== sanity test 398c: run fio to test AIO ================= 21:15:13 (1786410913) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.707455 s, 59.3 MB/s osc.lustre-OST0000-osc-ffff89e5c61d6800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144963: Mon Aug 10 21:18:06 2026 write: IOPS=15, BW=63.6KiB/s (65.1kB/s)(10.0MiB/161054msec); 0 zone resets slat (usec): min=36, max=16476, avg=221.83, stdev=666.43 clat (msec): min=297, max=2107, avg=1005.47, stdev=284.37 lat (msec): min=298, max=2107, avg=1005.70, stdev=284.41 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 567], 10.00th=[ 693], 20.00th=[ 793], | 30.00th=[ 860], 40.00th=[ 911], 50.00th=[ 969], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1385], 95.00th=[ 1519], | 99.00th=[ 1838], 99.50th=[ 1955], 99.90th=[ 2089], 99.95th=[ 2089], | 99.99th=[ 2106] bw ( KiB/s): min= 7, max= 128, per=24.96%, avg=63.39, stdev=28.26, samples=318 iops : min= 1, max= 32, avg=15.40, stdev= 7.09, samples=318 lat (msec) : 500=3.91%, 750=10.16%, 1000=40.62%, 2000=45.04%, >=2000=0.27% cpu : usr=0.03%, sys=0.26%, ctx=1846, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144964: Mon Aug 10 21:18:06 2026 write: IOPS=15, BW=63.8KiB/s (65.3kB/s)(10.0MiB/160581msec); 0 zone resets slat (usec): min=35, max=20416, avg=236.43, stdev=744.80 clat (msec): min=143, max=2074, avg=1003.02, stdev=290.85 lat (msec): min=143, max=2074, avg=1003.26, stdev=290.88 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 542], 10.00th=[ 693], 20.00th=[ 802], | 30.00th=[ 860], 40.00th=[ 919], 50.00th=[ 978], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1200], 90.00th=[ 1385], 95.00th=[ 1536], | 99.00th=[ 1905], 99.50th=[ 1938], 99.90th=[ 2005], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 254, per=25.77%, avg=65.46, stdev=32.84, samples=308 iops : min= 1, max= 63, avg=15.93, stdev= 8.25, samples=308 lat (msec) : 250=0.62%, 500=3.91%, 750=9.34%, 1000=39.84%, 2000=46.02% lat (msec) : >=2000=0.27% cpu : usr=0.05%, sys=0.25%, ctx=1963, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144965: Mon Aug 10 21:18:06 2026 write: IOPS=15, BW=63.7KiB/s (65.2kB/s)(10.0MiB/160780msec); 0 zone resets slat (usec): min=36, max=15369, avg=211.90, stdev=682.94 clat (msec): min=216, max=1987, avg=1004.30, stdev=289.04 lat (msec): min=216, max=1987, avg=1004.52, stdev=289.06 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 523], 10.00th=[ 684], 20.00th=[ 802], | 30.00th=[ 852], 40.00th=[ 927], 50.00th=[ 978], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1368], 95.00th=[ 1502], | 99.00th=[ 1871], 99.50th=[ 1938], 99.90th=[ 1989], 99.95th=[ 1989], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 240, per=25.20%, avg=64.00, stdev=31.87, samples=315 iops : min= 1, max= 60, avg=15.56, stdev= 7.99, samples=315 lat (msec) : 250=0.62%, 500=3.40%, 750=10.51%, 1000=39.96%, 2000=45.51% cpu : usr=0.04%, sys=0.25%, ctx=1707, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144966: Mon Aug 10 21:18:06 2026 write: IOPS=15, BW=63.6KiB/s (65.1kB/s)(10.0MiB/161055msec); 0 zone resets slat (usec): min=37, max=18903, avg=248.31, stdev=852.44 clat (msec): min=202, max=2108, avg=1005.64, stdev=286.82 lat (msec): min=202, max=2108, avg=1005.89, stdev=286.82 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 575], 10.00th=[ 684], 20.00th=[ 793], | 30.00th=[ 860], 40.00th=[ 911], 50.00th=[ 978], 60.00th=[ 1045], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1385], 95.00th=[ 1519], | 99.00th=[ 1905], 99.50th=[ 1972], 99.90th=[ 2106], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 7, max= 128, per=24.88%, avg=63.19, stdev=26.81, samples=319 iops : min= 1, max= 32, avg=15.35, stdev= 6.69, samples=319 lat (msec) : 250=0.08%, 500=3.32%, 750=11.64%, 1000=38.75%, 2000=45.86% lat (msec) : >=2000=0.35% cpu : usr=0.04%, sys=0.25%, ctx=2204, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=254KiB/s (260kB/s), 63.6KiB/s-63.8KiB/s (65.1kB/s-65.3kB/s), io=40.0MiB (41.9MB), run=160581-161055msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144998: Mon Aug 10 21:20:12 2026 read: IOPS=10, BW=43.4KiB/s (44.4kB/s)(5048KiB/116321msec) slat (usec): min=2677, max=54889, avg=15022.32, stdev=8306.10 clat (msec): min=28, max=1431, avg=445.79, stdev=151.61 lat (msec): min=45, max=1436, avg=460.81, stdev=153.26 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 264], 10.00th=[ 292], 20.00th=[ 342], | 30.00th=[ 368], 40.00th=[ 393], 50.00th=[ 418], 60.00th=[ 447], | 70.00th=[ 489], 80.00th=[ 542], 90.00th=[ 651], 95.00th=[ 726], | 99.00th=[ 953], 99.50th=[ 1045], 99.90th=[ 1334], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 7, max= 96, per=25.33%, avg=43.32, stdev=19.53, samples=228 iops : min= 1, max= 24, avg=10.34, stdev= 4.92, samples=228 write: IOPS=11, BW=44.6KiB/s (45.7kB/s)(5192KiB/116321msec); 0 zone resets slat (usec): min=54, max=19350, avg=338.03, stdev=1076.55 clat (msec): min=251, max=2202, avg=982.76, stdev=317.01 lat (msec): min=252, max=2202, avg=983.10, stdev=316.98 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 550], 10.00th=[ 642], 20.00th=[ 743], | 30.00th=[ 810], 40.00th=[ 869], 50.00th=[ 919], 60.00th=[ 995], | 70.00th=[ 1083], 80.00th=[ 1200], 90.00th=[ 1401], 95.00th=[ 1603], | 99.00th=[ 2039], 99.50th=[ 2140], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 103, per=25.03%, avg=44.55, stdev=20.38, samples=228 iops : min= 1, max= 25, avg=10.64, stdev= 5.13, samples=228 lat (msec) : 50=0.16%, 100=0.12%, 250=1.76%, 500=35.27%, 750=20.43% lat (msec) : 1000=22.30%, 2000=19.34%, >=2000=0.62% cpu : usr=0.10%, sys=0.68%, ctx=4743, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144999: Mon Aug 10 21:20:12 2026 read: IOPS=10, BW=42.6KiB/s (43.6kB/s)(4976KiB/116827msec) slat (usec): min=61, max=57589, avg=14929.86, stdev=8748.57 clat (msec): min=38, max=955, avg=436.19, stdev=133.99 lat (msec): min=44, max=979, avg=451.12, stdev=136.28 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 262], 10.00th=[ 296], 20.00th=[ 334], | 30.00th=[ 359], 40.00th=[ 393], 50.00th=[ 418], 60.00th=[ 443], | 70.00th=[ 481], 80.00th=[ 531], 90.00th=[ 625], 95.00th=[ 701], | 99.00th=[ 827], 99.50th=[ 885], 99.90th=[ 927], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 7, max= 119, per=24.97%, avg=42.69, stdev=22.41, samples=228 iops : min= 1, max= 29, avg=10.18, stdev= 5.64, samples=228 write: IOPS=11, BW=45.1KiB/s (46.1kB/s)(5264KiB/116827msec); 0 zone resets slat (usec): min=54, max=14872, avg=306.48, stdev=876.64 clat (msec): min=181, max=2353, avg=993.16, stdev=342.43 lat (msec): min=181, max=2353, avg=993.47, stdev=342.40 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 558], 10.00th=[ 634], 20.00th=[ 743], | 30.00th=[ 818], 40.00th=[ 869], 50.00th=[ 927], 60.00th=[ 995], | 70.00th=[ 1099], 80.00th=[ 1217], 90.00th=[ 1435], 95.00th=[ 1670], | 99.00th=[ 2165], 99.50th=[ 2265], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 97, per=25.13%, avg=44.73, stdev=20.60, samples=230 iops : min= 1, max= 24, avg=10.69, stdev= 5.18, samples=230 lat (msec) : 50=0.04%, 100=0.27%, 250=2.07%, 500=35.04%, 750=20.39% lat (msec) : 1000=21.76%, 2000=19.53%, >=2000=0.90% cpu : usr=0.09%, sys=0.59%, ctx=4322, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145000: Mon Aug 10 21:20:12 2026 read: IOPS=10, BW=42.7KiB/s (43.7kB/s)(4984KiB/116788msec) slat (usec): min=2391, max=68047, avg=14676.38, stdev=8502.40 clat (msec): min=25, max=1033, avg=439.48, stdev=137.02 lat (msec): min=36, max=1043, avg=454.17, stdev=138.57 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 266], 10.00th=[ 292], 20.00th=[ 334], | 30.00th=[ 363], 40.00th=[ 393], 50.00th=[ 418], 60.00th=[ 447], | 70.00th=[ 481], 80.00th=[ 531], 90.00th=[ 625], 95.00th=[ 709], | 99.00th=[ 844], 99.50th=[ 885], 99.90th=[ 1003], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 7, max= 111, per=25.19%, avg=43.07, stdev=21.79, samples=227 iops : min= 1, max= 27, avg=10.28, stdev= 5.47, samples=227 write: IOPS=11, BW=45.0KiB/s (46.1kB/s)(5256KiB/116788msec); 0 zone resets slat (usec): min=46, max=14462, avg=303.56, stdev=791.64 clat (msec): min=129, max=2388, avg=988.61, stdev=334.79 lat (msec): min=129, max=2388, avg=988.91, stdev=334.75 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 542], 10.00th=[ 642], 20.00th=[ 743], | 30.00th=[ 810], 40.00th=[ 869], 50.00th=[ 927], 60.00th=[ 995], | 70.00th=[ 1083], 80.00th=[ 1200], 90.00th=[ 1401], 95.00th=[ 1620], | 99.00th=[ 2198], 99.50th=[ 2299], 99.90th=[ 2366], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 95, per=24.84%, avg=44.21, stdev=20.11, samples=232 iops : min= 1, max= 23, avg=10.56, stdev= 5.05, samples=232 lat (msec) : 50=0.20%, 100=0.08%, 250=1.52%, 500=36.02%, 750=20.04% lat (msec) : 1000=21.60%, 2000=19.69%, >=2000=0.86% cpu : usr=0.08%, sys=0.65%, ctx=4615, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145001: Mon Aug 10 21:20:12 2026 read: IOPS=10, BW=43.5KiB/s (44.5kB/s)(5060KiB/116318msec) slat (usec): min=3144, max=64252, avg=14824.15, stdev=8576.03 clat (msec): min=43, max=1266, avg=440.04, stdev=145.27 lat (msec): min=55, max=1278, avg=454.87, stdev=147.13 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 259], 10.00th=[ 288], 20.00th=[ 330], | 30.00th=[ 359], 40.00th=[ 388], 50.00th=[ 418], 60.00th=[ 451], | 70.00th=[ 485], 80.00th=[ 535], 90.00th=[ 634], 95.00th=[ 718], | 99.00th=[ 902], 99.50th=[ 944], 99.90th=[ 1250], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 6, max= 119, per=25.89%, avg=44.27, stdev=20.24, samples=224 iops : min= 1, max= 29, avg=10.58, stdev= 5.09, samples=224 write: IOPS=11, BW=44.5KiB/s (45.6kB/s)(5180KiB/116318msec); 0 zone resets slat (usec): min=39, max=23338, avg=323.99, stdev=992.90 clat (msec): min=376, max=2364, avg=989.19, stdev=324.35 lat (msec): min=376, max=2364, avg=989.52, stdev=324.35 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 550], 10.00th=[ 642], 20.00th=[ 751], | 30.00th=[ 810], 40.00th=[ 869], 50.00th=[ 936], 60.00th=[ 995], | 70.00th=[ 1083], 80.00th=[ 1200], 90.00th=[ 1401], 95.00th=[ 1636], | 99.00th=[ 2123], 99.50th=[ 2198], 99.90th=[ 2333], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 103, per=24.96%, avg=44.43, stdev=18.85, samples=228 iops : min= 1, max= 25, avg=10.61, stdev= 4.71, samples=228 lat (msec) : 50=0.12%, 100=0.16%, 250=1.76%, 500=35.27%, 750=20.43% lat (msec) : 1000=22.07%, 2000=19.45%, >=2000=0.74% cpu : usr=0.11%, sys=0.70%, ctx=4587, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=172KiB/s (176kB/s), 42.6KiB/s-43.5KiB/s (43.6kB/s-44.5kB/s), io=19.6MiB (20.5MB), run=116318-116827msec WRITE: bw=179KiB/s (183kB/s), 44.5KiB/s-45.1KiB/s (45.6kB/s-46.1kB/s), io=20.4MiB (21.4MB), run=116318-116827msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145022: Mon Aug 10 21:20:23 2026 read: IOPS=1, BW=68.3MiB/s (71.6MB/s)(40.0MiB/586msec) slat (nsec): min=26493k, max=26493k, avg=26492592.00, stdev= 0.00 clat (nsec): min=542768k, max=542768k, avg=542767681.00, stdev= 0.00 lat (nsec): min=569264k, max=569264k, avg=569264378.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 542], 10.00th=[ 542], 20.00th=[ 542], | 30.00th=[ 542], 40.00th=[ 542], 50.00th=[ 542], 60.00th=[ 542], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.37%, ctx=6, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=68.3MiB/s (71.6MB/s), 68.3MiB/s-68.3MiB/s (71.6MB/s-71.6MB/s), io=40.0MiB (41.9MB), run=586-586msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout