== sanity test 398c: run fio to test AIO ================= 02:04:36 (1786946676) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.160628 s, 261 MB/s osc.lustre-OST0000-osc-ffff8af2f25d9800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=184322: Mon Aug 17 02:04:57 2026 write: IOPS=156, BW=625KiB/s (640kB/s)(10.0MiB/16393msec); 0 zone resets slat (usec): min=34, max=9421, avg=210.45, stdev=433.29 clat (msec): min=14, max=201, avg=101.84, stdev=25.26 lat (msec): min=14, max=201, avg=102.05, stdev=25.28 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 59], 10.00th=[ 71], 20.00th=[ 83], | 30.00th=[ 90], 40.00th=[ 96], 50.00th=[ 103], 60.00th=[ 108], | 70.00th=[ 114], 80.00th=[ 123], 90.00th=[ 134], 95.00th=[ 142], | 99.00th=[ 159], 99.50th=[ 197], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 201] bw ( KiB/s): min= 451, max= 957, per=24.95%, avg=622.94, stdev=115.68, samples=32 iops : min= 112, max= 239, avg=155.41, stdev=28.94, samples=32 lat (msec) : 20=0.04%, 50=2.70%, 100=43.59%, 250=53.67% cpu : usr=0.18%, sys=1.84%, ctx=2618, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184323: Mon Aug 17 02:04:57 2026 write: IOPS=157, BW=630KiB/s (645kB/s)(10.0MiB/16261msec); 0 zone resets slat (usec): min=34, max=7183, avg=237.44, stdev=387.88 clat (msec): min=8, max=219, avg=101.24, stdev=26.06 lat (msec): min=8, max=219, avg=101.48, stdev=26.06 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 56], 10.00th=[ 69], 20.00th=[ 82], | 30.00th=[ 89], 40.00th=[ 95], 50.00th=[ 102], 60.00th=[ 109], | 70.00th=[ 115], 80.00th=[ 123], 90.00th=[ 133], 95.00th=[ 144], | 99.00th=[ 161], 99.50th=[ 169], 99.90th=[ 201], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 423, max= 1107, per=25.07%, avg=626.09, stdev=137.27, samples=32 iops : min= 105, max= 276, avg=156.16, stdev=34.29, samples=32 lat (msec) : 10=0.04%, 20=0.12%, 50=3.01%, 100=44.73%, 250=52.11% cpu : usr=0.25%, sys=2.29%, ctx=3060, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184324: Mon Aug 17 02:04:57 2026 write: IOPS=156, BW=625KiB/s (640kB/s)(10.0MiB/16392msec); 0 zone resets slat (usec): min=41, max=6076, avg=207.02, stdev=353.94 clat (msec): min=9, max=199, avg=101.86, stdev=24.89 lat (msec): min=9, max=199, avg=102.07, stdev=24.90 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 59], 10.00th=[ 71], 20.00th=[ 83], | 30.00th=[ 90], 40.00th=[ 96], 50.00th=[ 102], 60.00th=[ 109], | 70.00th=[ 115], 80.00th=[ 123], 90.00th=[ 133], 95.00th=[ 142], | 99.00th=[ 159], 99.50th=[ 176], 99.90th=[ 184], 99.95th=[ 197], | 99.99th=[ 201] bw ( KiB/s): min= 487, max= 938, per=24.88%, avg=621.25, stdev=114.94, samples=32 iops : min= 121, max= 234, avg=155.00, stdev=28.73, samples=32 lat (msec) : 10=0.04%, 20=0.12%, 50=2.03%, 100=44.96%, 250=52.85% cpu : usr=0.26%, sys=1.70%, ctx=2798, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184325: Mon Aug 17 02:04:57 2026 write: IOPS=156, BW=624KiB/s (639kB/s)(10.0MiB/16403msec); 0 zone resets slat (usec): min=35, max=9543, avg=233.30, stdev=515.27 clat (msec): min=16, max=201, avg=101.96, stdev=25.37 lat (msec): min=17, max=201, avg=102.20, stdev=25.40 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 58], 10.00th=[ 73], 20.00th=[ 81], | 30.00th=[ 90], 40.00th=[ 96], 50.00th=[ 103], 60.00th=[ 108], | 70.00th=[ 115], 80.00th=[ 123], 90.00th=[ 133], 95.00th=[ 142], | 99.00th=[ 167], 99.50th=[ 180], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 403, max= 1040, per=24.84%, avg=620.37, stdev=132.14, samples=32 iops : min= 100, max= 260, avg=154.81, stdev=33.10, samples=32 lat (msec) : 20=0.08%, 50=2.54%, 100=43.32%, 250=54.06% cpu : usr=0.30%, sys=1.82%, ctx=2746, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2497KiB/s (2557kB/s), 624KiB/s-630KiB/s (639kB/s-645kB/s), io=40.0MiB (41.9MB), run=16261-16403msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=184337: Mon Aug 17 02:05:19 2026 read: IOPS=77, BW=311KiB/s (318kB/s)(5048KiB/16243msec) slat (usec): min=1998, max=27279, avg=7193.43, stdev=3532.63 clat (msec): min=6, max=162, avg=95.30, stdev=20.94 lat (msec): min=13, max=179, avg=102.50, stdev=21.78 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 66], 10.00th=[ 70], 20.00th=[ 78], | 30.00th=[ 84], 40.00th=[ 90], 50.00th=[ 94], 60.00th=[ 99], | 70.00th=[ 105], 80.00th=[ 112], 90.00th=[ 123], 95.00th=[ 134], | 99.00th=[ 155], 99.50th=[ 157], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 201, max= 432, per=25.13%, avg=310.31, stdev=48.53, samples=32 iops : min= 50, max= 108, avg=77.22, stdev=12.15, samples=32 write: IOPS=79, BW=320KiB/s (327kB/s)(5192KiB/16243msec); 0 zone resets slat (usec): min=41, max=7619, avg=232.34, stdev=423.83 clat (msec): min=52, max=174, avg=98.58, stdev=19.34 lat (msec): min=52, max=175, avg=98.81, stdev=19.34 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 72], 10.00th=[ 77], 20.00th=[ 83], | 30.00th=[ 87], 40.00th=[ 92], 50.00th=[ 96], 60.00th=[ 101], | 70.00th=[ 107], 80.00th=[ 115], 90.00th=[ 125], 95.00th=[ 136], | 99.00th=[ 155], 99.50th=[ 163], 99.90th=[ 171], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 215, max= 399, per=24.88%, avg=319.94, stdev=48.22, samples=32 iops : min= 53, max= 99, avg=79.66, stdev=12.03, samples=32 lat (msec) : 10=0.04%, 50=0.16%, 100=61.02%, 250=38.79% cpu : usr=0.20%, sys=2.78%, ctx=4847, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184338: Mon Aug 17 02:05:19 2026 read: IOPS=76, BW=307KiB/s (314kB/s)(4976KiB/16219msec) slat (usec): min=2028, max=32549, avg=7116.38, stdev=3580.32 clat (msec): min=14, max=195, avg=94.99, stdev=21.17 lat (msec): min=21, max=210, avg=102.11, stdev=22.06 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 66], 10.00th=[ 70], 20.00th=[ 78], | 30.00th=[ 84], 40.00th=[ 89], 50.00th=[ 93], 60.00th=[ 99], | 70.00th=[ 105], 80.00th=[ 111], 90.00th=[ 123], 95.00th=[ 134], | 99.00th=[ 157], 99.50th=[ 165], 99.90th=[ 182], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 199, max= 400, per=24.77%, avg=305.97, stdev=54.31, samples=32 iops : min= 49, max= 100, avg=76.16, stdev=13.60, samples=32 write: IOPS=81, BW=325KiB/s (332kB/s)(5264KiB/16219msec); 0 zone resets slat (usec): min=44, max=7312, avg=244.38, stdev=443.79 clat (msec): min=26, max=180, avg=99.07, stdev=21.46 lat (msec): min=27, max=180, avg=99.32, stdev=21.45 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 69], 10.00th=[ 74], 20.00th=[ 82], | 30.00th=[ 87], 40.00th=[ 92], 50.00th=[ 97], 60.00th=[ 102], | 70.00th=[ 108], 80.00th=[ 115], 90.00th=[ 127], 95.00th=[ 140], | 99.00th=[ 163], 99.50th=[ 174], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 232, max= 432, per=25.25%, avg=324.72, stdev=50.86, samples=32 iops : min= 58, max= 108, avg=80.84, stdev=12.70, samples=32 lat (msec) : 20=0.08%, 50=0.23%, 100=59.57%, 250=40.12% cpu : usr=0.31%, sys=2.94%, ctx=4813, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184339: Mon Aug 17 02:05:19 2026 read: IOPS=76, BW=307KiB/s (314kB/s)(4984KiB/16245msec) slat (usec): min=1605, max=26620, avg=7042.49, stdev=3479.05 clat (msec): min=19, max=183, avg=95.13, stdev=20.36 lat (msec): min=24, max=190, avg=102.17, stdev=21.25 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 65], 10.00th=[ 71], 20.00th=[ 80], | 30.00th=[ 85], 40.00th=[ 89], 50.00th=[ 94], 60.00th=[ 99], | 70.00th=[ 104], 80.00th=[ 110], 90.00th=[ 122], 95.00th=[ 136], | 99.00th=[ 157], 99.50th=[ 163], 99.90th=[ 176], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 188, max= 391, per=24.85%, avg=306.94, stdev=54.68, samples=32 iops : min= 47, max= 97, avg=76.41, stdev=13.67, samples=32 write: IOPS=80, BW=324KiB/s (331kB/s)(5256KiB/16245msec); 0 zone resets slat (usec): min=39, max=7929, avg=229.63, stdev=401.05 clat (msec): min=49, max=174, avg=98.89, stdev=20.63 lat (msec): min=50, max=174, avg=99.12, stdev=20.67 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 70], 10.00th=[ 75], 20.00th=[ 82], | 30.00th=[ 86], 40.00th=[ 92], 50.00th=[ 96], 60.00th=[ 103], | 70.00th=[ 109], 80.00th=[ 115], 90.00th=[ 126], 95.00th=[ 138], | 99.00th=[ 157], 99.50th=[ 163], 99.90th=[ 169], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 231, max= 447, per=25.07%, avg=322.41, stdev=52.18, samples=32 iops : min= 57, max= 111, avg=80.28, stdev=13.05, samples=32 lat (msec) : 20=0.04%, 50=0.08%, 100=59.96%, 250=39.92% cpu : usr=0.25%, sys=2.84%, ctx=4802, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184340: Mon Aug 17 02:05:19 2026 read: IOPS=78, BW=313KiB/s (320kB/s)(5060KiB/16173msec) slat (usec): min=58, max=26493, avg=7141.70, stdev=3594.25 clat (msec): min=16, max=179, avg=95.67, stdev=22.00 lat (msec): min=21, max=192, avg=102.81, stdev=22.79 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 66], 10.00th=[ 71], 20.00th=[ 79], | 30.00th=[ 84], 40.00th=[ 88], 50.00th=[ 93], 60.00th=[ 97], | 70.00th=[ 104], 80.00th=[ 113], 90.00th=[ 125], 95.00th=[ 140], | 99.00th=[ 159], 99.50th=[ 165], 99.90th=[ 169], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 239, max= 424, per=25.21%, avg=311.31, stdev=45.89, samples=32 iops : min= 59, max= 106, avg=77.50, stdev=11.49, samples=32 write: IOPS=80, BW=320KiB/s (328kB/s)(5180KiB/16173msec); 0 zone resets slat (usec): min=40, max=7925, avg=255.80, stdev=513.94 clat (msec): min=44, max=188, avg=98.75, stdev=21.22 lat (msec): min=47, max=188, avg=99.01, stdev=21.23 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 69], 10.00th=[ 75], 20.00th=[ 82], | 30.00th=[ 87], 40.00th=[ 91], 50.00th=[ 96], 60.00th=[ 102], | 70.00th=[ 107], 80.00th=[ 114], 90.00th=[ 126], 95.00th=[ 140], | 99.00th=[ 165], 99.50th=[ 171], 99.90th=[ 180], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 223, max= 429, per=24.77%, avg=318.59, stdev=53.94, samples=32 iops : min= 55, max= 107, avg=79.28, stdev=13.40, samples=32 lat (msec) : 20=0.08%, 50=0.23%, 100=60.62%, 250=39.06% cpu : usr=0.32%, sys=2.88%, ctx=4693, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1235KiB/s (1265kB/s), 307KiB/s-313KiB/s (314kB/s-320kB/s), io=19.6MiB (20.5MB), run=16173-16245msec WRITE: bw=1286KiB/s (1317kB/s), 320KiB/s-325KiB/s (327kB/s-332kB/s), io=20.4MiB (21.4MB), run=16173-16245msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=184346: Mon Aug 17 02:05:25 2026 read: IOPS=4, BW=163MiB/s (171MB/s)(40.0MiB/246msec) slat (nsec): min=26610k, max=26610k, avg=26610465.00, stdev= 0.00 clat (nsec): min=210170k, max=210170k, avg=210170496.00, stdev= 0.00 lat (nsec): min=236785k, max=236785k, avg=236784819.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 211], 10.00th=[ 211], 20.00th=[ 211], | 30.00th=[ 211], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 211], | 70.00th=[ 211], 80.00th=[ 211], 90.00th=[ 211], 95.00th=[ 211], | 99.00th=[ 211], 99.50th=[ 211], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.90%, ctx=5, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=163MiB/s (171MB/s), 163MiB/s-163MiB/s (171MB/s-171MB/s), io=40.0MiB (41.9MB), run=246-246msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout