== sanity test 398c: run fio to test AIO ================= 08:47:48 (1787834868) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.393 s, 107 MB/s osc.lustre-OST0000-osc-ffff8bad3117b000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180025: Thu Aug 27 08:48:21 2026 write: IOPS=107, BW=432KiB/s (442kB/s)(10.0MiB/23728msec); 0 zone resets slat (usec): min=29, max=14474, avg=187.31, stdev=572.88 clat (msec): min=17, max=250, avg=147.58, stdev=30.43 lat (msec): min=19, max=250, avg=147.76, stdev=30.43 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 108], 10.00th=[ 115], 20.00th=[ 124], | 30.00th=[ 131], 40.00th=[ 138], 50.00th=[ 144], 60.00th=[ 150], | 70.00th=[ 161], 80.00th=[ 171], 90.00th=[ 188], 95.00th=[ 201], | 99.00th=[ 232], 99.50th=[ 243], 99.90th=[ 251], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 269, max= 573, per=24.96%, avg=430.57, stdev=59.93, samples=47 iops : min= 67, max= 143, avg=107.36, stdev=14.95, samples=47 lat (msec) : 20=0.04%, 50=0.16%, 100=2.23%, 250=97.50%, 500=0.08% cpu : usr=0.20%, sys=0.94%, ctx=3024, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180026: Thu Aug 27 08:48:21 2026 write: IOPS=108, BW=434KiB/s (444kB/s)(10.0MiB/23614msec); 0 zone resets slat (usec): min=31, max=7958, avg=184.41, stdev=476.13 clat (msec): min=25, max=245, avg=147.28, stdev=31.07 lat (msec): min=25, max=245, avg=147.46, stdev=31.07 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 106], 10.00th=[ 115], 20.00th=[ 126], | 30.00th=[ 133], 40.00th=[ 140], 50.00th=[ 144], 60.00th=[ 150], | 70.00th=[ 161], 80.00th=[ 171], 90.00th=[ 188], 95.00th=[ 201], | 99.00th=[ 236], 99.50th=[ 241], 99.90th=[ 245], 99.95th=[ 245], | 99.99th=[ 247] bw ( KiB/s): min= 263, max= 696, per=25.08%, avg=432.70, stdev=70.26, samples=46 iops : min= 65, max= 174, avg=107.93, stdev=17.58, samples=46 lat (msec) : 50=0.27%, 100=3.05%, 250=96.68% cpu : usr=0.23%, sys=0.93%, ctx=2873, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180027: Thu Aug 27 08:48:21 2026 write: IOPS=107, BW=432KiB/s (442kB/s)(10.0MiB/23730msec); 0 zone resets slat (usec): min=28, max=8669, avg=186.62, stdev=445.56 clat (msec): min=21, max=256, avg=147.54, stdev=29.95 lat (msec): min=21, max=256, avg=147.73, stdev=29.95 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 105], 10.00th=[ 116], 20.00th=[ 125], | 30.00th=[ 132], 40.00th=[ 138], 50.00th=[ 144], 60.00th=[ 150], | 70.00th=[ 161], 80.00th=[ 171], 90.00th=[ 188], 95.00th=[ 203], | 99.00th=[ 230], 99.50th=[ 243], 99.90th=[ 257], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 285, max= 565, per=24.94%, avg=430.30, stdev=58.89, samples=47 iops : min= 71, max= 141, avg=107.28, stdev=14.69, samples=47 lat (msec) : 50=0.16%, 100=3.20%, 250=96.33%, 500=0.31% cpu : usr=0.18%, sys=1.03%, ctx=3181, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180028: Thu Aug 27 08:48:21 2026 write: IOPS=107, BW=431KiB/s (442kB/s)(10.0MiB/23733msec); 0 zone resets slat (usec): min=33, max=9478, avg=181.48, stdev=462.86 clat (msec): min=15, max=257, avg=147.44, stdev=30.58 lat (msec): min=16, max=257, avg=147.62, stdev=30.59 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 106], 10.00th=[ 115], 20.00th=[ 125], | 30.00th=[ 132], 40.00th=[ 138], 50.00th=[ 144], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 171], 90.00th=[ 190], 95.00th=[ 201], | 99.00th=[ 241], 99.50th=[ 243], 99.90th=[ 251], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 261, max= 558, per=24.93%, avg=429.98, stdev=58.14, samples=47 iops : min= 65, max= 139, avg=107.19, stdev=14.47, samples=47 lat (msec) : 20=0.04%, 50=0.04%, 100=3.09%, 250=96.72%, 500=0.12% cpu : usr=0.18%, sys=0.94%, ctx=3019, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1726KiB/s (1767kB/s), 431KiB/s-434KiB/s (442kB/s-444kB/s), io=40.0MiB (41.9MB), run=23614-23733msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180041: Thu Aug 27 08:48:50 2026 read: IOPS=54, BW=220KiB/s (225kB/s)(5048KiB/22988msec) slat (usec): min=2279, max=48274, avg=9852.52, stdev=5197.86 clat (msec): min=69, max=240, avg=135.94, stdev=28.23 lat (msec): min=78, max=255, avg=145.80, stdev=28.77 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 96], 10.00th=[ 104], 20.00th=[ 112], | 30.00th=[ 120], 40.00th=[ 127], 50.00th=[ 133], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 159], 90.00th=[ 174], 95.00th=[ 186], | 99.00th=[ 224], 99.50th=[ 230], 99.90th=[ 241], 99.95th=[ 241], | 99.99th=[ 241] bw ( KiB/s): min= 158, max= 360, per=25.01%, avg=218.11, stdev=37.75, samples=45 iops : min= 39, max= 90, avg=54.16, stdev= 9.47, samples=45 write: IOPS=56, BW=226KiB/s (231kB/s)(5192KiB/22988msec); 0 zone resets slat (usec): min=35, max=13733, avg=214.39, stdev=675.25 clat (msec): min=69, max=233, avg=139.38, stdev=26.15 lat (msec): min=69, max=233, avg=139.60, stdev=26.14 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 103], 10.00th=[ 109], 20.00th=[ 118], | 30.00th=[ 124], 40.00th=[ 130], 50.00th=[ 136], 60.00th=[ 144], | 70.00th=[ 150], 80.00th=[ 159], 90.00th=[ 176], 95.00th=[ 188], | 99.00th=[ 215], 99.50th=[ 224], 99.90th=[ 232], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 142, max= 302, per=24.83%, avg=225.42, stdev=40.68, samples=45 iops : min= 35, max= 75, avg=55.98, stdev=10.28, samples=45 lat (msec) : 100=5.82%, 250=94.18% cpu : usr=0.27%, sys=1.99%, ctx=4876, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180042: Thu Aug 27 08:48:50 2026 read: IOPS=54, BW=218KiB/s (224kB/s)(4976KiB/22790msec) slat (usec): min=2315, max=50806, avg=9732.92, stdev=4871.62 clat (msec): min=71, max=272, avg=135.14, stdev=26.63 lat (msec): min=78, max=286, avg=144.88, stdev=27.42 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 99], 10.00th=[ 105], 20.00th=[ 113], | 30.00th=[ 122], 40.00th=[ 127], 50.00th=[ 133], 60.00th=[ 140], | 70.00th=[ 146], 80.00th=[ 155], 90.00th=[ 171], 95.00th=[ 186], | 99.00th=[ 215], 99.50th=[ 220], 99.90th=[ 247], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 151, max= 278, per=24.97%, avg=217.78, stdev=35.91, samples=45 iops : min= 37, max= 69, avg=54.07, stdev= 9.02, samples=45 write: IOPS=57, BW=231KiB/s (237kB/s)(5264KiB/22790msec); 0 zone resets slat (usec): min=33, max=13788, avg=202.80, stdev=615.51 clat (msec): min=43, max=250, avg=138.14, stdev=24.25 lat (msec): min=43, max=250, avg=138.35, stdev=24.25 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 103], 10.00th=[ 110], 20.00th=[ 118], | 30.00th=[ 126], 40.00th=[ 130], 50.00th=[ 136], 60.00th=[ 142], | 70.00th=[ 150], 80.00th=[ 157], 90.00th=[ 169], 95.00th=[ 182], | 99.00th=[ 203], 99.50th=[ 215], 99.90th=[ 247], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 157, max= 368, per=25.48%, avg=231.33, stdev=42.67, samples=45 iops : min= 39, max= 92, avg=57.47, stdev=10.72, samples=45 lat (msec) : 50=0.04%, 100=5.27%, 250=94.61%, 500=0.08% cpu : usr=0.25%, sys=2.06%, ctx=4493, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180043: Thu Aug 27 08:48:50 2026 read: IOPS=54, BW=217KiB/s (222kB/s)(4984KiB/22987msec) slat (usec): min=2221, max=43564, avg=10009.73, stdev=4915.33 clat (msec): min=10, max=238, avg=134.77, stdev=27.05 lat (msec): min=21, max=245, avg=144.78, stdev=27.61 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 95], 10.00th=[ 103], 20.00th=[ 113], | 30.00th=[ 121], 40.00th=[ 126], 50.00th=[ 131], 60.00th=[ 138], | 70.00th=[ 148], 80.00th=[ 159], 90.00th=[ 171], 95.00th=[ 184], | 99.00th=[ 209], 99.50th=[ 226], 99.90th=[ 232], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 130, max= 360, per=24.80%, avg=216.24, stdev=43.71, samples=45 iops : min= 32, max= 90, avg=53.69, stdev=11.00, samples=45 write: IOPS=57, BW=229KiB/s (234kB/s)(5256KiB/22987msec); 0 zone resets slat (usec): min=35, max=10005, avg=202.15, stdev=492.49 clat (msec): min=74, max=251, avg=140.20, stdev=25.80 lat (msec): min=74, max=251, avg=140.40, stdev=25.81 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 103], 10.00th=[ 110], 20.00th=[ 120], | 30.00th=[ 125], 40.00th=[ 131], 50.00th=[ 138], 60.00th=[ 144], | 70.00th=[ 153], 80.00th=[ 163], 90.00th=[ 176], 95.00th=[ 188], | 99.00th=[ 215], 99.50th=[ 226], 99.90th=[ 232], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 143, max= 319, per=24.96%, avg=226.67, stdev=43.76, samples=45 iops : min= 35, max= 79, avg=56.29, stdev=10.97, samples=45 lat (msec) : 20=0.04%, 100=5.86%, 250=94.06%, 500=0.04% cpu : usr=0.28%, sys=2.02%, ctx=4726, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180044: Thu Aug 27 08:48:50 2026 read: IOPS=55, BW=222KiB/s (227kB/s)(5060KiB/22791msec) slat (usec): min=53, max=40251, avg=9720.09, stdev=5009.03 clat (msec): min=11, max=239, avg=135.15, stdev=28.11 lat (msec): min=18, max=249, avg=144.88, stdev=28.71 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 96], 10.00th=[ 103], 20.00th=[ 113], | 30.00th=[ 121], 40.00th=[ 127], 50.00th=[ 133], 60.00th=[ 140], | 70.00th=[ 146], 80.00th=[ 157], 90.00th=[ 174], 95.00th=[ 186], | 99.00th=[ 213], 99.50th=[ 220], 99.90th=[ 232], 99.95th=[ 241], | 99.99th=[ 241] bw ( KiB/s): min= 134, max= 320, per=25.42%, avg=221.64, stdev=38.34, samples=45 iops : min= 33, max= 80, avg=55.04, stdev= 9.68, samples=45 write: IOPS=56, BW=227KiB/s (233kB/s)(5180KiB/22791msec); 0 zone resets slat (usec): min=34, max=9162, avg=210.32, stdev=547.36 clat (msec): min=22, max=252, avg=139.43, stdev=26.26 lat (msec): min=22, max=252, avg=139.65, stdev=26.27 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 101], 10.00th=[ 110], 20.00th=[ 118], | 30.00th=[ 125], 40.00th=[ 131], 50.00th=[ 138], 60.00th=[ 144], | 70.00th=[ 153], 80.00th=[ 161], 90.00th=[ 176], 95.00th=[ 184], | 99.00th=[ 211], 99.50th=[ 222], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 126, max= 296, per=25.00%, avg=226.98, stdev=38.35, samples=45 iops : min= 31, max= 74, avg=56.36, stdev= 9.61, samples=45 lat (msec) : 20=0.16%, 50=0.16%, 100=5.90%, 250=93.71%, 500=0.08% cpu : usr=0.24%, sys=1.96%, ctx=4736, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=873KiB/s (894kB/s), 217KiB/s-222KiB/s (222kB/s-227kB/s), io=19.6MiB (20.5MB), run=22790-22988msec WRITE: bw=909KiB/s (931kB/s), 226KiB/s-231KiB/s (231kB/s-237kB/s), io=20.4MiB (21.4MB), run=22790-22988msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180051: Thu Aug 27 08:48:57 2026 read: IOPS=3, BW=126MiB/s (132MB/s)(40.0MiB/317msec) slat (nsec): min=35060k, max=35060k, avg=35060438.00, stdev= 0.00 clat (nsec): min=256362k, max=256362k, avg=256362400.00, stdev= 0.00 lat (nsec): min=291486k, max=291486k, avg=291485945.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 257], 5.00th=[ 257], 10.00th=[ 257], 20.00th=[ 257], | 30.00th=[ 257], 40.00th=[ 257], 50.00th=[ 257], 60.00th=[ 257], | 70.00th=[ 257], 80.00th=[ 257], 90.00th=[ 257], 95.00th=[ 257], | 99.00th=[ 257], 99.50th=[ 257], 99.90th=[ 257], 99.95th=[ 257], | 99.99th=[ 257] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.80%, ctx=9, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=40.0MiB (41.9MB), run=317-317msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout