== sanity test 398c: run fio to test AIO ================= 01:10:31 (1786943431) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.397441 s, 106 MB/s osc.lustre-OST0000-osc-ffff998b6a3fb800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=189098: Mon Aug 17 01:11:03 2026 write: IOPS=105, BW=421KiB/s (432kB/s)(10.0MiB/24300msec); 0 zone resets slat (usec): min=50, max=11612, avg=286.62, stdev=630.57 clat (msec): min=48, max=301, avg=151.11, stdev=36.13 lat (msec): min=49, max=301, avg=151.40, stdev=36.14 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 105], 10.00th=[ 112], 20.00th=[ 124], | 30.00th=[ 130], 40.00th=[ 136], 50.00th=[ 144], 60.00th=[ 155], | 70.00th=[ 167], 80.00th=[ 182], 90.00th=[ 201], 95.00th=[ 213], | 99.00th=[ 251], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 300] bw ( KiB/s): min= 247, max= 598, per=24.95%, avg=420.12, stdev=81.69, samples=48 iops : min= 61, max= 149, avg=104.60, stdev=20.51, samples=48 lat (msec) : 50=0.47%, 100=3.36%, 250=95.12%, 500=1.05% cpu : usr=0.27%, sys=1.23%, ctx=2252, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=189099: Mon Aug 17 01:11:03 2026 write: IOPS=105, BW=421KiB/s (431kB/s)(10.0MiB/24310msec); 0 zone resets slat (usec): min=48, max=11633, avg=258.84, stdev=604.81 clat (msec): min=49, max=294, avg=151.10, stdev=36.22 lat (msec): min=49, max=294, avg=151.36, stdev=36.20 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 101], 10.00th=[ 111], 20.00th=[ 123], | 30.00th=[ 130], 40.00th=[ 138], 50.00th=[ 144], 60.00th=[ 155], | 70.00th=[ 165], 80.00th=[ 182], 90.00th=[ 205], 95.00th=[ 215], | 99.00th=[ 247], 99.50th=[ 279], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 252, max= 574, per=24.87%, avg=418.81, stdev=89.44, samples=48 iops : min= 63, max= 143, avg=104.31, stdev=22.41, samples=48 lat (msec) : 50=0.16%, 100=4.88%, 250=94.14%, 500=0.82% cpu : usr=0.15%, sys=1.20%, ctx=2046, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=189100: Mon Aug 17 01:11:03 2026 write: IOPS=105, BW=423KiB/s (433kB/s)(10.0MiB/24211msec); 0 zone resets slat (usec): min=51, max=13165, avg=262.85, stdev=633.54 clat (msec): min=26, max=297, avg=150.92, stdev=36.61 lat (msec): min=26, max=299, avg=151.18, stdev=36.64 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 101], 10.00th=[ 110], 20.00th=[ 123], | 30.00th=[ 129], 40.00th=[ 136], 50.00th=[ 146], 60.00th=[ 155], | 70.00th=[ 167], 80.00th=[ 182], 90.00th=[ 203], 95.00th=[ 211], | 99.00th=[ 259], 99.50th=[ 279], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 247, max= 637, per=24.97%, avg=420.52, stdev=89.67, samples=48 iops : min= 61, max= 159, avg=104.69, stdev=22.49, samples=48 lat (msec) : 50=0.12%, 100=4.80%, 250=93.87%, 500=1.21% cpu : usr=0.16%, sys=1.33%, ctx=1837, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=189101: Mon Aug 17 01:11:03 2026 write: IOPS=105, BW=421KiB/s (431kB/s)(10.0MiB/24303msec); 0 zone resets slat (usec): min=51, max=11837, avg=316.73, stdev=789.09 clat (msec): min=24, max=303, avg=151.03, stdev=35.87 lat (msec): min=26, max=303, avg=151.35, stdev=35.86 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 105], 10.00th=[ 113], 20.00th=[ 122], | 30.00th=[ 129], 40.00th=[ 136], 50.00th=[ 146], 60.00th=[ 155], | 70.00th=[ 165], 80.00th=[ 180], 90.00th=[ 203], 95.00th=[ 213], | 99.00th=[ 247], 99.50th=[ 275], 99.90th=[ 292], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 255, max= 584, per=24.94%, avg=419.94, stdev=80.27, samples=48 iops : min= 63, max= 146, avg=104.56, stdev=20.12, samples=48 lat (msec) : 50=0.35%, 100=2.77%, 250=96.02%, 500=0.86% cpu : usr=0.16%, sys=1.42%, ctx=2356, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1685KiB/s (1725kB/s), 421KiB/s-423KiB/s (431kB/s-433kB/s), io=40.0MiB (41.9MB), run=24211-24310msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=189121: Mon Aug 17 01:11:34 2026 read: IOPS=51, BW=205KiB/s (210kB/s)(5048KiB/24592msec) slat (usec): min=2507, max=36233, avg=10719.06, stdev=5066.02 clat (msec): min=34, max=272, avg=145.77, stdev=30.69 lat (msec): min=47, max=283, avg=156.50, stdev=31.81 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 103], 10.00th=[ 111], 20.00th=[ 120], | 30.00th=[ 128], 40.00th=[ 134], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 171], 90.00th=[ 190], 95.00th=[ 201], | 99.00th=[ 224], 99.50th=[ 245], 99.90th=[ 262], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 107, max= 296, per=25.00%, avg=204.02, stdev=37.98, samples=48 iops : min= 26, max= 74, avg=50.56, stdev= 9.60, samples=48 write: IOPS=52, BW=211KiB/s (216kB/s)(5192KiB/24592msec); 0 zone resets slat (usec): min=62, max=12360, avg=322.76, stdev=643.60 clat (msec): min=21, max=248, avg=148.05, stdev=28.21 lat (msec): min=21, max=248, avg=148.38, stdev=28.24 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 106], 10.00th=[ 115], 20.00th=[ 125], | 30.00th=[ 132], 40.00th=[ 138], 50.00th=[ 146], 60.00th=[ 155], | 70.00th=[ 161], 80.00th=[ 171], 90.00th=[ 184], 95.00th=[ 199], | 99.00th=[ 230], 99.50th=[ 234], 99.90th=[ 236], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min= 122, max= 302, per=24.83%, avg=210.81, stdev=36.35, samples=48 iops : min= 30, max= 75, avg=52.27, stdev= 9.10, samples=48 lat (msec) : 50=0.08%, 100=3.20%, 250=96.52%, 500=0.20% cpu : usr=0.28%, sys=2.24%, ctx=4761, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=189122: Mon Aug 17 01:11:34 2026 read: IOPS=50, BW=204KiB/s (209kB/s)(4976KiB/24421msec) slat (usec): min=2806, max=38461, avg=10617.13, stdev=5199.99 clat (msec): min=7, max=291, avg=144.69, stdev=31.84 lat (msec): min=13, max=298, avg=155.32, stdev=32.92 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 100], 10.00th=[ 108], 20.00th=[ 117], | 30.00th=[ 126], 40.00th=[ 133], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 161], 80.00th=[ 174], 90.00th=[ 186], 95.00th=[ 201], | 99.00th=[ 228], 99.50th=[ 243], 99.90th=[ 271], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 114, max= 296, per=24.92%, avg=203.33, stdev=38.93, samples=48 iops : min= 28, max= 74, avg=50.40, stdev= 9.78, samples=48 write: IOPS=53, BW=216KiB/s (221kB/s)(5264KiB/24421msec); 0 zone resets slat (usec): min=50, max=15350, avg=314.39, stdev=744.56 clat (msec): min=72, max=256, avg=148.05, stdev=27.78 lat (msec): min=72, max=256, avg=148.36, stdev=27.76 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 105], 10.00th=[ 113], 20.00th=[ 124], | 30.00th=[ 132], 40.00th=[ 140], 50.00th=[ 148], 60.00th=[ 155], | 70.00th=[ 163], 80.00th=[ 171], 90.00th=[ 184], 95.00th=[ 197], | 99.00th=[ 222], 99.50th=[ 232], 99.90th=[ 253], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 127, max= 328, per=25.40%, avg=215.62, stdev=44.86, samples=48 iops : min= 31, max= 82, avg=53.48, stdev=11.30, samples=48 lat (msec) : 10=0.04%, 50=0.04%, 100=3.98%, 250=95.66%, 500=0.27% cpu : usr=0.29%, sys=2.24%, ctx=4864, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=189123: Mon Aug 17 01:11:34 2026 read: IOPS=50, BW=203KiB/s (208kB/s)(4984KiB/24591msec) slat (usec): min=2746, max=41902, avg=10596.93, stdev=5512.99 clat (msec): min=76, max=267, avg=144.55, stdev=30.70 lat (msec): min=81, max=283, avg=155.15, stdev=31.75 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 101], 10.00th=[ 109], 20.00th=[ 120], | 30.00th=[ 126], 40.00th=[ 133], 50.00th=[ 140], 60.00th=[ 148], | 70.00th=[ 159], 80.00th=[ 171], 90.00th=[ 188], 95.00th=[ 203], | 99.00th=[ 226], 99.50th=[ 232], 99.90th=[ 259], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 93, max= 312, per=24.81%, avg=202.44, stdev=45.40, samples=48 iops : min= 23, max= 78, avg=50.17, stdev=11.38, samples=48 write: IOPS=53, BW=214KiB/s (219kB/s)(5256KiB/24591msec); 0 zone resets slat (usec): min=54, max=11354, avg=322.25, stdev=666.59 clat (msec): min=81, max=247, avg=149.83, stdev=28.85 lat (msec): min=81, max=248, avg=150.16, stdev=28.86 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 107], 10.00th=[ 114], 20.00th=[ 126], | 30.00th=[ 133], 40.00th=[ 140], 50.00th=[ 148], 60.00th=[ 155], | 70.00th=[ 163], 80.00th=[ 176], 90.00th=[ 188], 95.00th=[ 201], | 99.00th=[ 232], 99.50th=[ 236], 99.90th=[ 245], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min= 135, max= 310, per=24.92%, avg=211.58, stdev=37.20, samples=48 iops : min= 33, max= 77, avg=52.46, stdev= 9.38, samples=48 lat (msec) : 100=3.48%, 250=96.41%, 500=0.12% cpu : usr=0.26%, sys=2.22%, ctx=4621, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=189124: Mon Aug 17 01:11:34 2026 read: IOPS=51, BW=207KiB/s (212kB/s)(5060KiB/24443msec) slat (usec): min=85, max=42615, avg=10450.44, stdev=5174.63 clat (msec): min=14, max=278, avg=144.86, stdev=31.92 lat (msec): min=14, max=303, avg=155.32, stdev=32.85 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 101], 10.00th=[ 108], 20.00th=[ 118], | 30.00th=[ 126], 40.00th=[ 134], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 171], 90.00th=[ 190], 95.00th=[ 199], | 99.00th=[ 222], 99.50th=[ 241], 99.90th=[ 275], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 126, max= 270, per=25.37%, avg=206.98, stdev=34.67, samples=48 iops : min= 31, max= 67, avg=51.31, stdev= 8.79, samples=48 write: IOPS=52, BW=212KiB/s (217kB/s)(5180KiB/24443msec); 0 zone resets slat (usec): min=55, max=7605, avg=310.93, stdev=576.40 clat (msec): min=44, max=255, avg=149.52, stdev=28.95 lat (msec): min=45, max=255, avg=149.84, stdev=28.97 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 107], 10.00th=[ 116], 20.00th=[ 125], | 30.00th=[ 132], 40.00th=[ 140], 50.00th=[ 148], 60.00th=[ 155], | 70.00th=[ 165], 80.00th=[ 176], 90.00th=[ 188], 95.00th=[ 201], | 99.00th=[ 224], 99.50th=[ 232], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 127, max= 318, per=24.89%, avg=211.29, stdev=37.81, samples=48 iops : min= 31, max= 79, avg=52.40, stdev= 9.52, samples=48 lat (msec) : 20=0.08%, 50=0.12%, 100=3.87%, 250=95.62%, 500=0.31% cpu : usr=0.23%, sys=2.23%, ctx=4736, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=816KiB/s (836kB/s), 203KiB/s-207KiB/s (208kB/s-212kB/s), io=19.6MiB (20.5MB), run=24421-24592msec WRITE: bw=850KiB/s (870kB/s), 211KiB/s-216KiB/s (216kB/s-221kB/s), io=20.4MiB (21.4MB), run=24421-24592msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=189132: Mon Aug 17 01:11:42 2026 read: IOPS=2, BW=106MiB/s (111MB/s)(40.0MiB/377msec) slat (nsec): min=38266k, max=38266k, avg=38265834.00, stdev= 0.00 clat (nsec): min=311861k, max=311861k, avg=311860839.00, stdev= 0.00 lat (nsec): min=350132k, max=350132k, avg=350131664.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 313], 10.00th=[ 313], 20.00th=[ 313], | 30.00th=[ 313], 40.00th=[ 313], 50.00th=[ 313], 60.00th=[ 313], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 313], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.19%, ctx=5, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=106MiB/s (111MB/s), 106MiB/s-106MiB/s (111MB/s-111MB/s), io=40.0MiB (41.9MB), run=377-377msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout