== sanity test 398c: run fio to test AIO ================= 10:51:39 (1788619899) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.301946 s, 139 MB/s osc.lustre-OST0000-osc-ffff9e85584bb800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180169: Sat Sep 5 10:52:14 2026 write: IOPS=102, BW=409KiB/s (418kB/s)(10.0MiB/25062msec); 0 zone resets slat (usec): min=21, max=15757, avg=216.69, stdev=833.54 clat (msec): min=15, max=280, avg=156.20, stdev=35.72 lat (msec): min=16, max=280, avg=156.42, stdev=35.70 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 107], 10.00th=[ 115], 20.00th=[ 128], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 163], | 70.00th=[ 174], 80.00th=[ 182], 90.00th=[ 207], 95.00th=[ 220], | 99.00th=[ 251], 99.50th=[ 262], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 255, max= 628, per=24.96%, avg=406.92, stdev=71.20, samples=49 iops : min= 63, max= 157, avg=101.33, stdev=17.86, samples=49 lat (msec) : 20=0.08%, 50=0.04%, 100=3.95%, 250=94.92%, 500=1.02% cpu : usr=0.21%, sys=0.99%, ctx=1869, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180170: Sat Sep 5 10:52:14 2026 write: IOPS=102, BW=409KiB/s (419kB/s)(10.0MiB/25043msec); 0 zone resets slat (usec): min=25, max=15485, avg=290.48, stdev=880.39 clat (msec): min=19, max=297, avg=154.90, stdev=36.20 lat (msec): min=19, max=297, avg=155.19, stdev=36.20 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 101], 10.00th=[ 113], 20.00th=[ 128], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 161], | 70.00th=[ 171], 80.00th=[ 184], 90.00th=[ 203], 95.00th=[ 218], | 99.00th=[ 249], 99.50th=[ 271], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 238, max= 636, per=25.01%, avg=407.73, stdev=71.53, samples=49 iops : min= 59, max= 159, avg=101.53, stdev=17.93, samples=49 lat (msec) : 20=0.08%, 50=0.35%, 100=4.49%, 250=94.14%, 500=0.94% cpu : usr=0.29%, sys=1.59%, ctx=3023, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180171: Sat Sep 5 10:52:14 2026 write: IOPS=101, BW=408KiB/s (417kB/s)(10.0MiB/25120msec); 0 zone resets slat (usec): min=20, max=6897, avg=188.21, stdev=501.30 clat (msec): min=22, max=293, avg=155.30, stdev=35.95 lat (msec): min=22, max=293, avg=155.49, stdev=35.94 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 99], 10.00th=[ 114], 20.00th=[ 130], | 30.00th=[ 138], 40.00th=[ 144], 50.00th=[ 150], 60.00th=[ 161], | 70.00th=[ 174], 80.00th=[ 182], 90.00th=[ 205], 95.00th=[ 220], | 99.00th=[ 259], 99.50th=[ 275], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 296] bw ( KiB/s): min= 230, max= 628, per=24.81%, avg=404.39, stdev=78.23, samples=49 iops : min= 57, max= 157, avg=100.71, stdev=19.61, samples=49 lat (msec) : 50=0.08%, 100=5.27%, 250=93.16%, 500=1.48% cpu : usr=0.20%, sys=0.95%, ctx=1873, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180172: Sat Sep 5 10:52:14 2026 write: IOPS=101, BW=408KiB/s (418kB/s)(10.0MiB/25104msec); 0 zone resets slat (usec): min=22, max=47377, avg=248.57, stdev=1176.22 clat (msec): min=24, max=283, avg=155.39, stdev=35.88 lat (msec): min=24, max=285, avg=155.64, stdev=35.83 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 102], 10.00th=[ 115], 20.00th=[ 129], | 30.00th=[ 138], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 161], | 70.00th=[ 171], 80.00th=[ 182], 90.00th=[ 203], 95.00th=[ 222], | 99.00th=[ 259], 99.50th=[ 275], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 255, max= 628, per=24.89%, avg=405.65, stdev=73.60, samples=49 iops : min= 63, max= 157, avg=101.00, stdev=18.44, samples=49 lat (msec) : 50=0.39%, 100=4.14%, 250=94.06%, 500=1.41% cpu : usr=0.24%, sys=1.15%, ctx=2192, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1631KiB/s (1670kB/s), 408KiB/s-409KiB/s (417kB/s-419kB/s), io=40.0MiB (41.9MB), run=25043-25120msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180186: Sat Sep 5 10:52:51 2026 read: IOPS=41, BW=167KiB/s (171kB/s)(5048KiB/30196msec) slat (usec): min=3265, max=45999, avg=13084.19, stdev=6496.11 clat (msec): min=22, max=377, avg=178.28, stdev=39.21 lat (msec): min=29, max=418, avg=191.37, stdev=40.44 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 122], 10.00th=[ 134], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 184], | 70.00th=[ 197], 80.00th=[ 211], 90.00th=[ 232], 95.00th=[ 247], | 99.00th=[ 279], 99.50th=[ 292], 99.90th=[ 347], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 100, max= 270, per=25.17%, avg=167.14, stdev=35.06, samples=59 iops : min= 25, max= 67, avg=41.29, stdev= 8.78, samples=59 write: IOPS=42, BW=172KiB/s (176kB/s)(5192KiB/30196msec); 0 zone resets slat (usec): min=25, max=16475, avg=284.24, stdev=820.05 clat (msec): min=39, max=345, avg=183.77, stdev=36.66 lat (msec): min=40, max=345, avg=184.06, stdev=36.66 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 127], 10.00th=[ 138], 20.00th=[ 153], | 30.00th=[ 165], 40.00th=[ 174], 50.00th=[ 182], 60.00th=[ 190], | 70.00th=[ 201], 80.00th=[ 218], 90.00th=[ 232], 95.00th=[ 243], | 99.00th=[ 275], 99.50th=[ 288], 99.90th=[ 321], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 110, max= 294, per=24.95%, avg=172.39, stdev=36.99, samples=59 iops : min= 27, max= 73, avg=42.59, stdev= 9.22, samples=59 lat (msec) : 50=0.12%, 100=0.90%, 250=94.88%, 500=4.10% cpu : usr=0.26%, sys=1.97%, ctx=4506, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180187: Sat Sep 5 10:52:51 2026 read: IOPS=41, BW=165KiB/s (169kB/s)(4976KiB/30195msec) slat (usec): min=3110, max=63412, avg=13253.12, stdev=6951.21 clat (msec): min=28, max=342, avg=178.97, stdev=38.97 lat (msec): min=39, max=352, avg=192.23, stdev=40.29 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 120], 10.00th=[ 132], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 211], 90.00th=[ 230], 95.00th=[ 245], | 99.00th=[ 288], 99.50th=[ 305], 99.90th=[ 334], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 100, max= 223, per=24.73%, avg=164.22, stdev=32.28, samples=59 iops : min= 25, max= 55, avg=40.54, stdev= 8.05, samples=59 write: IOPS=43, BW=174KiB/s (179kB/s)(5264KiB/30195msec); 0 zone resets slat (usec): min=32, max=16517, avg=276.78, stdev=768.26 clat (msec): min=66, max=342, avg=183.24, stdev=35.41 lat (msec): min=66, max=342, avg=183.52, stdev=35.42 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 128], 10.00th=[ 140], 20.00th=[ 157], | 30.00th=[ 165], 40.00th=[ 174], 50.00th=[ 182], 60.00th=[ 190], | 70.00th=[ 201], 80.00th=[ 211], 90.00th=[ 230], 95.00th=[ 243], | 99.00th=[ 271], 99.50th=[ 279], 99.90th=[ 321], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 104, max= 294, per=25.26%, avg=174.53, stdev=38.26, samples=59 iops : min= 26, max= 73, avg=43.14, stdev= 9.56, samples=59 lat (msec) : 50=0.08%, 100=1.02%, 250=95.43%, 500=3.48% cpu : usr=0.30%, sys=1.86%, ctx=4238, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180188: Sat Sep 5 10:52:51 2026 read: IOPS=41, BW=166KiB/s (170kB/s)(4984KiB/29942msec) slat (usec): min=37, max=42241, avg=13081.53, stdev=6448.85 clat (msec): min=48, max=314, avg=177.45, stdev=37.69 lat (msec): min=59, max=327, avg=190.54, stdev=38.94 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 121], 10.00th=[ 132], 20.00th=[ 146], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 184], | 70.00th=[ 197], 80.00th=[ 209], 90.00th=[ 230], 95.00th=[ 245], | 99.00th=[ 271], 99.50th=[ 279], 99.90th=[ 296], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 103, max= 262, per=25.11%, avg=166.75, stdev=37.93, samples=59 iops : min= 25, max= 65, avg=41.19, stdev= 9.52, samples=59 write: IOPS=43, BW=176KiB/s (180kB/s)(5256KiB/29942msec); 0 zone resets slat (usec): min=34, max=33106, avg=279.61, stdev=1170.23 clat (msec): min=51, max=349, avg=182.99, stdev=36.53 lat (msec): min=52, max=349, avg=183.27, stdev=36.51 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 125], 10.00th=[ 140], 20.00th=[ 155], | 30.00th=[ 163], 40.00th=[ 174], 50.00th=[ 182], 60.00th=[ 190], | 70.00th=[ 201], 80.00th=[ 213], 90.00th=[ 228], 95.00th=[ 243], | 99.00th=[ 275], 99.50th=[ 296], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 102, max= 246, per=25.30%, avg=174.83, stdev=33.50, samples=59 iops : min= 25, max= 61, avg=43.20, stdev= 8.35, samples=59 lat (msec) : 50=0.04%, 100=0.82%, 250=95.43%, 500=3.71% cpu : usr=0.29%, sys=1.93%, ctx=4391, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180189: Sat Sep 5 10:52:51 2026 read: IOPS=41, BW=168KiB/s (172kB/s)(5060KiB/30195msec) slat (msec): min=2, max=218, avg=13.25, stdev= 8.72 clat (msec): min=37, max=416, avg=178.67, stdev=39.62 lat (msec): min=58, max=448, avg=191.93, stdev=40.76 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 123], 10.00th=[ 132], 20.00th=[ 146], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 209], 90.00th=[ 230], 95.00th=[ 247], | 99.00th=[ 279], 99.50th=[ 288], 99.90th=[ 405], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 88, max= 247, per=25.21%, avg=167.39, stdev=32.24, samples=59 iops : min= 22, max= 61, avg=41.34, stdev= 8.04, samples=59 write: IOPS=42, BW=172KiB/s (176kB/s)(5180KiB/30195msec); 0 zone resets slat (usec): min=30, max=14086, avg=290.14, stdev=801.48 clat (msec): min=71, max=404, avg=183.28, stdev=39.42 lat (msec): min=71, max=406, avg=183.58, stdev=39.49 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 126], 10.00th=[ 138], 20.00th=[ 150], | 30.00th=[ 163], 40.00th=[ 174], 50.00th=[ 182], 60.00th=[ 190], | 70.00th=[ 201], 80.00th=[ 211], 90.00th=[ 234], 95.00th=[ 253], | 99.00th=[ 292], 99.50th=[ 317], 99.90th=[ 393], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 110, max= 270, per=24.71%, avg=170.75, stdev=37.43, samples=59 iops : min= 27, max= 67, avg=42.19, stdev= 9.35, samples=59 lat (msec) : 50=0.04%, 100=1.25%, 250=93.91%, 500=4.80% cpu : usr=0.32%, sys=2.00%, ctx=4528, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=665KiB/s (681kB/s), 165KiB/s-168KiB/s (169kB/s-172kB/s), io=19.6MiB (20.5MB), run=29942-30196msec WRITE: bw=692KiB/s (708kB/s), 172KiB/s-176KiB/s (176kB/s-180kB/s), io=20.4MiB (21.4MB), run=29942-30196msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180197: Sat Sep 5 10:53:01 2026 read: IOPS=1, BW=77.7MiB/s (81.4MB/s)(40.0MiB/515msec) slat (nsec): min=32944k, max=32944k, avg=32943566.00, stdev= 0.00 clat (nsec): min=461807k, max=461807k, avg=461807395.00, stdev= 0.00 lat (nsec): min=494756k, max=494756k, avg=494756321.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 464], 10.00th=[ 464], 20.00th=[ 464], | 30.00th=[ 464], 40.00th=[ 464], 50.00th=[ 464], 60.00th=[ 464], | 70.00th=[ 464], 80.00th=[ 464], 90.00th=[ 464], 95.00th=[ 464], | 99.00th=[ 464], 99.50th=[ 464], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.33%, ctx=7, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=77.7MiB/s (81.4MB/s), 77.7MiB/s-77.7MiB/s (81.4MB/s-81.4MB/s), io=40.0MiB (41.9MB), run=515-515msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout