== sanity test 398c: run fio to test AIO ================= 13:19:42 (1789060782) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.637426 s, 65.8 MB/s osc.lustre-OST0000-osc-ffff9cab5e6d3800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179879: Thu Sep 10 13:20:20 2026 write: IOPS=91, BW=366KiB/s (375kB/s)(10.0MiB/27980msec); 0 zone resets slat (usec): min=28, max=12485, avg=240.02, stdev=718.85 clat (msec): min=16, max=294, avg=174.43, stdev=37.38 lat (msec): min=16, max=294, avg=174.68, stdev=37.38 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 122], 10.00th=[ 130], 20.00th=[ 144], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 205], 90.00th=[ 222], 95.00th=[ 245], | 99.00th=[ 271], 99.50th=[ 275], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 254, max= 544, per=25.03%, avg=365.40, stdev=61.54, samples=55 iops : min= 63, max= 136, avg=90.93, stdev=15.47, samples=55 lat (msec) : 20=0.04%, 50=0.12%, 100=1.13%, 250=94.73%, 500=3.98% cpu : usr=0.21%, sys=1.04%, ctx=2885, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179880: Thu Sep 10 13:20:20 2026 write: IOPS=91, BW=365KiB/s (374kB/s)(10.0MiB/28052msec); 0 zone resets slat (usec): min=28, max=19379, avg=243.58, stdev=797.97 clat (msec): min=30, max=294, avg=174.17, stdev=36.23 lat (msec): min=30, max=294, avg=174.42, stdev=36.20 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 122], 10.00th=[ 132], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 190], 80.00th=[ 203], 90.00th=[ 224], 95.00th=[ 239], | 99.00th=[ 268], 99.50th=[ 284], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 296] bw ( KiB/s): min= 242, max= 520, per=24.85%, avg=362.84, stdev=59.34, samples=55 iops : min= 60, max= 130, avg=90.27, stdev=14.89, samples=55 lat (msec) : 50=0.12%, 100=0.82%, 250=96.09%, 500=2.97% cpu : usr=0.17%, sys=1.13%, ctx=3027, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179881: Thu Sep 10 13:20:20 2026 write: IOPS=91, BW=366KiB/s (374kB/s)(10.0MiB/28010msec); 0 zone resets slat (usec): min=28, max=14995, avg=206.16, stdev=654.07 clat (msec): min=16, max=293, avg=174.01, stdev=36.38 lat (msec): min=17, max=293, avg=174.22, stdev=36.36 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 124], 10.00th=[ 131], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 190], 80.00th=[ 203], 90.00th=[ 222], 95.00th=[ 243], | 99.00th=[ 271], 99.50th=[ 275], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 243, max= 512, per=24.89%, avg=363.44, stdev=56.48, samples=55 iops : min= 60, max= 128, avg=90.42, stdev=14.16, samples=55 lat (msec) : 20=0.08%, 100=0.74%, 250=95.70%, 500=3.48% cpu : usr=0.16%, sys=0.89%, ctx=2625, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179882: Thu Sep 10 13:20:20 2026 write: IOPS=91, BW=365KiB/s (374kB/s)(10.0MiB/28054msec); 0 zone resets slat (usec): min=28, max=12067, avg=150.96, stdev=483.49 clat (msec): min=78, max=293, avg=174.16, stdev=37.37 lat (msec): min=78, max=293, avg=174.31, stdev=37.37 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 121], 10.00th=[ 130], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 180], | 70.00th=[ 190], 80.00th=[ 205], 90.00th=[ 230], 95.00th=[ 241], | 99.00th=[ 268], 99.50th=[ 288], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 244, max= 512, per=24.81%, avg=362.16, stdev=64.47, samples=55 iops : min= 61, max= 128, avg=90.15, stdev=16.16, samples=55 lat (msec) : 100=0.94%, 250=96.72%, 500=2.34% cpu : usr=0.12%, sys=0.70%, ctx=1295, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1460KiB/s (1495kB/s), 365KiB/s-366KiB/s (374kB/s-375kB/s), io=40.0MiB (41.9MB), run=27980-28054msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179897: Thu Sep 10 13:20:58 2026 read: IOPS=41, BW=165KiB/s (169kB/s)(5048KiB/30618msec) slat (usec): min=3292, max=48815, avg=13685.28, stdev=7190.88 clat (msec): min=83, max=313, avg=182.04, stdev=40.48 lat (msec): min=99, max=328, avg=195.74, stdev=41.35 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 124], 10.00th=[ 134], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 215], 90.00th=[ 241], 95.00th=[ 259], | 99.00th=[ 288], 99.50th=[ 296], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 93, max= 247, per=25.09%, avg=163.87, stdev=34.19, samples=60 iops : min= 23, max= 61, avg=40.52, stdev= 8.54, samples=60 write: IOPS=42, BW=170KiB/s (174kB/s)(5192KiB/30618msec); 0 zone resets slat (usec): min=34, max=14942, avg=298.44, stdev=977.87 clat (msec): min=104, max=338, avg=183.70, stdev=37.36 lat (msec): min=104, max=338, avg=184.00, stdev=37.38 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 129], 10.00th=[ 140], 20.00th=[ 153], | 30.00th=[ 165], 40.00th=[ 174], 50.00th=[ 180], 60.00th=[ 188], | 70.00th=[ 197], 80.00th=[ 211], 90.00th=[ 236], 95.00th=[ 257], | 99.00th=[ 284], 99.50th=[ 292], 99.90th=[ 309], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 62, max= 247, per=24.93%, avg=169.50, stdev=40.91, samples=60 iops : min= 15, max= 61, avg=41.92, stdev=10.22, samples=60 lat (msec) : 100=0.35%, 250=93.44%, 500=6.21% cpu : usr=0.34%, sys=2.03%, ctx=4659, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179898: Thu Sep 10 13:20:58 2026 read: IOPS=40, BW=162KiB/s (166kB/s)(4976KiB/30690msec) slat (usec): min=2525, max=55410, avg=13901.96, stdev=7575.13 clat (msec): min=41, max=330, avg=180.61, stdev=41.35 lat (msec): min=59, max=350, avg=194.52, stdev=42.31 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 120], 10.00th=[ 132], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 186], | 70.00th=[ 199], 80.00th=[ 213], 90.00th=[ 239], 95.00th=[ 257], | 99.00th=[ 292], 99.50th=[ 309], 99.90th=[ 326], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 69, max= 245, per=24.62%, avg=160.75, stdev=36.34, samples=60 iops : min= 17, max= 61, avg=39.73, stdev= 9.09, samples=60 write: IOPS=42, BW=172KiB/s (176kB/s)(5264KiB/30690msec); 0 zone resets slat (usec): min=37, max=16796, avg=321.55, stdev=1036.33 clat (msec): min=29, max=314, avg=186.55, stdev=37.49 lat (msec): min=30, max=320, avg=186.88, stdev=37.51 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 129], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 176], 50.00th=[ 184], 60.00th=[ 194], | 70.00th=[ 205], 80.00th=[ 215], 90.00th=[ 236], 95.00th=[ 257], | 99.00th=[ 288], 99.50th=[ 296], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 80, max= 271, per=25.00%, avg=169.97, stdev=36.43, samples=60 iops : min= 20, max= 67, avg=42.03, stdev= 9.03, samples=60 lat (msec) : 50=0.12%, 100=0.70%, 250=92.85%, 500=6.33% cpu : usr=0.30%, sys=2.18%, ctx=4601, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179899: Thu Sep 10 13:20:58 2026 read: IOPS=40, BW=163KiB/s (167kB/s)(4984KiB/30621msec) slat (usec): min=2658, max=58223, avg=13698.08, stdev=7253.48 clat (msec): min=31, max=327, avg=180.42, stdev=39.05 lat (msec): min=40, max=347, avg=194.12, stdev=40.09 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 127], 10.00th=[ 136], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 199], 80.00th=[ 211], 90.00th=[ 234], 95.00th=[ 253], | 99.00th=[ 288], 99.50th=[ 296], 99.90th=[ 309], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 94, max= 277, per=24.89%, avg=162.55, stdev=37.36, samples=60 iops : min= 23, max= 69, avg=40.17, stdev= 9.32, samples=60 write: IOPS=42, BW=172KiB/s (176kB/s)(5256KiB/30621msec); 0 zone resets slat (usec): min=36, max=24756, avg=333.19, stdev=1101.27 clat (msec): min=103, max=350, avg=185.97, stdev=37.94 lat (msec): min=103, max=350, avg=186.30, stdev=37.90 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 130], 10.00th=[ 142], 20.00th=[ 155], | 30.00th=[ 165], 40.00th=[ 174], 50.00th=[ 182], 60.00th=[ 190], | 70.00th=[ 203], 80.00th=[ 215], 90.00th=[ 236], 95.00th=[ 257], | 99.00th=[ 296], 99.50th=[ 309], 99.90th=[ 338], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 85, max= 254, per=25.12%, avg=170.85, stdev=33.37, samples=60 iops : min= 21, max= 63, avg=42.27, stdev= 8.27, samples=60 lat (msec) : 50=0.04%, 100=0.23%, 250=93.75%, 500=5.98% cpu : usr=0.28%, sys=2.11%, ctx=4685, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179900: Thu Sep 10 13:20:58 2026 read: IOPS=41, BW=166KiB/s (170kB/s)(5060KiB/30437msec) slat (usec): min=51, max=55789, avg=13606.44, stdev=7157.41 clat (msec): min=25, max=329, avg=179.53, stdev=41.73 lat (msec): min=25, max=347, avg=193.14, stdev=42.96 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 118], 10.00th=[ 130], 20.00th=[ 144], | 30.00th=[ 157], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 199], 80.00th=[ 211], 90.00th=[ 236], 95.00th=[ 257], | 99.00th=[ 288], 99.50th=[ 300], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 72, max= 285, per=25.45%, avg=166.22, stdev=34.60, samples=60 iops : min= 18, max= 71, avg=41.10, stdev= 8.67, samples=60 write: IOPS=42, BW=170KiB/s (174kB/s)(5180KiB/30437msec); 0 zone resets slat (usec): min=30, max=13302, avg=324.53, stdev=881.73 clat (msec): min=91, max=338, avg=186.55, stdev=40.02 lat (msec): min=91, max=338, avg=186.88, stdev=39.98 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 131], 10.00th=[ 140], 20.00th=[ 155], | 30.00th=[ 165], 40.00th=[ 174], 50.00th=[ 182], 60.00th=[ 190], | 70.00th=[ 201], 80.00th=[ 215], 90.00th=[ 239], 95.00th=[ 266], | 99.00th=[ 309], 99.50th=[ 321], 99.90th=[ 334], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 80, max= 270, per=24.89%, avg=169.28, stdev=35.91, samples=60 iops : min= 20, max= 67, avg=41.85, stdev= 8.98, samples=60 lat (msec) : 50=0.08%, 100=0.62%, 250=92.62%, 500=6.68% cpu : usr=0.34%, sys=2.20%, ctx=4590, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=654KiB/s (670kB/s), 162KiB/s-166KiB/s (166kB/s-170kB/s), io=19.6MiB (20.5MB), run=30437-30690msec WRITE: bw=681KiB/s (697kB/s), 170KiB/s-172KiB/s (174kB/s-176kB/s), io=20.4MiB (21.4MB), run=30437-30690msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179909: Thu Sep 10 13:21:08 2026 read: IOPS=2, BW=83.3MiB/s (87.4MB/s)(40.0MiB/480msec) slat (nsec): min=27908k, max=27908k, avg=27907735.00, stdev= 0.00 clat (nsec): min=431228k, max=431228k, avg=431227684.00, stdev= 0.00 lat (nsec): min=459140k, max=459140k, avg=459139567.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 430], 10.00th=[ 430], 20.00th=[ 430], | 30.00th=[ 430], 40.00th=[ 430], 50.00th=[ 430], 60.00th=[ 430], | 70.00th=[ 430], 80.00th=[ 430], 90.00th=[ 430], 95.00th=[ 430], | 99.00th=[ 430], 99.50th=[ 430], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.92%, ctx=5, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=83.3MiB/s (87.4MB/s), 83.3MiB/s-83.3MiB/s (87.4MB/s-87.4MB/s), io=40.0MiB (41.9MB), run=480-480msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout