== sanity test 398c: run fio to test AIO ================= 17:39:15 (1775770755) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.87777 s, 47.8 MB/s osc.lustre-OST0000-osc-ffff8f6206136000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=182129: Thu Apr 9 17:40:00 2026 write: IOPS=84, BW=338KiB/s (346kB/s)(10.0MiB/30298msec); 0 zone resets slat (usec): min=39, max=12967, avg=274.11, stdev=824.08 clat (msec): min=19, max=385, avg=188.36, stdev=47.86 lat (msec): min=20, max=385, avg=188.64, stdev=47.88 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 123], 10.00th=[ 138], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 169], 50.00th=[ 180], 60.00th=[ 190], | 70.00th=[ 207], 80.00th=[ 228], 90.00th=[ 257], 95.00th=[ 284], | 99.00th=[ 313], 99.50th=[ 338], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 193, max= 483, per=24.86%, avg=335.90, stdev=73.15, samples=60 iops : min= 48, max= 120, avg=83.62, stdev=18.29, samples=60 lat (msec) : 20=0.04%, 100=0.27%, 250=87.62%, 500=12.07% cpu : usr=0.15%, sys=1.09%, ctx=2318, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182130: Thu Apr 9 17:40:00 2026 write: IOPS=84, BW=338KiB/s (346kB/s)(10.0MiB/30304msec); 0 zone resets slat (usec): min=38, max=11825, avg=244.43, stdev=750.56 clat (msec): min=32, max=368, avg=188.02, stdev=49.03 lat (msec): min=32, max=368, avg=188.27, stdev=49.05 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 129], 10.00th=[ 140], 20.00th=[ 150], | 30.00th=[ 159], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 228], 90.00th=[ 259], 95.00th=[ 284], | 99.00th=[ 330], 99.50th=[ 363], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 159, max= 513, per=24.85%, avg=335.73, stdev=76.36, samples=59 iops : min= 39, max= 128, avg=83.56, stdev=19.11, samples=59 lat (msec) : 50=0.43%, 100=0.20%, 250=86.25%, 500=13.12% cpu : usr=0.13%, sys=1.05%, ctx=1745, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182131: Thu Apr 9 17:40:00 2026 write: IOPS=84, BW=338KiB/s (346kB/s)(10.0MiB/30286msec); 0 zone resets slat (usec): min=41, max=19107, avg=309.29, stdev=1010.49 clat (msec): min=31, max=391, avg=188.05, stdev=49.24 lat (msec): min=32, max=391, avg=188.36, stdev=49.28 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 128], 10.00th=[ 136], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 169], 50.00th=[ 180], 60.00th=[ 188], | 70.00th=[ 203], 80.00th=[ 226], 90.00th=[ 257], 95.00th=[ 288], | 99.00th=[ 326], 99.50th=[ 363], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 246, max= 512, per=24.85%, avg=335.78, stdev=76.38, samples=59 iops : min= 61, max= 128, avg=83.58, stdev=19.13, samples=59 lat (msec) : 50=0.23%, 100=0.47%, 250=87.70%, 500=11.60% cpu : usr=0.17%, sys=1.08%, ctx=2143, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182132: Thu Apr 9 17:40:00 2026 write: IOPS=84, BW=338KiB/s (346kB/s)(10.0MiB/30294msec); 0 zone resets slat (usec): min=38, max=29928, avg=331.76, stdev=1067.08 clat (msec): min=31, max=399, avg=188.58, stdev=48.51 lat (msec): min=31, max=399, avg=188.92, stdev=48.55 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 130], 10.00th=[ 140], 20.00th=[ 150], | 30.00th=[ 159], 40.00th=[ 169], 50.00th=[ 180], 60.00th=[ 188], | 70.00th=[ 203], 80.00th=[ 228], 90.00th=[ 259], 95.00th=[ 288], | 99.00th=[ 342], 99.50th=[ 347], 99.90th=[ 397], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 150, max= 510, per=24.94%, avg=336.92, stdev=72.50, samples=59 iops : min= 37, max= 127, avg=83.88, stdev=18.17, samples=59 lat (msec) : 50=0.16%, 100=0.27%, 250=87.54%, 500=12.03% cpu : usr=0.19%, sys=1.20%, ctx=2726, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1352KiB/s (1384kB/s), 338KiB/s-338KiB/s (346kB/s-346kB/s), io=40.0MiB (41.9MB), run=30286-30304msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=182147: Thu Apr 9 17:40:45 2026 read: IOPS=37, BW=151KiB/s (155kB/s)(5048KiB/33421msec) slat (usec): min=3370, max=69290, avg=14808.84, stdev=8062.02 clat (msec): min=27, max=344, avg=195.15, stdev=43.83 lat (msec): min=35, max=384, avg=209.96, stdev=45.91 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 124], 10.00th=[ 140], 20.00th=[ 161], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 194], 60.00th=[ 205], | 70.00th=[ 220], 80.00th=[ 232], 90.00th=[ 251], 95.00th=[ 268], | 99.00th=[ 300], 99.50th=[ 309], 99.90th=[ 342], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 80, max= 294, per=25.38%, avg=152.03, stdev=34.18, samples=65 iops : min= 20, max= 73, avg=37.48, stdev= 8.56, samples=65 write: IOPS=38, BW=155KiB/s (159kB/s)(5192KiB/33421msec); 0 zone resets slat (usec): min=53, max=14185, avg=328.68, stdev=1053.13 clat (msec): min=72, max=351, avg=203.43, stdev=43.13 lat (msec): min=72, max=351, avg=203.76, stdev=43.21 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 132], 10.00th=[ 153], 20.00th=[ 169], | 30.00th=[ 182], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 213], | 70.00th=[ 226], 80.00th=[ 241], 90.00th=[ 257], 95.00th=[ 279], | 99.00th=[ 317], 99.50th=[ 321], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 88, max= 232, per=25.10%, avg=156.63, stdev=33.82, samples=65 iops : min= 22, max= 58, avg=38.65, stdev= 8.46, samples=65 lat (msec) : 50=0.04%, 100=1.05%, 250=86.48%, 500=12.42% cpu : usr=0.18%, sys=2.10%, ctx=4490, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182148: Thu Apr 9 17:40:45 2026 read: IOPS=37, BW=149KiB/s (153kB/s)(4976KiB/33374msec) slat (usec): min=60, max=58745, avg=14823.10, stdev=8078.21 clat (msec): min=25, max=333, avg=196.09, stdev=44.10 lat (msec): min=40, max=347, avg=210.92, stdev=45.48 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 125], 10.00th=[ 140], 20.00th=[ 161], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 194], 60.00th=[ 209], | 70.00th=[ 220], 80.00th=[ 236], 90.00th=[ 253], 95.00th=[ 264], | 99.00th=[ 309], 99.50th=[ 317], 99.90th=[ 330], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 63, max= 286, per=24.67%, avg=147.77, stdev=39.50, samples=66 iops : min= 15, max= 71, avg=36.41, stdev= 9.89, samples=66 write: IOPS=39, BW=158KiB/s (162kB/s)(5264KiB/33374msec); 0 zone resets slat (usec): min=52, max=12910, avg=298.95, stdev=855.22 clat (msec): min=59, max=357, avg=205.41, stdev=43.50 lat (msec): min=59, max=358, avg=205.71, stdev=43.53 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 133], 10.00th=[ 150], 20.00th=[ 169], | 30.00th=[ 184], 40.00th=[ 194], 50.00th=[ 205], 60.00th=[ 215], | 70.00th=[ 228], 80.00th=[ 243], 90.00th=[ 262], 95.00th=[ 279], | 99.00th=[ 317], 99.50th=[ 334], 99.90th=[ 347], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 80, max= 240, per=25.18%, avg=157.14, stdev=35.04, samples=66 iops : min= 20, max= 60, avg=38.77, stdev= 8.81, samples=66 lat (msec) : 50=0.04%, 100=0.82%, 250=86.02%, 500=13.12% cpu : usr=0.28%, sys=1.92%, ctx=4107, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182149: Thu Apr 9 17:40:45 2026 read: IOPS=37, BW=149KiB/s (153kB/s)(4984KiB/33447msec) slat (usec): min=3067, max=56501, avg=15020.09, stdev=8158.50 clat (msec): min=7, max=369, avg=196.66, stdev=44.32 lat (msec): min=11, max=381, avg=211.69, stdev=45.99 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 123], 10.00th=[ 142], 20.00th=[ 161], | 30.00th=[ 176], 40.00th=[ 188], 50.00th=[ 197], 60.00th=[ 207], | 70.00th=[ 220], 80.00th=[ 232], 90.00th=[ 253], 95.00th=[ 271], | 99.00th=[ 300], 99.50th=[ 313], 99.90th=[ 342], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 39, max= 239, per=24.70%, avg=147.95, stdev=39.58, samples=66 iops : min= 9, max= 59, avg=36.44, stdev= 9.88, samples=66 write: IOPS=39, BW=157KiB/s (161kB/s)(5256KiB/33447msec); 0 zone resets slat (usec): min=41, max=12433, avg=256.01, stdev=573.54 clat (msec): min=70, max=357, avg=203.22, stdev=45.31 lat (msec): min=70, max=358, avg=203.47, stdev=45.33 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 131], 10.00th=[ 146], 20.00th=[ 165], | 30.00th=[ 182], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 213], | 70.00th=[ 226], 80.00th=[ 241], 90.00th=[ 259], 95.00th=[ 279], | 99.00th=[ 321], 99.50th=[ 334], 99.90th=[ 355], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 39, max= 262, per=24.89%, avg=155.33, stdev=38.97, samples=66 iops : min= 9, max= 65, avg=38.32, stdev= 9.81, samples=66 lat (msec) : 10=0.04%, 100=1.25%, 250=86.21%, 500=12.50% cpu : usr=0.19%, sys=1.87%, ctx=4257, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182150: Thu Apr 9 17:40:45 2026 read: IOPS=37, BW=151KiB/s (155kB/s)(5060KiB/33459msec) slat (usec): min=2582, max=59055, avg=14813.74, stdev=7855.96 clat (msec): min=17, max=375, avg=196.69, stdev=47.58 lat (msec): min=25, max=386, avg=211.51, stdev=49.17 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 121], 10.00th=[ 140], 20.00th=[ 161], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 197], 60.00th=[ 207], | 70.00th=[ 220], 80.00th=[ 232], 90.00th=[ 255], 95.00th=[ 271], | 99.00th=[ 317], 99.50th=[ 338], 99.90th=[ 355], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 23, max= 215, per=24.89%, avg=149.08, stdev=35.87, samples=66 iops : min= 5, max= 53, avg=36.74, stdev= 9.05, samples=66 write: IOPS=38, BW=155KiB/s (159kB/s)(5180KiB/33459msec); 0 zone resets slat (usec): min=44, max=15425, avg=299.03, stdev=918.64 clat (msec): min=25, max=385, avg=202.91, stdev=46.11 lat (msec): min=25, max=386, avg=203.21, stdev=46.14 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 131], 10.00th=[ 150], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 215], | 70.00th=[ 228], 80.00th=[ 243], 90.00th=[ 262], 95.00th=[ 275], | 99.00th=[ 317], 99.50th=[ 330], 99.90th=[ 376], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 31, max= 318, per=24.41%, avg=152.35, stdev=42.27, samples=66 iops : min= 7, max= 79, avg=37.56, stdev=10.56, samples=66 lat (msec) : 20=0.04%, 50=0.35%, 100=1.72%, 250=84.41%, 500=13.48% cpu : usr=0.23%, sys=1.93%, ctx=4077, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=600KiB/s (614kB/s), 149KiB/s-151KiB/s (153kB/s-155kB/s), io=19.6MiB (20.5MB), run=33374-33459msec WRITE: bw=624KiB/s (639kB/s), 155KiB/s-158KiB/s (159kB/s-162kB/s), io=20.4MiB (21.4MB), run=33374-33459msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=182159: Thu Apr 9 17:40:58 2026 read: IOPS=1, BW=65.6MiB/s (68.8MB/s)(40.0MiB/610msec) slat (nsec): min=52571k, max=52571k, avg=52570792.00, stdev= 0.00 clat (nsec): min=528098k, max=528098k, avg=528098197.00, stdev= 0.00 lat (nsec): min=580673k, max=580673k, avg=580673318.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 527], 5.00th=[ 527], 10.00th=[ 527], 20.00th=[ 527], | 30.00th=[ 527], 40.00th=[ 527], 50.00th=[ 527], 60.00th=[ 527], | 70.00th=[ 527], 80.00th=[ 527], 90.00th=[ 527], 95.00th=[ 527], | 99.00th=[ 527], 99.50th=[ 527], 99.90th=[ 527], 99.95th=[ 527], | 99.99th=[ 527] lat (msec) : 750=100.00% cpu : usr=0.16%, sys=3.12%, ctx=10, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=65.6MiB/s (68.8MB/s), 65.6MiB/s-65.6MiB/s (68.8MB/s-68.8MB/s), io=40.0MiB (41.9MB), run=610-610msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout