== sanity test 398c: run fio to test AIO ================= 08:55:43 (1786020943) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.59606 s, 70.4 MB/s osc.lustre-OST0000-osc-ffff91c61145a000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144196: Thu Aug 6 09:00:53 2026 write: IOPS=8, BW=35.3KiB/s (36.2kB/s)(10.0MiB/289869msec); 0 zone resets slat (usec): min=47, max=645831, avg=470.58, stdev=12811.23 clat (msec): min=392, max=8440, avg=1809.48, stdev=920.27 lat (msec): min=392, max=8440, avg=1809.95, stdev=921.04 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 600], 10.00th=[ 701], 20.00th=[ 1011], | 30.00th=[ 1267], 40.00th=[ 1502], 50.00th=[ 1670], 60.00th=[ 1938], | 70.00th=[ 2165], 80.00th=[ 2500], 90.00th=[ 2937], 95.00th=[ 3406], | 99.00th=[ 4530], 99.50th=[ 5671], 99.90th=[ 8087], 99.95th=[ 8087], | 99.99th=[ 8423] bw ( KiB/s): min= 7, max= 128, per=28.80%, avg=40.61, stdev=28.99, samples=492 iops : min= 1, max= 32, avg= 9.68, stdev= 7.27, samples=492 lat (msec) : 500=2.30%, 750=8.71%, 1000=8.09%, 2000=43.44%, >=2000=37.46% cpu : usr=0.02%, sys=0.12%, ctx=1523, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144197: Thu Aug 6 09:00:53 2026 write: IOPS=8, BW=35.3KiB/s (36.2kB/s)(10.0MiB/289873msec); 0 zone resets slat (usec): min=49, max=27792, avg=237.79, stdev=1202.63 clat (msec): min=259, max=8623, avg=1809.41, stdev=933.82 lat (msec): min=259, max=8623, avg=1809.65, stdev=933.85 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 617], 10.00th=[ 718], 20.00th=[ 986], | 30.00th=[ 1267], 40.00th=[ 1502], 50.00th=[ 1720], 60.00th=[ 1938], | 70.00th=[ 2165], 80.00th=[ 2534], 90.00th=[ 2903], 95.00th=[ 3406], | 99.00th=[ 4396], 99.50th=[ 6074], 99.90th=[ 8221], 99.95th=[ 8423], | 99.99th=[ 8658] bw ( KiB/s): min= 7, max= 144, per=29.30%, avg=41.32, stdev=29.30, samples=484 iops : min= 1, max= 36, avg= 9.86, stdev= 7.38, samples=484 lat (msec) : 500=1.56%, 750=11.02%, 1000=7.89%, 2000=41.41%, >=2000=38.12% cpu : usr=0.02%, sys=0.13%, ctx=1672, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144198: Thu Aug 6 09:00:53 2026 write: IOPS=8, BW=35.4KiB/s (36.2kB/s)(10.0MiB/289576msec); 0 zone resets slat (usec): min=49, max=32678, avg=207.12, stdev=1024.51 clat (msec): min=200, max=8997, avg=1807.64, stdev=963.14 lat (msec): min=200, max=8997, avg=1807.85, stdev=963.16 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 584], 10.00th=[ 701], 20.00th=[ 986], | 30.00th=[ 1267], 40.00th=[ 1469], 50.00th=[ 1687], 60.00th=[ 1955], | 70.00th=[ 2165], 80.00th=[ 2500], 90.00th=[ 2937], 95.00th=[ 3373], | 99.00th=[ 4279], 99.50th=[ 7886], 99.90th=[ 8926], 99.95th=[ 9060], | 99.99th=[ 9060] bw ( KiB/s): min= 4, max= 136, per=28.53%, avg=40.23, stdev=30.43, samples=498 iops : min= 1, max= 34, avg= 9.59, stdev= 7.67, samples=498 lat (msec) : 250=0.20%, 500=1.68%, 750=9.69%, 1000=8.91%, 2000=41.80% lat (msec) : >=2000=37.73% cpu : usr=0.02%, sys=0.13%, ctx=1842, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144199: Thu Aug 6 09:00:53 2026 write: IOPS=8, BW=35.3KiB/s (36.2kB/s)(10.0MiB/289875msec); 0 zone resets slat (usec): min=43, max=23572, avg=227.13, stdev=1066.60 clat (msec): min=261, max=8885, avg=1809.81, stdev=954.20 lat (msec): min=261, max=8885, avg=1810.03, stdev=954.26 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 600], 10.00th=[ 718], 20.00th=[ 1028], | 30.00th=[ 1267], 40.00th=[ 1485], 50.00th=[ 1703], 60.00th=[ 1938], | 70.00th=[ 2198], 80.00th=[ 2500], 90.00th=[ 2869], 95.00th=[ 3306], | 99.00th=[ 4530], 99.50th=[ 7684], 99.90th=[ 8658], 99.95th=[ 8926], | 99.99th=[ 8926] bw ( KiB/s): min= 7, max= 191, per=27.63%, avg=38.96, stdev=26.96, samples=514 iops : min= 1, max= 47, avg= 9.27, stdev= 6.80, samples=514 lat (msec) : 500=1.91%, 750=10.51%, 1000=6.91%, 2000=42.46%, >=2000=38.20% cpu : usr=0.02%, sys=0.13%, ctx=1784, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=141KiB/s (145kB/s), 35.3KiB/s-35.4KiB/s (36.2kB/s-36.2kB/s), io=40.0MiB (41.9MB), run=289576-289875msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144261: Thu Aug 6 09:04:17 2026 read: IOPS=6, BW=27.2KiB/s (27.9kB/s)(5048KiB/185500msec) slat (msec): min=2, max=100, avg=19.73, stdev=13.57 clat (msec): min=18, max=1852, avg=725.50, stdev=225.76 lat (msec): min=25, max=1867, avg=745.23, stdev=228.57 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 305], 10.00th=[ 405], 20.00th=[ 542], | 30.00th=[ 642], 40.00th=[ 701], 50.00th=[ 751], 60.00th=[ 793], | 70.00th=[ 844], 80.00th=[ 902], 90.00th=[ 986], 95.00th=[ 1053], | 99.00th=[ 1217], 99.50th=[ 1250], 99.90th=[ 1703], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 7, max= 96, per=26.48%, avg=28.33, stdev=15.36, samples=347 iops : min= 1, max= 24, avg= 6.62, stdev= 3.87, samples=347 write: IOPS=6, BW=27.0KiB/s (28.7kB/s)(5192KiB/185500msec); 0 zone resets slat (usec): min=53, max=20384, avg=276.02, stdev=1077.96 clat (msec): min=565, max=2998, avg=1558.00, stdev=441.98 lat (msec): min=565, max=2998, avg=1558.28, stdev=442.02 clat percentiles (msec): | 1.00th=[ 651], 5.00th=[ 743], 10.00th=[ 978], 20.00th=[ 1167], | 30.00th=[ 1334], 40.00th=[ 1452], 50.00th=[ 1586], 60.00th=[ 1687], | 70.00th=[ 1787], 80.00th=[ 1921], 90.00th=[ 2106], 95.00th=[ 2265], | 99.00th=[ 2601], 99.50th=[ 2735], 99.90th=[ 2970], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 7, max= 88, per=25.49%, avg=28.55, stdev=16.21, samples=353 iops : min= 1, max= 22, avg= 6.67, stdev= 4.07, samples=353 lat (msec) : 20=0.04%, 50=0.08%, 100=0.16%, 250=0.82%, 500=6.60% lat (msec) : 750=19.88%, 1000=22.85%, 2000=42.03%, >=2000=7.54% cpu : usr=0.05%, sys=0.34%, ctx=4052, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144262: Thu Aug 6 09:04:17 2026 read: IOPS=6, BW=26.7KiB/s (27.3kB/s)(4976KiB/186299msec) slat (usec): min=69, max=122055, avg=19339.19, stdev=13289.40 clat (msec): min=38, max=1391, avg=724.77, stdev=221.73 lat (msec): min=42, max=1403, avg=744.11, stdev=224.23 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 326], 10.00th=[ 401], 20.00th=[ 531], | 30.00th=[ 634], 40.00th=[ 701], 50.00th=[ 751], 60.00th=[ 793], | 70.00th=[ 844], 80.00th=[ 902], 90.00th=[ 986], 95.00th=[ 1062], | 99.00th=[ 1234], 99.50th=[ 1284], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 7, max= 88, per=26.22%, avg=28.06, stdev=15.71, samples=344 iops : min= 1, max= 22, avg= 6.55, stdev= 3.95, samples=344 write: IOPS=7, BW=28.3KiB/s (28.9kB/s)(5264KiB/186299msec); 0 zone resets slat (usec): min=52, max=38170, avg=260.08, stdev=1316.67 clat (msec): min=198, max=2900, avg=1560.21, stdev=441.41 lat (msec): min=198, max=2900, avg=1560.47, stdev=441.46 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 760], 10.00th=[ 986], 20.00th=[ 1183], | 30.00th=[ 1351], 40.00th=[ 1469], 50.00th=[ 1586], 60.00th=[ 1687], | 70.00th=[ 1787], 80.00th=[ 1938], 90.00th=[ 2106], 95.00th=[ 2265], | 99.00th=[ 2567], 99.50th=[ 2735], 99.90th=[ 2903], 99.95th=[ 2903], | 99.99th=[ 2903] bw ( KiB/s): min= 7, max= 104, per=25.43%, avg=28.48, stdev=17.06, samples=359 iops : min= 1, max= 26, avg= 6.65, stdev= 4.28, samples=359 lat (msec) : 50=0.08%, 100=0.12%, 250=0.90%, 500=7.62%, 750=17.77% lat (msec) : 1000=23.05%, 2000=42.93%, >=2000=7.54% cpu : usr=0.05%, sys=0.33%, ctx=3747, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144263: Thu Aug 6 09:04:17 2026 read: IOPS=6, BW=26.8KiB/s (27.4kB/s)(4984KiB/186302msec) slat (msec): min=2, max=128, avg=19.98, stdev=13.23 clat (msec): min=23, max=1327, avg=711.95, stdev=218.72 lat (msec): min=34, max=1347, avg=731.93, stdev=221.62 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 321], 10.00th=[ 388], 20.00th=[ 523], | 30.00th=[ 617], 40.00th=[ 684], 50.00th=[ 726], 60.00th=[ 776], | 70.00th=[ 827], 80.00th=[ 885], 90.00th=[ 995], 95.00th=[ 1062], | 99.00th=[ 1183], 99.50th=[ 1234], 99.90th=[ 1301], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 5, max= 96, per=26.03%, avg=27.85, stdev=16.51, samples=349 iops : min= 1, max= 24, avg= 6.50, stdev= 4.13, samples=349 write: IOPS=7, BW=28.2KiB/s (28.9kB/s)(5256KiB/186302msec); 0 zone resets slat (usec): min=54, max=22502, avg=252.27, stdev=916.19 clat (msec): min=463, max=2764, avg=1570.88, stdev=445.77 lat (msec): min=463, max=2764, avg=1571.14, stdev=445.80 clat percentiles (msec): | 1.00th=[ 625], 5.00th=[ 776], 10.00th=[ 944], 20.00th=[ 1200], | 30.00th=[ 1334], 40.00th=[ 1452], 50.00th=[ 1569], 60.00th=[ 1703], | 70.00th=[ 1804], 80.00th=[ 1921], 90.00th=[ 2165], 95.00th=[ 2333], | 99.00th=[ 2601], 99.50th=[ 2635], 99.90th=[ 2769], 99.95th=[ 2769], | 99.99th=[ 2769] bw ( KiB/s): min= 7, max= 79, per=25.63%, avg=28.71, stdev=16.03, samples=355 iops : min= 1, max= 19, avg= 6.71, stdev= 4.03, samples=355 lat (msec) : 50=0.04%, 100=0.08%, 250=0.55%, 500=8.44%, 750=19.49% lat (msec) : 1000=21.21%, 2000=41.76%, >=2000=8.44% cpu : usr=0.06%, sys=0.35%, ctx=4059, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144264: Thu Aug 6 09:04:17 2026 read: IOPS=6, BW=27.3KiB/s (27.0kB/s)(5060KiB/185310msec) slat (usec): min=2750, max=99233, avg=19585.34, stdev=13006.26 clat (msec): min=18, max=2303, avg=721.92, stdev=235.14 lat (msec): min=33, max=2324, avg=741.51, stdev=238.34 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 321], 10.00th=[ 397], 20.00th=[ 518], | 30.00th=[ 625], 40.00th=[ 693], 50.00th=[ 735], 60.00th=[ 785], | 70.00th=[ 835], 80.00th=[ 894], 90.00th=[ 995], 95.00th=[ 1062], | 99.00th=[ 1234], 99.50th=[ 1334], 99.90th=[ 2265], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 111, per=26.91%, avg=28.79, stdev=16.27, samples=342 iops : min= 1, max= 27, avg= 6.73, stdev= 4.09, samples=342 write: IOPS=6, BW=27.0KiB/s (28.6kB/s)(5180KiB/185310msec); 0 zone resets slat (usec): min=51, max=23957, avg=334.06, stdev=1380.69 clat (msec): min=297, max=3239, avg=1561.57, stdev=438.41 lat (msec): min=304, max=3239, avg=1561.90, stdev=438.42 clat percentiles (msec): | 1.00th=[ 600], 5.00th=[ 751], 10.00th=[ 953], 20.00th=[ 1217], | 30.00th=[ 1368], 40.00th=[ 1469], 50.00th=[ 1586], 60.00th=[ 1687], | 70.00th=[ 1787], 80.00th=[ 1921], 90.00th=[ 2106], 95.00th=[ 2265], | 99.00th=[ 2567], 99.50th=[ 2601], 99.90th=[ 3239], 99.95th=[ 3239], | 99.99th=[ 3239] bw ( KiB/s): min= 7, max= 88, per=25.50%, avg=28.56, stdev=15.60, samples=352 iops : min= 1, max= 22, avg= 6.68, stdev= 3.95, samples=352 lat (msec) : 20=0.04%, 100=0.20%, 250=0.47%, 500=8.44%, 750=19.22% lat (msec) : 1000=22.15%, 2000=42.11%, >=2000=7.38% cpu : usr=0.06%, sys=0.35%, ctx=4300, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=108KiB/s (110kB/s), 26.7KiB/s-27.3KiB/s (27.3kB/s-27.0kB/s), io=19.6MiB (20.5MB), run=185310-186302msec WRITE: bw=112KiB/s (115kB/s), 27.0KiB/s-28.3KiB/s (28.6kB/s-28.9kB/s), io=20.4MiB (21.4MB), run=185310-186302msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144298: Thu Aug 6 09:04:30 2026 read: IOPS=1, BW=53.7MiB/s (56.3MB/s)(40.0MiB/745msec) slat (nsec): min=19576k, max=19576k, avg=19576246.00, stdev= 0.00 clat (nsec): min=710586k, max=710586k, avg=710585702.00, stdev= 0.00 lat (nsec): min=730166k, max=730166k, avg=730165740.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 709], 5.00th=[ 709], 10.00th=[ 709], 20.00th=[ 709], | 30.00th=[ 709], 40.00th=[ 709], 50.00th=[ 709], 60.00th=[ 709], | 70.00th=[ 709], 80.00th=[ 709], 90.00th=[ 709], 95.00th=[ 709], | 99.00th=[ 709], 99.50th=[ 709], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.34%, ctx=4, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=53.7MiB/s (56.3MB/s), 53.7MiB/s-53.7MiB/s (56.3MB/s-56.3MB/s), io=40.0MiB (41.9MB), run=745-745msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout