== sanity test 398c: run fio to test AIO ================= 01:48:46 (1787377726) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.404373 s, 104 MB/s osc.lustre-OST0000-osc-ffff8bb7ac777000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145704: Sat Aug 22 01:51:54 2026 write: IOPS=14, BW=57.7KiB/s (59.1kB/s)(10.0MiB/177400msec); 0 zone resets slat (usec): min=21, max=17197, avg=243.47, stdev=922.49 clat (msec): min=252, max=2366, avg=1107.58, stdev=306.46 lat (msec): min=252, max=2366, avg=1107.83, stdev=306.47 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 617], 10.00th=[ 768], 20.00th=[ 902], | 30.00th=[ 978], 40.00th=[ 1036], 50.00th=[ 1099], 60.00th=[ 1167], | 70.00th=[ 1250], 80.00th=[ 1334], 90.00th=[ 1469], 95.00th=[ 1603], | 99.00th=[ 1905], 99.50th=[ 2005], 99.90th=[ 2232], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 255, per=25.03%, avg=57.56, stdev=30.24, samples=349 iops : min= 1, max= 63, avg=13.87, stdev= 7.57, samples=349 lat (msec) : 500=3.79%, 750=5.12%, 1000=23.95%, 2000=66.64%, >=2000=0.51% cpu : usr=0.05%, sys=0.23%, ctx=1872, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145705: Sat Aug 22 01:51:54 2026 write: IOPS=14, BW=57.9KiB/s (59.2kB/s)(10.0MiB/176997msec); 0 zone resets slat (usec): min=23, max=17252, avg=218.44, stdev=784.12 clat (msec): min=158, max=2336, avg=1105.66, stdev=311.93 lat (msec): min=158, max=2336, avg=1105.88, stdev=311.96 clat percentiles (msec): | 1.00th=[ 253], 5.00th=[ 592], 10.00th=[ 776], 20.00th=[ 902], | 30.00th=[ 986], 40.00th=[ 1036], 50.00th=[ 1099], 60.00th=[ 1167], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1485], 95.00th=[ 1653], | 99.00th=[ 1905], 99.50th=[ 1972], 99.90th=[ 2123], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 255, per=25.14%, avg=57.82, stdev=31.46, samples=347 iops : min= 1, max= 63, avg=13.94, stdev= 7.87, samples=347 lat (msec) : 250=0.66%, 500=3.55%, 750=4.38%, 1000=23.52%, 2000=67.58% lat (msec) : >=2000=0.31% cpu : usr=0.05%, sys=0.21%, ctx=1574, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145706: Sat Aug 22 01:51:54 2026 write: IOPS=14, BW=57.8KiB/s (59.2kB/s)(10.0MiB/177168msec); 0 zone resets slat (usec): min=22, max=16013, avg=234.42, stdev=858.08 clat (msec): min=136, max=2340, avg=1106.42, stdev=308.24 lat (msec): min=136, max=2340, avg=1106.66, stdev=308.23 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 609], 10.00th=[ 785], 20.00th=[ 911], | 30.00th=[ 986], 40.00th=[ 1045], 50.00th=[ 1099], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1318], 90.00th=[ 1469], 95.00th=[ 1586], | 99.00th=[ 1921], 99.50th=[ 1972], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 255, per=24.94%, avg=57.37, stdev=30.37, samples=350 iops : min= 1, max= 63, avg=13.82, stdev= 7.60, samples=350 lat (msec) : 250=1.05%, 500=3.01%, 750=4.49%, 1000=23.91%, 2000=67.11% lat (msec) : >=2000=0.43% cpu : usr=0.05%, sys=0.23%, ctx=1850, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145707: Sat Aug 22 01:51:54 2026 write: IOPS=14, BW=57.8KiB/s (59.2kB/s)(10.0MiB/177244msec); 0 zone resets slat (usec): min=22, max=15828, avg=202.70, stdev=686.13 clat (msec): min=260, max=2346, avg=1106.76, stdev=310.13 lat (msec): min=260, max=2346, avg=1106.97, stdev=310.14 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 617], 10.00th=[ 776], 20.00th=[ 902], | 30.00th=[ 986], 40.00th=[ 1036], 50.00th=[ 1099], 60.00th=[ 1167], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1469], 95.00th=[ 1636], | 99.00th=[ 1921], 99.50th=[ 2005], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 255, per=25.42%, avg=58.48, stdev=31.59, samples=343 iops : min= 1, max= 63, avg=14.10, stdev= 7.91, samples=343 lat (msec) : 500=3.95%, 750=4.69%, 1000=24.73%, 2000=66.21%, >=2000=0.43% cpu : usr=0.05%, sys=0.23%, ctx=1782, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=231KiB/s (236kB/s), 57.7KiB/s-57.9KiB/s (59.1kB/s-59.2kB/s), io=40.0MiB (41.9MB), run=176997-177400msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145744: Sat Aug 22 01:54:11 2026 read: IOPS=9, BW=39.3KiB/s (40.3kB/s)(5048KiB/128392msec) slat (usec): min=31, max=88545, avg=16844.59, stdev=10035.78 clat (msec): min=15, max=1503, avg=512.43, stdev=194.64 lat (msec): min=34, max=1511, avg=529.29, stdev=197.36 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 266], 10.00th=[ 305], 20.00th=[ 359], | 30.00th=[ 397], 40.00th=[ 435], 50.00th=[ 477], 60.00th=[ 535], | 70.00th=[ 592], 80.00th=[ 667], 90.00th=[ 760], 95.00th=[ 852], | 99.00th=[ 1200], 99.50th=[ 1234], 99.90th=[ 1334], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 7, max= 119, per=25.38%, avg=39.34, stdev=21.69, samples=250 iops : min= 1, max= 29, avg= 9.30, stdev= 5.42, samples=250 write: IOPS=10, BW=40.4KiB/s (41.4kB/s)(5192KiB/128392msec); 0 zone resets slat (usec): min=25, max=11020, avg=257.72, stdev=726.48 clat (msec): min=217, max=2827, avg=1066.14, stdev=401.19 lat (msec): min=218, max=2827, avg=1066.41, stdev=401.16 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 542], 10.00th=[ 617], 20.00th=[ 743], | 30.00th=[ 827], 40.00th=[ 902], 50.00th=[ 978], 60.00th=[ 1083], | 70.00th=[ 1234], 80.00th=[ 1401], 90.00th=[ 1603], 95.00th=[ 1770], | 99.00th=[ 2198], 99.50th=[ 2534], 99.90th=[ 2836], 99.95th=[ 2836], | 99.99th=[ 2836] bw ( KiB/s): min= 7, max= 111, per=25.05%, avg=40.33, stdev=20.41, samples=251 iops : min= 1, max= 27, avg= 9.55, stdev= 5.14, samples=251 lat (msec) : 20=0.04%, 50=0.12%, 100=0.20%, 250=1.76%, 500=26.33% lat (msec) : 750=25.94%, 1000=20.16%, 2000=23.98%, >=2000=1.48% cpu : usr=0.11%, sys=0.55%, ctx=4420, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145745: Sat Aug 22 01:54:11 2026 read: IOPS=9, BW=38.5KiB/s (39.5kB/s)(4976KiB/129102msec) slat (usec): min=3159, max=75097, avg=16993.58, stdev=10200.15 clat (msec): min=23, max=1471, avg=503.05, stdev=186.27 lat (msec): min=29, max=1482, avg=520.05, stdev=189.06 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 271], 10.00th=[ 305], 20.00th=[ 359], | 30.00th=[ 397], 40.00th=[ 430], 50.00th=[ 464], 60.00th=[ 506], | 70.00th=[ 567], 80.00th=[ 642], 90.00th=[ 743], 95.00th=[ 860], | 99.00th=[ 1116], 99.50th=[ 1217], 99.90th=[ 1452], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 7, max= 111, per=25.41%, avg=39.38, stdev=21.55, samples=246 iops : min= 1, max= 27, avg= 9.30, stdev= 5.41, samples=246 write: IOPS=10, BW=40.8KiB/s (41.8kB/s)(5264KiB/129102msec); 0 zone resets slat (usec): min=23, max=20572, avg=305.47, stdev=1135.55 clat (msec): min=360, max=2803, avg=1073.96, stdev=400.19 lat (msec): min=361, max=2803, avg=1074.27, stdev=400.20 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 542], 10.00th=[ 617], 20.00th=[ 751], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 986], 60.00th=[ 1099], | 70.00th=[ 1217], 80.00th=[ 1401], 90.00th=[ 1620], 95.00th=[ 1804], | 99.00th=[ 2232], 99.50th=[ 2467], 99.90th=[ 2802], 99.95th=[ 2802], | 99.99th=[ 2802] bw ( KiB/s): min= 7, max= 103, per=25.67%, avg=41.33, stdev=19.95, samples=249 iops : min= 1, max= 25, avg= 9.80, stdev= 5.03, samples=249 lat (msec) : 50=0.12%, 100=0.08%, 250=1.37%, 500=28.36%, 750=24.26% lat (msec) : 1000=19.84%, 2000=24.45%, >=2000=1.52% cpu : usr=0.09%, sys=0.56%, ctx=4127, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145746: Sat Aug 22 01:54:11 2026 read: IOPS=9, BW=38.7KiB/s (39.6kB/s)(4984KiB/128907msec) slat (usec): min=3244, max=75137, avg=16841.65, stdev=10048.03 clat (msec): min=20, max=1503, avg=501.51, stdev=189.32 lat (msec): min=38, max=1511, avg=518.36, stdev=192.42 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 266], 10.00th=[ 296], 20.00th=[ 347], | 30.00th=[ 388], 40.00th=[ 430], 50.00th=[ 464], 60.00th=[ 510], | 70.00th=[ 575], 80.00th=[ 642], 90.00th=[ 751], 95.00th=[ 869], | 99.00th=[ 1070], 99.50th=[ 1200], 99.90th=[ 1318], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 7, max= 119, per=25.46%, avg=39.46, stdev=22.97, samples=247 iops : min= 1, max= 29, avg= 9.33, stdev= 5.77, samples=247 write: IOPS=10, BW=40.8KiB/s (41.8kB/s)(5256KiB/128907msec); 0 zone resets slat (usec): min=30, max=25442, avg=303.57, stdev=1235.83 clat (msec): min=344, max=2804, avg=1073.97, stdev=395.97 lat (msec): min=345, max=2804, avg=1074.27, stdev=395.95 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 558], 10.00th=[ 634], 20.00th=[ 743], | 30.00th=[ 827], 40.00th=[ 902], 50.00th=[ 995], 60.00th=[ 1099], | 70.00th=[ 1234], 80.00th=[ 1418], 90.00th=[ 1603], 95.00th=[ 1821], | 99.00th=[ 2198], 99.50th=[ 2265], 99.90th=[ 2802], 99.95th=[ 2802], | 99.99th=[ 2802] bw ( KiB/s): min= 7, max= 120, per=25.35%, avg=40.81, stdev=20.41, samples=250 iops : min= 1, max= 30, avg= 9.66, stdev= 5.14, samples=250 lat (msec) : 50=0.16%, 100=0.12%, 250=1.60%, 500=28.05%, 750=25.00% lat (msec) : 1000=18.71%, 2000=24.88%, >=2000=1.48% cpu : usr=0.09%, sys=0.55%, ctx=4237, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145747: Sat Aug 22 01:54:11 2026 read: IOPS=9, BW=39.4KiB/s (40.3kB/s)(5060KiB/128494msec) slat (usec): min=3046, max=97855, avg=16375.54, stdev=10092.33 clat (msec): min=44, max=1340, avg=508.20, stdev=187.64 lat (msec): min=50, max=1377, avg=524.58, stdev=190.50 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 275], 10.00th=[ 300], 20.00th=[ 355], | 30.00th=[ 397], 40.00th=[ 435], 50.00th=[ 477], 60.00th=[ 514], | 70.00th=[ 575], 80.00th=[ 651], 90.00th=[ 760], 95.00th=[ 860], | 99.00th=[ 1200], 99.50th=[ 1267], 99.90th=[ 1334], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 7, max= 119, per=25.43%, avg=39.41, stdev=20.55, samples=251 iops : min= 1, max= 29, avg= 9.31, stdev= 5.15, samples=251 write: IOPS=10, BW=40.3KiB/s (41.3kB/s)(5180KiB/128494msec); 0 zone resets slat (usec): min=24, max=23304, avg=282.92, stdev=1061.21 clat (msec): min=250, max=2778, avg=1071.34, stdev=392.17 lat (msec): min=251, max=2778, avg=1071.62, stdev=392.14 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 550], 10.00th=[ 642], 20.00th=[ 743], | 30.00th=[ 835], 40.00th=[ 911], 50.00th=[ 995], 60.00th=[ 1099], | 70.00th=[ 1234], 80.00th=[ 1401], 90.00th=[ 1586], 95.00th=[ 1787], | 99.00th=[ 2198], 99.50th=[ 2433], 99.90th=[ 2635], 99.95th=[ 2769], | 99.99th=[ 2769] bw ( KiB/s): min= 7, max= 104, per=25.34%, avg=40.80, stdev=20.48, samples=247 iops : min= 1, max= 26, avg= 9.66, stdev= 5.15, samples=247 lat (msec) : 50=0.12%, 100=0.04%, 250=1.25%, 500=28.55%, 750=24.80% lat (msec) : 1000=19.80%, 2000=24.06%, >=2000=1.37% cpu : usr=0.10%, sys=0.60%, ctx=4549, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=155KiB/s (159kB/s), 38.5KiB/s-39.4KiB/s (39.5kB/s-40.3kB/s), io=19.6MiB (20.5MB), run=128392-129102msec WRITE: bw=162KiB/s (166kB/s), 40.3KiB/s-40.8KiB/s (41.3kB/s-41.8kB/s), io=20.4MiB (21.4MB), run=128392-129102msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145772: Sat Aug 22 01:54:22 2026 read: IOPS=1, BW=52.8MiB/s (55.3MB/s)(40.0MiB/758msec) slat (nsec): min=42104k, max=42104k, avg=42103752.00, stdev= 0.00 clat (nsec): min=676697k, max=676697k, avg=676696607.00, stdev= 0.00 lat (nsec): min=718805k, max=718805k, avg=718804752.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 676], 5.00th=[ 676], 10.00th=[ 676], 20.00th=[ 676], | 30.00th=[ 676], 40.00th=[ 676], 50.00th=[ 676], 60.00th=[ 676], | 70.00th=[ 676], 80.00th=[ 676], 90.00th=[ 676], 95.00th=[ 676], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] lat (msec) : 750=100.00% cpu : usr=0.13%, sys=1.45%, ctx=32, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=52.8MiB/s (55.3MB/s), 52.8MiB/s-52.8MiB/s (55.3MB/s-55.3MB/s), io=40.0MiB (41.9MB), run=758-758msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout