== sanity test 398c: run fio to test AIO ================= 22:32:55 (1787797975) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.383417 s, 109 MB/s osc.lustre-OST0000-osc-ffff9a5ac60e9800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145632: Wed Aug 26 22:35:44 2026 write: IOPS=15, BW=63.9KiB/s (65.4kB/s)(10.0MiB/160358msec); 0 zone resets slat (usec): min=23, max=16789, avg=237.69, stdev=907.88 clat (msec): min=165, max=2336, avg=1001.15, stdev=272.98 lat (msec): min=166, max=2337, avg=1001.39, stdev=272.97 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 625], 10.00th=[ 726], 20.00th=[ 810], | 30.00th=[ 869], 40.00th=[ 927], 50.00th=[ 1003], 60.00th=[ 1053], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1720], 99.50th=[ 2089], 99.90th=[ 2299], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 219, per=24.80%, avg=63.25, stdev=28.99, samples=318 iops : min= 1, max= 54, avg=15.35, stdev= 7.25, samples=318 lat (msec) : 250=1.41%, 500=2.73%, 750=8.24%, 1000=38.16%, 2000=48.87% lat (msec) : >=2000=0.59% cpu : usr=0.05%, sys=0.24%, ctx=1765, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145633: Wed Aug 26 22:35:44 2026 write: IOPS=15, BW=63.8KiB/s (65.4kB/s)(10.0MiB/160404msec); 0 zone resets slat (usec): min=21, max=17951, avg=205.26, stdev=818.96 clat (msec): min=114, max=2296, avg=1002.00, stdev=271.55 lat (msec): min=114, max=2296, avg=1002.21, stdev=271.54 clat percentiles (msec): | 1.00th=[ 264], 5.00th=[ 617], 10.00th=[ 726], 20.00th=[ 810], | 30.00th=[ 869], 40.00th=[ 936], 50.00th=[ 1003], 60.00th=[ 1062], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1318], 95.00th=[ 1401], | 99.00th=[ 1821], 99.50th=[ 1955], 99.90th=[ 2198], 99.95th=[ 2265], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 232, per=24.95%, avg=63.63, stdev=31.88, samples=316 iops : min= 1, max= 58, avg=15.45, stdev= 7.98, samples=316 lat (msec) : 250=0.90%, 500=3.20%, 750=8.48%, 1000=36.76%, 2000=50.35% lat (msec) : >=2000=0.31% cpu : usr=0.04%, sys=0.23%, ctx=1755, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145634: Wed Aug 26 22:35:44 2026 write: IOPS=15, BW=63.9KiB/s (65.4kB/s)(10.0MiB/160363msec); 0 zone resets slat (usec): min=22, max=29603, avg=253.64, stdev=1115.02 clat (msec): min=220, max=2297, avg=1001.67, stdev=274.61 lat (msec): min=220, max=2297, avg=1001.92, stdev=274.65 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 617], 10.00th=[ 718], 20.00th=[ 802], | 30.00th=[ 869], 40.00th=[ 927], 50.00th=[ 1003], 60.00th=[ 1062], | 70.00th=[ 1116], 80.00th=[ 1183], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1871], 99.50th=[ 2106], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 224, per=25.03%, avg=63.83, stdev=31.62, samples=315 iops : min= 1, max= 56, avg=15.49, stdev= 7.92, samples=315 lat (msec) : 250=0.16%, 500=3.75%, 750=9.34%, 1000=36.68%, 2000=49.34% lat (msec) : >=2000=0.74% cpu : usr=0.05%, sys=0.25%, ctx=1855, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145635: Wed Aug 26 22:35:44 2026 write: IOPS=15, BW=63.9KiB/s (65.4kB/s)(10.0MiB/160272msec); 0 zone resets slat (usec): min=22, max=14975, avg=227.79, stdev=827.38 clat (msec): min=166, max=2297, avg=1000.52, stdev=278.16 lat (msec): min=166, max=2297, avg=1000.75, stdev=278.14 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 609], 10.00th=[ 709], 20.00th=[ 802], | 30.00th=[ 860], 40.00th=[ 936], 50.00th=[ 995], 60.00th=[ 1062], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1318], 95.00th=[ 1435], | 99.00th=[ 1854], 99.50th=[ 2089], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 253, per=25.19%, avg=64.25, stdev=31.06, samples=313 iops : min= 1, max= 63, avg=15.60, stdev= 7.81, samples=313 lat (msec) : 250=1.80%, 500=1.99%, 750=9.61%, 1000=37.50%, 2000=48.44% lat (msec) : >=2000=0.66% cpu : usr=0.06%, sys=0.23%, ctx=1786, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=255KiB/s (261kB/s), 63.8KiB/s-63.9KiB/s (65.4kB/s-65.4kB/s), io=40.0MiB (41.9MB), run=160272-160404msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145668: Wed Aug 26 22:37:48 2026 read: IOPS=10, BW=43.9KiB/s (44.0kB/s)(5048KiB/114904msec) slat (usec): min=3312, max=77308, avg=16130.74, stdev=9026.01 clat (msec): min=21, max=878, avg=447.47, stdev=123.00 lat (msec): min=31, max=898, avg=463.61, stdev=124.25 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 271], 10.00th=[ 309], 20.00th=[ 351], | 30.00th=[ 388], 40.00th=[ 414], 50.00th=[ 439], 60.00th=[ 464], | 70.00th=[ 498], 80.00th=[ 542], 90.00th=[ 600], 95.00th=[ 676], | 99.00th=[ 793], 99.50th=[ 802], 99.90th=[ 827], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 5, max= 103, per=24.92%, avg=43.36, stdev=19.88, samples=228 iops : min= 1, max= 25, avg=10.32, stdev= 4.97, samples=228 write: IOPS=11, BW=45.2KiB/s (46.3kB/s)(5192KiB/114904msec); 0 zone resets slat (usec): min=30, max=26807, avg=357.33, stdev=1375.63 clat (msec): min=376, max=2125, avg=961.03, stdev=228.27 lat (msec): min=382, max=2125, avg=961.39, stdev=228.30 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 600], 10.00th=[ 667], 20.00th=[ 768], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1011], | 70.00th=[ 1062], 80.00th=[ 1133], 90.00th=[ 1234], 95.00th=[ 1334], | 99.00th=[ 1536], 99.50th=[ 1670], 99.90th=[ 2089], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 5, max= 103, per=24.56%, avg=44.46, stdev=19.30, samples=228 iops : min= 1, max= 25, avg=10.59, stdev= 4.87, samples=228 lat (msec) : 50=0.20%, 100=0.23%, 250=1.05%, 500=33.71%, 750=22.23% lat (msec) : 1000=21.29%, 2000=21.13%, >=2000=0.16% cpu : usr=0.09%, sys=0.68%, ctx=4482, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145669: Wed Aug 26 22:37:48 2026 read: IOPS=10, BW=43.3KiB/s (44.3kB/s)(4976KiB/114971msec) slat (usec): min=3235, max=69532, avg=16473.55, stdev=9287.45 clat (msec): min=16, max=868, avg=446.78, stdev=119.16 lat (msec): min=22, max=888, avg=463.26, stdev=120.91 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 271], 10.00th=[ 305], 20.00th=[ 351], | 30.00th=[ 388], 40.00th=[ 414], 50.00th=[ 443], 60.00th=[ 468], | 70.00th=[ 498], 80.00th=[ 542], 90.00th=[ 592], 95.00th=[ 659], | 99.00th=[ 751], 99.50th=[ 785], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 7, max= 104, per=24.74%, avg=43.05, stdev=19.88, samples=226 iops : min= 1, max= 26, avg=10.24, stdev= 4.99, samples=226 write: IOPS=11, BW=45.8KiB/s (46.9kB/s)(5264KiB/114971msec); 0 zone resets slat (usec): min=31, max=42194, avg=335.61, stdev=1581.19 clat (msec): min=172, max=2048, avg=956.48, stdev=231.61 lat (msec): min=172, max=2048, avg=956.82, stdev=231.65 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 584], 10.00th=[ 651], 20.00th=[ 760], | 30.00th=[ 844], 40.00th=[ 902], 50.00th=[ 953], 60.00th=[ 1011], | 70.00th=[ 1062], 80.00th=[ 1133], 90.00th=[ 1234], 95.00th=[ 1334], | 99.00th=[ 1569], 99.50th=[ 1854], 99.90th=[ 1989], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 95, per=25.03%, avg=45.30, stdev=18.84, samples=227 iops : min= 1, max= 23, avg=10.81, stdev= 4.75, samples=227 lat (msec) : 20=0.08%, 50=0.08%, 100=0.31%, 250=1.13%, 500=33.48% lat (msec) : 750=22.30%, 1000=21.25%, 2000=21.33%, >=2000=0.04% cpu : usr=0.13%, sys=0.66%, ctx=4274, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145670: Wed Aug 26 22:37:48 2026 read: IOPS=10, BW=43.5KiB/s (44.5kB/s)(4984KiB/114626msec) slat (usec): min=55, max=67585, avg=16008.49, stdev=9032.50 clat (msec): min=11, max=1245, avg=443.86, stdev=124.76 lat (msec): min=41, max=1266, avg=459.88, stdev=126.29 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 275], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 376], 40.00th=[ 405], 50.00th=[ 430], 60.00th=[ 460], | 70.00th=[ 493], 80.00th=[ 535], 90.00th=[ 609], 95.00th=[ 676], | 99.00th=[ 760], 99.50th=[ 810], 99.90th=[ 1028], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 7, max= 112, per=25.06%, avg=43.60, stdev=20.11, samples=224 iops : min= 1, max= 28, avg=10.38, stdev= 5.06, samples=224 write: IOPS=11, BW=45.9KiB/s (46.0kB/s)(5256KiB/114626msec); 0 zone resets slat (usec): min=26, max=48022, avg=356.38, stdev=1715.41 clat (msec): min=175, max=2129, avg=958.00, stdev=239.81 lat (msec): min=177, max=2129, avg=958.36, stdev=239.79 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 584], 10.00th=[ 651], 20.00th=[ 760], | 30.00th=[ 844], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1011], | 70.00th=[ 1070], 80.00th=[ 1133], 90.00th=[ 1217], 95.00th=[ 1334], | 99.00th=[ 1636], 99.50th=[ 1854], 99.90th=[ 2056], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 111, per=25.19%, avg=45.60, stdev=19.68, samples=224 iops : min= 1, max= 27, avg=10.88, stdev= 4.93, samples=224 lat (msec) : 20=0.04%, 50=0.16%, 100=0.27%, 250=1.33%, 500=33.87% lat (msec) : 750=21.91%, 1000=20.82%, 2000=21.48%, >=2000=0.12% cpu : usr=0.10%, sys=0.65%, ctx=4334, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145671: Wed Aug 26 22:37:48 2026 read: IOPS=11, BW=44.1KiB/s (45.1kB/s)(5060KiB/114803msec) slat (usec): min=2985, max=63117, avg=15869.31, stdev=8951.15 clat (msec): min=18, max=1207, avg=449.77, stdev=121.76 lat (msec): min=33, max=1225, avg=465.65, stdev=123.02 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 279], 10.00th=[ 309], 20.00th=[ 359], | 30.00th=[ 388], 40.00th=[ 414], 50.00th=[ 439], 60.00th=[ 468], | 70.00th=[ 502], 80.00th=[ 542], 90.00th=[ 600], 95.00th=[ 667], | 99.00th=[ 768], 99.50th=[ 776], 99.90th=[ 1133], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 7, max= 111, per=25.69%, avg=44.70, stdev=19.58, samples=222 iops : min= 1, max= 27, avg=10.65, stdev= 4.91, samples=222 write: IOPS=11, BW=45.1KiB/s (46.2kB/s)(5180KiB/114803msec); 0 zone resets slat (usec): min=32, max=20544, avg=358.67, stdev=1245.85 clat (msec): min=311, max=2131, avg=959.55, stdev=240.49 lat (msec): min=311, max=2131, avg=959.91, stdev=240.54 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 584], 10.00th=[ 651], 20.00th=[ 751], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1020], | 70.00th=[ 1083], 80.00th=[ 1133], 90.00th=[ 1234], 95.00th=[ 1318], | 99.00th=[ 1737], 99.50th=[ 1854], 99.90th=[ 2123], 99.95th=[ 2140], | 99.99th=[ 2140] bw ( KiB/s): min= 7, max= 111, per=24.77%, avg=44.83, stdev=19.20, samples=225 iops : min= 1, max= 27, avg=10.68, stdev= 4.82, samples=225 lat (msec) : 20=0.04%, 50=0.12%, 100=0.23%, 250=1.05%, 500=33.71% lat (msec) : 750=23.59%, 1000=18.98%, 2000=22.19%, >=2000=0.08% cpu : usr=0.12%, sys=0.70%, ctx=4636, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=175KiB/s (179kB/s), 43.3KiB/s-44.1KiB/s (44.3kB/s-45.1kB/s), io=19.6MiB (20.5MB), run=114626-114971msec WRITE: bw=182KiB/s (186kB/s), 45.1KiB/s-45.9KiB/s (46.2kB/s-46.0kB/s), io=20.4MiB (21.4MB), run=114626-114971msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145691: Wed Aug 26 22:38:00 2026 read: IOPS=1, BW=51.3MiB/s (53.8MB/s)(40.0MiB/780msec) slat (nsec): min=59627k, max=59627k, avg=59626544.00, stdev= 0.00 clat (nsec): min=704130k, max=704130k, avg=704130498.00, stdev= 0.00 lat (nsec): min=763787k, max=763787k, avg=763787122.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 701], 5.00th=[ 701], 10.00th=[ 701], 20.00th=[ 701], | 30.00th=[ 701], 40.00th=[ 701], 50.00th=[ 701], 60.00th=[ 701], | 70.00th=[ 701], 80.00th=[ 701], 90.00th=[ 701], 95.00th=[ 701], | 99.00th=[ 701], 99.50th=[ 701], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 701] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.28%, ctx=44, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=51.3MiB/s (53.8MB/s), 51.3MiB/s-51.3MiB/s (53.8MB/s-53.8MB/s), io=40.0MiB (41.9MB), run=780-780msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout