== sanity test 398c: run fio to test AIO ================= 22:23:08 (1781230988) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.759297 s, 55.2 MB/s osc.lustre-OST0000-osc-ffff925698818000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146898: Thu Jun 11 22:26:01 2026 write: IOPS=15, BW=62.0KiB/s (64.5kB/s)(10.0MiB/162660msec); 0 zone resets slat (usec): min=37, max=37057, avg=313.05, stdev=1327.34 clat (msec): min=58, max=2116, avg=1015.82, stdev=295.80 lat (msec): min=58, max=2116, avg=1016.13, stdev=295.82 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 625], 10.00th=[ 718], 20.00th=[ 793], | 30.00th=[ 852], 40.00th=[ 919], 50.00th=[ 969], 60.00th=[ 1036], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1418], 95.00th=[ 1586], | 99.00th=[ 1804], 99.50th=[ 1871], 99.90th=[ 1989], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 230, per=25.13%, avg=63.07, stdev=28.98, samples=319 iops : min= 1, max= 57, avg=15.32, stdev= 7.26, samples=319 lat (msec) : 100=0.04%, 250=0.98%, 500=2.03%, 750=10.47%, 1000=40.39% lat (msec) : 2000=46.02%, >=2000=0.08% cpu : usr=0.05%, sys=0.27%, ctx=2117, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146899: Thu Jun 11 22:26:01 2026 write: IOPS=15, BW=62.9KiB/s (64.4kB/s)(10.0MiB/162714msec); 0 zone resets slat (usec): min=37, max=28294, avg=255.64, stdev=980.73 clat (msec): min=180, max=1982, avg=1015.82, stdev=290.28 lat (msec): min=180, max=1982, avg=1016.08, stdev=290.30 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 625], 10.00th=[ 718], 20.00th=[ 802], | 30.00th=[ 860], 40.00th=[ 911], 50.00th=[ 969], 60.00th=[ 1045], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1418], 95.00th=[ 1569], | 99.00th=[ 1787], 99.50th=[ 1905], 99.90th=[ 1955], 99.95th=[ 1955], | 99.99th=[ 1989] bw ( KiB/s): min= 8, max= 222, per=25.56%, avg=64.14, stdev=31.24, samples=314 iops : min= 2, max= 55, avg=15.58, stdev= 7.82, samples=314 lat (msec) : 250=0.82%, 500=2.27%, 750=10.16%, 1000=40.82%, 2000=45.94% cpu : usr=0.04%, sys=0.27%, ctx=1844, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146900: Thu Jun 11 22:26:01 2026 write: IOPS=15, BW=62.9KiB/s (64.4kB/s)(10.0MiB/162713msec); 0 zone resets slat (usec): min=37, max=24679, avg=274.45, stdev=1095.60 clat (msec): min=150, max=1979, avg=1015.47, stdev=290.62 lat (msec): min=150, max=1979, avg=1015.75, stdev=290.64 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 642], 10.00th=[ 718], 20.00th=[ 802], | 30.00th=[ 852], 40.00th=[ 911], 50.00th=[ 969], 60.00th=[ 1053], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1385], 95.00th=[ 1552], | 99.00th=[ 1821], 99.50th=[ 1888], 99.90th=[ 1972], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 225, per=25.70%, avg=64.51, stdev=30.37, samples=312 iops : min= 1, max= 56, avg=15.68, stdev= 7.60, samples=312 lat (msec) : 250=1.25%, 500=1.45%, 750=10.47%, 1000=41.29%, 2000=45.55% cpu : usr=0.04%, sys=0.28%, ctx=1823, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146901: Thu Jun 11 22:26:01 2026 write: IOPS=15, BW=62.0KiB/s (64.5kB/s)(10.0MiB/162648msec); 0 zone resets slat (usec): min=39, max=20456, avg=280.10, stdev=1074.86 clat (msec): min=161, max=2041, avg=1015.10, stdev=300.46 lat (msec): min=161, max=2057, avg=1015.38, stdev=300.54 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 617], 10.00th=[ 709], 20.00th=[ 793], | 30.00th=[ 844], 40.00th=[ 902], 50.00th=[ 969], 60.00th=[ 1045], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1418], 95.00th=[ 1603], | 99.00th=[ 1854], 99.50th=[ 1905], 99.90th=[ 1989], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 226, per=25.21%, avg=63.28, stdev=29.02, samples=318 iops : min= 1, max= 56, avg=15.37, stdev= 7.27, samples=318 lat (msec) : 250=1.25%, 500=2.07%, 750=10.55%, 1000=40.08%, 2000=45.98% lat (msec) : >=2000=0.08% cpu : usr=0.05%, sys=0.26%, ctx=2035, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=252KiB/s (258kB/s), 62.9KiB/s-62.0KiB/s (64.4kB/s-64.5kB/s), io=40.0MiB (41.9MB), run=162648-162714msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146935: Thu Jun 11 22:27:56 2026 read: IOPS=11, BW=46.9KiB/s (48.0kB/s)(5048KiB/107597msec) slat (usec): min=2668, max=60888, avg=15824.47, stdev=9384.46 clat (msec): min=15, max=1163, avg=427.19, stdev=147.53 lat (msec): min=19, max=1169, avg=443.02, stdev=150.28 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 220], 10.00th=[ 264], 20.00th=[ 313], | 30.00th=[ 347], 40.00th=[ 380], 50.00th=[ 414], 60.00th=[ 447], | 70.00th=[ 485], 80.00th=[ 527], 90.00th=[ 617], 95.00th=[ 693], | 99.00th=[ 885], 99.50th=[ 927], 99.90th=[ 1062], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 8, max= 119, per=24.90%, avg=46.32, stdev=23.26, samples=213 iops : min= 2, max= 29, avg=11.07, stdev= 5.85, samples=213 write: IOPS=12, BW=48.3KiB/s (49.4kB/s)(5192KiB/107597msec); 0 zone resets slat (usec): min=44, max=29938, avg=405.54, stdev=1422.67 clat (msec): min=300, max=2033, avg=891.44, stdev=273.45 lat (msec): min=300, max=2033, avg=891.85, stdev=273.49 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 468], 10.00th=[ 550], 20.00th=[ 659], | 30.00th=[ 743], 40.00th=[ 810], 50.00th=[ 869], 60.00th=[ 944], | 70.00th=[ 1011], 80.00th=[ 1099], 90.00th=[ 1234], 95.00th=[ 1334], | 99.00th=[ 1737], 99.50th=[ 1787], 99.90th=[ 1938], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 104, per=24.80%, avg=48.11, stdev=21.41, samples=211 iops : min= 1, max= 26, avg=11.51, stdev= 5.41, samples=211 lat (msec) : 20=0.04%, 50=0.16%, 100=0.12%, 250=3.55%, 500=35.70% lat (msec) : 750=24.06%, 1000=19.73%, 2000=16.60%, >=2000=0.04% cpu : usr=0.10%, sys=0.71%, ctx=4276, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146936: Thu Jun 11 22:27:56 2026 read: IOPS=11, BW=46.2KiB/s (47.4kB/s)(4976KiB/107591msec) slat (usec): min=2799, max=71017, avg=15958.63, stdev=9437.57 clat (msec): min=18, max=1172, avg=428.53, stdev=153.24 lat (msec): min=30, max=1177, avg=444.50, stdev=155.80 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 218], 10.00th=[ 259], 20.00th=[ 309], | 30.00th=[ 338], 40.00th=[ 380], 50.00th=[ 409], 60.00th=[ 443], | 70.00th=[ 481], 80.00th=[ 542], 90.00th=[ 625], 95.00th=[ 709], | 99.00th=[ 911], 99.50th=[ 936], 99.90th=[ 1083], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 7, max= 112, per=24.86%, avg=46.23, stdev=22.61, samples=210 iops : min= 1, max= 28, avg=11.05, stdev= 5.73, samples=210 write: IOPS=12, BW=48.9KiB/s (50.1kB/s)(5264KiB/107591msec); 0 zone resets slat (usec): min=42, max=20359, avg=406.88, stdev=1280.28 clat (msec): min=246, max=2003, avg=884.36, stdev=278.12 lat (msec): min=246, max=2011, avg=884.77, stdev=278.18 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 451], 10.00th=[ 550], 20.00th=[ 651], | 30.00th=[ 726], 40.00th=[ 810], 50.00th=[ 869], 60.00th=[ 944], | 70.00th=[ 1003], 80.00th=[ 1099], 90.00th=[ 1234], 95.00th=[ 1385], | 99.00th=[ 1636], 99.50th=[ 1720], 99.90th=[ 1938], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 119, per=24.94%, avg=48.38, stdev=23.24, samples=213 iops : min= 1, max= 29, avg=11.59, stdev= 5.85, samples=213 lat (msec) : 20=0.04%, 50=0.23%, 250=3.95%, 500=35.12%, 750=24.57% lat (msec) : 1000=20.12%, 2000=15.94%, >=2000=0.04% cpu : usr=0.09%, sys=0.71%, ctx=4461, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146937: Thu Jun 11 22:27:56 2026 read: IOPS=11, BW=46.3KiB/s (47.5kB/s)(4984KiB/107549msec) slat (usec): min=61, max=88815, avg=15808.58, stdev=9611.43 clat (msec): min=36, max=1410, avg=428.36, stdev=149.93 lat (msec): min=41, max=1428, avg=444.18, stdev=152.58 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 220], 10.00th=[ 264], 20.00th=[ 309], | 30.00th=[ 347], 40.00th=[ 380], 50.00th=[ 414], 60.00th=[ 443], | 70.00th=[ 481], 80.00th=[ 531], 90.00th=[ 625], 95.00th=[ 709], | 99.00th=[ 885], 99.50th=[ 927], 99.90th=[ 1053], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 7, max= 151, per=24.68%, avg=45.90, stdev=23.25, samples=213 iops : min= 1, max= 37, avg=10.96, stdev= 5.88, samples=213 write: IOPS=12, BW=48.9KiB/s (50.0kB/s)(5256KiB/107549msec); 0 zone resets slat (usec): min=55, max=18588, avg=404.14, stdev=1202.12 clat (msec): min=135, max=2405, avg=887.04, stdev=288.31 lat (msec): min=136, max=2406, avg=887.45, stdev=288.33 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 472], 10.00th=[ 550], 20.00th=[ 659], | 30.00th=[ 735], 40.00th=[ 802], 50.00th=[ 869], 60.00th=[ 936], | 70.00th=[ 1011], 80.00th=[ 1099], 90.00th=[ 1234], 95.00th=[ 1351], | 99.00th=[ 1787], 99.50th=[ 2005], 99.90th=[ 2265], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 112, per=24.89%, avg=48.28, stdev=21.74, samples=212 iops : min= 1, max= 28, avg=11.56, stdev= 5.48, samples=212 lat (msec) : 50=0.12%, 100=0.08%, 250=4.06%, 500=35.35%, 750=24.06% lat (msec) : 1000=20.31%, 2000=15.82%, >=2000=0.20% cpu : usr=0.10%, sys=0.74%, ctx=4801, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146938: Thu Jun 11 22:27:56 2026 read: IOPS=11, BW=47.1KiB/s (48.2kB/s)(5060KiB/107459msec) slat (usec): min=2877, max=72853, avg=15657.28, stdev=9691.95 clat (msec): min=19, max=1480, avg=424.83, stdev=154.61 lat (msec): min=28, max=1490, avg=440.49, stdev=156.94 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 215], 10.00th=[ 255], 20.00th=[ 305], | 30.00th=[ 338], 40.00th=[ 376], 50.00th=[ 409], 60.00th=[ 443], | 70.00th=[ 477], 80.00th=[ 523], 90.00th=[ 617], 95.00th=[ 701], | 99.00th=[ 911], 99.50th=[ 961], 99.90th=[ 1301], 99.95th=[ 1485], | 99.99th=[ 1485] bw ( KiB/s): min= 7, max= 128, per=25.64%, avg=47.70, stdev=22.07, samples=208 iops : min= 1, max= 32, avg=11.42, stdev= 5.55, samples=208 write: IOPS=12, BW=48.2KiB/s (49.4kB/s)(5180KiB/107459msec); 0 zone resets slat (usec): min=37, max=12728, avg=340.54, stdev=849.22 clat (msec): min=225, max=2930, avg=893.72, stdev=306.65 lat (msec): min=226, max=2930, avg=894.07, stdev=306.66 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 460], 10.00th=[ 550], 20.00th=[ 642], | 30.00th=[ 726], 40.00th=[ 810], 50.00th=[ 877], 60.00th=[ 936], | 70.00th=[ 1011], 80.00th=[ 1099], 90.00th=[ 1250], 95.00th=[ 1368], | 99.00th=[ 1787], 99.50th=[ 2333], 99.90th=[ 2937], 99.95th=[ 2937], | 99.99th=[ 2937] bw ( KiB/s): min= 7, max= 111, per=25.16%, avg=48.81, stdev=20.91, samples=207 iops : min= 1, max= 27, avg=11.69, stdev= 5.30, samples=207 lat (msec) : 20=0.04%, 50=0.12%, 100=0.16%, 250=4.41%, 500=35.27% lat (msec) : 750=24.53%, 1000=19.49%, 2000=15.70%, >=2000=0.27% cpu : usr=0.11%, sys=0.68%, ctx=4379, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=187KiB/s (191kB/s), 46.2KiB/s-47.1KiB/s (47.4kB/s-48.2kB/s), io=19.6MiB (20.5MB), run=107459-107597msec WRITE: bw=194KiB/s (199kB/s), 48.2KiB/s-48.9KiB/s (49.4kB/s-50.1kB/s), io=20.4MiB (21.4MB), run=107459-107597msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146958: Thu Jun 11 22:28:10 2026 read: IOPS=1, BW=59.1MiB/s (61.0MB/s)(40.0MiB/677msec) slat (nsec): min=66570k, max=66570k, avg=66570419.00, stdev= 0.00 clat (nsec): min=594487k, max=594487k, avg=594486557.00, stdev= 0.00 lat (nsec): min=661062k, max=661062k, avg=661061642.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 592], 5.00th=[ 592], 10.00th=[ 592], 20.00th=[ 592], | 30.00th=[ 592], 40.00th=[ 592], 50.00th=[ 592], 60.00th=[ 592], | 70.00th=[ 592], 80.00th=[ 592], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.48%, ctx=38, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=59.1MiB/s (61.0MB/s), 59.1MiB/s-59.1MiB/s (61.0MB/s-61.0MB/s), io=40.0MiB (41.9MB), run=677-677msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout