== sanity test 398c: run fio to test AIO ================= 21:31:42 (1780968702) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.467251 s, 89.8 MB/s osc.lustre-OST0000-osc-ffff91384566c800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=150566: Mon Jun 8 21:32:23 2026 write: IOPS=92, BW=369KiB/s (378kB/s)(10.0MiB/27770msec); 0 zone resets slat (usec): min=26, max=18444, avg=202.04, stdev=910.71 clat (msec): min=8, max=313, avg=172.86, stdev=42.21 lat (msec): min=8, max=313, avg=173.06, stdev=42.21 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 112], 10.00th=[ 124], 20.00th=[ 136], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 190], 80.00th=[ 209], 90.00th=[ 234], 95.00th=[ 253], | 99.00th=[ 275], 99.50th=[ 279], 99.90th=[ 284], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 241, max= 502, per=24.94%, avg=367.58, stdev=72.85, samples=55 iops : min= 60, max= 125, avg=91.55, stdev=18.25, samples=55 lat (msec) : 10=0.04%, 20=0.04%, 50=0.04%, 100=1.33%, 250=93.24% lat (msec) : 500=5.31% cpu : usr=0.22%, sys=0.83%, ctx=1015, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=150567: Mon Jun 8 21:32:23 2026 write: IOPS=92, BW=369KiB/s (378kB/s)(10.0MiB/27714msec); 0 zone resets slat (usec): min=19, max=29100, avg=213.02, stdev=984.00 clat (msec): min=17, max=307, avg=172.69, stdev=41.66 lat (msec): min=17, max=307, avg=172.90, stdev=41.62 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 113], 10.00th=[ 124], 20.00th=[ 138], | 30.00th=[ 150], 40.00th=[ 157], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 192], 80.00th=[ 209], 90.00th=[ 234], 95.00th=[ 247], | 99.00th=[ 279], 99.50th=[ 288], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 227, max= 512, per=24.87%, avg=366.59, stdev=75.81, samples=54 iops : min= 56, max= 128, avg=91.28, stdev=19.00, samples=54 lat (msec) : 20=0.04%, 50=0.16%, 100=1.56%, 250=94.02%, 500=4.22% cpu : usr=0.19%, sys=0.95%, ctx=1081, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=150568: Mon Jun 8 21:32:23 2026 write: IOPS=92, BW=369KiB/s (378kB/s)(10.0MiB/27773msec); 0 zone resets slat (usec): min=18, max=23264, avg=239.97, stdev=1104.79 clat (msec): min=31, max=293, avg=172.60, stdev=42.45 lat (msec): min=32, max=293, avg=172.84, stdev=42.46 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 112], 10.00th=[ 124], 20.00th=[ 140], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 192], 80.00th=[ 211], 90.00th=[ 228], 95.00th=[ 259], | 99.00th=[ 279], 99.50th=[ 284], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 227, max= 534, per=24.92%, avg=367.36, stdev=71.61, samples=55 iops : min= 56, max= 133, avg=91.49, stdev=17.89, samples=55 lat (msec) : 50=0.12%, 100=1.56%, 250=91.60%, 500=6.72% cpu : usr=0.28%, sys=0.94%, ctx=1025, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=150569: Mon Jun 8 21:32:23 2026 write: IOPS=92, BW=369KiB/s (377kB/s)(10.0MiB/27778msec); 0 zone resets slat (usec): min=20, max=32382, avg=249.06, stdev=1204.11 clat (msec): min=37, max=291, avg=172.60, stdev=42.18 lat (msec): min=38, max=292, avg=172.85, stdev=42.17 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 111], 10.00th=[ 124], 20.00th=[ 138], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 194], 80.00th=[ 211], 90.00th=[ 230], 95.00th=[ 251], | 99.00th=[ 279], 99.50th=[ 284], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 292] bw ( KiB/s): min= 228, max= 512, per=24.94%, avg=367.65, stdev=78.51, samples=55 iops : min= 57, max= 128, avg=91.56, stdev=19.64, samples=55 lat (msec) : 50=0.12%, 100=1.41%, 250=93.67%, 500=4.80% cpu : usr=0.21%, sys=0.98%, ctx=1048, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1475KiB/s (1510kB/s), 369KiB/s-369KiB/s (377kB/s-378kB/s), io=40.0MiB (41.9MB), run=27714-27778msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=150584: Mon Jun 8 21:33:00 2026 read: IOPS=47, BW=189KiB/s (194kB/s)(5048KiB/26699msec) slat (usec): min=1843, max=57917, avg=11507.40, stdev=7195.05 clat (msec): min=61, max=360, avg=157.41, stdev=45.65 lat (msec): min=65, max=373, avg=168.94, stdev=47.61 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 96], 10.00th=[ 107], 20.00th=[ 124], | 30.00th=[ 133], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 159], | 70.00th=[ 171], 80.00th=[ 190], 90.00th=[ 220], 95.00th=[ 247], | 99.00th=[ 309], 99.50th=[ 326], 99.90th=[ 334], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 89, max= 276, per=25.32%, avg=189.15, stdev=46.05, samples=52 iops : min= 22, max= 69, avg=46.85, stdev=11.48, samples=52 write: IOPS=48, BW=194KiB/s (199kB/s)(5192KiB/26699msec); 0 zone resets slat (usec): min=26, max=21421, avg=260.35, stdev=1016.17 clat (msec): min=36, max=345, avg=162.80, stdev=45.27 lat (msec): min=36, max=345, avg=163.07, stdev=45.28 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 102], 10.00th=[ 113], 20.00th=[ 128], | 30.00th=[ 138], 40.00th=[ 146], 50.00th=[ 157], 60.00th=[ 165], | 70.00th=[ 178], 80.00th=[ 192], 90.00th=[ 224], 95.00th=[ 257], | 99.00th=[ 296], 99.50th=[ 313], 99.90th=[ 326], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 81, max= 311, per=25.13%, avg=195.50, stdev=53.11, samples=52 iops : min= 20, max= 77, avg=48.46, stdev=13.28, samples=52 lat (msec) : 50=0.04%, 100=5.31%, 250=89.65%, 500=5.00% cpu : usr=0.37%, sys=2.04%, ctx=2685, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=150585: Mon Jun 8 21:33:00 2026 read: IOPS=46, BW=185KiB/s (190kB/s)(4976KiB/26853msec) slat (usec): min=1940, max=48130, avg=11765.85, stdev=6982.41 clat (msec): min=18, max=362, avg=157.51, stdev=46.33 lat (msec): min=22, max=394, avg=169.28, stdev=48.47 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 94], 10.00th=[ 106], 20.00th=[ 120], | 30.00th=[ 132], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 163], | 70.00th=[ 174], 80.00th=[ 192], 90.00th=[ 222], 95.00th=[ 249], | 99.00th=[ 292], 99.50th=[ 309], 99.90th=[ 359], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 37, max= 284, per=24.75%, avg=184.89, stdev=50.42, samples=53 iops : min= 9, max= 71, avg=45.79, stdev=12.59, samples=53 write: IOPS=49, BW=196KiB/s (201kB/s)(5264KiB/26853msec); 0 zone resets slat (usec): min=25, max=14565, avg=269.58, stdev=1022.11 clat (msec): min=31, max=333, avg=164.37, stdev=46.58 lat (msec): min=31, max=333, avg=164.64, stdev=46.60 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 97], 10.00th=[ 112], 20.00th=[ 129], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 169], | 70.00th=[ 182], 80.00th=[ 199], 90.00th=[ 230], 95.00th=[ 257], | 99.00th=[ 300], 99.50th=[ 313], 99.90th=[ 326], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 89, max= 334, per=25.25%, avg=196.42, stdev=53.29, samples=53 iops : min= 22, max= 83, avg=48.68, stdev=13.30, samples=53 lat (msec) : 20=0.04%, 50=0.16%, 100=6.56%, 250=87.81%, 500=5.43% cpu : usr=0.30%, sys=2.18%, ctx=2706, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=150586: Mon Jun 8 21:33:00 2026 read: IOPS=46, BW=186KiB/s (191kB/s)(4984KiB/26783msec) slat (usec): min=2091, max=51018, avg=11705.83, stdev=7304.71 clat (msec): min=56, max=381, avg=157.70, stdev=46.20 lat (msec): min=60, max=403, avg=169.41, stdev=48.41 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 95], 10.00th=[ 108], 20.00th=[ 122], | 30.00th=[ 131], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 161], | 70.00th=[ 176], 80.00th=[ 192], 90.00th=[ 220], 95.00th=[ 247], | 99.00th=[ 296], 99.50th=[ 313], 99.90th=[ 359], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 78, max= 304, per=24.87%, avg=185.77, stdev=50.87, samples=53 iops : min= 19, max= 76, avg=46.00, stdev=12.74, samples=53 write: IOPS=49, BW=196KiB/s (201kB/s)(5256KiB/26783msec); 0 zone resets slat (usec): min=22, max=16294, avg=245.24, stdev=868.44 clat (msec): min=65, max=427, avg=163.48, stdev=48.53 lat (msec): min=65, max=427, avg=163.73, stdev=48.53 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 102], 10.00th=[ 112], 20.00th=[ 126], | 30.00th=[ 138], 40.00th=[ 146], 50.00th=[ 157], 60.00th=[ 165], | 70.00th=[ 178], 80.00th=[ 197], 90.00th=[ 232], 95.00th=[ 255], | 99.00th=[ 317], 99.50th=[ 359], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 82, max= 301, per=25.13%, avg=195.51, stdev=52.67, samples=53 iops : min= 20, max= 75, avg=48.45, stdev=13.14, samples=53 lat (msec) : 100=5.70%, 250=89.22%, 500=5.08% cpu : usr=0.32%, sys=2.09%, ctx=2590, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=150587: Mon Jun 8 21:33:00 2026 read: IOPS=47, BW=188KiB/s (193kB/s)(5060KiB/26846msec) slat (usec): min=62, max=65260, avg=11823.12, stdev=7681.27 clat (msec): min=15, max=384, avg=158.21, stdev=46.88 lat (msec): min=15, max=401, avg=170.04, stdev=48.85 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 95], 10.00th=[ 110], 20.00th=[ 123], | 30.00th=[ 131], 40.00th=[ 142], 50.00th=[ 153], 60.00th=[ 163], | 70.00th=[ 174], 80.00th=[ 192], 90.00th=[ 218], 95.00th=[ 249], | 99.00th=[ 305], 99.50th=[ 330], 99.90th=[ 376], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 101, max= 301, per=25.21%, avg=188.28, stdev=46.43, samples=53 iops : min= 25, max= 75, avg=46.62, stdev=11.63, samples=53 write: IOPS=48, BW=193KiB/s (198kB/s)(5180KiB/26846msec); 0 zone resets slat (usec): min=22, max=26585, avg=312.47, stdev=1382.75 clat (msec): min=52, max=372, avg=164.82, stdev=45.73 lat (msec): min=52, max=372, avg=165.14, stdev=45.82 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 100], 10.00th=[ 113], 20.00th=[ 129], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 169], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 226], 95.00th=[ 251], | 99.00th=[ 309], 99.50th=[ 317], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 93, max= 270, per=24.78%, avg=192.79, stdev=44.88, samples=53 iops : min= 23, max= 67, avg=47.77, stdev=11.16, samples=53 lat (msec) : 20=0.12%, 50=0.08%, 100=5.78%, 250=88.91%, 500=5.12% cpu : usr=0.30%, sys=2.43%, ctx=2662, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=747KiB/s (765kB/s), 185KiB/s-189KiB/s (190kB/s-194kB/s), io=19.6MiB (20.5MB), run=26699-26853msec WRITE: bw=778KiB/s (797kB/s), 193KiB/s-196KiB/s (198kB/s-201kB/s), io=20.4MiB (21.4MB), run=26699-26853msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=150595: Mon Jun 8 21:33:10 2026 read: IOPS=2, BW=83.2MiB/s (87.2MB/s)(40.0MiB/481msec) slat (nsec): min=45061k, max=45061k, avg=45061173.00, stdev= 0.00 clat (nsec): min=408050k, max=408050k, avg=408049807.00, stdev= 0.00 lat (nsec): min=453144k, max=453144k, avg=453144253.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 409], 10.00th=[ 409], 20.00th=[ 409], | 30.00th=[ 409], 40.00th=[ 409], 50.00th=[ 409], 60.00th=[ 409], | 70.00th=[ 409], 80.00th=[ 409], 90.00th=[ 409], 95.00th=[ 409], | 99.00th=[ 409], 99.50th=[ 409], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.08%, ctx=3, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=83.2MiB/s (87.2MB/s), 83.2MiB/s-83.2MiB/s (87.2MB/s-87.2MB/s), io=40.0MiB (41.9MB), run=481-481msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck