== sanity test 398c: run fio to test AIO ================= 18:21:20 (1781821280) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.203992 s, 206 MB/s osc.lustre-OST0000-osc-ffff8b1ef2f30800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=187780: Thu Jun 18 18:21:38 2026 write: IOPS=189, BW=760KiB/s (778kB/s)(10.0MiB/13480msec); 0 zone resets slat (usec): min=50, max=31690, avg=258.66, stdev=681.82 clat (msec): min=10, max=166, avg=83.35, stdev=16.23 lat (msec): min=19, max=166, avg=83.61, stdev=16.20 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 71], | 30.00th=[ 75], 40.00th=[ 79], 50.00th=[ 82], 60.00th=[ 86], | 70.00th=[ 91], 80.00th=[ 97], 90.00th=[ 104], 95.00th=[ 111], | 99.00th=[ 131], 99.50th=[ 136], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 167] bw ( KiB/s): min= 622, max= 872, per=25.02%, avg=760.23, stdev=80.71, samples=26 iops : min= 155, max= 218, avg=189.85, stdev=20.24, samples=26 lat (msec) : 20=0.08%, 50=1.52%, 100=84.06%, 250=14.34% cpu : usr=0.30%, sys=3.15%, ctx=2754, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=187781: Thu Jun 18 18:21:38 2026 write: IOPS=191, BW=765KiB/s (784kB/s)(10.0MiB/13381msec); 0 zone resets slat (usec): min=37, max=6019, avg=251.72, stdev=315.59 clat (msec): min=7, max=177, avg=83.21, stdev=16.85 lat (msec): min=8, max=177, avg=83.46, stdev=16.85 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 63], 10.00th=[ 67], 20.00th=[ 71], | 30.00th=[ 75], 40.00th=[ 79], 50.00th=[ 82], 60.00th=[ 86], | 70.00th=[ 91], 80.00th=[ 97], 90.00th=[ 103], 95.00th=[ 109], | 99.00th=[ 130], 99.50th=[ 136], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 614, max= 955, per=25.20%, avg=765.46, stdev=88.72, samples=26 iops : min= 153, max= 238, avg=191.15, stdev=22.21, samples=26 lat (msec) : 10=0.04%, 20=0.04%, 50=1.76%, 100=83.98%, 250=14.18% cpu : usr=0.39%, sys=2.75%, ctx=2590, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=187782: Thu Jun 18 18:21:38 2026 write: IOPS=190, BW=760KiB/s (778kB/s)(10.0MiB/13471msec); 0 zone resets slat (usec): min=38, max=37314, avg=259.70, stdev=770.82 clat (msec): min=10, max=147, avg=83.31, stdev=16.56 lat (msec): min=11, max=148, avg=83.57, stdev=16.53 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 71], | 30.00th=[ 75], 40.00th=[ 79], 50.00th=[ 82], 60.00th=[ 86], | 70.00th=[ 92], 80.00th=[ 97], 90.00th=[ 103], 95.00th=[ 109], | 99.00th=[ 130], 99.50th=[ 138], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 148] bw ( KiB/s): min= 590, max= 870, per=25.07%, avg=761.77, stdev=78.55, samples=26 iops : min= 147, max= 217, avg=190.23, stdev=19.69, samples=26 lat (msec) : 20=0.08%, 50=1.64%, 100=84.41%, 250=13.87% cpu : usr=0.30%, sys=3.04%, ctx=2824, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=187783: Thu Jun 18 18:21:38 2026 write: IOPS=189, BW=760KiB/s (778kB/s)(10.0MiB/13480msec); 0 zone resets slat (usec): min=44, max=3942, avg=245.45, stdev=288.63 clat (msec): min=5, max=141, avg=83.34, stdev=16.02 lat (msec): min=5, max=141, avg=83.58, stdev=16.03 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 61], 10.00th=[ 67], 20.00th=[ 71], | 30.00th=[ 75], 40.00th=[ 79], 50.00th=[ 82], 60.00th=[ 86], | 70.00th=[ 91], 80.00th=[ 97], 90.00th=[ 104], 95.00th=[ 112], | 99.00th=[ 127], 99.50th=[ 131], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 574, max= 896, per=25.00%, avg=759.62, stdev=75.95, samples=26 iops : min= 143, max= 224, avg=189.69, stdev=19.03, samples=26 lat (msec) : 10=0.08%, 20=0.08%, 50=1.56%, 100=84.73%, 250=13.55% cpu : usr=0.28%, sys=2.82%, ctx=2873, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3039KiB/s (3112kB/s), 760KiB/s-765KiB/s (778kB/s-784kB/s), io=40.0MiB (41.9MB), run=13381-13480msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=187795: Thu Jun 18 18:21:55 2026 read: IOPS=102, BW=411KiB/s (421kB/s)(5048KiB/12278msec) slat (usec): min=37, max=15313, avg=4893.98, stdev=2001.31 clat (msec): min=13, max=137, avg=72.66, stdev=16.92 lat (msec): min=16, max=142, avg=77.56, stdev=17.67 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 49], 10.00th=[ 53], 20.00th=[ 59], | 30.00th=[ 64], 40.00th=[ 67], 50.00th=[ 71], 60.00th=[ 75], | 70.00th=[ 80], 80.00th=[ 87], 90.00th=[ 96], 95.00th=[ 104], | 99.00th=[ 122], 99.50th=[ 127], 99.90th=[ 131], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 288, max= 584, per=25.22%, avg=409.83, stdev=81.25, samples=24 iops : min= 72, max= 146, avg=102.33, stdev=20.36, samples=24 write: IOPS=105, BW=423KiB/s (433kB/s)(5192KiB/12278msec); 0 zone resets slat (usec): min=38, max=7219, avg=250.81, stdev=303.41 clat (msec): min=25, max=136, avg=75.35, stdev=16.01 lat (msec): min=25, max=136, avg=75.60, stdev=16.03 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 52], 10.00th=[ 56], 20.00th=[ 62], | 30.00th=[ 67], 40.00th=[ 70], 50.00th=[ 74], 60.00th=[ 79], | 70.00th=[ 83], 80.00th=[ 89], 90.00th=[ 97], 95.00th=[ 105], | 99.00th=[ 115], 99.50th=[ 123], 99.90th=[ 132], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 296, max= 584, per=25.00%, avg=422.92, stdev=68.26, samples=24 iops : min= 74, max= 146, avg=105.63, stdev=17.12, samples=24 lat (msec) : 20=0.04%, 50=5.31%, 100=87.23%, 250=7.42% cpu : usr=0.50%, sys=3.74%, ctx=4691, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=187796: Thu Jun 18 18:21:55 2026 read: IOPS=100, BW=403KiB/s (413kB/s)(4976KiB/12345msec) slat (usec): min=358, max=15902, avg=4926.72, stdev=1982.29 clat (msec): min=13, max=127, avg=72.08, stdev=16.11 lat (msec): min=17, max=132, avg=77.01, stdev=16.83 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 50], 10.00th=[ 54], 20.00th=[ 59], | 30.00th=[ 63], 40.00th=[ 66], 50.00th=[ 70], 60.00th=[ 74], | 70.00th=[ 80], 80.00th=[ 86], 90.00th=[ 95], 95.00th=[ 102], | 99.00th=[ 114], 99.50th=[ 120], 99.90th=[ 126], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 294, max= 560, per=24.77%, avg=402.54, stdev=75.64, samples=24 iops : min= 73, max= 140, avg=100.54, stdev=18.97, samples=24 write: IOPS=106, BW=426KiB/s (437kB/s)(5264KiB/12345msec); 0 zone resets slat (usec): min=37, max=3200, avg=246.49, stdev=272.50 clat (msec): min=16, max=137, avg=76.31, stdev=16.91 lat (msec): min=16, max=137, avg=76.56, stdev=16.92 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 53], 10.00th=[ 56], 20.00th=[ 62], | 30.00th=[ 66], 40.00th=[ 70], 50.00th=[ 74], 60.00th=[ 80], | 70.00th=[ 85], 80.00th=[ 91], 90.00th=[ 100], 95.00th=[ 107], | 99.00th=[ 121], 99.50th=[ 127], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 304, max= 568, per=25.13%, avg=425.12, stdev=75.66, samples=24 iops : min= 76, max= 142, avg=106.17, stdev=18.96, samples=24 lat (msec) : 20=0.08%, 50=4.45%, 100=87.70%, 250=7.77% cpu : usr=0.44%, sys=3.76%, ctx=4697, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=187797: Thu Jun 18 18:21:55 2026 read: IOPS=101, BW=404KiB/s (414kB/s)(4984KiB/12325msec) slat (usec): min=1524, max=16420, avg=5003.56, stdev=1991.48 clat (msec): min=24, max=132, avg=72.23, stdev=16.39 lat (msec): min=31, max=137, avg=77.24, stdev=17.04 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 50], 10.00th=[ 53], 20.00th=[ 58], | 30.00th=[ 63], 40.00th=[ 67], 50.00th=[ 71], 60.00th=[ 74], | 70.00th=[ 81], 80.00th=[ 86], 90.00th=[ 94], 95.00th=[ 103], | 99.00th=[ 121], 99.50th=[ 123], 99.90th=[ 128], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 215, max= 592, per=24.94%, avg=405.33, stdev=81.57, samples=24 iops : min= 53, max= 148, avg=101.21, stdev=20.49, samples=24 write: IOPS=106, BW=426KiB/s (437kB/s)(5256KiB/12325msec); 0 zone resets slat (usec): min=43, max=3519, avg=242.16, stdev=249.56 clat (msec): min=36, max=138, avg=75.79, stdev=16.75 lat (msec): min=36, max=138, avg=76.04, stdev=16.77 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 52], 10.00th=[ 56], 20.00th=[ 62], | 30.00th=[ 66], 40.00th=[ 70], 50.00th=[ 74], 60.00th=[ 80], | 70.00th=[ 84], 80.00th=[ 90], 90.00th=[ 99], 95.00th=[ 106], | 99.00th=[ 121], 99.50th=[ 126], 99.90th=[ 134], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 304, max= 664, per=25.12%, avg=424.96, stdev=84.50, samples=24 iops : min= 76, max= 166, avg=106.12, stdev=21.17, samples=24 lat (msec) : 50=5.16%, 100=87.42%, 250=7.42% cpu : usr=0.36%, sys=3.98%, ctx=4691, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=187798: Thu Jun 18 18:21:55 2026 read: IOPS=102, BW=410KiB/s (420kB/s)(5060KiB/12343msec) slat (usec): min=1529, max=16736, avg=5003.10, stdev=1998.57 clat (msec): min=13, max=137, avg=72.51, stdev=16.98 lat (msec): min=17, max=142, avg=77.52, stdev=17.73 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 49], 10.00th=[ 53], 20.00th=[ 58], | 30.00th=[ 63], 40.00th=[ 67], 50.00th=[ 71], 60.00th=[ 75], | 70.00th=[ 81], 80.00th=[ 87], 90.00th=[ 95], 95.00th=[ 104], | 99.00th=[ 121], 99.50th=[ 128], 99.90th=[ 136], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 270, max= 616, per=25.18%, avg=409.17, stdev=76.21, samples=24 iops : min= 67, max= 154, avg=102.21, stdev=19.09, samples=24 write: IOPS=104, BW=420KiB/s (430kB/s)(5180KiB/12343msec); 0 zone resets slat (usec): min=54, max=7054, avg=261.14, stdev=359.71 clat (msec): min=20, max=144, avg=75.55, stdev=16.53 lat (msec): min=20, max=144, avg=75.82, stdev=16.55 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 52], 10.00th=[ 57], 20.00th=[ 62], | 30.00th=[ 66], 40.00th=[ 70], 50.00th=[ 74], 60.00th=[ 79], | 70.00th=[ 84], 80.00th=[ 90], 90.00th=[ 97], 95.00th=[ 105], | 99.00th=[ 121], 99.50th=[ 130], 99.90th=[ 140], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 296, max= 568, per=24.68%, avg=417.54, stdev=81.24, samples=24 iops : min= 74, max= 142, avg=104.29, stdev=20.36, samples=24 lat (msec) : 20=0.04%, 50=5.43%, 100=86.95%, 250=7.58% cpu : usr=0.41%, sys=3.86%, ctx=4672, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1626KiB/s (1665kB/s), 403KiB/s-411KiB/s (413kB/s-421kB/s), io=19.6MiB (20.5MB), run=12278-12345msec WRITE: bw=1692KiB/s (1733kB/s), 420KiB/s-426KiB/s (430kB/s-437kB/s), io=20.4MiB (21.4MB), run=12278-12345msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=187803: Thu Jun 18 18:21:59 2026 read: IOPS=5, BW=208MiB/s (218MB/s)(40.0MiB/192msec) slat (nsec): min=13747k, max=13747k, avg=13747275.00, stdev= 0.00 clat (nsec): min=168607k, max=168607k, avg=168606523.00, stdev= 0.00 lat (nsec): min=182359k, max=182359k, avg=182358988.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 169], 10.00th=[ 169], 20.00th=[ 169], | 30.00th=[ 169], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 169], | 70.00th=[ 169], 80.00th=[ 169], 90.00th=[ 169], 95.00th=[ 169], | 99.00th=[ 169], 99.50th=[ 169], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.28%, ctx=5, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=208MiB/s (218MB/s), 208MiB/s-208MiB/s (218MB/s-218MB/s), io=40.0MiB (41.9MB), run=192-192msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout