== sanity test 398c: run fio to test AIO ================= 17:15:32 (1780089332) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.189851 s, 221 MB/s osc.lustre-OST0000-osc-ffff8c1007f9c800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=186032: Fri May 29 17:15:45 2026 write: IOPS=256, BW=1027KiB/s (1051kB/s)(10.0MiB/9973msec); 0 zone resets slat (usec): min=36, max=3005, avg=189.77, stdev=167.81 clat (msec): min=17, max=109, avg=61.73, stdev=13.09 lat (msec): min=18, max=109, avg=61.92, stdev=13.10 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 43], 10.00th=[ 46], 20.00th=[ 51], | 30.00th=[ 55], 40.00th=[ 57], 50.00th=[ 62], 60.00th=[ 64], | 70.00th=[ 68], 80.00th=[ 73], 90.00th=[ 79], 95.00th=[ 84], | 99.00th=[ 101], 99.50th=[ 104], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 110] bw ( KiB/s): min= 846, max= 1236, per=24.92%, avg=1023.63, stdev=115.61, samples=19 iops : min= 211, max= 309, avg=255.74, stdev=28.92, samples=19 lat (msec) : 20=0.08%, 50=18.16%, 100=80.62%, 250=1.13% cpu : usr=0.44%, sys=3.53%, ctx=2619, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=186033: Fri May 29 17:15:45 2026 write: IOPS=256, BW=1027KiB/s (1052kB/s)(10.0MiB/9972msec); 0 zone resets slat (usec): min=36, max=4390, avg=192.03, stdev=228.42 clat (msec): min=6, max=109, avg=61.66, stdev=13.12 lat (msec): min=6, max=109, avg=61.86, stdev=13.13 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 43], 10.00th=[ 46], 20.00th=[ 51], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 61], 60.00th=[ 65], | 70.00th=[ 68], 80.00th=[ 73], 90.00th=[ 79], 95.00th=[ 85], | 99.00th=[ 99], 99.50th=[ 105], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 800, max= 1272, per=24.93%, avg=1024.05, stdev=120.72, samples=19 iops : min= 200, max= 318, avg=255.84, stdev=30.21, samples=19 lat (msec) : 10=0.04%, 20=0.12%, 50=18.52%, 100=80.78%, 250=0.55% cpu : usr=0.40%, sys=3.41%, ctx=2533, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=186034: Fri May 29 17:15:45 2026 write: IOPS=256, BW=1027KiB/s (1052kB/s)(10.0MiB/9972msec); 0 zone resets slat (usec): min=37, max=3517, avg=185.14, stdev=204.68 clat (msec): min=9, max=109, avg=61.69, stdev=13.07 lat (msec): min=9, max=109, avg=61.88, stdev=13.07 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 43], 10.00th=[ 46], 20.00th=[ 51], | 30.00th=[ 55], 40.00th=[ 57], 50.00th=[ 61], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 73], 90.00th=[ 79], 95.00th=[ 85], | 99.00th=[ 101], 99.50th=[ 105], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 838, max= 1248, per=24.95%, avg=1024.58, stdev=115.57, samples=19 iops : min= 209, max= 312, avg=255.95, stdev=28.89, samples=19 lat (msec) : 10=0.04%, 20=0.08%, 50=18.59%, 100=80.27%, 250=1.02% cpu : usr=0.34%, sys=3.56%, ctx=2604, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=186035: Fri May 29 17:15:45 2026 write: IOPS=258, BW=1034KiB/s (1058kB/s)(10.0MiB/9907msec); 0 zone resets slat (usec): min=36, max=4350, avg=202.76, stdev=214.18 clat (msec): min=7, max=122, avg=61.63, stdev=13.30 lat (msec): min=8, max=122, avg=61.84, stdev=13.31 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 42], 10.00th=[ 46], 20.00th=[ 51], | 30.00th=[ 54], 40.00th=[ 58], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 68], 80.00th=[ 73], 90.00th=[ 79], 95.00th=[ 85], | 99.00th=[ 100], 99.50th=[ 109], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 846, max= 1294, per=25.10%, avg=1030.89, stdev=126.83, samples=19 iops : min= 211, max= 323, avg=257.53, stdev=31.68, samples=19 lat (msec) : 10=0.12%, 20=0.04%, 50=18.98%, 100=79.92%, 250=0.94% cpu : usr=0.41%, sys=3.63%, ctx=2683, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=4107KiB/s (4206kB/s), 1027KiB/s-1034KiB/s (1051kB/s-1058kB/s), io=40.0MiB (41.9MB), run=9907-9973msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=186046: Fri May 29 17:15:58 2026 read: IOPS=142, BW=572KiB/s (586kB/s)(5048KiB/8826msec) slat (usec): min=1192, max=14506, avg=3612.79, stdev=1531.33 clat (usec): min=10724, max=89656, avg=51914.03, stdev=11029.21 lat (usec): min=19282, max=96563, avg=55530.17, stdev=11422.78 clat percentiles (usec): | 1.00th=[30802], 5.00th=[36439], 10.00th=[39584], 20.00th=[43254], | 30.00th=[45351], 40.00th=[47973], 50.00th=[50594], 60.00th=[53216], | 70.00th=[56361], 80.00th=[60556], 90.00th=[66847], 95.00th=[72877], | 99.00th=[82314], 99.50th=[84411], 99.90th=[88605], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 440, max= 742, per=25.09%, avg=570.35, stdev=76.50, samples=17 iops : min= 110, max= 185, avg=142.53, stdev=19.08, samples=17 write: IOPS=147, BW=588KiB/s (602kB/s)(5192KiB/8826msec); 0 zone resets slat (usec): min=56, max=2048, avg=181.00, stdev=134.83 clat (usec): min=8109, max=89449, avg=53688.05, stdev=9910.98 lat (usec): min=8268, max=89551, avg=53870.25, stdev=9910.06 clat percentiles (usec): | 1.00th=[35390], 5.00th=[39060], 10.00th=[42206], 20.00th=[45876], | 30.00th=[47973], 40.00th=[50070], 50.00th=[52167], 60.00th=[55313], | 70.00th=[57934], 80.00th=[61604], 90.00th=[66847], 95.00th=[72877], | 99.00th=[80217], 99.50th=[81265], 99.90th=[85459], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 520, max= 686, per=24.90%, avg=589.29, stdev=49.77, samples=17 iops : min= 130, max= 171, avg=147.24, stdev=12.42, samples=17 lat (msec) : 10=0.04%, 20=0.08%, 50=44.10%, 100=55.78% cpu : usr=0.44%, sys=4.76%, ctx=4549, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=186047: Fri May 29 17:15:58 2026 read: IOPS=142, BW=569KiB/s (582kB/s)(4976KiB/8748msec) slat (usec): min=91, max=12484, avg=3583.93, stdev=1461.55 clat (msec): min=5, max=123, avg=52.04, stdev=11.85 lat (msec): min=5, max=126, avg=55.63, stdev=12.21 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 37], 10.00th=[ 40], 20.00th=[ 43], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 52], 60.00th=[ 54], | 70.00th=[ 57], 80.00th=[ 61], 90.00th=[ 68], 95.00th=[ 75], | 99.00th=[ 84], 99.50th=[ 95], 99.90th=[ 121], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 464, max= 758, per=25.06%, avg=569.53, stdev=74.68, samples=17 iops : min= 116, max= 189, avg=142.29, stdev=18.65, samples=17 write: IOPS=150, BW=602KiB/s (616kB/s)(5264KiB/8748msec); 0 zone resets slat (usec): min=57, max=2721, avg=189.85, stdev=160.58 clat (msec): min=21, max=123, avg=53.36, stdev=10.17 lat (msec): min=22, max=123, avg=53.55, stdev=10.17 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 41], 10.00th=[ 42], 20.00th=[ 45], | 30.00th=[ 47], 40.00th=[ 50], 50.00th=[ 53], 60.00th=[ 55], | 70.00th=[ 58], 80.00th=[ 62], 90.00th=[ 67], 95.00th=[ 72], | 99.00th=[ 83], 99.50th=[ 85], 99.90th=[ 107], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 480, max= 728, per=25.54%, avg=604.65, stdev=76.17, samples=17 iops : min= 120, max= 182, avg=151.06, stdev=18.99, samples=17 lat (msec) : 10=0.20%, 20=0.04%, 50=43.01%, 100=56.52%, 250=0.23% cpu : usr=0.46%, sys=4.65%, ctx=4655, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=186048: Fri May 29 17:15:58 2026 read: IOPS=141, BW=565KiB/s (579kB/s)(4984KiB/8819msec) slat (usec): min=1148, max=12318, avg=3643.47, stdev=1516.91 clat (usec): min=6853, max=96580, avg=52045.44, stdev=10881.71 lat (msec): min=12, max=100, avg=55.69, stdev=11.30 clat percentiles (usec): | 1.00th=[33162], 5.00th=[36963], 10.00th=[39584], 20.00th=[42206], | 30.00th=[45351], 40.00th=[48497], 50.00th=[51119], 60.00th=[53216], | 70.00th=[56361], 80.00th=[60556], 90.00th=[66847], 95.00th=[72877], | 99.00th=[81265], 99.50th=[82314], 99.90th=[93848], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 494, max= 726, per=24.99%, avg=568.12, stdev=78.31, samples=17 iops : min= 123, max= 181, avg=141.94, stdev=19.56, samples=17 write: IOPS=148, BW=596KiB/s (610kB/s)(5256KiB/8819msec); 0 zone resets slat (usec): min=45, max=4104, avg=195.53, stdev=227.39 clat (usec): min=11808, max=96130, avg=53741.67, stdev=10423.34 lat (usec): min=12038, max=96235, avg=53938.38, stdev=10427.02 clat percentiles (usec): | 1.00th=[35390], 5.00th=[39060], 10.00th=[41681], 20.00th=[45351], | 30.00th=[47449], 40.00th=[50070], 50.00th=[52691], 60.00th=[55313], | 70.00th=[58459], 80.00th=[61604], 90.00th=[68682], 95.00th=[72877], | 99.00th=[82314], 99.50th=[84411], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 480, max= 776, per=25.15%, avg=595.24, stdev=83.11, samples=17 iops : min= 120, max= 194, avg=148.71, stdev=20.73, samples=17 lat (msec) : 10=0.04%, 20=0.04%, 50=43.24%, 100=56.68% cpu : usr=0.26%, sys=4.82%, ctx=4500, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=186049: Fri May 29 17:15:58 2026 read: IOPS=143, BW=574KiB/s (588kB/s)(5060KiB/8813msec) slat (usec): min=1123, max=11532, avg=3628.00, stdev=1462.30 clat (usec): min=8737, max=88752, avg=51926.35, stdev=11039.89 lat (usec): min=11313, max=95873, avg=55558.16, stdev=11374.49 clat percentiles (usec): | 1.00th=[32113], 5.00th=[36963], 10.00th=[39060], 20.00th=[42206], | 30.00th=[44827], 40.00th=[47973], 50.00th=[50594], 60.00th=[53740], | 70.00th=[56361], 80.00th=[60556], 90.00th=[66847], 95.00th=[71828], | 99.00th=[83362], 99.50th=[86508], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 464, max= 766, per=25.31%, avg=575.35, stdev=72.55, samples=17 iops : min= 116, max= 191, avg=143.76, stdev=18.05, samples=17 write: IOPS=146, BW=588KiB/s (602kB/s)(5180KiB/8813msec); 0 zone resets slat (usec): min=54, max=4674, avg=187.05, stdev=185.95 clat (msec): min=25, max=101, avg=53.72, stdev=10.36 lat (msec): min=25, max=101, avg=53.91, stdev=10.36 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 45], | 30.00th=[ 48], 40.00th=[ 51], 50.00th=[ 53], 60.00th=[ 55], | 70.00th=[ 58], 80.00th=[ 63], 90.00th=[ 68], 95.00th=[ 73], | 99.00th=[ 82], 99.50th=[ 84], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 448, max= 712, per=24.84%, avg=588.00, stdev=64.06, samples=17 iops : min= 112, max= 178, avg=146.94, stdev=16.00, samples=17 lat (msec) : 10=0.04%, 50=44.06%, 100=55.82%, 250=0.08% cpu : usr=0.50%, sys=4.71%, ctx=4615, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2274KiB/s (2328kB/s), 565KiB/s-574KiB/s (579kB/s-588kB/s), io=19.6MiB (20.5MB), run=8748-8826msec WRITE: bw=2367KiB/s (2424kB/s), 588KiB/s-602KiB/s (602kB/s-616kB/s), io=20.4MiB (21.4MB), run=8748-8826msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=186054: Fri May 29 17:16:01 2026 read: IOPS=7, BW=284MiB/s (297MB/s)(40.0MiB/141msec) slat (nsec): min=14532k, max=14532k, avg=14531807.00, stdev= 0.00 clat (nsec): min=119133k, max=119133k, avg=119133476.00, stdev= 0.00 lat (nsec): min=133669k, max=133669k, avg=133669490.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 120], 50.00th=[ 120], 60.00th=[ 120], | 70.00th=[ 120], 80.00th=[ 120], 90.00th=[ 120], 95.00th=[ 120], | 99.00th=[ 120], 99.50th=[ 120], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=7.86%, ctx=12, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=284MiB/s (297MB/s), 284MiB/s-284MiB/s (297MB/s-297MB/s), io=40.0MiB (41.9MB), run=141-141msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout