== sanity test 398c: run fio to test AIO ================= 08:57:42 (1786193862) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.132628 s, 316 MB/s osc.lustre-OST0000-osc-ffff90df32376000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180201: Sat Aug 8 08:57:53 2026 write: IOPS=352, BW=1410KiB/s (1444kB/s)(10.0MiB/7261msec); 0 zone resets slat (usec): min=34, max=2355, avg=176.26, stdev=138.85 clat (usec): min=3915, max=78062, avg=44848.87, stdev=7538.64 lat (usec): min=4028, max=78142, avg=45026.39, stdev=7554.86 clat percentiles (usec): | 1.00th=[29754], 5.00th=[32900], 10.00th=[34866], 20.00th=[38011], | 30.00th=[40633], 40.00th=[42730], 50.00th=[44827], 60.00th=[46924], | 70.00th=[48497], 80.00th=[50594], 90.00th=[54264], 95.00th=[57410], | 99.00th=[62653], 99.50th=[66847], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 1232, max= 1557, per=24.90%, avg=1404.36, stdev=95.92, samples=14 iops : min= 308, max= 389, avg=351.07, stdev=23.95, samples=14 lat (msec) : 4=0.04%, 10=0.04%, 50=76.37%, 100=23.55% cpu : usr=0.54%, sys=4.59%, ctx=2356, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180202: Sat Aug 8 08:57:53 2026 write: IOPS=354, BW=1418KiB/s (1452kB/s)(10.0MiB/7221msec); 0 zone resets slat (usec): min=41, max=1349, avg=194.41, stdev=129.33 clat (usec): min=4048, max=92228, avg=44879.49, stdev=7941.11 lat (usec): min=4257, max=92459, avg=45075.22, stdev=7942.77 clat percentiles (usec): | 1.00th=[27395], 5.00th=[32375], 10.00th=[34866], 20.00th=[38536], | 30.00th=[40633], 40.00th=[42730], 50.00th=[44303], 60.00th=[46400], | 70.00th=[48497], 80.00th=[50594], 90.00th=[54789], 95.00th=[57410], | 99.00th=[65799], 99.50th=[72877], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 1248, max= 1770, per=25.11%, avg=1416.71, stdev=132.11, samples=14 iops : min= 312, max= 442, avg=354.14, stdev=32.92, samples=14 lat (msec) : 10=0.08%, 20=0.12%, 50=75.70%, 100=24.10% cpu : usr=0.46%, sys=5.03%, ctx=2599, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180203: Sat Aug 8 08:57:53 2026 write: IOPS=352, BW=1412KiB/s (1446kB/s)(10.0MiB/7253msec); 0 zone resets slat (usec): min=36, max=1513, avg=192.19, stdev=125.89 clat (usec): min=15191, max=77936, avg=44799.16, stdev=7573.21 lat (usec): min=15282, max=78229, avg=44992.70, stdev=7582.19 clat percentiles (usec): | 1.00th=[27657], 5.00th=[32375], 10.00th=[34866], 20.00th=[38536], | 30.00th=[41157], 40.00th=[42730], 50.00th=[44827], 60.00th=[46400], | 70.00th=[48497], 80.00th=[50594], 90.00th=[54789], 95.00th=[57410], | 99.00th=[63701], 99.50th=[66323], 99.90th=[77071], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 1248, max= 1654, per=24.93%, avg=1406.14, stdev=100.82, samples=14 iops : min= 312, max= 413, avg=351.50, stdev=25.11, samples=14 lat (msec) : 20=0.08%, 50=76.84%, 100=23.09% cpu : usr=0.54%, sys=5.28%, ctx=2541, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180204: Sat Aug 8 08:57:53 2026 write: IOPS=353, BW=1414KiB/s (1448kB/s)(10.0MiB/7243msec); 0 zone resets slat (usec): min=37, max=25257, avg=212.79, stdev=517.04 clat (usec): min=8647, max=77901, avg=44786.49, stdev=7738.07 lat (usec): min=8793, max=78020, avg=45000.48, stdev=7714.28 clat percentiles (usec): | 1.00th=[27395], 5.00th=[32375], 10.00th=[35390], 20.00th=[38536], | 30.00th=[41157], 40.00th=[42730], 50.00th=[44827], 60.00th=[46400], | 70.00th=[48497], 80.00th=[50594], 90.00th=[54789], 95.00th=[57934], | 99.00th=[63701], 99.50th=[67634], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 1240, max= 1673, per=25.00%, avg=1410.36, stdev=105.13, samples=14 iops : min= 310, max= 418, avg=352.57, stdev=26.23, samples=14 lat (msec) : 10=0.08%, 20=0.12%, 50=76.60%, 100=23.20% cpu : usr=0.57%, sys=5.63%, ctx=2552, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=5641KiB/s (5776kB/s), 1410KiB/s-1418KiB/s (1444kB/s-1452kB/s), io=40.0MiB (41.9MB), run=7221-7261msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180215: Sat Aug 8 08:58:03 2026 read: IOPS=176, BW=708KiB/s (725kB/s)(5048KiB/7132msec) slat (usec): min=1104, max=8959, avg=2886.34, stdev=949.16 clat (usec): min=7780, max=76269, avg=42433.69, stdev=8047.41 lat (usec): min=10268, max=78132, avg=45322.67, stdev=8295.86 clat percentiles (usec): | 1.00th=[27395], 5.00th=[30540], 10.00th=[32900], 20.00th=[35914], | 30.00th=[38011], 40.00th=[39584], 50.00th=[41681], 60.00th=[43779], | 70.00th=[45876], 80.00th=[49021], 90.00th=[53216], 95.00th=[56361], | 99.00th=[63177], 99.50th=[66847], 99.90th=[74974], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 488, max= 896, per=24.88%, avg=699.92, stdev=114.36, samples=13 iops : min= 122, max= 224, avg=174.92, stdev=28.62, samples=13 write: IOPS=181, BW=728KiB/s (745kB/s)(5192KiB/7132msec); 0 zone resets slat (usec): min=46, max=1043, avg=173.16, stdev=104.49 clat (usec): min=15734, max=73555, avg=43061.86, stdev=7427.36 lat (usec): min=16095, max=74270, avg=43236.47, stdev=7427.76 clat percentiles (usec): | 1.00th=[29230], 5.00th=[32375], 10.00th=[33817], 20.00th=[36963], | 30.00th=[39060], 40.00th=[40633], 50.00th=[42206], 60.00th=[44303], | 70.00th=[46400], 80.00th=[49021], 90.00th=[53216], 95.00th=[56886], | 99.00th=[63177], 99.50th=[66323], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 604, max= 878, per=24.73%, avg=724.46, stdev=72.89, samples=13 iops : min= 151, max= 219, avg=181.08, stdev=18.14, samples=13 lat (msec) : 10=0.04%, 20=0.16%, 50=82.73%, 100=17.07% cpu : usr=0.59%, sys=5.37%, ctx=4637, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180216: Sat Aug 8 08:58:03 2026 read: IOPS=174, BW=698KiB/s (715kB/s)(4976KiB/7129msec) slat (usec): min=979, max=7224, avg=2916.30, stdev=934.63 clat (usec): min=5194, max=73757, avg=42503.23, stdev=8211.05 lat (usec): min=7305, max=78457, avg=45421.67, stdev=8477.38 clat percentiles (usec): | 1.00th=[26608], 5.00th=[30540], 10.00th=[33162], 20.00th=[35914], | 30.00th=[38011], 40.00th=[39584], 50.00th=[41681], 60.00th=[43779], | 70.00th=[46400], 80.00th=[49546], 90.00th=[53216], 95.00th=[56886], | 99.00th=[63701], 99.50th=[66323], 99.90th=[72877], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 544, max= 832, per=24.44%, avg=687.38, stdev=87.61, samples=13 iops : min= 136, max= 208, avg=171.77, stdev=21.96, samples=13 write: IOPS=184, BW=738KiB/s (756kB/s)(5264KiB/7129msec); 0 zone resets slat (usec): min=56, max=1365, avg=179.29, stdev=126.36 clat (usec): min=13336, max=71544, avg=43187.11, stdev=7269.73 lat (usec): min=13527, max=71694, avg=43367.66, stdev=7270.92 clat percentiles (usec): | 1.00th=[28705], 5.00th=[32900], 10.00th=[34341], 20.00th=[36963], | 30.00th=[39060], 40.00th=[40633], 50.00th=[42206], 60.00th=[44303], | 70.00th=[46400], 80.00th=[49546], 90.00th=[52691], 95.00th=[56361], | 99.00th=[62129], 99.50th=[65274], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 576, max= 926, per=25.16%, avg=737.00, stdev=96.34, samples=13 iops : min= 144, max= 231, avg=184.15, stdev=24.06, samples=13 lat (msec) : 10=0.12%, 20=0.04%, 50=81.76%, 100=18.09% cpu : usr=0.59%, sys=5.49%, ctx=4658, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180217: Sat Aug 8 08:58:03 2026 read: IOPS=175, BW=702KiB/s (719kB/s)(4984KiB/7098msec) slat (usec): min=33, max=8822, avg=2920.30, stdev=970.67 clat (usec): min=5638, max=80070, avg=42308.54, stdev=8501.53 lat (usec): min=5743, max=84134, avg=45230.80, stdev=8824.40 clat percentiles (usec): | 1.00th=[23200], 5.00th=[30016], 10.00th=[32900], 20.00th=[35914], | 30.00th=[38011], 40.00th=[40109], 50.00th=[41681], 60.00th=[43779], | 70.00th=[45876], 80.00th=[49021], 90.00th=[52691], 95.00th=[56886], | 99.00th=[64750], 99.50th=[69731], 99.90th=[77071], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 592, max= 864, per=24.94%, avg=701.62, stdev=78.86, samples=13 iops : min= 148, max= 216, avg=175.31, stdev=19.77, samples=13 write: IOPS=185, BW=740KiB/s (758kB/s)(5256KiB/7098msec); 0 zone resets slat (usec): min=35, max=2180, avg=181.17, stdev=126.51 clat (usec): min=23907, max=74742, avg=43186.14, stdev=7622.37 lat (usec): min=24167, max=74902, avg=43368.77, stdev=7629.07 clat percentiles (usec): | 1.00th=[30016], 5.00th=[32375], 10.00th=[34341], 20.00th=[36439], | 30.00th=[38536], 40.00th=[40633], 50.00th=[42206], 60.00th=[44303], | 70.00th=[46400], 80.00th=[49021], 90.00th=[53216], 95.00th=[56886], | 99.00th=[65799], 99.50th=[70779], 99.90th=[73925], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 536, max= 928, per=24.96%, avg=731.00, stdev=108.55, samples=13 iops : min= 134, max= 232, avg=182.69, stdev=27.08, samples=13 lat (msec) : 10=0.20%, 20=0.20%, 50=83.28%, 100=16.33% cpu : usr=0.59%, sys=5.38%, ctx=4618, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180218: Sat Aug 8 08:58:03 2026 read: IOPS=177, BW=709KiB/s (727kB/s)(5060KiB/7132msec) slat (usec): min=1149, max=9360, avg=2918.74, stdev=976.29 clat (usec): min=16275, max=76296, avg=42209.21, stdev=7828.75 lat (usec): min=17681, max=79224, avg=45130.06, stdev=8108.64 clat percentiles (usec): | 1.00th=[26870], 5.00th=[30278], 10.00th=[32637], 20.00th=[35390], | 30.00th=[38011], 40.00th=[39584], 50.00th=[41681], 60.00th=[43779], | 70.00th=[45876], 80.00th=[48497], 90.00th=[52691], 95.00th=[56361], | 99.00th=[62653], 99.50th=[64226], 99.90th=[69731], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 520, max= 862, per=25.00%, avg=703.38, stdev=101.85, samples=13 iops : min= 130, max= 215, avg=175.77, stdev=25.45, samples=13 write: IOPS=181, BW=726KiB/s (744kB/s)(5180KiB/7132msec); 0 zone resets slat (usec): min=55, max=1226, avg=182.21, stdev=125.66 clat (usec): min=17687, max=72766, avg=43203.97, stdev=7455.76 lat (usec): min=17779, max=73064, avg=43387.32, stdev=7470.57 clat percentiles (usec): | 1.00th=[29754], 5.00th=[32637], 10.00th=[34341], 20.00th=[36439], | 30.00th=[39060], 40.00th=[40633], 50.00th=[42730], 60.00th=[44303], | 70.00th=[46400], 80.00th=[48497], 90.00th=[53216], 95.00th=[56886], | 99.00th=[65799], 99.50th=[70779], 99.90th=[71828], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 600, max= 872, per=24.50%, avg=717.54, stdev=81.58, samples=13 iops : min= 150, max= 218, avg=179.31, stdev=20.45, samples=13 lat (msec) : 20=0.23%, 50=83.98%, 100=15.78% cpu : usr=0.46%, sys=5.54%, ctx=4717, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2814KiB/s (2881kB/s), 698KiB/s-709KiB/s (715kB/s-727kB/s), io=19.6MiB (20.5MB), run=7098-7132msec WRITE: bw=2929KiB/s (3000kB/s), 726KiB/s-740KiB/s (744kB/s-758kB/s), io=20.4MiB (21.4MB), run=7098-7132msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180223: Sat Aug 8 08:58:07 2026 read: IOPS=8, BW=342MiB/s (358MB/s)(40.0MiB/117msec) slat (nsec): min=11362k, max=11362k, avg=11361899.00, stdev= 0.00 clat (nsec): min=100292k, max=100292k, avg=100291845.00, stdev= 0.00 lat (nsec): min=111658k, max=111658k, avg=111657829.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 101], 10.00th=[ 101], 20.00th=[ 101], | 30.00th=[ 101], 40.00th=[ 101], 50.00th=[ 101], 60.00th=[ 101], | 70.00th=[ 101], 80.00th=[ 101], 90.00th=[ 101], 95.00th=[ 101], | 99.00th=[ 101], 99.50th=[ 101], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=8.62%, ctx=11, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=342MiB/s (358MB/s), 342MiB/s-342MiB/s (358MB/s-358MB/s), io=40.0MiB (41.9MB), run=117-117msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout