== sanity test 398c: run fio to test AIO ================= 13:25:24 (1772475924) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.140236 s, 299 MB/s osc.lustre-OST0000-osc-ffff8a800b10e800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=182115: Mon Mar 2 13:25:35 2026 write: IOPS=373, BW=1492KiB/s (1528kB/s)(10.0MiB/6861msec); 0 zone resets slat (usec): min=40, max=2654, avg=183.80, stdev=117.66 clat (usec): min=22130, max=77850, avg=42316.72, stdev=8108.29 lat (usec): min=22376, max=77980, avg=42501.59, stdev=8116.96 clat percentiles (usec): | 1.00th=[25560], 5.00th=[30016], 10.00th=[32900], 20.00th=[35390], | 30.00th=[37487], 40.00th=[39584], 50.00th=[41681], 60.00th=[43779], | 70.00th=[46400], 80.00th=[49021], 90.00th=[53216], 95.00th=[55837], | 99.00th=[66323], 99.50th=[67634], 99.90th=[71828], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 1272, max= 1766, per=25.04%, avg=1494.69, stdev=140.58, samples=13 iops : min= 318, max= 441, avg=373.62, stdev=35.06, samples=13 lat (msec) : 50=82.97%, 100=17.03% cpu : usr=0.41%, sys=5.55%, ctx=2599, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182116: Mon Mar 2 13:25:35 2026 write: IOPS=376, BW=1505KiB/s (1542kB/s)(10.0MiB/6802msec); 0 zone resets slat (usec): min=54, max=3536, avg=185.01, stdev=152.81 clat (usec): min=6164, max=74572, avg=42258.35, stdev=8287.94 lat (usec): min=6311, max=74691, avg=42444.51, stdev=8293.52 clat percentiles (usec): | 1.00th=[23725], 5.00th=[29754], 10.00th=[32900], 20.00th=[35390], | 30.00th=[37487], 40.00th=[39060], 50.00th=[42206], 60.00th=[44303], | 70.00th=[46400], 80.00th=[49021], 90.00th=[52691], 95.00th=[56361], | 99.00th=[65274], 99.50th=[67634], 99.90th=[69731], 99.95th=[69731], | 99.99th=[74974] bw ( KiB/s): min= 1264, max= 1909, per=25.17%, avg=1502.62, stdev=165.87, samples=13 iops : min= 316, max= 477, avg=375.62, stdev=41.41, samples=13 lat (msec) : 10=0.12%, 20=0.12%, 50=82.19%, 100=17.58% cpu : usr=0.53%, sys=5.04%, ctx=2800, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182117: Mon Mar 2 13:25:35 2026 write: IOPS=373, BW=1494KiB/s (1530kB/s)(10.0MiB/6854msec); 0 zone resets slat (usec): min=35, max=1955, avg=179.44, stdev=105.97 clat (usec): min=4948, max=70943, avg=42229.50, stdev=8173.79 lat (usec): min=5298, max=71101, avg=42410.02, stdev=8183.24 clat percentiles (usec): | 1.00th=[24511], 5.00th=[29754], 10.00th=[32900], 20.00th=[35390], | 30.00th=[37487], 40.00th=[39584], 50.00th=[41681], 60.00th=[43779], | 70.00th=[46400], 80.00th=[49021], 90.00th=[52167], 95.00th=[56361], | 99.00th=[64750], 99.50th=[67634], 99.90th=[67634], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 1264, max= 1772, per=25.03%, avg=1493.92, stdev=144.27, samples=13 iops : min= 316, max= 443, avg=373.46, stdev=36.06, samples=13 lat (msec) : 10=0.04%, 50=82.89%, 100=17.07% cpu : usr=0.29%, sys=5.24%, ctx=2582, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182118: Mon Mar 2 13:25:35 2026 write: IOPS=373, BW=1492KiB/s (1528kB/s)(10.0MiB/6861msec); 0 zone resets slat (usec): min=45, max=1983, avg=164.70, stdev=104.40 clat (usec): min=7906, max=78017, avg=42316.63, stdev=7980.39 lat (usec): min=8078, max=78131, avg=42482.46, stdev=7982.94 clat percentiles (usec): | 1.00th=[25297], 5.00th=[29754], 10.00th=[33424], 20.00th=[35390], | 30.00th=[37487], 40.00th=[39584], 50.00th=[42206], 60.00th=[44303], | 70.00th=[46400], 80.00th=[49021], 90.00th=[52691], 95.00th=[56361], | 99.00th=[63701], 99.50th=[66323], 99.90th=[67634], 99.95th=[74974], | 99.99th=[78119] bw ( KiB/s): min= 1224, max= 1766, per=25.02%, avg=1493.54, stdev=140.62, samples=13 iops : min= 306, max= 441, avg=373.31, stdev=35.07, samples=13 lat (msec) : 10=0.04%, 50=83.44%, 100=16.52% cpu : usr=0.55%, sys=4.40%, ctx=2663, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=5970KiB/s (6113kB/s), 1492KiB/s-1505KiB/s (1528kB/s-1542kB/s), io=40.0MiB (41.9MB), run=6802-6861msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=182128: Mon Mar 2 13:25:46 2026 read: IOPS=170, BW=680KiB/s (697kB/s)(5048KiB/7419msec) slat (usec): min=56, max=12676, avg=3013.07, stdev=1283.00 clat (msec): min=4, max=100, avg=44.36, stdev=10.21 lat (msec): min=8, max=102, avg=47.38, stdev=10.52 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 32], 10.00th=[ 34], 20.00th=[ 37], | 30.00th=[ 40], 40.00th=[ 42], 50.00th=[ 44], 60.00th=[ 46], | 70.00th=[ 48], 80.00th=[ 52], 90.00th=[ 56], 95.00th=[ 62], | 99.00th=[ 79], 99.50th=[ 83], 99.90th=[ 100], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 584, max= 800, per=25.17%, avg=678.21, stdev=69.52, samples=14 iops : min= 146, max= 200, avg=169.50, stdev=17.32, samples=14 write: IOPS=174, BW=700KiB/s (717kB/s)(5192KiB/7419msec); 0 zone resets slat (usec): min=40, max=3884, avg=175.23, stdev=177.29 clat (usec): min=19364, max=95484, avg=45049.31, stdev=8789.49 lat (usec): min=19646, max=95626, avg=45225.55, stdev=8788.06 clat percentiles (usec): | 1.00th=[27657], 5.00th=[33817], 10.00th=[35390], 20.00th=[37487], | 30.00th=[39584], 40.00th=[42206], 50.00th=[43779], 60.00th=[45876], | 70.00th=[48497], 80.00th=[51643], 90.00th=[56886], 95.00th=[60556], | 99.00th=[68682], 99.50th=[82314], 99.90th=[91751], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 552, max= 792, per=24.88%, avg=697.86, stdev=64.63, samples=14 iops : min= 138, max= 198, avg=174.43, stdev=16.15, samples=14 lat (msec) : 10=0.20%, 20=0.35%, 50=75.20%, 100=24.22%, 250=0.04% cpu : usr=0.38%, sys=5.08%, ctx=4473, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182129: Mon Mar 2 13:25:46 2026 read: IOPS=167, BW=668KiB/s (685kB/s)(4976KiB/7444msec) slat (usec): min=1026, max=11300, avg=3025.91, stdev=1208.32 clat (usec): min=9417, max=96577, avg=44163.61, stdev=9193.70 lat (msec): min=19, max=102, avg=47.19, stdev= 9.42 clat percentiles (usec): | 1.00th=[27657], 5.00th=[31589], 10.00th=[34341], 20.00th=[36963], | 30.00th=[39060], 40.00th=[40633], 50.00th=[43254], 60.00th=[45351], | 70.00th=[47973], 80.00th=[50594], 90.00th=[54789], 95.00th=[58983], | 99.00th=[73925], 99.50th=[78119], 99.90th=[93848], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 568, max= 808, per=24.60%, avg=662.79, stdev=74.88, samples=14 iops : min= 142, max= 202, avg=165.64, stdev=18.72, samples=14 write: IOPS=176, BW=707KiB/s (724kB/s)(5264KiB/7444msec); 0 zone resets slat (usec): min=56, max=3388, avg=174.40, stdev=148.15 clat (msec): min=24, max=101, avg=45.17, stdev= 8.54 lat (msec): min=24, max=101, avg=45.35, stdev= 8.55 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 34], 10.00th=[ 36], 20.00th=[ 39], | 30.00th=[ 41], 40.00th=[ 43], 50.00th=[ 44], 60.00th=[ 46], | 70.00th=[ 48], 80.00th=[ 52], 90.00th=[ 57], 95.00th=[ 61], | 99.00th=[ 71], 99.50th=[ 81], 99.90th=[ 96], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 608, max= 832, per=25.23%, avg=707.57, stdev=67.95, samples=14 iops : min= 152, max= 208, avg=176.79, stdev=17.06, samples=14 lat (msec) : 10=0.04%, 50=77.30%, 100=22.62%, 250=0.04% cpu : usr=0.55%, sys=5.00%, ctx=4463, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182130: Mon Mar 2 13:25:46 2026 read: IOPS=167, BW=670KiB/s (686kB/s)(4984KiB/7444msec) slat (usec): min=1056, max=11580, avg=3012.04, stdev=1200.47 clat (usec): min=21061, max=98381, avg=44828.85, stdev=9146.37 lat (msec): min=24, max=102, avg=47.84, stdev= 9.35 clat percentiles (usec): | 1.00th=[28181], 5.00th=[32113], 10.00th=[34341], 20.00th=[36963], | 30.00th=[39584], 40.00th=[41681], 50.00th=[43779], 60.00th=[46400], | 70.00th=[48497], 80.00th=[51643], 90.00th=[57410], 95.00th=[61080], | 99.00th=[68682], 99.50th=[74974], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 593, max= 808, per=24.89%, avg=670.57, stdev=60.39, samples=14 iops : min= 148, max= 202, avg=167.57, stdev=15.07, samples=14 write: IOPS=176, BW=706KiB/s (723kB/s)(5256KiB/7444msec); 0 zone resets slat (usec): min=50, max=4917, avg=177.46, stdev=229.40 clat (usec): min=25512, max=92535, avg=44598.42, stdev=8605.85 lat (usec): min=25658, max=92633, avg=44776.97, stdev=8608.28 clat percentiles (usec): | 1.00th=[30278], 5.00th=[33817], 10.00th=[34866], 20.00th=[37487], | 30.00th=[39584], 40.00th=[41157], 50.00th=[43254], 60.00th=[44827], | 70.00th=[47449], 80.00th=[50594], 90.00th=[56361], 95.00th=[61080], | 99.00th=[70779], 99.50th=[81265], 99.90th=[89654], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 584, max= 840, per=24.92%, avg=699.07, stdev=81.77, samples=14 iops : min= 146, max= 210, avg=174.71, stdev=20.47, samples=14 lat (msec) : 50=76.68%, 100=23.32% cpu : usr=0.44%, sys=5.01%, ctx=4497, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182131: Mon Mar 2 13:25:46 2026 read: IOPS=169, BW=679KiB/s (696kB/s)(5060KiB/7448msec) slat (usec): min=1061, max=10152, avg=2996.24, stdev=1177.95 clat (msec): min=9, max=113, avg=44.36, stdev= 9.49 lat (msec): min=10, max=117, avg=47.36, stdev= 9.71 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 32], 10.00th=[ 34], 20.00th=[ 37], | 30.00th=[ 40], 40.00th=[ 42], 50.00th=[ 44], 60.00th=[ 46], | 70.00th=[ 48], 80.00th=[ 52], 90.00th=[ 57], 95.00th=[ 61], | 99.00th=[ 71], 99.50th=[ 77], 99.90th=[ 99], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 568, max= 784, per=24.96%, avg=672.50, stdev=69.32, samples=14 iops : min= 142, max= 196, avg=168.07, stdev=17.40, samples=14 write: IOPS=173, BW=695KiB/s (712kB/s)(5180KiB/7448msec); 0 zone resets slat (usec): min=37, max=1981, avg=164.91, stdev=106.06 clat (msec): min=10, max=104, avg=44.93, stdev= 9.38 lat (msec): min=10, max=104, avg=45.09, stdev= 9.39 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 33], 10.00th=[ 35], 20.00th=[ 39], | 30.00th=[ 40], 40.00th=[ 43], 50.00th=[ 44], 60.00th=[ 46], | 70.00th=[ 48], 80.00th=[ 51], 90.00th=[ 56], 95.00th=[ 61], | 99.00th=[ 75], 99.50th=[ 97], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 544, max= 792, per=24.67%, avg=691.93, stdev=84.61, samples=14 iops : min= 136, max= 198, avg=172.93, stdev=21.12, samples=14 lat (msec) : 10=0.04%, 20=0.35%, 50=76.88%, 100=22.58%, 250=0.16% cpu : usr=0.48%, sys=5.05%, ctx=4478, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2694KiB/s (2759kB/s), 668KiB/s-680KiB/s (685kB/s-697kB/s), io=19.6MiB (20.5MB), run=7419-7448msec WRITE: bw=2805KiB/s (2872kB/s), 695KiB/s-707KiB/s (712kB/s-724kB/s), io=20.4MiB (21.4MB), run=7419-7448msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=182136: Mon Mar 2 13:25:50 2026 read: IOPS=6, BW=258MiB/s (271MB/s)(40.0MiB/155msec) slat (nsec): min=10490k, max=10490k, avg=10490347.00, stdev= 0.00 clat (nsec): min=132120k, max=132120k, avg=132119925.00, stdev= 0.00 lat (nsec): min=142614k, max=142614k, avg=142614163.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 132], 10.00th=[ 132], 20.00th=[ 132], | 30.00th=[ 132], 40.00th=[ 132], 50.00th=[ 132], 60.00th=[ 132], | 70.00th=[ 132], 80.00th=[ 132], 90.00th=[ 132], 95.00th=[ 132], | 99.00th=[ 132], 99.50th=[ 132], 99.90th=[ 132], 99.95th=[ 132], | 99.99th=[ 132] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.49%, ctx=3, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=258MiB/s (271MB/s), 258MiB/s-258MiB/s (271MB/s-271MB/s), io=40.0MiB (41.9MB), run=155-155msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout