== sanity test 398c: run fio to test AIO ================= 01:44:49 (1783143889) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.17166 s, 244 MB/s osc.lustre-OST0000-osc-ffff893d89db2000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=181365: Sat Jul 4 01:45:06 2026 write: IOPS=200, BW=803KiB/s (822kB/s)(10.0MiB/12749msec); 0 zone resets slat (usec): min=35, max=4402, avg=223.85, stdev=311.22 clat (msec): min=10, max=170, avg=79.32, stdev=16.82 lat (msec): min=11, max=170, avg=79.55, stdev=16.82 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 56], 10.00th=[ 59], 20.00th=[ 66], | 30.00th=[ 71], 40.00th=[ 74], 50.00th=[ 79], 60.00th=[ 82], | 70.00th=[ 87], 80.00th=[ 92], 90.00th=[ 101], 95.00th=[ 108], | 99.00th=[ 134], 99.50th=[ 140], 99.90th=[ 159], 99.95th=[ 167], | 99.99th=[ 171] bw ( KiB/s): min= 592, max= 984, per=25.20%, avg=804.60, stdev=83.83, samples=25 iops : min= 148, max= 246, avg=200.96, stdev=20.95, samples=25 lat (msec) : 20=0.04%, 50=1.56%, 100=88.12%, 250=10.27% cpu : usr=0.25%, sys=2.72%, ctx=2860, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181366: Sat Jul 4 01:45:06 2026 write: IOPS=199, BW=798KiB/s (817kB/s)(10.0MiB/12827msec); 0 zone resets slat (usec): min=37, max=43066, avg=221.06, stdev=890.34 clat (msec): min=17, max=142, avg=79.36, stdev=16.29 lat (msec): min=18, max=142, avg=79.58, stdev=16.25 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 56], 10.00th=[ 60], 20.00th=[ 66], | 30.00th=[ 70], 40.00th=[ 75], 50.00th=[ 79], 60.00th=[ 83], | 70.00th=[ 87], 80.00th=[ 92], 90.00th=[ 102], 95.00th=[ 108], | 99.00th=[ 126], 99.50th=[ 136], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 142] bw ( KiB/s): min= 624, max= 958, per=24.93%, avg=796.04, stdev=81.57, samples=25 iops : min= 156, max= 239, avg=198.80, stdev=20.38, samples=25 lat (msec) : 20=0.04%, 50=1.37%, 100=87.34%, 250=11.25% cpu : usr=0.32%, sys=2.83%, ctx=2810, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181367: Sat Jul 4 01:45:06 2026 write: IOPS=199, BW=798KiB/s (817kB/s)(10.0MiB/12827msec); 0 zone resets slat (usec): min=36, max=28702, avg=205.61, stdev=622.42 clat (msec): min=9, max=139, avg=79.37, stdev=16.12 lat (msec): min=10, max=139, avg=79.57, stdev=16.09 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 57], 10.00th=[ 61], 20.00th=[ 66], | 30.00th=[ 71], 40.00th=[ 75], 50.00th=[ 79], 60.00th=[ 82], | 70.00th=[ 87], 80.00th=[ 91], 90.00th=[ 102], 95.00th=[ 109], | 99.00th=[ 129], 99.50th=[ 133], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 140] bw ( KiB/s): min= 584, max= 992, per=25.04%, avg=799.44, stdev=96.88, samples=25 iops : min= 146, max= 248, avg=199.64, stdev=24.23, samples=25 lat (msec) : 10=0.04%, 20=0.04%, 50=1.91%, 100=87.15%, 250=10.86% cpu : usr=0.34%, sys=2.47%, ctx=2674, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181368: Sat Jul 4 01:45:06 2026 write: IOPS=199, BW=798KiB/s (818kB/s)(10.0MiB/12825msec); 0 zone resets slat (usec): min=36, max=4746, avg=220.16, stdev=311.79 clat (msec): min=17, max=140, avg=79.25, stdev=16.17 lat (msec): min=17, max=141, avg=79.47, stdev=16.18 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 55], 10.00th=[ 61], 20.00th=[ 66], | 30.00th=[ 70], 40.00th=[ 75], 50.00th=[ 79], 60.00th=[ 82], | 70.00th=[ 87], 80.00th=[ 92], 90.00th=[ 101], 95.00th=[ 110], | 99.00th=[ 126], 99.50th=[ 130], 99.90th=[ 132], 99.95th=[ 134], | 99.99th=[ 142] bw ( KiB/s): min= 640, max= 976, per=25.01%, avg=798.48, stdev=82.30, samples=25 iops : min= 160, max= 244, avg=199.40, stdev=20.61, samples=25 lat (msec) : 20=0.04%, 50=1.56%, 100=88.24%, 250=10.16% cpu : usr=0.35%, sys=2.76%, ctx=2868, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3193KiB/s (3270kB/s), 798KiB/s-803KiB/s (817kB/s-822kB/s), io=40.0MiB (41.9MB), run=12749-12827msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=181379: Sat Jul 4 01:45:18 2026 read: IOPS=156, BW=628KiB/s (643kB/s)(5048KiB/8042msec) slat (usec): min=1357, max=10211, avg=3288.42, stdev=1273.06 clat (usec): min=25191, max=92363, avg=47541.03, stdev=9591.88 lat (usec): min=29342, max=97640, avg=50832.31, stdev=9948.03 clat percentiles (usec): | 1.00th=[30540], 5.00th=[34341], 10.00th=[36439], 20.00th=[39584], | 30.00th=[42206], 40.00th=[43779], 50.00th=[46400], 60.00th=[48497], | 70.00th=[51119], 80.00th=[54264], 90.00th=[60556], 95.00th=[66323], | 99.00th=[73925], 99.50th=[78119], 99.90th=[91751], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 478, max= 752, per=24.98%, avg=623.33, stdev=92.88, samples=15 iops : min= 119, max= 188, avg=155.73, stdev=23.28, samples=15 write: IOPS=161, BW=646KiB/s (661kB/s)(5192KiB/8042msec); 0 zone resets slat (usec): min=56, max=5199, avg=190.60, stdev=188.62 clat (usec): min=28824, max=89805, avg=48624.11, stdev=8455.11 lat (usec): min=29002, max=89949, avg=48815.97, stdev=8462.67 clat percentiles (usec): | 1.00th=[32637], 5.00th=[36963], 10.00th=[39584], 20.00th=[42206], | 30.00th=[43779], 40.00th=[45351], 50.00th=[46924], 60.00th=[49021], | 70.00th=[51643], 80.00th=[54789], 90.00th=[60031], 95.00th=[64750], | 99.00th=[72877], 99.50th=[74974], 99.90th=[85459], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 440, max= 824, per=24.76%, avg=643.07, stdev=104.78, samples=15 iops : min= 110, max= 206, avg=160.67, stdev=26.26, samples=15 lat (msec) : 50=65.20%, 100=34.80% cpu : usr=0.56%, sys=4.96%, ctx=4348, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181380: Sat Jul 4 01:45:18 2026 read: IOPS=154, BW=619KiB/s (634kB/s)(4976KiB/8035msec) slat (usec): min=1217, max=10698, avg=3300.52, stdev=1286.45 clat (usec): min=10984, max=88619, avg=47499.08, stdev=9462.39 lat (usec): min=18291, max=93976, avg=50802.82, stdev=9797.12 clat percentiles (usec): | 1.00th=[30278], 5.00th=[33817], 10.00th=[36439], 20.00th=[39584], | 30.00th=[42206], 40.00th=[44303], 50.00th=[46400], 60.00th=[48497], | 70.00th=[51119], 80.00th=[54789], 90.00th=[60556], 95.00th=[64750], | 99.00th=[72877], 99.50th=[77071], 99.90th=[84411], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 384, max= 712, per=24.47%, avg=610.60, stdev=92.22, samples=15 iops : min= 96, max= 178, avg=152.53, stdev=23.12, samples=15 write: IOPS=163, BW=655KiB/s (671kB/s)(5264KiB/8035msec); 0 zone resets slat (usec): min=49, max=2074, avg=179.10, stdev=137.90 clat (usec): min=15212, max=87375, avg=48741.19, stdev=8992.59 lat (usec): min=15330, max=87614, avg=48921.73, stdev=9002.14 clat percentiles (usec): | 1.00th=[32900], 5.00th=[36439], 10.00th=[38536], 20.00th=[41681], | 30.00th=[43254], 40.00th=[45351], 50.00th=[47449], 60.00th=[49021], | 70.00th=[51643], 80.00th=[55837], 90.00th=[61604], 95.00th=[66323], | 99.00th=[73925], 99.50th=[78119], 99.90th=[84411], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 516, max= 796, per=25.31%, avg=657.27, stdev=90.58, samples=15 iops : min= 129, max= 199, avg=164.27, stdev=22.70, samples=15 lat (msec) : 20=0.08%, 50=64.41%, 100=35.51% cpu : usr=0.49%, sys=4.85%, ctx=4296, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181381: Sat Jul 4 01:45:18 2026 read: IOPS=154, BW=620KiB/s (635kB/s)(4984KiB/8041msec) slat (usec): min=1290, max=9454, avg=3289.52, stdev=1194.79 clat (usec): min=5680, max=84189, avg=47856.42, stdev=9532.21 lat (usec): min=7680, max=88766, avg=51148.86, stdev=9863.60 clat percentiles (usec): | 1.00th=[29754], 5.00th=[33817], 10.00th=[36963], 20.00th=[40109], | 30.00th=[42206], 40.00th=[44827], 50.00th=[46924], 60.00th=[49021], | 70.00th=[51643], 80.00th=[54789], 90.00th=[61080], 95.00th=[65799], | 99.00th=[74974], 99.50th=[76022], 99.90th=[83362], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 427, max= 776, per=24.83%, avg=619.40, stdev=92.01, samples=15 iops : min= 106, max= 194, avg=154.73, stdev=23.09, samples=15 write: IOPS=163, BW=654KiB/s (669kB/s)(5256KiB/8041msec); 0 zone resets slat (usec): min=42, max=6045, avg=192.71, stdev=236.05 clat (usec): min=9793, max=89160, avg=48276.02, stdev=8779.51 lat (usec): min=10311, max=89351, avg=48470.06, stdev=8781.98 clat percentiles (usec): | 1.00th=[31851], 5.00th=[36439], 10.00th=[38536], 20.00th=[41157], | 30.00th=[43254], 40.00th=[45351], 50.00th=[46924], 60.00th=[49021], | 70.00th=[51119], 80.00th=[54789], 90.00th=[60031], 95.00th=[63701], | 99.00th=[74974], 99.50th=[80217], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 408, max= 824, per=24.74%, avg=642.47, stdev=115.54, samples=15 iops : min= 102, max= 206, avg=160.53, stdev=28.93, samples=15 lat (msec) : 10=0.08%, 20=0.08%, 50=64.22%, 100=35.62% cpu : usr=0.47%, sys=4.95%, ctx=4318, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181382: Sat Jul 4 01:45:18 2026 read: IOPS=158, BW=632KiB/s (647kB/s)(5060KiB/8005msec) slat (usec): min=86, max=12293, avg=3325.20, stdev=1351.17 clat (usec): min=7988, max=88848, avg=47585.07, stdev=9841.02 lat (usec): min=8846, max=92464, avg=50913.33, stdev=10196.68 clat percentiles (usec): | 1.00th=[30540], 5.00th=[34341], 10.00th=[36963], 20.00th=[39584], | 30.00th=[42206], 40.00th=[44303], 50.00th=[46400], 60.00th=[48497], | 70.00th=[51643], 80.00th=[54264], 90.00th=[60556], 95.00th=[66323], | 99.00th=[76022], 99.50th=[78119], 99.90th=[85459], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 480, max= 768, per=25.15%, avg=627.60, stdev=85.59, samples=15 iops : min= 120, max= 192, avg=156.80, stdev=21.46, samples=15 write: IOPS=161, BW=647KiB/s (663kB/s)(5180KiB/8005msec); 0 zone resets slat (usec): min=47, max=2184, avg=180.23, stdev=144.19 clat (usec): min=31248, max=84488, avg=48755.22, stdev=8795.62 lat (usec): min=31432, max=84628, avg=48936.61, stdev=8799.08 clat percentiles (usec): | 1.00th=[33817], 5.00th=[36963], 10.00th=[39060], 20.00th=[41681], | 30.00th=[43254], 40.00th=[45351], 50.00th=[47449], 60.00th=[49546], | 70.00th=[51643], 80.00th=[55313], 90.00th=[61080], 95.00th=[65799], | 99.00th=[74974], 99.50th=[78119], 99.90th=[82314], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 456, max= 752, per=24.90%, avg=646.60, stdev=83.89, samples=15 iops : min= 114, max= 188, avg=161.53, stdev=21.02, samples=15 lat (msec) : 10=0.16%, 20=0.12%, 50=63.44%, 100=36.29% cpu : usr=0.56%, sys=4.80%, ctx=4379, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2495KiB/s (2555kB/s), 619KiB/s-632KiB/s (634kB/s-647kB/s), io=19.6MiB (20.5MB), run=8005-8042msec WRITE: bw=2598KiB/s (2660kB/s), 646KiB/s-655KiB/s (661kB/s-671kB/s), io=20.4MiB (21.4MB), run=8005-8042msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=181387: Sat Jul 4 01:45:21 2026 read: IOPS=6, BW=244MiB/s (256MB/s)(40.0MiB/164msec) slat (nsec): min=13547k, max=13547k, avg=13547222.00, stdev= 0.00 clat (nsec): min=143693k, max=143693k, avg=143692569.00, stdev= 0.00 lat (nsec): min=157245k, max=157245k, avg=157244623.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 144], 50.00th=[ 144], 60.00th=[ 144], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.13%, ctx=11, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=244MiB/s (256MB/s), 244MiB/s-244MiB/s (256MB/s-256MB/s), io=40.0MiB (41.9MB), run=164-164msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout