== sanity test 398c: run fio to test AIO ================= 13:23:48 (1772475828) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.179452 s, 234 MB/s osc.lustre-OST0000-osc-ffff9b0a32f9d800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=181553: Mon Mar 2 13:24:02 2026 write: IOPS=268, BW=1074KiB/s (1100kB/s)(10.0MiB/9534msec); 0 zone resets slat (usec): min=37, max=18159, avg=203.04, stdev=411.45 clat (msec): min=7, max=101, avg=59.00, stdev=14.51 lat (msec): min=8, max=102, avg=59.20, stdev=14.50 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 37], 10.00th=[ 42], 20.00th=[ 46], | 30.00th=[ 50], 40.00th=[ 55], 50.00th=[ 59], 60.00th=[ 63], | 70.00th=[ 66], 80.00th=[ 70], 90.00th=[ 80], 95.00th=[ 87], | 99.00th=[ 95], 99.50th=[ 101], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 103] bw ( KiB/s): min= 886, max= 1408, per=24.61%, avg=1054.89, stdev=156.15, samples=18 iops : min= 221, max= 352, avg=263.61, stdev=39.05, samples=18 lat (msec) : 10=0.04%, 20=0.08%, 50=31.72%, 100=67.77%, 250=0.39% cpu : usr=0.37%, sys=3.69%, ctx=2675, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181554: Mon Mar 2 13:24:02 2026 write: IOPS=267, BW=1072KiB/s (1097kB/s)(10.0MiB/9556msec); 0 zone resets slat (usec): min=38, max=4061, avg=204.87, stdev=220.18 clat (msec): min=7, max=101, avg=59.07, stdev=14.63 lat (msec): min=7, max=102, avg=59.27, stdev=14.64 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 46], | 30.00th=[ 50], 40.00th=[ 54], 50.00th=[ 59], 60.00th=[ 63], | 70.00th=[ 66], 80.00th=[ 71], 90.00th=[ 79], 95.00th=[ 87], | 99.00th=[ 95], 99.50th=[ 97], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 102] bw ( KiB/s): min= 824, max= 1408, per=24.53%, avg=1051.33, stdev=155.84, samples=18 iops : min= 206, max= 352, avg=262.72, stdev=38.97, samples=18 lat (msec) : 10=0.08%, 20=0.08%, 50=32.23%, 100=67.54%, 250=0.08% cpu : usr=0.39%, sys=3.94%, ctx=2833, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181555: Mon Mar 2 13:24:02 2026 write: IOPS=267, BW=1072KiB/s (1097kB/s)(10.0MiB/9556msec); 0 zone resets slat (usec): min=34, max=2192, avg=147.95, stdev=138.84 clat (msec): min=18, max=103, avg=59.10, stdev=14.93 lat (msec): min=19, max=104, avg=59.25, stdev=14.92 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 47], | 30.00th=[ 49], 40.00th=[ 53], 50.00th=[ 58], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 72], 90.00th=[ 80], 95.00th=[ 87], | 99.00th=[ 96], 99.50th=[ 96], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 792, max= 1408, per=24.48%, avg=1049.17, stdev=160.66, samples=18 iops : min= 198, max= 352, avg=262.17, stdev=40.18, samples=18 lat (msec) : 20=0.04%, 50=34.77%, 100=64.84%, 250=0.35% cpu : usr=0.38%, sys=2.83%, ctx=1922, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181556: Mon Mar 2 13:24:02 2026 write: IOPS=269, BW=1077KiB/s (1103kB/s)(10.0MiB/9507msec); 0 zone resets slat (usec): min=46, max=2864, avg=201.16, stdev=165.70 clat (msec): min=9, max=101, avg=59.13, stdev=14.36 lat (msec): min=9, max=101, avg=59.34, stdev=14.36 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 46], | 30.00th=[ 50], 40.00th=[ 55], 50.00th=[ 59], 60.00th=[ 63], | 70.00th=[ 66], 80.00th=[ 71], 90.00th=[ 79], 95.00th=[ 87], | 99.00th=[ 95], 99.50th=[ 96], 99.90th=[ 100], 99.95th=[ 101], | 99.99th=[ 102] bw ( KiB/s): min= 790, max= 1408, per=24.71%, avg=1059.00, stdev=163.01, samples=18 iops : min= 197, max= 352, avg=264.67, stdev=40.79, samples=18 lat (msec) : 10=0.04%, 20=0.12%, 50=31.45%, 100=68.36%, 250=0.04% cpu : usr=0.49%, sys=3.73%, ctx=2735, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=4286KiB/s (4389kB/s), 1072KiB/s-1077KiB/s (1097kB/s-1103kB/s), io=40.0MiB (41.9MB), run=9507-9556msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=181567: Mon Mar 2 13:24:14 2026 read: IOPS=150, BW=601KiB/s (615kB/s)(5048KiB/8400msec) slat (usec): min=1302, max=16857, avg=3517.08, stdev=1429.65 clat (usec): min=25824, max=89143, avg=49736.61, stdev=10813.79 lat (msec): min=30, max=106, avg=53.26, stdev=11.22 clat percentiles (usec): | 1.00th=[31589], 5.00th=[35390], 10.00th=[37487], 20.00th=[40633], | 30.00th=[43254], 40.00th=[45351], 50.00th=[47973], 60.00th=[50594], | 70.00th=[53740], 80.00th=[58983], 90.00th=[65799], 95.00th=[70779], | 99.00th=[80217], 99.50th=[82314], 99.90th=[87557], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 456, max= 752, per=25.15%, avg=600.88, stdev=81.16, samples=16 iops : min= 114, max= 188, avg=150.19, stdev=20.32, samples=16 write: IOPS=154, BW=618KiB/s (633kB/s)(5192KiB/8400msec); 0 zone resets slat (usec): min=46, max=3092, avg=195.76, stdev=186.35 clat (usec): min=27317, max=91308, avg=51027.24, stdev=10383.93 lat (usec): min=27514, max=91423, avg=51224.16, stdev=10392.06 clat percentiles (usec): | 1.00th=[33162], 5.00th=[36439], 10.00th=[39060], 20.00th=[42206], | 30.00th=[44303], 40.00th=[46924], 50.00th=[49546], 60.00th=[52167], | 70.00th=[54789], 80.00th=[58983], 90.00th=[65274], 95.00th=[71828], | 99.00th=[81265], 99.50th=[83362], 99.90th=[86508], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 471, max= 888, per=24.71%, avg=614.62, stdev=111.01, samples=16 iops : min= 117, max= 222, avg=153.56, stdev=27.80, samples=16 lat (msec) : 50=55.66%, 100=44.34% cpu : usr=0.58%, sys=5.02%, ctx=4463, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181568: Mon Mar 2 13:24:14 2026 read: IOPS=149, BW=598KiB/s (612kB/s)(4976KiB/8323msec) slat (usec): min=54, max=12657, avg=3485.51, stdev=1356.16 clat (usec): min=5871, max=95332, avg=49845.16, stdev=11294.80 lat (msec): min=5, max=100, avg=53.33, stdev=11.73 clat percentiles (usec): | 1.00th=[30016], 5.00th=[34341], 10.00th=[37487], 20.00th=[41157], | 30.00th=[43254], 40.00th=[45876], 50.00th=[47973], 60.00th=[50594], | 70.00th=[53740], 80.00th=[58983], 90.00th=[65799], 95.00th=[71828], | 99.00th=[80217], 99.50th=[82314], 99.90th=[91751], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 480, max= 728, per=25.01%, avg=597.44, stdev=81.45, samples=16 iops : min= 120, max= 182, avg=149.31, stdev=20.43, samples=16 write: IOPS=158, BW=632KiB/s (648kB/s)(5264KiB/8323msec); 0 zone resets slat (usec): min=59, max=2693, avg=199.37, stdev=157.72 clat (usec): min=7881, max=92149, avg=50400.13, stdev=10493.94 lat (usec): min=8097, max=92310, avg=50600.74, stdev=10498.17 clat percentiles (usec): | 1.00th=[30278], 5.00th=[36439], 10.00th=[38536], 20.00th=[41681], | 30.00th=[44303], 40.00th=[46400], 50.00th=[48497], 60.00th=[51119], | 70.00th=[54264], 80.00th=[58459], 90.00th=[66323], 95.00th=[70779], | 99.00th=[78119], 99.50th=[81265], 99.90th=[89654], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 456, max= 888, per=25.38%, avg=631.12, stdev=111.10, samples=16 iops : min= 114, max= 222, avg=157.75, stdev=27.79, samples=16 lat (msec) : 10=0.23%, 20=0.08%, 50=57.03%, 100=42.66% cpu : usr=0.66%, sys=5.05%, ctx=4428, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181569: Mon Mar 2 13:24:14 2026 read: IOPS=148, BW=594KiB/s (608kB/s)(4984KiB/8392msec) slat (usec): min=1401, max=12887, avg=3556.97, stdev=1461.67 clat (usec): min=25114, max=85570, avg=49849.69, stdev=10236.91 lat (usec): min=28115, max=95678, avg=53410.32, stdev=10692.26 clat percentiles (usec): | 1.00th=[31065], 5.00th=[35390], 10.00th=[38011], 20.00th=[41157], | 30.00th=[43779], 40.00th=[46400], 50.00th=[48497], 60.00th=[50594], | 70.00th=[53740], 80.00th=[57934], 90.00th=[64226], 95.00th=[69731], | 99.00th=[78119], 99.50th=[79168], 99.90th=[82314], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 408, max= 776, per=24.91%, avg=595.13, stdev=98.26, samples=16 iops : min= 102, max= 194, avg=148.69, stdev=24.64, samples=16 write: IOPS=156, BW=626KiB/s (641kB/s)(5256KiB/8392msec); 0 zone resets slat (usec): min=54, max=3904, avg=200.82, stdev=216.94 clat (usec): min=29376, max=89096, avg=50635.62, stdev=9907.99 lat (usec): min=29548, max=89213, avg=50837.62, stdev=9910.19 clat percentiles (usec): | 1.00th=[33424], 5.00th=[37487], 10.00th=[39584], 20.00th=[42206], | 30.00th=[44827], 40.00th=[46924], 50.00th=[49021], 60.00th=[51119], | 70.00th=[54264], 80.00th=[58459], 90.00th=[65274], 95.00th=[69731], | 99.00th=[77071], 99.50th=[80217], 99.90th=[84411], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 480, max= 824, per=24.94%, avg=620.38, stdev=82.59, samples=16 iops : min= 120, max= 206, avg=155.06, stdev=20.65, samples=16 lat (msec) : 50=55.43%, 100=44.57% cpu : usr=0.43%, sys=5.27%, ctx=4363, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181570: Mon Mar 2 13:24:14 2026 read: IOPS=150, BW=603KiB/s (617kB/s)(5060KiB/8392msec) slat (usec): min=1343, max=11724, avg=3474.66, stdev=1347.72 clat (usec): min=9289, max=94344, avg=49797.39, stdev=10812.97 lat (msec): min=13, max=102, avg=53.28, stdev=11.20 clat percentiles (usec): | 1.00th=[30016], 5.00th=[34866], 10.00th=[37487], 20.00th=[41157], | 30.00th=[43779], 40.00th=[45876], 50.00th=[48497], 60.00th=[50594], | 70.00th=[53740], 80.00th=[57934], 90.00th=[64226], 95.00th=[69731], | 99.00th=[80217], 99.50th=[87557], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 416, max= 792, per=25.22%, avg=602.56, stdev=93.75, samples=16 iops : min= 104, max= 198, avg=150.56, stdev=23.46, samples=16 write: IOPS=154, BW=617KiB/s (632kB/s)(5180KiB/8392msec); 0 zone resets slat (usec): min=42, max=3771, avg=198.11, stdev=172.20 clat (usec): min=26519, max=91550, avg=50901.52, stdev=10261.20 lat (usec): min=26847, max=91654, avg=51100.98, stdev=10250.99 clat percentiles (usec): | 1.00th=[32900], 5.00th=[37487], 10.00th=[39584], 20.00th=[42206], | 30.00th=[44827], 40.00th=[46400], 50.00th=[49021], 60.00th=[52167], | 70.00th=[54789], 80.00th=[58459], 90.00th=[65799], 95.00th=[70779], | 99.00th=[80217], 99.50th=[84411], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 472, max= 744, per=24.70%, avg=614.37, stdev=84.26, samples=16 iops : min= 118, max= 186, avg=153.56, stdev=21.09, samples=16 lat (msec) : 10=0.04%, 50=55.27%, 100=44.69% cpu : usr=0.56%, sys=5.17%, ctx=4455, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2389KiB/s (2446kB/s), 594KiB/s-603KiB/s (608kB/s-617kB/s), io=19.6MiB (20.5MB), run=8323-8400msec WRITE: bw=2487KiB/s (2547kB/s), 617KiB/s-632KiB/s (632kB/s-648kB/s), io=20.4MiB (21.4MB), run=8323-8400msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=181575: Mon Mar 2 13:24:18 2026 read: IOPS=4, BW=196MiB/s (206MB/s)(40.0MiB/204msec) slat (nsec): min=14354k, max=14354k, avg=14354195.00, stdev= 0.00 clat (nsec): min=183514k, max=183514k, avg=183514334.00, stdev= 0.00 lat (nsec): min=197874k, max=197874k, avg=197873833.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 184], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.37%, ctx=11, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=196MiB/s (206MB/s), 196MiB/s-196MiB/s (206MB/s-206MB/s), io=40.0MiB (41.9MB), run=204-204msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout