== sanity test 398c: run fio to test AIO ================= 13:48:16 (1763491696) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.17492 s, 240 MB/s osc.lustre-OST0000-osc-ffff973703476800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=183219: Tue Nov 18 13:48:30 2025 write: IOPS=243, BW=974KiB/s (998kB/s)(10.0MiB/10508msec); 0 zone resets slat (usec): min=33, max=2381, avg=141.89, stdev=123.01 clat (msec): min=28, max=116, avg=65.05, stdev=11.63 lat (msec): min=29, max=117, avg=65.19, stdev=11.63 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 46], 10.00th=[ 52], 20.00th=[ 57], | 30.00th=[ 60], 40.00th=[ 63], 50.00th=[ 66], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 74], 90.00th=[ 80], 95.00th=[ 84], | 99.00th=[ 95], 99.50th=[ 104], 99.90th=[ 109], 99.95th=[ 113], | 99.99th=[ 117] bw ( KiB/s): min= 776, max= 1254, per=24.88%, avg=969.60, stdev=103.07, samples=20 iops : min= 194, max= 313, avg=242.30, stdev=25.68, samples=20 lat (msec) : 50=8.28%, 100=91.05%, 250=0.66% cpu : usr=0.47%, sys=2.69%, ctx=2843, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183220: Tue Nov 18 13:48:30 2025 write: IOPS=243, BW=975KiB/s (998kB/s)(10.0MiB/10503msec); 0 zone resets slat (usec): min=24, max=2277, avg=131.35, stdev=131.38 clat (msec): min=31, max=108, avg=65.07, stdev=11.68 lat (msec): min=32, max=108, avg=65.21, stdev=11.69 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 45], 10.00th=[ 52], 20.00th=[ 57], | 30.00th=[ 60], 40.00th=[ 63], 50.00th=[ 65], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 74], 90.00th=[ 80], 95.00th=[ 84], | 99.00th=[ 99], 99.50th=[ 103], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 109] bw ( KiB/s): min= 768, max= 1254, per=24.89%, avg=970.00, stdev=105.65, samples=20 iops : min= 192, max= 313, avg=242.40, stdev=26.34, samples=20 lat (msec) : 50=8.44%, 100=90.74%, 250=0.82% cpu : usr=0.52%, sys=2.58%, ctx=2816, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183221: Tue Nov 18 13:48:30 2025 write: IOPS=244, BW=978KiB/s (1001kB/s)(10.0MiB/10475msec); 0 zone resets slat (usec): min=34, max=21957, avg=152.66, stdev=453.53 clat (msec): min=9, max=122, avg=64.92, stdev=12.37 lat (msec): min=9, max=122, avg=65.07, stdev=12.35 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 44], 10.00th=[ 51], 20.00th=[ 57], | 30.00th=[ 60], 40.00th=[ 63], 50.00th=[ 66], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 74], 90.00th=[ 80], 95.00th=[ 84], | 99.00th=[ 99], 99.50th=[ 105], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 123] bw ( KiB/s): min= 776, max= 1313, per=25.00%, avg=974.15, stdev=113.53, samples=20 iops : min= 194, max= 328, avg=243.45, stdev=28.34, samples=20 lat (msec) : 10=0.04%, 20=0.16%, 50=8.83%, 100=90.20%, 250=0.78% cpu : usr=0.47%, sys=2.94%, ctx=2968, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183222: Tue Nov 18 13:48:30 2025 write: IOPS=245, BW=981KiB/s (1004kB/s)(10.0MiB/10441msec); 0 zone resets slat (usec): min=22, max=2291, avg=135.76, stdev=132.47 clat (msec): min=10, max=138, avg=65.03, stdev=12.19 lat (msec): min=11, max=138, avg=65.17, stdev=12.20 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 45], 10.00th=[ 52], 20.00th=[ 57], | 30.00th=[ 60], 40.00th=[ 63], 50.00th=[ 66], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 74], 90.00th=[ 80], 95.00th=[ 84], | 99.00th=[ 101], 99.50th=[ 104], 99.90th=[ 121], 99.95th=[ 121], | 99.99th=[ 140] bw ( KiB/s): min= 776, max= 1418, per=25.08%, avg=977.40, stdev=128.59, samples=20 iops : min= 194, max= 354, avg=244.25, stdev=32.05, samples=20 lat (msec) : 20=0.08%, 50=7.93%, 100=90.94%, 250=1.05% cpu : usr=0.40%, sys=2.62%, ctx=2761, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3898KiB/s (3992kB/s), 974KiB/s-981KiB/s (998kB/s-1004kB/s), io=40.0MiB (41.9MB), run=10441-10508msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=183233: Tue Nov 18 13:48:42 2025 read: IOPS=149, BW=598KiB/s (612kB/s)(5048KiB/8440msec) slat (usec): min=38, max=17064, avg=3590.87, stdev=1377.81 clat (msec): min=12, max=100, avg=50.01, stdev=10.52 lat (msec): min=15, max=104, avg=53.61, stdev=10.97 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 36], 10.00th=[ 39], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 55], 80.00th=[ 59], 90.00th=[ 65], 95.00th=[ 70], | 99.00th=[ 78], 99.50th=[ 82], 99.90th=[ 97], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 526, max= 760, per=25.50%, avg=602.56, stdev=69.19, samples=16 iops : min= 131, max= 190, avg=150.50, stdev=17.34, samples=16 write: IOPS=153, BW=615KiB/s (630kB/s)(5192KiB/8440msec); 0 zone resets slat (usec): min=34, max=1455, avg=109.32, stdev=103.87 clat (msec): min=6, max=110, avg=51.38, stdev= 9.52 lat (msec): min=7, max=110, avg=51.49, stdev= 9.53 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 49], 50.00th=[ 51], 60.00th=[ 54], | 70.00th=[ 56], 80.00th=[ 59], 90.00th=[ 64], 95.00th=[ 68], | 99.00th=[ 80], 99.50th=[ 82], 99.90th=[ 101], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 400, max= 774, per=25.09%, avg=617.19, stdev=84.63, samples=16 iops : min= 100, max= 193, avg=154.12, stdev=21.15, samples=16 lat (msec) : 10=0.04%, 20=0.08%, 50=52.34%, 100=47.42%, 250=0.12% cpu : usr=0.45%, sys=3.77%, ctx=4633, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183234: Tue Nov 18 13:48:42 2025 read: IOPS=147, BW=589KiB/s (603kB/s)(4976KiB/8454msec) slat (usec): min=45, max=10311, avg=3584.10, stdev=1252.21 clat (usec): min=24356, max=95046, avg=50494.96, stdev=10459.27 lat (msec): min=29, max=100, avg=54.08, stdev=10.82 clat percentiles (usec): | 1.00th=[31589], 5.00th=[35914], 10.00th=[38536], 20.00th=[41681], | 30.00th=[44303], 40.00th=[46400], 50.00th=[49021], 60.00th=[51643], | 70.00th=[54789], 80.00th=[58983], 90.00th=[64750], 95.00th=[69731], | 99.00th=[79168], 99.50th=[86508], 99.90th=[93848], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 448, max= 664, per=24.95%, avg=589.56, stdev=60.97, samples=16 iops : min= 112, max= 166, avg=147.25, stdev=15.28, samples=16 write: IOPS=155, BW=623KiB/s (638kB/s)(5264KiB/8454msec); 0 zone resets slat (usec): min=30, max=4204, avg=102.82, stdev=130.60 clat (msec): min=18, max=118, avg=51.37, stdev= 9.67 lat (msec): min=18, max=119, avg=51.47, stdev= 9.68 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 51], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 59], 90.00th=[ 64], 95.00th=[ 69], | 99.00th=[ 78], 99.50th=[ 84], 99.90th=[ 114], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 400, max= 734, per=25.49%, avg=627.06, stdev=88.65, samples=16 iops : min= 100, max= 183, avg=156.62, stdev=22.14, samples=16 lat (msec) : 20=0.08%, 50=51.91%, 100=47.93%, 250=0.08% cpu : usr=0.43%, sys=3.61%, ctx=4547, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183235: Tue Nov 18 13:48:42 2025 read: IOPS=146, BW=587KiB/s (601kB/s)(4984KiB/8492msec) slat (usec): min=1251, max=18477, avg=3677.93, stdev=1403.16 clat (usec): min=9514, max=93208, avg=50511.12, stdev=10128.09 lat (usec): min=13090, max=99050, avg=54192.15, stdev=10519.99 clat percentiles (usec): | 1.00th=[31851], 5.00th=[36439], 10.00th=[38536], 20.00th=[41681], | 30.00th=[44827], 40.00th=[46924], 50.00th=[49021], 60.00th=[52167], | 70.00th=[55313], 80.00th=[58983], 90.00th=[64226], 95.00th=[67634], | 99.00th=[78119], 99.50th=[80217], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 463, max= 686, per=25.03%, avg=591.44, stdev=64.98, samples=16 iops : min= 115, max= 171, avg=147.69, stdev=16.31, samples=16 write: IOPS=154, BW=619KiB/s (634kB/s)(5256KiB/8492msec); 0 zone resets slat (usec): min=34, max=2758, avg=113.44, stdev=131.82 clat (usec): min=13325, max=89454, avg=51230.47, stdev=9193.73 lat (usec): min=13413, max=89563, avg=51345.04, stdev=9195.64 clat percentiles (usec): | 1.00th=[34866], 5.00th=[38011], 10.00th=[40633], 20.00th=[43254], | 30.00th=[45876], 40.00th=[47973], 50.00th=[50070], 60.00th=[52167], | 70.00th=[55313], 80.00th=[58983], 90.00th=[63701], 95.00th=[67634], | 99.00th=[76022], 99.50th=[78119], 99.90th=[82314], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 408, max= 776, per=25.03%, avg=615.75, stdev=91.60, samples=16 iops : min= 102, max= 194, avg=153.81, stdev=22.90, samples=16 lat (msec) : 10=0.04%, 20=0.04%, 50=50.78%, 100=49.14% cpu : usr=0.39%, sys=3.79%, ctx=4586, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183236: Tue Nov 18 13:48:42 2025 read: IOPS=148, BW=596KiB/s (610kB/s)(5060KiB/8492msec) slat (usec): min=1372, max=17180, avg=3629.67, stdev=1323.31 clat (usec): min=28686, max=93750, avg=50279.24, stdev=9955.00 lat (usec): min=31194, max=98277, avg=53912.02, stdev=10320.68 clat percentiles (usec): | 1.00th=[32375], 5.00th=[35914], 10.00th=[38536], 20.00th=[41681], | 30.00th=[43779], 40.00th=[46400], 50.00th=[49021], 60.00th=[51643], | 70.00th=[55313], 80.00th=[58983], 90.00th=[64226], 95.00th=[68682], | 99.00th=[74974], 99.50th=[80217], 99.90th=[86508], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 432, max= 718, per=25.30%, avg=597.94, stdev=72.07, samples=16 iops : min= 108, max= 179, avg=149.31, stdev=18.06, samples=16 write: IOPS=152, BW=610KiB/s (625kB/s)(5180KiB/8492msec); 0 zone resets slat (usec): min=33, max=1112, avg=106.30, stdev=80.97 clat (msec): min=14, max=100, avg=51.55, stdev= 9.38 lat (msec): min=15, max=100, avg=51.65, stdev= 9.38 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 49], 50.00th=[ 51], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 59], 90.00th=[ 65], 95.00th=[ 68], | 99.00th=[ 80], 99.50th=[ 81], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 504, max= 728, per=24.94%, avg=613.44, stdev=68.35, samples=16 iops : min= 126, max= 182, avg=153.19, stdev=17.16, samples=16 lat (msec) : 20=0.04%, 50=50.31%, 100=49.57%, 250=0.08% cpu : usr=0.47%, sys=3.65%, ctx=4588, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2363KiB/s (2420kB/s), 587KiB/s-598KiB/s (601kB/s-612kB/s), io=19.6MiB (20.5MB), run=8440-8492msec WRITE: bw=2460KiB/s (2519kB/s), 610KiB/s-623KiB/s (625kB/s-638kB/s), io=20.4MiB (21.4MB), run=8440-8492msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=183241: Tue Nov 18 13:48:46 2025 read: IOPS=7, BW=284MiB/s (297MB/s)(40.0MiB/141msec) slat (nsec): min=13148k, max=13148k, avg=13147745.00, stdev= 0.00 clat (nsec): min=120261k, max=120261k, avg=120260933.00, stdev= 0.00 lat (nsec): min=133413k, max=133413k, avg=133412743.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 121], 10.00th=[ 121], 20.00th=[ 121], | 30.00th=[ 121], 40.00th=[ 121], 50.00th=[ 121], 60.00th=[ 121], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 121], 95.00th=[ 121], | 99.00th=[ 121], 99.50th=[ 121], 99.90th=[ 121], 99.95th=[ 121], | 99.99th=[ 121] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=8.57%, ctx=12, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=284MiB/s (297MB/s), 284MiB/s-284MiB/s (297MB/s-297MB/s), io=40.0MiB (41.9MB), run=141-141msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout