== sanity test 398c: run fio to test AIO ================= 21:55:19 (1787104519) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.159685 s, 263 MB/s osc.lustre-OST0000-osc-ffff9ffe797f3000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=185653: Tue Aug 18 21:55:38 2026 write: IOPS=170, BW=682KiB/s (698kB/s)(10.0MiB/15020msec); 0 zone resets slat (usec): min=35, max=7406, avg=221.46, stdev=371.04 clat (msec): min=15, max=161, avg=93.55, stdev=20.75 lat (msec): min=15, max=161, avg=93.77, stdev=20.77 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 78], | 30.00th=[ 83], 40.00th=[ 87], 50.00th=[ 92], 60.00th=[ 97], | 70.00th=[ 104], 80.00th=[ 111], 90.00th=[ 123], 95.00th=[ 131], | 99.00th=[ 146], 99.50th=[ 153], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 163] bw ( KiB/s): min= 520, max= 920, per=25.03%, avg=679.55, stdev=104.84, samples=29 iops : min= 130, max= 230, avg=169.79, stdev=26.16, samples=29 lat (msec) : 20=0.08%, 50=1.99%, 100=63.55%, 250=34.38% cpu : usr=0.27%, sys=2.18%, ctx=2582, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=185654: Tue Aug 18 21:55:38 2026 write: IOPS=170, BW=681KiB/s (698kB/s)(10.0MiB/15031msec); 0 zone resets slat (usec): min=35, max=7280, avg=239.99, stdev=448.44 clat (msec): min=9, max=160, avg=93.42, stdev=20.79 lat (msec): min=9, max=160, avg=93.66, stdev=20.81 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 63], 10.00th=[ 70], 20.00th=[ 77], | 30.00th=[ 83], 40.00th=[ 87], 50.00th=[ 91], 60.00th=[ 96], | 70.00th=[ 103], 80.00th=[ 111], 90.00th=[ 122], 95.00th=[ 130], | 99.00th=[ 144], 99.50th=[ 159], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 512, max= 952, per=25.03%, avg=679.45, stdev=107.14, samples=29 iops : min= 128, max= 238, avg=169.76, stdev=26.76, samples=29 lat (msec) : 10=0.04%, 20=0.04%, 50=1.72%, 100=63.91%, 250=34.30% cpu : usr=0.28%, sys=2.26%, ctx=2770, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=185655: Tue Aug 18 21:55:38 2026 write: IOPS=169, BW=679KiB/s (695kB/s)(10.0MiB/15078msec); 0 zone resets slat (usec): min=36, max=43103, avg=236.48, stdev=910.99 clat (msec): min=16, max=162, avg=93.37, stdev=20.66 lat (msec): min=16, max=162, avg=93.61, stdev=20.65 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 64], 10.00th=[ 70], 20.00th=[ 77], | 30.00th=[ 83], 40.00th=[ 87], 50.00th=[ 91], 60.00th=[ 96], | 70.00th=[ 103], 80.00th=[ 111], 90.00th=[ 122], 95.00th=[ 131], | 99.00th=[ 148], 99.50th=[ 155], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 163] bw ( KiB/s): min= 512, max= 968, per=24.86%, avg=674.83, stdev=108.13, samples=29 iops : min= 128, max= 242, avg=168.62, stdev=27.02, samples=29 lat (msec) : 20=0.04%, 50=1.60%, 100=65.51%, 250=32.85% cpu : usr=0.31%, sys=2.39%, ctx=2785, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=185656: Tue Aug 18 21:55:38 2026 write: IOPS=169, BW=679KiB/s (695kB/s)(10.0MiB/15083msec); 0 zone resets slat (usec): min=35, max=10734, avg=240.40, stdev=454.60 clat (msec): min=16, max=156, avg=93.38, stdev=20.57 lat (msec): min=17, max=157, avg=93.62, stdev=20.59 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 63], 10.00th=[ 70], 20.00th=[ 77], | 30.00th=[ 82], 40.00th=[ 87], 50.00th=[ 92], 60.00th=[ 97], | 70.00th=[ 104], 80.00th=[ 111], 90.00th=[ 123], 95.00th=[ 130], | 99.00th=[ 144], 99.50th=[ 148], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 512, max= 952, per=24.83%, avg=674.24, stdev=110.05, samples=29 iops : min= 128, max= 238, avg=168.48, stdev=27.49, samples=29 lat (msec) : 20=0.04%, 50=1.48%, 100=64.18%, 250=34.30% cpu : usr=0.36%, sys=2.51%, ctx=2803, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2716KiB/s (2781kB/s), 679KiB/s-682KiB/s (695kB/s-698kB/s), io=40.0MiB (41.9MB), run=15020-15083msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=185668: Tue Aug 18 21:55:52 2026 read: IOPS=112, BW=449KiB/s (460kB/s)(5048KiB/11232msec) slat (usec): min=1125, max=21481, avg=4688.24, stdev=2212.89 clat (msec): min=10, max=126, avg=66.15, stdev=14.81 lat (msec): min=14, max=134, avg=70.84, stdev=15.39 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 43], 10.00th=[ 47], 20.00th=[ 54], | 30.00th=[ 59], 40.00th=[ 63], 50.00th=[ 66], 60.00th=[ 70], | 70.00th=[ 74], 80.00th=[ 80], 90.00th=[ 85], 95.00th=[ 92], | 99.00th=[ 105], 99.50th=[ 108], 99.90th=[ 124], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 346, max= 544, per=25.14%, avg=448.23, stdev=50.00, samples=22 iops : min= 86, max= 136, avg=111.82, stdev=12.56, samples=22 write: IOPS=115, BW=462KiB/s (473kB/s)(5192KiB/11232msec); 0 zone resets slat (usec): min=53, max=4436, avg=200.74, stdev=281.57 clat (msec): min=21, max=122, avg=68.30, stdev=14.82 lat (msec): min=21, max=122, avg=68.51, stdev=14.83 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 45], 10.00th=[ 50], 20.00th=[ 56], | 30.00th=[ 61], 40.00th=[ 65], 50.00th=[ 68], 60.00th=[ 72], | 70.00th=[ 75], 80.00th=[ 81], 90.00th=[ 88], 95.00th=[ 95], | 99.00th=[ 109], 99.50th=[ 112], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 343, max= 568, per=24.90%, avg=462.41, stdev=56.47, samples=22 iops : min= 85, max= 142, avg=115.36, stdev=14.28, samples=22 lat (msec) : 20=0.08%, 50=12.62%, 100=85.39%, 250=1.91% cpu : usr=0.36%, sys=3.68%, ctx=4580, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=185669: Tue Aug 18 21:55:52 2026 read: IOPS=110, BW=442KiB/s (453kB/s)(4976KiB/11247msec) slat (usec): min=1329, max=16653, avg=4721.05, stdev=2174.58 clat (msec): min=5, max=118, avg=66.86, stdev=15.47 lat (msec): min=11, max=126, avg=71.59, stdev=15.95 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 44], 10.00th=[ 47], 20.00th=[ 53], | 30.00th=[ 59], 40.00th=[ 63], 50.00th=[ 67], 60.00th=[ 70], | 70.00th=[ 74], 80.00th=[ 80], 90.00th=[ 88], 95.00th=[ 94], | 99.00th=[ 107], 99.50th=[ 112], 99.90th=[ 116], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 333, max= 512, per=24.62%, avg=439.00, stdev=48.30, samples=22 iops : min= 83, max= 128, avg=109.50, stdev=12.23, samples=22 write: IOPS=117, BW=468KiB/s (479kB/s)(5264KiB/11247msec); 0 zone resets slat (usec): min=54, max=3538, avg=189.87, stdev=219.05 clat (msec): min=25, max=113, avg=67.96, stdev=14.30 lat (msec): min=25, max=113, avg=68.15, stdev=14.31 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 45], 10.00th=[ 49], 20.00th=[ 56], | 30.00th=[ 61], 40.00th=[ 65], 50.00th=[ 68], 60.00th=[ 71], | 70.00th=[ 75], 80.00th=[ 81], 90.00th=[ 87], 95.00th=[ 92], | 99.00th=[ 102], 99.50th=[ 106], 99.90th=[ 112], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 375, max= 616, per=25.15%, avg=467.05, stdev=70.53, samples=22 iops : min= 93, max= 154, avg=116.50, stdev=17.65, samples=22 lat (msec) : 10=0.04%, 50=13.20%, 100=84.57%, 250=2.19% cpu : usr=0.32%, sys=3.62%, ctx=4482, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=185670: Tue Aug 18 21:55:52 2026 read: IOPS=111, BW=445KiB/s (455kB/s)(4984KiB/11208msec) slat (usec): min=59, max=23288, avg=4661.72, stdev=2294.49 clat (msec): min=31, max=126, avg=66.62, stdev=15.02 lat (msec): min=32, max=139, avg=71.28, stdev=15.61 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 44], 10.00th=[ 48], 20.00th=[ 54], | 30.00th=[ 58], 40.00th=[ 63], 50.00th=[ 67], 60.00th=[ 70], | 70.00th=[ 74], 80.00th=[ 79], 90.00th=[ 87], 95.00th=[ 94], | 99.00th=[ 103], 99.50th=[ 108], 99.90th=[ 123], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 344, max= 576, per=24.97%, avg=445.27, stdev=60.51, samples=22 iops : min= 86, max= 144, avg=111.09, stdev=15.24, samples=22 write: IOPS=117, BW=469KiB/s (480kB/s)(5256KiB/11208msec); 0 zone resets slat (usec): min=37, max=3152, avg=200.37, stdev=230.85 clat (msec): min=19, max=114, avg=68.36, stdev=14.65 lat (msec): min=19, max=114, avg=68.56, stdev=14.66 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 45], 10.00th=[ 50], 20.00th=[ 56], | 30.00th=[ 61], 40.00th=[ 65], 50.00th=[ 68], 60.00th=[ 72], | 70.00th=[ 75], 80.00th=[ 81], 90.00th=[ 88], 95.00th=[ 93], | 99.00th=[ 107], 99.50th=[ 110], 99.90th=[ 114], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 365, max= 576, per=25.17%, avg=467.32, stdev=58.38, samples=22 iops : min= 91, max= 144, avg=116.59, stdev=14.69, samples=22 lat (msec) : 20=0.04%, 50=12.34%, 100=85.78%, 250=1.84% cpu : usr=0.34%, sys=3.64%, ctx=4513, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=185671: Tue Aug 18 21:55:52 2026 read: IOPS=112, BW=450KiB/s (461kB/s)(5060KiB/11250msec) slat (usec): min=1178, max=19032, avg=4666.18, stdev=2195.20 clat (msec): min=29, max=119, avg=66.07, stdev=14.93 lat (msec): min=34, max=123, avg=70.74, stdev=15.48 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 43], 10.00th=[ 47], 20.00th=[ 54], | 30.00th=[ 58], 40.00th=[ 62], 50.00th=[ 66], 60.00th=[ 70], | 70.00th=[ 73], 80.00th=[ 79], 90.00th=[ 86], 95.00th=[ 92], | 99.00th=[ 105], 99.50th=[ 109], 99.90th=[ 118], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 326, max= 600, per=25.15%, avg=448.41, stdev=63.81, samples=22 iops : min= 81, max= 150, avg=111.86, stdev=16.04, samples=22 write: IOPS=115, BW=460KiB/s (471kB/s)(5180KiB/11250msec); 0 zone resets slat (usec): min=56, max=7847, avg=186.00, stdev=302.48 clat (msec): min=15, max=112, avg=68.34, stdev=14.69 lat (msec): min=15, max=113, avg=68.53, stdev=14.69 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 45], 10.00th=[ 50], 20.00th=[ 55], | 30.00th=[ 62], 40.00th=[ 65], 50.00th=[ 68], 60.00th=[ 72], | 70.00th=[ 77], 80.00th=[ 81], 90.00th=[ 87], 95.00th=[ 94], | 99.00th=[ 106], 99.50th=[ 109], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 365, max= 632, per=24.66%, avg=457.86, stdev=57.15, samples=22 iops : min= 91, max= 158, avg=114.23, stdev=14.38, samples=22 lat (msec) : 20=0.04%, 50=13.28%, 100=84.61%, 250=2.07% cpu : usr=0.34%, sys=3.66%, ctx=4495, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1784KiB/s (1827kB/s), 442KiB/s-450KiB/s (453kB/s-461kB/s), io=19.6MiB (20.5MB), run=11208-11250msec WRITE: bw=1857KiB/s (1902kB/s), 460KiB/s-469KiB/s (471kB/s-480kB/s), io=20.4MiB (21.4MB), run=11208-11250msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=185676: Tue Aug 18 21:55:56 2026 read: IOPS=7, BW=299MiB/s (313MB/s)(40.0MiB/134msec) slat (nsec): min=11269k, max=11269k, avg=11268855.00, stdev= 0.00 clat (nsec): min=117187k, max=117187k, avg=117186913.00, stdev= 0.00 lat (nsec): min=128460k, max=128460k, avg=128460406.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 117], 10.00th=[ 117], 20.00th=[ 117], | 30.00th=[ 117], 40.00th=[ 117], 50.00th=[ 117], 60.00th=[ 117], | 70.00th=[ 117], 80.00th=[ 117], 90.00th=[ 117], 95.00th=[ 117], | 99.00th=[ 117], 99.50th=[ 117], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=7.52%, ctx=8, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=299MiB/s (313MB/s), 299MiB/s-299MiB/s (313MB/s-313MB/s), io=40.0MiB (41.9MB), run=134-134msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout