== sanity test 398c: run fio to test AIO ================= 06:35:42 (1762169742) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.191691 s, 219 MB/s osc.lustre-OST0000-osc-ffff8acb87379800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=188490: Mon Nov 3 06:35:58 2025 write: IOPS=218, BW=874KiB/s (895kB/s)(10.0MiB/11714msec); 0 zone resets slat (usec): min=30, max=1634, avg=122.62, stdev=134.45 clat (msec): min=40, max=134, avg=72.55, stdev=15.19 lat (msec): min=40, max=134, avg=72.68, stdev=15.19 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 52], 10.00th=[ 56], 20.00th=[ 61], | 30.00th=[ 64], 40.00th=[ 67], 50.00th=[ 70], 60.00th=[ 73], | 70.00th=[ 79], 80.00th=[ 85], 90.00th=[ 94], 95.00th=[ 100], | 99.00th=[ 122], 99.50th=[ 126], 99.90th=[ 134], 99.95th=[ 134], | 99.99th=[ 134] bw ( KiB/s): min= 576, max= 1128, per=25.00%, avg=874.17, stdev=124.62, samples=23 iops : min= 144, max= 282, avg=218.43, stdev=31.17, samples=23 lat (msec) : 50=2.89%, 100=92.42%, 250=4.69% cpu : usr=0.37%, sys=2.12%, ctx=2799, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=188491: Mon Nov 3 06:35:58 2025 write: IOPS=218, BW=875KiB/s (896kB/s)(10.0MiB/11697msec); 0 zone resets slat (usec): min=31, max=34651, avg=142.03, stdev=705.15 clat (msec): min=9, max=133, avg=72.47, stdev=15.43 lat (msec): min=9, max=133, avg=72.61, stdev=15.42 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 52], 10.00th=[ 57], 20.00th=[ 61], | 30.00th=[ 64], 40.00th=[ 66], 50.00th=[ 70], 60.00th=[ 74], | 70.00th=[ 80], 80.00th=[ 85], 90.00th=[ 94], 95.00th=[ 100], | 99.00th=[ 122], 99.50th=[ 128], 99.90th=[ 134], 99.95th=[ 134], | 99.99th=[ 134] bw ( KiB/s): min= 568, max= 1112, per=25.02%, avg=874.61, stdev=121.81, samples=23 iops : min= 142, max= 278, avg=218.52, stdev=30.48, samples=23 lat (msec) : 10=0.04%, 20=0.04%, 50=3.48%, 100=92.11%, 250=4.34% cpu : usr=0.32%, sys=2.22%, ctx=2914, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=188492: Mon Nov 3 06:35:58 2025 write: IOPS=219, BW=878KiB/s (899kB/s)(10.0MiB/11669msec); 0 zone resets slat (usec): min=24, max=5545, avg=134.70, stdev=231.74 clat (msec): min=16, max=133, avg=72.70, stdev=15.66 lat (msec): min=16, max=133, avg=72.84, stdev=15.66 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 52], 10.00th=[ 56], 20.00th=[ 61], | 30.00th=[ 64], 40.00th=[ 67], 50.00th=[ 70], 60.00th=[ 73], | 70.00th=[ 80], 80.00th=[ 86], 90.00th=[ 96], 95.00th=[ 102], | 99.00th=[ 120], 99.50th=[ 127], 99.90th=[ 132], 99.95th=[ 133], | 99.99th=[ 134] bw ( KiB/s): min= 528, max= 1112, per=25.03%, avg=875.04, stdev=134.67, samples=23 iops : min= 132, max= 278, avg=218.65, stdev=33.69, samples=23 lat (msec) : 20=0.08%, 50=3.44%, 100=90.78%, 250=5.70% cpu : usr=0.30%, sys=2.10%, ctx=2960, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=188493: Mon Nov 3 06:35:58 2025 write: IOPS=218, BW=875KiB/s (896kB/s)(10.0MiB/11709msec); 0 zone resets slat (usec): min=24, max=29757, avg=139.51, stdev=607.05 clat (msec): min=12, max=138, avg=72.54, stdev=15.37 lat (msec): min=12, max=138, avg=72.68, stdev=15.36 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 52], 10.00th=[ 57], 20.00th=[ 61], | 30.00th=[ 64], 40.00th=[ 67], 50.00th=[ 70], 60.00th=[ 74], | 70.00th=[ 80], 80.00th=[ 85], 90.00th=[ 93], 95.00th=[ 101], | 99.00th=[ 121], 99.50th=[ 125], 99.90th=[ 134], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 552, max= 1120, per=24.97%, avg=872.83, stdev=125.76, samples=23 iops : min= 138, max= 280, avg=218.09, stdev=31.45, samples=23 lat (msec) : 20=0.08%, 50=3.87%, 100=91.13%, 250=4.92% cpu : usr=0.32%, sys=2.31%, ctx=2923, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3497KiB/s (3581kB/s), 874KiB/s-878KiB/s (895kB/s-899kB/s), io=40.0MiB (41.9MB), run=11669-11714msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=188504: Mon Nov 3 06:36:12 2025 read: IOPS=121, BW=486KiB/s (498kB/s)(5048KiB/10384msec) slat (usec): min=1421, max=25773, avg=4207.58, stdev=1782.43 clat (msec): min=33, max=170, avg=61.52, stdev=14.22 lat (msec): min=38, max=176, avg=65.74, stdev=14.64 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 44], 10.00th=[ 47], 20.00th=[ 51], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 66], 80.00th=[ 71], 90.00th=[ 79], 95.00th=[ 86], | 99.00th=[ 109], 99.50th=[ 114], 99.90th=[ 167], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 360, max= 584, per=25.33%, avg=489.40, stdev=59.20, samples=20 iops : min= 90, max= 146, avg=122.15, stdev=14.77, samples=20 write: IOPS=125, BW=500KiB/s (512kB/s)(5192KiB/10384msec); 0 zone resets slat (usec): min=29, max=4310, avg=122.41, stdev=174.16 clat (msec): min=37, max=154, avg=63.20, stdev=12.20 lat (msec): min=37, max=154, avg=63.32, stdev=12.21 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 48], 10.00th=[ 51], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 68], 80.00th=[ 72], 90.00th=[ 80], 95.00th=[ 87], | 99.00th=[ 100], 99.50th=[ 105], 99.90th=[ 140], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 407, max= 632, per=25.16%, avg=506.05, stdev=54.81, samples=20 iops : min= 101, max= 158, avg=126.30, stdev=13.78, samples=20 lat (msec) : 50=14.02%, 100=84.73%, 250=1.25% cpu : usr=0.42%, sys=3.33%, ctx=4669, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=188505: Mon Nov 3 06:36:12 2025 read: IOPS=119, BW=480KiB/s (491kB/s)(4976KiB/10371msec) slat (usec): min=1377, max=12677, avg=4189.47, stdev=1592.33 clat (msec): min=11, max=146, avg=61.44, stdev=13.64 lat (msec): min=15, max=156, avg=65.63, stdev=14.12 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 44], 10.00th=[ 47], 20.00th=[ 51], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 71], 90.00th=[ 80], 95.00th=[ 87], | 99.00th=[ 103], 99.50th=[ 112], 99.90th=[ 144], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 384, max= 568, per=24.96%, avg=482.25, stdev=50.94, samples=20 iops : min= 96, max= 142, avg=120.35, stdev=12.71, samples=20 write: IOPS=126, BW=508KiB/s (520kB/s)(5264KiB/10371msec); 0 zone resets slat (usec): min=32, max=2502, avg=119.27, stdev=154.94 clat (msec): min=38, max=146, avg=63.19, stdev=12.37 lat (msec): min=38, max=146, avg=63.31, stdev=12.38 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 48], 10.00th=[ 50], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 64], | 70.00th=[ 68], 80.00th=[ 72], 90.00th=[ 81], 95.00th=[ 86], | 99.00th=[ 105], 99.50th=[ 111], 99.90th=[ 122], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 400, max= 624, per=25.47%, avg=512.25, stdev=64.03, samples=20 iops : min= 100, max= 156, avg=127.85, stdev=16.06, samples=20 lat (msec) : 20=0.04%, 50=14.92%, 100=83.71%, 250=1.33% cpu : usr=0.41%, sys=3.33%, ctx=4628, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=188506: Mon Nov 3 06:36:12 2025 read: IOPS=120, BW=480KiB/s (492kB/s)(4984KiB/10381msec) slat (usec): min=1539, max=23191, avg=4245.18, stdev=1728.61 clat (msec): min=13, max=126, avg=61.34, stdev=12.81 lat (msec): min=18, max=134, avg=65.59, stdev=13.27 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 44], 10.00th=[ 47], 20.00th=[ 51], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 61], 60.00th=[ 64], | 70.00th=[ 67], 80.00th=[ 71], 90.00th=[ 79], 95.00th=[ 85], | 99.00th=[ 99], 99.50th=[ 106], 99.90th=[ 114], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 398, max= 632, per=25.08%, avg=484.60, stdev=66.22, samples=20 iops : min= 99, max= 158, avg=120.95, stdev=16.61, samples=20 write: IOPS=126, BW=506KiB/s (518kB/s)(5256KiB/10381msec); 0 zone resets slat (usec): min=31, max=5756, avg=121.83, stdev=214.29 clat (msec): min=18, max=126, avg=63.36, stdev=12.57 lat (msec): min=18, max=126, avg=63.49, stdev=12.58 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 47], 10.00th=[ 51], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 72], 90.00th=[ 81], 95.00th=[ 86], | 99.00th=[ 104], 99.50th=[ 110], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 368, max= 616, per=25.21%, avg=506.90, stdev=62.36, samples=20 iops : min= 92, max= 154, avg=126.55, stdev=15.57, samples=20 lat (msec) : 20=0.12%, 50=13.98%, 100=84.88%, 250=1.02% cpu : usr=0.39%, sys=3.36%, ctx=4525, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=188507: Mon Nov 3 06:36:12 2025 read: IOPS=122, BW=490KiB/s (501kB/s)(5060KiB/10334msec) slat (usec): min=49, max=19540, avg=4246.86, stdev=1773.59 clat (msec): min=8, max=159, avg=61.10, stdev=14.34 lat (msec): min=8, max=166, avg=65.35, stdev=14.82 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 44], 10.00th=[ 47], 20.00th=[ 51], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 59], 60.00th=[ 63], | 70.00th=[ 66], 80.00th=[ 71], 90.00th=[ 78], 95.00th=[ 86], | 99.00th=[ 104], 99.50th=[ 124], 99.90th=[ 155], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 328, max= 606, per=25.54%, avg=493.50, stdev=69.28, samples=20 iops : min= 82, max= 151, avg=123.15, stdev=17.29, samples=20 write: IOPS=125, BW=501KiB/s (513kB/s)(5180KiB/10334msec); 0 zone resets slat (usec): min=35, max=1729, avg=123.95, stdev=152.28 clat (msec): min=20, max=185, avg=63.50, stdev=13.78 lat (msec): min=20, max=185, avg=63.63, stdev=13.78 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 47], 10.00th=[ 50], 20.00th=[ 54], | 30.00th=[ 57], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 68], 80.00th=[ 72], 90.00th=[ 81], 95.00th=[ 88], | 99.00th=[ 105], 99.50th=[ 113], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 368, max= 568, per=25.11%, avg=504.90, stdev=49.01, samples=20 iops : min= 92, max= 142, avg=126.00, stdev=12.33, samples=20 lat (msec) : 10=0.12%, 20=0.12%, 50=13.95%, 100=84.45%, 250=1.37% cpu : usr=0.43%, sys=3.40%, ctx=4700, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1933KiB/s (1979kB/s), 480KiB/s-490KiB/s (491kB/s-501kB/s), io=19.6MiB (20.5MB), run=10334-10384msec WRITE: bw=2012KiB/s (2060kB/s), 500KiB/s-508KiB/s (512kB/s-520kB/s), io=20.4MiB (21.4MB), run=10334-10384msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=188512: Mon Nov 3 06:36:16 2025 read: IOPS=5, BW=237MiB/s (248MB/s)(40.0MiB/169msec) slat (nsec): min=12022k, max=12022k, avg=12021951.00, stdev= 0.00 clat (nsec): min=147025k, max=147025k, avg=147024725.00, stdev= 0.00 lat (nsec): min=159051k, max=159051k, avg=159050563.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 148], 10.00th=[ 148], 20.00th=[ 148], | 30.00th=[ 148], 40.00th=[ 148], 50.00th=[ 148], 60.00th=[ 148], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.55%, ctx=9, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=237MiB/s (248MB/s), 237MiB/s-237MiB/s (248MB/s-248MB/s), io=40.0MiB (41.9MB), run=169-169msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout