== sanity test 398c: run fio to test AIO ================= 23:41:03 (1764736863) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.206085 s, 204 MB/s osc.lustre-OST0000-osc-ffff9c82917de000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=195464: Tue Dec 2 23:41:27 2025 write: IOPS=139, BW=559KiB/s (572kB/s)(10.0MiB/18331msec); 0 zone resets slat (usec): min=24, max=5352, avg=143.97, stdev=297.18 clat (msec): min=7, max=219, avg=114.29, stdev=24.71 lat (msec): min=8, max=219, avg=114.43, stdev=24.72 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 78], 10.00th=[ 84], 20.00th=[ 96], | 30.00th=[ 105], 40.00th=[ 110], 50.00th=[ 115], 60.00th=[ 121], | 70.00th=[ 126], 80.00th=[ 133], 90.00th=[ 144], 95.00th=[ 153], | 99.00th=[ 174], 99.50th=[ 199], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 220] bw ( KiB/s): min= 383, max= 885, per=25.15%, avg=558.36, stdev=85.74, samples=36 iops : min= 95, max= 221, avg=139.33, stdev=21.44, samples=36 lat (msec) : 10=0.04%, 50=1.25%, 100=24.14%, 250=74.57% cpu : usr=0.17%, sys=1.40%, ctx=2828, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=195465: Tue Dec 2 23:41:27 2025 write: IOPS=139, BW=557KiB/s (570kB/s)(10.0MiB/18390msec); 0 zone resets slat (usec): min=28, max=10983, avg=174.96, stdev=404.39 clat (msec): min=15, max=217, avg=114.44, stdev=23.93 lat (msec): min=15, max=217, avg=114.62, stdev=23.92 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 79], 10.00th=[ 86], 20.00th=[ 95], | 30.00th=[ 104], 40.00th=[ 110], 50.00th=[ 115], 60.00th=[ 121], | 70.00th=[ 127], 80.00th=[ 133], 90.00th=[ 144], 95.00th=[ 153], | 99.00th=[ 174], 99.50th=[ 190], 99.90th=[ 199], 99.95th=[ 201], | 99.99th=[ 218] bw ( KiB/s): min= 391, max= 801, per=25.08%, avg=556.69, stdev=77.57, samples=36 iops : min= 97, max= 200, avg=138.92, stdev=19.41, samples=36 lat (msec) : 20=0.20%, 50=0.86%, 100=24.22%, 250=74.73% cpu : usr=0.27%, sys=1.64%, ctx=3176, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=195466: Tue Dec 2 23:41:27 2025 write: IOPS=139, BW=556KiB/s (570kB/s)(10.0MiB/18409msec); 0 zone resets slat (usec): min=24, max=4866, avg=146.41, stdev=320.36 clat (msec): min=14, max=198, avg=114.59, stdev=23.58 lat (msec): min=14, max=198, avg=114.74, stdev=23.59 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 78], 10.00th=[ 84], 20.00th=[ 95], | 30.00th=[ 105], 40.00th=[ 110], 50.00th=[ 115], 60.00th=[ 122], | 70.00th=[ 127], 80.00th=[ 132], 90.00th=[ 142], 95.00th=[ 153], | 99.00th=[ 178], 99.50th=[ 180], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 431, max= 750, per=24.90%, avg=552.81, stdev=74.26, samples=36 iops : min= 107, max= 187, avg=137.94, stdev=18.62, samples=36 lat (msec) : 20=0.04%, 50=0.43%, 100=23.24%, 250=76.29% cpu : usr=0.23%, sys=1.41%, ctx=2545, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=195467: Tue Dec 2 23:41:27 2025 write: IOPS=138, BW=555KiB/s (569kB/s)(10.0MiB/18444msec); 0 zone resets slat (usec): min=28, max=6078, avg=144.72, stdev=313.35 clat (msec): min=17, max=198, avg=114.81, stdev=23.08 lat (msec): min=18, max=199, avg=114.96, stdev=23.08 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 80], 10.00th=[ 86], 20.00th=[ 95], | 30.00th=[ 104], 40.00th=[ 110], 50.00th=[ 116], 60.00th=[ 121], | 70.00th=[ 127], 80.00th=[ 134], 90.00th=[ 144], 95.00th=[ 153], | 99.00th=[ 171], 99.50th=[ 180], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 416, max= 718, per=24.87%, avg=552.22, stdev=71.06, samples=36 iops : min= 104, max= 179, avg=137.78, stdev=17.74, samples=36 lat (msec) : 20=0.04%, 50=0.16%, 100=25.82%, 250=73.98% cpu : usr=0.22%, sys=1.38%, ctx=2988, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2221KiB/s (2274kB/s), 555KiB/s-559KiB/s (569kB/s-572kB/s), io=40.0MiB (41.9MB), run=18331-18444msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=195479: Tue Dec 2 23:41:49 2025 read: IOPS=72, BW=290KiB/s (297kB/s)(5048KiB/17393msec) slat (usec): min=31, max=29262, avg=7525.60, stdev=3493.39 clat (msec): min=10, max=190, avg=102.68, stdev=26.88 lat (msec): min=15, max=203, avg=110.21, stdev=28.10 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 66], 10.00th=[ 72], 20.00th=[ 81], | 30.00th=[ 86], 40.00th=[ 93], 50.00th=[ 101], 60.00th=[ 108], | 70.00th=[ 115], 80.00th=[ 123], 90.00th=[ 140], 95.00th=[ 155], | 99.00th=[ 178], 99.50th=[ 184], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 199, max= 400, per=25.14%, avg=289.88, stdev=57.65, samples=34 iops : min= 49, max= 100, avg=72.09, stdev=14.54, samples=34 write: IOPS=74, BW=299KiB/s (306kB/s)(5192KiB/17393msec); 0 zone resets slat (usec): min=24, max=3045, avg=143.18, stdev=230.21 clat (msec): min=47, max=195, avg=106.17, stdev=25.66 lat (msec): min=47, max=195, avg=106.32, stdev=25.67 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 71], 10.00th=[ 77], 20.00th=[ 85], | 30.00th=[ 91], 40.00th=[ 97], 50.00th=[ 103], 60.00th=[ 110], | 70.00th=[ 117], 80.00th=[ 127], 90.00th=[ 142], 95.00th=[ 155], | 99.00th=[ 178], 99.50th=[ 182], 99.90th=[ 194], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 174, max= 456, per=24.89%, avg=298.94, stdev=69.79, samples=34 iops : min= 43, max= 114, avg=74.35, stdev=17.45, samples=34 lat (msec) : 20=0.04%, 50=0.51%, 100=47.58%, 250=51.88% cpu : usr=0.26%, sys=2.43%, ctx=4637, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=195480: Tue Dec 2 23:41:49 2025 read: IOPS=72, BW=291KiB/s (297kB/s)(4976KiB/17129msec) slat (usec): min=32, max=21631, avg=7326.63, stdev=3303.49 clat (msec): min=18, max=223, avg=100.85, stdev=27.75 lat (msec): min=33, max=235, avg=108.18, stdev=28.86 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 64], 10.00th=[ 69], 20.00th=[ 79], | 30.00th=[ 85], 40.00th=[ 91], 50.00th=[ 97], 60.00th=[ 105], | 70.00th=[ 113], 80.00th=[ 123], 90.00th=[ 138], 95.00th=[ 155], | 99.00th=[ 182], 99.50th=[ 188], 99.90th=[ 218], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 182, max= 400, per=25.25%, avg=291.09, stdev=61.08, samples=33 iops : min= 45, max= 100, avg=72.36, stdev=15.35, samples=33 write: IOPS=76, BW=307KiB/s (315kB/s)(5264KiB/17129msec); 0 zone resets slat (usec): min=28, max=6400, avg=168.30, stdev=364.42 clat (msec): min=18, max=266, avg=105.39, stdev=27.32 lat (msec): min=18, max=266, avg=105.56, stdev=27.34 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 69], 10.00th=[ 77], 20.00th=[ 85], | 30.00th=[ 90], 40.00th=[ 95], 50.00th=[ 103], 60.00th=[ 109], | 70.00th=[ 116], 80.00th=[ 125], 90.00th=[ 144], 95.00th=[ 155], | 99.00th=[ 184], 99.50th=[ 190], 99.90th=[ 236], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 184, max= 431, per=25.66%, avg=308.21, stdev=71.44, samples=33 iops : min= 46, max= 107, avg=76.64, stdev=17.92, samples=33 lat (msec) : 20=0.08%, 50=1.13%, 100=49.49%, 250=49.26%, 500=0.04% cpu : usr=0.41%, sys=2.38%, ctx=4627, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=195481: Tue Dec 2 23:41:49 2025 read: IOPS=71, BW=287KiB/s (294kB/s)(4984KiB/17387msec) slat (usec): min=2102, max=25057, avg=7491.25, stdev=3282.66 clat (msec): min=33, max=195, avg=101.62, stdev=26.60 lat (msec): min=38, max=212, avg=109.12, stdev=27.77 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 67], 10.00th=[ 72], 20.00th=[ 80], | 30.00th=[ 86], 40.00th=[ 92], 50.00th=[ 97], 60.00th=[ 104], | 70.00th=[ 113], 80.00th=[ 122], 90.00th=[ 140], 95.00th=[ 157], | 99.00th=[ 176], 99.50th=[ 180], 99.90th=[ 194], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 159, max= 440, per=24.93%, avg=287.41, stdev=71.79, samples=34 iops : min= 39, max= 110, avg=71.47, stdev=18.04, samples=34 write: IOPS=75, BW=302KiB/s (310kB/s)(5256KiB/17387msec); 0 zone resets slat (usec): min=32, max=3548, avg=144.05, stdev=217.35 clat (msec): min=47, max=198, avg=106.78, stdev=26.39 lat (msec): min=47, max=198, avg=106.92, stdev=26.40 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 71], 10.00th=[ 78], 20.00th=[ 85], | 30.00th=[ 91], 40.00th=[ 97], 50.00th=[ 104], 60.00th=[ 110], | 70.00th=[ 117], 80.00th=[ 127], 90.00th=[ 144], 95.00th=[ 159], | 99.00th=[ 180], 99.50th=[ 186], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 174, max= 416, per=25.13%, avg=301.85, stdev=60.21, samples=34 iops : min= 43, max= 104, avg=75.09, stdev=15.07, samples=34 lat (msec) : 50=0.16%, 100=49.53%, 250=50.31% cpu : usr=0.28%, sys=2.42%, ctx=4761, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=195482: Tue Dec 2 23:41:49 2025 read: IOPS=72, BW=291KiB/s (298kB/s)(5060KiB/17388msec) slat (usec): min=2036, max=84131, avg=7685.60, stdev=4128.88 clat (msec): min=14, max=216, avg=102.37, stdev=27.46 lat (msec): min=21, max=232, avg=110.06, stdev=28.76 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 66], 10.00th=[ 72], 20.00th=[ 80], | 30.00th=[ 86], 40.00th=[ 92], 50.00th=[ 97], 60.00th=[ 105], | 70.00th=[ 113], 80.00th=[ 126], 90.00th=[ 142], 95.00th=[ 157], | 99.00th=[ 180], 99.50th=[ 188], 99.90th=[ 203], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 166, max= 416, per=25.26%, avg=291.29, stdev=61.38, samples=34 iops : min= 41, max= 104, avg=72.44, stdev=15.37, samples=34 write: IOPS=74, BW=298KiB/s (305kB/s)(5180KiB/17388msec); 0 zone resets slat (usec): min=29, max=5888, avg=154.01, stdev=310.33 clat (msec): min=19, max=208, avg=106.04, stdev=26.50 lat (msec): min=19, max=208, avg=106.19, stdev=26.51 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 69], 10.00th=[ 77], 20.00th=[ 84], | 30.00th=[ 89], 40.00th=[ 96], 50.00th=[ 104], 60.00th=[ 110], | 70.00th=[ 118], 80.00th=[ 127], 90.00th=[ 144], 95.00th=[ 157], | 99.00th=[ 176], 99.50th=[ 184], 99.90th=[ 190], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 166, max= 464, per=24.74%, avg=297.09, stdev=71.96, samples=34 iops : min= 41, max= 116, avg=73.88, stdev=18.11, samples=34 lat (msec) : 20=0.08%, 50=0.23%, 100=49.26%, 250=50.43% cpu : usr=0.26%, sys=2.45%, ctx=4595, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1154KiB/s (1181kB/s), 287KiB/s-291KiB/s (294kB/s-298kB/s), io=19.6MiB (20.5MB), run=17129-17393msec WRITE: bw=1201KiB/s (1230kB/s), 298KiB/s-307KiB/s (305kB/s-315kB/s), io=20.4MiB (21.4MB), run=17129-17393msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=195489: Tue Dec 2 23:41:56 2025 read: IOPS=2, BW=118MiB/s (124MB/s)(40.0MiB/338msec) slat (nsec): min=30139k, max=30139k, avg=30138882.00, stdev= 0.00 clat (nsec): min=294550k, max=294550k, avg=294549958.00, stdev= 0.00 lat (nsec): min=324693k, max=324693k, avg=324693352.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 296], 10.00th=[ 296], 20.00th=[ 296], | 30.00th=[ 296], 40.00th=[ 296], 50.00th=[ 296], 60.00th=[ 296], | 70.00th=[ 296], 80.00th=[ 296], 90.00th=[ 296], 95.00th=[ 296], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.86%, ctx=14, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=118MiB/s (124MB/s), 118MiB/s-118MiB/s (124MB/s-124MB/s), io=40.0MiB (41.9MB), run=338-338msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout