== sanity test 398c: run fio to test AIO ================= 10:51:16 (1782485476) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.392681 s, 107 MB/s osc.lustre-OST0000-osc-ffff8e6a9845c800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=182608: Fri Jun 26 10:51:45 2026 write: IOPS=118, BW=475KiB/s (486kB/s)(10.0MiB/21557msec); 0 zone resets slat (usec): min=38, max=10506, avg=248.44, stdev=622.64 clat (msec): min=12, max=271, avg=133.52, stdev=29.98 lat (msec): min=12, max=271, avg=133.78, stdev=29.98 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 90], 10.00th=[ 97], 20.00th=[ 108], | 30.00th=[ 117], 40.00th=[ 126], 50.00th=[ 133], 60.00th=[ 140], | 70.00th=[ 146], 80.00th=[ 157], 90.00th=[ 171], 95.00th=[ 186], | 99.00th=[ 209], 99.50th=[ 253], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 343, max= 746, per=25.06%, avg=475.86, stdev=78.71, samples=42 iops : min= 85, max= 186, avg=118.62, stdev=19.72, samples=42 lat (msec) : 20=0.08%, 50=0.16%, 100=12.62%, 250=86.56%, 500=0.59% cpu : usr=0.20%, sys=1.51%, ctx=2816, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182609: Fri Jun 26 10:51:45 2026 write: IOPS=118, BW=475KiB/s (486kB/s)(10.0MiB/21568msec); 0 zone resets slat (usec): min=40, max=14514, avg=257.06, stdev=714.20 clat (msec): min=13, max=255, avg=133.57, stdev=29.00 lat (msec): min=16, max=256, avg=133.83, stdev=28.99 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 91], 10.00th=[ 99], 20.00th=[ 109], | 30.00th=[ 118], 40.00th=[ 126], 50.00th=[ 132], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 157], 90.00th=[ 169], 95.00th=[ 184], | 99.00th=[ 203], 99.50th=[ 245], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 257] bw ( KiB/s): min= 319, max= 672, per=24.92%, avg=473.21, stdev=74.54, samples=42 iops : min= 79, max= 168, avg=117.95, stdev=18.69, samples=42 lat (msec) : 20=0.08%, 50=0.16%, 100=11.17%, 250=88.20%, 500=0.39% cpu : usr=0.21%, sys=1.42%, ctx=2784, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182610: Fri Jun 26 10:51:45 2026 write: IOPS=118, BW=475KiB/s (486kB/s)(10.0MiB/21558msec); 0 zone resets slat (usec): min=35, max=10537, avg=232.17, stdev=541.16 clat (msec): min=29, max=248, avg=133.67, stdev=28.00 lat (msec): min=29, max=248, avg=133.91, stdev=28.00 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 92], 10.00th=[ 101], 20.00th=[ 110], | 30.00th=[ 118], 40.00th=[ 125], 50.00th=[ 132], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 157], 90.00th=[ 171], 95.00th=[ 182], | 99.00th=[ 215], 99.50th=[ 224], 99.90th=[ 234], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min= 367, max= 648, per=24.98%, avg=474.43, stdev=75.58, samples=42 iops : min= 91, max= 162, avg=118.29, stdev=18.90, samples=42 lat (msec) : 50=0.08%, 100=9.53%, 250=90.39% cpu : usr=0.16%, sys=1.41%, ctx=2509, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182611: Fri Jun 26 10:51:45 2026 write: IOPS=119, BW=479KiB/s (490kB/s)(10.0MiB/21389msec); 0 zone resets slat (usec): min=37, max=12562, avg=285.23, stdev=757.09 clat (msec): min=17, max=297, avg=133.26, stdev=29.72 lat (msec): min=17, max=298, avg=133.54, stdev=29.69 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 89], 10.00th=[ 97], 20.00th=[ 109], | 30.00th=[ 117], 40.00th=[ 126], 50.00th=[ 133], 60.00th=[ 140], | 70.00th=[ 146], 80.00th=[ 157], 90.00th=[ 171], 95.00th=[ 182], | 99.00th=[ 218], 99.50th=[ 241], 99.90th=[ 284], 99.95th=[ 296], | 99.99th=[ 300] bw ( KiB/s): min= 367, max= 744, per=25.17%, avg=478.00, stdev=80.20, samples=42 iops : min= 91, max= 186, avg=119.17, stdev=20.12, samples=42 lat (msec) : 20=0.04%, 50=0.16%, 100=11.88%, 250=87.62%, 500=0.31% cpu : usr=0.23%, sys=1.60%, ctx=2842, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1899KiB/s (1945kB/s), 475KiB/s-479KiB/s (486kB/s-490kB/s), io=40.0MiB (41.9MB), run=21389-21568msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=182624: Fri Jun 26 10:52:15 2026 read: IOPS=52, BW=211KiB/s (216kB/s)(5048KiB/23879msec) slat (usec): min=2453, max=45474, avg=10746.49, stdev=5987.42 clat (msec): min=24, max=252, avg=140.06, stdev=33.41 lat (msec): min=34, max=273, avg=150.81, stdev=35.08 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 92], 10.00th=[ 102], 20.00th=[ 112], | 30.00th=[ 121], 40.00th=[ 128], 50.00th=[ 136], 60.00th=[ 146], | 70.00th=[ 157], 80.00th=[ 169], 90.00th=[ 186], 95.00th=[ 201], | 99.00th=[ 230], 99.50th=[ 239], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 83, max= 327, per=25.10%, avg=210.81, stdev=47.23, samples=47 iops : min= 20, max= 81, avg=52.32, stdev=11.85, samples=47 write: IOPS=54, BW=217KiB/s (223kB/s)(5192KiB/23879msec); 0 zone resets slat (usec): min=43, max=12724, avg=271.54, stdev=723.90 clat (msec): min=76, max=260, avg=144.67, stdev=32.04 lat (msec): min=76, max=260, avg=144.95, stdev=32.04 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 99], 10.00th=[ 106], 20.00th=[ 116], | 30.00th=[ 125], 40.00th=[ 134], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 161], 80.00th=[ 174], 90.00th=[ 190], 95.00th=[ 203], | 99.00th=[ 230], 99.50th=[ 234], 99.90th=[ 253], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 76, max= 368, per=24.84%, avg=217.11, stdev=52.83, samples=47 iops : min= 19, max= 92, avg=53.91, stdev=13.24, samples=47 lat (msec) : 50=0.12%, 100=7.46%, 250=92.19%, 500=0.23% cpu : usr=0.29%, sys=2.26%, ctx=4634, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182625: Fri Jun 26 10:52:15 2026 read: IOPS=52, BW=209KiB/s (214kB/s)(4976KiB/23799msec) slat (usec): min=2601, max=47986, avg=10560.72, stdev=5833.26 clat (msec): min=25, max=259, avg=139.98, stdev=32.75 lat (msec): min=28, max=270, avg=150.55, stdev=34.16 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 94], 10.00th=[ 103], 20.00th=[ 112], | 30.00th=[ 121], 40.00th=[ 129], 50.00th=[ 136], 60.00th=[ 146], | 70.00th=[ 157], 80.00th=[ 167], 90.00th=[ 186], 95.00th=[ 199], | 99.00th=[ 226], 99.50th=[ 230], 99.90th=[ 236], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 83, max= 312, per=24.85%, avg=208.74, stdev=45.22, samples=47 iops : min= 20, max= 78, avg=51.81, stdev=11.34, samples=47 write: IOPS=55, BW=221KiB/s (226kB/s)(5264KiB/23799msec); 0 zone resets slat (usec): min=40, max=14976, avg=282.02, stdev=701.15 clat (msec): min=24, max=258, avg=144.61, stdev=31.06 lat (msec): min=25, max=258, avg=144.90, stdev=31.09 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 101], 10.00th=[ 107], 20.00th=[ 116], | 30.00th=[ 125], 40.00th=[ 133], 50.00th=[ 142], 60.00th=[ 153], | 70.00th=[ 163], 80.00th=[ 174], 90.00th=[ 186], 95.00th=[ 197], | 99.00th=[ 224], 99.50th=[ 230], 99.90th=[ 257], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 119, max= 336, per=25.32%, avg=221.32, stdev=51.58, samples=47 iops : min= 29, max= 84, avg=54.94, stdev=12.95, samples=47 lat (msec) : 50=0.16%, 100=6.60%, 250=93.05%, 500=0.20% cpu : usr=0.24%, sys=2.45%, ctx=4819, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182626: Fri Jun 26 10:52:15 2026 read: IOPS=52, BW=209KiB/s (214kB/s)(4984KiB/23867msec) slat (usec): min=2824, max=50199, avg=10519.92, stdev=5764.74 clat (msec): min=33, max=268, avg=140.78, stdev=33.25 lat (msec): min=39, max=291, avg=151.31, stdev=34.64 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 93], 10.00th=[ 102], 20.00th=[ 112], | 30.00th=[ 121], 40.00th=[ 129], 50.00th=[ 138], 60.00th=[ 146], | 70.00th=[ 159], 80.00th=[ 169], 90.00th=[ 188], 95.00th=[ 199], | 99.00th=[ 226], 99.50th=[ 241], 99.90th=[ 266], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 90, max= 343, per=24.85%, avg=208.74, stdev=47.78, samples=47 iops : min= 22, max= 85, avg=51.81, stdev=11.97, samples=47 write: IOPS=55, BW=220KiB/s (226kB/s)(5256KiB/23867msec); 0 zone resets slat (usec): min=47, max=14401, avg=293.41, stdev=778.65 clat (msec): min=33, max=247, avg=144.71, stdev=30.86 lat (msec): min=33, max=247, avg=145.01, stdev=30.85 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 99], 10.00th=[ 107], 20.00th=[ 117], | 30.00th=[ 127], 40.00th=[ 136], 50.00th=[ 144], 60.00th=[ 153], | 70.00th=[ 161], 80.00th=[ 171], 90.00th=[ 186], 95.00th=[ 199], | 99.00th=[ 215], 99.50th=[ 230], 99.90th=[ 247], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min= 119, max= 359, per=25.04%, avg=218.81, stdev=52.24, samples=47 iops : min= 29, max= 89, avg=54.32, stdev=13.12, samples=47 lat (msec) : 50=0.23%, 100=7.62%, 250=91.99%, 500=0.16% cpu : usr=0.31%, sys=2.31%, ctx=4515, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182627: Fri Jun 26 10:52:15 2026 read: IOPS=53, BW=213KiB/s (218kB/s)(5060KiB/23765msec) slat (usec): min=94, max=39600, avg=10567.35, stdev=5628.93 clat (msec): min=19, max=259, avg=140.58, stdev=34.10 lat (msec): min=19, max=277, avg=151.15, stdev=35.57 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 92], 10.00th=[ 100], 20.00th=[ 112], | 30.00th=[ 122], 40.00th=[ 129], 50.00th=[ 136], 60.00th=[ 146], | 70.00th=[ 159], 80.00th=[ 171], 90.00th=[ 186], 95.00th=[ 201], | 99.00th=[ 230], 99.50th=[ 239], 99.90th=[ 253], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 141, max= 336, per=25.29%, avg=212.47, stdev=44.07, samples=47 iops : min= 35, max= 84, avg=52.74, stdev=11.05, samples=47 write: IOPS=54, BW=218KiB/s (223kB/s)(5180KiB/23765msec); 0 zone resets slat (usec): min=45, max=6960, avg=275.84, stdev=569.35 clat (msec): min=80, max=260, avg=145.04, stdev=30.18 lat (msec): min=80, max=260, avg=145.32, stdev=30.22 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 102], 10.00th=[ 108], 20.00th=[ 117], | 30.00th=[ 128], 40.00th=[ 134], 50.00th=[ 142], 60.00th=[ 153], | 70.00th=[ 161], 80.00th=[ 169], 90.00th=[ 184], 95.00th=[ 199], | 99.00th=[ 230], 99.50th=[ 236], 99.90th=[ 251], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 104, max= 296, per=24.89%, avg=217.55, stdev=43.14, samples=47 iops : min= 26, max= 74, avg=54.02, stdev=10.79, samples=47 lat (msec) : 20=0.04%, 50=0.16%, 100=7.23%, 250=92.42%, 500=0.16% cpu : usr=0.31%, sys=2.30%, ctx=4727, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=840KiB/s (861kB/s), 209KiB/s-213KiB/s (214kB/s-218kB/s), io=19.6MiB (20.5MB), run=23765-23879msec WRITE: bw=875KiB/s (896kB/s), 217KiB/s-221KiB/s (223kB/s-226kB/s), io=20.4MiB (21.4MB), run=23765-23879msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=182634: Fri Jun 26 10:52:24 2026 read: IOPS=2, BW=101MiB/s (106MB/s)(40.0MiB/396msec) slat (nsec): min=40875k, max=40875k, avg=40875322.00, stdev= 0.00 clat (nsec): min=330121k, max=330121k, avg=330120893.00, stdev= 0.00 lat (nsec): min=371001k, max=371001k, avg=371001392.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 330], | 30.00th=[ 330], 40.00th=[ 330], 50.00th=[ 330], 60.00th=[ 330], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] lat (msec) : 500=100.00% cpu : usr=0.25%, sys=4.56%, ctx=8, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=101MiB/s (106MB/s), 101MiB/s-101MiB/s (106MB/s-106MB/s), io=40.0MiB (41.9MB), run=396-396msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout