== sanity test 398c: run fio to test AIO ================= 06:17:00 (1786184220) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.673877 s, 62.2 MB/s osc.lustre-OST0000-osc-ffff8ad8b0e80800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180127: Sat Aug 8 06:17:41 2026 write: IOPS=85, BW=342KiB/s (350kB/s)(10.0MiB/29968msec); 0 zone resets slat (usec): min=35, max=27796, avg=319.44, stdev=1221.35 clat (msec): min=20, max=430, avg=185.68, stdev=59.70 lat (msec): min=21, max=431, avg=186.00, stdev=59.73 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 100], 10.00th=[ 108], 20.00th=[ 136], | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 182], 60.00th=[ 194], | 70.00th=[ 215], 80.00th=[ 234], 90.00th=[ 268], 95.00th=[ 284], | 99.00th=[ 351], 99.50th=[ 401], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 125, max= 654, per=25.08%, avg=342.31, stdev=105.41, samples=59 iops : min= 31, max= 163, avg=85.14, stdev=26.37, samples=59 lat (msec) : 50=0.12%, 100=5.20%, 250=79.77%, 500=14.92% cpu : usr=0.17%, sys=1.18%, ctx=2833, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180128: Sat Aug 8 06:17:41 2026 write: IOPS=85, BW=341KiB/s (350kB/s)(10.0MiB/29992msec); 0 zone resets slat (usec): min=35, max=19017, avg=298.32, stdev=923.09 clat (msec): min=62, max=423, avg=185.73, stdev=57.39 lat (msec): min=63, max=423, avg=186.04, stdev=57.42 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 102], 10.00th=[ 111], 20.00th=[ 136], | 30.00th=[ 153], 40.00th=[ 169], 50.00th=[ 182], 60.00th=[ 197], | 70.00th=[ 213], 80.00th=[ 232], 90.00th=[ 266], 95.00th=[ 284], | 99.00th=[ 326], 99.50th=[ 363], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 422] bw ( KiB/s): min= 125, max= 638, per=24.96%, avg=340.71, stdev=106.29, samples=59 iops : min= 31, max= 159, avg=84.73, stdev=26.59, samples=59 lat (msec) : 100=4.77%, 250=81.41%, 500=13.83% cpu : usr=0.13%, sys=0.99%, ctx=2107, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180129: Sat Aug 8 06:17:41 2026 write: IOPS=86, BW=345KiB/s (353kB/s)(10.0MiB/29691msec); 0 zone resets slat (usec): min=35, max=27619, avg=307.60, stdev=1187.87 clat (msec): min=42, max=421, avg=185.11, stdev=60.49 lat (msec): min=42, max=421, avg=185.42, stdev=60.52 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 95], 10.00th=[ 107], 20.00th=[ 132], | 30.00th=[ 150], 40.00th=[ 167], 50.00th=[ 182], 60.00th=[ 194], | 70.00th=[ 215], 80.00th=[ 236], 90.00th=[ 268], 95.00th=[ 288], | 99.00th=[ 351], 99.50th=[ 405], 99.90th=[ 414], 99.95th=[ 418], | 99.99th=[ 422] bw ( KiB/s): min= 166, max= 662, per=25.12%, avg=342.88, stdev=103.91, samples=59 iops : min= 41, max= 165, avg=85.29, stdev=25.98, samples=59 lat (msec) : 50=0.04%, 100=6.29%, 250=79.53%, 500=14.14% cpu : usr=0.16%, sys=1.31%, ctx=2647, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180130: Sat Aug 8 06:17:41 2026 write: IOPS=85, BW=341KiB/s (350kB/s)(10.0MiB/29986msec); 0 zone resets slat (usec): min=37, max=17667, avg=297.56, stdev=958.22 clat (msec): min=36, max=402, avg=185.41, stdev=58.26 lat (msec): min=36, max=402, avg=185.71, stdev=58.26 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 102], 10.00th=[ 109], 20.00th=[ 133], | 30.00th=[ 155], 40.00th=[ 169], 50.00th=[ 182], 60.00th=[ 194], | 70.00th=[ 215], 80.00th=[ 239], 90.00th=[ 266], 95.00th=[ 288], | 99.00th=[ 330], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 405] bw ( KiB/s): min= 150, max= 710, per=25.30%, avg=345.36, stdev=102.88, samples=58 iops : min= 37, max= 177, avg=85.88, stdev=25.72, samples=58 lat (msec) : 50=0.08%, 100=4.14%, 250=82.50%, 500=13.28% cpu : usr=0.15%, sys=1.02%, ctx=2380, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1366KiB/s (1398kB/s), 341KiB/s-345KiB/s (350kB/s-353kB/s), io=40.0MiB (41.9MB), run=29691-29992msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180145: Sat Aug 8 06:18:25 2026 read: IOPS=38, BW=154KiB/s (158kB/s)(5048KiB/32811msec) slat (usec): min=85, max=81163, avg=14438.42, stdev=9012.01 clat (msec): min=14, max=434, avg=194.13, stdev=61.45 lat (msec): min=30, max=451, avg=208.57, stdev=64.39 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 102], 10.00th=[ 121], 20.00th=[ 144], | 30.00th=[ 161], 40.00th=[ 176], 50.00th=[ 188], 60.00th=[ 205], | 70.00th=[ 222], 80.00th=[ 243], 90.00th=[ 279], 95.00th=[ 305], | 99.00th=[ 368], 99.50th=[ 393], 99.90th=[ 422], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 63, max= 289, per=25.36%, avg=152.65, stdev=49.67, samples=65 iops : min= 15, max= 72, avg=37.66, stdev=12.45, samples=65 write: IOPS=39, BW=158KiB/s (162kB/s)(5192KiB/32811msec); 0 zone resets slat (usec): min=38, max=12146, avg=242.96, stdev=511.08 clat (msec): min=72, max=454, avg=200.82, stdev=60.77 lat (msec): min=72, max=454, avg=201.06, stdev=60.78 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 111], 10.00th=[ 125], 20.00th=[ 148], | 30.00th=[ 167], 40.00th=[ 180], 50.00th=[ 194], 60.00th=[ 211], | 70.00th=[ 230], 80.00th=[ 251], 90.00th=[ 284], 95.00th=[ 305], | 99.00th=[ 372], 99.50th=[ 414], 99.90th=[ 435], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 47, max= 302, per=25.11%, avg=157.42, stdev=45.55, samples=65 iops : min= 11, max= 75, avg=38.83, stdev=11.45, samples=65 lat (msec) : 20=0.04%, 50=0.16%, 100=3.55%, 250=77.77%, 500=18.48% cpu : usr=0.27%, sys=1.84%, ctx=4440, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180146: Sat Aug 8 06:18:25 2026 read: IOPS=37, BW=150KiB/s (153kB/s)(4976KiB/33280msec) slat (usec): min=2494, max=68071, avg=14748.19, stdev=9121.93 clat (msec): min=23, max=442, avg=196.58, stdev=60.96 lat (msec): min=54, max=451, avg=211.33, stdev=63.99 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 110], 10.00th=[ 122], 20.00th=[ 144], | 30.00th=[ 161], 40.00th=[ 176], 50.00th=[ 190], 60.00th=[ 207], | 70.00th=[ 226], 80.00th=[ 247], 90.00th=[ 279], 95.00th=[ 309], | 99.00th=[ 372], 99.50th=[ 384], 99.90th=[ 435], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 55, max= 289, per=24.58%, avg=147.97, stdev=49.19, samples=66 iops : min= 13, max= 72, avg=36.45, stdev=12.37, samples=66 write: IOPS=39, BW=158KiB/s (162kB/s)(5264KiB/33280msec); 0 zone resets slat (usec): min=54, max=29278, avg=318.75, stdev=1134.89 clat (msec): min=43, max=457, avg=201.80, stdev=59.64 lat (msec): min=44, max=457, avg=202.12, stdev=59.63 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 113], 10.00th=[ 129], 20.00th=[ 150], | 30.00th=[ 167], 40.00th=[ 182], 50.00th=[ 197], 60.00th=[ 213], | 70.00th=[ 230], 80.00th=[ 251], 90.00th=[ 284], 95.00th=[ 305], | 99.00th=[ 351], 99.50th=[ 409], 99.90th=[ 443], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 55, max= 257, per=25.11%, avg=157.45, stdev=44.51, samples=66 iops : min= 13, max= 64, avg=38.86, stdev=11.15, samples=66 lat (msec) : 50=0.08%, 100=2.81%, 250=77.93%, 500=19.18% cpu : usr=0.23%, sys=1.86%, ctx=4393, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180147: Sat Aug 8 06:18:25 2026 read: IOPS=37, BW=151KiB/s (154kB/s)(4984KiB/33082msec) slat (usec): min=1907, max=71629, avg=14255.40, stdev=8824.13 clat (msec): min=20, max=509, avg=193.75, stdev=60.28 lat (msec): min=34, max=524, avg=208.01, stdev=63.08 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 105], 10.00th=[ 120], 20.00th=[ 144], | 30.00th=[ 161], 40.00th=[ 176], 50.00th=[ 188], 60.00th=[ 203], | 70.00th=[ 222], 80.00th=[ 245], 90.00th=[ 275], 95.00th=[ 300], | 99.00th=[ 342], 99.50th=[ 388], 99.90th=[ 498], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 63, max= 288, per=25.01%, avg=150.58, stdev=50.99, samples=65 iops : min= 15, max= 72, avg=37.09, stdev=12.80, samples=65 write: IOPS=39, BW=159KiB/s (163kB/s)(5256KiB/33082msec); 0 zone resets slat (usec): min=42, max=12573, avg=296.92, stdev=882.02 clat (msec): min=75, max=564, avg=202.29, stdev=61.55 lat (msec): min=75, max=564, avg=202.59, stdev=61.56 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 111], 10.00th=[ 127], 20.00th=[ 150], | 30.00th=[ 169], 40.00th=[ 182], 50.00th=[ 197], 60.00th=[ 213], | 70.00th=[ 230], 80.00th=[ 249], 90.00th=[ 275], 95.00th=[ 305], | 99.00th=[ 380], 99.50th=[ 426], 99.90th=[ 527], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 55, max= 289, per=25.26%, avg=158.35, stdev=47.97, samples=65 iops : min= 13, max= 72, avg=39.05, stdev=12.01, samples=65 lat (msec) : 50=0.08%, 100=2.70%, 250=78.59%, 500=18.48%, 750=0.16% cpu : usr=0.23%, sys=1.83%, ctx=4395, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180148: Sat Aug 8 06:18:25 2026 read: IOPS=38, BW=152KiB/s (156kB/s)(5060KiB/33281msec) slat (usec): min=1768, max=76092, avg=14415.35, stdev=9184.82 clat (msec): min=34, max=432, avg=195.24, stdev=61.89 lat (msec): min=49, max=447, avg=209.66, stdev=65.10 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 106], 10.00th=[ 120], 20.00th=[ 142], | 30.00th=[ 161], 40.00th=[ 174], 50.00th=[ 190], 60.00th=[ 207], | 70.00th=[ 224], 80.00th=[ 249], 90.00th=[ 279], 95.00th=[ 305], | 99.00th=[ 363], 99.50th=[ 393], 99.90th=[ 422], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 63, max= 320, per=25.06%, avg=150.85, stdev=49.87, samples=66 iops : min= 15, max= 80, avg=37.20, stdev=12.53, samples=66 write: IOPS=38, BW=156KiB/s (159kB/s)(5180KiB/33281msec); 0 zone resets slat (usec): min=49, max=15342, avg=288.00, stdev=821.96 clat (msec): min=49, max=463, avg=203.16, stdev=61.49 lat (msec): min=49, max=464, avg=203.45, stdev=61.51 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 111], 10.00th=[ 127], 20.00th=[ 150], | 30.00th=[ 171], 40.00th=[ 184], 50.00th=[ 197], 60.00th=[ 213], | 70.00th=[ 230], 80.00th=[ 253], 90.00th=[ 284], 95.00th=[ 313], | 99.00th=[ 359], 99.50th=[ 414], 99.90th=[ 460], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 71, max= 302, per=24.66%, avg=154.59, stdev=43.67, samples=66 iops : min= 17, max= 75, avg=38.14, stdev=10.98, samples=66 lat (msec) : 50=0.23%, 100=2.93%, 250=76.72%, 500=20.12% cpu : usr=0.20%, sys=1.88%, ctx=4256, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=603KiB/s (617kB/s), 150KiB/s-154KiB/s (153kB/s-158kB/s), io=19.6MiB (20.5MB), run=32811-33281msec WRITE: bw=628KiB/s (643kB/s), 156KiB/s-159KiB/s (159kB/s-163kB/s), io=20.4MiB (21.4MB), run=32811-33281msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180159: Sat Aug 8 06:18:37 2026 read: IOPS=1, BW=61.5MiB/s (64.5MB/s)(40.0MiB/650msec) slat (nsec): min=37283k, max=37283k, avg=37282823.00, stdev= 0.00 clat (nsec): min=581274k, max=581274k, avg=581274471.00, stdev= 0.00 lat (nsec): min=618562k, max=618562k, avg=618561712.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 584], 5.00th=[ 584], 10.00th=[ 584], 20.00th=[ 584], | 30.00th=[ 584], 40.00th=[ 584], 50.00th=[ 584], 60.00th=[ 584], | 70.00th=[ 584], 80.00th=[ 584], 90.00th=[ 584], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.62%, ctx=6, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=61.5MiB/s (64.5MB/s), 61.5MiB/s-61.5MiB/s (64.5MB/s-64.5MB/s), io=40.0MiB (41.9MB), run=650-650msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout