== sanity test 398c: run fio to test AIO ================= 09:28:19 (1789306099) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.26768 s, 157 MB/s osc.lustre-OST0000-osc-ffff923830262800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146689: Sun Sep 13 09:29:45 2026 write: IOPS=31, BW=126KiB/s (129kB/s)(10.0MiB/80989msec); 0 zone resets slat (usec): min=21, max=3604, avg=155.46, stdev=292.92 clat (msec): min=145, max=856, avg=505.82, stdev=120.71 lat (msec): min=145, max=856, avg=505.97, stdev=120.72 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 305], 10.00th=[ 363], 20.00th=[ 405], | 30.00th=[ 443], 40.00th=[ 481], 50.00th=[ 518], 60.00th=[ 550], | 70.00th=[ 567], 80.00th=[ 609], 90.00th=[ 651], 95.00th=[ 693], | 99.00th=[ 793], 99.50th=[ 810], 99.90th=[ 835], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 39, max= 313, per=24.90%, avg=125.51, stdev=36.16, samples=161 iops : min= 9, max= 78, avg=31.02, stdev= 9.09, samples=161 lat (msec) : 250=3.20%, 500=42.66%, 750=52.42%, 1000=1.72% cpu : usr=0.06%, sys=0.35%, ctx=2436, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146690: Sun Sep 13 09:29:45 2026 write: IOPS=31, BW=126KiB/s (129kB/s)(10.0MiB/81125msec); 0 zone resets slat (usec): min=21, max=9461, avg=158.00, stdev=363.74 clat (msec): min=125, max=858, avg=506.48, stdev=119.81 lat (msec): min=125, max=859, avg=506.64, stdev=119.81 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 313], 10.00th=[ 359], 20.00th=[ 405], | 30.00th=[ 447], 40.00th=[ 481], 50.00th=[ 518], 60.00th=[ 550], | 70.00th=[ 575], 80.00th=[ 600], 90.00th=[ 651], 95.00th=[ 693], | 99.00th=[ 768], 99.50th=[ 793], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 860] bw ( KiB/s): min= 55, max= 299, per=24.90%, avg=125.52, stdev=33.38, samples=161 iops : min= 13, max= 74, avg=31.02, stdev= 8.39, samples=161 lat (msec) : 250=3.12%, 500=41.52%, 750=53.75%, 1000=1.60% cpu : usr=0.06%, sys=0.34%, ctx=2378, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146691: Sun Sep 13 09:29:45 2026 write: IOPS=31, BW=126KiB/s (129kB/s)(10.0MiB/81057msec); 0 zone resets slat (usec): min=24, max=9598, avg=154.18, stdev=359.02 clat (msec): min=186, max=850, avg=505.84, stdev=119.51 lat (msec): min=186, max=851, avg=505.99, stdev=119.53 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 305], 10.00th=[ 355], 20.00th=[ 405], | 30.00th=[ 447], 40.00th=[ 485], 50.00th=[ 518], 60.00th=[ 542], | 70.00th=[ 567], 80.00th=[ 600], 90.00th=[ 659], 95.00th=[ 693], | 99.00th=[ 768], 99.50th=[ 793], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 852] bw ( KiB/s): min= 31, max= 256, per=24.93%, avg=125.66, stdev=33.98, samples=161 iops : min= 7, max= 64, avg=31.06, stdev= 8.57, samples=161 lat (msec) : 250=3.75%, 500=41.60%, 750=53.16%, 1000=1.48% cpu : usr=0.06%, sys=0.36%, ctx=2318, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146692: Sun Sep 13 09:29:45 2026 write: IOPS=31, BW=126KiB/s (129kB/s)(10.0MiB/81123msec); 0 zone resets slat (usec): min=23, max=4594, avg=164.97, stdev=300.78 clat (msec): min=47, max=859, avg=506.35, stdev=120.73 lat (msec): min=47, max=859, avg=506.52, stdev=120.73 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 309], 10.00th=[ 359], 20.00th=[ 405], | 30.00th=[ 443], 40.00th=[ 481], 50.00th=[ 514], 60.00th=[ 550], | 70.00th=[ 575], 80.00th=[ 600], 90.00th=[ 651], 95.00th=[ 701], | 99.00th=[ 785], 99.50th=[ 802], 99.90th=[ 844], 99.95th=[ 852], | 99.99th=[ 860] bw ( KiB/s): min= 56, max= 287, per=24.93%, avg=125.64, stdev=34.61, samples=161 iops : min= 14, max= 71, avg=31.05, stdev= 8.69, samples=161 lat (msec) : 50=0.04%, 100=0.08%, 250=3.44%, 500=42.93%, 750=51.56% lat (msec) : 1000=1.95% cpu : usr=0.05%, sys=0.34%, ctx=3084, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=505KiB/s (517kB/s), 126KiB/s-126KiB/s (129kB/s-129kB/s), io=40.0MiB (41.9MB), run=80989-81125msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146713: Sun Sep 13 09:30:38 2026 read: IOPS=26, BW=105KiB/s (107kB/s)(5048KiB/48168msec) slat (usec): min=33, max=19872, avg=5487.54, stdev=2367.25 clat (msec): min=7, max=536, avg=193.68, stdev=49.32 lat (msec): min=10, max=545, avg=199.18, stdev=49.92 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 125], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 194], 60.00th=[ 205], | 70.00th=[ 215], 80.00th=[ 228], 90.00th=[ 249], 95.00th=[ 266], | 99.00th=[ 317], 99.50th=[ 376], 99.90th=[ 493], 99.95th=[ 535], | 99.99th=[ 535] bw ( KiB/s): min= 31, max= 176, per=25.33%, avg=104.63, stdev=32.46, samples=95 iops : min= 7, max= 44, avg=25.84, stdev= 8.17, samples=95 write: IOPS=26, BW=108KiB/s (110kB/s)(5192KiB/48168msec); 0 zone resets slat (usec): min=32, max=6169, avg=172.80, stdev=323.45 clat (msec): min=67, max=790, avg=399.42, stdev=114.96 lat (msec): min=67, max=790, avg=399.59, stdev=114.97 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 234], 10.00th=[ 262], 20.00th=[ 300], | 30.00th=[ 334], 40.00th=[ 359], 50.00th=[ 393], 60.00th=[ 426], | 70.00th=[ 451], 80.00th=[ 489], 90.00th=[ 558], 95.00th=[ 609], | 99.00th=[ 676], 99.50th=[ 684], 99.90th=[ 785], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 47, max= 184, per=25.00%, avg=107.51, stdev=28.62, samples=95 iops : min= 11, max= 46, avg=26.56, stdev= 7.17, samples=95 lat (msec) : 10=0.08%, 20=0.43%, 50=0.27%, 100=0.35%, 250=47.19% lat (msec) : 500=42.19%, 750=9.34%, 1000=0.16% cpu : usr=0.12%, sys=0.93%, ctx=4951, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146714: Sun Sep 13 09:30:38 2026 read: IOPS=25, BW=102KiB/s (105kB/s)(4976KiB/48550msec) slat (usec): min=1941, max=18812, avg=5743.99, stdev=2506.32 clat (msec): min=4, max=415, avg=192.84, stdev=48.15 lat (msec): min=11, max=421, avg=198.59, stdev=48.53 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 126], 10.00th=[ 140], 20.00th=[ 159], | 30.00th=[ 171], 40.00th=[ 180], 50.00th=[ 190], 60.00th=[ 201], | 70.00th=[ 213], 80.00th=[ 230], 90.00th=[ 251], 95.00th=[ 271], | 99.00th=[ 326], 99.50th=[ 342], 99.90th=[ 414], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 32, max= 184, per=24.76%, avg=102.26, stdev=35.53, samples=96 iops : min= 8, max= 46, avg=25.24, stdev= 8.92, samples=96 write: IOPS=27, BW=108KiB/s (111kB/s)(5264KiB/48550msec); 0 zone resets slat (usec): min=26, max=5140, avg=172.04, stdev=280.12 clat (msec): min=147, max=784, avg=401.41, stdev=111.06 lat (msec): min=147, max=784, avg=401.58, stdev=111.06 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 241], 10.00th=[ 268], 20.00th=[ 305], | 30.00th=[ 338], 40.00th=[ 363], 50.00th=[ 393], 60.00th=[ 422], | 70.00th=[ 456], 80.00th=[ 489], 90.00th=[ 558], 95.00th=[ 609], | 99.00th=[ 684], 99.50th=[ 743], 99.90th=[ 768], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 39, max= 191, per=25.19%, avg=108.32, stdev=29.09, samples=96 iops : min= 9, max= 47, avg=26.75, stdev= 7.26, samples=96 lat (msec) : 10=0.08%, 20=0.12%, 50=0.51%, 100=0.20%, 250=46.17% lat (msec) : 500=43.36%, 750=9.38%, 1000=0.20% cpu : usr=0.11%, sys=0.92%, ctx=5142, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146715: Sun Sep 13 09:30:38 2026 read: IOPS=25, BW=103KiB/s (105kB/s)(4984KiB/48575msec) slat (usec): min=1706, max=20707, avg=5714.72, stdev=2520.31 clat (msec): min=5, max=324, avg=193.11, stdev=42.92 lat (msec): min=9, max=335, avg=198.83, stdev=43.36 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 131], 10.00th=[ 144], 20.00th=[ 159], | 30.00th=[ 171], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 215], 80.00th=[ 230], 90.00th=[ 247], 95.00th=[ 266], | 99.00th=[ 296], 99.50th=[ 305], 99.90th=[ 321], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 31, max= 192, per=24.80%, avg=102.41, stdev=36.15, samples=96 iops : min= 7, max= 48, avg=25.27, stdev= 9.07, samples=96 write: IOPS=27, BW=108KiB/s (111kB/s)(5256KiB/48575msec); 0 zone resets slat (usec): min=32, max=4605, avg=170.95, stdev=297.76 clat (msec): min=114, max=768, avg=401.82, stdev=112.81 lat (msec): min=114, max=768, avg=402.00, stdev=112.81 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 241], 10.00th=[ 264], 20.00th=[ 300], | 30.00th=[ 334], 40.00th=[ 363], 50.00th=[ 393], 60.00th=[ 422], | 70.00th=[ 456], 80.00th=[ 493], 90.00th=[ 550], 95.00th=[ 609], | 99.00th=[ 684], 99.50th=[ 709], 99.90th=[ 760], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 47, max= 200, per=24.92%, avg=107.16, stdev=27.58, samples=96 iops : min= 11, max= 50, avg=26.46, stdev= 6.90, samples=96 lat (msec) : 10=0.12%, 20=0.04%, 50=0.27%, 100=0.08%, 250=47.11% lat (msec) : 500=42.81%, 750=9.49%, 1000=0.08% cpu : usr=0.12%, sys=0.89%, ctx=4569, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146716: Sun Sep 13 09:30:38 2026 read: IOPS=26, BW=104KiB/s (107kB/s)(5060KiB/48462msec) slat (usec): min=1886, max=18640, avg=5692.61, stdev=2488.43 clat (msec): min=7, max=351, avg=194.69, stdev=43.66 lat (msec): min=11, max=358, avg=200.38, stdev=44.12 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 130], 10.00th=[ 144], 20.00th=[ 159], | 30.00th=[ 171], 40.00th=[ 184], 50.00th=[ 194], 60.00th=[ 205], | 70.00th=[ 215], 80.00th=[ 230], 90.00th=[ 249], 95.00th=[ 271], | 99.00th=[ 300], 99.50th=[ 313], 99.90th=[ 347], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 31, max= 191, per=25.26%, avg=104.32, stdev=34.19, samples=96 iops : min= 7, max= 47, avg=25.75, stdev= 8.57, samples=96 write: IOPS=26, BW=107KiB/s (109kB/s)(5180KiB/48462msec); 0 zone resets slat (usec): min=29, max=5910, avg=170.00, stdev=320.64 clat (msec): min=151, max=796, avg=401.97, stdev=112.47 lat (msec): min=151, max=796, avg=402.14, stdev=112.48 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 241], 10.00th=[ 266], 20.00th=[ 296], | 30.00th=[ 334], 40.00th=[ 368], 50.00th=[ 397], 60.00th=[ 422], | 70.00th=[ 451], 80.00th=[ 489], 90.00th=[ 558], 95.00th=[ 609], | 99.00th=[ 693], 99.50th=[ 768], 99.90th=[ 793], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 48, max= 176, per=24.72%, avg=106.29, stdev=25.92, samples=96 iops : min= 12, max= 44, avg=26.25, stdev= 6.50, samples=96 lat (msec) : 10=0.08%, 20=0.04%, 50=0.20%, 100=0.08%, 250=47.77% lat (msec) : 500=42.81%, 750=8.75%, 1000=0.27% cpu : usr=0.13%, sys=0.91%, ctx=5056, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=413KiB/s (423kB/s), 102KiB/s-105KiB/s (105kB/s-107kB/s), io=19.6MiB (20.5MB), run=48168-48575msec WRITE: bw=430KiB/s (440kB/s), 107KiB/s-108KiB/s (109kB/s-111kB/s), io=20.4MiB (21.4MB), run=48168-48575msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146727: Sun Sep 13 09:30:43 2026 read: IOPS=4, BW=171MiB/s (179MB/s)(40.0MiB/234msec) slat (nsec): min=16911k, max=16911k, avg=16911002.00, stdev= 0.00 clat (nsec): min=208377k, max=208377k, avg=208377040.00, stdev= 0.00 lat (nsec): min=225316k, max=225316k, avg=225316164.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 209], 10.00th=[ 209], 20.00th=[ 209], | 30.00th=[ 209], 40.00th=[ 209], 50.00th=[ 209], 60.00th=[ 209], | 70.00th=[ 209], 80.00th=[ 209], 90.00th=[ 209], 95.00th=[ 209], | 99.00th=[ 209], 99.50th=[ 209], 99.90th=[ 209], 99.95th=[ 209], | 99.99th=[ 209] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.72%, ctx=36, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=171MiB/s (179MB/s), 171MiB/s-171MiB/s (179MB/s-179MB/s), io=40.0MiB (41.9MB), run=234-234msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout