== sanity test 398c: run fio to test AIO ================= 01:26:51 (1786944411) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.431239 s, 97.3 MB/s osc.lustre-OST0000-osc-ffff8ba0df9d6800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=148344: Mon Aug 17 01:29:30 2026 write: IOPS=16, BW=66.7KiB/s (68.3kB/s)(10.0MiB/153545msec); 0 zone resets slat (usec): min=36, max=20334, avg=240.44, stdev=815.64 clat (msec): min=149, max=2711, avg=958.61, stdev=464.40 lat (msec): min=149, max=2711, avg=958.85, stdev=464.41 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 330], 10.00th=[ 472], 20.00th=[ 584], | 30.00th=[ 667], 40.00th=[ 760], 50.00th=[ 844], 60.00th=[ 969], | 70.00th=[ 1133], 80.00th=[ 1318], 90.00th=[ 1620], 95.00th=[ 1921], | 99.00th=[ 2299], 99.50th=[ 2366], 99.90th=[ 2702], 99.95th=[ 2702], | 99.99th=[ 2702] bw ( KiB/s): min= 7, max= 256, per=24.96%, avg=66.39, stdev=38.11, samples=303 iops : min= 1, max= 64, avg=16.14, stdev= 9.59, samples=303 lat (msec) : 250=1.37%, 500=10.59%, 750=27.19%, 1000=22.58%, 2000=34.34% lat (msec) : >=2000=3.95% cpu : usr=0.04%, sys=0.29%, ctx=1866, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148345: Mon Aug 17 01:29:30 2026 write: IOPS=16, BW=66.7KiB/s (68.3kB/s)(10.0MiB/153430msec); 0 zone resets slat (usec): min=36, max=32437, avg=239.26, stdev=870.35 clat (msec): min=124, max=2621, avg=958.35, stdev=472.09 lat (msec): min=124, max=2621, avg=958.59, stdev=472.13 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 351], 10.00th=[ 464], 20.00th=[ 584], | 30.00th=[ 667], 40.00th=[ 751], 50.00th=[ 844], 60.00th=[ 969], | 70.00th=[ 1133], 80.00th=[ 1318], 90.00th=[ 1620], 95.00th=[ 1921], | 99.00th=[ 2333], 99.50th=[ 2400], 99.90th=[ 2567], 99.95th=[ 2635], | 99.99th=[ 2635] bw ( KiB/s): min= 7, max= 312, per=24.93%, avg=66.30, stdev=40.67, samples=303 iops : min= 1, max= 78, avg=16.12, stdev=10.24, samples=303 lat (msec) : 250=1.88%, 500=10.51%, 750=27.54%, 1000=21.99%, 2000=34.14% lat (msec) : >=2000=3.95% cpu : usr=0.03%, sys=0.27%, ctx=1952, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148346: Mon Aug 17 01:29:30 2026 write: IOPS=16, BW=66.7KiB/s (68.3kB/s)(10.0MiB/153431msec); 0 zone resets slat (usec): min=35, max=17035, avg=230.00, stdev=632.65 clat (msec): min=92, max=2622, avg=958.19, stdev=468.67 lat (msec): min=93, max=2622, avg=958.42, stdev=468.65 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 347], 10.00th=[ 464], 20.00th=[ 575], | 30.00th=[ 667], 40.00th=[ 743], 50.00th=[ 844], 60.00th=[ 969], | 70.00th=[ 1116], 80.00th=[ 1318], 90.00th=[ 1620], 95.00th=[ 1921], | 99.00th=[ 2366], 99.50th=[ 2433], 99.90th=[ 2601], 99.95th=[ 2635], | 99.99th=[ 2635] bw ( KiB/s): min= 7, max= 261, per=25.37%, avg=67.48, stdev=40.11, samples=298 iops : min= 1, max= 65, avg=16.42, stdev=10.10, samples=298 lat (msec) : 100=0.04%, 250=1.48%, 500=10.86%, 750=28.05%, 1000=22.07% lat (msec) : 2000=33.63%, >=2000=3.87% cpu : usr=0.04%, sys=0.27%, ctx=2046, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148347: Mon Aug 17 01:29:30 2026 write: IOPS=16, BW=66.6KiB/s (68.2kB/s)(10.0MiB/153640msec); 0 zone resets slat (usec): min=36, max=38142, avg=230.34, stdev=926.59 clat (msec): min=156, max=2690, avg=959.31, stdev=469.01 lat (msec): min=156, max=2691, avg=959.54, stdev=469.04 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 342], 10.00th=[ 481], 20.00th=[ 584], | 30.00th=[ 659], 40.00th=[ 751], 50.00th=[ 844], 60.00th=[ 969], | 70.00th=[ 1133], 80.00th=[ 1318], 90.00th=[ 1636], 95.00th=[ 1921], | 99.00th=[ 2333], 99.50th=[ 2433], 99.90th=[ 2601], 99.95th=[ 2702], | 99.99th=[ 2702] bw ( KiB/s): min= 7, max= 256, per=25.55%, avg=67.96, stdev=41.40, samples=296 iops : min= 1, max= 64, avg=16.53, stdev=10.42, samples=296 lat (msec) : 250=1.52%, 500=10.39%, 750=27.93%, 1000=22.66%, 2000=33.32% lat (msec) : >=2000=4.18% cpu : usr=0.04%, sys=0.29%, ctx=2001, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=267KiB/s (273kB/s), 66.6KiB/s-66.7KiB/s (68.2kB/s-68.3kB/s), io=40.0MiB (41.9MB), run=153430-153640msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148379: Mon Aug 17 01:31:14 2026 read: IOPS=13, BW=52.2KiB/s (53.5kB/s)(5048KiB/96674msec) slat (usec): min=2734, max=73676, avg=13485.27, stdev=7495.95 clat (msec): min=12, max=751, avg=365.49, stdev=91.91 lat (msec): min=16, max=767, avg=378.98, stdev=92.77 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 239], 10.00th=[ 262], 20.00th=[ 292], | 30.00th=[ 317], 40.00th=[ 338], 50.00th=[ 355], 60.00th=[ 376], | 70.00th=[ 401], 80.00th=[ 430], 90.00th=[ 481], 95.00th=[ 542], | 99.00th=[ 625], 99.50th=[ 659], 99.90th=[ 743], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 7, max= 119, per=24.92%, avg=51.58, stdev=23.19, samples=192 iops : min= 1, max= 29, avg=12.40, stdev= 5.80, samples=192 write: IOPS=13, BW=53.7KiB/s (54.0kB/s)(5192KiB/96674msec); 0 zone resets slat (usec): min=42, max=20499, avg=327.78, stdev=919.15 clat (msec): min=239, max=1696, avg=819.72, stdev=228.48 lat (msec): min=239, max=1696, avg=820.05, stdev=228.47 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 477], 10.00th=[ 550], 20.00th=[ 642], | 30.00th=[ 709], 40.00th=[ 751], 50.00th=[ 802], 60.00th=[ 860], | 70.00th=[ 919], 80.00th=[ 995], 90.00th=[ 1099], 95.00th=[ 1234], | 99.00th=[ 1469], 99.50th=[ 1502], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 120, per=24.67%, avg=53.28, stdev=21.92, samples=191 iops : min= 1, max= 30, avg=12.83, stdev= 5.53, samples=191 lat (msec) : 20=0.08%, 50=0.04%, 100=0.04%, 250=3.87%, 500=44.41% lat (msec) : 750=21.13%, 1000=21.13%, 2000=9.30% cpu : usr=0.08%, sys=0.76%, ctx=4512, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148380: Mon Aug 17 01:31:14 2026 read: IOPS=12, BW=51.5KiB/s (52.7kB/s)(4976KiB/96675msec) slat (usec): min=2648, max=78339, avg=13497.94, stdev=7870.34 clat (msec): min=17, max=791, avg=360.73, stdev=94.75 lat (msec): min=29, max=801, avg=374.23, stdev=96.22 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 228], 10.00th=[ 251], 20.00th=[ 292], | 30.00th=[ 313], 40.00th=[ 334], 50.00th=[ 355], 60.00th=[ 380], | 70.00th=[ 405], 80.00th=[ 430], 90.00th=[ 472], 95.00th=[ 518], | 99.00th=[ 642], 99.50th=[ 659], 99.90th=[ 709], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 7, max= 119, per=24.54%, avg=50.81, stdev=24.00, samples=192 iops : min= 1, max= 29, avg=12.20, stdev= 6.03, samples=192 write: IOPS=13, BW=54.5KiB/s (55.8kB/s)(5264KiB/96675msec); 0 zone resets slat (usec): min=44, max=15465, avg=281.96, stdev=683.26 clat (msec): min=226, max=1697, avg=819.16, stdev=236.30 lat (msec): min=226, max=1698, avg=819.45, stdev=236.31 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 443], 10.00th=[ 535], 20.00th=[ 634], | 30.00th=[ 709], 40.00th=[ 751], 50.00th=[ 802], 60.00th=[ 860], | 70.00th=[ 927], 80.00th=[ 1003], 90.00th=[ 1099], 95.00th=[ 1217], | 99.00th=[ 1502], 99.50th=[ 1586], 99.90th=[ 1687], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 120, per=24.87%, avg=53.72, stdev=22.11, samples=192 iops : min= 1, max= 30, avg=12.94, stdev= 5.58, samples=192 lat (msec) : 20=0.04%, 50=0.27%, 100=0.20%, 250=4.30%, 500=44.30% lat (msec) : 750=19.34%, 1000=21.33%, 2000=10.23% cpu : usr=0.11%, sys=0.71%, ctx=4320, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148381: Mon Aug 17 01:31:14 2026 read: IOPS=12, BW=51.7KiB/s (52.9kB/s)(4984KiB/96471msec) slat (usec): min=58, max=72122, avg=13299.83, stdev=7695.93 clat (msec): min=17, max=874, avg=363.24, stdev=95.77 lat (msec): min=23, max=883, avg=376.55, stdev=97.03 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 239], 10.00th=[ 262], 20.00th=[ 288], | 30.00th=[ 309], 40.00th=[ 334], 50.00th=[ 355], 60.00th=[ 380], | 70.00th=[ 405], 80.00th=[ 426], 90.00th=[ 481], 95.00th=[ 527], | 99.00th=[ 625], 99.50th=[ 659], 99.90th=[ 869], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 7, max= 112, per=24.83%, avg=51.39, stdev=22.10, samples=191 iops : min= 1, max= 28, avg=12.36, stdev= 5.57, samples=191 write: IOPS=13, BW=54.5KiB/s (55.8kB/s)(5256KiB/96471msec); 0 zone resets slat (usec): min=43, max=20277, avg=289.49, stdev=886.69 clat (msec): min=227, max=1675, avg=816.38, stdev=239.16 lat (msec): min=227, max=1675, avg=816.68, stdev=239.16 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 451], 10.00th=[ 531], 20.00th=[ 625], | 30.00th=[ 701], 40.00th=[ 751], 50.00th=[ 802], 60.00th=[ 860], | 70.00th=[ 919], 80.00th=[ 995], 90.00th=[ 1099], 95.00th=[ 1234], | 99.00th=[ 1552], 99.50th=[ 1620], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 8, max= 135, per=24.99%, avg=53.97, stdev=22.75, samples=190 iops : min= 2, max= 33, avg=13.01, stdev= 5.74, samples=190 lat (msec) : 20=0.04%, 50=0.12%, 100=0.20%, 250=3.44%, 500=44.65% lat (msec) : 750=20.35%, 1000=21.52%, 2000=9.69% cpu : usr=0.09%, sys=0.71%, ctx=4075, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148382: Mon Aug 17 01:31:14 2026 read: IOPS=13, BW=52.4KiB/s (53.7kB/s)(5060KiB/96474msec) slat (usec): min=3070, max=72924, avg=13317.47, stdev=7649.85 clat (msec): min=15, max=857, avg=360.75, stdev=94.43 lat (msec): min=22, max=883, avg=374.08, stdev=95.70 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 230], 10.00th=[ 253], 20.00th=[ 284], | 30.00th=[ 309], 40.00th=[ 330], 50.00th=[ 351], 60.00th=[ 376], | 70.00th=[ 397], 80.00th=[ 426], 90.00th=[ 477], 95.00th=[ 535], | 99.00th=[ 634], 99.50th=[ 659], 99.90th=[ 709], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 7, max= 143, per=25.33%, avg=52.44, stdev=24.48, samples=190 iops : min= 1, max= 35, avg=12.62, stdev= 6.17, samples=190 write: IOPS=13, BW=53.7KiB/s (54.0kB/s)(5180KiB/96474msec); 0 zone resets slat (usec): min=39, max=12565, avg=284.11, stdev=667.71 clat (msec): min=225, max=1728, avg=823.44, stdev=235.72 lat (msec): min=227, max=1729, avg=823.72, stdev=235.74 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 472], 10.00th=[ 535], 20.00th=[ 634], | 30.00th=[ 718], 40.00th=[ 760], 50.00th=[ 810], 60.00th=[ 869], | 70.00th=[ 919], 80.00th=[ 978], 90.00th=[ 1099], 95.00th=[ 1250], | 99.00th=[ 1552], 99.50th=[ 1603], 99.90th=[ 1720], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 15, max= 128, per=24.53%, avg=52.98, stdev=21.64, samples=191 iops : min= 3, max= 32, avg=12.75, stdev= 5.45, samples=191 lat (msec) : 20=0.04%, 50=0.16%, 100=0.12%, 250=4.26%, 500=44.41% lat (msec) : 750=19.80%, 1000=22.07%, 2000=9.14% cpu : usr=0.09%, sys=0.74%, ctx=4181, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=208KiB/s (213kB/s), 51.5KiB/s-52.4KiB/s (52.7kB/s-53.7kB/s), io=19.6MiB (20.5MB), run=96471-96675msec WRITE: bw=216KiB/s (221kB/s), 53.7KiB/s-54.5KiB/s (54.0kB/s-55.8kB/s), io=20.4MiB (21.4MB), run=96471-96675msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=148406: Mon Aug 17 01:31:24 2026 read: IOPS=2, BW=80.6MiB/s (84.6MB/s)(40.0MiB/496msec) slat (nsec): min=38043k, max=38043k, avg=38043294.00, stdev= 0.00 clat (nsec): min=431105k, max=431105k, avg=431104932.00, stdev= 0.00 lat (nsec): min=469154k, max=469154k, avg=469153543.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 430], 10.00th=[ 430], 20.00th=[ 430], | 30.00th=[ 430], 40.00th=[ 430], 50.00th=[ 430], 60.00th=[ 430], | 70.00th=[ 430], 80.00th=[ 430], 90.00th=[ 430], 95.00th=[ 430], | 99.00th=[ 430], 99.50th=[ 430], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] lat (msec) : 500=100.00% cpu : usr=0.20%, sys=3.23%, ctx=33, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=80.6MiB/s (84.6MB/s), 80.6MiB/s-80.6MiB/s (84.6MB/s-84.6MB/s), io=40.0MiB (41.9MB), run=496-496msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout