== sanity test 398c: run fio to test AIO ================= 02:05:46 (1786341946) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.282737 s, 148 MB/s osc.lustre-OST0000-osc-ffff93abcd986000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145753: Mon Aug 10 02:07:12 2026 write: IOPS=31, BW=127KiB/s (130kB/s)(10.0MiB/80528msec); 0 zone resets slat (usec): min=36, max=6561, avg=252.44, stdev=423.84 clat (msec): min=136, max=869, avg=502.46, stdev=119.22 lat (msec): min=136, max=870, avg=502.72, stdev=119.23 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 326], 10.00th=[ 359], 20.00th=[ 401], | 30.00th=[ 443], 40.00th=[ 468], 50.00th=[ 502], 60.00th=[ 535], | 70.00th=[ 575], 80.00th=[ 609], 90.00th=[ 659], 95.00th=[ 684], | 99.00th=[ 751], 99.50th=[ 776], 99.90th=[ 802], 99.95th=[ 835], | 99.99th=[ 869] bw ( KiB/s): min= 32, max= 344, per=24.90%, avg=126.49, stdev=36.78, samples=160 iops : min= 8, max= 86, avg=31.29, stdev= 9.23, samples=160 lat (msec) : 250=3.16%, 500=46.45%, 750=49.41%, 1000=0.98% cpu : usr=0.07%, sys=0.53%, ctx=3058, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145754: Mon Aug 10 02:07:12 2026 write: IOPS=31, BW=127KiB/s (130kB/s)(10.0MiB/80461msec); 0 zone resets slat (usec): min=36, max=7396, avg=264.44, stdev=474.76 clat (msec): min=93, max=1004, avg=502.35, stdev=120.54 lat (msec): min=95, max=1004, avg=502.62, stdev=120.56 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 313], 10.00th=[ 359], 20.00th=[ 409], | 30.00th=[ 443], 40.00th=[ 472], 50.00th=[ 502], 60.00th=[ 535], | 70.00th=[ 575], 80.00th=[ 609], 90.00th=[ 651], 95.00th=[ 684], | 99.00th=[ 751], 99.50th=[ 785], 99.90th=[ 944], 99.95th=[ 969], | 99.99th=[ 1003] bw ( KiB/s): min= 40, max= 318, per=24.87%, avg=126.35, stdev=37.01, samples=160 iops : min= 10, max= 79, avg=31.26, stdev= 9.27, samples=160 lat (msec) : 100=0.04%, 250=3.28%, 500=45.51%, 750=50.08%, 1000=1.05% lat (msec) : 2000=0.04% cpu : usr=0.07%, sys=0.48%, ctx=3011, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145755: Mon Aug 10 02:07:12 2026 write: IOPS=31, BW=127KiB/s (130kB/s)(10.0MiB/80409msec); 0 zone resets slat (usec): min=37, max=6300, avg=248.55, stdev=433.72 clat (msec): min=94, max=1082, avg=501.79, stdev=121.25 lat (msec): min=95, max=1082, avg=502.04, stdev=121.28 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 313], 10.00th=[ 359], 20.00th=[ 401], | 30.00th=[ 439], 40.00th=[ 468], 50.00th=[ 502], 60.00th=[ 535], | 70.00th=[ 575], 80.00th=[ 609], 90.00th=[ 651], 95.00th=[ 684], | 99.00th=[ 751], 99.50th=[ 776], 99.90th=[ 802], 99.95th=[ 818], | 99.99th=[ 1083] bw ( KiB/s): min= 40, max= 318, per=24.87%, avg=126.33, stdev=36.02, samples=160 iops : min= 10, max= 79, avg=31.26, stdev= 9.02, samples=160 lat (msec) : 100=0.04%, 250=2.58%, 500=46.91%, 750=49.30%, 1000=1.13% lat (msec) : 2000=0.04% cpu : usr=0.07%, sys=0.51%, ctx=2991, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145756: Mon Aug 10 02:07:12 2026 write: IOPS=31, BW=127KiB/s (130kB/s)(10.0MiB/80459msec); 0 zone resets slat (usec): min=37, max=7725, avg=264.44, stdev=477.53 clat (msec): min=99, max=896, avg=502.03, stdev=119.04 lat (msec): min=99, max=896, avg=502.29, stdev=119.04 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 313], 10.00th=[ 359], 20.00th=[ 405], | 30.00th=[ 443], 40.00th=[ 472], 50.00th=[ 502], 60.00th=[ 535], | 70.00th=[ 575], 80.00th=[ 609], 90.00th=[ 651], 95.00th=[ 684], | 99.00th=[ 760], 99.50th=[ 785], 99.90th=[ 810], 99.95th=[ 818], | 99.99th=[ 894] bw ( KiB/s): min= 63, max= 272, per=24.89%, avg=126.42, stdev=34.33, samples=160 iops : min= 15, max= 68, avg=31.27, stdev= 8.61, samples=160 lat (msec) : 100=0.04%, 250=3.12%, 500=46.41%, 750=49.14%, 1000=1.29% cpu : usr=0.08%, sys=0.51%, ctx=3215, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=509KiB/s (521kB/s), 127KiB/s-127KiB/s (130kB/s-130kB/s), io=40.0MiB (41.9MB), run=80409-80528msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145778: Mon Aug 10 02:08:10 2026 read: IOPS=23, BW=94.4KiB/s (96.6kB/s)(5048KiB/53496msec) slat (usec): min=1500, max=28500, avg=7217.19, stdev=3487.19 clat (msec): min=30, max=438, avg=206.49, stdev=55.70 lat (msec): min=37, max=452, avg=213.71, stdev=56.43 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 131], 10.00th=[ 146], 20.00th=[ 163], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 228], 80.00th=[ 247], 90.00th=[ 275], 95.00th=[ 305], | 99.00th=[ 372], 99.50th=[ 409], 99.90th=[ 435], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 24, max= 208, per=25.19%, avg=94.21, stdev=33.94, samples=106 iops : min= 6, max= 52, avg=23.25, stdev= 8.46, samples=106 write: IOPS=24, BW=97.1KiB/s (99.4kB/s)(5192KiB/53496msec); 0 zone resets slat (usec): min=44, max=15447, avg=286.02, stdev=673.73 clat (msec): min=113, max=772, avg=449.63, stdev=118.82 lat (msec): min=113, max=772, avg=449.92, stdev=118.85 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 279], 10.00th=[ 305], 20.00th=[ 342], | 30.00th=[ 372], 40.00th=[ 409], 50.00th=[ 443], 60.00th=[ 472], | 70.00th=[ 510], 80.00th=[ 558], 90.00th=[ 625], 95.00th=[ 651], | 99.00th=[ 735], 99.50th=[ 751], 99.90th=[ 776], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 39, max= 168, per=24.79%, avg=96.70, stdev=26.41, samples=106 iops : min= 9, max= 42, avg=23.89, stdev= 6.65, samples=106 lat (msec) : 50=0.27%, 100=0.55%, 250=40.82%, 500=41.76%, 750=16.29% lat (msec) : 1000=0.31% cpu : usr=0.14%, sys=1.10%, ctx=5247, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145779: Mon Aug 10 02:08:10 2026 read: IOPS=23, BW=92.9KiB/s (95.2kB/s)(4976KiB/53542msec) slat (usec): min=1853, max=28253, avg=7366.70, stdev=3651.15 clat (msec): min=7, max=415, avg=208.12, stdev=52.77 lat (msec): min=12, max=422, avg=215.50, stdev=53.65 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 134], 10.00th=[ 148], 20.00th=[ 165], | 30.00th=[ 182], 40.00th=[ 192], 50.00th=[ 203], 60.00th=[ 218], | 70.00th=[ 230], 80.00th=[ 249], 90.00th=[ 279], 95.00th=[ 296], | 99.00th=[ 359], 99.50th=[ 380], 99.90th=[ 409], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 24, max= 168, per=24.64%, avg=92.17, stdev=32.95, samples=106 iops : min= 6, max= 42, avg=22.75, stdev= 8.21, samples=106 write: IOPS=24, BW=98.3KiB/s (101kB/s)(5264KiB/53542msec); 0 zone resets slat (usec): min=46, max=10319, avg=285.02, stdev=547.86 clat (msec): min=120, max=780, avg=445.58, stdev=118.62 lat (msec): min=120, max=780, avg=445.87, stdev=118.62 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 271], 10.00th=[ 300], 20.00th=[ 347], | 30.00th=[ 372], 40.00th=[ 401], 50.00th=[ 435], 60.00th=[ 468], | 70.00th=[ 510], 80.00th=[ 558], 90.00th=[ 609], 95.00th=[ 642], | 99.00th=[ 735], 99.50th=[ 768], 99.90th=[ 776], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 40, max= 192, per=25.11%, avg=97.92, stdev=28.63, samples=106 iops : min= 10, max= 48, avg=24.18, stdev= 7.19, samples=106 lat (msec) : 10=0.04%, 20=0.08%, 50=0.20%, 100=0.20%, 250=40.23% lat (msec) : 500=42.97%, 750=15.82%, 1000=0.47% cpu : usr=0.12%, sys=1.14%, ctx=5242, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145780: Mon Aug 10 02:08:10 2026 read: IOPS=23, BW=93.1KiB/s (95.4kB/s)(4984KiB/53517msec) slat (usec): min=59, max=29342, avg=7201.52, stdev=3682.30 clat (msec): min=7, max=498, avg=207.80, stdev=56.54 lat (msec): min=15, max=509, avg=215.01, stdev=57.25 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 130], 10.00th=[ 146], 20.00th=[ 163], | 30.00th=[ 180], 40.00th=[ 192], 50.00th=[ 203], 60.00th=[ 215], | 70.00th=[ 230], 80.00th=[ 249], 90.00th=[ 279], 95.00th=[ 305], | 99.00th=[ 363], 99.50th=[ 422], 99.90th=[ 493], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 24, max= 208, per=24.89%, avg=93.08, stdev=34.88, samples=106 iops : min= 6, max= 52, avg=22.97, stdev= 8.69, samples=106 write: IOPS=24, BW=98.2KiB/s (101kB/s)(5256KiB/53517msec); 0 zone resets slat (usec): min=40, max=7351, avg=287.51, stdev=512.08 clat (msec): min=89, max=801, avg=447.04, stdev=118.43 lat (msec): min=89, max=801, avg=447.33, stdev=118.44 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 268], 10.00th=[ 305], 20.00th=[ 351], | 30.00th=[ 376], 40.00th=[ 409], 50.00th=[ 439], 60.00th=[ 472], | 70.00th=[ 510], 80.00th=[ 550], 90.00th=[ 600], 95.00th=[ 651], | 99.00th=[ 726], 99.50th=[ 751], 99.90th=[ 785], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 39, max= 151, per=25.04%, avg=97.67, stdev=25.76, samples=106 iops : min= 9, max= 37, avg=24.12, stdev= 6.48, samples=106 lat (msec) : 10=0.04%, 20=0.08%, 50=0.23%, 100=0.47%, 250=40.43% lat (msec) : 500=41.95%, 750=16.52%, 1000=0.27% cpu : usr=0.13%, sys=1.16%, ctx=5270, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145781: Mon Aug 10 02:08:10 2026 read: IOPS=23, BW=94.7KiB/s (96.9kB/s)(5060KiB/53449msec) slat (usec): min=1663, max=25601, avg=7271.39, stdev=3590.49 clat (msec): min=27, max=449, avg=209.72, stdev=53.22 lat (msec): min=32, max=459, avg=217.00, stdev=54.11 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 138], 10.00th=[ 150], 20.00th=[ 169], | 30.00th=[ 182], 40.00th=[ 192], 50.00th=[ 205], 60.00th=[ 218], | 70.00th=[ 232], 80.00th=[ 251], 90.00th=[ 279], 95.00th=[ 296], | 99.00th=[ 368], 99.50th=[ 380], 99.90th=[ 405], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 15, max= 200, per=25.28%, avg=94.54, stdev=31.68, samples=106 iops : min= 3, max= 50, avg=23.34, stdev= 7.99, samples=106 write: IOPS=24, BW=96.9KiB/s (99.2kB/s)(5180KiB/53449msec); 0 zone resets slat (usec): min=39, max=7243, avg=271.13, stdev=435.71 clat (msec): min=103, max=787, avg=446.79, stdev=118.33 lat (msec): min=103, max=787, avg=447.06, stdev=118.35 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 271], 10.00th=[ 309], 20.00th=[ 351], | 30.00th=[ 376], 40.00th=[ 409], 50.00th=[ 439], 60.00th=[ 468], | 70.00th=[ 506], 80.00th=[ 550], 90.00th=[ 609], 95.00th=[ 651], | 99.00th=[ 743], 99.50th=[ 768], 99.90th=[ 776], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 40, max= 159, per=24.69%, avg=96.27, stdev=25.22, samples=106 iops : min= 10, max= 39, avg=23.77, stdev= 6.29, samples=106 lat (msec) : 50=0.39%, 100=0.23%, 250=40.00%, 500=43.48%, 750=15.47% lat (msec) : 1000=0.43% cpu : usr=0.15%, sys=1.09%, ctx=5258, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=375KiB/s (384kB/s), 92.9KiB/s-94.7KiB/s (95.2kB/s-96.9kB/s), io=19.6MiB (20.5MB), run=53449-53542msec WRITE: bw=390KiB/s (400kB/s), 96.9KiB/s-98.3KiB/s (99.2kB/s-101kB/s), io=20.4MiB (21.4MB), run=53449-53542msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145793: Mon Aug 10 02:08:15 2026 read: IOPS=3, BW=150MiB/s (158MB/s)(40.0MiB/266msec) slat (nsec): min=22693k, max=22693k, avg=22692754.00, stdev= 0.00 clat (nsec): min=223664k, max=223664k, avg=223664413.00, stdev= 0.00 lat (nsec): min=246362k, max=246362k, avg=246361682.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 224], 10.00th=[ 224], 20.00th=[ 224], | 30.00th=[ 224], 40.00th=[ 224], 50.00th=[ 224], 60.00th=[ 224], | 70.00th=[ 224], 80.00th=[ 224], 90.00th=[ 224], 95.00th=[ 224], | 99.00th=[ 224], 99.50th=[ 224], 99.90th=[ 224], 99.95th=[ 224], | 99.99th=[ 224] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.15%, ctx=5, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=150MiB/s (158MB/s), 150MiB/s-150MiB/s (158MB/s-158MB/s), io=40.0MiB (41.9MB), run=266-266msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout