== sanity test 398c: run fio to test AIO ================= 22:16:02 (1763349362) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.509867 s, 82.3 MB/s osc.lustre-OST0000-osc-ffff9710001c8800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=148071: Sun Nov 16 22:18:44 2025 write: IOPS=16, BW=66.3KiB/s (67.9kB/s)(10.0MiB/154333msec); 0 zone resets slat (usec): min=27, max=7659, avg=233.30, stdev=567.76 clat (msec): min=119, max=1650, avg=963.62, stdev=236.68 lat (msec): min=119, max=1650, avg=963.86, stdev=236.70 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 575], 10.00th=[ 693], 20.00th=[ 810], | 30.00th=[ 869], 40.00th=[ 919], 50.00th=[ 978], 60.00th=[ 1028], | 70.00th=[ 1070], 80.00th=[ 1133], 90.00th=[ 1250], 95.00th=[ 1334], | 99.00th=[ 1485], 99.50th=[ 1519], 99.90th=[ 1586], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 7, max= 254, per=24.71%, avg=65.48, stdev=26.09, samples=307 iops : min= 1, max= 63, avg=15.84, stdev= 6.57, samples=307 lat (msec) : 250=1.91%, 500=1.95%, 750=10.94%, 1000=40.51%, 2000=44.69% cpu : usr=0.05%, sys=0.31%, ctx=1858, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148072: Sun Nov 16 22:18:44 2025 write: IOPS=16, BW=66.4KiB/s (68.0kB/s)(10.0MiB/154171msec); 0 zone resets slat (usec): min=27, max=11594, avg=217.31, stdev=576.65 clat (msec): min=89, max=1639, avg=962.86, stdev=241.31 lat (msec): min=90, max=1639, avg=963.08, stdev=241.31 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 567], 10.00th=[ 693], 20.00th=[ 802], | 30.00th=[ 869], 40.00th=[ 919], 50.00th=[ 978], 60.00th=[ 1028], | 70.00th=[ 1083], 80.00th=[ 1150], 90.00th=[ 1250], 95.00th=[ 1351], | 99.00th=[ 1502], 99.50th=[ 1536], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1636] bw ( KiB/s): min= 7, max= 269, per=24.78%, avg=65.66, stdev=31.02, samples=306 iops : min= 1, max= 67, avg=15.89, stdev= 7.79, samples=306 lat (msec) : 100=0.04%, 250=1.80%, 500=2.27%, 750=10.27%, 1000=40.86% lat (msec) : 2000=44.77% cpu : usr=0.04%, sys=0.30%, ctx=1682, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148073: Sun Nov 16 22:18:44 2025 write: IOPS=16, BW=66.5KiB/s (68.1kB/s)(10.0MiB/153986msec); 0 zone resets slat (usec): min=28, max=15062, avg=227.68, stdev=592.24 clat (msec): min=182, max=1638, avg=961.28, stdev=242.49 lat (msec): min=182, max=1639, avg=961.51, stdev=242.51 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 567], 10.00th=[ 684], 20.00th=[ 802], | 30.00th=[ 869], 40.00th=[ 927], 50.00th=[ 978], 60.00th=[ 1028], | 70.00th=[ 1070], 80.00th=[ 1133], 90.00th=[ 1250], 95.00th=[ 1351], | 99.00th=[ 1502], 99.50th=[ 1586], 99.90th=[ 1620], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 7, max= 254, per=24.78%, avg=65.67, stdev=31.17, samples=306 iops : min= 1, max= 63, avg=15.90, stdev= 7.80, samples=306 lat (msec) : 250=1.88%, 500=2.30%, 750=10.94%, 1000=40.08%, 2000=44.80% cpu : usr=0.05%, sys=0.30%, ctx=1569, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148074: Sun Nov 16 22:18:44 2025 write: IOPS=16, BW=66.4KiB/s (67.9kB/s)(10.0MiB/154327msec); 0 zone resets slat (usec): min=27, max=7861, avg=196.11, stdev=473.74 clat (msec): min=187, max=1636, avg=963.61, stdev=233.55 lat (msec): min=187, max=1637, avg=963.81, stdev=233.54 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 584], 10.00th=[ 693], 20.00th=[ 810], | 30.00th=[ 877], 40.00th=[ 936], 50.00th=[ 978], 60.00th=[ 1028], | 70.00th=[ 1062], 80.00th=[ 1133], 90.00th=[ 1234], 95.00th=[ 1334], | 99.00th=[ 1485], 99.50th=[ 1519], 99.90th=[ 1586], 99.95th=[ 1620], | 99.99th=[ 1636] bw ( KiB/s): min= 7, max= 254, per=25.55%, avg=67.70, stdev=38.62, samples=297 iops : min= 1, max= 63, avg=16.40, stdev= 9.66, samples=297 lat (msec) : 250=1.88%, 500=2.15%, 750=10.39%, 1000=40.47%, 2000=45.12% cpu : usr=0.06%, sys=0.26%, ctx=1543, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=265KiB/s (272kB/s), 66.3KiB/s-66.5KiB/s (67.9kB/s-68.1kB/s), io=40.0MiB (41.9MB), run=153986-154333msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148106: Sun Nov 16 22:20:55 2025 read: IOPS=10, BW=40.8KiB/s (41.8kB/s)(5048KiB/123630msec) slat (usec): min=3442, max=41522, avg=14267.62, stdev=5980.23 clat (msec): min=23, max=950, avg=488.30, stdev=110.79 lat (msec): min=35, max=977, avg=502.57, stdev=111.38 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 338], 10.00th=[ 368], 20.00th=[ 401], | 30.00th=[ 426], 40.00th=[ 456], 50.00th=[ 481], 60.00th=[ 514], | 70.00th=[ 542], 80.00th=[ 575], 90.00th=[ 617], 95.00th=[ 676], | 99.00th=[ 810], 99.50th=[ 835], 99.90th=[ 927], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 7, max= 104, per=25.17%, avg=40.78, stdev=18.59, samples=242 iops : min= 1, max= 26, avg= 9.68, stdev= 4.69, samples=242 write: IOPS=10, BW=41.0KiB/s (43.0kB/s)(5192KiB/123630msec); 0 zone resets slat (usec): min=30, max=8486, avg=283.56, stdev=693.01 clat (msec): min=465, max=1898, avg=1031.32, stdev=218.39 lat (msec): min=465, max=1898, avg=1031.60, stdev=218.43 clat percentiles (msec): | 1.00th=[ 558], 5.00th=[ 684], 10.00th=[ 768], 20.00th=[ 860], | 30.00th=[ 919], 40.00th=[ 969], 50.00th=[ 1020], 60.00th=[ 1070], | 70.00th=[ 1133], 80.00th=[ 1200], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1620], 99.50th=[ 1720], 99.90th=[ 1854], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 88, per=24.86%, avg=41.77, stdev=18.20, samples=243 iops : min= 1, max= 22, avg= 9.93, stdev= 4.57, samples=243 lat (msec) : 50=0.08%, 100=0.23%, 250=0.23%, 500=26.76%, 750=25.82% lat (msec) : 1000=19.69%, 2000=27.19% cpu : usr=0.10%, sys=0.60%, ctx=4362, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148107: Sun Nov 16 22:20:55 2025 read: IOPS=10, BW=40.2KiB/s (41.2kB/s)(4976KiB/123671msec) slat (usec): min=2925, max=44794, avg=14377.30, stdev=5860.49 clat (msec): min=25, max=889, avg=487.30, stdev=108.26 lat (msec): min=40, max=907, avg=501.69, stdev=108.86 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 326], 10.00th=[ 355], 20.00th=[ 401], | 30.00th=[ 430], 40.00th=[ 460], 50.00th=[ 489], 60.00th=[ 518], | 70.00th=[ 542], 80.00th=[ 575], 90.00th=[ 609], 95.00th=[ 659], | 99.00th=[ 768], 99.50th=[ 827], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 7, max= 112, per=25.08%, avg=40.64, stdev=20.12, samples=239 iops : min= 1, max= 28, avg= 9.64, stdev= 5.08, samples=239 write: IOPS=10, BW=42.6KiB/s (43.6kB/s)(5264KiB/123671msec); 0 zone resets slat (usec): min=29, max=12390, avg=256.41, stdev=619.17 clat (msec): min=276, max=1842, avg=1026.65, stdev=227.65 lat (msec): min=276, max=1842, avg=1026.91, stdev=227.65 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 651], 10.00th=[ 735], 20.00th=[ 852], | 30.00th=[ 919], 40.00th=[ 969], 50.00th=[ 1020], 60.00th=[ 1070], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1334], 95.00th=[ 1401], | 99.00th=[ 1586], 99.50th=[ 1720], 99.90th=[ 1821], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 103, per=25.29%, avg=42.49, stdev=19.01, samples=242 iops : min= 1, max= 25, avg=10.11, stdev= 4.76, samples=242 lat (msec) : 50=0.08%, 100=0.20%, 250=0.27%, 500=26.21%, 750=26.95% lat (msec) : 1000=18.91%, 2000=27.38% cpu : usr=0.09%, sys=0.61%, ctx=4493, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148108: Sun Nov 16 22:20:55 2025 read: IOPS=10, BW=40.3KiB/s (41.3kB/s)(4984KiB/123628msec) slat (usec): min=3080, max=85750, avg=14592.56, stdev=6625.15 clat (msec): min=14, max=937, avg=486.55, stdev=109.05 lat (msec): min=18, max=952, avg=501.15, stdev=109.74 clat percentiles (msec): | 1.00th=[ 259], 5.00th=[ 326], 10.00th=[ 359], 20.00th=[ 401], | 30.00th=[ 430], 40.00th=[ 460], 50.00th=[ 485], 60.00th=[ 510], | 70.00th=[ 542], 80.00th=[ 567], 90.00th=[ 617], 95.00th=[ 667], | 99.00th=[ 802], 99.50th=[ 835], 99.90th=[ 919], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 7, max= 104, per=25.03%, avg=40.55, stdev=19.00, samples=241 iops : min= 1, max= 26, avg= 9.63, stdev= 4.79, samples=241 write: IOPS=10, BW=42.5KiB/s (43.5kB/s)(5256KiB/123628msec); 0 zone resets slat (usec): min=32, max=13715, avg=281.43, stdev=705.67 clat (msec): min=223, max=1886, avg=1027.37, stdev=221.80 lat (msec): min=224, max=1886, avg=1027.66, stdev=221.81 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 659], 10.00th=[ 751], 20.00th=[ 852], | 30.00th=[ 911], 40.00th=[ 969], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1133], 80.00th=[ 1200], 90.00th=[ 1301], 95.00th=[ 1418], | 99.00th=[ 1586], 99.50th=[ 1720], 99.90th=[ 1871], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 6, max= 87, per=24.99%, avg=41.98, stdev=18.52, samples=244 iops : min= 1, max= 21, avg= 9.98, stdev= 4.68, samples=244 lat (msec) : 20=0.08%, 50=0.04%, 100=0.12%, 250=0.16%, 500=27.89% lat (msec) : 750=24.77%, 1000=18.95%, 2000=28.01% cpu : usr=0.11%, sys=0.64%, ctx=4226, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148109: Sun Nov 16 22:20:55 2025 read: IOPS=10, BW=41.2KiB/s (42.1kB/s)(5060KiB/122950msec) slat (usec): min=54, max=74730, avg=14425.37, stdev=6458.39 clat (msec): min=12, max=920, avg=489.43, stdev=110.47 lat (msec): min=23, max=936, avg=503.86, stdev=111.20 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 326], 10.00th=[ 363], 20.00th=[ 405], | 30.00th=[ 435], 40.00th=[ 460], 50.00th=[ 489], 60.00th=[ 518], | 70.00th=[ 550], 80.00th=[ 575], 90.00th=[ 617], 95.00th=[ 667], | 99.00th=[ 768], 99.50th=[ 793], 99.90th=[ 869], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 7, max= 95, per=25.36%, avg=41.08, stdev=17.99, samples=241 iops : min= 1, max= 23, avg= 9.76, stdev= 4.55, samples=241 write: IOPS=10, BW=42.1KiB/s (43.1kB/s)(5180KiB/122950msec); 0 zone resets slat (usec): min=32, max=12438, avg=274.00, stdev=676.72 clat (msec): min=193, max=1841, avg=1025.70, stdev=230.15 lat (msec): min=194, max=1841, avg=1025.98, stdev=230.14 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 651], 10.00th=[ 743], 20.00th=[ 852], | 30.00th=[ 911], 40.00th=[ 969], 50.00th=[ 1028], 60.00th=[ 1070], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1586], 99.50th=[ 1670], 99.90th=[ 1838], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 96, per=24.72%, avg=41.53, stdev=17.02, samples=243 iops : min= 1, max= 24, avg= 9.87, stdev= 4.29, samples=243 lat (msec) : 20=0.04%, 50=0.12%, 100=0.27%, 250=0.31%, 500=26.45% lat (msec) : 750=26.95%, 1000=18.59%, 2000=27.27% cpu : usr=0.08%, sys=0.66%, ctx=4375, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=162KiB/s (166kB/s), 40.2KiB/s-41.2KiB/s (41.2kB/s-42.1kB/s), io=19.6MiB (20.5MB), run=122950-123671msec WRITE: bw=169KiB/s (173kB/s), 41.0KiB/s-42.6KiB/s (43.0kB/s-43.6kB/s), io=20.4MiB (21.4MB), run=122950-123671msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=148131: Sun Nov 16 22:21:04 2025 read: IOPS=1, BW=78.4MiB/s (82.2MB/s)(40.0MiB/510msec) slat (nsec): min=22448k, max=22448k, avg=22448274.00, stdev= 0.00 clat (nsec): min=460969k, max=460969k, avg=460968863.00, stdev= 0.00 lat (nsec): min=483422k, max=483422k, avg=483422431.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 460], 10.00th=[ 460], 20.00th=[ 460], | 30.00th=[ 460], 40.00th=[ 460], 50.00th=[ 460], 60.00th=[ 460], | 70.00th=[ 460], 80.00th=[ 460], 90.00th=[ 460], 95.00th=[ 460], | 99.00th=[ 460], 99.50th=[ 460], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.75%, ctx=25, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=78.4MiB/s (82.2MB/s), 78.4MiB/s-78.4MiB/s (82.2MB/s-82.2MB/s), io=40.0MiB (41.9MB), run=510-510msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout