== sanity test 398c: run fio to test AIO ================= 02:12:58 (1761286378) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.290707 s, 144 MB/s osc.lustre-OST0000-osc-ffff9a0f43f83000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=158758: Fri Oct 24 02:15:51 2025 write: IOPS=15, BW=61.5KiB/s (62.0kB/s)(10.0MiB/166539msec); 0 zone resets slat (usec): min=24, max=10481, avg=205.24, stdev=568.11 clat (msec): min=187, max=1712, avg=1039.61, stdev=238.49 lat (msec): min=187, max=1715, avg=1039.82, stdev=238.49 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 659], 10.00th=[ 760], 20.00th=[ 860], | 30.00th=[ 927], 40.00th=[ 986], 50.00th=[ 1036], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1234], 90.00th=[ 1334], 95.00th=[ 1435], | 99.00th=[ 1586], 99.50th=[ 1636], 99.90th=[ 1703], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min= 15, max= 175, per=24.88%, avg=60.96, stdev=23.05, samples=330 iops : min= 3, max= 43, avg=14.76, stdev= 5.76, samples=330 lat (msec) : 250=0.62%, 500=1.29%, 750=7.81%, 1000=33.24%, 2000=57.03% cpu : usr=0.05%, sys=0.26%, ctx=1672, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=158759: Fri Oct 24 02:15:51 2025 write: IOPS=15, BW=61.6KiB/s (63.0kB/s)(10.0MiB/166335msec); 0 zone resets slat (usec): min=28, max=11371, avg=233.32, stdev=612.31 clat (msec): min=188, max=1696, avg=1038.98, stdev=240.38 lat (msec): min=188, max=1696, avg=1039.21, stdev=240.40 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 651], 10.00th=[ 751], 20.00th=[ 852], | 30.00th=[ 927], 40.00th=[ 986], 50.00th=[ 1036], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1234], 90.00th=[ 1334], 95.00th=[ 1435], | 99.00th=[ 1603], 99.50th=[ 1653], 99.90th=[ 1687], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 183, per=24.95%, avg=61.12, stdev=27.70, samples=329 iops : min= 1, max= 45, avg=14.80, stdev= 6.93, samples=329 lat (msec) : 250=0.62%, 500=1.25%, 750=7.89%, 1000=32.77%, 2000=57.46% cpu : usr=0.05%, sys=0.27%, ctx=2055, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=158760: Fri Oct 24 02:15:51 2025 write: IOPS=15, BW=61.4KiB/s (62.9kB/s)(10.0MiB/166647msec); 0 zone resets slat (usec): min=28, max=13622, avg=212.41, stdev=645.03 clat (msec): min=93, max=1751, avg=1040.25, stdev=239.65 lat (msec): min=93, max=1751, avg=1040.47, stdev=239.64 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 667], 10.00th=[ 760], 20.00th=[ 860], | 30.00th=[ 927], 40.00th=[ 986], 50.00th=[ 1036], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1234], 90.00th=[ 1334], 95.00th=[ 1435], | 99.00th=[ 1586], 99.50th=[ 1620], 99.90th=[ 1687], 99.95th=[ 1687], | 99.99th=[ 1754] bw ( KiB/s): min= 7, max= 135, per=25.10%, avg=61.50, stdev=31.26, samples=327 iops : min= 1, max= 33, avg=14.90, stdev= 7.81, samples=327 lat (msec) : 100=0.04%, 250=0.66%, 500=1.05%, 750=7.03%, 1000=34.69% lat (msec) : 2000=56.52% cpu : usr=0.04%, sys=0.27%, ctx=1794, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=158761: Fri Oct 24 02:15:51 2025 write: IOPS=15, BW=61.4KiB/s (62.9kB/s)(10.0MiB/166648msec); 0 zone resets slat (usec): min=28, max=11129, avg=210.68, stdev=613.50 clat (msec): min=190, max=1750, avg=1040.38, stdev=234.41 lat (msec): min=190, max=1750, avg=1040.60, stdev=234.41 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 667], 10.00th=[ 768], 20.00th=[ 860], | 30.00th=[ 919], 40.00th=[ 978], 50.00th=[ 1036], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1234], 90.00th=[ 1351], 95.00th=[ 1435], | 99.00th=[ 1569], 99.50th=[ 1603], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1754] bw ( KiB/s): min= 7, max= 128, per=24.88%, avg=60.97, stdev=26.71, samples=330 iops : min= 1, max= 32, avg=14.76, stdev= 6.69, samples=330 lat (msec) : 250=0.16%, 500=1.21%, 750=7.07%, 1000=36.13%, 2000=55.43% cpu : usr=0.05%, sys=0.27%, ctx=1761, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=246KiB/s (252kB/s), 61.4KiB/s-61.6KiB/s (62.9kB/s-63.0kB/s), io=40.0MiB (41.9MB), run=166335-166648msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=158795: Fri Oct 24 02:17:25 2025 read: IOPS=14, BW=58.0KiB/s (59.4kB/s)(5048KiB/87008msec) slat (usec): min=33, max=58689, avg=10323.63, stdev=4553.66 clat (msec): min=13, max=930, avg=340.53, stdev=83.79 lat (msec): min=20, max=946, avg=350.86, stdev=84.63 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 228], 10.00th=[ 249], 20.00th=[ 279], | 30.00th=[ 300], 40.00th=[ 321], 50.00th=[ 342], 60.00th=[ 359], | 70.00th=[ 376], 80.00th=[ 401], 90.00th=[ 435], 95.00th=[ 464], | 99.00th=[ 535], 99.50th=[ 575], 99.90th=[ 885], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 15, max= 151, per=25.16%, avg=57.62, stdev=24.55, samples=172 iops : min= 3, max= 37, avg=13.89, stdev= 6.19, samples=172 write: IOPS=14, BW=59.7KiB/s (61.1kB/s)(5192KiB/87008msec); 0 zone resets slat (usec): min=33, max=10598, avg=248.02, stdev=514.55 clat (msec): min=90, max=1555, avg=730.24, stdev=177.80 lat (msec): min=90, max=1555, avg=730.49, stdev=177.81 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 443], 10.00th=[ 506], 20.00th=[ 584], | 30.00th=[ 634], 40.00th=[ 684], 50.00th=[ 726], 60.00th=[ 768], | 70.00th=[ 818], 80.00th=[ 877], 90.00th=[ 953], 95.00th=[ 1020], | 99.00th=[ 1167], 99.50th=[ 1200], 99.90th=[ 1536], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 15, max= 115, per=24.71%, avg=59.06, stdev=19.30, samples=172 iops : min= 3, max= 28, avg=14.24, stdev= 4.83, samples=172 lat (msec) : 20=0.08%, 50=0.04%, 100=0.59%, 250=4.69%, 500=47.93% lat (msec) : 750=23.83%, 1000=19.77%, 2000=3.09% cpu : usr=0.11%, sys=0.82%, ctx=4798, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=158796: Fri Oct 24 02:17:25 2025 read: IOPS=14, BW=57.1KiB/s (58.4kB/s)(4976KiB/87201msec) slat (usec): min=3178, max=37503, avg=10372.36, stdev=4588.16 clat (msec): min=22, max=733, avg=343.01, stdev=74.28 lat (msec): min=29, max=750, avg=353.39, stdev=74.89 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 230], 10.00th=[ 251], 20.00th=[ 284], | 30.00th=[ 305], 40.00th=[ 326], 50.00th=[ 342], 60.00th=[ 363], | 70.00th=[ 380], 80.00th=[ 401], 90.00th=[ 435], 95.00th=[ 460], | 99.00th=[ 514], 99.50th=[ 575], 99.90th=[ 726], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 7, max= 120, per=24.79%, avg=56.78, stdev=23.20, samples=172 iops : min= 1, max= 30, avg=13.68, stdev= 5.82, samples=172 write: IOPS=15, BW=60.4KiB/s (61.8kB/s)(5264KiB/87201msec); 0 zone resets slat (usec): min=33, max=13038, avg=281.96, stdev=666.51 clat (msec): min=233, max=1213, avg=724.35, stdev=177.22 lat (msec): min=233, max=1213, avg=724.63, stdev=177.21 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 439], 10.00th=[ 493], 20.00th=[ 575], | 30.00th=[ 625], 40.00th=[ 676], 50.00th=[ 726], 60.00th=[ 776], | 70.00th=[ 818], 80.00th=[ 877], 90.00th=[ 953], 95.00th=[ 1011], | 99.00th=[ 1133], 99.50th=[ 1167], 99.90th=[ 1200], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 23, max= 126, per=24.97%, avg=59.67, stdev=19.84, samples=173 iops : min= 5, max= 31, avg=14.40, stdev= 4.96, samples=173 lat (msec) : 50=0.12%, 100=0.16%, 250=4.45%, 500=48.79%, 750=23.48% lat (msec) : 1000=20.08%, 2000=2.93% cpu : usr=0.10%, sys=0.86%, ctx=4950, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=158797: Fri Oct 24 02:17:25 2025 read: IOPS=14, BW=57.1KiB/s (58.4kB/s)(4984KiB/87341msec) slat (usec): min=2923, max=59712, avg=10402.63, stdev=4608.23 clat (msec): min=10, max=602, avg=345.64, stdev=73.34 lat (msec): min=15, max=613, avg=356.05, stdev=74.03 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 236], 10.00th=[ 262], 20.00th=[ 288], | 30.00th=[ 309], 40.00th=[ 326], 50.00th=[ 347], 60.00th=[ 363], | 70.00th=[ 384], 80.00th=[ 401], 90.00th=[ 430], 95.00th=[ 464], | 99.00th=[ 531], 99.50th=[ 567], 99.90th=[ 584], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 8, max= 120, per=24.69%, avg=56.53, stdev=23.80, samples=173 iops : min= 2, max= 30, avg=13.61, stdev= 5.95, samples=173 write: IOPS=15, BW=60.2KiB/s (61.6kB/s)(5256KiB/87341msec); 0 zone resets slat (usec): min=31, max=7727, avg=263.64, stdev=542.66 clat (msec): min=143, max=1218, avg=724.01, stdev=174.40 lat (msec): min=143, max=1218, avg=724.27, stdev=174.40 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 447], 10.00th=[ 493], 20.00th=[ 575], | 30.00th=[ 634], 40.00th=[ 684], 50.00th=[ 726], 60.00th=[ 776], | 70.00th=[ 818], 80.00th=[ 869], 90.00th=[ 961], 95.00th=[ 1020], | 99.00th=[ 1133], 99.50th=[ 1150], 99.90th=[ 1183], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 15, max= 112, per=24.69%, avg=59.00, stdev=20.53, samples=173 iops : min= 3, max= 28, avg=14.24, stdev= 5.16, samples=173 lat (msec) : 20=0.12%, 50=0.12%, 100=0.16%, 250=3.75%, 500=49.30% lat (msec) : 750=24.06%, 1000=19.41%, 2000=3.09% cpu : usr=0.12%, sys=0.82%, ctx=4850, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=158798: Fri Oct 24 02:17:25 2025 read: IOPS=14, BW=58.2KiB/s (59.6kB/s)(5060KiB/86969msec) slat (usec): min=2582, max=60162, avg=10340.88, stdev=4502.25 clat (msec): min=53, max=846, avg=344.61, stdev=74.96 lat (msec): min=62, max=854, avg=354.95, stdev=75.48 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 234], 10.00th=[ 257], 20.00th=[ 284], | 30.00th=[ 309], 40.00th=[ 326], 50.00th=[ 342], 60.00th=[ 359], | 70.00th=[ 380], 80.00th=[ 405], 90.00th=[ 439], 95.00th=[ 464], | 99.00th=[ 523], 99.50th=[ 567], 99.90th=[ 793], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 15, max= 159, per=25.28%, avg=57.89, stdev=23.43, samples=172 iops : min= 3, max= 39, avg=13.95, stdev= 5.89, samples=172 write: IOPS=14, BW=59.6KiB/s (60.0kB/s)(5180KiB/86969msec); 0 zone resets slat (usec): min=32, max=6778, avg=250.29, stdev=431.87 clat (msec): min=129, max=1701, avg=725.79, stdev=182.63 lat (msec): min=130, max=1701, avg=726.05, stdev=182.62 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 443], 10.00th=[ 498], 20.00th=[ 575], | 30.00th=[ 634], 40.00th=[ 676], 50.00th=[ 718], 60.00th=[ 768], | 70.00th=[ 818], 80.00th=[ 869], 90.00th=[ 953], 95.00th=[ 1011], | 99.00th=[ 1167], 99.50th=[ 1368], 99.90th=[ 1653], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 16, max= 119, per=24.64%, avg=58.88, stdev=20.15, samples=172 iops : min= 4, max= 29, avg=14.20, stdev= 5.04, samples=172 lat (msec) : 100=0.35%, 250=3.71%, 500=49.69%, 750=23.87%, 1000=19.30% lat (msec) : 2000=3.09% cpu : usr=0.13%, sys=0.82%, ctx=4785, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=230KiB/s (235kB/s), 57.1KiB/s-58.2KiB/s (58.4kB/s-59.6kB/s), io=19.6MiB (20.5MB), run=86969-87341msec WRITE: bw=239KiB/s (245kB/s), 59.6KiB/s-60.4KiB/s (60.0kB/s-61.8kB/s), io=20.4MiB (21.4MB), run=86969-87341msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=158814: Fri Oct 24 02:17:33 2025 read: IOPS=2, BW=109MiB/s (115MB/s)(40.0MiB/366msec) slat (nsec): min=29486k, max=29486k, avg=29486174.00, stdev= 0.00 clat (nsec): min=311247k, max=311247k, avg=311246717.00, stdev= 0.00 lat (nsec): min=340742k, max=340742k, avg=340741518.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 313], 10.00th=[ 313], 20.00th=[ 313], | 30.00th=[ 313], 40.00th=[ 313], 50.00th=[ 313], 60.00th=[ 313], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 313], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=4.66%, ctx=43, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=109MiB/s (115MB/s), 109MiB/s-109MiB/s (115MB/s-115MB/s), io=40.0MiB (41.9MB), run=366-366msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout