== sanity test 398c: run fio to test AIO ================= 22:27:47 (1746152867) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.320587 s, 131 MB/s osc.lustre-OST0000-osc-ffff88009ea79000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=1682: Thu May 1 22:30:32 2025 write: IOPS=15, BW=63.4KiB/s (64.9kB/s)(10.0MiB/161558msec) slat (usec): min=13, max=5999, avg=202.98, stdev=484.74 clat (msec): min=317, max=1684, avg=1008.31, stdev=188.38 lat (msec): min=317, max=1684, avg=1008.51, stdev=188.38 clat percentiles (msec): | 1.00th=[ 558], 5.00th=[ 701], 10.00th=[ 768], 20.00th=[ 852], | 30.00th=[ 919], 40.00th=[ 978], 50.00th=[ 1020], 60.00th=[ 1070], | 70.00th=[ 1099], 80.00th=[ 1150], 90.00th=[ 1234], 95.00th=[ 1284], | 99.00th=[ 1536], 99.50th=[ 1586], 99.90th=[ 1653], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 128, per=25.79%, avg=65.26, stdev=24.63, samples=310 iops : min= 1, max= 32, avg=16.04, stdev= 6.16, samples=310 lat (msec) : 500=0.98%, 750=7.46%, 1000=37.03% cpu : usr=0.07%, sys=0.39%, ctx=1336, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=1683: Thu May 1 22:30:32 2025 write: IOPS=15, BW=63.5KiB/s (65.0kB/s)(10.0MiB/161297msec) slat (usec): min=13, max=8721, avg=233.24, stdev=668.14 clat (msec): min=314, max=1841, avg=1006.52, stdev=192.07 lat (msec): min=314, max=1841, avg=1006.75, stdev=192.09 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 718], 10.00th=[ 760], 20.00th=[ 844], | 30.00th=[ 911], 40.00th=[ 978], 50.00th=[ 1020], 60.00th=[ 1070], | 70.00th=[ 1099], 80.00th=[ 1150], 90.00th=[ 1217], 95.00th=[ 1301], | 99.00th=[ 1552], 99.50th=[ 1569], 99.90th=[ 1770], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 256, per=26.25%, avg=66.41, stdev=32.91, samples=305 iops : min= 1, max= 64, avg=16.35, stdev= 8.22, samples=305 lat (msec) : 500=1.25%, 750=7.30%, 1000=37.23% cpu : usr=0.08%, sys=0.37%, ctx=1257, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=1684: Thu May 1 22:30:32 2025 write: IOPS=15, BW=63.4KiB/s (64.9kB/s)(10.0MiB/161503msec) slat (usec): min=11, max=12122, avg=219.54, stdev=612.43 clat (msec): min=434, max=1728, avg=1008.11, stdev=184.38 lat (msec): min=434, max=1728, avg=1008.34, stdev=184.41 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 709], 10.00th=[ 768], 20.00th=[ 852], | 30.00th=[ 919], 40.00th=[ 978], 50.00th=[ 1020], 60.00th=[ 1062], | 70.00th=[ 1099], 80.00th=[ 1150], 90.00th=[ 1217], 95.00th=[ 1284], | 99.00th=[ 1502], 99.50th=[ 1552], 99.90th=[ 1703], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 8, max= 128, per=25.23%, avg=63.82, stdev=28.27, samples=317 iops : min= 2, max= 32, avg=15.71, stdev= 7.08, samples=317 lat (msec) : 500=1.37%, 750=5.86%, 1000=37.81% cpu : usr=0.08%, sys=0.41%, ctx=1382, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=1685: Thu May 1 22:30:32 2025 write: IOPS=15, BW=63.6KiB/s (65.1kB/s)(10.0MiB/160999msec) slat (usec): min=13, max=9009, avg=227.90, stdev=598.69 clat (msec): min=81, max=1772, avg=1004.98, stdev=200.05 lat (msec): min=81, max=1775, avg=1005.21, stdev=200.07 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 701], 10.00th=[ 768], 20.00th=[ 844], | 30.00th=[ 911], 40.00th=[ 978], 50.00th=[ 1028], 60.00th=[ 1062], | 70.00th=[ 1099], 80.00th=[ 1150], 90.00th=[ 1217], 95.00th=[ 1284], | 99.00th=[ 1536], 99.50th=[ 1670], 99.90th=[ 1770], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 7, max= 128, per=24.95%, avg=63.13, stdev=21.93, samples=321 iops : min= 1, max= 32, avg=15.56, stdev= 5.47, samples=321 lat (msec) : 100=0.04%, 250=0.59%, 500=0.62%, 750=7.38%, 1000=36.29% cpu : usr=0.07%, sys=0.43%, ctx=1474, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=254KiB/s (260kB/s), 63.4KiB/s-63.6KiB/s (64.9kB/s-65.1kB/s), io=40.0MiB (41.9MB), run=160999-161558msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1825: Thu May 1 22:32:18 2025 read: IOPS=12, BW=48.5KiB/s (49.6kB/s)(5048KiB/104124msec) slat (usec): min=2382, max=39203, avg=9241.04, stdev=4364.71 clat (msec): min=16, max=776, avg=420.38, stdev=87.32 lat (msec): min=18, max=786, avg=429.62, stdev=87.97 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 288], 10.00th=[ 317], 20.00th=[ 351], | 30.00th=[ 376], 40.00th=[ 397], 50.00th=[ 414], 60.00th=[ 439], | 70.00th=[ 460], 80.00th=[ 493], 90.00th=[ 531], 95.00th=[ 558], | 99.00th=[ 659], 99.50th=[ 667], 99.90th=[ 768], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 7, max= 111, per=24.91%, avg=47.83, stdev=20.21, samples=208 iops : min= 1, max= 27, avg=11.71, stdev= 5.06, samples=208 write: IOPS=12, BW=49.9KiB/s (51.1kB/s)(5192KiB/104124msec) slat (usec): min=16, max=8046, avg=253.19, stdev=662.82 clat (msec): min=209, max=1575, avg=863.24, stdev=218.44 lat (msec): min=209, max=1575, avg=863.49, stdev=218.43 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 527], 10.00th=[ 592], 20.00th=[ 667], | 30.00th=[ 735], 40.00th=[ 802], 50.00th=[ 852], 60.00th=[ 919], | 70.00th=[ 978], 80.00th=[ 1045], 90.00th=[ 1167], 95.00th=[ 1267], | 99.00th=[ 1368], 99.50th=[ 1418], 99.90th=[ 1569], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 8, max= 120, per=24.77%, avg=49.55, stdev=19.64, samples=207 iops : min= 2, max= 30, avg=12.14, stdev= 4.98, samples=207 lat (msec) : 20=0.04%, 50=0.04%, 100=0.12%, 250=0.59%, 500=42.07% lat (msec) : 750=22.62%, 1000=21.72% cpu : usr=0.23%, sys=1.09%, ctx=2755, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1826: Thu May 1 22:32:18 2025 read: IOPS=11, BW=47.8KiB/s (48.0kB/s)(4976KiB/104061msec) slat (usec): min=2130, max=28641, avg=9065.47, stdev=4193.35 clat (msec): min=11, max=773, avg=419.88, stdev=92.40 lat (msec): min=19, max=777, avg=428.95, stdev=93.22 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 288], 10.00th=[ 313], 20.00th=[ 347], | 30.00th=[ 372], 40.00th=[ 393], 50.00th=[ 414], 60.00th=[ 435], | 70.00th=[ 460], 80.00th=[ 489], 90.00th=[ 542], 95.00th=[ 584], | 99.00th=[ 651], 99.50th=[ 684], 99.90th=[ 760], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 7, max= 120, per=24.60%, avg=47.23, stdev=21.43, samples=207 iops : min= 1, max= 30, avg=11.53, stdev= 5.36, samples=207 write: IOPS=12, BW=50.6KiB/s (51.8kB/s)(5264KiB/104061msec) slat (usec): min=15, max=5780, avg=249.80, stdev=598.90 clat (msec): min=363, max=1585, avg=857.96, stdev=223.22 lat (msec): min=363, max=1586, avg=858.22, stdev=223.25 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 510], 10.00th=[ 558], 20.00th=[ 659], | 30.00th=[ 735], 40.00th=[ 793], 50.00th=[ 852], 60.00th=[ 902], | 70.00th=[ 969], 80.00th=[ 1045], 90.00th=[ 1167], 95.00th=[ 1250], | 99.00th=[ 1385], 99.50th=[ 1502], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 7, max= 112, per=25.11%, avg=50.22, stdev=19.05, samples=206 iops : min= 1, max= 28, avg=12.28, stdev= 4.76, samples=206 lat (msec) : 20=0.04%, 100=0.20%, 250=0.78%, 500=40.94%, 750=23.95% lat (msec) : 1000=20.94% cpu : usr=0.21%, sys=1.11%, ctx=2721, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1827: Thu May 1 22:32:18 2025 read: IOPS=11, BW=47.0KiB/s (49.1kB/s)(4984KiB/103888msec) slat (usec): min=1825, max=26745, avg=9232.54, stdev=4171.22 clat (msec): min=10, max=762, avg=417.66, stdev=90.24 lat (msec): min=13, max=771, avg=426.91, stdev=90.92 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 284], 10.00th=[ 313], 20.00th=[ 347], | 30.00th=[ 372], 40.00th=[ 393], 50.00th=[ 414], 60.00th=[ 435], | 70.00th=[ 456], 80.00th=[ 489], 90.00th=[ 531], 95.00th=[ 575], | 99.00th=[ 642], 99.50th=[ 667], 99.90th=[ 751], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 7, max= 128, per=24.93%, avg=47.87, stdev=21.18, samples=206 iops : min= 1, max= 32, avg=11.65, stdev= 5.29, samples=206 write: IOPS=12, BW=50.6KiB/s (51.8kB/s)(5256KiB/103888msec) slat (usec): min=13, max=10078, avg=273.63, stdev=671.00 clat (msec): min=256, max=1627, avg=857.85, stdev=216.10 lat (msec): min=256, max=1627, avg=858.13, stdev=216.10 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 527], 10.00th=[ 592], 20.00th=[ 659], | 30.00th=[ 726], 40.00th=[ 785], 50.00th=[ 852], 60.00th=[ 911], | 70.00th=[ 961], 80.00th=[ 1036], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1368], 99.50th=[ 1485], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 7, max= 104, per=25.02%, avg=50.04, stdev=18.51, samples=206 iops : min= 1, max= 26, avg=12.19, stdev= 4.69, samples=206 lat (msec) : 20=0.08%, 50=0.04%, 100=0.16%, 250=0.55%, 500=41.17% lat (msec) : 750=23.44%, 1000=22.23% cpu : usr=0.21%, sys=1.10%, ctx=2795, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1828: Thu May 1 22:32:18 2025 read: IOPS=12, BW=48.9KiB/s (50.0kB/s)(5060KiB/103552msec) slat (usec): min=63, max=30215, avg=9206.85, stdev=4405.92 clat (msec): min=9, max=1315, avg=421.88, stdev=102.86 lat (msec): min=9, max=1335, avg=431.09, stdev=103.78 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 284], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 376], 40.00th=[ 397], 50.00th=[ 418], 60.00th=[ 443], | 70.00th=[ 464], 80.00th=[ 489], 90.00th=[ 542], 95.00th=[ 575], | 99.00th=[ 667], 99.50th=[ 709], 99.90th=[ 1083], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 7, max= 96, per=25.57%, avg=49.09, stdev=20.19, samples=204 iops : min= 1, max= 24, avg=12.01, stdev= 5.10, samples=204 write: IOPS=12, BW=50.0KiB/s (51.2kB/s)(5180KiB/103552msec) slat (usec): min=11, max=6227, avg=264.37, stdev=591.39 clat (msec): min=135, max=2277, avg=857.03, stdev=230.30 lat (msec): min=135, max=2277, avg=857.30, stdev=230.31 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 523], 10.00th=[ 584], 20.00th=[ 659], | 30.00th=[ 726], 40.00th=[ 785], 50.00th=[ 852], 60.00th=[ 911], | 70.00th=[ 969], 80.00th=[ 1028], 90.00th=[ 1133], 95.00th=[ 1250], | 99.00th=[ 1418], 99.50th=[ 1653], 99.90th=[ 2198], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 112, per=24.88%, avg=49.75, stdev=20.07, samples=205 iops : min= 1, max= 28, avg=12.18, stdev= 5.06, samples=205 lat (msec) : 10=0.04%, 20=0.12%, 50=0.20%, 100=0.27%, 250=0.59% lat (msec) : 500=40.94%, 750=24.22%, 1000=21.25% cpu : usr=0.17%, sys=1.15%, ctx=2733, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=193KiB/s (197kB/s), 47.8KiB/s-48.9KiB/s (48.0kB/s-50.0kB/s), io=19.6MiB (20.5MB), run=103552-104124msec WRITE: bw=201KiB/s (205kB/s), 49.9KiB/s-50.6KiB/s (51.1kB/s-51.8kB/s), io=20.4MiB (21.4MB), run=103552-104124msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=1916: Thu May 1 22:32:21 2025 read: IOPS=2, BW=96.2MiB/s (101MB/s)(40.0MiB/416msec) slat (nsec): min=5649.0k, max=5649.0k, avg=5649008.00, stdev= 0.00 clat (nsec): min=393648k, max=393648k, avg=393648085.00, stdev= 0.00 lat (nsec): min=399323k, max=399323k, avg=399323042.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 393], 10.00th=[ 393], 20.00th=[ 393], | 30.00th=[ 393], 40.00th=[ 393], 50.00th=[ 393], 60.00th=[ 393], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] lat (msec) : 500=100.00% cpu : usr=0.24%, sys=0.72%, ctx=6, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=96.2MiB/s (101MB/s), 96.2MiB/s-96.2MiB/s (101MB/s-101MB/s), io=40.0MiB (41.9MB), run=416-416msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout