== sanity test 398c: run fio to test AIO ================= 23:14:11 (1746155651) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.305311 s, 137 MB/s osc.lustre-OST0000-osc-ffff8800b582e800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=557: Thu May 1 23:16:57 2025 write: IOPS=15, BW=63.1KiB/s (64.7kB/s)(10.0MiB/162181msec) slat (usec): min=12, max=9103, avg=224.96, stdev=638.64 clat (msec): min=413, max=1713, avg=1011.90, stdev=187.62 lat (msec): min=413, max=1716, avg=1012.13, stdev=187.63 clat percentiles (msec): | 1.00th=[ 584], 5.00th=[ 676], 10.00th=[ 768], 20.00th=[ 860], | 30.00th=[ 919], 40.00th=[ 969], 50.00th=[ 1020], 60.00th=[ 1062], | 70.00th=[ 1116], 80.00th=[ 1167], 90.00th=[ 1250], 95.00th=[ 1301], | 99.00th=[ 1418], 99.50th=[ 1502], 99.90th=[ 1653], 99.95th=[ 1670], | 99.99th=[ 1720] bw ( KiB/s): min= 7, max= 128, per=25.23%, avg=63.58, stdev=27.49, samples=318 iops : min= 1, max= 32, avg=15.58, stdev= 6.89, samples=318 lat (msec) : 500=0.62%, 750=7.77%, 1000=37.73% cpu : usr=0.07%, sys=0.40%, ctx=1431, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=558: Thu May 1 23:16:57 2025 write: IOPS=15, BW=63.2KiB/s (64.7kB/s)(10.0MiB/162022msec) slat (usec): min=11, max=10921, avg=217.79, stdev=606.08 clat (msec): min=131, max=1716, avg=1010.87, stdev=192.83 lat (msec): min=131, max=1716, avg=1011.09, stdev=192.84 clat percentiles (msec): | 1.00th=[ 592], 5.00th=[ 693], 10.00th=[ 760], 20.00th=[ 852], | 30.00th=[ 919], 40.00th=[ 969], 50.00th=[ 1020], 60.00th=[ 1062], | 70.00th=[ 1116], 80.00th=[ 1167], 90.00th=[ 1250], 95.00th=[ 1318], | 99.00th=[ 1435], 99.50th=[ 1485], 99.90th=[ 1653], 99.95th=[ 1670], | 99.99th=[ 1720] bw ( KiB/s): min= 7, max= 128, per=24.96%, avg=62.90, stdev=26.74, samples=322 iops : min= 1, max= 32, avg=15.48, stdev= 6.71, samples=322 lat (msec) : 250=0.04%, 500=0.66%, 750=8.83%, 1000=36.91% cpu : usr=0.09%, sys=0.38%, ctx=1368, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=559: Thu May 1 23:16:57 2025 write: IOPS=15, BW=63.2KiB/s (64.7kB/s)(10.0MiB/162149msec) slat (usec): min=12, max=10402, avg=201.72, stdev=604.16 clat (msec): min=402, max=1659, avg=1011.68, stdev=189.94 lat (msec): min=403, max=1660, avg=1011.89, stdev=189.94 clat percentiles (msec): | 1.00th=[ 592], 5.00th=[ 701], 10.00th=[ 776], 20.00th=[ 852], | 30.00th=[ 911], 40.00th=[ 961], 50.00th=[ 1011], 60.00th=[ 1070], | 70.00th=[ 1116], 80.00th=[ 1183], 90.00th=[ 1250], 95.00th=[ 1318], | 99.00th=[ 1435], 99.50th=[ 1485], 99.90th=[ 1569], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 7, max= 128, per=25.60%, avg=64.51, stdev=26.60, samples=314 iops : min= 1, max= 32, avg=15.90, stdev= 6.66, samples=314 lat (msec) : 500=0.70%, 750=6.99%, 1000=40.12% cpu : usr=0.09%, sys=0.35%, ctx=1329, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=560: Thu May 1 23:16:57 2025 write: IOPS=15, BW=63.3KiB/s (64.8kB/s)(10.0MiB/161719msec) slat (usec): min=12, max=8874, avg=213.96, stdev=567.26 clat (msec): min=161, max=1629, avg=1009.71, stdev=199.45 lat (msec): min=161, max=1629, avg=1009.93, stdev=199.45 clat percentiles (msec): | 1.00th=[ 498], 5.00th=[ 659], 10.00th=[ 751], 20.00th=[ 852], | 30.00th=[ 919], 40.00th=[ 969], 50.00th=[ 1020], 60.00th=[ 1070], | 70.00th=[ 1116], 80.00th=[ 1183], 90.00th=[ 1267], 95.00th=[ 1318], | 99.00th=[ 1435], 99.50th=[ 1485], 99.90th=[ 1552], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 8, max= 128, per=25.42%, avg=64.05, stdev=25.94, samples=316 iops : min= 2, max= 32, avg=15.72, stdev= 6.47, samples=316 lat (msec) : 250=0.20%, 500=0.82%, 750=8.83%, 1000=36.88% cpu : usr=0.07%, sys=0.39%, ctx=1339, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=253KiB/s (259kB/s), 63.1KiB/s-63.3KiB/s (64.7kB/s-64.8kB/s), io=40.0MiB (41.9MB), run=161719-162181msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=705: Thu May 1 23:18:34 2025 read: IOPS=13, BW=52.9KiB/s (54.2kB/s)(5048KiB/95405msec) slat (usec): min=1596, max=59341, avg=8331.55, stdev=4642.60 clat (msec): min=11, max=670, avg=390.22, stdev=85.51 lat (msec): min=16, max=679, avg=398.56, stdev=86.31 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 259], 10.00th=[ 284], 20.00th=[ 317], | 30.00th=[ 342], 40.00th=[ 368], 50.00th=[ 388], 60.00th=[ 409], | 70.00th=[ 430], 80.00th=[ 464], 90.00th=[ 506], 95.00th=[ 535], | 99.00th=[ 600], 99.50th=[ 617], 99.90th=[ 659], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 8, max= 119, per=25.09%, avg=52.70, stdev=23.66, samples=189 iops : min= 2, max= 29, avg=12.95, stdev= 5.91, samples=189 write: IOPS=13, BW=54.4KiB/s (55.7kB/s)(5192KiB/95405msec) slat (usec): min=13, max=11238, avg=235.57, stdev=618.69 clat (msec): min=249, max=1423, avg=786.44, stdev=212.81 lat (msec): min=249, max=1424, avg=786.68, stdev=212.83 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 468], 10.00th=[ 514], 20.00th=[ 609], | 30.00th=[ 667], 40.00th=[ 718], 50.00th=[ 768], 60.00th=[ 818], | 70.00th=[ 894], 80.00th=[ 969], 90.00th=[ 1070], 95.00th=[ 1183], | 99.00th=[ 1301], 99.50th=[ 1368], 99.90th=[ 1418], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 7, max= 120, per=24.74%, avg=53.94, stdev=20.62, samples=190 iops : min= 1, max= 30, avg=13.26, stdev= 5.20, samples=190 lat (msec) : 20=0.04%, 100=0.08%, 250=1.80%, 500=46.13%, 750=25.23% lat (msec) : 1000=18.44% cpu : usr=0.27%, sys=1.07%, ctx=2721, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=706: Thu May 1 23:18:34 2025 read: IOPS=13, BW=52.3KiB/s (53.6kB/s)(4976KiB/95112msec) slat (usec): min=1984, max=66520, avg=8230.51, stdev=4543.93 clat (msec): min=5, max=712, avg=388.09, stdev=89.36 lat (msec): min=9, max=719, avg=396.33, stdev=90.27 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 251], 10.00th=[ 284], 20.00th=[ 317], | 30.00th=[ 342], 40.00th=[ 363], 50.00th=[ 384], 60.00th=[ 409], | 70.00th=[ 430], 80.00th=[ 464], 90.00th=[ 506], 95.00th=[ 535], | 99.00th=[ 600], 99.50th=[ 617], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 7, max= 128, per=24.78%, avg=52.05, stdev=24.30, samples=189 iops : min= 1, max= 32, avg=12.78, stdev= 6.13, samples=189 write: IOPS=13, BW=55.3KiB/s (56.7kB/s)(5264KiB/95112msec) slat (usec): min=14, max=4340, avg=243.74, stdev=527.10 clat (msec): min=115, max=1434, avg=780.06, stdev=221.82 lat (msec): min=115, max=1435, avg=780.31, stdev=221.86 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 439], 10.00th=[ 514], 20.00th=[ 592], | 30.00th=[ 651], 40.00th=[ 701], 50.00th=[ 760], 60.00th=[ 827], | 70.00th=[ 885], 80.00th=[ 969], 90.00th=[ 1083], 95.00th=[ 1200], | 99.00th=[ 1334], 99.50th=[ 1351], 99.90th=[ 1418], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 8, max= 144, per=25.12%, avg=54.76, stdev=22.54, samples=190 iops : min= 2, max= 36, avg=13.46, stdev= 5.64, samples=190 lat (msec) : 10=0.04%, 20=0.04%, 50=0.04%, 100=0.20%, 250=2.11% lat (msec) : 500=45.27%, 750=26.25%, 1000=17.34% cpu : usr=0.18%, sys=1.21%, ctx=2763, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=707: Thu May 1 23:18:34 2025 read: IOPS=13, BW=52.3KiB/s (53.6kB/s)(4984KiB/95290msec) slat (usec): min=37, max=54867, avg=8169.53, stdev=4311.34 clat (msec): min=9, max=894, avg=386.10, stdev=92.86 lat (msec): min=12, max=908, avg=394.28, stdev=93.71 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 249], 10.00th=[ 275], 20.00th=[ 313], | 30.00th=[ 338], 40.00th=[ 363], 50.00th=[ 384], 60.00th=[ 405], | 70.00th=[ 435], 80.00th=[ 464], 90.00th=[ 502], 95.00th=[ 535], | 99.00th=[ 609], 99.50th=[ 625], 99.90th=[ 885], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 8, max= 120, per=24.75%, avg=51.97, stdev=24.15, samples=190 iops : min= 2, max= 30, avg=12.75, stdev= 6.03, samples=190 write: IOPS=13, BW=55.2KiB/s (56.5kB/s)(5256KiB/95290msec) slat (usec): min=13, max=9272, avg=243.01, stdev=607.27 clat (msec): min=121, max=1468, avg=785.34, stdev=215.51 lat (msec): min=122, max=1468, avg=785.59, stdev=215.54 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 460], 10.00th=[ 523], 20.00th=[ 592], | 30.00th=[ 659], 40.00th=[ 718], 50.00th=[ 776], 60.00th=[ 835], | 70.00th=[ 894], 80.00th=[ 961], 90.00th=[ 1070], 95.00th=[ 1167], | 99.00th=[ 1301], 99.50th=[ 1351], 99.90th=[ 1418], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 15, max= 96, per=25.02%, avg=54.54, stdev=18.91, samples=190 iops : min= 3, max= 24, avg=13.39, stdev= 4.79, samples=190 lat (msec) : 10=0.04%, 20=0.08%, 50=0.12%, 100=0.16%, 250=2.50% lat (msec) : 500=45.08%, 750=24.02%, 1000=19.73% cpu : usr=0.21%, sys=1.14%, ctx=2714, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=708: Thu May 1 23:18:34 2025 read: IOPS=13, BW=53.1KiB/s (54.4kB/s)(5060KiB/95290msec) slat (usec): min=1732, max=31764, avg=8164.19, stdev=4149.52 clat (msec): min=43, max=888, avg=391.43, stdev=92.22 lat (msec): min=54, max=895, avg=399.60, stdev=92.91 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 257], 10.00th=[ 288], 20.00th=[ 321], | 30.00th=[ 342], 40.00th=[ 359], 50.00th=[ 384], 60.00th=[ 405], | 70.00th=[ 439], 80.00th=[ 468], 90.00th=[ 510], 95.00th=[ 542], | 99.00th=[ 592], 99.50th=[ 651], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 7, max= 128, per=25.17%, avg=52.85, stdev=23.91, samples=189 iops : min= 1, max= 32, avg=12.95, stdev= 5.99, samples=189 write: IOPS=13, BW=54.4KiB/s (55.7kB/s)(5180KiB/95290msec) slat (usec): min=15, max=7211, avg=237.32, stdev=590.08 clat (msec): min=277, max=1416, avg=785.18, stdev=215.91 lat (msec): min=277, max=1416, avg=785.42, stdev=215.94 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 451], 10.00th=[ 510], 20.00th=[ 600], | 30.00th=[ 667], 40.00th=[ 718], 50.00th=[ 768], 60.00th=[ 827], | 70.00th=[ 894], 80.00th=[ 969], 90.00th=[ 1070], 95.00th=[ 1183], | 99.00th=[ 1334], 99.50th=[ 1368], 99.90th=[ 1418], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 8, max= 144, per=24.72%, avg=53.88, stdev=21.09, samples=190 iops : min= 2, max= 36, avg=13.21, stdev= 5.32, samples=190 lat (msec) : 50=0.04%, 100=0.23%, 250=1.60%, 500=46.41%, 750=24.88% lat (msec) : 1000=18.59% cpu : usr=0.18%, sys=1.21%, ctx=2710, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=210KiB/s (215kB/s), 52.3KiB/s-53.1KiB/s (53.6kB/s-54.4kB/s), io=19.6MiB (20.5MB), run=95112-95405msec WRITE: bw=219KiB/s (224kB/s), 54.4KiB/s-55.3KiB/s (55.7kB/s-56.7kB/s), io=20.4MiB (21.4MB), run=95112-95405msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=792: Thu May 1 23:18:37 2025 read: IOPS=1, BW=62.8MiB/s (65.8MB/s)(40.0MiB/637msec) slat (nsec): min=6182.3k, max=6182.3k, avg=6182307.00, stdev= 0.00 clat (nsec): min=598600k, max=598600k, avg=598599914.00, stdev= 0.00 lat (nsec): min=604808k, max=604808k, avg=604807851.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 600], 5.00th=[ 600], 10.00th=[ 600], 20.00th=[ 600], | 30.00th=[ 600], 40.00th=[ 600], 50.00th=[ 600], 60.00th=[ 600], | 70.00th=[ 600], 80.00th=[ 600], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] lat (msec) : 750=100.00% cpu : usr=0.16%, sys=0.63%, ctx=6, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=62.8MiB/s (65.8MB/s), 62.8MiB/s-62.8MiB/s (65.8MB/s-65.8MB/s), io=40.0MiB (41.9MB), run=637-637msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout