== sanity test 398c: run fio to test AIO ================= 22:22:57 (1712024577) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.147712 s, 284 MB/s osc.lustre-OST0000-osc-ffff8800ad80f800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=27098: Mon Apr 1 22:24:48 2024 write: IOPS=23, BW=94.4KiB/s (96.7kB/s)(10.0MiB/108470msec) slat (usec): min=15, max=6898, avg=153.89, stdev=394.80 clat (msec): min=49, max=1342, avg=677.07, stdev=159.65 lat (msec): min=49, max=1342, avg=677.22, stdev=159.66 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 451], 10.00th=[ 502], 20.00th=[ 558], | 30.00th=[ 592], 40.00th=[ 634], 50.00th=[ 667], 60.00th=[ 701], | 70.00th=[ 751], 80.00th=[ 793], 90.00th=[ 885], 95.00th=[ 953], | 99.00th=[ 1099], 99.50th=[ 1150], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1351] bw ( KiB/s): min= 23, max= 231, per=24.94%, avg=94.02, stdev=29.52, samples=216 iops : min= 5, max= 57, avg=23.32, stdev= 7.36, samples=216 lat (msec) : 50=0.04%, 100=0.08%, 250=0.59%, 500=9.30%, 750=60.35% lat (msec) : 1000=26.72% cpu : usr=0.07%, sys=0.52%, ctx=1474, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=27099: Mon Apr 1 22:24:48 2024 write: IOPS=23, BW=94.4KiB/s (96.6kB/s)(10.0MiB/108516msec) slat (usec): min=14, max=3559, avg=147.59, stdev=307.18 clat (msec): min=202, max=1256, avg=677.51, stdev=153.56 lat (msec): min=202, max=1257, avg=677.66, stdev=153.58 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 456], 10.00th=[ 506], 20.00th=[ 558], | 30.00th=[ 592], 40.00th=[ 625], 50.00th=[ 667], 60.00th=[ 709], | 70.00th=[ 751], 80.00th=[ 802], 90.00th=[ 877], 95.00th=[ 936], | 99.00th=[ 1099], 99.50th=[ 1116], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1250] bw ( KiB/s): min= 8, max= 255, per=24.83%, avg=93.62, stdev=30.53, samples=217 iops : min= 2, max= 63, avg=23.25, stdev= 7.60, samples=217 lat (msec) : 250=0.62%, 500=8.59%, 750=61.02%, 1000=26.91% cpu : usr=0.10%, sys=0.48%, ctx=1397, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=27100: Mon Apr 1 22:24:48 2024 write: IOPS=23, BW=94.4KiB/s (96.6kB/s)(10.0MiB/108516msec) slat (usec): min=15, max=7132, avg=146.78, stdev=343.20 clat (msec): min=72, max=1231, avg=677.48, stdev=157.35 lat (msec): min=72, max=1231, avg=677.63, stdev=157.35 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 451], 10.00th=[ 498], 20.00th=[ 558], | 30.00th=[ 592], 40.00th=[ 625], 50.00th=[ 667], 60.00th=[ 709], | 70.00th=[ 743], 80.00th=[ 802], 90.00th=[ 894], 95.00th=[ 969], | 99.00th=[ 1045], 99.50th=[ 1099], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1234] bw ( KiB/s): min= 23, max= 240, per=24.82%, avg=93.59, stdev=29.80, samples=217 iops : min= 5, max= 60, avg=23.22, stdev= 7.46, samples=217 lat (msec) : 100=0.08%, 250=0.78%, 500=9.49%, 750=60.70%, 1000=26.17% cpu : usr=0.09%, sys=0.49%, ctx=1433, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=27101: Mon Apr 1 22:24:48 2024 write: IOPS=23, BW=94.4KiB/s (96.6kB/s)(10.0MiB/108516msec) slat (usec): min=12, max=3955, avg=146.80, stdev=320.55 clat (msec): min=48, max=1338, avg=677.39, stdev=156.87 lat (msec): min=49, max=1339, avg=677.53, stdev=156.88 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 447], 10.00th=[ 502], 20.00th=[ 558], | 30.00th=[ 592], 40.00th=[ 634], 50.00th=[ 667], 60.00th=[ 701], | 70.00th=[ 760], 80.00th=[ 810], 90.00th=[ 877], 95.00th=[ 944], | 99.00th=[ 1070], 99.50th=[ 1116], 99.90th=[ 1167], 99.95th=[ 1234], | 99.99th=[ 1334] bw ( KiB/s): min= 15, max= 215, per=24.93%, avg=93.98, stdev=31.35, samples=216 iops : min= 3, max= 53, avg=23.28, stdev= 7.81, samples=216 lat (msec) : 50=0.08%, 100=0.08%, 250=0.70%, 500=9.14%, 750=58.71% lat (msec) : 1000=28.98% cpu : usr=0.12%, sys=0.46%, ctx=1446, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=377KiB/s (387kB/s), 94.4KiB/s-94.4KiB/s (96.6kB/s-96.7kB/s), io=40.0MiB (41.9MB), run=108470-108516msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27198: Mon Apr 1 22:26:01 2024 read: IOPS=17, BW=70.7KiB/s (72.4kB/s)(5048KiB/71433msec) slat (usec): min=15, max=24708, avg=5758.28, stdev=2520.24 clat (msec): min=44, max=476, avg=285.43, stdev=57.73 lat (msec): min=48, max=485, avg=291.20, stdev=58.15 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 197], 10.00th=[ 220], 20.00th=[ 241], | 30.00th=[ 255], 40.00th=[ 271], 50.00th=[ 288], 60.00th=[ 300], | 70.00th=[ 317], 80.00th=[ 330], 90.00th=[ 359], 95.00th=[ 380], | 99.00th=[ 409], 99.50th=[ 435], 99.90th=[ 477], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 16, max= 143, per=25.16%, avg=70.18, stdev=26.42, samples=142 iops : min= 4, max= 35, avg=17.25, stdev= 6.63, samples=142 write: IOPS=18, BW=72.7KiB/s (74.4kB/s)(5192KiB/71433msec) slat (usec): min=15, max=5011, avg=162.92, stdev=326.93 clat (msec): min=84, max=1125, avg=597.00, stdev=149.20 lat (msec): min=84, max=1126, avg=597.16, stdev=149.21 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 368], 10.00th=[ 414], 20.00th=[ 477], | 30.00th=[ 510], 40.00th=[ 550], 50.00th=[ 592], 60.00th=[ 625], | 70.00th=[ 676], 80.00th=[ 726], 90.00th=[ 793], 95.00th=[ 852], | 99.00th=[ 953], 99.50th=[ 1011], 99.90th=[ 1116], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 31, max= 144, per=24.79%, avg=71.88, stdev=21.40, samples=142 iops : min= 7, max= 36, avg=17.67, stdev= 5.35, samples=142 lat (msec) : 50=0.08%, 100=0.47%, 250=12.73%, 500=50.00%, 750=28.48% lat (msec) : 1000=7.93% cpu : usr=0.06%, sys=1.51%, ctx=2767, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27199: Mon Apr 1 22:26:01 2024 read: IOPS=17, BW=69.3KiB/s (70.0kB/s)(4976KiB/71797msec) slat (usec): min=22, max=19573, avg=5809.23, stdev=2444.33 clat (msec): min=5, max=513, avg=286.35, stdev=58.54 lat (msec): min=6, max=518, avg=292.16, stdev=58.73 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 203], 10.00th=[ 224], 20.00th=[ 245], | 30.00th=[ 257], 40.00th=[ 271], 50.00th=[ 288], 60.00th=[ 300], | 70.00th=[ 313], 80.00th=[ 330], 90.00th=[ 359], 95.00th=[ 384], | 99.00th=[ 435], 99.50th=[ 464], 99.90th=[ 472], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 15, max= 136, per=24.61%, avg=68.65, stdev=26.74, samples=143 iops : min= 3, max= 34, avg=16.92, stdev= 6.73, samples=143 write: IOPS=18, BW=73.3KiB/s (75.1kB/s)(5264KiB/71797msec) slat (usec): min=17, max=4090, avg=165.01, stdev=334.63 clat (msec): min=114, max=1132, avg=596.32, stdev=152.41 lat (msec): min=114, max=1132, avg=596.49, stdev=152.42 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 359], 10.00th=[ 401], 20.00th=[ 464], | 30.00th=[ 510], 40.00th=[ 550], 50.00th=[ 592], 60.00th=[ 625], | 70.00th=[ 676], 80.00th=[ 726], 90.00th=[ 793], 95.00th=[ 852], | 99.00th=[ 961], 99.50th=[ 1045], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 15, max= 144, per=25.14%, avg=72.91, stdev=21.31, samples=143 iops : min= 3, max= 36, avg=17.99, stdev= 5.34, samples=143 lat (msec) : 10=0.08%, 20=0.04%, 50=0.27%, 100=0.12%, 250=11.21% lat (msec) : 500=51.09%, 750=28.98%, 1000=7.81% cpu : usr=0.20%, sys=1.36%, ctx=2752, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27200: Mon Apr 1 22:26:01 2024 read: IOPS=17, BW=69.4KiB/s (71.0kB/s)(4984KiB/71859msec) slat (usec): min=31, max=22218, avg=5650.32, stdev=2373.20 clat (msec): min=8, max=471, avg=284.57, stdev=52.46 lat (msec): min=8, max=481, avg=290.23, stdev=52.76 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 205], 10.00th=[ 226], 20.00th=[ 245], | 30.00th=[ 257], 40.00th=[ 271], 50.00th=[ 288], 60.00th=[ 296], | 70.00th=[ 313], 80.00th=[ 326], 90.00th=[ 351], 95.00th=[ 368], | 99.00th=[ 405], 99.50th=[ 435], 99.90th=[ 468], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 8, max= 176, per=24.86%, avg=69.36, stdev=27.07, samples=142 iops : min= 2, max= 44, avg=17.13, stdev= 6.80, samples=142 write: IOPS=18, BW=73.1KiB/s (74.9kB/s)(5256KiB/71859msec) slat (usec): min=16, max=3365, avg=167.80, stdev=326.75 clat (msec): min=140, max=1151, avg=599.43, stdev=152.43 lat (msec): min=140, max=1151, avg=599.60, stdev=152.43 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 363], 10.00th=[ 414], 20.00th=[ 468], | 30.00th=[ 514], 40.00th=[ 550], 50.00th=[ 592], 60.00th=[ 634], | 70.00th=[ 676], 80.00th=[ 735], 90.00th=[ 802], 95.00th=[ 844], | 99.00th=[ 978], 99.50th=[ 1083], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 16, max= 120, per=24.94%, avg=72.31, stdev=21.00, samples=143 iops : min= 4, max= 30, avg=17.87, stdev= 5.28, samples=143 lat (msec) : 10=0.08%, 50=0.12%, 100=0.12%, 250=11.41%, 500=50.35% lat (msec) : 750=29.41%, 1000=8.09% cpu : usr=0.16%, sys=1.38%, ctx=2705, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27201: Mon Apr 1 22:26:01 2024 read: IOPS=17, BW=70.8KiB/s (72.5kB/s)(5060KiB/71432msec) slat (usec): min=24, max=24111, avg=5652.44, stdev=2465.56 clat (msec): min=7, max=804, avg=286.62, stdev=62.47 lat (msec): min=10, max=807, avg=292.28, stdev=62.86 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 199], 10.00th=[ 222], 20.00th=[ 243], | 30.00th=[ 257], 40.00th=[ 275], 50.00th=[ 288], 60.00th=[ 300], | 70.00th=[ 317], 80.00th=[ 334], 90.00th=[ 355], 95.00th=[ 376], | 99.00th=[ 405], 99.50th=[ 451], 99.90th=[ 743], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 7, max= 151, per=25.36%, avg=70.75, stdev=26.40, samples=142 iops : min= 1, max= 37, avg=17.46, stdev= 6.64, samples=142 write: IOPS=18, BW=72.5KiB/s (74.3kB/s)(5180KiB/71432msec) slat (usec): min=17, max=6048, avg=163.10, stdev=357.51 clat (msec): min=193, max=1156, avg=596.45, stdev=150.63 lat (msec): min=193, max=1156, avg=596.61, stdev=150.63 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 359], 10.00th=[ 405], 20.00th=[ 472], | 30.00th=[ 510], 40.00th=[ 550], 50.00th=[ 592], 60.00th=[ 625], | 70.00th=[ 676], 80.00th=[ 718], 90.00th=[ 793], 95.00th=[ 852], | 99.00th=[ 978], 99.50th=[ 1062], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 31, max= 127, per=24.84%, avg=72.04, stdev=20.38, samples=142 iops : min= 7, max= 31, avg=17.79, stdev= 5.10, samples=142 lat (msec) : 10=0.08%, 50=0.23%, 100=0.23%, 250=11.68%, 500=50.98% lat (msec) : 750=28.48%, 1000=7.89% cpu : usr=0.04%, sys=1.46%, ctx=2745, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=279KiB/s (286kB/s), 69.3KiB/s-70.8KiB/s (70.0kB/s-72.5kB/s), io=19.6MiB (20.5MB), run=71432-71859msec WRITE: bw=291KiB/s (298kB/s), 72.5KiB/s-73.3KiB/s (74.3kB/s-75.1kB/s), io=20.4MiB (21.4MB), run=71432-71859msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=27263: Mon Apr 1 22:26:03 2024 read: IOPS=2, BW=99.8MiB/s (105MB/s)(40.0MiB/401msec) slat (nsec): min=13073k, max=13073k, avg=13072656.00, stdev= 0.00 clat (nsec): min=385680k, max=385680k, avg=385680378.00, stdev= 0.00 lat (nsec): min=398781k, max=398781k, avg=398780816.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 384], 10.00th=[ 384], 20.00th=[ 384], | 30.00th=[ 384], 40.00th=[ 384], 50.00th=[ 384], 60.00th=[ 384], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] lat (msec) : 500=100.00% cpu : usr=0.25%, sys=1.75%, ctx=2, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=99.8MiB/s (105MB/s), 99.8MiB/s-99.8MiB/s (105MB/s-105MB/s), io=40.0MiB (41.9MB), run=401-401msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout