== sanity test 398c: run fio to test AIO ================= 01:15:43 (1746076543) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.22879 s, 183 MB/s osc.lustre-OST0000-osc-ffff88012d0a7000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=747: Thu May 1 01:18:02 2025 write: IOPS=18, BW=75.3KiB/s (77.1kB/s)(10.0MiB/135941msec) slat (usec): min=15, max=6272, avg=171.20, stdev=447.09 clat (msec): min=195, max=1456, avg=848.61, stdev=186.32 lat (msec): min=195, max=1457, avg=848.79, stdev=186.35 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 701], | 30.00th=[ 751], 40.00th=[ 802], 50.00th=[ 844], 60.00th=[ 894], | 70.00th=[ 936], 80.00th=[ 995], 90.00th=[ 1083], 95.00th=[ 1167], | 99.00th=[ 1334], 99.50th=[ 1368], 99.90th=[ 1401], 99.95th=[ 1418], | 99.99th=[ 1452] bw ( KiB/s): min= 7, max= 128, per=25.01%, avg=75.29, stdev=31.58, samples=269 iops : min= 1, max= 32, avg=18.61, stdev= 7.92, samples=269 lat (msec) : 250=0.62%, 500=0.98%, 750=28.24%, 1000=51.37% cpu : usr=0.08%, sys=0.42%, ctx=1325, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=748: Thu May 1 01:18:02 2025 write: IOPS=18, BW=75.4KiB/s (77.2kB/s)(10.0MiB/135861msec) slat (usec): min=15, max=6003, avg=164.47, stdev=442.82 clat (msec): min=72, max=1460, avg=848.46, stdev=194.01 lat (msec): min=72, max=1460, avg=848.62, stdev=194.02 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 550], 10.00th=[ 625], 20.00th=[ 701], | 30.00th=[ 751], 40.00th=[ 802], 50.00th=[ 844], 60.00th=[ 894], | 70.00th=[ 944], 80.00th=[ 1003], 90.00th=[ 1083], 95.00th=[ 1167], | 99.00th=[ 1334], 99.50th=[ 1385], 99.90th=[ 1435], 99.95th=[ 1452], | 99.99th=[ 1469] bw ( KiB/s): min= 15, max= 255, per=24.85%, avg=74.80, stdev=28.91, samples=271 iops : min= 3, max= 63, avg=18.45, stdev= 7.23, samples=271 lat (msec) : 100=0.62%, 500=1.52%, 750=28.44%, 1000=49.34% cpu : usr=0.08%, sys=0.40%, ctx=1350, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=749: Thu May 1 01:18:02 2025 write: IOPS=18, BW=75.3KiB/s (77.1kB/s)(10.0MiB/136020msec) slat (usec): min=11, max=14888, avg=169.57, stdev=537.28 clat (msec): min=192, max=1485, avg=849.12, stdev=188.08 lat (msec): min=192, max=1486, avg=849.29, stdev=188.11 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 701], | 30.00th=[ 751], 40.00th=[ 802], 50.00th=[ 844], 60.00th=[ 894], | 70.00th=[ 936], 80.00th=[ 995], 90.00th=[ 1099], 95.00th=[ 1167], | 99.00th=[ 1334], 99.50th=[ 1368], 99.90th=[ 1452], 99.95th=[ 1469], | 99.99th=[ 1485] bw ( KiB/s): min= 7, max= 128, per=24.71%, avg=74.38, stdev=26.84, samples=272 iops : min= 1, max= 32, avg=18.25, stdev= 6.73, samples=272 lat (msec) : 250=0.62%, 500=1.95%, 750=27.54%, 1000=51.37% cpu : usr=0.09%, sys=0.42%, ctx=1345, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=750: Thu May 1 01:18:02 2025 write: IOPS=18, BW=75.4KiB/s (77.2kB/s)(10.0MiB/135896msec) slat (usec): min=15, max=25140, avg=184.83, stdev=709.27 clat (msec): min=49, max=1486, avg=848.52, stdev=190.48 lat (msec): min=49, max=1486, avg=848.71, stdev=190.48 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 542], 10.00th=[ 625], 20.00th=[ 701], | 30.00th=[ 751], 40.00th=[ 802], 50.00th=[ 844], 60.00th=[ 894], | 70.00th=[ 936], 80.00th=[ 1003], 90.00th=[ 1099], 95.00th=[ 1167], | 99.00th=[ 1334], 99.50th=[ 1385], 99.90th=[ 1435], 99.95th=[ 1452], | 99.99th=[ 1485] bw ( KiB/s): min= 8, max= 152, per=24.85%, avg=74.79, stdev=27.70, samples=271 iops : min= 2, max= 38, avg=18.45, stdev= 6.92, samples=271 lat (msec) : 50=0.04%, 100=0.04%, 250=0.55%, 500=1.60%, 750=27.50% lat (msec) : 1000=50.70% cpu : usr=0.08%, sys=0.43%, ctx=1445, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=301KiB/s (308kB/s), 75.3KiB/s-75.4KiB/s (77.1kB/s-77.2kB/s), io=40.0MiB (41.9MB), run=135861-136020msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=895: Thu May 1 01:19:18 2025 read: IOPS=16, BW=67.4KiB/s (68.0kB/s)(5048KiB/74920msec) slat (usec): min=1747, max=35609, avg=7528.00, stdev=3512.59 clat (msec): min=11, max=646, avg=304.81, stdev=67.44 lat (msec): min=14, max=653, avg=312.35, stdev=68.19 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 207], 10.00th=[ 226], 20.00th=[ 251], | 30.00th=[ 271], 40.00th=[ 288], 50.00th=[ 305], 60.00th=[ 317], | 70.00th=[ 334], 80.00th=[ 355], 90.00th=[ 384], 95.00th=[ 409], | 99.00th=[ 493], 99.50th=[ 506], 99.90th=[ 634], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 15, max= 144, per=25.00%, avg=66.74, stdev=23.46, samples=149 iops : min= 3, max= 36, avg=16.45, stdev= 5.89, samples=149 write: IOPS=17, BW=69.3KiB/s (70.0kB/s)(5192KiB/74920msec) slat (usec): min=16, max=5325, avg=166.84, stdev=356.36 clat (msec): min=209, max=1124, avg=617.94, stdev=164.60 lat (msec): min=210, max=1124, avg=618.11, stdev=164.61 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 368], 10.00th=[ 422], 20.00th=[ 481], | 30.00th=[ 527], 40.00th=[ 558], 50.00th=[ 600], 60.00th=[ 634], | 70.00th=[ 693], 80.00th=[ 760], 90.00th=[ 852], 95.00th=[ 927], | 99.00th=[ 1011], 99.50th=[ 1036], 99.90th=[ 1099], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 23, max= 136, per=24.70%, avg=68.68, stdev=22.97, samples=149 iops : min= 5, max= 34, avg=16.93, stdev= 5.77, samples=149 lat (msec) : 20=0.04%, 50=0.04%, 100=0.23%, 250=9.65%, 500=51.05% lat (msec) : 750=27.93%, 1000=10.43% cpu : usr=0.23%, sys=1.29%, ctx=2711, majf=0, minf=35 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=896: Thu May 1 01:19:18 2025 read: IOPS=16, BW=66.4KiB/s (68.0kB/s)(4976KiB/74918msec) slat (usec): min=36, max=28730, avg=7452.44, stdev=3451.61 clat (msec): min=12, max=617, avg=306.78, stdev=72.84 lat (msec): min=12, max=623, avg=314.24, stdev=73.44 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 199], 10.00th=[ 222], 20.00th=[ 247], | 30.00th=[ 271], 40.00th=[ 288], 50.00th=[ 305], 60.00th=[ 321], | 70.00th=[ 342], 80.00th=[ 368], 90.00th=[ 393], 95.00th=[ 422], | 99.00th=[ 498], 99.50th=[ 510], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 8, max= 152, per=24.64%, avg=65.78, stdev=26.43, samples=149 iops : min= 2, max= 38, avg=16.26, stdev= 6.59, samples=149 write: IOPS=17, BW=70.3KiB/s (71.9kB/s)(5264KiB/74918msec) slat (usec): min=16, max=5790, avg=201.82, stdev=503.13 clat (msec): min=81, max=1120, avg=613.09, stdev=168.01 lat (msec): min=81, max=1120, avg=613.29, stdev=168.04 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 359], 10.00th=[ 418], 20.00th=[ 477], | 30.00th=[ 527], 40.00th=[ 558], 50.00th=[ 600], 60.00th=[ 634], | 70.00th=[ 693], 80.00th=[ 751], 90.00th=[ 844], 95.00th=[ 911], | 99.00th=[ 1020], 99.50th=[ 1062], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 24, max= 152, per=25.17%, avg=69.97, stdev=23.63, samples=149 iops : min= 6, max= 38, avg=17.30, stdev= 5.94, samples=149 lat (msec) : 20=0.08%, 50=0.16%, 100=0.51%, 250=9.96%, 500=50.00% lat (msec) : 750=28.63%, 1000=9.88% cpu : usr=0.32%, sys=1.26%, ctx=2689, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=897: Thu May 1 01:19:18 2025 read: IOPS=16, BW=66.3KiB/s (67.9kB/s)(4984KiB/75149msec) slat (usec): min=1718, max=27543, avg=7299.41, stdev=3371.67 clat (msec): min=11, max=568, avg=305.95, stdev=68.66 lat (msec): min=17, max=576, avg=313.25, stdev=69.17 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 209], 10.00th=[ 228], 20.00th=[ 253], | 30.00th=[ 271], 40.00th=[ 284], 50.00th=[ 300], 60.00th=[ 317], | 70.00th=[ 334], 80.00th=[ 359], 90.00th=[ 393], 95.00th=[ 426], | 99.00th=[ 493], 99.50th=[ 498], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 15, max= 152, per=24.61%, avg=65.71, stdev=26.71, samples=150 iops : min= 3, max= 38, avg=16.10, stdev= 6.69, samples=150 write: IOPS=17, BW=69.9KiB/s (71.6kB/s)(5256KiB/75149msec) slat (usec): min=14, max=11769, avg=190.98, stdev=545.57 clat (msec): min=154, max=1128, avg=616.25, stdev=168.46 lat (msec): min=154, max=1128, avg=616.45, stdev=168.46 clat percentiles (msec): | 1.00th=[ 264], 5.00th=[ 363], 10.00th=[ 405], 20.00th=[ 485], | 30.00th=[ 527], 40.00th=[ 558], 50.00th=[ 600], 60.00th=[ 642], | 70.00th=[ 693], 80.00th=[ 768], 90.00th=[ 852], 95.00th=[ 919], | 99.00th=[ 1036], 99.50th=[ 1070], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 24, max= 160, per=24.81%, avg=68.96, stdev=22.55, samples=150 iops : min= 6, max= 40, avg=16.91, stdev= 5.64, samples=150 lat (msec) : 20=0.04%, 50=0.04%, 100=0.12%, 250=9.45%, 500=50.78% lat (msec) : 750=28.32%, 1000=10.35% cpu : usr=0.28%, sys=1.28%, ctx=2685, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=898: Thu May 1 01:19:18 2025 read: IOPS=16, BW=67.7KiB/s (69.3kB/s)(5060KiB/74755msec) slat (usec): min=1740, max=36275, avg=7351.70, stdev=3274.51 clat (msec): min=33, max=947, avg=306.46, stdev=72.59 lat (msec): min=39, max=953, avg=313.82, stdev=73.16 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 207], 10.00th=[ 226], 20.00th=[ 249], | 30.00th=[ 271], 40.00th=[ 284], 50.00th=[ 300], 60.00th=[ 321], | 70.00th=[ 338], 80.00th=[ 363], 90.00th=[ 388], 95.00th=[ 422], | 99.00th=[ 502], 99.50th=[ 531], 99.90th=[ 726], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 8, max= 160, per=25.20%, avg=67.28, stdev=25.93, samples=149 iops : min= 2, max= 40, avg=16.54, stdev= 6.52, samples=149 write: IOPS=17, BW=69.3KiB/s (70.0kB/s)(5180KiB/74755msec) slat (usec): min=16, max=4273, avg=183.24, stdev=408.92 clat (msec): min=228, max=1306, avg=615.33, stdev=167.45 lat (msec): min=228, max=1306, avg=615.51, stdev=167.48 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 359], 10.00th=[ 418], 20.00th=[ 477], | 30.00th=[ 527], 40.00th=[ 558], 50.00th=[ 592], 60.00th=[ 634], | 70.00th=[ 693], 80.00th=[ 768], 90.00th=[ 844], 95.00th=[ 902], | 99.00th=[ 1028], 99.50th=[ 1116], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 16, max= 136, per=24.68%, avg=68.61, stdev=21.97, samples=149 iops : min= 4, max= 34, avg=16.88, stdev= 5.50, samples=149 lat (msec) : 50=0.12%, 100=0.04%, 250=10.23%, 500=50.78%, 750=27.42% lat (msec) : 1000=10.59% cpu : usr=0.22%, sys=1.33%, ctx=2709, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=267KiB/s (273kB/s), 66.3KiB/s-67.7KiB/s (67.9kB/s-69.3kB/s), io=19.6MiB (20.5MB), run=74755-75149msec WRITE: bw=278KiB/s (285kB/s), 69.3KiB/s-70.3KiB/s (70.0kB/s-71.9kB/s), io=20.4MiB (21.4MB), run=74755-75149msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=966: Thu May 1 01:19:21 2025 read: IOPS=2, BW=87.7MiB/s (91.0MB/s)(40.0MiB/456msec) slat (nsec): min=14080k, max=14080k, avg=14079621.00, stdev= 0.00 clat (nsec): min=430817k, max=430817k, avg=430817327.00, stdev= 0.00 lat (nsec): min=444922k, max=444922k, avg=444922429.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 430], 10.00th=[ 430], 20.00th=[ 430], | 30.00th=[ 430], 40.00th=[ 430], 50.00th=[ 430], 60.00th=[ 430], | 70.00th=[ 430], 80.00th=[ 430], 90.00th=[ 430], 95.00th=[ 430], | 99.00th=[ 430], 99.50th=[ 430], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.20%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=87.7MiB/s (91.0MB/s), 87.7MiB/s-87.7MiB/s (91.0MB/s-91.0MB/s), io=40.0MiB (41.9MB), run=456-456msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout