== sanity test 398c: run fio to test AIO ================= 03:00:10 (1746082810) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.123705 s, 339 MB/s osc.lustre-OST0000-osc-ffff8800b5930000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=5693: Thu May 1 03:01:45 2025 write: IOPS=27, BW=109KiB/s (112kB/s)(10.0MiB/93603msec) slat (usec): min=12, max=5020, avg=142.44, stdev=300.53 clat (msec): min=113, max=1120, avg=584.02, stdev=166.20 lat (msec): min=113, max=1120, avg=584.17, stdev=166.22 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 300], 10.00th=[ 372], 20.00th=[ 447], | 30.00th=[ 498], 40.00th=[ 542], 50.00th=[ 584], 60.00th=[ 625], | 70.00th=[ 676], 80.00th=[ 735], 90.00th=[ 802], 95.00th=[ 852], | 99.00th=[ 953], 99.50th=[ 986], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1116] bw ( KiB/s): min= 23, max= 256, per=24.84%, avg=108.53, stdev=37.86, samples=187 iops : min= 5, max= 64, avg=26.88, stdev= 9.47, samples=187 lat (msec) : 250=2.62%, 500=27.66%, 750=52.85%, 1000=16.48% cpu : usr=0.08%, sys=0.59%, ctx=1539, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=5694: Thu May 1 03:01:45 2025 write: IOPS=27, BW=109KiB/s (112kB/s)(10.0MiB/93535msec) slat (usec): min=12, max=3880, avg=137.53, stdev=290.14 clat (msec): min=114, max=1324, avg=583.45, stdev=167.13 lat (msec): min=114, max=1324, avg=583.59, stdev=167.15 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 305], 10.00th=[ 372], 20.00th=[ 447], | 30.00th=[ 493], 40.00th=[ 542], 50.00th=[ 584], 60.00th=[ 625], | 70.00th=[ 676], 80.00th=[ 726], 90.00th=[ 802], 95.00th=[ 852], | 99.00th=[ 953], 99.50th=[ 995], 99.90th=[ 1183], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 40, max= 264, per=24.97%, avg=109.12, stdev=37.05, samples=186 iops : min= 10, max= 66, avg=27.03, stdev= 9.27, samples=186 lat (msec) : 250=2.81%, 500=28.09%, 750=53.71%, 1000=15.00% cpu : usr=0.13%, sys=0.53%, ctx=1575, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=5695: Thu May 1 03:01:45 2025 write: IOPS=27, BW=109KiB/s (112kB/s)(10.0MiB/93535msec) slat (usec): min=14, max=5748, avg=148.32, stdev=321.23 clat (msec): min=88, max=1366, avg=583.72, stdev=169.43 lat (msec): min=88, max=1366, avg=583.87, stdev=169.44 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 300], 10.00th=[ 368], 20.00th=[ 447], | 30.00th=[ 498], 40.00th=[ 542], 50.00th=[ 584], 60.00th=[ 625], | 70.00th=[ 676], 80.00th=[ 726], 90.00th=[ 810], 95.00th=[ 860], | 99.00th=[ 961], 99.50th=[ 1003], 99.90th=[ 1234], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min= 32, max= 280, per=24.99%, avg=109.22, stdev=38.19, samples=186 iops : min= 8, max= 70, avg=27.12, stdev= 9.54, samples=186 lat (msec) : 100=0.12%, 250=2.81%, 500=27.62%, 750=53.59%, 1000=15.31% cpu : usr=0.15%, sys=0.53%, ctx=1664, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=5696: Thu May 1 03:01:45 2025 write: IOPS=27, BW=109KiB/s (112kB/s)(10.0MiB/93568msec) slat (usec): min=12, max=5473, avg=137.17, stdev=303.99 clat (msec): min=94, max=1048, avg=583.42, stdev=163.96 lat (msec): min=94, max=1048, avg=583.56, stdev=163.98 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 305], 10.00th=[ 372], 20.00th=[ 451], | 30.00th=[ 502], 40.00th=[ 542], 50.00th=[ 584], 60.00th=[ 625], | 70.00th=[ 667], 80.00th=[ 726], 90.00th=[ 802], 95.00th=[ 852], | 99.00th=[ 953], 99.50th=[ 978], 99.90th=[ 1020], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 15, max= 256, per=24.84%, avg=108.55, stdev=36.67, samples=187 iops : min= 3, max= 64, avg=26.89, stdev= 9.18, samples=187 lat (msec) : 100=0.62%, 250=1.41%, 500=27.97%, 750=53.95%, 1000=15.66% cpu : usr=0.10%, sys=0.55%, ctx=1560, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=438KiB/s (448kB/s), 109KiB/s-109KiB/s (112kB/s-112kB/s), io=40.0MiB (41.9MB), run=93535-93603msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5781: Thu May 1 03:02:43 2025 read: IOPS=22, BW=90.2KiB/s (92.3kB/s)(5048KiB/55979msec) slat (usec): min=1401, max=15816, avg=4969.40, stdev=1987.45 clat (msec): min=10, max=448, avg=228.54, stdev=47.98 lat (msec): min=15, max=455, avg=233.52, stdev=48.31 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 161], 10.00th=[ 174], 20.00th=[ 192], | 30.00th=[ 201], 40.00th=[ 213], 50.00th=[ 226], 60.00th=[ 239], | 70.00th=[ 251], 80.00th=[ 271], 90.00th=[ 288], 95.00th=[ 309], | 99.00th=[ 351], 99.50th=[ 368], 99.90th=[ 443], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 24, max= 176, per=25.03%, avg=89.60, stdev=30.64, samples=111 iops : min= 6, max= 44, avg=22.14, stdev= 7.68, samples=111 write: IOPS=23, BW=92.7KiB/s (94.0kB/s)(5192KiB/55979msec) slat (usec): min=16, max=3581, avg=152.89, stdev=286.74 clat (msec): min=131, max=891, avg=461.50, stdev=122.98 lat (msec): min=131, max=892, avg=461.66, stdev=122.99 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 271], 10.00th=[ 305], 20.00th=[ 347], | 30.00th=[ 393], 40.00th=[ 426], 50.00th=[ 456], 60.00th=[ 489], | 70.00th=[ 523], 80.00th=[ 567], 90.00th=[ 625], 95.00th=[ 667], | 99.00th=[ 785], 99.50th=[ 810], 99.90th=[ 844], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 32, max= 176, per=24.75%, avg=92.05, stdev=28.43, samples=111 iops : min= 8, max= 44, avg=22.76, stdev= 7.14, samples=111 lat (msec) : 20=0.04%, 50=0.20%, 100=0.16%, 250=34.80%, 500=46.37% lat (msec) : 750=17.30%, 1000=1.13% cpu : usr=0.26%, sys=1.59%, ctx=2711, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5782: Thu May 1 03:02:43 2025 read: IOPS=22, BW=88.8KiB/s (90.9kB/s)(4976KiB/56024msec) slat (usec): min=1413, max=18315, avg=5089.23, stdev=2130.37 clat (msec): min=4, max=373, avg=227.56, stdev=51.05 lat (msec): min=8, max=378, avg=232.65, stdev=51.46 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 159], 10.00th=[ 171], 20.00th=[ 188], | 30.00th=[ 201], 40.00th=[ 213], 50.00th=[ 226], 60.00th=[ 236], | 70.00th=[ 251], 80.00th=[ 271], 90.00th=[ 296], 95.00th=[ 317], | 99.00th=[ 355], 99.50th=[ 363], 99.90th=[ 372], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 8, max= 152, per=24.51%, avg=87.76, stdev=32.40, samples=111 iops : min= 2, max= 38, avg=21.76, stdev= 8.09, samples=111 write: IOPS=23, BW=93.0KiB/s (96.2kB/s)(5264KiB/56024msec) slat (usec): min=14, max=2448, avg=143.57, stdev=241.17 clat (msec): min=173, max=924, avg=460.13, stdev=123.00 lat (msec): min=173, max=924, avg=460.28, stdev=123.02 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 271], 10.00th=[ 305], 20.00th=[ 347], | 30.00th=[ 380], 40.00th=[ 422], 50.00th=[ 460], 60.00th=[ 489], | 70.00th=[ 527], 80.00th=[ 558], 90.00th=[ 625], 95.00th=[ 667], | 99.00th=[ 793], 99.50th=[ 818], 99.90th=[ 844], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 32, max= 176, per=25.02%, avg=93.09, stdev=26.27, samples=111 iops : min= 8, max= 44, avg=23.09, stdev= 6.54, samples=111 lat (msec) : 10=0.08%, 20=0.12%, 50=0.23%, 100=0.12%, 250=34.45% lat (msec) : 500=45.94%, 750=18.20%, 1000=0.86% cpu : usr=0.32%, sys=1.51%, ctx=2710, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5783: Thu May 1 03:02:43 2025 read: IOPS=22, BW=89.0KiB/s (91.2kB/s)(4984KiB/55978msec) slat (usec): min=1553, max=21127, avg=5106.59, stdev=2065.29 clat (msec): min=6, max=446, avg=227.13, stdev=50.72 lat (msec): min=10, max=454, avg=232.24, stdev=51.10 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 157], 10.00th=[ 169], 20.00th=[ 188], | 30.00th=[ 201], 40.00th=[ 213], 50.00th=[ 226], 60.00th=[ 236], | 70.00th=[ 251], 80.00th=[ 271], 90.00th=[ 292], 95.00th=[ 317], | 99.00th=[ 351], 99.50th=[ 363], 99.90th=[ 443], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 23, max= 176, per=24.85%, avg=88.95, stdev=31.08, samples=111 iops : min= 5, max= 44, avg=22.04, stdev= 7.76, samples=111 write: IOPS=23, BW=93.9KiB/s (96.1kB/s)(5256KiB/55978msec) slat (usec): min=15, max=2813, avg=141.89, stdev=239.03 clat (msec): min=111, max=829, avg=459.80, stdev=122.91 lat (msec): min=111, max=829, avg=459.95, stdev=122.92 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 266], 10.00th=[ 300], 20.00th=[ 351], | 30.00th=[ 388], 40.00th=[ 422], 50.00th=[ 460], 60.00th=[ 493], | 70.00th=[ 523], 80.00th=[ 558], 90.00th=[ 625], 95.00th=[ 667], | 99.00th=[ 785], 99.50th=[ 802], 99.90th=[ 827], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 40, max= 176, per=25.02%, avg=93.06, stdev=26.28, samples=111 iops : min= 10, max= 44, avg=23.06, stdev= 6.58, samples=111 lat (msec) : 10=0.12%, 20=0.04%, 50=0.12%, 100=0.23%, 250=35.55% lat (msec) : 500=44.49%, 750=18.52%, 1000=0.94% cpu : usr=0.35%, sys=1.51%, ctx=2745, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5784: Thu May 1 03:02:43 2025 read: IOPS=22, BW=90.8KiB/s (92.0kB/s)(5060KiB/55732msec) slat (usec): min=42, max=21007, avg=4962.42, stdev=2103.32 clat (msec): min=7, max=585, avg=229.31, stdev=49.11 lat (msec): min=12, max=591, avg=234.28, stdev=49.64 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 161], 10.00th=[ 176], 20.00th=[ 192], | 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 226], 60.00th=[ 236], | 70.00th=[ 249], 80.00th=[ 268], 90.00th=[ 288], 95.00th=[ 309], | 99.00th=[ 359], 99.50th=[ 368], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 16, max= 176, per=25.28%, avg=90.51, stdev=29.71, samples=111 iops : min= 4, max= 44, avg=22.46, stdev= 7.48, samples=111 write: IOPS=23, BW=92.9KiB/s (95.2kB/s)(5180KiB/55732msec) slat (usec): min=17, max=4245, avg=138.80, stdev=254.56 clat (msec): min=89, max=844, avg=459.15, stdev=124.84 lat (msec): min=89, max=844, avg=459.29, stdev=124.82 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 271], 10.00th=[ 300], 20.00th=[ 347], | 30.00th=[ 380], 40.00th=[ 422], 50.00th=[ 460], 60.00th=[ 489], | 70.00th=[ 523], 80.00th=[ 567], 90.00th=[ 617], 95.00th=[ 676], | 99.00th=[ 785], 99.50th=[ 802], 99.90th=[ 835], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 40, max= 160, per=24.85%, avg=92.46, stdev=23.98, samples=111 iops : min= 10, max= 40, avg=22.95, stdev= 5.98, samples=111 lat (msec) : 10=0.04%, 20=0.08%, 50=0.12%, 100=0.12%, 250=35.62% lat (msec) : 500=45.47%, 750=17.50%, 1000=1.05% cpu : usr=0.43%, sys=1.38%, ctx=2698, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=358KiB/s (367kB/s), 88.8KiB/s-90.8KiB/s (90.9kB/s-92.0kB/s), io=19.6MiB (20.5MB), run=55732-56024msec WRITE: bw=373KiB/s (382kB/s), 92.7KiB/s-93.0KiB/s (94.0kB/s-96.2kB/s), io=20.4MiB (21.4MB), run=55732-56024msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=5832: Thu May 1 03:02:44 2025 read: IOPS=5, BW=213MiB/s (223MB/s)(40.0MiB/188msec) slat (nsec): min=7609.8k, max=7609.8k, avg=7609842.00, stdev= 0.00 clat (nsec): min=173564k, max=173564k, avg=173563531.00, stdev= 0.00 lat (nsec): min=181193k, max=181193k, avg=181192995.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 174], 50.00th=[ 174], 60.00th=[ 174], | 70.00th=[ 174], 80.00th=[ 174], 90.00th=[ 174], 95.00th=[ 174], | 99.00th=[ 174], 99.50th=[ 174], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] lat (msec) : 250=100.00% cpu : usr=0.53%, sys=3.21%, ctx=5, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=213MiB/s (223MB/s), 213MiB/s-213MiB/s (223MB/s-223MB/s), io=40.0MiB (41.9MB), run=188-188msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout