== sanity test 398c: run fio to test AIO ================= 18:34:43 (1746138883) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.157816 s, 266 MB/s osc.lustre-OST0000-osc-ffff880094be7800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=5109: Thu May 1 18:36:15 2025 write: IOPS=28, BW=113KiB/s (116kB/s)(10.0MiB/90278msec) slat (usec): min=12, max=4600, avg=137.34, stdev=274.78 clat (msec): min=75, max=984, avg=563.53, stdev=144.90 lat (msec): min=75, max=984, avg=563.67, stdev=144.92 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 334], 10.00th=[ 380], 20.00th=[ 443], | 30.00th=[ 485], 40.00th=[ 523], 50.00th=[ 558], 60.00th=[ 600], | 70.00th=[ 634], 80.00th=[ 693], 90.00th=[ 760], 95.00th=[ 802], | 99.00th=[ 885], 99.50th=[ 927], 99.90th=[ 969], 99.95th=[ 978], | 99.99th=[ 986] bw ( KiB/s): min= 15, max= 255, per=24.86%, avg=112.61, stdev=38.45, samples=180 iops : min= 3, max= 63, avg=27.91, stdev= 9.66, samples=180 lat (msec) : 100=0.62%, 250=0.39%, 500=32.89%, 750=55.62%, 1000=10.47% cpu : usr=0.15%, sys=0.53%, ctx=1412, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=5110: Thu May 1 18:36:15 2025 write: IOPS=28, BW=114KiB/s (116kB/s)(10.0MiB/90182msec) slat (usec): min=12, max=4985, avg=141.68, stdev=309.10 clat (msec): min=90, max=980, avg=562.98, stdev=146.60 lat (msec): min=90, max=980, avg=563.13, stdev=146.62 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 334], 10.00th=[ 380], 20.00th=[ 443], | 30.00th=[ 485], 40.00th=[ 523], 50.00th=[ 558], 60.00th=[ 600], | 70.00th=[ 642], 80.00th=[ 693], 90.00th=[ 760], 95.00th=[ 802], | 99.00th=[ 885], 99.50th=[ 919], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 978] bw ( KiB/s): min= 23, max= 255, per=24.91%, avg=112.86, stdev=34.24, samples=180 iops : min= 5, max= 63, avg=28.03, stdev= 8.58, samples=180 lat (msec) : 100=0.62%, 250=0.70%, 500=33.16%, 750=53.95%, 1000=11.56% cpu : usr=0.09%, sys=0.59%, ctx=1499, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=5111: Thu May 1 18:36:15 2025 write: IOPS=28, BW=114KiB/s (116kB/s)(10.0MiB/90150msec) slat (usec): min=15, max=4026, avg=133.60, stdev=278.10 clat (msec): min=76, max=983, avg=562.66, stdev=148.76 lat (msec): min=76, max=983, avg=562.79, stdev=148.78 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 326], 10.00th=[ 372], 20.00th=[ 443], | 30.00th=[ 481], 40.00th=[ 518], 50.00th=[ 558], 60.00th=[ 600], | 70.00th=[ 642], 80.00th=[ 693], 90.00th=[ 760], 95.00th=[ 802], | 99.00th=[ 877], 99.50th=[ 911], 99.90th=[ 961], 99.95th=[ 969], | 99.99th=[ 986] bw ( KiB/s): min= 48, max= 256, per=25.05%, avg=113.47, stdev=35.65, samples=179 iops : min= 12, max= 64, avg=28.17, stdev= 8.93, samples=179 lat (msec) : 100=0.62%, 250=0.62%, 500=34.41%, 750=52.97%, 1000=11.37% cpu : usr=0.13%, sys=0.51%, ctx=1467, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=5112: Thu May 1 18:36:15 2025 write: IOPS=28, BW=113KiB/s (116kB/s)(10.0MiB/90241msec) slat (usec): min=12, max=4992, avg=148.02, stdev=331.58 clat (msec): min=43, max=966, avg=563.39, stdev=148.04 lat (msec): min=45, max=966, avg=563.54, stdev=148.05 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 334], 10.00th=[ 384], 20.00th=[ 447], | 30.00th=[ 485], 40.00th=[ 523], 50.00th=[ 558], 60.00th=[ 600], | 70.00th=[ 642], 80.00th=[ 693], 90.00th=[ 768], 95.00th=[ 810], | 99.00th=[ 911], 99.50th=[ 936], 99.90th=[ 953], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 24, max= 280, per=24.91%, avg=112.83, stdev=35.96, samples=180 iops : min= 6, max= 70, avg=28.00, stdev= 8.98, samples=180 lat (msec) : 50=0.35%, 100=0.16%, 250=0.59%, 500=33.59%, 750=54.10% lat (msec) : 1000=11.21% cpu : usr=0.21%, sys=0.49%, ctx=1507, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=454KiB/s (465kB/s), 113KiB/s-114KiB/s (116kB/s-116kB/s), io=40.0MiB (41.9MB), run=90150-90278msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5194: Thu May 1 18:37:11 2025 read: IOPS=23, BW=93.2KiB/s (95.4kB/s)(5048KiB/54176msec) slat (usec): min=1282, max=16335, avg=4531.29, stdev=1869.74 clat (msec): min=9, max=508, avg=212.39, stdev=46.08 lat (msec): min=13, max=510, avg=216.92, stdev=46.42 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 144], 10.00th=[ 163], 20.00th=[ 178], | 30.00th=[ 190], 40.00th=[ 201], 50.00th=[ 211], 60.00th=[ 222], | 70.00th=[ 232], 80.00th=[ 249], 90.00th=[ 266], 95.00th=[ 288], | 99.00th=[ 326], 99.50th=[ 334], 99.90th=[ 506], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 32, max= 232, per=24.98%, avg=92.44, stdev=35.32, samples=108 iops : min= 8, max= 58, avg=22.90, stdev= 8.85, samples=108 write: IOPS=23, BW=95.8KiB/s (98.1kB/s)(5192KiB/54176msec) slat (usec): min=15, max=3070, avg=150.17, stdev=280.20 clat (msec): min=155, max=848, avg=455.89, stdev=116.09 lat (msec): min=155, max=848, avg=456.05, stdev=116.09 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 271], 10.00th=[ 309], 20.00th=[ 359], | 30.00th=[ 388], 40.00th=[ 422], 50.00th=[ 456], 60.00th=[ 485], | 70.00th=[ 514], 80.00th=[ 550], 90.00th=[ 609], 95.00th=[ 651], | 99.00th=[ 760], 99.50th=[ 785], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 40, max= 160, per=24.71%, avg=95.14, stdev=22.81, samples=108 iops : min= 10, max= 40, avg=23.57, stdev= 5.68, samples=108 lat (msec) : 10=0.04%, 20=0.04%, 50=0.20%, 100=0.12%, 250=41.41% lat (msec) : 500=40.74%, 750=16.76%, 1000=0.70% cpu : usr=0.24%, sys=1.64%, ctx=2723, majf=0, minf=35 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5195: Thu May 1 18:37:11 2025 read: IOPS=23, BW=92.2KiB/s (94.4kB/s)(4976KiB/53960msec) slat (usec): min=23, max=17430, avg=4646.23, stdev=1971.54 clat (msec): min=3, max=415, avg=214.44, stdev=48.16 lat (msec): min=3, max=420, avg=219.10, stdev=48.54 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 144], 10.00th=[ 161], 20.00th=[ 178], | 30.00th=[ 192], 40.00th=[ 203], 50.00th=[ 213], 60.00th=[ 224], | 70.00th=[ 236], 80.00th=[ 253], 90.00th=[ 275], 95.00th=[ 288], | 99.00th=[ 326], 99.50th=[ 342], 99.90th=[ 414], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 31, max= 224, per=24.77%, avg=91.65, stdev=31.93, samples=107 iops : min= 7, max= 56, avg=22.74, stdev= 8.02, samples=107 write: IOPS=24, BW=97.6KiB/s (99.9kB/s)(5264KiB/53960msec) slat (usec): min=14, max=2629, avg=142.70, stdev=244.64 clat (msec): min=49, max=838, avg=448.32, stdev=122.27 lat (msec): min=49, max=838, avg=448.47, stdev=122.27 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 251], 10.00th=[ 300], 20.00th=[ 351], | 30.00th=[ 388], 40.00th=[ 418], 50.00th=[ 443], 60.00th=[ 477], | 70.00th=[ 510], 80.00th=[ 550], 90.00th=[ 600], 95.00th=[ 651], | 99.00th=[ 776], 99.50th=[ 818], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 31, max= 216, per=25.22%, avg=97.10, stdev=28.10, samples=107 iops : min= 7, max= 54, avg=24.10, stdev= 7.03, samples=107 lat (msec) : 4=0.04%, 10=0.04%, 20=0.08%, 50=0.39%, 100=0.43% lat (msec) : 250=39.92%, 500=42.42%, 750=16.09%, 1000=0.59% cpu : usr=0.37%, sys=1.52%, ctx=2685, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5196: Thu May 1 18:37:11 2025 read: IOPS=22, BW=91.9KiB/s (94.1kB/s)(4984KiB/54215msec) slat (usec): min=1299, max=18149, avg=4618.25, stdev=2001.69 clat (msec): min=36, max=376, avg=212.89, stdev=42.61 lat (msec): min=41, max=380, avg=217.52, stdev=42.92 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 148], 10.00th=[ 163], 20.00th=[ 180], | 30.00th=[ 192], 40.00th=[ 203], 50.00th=[ 213], 60.00th=[ 222], | 70.00th=[ 234], 80.00th=[ 249], 90.00th=[ 268], 95.00th=[ 284], | 99.00th=[ 317], 99.50th=[ 326], 99.90th=[ 368], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 15, max= 224, per=24.75%, avg=91.58, stdev=35.83, samples=108 iops : min= 3, max= 56, avg=22.72, stdev= 8.96, samples=108 write: IOPS=24, BW=96.9KiB/s (99.3kB/s)(5256KiB/54215msec) slat (usec): min=16, max=3621, avg=151.29, stdev=262.19 clat (msec): min=153, max=820, avg=452.78, stdev=117.11 lat (msec): min=153, max=821, avg=452.94, stdev=117.13 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 257], 10.00th=[ 305], 20.00th=[ 355], | 30.00th=[ 388], 40.00th=[ 422], 50.00th=[ 451], 60.00th=[ 481], | 70.00th=[ 510], 80.00th=[ 550], 90.00th=[ 600], 95.00th=[ 642], | 99.00th=[ 776], 99.50th=[ 793], 99.90th=[ 793], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 48, max= 191, per=24.94%, avg=96.01, stdev=23.52, samples=108 iops : min= 12, max= 47, avg=23.83, stdev= 5.87, samples=108 lat (msec) : 50=0.23%, 100=0.04%, 250=41.37%, 500=41.33%, 750=16.33% lat (msec) : 1000=0.70% cpu : usr=0.12%, sys=1.76%, ctx=2746, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5197: Thu May 1 18:37:11 2025 read: IOPS=23, BW=94.2KiB/s (96.5kB/s)(5060KiB/53697msec) slat (usec): min=1205, max=16063, avg=4624.91, stdev=1851.30 clat (msec): min=6, max=331, avg=214.81, stdev=44.46 lat (msec): min=10, max=336, avg=219.44, stdev=44.79 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 150], 10.00th=[ 163], 20.00th=[ 180], | 30.00th=[ 194], 40.00th=[ 203], 50.00th=[ 213], 60.00th=[ 226], | 70.00th=[ 236], 80.00th=[ 253], 90.00th=[ 275], 95.00th=[ 288], | 99.00th=[ 317], 99.50th=[ 326], 99.90th=[ 330], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 32, max= 184, per=25.40%, avg=93.99, stdev=29.35, samples=107 iops : min= 8, max= 46, avg=23.28, stdev= 7.37, samples=107 write: IOPS=24, BW=96.5KiB/s (98.8kB/s)(5180KiB/53697msec) slat (usec): min=13, max=2804, avg=158.75, stdev=269.60 clat (msec): min=157, max=792, avg=448.11, stdev=113.55 lat (msec): min=158, max=793, avg=448.27, stdev=113.55 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 264], 10.00th=[ 300], 20.00th=[ 355], | 30.00th=[ 388], 40.00th=[ 418], 50.00th=[ 447], 60.00th=[ 481], | 70.00th=[ 510], 80.00th=[ 542], 90.00th=[ 592], 95.00th=[ 634], | 99.00th=[ 718], 99.50th=[ 751], 99.90th=[ 785], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 47, max= 168, per=24.84%, avg=95.63, stdev=25.49, samples=107 iops : min= 11, max= 42, avg=23.69, stdev= 6.40, samples=107 lat (msec) : 10=0.04%, 20=0.08%, 50=0.16%, 100=0.16%, 250=40.35% lat (msec) : 500=42.93%, 750=15.98%, 1000=0.31% cpu : usr=0.09%, sys=1.86%, ctx=2760, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=370KiB/s (379kB/s), 91.9KiB/s-94.2KiB/s (94.1kB/s-96.5kB/s), io=19.6MiB (20.5MB), run=53697-54215msec WRITE: bw=385KiB/s (395kB/s), 95.8KiB/s-97.6KiB/s (98.1kB/s-99.9kB/s), io=20.4MiB (21.4MB), run=53697-54215msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=5244: Thu May 1 18:37:12 2025 read: IOPS=3, BW=159MiB/s (166MB/s)(40.0MiB/252msec) slat (nsec): min=11536k, max=11536k, avg=11536466.00, stdev= 0.00 clat (nsec): min=231433k, max=231433k, avg=231433433.00, stdev= 0.00 lat (nsec): min=242993k, max=242993k, avg=242993414.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 232], 10.00th=[ 232], 20.00th=[ 232], | 30.00th=[ 232], 40.00th=[ 232], 50.00th=[ 232], 60.00th=[ 232], | 70.00th=[ 232], 80.00th=[ 232], 90.00th=[ 232], 95.00th=[ 232], | 99.00th=[ 232], 99.50th=[ 232], 99.90th=[ 232], 99.95th=[ 232], | 99.99th=[ 232] lat (msec) : 250=100.00% cpu : usr=0.80%, sys=2.79%, ctx=36, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=159MiB/s (166MB/s), 159MiB/s-159MiB/s (166MB/s-166MB/s), io=40.0MiB (41.9MB), run=252-252msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout