== sanity test 398c: run fio to test AIO ================= 03:14:34 (1746083674) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.160166 s, 262 MB/s osc.lustre-OST0000-osc-ffff8800875ee800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=2960: Thu May 1 03:16:08 2025 write: IOPS=27, BW=111KiB/s (114kB/s)(10.0MiB/92247msec) slat (usec): min=11, max=4940, avg=125.91, stdev=274.14 clat (msec): min=205, max=1048, avg=575.84, stdev=133.71 lat (msec): min=205, max=1053, avg=575.97, stdev=133.72 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 376], 10.00th=[ 422], 20.00th=[ 472], | 30.00th=[ 502], 40.00th=[ 542], 50.00th=[ 567], 60.00th=[ 592], | 70.00th=[ 634], 80.00th=[ 684], 90.00th=[ 751], 95.00th=[ 818], | 99.00th=[ 919], 99.50th=[ 978], 99.90th=[ 1020], 99.95th=[ 1020], | 99.99th=[ 1045] bw ( KiB/s): min= 39, max= 255, per=24.85%, avg=110.32, stdev=27.80, samples=184 iops : min= 9, max= 63, avg=27.34, stdev= 7.00, samples=184 lat (msec) : 250=0.62%, 500=27.97%, 750=61.45%, 1000=9.80% cpu : usr=0.11%, sys=0.51%, ctx=1445, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=2961: Thu May 1 03:16:08 2025 write: IOPS=27, BW=111KiB/s (114kB/s)(10.0MiB/92246msec) slat (usec): min=11, max=19677, avg=136.36, stdev=482.62 clat (msec): min=205, max=1054, avg=575.83, stdev=135.85 lat (msec): min=205, max=1054, avg=575.97, stdev=135.86 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 376], 10.00th=[ 426], 20.00th=[ 472], | 30.00th=[ 506], 40.00th=[ 542], 50.00th=[ 567], 60.00th=[ 592], | 70.00th=[ 625], 80.00th=[ 676], 90.00th=[ 751], 95.00th=[ 827], | 99.00th=[ 961], 99.50th=[ 986], 99.90th=[ 1036], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 40, max= 256, per=24.87%, avg=110.42, stdev=30.73, samples=184 iops : min= 10, max= 64, avg=27.43, stdev= 7.71, samples=184 lat (msec) : 250=0.66%, 500=28.40%, 750=60.43%, 1000=10.12% cpu : usr=0.14%, sys=0.48%, ctx=1369, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=2962: Thu May 1 03:16:08 2025 write: IOPS=27, BW=111KiB/s (114kB/s)(10.0MiB/92168msec) slat (usec): min=11, max=6570, avg=131.95, stdev=310.26 clat (msec): min=53, max=1214, avg=575.54, stdev=139.87 lat (msec): min=53, max=1214, avg=575.67, stdev=139.89 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 368], 10.00th=[ 422], 20.00th=[ 472], | 30.00th=[ 506], 40.00th=[ 535], 50.00th=[ 567], 60.00th=[ 600], | 70.00th=[ 634], 80.00th=[ 684], 90.00th=[ 760], 95.00th=[ 810], | 99.00th=[ 944], 99.50th=[ 986], 99.90th=[ 1167], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 32, max= 256, per=24.83%, avg=110.26, stdev=32.91, samples=184 iops : min= 8, max= 64, avg=27.28, stdev= 8.27, samples=184 lat (msec) : 100=0.12%, 250=1.17%, 500=27.07%, 750=60.70%, 1000=10.70% cpu : usr=0.08%, sys=0.56%, ctx=1437, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=2963: Thu May 1 03:16:08 2025 write: IOPS=27, BW=111KiB/s (114kB/s)(10.0MiB/92167msec) slat (usec): min=14, max=19269, avg=136.51, stdev=473.92 clat (msec): min=183, max=1450, avg=575.36, stdev=140.90 lat (msec): min=183, max=1450, avg=575.50, stdev=140.90 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 376], 10.00th=[ 422], 20.00th=[ 468], | 30.00th=[ 506], 40.00th=[ 535], 50.00th=[ 558], 60.00th=[ 592], | 70.00th=[ 634], 80.00th=[ 676], 90.00th=[ 751], 95.00th=[ 818], | 99.00th=[ 953], 99.50th=[ 1003], 99.90th=[ 1385], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 23, max= 256, per=24.97%, avg=110.87, stdev=34.97, samples=183 iops : min= 5, max= 64, avg=27.44, stdev= 8.81, samples=183 lat (msec) : 250=1.25%, 500=27.07%, 750=61.29%, 1000=9.92% cpu : usr=0.08%, sys=0.54%, ctx=1436, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=444KiB/s (455kB/s), 111KiB/s-111KiB/s (114kB/s-114kB/s), io=40.0MiB (41.9MB), run=92167-92247msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=3047: Thu May 1 03:17:06 2025 read: IOPS=22, BW=89.7KiB/s (91.9kB/s)(5048KiB/56267msec) slat (usec): min=1055, max=27307, avg=5133.56, stdev=2473.30 clat (msec): min=7, max=481, avg=236.35, stdev=53.54 lat (msec): min=10, max=486, avg=241.49, stdev=54.09 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 161], 10.00th=[ 174], 20.00th=[ 192], | 30.00th=[ 209], 40.00th=[ 224], 50.00th=[ 234], 60.00th=[ 247], | 70.00th=[ 262], 80.00th=[ 279], 90.00th=[ 300], 95.00th=[ 330], | 99.00th=[ 372], 99.50th=[ 401], 99.90th=[ 443], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 32, max= 160, per=25.19%, avg=89.44, stdev=30.49, samples=112 iops : min= 8, max= 40, avg=22.15, stdev= 7.63, samples=112 write: IOPS=23, BW=92.3KiB/s (94.5kB/s)(5192KiB/56267msec) slat (usec): min=15, max=5441, avg=156.48, stdev=347.00 clat (msec): min=165, max=809, avg=457.06, stdev=117.52 lat (msec): min=165, max=809, avg=457.22, stdev=117.54 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 268], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 388], 40.00th=[ 418], 50.00th=[ 451], 60.00th=[ 489], | 70.00th=[ 523], 80.00th=[ 558], 90.00th=[ 609], 95.00th=[ 659], | 99.00th=[ 726], 99.50th=[ 793], 99.90th=[ 802], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 47, max= 191, per=24.81%, avg=91.79, stdev=25.64, samples=112 iops : min= 11, max= 47, avg=22.74, stdev= 6.39, samples=112 lat (msec) : 10=0.08%, 20=0.08%, 50=0.16%, 100=0.12%, 250=31.76% lat (msec) : 500=49.26%, 750=18.24%, 1000=0.31% cpu : usr=0.13%, sys=1.68%, ctx=2738, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3048: Thu May 1 03:17:06 2025 read: IOPS=22, BW=88.3KiB/s (90.4kB/s)(4976KiB/56343msec) slat (usec): min=17, max=25140, avg=5165.37, stdev=2445.23 clat (msec): min=10, max=554, avg=234.02, stdev=54.99 lat (msec): min=11, max=559, avg=239.19, stdev=55.54 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 155], 10.00th=[ 171], 20.00th=[ 188], | 30.00th=[ 205], 40.00th=[ 222], 50.00th=[ 232], 60.00th=[ 245], | 70.00th=[ 259], 80.00th=[ 279], 90.00th=[ 305], 95.00th=[ 326], | 99.00th=[ 372], 99.50th=[ 376], 99.90th=[ 550], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 15, max= 199, per=24.71%, avg=87.71, stdev=32.15, samples=112 iops : min= 3, max= 49, avg=21.71, stdev= 8.03, samples=112 write: IOPS=23, BW=93.4KiB/s (95.7kB/s)(5264KiB/56343msec) slat (usec): min=16, max=3124, avg=152.41, stdev=291.76 clat (msec): min=86, max=798, avg=458.34, stdev=119.73 lat (msec): min=86, max=798, avg=458.49, stdev=119.72 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 271], 10.00th=[ 317], 20.00th=[ 359], | 30.00th=[ 388], 40.00th=[ 422], 50.00th=[ 451], 60.00th=[ 481], | 70.00th=[ 518], 80.00th=[ 567], 90.00th=[ 625], 95.00th=[ 667], | 99.00th=[ 735], 99.50th=[ 760], 99.90th=[ 785], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 32, max= 176, per=25.09%, avg=92.85, stdev=24.80, samples=112 iops : min= 8, max= 44, avg=22.99, stdev= 6.19, samples=112 lat (msec) : 20=0.04%, 50=0.27%, 100=0.23%, 250=32.30%, 500=48.98% lat (msec) : 750=17.85%, 1000=0.31% cpu : usr=0.17%, sys=1.61%, ctx=2715, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3049: Thu May 1 03:17:06 2025 read: IOPS=22, BW=88.3KiB/s (90.4kB/s)(4984KiB/56455msec) slat (usec): min=1430, max=21742, avg=5160.65, stdev=2367.24 clat (msec): min=9, max=423, avg=235.24, stdev=53.71 lat (msec): min=14, max=428, avg=240.41, stdev=54.20 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 157], 10.00th=[ 171], 20.00th=[ 190], | 30.00th=[ 207], 40.00th=[ 222], 50.00th=[ 234], 60.00th=[ 245], | 70.00th=[ 259], 80.00th=[ 279], 90.00th=[ 305], 95.00th=[ 330], | 99.00th=[ 368], 99.50th=[ 376], 99.90th=[ 388], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 24, max= 200, per=24.75%, avg=87.86, stdev=34.60, samples=112 iops : min= 6, max= 50, avg=21.75, stdev= 8.75, samples=112 write: IOPS=23, BW=93.1KiB/s (95.3kB/s)(5256KiB/56455msec) slat (usec): min=15, max=4071, avg=141.45, stdev=274.17 clat (msec): min=113, max=840, avg=458.18, stdev=118.70 lat (msec): min=113, max=841, avg=458.33, stdev=118.71 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 284], 10.00th=[ 313], 20.00th=[ 355], | 30.00th=[ 388], 40.00th=[ 426], 50.00th=[ 451], 60.00th=[ 481], | 70.00th=[ 514], 80.00th=[ 567], 90.00th=[ 625], 95.00th=[ 667], | 99.00th=[ 743], 99.50th=[ 768], 99.90th=[ 835], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 40, max= 160, per=24.83%, avg=91.86, stdev=24.45, samples=112 iops : min= 10, max= 40, avg=22.75, stdev= 6.19, samples=112 lat (msec) : 10=0.04%, 20=0.04%, 50=0.20%, 100=0.16%, 250=32.15% lat (msec) : 500=50.23%, 750=16.76%, 1000=0.43% cpu : usr=0.07%, sys=1.65%, ctx=2639, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3050: Thu May 1 03:17:06 2025 read: IOPS=22, BW=89.9KiB/s (92.1kB/s)(5060KiB/56267msec) slat (usec): min=1371, max=23497, avg=5140.66, stdev=2276.15 clat (msec): min=3, max=560, avg=239.31, stdev=57.44 lat (msec): min=5, max=563, avg=244.46, stdev=57.83 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 157], 10.00th=[ 176], 20.00th=[ 194], | 30.00th=[ 211], 40.00th=[ 226], 50.00th=[ 236], 60.00th=[ 249], | 70.00th=[ 264], 80.00th=[ 279], 90.00th=[ 309], 95.00th=[ 338], | 99.00th=[ 388], 99.50th=[ 422], 99.90th=[ 527], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 8, max= 168, per=25.24%, avg=89.61, stdev=30.15, samples=112 iops : min= 2, max= 42, avg=22.21, stdev= 7.54, samples=112 write: IOPS=23, BW=92.1KiB/s (94.3kB/s)(5180KiB/56267msec) slat (usec): min=14, max=4657, avg=145.53, stdev=293.95 clat (msec): min=51, max=811, avg=455.10, stdev=118.67 lat (msec): min=51, max=811, avg=455.25, stdev=118.67 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 271], 10.00th=[ 317], 20.00th=[ 355], | 30.00th=[ 388], 40.00th=[ 414], 50.00th=[ 447], 60.00th=[ 477], | 70.00th=[ 514], 80.00th=[ 558], 90.00th=[ 617], 95.00th=[ 659], | 99.00th=[ 751], 99.50th=[ 785], 99.90th=[ 810], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 40, max= 152, per=24.78%, avg=91.68, stdev=23.74, samples=112 iops : min= 10, max= 38, avg=22.73, stdev= 5.99, samples=112 lat (msec) : 4=0.04%, 10=0.04%, 20=0.04%, 50=0.23%, 100=0.08% lat (msec) : 250=31.56%, 500=50.74%, 750=16.72%, 1000=0.55% cpu : usr=0.31%, sys=1.46%, ctx=2764, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=355KiB/s (364kB/s), 88.3KiB/s-89.9KiB/s (90.4kB/s-92.1kB/s), io=19.6MiB (20.5MB), run=56267-56455msec WRITE: bw=370KiB/s (379kB/s), 92.1KiB/s-93.4KiB/s (94.3kB/s-95.7kB/s), io=20.4MiB (21.4MB), run=56267-56455msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=3099: Thu May 1 03:17:08 2025 read: IOPS=2, BW=114MiB/s (119MB/s)(40.0MiB/351msec) slat (nsec): min=13290k, max=13290k, avg=13290047.00, stdev= 0.00 clat (nsec): min=326470k, max=326470k, avg=326470053.00, stdev= 0.00 lat (nsec): min=339783k, max=339783k, avg=339782563.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 326], 10.00th=[ 326], 20.00th=[ 326], | 30.00th=[ 326], 40.00th=[ 326], 50.00th=[ 326], 60.00th=[ 326], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] lat (msec) : 500=100.00% cpu : usr=0.57%, sys=2.29%, ctx=19, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=114MiB/s (119MB/s), 114MiB/s-114MiB/s (119MB/s-119MB/s), io=40.0MiB (41.9MB), run=351-351msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout