== sanity test 398c: run fio to test AIO ================= 19:54:46 (1746143686) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.169402 s, 248 MB/s osc.lustre-OST0000-osc-ffff88012fd03800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=3870: Thu May 1 19:56:31 2025 write: IOPS=24, BW=99.0KiB/s (101kB/s)(10.0MiB/103412msec) slat (usec): min=11, max=5159, avg=146.86, stdev=359.06 clat (msec): min=121, max=1220, avg=645.52, stdev=151.62 lat (msec): min=121, max=1220, avg=645.67, stdev=151.65 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 368], 10.00th=[ 464], 20.00th=[ 518], | 30.00th=[ 575], 40.00th=[ 609], 50.00th=[ 659], 60.00th=[ 693], | 70.00th=[ 726], 80.00th=[ 768], 90.00th=[ 827], 95.00th=[ 877], | 99.00th=[ 986], 99.50th=[ 1099], 99.90th=[ 1167], 99.95th=[ 1183], | 99.99th=[ 1217] bw ( KiB/s): min= 16, max= 256, per=24.77%, avg=98.10, stdev=33.20, samples=206 iops : min= 4, max= 64, avg=24.30, stdev= 8.27, samples=206 lat (msec) : 250=0.70%, 500=16.37%, 750=58.59%, 1000=23.59% cpu : usr=0.15%, sys=0.44%, ctx=1374, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=3871: Thu May 1 19:56:31 2025 write: IOPS=24, BW=99.0KiB/s (101kB/s)(10.0MiB/103385msec) slat (usec): min=11, max=13496, avg=155.47, stdev=444.33 clat (msec): min=107, max=1221, avg=645.27, stdev=153.90 lat (msec): min=107, max=1221, avg=645.43, stdev=153.89 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 380], 10.00th=[ 460], 20.00th=[ 518], | 30.00th=[ 567], 40.00th=[ 617], 50.00th=[ 659], 60.00th=[ 693], | 70.00th=[ 735], 80.00th=[ 776], 90.00th=[ 818], 95.00th=[ 877], | 99.00th=[ 995], 99.50th=[ 1099], 99.90th=[ 1167], 99.95th=[ 1183], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 256, per=24.85%, avg=98.41, stdev=31.53, samples=206 iops : min= 1, max= 64, avg=24.34, stdev= 7.94, samples=206 lat (msec) : 250=1.25%, 500=15.78%, 750=58.55%, 1000=23.59% cpu : usr=0.07%, sys=0.57%, ctx=1494, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=3872: Thu May 1 19:56:31 2025 write: IOPS=24, BW=99.1KiB/s (102kB/s)(10.0MiB/103292msec) slat (usec): min=12, max=4362, avg=135.65, stdev=313.78 clat (msec): min=51, max=1220, avg=644.95, stdev=153.56 lat (msec): min=52, max=1220, avg=645.09, stdev=153.57 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 359], 10.00th=[ 468], 20.00th=[ 518], | 30.00th=[ 575], 40.00th=[ 617], 50.00th=[ 659], 60.00th=[ 693], | 70.00th=[ 726], 80.00th=[ 776], 90.00th=[ 827], 95.00th=[ 869], | 99.00th=[ 995], 99.50th=[ 1053], 99.90th=[ 1183], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 16, max= 256, per=24.87%, avg=98.49, stdev=34.57, samples=206 iops : min= 4, max= 64, avg=24.36, stdev= 8.65, samples=206 lat (msec) : 100=0.04%, 250=1.17%, 500=14.49%, 750=60.12%, 1000=23.32% cpu : usr=0.17%, sys=0.41%, ctx=1360, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=3873: Thu May 1 19:56:31 2025 write: IOPS=24, BW=99.0KiB/s (101kB/s)(10.0MiB/103386msec) slat (usec): min=13, max=5928, avg=157.09, stdev=370.07 clat (msec): min=106, max=1240, avg=645.31, stdev=155.64 lat (msec): min=107, max=1240, avg=645.47, stdev=155.65 clat percentiles (msec): | 1.00th=[ 262], 5.00th=[ 363], 10.00th=[ 456], 20.00th=[ 514], | 30.00th=[ 567], 40.00th=[ 617], 50.00th=[ 651], 60.00th=[ 693], | 70.00th=[ 735], 80.00th=[ 776], 90.00th=[ 827], 95.00th=[ 869], | 99.00th=[ 995], 99.50th=[ 1099], 99.90th=[ 1200], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 7, max= 248, per=24.83%, avg=98.33, stdev=32.51, samples=206 iops : min= 1, max= 62, avg=24.32, stdev= 8.17, samples=206 lat (msec) : 250=0.98%, 500=16.76%, 750=57.19%, 1000=24.30% cpu : usr=0.17%, sys=0.45%, ctx=1481, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=396KiB/s (406kB/s), 99.0KiB/s-99.1KiB/s (101kB/s-102kB/s), io=40.0MiB (41.9MB), run=103292-103412msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=3966: Thu May 1 19:57:32 2025 read: IOPS=21, BW=85.3KiB/s (87.3kB/s)(5048KiB/59189msec) slat (usec): min=24, max=18061, avg=5156.88, stdev=2341.26 clat (msec): min=3, max=764, avg=245.05, stdev=65.18 lat (msec): min=9, max=774, avg=250.21, stdev=65.82 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 159], 10.00th=[ 178], 20.00th=[ 199], | 30.00th=[ 215], 40.00th=[ 230], 50.00th=[ 245], 60.00th=[ 257], | 70.00th=[ 275], 80.00th=[ 292], 90.00th=[ 313], 95.00th=[ 342], | 99.00th=[ 388], 99.50th=[ 414], 99.90th=[ 735], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 16, max= 160, per=25.25%, avg=85.08, stdev=32.24, samples=118 iops : min= 4, max= 40, avg=21.12, stdev= 8.05, samples=118 write: IOPS=21, BW=87.7KiB/s (89.8kB/s)(5192KiB/59189msec) slat (usec): min=15, max=3252, avg=150.80, stdev=293.05 clat (msec): min=60, max=1206, avg=485.53, stdev=144.37 lat (msec): min=60, max=1206, avg=485.68, stdev=144.38 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 259], 10.00th=[ 309], 20.00th=[ 359], | 30.00th=[ 401], 40.00th=[ 443], 50.00th=[ 477], 60.00th=[ 518], | 70.00th=[ 550], 80.00th=[ 617], 90.00th=[ 676], 95.00th=[ 718], | 99.00th=[ 818], 99.50th=[ 894], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 32, max= 160, per=25.01%, avg=87.79, stdev=26.69, samples=117 iops : min= 8, max= 40, avg=21.79, stdev= 6.71, samples=117 lat (msec) : 4=0.04%, 10=0.12%, 20=0.04%, 50=0.31%, 100=0.39% lat (msec) : 250=27.62%, 500=49.06%, 750=20.82%, 1000=1.41% cpu : usr=0.24%, sys=1.54%, ctx=2706, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3967: Thu May 1 19:57:32 2025 read: IOPS=20, BW=83.7KiB/s (85.7kB/s)(4976KiB/59435msec) slat (usec): min=1297, max=20319, avg=5274.86, stdev=2401.15 clat (msec): min=6, max=546, avg=246.06, stdev=56.48 lat (msec): min=11, max=548, avg=251.34, stdev=57.05 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 161], 10.00th=[ 180], 20.00th=[ 201], | 30.00th=[ 218], 40.00th=[ 232], 50.00th=[ 247], 60.00th=[ 259], | 70.00th=[ 275], 80.00th=[ 292], 90.00th=[ 317], 95.00th=[ 330], | 99.00th=[ 380], 99.50th=[ 388], 99.90th=[ 489], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 24, max= 152, per=24.88%, avg=83.84, stdev=28.45, samples=118 iops : min= 6, max= 38, avg=20.74, stdev= 7.11, samples=118 write: IOPS=22, BW=88.6KiB/s (90.7kB/s)(5264KiB/59435msec) slat (usec): min=13, max=4758, avg=163.61, stdev=311.28 clat (msec): min=176, max=1106, avg=483.99, stdev=135.81 lat (msec): min=177, max=1106, avg=484.15, stdev=135.82 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 271], 10.00th=[ 313], 20.00th=[ 368], | 30.00th=[ 409], 40.00th=[ 443], 50.00th=[ 477], 60.00th=[ 510], | 70.00th=[ 550], 80.00th=[ 600], 90.00th=[ 659], 95.00th=[ 718], | 99.00th=[ 810], 99.50th=[ 877], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 39, max= 160, per=25.10%, avg=88.11, stdev=28.63, samples=118 iops : min= 9, max= 40, avg=21.81, stdev= 7.23, samples=118 lat (msec) : 10=0.04%, 20=0.04%, 50=0.20%, 100=0.31%, 250=26.80% lat (msec) : 500=50.20%, 750=20.94%, 1000=1.33% cpu : usr=0.11%, sys=1.71%, ctx=2725, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3968: Thu May 1 19:57:32 2025 read: IOPS=20, BW=83.7KiB/s (85.8kB/s)(4984KiB/59517msec) slat (usec): min=1284, max=19557, avg=5105.87, stdev=2212.31 clat (msec): min=6, max=510, avg=243.96, stdev=56.21 lat (msec): min=12, max=513, avg=249.08, stdev=56.76 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 157], 10.00th=[ 178], 20.00th=[ 201], | 30.00th=[ 218], 40.00th=[ 230], 50.00th=[ 245], 60.00th=[ 257], | 70.00th=[ 271], 80.00th=[ 292], 90.00th=[ 313], 95.00th=[ 334], | 99.00th=[ 380], 99.50th=[ 397], 99.90th=[ 506], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 15, max= 208, per=24.84%, avg=83.70, stdev=34.19, samples=118 iops : min= 3, max= 52, avg=20.70, stdev= 8.62, samples=118 write: IOPS=22, BW=88.3KiB/s (90.4kB/s)(5256KiB/59517msec) slat (usec): min=17, max=6988, avg=164.81, stdev=351.77 clat (msec): min=81, max=848, avg=487.59, stdev=132.82 lat (msec): min=81, max=848, avg=487.76, stdev=132.83 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 271], 10.00th=[ 321], 20.00th=[ 372], | 30.00th=[ 418], 40.00th=[ 447], 50.00th=[ 481], 60.00th=[ 518], | 70.00th=[ 558], 80.00th=[ 609], 90.00th=[ 667], 95.00th=[ 718], | 99.00th=[ 785], 99.50th=[ 802], 99.90th=[ 827], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 23, max= 152, per=24.87%, avg=87.30, stdev=25.72, samples=118 iops : min= 5, max= 38, avg=21.60, stdev= 6.48, samples=118 lat (msec) : 10=0.04%, 20=0.12%, 50=0.16%, 100=0.12%, 250=27.97% lat (msec) : 500=48.44%, 750=21.91%, 1000=1.25% cpu : usr=0.14%, sys=1.66%, ctx=2707, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3969: Thu May 1 19:57:32 2025 read: IOPS=21, BW=85.3KiB/s (87.4kB/s)(5060KiB/59305msec) slat (usec): min=1207, max=34947, avg=5283.00, stdev=2577.95 clat (msec): min=9, max=787, avg=248.33, stdev=63.46 lat (msec): min=13, max=791, avg=253.61, stdev=63.97 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 161], 10.00th=[ 178], 20.00th=[ 201], | 30.00th=[ 220], 40.00th=[ 234], 50.00th=[ 247], 60.00th=[ 259], | 70.00th=[ 275], 80.00th=[ 292], 90.00th=[ 317], 95.00th=[ 338], | 99.00th=[ 397], 99.50th=[ 414], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 31, max= 168, per=25.30%, avg=85.25, stdev=30.12, samples=118 iops : min= 7, max= 42, avg=21.13, stdev= 7.56, samples=118 write: IOPS=21, BW=87.3KiB/s (89.4kB/s)(5180KiB/59305msec) slat (usec): min=15, max=13406, avg=159.80, stdev=446.53 clat (msec): min=167, max=1150, avg=483.54, stdev=138.48 lat (msec): min=167, max=1150, avg=483.70, stdev=138.50 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 264], 10.00th=[ 317], 20.00th=[ 368], | 30.00th=[ 405], 40.00th=[ 439], 50.00th=[ 472], 60.00th=[ 510], | 70.00th=[ 550], 80.00th=[ 600], 90.00th=[ 659], 95.00th=[ 709], | 99.00th=[ 793], 99.50th=[ 1053], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 31, max= 176, per=24.73%, avg=86.81, stdev=27.24, samples=118 iops : min= 7, max= 44, avg=21.52, stdev= 6.88, samples=118 lat (msec) : 10=0.04%, 20=0.08%, 50=0.20%, 250=27.23%, 500=50.51% lat (msec) : 750=20.59%, 1000=1.09% cpu : usr=0.31%, sys=1.50%, ctx=2715, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=337KiB/s (345kB/s), 83.7KiB/s-85.3KiB/s (85.7kB/s-87.4kB/s), io=19.6MiB (20.5MB), run=59189-59517msec WRITE: bw=351KiB/s (359kB/s), 87.3KiB/s-88.6KiB/s (89.4kB/s-90.7kB/s), io=20.4MiB (21.4MB), run=59189-59517msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=4021: Thu May 1 19:57:34 2025 read: IOPS=3, BW=149MiB/s (156MB/s)(40.0MiB/269msec) slat (nsec): min=6225.3k, max=6225.3k, avg=6225308.00, stdev= 0.00 clat (nsec): min=237158k, max=237158k, avg=237158373.00, stdev= 0.00 lat (nsec): min=243412k, max=243412k, avg=243411526.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 239], 10.00th=[ 239], 20.00th=[ 239], | 30.00th=[ 239], 40.00th=[ 239], 50.00th=[ 239], 60.00th=[ 239], | 70.00th=[ 239], 80.00th=[ 239], 90.00th=[ 239], 95.00th=[ 239], | 99.00th=[ 239], 99.50th=[ 239], 99.90th=[ 239], 99.95th=[ 239], | 99.99th=[ 239] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=1.49%, ctx=26, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=149MiB/s (156MB/s), 149MiB/s-149MiB/s (156MB/s-156MB/s), io=40.0MiB (41.9MB), run=269-269msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout