== sanity test 398c: run fio to test AIO ================= 00:32:20 (1746160340) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.273547 s, 153 MB/s osc.lustre-OST0000-osc-ffff88012d513000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=5172: Fri May 2 00:34:15 2025 write: IOPS=23, BW=92.0KiB/s (94.2kB/s)(10.0MiB/111293msec) slat (usec): min=13, max=4013, avg=146.23, stdev=302.45 clat (msec): min=229, max=1165, avg=694.41, stdev=149.98 lat (msec): min=229, max=1166, avg=694.56, stdev=149.99 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 464], 10.00th=[ 506], 20.00th=[ 575], | 30.00th=[ 609], 40.00th=[ 651], 50.00th=[ 693], 60.00th=[ 726], | 70.00th=[ 768], 80.00th=[ 810], 90.00th=[ 894], 95.00th=[ 953], | 99.00th=[ 1070], 99.50th=[ 1099], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1167] bw ( KiB/s): min= 8, max= 167, per=24.80%, avg=91.27, stdev=30.12, samples=222 iops : min= 2, max= 41, avg=22.54, stdev= 7.55, samples=222 lat (msec) : 250=0.04%, 500=9.22%, 750=55.82%, 1000=31.64% cpu : usr=0.08%, sys=0.50%, ctx=1404, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=5173: Fri May 2 00:34:15 2025 write: IOPS=23, BW=92.0KiB/s (94.3kB/s)(10.0MiB/111249msec) slat (usec): min=15, max=5283, avg=160.23, stdev=351.07 clat (msec): min=189, max=1140, avg=694.42, stdev=149.96 lat (msec): min=189, max=1140, avg=694.58, stdev=149.96 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 460], 10.00th=[ 506], 20.00th=[ 575], | 30.00th=[ 617], 40.00th=[ 651], 50.00th=[ 684], 60.00th=[ 726], | 70.00th=[ 760], 80.00th=[ 810], 90.00th=[ 902], 95.00th=[ 978], | 99.00th=[ 1045], 99.50th=[ 1062], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1133] bw ( KiB/s): min= 7, max= 183, per=25.06%, avg=92.20, stdev=31.96, samples=220 iops : min= 1, max= 45, avg=22.80, stdev= 7.99, samples=220 lat (msec) : 250=0.04%, 500=9.49%, 750=57.27%, 1000=29.77% cpu : usr=0.09%, sys=0.51%, ctx=1356, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=5174: Fri May 2 00:34:15 2025 write: IOPS=23, BW=92.0KiB/s (94.2kB/s)(10.0MiB/111292msec) slat (usec): min=14, max=5605, avg=159.68, stdev=340.52 clat (msec): min=169, max=1158, avg=694.31, stdev=151.21 lat (msec): min=169, max=1158, avg=694.48, stdev=151.22 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 460], 10.00th=[ 514], 20.00th=[ 567], | 30.00th=[ 617], 40.00th=[ 651], 50.00th=[ 693], 60.00th=[ 726], | 70.00th=[ 768], 80.00th=[ 810], 90.00th=[ 894], 95.00th=[ 961], | 99.00th=[ 1053], 99.50th=[ 1083], 99.90th=[ 1150], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 8, max= 160, per=25.19%, avg=92.69, stdev=27.48, samples=219 iops : min= 2, max= 40, avg=22.96, stdev= 6.87, samples=219 lat (msec) : 250=0.08%, 500=8.59%, 750=57.42%, 1000=30.51% cpu : usr=0.08%, sys=0.53%, ctx=1409, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=5175: Fri May 2 00:34:15 2025 write: IOPS=23, BW=92.1KiB/s (94.4kB/s)(10.0MiB/111127msec) slat (usec): min=13, max=5644, avg=173.07, stdev=367.77 clat (msec): min=83, max=1191, avg=693.69, stdev=154.31 lat (msec): min=83, max=1191, avg=693.86, stdev=154.31 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 468], 10.00th=[ 510], 20.00th=[ 567], | 30.00th=[ 609], 40.00th=[ 651], 50.00th=[ 693], 60.00th=[ 726], | 70.00th=[ 768], 80.00th=[ 810], 90.00th=[ 894], 95.00th=[ 969], | 99.00th=[ 1083], 99.50th=[ 1116], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 15, max= 183, per=24.83%, avg=91.38, stdev=28.76, samples=222 iops : min= 3, max= 45, avg=22.60, stdev= 7.20, samples=222 lat (msec) : 100=0.12%, 250=0.27%, 500=8.52%, 750=57.03%, 1000=30.59% cpu : usr=0.14%, sys=0.51%, ctx=1508, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=368KiB/s (377kB/s), 92.0KiB/s-92.1KiB/s (94.2kB/s-94.4kB/s), io=40.0MiB (41.9MB), run=111127-111293msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5274: Fri May 2 00:35:21 2025 read: IOPS=19, BW=77.2KiB/s (79.0kB/s)(5048KiB/65430msec) slat (usec): min=19, max=30599, avg=5843.36, stdev=2938.30 clat (msec): min=7, max=751, avg=264.32, stdev=70.48 lat (msec): min=9, max=758, avg=270.17, stdev=71.35 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 169], 10.00th=[ 188], 20.00th=[ 211], | 30.00th=[ 226], 40.00th=[ 243], 50.00th=[ 259], 60.00th=[ 275], | 70.00th=[ 296], 80.00th=[ 317], 90.00th=[ 355], 95.00th=[ 380], | 99.00th=[ 456], 99.50th=[ 498], 99.90th=[ 634], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 15, max= 175, per=25.19%, avg=77.08, stdev=29.88, samples=130 iops : min= 3, max= 43, avg=19.07, stdev= 7.48, samples=130 write: IOPS=19, BW=79.4KiB/s (81.3kB/s)(5192KiB/65430msec) slat (usec): min=15, max=5886, avg=181.36, stdev=394.15 clat (msec): min=77, max=1057, avg=543.01, stdev=151.64 lat (msec): min=77, max=1057, avg=543.19, stdev=151.65 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 305], 10.00th=[ 359], 20.00th=[ 418], | 30.00th=[ 464], 40.00th=[ 498], 50.00th=[ 535], 60.00th=[ 575], | 70.00th=[ 617], 80.00th=[ 667], 90.00th=[ 743], 95.00th=[ 793], | 99.00th=[ 969], 99.50th=[ 986], 99.90th=[ 1045], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 23, max= 160, per=24.84%, avg=78.99, stdev=26.18, samples=130 iops : min= 5, max= 40, avg=19.55, stdev= 6.58, samples=130 lat (msec) : 10=0.04%, 20=0.08%, 50=0.08%, 100=0.74%, 250=22.19% lat (msec) : 500=46.95%, 750=25.31%, 1000=4.45% cpu : usr=0.35%, sys=1.40%, ctx=2812, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5275: Fri May 2 00:35:21 2025 read: IOPS=18, BW=75.9KiB/s (77.8kB/s)(4976KiB/65531msec) slat (usec): min=1782, max=34790, avg=5875.38, stdev=2847.62 clat (msec): min=4, max=509, avg=262.67, stdev=63.99 lat (msec): min=7, max=521, avg=268.55, stdev=64.93 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 176], 10.00th=[ 192], 20.00th=[ 213], | 30.00th=[ 230], 40.00th=[ 245], 50.00th=[ 257], 60.00th=[ 275], | 70.00th=[ 292], 80.00th=[ 313], 90.00th=[ 347], 95.00th=[ 372], | 99.00th=[ 418], 99.50th=[ 447], 99.90th=[ 472], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 8, max= 176, per=24.65%, avg=75.42, stdev=30.71, samples=130 iops : min= 2, max= 44, avg=18.69, stdev= 7.65, samples=130 write: IOPS=20, BW=80.3KiB/s (82.3kB/s)(5264KiB/65531msec) slat (usec): min=16, max=3257, avg=155.43, stdev=278.81 clat (msec): min=109, max=1041, avg=541.55, stdev=150.95 lat (msec): min=110, max=1041, avg=541.71, stdev=150.96 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 309], 10.00th=[ 359], 20.00th=[ 409], | 30.00th=[ 460], 40.00th=[ 502], 50.00th=[ 542], 60.00th=[ 575], | 70.00th=[ 609], 80.00th=[ 667], 90.00th=[ 743], 95.00th=[ 793], | 99.00th=[ 961], 99.50th=[ 995], 99.90th=[ 1036], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 24, max= 144, per=25.13%, avg=79.91, stdev=25.19, samples=130 iops : min= 6, max= 36, avg=19.82, stdev= 6.30, samples=130 lat (msec) : 10=0.08%, 20=0.08%, 50=0.35%, 100=0.04%, 250=22.30% lat (msec) : 500=46.09%, 750=26.41%, 1000=4.49% cpu : usr=0.26%, sys=1.49%, ctx=2758, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5276: Fri May 2 00:35:21 2025 read: IOPS=19, BW=76.1KiB/s (77.9kB/s)(4984KiB/65528msec) slat (usec): min=1645, max=33865, avg=5928.92, stdev=3067.38 clat (msec): min=6, max=644, avg=263.72, stdev=67.32 lat (msec): min=8, max=649, avg=269.65, stdev=68.21 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 169], 10.00th=[ 190], 20.00th=[ 215], | 30.00th=[ 230], 40.00th=[ 245], 50.00th=[ 259], 60.00th=[ 275], | 70.00th=[ 292], 80.00th=[ 317], 90.00th=[ 351], 95.00th=[ 372], | 99.00th=[ 435], 99.50th=[ 493], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 15, max= 168, per=24.88%, avg=76.13, stdev=31.04, samples=130 iops : min= 3, max= 42, avg=18.82, stdev= 7.78, samples=130 write: IOPS=20, BW=80.2KiB/s (82.1kB/s)(5256KiB/65528msec) slat (usec): min=14, max=4712, avg=169.85, stdev=342.17 clat (msec): min=53, max=1078, avg=540.85, stdev=149.63 lat (msec): min=53, max=1078, avg=541.02, stdev=149.65 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 317], 10.00th=[ 359], 20.00th=[ 414], | 30.00th=[ 460], 40.00th=[ 498], 50.00th=[ 535], 60.00th=[ 575], | 70.00th=[ 609], 80.00th=[ 659], 90.00th=[ 735], 95.00th=[ 793], | 99.00th=[ 936], 99.50th=[ 1003], 99.90th=[ 1062], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 24, max= 152, per=25.03%, avg=79.58, stdev=25.09, samples=130 iops : min= 6, max= 38, avg=19.68, stdev= 6.29, samples=130 lat (msec) : 10=0.08%, 20=0.04%, 50=0.35%, 100=0.20%, 250=21.99% lat (msec) : 500=47.03%, 750=26.02%, 1000=4.02% cpu : usr=0.15%, sys=1.54%, ctx=2787, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5277: Fri May 2 00:35:21 2025 read: IOPS=19, BW=77.5KiB/s (79.4kB/s)(5060KiB/65292msec) slat (usec): min=1447, max=26810, avg=5815.24, stdev=2666.34 clat (msec): min=8, max=553, avg=264.32, stdev=64.27 lat (msec): min=11, max=560, avg=270.14, stdev=65.00 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 176], 10.00th=[ 192], 20.00th=[ 213], | 30.00th=[ 228], 40.00th=[ 243], 50.00th=[ 257], 60.00th=[ 275], | 70.00th=[ 296], 80.00th=[ 317], 90.00th=[ 351], 95.00th=[ 376], | 99.00th=[ 418], 99.50th=[ 443], 99.90th=[ 493], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 24, max= 168, per=25.24%, avg=77.23, stdev=29.44, samples=130 iops : min= 6, max= 42, avg=19.03, stdev= 7.39, samples=130 write: IOPS=19, BW=79.3KiB/s (81.2kB/s)(5180KiB/65292msec) slat (usec): min=17, max=4244, avg=165.88, stdev=327.48 clat (msec): min=148, max=1031, avg=541.27, stdev=150.04 lat (msec): min=148, max=1031, avg=541.43, stdev=150.03 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 317], 10.00th=[ 363], 20.00th=[ 414], | 30.00th=[ 456], 40.00th=[ 489], 50.00th=[ 523], 60.00th=[ 567], | 70.00th=[ 617], 80.00th=[ 667], 90.00th=[ 743], 95.00th=[ 793], | 99.00th=[ 919], 99.50th=[ 969], 99.90th=[ 1020], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 23, max= 159, per=24.73%, avg=78.65, stdev=23.96, samples=130 iops : min= 5, max= 39, avg=19.38, stdev= 6.02, samples=130 lat (msec) : 10=0.04%, 20=0.08%, 50=0.16%, 100=0.08%, 250=23.12% lat (msec) : 500=47.54%, 750=24.34%, 1000=4.53% cpu : usr=0.21%, sys=1.50%, ctx=2761, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=306KiB/s (314kB/s), 75.9KiB/s-77.5KiB/s (77.8kB/s-79.4kB/s), io=19.6MiB (20.5MB), run=65292-65531msec WRITE: bw=319KiB/s (326kB/s), 79.3KiB/s-80.3KiB/s (81.2kB/s-82.3kB/s), io=20.4MiB (21.4MB), run=65292-65531msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=5334: Fri May 2 00:35:23 2025 read: IOPS=3, BW=127MiB/s (134MB/s)(40.0MiB/314msec) slat (nsec): min=2561.5k, max=2561.5k, avg=2561544.00, stdev= 0.00 clat (nsec): min=300689k, max=300689k, avg=300688948.00, stdev= 0.00 lat (nsec): min=303276k, max=303276k, avg=303275869.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=0.64%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=127MiB/s (134MB/s), 127MiB/s-127MiB/s (134MB/s-134MB/s), io=40.0MiB (41.9MB), run=314-314msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout