== sanity test 398c: run fio to test AIO ================= 21:02:20 (1746147740) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.132514 s, 317 MB/s osc.lustre-OST0000-osc-ffff880093c45000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=673: Thu May 1 21:03:42 2025 write: IOPS=31, BW=127KiB/s (130kB/s)(10.0MiB/80628msec) slat (usec): min=11, max=4051, avg=123.87, stdev=291.58 clat (msec): min=170, max=963, avg=503.38, stdev=140.62 lat (msec): min=170, max=963, avg=503.51, stdev=140.63 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 288], 10.00th=[ 334], 20.00th=[ 384], | 30.00th=[ 422], 40.00th=[ 456], 50.00th=[ 498], 60.00th=[ 527], | 70.00th=[ 567], 80.00th=[ 617], 90.00th=[ 693], 95.00th=[ 751], | 99.00th=[ 852], 99.50th=[ 885], 99.90th=[ 953], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 56, max= 384, per=24.83%, avg=126.16, stdev=41.05, samples=161 iops : min= 14, max= 96, avg=31.32, stdev=10.29, samples=161 lat (msec) : 250=2.27%, 500=48.52%, 750=43.95%, 1000=5.27% cpu : usr=0.05%, sys=0.63%, ctx=1410, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=674: Thu May 1 21:03:42 2025 write: IOPS=31, BW=127KiB/s (130kB/s)(10.0MiB/80492msec) slat (usec): min=10, max=4374, avg=131.09, stdev=289.59 clat (msec): min=69, max=972, avg=502.52, stdev=142.82 lat (msec): min=69, max=974, avg=502.66, stdev=142.84 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 292], 10.00th=[ 326], 20.00th=[ 384], | 30.00th=[ 426], 40.00th=[ 456], 50.00th=[ 493], 60.00th=[ 531], | 70.00th=[ 567], 80.00th=[ 617], 90.00th=[ 701], 95.00th=[ 768], | 99.00th=[ 852], 99.50th=[ 869], 99.90th=[ 961], 99.95th=[ 961], | 99.99th=[ 969] bw ( KiB/s): min= 15, max= 264, per=24.96%, avg=126.79, stdev=39.82, samples=160 iops : min= 3, max= 66, avg=31.51, stdev=10.00, samples=160 lat (msec) : 100=0.08%, 250=2.77%, 500=48.98%, 750=42.19%, 1000=5.98% cpu : usr=0.14%, sys=0.55%, ctx=1448, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=675: Thu May 1 21:03:42 2025 write: IOPS=31, BW=127KiB/s (130kB/s)(10.0MiB/80492msec) slat (usec): min=10, max=4030, avg=123.14, stdev=240.35 clat (msec): min=59, max=963, avg=502.57, stdev=142.94 lat (msec): min=59, max=963, avg=502.70, stdev=142.96 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 288], 10.00th=[ 330], 20.00th=[ 384], | 30.00th=[ 426], 40.00th=[ 460], 50.00th=[ 498], 60.00th=[ 527], | 70.00th=[ 567], 80.00th=[ 617], 90.00th=[ 693], 95.00th=[ 760], | 99.00th=[ 844], 99.50th=[ 877], 99.90th=[ 953], 99.95th=[ 953], | 99.99th=[ 961] bw ( KiB/s): min= 56, max= 256, per=24.94%, avg=126.72, stdev=39.42, samples=160 iops : min= 14, max= 64, avg=31.47, stdev= 9.80, samples=160 lat (msec) : 100=0.62%, 250=1.91%, 500=49.18%, 750=42.73%, 1000=5.55% cpu : usr=0.14%, sys=0.59%, ctx=1530, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=676: Thu May 1 21:03:42 2025 write: IOPS=31, BW=127KiB/s (130kB/s)(10.0MiB/80587msec) slat (usec): min=10, max=3617, avg=122.31, stdev=246.03 clat (msec): min=170, max=974, avg=503.12, stdev=139.79 lat (msec): min=170, max=974, avg=503.25, stdev=139.80 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 292], 10.00th=[ 330], 20.00th=[ 388], | 30.00th=[ 422], 40.00th=[ 460], 50.00th=[ 493], 60.00th=[ 523], | 70.00th=[ 567], 80.00th=[ 625], 90.00th=[ 701], 95.00th=[ 760], | 99.00th=[ 844], 99.50th=[ 869], 99.90th=[ 919], 99.95th=[ 944], | 99.99th=[ 978] bw ( KiB/s): min= 55, max= 312, per=24.84%, avg=126.19, stdev=39.10, samples=161 iops : min= 13, max= 78, avg=31.35, stdev= 9.82, samples=161 lat (msec) : 250=1.84%, 500=50.66%, 750=41.88%, 1000=5.62% cpu : usr=0.07%, sys=0.61%, ctx=1564, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=508KiB/s (520kB/s), 127KiB/s-127KiB/s (130kB/s-130kB/s), io=40.0MiB (41.9MB), run=80492-80628msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=753: Thu May 1 21:04:39 2025 read: IOPS=22, BW=91.2KiB/s (93.4kB/s)(5048KiB/55372msec) slat (usec): min=1496, max=18136, avg=5086.68, stdev=2231.58 clat (msec): min=3, max=519, avg=226.53, stdev=53.36 lat (msec): min=10, max=524, avg=231.62, stdev=53.71 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 148], 10.00th=[ 167], 20.00th=[ 186], | 30.00th=[ 201], 40.00th=[ 213], 50.00th=[ 224], 60.00th=[ 236], | 70.00th=[ 249], 80.00th=[ 268], 90.00th=[ 292], 95.00th=[ 313], | 99.00th=[ 368], 99.50th=[ 409], 99.90th=[ 468], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 32, max= 216, per=25.08%, avg=90.79, stdev=31.16, samples=110 iops : min= 8, max= 54, avg=22.45, stdev= 7.83, samples=110 write: IOPS=23, BW=93.8KiB/s (96.0kB/s)(5192KiB/55372msec) slat (usec): min=14, max=3209, avg=141.03, stdev=282.11 clat (msec): min=205, max=860, avg=456.20, stdev=109.55 lat (msec): min=206, max=861, avg=456.35, stdev=109.58 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 279], 10.00th=[ 313], 20.00th=[ 363], | 30.00th=[ 397], 40.00th=[ 426], 50.00th=[ 451], 60.00th=[ 481], | 70.00th=[ 510], 80.00th=[ 550], 90.00th=[ 592], 95.00th=[ 642], | 99.00th=[ 768], 99.50th=[ 810], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 39, max= 152, per=24.76%, avg=93.34, stdev=23.69, samples=110 iops : min= 9, max= 38, avg=23.08, stdev= 5.98, samples=110 lat (msec) : 4=0.04%, 10=0.08%, 20=0.04%, 50=0.20%, 100=0.20% lat (msec) : 250=34.92%, 500=47.19%, 750=16.80%, 1000=0.55% cpu : usr=0.17%, sys=1.66%, ctx=2724, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=754: Thu May 1 21:04:39 2025 read: IOPS=22, BW=89.0KiB/s (92.2kB/s)(4976KiB/55294msec) slat (usec): min=1585, max=17120, avg=5069.63, stdev=2228.16 clat (msec): min=4, max=443, avg=222.79, stdev=50.71 lat (msec): min=7, max=448, avg=227.86, stdev=50.98 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 150], 10.00th=[ 169], 20.00th=[ 184], | 30.00th=[ 197], 40.00th=[ 209], 50.00th=[ 222], 60.00th=[ 232], | 70.00th=[ 245], 80.00th=[ 262], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 355], 99.50th=[ 384], 99.90th=[ 439], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 16, max= 151, per=24.70%, avg=89.42, stdev=28.73, samples=110 iops : min= 4, max= 37, avg=22.16, stdev= 7.14, samples=110 write: IOPS=23, BW=95.2KiB/s (97.5kB/s)(5264KiB/55294msec) slat (usec): min=14, max=3144, avg=147.43, stdev=301.08 clat (msec): min=188, max=914, avg=455.77, stdev=109.76 lat (msec): min=188, max=914, avg=455.92, stdev=109.76 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 288], 10.00th=[ 317], 20.00th=[ 359], | 30.00th=[ 393], 40.00th=[ 422], 50.00th=[ 447], 60.00th=[ 477], | 70.00th=[ 514], 80.00th=[ 550], 90.00th=[ 600], 95.00th=[ 642], | 99.00th=[ 718], 99.50th=[ 810], 99.90th=[ 852], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 32, max= 144, per=25.13%, avg=94.73, stdev=22.64, samples=110 iops : min= 8, max= 36, avg=23.49, stdev= 5.70, samples=110 lat (msec) : 10=0.04%, 50=0.35%, 100=0.12%, 250=36.02%, 500=46.48% lat (msec) : 750=16.64%, 1000=0.35% cpu : usr=0.20%, sys=1.62%, ctx=2733, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=755: Thu May 1 21:04:39 2025 read: IOPS=22, BW=90.2KiB/s (92.4kB/s)(4984KiB/55241msec) slat (usec): min=23, max=18439, avg=4979.09, stdev=2233.83 clat (msec): min=3, max=603, avg=223.18, stdev=52.65 lat (msec): min=7, max=608, avg=228.16, stdev=53.00 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 148], 10.00th=[ 167], 20.00th=[ 184], | 30.00th=[ 197], 40.00th=[ 211], 50.00th=[ 224], 60.00th=[ 234], | 70.00th=[ 247], 80.00th=[ 262], 90.00th=[ 279], 95.00th=[ 305], | 99.00th=[ 355], 99.50th=[ 363], 99.90th=[ 567], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 23, max= 208, per=24.87%, avg=90.03, stdev=34.61, samples=110 iops : min= 5, max= 52, avg=22.28, stdev= 8.71, samples=110 write: IOPS=23, BW=95.1KiB/s (97.4kB/s)(5256KiB/55241msec) slat (usec): min=12, max=3641, avg=152.85, stdev=312.52 clat (msec): min=64, max=873, avg=455.76, stdev=114.10 lat (msec): min=66, max=873, avg=455.91, stdev=114.12 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 279], 10.00th=[ 313], 20.00th=[ 363], | 30.00th=[ 393], 40.00th=[ 426], 50.00th=[ 451], 60.00th=[ 481], | 70.00th=[ 510], 80.00th=[ 550], 90.00th=[ 600], 95.00th=[ 642], | 99.00th=[ 760], 99.50th=[ 835], 99.90th=[ 869], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 31, max= 151, per=25.06%, avg=94.46, stdev=24.09, samples=110 iops : min= 7, max= 37, avg=23.39, stdev= 6.02, samples=110 lat (msec) : 4=0.04%, 10=0.08%, 50=0.35%, 100=0.12%, 250=36.02% lat (msec) : 500=46.25%, 750=16.52%, 1000=0.62% cpu : usr=0.20%, sys=1.64%, ctx=2718, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=756: Thu May 1 21:04:39 2025 read: IOPS=22, BW=91.5KiB/s (93.6kB/s)(5060KiB/55329msec) slat (usec): min=1268, max=18110, avg=5041.96, stdev=2154.22 clat (msec): min=5, max=442, avg=224.59, stdev=50.39 lat (msec): min=12, max=448, avg=229.64, stdev=50.64 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 150], 10.00th=[ 167], 20.00th=[ 184], | 30.00th=[ 199], 40.00th=[ 209], 50.00th=[ 222], 60.00th=[ 234], | 70.00th=[ 247], 80.00th=[ 264], 90.00th=[ 288], 95.00th=[ 309], | 99.00th=[ 351], 99.50th=[ 376], 99.90th=[ 430], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 16, max= 184, per=25.23%, avg=91.34, stdev=31.78, samples=110 iops : min= 4, max= 46, avg=22.61, stdev= 7.98, samples=110 write: IOPS=23, BW=93.6KiB/s (95.9kB/s)(5180KiB/55329msec) slat (usec): min=11, max=4777, avg=149.48, stdev=312.82 clat (msec): min=175, max=907, avg=458.16, stdev=112.24 lat (msec): min=175, max=907, avg=458.31, stdev=112.26 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 284], 10.00th=[ 326], 20.00th=[ 363], | 30.00th=[ 397], 40.00th=[ 422], 50.00th=[ 447], 60.00th=[ 481], | 70.00th=[ 514], 80.00th=[ 550], 90.00th=[ 609], 95.00th=[ 651], | 99.00th=[ 760], 99.50th=[ 835], 99.90th=[ 869], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 39, max= 144, per=24.69%, avg=93.08, stdev=22.15, samples=110 iops : min= 9, max= 36, avg=23.05, stdev= 5.53, samples=110 lat (msec) : 10=0.04%, 50=0.12%, 100=0.27%, 250=36.02%, 500=46.45% lat (msec) : 750=16.48%, 1000=0.62% cpu : usr=0.43%, sys=1.39%, ctx=2754, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=362KiB/s (371kB/s), 89.0KiB/s-91.5KiB/s (92.2kB/s-93.6kB/s), io=19.6MiB (20.5MB), run=55241-55372msec WRITE: bw=377KiB/s (386kB/s), 93.6KiB/s-95.2KiB/s (95.9kB/s-97.5kB/s), io=20.4MiB (21.4MB), run=55241-55372msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=805: Thu May 1 21:04:41 2025 read: IOPS=2, BW=115MiB/s (121MB/s)(40.0MiB/347msec) slat (nsec): min=6051.6k, max=6051.6k, avg=6051620.00, stdev= 0.00 clat (nsec): min=331904k, max=331904k, avg=331904370.00, stdev= 0.00 lat (nsec): min=337980k, max=337980k, avg=337979640.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 334], 10.00th=[ 334], 20.00th=[ 334], | 30.00th=[ 334], 40.00th=[ 334], 50.00th=[ 334], 60.00th=[ 334], | 70.00th=[ 334], 80.00th=[ 334], 90.00th=[ 334], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] lat (msec) : 500=100.00% cpu : usr=0.29%, sys=1.16%, ctx=13, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=115MiB/s (121MB/s), 115MiB/s-115MiB/s (121MB/s-121MB/s), io=40.0MiB (41.9MB), run=347-347msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout