== sanity test 398c: run fio to test AIO ================= 16:53:40 (1730580820) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.13667 s, 307 MB/s osc.lustre-OST0000-osc-ffff8800aaa55000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=29594: Sat Nov 2 16:54:16 2024 write: IOPS=72, BW=292KiB/s (299kB/s)(10.0MiB/35119msec) slat (usec): min=13, max=480, avg=54.00, stdev=32.92 clat (msec): min=101, max=434, avg=219.11, stdev=41.30 lat (msec): min=101, max=434, avg=219.16, stdev=41.30 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 153], 10.00th=[ 167], 20.00th=[ 186], | 30.00th=[ 199], 40.00th=[ 211], 50.00th=[ 220], 60.00th=[ 230], | 70.00th=[ 241], 80.00th=[ 253], 90.00th=[ 271], 95.00th=[ 284], | 99.00th=[ 313], 99.50th=[ 326], 99.90th=[ 388], 99.95th=[ 422], | 99.99th=[ 435] bw ( KiB/s): min= 216, max= 432, per=24.98%, avg=290.74, stdev=41.13, samples=70 iops : min= 54, max= 108, avg=72.60, stdev=10.27, samples=70 lat (msec) : 250=78.32%, 500=21.68% cpu : usr=0.15%, sys=0.44%, ctx=1911, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=29595: Sat Nov 2 16:54:16 2024 write: IOPS=72, BW=291KiB/s (298kB/s)(10.0MiB/35138msec) slat (usec): min=13, max=299, avg=55.50, stdev=31.59 clat (msec): min=89, max=337, avg=219.13, stdev=40.03 lat (msec): min=89, max=337, avg=219.19, stdev=40.04 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 155], 10.00th=[ 167], 20.00th=[ 186], | 30.00th=[ 201], 40.00th=[ 211], 50.00th=[ 220], 60.00th=[ 230], | 70.00th=[ 241], 80.00th=[ 251], 90.00th=[ 271], 95.00th=[ 284], | 99.00th=[ 317], 99.50th=[ 326], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 192, max= 392, per=24.96%, avg=290.53, stdev=37.87, samples=70 iops : min= 48, max= 98, avg=72.56, stdev= 9.48, samples=70 lat (msec) : 100=0.04%, 250=79.53%, 500=20.43% cpu : usr=0.11%, sys=0.48%, ctx=2115, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=29596: Sat Nov 2 16:54:16 2024 write: IOPS=72, BW=291KiB/s (298kB/s)(10.0MiB/35163msec) slat (usec): min=14, max=259, avg=57.23, stdev=34.52 clat (msec): min=84, max=337, avg=219.30, stdev=40.12 lat (msec): min=84, max=337, avg=219.36, stdev=40.12 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 157], 10.00th=[ 167], 20.00th=[ 186], | 30.00th=[ 201], 40.00th=[ 211], 50.00th=[ 220], 60.00th=[ 230], | 70.00th=[ 241], 80.00th=[ 253], 90.00th=[ 271], 95.00th=[ 288], | 99.00th=[ 313], 99.50th=[ 321], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 338] bw ( KiB/s): min= 192, max= 384, per=24.90%, avg=289.83, stdev=40.46, samples=70 iops : min= 48, max= 96, avg=72.37, stdev=10.12, samples=70 lat (msec) : 100=0.08%, 250=78.20%, 500=21.72% cpu : usr=0.10%, sys=0.49%, ctx=2218, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=29597: Sat Nov 2 16:54:16 2024 write: IOPS=73, BW=292KiB/s (299kB/s)(10.0MiB/35026msec) slat (usec): min=13, max=444, avg=62.34, stdev=37.93 clat (msec): min=15, max=455, avg=218.64, stdev=43.48 lat (msec): min=15, max=455, avg=218.71, stdev=43.47 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 150], 10.00th=[ 167], 20.00th=[ 186], | 30.00th=[ 199], 40.00th=[ 209], 50.00th=[ 220], 60.00th=[ 230], | 70.00th=[ 241], 80.00th=[ 253], 90.00th=[ 271], 95.00th=[ 288], | 99.00th=[ 317], 99.50th=[ 330], 99.90th=[ 439], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 184, max= 464, per=24.98%, avg=290.79, stdev=46.46, samples=70 iops : min= 46, max= 116, avg=72.64, stdev=11.61, samples=70 lat (msec) : 20=0.04%, 50=0.12%, 100=0.55%, 250=77.34%, 500=21.95% cpu : usr=0.08%, sys=0.53%, ctx=2650, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1165KiB/s (1193kB/s), 291KiB/s-292KiB/s (298kB/s-299kB/s), io=40.0MiB (41.9MB), run=35026-35163msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29634: Sat Nov 2 16:54:39 2024 read: IOPS=59, BW=236KiB/s (242kB/s)(5048KiB/21361msec) slat (usec): min=730, max=4538, avg=1455.01, stdev=397.81 clat (usec): min=1763, max=169162, avg=85637.30, stdev=17698.93 lat (msec): min=3, max=171, avg=87.09, stdev=17.78 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 61], 10.00th=[ 66], 20.00th=[ 72], | 30.00th=[ 78], 40.00th=[ 81], 50.00th=[ 85], 60.00th=[ 89], | 70.00th=[ 94], 80.00th=[ 101], 90.00th=[ 107], 95.00th=[ 114], | 99.00th=[ 128], 99.50th=[ 140], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 144, max= 359, per=24.96%, avg=234.14, stdev=44.51, samples=42 iops : min= 36, max= 89, avg=58.43, stdev=11.12, samples=42 write: IOPS=60, BW=243KiB/s (249kB/s)(5192KiB/21361msec) slat (usec): min=14, max=327, avg=50.55, stdev=25.43 clat (msec): min=29, max=314, avg=178.11, stdev=44.14 lat (msec): min=29, max=314, avg=178.16, stdev=44.14 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 109], 10.00th=[ 123], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 218], 90.00th=[ 239], 95.00th=[ 255], | 99.00th=[ 288], 99.50th=[ 305], 99.90th=[ 313], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 151, max= 304, per=24.65%, avg=240.81, stdev=36.42, samples=42 iops : min= 37, max= 76, avg=60.10, stdev= 9.10, samples=42 lat (msec) : 2=0.04%, 4=0.12%, 10=0.04%, 20=0.12%, 50=0.31% lat (msec) : 100=39.57%, 250=56.72%, 500=3.09% cpu : usr=0.13%, sys=1.15%, ctx=3059, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29635: Sat Nov 2 16:54:39 2024 read: IOPS=58, BW=233KiB/s (238kB/s)(4976KiB/21376msec) slat (usec): min=682, max=3775, avg=1438.52, stdev=396.28 clat (msec): min=2, max=169, avg=85.99, stdev=18.48 lat (msec): min=3, max=172, avg=87.43, stdev=18.60 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 62], 10.00th=[ 65], 20.00th=[ 73], | 30.00th=[ 78], 40.00th=[ 81], 50.00th=[ 85], 60.00th=[ 89], | 70.00th=[ 95], 80.00th=[ 101], 90.00th=[ 108], 95.00th=[ 117], | 99.00th=[ 136], 99.50th=[ 140], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 119, max= 320, per=24.60%, avg=230.71, stdev=52.57, samples=42 iops : min= 29, max= 80, avg=57.57, stdev=13.17, samples=42 write: IOPS=61, BW=246KiB/s (252kB/s)(5264KiB/21376msec) slat (usec): min=13, max=229, avg=53.53, stdev=28.02 clat (msec): min=23, max=314, avg=176.91, stdev=45.21 lat (msec): min=23, max=314, avg=176.96, stdev=45.21 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 102], 10.00th=[ 116], 20.00th=[ 140], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 213], 90.00th=[ 236], 95.00th=[ 257], | 99.00th=[ 288], 99.50th=[ 300], 99.90th=[ 305], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 184, max= 383, per=24.90%, avg=243.29, stdev=41.78, samples=42 iops : min= 46, max= 95, avg=60.71, stdev=10.40, samples=42 lat (msec) : 4=0.08%, 20=0.35%, 50=0.51%, 100=40.04%, 250=55.94% lat (msec) : 500=3.09% cpu : usr=0.15%, sys=1.09%, ctx=3403, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29636: Sat Nov 2 16:54:39 2024 read: IOPS=58, BW=234KiB/s (239kB/s)(4984KiB/21329msec) slat (usec): min=27, max=4146, avg=1426.58, stdev=399.55 clat (msec): min=2, max=193, avg=85.19, stdev=18.25 lat (msec): min=3, max=195, avg=86.62, stdev=18.36 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 60], 10.00th=[ 66], 20.00th=[ 72], | 30.00th=[ 77], 40.00th=[ 81], 50.00th=[ 85], 60.00th=[ 89], | 70.00th=[ 94], 80.00th=[ 101], 90.00th=[ 106], 95.00th=[ 115], | 99.00th=[ 128], 99.50th=[ 140], 99.90th=[ 192], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 120, max= 376, per=24.98%, avg=234.29, stdev=55.71, samples=42 iops : min= 30, max= 94, avg=58.57, stdev=13.93, samples=42 write: IOPS=61, BW=246KiB/s (252kB/s)(5256KiB/21329msec) slat (usec): min=15, max=1027, avg=52.61, stdev=38.21 clat (msec): min=28, max=316, avg=177.34, stdev=44.72 lat (msec): min=28, max=317, avg=177.39, stdev=44.72 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 109], 10.00th=[ 122], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 201], 80.00th=[ 215], 90.00th=[ 236], 95.00th=[ 259], | 99.00th=[ 284], 99.50th=[ 296], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 152, max= 312, per=25.07%, avg=244.95, stdev=36.04, samples=42 iops : min= 38, max= 78, avg=61.24, stdev= 9.01, samples=42 lat (msec) : 4=0.12%, 10=0.04%, 20=0.27%, 50=0.51%, 100=39.61% lat (msec) : 250=55.98%, 500=3.48% cpu : usr=0.16%, sys=1.09%, ctx=3170, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29637: Sat Nov 2 16:54:39 2024 read: IOPS=59, BW=237KiB/s (243kB/s)(5060KiB/21329msec) slat (usec): min=685, max=4440, avg=1437.42, stdev=397.84 clat (msec): min=2, max=170, avg=85.89, stdev=18.16 lat (msec): min=3, max=173, avg=87.33, stdev=18.26 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 61], 10.00th=[ 66], 20.00th=[ 72], | 30.00th=[ 78], 40.00th=[ 82], 50.00th=[ 85], 60.00th=[ 89], | 70.00th=[ 95], 80.00th=[ 102], 90.00th=[ 108], 95.00th=[ 116], | 99.00th=[ 131], 99.50th=[ 138], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 151, max= 359, per=25.27%, avg=237.07, stdev=50.98, samples=42 iops : min= 37, max= 89, avg=59.21, stdev=12.74, samples=42 write: IOPS=60, BW=243KiB/s (249kB/s)(5180KiB/21329msec) slat (usec): min=14, max=406, avg=51.87, stdev=28.90 clat (msec): min=64, max=315, avg=177.64, stdev=43.41 lat (msec): min=64, max=315, avg=177.69, stdev=43.41 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 107], 10.00th=[ 121], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 211], 90.00th=[ 234], 95.00th=[ 253], | 99.00th=[ 288], 99.50th=[ 300], 99.90th=[ 305], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 168, max= 336, per=24.66%, avg=240.88, stdev=39.63, samples=42 iops : min= 42, max= 84, avg=60.17, stdev= 9.88, samples=42 lat (msec) : 4=0.04%, 10=0.04%, 20=0.39%, 50=0.12%, 100=39.45% lat (msec) : 250=57.15%, 500=2.81% cpu : usr=0.16%, sys=1.09%, ctx=3207, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=939KiB/s (961kB/s), 233KiB/s-237KiB/s (238kB/s-243kB/s), io=19.6MiB (20.5MB), run=21329-21376msec WRITE: bw=977KiB/s (1001kB/s), 243KiB/s-246KiB/s (249kB/s-252kB/s), io=20.4MiB (21.4MB), run=21329-21376msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=29658: Sat Nov 2 16:54:40 2024 read: IOPS=7, BW=301MiB/s (315MB/s)(40.0MiB/133msec) slat (nsec): min=5504.4k, max=5504.4k, avg=5504376.00, stdev= 0.00 clat (nsec): min=120048k, max=120048k, avg=120047530.00, stdev= 0.00 lat (nsec): min=125561k, max=125561k, avg=125560787.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 121], 10.00th=[ 121], 20.00th=[ 121], | 30.00th=[ 121], 40.00th=[ 121], 50.00th=[ 121], 60.00th=[ 121], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 121], 95.00th=[ 121], | 99.00th=[ 121], 99.50th=[ 121], 99.90th=[ 121], 99.95th=[ 121], | 99.99th=[ 121] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.55%, ctx=15, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=301MiB/s (315MB/s), 301MiB/s-301MiB/s (315MB/s-315MB/s), io=40.0MiB (41.9MB), run=133-133msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout