== sanity test 398c: run fio to test AIO ================= 07:20:22 (1730809222) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.107188 s, 391 MB/s osc.lustre-OST0000-osc-ffff8800a8c48800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=27874: Tue Nov 5 07:21:08 2024 write: IOPS=58, BW=234KiB/s (239kB/s)(10.0MiB/43809msec) slat (usec): min=17, max=641, avg=104.45, stdev=51.75 clat (msec): min=39, max=409, avg=273.51, stdev=53.73 lat (msec): min=39, max=409, avg=273.62, stdev=53.73 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 182], 10.00th=[ 207], 20.00th=[ 230], | 30.00th=[ 247], 40.00th=[ 259], 50.00th=[ 275], 60.00th=[ 288], | 70.00th=[ 305], 80.00th=[ 321], 90.00th=[ 342], 95.00th=[ 359], | 99.00th=[ 384], 99.50th=[ 388], 99.90th=[ 401], 99.95th=[ 405], | 99.99th=[ 409] bw ( KiB/s): min= 159, max= 383, per=25.02%, avg=232.98, stdev=41.40, samples=87 iops : min= 39, max= 95, avg=58.15, stdev=10.33, samples=87 lat (msec) : 50=0.20%, 100=0.23%, 250=32.42%, 500=67.15% cpu : usr=0.15%, sys=0.70%, ctx=2354, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=27875: Tue Nov 5 07:21:08 2024 write: IOPS=58, BW=233KiB/s (239kB/s)(10.0MiB/43954msec) slat (usec): min=22, max=484, avg=98.35, stdev=52.41 clat (msec): min=46, max=409, avg=274.19, stdev=52.49 lat (msec): min=46, max=409, avg=274.29, stdev=52.49 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 190], 10.00th=[ 203], 20.00th=[ 230], | 30.00th=[ 247], 40.00th=[ 259], 50.00th=[ 271], 60.00th=[ 288], | 70.00th=[ 305], 80.00th=[ 321], 90.00th=[ 342], 95.00th=[ 359], | 99.00th=[ 384], 99.50th=[ 393], 99.90th=[ 405], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 128, max= 383, per=24.83%, avg=231.14, stdev=40.02, samples=87 iops : min= 32, max= 95, avg=57.69, stdev= 9.98, samples=87 lat (msec) : 50=0.08%, 100=0.04%, 250=32.70%, 500=67.19% cpu : usr=0.11%, sys=0.67%, ctx=2117, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=27876: Tue Nov 5 07:21:08 2024 write: IOPS=58, BW=233KiB/s (239kB/s)(10.0MiB/43940msec) slat (usec): min=19, max=526, avg=96.87, stdev=50.38 clat (msec): min=62, max=408, avg=274.17, stdev=52.42 lat (msec): min=62, max=408, avg=274.27, stdev=52.43 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 184], 10.00th=[ 209], 20.00th=[ 230], | 30.00th=[ 247], 40.00th=[ 259], 50.00th=[ 275], 60.00th=[ 288], | 70.00th=[ 309], 80.00th=[ 326], 90.00th=[ 342], 95.00th=[ 355], | 99.00th=[ 380], 99.50th=[ 393], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 136, max= 384, per=24.97%, avg=232.43, stdev=43.43, samples=87 iops : min= 34, max= 96, avg=58.01, stdev=10.82, samples=87 lat (msec) : 100=0.04%, 250=31.99%, 500=67.97% cpu : usr=0.11%, sys=0.69%, ctx=1822, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=27877: Tue Nov 5 07:21:08 2024 write: IOPS=58, BW=233KiB/s (239kB/s)(10.0MiB/43897msec) slat (usec): min=27, max=455, avg=95.06, stdev=47.70 clat (msec): min=117, max=407, avg=273.83, stdev=51.55 lat (msec): min=117, max=407, avg=273.93, stdev=51.56 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 182], 10.00th=[ 205], 20.00th=[ 230], | 30.00th=[ 249], 40.00th=[ 262], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 305], 80.00th=[ 317], 90.00th=[ 342], 95.00th=[ 359], | 99.00th=[ 380], 99.50th=[ 388], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 409] bw ( KiB/s): min= 144, max= 383, per=24.96%, avg=232.38, stdev=40.67, samples=87 iops : min= 36, max= 95, avg=58.03, stdev=10.14, samples=87 lat (msec) : 250=31.29%, 500=68.71% cpu : usr=0.11%, sys=0.69%, ctx=1882, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=932KiB/s (954kB/s), 233KiB/s-234KiB/s (239kB/s-239kB/s), io=40.0MiB (41.9MB), run=43809-43954msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27922: Tue Nov 5 07:21:29 2024 read: IOPS=61, BW=246KiB/s (252kB/s)(5048KiB/20491msec) slat (usec): min=23, max=3734, avg=1513.11, stdev=508.81 clat (msec): min=2, max=170, avg=83.88, stdev=27.62 lat (msec): min=3, max=172, avg=85.40, stdev=27.96 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 50], 10.00th=[ 55], 20.00th=[ 61], | 30.00th=[ 66], 40.00th=[ 72], 50.00th=[ 80], 60.00th=[ 87], | 70.00th=[ 97], 80.00th=[ 110], 90.00th=[ 124], 95.00th=[ 134], | 99.00th=[ 150], 99.50th=[ 159], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 112, max= 432, per=25.03%, avg=244.50, stdev=78.18, samples=40 iops : min= 28, max= 108, avg=61.05, stdev=19.55, samples=40 write: IOPS=63, BW=253KiB/s (259kB/s)(5192KiB/20491msec) slat (usec): min=16, max=628, avg=74.49, stdev=47.03 clat (msec): min=30, max=416, avg=169.32, stdev=57.93 lat (msec): min=30, max=416, avg=169.39, stdev=57.94 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 95], 10.00th=[ 106], 20.00th=[ 120], | 30.00th=[ 134], 40.00th=[ 148], 50.00th=[ 161], 60.00th=[ 176], | 70.00th=[ 192], 80.00th=[ 211], 90.00th=[ 251], 95.00th=[ 288], | 99.00th=[ 338], 99.50th=[ 363], 99.90th=[ 388], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 136, max= 384, per=24.53%, avg=249.50, stdev=68.07, samples=40 iops : min= 34, max= 96, avg=62.30, stdev=17.06, samples=40 lat (msec) : 4=0.16%, 10=0.16%, 20=0.16%, 50=2.50%, 100=36.05% lat (msec) : 250=55.86%, 500=5.12% cpu : usr=0.20%, sys=1.57%, ctx=3513, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27923: Tue Nov 5 07:21:29 2024 read: IOPS=60, BW=243KiB/s (248kB/s)(4976KiB/20511msec) slat (usec): min=705, max=4481, avg=1523.01, stdev=506.04 clat (usec): min=1617, max=168710, avg=82929.65, stdev=26373.86 lat (msec): min=2, max=170, avg=84.46, stdev=26.70 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 51], 10.00th=[ 55], 20.00th=[ 62], | 30.00th=[ 66], 40.00th=[ 71], 50.00th=[ 78], 60.00th=[ 86], | 70.00th=[ 95], 80.00th=[ 108], 90.00th=[ 122], 95.00th=[ 131], | 99.00th=[ 150], 99.50th=[ 155], 99.90th=[ 161], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 120, max= 416, per=24.66%, avg=240.93, stdev=79.97, samples=41 iops : min= 30, max= 104, avg=60.20, stdev=20.00, samples=41 write: IOPS=64, BW=257KiB/s (263kB/s)(5264KiB/20511msec) slat (usec): min=15, max=1445, avg=74.81, stdev=55.88 clat (msec): min=54, max=388, avg=169.15, stdev=56.31 lat (msec): min=54, max=388, avg=169.23, stdev=56.32 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 93], 10.00th=[ 105], 20.00th=[ 122], | 30.00th=[ 136], 40.00th=[ 150], 50.00th=[ 161], 60.00th=[ 176], | 70.00th=[ 192], 80.00th=[ 209], 90.00th=[ 245], 95.00th=[ 279], | 99.00th=[ 338], 99.50th=[ 368], 99.90th=[ 376], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 112, max= 392, per=25.13%, avg=255.56, stdev=67.92, samples=41 iops : min= 28, max= 98, avg=63.85, stdev=16.99, samples=41 lat (msec) : 2=0.08%, 10=0.04%, 20=0.20%, 50=1.72%, 100=38.40% lat (msec) : 250=54.84%, 500=4.73% cpu : usr=0.21%, sys=1.61%, ctx=3270, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27924: Tue Nov 5 07:21:29 2024 read: IOPS=60, BW=243KiB/s (248kB/s)(4984KiB/20538msec) slat (usec): min=672, max=4551, avg=1532.40, stdev=509.69 clat (msec): min=5, max=168, avg=83.99, stdev=27.96 lat (msec): min=7, max=170, avg=85.52, stdev=28.29 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 49], 10.00th=[ 54], 20.00th=[ 60], | 30.00th=[ 66], 40.00th=[ 71], 50.00th=[ 79], 60.00th=[ 87], | 70.00th=[ 99], 80.00th=[ 111], 90.00th=[ 125], 95.00th=[ 138], | 99.00th=[ 155], 99.50th=[ 157], 99.90th=[ 161], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 120, max= 456, per=24.77%, avg=242.05, stdev=78.24, samples=41 iops : min= 30, max= 114, avg=60.44, stdev=19.54, samples=41 write: IOPS=63, BW=256KiB/s (262kB/s)(5256KiB/20538msec) slat (usec): min=13, max=1602, avg=75.06, stdev=61.34 clat (msec): min=35, max=415, avg=168.52, stdev=58.12 lat (msec): min=35, max=415, avg=168.60, stdev=58.14 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 91], 10.00th=[ 103], 20.00th=[ 118], | 30.00th=[ 134], 40.00th=[ 148], 50.00th=[ 161], 60.00th=[ 176], | 70.00th=[ 192], 80.00th=[ 211], 90.00th=[ 253], 95.00th=[ 279], | 99.00th=[ 338], 99.50th=[ 351], 99.90th=[ 368], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 144, max= 416, per=24.95%, avg=253.76, stdev=74.22, samples=41 iops : min= 36, max= 104, avg=63.37, stdev=18.61, samples=41 lat (msec) : 10=0.04%, 20=0.08%, 50=3.05%, 100=36.33%, 250=55.08% lat (msec) : 500=5.43% cpu : usr=0.17%, sys=1.65%, ctx=3499, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27925: Tue Nov 5 07:21:29 2024 read: IOPS=61, BW=248KiB/s (254kB/s)(5060KiB/20434msec) slat (usec): min=680, max=4603, avg=1526.92, stdev=522.57 clat (usec): min=1271, max=160541, avg=82668.98, stdev=27513.42 lat (msec): min=2, max=162, avg=84.20, stdev=27.85 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 48], 10.00th=[ 54], 20.00th=[ 59], | 30.00th=[ 66], 40.00th=[ 70], 50.00th=[ 78], 60.00th=[ 86], | 70.00th=[ 97], 80.00th=[ 108], 90.00th=[ 123], 95.00th=[ 133], | 99.00th=[ 153], 99.50th=[ 157], 99.90th=[ 159], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 112, max= 472, per=25.21%, avg=246.35, stdev=85.03, samples=40 iops : min= 28, max= 118, avg=61.55, stdev=21.26, samples=40 write: IOPS=63, BW=253KiB/s (260kB/s)(5180KiB/20434msec) slat (usec): min=14, max=438, avg=72.88, stdev=40.10 clat (msec): min=57, max=398, avg=169.66, stdev=57.97 lat (msec): min=57, max=398, avg=169.74, stdev=57.99 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 92], 10.00th=[ 104], 20.00th=[ 118], | 30.00th=[ 136], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 176], | 70.00th=[ 192], 80.00th=[ 211], 90.00th=[ 253], 95.00th=[ 284], | 99.00th=[ 338], 99.50th=[ 355], 99.90th=[ 384], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 144, max= 400, per=24.66%, avg=250.75, stdev=67.03, samples=40 iops : min= 36, max= 100, avg=62.65, stdev=16.76, samples=40 lat (msec) : 2=0.08%, 20=0.27%, 50=2.85%, 100=37.11%, 250=54.30% lat (msec) : 500=5.39% cpu : usr=0.17%, sys=1.66%, ctx=3221, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=977KiB/s (1001kB/s), 243KiB/s-248KiB/s (248kB/s-254kB/s), io=19.6MiB (20.5MB), run=20434-20538msec WRITE: bw=1017KiB/s (1042kB/s), 253KiB/s-257KiB/s (259kB/s-263kB/s), io=20.4MiB (21.4MB), run=20434-20538msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=27945: Tue Nov 5 07:21:31 2024 read: IOPS=6, BW=256MiB/s (269MB/s)(40.0MiB/156msec) slat (nsec): min=5880.8k, max=5880.8k, avg=5880847.00, stdev= 0.00 clat (nsec): min=143946k, max=143946k, avg=143946113.00, stdev= 0.00 lat (nsec): min=149838k, max=149838k, avg=149837864.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 144], 50.00th=[ 144], 60.00th=[ 144], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=3.23%, ctx=15, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=256MiB/s (269MB/s), 256MiB/s-256MiB/s (269MB/s-269MB/s), io=40.0MiB (41.9MB), run=156-156msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout