== sanity test 398c: run fio to test AIO ================= 10:03:25 (1743516205) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.102432 s, 409 MB/s osc.lustre-OST0000-osc-ffff88012aab6000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=9968: Tue Apr 1 10:03:51 2025 write: IOPS=107, BW=428KiB/s (439kB/s)(10.0MiB/23910msec) slat (usec): min=12, max=499, avg=64.36, stdev=41.72 clat (msec): min=58, max=263, avg=149.14, stdev=27.16 lat (msec): min=58, max=263, avg=149.20, stdev=27.16 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 106], 10.00th=[ 114], 20.00th=[ 127], | 30.00th=[ 136], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 157], | 70.00th=[ 165], 80.00th=[ 174], 90.00th=[ 184], 95.00th=[ 192], | 99.00th=[ 211], 99.50th=[ 224], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 359, max= 640, per=25.06%, avg=429.00, stdev=56.62, samples=47 iops : min= 89, max= 160, avg=107.17, stdev=14.21, samples=47 lat (msec) : 100=4.06%, 250=95.74%, 500=0.20% cpu : usr=0.15%, sys=0.75%, ctx=2121, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=9969: Tue Apr 1 10:03:51 2025 write: IOPS=107, BW=429KiB/s (439kB/s)(10.0MiB/23868msec) slat (usec): min=13, max=743, avg=67.49, stdev=43.37 clat (msec): min=23, max=275, avg=148.96, stdev=27.44 lat (msec): min=23, max=275, avg=149.03, stdev=27.44 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 105], 10.00th=[ 114], 20.00th=[ 128], | 30.00th=[ 136], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 157], | 70.00th=[ 165], 80.00th=[ 174], 90.00th=[ 184], 95.00th=[ 190], | 99.00th=[ 213], 99.50th=[ 215], 99.90th=[ 249], 99.95th=[ 257], | 99.99th=[ 275] bw ( KiB/s): min= 344, max= 624, per=25.11%, avg=429.85, stdev=56.90, samples=47 iops : min= 86, max= 156, avg=107.38, stdev=14.28, samples=47 lat (msec) : 50=0.39%, 100=3.91%, 250=95.62%, 500=0.08% cpu : usr=0.16%, sys=0.78%, ctx=2455, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=9970: Tue Apr 1 10:03:51 2025 write: IOPS=107, BW=428KiB/s (439kB/s)(10.0MiB/23909msec) slat (usec): min=12, max=524, avg=63.85, stdev=41.00 clat (msec): min=65, max=278, avg=149.17, stdev=26.69 lat (msec): min=65, max=278, avg=149.23, stdev=26.69 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 106], 10.00th=[ 114], 20.00th=[ 127], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 150], 60.00th=[ 157], | 70.00th=[ 165], 80.00th=[ 174], 90.00th=[ 184], 95.00th=[ 192], | 99.00th=[ 207], 99.50th=[ 215], 99.90th=[ 245], 99.95th=[ 271], | 99.99th=[ 279] bw ( KiB/s): min= 352, max= 632, per=25.04%, avg=428.74, stdev=56.64, samples=47 iops : min= 88, max= 158, avg=107.17, stdev=14.17, samples=47 lat (msec) : 100=4.06%, 250=95.86%, 500=0.08% cpu : usr=0.20%, sys=0.69%, ctx=2218, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=9971: Tue Apr 1 10:03:51 2025 write: IOPS=107, BW=428KiB/s (438kB/s)(10.0MiB/23923msec) slat (usec): min=12, max=413, avg=64.80, stdev=41.17 clat (msec): min=23, max=223, avg=149.10, stdev=26.31 lat (msec): min=23, max=223, avg=149.17, stdev=26.31 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 106], 10.00th=[ 114], 20.00th=[ 127], | 30.00th=[ 136], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 157], | 70.00th=[ 165], 80.00th=[ 174], 90.00th=[ 184], 95.00th=[ 190], | 99.00th=[ 205], 99.50th=[ 211], 99.90th=[ 215], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 344, max= 632, per=24.93%, avg=426.85, stdev=55.54, samples=47 iops : min= 86, max= 158, avg=106.68, stdev=13.90, samples=47 lat (msec) : 50=0.08%, 100=3.32%, 250=96.60% cpu : usr=0.13%, sys=0.76%, ctx=2348, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1712KiB/s (1753kB/s), 428KiB/s-429KiB/s (438kB/s-439kB/s), io=40.0MiB (41.9MB), run=23868-23923msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10000: Tue Apr 1 10:04:06 2025 read: IOPS=88, BW=353KiB/s (362kB/s)(5048KiB/14286msec) slat (usec): min=720, max=4958, avg=1176.46, stdev=302.83 clat (msec): min=4, max=141, avg=58.38, stdev=12.62 lat (msec): min=5, max=142, avg=59.56, stdev=12.67 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 42], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 67], 90.00th=[ 73], 95.00th=[ 79], | 99.00th=[ 101], 99.50th=[ 112], 99.90th=[ 126], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 264, max= 448, per=25.27%, avg=354.50, stdev=59.35, samples=28 iops : min= 66, max= 112, avg=88.57, stdev=14.87, samples=28 write: IOPS=90, BW=363KiB/s (372kB/s)(5192KiB/14286msec) slat (usec): min=13, max=576, avg=60.73, stdev=37.33 clat (msec): min=52, max=208, avg=117.73, stdev=25.17 lat (msec): min=52, max=208, avg=117.79, stdev=25.17 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 78], 10.00th=[ 85], 20.00th=[ 95], | 30.00th=[ 104], 40.00th=[ 110], 50.00th=[ 117], 60.00th=[ 125], | 70.00th=[ 133], 80.00th=[ 140], 90.00th=[ 150], 95.00th=[ 159], | 99.00th=[ 182], 99.50th=[ 186], 99.90th=[ 201], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 272, max= 424, per=24.87%, avg=363.07, stdev=38.39, samples=28 iops : min= 68, max= 106, avg=90.71, stdev= 9.54, samples=28 lat (msec) : 10=0.27%, 20=0.27%, 50=9.69%, 100=51.52%, 250=38.24% cpu : usr=0.22%, sys=1.75%, ctx=3858, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10001: Tue Apr 1 10:04:06 2025 read: IOPS=86, BW=348KiB/s (356kB/s)(4976KiB/14302msec) slat (usec): min=701, max=3040, avg=1153.78, stdev=278.93 clat (msec): min=8, max=117, avg=58.12, stdev=11.89 lat (msec): min=9, max=119, avg=59.28, stdev=11.92 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 41], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 67], 90.00th=[ 72], 95.00th=[ 78], | 99.00th=[ 91], 99.50th=[ 104], 99.90th=[ 116], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 255, max= 504, per=24.80%, avg=347.96, stdev=64.37, samples=28 iops : min= 63, max= 126, avg=86.96, stdev=16.13, samples=28 write: IOPS=92, BW=368KiB/s (377kB/s)(5264KiB/14302msec) slat (usec): min=13, max=440, avg=59.77, stdev=32.40 clat (msec): min=44, max=204, avg=117.55, stdev=24.61 lat (msec): min=44, max=204, avg=117.61, stdev=24.61 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 80], 10.00th=[ 88], 20.00th=[ 96], | 30.00th=[ 104], 40.00th=[ 111], 50.00th=[ 117], 60.00th=[ 124], | 70.00th=[ 130], 80.00th=[ 140], 90.00th=[ 150], 95.00th=[ 161], | 99.00th=[ 176], 99.50th=[ 186], 99.90th=[ 199], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 288, max= 456, per=25.05%, avg=365.68, stdev=34.53, samples=28 iops : min= 72, max= 114, avg=91.39, stdev= 8.61, samples=28 lat (msec) : 10=0.04%, 20=0.31%, 50=10.66%, 100=50.35%, 250=38.63% cpu : usr=0.31%, sys=1.66%, ctx=3581, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10002: Tue Apr 1 10:04:06 2025 read: IOPS=87, BW=349KiB/s (357kB/s)(4984KiB/14286msec) slat (usec): min=19, max=5013, avg=1167.78, stdev=321.15 clat (msec): min=2, max=151, avg=58.22, stdev=12.46 lat (msec): min=2, max=151, avg=59.39, stdev=12.52 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 42], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 67], 90.00th=[ 72], 95.00th=[ 79], | 99.00th=[ 91], 99.50th=[ 104], 99.90th=[ 126], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 176, max= 502, per=25.10%, avg=352.14, stdev=71.49, samples=28 iops : min= 44, max= 125, avg=87.96, stdev=17.79, samples=28 write: IOPS=91, BW=368KiB/s (377kB/s)(5256KiB/14286msec) slat (usec): min=15, max=1043, avg=57.62, stdev=39.42 clat (msec): min=25, max=207, avg=117.44, stdev=26.04 lat (msec): min=25, max=207, avg=117.49, stdev=26.04 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 75], 10.00th=[ 86], 20.00th=[ 95], | 30.00th=[ 104], 40.00th=[ 110], 50.00th=[ 117], 60.00th=[ 125], | 70.00th=[ 131], 80.00th=[ 140], 90.00th=[ 150], 95.00th=[ 159], | 99.00th=[ 184], 99.50th=[ 188], 99.90th=[ 201], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 280, max= 456, per=25.10%, avg=366.46, stdev=40.27, samples=28 iops : min= 70, max= 114, avg=91.54, stdev=10.16, samples=28 lat (msec) : 4=0.12%, 10=0.12%, 20=0.16%, 50=10.62%, 100=50.98% lat (msec) : 250=38.01% cpu : usr=0.29%, sys=1.69%, ctx=3134, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10003: Tue Apr 1 10:04:06 2025 read: IOPS=88, BW=354KiB/s (362kB/s)(5060KiB/14302msec) slat (usec): min=678, max=3259, avg=1169.07, stdev=271.75 clat (usec): min=1891, max=114438, avg=58663.55, stdev=11777.53 lat (msec): min=3, max=117, avg=59.83, stdev=11.83 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 42], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 64], 80.00th=[ 67], 90.00th=[ 73], 95.00th=[ 79], | 99.00th=[ 91], 99.50th=[ 106], 99.90th=[ 110], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 208, max= 472, per=25.27%, avg=354.54, stdev=65.06, samples=28 iops : min= 52, max= 118, avg=88.61, stdev=16.24, samples=28 write: IOPS=90, BW=362KiB/s (371kB/s)(5180KiB/14302msec) slat (usec): min=12, max=1051, avg=59.68, stdev=46.31 clat (msec): min=33, max=209, avg=117.82, stdev=25.42 lat (msec): min=33, max=209, avg=117.88, stdev=25.43 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 79], 10.00th=[ 86], 20.00th=[ 95], | 30.00th=[ 104], 40.00th=[ 111], 50.00th=[ 117], 60.00th=[ 125], | 70.00th=[ 131], 80.00th=[ 140], 90.00th=[ 150], 95.00th=[ 159], | 99.00th=[ 186], 99.50th=[ 197], 99.90th=[ 201], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 288, max= 456, per=24.60%, avg=359.11, stdev=39.48, samples=28 iops : min= 72, max= 114, avg=89.75, stdev= 9.89, samples=28 lat (msec) : 2=0.04%, 10=0.08%, 20=0.12%, 50=10.47%, 100=51.80% lat (msec) : 250=37.50% cpu : usr=0.25%, sys=1.70%, ctx=3687, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1403KiB/s (1437kB/s), 348KiB/s-354KiB/s (356kB/s-362kB/s), io=19.6MiB (20.5MB), run=14286-14302msec WRITE: bw=1461KiB/s (1496kB/s), 362KiB/s-368KiB/s (371kB/s-377kB/s), io=20.4MiB (21.4MB), run=14286-14302msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=10018: Tue Apr 1 10:04:08 2025 read: IOPS=12, BW=506MiB/s (531MB/s)(40.0MiB/79msec) slat (nsec): min=7193.8k, max=7193.8k, avg=7193842.00, stdev= 0.00 clat (nsec): min=67291k, max=67291k, avg=67291071.00, stdev= 0.00 lat (nsec): min=74496k, max=74496k, avg=74496353.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[67634], 5.00th=[67634], 10.00th=[67634], 20.00th=[67634], | 30.00th=[67634], 40.00th=[67634], 50.00th=[67634], 60.00th=[67634], | 70.00th=[67634], 80.00th=[67634], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=10.26%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=506MiB/s (531MB/s), 506MiB/s-506MiB/s (531MB/s-531MB/s), io=40.0MiB (41.9MB), run=79-79msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout