== sanity test 398c: run fio to test AIO ================= 09:35:05 (1721223305)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.149869 s, 280 MB/s
osc.lustre-OST0000-osc-ffff88012b0a1800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=22217: Wed Jul 17 09:35:13 2024
  write: IOPS=453, BW=1814KiB/s (1858kB/s)(10.0MiB/5644msec)
    slat (usec): min=22, max=34600, avg=92.66, stdev=683.49
    clat (usec): min=5460, max=56497, avg=34912.83, stdev=3457.41
     lat (usec): min=5634, max=56547, avg=35007.42, stdev=3448.68
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28967], 10.00th=[30278], 20.00th=[32113],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[35914], 60.00th=[36439],
     | 70.00th=[36963], 80.00th=[37487], 90.00th=[38536], 95.00th=[39060],
     | 99.00th=[40633], 99.50th=[41157], 99.90th=[52691], 99.95th=[52691],
     | 99.99th=[56361]
   bw (  KiB/s): min= 1688, max= 1888, per=24.93%, avg=1809.09, stdev=57.05, samples=11
   iops        : min=  422, max=  472, avg=452.27, stdev=14.26, samples=11
  lat (msec)   : 10=0.04%, 20=0.31%, 50=99.53%, 100=0.12%
  cpu          : usr=0.62%, sys=4.59%, ctx=1722, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=22218: Wed Jul 17 09:35:13 2024
  write: IOPS=454, BW=1818KiB/s (1862kB/s)(10.0MiB/5632msec)
    slat (usec): min=22, max=253, avg=69.77, stdev=37.80
    clat (usec): min=24183, max=55546, avg=35078.70, stdev=3331.07
     lat (usec): min=24270, max=55716, avg=35150.10, stdev=3331.88
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[28967], 10.00th=[30802], 20.00th=[32113],
     | 30.00th=[33162], 40.00th=[34866], 50.00th=[35914], 60.00th=[36439],
     | 70.00th=[36963], 80.00th=[37487], 90.00th=[38536], 95.00th=[39060],
     | 99.00th=[43254], 99.50th=[43779], 99.90th=[51643], 99.95th=[55313],
     | 99.99th=[55313]
   bw (  KiB/s): min= 1664, max= 1920, per=25.04%, avg=1817.09, stdev=80.92, samples=11
   iops        : min=  416, max=  480, avg=454.27, stdev=20.23, samples=11
  lat (msec)   : 50=99.88%, 100=0.12%
  cpu          : usr=0.71%, sys=3.85%, ctx=1193, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=22219: Wed Jul 17 09:35:13 2024
  write: IOPS=455, BW=1822KiB/s (1865kB/s)(10.0MiB/5621msec)
    slat (usec): min=23, max=493, avg=83.57, stdev=43.38
    clat (usec): min=7548, max=60735, avg=35002.63, stdev=3521.82
     lat (usec): min=7618, max=60846, avg=35088.06, stdev=3522.09
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28705], 10.00th=[30802], 20.00th=[32113],
     | 30.00th=[33162], 40.00th=[34866], 50.00th=[35914], 60.00th=[36439],
     | 70.00th=[36963], 80.00th=[37487], 90.00th=[38536], 95.00th=[39060],
     | 99.00th=[41157], 99.50th=[43779], 99.90th=[56886], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min= 1672, max= 1904, per=25.10%, avg=1821.45, stdev=66.27, samples=11
   iops        : min=  418, max=  476, avg=455.36, stdev=16.57, samples=11
  lat (msec)   : 10=0.08%, 20=0.20%, 50=99.53%, 100=0.20%
  cpu          : usr=0.66%, sys=4.84%, ctx=1895, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=22220: Wed Jul 17 09:35:13 2024
  write: IOPS=453, BW=1815KiB/s (1859kB/s)(10.0MiB/5642msec)
    slat (usec): min=23, max=33894, avg=98.59, stdev=669.48
    clat (usec): min=7496, max=62352, avg=34890.36, stdev=3574.05
     lat (usec): min=7612, max=62389, avg=34990.99, stdev=3564.16
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28967], 10.00th=[30540], 20.00th=[32113],
     | 30.00th=[33162], 40.00th=[34866], 50.00th=[35914], 60.00th=[36439],
     | 70.00th=[36963], 80.00th=[37487], 90.00th=[38536], 95.00th=[39060],
     | 99.00th=[40109], 99.50th=[41157], 99.90th=[62129], 99.95th=[62129],
     | 99.99th=[62129]
   bw (  KiB/s): min= 1688, max= 1872, per=25.01%, avg=1814.91, stdev=61.56, samples=11
   iops        : min=  422, max=  468, avg=453.73, stdev=15.39, samples=11
  lat (msec)   : 10=0.12%, 20=0.27%, 50=99.41%, 100=0.20%
  cpu          : usr=0.59%, sys=5.67%, ctx=1985, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7257KiB/s (7431kB/s), 1814KiB/s-1822KiB/s (1858kB/s-1865kB/s), io=40.0MiB (41.9MB), run=5621-5644msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=22234: Wed Jul 17 09:35:16 2024
   read: IOPS=646, BW=2587KiB/s (2649kB/s)(5048KiB/1951msec)
    slat (usec): min=430, max=2016, avg=786.34, stdev=250.30
    clat (usec): min=3944, max=31805, avg=11558.72, stdev=3365.87
     lat (usec): min=4577, max=33060, avg=12347.31, stdev=3537.93
    clat percentiles (usec):
     |  1.00th=[ 7177],  5.00th=[ 8455], 10.00th=[ 8979], 20.00th=[ 9503],
     | 30.00th=[10028], 40.00th=[10421], 50.00th=[10814], 60.00th=[11338],
     | 70.00th=[11731], 80.00th=[12256], 90.00th=[14091], 95.00th=[19268],
     | 99.00th=[25297], 99.50th=[26608], 99.90th=[31851], 99.95th=[31851],
     | 99.99th=[31851]
   bw (  KiB/s): min= 1792, max= 2840, per=24.19%, avg=2488.00, stdev=602.77, samples=3
   iops        : min=  448, max=  710, avg=622.00, stdev=150.69, samples=3
  write: IOPS=665, BW=2661KiB/s (2725kB/s)(5192KiB/1951msec)
    slat (usec): min=11, max=248, avg=42.83, stdev=25.20
    clat (usec): min=2203, max=28583, avg=11787.84, stdev=3082.28
     lat (usec): min=2238, max=28647, avg=11831.66, stdev=3090.65
    clat percentiles (usec):
     |  1.00th=[ 8586],  5.00th=[ 9372], 10.00th=[ 9765], 20.00th=[10159],
     | 30.00th=[10552], 40.00th=[10814], 50.00th=[10945], 60.00th=[11338],
     | 70.00th=[11600], 80.00th=[12125], 90.00th=[13698], 95.00th=[20055],
     | 99.00th=[24249], 99.50th=[26346], 99.90th=[27132], 99.95th=[28705],
     | 99.99th=[28705]
   bw (  KiB/s): min= 2000, max= 2968, per=24.63%, avg=2637.33, stdev=552.08, samples=3
   iops        : min=  500, max=  742, avg=659.33, stdev=138.02, samples=3
  lat (msec)   : 4=0.12%, 10=22.30%, 20=73.05%, 50=4.53%
  cpu          : usr=1.49%, sys=11.59%, ctx=2257, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=22235: Wed Jul 17 09:35:16 2024
   read: IOPS=638, BW=2556KiB/s (2617kB/s)(4976KiB/1947msec)
    slat (usec): min=429, max=2191, avg=786.31, stdev=252.17
    clat (usec): min=1775, max=31646, avg=11475.79, stdev=3328.67
     lat (usec): min=3392, max=33044, avg=12264.09, stdev=3503.89
    clat percentiles (usec):
     |  1.00th=[ 7635],  5.00th=[ 8455], 10.00th=[ 8848], 20.00th=[ 9372],
     | 30.00th=[ 9896], 40.00th=[10421], 50.00th=[10814], 60.00th=[11207],
     | 70.00th=[11600], 80.00th=[12125], 90.00th=[14091], 95.00th=[19792],
     | 99.00th=[25297], 99.50th=[25822], 99.90th=[31589], 99.95th=[31589],
     | 99.99th=[31589]
   bw (  KiB/s): min= 1812, max= 2864, per=24.15%, avg=2484.00, stdev=583.63, samples=3
   iops        : min=  453, max=  716, avg=621.00, stdev=145.91, samples=3
  write: IOPS=675, BW=2704KiB/s (2769kB/s)(5264KiB/1947msec)
    slat (usec): min=11, max=683, avg=44.25, stdev=34.13
    clat (usec): min=5760, max=29094, avg=11876.40, stdev=3111.48
     lat (usec): min=5785, max=29212, avg=11921.65, stdev=3124.31
    clat percentiles (usec):
     |  1.00th=[ 8455],  5.00th=[ 9372], 10.00th=[ 9765], 20.00th=[10290],
     | 30.00th=[10552], 40.00th=[10814], 50.00th=[11076], 60.00th=[11338],
     | 70.00th=[11731], 80.00th=[12125], 90.00th=[14222], 95.00th=[20317],
     | 99.00th=[24773], 99.50th=[25297], 99.90th=[28705], 99.95th=[28967],
     | 99.99th=[28967]
   bw (  KiB/s): min= 2059, max= 3040, per=25.16%, avg=2694.33, stdev=550.93, samples=3
   iops        : min=  514, max=  760, avg=673.33, stdev=138.16, samples=3
  lat (msec)   : 2=0.04%, 10=23.24%, 20=71.95%, 50=4.77%
  cpu          : usr=1.49%, sys=11.56%, ctx=2224, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=22236: Wed Jul 17 09:35:16 2024
   read: IOPS=641, BW=2565KiB/s (2627kB/s)(4984KiB/1943msec)
    slat (usec): min=26, max=1974, avg=783.01, stdev=261.10
    clat (usec): min=5512, max=33111, avg=11526.58, stdev=3374.76
     lat (usec): min=6652, max=34690, avg=12311.90, stdev=3555.64
    clat percentiles (usec):
     |  1.00th=[ 7701],  5.00th=[ 8455], 10.00th=[ 8848], 20.00th=[ 9503],
     | 30.00th=[ 9896], 40.00th=[10421], 50.00th=[10814], 60.00th=[11207],
     | 70.00th=[11600], 80.00th=[12125], 90.00th=[14615], 95.00th=[19792],
     | 99.00th=[25822], 99.50th=[26608], 99.90th=[31327], 99.95th=[33162],
     | 99.99th=[33162]
   bw (  KiB/s): min= 1972, max= 2880, per=24.64%, avg=2534.67, stdev=491.47, samples=3
   iops        : min=  493, max=  720, avg=633.67, stdev=122.87, samples=3
  write: IOPS=676, BW=2705KiB/s (2770kB/s)(5256KiB/1943msec)
    slat (usec): min=11, max=326, avg=43.83, stdev=27.27
    clat (usec): min=3488, max=29593, avg=11853.23, stdev=3282.12
     lat (usec): min=3557, max=29667, avg=11898.01, stdev=3290.21
    clat percentiles (usec):
     |  1.00th=[ 8586],  5.00th=[ 9372], 10.00th=[ 9765], 20.00th=[10159],
     | 30.00th=[10552], 40.00th=[10814], 50.00th=[11076], 60.00th=[11338],
     | 70.00th=[11600], 80.00th=[12125], 90.00th=[13960], 95.00th=[20579],
     | 99.00th=[25297], 99.50th=[27132], 99.90th=[29230], 99.95th=[29492],
     | 99.99th=[29492]
   bw (  KiB/s): min= 1908, max= 3072, per=24.62%, avg=2636.00, stdev=634.56, samples=3
   iops        : min=  477, max=  768, avg=659.00, stdev=158.64, samples=3
  lat (msec)   : 4=0.04%, 10=23.32%, 20=71.45%, 50=5.20%
  cpu          : usr=1.24%, sys=11.59%, ctx=2233, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=22237: Wed Jul 17 09:35:16 2024
   read: IOPS=648, BW=2595KiB/s (2657kB/s)(5060KiB/1950msec)
    slat (usec): min=431, max=2144, avg=788.78, stdev=257.87
    clat (usec): min=6395, max=32278, avg=11513.23, stdev=3280.91
     lat (usec): min=7113, max=33353, avg=12303.98, stdev=3465.39
    clat percentiles (usec):
     |  1.00th=[ 7635],  5.00th=[ 8455], 10.00th=[ 8848], 20.00th=[ 9503],
     | 30.00th=[ 9896], 40.00th=[10421], 50.00th=[10945], 60.00th=[11338],
     | 70.00th=[11731], 80.00th=[12256], 90.00th=[13960], 95.00th=[19530],
     | 99.00th=[24511], 99.50th=[25822], 99.90th=[31589], 99.95th=[32375],
     | 99.99th=[32375]
   bw (  KiB/s): min= 1852, max= 2896, per=24.56%, avg=2526.67, stdev=585.15, samples=3
   iops        : min=  463, max=  724, avg=631.67, stdev=146.29, samples=3
  write: IOPS=664, BW=2656KiB/s (2720kB/s)(5180KiB/1950msec)
    slat (usec): min=12, max=691, avg=44.57, stdev=37.70
    clat (usec): min=7118, max=30645, avg=11843.50, stdev=3184.54
     lat (usec): min=7168, max=30720, avg=11889.00, stdev=3194.69
    clat percentiles (usec):
     |  1.00th=[ 8717],  5.00th=[ 9372], 10.00th=[ 9765], 20.00th=[10159],
     | 30.00th=[10552], 40.00th=[10814], 50.00th=[11076], 60.00th=[11338],
     | 70.00th=[11731], 80.00th=[11994], 90.00th=[13698], 95.00th=[20841],
     | 99.00th=[24511], 99.50th=[25822], 99.90th=[29230], 99.95th=[30540],
     | 99.99th=[30540]
   bw (  KiB/s): min= 1948, max= 3000, per=24.32%, avg=2604.00, stdev=572.17, samples=3
   iops        : min=  487, max=  750, avg=651.00, stdev=143.04, samples=3
  lat (msec)   : 10=23.95%, 20=71.02%, 50=5.04%
  cpu          : usr=1.28%, sys=11.70%, ctx=2283, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=10.0MiB/s (10.5MB/s), 2556KiB/s-2595KiB/s (2617kB/s-2657kB/s), io=19.6MiB (20.5MB), run=1943-1951msec
  WRITE: bw=10.5MiB/s (10.0MB/s), 2656KiB/s-2705KiB/s (2720kB/s-2770kB/s), io=20.4MiB (21.4MB), run=1943-1951msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=22242: Wed Jul 17 09:35:17 2024
   read: IOPS=5, BW=234MiB/s (245MB/s)(40.0MiB/171msec)
    slat (nsec): min=7022.8k, max=7022.8k, avg=7022781.00, stdev= 0.00
    clat (nsec): min=158920k, max=158920k, avg=158919656.00, stdev= 0.00
     lat (nsec): min=165952k, max=165952k, avg=165952164.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  159],  5.00th=[  159], 10.00th=[  159], 20.00th=[  159],
     | 30.00th=[  159], 40.00th=[  159], 50.00th=[  159], 60.00th=[  159],
     | 70.00th=[  159], 80.00th=[  159], 90.00th=[  159], 95.00th=[  159],
     | 99.00th=[  159], 99.50th=[  159], 99.90th=[  159], 99.95th=[  159],
     | 99.99th=[  159]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=4.71%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=234MiB/s (245MB/s), 234MiB/s-234MiB/s (245MB/s-245MB/s), io=40.0MiB (41.9MB), run=171-171msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck