== sanity test 398c: run fio to test AIO ================= 11:00:55 (1721228455)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.136651 s, 307 MB/s
osc.lustre-OST0000-osc-ffff8800b6fe0800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=20436: Wed Jul 17 11:01:02 2024
  write: IOPS=458, BW=1835KiB/s (1880kB/s)(10.0MiB/5579msec)
    slat (usec): min=12, max=362, avg=77.70, stdev=39.83
    clat (usec): min=3705, max=78981, avg=34741.42, stdev=6714.10
     lat (usec): min=3818, max=79114, avg=34820.92, stdev=6716.67
    clat percentiles (usec):
     |  1.00th=[16909],  5.00th=[21627], 10.00th=[24249], 20.00th=[29230],
     | 30.00th=[33424], 40.00th=[35390], 50.00th=[36963], 60.00th=[38011],
     | 70.00th=[39060], 80.00th=[39584], 90.00th=[40109], 95.00th=[40633],
     | 99.00th=[43779], 99.50th=[48497], 99.90th=[71828], 99.95th=[74974],
     | 99.99th=[79168]
   bw (  KiB/s): min= 1584, max= 2544, per=25.21%, avg=1839.64, stdev=292.21, samples=11
   iops        : min=  396, max=  636, avg=459.91, stdev=73.05, samples=11
  lat (msec)   : 4=0.08%, 10=0.20%, 20=3.09%, 50=96.29%, 100=0.35%
  cpu          : usr=0.72%, sys=4.48%, ctx=1943, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=20437: Wed Jul 17 11:01:02 2024
  write: IOPS=456, BW=1827KiB/s (1870kB/s)(10.0MiB/5606msec)
    slat (usec): min=13, max=405, avg=68.97, stdev=38.36
    clat (usec): min=12994, max=65987, avg=34895.85, stdev=6308.21
     lat (usec): min=13091, max=66144, avg=34966.56, stdev=6314.29
    clat percentiles (usec):
     |  1.00th=[17171],  5.00th=[22152], 10.00th=[24511], 20.00th=[29230],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[37487], 60.00th=[38011],
     | 70.00th=[39060], 80.00th=[39584], 90.00th=[40109], 95.00th=[40633],
     | 99.00th=[45351], 99.50th=[45351], 99.90th=[65799], 99.95th=[65799],
     | 99.99th=[65799]
   bw (  KiB/s): min= 1584, max= 2440, per=25.08%, avg=1830.18, stdev=271.38, samples=11
   iops        : min=  396, max=  610, avg=457.55, stdev=67.85, samples=11
  lat (msec)   : 20=2.30%, 50=97.34%, 100=0.35%
  cpu          : usr=0.59%, sys=3.89%, ctx=1518, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=20438: Wed Jul 17 11:01:02 2024
  write: IOPS=456, BW=1826KiB/s (1869kB/s)(10.0MiB/5609msec)
    slat (usec): min=15, max=633, avg=77.16, stdev=43.91
    clat (usec): min=4612, max=77703, avg=34910.30, stdev=6518.55
     lat (usec): min=4718, max=77737, avg=34989.16, stdev=6524.54
    clat percentiles (usec):
     |  1.00th=[17171],  5.00th=[22152], 10.00th=[24249], 20.00th=[29230],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[37487], 60.00th=[38011],
     | 70.00th=[39060], 80.00th=[39584], 90.00th=[40109], 95.00th=[40633],
     | 99.00th=[45351], 99.50th=[45876], 99.90th=[73925], 99.95th=[73925],
     | 99.99th=[78119]
   bw (  KiB/s): min= 1584, max= 2432, per=25.11%, avg=1832.36, stdev=269.86, samples=11
   iops        : min=  396, max=  608, avg=458.09, stdev=67.47, samples=11
  lat (msec)   : 10=0.08%, 20=2.11%, 50=97.38%, 100=0.43%
  cpu          : usr=0.89%, sys=4.10%, ctx=1945, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=20439: Wed Jul 17 11:01:02 2024
  write: IOPS=456, BW=1825KiB/s (1868kB/s)(10.0MiB/5612msec)
    slat (usec): min=16, max=400, avg=73.04, stdev=39.74
    clat (usec): min=16502, max=72238, avg=34922.03, stdev=6278.61
     lat (usec): min=16582, max=72317, avg=34996.78, stdev=6286.47
    clat percentiles (usec):
     |  1.00th=[17695],  5.00th=[22152], 10.00th=[24511], 20.00th=[29230],
     | 30.00th=[33424], 40.00th=[35390], 50.00th=[37487], 60.00th=[38011],
     | 70.00th=[39060], 80.00th=[39584], 90.00th=[40109], 95.00th=[40633],
     | 99.00th=[43779], 99.50th=[44303], 99.90th=[68682], 99.95th=[68682],
     | 99.99th=[71828]
   bw (  KiB/s): min= 1552, max= 2432, per=25.05%, avg=1828.00, stdev=272.52, samples=11
   iops        : min=  388, max=  608, avg=457.00, stdev=68.13, samples=11
  lat (msec)   : 20=2.03%, 50=97.70%, 100=0.27%
  cpu          : usr=0.75%, sys=3.80%, ctx=1862, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7299KiB/s (7474kB/s), 1825KiB/s-1835KiB/s (1868kB/s-1880kB/s), io=40.0MiB (41.9MB), run=5579-5612msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=20453: Wed Jul 17 11:01:06 2024
   read: IOPS=542, BW=2169KiB/s (2221kB/s)(5048KiB/2327msec)
    slat (usec): min=443, max=2594, avg=856.66, stdev=280.80
    clat (usec): min=2695, max=32735, avg=13742.62, stdev=3617.01
     lat (usec): min=3199, max=34279, avg=14601.44, stdev=3806.47
    clat percentiles (usec):
     |  1.00th=[ 8717],  5.00th=[ 9765], 10.00th=[10421], 20.00th=[11207],
     | 30.00th=[11863], 40.00th=[12387], 50.00th=[13042], 60.00th=[13566],
     | 70.00th=[14353], 80.00th=[15401], 90.00th=[17433], 95.00th=[22152],
     | 99.00th=[26870], 99.50th=[28967], 99.90th=[32637], 99.95th=[32637],
     | 99.99th=[32637]
   bw (  KiB/s): min= 1480, max= 2312, per=24.08%, avg=2076.75, stdev=399.90, samples=4
   iops        : min=  370, max=  578, avg=519.00, stdev=99.89, samples=4
  write: IOPS=557, BW=2231KiB/s (2285kB/s)(5192KiB/2327msec)
    slat (usec): min=12, max=774, avg=42.75, stdev=33.67
    clat (usec): min=1678, max=31292, avg=14210.15, stdev=3344.47
     lat (usec): min=1710, max=31346, avg=14253.92, stdev=3352.52
    clat percentiles (usec):
     |  1.00th=[ 9765],  5.00th=[10683], 10.00th=[11207], 20.00th=[11863],
     | 30.00th=[12387], 40.00th=[12911], 50.00th=[13435], 60.00th=[14091],
     | 70.00th=[14746], 80.00th=[15533], 90.00th=[17171], 95.00th=[22676],
     | 99.00th=[26084], 99.50th=[27132], 99.90th=[29230], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 1544, max= 2568, per=24.40%, avg=2190.75, stdev=451.49, samples=4
   iops        : min=  386, max=  642, avg=547.50, stdev=112.75, samples=4
  lat (msec)   : 2=0.04%, 4=0.04%, 10=4.02%, 20=88.91%, 50=6.99%
  cpu          : usr=1.33%, sys=9.42%, ctx=2208, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=20454: Wed Jul 17 11:01:06 2024
   read: IOPS=535, BW=2141KiB/s (2193kB/s)(4976KiB/2324msec)
    slat (usec): min=421, max=2717, avg=856.42, stdev=286.27
    clat (usec): min=1859, max=29883, avg=13747.79, stdev=3677.14
     lat (usec): min=2469, max=31633, avg=14606.42, stdev=3854.29
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[ 9765], 10.00th=[10290], 20.00th=[11076],
     | 30.00th=[11731], 40.00th=[12256], 50.00th=[12911], 60.00th=[13698],
     | 70.00th=[14484], 80.00th=[15401], 90.00th=[17695], 95.00th=[22414],
     | 99.00th=[27395], 99.50th=[28443], 99.90th=[29492], 99.95th=[29754],
     | 99.99th=[29754]
   bw (  KiB/s): min= 1392, max= 2368, per=23.99%, avg=2068.75, stdev=454.85, samples=4
   iops        : min=  348, max=  592, avg=517.00, stdev=113.63, samples=4
  write: IOPS=566, BW=2265KiB/s (2319kB/s)(5264KiB/2324msec)
    slat (usec): min=11, max=622, avg=41.17, stdev=28.39
    clat (usec): min=8834, max=32303, avg=14225.73, stdev=3511.82
     lat (usec): min=8864, max=32335, avg=14267.77, stdev=3518.92
    clat percentiles (usec):
     |  1.00th=[ 9896],  5.00th=[10421], 10.00th=[11076], 20.00th=[11863],
     | 30.00th=[12387], 40.00th=[12911], 50.00th=[13435], 60.00th=[14091],
     | 70.00th=[14746], 80.00th=[15664], 90.00th=[17695], 95.00th=[22938],
     | 99.00th=[27395], 99.50th=[28181], 99.90th=[30278], 99.95th=[32375],
     | 99.99th=[32375]
   bw (  KiB/s): min= 1656, max= 2576, per=24.76%, avg=2222.75, stdev=397.36, samples=4
   iops        : min=  414, max=  644, avg=555.50, stdev=99.24, samples=4
  lat (msec)   : 2=0.04%, 10=4.41%, 20=88.32%, 50=7.23%
  cpu          : usr=1.21%, sys=9.34%, ctx=2195, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=20455: Wed Jul 17 11:01:06 2024
   read: IOPS=535, BW=2144KiB/s (2195kB/s)(4984KiB/2325msec)
    slat (usec): min=471, max=2442, avg=851.86, stdev=280.26
    clat (usec): min=874, max=31245, avg=13702.23, stdev=3612.37
     lat (usec): min=1718, max=33042, avg=14556.36, stdev=3789.70
    clat percentiles (usec):
     |  1.00th=[ 8717],  5.00th=[ 9634], 10.00th=[10290], 20.00th=[11207],
     | 30.00th=[11731], 40.00th=[12518], 50.00th=[12911], 60.00th=[13566],
     | 70.00th=[14353], 80.00th=[15270], 90.00th=[17957], 95.00th=[21627],
     | 99.00th=[27132], 99.50th=[29230], 99.90th=[30278], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 1520, max= 2339, per=24.15%, avg=2082.75, stdev=378.31, samples=4
   iops        : min=  380, max=  584, avg=520.50, stdev=94.41, samples=4
  write: IOPS=565, BW=2261KiB/s (2315kB/s)(5256KiB/2325msec)
    slat (usec): min=11, max=737, avg=42.72, stdev=35.84
    clat (usec): min=8514, max=28497, avg=14267.43, stdev=3455.42
     lat (usec): min=8545, max=28642, avg=14311.19, stdev=3466.69
    clat percentiles (usec):
     |  1.00th=[ 9634],  5.00th=[10683], 10.00th=[11076], 20.00th=[11863],
     | 30.00th=[12387], 40.00th=[13042], 50.00th=[13435], 60.00th=[13960],
     | 70.00th=[14615], 80.00th=[15664], 90.00th=[17695], 95.00th=[22676],
     | 99.00th=[26608], 99.50th=[27395], 99.90th=[28181], 99.95th=[28443],
     | 99.99th=[28443]
   bw (  KiB/s): min= 1536, max= 2672, per=24.53%, avg=2202.75, stdev=479.36, samples=4
   iops        : min=  384, max=  668, avg=550.50, stdev=119.82, samples=4
  lat (usec)   : 1000=0.04%
  lat (msec)   : 10=4.06%, 20=88.75%, 50=7.15%
  cpu          : usr=1.12%, sys=9.42%, ctx=2174, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=20456: Wed Jul 17 11:01:06 2024
   read: IOPS=545, BW=2184KiB/s (2236kB/s)(5060KiB/2317msec)
    slat (usec): min=25, max=2309, avg=856.65, stdev=287.17
    clat (usec): min=1917, max=32454, avg=13680.18, stdev=3703.15
     lat (usec): min=1948, max=34072, avg=14538.68, stdev=3893.21
    clat percentiles (usec):
     |  1.00th=[ 8717],  5.00th=[ 9634], 10.00th=[10159], 20.00th=[11076],
     | 30.00th=[11731], 40.00th=[12387], 50.00th=[12911], 60.00th=[13566],
     | 70.00th=[14353], 80.00th=[15401], 90.00th=[18220], 95.00th=[22152],
     | 99.00th=[26346], 99.50th=[26870], 99.90th=[30016], 99.95th=[32375],
     | 99.99th=[32375]
   bw (  KiB/s): min= 1568, max= 2464, per=24.61%, avg=2122.00, stdev=388.05, samples=4
   iops        : min=  392, max=  616, avg=530.50, stdev=97.01, samples=4
  write: IOPS=558, BW=2236KiB/s (2289kB/s)(5180KiB/2317msec)
    slat (usec): min=12, max=514, avg=43.30, stdev=33.79
    clat (usec): min=6033, max=31141, avg=14287.16, stdev=3588.24
     lat (usec): min=6256, max=31168, avg=14331.40, stdev=3596.43
    clat percentiles (usec):
     |  1.00th=[ 9765],  5.00th=[10683], 10.00th=[11076], 20.00th=[11863],
     | 30.00th=[12387], 40.00th=[12911], 50.00th=[13435], 60.00th=[13960],
     | 70.00th=[14746], 80.00th=[15795], 90.00th=[17695], 95.00th=[23462],
     | 99.00th=[27132], 99.50th=[28705], 99.90th=[30540], 99.95th=[31065],
     | 99.99th=[31065]
   bw (  KiB/s): min= 1544, max= 2480, per=24.35%, avg=2186.00, stdev=439.99, samples=4
   iops        : min=  386, max=  620, avg=546.50, stdev=110.00, samples=4
  lat (msec)   : 2=0.04%, 4=0.20%, 10=4.57%, 20=87.85%, 50=7.34%
  cpu          : usr=0.95%, sys=9.67%, ctx=2216, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=8624KiB/s (8831kB/s), 2141KiB/s-2184KiB/s (2193kB/s-2236kB/s), io=19.6MiB (20.5MB), run=2317-2327msec
  WRITE: bw=8978KiB/s (9194kB/s), 2231KiB/s-2265KiB/s (2285kB/s-2319kB/s), io=20.4MiB (21.4MB), run=2317-2327msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=20462: Wed Jul 17 11:01:07 2024
   read: IOPS=6, BW=263MiB/s (276MB/s)(40.0MiB/152msec)
    slat (nsec): min=6772.6k, max=6772.6k, avg=6772568.00, stdev= 0.00
    clat (nsec): min=139234k, max=139234k, avg=139234355.00, stdev= 0.00
     lat (nsec): min=146016k, max=146016k, avg=146015543.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  140],  5.00th=[  140], 10.00th=[  140], 20.00th=[  140],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  140],
     | 99.99th=[  140]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=3.97%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=263MiB/s (276MB/s), 263MiB/s-263MiB/s (276MB/s-276MB/s), io=40.0MiB (41.9MB), run=152-152msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck