== sanity test 398c: run fio to test AIO ================= 13:42:30 (1721410950)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.121833 s, 344 MB/s
osc.lustre-OST0000-osc-ffff88012b0df000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=18575: Fri Jul 19 13:42:35 2024
  write: IOPS=636, BW=2545KiB/s (2606kB/s)(10.0MiB/4023msec)
    slat (usec): min=12, max=508, avg=69.83, stdev=43.99
    clat (usec): min=3474, max=59577, avg=24999.73, stdev=8735.27
     lat (usec): min=3508, max=59678, avg=25071.26, stdev=8755.55
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[13304], 10.00th=[15139], 20.00th=[16712],
     | 30.00th=[17957], 40.00th=[20055], 50.00th=[23725], 60.00th=[27132],
     | 70.00th=[30802], 80.00th=[35390], 90.00th=[36963], 95.00th=[38011],
     | 99.00th=[44827], 99.50th=[50594], 99.90th=[55837], 99.95th=[55837],
     | 99.99th=[59507]
   bw (  KiB/s): min= 1720, max= 3640, per=25.02%, avg=2545.00, stdev=732.81, samples=8
   iops        : min=  430, max=  910, avg=636.25, stdev=183.20, samples=8
  lat (msec)   : 4=0.08%, 10=0.27%, 20=39.49%, 50=59.34%, 100=0.82%
  cpu          : usr=0.75%, sys=4.85%, ctx=2069, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18576: Fri Jul 19 13:42:35 2024
  write: IOPS=636, BW=2545KiB/s (2606kB/s)(10.0MiB/4023msec)
    slat (usec): min=12, max=620, avg=65.84, stdev=44.08
    clat (usec): min=2400, max=51480, avg=25009.10, stdev=8617.39
     lat (usec): min=2444, max=51507, avg=25076.58, stdev=8637.82
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[13173], 10.00th=[15270], 20.00th=[16909],
     | 30.00th=[17957], 40.00th=[20317], 50.00th=[23725], 60.00th=[27132],
     | 70.00th=[31065], 80.00th=[34866], 90.00th=[36963], 95.00th=[38011],
     | 99.00th=[40109], 99.50th=[50594], 99.90th=[51119], 99.95th=[51643],
     | 99.99th=[51643]
   bw (  KiB/s): min= 1720, max= 3616, per=24.97%, avg=2540.00, stdev=725.07, samples=8
   iops        : min=  430, max=  904, avg=635.00, stdev=181.27, samples=8
  lat (msec)   : 4=0.08%, 20=39.45%, 50=59.84%, 100=0.62%
  cpu          : usr=0.52%, sys=4.55%, ctx=1870, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18577: Fri Jul 19 13:42:35 2024
  write: IOPS=635, BW=2543KiB/s (2604kB/s)(10.0MiB/4027msec)
    slat (usec): min=12, max=526, avg=63.63, stdev=39.58
    clat (usec): min=8077, max=59766, avg=25029.02, stdev=8742.27
     lat (usec): min=8123, max=59970, avg=25094.35, stdev=8759.99
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[13435], 10.00th=[15139], 20.00th=[16909],
     | 30.00th=[17957], 40.00th=[20055], 50.00th=[23725], 60.00th=[27132],
     | 70.00th=[31065], 80.00th=[35390], 90.00th=[36963], 95.00th=[38011],
     | 99.00th=[40633], 99.50th=[50594], 99.90th=[59507], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min= 1680, max= 3664, per=25.02%, avg=2545.00, stdev=738.50, samples=8
   iops        : min=  420, max=  916, avg=636.25, stdev=184.62, samples=8
  lat (msec)   : 10=0.35%, 20=39.53%, 50=59.26%, 100=0.86%
  cpu          : usr=0.55%, sys=4.77%, ctx=1699, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18578: Fri Jul 19 13:42:35 2024
  write: IOPS=637, BW=2551KiB/s (2612kB/s)(10.0MiB/4014msec)
    slat (usec): min=12, max=720, avg=65.65, stdev=44.46
    clat (usec): min=8124, max=68193, avg=24971.52, stdev=8825.78
     lat (usec): min=8143, max=68221, avg=25038.83, stdev=8840.30
    clat percentiles (usec):
     |  1.00th=[10814],  5.00th=[13304], 10.00th=[15139], 20.00th=[16712],
     | 30.00th=[17957], 40.00th=[19792], 50.00th=[23462], 60.00th=[26870],
     | 70.00th=[31065], 80.00th=[35390], 90.00th=[36963], 95.00th=[38011],
     | 99.00th=[46924], 99.50th=[48497], 99.90th=[67634], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min= 1712, max= 3728, per=25.02%, avg=2545.00, stdev=756.10, samples=8
   iops        : min=  428, max=  932, avg=636.25, stdev=189.03, samples=8
  lat (msec)   : 10=0.20%, 20=40.27%, 50=59.06%, 100=0.47%
  cpu          : usr=0.67%, sys=4.59%, ctx=1862, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=9.93MiB/s (10.4MB/s), 2543KiB/s-2551KiB/s (2604kB/s-2612kB/s), io=40.0MiB (41.9MB), run=4014-4027msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=18591: Fri Jul 19 13:42:38 2024
   read: IOPS=654, BW=2618KiB/s (2681kB/s)(5048KiB/1928msec)
    slat (usec): min=422, max=1914, avg=728.05, stdev=158.01
    clat (usec): min=6597, max=20639, avg=11318.50, stdev=1919.34
     lat (usec): min=7111, max=21635, avg=12048.33, stdev=1982.93
    clat percentiles (usec):
     |  1.00th=[ 7635],  5.00th=[ 8586], 10.00th=[ 9241], 20.00th=[ 9765],
     | 30.00th=[10290], 40.00th=[10683], 50.00th=[11207], 60.00th=[11600],
     | 70.00th=[11994], 80.00th=[12518], 90.00th=[13435], 95.00th=[14746],
     | 99.00th=[17957], 99.50th=[18482], 99.90th=[20317], 99.95th=[20579],
     | 99.99th=[20579]
   bw (  KiB/s): min= 2360, max= 2688, per=24.65%, avg=2565.33, stdev=178.95, samples=3
   iops        : min=  590, max=  672, avg=641.33, stdev=44.74, samples=3
  write: IOPS=673, BW=2693KiB/s (2758kB/s)(5192KiB/1928msec)
    slat (usec): min=12, max=752, avg=41.53, stdev=31.40
    clat (usec): min=7371, max=20564, avg=11790.13, stdev=1676.68
     lat (usec): min=7413, max=20628, avg=11832.78, stdev=1679.88
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[ 9634], 10.00th=[10159], 20.00th=[10552],
     | 30.00th=[10945], 40.00th=[11338], 50.00th=[11600], 60.00th=[11863],
     | 70.00th=[12256], 80.00th=[12649], 90.00th=[13435], 95.00th=[15270],
     | 99.00th=[18220], 99.50th=[18744], 99.90th=[19530], 99.95th=[20579],
     | 99.99th=[20579]
   bw (  KiB/s): min= 2544, max= 2840, per=25.03%, avg=2712.00, stdev=152.00, samples=3
   iops        : min=  636, max=  710, avg=678.00, stdev=38.00, samples=3
  lat (msec)   : 10=16.45%, 20=83.44%, 50=0.12%
  cpu          : usr=1.56%, sys=9.08%, ctx=2897, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18592: Fri Jul 19 13:42:38 2024
   read: IOPS=648, BW=2596KiB/s (2658kB/s)(4976KiB/1917msec)
    slat (usec): min=24, max=2127, avg=723.87, stdev=167.23
    clat (usec): min=658, max=23561, avg=11285.13, stdev=2090.57
     lat (usec): min=1429, max=24679, avg=12010.61, stdev=2152.14
    clat percentiles (usec):
     |  1.00th=[ 6980],  5.00th=[ 8586], 10.00th=[ 9241], 20.00th=[ 9896],
     | 30.00th=[10290], 40.00th=[10683], 50.00th=[11076], 60.00th=[11469],
     | 70.00th=[11994], 80.00th=[12649], 90.00th=[13435], 95.00th=[14615],
     | 99.00th=[18482], 99.50th=[19792], 99.90th=[21890], 99.95th=[23462],
     | 99.99th=[23462]
   bw (  KiB/s): min= 2384, max= 2632, per=24.49%, avg=2549.33, stdev=143.18, samples=3
   iops        : min=  596, max=  658, avg=637.33, stdev=35.80, samples=3
  write: IOPS=686, BW=2746KiB/s (2812kB/s)(5264KiB/1917msec)
    slat (usec): min=13, max=602, avg=42.91, stdev=30.49
    clat (usec): min=5713, max=21948, avg=11827.89, stdev=1755.08
     lat (usec): min=5771, max=21997, avg=11871.86, stdev=1755.92
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[ 9503], 10.00th=[10028], 20.00th=[10552],
     | 30.00th=[10945], 40.00th=[11338], 50.00th=[11731], 60.00th=[11994],
     | 70.00th=[12256], 80.00th=[12649], 90.00th=[13435], 95.00th=[15008],
     | 99.00th=[18744], 99.50th=[19530], 99.90th=[21103], 99.95th=[21890],
     | 99.99th=[21890]
   bw (  KiB/s): min= 2624, max= 2888, per=25.57%, avg=2770.67, stdev=134.42, samples=3
   iops        : min=  656, max=  722, avg=692.67, stdev=33.61, samples=3
  lat (usec)   : 750=0.04%
  lat (msec)   : 2=0.08%, 4=0.12%, 10=16.41%, 20=82.97%, 50=0.39%
  cpu          : usr=1.46%, sys=9.29%, ctx=2862, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18593: Fri Jul 19 13:42:38 2024
   read: IOPS=646, BW=2586KiB/s (2648kB/s)(4984KiB/1927msec)
    slat (usec): min=393, max=2191, avg=726.36, stdev=164.11
    clat (usec): min=2040, max=21530, avg=11357.33, stdev=1953.80
     lat (usec): min=2605, max=23421, avg=12085.33, stdev=2013.67
    clat percentiles (usec):
     |  1.00th=[ 7898],  5.00th=[ 8848], 10.00th=[ 9241], 20.00th=[ 9896],
     | 30.00th=[10290], 40.00th=[10683], 50.00th=[11076], 60.00th=[11600],
     | 70.00th=[11994], 80.00th=[12649], 90.00th=[13566], 95.00th=[14615],
     | 99.00th=[18482], 99.50th=[19268], 99.90th=[20841], 99.95th=[21627],
     | 99.99th=[21627]
   bw (  KiB/s): min= 2472, max= 2688, per=24.93%, avg=2594.67, stdev=110.95, samples=3
   iops        : min=  618, max=  672, avg=648.67, stdev=27.74, samples=3
  write: IOPS=681, BW=2728KiB/s (2793kB/s)(5256KiB/1927msec)
    slat (usec): min=11, max=316, avg=42.33, stdev=27.54
    clat (usec): min=1825, max=20652, avg=11747.39, stdev=1760.14
     lat (usec): min=2000, max=20670, avg=11790.96, stdev=1763.83
    clat percentiles (usec):
     |  1.00th=[ 8455],  5.00th=[ 9634], 10.00th=[ 9896], 20.00th=[10552],
     | 30.00th=[10814], 40.00th=[11338], 50.00th=[11600], 60.00th=[11863],
     | 70.00th=[12256], 80.00th=[12649], 90.00th=[13566], 95.00th=[14746],
     | 99.00th=[18220], 99.50th=[19006], 99.90th=[20317], 99.95th=[20579],
     | 99.99th=[20579]
   bw (  KiB/s): min= 2464, max= 2824, per=24.81%, avg=2688.00, stdev=195.47, samples=3
   iops        : min=  616, max=  706, avg=672.00, stdev=48.87, samples=3
  lat (msec)   : 2=0.04%, 4=0.08%, 10=16.88%, 20=82.73%, 50=0.27%
  cpu          : usr=1.35%, sys=9.19%, ctx=2851, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18594: Fri Jul 19 13:42:38 2024
   read: IOPS=656, BW=2626KiB/s (2689kB/s)(5060KiB/1927msec)
    slat (usec): min=406, max=2420, avg=725.29, stdev=167.79
    clat (usec): min=6558, max=21635, avg=11311.55, stdev=1955.63
     lat (usec): min=7251, max=23388, avg=12038.52, stdev=2010.53
    clat percentiles (usec):
     |  1.00th=[ 7439],  5.00th=[ 8717], 10.00th=[ 9241], 20.00th=[ 9765],
     | 30.00th=[10290], 40.00th=[10683], 50.00th=[11207], 60.00th=[11600],
     | 70.00th=[11994], 80.00th=[12649], 90.00th=[13304], 95.00th=[14615],
     | 99.00th=[18482], 99.50th=[19530], 99.90th=[20317], 99.95th=[21627],
     | 99.99th=[21627]
   bw (  KiB/s): min= 2344, max= 2776, per=25.01%, avg=2602.67, stdev=228.29, samples=3
   iops        : min=  586, max=  694, avg=650.67, stdev=57.07, samples=3
  write: IOPS=672, BW=2688KiB/s (2753kB/s)(5180KiB/1927msec)
    slat (usec): min=12, max=723, avg=41.57, stdev=34.48
    clat (usec): min=4245, max=19933, avg=11813.59, stdev=1775.09
     lat (usec): min=4288, max=19987, avg=11856.13, stdev=1783.15
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[ 9503], 10.00th=[10028], 20.00th=[10552],
     | 30.00th=[10945], 40.00th=[11338], 50.00th=[11600], 60.00th=[11994],
     | 70.00th=[12256], 80.00th=[12780], 90.00th=[13566], 95.00th=[15008],
     | 99.00th=[18482], 99.50th=[18744], 99.90th=[19530], 99.95th=[20055],
     | 99.99th=[20055]
   bw (  KiB/s): min= 2544, max= 2792, per=24.63%, avg=2669.33, stdev=124.02, samples=3
   iops        : min=  636, max=  698, avg=667.33, stdev=31.01, samples=3
  lat (msec)   : 10=17.27%, 20=82.66%, 50=0.08%
  cpu          : usr=1.61%, sys=8.77%, ctx=2907, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=10.2MiB/s (10.7MB/s), 2586KiB/s-2626KiB/s (2648kB/s-2689kB/s), io=19.6MiB (20.5MB), run=1917-1928msec
  WRITE: bw=10.6MiB/s (11.1MB/s), 2688KiB/s-2746KiB/s (2753kB/s-2812kB/s), io=20.4MiB (21.4MB), run=1917-1928msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=18599: Fri Jul 19 13:42:40 2024
   read: IOPS=9, BW=377MiB/s (396MB/s)(40.0MiB/106msec)
    slat (nsec): min=7404.7k, max=7404.7k, avg=7404698.00, stdev= 0.00
    clat (nsec): min=94960k, max=94960k, avg=94959787.00, stdev= 0.00
     lat (nsec): min=102376k, max=102376k, avg=102376315.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[94897],  5.00th=[94897], 10.00th=[94897], 20.00th=[94897],
     | 30.00th=[94897], 40.00th=[94897], 50.00th=[94897], 60.00th=[94897],
     | 70.00th=[94897], 80.00th=[94897], 90.00th=[94897], 95.00th=[94897],
     | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897],
     | 99.99th=[94897]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.67%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=377MiB/s (396MB/s), 377MiB/s-377MiB/s (396MB/s-396MB/s), io=40.0MiB (41.9MB), run=106-106msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck