== sanity test 398c: run fio to test AIO ================= 08:18:52 (1717157932)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.10244 s, 409 MB/s
osc.lustre-OST0000-osc-ffff880136b03000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=20242: Fri May 31 08:19:25 2024
  write: IOPS=81, BW=327KiB/s (335kB/s)(10.0MiB/31275msec)
    slat (usec): min=11, max=324, avg=47.27, stdev=25.67
    clat (msec): min=38, max=358, avg=195.33, stdev=46.26
     lat (msec): min=38, max=358, avg=195.38, stdev=46.26
    clat percentiles (msec):
     |  1.00th=[  102],  5.00th=[  132], 10.00th=[  142], 20.00th=[  157],
     | 30.00th=[  169], 40.00th=[  180], 50.00th=[  190], 60.00th=[  201],
     | 70.00th=[  209], 80.00th=[  230], 90.00th=[  259], 95.00th=[  292],
     | 99.00th=[  326], 99.50th=[  330], 99.90th=[  342], 99.95th=[  347],
     | 99.99th=[  359]
   bw (  KiB/s): min=  192, max=  472, per=24.98%, avg=326.73, stdev=64.43, samples=62
   iops        : min=   48, max=  118, avg=81.60, stdev=16.12, samples=62
  lat (msec)   : 50=0.08%, 100=0.86%, 250=85.98%, 500=13.09%
  cpu          : usr=0.10%, sys=0.51%, ctx=2073, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=20243: Fri May 31 08:19:25 2024
  write: IOPS=81, BW=327KiB/s (335kB/s)(10.0MiB/31312msec)
    slat (usec): min=11, max=479, avg=47.58, stdev=25.47
    clat (msec): min=79, max=366, avg=195.54, stdev=46.09
     lat (msec): min=79, max=366, avg=195.59, stdev=46.09
    clat percentiles (msec):
     |  1.00th=[  106],  5.00th=[  133], 10.00th=[  144], 20.00th=[  159],
     | 30.00th=[  171], 40.00th=[  180], 50.00th=[  190], 60.00th=[  199],
     | 70.00th=[  209], 80.00th=[  230], 90.00th=[  257], 95.00th=[  288],
     | 99.00th=[  330], 99.50th=[  351], 99.90th=[  359], 99.95th=[  363],
     | 99.99th=[  368]
   bw (  KiB/s): min=  176, max=  440, per=24.93%, avg=326.06, stdev=62.13, samples=62
   iops        : min=   44, max=  110, avg=81.52, stdev=15.53, samples=62
  lat (msec)   : 100=0.70%, 250=86.76%, 500=12.54%
  cpu          : usr=0.10%, sys=0.53%, ctx=2162, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=20244: Fri May 31 08:19:25 2024
  write: IOPS=81, BW=327KiB/s (335kB/s)(10.0MiB/31312msec)
    slat (usec): min=12, max=277, avg=47.72, stdev=23.74
    clat (msec): min=79, max=364, avg=195.53, stdev=46.28
     lat (msec): min=79, max=364, avg=195.58, stdev=46.28
    clat percentiles (msec):
     |  1.00th=[  111],  5.00th=[  133], 10.00th=[  142], 20.00th=[  159],
     | 30.00th=[  169], 40.00th=[  180], 50.00th=[  190], 60.00th=[  199],
     | 70.00th=[  209], 80.00th=[  230], 90.00th=[  257], 95.00th=[  288],
     | 99.00th=[  330], 99.50th=[  338], 99.90th=[  355], 99.95th=[  363],
     | 99.99th=[  363]
   bw (  KiB/s): min=  200, max=  456, per=24.90%, avg=325.74, stdev=62.05, samples=62
   iops        : min=   50, max=  114, avg=81.39, stdev=15.49, samples=62
  lat (msec)   : 100=0.70%, 250=86.37%, 500=12.93%
  cpu          : usr=0.07%, sys=0.56%, ctx=2319, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=20245: Fri May 31 08:19:25 2024
  write: IOPS=81, BW=327KiB/s (335kB/s)(10.0MiB/31312msec)
    slat (usec): min=12, max=277, avg=46.76, stdev=24.23
    clat (msec): min=65, max=366, avg=195.53, stdev=46.05
     lat (msec): min=65, max=366, avg=195.58, stdev=46.05
    clat percentiles (msec):
     |  1.00th=[  106],  5.00th=[  132], 10.00th=[  144], 20.00th=[  159],
     | 30.00th=[  171], 40.00th=[  180], 50.00th=[  190], 60.00th=[  199],
     | 70.00th=[  209], 80.00th=[  230], 90.00th=[  257], 95.00th=[  288],
     | 99.00th=[  330], 99.50th=[  342], 99.90th=[  363], 99.95th=[  363],
     | 99.99th=[  368]
   bw (  KiB/s): min=  200, max=  448, per=24.89%, avg=325.50, stdev=61.11, samples=62
   iops        : min=   50, max=  112, avg=81.34, stdev=15.27, samples=62
  lat (msec)   : 100=0.82%, 250=86.95%, 500=12.23%
  cpu          : usr=0.14%, sys=0.49%, ctx=2228, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1308KiB/s (1340kB/s), 327KiB/s-327KiB/s (335kB/s-335kB/s), io=40.0MiB (41.9MB), run=31275-31312msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=20280: Fri May 31 08:19:44 2024
   read: IOPS=71, BW=288KiB/s (295kB/s)(5048KiB/17538msec)
    slat (usec): min=479, max=2228, avg=788.29, stdev=211.76
    clat (msec): min=2, max=169, avg=75.07, stdev=15.28
     lat (msec): min=3, max=170, avg=75.86, stdev=15.32
    clat percentiles (msec):
     |  1.00th=[   47],  5.00th=[   54], 10.00th=[   59], 20.00th=[   64],
     | 30.00th=[   67], 40.00th=[   72], 50.00th=[   75], 60.00th=[   78],
     | 70.00th=[   82], 80.00th=[   87], 90.00th=[   92], 95.00th=[   99],
     | 99.00th=[  113], 99.50th=[  133], 99.90th=[  169], 99.95th=[  169],
     | 99.99th=[  169]
   bw (  KiB/s): min=  152, max=  400, per=25.24%, avg=287.29, stdev=58.06, samples=35
   iops        : min=   38, max=  100, avg=71.80, stdev=14.52, samples=35
  write: IOPS=74, BW=296KiB/s (303kB/s)(5192KiB/17538msec)
    slat (usec): min=12, max=176, avg=44.07, stdev=21.44
    clat (msec): min=75, max=243, avg=142.12, stdev=30.86
     lat (msec): min=75, max=243, avg=142.17, stdev=30.86
    clat percentiles (msec):
     |  1.00th=[   86],  5.00th=[   96], 10.00th=[  105], 20.00th=[  116],
     | 30.00th=[  125], 40.00th=[  132], 50.00th=[  138], 60.00th=[  146],
     | 70.00th=[  157], 80.00th=[  167], 90.00th=[  184], 95.00th=[  199],
     | 99.00th=[  226], 99.50th=[  232], 99.90th=[  243], 99.95th=[  245],
     | 99.99th=[  245]
   bw (  KiB/s): min=  184, max=  400, per=24.84%, avg=294.37, stdev=39.57, samples=35
   iops        : min=   46, max=  100, avg=73.57, stdev= 9.90, samples=35
  lat (msec)   : 4=0.04%, 20=0.31%, 50=0.78%, 100=49.41%, 250=49.45%
  cpu          : usr=0.23%, sys=1.26%, ctx=2768, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=20281: Fri May 31 08:19:44 2024
   read: IOPS=70, BW=283KiB/s (290kB/s)(4976KiB/17574msec)
    slat (usec): min=22, max=2923, avg=793.23, stdev=236.38
    clat (usec): min=1138, max=169257, avg=75329.31, stdev=15074.75
     lat (usec): min=1294, max=171045, avg=76124.35, stdev=15126.66
    clat percentiles (msec):
     |  1.00th=[   47],  5.00th=[   56], 10.00th=[   61], 20.00th=[   64],
     | 30.00th=[   68], 40.00th=[   72], 50.00th=[   74], 60.00th=[   77],
     | 70.00th=[   81], 80.00th=[   87], 90.00th=[   93], 95.00th=[  101],
     | 99.00th=[  116], 99.50th=[  123], 99.90th=[  169], 99.95th=[  169],
     | 99.99th=[  169]
   bw (  KiB/s): min=  152, max=  392, per=24.86%, avg=282.94, stdev=58.26, samples=35
   iops        : min=   38, max=   98, avg=70.71, stdev=14.58, samples=35
  write: IOPS=74, BW=300KiB/s (307kB/s)(5264KiB/17574msec)
    slat (usec): min=12, max=274, avg=45.78, stdev=23.75
    clat (msec): min=21, max=262, avg=141.53, stdev=31.40
     lat (msec): min=21, max=262, avg=141.58, stdev=31.41
    clat percentiles (msec):
     |  1.00th=[   83],  5.00th=[   95], 10.00th=[  105], 20.00th=[  115],
     | 30.00th=[  125], 40.00th=[  131], 50.00th=[  138], 60.00th=[  148],
     | 70.00th=[  159], 80.00th=[  169], 90.00th=[  184], 95.00th=[  197],
     | 99.00th=[  222], 99.50th=[  232], 99.90th=[  245], 99.95th=[  264],
     | 99.99th=[  264]
   bw (  KiB/s): min=  232, max=  432, per=25.21%, avg=298.71, stdev=46.36, samples=35
   iops        : min=   58, max=  108, avg=74.66, stdev=11.56, samples=35
  lat (msec)   : 2=0.08%, 20=0.12%, 50=0.90%, 100=48.98%, 250=49.88%
  lat (msec)   : 500=0.04%
  cpu          : usr=0.18%, sys=1.32%, ctx=2733, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=20282: Fri May 31 08:19:44 2024
   read: IOPS=70, BW=283KiB/s (290kB/s)(4984KiB/17622msec)
    slat (usec): min=435, max=2672, avg=792.28, stdev=219.49
    clat (msec): min=11, max=168, avg=75.79, stdev=15.57
     lat (msec): min=12, max=169, avg=76.58, stdev=15.62
    clat percentiles (msec):
     |  1.00th=[   43],  5.00th=[   56], 10.00th=[   60], 20.00th=[   64],
     | 30.00th=[   68], 40.00th=[   72], 50.00th=[   75], 60.00th=[   79],
     | 70.00th=[   82], 80.00th=[   88], 90.00th=[   93], 95.00th=[  101],
     | 99.00th=[  116], 99.50th=[  140], 99.90th=[  167], 99.95th=[  169],
     | 99.99th=[  169]
   bw (  KiB/s): min=  192, max=  440, per=24.76%, avg=281.80, stdev=56.89, samples=35
   iops        : min=   48, max=  110, avg=70.43, stdev=14.22, samples=35
  write: IOPS=74, BW=298KiB/s (305kB/s)(5256KiB/17622msec)
    slat (usec): min=13, max=231, avg=45.58, stdev=20.49
    clat (msec): min=35, max=251, avg=141.71, stdev=31.93
     lat (msec): min=35, max=251, avg=141.76, stdev=31.93
    clat percentiles (msec):
     |  1.00th=[   77],  5.00th=[   95], 10.00th=[  104], 20.00th=[  114],
     | 30.00th=[  124], 40.00th=[  132], 50.00th=[  140], 60.00th=[  148],
     | 70.00th=[  159], 80.00th=[  169], 90.00th=[  186], 95.00th=[  199],
     | 99.00th=[  220], 99.50th=[  232], 99.90th=[  243], 99.95th=[  251],
     | 99.99th=[  251]
   bw (  KiB/s): min=  192, max=  376, per=24.86%, avg=294.60, stdev=41.64, samples=35
   iops        : min=   48, max=   94, avg=73.63, stdev=10.42, samples=35
  lat (msec)   : 20=0.31%, 50=0.82%, 100=48.79%, 250=50.04%, 500=0.04%
  cpu          : usr=0.16%, sys=1.37%, ctx=2659, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=20283: Fri May 31 08:19:44 2024
   read: IOPS=72, BW=288KiB/s (295kB/s)(5060KiB/17561msec)
    slat (usec): min=459, max=2735, avg=800.82, stdev=238.70
    clat (msec): min=12, max=170, avg=75.71, stdev=15.14
     lat (msec): min=13, max=171, avg=76.51, stdev=15.20
    clat percentiles (msec):
     |  1.00th=[   46],  5.00th=[   56], 10.00th=[   61], 20.00th=[   64],
     | 30.00th=[   68], 40.00th=[   72], 50.00th=[   75], 60.00th=[   78],
     | 70.00th=[   82], 80.00th=[   88], 90.00th=[   93], 95.00th=[  100],
     | 99.00th=[  116], 99.50th=[  138], 99.90th=[  171], 99.95th=[  171],
     | 99.99th=[  171]
   bw (  KiB/s): min=  199, max=  392, per=25.33%, avg=288.20, stdev=52.49, samples=35
   iops        : min=   49, max=   98, avg=72.03, stdev=13.16, samples=35
  write: IOPS=73, BW=295KiB/s (302kB/s)(5180KiB/17561msec)
    slat (usec): min=13, max=881, avg=47.67, stdev=34.65
    clat (msec): min=53, max=251, avg=141.96, stdev=30.85
     lat (msec): min=53, max=251, avg=142.01, stdev=30.85
    clat percentiles (msec):
     |  1.00th=[   83],  5.00th=[   99], 10.00th=[  105], 20.00th=[  115],
     | 30.00th=[  125], 40.00th=[  132], 50.00th=[  138], 60.00th=[  146],
     | 70.00th=[  157], 80.00th=[  169], 90.00th=[  184], 95.00th=[  194],
     | 99.00th=[  228], 99.50th=[  249], 99.90th=[  251], 99.95th=[  251],
     | 99.99th=[  251]
   bw (  KiB/s): min=  192, max=  367, per=24.76%, avg=293.46, stdev=41.55, samples=35
   iops        : min=   48, max=   91, avg=73.34, stdev=10.35, samples=35
  lat (msec)   : 20=0.16%, 50=1.02%, 100=49.06%, 250=49.65%, 500=0.12%
  cpu          : usr=0.16%, sys=1.38%, ctx=2774, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1139KiB/s (1166kB/s), 283KiB/s-288KiB/s (290kB/s-295kB/s), io=19.6MiB (20.5MB), run=17538-17622msec
  WRITE: bw=1186KiB/s (1214kB/s), 295KiB/s-300KiB/s (302kB/s-307kB/s), io=20.4MiB (21.4MB), run=17538-17622msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=20301: Fri May 31 08:19:45 2024
   read: IOPS=5, BW=235MiB/s (247MB/s)(40.0MiB/170msec)
    slat (nsec): min=14871k, max=14871k, avg=14871413.00, stdev= 0.00
    clat (nsec): min=149171k, max=149171k, avg=149171496.00, stdev= 0.00
     lat (nsec): min=164063k, max=164063k, avg=164062949.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  150],  5.00th=[  150], 10.00th=[  150], 20.00th=[  150],
     | 30.00th=[  150], 40.00th=[  150], 50.00th=[  150], 60.00th=[  150],
     | 70.00th=[  150], 80.00th=[  150], 90.00th=[  150], 95.00th=[  150],
     | 99.00th=[  150], 99.50th=[  150], 99.90th=[  150], 99.95th=[  150],
     | 99.99th=[  150]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=8.88%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=235MiB/s (247MB/s), 235MiB/s-235MiB/s (247MB/s-247MB/s), io=40.0MiB (41.9MB), run=170-170msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck