== sanity test 398c: run fio to test AIO ================= 16:03:06 (1715457786)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.160881 s, 261 MB/s
osc.lustre-OST0000-osc-ffff880087ce0000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=22293: Sat May 11 16:03:34 2024
  write: IOPS=97, BW=391KiB/s (401kB/s)(10.0MiB/26181msec)
    slat (usec): min=11, max=277, avg=50.28, stdev=27.09
    clat (msec): min=81, max=265, avg=163.51, stdev=28.43
     lat (msec): min=81, max=265, avg=163.56, stdev=28.43
    clat percentiles (msec):
     |  1.00th=[  108],  5.00th=[  116], 10.00th=[  125], 20.00th=[  140],
     | 30.00th=[  148], 40.00th=[  157], 50.00th=[  163], 60.00th=[  171],
     | 70.00th=[  178], 80.00th=[  188], 90.00th=[  201], 95.00th=[  211],
     | 99.00th=[  232], 99.50th=[  241], 99.90th=[  255], 99.95th=[  264],
     | 99.99th=[  266]
   bw (  KiB/s): min=  312, max=  528, per=24.91%, avg=389.60, stdev=46.69, samples=52
   iops        : min=   78, max=  132, avg=97.33, stdev=11.64, samples=52
  lat (msec)   : 100=0.43%, 250=99.34%, 500=0.23%
  cpu          : usr=0.11%, sys=0.55%, ctx=2448, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=22294: Sat May 11 16:03:34 2024
  write: IOPS=98, BW=393KiB/s (403kB/s)(10.0MiB/26039msec)
    slat (usec): min=12, max=304, avg=50.28, stdev=26.05
    clat (msec): min=26, max=289, avg=162.61, stdev=30.12
     lat (msec): min=26, max=289, avg=162.66, stdev=30.12
    clat percentiles (msec):
     |  1.00th=[   99],  5.00th=[  113], 10.00th=[  123], 20.00th=[  138],
     | 30.00th=[  148], 40.00th=[  157], 50.00th=[  163], 60.00th=[  169],
     | 70.00th=[  178], 80.00th=[  188], 90.00th=[  201], 95.00th=[  213],
     | 99.00th=[  236], 99.50th=[  247], 99.90th=[  279], 99.95th=[  292],
     | 99.99th=[  292]
   bw (  KiB/s): min=  312, max=  552, per=25.03%, avg=391.50, stdev=50.92, samples=52
   iops        : min=   78, max=  138, avg=97.85, stdev=12.74, samples=52
  lat (msec)   : 50=0.04%, 100=1.17%, 250=98.48%, 500=0.31%
  cpu          : usr=0.12%, sys=0.60%, ctx=2114, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=22295: Sat May 11 16:03:34 2024
  write: IOPS=97, BW=391KiB/s (400kB/s)(10.0MiB/26183msec)
    slat (usec): min=11, max=206, avg=48.24, stdev=26.51
    clat (msec): min=81, max=263, avg=163.51, stdev=28.67
     lat (msec): min=81, max=263, avg=163.56, stdev=28.67
    clat percentiles (msec):
     |  1.00th=[  106],  5.00th=[  115], 10.00th=[  126], 20.00th=[  140],
     | 30.00th=[  150], 40.00th=[  157], 50.00th=[  163], 60.00th=[  171],
     | 70.00th=[  178], 80.00th=[  186], 90.00th=[  201], 95.00th=[  211],
     | 99.00th=[  239], 99.50th=[  241], 99.90th=[  253], 99.95th=[  255],
     | 99.99th=[  264]
   bw (  KiB/s): min=  312, max=  528, per=24.93%, avg=389.96, stdev=47.69, samples=52
   iops        : min=   78, max=  132, avg=97.46, stdev=11.93, samples=52
  lat (msec)   : 100=0.66%, 250=99.18%, 500=0.16%
  cpu          : usr=0.13%, sys=0.53%, ctx=2034, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=22296: Sat May 11 16:03:34 2024
  write: IOPS=97, BW=392KiB/s (401kB/s)(10.0MiB/26141msec)
    slat (usec): min=11, max=239, avg=48.74, stdev=25.30
    clat (msec): min=43, max=297, avg=163.24, stdev=29.78
     lat (msec): min=43, max=297, avg=163.29, stdev=29.78
    clat percentiles (msec):
     |  1.00th=[  102],  5.00th=[  115], 10.00th=[  125], 20.00th=[  138],
     | 30.00th=[  148], 40.00th=[  157], 50.00th=[  163], 60.00th=[  169],
     | 70.00th=[  178], 80.00th=[  188], 90.00th=[  201], 95.00th=[  215],
     | 99.00th=[  239], 99.50th=[  247], 99.90th=[  288], 99.95th=[  288],
     | 99.99th=[  296]
   bw (  KiB/s): min=  296, max=  592, per=25.03%, avg=391.50, stdev=53.77, samples=52
   iops        : min=   74, max=  148, avg=97.85, stdev=13.45, samples=52
  lat (msec)   : 50=0.04%, 100=0.62%, 250=98.91%, 500=0.43%
  cpu          : usr=0.11%, sys=0.59%, ctx=1996, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1564KiB/s (1602kB/s), 391KiB/s-393KiB/s (400kB/s-403kB/s), io=40.0MiB (41.9MB), run=26039-26183msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=22326: Sat May 11 16:03:49 2024
   read: IOPS=85, BW=342KiB/s (350kB/s)(5048KiB/14767msec)
    slat (usec): min=77, max=2666, avg=765.68, stdev=218.91
    clat (msec): min=9, max=162, avg=63.35, stdev=14.32
     lat (msec): min=9, max=163, avg=64.12, stdev=14.37
    clat percentiles (msec):
     |  1.00th=[   15],  5.00th=[   45], 10.00th=[   50], 20.00th=[   54],
     | 30.00th=[   58], 40.00th=[   62], 50.00th=[   64], 60.00th=[   66],
     | 70.00th=[   69], 80.00th=[   72], 90.00th=[   77], 95.00th=[   83],
     | 99.00th=[  105], 99.50th=[  150], 99.90th=[  163], 99.95th=[  163],
     | 99.99th=[  163]
   bw (  KiB/s): min=  192, max=  440, per=25.25%, avg=342.86, stdev=57.23, samples=29
   iops        : min=   48, max=  110, avg=85.69, stdev=14.27, samples=29
  write: IOPS=87, BW=352KiB/s (360kB/s)(5192KiB/14767msec)
    slat (usec): min=11, max=651, avg=47.15, stdev=28.13
    clat (msec): min=29, max=223, avg=119.46, stdev=26.78
     lat (msec): min=30, max=223, avg=119.51, stdev=26.77
    clat percentiles (msec):
     |  1.00th=[   67],  5.00th=[   81], 10.00th=[   89], 20.00th=[   99],
     | 30.00th=[  104], 40.00th=[  111], 50.00th=[  117], 60.00th=[  124],
     | 70.00th=[  130], 80.00th=[  140], 90.00th=[  157], 95.00th=[  171],
     | 99.00th=[  194], 99.50th=[  199], 99.90th=[  224], 99.95th=[  224],
     | 99.99th=[  224]
   bw (  KiB/s): min=  304, max=  424, per=24.87%, avg=351.41, stdev=37.91, samples=29
   iops        : min=   76, max=  106, avg=87.83, stdev= 9.50, samples=29
  lat (msec)   : 10=0.08%, 20=0.47%, 50=5.00%, 100=55.39%, 250=39.06%
  cpu          : usr=0.27%, sys=1.38%, ctx=3001, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=22327: Sat May 11 16:03:49 2024
   read: IOPS=84, BW=337KiB/s (345kB/s)(4976KiB/14776msec)
    slat (usec): min=457, max=2534, avg=760.23, stdev=207.21
    clat (usec): min=1451, max=151899, avg=63271.24, stdev=12435.33
     lat (msec): min=2, max=154, avg=64.03, stdev=12.48
    clat percentiles (msec):
     |  1.00th=[   41],  5.00th=[   46], 10.00th=[   51], 20.00th=[   54],
     | 30.00th=[   57], 40.00th=[   61], 50.00th=[   63], 60.00th=[   66],
     | 70.00th=[   69], 80.00th=[   72], 90.00th=[   78], 95.00th=[   83],
     | 99.00th=[   94], 99.50th=[  102], 99.90th=[  150], 99.95th=[  153],
     | 99.99th=[  153]
   bw (  KiB/s): min=  200, max=  448, per=24.76%, avg=336.28, stdev=60.11, samples=29
   iops        : min=   50, max=  112, avg=84.07, stdev=15.03, samples=29
  write: IOPS=89, BW=356KiB/s (365kB/s)(5264KiB/14776msec)
    slat (usec): min=11, max=200, avg=44.94, stdev=20.14
    clat (msec): min=34, max=203, avg=118.87, stdev=25.11
     lat (msec): min=34, max=203, avg=118.91, stdev=25.11
    clat percentiles (msec):
     |  1.00th=[   70],  5.00th=[   82], 10.00th=[   89], 20.00th=[   97],
     | 30.00th=[  105], 40.00th=[  110], 50.00th=[  117], 60.00th=[  124],
     | 70.00th=[  130], 80.00th=[  138], 90.00th=[  155], 95.00th=[  167],
     | 99.00th=[  184], 99.50th=[  190], 99.90th=[  205], 99.95th=[  205],
     | 99.99th=[  205]
   bw (  KiB/s): min=  296, max=  424, per=25.09%, avg=354.48, stdev=38.73, samples=29
   iops        : min=   74, max=  106, avg=88.62, stdev= 9.68, samples=29
  lat (msec)   : 2=0.04%, 4=0.04%, 20=0.12%, 50=4.77%, 100=55.90%
  lat (msec)   : 250=39.14%
  cpu          : usr=0.17%, sys=1.46%, ctx=2844, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=22328: Sat May 11 16:03:49 2024
   read: IOPS=84, BW=338KiB/s (346kB/s)(4984KiB/14767msec)
    slat (usec): min=448, max=3176, avg=759.26, stdev=201.73
    clat (msec): min=9, max=155, avg=63.81, stdev=12.92
     lat (msec): min=9, max=157, avg=64.57, stdev=12.98
    clat percentiles (msec):
     |  1.00th=[   39],  5.00th=[   47], 10.00th=[   52], 20.00th=[   55],
     | 30.00th=[   58], 40.00th=[   61], 50.00th=[   64], 60.00th=[   66],
     | 70.00th=[   69], 80.00th=[   72], 90.00th=[   78], 95.00th=[   83],
     | 99.00th=[  100], 99.50th=[  146], 99.90th=[  153], 99.95th=[  157],
     | 99.99th=[  157]
   bw (  KiB/s): min=  232, max=  448, per=25.05%, avg=340.14, stdev=53.09, samples=29
   iops        : min=   58, max=  112, avg=85.03, stdev=13.27, samples=29
  write: IOPS=88, BW=356KiB/s (364kB/s)(5256KiB/14767msec)
    slat (usec): min=11, max=247, avg=45.72, stdev=20.32
    clat (msec): min=46, max=225, avg=118.24, stdev=26.02
     lat (msec): min=46, max=225, avg=118.29, stdev=26.02
    clat percentiles (msec):
     |  1.00th=[   69],  5.00th=[   82], 10.00th=[   87], 20.00th=[   96],
     | 30.00th=[  104], 40.00th=[  109], 50.00th=[  115], 60.00th=[  123],
     | 70.00th=[  130], 80.00th=[  140], 90.00th=[  155], 95.00th=[  167],
     | 99.00th=[  188], 99.50th=[  194], 99.90th=[  205], 99.95th=[  226],
     | 99.99th=[  226]
   bw (  KiB/s): min=  272, max=  456, per=25.09%, avg=354.48, stdev=38.37, samples=29
   iops        : min=   68, max=  114, avg=88.62, stdev= 9.59, samples=29
  lat (msec)   : 10=0.08%, 20=0.16%, 50=3.91%, 100=57.30%, 250=38.55%
  cpu          : usr=0.24%, sys=1.48%, ctx=2837, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=22329: Sat May 11 16:03:49 2024
   read: IOPS=85, BW=343KiB/s (351kB/s)(5060KiB/14750msec)
    slat (usec): min=425, max=2620, avg=764.30, stdev=211.93
    clat (usec): min=762, max=154783, avg=63092.12, stdev=13759.05
     lat (usec): min=1315, max=157406, avg=63858.15, stdev=13819.84
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   46], 10.00th=[   50], 20.00th=[   54],
     | 30.00th=[   58], 40.00th=[   61], 50.00th=[   64], 60.00th=[   66],
     | 70.00th=[   68], 80.00th=[   71], 90.00th=[   77], 95.00th=[   82],
     | 99.00th=[  104], 99.50th=[  146], 99.90th=[  153], 99.95th=[  155],
     | 99.99th=[  155]
   bw (  KiB/s): min=  240, max=  480, per=25.43%, avg=345.38, stdev=52.20, samples=29
   iops        : min=   60, max=  120, avg=86.34, stdev=13.05, samples=29
  write: IOPS=87, BW=351KiB/s (360kB/s)(5180KiB/14750msec)
    slat (usec): min=12, max=218, avg=45.73, stdev=21.67
    clat (msec): min=46, max=217, avg=119.53, stdev=26.44
     lat (msec): min=47, max=217, avg=119.58, stdev=26.44
    clat percentiles (msec):
     |  1.00th=[   70],  5.00th=[   81], 10.00th=[   89], 20.00th=[   97],
     | 30.00th=[  105], 40.00th=[  110], 50.00th=[  117], 60.00th=[  123],
     | 70.00th=[  130], 80.00th=[  142], 90.00th=[  155], 95.00th=[  171],
     | 99.00th=[  197], 99.50th=[  205], 99.90th=[  218], 99.95th=[  218],
     | 99.99th=[  218]
   bw (  KiB/s): min=  296, max=  416, per=24.79%, avg=350.34, stdev=30.94, samples=29
   iops        : min=   74, max=  104, avg=87.59, stdev= 7.73, samples=29
  lat (usec)   : 1000=0.12%
  lat (msec)   : 2=0.04%, 10=0.08%, 20=0.31%, 50=4.53%, 100=55.62%
  lat (msec)   : 250=39.30%
  cpu          : usr=0.18%, sys=1.44%, ctx=3108, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1358KiB/s (1391kB/s), 337KiB/s-343KiB/s (345kB/s-351kB/s), io=19.6MiB (20.5MB), run=14750-14776msec
  WRITE: bw=1414KiB/s (1448kB/s), 351KiB/s-356KiB/s (360kB/s-365kB/s), io=20.4MiB (21.4MB), run=14750-14776msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=22345: Sat May 11 16:03:51 2024
   read: IOPS=10, BW=421MiB/s (442MB/s)(40.0MiB/95msec)
    slat (nsec): min=7427.1k, max=7427.1k, avg=7427105.00, stdev= 0.00
    clat (nsec): min=82509k, max=82509k, avg=82508794.00, stdev= 0.00
     lat (nsec): min=89945k, max=89945k, avg=89944925.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[82314],  5.00th=[82314], 10.00th=[82314], 20.00th=[82314],
     | 30.00th=[82314], 40.00th=[82314], 50.00th=[82314], 60.00th=[82314],
     | 70.00th=[82314], 80.00th=[82314], 90.00th=[82314], 95.00th=[82314],
     | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314],
     | 99.99th=[82314]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.45%, ctx=15, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=421MiB/s (442MB/s), 421MiB/s-421MiB/s (442MB/s-442MB/s), io=40.0MiB (41.9MB), run=95-95msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck