== sanity test 398b: DIO and buffer IO race ============== 10:51:34 (1721227894)
/usr/bin/fio
12+0 records in
12+0 records out
12582912 bytes (13 MB) copied, 0.104889 s, 120 MB/s
mix direct rw 12M to OST0 by fio with 4 jobs...
mix buffer rw 12M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=26733: Wed Jul 17 10:51:37 2024
   read: IOPS=179, BW=716KiB/s (734kB/s)(1480KiB/2066msec)
    clat (usec): min=3, max=65889, avg=1541.35, stdev=4465.01
     lat (usec): min=3, max=65890, avg=1542.06, stdev=4464.99
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   11], 40.00th=[   15], 50.00th=[   24], 60.00th=[  971],
     | 70.00th=[ 1696], 80.00th=[ 2212], 90.00th=[ 2900], 95.00th=[ 5735],
     | 99.00th=[18482], 99.50th=[27919], 99.90th=[65799], 99.95th=[65799],
     | 99.99th=[65799]
   bw (  KiB/s): min=  568, max=  856, per=23.40%, avg=678.00, stdev=125.03, samples=4
   iops        : min=  142, max=  214, avg=169.50, stdev=31.26, samples=4
  write: IOPS=192, BW=771KiB/s (789kB/s)(1592KiB/2066msec)
    clat (usec): min=85, max=37537, avg=3715.61, stdev=5247.03
     lat (usec): min=86, max=37537, avg=3716.23, stdev=5247.05
    clat percentiles (usec):
     |  1.00th=[  101],  5.00th=[  108], 10.00th=[  115], 20.00th=[  131],
     | 30.00th=[  194], 40.00th=[  799], 50.00th=[ 1270], 60.00th=[ 1860],
     | 70.00th=[ 3130], 80.00th=[ 8356], 90.00th=[11731], 95.00th=[14484],
     | 99.00th=[21365], 99.50th=[25035], 99.90th=[37487], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min=  640, max=  864, per=24.07%, avg=734.00, stdev=105.80, samples=4
   iops        : min=  160, max=  216, avg=183.50, stdev=26.45, samples=4
  lat (usec)   : 4=0.52%, 10=13.80%, 20=8.33%, 50=3.39%, 100=1.43%
  lat (usec)   : 250=16.28%, 500=1.56%, 750=1.82%, 1000=5.21%
  lat (msec)   : 2=15.89%, 4=14.58%, 10=8.85%, 20=7.29%, 50=0.91%
  lat (msec)   : 100=0.13%
  cpu          : usr=0.24%, sys=7.12%, ctx=933, majf=0, minf=36
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=26734: Wed Jul 17 10:51:37 2024
   read: IOPS=175, BW=701KiB/s (718kB/s)(1436KiB/2048msec)
    clat (usec): min=3, max=27058, avg=1272.30, stdev=2667.76
     lat (usec): min=4, max=27059, avg=1272.83, stdev=2667.78
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    7], 20.00th=[    9],
     | 30.00th=[   11], 40.00th=[   16], 50.00th=[   52], 60.00th=[ 1090],
     | 70.00th=[ 1582], 80.00th=[ 2114], 90.00th=[ 2737], 95.00th=[ 3556],
     | 99.00th=[13304], 99.50th=[25035], 99.90th=[27132], 99.95th=[27132],
     | 99.99th=[27132]
   bw (  KiB/s): min=  520, max=  808, per=24.29%, avg=704.00, stdev=132.26, samples=4
   iops        : min=  130, max=  202, avg=176.00, stdev=33.07, samples=4
  write: IOPS=199, BW=799KiB/s (818kB/s)(1636KiB/2048msec)
    clat (usec): min=90, max=53820, avg=3848.59, stdev=5821.40
     lat (usec): min=90, max=53821, avg=3849.58, stdev=5821.24
    clat percentiles (usec):
     |  1.00th=[  100],  5.00th=[  106], 10.00th=[  116], 20.00th=[  139],
     | 30.00th=[  239], 40.00th=[  832], 50.00th=[ 1319], 60.00th=[ 1844],
     | 70.00th=[ 3195], 80.00th=[ 8225], 90.00th=[12125], 95.00th=[14353],
     | 99.00th=[21890], 99.50th=[25297], 99.90th=[53740], 99.95th=[53740],
     | 99.99th=[53740]
   bw (  KiB/s): min=  744, max=  936, per=26.11%, avg=796.00, stdev=93.41, samples=4
   iops        : min=  186, max=  234, avg=199.00, stdev=23.35, samples=4
  lat (usec)   : 4=0.65%, 10=12.63%, 20=7.81%, 50=2.21%, 100=1.56%
  lat (usec)   : 250=16.41%, 500=2.21%, 750=1.43%, 1000=6.12%
  lat (msec)   : 2=18.49%, 4=13.80%, 10=7.81%, 20=7.68%, 50=1.04%
  lat (msec)   : 100=0.13%
  cpu          : usr=0.20%, sys=6.94%, ctx=933, majf=0, minf=35
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=26735: Wed Jul 17 10:51:37 2024
   read: IOPS=189, BW=758KiB/s (777kB/s)(1564KiB/2062msec)
    clat (usec): min=3, max=24235, avg=1357.96, stdev=2875.88
     lat (usec): min=3, max=24236, avg=1358.45, stdev=2875.90
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    7], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   14], 50.00th=[   26], 60.00th=[ 1012],
     | 70.00th=[ 1582], 80.00th=[ 2212], 90.00th=[ 2900], 95.00th=[ 3720],
     | 99.00th=[16712], 99.50th=[20055], 99.90th=[24249], 99.95th=[24249],
     | 99.99th=[24249]
   bw (  KiB/s): min=  640, max=  824, per=25.12%, avg=728.00, stdev=97.54, samples=4
   iops        : min=  160, max=  206, avg=182.00, stdev=24.39, samples=4
  write: IOPS=182, BW=731KiB/s (749kB/s)(1508KiB/2062msec)
    clat (usec): min=90, max=58978, avg=4039.73, stdev=6103.23
     lat (usec): min=91, max=58979, avg=4040.31, stdev=6103.25
    clat percentiles (usec):
     |  1.00th=[   96],  5.00th=[  105], 10.00th=[  115], 20.00th=[  135],
     | 30.00th=[  194], 40.00th=[  898], 50.00th=[ 1319], 60.00th=[ 2089],
     | 70.00th=[ 3294], 80.00th=[ 7832], 90.00th=[12518], 95.00th=[16057],
     | 99.00th=[22938], 99.50th=[27919], 99.90th=[58983], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min=  520, max=  760, per=22.63%, avg=690.00, stdev=113.77, samples=4
   iops        : min=  130, max=  190, avg=172.50, stdev=28.44, samples=4
  lat (usec)   : 4=0.91%, 10=14.97%, 20=7.94%, 50=2.86%, 100=2.34%
  lat (usec)   : 250=15.36%, 500=1.17%, 750=1.04%, 1000=4.69%
  lat (msec)   : 2=16.02%, 4=16.28%, 10=7.68%, 20=7.16%, 50=1.43%
  lat (msec)   : 100=0.13%
  cpu          : usr=0.19%, sys=7.71%, ctx=966, majf=0, minf=33
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=26737: Wed Jul 17 10:51:37 2024
   read: IOPS=186, BW=747KiB/s (765kB/s)(1508KiB/2018msec)
    clat (usec): min=3, max=23043, avg=1421.20, stdev=2915.62
     lat (usec): min=3, max=23043, avg=1422.34, stdev=2915.34
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    6], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   14], 50.00th=[   26], 60.00th=[ 1237],
     | 70.00th=[ 1762], 80.00th=[ 2278], 90.00th=[ 2933], 95.00th=[ 4424],
     | 99.00th=[19268], 99.50th=[22414], 99.90th=[22938], 99.95th=[22938],
     | 99.99th=[22938]
   bw (  KiB/s): min=  632, max=  888, per=25.47%, avg=738.00, stdev=109.96, samples=4
   iops        : min=  158, max=  222, avg=184.50, stdev=27.49, samples=4
  write: IOPS=193, BW=775KiB/s (794kB/s)(1564KiB/2018msec)
    clat (usec): min=89, max=29574, avg=3752.40, stdev=5601.97
     lat (usec): min=90, max=29575, avg=3752.99, stdev=5601.99
    clat percentiles (usec):
     |  1.00th=[   97],  5.00th=[  109], 10.00th=[  116], 20.00th=[  133],
     | 30.00th=[  194], 40.00th=[  807], 50.00th=[ 1205], 60.00th=[ 1827],
     | 70.00th=[ 3032], 80.00th=[ 7111], 90.00th=[11863], 95.00th=[16450],
     | 99.00th=[23725], 99.50th=[26608], 99.90th=[29492], 99.95th=[29492],
     | 99.99th=[29492]
   bw (  KiB/s): min=  648, max=  936, per=25.12%, avg=766.00, stdev=122.44, samples=4
   iops        : min=  162, max=  234, avg=191.50, stdev=30.61, samples=4
  lat (usec)   : 4=1.30%, 10=14.45%, 20=7.29%, 50=1.69%, 100=2.86%
  lat (usec)   : 250=16.41%, 500=1.69%, 750=1.56%, 1000=2.99%
  lat (msec)   : 2=18.36%, 4=15.36%, 10=7.42%, 20=6.77%, 50=1.82%
  cpu          : usr=0.20%, sys=7.88%, ctx=955, majf=0, minf=35
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2898KiB/s (2968kB/s), 701KiB/s-758KiB/s (718kB/s-777kB/s), io=5988KiB (6132kB), run=2018-2066msec
  WRITE: bw=3049KiB/s (3123kB/s), 731KiB/s-799KiB/s (749kB/s-818kB/s), io=6300KiB (6451kB), run=2018-2066msec

rand-rw: (groupid=0, jobs=1): err= 0: pid=26736: Wed Jul 17 10:51:44 2024
   read: IOPS=43, BW=176KiB/s (180kB/s)(1480KiB/8414msec)
    clat (usec): min=632, max=39089, avg=2109.17, stdev=3754.77
     lat (usec): min=633, max=39090, avg=2109.93, stdev=3754.78
    clat percentiles (usec):
     |  1.00th=[  701],  5.00th=[  816], 10.00th=[  848], 20.00th=[  914],
     | 30.00th=[  963], 40.00th=[ 1012], 50.00th=[ 1090], 60.00th=[ 1237],
     | 70.00th=[ 1385], 80.00th=[ 1745], 90.00th=[ 2311], 95.00th=[ 9110],
     | 99.00th=[21103], 99.50th=[24511], 99.90th=[39060], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min=  104, max=  304, per=24.92%, avg=174.44, stdev=53.19, samples=16
   iops        : min=   26, max=   76, avg=43.56, stdev=13.32, samples=16
  write: IOPS=47, BW=189KiB/s (194kB/s)(1592KiB/8414msec)
    clat (usec): min=10556, max=46589, avg=19139.56, stdev=5512.00
     lat (usec): min=10557, max=46590, avg=19140.33, stdev=5512.02
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[11731], 10.00th=[12387], 20.00th=[13435],
     | 30.00th=[15664], 40.00th=[17957], 50.00th=[19530], 60.00th=[20579],
     | 70.00th=[21627], 80.00th=[22676], 90.00th=[25035], 95.00th=[28967],
     | 99.00th=[38536], 99.50th=[40633], 99.90th=[46400], 99.95th=[46400],
     | 99.99th=[46400]
   bw (  KiB/s): min=  104, max=  224, per=25.43%, avg=187.44, stdev=34.64, samples=16
   iops        : min=   26, max=   56, avg=46.81, stdev= 8.63, samples=16
  lat (usec)   : 750=0.91%, 1000=17.58%
  lat (msec)   : 2=22.40%, 4=3.91%, 10=1.17%, 20=29.43%, 50=24.61%
  cpu          : usr=0.12%, sys=1.27%, ctx=812, majf=0, minf=33
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=26738: Wed Jul 17 10:51:44 2024
   read: IOPS=42, BW=168KiB/s (172kB/s)(1436KiB/8545msec)
    clat (usec): min=644, max=24335, avg=1913.94, stdev=3043.54
     lat (usec): min=644, max=24336, avg=1914.60, stdev=3043.54
    clat percentiles (usec):
     |  1.00th=[  734],  5.00th=[  791], 10.00th=[  848], 20.00th=[  914],
     | 30.00th=[  955], 40.00th=[ 1012], 50.00th=[ 1090], 60.00th=[ 1237],
     | 70.00th=[ 1385], 80.00th=[ 1614], 90.00th=[ 2442], 95.00th=[ 6783],
     | 99.00th=[19268], 99.50th=[22676], 99.90th=[24249], 99.95th=[24249],
     | 99.99th=[24249]
   bw (  KiB/s): min=   72, max=  280, per=23.76%, avg=166.31, stdev=54.90, samples=16
   iops        : min=   18, max=   70, avg=41.50, stdev=13.74, samples=16
  write: IOPS=47, BW=191KiB/s (196kB/s)(1636KiB/8545msec)
    clat (usec): min=10467, max=86288, avg=19193.19, stdev=6735.18
     lat (usec): min=10468, max=86289, avg=19194.01, stdev=6735.20
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[11469], 10.00th=[12256], 20.00th=[13304],
     | 30.00th=[15401], 40.00th=[17957], 50.00th=[19268], 60.00th=[20317],
     | 70.00th=[21365], 80.00th=[22152], 90.00th=[24511], 95.00th=[28705],
     | 99.00th=[41157], 99.50th=[47449], 99.90th=[86508], 99.95th=[86508],
     | 99.99th=[86508]
   bw (  KiB/s): min=  112, max=  224, per=25.52%, avg=188.06, stdev=30.34, samples=16
   iops        : min=   28, max=   56, avg=46.94, stdev= 7.55, samples=16
  lat (usec)   : 750=1.04%, 1000=16.41%
  lat (msec)   : 2=22.92%, 4=3.65%, 10=0.91%, 20=31.64%, 50=23.18%
  lat (msec)   : 100=0.26%
  cpu          : usr=0.13%, sys=1.29%, ctx=821, majf=0, minf=31
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=26739: Wed Jul 17 10:51:44 2024
   read: IOPS=49, BW=197KiB/s (202kB/s)(1564KiB/7943msec)
    clat (usec): min=648, max=38322, avg=1866.37, stdev=3097.69
     lat (usec): min=648, max=38323, avg=1866.99, stdev=3097.70
    clat percentiles (usec):
     |  1.00th=[  685],  5.00th=[  775], 10.00th=[  832], 20.00th=[  898],
     | 30.00th=[  947], 40.00th=[ 1012], 50.00th=[ 1090], 60.00th=[ 1205],
     | 70.00th=[ 1369], 80.00th=[ 1631], 90.00th=[ 2376], 95.00th=[ 4817],
     | 99.00th=[16909], 99.50th=[19006], 99.90th=[38536], 99.95th=[38536],
     | 99.99th=[38536]
   bw (  KiB/s): min=   96, max=  352, per=28.79%, avg=201.53, stdev=77.32, samples=15
   iops        : min=   24, max=   88, avg=50.33, stdev=19.32, samples=15
  write: IOPS=47, BW=190KiB/s (194kB/s)(1508KiB/7943msec)
    clat (usec): min=10885, max=80008, avg=19077.22, stdev=6898.65
     lat (usec): min=10886, max=80011, avg=19077.98, stdev=6898.71
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[11731], 10.00th=[12387], 20.00th=[13304],
     | 30.00th=[14877], 40.00th=[16909], 50.00th=[19268], 60.00th=[20055],
     | 70.00th=[21103], 80.00th=[22414], 90.00th=[24773], 95.00th=[28967],
     | 99.00th=[44827], 99.50th=[70779], 99.90th=[80217], 99.95th=[80217],
     | 99.99th=[80217]
   bw (  KiB/s): min=  112, max=  216, per=25.68%, avg=189.27, stdev=29.99, samples=15
   iops        : min=   28, max=   54, avg=47.27, stdev= 7.47, samples=15
  lat (usec)   : 750=1.56%, 1000=18.23%
  lat (msec)   : 2=24.35%, 4=3.91%, 10=0.78%, 20=30.73%, 50=20.05%
  lat (msec)   : 100=0.39%
  cpu          : usr=0.04%, sys=1.49%, ctx=812, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=26740: Wed Jul 17 10:51:44 2024
   read: IOPS=45, BW=182KiB/s (186kB/s)(1508KiB/8285msec)
    clat (usec): min=643, max=24071, avg=1898.96, stdev=2685.96
     lat (usec): min=644, max=24072, avg=1899.57, stdev=2685.96
    clat percentiles (usec):
     |  1.00th=[  709],  5.00th=[  783], 10.00th=[  840], 20.00th=[  914],
     | 30.00th=[  971], 40.00th=[ 1045], 50.00th=[ 1172], 60.00th=[ 1319],
     | 70.00th=[ 1516], 80.00th=[ 1745], 90.00th=[ 2507], 95.00th=[ 6587],
     | 99.00th=[16319], 99.50th=[21365], 99.90th=[23987], 99.95th=[23987],
     | 99.99th=[23987]
   bw (  KiB/s): min=   72, max=  264, per=25.63%, avg=179.44, stdev=55.44, samples=16
   iops        : min=   18, max=   66, avg=44.81, stdev=13.81, samples=16
  write: IOPS=47, BW=189KiB/s (193kB/s)(1564KiB/8285msec)
    clat (usec): min=10482, max=63885, avg=19317.38, stdev=6305.32
     lat (usec): min=10483, max=63887, avg=19318.15, stdev=6305.35
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[12125], 10.00th=[12518], 20.00th=[13435],
     | 30.00th=[15926], 40.00th=[17957], 50.00th=[19530], 60.00th=[20579],
     | 70.00th=[21365], 80.00th=[22938], 90.00th=[24773], 95.00th=[26608],
     | 99.00th=[44303], 99.50th=[63177], 99.90th=[63701], 99.95th=[63701],
     | 99.99th=[63701]
   bw (  KiB/s): min=  120, max=  240, per=25.43%, avg=187.44, stdev=32.39, samples=16
   iops        : min=   30, max=   60, avg=46.81, stdev= 8.09, samples=16
  lat (usec)   : 750=1.04%, 1000=16.02%
  lat (msec)   : 2=25.00%, 4=3.65%, 10=1.95%, 20=29.43%, 50=22.53%
  lat (msec)   : 100=0.39%
  cpu          : usr=0.07%, sys=1.34%, ctx=815, majf=0, minf=33
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=701KiB/s (718kB/s), 168KiB/s-197KiB/s (172kB/s-202kB/s), io=5988KiB (6132kB), run=7943-8545msec
  WRITE: bw=737KiB/s (755kB/s), 189KiB/s-191KiB/s (193kB/s-196kB/s), io=6300KiB (6451kB), run=7943-8545msec