== sanity test 398b: DIO and buffer IO race ============== 13:42:15 (1736361735)
/usr/bin/fio
12+0 records in
12+0 records out
12582912 bytes (13 MB) copied, 0.10218 s, 123 MB/s
mix direct rw 12M to OST0 by fio with 4 jobs...
mix buffer rw 12M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=23702: Wed Jan  8 13:42:18 2025
   read: IOPS=205, BW=823KiB/s (842kB/s)(1480KiB/1799msec)
    clat (usec): min=3, max=85772, avg=1392.02, stdev=4972.65
     lat (usec): min=3, max=85774, avg=1392.72, stdev=4972.65
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    7], 20.00th=[    9],
     | 30.00th=[   11], 40.00th=[   16], 50.00th=[   28], 60.00th=[  840],
     | 70.00th=[ 1565], 80.00th=[ 2147], 90.00th=[ 2966], 95.00th=[ 3916],
     | 99.00th=[11600], 99.50th=[26870], 99.90th=[85459], 99.95th=[85459],
     | 99.99th=[85459]
   bw (  KiB/s): min=  662, max=  912, per=23.70%, avg=788.67, stdev=125.03, samples=3
   iops        : min=  165, max=  228, avg=197.00, stdev=31.51, samples=3
  write: IOPS=221, BW=885KiB/s (906kB/s)(1592KiB/1799msec)
    clat (usec): min=92, max=22346, avg=3200.65, stdev=4537.15
     lat (usec): min=93, max=22346, avg=3201.31, stdev=4537.13
    clat percentiles (usec):
     |  1.00th=[   96],  5.00th=[  111], 10.00th=[  124], 20.00th=[  147],
     | 30.00th=[  190], 40.00th=[  832], 50.00th=[ 1270], 60.00th=[ 1909],
     | 70.00th=[ 2671], 80.00th=[ 5800], 90.00th=[ 9765], 95.00th=[14091],
     | 99.00th=[19530], 99.50th=[21627], 99.90th=[22414], 99.95th=[22414],
     | 99.99th=[22414]
   bw (  KiB/s): min=  670, max=  960, per=24.51%, avg=858.00, stdev=163.01, samples=3
   iops        : min=  167, max=  240, avg=214.33, stdev=41.04, samples=3
  lat (usec)   : 4=0.52%, 10=11.85%, 20=9.90%, 50=4.56%, 100=1.82%
  lat (usec)   : 250=17.06%, 500=1.17%, 750=0.91%, 1000=5.08%
  lat (msec)   : 2=16.02%, 4=15.89%, 10=9.64%, 20=4.95%, 50=0.52%
  lat (msec)   : 100=0.13%
  cpu          : usr=0.39%, sys=8.90%, ctx=906, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23703: Wed Jan  8 13:42:18 2025
   read: IOPS=204, BW=819KiB/s (839kB/s)(1436KiB/1753msec)
    clat (usec): min=3, max=28200, avg=1172.80, stdev=2400.88
     lat (usec): min=3, max=28201, avg=1173.68, stdev=2400.86
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    5], 10.00th=[    8], 20.00th=[   10],
     | 30.00th=[   14], 40.00th=[   20], 50.00th=[   39], 60.00th=[  963],
     | 70.00th=[ 1565], 80.00th=[ 2089], 90.00th=[ 2671], 95.00th=[ 3851],
     | 99.00th=[10290], 99.50th=[23462], 99.90th=[28181], 99.95th=[28181],
     | 99.99th=[28181]
   bw (  KiB/s): min=  592, max= 1064, per=24.04%, avg=800.00, stdev=240.93, samples=3
   iops        : min=  148, max=  266, avg=200.00, stdev=60.23, samples=3
  write: IOPS=233, BW=933KiB/s (956kB/s)(1636KiB/1753msec)
    clat (usec): min=92, max=35399, avg=3227.36, stdev=4582.38
     lat (usec): min=92, max=35400, avg=3228.56, stdev=4582.16
    clat percentiles (usec):
     |  1.00th=[  108],  5.00th=[  114], 10.00th=[  127], 20.00th=[  172],
     | 30.00th=[  281], 40.00th=[  840], 50.00th=[ 1336], 60.00th=[ 1958],
     | 70.00th=[ 3130], 80.00th=[ 6456], 90.00th=[ 9110], 95.00th=[11994],
     | 99.00th=[19792], 99.50th=[22676], 99.90th=[35390], 99.95th=[35390],
     | 99.99th=[35390]
   bw (  KiB/s): min=  784, max= 1072, per=25.75%, avg=901.33, stdev=151.23, samples=3
   iops        : min=  196, max=  268, avg=225.33, stdev=37.81, samples=3
  lat (usec)   : 4=0.39%, 10=9.24%, 20=9.24%, 50=4.56%, 100=1.30%
  lat (usec)   : 250=15.62%, 500=3.12%, 750=2.34%, 1000=5.47%
  lat (msec)   : 2=17.84%, 4=14.97%, 10=10.81%, 20=4.30%, 50=0.78%
  cpu          : usr=0.11%, sys=9.53%, ctx=961, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23704: Wed Jan  8 13:42:18 2025
   read: IOPS=220, BW=882KiB/s (903kB/s)(1564KiB/1773msec)
    clat (usec): min=3, max=24014, avg=1161.86, stdev=2296.31
     lat (usec): min=3, max=24014, avg=1162.58, stdev=2296.26
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    6], 20.00th=[    9],
     | 30.00th=[   11], 40.00th=[   13], 50.00th=[   25], 60.00th=[  947],
     | 70.00th=[ 1582], 80.00th=[ 2073], 90.00th=[ 2638], 95.00th=[ 3687],
     | 99.00th=[12911], 99.50th=[19792], 99.90th=[23987], 99.95th=[23987],
     | 99.99th=[23987]
   bw (  KiB/s): min=  616, max=  952, per=24.12%, avg=802.67, stdev=171.08, samples=3
   iops        : min=  154, max=  238, avg=200.67, stdev=42.77, samples=3
  write: IOPS=212, BW=851KiB/s (871kB/s)(1508KiB/1773msec)
    clat (usec): min=87, max=31576, avg=3470.09, stdev=4381.21
     lat (usec): min=88, max=31577, avg=3471.03, stdev=4381.03
    clat percentiles (usec):
     |  1.00th=[  101],  5.00th=[  113], 10.00th=[  128], 20.00th=[  176],
     | 30.00th=[  570], 40.00th=[ 1029], 50.00th=[ 1598], 60.00th=[ 2212],
     | 70.00th=[ 4293], 80.00th=[ 6783], 90.00th=[ 9634], 95.00th=[11469],
     | 99.00th=[18482], 99.50th=[22676], 99.90th=[31589], 99.95th=[31589],
     | 99.99th=[31589]
   bw (  KiB/s): min=  728, max=  968, per=23.08%, avg=808.00, stdev=138.56, samples=3
   iops        : min=  182, max=  242, avg=202.00, stdev=34.64, samples=3
  lat (usec)   : 4=0.91%, 10=13.67%, 20=8.98%, 50=4.17%, 100=0.91%
  lat (usec)   : 250=12.76%, 500=1.69%, 750=1.82%, 1000=4.82%
  lat (msec)   : 2=18.36%, 4=14.58%, 10=12.63%, 20=4.17%, 50=0.52%
  cpu          : usr=0.17%, sys=8.97%, ctx=958, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23705: Wed Jan  8 13:42:18 2025
   read: IOPS=209, BW=839KiB/s (859kB/s)(1508KiB/1797msec)
    clat (usec): min=3, max=14508, avg=966.86, stdev=1753.39
     lat (usec): min=3, max=14508, avg=967.41, stdev=1753.38
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    7], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   12], 50.00th=[   24], 60.00th=[  857],
     | 70.00th=[ 1483], 80.00th=[ 2024], 90.00th=[ 2540], 95.00th=[ 2966],
     | 99.00th=[12125], 99.50th=[14353], 99.90th=[14484], 99.95th=[14484],
     | 99.99th=[14484]
   bw (  KiB/s): min=  480, max= 1040, per=22.52%, avg=749.33, stdev=280.61, samples=3
   iops        : min=  120, max=  260, avg=187.33, stdev=70.15, samples=3
  write: IOPS=217, BW=870KiB/s (891kB/s)(1564KiB/1797msec)
    clat (usec): min=89, max=66516, avg=3641.83, stdev=5998.90
     lat (usec): min=89, max=66518, avg=3642.48, stdev=5998.99
    clat percentiles (usec):
     |  1.00th=[   94],  5.00th=[  114], 10.00th=[  122], 20.00th=[  159],
     | 30.00th=[  285], 40.00th=[  857], 50.00th=[ 1418], 60.00th=[ 2180],
     | 70.00th=[ 4080], 80.00th=[ 6194], 90.00th=[ 9896], 95.00th=[13173],
     | 99.00th=[24773], 99.50th=[53740], 99.90th=[66323], 99.95th=[66323],
     | 99.99th=[66323]
   bw (  KiB/s): min=  536, max=  968, per=23.16%, avg=810.67, stdev=238.71, samples=3
   iops        : min=  134, max=  242, avg=202.67, stdev=59.68, samples=3
  lat (usec)   : 4=0.91%, 10=14.58%, 20=8.33%, 50=4.04%, 100=1.69%
  lat (usec)   : 250=13.54%, 500=2.73%, 750=1.82%, 1000=5.73%
  lat (msec)   : 2=14.58%, 4=15.62%, 10=10.81%, 20=5.08%, 50=0.26%
  lat (msec)   : 100=0.26%
  cpu          : usr=0.17%, sys=8.46%, ctx=942, majf=0, minf=35
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3329KiB/s (3408kB/s), 819KiB/s-882KiB/s (839kB/s-903kB/s), io=5988KiB (6132kB), run=1753-1799msec
  WRITE: bw=3502KiB/s (3586kB/s), 851KiB/s-933KiB/s (871kB/s-956kB/s), io=6300KiB (6451kB), run=1753-1799msec

rand-rw: (groupid=0, jobs=1): err= 0: pid=23698: Wed Jan  8 13:42:23 2025
   read: IOPS=54, BW=220KiB/s (225kB/s)(1480KiB/6737msec)
    clat (usec): min=612, max=66176, avg=1971.91, stdev=4381.84
     lat (usec): min=612, max=66178, avg=1972.58, stdev=4381.87
    clat percentiles (usec):
     |  1.00th=[  676],  5.00th=[  725], 10.00th=[  775], 20.00th=[  840],
     | 30.00th=[  881], 40.00th=[  947], 50.00th=[ 1012], 60.00th=[ 1156],
     | 70.00th=[ 1336], 80.00th=[ 1598], 90.00th=[ 2376], 95.00th=[ 7963],
     | 99.00th=[14484], 99.50th=[33424], 99.90th=[66323], 99.95th=[66323],
     | 99.99th=[66323]
   bw (  KiB/s): min=  112, max=  344, per=24.76%, avg=214.69, stdev=60.22, samples=13
   iops        : min=   28, max=   86, avg=53.62, stdev=15.00, samples=13
  write: IOPS=59, BW=236KiB/s (242kB/s)(1592KiB/6737msec)
    clat (usec): min=7638, max=42718, avg=15053.75, stdev=5169.27
     lat (usec): min=7639, max=42721, avg=15054.52, stdev=5169.31
    clat percentiles (usec):
     |  1.00th=[ 8029],  5.00th=[ 8717], 10.00th=[ 9372], 20.00th=[10552],
     | 30.00th=[11600], 40.00th=[13435], 50.00th=[14746], 60.00th=[15795],
     | 70.00th=[16909], 80.00th=[18220], 90.00th=[20055], 95.00th=[24511],
     | 99.00th=[32637], 99.50th=[39060], 99.90th=[42730], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min=  120, max=  280, per=25.77%, avg=235.00, stdev=46.40, samples=13
   iops        : min=   30, max=   70, avg=58.69, stdev=11.54, samples=13
  lat (usec)   : 750=3.52%, 1000=19.53%
  lat (msec)   : 2=18.75%, 4=2.73%, 10=10.16%, 20=39.84%, 50=5.34%
  lat (msec)   : 100=0.13%
  cpu          : usr=0.18%, sys=1.56%, ctx=827, majf=0, minf=33
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23699: Wed Jan  8 13:42:23 2025
   read: IOPS=52, BW=208KiB/s (213kB/s)(1436KiB/6902msec)
    clat (usec): min=659, max=21220, avg=1626.63, stdev=2163.21
     lat (usec): min=660, max=21221, avg=1627.32, stdev=2163.20
    clat percentiles (usec):
     |  1.00th=[  693],  5.00th=[  725], 10.00th=[  775], 20.00th=[  824],
     | 30.00th=[  889], 40.00th=[  979], 50.00th=[ 1074], 60.00th=[ 1205],
     | 70.00th=[ 1352], 80.00th=[ 1647], 90.00th=[ 2474], 95.00th=[ 3884],
     | 99.00th=[12125], 99.50th=[19530], 99.90th=[21103], 99.95th=[21103],
     | 99.99th=[21103]
   bw (  KiB/s): min=  104, max=  336, per=23.77%, avg=206.08, stdev=64.25, samples=13
   iops        : min=   26, max=   84, avg=51.46, stdev=16.06, samples=13
  write: IOPS=59, BW=237KiB/s (243kB/s)(1636KiB/6902msec)
    clat (usec): min=7652, max=62926, avg=15399.31, stdev=5576.19
     lat (usec): min=7653, max=62927, avg=15400.13, stdev=5576.20
    clat percentiles (usec):
     |  1.00th=[ 8291],  5.00th=[ 8848], 10.00th=[ 9503], 20.00th=[10945],
     | 30.00th=[12780], 40.00th=[14091], 50.00th=[15139], 60.00th=[16057],
     | 70.00th=[16909], 80.00th=[17957], 90.00th=[19792], 95.00th=[24249],
     | 99.00th=[36439], 99.50th=[45351], 99.90th=[63177], 99.95th=[63177],
     | 99.99th=[63177]
   bw (  KiB/s): min=  160, max=  271, per=25.63%, avg=233.77, stdev=32.26, samples=13
   iops        : min=   40, max=   67, avg=58.38, stdev= 8.00, samples=13
  lat (usec)   : 750=3.65%, 1000=16.54%
  lat (msec)   : 2=20.44%, 4=4.04%, 10=8.46%, 20=41.67%, 50=5.08%
  lat (msec)   : 100=0.13%
  cpu          : usr=0.09%, sys=1.70%, ctx=813, majf=0, minf=33
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23700: Wed Jan  8 13:42:23 2025
   read: IOPS=60, BW=240KiB/s (246kB/s)(1564KiB/6514msec)
    clat (usec): min=624, max=19363, avg=1848.84, stdev=2609.96
     lat (usec): min=625, max=19364, avg=1849.50, stdev=2609.97
    clat percentiles (usec):
     |  1.00th=[  676],  5.00th=[  734], 10.00th=[  783], 20.00th=[  832],
     | 30.00th=[  881], 40.00th=[  922], 50.00th=[ 1029], 60.00th=[ 1172],
     | 70.00th=[ 1336], 80.00th=[ 1680], 90.00th=[ 2737], 95.00th=[ 8160],
     | 99.00th=[14222], 99.50th=[18744], 99.90th=[19268], 99.95th=[19268],
     | 99.99th=[19268]
   bw (  KiB/s): min=  120, max=  384, per=27.74%, avg=240.54, stdev=71.81, samples=13
   iops        : min=   30, max=   96, avg=60.08, stdev=17.96, samples=13
  write: IOPS=57, BW=232KiB/s (237kB/s)(1508KiB/6514msec)
    clat (usec): min=7624, max=60759, avg=15312.36, stdev=5162.90
     lat (usec): min=7624, max=60760, avg=15313.18, stdev=5162.91
    clat percentiles (usec):
     |  1.00th=[ 8291],  5.00th=[ 8848], 10.00th=[10028], 20.00th=[10945],
     | 30.00th=[12518], 40.00th=[13960], 50.00th=[15139], 60.00th=[16057],
     | 70.00th=[16909], 80.00th=[17957], 90.00th=[20055], 95.00th=[23462],
     | 99.00th=[33817], 99.50th=[37487], 99.90th=[60556], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  160, max=  287, per=25.36%, avg=231.31, stdev=33.24, samples=13
   iops        : min=   40, max=   71, avg=57.77, stdev= 8.21, samples=13
  lat (usec)   : 750=2.99%, 1000=20.96%
  lat (msec)   : 2=19.92%, 4=2.73%, 10=7.68%, 20=40.49%, 50=5.08%
  lat (msec)   : 100=0.13%
  cpu          : usr=0.14%, sys=1.69%, ctx=835, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23701: Wed Jan  8 13:42:23 2025
   read: IOPS=55, BW=221KiB/s (227kB/s)(1508KiB/6813msec)
    clat (usec): min=640, max=34954, avg=1814.70, stdev=3120.32
     lat (usec): min=641, max=34954, avg=1815.51, stdev=3120.29
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  742], 10.00th=[  775], 20.00th=[  840],
     | 30.00th=[  906], 40.00th=[  971], 50.00th=[ 1045], 60.00th=[ 1156],
     | 70.00th=[ 1287], 80.00th=[ 1450], 90.00th=[ 2089], 95.00th=[ 7701],
     | 99.00th=[15795], 99.50th=[21627], 99.90th=[34866], 99.95th=[34866],
     | 99.99th=[34866]
   bw (  KiB/s): min=  112, max=  328, per=24.62%, avg=213.46, stdev=61.54, samples=13
   iops        : min=   28, max=   82, avg=53.31, stdev=15.43, samples=13
  write: IOPS=57, BW=230KiB/s (235kB/s)(1564KiB/6813msec)
    clat (usec): min=7258, max=66355, avg=15647.79, stdev=6267.20
     lat (usec): min=7258, max=66356, avg=15648.63, stdev=6267.21
    clat percentiles (usec):
     |  1.00th=[ 8029],  5.00th=[ 8717], 10.00th=[ 9503], 20.00th=[10814],
     | 30.00th=[12387], 40.00th=[14222], 50.00th=[15270], 60.00th=[16319],
     | 70.00th=[17171], 80.00th=[18482], 90.00th=[20317], 95.00th=[24511],
     | 99.00th=[41681], 99.50th=[61604], 99.90th=[66323], 99.95th=[66323],
     | 99.99th=[66323]
   bw (  KiB/s): min=  128, max=  287, per=24.62%, avg=224.54, stdev=43.14, samples=13
   iops        : min=   32, max=   71, avg=56.08, stdev=10.70, samples=13
  lat (usec)   : 750=3.12%, 1000=18.36%
  lat (msec)   : 2=21.87%, 4=2.73%, 10=8.20%, 20=39.97%, 50=5.47%
  lat (msec)   : 100=0.26%
  cpu          : usr=0.16%, sys=1.56%, ctx=820, majf=0, minf=31
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=868KiB/s (888kB/s), 208KiB/s-240KiB/s (213kB/s-246kB/s), io=5988KiB (6132kB), run=6514-6902msec
  WRITE: bw=913KiB/s (935kB/s), 230KiB/s-237KiB/s (235kB/s-243kB/s), io=6300KiB (6451kB), run=6514-6902msec