== sanity test 398b: DIO and buffer IO race ============== 12:02:07 (1730826127)
/usr/bin/fio
12+0 records in
12+0 records out
12582912 bytes (13 MB) copied, 0.103052 s, 122 MB/s
mix direct rw 12M to OST0 by fio with 4 jobs...
mix buffer rw 12M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=19654: Tue Nov  5 12:02:10 2024
   read: IOPS=251, BW=1007KiB/s (1031kB/s)(1480KiB/1470msec)
    clat (usec): min=3, max=42328, avg=1207.95, stdev=3094.61
     lat (usec): min=3, max=42329, avg=1208.66, stdev=3094.66
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    6], 10.00th=[    8], 20.00th=[   10],
     | 30.00th=[   13], 40.00th=[   18], 50.00th=[   31], 60.00th=[  857],
     | 70.00th=[ 1516], 80.00th=[ 1958], 90.00th=[ 2737], 95.00th=[ 3326],
     | 99.00th=[15926], 99.50th=[22676], 99.90th=[42206], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min= 1064, max= 1136, per=28.94%, avg=1100.00, stdev=50.91, samples=2
   iops        : min=  266, max=  284, avg=275.00, stdev=12.73, samples=2
  write: IOPS=270, BW=1083KiB/s (1109kB/s)(1592KiB/1470msec)
    clat (usec): min=100, max=21056, avg=2530.80, stdev=3832.86
     lat (usec): min=100, max=21056, avg=2531.88, stdev=3832.69
    clat percentiles (usec):
     |  1.00th=[  108],  5.00th=[  125], 10.00th=[  141], 20.00th=[  167],
     | 30.00th=[  210], 40.00th=[  310], 50.00th=[  914], 60.00th=[ 1483],
     | 70.00th=[ 2147], 80.00th=[ 3720], 90.00th=[ 8291], 95.00th=[10814],
     | 99.00th=[17957], 99.50th=[19530], 99.90th=[21103], 99.95th=[21103],
     | 99.99th=[21103]
   bw (  KiB/s): min=  992, max= 1344, per=29.20%, avg=1168.00, stdev=248.90, samples=2
   iops        : min=  248, max=  336, avg=292.00, stdev=62.23, samples=2
  lat (usec)   : 4=0.26%, 10=10.03%, 20=9.51%, 50=6.12%, 100=0.39%
  lat (usec)   : 250=19.27%, 500=4.30%, 750=1.82%, 1000=5.08%
  lat (msec)   : 2=16.80%, 4=14.58%, 10=7.68%, 20=3.65%, 50=0.52%
  cpu          : usr=0.34%, sys=14.23%, ctx=854, majf=0, minf=35
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19655: Tue Nov  5 12:02:10 2024
   read: IOPS=227, BW=912KiB/s (934kB/s)(1436KiB/1575msec)
    clat (usec): min=4, max=13078, avg=1056.59, stdev=1849.21
     lat (usec): min=4, max=13079, avg=1057.24, stdev=1849.27
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    6], 10.00th=[    9], 20.00th=[   10],
     | 30.00th=[   12], 40.00th=[   16], 50.00th=[   27], 60.00th=[  775],
     | 70.00th=[ 1385], 80.00th=[ 1991], 90.00th=[ 2868], 95.00th=[ 3818],
     | 99.00th=[ 8979], 99.50th=[12911], 99.90th=[13042], 99.95th=[13042],
     | 99.99th=[13042]
   bw (  KiB/s): min=  760, max= 1104, per=24.06%, avg=914.67, stdev=174.60, samples=3
   iops        : min=  190, max=  276, avg=228.67, stdev=43.65, samples=3
  write: IOPS=259, BW=1039KiB/s (1064kB/s)(1636KiB/1575msec)
    clat (usec): min=109, max=30767, avg=2898.03, stdev=4548.49
     lat (usec): min=109, max=30768, avg=2898.77, stdev=4548.48
    clat percentiles (usec):
     |  1.00th=[  116],  5.00th=[  133], 10.00th=[  145], 20.00th=[  167],
     | 30.00th=[  229], 40.00th=[  445], 50.00th=[ 1020], 60.00th=[ 1549],
     | 70.00th=[ 2245], 80.00th=[ 4817], 90.00th=[ 8586], 95.00th=[11600],
     | 99.00th=[21103], 99.50th=[25822], 99.90th=[30802], 99.95th=[30802],
     | 99.99th=[30802]
   bw (  KiB/s): min=  856, max= 1200, per=25.53%, avg=1021.33, stdev=172.39, samples=3
   iops        : min=  214, max=  300, avg=255.33, stdev=43.10, samples=3
  lat (usec)   : 10=9.90%, 20=11.46%, 50=5.08%, 100=0.39%, 250=18.49%
  lat (usec)   : 500=4.17%, 750=1.04%, 1000=5.47%
  lat (msec)   : 2=17.32%, 4=12.37%, 10=10.68%, 20=2.73%, 50=0.91%
  cpu          : usr=0.57%, sys=10.48%, ctx=852, majf=0, minf=32
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19656: Tue Nov  5 12:02:10 2024
   read: IOPS=248, BW=996KiB/s (1019kB/s)(1564KiB/1571msec)
    clat (usec): min=3, max=28697, avg=1116.04, stdev=2624.67
     lat (usec): min=4, max=28698, avg=1116.99, stdev=2624.64
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    6], 10.00th=[    8], 20.00th=[   10],
     | 30.00th=[   11], 40.00th=[   15], 50.00th=[   25], 60.00th=[  816],
     | 70.00th=[ 1287], 80.00th=[ 1991], 90.00th=[ 2540], 95.00th=[ 4113],
     | 99.00th=[11338], 99.50th=[24773], 99.90th=[28705], 99.95th=[28705],
     | 99.99th=[28705]
   bw (  KiB/s): min=  872, max= 1128, per=26.52%, avg=1008.00, stdev=128.75, samples=3
   iops        : min=  218, max=  282, avg=252.00, stdev=32.19, samples=3
  write: IOPS=239, BW=960KiB/s (983kB/s)(1508KiB/1571msec)
    clat (usec): min=112, max=22721, avg=2964.95, stdev=4201.55
     lat (usec): min=112, max=22721, avg=2965.71, stdev=4201.58
    clat percentiles (usec):
     |  1.00th=[  115],  5.00th=[  129], 10.00th=[  143], 20.00th=[  174],
     | 30.00th=[  208], 40.00th=[  611], 50.00th=[ 1074], 60.00th=[ 1778],
     | 70.00th=[ 2671], 80.00th=[ 5669], 90.00th=[ 9241], 95.00th=[12125],
     | 99.00th=[18744], 99.50th=[19530], 99.90th=[22676], 99.95th=[22676],
     | 99.99th=[22676]
   bw (  KiB/s): min=  896, max= 1032, per=23.67%, avg=946.67, stdev=74.33, samples=3
   iops        : min=  224, max=  258, avg=236.67, stdev=18.58, samples=3
  lat (usec)   : 4=0.13%, 10=11.98%, 20=10.81%, 50=5.86%, 100=0.65%
  lat (usec)   : 250=16.93%, 500=2.73%, 750=1.69%, 1000=5.47%
  lat (msec)   : 2=15.76%, 4=13.67%, 10=9.90%, 20=3.91%, 50=0.52%
  cpu          : usr=0.38%, sys=10.76%, ctx=889, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19657: Tue Nov  5 12:02:10 2024
   read: IOPS=248, BW=993KiB/s (1017kB/s)(1508KiB/1519msec)
    clat (usec): min=3, max=20033, avg=1170.05, stdev=2110.06
     lat (usec): min=4, max=20033, avg=1170.82, stdev=2110.16
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    7], 10.00th=[    8], 20.00th=[   10],
     | 30.00th=[   12], 40.00th=[   18], 50.00th=[   29], 60.00th=[ 1029],
     | 70.00th=[ 1582], 80.00th=[ 2024], 90.00th=[ 2835], 95.00th=[ 4424],
     | 99.00th=[11469], 99.50th=[13566], 99.90th=[20055], 99.95th=[20055],
     | 99.99th=[20055]
   bw (  KiB/s): min=  904, max= 1112, per=26.03%, avg=989.33, stdev=108.91, samples=3
   iops        : min=  226, max=  278, avg=247.33, stdev=27.23, samples=3
  write: IOPS=257, BW=1030KiB/s (1054kB/s)(1564KiB/1519msec)
    clat (usec): min=106, max=49251, avg=2709.79, stdev=4494.07
     lat (usec): min=107, max=49252, avg=2710.71, stdev=4494.01
    clat percentiles (usec):
     |  1.00th=[  109],  5.00th=[  124], 10.00th=[  139], 20.00th=[  165],
     | 30.00th=[  225], 40.00th=[  367], 50.00th=[  996], 60.00th=[ 1565],
     | 70.00th=[ 2409], 80.00th=[ 4293], 90.00th=[ 8291], 95.00th=[10552],
     | 99.00th=[20317], 99.50th=[24249], 99.90th=[49021], 99.95th=[49021],
     | 99.99th=[49021]
   bw (  KiB/s): min=  968, max= 1096, per=25.80%, avg=1032.00, stdev=64.00, samples=3
   iops        : min=  242, max=  274, avg=258.00, stdev=16.00, samples=3
  lat (usec)   : 4=0.39%, 10=10.29%, 20=10.68%, 50=5.08%, 100=0.52%
  lat (usec)   : 250=17.45%, 500=3.91%, 750=1.30%, 1000=5.34%
  lat (msec)   : 2=17.71%, 4=14.32%, 10=9.51%, 20=2.86%, 50=0.65%
  cpu          : usr=0.40%, sys=11.53%, ctx=866, majf=0, minf=35
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3802KiB/s (3893kB/s), 912KiB/s-1007KiB/s (934kB/s-1031kB/s), io=5988KiB (6132kB), run=1470-1575msec
  WRITE: bw=4000KiB/s (4096kB/s), 960KiB/s-1083KiB/s (983kB/s-1109kB/s), io=6300KiB (6451kB), run=1470-1575msec

rand-rw: (groupid=0, jobs=1): err= 0: pid=19658: Tue Nov  5 12:02:16 2024
   read: IOPS=51, BW=206KiB/s (211kB/s)(1480KiB/7194msec)
    clat (usec): min=649, max=65926, avg=2169.41, stdev=4143.40
     lat (usec): min=650, max=65928, avg=2170.41, stdev=4143.45
    clat percentiles (usec):
     |  1.00th=[  717],  5.00th=[  898], 10.00th=[  947], 20.00th=[ 1037],
     | 30.00th=[ 1123], 40.00th=[ 1188], 50.00th=[ 1270], 60.00th=[ 1401],
     | 70.00th=[ 1631], 80.00th=[ 2212], 90.00th=[ 3097], 95.00th=[ 5407],
     | 99.00th=[13960], 99.50th=[25560], 99.90th=[65799], 99.95th=[65799],
     | 99.99th=[65799]
   bw (  KiB/s): min=  112, max=  312, per=25.04%, avg=202.79, stdev=54.18, samples=14
   iops        : min=   28, max=   78, avg=50.64, stdev=13.59, samples=14
  write: IOPS=55, BW=221KiB/s (227kB/s)(1592KiB/7194msec)
    clat (usec): min=9013, max=43328, avg=16029.43, stdev=4597.66
     lat (usec): min=9014, max=43328, avg=16030.56, stdev=4597.74
    clat percentiles (usec):
     |  1.00th=[ 9634],  5.00th=[10290], 10.00th=[10814], 20.00th=[11600],
     | 30.00th=[12911], 40.00th=[14615], 50.00th=[15926], 60.00th=[17171],
     | 70.00th=[18220], 80.00th=[19268], 90.00th=[20317], 95.00th=[22152],
     | 99.00th=[33817], 99.50th=[38536], 99.90th=[43254], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min=  120, max=  256, per=25.88%, avg=220.50, stdev=35.28, samples=14
   iops        : min=   30, max=   64, avg=55.07, stdev= 8.79, samples=14
  lat (usec)   : 750=0.65%, 1000=6.77%
  lat (msec)   : 2=30.34%, 4=7.29%, 10=2.60%, 20=44.92%, 50=7.29%
  lat (msec)   : 100=0.13%
  cpu          : usr=0.03%, sys=2.45%, ctx=861, majf=0, minf=33
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19659: Tue Nov  5 12:02:16 2024
   read: IOPS=48, BW=194KiB/s (199kB/s)(1436KiB/7392msec)
    clat (usec): min=682, max=66413, avg=2313.24, stdev=4533.04
     lat (usec): min=682, max=66414, avg=2314.39, stdev=4533.05
    clat percentiles (usec):
     |  1.00th=[  766],  5.00th=[  881], 10.00th=[  963], 20.00th=[ 1057],
     | 30.00th=[ 1172], 40.00th=[ 1237], 50.00th=[ 1336], 60.00th=[ 1483],
     | 70.00th=[ 1631], 80.00th=[ 2147], 90.00th=[ 3654], 95.00th=[ 4817],
     | 99.00th=[19792], 99.50th=[33817], 99.90th=[66323], 99.95th=[66323],
     | 99.99th=[66323]
   bw (  KiB/s): min=   80, max=  296, per=24.27%, avg=196.57, stdev=55.93, samples=14
   iops        : min=   20, max=   74, avg=49.14, stdev=13.98, samples=14
  write: IOPS=55, BW=221KiB/s (227kB/s)(1636KiB/7392msec)
    clat (usec): min=8780, max=56374, avg=16016.08, stdev=4892.24
     lat (usec): min=8781, max=56376, avg=16017.23, stdev=4892.33
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[10028], 10.00th=[10945], 20.00th=[11731],
     | 30.00th=[12780], 40.00th=[14222], 50.00th=[15795], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[19530], 90.00th=[20579], 95.00th=[21890],
     | 99.00th=[34341], 99.50th=[37487], 99.90th=[56361], 99.95th=[56361],
     | 99.99th=[56361]
   bw (  KiB/s): min=  136, max=  272, per=25.75%, avg=219.43, stdev=33.61, samples=14
   iops        : min=   34, max=   68, avg=54.86, stdev= 8.40, samples=14
  lat (usec)   : 750=0.39%, 1000=5.86%
  lat (msec)   : 2=30.21%, 4=7.03%, 10=3.78%, 20=44.40%, 50=8.07%
  lat (msec)   : 100=0.26%
  cpu          : usr=0.07%, sys=2.37%, ctx=872, majf=0, minf=33
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19660: Tue Nov  5 12:02:16 2024
   read: IOPS=55, BW=223KiB/s (229kB/s)(1564KiB/7005msec)
    clat (usec): min=727, max=19585, avg=1959.02, stdev=2130.97
     lat (usec): min=728, max=19586, avg=1960.02, stdev=2130.97
    clat percentiles (usec):
     |  1.00th=[  807],  5.00th=[  898], 10.00th=[  971], 20.00th=[ 1057],
     | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1319], 60.00th=[ 1434],
     | 70.00th=[ 1663], 80.00th=[ 2180], 90.00th=[ 3359], 95.00th=[ 3916],
     | 99.00th=[12387], 99.50th=[14746], 99.90th=[19530], 99.95th=[19530],
     | 99.99th=[19530]
   bw (  KiB/s): min=  104, max=  352, per=27.57%, avg=223.36, stdev=74.81, samples=14
   iops        : min=   26, max=   88, avg=55.79, stdev=18.75, samples=14
  write: IOPS=53, BW=215KiB/s (220kB/s)(1508KiB/7005msec)
    clat (usec): min=8729, max=65647, avg=16522.33, stdev=5884.71
     lat (usec): min=8730, max=65648, avg=16523.51, stdev=5884.71
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[10028], 10.00th=[10814], 20.00th=[11731],
     | 30.00th=[12780], 40.00th=[14877], 50.00th=[16188], 60.00th=[17433],
     | 70.00th=[18482], 80.00th=[19530], 90.00th=[21103], 95.00th=[26870],
     | 99.00th=[38011], 99.50th=[53740], 99.90th=[65799], 99.95th=[65799],
     | 99.99th=[65799]
   bw (  KiB/s): min=  128, max=  256, per=25.21%, avg=214.79, stdev=31.28, samples=14
   iops        : min=   32, max=   64, avg=53.64, stdev= 7.76, samples=14
  lat (usec)   : 750=0.13%, 1000=6.77%
  lat (msec)   : 2=32.55%, 4=9.11%, 10=2.99%, 20=40.36%, 50=7.81%
  lat (msec)   : 100=0.26%
  cpu          : usr=0.14%, sys=2.40%, ctx=887, majf=0, minf=32
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19661: Tue Nov  5 12:02:16 2024
   read: IOPS=52, BW=210KiB/s (215kB/s)(1508KiB/7167msec)
    clat (usec): min=619, max=31761, avg=2337.26, stdev=3666.19
     lat (usec): min=620, max=31761, avg=2338.20, stdev=3666.17
    clat percentiles (usec):
     |  1.00th=[  717],  5.00th=[  865], 10.00th=[  930], 20.00th=[ 1057],
     | 30.00th=[ 1139], 40.00th=[ 1237], 50.00th=[ 1336], 60.00th=[ 1500],
     | 70.00th=[ 1696], 80.00th=[ 2147], 90.00th=[ 3458], 95.00th=[ 6915],
     | 99.00th=[22938], 99.50th=[25035], 99.90th=[31851], 99.95th=[31851],
     | 99.99th=[31851]
   bw (  KiB/s): min=   96, max=  296, per=26.02%, avg=210.79, stdev=66.70, samples=14
   iops        : min=   24, max=   74, avg=52.64, stdev=16.74, samples=14
  write: IOPS=54, BW=218KiB/s (223kB/s)(1564KiB/7167msec)
    clat (usec): min=8807, max=64456, avg=16047.83, stdev=5559.53
     lat (usec): min=8808, max=64457, avg=16048.99, stdev=5559.57
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[10421], 10.00th=[10945], 20.00th=[11600],
     | 30.00th=[12649], 40.00th=[14353], 50.00th=[15795], 60.00th=[16712],
     | 70.00th=[17957], 80.00th=[19006], 90.00th=[20317], 95.00th=[22414],
     | 99.00th=[38536], 99.50th=[53740], 99.90th=[64226], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  128, max=  248, per=25.61%, avg=218.21, stdev=36.87, samples=14
   iops        : min=   32, max=   62, avg=54.50, stdev= 9.29, samples=14
  lat (usec)   : 750=0.78%, 1000=7.16%
  lat (msec)   : 2=30.47%, 4=6.90%, 10=3.52%, 20=43.75%, 50=7.16%
  lat (msec)   : 100=0.26%
  cpu          : usr=0.06%, sys=2.43%, ctx=873, majf=0, minf=33
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=810KiB/s (830kB/s), 194KiB/s-223KiB/s (199kB/s-229kB/s), io=5988KiB (6132kB), run=7005-7392msec
  WRITE: bw=852KiB/s (873kB/s), 215KiB/s-221KiB/s (220kB/s-227kB/s), io=6300KiB (6451kB), run=7005-7392msec