== sanity test 398b: DIO and buffer IO race ============== 19:11:33 (1717024293)
/usr/bin/fio
12+0 records in
12+0 records out
12582912 bytes (13 MB) copied, 0.0752862 s, 167 MB/s
mix direct rw 12M to OST0 by fio with 4 jobs...
mix buffer rw 12M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=16122: Wed May 29 19:11:36 2024
   read: IOPS=284, BW=1139KiB/s (1167kB/s)(1480KiB/1299msec)
    clat (usec): min=4, max=31616, avg=1974.25, stdev=2729.02
     lat (usec): min=4, max=31617, avg=1975.21, stdev=2728.96
    clat percentiles (usec):
     |  1.00th=[    6],  5.00th=[    8], 10.00th=[   10], 20.00th=[   13],
     | 30.00th=[   15], 40.00th=[   24], 50.00th=[ 1680], 60.00th=[ 2311],
     | 70.00th=[ 2835], 80.00th=[ 3523], 90.00th=[ 4178], 95.00th=[ 5145],
     | 99.00th=[13435], 99.50th=[16909], 99.90th=[31589], 99.95th=[31589],
     | 99.99th=[31589]
   bw (  KiB/s): min=  912, max= 1096, per=23.26%, avg=1004.00, stdev=130.11, samples=2
   iops        : min=  228, max=  274, avg=251.00, stdev=32.53, samples=2
  write: IOPS=306, BW=1226KiB/s (1255kB/s)(1592KiB/1299msec)
    clat (usec): min=87, max=13061, avg=1407.65, stdev=1827.42
     lat (usec): min=88, max=13061, avg=1408.59, stdev=1827.30
    clat percentiles (usec):
     |  1.00th=[   99],  5.00th=[  122], 10.00th=[  149], 20.00th=[  174],
     | 30.00th=[  192], 40.00th=[  247], 50.00th=[  799], 60.00th=[ 1123],
     | 70.00th=[ 1549], 80.00th=[ 2343], 90.00th=[ 3884], 95.00th=[ 5342],
     | 99.00th=[ 8848], 99.50th=[10945], 99.90th=[13042], 99.95th=[13042],
     | 99.99th=[13042]
   bw (  KiB/s): min=  912, max= 1312, per=24.48%, avg=1112.00, stdev=282.84, samples=2
   iops        : min=  228, max=  328, avg=278.00, stdev=70.71, samples=2
  lat (usec)   : 10=6.12%, 20=12.76%, 50=1.17%, 100=0.52%, 250=20.57%
  lat (usec)   : 500=3.78%, 750=1.04%, 1000=4.56%
  lat (msec)   : 2=14.32%, 4=24.48%, 10=9.77%, 20=0.78%, 50=0.13%
  cpu          : usr=0.54%, sys=15.49%, ctx=905, majf=0, minf=35
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16123: Wed May 29 19:11:36 2024
   read: IOPS=258, BW=1035KiB/s (1060kB/s)(1436KiB/1387msec)
    clat (usec): min=4, max=32751, avg=2121.65, stdev=2760.54
     lat (usec): min=5, max=32751, avg=2122.43, stdev=2760.61
    clat percentiles (usec):
     |  1.00th=[    6],  5.00th=[    8], 10.00th=[   10], 20.00th=[   13],
     | 30.00th=[   16], 40.00th=[ 1172], 50.00th=[ 1909], 60.00th=[ 2409],
     | 70.00th=[ 3032], 80.00th=[ 3523], 90.00th=[ 4490], 95.00th=[ 5473],
     | 99.00th=[10683], 99.50th=[17957], 99.90th=[32637], 99.95th=[32637],
     | 99.99th=[32637]
   bw (  KiB/s): min=  856, max= 1016, per=21.68%, avg=936.00, stdev=113.14, samples=2
   iops        : min=  214, max=  254, avg=234.00, stdev=28.28, samples=2
  write: IOPS=294, BW=1180KiB/s (1208kB/s)(1636KiB/1387msec)
    clat (usec): min=94, max=21198, avg=1512.34, stdev=2087.70
     lat (usec): min=94, max=21198, avg=1513.11, stdev=2087.68
    clat percentiles (usec):
     |  1.00th=[  110],  5.00th=[  122], 10.00th=[  147], 20.00th=[  174],
     | 30.00th=[  192], 40.00th=[  273], 50.00th=[ 1012], 60.00th=[ 1221],
     | 70.00th=[ 1598], 80.00th=[ 2376], 90.00th=[ 3949], 95.00th=[ 5473],
     | 99.00th=[ 7832], 99.50th=[13829], 99.90th=[21103], 99.95th=[21103],
     | 99.99th=[21103]
   bw (  KiB/s): min=  960, max= 1168, per=23.43%, avg=1064.00, stdev=147.08, samples=2
   iops        : min=  240, max=  292, avg=266.00, stdev=36.77, samples=2
  lat (usec)   : 10=5.73%, 20=10.81%, 50=0.91%, 100=0.13%, 250=20.44%
  lat (usec)   : 500=2.34%, 750=1.69%, 1000=2.86%
  lat (msec)   : 2=19.40%, 4=23.83%, 10=10.81%, 20=0.78%, 50=0.26%
  cpu          : usr=0.72%, sys=14.86%, ctx=933, majf=0, minf=32
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16124: Wed May 29 19:11:36 2024
   read: IOPS=293, BW=1176KiB/s (1204kB/s)(1564KiB/1330msec)
    clat (usec): min=5, max=16481, avg=2013.77, stdev=2365.73
     lat (usec): min=5, max=16482, avg=2014.45, stdev=2365.75
    clat percentiles (usec):
     |  1.00th=[    7],  5.00th=[    9], 10.00th=[   10], 20.00th=[   13],
     | 30.00th=[   16], 40.00th=[ 1057], 50.00th=[ 1844], 60.00th=[ 2311],
     | 70.00th=[ 2802], 80.00th=[ 3392], 90.00th=[ 4178], 95.00th=[ 5735],
     | 99.00th=[11994], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16450]
   bw (  KiB/s): min=  984, max= 1128, per=24.46%, avg=1056.00, stdev=101.82, samples=2
   iops        : min=  246, max=  282, avg=264.00, stdev=25.46, samples=2
  write: IOPS=283, BW=1134KiB/s (1161kB/s)(1508KiB/1330msec)
    clat (usec): min=94, max=17804, avg=1420.47, stdev=1825.84
     lat (usec): min=95, max=17805, avg=1421.33, stdev=1825.81
    clat percentiles (usec):
     |  1.00th=[  102],  5.00th=[  141], 10.00th=[  153], 20.00th=[  169],
     | 30.00th=[  184], 40.00th=[  223], 50.00th=[  930], 60.00th=[ 1221],
     | 70.00th=[ 1745], 80.00th=[ 2474], 90.00th=[ 3621], 95.00th=[ 4359],
     | 99.00th=[ 7308], 99.50th=[12387], 99.90th=[17695], 99.95th=[17695],
     | 99.99th=[17695]
   bw (  KiB/s): min= 1008, max= 1016, per=22.28%, avg=1012.00, stdev= 5.66, samples=2
   iops        : min=  252, max=  254, avg=253.00, stdev= 1.41, samples=2
  lat (usec)   : 10=5.21%, 20=12.63%, 50=0.65%, 100=1.04%, 250=20.18%
  lat (usec)   : 500=1.95%, 750=0.78%, 1000=2.99%
  lat (msec)   : 2=18.62%, 4=26.04%, 10=8.85%, 20=1.04%
  cpu          : usr=0.15%, sys=15.80%, ctx=936, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16125: Wed May 29 19:11:36 2024
   read: IOPS=273, BW=1096KiB/s (1122kB/s)(1508KiB/1376msec)
    clat (usec): min=4, max=14773, avg=2089.39, stdev=2283.88
     lat (usec): min=4, max=14774, avg=2090.05, stdev=2283.89
    clat percentiles (usec):
     |  1.00th=[    6],  5.00th=[    8], 10.00th=[   10], 20.00th=[   12],
     | 30.00th=[   16], 40.00th=[ 1139], 50.00th=[ 2024], 60.00th=[ 2540],
     | 70.00th=[ 2999], 80.00th=[ 3458], 90.00th=[ 4228], 95.00th=[ 5997],
     | 99.00th=[12518], 99.50th=[14746], 99.90th=[14746], 99.95th=[14746],
     | 99.99th=[14746]
   bw (  KiB/s): min=  888, max= 1144, per=23.53%, avg=1016.00, stdev=181.02, samples=2
   iops        : min=  222, max=  286, avg=254.00, stdev=45.25, samples=2
  write: IOPS=284, BW=1137KiB/s (1164kB/s)(1564KiB/1376msec)
    clat (usec): min=93, max=21461, avg=1485.34, stdev=2098.08
     lat (usec): min=93, max=21462, avg=1486.14, stdev=2098.08
    clat percentiles (usec):
     |  1.00th=[  106],  5.00th=[  123], 10.00th=[  149], 20.00th=[  167],
     | 30.00th=[  194], 40.00th=[  269], 50.00th=[  930], 60.00th=[ 1156],
     | 70.00th=[ 1598], 80.00th=[ 2540], 90.00th=[ 3851], 95.00th=[ 5014],
     | 99.00th=[ 9503], 99.50th=[15795], 99.90th=[21365], 99.95th=[21365],
     | 99.99th=[21365]
   bw (  KiB/s): min=  912, max= 1192, per=23.16%, avg=1052.00, stdev=197.99, samples=2
   iops        : min=  228, max=  298, avg=263.00, stdev=49.50, samples=2
  lat (usec)   : 10=5.99%, 20=10.94%, 50=0.65%, 100=0.78%, 250=19.66%
  lat (usec)   : 500=3.39%, 750=0.78%, 1000=2.99%
  lat (msec)   : 2=17.19%, 4=27.08%, 10=9.37%, 20=1.04%, 50=0.13%
  cpu          : usr=0.58%, sys=14.04%, ctx=950, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=4317KiB/s (4421kB/s), 1035KiB/s-1176KiB/s (1060kB/s-1204kB/s), io=5988KiB (6132kB), run=1299-1387msec
  WRITE: bw=4542KiB/s (4651kB/s), 1134KiB/s-1226KiB/s (1161kB/s-1255kB/s), io=6300KiB (6451kB), run=1299-1387msec

rand-rw: (groupid=0, jobs=1): err= 0: pid=16118: Wed May 29 19:11:37 2024
   read: IOPS=181, BW=725KiB/s (743kB/s)(1480KiB/2041msec)
    clat (usec): min=625, max=16696, avg=1820.49, stdev=1693.62
     lat (usec): min=625, max=16697, avg=1821.23, stdev=1693.66
    clat percentiles (usec):
     |  1.00th=[  685],  5.00th=[  742], 10.00th=[  799], 20.00th=[  873],
     | 30.00th=[  930], 40.00th=[ 1004], 50.00th=[ 1090], 60.00th=[ 1287],
     | 70.00th=[ 1680], 80.00th=[ 2409], 90.00th=[ 4047], 95.00th=[ 5145],
     | 99.00th=[ 7963], 99.50th=[ 8717], 99.90th=[16712], 99.95th=[16712],
     | 99.99th=[16712]
   bw (  KiB/s): min=  488, max= 1208, per=24.34%, avg=714.00, stdev=334.11, samples=4
   iops        : min=  122, max=  302, avg=178.50, stdev=83.53, samples=4
  write: IOPS=195, BW=780KiB/s (799kB/s)(1592KiB/2041msec)
    clat (usec): min=1257, max=17717, avg=3399.90, stdev=2182.36
     lat (usec): min=1258, max=17718, avg=3400.80, stdev=2182.46
    clat percentiles (usec):
     |  1.00th=[ 1336],  5.00th=[ 1598], 10.00th=[ 1795], 20.00th=[ 1975],
     | 30.00th=[ 2114], 40.00th=[ 2278], 50.00th=[ 2442], 60.00th=[ 2769],
     | 70.00th=[ 3621], 80.00th=[ 4686], 90.00th=[ 6259], 95.00th=[ 7832],
     | 99.00th=[11600], 99.50th=[12649], 99.90th=[17695], 99.95th=[17695],
     | 99.99th=[17695]
   bw (  KiB/s): min=  504, max= 1328, per=25.15%, avg=776.00, stdev=382.05, samples=4
   iops        : min=  126, max=  332, avg=194.00, stdev=95.51, samples=4
  lat (usec)   : 750=2.47%, 1000=16.67%
  lat (msec)   : 2=27.99%, 4=34.51%, 10=16.93%, 20=1.43%
  cpu          : usr=0.20%, sys=5.78%, ctx=827, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16119: Wed May 29 19:11:37 2024
   read: IOPS=178, BW=715KiB/s (733kB/s)(1436KiB/2007msec)
    clat (usec): min=636, max=17246, avg=1679.88, stdev=1613.48
     lat (usec): min=637, max=17247, avg=1680.62, stdev=1613.52
    clat percentiles (usec):
     |  1.00th=[  652],  5.00th=[  725], 10.00th=[  775], 20.00th=[  848],
     | 30.00th=[  914], 40.00th=[  971], 50.00th=[ 1057], 60.00th=[ 1172],
     | 70.00th=[ 1483], 80.00th=[ 1975], 90.00th=[ 3982], 95.00th=[ 4948],
     | 99.00th=[ 7570], 99.50th=[ 8586], 99.90th=[17171], 99.95th=[17171],
     | 99.99th=[17171]
   bw (  KiB/s): min=  496, max= 1200, per=24.28%, avg=712.00, stdev=332.75, samples=4
   iops        : min=  124, max=  300, avg=178.00, stdev=83.19, samples=4
  write: IOPS=203, BW=815KiB/s (835kB/s)(1636KiB/2007msec)
    clat (usec): min=1145, max=17463, avg=3399.39, stdev=2258.52
     lat (usec): min=1146, max=17464, avg=3400.23, stdev=2258.57
    clat percentiles (usec):
     |  1.00th=[ 1270],  5.00th=[ 1516], 10.00th=[ 1696], 20.00th=[ 1958],
     | 30.00th=[ 2114], 40.00th=[ 2245], 50.00th=[ 2507], 60.00th=[ 2900],
     | 70.00th=[ 3752], 80.00th=[ 4555], 90.00th=[ 6325], 95.00th=[ 7767],
     | 99.00th=[11731], 99.50th=[14615], 99.90th=[17433], 99.95th=[17433],
     | 99.99th=[17433]
   bw (  KiB/s): min=  560, max= 1384, per=26.38%, avg=814.00, stdev=382.69, samples=4
   iops        : min=  140, max=  346, avg=203.50, stdev=95.67, samples=4
  lat (usec)   : 750=3.39%, 1000=17.06%
  lat (msec)   : 2=29.69%, 4=32.16%, 10=16.54%, 20=1.17%
  cpu          : usr=0.70%, sys=5.28%, ctx=825, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16120: Wed May 29 19:11:37 2024
   read: IOPS=194, BW=779KiB/s (798kB/s)(1564KiB/2007msec)
    clat (usec): min=625, max=14243, avg=1876.07, stdev=1824.44
     lat (usec): min=625, max=14244, avg=1876.76, stdev=1824.47
    clat percentiles (usec):
     |  1.00th=[  644],  5.00th=[  742], 10.00th=[  783], 20.00th=[  848],
     | 30.00th=[  914], 40.00th=[  996], 50.00th=[ 1090], 60.00th=[ 1303],
     | 70.00th=[ 1614], 80.00th=[ 2343], 90.00th=[ 4621], 95.00th=[ 6063],
     | 99.00th=[ 9634], 99.50th=[10159], 99.90th=[14222], 99.95th=[14222],
     | 99.99th=[14222]
   bw (  KiB/s): min=  440, max= 1424, per=26.66%, avg=782.00, stdev=438.68, samples=4
   iops        : min=  110, max=  356, avg=195.50, stdev=109.67, samples=4
  write: IOPS=187, BW=751KiB/s (769kB/s)(1508KiB/2007msec)
    clat (usec): min=1235, max=20546, avg=3343.72, stdev=2260.27
     lat (usec): min=1236, max=20548, avg=3344.58, stdev=2260.31
    clat percentiles (usec):
     |  1.00th=[ 1303],  5.00th=[ 1549], 10.00th=[ 1729], 20.00th=[ 1893],
     | 30.00th=[ 2089], 40.00th=[ 2245], 50.00th=[ 2442], 60.00th=[ 2933],
     | 70.00th=[ 3523], 80.00th=[ 4424], 90.00th=[ 5866], 95.00th=[ 7570],
     | 99.00th=[13435], 99.50th=[14615], 99.90th=[20579], 99.95th=[20579],
     | 99.99th=[20579]
   bw (  KiB/s): min=  512, max= 1248, per=24.24%, avg=748.00, stdev=344.50, samples=4
   iops        : min=  128, max=  312, avg=187.00, stdev=86.12, samples=4
  lat (usec)   : 750=2.73%, 1000=18.36%
  lat (msec)   : 2=30.60%, 4=29.56%, 10=17.32%, 20=1.30%, 50=0.13%
  cpu          : usr=0.50%, sys=5.43%, ctx=821, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16121: Wed May 29 19:11:37 2024
   read: IOPS=190, BW=762KiB/s (780kB/s)(1508KiB/1980msec)
    clat (usec): min=593, max=15668, avg=1857.41, stdev=1868.70
     lat (usec): min=593, max=15668, avg=1858.53, stdev=1868.52
    clat percentiles (usec):
     |  1.00th=[  676],  5.00th=[  734], 10.00th=[  791], 20.00th=[  857],
     | 30.00th=[  914], 40.00th=[  996], 50.00th=[ 1123], 60.00th=[ 1287],
     | 70.00th=[ 1565], 80.00th=[ 2311], 90.00th=[ 3851], 95.00th=[ 5932],
     | 99.00th=[ 9503], 99.50th=[13829], 99.90th=[15664], 99.95th=[15664],
     | 99.99th=[15664]
   bw (  KiB/s): min=  536, max=  624, per=20.18%, avg=592.00, stdev=48.66, samples=3
   iops        : min=  134, max=  156, avg=148.00, stdev=12.17, samples=3
  write: IOPS=197, BW=790KiB/s (809kB/s)(1564KiB/1980msec)
    clat (usec): min=1077, max=20323, avg=3248.36, stdev=2079.71
     lat (usec): min=1078, max=20325, avg=3249.23, stdev=2079.81
    clat percentiles (usec):
     |  1.00th=[ 1237],  5.00th=[ 1516], 10.00th=[ 1745], 20.00th=[ 1909],
     | 30.00th=[ 2057], 40.00th=[ 2245], 50.00th=[ 2442], 60.00th=[ 2737],
     | 70.00th=[ 3326], 80.00th=[ 4555], 90.00th=[ 5866], 95.00th=[ 7111],
     | 99.00th=[10290], 99.50th=[13304], 99.90th=[20317], 99.95th=[20317],
     | 99.99th=[20317]
   bw (  KiB/s): min=  544, max=  704, per=19.87%, avg=613.33, stdev=82.11, samples=3
   iops        : min=  136, max=  176, avg=153.33, stdev=20.53, samples=3
  lat (usec)   : 750=2.86%, 1000=16.80%
  lat (msec)   : 2=30.60%, 4=33.33%, 10=15.49%, 20=0.78%, 50=0.13%
  cpu          : usr=0.15%, sys=6.01%, ctx=815, majf=0, minf=30
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2934KiB/s (3004kB/s), 715KiB/s-779KiB/s (733kB/s-798kB/s), io=5988KiB (6132kB), run=1980-2041msec
  WRITE: bw=3087KiB/s (3161kB/s), 751KiB/s-815KiB/s (769kB/s-835kB/s), io=6300KiB (6451kB), run=1980-2041msec