== sanity test 398b: DIO and buffer IO race ============== 19:37:03 (1736987823)
/usr/bin/fio
12+0 records in
12+0 records out
12582912 bytes (13 MB) copied, 0.067988 s, 185 MB/s
mix direct rw 12M to OST0 by fio with 4 jobs...
mix buffer rw 12M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=18336: Wed Jan 15 19:37:05 2025
   read: IOPS=441, BW=1766KiB/s (1808kB/s)(1480KiB/838msec)
    clat (usec): min=3, max=26941, avg=1263.77, stdev=2788.80
     lat (usec): min=3, max=26941, avg=1264.97, stdev=2788.55
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    5], 10.00th=[    6], 20.00th=[    8],
     | 30.00th=[    9], 40.00th=[   11], 50.00th=[   13], 60.00th=[   21],
     | 70.00th=[ 1598], 80.00th=[ 2311], 90.00th=[ 3425], 95.00th=[ 4621],
     | 99.00th=[16712], 99.50th=[23725], 99.90th=[26870], 99.95th=[26870],
     | 99.99th=[26870]
   bw (  KiB/s): min= 2128, max= 2128, per=29.78%, avg=2128.00, stdev= 0.00, samples=1
   iops        : min=  532, max=  532, avg=532.00, stdev= 0.00, samples=1
  write: IOPS=474, BW=1900KiB/s (1945kB/s)(1592KiB/838msec)
    clat (usec): min=89, max=31248, avg=909.99, stdev=2060.45
     lat (usec): min=90, max=31249, avg=911.09, stdev=2060.54
    clat percentiles (usec):
     |  1.00th=[   91],  5.00th=[   99], 10.00th=[  106], 20.00th=[  124],
     | 30.00th=[  143], 40.00th=[  167], 50.00th=[  188], 60.00th=[  231],
     | 70.00th=[  766], 80.00th=[ 1221], 90.00th=[ 2474], 95.00th=[ 4146],
     | 99.00th=[ 6849], 99.50th=[ 8717], 99.90th=[31327], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 2296, max= 2296, per=30.54%, avg=2296.00, stdev= 0.00, samples=1
   iops        : min=  574, max=  574, avg=574.00, stdev= 0.00, samples=1
  lat (usec)   : 4=0.39%, 10=18.10%, 20=10.16%, 50=1.30%, 100=3.52%
  lat (usec)   : 250=29.30%, 500=3.39%, 750=0.65%, 1000=3.91%
  lat (msec)   : 2=9.77%, 4=13.80%, 10=4.69%, 20=0.65%, 50=0.39%
  cpu          : usr=0.84%, sys=22.22%, ctx=668, majf=0, minf=33
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18337: Wed Jan 15 19:37:05 2025
   read: IOPS=443, BW=1773KiB/s (1815kB/s)(1436KiB/810msec)
    clat (usec): min=3, max=15533, avg=1167.83, stdev=2085.41
     lat (usec): min=3, max=15533, avg=1168.49, stdev=2085.53
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    4], 10.00th=[    5], 20.00th=[    6],
     | 30.00th=[    7], 40.00th=[    9], 50.00th=[   11], 60.00th=[   55],
     | 70.00th=[ 1663], 80.00th=[ 2409], 90.00th=[ 3294], 95.00th=[ 4490],
     | 99.00th=[12125], 99.50th=[15008], 99.90th=[15533], 99.95th=[15533],
     | 99.99th=[15533]
   bw (  KiB/s): min= 2136, max= 2136, per=29.90%, avg=2136.00, stdev= 0.00, samples=1
   iops        : min=  534, max=  534, avg=534.00, stdev= 0.00, samples=1
  write: IOPS=504, BW=2020KiB/s (2068kB/s)(1636KiB/810msec)
    clat (usec): min=79, max=32252, avg=927.23, stdev=2371.26
     lat (usec): min=80, max=32253, avg=927.91, stdev=2371.37
    clat percentiles (usec):
     |  1.00th=[   82],  5.00th=[   87], 10.00th=[   93], 20.00th=[  104],
     | 30.00th=[  117], 40.00th=[  137], 50.00th=[  159], 60.00th=[  198],
     | 70.00th=[  742], 80.00th=[ 1205], 90.00th=[ 2343], 95.00th=[ 3687],
     | 99.00th=[ 9503], 99.50th=[12387], 99.90th=[32375], 99.95th=[32375],
     | 99.99th=[32375]
   bw (  KiB/s): min= 2320, max= 2320, per=30.86%, avg=2320.00, stdev= 0.00, samples=1
   iops        : min=  580, max=  580, avg=580.00, stdev= 0.00, samples=1
  lat (usec)   : 4=3.26%, 10=18.10%, 20=6.25%, 50=0.39%, 100=9.51%
  lat (usec)   : 250=25.52%, 500=1.82%, 750=1.56%, 1000=4.43%
  lat (msec)   : 2=10.68%, 4=13.54%, 10=3.91%, 20=0.78%, 50=0.26%
  cpu          : usr=0.25%, sys=20.89%, ctx=643, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18338: Wed Jan 15 19:37:05 2025
   read: IOPS=756, BW=3025KiB/s (3098kB/s)(1564KiB/517msec)
    clat (usec): min=2, max=29059, avg=737.49, stdev=2390.72
     lat (usec): min=3, max=29060, avg=738.07, stdev=2390.84
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    4], 10.00th=[    4], 20.00th=[    5],
     | 30.00th=[    6], 40.00th=[    7], 50.00th=[    7], 60.00th=[    8],
     | 70.00th=[   10], 80.00th=[  807], 90.00th=[ 2474], 95.00th=[ 3261],
     | 99.00th=[12649], 99.50th=[20317], 99.90th=[28967], 99.95th=[28967],
     | 99.99th=[28967]
   bw (  KiB/s): min= 3112, max= 3112, per=43.55%, avg=3112.00, stdev= 0.00, samples=1
   iops        : min=  778, max=  778, avg=778.00, stdev= 0.00, samples=1
  write: IOPS=729, BW=2917KiB/s (2987kB/s)(1508KiB/517msec)
    clat (usec): min=74, max=29733, avg=586.04, stdev=2212.92
     lat (usec): min=75, max=29734, avg=586.62, stdev=2212.94
    clat percentiles (usec):
     |  1.00th=[   78],  5.00th=[   85], 10.00th=[   90], 20.00th=[   99],
     | 30.00th=[  106], 40.00th=[  113], 50.00th=[  119], 60.00th=[  137],
     | 70.00th=[  159], 80.00th=[  210], 90.00th=[ 1205], 95.00th=[ 2409],
     | 99.00th=[ 9110], 99.50th=[18744], 99.90th=[29754], 99.95th=[29754],
     | 99.99th=[29754]
   bw (  KiB/s): min= 2944, max= 2944, per=39.16%, avg=2944.00, stdev= 0.00, samples=1
   iops        : min=  736, max=  736, avg=736.00, stdev= 0.00, samples=1
  lat (usec)   : 4=6.51%, 10=29.17%, 20=4.04%, 50=0.65%, 100=10.55%
  lat (usec)   : 250=30.47%, 500=0.78%, 750=0.39%, 1000=1.82%
  lat (msec)   : 2=5.60%, 4=7.29%, 10=1.69%, 20=0.65%, 50=0.39%
  cpu          : usr=0.58%, sys=22.09%, ctx=391, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18339: Wed Jan 15 19:37:05 2025
   read: IOPS=588, BW=2353KiB/s (2409kB/s)(1508KiB/641msec)
    clat (usec): min=3, max=15143, avg=955.46, stdev=2167.54
     lat (usec): min=3, max=15144, avg=955.98, stdev=2167.65
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    4], 10.00th=[    5], 20.00th=[    6],
     | 30.00th=[    8], 40.00th=[    9], 50.00th=[   10], 60.00th=[   12],
     | 70.00th=[   17], 80.00th=[ 1876], 90.00th=[ 2999], 95.00th=[ 4359],
     | 99.00th=[12125], 99.50th=[15139], 99.90th=[15139], 99.95th=[15139],
     | 99.99th=[15139]
   bw (  KiB/s): min= 2576, max= 2576, per=36.05%, avg=2576.00, stdev= 0.00, samples=1
   iops        : min=  644, max=  644, avg=644.00, stdev= 0.00, samples=1
  write: IOPS=609, BW=2440KiB/s (2498kB/s)(1564KiB/641msec)
    clat (usec): min=80, max=29854, avg=694.99, stdev=1933.82
     lat (usec): min=80, max=29855, avg=695.55, stdev=1933.85
    clat percentiles (usec):
     |  1.00th=[   85],  5.00th=[   97], 10.00th=[  104], 20.00th=[  120],
     | 30.00th=[  131], 40.00th=[  149], 50.00th=[  165], 60.00th=[  184],
     | 70.00th=[  233], 80.00th=[  824], 90.00th=[ 1582], 95.00th=[ 3195],
     | 99.00th=[ 7242], 99.50th=[ 9765], 99.90th=[29754], 99.95th=[29754],
     | 99.99th=[29754]
   bw (  KiB/s): min= 2688, max= 2688, per=35.76%, avg=2688.00, stdev= 0.00, samples=1
   iops        : min=  672, max=  672, avg=672.00, stdev= 0.00, samples=1
  lat (usec)   : 4=3.52%, 10=22.14%, 20=8.72%, 50=0.52%, 100=3.78%
  lat (usec)   : 250=33.07%, 500=2.60%, 750=0.39%, 1000=4.17%
  lat (msec)   : 2=7.81%, 4=8.07%, 10=4.17%, 20=0.91%, 50=0.13%
  cpu          : usr=0.16%, sys=20.31%, ctx=501, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7146KiB/s (7317kB/s), 1766KiB/s-3025KiB/s (1808kB/s-3098kB/s), io=5988KiB (6132kB), run=517-838msec
  WRITE: bw=7518KiB/s (7698kB/s), 1900KiB/s-2917KiB/s (1945kB/s-2987kB/s), io=6300KiB (6451kB), run=517-838msec

rand-rw: (groupid=0, jobs=1): err= 0: pid=18341: Wed Jan 15 19:37:06 2025
   read: IOPS=219, BW=879KiB/s (900kB/s)(1480KiB/1684msec)
    clat (usec): min=550, max=23353, avg=1581.85, stdev=2002.89
     lat (usec): min=551, max=23354, avg=1582.49, stdev=2002.91
    clat percentiles (usec):
     |  1.00th=[  627],  5.00th=[  701], 10.00th=[  758], 20.00th=[  824],
     | 30.00th=[  873], 40.00th=[  947], 50.00th=[ 1012], 60.00th=[ 1090],
     | 70.00th=[ 1254], 80.00th=[ 1516], 90.00th=[ 2606], 95.00th=[ 4293],
     | 99.00th=[13304], 99.50th=[13566], 99.90th=[23462], 99.95th=[23462],
     | 99.99th=[23462]
   bw (  KiB/s): min=  496, max= 1120, per=23.63%, avg=840.00, stdev=316.88, samples=3
   iops        : min=  124, max=  280, avg=210.00, stdev=79.22, samples=3
  write: IOPS=236, BW=945KiB/s (968kB/s)(1592KiB/1684msec)
    clat (usec): min=1191, max=32171, avg=2735.18, stdev=2176.93
     lat (usec): min=1192, max=32172, avg=2736.04, stdev=2176.91
    clat percentiles (usec):
     |  1.00th=[ 1270],  5.00th=[ 1516], 10.00th=[ 1696], 20.00th=[ 1909],
     | 30.00th=[ 2057], 40.00th=[ 2147], 50.00th=[ 2212], 60.00th=[ 2376],
     | 70.00th=[ 2573], 80.00th=[ 2900], 90.00th=[ 4080], 95.00th=[ 5473],
     | 99.00th=[12387], 99.50th=[16712], 99.90th=[32113], 99.95th=[32113],
     | 99.99th=[32113]
   bw (  KiB/s): min=  528, max= 1280, per=24.31%, avg=909.33, stdev=376.11, samples=3
   iops        : min=  132, max=  320, avg=227.33, stdev=94.03, samples=3
  lat (usec)   : 750=4.17%, 1000=18.49%
  lat (msec)   : 2=31.77%, 4=37.89%, 10=6.38%, 20=1.04%, 50=0.26%
  cpu          : usr=0.42%, sys=6.30%, ctx=811, majf=0, minf=32
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18342: Wed Jan 15 19:37:06 2025
   read: IOPS=214, BW=858KiB/s (878kB/s)(1436KiB/1674msec)
    clat (usec): min=589, max=28778, avg=1518.54, stdev=2150.95
     lat (usec): min=590, max=28778, avg=1519.16, stdev=2150.96
    clat percentiles (usec):
     |  1.00th=[  627],  5.00th=[  685], 10.00th=[  750], 20.00th=[  824],
     | 30.00th=[  873], 40.00th=[  922], 50.00th=[  988], 60.00th=[ 1074],
     | 70.00th=[ 1205], 80.00th=[ 1549], 90.00th=[ 2868], 95.00th=[ 4146],
     | 99.00th=[ 7504], 99.50th=[23725], 99.90th=[28705], 99.95th=[28705],
     | 99.99th=[28705]
   bw (  KiB/s): min=  448, max= 1248, per=23.33%, avg=829.33, stdev=401.30, samples=3
   iops        : min=  112, max=  312, avg=207.33, stdev=100.33, samples=3
  write: IOPS=244, BW=977KiB/s (1001kB/s)(1636KiB/1674msec)
    clat (usec): min=1139, max=21340, avg=2744.76, stdev=1902.00
     lat (usec): min=1140, max=21341, avg=2745.65, stdev=1901.97
    clat percentiles (usec):
     |  1.00th=[ 1270],  5.00th=[ 1549], 10.00th=[ 1729], 20.00th=[ 1942],
     | 30.00th=[ 2040], 40.00th=[ 2114], 50.00th=[ 2245], 60.00th=[ 2343],
     | 70.00th=[ 2540], 80.00th=[ 2999], 90.00th=[ 4359], 95.00th=[ 5800],
     | 99.00th=[10421], 99.50th=[12125], 99.90th=[21365], 99.95th=[21365],
     | 99.99th=[21365]
   bw (  KiB/s): min=  520, max= 1288, per=25.16%, avg=941.33, stdev=389.41, samples=3
   iops        : min=  130, max=  322, avg=235.33, stdev=97.35, samples=3
  lat (usec)   : 750=4.69%, 1000=19.27%
  lat (msec)   : 2=29.95%, 4=38.15%, 10=7.03%, 20=0.39%, 50=0.52%
  cpu          : usr=0.36%, sys=6.28%, ctx=800, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18343: Wed Jan 15 19:37:06 2025
   read: IOPS=238, BW=955KiB/s (978kB/s)(1564KiB/1638msec)
    clat (usec): min=583, max=14172, avg=1437.60, stdev=1537.17
     lat (usec): min=584, max=14173, avg=1438.23, stdev=1537.18
    clat percentiles (usec):
     |  1.00th=[  652],  5.00th=[  734], 10.00th=[  775], 20.00th=[  840],
     | 30.00th=[  881], 40.00th=[  930], 50.00th=[  988], 60.00th=[ 1057],
     | 70.00th=[ 1156], 80.00th=[ 1385], 90.00th=[ 2089], 95.00th=[ 4424],
     | 99.00th=[10028], 99.50th=[12125], 99.90th=[14222], 99.95th=[14222],
     | 99.99th=[14222]
   bw (  KiB/s): min=  432, max= 1320, per=25.88%, avg=920.00, stdev=450.49, samples=3
   iops        : min=  108, max=  330, avg=230.00, stdev=112.62, samples=3
  write: IOPS=230, BW=921KiB/s (943kB/s)(1508KiB/1638msec)
    clat (usec): min=1134, max=32142, avg=2828.23, stdev=2644.71
     lat (usec): min=1135, max=32143, avg=2829.03, stdev=2644.75
    clat percentiles (usec):
     |  1.00th=[ 1221],  5.00th=[ 1516], 10.00th=[ 1729], 20.00th=[ 1909],
     | 30.00th=[ 2008], 40.00th=[ 2114], 50.00th=[ 2212], 60.00th=[ 2343],
     | 70.00th=[ 2540], 80.00th=[ 2868], 90.00th=[ 4424], 95.00th=[ 5735],
     | 99.00th=[12649], 99.50th=[28443], 99.90th=[32113], 99.95th=[32113],
     | 99.99th=[32113]
   bw (  KiB/s): min=  520, max= 1224, per=23.81%, avg=890.67, stdev=353.48, samples=3
   iops        : min=  130, max=  306, avg=222.67, stdev=88.37, samples=3
  lat (usec)   : 750=3.65%, 1000=23.05%
  lat (msec)   : 2=32.42%, 4=32.29%, 10=7.55%, 20=0.65%, 50=0.39%
  cpu          : usr=0.67%, sys=6.11%, ctx=807, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18344: Wed Jan 15 19:37:06 2025
   read: IOPS=229, BW=919KiB/s (941kB/s)(1508KiB/1641msec)
    clat (usec): min=573, max=24658, avg=1415.31, stdev=1633.18
     lat (usec): min=574, max=24659, avg=1415.94, stdev=1633.20
    clat percentiles (usec):
     |  1.00th=[  619],  5.00th=[  709], 10.00th=[  766], 20.00th=[  824],
     | 30.00th=[  881], 40.00th=[  914], 50.00th=[  955], 60.00th=[ 1029],
     | 70.00th=[ 1156], 80.00th=[ 1418], 90.00th=[ 2376], 95.00th=[ 4293],
     | 99.00th=[ 6849], 99.50th=[ 7963], 99.90th=[24773], 99.95th=[24773],
     | 99.99th=[24773]
   bw (  KiB/s): min=  495, max= 1144, per=24.82%, avg=882.33, stdev=342.26, samples=3
   iops        : min=  123, max=  286, avg=220.33, stdev=85.99, samples=3
  write: IOPS=238, BW=953KiB/s (976kB/s)(1564KiB/1641msec)
    clat (usec): min=1149, max=32139, avg=2812.36, stdev=2557.50
     lat (usec): min=1149, max=32140, avg=2813.15, stdev=2557.52
    clat percentiles (usec):
     |  1.00th=[ 1205],  5.00th=[ 1434], 10.00th=[ 1598], 20.00th=[ 1860],
     | 30.00th=[ 2024], 40.00th=[ 2114], 50.00th=[ 2245], 60.00th=[ 2376],
     | 70.00th=[ 2606], 80.00th=[ 3097], 90.00th=[ 4424], 95.00th=[ 5866],
     | 99.00th=[12518], 99.50th=[32113], 99.90th=[32113], 99.95th=[32113],
     | 99.99th=[32113]
   bw (  KiB/s): min=  542, max= 1304, per=24.72%, avg=924.67, stdev=381.01, samples=3
   iops        : min=  135, max=  326, avg=231.00, stdev=95.50, samples=3
  lat (usec)   : 750=3.65%, 1000=24.61%
  lat (msec)   : 2=28.78%, 4=34.11%, 10=8.20%, 20=0.26%, 50=0.39%
  cpu          : usr=0.24%, sys=6.10%, ctx=808, majf=0, minf=32
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3556KiB/s (3641kB/s), 858KiB/s-955KiB/s (878kB/s-978kB/s), io=5988KiB (6132kB), run=1638-1684msec
  WRITE: bw=3741KiB/s (3831kB/s), 921KiB/s-977KiB/s (943kB/s-1001kB/s), io=6300KiB (6451kB), run=1638-1684msec