== sanity test 398b: DIO and buffer IO race ============== 18:05:57 (1718402757)
/usr/bin/fio
12+0 records in
12+0 records out
12582912 bytes (13 MB) copied, 0.0736137 s, 171 MB/s
mix direct rw 12M to OST0 by fio with 4 jobs...
mix buffer rw 12M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=19519: Fri Jun 14 18:06:00 2024
   read: IOPS=173, BW=693KiB/s (710kB/s)(1480KiB/2136msec)
    clat (usec): min=3, max=34309, avg=1526.48, stdev=3513.77
     lat (usec): min=4, max=34310, avg=1527.48, stdev=3513.77
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    7], 10.00th=[    8], 20.00th=[   10],
     | 30.00th=[   13], 40.00th=[   17], 50.00th=[   30], 60.00th=[ 1156],
     | 70.00th=[ 1811], 80.00th=[ 2311], 90.00th=[ 3261], 95.00th=[ 4178],
     | 99.00th=[23200], 99.50th=[27657], 99.90th=[34341], 99.95th=[34341],
     | 99.99th=[34341]
   bw (  KiB/s): min=  560, max=  854, per=24.89%, avg=697.50, stdev=143.77, samples=4
   iops        : min=  140, max=  213, avg=174.25, stdev=35.76, samples=4
  write: IOPS=186, BW=745KiB/s (763kB/s)(1592KiB/2136msec)
    clat (usec): min=89, max=32364, avg=3921.28, stdev=6254.96
     lat (usec): min=89, max=32364, avg=3922.47, stdev=6254.72
    clat percentiles (usec):
     |  1.00th=[   96],  5.00th=[  115], 10.00th=[  129], 20.00th=[  163],
     | 30.00th=[  186], 40.00th=[  239], 50.00th=[ 1106], 60.00th=[ 1631],
     | 70.00th=[ 2769], 80.00th=[ 8356], 90.00th=[12911], 95.00th=[16909],
     | 99.00th=[29492], 99.50th=[31065], 99.90th=[32375], 99.95th=[32375],
     | 99.99th=[32375]
   bw (  KiB/s): min=  584, max=  934, per=25.42%, avg=749.50, stdev=168.26, samples=4
   iops        : min=  146, max=  233, avg=187.25, stdev=41.88, samples=4
  lat (usec)   : 4=0.26%, 10=9.64%, 20=10.42%, 50=5.08%, 100=1.04%
  lat (usec)   : 250=20.70%, 500=1.82%, 750=0.65%, 1000=2.99%
  lat (msec)   : 2=16.67%, 4=14.32%, 10=7.29%, 20=6.90%, 50=2.21%
  cpu          : usr=0.14%, sys=7.68%, ctx=860, majf=0, minf=35
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19520: Fri Jun 14 18:06:00 2024
   read: IOPS=169, BW=678KiB/s (694kB/s)(1436KiB/2118msec)
    clat (usec): min=3, max=27826, avg=1291.40, stdev=3120.94
     lat (usec): min=4, max=27827, avg=1292.05, stdev=3120.95
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    7], 10.00th=[    8], 20.00th=[   10],
     | 30.00th=[   12], 40.00th=[   15], 50.00th=[   20], 60.00th=[   51],
     | 70.00th=[ 1450], 80.00th=[ 2212], 90.00th=[ 3097], 95.00th=[ 4228],
     | 99.00th=[20841], 99.50th=[24773], 99.90th=[27919], 99.95th=[27919],
     | 99.99th=[27919]
   bw (  KiB/s): min=  560, max=  928, per=24.27%, avg=680.00, stdev=170.58, samples=4
   iops        : min=  140, max=  232, avg=170.00, stdev=42.65, samples=4
  write: IOPS=193, BW=772KiB/s (791kB/s)(1636KiB/2118msec)
    clat (usec): min=91, max=40293, avg=4012.24, stdev=7101.39
     lat (usec): min=92, max=40293, avg=4013.32, stdev=7101.78
    clat percentiles (usec):
     |  1.00th=[  102],  5.00th=[  119], 10.00th=[  135], 20.00th=[  165],
     | 30.00th=[  180], 40.00th=[  200], 50.00th=[  578], 60.00th=[ 1254],
     | 70.00th=[ 2474], 80.00th=[ 6915], 90.00th=[14484], 95.00th=[20317],
     | 99.00th=[33162], 99.50th=[35390], 99.90th=[40109], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min=  624, max= 1088, per=25.92%, avg=764.00, stdev=218.75, samples=4
   iops        : min=  156, max=  272, avg=191.00, stdev=54.69, samples=4
  lat (usec)   : 4=0.39%, 10=10.03%, 20=13.02%, 50=4.56%, 100=1.04%
  lat (usec)   : 250=25.13%, 500=1.30%, 750=0.52%, 1000=2.73%
  lat (msec)   : 2=13.41%, 4=13.02%, 10=5.08%, 20=6.38%, 50=3.39%
  cpu          : usr=0.14%, sys=6.90%, ctx=786, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19521: Fri Jun 14 18:06:00 2024
   read: IOPS=182, BW=732KiB/s (749kB/s)(1564KiB/2137msec)
    clat (usec): min=3, max=31187, avg=1505.35, stdev=3549.89
     lat (usec): min=3, max=31187, avg=1505.94, stdev=3549.90
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    6], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   11], 40.00th=[   14], 50.00th=[   24], 60.00th=[ 1045],
     | 70.00th=[ 1631], 80.00th=[ 2343], 90.00th=[ 3163], 95.00th=[ 4228],
     | 99.00th=[21365], 99.50th=[28705], 99.90th=[31065], 99.95th=[31065],
     | 99.99th=[31065]
   bw (  KiB/s): min=  648, max=  800, per=25.98%, avg=728.00, stdev=68.82, samples=4
   iops        : min=  162, max=  200, avg=182.00, stdev=17.20, samples=4
  write: IOPS=176, BW=706KiB/s (723kB/s)(1508KiB/2137msec)
    clat (usec): min=83, max=57609, avg=4077.28, stdev=6963.85
     lat (usec): min=83, max=57609, avg=4078.05, stdev=6963.84
    clat percentiles (usec):
     |  1.00th=[   88],  5.00th=[  105], 10.00th=[  120], 20.00th=[  153],
     | 30.00th=[  178], 40.00th=[  237], 50.00th=[  963], 60.00th=[ 1598],
     | 70.00th=[ 2606], 80.00th=[ 7832], 90.00th=[13435], 95.00th=[18744],
     | 99.00th=[28967], 99.50th=[39060], 99.90th=[57410], 99.95th=[57410],
     | 99.99th=[57410]
   bw (  KiB/s): min=  568, max=  856, per=23.41%, avg=690.00, stdev=125.54, samples=4
   iops        : min=  142, max=  214, avg=172.50, stdev=31.38, samples=4
  lat (usec)   : 4=0.52%, 10=12.76%, 20=10.94%, 50=4.17%, 100=1.17%
  lat (usec)   : 250=19.79%, 500=1.69%, 750=1.17%, 1000=2.99%
  lat (msec)   : 2=13.54%, 4=15.62%, 10=5.86%, 20=7.03%, 50=2.60%
  lat (msec)   : 100=0.13%
  cpu          : usr=0.28%, sys=8.52%, ctx=843, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19522: Fri Jun 14 18:06:00 2024
   read: IOPS=185, BW=743KiB/s (761kB/s)(1508KiB/2030msec)
    clat (usec): min=3, max=20638, avg=1270.10, stdev=2214.59
     lat (usec): min=3, max=20639, avg=1271.04, stdev=2214.40
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    7], 10.00th=[    8], 20.00th=[   10],
     | 30.00th=[   12], 40.00th=[   16], 50.00th=[   30], 60.00th=[ 1139],
     | 70.00th=[ 1778], 80.00th=[ 2311], 90.00th=[ 3425], 95.00th=[ 4113],
     | 99.00th=[12256], 99.50th=[19268], 99.90th=[20579], 99.95th=[20579],
     | 99.99th=[20579]
   bw (  KiB/s): min=  680, max=  784, per=25.96%, avg=727.50, stdev=42.97, samples=4
   iops        : min=  170, max=  196, avg=181.75, stdev=10.78, samples=4
  write: IOPS=192, BW=770KiB/s (789kB/s)(1564KiB/2030msec)
    clat (usec): min=85, max=44964, avg=3944.46, stdev=6781.75
     lat (usec): min=86, max=44965, avg=3945.19, stdev=6781.76
    clat percentiles (usec):
     |  1.00th=[   98],  5.00th=[  113], 10.00th=[  126], 20.00th=[  165],
     | 30.00th=[  186], 40.00th=[  241], 50.00th=[  898], 60.00th=[ 1565],
     | 70.00th=[ 2573], 80.00th=[ 6652], 90.00th=[12911], 95.00th=[18220],
     | 99.00th=[30540], 99.50th=[36439], 99.90th=[44827], 99.95th=[44827],
     | 99.99th=[44827]
   bw (  KiB/s): min=  640, max=  902, per=25.83%, avg=761.50, stdev=107.72, samples=4
   iops        : min=  160, max=  225, avg=190.25, stdev=26.71, samples=4
  lat (usec)   : 4=0.39%, 10=10.81%, 20=10.29%, 50=4.43%, 100=0.91%
  lat (usec)   : 250=21.35%, 500=2.86%, 750=0.52%, 1000=3.52%
  lat (msec)   : 2=14.19%, 4=15.23%, 10=7.55%, 20=5.47%, 50=2.47%
  cpu          : usr=0.25%, sys=8.67%, ctx=854, majf=0, minf=33
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2802KiB/s (2869kB/s), 678KiB/s-743KiB/s (694kB/s-761kB/s), io=5988KiB (6132kB), run=2030-2137msec
  WRITE: bw=2948KiB/s (3019kB/s), 706KiB/s-772KiB/s (723kB/s-791kB/s), io=6300KiB (6451kB), run=2030-2137msec

rand-rw: (groupid=0, jobs=1): err= 0: pid=19524: Fri Jun 14 18:06:09 2024
   read: IOPS=34, BW=140KiB/s (143kB/s)(1480KiB/10580msec)
    clat (usec): min=671, max=32548, avg=2492.70, stdev=3664.99
     lat (usec): min=672, max=32549, avg=2493.81, stdev=3665.08
    clat percentiles (usec):
     |  1.00th=[  758],  5.00th=[  824], 10.00th=[  914], 20.00th=[ 1012],
     | 30.00th=[ 1139], 40.00th=[ 1270], 50.00th=[ 1500], 60.00th=[ 1827],
     | 70.00th=[ 2212], 80.00th=[ 2671], 90.00th=[ 3326], 95.00th=[ 7111],
     | 99.00th=[20579], 99.50th=[32375], 99.90th=[32637], 99.95th=[32637],
     | 99.99th=[32637]
   bw (  KiB/s): min=   80, max=  272, per=24.74%, avg=139.55, stdev=50.79, samples=20
   iops        : min=   20, max=   68, avg=34.85, stdev=12.67, samples=20
  write: IOPS=37, BW=150KiB/s (154kB/s)(1592KiB/10580msec)
    clat (usec): min=12471, max=59648, avg=24239.29, stdev=7290.86
     lat (usec): min=12472, max=59650, avg=24240.67, stdev=7290.89
    clat percentiles (usec):
     |  1.00th=[13042],  5.00th=[15008], 10.00th=[16057], 20.00th=[17695],
     | 30.00th=[19792], 40.00th=[21627], 50.00th=[23462], 60.00th=[25035],
     | 70.00th=[26608], 80.00th=[29230], 90.00th=[32637], 95.00th=[38536],
     | 99.00th=[51643], 99.50th=[53740], 99.90th=[59507], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min=  104, max=  176, per=25.22%, avg=149.55, stdev=21.28, samples=20
   iops        : min=   26, max=   44, avg=37.35, stdev= 5.33, samples=20
  lat (usec)   : 750=0.39%, 1000=8.72%
  lat (msec)   : 2=22.14%, 4=13.02%, 10=1.95%, 20=17.06%, 50=36.20%
  lat (msec)   : 100=0.52%
  cpu          : usr=0.13%, sys=1.42%, ctx=896, majf=0, minf=33
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19525: Fri Jun 14 18:06:09 2024
   read: IOPS=33, BW=135KiB/s (139kB/s)(1436KiB/10610msec)
    clat (usec): min=687, max=30903, avg=2537.93, stdev=3344.16
     lat (usec): min=687, max=30903, avg=2539.08, stdev=3344.13
    clat percentiles (usec):
     |  1.00th=[  824],  5.00th=[  873], 10.00th=[  938], 20.00th=[ 1037],
     | 30.00th=[ 1156], 40.00th=[ 1336], 50.00th=[ 1614], 60.00th=[ 1991],
     | 70.00th=[ 2278], 80.00th=[ 2671], 90.00th=[ 3425], 95.00th=[ 9241],
     | 99.00th=[18220], 99.50th=[20841], 99.90th=[30802], 99.95th=[30802],
     | 99.99th=[30802]
   bw (  KiB/s): min=   72, max=  239, per=24.16%, avg=136.25, stdev=45.50, samples=20
   iops        : min=   18, max=   59, avg=33.95, stdev=11.31, samples=20
  write: IOPS=38, BW=154KiB/s (158kB/s)(1636KiB/10610msec)
    clat (usec): min=11176, max=70918, avg=23685.65, stdev=7124.30
     lat (usec): min=11177, max=70919, avg=23686.94, stdev=7124.33
    clat percentiles (usec):
     |  1.00th=[13173],  5.00th=[14877], 10.00th=[15664], 20.00th=[17171],
     | 30.00th=[19268], 40.00th=[21365], 50.00th=[22938], 60.00th=[24511],
     | 70.00th=[26608], 80.00th=[28443], 90.00th=[32113], 95.00th=[35914],
     | 99.00th=[44827], 99.50th=[51119], 99.90th=[70779], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min=  112, max=  176, per=25.74%, avg=152.65, stdev=17.55, samples=20
   iops        : min=   28, max=   44, avg=38.05, stdev= 4.37, samples=20
  lat (usec)   : 750=0.26%, 1000=7.42%
  lat (msec)   : 2=20.57%, 4=14.71%, 10=1.56%, 20=18.62%, 50=36.46%
  lat (msec)   : 100=0.39%
  cpu          : usr=0.22%, sys=1.35%, ctx=925, majf=0, minf=31
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19526: Fri Jun 14 18:06:09 2024
   read: IOPS=38, BW=152KiB/s (156kB/s)(1564KiB/10282msec)
    clat (usec): min=694, max=31546, avg=2606.88, stdev=3873.93
     lat (usec): min=695, max=31546, avg=2608.06, stdev=3873.92
    clat percentiles (usec):
     |  1.00th=[  775],  5.00th=[  873], 10.00th=[  955], 20.00th=[ 1045],
     | 30.00th=[ 1156], 40.00th=[ 1319], 50.00th=[ 1614], 60.00th=[ 1893],
     | 70.00th=[ 2245], 80.00th=[ 2671], 90.00th=[ 3392], 95.00th=[ 9503],
     | 99.00th=[26084], 99.50th=[28967], 99.90th=[31589], 99.95th=[31589],
     | 99.99th=[31589]
   bw (  KiB/s): min=   72, max=  280, per=27.50%, avg=155.10, stdev=59.76, samples=20
   iops        : min=   18, max=   70, avg=38.70, stdev=14.97, samples=20
  write: IOPS=36, BW=147KiB/s (150kB/s)(1508KiB/10282msec)
    clat (usec): min=11961, max=51574, avg=24542.00, stdev=7144.25
     lat (usec): min=11961, max=51575, avg=24543.58, stdev=7144.16
    clat percentiles (usec):
     |  1.00th=[13173],  5.00th=[14353], 10.00th=[15664], 20.00th=[18220],
     | 30.00th=[20841], 40.00th=[22414], 50.00th=[23987], 60.00th=[25822],
     | 70.00th=[26870], 80.00th=[28967], 90.00th=[33424], 95.00th=[38536],
     | 99.00th=[47449], 99.50th=[49546], 99.90th=[51643], 99.95th=[51643],
     | 99.99th=[51643]
   bw (  KiB/s): min=  111, max=  176, per=24.74%, avg=146.70, stdev=17.19, samples=20
   iops        : min=   27, max=   44, avg=36.60, stdev= 4.39, samples=20
  lat (usec)   : 750=0.13%, 1000=7.16%
  lat (msec)   : 2=24.74%, 4=14.71%, 10=1.82%, 20=14.71%, 50=36.59%
  lat (msec)   : 100=0.13%
  cpu          : usr=0.15%, sys=1.48%, ctx=935, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19527: Fri Jun 14 18:06:09 2024
   read: IOPS=36, BW=147KiB/s (150kB/s)(1508KiB/10273msec)
    clat (usec): min=721, max=33126, avg=2622.95, stdev=3760.81
     lat (usec): min=721, max=33126, avg=2624.04, stdev=3760.82
    clat percentiles (usec):
     |  1.00th=[  758],  5.00th=[  873], 10.00th=[  938], 20.00th=[ 1074],
     | 30.00th=[ 1172], 40.00th=[ 1352], 50.00th=[ 1582], 60.00th=[ 2024],
     | 70.00th=[ 2278], 80.00th=[ 2704], 90.00th=[ 3490], 95.00th=[ 9503],
     | 99.00th=[23462], 99.50th=[29492], 99.90th=[33162], 99.95th=[33162],
     | 99.99th=[33162]
   bw (  KiB/s): min=   64, max=  216, per=25.59%, avg=144.35, stdev=41.21, samples=20
   iops        : min=   16, max=   54, avg=36.05, stdev=10.28, samples=20
  write: IOPS=38, BW=152KiB/s (156kB/s)(1564KiB/10273msec)
    clat (usec): min=12391, max=72902, avg=23718.73, stdev=7301.19
     lat (usec): min=12393, max=72903, avg=23720.13, stdev=7301.22
    clat percentiles (usec):
     |  1.00th=[13173],  5.00th=[14877], 10.00th=[15664], 20.00th=[17433],
     | 30.00th=[19530], 40.00th=[21365], 50.00th=[23200], 60.00th=[24511],
     | 70.00th=[26346], 80.00th=[28181], 90.00th=[31589], 95.00th=[35914],
     | 99.00th=[51643], 99.50th=[57410], 99.90th=[72877], 99.95th=[72877],
     | 99.99th=[72877]
   bw (  KiB/s): min=  120, max=  176, per=25.56%, avg=151.55, stdev=15.07, samples=20
   iops        : min=   30, max=   44, avg=37.85, stdev= 3.82, samples=20
  lat (usec)   : 750=0.39%, 1000=6.77%
  lat (msec)   : 2=21.87%, 4=16.02%, 10=1.82%, 20=18.36%, 50=34.24%
  lat (msec)   : 100=0.52%
  cpu          : usr=0.18%, sys=1.41%, ctx=912, majf=0, minf=33
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=564KiB/s (578kB/s), 135KiB/s-152KiB/s (139kB/s-156kB/s), io=5988KiB (6132kB), run=10273-10610msec
  WRITE: bw=594KiB/s (608kB/s), 147KiB/s-154KiB/s (150kB/s-158kB/s), io=6300KiB (6451kB), run=10273-10610msec