== sanity test 398b: DIO and buffer IO race ============== 18:11:28 (1718403088)
/usr/bin/fio
12+0 records in
12+0 records out
12582912 bytes (13 MB) copied, 0.0827163 s, 152 MB/s
mix direct rw 12M to OST0 by fio with 4 jobs...
mix buffer rw 12M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=14254: Fri Jun 14 18:11:31 2024
   read: IOPS=321, BW=1285KiB/s (1316kB/s)(1480KiB/1152msec)
    clat (usec): min=3, max=26967, avg=1753.17, stdev=2887.37
     lat (usec): min=3, max=26968, avg=1753.71, stdev=2887.39
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    7], 10.00th=[    8], 20.00th=[   10],
     | 30.00th=[   12], 40.00th=[  123], 50.00th=[ 1254], 60.00th=[ 1762],
     | 70.00th=[ 2278], 80.00th=[ 2671], 90.00th=[ 3458], 95.00th=[ 5211],
     | 99.00th=[17695], 99.50th=[25035], 99.90th=[26870], 99.95th=[26870],
     | 99.99th=[26870]
   bw (  KiB/s): min=  688, max= 1564, per=21.93%, avg=1126.00, stdev=619.43, samples=2
   iops        : min=  172, max=  391, avg=281.50, stdev=154.86, samples=2
  write: IOPS=345, BW=1382KiB/s (1415kB/s)(1592KiB/1152msec)
    clat (usec): min=86, max=22596, avg=1240.84, stdev=2075.94
     lat (usec): min=87, max=22597, avg=1243.34, stdev=2075.64
    clat percentiles (usec):
     |  1.00th=[   93],  5.00th=[  103], 10.00th=[  112], 20.00th=[  133],
     | 30.00th=[  172], 40.00th=[  217], 50.00th=[  424], 60.00th=[  898],
     | 70.00th=[ 1188], 80.00th=[ 1844], 90.00th=[ 3458], 95.00th=[ 4555],
     | 99.00th=[ 8979], 99.50th=[18744], 99.90th=[22676], 99.95th=[22676],
     | 99.99th=[22676]
   bw (  KiB/s): min=  720, max= 1684, per=22.25%, avg=1202.00, stdev=681.65, samples=2
   iops        : min=  180, max=  421, avg=300.50, stdev=170.41, samples=2
  lat (usec)   : 4=0.39%, 10=11.33%, 20=6.64%, 50=0.39%, 100=1.82%
  lat (usec)   : 250=21.87%, 500=3.26%, 750=1.43%, 1000=8.72%
  lat (msec)   : 2=18.62%, 4=17.84%, 10=6.51%, 20=0.65%, 50=0.52%
  cpu          : usr=0.26%, sys=17.20%, ctx=1002, majf=0, minf=33
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14255: Fri Jun 14 18:11:31 2024
   read: IOPS=311, BW=1245KiB/s (1275kB/s)(1436KiB/1153msec)
    clat (usec): min=3, max=34387, avg=1800.93, stdev=3247.08
     lat (usec): min=3, max=34388, avg=1801.49, stdev=3247.09
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    7], 10.00th=[    8], 20.00th=[   10],
     | 30.00th=[   12], 40.00th=[   18], 50.00th=[ 1221], 60.00th=[ 1762],
     | 70.00th=[ 2147], 80.00th=[ 2704], 90.00th=[ 3654], 95.00th=[ 5342],
     | 99.00th=[14746], 99.50th=[25297], 99.90th=[34341], 99.95th=[34341],
     | 99.99th=[34341]
   bw (  KiB/s): min=  808, max= 1416, per=21.66%, avg=1112.00, stdev=429.92, samples=2
   iops        : min=  202, max=  354, avg=278.00, stdev=107.48, samples=2
  write: IOPS=354, BW=1419KiB/s (1453kB/s)(1636KiB/1153msec)
    clat (usec): min=90, max=26567, avg=1211.10, stdev=2061.03
     lat (usec): min=91, max=26567, avg=1212.07, stdev=2060.85
    clat percentiles (usec):
     |  1.00th=[   94],  5.00th=[  102], 10.00th=[  112], 20.00th=[  133],
     | 30.00th=[  163], 40.00th=[  206], 50.00th=[  445], 60.00th=[  914],
     | 70.00th=[ 1237], 80.00th=[ 1942], 90.00th=[ 3097], 95.00th=[ 4146],
     | 99.00th=[ 6325], 99.50th=[ 8848], 99.90th=[26608], 99.95th=[26608],
     | 99.99th=[26608]
   bw (  KiB/s): min=  920, max= 1488, per=22.28%, avg=1204.00, stdev=401.64, samples=2
   iops        : min=  230, max=  372, avg=301.00, stdev=100.41, samples=2
  lat (usec)   : 4=0.13%, 10=9.90%, 20=8.72%, 100=2.34%, 250=21.35%
  lat (usec)   : 500=4.17%, 750=1.69%, 1000=5.99%
  lat (msec)   : 2=19.40%, 4=19.40%, 10=5.60%, 20=0.78%, 50=0.52%
  cpu          : usr=0.43%, sys=16.23%, ctx=930, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14256: Fri Jun 14 18:11:31 2024
   read: IOPS=335, BW=1341KiB/s (1374kB/s)(1564KiB/1166msec)
    clat (usec): min=3, max=26961, avg=1734.06, stdev=2791.76
     lat (usec): min=3, max=26962, avg=1734.75, stdev=2791.80
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    6], 10.00th=[    7], 20.00th=[   10],
     | 30.00th=[   12], 40.00th=[   31], 50.00th=[ 1369], 60.00th=[ 1778],
     | 70.00th=[ 2278], 80.00th=[ 2769], 90.00th=[ 3490], 95.00th=[ 4686],
     | 99.00th=[16319], 99.50th=[25560], 99.90th=[26870], 99.95th=[26870],
     | 99.99th=[26870]
   bw (  KiB/s): min=  784, max= 1376, per=21.03%, avg=1080.00, stdev=418.61, samples=2
   iops        : min=  196, max=  344, avg=270.00, stdev=104.65, samples=2
  write: IOPS=323, BW=1293KiB/s (1324kB/s)(1508KiB/1166msec)
    clat (usec): min=91, max=26744, avg=1260.57, stdev=2548.30
     lat (usec): min=91, max=26745, avg=1263.27, stdev=2547.72
    clat percentiles (usec):
     |  1.00th=[   95],  5.00th=[  105], 10.00th=[  113], 20.00th=[  131],
     | 30.00th=[  176], 40.00th=[  221], 50.00th=[  660], 60.00th=[  898],
     | 70.00th=[ 1123], 80.00th=[ 1827], 90.00th=[ 2900], 95.00th=[ 4015],
     | 99.00th=[15270], 99.50th=[23200], 99.90th=[26870], 99.95th=[26870],
     | 99.99th=[26870]
   bw (  KiB/s): min=  888, max= 1232, per=19.62%, avg=1060.00, stdev=243.24, samples=2
   iops        : min=  222, max=  308, avg=265.00, stdev=60.81, samples=2
  lat (usec)   : 4=0.91%, 10=11.20%, 20=8.07%, 50=0.26%, 100=1.43%
  lat (usec)   : 250=19.92%, 500=2.86%, 750=2.08%, 1000=8.33%
  lat (msec)   : 2=17.97%, 4=21.09%, 10=4.56%, 20=0.65%, 50=0.65%
  cpu          : usr=0.43%, sys=18.71%, ctx=957, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14257: Fri Jun 14 18:11:31 2024
   read: IOPS=327, BW=1309KiB/s (1340kB/s)(1508KiB/1152msec)
    clat (usec): min=3, max=33148, avg=1766.28, stdev=3143.71
     lat (usec): min=3, max=33197, avg=1766.96, stdev=3144.99
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    7], 10.00th=[    9], 20.00th=[   11],
     | 30.00th=[   13], 40.00th=[  159], 50.00th=[ 1467], 60.00th=[ 1876],
     | 70.00th=[ 2212], 80.00th=[ 2606], 90.00th=[ 3392], 95.00th=[ 5014],
     | 99.00th=[18482], 99.50th=[30540], 99.90th=[33162], 99.95th=[33162],
     | 99.99th=[33162]
   bw (  KiB/s): min=  760, max= 1461, per=21.63%, avg=1110.50, stdev=495.68, samples=2
   iops        : min=  190, max=  365, avg=277.50, stdev=123.74, samples=2
  write: IOPS=339, BW=1358KiB/s (1390kB/s)(1564KiB/1152msec)
    clat (usec): min=78, max=25716, avg=1217.94, stdev=2023.20
     lat (usec): min=78, max=25716, avg=1218.75, stdev=2023.22
    clat percentiles (usec):
     |  1.00th=[   92],  5.00th=[  105], 10.00th=[  118], 20.00th=[  139],
     | 30.00th=[  169], 40.00th=[  206], 50.00th=[  537], 60.00th=[  865],
     | 70.00th=[ 1172], 80.00th=[ 2008], 90.00th=[ 3163], 95.00th=[ 4359],
     | 99.00th=[10028], 99.50th=[11338], 99.90th=[25822], 99.95th=[25822],
     | 99.99th=[25822]
   bw (  KiB/s): min=  808, max= 1588, per=22.17%, avg=1198.00, stdev=551.54, samples=2
   iops        : min=  202, max=  397, avg=299.50, stdev=137.89, samples=2
  lat (usec)   : 4=0.13%, 10=9.11%, 20=8.59%, 50=0.52%, 100=2.34%
  lat (usec)   : 250=21.48%, 500=2.99%, 750=1.82%, 1000=7.03%
  lat (msec)   : 2=18.23%, 4=21.61%, 10=4.82%, 20=0.78%, 50=0.52%
  cpu          : usr=0.26%, sys=17.38%, ctx=954, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=5136KiB/s (5259kB/s), 1245KiB/s-1341KiB/s (1275kB/s-1374kB/s), io=5988KiB (6132kB), run=1152-1166msec
  WRITE: bw=5403KiB/s (5533kB/s), 1293KiB/s-1419KiB/s (1324kB/s-1453kB/s), io=6300KiB (6451kB), run=1152-1166msec

rand-rw: (groupid=0, jobs=1): err= 0: pid=14258: Fri Jun 14 18:11:32 2024
   read: IOPS=206, BW=825KiB/s (844kB/s)(1480KiB/1795msec)
    clat (usec): min=577, max=36688, avg=1710.66, stdev=2735.09
     lat (usec): min=578, max=36688, avg=1711.28, stdev=2735.09
    clat percentiles (usec):
     |  1.00th=[  603],  5.00th=[  668], 10.00th=[  725], 20.00th=[  783],
     | 30.00th=[  832], 40.00th=[  889], 50.00th=[  938], 60.00th=[ 1057],
     | 70.00th=[ 1385], 80.00th=[ 1827], 90.00th=[ 3130], 95.00th=[ 4883],
     | 99.00th=[11338], 99.50th=[25297], 99.90th=[36439], 99.95th=[36439],
     | 99.99th=[36439]
   bw (  KiB/s): min=  480, max= 1040, per=22.64%, avg=741.33, stdev=281.86, samples=3
   iops        : min=  120, max=  260, avg=185.33, stdev=70.47, samples=3
  write: IOPS=221, BW=887KiB/s (908kB/s)(1592KiB/1795msec)
    clat (usec): min=1133, max=25277, avg=2901.57, stdev=2311.97
     lat (usec): min=1134, max=25277, avg=2902.30, stdev=2311.97
    clat percentiles (usec):
     |  1.00th=[ 1221],  5.00th=[ 1450], 10.00th=[ 1647], 20.00th=[ 1876],
     | 30.00th=[ 1975], 40.00th=[ 2114], 50.00th=[ 2245], 60.00th=[ 2409],
     | 70.00th=[ 2966], 80.00th=[ 3425], 90.00th=[ 4621], 95.00th=[ 5735],
     | 99.00th=[16188], 99.50th=[24773], 99.90th=[25297], 99.95th=[25297],
     | 99.99th=[25297]
   bw (  KiB/s): min=  504, max= 1224, per=22.83%, avg=786.67, stdev=384.11, samples=3
   iops        : min=  126, max=  306, avg=196.67, stdev=96.03, samples=3
  lat (usec)   : 750=5.99%, 1000=20.57%
  lat (msec)   : 2=29.30%, 4=33.33%, 10=9.64%, 20=0.65%, 50=0.52%
  cpu          : usr=0.39%, sys=5.52%, ctx=809, majf=0, minf=32
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14259: Fri Jun 14 18:11:32 2024
   read: IOPS=196, BW=786KiB/s (804kB/s)(1436KiB/1828msec)
    clat (usec): min=569, max=34040, avg=1665.28, stdev=2530.00
     lat (usec): min=569, max=34041, avg=1666.01, stdev=2529.98
    clat percentiles (usec):
     |  1.00th=[  627],  5.00th=[  676], 10.00th=[  717], 20.00th=[  766],
     | 30.00th=[  807], 40.00th=[  857], 50.00th=[  914], 60.00th=[  996],
     | 70.00th=[ 1237], 80.00th=[ 1827], 90.00th=[ 3523], 95.00th=[ 4621],
     | 99.00th=[15401], 99.50th=[17433], 99.90th=[33817], 99.95th=[33817],
     | 99.99th=[33817]
   bw (  KiB/s): min=  448, max= 1128, per=21.66%, avg=709.33, stdev=366.29, samples=3
   iops        : min=  112, max=  282, avg=177.33, stdev=91.57, samples=3
  write: IOPS=223, BW=895KiB/s (916kB/s)(1636KiB/1828msec)
    clat (usec): min=1238, max=25226, avg=2991.33, stdev=2206.22
     lat (usec): min=1239, max=25227, avg=2992.57, stdev=2205.99
    clat percentiles (usec):
     |  1.00th=[ 1303],  5.00th=[ 1450], 10.00th=[ 1663], 20.00th=[ 1876],
     | 30.00th=[ 1975], 40.00th=[ 2057], 50.00th=[ 2212], 60.00th=[ 2442],
     | 70.00th=[ 2933], 80.00th=[ 3589], 90.00th=[ 5342], 95.00th=[ 7046],
     | 99.00th=[11863], 99.50th=[13829], 99.90th=[25297], 99.95th=[25297],
     | 99.99th=[25297]
   bw (  KiB/s): min=  528, max= 1096, per=22.21%, avg=765.33, stdev=295.28, samples=3
   iops        : min=  132, max=  274, avg=191.33, stdev=73.82, samples=3
  lat (usec)   : 750=7.68%, 1000=20.44%
  lat (msec)   : 2=27.34%, 4=32.16%, 10=10.94%, 20=1.17%, 50=0.26%
  cpu          : usr=0.49%, sys=5.53%, ctx=827, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14260: Fri Jun 14 18:11:32 2024
   read: IOPS=215, BW=860KiB/s (881kB/s)(1564KiB/1818msec)
    clat (usec): min=566, max=21238, avg=1562.16, stdev=1734.54
     lat (usec): min=566, max=21239, avg=1562.76, stdev=1734.56
    clat percentiles (usec):
     |  1.00th=[  603],  5.00th=[  693], 10.00th=[  725], 20.00th=[  791],
     | 30.00th=[  832], 40.00th=[  873], 50.00th=[  955], 60.00th=[ 1045],
     | 70.00th=[ 1270], 80.00th=[ 1811], 90.00th=[ 3195], 95.00th=[ 4817],
     | 99.00th=[ 9372], 99.50th=[11863], 99.90th=[21365], 99.95th=[21365],
     | 99.99th=[21365]
   bw (  KiB/s): min=  376, max= 1144, per=23.12%, avg=757.33, stdev=384.03, samples=3
   iops        : min=   94, max=  286, avg=189.33, stdev=96.01, samples=3
  write: IOPS=207, BW=829KiB/s (849kB/s)(1508KiB/1818msec)
    clat (usec): min=1238, max=36281, avg=3182.98, stdev=3225.11
     lat (usec): min=1239, max=36282, avg=3184.01, stdev=3225.16
    clat percentiles (usec):
     |  1.00th=[ 1303],  5.00th=[ 1516], 10.00th=[ 1745], 20.00th=[ 1876],
     | 30.00th=[ 1975], 40.00th=[ 2089], 50.00th=[ 2212], 60.00th=[ 2442],
     | 70.00th=[ 2999], 80.00th=[ 3687], 90.00th=[ 5211], 95.00th=[ 6849],
     | 99.00th=[18220], 99.50th=[34866], 99.90th=[36439], 99.95th=[36439],
     | 99.99th=[36439]
   bw (  KiB/s): min=  480, max= 1104, per=20.97%, avg=722.67, stdev=334.31, samples=3
   iops        : min=  120, max=  276, avg=180.67, stdev=83.58, samples=3
  lat (usec)   : 750=7.16%, 1000=21.35%
  lat (msec)   : 2=28.78%, 4=30.99%, 10=10.29%, 20=0.91%, 50=0.52%
  cpu          : usr=0.17%, sys=5.39%, ctx=828, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14261: Fri Jun 14 18:11:32 2024
   read: IOPS=209, BW=836KiB/s (856kB/s)(1508KiB/1803msec)
    clat (usec): min=551, max=16861, avg=1660.31, stdev=1842.72
     lat (usec): min=552, max=16861, avg=1660.91, stdev=1842.72
    clat percentiles (usec):
     |  1.00th=[  603],  5.00th=[  701], 10.00th=[  742], 20.00th=[  799],
     | 30.00th=[  840], 40.00th=[  889], 50.00th=[  971], 60.00th=[ 1106],
     | 70.00th=[ 1369], 80.00th=[ 2089], 90.00th=[ 3556], 95.00th=[ 4883],
     | 99.00th=[10552], 99.50th=[16581], 99.90th=[16909], 99.95th=[16909],
     | 99.99th=[16909]
   bw (  KiB/s): min=  456, max=  992, per=21.90%, avg=717.33, stdev=268.25, samples=3
   iops        : min=  114, max=  248, avg=179.33, stdev=67.06, samples=3
  write: IOPS=216, BW=867KiB/s (888kB/s)(1564KiB/1803msec)
    clat (usec): min=1141, max=33910, avg=2992.76, stdev=2559.04
     lat (usec): min=1142, max=33911, avg=2993.48, stdev=2559.06
    clat percentiles (usec):
     |  1.00th=[ 1237],  5.00th=[ 1434], 10.00th=[ 1680], 20.00th=[ 1876],
     | 30.00th=[ 1991], 40.00th=[ 2057], 50.00th=[ 2212], 60.00th=[ 2409],
     | 70.00th=[ 2704], 80.00th=[ 3589], 90.00th=[ 5014], 95.00th=[ 6587],
     | 99.00th=[16057], 99.50th=[20317], 99.90th=[33817], 99.95th=[33817],
     | 99.99th=[33817]
   bw (  KiB/s): min=  496, max= 1176, per=22.60%, avg=778.67, stdev=354.21, samples=3
   iops        : min=  124, max=  294, avg=194.67, stdev=88.55, samples=3
  lat (usec)   : 750=6.12%, 1000=20.31%
  lat (msec)   : 2=28.26%, 4=32.81%, 10=11.07%, 20=1.17%, 50=0.26%
  cpu          : usr=0.28%, sys=5.55%, ctx=823, majf=0, minf=32
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3276KiB/s (3354kB/s), 786KiB/s-860KiB/s (804kB/s-881kB/s), io=5988KiB (6132kB), run=1795-1828msec
  WRITE: bw=3446KiB/s (3529kB/s), 829KiB/s-895KiB/s (849kB/s-916kB/s), io=6300KiB (6451kB), run=1795-1828msec