== sanity test 398b: DIO and buffer IO race ============== 00:48:07 (1722487687)
/usr/bin/fio
12+0 records in
12+0 records out
12582912 bytes (13 MB) copied, 0.102515 s, 123 MB/s
mix direct rw 12M to OST0 by fio with 4 jobs...
mix buffer rw 12M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=17808: Thu Aug  1 00:48:09 2024
   read: IOPS=396, BW=1586KiB/s (1624kB/s)(1480KiB/933msec)
    clat (usec): min=3, max=21813, avg=1440.07, stdev=2218.93
     lat (usec): min=3, max=21814, avg=1440.58, stdev=2219.00
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    6], 10.00th=[    7], 20.00th=[   10],
     | 30.00th=[   11], 40.00th=[   43], 50.00th=[ 1156], 60.00th=[ 1549],
     | 70.00th=[ 1909], 80.00th=[ 2212], 90.00th=[ 2868], 95.00th=[ 4490],
     | 99.00th=[10421], 99.50th=[20579], 99.90th=[21890], 99.95th=[21890],
     | 99.99th=[21890]
   bw (  KiB/s): min= 1240, max= 1240, per=21.87%, avg=1240.00, stdev= 0.00, samples=1
   iops        : min=  310, max=  310, avg=310.00, stdev= 0.00, samples=1
  write: IOPS=426, BW=1706KiB/s (1747kB/s)(1592KiB/933msec)
    clat (usec): min=90, max=7342, avg=959.18, stdev=1190.36
     lat (usec): min=90, max=7343, avg=959.76, stdev=1190.38
    clat percentiles (usec):
     |  1.00th=[   93],  5.00th=[  103], 10.00th=[  112], 20.00th=[  122],
     | 30.00th=[  157], 40.00th=[  210], 50.00th=[  586], 60.00th=[  832],
     | 70.00th=[ 1029], 80.00th=[ 1582], 90.00th=[ 2409], 95.00th=[ 3720],
     | 99.00th=[ 5932], 99.50th=[ 7308], 99.90th=[ 7373], 99.95th=[ 7373],
     | 99.99th=[ 7373]
   bw (  KiB/s): min= 1256, max= 1256, per=21.06%, avg=1256.00, stdev= 0.00, samples=1
   iops        : min=  314, max=  314, avg=314.00, stdev= 0.00, samples=1
  lat (usec)   : 4=0.65%, 10=12.37%, 20=5.99%, 50=0.26%, 100=2.47%
  lat (usec)   : 250=20.96%, 500=2.34%, 750=5.73%, 1000=7.29%
  lat (msec)   : 2=22.92%, 4=14.32%, 10=4.17%, 20=0.26%, 50=0.26%
  cpu          : usr=0.32%, sys=17.06%, ctx=921, majf=0, minf=35
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17809: Thu Aug  1 00:48:09 2024
   read: IOPS=339, BW=1360KiB/s (1392kB/s)(1436KiB/1056msec)
    clat (usec): min=3, max=24562, avg=1528.80, stdev=1912.29
     lat (usec): min=3, max=24563, avg=1530.07, stdev=1912.35
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    6], 10.00th=[    7], 20.00th=[   10],
     | 30.00th=[   12], 40.00th=[  898], 50.00th=[ 1401], 60.00th=[ 1811],
     | 70.00th=[ 2114], 80.00th=[ 2540], 90.00th=[ 3064], 95.00th=[ 4015],
     | 99.00th=[ 8225], 99.50th=[ 9241], 99.90th=[24511], 99.95th=[24511],
     | 99.99th=[24511]
   bw (  KiB/s): min=  944, max=  944, per=16.65%, avg=944.00, stdev= 0.00, samples=1
   iops        : min=  236, max=  236, avg=236.00, stdev= 0.00, samples=1
  write: IOPS=387, BW=1549KiB/s (1586kB/s)(1636KiB/1056msec)
    clat (usec): min=89, max=29565, avg=1206.01, stdev=1935.04
     lat (usec): min=90, max=29566, avg=1206.78, stdev=1935.06
    clat percentiles (usec):
     |  1.00th=[  100],  5.00th=[  108], 10.00th=[  116], 20.00th=[  141],
     | 30.00th=[  200], 40.00th=[  611], 50.00th=[  783], 60.00th=[  955],
     | 70.00th=[ 1401], 80.00th=[ 1811], 90.00th=[ 2638], 95.00th=[ 3785],
     | 99.00th=[ 7046], 99.50th=[ 7635], 99.90th=[29492], 99.95th=[29492],
     | 99.99th=[29492]
   bw (  KiB/s): min= 1152, max= 1152, per=19.31%, avg=1152.00, stdev= 0.00, samples=1
   iops        : min=  288, max=  288, avg=288.00, stdev= 0.00, samples=1
  lat (usec)   : 4=0.78%, 10=10.55%, 20=3.91%, 50=0.26%, 100=1.04%
  lat (usec)   : 250=17.06%, 500=2.73%, 750=6.51%, 1000=10.03%
  lat (msec)   : 2=22.92%, 4=19.53%, 10=4.30%, 20=0.13%, 50=0.26%
  cpu          : usr=0.76%, sys=16.49%, ctx=1062, majf=0, minf=32
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17810: Thu Aug  1 00:48:09 2024
   read: IOPS=414, BW=1657KiB/s (1697kB/s)(1564KiB/944msec)
    clat (usec): min=3, max=25013, avg=1451.35, stdev=2175.75
     lat (usec): min=3, max=25013, avg=1451.90, stdev=2175.78
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    6], 10.00th=[    8], 20.00th=[   10],
     | 30.00th=[   11], 40.00th=[  742], 50.00th=[ 1303], 60.00th=[ 1647],
     | 70.00th=[ 2024], 80.00th=[ 2343], 90.00th=[ 2835], 95.00th=[ 3490],
     | 99.00th=[ 9241], 99.50th=[22676], 99.90th=[25035], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 1264, max= 1264, per=22.29%, avg=1264.00, stdev= 0.00, samples=1
   iops        : min=  316, max=  316, avg=316.00, stdev= 0.00, samples=1
  write: IOPS=399, BW=1597KiB/s (1636kB/s)(1508KiB/944msec)
    clat (usec): min=86, max=25499, avg=961.57, stdev=1649.05
     lat (usec): min=86, max=25500, avg=962.21, stdev=1649.08
    clat percentiles (usec):
     |  1.00th=[   93],  5.00th=[  100], 10.00th=[  108], 20.00th=[  120],
     | 30.00th=[  147], 40.00th=[  194], 50.00th=[  424], 60.00th=[  832],
     | 70.00th=[ 1074], 80.00th=[ 1647], 90.00th=[ 2409], 95.00th=[ 3064],
     | 99.00th=[ 5080], 99.50th=[ 7177], 99.90th=[25560], 99.95th=[25560],
     | 99.99th=[25560]
   bw (  KiB/s): min= 1248, max= 1248, per=20.92%, avg=1248.00, stdev= 0.00, samples=1
   iops        : min=  312, max=  312, avg=312.00, stdev= 0.00, samples=1
  lat (usec)   : 4=0.39%, 10=12.63%, 20=5.60%, 50=0.26%, 100=2.60%
  lat (usec)   : 250=19.92%, 500=2.73%, 750=4.43%, 1000=5.60%
  lat (msec)   : 2=23.70%, 4=19.27%, 10=2.34%, 20=0.13%, 50=0.39%
  cpu          : usr=0.11%, sys=20.47%, ctx=988, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17811: Thu Aug  1 00:48:09 2024
   read: IOPS=414, BW=1657KiB/s (1697kB/s)(1508KiB/910msec)
    clat (usec): min=3, max=19895, avg=1400.29, stdev=1737.65
     lat (usec): min=3, max=19896, avg=1401.13, stdev=1737.43
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    7], 10.00th=[    8], 20.00th=[   10],
     | 30.00th=[   11], 40.00th=[  424], 50.00th=[ 1221], 60.00th=[ 1631],
     | 70.00th=[ 2057], 80.00th=[ 2442], 90.00th=[ 2999], 95.00th=[ 3916],
     | 99.00th=[ 6783], 99.50th=[ 7963], 99.90th=[19792], 99.95th=[19792],
     | 99.99th=[19792]
   bw (  KiB/s): min= 1376, max= 1376, per=24.27%, avg=1376.00, stdev= 0.00, samples=1
   iops        : min=  344, max=  344, avg=344.00, stdev= 0.00, samples=1
  write: IOPS=429, BW=1719KiB/s (1760kB/s)(1564KiB/910msec)
    clat (usec): min=87, max=18637, avg=943.50, stdev=1448.18
     lat (usec): min=88, max=18639, avg=944.11, stdev=1448.25
    clat percentiles (usec):
     |  1.00th=[   92],  5.00th=[  100], 10.00th=[  106], 20.00th=[  119],
     | 30.00th=[  137], 40.00th=[  192], 50.00th=[  392], 60.00th=[  799],
     | 70.00th=[  996], 80.00th=[ 1549], 90.00th=[ 2311], 95.00th=[ 3163],
     | 99.00th=[ 6718], 99.50th=[ 7242], 99.90th=[18744], 99.95th=[18744],
     | 99.99th=[18744]
   bw (  KiB/s): min= 1376, max= 1376, per=23.07%, avg=1376.00, stdev= 0.00, samples=1
   iops        : min=  344, max=  344, avg=344.00, stdev= 0.00, samples=1
  lat (usec)   : 4=0.26%, 10=12.89%, 20=5.08%, 50=0.78%, 100=2.99%
  lat (usec)   : 250=21.22%, 500=2.21%, 750=4.17%, 1000=9.24%
  lat (msec)   : 2=19.53%, 4=18.10%, 10=3.26%, 20=0.26%
  cpu          : usr=0.55%, sys=17.71%, ctx=865, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=5670KiB/s (5807kB/s), 1360KiB/s-1657KiB/s (1392kB/s-1697kB/s), io=5988KiB (6132kB), run=910-1056msec
  WRITE: bw=5966KiB/s (6109kB/s), 1549KiB/s-1719KiB/s (1586kB/s-1760kB/s), io=6300KiB (6451kB), run=910-1056msec

rand-rw: (groupid=0, jobs=1): err= 0: pid=17804: Thu Aug  1 00:48:10 2024
   read: IOPS=249, BW=998KiB/s (1022kB/s)(1480KiB/1483msec)
    clat (usec): min=508, max=17830, avg=1355.88, stdev=1549.33
     lat (usec): min=509, max=17832, avg=1356.45, stdev=1549.41
    clat percentiles (usec):
     |  1.00th=[  537],  5.00th=[  562], 10.00th=[  611], 20.00th=[  660],
     | 30.00th=[  717], 40.00th=[  758], 50.00th=[  840], 60.00th=[  947],
     | 70.00th=[ 1205], 80.00th=[ 1713], 90.00th=[ 2671], 95.00th=[ 3392],
     | 99.00th=[ 5538], 99.50th=[15270], 99.90th=[17957], 99.95th=[17957],
     | 99.99th=[17957]
   bw (  KiB/s): min=  688, max=  904, per=20.33%, avg=796.00, stdev=152.74, samples=2
   iops        : min=  172, max=  226, avg=199.00, stdev=38.18, samples=2
  write: IOPS=268, BW=1073KiB/s (1099kB/s)(1592KiB/1483msec)
    clat (usec): min=1068, max=25740, avg=2420.51, stdev=1757.15
     lat (usec): min=1069, max=25741, avg=2421.20, stdev=1757.17
    clat percentiles (usec):
     |  1.00th=[ 1123],  5.00th=[ 1221], 10.00th=[ 1401], 20.00th=[ 1565],
     | 30.00th=[ 1663], 40.00th=[ 1795], 50.00th=[ 1942], 60.00th=[ 2114],
     | 70.00th=[ 2442], 80.00th=[ 2900], 90.00th=[ 3916], 95.00th=[ 4686],
     | 99.00th=[ 9765], 99.50th=[11731], 99.90th=[25822], 99.95th=[25822],
     | 99.99th=[25822]
   bw (  KiB/s): min=  696, max= 1000, per=20.58%, avg=848.00, stdev=214.96, samples=2
   iops        : min=  174, max=  250, avg=212.00, stdev=53.74, samples=2
  lat (usec)   : 750=17.97%, 1000=12.50%
  lat (msec)   : 2=38.02%, 4=25.13%, 10=5.73%, 20=0.52%, 50=0.13%
  cpu          : usr=0.54%, sys=6.21%, ctx=828, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17805: Thu Aug  1 00:48:10 2024
   read: IOPS=234, BW=939KiB/s (962kB/s)(1436KiB/1529msec)
    clat (usec): min=505, max=19508, avg=1297.09, stdev=1342.79
     lat (usec): min=505, max=19509, avg=1297.66, stdev=1342.85
    clat percentiles (usec):
     |  1.00th=[  537],  5.00th=[  562], 10.00th=[  611], 20.00th=[  668],
     | 30.00th=[  725], 40.00th=[  783], 50.00th=[  848], 60.00th=[  930],
     | 70.00th=[ 1205], 80.00th=[ 1598], 90.00th=[ 2540], 95.00th=[ 3556],
     | 99.00th=[ 4817], 99.50th=[ 6194], 99.90th=[19530], 99.95th=[19530],
     | 99.99th=[19530]
   bw (  KiB/s): min=  632, max= 1424, per=23.97%, avg=938.67, stdev=425.16, samples=3
   iops        : min=  158, max=  356, avg=234.67, stdev=106.29, samples=3
  write: IOPS=267, BW=1070KiB/s (1096kB/s)(1636KiB/1529msec)
    clat (usec): min=1069, max=24850, avg=2560.60, stdev=1763.29
     lat (usec): min=1069, max=24851, avg=2561.26, stdev=1763.34
    clat percentiles (usec):
     |  1.00th=[ 1106],  5.00th=[ 1237], 10.00th=[ 1418], 20.00th=[ 1598],
     | 30.00th=[ 1696], 40.00th=[ 1795], 50.00th=[ 1942], 60.00th=[ 2245],
     | 70.00th=[ 2671], 80.00th=[ 3228], 90.00th=[ 4424], 95.00th=[ 5538],
     | 99.00th=[ 7963], 99.50th=[ 8455], 99.90th=[24773], 99.95th=[24773],
     | 99.99th=[24773]
   bw (  KiB/s): min=  680, max= 1528, per=25.76%, avg=1061.33, stdev=430.39, samples=3
   iops        : min=  170, max=  382, avg=265.33, stdev=107.60, samples=3
  lat (usec)   : 750=16.54%, 1000=13.15%
  lat (msec)   : 2=38.41%, 4=23.31%, 10=8.33%, 20=0.13%, 50=0.13%
  cpu          : usr=0.46%, sys=6.22%, ctx=825, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17806: Thu Aug  1 00:48:10 2024
   read: IOPS=262, BW=1052KiB/s (1077kB/s)(1564KiB/1487msec)
    clat (usec): min=488, max=12086, avg=1356.52, stdev=1297.13
     lat (usec): min=489, max=12086, avg=1357.08, stdev=1297.16
    clat percentiles (usec):
     |  1.00th=[  515],  5.00th=[  578], 10.00th=[  603], 20.00th=[  668],
     | 30.00th=[  709], 40.00th=[  783], 50.00th=[  857], 60.00th=[  979],
     | 70.00th=[ 1156], 80.00th=[ 1827], 90.00th=[ 2900], 95.00th=[ 3720],
     | 99.00th=[ 7439], 99.50th=[ 8586], 99.90th=[12125], 99.95th=[12125],
     | 99.99th=[12125]
   bw (  KiB/s): min=  616, max=  960, per=20.12%, avg=788.00, stdev=243.24, samples=2
   iops        : min=  154, max=  240, avg=197.00, stdev=60.81, samples=2
  write: IOPS=253, BW=1014KiB/s (1038kB/s)(1508KiB/1487msec)
    clat (usec): min=1038, max=32383, avg=2517.87, stdev=1995.11
     lat (usec): min=1038, max=32387, avg=2518.59, stdev=1995.25
    clat percentiles (usec):
     |  1.00th=[ 1090],  5.00th=[ 1270], 10.00th=[ 1467], 20.00th=[ 1631],
     | 30.00th=[ 1696], 40.00th=[ 1795], 50.00th=[ 1926], 60.00th=[ 2212],
     | 70.00th=[ 2638], 80.00th=[ 3195], 90.00th=[ 3916], 95.00th=[ 5211],
     | 99.00th=[ 8291], 99.50th=[ 9241], 99.90th=[32375], 99.95th=[32375],
     | 99.99th=[32375]
   bw (  KiB/s): min=  704, max=  928, per=19.81%, avg=816.00, stdev=158.39, samples=2
   iops        : min=  176, max=  232, avg=204.00, stdev=39.60, samples=2
  lat (usec)   : 500=0.26%, 750=17.71%, 1000=13.93%
  lat (msec)   : 2=36.20%, 4=25.26%, 10=6.38%, 20=0.13%, 50=0.13%
  cpu          : usr=0.47%, sys=5.72%, ctx=828, majf=0, minf=32
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17807: Thu Aug  1 00:48:10 2024
   read: IOPS=250, BW=1001KiB/s (1025kB/s)(1508KiB/1506msec)
    clat (usec): min=489, max=14741, avg=1351.78, stdev=1349.13
     lat (usec): min=489, max=14741, avg=1352.57, stdev=1349.07
    clat percentiles (usec):
     |  1.00th=[  523],  5.00th=[  570], 10.00th=[  611], 20.00th=[  660],
     | 30.00th=[  725], 40.00th=[  791], 50.00th=[  840], 60.00th=[  971],
     | 70.00th=[ 1270], 80.00th=[ 1713], 90.00th=[ 2671], 95.00th=[ 3720],
     | 99.00th=[ 7439], 99.50th=[ 9634], 99.90th=[14746], 99.95th=[14746],
     | 99.99th=[14746]
   bw (  KiB/s): min=  672, max= 1408, per=25.40%, avg=994.67, stdev=376.28, samples=3
   iops        : min=  168, max=  352, avg=248.67, stdev=94.07, samples=3
  write: IOPS=259, BW=1039KiB/s (1063kB/s)(1564KiB/1506msec)
    clat (usec): min=965, max=28811, avg=2503.18, stdev=1881.76
     lat (usec): min=965, max=28813, avg=2503.84, stdev=1881.83
    clat percentiles (usec):
     |  1.00th=[ 1106],  5.00th=[ 1205], 10.00th=[ 1401], 20.00th=[ 1582],
     | 30.00th=[ 1696], 40.00th=[ 1795], 50.00th=[ 1975], 60.00th=[ 2180],
     | 70.00th=[ 2638], 80.00th=[ 3097], 90.00th=[ 4146], 95.00th=[ 4948],
     | 99.00th=[ 8848], 99.50th=[11731], 99.90th=[28705], 99.95th=[28705],
     | 99.99th=[28705]
   bw (  KiB/s): min=  680, max= 1552, per=25.24%, avg=1040.00, stdev=455.44, samples=3
   iops        : min=  170, max=  388, avg=260.00, stdev=113.86, samples=3
  lat (usec)   : 500=0.13%, 750=16.67%, 1000=14.06%
  lat (msec)   : 2=36.59%, 4=25.13%, 10=7.03%, 20=0.26%, 50=0.13%
  cpu          : usr=0.33%, sys=5.85%, ctx=851, majf=0, minf=32
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3916KiB/s (4010kB/s), 939KiB/s-1052KiB/s (962kB/s-1077kB/s), io=5988KiB (6132kB), run=1483-1529msec
  WRITE: bw=4120KiB/s (4219kB/s), 1014KiB/s-1073KiB/s (1038kB/s-1099kB/s), io=6300KiB (6451kB), run=1483-1529msec