== sanity test 398b: DIO and buffer IO race ============== 18:13:42 (1718403222)
/usr/bin/fio
12+0 records in
12+0 records out
12582912 bytes (13 MB) copied, 0.107023 s, 118 MB/s
mix direct rw 12M to OST0 by fio with 4 jobs...
mix buffer rw 12M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=14005: Fri Jun 14 18:13:44 2024
   read: IOPS=325, BW=1302KiB/s (1333kB/s)(1480KiB/1137msec)
    clat (usec): min=3, max=32131, avg=1797.33, stdev=2597.74
     lat (usec): min=3, max=32132, avg=1797.99, stdev=2597.81
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    7], 10.00th=[    8], 20.00th=[   11],
     | 30.00th=[   16], 40.00th=[  889], 50.00th=[ 1483], 60.00th=[ 1991],
     | 70.00th=[ 2442], 80.00th=[ 2933], 90.00th=[ 3523], 95.00th=[ 4883],
     | 99.00th=[10683], 99.50th=[21890], 99.90th=[32113], 99.95th=[32113],
     | 99.99th=[32113]
   bw (  KiB/s): min=  960, max= 1552, per=25.36%, avg=1256.00, stdev=418.61, samples=2
   iops        : min=  240, max=  388, avg=314.00, stdev=104.65, samples=2
  write: IOPS=350, BW=1400KiB/s (1434kB/s)(1592KiB/1137msec)
    clat (usec): min=90, max=14501, avg=1144.89, stdev=1552.99
     lat (usec): min=90, max=14503, avg=1146.28, stdev=1553.08
    clat percentiles (usec):
     |  1.00th=[   93],  5.00th=[  104], 10.00th=[  119], 20.00th=[  147],
     | 30.00th=[  172], 40.00th=[  212], 50.00th=[  717], 60.00th=[  963],
     | 70.00th=[ 1303], 80.00th=[ 1926], 90.00th=[ 2868], 95.00th=[ 4178],
     | 99.00th=[ 7439], 99.50th=[ 9372], 99.90th=[14484], 99.95th=[14484],
     | 99.99th=[14484]
   bw (  KiB/s): min=  920, max= 1808, per=26.18%, avg=1364.00, stdev=627.91, samples=2
   iops        : min=  230, max=  452, avg=341.00, stdev=156.98, samples=2
  lat (usec)   : 4=0.52%, 10=8.33%, 20=7.16%, 50=1.04%, 100=1.82%
  lat (usec)   : 250=20.57%, 500=3.52%, 750=1.82%, 1000=7.55%
  lat (msec)   : 2=18.62%, 4=22.79%, 10=5.60%, 20=0.39%, 50=0.26%
  cpu          : usr=0.18%, sys=18.75%, ctx=916, majf=0, minf=36
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14006: Fri Jun 14 18:13:44 2024
   read: IOPS=306, BW=1226KiB/s (1256kB/s)(1436KiB/1171msec)
    clat (usec): min=3, max=37516, avg=1904.43, stdev=3071.60
     lat (usec): min=3, max=37517, avg=1905.50, stdev=3071.34
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    8], 10.00th=[    9], 20.00th=[   11],
     | 30.00th=[   15], 40.00th=[ 1037], 50.00th=[ 1565], 60.00th=[ 1991],
     | 70.00th=[ 2474], 80.00th=[ 2900], 90.00th=[ 3687], 95.00th=[ 4490],
     | 99.00th=[11207], 99.50th=[29230], 99.90th=[37487], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min=  728, max= 1472, per=22.21%, avg=1100.00, stdev=526.09, samples=2
   iops        : min=  182, max=  368, avg=275.00, stdev=131.52, samples=2
  write: IOPS=349, BW=1397KiB/s (1431kB/s)(1636KiB/1171msec)
    clat (usec): min=90, max=28376, avg=1163.46, stdev=1938.27
     lat (usec): min=90, max=28377, avg=1164.46, stdev=1938.21
    clat percentiles (usec):
     |  1.00th=[   95],  5.00th=[  105], 10.00th=[  117], 20.00th=[  141],
     | 30.00th=[  167], 40.00th=[  223], 50.00th=[  545], 60.00th=[  881],
     | 70.00th=[ 1172], 80.00th=[ 1844], 90.00th=[ 3097], 95.00th=[ 4293],
     | 99.00th=[ 6259], 99.50th=[ 7242], 99.90th=[28443], 99.95th=[28443],
     | 99.99th=[28443]
   bw (  KiB/s): min=  816, max= 1592, per=23.11%, avg=1204.00, stdev=548.71, samples=2
   iops        : min=  204, max=  398, avg=301.00, stdev=137.18, samples=2
  lat (usec)   : 4=0.52%, 10=7.81%, 20=7.03%, 50=0.13%, 100=2.47%
  lat (usec)   : 250=20.57%, 500=4.17%, 750=2.08%, 1000=8.20%
  lat (msec)   : 2=18.75%, 4=21.22%, 10=6.25%, 20=0.39%, 50=0.39%
  cpu          : usr=0.43%, sys=18.46%, ctx=943, majf=0, minf=33
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14007: Fri Jun 14 18:13:44 2024
   read: IOPS=323, BW=1294KiB/s (1325kB/s)(1564KiB/1209msec)
    clat (usec): min=3, max=17603, avg=1647.13, stdev=1912.54
     lat (usec): min=3, max=17604, avg=1647.78, stdev=1912.55
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    8], 10.00th=[    9], 20.00th=[   11],
     | 30.00th=[   15], 40.00th=[ 1074], 50.00th=[ 1582], 60.00th=[ 1942],
     | 70.00th=[ 2311], 80.00th=[ 2769], 90.00th=[ 3326], 95.00th=[ 4490],
     | 99.00th=[ 7504], 99.50th=[17171], 99.90th=[17695], 99.95th=[17695],
     | 99.99th=[17695]
   bw (  KiB/s): min=  832, max= 1440, per=22.94%, avg=1136.00, stdev=429.92, samples=2
   iops        : min=  208, max=  360, avg=284.00, stdev=107.48, samples=2
  write: IOPS=311, BW=1247KiB/s (1277kB/s)(1508KiB/1209msec)
    clat (usec): min=85, max=32601, avg=1451.66, stdev=2317.70
     lat (usec): min=86, max=32602, avg=1452.37, stdev=2317.75
    clat percentiles (usec):
     |  1.00th=[   97],  5.00th=[  110], 10.00th=[  123], 20.00th=[  163],
     | 30.00th=[  229], 40.00th=[  529], 50.00th=[  898], 60.00th=[ 1156],
     | 70.00th=[ 1647], 80.00th=[ 2180], 90.00th=[ 3523], 95.00th=[ 4555],
     | 99.00th=[ 8455], 99.50th=[16188], 99.90th=[32637], 99.95th=[32637],
     | 99.99th=[32637]
   bw (  KiB/s): min=  896, max= 1312, per=21.19%, avg=1104.00, stdev=294.16, samples=2
   iops        : min=  224, max=  328, avg=276.00, stdev=73.54, samples=2
  lat (usec)   : 4=0.26%, 10=8.20%, 20=8.59%, 50=0.39%, 100=1.30%
  lat (usec)   : 250=15.49%, 500=3.91%, 750=2.47%, 1000=6.64%
  lat (msec)   : 2=21.74%, 4=23.70%, 10=6.77%, 20=0.39%, 50=0.13%
  cpu          : usr=0.41%, sys=17.55%, ctx=949, majf=0, minf=35
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14008: Fri Jun 14 18:13:44 2024
   read: IOPS=334, BW=1337KiB/s (1369kB/s)(1508KiB/1128msec)
    clat (usec): min=3, max=12717, avg=1667.67, stdev=1852.93
     lat (usec): min=3, max=12718, avg=1668.51, stdev=1853.06
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    7], 10.00th=[    9], 20.00th=[   11],
     | 30.00th=[   15], 40.00th=[  791], 50.00th=[ 1483], 60.00th=[ 1844],
     | 70.00th=[ 2376], 80.00th=[ 2835], 90.00th=[ 3621], 95.00th=[ 5080],
     | 99.00th=[ 9372], 99.50th=[10945], 99.90th=[12780], 99.95th=[12780],
     | 99.99th=[12780]
   bw (  KiB/s): min=  896, max= 1552, per=24.72%, avg=1224.00, stdev=463.86, samples=2
   iops        : min=  224, max=  388, avg=306.00, stdev=115.97, samples=2
  write: IOPS=346, BW=1387KiB/s (1420kB/s)(1564KiB/1128msec)
    clat (usec): min=83, max=31230, avg=1224.43, stdev=2109.52
     lat (usec): min=83, max=31231, avg=1226.39, stdev=2109.70
    clat percentiles (usec):
     |  1.00th=[   89],  5.00th=[  108], 10.00th=[  122], 20.00th=[  149],
     | 30.00th=[  186], 40.00th=[  289], 50.00th=[  775], 60.00th=[ 1057],
     | 70.00th=[ 1401], 80.00th=[ 1926], 90.00th=[ 2737], 95.00th=[ 3654],
     | 99.00th=[ 6783], 99.50th=[16450], 99.90th=[31327], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min=  928, max= 1664, per=24.88%, avg=1296.00, stdev=520.43, samples=2
   iops        : min=  232, max=  416, avg=324.00, stdev=130.11, samples=2
  lat (usec)   : 4=0.26%, 10=8.46%, 20=7.16%, 50=1.04%, 100=2.21%
  lat (usec)   : 250=17.97%, 500=4.17%, 750=2.73%, 1000=5.60%
  lat (msec)   : 2=22.92%, 4=21.87%, 10=5.08%, 20=0.39%, 50=0.13%
  cpu          : usr=0.44%, sys=16.95%, ctx=956, majf=0, minf=35
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=4953KiB/s (5072kB/s), 1226KiB/s-1337KiB/s (1256kB/s-1369kB/s), io=5988KiB (6132kB), run=1128-1209msec
  WRITE: bw=5211KiB/s (5336kB/s), 1247KiB/s-1400KiB/s (1277kB/s-1434kB/s), io=6300KiB (6451kB), run=1128-1209msec

rand-rw: (groupid=0, jobs=1): err= 0: pid=14009: Fri Jun 14 18:13:45 2024
   read: IOPS=213, BW=855KiB/s (876kB/s)(1480KiB/1731msec)
    clat (usec): min=559, max=58523, avg=1685.39, stdev=3357.17
     lat (usec): min=559, max=58525, avg=1686.01, stdev=3357.24
    clat percentiles (usec):
     |  1.00th=[  578],  5.00th=[  660], 10.00th=[  701], 20.00th=[  766],
     | 30.00th=[  807], 40.00th=[  873], 50.00th=[  963], 60.00th=[ 1045],
     | 70.00th=[ 1319], 80.00th=[ 1926], 90.00th=[ 3163], 95.00th=[ 4490],
     | 99.00th=[ 9896], 99.50th=[14746], 99.90th=[58459], 99.95th=[58459],
     | 99.99th=[58459]
   bw (  KiB/s): min=  456, max= 1064, per=22.39%, avg=767.33, stdev=304.27, samples=3
   iops        : min=  114, max=  266, avg=191.67, stdev=76.05, samples=3
  write: IOPS=229, BW=920KiB/s (942kB/s)(1592KiB/1731msec)
    clat (usec): min=1072, max=9091, avg=2758.91, stdev=1515.59
     lat (usec): min=1073, max=9091, avg=2759.72, stdev=1515.60
    clat percentiles (usec):
     |  1.00th=[ 1205],  5.00th=[ 1450], 10.00th=[ 1598], 20.00th=[ 1729],
     | 30.00th=[ 1827], 40.00th=[ 1958], 50.00th=[ 2114], 60.00th=[ 2376],
     | 70.00th=[ 2966], 80.00th=[ 3687], 90.00th=[ 4948], 95.00th=[ 6194],
     | 99.00th=[ 8029], 99.50th=[ 8586], 99.90th=[ 9110], 99.95th=[ 9110],
     | 99.99th=[ 9110]
   bw (  KiB/s): min=  496, max= 1232, per=22.98%, avg=828.67, stdev=373.05, samples=3
   iops        : min=  124, max=  308, avg=207.00, stdev=93.31, samples=3
  lat (usec)   : 750=8.46%, 1000=17.45%
  lat (msec)   : 2=35.03%, 4=27.21%, 10=11.46%, 20=0.26%, 100=0.13%
  cpu          : usr=0.46%, sys=5.72%, ctx=818, majf=0, minf=32
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14010: Fri Jun 14 18:13:45 2024
   read: IOPS=205, BW=822KiB/s (842kB/s)(1436KiB/1747msec)
    clat (usec): min=535, max=57724, avg=1610.55, stdev=3255.33
     lat (usec): min=536, max=57726, avg=1611.39, stdev=3255.32
    clat percentiles (usec):
     |  1.00th=[  562],  5.00th=[  644], 10.00th=[  676], 20.00th=[  734],
     | 30.00th=[  766], 40.00th=[  824], 50.00th=[  873], 60.00th=[  971],
     | 70.00th=[ 1205], 80.00th=[ 1811], 90.00th=[ 3523], 95.00th=[ 4490],
     | 99.00th=[ 7570], 99.50th=[ 8979], 99.90th=[57934], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min=  440, max= 1248, per=22.02%, avg=754.67, stdev=432.62, samples=3
   iops        : min=  110, max=  312, avg=188.67, stdev=108.15, samples=3
  write: IOPS=234, BW=936KiB/s (959kB/s)(1636KiB/1747msec)
    clat (usec): min=1022, max=24883, avg=2837.56, stdev=2046.43
     lat (usec): min=1022, max=24884, avg=2838.35, stdev=2046.50
    clat percentiles (usec):
     |  1.00th=[ 1139],  5.00th=[ 1319], 10.00th=[ 1516], 20.00th=[ 1680],
     | 30.00th=[ 1827], 40.00th=[ 1958], 50.00th=[ 2089], 60.00th=[ 2376],
     | 70.00th=[ 2966], 80.00th=[ 3752], 90.00th=[ 5014], 95.00th=[ 5932],
     | 99.00th=[10421], 99.50th=[12387], 99.90th=[24773], 99.95th=[24773],
     | 99.99th=[24773]
   bw (  KiB/s): min=  488, max= 1176, per=22.56%, avg=813.33, stdev=345.52, samples=3
   iops        : min=  122, max=  294, avg=203.33, stdev=86.38, samples=3
  lat (usec)   : 750=12.11%, 1000=16.41%
  lat (msec)   : 2=33.07%, 4=24.74%, 10=12.89%, 20=0.52%, 50=0.13%
  lat (msec)   : 100=0.13%
  cpu          : usr=0.46%, sys=5.44%, ctx=816, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14011: Fri Jun 14 18:13:45 2024
   read: IOPS=230, BW=921KiB/s (943kB/s)(1564KiB/1698msec)
    clat (usec): min=536, max=10135, avg=1518.30, stdev=1428.35
     lat (usec): min=537, max=10137, avg=1518.95, stdev=1428.39
    clat percentiles (usec):
     |  1.00th=[  570],  5.00th=[  627], 10.00th=[  676], 20.00th=[  725],
     | 30.00th=[  775], 40.00th=[  824], 50.00th=[  914], 60.00th=[  988],
     | 70.00th=[ 1303], 80.00th=[ 1975], 90.00th=[ 3687], 95.00th=[ 4621],
     | 99.00th=[ 7177], 99.50th=[ 8848], 99.90th=[10159], 99.95th=[10159],
     | 99.99th=[10159]
   bw (  KiB/s): min=  432, max= 1232, per=24.04%, avg=824.00, stdev=400.24, samples=3
   iops        : min=  108, max=  308, avg=206.00, stdev=100.06, samples=3
  write: IOPS=222, BW=888KiB/s (909kB/s)(1508KiB/1698msec)
    clat (usec): min=1136, max=56796, avg=2908.39, stdev=3264.69
     lat (usec): min=1137, max=56798, avg=2909.12, stdev=3264.76
    clat percentiles (usec):
     |  1.00th=[ 1172],  5.00th=[ 1319], 10.00th=[ 1549], 20.00th=[ 1696],
     | 30.00th=[ 1795], 40.00th=[ 1909], 50.00th=[ 2114], 60.00th=[ 2409],
     | 70.00th=[ 2835], 80.00th=[ 3556], 90.00th=[ 4948], 95.00th=[ 6718],
     | 99.00th=[ 9765], 99.50th=[15008], 99.90th=[56886], 99.95th=[56886],
     | 99.99th=[56886]
   bw (  KiB/s): min=  504, max= 1272, per=22.63%, avg=816.00, stdev=403.74, samples=3
   iops        : min=  126, max=  318, avg=204.00, stdev=100.94, samples=3
  lat (usec)   : 750=12.63%, 1000=18.49%
  lat (msec)   : 2=31.64%, 4=25.26%, 10=11.46%, 20=0.39%, 100=0.13%
  cpu          : usr=0.29%, sys=6.07%, ctx=820, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=14012: Fri Jun 14 18:13:45 2024
   read: IOPS=218, BW=875KiB/s (896kB/s)(1508KiB/1723msec)
    clat (usec): min=507, max=7164, avg=1524.85, stdev=1269.19
     lat (usec): min=508, max=7164, avg=1525.54, stdev=1269.22
    clat percentiles (usec):
     |  1.00th=[  570],  5.00th=[  627], 10.00th=[  685], 20.00th=[  734],
     | 30.00th=[  783], 40.00th=[  840], 50.00th=[  930], 60.00th=[ 1090],
     | 70.00th=[ 1467], 80.00th=[ 2008], 90.00th=[ 3720], 95.00th=[ 4359],
     | 99.00th=[ 6128], 99.50th=[ 7046], 99.90th=[ 7177], 99.95th=[ 7177],
     | 99.99th=[ 7177]
   bw (  KiB/s): min=  472, max= 1040, per=22.18%, avg=760.00, stdev=284.08, samples=3
   iops        : min=  118, max=  260, avg=190.00, stdev=71.02, samples=3
  write: IOPS=226, BW=908KiB/s (930kB/s)(1564KiB/1723msec)
    clat (usec): min=1125, max=38738, avg=2917.63, stdev=2582.51
     lat (usec): min=1126, max=38740, avg=2918.52, stdev=2582.66
    clat percentiles (usec):
     |  1.00th=[ 1221],  5.00th=[ 1434], 10.00th=[ 1582], 20.00th=[ 1745],
     | 30.00th=[ 1827], 40.00th=[ 1958], 50.00th=[ 2147], 60.00th=[ 2442],
     | 70.00th=[ 2966], 80.00th=[ 3654], 90.00th=[ 4948], 95.00th=[ 6456],
     | 99.00th=[12780], 99.50th=[17957], 99.90th=[38536], 99.95th=[38536],
     | 99.99th=[38536]
   bw (  KiB/s): min=  536, max= 1240, per=22.92%, avg=826.67, stdev=367.68, samples=3
   iops        : min=  134, max=  310, avg=206.67, stdev=91.92, samples=3
  lat (usec)   : 750=11.46%, 1000=15.62%
  lat (msec)   : 2=33.46%, 4=27.73%, 10=11.07%, 20=0.52%, 50=0.13%
  cpu          : usr=0.41%, sys=5.63%, ctx=819, majf=0, minf=32
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3428KiB/s (3510kB/s), 822KiB/s-921KiB/s (842kB/s-943kB/s), io=5988KiB (6132kB), run=1698-1747msec
  WRITE: bw=3606KiB/s (3693kB/s), 888KiB/s-936KiB/s (909kB/s-959kB/s), io=6300KiB (6451kB), run=1698-1747msec