== sanity test 398b: DIO and buffer IO race ============== 06:56:53 (1721386613)
/usr/bin/fio
12+0 records in
12+0 records out
12582912 bytes (13 MB) copied, 0.100637 s, 125 MB/s
mix direct rw 12M to OST0 by fio with 4 jobs...
mix buffer rw 12M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=21878: Fri Jul 19 06:56:55 2024
   read: IOPS=325, BW=1302KiB/s (1333kB/s)(1480KiB/1137msec)
    clat (usec): min=3, max=26095, avg=1639.79, stdev=2224.86
     lat (usec): min=3, max=26096, avg=1641.12, stdev=2225.11
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    7], 10.00th=[    9], 20.00th=[   11],
     | 30.00th=[   13], 40.00th=[   17], 50.00th=[ 1221], 60.00th=[ 1876],
     | 70.00th=[ 2376], 80.00th=[ 2933], 90.00th=[ 3851], 95.00th=[ 4686],
     | 99.00th=[10159], 99.50th=[11731], 99.90th=[26084], 99.95th=[26084],
     | 99.99th=[26084]
   bw (  KiB/s): min= 1024, max= 1413, per=23.14%, avg=1218.50, stdev=275.06, samples=2
   iops        : min=  256, max=  353, avg=304.50, stdev=68.59, samples=2
  write: IOPS=350, BW=1400KiB/s (1434kB/s)(1592KiB/1137msec)
    clat (usec): min=91, max=39280, avg=1285.70, stdev=2506.28
     lat (usec): min=91, max=39281, avg=1286.59, stdev=2506.33
    clat percentiles (usec):
     |  1.00th=[   98],  5.00th=[  114], 10.00th=[  139], 20.00th=[  167],
     | 30.00th=[  190], 40.00th=[  269], 50.00th=[  717], 60.00th=[ 1012],
     | 70.00th=[ 1254], 80.00th=[ 2057], 90.00th=[ 2999], 95.00th=[ 3818],
     | 99.00th=[ 9634], 99.50th=[16712], 99.90th=[39060], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min=  960, max= 1740, per=24.37%, avg=1350.00, stdev=551.54, samples=2
   iops        : min=  240, max=  435, avg=337.50, stdev=137.89, samples=2
  lat (usec)   : 4=0.39%, 10=7.81%, 20=11.72%, 50=0.65%, 100=1.17%
  lat (usec)   : 250=19.01%, 500=5.34%, 750=1.17%, 1000=6.25%
  lat (msec)   : 2=17.58%, 4=23.18%, 10=4.82%, 20=0.65%, 50=0.26%
  cpu          : usr=0.62%, sys=17.08%, ctx=840, majf=0, minf=35
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21880: Fri Jul 19 06:56:55 2024
   read: IOPS=315, BW=1263KiB/s (1293kB/s)(1436KiB/1137msec)
    clat (usec): min=3, max=28460, avg=1785.19, stdev=2738.90
     lat (usec): min=4, max=28461, avg=1785.80, stdev=2738.94
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    7], 10.00th=[    8], 20.00th=[   11],
     | 30.00th=[   13], 40.00th=[   21], 50.00th=[ 1401], 60.00th=[ 1942],
     | 70.00th=[ 2442], 80.00th=[ 2933], 90.00th=[ 3982], 95.00th=[ 4686],
     | 99.00th=[13698], 99.50th=[24511], 99.90th=[28443], 99.95th=[28443],
     | 99.99th=[28443]
   bw (  KiB/s): min=  840, max= 1544, per=22.64%, avg=1192.00, stdev=497.80, samples=2
   iops        : min=  210, max=  386, avg=298.00, stdev=124.45, samples=2
  write: IOPS=359, BW=1439KiB/s (1473kB/s)(1636KiB/1137msec)
    clat (usec): min=84, max=32564, avg=1188.10, stdev=2185.88
     lat (usec): min=84, max=32564, avg=1189.06, stdev=2185.85
    clat percentiles (usec):
     |  1.00th=[   92],  5.00th=[  105], 10.00th=[  116], 20.00th=[  143],
     | 30.00th=[  178], 40.00th=[  219], 50.00th=[  379], 60.00th=[  988],
     | 70.00th=[ 1221], 80.00th=[ 1991], 90.00th=[ 3032], 95.00th=[ 3818],
     | 99.00th=[ 5473], 99.50th=[ 5735], 99.90th=[32637], 99.95th=[32637],
     | 99.99th=[32637]
   bw (  KiB/s): min=  944, max= 1696, per=23.83%, avg=1320.00, stdev=531.74, samples=2
   iops        : min=  236, max=  424, avg=330.00, stdev=132.94, samples=2
  lat (usec)   : 4=0.26%, 10=8.85%, 20=9.51%, 50=0.78%, 100=2.08%
  lat (usec)   : 250=21.48%, 500=4.56%, 750=0.78%, 1000=5.08%
  lat (msec)   : 2=18.10%, 4=21.35%, 10=6.25%, 20=0.52%, 50=0.39%
  cpu          : usr=0.09%, sys=18.75%, ctx=895, majf=0, minf=32
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21882: Fri Jul 19 06:56:55 2024
   read: IOPS=348, BW=1394KiB/s (1427kB/s)(1564KiB/1122msec)
    clat (usec): min=3, max=14417, avg=1575.96, stdev=1785.42
     lat (usec): min=3, max=14418, avg=1576.60, stdev=1785.44
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    7], 10.00th=[    8], 20.00th=[   10],
     | 30.00th=[   13], 40.00th=[   42], 50.00th=[ 1401], 60.00th=[ 1975],
     | 70.00th=[ 2409], 80.00th=[ 2835], 90.00th=[ 3425], 95.00th=[ 4228],
     | 99.00th=[ 8455], 99.50th=[ 9765], 99.90th=[14353], 99.95th=[14353],
     | 99.99th=[14353]
   bw (  KiB/s): min=  872, max= 1648, per=23.93%, avg=1260.00, stdev=548.71, samples=2
   iops        : min=  218, max=  412, avg=315.00, stdev=137.18, samples=2
  write: IOPS=336, BW=1344KiB/s (1376kB/s)(1508KiB/1122msec)
    clat (usec): min=85, max=32387, avg=1306.60, stdev=2877.16
     lat (usec): min=85, max=32389, avg=1307.41, stdev=2877.26
    clat percentiles (usec):
     |  1.00th=[   92],  5.00th=[  105], 10.00th=[  117], 20.00th=[  151],
     | 30.00th=[  178], 40.00th=[  208], 50.00th=[  453], 60.00th=[  938],
     | 70.00th=[ 1205], 80.00th=[ 1745], 90.00th=[ 2933], 95.00th=[ 4359],
     | 99.00th=[ 9372], 99.50th=[32113], 99.90th=[32375], 99.95th=[32375],
     | 99.99th=[32375]
   bw (  KiB/s): min=  896, max= 1600, per=22.53%, avg=1248.00, stdev=497.80, samples=2
   iops        : min=  224, max=  400, avg=312.00, stdev=124.45, samples=2
  lat (usec)   : 4=0.26%, 10=10.42%, 20=9.11%, 50=0.65%, 100=1.69%
  lat (usec)   : 250=20.57%, 500=2.99%, 750=1.04%, 1000=5.73%
  lat (msec)   : 2=18.88%, 4=22.66%, 10=5.47%, 20=0.13%, 50=0.39%
  cpu          : usr=0.62%, sys=16.50%, ctx=795, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21884: Fri Jul 19 06:56:55 2024
   read: IOPS=341, BW=1366KiB/s (1399kB/s)(1508KiB/1104msec)
    clat (usec): min=3, max=43522, avg=1717.21, stdev=3294.16
     lat (usec): min=3, max=43523, avg=1718.07, stdev=3294.08
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    6], 10.00th=[    8], 20.00th=[   10],
     | 30.00th=[   12], 40.00th=[   17], 50.00th=[ 1237], 60.00th=[ 1893],
     | 70.00th=[ 2376], 80.00th=[ 2868], 90.00th=[ 3392], 95.00th=[ 4228],
     | 99.00th=[13042], 99.50th=[26346], 99.90th=[43779], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min=  864, max= 1464, per=22.10%, avg=1164.00, stdev=424.26, samples=2
   iops        : min=  216, max=  366, avg=291.00, stdev=106.07, samples=2
  write: IOPS=354, BW=1417KiB/s (1451kB/s)(1564KiB/1104msec)
    clat (usec): min=78, max=9781, avg=1130.57, stdev=1478.57
     lat (usec): min=79, max=9782, avg=1131.86, stdev=1478.62
    clat percentiles (usec):
     |  1.00th=[   85],  5.00th=[  100], 10.00th=[  115], 20.00th=[  141],
     | 30.00th=[  172], 40.00th=[  215], 50.00th=[  347], 60.00th=[  963],
     | 70.00th=[ 1221], 80.00th=[ 1958], 90.00th=[ 3032], 95.00th=[ 4359],
     | 99.00th=[ 6849], 99.50th=[ 8848], 99.90th=[ 9765], 99.95th=[ 9765],
     | 99.99th=[ 9765]
   bw (  KiB/s): min=  912, max= 1600, per=22.67%, avg=1256.00, stdev=486.49, samples=2
   iops        : min=  228, max=  400, avg=314.00, stdev=121.62, samples=2
  lat (usec)   : 4=0.91%, 10=10.81%, 20=8.72%, 50=0.52%, 100=2.86%
  lat (usec)   : 250=20.44%, 500=4.56%, 750=1.17%, 1000=4.04%
  lat (msec)   : 2=17.58%, 4=22.66%, 10=5.08%, 20=0.26%, 50=0.39%
  cpu          : usr=0.18%, sys=19.85%, ctx=894, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=5266KiB/s (5393kB/s), 1263KiB/s-1394KiB/s (1293kB/s-1427kB/s), io=5988KiB (6132kB), run=1104-1137msec
  WRITE: bw=5541KiB/s (5674kB/s), 1344KiB/s-1439KiB/s (1376kB/s-1473kB/s), io=6300KiB (6451kB), run=1104-1137msec

rand-rw: (groupid=0, jobs=1): err= 0: pid=21879: Fri Jul 19 06:56:56 2024
   read: IOPS=195, BW=783KiB/s (802kB/s)(1480KiB/1889msec)
    clat (usec): min=594, max=26830, avg=1640.30, stdev=2083.48
     lat (usec): min=595, max=26832, avg=1641.10, stdev=2083.50
    clat percentiles (usec):
     |  1.00th=[  693],  5.00th=[  734], 10.00th=[  766], 20.00th=[  840],
     | 30.00th=[  889], 40.00th=[  955], 50.00th=[ 1020], 60.00th=[ 1090],
     | 70.00th=[ 1303], 80.00th=[ 1762], 90.00th=[ 3130], 95.00th=[ 4686],
     | 99.00th=[ 8291], 99.50th=[16319], 99.90th=[26870], 99.95th=[26870],
     | 99.99th=[26870]
   bw (  KiB/s): min=  432, max=  776, per=20.64%, avg=645.33, stdev=186.30, samples=3
   iops        : min=  108, max=  194, avg=161.33, stdev=46.58, samples=3
  write: IOPS=210, BW=843KiB/s (863kB/s)(1592KiB/1889msec)
    clat (usec): min=1180, max=50911, avg=3174.77, stdev=3305.01
     lat (usec): min=1181, max=50912, avg=3175.71, stdev=3305.00
    clat percentiles (usec):
     |  1.00th=[ 1336],  5.00th=[ 1565], 10.00th=[ 1745], 20.00th=[ 2008],
     | 30.00th=[ 2114], 40.00th=[ 2212], 50.00th=[ 2343], 60.00th=[ 2606],
     | 70.00th=[ 3163], 80.00th=[ 3818], 90.00th=[ 5145], 95.00th=[ 6587],
     | 99.00th=[ 9896], 99.50th=[36439], 99.90th=[51119], 99.95th=[51119],
     | 99.99th=[51119]
   bw (  KiB/s): min=  464, max= 1064, per=22.05%, avg=725.33, stdev=307.38, samples=3
   iops        : min=  116, max=  266, avg=181.33, stdev=76.85, samples=3
  lat (usec)   : 750=3.26%, 1000=20.05%
  lat (msec)   : 2=27.47%, 4=36.59%, 10=11.85%, 20=0.39%, 50=0.26%
  lat (msec)   : 100=0.13%
  cpu          : usr=0.11%, sys=6.09%, ctx=830, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21881: Fri Jul 19 06:56:56 2024
   read: IOPS=190, BW=762KiB/s (780kB/s)(1436KiB/1885msec)
    clat (usec): min=555, max=27877, avg=1648.02, stdev=2095.30
     lat (usec): min=556, max=27879, avg=1648.68, stdev=2095.35
    clat percentiles (usec):
     |  1.00th=[  676],  5.00th=[  758], 10.00th=[  791], 20.00th=[  857],
     | 30.00th=[  898], 40.00th=[  955], 50.00th=[ 1012], 60.00th=[ 1106],
     | 70.00th=[ 1369], 80.00th=[ 1860], 90.00th=[ 3195], 95.00th=[ 4883],
     | 99.00th=[ 7111], 99.50th=[19792], 99.90th=[27919], 99.95th=[27919],
     | 99.99th=[27919]
   bw (  KiB/s): min=  448, max=  968, per=21.16%, avg=661.33, stdev=272.27, samples=3
   iops        : min=  112, max=  242, avg=165.33, stdev=68.07, samples=3
  write: IOPS=216, BW=868KiB/s (889kB/s)(1636KiB/1885msec)
    clat (usec): min=1287, max=27041, avg=3111.76, stdev=2168.06
     lat (usec): min=1287, max=27043, avg=3112.59, stdev=2168.10
    clat percentiles (usec):
     |  1.00th=[ 1352],  5.00th=[ 1549], 10.00th=[ 1795], 20.00th=[ 1991],
     | 30.00th=[ 2114], 40.00th=[ 2245], 50.00th=[ 2376], 60.00th=[ 2638],
     | 70.00th=[ 3130], 80.00th=[ 3884], 90.00th=[ 5211], 95.00th=[ 6390],
     | 99.00th=[11731], 99.50th=[12125], 99.90th=[27132], 99.95th=[27132],
     | 99.99th=[27132]
   bw (  KiB/s): min=  528, max=  960, per=22.46%, avg=738.67, stdev=216.20, samples=3
   iops        : min=  132, max=  240, avg=184.67, stdev=54.05, samples=3
  lat (usec)   : 750=2.08%, 1000=20.18%
  lat (msec)   : 2=27.08%, 4=36.98%, 10=12.50%, 20=0.91%, 50=0.26%
  cpu          : usr=0.27%, sys=5.94%, ctx=840, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21883: Fri Jul 19 06:56:56 2024
   read: IOPS=208, BW=836KiB/s (856kB/s)(1564KiB/1871msec)
    clat (usec): min=634, max=29543, avg=1721.25, stdev=2158.61
     lat (usec): min=635, max=29545, avg=1721.95, stdev=2158.67
    clat percentiles (usec):
     |  1.00th=[  660],  5.00th=[  734], 10.00th=[  775], 20.00th=[  824],
     | 30.00th=[  881], 40.00th=[  947], 50.00th=[  996], 60.00th=[ 1106],
     | 70.00th=[ 1336], 80.00th=[ 2089], 90.00th=[ 3458], 95.00th=[ 5080],
     | 99.00th=[10421], 99.50th=[15401], 99.90th=[29492], 99.95th=[29492],
     | 99.99th=[29492]
   bw (  KiB/s): min=  432, max=  976, per=22.52%, avg=704.00, stdev=272.00, samples=3
   iops        : min=  108, max=  244, avg=176.00, stdev=68.00, samples=3
  write: IOPS=201, BW=806KiB/s (825kB/s)(1508KiB/1871msec)
    clat (usec): min=1279, max=31496, avg=3146.44, stdev=2485.19
     lat (usec): min=1280, max=31500, avg=3147.28, stdev=2485.31
    clat percentiles (usec):
     |  1.00th=[ 1336],  5.00th=[ 1500], 10.00th=[ 1745], 20.00th=[ 1991],
     | 30.00th=[ 2114], 40.00th=[ 2245], 50.00th=[ 2376], 60.00th=[ 2606],
     | 70.00th=[ 3064], 80.00th=[ 3752], 90.00th=[ 5538], 95.00th=[ 7111],
     | 99.00th=[10945], 99.50th=[21890], 99.90th=[31589], 99.95th=[31589],
     | 99.99th=[31589]
   bw (  KiB/s): min=  504, max=  944, per=20.51%, avg=674.67, stdev=236.01, samples=3
   iops        : min=  126, max=  236, avg=168.67, stdev=59.00, samples=3
  lat (usec)   : 750=3.39%, 1000=22.53%
  lat (msec)   : 2=24.48%, 4=36.98%, 10=11.59%, 20=0.65%, 50=0.39%
  cpu          : usr=0.48%, sys=5.67%, ctx=811, majf=0, minf=32
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21885: Fri Jul 19 06:56:56 2024
   read: IOPS=196, BW=787KiB/s (806kB/s)(1508KiB/1915msec)
    clat (usec): min=614, max=12040, avg=1725.24, stdev=1589.12
     lat (usec): min=615, max=12041, avg=1726.13, stdev=1589.14
    clat percentiles (usec):
     |  1.00th=[  660],  5.00th=[  717], 10.00th=[  775], 20.00th=[  857],
     | 30.00th=[  906], 40.00th=[  963], 50.00th=[ 1045], 60.00th=[ 1172],
     | 70.00th=[ 1401], 80.00th=[ 2212], 90.00th=[ 4113], 95.00th=[ 5342],
     | 99.00th=[ 7832], 99.50th=[ 8356], 99.90th=[11994], 99.95th=[11994],
     | 99.99th=[11994]
   bw (  KiB/s): min=  416, max=  880, per=21.07%, avg=658.67, stdev=232.73, samples=3
   iops        : min=  104, max=  220, avg=164.67, stdev=58.18, samples=3
  write: IOPS=204, BW=817KiB/s (836kB/s)(1564KiB/1915msec)
    clat (usec): min=1296, max=41594, avg=3185.06, stdev=2912.22
     lat (usec): min=1296, max=41596, avg=3186.48, stdev=2912.23
    clat percentiles (usec):
     |  1.00th=[ 1401],  5.00th=[ 1614], 10.00th=[ 1827], 20.00th=[ 2008],
     | 30.00th=[ 2114], 40.00th=[ 2245], 50.00th=[ 2376], 60.00th=[ 2606],
     | 70.00th=[ 3130], 80.00th=[ 3687], 90.00th=[ 4948], 95.00th=[ 6652],
     | 99.00th=[11600], 99.50th=[27132], 99.90th=[41681], 99.95th=[41681],
     | 99.99th=[41681]
   bw (  KiB/s): min=  456, max=  984, per=20.67%, avg=680.00, stdev=272.94, samples=3
   iops        : min=  114, max=  246, avg=170.00, stdev=68.23, samples=3
  lat (usec)   : 750=3.78%, 1000=18.23%
  lat (msec)   : 2=25.65%, 4=38.80%, 10=12.37%, 20=0.91%, 50=0.26%
  cpu          : usr=0.21%, sys=6.22%, ctx=854, majf=0, minf=33
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3127KiB/s (3202kB/s), 762KiB/s-836KiB/s (780kB/s-856kB/s), io=5988KiB (6132kB), run=1871-1915msec
  WRITE: bw=3290KiB/s (3369kB/s), 806KiB/s-868KiB/s (825kB/s-889kB/s), io=6300KiB (6451kB), run=1871-1915msec