== sanity test 398b: DIO and buffer IO race ============== 14:36:15 (1731353775)
/usr/bin/fio
48+0 records in
48+0 records out
50331648 bytes (50 MB) copied, 2.14804 s, 23.4 MB/s
mix direct rw 4096 by fio with 4 jobs...
mix buffer rw 4096 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=28548: Mon Nov 11 15:30:23 2024
   read: IOPS=0, BW=1891B/s (1891B/s)(5992KiB/3243288msec)
    clat (usec): min=58, max=50080k, avg=171465.91, stdev=2886345.64
     lat (usec): min=58, max=50080k, avg=171466.78, stdev=2886345.64
    clat percentiles (usec):
     |  1.00th=[      74],  5.00th=[    1991], 10.00th=[    2180],
     | 20.00th=[    2507], 30.00th=[    2900], 40.00th=[    3425],
     | 50.00th=[    3851], 60.00th=[    4293], 70.00th=[    4752],
     | 80.00th=[    5407], 90.00th=[    6980], 95.00th=[    8586],
     | 99.00th=[   24773], 99.50th=[   46924], 99.90th=[17112761],
     | 99.95th=[17112761], 99.99th=[17112761]
   bw (  KiB/s): min=    7, max=  216, per=100.00%, avg=86.85, stdev=55.50, samples=136
   iops        : min=    1, max=   54, avg=21.62, stdev=13.90, samples=136
  write: IOPS=0, BW=1987B/s (1987B/s)(6296KiB/3243288msec)
    clat (usec): min=485, max=50112k, avg=1897288.95, stdev=9499355.88
     lat (usec): min=486, max=50112k, avg=1897289.88, stdev=9499355.89
    clat percentiles (usec):
     |  1.00th=[     537],  5.00th=[     717], 10.00th=[    1844],
     | 20.00th=[    3523], 30.00th=[    5866], 40.00th=[   11731],
     | 50.00th=[   19268], 60.00th=[   26084], 70.00th=[   33424],
     | 80.00th=[   43254], 90.00th=[   61080], 95.00th=[   94897],
     | 99.00th=[17112761], 99.50th=[17112761], 99.90th=[17112761],
     | 99.95th=[17112761], 99.99th=[17112761]
   bw (  KiB/s): min=    7, max=  208, per=100.00%, avg=85.43, stdev=53.72, samples=145
   iops        : min=    1, max=   52, avg=21.28, stdev=13.44, samples=145
  lat (usec)   : 100=0.88%, 250=0.36%, 500=0.16%, 750=2.60%, 1000=1.40%
  lat (msec)   : 2=2.67%, 4=29.26%, 10=29.17%, 20=7.85%, 50=17.55%
  lat (msec)   : 100=5.50%, 250=0.52%
  cpu          : usr=0.00%, sys=0.12%, ctx=9199, majf=0, minf=37
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=28549: Mon Nov 11 15:30:23 2024
   read: IOPS=0, BW=1875B/s (1875B/s)(5940KiB/3242990msec)
    clat (usec): min=60, max=50052k, avg=240251.72, stdev=3426904.11
     lat (usec): min=61, max=50052k, avg=240253.14, stdev=3426904.07
    clat percentiles (usec):
     |  1.00th=[     110],  5.00th=[    2008], 10.00th=[    2180],
     | 20.00th=[    2474], 30.00th=[    2900], 40.00th=[    3425],
     | 50.00th=[    3916], 60.00th=[    4293], 70.00th=[    4817],
     | 80.00th=[    5473], 90.00th=[    6980], 95.00th=[    8717],
     | 99.00th=[   28705], 99.50th=[   86508], 99.90th=[17112761],
     | 99.95th=[17112761], 99.99th=[17112761]
   bw (  KiB/s): min=    7, max=  256, per=100.00%, avg=86.94, stdev=58.40, samples=136
   iops        : min=    1, max=   64, avg=21.65, stdev=14.63, samples=136
  write: IOPS=0, BW=2004B/s (2004B/s)(6348KiB/3242990msec)
    clat (usec): min=481, max=100022k, avg=1818596.10, stdev=9474807.88
     lat (usec): min=481, max=100022k, avg=1818597.11, stdev=9474807.87
    clat percentiles (usec):
     |  1.00th=[     537],  5.00th=[     717], 10.00th=[    1893],
     | 20.00th=[    3458], 30.00th=[    6390], 40.00th=[   10814],
     | 50.00th=[   18482], 60.00th=[   25035], 70.00th=[   32900],
     | 80.00th=[   43254], 90.00th=[   57934], 95.00th=[   87557],
     | 99.00th=[17112761], 99.50th=[17112761], 99.90th=[17112761],
     | 99.95th=[17112761], 99.99th=[17112761]
   bw (  KiB/s): min=    8, max=  200, per=100.00%, avg=88.68, stdev=54.16, samples=142
   iops        : min=    2, max=   50, avg=22.09, stdev=13.56, samples=142
  lat (usec)   : 100=0.46%, 250=0.36%, 500=0.13%, 750=2.70%, 1000=1.24%
  lat (msec)   : 2=2.99%, 4=29.17%, 10=29.07%, 20=8.40%, 50=17.90%
  lat (msec)   : 100=5.21%, 250=0.33%
  cpu          : usr=0.00%, sys=0.12%, ctx=9207, majf=0, minf=36
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=28551: Mon Nov 11 15:30:23 2024
   read: IOPS=0, BW=1902B/s (1902B/s)(6028KiB/3244198msec)
    clat (usec): min=57, max=50053k, avg=170460.97, stdev=2877454.78
     lat (usec): min=58, max=50053k, avg=170461.83, stdev=2877454.79
    clat percentiles (usec):
     |  1.00th=[      77],  5.00th=[    1942], 10.00th=[    2147],
     | 20.00th=[    2474], 30.00th=[    2835], 40.00th=[    3392],
     | 50.00th=[    3818], 60.00th=[    4228], 70.00th=[    4752],
     | 80.00th=[    5538], 90.00th=[    6980], 95.00th=[    8979],
     | 99.00th=[   26084], 99.50th=[   40633], 99.90th=[17112761],
     | 99.95th=[17112761], 99.99th=[17112761]
   bw (  KiB/s): min=    8, max=  240, per=100.00%, avg=85.93, stdev=55.28, samples=139
   iops        : min=    2, max=   60, avg=21.36, stdev=13.85, samples=139
  write: IOPS=0, BW=1975B/s (1975B/s)(6260KiB/3244198msec)
    clat (usec): min=482, max=50228k, avg=1908762.23, stdev=9525862.39
     lat (usec): min=483, max=50228k, avg=1908763.17, stdev=9525862.39
    clat percentiles (usec):
     |  1.00th=[     545],  5.00th=[     750], 10.00th=[    1844],
     | 20.00th=[    3752], 30.00th=[    6783], 40.00th=[   11469],
     | 50.00th=[   19268], 60.00th=[   26608], 70.00th=[   34866],
     | 80.00th=[   43779], 90.00th=[   63701], 95.00th=[  103285],
     | 99.00th=[17112761], 99.50th=[17112761], 99.90th=[17112761],
     | 99.95th=[17112761], 99.99th=[17112761]
   bw (  KiB/s): min=    7, max=  216, per=100.00%, avg=84.26, stdev=54.00, samples=147
   iops        : min=    1, max=   54, avg=20.94, stdev=13.51, samples=147
  lat (usec)   : 100=0.78%, 250=0.55%, 500=0.03%, 750=2.57%, 1000=1.40%
  lat (msec)   : 2=3.03%, 4=28.84%, 10=28.91%, 20=8.40%, 50=17.22%
  lat (msec)   : 100=5.50%, 250=0.68%
  cpu          : usr=0.00%, sys=0.13%, ctx=9279, majf=0, minf=36
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=28553: Mon Nov 11 15:30:23 2024
   read: IOPS=0, BW=1911B/s (1911B/s)(6052KiB/3242680msec)
    clat (usec): min=59, max=50031k, avg=202770.41, stdev=3143007.19
     lat (usec): min=59, max=50031k, avg=202771.28, stdev=3143007.19
    clat percentiles (usec):
     |  1.00th=[     119],  5.00th=[    1975], 10.00th=[    2180],
     | 20.00th=[    2507], 30.00th=[    2966], 40.00th=[    3425],
     | 50.00th=[    3884], 60.00th=[    4228], 70.00th=[    4752],
     | 80.00th=[    5473], 90.00th=[    6915], 95.00th=[    9241],
     | 99.00th=[   24249], 99.50th=[   60031], 99.90th=[17112761],
     | 99.95th=[17112761], 99.99th=[17112761]
   bw (  KiB/s): min=    7, max=  272, per=100.00%, avg=88.62, stdev=58.77, samples=136
   iops        : min=    1, max=   68, avg=22.04, stdev=14.73, samples=136
  write: IOPS=0, BW=1969B/s (1969B/s)(6236KiB/3242680msec)
    clat (usec): min=486, max=50112k, avg=1883122.55, stdev=9466048.71
     lat (usec): min=487, max=50112k, avg=1883123.68, stdev=9466048.72
    clat percentiles (usec):
     |  1.00th=[     529],  5.00th=[     717], 10.00th=[    1729],
     | 20.00th=[    3654], 30.00th=[    6521], 40.00th=[   11207],
     | 50.00th=[   18220], 60.00th=[   25822], 70.00th=[   33817],
     | 80.00th=[   42730], 90.00th=[   58983], 95.00th=[   89654],
     | 99.00th=[17112761], 99.50th=[17112761], 99.90th=[17112761],
     | 99.95th=[17112761], 99.99th=[17112761]
   bw (  KiB/s): min=    7, max=  256, per=100.00%, avg=84.94, stdev=55.22, samples=146
   iops        : min=    1, max=   64, avg=21.12, stdev=13.85, samples=146
  lat (usec)   : 100=0.36%, 250=0.39%, 500=0.03%, 750=2.77%, 1000=1.30%
  lat (msec)   : 2=3.16%, 4=29.56%, 10=29.07%, 20=8.40%, 50=17.45%
  lat (msec)   : 100=5.01%, 250=0.42%
  cpu          : usr=0.00%, sys=0.12%, ctx=9139, majf=0, minf=33
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7579B/s (7579B/s), 1875B/s-1911B/s (1875B/s-1911B/s), io=23.4MiB (24.6MB), run=3242680-3244198msec
  WRITE: bw=7935B/s (7935B/s), 1969B/s-2004B/s (1969B/s-2004B/s), io=24.6MiB (25.7MB), run=3242680-3244198msec

rand-rw: (groupid=0, jobs=1): err= 0: pid=28550: Mon Nov 11 15:30:24 2024
   read: IOPS=0, BW=1890B/s (1890B/s)(5992KiB/3244778msec)
    clat (usec): min=1588, max=50078k, avg=773274.75, stdev=6149881.22
     lat (usec): min=1589, max=50078k, avg=773275.65, stdev=6149881.22
    clat percentiles (usec):
     |  1.00th=[    1795],  5.00th=[    1975], 10.00th=[    2073],
     | 20.00th=[    2245], 30.00th=[    2442], 40.00th=[    2671],
     | 50.00th=[    2999], 60.00th=[    3490], 70.00th=[    4146],
     | 80.00th=[    5342], 90.00th=[    9110], 95.00th=[   27657],
     | 99.00th=[17112761], 99.50th=[17112761], 99.90th=[17112761],
     | 99.95th=[17112761], 99.99th=[17112761]
   bw (  KiB/s): min=    8, max=  224, per=100.00%, avg=84.76, stdev=55.50, samples=140
   iops        : min=    2, max=   56, avg=21.16, stdev=13.86, samples=140
  write: IOPS=0, BW=1986B/s (1986B/s)(6296KiB/3244778msec)
    clat (msec): min=9, max=99937, avg=1325.49, stdev=8165.14
     lat (msec): min=9, max=99937, avg=1325.49, stdev=8165.14
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[   13], 10.00th=[   14], 20.00th=[   17],
     | 30.00th=[   18], 40.00th=[   20], 50.00th=[   21], 60.00th=[   23],
     | 70.00th=[   25], 80.00th=[   28], 90.00th=[   34], 95.00th=[   50],
     | 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113],
     | 99.99th=[17113]
   bw (  KiB/s): min=    8, max=  200, per=100.00%, avg=84.13, stdev=55.20, samples=148
   iops        : min=    2, max=   50, avg=21.01, stdev=13.79, samples=148
  lat (msec)   : 2=3.09%, 4=29.92%, 10=11.36%, 20=23.89%, 50=27.93%
  lat (msec)   : 100=1.56%, 250=0.20%
  cpu          : usr=0.00%, sys=0.07%, ctx=3629, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=28552: Mon Nov 11 15:30:24 2024
   read: IOPS=0, BW=1874B/s (1874B/s)(5940KiB/3244891msec)
    clat (usec): min=1702, max=50075k, avg=679081.76, stdev=5766236.33
     lat (usec): min=1703, max=50075k, avg=679082.69, stdev=5766236.32
    clat percentiles (usec):
     |  1.00th=[    1795],  5.00th=[    1958], 10.00th=[    2073],
     | 20.00th=[    2245], 30.00th=[    2409], 40.00th=[    2671],
     | 50.00th=[    3064], 60.00th=[    3556], 70.00th=[    4228],
     | 80.00th=[    5342], 90.00th=[   10683], 95.00th=[   24773],
     | 99.00th=[17112761], 99.50th=[17112761], 99.90th=[17112761],
     | 99.95th=[17112761], 99.99th=[17112761]
   bw (  KiB/s): min=    7, max=  264, per=100.00%, avg=86.38, stdev=59.85, samples=136
   iops        : min=    1, max=   66, avg=21.57, stdev=14.98, samples=136
  write: IOPS=0, BW=2003B/s (2003B/s)(6348KiB/3244891msec)
    clat (msec): min=10, max=50142, avg=1409.16, stdev=8214.95
     lat (msec): min=10, max=50142, avg=1409.16, stdev=8214.95
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[   13], 10.00th=[   15], 20.00th=[   17],
     | 30.00th=[   18], 40.00th=[   20], 50.00th=[   21], 60.00th=[   23],
     | 70.00th=[   25], 80.00th=[   27], 90.00th=[   34], 95.00th=[   50],
     | 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113],
     | 99.99th=[17113]
   bw (  KiB/s): min=    8, max=  192, per=100.00%, avg=84.97, stdev=55.25, samples=147
   iops        : min=    2, max=   48, avg=21.21, stdev=13.83, samples=147
  lat (msec)   : 2=3.19%, 4=29.00%, 10=11.17%, 20=25.23%, 50=27.90%
  lat (msec)   : 100=1.30%, 250=0.13%
  cpu          : usr=0.00%, sys=0.07%, ctx=3552, majf=0, minf=35
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=28554: Mon Nov 11 15:30:24 2024
   read: IOPS=0, BW=1902B/s (1902B/s)(6028KiB/3244597msec)
    clat (usec): min=1743, max=100031k, avg=768301.77, stdev=6396566.32
     lat (usec): min=1744, max=100031k, avg=768302.67, stdev=6396566.32
    clat percentiles (usec):
     |  1.00th=[    1844],  5.00th=[    1958], 10.00th=[    2057],
     | 20.00th=[    2245], 30.00th=[    2474], 40.00th=[    2704],
     | 50.00th=[    3032], 60.00th=[    3523], 70.00th=[    4178],
     | 80.00th=[    5276], 90.00th=[    9241], 95.00th=[   23462],
     | 99.00th=[17112761], 99.50th=[17112761], 99.90th=[17112761],
     | 99.95th=[17112761], 99.99th=[17112761]
   bw (  KiB/s): min=    8, max=  232, per=100.00%, avg=87.66, stdev=58.62, samples=137
   iops        : min=    2, max=   58, avg=21.88, stdev=14.67, samples=137
  write: IOPS=0, BW=1975B/s (1975B/s)(6260KiB/3244597msec)
    clat (msec): min=10, max=50104, avg=1333.34, stdev=7992.37
     lat (msec): min=10, max=50104, avg=1333.34, stdev=7992.37
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[   14], 10.00th=[   15], 20.00th=[   17],
     | 30.00th=[   18], 40.00th=[   20], 50.00th=[   21], 60.00th=[   23],
     | 70.00th=[   25], 80.00th=[   28], 90.00th=[   35], 95.00th=[   53],
     | 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113],
     | 99.99th=[17113]
   bw (  KiB/s): min=    7, max=  208, per=100.00%, avg=87.78, stdev=54.98, samples=142
   iops        : min=    1, max=   52, avg=21.91, stdev=13.77, samples=142
  lat (msec)   : 2=3.35%, 4=29.75%, 10=11.36%, 20=24.61%, 50=27.25%
  lat (msec)   : 100=1.53%, 250=0.10%
  cpu          : usr=0.00%, sys=0.07%, ctx=3634, majf=0, minf=37
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=28555: Mon Nov 11 15:30:24 2024
   read: IOPS=0, BW=1910B/s (1910B/s)(6052KiB/3244396msec)
    clat (usec): min=1652, max=50079k, avg=600518.88, stdev=5424252.06
     lat (usec): min=1652, max=50079k, avg=600519.80, stdev=5424252.05
    clat percentiles (usec):
     |  1.00th=[    1811],  5.00th=[    1942], 10.00th=[    2073],
     | 20.00th=[    2245], 30.00th=[    2474], 40.00th=[    2671],
     | 50.00th=[    3064], 60.00th=[    3556], 70.00th=[    4228],
     | 80.00th=[    5342], 90.00th=[    9503], 95.00th=[   25822],
     | 99.00th=[17112761], 99.50th=[17112761], 99.90th=[17112761],
     | 99.95th=[17112761], 99.99th=[17112761]
   bw (  KiB/s): min=    7, max=  240, per=100.00%, avg=84.54, stdev=57.25, samples=141
   iops        : min=    1, max=   60, avg=21.05, stdev=14.32, samples=141
  write: IOPS=0, BW=1968B/s (1968B/s)(6236KiB/3244396msec)
    clat (msec): min=10, max=50132, avg=1498.21, stdev=8466.07
     lat (msec): min=10, max=50132, avg=1498.21, stdev=8466.07
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[   14], 10.00th=[   15], 20.00th=[   17],
     | 30.00th=[   18], 40.00th=[   20], 50.00th=[   21], 60.00th=[   23],
     | 70.00th=[   25], 80.00th=[   28], 90.00th=[   34], 95.00th=[   52],
     | 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113],
     | 99.99th=[17113]
   bw (  KiB/s): min=    7, max=  199, per=100.00%, avg=84.32, stdev=55.52, samples=146
   iops        : min=    1, max=   49, avg=20.99, stdev=13.89, samples=146
  lat (msec)   : 2=3.55%, 4=29.56%, 10=11.43%, 20=24.74%, 50=27.18%
  lat (msec)   : 100=1.37%, 250=0.10%
  cpu          : usr=0.00%, sys=0.07%, ctx=3589, majf=0, minf=36
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7577B/s (7577B/s), 1874B/s-1910B/s (1874B/s-1910B/s), io=23.4MiB (24.6MB), run=3244396-3244891msec
  WRITE: bw=7933B/s (7933B/s), 1968B/s-2003B/s (1968B/s-2003B/s), io=24.6MiB (25.7MB), run=3244396-3244891msec
mix direct rw 16384 by fio with 4 jobs...
mix buffer rw 16384 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes
rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes