== sanity test 398b: DIO and buffer IO race ============== 19:32:48 (1736987568)
/usr/bin/fio
12+0 records in
12+0 records out
12582912 bytes (13 MB) copied, 0.0758564 s, 166 MB/s
mix direct rw 12M to OST0 by fio with 4 jobs...
mix buffer rw 12M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=17480: Wed Jan 15 19:32:49 2025
   read: IOPS=2283, BW=9136KiB/s (9355kB/s)(1480KiB/162msec)
    clat (usec): min=4, max=7054, avg=134.07, stdev=715.06
     lat (usec): min=4, max=7055, avg=134.62, stdev=715.09
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    6], 10.00th=[    6], 20.00th=[    7],
     | 30.00th=[    8], 40.00th=[    9], 50.00th=[    9], 60.00th=[   10],
     | 70.00th=[   10], 80.00th=[   12], 90.00th=[   15], 95.00th=[   25],
     | 99.00th=[ 5145], 99.50th=[ 5800], 99.90th=[ 7046], 99.95th=[ 7046],
     | 99.99th=[ 7046]
  write: IOPS=2456, BW=9827KiB/s (10.1MB/s)(1592KiB/162msec)
    clat (usec): min=90, max=21718, avg=263.96, stdev=1136.79
     lat (usec): min=90, max=21718, avg=264.62, stdev=1136.79
    clat percentiles (usec):
     |  1.00th=[   97],  5.00th=[  125], 10.00th=[  135], 20.00th=[  143],
     | 30.00th=[  151], 40.00th=[  161], 50.00th=[  172], 60.00th=[  184],
     | 70.00th=[  196], 80.00th=[  208], 90.00th=[  235], 95.00th=[  262],
     | 99.00th=[ 1745], 99.50th=[ 6652], 99.90th=[21627], 99.95th=[21627],
     | 99.99th=[21627]
  lat (usec)   : 10=34.37%, 20=10.55%, 50=1.30%, 100=0.91%, 250=47.92%
  lat (usec)   : 500=1.82%, 1000=0.52%
  lat (msec)   : 2=1.17%, 4=0.65%, 10=0.65%, 50=0.13%
  cpu          : usr=2.48%, sys=52.17%, ctx=117, majf=0, minf=33
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17481: Wed Jan 15 19:32:49 2025
   read: IOPS=4174, BW=16.3MiB/s (17.1MB/s)(1436KiB/86msec)
    clat (usec): min=3, max=5558, avg=40.10, stdev=371.74
     lat (usec): min=3, max=5559, avg=40.54, stdev=371.77
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    4], 10.00th=[    5], 20.00th=[    5],
     | 30.00th=[    6], 40.00th=[    7], 50.00th=[    7], 60.00th=[    8],
     | 70.00th=[    9], 80.00th=[   10], 90.00th=[   12], 95.00th=[   14],
     | 99.00th=[   29], 99.50th=[ 3163], 99.90th=[ 5538], 99.95th=[ 5538],
     | 99.99th=[ 5538]
  write: IOPS=4755, BW=18.6MiB/s (19.5MB/s)(1636KiB/86msec)
    clat (usec): min=76, max=1778, avg=150.04, stdev=93.88
     lat (usec): min=76, max=1779, avg=150.56, stdev=93.93
    clat percentiles (usec):
     |  1.00th=[   79],  5.00th=[   88], 10.00th=[   94], 20.00th=[  104],
     | 30.00th=[  114], 40.00th=[  124], 50.00th=[  139], 60.00th=[  149],
     | 70.00th=[  163], 80.00th=[  188], 90.00th=[  219], 95.00th=[  237],
     | 99.00th=[  306], 99.50th=[  322], 99.90th=[ 1778], 99.95th=[ 1778],
     | 99.99th=[ 1778]
  lat (usec)   : 4=2.47%, 10=36.59%, 20=6.64%, 50=0.65%, 100=7.94%
  lat (usec)   : 250=43.75%, 500=1.43%
  lat (msec)   : 2=0.13%, 4=0.26%, 10=0.13%
  cpu          : usr=0.00%, sys=81.18%, ctx=37, majf=0, minf=33
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17482: Wed Jan 15 19:32:49 2025
   read: IOPS=4159, BW=16.2MiB/s (17.0MB/s)(1564KiB/94msec)
    clat (usec): min=4, max=7037, avg=26.57, stdev=355.46
     lat (usec): min=5, max=7037, avg=27.11, stdev=355.45
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    6], 10.00th=[    6], 20.00th=[    7],
     | 30.00th=[    7], 40.00th=[    8], 50.00th=[    9], 60.00th=[   10],
     | 70.00th=[   10], 80.00th=[   11], 90.00th=[   13], 95.00th=[   14],
     | 99.00th=[   22], 99.50th=[   25], 99.90th=[ 7046], 99.95th=[ 7046],
     | 99.99th=[ 7046]
  write: IOPS=4010, BW=15.7MiB/s (16.4MB/s)(1508KiB/94msec)
    clat (usec): min=120, max=4047, avg=194.01, stdev=238.98
     lat (usec): min=120, max=4048, avg=194.74, stdev=239.00
    clat percentiles (usec):
     |  1.00th=[  124],  5.00th=[  131], 10.00th=[  137], 20.00th=[  143],
     | 30.00th=[  153], 40.00th=[  161], 50.00th=[  172], 60.00th=[  184],
     | 70.00th=[  192], 80.00th=[  208], 90.00th=[  231], 95.00th=[  245],
     | 99.00th=[  281], 99.50th=[ 2638], 99.90th=[ 4047], 99.95th=[ 4047],
     | 99.99th=[ 4047]
  lat (usec)   : 10=39.45%, 20=10.94%, 50=0.39%, 250=47.01%, 500=1.82%
  lat (msec)   : 4=0.13%, 10=0.26%
  cpu          : usr=2.15%, sys=77.42%, ctx=85, majf=0, minf=33
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17483: Wed Jan 15 19:32:49 2025
   read: IOPS=1885, BW=7540KiB/s (7721kB/s)(1508KiB/200msec)
    clat (usec): min=4, max=5937, avg=163.42, stdev=727.42
     lat (usec): min=4, max=5938, avg=164.15, stdev=727.77
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    6], 10.00th=[    7], 20.00th=[    8],
     | 30.00th=[    8], 40.00th=[    9], 50.00th=[   10], 60.00th=[   11],
     | 70.00th=[   12], 80.00th=[   13], 90.00th=[   17], 95.00th=[  227],
     | 99.00th=[ 3785], 99.50th=[ 5538], 99.90th=[ 5932], 99.95th=[ 5932],
     | 99.99th=[ 5932]
  write: IOPS=1955, BW=7820KiB/s (8008kB/s)(1564KiB/200msec)
    clat (usec): min=91, max=20961, avg=323.74, stdev=1135.01
     lat (usec): min=92, max=20962, avg=324.34, stdev=1135.03
    clat percentiles (usec):
     |  1.00th=[   95],  5.00th=[  115], 10.00th=[  141], 20.00th=[  159],
     | 30.00th=[  165], 40.00th=[  176], 50.00th=[  186], 60.00th=[  196],
     | 70.00th=[  208], 80.00th=[  229], 90.00th=[  269], 95.00th=[  351],
     | 99.00th=[ 2933], 99.50th=[ 4817], 99.90th=[20841], 99.95th=[20841],
     | 99.99th=[20841]
  lat (usec)   : 10=24.87%, 20=20.18%, 50=1.30%, 100=1.17%, 250=43.10%
  lat (usec)   : 500=4.56%, 750=0.13%, 1000=0.26%
  lat (msec)   : 2=0.91%, 4=2.86%, 10=0.52%, 50=0.13%
  cpu          : usr=3.02%, sys=44.72%, ctx=165, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=29.2MiB/s (30.7MB/s), 7540KiB/s-16.3MiB/s (7721kB/s-17.1MB/s), io=5988KiB (6132kB), run=86-200msec
  WRITE: bw=30.8MiB/s (32.3MB/s), 7820KiB/s-18.6MiB/s (8008kB/s-19.5MB/s), io=6300KiB (6451kB), run=86-200msec

rand-rw: (groupid=0, jobs=1): err= 0: pid=17485: Wed Jan 15 19:32:50 2025
   read: IOPS=255, BW=1021KiB/s (1046kB/s)(1480KiB/1449msec)
    clat (usec): min=607, max=26438, avg=1428.94, stdev=1559.03
     lat (usec): min=607, max=26439, avg=1429.57, stdev=1559.04
    clat percentiles (usec):
     |  1.00th=[  627],  5.00th=[  717], 10.00th=[  758], 20.00th=[  816],
     | 30.00th=[  889], 40.00th=[  947], 50.00th=[ 1074], 60.00th=[ 1450],
     | 70.00th=[ 1647], 80.00th=[ 1811], 90.00th=[ 1991], 95.00th=[ 2212],
     | 99.00th=[ 5145], 99.50th=[10028], 99.90th=[26346], 99.95th=[26346],
     | 99.99th=[26346]
   bw (  KiB/s): min=  880, max= 1088, per=23.90%, avg=984.00, stdev=147.08, samples=2
   iops        : min=  220, max=  272, avg=246.00, stdev=36.77, samples=2
  write: IOPS=274, BW=1099KiB/s (1125kB/s)(1592KiB/1449msec)
    clat (usec): min=1182, max=5081, avg=2293.70, stdev=597.38
     lat (usec): min=1183, max=5082, avg=2294.50, stdev=597.43
    clat percentiles (usec):
     |  1.00th=[ 1287],  5.00th=[ 1401], 10.00th=[ 1663], 20.00th=[ 1827],
     | 30.00th=[ 1942], 40.00th=[ 2057], 50.00th=[ 2180], 60.00th=[ 2311],
     | 70.00th=[ 2540], 80.00th=[ 2802], 90.00th=[ 3130], 95.00th=[ 3326],
     | 99.00th=[ 4228], 99.50th=[ 4621], 99.90th=[ 5080], 99.95th=[ 5080],
     | 99.99th=[ 5080]
   bw (  KiB/s): min= 1008, max= 1176, per=25.21%, avg=1092.00, stdev=118.79, samples=2
   iops        : min=  252, max=  294, avg=273.00, stdev=29.70, samples=2
  lat (usec)   : 750=3.65%, 1000=17.45%
  lat (msec)   : 2=40.76%, 4=36.85%, 10=1.04%, 20=0.13%, 50=0.13%
  cpu          : usr=0.62%, sys=7.39%, ctx=943, majf=0, minf=34
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17486: Wed Jan 15 19:32:50 2025
   read: IOPS=246, BW=988KiB/s (1011kB/s)(1436KiB/1454msec)
    clat (usec): min=576, max=26810, avg=1382.92, stdev=1450.09
     lat (usec): min=576, max=26811, avg=1383.59, stdev=1450.19
    clat percentiles (usec):
     |  1.00th=[  619],  5.00th=[  701], 10.00th=[  766], 20.00th=[  824],
     | 30.00th=[  881], 40.00th=[  947], 50.00th=[ 1106], 60.00th=[ 1500],
     | 70.00th=[ 1680], 80.00th=[ 1827], 90.00th=[ 2008], 95.00th=[ 2212],
     | 99.00th=[ 2671], 99.50th=[ 4113], 99.90th=[26870], 99.95th=[26870],
     | 99.99th=[26870]
   bw (  KiB/s): min=  928, max= 1040, per=23.90%, avg=984.00, stdev=79.20, samples=2
   iops        : min=  232, max=  260, avg=246.00, stdev=19.80, samples=2
  write: IOPS=281, BW=1125KiB/s (1152kB/s)(1636KiB/1454msec)
    clat (usec): min=1174, max=5801, avg=2320.95, stdev=662.94
     lat (usec): min=1174, max=5801, avg=2321.73, stdev=663.01
    clat percentiles (usec):
     |  1.00th=[ 1270],  5.00th=[ 1401], 10.00th=[ 1614], 20.00th=[ 1827],
     | 30.00th=[ 1942], 40.00th=[ 2040], 50.00th=[ 2180], 60.00th=[ 2311],
     | 70.00th=[ 2606], 80.00th=[ 2868], 90.00th=[ 3163], 95.00th=[ 3425],
     | 99.00th=[ 4359], 99.50th=[ 5211], 99.90th=[ 5800], 99.95th=[ 5800],
     | 99.99th=[ 5800]
   bw (  KiB/s): min= 1080, max= 1120, per=25.39%, avg=1100.00, stdev=28.28, samples=2
   iops        : min=  270, max=  280, avg=275.00, stdev= 7.07, samples=2
  lat (usec)   : 750=4.43%, 1000=16.54%
  lat (msec)   : 2=39.58%, 4=38.41%, 10=0.91%, 50=0.13%
  cpu          : usr=0.48%, sys=7.57%, ctx=953, majf=0, minf=33
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17487: Wed Jan 15 19:32:50 2025
   read: IOPS=277, BW=1112KiB/s (1138kB/s)(1564KiB/1407msec)
    clat (usec): min=571, max=4446, avg=1307.71, stdev=574.34
     lat (usec): min=571, max=4446, avg=1308.39, stdev=574.31
    clat percentiles (usec):
     |  1.00th=[  603],  5.00th=[  725], 10.00th=[  766], 20.00th=[  824],
     | 30.00th=[  873], 40.00th=[  938], 50.00th=[ 1037], 60.00th=[ 1434],
     | 70.00th=[ 1647], 80.00th=[ 1827], 90.00th=[ 2024], 95.00th=[ 2278],
     | 99.00th=[ 3359], 99.50th=[ 3720], 99.90th=[ 4424], 99.95th=[ 4424],
     | 99.99th=[ 4424]
   bw (  KiB/s): min= 1080, max= 1144, per=27.00%, avg=1112.00, stdev=45.25, samples=2
   iops        : min=  270, max=  286, avg=278.00, stdev=11.31, samples=2
  write: IOPS=267, BW=1072KiB/s (1098kB/s)(1508KiB/1407msec)
    clat (usec): min=1172, max=32167, avg=2353.18, stdev=1672.17
     lat (usec): min=1172, max=32168, avg=2354.00, stdev=1672.20
    clat percentiles (usec):
     |  1.00th=[ 1254],  5.00th=[ 1401], 10.00th=[ 1631], 20.00th=[ 1860],
     | 30.00th=[ 1975], 40.00th=[ 2057], 50.00th=[ 2147], 60.00th=[ 2245],
     | 70.00th=[ 2409], 80.00th=[ 2769], 90.00th=[ 3064], 95.00th=[ 3294],
     | 99.00th=[ 5276], 99.50th=[ 7832], 99.90th=[32113], 99.95th=[32113],
     | 99.99th=[32113]
   bw (  KiB/s): min= 1056, max= 1096, per=24.84%, avg=1076.00, stdev=28.28, samples=2
   iops        : min=  264, max=  274, avg=269.00, stdev= 7.07, samples=2
  lat (usec)   : 750=4.17%, 1000=19.40%
  lat (msec)   : 2=38.02%, 4=37.63%, 10=0.65%, 50=0.13%
  cpu          : usr=0.57%, sys=7.68%, ctx=951, majf=0, minf=32
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17488: Wed Jan 15 19:32:50 2025
   read: IOPS=265, BW=1063KiB/s (1089kB/s)(1508KiB/1418msec)
    clat (usec): min=601, max=4920, avg=1318.52, stdev=633.64
     lat (usec): min=601, max=4920, avg=1319.14, stdev=633.65
    clat percentiles (usec):
     |  1.00th=[  635],  5.00th=[  709], 10.00th=[  742], 20.00th=[  807],
     | 30.00th=[  865], 40.00th=[  947], 50.00th=[ 1037], 60.00th=[ 1401],
     | 70.00th=[ 1614], 80.00th=[ 1811], 90.00th=[ 2040], 95.00th=[ 2245],
     | 99.00th=[ 4146], 99.50th=[ 4490], 99.90th=[ 4948], 99.95th=[ 4948],
     | 99.99th=[ 4948]
   bw (  KiB/s): min=  984, max= 1104, per=25.35%, avg=1044.00, stdev=84.85, samples=2
   iops        : min=  246, max=  276, avg=261.00, stdev=21.21, samples=2
  write: IOPS=275, BW=1103KiB/s (1129kB/s)(1564KiB/1418msec)
    clat (usec): min=1258, max=33147, avg=2336.08, stdev=1665.82
     lat (usec): min=1259, max=33148, avg=2336.87, stdev=1665.84
    clat percentiles (usec):
     |  1.00th=[ 1303],  5.00th=[ 1450], 10.00th=[ 1614], 20.00th=[ 1827],
     | 30.00th=[ 1926], 40.00th=[ 2024], 50.00th=[ 2114], 60.00th=[ 2212],
     | 70.00th=[ 2474], 80.00th=[ 2769], 90.00th=[ 3097], 95.00th=[ 3294],
     | 99.00th=[ 4359], 99.50th=[ 4883], 99.90th=[33162], 99.95th=[33162],
     | 99.99th=[33162]
   bw (  KiB/s): min= 1064, max= 1168, per=25.76%, avg=1116.00, stdev=73.54, samples=2
   iops        : min=  266, max=  292, avg=279.00, stdev=18.38, samples=2
  lat (usec)   : 750=5.47%, 1000=17.58%
  lat (msec)   : 2=39.58%, 4=36.20%, 10=1.04%, 50=0.13%
  cpu          : usr=0.78%, sys=7.34%, ctx=928, majf=0, minf=32
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=4118KiB/s (4217kB/s), 988KiB/s-1112KiB/s (1011kB/s-1138kB/s), io=5988KiB (6132kB), run=1407-1454msec
  WRITE: bw=4333KiB/s (4437kB/s), 1072KiB/s-1125KiB/s (1098kB/s-1152kB/s), io=6300KiB (6451kB), run=1407-1454msec