== sanity test 398b: DIO and buffer IO race ============== 11:10:17 (1761318617) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.177837 s, 70.8 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148410: Fri Oct 24 11:10:31 2025 read: IOPS=52, BW=209KiB/s (214kB/s)(1480KiB/7082msec) clat (usec): min=6, max=85101, avg=11330.95, stdev=13204.31 lat (usec): min=6, max=85102, avg=11331.93, stdev=13204.43 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 13], 10.00th=[ 18], 20.00th=[ 26], | 30.00th=[ 39], 40.00th=[ 5604], 50.00th=[ 8979], 60.00th=[12649], | 70.00th=[15795], 80.00th=[19530], 90.00th=[25822], 95.00th=[32900], | 99.00th=[63177], 99.50th=[78119], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 98, max= 272, per=24.03%, avg=200.92, stdev=52.16, samples=13 iops : min= 24, max= 68, avg=49.62, stdev=13.09, samples=13 write: IOPS=56, BW=225KiB/s (230kB/s)(1592KiB/7082msec); 0 zone resets clat (usec): min=161, max=69244, avg=7107.64, stdev=9448.66 lat (usec): min=161, max=69244, avg=7108.51, stdev=9448.69 clat percentiles (usec): | 1.00th=[ 174], 5.00th=[ 198], 10.00th=[ 219], 20.00th=[ 249], | 30.00th=[ 318], 40.00th=[ 627], 50.00th=[ 3720], 60.00th=[ 5866], | 70.00th=[ 8979], 80.00th=[12518], 90.00th=[19792], 95.00th=[28705], | 99.00th=[38536], 99.50th=[42206], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 120, max= 357, per=24.96%, avg=219.62, stdev=73.48, samples=13 iops : min= 30, max= 89, avg=54.38, stdev=18.33, samples=13 lat (usec) : 10=0.91%, 20=5.60%, 50=10.16%, 100=0.78%, 250=10.94% lat (usec) : 500=9.24%, 750=2.21%, 1000=0.26% lat (msec) : 2=1.69%, 4=3.26%, 10=18.10%, 20=22.66%, 50=13.02% lat (msec) : 100=1.17% cpu : usr=0.16%, sys=5.03%, ctx=858, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148411: Fri Oct 24 11:10:31 2025 read: IOPS=50, BW=203KiB/s (208kB/s)(1436KiB/7063msec) clat (usec): min=6, max=106100, avg=10662.01, stdev=12673.12 lat (usec): min=6, max=106101, avg=10665.66, stdev=12672.45 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 14], 10.00th=[ 19], 20.00th=[ 26], | 30.00th=[ 33], 40.00th=[ 5145], 50.00th=[ 8979], 60.00th=[ 11994], | 70.00th=[ 15139], 80.00th=[ 19268], 90.00th=[ 23987], 95.00th=[ 29230], | 99.00th=[ 63701], 99.50th=[ 83362], 99.90th=[106431], 99.95th=[106431], | 99.99th=[106431] bw ( KiB/s): min= 70, max= 365, per=24.35%, avg=203.54, stdev=76.92, samples=13 iops : min= 17, max= 91, avg=50.31, stdev=19.31, samples=13 write: IOPS=57, BW=232KiB/s (237kB/s)(1636KiB/7063msec); 0 zone resets clat (usec): min=143, max=61826, avg=7767.70, stdev=9557.56 lat (usec): min=144, max=61827, avg=7768.63, stdev=9557.58 clat percentiles (usec): | 1.00th=[ 167], 5.00th=[ 200], 10.00th=[ 217], 20.00th=[ 258], | 30.00th=[ 334], 40.00th=[ 1090], 50.00th=[ 4817], 60.00th=[ 6718], | 70.00th=[ 9503], 80.00th=[15008], 90.00th=[20841], 95.00th=[28705], | 99.00th=[36439], 99.50th=[38536], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 111, max= 327, per=25.94%, avg=228.23, stdev=61.57, samples=13 iops : min= 27, max= 81, avg=56.46, stdev=15.38, samples=13 lat (usec) : 10=1.30%, 20=4.95%, 50=10.42%, 100=0.65%, 250=10.03% lat (usec) : 500=9.77%, 750=0.78%, 1000=0.91% lat (msec) : 2=2.08%, 4=2.34%, 10=19.01%, 20=23.96%, 50=13.02% lat (msec) : 100=0.65%, 250=0.13% cpu : usr=0.08%, sys=5.25%, ctx=830, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148412: Fri Oct 24 11:10:31 2025 read: IOPS=54, BW=218KiB/s (224kB/s)(1564KiB/7159msec) clat (usec): min=6, max=101887, avg=10264.37, stdev=11612.99 lat (usec): min=6, max=101887, avg=10265.29, stdev=11612.98 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 13], 10.00th=[ 19], 20.00th=[ 26], | 30.00th=[ 31], 40.00th=[ 4359], 50.00th=[ 8356], 60.00th=[ 12256], | 70.00th=[ 15401], 80.00th=[ 18744], 90.00th=[ 23462], 95.00th=[ 29492], | 99.00th=[ 42206], 99.50th=[ 79168], 99.90th=[102237], 99.95th=[102237], | 99.99th=[102237] bw ( KiB/s): min= 63, max= 343, per=26.39%, avg=220.64, stdev=83.22, samples=14 iops : min= 15, max= 85, avg=54.57, stdev=20.91, samples=14 write: IOPS=52, BW=211KiB/s (216kB/s)(1508KiB/7159msec); 0 zone resets clat (usec): min=169, max=168886, avg=8185.97, stdev=12878.40 lat (usec): min=170, max=168888, avg=8187.95, stdev=12877.87 clat percentiles (usec): | 1.00th=[ 180], 5.00th=[ 202], 10.00th=[ 235], 20.00th=[ 285], | 30.00th=[ 375], 40.00th=[ 947], 50.00th=[ 4490], 60.00th=[ 6915], | 70.00th=[ 9372], 80.00th=[ 14877], 90.00th=[ 21365], 95.00th=[ 28181], | 99.00th=[ 52167], 99.50th=[ 54264], 99.90th=[168821], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 77, max= 384, per=23.75%, avg=209.00, stdev=79.46, samples=14 iops : min= 19, max= 96, avg=51.71, stdev=19.94, samples=14 lat (usec) : 10=1.30%, 20=4.95%, 50=12.37%, 100=0.39%, 250=6.38% lat (usec) : 500=11.33%, 750=1.17%, 1000=1.17% lat (msec) : 2=1.56%, 4=3.26%, 10=18.88%, 20=23.05%, 50=13.41% lat (msec) : 100=0.52%, 250=0.26% cpu : usr=0.20%, sys=5.10%, ctx=850, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148413: Fri Oct 24 11:10:31 2025 read: IOPS=54, BW=216KiB/s (221kB/s)(1508KiB/6979msec) clat (usec): min=6, max=112983, avg=10837.39, stdev=13854.93 lat (usec): min=6, max=112984, avg=10838.33, stdev=13855.09 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 13], 10.00th=[ 17], 20.00th=[ 25], | 30.00th=[ 32], 40.00th=[ 167], 50.00th=[ 8225], 60.00th=[ 12649], | 70.00th=[ 15139], 80.00th=[ 18482], 90.00th=[ 24511], 95.00th=[ 31327], | 99.00th=[ 67634], 99.50th=[103285], 99.90th=[112722], 99.95th=[112722], | 99.99th=[112722] bw ( KiB/s): min= 91, max= 310, per=25.51%, avg=213.23, stdev=61.36, samples=13 iops : min= 22, max= 77, avg=52.69, stdev=15.41, samples=13 write: IOPS=56, BW=224KiB/s (229kB/s)(1564KiB/6979msec); 0 zone resets clat (usec): min=163, max=42193, avg=7298.32, stdev=9470.04 lat (usec): min=163, max=42193, avg=7299.24, stdev=9470.06 clat percentiles (usec): | 1.00th=[ 176], 5.00th=[ 200], 10.00th=[ 217], 20.00th=[ 265], | 30.00th=[ 326], 40.00th=[ 766], 50.00th=[ 4146], 60.00th=[ 6456], | 70.00th=[ 8979], 80.00th=[12649], 90.00th=[20579], 95.00th=[29492], | 99.00th=[40109], 99.50th=[41681], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 98, max= 341, per=25.40%, avg=223.54, stdev=77.68, samples=13 iops : min= 24, max= 85, avg=55.31, stdev=19.55, samples=13 lat (usec) : 10=1.30%, 20=5.99%, 50=11.46%, 100=0.65%, 250=8.98% lat (usec) : 500=10.16%, 750=1.56%, 1000=0.91% lat (msec) : 2=1.95%, 4=2.34%, 10=18.36%, 20=22.79%, 50=12.76% lat (msec) : 100=0.52%, 250=0.26% cpu : usr=0.17%, sys=5.03%, ctx=816, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=836KiB/s (857kB/s), 203KiB/s-218KiB/s (208kB/s-224kB/s), io=5988KiB (6132kB), run=6979-7159msec WRITE: bw=880KiB/s (901kB/s), 211KiB/s-232KiB/s (216kB/s-237kB/s), io=6300KiB (6451kB), run=6979-7159msec rand-rw: (groupid=0, jobs=1): err= 0: pid=148414: Fri Oct 24 11:10:38 2025 read: IOPS=27, BW=110KiB/s (112kB/s)(1480KiB/13502msec) clat (msec): min=2, max=117, avg=11.97, stdev=11.10 lat (msec): min=2, max=117, avg=11.97, stdev=11.10 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 15], 90.00th=[ 26], 95.00th=[ 35], | 99.00th=[ 53], 99.50th=[ 59], 99.90th=[ 118], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 39, max= 174, per=23.97%, avg=106.19, stdev=34.63, samples=26 iops : min= 9, max= 43, avg=26.04, stdev= 8.67, samples=26 write: IOPS=29, BW=118KiB/s (121kB/s)(1592KiB/13502msec); 0 zone resets clat (usec): min=4885, max=84532, avg=22691.23, stdev=10616.21 lat (usec): min=4887, max=84534, avg=22692.87, stdev=10616.14 clat percentiles (usec): | 1.00th=[ 7242], 5.00th=[10159], 10.00th=[12518], 20.00th=[14353], | 30.00th=[16450], 40.00th=[18220], 50.00th=[20055], 60.00th=[22676], | 70.00th=[25035], 80.00th=[29492], 90.00th=[37487], 95.00th=[45351], | 99.00th=[58459], 99.50th=[71828], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 51, max= 166, per=24.98%, avg=116.38, stdev=34.54, samples=26 iops : min= 12, max= 41, avg=28.58, stdev= 8.66, samples=26 lat (msec) : 4=3.26%, 10=27.86%, 20=36.33%, 50=30.86%, 100=1.56% lat (msec) : 250=0.13% cpu : usr=0.15%, sys=1.54%, ctx=819, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148415: Fri Oct 24 11:10:38 2025 read: IOPS=26, BW=107KiB/s (110kB/s)(1436KiB/13418msec) clat (msec): min=2, max=117, avg=11.44, stdev=10.34 lat (msec): min=2, max=117, avg=11.44, stdev=10.34 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 15], 90.00th=[ 25], 95.00th=[ 32], | 99.00th=[ 52], 99.50th=[ 56], 99.90th=[ 118], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 44, max= 182, per=24.05%, avg=106.54, stdev=41.19, samples=26 iops : min= 11, max= 45, avg=26.15, stdev=10.32, samples=26 write: IOPS=30, BW=122KiB/s (125kB/s)(1636KiB/13418msec); 0 zone resets clat (usec): min=6014, max=79011, avg=22670.66, stdev=10491.30 lat (usec): min=6016, max=79012, avg=22672.29, stdev=10491.52 clat percentiles (usec): | 1.00th=[ 8848], 5.00th=[10945], 10.00th=[12649], 20.00th=[14877], | 30.00th=[16712], 40.00th=[18744], 50.00th=[20317], 60.00th=[22676], | 70.00th=[24773], 80.00th=[27657], 90.00th=[36439], 95.00th=[45351], | 99.00th=[58983], 99.50th=[63177], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 71, max= 191, per=25.69%, avg=119.69, stdev=31.75, samples=26 iops : min= 17, max= 47, avg=29.42, stdev= 7.94, samples=26 lat (msec) : 4=2.21%, 10=27.60%, 20=37.11%, 50=30.73%, 100=2.21% lat (msec) : 250=0.13% cpu : usr=0.20%, sys=1.48%, ctx=823, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148416: Fri Oct 24 11:10:38 2025 read: IOPS=29, BW=116KiB/s (119kB/s)(1564KiB/13437msec) clat (usec): min=2576, max=71250, avg=11911.55, stdev=9895.95 lat (usec): min=2577, max=71252, avg=11914.65, stdev=9894.78 clat percentiles (usec): | 1.00th=[ 3425], 5.00th=[ 4424], 10.00th=[ 5014], 20.00th=[ 5997], | 30.00th=[ 6652], 40.00th=[ 7373], 50.00th=[ 8586], 60.00th=[ 9896], | 70.00th=[11469], 80.00th=[14877], 90.00th=[23462], 95.00th=[35390], | 99.00th=[53216], 99.50th=[57934], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 43, max= 216, per=26.14%, avg=115.81, stdev=44.22, samples=26 iops : min= 10, max= 54, avg=28.42, stdev=11.11, samples=26 write: IOPS=28, BW=112KiB/s (115kB/s)(1508KiB/13437msec); 0 zone resets clat (msec): min=4, max=129, avg=23.17, stdev=12.72 lat (msec): min=4, max=129, avg=23.17, stdev=12.72 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 23], | 70.00th=[ 26], 80.00th=[ 29], 90.00th=[ 36], 95.00th=[ 47], | 99.00th=[ 68], 99.50th=[ 112], 99.90th=[ 130], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 43, max= 174, per=23.57%, avg=109.85, stdev=35.44, samples=26 iops : min= 10, max= 43, avg=26.92, stdev= 8.90, samples=26 lat (msec) : 4=2.21%, 10=31.38%, 20=34.24%, 50=30.08%, 100=1.82% lat (msec) : 250=0.26% cpu : usr=0.19%, sys=1.51%, ctx=816, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148417: Fri Oct 24 11:10:38 2025 read: IOPS=28, BW=114KiB/s (116kB/s)(1508KiB/13261msec) clat (msec): min=2, max=102, avg=11.00, stdev=10.32 lat (msec): min=2, max=102, avg=11.00, stdev=10.32 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 23], 95.00th=[ 34], | 99.00th=[ 53], 99.50th=[ 57], 99.90th=[ 103], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min= 39, max= 207, per=25.39%, avg=112.46, stdev=39.85, samples=26 iops : min= 9, max= 51, avg=27.58, stdev=10.04, samples=26 write: IOPS=29, BW=118KiB/s (121kB/s)(1564KiB/13261msec); 0 zone resets clat (usec): min=5299, max=97288, avg=23188.07, stdev=11418.38 lat (usec): min=5300, max=97292, avg=23189.81, stdev=11418.31 clat percentiles (usec): | 1.00th=[ 8291], 5.00th=[10552], 10.00th=[13173], 20.00th=[15139], | 30.00th=[16909], 40.00th=[19006], 50.00th=[20579], 60.00th=[22152], | 70.00th=[25560], 80.00th=[28443], 90.00th=[35914], 95.00th=[47973], | 99.00th=[66847], 99.50th=[73925], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 55, max= 175, per=25.23%, avg=117.58, stdev=38.41, samples=26 iops : min= 13, max= 43, avg=28.88, stdev= 9.61, samples=26 lat (msec) : 4=3.26%, 10=32.16%, 20=32.42%, 50=29.56%, 100=2.47% lat (msec) : 250=0.13% cpu : usr=0.24%, sys=1.52%, ctx=828, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=443KiB/s (454kB/s), 107KiB/s-116KiB/s (110kB/s-119kB/s), io=5988KiB (6132kB), run=13261-13502msec WRITE: bw=467KiB/s (478kB/s), 112KiB/s-122KiB/s (115kB/s-125kB/s), io=6300KiB (6451kB), run=13261-13502msec