== sanity test 398b: DIO and buffer IO race ============== 04:54:07 (1761123247) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.207407 s, 60.7 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=122327: Wed Oct 22 04:54:24 2025 read: IOPS=46, BW=186KiB/s (191kB/s)(1480KiB/7947msec) clat (usec): min=8, max=88965, avg=7212.97, stdev=9907.45 lat (usec): min=8, max=88967, avg=7213.73, stdev=9907.54 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 17], 20.00th=[ 23], | 30.00th=[ 31], 40.00th=[ 50], 50.00th=[ 3392], 60.00th=[ 7898], | 70.00th=[11469], 80.00th=[14091], 90.00th=[17957], 95.00th=[22152], | 99.00th=[43779], 99.50th=[56886], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 88, max= 272, per=24.68%, avg=185.13, stdev=62.01, samples=15 iops : min= 22, max= 68, avg=45.87, stdev=15.58, samples=15 write: IOPS=50, BW=200KiB/s (205kB/s)(1592KiB/7947msec); 0 zone resets clat (usec): min=159, max=135876, avg=13120.30, stdev=17338.31 lat (usec): min=160, max=135876, avg=13121.18, stdev=17338.31 clat percentiles (usec): | 1.00th=[ 182], 5.00th=[ 221], 10.00th=[ 262], 20.00th=[ 420], | 30.00th=[ 873], 40.00th=[ 3884], 50.00th=[ 7177], 60.00th=[ 10683], | 70.00th=[ 15008], 80.00th=[ 21890], 90.00th=[ 35914], 95.00th=[ 51119], | 99.00th=[ 70779], 99.50th=[ 80217], 99.90th=[135267], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 143, max= 272, per=25.65%, avg=202.40, stdev=38.91, samples=15 iops : min= 35, max= 68, avg=50.20, stdev= 9.83, samples=15 lat (usec) : 10=0.78%, 20=6.12%, 50=12.37%, 100=2.60%, 250=5.34% lat (usec) : 500=8.59%, 750=2.21%, 1000=0.91% lat (msec) : 2=2.60%, 4=4.04%, 10=16.67%, 20=22.92%, 50=11.59% lat (msec) : 100=3.12%, 250=0.13% cpu : usr=0.24%, sys=5.01%, ctx=943, majf=0, minf=18 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=122328: Wed Oct 22 04:54:24 2025 read: IOPS=45, BW=180KiB/s (184kB/s)(1436KiB/7976msec) clat (usec): min=6, max=77843, avg=7457.44, stdev=10257.16 lat (usec): min=6, max=77844, avg=7458.42, stdev=10257.03 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 13], 10.00th=[ 17], 20.00th=[ 26], | 30.00th=[ 33], 40.00th=[ 50], 50.00th=[ 4490], 60.00th=[ 7635], | 70.00th=[10552], 80.00th=[14746], 90.00th=[19530], 95.00th=[23200], | 99.00th=[59507], 99.50th=[68682], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 95, max= 256, per=23.44%, avg=175.80, stdev=49.99, samples=15 iops : min= 23, max= 64, avg=43.53, stdev=12.63, samples=15 write: IOPS=51, BW=205KiB/s (210kB/s)(1636KiB/7976msec); 0 zone resets clat (usec): min=164, max=116773, avg=12826.43, stdev=17209.72 lat (usec): min=164, max=116774, avg=12827.33, stdev=17209.76 clat percentiles (usec): | 1.00th=[ 190], 5.00th=[ 229], 10.00th=[ 269], 20.00th=[ 371], | 30.00th=[ 734], 40.00th=[ 3261], 50.00th=[ 6587], 60.00th=[ 10552], | 70.00th=[ 15139], 80.00th=[ 22676], 90.00th=[ 35914], 95.00th=[ 45351], | 99.00th=[ 77071], 99.50th=[ 94897], 99.90th=[116917], 99.95th=[116917], | 99.99th=[116917] bw ( KiB/s): min= 103, max= 312, per=24.97%, avg=197.00, stdev=50.61, samples=15 iops : min= 25, max= 78, avg=48.87, stdev=12.77, samples=15 lat (usec) : 10=0.52%, 20=5.99%, 50=12.11%, 100=1.82%, 250=4.95% lat (usec) : 500=10.16%, 750=2.73%, 1000=0.78% lat (msec) : 2=2.60%, 4=4.04%, 10=17.71%, 20=19.79%, 50=14.06% lat (msec) : 100=2.60%, 250=0.13% cpu : usr=0.24%, sys=5.28%, ctx=944, majf=0, minf=17 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=122329: Wed Oct 22 04:54:24 2025 read: IOPS=51, BW=206KiB/s (211kB/s)(1564KiB/7599msec) clat (usec): min=6, max=78965, avg=7877.49, stdev=10636.63 lat (usec): min=6, max=78965, avg=7878.22, stdev=10636.62 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 14], 10.00th=[ 18], 20.00th=[ 25], | 30.00th=[ 36], 40.00th=[ 63], 50.00th=[ 4883], 60.00th=[ 8160], | 70.00th=[12256], 80.00th=[14484], 90.00th=[19268], 95.00th=[26870], | 99.00th=[59507], 99.50th=[66847], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 104, max= 351, per=27.46%, avg=205.93, stdev=64.32, samples=15 iops : min= 26, max= 87, avg=51.07, stdev=16.06, samples=15 write: IOPS=49, BW=198KiB/s (203kB/s)(1508KiB/7599msec); 0 zone resets clat (usec): min=117, max=129915, avg=11908.57, stdev=15986.57 lat (usec): min=118, max=129917, avg=11909.58, stdev=15986.56 clat percentiles (usec): | 1.00th=[ 180], 5.00th=[ 223], 10.00th=[ 258], 20.00th=[ 343], | 30.00th=[ 586], 40.00th=[ 3032], 50.00th=[ 5538], 60.00th=[ 9896], | 70.00th=[ 13566], 80.00th=[ 21103], 90.00th=[ 34341], 95.00th=[ 42730], | 99.00th=[ 63177], 99.50th=[ 85459], 99.90th=[129500], 99.95th=[129500], | 99.99th=[129500] bw ( KiB/s): min= 127, max= 296, per=25.00%, avg=197.27, stdev=49.55, samples=15 iops : min= 31, max= 74, avg=48.87, stdev=12.47, samples=15 lat (usec) : 10=1.43%, 20=5.99%, 50=11.20%, 100=2.86%, 250=5.47% lat (usec) : 500=8.98%, 750=3.52%, 1000=1.30% lat (msec) : 2=1.56%, 4=3.52%, 10=17.06%, 20=22.01%, 50=13.02% lat (msec) : 100=1.95%, 250=0.13% cpu : usr=0.22%, sys=5.75%, ctx=920, majf=0, minf=18 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=122330: Wed Oct 22 04:54:24 2025 read: IOPS=55, BW=221KiB/s (226kB/s)(1508KiB/6836msec) clat (usec): min=5, max=93403, avg=7356.91, stdev=11056.80 lat (usec): min=5, max=93404, avg=7357.59, stdev=11056.86 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 13], 10.00th=[ 17], 20.00th=[ 24], | 30.00th=[ 30], 40.00th=[ 45], 50.00th=[ 603], 60.00th=[ 7308], | 70.00th=[10683], 80.00th=[14353], 90.00th=[19268], 95.00th=[23987], | 99.00th=[47449], 99.50th=[76022], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 143, max= 386, per=28.97%, avg=217.31, stdev=64.79, samples=13 iops : min= 35, max= 96, avg=53.85, stdev=16.29, samples=13 write: IOPS=57, BW=229KiB/s (234kB/s)(1564KiB/6836msec); 0 zone resets clat (usec): min=128, max=74707, avg=10238.23, stdev=14603.27 lat (usec): min=129, max=74709, avg=10239.09, stdev=14603.29 clat percentiles (usec): | 1.00th=[ 163], 5.00th=[ 219], 10.00th=[ 249], 20.00th=[ 347], | 30.00th=[ 553], 40.00th=[ 1106], 50.00th=[ 3261], 60.00th=[ 6718], | 70.00th=[11994], 80.00th=[17957], 90.00th=[30540], 95.00th=[43779], | 99.00th=[67634], 99.50th=[71828], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 136, max= 414, per=29.05%, avg=229.23, stdev=66.97, samples=13 iops : min= 34, max= 103, avg=56.85, stdev=16.70, samples=13 lat (usec) : 10=1.04%, 20=6.64%, 50=12.76%, 100=2.34%, 250=5.86% lat (usec) : 500=9.64%, 750=4.69%, 1000=2.34% lat (msec) : 2=3.39%, 4=4.04%, 10=13.80%, 20=20.44%, 50=10.94% lat (msec) : 100=2.08% cpu : usr=0.31%, sys=5.22%, ctx=855, majf=0, minf=17 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=751KiB/s (769kB/s), 180KiB/s-221KiB/s (184kB/s-226kB/s), io=5988KiB (6132kB), run=6836-7976msec WRITE: bw=790KiB/s (809kB/s), 198KiB/s-229KiB/s (203kB/s-234kB/s), io=6300KiB (6451kB), run=6836-7976msec rand-rw: (groupid=0, jobs=1): err= 0: pid=122332: Wed Oct 22 04:54:48 2025 read: IOPS=11, BW=46.3KiB/s (47.4kB/s)(1480KiB/31978msec) clat (usec): min=2927, max=85001, avg=12101.69, stdev=12100.52 lat (usec): min=2927, max=85002, avg=12103.69, stdev=12100.18 clat percentiles (usec): | 1.00th=[ 3064], 5.00th=[ 3654], 10.00th=[ 4228], 20.00th=[ 5145], | 30.00th=[ 6259], 40.00th=[ 6980], 50.00th=[ 7963], 60.00th=[ 9503], | 70.00th=[11338], 80.00th=[16057], 90.00th=[23200], 95.00th=[30016], | 99.00th=[71828], 99.50th=[84411], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 7, max= 127, per=24.62%, avg=45.30, stdev=20.98, samples=63 iops : min= 1, max= 31, avg=10.83, stdev= 5.34, samples=63 write: IOPS=12, BW=49.8KiB/s (50.0kB/s)(1592KiB/31978msec); 0 zone resets clat (msec): min=31, max=197, avg=69.02, stdev=23.06 lat (msec): min=31, max=197, avg=69.02, stdev=23.06 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 41], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 56], 40.00th=[ 61], 50.00th=[ 65], 60.00th=[ 70], | 70.00th=[ 75], 80.00th=[ 85], 90.00th=[ 101], 95.00th=[ 110], | 99.00th=[ 148], 99.50th=[ 188], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 31, max= 72, per=25.40%, avg=49.02, stdev=10.67, samples=63 iops : min= 7, max= 18, avg=11.75, stdev= 2.74, samples=63 lat (msec) : 4=3.39%, 10=26.69%, 20=11.20%, 50=16.02%, 100=37.63% lat (msec) : 250=5.08% cpu : usr=0.08%, sys=0.98%, ctx=934, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=122333: Wed Oct 22 04:54:48 2025 read: IOPS=11, BW=44.2KiB/s (45.2kB/s)(1436KiB/32510msec) clat (msec): min=2, max=116, avg=11.71, stdev=11.72 lat (msec): min=2, max=116, avg=11.71, stdev=11.72 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 16], 90.00th=[ 22], 95.00th=[ 30], | 99.00th=[ 71], 99.50th=[ 91], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 7, max= 120, per=24.04%, avg=44.24, stdev=23.85, samples=63 iops : min= 1, max= 30, avg=10.56, stdev= 5.97, samples=63 write: IOPS=12, BW=50.3KiB/s (51.5kB/s)(1636KiB/32510msec); 0 zone resets clat (msec): min=27, max=178, avg=69.14, stdev=22.35 lat (msec): min=28, max=178, avg=69.15, stdev=22.35 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 40], 10.00th=[ 44], 20.00th=[ 52], | 30.00th=[ 56], 40.00th=[ 61], 50.00th=[ 67], 60.00th=[ 72], | 70.00th=[ 78], 80.00th=[ 85], 90.00th=[ 100], 95.00th=[ 110], | 99.00th=[ 134], 99.50th=[ 142], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 31, max= 80, per=25.71%, avg=49.62, stdev=10.55, samples=64 iops : min= 7, max= 20, avg=11.91, stdev= 2.74, samples=64 lat (msec) : 4=3.39%, 10=25.13%, 20=12.63%, 50=14.58%, 100=39.06% lat (msec) : 250=5.21% cpu : usr=0.09%, sys=0.97%, ctx=940, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=122334: Wed Oct 22 04:54:48 2025 read: IOPS=12, BW=50.3KiB/s (51.5kB/s)(1564KiB/31088msec) clat (msec): min=2, max=124, avg=12.42, stdev=12.37 lat (msec): min=2, max=124, avg=12.42, stdev=12.37 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 16], 90.00th=[ 24], 95.00th=[ 32], | 99.00th=[ 71], 99.50th=[ 73], 99.90th=[ 125], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 7, max= 143, per=27.39%, avg=50.39, stdev=27.85, samples=61 iops : min= 1, max= 35, avg=12.08, stdev= 6.92, samples=61 write: IOPS=12, BW=48.5KiB/s (49.7kB/s)(1508KiB/31088msec); 0 zone resets clat (msec): min=28, max=163, avg=69.51, stdev=21.60 lat (msec): min=28, max=163, avg=69.51, stdev=21.61 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 40], 10.00th=[ 45], 20.00th=[ 51], | 30.00th=[ 57], 40.00th=[ 63], 50.00th=[ 68], 60.00th=[ 72], | 70.00th=[ 78], 80.00th=[ 85], 90.00th=[ 99], 95.00th=[ 111], | 99.00th=[ 148], 99.50th=[ 155], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 31, max= 64, per=24.81%, avg=47.89, stdev= 8.97, samples=61 iops : min= 7, max= 16, avg=11.46, stdev= 2.32, samples=61 lat (msec) : 4=2.99%, 10=27.21%, 20=13.54%, 50=15.36%, 100=36.07% lat (msec) : 250=4.82% cpu : usr=0.09%, sys=1.06%, ctx=943, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=122335: Wed Oct 22 04:54:48 2025 read: IOPS=11, BW=46.0KiB/s (48.1kB/s)(1508KiB/32090msec) clat (usec): min=2465, max=89350, avg=12648.85, stdev=11688.67 lat (usec): min=2466, max=89350, avg=12650.61, stdev=11689.01 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3949], 10.00th=[ 4555], 20.00th=[ 5669], | 30.00th=[ 6521], 40.00th=[ 7439], 50.00th=[ 8848], 60.00th=[10290], | 70.00th=[12387], 80.00th=[16909], 90.00th=[24773], 95.00th=[35390], | 99.00th=[69731], 99.50th=[69731], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 7, max= 88, per=25.07%, avg=46.13, stdev=21.13, samples=63 iops : min= 1, max= 22, avg=11.03, stdev= 5.35, samples=63 write: IOPS=12, BW=48.7KiB/s (49.9kB/s)(1564KiB/32090msec); 0 zone resets clat (msec): min=28, max=187, avg=69.82, stdev=21.97 lat (msec): min=28, max=187, avg=69.83, stdev=21.97 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 40], 10.00th=[ 45], 20.00th=[ 52], | 30.00th=[ 56], 40.00th=[ 62], 50.00th=[ 68], 60.00th=[ 74], | 70.00th=[ 80], 80.00th=[ 87], 90.00th=[ 101], 95.00th=[ 112], | 99.00th=[ 125], 99.50th=[ 131], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 23, max= 64, per=24.82%, avg=47.90, stdev= 9.15, samples=63 iops : min= 5, max= 16, avg=11.48, stdev= 2.42, samples=63 lat (msec) : 4=2.86%, 10=25.91%, 20=12.76%, 50=15.76%, 100=37.63% lat (msec) : 250=5.08% cpu : usr=0.09%, sys=1.00%, ctx=947, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=184KiB/s (189kB/s), 44.2KiB/s-50.3KiB/s (45.2kB/s-51.5kB/s), io=5988KiB (6132kB), run=31088-32510msec WRITE: bw=194KiB/s (198kB/s), 48.5KiB/s-50.3KiB/s (49.7kB/s-51.5kB/s), io=6300KiB (6451kB), run=31088-32510msec