== sanity test 398b: DIO and buffer IO race ============== 12:41:26 (1776271286) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.282067 s, 44.6 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148374: Wed Apr 15 12:41:42 2026 read: IOPS=51, BW=206KiB/s (211kB/s)(1480KiB/7192msec) clat (usec): min=6, max=90901, avg=10716.08, stdev=12477.85 lat (usec): min=7, max=90901, avg=10716.79, stdev=12477.85 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 14], 10.00th=[ 19], 20.00th=[ 26], | 30.00th=[ 33], 40.00th=[ 3884], 50.00th=[ 8717], 60.00th=[11863], | 70.00th=[15664], 80.00th=[19268], 90.00th=[24511], 95.00th=[30278], | 99.00th=[62653], 99.50th=[67634], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 88, max= 303, per=24.76%, avg=199.07, stdev=67.94, samples=14 iops : min= 22, max= 75, avg=49.36, stdev=16.90, samples=14 write: IOPS=55, BW=221KiB/s (227kB/s)(1592KiB/7192msec); 0 zone resets clat (usec): min=138, max=119983, avg=7957.61, stdev=11726.74 lat (usec): min=139, max=119985, avg=7958.49, stdev=11726.78 clat percentiles (usec): | 1.00th=[ 169], 5.00th=[ 198], 10.00th=[ 215], 20.00th=[ 265], | 30.00th=[ 379], 40.00th=[ 1598], 50.00th=[ 4047], 60.00th=[ 6325], | 70.00th=[ 9503], 80.00th=[ 13960], 90.00th=[ 21365], 95.00th=[ 26870], | 99.00th=[ 52691], 99.50th=[ 88605], 99.90th=[120062], 99.95th=[120062], | 99.99th=[120062] bw ( KiB/s): min= 66, max= 335, per=25.72%, avg=217.57, stdev=69.24, samples=14 iops : min= 16, max= 83, avg=54.00, stdev=17.33, samples=14 lat (usec) : 10=1.04%, 20=4.69%, 50=11.20%, 100=0.78%, 250=8.59% lat (usec) : 500=9.24%, 750=1.43%, 1000=0.65% lat (msec) : 2=2.34%, 4=5.08%, 10=17.84%, 20=22.79%, 50=13.02% lat (msec) : 100=1.17%, 250=0.13% cpu : usr=0.28%, sys=5.19%, ctx=916, majf=0, minf=19 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148375: Wed Apr 15 12:41:42 2026 read: IOPS=48, BW=196KiB/s (201kB/s)(1436KiB/7327msec) clat (usec): min=6, max=103525, avg=12214.35, stdev=14244.33 lat (usec): min=6, max=103525, avg=12215.09, stdev=14244.33 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 16], 10.00th=[ 20], 20.00th=[ 30], | 30.00th=[ 123], 40.00th=[ 6521], 50.00th=[ 10421], 60.00th=[ 13173], | 70.00th=[ 15926], 80.00th=[ 19792], 90.00th=[ 26346], 95.00th=[ 34866], | 99.00th=[ 84411], 99.50th=[102237], 99.90th=[103285], 99.95th=[103285], | 99.99th=[103285] bw ( KiB/s): min= 65, max= 314, per=24.26%, avg=195.07, stdev=65.93, samples=14 iops : min= 16, max= 78, avg=48.36, stdev=16.52, samples=14 write: IOPS=55, BW=223KiB/s (229kB/s)(1636KiB/7327msec); 0 zone resets clat (usec): min=143, max=59412, avg=7044.30, stdev=9326.10 lat (usec): min=144, max=59413, avg=7045.57, stdev=9326.09 clat percentiles (usec): | 1.00th=[ 178], 5.00th=[ 206], 10.00th=[ 235], 20.00th=[ 293], | 30.00th=[ 424], 40.00th=[ 1057], 50.00th=[ 3523], 60.00th=[ 5604], | 70.00th=[ 8455], 80.00th=[12911], 90.00th=[18744], 95.00th=[27657], | 99.00th=[39584], 99.50th=[45876], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 124, max= 362, per=25.72%, avg=217.57, stdev=65.56, samples=14 iops : min= 31, max= 90, avg=54.00, stdev=16.45, samples=14 lat (usec) : 10=1.17%, 20=3.52%, 50=8.85%, 100=0.26%, 250=7.42% lat (usec) : 500=10.55%, 750=2.21%, 1000=1.82% lat (msec) : 2=1.82%, 4=5.73%, 10=19.01%, 20=23.96%, 50=12.50% lat (msec) : 100=0.91%, 250=0.26% cpu : usr=0.20%, sys=5.79%, ctx=969, majf=0, minf=17 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148377: Wed Apr 15 12:41:42 2026 read: IOPS=52, BW=210KiB/s (215kB/s)(1564KiB/7440msec) clat (usec): min=6, max=63685, avg=11308.55, stdev=11736.80 lat (usec): min=6, max=63685, avg=11311.03, stdev=11740.08 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 15], 10.00th=[ 21], 20.00th=[ 29], | 30.00th=[ 52], 40.00th=[ 6783], 50.00th=[ 9634], 60.00th=[12518], | 70.00th=[15926], 80.00th=[19530], 90.00th=[25035], 95.00th=[34341], | 99.00th=[58459], 99.50th=[59507], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 88, max= 335, per=26.25%, avg=211.07, stdev=69.25, samples=14 iops : min= 22, max= 83, avg=52.36, stdev=17.31, samples=14 write: IOPS=50, BW=203KiB/s (208kB/s)(1508KiB/7440msec); 0 zone resets clat (usec): min=147, max=60824, avg=7876.66, stdev=8980.53 lat (usec): min=148, max=60825, avg=7877.68, stdev=8980.56 clat percentiles (usec): | 1.00th=[ 172], 5.00th=[ 200], 10.00th=[ 229], 20.00th=[ 338], | 30.00th=[ 938], 40.00th=[ 2999], 50.00th=[ 5145], 60.00th=[ 7635], | 70.00th=[10028], 80.00th=[14091], 90.00th=[19792], 95.00th=[24249], | 99.00th=[40109], 99.50th=[44827], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 95, max= 322, per=23.38%, avg=197.79, stdev=59.61, samples=14 iops : min= 23, max= 80, avg=49.00, stdev=14.98, samples=14 lat (usec) : 10=0.78%, 20=4.17%, 50=10.16%, 100=0.78%, 250=6.64% lat (usec) : 500=6.90%, 750=1.43%, 1000=0.78% lat (msec) : 2=2.73%, 4=5.08%, 10=20.44%, 20=25.52%, 50=13.67% lat (msec) : 100=0.91% cpu : usr=0.20%, sys=5.03%, ctx=980, majf=0, minf=18 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148378: Wed Apr 15 12:41:42 2026 read: IOPS=52, BW=210KiB/s (215kB/s)(1508KiB/7193msec) clat (usec): min=5, max=120440, avg=11240.34, stdev=14002.52 lat (usec): min=5, max=120442, avg=11241.11, stdev=14002.61 clat percentiles (usec): | 1.00th=[ 10], 5.00th=[ 13], 10.00th=[ 18], 20.00th=[ 26], | 30.00th=[ 36], 40.00th=[ 4883], 50.00th=[ 9110], 60.00th=[ 11994], | 70.00th=[ 15008], 80.00th=[ 18220], 90.00th=[ 26870], 95.00th=[ 34341], | 99.00th=[ 69731], 99.50th=[ 95945], 99.90th=[120062], 99.95th=[120062], | 99.99th=[120062] bw ( KiB/s): min= 88, max= 327, per=26.04%, avg=209.36, stdev=72.41, samples=14 iops : min= 22, max= 81, avg=51.93, stdev=18.07, samples=14 write: IOPS=54, BW=217KiB/s (223kB/s)(1564KiB/7193msec); 0 zone resets clat (usec): min=120, max=49539, avg=7470.55, stdev=9392.18 lat (usec): min=121, max=49539, avg=7477.79, stdev=9388.13 clat percentiles (usec): | 1.00th=[ 145], 5.00th=[ 188], 10.00th=[ 217], 20.00th=[ 273], | 30.00th=[ 392], 40.00th=[ 865], 50.00th=[ 4555], 60.00th=[ 6521], | 70.00th=[ 8455], 80.00th=[13566], 90.00th=[21365], 95.00th=[27132], | 99.00th=[40109], 99.50th=[46400], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 102, max= 360, per=25.68%, avg=217.29, stdev=70.93, samples=14 iops : min= 25, max= 90, avg=53.86, stdev=17.78, samples=14 lat (usec) : 10=1.04%, 20=5.08%, 50=10.68%, 100=0.65%, 250=8.98% lat (usec) : 500=8.85%, 750=3.12%, 1000=0.91% lat (msec) : 2=1.43%, 4=3.12%, 10=18.75%, 20=22.66%, 50=13.80% lat (msec) : 100=0.78%, 250=0.13% cpu : usr=0.19%, sys=5.23%, ctx=947, majf=0, minf=18 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=805KiB/s (824kB/s), 196KiB/s-210KiB/s (201kB/s-215kB/s), io=5988KiB (6132kB), run=7192-7440msec WRITE: bw=847KiB/s (867kB/s), 203KiB/s-223KiB/s (208kB/s-229kB/s), io=6300KiB (6451kB), run=7192-7440msec rand-rw: (groupid=0, jobs=1): err= 0: pid=148379: Wed Apr 15 12:41:47 2026 read: IOPS=31, BW=128KiB/s (131kB/s)(1480KiB/11593msec) clat (msec): min=2, max=116, avg=11.32, stdev=11.32 lat (msec): min=2, max=116, avg=11.33, stdev=11.32 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 12], 80.00th=[ 16], 90.00th=[ 24], 95.00th=[ 34], | 99.00th=[ 53], 99.50th=[ 85], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 55, max= 231, per=24.53%, avg=124.14, stdev=52.31, samples=22 iops : min= 13, max= 57, avg=30.64, stdev=13.11, samples=22 write: IOPS=34, BW=137KiB/s (141kB/s)(1592KiB/11593msec); 0 zone resets clat (usec): min=4690, max=65347, avg=18523.76, stdev=10003.64 lat (usec): min=4692, max=65349, avg=18525.43, stdev=10003.61 clat percentiles (usec): | 1.00th=[ 5276], 5.00th=[ 7242], 10.00th=[ 8717], 20.00th=[10683], | 30.00th=[12125], 40.00th=[14091], 50.00th=[15664], 60.00th=[18220], | 70.00th=[20841], 80.00th=[25297], 90.00th=[33424], 95.00th=[38011], | 99.00th=[51119], 99.50th=[61080], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 53, max= 239, per=25.16%, avg=134.09, stdev=54.44, samples=22 iops : min= 13, max= 59, avg=33.18, stdev=13.65, samples=22 lat (msec) : 4=7.29%, 10=32.55%, 20=36.98%, 50=22.14%, 100=0.91% lat (msec) : 250=0.13% cpu : usr=0.16%, sys=1.79%, ctx=854, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148380: Wed Apr 15 12:41:47 2026 read: IOPS=30, BW=122KiB/s (125kB/s)(1436KiB/11768msec) clat (usec): min=1909, max=116515, avg=10994.61, stdev=11098.32 lat (usec): min=1910, max=116518, avg=10996.00, stdev=11098.25 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 12], 80.00th=[ 16], 90.00th=[ 24], 95.00th=[ 31], | 99.00th=[ 52], 99.50th=[ 58], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 39, max= 231, per=24.00%, avg=121.43, stdev=57.54, samples=23 iops : min= 9, max= 57, avg=30.00, stdev=14.40, samples=23 write: IOPS=34, BW=139KiB/s (142kB/s)(1636KiB/11768msec); 0 zone resets clat (msec): min=4, max=121, avg=19.06, stdev=11.49 lat (msec): min=4, max=121, avg=19.07, stdev=11.49 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 12], | 30.00th=[ 13], 40.00th=[ 14], 50.00th=[ 16], 60.00th=[ 20], | 70.00th=[ 22], 80.00th=[ 26], 90.00th=[ 33], 95.00th=[ 39], | 99.00th=[ 61], 99.50th=[ 66], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 79, max= 272, per=25.84%, avg=137.74, stdev=54.43, samples=23 iops : min= 19, max= 68, avg=34.09, stdev=13.70, samples=23 lat (msec) : 2=0.26%, 4=7.29%, 10=30.86%, 20=36.07%, 50=23.96% lat (msec) : 100=1.30%, 250=0.26% cpu : usr=0.17%, sys=1.79%, ctx=842, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148381: Wed Apr 15 12:41:47 2026 read: IOPS=33, BW=132KiB/s (136kB/s)(1564KiB/11818msec) clat (usec): min=1949, max=70353, avg=11085.96, stdev=10356.25 lat (usec): min=1950, max=70355, avg=11089.86, stdev=10354.91 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 3163], 10.00th=[ 3654], 20.00th=[ 4424], | 30.00th=[ 5014], 40.00th=[ 5932], 50.00th=[ 6849], 60.00th=[ 8717], | 70.00th=[11076], 80.00th=[15664], 90.00th=[24511], 95.00th=[34341], | 99.00th=[53216], 99.50th=[57934], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 48, max= 280, per=26.01%, avg=131.61, stdev=69.05, samples=23 iops : min= 12, max= 70, avg=32.57, stdev=17.34, samples=23 write: IOPS=31, BW=128KiB/s (131kB/s)(1508KiB/11818msec); 0 zone resets clat (msec): min=4, max=125, avg=19.79, stdev=12.13 lat (msec): min=4, max=125, avg=19.79, stdev=12.13 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 12], | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 20], | 70.00th=[ 22], 80.00th=[ 27], 90.00th=[ 35], 95.00th=[ 46], | 99.00th=[ 63], 99.50th=[ 67], 99.90th=[ 126], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 71, max= 231, per=23.40%, avg=124.70, stdev=51.57, samples=23 iops : min= 17, max= 57, avg=30.83, stdev=12.90, samples=23 lat (msec) : 2=0.13%, 4=6.77%, 10=32.29%, 20=35.03%, 50=23.83% lat (msec) : 100=1.82%, 250=0.13% cpu : usr=0.16%, sys=1.68%, ctx=844, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148382: Wed Apr 15 12:41:47 2026 read: IOPS=32, BW=129KiB/s (132kB/s)(1508KiB/11727msec) clat (usec): min=1817, max=95707, avg=11383.13, stdev=10685.71 lat (usec): min=1818, max=95708, avg=11384.38, stdev=10685.69 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 3261], 10.00th=[ 3621], 20.00th=[ 4424], | 30.00th=[ 5276], 40.00th=[ 6128], 50.00th=[ 7242], 60.00th=[ 9503], | 70.00th=[11600], 80.00th=[17171], 90.00th=[23462], 95.00th=[33817], | 99.00th=[49546], 99.50th=[66323], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 46, max= 272, per=25.24%, avg=127.74, stdev=54.14, samples=23 iops : min= 11, max= 68, avg=31.57, stdev=13.62, samples=23 write: IOPS=33, BW=133KiB/s (137kB/s)(1564KiB/11727msec); 0 zone resets clat (msec): min=3, max=105, avg=18.95, stdev=11.70 lat (msec): min=3, max=105, avg=18.96, stdev=11.70 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 14], 50.00th=[ 16], 60.00th=[ 19], | 70.00th=[ 22], 80.00th=[ 25], 90.00th=[ 34], 95.00th=[ 44], | 99.00th=[ 61], 99.50th=[ 71], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 53, max= 231, per=24.99%, avg=133.22, stdev=53.35, samples=23 iops : min= 13, max= 57, avg=32.96, stdev=13.35, samples=23 lat (msec) : 2=0.13%, 4=7.68%, 10=31.38%, 20=35.68%, 50=23.31% lat (msec) : 100=1.69%, 250=0.13% cpu : usr=0.13%, sys=1.65%, ctx=819, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=507KiB/s (519kB/s), 122KiB/s-132KiB/s (125kB/s-136kB/s), io=5988KiB (6132kB), run=11593-11818msec WRITE: bw=533KiB/s (546kB/s), 128KiB/s-139KiB/s (131kB/s-142kB/s), io=6300KiB (6451kB), run=11593-11818msec