== sanity test 398b: DIO and buffer IO race ============== 10:12:22 (1761315142) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.189036 s, 66.6 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=151020: Fri Oct 24 10:12:41 2025 read: IOPS=44, BW=179KiB/s (183kB/s)(1480KiB/8279msec) clat (usec): min=5, max=112907, avg=13253.58, stdev=15411.02 lat (usec): min=6, max=112908, avg=13254.32, stdev=15411.05 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 18], 20.00th=[ 26], | 30.00th=[ 34], 40.00th=[ 6718], 50.00th=[ 11338], 60.00th=[ 14877], | 70.00th=[ 18744], 80.00th=[ 22414], 90.00th=[ 27919], 95.00th=[ 38536], | 99.00th=[ 78119], 99.50th=[112722], 99.90th=[112722], 99.95th=[112722], | 99.99th=[112722] bw ( KiB/s): min= 87, max= 351, per=26.68%, avg=179.81, stdev=62.39, samples=16 iops : min= 21, max= 87, avg=44.31, stdev=15.56, samples=16 write: IOPS=48, BW=192KiB/s (197kB/s)(1592KiB/8279msec); 0 zone resets clat (usec): min=158, max=56794, avg=8432.35, stdev=11018.04 lat (usec): min=158, max=56796, avg=8433.53, stdev=11018.23 clat percentiles (usec): | 1.00th=[ 165], 5.00th=[ 190], 10.00th=[ 200], 20.00th=[ 225], | 30.00th=[ 269], 40.00th=[ 433], 50.00th=[ 4113], 60.00th=[ 7177], | 70.00th=[10945], 80.00th=[15926], 90.00th=[24249], 95.00th=[30802], | 99.00th=[51119], 99.50th=[55313], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 100, max= 343, per=27.35%, avg=193.94, stdev=85.72, samples=16 iops : min= 25, max= 85, avg=47.88, stdev=21.35, samples=16 lat (usec) : 10=1.56%, 20=4.17%, 50=10.94%, 100=0.26%, 250=13.28% lat (usec) : 500=7.68%, 750=0.65%, 1000=0.78% lat (msec) : 2=2.47%, 4=1.69%, 10=14.06%, 20=23.18%, 50=17.32% lat (msec) : 100=1.69%, 250=0.26% cpu : usr=0.10%, sys=4.17%, ctx=812, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151021: Fri Oct 24 10:12:41 2025 read: IOPS=43, BW=173KiB/s (177kB/s)(1436KiB/8296msec) clat (usec): min=8, max=118026, avg=13127.85, stdev=14858.14 lat (usec): min=8, max=118027, avg=13129.78, stdev=14860.09 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 16], 10.00th=[ 22], 20.00th=[ 28], | 30.00th=[ 40], 40.00th=[ 7504], 50.00th=[ 11338], 60.00th=[ 13829], | 70.00th=[ 18744], 80.00th=[ 22152], 90.00th=[ 28967], 95.00th=[ 39060], | 99.00th=[ 69731], 99.50th=[101188], 99.90th=[117965], 99.95th=[117965], | 99.99th=[117965] bw ( KiB/s): min= 87, max= 255, per=25.70%, avg=173.19, stdev=53.16, samples=16 iops : min= 21, max= 63, avg=42.62, stdev=13.28, samples=16 write: IOPS=49, BW=197KiB/s (202kB/s)(1636KiB/8296msec); 0 zone resets clat (usec): min=160, max=62101, avg=8700.77, stdev=10859.19 lat (usec): min=161, max=62102, avg=8701.75, stdev=10859.16 clat percentiles (usec): | 1.00th=[ 169], 5.00th=[ 190], 10.00th=[ 215], 20.00th=[ 253], | 30.00th=[ 310], 40.00th=[ 1926], 50.00th=[ 5669], 60.00th=[ 7570], | 70.00th=[10814], 80.00th=[15533], 90.00th=[21365], 95.00th=[30802], | 99.00th=[46924], 99.50th=[50070], 99.90th=[62129], 99.95th=[62129], | 99.99th=[62129] bw ( KiB/s): min= 127, max= 319, per=27.78%, avg=196.94, stdev=58.81, samples=16 iops : min= 31, max= 79, avg=48.56, stdev=14.70, samples=16 lat (usec) : 10=0.52%, 20=3.26%, 50=10.81%, 100=0.91%, 250=10.42% lat (usec) : 500=8.46%, 750=0.39%, 1000=0.39% lat (msec) : 2=2.34%, 4=1.82%, 10=18.75%, 20=22.66%, 50=17.97% lat (msec) : 100=1.04%, 250=0.26% cpu : usr=0.18%, sys=4.09%, ctx=832, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151022: Fri Oct 24 10:12:41 2025 read: IOPS=44, BW=176KiB/s (180kB/s)(1564KiB/8877msec) clat (usec): min=6, max=92157, avg=13523.45, stdev=13638.64 lat (usec): min=6, max=92158, avg=13524.31, stdev=13638.61 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 18], 10.00th=[ 22], 20.00th=[ 32], | 30.00th=[ 111], 40.00th=[ 8455], 50.00th=[11731], 60.00th=[15401], | 70.00th=[18482], 80.00th=[22676], 90.00th=[29754], 95.00th=[36439], | 99.00th=[60031], 99.50th=[90702], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 110, max= 327, per=26.06%, avg=175.65, stdev=55.19, samples=17 iops : min= 27, max= 81, avg=43.24, stdev=13.76, samples=17 write: IOPS=42, BW=170KiB/s (174kB/s)(1508KiB/8877msec); 0 zone resets clat (usec): min=160, max=67308, avg=9473.51, stdev=11478.51 lat (usec): min=161, max=67309, avg=9476.38, stdev=11481.30 clat percentiles (usec): | 1.00th=[ 172], 5.00th=[ 196], 10.00th=[ 215], 20.00th=[ 255], | 30.00th=[ 338], 40.00th=[ 3032], 50.00th=[ 6718], 60.00th=[ 8586], | 70.00th=[11731], 80.00th=[16909], 90.00th=[25560], 95.00th=[33817], | 99.00th=[50594], 99.50th=[57410], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 63, max= 279, per=23.49%, avg=166.53, stdev=63.38, samples=17 iops : min= 15, max= 69, avg=41.00, stdev=15.80, samples=17 lat (usec) : 10=1.17%, 20=3.12%, 50=10.68%, 100=0.26%, 250=9.24% lat (usec) : 500=7.94%, 750=0.91%, 1000=0.39% lat (msec) : 2=0.65%, 4=2.34%, 10=18.10%, 20=24.35%, 50=19.53% lat (msec) : 100=1.30% cpu : usr=0.18%, sys=3.79%, ctx=887, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151023: Fri Oct 24 10:12:41 2025 read: IOPS=45, BW=183KiB/s (187kB/s)(1508KiB/8254msec) clat (usec): min=7, max=114301, avg=13275.12, stdev=15170.89 lat (usec): min=7, max=114303, avg=13275.90, stdev=15170.86 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 13], 10.00th=[ 17], 20.00th=[ 24], | 30.00th=[ 33], 40.00th=[ 6259], 50.00th=[ 10945], 60.00th=[ 15926], | 70.00th=[ 19006], 80.00th=[ 22152], 90.00th=[ 30016], 95.00th=[ 37487], | 99.00th=[ 74974], 99.50th=[107480], 99.90th=[114820], 99.95th=[114820], | 99.99th=[114820] bw ( KiB/s): min= 84, max= 302, per=25.79%, avg=173.80, stdev=60.55, samples=15 iops : min= 21, max= 75, avg=42.80, stdev=15.10, samples=15 write: IOPS=47, BW=189KiB/s (194kB/s)(1564KiB/8254msec); 0 zone resets clat (usec): min=149, max=59311, avg=8264.71, stdev=10965.17 lat (usec): min=150, max=59311, avg=8265.59, stdev=10965.16 clat percentiles (usec): | 1.00th=[ 172], 5.00th=[ 194], 10.00th=[ 210], 20.00th=[ 237], | 30.00th=[ 277], 40.00th=[ 371], 50.00th=[ 4424], 60.00th=[ 7242], | 70.00th=[10159], 80.00th=[15926], 90.00th=[22152], 95.00th=[31589], | 99.00th=[51119], 99.50th=[56361], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 110, max= 278, per=25.92%, avg=183.80, stdev=52.60, samples=15 iops : min= 27, max= 69, avg=45.27, stdev=13.13, samples=15 lat (usec) : 10=1.43%, 20=5.99%, 50=10.03%, 100=0.39%, 250=11.98% lat (usec) : 500=9.64%, 750=0.39%, 1000=0.65% lat (msec) : 2=1.17%, 4=1.69%, 10=13.93%, 20=22.40%, 50=18.49% lat (msec) : 100=1.56%, 250=0.26% cpu : usr=0.18%, sys=3.91%, ctx=806, majf=0, minf=17 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=675KiB/s (691kB/s), 173KiB/s-183KiB/s (177kB/s-187kB/s), io=5988KiB (6132kB), run=8254-8877msec WRITE: bw=710KiB/s (727kB/s), 170KiB/s-197KiB/s (174kB/s-202kB/s), io=6300KiB (6451kB), run=8254-8877msec rand-rw: (groupid=0, jobs=1): err= 0: pid=151015: Fri Oct 24 10:12:47 2025 read: IOPS=24, BW=98.7KiB/s (101kB/s)(1480KiB/14990msec) clat (usec): min=2779, max=77259, avg=13202.54, stdev=11467.49 lat (usec): min=2779, max=77261, avg=13205.07, stdev=11466.49 clat percentiles (usec): | 1.00th=[ 3687], 5.00th=[ 4293], 10.00th=[ 5080], 20.00th=[ 5866], | 30.00th=[ 6718], 40.00th=[ 7767], 50.00th=[ 8979], 60.00th=[10290], | 70.00th=[12649], 80.00th=[17433], 90.00th=[29230], 95.00th=[39060], | 99.00th=[52691], 99.50th=[69731], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 47, max= 187, per=24.49%, avg=97.72, stdev=38.09, samples=29 iops : min= 11, max= 46, avg=23.76, stdev= 9.56, samples=29 write: IOPS=26, BW=106KiB/s (109kB/s)(1592KiB/14990msec); 0 zone resets clat (usec): min=5648, max=93072, avg=25229.19, stdev=14335.86 lat (usec): min=5649, max=93074, avg=25230.56, stdev=14335.80 clat percentiles (usec): | 1.00th=[ 8455], 5.00th=[10683], 10.00th=[12518], 20.00th=[14615], | 30.00th=[16712], 40.00th=[18744], 50.00th=[21103], 60.00th=[23725], | 70.00th=[27919], 80.00th=[32375], 90.00th=[44303], 95.00th=[53740], | 99.00th=[82314], 99.50th=[90702], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 23, max= 191, per=25.23%, avg=105.97, stdev=47.22, samples=29 iops : min= 5, max= 47, avg=25.86, stdev=11.84, samples=29 lat (msec) : 4=1.04%, 10=28.52%, 20=33.98%, 50=31.64%, 100=4.82% cpu : usr=0.17%, sys=1.35%, ctx=823, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151016: Fri Oct 24 10:12:47 2025 read: IOPS=24, BW=96.7KiB/s (99.1kB/s)(1436KiB/14844msec) clat (usec): min=2504, max=91028, avg=12732.52, stdev=11470.59 lat (usec): min=2504, max=91029, avg=12733.48, stdev=11470.59 clat percentiles (usec): | 1.00th=[ 3326], 5.00th=[ 4080], 10.00th=[ 4948], 20.00th=[ 5735], | 30.00th=[ 6587], 40.00th=[ 7242], 50.00th=[ 8455], 60.00th=[10683], | 70.00th=[13435], 80.00th=[16188], 90.00th=[25560], 95.00th=[36439], | 99.00th=[64750], 99.50th=[71828], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 16, max= 192, per=24.04%, avg=95.93, stdev=48.35, samples=29 iops : min= 4, max= 48, avg=23.41, stdev=12.15, samples=29 write: IOPS=27, BW=110KiB/s (113kB/s)(1636KiB/14844msec); 0 zone resets clat (usec): min=5931, max=94813, avg=24925.92, stdev=13968.63 lat (usec): min=5933, max=94814, avg=24927.49, stdev=13968.33 clat percentiles (usec): | 1.00th=[ 8455], 5.00th=[10552], 10.00th=[12256], 20.00th=[14484], | 30.00th=[16450], 40.00th=[18482], 50.00th=[20579], 60.00th=[23725], | 70.00th=[26608], 80.00th=[34341], 90.00th=[44827], 95.00th=[53740], | 99.00th=[71828], 99.50th=[76022], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 47, max= 187, per=25.84%, avg=108.52, stdev=42.39, samples=29 iops : min= 11, max= 46, avg=26.55, stdev=10.65, samples=29 lat (msec) : 4=2.08%, 10=26.82%, 20=36.20%, 50=30.47%, 100=4.43% cpu : usr=0.11%, sys=1.26%, ctx=822, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151017: Fri Oct 24 10:12:47 2025 read: IOPS=26, BW=107KiB/s (109kB/s)(1564KiB/14648msec) clat (usec): min=2698, max=60036, avg=12949.75, stdev=10927.29 lat (usec): min=2708, max=60038, avg=12951.67, stdev=10926.97 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 4080], 10.00th=[ 4621], 20.00th=[ 5866], | 30.00th=[ 6915], 40.00th=[ 7767], 50.00th=[ 8717], 60.00th=[ 9765], | 70.00th=[12125], 80.00th=[17957], 90.00th=[29754], 95.00th=[38011], | 99.00th=[55837], 99.50th=[56886], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 39, max= 266, per=26.81%, avg=106.96, stdev=59.26, samples=28 iops : min= 9, max= 66, avg=26.11, stdev=14.83, samples=28 write: IOPS=25, BW=103KiB/s (105kB/s)(1508KiB/14648msec); 0 zone resets clat (msec): min=6, max=100, avg=25.20, stdev=13.84 lat (msec): min=6, max=100, avg=25.20, stdev=13.84 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 24], | 70.00th=[ 29], 80.00th=[ 35], 90.00th=[ 44], 95.00th=[ 53], | 99.00th=[ 78], 99.50th=[ 87], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 46, max= 183, per=24.21%, avg=101.68, stdev=44.58, samples=28 iops : min= 11, max= 45, avg=24.79, stdev=11.19, samples=28 lat (msec) : 4=2.34%, 10=30.34%, 20=32.42%, 50=31.25%, 100=3.52% lat (msec) : 250=0.13% cpu : usr=0.12%, sys=1.33%, ctx=835, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151018: Fri Oct 24 10:12:47 2025 read: IOPS=25, BW=102KiB/s (105kB/s)(1508KiB/14772msec) clat (usec): min=2153, max=76926, avg=13267.38, stdev=11600.13 lat (usec): min=2154, max=76927, avg=13269.04, stdev=11599.98 clat percentiles (usec): | 1.00th=[ 3097], 5.00th=[ 4228], 10.00th=[ 4948], 20.00th=[ 5866], | 30.00th=[ 6980], 40.00th=[ 7701], 50.00th=[ 8979], 60.00th=[10683], | 70.00th=[12649], 80.00th=[18744], 90.00th=[27395], 95.00th=[36439], | 99.00th=[59507], 99.50th=[72877], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 39, max= 215, per=25.08%, avg=100.07, stdev=45.20, samples=28 iops : min= 9, max= 53, avg=24.39, stdev=11.34, samples=28 write: IOPS=26, BW=106KiB/s (108kB/s)(1564KiB/14772msec); 0 zone resets clat (msec): min=6, max=132, avg=24.77, stdev=14.15 lat (msec): min=6, max=132, avg=24.77, stdev=14.15 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 24], | 70.00th=[ 28], 80.00th=[ 34], 90.00th=[ 43], 95.00th=[ 50], | 99.00th=[ 71], 99.50th=[ 92], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 47, max= 191, per=24.88%, avg=104.50, stdev=44.72, samples=28 iops : min= 11, max= 47, avg=25.50, stdev=11.27, samples=28 lat (msec) : 4=1.95%, 10=27.86%, 20=34.64%, 50=31.90%, 100=3.52% lat (msec) : 250=0.13% cpu : usr=0.16%, sys=1.34%, ctx=823, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=399KiB/s (409kB/s), 96.7KiB/s-107KiB/s (99.1kB/s-109kB/s), io=5988KiB (6132kB), run=14648-14990msec WRITE: bw=420KiB/s (430kB/s), 103KiB/s-110KiB/s (105kB/s-113kB/s), io=6300KiB (6451kB), run=14648-14990msec