== sanity test 398b: DIO and buffer IO race ============== 16:47:46 (1771969666) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.0763628 s, 165 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=121605: Tue Feb 24 16:47:54 2026 read: IOPS=180, BW=723KiB/s (740kB/s)(1480KiB/2047msec) clat (usec): min=5, max=37078, avg=1870.85, stdev=3970.80 lat (usec): min=5, max=37078, avg=1871.29, stdev=3970.79 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 21], 50.00th=[ 24], 60.00th=[ 37], | 70.00th=[ 2040], 80.00th=[ 3589], 90.00th=[ 5538], 95.00th=[ 7439], | 99.00th=[22152], 99.50th=[27395], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 400, max= 1146, per=28.83%, avg=710.00, stdev=388.63, samples=3 iops : min= 100, max= 286, avg=177.33, stdev=96.88, samples=3 write: IOPS=194, BW=778KiB/s (796kB/s)(1592KiB/2047msec); 0 zone resets clat (usec): min=126, max=40655, avg=3363.69, stdev=5819.73 lat (usec): min=126, max=40656, avg=3364.17, stdev=5819.76 clat percentiles (usec): | 1.00th=[ 139], 5.00th=[ 178], 10.00th=[ 243], 20.00th=[ 338], | 30.00th=[ 437], 40.00th=[ 486], 50.00th=[ 553], 60.00th=[ 1237], | 70.00th=[ 2474], 80.00th=[ 4817], 90.00th=[11863], 95.00th=[17433], | 99.00th=[26608], 99.50th=[32375], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 520, max= 1206, per=29.92%, avg=775.33, stdev=375.11, samples=3 iops : min= 130, max= 301, avg=193.67, stdev=93.49, samples=3 lat (usec) : 10=3.26%, 20=15.89%, 50=11.72%, 100=0.91%, 250=5.86% lat (usec) : 500=17.32%, 750=7.03%, 1000=0.78% lat (msec) : 2=5.08%, 4=10.94%, 10=13.93%, 20=4.69%, 50=2.60% cpu : usr=0.49%, sys=11.44%, ctx=691, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=121606: Tue Feb 24 16:47:54 2026 read: IOPS=161, BW=647KiB/s (663kB/s)(1436KiB/2219msec) clat (usec): min=6, max=47997, avg=1889.74, stdev=4297.57 lat (usec): min=6, max=47997, avg=1890.23, stdev=4297.67 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 21], 50.00th=[ 29], 60.00th=[ 60], | 70.00th=[ 2311], 80.00th=[ 3359], 90.00th=[ 5342], 95.00th=[ 7373], | 99.00th=[18744], 99.50th=[44827], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 376, max= 1206, per=25.40%, avg=625.50, stdev=389.36, samples=4 iops : min= 94, max= 301, avg=156.25, stdev=97.09, samples=4 write: IOPS=184, BW=737KiB/s (755kB/s)(1636KiB/2219msec); 0 zone resets clat (usec): min=134, max=38748, avg=3729.33, stdev=5679.65 lat (usec): min=134, max=38749, avg=3729.83, stdev=5679.70 clat percentiles (usec): | 1.00th=[ 176], 5.00th=[ 225], 10.00th=[ 285], 20.00th=[ 420], | 30.00th=[ 474], 40.00th=[ 515], 50.00th=[ 635], 60.00th=[ 1975], | 70.00th=[ 3654], 80.00th=[ 6063], 90.00th=[11994], 95.00th=[15664], | 99.00th=[22938], 99.50th=[30540], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 456, max= 1386, per=26.96%, avg=698.50, stdev=458.53, samples=4 iops : min= 114, max= 346, avg=174.50, stdev=114.38, samples=4 lat (usec) : 10=1.04%, 20=17.19%, 50=8.98%, 100=1.17%, 250=4.04% lat (usec) : 500=16.41%, 750=7.16%, 1000=0.26% lat (msec) : 2=7.03%, 4=14.32%, 10=14.84%, 20=5.60%, 50=1.95% cpu : usr=0.14%, sys=12.35%, ctx=698, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=121607: Tue Feb 24 16:47:54 2026 read: IOPS=160, BW=643KiB/s (659kB/s)(1564KiB/2431msec) clat (usec): min=5, max=34283, avg=2390.06, stdev=4065.30 lat (usec): min=5, max=34284, avg=2390.47, stdev=4065.31 clat percentiles (usec): | 1.00th=[ 7], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 16], | 30.00th=[ 20], 40.00th=[ 28], 50.00th=[ 45], 60.00th=[ 2409], | 70.00th=[ 3294], 80.00th=[ 4293], 90.00th=[ 5997], 95.00th=[ 7635], | 99.00th=[20579], 99.50th=[32637], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 416, max= 760, per=22.17%, avg=546.00, stdev=153.66, samples=4 iops : min= 104, max= 190, avg=136.50, stdev=38.41, samples=4 write: IOPS=155, BW=620KiB/s (635kB/s)(1508KiB/2431msec); 0 zone resets clat (usec): min=124, max=39759, avg=3939.54, stdev=6227.16 lat (usec): min=124, max=39759, avg=3940.05, stdev=6227.19 clat percentiles (usec): | 1.00th=[ 141], 5.00th=[ 165], 10.00th=[ 215], 20.00th=[ 314], | 30.00th=[ 437], 40.00th=[ 478], 50.00th=[ 1156], 60.00th=[ 2278], | 70.00th=[ 3654], 80.00th=[ 5735], 90.00th=[12125], 95.00th=[16319], | 99.00th=[29492], 99.50th=[35914], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 416, max= 813, per=20.97%, avg=543.25, stdev=181.88, samples=4 iops : min= 104, max= 203, avg=135.75, stdev=45.35, samples=4 lat (usec) : 10=2.60%, 20=13.67%, 50=9.64%, 100=0.78%, 250=7.29% lat (usec) : 500=14.58%, 750=2.86%, 1000=0.13% lat (msec) : 2=5.99%, 4=17.06%, 10=17.19%, 20=5.86%, 50=2.34% cpu : usr=0.29%, sys=11.07%, ctx=810, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=121608: Tue Feb 24 16:47:54 2026 read: IOPS=217, BW=869KiB/s (890kB/s)(1508KiB/1736msec) clat (usec): min=4, max=43880, avg=1463.15, stdev=3703.77 lat (usec): min=4, max=43880, avg=1463.48, stdev=3703.82 clat percentiles (usec): | 1.00th=[ 6], 5.00th=[ 9], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 22], | 70.00th=[ 42], 80.00th=[ 2966], 90.00th=[ 4817], 95.00th=[ 6915], | 99.00th=[13173], 99.50th=[36439], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 360, max= 1649, per=34.01%, avg=837.67, stdev=706.31, samples=3 iops : min= 90, max= 412, avg=209.33, stdev=176.44, samples=3 write: IOPS=225, BW=901KiB/s (923kB/s)(1564KiB/1736msec); 0 zone resets clat (usec): min=115, max=33413, avg=2994.06, stdev=5649.01 lat (usec): min=115, max=33414, avg=2994.52, stdev=5649.05 clat percentiles (usec): | 1.00th=[ 125], 5.00th=[ 159], 10.00th=[ 221], 20.00th=[ 318], | 30.00th=[ 412], 40.00th=[ 449], 50.00th=[ 482], 60.00th=[ 529], | 70.00th=[ 1450], 80.00th=[ 4424], 90.00th=[ 9765], 95.00th=[16712], | 99.00th=[30278], 99.50th=[32113], 99.90th=[33424], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 448, max= 1702, per=33.73%, avg=874.00, stdev=717.17, samples=3 iops : min= 112, max= 425, avg=218.33, stdev=179.00, samples=3 lat (usec) : 10=4.69%, 20=22.92%, 50=7.29%, 100=0.39%, 250=6.90% lat (usec) : 500=20.96%, 750=6.12%, 1000=0.26% lat (msec) : 2=4.69%, 4=8.59%, 10=11.46%, 20=4.17%, 50=1.56% cpu : usr=0.17%, sys=12.74%, ctx=532, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2463KiB/s (2522kB/s), 643KiB/s-869KiB/s (659kB/s-890kB/s), io=5988KiB (6132kB), run=1736-2431msec WRITE: bw=2592KiB/s (2654kB/s), 620KiB/s-901KiB/s (635kB/s-923kB/s), io=6300KiB (6451kB), run=1736-2431msec rand-rw: (groupid=0, jobs=1): err= 0: pid=121610: Tue Feb 24 16:48:03 2026 read: IOPS=34, BW=137KiB/s (140kB/s)(1480KiB/10818msec) clat (usec): min=956, max=35555, avg=4084.74, stdev=4533.72 lat (usec): min=956, max=35556, avg=4085.39, stdev=4533.74 clat percentiles (usec): | 1.00th=[ 1029], 5.00th=[ 1172], 10.00th=[ 1254], 20.00th=[ 1500], | 30.00th=[ 1745], 40.00th=[ 2278], 50.00th=[ 2999], 60.00th=[ 3556], | 70.00th=[ 4113], 80.00th=[ 5080], 90.00th=[ 6980], 95.00th=[11469], | 99.00th=[28705], 99.50th=[32637], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 72, max= 248, per=24.61%, avg=135.38, stdev=42.62, samples=21 iops : min= 18, max= 62, avg=33.81, stdev=10.69, samples=21 write: IOPS=36, BW=147KiB/s (151kB/s)(1592KiB/10818msec); 0 zone resets clat (usec): min=10540, max=73443, avg=23354.79, stdev=7292.14 lat (usec): min=10541, max=73444, avg=23355.53, stdev=7292.17 clat percentiles (usec): | 1.00th=[11338], 5.00th=[15008], 10.00th=[16319], 20.00th=[17957], | 30.00th=[19006], 40.00th=[20055], 50.00th=[21365], 60.00th=[23462], | 70.00th=[25297], 80.00th=[28443], 90.00th=[32900], 95.00th=[38011], | 99.00th=[47449], 99.50th=[54789], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 84, max= 176, per=25.47%, avg=147.24, stdev=26.82, samples=21 iops : min= 21, max= 44, avg=36.81, stdev= 6.71, samples=21 lat (usec) : 1000=0.26% lat (msec) : 2=17.32%, 4=15.10%, 10=12.76%, 20=22.53%, 50=31.77% lat (msec) : 100=0.26% cpu : usr=0.10%, sys=1.71%, ctx=1051, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=121611: Tue Feb 24 16:48:03 2026 read: IOPS=32, BW=132KiB/s (135kB/s)(1436KiB/10886msec) clat (usec): min=949, max=61105, avg=4169.44, stdev=4508.84 lat (usec): min=949, max=61106, avg=4170.03, stdev=4508.86 clat percentiles (usec): | 1.00th=[ 1012], 5.00th=[ 1156], 10.00th=[ 1303], 20.00th=[ 1549], | 30.00th=[ 1991], 40.00th=[ 2704], 50.00th=[ 3359], 60.00th=[ 3949], | 70.00th=[ 4555], 80.00th=[ 5800], 90.00th=[ 6915], 95.00th=[ 9372], | 99.00th=[20841], 99.50th=[33424], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 58, max= 208, per=24.12%, avg=132.67, stdev=40.02, samples=21 iops : min= 14, max= 52, avg=33.14, stdev=10.05, samples=21 write: IOPS=37, BW=150KiB/s (154kB/s)(1636KiB/10886msec); 0 zone resets clat (usec): min=9257, max=54293, avg=22928.70, stdev=6611.65 lat (usec): min=9257, max=54294, avg=22929.45, stdev=6611.66 clat percentiles (usec): | 1.00th=[10028], 5.00th=[14091], 10.00th=[15926], 20.00th=[17695], | 30.00th=[19006], 40.00th=[20055], 50.00th=[22152], 60.00th=[23987], | 70.00th=[25560], 80.00th=[28443], 90.00th=[31065], 95.00th=[33162], | 99.00th=[46400], 99.50th=[49021], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 97, max= 176, per=25.71%, avg=148.62, stdev=20.82, samples=21 iops : min= 24, max= 44, avg=37.14, stdev= 5.24, samples=21 lat (usec) : 1000=0.26% lat (msec) : 2=14.06%, 4=14.45%, 10=16.41%, 20=22.14%, 50=32.42% lat (msec) : 100=0.26% cpu : usr=0.11%, sys=1.75%, ctx=1067, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=121613: Tue Feb 24 16:48:03 2026 read: IOPS=36, BW=148KiB/s (151kB/s)(1564KiB/10579msec) clat (usec): min=976, max=47589, avg=4144.62, stdev=4166.50 lat (usec): min=976, max=47590, avg=4145.30, stdev=4166.51 clat percentiles (usec): | 1.00th=[ 1057], 5.00th=[ 1188], 10.00th=[ 1319], 20.00th=[ 1614], | 30.00th=[ 1942], 40.00th=[ 2474], 50.00th=[ 3163], 60.00th=[ 3785], | 70.00th=[ 4490], 80.00th=[ 5473], 90.00th=[ 6849], 95.00th=[ 9765], | 99.00th=[21627], 99.50th=[28443], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 72, max= 272, per=27.47%, avg=151.10, stdev=59.71, samples=20 iops : min= 18, max= 68, avg=37.75, stdev=14.96, samples=20 write: IOPS=35, BW=143KiB/s (146kB/s)(1508KiB/10579msec); 0 zone resets clat (usec): min=10899, max=54690, avg=23738.88, stdev=7340.49 lat (usec): min=10900, max=54691, avg=23739.65, stdev=7340.52 clat percentiles (usec): | 1.00th=[11469], 5.00th=[14615], 10.00th=[16712], 20.00th=[18220], | 30.00th=[19530], 40.00th=[20317], 50.00th=[22152], 60.00th=[24249], | 70.00th=[26346], 80.00th=[28705], 90.00th=[32900], 95.00th=[36963], | 99.00th=[51643], 99.50th=[54264], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 91, max= 168, per=24.59%, avg=142.15, stdev=20.99, samples=20 iops : min= 22, max= 42, avg=35.50, stdev= 5.35, samples=20 lat (usec) : 1000=0.13% lat (msec) : 2=16.02%, 4=15.49%, 10=16.80%, 20=19.53%, 50=31.38% lat (msec) : 100=0.65% cpu : usr=0.16%, sys=1.76%, ctx=1073, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=121614: Tue Feb 24 16:48:03 2026 read: IOPS=34, BW=139KiB/s (143kB/s)(1508KiB/10828msec) clat (usec): min=928, max=76085, avg=4146.68, stdev=5523.44 lat (usec): min=928, max=76085, avg=4147.36, stdev=5523.48 clat percentiles (usec): | 1.00th=[ 1037], 5.00th=[ 1188], 10.00th=[ 1287], 20.00th=[ 1532], | 30.00th=[ 1942], 40.00th=[ 2540], 50.00th=[ 3032], 60.00th=[ 3752], | 70.00th=[ 4424], 80.00th=[ 5145], 90.00th=[ 6521], 95.00th=[ 9634], | 99.00th=[24511], 99.50th=[49546], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 48, max= 208, per=25.20%, avg=138.62, stdev=49.98, samples=21 iops : min= 12, max= 52, avg=34.62, stdev=12.55, samples=21 write: IOPS=36, BW=144KiB/s (148kB/s)(1564KiB/10828msec); 0 zone resets clat (usec): min=10784, max=64145, avg=23667.93, stdev=7777.19 lat (usec): min=10785, max=64146, avg=23668.73, stdev=7777.27 clat percentiles (usec): | 1.00th=[11076], 5.00th=[14615], 10.00th=[16450], 20.00th=[17957], | 30.00th=[19006], 40.00th=[20579], 50.00th=[21890], 60.00th=[23987], | 70.00th=[25822], 80.00th=[28443], 90.00th=[32900], 95.00th=[36439], | 99.00th=[53216], 99.50th=[60031], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 78, max= 176, per=24.96%, avg=144.29, stdev=26.47, samples=21 iops : min= 19, max= 44, avg=36.05, stdev= 6.68, samples=21 lat (usec) : 1000=0.13% lat (msec) : 2=15.10%, 4=15.89%, 10=15.89%, 20=20.96%, 50=31.12% lat (msec) : 100=0.91% cpu : usr=0.10%, sys=1.74%, ctx=1069, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=550KiB/s (563kB/s), 132KiB/s-148KiB/s (135kB/s-151kB/s), io=5988KiB (6132kB), run=10579-10886msec WRITE: bw=579KiB/s (593kB/s), 143KiB/s-150KiB/s (146kB/s-154kB/s), io=6300KiB (6451kB), run=10579-10886msec