== sanity test 398b: DIO and buffer IO race ============== 22:01:25 (1767841285) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.0890289 s, 141 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=143282: Wed Jan 7 22:01:34 2026 read: IOPS=130, BW=521KiB/s (534kB/s)(1480KiB/2838msec) clat (usec): min=4, max=32211, avg=2813.35, stdev=4588.25 lat (usec): min=4, max=32211, avg=2814.03, stdev=4588.22 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 12], 10.00th=[ 16], 20.00th=[ 20], | 30.00th=[ 27], 40.00th=[ 47], 50.00th=[ 1729], 60.00th=[ 2442], | 70.00th=[ 3261], 80.00th=[ 4080], 90.00th=[ 5604], 95.00th=[13304], | 99.00th=[21365], 99.50th=[26870], 99.90th=[32113], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 374, max= 656, per=23.44%, avg=494.40, stdev=131.02, samples=5 iops : min= 93, max= 164, avg=123.40, stdev=32.95, samples=5 write: IOPS=140, BW=561KiB/s (574kB/s)(1592KiB/2838msec); 0 zone resets clat (usec): min=106, max=25644, avg=4480.31, stdev=5334.95 lat (usec): min=107, max=25644, avg=4481.07, stdev=5335.18 clat percentiles (usec): | 1.00th=[ 127], 5.00th=[ 172], 10.00th=[ 212], 20.00th=[ 281], | 30.00th=[ 400], 40.00th=[ 1467], 50.00th=[ 2442], 60.00th=[ 3621], | 70.00th=[ 5145], 80.00th=[ 8225], 90.00th=[12256], 95.00th=[17171], | 99.00th=[22414], 99.50th=[24249], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 436, max= 648, per=24.31%, avg=539.40, stdev=81.26, samples=5 iops : min= 109, max= 162, avg=134.80, stdev=20.29, samples=5 lat (usec) : 10=1.17%, 20=8.46%, 50=9.90%, 100=1.56%, 250=8.72% lat (usec) : 500=9.38%, 750=0.91%, 1000=0.65% lat (msec) : 2=7.81%, 4=22.01%, 10=18.62%, 20=8.98%, 50=1.82% cpu : usr=0.49%, sys=12.27%, ctx=1053, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=143283: Wed Jan 7 22:01:34 2026 read: IOPS=130, BW=520KiB/s (533kB/s)(1436KiB/2759msec) clat (usec): min=7, max=33371, avg=3115.93, stdev=4829.85 lat (usec): min=7, max=33372, avg=3116.36, stdev=4829.86 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 13], 10.00th=[ 16], 20.00th=[ 21], | 30.00th=[ 31], 40.00th=[ 48], 50.00th=[ 2114], 60.00th=[ 2966], | 70.00th=[ 3720], 80.00th=[ 4883], 90.00th=[ 6783], 95.00th=[ 9896], | 99.00th=[27395], 99.50th=[33162], 99.90th=[33424], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 436, max= 589, per=24.82%, avg=523.40, stdev=59.56, samples=5 iops : min= 109, max= 147, avg=130.80, stdev=14.82, samples=5 write: IOPS=148, BW=593KiB/s (607kB/s)(1636KiB/2759msec); 0 zone resets clat (usec): min=109, max=41731, avg=3976.43, stdev=5390.75 lat (usec): min=109, max=41732, avg=3976.99, stdev=5390.78 clat percentiles (usec): | 1.00th=[ 137], 5.00th=[ 186], 10.00th=[ 219], 20.00th=[ 289], | 30.00th=[ 379], 40.00th=[ 1287], 50.00th=[ 2024], 60.00th=[ 2933], | 70.00th=[ 4178], 80.00th=[ 6849], 90.00th=[10814], 95.00th=[14484], | 99.00th=[22938], 99.50th=[26346], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 504, max= 680, per=26.65%, avg=591.40, stdev=78.73, samples=5 iops : min= 126, max= 170, avg=147.80, stdev=19.68, samples=5 lat (usec) : 10=1.17%, 20=7.68%, 50=9.90%, 100=0.65%, 250=8.33% lat (usec) : 500=10.68%, 750=0.91%, 1000=0.78% lat (msec) : 2=9.24%, 4=20.70%, 10=21.61%, 20=6.38%, 50=1.95% cpu : usr=0.25%, sys=14.94%, ctx=1080, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=143284: Wed Jan 7 22:01:34 2026 read: IOPS=153, BW=615KiB/s (630kB/s)(1564KiB/2544msec) clat (usec): min=7, max=30008, avg=2896.07, stdev=3947.22 lat (usec): min=7, max=30009, avg=2896.59, stdev=3947.24 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 13], 10.00th=[ 16], 20.00th=[ 21], | 30.00th=[ 31], 40.00th=[ 66], 50.00th=[ 1975], 60.00th=[ 2933], | 70.00th=[ 3851], 80.00th=[ 5014], 90.00th=[ 7373], 95.00th=[ 8717], | 99.00th=[22152], 99.50th=[24511], 99.90th=[30016], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 512, max= 752, per=27.47%, avg=579.25, stdev=115.38, samples=4 iops : min= 128, max= 188, avg=144.75, stdev=28.88, samples=4 write: IOPS=148, BW=593KiB/s (607kB/s)(1508KiB/2544msec); 0 zone resets clat (usec): min=104, max=41989, avg=3705.95, stdev=5265.15 lat (usec): min=104, max=41989, avg=3706.69, stdev=5265.25 clat percentiles (usec): | 1.00th=[ 121], 5.00th=[ 163], 10.00th=[ 200], 20.00th=[ 249], | 30.00th=[ 314], 40.00th=[ 404], 50.00th=[ 1418], 60.00th=[ 2769], | 70.00th=[ 4047], 80.00th=[ 6521], 90.00th=[10683], 95.00th=[14615], | 99.00th=[22676], 99.50th=[23200], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 456, max= 806, per=25.81%, avg=572.75, stdev=161.69, samples=4 iops : min= 114, max= 201, avg=143.00, stdev=40.19, samples=4 lat (usec) : 10=1.43%, 20=8.46%, 50=10.29%, 100=0.65%, 250=10.81% lat (usec) : 500=11.72%, 750=0.91%, 1000=0.13% lat (msec) : 2=8.72%, 4=16.93%, 10=22.79%, 20=5.73%, 50=1.43% cpu : usr=0.24%, sys=17.81%, ctx=984, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=143285: Wed Jan 7 22:01:34 2026 read: IOPS=142, BW=570KiB/s (583kB/s)(1508KiB/2647msec) clat (usec): min=5, max=55835, avg=2868.21, stdev=4930.22 lat (usec): min=5, max=55836, avg=2868.68, stdev=4930.24 clat percentiles (usec): | 1.00th=[ 7], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 20], | 30.00th=[ 30], 40.00th=[ 51], 50.00th=[ 1975], 60.00th=[ 2704], | 70.00th=[ 3589], 80.00th=[ 4359], 90.00th=[ 5866], 95.00th=[ 9634], | 99.00th=[25822], 99.50th=[31851], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 484, max= 728, per=27.10%, avg=571.60, stdev=100.50, samples=5 iops : min= 121, max= 182, avg=142.80, stdev=25.23, samples=5 write: IOPS=147, BW=591KiB/s (605kB/s)(1564KiB/2647msec); 0 zone resets clat (usec): min=105, max=30986, avg=3972.83, stdev=5296.73 lat (usec): min=105, max=30987, avg=3973.63, stdev=5296.57 clat percentiles (usec): | 1.00th=[ 121], 5.00th=[ 190], 10.00th=[ 219], 20.00th=[ 297], | 30.00th=[ 371], 40.00th=[ 660], 50.00th=[ 1729], 60.00th=[ 3032], | 70.00th=[ 4359], 80.00th=[ 6783], 90.00th=[11863], 95.00th=[15008], | 99.00th=[26084], 99.50th=[27657], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 511, max= 688, per=26.79%, avg=594.40, stdev=65.57, samples=5 iops : min= 127, max= 172, avg=148.40, stdev=16.62, samples=5 lat (usec) : 10=1.82%, 20=9.11%, 50=8.59%, 100=1.56%, 250=7.81% lat (usec) : 500=12.11%, 750=1.43%, 1000=0.65% lat (msec) : 2=8.46%, 4=20.31%, 10=19.66%, 20=6.90%, 50=1.43% lat (msec) : 100=0.13% cpu : usr=0.26%, sys=14.74%, ctx=996, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2110KiB/s (2161kB/s), 520KiB/s-615KiB/s (533kB/s-630kB/s), io=5988KiB (6132kB), run=2544-2838msec WRITE: bw=2220KiB/s (2273kB/s), 561KiB/s-593KiB/s (574kB/s-607kB/s), io=6300KiB (6451kB), run=2544-2838msec rand-rw: (groupid=0, jobs=1): err= 0: pid=143287: Wed Jan 7 22:01:40 2026 read: IOPS=41, BW=166KiB/s (170kB/s)(1480KiB/8925msec) clat (usec): min=935, max=41695, avg=3583.65, stdev=5036.72 lat (usec): min=936, max=41696, avg=3584.24, stdev=5036.72 clat percentiles (usec): | 1.00th=[ 1004], 5.00th=[ 1090], 10.00th=[ 1172], 20.00th=[ 1352], | 30.00th=[ 1582], 40.00th=[ 1778], 50.00th=[ 2040], 60.00th=[ 2540], | 70.00th=[ 2999], 80.00th=[ 3621], 90.00th=[ 5407], 95.00th=[13173], | 99.00th=[31065], 99.50th=[33162], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 104, max= 264, per=24.61%, avg=163.88, stdev=46.96, samples=17 iops : min= 26, max= 66, avg=40.88, stdev=11.82, samples=17 write: IOPS=44, BW=178KiB/s (183kB/s)(1592KiB/8925msec); 0 zone resets clat (usec): min=10551, max=51330, avg=19061.57, stdev=5766.32 lat (usec): min=10552, max=51331, avg=19062.54, stdev=5766.42 clat percentiles (usec): | 1.00th=[11338], 5.00th=[12256], 10.00th=[13042], 20.00th=[14091], | 30.00th=[15270], 40.00th=[16909], 50.00th=[18220], 60.00th=[19792], | 70.00th=[21103], 80.00th=[22676], 90.00th=[25822], 95.00th=[28967], | 99.00th=[42206], 99.50th=[51119], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 99, max= 240, per=25.09%, avg=175.65, stdev=39.27, samples=17 iops : min= 24, max= 60, avg=43.82, stdev= 9.99, samples=17 lat (usec) : 1000=0.52% lat (msec) : 2=22.79%, 4=17.32%, 10=4.43%, 20=33.72%, 50=20.96% lat (msec) : 100=0.26% cpu : usr=0.13%, sys=1.96%, ctx=885, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=143288: Wed Jan 7 22:01:40 2026 read: IOPS=39, BW=160KiB/s (164kB/s)(1436KiB/8988msec) clat (usec): min=906, max=41610, avg=3203.29, stdev=4205.26 lat (usec): min=907, max=41611, avg=3203.89, stdev=4205.38 clat percentiles (usec): | 1.00th=[ 963], 5.00th=[ 1057], 10.00th=[ 1172], 20.00th=[ 1336], | 30.00th=[ 1500], 40.00th=[ 1762], 50.00th=[ 2089], 60.00th=[ 2507], | 70.00th=[ 2900], 80.00th=[ 3621], 90.00th=[ 5800], 95.00th=[ 8586], | 99.00th=[20317], 99.50th=[40633], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 68, max= 272, per=24.05%, avg=160.18, stdev=57.54, samples=17 iops : min= 17, max= 68, avg=40.00, stdev=14.37, samples=17 write: IOPS=45, BW=182KiB/s (186kB/s)(1636KiB/8988msec); 0 zone resets clat (usec): min=9285, max=52130, avg=19133.42, stdev=6332.02 lat (usec): min=9286, max=52130, avg=19134.48, stdev=6331.87 clat percentiles (usec): | 1.00th=[10290], 5.00th=[12256], 10.00th=[12911], 20.00th=[13960], | 30.00th=[15270], 40.00th=[16581], 50.00th=[18220], 60.00th=[19530], | 70.00th=[21103], 80.00th=[22938], 90.00th=[25822], 95.00th=[31589], | 99.00th=[41157], 99.50th=[46924], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 111, max= 216, per=25.62%, avg=179.35, stdev=35.56, samples=17 iops : min= 27, max= 54, avg=44.76, stdev= 9.04, samples=17 lat (usec) : 1000=1.04% lat (msec) : 2=21.35%, 4=16.54%, 10=6.51%, 20=34.77%, 50=19.66% lat (msec) : 100=0.13% cpu : usr=0.14%, sys=1.98%, ctx=883, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=143289: Wed Jan 7 22:01:40 2026 read: IOPS=45, BW=181KiB/s (185kB/s)(1564KiB/8662msec) clat (usec): min=930, max=30989, avg=3068.73, stdev=3562.43 lat (usec): min=930, max=30990, avg=3069.46, stdev=3562.85 clat percentiles (usec): | 1.00th=[ 979], 5.00th=[ 1045], 10.00th=[ 1123], 20.00th=[ 1336], | 30.00th=[ 1500], 40.00th=[ 1696], 50.00th=[ 2008], 60.00th=[ 2474], | 70.00th=[ 2868], 80.00th=[ 3523], 90.00th=[ 5473], 95.00th=[ 7767], | 99.00th=[22152], 99.50th=[25560], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 96, max= 344, per=27.50%, avg=183.18, stdev=79.59, samples=17 iops : min= 24, max= 86, avg=45.71, stdev=19.99, samples=17 write: IOPS=43, BW=174KiB/s (178kB/s)(1508KiB/8662msec); 0 zone resets clat (usec): min=11083, max=61768, avg=19758.31, stdev=6384.57 lat (usec): min=11084, max=61769, avg=19759.23, stdev=6384.60 clat percentiles (usec): | 1.00th=[11863], 5.00th=[12780], 10.00th=[13435], 20.00th=[14746], | 30.00th=[16188], 40.00th=[17433], 50.00th=[19006], 60.00th=[20317], | 70.00th=[21627], 80.00th=[22938], 90.00th=[25297], 95.00th=[33162], | 99.00th=[47449], 99.50th=[48497], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 122, max= 208, per=24.82%, avg=173.71, stdev=28.54, samples=17 iops : min= 30, max= 52, avg=43.35, stdev= 7.27, samples=17 lat (usec) : 1000=1.17% lat (msec) : 2=24.09%, 4=17.19%, 10=6.38%, 20=29.82%, 50=21.22% lat (msec) : 100=0.13% cpu : usr=0.24%, sys=1.95%, ctx=902, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=143290: Wed Jan 7 22:01:40 2026 read: IOPS=42, BW=172KiB/s (176kB/s)(1508KiB/8787msec) clat (usec): min=917, max=24948, avg=3035.79, stdev=3034.26 lat (usec): min=917, max=24949, avg=3036.48, stdev=3034.33 clat percentiles (usec): | 1.00th=[ 979], 5.00th=[ 1106], 10.00th=[ 1188], 20.00th=[ 1369], | 30.00th=[ 1549], 40.00th=[ 1795], 50.00th=[ 2147], 60.00th=[ 2606], | 70.00th=[ 3032], 80.00th=[ 3851], 90.00th=[ 5080], 95.00th=[ 7963], | 99.00th=[19268], 99.50th=[24511], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 88, max= 272, per=25.45%, avg=169.53, stdev=52.81, samples=17 iops : min= 22, max= 68, avg=42.29, stdev=13.26, samples=17 write: IOPS=44, BW=178KiB/s (182kB/s)(1564KiB/8787msec); 0 zone resets clat (usec): min=10472, max=59501, avg=19517.28, stdev=6505.58 lat (usec): min=10473, max=59502, avg=19518.12, stdev=6505.62 clat percentiles (usec): | 1.00th=[11338], 5.00th=[12518], 10.00th=[13304], 20.00th=[14222], | 30.00th=[15401], 40.00th=[16909], 50.00th=[18482], 60.00th=[20055], | 70.00th=[21365], 80.00th=[22938], 90.00th=[26608], 95.00th=[29754], | 99.00th=[51119], 99.50th=[53740], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 122, max= 216, per=25.35%, avg=177.47, stdev=30.48, samples=17 iops : min= 30, max= 54, avg=44.29, stdev= 7.74, samples=17 lat (usec) : 1000=0.52% lat (msec) : 2=22.27%, 4=17.19%, 10=7.42%, 20=31.51%, 50=20.57% lat (msec) : 100=0.52% cpu : usr=0.17%, sys=1.99%, ctx=901, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=666KiB/s (682kB/s), 160KiB/s-181KiB/s (164kB/s-185kB/s), io=5988KiB (6132kB), run=8662-8988msec WRITE: bw=701KiB/s (718kB/s), 174KiB/s-182KiB/s (178kB/s-186kB/s), io=6300KiB (6451kB), run=8662-8988msec