== sanity test 398b: DIO and buffer IO race ============== 09:54:15 (1761314055) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.242513 s, 51.9 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148979: Fri Oct 24 09:54:30 2025 read: IOPS=65, BW=261KiB/s (267kB/s)(1480KiB/5667msec) clat (usec): min=6, max=49481, avg=8352.89, stdev=8640.71 lat (usec): min=6, max=49481, avg=8353.67, stdev=8640.68 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 15], 10.00th=[ 19], 20.00th=[ 24], | 30.00th=[ 32], 40.00th=[ 4146], 50.00th=[ 7439], 60.00th=[10028], | 70.00th=[12125], 80.00th=[14615], 90.00th=[19530], 95.00th=[23462], | 99.00th=[36439], 99.50th=[48497], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 87, max= 398, per=26.09%, avg=253.09, stdev=90.08, samples=11 iops : min= 21, max= 99, avg=62.82, stdev=22.63, samples=11 write: IOPS=70, BW=281KiB/s (288kB/s)(1592KiB/5667msec); 0 zone resets clat (usec): min=153, max=61211, avg=6333.79, stdev=8434.82 lat (usec): min=153, max=61211, avg=6338.15, stdev=8438.70 clat percentiles (usec): | 1.00th=[ 178], 5.00th=[ 202], 10.00th=[ 212], 20.00th=[ 255], | 30.00th=[ 326], 40.00th=[ 938], 50.00th=[ 3949], 60.00th=[ 5211], | 70.00th=[ 7373], 80.00th=[11469], 90.00th=[16909], 95.00th=[22676], | 99.00th=[43254], 99.50th=[50594], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 127, max= 430, per=27.05%, avg=276.18, stdev=101.38, samples=11 iops : min= 31, max= 107, avg=68.64, stdev=25.41, samples=11 lat (usec) : 10=1.04%, 20=5.47%, 50=10.42%, 100=0.39%, 250=9.77% lat (usec) : 500=9.90%, 750=0.91%, 1000=1.30% lat (msec) : 2=1.69%, 4=4.43%, 10=23.05%, 20=23.31%, 50=8.07% lat (msec) : 100=0.26% cpu : usr=0.21%, sys=5.75%, ctx=828, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148980: Fri Oct 24 09:54:30 2025 read: IOPS=59, BW=239KiB/s (245kB/s)(1436KiB/6014msec) clat (usec): min=7, max=68565, avg=9257.73, stdev=9487.89 lat (usec): min=7, max=68566, avg=9259.03, stdev=9487.79 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 14], 10.00th=[ 19], 20.00th=[ 25], | 30.00th=[ 34], 40.00th=[ 5932], 50.00th=[ 8356], 60.00th=[11207], | 70.00th=[13566], 80.00th=[15926], 90.00th=[20579], 95.00th=[24249], | 99.00th=[36963], 99.50th=[52167], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 151, max= 383, per=24.52%, avg=237.82, stdev=63.58, samples=11 iops : min= 37, max= 95, avg=59.00, stdev=15.89, samples=11 write: IOPS=68, BW=272KiB/s (279kB/s)(1636KiB/6014msec); 0 zone resets clat (usec): min=157, max=41285, avg=6462.24, stdev=8093.00 lat (usec): min=157, max=41286, avg=6463.11, stdev=8093.01 clat percentiles (usec): | 1.00th=[ 180], 5.00th=[ 200], 10.00th=[ 221], 20.00th=[ 273], | 30.00th=[ 375], 40.00th=[ 1221], 50.00th=[ 3949], 60.00th=[ 5538], | 70.00th=[ 7504], 80.00th=[10945], 90.00th=[18220], 95.00th=[24773], | 99.00th=[33817], 99.50th=[38536], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 159, max= 366, per=25.95%, avg=264.91, stdev=63.76, samples=11 iops : min= 39, max= 91, avg=65.82, stdev=15.90, samples=11 lat (usec) : 10=1.04%, 20=4.95%, 50=10.03%, 100=0.13%, 250=9.24% lat (usec) : 500=9.11%, 750=1.04%, 1000=1.43% lat (msec) : 2=2.73%, 4=3.78%, 10=24.09%, 20=22.79%, 50=9.38% lat (msec) : 100=0.26% cpu : usr=0.22%, sys=5.97%, ctx=871, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148981: Fri Oct 24 09:54:30 2025 read: IOPS=63, BW=254KiB/s (260kB/s)(1564KiB/6167msec) clat (usec): min=7, max=59484, avg=9476.20, stdev=9836.34 lat (usec): min=7, max=59484, avg=9477.22, stdev=9836.14 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 14], 10.00th=[ 20], 20.00th=[ 26], | 30.00th=[ 46], 40.00th=[ 5866], 50.00th=[ 8291], 60.00th=[10683], | 70.00th=[13173], 80.00th=[15926], 90.00th=[19792], 95.00th=[25297], | 99.00th=[46400], 99.50th=[54789], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 137, max= 415, per=26.07%, avg=252.83, stdev=93.28, samples=12 iops : min= 34, max= 103, avg=62.83, stdev=23.30, samples=12 write: IOPS=61, BW=245KiB/s (250kB/s)(1508KiB/6167msec); 0 zone resets clat (usec): min=166, max=71793, avg=6400.48, stdev=8651.57 lat (usec): min=167, max=71794, avg=6401.49, stdev=8651.61 clat percentiles (usec): | 1.00th=[ 182], 5.00th=[ 196], 10.00th=[ 219], 20.00th=[ 260], | 30.00th=[ 347], 40.00th=[ 832], 50.00th=[ 3982], 60.00th=[ 5800], | 70.00th=[ 7898], 80.00th=[10945], 90.00th=[16319], 95.00th=[22152], | 99.00th=[41157], 99.50th=[60556], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 151, max= 398, per=23.57%, avg=240.67, stdev=72.21, samples=12 iops : min= 37, max= 99, avg=59.75, stdev=18.16, samples=12 lat (usec) : 10=1.04%, 20=4.43%, 50=10.03%, 100=0.13%, 250=8.85% lat (usec) : 500=9.64%, 750=1.30%, 1000=0.52% lat (msec) : 2=1.95%, 4=4.95%, 10=24.22%, 20=25.26%, 50=7.16% lat (msec) : 100=0.52% cpu : usr=0.24%, sys=5.87%, ctx=900, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148982: Fri Oct 24 09:54:30 2025 read: IOPS=65, BW=262KiB/s (269kB/s)(1508KiB/5750msec) clat (usec): min=6, max=56121, avg=8931.92, stdev=9521.69 lat (usec): min=6, max=56123, avg=8932.63, stdev=9521.73 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 12], 10.00th=[ 16], 20.00th=[ 23], | 30.00th=[ 30], 40.00th=[ 4555], 50.00th=[ 7963], 60.00th=[10552], | 70.00th=[13042], 80.00th=[15795], 90.00th=[21103], 95.00th=[26346], | 99.00th=[41157], 99.50th=[52691], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 158, max= 424, per=26.76%, avg=259.55, stdev=88.95, samples=11 iops : min= 39, max= 106, avg=64.45, stdev=22.22, samples=11 write: IOPS=68, BW=272KiB/s (279kB/s)(1564KiB/5750msec); 0 zone resets clat (usec): min=147, max=42757, avg=6028.64, stdev=7876.05 lat (usec): min=147, max=42759, avg=6029.78, stdev=7876.22 clat percentiles (usec): | 1.00th=[ 172], 5.00th=[ 194], 10.00th=[ 215], 20.00th=[ 253], | 30.00th=[ 318], 40.00th=[ 578], 50.00th=[ 2409], 60.00th=[ 4883], | 70.00th=[ 7832], 80.00th=[11076], 90.00th=[16188], 95.00th=[24249], | 99.00th=[32637], 99.50th=[36963], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 172, max= 447, per=26.83%, avg=273.91, stdev=91.29, samples=11 iops : min= 43, max= 111, avg=68.09, stdev=22.72, samples=11 lat (usec) : 10=1.82%, 20=5.86%, 50=10.29%, 100=0.39%, 250=10.29% lat (usec) : 500=9.51%, 750=1.95%, 1000=1.17% lat (msec) : 2=2.21%, 4=3.39%, 10=21.35%, 20=22.01%, 50=9.51% lat (msec) : 100=0.26% cpu : usr=0.14%, sys=5.81%, ctx=790, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=971KiB/s (994kB/s), 239KiB/s-262KiB/s (245kB/s-269kB/s), io=5988KiB (6132kB), run=5667-6167msec WRITE: bw=1022KiB/s (1046kB/s), 245KiB/s-281KiB/s (250kB/s-288kB/s), io=6300KiB (6451kB), run=5667-6167msec rand-rw: (groupid=0, jobs=1): err= 0: pid=148984: Fri Oct 24 09:54:35 2025 read: IOPS=34, BW=136KiB/s (139kB/s)(1480KiB/10876msec) clat (usec): min=2495, max=65897, avg=9857.90, stdev=8865.88 lat (usec): min=2495, max=65898, avg=9858.91, stdev=8865.86 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 3294], 10.00th=[ 3720], 20.00th=[ 4424], | 30.00th=[ 5080], 40.00th=[ 5866], 50.00th=[ 6652], 60.00th=[ 7832], | 70.00th=[ 9634], 80.00th=[11863], 90.00th=[20841], 95.00th=[28967], | 99.00th=[44827], 99.50th=[62653], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 55, max= 231, per=24.36%, avg=134.00, stdev=45.11, samples=21 iops : min= 13, max= 57, avg=33.05, stdev=11.28, samples=21 write: IOPS=36, BW=146KiB/s (150kB/s)(1592KiB/10876msec); 0 zone resets clat (usec): min=5591, max=96330, avg=18068.31, stdev=9776.12 lat (usec): min=5593, max=96331, avg=18069.66, stdev=9776.09 clat percentiles (usec): | 1.00th=[ 6194], 5.00th=[ 7701], 10.00th=[ 9241], 20.00th=[11207], | 30.00th=[12649], 40.00th=[13960], 50.00th=[15270], 60.00th=[17433], | 70.00th=[20317], 80.00th=[23462], 90.00th=[28967], 95.00th=[35914], | 99.00th=[51119], 99.50th=[71828], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 55, max= 240, per=25.25%, avg=146.19, stdev=55.10, samples=21 iops : min= 13, max= 60, avg=36.10, stdev=13.83, samples=21 lat (msec) : 4=7.81%, 10=34.38%, 20=35.68%, 50=21.09%, 100=1.04% cpu : usr=0.14%, sys=1.89%, ctx=819, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148985: Fri Oct 24 09:54:35 2025 read: IOPS=33, BW=135KiB/s (138kB/s)(1436KiB/10665msec) clat (usec): min=2080, max=68408, avg=9101.67, stdev=8062.40 lat (usec): min=2081, max=68410, avg=9102.58, stdev=8062.39 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 3195], 10.00th=[ 3720], 20.00th=[ 4359], | 30.00th=[ 4883], 40.00th=[ 5473], 50.00th=[ 6325], 60.00th=[ 7439], | 70.00th=[ 8848], 80.00th=[11076], 90.00th=[18744], 95.00th=[27132], | 99.00th=[36963], 99.50th=[57934], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 41, max= 239, per=24.39%, avg=134.14, stdev=52.63, samples=21 iops : min= 10, max= 59, avg=33.10, stdev=13.13, samples=21 write: IOPS=38, BW=153KiB/s (157kB/s)(1636KiB/10665msec); 0 zone resets clat (usec): min=5049, max=91309, avg=18016.27, stdev=9202.06 lat (usec): min=5051, max=91310, avg=18017.78, stdev=9201.79 clat percentiles (usec): | 1.00th=[ 6063], 5.00th=[ 8291], 10.00th=[ 9503], 20.00th=[11338], | 30.00th=[13042], 40.00th=[14484], 50.00th=[15795], 60.00th=[17171], | 70.00th=[20055], 80.00th=[23725], 90.00th=[28967], 95.00th=[33817], | 99.00th=[50594], 99.50th=[54789], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 97, max= 240, per=26.45%, avg=153.14, stdev=43.59, samples=21 iops : min= 24, max= 60, avg=37.86, stdev=10.93, samples=21 lat (msec) : 4=6.90%, 10=35.68%, 20=36.98%, 50=19.53%, 100=0.91% cpu : usr=0.23%, sys=1.81%, ctx=822, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148986: Fri Oct 24 09:54:35 2025 read: IOPS=37, BW=148KiB/s (152kB/s)(1564KiB/10549msec) clat (usec): min=1889, max=41814, avg=8927.65, stdev=7285.42 lat (usec): min=1890, max=41815, avg=8928.62, stdev=7285.39 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 3130], 10.00th=[ 3621], 20.00th=[ 4293], | 30.00th=[ 4883], 40.00th=[ 5604], 50.00th=[ 6259], 60.00th=[ 7177], | 70.00th=[ 8717], 80.00th=[11731], 90.00th=[18482], 95.00th=[27132], | 99.00th=[37487], 99.50th=[38011], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 62, max= 264, per=27.02%, avg=148.60, stdev=61.09, samples=20 iops : min= 15, max= 66, avg=36.65, stdev=15.23, samples=20 write: IOPS=35, BW=143KiB/s (146kB/s)(1508KiB/10549msec); 0 zone resets clat (usec): min=6259, max=78256, avg=18636.14, stdev=9877.55 lat (usec): min=6260, max=78259, avg=18637.54, stdev=9877.61 clat percentiles (usec): | 1.00th=[ 7177], 5.00th=[ 8586], 10.00th=[ 9634], 20.00th=[11207], | 30.00th=[13042], 40.00th=[14353], 50.00th=[15926], 60.00th=[17957], | 70.00th=[20317], 80.00th=[23987], 90.00th=[30540], 95.00th=[39584], | 99.00th=[53216], 99.50th=[63177], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 69, max= 207, per=24.15%, avg=139.85, stdev=46.25, samples=20 iops : min= 17, max= 51, avg=34.50, stdev=11.56, samples=20 lat (msec) : 2=0.13%, 4=7.03%, 10=36.33%, 20=36.72%, 50=19.01% lat (msec) : 100=0.78% cpu : usr=0.21%, sys=1.84%, ctx=817, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148987: Fri Oct 24 09:54:35 2025 read: IOPS=35, BW=141KiB/s (145kB/s)(1508KiB/10686msec) clat (usec): min=2181, max=66564, avg=9925.63, stdev=9092.73 lat (usec): min=2182, max=66564, avg=9926.99, stdev=9092.70 clat percentiles (usec): | 1.00th=[ 2278], 5.00th=[ 3130], 10.00th=[ 3556], 20.00th=[ 4178], | 30.00th=[ 4817], 40.00th=[ 5473], 50.00th=[ 6325], 60.00th=[ 7767], | 70.00th=[ 9503], 80.00th=[13829], 90.00th=[22676], 95.00th=[30016], | 99.00th=[44303], 99.50th=[62129], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 64, max= 312, per=25.50%, avg=140.24, stdev=59.34, samples=21 iops : min= 16, max= 78, avg=34.62, stdev=14.91, samples=21 write: IOPS=36, BW=146KiB/s (150kB/s)(1564KiB/10686msec); 0 zone resets clat (usec): min=3624, max=68167, avg=17693.85, stdev=9671.92 lat (usec): min=3625, max=68169, avg=17695.16, stdev=9671.94 clat percentiles (usec): | 1.00th=[ 5276], 5.00th=[ 8029], 10.00th=[ 9372], 20.00th=[11076], | 30.00th=[12518], 40.00th=[13698], 50.00th=[15139], 60.00th=[16450], | 70.00th=[19006], 80.00th=[23462], 90.00th=[28705], 95.00th=[35914], | 99.00th=[60556], 99.50th=[65274], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 71, max= 231, per=25.24%, avg=146.14, stdev=53.61, samples=21 iops : min= 17, max= 57, avg=36.10, stdev=13.42, samples=21 lat (msec) : 4=8.59%, 10=33.72%, 20=38.54%, 50=17.84%, 100=1.30% cpu : usr=0.15%, sys=1.89%, ctx=822, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=551KiB/s (564kB/s), 135KiB/s-148KiB/s (138kB/s-152kB/s), io=5988KiB (6132kB), run=10549-10876msec WRITE: bw=579KiB/s (593kB/s), 143KiB/s-153KiB/s (146kB/s-157kB/s), io=6300KiB (6451kB), run=10549-10876msec