== sanity test 398b: DIO and buffer IO race ============== 10:17:35 (1761315455) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.153606 s, 81.9 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=149003: Fri Oct 24 10:17:48 2025 read: IOPS=67, BW=269KiB/s (275kB/s)(1480KiB/5507msec) clat (usec): min=5, max=82233, avg=8578.26, stdev=10341.01 lat (usec): min=5, max=82233, avg=8578.97, stdev=10341.01 clat percentiles (usec): | 1.00th=[ 10], 5.00th=[ 14], 10.00th=[ 17], 20.00th=[ 22], | 30.00th=[ 29], 40.00th=[ 3884], 50.00th=[ 6980], 60.00th=[ 9896], | 70.00th=[12256], 80.00th=[15270], 90.00th=[18744], 95.00th=[23200], | 99.00th=[48497], 99.50th=[74974], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 116, max= 304, per=24.53%, avg=258.10, stdev=59.69, samples=10 iops : min= 29, max= 76, avg=64.20, stdev=14.94, samples=10 write: IOPS=72, BW=289KiB/s (296kB/s)(1592KiB/5507msec); 0 zone resets clat (usec): min=121, max=75306, avg=5827.88, stdev=8106.03 lat (usec): min=121, max=75307, avg=5828.68, stdev=8106.02 clat percentiles (usec): | 1.00th=[ 165], 5.00th=[ 186], 10.00th=[ 204], 20.00th=[ 247], | 30.00th=[ 310], 40.00th=[ 553], 50.00th=[ 3130], 60.00th=[ 5014], | 70.00th=[ 7177], 80.00th=[10683], 90.00th=[15926], 95.00th=[19792], | 99.00th=[39060], 99.50th=[49021], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 132, max= 400, per=25.34%, avg=280.50, stdev=76.40, samples=10 iops : min= 33, max= 100, avg=69.80, stdev=19.03, samples=10 lat (usec) : 10=1.04%, 20=7.03%, 50=9.77%, 100=0.26%, 250=10.81% lat (usec) : 500=10.16%, 750=1.30%, 1000=0.52% lat (msec) : 2=1.56%, 4=5.08%, 10=22.40%, 20=23.31%, 50=6.25% lat (msec) : 100=0.52% cpu : usr=0.13%, sys=6.18%, ctx=825, majf=0, minf=17 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149004: Fri Oct 24 10:17:48 2025 read: IOPS=66, BW=267KiB/s (273kB/s)(1436KiB/5383msec) clat (usec): min=6, max=81948, avg=8404.35, stdev=10097.16 lat (usec): min=6, max=81949, avg=8405.15, stdev=10097.20 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 13], 10.00th=[ 17], 20.00th=[ 22], | 30.00th=[ 29], 40.00th=[ 2507], 50.00th=[ 6849], 60.00th=[ 9765], | 70.00th=[11731], 80.00th=[14222], 90.00th=[19530], 95.00th=[24511], | 99.00th=[49021], 99.50th=[71828], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 109, max= 438, per=25.38%, avg=267.00, stdev=100.87, samples=10 iops : min= 27, max= 109, avg=66.40, stdev=25.21, samples=10 write: IOPS=75, BW=304KiB/s (311kB/s)(1636KiB/5383msec); 0 zone resets clat (usec): min=155, max=51394, avg=5749.91, stdev=7906.78 lat (usec): min=155, max=51396, avg=5752.55, stdev=7906.88 clat percentiles (usec): | 1.00th=[ 176], 5.00th=[ 190], 10.00th=[ 210], 20.00th=[ 265], | 30.00th=[ 322], 40.00th=[ 603], 50.00th=[ 2802], 60.00th=[ 4621], | 70.00th=[ 6652], 80.00th=[10814], 90.00th=[16057], 95.00th=[19792], | 99.00th=[40633], 99.50th=[46400], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 187, max= 454, per=27.06%, avg=299.60, stdev=81.75, samples=10 iops : min= 46, max= 113, avg=74.50, stdev=20.57, samples=10 lat (usec) : 10=0.91%, 20=6.90%, 50=9.38%, 100=0.13%, 250=9.24% lat (usec) : 500=10.68%, 750=2.60%, 1000=1.69% lat (msec) : 2=2.60%, 4=5.08%, 10=20.96%, 20=23.05%, 50=6.38% lat (msec) : 100=0.39% cpu : usr=0.26%, sys=5.89%, ctx=821, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149005: Fri Oct 24 10:17:48 2025 read: IOPS=68, BW=275KiB/s (282kB/s)(1564KiB/5687msec) clat (usec): min=6, max=41460, avg=7521.17, stdev=8059.74 lat (usec): min=6, max=41461, avg=7526.26, stdev=8065.20 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 11], 10.00th=[ 16], 20.00th=[ 21], | 30.00th=[ 26], 40.00th=[ 408], 50.00th=[ 6128], 60.00th=[ 9241], | 70.00th=[11600], 80.00th=[13960], 90.00th=[17957], 95.00th=[23200], | 99.00th=[33817], 99.50th=[39060], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 87, max= 438, per=26.18%, avg=275.36, stdev=100.68, samples=11 iops : min= 21, max= 109, avg=68.55, stdev=25.17, samples=11 write: IOPS=66, BW=265KiB/s (272kB/s)(1508KiB/5687msec); 0 zone resets clat (usec): min=164, max=123466, avg=7254.57, stdev=10508.55 lat (usec): min=165, max=123469, avg=7255.54, stdev=10508.57 clat percentiles (usec): | 1.00th=[ 182], 5.00th=[ 210], 10.00th=[ 231], 20.00th=[ 293], | 30.00th=[ 441], 40.00th=[ 1827], 50.00th=[ 4359], 60.00th=[ 6456], | 70.00th=[ 8586], 80.00th=[ 11994], 90.00th=[ 18482], 95.00th=[ 24511], | 99.00th=[ 41681], 99.50th=[ 58983], 99.90th=[123208], 99.95th=[123208], | 99.99th=[123208] bw ( KiB/s): min= 135, max= 446, per=23.54%, avg=260.64, stdev=81.68, samples=11 iops : min= 33, max= 111, avg=64.82, stdev=20.41, samples=11 lat (usec) : 10=1.95%, 20=7.94%, 50=9.24%, 100=0.65%, 250=6.64% lat (usec) : 500=9.38%, 750=1.43%, 1000=1.04% lat (msec) : 2=2.34%, 4=4.30%, 10=24.61%, 20=23.18%, 50=7.03% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.18%, sys=5.63%, ctx=853, majf=0, minf=17 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149006: Fri Oct 24 10:17:48 2025 read: IOPS=68, BW=272KiB/s (279kB/s)(1508KiB/5544msec) clat (usec): min=6, max=88683, avg=8546.67, stdev=9752.95 lat (usec): min=6, max=88683, avg=8547.37, stdev=9752.94 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 15], 20.00th=[ 22], | 30.00th=[ 29], 40.00th=[ 3785], 50.00th=[ 7635], 60.00th=[ 9503], | 70.00th=[12125], 80.00th=[15139], 90.00th=[20841], 95.00th=[25560], | 99.00th=[36439], 99.50th=[44303], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 109, max= 392, per=25.77%, avg=271.09, stdev=78.95, samples=11 iops : min= 27, max= 98, avg=67.45, stdev=19.88, samples=11 write: IOPS=70, BW=282KiB/s (289kB/s)(1564KiB/5544msec); 0 zone resets clat (usec): min=119, max=122873, avg=5911.45, stdev=9597.49 lat (usec): min=119, max=122874, avg=5912.37, stdev=9597.65 clat percentiles (usec): | 1.00th=[ 161], 5.00th=[ 192], 10.00th=[ 217], 20.00th=[ 249], | 30.00th=[ 306], 40.00th=[ 545], 50.00th=[ 2638], 60.00th=[ 4686], | 70.00th=[ 6783], 80.00th=[ 10028], 90.00th=[ 15139], 95.00th=[ 23200], | 99.00th=[ 36963], 99.50th=[ 40633], 99.90th=[123208], 99.95th=[123208], | 99.99th=[123208] bw ( KiB/s): min= 141, max= 432, per=25.61%, avg=283.45, stdev=91.66, samples=11 iops : min= 35, max= 108, avg=70.55, stdev=22.98, samples=11 lat (usec) : 10=1.43%, 20=7.03%, 50=9.77%, 100=0.39%, 250=10.68% lat (usec) : 500=9.51%, 750=1.95%, 1000=1.17% lat (msec) : 2=1.82%, 4=4.69%, 10=22.92%, 20=19.66%, 50=8.72% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.22%, sys=5.70%, ctx=844, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1053KiB/s (1078kB/s), 267KiB/s-275KiB/s (273kB/s-282kB/s), io=5988KiB (6132kB), run=5383-5687msec WRITE: bw=1108KiB/s (1134kB/s), 265KiB/s-304KiB/s (272kB/s-311kB/s), io=6300KiB (6451kB), run=5383-5687msec rand-rw: (groupid=0, jobs=1): err= 0: pid=148998: Fri Oct 24 10:17:54 2025 read: IOPS=32, BW=130KiB/s (134kB/s)(1480KiB/11351msec) clat (usec): min=2213, max=44889, avg=9568.33, stdev=6951.95 lat (usec): min=2214, max=44892, avg=9569.40, stdev=6951.86 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3425], 10.00th=[ 3982], 20.00th=[ 4686], | 30.00th=[ 5473], 40.00th=[ 6063], 50.00th=[ 7177], 60.00th=[ 8291], | 70.00th=[ 9896], 80.00th=[12780], 90.00th=[18744], 95.00th=[25297], | 99.00th=[35390], 99.50th=[40633], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 64, max= 227, per=24.80%, avg=128.73, stdev=44.51, samples=22 iops : min= 16, max= 56, avg=31.68, stdev=10.96, samples=22 write: IOPS=35, BW=140KiB/s (144kB/s)(1592KiB/11351msec); 0 zone resets clat (msec): min=4, max=131, avg=19.52, stdev=10.93 lat (msec): min=4, max=131, avg=19.52, stdev=10.93 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 23], 80.00th=[ 26], 90.00th=[ 31], 95.00th=[ 37], | 99.00th=[ 51], 99.50th=[ 103], 99.90th=[ 132], 99.95th=[ 132], | 99.99th=[ 132] bw ( KiB/s): min= 79, max= 239, per=25.65%, avg=140.05, stdev=39.26, samples=22 iops : min= 19, max= 59, avg=34.55, stdev= 9.68, samples=22 lat (msec) : 4=5.08%, 10=33.46%, 20=37.89%, 50=22.92%, 100=0.39% lat (msec) : 250=0.26% cpu : usr=0.25%, sys=1.69%, ctx=845, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148999: Fri Oct 24 10:17:54 2025 read: IOPS=31, BW=125KiB/s (128kB/s)(1436KiB/11533msec) clat (msec): min=2, max=115, avg=10.10, stdev= 9.14 lat (msec): min=2, max=115, avg=10.10, stdev= 9.14 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 13], 90.00th=[ 20], 95.00th=[ 25], | 99.00th=[ 44], 99.50th=[ 57], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 55, max= 207, per=23.86%, avg=123.82, stdev=41.11, samples=22 iops : min= 13, max= 51, avg=30.45, stdev=10.23, samples=22 write: IOPS=35, BW=142KiB/s (145kB/s)(1636KiB/11533msec); 0 zone resets clat (usec): min=5012, max=63260, avg=19262.84, stdev=8258.47 lat (usec): min=5013, max=63261, avg=19264.38, stdev=8258.48 clat percentiles (usec): | 1.00th=[ 6390], 5.00th=[ 8717], 10.00th=[10421], 20.00th=[12649], | 30.00th=[14222], 40.00th=[15926], 50.00th=[17695], 60.00th=[19792], | 70.00th=[21890], 80.00th=[24511], 90.00th=[29754], 95.00th=[34866], | 99.00th=[45876], 99.50th=[49021], 99.90th=[63177], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 80, max= 207, per=25.67%, avg=140.18, stdev=29.74, samples=22 iops : min= 20, max= 51, avg=34.55, stdev= 7.29, samples=22 lat (msec) : 4=4.95%, 10=30.73%, 20=39.19%, 50=24.74%, 100=0.26% lat (msec) : 250=0.13% cpu : usr=0.11%, sys=1.73%, ctx=818, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149000: Fri Oct 24 10:17:54 2025 read: IOPS=34, BW=139KiB/s (142kB/s)(1564KiB/11286msec) clat (usec): min=1869, max=149093, avg=9609.67, stdev=9965.98 lat (usec): min=1869, max=149095, avg=9610.80, stdev=9965.91 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 13], 90.00th=[ 18], 95.00th=[ 24], | 99.00th=[ 38], 99.50th=[ 63], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 80, max= 287, per=26.99%, avg=140.09, stdev=55.09, samples=22 iops : min= 20, max= 71, avg=34.55, stdev=13.68, samples=22 write: IOPS=33, BW=134KiB/s (137kB/s)(1508KiB/11286msec); 0 zone resets clat (usec): min=4783, max=57975, avg=19860.69, stdev=8443.11 lat (usec): min=4785, max=57977, avg=19863.75, stdev=8442.05 clat percentiles (usec): | 1.00th=[ 5997], 5.00th=[ 9503], 10.00th=[11469], 20.00th=[13042], | 30.00th=[15008], 40.00th=[16581], 50.00th=[17957], 60.00th=[20055], | 70.00th=[22414], 80.00th=[25297], 90.00th=[30540], 95.00th=[36439], | 99.00th=[50070], 99.50th=[54789], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 72, max= 199, per=24.32%, avg=132.77, stdev=34.54, samples=22 iops : min= 18, max= 49, avg=32.73, stdev= 8.53, samples=22 lat (msec) : 2=0.13%, 4=6.90%, 10=30.86%, 20=38.15%, 50=23.18% lat (msec) : 100=0.65%, 250=0.13% cpu : usr=0.13%, sys=1.69%, ctx=856, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149001: Fri Oct 24 10:17:54 2025 read: IOPS=32, BW=132KiB/s (135kB/s)(1508KiB/11458msec) clat (msec): min=2, max=141, avg=10.18, stdev=10.11 lat (msec): min=2, max=141, avg=10.19, stdev=10.11 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 10], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 21], 95.00th=[ 25], | 99.00th=[ 50], 99.50th=[ 60], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 46, max= 199, per=25.00%, avg=129.73, stdev=45.14, samples=22 iops : min= 11, max= 49, avg=31.91, stdev=11.11, samples=22 write: IOPS=34, BW=136KiB/s (140kB/s)(1564KiB/11458msec); 0 zone resets clat (usec): min=4283, max=62480, avg=19354.50, stdev=8407.17 lat (usec): min=4285, max=62481, avg=19355.78, stdev=8407.19 clat percentiles (usec): | 1.00th=[ 7111], 5.00th=[ 9241], 10.00th=[10945], 20.00th=[13173], | 30.00th=[14484], 40.00th=[15926], 50.00th=[17695], 60.00th=[19006], | 70.00th=[21365], 80.00th=[25297], 90.00th=[29492], 95.00th=[36963], | 99.00th=[50070], 99.50th=[53216], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 87, max= 207, per=24.92%, avg=136.05, stdev=34.95, samples=22 iops : min= 21, max= 51, avg=33.55, stdev= 8.67, samples=22 lat (msec) : 4=6.51%, 10=29.43%, 20=40.76%, 50=22.53%, 100=0.65% lat (msec) : 250=0.13% cpu : usr=0.15%, sys=1.68%, ctx=825, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=519KiB/s (532kB/s), 125KiB/s-139KiB/s (128kB/s-142kB/s), io=5988KiB (6132kB), run=11286-11533msec WRITE: bw=546KiB/s (559kB/s), 134KiB/s-142KiB/s (137kB/s-145kB/s), io=6300KiB (6451kB), run=11286-11533msec