== sanity test 398b: DIO and buffer IO race ============== 09:23:53 (1774531433) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.106876 s, 118 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147782: Thu Mar 26 09:24:02 2026 read: IOPS=131, BW=528KiB/s (540kB/s)(1480KiB/2804msec) clat (usec): min=4, max=45656, avg=4366.27, stdev=5195.78 lat (usec): min=4, max=45657, avg=4366.74, stdev=5195.81 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 13], 10.00th=[ 16], 20.00th=[ 22], | 30.00th=[ 31], 40.00th=[ 2180], 50.00th=[ 3687], 60.00th=[ 4817], | 70.00th=[ 5932], 80.00th=[ 6849], 90.00th=[ 9372], 95.00th=[12256], | 99.00th=[26870], 99.50th=[31065], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 328, max= 688, per=23.65%, avg=505.00, stdev=138.88, samples=5 iops : min= 82, max= 172, avg=126.20, stdev=34.69, samples=5 write: IOPS=141, BW=568KiB/s (581kB/s)(1592KiB/2804msec); 0 zone resets clat (usec): min=129, max=48790, avg=2966.62, stdev=4083.36 lat (usec): min=129, max=48790, avg=2967.18, stdev=4083.35 clat percentiles (usec): | 1.00th=[ 149], 5.00th=[ 204], 10.00th=[ 258], 20.00th=[ 318], | 30.00th=[ 383], 40.00th=[ 1467], 50.00th=[ 2073], 60.00th=[ 2573], | 70.00th=[ 3523], 80.00th=[ 5014], 90.00th=[ 6915], 95.00th=[ 8848], | 99.00th=[13960], 99.50th=[37487], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 367, max= 696, per=24.55%, avg=551.40, stdev=140.93, samples=5 iops : min= 91, max= 174, avg=137.60, stdev=35.56, samples=5 lat (usec) : 10=0.78%, 20=7.81%, 50=7.03%, 100=0.39%, 250=4.82% lat (usec) : 500=13.28%, 750=1.56%, 1000=1.04% lat (msec) : 2=7.03%, 4=19.79%, 10=30.47%, 20=4.95%, 50=1.04% cpu : usr=0.18%, sys=13.49%, ctx=961, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147783: Thu Mar 26 09:24:02 2026 read: IOPS=128, BW=514KiB/s (526kB/s)(1436KiB/2796msec) clat (usec): min=5, max=34221, avg=4419.17, stdev=4872.63 lat (usec): min=5, max=34221, avg=4419.64, stdev=4872.64 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 15], 10.00th=[ 18], 20.00th=[ 23], | 30.00th=[ 100], 40.00th=[ 2737], 50.00th=[ 3851], 60.00th=[ 4948], | 70.00th=[ 6063], 80.00th=[ 7046], 90.00th=[ 8979], 95.00th=[10814], | 99.00th=[28443], 99.50th=[34341], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 367, max= 712, per=24.25%, avg=517.80, stdev=146.56, samples=5 iops : min= 91, max= 178, avg=129.20, stdev=36.90, samples=5 write: IOPS=146, BW=585KiB/s (599kB/s)(1636KiB/2796msec); 0 zone resets clat (usec): min=130, max=20733, avg=2935.61, stdev=3263.84 lat (usec): min=130, max=20734, avg=2936.18, stdev=3263.84 clat percentiles (usec): | 1.00th=[ 157], 5.00th=[ 212], 10.00th=[ 243], 20.00th=[ 302], | 30.00th=[ 420], 40.00th=[ 1483], 50.00th=[ 1958], 60.00th=[ 2540], | 70.00th=[ 3556], 80.00th=[ 4817], 90.00th=[ 7308], 95.00th=[ 9634], | 99.00th=[13829], 99.50th=[14877], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 444, max= 832, per=26.02%, avg=584.40, stdev=149.55, samples=5 iops : min= 111, max= 208, avg=146.00, stdev=37.46, samples=5 lat (usec) : 10=0.78%, 20=6.51%, 50=6.51%, 100=0.26%, 250=6.38% lat (usec) : 500=11.46%, 750=1.30%, 1000=0.52% lat (msec) : 2=8.85%, 4=20.70%, 10=31.51%, 20=4.30%, 50=0.91% cpu : usr=0.32%, sys=12.31%, ctx=962, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147784: Thu Mar 26 09:24:02 2026 read: IOPS=140, BW=561KiB/s (574kB/s)(1564KiB/2788msec) clat (usec): min=5, max=59556, avg=4179.01, stdev=4771.83 lat (usec): min=5, max=59556, avg=4179.58, stdev=4771.77 clat percentiles (usec): | 1.00th=[ 7], 5.00th=[ 13], 10.00th=[ 17], 20.00th=[ 22], | 30.00th=[ 34], 40.00th=[ 2966], 50.00th=[ 4015], 60.00th=[ 5014], | 70.00th=[ 5735], 80.00th=[ 6783], 90.00th=[ 8291], 95.00th=[ 9896], | 99.00th=[19268], 99.50th=[24249], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 386, max= 728, per=26.97%, avg=575.80, stdev=135.35, samples=5 iops : min= 96, max= 182, avg=143.80, stdev=34.02, samples=5 write: IOPS=135, BW=541KiB/s (554kB/s)(1508KiB/2788msec); 0 zone resets clat (usec): min=130, max=55107, avg=3038.13, stdev=4285.21 lat (usec): min=130, max=55108, avg=3038.80, stdev=4285.21 clat percentiles (usec): | 1.00th=[ 167], 5.00th=[ 204], 10.00th=[ 227], 20.00th=[ 281], | 30.00th=[ 355], 40.00th=[ 775], 50.00th=[ 1942], 60.00th=[ 2442], | 70.00th=[ 3458], 80.00th=[ 5014], 90.00th=[ 7570], 95.00th=[ 9896], | 99.00th=[15008], 99.50th=[22938], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 422, max= 640, per=23.84%, avg=535.40, stdev=83.36, samples=5 iops : min= 105, max= 160, avg=133.60, stdev=21.10, samples=5 lat (usec) : 10=1.30%, 20=7.94%, 50=7.29%, 100=0.13%, 250=7.16% lat (usec) : 500=10.42%, 750=2.08%, 1000=0.13% lat (msec) : 2=6.64%, 4=18.23%, 10=33.85%, 20=4.17%, 50=0.39% lat (msec) : 100=0.26% cpu : usr=0.25%, sys=12.02%, ctx=967, majf=0, minf=17 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147785: Thu Mar 26 09:24:02 2026 read: IOPS=142, BW=568KiB/s (582kB/s)(1508KiB/2653msec) clat (usec): min=6, max=32617, avg=3925.28, stdev=4232.91 lat (usec): min=6, max=32618, avg=3925.72, stdev=4232.94 clat percentiles (usec): | 1.00th=[ 10], 5.00th=[ 13], 10.00th=[ 18], 20.00th=[ 22], | 30.00th=[ 28], 40.00th=[ 2606], 50.00th=[ 3458], 60.00th=[ 4293], | 70.00th=[ 5407], 80.00th=[ 6718], 90.00th=[ 8455], 95.00th=[11338], | 99.00th=[17171], 99.50th=[28443], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 400, max= 888, per=27.03%, avg=577.00, stdev=186.66, samples=5 iops : min= 100, max= 222, avg=144.20, stdev=46.66, samples=5 write: IOPS=147, BW=590KiB/s (604kB/s)(1564KiB/2653msec); 0 zone resets clat (usec): min=140, max=46554, avg=2977.24, stdev=4266.38 lat (usec): min=140, max=46555, avg=2977.81, stdev=4266.37 clat percentiles (usec): | 1.00th=[ 176], 5.00th=[ 202], 10.00th=[ 229], 20.00th=[ 285], | 30.00th=[ 334], 40.00th=[ 457], 50.00th=[ 1762], 60.00th=[ 2311], | 70.00th=[ 3261], 80.00th=[ 5014], 90.00th=[ 7832], 95.00th=[10028], | 99.00th=[18744], 99.50th=[27919], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 451, max= 872, per=26.79%, avg=601.80, stdev=167.96, samples=5 iops : min= 112, max= 218, avg=150.20, stdev=42.24, samples=5 lat (usec) : 10=0.91%, 20=6.90%, 50=8.46%, 250=7.29%, 500=14.19% lat (usec) : 750=1.30%, 1000=0.39% lat (msec) : 2=6.12%, 4=19.92%, 10=28.91%, 20=4.95%, 50=0.65% cpu : usr=0.45%, sys=11.99%, ctx=929, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2136KiB/s (2187kB/s), 514KiB/s-568KiB/s (526kB/s-582kB/s), io=5988KiB (6132kB), run=2653-2804msec WRITE: bw=2247KiB/s (2301kB/s), 541KiB/s-590KiB/s (554kB/s-604kB/s), io=6300KiB (6451kB), run=2653-2804msec rand-rw: (groupid=0, jobs=1): err= 0: pid=147777: Thu Mar 26 09:24:04 2026 read: IOPS=86, BW=348KiB/s (356kB/s)(1480KiB/4255msec) clat (usec): min=968, max=50406, avg=3940.01, stdev=5078.06 lat (usec): min=969, max=50407, avg=3940.75, stdev=5078.05 clat percentiles (usec): | 1.00th=[ 1012], 5.00th=[ 1205], 10.00th=[ 1287], 20.00th=[ 1467], | 30.00th=[ 1631], 40.00th=[ 1942], 50.00th=[ 2245], 60.00th=[ 2802], | 70.00th=[ 3687], 80.00th=[ 5014], 90.00th=[ 8094], 95.00th=[11731], | 99.00th=[27657], 99.50th=[47973], 99.90th=[50594], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 178, max= 560, per=24.46%, avg=344.12, stdev=151.18, samples=8 iops : min= 44, max= 140, avg=85.88, stdev=37.97, samples=8 write: IOPS=93, BW=374KiB/s (383kB/s)(1592KiB/4255msec); 0 zone resets clat (usec): min=1801, max=37455, avg=6981.26, stdev=4648.28 lat (usec): min=1802, max=37456, avg=6982.16, stdev=4648.28 clat percentiles (usec): | 1.00th=[ 2180], 5.00th=[ 2868], 10.00th=[ 3163], 20.00th=[ 3785], | 30.00th=[ 4228], 40.00th=[ 4948], 50.00th=[ 5669], 60.00th=[ 6456], | 70.00th=[ 7439], 80.00th=[ 9634], 90.00th=[12125], 95.00th=[15139], | 99.00th=[29754], 99.50th=[34341], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 205, max= 648, per=25.10%, avg=371.50, stdev=174.08, samples=8 iops : min= 51, max= 162, avg=92.75, stdev=43.65, samples=8 lat (usec) : 1000=0.39% lat (msec) : 2=19.92%, 4=27.60%, 10=38.93%, 20=11.20%, 50=1.82% lat (msec) : 100=0.13% cpu : usr=0.24%, sys=3.83%, ctx=830, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147778: Thu Mar 26 09:24:04 2026 read: IOPS=85, BW=341KiB/s (350kB/s)(1436KiB/4205msec) clat (usec): min=935, max=33604, avg=3628.98, stdev=3776.14 lat (usec): min=935, max=33605, avg=3629.54, stdev=3776.18 clat percentiles (usec): | 1.00th=[ 1057], 5.00th=[ 1188], 10.00th=[ 1287], 20.00th=[ 1467], | 30.00th=[ 1680], 40.00th=[ 1942], 50.00th=[ 2245], 60.00th=[ 2769], | 70.00th=[ 3458], 80.00th=[ 5211], 90.00th=[ 7832], 95.00th=[10552], | 99.00th=[17433], 99.50th=[33424], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 212, max= 592, per=24.48%, avg=344.37, stdev=155.77, samples=8 iops : min= 53, max= 148, avg=86.00, stdev=39.00, samples=8 write: IOPS=97, BW=389KiB/s (398kB/s)(1636KiB/4205msec); 0 zone resets clat (usec): min=2011, max=33285, avg=7037.27, stdev=4313.15 lat (usec): min=2011, max=33286, avg=7038.03, stdev=4313.20 clat percentiles (usec): | 1.00th=[ 2212], 5.00th=[ 2802], 10.00th=[ 3195], 20.00th=[ 3621], | 30.00th=[ 4080], 40.00th=[ 4686], 50.00th=[ 5604], 60.00th=[ 6718], | 70.00th=[ 8356], 80.00th=[ 9896], 90.00th=[12649], 95.00th=[16319], | 99.00th=[22676], 99.50th=[23462], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 258, max= 704, per=26.22%, avg=388.12, stdev=156.87, samples=8 iops : min= 64, max= 176, avg=96.88, stdev=39.36, samples=8 lat (usec) : 1000=0.13% lat (msec) : 2=19.79%, 4=29.04%, 10=38.28%, 20=11.59%, 50=1.17% cpu : usr=0.31%, sys=3.90%, ctx=857, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147779: Thu Mar 26 09:24:04 2026 read: IOPS=93, BW=374KiB/s (383kB/s)(1564KiB/4187msec) clat (usec): min=1018, max=50941, avg=3752.01, stdev=4166.82 lat (usec): min=1018, max=50942, avg=3753.29, stdev=4166.51 clat percentiles (usec): | 1.00th=[ 1037], 5.00th=[ 1254], 10.00th=[ 1336], 20.00th=[ 1467], | 30.00th=[ 1745], 40.00th=[ 1958], 50.00th=[ 2212], 60.00th=[ 2671], | 70.00th=[ 3621], 80.00th=[ 4948], 90.00th=[ 7832], 95.00th=[12518], | 99.00th=[16450], 99.50th=[21627], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 199, max= 728, per=26.99%, avg=379.75, stdev=186.51, samples=8 iops : min= 49, max= 182, avg=94.75, stdev=46.82, samples=8 write: IOPS=90, BW=360KiB/s (369kB/s)(1508KiB/4187msec); 0 zone resets clat (usec): min=2222, max=39238, avg=7147.78, stdev=4785.78 lat (usec): min=2223, max=39242, avg=7148.94, stdev=4785.93 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2868], 10.00th=[ 3130], 20.00th=[ 3654], | 30.00th=[ 4146], 40.00th=[ 4621], 50.00th=[ 5407], 60.00th=[ 6587], | 70.00th=[ 8291], 80.00th=[10028], 90.00th=[13042], 95.00th=[16909], | 99.00th=[24511], 99.50th=[29754], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 240, max= 584, per=24.19%, avg=358.00, stdev=143.62, samples=8 iops : min= 60, max= 146, avg=89.38, stdev=36.01, samples=8 lat (msec) : 2=21.61%, 4=28.52%, 10=36.07%, 20=12.37%, 50=1.30% lat (msec) : 100=0.13% cpu : usr=0.38%, sys=3.63%, ctx=828, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147780: Thu Mar 26 09:24:04 2026 read: IOPS=89, BW=358KiB/s (367kB/s)(1508KiB/4210msec) clat (usec): min=1051, max=46200, avg=4039.82, stdev=4394.62 lat (usec): min=1051, max=46201, avg=4040.50, stdev=4394.59 clat percentiles (usec): | 1.00th=[ 1106], 5.00th=[ 1221], 10.00th=[ 1336], 20.00th=[ 1532], | 30.00th=[ 1696], 40.00th=[ 1975], 50.00th=[ 2343], 60.00th=[ 2966], | 70.00th=[ 3785], 80.00th=[ 5473], 90.00th=[ 9503], 95.00th=[13304], | 99.00th=[17433], 99.50th=[27919], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 192, max= 648, per=25.22%, avg=354.88, stdev=146.91, samples=8 iops : min= 48, max= 162, avg=88.63, stdev=36.80, samples=8 write: IOPS=92, BW=371KiB/s (380kB/s)(1564KiB/4210msec); 0 zone resets clat (usec): min=1772, max=42572, avg=6810.93, stdev=4277.58 lat (usec): min=1773, max=42573, avg=6811.74, stdev=4277.62 clat percentiles (usec): | 1.00th=[ 2057], 5.00th=[ 2573], 10.00th=[ 3163], 20.00th=[ 3654], | 30.00th=[ 4113], 40.00th=[ 4686], 50.00th=[ 5407], 60.00th=[ 6325], | 70.00th=[ 7898], 80.00th=[ 9896], 90.00th=[12649], 95.00th=[14746], | 99.00th=[20317], 99.50th=[23725], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 245, max= 616, per=25.03%, avg=370.50, stdev=161.16, samples=8 iops : min= 61, max= 154, avg=92.50, stdev=40.39, samples=8 lat (msec) : 2=20.44%, 4=27.73%, 10=37.76%, 20=13.15%, 50=0.91% cpu : usr=0.38%, sys=3.73%, ctx=837, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1407KiB/s (1441kB/s), 341KiB/s-374KiB/s (350kB/s-383kB/s), io=5988KiB (6132kB), run=4187-4255msec WRITE: bw=1481KiB/s (1516kB/s), 360KiB/s-389KiB/s (369kB/s-398kB/s), io=6300KiB (6451kB), run=4187-4255msec