== sanity test 398b: DIO and buffer IO race ============== 17:07:34 (1771970854) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.111701 s, 113 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147646: Tue Feb 24 17:07:42 2026 read: IOPS=173, BW=695KiB/s (712kB/s)(1480KiB/2129msec) clat (usec): min=6, max=41074, avg=3153.29, stdev=3811.68 lat (usec): min=6, max=41075, avg=3154.10, stdev=3811.76 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 13], 10.00th=[ 16], 20.00th=[ 21], | 30.00th=[ 27], 40.00th=[ 2024], 50.00th=[ 3064], 60.00th=[ 3687], | 70.00th=[ 4228], 80.00th=[ 5145], 90.00th=[ 6587], 95.00th=[ 8455], | 99.00th=[10945], 99.50th=[36439], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 560, max= 824, per=24.99%, avg=686.00, stdev=117.28, samples=4 iops : min= 140, max= 206, avg=171.50, stdev=29.32, samples=4 write: IOPS=186, BW=748KiB/s (766kB/s)(1592KiB/2129msec); 0 zone resets clat (usec): min=133, max=18877, avg=2397.67, stdev=2815.72 lat (usec): min=134, max=18877, avg=2398.66, stdev=2815.50 clat percentiles (usec): | 1.00th=[ 161], 5.00th=[ 219], 10.00th=[ 255], 20.00th=[ 310], | 30.00th=[ 396], 40.00th=[ 742], 50.00th=[ 1532], 60.00th=[ 1958], | 70.00th=[ 2769], 80.00th=[ 4146], 90.00th=[ 6128], 95.00th=[ 8291], | 99.00th=[14091], 99.50th=[16909], 99.90th=[19006], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 657, max= 888, per=25.91%, avg=748.25, stdev=101.00, samples=4 iops : min= 164, max= 222, avg=187.00, stdev=25.32, samples=4 lat (usec) : 10=1.04%, 20=8.59%, 50=6.38%, 100=0.39%, 250=5.21% lat (usec) : 500=13.80%, 750=2.60%, 1000=0.65% lat (msec) : 2=11.72%, 4=22.40%, 10=25.00%, 20=1.95%, 50=0.26% cpu : usr=0.38%, sys=15.79%, ctx=996, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147647: Tue Feb 24 17:07:42 2026 read: IOPS=164, BW=658KiB/s (674kB/s)(1436KiB/2181msec) clat (usec): min=8, max=19403, avg=3419.88, stdev=3225.79 lat (usec): min=9, max=19403, avg=3420.34, stdev=3225.79 clat percentiles (usec): | 1.00th=[ 10], 5.00th=[ 15], 10.00th=[ 18], 20.00th=[ 23], | 30.00th=[ 1303], 40.00th=[ 2737], 50.00th=[ 3359], 60.00th=[ 3818], | 70.00th=[ 4490], 80.00th=[ 5276], 90.00th=[ 7373], 95.00th=[ 9241], | 99.00th=[14615], 99.50th=[17957], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 558, max= 792, per=23.88%, avg=655.50, stdev=106.94, samples=4 iops : min= 139, max= 198, avg=163.75, stdev=26.89, samples=4 write: IOPS=187, BW=750KiB/s (768kB/s)(1636KiB/2181msec); 0 zone resets clat (usec): min=130, max=43715, avg=2312.77, stdev=3119.21 lat (usec): min=130, max=43716, avg=2313.35, stdev=3119.18 clat percentiles (usec): | 1.00th=[ 159], 5.00th=[ 198], 10.00th=[ 225], 20.00th=[ 297], | 30.00th=[ 379], 40.00th=[ 1090], 50.00th=[ 1565], 60.00th=[ 1991], | 70.00th=[ 2540], 80.00th=[ 3884], 90.00th=[ 5800], 95.00th=[ 6783], | 99.00th=[10290], 99.50th=[12911], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 578, max= 848, per=25.50%, avg=736.50, stdev=120.09, samples=4 iops : min= 144, max= 212, avg=184.00, stdev=30.24, samples=4 lat (usec) : 10=0.52%, 20=7.03%, 50=5.47%, 100=0.52%, 250=8.59% lat (usec) : 500=10.55%, 750=1.30%, 1000=1.04% lat (msec) : 2=13.02%, 4=24.35%, 10=25.26%, 20=2.21%, 50=0.13% cpu : usr=0.18%, sys=14.17%, ctx=1036, majf=0, minf=17 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147648: Tue Feb 24 17:07:42 2026 read: IOPS=196, BW=786KiB/s (804kB/s)(1564KiB/1991msec) clat (usec): min=6, max=44554, avg=2978.00, stdev=3623.19 lat (usec): min=6, max=44554, avg=2978.56, stdev=3623.19 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 12], 10.00th=[ 15], 20.00th=[ 18], | 30.00th=[ 24], 40.00th=[ 1549], 50.00th=[ 2769], 60.00th=[ 3523], | 70.00th=[ 4359], 80.00th=[ 5014], 90.00th=[ 6259], 95.00th=[ 8356], | 99.00th=[14222], 99.50th=[16450], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 571, max= 856, per=26.85%, avg=737.00, stdev=148.20, samples=3 iops : min= 142, max= 214, avg=184.00, stdev=37.47, samples=3 write: IOPS=189, BW=757KiB/s (776kB/s)(1508KiB/1991msec); 0 zone resets clat (usec): min=145, max=41374, avg=2166.55, stdev=3118.54 lat (usec): min=145, max=41375, avg=2167.13, stdev=3118.57 clat percentiles (usec): | 1.00th=[ 174], 5.00th=[ 206], 10.00th=[ 239], 20.00th=[ 297], | 30.00th=[ 359], 40.00th=[ 537], 50.00th=[ 1467], 60.00th=[ 1795], | 70.00th=[ 2311], 80.00th=[ 3523], 90.00th=[ 5342], 95.00th=[ 7373], | 99.00th=[10552], 99.50th=[18482], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 644, max= 784, per=25.35%, avg=732.00, stdev=76.63, samples=3 iops : min= 161, max= 196, avg=183.00, stdev=19.16, samples=3 lat (usec) : 10=0.91%, 20=11.85%, 50=5.86%, 100=0.39%, 250=6.51% lat (usec) : 500=13.15%, 750=1.69%, 1000=1.04% lat (msec) : 2=12.24%, 4=20.70%, 10=23.70%, 20=1.69%, 50=0.26% cpu : usr=0.45%, sys=14.67%, ctx=940, majf=0, minf=17 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147649: Tue Feb 24 17:07:42 2026 read: IOPS=186, BW=747KiB/s (764kB/s)(1508KiB/2020msec) clat (usec): min=4, max=16548, avg=2963.84, stdev=2821.95 lat (usec): min=4, max=16548, avg=2964.26, stdev=2821.94 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 13], 10.00th=[ 15], 20.00th=[ 20], | 30.00th=[ 25], 40.00th=[ 2147], 50.00th=[ 3064], 60.00th=[ 3720], | 70.00th=[ 4293], 80.00th=[ 5211], 90.00th=[ 6521], 95.00th=[ 7504], | 99.00th=[11600], 99.50th=[14746], 99.90th=[16581], 99.95th=[16581], | 99.99th=[16581] bw ( KiB/s): min= 631, max= 744, per=25.15%, avg=690.33, stdev=56.71, samples=3 iops : min= 157, max= 186, avg=172.33, stdev=14.57, samples=3 write: IOPS=193, BW=774KiB/s (793kB/s)(1564KiB/2020msec); 0 zone resets clat (usec): min=117, max=46415, avg=2289.10, stdev=3454.83 lat (usec): min=117, max=46415, avg=2289.65, stdev=3454.82 clat percentiles (usec): | 1.00th=[ 159], 5.00th=[ 196], 10.00th=[ 227], 20.00th=[ 289], | 30.00th=[ 359], 40.00th=[ 502], 50.00th=[ 1352], 60.00th=[ 1778], | 70.00th=[ 2507], 80.00th=[ 3949], 90.00th=[ 5866], 95.00th=[ 7242], | 99.00th=[12387], 99.50th=[23462], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 671, max= 896, per=25.93%, avg=749.00, stdev=127.39, samples=3 iops : min= 167, max= 224, avg=187.00, stdev=32.08, samples=3 lat (usec) : 10=1.17%, 20=10.55%, 50=5.34%, 100=0.26%, 250=7.55% lat (usec) : 500=13.15%, 750=2.08%, 1000=0.91% lat (msec) : 2=11.46%, 4=20.05%, 10=26.04%, 20=1.17%, 50=0.26% cpu : usr=0.54%, sys=13.82%, ctx=939, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2746KiB/s (2811kB/s), 658KiB/s-786KiB/s (674kB/s-804kB/s), io=5988KiB (6132kB), run=1991-2181msec WRITE: bw=2889KiB/s (2958kB/s), 748KiB/s-774KiB/s (766kB/s-793kB/s), io=6300KiB (6451kB), run=1991-2181msec rand-rw: (groupid=0, jobs=1): err= 0: pid=147641: Tue Feb 24 17:07:43 2026 read: IOPS=109, BW=438KiB/s (448kB/s)(1480KiB/3380msec) clat (usec): min=1006, max=27377, avg=3095.90, stdev=2758.59 lat (usec): min=1006, max=27379, avg=3096.42, stdev=2758.64 clat percentiles (usec): | 1.00th=[ 1029], 5.00th=[ 1106], 10.00th=[ 1172], 20.00th=[ 1336], | 30.00th=[ 1483], 40.00th=[ 1680], 50.00th=[ 1926], 60.00th=[ 2376], | 70.00th=[ 3195], 80.00th=[ 4490], 90.00th=[ 7177], 95.00th=[ 8848], | 99.00th=[10945], 99.50th=[14484], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 280, max= 680, per=24.33%, avg=425.83, stdev=150.65, samples=6 iops : min= 70, max= 170, avg=106.33, stdev=37.77, samples=6 write: IOPS=117, BW=471KiB/s (482kB/s)(1592KiB/3380msec); 0 zone resets clat (usec): min=1707, max=46487, avg=5580.21, stdev=3580.54 lat (usec): min=1707, max=46488, avg=5580.90, stdev=3580.56 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2474], 10.00th=[ 2900], 20.00th=[ 3261], | 30.00th=[ 3621], 40.00th=[ 4047], 50.00th=[ 4490], 60.00th=[ 5211], | 70.00th=[ 6259], 80.00th=[ 7439], 90.00th=[ 9503], 95.00th=[11600], | 99.00th=[17433], 99.50th=[20317], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 304, max= 696, per=25.14%, avg=463.00, stdev=182.74, samples=6 iops : min= 76, max= 174, avg=115.67, stdev=45.74, samples=6 lat (msec) : 2=26.04%, 4=31.38%, 10=37.11%, 20=5.08%, 50=0.39% cpu : usr=0.24%, sys=4.44%, ctx=830, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147642: Tue Feb 24 17:07:43 2026 read: IOPS=104, BW=420KiB/s (430kB/s)(1436KiB/3420msec) clat (usec): min=965, max=21747, avg=3064.77, stdev=2771.73 lat (usec): min=965, max=21747, avg=3065.28, stdev=2771.77 clat percentiles (usec): | 1.00th=[ 1004], 5.00th=[ 1074], 10.00th=[ 1172], 20.00th=[ 1319], | 30.00th=[ 1450], 40.00th=[ 1663], 50.00th=[ 1876], 60.00th=[ 2376], | 70.00th=[ 3228], 80.00th=[ 4178], 90.00th=[ 6915], 95.00th=[ 8717], | 99.00th=[13173], 99.50th=[17695], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 248, max= 640, per=23.11%, avg=404.50, stdev=181.79, samples=6 iops : min= 62, max= 160, avg=101.00, stdev=45.51, samples=6 write: IOPS=119, BW=478KiB/s (490kB/s)(1636KiB/3420msec); 0 zone resets clat (usec): min=1610, max=43130, avg=5627.09, stdev=3669.63 lat (usec): min=1610, max=43130, avg=5627.74, stdev=3669.66 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2442], 10.00th=[ 2802], 20.00th=[ 3294], | 30.00th=[ 3556], 40.00th=[ 3949], 50.00th=[ 4555], 60.00th=[ 5342], | 70.00th=[ 6128], 80.00th=[ 7177], 90.00th=[ 9765], 95.00th=[12256], | 99.00th=[17433], 99.50th=[19792], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 352, max= 712, per=24.86%, avg=458.00, stdev=157.50, samples=6 iops : min= 88, max= 178, avg=114.50, stdev=39.37, samples=6 lat (usec) : 1000=0.39% lat (msec) : 2=25.91%, 4=32.42%, 10=34.38%, 20=6.51%, 50=0.39% cpu : usr=0.44%, sys=4.18%, ctx=836, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147643: Tue Feb 24 17:07:43 2026 read: IOPS=115, BW=463KiB/s (474kB/s)(1564KiB/3377msec) clat (usec): min=918, max=15227, avg=3031.91, stdev=2681.21 lat (usec): min=918, max=15228, avg=3032.49, stdev=2681.23 clat percentiles (usec): | 1.00th=[ 1004], 5.00th=[ 1090], 10.00th=[ 1156], 20.00th=[ 1319], | 30.00th=[ 1450], 40.00th=[ 1614], 50.00th=[ 1860], 60.00th=[ 2180], | 70.00th=[ 2868], 80.00th=[ 4178], 90.00th=[ 7177], 95.00th=[ 9241], | 99.00th=[13173], 99.50th=[14484], 99.90th=[15270], 99.95th=[15270], | 99.99th=[15270] bw ( KiB/s): min= 276, max= 712, per=26.02%, avg=455.33, stdev=199.01, samples=6 iops : min= 69, max= 178, avg=113.83, stdev=49.75, samples=6 write: IOPS=111, BW=447KiB/s (457kB/s)(1508KiB/3377msec); 0 zone resets clat (usec): min=1891, max=37287, avg=5764.04, stdev=3723.02 lat (usec): min=1892, max=37287, avg=5765.14, stdev=3722.99 clat percentiles (usec): | 1.00th=[ 2147], 5.00th=[ 2606], 10.00th=[ 2933], 20.00th=[ 3195], | 30.00th=[ 3490], 40.00th=[ 3916], 50.00th=[ 4621], 60.00th=[ 5407], | 70.00th=[ 6521], 80.00th=[ 7898], 90.00th=[10028], 95.00th=[12649], | 99.00th=[21890], 99.50th=[28181], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 256, max= 688, per=23.74%, avg=437.33, stdev=178.14, samples=6 iops : min= 64, max= 172, avg=109.33, stdev=44.54, samples=6 lat (usec) : 1000=0.39% lat (msec) : 2=26.95%, 4=33.07%, 10=32.94%, 20=6.12%, 50=0.52% cpu : usr=0.27%, sys=4.38%, ctx=840, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147644: Tue Feb 24 17:07:43 2026 read: IOPS=110, BW=443KiB/s (453kB/s)(1508KiB/3406msec) clat (usec): min=909, max=18527, avg=3064.68, stdev=2717.95 lat (usec): min=910, max=18528, avg=3065.29, stdev=2717.90 clat percentiles (usec): | 1.00th=[ 996], 5.00th=[ 1123], 10.00th=[ 1205], 20.00th=[ 1319], | 30.00th=[ 1450], 40.00th=[ 1598], 50.00th=[ 1909], 60.00th=[ 2442], | 70.00th=[ 3064], 80.00th=[ 4293], 90.00th=[ 6849], 95.00th=[ 8356], | 99.00th=[14746], 99.50th=[15008], 99.90th=[18482], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 303, max= 704, per=24.52%, avg=429.17, stdev=166.71, samples=6 iops : min= 75, max= 176, avg=107.17, stdev=41.79, samples=6 write: IOPS=114, BW=459KiB/s (470kB/s)(1564KiB/3406msec); 0 zone resets clat (usec): min=1815, max=45965, avg=5712.84, stdev=3904.73 lat (usec): min=1815, max=45966, avg=5713.63, stdev=3904.74 clat percentiles (usec): | 1.00th=[ 2057], 5.00th=[ 2442], 10.00th=[ 2868], 20.00th=[ 3163], | 30.00th=[ 3556], 40.00th=[ 4015], 50.00th=[ 4686], 60.00th=[ 5407], | 70.00th=[ 6325], 80.00th=[ 7373], 90.00th=[ 9372], 95.00th=[12125], | 99.00th=[18482], 99.50th=[26870], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 304, max= 704, per=24.23%, avg=446.33, stdev=194.21, samples=6 iops : min= 76, max= 176, avg=111.50, stdev=48.60, samples=6 lat (usec) : 1000=0.52% lat (msec) : 2=25.13%, 4=32.68%, 10=35.94%, 20=5.34%, 50=0.39% cpu : usr=0.29%, sys=4.58%, ctx=849, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1751KiB/s (1793kB/s), 420KiB/s-463KiB/s (430kB/s-474kB/s), io=5988KiB (6132kB), run=3377-3420msec WRITE: bw=1842KiB/s (1886kB/s), 447KiB/s-478KiB/s (457kB/s-490kB/s), io=6300KiB (6451kB), run=3377-3420msec