== sanity test 398b: DIO and buffer IO race ============== 14:11:20 (1772305880) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.087779 s, 143 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=121944: Sat Feb 28 14:11:29 2026 read: IOPS=126, BW=505KiB/s (517kB/s)(1480KiB/2929msec) clat (usec): min=5, max=34537, avg=2251.25, stdev=3635.07 lat (usec): min=5, max=34538, avg=2251.88, stdev=3634.97 clat percentiles (usec): | 1.00th=[ 7], 5.00th=[ 11], 10.00th=[ 14], 20.00th=[ 18], | 30.00th=[ 24], 40.00th=[ 42], 50.00th=[ 1385], 60.00th=[ 2409], | 70.00th=[ 3130], 80.00th=[ 4146], 90.00th=[ 5211], 95.00th=[ 6456], | 99.00th=[17433], 99.50th=[31589], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 320, max= 586, per=24.70%, avg=502.80, stdev=117.57, samples=5 iops : min= 80, max= 146, avg=125.60, stdev=29.31, samples=5 write: IOPS=135, BW=544KiB/s (557kB/s)(1592KiB/2929msec); 0 zone resets clat (usec): min=129, max=45111, avg=5215.23, stdev=6598.23 lat (usec): min=129, max=45112, avg=5215.84, stdev=6598.21 clat percentiles (usec): | 1.00th=[ 139], 5.00th=[ 215], 10.00th=[ 269], 20.00th=[ 388], | 30.00th=[ 676], 40.00th=[ 1713], 50.00th=[ 2409], 60.00th=[ 3458], | 70.00th=[ 6063], 80.00th=[ 9503], 90.00th=[15533], 95.00th=[18744], | 99.00th=[30016], 99.50th=[34341], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 416, max= 616, per=25.64%, avg=549.20, stdev=78.73, samples=5 iops : min= 104, max= 154, avg=137.20, stdev=19.63, samples=5 lat (usec) : 10=2.21%, 20=9.11%, 50=10.29%, 100=1.04%, 250=4.95% lat (usec) : 500=10.16%, 750=1.56%, 1000=1.56% lat (msec) : 2=9.77%, 4=19.53%, 10=19.01%, 20=8.46%, 50=2.34% cpu : usr=0.34%, sys=10.48%, ctx=1167, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=121945: Sat Feb 28 14:11:29 2026 read: IOPS=123, BW=495KiB/s (507kB/s)(1436KiB/2903msec) clat (usec): min=6, max=43653, avg=2482.21, stdev=4408.40 lat (usec): min=7, max=43654, avg=2482.72, stdev=4408.49 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 13], 10.00th=[ 15], 20.00th=[ 20], | 30.00th=[ 26], 40.00th=[ 44], 50.00th=[ 1549], 60.00th=[ 2311], | 70.00th=[ 3195], 80.00th=[ 3982], 90.00th=[ 5538], 95.00th=[ 7111], | 99.00th=[23725], 99.50th=[32375], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 384, max= 672, per=24.49%, avg=498.60, stdev=112.88, samples=5 iops : min= 96, max= 168, avg=124.60, stdev=28.26, samples=5 write: IOPS=140, BW=564KiB/s (577kB/s)(1636KiB/2903msec); 0 zone resets clat (usec): min=149, max=27328, avg=4883.60, stdev=5692.64 lat (usec): min=149, max=27329, avg=4884.20, stdev=5692.65 clat percentiles (usec): | 1.00th=[ 186], 5.00th=[ 223], 10.00th=[ 277], 20.00th=[ 351], | 30.00th=[ 701], 40.00th=[ 1893], 50.00th=[ 2540], 60.00th=[ 3785], | 70.00th=[ 5735], 80.00th=[ 8848], 90.00th=[13435], 95.00th=[16909], | 99.00th=[26084], 99.50th=[26346], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 503, max= 624, per=26.21%, avg=561.40, stdev=50.06, samples=5 iops : min= 125, max= 156, avg=140.20, stdev=12.74, samples=5 lat (usec) : 10=1.17%, 20=8.72%, 50=10.16%, 100=0.78%, 250=5.21% lat (usec) : 500=10.29%, 750=2.34%, 1000=0.39% lat (msec) : 2=9.24%, 4=22.40%, 10=18.62%, 20=8.33%, 50=2.34% cpu : usr=0.17%, sys=10.99%, ctx=1123, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=121947: Sat Feb 28 14:11:29 2026 read: IOPS=151, BW=608KiB/s (622kB/s)(1564KiB/2573msec) clat (usec): min=4, max=27454, avg=2179.86, stdev=3267.70 lat (usec): min=5, max=27455, avg=2180.34, stdev=3267.71 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 13], 10.00th=[ 16], 20.00th=[ 19], | 30.00th=[ 28], 40.00th=[ 41], 50.00th=[ 1614], 60.00th=[ 2278], | 70.00th=[ 3097], 80.00th=[ 3916], 90.00th=[ 5014], 95.00th=[ 6521], | 99.00th=[22938], 99.50th=[24773], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 541, max= 712, per=30.30%, avg=617.00, stdev=70.86, samples=5 iops : min= 135, max= 178, avg=154.20, stdev=17.78, samples=5 write: IOPS=146, BW=586KiB/s (600kB/s)(1508KiB/2573msec); 0 zone resets clat (usec): min=158, max=29701, avg=4515.50, stdev=5531.92 lat (usec): min=159, max=29702, avg=4516.29, stdev=5531.80 clat percentiles (usec): | 1.00th=[ 172], 5.00th=[ 239], 10.00th=[ 297], 20.00th=[ 404], | 30.00th=[ 922], 40.00th=[ 1663], 50.00th=[ 2376], 60.00th=[ 3458], | 70.00th=[ 5145], 80.00th=[ 7373], 90.00th=[12387], 95.00th=[16319], | 99.00th=[24511], 99.50th=[28443], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 504, max= 639, per=27.33%, avg=585.40, stdev=49.50, samples=5 iops : min= 126, max= 159, avg=146.20, stdev=12.17, samples=5 lat (usec) : 10=1.04%, 20=9.77%, 50=11.59%, 100=1.04%, 250=3.65% lat (usec) : 500=9.38%, 750=1.82%, 1000=1.04% lat (msec) : 2=11.59%, 4=22.14%, 10=19.53%, 20=5.34%, 50=2.08% cpu : usr=0.16%, sys=11.16%, ctx=1118, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=121949: Sat Feb 28 14:11:29 2026 read: IOPS=128, BW=513KiB/s (525kB/s)(1508KiB/2941msec) clat (usec): min=4, max=24645, avg=2098.95, stdev=3168.39 lat (usec): min=5, max=24645, avg=2099.41, stdev=3168.39 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 15], 20.00th=[ 18], | 30.00th=[ 23], 40.00th=[ 33], 50.00th=[ 273], 60.00th=[ 2343], | 70.00th=[ 3064], 80.00th=[ 4080], 90.00th=[ 5080], 95.00th=[ 6325], | 99.00th=[21103], 99.50th=[24249], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 440, max= 632, per=25.18%, avg=512.60, stdev=86.31, samples=5 iops : min= 110, max= 158, avg=128.00, stdev=21.68, samples=5 write: IOPS=132, BW=532KiB/s (545kB/s)(1564KiB/2941msec); 0 zone resets clat (usec): min=128, max=33191, avg=5452.13, stdev=6573.21 lat (usec): min=128, max=33192, avg=5452.72, stdev=6573.23 clat percentiles (usec): | 1.00th=[ 143], 5.00th=[ 204], 10.00th=[ 262], 20.00th=[ 371], | 30.00th=[ 775], 40.00th=[ 1827], 50.00th=[ 2835], 60.00th=[ 4424], | 70.00th=[ 6325], 80.00th=[ 9372], 90.00th=[15270], 95.00th=[19268], | 99.00th=[29754], 99.50th=[32113], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 376, max= 688, per=24.81%, avg=531.40, stdev=112.61, samples=5 iops : min= 94, max= 172, avg=132.80, stdev=28.16, samples=5 lat (usec) : 10=1.56%, 20=11.85%, 50=9.77%, 100=0.78%, 250=4.82% lat (usec) : 500=10.03%, 750=1.43%, 1000=0.91% lat (msec) : 2=8.33%, 4=18.62%, 10=22.14%, 20=7.16%, 50=2.60% cpu : usr=0.17%, sys=9.83%, ctx=1099, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2036KiB/s (2085kB/s), 495KiB/s-608KiB/s (507kB/s-622kB/s), io=5988KiB (6132kB), run=2573-2941msec WRITE: bw=2142KiB/s (2194kB/s), 532KiB/s-586KiB/s (545kB/s-600kB/s), io=6300KiB (6451kB), run=2573-2941msec rand-rw: (groupid=0, jobs=1): err= 0: pid=121943: Sat Feb 28 14:11:37 2026 read: IOPS=34, BW=138KiB/s (141kB/s)(1480KiB/10737msec) clat (usec): min=1026, max=64736, avg=3222.73, stdev=5723.18 lat (usec): min=1026, max=64737, avg=3223.67, stdev=5723.19 clat percentiles (usec): | 1.00th=[ 1123], 5.00th=[ 1270], 10.00th=[ 1319], 20.00th=[ 1450], | 30.00th=[ 1598], 40.00th=[ 1745], 50.00th=[ 1926], 60.00th=[ 2147], | 70.00th=[ 2474], 80.00th=[ 3294], 90.00th=[ 4817], 95.00th=[ 7177], | 99.00th=[31065], 99.50th=[62653], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 88, max= 288, per=24.61%, avg=135.62, stdev=45.22, samples=21 iops : min= 22, max= 72, avg=33.90, stdev=11.30, samples=21 write: IOPS=37, BW=148KiB/s (152kB/s)(1592KiB/10737msec); 0 zone resets clat (usec): min=11009, max=61484, avg=23924.62, stdev=6959.12 lat (usec): min=11010, max=61485, avg=23925.67, stdev=6959.13 clat percentiles (usec): | 1.00th=[12256], 5.00th=[15533], 10.00th=[16450], 20.00th=[17957], | 30.00th=[19268], 40.00th=[20841], 50.00th=[22938], 60.00th=[25035], | 70.00th=[26608], 80.00th=[29230], 90.00th=[32900], 95.00th=[34866], | 99.00th=[45876], 99.50th=[52691], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 97, max= 176, per=25.43%, avg=147.48, stdev=23.70, samples=21 iops : min= 24, max= 44, avg=36.86, stdev= 5.95, samples=21 lat (msec) : 2=26.04%, 4=15.49%, 10=4.95%, 20=19.14%, 50=33.85% lat (msec) : 100=0.52% cpu : usr=0.13%, sys=1.56%, ctx=826, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=121946: Sat Feb 28 14:11:37 2026 read: IOPS=33, BW=132KiB/s (135kB/s)(1436KiB/10856msec) clat (usec): min=1032, max=51212, avg=3370.73, stdev=4641.12 lat (usec): min=1033, max=51213, avg=3371.84, stdev=4641.61 clat percentiles (usec): | 1.00th=[ 1123], 5.00th=[ 1237], 10.00th=[ 1319], 20.00th=[ 1450], | 30.00th=[ 1598], 40.00th=[ 1795], 50.00th=[ 2008], 60.00th=[ 2245], | 70.00th=[ 2802], 80.00th=[ 3687], 90.00th=[ 5538], 95.00th=[11338], | 99.00th=[23462], 99.50th=[31851], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 72, max= 208, per=23.81%, avg=131.19, stdev=39.90, samples=21 iops : min= 18, max= 52, avg=32.76, stdev=10.03, samples=21 write: IOPS=37, BW=151KiB/s (154kB/s)(1636KiB/10856msec); 0 zone resets clat (usec): min=10635, max=59539, avg=23530.82, stdev=6745.94 lat (usec): min=10636, max=59541, avg=23531.98, stdev=6745.81 clat percentiles (usec): | 1.00th=[11469], 5.00th=[13960], 10.00th=[16057], 20.00th=[17957], | 30.00th=[19268], 40.00th=[20579], 50.00th=[22938], 60.00th=[24773], | 70.00th=[26608], 80.00th=[28443], 90.00th=[32375], 95.00th=[35390], | 99.00th=[42730], 99.50th=[44303], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 96, max= 176, per=25.48%, avg=147.76, stdev=21.16, samples=21 iops : min= 24, max= 44, avg=36.90, stdev= 5.33, samples=21 lat (msec) : 2=23.05%, 4=15.36%, 10=5.60%, 20=21.09%, 50=34.64% lat (msec) : 100=0.26% cpu : usr=0.15%, sys=1.56%, ctx=823, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=121948: Sat Feb 28 14:11:37 2026 read: IOPS=37, BW=152KiB/s (155kB/s)(1564KiB/10311msec) clat (usec): min=1056, max=29668, avg=3262.40, stdev=3795.66 lat (usec): min=1056, max=29669, avg=3263.23, stdev=3795.68 clat percentiles (usec): | 1.00th=[ 1123], 5.00th=[ 1237], 10.00th=[ 1352], 20.00th=[ 1483], | 30.00th=[ 1663], 40.00th=[ 1844], 50.00th=[ 2024], 60.00th=[ 2245], | 70.00th=[ 2671], 80.00th=[ 3425], 90.00th=[ 5866], 95.00th=[10552], | 99.00th=[22152], 99.50th=[25035], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 48, max= 296, per=28.10%, avg=154.85, stdev=70.29, samples=20 iops : min= 12, max= 74, avg=38.70, stdev=17.58, samples=20 write: IOPS=36, BW=146KiB/s (150kB/s)(1508KiB/10311msec); 0 zone resets clat (usec): min=11558, max=77365, avg=23912.03, stdev=7231.59 lat (usec): min=11558, max=77366, avg=23912.99, stdev=7231.63 clat percentiles (usec): | 1.00th=[12125], 5.00th=[15664], 10.00th=[16319], 20.00th=[17957], | 30.00th=[19268], 40.00th=[20579], 50.00th=[22938], 60.00th=[24773], | 70.00th=[26870], 80.00th=[29492], 90.00th=[32113], 95.00th=[37487], | 99.00th=[43254], 99.50th=[52167], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 112, max= 176, per=25.24%, avg=146.40, stdev=19.61, samples=20 iops : min= 28, max= 44, avg=36.60, stdev= 4.90, samples=20 lat (msec) : 2=24.87%, 4=17.58%, 10=5.60%, 20=19.92%, 50=31.77% lat (msec) : 100=0.26% cpu : usr=0.12%, sys=1.65%, ctx=819, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=121950: Sat Feb 28 14:11:37 2026 read: IOPS=35, BW=143KiB/s (146kB/s)(1508KiB/10570msec) clat (usec): min=1074, max=39048, avg=2896.15, stdev=3697.72 lat (usec): min=1075, max=39049, avg=2896.95, stdev=3697.73 clat percentiles (usec): | 1.00th=[ 1172], 5.00th=[ 1237], 10.00th=[ 1303], 20.00th=[ 1450], | 30.00th=[ 1598], 40.00th=[ 1713], 50.00th=[ 1926], 60.00th=[ 2147], | 70.00th=[ 2540], 80.00th=[ 2999], 90.00th=[ 4293], 95.00th=[ 7898], | 99.00th=[22938], 99.50th=[29754], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 72, max= 192, per=25.73%, avg=141.76, stdev=33.52, samples=21 iops : min= 18, max= 48, avg=35.43, stdev= 8.39, samples=21 write: IOPS=36, BW=148KiB/s (152kB/s)(1564KiB/10570msec); 0 zone resets clat (usec): min=11444, max=66407, avg=24204.56, stdev=7275.55 lat (usec): min=11445, max=66408, avg=24205.61, stdev=7275.47 clat percentiles (usec): | 1.00th=[12518], 5.00th=[15008], 10.00th=[16909], 20.00th=[18482], | 30.00th=[19268], 40.00th=[21103], 50.00th=[23200], 60.00th=[25035], | 70.00th=[26870], 80.00th=[29230], 90.00th=[32900], 95.00th=[36963], | 99.00th=[51643], 99.50th=[61604], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 112, max= 184, per=25.54%, avg=148.14, stdev=20.68, samples=21 iops : min= 28, max= 46, avg=37.00, stdev= 5.20, samples=21 lat (msec) : 2=26.95%, 4=16.54%, 10=3.65%, 20=18.49%, 50=33.85% lat (msec) : 100=0.52% cpu : usr=0.07%, sys=1.67%, ctx=824, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=552KiB/s (565kB/s), 132KiB/s-152KiB/s (135kB/s-155kB/s), io=5988KiB (6132kB), run=10311-10856msec WRITE: bw=580KiB/s (594kB/s), 146KiB/s-151KiB/s (150kB/s-154kB/s), io=6300KiB (6451kB), run=10311-10856msec