== sanity test 398b: DIO and buffer IO race ============== 21:55:39 (1767840939) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.0883243 s, 142 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=121260: Wed Jan 7 21:55:48 2026 read: IOPS=134, BW=537KiB/s (550kB/s)(1480KiB/2757msec) clat (usec): min=5, max=47874, avg=2502.14, stdev=4417.89 lat (usec): min=5, max=47874, avg=2502.62, stdev=4417.88 clat percentiles (usec): | 1.00th=[ 10], 5.00th=[ 14], 10.00th=[ 16], 20.00th=[ 23], | 30.00th=[ 31], 40.00th=[ 45], 50.00th=[ 1549], 60.00th=[ 2638], | 70.00th=[ 3326], 80.00th=[ 4146], 90.00th=[ 5342], 95.00th=[ 6652], | 99.00th=[24511], 99.50th=[31851], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 408, max= 608, per=27.90%, avg=533.80, stdev=74.67, samples=5 iops : min= 102, max= 152, avg=133.40, stdev=18.65, samples=5 write: IOPS=144, BW=577KiB/s (591kB/s)(1592KiB/2757msec); 0 zone resets clat (usec): min=149, max=30297, avg=4585.37, stdev=5229.92 lat (usec): min=150, max=30298, avg=4585.97, stdev=5229.92 clat percentiles (usec): | 1.00th=[ 194], 5.00th=[ 249], 10.00th=[ 322], 20.00th=[ 404], | 30.00th=[ 1090], 40.00th=[ 1795], 50.00th=[ 2671], 60.00th=[ 3916], | 70.00th=[ 5473], 80.00th=[ 7570], 90.00th=[12911], 95.00th=[16319], | 99.00th=[21627], 99.50th=[26870], 99.90th=[30278], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 544, max= 608, per=28.42%, avg=571.80, stdev=30.53, samples=5 iops : min= 136, max= 152, avg=142.80, stdev= 7.69, samples=5 lat (usec) : 10=0.52%, 20=7.03%, 50=14.06%, 100=0.78%, 250=3.26% lat (usec) : 500=10.29%, 750=1.95%, 1000=0.65% lat (msec) : 2=9.24%, 4=21.22%, 10=22.40%, 20=6.77%, 50=1.82% cpu : usr=0.29%, sys=11.25%, ctx=1061, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=121261: Wed Jan 7 21:55:48 2026 read: IOPS=114, BW=459KiB/s (470kB/s)(1436KiB/3130msec) clat (usec): min=5, max=61332, avg=2743.13, stdev=4332.98 lat (usec): min=5, max=61332, avg=2743.59, stdev=4332.98 clat percentiles (usec): | 1.00th=[ 6], 5.00th=[ 13], 10.00th=[ 18], 20.00th=[ 21], | 30.00th=[ 32], 40.00th=[ 94], 50.00th=[ 2311], 60.00th=[ 3228], | 70.00th=[ 3851], 80.00th=[ 4424], 90.00th=[ 5800], 95.00th=[ 7046], | 99.00th=[17433], 99.50th=[24249], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 209, max= 592, per=24.19%, avg=462.83, stdev=154.83, samples=6 iops : min= 52, max= 148, avg=115.67, stdev=38.79, samples=6 write: IOPS=130, BW=523KiB/s (535kB/s)(1636KiB/3130msec); 0 zone resets clat (usec): min=160, max=48951, avg=5226.98, stdev=6353.68 lat (usec): min=160, max=48952, avg=5227.65, stdev=6353.64 clat percentiles (usec): | 1.00th=[ 192], 5.00th=[ 245], 10.00th=[ 277], 20.00th=[ 388], | 30.00th=[ 1139], 40.00th=[ 1811], 50.00th=[ 2868], 60.00th=[ 3949], | 70.00th=[ 6128], 80.00th=[ 9372], 90.00th=[14353], 95.00th=[16909], | 99.00th=[27132], 99.50th=[29754], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 249, max= 792, per=26.25%, avg=528.17, stdev=184.01, samples=6 iops : min= 62, max= 198, avg=132.00, stdev=46.08, samples=6 lat (usec) : 10=1.30%, 20=7.03%, 50=9.24%, 100=1.17%, 250=4.17% lat (usec) : 500=10.16%, 750=2.21%, 1000=0.39% lat (msec) : 2=8.98%, 4=20.83%, 10=23.96%, 20=8.59%, 50=1.82% lat (msec) : 100=0.13% cpu : usr=0.16%, sys=11.12%, ctx=1226, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=121262: Wed Jan 7 21:55:48 2026 read: IOPS=141, BW=564KiB/s (578kB/s)(1564KiB/2773msec) clat (usec): min=6, max=27113, avg=2700.65, stdev=3682.54 lat (usec): min=6, max=27114, avg=2701.20, stdev=3682.58 clat percentiles (usec): | 1.00th=[ 11], 5.00th=[ 15], 10.00th=[ 18], 20.00th=[ 26], | 30.00th=[ 41], 40.00th=[ 102], 50.00th=[ 2180], 60.00th=[ 2868], | 70.00th=[ 3490], 80.00th=[ 4178], 90.00th=[ 5866], 95.00th=[ 8094], | 99.00th=[21103], 99.50th=[22938], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 464, max= 720, per=30.04%, avg=574.60, stdev=124.80, samples=5 iops : min= 116, max= 180, avg=143.60, stdev=31.25, samples=5 write: IOPS=135, BW=544KiB/s (557kB/s)(1508KiB/2773msec); 0 zone resets clat (usec): min=146, max=33161, avg=4536.29, stdev=5280.95 lat (usec): min=147, max=33162, avg=4536.96, stdev=5280.95 clat percentiles (usec): | 1.00th=[ 186], 5.00th=[ 231], 10.00th=[ 302], 20.00th=[ 420], | 30.00th=[ 758], 40.00th=[ 1909], 50.00th=[ 2802], 60.00th=[ 3785], | 70.00th=[ 5211], 80.00th=[ 7701], 90.00th=[11994], 95.00th=[14746], | 99.00th=[26608], 99.50th=[33162], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 440, max= 608, per=26.56%, avg=534.40, stdev=67.74, samples=5 iops : min= 110, max= 152, avg=133.60, stdev=16.94, samples=5 lat (usec) : 10=0.26%, 20=6.77%, 50=10.55%, 100=2.73%, 250=3.78% lat (usec) : 500=9.64%, 750=2.34%, 1000=0.78% lat (msec) : 2=8.20%, 4=24.74%, 10=20.83%, 20=7.81%, 50=1.56% cpu : usr=0.29%, sys=11.33%, ctx=1175, majf=0, minf=17 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=121264: Wed Jan 7 21:55:48 2026 read: IOPS=134, BW=537KiB/s (550kB/s)(1508KiB/2809msec) clat (usec): min=8, max=43518, avg=2337.81, stdev=3939.89 lat (usec): min=8, max=43519, avg=2338.37, stdev=3939.93 clat percentiles (usec): | 1.00th=[ 10], 5.00th=[ 14], 10.00th=[ 16], 20.00th=[ 20], | 30.00th=[ 29], 40.00th=[ 47], 50.00th=[ 1598], 60.00th=[ 2606], | 70.00th=[ 3458], 80.00th=[ 4359], 90.00th=[ 5342], 95.00th=[ 6063], | 99.00th=[17433], 99.50th=[43254], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 440, max= 624, per=26.93%, avg=515.20, stdev=70.56, samples=5 iops : min= 110, max= 156, avg=128.80, stdev=17.64, samples=5 write: IOPS=139, BW=557KiB/s (570kB/s)(1564KiB/2809msec); 0 zone resets clat (usec): min=166, max=30353, avg=4911.71, stdev=5675.13 lat (usec): min=167, max=30353, avg=4912.31, stdev=5675.13 clat percentiles (usec): | 1.00th=[ 178], 5.00th=[ 239], 10.00th=[ 302], 20.00th=[ 420], | 30.00th=[ 1221], 40.00th=[ 1909], 50.00th=[ 2704], 60.00th=[ 3851], | 70.00th=[ 5473], 80.00th=[ 8717], 90.00th=[13304], 95.00th=[17171], | 99.00th=[26608], 99.50th=[27657], 99.90th=[30278], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 520, max= 576, per=27.07%, avg=544.60, stdev=25.49, samples=5 iops : min= 130, max= 144, avg=136.00, stdev= 6.48, samples=5 lat (usec) : 10=0.78%, 20=9.11%, 50=10.81%, 100=1.43%, 250=3.52% lat (usec) : 500=8.85%, 750=1.69%, 1000=1.56% lat (msec) : 2=10.81%, 4=20.70%, 10=21.22%, 20=7.94%, 50=1.56% cpu : usr=0.36%, sys=11.72%, ctx=1146, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1913KiB/s (1959kB/s), 459KiB/s-564KiB/s (470kB/s-578kB/s), io=5988KiB (6132kB), run=2757-3130msec WRITE: bw=2013KiB/s (2061kB/s), 523KiB/s-577KiB/s (535kB/s-591kB/s), io=6300KiB (6451kB), run=2757-3130msec rand-rw: (groupid=0, jobs=1): err= 0: pid=121255: Wed Jan 7 21:55:55 2026 read: IOPS=37, BW=151KiB/s (155kB/s)(1480KiB/9802msec) clat (usec): min=927, max=40244, avg=3246.07, stdev=4409.92 lat (usec): min=928, max=40245, avg=3246.93, stdev=4409.90 clat percentiles (usec): | 1.00th=[ 1037], 5.00th=[ 1156], 10.00th=[ 1237], 20.00th=[ 1369], | 30.00th=[ 1500], 40.00th=[ 1680], 50.00th=[ 1860], 60.00th=[ 2073], | 70.00th=[ 2474], 80.00th=[ 3425], 90.00th=[ 5145], 95.00th=[12125], | 99.00th=[24511], 99.50th=[32900], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 93, max= 240, per=24.29%, avg=147.21, stdev=44.29, samples=19 iops : min= 23, max= 60, avg=36.79, stdev=11.09, samples=19 write: IOPS=40, BW=162KiB/s (166kB/s)(1592KiB/9802msec); 0 zone resets clat (usec): min=9163, max=69435, avg=21561.15, stdev=7261.96 lat (usec): min=9165, max=69435, avg=21562.10, stdev=7261.97 clat percentiles (usec): | 1.00th=[10814], 5.00th=[12649], 10.00th=[13960], 20.00th=[15664], | 30.00th=[17433], 40.00th=[18744], 50.00th=[20055], 60.00th=[21627], | 70.00th=[23725], 80.00th=[27132], 90.00th=[30278], 95.00th=[34341], | 99.00th=[46924], 99.50th=[52691], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 106, max= 224, per=25.27%, avg=160.95, stdev=33.85, samples=19 iops : min= 26, max= 56, avg=40.21, stdev= 8.51, samples=19 lat (usec) : 1000=0.13% lat (msec) : 2=27.47%, 4=13.15%, 10=4.82%, 20=27.60%, 50=26.43% lat (msec) : 100=0.39% cpu : usr=0.18%, sys=1.68%, ctx=825, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=121256: Wed Jan 7 21:55:55 2026 read: IOPS=36, BW=145KiB/s (149kB/s)(1436KiB/9878msec) clat (usec): min=950, max=23536, avg=2982.09, stdev=3425.38 lat (usec): min=950, max=23537, avg=2982.87, stdev=3425.40 clat percentiles (usec): | 1.00th=[ 971], 5.00th=[ 1139], 10.00th=[ 1188], 20.00th=[ 1385], | 30.00th=[ 1500], 40.00th=[ 1663], 50.00th=[ 1893], 60.00th=[ 2073], | 70.00th=[ 2573], 80.00th=[ 3326], 90.00th=[ 4686], 95.00th=[12125], | 99.00th=[18220], 99.50th=[21103], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 56, max= 224, per=23.83%, avg=144.42, stdev=48.84, samples=19 iops : min= 14, max= 56, avg=36.11, stdev=12.21, samples=19 write: IOPS=41, BW=166KiB/s (170kB/s)(1636KiB/9878msec); 0 zone resets clat (usec): min=8532, max=53226, avg=21487.58, stdev=7127.36 lat (usec): min=8533, max=53226, avg=21488.60, stdev=7127.40 clat percentiles (usec): | 1.00th=[ 9241], 5.00th=[12649], 10.00th=[13698], 20.00th=[16057], | 30.00th=[17433], 40.00th=[18482], 50.00th=[20055], 60.00th=[21627], | 70.00th=[23725], 80.00th=[26608], 90.00th=[30802], 95.00th=[33817], | 99.00th=[46924], 99.50th=[50070], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 112, max= 224, per=25.50%, avg=162.42, stdev=31.12, samples=19 iops : min= 28, max= 56, avg=40.58, stdev= 7.81, samples=19 lat (usec) : 1000=0.52% lat (msec) : 2=25.52%, 4=14.84%, 10=4.30%, 20=27.86%, 50=26.56% lat (msec) : 100=0.39% cpu : usr=0.19%, sys=1.70%, ctx=824, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=121257: Wed Jan 7 21:55:55 2026 read: IOPS=41, BW=166KiB/s (170kB/s)(1564KiB/9417msec) clat (usec): min=880, max=26499, avg=2903.37, stdev=3480.20 lat (usec): min=881, max=26500, avg=2904.47, stdev=3480.70 clat percentiles (usec): | 1.00th=[ 1020], 5.00th=[ 1139], 10.00th=[ 1221], 20.00th=[ 1336], | 30.00th=[ 1500], 40.00th=[ 1663], 50.00th=[ 1844], 60.00th=[ 2057], | 70.00th=[ 2409], 80.00th=[ 3195], 90.00th=[ 4883], 95.00th=[ 8717], | 99.00th=[20841], 99.50th=[22414], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 72, max= 304, per=27.69%, avg=167.78, stdev=69.86, samples=18 iops : min= 18, max= 76, avg=41.94, stdev=17.46, samples=18 write: IOPS=40, BW=160KiB/s (164kB/s)(1508KiB/9417msec); 0 zone resets clat (usec): min=9738, max=53310, avg=21917.11, stdev=7422.75 lat (usec): min=9738, max=53311, avg=21918.12, stdev=7422.76 clat percentiles (usec): | 1.00th=[11076], 5.00th=[12649], 10.00th=[14353], 20.00th=[16057], | 30.00th=[17433], 40.00th=[18744], 50.00th=[20579], 60.00th=[22414], | 70.00th=[24249], 80.00th=[26870], 90.00th=[31589], 95.00th=[35390], | 99.00th=[50594], 99.50th=[52167], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 96, max= 216, per=24.82%, avg=158.11, stdev=31.35, samples=18 iops : min= 24, max= 54, avg=39.50, stdev= 7.87, samples=18 lat (usec) : 1000=0.13% lat (msec) : 2=29.69%, 4=14.32%, 10=4.82%, 20=24.35%, 50=26.17% lat (msec) : 100=0.52% cpu : usr=0.22%, sys=1.75%, ctx=844, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=121258: Wed Jan 7 21:55:55 2026 read: IOPS=38, BW=155KiB/s (159kB/s)(1508KiB/9710msec) clat (usec): min=836, max=46063, avg=3335.20, stdev=4444.59 lat (usec): min=836, max=46064, avg=3335.98, stdev=4444.67 clat percentiles (usec): | 1.00th=[ 963], 5.00th=[ 1123], 10.00th=[ 1188], 20.00th=[ 1418], | 30.00th=[ 1549], 40.00th=[ 1647], 50.00th=[ 1860], 60.00th=[ 2147], | 70.00th=[ 2606], 80.00th=[ 3687], 90.00th=[ 6980], 95.00th=[11469], | 99.00th=[19006], 99.50th=[37487], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 72, max= 264, per=25.47%, avg=154.37, stdev=51.08, samples=19 iops : min= 18, max= 66, avg=38.58, stdev=12.79, samples=19 write: IOPS=40, BW=161KiB/s (165kB/s)(1564KiB/9710msec); 0 zone resets clat (usec): min=10282, max=55166, avg=21573.96, stdev=6418.15 lat (usec): min=10283, max=55167, avg=21574.93, stdev=6418.19 clat percentiles (usec): | 1.00th=[11863], 5.00th=[13435], 10.00th=[14877], 20.00th=[16319], | 30.00th=[17433], 40.00th=[18744], 50.00th=[20055], 60.00th=[21890], | 70.00th=[24249], 80.00th=[26870], 90.00th=[29754], 95.00th=[32900], | 99.00th=[38536], 99.50th=[51643], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 107, max= 216, per=25.21%, avg=160.58, stdev=27.05, samples=19 iops : min= 26, max= 54, avg=40.11, stdev= 6.85, samples=19 lat (usec) : 1000=0.52% lat (msec) : 2=27.08%, 4=12.11%, 10=6.38%, 20=28.13%, 50=25.39% lat (msec) : 100=0.39% cpu : usr=0.11%, sys=1.79%, ctx=850, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=606KiB/s (621kB/s), 145KiB/s-166KiB/s (149kB/s-170kB/s), io=5988KiB (6132kB), run=9417-9878msec WRITE: bw=638KiB/s (653kB/s), 160KiB/s-166KiB/s (164kB/s-170kB/s), io=6300KiB (6451kB), run=9417-9878msec