== sanity test 398b: DIO and buffer IO race ============== 09:49:26 (1761313766) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.322376 s, 39.0 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148369: Fri Oct 24 09:49:43 2025 read: IOPS=53, BW=213KiB/s (219kB/s)(1480KiB/6933msec) clat (usec): min=7, max=108974, avg=11110.85, stdev=12866.77 lat (usec): min=8, max=108975, avg=11111.45, stdev=12866.79 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 17], 20.00th=[ 24], | 30.00th=[ 32], 40.00th=[ 5735], 50.00th=[ 9634], 60.00th=[ 11731], | 70.00th=[ 14746], 80.00th=[ 18482], 90.00th=[ 24773], 95.00th=[ 33817], | 99.00th=[ 59507], 99.50th=[ 66323], 99.90th=[108528], 99.95th=[108528], | 99.99th=[108528] bw ( KiB/s): min= 109, max= 286, per=25.99%, avg=203.77, stdev=61.28, samples=13 iops : min= 27, max= 71, avg=50.38, stdev=15.32, samples=13 write: IOPS=57, BW=230KiB/s (235kB/s)(1592KiB/6933msec); 0 zone resets clat (usec): min=141, max=63716, avg=7053.71, stdev=9172.15 lat (usec): min=141, max=63716, avg=7054.55, stdev=9172.11 clat percentiles (usec): | 1.00th=[ 161], 5.00th=[ 186], 10.00th=[ 210], 20.00th=[ 262], | 30.00th=[ 359], 40.00th=[ 1303], 50.00th=[ 4228], 60.00th=[ 5997], | 70.00th=[ 8717], 80.00th=[12387], 90.00th=[17433], 95.00th=[26084], | 99.00th=[40109], 99.50th=[55313], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 125, max= 392, per=26.90%, avg=221.62, stdev=99.68, samples=13 iops : min= 31, max= 98, avg=54.85, stdev=24.91, samples=13 lat (usec) : 10=0.78%, 20=5.99%, 50=9.38%, 100=0.26%, 250=9.51% lat (usec) : 500=8.72%, 750=1.04%, 1000=0.78% lat (msec) : 2=2.47%, 4=4.04%, 10=19.14%, 20=26.04%, 50=10.68% lat (msec) : 100=1.04%, 250=0.13% cpu : usr=0.20%, sys=4.44%, ctx=872, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148370: Fri Oct 24 09:49:43 2025 read: IOPS=51, BW=206KiB/s (211kB/s)(1436KiB/6966msec) clat (usec): min=6, max=108140, avg=11058.09, stdev=12905.43 lat (usec): min=6, max=108140, avg=11059.68, stdev=12904.65 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 13], 10.00th=[ 17], 20.00th=[ 24], | 30.00th=[ 33], 40.00th=[ 6456], 50.00th=[ 9765], 60.00th=[ 11863], | 70.00th=[ 14746], 80.00th=[ 18482], 90.00th=[ 23725], 95.00th=[ 32900], | 99.00th=[ 53216], 99.50th=[105382], 99.90th=[108528], 99.95th=[108528], | 99.99th=[108528] bw ( KiB/s): min= 70, max= 311, per=25.77%, avg=202.00, stdev=73.04, samples=13 iops : min= 17, max= 77, avg=49.92, stdev=18.32, samples=13 write: IOPS=58, BW=235KiB/s (240kB/s)(1636KiB/6966msec); 0 zone resets clat (usec): min=156, max=49475, avg=7297.69, stdev=8565.40 lat (usec): min=156, max=49476, avg=7298.47, stdev=8565.44 clat percentiles (usec): | 1.00th=[ 174], 5.00th=[ 192], 10.00th=[ 210], 20.00th=[ 262], | 30.00th=[ 334], 40.00th=[ 1991], 50.00th=[ 5145], 60.00th=[ 6718], | 70.00th=[ 9765], 80.00th=[12911], 90.00th=[18482], 95.00th=[25822], | 99.00th=[39060], 99.50th=[40633], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 133, max= 359, per=27.41%, avg=225.85, stdev=70.47, samples=13 iops : min= 33, max= 89, avg=55.92, stdev=17.51, samples=13 lat (usec) : 10=1.56%, 20=5.99%, 50=8.20%, 250=9.64%, 500=8.46% lat (usec) : 750=1.43%, 1000=0.91% lat (msec) : 2=1.17%, 4=2.99%, 10=21.74%, 20=25.26%, 50=12.11% lat (msec) : 100=0.26%, 250=0.26% cpu : usr=0.16%, sys=4.58%, ctx=874, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148371: Fri Oct 24 09:49:43 2025 read: IOPS=51, BW=205KiB/s (210kB/s)(1564KiB/7637msec) clat (usec): min=5, max=75537, avg=11327.01, stdev=12058.81 lat (usec): min=5, max=75539, avg=11327.69, stdev=12058.87 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 12], 10.00th=[ 18], 20.00th=[ 24], | 30.00th=[ 35], 40.00th=[ 7046], 50.00th=[ 9896], 60.00th=[12518], | 70.00th=[15533], 80.00th=[19268], 90.00th=[23987], 95.00th=[32900], | 99.00th=[66323], 99.50th=[71828], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 71, max= 341, per=25.91%, avg=203.13, stdev=84.61, samples=15 iops : min= 17, max= 85, avg=50.20, stdev=21.27, samples=15 write: IOPS=49, BW=197KiB/s (202kB/s)(1508KiB/7637msec); 0 zone resets clat (usec): min=141, max=101384, avg=8483.12, stdev=11200.45 lat (usec): min=141, max=101385, avg=8483.97, stdev=11200.44 clat percentiles (usec): | 1.00th=[ 172], 5.00th=[ 198], 10.00th=[ 221], 20.00th=[ 281], | 30.00th=[ 445], 40.00th=[ 3523], 50.00th=[ 5735], 60.00th=[ 7570], | 70.00th=[ 9634], 80.00th=[ 13304], 90.00th=[ 20841], 95.00th=[ 32900], | 99.00th=[ 47449], 99.50th=[ 56361], 99.90th=[101188], 99.95th=[101188], | 99.99th=[101188] bw ( KiB/s): min= 86, max= 280, per=23.50%, avg=193.67, stdev=63.95, samples=15 iops : min= 21, max= 70, avg=47.80, stdev=16.05, samples=15 lat (usec) : 10=1.95%, 20=5.86%, 50=8.72%, 250=8.46%, 500=6.77% lat (usec) : 750=0.78%, 1000=1.17% lat (msec) : 2=1.04%, 4=3.26%, 10=22.53%, 20=25.39%, 50=13.15% lat (msec) : 100=0.78%, 250=0.13% cpu : usr=0.21%, sys=4.61%, ctx=971, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148372: Fri Oct 24 09:49:43 2025 read: IOPS=54, BW=219KiB/s (224kB/s)(1508KiB/6886msec) clat (usec): min=7, max=135646, avg=10691.37, stdev=13121.59 lat (usec): min=7, max=135648, avg=10692.63, stdev=13121.46 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 15], 10.00th=[ 18], 20.00th=[ 25], | 30.00th=[ 32], 40.00th=[ 6325], 50.00th=[ 9241], 60.00th=[ 11994], | 70.00th=[ 14353], 80.00th=[ 17433], 90.00th=[ 22938], 95.00th=[ 30016], | 99.00th=[ 53216], 99.50th=[103285], 99.90th=[135267], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 55, max= 343, per=26.81%, avg=210.15, stdev=86.70, samples=13 iops : min= 13, max= 85, avg=51.92, stdev=21.74, samples=13 write: IOPS=56, BW=227KiB/s (233kB/s)(1564KiB/6886msec); 0 zone resets clat (usec): min=150, max=57542, avg=7262.90, stdev=9743.70 lat (usec): min=150, max=57542, avg=7264.12, stdev=9743.81 clat percentiles (usec): | 1.00th=[ 165], 5.00th=[ 188], 10.00th=[ 204], 20.00th=[ 243], | 30.00th=[ 326], 40.00th=[ 1270], 50.00th=[ 4228], 60.00th=[ 5997], | 70.00th=[ 8455], 80.00th=[12387], 90.00th=[19006], 95.00th=[27919], | 99.00th=[50070], 99.50th=[54264], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 78, max= 336, per=26.98%, avg=222.31, stdev=89.04, samples=13 iops : min= 19, max= 84, avg=55.00, stdev=22.27, samples=13 lat (usec) : 10=0.78%, 20=5.08%, 50=11.07%, 100=0.39%, 250=11.07% lat (usec) : 500=7.16%, 750=1.56%, 1000=0.65% lat (msec) : 2=1.43%, 4=3.91%, 10=21.74%, 20=22.53%, 50=11.72% lat (msec) : 100=0.65%, 250=0.26% cpu : usr=0.15%, sys=5.29%, ctx=936, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=784KiB/s (803kB/s), 205KiB/s-219KiB/s (210kB/s-224kB/s), io=5988KiB (6132kB), run=6886-7637msec WRITE: bw=825KiB/s (845kB/s), 197KiB/s-235KiB/s (202kB/s-240kB/s), io=6300KiB (6451kB), run=6886-7637msec rand-rw: (groupid=0, jobs=1): err= 0: pid=148364: Fri Oct 24 09:49:47 2025 read: IOPS=31, BW=126KiB/s (129kB/s)(1480KiB/11714msec) clat (usec): min=2098, max=75346, avg=10192.97, stdev=9063.29 lat (usec): min=2099, max=75347, avg=10194.61, stdev=9062.95 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 3163], 10.00th=[ 3851], 20.00th=[ 4555], | 30.00th=[ 5342], 40.00th=[ 6194], 50.00th=[ 7242], 60.00th=[ 8356], | 70.00th=[10552], 80.00th=[12780], 90.00th=[20841], 95.00th=[28181], | 99.00th=[48497], 99.50th=[55837], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 47, max= 255, per=24.66%, avg=125.04, stdev=56.81, samples=23 iops : min= 11, max= 63, avg=30.78, stdev=14.20, samples=23 write: IOPS=33, BW=136KiB/s (139kB/s)(1592KiB/11714msec); 0 zone resets clat (usec): min=5182, max=67324, avg=19818.12, stdev=10658.76 lat (usec): min=5184, max=67324, avg=19819.31, stdev=10658.73 clat percentiles (usec): | 1.00th=[ 6849], 5.00th=[ 8291], 10.00th=[ 9765], 20.00th=[11469], | 30.00th=[13042], 40.00th=[14877], 50.00th=[16909], 60.00th=[18744], | 70.00th=[21365], 80.00th=[26608], 90.00th=[34341], 95.00th=[42730], | 99.00th=[57410], 99.50th=[63177], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 56, max= 231, per=25.33%, avg=135.26, stdev=52.81, samples=23 iops : min= 14, max= 57, avg=33.35, stdev=13.22, samples=23 lat (msec) : 4=5.73%, 10=32.94%, 20=36.72%, 50=23.31%, 100=1.30% cpu : usr=0.13%, sys=1.49%, ctx=828, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148365: Fri Oct 24 09:49:47 2025 read: IOPS=30, BW=122KiB/s (125kB/s)(1436KiB/11788msec) clat (usec): min=1986, max=90619, avg=10417.66, stdev=9820.84 lat (usec): min=1987, max=90623, avg=10418.73, stdev=9820.90 clat percentiles (usec): | 1.00th=[ 2245], 5.00th=[ 2802], 10.00th=[ 3458], 20.00th=[ 4752], | 30.00th=[ 5473], 40.00th=[ 6390], 50.00th=[ 7242], 60.00th=[ 8225], | 70.00th=[10028], 80.00th=[13566], 90.00th=[22676], 95.00th=[30278], | 99.00th=[46924], 99.50th=[69731], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 42, max= 247, per=23.88%, avg=121.09, stdev=58.48, samples=23 iops : min= 10, max= 61, avg=29.78, stdev=14.61, samples=23 write: IOPS=34, BW=139KiB/s (142kB/s)(1636KiB/11788msec); 0 zone resets clat (usec): min=5119, max=84876, avg=19590.45, stdev=11281.44 lat (usec): min=5120, max=84877, avg=19591.76, stdev=11281.34 clat percentiles (usec): | 1.00th=[ 6063], 5.00th=[ 8160], 10.00th=[ 9241], 20.00th=[11207], | 30.00th=[12649], 40.00th=[14353], 50.00th=[16450], 60.00th=[19006], | 70.00th=[22414], 80.00th=[27132], 90.00th=[33424], 95.00th=[41157], | 99.00th=[58983], 99.50th=[68682], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 63, max= 248, per=25.62%, avg=136.83, stdev=52.22, samples=23 iops : min= 15, max= 62, avg=33.74, stdev=13.10, samples=23 lat (msec) : 2=0.13%, 4=6.12%, 10=33.33%, 20=35.03%, 50=23.70% lat (msec) : 100=1.69% cpu : usr=0.20%, sys=1.54%, ctx=823, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148366: Fri Oct 24 09:49:47 2025 read: IOPS=33, BW=134KiB/s (138kB/s)(1564KiB/11634msec) clat (usec): min=1879, max=52143, avg=10173.20, stdev=9130.65 lat (usec): min=1880, max=52144, avg=10174.18, stdev=9130.62 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 3163], 10.00th=[ 3589], 20.00th=[ 4424], | 30.00th=[ 5145], 40.00th=[ 5997], 50.00th=[ 6783], 60.00th=[ 7701], | 70.00th=[ 9765], 80.00th=[12911], 90.00th=[22938], 95.00th=[31589], | 99.00th=[46924], 99.50th=[49021], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 47, max= 303, per=26.58%, avg=134.74, stdev=70.45, samples=23 iops : min= 11, max= 75, avg=33.22, stdev=17.57, samples=23 write: IOPS=32, BW=130KiB/s (133kB/s)(1508KiB/11634msec); 0 zone resets clat (usec): min=4335, max=73646, avg=20167.40, stdev=11778.17 lat (usec): min=4337, max=73647, avg=20168.90, stdev=11777.89 clat percentiles (usec): | 1.00th=[ 5407], 5.00th=[ 7767], 10.00th=[ 9241], 20.00th=[11076], | 30.00th=[13042], 40.00th=[14484], 50.00th=[17171], 60.00th=[19792], | 70.00th=[23200], 80.00th=[27132], 90.00th=[33817], 95.00th=[44827], | 99.00th=[68682], 99.50th=[71828], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 63, max= 240, per=24.19%, avg=129.17, stdev=52.50, samples=23 iops : min= 15, max= 60, avg=31.83, stdev=13.16, samples=23 lat (msec) : 2=0.26%, 4=6.90%, 10=35.81%, 20=32.03%, 50=23.44% lat (msec) : 100=1.56% cpu : usr=0.14%, sys=1.56%, ctx=833, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148367: Fri Oct 24 09:49:47 2025 read: IOPS=31, BW=128KiB/s (131kB/s)(1508KiB/11796msec) clat (usec): min=2122, max=93217, avg=11279.61, stdev=11081.30 lat (usec): min=2123, max=93218, avg=11280.47, stdev=11081.28 clat percentiles (usec): | 1.00th=[ 2343], 5.00th=[ 2933], 10.00th=[ 3523], 20.00th=[ 4359], | 30.00th=[ 5080], 40.00th=[ 5735], 50.00th=[ 7177], 60.00th=[ 9110], | 70.00th=[11076], 80.00th=[16450], 90.00th=[24511], 95.00th=[36439], | 99.00th=[54264], 99.50th=[56886], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 39, max= 247, per=24.55%, avg=124.48, stdev=59.47, samples=23 iops : min= 9, max= 61, avg=30.61, stdev=14.89, samples=23 write: IOPS=33, BW=133KiB/s (136kB/s)(1564KiB/11796msec); 0 zone resets clat (usec): min=4740, max=86325, avg=19174.48, stdev=11097.02 lat (usec): min=4741, max=86328, avg=19175.67, stdev=11097.01 clat percentiles (usec): | 1.00th=[ 5997], 5.00th=[ 7635], 10.00th=[ 9110], 20.00th=[11076], | 30.00th=[12649], 40.00th=[14091], 50.00th=[15664], 60.00th=[18220], | 70.00th=[21627], 80.00th=[26346], 90.00th=[33162], 95.00th=[41157], | 99.00th=[61080], 99.50th=[72877], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 55, max= 240, per=24.46%, avg=130.61, stdev=58.47, samples=23 iops : min= 13, max= 60, avg=32.17, stdev=14.65, samples=23 lat (msec) : 4=7.94%, 10=31.38%, 20=36.72%, 50=22.27%, 100=1.69% cpu : usr=0.13%, sys=1.57%, ctx=800, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=508KiB/s (520kB/s), 122KiB/s-134KiB/s (125kB/s-138kB/s), io=5988KiB (6132kB), run=11634-11796msec WRITE: bw=534KiB/s (547kB/s), 130KiB/s-139KiB/s (133kB/s-142kB/s), io=6300KiB (6451kB), run=11634-11796msec