== sanity test 398b: DIO and buffer IO race ============== 10:31:37 (1761316297) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.329877 s, 38.1 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148504: Fri Oct 24 10:31:57 2025 read: IOPS=45, BW=181KiB/s (186kB/s)(1480KiB/8168msec) clat (usec): min=4, max=111290, avg=12899.64, stdev=17316.77 lat (usec): min=5, max=111290, avg=12900.25, stdev=17316.82 clat percentiles (usec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 11], 20.00th=[ 16], | 30.00th=[ 22], 40.00th=[ 30], 50.00th=[ 110], 60.00th=[ 13566], | 70.00th=[ 19268], 80.00th=[ 26346], 90.00th=[ 34341], 95.00th=[ 46924], | 99.00th=[ 71828], 99.50th=[ 90702], 99.90th=[111674], 99.95th=[111674], | 99.99th=[111674] bw ( KiB/s): min= 63, max= 477, per=26.31%, avg=166.80, stdev=98.14, samples=15 iops : min= 15, max= 119, avg=41.13, stdev=24.61, samples=15 write: IOPS=48, BW=195KiB/s (200kB/s)(1592KiB/8168msec); 0 zone resets clat (usec): min=148, max=91998, avg=8292.99, stdev=12319.44 lat (usec): min=148, max=91998, avg=8293.72, stdev=12319.49 clat percentiles (usec): | 1.00th=[ 157], 5.00th=[ 174], 10.00th=[ 188], 20.00th=[ 215], | 30.00th=[ 251], 40.00th=[ 355], 50.00th=[ 1614], 60.00th=[ 6456], | 70.00th=[10159], 80.00th=[16909], 90.00th=[24511], 95.00th=[33162], | 99.00th=[55837], 99.50th=[78119], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 102, max= 483, per=27.37%, avg=182.53, stdev=97.25, samples=15 iops : min= 25, max= 120, avg=45.07, stdev=24.27, samples=15 lat (usec) : 10=3.91%, 20=9.51%, 50=9.77%, 100=0.78%, 250=15.89% lat (usec) : 500=7.29%, 750=0.91%, 1000=0.65% lat (msec) : 2=2.47%, 4=2.34%, 10=9.24%, 20=16.41%, 50=18.23% lat (msec) : 100=2.47%, 250=0.13% cpu : usr=0.15%, sys=3.78%, ctx=647, majf=0, minf=18 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148505: Fri Oct 24 10:31:57 2025 read: IOPS=40, BW=161KiB/s (165kB/s)(1436KiB/8937msec) clat (usec): min=6, max=121643, avg=14549.39, stdev=18321.20 lat (usec): min=7, max=121644, avg=14550.04, stdev=18321.26 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 11], 10.00th=[ 14], 20.00th=[ 21], | 30.00th=[ 27], 40.00th=[ 38], 50.00th=[ 9110], 60.00th=[ 16319], | 70.00th=[ 20841], 80.00th=[ 25822], 90.00th=[ 39060], 95.00th=[ 46924], | 99.00th=[ 81265], 99.50th=[101188], 99.90th=[122160], 99.95th=[122160], | 99.99th=[122160] bw ( KiB/s): min= 63, max= 308, per=24.71%, avg=156.65, stdev=67.60, samples=17 iops : min= 15, max= 77, avg=38.59, stdev=17.03, samples=17 write: IOPS=45, BW=183KiB/s (187kB/s)(1636KiB/8937msec); 0 zone resets clat (usec): min=147, max=72681, avg=8992.02, stdev=12281.89 lat (usec): min=148, max=72682, avg=8992.84, stdev=12281.92 clat percentiles (usec): | 1.00th=[ 167], 5.00th=[ 186], 10.00th=[ 196], 20.00th=[ 225], | 30.00th=[ 297], 40.00th=[ 429], 50.00th=[ 4178], 60.00th=[ 7373], | 70.00th=[10683], 80.00th=[15926], 90.00th=[27919], 95.00th=[33817], | 99.00th=[53740], 99.50th=[59507], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 86, max= 347, per=26.78%, avg=178.65, stdev=71.43, samples=17 iops : min= 21, max= 86, avg=44.06, stdev=17.84, samples=17 lat (usec) : 10=2.21%, 20=7.03%, 50=10.94%, 100=0.26%, 250=12.50% lat (usec) : 500=9.51%, 750=0.26%, 1000=0.52% lat (msec) : 2=2.08%, 4=2.08%, 10=12.89%, 20=15.63%, 50=21.09% lat (msec) : 100=2.73%, 250=0.26% cpu : usr=0.17%, sys=3.68%, ctx=687, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148506: Fri Oct 24 10:31:57 2025 read: IOPS=41, BW=166KiB/s (170kB/s)(1564KiB/9437msec) clat (usec): min=5, max=206877, avg=15208.43, stdev=19696.97 lat (usec): min=5, max=206877, avg=15209.13, stdev=19697.02 clat percentiles (usec): | 1.00th=[ 7], 5.00th=[ 10], 10.00th=[ 13], 20.00th=[ 20], | 30.00th=[ 26], 40.00th=[ 37], 50.00th=[ 12649], 60.00th=[ 17433], | 70.00th=[ 22938], 80.00th=[ 26870], 90.00th=[ 35914], 95.00th=[ 44827], | 99.00th=[ 82314], 99.50th=[112722], 99.90th=[206570], 99.95th=[206570], | 99.99th=[206570] bw ( KiB/s): min= 63, max= 282, per=25.68%, avg=162.83, stdev=53.53, samples=18 iops : min= 15, max= 70, avg=40.17, stdev=13.35, samples=18 write: IOPS=39, BW=160KiB/s (164kB/s)(1508KiB/9437msec); 0 zone resets clat (usec): min=145, max=90551, avg=9029.67, stdev=13351.72 lat (usec): min=145, max=90551, avg=9030.52, stdev=13351.76 clat percentiles (usec): | 1.00th=[ 161], 5.00th=[ 182], 10.00th=[ 196], 20.00th=[ 229], | 30.00th=[ 289], 40.00th=[ 379], 50.00th=[ 2540], 60.00th=[ 7373], | 70.00th=[10290], 80.00th=[16188], 90.00th=[25822], 95.00th=[33424], | 99.00th=[66847], 99.50th=[78119], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 78, max= 344, per=22.98%, avg=153.28, stdev=65.92, samples=18 iops : min= 19, max= 86, avg=37.78, stdev=16.58, samples=18 lat (usec) : 10=3.26%, 20=7.03%, 50=11.07%, 100=0.26%, 250=11.72% lat (usec) : 500=9.11%, 750=0.91%, 1000=0.65% lat (msec) : 2=1.69%, 4=1.95%, 10=10.03%, 20=16.80%, 50=22.27% lat (msec) : 100=2.99%, 250=0.26% cpu : usr=0.16%, sys=3.92%, ctx=748, majf=0, minf=17 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148507: Fri Oct 24 10:31:57 2025 read: IOPS=52, BW=209KiB/s (214kB/s)(1508KiB/7229msec) clat (usec): min=6, max=122341, avg=11015.96, stdev=15602.09 lat (usec): min=6, max=122342, avg=11016.54, stdev=15602.17 clat percentiles (usec): | 1.00th=[ 7], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 15], | 30.00th=[ 20], 40.00th=[ 25], 50.00th=[ 38], 60.00th=[ 10945], | 70.00th=[ 17957], 80.00th=[ 23462], 90.00th=[ 31065], 95.00th=[ 37487], | 99.00th=[ 65799], 99.50th=[ 74974], 99.90th=[122160], 99.95th=[122160], | 99.99th=[122160] bw ( KiB/s): min= 86, max= 521, per=29.57%, avg=187.46, stdev=118.80, samples=13 iops : min= 21, max= 130, avg=46.31, stdev=29.75, samples=13 write: IOPS=54, BW=216KiB/s (222kB/s)(1564KiB/7229msec); 0 zone resets clat (usec): min=152, max=84081, avg=7648.54, stdev=12353.14 lat (usec): min=152, max=84082, avg=7649.31, stdev=12353.22 clat percentiles (usec): | 1.00th=[ 165], 5.00th=[ 178], 10.00th=[ 196], 20.00th=[ 223], | 30.00th=[ 260], 40.00th=[ 322], 50.00th=[ 914], 60.00th=[ 5145], | 70.00th=[ 8586], 80.00th=[13042], 90.00th=[23200], 95.00th=[32637], | 99.00th=[63701], 99.50th=[81265], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 86, max= 555, per=29.65%, avg=197.77, stdev=119.25, samples=13 iops : min= 21, max= 138, avg=48.85, stdev=29.76, samples=13 lat (usec) : 10=4.04%, 20=10.81%, 50=10.94%, 100=0.52%, 250=14.45% lat (usec) : 500=10.03%, 750=1.30%, 1000=0.39% lat (msec) : 2=1.69%, 4=2.08%, 10=10.03%, 20=14.45%, 50=17.71% lat (msec) : 100=1.43%, 250=0.13% cpu : usr=0.21%, sys=4.18%, ctx=591, majf=0, minf=17 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=635KiB/s (650kB/s), 161KiB/s-209KiB/s (165kB/s-214kB/s), io=5988KiB (6132kB), run=7229-9437msec WRITE: bw=668KiB/s (684kB/s), 160KiB/s-216KiB/s (164kB/s-222kB/s), io=6300KiB (6451kB), run=7229-9437msec rand-rw: (groupid=0, jobs=1): err= 0: pid=148509: Fri Oct 24 10:32:07 2025 read: IOPS=19, BW=78.8KiB/s (80.7kB/s)(1480KiB/18789msec) clat (msec): min=3, max=224, avg=16.63, stdev=17.57 lat (msec): min=3, max=224, avg=16.63, stdev=17.57 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 15], | 70.00th=[ 17], 80.00th=[ 20], 90.00th=[ 29], 95.00th=[ 47], | 99.00th=[ 82], 99.50th=[ 101], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 33, max= 143, per=24.58%, avg=77.42, stdev=26.66, samples=36 iops : min= 8, max= 35, avg=18.78, stdev= 6.63, samples=36 write: IOPS=21, BW=84.7KiB/s (86.8kB/s)(1592KiB/18789msec); 0 zone resets clat (msec): min=6, max=121, avg=31.62, stdev=15.91 lat (msec): min=6, max=121, avg=31.62, stdev=15.91 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 14], 10.00th=[ 17], 20.00th=[ 20], | 30.00th=[ 23], 40.00th=[ 26], 50.00th=[ 29], 60.00th=[ 32], | 70.00th=[ 35], 80.00th=[ 41], 90.00th=[ 52], 95.00th=[ 62], | 99.00th=[ 104], 99.50th=[ 107], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 23, max= 135, per=25.47%, avg=84.31, stdev=31.90, samples=36 iops : min= 5, max= 33, avg=20.50, stdev= 7.94, samples=36 lat (msec) : 4=0.39%, 10=18.10%, 20=31.64%, 50=42.32%, 100=6.77% lat (msec) : 250=0.78% cpu : usr=0.13%, sys=1.31%, ctx=835, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148510: Fri Oct 24 10:32:07 2025 read: IOPS=18, BW=75.6KiB/s (77.4kB/s)(1436KiB/19001msec) clat (msec): min=3, max=215, avg=16.60, stdev=17.31 lat (msec): min=3, max=215, avg=16.60, stdev=17.31 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 33], 95.00th=[ 52], | 99.00th=[ 83], 99.50th=[ 93], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 11, max= 143, per=23.88%, avg=75.22, stdev=36.18, samples=37 iops : min= 2, max= 35, avg=18.22, stdev= 9.06, samples=37 write: IOPS=21, BW=86.1KiB/s (88.2kB/s)(1636KiB/19001msec); 0 zone resets clat (msec): min=6, max=108, avg=31.77, stdev=16.98 lat (msec): min=6, max=108, avg=31.77, stdev=16.98 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 14], 10.00th=[ 16], 20.00th=[ 20], | 30.00th=[ 22], 40.00th=[ 25], 50.00th=[ 28], 60.00th=[ 31], | 70.00th=[ 36], 80.00th=[ 43], 90.00th=[ 55], 95.00th=[ 63], | 99.00th=[ 96], 99.50th=[ 102], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 23, max= 168, per=25.81%, avg=85.43, stdev=33.64, samples=37 iops : min= 5, max= 42, avg=20.76, stdev= 8.47, samples=37 lat (msec) : 4=0.26%, 10=18.62%, 20=30.86%, 50=41.02%, 100=8.59% lat (msec) : 250=0.65% cpu : usr=0.19%, sys=1.29%, ctx=842, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148511: Fri Oct 24 10:32:07 2025 read: IOPS=21, BW=84.7KiB/s (86.7kB/s)(1564KiB/18471msec) clat (usec): min=3132, max=68186, avg=14581.72, stdev=11224.57 lat (usec): min=3133, max=68188, avg=14582.89, stdev=11224.49 clat percentiles (usec): | 1.00th=[ 3589], 5.00th=[ 4817], 10.00th=[ 5932], 20.00th=[ 7177], | 30.00th=[ 8225], 40.00th=[ 9634], 50.00th=[11338], 60.00th=[13173], | 70.00th=[14615], 80.00th=[18744], 90.00th=[25822], 95.00th=[40109], | 99.00th=[62653], 99.50th=[66323], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 22, max= 223, per=27.00%, avg=85.06, stdev=45.45, samples=36 iops : min= 5, max= 55, avg=20.69, stdev=11.40, samples=36 write: IOPS=20, BW=81.6KiB/s (83.6kB/s)(1508KiB/18471msec); 0 zone resets clat (msec): min=8, max=228, avg=33.74, stdev=20.12 lat (msec): min=8, max=228, avg=33.75, stdev=20.12 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 15], 10.00th=[ 18], 20.00th=[ 21], | 30.00th=[ 24], 40.00th=[ 26], 50.00th=[ 29], 60.00th=[ 32], | 70.00th=[ 37], 80.00th=[ 45], 90.00th=[ 58], 95.00th=[ 70], | 99.00th=[ 96], 99.50th=[ 167], 99.90th=[ 230], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 22, max= 151, per=24.56%, avg=81.28, stdev=30.52, samples=36 iops : min= 5, max= 37, avg=19.72, stdev= 7.61, samples=36 lat (msec) : 4=0.91%, 10=20.31%, 20=29.95%, 50=40.49%, 100=7.94% lat (msec) : 250=0.39% cpu : usr=0.13%, sys=1.36%, ctx=854, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148512: Fri Oct 24 10:32:07 2025 read: IOPS=20, BW=80.5KiB/s (82.4kB/s)(1508KiB/18733msec) clat (msec): min=3, max=113, avg=16.66, stdev=15.36 lat (msec): min=3, max=113, avg=16.66, stdev=15.36 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 9], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 17], 80.00th=[ 21], 90.00th=[ 30], 95.00th=[ 50], | 99.00th=[ 93], 99.50th=[ 103], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 23, max= 159, per=25.03%, avg=78.83, stdev=31.84, samples=36 iops : min= 5, max= 39, avg=19.11, stdev= 7.97, samples=36 write: IOPS=20, BW=83.5KiB/s (85.5kB/s)(1564KiB/18733msec); 0 zone resets clat (msec): min=6, max=190, avg=31.73, stdev=16.83 lat (msec): min=6, max=190, avg=31.73, stdev=16.83 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 16], 10.00th=[ 18], 20.00th=[ 20], | 30.00th=[ 24], 40.00th=[ 26], 50.00th=[ 28], 60.00th=[ 31], | 70.00th=[ 35], 80.00th=[ 42], 90.00th=[ 50], 95.00th=[ 60], | 99.00th=[ 100], 99.50th=[ 111], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 23, max= 158, per=25.10%, avg=83.08, stdev=32.71, samples=36 iops : min= 5, max= 39, avg=20.17, stdev= 8.20, samples=36 lat (msec) : 4=0.39%, 10=17.32%, 20=31.38%, 50=43.62%, 100=6.51% lat (msec) : 250=0.78% cpu : usr=0.11%, sys=1.27%, ctx=840, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=315KiB/s (323kB/s), 75.6KiB/s-84.7KiB/s (77.4kB/s-86.7kB/s), io=5988KiB (6132kB), run=18471-19001msec WRITE: bw=332KiB/s (340kB/s), 81.6KiB/s-86.1KiB/s (83.6kB/s-88.2kB/s), io=6300KiB (6451kB), run=18471-19001msec