== sanity test 398b: DIO and buffer IO race ============== 05:22:35 (1761124955) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.226912 s, 55.5 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147725: Wed Oct 22 05:22:53 2025 read: IOPS=44, BW=179KiB/s (183kB/s)(1480KiB/8285msec) clat (usec): min=7, max=92815, avg=12100.35, stdev=13852.52 lat (usec): min=8, max=92815, avg=12104.33, stdev=13849.94 clat percentiles (usec): | 1.00th=[ 10], 5.00th=[ 13], 10.00th=[ 17], 20.00th=[ 25], | 30.00th=[ 34], 40.00th=[ 1467], 50.00th=[10159], 60.00th=[13829], | 70.00th=[17171], 80.00th=[21365], 90.00th=[29230], 95.00th=[36963], | 99.00th=[64226], 99.50th=[83362], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 112, max= 285, per=27.26%, avg=179.94, stdev=54.82, samples=16 iops : min= 28, max= 71, avg=44.44, stdev=13.73, samples=16 write: IOPS=48, BW=192KiB/s (197kB/s)(1592KiB/8285msec); 0 zone resets clat (usec): min=158, max=116475, avg=9509.79, stdev=12909.51 lat (usec): min=159, max=116476, avg=9510.89, stdev=12909.49 clat percentiles (usec): | 1.00th=[ 178], 5.00th=[ 215], 10.00th=[ 253], 20.00th=[ 330], | 30.00th=[ 529], 40.00th=[ 1778], 50.00th=[ 5342], 60.00th=[ 8356], | 70.00th=[ 11994], 80.00th=[ 15795], 90.00th=[ 26870], 95.00th=[ 34866], | 99.00th=[ 57410], 99.50th=[ 76022], 99.90th=[116917], 99.95th=[116917], | 99.99th=[116917] bw ( KiB/s): min= 55, max= 294, per=27.72%, avg=192.62, stdev=60.37, samples=16 iops : min= 13, max= 73, avg=47.62, stdev=15.16, samples=16 lat (usec) : 10=0.65%, 20=6.38%, 50=9.90%, 100=1.04%, 250=5.60% lat (usec) : 500=9.90%, 750=3.52%, 1000=0.78% lat (msec) : 2=2.86%, 4=2.34%, 10=14.84%, 20=24.22%, 50=16.28% lat (msec) : 100=1.56%, 250=0.13% cpu : usr=0.12%, sys=5.26%, ctx=886, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147726: Wed Oct 22 05:22:53 2025 read: IOPS=40, BW=163KiB/s (167kB/s)(1436KiB/8794msec) clat (usec): min=8, max=94348, avg=13741.41, stdev=14178.46 lat (usec): min=8, max=94349, avg=13742.29, stdev=14178.50 clat percentiles (usec): | 1.00th=[ 10], 5.00th=[ 14], 10.00th=[ 20], 20.00th=[ 29], | 30.00th=[ 53], 40.00th=[ 8356], 50.00th=[12256], 60.00th=[15664], | 70.00th=[18744], 80.00th=[23462], 90.00th=[31065], 95.00th=[39060], | 99.00th=[63701], 99.50th=[82314], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 100, max= 262, per=24.74%, avg=163.29, stdev=45.50, samples=17 iops : min= 25, max= 65, avg=40.29, stdev=11.27, samples=17 write: IOPS=46, BW=186KiB/s (191kB/s)(1636KiB/8794msec); 0 zone resets clat (usec): min=170, max=82827, avg=9355.99, stdev=12102.50 lat (usec): min=171, max=82827, avg=9362.31, stdev=12099.57 clat percentiles (usec): | 1.00th=[ 194], 5.00th=[ 219], 10.00th=[ 251], 20.00th=[ 334], | 30.00th=[ 490], 40.00th=[ 1205], 50.00th=[ 5342], 60.00th=[ 8455], | 70.00th=[11600], 80.00th=[17171], 90.00th=[25297], 95.00th=[31065], | 99.00th=[53740], 99.50th=[56886], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 127, max= 272, per=26.83%, avg=186.47, stdev=40.56, samples=17 iops : min= 31, max= 68, avg=46.06, stdev=10.19, samples=17 lat (usec) : 10=0.52%, 20=4.17%, 50=9.11%, 100=0.78%, 250=5.73% lat (usec) : 500=11.20%, 750=3.52%, 1000=1.30% lat (msec) : 2=2.34%, 4=2.08%, 10=15.23%, 20=23.83%, 50=18.36% lat (msec) : 100=1.82% cpu : usr=0.11%, sys=5.61%, ctx=952, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147727: Wed Oct 22 05:22:53 2025 read: IOPS=43, BW=173KiB/s (177kB/s)(1564KiB/9060msec) clat (usec): min=4, max=118236, avg=13939.78, stdev=14499.57 lat (usec): min=4, max=118236, avg=13940.76, stdev=14499.51 clat percentiles (usec): | 1.00th=[ 11], 5.00th=[ 16], 10.00th=[ 21], 20.00th=[ 30], | 30.00th=[ 46], 40.00th=[ 8455], 50.00th=[ 12518], 60.00th=[ 16450], | 70.00th=[ 19530], 80.00th=[ 24773], 90.00th=[ 29754], 95.00th=[ 38536], | 99.00th=[ 56886], 99.50th=[ 83362], 99.90th=[117965], 99.95th=[117965], | 99.99th=[117965] bw ( KiB/s): min= 88, max= 264, per=26.42%, avg=174.35, stdev=48.87, samples=17 iops : min= 22, max= 66, avg=43.00, stdev=12.20, samples=17 write: IOPS=41, BW=166KiB/s (170kB/s)(1508KiB/9060msec); 0 zone resets clat (usec): min=106, max=61590, avg=9495.84, stdev=10980.42 lat (usec): min=107, max=61591, avg=9497.02, stdev=10980.44 clat percentiles (usec): | 1.00th=[ 169], 5.00th=[ 215], 10.00th=[ 265], 20.00th=[ 412], | 30.00th=[ 594], 40.00th=[ 2442], 50.00th=[ 6521], 60.00th=[ 9372], | 70.00th=[12256], 80.00th=[16450], 90.00th=[23987], 95.00th=[32113], | 99.00th=[50070], 99.50th=[55313], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 87, max= 295, per=23.37%, avg=162.41, stdev=67.65, samples=17 iops : min= 21, max= 73, avg=40.00, stdev=16.95, samples=17 lat (usec) : 10=0.26%, 20=4.56%, 50=10.68%, 100=1.04%, 250=4.43% lat (usec) : 500=8.07%, 750=3.52%, 1000=1.04% lat (msec) : 2=2.34%, 4=2.34%, 10=15.76%, 20=23.44%, 50=21.35% lat (msec) : 100=1.04%, 250=0.13% cpu : usr=0.22%, sys=5.32%, ctx=970, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147728: Wed Oct 22 05:22:53 2025 read: IOPS=45, BW=183KiB/s (187kB/s)(1508KiB/8256msec) clat (usec): min=7, max=100265, avg=11899.50, stdev=13636.66 lat (usec): min=7, max=100266, avg=11900.29, stdev=13636.65 clat percentiles (usec): | 1.00th=[ 10], 5.00th=[ 13], 10.00th=[ 15], 20.00th=[ 23], | 30.00th=[ 30], 40.00th=[ 221], 50.00th=[ 10159], 60.00th=[ 13304], | 70.00th=[ 17695], 80.00th=[ 21890], 90.00th=[ 28443], 95.00th=[ 33424], | 99.00th=[ 59507], 99.50th=[ 78119], 99.90th=[100140], 99.95th=[100140], | 99.99th=[100140] bw ( KiB/s): min= 89, max= 256, per=27.24%, avg=179.80, stdev=50.17, samples=15 iops : min= 22, max= 64, avg=44.40, stdev=12.53, samples=15 write: IOPS=47, BW=189KiB/s (194kB/s)(1564KiB/8256msec); 0 zone resets clat (usec): min=158, max=130468, avg=9586.94, stdev=13417.71 lat (usec): min=159, max=130468, avg=9588.42, stdev=13417.38 clat percentiles (usec): | 1.00th=[ 182], 5.00th=[ 215], 10.00th=[ 243], 20.00th=[ 322], | 30.00th=[ 474], 40.00th=[ 1844], 50.00th=[ 5276], 60.00th=[ 7635], | 70.00th=[ 11076], 80.00th=[ 16319], 90.00th=[ 26084], 95.00th=[ 35914], | 99.00th=[ 56361], 99.50th=[ 57410], 99.90th=[130548], 99.95th=[130548], | 99.99th=[130548] bw ( KiB/s): min= 95, max= 287, per=27.21%, avg=189.13, stdev=60.47, samples=15 iops : min= 23, max= 71, avg=46.67, stdev=15.23, samples=15 lat (usec) : 10=1.43%, 20=6.12%, 50=11.07%, 100=0.39%, 250=6.12% lat (usec) : 500=10.94%, 750=1.69%, 1000=1.04% lat (msec) : 2=1.95%, 4=2.47%, 10=15.63%, 20=21.88%, 50=17.45% lat (msec) : 100=1.56%, 250=0.26% cpu : usr=0.24%, sys=5.77%, ctx=911, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=661KiB/s (677kB/s), 163KiB/s-183KiB/s (167kB/s-187kB/s), io=5988KiB (6132kB), run=8256-9060msec WRITE: bw=695KiB/s (712kB/s), 166KiB/s-192KiB/s (170kB/s-197kB/s), io=6300KiB (6451kB), run=8256-9060msec rand-rw: (groupid=0, jobs=1): err= 0: pid=147720: Wed Oct 22 05:22:58 2025 read: IOPS=25, BW=102KiB/s (105kB/s)(1480KiB/14473msec) clat (usec): min=2842, max=96099, avg=13655.48, stdev=13116.89 lat (usec): min=2843, max=96101, avg=13658.38, stdev=13115.87 clat percentiles (usec): | 1.00th=[ 2999], 5.00th=[ 4146], 10.00th=[ 4752], 20.00th=[ 5669], | 30.00th=[ 6980], 40.00th=[ 7701], 50.00th=[ 8717], 60.00th=[ 9765], | 70.00th=[12387], 80.00th=[17957], 90.00th=[30278], 95.00th=[39060], | 99.00th=[67634], 99.50th=[71828], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 40, max= 191, per=24.38%, avg=98.75, stdev=41.19, samples=28 iops : min= 10, max= 47, avg=24.21, stdev=10.34, samples=28 write: IOPS=27, BW=110KiB/s (113kB/s)(1592KiB/14473msec); 0 zone resets clat (usec): min=6435, max=83228, avg=23522.54, stdev=13236.22 lat (usec): min=6435, max=83229, avg=23527.28, stdev=13235.14 clat percentiles (usec): | 1.00th=[ 7963], 5.00th=[10421], 10.00th=[11863], 20.00th=[13698], | 30.00th=[15795], 40.00th=[17695], 50.00th=[19792], 60.00th=[21890], | 70.00th=[26084], 80.00th=[30278], 90.00th=[39060], 95.00th=[53740], | 99.00th=[71828], 99.50th=[79168], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 23, max= 183, per=25.40%, avg=108.21, stdev=45.74, samples=28 iops : min= 5, max= 45, avg=26.61, stdev=11.51, samples=28 lat (msec) : 4=1.82%, 10=29.17%, 20=35.68%, 50=29.04%, 100=4.30% cpu : usr=0.19%, sys=1.87%, ctx=855, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147721: Wed Oct 22 05:22:58 2025 read: IOPS=24, BW=97.3KiB/s (99.6kB/s)(1436KiB/14757msec) clat (usec): min=2543, max=81948, avg=12791.93, stdev=12367.72 lat (usec): min=2544, max=81951, avg=12793.46, stdev=12367.50 clat percentiles (usec): | 1.00th=[ 3294], 5.00th=[ 3851], 10.00th=[ 4555], 20.00th=[ 5473], | 30.00th=[ 6259], 40.00th=[ 6980], 50.00th=[ 8291], 60.00th=[ 9634], | 70.00th=[11731], 80.00th=[16450], 90.00th=[27657], 95.00th=[42206], | 99.00th=[64226], 99.50th=[74974], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 31, max= 183, per=23.76%, avg=96.24, stdev=49.14, samples=29 iops : min= 7, max= 45, avg=23.59, stdev=12.38, samples=29 write: IOPS=27, BW=111KiB/s (114kB/s)(1636KiB/14757msec); 0 zone resets clat (msec): min=5, max=143, avg=24.74, stdev=15.43 lat (msec): min=5, max=143, avg=24.74, stdev=15.43 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 21], 60.00th=[ 24], | 70.00th=[ 28], 80.00th=[ 35], 90.00th=[ 44], 95.00th=[ 52], | 99.00th=[ 74], 99.50th=[ 91], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 47, max= 223, per=25.67%, avg=109.34, stdev=44.99, samples=29 iops : min= 11, max= 55, avg=26.86, stdev=11.27, samples=29 lat (msec) : 4=3.12%, 10=27.99%, 20=34.90%, 50=29.95%, 100=3.78% lat (msec) : 250=0.26% cpu : usr=0.23%, sys=1.73%, ctx=850, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147722: Wed Oct 22 05:22:58 2025 read: IOPS=27, BW=108KiB/s (111kB/s)(1564KiB/14477msec) clat (msec): min=2, max=106, avg=13.46, stdev=13.84 lat (msec): min=2, max=106, avg=13.46, stdev=13.84 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 17], 90.00th=[ 30], 95.00th=[ 41], | 99.00th=[ 81], 99.50th=[ 106], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 39, max= 256, per=26.46%, avg=107.14, stdev=56.41, samples=28 iops : min= 9, max= 64, avg=26.32, stdev=14.19, samples=28 write: IOPS=26, BW=104KiB/s (107kB/s)(1508KiB/14477msec); 0 zone resets clat (usec): min=5734, max=88719, avg=24225.65, stdev=13758.21 lat (usec): min=5735, max=88721, avg=24227.93, stdev=13759.50 clat percentiles (usec): | 1.00th=[ 7111], 5.00th=[ 9896], 10.00th=[11994], 20.00th=[14222], | 30.00th=[15795], 40.00th=[18220], 50.00th=[20317], 60.00th=[22938], | 70.00th=[25822], 80.00th=[31589], 90.00th=[43254], 95.00th=[56886], | 99.00th=[71828], 99.50th=[74974], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 39, max= 191, per=23.66%, avg=100.79, stdev=39.94, samples=28 iops : min= 9, max= 47, avg=24.75, stdev=10.03, samples=28 lat (msec) : 4=2.08%, 10=30.86%, 20=33.98%, 50=28.13%, 100=4.69% lat (msec) : 250=0.26% cpu : usr=0.17%, sys=1.87%, ctx=848, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147723: Wed Oct 22 05:22:58 2025 read: IOPS=25, BW=104KiB/s (106kB/s)(1508KiB/14560msec) clat (usec): min=2384, max=85070, avg=13441.70, stdev=12427.13 lat (usec): min=2385, max=85071, avg=13443.29, stdev=12429.21 clat percentiles (usec): | 1.00th=[ 3064], 5.00th=[ 4080], 10.00th=[ 4555], 20.00th=[ 5538], | 30.00th=[ 6521], 40.00th=[ 7570], 50.00th=[ 9110], 60.00th=[10814], | 70.00th=[12780], 80.00th=[17433], 90.00th=[28967], 95.00th=[43779], | 99.00th=[61604], 99.50th=[68682], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 55, max= 215, per=24.69%, avg=100.00, stdev=42.01, samples=28 iops : min= 13, max= 53, avg=24.54, stdev=10.49, samples=28 write: IOPS=26, BW=107KiB/s (110kB/s)(1564KiB/14560msec); 0 zone resets clat (msec): min=4, max=111, avg=24.09, stdev=14.55 lat (msec): min=4, max=111, avg=24.09, stdev=14.55 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 23], | 70.00th=[ 27], 80.00th=[ 33], 90.00th=[ 42], 95.00th=[ 51], | 99.00th=[ 83], 99.50th=[ 100], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 47, max= 184, per=24.54%, avg=104.54, stdev=45.20, samples=28 iops : min= 11, max= 46, avg=25.68, stdev=11.38, samples=28 lat (msec) : 4=2.08%, 10=27.99%, 20=36.85%, 50=28.65%, 100=4.30% lat (msec) : 250=0.13% cpu : usr=0.23%, sys=1.67%, ctx=831, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=406KiB/s (416kB/s), 97.3KiB/s-108KiB/s (99.6kB/s-111kB/s), io=5988KiB (6132kB), run=14473-14757msec WRITE: bw=427KiB/s (437kB/s), 104KiB/s-111KiB/s (107kB/s-114kB/s), io=6300KiB (6451kB), run=14473-14757msec