== sanity test 398b: DIO and buffer IO race ============== 10:39:13 (1761316753) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.227314 s, 55.4 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=151240: Fri Oct 24 10:39:32 2025 read: IOPS=46, BW=187KiB/s (192kB/s)(1480KiB/7900msec) clat (usec): min=5, max=171330, avg=12424.79, stdev=18964.90 lat (usec): min=6, max=171331, avg=12425.58, stdev=18964.96 clat percentiles (usec): | 1.00th=[ 7], 5.00th=[ 12], 10.00th=[ 15], 20.00th=[ 21], | 30.00th=[ 29], 40.00th=[ 44], 50.00th=[ 8717], 60.00th=[ 13042], | 70.00th=[ 16319], 80.00th=[ 20579], 90.00th=[ 27132], 95.00th=[ 36439], | 99.00th=[119014], 99.50th=[154141], 99.90th=[170918], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 77, max= 286, per=24.33%, avg=184.20, stdev=65.65, samples=15 iops : min= 19, max= 71, avg=45.60, stdev=16.43, samples=15 write: IOPS=50, BW=202KiB/s (206kB/s)(1592KiB/7900msec); 0 zone resets clat (usec): min=147, max=57476, avg=8260.04, stdev=10277.52 lat (usec): min=147, max=57477, avg=8261.74, stdev=10277.22 clat percentiles (usec): | 1.00th=[ 155], 5.00th=[ 180], 10.00th=[ 198], 20.00th=[ 231], | 30.00th=[ 306], 40.00th=[ 1205], 50.00th=[ 5014], 60.00th=[ 7635], | 70.00th=[10683], 80.00th=[16057], 90.00th=[21103], 95.00th=[27919], | 99.00th=[52167], 99.50th=[54789], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 71, max= 341, per=25.26%, avg=201.33, stdev=82.12, samples=15 iops : min= 17, max= 85, avg=49.87, stdev=20.61, samples=15 lat (usec) : 10=1.82%, 20=7.29%, 50=10.29%, 100=0.26%, 250=13.15% lat (usec) : 500=6.12%, 750=0.78%, 1000=0.91% lat (msec) : 2=2.21%, 4=2.21%, 10=16.02%, 20=22.66%, 50=14.71% lat (msec) : 100=1.04%, 250=0.52% cpu : usr=0.18%, sys=4.79%, ctx=763, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151241: Fri Oct 24 10:39:32 2025 read: IOPS=45, BW=183KiB/s (188kB/s)(1436KiB/7836msec) clat (usec): min=6, max=214916, avg=13080.87, stdev=17454.06 lat (usec): min=6, max=214916, avg=13081.76, stdev=17454.04 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 13], 10.00th=[ 18], 20.00th=[ 25], | 30.00th=[ 31], 40.00th=[ 6587], 50.00th=[ 11207], 60.00th=[ 14353], | 70.00th=[ 17433], 80.00th=[ 21890], 90.00th=[ 27657], 95.00th=[ 37487], | 99.00th=[ 72877], 99.50th=[ 90702], 99.90th=[214959], 99.95th=[214959], | 99.99th=[214959] bw ( KiB/s): min= 54, max= 341, per=24.49%, avg=185.40, stdev=71.96, samples=15 iops : min= 13, max= 85, avg=45.87, stdev=18.03, samples=15 write: IOPS=52, BW=209KiB/s (214kB/s)(1636KiB/7836msec); 0 zone resets clat (usec): min=151, max=44932, avg=7633.76, stdev=9485.50 lat (usec): min=152, max=44933, avg=7634.58, stdev=9485.52 clat percentiles (usec): | 1.00th=[ 167], 5.00th=[ 190], 10.00th=[ 204], 20.00th=[ 239], | 30.00th=[ 273], 40.00th=[ 433], 50.00th=[ 4817], 60.00th=[ 7308], | 70.00th=[ 9765], 80.00th=[14222], 90.00th=[21365], 95.00th=[26346], | 99.00th=[41157], 99.50th=[41157], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 102, max= 389, per=26.26%, avg=209.27, stdev=75.34, samples=15 iops : min= 25, max= 97, avg=51.87, stdev=18.89, samples=15 lat (usec) : 10=1.43%, 20=5.34%, 50=8.72%, 100=1.04%, 250=12.11% lat (usec) : 500=9.51%, 750=0.78%, 1000=0.78% lat (msec) : 2=1.17%, 4=1.43%, 10=17.06%, 20=23.83%, 50=15.76% lat (msec) : 100=0.91%, 250=0.13% cpu : usr=0.17%, sys=3.65%, ctx=839, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151242: Fri Oct 24 10:39:32 2025 read: IOPS=50, BW=202KiB/s (207kB/s)(1564KiB/7745msec) clat (usec): min=5, max=77364, avg=11698.48, stdev=13314.55 lat (usec): min=6, max=77365, avg=11699.21, stdev=13314.55 clat percentiles (usec): | 1.00th=[ 7], 5.00th=[ 10], 10.00th=[ 14], 20.00th=[ 21], | 30.00th=[ 29], 40.00th=[ 4015], 50.00th=[ 9110], 60.00th=[13698], | 70.00th=[17171], 80.00th=[21103], 90.00th=[27132], 95.00th=[33817], | 99.00th=[68682], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 79, max= 373, per=27.15%, avg=205.53, stdev=95.66, samples=15 iops : min= 19, max= 93, avg=50.93, stdev=23.90, samples=15 write: IOPS=48, BW=195KiB/s (199kB/s)(1508KiB/7745msec); 0 zone resets clat (usec): min=148, max=66588, avg=8371.75, stdev=11693.66 lat (usec): min=148, max=66589, avg=8372.58, stdev=11693.70 clat percentiles (usec): | 1.00th=[ 167], 5.00th=[ 184], 10.00th=[ 198], 20.00th=[ 227], | 30.00th=[ 265], 40.00th=[ 351], 50.00th=[ 3785], 60.00th=[ 7504], | 70.00th=[10421], 80.00th=[14877], 90.00th=[21890], 95.00th=[31589], | 99.00th=[56886], 99.50th=[65274], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 94, max= 318, per=24.53%, avg=195.47, stdev=79.20, samples=15 iops : min= 23, max= 79, avg=48.40, stdev=19.87, samples=15 lat (usec) : 10=2.86%, 20=7.03%, 50=9.64%, 100=0.13%, 250=12.50% lat (usec) : 500=8.72%, 750=0.26%, 1000=0.39% lat (msec) : 2=1.69%, 4=1.69%, 10=15.10%, 20=22.40%, 50=15.76% lat (msec) : 100=1.82% cpu : usr=0.13%, sys=3.67%, ctx=758, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151243: Fri Oct 24 10:39:32 2025 read: IOPS=50, BW=200KiB/s (205kB/s)(1508KiB/7538msec) clat (usec): min=6, max=103246, avg=10921.64, stdev=12888.62 lat (usec): min=6, max=103247, avg=10922.30, stdev=12888.63 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 11], 10.00th=[ 15], 20.00th=[ 21], | 30.00th=[ 27], 40.00th=[ 37], 50.00th=[ 8455], 60.00th=[ 13698], | 70.00th=[ 16581], 80.00th=[ 21103], 90.00th=[ 26870], 95.00th=[ 32375], | 99.00th=[ 44827], 99.50th=[ 85459], 99.90th=[103285], 99.95th=[103285], | 99.99th=[103285] bw ( KiB/s): min= 78, max= 336, per=26.25%, avg=198.71, stdev=73.15, samples=14 iops : min= 19, max= 84, avg=49.21, stdev=18.36, samples=14 write: IOPS=51, BW=207KiB/s (212kB/s)(1564KiB/7538msec); 0 zone resets clat (usec): min=147, max=55993, avg=8684.00, stdev=11375.80 lat (usec): min=147, max=55993, avg=8684.94, stdev=11375.87 clat percentiles (usec): | 1.00th=[ 165], 5.00th=[ 188], 10.00th=[ 204], 20.00th=[ 237], | 30.00th=[ 306], 40.00th=[ 1057], 50.00th=[ 4555], 60.00th=[ 7767], | 70.00th=[10552], 80.00th=[15533], 90.00th=[25560], 95.00th=[33162], | 99.00th=[52691], 99.50th=[55313], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 96, max= 373, per=26.02%, avg=207.36, stdev=82.09, samples=14 iops : min= 24, max= 93, avg=51.43, stdev=20.55, samples=14 lat (usec) : 10=1.69%, 20=7.29%, 50=11.98%, 100=0.39%, 250=11.85% lat (usec) : 500=7.16%, 750=0.39%, 1000=0.91% lat (msec) : 2=2.47%, 4=2.47%, 10=13.67%, 20=22.01%, 50=16.80% lat (msec) : 100=0.78%, 250=0.13% cpu : usr=0.19%, sys=3.37%, ctx=732, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=758KiB/s (776kB/s), 183KiB/s-202KiB/s (188kB/s-207kB/s), io=5988KiB (6132kB), run=7538-7900msec WRITE: bw=797KiB/s (817kB/s), 195KiB/s-209KiB/s (199kB/s-214kB/s), io=6300KiB (6451kB), run=7538-7900msec rand-rw: (groupid=0, jobs=1): err= 0: pid=151235: Fri Oct 24 10:39:40 2025 read: IOPS=24, BW=99.1KiB/s (102kB/s)(1480KiB/14930msec) clat (usec): min=2567, max=82214, avg=12656.65, stdev=10559.27 lat (usec): min=2568, max=82217, avg=12658.75, stdev=10558.86 clat percentiles (usec): | 1.00th=[ 3326], 5.00th=[ 4621], 10.00th=[ 5276], 20.00th=[ 6194], | 30.00th=[ 6980], 40.00th=[ 7767], 50.00th=[ 8979], 60.00th=[10421], | 70.00th=[13042], 80.00th=[17171], 90.00th=[22938], 95.00th=[33424], | 99.00th=[65274], 99.50th=[80217], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 31, max= 239, per=25.39%, avg=98.00, stdev=41.94, samples=29 iops : min= 7, max= 59, avg=23.93, stdev=10.51, samples=29 write: IOPS=26, BW=107KiB/s (109kB/s)(1592KiB/14930msec); 0 zone resets clat (msec): min=5, max=115, avg=25.62, stdev=13.35 lat (msec): min=5, max=115, avg=25.62, stdev=13.35 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 26], | 70.00th=[ 29], 80.00th=[ 34], 90.00th=[ 44], 95.00th=[ 51], | 99.00th=[ 73], 99.50th=[ 100], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 52, max= 191, per=26.23%, avg=106.48, stdev=39.89, samples=29 iops : min= 13, max= 47, avg=26.10, stdev= 9.95, samples=29 lat (msec) : 4=0.78%, 10=28.52%, 20=31.12%, 50=36.07%, 100=3.39% lat (msec) : 250=0.13% cpu : usr=0.11%, sys=1.41%, ctx=818, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151236: Fri Oct 24 10:39:40 2025 read: IOPS=23, BW=92.7KiB/s (94.9kB/s)(1436KiB/15497msec) clat (msec): min=2, max=191, avg=14.29, stdev=15.94 lat (msec): min=2, max=191, avg=14.29, stdev=15.94 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 14], 80.00th=[ 19], 90.00th=[ 31], 95.00th=[ 41], | 99.00th=[ 83], 99.50th=[ 100], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 22, max= 191, per=23.79%, avg=91.83, stdev=45.69, samples=30 iops : min= 5, max= 47, avg=22.40, stdev=11.47, samples=30 write: IOPS=26, BW=106KiB/s (108kB/s)(1636KiB/15497msec); 0 zone resets clat (usec): min=5235, max=84343, avg=25262.04, stdev=12755.69 lat (usec): min=5236, max=84344, avg=25263.41, stdev=12755.73 clat percentiles (usec): | 1.00th=[ 8356], 5.00th=[10028], 10.00th=[13173], 20.00th=[15664], | 30.00th=[17433], 40.00th=[19530], 50.00th=[22152], 60.00th=[24773], | 70.00th=[28443], 80.00th=[33817], 90.00th=[42730], 95.00th=[49546], | 99.00th=[67634], 99.50th=[70779], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 39, max= 190, per=25.39%, avg=103.10, stdev=39.44, samples=30 iops : min= 9, max= 47, avg=25.23, stdev= 9.88, samples=30 lat (msec) : 4=1.43%, 10=26.30%, 20=33.20%, 50=35.16%, 100=3.78% lat (msec) : 250=0.13% cpu : usr=0.18%, sys=1.29%, ctx=824, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151237: Fri Oct 24 10:39:40 2025 read: IOPS=26, BW=105KiB/s (107kB/s)(1564KiB/14963msec) clat (usec): min=2711, max=68408, avg=13158.65, stdev=11090.26 lat (usec): min=2713, max=68409, avg=13160.22, stdev=11089.88 clat percentiles (usec): | 1.00th=[ 3392], 5.00th=[ 4015], 10.00th=[ 4817], 20.00th=[ 5932], | 30.00th=[ 6980], 40.00th=[ 8225], 50.00th=[ 9241], 60.00th=[10945], | 70.00th=[13698], 80.00th=[17957], 90.00th=[25560], 95.00th=[38536], | 99.00th=[60031], 99.50th=[64226], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 39, max= 238, per=27.08%, avg=104.52, stdev=50.87, samples=29 iops : min= 9, max= 59, avg=25.62, stdev=12.72, samples=29 write: IOPS=25, BW=101KiB/s (103kB/s)(1508KiB/14963msec); 0 zone resets clat (msec): min=5, max=122, avg=25.92, stdev=14.37 lat (msec): min=5, max=122, avg=25.93, stdev=14.37 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 26], | 70.00th=[ 29], 80.00th=[ 33], 90.00th=[ 41], 95.00th=[ 55], | 99.00th=[ 82], 99.50th=[ 96], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 47, max= 183, per=24.46%, avg=99.31, stdev=37.81, samples=29 iops : min= 11, max= 45, avg=24.28, stdev= 9.46, samples=29 lat (msec) : 4=2.47%, 10=28.39%, 20=31.64%, 50=33.72%, 100=3.65% lat (msec) : 250=0.13% cpu : usr=0.11%, sys=1.28%, ctx=811, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151238: Fri Oct 24 10:39:40 2025 read: IOPS=24, BW=98.2KiB/s (101kB/s)(1508KiB/15359msec) clat (usec): min=2468, max=58951, avg=13922.22, stdev=12005.12 lat (usec): min=2469, max=58952, avg=13927.32, stdev=12002.62 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3916], 10.00th=[ 4883], 20.00th=[ 5932], | 30.00th=[ 6849], 40.00th=[ 7767], 50.00th=[ 9110], 60.00th=[10683], | 70.00th=[13829], 80.00th=[19792], 90.00th=[34866], 95.00th=[42206], | 99.00th=[54789], 99.50th=[57934], 99.90th=[58983], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 31, max= 182, per=25.00%, avg=96.50, stdev=40.79, samples=30 iops : min= 7, max= 45, avg=23.60, stdev=10.15, samples=30 write: IOPS=25, BW=102KiB/s (104kB/s)(1564KiB/15359msec); 0 zone resets clat (msec): min=5, max=102, avg=25.71, stdev=13.86 lat (msec): min=5, max=102, avg=25.71, stdev=13.86 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 26], | 70.00th=[ 29], 80.00th=[ 34], 90.00th=[ 42], 95.00th=[ 51], | 99.00th=[ 77], 99.50th=[ 99], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 47, max= 175, per=24.82%, avg=100.77, stdev=40.28, samples=30 iops : min= 11, max= 43, avg=24.63, stdev=10.08, samples=30 lat (msec) : 4=2.47%, 10=26.95%, 20=30.34%, 50=36.59%, 100=3.52% lat (msec) : 250=0.13% cpu : usr=0.11%, sys=1.25%, ctx=824, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=386KiB/s (396kB/s), 92.7KiB/s-105KiB/s (94.9kB/s-107kB/s), io=5988KiB (6132kB), run=14930-15497msec WRITE: bw=407KiB/s (416kB/s), 101KiB/s-107KiB/s (103kB/s-109kB/s), io=6300KiB (6451kB), run=14930-15497msec