== sanity test 398b: DIO and buffer IO race ============== 11:03:31 (1736265811) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.0746211 s, 169 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=903: Tue Jan 7 11:03:34 2025 read: IOPS=185, BW=742KiB/s (760kB/s)(1480KiB/1995msec) clat (usec): min=4, max=21588, avg=1524.12, stdev=2658.40 lat (usec): min=4, max=21589, avg=1524.80, stdev=2658.39 clat percentiles (usec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 13], | 30.00th=[ 18], 40.00th=[ 27], 50.00th=[ 857], 60.00th=[ 1565], | 70.00th=[ 2114], 80.00th=[ 2638], 90.00th=[ 3294], 95.00th=[ 4146], | 99.00th=[16450], 99.50th=[20841], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 680, max= 840, per=25.50%, avg=749.33, stdev=82.11, samples=3 iops : min= 170, max= 210, avg=187.33, stdev=20.53, samples=3 write: IOPS=199, BW=798KiB/s (817kB/s)(1592KiB/1995msec) clat (usec): min=97, max=45745, avg=3555.69, stdev=5727.42 lat (usec): min=97, max=45745, avg=3556.53, stdev=5727.54 clat percentiles (usec): | 1.00th=[ 108], 5.00th=[ 131], 10.00th=[ 161], 20.00th=[ 184], | 30.00th=[ 285], 40.00th=[ 947], 50.00th=[ 1385], 60.00th=[ 2089], | 70.00th=[ 3097], 80.00th=[ 5932], 90.00th=[ 9634], 95.00th=[13173], | 99.00th=[30278], 99.50th=[40633], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 672, max= 872, per=25.71%, avg=794.67, stdev=107.43, samples=3 iops : min= 168, max= 218, avg=198.67, stdev=26.86, samples=3 lat (usec) : 10=5.08%, 20=11.20%, 50=6.51%, 100=0.52%, 250=15.23% lat (usec) : 500=2.21%, 750=1.04%, 1000=4.30% lat (msec) : 2=17.32%, 4=20.96%, 10=9.77%, 20=4.43%, 50=1.43% cpu : usr=0.40%, sys=8.83%, ctx=991, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=907: Tue Jan 7 11:03:34 2025 read: IOPS=176, BW=705KiB/s (722kB/s)(1436KiB/2038msec) clat (usec): min=3, max=32568, avg=1622.11, stdev=2983.70 lat (usec): min=4, max=32569, avg=1623.16, stdev=2983.59 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 13], | 30.00th=[ 19], 40.00th=[ 29], 50.00th=[ 693], 60.00th=[ 1647], | 70.00th=[ 2147], 80.00th=[ 2671], 90.00th=[ 3523], 95.00th=[ 4555], | 99.00th=[13435], 99.50th=[21103], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 440, max= 960, per=23.96%, avg=704.00, stdev=244.32, samples=4 iops : min= 110, max= 240, avg=176.00, stdev=61.08, samples=4 write: IOPS=200, BW=803KiB/s (822kB/s)(1636KiB/2038msec) clat (usec): min=94, max=42687, avg=3522.10, stdev=5293.96 lat (usec): min=95, max=42688, avg=3523.13, stdev=5293.80 clat percentiles (usec): | 1.00th=[ 109], 5.00th=[ 130], 10.00th=[ 155], 20.00th=[ 178], | 30.00th=[ 235], 40.00th=[ 938], 50.00th=[ 1418], 60.00th=[ 2212], | 70.00th=[ 3032], 80.00th=[ 5735], 90.00th=[10683], 95.00th=[13435], | 99.00th=[26084], 99.50th=[29754], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 488, max= 1000, per=25.88%, avg=800.00, stdev=245.36, samples=4 iops : min= 122, max= 250, avg=200.00, stdev=61.34, samples=4 lat (usec) : 4=0.13%, 10=5.47%, 20=8.98%, 50=7.42%, 100=0.65% lat (usec) : 250=16.93%, 500=1.82%, 750=1.30%, 1000=3.78% lat (msec) : 2=15.49%, 4=20.57%, 10=10.16%, 20=6.12%, 50=1.17% cpu : usr=0.29%, sys=9.43%, ctx=950, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=909: Tue Jan 7 11:03:34 2025 read: IOPS=192, BW=769KiB/s (787kB/s)(1564KiB/2035msec) clat (usec): min=3, max=20474, avg=1729.18, stdev=2846.03 lat (usec): min=4, max=20474, avg=1730.17, stdev=2846.00 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 12], | 30.00th=[ 17], 40.00th=[ 28], 50.00th=[ 1156], 60.00th=[ 1795], | 70.00th=[ 2114], 80.00th=[ 2573], 90.00th=[ 3392], 95.00th=[ 5473], | 99.00th=[17171], 99.50th=[19268], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 576, max= 960, per=26.21%, avg=770.00, stdev=156.82, samples=4 iops : min= 144, max= 240, avg=192.50, stdev=39.20, samples=4 write: IOPS=185, BW=741KiB/s (759kB/s)(1508KiB/2035msec) clat (usec): min=103, max=28139, avg=3579.65, stdev=4714.32 lat (usec): min=104, max=28140, avg=3580.55, stdev=4714.51 clat percentiles (usec): | 1.00th=[ 113], 5.00th=[ 139], 10.00th=[ 157], 20.00th=[ 184], | 30.00th=[ 310], 40.00th=[ 1029], 50.00th=[ 1647], 60.00th=[ 2311], | 70.00th=[ 3654], 80.00th=[ 7046], 90.00th=[10028], 95.00th=[12256], | 99.00th=[22414], 99.50th=[27395], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 672, max= 832, per=23.62%, avg=730.00, stdev=73.72, samples=4 iops : min= 168, max= 208, avg=182.50, stdev=18.43, samples=4 lat (usec) : 4=0.13%, 10=6.25%, 20=11.07%, 50=4.95%, 100=0.26% lat (usec) : 250=14.06%, 500=1.82%, 750=0.65%, 1000=4.43% lat (msec) : 2=17.19%, 4=20.96%, 10=11.46%, 20=5.99%, 50=0.78% cpu : usr=0.44%, sys=9.73%, ctx=1025, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=911: Tue Jan 7 11:03:34 2025 read: IOPS=199, BW=798KiB/s (817kB/s)(1508KiB/1889msec) clat (usec): min=3, max=19716, avg=1546.46, stdev=2787.58 lat (usec): min=4, max=19716, avg=1547.33, stdev=2787.47 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 13], | 30.00th=[ 18], 40.00th=[ 25], 50.00th=[ 135], 60.00th=[ 1565], | 70.00th=[ 2024], 80.00th=[ 2474], 90.00th=[ 3163], 95.00th=[ 4293], | 99.00th=[18220], 99.50th=[19006], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 640, max= 832, per=24.51%, avg=720.00, stdev=99.92, samples=3 iops : min= 160, max= 208, avg=180.00, stdev=24.98, samples=3 write: IOPS=206, BW=828KiB/s (848kB/s)(1564KiB/1889msec) clat (usec): min=98, max=42482, avg=3316.34, stdev=4945.20 lat (usec): min=99, max=42483, avg=3317.44, stdev=4944.98 clat percentiles (usec): | 1.00th=[ 106], 5.00th=[ 133], 10.00th=[ 155], 20.00th=[ 176], | 30.00th=[ 210], 40.00th=[ 898], 50.00th=[ 1418], 60.00th=[ 2024], | 70.00th=[ 2999], 80.00th=[ 5342], 90.00th=[10290], 95.00th=[14091], | 99.00th=[19006], 99.50th=[31327], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 712, max= 880, per=25.71%, avg=794.67, stdev=84.03, samples=3 iops : min= 178, max= 220, avg=198.67, stdev=21.01, samples=3 lat (usec) : 4=0.26%, 10=5.34%, 20=11.07%, 50=7.03%, 100=0.65% lat (usec) : 250=16.67%, 500=1.30%, 750=0.78%, 1000=4.17% lat (msec) : 2=17.45%, 4=19.92%, 10=8.72%, 20=6.38%, 50=0.26% cpu : usr=0.26%, sys=9.22%, ctx=940, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2938KiB/s (3009kB/s), 705KiB/s-798KiB/s (722kB/s-817kB/s), io=5988KiB (6132kB), run=1889-2038msec WRITE: bw=3091KiB/s (3165kB/s), 741KiB/s-828KiB/s (759kB/s-848kB/s), io=6300KiB (6451kB), run=1889-2038msec rand-rw: (groupid=0, jobs=1): err= 0: pid=905: Tue Jan 7 11:03:40 2025 read: IOPS=49, BW=199KiB/s (204kB/s)(1480KiB/7447msec) clat (usec): min=705, max=21460, avg=1978.29, stdev=2466.20 lat (usec): min=705, max=21460, avg=1979.35, stdev=2466.27 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 840], 10.00th=[ 881], 20.00th=[ 938], | 30.00th=[ 1020], 40.00th=[ 1123], 50.00th=[ 1237], 60.00th=[ 1385], | 70.00th=[ 1549], 80.00th=[ 1860], 90.00th=[ 3228], 95.00th=[ 7767], | 99.00th=[14746], 99.50th=[15401], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 127, max= 320, per=25.26%, avg=199.29, stdev=56.84, samples=14 iops : min= 31, max= 80, avg=49.71, stdev=14.19, samples=14 write: IOPS=53, BW=214KiB/s (219kB/s)(1592KiB/7447msec) clat (usec): min=8025, max=59154, avg=16835.03, stdev=6151.65 lat (usec): min=8026, max=59155, avg=16836.33, stdev=6151.75 clat percentiles (usec): | 1.00th=[ 8717], 5.00th=[ 9765], 10.00th=[10159], 20.00th=[11600], | 30.00th=[12780], 40.00th=[14615], 50.00th=[16909], 60.00th=[17957], | 70.00th=[19006], 80.00th=[20317], 90.00th=[22676], 95.00th=[26608], | 99.00th=[43779], 99.50th=[50594], 99.90th=[58983], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 168, max= 264, per=25.73%, avg=213.57, stdev=31.34, samples=14 iops : min= 42, max= 66, avg=53.29, stdev= 7.88, samples=14 lat (usec) : 750=0.13%, 1000=13.28% lat (msec) : 2=26.17%, 4=4.56%, 10=6.25%, 20=38.02%, 50=11.33% lat (msec) : 100=0.26% cpu : usr=0.27%, sys=1.67%, ctx=834, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=906: Tue Jan 7 11:03:40 2025 read: IOPS=47, BW=189KiB/s (194kB/s)(1436KiB/7584msec) clat (usec): min=703, max=33971, avg=2115.96, stdev=3160.00 lat (usec): min=704, max=33972, avg=2116.87, stdev=3160.01 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 816], 10.00th=[ 873], 20.00th=[ 938], | 30.00th=[ 996], 40.00th=[ 1106], 50.00th=[ 1188], 60.00th=[ 1287], | 70.00th=[ 1467], 80.00th=[ 1844], 90.00th=[ 3130], 95.00th=[10028], | 99.00th=[15401], 99.50th=[17957], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 80, max= 312, per=23.79%, avg=187.73, stdev=62.11, samples=15 iops : min= 20, max= 78, avg=46.93, stdev=15.53, samples=15 write: IOPS=53, BW=216KiB/s (221kB/s)(1636KiB/7584msec) clat (usec): min=8736, max=69704, avg=16653.38, stdev=5843.36 lat (usec): min=8737, max=69705, avg=16654.53, stdev=5843.37 clat percentiles (usec): | 1.00th=[ 8979], 5.00th=[ 9896], 10.00th=[10421], 20.00th=[11338], | 30.00th=[12780], 40.00th=[14877], 50.00th=[16450], 60.00th=[17695], | 70.00th=[19006], 80.00th=[20055], 90.00th=[22152], 95.00th=[26084], | 99.00th=[35914], 99.50th=[37487], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 144, max= 256, per=25.77%, avg=213.87, stdev=31.05, samples=15 iops : min= 36, max= 64, avg=53.47, stdev= 7.76, samples=15 lat (usec) : 750=0.52%, 1000=13.93% lat (msec) : 2=24.35%, 4=4.04%, 10=4.43%, 20=41.15%, 50=11.46% lat (msec) : 100=0.13% cpu : usr=0.18%, sys=1.74%, ctx=822, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=908: Tue Jan 7 11:03:40 2025 read: IOPS=53, BW=214KiB/s (219kB/s)(1564KiB/7312msec) clat (usec): min=701, max=23298, avg=2143.51, stdev=3041.37 lat (usec): min=702, max=23298, avg=2144.49, stdev=3041.39 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 840], 10.00th=[ 881], 20.00th=[ 955], | 30.00th=[ 1029], 40.00th=[ 1123], 50.00th=[ 1221], 60.00th=[ 1336], | 70.00th=[ 1549], 80.00th=[ 1795], 90.00th=[ 3752], 95.00th=[ 8848], | 99.00th=[18220], 99.50th=[21627], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 112, max= 336, per=27.59%, avg=217.71, stdev=73.67, samples=14 iops : min= 28, max= 84, avg=54.43, stdev=18.42, samples=14 write: IOPS=51, BW=206KiB/s (211kB/s)(1508KiB/7312msec) clat (usec): min=8647, max=66196, avg=17143.04, stdev=6424.32 lat (usec): min=8648, max=66197, avg=17144.13, stdev=6424.32 clat percentiles (usec): | 1.00th=[ 9241], 5.00th=[10028], 10.00th=[10683], 20.00th=[11731], | 30.00th=[13173], 40.00th=[15270], 50.00th=[16712], 60.00th=[17957], | 70.00th=[18744], 80.00th=[20055], 90.00th=[23200], 95.00th=[27657], | 99.00th=[44827], 99.50th=[46924], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 144, max= 256, per=24.85%, avg=206.29, stdev=33.53, samples=14 iops : min= 36, max= 64, avg=51.57, stdev= 8.38, samples=14 lat (usec) : 750=0.26%, 1000=13.54% lat (msec) : 2=28.65%, 4=3.39%, 10=5.08%, 20=38.41%, 50=10.55% lat (msec) : 100=0.13% cpu : usr=0.08%, sys=1.91%, ctx=841, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=910: Tue Jan 7 11:03:40 2025 read: IOPS=50, BW=202KiB/s (207kB/s)(1508KiB/7473msec) clat (usec): min=719, max=32525, avg=2432.14, stdev=3923.49 lat (usec): min=720, max=32526, avg=2433.08, stdev=3923.49 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 848], 10.00th=[ 889], 20.00th=[ 955], | 30.00th=[ 1037], 40.00th=[ 1123], 50.00th=[ 1221], 60.00th=[ 1336], | 70.00th=[ 1500], 80.00th=[ 1876], 90.00th=[ 4080], 95.00th=[11076], | 99.00th=[22414], 99.50th=[25560], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 112, max= 296, per=24.68%, avg=194.71, stdev=55.49, samples=14 iops : min= 28, max= 74, avg=48.57, stdev=13.95, samples=14 write: IOPS=52, BW=209KiB/s (214kB/s)(1564KiB/7473msec) clat (usec): min=8081, max=50321, avg=16721.13, stdev=5660.74 lat (usec): min=8082, max=50322, avg=16722.48, stdev=5660.62 clat percentiles (usec): | 1.00th=[ 9110], 5.00th=[10028], 10.00th=[10552], 20.00th=[11731], | 30.00th=[13304], 40.00th=[15008], 50.00th=[16188], 60.00th=[17695], | 70.00th=[18744], 80.00th=[20055], 90.00th=[22152], 95.00th=[26346], | 99.00th=[39060], 99.50th=[48497], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 136, max= 264, per=24.70%, avg=205.00, stdev=38.67, samples=14 iops : min= 34, max= 66, avg=51.14, stdev= 9.70, samples=14 lat (usec) : 750=0.13%, 1000=13.02% lat (msec) : 2=27.08%, 4=3.91%, 10=4.56%, 20=40.23%, 50=10.94% lat (msec) : 100=0.13% cpu : usr=0.23%, sys=1.73%, ctx=844, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=790KiB/s (809kB/s), 189KiB/s-214KiB/s (194kB/s-219kB/s), io=5988KiB (6132kB), run=7312-7584msec WRITE: bw=831KiB/s (851kB/s), 206KiB/s-216KiB/s (211kB/s-221kB/s), io=6300KiB (6451kB), run=7312-7584msec