== sanity test 398b: DIO and buffer IO race ============== 18:23:46 (1715207026) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.103295 s, 122 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20847: Wed May 8 18:23:49 2024 read: IOPS=184, BW=737KiB/s (755kB/s)(1480KiB/2007msec) clat (usec): min=3, max=31286, avg=1677.68, stdev=3896.94 lat (usec): min=3, max=31286, avg=1678.52, stdev=3896.83 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 13], 40.00th=[ 19], 50.00th=[ 34], 60.00th=[ 1156], | 70.00th=[ 1893], 80.00th=[ 2573], 90.00th=[ 3294], 95.00th=[ 5014], | 99.00th=[25822], 99.50th=[28705], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 672, max= 840, per=24.71%, avg=736.00, stdev=74.19, samples=4 iops : min= 168, max= 210, avg=184.00, stdev=18.55, samples=4 write: IOPS=198, BW=793KiB/s (812kB/s)(1592KiB/2007msec) clat (usec): min=92, max=39644, avg=3460.93, stdev=5679.32 lat (usec): min=93, max=39646, avg=3461.64, stdev=5679.45 clat percentiles (usec): | 1.00th=[ 99], 5.00th=[ 110], 10.00th=[ 121], 20.00th=[ 139], | 30.00th=[ 176], 40.00th=[ 474], 50.00th=[ 1020], 60.00th=[ 1582], | 70.00th=[ 2343], 80.00th=[ 5080], 90.00th=[11076], 95.00th=[15401], | 99.00th=[23987], 99.50th=[27132], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 728, max= 872, per=25.34%, avg=794.00, stdev=63.46, samples=4 iops : min= 182, max= 218, avg=198.50, stdev=15.86, samples=4 lat (usec) : 4=0.39%, 10=10.29%, 20=9.24%, 50=4.69%, 100=2.34% lat (usec) : 250=19.53%, 500=0.65%, 750=1.30%, 1000=4.56% lat (msec) : 2=14.97%, 4=17.06%, 10=7.03%, 20=5.47%, 50=2.47% cpu : usr=0.35%, sys=7.73%, ctx=839, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20848: Wed May 8 18:23:49 2024 read: IOPS=182, BW=731KiB/s (749kB/s)(1436KiB/1964msec) clat (usec): min=3, max=29919, avg=1638.46, stdev=3433.45 lat (usec): min=3, max=29919, avg=1639.16, stdev=3433.51 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 12], 40.00th=[ 20], 50.00th=[ 40], 60.00th=[ 1336], | 70.00th=[ 1844], 80.00th=[ 2409], 90.00th=[ 3195], 95.00th=[ 5866], | 99.00th=[20317], 99.50th=[26870], 99.90th=[30016], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 720, max= 832, per=25.60%, avg=762.67, stdev=60.58, samples=3 iops : min= 180, max= 208, avg=190.67, stdev=15.14, samples=3 write: IOPS=208, BW=833KiB/s (853kB/s)(1636KiB/1964msec) clat (usec): min=94, max=32060, avg=3334.24, stdev=5336.75 lat (usec): min=95, max=32060, avg=3335.18, stdev=5336.62 clat percentiles (usec): | 1.00th=[ 99], 5.00th=[ 110], 10.00th=[ 118], 20.00th=[ 143], | 30.00th=[ 182], 40.00th=[ 330], 50.00th=[ 963], 60.00th=[ 1598], | 70.00th=[ 2376], 80.00th=[ 5473], 90.00th=[11338], 95.00th=[15401], | 99.00th=[22152], 99.50th=[23200], 99.90th=[32113], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 656, max= 952, per=26.04%, avg=816.00, stdev=149.45, samples=3 iops : min= 164, max= 238, avg=204.00, stdev=37.36, samples=3 lat (usec) : 4=0.13%, 10=10.94%, 20=8.20%, 50=4.30%, 100=1.04% lat (usec) : 250=19.27%, 500=2.08%, 750=1.04%, 1000=5.99% lat (msec) : 2=16.54%, 4=15.62%, 10=7.16%, 20=5.73%, 50=1.95% cpu : usr=0.15%, sys=8.76%, ctx=875, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20849: Wed May 8 18:23:49 2024 read: IOPS=203, BW=813KiB/s (833kB/s)(1564KiB/1923msec) clat (usec): min=3, max=26123, avg=1549.48, stdev=3182.57 lat (usec): min=4, max=26123, avg=1550.17, stdev=3182.60 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 13], 40.00th=[ 19], 50.00th=[ 96], 60.00th=[ 1270], | 70.00th=[ 1860], 80.00th=[ 2376], 90.00th=[ 3130], 95.00th=[ 4621], | 99.00th=[17695], 99.50th=[25297], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 704, max= 944, per=26.94%, avg=802.67, stdev=125.56, samples=3 iops : min= 176, max= 236, avg=200.67, stdev=31.39, samples=3 write: IOPS=196, BW=784KiB/s (803kB/s)(1508KiB/1923msec) clat (usec): min=94, max=30134, avg=3466.04, stdev=5324.39 lat (usec): min=94, max=30135, avg=3467.09, stdev=5324.64 clat percentiles (usec): | 1.00th=[ 98], 5.00th=[ 110], 10.00th=[ 121], 20.00th=[ 151], | 30.00th=[ 188], 40.00th=[ 347], 50.00th=[ 955], 60.00th=[ 1614], | 70.00th=[ 2737], 80.00th=[ 6521], 90.00th=[12256], 95.00th=[14746], | 99.00th=[22938], 99.50th=[24511], 99.90th=[30016], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 720, max= 904, per=25.70%, avg=805.33, stdev=92.72, samples=3 iops : min= 180, max= 226, avg=201.33, stdev=23.18, samples=3 lat (usec) : 4=0.13%, 10=11.07%, 20=10.16%, 50=3.65%, 100=1.56% lat (usec) : 250=17.58%, 500=2.47%, 750=1.43%, 1000=5.08% lat (msec) : 2=15.10%, 4=17.19%, 10=6.12%, 20=7.03%, 50=1.43% cpu : usr=0.16%, sys=9.52%, ctx=883, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20850: Wed May 8 18:23:49 2024 read: IOPS=187, BW=750KiB/s (768kB/s)(1508KiB/2010msec) clat (usec): min=3, max=21213, avg=1637.65, stdev=2891.81 lat (usec): min=3, max=21214, avg=1638.23, stdev=2891.82 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 13], 40.00th=[ 20], 50.00th=[ 149], 60.00th=[ 1450], | 70.00th=[ 1844], 80.00th=[ 2507], 90.00th=[ 3654], 95.00th=[ 6652], | 99.00th=[15926], 99.50th=[18744], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 448, max= 912, per=24.84%, avg=740.00, stdev=208.36, samples=4 iops : min= 112, max= 228, avg=185.00, stdev=52.09, samples=4 write: IOPS=194, BW=778KiB/s (797kB/s)(1564KiB/2010msec) clat (usec): min=88, max=26681, avg=3538.66, stdev=5575.63 lat (usec): min=88, max=26682, avg=3539.58, stdev=5575.69 clat percentiles (usec): | 1.00th=[ 97], 5.00th=[ 110], 10.00th=[ 121], 20.00th=[ 141], | 30.00th=[ 188], 40.00th=[ 758], 50.00th=[ 1106], 60.00th=[ 1582], | 70.00th=[ 2540], 80.00th=[ 6128], 90.00th=[11600], 95.00th=[15664], | 99.00th=[24773], 99.50th=[26608], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 616, max= 880, per=24.63%, avg=772.00, stdev=112.48, samples=4 iops : min= 154, max= 220, avg=193.00, stdev=28.12, samples=4 lat (usec) : 4=0.26%, 10=10.68%, 20=8.85%, 50=3.52%, 100=1.82% lat (usec) : 250=17.71%, 500=1.43%, 750=1.04%, 1000=4.43% lat (msec) : 2=19.40%, 4=14.84%, 10=7.81%, 20=6.38%, 50=1.82% cpu : usr=0.15%, sys=8.51%, ctx=922, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2979KiB/s (3051kB/s), 731KiB/s-813KiB/s (749kB/s-833kB/s), io=5988KiB (6132kB), run=1923-2010msec WRITE: bw=3134KiB/s (3210kB/s), 778KiB/s-833KiB/s (797kB/s-853kB/s), io=6300KiB (6451kB), run=1923-2010msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20852: Wed May 8 18:23:56 2024 read: IOPS=43, BW=174KiB/s (179kB/s)(1480KiB/8487msec) clat (usec): min=657, max=69190, avg=2214.26, stdev=4735.38 lat (usec): min=657, max=69191, avg=2214.97, stdev=4735.39 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 758], 10.00th=[ 791], 20.00th=[ 840], | 30.00th=[ 881], 40.00th=[ 947], 50.00th=[ 1012], 60.00th=[ 1123], | 70.00th=[ 1336], 80.00th=[ 1729], 90.00th=[ 3064], 95.00th=[10814], | 99.00th=[19530], 99.50th=[20841], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 104, max= 304, per=24.78%, avg=172.00, stdev=58.35, samples=16 iops : min= 26, max= 76, avg=43.00, stdev=14.59, samples=16 write: IOPS=46, BW=188KiB/s (192kB/s)(1592KiB/8487msec) clat (usec): min=10882, max=52153, avg=19237.92, stdev=5828.15 lat (usec): min=10883, max=52154, avg=19238.79, stdev=5828.15 clat percentiles (usec): | 1.00th=[11207], 5.00th=[11731], 10.00th=[12518], 20.00th=[13698], | 30.00th=[15008], 40.00th=[17433], 50.00th=[19268], 60.00th=[20579], | 70.00th=[21890], 80.00th=[23200], 90.00th=[25560], 95.00th=[27657], | 99.00th=[40109], 99.50th=[51643], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 120, max= 224, per=25.44%, avg=186.00, stdev=31.26, samples=16 iops : min= 30, max= 56, avg=46.50, stdev= 7.81, samples=16 lat (usec) : 750=2.34%, 1000=21.22% lat (msec) : 2=16.93%, 4=3.52%, 10=1.43%, 20=30.21%, 50=23.96% lat (msec) : 100=0.39% cpu : usr=0.11%, sys=1.36%, ctx=823, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20853: Wed May 8 18:23:56 2024 read: IOPS=41, BW=167KiB/s (171kB/s)(1436KiB/8617msec) clat (usec): min=591, max=26774, avg=1980.50, stdev=3163.00 lat (usec): min=591, max=26775, avg=1981.23, stdev=3163.00 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 766], 10.00th=[ 807], 20.00th=[ 865], | 30.00th=[ 938], 40.00th=[ 1012], 50.00th=[ 1090], 60.00th=[ 1237], | 70.00th=[ 1352], 80.00th=[ 1680], 90.00th=[ 2606], 95.00th=[ 8717], | 99.00th=[20841], 99.50th=[23462], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 72, max= 256, per=23.86%, avg=165.59, stdev=50.28, samples=17 iops : min= 18, max= 64, avg=41.35, stdev=12.58, samples=17 write: IOPS=47, BW=190KiB/s (194kB/s)(1636KiB/8617msec) clat (usec): min=10099, max=49582, avg=19306.70, stdev=5703.41 lat (usec): min=10100, max=49583, avg=19307.63, stdev=5703.45 clat percentiles (usec): | 1.00th=[10814], 5.00th=[11863], 10.00th=[12649], 20.00th=[13829], | 30.00th=[15926], 40.00th=[17957], 50.00th=[19006], 60.00th=[20317], | 70.00th=[21890], 80.00th=[23200], 90.00th=[25035], 95.00th=[28967], | 99.00th=[37487], 99.50th=[40109], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 128, max= 216, per=25.67%, avg=187.65, stdev=26.19, samples=17 iops : min= 32, max= 54, avg=46.88, stdev= 6.54, samples=17 lat (usec) : 750=1.69%, 1000=16.93% lat (msec) : 2=20.83%, 4=4.04%, 10=1.04%, 20=32.03%, 50=23.44% cpu : usr=0.09%, sys=1.39%, ctx=814, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20854: Wed May 8 18:23:56 2024 read: IOPS=47, BW=192KiB/s (197kB/s)(1564KiB/8146msec) clat (usec): min=600, max=20193, avg=1772.50, stdev=2764.41 lat (usec): min=600, max=20193, avg=1773.24, stdev=2764.42 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 848], | 30.00th=[ 906], 40.00th=[ 979], 50.00th=[ 1090], 60.00th=[ 1172], | 70.00th=[ 1303], 80.00th=[ 1565], 90.00th=[ 2114], 95.00th=[ 5014], | 99.00th=[18220], 99.50th=[19792], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 96, max= 344, per=28.03%, avg=194.50, stdev=74.40, samples=16 iops : min= 24, max= 86, avg=48.62, stdev=18.60, samples=16 write: IOPS=46, BW=185KiB/s (190kB/s)(1508KiB/8146msec) clat (usec): min=11263, max=59380, avg=19744.97, stdev=6153.16 lat (usec): min=11264, max=59380, avg=19746.03, stdev=6153.12 clat percentiles (usec): | 1.00th=[11338], 5.00th=[12256], 10.00th=[12780], 20.00th=[14091], | 30.00th=[15664], 40.00th=[17957], 50.00th=[19792], 60.00th=[20841], | 70.00th=[21890], 80.00th=[23462], 90.00th=[25822], 95.00th=[30540], | 99.00th=[41157], 99.50th=[49021], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 128, max= 216, per=25.31%, avg=185.00, stdev=28.15, samples=16 iops : min= 32, max= 54, avg=46.25, stdev= 7.04, samples=16 lat (usec) : 750=3.39%, 1000=18.10% lat (msec) : 2=23.96%, 4=2.73%, 10=0.91%, 20=26.95%, 50=23.83% lat (msec) : 100=0.13% cpu : usr=0.15%, sys=1.38%, ctx=844, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20855: Wed May 8 18:23:56 2024 read: IOPS=44, BW=178KiB/s (183kB/s)(1508KiB/8456msec) clat (usec): min=632, max=65439, avg=2359.62, stdev=5063.79 lat (usec): min=633, max=65440, avg=2360.39, stdev=5063.78 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 750], 10.00th=[ 799], 20.00th=[ 848], | 30.00th=[ 930], 40.00th=[ 996], 50.00th=[ 1090], 60.00th=[ 1270], | 70.00th=[ 1450], 80.00th=[ 1713], 90.00th=[ 3064], 95.00th=[11600], | 99.00th=[23200], 99.50th=[35914], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 80, max= 280, per=24.85%, avg=172.44, stdev=56.24, samples=16 iops : min= 20, max= 70, avg=43.06, stdev=14.05, samples=16 write: IOPS=46, BW=185KiB/s (189kB/s)(1564KiB/8456msec) clat (usec): min=10141, max=60961, avg=19330.57, stdev=6129.21 lat (usec): min=10142, max=60961, avg=19331.46, stdev=6129.24 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11863], 10.00th=[12649], 20.00th=[13566], | 30.00th=[15270], 40.00th=[17695], 50.00th=[19268], 60.00th=[20317], | 70.00th=[21627], 80.00th=[23200], 90.00th=[25560], 95.00th=[28181], | 99.00th=[42206], 99.50th=[56361], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 112, max= 216, per=24.82%, avg=181.44, stdev=34.97, samples=16 iops : min= 28, max= 54, avg=45.31, stdev= 8.71, samples=16 lat (usec) : 750=2.47%, 1000=17.45% lat (msec) : 2=21.61%, 4=3.52%, 10=1.43%, 20=30.34%, 50=22.79% lat (msec) : 100=0.39% cpu : usr=0.14%, sys=1.38%, ctx=838, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=695KiB/s (712kB/s), 167KiB/s-192KiB/s (171kB/s-197kB/s), io=5988KiB (6132kB), run=8146-8617msec WRITE: bw=731KiB/s (749kB/s), 185KiB/s-190KiB/s (189kB/s-194kB/s), io=6300KiB (6451kB), run=8146-8617msec