== sanity test 398b: DIO and buffer IO race ============== 13:06:50 (1736273210) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.0681049 s, 185 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24102: Tue Jan 7 13:06:53 2025 read: IOPS=203, BW=814KiB/s (834kB/s)(1480KiB/1818msec) clat (usec): min=4, max=23268, avg=1292.47, stdev=2330.23 lat (usec): min=4, max=23268, avg=1293.07, stdev=2330.23 clat percentiles (usec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 14], 40.00th=[ 21], 50.00th=[ 46], 60.00th=[ 1205], | 70.00th=[ 1942], 80.00th=[ 2311], 90.00th=[ 2835], 95.00th=[ 3556], | 99.00th=[11994], 99.50th=[17957], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 680, max= 896, per=24.48%, avg=802.67, stdev=110.95, samples=3 iops : min= 170, max= 224, avg=200.67, stdev=27.74, samples=3 write: IOPS=218, BW=876KiB/s (897kB/s)(1592KiB/1818msec) clat (usec): min=80, max=23642, avg=3350.37, stdev=4313.12 lat (usec): min=81, max=23642, avg=3351.05, stdev=4313.11 clat percentiles (usec): | 1.00th=[ 96], 5.00th=[ 119], 10.00th=[ 133], 20.00th=[ 180], | 30.00th=[ 297], 40.00th=[ 947], 50.00th=[ 1401], 60.00th=[ 2147], | 70.00th=[ 3556], 80.00th=[ 6980], 90.00th=[10028], 95.00th=[12518], | 99.00th=[17433], 99.50th=[19792], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 688, max= 984, per=25.43%, avg=877.33, stdev=164.41, samples=3 iops : min= 172, max= 246, avg=219.33, stdev=41.10, samples=3 lat (usec) : 10=8.20%, 20=10.81%, 50=5.47%, 100=0.91%, 250=14.71% lat (usec) : 500=1.56%, 750=2.34%, 1000=4.82% lat (msec) : 2=16.02%, 4=19.14%, 10=10.03%, 20=5.73%, 50=0.26% cpu : usr=0.22%, sys=9.19%, ctx=968, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24103: Tue Jan 7 13:06:53 2025 read: IOPS=214, BW=857KiB/s (877kB/s)(1436KiB/1676msec) clat (usec): min=3, max=18997, avg=1102.67, stdev=1754.55 lat (usec): min=3, max=18998, avg=1103.59, stdev=1754.36 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 14], 40.00th=[ 22], 50.00th=[ 61], 60.00th=[ 1205], | 70.00th=[ 1647], 80.00th=[ 2073], 90.00th=[ 2769], 95.00th=[ 3163], | 99.00th=[ 7701], 99.50th=[13566], 99.90th=[19006], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 784, max= 1000, per=26.76%, avg=877.33, stdev=110.95, samples=3 iops : min= 196, max= 250, avg=219.33, stdev=27.74, samples=3 write: IOPS=244, BW=976KiB/s (1000kB/s)(1636KiB/1676msec) clat (usec): min=88, max=31319, avg=3113.04, stdev=4423.63 lat (usec): min=89, max=31320, avg=3113.75, stdev=4423.62 clat percentiles (usec): | 1.00th=[ 99], 5.00th=[ 113], 10.00th=[ 125], 20.00th=[ 163], | 30.00th=[ 190], 40.00th=[ 750], 50.00th=[ 1090], 60.00th=[ 1778], | 70.00th=[ 2999], 80.00th=[ 6259], 90.00th=[ 9372], 95.00th=[12125], | 99.00th=[17695], 99.50th=[22676], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 888, max= 1104, per=28.37%, avg=978.67, stdev=112.10, samples=3 iops : min= 222, max= 276, avg=244.67, stdev=28.02, samples=3 lat (usec) : 4=0.39%, 10=7.03%, 20=10.42%, 50=5.34%, 100=1.69% lat (usec) : 250=17.71%, 500=1.43%, 750=1.82%, 1000=5.73% lat (msec) : 2=18.36%, 4=15.62%, 10=9.37%, 20=4.69%, 50=0.39% cpu : usr=0.36%, sys=9.25%, ctx=952, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24104: Tue Jan 7 13:06:53 2025 read: IOPS=214, BW=857KiB/s (877kB/s)(1564KiB/1826msec) clat (usec): min=3, max=21358, avg=1407.85, stdev=2872.60 lat (usec): min=4, max=21358, avg=1408.81, stdev=2872.43 clat percentiles (usec): | 1.00th=[ 4], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 13], 40.00th=[ 18], 50.00th=[ 47], 60.00th=[ 1254], | 70.00th=[ 1844], 80.00th=[ 2311], 90.00th=[ 2900], 95.00th=[ 3752], | 99.00th=[19530], 99.50th=[21103], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 544, max= 920, per=24.15%, avg=792.00, stdev=214.81, samples=3 iops : min= 136, max= 230, avg=198.00, stdev=53.70, samples=3 write: IOPS=206, BW=826KiB/s (846kB/s)(1508KiB/1826msec) clat (usec): min=96, max=38892, avg=3361.99, stdev=4872.44 lat (usec): min=96, max=38893, avg=3363.19, stdev=4872.30 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 115], 10.00th=[ 133], 20.00th=[ 169], | 30.00th=[ 221], 40.00th=[ 824], 50.00th=[ 1205], 60.00th=[ 1975], | 70.00th=[ 3326], 80.00th=[ 6718], 90.00th=[ 9634], 95.00th=[13173], | 99.00th=[19792], 99.50th=[28967], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 640, max= 912, per=23.11%, avg=797.33, stdev=140.93, samples=3 iops : min= 160, max= 228, avg=199.33, stdev=35.23, samples=3 lat (usec) : 4=0.65%, 10=8.46%, 20=12.24%, 50=4.17%, 100=1.17% lat (usec) : 250=16.41%, 500=0.91%, 750=1.82%, 1000=5.60% lat (msec) : 2=15.76%, 4=17.58%, 10=9.37%, 20=5.21%, 50=0.65% cpu : usr=0.16%, sys=8.93%, ctx=925, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24105: Tue Jan 7 13:06:53 2025 read: IOPS=231, BW=926KiB/s (948kB/s)(1508KiB/1629msec) clat (usec): min=3, max=27485, avg=1335.03, stdev=2639.03 lat (usec): min=3, max=27486, avg=1335.91, stdev=2639.00 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 11], | 30.00th=[ 15], 40.00th=[ 24], 50.00th=[ 143], 60.00th=[ 1352], | 70.00th=[ 1745], 80.00th=[ 2311], 90.00th=[ 2966], 95.00th=[ 3523], | 99.00th=[18220], 99.50th=[22676], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 840, max= 928, per=27.33%, avg=896.00, stdev=48.66, samples=3 iops : min= 210, max= 232, avg=224.00, stdev=12.17, samples=3 write: IOPS=240, BW=960KiB/s (983kB/s)(1564KiB/1629msec) clat (usec): min=98, max=19150, avg=2860.69, stdev=3998.60 lat (usec): min=98, max=19151, avg=2861.38, stdev=3998.60 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 112], 10.00th=[ 139], 20.00th=[ 167], | 30.00th=[ 194], 40.00th=[ 709], 50.00th=[ 988], 60.00th=[ 1467], | 70.00th=[ 2507], 80.00th=[ 5669], 90.00th=[ 9110], 95.00th=[11731], | 99.00th=[16712], 99.50th=[18744], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 880, max= 1016, per=27.44%, avg=946.67, stdev=68.04, samples=3 iops : min= 220, max= 254, avg=236.67, stdev=17.01, samples=3 lat (usec) : 4=0.13%, 10=8.07%, 20=9.64%, 50=5.86%, 100=0.91% lat (usec) : 250=17.71%, 500=1.17%, 750=2.47%, 1000=5.60% lat (msec) : 2=19.27%, 4=15.76%, 10=8.20%, 20=4.82%, 50=0.39% cpu : usr=0.37%, sys=11.18%, ctx=918, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3279KiB/s (3358kB/s), 814KiB/s-926KiB/s (834kB/s-948kB/s), io=5988KiB (6132kB), run=1629-1826msec WRITE: bw=3450KiB/s (3533kB/s), 826KiB/s-976KiB/s (846kB/s-1000kB/s), io=6300KiB (6451kB), run=1629-1826msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24097: Tue Jan 7 13:06:58 2025 read: IOPS=56, BW=225KiB/s (230kB/s)(1480KiB/6579msec) clat (usec): min=616, max=31948, avg=1768.85, stdev=2684.09 lat (usec): min=617, max=31949, avg=1769.64, stdev=2684.09 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 750], 10.00th=[ 783], 20.00th=[ 848], | 30.00th=[ 898], 40.00th=[ 955], 50.00th=[ 1057], 60.00th=[ 1172], | 70.00th=[ 1319], 80.00th=[ 1614], 90.00th=[ 2376], 95.00th=[ 6259], | 99.00th=[14484], 99.50th=[17171], 99.90th=[31851], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 136, max= 368, per=25.07%, avg=222.15, stdev=62.67, samples=13 iops : min= 34, max= 92, avg=55.54, stdev=15.84, samples=13 write: IOPS=60, BW=242KiB/s (248kB/s)(1592KiB/6579msec) clat (usec): min=7100, max=40756, avg=14853.55, stdev=4953.58 lat (usec): min=7100, max=40757, avg=14854.50, stdev=4953.60 clat percentiles (usec): | 1.00th=[ 7635], 5.00th=[ 8356], 10.00th=[ 8848], 20.00th=[10159], | 30.00th=[11600], 40.00th=[13566], 50.00th=[14877], 60.00th=[15795], | 70.00th=[16909], 80.00th=[18220], 90.00th=[20317], 95.00th=[22938], | 99.00th=[32375], 99.50th=[39584], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 184, max= 288, per=25.82%, avg=240.62, stdev=37.16, samples=13 iops : min= 46, max= 72, avg=60.15, stdev= 9.29, samples=13 lat (usec) : 750=2.34%, 1000=18.88% lat (msec) : 2=20.83%, 4=3.12%, 10=11.85%, 20=36.33%, 50=6.64% cpu : usr=0.06%, sys=1.92%, ctx=832, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24098: Tue Jan 7 13:06:58 2025 read: IOPS=53, BW=213KiB/s (218kB/s)(1436KiB/6755msec) clat (usec): min=658, max=24879, avg=1848.24, stdev=2676.73 lat (usec): min=659, max=24880, avg=1849.06, stdev=2676.72 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 758], 10.00th=[ 791], 20.00th=[ 857], | 30.00th=[ 906], 40.00th=[ 996], 50.00th=[ 1074], 60.00th=[ 1172], | 70.00th=[ 1369], 80.00th=[ 1680], 90.00th=[ 2835], 95.00th=[ 7111], | 99.00th=[13566], 99.50th=[22938], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 112, max= 352, per=23.96%, avg=212.31, stdev=59.47, samples=13 iops : min= 28, max= 88, avg=53.08, stdev=14.87, samples=13 write: IOPS=60, BW=242KiB/s (248kB/s)(1636KiB/6755msec) clat (usec): min=7287, max=39102, avg=14867.49, stdev=5107.28 lat (usec): min=7288, max=39103, avg=14868.50, stdev=5107.28 clat percentiles (usec): | 1.00th=[ 7635], 5.00th=[ 8586], 10.00th=[ 9110], 20.00th=[10159], | 30.00th=[11338], 40.00th=[13304], 50.00th=[14615], 60.00th=[15795], | 70.00th=[16712], 80.00th=[17957], 90.00th=[20317], 95.00th=[24249], | 99.00th=[34341], 99.50th=[35390], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 176, max= 280, per=25.62%, avg=238.77, stdev=35.45, samples=13 iops : min= 44, max= 70, avg=59.69, stdev= 8.86, samples=13 lat (usec) : 750=2.08%, 1000=17.06% lat (msec) : 2=20.57%, 4=3.39%, 10=12.50%, 20=38.54%, 50=5.86% cpu : usr=0.10%, sys=1.85%, ctx=841, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24099: Tue Jan 7 13:06:58 2025 read: IOPS=61, BW=245KiB/s (251kB/s)(1564KiB/6379msec) clat (usec): min=637, max=24114, avg=1932.02, stdev=2819.46 lat (usec): min=638, max=24115, avg=1932.85, stdev=2819.47 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 750], 10.00th=[ 775], 20.00th=[ 840], | 30.00th=[ 898], 40.00th=[ 955], 50.00th=[ 1037], 60.00th=[ 1188], | 70.00th=[ 1319], 80.00th=[ 1647], 90.00th=[ 3425], 95.00th=[ 8029], | 99.00th=[16057], 99.50th=[20579], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 144, max= 440, per=28.22%, avg=250.00, stdev=83.57, samples=12 iops : min= 36, max= 110, avg=62.50, stdev=20.89, samples=12 write: IOPS=59, BW=236KiB/s (242kB/s)(1508KiB/6379msec) clat (usec): min=7401, max=40727, avg=14882.47, stdev=4962.35 lat (usec): min=7402, max=40728, avg=14883.47, stdev=4962.36 clat percentiles (usec): | 1.00th=[ 7701], 5.00th=[ 8291], 10.00th=[ 9110], 20.00th=[10421], | 30.00th=[11863], 40.00th=[13042], 50.00th=[14746], 60.00th=[15664], | 70.00th=[16909], 80.00th=[17957], 90.00th=[20055], 95.00th=[23462], | 99.00th=[32113], 99.50th=[39060], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 184, max= 296, per=25.04%, avg=233.33, stdev=36.40, samples=12 iops : min= 46, max= 74, avg=58.33, stdev= 9.10, samples=12 lat (usec) : 750=2.73%, 1000=19.27% lat (msec) : 2=21.22%, 4=3.12%, 10=11.46%, 20=36.46%, 50=5.73% cpu : usr=0.13%, sys=1.91%, ctx=838, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24100: Tue Jan 7 13:06:58 2025 read: IOPS=56, BW=227KiB/s (232kB/s)(1508KiB/6642msec) clat (usec): min=634, max=29769, avg=2056.54, stdev=3504.73 lat (usec): min=635, max=29770, avg=2057.33, stdev=3504.74 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 750], 10.00th=[ 791], 20.00th=[ 840], | 30.00th=[ 889], 40.00th=[ 971], 50.00th=[ 1074], 60.00th=[ 1221], | 70.00th=[ 1369], 80.00th=[ 1663], 90.00th=[ 2966], 95.00th=[ 8455], | 99.00th=[20317], 99.50th=[29492], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 152, max= 320, per=25.42%, avg=225.23, stdev=53.65, samples=13 iops : min= 38, max= 80, avg=56.31, stdev=13.41, samples=13 write: IOPS=58, BW=235KiB/s (241kB/s)(1564KiB/6642msec) clat (usec): min=7069, max=38339, avg=14969.41, stdev=5164.04 lat (usec): min=7070, max=38340, avg=14970.35, stdev=5164.05 clat percentiles (usec): | 1.00th=[ 7570], 5.00th=[ 8455], 10.00th=[ 9110], 20.00th=[10290], | 30.00th=[11731], 40.00th=[13829], 50.00th=[14746], 60.00th=[15795], | 70.00th=[16909], 80.00th=[17957], 90.00th=[20055], 95.00th=[23462], | 99.00th=[34866], 99.50th=[38011], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 168, max= 288, per=25.16%, avg=234.46, stdev=38.70, samples=13 iops : min= 42, max= 72, avg=58.62, stdev= 9.67, samples=13 lat (usec) : 750=2.08%, 1000=19.14% lat (msec) : 2=20.83%, 4=3.12%, 10=11.07%, 20=38.15%, 50=5.60% cpu : usr=0.14%, sys=1.82%, ctx=845, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=886KiB/s (908kB/s), 213KiB/s-245KiB/s (218kB/s-251kB/s), io=5988KiB (6132kB), run=6379-6755msec WRITE: bw=933KiB/s (955kB/s), 235KiB/s-242KiB/s (241kB/s-248kB/s), io=6300KiB (6451kB), run=6379-6755msec