== sanity test 398b: DIO and buffer IO race ============== 19:27:27 (1736987247) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.0746467 s, 169 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23681: Wed Jan 15 19:27:30 2025 read: IOPS=173, BW=694KiB/s (711kB/s)(1480KiB/2133msec) clat (usec): min=4, max=26248, avg=1700.45, stdev=3418.07 lat (usec): min=4, max=26248, avg=1701.39, stdev=3417.98 clat percentiles (usec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 13], | 30.00th=[ 17], 40.00th=[ 27], 50.00th=[ 49], 60.00th=[ 1565], | 70.00th=[ 2311], 80.00th=[ 2868], 90.00th=[ 3589], 95.00th=[ 4146], | 99.00th=[22152], 99.50th=[26346], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 600, max= 704, per=24.78%, avg=658.00, stdev=43.51, samples=4 iops : min= 150, max= 176, avg=164.50, stdev=10.88, samples=4 write: IOPS=186, BW=746KiB/s (764kB/s)(1592KiB/2133msec) clat (usec): min=89, max=42824, avg=3748.34, stdev=5338.56 lat (usec): min=90, max=42825, avg=3749.17, stdev=5338.59 clat percentiles (usec): | 1.00th=[ 99], 5.00th=[ 121], 10.00th=[ 139], 20.00th=[ 178], | 30.00th=[ 210], 40.00th=[ 881], 50.00th=[ 1500], 60.00th=[ 2245], | 70.00th=[ 3654], 80.00th=[ 7308], 90.00th=[11207], 95.00th=[15008], | 99.00th=[21890], 99.50th=[25822], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 600, max= 880, per=25.85%, avg=722.00, stdev=117.82, samples=4 iops : min= 150, max= 220, avg=180.50, stdev=29.46, samples=4 lat (usec) : 10=5.86%, 20=10.68%, 50=7.55%, 100=1.56%, 250=17.45% lat (usec) : 500=1.95%, 750=0.78%, 1000=2.34% lat (msec) : 2=13.41%, 4=20.57%, 10=9.90%, 20=6.38%, 50=1.56% cpu : usr=0.19%, sys=8.44%, ctx=969, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23682: Wed Jan 15 19:27:30 2025 read: IOPS=162, BW=648KiB/s (664kB/s)(1436KiB/2216msec) clat (usec): min=3, max=29285, avg=1782.83, stdev=2826.29 lat (usec): min=4, max=29286, avg=1783.83, stdev=2826.11 clat percentiles (usec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 13], | 30.00th=[ 20], 40.00th=[ 34], 50.00th=[ 1172], 60.00th=[ 1762], | 70.00th=[ 2409], 80.00th=[ 2966], 90.00th=[ 3884], 95.00th=[ 5080], | 99.00th=[14484], 99.50th=[17171], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 528, max= 776, per=24.48%, avg=650.00, stdev=101.48, samples=4 iops : min= 132, max= 194, avg=162.50, stdev=25.37, samples=4 write: IOPS=184, BW=738KiB/s (756kB/s)(1636KiB/2216msec) clat (usec): min=87, max=51074, avg=3813.96, stdev=5574.25 lat (usec): min=88, max=51075, avg=3814.98, stdev=5574.37 clat percentiles (usec): | 1.00th=[ 109], 5.00th=[ 139], 10.00th=[ 161], 20.00th=[ 188], | 30.00th=[ 269], 40.00th=[ 947], 50.00th=[ 1418], 60.00th=[ 2114], | 70.00th=[ 3392], 80.00th=[ 7439], 90.00th=[11338], 95.00th=[15401], | 99.00th=[21890], 99.50th=[23200], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 688, max= 776, per=26.28%, avg=734.00, stdev=44.48, samples=4 iops : min= 172, max= 194, avg=183.50, stdev=11.12, samples=4 lat (usec) : 4=0.13%, 10=5.34%, 20=8.72%, 50=6.12%, 100=0.39% lat (usec) : 250=15.49%, 500=2.86%, 750=0.65%, 1000=3.39% lat (msec) : 2=17.58%, 4=19.92%, 10=11.33%, 20=7.03%, 50=0.91% lat (msec) : 100=0.13% cpu : usr=0.23%, sys=10.20%, ctx=1079, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23684: Wed Jan 15 19:27:30 2025 read: IOPS=179, BW=716KiB/s (733kB/s)(1564KiB/2184msec) clat (usec): min=3, max=24845, avg=1891.98, stdev=3519.82 lat (usec): min=4, max=24845, avg=1893.08, stdev=3519.74 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 13], | 30.00th=[ 17], 40.00th=[ 29], 50.00th=[ 1029], 60.00th=[ 1663], | 70.00th=[ 2212], 80.00th=[ 2900], 90.00th=[ 3589], 95.00th=[ 6128], | 99.00th=[22676], 99.50th=[22938], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 584, max= 808, per=27.42%, avg=728.00, stdev=99.49, samples=4 iops : min= 146, max= 202, avg=182.00, stdev=24.87, samples=4 write: IOPS=172, BW=690KiB/s (707kB/s)(1508KiB/2184msec) clat (usec): min=100, max=55027, avg=3805.95, stdev=5777.15 lat (usec): min=101, max=55028, avg=3807.24, stdev=5777.47 clat percentiles (usec): | 1.00th=[ 112], 5.00th=[ 145], 10.00th=[ 161], 20.00th=[ 184], | 30.00th=[ 235], 40.00th=[ 889], 50.00th=[ 1532], 60.00th=[ 2311], | 70.00th=[ 3359], 80.00th=[ 6915], 90.00th=[11076], 95.00th=[16188], | 99.00th=[21890], 99.50th=[32375], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 624, max= 792, per=24.63%, avg=688.00, stdev=75.89, samples=4 iops : min= 156, max= 198, avg=172.00, stdev=18.97, samples=4 lat (usec) : 4=0.13%, 10=5.08%, 20=12.24%, 50=5.47%, 100=1.04% lat (usec) : 250=15.76%, 500=2.60%, 750=0.91%, 1000=2.47% lat (msec) : 2=16.54%, 4=19.79%, 10=9.64%, 20=6.64%, 50=1.56% lat (msec) : 100=0.13% cpu : usr=0.46%, sys=8.84%, ctx=1035, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23686: Wed Jan 15 19:27:30 2025 read: IOPS=167, BW=669KiB/s (685kB/s)(1508KiB/2255msec) clat (usec): min=3, max=29170, avg=1794.15, stdev=3428.37 lat (usec): min=3, max=29171, avg=1794.96, stdev=3428.36 clat percentiles (usec): | 1.00th=[ 4], 5.00th=[ 7], 10.00th=[ 9], 20.00th=[ 12], | 30.00th=[ 15], 40.00th=[ 28], 50.00th=[ 865], 60.00th=[ 1713], | 70.00th=[ 2212], 80.00th=[ 2900], 90.00th=[ 3523], 95.00th=[ 5014], | 99.00th=[22152], 99.50th=[24511], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 448, max= 688, per=22.52%, avg=598.00, stdev=105.20, samples=4 iops : min= 112, max= 172, avg=149.50, stdev=26.30, samples=4 write: IOPS=173, BW=694KiB/s (710kB/s)(1564KiB/2255msec) clat (usec): min=93, max=53226, avg=3995.20, stdev=5710.25 lat (usec): min=93, max=53227, avg=3997.41, stdev=5709.67 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 133], 10.00th=[ 161], 20.00th=[ 192], | 30.00th=[ 441], 40.00th=[ 1106], 50.00th=[ 1647], 60.00th=[ 2409], | 70.00th=[ 4015], 80.00th=[ 7373], 90.00th=[11207], 95.00th=[13829], | 99.00th=[26608], 99.50th=[36439], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 472, max= 736, per=23.06%, avg=644.00, stdev=120.89, samples=4 iops : min= 118, max= 184, avg=161.00, stdev=30.22, samples=4 lat (usec) : 4=1.04%, 10=6.51%, 20=9.77%, 50=5.08%, 100=1.17% lat (usec) : 250=14.45%, 500=2.21%, 750=0.52%, 1000=2.99% lat (msec) : 2=15.49%, 4=21.74%, 10=10.68%, 20=7.03%, 50=1.17% lat (msec) : 100=0.13% cpu : usr=0.27%, sys=8.47%, ctx=1023, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2655KiB/s (2719kB/s), 648KiB/s-716KiB/s (664kB/s-733kB/s), io=5988KiB (6132kB), run=2133-2255msec WRITE: bw=2794KiB/s (2861kB/s), 690KiB/s-746KiB/s (707kB/s-764kB/s), io=6300KiB (6451kB), run=2133-2255msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23683: Wed Jan 15 19:27:36 2025 read: IOPS=46, BW=187KiB/s (191kB/s)(1480KiB/7932msec) clat (usec): min=701, max=29043, avg=2272.10, stdev=3496.77 lat (usec): min=701, max=29044, avg=2273.06, stdev=3496.80 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 824], 10.00th=[ 865], 20.00th=[ 930], | 30.00th=[ 1004], 40.00th=[ 1090], 50.00th=[ 1205], 60.00th=[ 1385], | 70.00th=[ 1582], 80.00th=[ 2024], 90.00th=[ 3556], 95.00th=[ 9110], | 99.00th=[19792], 99.50th=[22676], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 112, max= 296, per=24.63%, avg=183.47, stdev=51.62, samples=15 iops : min= 28, max= 74, avg=45.87, stdev=12.91, samples=15 write: IOPS=50, BW=201KiB/s (206kB/s)(1592KiB/7932msec) clat (usec): min=8831, max=58842, avg=17774.91, stdev=6289.84 lat (usec): min=8831, max=58843, avg=17776.10, stdev=6289.82 clat percentiles (usec): | 1.00th=[ 9503], 5.00th=[10683], 10.00th=[11207], 20.00th=[12780], | 30.00th=[14746], 40.00th=[16450], 50.00th=[17433], 60.00th=[18482], | 70.00th=[19268], 80.00th=[20579], 90.00th=[23200], 95.00th=[26870], | 99.00th=[50070], 99.50th=[57410], 99.90th=[58983], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 120, max= 240, per=25.24%, avg=197.87, stdev=38.30, samples=15 iops : min= 30, max= 60, avg=49.47, stdev= 9.58, samples=15 lat (usec) : 750=0.78%, 1000=13.28% lat (msec) : 2=24.35%, 4=5.60%, 10=3.39%, 20=40.10%, 50=11.98% lat (msec) : 100=0.52% cpu : usr=0.23%, sys=1.63%, ctx=831, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23685: Wed Jan 15 19:27:36 2025 read: IOPS=44, BW=179KiB/s (183kB/s)(1436KiB/8032msec) clat (usec): min=691, max=34924, avg=2286.92, stdev=3411.86 lat (usec): min=692, max=34925, avg=2287.86, stdev=3411.90 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 832], 10.00th=[ 881], 20.00th=[ 947], | 30.00th=[ 1020], 40.00th=[ 1106], 50.00th=[ 1237], 60.00th=[ 1352], | 70.00th=[ 1565], 80.00th=[ 2180], 90.00th=[ 3982], 95.00th=[ 9110], | 99.00th=[17171], 99.50th=[18482], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 72, max= 312, per=23.83%, avg=177.53, stdev=67.91, samples=15 iops : min= 18, max= 78, avg=44.33, stdev=16.88, samples=15 write: IOPS=50, BW=204KiB/s (209kB/s)(1636KiB/8032msec) clat (usec): min=9565, max=79473, avg=17589.72, stdev=6934.99 lat (usec): min=9566, max=79475, avg=17590.88, stdev=6935.00 clat percentiles (usec): | 1.00th=[ 9765], 5.00th=[10421], 10.00th=[10945], 20.00th=[12256], | 30.00th=[14091], 40.00th=[15795], 50.00th=[17171], 60.00th=[17957], | 70.00th=[19006], 80.00th=[20317], 90.00th=[22676], 95.00th=[27919], | 99.00th=[44827], 99.50th=[52691], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 136, max= 240, per=25.37%, avg=198.87, stdev=36.97, samples=15 iops : min= 34, max= 60, avg=49.67, stdev= 9.21, samples=15 lat (usec) : 750=0.26%, 1000=12.63% lat (msec) : 2=23.31%, 4=5.99%, 10=3.65%, 20=42.58%, 50=11.20% lat (msec) : 100=0.39% cpu : usr=0.26%, sys=1.61%, ctx=828, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23687: Wed Jan 15 19:27:36 2025 read: IOPS=51, BW=206KiB/s (211kB/s)(1564KiB/7583msec) clat (usec): min=699, max=26189, avg=2378.13, stdev=3645.92 lat (usec): min=699, max=26190, avg=2379.12, stdev=3646.16 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 807], 10.00th=[ 865], 20.00th=[ 930], | 30.00th=[ 988], 40.00th=[ 1074], 50.00th=[ 1188], 60.00th=[ 1319], | 70.00th=[ 1532], 80.00th=[ 1909], 90.00th=[ 3916], 95.00th=[11469], | 99.00th=[18744], 99.50th=[19792], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 96, max= 368, per=27.91%, avg=207.93, stdev=81.19, samples=15 iops : min= 24, max= 92, avg=51.93, stdev=20.30, samples=15 write: IOPS=49, BW=199KiB/s (204kB/s)(1508KiB/7583msec) clat (usec): min=8768, max=80380, avg=17611.66, stdev=6054.80 lat (usec): min=8768, max=80381, avg=17612.77, stdev=6054.87 clat percentiles (usec): | 1.00th=[ 9765], 5.00th=[10683], 10.00th=[11207], 20.00th=[12518], | 30.00th=[14484], 40.00th=[16581], 50.00th=[17695], 60.00th=[18482], | 70.00th=[19530], 80.00th=[20579], 90.00th=[22676], 95.00th=[27132], | 99.00th=[40109], 99.50th=[41681], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 112, max= 240, per=25.30%, avg=198.33, stdev=35.58, samples=15 iops : min= 28, max= 60, avg=49.53, stdev= 8.92, samples=15 lat (usec) : 750=0.39%, 1000=15.62% lat (msec) : 2=25.26%, 4=4.69%, 10=2.73%, 20=39.45%, 50=11.72% lat (msec) : 100=0.13% cpu : usr=0.21%, sys=1.74%, ctx=847, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23688: Wed Jan 15 19:27:36 2025 read: IOPS=49, BW=196KiB/s (201kB/s)(1508KiB/7687msec) clat (usec): min=698, max=32872, avg=2260.47, stdev=3680.37 lat (usec): min=698, max=32873, avg=2261.43, stdev=3680.40 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 816], 10.00th=[ 865], 20.00th=[ 938], | 30.00th=[ 1004], 40.00th=[ 1090], 50.00th=[ 1237], 60.00th=[ 1352], | 70.00th=[ 1565], 80.00th=[ 1942], 90.00th=[ 3359], 95.00th=[ 8848], | 99.00th=[23725], 99.50th=[30540], 99.90th=[32900], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 112, max= 304, per=25.84%, avg=192.53, stdev=60.43, samples=15 iops : min= 28, max= 76, avg=48.13, stdev=15.11, samples=15 write: IOPS=50, BW=203KiB/s (208kB/s)(1564KiB/7687msec) clat (usec): min=8943, max=56927, avg=17435.46, stdev=5636.72 lat (usec): min=8944, max=56929, avg=17436.63, stdev=5636.70 clat percentiles (usec): | 1.00th=[ 9503], 5.00th=[10421], 10.00th=[11207], 20.00th=[12387], | 30.00th=[14091], 40.00th=[16188], 50.00th=[17433], 60.00th=[18220], | 70.00th=[19268], 80.00th=[20579], 90.00th=[22938], 95.00th=[26346], | 99.00th=[40109], 99.50th=[46400], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 128, max= 256, per=25.78%, avg=202.13, stdev=40.84, samples=15 iops : min= 32, max= 64, avg=50.53, stdev=10.21, samples=15 lat (usec) : 750=0.26%, 1000=14.06% lat (msec) : 2=25.26%, 4=5.08%, 10=3.39%, 20=39.45%, 50=12.37% lat (msec) : 100=0.13% cpu : usr=0.25%, sys=1.67%, ctx=828, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=746KiB/s (763kB/s), 179KiB/s-206KiB/s (183kB/s-211kB/s), io=5988KiB (6132kB), run=7583-8032msec WRITE: bw=784KiB/s (803kB/s), 199KiB/s-204KiB/s (204kB/s-209kB/s), io=6300KiB (6451kB), run=7583-8032msec