== sanity test 398b: DIO and buffer IO race ============== 19:13:32 (1717024412) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.0763296 s, 165 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=19515: Wed May 29 19:13:37 2024 read: IOPS=112, BW=452KiB/s (462kB/s)(1480KiB/3277msec) clat (usec): min=3, max=45579, avg=2330.05, stdev=4941.93 lat (usec): min=4, max=45580, avg=2330.82, stdev=4941.95 clat percentiles (usec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 13], | 30.00th=[ 19], 40.00th=[ 30], 50.00th=[ 914], 60.00th=[ 1926], | 70.00th=[ 2704], 80.00th=[ 3425], 90.00th=[ 4883], 95.00th=[ 6521], | 99.00th=[29754], 99.50th=[36963], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 272, max= 520, per=23.13%, avg=422.67, stdev=100.61, samples=6 iops : min= 68, max= 130, avg=105.67, stdev=25.15, samples=6 write: IOPS=121, BW=486KiB/s (497kB/s)(1592KiB/3277msec) clat (usec): min=92, max=44315, avg=6044.85, stdev=8292.55 lat (usec): min=93, max=44316, avg=6045.73, stdev=8292.58 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 139], 10.00th=[ 151], 20.00th=[ 172], | 30.00th=[ 420], 40.00th=[ 1287], 50.00th=[ 2114], 60.00th=[ 3425], | 70.00th=[ 6783], 80.00th=[11863], 90.00th=[17695], 95.00th=[24511], | 99.00th=[38536], 99.50th=[40633], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 352, max= 536, per=24.07%, avg=462.67, stdev=64.08, samples=6 iops : min= 88, max= 134, avg=115.67, stdev=16.02, samples=6 lat (usec) : 4=0.13%, 10=5.34%, 20=9.77%, 50=5.99%, 100=1.30% lat (usec) : 250=15.10%, 500=2.21%, 750=1.04%, 1000=2.08% lat (msec) : 2=11.33%, 4=19.66%, 10=11.85%, 20=9.51%, 50=4.69% cpu : usr=0.12%, sys=7.36%, ctx=1032, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19516: Wed May 29 19:13:37 2024 read: IOPS=111, BW=445KiB/s (455kB/s)(1436KiB/3230msec) clat (usec): min=4, max=45332, avg=2752.16, stdev=5924.22 lat (usec): min=4, max=45333, avg=2752.99, stdev=5924.22 clat percentiles (usec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 13], | 30.00th=[ 22], 40.00th=[ 36], 50.00th=[ 1205], 60.00th=[ 2089], | 70.00th=[ 2868], 80.00th=[ 3589], 90.00th=[ 5276], 95.00th=[ 8717], | 99.00th=[34866], 99.50th=[40633], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 280, max= 616, per=24.01%, avg=438.67, stdev=125.21, samples=6 iops : min= 70, max= 154, avg=109.67, stdev=31.30, samples=6 write: IOPS=126, BW=507KiB/s (519kB/s)(1636KiB/3230msec) clat (usec): min=95, max=59318, avg=5457.86, stdev=8535.12 lat (usec): min=96, max=59320, avg=5458.74, stdev=8535.15 clat percentiles (usec): | 1.00th=[ 115], 5.00th=[ 143], 10.00th=[ 151], 20.00th=[ 178], | 30.00th=[ 255], 40.00th=[ 1004], 50.00th=[ 1795], 60.00th=[ 2802], | 70.00th=[ 4621], 80.00th=[10028], 90.00th=[16909], 95.00th=[21627], | 99.00th=[42206], 99.50th=[49546], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 336, max= 624, per=25.53%, avg=490.67, stdev=120.12, samples=6 iops : min= 84, max= 156, avg=122.67, stdev=30.03, samples=6 lat (usec) : 10=5.47%, 20=8.46%, 50=6.64%, 100=0.39%, 250=16.54% lat (usec) : 500=2.86%, 750=0.78%, 1000=2.86% lat (msec) : 2=10.94%, 4=20.31%, 10=11.98%, 20=8.46%, 50=4.04% lat (msec) : 100=0.26% cpu : usr=0.46%, sys=6.91%, ctx=987, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19517: Wed May 29 19:13:37 2024 read: IOPS=121, BW=486KiB/s (498kB/s)(1564KiB/3217msec) clat (usec): min=3, max=66254, avg=2376.88, stdev=5156.21 lat (usec): min=3, max=66256, avg=2377.75, stdev=5156.19 clat percentiles (usec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 12], | 30.00th=[ 18], 40.00th=[ 28], 50.00th=[ 200], 60.00th=[ 1844], | 70.00th=[ 2704], 80.00th=[ 3556], 90.00th=[ 5145], 95.00th=[ 6980], | 99.00th=[25297], 99.50th=[28443], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 200, max= 632, per=25.32%, avg=462.67, stdev=179.99, samples=6 iops : min= 50, max= 158, avg=115.67, stdev=45.00, samples=6 write: IOPS=117, BW=469KiB/s (480kB/s)(1508KiB/3217msec) clat (usec): min=84, max=52295, avg=6045.03, stdev=8908.66 lat (usec): min=84, max=52297, avg=6045.85, stdev=8908.71 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 137], 10.00th=[ 147], 20.00th=[ 174], | 30.00th=[ 260], 40.00th=[ 1057], 50.00th=[ 1663], 60.00th=[ 2999], | 70.00th=[ 6521], 80.00th=[12649], 90.00th=[17171], 95.00th=[23725], | 99.00th=[42730], 99.50th=[52167], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 280, max= 560, per=23.24%, avg=446.67, stdev=116.20, samples=6 iops : min= 70, max= 140, avg=111.67, stdev=29.05, samples=6 lat (usec) : 4=0.26%, 10=7.42%, 20=8.98%, 50=7.42%, 100=1.56% lat (usec) : 250=14.45%, 500=2.60%, 750=0.52%, 1000=2.34% lat (msec) : 2=11.85%, 4=16.41%, 10=12.37%, 20=9.24%, 50=4.17% lat (msec) : 100=0.39% cpu : usr=0.34%, sys=6.62%, ctx=948, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19518: Wed May 29 19:13:37 2024 read: IOPS=120, BW=481KiB/s (492kB/s)(1508KiB/3137msec) clat (usec): min=4, max=49797, avg=2363.23, stdev=4562.57 lat (usec): min=4, max=49797, avg=2364.43, stdev=4562.63 clat percentiles (usec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 20], 40.00th=[ 34], 50.00th=[ 1270], 60.00th=[ 2114], | 70.00th=[ 2737], 80.00th=[ 3490], 90.00th=[ 5276], 95.00th=[ 7308], | 99.00th=[25560], 99.50th=[32375], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 304, max= 590, per=25.01%, avg=457.00, stdev=119.76, samples=6 iops : min= 76, max= 147, avg=114.17, stdev=29.83, samples=6 write: IOPS=124, BW=499KiB/s (511kB/s)(1564KiB/3137msec) clat (usec): min=100, max=55610, avg=5717.52, stdev=8781.93 lat (usec): min=101, max=55612, avg=5718.54, stdev=8781.92 clat percentiles (usec): | 1.00th=[ 109], 5.00th=[ 143], 10.00th=[ 151], 20.00th=[ 178], | 30.00th=[ 255], 40.00th=[ 1139], 50.00th=[ 1860], 60.00th=[ 2868], | 70.00th=[ 4817], 80.00th=[10159], 90.00th=[17171], 95.00th=[24773], | 99.00th=[40109], 99.50th=[49021], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 384, max= 648, per=25.03%, avg=481.00, stdev=105.04, samples=6 iops : min= 96, max= 162, avg=120.17, stdev=26.19, samples=6 lat (usec) : 10=4.56%, 20=10.29%, 50=6.12%, 100=1.43%, 250=15.89% lat (usec) : 500=2.21%, 750=1.43%, 1000=1.43% lat (msec) : 2=12.24%, 4=19.01%, 10=13.41%, 20=7.42%, 50=4.43% lat (msec) : 100=0.13% cpu : usr=0.22%, sys=7.17%, ctx=989, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1827KiB/s (1871kB/s), 445KiB/s-486KiB/s (455kB/s-498kB/s), io=5988KiB (6132kB), run=3137-3277msec WRITE: bw=1922KiB/s (1969kB/s), 469KiB/s-507KiB/s (480kB/s-519kB/s), io=6300KiB (6451kB), run=3137-3277msec rand-rw: (groupid=0, jobs=1): err= 0: pid=19511: Wed May 29 19:13:44 2024 read: IOPS=36, BW=144KiB/s (148kB/s)(1480KiB/10271msec) clat (usec): min=590, max=31471, avg=2853.49, stdev=4677.64 lat (usec): min=591, max=31472, avg=2854.60, stdev=4677.68 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 807], 10.00th=[ 857], 20.00th=[ 938], | 30.00th=[ 1045], 40.00th=[ 1156], 50.00th=[ 1352], 60.00th=[ 1549], | 70.00th=[ 1713], 80.00th=[ 2278], 90.00th=[ 5800], 95.00th=[13435], | 99.00th=[26608], 99.50th=[30278], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 64, max= 280, per=24.52%, avg=140.75, stdev=56.22, samples=20 iops : min= 16, max= 70, avg=35.15, stdev=14.09, samples=20 write: IOPS=38, BW=155KiB/s (159kB/s)(1592KiB/10271msec) clat (usec): min=11079, max=64194, avg=23115.27, stdev=8962.08 lat (usec): min=11080, max=64195, avg=23116.60, stdev=8962.21 clat percentiles (usec): | 1.00th=[11338], 5.00th=[12387], 10.00th=[13566], 20.00th=[15926], | 30.00th=[17695], 40.00th=[19792], 50.00th=[21365], 60.00th=[22938], | 70.00th=[25297], 80.00th=[28443], 90.00th=[35914], 95.00th=[42730], | 99.00th=[53740], 99.50th=[58459], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 80, max= 200, per=25.42%, avg=153.55, stdev=39.81, samples=20 iops : min= 20, max= 50, avg=38.35, stdev=10.00, samples=20 lat (usec) : 750=0.78%, 1000=11.59% lat (msec) : 2=24.61%, 4=5.21%, 10=2.47%, 20=24.22%, 50=30.47% lat (msec) : 100=0.65% cpu : usr=0.28%, sys=1.42%, ctx=838, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19512: Wed May 29 19:13:44 2024 read: IOPS=34, BW=138KiB/s (141kB/s)(1436KiB/10424msec) clat (usec): min=661, max=42282, avg=2608.00, stdev=4305.93 lat (usec): min=661, max=42283, avg=2609.04, stdev=4306.00 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 816], 10.00th=[ 889], 20.00th=[ 1029], | 30.00th=[ 1123], 40.00th=[ 1237], 50.00th=[ 1385], 60.00th=[ 1532], | 70.00th=[ 1860], 80.00th=[ 2376], 90.00th=[ 4146], 95.00th=[10683], | 99.00th=[24773], 99.50th=[27132], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 24, max= 240, per=23.70%, avg=136.05, stdev=56.89, samples=20 iops : min= 6, max= 60, avg=33.90, stdev=14.19, samples=20 write: IOPS=39, BW=157KiB/s (161kB/s)(1636KiB/10424msec) clat (usec): min=10430, max=66023, avg=23149.08, stdev=9522.25 lat (usec): min=10430, max=66024, avg=23150.38, stdev=9522.24 clat percentiles (usec): | 1.00th=[11076], 5.00th=[12387], 10.00th=[13566], 20.00th=[15795], | 30.00th=[17695], 40.00th=[19792], 50.00th=[21627], 60.00th=[23200], | 70.00th=[25297], 80.00th=[28443], 90.00th=[34341], 95.00th=[40633], | 99.00th=[63177], 99.50th=[64226], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 80, max= 207, per=25.37%, avg=153.25, stdev=37.84, samples=20 iops : min= 20, max= 51, avg=38.20, stdev= 9.42, samples=20 lat (usec) : 750=0.78%, 1000=7.81% lat (msec) : 2=25.39%, 4=7.94%, 10=2.34%, 20=24.09%, 50=29.82% lat (msec) : 100=1.82% cpu : usr=0.14%, sys=1.50%, ctx=830, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19513: Wed May 29 19:13:44 2024 read: IOPS=39, BW=159KiB/s (162kB/s)(1564KiB/9866msec) clat (usec): min=642, max=38147, avg=2453.57, stdev=3862.29 lat (usec): min=642, max=38149, avg=2454.71, stdev=3862.34 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 799], 10.00th=[ 865], 20.00th=[ 996], | 30.00th=[ 1090], 40.00th=[ 1188], 50.00th=[ 1319], 60.00th=[ 1516], | 70.00th=[ 1696], 80.00th=[ 2147], 90.00th=[ 4113], 95.00th=[10814], | 99.00th=[23725], 99.50th=[26870], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 64, max= 280, per=27.85%, avg=159.84, stdev=70.82, samples=19 iops : min= 16, max= 70, avg=39.84, stdev=17.69, samples=19 write: IOPS=38, BW=153KiB/s (157kB/s)(1508KiB/9866msec) clat (usec): min=11030, max=89639, avg=23576.58, stdev=9598.15 lat (usec): min=11031, max=89641, avg=23578.65, stdev=9598.96 clat percentiles (usec): | 1.00th=[11731], 5.00th=[12911], 10.00th=[13829], 20.00th=[16450], | 30.00th=[18482], 40.00th=[20317], 50.00th=[22152], 60.00th=[23462], | 70.00th=[26084], 80.00th=[28443], 90.00th=[34341], 95.00th=[41157], | 99.00th=[54789], 99.50th=[86508], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 80, max= 199, per=25.00%, avg=151.00, stdev=35.68, samples=19 iops : min= 20, max= 49, avg=37.63, stdev= 8.86, samples=19 lat (usec) : 750=1.04%, 1000=9.51% lat (msec) : 2=29.17%, 4=5.73%, 10=2.60%, 20=21.22%, 50=29.56% lat (msec) : 100=1.17% cpu : usr=0.09%, sys=1.62%, ctx=843, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19514: Wed May 29 19:13:44 2024 read: IOPS=36, BW=147KiB/s (150kB/s)(1508KiB/10289msec) clat (usec): min=663, max=32679, avg=2659.08, stdev=4129.00 lat (usec): min=664, max=32680, avg=2660.51, stdev=4129.24 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 783], 10.00th=[ 824], 20.00th=[ 979], | 30.00th=[ 1090], 40.00th=[ 1270], 50.00th=[ 1401], 60.00th=[ 1598], | 70.00th=[ 1811], 80.00th=[ 2376], 90.00th=[ 4817], 95.00th=[12387], | 99.00th=[22152], 99.50th=[31851], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 56, max= 232, per=25.08%, avg=143.95, stdev=54.61, samples=20 iops : min= 14, max= 58, avg=35.95, stdev=13.69, samples=20 write: IOPS=38, BW=152KiB/s (156kB/s)(1564KiB/10289msec) clat (msec): min=10, max=112, avg=23.71, stdev=10.16 lat (msec): min=10, max=112, avg=23.71, stdev=10.16 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 29], 90.00th=[ 34], 95.00th=[ 40], | 99.00th=[ 63], 99.50th=[ 91], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 87, max= 200, per=24.89%, avg=150.35, stdev=37.49, samples=20 iops : min= 21, max= 50, avg=37.55, stdev= 9.44, samples=20 lat (usec) : 750=1.04%, 1000=9.64% lat (msec) : 2=26.43%, 4=5.99%, 10=2.73%, 20=22.40%, 50=30.47% lat (msec) : 100=1.17%, 250=0.13% cpu : usr=0.20%, sys=1.47%, ctx=847, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=574KiB/s (588kB/s), 138KiB/s-159KiB/s (141kB/s-162kB/s), io=5988KiB (6132kB), run=9866-10424msec WRITE: bw=604KiB/s (619kB/s), 152KiB/s-157KiB/s (156kB/s-161kB/s), io=6300KiB (6451kB), run=9866-10424msec