== sanity test 398b: DIO and buffer IO race ============== 19:31:24 (1736987484) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.0632033 s, 199 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23329: Wed Jan 15 19:31:27 2025 read: IOPS=204, BW=819KiB/s (839kB/s)(1480KiB/1806msec) clat (usec): min=4, max=41230, avg=1438.07, stdev=3439.96 lat (usec): min=4, max=41230, avg=1438.75, stdev=3439.97 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 11], | 30.00th=[ 13], 40.00th=[ 17], 50.00th=[ 29], 60.00th=[ 955], | 70.00th=[ 1778], 80.00th=[ 2343], 90.00th=[ 3228], 95.00th=[ 4178], | 99.00th=[19530], 99.50th=[27132], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 688, max= 936, per=23.78%, avg=781.33, stdev=134.90, samples=3 iops : min= 172, max= 234, avg=195.33, stdev=33.72, samples=3 write: IOPS=220, BW=882KiB/s (903kB/s)(1592KiB/1806msec) clat (usec): min=94, max=24990, avg=3171.91, stdev=4155.65 lat (usec): min=95, max=24991, avg=3173.00, stdev=4155.87 clat percentiles (usec): | 1.00th=[ 102], 5.00th=[ 126], 10.00th=[ 147], 20.00th=[ 176], | 30.00th=[ 249], 40.00th=[ 914], 50.00th=[ 1500], 60.00th=[ 2245], | 70.00th=[ 3785], 80.00th=[ 5735], 90.00th=[ 8979], 95.00th=[12256], | 99.00th=[19006], 99.50th=[22152], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 696, max= 1048, per=24.30%, avg=840.00, stdev=184.52, samples=3 iops : min= 174, max= 262, avg=210.00, stdev=46.13, samples=3 lat (usec) : 10=8.72%, 20=12.50%, 50=5.47%, 100=0.78%, 250=15.62% lat (usec) : 500=3.65%, 750=1.04%, 1000=2.99% lat (msec) : 2=14.84%, 4=16.67%, 10=12.24%, 20=4.82%, 50=0.65% cpu : usr=0.44%, sys=9.75%, ctx=994, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23330: Wed Jan 15 19:31:27 2025 read: IOPS=201, BW=805KiB/s (825kB/s)(1436KiB/1783msec) clat (usec): min=3, max=24257, avg=1342.40, stdev=2769.88 lat (usec): min=4, max=24257, avg=1343.31, stdev=2769.77 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 12], | 30.00th=[ 15], 40.00th=[ 21], 50.00th=[ 36], 60.00th=[ 1020], | 70.00th=[ 1745], 80.00th=[ 2409], 90.00th=[ 3195], 95.00th=[ 3687], | 99.00th=[16909], 99.50th=[22152], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 704, max= 880, per=24.51%, avg=805.33, stdev=90.98, samples=3 iops : min= 176, max= 220, avg=201.33, stdev=22.74, samples=3 write: IOPS=229, BW=918KiB/s (940kB/s)(1636KiB/1783msec) clat (usec): min=96, max=35031, avg=3153.54, stdev=4576.51 lat (usec): min=97, max=35032, avg=3154.71, stdev=4576.48 clat percentiles (usec): | 1.00th=[ 106], 5.00th=[ 125], 10.00th=[ 141], 20.00th=[ 169], | 30.00th=[ 200], 40.00th=[ 791], 50.00th=[ 1467], 60.00th=[ 2212], | 70.00th=[ 2999], 80.00th=[ 5538], 90.00th=[ 8848], 95.00th=[12125], | 99.00th=[18744], 99.50th=[20579], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 776, max= 1024, per=26.38%, avg=912.00, stdev=125.73, samples=3 iops : min= 194, max= 256, avg=228.00, stdev=31.43, samples=3 lat (usec) : 4=0.13%, 10=6.64%, 20=11.46%, 50=6.12%, 100=1.04% lat (usec) : 250=18.62%, 500=1.82%, 750=1.82%, 1000=3.39% lat (msec) : 2=13.80%, 4=21.09%, 10=8.46%, 20=4.95%, 50=0.65% cpu : usr=0.51%, sys=10.77%, ctx=908, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23331: Wed Jan 15 19:31:27 2025 read: IOPS=214, BW=858KiB/s (879kB/s)(1564KiB/1822msec) clat (usec): min=3, max=26639, avg=1443.52, stdev=2738.87 lat (usec): min=3, max=26640, avg=1444.40, stdev=2738.87 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 14], 40.00th=[ 21], 50.00th=[ 93], 60.00th=[ 1483], | 70.00th=[ 1991], 80.00th=[ 2442], 90.00th=[ 3195], 95.00th=[ 4080], | 99.00th=[17171], 99.50th=[24773], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 584, max= 952, per=23.94%, avg=786.67, stdev=186.82, samples=3 iops : min= 146, max= 238, avg=196.67, stdev=46.70, samples=3 write: IOPS=206, BW=828KiB/s (848kB/s)(1508KiB/1822msec) clat (usec): min=82, max=38695, avg=3303.17, stdev=4375.08 lat (usec): min=83, max=38695, avg=3304.11, stdev=4374.97 clat percentiles (usec): | 1.00th=[ 113], 5.00th=[ 135], 10.00th=[ 159], 20.00th=[ 184], | 30.00th=[ 208], 40.00th=[ 898], 50.00th=[ 1532], 60.00th=[ 2573], | 70.00th=[ 4015], 80.00th=[ 6194], 90.00th=[ 9110], 95.00th=[11207], | 99.00th=[17695], 99.50th=[24511], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 704, max= 936, per=22.76%, avg=786.67, stdev=129.57, samples=3 iops : min= 176, max= 234, avg=196.67, stdev=32.39, samples=3 lat (usec) : 4=0.13%, 10=8.33%, 20=11.33%, 50=5.08%, 100=0.78% lat (usec) : 250=16.54%, 500=2.34%, 750=0.78%, 1000=3.12% lat (msec) : 2=14.84%, 4=19.14%, 10=13.15%, 20=3.78%, 50=0.65% cpu : usr=0.33%, sys=9.67%, ctx=986, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23332: Wed Jan 15 19:31:27 2025 read: IOPS=211, BW=848KiB/s (868kB/s)(1508KiB/1779msec) clat (usec): min=3, max=32997, avg=1279.31, stdev=2604.92 lat (usec): min=4, max=32998, avg=1280.50, stdev=2604.69 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 14], 40.00th=[ 18], 50.00th=[ 31], 60.00th=[ 1172], | 70.00th=[ 1893], 80.00th=[ 2278], 90.00th=[ 3032], 95.00th=[ 3851], | 99.00th=[11600], 99.50th=[18744], 99.90th=[32900], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 720, max= 880, per=24.10%, avg=792.00, stdev=81.19, samples=3 iops : min= 180, max= 220, avg=198.00, stdev=20.30, samples=3 write: IOPS=219, BW=879KiB/s (900kB/s)(1564KiB/1779msec) clat (usec): min=96, max=51468, avg=3288.52, stdev=4763.48 lat (usec): min=97, max=51469, avg=3289.29, stdev=4763.49 clat percentiles (usec): | 1.00th=[ 99], 5.00th=[ 129], 10.00th=[ 155], 20.00th=[ 182], | 30.00th=[ 285], 40.00th=[ 898], 50.00th=[ 1303], 60.00th=[ 2212], | 70.00th=[ 3425], 80.00th=[ 6194], 90.00th=[ 9110], 95.00th=[12256], | 99.00th=[19006], 99.50th=[23200], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 816, max= 888, per=24.68%, avg=853.33, stdev=36.07, samples=3 iops : min= 204, max= 222, avg=213.33, stdev= 9.02, samples=3 lat (usec) : 4=0.13%, 10=7.68%, 20=12.50%, 50=6.12%, 100=1.30% lat (usec) : 250=14.58%, 500=2.86%, 750=1.43%, 1000=3.65% lat (msec) : 2=15.76%, 4=17.45%, 10=11.59%, 20=4.56%, 50=0.26% lat (msec) : 100=0.13% cpu : usr=0.28%, sys=9.67%, ctx=936, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3286KiB/s (3365kB/s), 805KiB/s-858KiB/s (825kB/s-879kB/s), io=5988KiB (6132kB), run=1779-1822msec WRITE: bw=3458KiB/s (3541kB/s), 828KiB/s-918KiB/s (848kB/s-940kB/s), io=6300KiB (6451kB), run=1779-1822msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23333: Wed Jan 15 19:31:32 2025 read: IOPS=55, BW=223KiB/s (228kB/s)(1480KiB/6648msec) clat (usec): min=723, max=97996, avg=2165.61, stdev=5585.81 lat (usec): min=723, max=97996, avg=2166.46, stdev=5585.82 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 832], 10.00th=[ 865], 20.00th=[ 922], | 30.00th=[ 1004], 40.00th=[ 1074], 50.00th=[ 1139], 60.00th=[ 1270], | 70.00th=[ 1467], 80.00th=[ 1893], 90.00th=[ 3032], 95.00th=[ 6063], | 99.00th=[17433], 99.50th=[23462], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 128, max= 328, per=25.10%, avg=219.62, stdev=54.14, samples=13 iops : min= 32, max= 82, avg=54.85, stdev=13.55, samples=13 write: IOPS=59, BW=239KiB/s (245kB/s)(1592KiB/6648msec) clat (usec): min=8048, max=42546, avg=14668.04, stdev=4744.63 lat (usec): min=8049, max=42547, avg=14669.03, stdev=4744.66 clat percentiles (usec): | 1.00th=[ 8291], 5.00th=[ 9110], 10.00th=[ 9634], 20.00th=[10814], | 30.00th=[11863], 40.00th=[13042], 50.00th=[14222], 60.00th=[15401], | 70.00th=[16188], 80.00th=[17433], 90.00th=[19006], 95.00th=[21365], | 99.00th=[37487], 99.50th=[40633], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 136, max= 295, per=25.94%, avg=238.69, stdev=44.66, samples=13 iops : min= 34, max= 73, avg=59.62, stdev=11.09, samples=13 lat (usec) : 750=0.26%, 1000=13.93% lat (msec) : 2=24.74%, 4=5.47%, 10=8.59%, 20=43.10%, 50=3.78% lat (msec) : 100=0.13% cpu : usr=0.20%, sys=2.03%, ctx=826, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23334: Wed Jan 15 19:31:32 2025 read: IOPS=52, BW=210KiB/s (215kB/s)(1436KiB/6843msec) clat (usec): min=671, max=23710, avg=2074.05, stdev=2807.77 lat (usec): min=671, max=23710, avg=2075.05, stdev=2807.76 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 816], 10.00th=[ 865], 20.00th=[ 947], | 30.00th=[ 1029], 40.00th=[ 1139], 50.00th=[ 1237], 60.00th=[ 1385], | 70.00th=[ 1532], 80.00th=[ 1876], 90.00th=[ 2900], 95.00th=[ 8455], | 99.00th=[16057], 99.50th=[21365], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 80, max= 336, per=23.56%, avg=206.15, stdev=72.60, samples=13 iops : min= 20, max= 84, avg=51.54, stdev=18.15, samples=13 write: IOPS=59, BW=239KiB/s (245kB/s)(1636KiB/6843msec) clat (usec): min=7305, max=80104, avg=14885.85, stdev=5605.36 lat (usec): min=7305, max=80105, avg=14886.94, stdev=5605.32 clat percentiles (usec): | 1.00th=[ 7832], 5.00th=[ 8586], 10.00th=[ 9372], 20.00th=[10683], | 30.00th=[11994], 40.00th=[13435], 50.00th=[14615], 60.00th=[15401], | 70.00th=[16188], 80.00th=[17695], 90.00th=[20317], 95.00th=[22938], | 99.00th=[30802], 99.50th=[35914], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 144, max= 280, per=25.35%, avg=233.23, stdev=39.71, samples=13 iops : min= 36, max= 70, avg=58.31, stdev= 9.93, samples=13 lat (usec) : 750=0.13%, 1000=12.11% lat (msec) : 2=25.91%, 4=4.69%, 10=9.64%, 20=41.80%, 50=5.60% lat (msec) : 100=0.13% cpu : usr=0.23%, sys=1.99%, ctx=814, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23335: Wed Jan 15 19:31:32 2025 read: IOPS=59, BW=236KiB/s (242kB/s)(1564KiB/6615msec) clat (usec): min=673, max=20495, avg=1905.15, stdev=2419.17 lat (usec): min=673, max=20496, avg=1906.01, stdev=2419.19 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 799], 10.00th=[ 857], 20.00th=[ 922], | 30.00th=[ 1004], 40.00th=[ 1090], 50.00th=[ 1221], 60.00th=[ 1352], | 70.00th=[ 1516], 80.00th=[ 1860], 90.00th=[ 2606], 95.00th=[ 7439], | 99.00th=[14353], 99.50th=[18744], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 104, max= 376, per=27.36%, avg=239.38, stdev=83.04, samples=13 iops : min= 26, max= 94, avg=59.85, stdev=20.76, samples=13 write: IOPS=56, BW=228KiB/s (233kB/s)(1508KiB/6615msec) clat (msec): min=8, max=110, avg=15.55, stdev= 6.74 lat (msec): min=8, max=110, avg=15.55, stdev= 6.74 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 13], 40.00th=[ 14], 50.00th=[ 16], 60.00th=[ 16], | 70.00th=[ 17], 80.00th=[ 18], 90.00th=[ 21], 95.00th=[ 24], | 99.00th=[ 35], 99.50th=[ 46], 99.90th=[ 111], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 128, max= 272, per=24.75%, avg=227.69, stdev=39.95, samples=13 iops : min= 32, max= 68, avg=56.92, stdev= 9.99, samples=13 lat (usec) : 750=0.78%, 1000=14.32% lat (msec) : 2=26.82%, 4=5.47%, 10=6.12%, 20=41.54%, 50=4.82% lat (msec) : 250=0.13% cpu : usr=0.15%, sys=2.07%, ctx=828, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23336: Wed Jan 15 19:31:32 2025 read: IOPS=57, BW=229KiB/s (234kB/s)(1508KiB/6593msec) clat (usec): min=728, max=21210, avg=1944.44, stdev=2604.71 lat (usec): min=729, max=21211, avg=1945.94, stdev=2604.57 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 840], 10.00th=[ 889], 20.00th=[ 947], | 30.00th=[ 1020], 40.00th=[ 1090], 50.00th=[ 1221], 60.00th=[ 1336], | 70.00th=[ 1450], 80.00th=[ 1876], 90.00th=[ 2737], 95.00th=[ 6980], | 99.00th=[15795], 99.50th=[17695], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 112, max= 296, per=26.01%, avg=227.62, stdev=63.31, samples=13 iops : min= 28, max= 74, avg=56.85, stdev=15.78, samples=13 write: IOPS=59, BW=237KiB/s (243kB/s)(1564KiB/6593msec) clat (usec): min=8158, max=55375, avg=14965.92, stdev=4843.70 lat (usec): min=8159, max=55376, avg=14967.02, stdev=4843.75 clat percentiles (usec): | 1.00th=[ 8291], 5.00th=[ 9372], 10.00th=[10028], 20.00th=[11469], | 30.00th=[12649], 40.00th=[13435], 50.00th=[14484], 60.00th=[15401], | 70.00th=[16188], 80.00th=[17171], 90.00th=[19792], 95.00th=[22152], | 99.00th=[30802], 99.50th=[49021], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 144, max= 296, per=25.81%, avg=237.46, stdev=45.62, samples=13 iops : min= 36, max= 74, avg=59.31, stdev=11.38, samples=13 lat (usec) : 750=0.13%, 1000=13.15% lat (msec) : 2=26.82%, 4=6.25%, 10=6.38%, 20=42.58%, 50=4.56% lat (msec) : 100=0.13% cpu : usr=0.15%, sys=2.11%, ctx=820, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=875KiB/s (896kB/s), 210KiB/s-236KiB/s (215kB/s-242kB/s), io=5988KiB (6132kB), run=6593-6843msec WRITE: bw=921KiB/s (943kB/s), 228KiB/s-239KiB/s (233kB/s-245kB/s), io=6300KiB (6451kB), run=6593-6843msec