== sanity test 398b: DIO and buffer IO race ============== 18:31:18 (1748989878) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.0556875 s, 226 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=3224: Tue Jun 3 18:31:21 2025 read: IOPS=471, BW=1885KiB/s (1931kB/s)(1480KiB/785msec) clat (usec): min=3, max=28440, avg=661.88, stdev=2261.45 lat (usec): min=3, max=28441, avg=662.36, stdev=2261.51 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 13], | 70.00th=[ 21], 80.00th=[ 1057], 90.00th=[ 2073], 95.00th=[ 2835], | 99.00th=[ 7439], 99.50th=[25297], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 2496, max= 2496, per=52.90%, avg=2496.00, stdev= 0.00, samples=1 iops : min= 624, max= 624, avg=624.00, stdev= 0.00, samples=1 write: IOPS=507, BW=2028KiB/s (2077kB/s)(1592KiB/785msec) clat (usec): min=84, max=39892, avg=1316.95, stdev=3181.62 lat (usec): min=85, max=39892, avg=1317.51, stdev=3181.69 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 98], 10.00th=[ 103], 20.00th=[ 111], | 30.00th=[ 120], 40.00th=[ 133], 50.00th=[ 147], 60.00th=[ 178], | 70.00th=[ 523], 80.00th=[ 1352], 90.00th=[ 4113], 95.00th=[ 8356], | 99.00th=[12649], 99.50th=[14615], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 2728, max= 2728, per=54.96%, avg=2728.00, stdev= 0.00, samples=1 iops : min= 682, max= 682, avg=682.00, stdev= 0.00, samples=1 lat (usec) : 4=0.13%, 10=26.69%, 20=6.51%, 50=3.65%, 100=4.04% lat (usec) : 250=31.90%, 500=1.17%, 750=0.78%, 1000=2.34% lat (msec) : 2=9.11%, 4=7.29%, 10=4.95%, 20=1.04%, 50=0.39% cpu : usr=0.89%, sys=13.52%, ctx=446, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3225: Tue Jun 3 18:31:21 2025 read: IOPS=419, BW=1678KiB/s (1718kB/s)(1436KiB/856msec) clat (usec): min=3, max=24755, avg=690.39, stdev=1813.97 lat (usec): min=4, max=24755, avg=691.14, stdev=1814.31 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 13], 60.00th=[ 17], | 70.00th=[ 30], 80.00th=[ 1434], 90.00th=[ 2507], 95.00th=[ 3032], | 99.00th=[ 7504], 99.50th=[ 9896], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 2328, max= 2328, per=49.34%, avg=2328.00, stdev= 0.00, samples=1 iops : min= 582, max= 582, avg=582.00, stdev= 0.00, samples=1 write: IOPS=477, BW=1911KiB/s (1957kB/s)(1636KiB/856msec) clat (usec): min=92, max=23363, avg=1449.37, stdev=2863.64 lat (usec): min=92, max=23364, avg=1450.14, stdev=2863.65 clat percentiles (usec): | 1.00th=[ 102], 5.00th=[ 113], 10.00th=[ 120], 20.00th=[ 135], | 30.00th=[ 151], 40.00th=[ 167], 50.00th=[ 196], 60.00th=[ 245], | 70.00th=[ 930], 80.00th=[ 2024], 90.00th=[ 5080], 95.00th=[ 7767], | 99.00th=[11207], 99.50th=[15795], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 2512, max= 2512, per=50.60%, avg=2512.00, stdev= 0.00, samples=1 iops : min= 628, max= 628, avg=628.00, stdev= 0.00, samples=1 lat (usec) : 4=0.13%, 10=19.14%, 20=10.42%, 50=4.30%, 100=0.91% lat (usec) : 250=32.68%, 500=3.39%, 750=0.78%, 1000=2.34% lat (msec) : 2=7.94%, 4=10.16%, 10=5.99%, 20=1.56%, 50=0.26% cpu : usr=0.58%, sys=15.32%, ctx=515, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3226: Tue Jun 3 18:31:21 2025 read: IOPS=308, BW=1232KiB/s (1262kB/s)(1564KiB/1269msec) clat (usec): min=3, max=34613, avg=1288.80, stdev=2617.21 lat (usec): min=3, max=34614, avg=1289.38, stdev=2617.25 clat percentiles (usec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 16], 50.00th=[ 24], 60.00th=[ 1106], | 70.00th=[ 1778], 80.00th=[ 2343], 90.00th=[ 3130], 95.00th=[ 3818], | 99.00th=[13304], 99.50th=[15795], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 944, max= 1432, per=25.18%, avg=1188.00, stdev=345.07, samples=2 iops : min= 236, max= 358, avg=297.00, stdev=86.27, samples=2 write: IOPS=297, BW=1188KiB/s (1217kB/s)(1508KiB/1269msec) clat (usec): min=82, max=41434, avg=1990.80, stdev=3583.79 lat (usec): min=83, max=41435, avg=1991.67, stdev=3584.12 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 98], 10.00th=[ 106], 20.00th=[ 123], | 30.00th=[ 159], 40.00th=[ 206], 50.00th=[ 338], 60.00th=[ 1090], | 70.00th=[ 1893], 80.00th=[ 2868], 90.00th=[ 6521], 95.00th=[ 8979], | 99.00th=[13829], 99.50th=[19006], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 960, max= 1448, per=24.25%, avg=1204.00, stdev=345.07, samples=2 iops : min= 240, max= 362, avg=301.00, stdev=86.27, samples=2 lat (usec) : 4=1.04%, 10=13.93%, 20=8.59%, 50=4.30%, 100=4.04% lat (usec) : 250=19.92%, 500=2.60%, 750=1.04%, 1000=2.47% lat (msec) : 2=14.97%, 4=17.06%, 10=7.68%, 20=2.08%, 50=0.26% cpu : usr=0.47%, sys=15.52%, ctx=761, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3227: Tue Jun 3 18:31:21 2025 read: IOPS=329, BW=1317KiB/s (1349kB/s)(1508KiB/1145msec) clat (usec): min=3, max=27461, avg=1014.73, stdev=2120.12 lat (usec): min=3, max=27462, avg=1015.39, stdev=2120.13 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 14], 40.00th=[ 19], 50.00th=[ 27], 60.00th=[ 74], | 70.00th=[ 1450], 80.00th=[ 1991], 90.00th=[ 2769], 95.00th=[ 3326], | 99.00th=[11076], 99.50th=[13698], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 1048, max= 1528, per=27.30%, avg=1288.00, stdev=339.41, samples=2 iops : min= 262, max= 382, avg=322.00, stdev=84.85, samples=2 write: IOPS=341, BW=1366KiB/s (1399kB/s)(1564KiB/1145msec) clat (usec): min=94, max=22343, avg=1915.36, stdev=3138.06 lat (usec): min=95, max=22343, avg=1916.77, stdev=3138.14 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 116], 10.00th=[ 128], 20.00th=[ 157], | 30.00th=[ 178], 40.00th=[ 212], 50.00th=[ 318], 60.00th=[ 947], | 70.00th=[ 1647], 80.00th=[ 2704], 90.00th=[ 6259], 95.00th=[ 8717], | 99.00th=[14222], 99.50th=[19268], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 1160, max= 1520, per=26.99%, avg=1340.00, stdev=254.56, samples=2 iops : min= 290, max= 380, avg=335.00, stdev=63.64, samples=2 lat (usec) : 4=0.26%, 10=9.77%, 20=11.33%, 50=7.42%, 100=1.17% lat (usec) : 250=23.18%, 500=3.12%, 750=1.69%, 1000=4.43% lat (msec) : 2=14.45%, 4=13.67%, 10=7.29%, 20=1.95%, 50=0.26% cpu : usr=0.87%, sys=15.12%, ctx=727, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4719KiB/s (4832kB/s), 1232KiB/s-1885KiB/s (1262kB/s-1931kB/s), io=5988KiB (6132kB), run=785-1269msec WRITE: bw=4965KiB/s (5084kB/s), 1188KiB/s-2028KiB/s (1217kB/s-2077kB/s), io=6300KiB (6451kB), run=785-1269msec rand-rw: (groupid=0, jobs=1): err= 0: pid=3229: Tue Jun 3 18:31:25 2025 read: IOPS=66, BW=266KiB/s (273kB/s)(1480KiB/5560msec) clat (usec): min=578, max=21007, avg=1821.06, stdev=2014.60 lat (usec): min=578, max=21008, avg=1821.84, stdev=2014.63 clat percentiles (usec): | 1.00th=[ 635], 5.00th=[ 734], 10.00th=[ 775], 20.00th=[ 857], | 30.00th=[ 930], 40.00th=[ 988], 50.00th=[ 1090], 60.00th=[ 1352], | 70.00th=[ 1795], 80.00th=[ 2376], 90.00th=[ 2933], 95.00th=[ 4686], | 99.00th=[11600], 99.50th=[13304], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 152, max= 392, per=25.20%, avg=264.36, stdev=73.82, samples=11 iops : min= 38, max= 98, avg=65.91, stdev=18.52, samples=11 write: IOPS=71, BW=286KiB/s (293kB/s)(1592KiB/5560msec) clat (usec): min=6570, max=45428, avg=12250.87, stdev=3890.08 lat (usec): min=6571, max=45429, avg=12251.81, stdev=3890.07 clat percentiles (usec): | 1.00th=[ 6849], 5.00th=[ 7570], 10.00th=[ 8029], 20.00th=[ 8979], | 30.00th=[10028], 40.00th=[10945], 50.00th=[11863], 60.00th=[12649], | 70.00th=[13566], 80.00th=[14615], 90.00th=[16188], 95.00th=[19006], | 99.00th=[25560], 99.50th=[26084], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 192, max= 336, per=25.88%, avg=285.45, stdev=49.73, samples=11 iops : min= 48, max= 84, avg=71.18, stdev=12.42, samples=11 lat (usec) : 750=3.12%, 1000=16.54% lat (msec) : 2=16.02%, 4=9.64%, 10=17.71%, 20=35.16%, 50=1.82% cpu : usr=0.16%, sys=2.43%, ctx=923, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3230: Tue Jun 3 18:31:25 2025 read: IOPS=62, BW=252KiB/s (258kB/s)(1436KiB/5707msec) clat (usec): min=598, max=21714, avg=1859.37, stdev=2351.29 lat (usec): min=599, max=21715, avg=1860.18, stdev=2351.29 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 758], 10.00th=[ 807], 20.00th=[ 865], | 30.00th=[ 930], 40.00th=[ 996], 50.00th=[ 1139], 60.00th=[ 1385], | 70.00th=[ 1860], 80.00th=[ 2311], 90.00th=[ 2868], 95.00th=[ 3851], | 99.00th=[14222], 99.50th=[19530], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 136, max= 376, per=23.91%, avg=250.82, stdev=71.82, samples=11 iops : min= 34, max= 94, avg=62.64, stdev=17.93, samples=11 write: IOPS=71, BW=287KiB/s (294kB/s)(1636KiB/5707msec) clat (usec): min=6644, max=54206, avg=12299.13, stdev=3953.19 lat (usec): min=6645, max=54207, avg=12300.25, stdev=3953.02 clat percentiles (usec): | 1.00th=[ 6915], 5.00th=[ 7701], 10.00th=[ 8225], 20.00th=[ 9241], | 30.00th=[ 9896], 40.00th=[10945], 50.00th=[11863], 60.00th=[12780], | 70.00th=[13829], 80.00th=[14877], 90.00th=[16450], 95.00th=[18744], | 99.00th=[21890], 99.50th=[22676], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 216, max= 328, per=25.71%, avg=283.55, stdev=33.97, samples=11 iops : min= 54, max= 82, avg=70.82, stdev= 8.47, samples=11 lat (usec) : 750=2.21%, 1000=16.93% lat (msec) : 2=16.15%, 4=9.24%, 10=17.84%, 20=36.33%, 50=1.17% lat (msec) : 100=0.13% cpu : usr=0.16%, sys=2.37%, ctx=917, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3231: Tue Jun 3 18:31:25 2025 read: IOPS=74, BW=298KiB/s (305kB/s)(1564KiB/5250msec) clat (usec): min=639, max=24174, avg=1557.35, stdev=1638.87 lat (usec): min=640, max=24176, avg=1558.16, stdev=1638.91 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 766], 10.00th=[ 783], 20.00th=[ 832], | 30.00th=[ 881], 40.00th=[ 947], 50.00th=[ 1057], 60.00th=[ 1188], | 70.00th=[ 1418], 80.00th=[ 2212], 90.00th=[ 2737], 95.00th=[ 3163], | 99.00th=[ 9110], 99.50th=[11731], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 168, max= 424, per=28.96%, avg=303.80, stdev=95.62, samples=10 iops : min= 42, max= 106, avg=75.80, stdev=23.79, samples=10 write: IOPS=71, BW=287KiB/s (294kB/s)(1508KiB/5250msec) clat (usec): min=6763, max=67038, avg=12284.98, stdev=4724.45 lat (usec): min=6764, max=67038, avg=12286.18, stdev=4724.59 clat percentiles (usec): | 1.00th=[ 6980], 5.00th=[ 7439], 10.00th=[ 8160], 20.00th=[ 8979], | 30.00th=[10028], 40.00th=[10945], 50.00th=[11731], 60.00th=[12518], | 70.00th=[13435], 80.00th=[14353], 90.00th=[15795], 95.00th=[17957], | 99.00th=[29230], 99.50th=[38536], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 208, max= 328, per=26.02%, avg=287.00, stdev=44.51, samples=10 iops : min= 52, max= 82, avg=71.60, stdev=11.12, samples=10 lat (usec) : 750=1.43%, 1000=21.87% lat (msec) : 2=16.28%, 4=9.77%, 10=15.10%, 20=33.85%, 50=1.56% lat (msec) : 100=0.13% cpu : usr=0.19%, sys=2.55%, ctx=931, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3232: Tue Jun 3 18:31:25 2025 read: IOPS=67, BW=271KiB/s (278kB/s)(1508KiB/5564msec) clat (usec): min=636, max=24019, avg=1859.51, stdev=2617.02 lat (usec): min=637, max=24020, avg=1860.27, stdev=2617.03 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 742], 10.00th=[ 783], 20.00th=[ 840], | 30.00th=[ 906], 40.00th=[ 979], 50.00th=[ 1090], 60.00th=[ 1237], | 70.00th=[ 1582], 80.00th=[ 2114], 90.00th=[ 2802], 95.00th=[ 5473], | 99.00th=[15139], 99.50th=[21365], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 152, max= 368, per=25.71%, avg=269.73, stdev=68.59, samples=11 iops : min= 38, max= 92, avg=67.36, stdev=17.11, samples=11 write: IOPS=70, BW=281KiB/s (288kB/s)(1564KiB/5564msec) clat (usec): min=6733, max=33078, avg=12415.92, stdev=3858.62 lat (usec): min=6734, max=33079, avg=12416.92, stdev=3858.63 clat percentiles (usec): | 1.00th=[ 7308], 5.00th=[ 7898], 10.00th=[ 8291], 20.00th=[ 8979], | 30.00th=[ 9896], 40.00th=[11076], 50.00th=[12256], 60.00th=[12911], | 70.00th=[13698], 80.00th=[14746], 90.00th=[16319], 95.00th=[19006], | 99.00th=[29492], 99.50th=[33162], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 184, max= 336, per=25.51%, avg=281.36, stdev=43.70, samples=11 iops : min= 46, max= 84, avg=70.27, stdev=10.97, samples=11 lat (usec) : 750=2.73%, 1000=17.97% lat (msec) : 2=17.97%, 4=7.68%, 10=17.71%, 20=33.33%, 50=2.60% cpu : usr=0.23%, sys=2.34%, ctx=911, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1049KiB/s (1074kB/s), 252KiB/s-298KiB/s (258kB/s-305kB/s), io=5988KiB (6132kB), run=5250-5707msec WRITE: bw=1104KiB/s (1130kB/s), 281KiB/s-287KiB/s (288kB/s-294kB/s), io=6300KiB (6451kB), run=5250-5707msec