== sanity test 398b: DIO and buffer IO race ============== 11:36:12 (1736267772) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.0685166 s, 184 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24834: Tue Jan 7 11:36:15 2025 read: IOPS=180, BW=722KiB/s (740kB/s)(1480KiB/2049msec) clat (usec): min=3, max=18918, avg=1453.28, stdev=2503.05 lat (usec): min=4, max=18919, avg=1453.90, stdev=2503.07 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 14], 40.00th=[ 21], 50.00th=[ 38], 60.00th=[ 1303], | 70.00th=[ 1975], 80.00th=[ 2474], 90.00th=[ 3687], 95.00th=[ 5014], | 99.00th=[15533], 99.50th=[16450], 99.90th=[19006], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 624, max= 784, per=24.44%, avg=714.00, stdev=66.41, samples=4 iops : min= 156, max= 196, avg=178.50, stdev=16.60, samples=4 write: IOPS=194, BW=777KiB/s (796kB/s)(1592KiB/2049msec) clat (usec): min=97, max=39253, avg=3765.78, stdev=5470.80 lat (usec): min=98, max=39253, avg=3766.50, stdev=5470.80 clat percentiles (usec): | 1.00th=[ 111], 5.00th=[ 133], 10.00th=[ 145], 20.00th=[ 172], | 30.00th=[ 306], 40.00th=[ 1156], 50.00th=[ 1729], 60.00th=[ 2311], | 70.00th=[ 3228], 80.00th=[ 5932], 90.00th=[11863], 95.00th=[15533], | 99.00th=[23462], 99.50th=[33424], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 624, max= 856, per=25.18%, avg=774.00, stdev=109.76, samples=4 iops : min= 156, max= 214, avg=193.50, stdev=27.44, samples=4 lat (usec) : 4=0.13%, 10=7.68%, 20=10.94%, 50=5.99%, 100=0.65% lat (usec) : 250=15.10%, 500=1.17%, 750=0.39%, 1000=3.39% lat (msec) : 2=16.93%, 4=20.18%, 10=10.68%, 20=5.60%, 50=1.17% cpu : usr=0.29%, sys=8.35%, ctx=967, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24836: Tue Jan 7 11:36:15 2025 read: IOPS=180, BW=723KiB/s (740kB/s)(1436KiB/1986msec) clat (usec): min=3, max=21266, avg=1566.29, stdev=2556.32 lat (usec): min=3, max=21267, avg=1567.19, stdev=2556.48 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 9], 20.00th=[ 12], | 30.00th=[ 16], 40.00th=[ 27], 50.00th=[ 930], 60.00th=[ 1467], | 70.00th=[ 2008], 80.00th=[ 2737], 90.00th=[ 3589], 95.00th=[ 4490], | 99.00th=[13173], 99.50th=[21103], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 656, max= 888, per=25.55%, avg=746.67, stdev=124.02, samples=3 iops : min= 164, max= 222, avg=186.67, stdev=31.01, samples=3 write: IOPS=205, BW=824KiB/s (844kB/s)(1636KiB/1986msec) clat (usec): min=89, max=39655, avg=3458.83, stdev=4845.99 lat (usec): min=89, max=39656, avg=3459.81, stdev=4846.13 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 126], 10.00th=[ 145], 20.00th=[ 169], | 30.00th=[ 251], 40.00th=[ 1045], 50.00th=[ 1614], 60.00th=[ 2409], | 70.00th=[ 3556], 80.00th=[ 6128], 90.00th=[10290], 95.00th=[12780], | 99.00th=[19530], 99.50th=[26084], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 752, max= 888, per=26.11%, avg=802.67, stdev=74.33, samples=3 iops : min= 188, max= 222, avg=200.67, stdev=18.58, samples=3 lat (usec) : 4=0.26%, 10=6.51%, 20=9.90%, 50=4.56%, 100=1.82% lat (usec) : 250=15.23%, 500=1.56%, 750=0.91%, 1000=3.65% lat (msec) : 2=17.45%, 4=19.40%, 10=12.37%, 20=5.60%, 50=0.78% cpu : usr=0.05%, sys=9.87%, ctx=1022, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24837: Tue Jan 7 11:36:15 2025 read: IOPS=198, BW=794KiB/s (813kB/s)(1564KiB/1969msec) clat (usec): min=3, max=36939, avg=1624.40, stdev=3529.43 lat (usec): min=3, max=36941, avg=1625.02, stdev=3529.46 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 14], 40.00th=[ 18], 50.00th=[ 34], 60.00th=[ 1352], | 70.00th=[ 1975], 80.00th=[ 2573], 90.00th=[ 3425], 95.00th=[ 4490], | 99.00th=[19268], 99.50th=[28967], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 576, max= 832, per=25.28%, avg=738.67, stdev=141.38, samples=3 iops : min= 144, max= 208, avg=184.67, stdev=35.35, samples=3 write: IOPS=191, BW=766KiB/s (784kB/s)(1508KiB/1969msec) clat (usec): min=92, max=38072, avg=3517.22, stdev=5224.68 lat (usec): min=93, max=38072, avg=3517.99, stdev=5224.66 clat percentiles (usec): | 1.00th=[ 108], 5.00th=[ 137], 10.00th=[ 149], 20.00th=[ 167], | 30.00th=[ 237], 40.00th=[ 947], 50.00th=[ 1303], 60.00th=[ 2180], | 70.00th=[ 3392], 80.00th=[ 5735], 90.00th=[10159], 95.00th=[13960], | 99.00th=[27657], 99.50th=[31065], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 688, max= 728, per=23.08%, avg=709.33, stdev=20.13, samples=3 iops : min= 172, max= 182, avg=177.33, stdev= 5.03, samples=3 lat (usec) : 4=0.26%, 10=8.85%, 20=12.63%, 50=4.56%, 100=1.17% lat (usec) : 250=15.23%, 500=1.30%, 750=1.04%, 1000=3.91% lat (msec) : 2=15.10%, 4=19.79%, 10=9.64%, 20=5.34%, 50=1.17% cpu : usr=0.46%, sys=8.84%, ctx=931, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24838: Tue Jan 7 11:36:15 2025 read: IOPS=191, BW=767KiB/s (786kB/s)(1508KiB/1965msec) clat (usec): min=3, max=24061, avg=1520.68, stdev=2926.96 lat (usec): min=3, max=24061, avg=1521.28, stdev=2926.97 clat percentiles (usec): | 1.00th=[ 4], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 11], | 30.00th=[ 13], 40.00th=[ 18], 50.00th=[ 33], 60.00th=[ 1336], | 70.00th=[ 2057], 80.00th=[ 2507], 90.00th=[ 3392], 95.00th=[ 4883], | 99.00th=[17171], 99.50th=[22152], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 624, max= 734, per=23.80%, avg=695.33, stdev=61.85, samples=3 iops : min= 156, max= 183, avg=173.67, stdev=15.31, samples=3 write: IOPS=198, BW=796KiB/s (815kB/s)(1564KiB/1965msec) clat (usec): min=95, max=37949, avg=3540.44, stdev=5071.54 lat (usec): min=96, max=37950, avg=3541.30, stdev=5071.52 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 127], 10.00th=[ 141], 20.00th=[ 165], | 30.00th=[ 247], 40.00th=[ 963], 50.00th=[ 1450], 60.00th=[ 2114], | 70.00th=[ 3195], 80.00th=[ 6390], 90.00th=[10290], 95.00th=[14877], | 99.00th=[21103], 99.50th=[27919], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 670, max= 768, per=23.49%, avg=722.00, stdev=49.27, samples=3 iops : min= 167, max= 192, avg=180.33, stdev=12.58, samples=3 lat (usec) : 4=0.78%, 10=8.72%, 20=11.07%, 50=5.47%, 100=1.04% lat (usec) : 250=15.36%, 500=2.08%, 750=0.78%, 1000=2.86% lat (msec) : 2=15.49%, 4=19.40%, 10=9.90%, 20=5.86%, 50=1.17% cpu : usr=0.36%, sys=9.42%, ctx=964, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2922KiB/s (2993kB/s), 722KiB/s-794KiB/s (740kB/s-813kB/s), io=5988KiB (6132kB), run=1965-2049msec WRITE: bw=3075KiB/s (3148kB/s), 766KiB/s-824KiB/s (784kB/s-844kB/s), io=6300KiB (6451kB), run=1965-2049msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24831: Tue Jan 7 11:36:21 2025 read: IOPS=48, BW=194KiB/s (198kB/s)(1480KiB/7640msec) clat (usec): min=696, max=24244, avg=2239.79, stdev=3130.82 lat (usec): min=696, max=24245, avg=2240.84, stdev=3130.84 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 857], 10.00th=[ 930], 20.00th=[ 1020], | 30.00th=[ 1090], 40.00th=[ 1221], 50.00th=[ 1352], 60.00th=[ 1467], | 70.00th=[ 1680], 80.00th=[ 1991], 90.00th=[ 3425], 95.00th=[ 8225], | 99.00th=[19792], 99.50th=[23987], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 128, max= 304, per=24.95%, avg=190.87, stdev=48.60, samples=15 iops : min= 32, max= 76, avg=47.67, stdev=12.12, samples=15 write: IOPS=52, BW=208KiB/s (213kB/s)(1592KiB/7640msec) clat (usec): min=8638, max=48681, avg=17075.37, stdev=5487.24 lat (usec): min=8641, max=48682, avg=17076.68, stdev=5487.26 clat percentiles (usec): | 1.00th=[ 9503], 5.00th=[10552], 10.00th=[11338], 20.00th=[12649], | 30.00th=[13698], 40.00th=[14877], 50.00th=[16188], 60.00th=[17695], | 70.00th=[19268], 80.00th=[20579], 90.00th=[22938], 95.00th=[25822], | 99.00th=[40633], 99.50th=[43779], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 128, max= 240, per=25.70%, avg=206.87, stdev=32.97, samples=15 iops : min= 32, max= 60, avg=51.67, stdev= 8.24, samples=15 lat (usec) : 750=0.39%, 1000=8.33% lat (msec) : 2=29.82%, 4=5.08%, 10=4.17%, 20=39.58%, 50=12.63% cpu : usr=0.27%, sys=1.91%, ctx=814, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24832: Tue Jan 7 11:36:21 2025 read: IOPS=45, BW=184KiB/s (188kB/s)(1436KiB/7821msec) clat (usec): min=675, max=25485, avg=2245.69, stdev=2971.51 lat (usec): min=676, max=25487, avg=2246.71, stdev=2971.50 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 840], 10.00th=[ 889], 20.00th=[ 1029], | 30.00th=[ 1123], 40.00th=[ 1254], 50.00th=[ 1385], 60.00th=[ 1549], | 70.00th=[ 1680], 80.00th=[ 2057], 90.00th=[ 3425], 95.00th=[ 9765], | 99.00th=[16450], 99.50th=[18744], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 80, max= 272, per=23.89%, avg=182.73, stdev=52.74, samples=15 iops : min= 20, max= 68, avg=45.53, stdev=13.08, samples=15 write: IOPS=52, BW=209KiB/s (214kB/s)(1636KiB/7821msec) clat (usec): min=8463, max=43153, avg=17112.28, stdev=5551.32 lat (usec): min=8464, max=43154, avg=17113.57, stdev=5551.35 clat percentiles (usec): | 1.00th=[ 9503], 5.00th=[10421], 10.00th=[11207], 20.00th=[12387], | 30.00th=[13304], 40.00th=[15533], 50.00th=[16581], 60.00th=[17695], | 70.00th=[19006], 80.00th=[20841], 90.00th=[22676], 95.00th=[26346], | 99.00th=[37487], 99.50th=[41157], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 143, max= 248, per=25.35%, avg=204.07, stdev=36.84, samples=15 iops : min= 35, max= 62, avg=50.87, stdev= 9.27, samples=15 lat (usec) : 750=0.39%, 1000=8.20% lat (msec) : 2=28.12%, 4=5.73%, 10=4.56%, 20=40.89%, 50=12.11% cpu : usr=0.24%, sys=1.89%, ctx=818, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24833: Tue Jan 7 11:36:21 2025 read: IOPS=53, BW=213KiB/s (218kB/s)(1564KiB/7334msec) clat (usec): min=695, max=34317, avg=1942.19, stdev=2767.16 lat (usec): min=696, max=34318, avg=1943.20, stdev=2767.20 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 857], 10.00th=[ 906], 20.00th=[ 996], | 30.00th=[ 1074], 40.00th=[ 1156], 50.00th=[ 1254], 60.00th=[ 1369], | 70.00th=[ 1549], 80.00th=[ 1844], 90.00th=[ 2737], 95.00th=[ 5276], | 99.00th=[16188], 99.50th=[18744], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 112, max= 360, per=28.38%, avg=217.14, stdev=78.37, samples=14 iops : min= 28, max= 90, avg=54.29, stdev=19.59, samples=14 write: IOPS=51, BW=206KiB/s (211kB/s)(1508KiB/7334msec) clat (usec): min=9005, max=57305, avg=17399.23, stdev=5868.88 lat (usec): min=9006, max=57306, avg=17400.46, stdev=5868.87 clat percentiles (usec): | 1.00th=[ 9503], 5.00th=[10814], 10.00th=[11731], 20.00th=[12911], | 30.00th=[13960], 40.00th=[15270], 50.00th=[16450], 60.00th=[17433], | 70.00th=[18744], 80.00th=[20579], 90.00th=[22676], 95.00th=[27395], | 99.00th=[41157], 99.50th=[50070], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 144, max= 232, per=25.48%, avg=205.14, stdev=31.00, samples=14 iops : min= 36, max= 58, avg=51.29, stdev= 7.75, samples=14 lat (usec) : 750=0.26%, 1000=10.29% lat (msec) : 2=32.42%, 4=4.43%, 10=2.99%, 20=37.50%, 50=11.98% lat (msec) : 100=0.13% cpu : usr=0.10%, sys=2.15%, ctx=836, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24835: Tue Jan 7 11:36:21 2025 read: IOPS=50, BW=200KiB/s (205kB/s)(1508KiB/7540msec) clat (usec): min=667, max=24217, avg=2080.56, stdev=2893.20 lat (usec): min=667, max=24218, avg=2081.53, stdev=2893.21 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 840], 10.00th=[ 898], 20.00th=[ 988], | 30.00th=[ 1090], 40.00th=[ 1188], 50.00th=[ 1287], 60.00th=[ 1385], | 70.00th=[ 1532], 80.00th=[ 1860], 90.00th=[ 2933], 95.00th=[ 8160], | 99.00th=[16581], 99.50th=[18220], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 112, max= 320, per=25.99%, avg=198.80, stdev=55.88, samples=15 iops : min= 28, max= 80, avg=49.60, stdev=14.01, samples=15 write: IOPS=51, BW=207KiB/s (212kB/s)(1564KiB/7540msec) clat (usec): min=8432, max=47291, avg=17232.38, stdev=5449.98 lat (usec): min=8434, max=47292, avg=17233.87, stdev=5450.04 clat percentiles (usec): | 1.00th=[ 9634], 5.00th=[10683], 10.00th=[11600], 20.00th=[12780], | 30.00th=[13960], 40.00th=[15533], 50.00th=[16909], 60.00th=[17957], | 70.00th=[19268], 80.00th=[20579], 90.00th=[22152], 95.00th=[25560], | 99.00th=[40633], 99.50th=[43779], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 136, max= 256, per=25.76%, avg=207.33, stdev=36.31, samples=15 iops : min= 34, max= 64, avg=51.73, stdev= 9.06, samples=15 lat (usec) : 750=0.52%, 1000=10.16% lat (msec) : 2=30.08%, 4=4.95%, 10=2.73%, 20=39.06%, 50=12.50% cpu : usr=0.15%, sys=2.06%, ctx=827, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=766KiB/s (784kB/s), 184KiB/s-213KiB/s (188kB/s-218kB/s), io=5988KiB (6132kB), run=7334-7821msec WRITE: bw=806KiB/s (825kB/s), 206KiB/s-209KiB/s (211kB/s-214kB/s), io=6300KiB (6451kB), run=7334-7821msec