== sanity test 398b: DIO and buffer IO race ============== 19:33:50 (1736987630) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.0691654 s, 182 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=26127: Wed Jan 15 19:33:53 2025 read: IOPS=198, BW=794KiB/s (813kB/s)(1480KiB/1865msec) clat (usec): min=3, max=42908, avg=1286.99, stdev=2909.71 lat (usec): min=3, max=42908, avg=1287.61, stdev=2909.71 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 11], | 30.00th=[ 13], 40.00th=[ 19], 50.00th=[ 28], 60.00th=[ 1029], | 70.00th=[ 1663], 80.00th=[ 2311], 90.00th=[ 3130], 95.00th=[ 3785], | 99.00th=[12387], 99.50th=[14484], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 632, max= 1040, per=26.53%, avg=800.00, stdev=213.32, samples=3 iops : min= 158, max= 260, avg=200.00, stdev=53.33, samples=3 write: IOPS=213, BW=854KiB/s (874kB/s)(1592KiB/1865msec) clat (usec): min=89, max=28341, avg=3452.35, stdev=5101.58 lat (usec): min=90, max=28342, avg=3453.08, stdev=5101.62 clat percentiles (usec): | 1.00th=[ 97], 5.00th=[ 110], 10.00th=[ 121], 20.00th=[ 157], | 30.00th=[ 194], 40.00th=[ 494], 50.00th=[ 1450], 60.00th=[ 2040], | 70.00th=[ 2868], 80.00th=[ 6194], 90.00th=[11076], 95.00th=[15270], | 99.00th=[22414], 99.50th=[23725], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 784, max= 984, per=27.31%, avg=866.67, stdev=104.41, samples=3 iops : min= 196, max= 246, avg=216.67, stdev=26.10, samples=3 lat (usec) : 4=0.39%, 10=7.55%, 20=12.76%, 50=5.73%, 100=1.56% lat (usec) : 250=19.01%, 500=1.56%, 750=0.91%, 1000=1.82% lat (msec) : 2=16.02%, 4=17.45%, 10=8.85%, 20=5.21%, 50=1.17% cpu : usr=0.32%, sys=9.07%, ctx=932, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26128: Wed Jan 15 19:33:53 2025 read: IOPS=187, BW=749KiB/s (767kB/s)(1436KiB/1917msec) clat (usec): min=3, max=30149, avg=1469.25, stdev=3240.76 lat (usec): min=4, max=30149, avg=1469.97, stdev=3240.78 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 16], 50.00th=[ 33], 60.00th=[ 1156], | 70.00th=[ 1762], 80.00th=[ 2311], 90.00th=[ 3490], 95.00th=[ 4621], | 99.00th=[17171], 99.50th=[27132], 99.90th=[30278], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 656, max= 1072, per=26.50%, avg=799.33, stdev=236.24, samples=3 iops : min= 164, max= 268, avg=199.67, stdev=59.20, samples=3 write: IOPS=213, BW=853KiB/s (874kB/s)(1636KiB/1917msec) clat (usec): min=88, max=45016, avg=3362.82, stdev=5343.28 lat (usec): min=89, max=45016, avg=3364.65, stdev=5343.11 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 131], 10.00th=[ 145], 20.00th=[ 165], | 30.00th=[ 184], 40.00th=[ 302], 50.00th=[ 1123], 60.00th=[ 1860], | 70.00th=[ 2933], 80.00th=[ 5866], 90.00th=[10945], 95.00th=[13566], | 99.00th=[20841], 99.50th=[25297], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 648, max= 1304, per=28.47%, avg=903.33, stdev=351.32, samples=3 iops : min= 162, max= 326, avg=225.67, stdev=87.93, samples=3 lat (usec) : 4=0.13%, 10=8.98%, 20=11.33%, 50=4.43%, 100=1.30% lat (usec) : 250=20.57%, 500=1.17%, 750=1.30%, 1000=2.99% lat (msec) : 2=15.62%, 4=15.23%, 10=9.64%, 20=5.99%, 50=1.30% cpu : usr=0.42%, sys=8.92%, ctx=863, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26129: Wed Jan 15 19:33:53 2025 read: IOPS=196, BW=788KiB/s (807kB/s)(1564KiB/1985msec) clat (usec): min=4, max=29025, avg=1606.78, stdev=3187.86 lat (usec): min=4, max=29025, avg=1607.46, stdev=3187.86 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 15], 40.00th=[ 24], 50.00th=[ 151], 60.00th=[ 1582], | 70.00th=[ 2114], 80.00th=[ 2573], 90.00th=[ 3326], 95.00th=[ 4228], | 99.00th=[21103], 99.50th=[25560], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 616, max= 760, per=23.34%, avg=704.00, stdev=77.15, samples=3 iops : min= 154, max= 190, avg=176.00, stdev=19.29, samples=3 write: IOPS=189, BW=760KiB/s (778kB/s)(1508KiB/1985msec) clat (usec): min=94, max=38869, avg=3564.01, stdev=5431.41 lat (usec): min=94, max=38871, avg=3564.83, stdev=5431.41 clat percentiles (usec): | 1.00th=[ 108], 5.00th=[ 129], 10.00th=[ 149], 20.00th=[ 176], | 30.00th=[ 208], 40.00th=[ 465], 50.00th=[ 1221], 60.00th=[ 1876], | 70.00th=[ 2835], 80.00th=[ 6980], 90.00th=[10814], 95.00th=[13829], | 99.00th=[24249], 99.50th=[26870], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 624, max= 728, per=21.43%, avg=680.00, stdev=52.46, samples=3 iops : min= 156, max= 182, avg=170.00, stdev=13.11, samples=3 lat (usec) : 10=8.46%, 20=9.77%, 50=6.77%, 100=0.52%, 250=17.06% lat (usec) : 500=2.99%, 750=0.13%, 1000=2.86% lat (msec) : 2=16.28%, 4=19.27%, 10=9.64%, 20=4.17%, 50=2.08% cpu : usr=0.35%, sys=9.07%, ctx=959, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26130: Wed Jan 15 19:33:53 2025 read: IOPS=213, BW=853KiB/s (874kB/s)(1508KiB/1767msec) clat (usec): min=4, max=23673, avg=1376.36, stdev=3004.29 lat (usec): min=4, max=23675, avg=1377.38, stdev=3004.16 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 11], | 30.00th=[ 13], 40.00th=[ 17], 50.00th=[ 27], 60.00th=[ 306], | 70.00th=[ 1647], 80.00th=[ 2409], 90.00th=[ 3294], 95.00th=[ 4817], | 99.00th=[22152], 99.50th=[22414], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 616, max= 1176, per=29.09%, avg=877.33, stdev=281.86, samples=3 iops : min= 154, max= 294, avg=219.33, stdev=70.47, samples=3 write: IOPS=221, BW=885KiB/s (906kB/s)(1564KiB/1767msec) clat (usec): min=81, max=25298, avg=3152.58, stdev=4762.39 lat (usec): min=81, max=25300, avg=3153.74, stdev=4762.18 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 110], 10.00th=[ 139], 20.00th=[ 167], | 30.00th=[ 192], 40.00th=[ 306], 50.00th=[ 1074], 60.00th=[ 1827], | 70.00th=[ 2999], 80.00th=[ 5342], 90.00th=[ 9634], 95.00th=[13042], | 99.00th=[21627], 99.50th=[25035], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 688, max= 1200, per=28.91%, avg=917.33, stdev=260.13, samples=3 iops : min= 172, max= 300, avg=229.33, stdev=65.03, samples=3 lat (usec) : 10=9.64%, 20=12.50%, 50=5.86%, 100=1.95%, 250=19.01% lat (usec) : 500=1.95%, 750=0.78%, 1000=3.12% lat (msec) : 2=13.80%, 4=16.15%, 10=9.77%, 20=3.91%, 50=1.56% cpu : usr=0.34%, sys=10.48%, ctx=822, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3017KiB/s (3089kB/s), 749KiB/s-853KiB/s (767kB/s-874kB/s), io=5988KiB (6132kB), run=1767-1985msec WRITE: bw=3174KiB/s (3250kB/s), 760KiB/s-885KiB/s (778kB/s-906kB/s), io=6300KiB (6451kB), run=1767-1985msec rand-rw: (groupid=0, jobs=1): err= 0: pid=26132: Wed Jan 15 19:34:00 2025 read: IOPS=43, BW=174KiB/s (178kB/s)(1480KiB/8504msec) clat (usec): min=812, max=36423, avg=2585.91, stdev=3856.72 lat (usec): min=813, max=36424, avg=2587.16, stdev=3856.63 clat percentiles (usec): | 1.00th=[ 832], 5.00th=[ 906], 10.00th=[ 955], 20.00th=[ 1045], | 30.00th=[ 1172], 40.00th=[ 1303], 50.00th=[ 1450], 60.00th=[ 1680], | 70.00th=[ 1909], 80.00th=[ 2704], 90.00th=[ 3752], 95.00th=[10290], | 99.00th=[21890], 99.50th=[27132], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 104, max= 272, per=24.96%, avg=173.50, stdev=49.81, samples=16 iops : min= 26, max= 68, avg=43.38, stdev=12.45, samples=16 write: IOPS=46, BW=187KiB/s (192kB/s)(1592KiB/8504msec) clat (usec): min=9860, max=53123, avg=18936.48, stdev=6228.64 lat (usec): min=9861, max=53124, avg=18938.03, stdev=6228.70 clat percentiles (usec): | 1.00th=[10552], 5.00th=[11600], 10.00th=[12125], 20.00th=[13435], | 30.00th=[15139], 40.00th=[16712], 50.00th=[18744], 60.00th=[20055], | 70.00th=[21103], 80.00th=[22676], 90.00th=[24249], 95.00th=[30540], | 99.00th=[46400], 99.50th=[46400], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 128, max= 216, per=25.44%, avg=186.00, stdev=28.70, samples=16 iops : min= 32, max= 54, avg=46.50, stdev= 7.17, samples=16 lat (usec) : 1000=7.03% lat (msec) : 2=27.34%, 4=9.77%, 10=1.69%, 20=33.07%, 50=20.96% lat (msec) : 100=0.13% cpu : usr=0.25%, sys=1.71%, ctx=894, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26133: Wed Jan 15 19:34:00 2025 read: IOPS=41, BW=167KiB/s (171kB/s)(1436KiB/8608msec) clat (usec): min=757, max=35081, avg=2771.89, stdev=4206.30 lat (usec): min=758, max=35082, avg=2772.99, stdev=4206.25 clat percentiles (usec): | 1.00th=[ 840], 5.00th=[ 930], 10.00th=[ 988], 20.00th=[ 1090], | 30.00th=[ 1172], 40.00th=[ 1319], 50.00th=[ 1500], 60.00th=[ 1811], | 70.00th=[ 2180], 80.00th=[ 2900], 90.00th=[ 4113], 95.00th=[10945], | 99.00th=[25035], 99.50th=[29230], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 104, max= 240, per=23.83%, avg=165.65, stdev=40.77, samples=17 iops : min= 26, max= 60, avg=41.41, stdev=10.19, samples=17 write: IOPS=47, BW=190KiB/s (195kB/s)(1636KiB/8608msec) clat (usec): min=9831, max=54921, avg=18584.52, stdev=5779.37 lat (usec): min=9832, max=54922, avg=18585.88, stdev=5779.24 clat percentiles (usec): | 1.00th=[10552], 5.00th=[11338], 10.00th=[11994], 20.00th=[13304], | 30.00th=[14746], 40.00th=[16712], 50.00th=[18220], 60.00th=[19792], | 70.00th=[21103], 80.00th=[22414], 90.00th=[24773], 95.00th=[27657], | 99.00th=[36439], 99.50th=[38536], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 144, max= 232, per=25.75%, avg=188.24, stdev=24.19, samples=17 iops : min= 36, max= 58, avg=47.06, stdev= 6.05, samples=17 lat (usec) : 1000=4.95% lat (msec) : 2=25.13%, 4=11.85%, 10=2.47%, 20=34.37%, 50=21.09% lat (msec) : 100=0.13% cpu : usr=0.19%, sys=1.84%, ctx=904, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26134: Wed Jan 15 19:34:00 2025 read: IOPS=47, BW=190KiB/s (194kB/s)(1564KiB/8250msec) clat (usec): min=735, max=27181, avg=2457.09, stdev=3098.74 lat (usec): min=736, max=27182, avg=2458.12, stdev=3098.86 clat percentiles (usec): | 1.00th=[ 807], 5.00th=[ 881], 10.00th=[ 955], 20.00th=[ 1074], | 30.00th=[ 1172], 40.00th=[ 1336], 50.00th=[ 1516], 60.00th=[ 1647], | 70.00th=[ 2008], 80.00th=[ 2671], 90.00th=[ 3949], 95.00th=[10421], | 99.00th=[16188], 99.50th=[21103], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 104, max= 343, per=27.68%, avg=192.38, stdev=72.21, samples=16 iops : min= 26, max= 85, avg=48.00, stdev=17.95, samples=16 write: IOPS=45, BW=183KiB/s (187kB/s)(1508KiB/8250msec) clat (usec): min=10304, max=55184, avg=19307.41, stdev=6303.55 lat (usec): min=10305, max=55185, avg=19308.63, stdev=6303.56 clat percentiles (usec): | 1.00th=[10552], 5.00th=[11600], 10.00th=[12387], 20.00th=[13698], | 30.00th=[15008], 40.00th=[17957], 50.00th=[19268], 60.00th=[20579], | 70.00th=[21627], 80.00th=[23200], 90.00th=[25560], 95.00th=[28443], | 99.00th=[45351], 99.50th=[51119], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 128, max= 216, per=25.02%, avg=182.88, stdev=25.42, samples=16 iops : min= 32, max= 54, avg=45.62, stdev= 6.34, samples=16 lat (usec) : 750=0.26%, 1000=6.51% lat (msec) : 2=28.65%, 4=10.55%, 10=2.21%, 20=30.60%, 50=20.96% lat (msec) : 100=0.26% cpu : usr=0.07%, sys=2.00%, ctx=919, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26135: Wed Jan 15 19:34:00 2025 read: IOPS=44, BW=180KiB/s (184kB/s)(1508KiB/8388msec) clat (usec): min=754, max=46510, avg=2738.77, stdev=3996.55 lat (usec): min=755, max=46511, avg=2739.83, stdev=3996.54 clat percentiles (usec): | 1.00th=[ 840], 5.00th=[ 906], 10.00th=[ 963], 20.00th=[ 1074], | 30.00th=[ 1188], 40.00th=[ 1336], 50.00th=[ 1582], 60.00th=[ 1811], | 70.00th=[ 2212], 80.00th=[ 2999], 90.00th=[ 3949], 95.00th=[11469], | 99.00th=[19268], 99.50th=[22938], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 104, max= 296, per=25.52%, avg=177.38, stdev=53.60, samples=16 iops : min= 26, max= 74, avg=44.25, stdev=13.49, samples=16 write: IOPS=46, BW=186KiB/s (191kB/s)(1564KiB/8388msec) clat (usec): min=8878, max=40477, avg=18784.14, stdev=5379.63 lat (usec): min=8879, max=40477, avg=18785.40, stdev=5379.66 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11731], 10.00th=[12387], 20.00th=[13435], | 30.00th=[14877], 40.00th=[17433], 50.00th=[19268], 60.00th=[20317], | 70.00th=[21103], 80.00th=[22414], 90.00th=[24511], 95.00th=[27132], | 99.00th=[36963], 99.50th=[40633], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 136, max= 224, per=25.22%, avg=184.38, stdev=25.26, samples=16 iops : min= 34, max= 56, avg=46.00, stdev= 6.36, samples=16 lat (usec) : 1000=6.77% lat (msec) : 2=25.13%, 4=12.37%, 10=2.08%, 20=31.77%, 50=21.87% cpu : usr=0.13%, sys=1.94%, ctx=931, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=696KiB/s (712kB/s), 167KiB/s-190KiB/s (171kB/s-194kB/s), io=5988KiB (6132kB), run=8250-8608msec WRITE: bw=732KiB/s (749kB/s), 183KiB/s-190KiB/s (187kB/s-195kB/s), io=6300KiB (6451kB), run=8250-8608msec