== sanity test 398b: DIO and buffer IO race ============== 15:36:14 (1734122174) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.40399 s, 35.8 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31578: Fri Dec 13 15:36:31 2024 read: IOPS=113, BW=452KiB/s (463kB/s)(5992KiB/13252msec) clat (usec): min=48, max=179622, avg=4207.65, stdev=5710.24 lat (usec): min=48, max=179622, avg=4208.34, stdev=5710.24 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 64], 10.00th=[ 77], 20.00th=[ 2008], | 30.00th=[ 2540], 40.00th=[ 2900], 50.00th=[ 3392], 60.00th=[ 4015], | 70.00th=[ 4948], 80.00th=[ 6325], 90.00th=[ 8029], 95.00th=[ 9896], | 99.00th=[ 15008], 99.50th=[ 21365], 99.90th=[ 33162], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 272, max= 648, per=25.11%, avg=450.04, stdev=74.19, samples=26 iops : min= 68, max= 162, avg=112.42, stdev=18.56, samples=26 write: IOPS=118, BW=475KiB/s (487kB/s)(6296KiB/13252msec) clat (usec): min=436, max=39419, avg=4333.47, stdev=4163.28 lat (usec): min=437, max=39420, avg=4334.56, stdev=4163.49 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 529], 10.00th=[ 562], 20.00th=[ 709], | 30.00th=[ 996], 40.00th=[ 2212], 50.00th=[ 3523], 60.00th=[ 4424], | 70.00th=[ 5473], 80.00th=[ 7046], 90.00th=[ 9765], 95.00th=[12125], | 99.00th=[18744], 99.50th=[21365], 99.90th=[31851], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 352, max= 656, per=25.17%, avg=472.15, stdev=88.73, samples=26 iops : min= 88, max= 164, avg=117.96, stdev=22.21, samples=26 lat (usec) : 50=0.03%, 100=6.77%, 250=1.63%, 500=0.62%, 750=10.84% lat (usec) : 1000=4.30% lat (msec) : 2=5.50%, 4=27.70%, 10=35.51%, 20=6.48%, 50=0.59% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=24.13%, ctx=6251, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31579: Fri Dec 13 15:36:31 2024 read: IOPS=110, BW=443KiB/s (454kB/s)(5940KiB/13397msec) clat (usec): min=48, max=48901, avg=4065.88, stdev=3494.86 lat (usec): min=49, max=48902, avg=4066.68, stdev=3494.69 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 62], 10.00th=[ 74], 20.00th=[ 1975], | 30.00th=[ 2474], 40.00th=[ 2966], 50.00th=[ 3425], 60.00th=[ 3982], | 70.00th=[ 5145], 80.00th=[ 6325], 90.00th=[ 8094], 95.00th=[ 9765], | 99.00th=[15139], 99.50th=[19006], 99.90th=[34866], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 208, max= 688, per=24.65%, avg=441.65, stdev=96.35, samples=26 iops : min= 52, max= 172, avg=110.31, stdev=24.10, samples=26 write: IOPS=118, BW=474KiB/s (485kB/s)(6348KiB/13397msec) clat (usec): min=429, max=175913, avg=4558.98, stdev=6107.27 lat (usec): min=430, max=175915, avg=4560.26, stdev=6107.13 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 676], | 30.00th=[ 963], 40.00th=[ 1975], 50.00th=[ 3589], 60.00th=[ 4555], | 70.00th=[ 5866], 80.00th=[ 7701], 90.00th=[ 10159], 95.00th=[ 11994], | 99.00th=[ 16712], 99.50th=[ 21890], 99.90th=[ 47449], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 256, max= 728, per=24.85%, avg=466.27, stdev=99.41, samples=26 iops : min= 64, max= 182, avg=116.46, stdev=24.88, samples=26 lat (usec) : 50=0.23%, 100=7.26%, 250=1.53%, 500=0.49%, 750=11.43% lat (usec) : 1000=4.20% lat (msec) : 2=5.37%, 4=26.60%, 10=35.22%, 20=7.13%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=23.89%, ctx=6328, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31580: Fri Dec 13 15:36:31 2024 read: IOPS=112, BW=450KiB/s (461kB/s)(6028KiB/13386msec) clat (usec): min=48, max=198674, avg=4028.57, stdev=5996.72 lat (usec): min=48, max=198675, avg=4029.24, stdev=5996.73 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 62], 10.00th=[ 74], 20.00th=[ 143], | 30.00th=[ 2409], 40.00th=[ 2868], 50.00th=[ 3359], 60.00th=[ 3916], | 70.00th=[ 4752], 80.00th=[ 6194], 90.00th=[ 8094], 95.00th=[ 9634], | 99.00th=[ 12780], 99.50th=[ 19268], 99.90th=[ 34866], 99.95th=[198181], | 99.99th=[198181] bw ( KiB/s): min= 256, max= 624, per=24.70%, avg=442.62, stdev=85.29, samples=26 iops : min= 64, max= 156, avg=110.54, stdev=21.39, samples=26 write: IOPS=116, BW=468KiB/s (479kB/s)(6260KiB/13386msec) clat (usec): min=432, max=57414, avg=4597.06, stdev=4355.50 lat (usec): min=433, max=57414, avg=4597.84, stdev=4355.52 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 537], 10.00th=[ 553], 20.00th=[ 685], | 30.00th=[ 988], 40.00th=[ 2474], 50.00th=[ 3818], 60.00th=[ 4817], | 70.00th=[ 6128], 80.00th=[ 7635], 90.00th=[10159], 95.00th=[12649], | 99.00th=[18482], 99.50th=[21103], 99.90th=[26870], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 312, max= 600, per=24.43%, avg=458.23, stdev=70.52, samples=26 iops : min= 78, max= 150, avg=114.46, stdev=17.66, samples=26 lat (usec) : 50=0.13%, 100=8.24%, 250=1.69%, 500=0.65%, 750=11.07% lat (usec) : 1000=3.91% lat (msec) : 2=4.39%, 4=26.04%, 10=36.39%, 20=6.93%, 50=0.49% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.19%, sys=23.87%, ctx=6090, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31581: Fri Dec 13 15:36:31 2024 read: IOPS=113, BW=452KiB/s (463kB/s)(6052KiB/13389msec) clat (usec): min=48, max=23974, avg=3965.31, stdev=3171.82 lat (usec): min=48, max=23975, avg=3966.02, stdev=3171.80 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 62], 10.00th=[ 73], 20.00th=[ 1745], | 30.00th=[ 2376], 40.00th=[ 2900], 50.00th=[ 3392], 60.00th=[ 4015], | 70.00th=[ 4948], 80.00th=[ 6456], 90.00th=[ 8291], 95.00th=[ 9634], | 99.00th=[13304], 99.50th=[16909], 99.90th=[22938], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 248, max= 656, per=24.70%, avg=442.62, stdev=102.09, samples=26 iops : min= 62, max= 164, avg=110.58, stdev=25.55, samples=26 write: IOPS=116, BW=466KiB/s (477kB/s)(6236KiB/13389msec) clat (usec): min=441, max=171169, avg=4662.84, stdev=6075.75 lat (usec): min=441, max=171170, avg=4664.25, stdev=6075.62 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 529], 10.00th=[ 562], 20.00th=[ 717], | 30.00th=[ 996], 40.00th=[ 2245], 50.00th=[ 3523], 60.00th=[ 4686], | 70.00th=[ 5997], 80.00th=[ 7635], 90.00th=[ 10421], 95.00th=[ 13173], | 99.00th=[ 19530], 99.50th=[ 24773], 99.90th=[ 30278], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 304, max= 632, per=24.42%, avg=458.04, stdev=85.56, samples=26 iops : min= 76, max= 158, avg=114.42, stdev=21.46, samples=26 lat (usec) : 50=0.20%, 100=7.52%, 250=1.82%, 500=0.68%, 750=10.77% lat (usec) : 1000=4.04% lat (msec) : 2=5.31%, 4=26.60%, 10=35.51%, 20=7.00%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=23.94%, ctx=6402, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1792KiB/s (1835kB/s), 443KiB/s-452KiB/s (454kB/s-463kB/s), io=23.4MiB (24.6MB), run=13252-13397msec WRITE: bw=1877KiB/s (1922kB/s), 466KiB/s-475KiB/s (477kB/s-487kB/s), io=24.6MiB (25.7MB), run=13252-13397msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31582: Fri Dec 13 15:36:53 2024 read: IOPS=42, BW=170KiB/s (174kB/s)(5992KiB/35185msec) clat (usec): min=1607, max=184294, avg=3978.37, stdev=5614.95 lat (usec): min=1608, max=184297, avg=3979.18, stdev=5614.98 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1893], 10.00th=[ 1975], 20.00th=[ 2089], | 30.00th=[ 2212], 40.00th=[ 2376], 50.00th=[ 2704], 60.00th=[ 3195], | 70.00th=[ 3720], 80.00th=[ 4424], 90.00th=[ 7504], 95.00th=[ 10290], | 99.00th=[ 18482], 99.50th=[ 20055], 99.90th=[ 38011], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 47, max= 303, per=25.10%, avg=169.67, stdev=64.26, samples=70 iops : min= 11, max= 75, avg=42.30, stdev=16.06, samples=70 write: IOPS=44, BW=179KiB/s (183kB/s)(6296KiB/35185msec) clat (usec): min=8360, max=70316, avg=18493.98, stdev=7170.97 lat (usec): min=8361, max=70317, avg=18494.90, stdev=7170.96 clat percentiles (usec): | 1.00th=[ 9634], 5.00th=[10814], 10.00th=[11600], 20.00th=[12911], | 30.00th=[14091], 40.00th=[15401], 50.00th=[17171], 60.00th=[18482], | 70.00th=[19792], 80.00th=[22152], 90.00th=[28181], 95.00th=[32900], | 99.00th=[42730], 99.50th=[47449], 99.90th=[65799], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 48, max= 247, per=25.19%, avg=178.36, stdev=53.20, samples=70 iops : min= 12, max= 61, avg=44.47, stdev=13.28, samples=70 lat (msec) : 2=5.92%, 4=30.63%, 10=10.45%, 20=37.96%, 50=14.84% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.09%, sys=5.69%, ctx=3628, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31583: Fri Dec 13 15:36:53 2024 read: IOPS=41, BW=167KiB/s (171kB/s)(5940KiB/35485msec) clat (usec): min=1614, max=173305, avg=3900.39, stdev=5264.58 lat (usec): min=1615, max=173307, avg=3901.15, stdev=5264.61 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1893], 10.00th=[ 1975], 20.00th=[ 2114], | 30.00th=[ 2245], 40.00th=[ 2442], 50.00th=[ 2704], 60.00th=[ 3195], | 70.00th=[ 3752], 80.00th=[ 4424], 90.00th=[ 7177], 95.00th=[ 10159], | 99.00th=[ 15270], 99.50th=[ 17433], 99.90th=[ 37487], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 16, max= 336, per=24.74%, avg=167.24, stdev=72.01, samples=70 iops : min= 4, max= 84, avg=41.76, stdev=18.02, samples=70 write: IOPS=44, BW=179KiB/s (183kB/s)(6348KiB/35485msec) clat (usec): min=8670, max=66219, avg=18636.05, stdev=7088.38 lat (usec): min=8671, max=66220, avg=18636.94, stdev=7088.38 clat percentiles (usec): | 1.00th=[ 9896], 5.00th=[10814], 10.00th=[11731], 20.00th=[13042], | 30.00th=[14222], 40.00th=[15664], 50.00th=[17171], 60.00th=[18482], | 70.00th=[20055], 80.00th=[22676], 90.00th=[28443], 95.00th=[32637], | 99.00th=[43254], 99.50th=[46400], 99.90th=[58459], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 64, max= 248, per=25.09%, avg=177.64, stdev=52.60, samples=70 iops : min= 16, max= 62, avg=44.36, stdev=13.17, samples=70 lat (msec) : 2=6.25%, 4=29.98%, 10=10.42%, 20=37.30%, 50=15.79% lat (msec) : 100=0.23%, 250=0.03% cpu : usr=0.12%, sys=5.53%, ctx=3569, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31584: Fri Dec 13 15:36:53 2024 read: IOPS=43, BW=172KiB/s (176kB/s)(6028KiB/35030msec) clat (usec): min=1625, max=37784, avg=4076.86, stdev=3242.97 lat (usec): min=1626, max=37785, avg=4077.62, stdev=3242.98 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1876], 10.00th=[ 1991], 20.00th=[ 2147], | 30.00th=[ 2278], 40.00th=[ 2507], 50.00th=[ 2835], 60.00th=[ 3425], | 70.00th=[ 3884], 80.00th=[ 4883], 90.00th=[ 8225], 95.00th=[10683], | 99.00th=[16319], 99.50th=[19792], 99.90th=[28443], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 56, max= 360, per=25.43%, avg=171.90, stdev=63.02, samples=70 iops : min= 14, max= 90, avg=42.90, stdev=15.74, samples=70 write: IOPS=44, BW=179KiB/s (183kB/s)(6260KiB/35030msec) clat (msec): min=8, max=188, avg=18.38, stdev= 7.83 lat (msec): min=8, max=188, avg=18.38, stdev= 7.83 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 19], | 70.00th=[ 20], 80.00th=[ 23], 90.00th=[ 28], 95.00th=[ 32], | 99.00th=[ 40], 99.50th=[ 42], 99.90th=[ 59], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 64, max= 264, per=25.22%, avg=178.53, stdev=54.72, samples=70 iops : min= 16, max= 66, avg=44.56, stdev=13.69, samples=70 lat (msec) : 2=5.40%, 4=29.95%, 10=11.36%, 20=38.38%, 50=14.84% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.09%, sys=5.67%, ctx=3576, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31585: Fri Dec 13 15:36:53 2024 read: IOPS=43, BW=174KiB/s (178kB/s)(6052KiB/34853msec) clat (usec): min=1484, max=37386, avg=3881.52, stdev=2999.06 lat (usec): min=1484, max=37387, avg=3882.36, stdev=2999.17 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1893], 10.00th=[ 1975], 20.00th=[ 2114], | 30.00th=[ 2245], 40.00th=[ 2442], 50.00th=[ 2769], 60.00th=[ 3294], | 70.00th=[ 3785], 80.00th=[ 4686], 90.00th=[ 7635], 95.00th=[10159], | 99.00th=[15008], 99.50th=[18220], 99.90th=[27132], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 56, max= 328, per=25.58%, avg=172.90, stdev=65.46, samples=69 iops : min= 14, max= 82, avg=43.16, stdev=16.36, samples=69 write: IOPS=44, BW=179KiB/s (183kB/s)(6236KiB/34853msec) clat (msec): min=8, max=177, avg=18.50, stdev= 7.92 lat (msec): min=8, max=177, avg=18.50, stdev= 7.92 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 19], | 70.00th=[ 20], 80.00th=[ 23], 90.00th=[ 28], 95.00th=[ 34], | 99.00th=[ 42], 99.50th=[ 44], 99.90th=[ 64], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 64, max= 256, per=25.21%, avg=178.46, stdev=52.59, samples=69 iops : min= 16, max= 64, avg=44.55, stdev=13.14, samples=69 lat (msec) : 2=6.32%, 4=29.75%, 10=11.52%, 20=37.60%, 50=14.71% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.11%, sys=5.63%, ctx=3559, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=677KiB/s (693kB/s), 167KiB/s-174KiB/s (171kB/s-178kB/s), io=23.4MiB (24.6MB), run=34853-35485msec WRITE: bw=708KiB/s (725kB/s), 179KiB/s-179KiB/s (183kB/s-183kB/s), io=24.6MiB (25.7MB), run=34853-35485msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31619: Fri Dec 13 15:36:59 2024 read: IOPS=76, BW=1217KiB/s (1247kB/s)(5920KiB/4863msec) clat (usec): min=51, max=90351, avg=9745.85, stdev=6598.19 lat (usec): min=51, max=90352, avg=9746.60, stdev=6598.29 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 67], 10.00th=[ 78], 20.00th=[ 7832], | 30.00th=[ 8848], 40.00th=[ 9634], 50.00th=[10290], 60.00th=[10945], | 70.00th=[11863], 80.00th=[13042], 90.00th=[15008], 95.00th=[16319], | 99.00th=[21103], 99.50th=[29754], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 768, max= 1344, per=24.33%, avg=1176.89, stdev=169.16, samples=9 iops : min= 48, max= 84, avg=73.56, stdev=10.57, samples=9 write: IOPS=81, BW=1309KiB/s (1341kB/s)(6368KiB/4863msec) clat (usec): min=653, max=19633, avg=3067.60, stdev=2980.70 lat (usec): min=655, max=19635, avg=3068.62, stdev=2980.74 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 750], 10.00th=[ 799], 20.00th=[ 848], | 30.00th=[ 963], 40.00th=[ 1205], 50.00th=[ 2278], 60.00th=[ 2966], | 70.00th=[ 3720], 80.00th=[ 4817], 90.00th=[ 6390], 95.00th=[ 8848], | 99.00th=[15533], 99.50th=[18482], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 832, max= 1696, per=25.22%, avg=1283.56, stdev=316.96, samples=9 iops : min= 52, max= 106, avg=80.22, stdev=19.81, samples=9 lat (usec) : 100=6.51%, 250=1.82%, 750=2.60%, 1000=13.80% lat (msec) : 2=8.72%, 4=13.54%, 10=24.87%, 20=27.47%, 50=0.52% lat (msec) : 100=0.13% cpu : usr=0.21%, sys=21.49%, ctx=2175, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31620: Fri Dec 13 15:36:59 2024 read: IOPS=74, BW=1200KiB/s (1228kB/s)(5744KiB/4788msec) clat (usec): min=61, max=27638, avg=9468.39, stdev=4886.30 lat (usec): min=61, max=27639, avg=9469.03, stdev=4886.30 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 69], 10.00th=[ 81], 20.00th=[ 7635], | 30.00th=[ 8979], 40.00th=[ 9634], 50.00th=[10421], 60.00th=[11076], | 70.00th=[11863], 80.00th=[12911], 90.00th=[14222], 95.00th=[15533], | 99.00th=[19792], 99.50th=[21103], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 768, max= 1408, per=24.84%, avg=1201.56, stdev=207.09, samples=9 iops : min= 48, max= 88, avg=75.00, stdev=13.06, samples=9 write: IOPS=85, BW=1367KiB/s (1400kB/s)(6544KiB/4788msec) clat (usec): min=597, max=73903, avg=3328.30, stdev=4816.90 lat (usec): min=598, max=73904, avg=3330.53, stdev=4816.49 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 725], 10.00th=[ 783], 20.00th=[ 857], | 30.00th=[ 955], 40.00th=[ 1188], 50.00th=[ 2245], 60.00th=[ 2835], | 70.00th=[ 3425], 80.00th=[ 4752], 90.00th=[ 6915], 95.00th=[ 9765], | 99.00th=[15664], 99.50th=[18744], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 1024, max= 1728, per=26.68%, avg=1357.78, stdev=227.57, samples=9 iops : min= 64, max= 108, avg=84.78, stdev=14.14, samples=9 lat (usec) : 100=5.86%, 250=1.69%, 500=0.13%, 750=3.12%, 1000=14.19% lat (msec) : 2=9.11%, 4=13.15%, 10=24.35%, 20=27.73%, 50=0.52% lat (msec) : 100=0.13% cpu : usr=0.21%, sys=21.52%, ctx=2085, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31621: Fri Dec 13 15:36:59 2024 read: IOPS=78, BW=1264KiB/s (1294kB/s)(6256KiB/4951msec) clat (usec): min=52, max=79507, avg=9523.49, stdev=6019.85 lat (usec): min=53, max=79508, avg=9524.14, stdev=6019.87 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 67], 10.00th=[ 80], 20.00th=[ 7504], | 30.00th=[ 8717], 40.00th=[ 9765], 50.00th=[10421], 60.00th=[11207], | 70.00th=[11994], 80.00th=[13173], 90.00th=[14222], 95.00th=[15533], | 99.00th=[18220], 99.50th=[19792], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 768, max= 1792, per=25.35%, avg=1226.00, stdev=289.72, samples=9 iops : min= 48, max= 112, avg=76.44, stdev=18.10, samples=9 write: IOPS=76, BW=1218KiB/s (1248kB/s)(6032KiB/4951msec) clat (usec): min=672, max=28795, avg=3134.34, stdev=3461.67 lat (usec): min=673, max=28796, avg=3136.08, stdev=3464.41 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 734], 10.00th=[ 791], 20.00th=[ 840], | 30.00th=[ 930], 40.00th=[ 1139], 50.00th=[ 1811], 60.00th=[ 2868], | 70.00th=[ 3556], 80.00th=[ 4948], 90.00th=[ 6521], 95.00th=[ 8586], | 99.00th=[18482], 99.50th=[24511], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 704, max= 1568, per=23.32%, avg=1186.89, stdev=301.57, samples=9 iops : min= 44, max= 98, avg=74.00, stdev=18.71, samples=9 lat (usec) : 100=7.16%, 250=1.69%, 500=0.26%, 750=2.73%, 1000=14.58% lat (msec) : 2=7.42%, 4=11.85%, 10=24.35%, 20=29.43%, 50=0.39% lat (msec) : 100=0.13% cpu : usr=0.06%, sys=21.15%, ctx=2137, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31622: Fri Dec 13 15:36:59 2024 read: IOPS=76, BW=1221KiB/s (1251kB/s)(6032KiB/4939msec) clat (usec): min=53, max=101557, avg=9679.92, stdev=6937.38 lat (usec): min=53, max=101558, avg=9680.57, stdev=6937.40 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 73], 10.00th=[ 83], 20.00th=[ 7439], | 30.00th=[ 8717], 40.00th=[ 9634], 50.00th=[ 10421], 60.00th=[ 11076], | 70.00th=[ 11863], 80.00th=[ 12780], 90.00th=[ 14615], 95.00th=[ 16319], | 99.00th=[ 21103], 99.50th=[ 27657], 99.90th=[101188], 99.95th=[101188], | 99.99th=[101188] bw ( KiB/s): min= 864, max= 1536, per=24.83%, avg=1201.11, stdev=200.62, samples=9 iops : min= 54, max= 96, avg=74.89, stdev=12.56, samples=9 write: IOPS=79, BW=1267KiB/s (1297kB/s)(6256KiB/4939msec) clat (usec): min=616, max=25289, avg=3257.08, stdev=3549.24 lat (usec): min=617, max=25290, avg=3261.32, stdev=3548.22 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 734], 10.00th=[ 791], 20.00th=[ 857], | 30.00th=[ 930], 40.00th=[ 1188], 50.00th=[ 2147], 60.00th=[ 2999], | 70.00th=[ 3851], 80.00th=[ 5080], 90.00th=[ 6718], 95.00th=[ 8717], | 99.00th=[19530], 99.50th=[22152], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 960, max= 1596, per=24.79%, avg=1261.44, stdev=200.90, samples=9 iops : min= 60, max= 99, avg=78.67, stdev=12.38, samples=9 lat (usec) : 100=6.25%, 250=1.95%, 500=0.13%, 750=3.12%, 1000=14.97% lat (msec) : 2=7.29%, 4=11.33%, 10=24.48%, 20=29.17%, 50=1.17% lat (msec) : 250=0.13% cpu : usr=0.12%, sys=22.40%, ctx=2155, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4838KiB/s (4954kB/s), 1200KiB/s-1264KiB/s (1228kB/s-1294kB/s), io=23.4MiB (24.5MB), run=4788-4951msec WRITE: bw=5090KiB/s (5212kB/s), 1218KiB/s-1367KiB/s (1248kB/s-1400kB/s), io=24.6MiB (25.8MB), run=4788-4951msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31624: Fri Dec 13 15:37:03 2024 read: IOPS=41, BW=669KiB/s (686kB/s)(5920KiB/8843msec) clat (usec): min=1966, max=95004, avg=4286.94, stdev=5789.76 lat (usec): min=1967, max=95005, avg=4287.73, stdev=5789.77 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2114], 10.00th=[ 2180], 20.00th=[ 2278], | 30.00th=[ 2409], 40.00th=[ 2540], 50.00th=[ 2704], 60.00th=[ 3032], | 70.00th=[ 3392], 80.00th=[ 4621], 90.00th=[ 7898], 95.00th=[11600], | 99.00th=[20055], 99.50th=[24511], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 416, max= 1184, per=25.03%, avg=656.82, stdev=216.60, samples=17 iops : min= 26, max= 74, avg=41.00, stdev=13.46, samples=17 write: IOPS=45, BW=720KiB/s (737kB/s)(6368KiB/8843msec) clat (usec): min=9686, max=50415, avg=18160.09, stdev=5498.95 lat (usec): min=9688, max=50416, avg=18161.73, stdev=5498.90 clat percentiles (usec): | 1.00th=[ 9896], 5.00th=[11338], 10.00th=[12256], 20.00th=[13698], | 30.00th=[14615], 40.00th=[16188], 50.00th=[17433], 60.00th=[18482], | 70.00th=[20055], 80.00th=[21627], 90.00th=[25560], 95.00th=[28705], | 99.00th=[34341], 99.50th=[40109], 99.90th=[50594], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 384, max= 928, per=25.77%, avg=711.41, stdev=162.72, samples=17 iops : min= 24, max= 58, avg=44.41, stdev=10.23, samples=17 lat (msec) : 2=0.52%, 4=36.20%, 10=8.59%, 20=38.54%, 50=15.89% lat (msec) : 100=0.26% cpu : usr=0.19%, sys=6.04%, ctx=876, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31625: Fri Dec 13 15:37:03 2024 read: IOPS=39, BW=629KiB/s (644kB/s)(5744KiB/9127msec) clat (usec): min=1778, max=31085, avg=4171.61, stdev=3714.59 lat (usec): min=1779, max=31087, avg=4172.41, stdev=3714.56 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2114], 10.00th=[ 2212], 20.00th=[ 2311], | 30.00th=[ 2442], 40.00th=[ 2606], 50.00th=[ 2737], 60.00th=[ 2966], | 70.00th=[ 3458], 80.00th=[ 4948], 90.00th=[ 7635], 95.00th=[11863], | 99.00th=[21627], 99.50th=[24249], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 192, max= 1088, per=23.96%, avg=628.78, stdev=238.26, samples=18 iops : min= 12, max= 68, avg=39.17, stdev=14.83, samples=18 write: IOPS=44, BW=717KiB/s (734kB/s)(6544KiB/9127msec) clat (usec): min=9508, max=50059, avg=18385.54, stdev=5548.25 lat (usec): min=9509, max=50061, avg=18387.15, stdev=5548.30 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11207], 10.00th=[12387], 20.00th=[13829], | 30.00th=[15139], 40.00th=[16450], 50.00th=[17695], 60.00th=[18744], | 70.00th=[20055], 80.00th=[22152], 90.00th=[25035], 95.00th=[28967], | 99.00th=[37487], 99.50th=[39584], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 416, max= 960, per=25.80%, avg=712.39, stdev=152.21, samples=18 iops : min= 26, max= 60, avg=44.39, stdev= 9.38, samples=18 lat (msec) : 2=0.65%, 4=33.98%, 10=8.85%, 20=39.58%, 50=16.80% lat (msec) : 100=0.13% cpu : usr=0.03%, sys=6.27%, ctx=915, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31626: Fri Dec 13 15:37:03 2024 read: IOPS=44, BW=714KiB/s (731kB/s)(6256KiB/8758msec) clat (usec): min=1929, max=22947, avg=4031.83, stdev=3300.33 lat (usec): min=1930, max=22948, avg=4032.58, stdev=3300.31 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2114], 10.00th=[ 2212], 20.00th=[ 2311], | 30.00th=[ 2409], 40.00th=[ 2573], 50.00th=[ 2769], 60.00th=[ 2999], | 70.00th=[ 3425], 80.00th=[ 4883], 90.00th=[ 7767], 95.00th=[11076], | 99.00th=[20317], 99.50th=[21365], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 416, max= 1248, per=27.40%, avg=718.94, stdev=252.94, samples=17 iops : min= 26, max= 78, avg=44.88, stdev=15.78, samples=17 write: IOPS=43, BW=689KiB/s (705kB/s)(6032KiB/8758msec) clat (usec): min=9422, max=83778, avg=18821.33, stdev=6380.74 lat (usec): min=9423, max=83783, avg=18823.02, stdev=6380.93 clat percentiles (usec): | 1.00th=[10552], 5.00th=[11600], 10.00th=[12256], 20.00th=[13960], | 30.00th=[15139], 40.00th=[16909], 50.00th=[17957], 60.00th=[19268], | 70.00th=[20579], 80.00th=[21890], 90.00th=[26084], 95.00th=[29754], | 99.00th=[35914], 99.50th=[43254], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 448, max= 928, per=24.81%, avg=685.06, stdev=140.75, samples=17 iops : min= 28, max= 58, avg=42.76, stdev= 8.74, samples=17 lat (msec) : 2=0.39%, 4=37.89%, 10=9.77%, 20=35.16%, 50=16.67% lat (msec) : 100=0.13% cpu : usr=0.10%, sys=6.38%, ctx=878, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31627: Fri Dec 13 15:37:03 2024 read: IOPS=41, BW=670KiB/s (687kB/s)(6032KiB/8997msec) clat (usec): min=1848, max=29844, avg=4153.95, stdev=3293.80 lat (usec): min=1849, max=29844, avg=4154.73, stdev=3293.86 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2089], 10.00th=[ 2180], 20.00th=[ 2311], | 30.00th=[ 2442], 40.00th=[ 2606], 50.00th=[ 2737], 60.00th=[ 2966], | 70.00th=[ 3851], 80.00th=[ 5407], 90.00th=[ 8225], 95.00th=[11469], | 99.00th=[15926], 99.50th=[20841], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 352, max= 1184, per=24.81%, avg=651.06, stdev=186.11, samples=17 iops : min= 22, max= 74, avg=40.59, stdev=11.59, samples=17 write: IOPS=43, BW=695KiB/s (712kB/s)(6256KiB/8997msec) clat (usec): min=9787, max=65252, avg=18690.93, stdev=6071.42 lat (usec): min=9789, max=65255, avg=18692.66, stdev=6071.43 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11338], 10.00th=[12387], 20.00th=[13698], | 30.00th=[15008], 40.00th=[16319], 50.00th=[17695], 60.00th=[19006], | 70.00th=[20579], 80.00th=[23200], 90.00th=[26084], 95.00th=[29492], | 99.00th=[36963], 99.50th=[46924], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 384, max= 960, per=24.81%, avg=684.94, stdev=168.57, samples=17 iops : min= 24, max= 60, avg=42.71, stdev=10.53, samples=17 lat (msec) : 2=0.78%, 4=34.11%, 10=11.07%, 20=36.46%, 50=17.45% lat (msec) : 100=0.13% cpu : usr=0.11%, sys=6.34%, ctx=890, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2624KiB/s (2687kB/s), 629KiB/s-714KiB/s (644kB/s-731kB/s), io=23.4MiB (24.5MB), run=8758-9127msec WRITE: bw=2761KiB/s (2827kB/s), 689KiB/s-720KiB/s (705kB/s-737kB/s), io=24.6MiB (25.8MB), run=8758-9127msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31642: Fri Dec 13 15:37:05 2024 read: IOPS=13, BW=13.9MiB/s (14.6MB/s)(7168KiB/504msec) clat (usec): min=12723, max=85350, avg=27990.24, stdev=25591.91 lat (usec): min=12724, max=85351, avg=27991.61, stdev=25591.77 clat percentiles (usec): | 1.00th=[12780], 5.00th=[12780], 10.00th=[12780], 20.00th=[15401], | 30.00th=[17957], 40.00th=[17957], 50.00th=[18482], 60.00th=[21103], | 70.00th=[21103], 80.00th=[24773], 90.00th=[85459], 95.00th=[85459], | 99.00th=[85459], 99.50th=[85459], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min=12288, max=12288, per=61.08%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9.92MiB/s (10.4MB/s)(5120KiB/504msec) clat (usec): min=28923, max=84300, avg=58996.73, stdev=22619.10 lat (usec): min=29014, max=84324, avg=59058.67, stdev=22593.15 clat percentiles (usec): | 1.00th=[28967], 5.00th=[28967], 10.00th=[28967], 20.00th=[28967], | 30.00th=[42730], 40.00th=[42730], 50.00th=[66847], 60.00th=[66847], | 70.00th=[71828], 80.00th=[71828], 90.00th=[84411], 95.00th=[84411], | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min=10240, max=10240, per=36.36%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=33.33%, 100=33.33% cpu : usr=0.00%, sys=14.31%, ctx=57, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31643: Fri Dec 13 15:37:05 2024 read: IOPS=1, BW=2012KiB/s (2060kB/s)(2048KiB/1018msec) clat (usec): min=15675, max=40928, avg=28301.99, stdev=17856.18 lat (usec): min=15677, max=40929, avg=28303.63, stdev=17855.68 clat percentiles (usec): | 1.00th=[15664], 5.00th=[15664], 10.00th=[15664], 20.00th=[15664], | 30.00th=[15664], 40.00th=[15664], 50.00th=[15664], 60.00th=[41157], | 70.00th=[41157], 80.00th=[41157], 90.00th=[41157], 95.00th=[41157], | 99.00th=[41157], 99.50th=[41157], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 4096, max= 4096, per=20.36%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9.82MiB/s (10.3MB/s)(10.0MiB/1018msec) clat (msec): min=28, max=515, avg=95.33, stdev=149.23 lat (msec): min=28, max=516, avg=95.41, stdev=149.23 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 34], | 30.00th=[ 35], 40.00th=[ 36], 50.00th=[ 39], 60.00th=[ 50], | 70.00th=[ 72], 80.00th=[ 72], 90.00th=[ 95], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min=16384, max=16384, per=58.17%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 16, max= 16, avg=16.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=58.33%, 100=25.00%, 750=8.33% cpu : usr=0.10%, sys=9.14%, ctx=41, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31644: Fri Dec 13 15:37:05 2024 read: IOPS=6, BW=6592KiB/s (6750kB/s)(6144KiB/932msec) clat (msec): min=15, max=475, avg=103.80, stdev=182.10 lat (msec): min=15, max=475, avg=103.80, stdev=182.10 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 24], | 30.00th=[ 24], 40.00th=[ 30], 50.00th=[ 30], 60.00th=[ 39], | 70.00th=[ 41], 80.00th=[ 41], 90.00th=[ 477], 95.00th=[ 477], | 99.00th=[ 477], 99.50th=[ 477], 99.90th=[ 477], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min=10219, max=10219, per=50.80%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6592KiB/s (6750kB/s)(6144KiB/932msec) clat (msec): min=26, max=110, avg=48.72, stdev=31.03 lat (msec): min=26, max=110, avg=48.77, stdev=31.02 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 40], 50.00th=[ 40], 60.00th=[ 43], | 70.00th=[ 46], 80.00th=[ 46], 90.00th=[ 110], 95.00th=[ 110], | 99.00th=[ 110], 99.50th=[ 110], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min=12263, max=12263, per=43.54%, avg=12263.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=75.00%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=7.52%, ctx=37, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31645: Fri Dec 13 15:37:05 2024 read: IOPS=5, BW=5401KiB/s (5530kB/s)(5120KiB/948msec) clat (msec): min=12, max=489, avg=117.94, stdev=207.87 lat (msec): min=12, max=489, avg=117.94, stdev=207.87 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 12], | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 34], 60.00th=[ 34], | 70.00th=[ 37], 80.00th=[ 37], 90.00th=[ 489], 95.00th=[ 489], | 99.00th=[ 489], 99.50th=[ 489], 99.90th=[ 489], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 8192, max= 8192, per=40.72%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7561KiB/s (7743kB/s)(7168KiB/948msec) clat (usec): min=35467, max=70311, avg=47327.73, stdev=13102.17 lat (usec): min=35535, max=70346, avg=47397.00, stdev=13092.51 clat percentiles (usec): | 1.00th=[35390], 5.00th=[35390], 10.00th=[35390], 20.00th=[35390], | 30.00th=[35390], 40.00th=[35390], 50.00th=[46924], 60.00th=[52691], | 70.00th=[52691], 80.00th=[54789], 90.00th=[70779], 95.00th=[70779], | 99.00th=[70779], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min=14336, max=14336, per=50.90%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=50.00%, 100=25.00%, 500=8.33% cpu : usr=0.00%, sys=6.76%, ctx=51, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.6MiB/s (20.6MB/s), 2012KiB/s-13.9MiB/s (2060kB/s-14.6MB/s), io=20.0MiB (20.0MB), run=504-1018msec WRITE: bw=27.5MiB/s (28.8MB/s), 6592KiB/s-9.92MiB/s (6750kB/s-10.4MB/s), io=28.0MiB (29.4MB), run=504-1018msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31647: Fri Dec 13 15:37:06 2024 read: IOPS=3, BW=3615KiB/s (3701kB/s)(7168KiB/1983msec) clat (usec): min=464, max=671550, avg=105615.82, stdev=250232.22 lat (usec): min=465, max=671550, avg=105616.95, stdev=250231.98 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 465], 10.00th=[ 465], 20.00th=[ 490], | 30.00th=[ 515], 40.00th=[ 515], 50.00th=[ 652], 60.00th=[ 14877], | 70.00th=[ 14877], 80.00th=[ 50594], 90.00th=[675283], 95.00th=[675283], | 99.00th=[675283], 99.50th=[675283], 99.90th=[675283], 99.95th=[675283], | 99.99th=[675283] bw ( KiB/s): min= 2048, max= 4096, per=29.76%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2582KiB/s (2644kB/s)(5120KiB/1983msec) clat (msec): min=16, max=1026, avg=226.54, stdev=447.41 lat (msec): min=16, max=1026, avg=226.61, stdev=447.41 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 17], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 29], 60.00th=[ 29], | 70.00th=[ 35], 80.00th=[ 35], 90.00th=[ 1028], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] lat (usec) : 500=16.67%, 750=16.67% lat (msec) : 20=16.67%, 50=25.00%, 100=8.33%, 750=8.33% cpu : usr=0.00%, sys=11.81%, ctx=342, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31648: Fri Dec 13 15:37:06 2024 read: IOPS=1, BW=1032KiB/s (1057kB/s)(2048KiB/1984msec) clat (usec): min=4925, max=71081, avg=38003.56, stdev=46779.39 lat (usec): min=4927, max=71082, avg=38004.78, stdev=46779.04 clat percentiles (usec): | 1.00th=[ 4948], 5.00th=[ 4948], 10.00th=[ 4948], 20.00th=[ 4948], | 30.00th=[ 4948], 40.00th=[ 4948], 50.00th=[ 4948], 60.00th=[70779], | 70.00th=[70779], 80.00th=[70779], 90.00th=[70779], 95.00th=[70779], | 99.00th=[70779], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 2048, max= 2048, per=19.84%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5161KiB/s (5285kB/s)(10.0MiB/1984msec) clat (msec): min=17, max=1543, avg=185.86, stdev=477.40 lat (msec): min=17, max=1543, avg=185.94, stdev=477.39 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 21], | 30.00th=[ 24], 40.00th=[ 27], 50.00th=[ 31], 60.00th=[ 32], | 70.00th=[ 38], 80.00th=[ 38], 90.00th=[ 93], 95.00th=[ 1536], | 99.00th=[ 1536], 99.50th=[ 1536], 99.90th=[ 1536], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 6144, max= 6144, per=42.52%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 10=8.33%, 20=8.33%, 50=58.33%, 100=16.67% cpu : usr=0.00%, sys=13.57%, ctx=62, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31649: Fri Dec 13 15:37:06 2024 read: IOPS=3, BW=3370KiB/s (3451kB/s)(6144KiB/1823msec) clat (usec): min=785, max=666265, avg=157752.67, stdev=258964.45 lat (usec): min=786, max=666266, avg=157753.44, stdev=258964.43 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 783], 10.00th=[ 783], 20.00th=[ 16909], | 30.00th=[ 16909], 40.00th=[ 20317], 50.00th=[ 20317], 60.00th=[ 48497], | 70.00th=[193987], 80.00th=[193987], 90.00th=[666895], 95.00th=[666895], | 99.00th=[666895], 99.50th=[666895], 99.90th=[666895], 99.95th=[666895], | 99.99th=[666895] bw ( KiB/s): min= 2048, max= 4096, per=33.07%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 write: IOPS=3, BW=3370KiB/s (3451kB/s)(6144KiB/1823msec) clat (msec): min=23, max=663, avg=144.12, stdev=255.74 lat (msec): min=23, max=663, avg=144.19, stdev=255.74 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 29], 50.00th=[ 29], 60.00th=[ 32], | 70.00th=[ 90], 80.00th=[ 90], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 2048, max= 8192, per=35.43%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (usec) : 1000=8.33% lat (msec) : 20=8.33%, 50=50.00%, 100=8.33%, 250=8.33%, 750=16.67% cpu : usr=0.05%, sys=21.95%, ctx=379, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31650: Fri Dec 13 15:37:06 2024 read: IOPS=3, BW=3135KiB/s (3211kB/s)(5120KiB/1633msec) clat (msec): min=4, max=663, avg=271.47, stdev=348.01 lat (msec): min=4, max=663, avg=271.48, stdev=348.01 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 45], 60.00th=[ 45], | 70.00th=[ 642], 80.00th=[ 642], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 2048, max= 2048, per=19.84%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4389KiB/s (4495kB/s)(7168KiB/1633msec) clat (usec): min=15218, max=66369, avg=33560.35, stdev=20019.44 lat (usec): min=15241, max=66445, avg=33630.43, stdev=20031.14 clat percentiles (usec): | 1.00th=[15270], 5.00th=[15270], 10.00th=[15270], 20.00th=[15533], | 30.00th=[15795], 40.00th=[15795], 50.00th=[32375], 60.00th=[38011], | 70.00th=[38011], 80.00th=[51643], 90.00th=[66323], 95.00th=[66323], | 99.00th=[66323], 99.50th=[66323], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 8192, max= 8192, per=56.69%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 10=16.67%, 20=25.00%, 50=25.00%, 100=16.67%, 750=16.67% cpu : usr=0.00%, sys=53.31%, ctx=386, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.1MiB/s (10.6MB/s), 1032KiB/s-3615KiB/s (1057kB/s-3701kB/s), io=20.0MiB (20.0MB), run=1633-1984msec WRITE: bw=14.1MiB/s (14.8MB/s), 2582KiB/s-5161KiB/s (2644kB/s-5285kB/s), io=28.0MiB (29.4MB), run=1633-1984msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31658: Fri Dec 13 15:37:11 2024 read: IOPS=0, BW=3156KiB/s (3232kB/s)(12.0MiB/3893msec) clat (msec): min=76, max=2315, avg=1295.61, stdev=1132.90 lat (msec): min=76, max=2315, avg=1295.61, stdev=1132.90 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 77], 10.00th=[ 77], 20.00th=[ 77], | 30.00th=[ 77], 40.00th=[ 1502], 50.00th=[ 1502], 60.00th=[ 1502], | 70.00th=[ 2333], 80.00th=[ 2333], 90.00th=[ 2333], 95.00th=[ 2333], | 99.00th=[ 2333], 99.50th=[ 2333], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 5752, max= 8192, per=100.00%, avg=6972.00, stdev=1725.34, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 100=33.33% cpu : usr=0.03%, sys=27.42%, ctx=1272, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31660: Fri Dec 13 15:37:11 2024 read: IOPS=0, BW=1932KiB/s (1978kB/s)(4096KiB/2120msec) clat (nsec): min=81648k, max=81648k, avg=81648037.00, stdev= 0.00 lat (nsec): min=81650k, max=81650k, avg=81650334.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[81265], 5.00th=[81265], 10.00th=[81265], 20.00th=[81265], | 30.00th=[81265], 40.00th=[81265], 50.00th=[81265], 60.00th=[81265], | 70.00th=[81265], 80.00th=[81265], 90.00th=[81265], 95.00th=[81265], | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3864KiB/s (3957kB/s)(8192KiB/2120msec) clat (msec): min=865, max=1141, avg=1003.49, stdev=194.48 lat (msec): min=866, max=1141, avg=1003.69, stdev=194.55 clat percentiles (msec): | 1.00th=[ 869], 5.00th=[ 869], 10.00th=[ 869], 20.00th=[ 869], | 30.00th=[ 869], 40.00th=[ 869], 50.00th=[ 869], 60.00th=[ 1150], | 70.00th=[ 1150], 80.00th=[ 1150], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 1000=33.33% cpu : usr=0.05%, sys=12.03%, ctx=308, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31662: Fri Dec 13 15:37:11 2024 write: IOPS=0, BW=2967KiB/s (3039kB/s)(12.0MiB/4141msec) clat (msec): min=949, max=1943, avg=1365.91, stdev=516.34 lat (msec): min=949, max=1943, avg=1366.45, stdev=516.21 clat percentiles (msec): | 1.00th=[ 953], 5.00th=[ 953], 10.00th=[ 953], 20.00th=[ 953], | 30.00th=[ 953], 40.00th=[ 1200], 50.00th=[ 1200], 60.00th=[ 1200], | 70.00th=[ 1938], 80.00th=[ 1938], 90.00th=[ 1938], 95.00th=[ 1938], | 99.00th=[ 1938], 99.50th=[ 1938], 99.90th=[ 1938], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 8159, max= 8192, per=100.00%, avg=8175.50, stdev=23.33, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 1000=33.33% cpu : usr=0.00%, sys=7.00%, ctx=50, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31663: Fri Dec 13 15:37:11 2024 write: IOPS=1, BW=6318KiB/s (6469kB/s)(12.0MiB/1945msec) clat (msec): min=94, max=1681, avg=624.29, stdev=915.52 lat (msec): min=94, max=1681, avg=624.62, stdev=915.35 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 94], 10.00th=[ 94], 20.00th=[ 94], | 30.00th=[ 94], 40.00th=[ 97], 50.00th=[ 97], 60.00th=[ 97], | 70.00th=[ 1687], 80.00th=[ 1687], 90.00th=[ 1687], 95.00th=[ 1687], | 99.00th=[ 1687], 99.50th=[ 1687], 99.90th=[ 1687], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=66.67% cpu : usr=0.00%, sys=16.31%, ctx=340, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4209KiB/s (4310kB/s), 1932KiB/s-3156KiB/s (1978kB/s-3232kB/s), io=16.0MiB (16.8MB), run=2120-3893msec WRITE: bw=7913KiB/s (8103kB/s), 2967KiB/s-6318KiB/s (3039kB/s-6469kB/s), io=32.0MiB (33.6MB), run=1945-4141msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31659: Fri Dec 13 15:37:11 2024 read: IOPS=0, BW=2964KiB/s (3035kB/s)(12.0MiB/4146msec) clat (msec): min=518, max=1866, avg=1380.04, stdev=748.19 lat (msec): min=518, max=1866, avg=1380.05, stdev=748.19 clat percentiles (msec): | 1.00th=[ 518], 5.00th=[ 518], 10.00th=[ 518], 20.00th=[ 518], | 30.00th=[ 518], 40.00th=[ 1754], 50.00th=[ 1754], 60.00th=[ 1754], | 70.00th=[ 1871], 80.00th=[ 1871], 90.00th=[ 1871], 95.00th=[ 1871], | 99.00th=[ 1871], 99.50th=[ 1871], 99.90th=[ 1871], 99.95th=[ 1871], | 99.99th=[ 1871] bw ( KiB/s): min= 5843, max= 8192, per=100.00%, avg=7017.50, stdev=1660.99, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 750=33.33% cpu : usr=0.00%, sys=1.47%, ctx=39, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31661: Fri Dec 13 15:37:11 2024 read: IOPS=0, BW=1001KiB/s (1025kB/s)(4096KiB/4093msec) clat (nsec): min=1723.5M, max=1723.5M, avg=1723517882.00, stdev= 0.00 lat (nsec): min=1723.5M, max=1723.5M, avg=1723519295.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1720], 5.00th=[ 1720], 10.00th=[ 1720], 20.00th=[ 1720], | 30.00th=[ 1720], 40.00th=[ 1720], 50.00th=[ 1720], 60.00th=[ 1720], | 70.00th=[ 1720], 80.00th=[ 1720], 90.00th=[ 1720], 95.00th=[ 1720], | 99.00th=[ 1720], 99.50th=[ 1720], 99.90th=[ 1720], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2001KiB/s (2050kB/s)(8192KiB/4093msec) clat (msec): min=130, max=2221, avg=1176.10, stdev=1478.82 lat (msec): min=130, max=2222, avg=1176.48, stdev=1478.65 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 131], 10.00th=[ 131], 20.00th=[ 131], | 30.00th=[ 131], 40.00th=[ 131], 50.00th=[ 131], 60.00th=[ 2232], | 70.00th=[ 2232], 80.00th=[ 2232], 90.00th=[ 2232], 95.00th=[ 2232], | 99.00th=[ 2232], 99.50th=[ 2232], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2232] lat (msec) : 250=33.33% cpu : usr=0.00%, sys=2.86%, ctx=40, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31664: Fri Dec 13 15:37:11 2024 write: IOPS=0, BW=2913KiB/s (2983kB/s)(12.0MiB/4218msec) clat (msec): min=853, max=1885, avg=1400.52, stdev=518.78 lat (msec): min=854, max=1886, avg=1400.92, stdev=518.71 clat percentiles (msec): | 1.00th=[ 852], 5.00th=[ 852], 10.00th=[ 852], 20.00th=[ 852], | 30.00th=[ 852], 40.00th=[ 1469], 50.00th=[ 1469], 60.00th=[ 1469], | 70.00th=[ 1888], 80.00th=[ 1888], 90.00th=[ 1888], 95.00th=[ 1888], | 99.00th=[ 1888], 99.50th=[ 1888], 99.90th=[ 1888], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 5657, max= 8192, per=89.14%, avg=6924.50, stdev=1792.52, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 1000=33.33% cpu : usr=0.02%, sys=2.21%, ctx=38, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31665: Fri Dec 13 15:37:11 2024 write: IOPS=1, BW=5245KiB/s (5370kB/s)(12.0MiB/2343msec) clat (msec): min=71, max=1127, avg=766.51, stdev=601.68 lat (msec): min=72, max=1127, avg=766.76, stdev=601.65 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 72], 10.00th=[ 72], 20.00th=[ 72], | 30.00th=[ 72], 40.00th=[ 1099], 50.00th=[ 1099], 60.00th=[ 1099], | 70.00th=[ 1133], 80.00th=[ 1133], 90.00th=[ 1133], 95.00th=[ 1133], | 99.00th=[ 1133], 99.50th=[ 1133], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33% cpu : usr=0.09%, sys=5.47%, ctx=192, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3952KiB/s (4047kB/s), 1001KiB/s-2964KiB/s (1025kB/s-3035kB/s), io=16.0MiB (16.8MB), run=4093-4146msec WRITE: bw=7769KiB/s (7955kB/s), 2001KiB/s-5245KiB/s (2050kB/s-5370kB/s), io=32.0MiB (33.6MB), run=2343-4218msec