== sanity test 398b: DIO and buffer IO race ============== 07:38:36 (1715341116) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.23323 s, 40.8 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=502: Fri May 10 07:38:52 2024 read: IOPS=115, BW=461KiB/s (472kB/s)(5992KiB/12995msec) clat (usec): min=48, max=194314, avg=3978.94, stdev=5947.79 lat (usec): min=48, max=194315, avg=3980.02, stdev=5947.80 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 61], 10.00th=[ 72], 20.00th=[ 121], | 30.00th=[ 2311], 40.00th=[ 2704], 50.00th=[ 3195], 60.00th=[ 3982], | 70.00th=[ 5014], 80.00th=[ 6325], 90.00th=[ 7832], 95.00th=[ 8848], | 99.00th=[ 15139], 99.50th=[ 17433], 99.90th=[ 38536], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 232, max= 736, per=24.58%, avg=453.96, stdev=93.55, samples=25 iops : min= 58, max= 184, avg=113.40, stdev=23.42, samples=25 write: IOPS=121, BW=484KiB/s (496kB/s)(6296KiB/12995msec) clat (usec): min=479, max=37609, avg=4403.82, stdev=4055.32 lat (usec): min=480, max=37610, avg=4404.60, stdev=4055.27 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 545], 10.00th=[ 578], 20.00th=[ 685], | 30.00th=[ 1037], 40.00th=[ 2507], 50.00th=[ 3621], 60.00th=[ 4555], | 70.00th=[ 5604], 80.00th=[ 7308], 90.00th=[10028], 95.00th=[12125], | 99.00th=[17957], 99.50th=[19006], 99.90th=[27132], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 288, max= 632, per=24.58%, avg=475.32, stdev=87.84, samples=25 iops : min= 72, max= 158, avg=118.76, stdev=21.92, samples=25 lat (usec) : 50=0.20%, 100=8.59%, 250=1.40%, 500=0.39%, 750=11.85% lat (usec) : 1000=3.19% lat (msec) : 2=5.01%, 4=26.33%, 10=36.36%, 20=6.32%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=23.93%, ctx=6525, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=504: Fri May 10 07:38:52 2024 read: IOPS=115, BW=462KiB/s (473kB/s)(5940KiB/12847msec) clat (usec): min=46, max=53769, avg=3825.50, stdev=3444.87 lat (usec): min=46, max=53770, avg=3827.06, stdev=3444.65 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 60], 10.00th=[ 70], 20.00th=[ 184], | 30.00th=[ 2376], 40.00th=[ 2769], 50.00th=[ 3228], 60.00th=[ 3785], | 70.00th=[ 4817], 80.00th=[ 6063], 90.00th=[ 7767], 95.00th=[ 9110], | 99.00th=[13042], 99.50th=[15008], 99.90th=[43254], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 256, max= 680, per=25.10%, avg=463.64, stdev=102.39, samples=25 iops : min= 64, max= 170, avg=115.88, stdev=25.62, samples=25 write: IOPS=123, BW=494KiB/s (506kB/s)(6348KiB/12847msec) clat (usec): min=458, max=166982, avg=4449.35, stdev=5895.83 lat (usec): min=459, max=166983, avg=4451.18, stdev=5896.06 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 545], 10.00th=[ 570], 20.00th=[ 660], | 30.00th=[ 1029], 40.00th=[ 2442], 50.00th=[ 3589], 60.00th=[ 4555], | 70.00th=[ 5538], 80.00th=[ 7046], 90.00th=[ 9503], 95.00th=[ 11863], | 99.00th=[ 17433], 99.50th=[ 21627], 99.90th=[ 52167], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 328, max= 816, per=25.28%, avg=488.92, stdev=102.87, samples=25 iops : min= 82, max= 204, avg=122.20, stdev=25.74, samples=25 lat (usec) : 50=0.33%, 100=8.11%, 250=1.33%, 500=0.42%, 750=12.08% lat (usec) : 1000=3.16% lat (msec) : 2=5.11%, 4=27.83%, 10=35.61%, 20=5.53%, 50=0.39% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.19%, sys=24.01%, ctx=6585, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=506: Fri May 10 07:38:52 2024 read: IOPS=118, BW=472KiB/s (483kB/s)(6028KiB/12770msec) clat (usec): min=48, max=37266, avg=3654.57, stdev=3215.67 lat (usec): min=48, max=37266, avg=3655.57, stdev=3215.79 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 61], 10.00th=[ 71], 20.00th=[ 119], | 30.00th=[ 2278], 40.00th=[ 2671], 50.00th=[ 3130], 60.00th=[ 3720], | 70.00th=[ 4621], 80.00th=[ 5932], 90.00th=[ 7439], 95.00th=[ 8356], | 99.00th=[12256], 99.50th=[17695], 99.90th=[36439], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 232, max= 720, per=25.45%, avg=470.04, stdev=100.41, samples=25 iops : min= 58, max= 180, avg=117.48, stdev=25.10, samples=25 write: IOPS=122, BW=490KiB/s (502kB/s)(6260KiB/12770msec) clat (usec): min=478, max=214827, avg=4574.65, stdev=6794.96 lat (usec): min=479, max=214828, avg=4575.34, stdev=6794.96 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 537], 10.00th=[ 570], 20.00th=[ 676], | 30.00th=[ 1057], 40.00th=[ 2507], 50.00th=[ 3687], 60.00th=[ 4621], | 70.00th=[ 5735], 80.00th=[ 7111], 90.00th=[ 9503], 95.00th=[ 12387], | 99.00th=[ 19792], 99.50th=[ 22152], 99.90th=[ 36439], 99.95th=[214959], | 99.99th=[214959] bw ( KiB/s): min= 312, max= 680, per=25.18%, avg=487.00, stdev=96.72, samples=25 iops : min= 78, max= 170, avg=121.72, stdev=24.19, samples=25 lat (usec) : 50=0.36%, 100=8.69%, 250=1.43%, 500=0.39%, 750=11.33% lat (usec) : 1000=3.19% lat (msec) : 2=5.53%, 4=27.25%, 10=36.23%, 20=5.01%, 50=0.55% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=24.01%, ctx=6431, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=507: Fri May 10 07:38:52 2024 read: IOPS=118, BW=473KiB/s (485kB/s)(6052KiB/12790msec) clat (usec): min=47, max=38484, avg=3790.29, stdev=3077.74 lat (usec): min=47, max=38484, avg=3791.53, stdev=3077.69 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 60], 10.00th=[ 70], 20.00th=[ 441], | 30.00th=[ 2343], 40.00th=[ 2704], 50.00th=[ 3261], 60.00th=[ 3982], | 70.00th=[ 4948], 80.00th=[ 5932], 90.00th=[ 7701], 95.00th=[ 8848], | 99.00th=[12518], 99.50th=[15401], 99.90th=[26608], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 264, max= 680, per=25.51%, avg=471.16, stdev=88.86, samples=25 iops : min= 66, max= 170, avg=117.72, stdev=22.21, samples=25 write: IOPS=121, BW=488KiB/s (499kB/s)(6236KiB/12790msec) clat (usec): min=435, max=165977, avg=4458.59, stdev=5845.28 lat (usec): min=436, max=165978, avg=4459.58, stdev=5845.20 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 660], | 30.00th=[ 1012], 40.00th=[ 2245], 50.00th=[ 3458], 60.00th=[ 4293], | 70.00th=[ 5538], 80.00th=[ 7242], 90.00th=[ 10290], 95.00th=[ 12387], | 99.00th=[ 19006], 99.50th=[ 21103], 99.90th=[ 34341], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 320, max= 664, per=25.11%, avg=485.56, stdev=80.69, samples=25 iops : min= 80, max= 166, avg=121.32, stdev=20.18, samples=25 lat (usec) : 50=0.36%, 100=8.43%, 250=0.85%, 500=0.62%, 750=11.82% lat (usec) : 1000=2.99% lat (msec) : 2=5.31%, 4=28.29%, 10=34.47%, 20=6.41%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=24.07%, ctx=6468, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1848KiB/s (1892kB/s), 461KiB/s-473KiB/s (472kB/s-485kB/s), io=23.4MiB (24.6MB), run=12770-12995msec WRITE: bw=1935KiB/s (1981kB/s), 484KiB/s-494KiB/s (496kB/s-506kB/s), io=24.6MiB (25.7MB), run=12770-12995msec rand-rw: (groupid=0, jobs=1): err= 0: pid=500: Fri May 10 07:39:21 2024 read: IOPS=35, BW=142KiB/s (146kB/s)(5992KiB/42051msec) clat (usec): min=1674, max=221025, avg=3660.83, stdev=6603.52 lat (usec): min=1675, max=221026, avg=3661.61, stdev=6603.52 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1893], 10.00th=[ 1958], 20.00th=[ 2040], | 30.00th=[ 2147], 40.00th=[ 2245], 50.00th=[ 2376], 60.00th=[ 2540], | 70.00th=[ 2769], 80.00th=[ 3392], 90.00th=[ 6652], 95.00th=[ 10421], | 99.00th=[ 19268], 99.50th=[ 25822], 99.90th=[ 40109], 99.95th=[221250], | 99.99th=[221250] bw ( KiB/s): min= 40, max= 256, per=25.18%, avg=142.54, stdev=53.98, samples=84 iops : min= 10, max= 64, avg=35.54, stdev=13.53, samples=84 write: IOPS=37, BW=150KiB/s (153kB/s)(6296KiB/42051msec) clat (usec): min=11730, max=64755, avg=23165.86, stdev=7978.40 lat (usec): min=11731, max=64756, avg=23166.72, stdev=7978.43 clat percentiles (usec): | 1.00th=[13304], 5.00th=[14222], 10.00th=[14877], 20.00th=[16188], | 30.00th=[17957], 40.00th=[20055], 50.00th=[22414], 60.00th=[23725], | 70.00th=[25297], 80.00th=[27395], 90.00th=[33424], 95.00th=[39584], | 99.00th=[49546], 99.50th=[55313], 99.90th=[63177], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 40, max= 208, per=25.25%, avg=149.49, stdev=39.21, samples=84 iops : min= 10, max= 52, avg=37.27, stdev= 9.84, samples=84 lat (msec) : 2=7.03%, 4=33.59%, 10=5.53%, 20=22.27%, 50=31.05% lat (msec) : 100=0.49%, 250=0.03% cpu : usr=0.10%, sys=4.44%, ctx=3404, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=501: Fri May 10 07:39:21 2024 read: IOPS=35, BW=140KiB/s (143kB/s)(5940KiB/42396msec) clat (usec): min=1638, max=170955, avg=3534.41, stdev=5326.49 lat (usec): min=1639, max=170958, avg=3535.12, stdev=5326.56 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1893], 10.00th=[ 1958], 20.00th=[ 2057], | 30.00th=[ 2147], 40.00th=[ 2245], 50.00th=[ 2343], 60.00th=[ 2540], | 70.00th=[ 2769], 80.00th=[ 3458], 90.00th=[ 6521], 95.00th=[ 9634], | 99.00th=[ 15270], 99.50th=[ 20055], 99.90th=[ 39584], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 16, max= 272, per=24.74%, avg=140.05, stdev=55.26, samples=84 iops : min= 4, max= 68, avg=34.90, stdev=13.86, samples=84 write: IOPS=37, BW=150KiB/s (153kB/s)(6348KiB/42396msec) clat (usec): min=12448, max=83512, avg=23342.18, stdev=7840.28 lat (usec): min=12449, max=83514, avg=23343.03, stdev=7840.33 clat percentiles (usec): | 1.00th=[13435], 5.00th=[14091], 10.00th=[14877], 20.00th=[16450], | 30.00th=[18482], 40.00th=[20579], 50.00th=[22414], 60.00th=[23987], | 70.00th=[25297], 80.00th=[27657], 90.00th=[33817], 95.00th=[39060], | 99.00th=[47973], 99.50th=[52691], 99.90th=[64226], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 40, max= 200, per=25.19%, avg=149.10, stdev=38.05, samples=84 iops : min= 10, max= 50, avg=37.17, stdev= 9.54, samples=84 lat (msec) : 2=7.19%, 4=33.11%, 10=5.79%, 20=20.83%, 50=32.68% lat (msec) : 100=0.36%, 250=0.03% cpu : usr=0.07%, sys=4.38%, ctx=3386, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=503: Fri May 10 07:39:21 2024 read: IOPS=35, BW=142KiB/s (146kB/s)(6028KiB/42348msec) clat (usec): min=1690, max=39165, avg=3614.75, stdev=3561.36 lat (usec): min=1691, max=39166, avg=3615.48, stdev=3561.33 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1893], 10.00th=[ 1958], 20.00th=[ 2057], | 30.00th=[ 2147], 40.00th=[ 2245], 50.00th=[ 2376], 60.00th=[ 2540], | 70.00th=[ 2835], 80.00th=[ 3654], 90.00th=[ 7439], 95.00th=[10159], | 99.00th=[18482], 99.50th=[25822], 99.90th=[38011], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 40, max= 272, per=25.10%, avg=142.05, stdev=50.95, samples=84 iops : min= 10, max= 68, avg=35.40, stdev=12.75, samples=84 write: IOPS=36, BW=148KiB/s (151kB/s)(6260KiB/42348msec) clat (msec): min=12, max=185, avg=23.51, stdev= 8.52 lat (msec): min=12, max=185, avg=23.51, stdev= 8.52 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 22], 50.00th=[ 23], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 28], 90.00th=[ 34], 95.00th=[ 39], | 99.00th=[ 50], 99.50th=[ 56], 99.90th=[ 72], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 56, max= 192, per=24.83%, avg=147.00, stdev=37.19, samples=84 iops : min= 14, max= 48, avg=36.64, stdev= 9.33, samples=84 lat (msec) : 2=7.13%, 4=33.20%, 10=6.25%, 20=19.89%, 50=33.07% lat (msec) : 100=0.42%, 250=0.03% cpu : usr=0.09%, sys=4.36%, ctx=3454, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=505: Fri May 10 07:39:21 2024 read: IOPS=36, BW=144KiB/s (148kB/s)(6052KiB/41965msec) clat (usec): min=1708, max=30979, avg=3426.49, stdev=2958.42 lat (usec): min=1708, max=30980, avg=3427.16, stdev=2958.42 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1909], 10.00th=[ 1958], 20.00th=[ 2073], | 30.00th=[ 2180], 40.00th=[ 2278], 50.00th=[ 2376], 60.00th=[ 2540], | 70.00th=[ 2802], 80.00th=[ 3425], 90.00th=[ 6587], 95.00th=[ 9765], | 99.00th=[16057], 99.50th=[20841], 99.90th=[25822], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 16, max= 272, per=25.38%, avg=143.66, stdev=55.64, samples=83 iops : min= 4, max= 68, avg=35.81, stdev=13.93, samples=83 write: IOPS=37, BW=149KiB/s (152kB/s)(6236KiB/41965msec) clat (msec): min=12, max=185, avg=23.52, stdev= 8.95 lat (msec): min=12, max=185, avg=23.53, stdev= 8.95 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 28], 90.00th=[ 34], 95.00th=[ 40], | 99.00th=[ 50], 99.50th=[ 57], 99.90th=[ 95], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 40, max= 200, per=25.05%, avg=148.29, stdev=36.60, samples=83 iops : min= 10, max= 50, avg=36.96, stdev= 9.17, samples=83 lat (msec) : 2=6.58%, 4=34.57%, 10=5.92%, 20=19.99%, 50=32.52% lat (msec) : 100=0.39%, 250=0.03% cpu : usr=0.09%, sys=4.43%, ctx=3419, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=566KiB/s (580kB/s), 140KiB/s-144KiB/s (143kB/s-148kB/s), io=23.4MiB (24.6MB), run=41965-42396msec WRITE: bw=593KiB/s (607kB/s), 148KiB/s-150KiB/s (151kB/s-153kB/s), io=24.6MiB (25.7MB), run=41965-42396msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=552: Fri May 10 07:39:27 2024 read: IOPS=81, BW=1308KiB/s (1339kB/s)(5920KiB/4527msec) clat (usec): min=51, max=41952, avg=9023.12, stdev=5733.42 lat (usec): min=51, max=41953, avg=9023.72, stdev=5733.45 clat percentiles (usec): | 1.00th=[ 54], 5.00th=[ 61], 10.00th=[ 68], 20.00th=[ 110], | 30.00th=[ 8586], 40.00th=[ 9503], 50.00th=[10290], 60.00th=[10683], | 70.00th=[11338], 80.00th=[12780], 90.00th=[14353], 95.00th=[15795], | 99.00th=[25822], 99.50th=[26084], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 800, max= 1760, per=25.09%, avg=1308.00, stdev=278.83, samples=9 iops : min= 50, max= 110, avg=81.67, stdev=17.36, samples=9 write: IOPS=87, BW=1407KiB/s (1440kB/s)(6368KiB/4527msec) clat (usec): min=640, max=43099, avg=2963.46, stdev=4207.08 lat (usec): min=640, max=43100, avg=2964.34, stdev=4207.10 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 734], 10.00th=[ 750], 20.00th=[ 807], | 30.00th=[ 873], 40.00th=[ 988], 50.00th=[ 1287], 60.00th=[ 2474], | 70.00th=[ 3163], 80.00th=[ 3982], 90.00th=[ 6456], 95.00th=[ 7898], | 99.00th=[29230], 99.50th=[30278], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 928, max= 1856, per=25.66%, avg=1407.67, stdev=343.16, samples=9 iops : min= 58, max= 116, avg=87.89, stdev=21.45, samples=9 lat (usec) : 100=9.24%, 250=1.43%, 750=4.69%, 1000=16.67% lat (msec) : 2=7.42%, 4=13.02%, 10=19.27%, 20=26.56%, 50=1.69% cpu : usr=0.11%, sys=22.38%, ctx=1949, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=554: Fri May 10 07:39:27 2024 read: IOPS=78, BW=1260KiB/s (1290kB/s)(5744KiB/4558msec) clat (usec): min=52, max=48899, avg=9006.80, stdev=5868.34 lat (usec): min=53, max=48899, avg=9007.43, stdev=5868.33 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 63], 10.00th=[ 73], 20.00th=[ 111], | 30.00th=[ 8717], 40.00th=[ 9503], 50.00th=[10028], 60.00th=[10814], | 70.00th=[11863], 80.00th=[12780], 90.00th=[14222], 95.00th=[16450], | 99.00th=[21103], 99.50th=[30802], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 832, max= 1600, per=24.14%, avg=1258.33, stdev=243.62, samples=9 iops : min= 52, max= 100, avg=78.56, stdev=15.21, samples=9 write: IOPS=89, BW=1436KiB/s (1470kB/s)(6544KiB/4558msec) clat (usec): min=639, max=65177, avg=3221.09, stdev=4496.65 lat (usec): min=639, max=65179, avg=3222.05, stdev=4496.70 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 734], 10.00th=[ 775], 20.00th=[ 840], | 30.00th=[ 898], 40.00th=[ 1012], 50.00th=[ 1614], 60.00th=[ 2835], | 70.00th=[ 3523], 80.00th=[ 4883], 90.00th=[ 7242], 95.00th=[ 9110], | 99.00th=[17695], 99.50th=[19792], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 1024, max= 1920, per=26.12%, avg=1432.56, stdev=235.62, samples=9 iops : min= 64, max= 120, avg=89.44, stdev=14.74, samples=9 lat (usec) : 100=8.85%, 250=1.43%, 500=0.13%, 750=3.52%, 1000=17.84% lat (msec) : 2=6.64%, 4=13.02%, 10=21.87%, 20=25.65%, 50=0.91% lat (msec) : 100=0.13% cpu : usr=0.11%, sys=21.92%, ctx=1890, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=556: Fri May 10 07:39:27 2024 read: IOPS=85, BW=1362KiB/s (1394kB/s)(6256KiB/4594msec) clat (usec): min=51, max=33829, avg=8705.90, stdev=5718.94 lat (usec): min=51, max=33829, avg=8706.45, stdev=5718.93 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 61], 10.00th=[ 65], 20.00th=[ 103], | 30.00th=[ 7898], 40.00th=[ 9241], 50.00th=[10159], 60.00th=[10945], | 70.00th=[11863], 80.00th=[12649], 90.00th=[13960], 95.00th=[16057], | 99.00th=[24249], 99.50th=[32375], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 896, max= 1824, per=25.92%, avg=1351.11, stdev=278.41, samples=9 iops : min= 56, max= 114, avg=84.44, stdev=17.40, samples=9 write: IOPS=82, BW=1313KiB/s (1345kB/s)(6032KiB/4594msec) clat (usec): min=624, max=55664, avg=3136.88, stdev=3994.40 lat (usec): min=624, max=55665, avg=3137.86, stdev=3994.41 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 709], 10.00th=[ 766], 20.00th=[ 816], | 30.00th=[ 947], 40.00th=[ 1172], 50.00th=[ 2040], 60.00th=[ 2900], | 70.00th=[ 3425], 80.00th=[ 4948], 90.00th=[ 6783], 95.00th=[ 8586], | 99.00th=[15664], 99.50th=[20317], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 864, max= 1760, per=23.40%, avg=1283.56, stdev=257.22, samples=9 iops : min= 54, max= 110, avg=80.22, stdev=16.08, samples=9 lat (usec) : 100=9.77%, 250=2.47%, 500=0.13%, 750=4.43%, 1000=12.63% lat (msec) : 2=7.68%, 4=12.37%, 10=22.66%, 20=26.82%, 50=0.91% lat (msec) : 100=0.13% cpu : usr=0.17%, sys=21.66%, ctx=2014, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=558: Fri May 10 07:39:27 2024 read: IOPS=82, BW=1325KiB/s (1357kB/s)(6032KiB/4552msec) clat (usec): min=50, max=52222, avg=8793.43, stdev=5909.22 lat (usec): min=51, max=52222, avg=8794.05, stdev=5909.24 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 61], 10.00th=[ 65], 20.00th=[ 113], | 30.00th=[ 7898], 40.00th=[ 8979], 50.00th=[ 9765], 60.00th=[10945], | 70.00th=[11863], 80.00th=[12780], 90.00th=[14353], 95.00th=[15664], | 99.00th=[24511], 99.50th=[26870], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 960, max= 1760, per=25.03%, avg=1304.89, stdev=231.74, samples=9 iops : min= 60, max= 110, avg=81.56, stdev=14.48, samples=9 write: IOPS=85, BW=1374KiB/s (1407kB/s)(6256KiB/4552msec) clat (usec): min=645, max=58705, avg=3144.28, stdev=4480.81 lat (usec): min=646, max=58707, avg=3145.37, stdev=4481.05 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 709], 10.00th=[ 750], 20.00th=[ 799], | 30.00th=[ 848], 40.00th=[ 930], 50.00th=[ 1221], 60.00th=[ 2638], | 70.00th=[ 3589], 80.00th=[ 5276], 90.00th=[ 7308], 95.00th=[ 9110], | 99.00th=[16712], 99.50th=[36439], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 1056, max= 2048, per=24.76%, avg=1358.22, stdev=356.02, samples=9 iops : min= 66, max= 128, avg=84.89, stdev=22.25, samples=9 lat (usec) : 100=9.37%, 250=1.82%, 750=5.47%, 1000=17.45% lat (msec) : 2=6.25%, 4=8.98%, 10=25.52%, 20=23.96%, 50=0.91% lat (msec) : 100=0.26% cpu : usr=0.15%, sys=20.87%, ctx=1981, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5214KiB/s (5339kB/s), 1260KiB/s-1362KiB/s (1290kB/s-1394kB/s), io=23.4MiB (24.5MB), run=4527-4594msec WRITE: bw=5485KiB/s (5617kB/s), 1313KiB/s-1436KiB/s (1345kB/s-1470kB/s), io=24.6MiB (25.8MB), run=4527-4594msec rand-rw: (groupid=0, jobs=1): err= 0: pid=553: Fri May 10 07:39:33 2024 read: IOPS=34, BW=555KiB/s (568kB/s)(5920KiB/10672msec) clat (usec): min=1879, max=42037, avg=4144.70, stdev=3922.97 lat (usec): min=1880, max=42038, avg=4145.30, stdev=3922.98 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2073], 10.00th=[ 2147], 20.00th=[ 2278], | 30.00th=[ 2409], 40.00th=[ 2540], 50.00th=[ 2769], 60.00th=[ 2999], | 70.00th=[ 3523], 80.00th=[ 4621], 90.00th=[ 7898], 95.00th=[11863], | 99.00th=[19792], 99.50th=[22152], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 320, max= 800, per=24.78%, avg=545.48, stdev=129.05, samples=21 iops : min= 20, max= 50, avg=34.05, stdev= 8.13, samples=21 write: IOPS=37, BW=597KiB/s (611kB/s)(6368KiB/10672msec) clat (usec): min=12803, max=47999, avg=22942.01, stdev=6483.57 lat (usec): min=12804, max=48001, avg=22943.28, stdev=6483.50 clat percentiles (usec): | 1.00th=[13173], 5.00th=[14222], 10.00th=[15008], 20.00th=[16581], | 30.00th=[19006], 40.00th=[21103], 50.00th=[22676], 60.00th=[23987], | 70.00th=[25297], 80.00th=[27132], 90.00th=[32375], 95.00th=[34866], | 99.00th=[43254], 99.50th=[47973], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 384, max= 768, per=25.60%, avg=592.67, stdev=134.92, samples=21 iops : min= 24, max= 48, avg=37.00, stdev= 8.46, samples=21 lat (msec) : 2=0.39%, 4=35.94%, 10=8.59%, 20=20.44%, 50=34.64% cpu : usr=0.04%, sys=4.99%, ctx=855, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=555: Fri May 10 07:39:33 2024 read: IOPS=32, BW=528KiB/s (541kB/s)(5744KiB/10881msec) clat (usec): min=1902, max=24902, avg=4166.45, stdev=3863.67 lat (usec): min=1903, max=24902, avg=4167.06, stdev=3863.68 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2114], 10.00th=[ 2147], 20.00th=[ 2278], | 30.00th=[ 2376], 40.00th=[ 2507], 50.00th=[ 2638], 60.00th=[ 2835], | 70.00th=[ 3195], 80.00th=[ 4817], 90.00th=[ 8029], 95.00th=[12911], | 99.00th=[20841], 99.50th=[21890], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 192, max= 928, per=23.87%, avg=525.38, stdev=207.21, samples=21 iops : min= 12, max= 58, avg=32.67, stdev=12.94, samples=21 write: IOPS=37, BW=601KiB/s (616kB/s)(6544KiB/10881msec) clat (usec): min=13078, max=98656, avg=22929.86, stdev=7770.71 lat (usec): min=13079, max=98658, avg=22931.01, stdev=7770.77 clat percentiles (usec): | 1.00th=[13304], 5.00th=[14091], 10.00th=[14877], 20.00th=[16319], | 30.00th=[18220], 40.00th=[19792], 50.00th=[21890], 60.00th=[23462], | 70.00th=[25297], 80.00th=[28443], 90.00th=[32375], 95.00th=[35914], | 99.00th=[42730], 99.50th=[46924], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 416, max= 768, per=25.59%, avg=592.38, stdev=129.72, samples=21 iops : min= 26, max= 48, avg=36.86, stdev= 8.06, samples=21 lat (msec) : 2=0.39%, 4=35.29%, 10=7.29%, 20=24.74%, 50=32.16% lat (msec) : 100=0.13% cpu : usr=0.10%, sys=4.84%, ctx=880, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=557: Fri May 10 07:39:33 2024 read: IOPS=37, BW=597KiB/s (612kB/s)(6256KiB/10476msec) clat (usec): min=1916, max=27082, avg=4112.12, stdev=3847.08 lat (usec): min=1917, max=27083, avg=4112.73, stdev=3847.08 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2114], 10.00th=[ 2180], 20.00th=[ 2245], | 30.00th=[ 2376], 40.00th=[ 2507], 50.00th=[ 2704], 60.00th=[ 2933], | 70.00th=[ 3261], 80.00th=[ 4948], 90.00th=[ 7242], 95.00th=[13173], | 99.00th=[22152], 99.50th=[25297], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 288, max= 1117, per=27.53%, avg=606.00, stdev=243.45, samples=20 iops : min= 18, max= 69, avg=37.70, stdev=15.15, samples=20 write: IOPS=35, BW=576KiB/s (590kB/s)(6032KiB/10476msec) clat (usec): min=13100, max=52351, avg=23502.84, stdev=7303.56 lat (usec): min=13101, max=52352, avg=23504.01, stdev=7303.55 clat percentiles (usec): | 1.00th=[13304], 5.00th=[14353], 10.00th=[15008], 20.00th=[16909], | 30.00th=[19268], 40.00th=[20841], 50.00th=[22676], 60.00th=[23987], | 70.00th=[25822], 80.00th=[28967], 90.00th=[33162], 95.00th=[37487], | 99.00th=[47973], 99.50th=[52167], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 384, max= 800, per=24.59%, avg=569.30, stdev=129.60, samples=20 iops : min= 24, max= 50, avg=35.40, stdev= 8.20, samples=20 lat (msec) : 2=0.65%, 4=38.54%, 10=8.46%, 20=19.14%, 50=32.81% lat (msec) : 100=0.39% cpu : usr=0.10%, sys=4.97%, ctx=873, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=559: Fri May 10 07:39:33 2024 read: IOPS=35, BW=570KiB/s (584kB/s)(6032KiB/10575msec) clat (usec): min=1873, max=24013, avg=4150.89, stdev=3685.46 lat (usec): min=1874, max=24014, avg=4151.60, stdev=3685.45 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2089], 10.00th=[ 2147], 20.00th=[ 2245], | 30.00th=[ 2409], 40.00th=[ 2507], 50.00th=[ 2671], 60.00th=[ 2900], | 70.00th=[ 3523], 80.00th=[ 4817], 90.00th=[ 7635], 95.00th=[13435], | 99.00th=[19268], 99.50th=[22938], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 192, max= 832, per=25.69%, avg=565.33, stdev=157.85, samples=21 iops : min= 12, max= 52, avg=35.33, stdev= 9.87, samples=21 write: IOPS=36, BW=592KiB/s (606kB/s)(6256KiB/10575msec) clat (usec): min=11854, max=67314, avg=23027.24, stdev=7543.81 lat (usec): min=11855, max=67315, avg=23028.47, stdev=7543.78 clat percentiles (usec): | 1.00th=[13042], 5.00th=[14091], 10.00th=[14877], 20.00th=[16581], | 30.00th=[18220], 40.00th=[20579], 50.00th=[22152], 60.00th=[23462], | 70.00th=[24773], 80.00th=[27132], 90.00th=[33424], 95.00th=[35914], | 99.00th=[48497], 99.50th=[54264], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 384, max= 800, per=25.54%, avg=591.24, stdev=141.17, samples=21 iops : min= 24, max= 50, avg=36.95, stdev= 8.82, samples=21 lat (msec) : 2=1.17%, 4=35.68%, 10=8.20%, 20=22.14%, 50=32.42% lat (msec) : 100=0.39% cpu : usr=0.02%, sys=5.04%, ctx=866, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2201KiB/s (2254kB/s), 528KiB/s-597KiB/s (541kB/s-612kB/s), io=23.4MiB (24.5MB), run=10476-10881msec WRITE: bw=2316KiB/s (2372kB/s), 576KiB/s-601KiB/s (590kB/s-616kB/s), io=24.6MiB (25.8MB), run=10476-10881msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=578: Fri May 10 07:39:38 2024 read: IOPS=2, BW=2214KiB/s (2267kB/s)(7168KiB/3238msec) clat (usec): min=437, max=949141, avg=343680.49, stdev=406594.50 lat (usec): min=438, max=949144, avg=343681.90, stdev=406594.85 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 437], 10.00th=[ 437], 20.00th=[ 545], | 30.00th=[ 43254], 40.00th=[ 43254], 50.00th=[202376], 60.00th=[337642], | 70.00th=[337642], 80.00th=[868221], 90.00th=[952108], 95.00th=[952108], | 99.00th=[952108], 99.50th=[952108], 99.90th=[952108], 99.95th=[952108], | 99.99th=[952108] bw ( KiB/s): min= 2023, max= 4096, per=49.32%, avg=2553.75, stdev=1028.23, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 write: IOPS=1, BW=1581KiB/s (1619kB/s)(5120KiB/3238msec) clat (msec): min=35, max=596, avg=165.13, stdev=242.08 lat (msec): min=35, max=596, avg=165.20, stdev=242.06 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 45], 40.00th=[ 45], 50.00th=[ 53], 60.00th=[ 53], | 70.00th=[ 97], 80.00th=[ 97], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 2023, max= 4031, per=41.76%, avg=3027.00, stdev=1419.87, samples=2 iops : min= 1, max= 3, avg= 2.00, stdev= 1.41, samples=2 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=25.00%, 100=16.67%, 250=8.33%, 500=8.33%, 750=8.33% lat (msec) : 1000=16.67% cpu : usr=0.00%, sys=20.42%, ctx=759, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=580: Fri May 10 07:39:38 2024 read: IOPS=0, BW=825KiB/s (845kB/s)(2048KiB/2481msec) clat (usec): min=442, max=56341, avg=28391.93, stdev=39526.53 lat (usec): min=443, max=56342, avg=28392.72, stdev=39526.73 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 445], 10.00th=[ 445], 20.00th=[ 445], | 30.00th=[ 445], 40.00th=[ 445], 50.00th=[ 445], 60.00th=[56361], | 70.00th=[56361], 80.00th=[56361], 90.00th=[56361], 95.00th=[56361], | 99.00th=[56361], 99.50th=[56361], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 2048, max= 2048, per=39.55%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4127KiB/s (4226kB/s)(10.0MiB/2481msec) clat (msec): min=22, max=995, avg=241.80, stdev=371.81 lat (msec): min=22, max=995, avg=241.87, stdev=371.80 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 26], | 30.00th=[ 41], 40.00th=[ 44], 50.00th=[ 50], 60.00th=[ 67], | 70.00th=[ 230], 80.00th=[ 230], 90.00th=[ 877], 95.00th=[ 995], | 99.00th=[ 995], 99.50th=[ 995], 99.90th=[ 995], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 4096, max= 4096, per=56.50%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=41.67%, 100=25.00%, 250=8.33%, 1000=16.67% cpu : usr=0.00%, sys=10.44%, ctx=116, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=582: Fri May 10 07:39:38 2024 read: IOPS=1, BW=1553KiB/s (1591kB/s)(6144KiB/3955msec) clat (usec): min=430, max=831392, avg=249134.33, stdev=346266.51 lat (usec): min=431, max=831393, avg=249135.66, stdev=346266.50 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 433], 10.00th=[ 433], 20.00th=[ 21890], | 30.00th=[ 21890], 40.00th=[ 41681], 50.00th=[ 41681], 60.00th=[ 77071], | 70.00th=[522191], 80.00th=[522191], 90.00th=[834667], 95.00th=[834667], | 99.00th=[834667], 99.50th=[834667], 99.90th=[834667], 99.95th=[834667], | 99.99th=[834667] bw ( KiB/s): min= 2048, max= 8192, per=98.88%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=1, BW=1553KiB/s (1591kB/s)(6144KiB/3955msec) clat (msec): min=61, max=1919, avg=409.59, stdev=743.38 lat (msec): min=61, max=1919, avg=409.64, stdev=743.39 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 70], | 30.00th=[ 70], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 77], | 70.00th=[ 253], 80.00th=[ 253], 90.00th=[ 1921], 95.00th=[ 1921], | 99.00th=[ 1921], 99.50th=[ 1921], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 2048, max= 4096, per=42.38%, avg=3072.00, stdev=1182.41, samples=4 iops : min= 2, max= 4, avg= 3.00, stdev= 1.15, samples=4 lat (usec) : 500=8.33% lat (msec) : 50=16.67%, 100=41.67%, 500=8.33%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=10.29%, ctx=632, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=583: Fri May 10 07:39:38 2024 read: IOPS=1, BW=1298KiB/s (1329kB/s)(5120KiB/3946msec) clat (msec): min=20, max=669, avg=204.63, stdev=271.49 lat (msec): min=20, max=669, avg=204.63, stdev=271.49 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 73], 60.00th=[ 73], | 70.00th=[ 222], 80.00th=[ 222], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1817KiB/s (1860kB/s)(7168KiB/3946msec) clat (msec): min=21, max=1901, avg=416.71, stdev=687.48 lat (msec): min=21, max=1901, avg=416.77, stdev=687.48 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 27], | 30.00th=[ 67], 40.00th=[ 67], 50.00th=[ 115], 60.00th=[ 161], | 70.00th=[ 161], 80.00th=[ 625], 90.00th=[ 1905], 95.00th=[ 1905], | 99.00th=[ 1905], 99.50th=[ 1905], 99.90th=[ 1905], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 2043, max= 6144, per=49.42%, avg=3582.75, stdev=1962.12, samples=4 iops : min= 1, max= 6, avg= 3.25, stdev= 2.22, samples=4 lat (msec) : 50=33.33%, 100=16.67%, 250=25.00%, 750=16.67% cpu : usr=0.00%, sys=9.93%, ctx=371, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5178KiB/s (5303kB/s), 825KiB/s-2214KiB/s (845kB/s-2267kB/s), io=20.0MiB (20.0MB), run=2481-3955msec WRITE: bw=7250KiB/s (7424kB/s), 1553KiB/s-4127KiB/s (1591kB/s-4226kB/s), io=28.0MiB (29.4MB), run=2481-3955msec rand-rw: (groupid=0, jobs=1): err= 0: pid=576: Fri May 10 07:39:38 2024 read: IOPS=2, BW=2977KiB/s (3048kB/s)(7168KiB/2408msec) clat (msec): min=22, max=890, avg=277.45, stdev=415.20 lat (msec): min=22, max=890, avg=277.45, stdev=415.20 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 24], | 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 45], 60.00th=[ 48], | 70.00th=[ 48], 80.00th=[ 877], 90.00th=[ 894], 95.00th=[ 894], | 99.00th=[ 894], 99.50th=[ 894], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 2048, max= 4096, per=68.47%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 write: IOPS=2, BW=2126KiB/s (2177kB/s)(5120KiB/2408msec) clat (msec): min=38, max=244, avg=90.58, stdev=86.79 lat (msec): min=38, max=244, avg=90.64, stdev=86.78 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 39], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 58], 60.00th=[ 58], | 70.00th=[ 70], 80.00th=[ 70], 90.00th=[ 245], 95.00th=[ 245], | 99.00th=[ 245], 99.50th=[ 245], 99.90th=[ 245], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 2048, max= 2048, per=29.35%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=16.67%, 250=8.33%, 1000=16.67% cpu : usr=0.00%, sys=5.36%, ctx=38, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=577: Fri May 10 07:39:38 2024 read: IOPS=0, BW=499KiB/s (511kB/s)(2048KiB/4108msec) clat (usec): min=27638, max=56128, avg=41883.35, stdev=20145.89 lat (usec): min=27642, max=56129, avg=41886.03, stdev=20143.55 clat percentiles (usec): | 1.00th=[27657], 5.00th=[27657], 10.00th=[27657], 20.00th=[27657], | 30.00th=[27657], 40.00th=[27657], 50.00th=[27657], 60.00th=[56361], | 70.00th=[56361], 80.00th=[56361], 90.00th=[56361], 95.00th=[56361], | 99.00th=[56361], 99.50th=[56361], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 2048, max= 2048, per=41.08%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2493KiB/s (2553kB/s)(10.0MiB/4108msec) clat (msec): min=38, max=991, avg=401.11, stdev=397.57 lat (msec): min=38, max=991, avg=401.20, stdev=397.55 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 42], | 30.00th=[ 66], 40.00th=[ 70], 50.00th=[ 77], 60.00th=[ 393], | 70.00th=[ 869], 80.00th=[ 869], 90.00th=[ 885], 95.00th=[ 995], | 99.00th=[ 995], 99.50th=[ 995], 99.90th=[ 995], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 2048, max= 4096, per=46.93%, avg=3275.00, stdev=1120.10, samples=5 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=5 lat (msec) : 50=25.00%, 100=33.33%, 500=8.33%, 750=8.33%, 1000=25.00% cpu : usr=0.02%, sys=3.85%, ctx=52, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=579: Fri May 10 07:39:38 2024 read: IOPS=1, BW=1504KiB/s (1540kB/s)(6144KiB/4085msec) clat (msec): min=32, max=767, avg=279.40, stdev=331.50 lat (msec): min=32, max=767, avg=279.40, stdev=331.50 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 40], | 30.00th=[ 40], 40.00th=[ 66], 50.00th=[ 66], 60.00th=[ 138], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 768], 95.00th=[ 768], | 99.00th=[ 768], 99.50th=[ 768], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 2043, max= 8192, per=100.00%, avg=5117.50, stdev=4348.00, samples=2 iops : min= 1, max= 8, avg= 4.50, stdev= 4.95, samples=2 write: IOPS=1, BW=1504KiB/s (1540kB/s)(6144KiB/4085msec) clat (msec): min=47, max=1876, avg=399.33, stdev=726.15 lat (msec): min=47, max=1876, avg=399.39, stdev=726.16 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 79], 50.00th=[ 79], 60.00th=[ 142], | 70.00th=[ 205], 80.00th=[ 205], 90.00th=[ 1871], 95.00th=[ 1871], | 99.00th=[ 1871], 99.50th=[ 1871], 99.90th=[ 1871], 99.95th=[ 1871], | 99.99th=[ 1871] bw ( KiB/s): min= 2043, max= 4096, per=44.00%, avg=3070.75, stdev=1183.86, samples=4 iops : min= 1, max= 4, avg= 2.75, stdev= 1.50, samples=4 lat (msec) : 50=33.33%, 100=16.67%, 250=25.00%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=2.64%, ctx=33, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=581: Fri May 10 07:39:38 2024 read: IOPS=1, BW=1278KiB/s (1309kB/s)(5120KiB/4005msec) clat (msec): min=14, max=821, avg=187.40, stdev=355.04 lat (msec): min=14, max=821, avg=187.40, stdev=355.04 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 17], 50.00th=[ 32], 60.00th=[ 32], | 70.00th=[ 52], 80.00th=[ 52], 90.00th=[ 818], 95.00th=[ 818], | 99.00th=[ 818], 99.50th=[ 818], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1790KiB/s (1833kB/s)(7168KiB/4005msec) clat (msec): min=58, max=1841, avg=435.65, stdev=651.58 lat (msec): min=58, max=1841, avg=435.72, stdev=651.58 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 62], | 30.00th=[ 68], 40.00th=[ 68], 50.00th=[ 104], 60.00th=[ 313], | 70.00th=[ 313], 80.00th=[ 600], 90.00th=[ 1838], 95.00th=[ 1838], | 99.00th=[ 1838], 99.50th=[ 1838], 99.90th=[ 1838], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 2043, max= 6144, per=41.07%, avg=2866.20, stdev=1832.35, samples=5 iops : min= 1, max= 6, avg= 2.60, stdev= 1.95, samples=5 lat (msec) : 20=16.67%, 50=8.33%, 100=33.33%, 250=8.33%, 500=8.33% lat (msec) : 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=3.57%, ctx=43, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4985KiB/s (5105kB/s), 499KiB/s-2977KiB/s (511kB/s-3048kB/s), io=20.0MiB (20.0MB), run=2408-4108msec WRITE: bw=6980KiB/s (7147kB/s), 1504KiB/s-2493KiB/s (1540kB/s-2553kB/s), io=28.0MiB (29.4MB), run=2408-4108msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=592: Fri May 10 07:39:41 2024 read: IOPS=1, BW=6533KiB/s (6689kB/s)(12.0MiB/1881msec) clat (msec): min=252, max=842, avg=625.67, stdev=324.50 lat (msec): min=252, max=842, avg=625.67, stdev=324.50 clat percentiles (msec): | 1.00th=[ 253], 5.00th=[ 253], 10.00th=[ 253], 20.00th=[ 253], | 30.00th=[ 253], 40.00th=[ 785], 50.00th=[ 785], 60.00th=[ 785], | 70.00th=[ 844], 80.00th=[ 844], 90.00th=[ 844], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 7501, max= 8192, per=95.21%, avg=7846.50, stdev=488.61, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 1000=66.67% cpu : usr=0.05%, sys=23.30%, ctx=646, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=593: Fri May 10 07:39:41 2024 read: IOPS=0, BW=2060KiB/s (2110kB/s)(4096KiB/1988msec) clat (nsec): min=126651k, max=126651k, avg=126650981.00, stdev= 0.00 lat (nsec): min=126654k, max=126654k, avg=126653506.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 127], 10.00th=[ 127], 20.00th=[ 127], | 30.00th=[ 127], 40.00th=[ 127], 50.00th=[ 127], 60.00th=[ 127], | 70.00th=[ 127], 80.00th=[ 127], 90.00th=[ 127], 95.00th=[ 127], | 99.00th=[ 127], 99.50th=[ 127], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 8175, max= 8175, per=99.20%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4121KiB/s (4220kB/s)(8192KiB/1988msec) clat (msec): min=96, max=1756, avg=926.12, stdev=1173.77 lat (msec): min=96, max=1756, avg=926.37, stdev=1173.73 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 96], 10.00th=[ 96], 20.00th=[ 96], | 30.00th=[ 96], 40.00th=[ 96], 50.00th=[ 96], 60.00th=[ 1754], | 70.00th=[ 1754], 80.00th=[ 1754], 90.00th=[ 1754], 95.00th=[ 1754], | 99.00th=[ 1754], 99.50th=[ 1754], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=11.73%, ctx=49, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=594: Fri May 10 07:39:41 2024 write: IOPS=1, BW=5905KiB/s (6047kB/s)(12.0MiB/2081msec) clat (msec): min=157, max=1084, avg=686.62, stdev=477.02 lat (msec): min=158, max=1084, avg=686.95, stdev=476.95 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 818], 50.00th=[ 818], 60.00th=[ 818], | 70.00th=[ 1083], 80.00th=[ 1083], 90.00th=[ 1083], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7262, max= 8192, per=49.07%, avg=7727.00, stdev=657.61, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=12.40%, ctx=68, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=595: Fri May 10 07:39:41 2024 write: IOPS=2, BW=8840KiB/s (9052kB/s)(12.0MiB/1390msec) clat (msec): min=112, max=1123, avg=454.53, stdev=579.82 lat (msec): min=112, max=1124, avg=454.86, stdev=579.77 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 113], 10.00th=[ 113], 20.00th=[ 113], | 30.00th=[ 113], 40.00th=[ 128], 50.00th=[ 128], 60.00th=[ 128], | 70.00th=[ 1116], 80.00th=[ 1116], 90.00th=[ 1116], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min=16351, max=16351, per=100.00%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=18.21%, ctx=104, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8241KiB/s (8439kB/s), 2060KiB/s-6533KiB/s (2110kB/s-6689kB/s), io=16.0MiB (16.8MB), run=1881-1988msec WRITE: bw=15.4MiB/s (16.1MB/s), 4121KiB/s-8840KiB/s (4220kB/s-9052kB/s), io=32.0MiB (33.6MB), run=1390-2081msec rand-rw: (groupid=0, jobs=1): err= 0: pid=596: Fri May 10 07:39:41 2024 read: IOPS=1, BW=5481KiB/s (5612kB/s)(12.0MiB/2242msec) clat (msec): min=308, max=1173, avg=746.88, stdev=432.43 lat (msec): min=308, max=1173, avg=746.88, stdev=432.43 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 309], 10.00th=[ 309], 20.00th=[ 309], | 30.00th=[ 309], 40.00th=[ 760], 50.00th=[ 760], 60.00th=[ 760], | 70.00th=[ 1167], 80.00th=[ 1167], 90.00th=[ 1167], 95.00th=[ 1167], | 99.00th=[ 1167], 99.50th=[ 1167], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=3.70%, ctx=39, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=597: Fri May 10 07:39:41 2024 read: IOPS=0, BW=1907KiB/s (1953kB/s)(4096KiB/2148msec) clat (nsec): min=1142.6M, max=1142.6M, avg=1142635896.00, stdev= 0.00 lat (nsec): min=1142.6M, max=1142.6M, avg=1142638314.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1150], 5.00th=[ 1150], 10.00th=[ 1150], 20.00th=[ 1150], | 30.00th=[ 1150], 40.00th=[ 1150], 50.00th=[ 1150], 60.00th=[ 1150], | 70.00th=[ 1150], 80.00th=[ 1150], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3814KiB/s (3905kB/s)(8192KiB/2148msec) clat (msec): min=76, max=877, avg=476.61, stdev=566.37 lat (msec): min=76, max=877, avg=476.89, stdev=566.33 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 77], 10.00th=[ 77], 20.00th=[ 77], | 30.00th=[ 77], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 877], | 70.00th=[ 877], 80.00th=[ 877], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] lat (msec) : 100=33.33%, 1000=33.33% cpu : usr=0.00%, sys=5.17%, ctx=30, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=598: Fri May 10 07:39:41 2024 write: IOPS=1, BW=5563KiB/s (5696kB/s)(12.0MiB/2209msec) clat (msec): min=322, max=1129, avg=734.06, stdev=403.96 lat (msec): min=322, max=1129, avg=734.36, stdev=403.84 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 321], 10.00th=[ 321], 20.00th=[ 321], | 30.00th=[ 321], 40.00th=[ 751], 50.00th=[ 751], 60.00th=[ 751], | 70.00th=[ 1133], 80.00th=[ 1133], 90.00th=[ 1133], 95.00th=[ 1133], | 99.00th=[ 1133], 99.50th=[ 1133], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=5.25%, ctx=72, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=599: Fri May 10 07:39:41 2024 write: IOPS=1, BW=5442KiB/s (5573kB/s)(12.0MiB/2258msec) clat (msec): min=157, max=1887, avg=740.81, stdev=993.45 lat (msec): min=157, max=1889, avg=741.44, stdev=994.02 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 178], 50.00th=[ 178], 60.00th=[ 178], | 70.00th=[ 1888], 80.00th=[ 1888], 90.00th=[ 1888], 95.00th=[ 1888], | 99.00th=[ 1888], 99.50th=[ 1888], 99.90th=[ 1888], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 8192, max= 8192, per=56.45%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.13%, sys=6.20%, ctx=190, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7308KiB/s (7483kB/s), 1907KiB/s-5481KiB/s (1953kB/s-5612kB/s), io=16.0MiB (16.8MB), run=2148-2242msec WRITE: bw=14.2MiB/s (14.9MB/s), 3814KiB/s-5563KiB/s (3905kB/s-5696kB/s), io=32.0MiB (33.6MB), run=2148-2258msec