== sanity test 398b: DIO and buffer IO race ============== 15:23:21 (1726860201) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.04041 s, 48.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25047: Fri Sep 20 15:23:36 2024 read: IOPS=119, BW=478KiB/s (489kB/s)(5992KiB/12543msec) clat (usec): min=47, max=74651, avg=3786.47, stdev=3715.52 lat (usec): min=47, max=74652, avg=3787.55, stdev=3715.52 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 61], 10.00th=[ 73], 20.00th=[ 1663], | 30.00th=[ 2409], 40.00th=[ 2769], 50.00th=[ 3195], 60.00th=[ 3785], | 70.00th=[ 4621], 80.00th=[ 5735], 90.00th=[ 7439], 95.00th=[ 9110], | 99.00th=[12649], 99.50th=[15401], 99.90th=[58983], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 184, max= 712, per=25.14%, avg=475.48, stdev=97.22, samples=25 iops : min= 46, max= 178, avg=118.72, stdev=24.29, samples=25 write: IOPS=125, BW=502KiB/s (514kB/s)(6296KiB/12543msec) clat (usec): min=448, max=138502, avg=4296.62, stdev=5236.74 lat (usec): min=448, max=138503, avg=4297.65, stdev=5236.63 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 701], | 30.00th=[ 1106], 40.00th=[ 2278], 50.00th=[ 3425], 60.00th=[ 4359], | 70.00th=[ 5473], 80.00th=[ 6849], 90.00th=[ 9110], 95.00th=[ 11207], | 99.00th=[ 18220], 99.50th=[ 21627], 99.90th=[ 37487], 99.95th=[139461], | 99.99th=[139461] bw ( KiB/s): min= 208, max= 712, per=25.22%, avg=499.40, stdev=92.75, samples=25 iops : min= 52, max= 178, avg=124.72, stdev=23.20, samples=25 lat (usec) : 50=0.36%, 100=7.55%, 250=1.46%, 500=0.88%, 750=10.71% lat (usec) : 1000=3.35% lat (msec) : 2=5.96%, 4=29.30%, 10=34.41%, 20=5.47%, 50=0.46% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.21%, sys=24.12%, ctx=6494, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25048: Fri Sep 20 15:23:36 2024 read: IOPS=116, BW=468KiB/s (479kB/s)(5940KiB/12693msec) clat (usec): min=45, max=26398, avg=3718.96, stdev=3006.26 lat (usec): min=45, max=26399, avg=3719.68, stdev=3006.44 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 60], 10.00th=[ 72], 20.00th=[ 231], | 30.00th=[ 2311], 40.00th=[ 2737], 50.00th=[ 3163], 60.00th=[ 3720], | 70.00th=[ 4752], 80.00th=[ 5932], 90.00th=[ 7635], 95.00th=[ 9110], | 99.00th=[13304], 99.50th=[14746], 99.90th=[24511], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 208, max= 816, per=24.65%, avg=466.12, stdev=111.19, samples=25 iops : min= 52, max= 204, avg=116.48, stdev=27.81, samples=25 write: IOPS=125, BW=500KiB/s (512kB/s)(6348KiB/12693msec) clat (usec): min=436, max=168919, avg=4451.90, stdev=5886.31 lat (usec): min=436, max=168920, avg=4452.73, stdev=5886.24 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 529], 10.00th=[ 562], 20.00th=[ 701], | 30.00th=[ 1057], 40.00th=[ 2442], 50.00th=[ 3490], 60.00th=[ 4359], | 70.00th=[ 5538], 80.00th=[ 7111], 90.00th=[ 9503], 95.00th=[ 12125], | 99.00th=[ 17957], 99.50th=[ 20579], 99.90th=[ 55837], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 264, max= 792, per=24.97%, avg=494.32, stdev=116.02, samples=25 iops : min= 66, max= 198, avg=123.52, stdev=29.04, samples=25 lat (usec) : 50=0.55%, 100=7.45%, 250=1.66%, 500=0.68%, 750=10.97% lat (usec) : 1000=3.22% lat (msec) : 2=5.92%, 4=29.04%, 10=34.44%, 20=5.70%, 50=0.29% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.14%, sys=23.72%, ctx=6440, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25049: Fri Sep 20 15:23:36 2024 read: IOPS=118, BW=475KiB/s (487kB/s)(6028KiB/12681msec) clat (usec): min=46, max=34461, avg=3694.28, stdev=3151.14 lat (usec): min=46, max=34462, avg=3695.56, stdev=3151.91 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 60], 10.00th=[ 72], 20.00th=[ 157], | 30.00th=[ 2343], 40.00th=[ 2769], 50.00th=[ 3195], 60.00th=[ 3785], | 70.00th=[ 4555], 80.00th=[ 5800], 90.00th=[ 7177], 95.00th=[ 8717], | 99.00th=[13566], 99.50th=[17171], 99.90th=[29492], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 192, max= 712, per=24.81%, avg=469.12, stdev=97.72, samples=25 iops : min= 48, max= 178, avg=117.16, stdev=24.42, samples=25 write: IOPS=123, BW=494KiB/s (505kB/s)(6260KiB/12681msec) clat (usec): min=421, max=190387, avg=4478.75, stdev=6231.09 lat (usec): min=422, max=190389, avg=4479.69, stdev=6231.15 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 562], 20.00th=[ 701], | 30.00th=[ 1123], 40.00th=[ 2606], 50.00th=[ 3556], 60.00th=[ 4359], | 70.00th=[ 5604], 80.00th=[ 7046], 90.00th=[ 9634], 95.00th=[ 12125], | 99.00th=[ 18220], 99.50th=[ 22676], 99.90th=[ 34341], 99.95th=[189793], | 99.99th=[189793] bw ( KiB/s): min= 272, max= 728, per=24.53%, avg=485.76, stdev=87.72, samples=25 iops : min= 68, max= 182, avg=121.32, stdev=21.90, samples=25 lat (usec) : 50=0.72%, 100=7.98%, 250=1.37%, 500=0.68%, 750=10.74% lat (usec) : 1000=3.19% lat (msec) : 2=5.83%, 4=29.17%, 10=34.11%, 20=5.66%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=23.82%, ctx=6431, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25050: Fri Sep 20 15:23:36 2024 read: IOPS=120, BW=484KiB/s (496kB/s)(6052KiB/12505msec) clat (usec): min=47, max=171325, avg=3884.45, stdev=5355.61 lat (usec): min=48, max=171325, avg=3885.55, stdev=5355.57 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 60], 10.00th=[ 73], 20.00th=[ 1762], | 30.00th=[ 2409], 40.00th=[ 2769], 50.00th=[ 3195], 60.00th=[ 3752], | 70.00th=[ 4686], 80.00th=[ 5932], 90.00th=[ 7767], 95.00th=[ 9110], | 99.00th=[ 13698], 99.50th=[ 15139], 99.90th=[ 49021], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 248, max= 664, per=25.59%, avg=483.92, stdev=92.41, samples=25 iops : min= 62, max= 166, avg=120.84, stdev=23.15, samples=25 write: IOPS=124, BW=499KiB/s (511kB/s)(6236KiB/12505msec) clat (usec): min=450, max=37667, avg=4185.35, stdev=4017.62 lat (usec): min=451, max=37668, avg=4186.61, stdev=4017.33 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 529], 10.00th=[ 562], 20.00th=[ 709], | 30.00th=[ 979], 40.00th=[ 2073], 50.00th=[ 3458], 60.00th=[ 4228], | 70.00th=[ 5407], 80.00th=[ 6915], 90.00th=[ 9372], 95.00th=[11994], | 99.00th=[15926], 99.50th=[23987], 99.90th=[32900], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 336, max= 648, per=25.17%, avg=498.28, stdev=67.66, samples=25 iops : min= 84, max= 162, avg=124.44, stdev=16.95, samples=25 lat (usec) : 50=0.46%, 100=7.78%, 250=1.30%, 500=0.62%, 750=10.35% lat (usec) : 1000=4.52% lat (msec) : 2=5.53%, 4=29.46%, 10=34.21%, 20=5.27%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=24.38%, ctx=6355, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1892KiB/s (1937kB/s), 468KiB/s-484KiB/s (479kB/s-496kB/s), io=23.4MiB (24.6MB), run=12505-12693msec WRITE: bw=1981KiB/s (2028kB/s), 494KiB/s-502KiB/s (505kB/s-514kB/s), io=24.6MiB (25.7MB), run=12505-12693msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25052: Fri Sep 20 15:23:57 2024 read: IOPS=45, BW=183KiB/s (188kB/s)(5992KiB/32680msec) clat (usec): min=1568, max=193212, avg=3602.23, stdev=5849.08 lat (usec): min=1568, max=193214, avg=3602.97, stdev=5849.10 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1827], 10.00th=[ 1893], 20.00th=[ 1991], | 30.00th=[ 2073], 40.00th=[ 2180], 50.00th=[ 2311], 60.00th=[ 2474], | 70.00th=[ 2868], 80.00th=[ 3752], 90.00th=[ 6849], 95.00th=[ 9503], | 99.00th=[ 17957], 99.50th=[ 23462], 99.90th=[ 34341], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 56, max= 344, per=25.07%, avg=182.98, stdev=71.54, samples=65 iops : min= 14, max= 86, avg=45.63, stdev=17.89, samples=65 write: IOPS=48, BW=193KiB/s (197kB/s)(6296KiB/32680msec) clat (usec): min=8257, max=55984, avg=17290.87, stdev=6723.70 lat (usec): min=8258, max=55985, avg=17291.78, stdev=6723.71 clat percentiles (usec): | 1.00th=[ 8979], 5.00th=[10028], 10.00th=[10683], 20.00th=[11731], | 30.00th=[12911], 40.00th=[14484], 50.00th=[16057], 60.00th=[17433], | 70.00th=[18744], 80.00th=[20841], 90.00th=[27132], 95.00th=[31589], | 99.00th=[39584], 99.50th=[41681], 99.90th=[51643], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 72, max= 264, per=25.19%, avg=192.46, stdev=60.14, samples=65 iops : min= 18, max= 66, avg=48.00, stdev=15.04, samples=65 lat (msec) : 2=10.48%, 4=28.97%, 10=9.93%, 20=38.22%, 50=12.27% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.09%, sys=5.98%, ctx=3454, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25053: Fri Sep 20 15:23:57 2024 read: IOPS=45, BW=181KiB/s (185kB/s)(5940KiB/32862msec) clat (usec): min=1605, max=136545, avg=3609.64, stdev=4637.68 lat (usec): min=1606, max=136547, avg=3610.52, stdev=4637.74 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1827], 10.00th=[ 1876], 20.00th=[ 1958], | 30.00th=[ 2057], 40.00th=[ 2147], 50.00th=[ 2311], 60.00th=[ 2507], | 70.00th=[ 2868], 80.00th=[ 3851], 90.00th=[ 7504], 95.00th=[ 10421], | 99.00th=[ 15926], 99.50th=[ 19268], 99.90th=[ 33817], 99.95th=[137364], | 99.99th=[137364] bw ( KiB/s): min= 16, max= 368, per=24.78%, avg=180.91, stdev=77.11, samples=65 iops : min= 4, max= 92, avg=45.12, stdev=19.29, samples=65 write: IOPS=48, BW=193KiB/s (198kB/s)(6348KiB/32862msec) clat (usec): min=8330, max=85199, avg=17286.05, stdev=6841.69 lat (usec): min=8331, max=85201, avg=17286.99, stdev=6841.65 clat percentiles (usec): | 1.00th=[ 9110], 5.00th=[10290], 10.00th=[10814], 20.00th=[11994], | 30.00th=[13173], 40.00th=[14615], 50.00th=[16057], 60.00th=[17171], | 70.00th=[18482], 80.00th=[20317], 90.00th=[25822], 95.00th=[30540], | 99.00th=[41157], 99.50th=[44303], 99.90th=[73925], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 80, max= 272, per=25.21%, avg=192.60, stdev=58.32, samples=65 iops : min= 20, max= 68, avg=48.05, stdev=14.59, samples=65 lat (msec) : 2=12.01%, 4=26.86%, 10=8.69%, 20=40.95%, 50=11.36% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.12%, sys=5.89%, ctx=3426, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25054: Fri Sep 20 15:23:57 2024 read: IOPS=45, BW=183KiB/s (188kB/s)(6028KiB/32870msec) clat (usec): min=1560, max=40207, avg=3564.94, stdev=3178.17 lat (usec): min=1561, max=40208, avg=3565.73, stdev=3178.16 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1811], 10.00th=[ 1876], 20.00th=[ 1991], | 30.00th=[ 2089], 40.00th=[ 2212], 50.00th=[ 2343], 60.00th=[ 2573], | 70.00th=[ 2966], 80.00th=[ 4113], 90.00th=[ 7308], 95.00th=[ 9896], | 99.00th=[15401], 99.50th=[19268], 99.90th=[34341], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 39, max= 367, per=25.13%, avg=183.46, stdev=67.40, samples=65 iops : min= 9, max= 91, avg=45.74, stdev=16.85, samples=65 write: IOPS=47, BW=190KiB/s (195kB/s)(6260KiB/32870msec) clat (msec): min=8, max=188, avg=17.55, stdev= 7.88 lat (msec): min=8, max=188, avg=17.55, stdev= 7.88 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 27], 95.00th=[ 31], | 99.00th=[ 40], 99.50th=[ 46], 99.90th=[ 74], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 87, max= 256, per=24.87%, avg=189.98, stdev=58.25, samples=65 iops : min= 21, max= 64, avg=47.37, stdev=14.57, samples=65 lat (msec) : 2=10.25%, 4=28.74%, 10=9.28%, 20=39.65%, 50=11.98% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.12%, sys=5.96%, ctx=3454, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25055: Fri Sep 20 15:23:57 2024 read: IOPS=46, BW=185KiB/s (189kB/s)(6052KiB/32723msec) clat (usec): min=1562, max=28858, avg=3466.77, stdev=3077.56 lat (usec): min=1563, max=28859, avg=3467.55, stdev=3077.56 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1827], 10.00th=[ 1876], 20.00th=[ 1958], | 30.00th=[ 2057], 40.00th=[ 2180], 50.00th=[ 2278], 60.00th=[ 2507], | 70.00th=[ 2835], 80.00th=[ 3851], 90.00th=[ 7046], 95.00th=[ 9372], | 99.00th=[15926], 99.50th=[21627], 99.90th=[28181], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 40, max= 336, per=25.31%, avg=184.78, stdev=73.75, samples=65 iops : min= 10, max= 84, avg=46.14, stdev=18.44, samples=65 write: IOPS=47, BW=191KiB/s (195kB/s)(6236KiB/32723msec) clat (msec): min=8, max=145, avg=17.58, stdev= 7.48 lat (msec): min=8, max=145, avg=17.58, stdev= 7.48 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 22], 90.00th=[ 27], 95.00th=[ 32], | 99.00th=[ 40], 99.50th=[ 45], 99.90th=[ 77], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 72, max= 264, per=24.91%, avg=190.32, stdev=57.05, samples=65 iops : min= 18, max= 66, avg=47.52, stdev=14.29, samples=65 lat (msec) : 2=11.56%, 4=28.19%, 10=9.21%, 20=38.28%, 50=12.60% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.13%, sys=5.92%, ctx=3469, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=731KiB/s (748kB/s), 181KiB/s-185KiB/s (185kB/s-189kB/s), io=23.4MiB (24.6MB), run=32680-32870msec WRITE: bw=765KiB/s (783kB/s), 190KiB/s-193KiB/s (195kB/s-198kB/s), io=24.6MiB (25.7MB), run=32680-32870msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25095: Fri Sep 20 15:24:03 2024 read: IOPS=76, BW=1221KiB/s (1250kB/s)(5920KiB/4850msec) clat (usec): min=52, max=40665, avg=9630.15, stdev=5225.48 lat (usec): min=52, max=40666, avg=9630.86, stdev=5225.44 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 64], 10.00th=[ 77], 20.00th=[ 7635], | 30.00th=[ 8717], 40.00th=[ 9634], 50.00th=[10421], 60.00th=[10945], | 70.00th=[11863], 80.00th=[12518], 90.00th=[14222], 95.00th=[15795], | 99.00th=[22938], 99.50th=[40633], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 736, max= 1472, per=26.14%, avg=1191.11, stdev=228.96, samples=9 iops : min= 46, max= 92, avg=74.44, stdev=14.31, samples=9 write: IOPS=82, BW=1313KiB/s (1345kB/s)(6368KiB/4850msec) clat (usec): min=628, max=25660, avg=3015.57, stdev=3307.01 lat (usec): min=629, max=25661, avg=3016.83, stdev=3307.12 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 750], 10.00th=[ 775], 20.00th=[ 832], | 30.00th=[ 922], 40.00th=[ 1090], 50.00th=[ 1631], 60.00th=[ 2671], | 70.00th=[ 3458], 80.00th=[ 4621], 90.00th=[ 6521], 95.00th=[ 8455], | 99.00th=[17171], 99.50th=[20317], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 800, max= 1824, per=27.15%, avg=1301.33, stdev=335.24, samples=9 iops : min= 50, max= 114, avg=81.33, stdev=20.95, samples=9 lat (usec) : 100=6.12%, 250=0.91%, 500=0.13%, 750=2.60%, 1000=16.28% lat (msec) : 2=8.33%, 4=11.72%, 10=25.65%, 20=27.34%, 50=0.91% cpu : usr=0.27%, sys=21.82%, ctx=2095, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25096: Fri Sep 20 15:24:03 2024 read: IOPS=71, BW=1147KiB/s (1175kB/s)(5744KiB/5006msec) clat (usec): min=51, max=31699, avg=9702.63, stdev=5309.84 lat (usec): min=52, max=31699, avg=9703.27, stdev=5309.83 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 64], 10.00th=[ 78], 20.00th=[ 7635], | 30.00th=[ 8717], 40.00th=[ 9634], 50.00th=[10552], 60.00th=[11338], | 70.00th=[12125], 80.00th=[13173], 90.00th=[15008], 95.00th=[17171], | 99.00th=[22938], 99.50th=[25560], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 416, max= 1664, per=24.98%, avg=1138.30, stdev=311.58, samples=10 iops : min= 26, max= 104, avg=70.90, stdev=19.42, samples=10 write: IOPS=81, BW=1307KiB/s (1339kB/s)(6544KiB/5006msec) clat (usec): min=655, max=94601, avg=3511.23, stdev=5918.18 lat (usec): min=656, max=94602, avg=3512.99, stdev=5917.97 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 750], 10.00th=[ 791], 20.00th=[ 881], | 30.00th=[ 955], 40.00th=[ 1205], 50.00th=[ 2212], 60.00th=[ 2835], | 70.00th=[ 3392], 80.00th=[ 4948], 90.00th=[ 7177], 95.00th=[ 9896], | 99.00th=[22414], 99.50th=[26608], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 672, max= 2016, per=27.15%, avg=1301.40, stdev=436.28, samples=10 iops : min= 42, max= 126, avg=81.10, stdev=27.21, samples=10 lat (usec) : 100=6.25%, 250=1.69%, 750=2.73%, 1000=14.84% lat (msec) : 2=8.33%, 4=14.19%, 10=22.40%, 20=27.73%, 50=1.69% lat (msec) : 100=0.13% cpu : usr=0.16%, sys=20.62%, ctx=2063, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25097: Fri Sep 20 15:24:03 2024 read: IOPS=74, BW=1190KiB/s (1219kB/s)(6256KiB/5257msec) clat (usec): min=52, max=54731, avg=10177.92, stdev=5704.85 lat (usec): min=52, max=54732, avg=10178.70, stdev=5704.89 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 69], 10.00th=[ 90], 20.00th=[ 7963], | 30.00th=[ 8979], 40.00th=[ 9896], 50.00th=[10683], 60.00th=[11469], | 70.00th=[12256], 80.00th=[13042], 90.00th=[15008], 95.00th=[16581], | 99.00th=[28443], 99.50th=[33817], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 768, max= 1589, per=25.74%, avg=1172.70, stdev=210.13, samples=10 iops : min= 48, max= 99, avg=73.10, stdev=13.14, samples=10 write: IOPS=71, BW=1147KiB/s (1175kB/s)(6032KiB/5257msec) clat (usec): min=669, max=65315, avg=3340.25, stdev=4751.66 lat (usec): min=669, max=65317, avg=3341.41, stdev=4751.68 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 857], | 30.00th=[ 979], 40.00th=[ 1205], 50.00th=[ 1926], 60.00th=[ 2769], | 70.00th=[ 3392], 80.00th=[ 4686], 90.00th=[ 6456], 95.00th=[10814], | 99.00th=[20841], 99.50th=[22676], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 800, max= 1504, per=23.27%, avg=1115.50, stdev=256.46, samples=10 iops : min= 50, max= 94, avg=69.50, stdev=16.07, samples=10 lat (usec) : 100=6.25%, 250=1.04%, 750=2.47%, 1000=12.76% lat (msec) : 2=9.37%, 4=12.37%, 10=22.92%, 20=30.86%, 50=1.69% lat (msec) : 100=0.26% cpu : usr=0.17%, sys=21.00%, ctx=2213, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25098: Fri Sep 20 15:24:03 2024 read: IOPS=77, BW=1233KiB/s (1263kB/s)(6032KiB/4892msec) clat (usec): min=53, max=28363, avg=9581.78, stdev=4816.14 lat (usec): min=53, max=28364, avg=9582.45, stdev=4816.16 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 64], 10.00th=[ 84], 20.00th=[ 7898], | 30.00th=[ 8848], 40.00th=[ 9503], 50.00th=[10290], 60.00th=[11076], | 70.00th=[11994], 80.00th=[12911], 90.00th=[14353], 95.00th=[15795], | 99.00th=[20055], 99.50th=[22414], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 928, max= 1373, per=26.13%, avg=1190.44, stdev=159.84, samples=9 iops : min= 58, max= 85, avg=74.22, stdev= 9.78, samples=9 write: IOPS=79, BW=1279KiB/s (1310kB/s)(6256KiB/4892msec) clat (usec): min=624, max=26255, avg=3196.63, stdev=3689.85 lat (usec): min=625, max=26256, avg=3197.74, stdev=3689.88 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 758], 10.00th=[ 791], 20.00th=[ 840], | 30.00th=[ 922], 40.00th=[ 1106], 50.00th=[ 2114], 60.00th=[ 2737], | 70.00th=[ 3392], 80.00th=[ 4817], 90.00th=[ 6915], 95.00th=[10159], | 99.00th=[21365], 99.50th=[24773], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 958, max= 1856, per=26.32%, avg=1261.67, stdev=266.95, samples=9 iops : min= 59, max= 116, avg=78.67, stdev=16.77, samples=9 lat (usec) : 100=5.73%, 250=1.69%, 750=2.21%, 1000=15.49% lat (msec) : 2=7.81%, 4=13.02%, 10=24.87%, 20=28.26%, 50=0.91% cpu : usr=0.12%, sys=21.92%, ctx=2100, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4556KiB/s (4666kB/s), 1147KiB/s-1233KiB/s (1175kB/s-1263kB/s), io=23.4MiB (24.5MB), run=4850-5257msec WRITE: bw=4794KiB/s (4909kB/s), 1147KiB/s-1313KiB/s (1175kB/s-1345kB/s), io=24.6MiB (25.8MB), run=4850-5257msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25099: Fri Sep 20 15:24:06 2024 read: IOPS=42, BW=678KiB/s (694kB/s)(5920KiB/8738msec) clat (usec): min=1879, max=42912, avg=4216.77, stdev=4011.55 lat (usec): min=1880, max=42913, avg=4217.62, stdev=4011.69 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2089], 10.00th=[ 2147], 20.00th=[ 2245], | 30.00th=[ 2343], 40.00th=[ 2507], 50.00th=[ 2671], 60.00th=[ 2999], | 70.00th=[ 3654], 80.00th=[ 5014], 90.00th=[ 8356], 95.00th=[11731], | 99.00th=[19268], 99.50th=[30540], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 352, max= 1280, per=24.72%, avg=660.71, stdev=221.97, samples=17 iops : min= 22, max= 80, avg=41.29, stdev=13.87, samples=17 write: IOPS=45, BW=729KiB/s (746kB/s)(6368KiB/8738msec) clat (usec): min=9111, max=45610, avg=17850.45, stdev=5962.89 lat (usec): min=9112, max=45612, avg=17852.33, stdev=5962.70 clat percentiles (usec): | 1.00th=[ 9372], 5.00th=[10814], 10.00th=[11469], 20.00th=[12780], | 30.00th=[14091], 40.00th=[15270], 50.00th=[16712], 60.00th=[18220], | 70.00th=[19792], 80.00th=[21890], 90.00th=[26084], 95.00th=[28705], | 99.00th=[39060], 99.50th=[42730], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 320, max= 992, per=25.63%, avg=720.94, stdev=194.36, samples=17 iops : min= 20, max= 62, avg=45.06, stdev=12.15, samples=17 lat (msec) : 2=0.65%, 4=34.37%, 10=10.68%, 20=39.32%, 50=14.97% cpu : usr=0.16%, sys=6.50%, ctx=898, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25100: Fri Sep 20 15:24:06 2024 read: IOPS=40, BW=641KiB/s (657kB/s)(5744KiB/8958msec) clat (usec): min=1856, max=27570, avg=4060.60, stdev=3574.17 lat (usec): min=1857, max=27570, avg=4061.34, stdev=3574.19 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 1991], 10.00th=[ 2114], 20.00th=[ 2245], | 30.00th=[ 2343], 40.00th=[ 2474], 50.00th=[ 2671], 60.00th=[ 2868], | 70.00th=[ 3294], 80.00th=[ 4686], 90.00th=[ 8291], 95.00th=[12387], | 99.00th=[19792], 99.50th=[20841], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 224, max= 1248, per=23.58%, avg=630.24, stdev=289.45, samples=17 iops : min= 14, max= 78, avg=39.24, stdev=18.07, samples=17 write: IOPS=45, BW=731KiB/s (748kB/s)(6544KiB/8958msec) clat (msec): min=8, max=122, avg=18.28, stdev= 8.02 lat (msec): min=8, max=122, avg=18.28, stdev= 8.02 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 19], | 70.00th=[ 20], 80.00th=[ 23], 90.00th=[ 26], 95.00th=[ 31], | 99.00th=[ 40], 99.50th=[ 42], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 448, max= 1024, per=25.42%, avg=714.94, stdev=187.56, samples=17 iops : min= 28, max= 64, avg=44.53, stdev=11.71, samples=17 lat (msec) : 2=2.47%, 4=33.46%, 10=7.94%, 20=40.49%, 50=15.49% lat (msec) : 250=0.13% cpu : usr=0.10%, sys=6.37%, ctx=886, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25101: Fri Sep 20 15:24:06 2024 read: IOPS=45, BW=729KiB/s (747kB/s)(6256KiB/8577msec) clat (usec): min=1784, max=23082, avg=4205.43, stdev=3455.36 lat (usec): min=1785, max=23083, avg=4206.31, stdev=3455.34 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2008], 10.00th=[ 2089], 20.00th=[ 2212], | 30.00th=[ 2376], 40.00th=[ 2540], 50.00th=[ 2769], 60.00th=[ 3032], | 70.00th=[ 3851], 80.00th=[ 5014], 90.00th=[ 9372], 95.00th=[11994], | 99.00th=[21103], 99.50th=[22152], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 384, max= 1472, per=27.46%, avg=734.12, stdev=292.73, samples=17 iops : min= 24, max= 92, avg=45.88, stdev=18.30, samples=17 write: IOPS=43, BW=703KiB/s (720kB/s)(6032KiB/8577msec) clat (usec): min=9102, max=40817, avg=18149.76, stdev=5823.15 lat (usec): min=9104, max=40820, avg=18151.24, stdev=5823.20 clat percentiles (usec): | 1.00th=[ 9503], 5.00th=[10945], 10.00th=[11731], 20.00th=[13304], | 30.00th=[14877], 40.00th=[16188], 50.00th=[17171], 60.00th=[18220], | 70.00th=[19792], 80.00th=[22152], 90.00th=[25560], 95.00th=[28705], | 99.00th=[38536], 99.50th=[40109], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 448, max= 928, per=24.76%, avg=696.47, stdev=171.03, samples=17 iops : min= 28, max= 58, avg=43.53, stdev=10.69, samples=17 lat (msec) : 2=2.47%, 4=33.72%, 10=11.72%, 20=37.50%, 50=14.58% cpu : usr=0.08%, sys=7.03%, ctx=903, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25102: Fri Sep 20 15:24:06 2024 read: IOPS=43, BW=694KiB/s (711kB/s)(6032KiB/8691msec) clat (usec): min=1796, max=29702, avg=4086.27, stdev=3566.26 lat (usec): min=1797, max=29703, avg=4087.02, stdev=3566.27 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2024], 10.00th=[ 2114], 20.00th=[ 2245], | 30.00th=[ 2376], 40.00th=[ 2507], 50.00th=[ 2704], 60.00th=[ 2999], | 70.00th=[ 3425], 80.00th=[ 4686], 90.00th=[ 8455], 95.00th=[11469], | 99.00th=[18744], 99.50th=[22676], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 351, max= 1024, per=25.41%, avg=679.24, stdev=195.33, samples=17 iops : min= 21, max= 64, avg=42.29, stdev=12.34, samples=17 write: IOPS=44, BW=720KiB/s (737kB/s)(6256KiB/8691msec) clat (usec): min=9132, max=47249, avg=18063.94, stdev=5698.80 lat (usec): min=9134, max=47253, avg=18065.53, stdev=5698.89 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11207], 10.00th=[11863], 20.00th=[13042], | 30.00th=[14615], 40.00th=[15926], 50.00th=[17171], 60.00th=[18482], | 70.00th=[19530], 80.00th=[21890], 90.00th=[25560], 95.00th=[28705], | 99.00th=[37487], 99.50th=[40633], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 383, max= 958, per=25.42%, avg=715.00, stdev=179.56, samples=17 iops : min= 23, max= 59, avg=44.53, stdev=11.28, samples=17 lat (msec) : 2=1.56%, 4=36.20%, 10=7.81%, 20=39.84%, 50=14.58% cpu : usr=0.08%, sys=6.58%, ctx=901, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2674KiB/s (2738kB/s), 641KiB/s-729KiB/s (657kB/s-747kB/s), io=23.4MiB (24.5MB), run=8577-8958msec WRITE: bw=2813KiB/s (2881kB/s), 703KiB/s-731KiB/s (720kB/s-748kB/s), io=24.6MiB (25.8MB), run=8577-8958msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25118: Fri Sep 20 15:24:12 2024 read: IOPS=1, BW=1514KiB/s (1551kB/s)(7168KiB/4733msec) clat (msec): min=8, max=889, avg=619.48, stdev=405.03 lat (msec): min=8, max=889, avg=619.48, stdev=405.03 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 9], 10.00th=[ 9], 20.00th=[ 48], | 30.00th=[ 810], 40.00th=[ 810], 50.00th=[ 844], 60.00th=[ 860], | 70.00th=[ 860], 80.00th=[ 885], 90.00th=[ 894], 95.00th=[ 894], | 99.00th=[ 894], 99.50th=[ 894], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 1768, max= 4096, per=55.65%, avg=2400.60, stdev=955.39, samples=5 iops : min= 1, max= 4, avg= 2.00, stdev= 1.22, samples=5 write: IOPS=1, BW=1082KiB/s (1108kB/s)(5120KiB/4733msec) clat (msec): min=29, max=162, avg=67.41, stdev=54.13 lat (msec): min=29, max=162, avg=67.48, stdev=54.14 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 30], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 50], 60.00th=[ 50], | 70.00th=[ 57], 80.00th=[ 57], 90.00th=[ 163], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 1768, max= 4096, per=40.05%, avg=2418.75, stdev=1125.66, samples=4 iops : min= 1, max= 4, avg= 1.75, stdev= 1.50, samples=4 lat (msec) : 10=8.33%, 50=33.33%, 100=8.33%, 250=8.33%, 1000=41.67% cpu : usr=0.02%, sys=17.56%, ctx=1458, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25120: Fri Sep 20 15:24:12 2024 read: IOPS=0, BW=657KiB/s (673kB/s)(2048KiB/3117msec) clat (msec): min=78, max=656, avg=367.35, stdev=409.16 lat (msec): min=78, max=656, avg=367.36, stdev=409.16 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 79], 10.00th=[ 79], 20.00th=[ 79], | 30.00th=[ 79], 40.00th=[ 79], 50.00th=[ 79], 60.00th=[ 659], | 70.00th=[ 659], 80.00th=[ 659], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 2048, max= 2048, per=47.47%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3285KiB/s (3364kB/s)(10.0MiB/3117msec) clat (msec): min=22, max=1930, avg=233.34, stdev=596.61 lat (msec): min=22, max=1930, avg=233.40, stdev=596.61 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 25], | 30.00th=[ 26], 40.00th=[ 28], 50.00th=[ 41], 60.00th=[ 42], | 70.00th=[ 66], 80.00th=[ 66], 90.00th=[ 96], 95.00th=[ 1938], | 99.00th=[ 1938], 99.50th=[ 1938], 99.90th=[ 1938], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 4096, max=10240, per=100.00%, avg=7168.00, stdev=4344.46, samples=2 iops : min= 4, max= 10, avg= 7.00, stdev= 4.24, samples=2 lat (msec) : 50=50.00%, 100=33.33%, 750=8.33% cpu : usr=0.00%, sys=10.30%, ctx=334, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25121: Fri Sep 20 15:24:12 2024 read: IOPS=1, BW=1294KiB/s (1325kB/s)(6144KiB/4747msec) clat (usec): min=373, max=831423, avg=289369.14, stdev=411626.31 lat (usec): min=374, max=831425, avg=289370.83, stdev=411625.97 clat percentiles (usec): | 1.00th=[ 375], 5.00th=[ 375], 10.00th=[ 375], 20.00th=[ 734], | 30.00th=[ 734], 40.00th=[ 840], 50.00th=[ 840], 60.00th=[ 96994], | 70.00th=[809501], 80.00th=[809501], 90.00th=[834667], 95.00th=[834667], | 99.00th=[834667], 99.50th=[834667], 99.90th=[834667], 99.95th=[834667], | 99.99th=[834667] bw ( KiB/s): min= 4096, max= 5278, per=100.00%, avg=4687.00, stdev=835.80, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=1, BW=1294KiB/s (1325kB/s)(6144KiB/4747msec) clat (msec): min=32, max=1875, avg=500.33, stdev=753.86 lat (msec): min=32, max=1875, avg=500.38, stdev=753.87 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 54], 50.00th=[ 54], 60.00th=[ 92], | 70.00th=[ 902], 80.00th=[ 902], 90.00th=[ 1871], 95.00th=[ 1871], | 99.00th=[ 1871], 99.50th=[ 1871], 99.90th=[ 1871], 99.95th=[ 1871], | 99.99th=[ 1871] bw ( KiB/s): min= 2048, max= 4096, per=48.47%, avg=2927.50, stdev=1042.61, samples=4 iops : min= 2, max= 4, avg= 2.75, stdev= 0.96, samples=4 lat (usec) : 500=8.33%, 750=8.33%, 1000=8.33% lat (msec) : 50=16.67%, 100=25.00%, 1000=25.00% cpu : usr=0.00%, sys=9.27%, ctx=615, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25122: Fri Sep 20 15:24:12 2024 read: IOPS=1, BW=1649KiB/s (1689kB/s)(5120KiB/3105msec) clat (usec): min=378, max=835095, avg=177641.10, stdev=368198.62 lat (usec): min=378, max=835096, avg=177641.93, stdev=368198.58 clat percentiles (usec): | 1.00th=[ 379], 5.00th=[ 379], 10.00th=[ 379], 20.00th=[ 379], | 30.00th=[ 482], 40.00th=[ 482], 50.00th=[ 502], 60.00th=[ 502], | 70.00th=[ 51643], 80.00th=[ 51643], 90.00th=[834667], 95.00th=[834667], | 99.00th=[834667], 99.50th=[834667], 99.90th=[834667], 99.95th=[834667], | 99.99th=[834667] bw ( KiB/s): min= 2048, max= 6144, per=94.95%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=2, BW=2309KiB/s (2364kB/s)(7168KiB/3105msec) clat (msec): min=16, max=1847, avg=311.82, stdev=678.07 lat (msec): min=16, max=1847, avg=311.89, stdev=678.08 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 21], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 74], 60.00th=[ 79], | 70.00th=[ 79], 80.00th=[ 111], 90.00th=[ 1854], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 2048, max= 4096, per=42.38%, avg=2560.00, stdev=1024.00, samples=4 iops : min= 2, max= 4, avg= 2.50, stdev= 1.00, samples=4 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 20=8.33%, 50=16.67%, 100=25.00%, 250=8.33%, 1000=8.33% cpu : usr=0.03%, sys=13.08%, ctx=300, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4314KiB/s (4418kB/s), 657KiB/s-1649KiB/s (673kB/s-1689kB/s), io=20.0MiB (20.0MB), run=3105-4747msec WRITE: bw=6040KiB/s (6185kB/s), 1082KiB/s-3285KiB/s (1108kB/s-3364kB/s), io=28.0MiB (29.4MB), run=3105-4747msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25115: Fri Sep 20 15:24:12 2024 read: IOPS=1, BW=1486KiB/s (1522kB/s)(7168KiB/4824msec) clat (msec): min=15, max=2613, avg=413.10, stdev=971.27 lat (msec): min=15, max=2613, avg=413.10, stdev=971.27 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 22], 40.00th=[ 22], 50.00th=[ 45], 60.00th=[ 50], | 70.00th=[ 50], 80.00th=[ 131], 90.00th=[ 2601], 95.00th=[ 2601], | 99.00th=[ 2601], 99.50th=[ 2601], 99.90th=[ 2601], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 1777, max= 6144, per=94.36%, avg=4005.67, stdev=2184.90, samples=3 iops : min= 1, max= 6, avg= 3.67, stdev= 2.52, samples=3 write: IOPS=1, BW=1061KiB/s (1087kB/s)(5120KiB/4824msec) clat (msec): min=54, max=906, avg=379.68, stdev=443.55 lat (msec): min=54, max=906, avg=379.75, stdev=443.53 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 55], 10.00th=[ 55], 20.00th=[ 55], | 30.00th=[ 56], 40.00th=[ 56], 50.00th=[ 61], 60.00th=[ 61], | 70.00th=[ 827], 80.00th=[ 827], 90.00th=[ 911], 95.00th=[ 911], | 99.00th=[ 911], 99.50th=[ 911], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 2048, max= 3555, per=42.91%, avg=2550.33, stdev=870.07, samples=3 iops : min= 2, max= 3, avg= 2.33, stdev= 0.58, samples=3 lat (msec) : 20=16.67%, 50=25.00%, 100=25.00%, 250=8.33%, 1000=16.67% cpu : usr=0.00%, sys=2.78%, ctx=59, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25116: Fri Sep 20 15:24:12 2024 read: IOPS=0, BW=641KiB/s (656kB/s)(2048KiB/3195msec) clat (msec): min=75, max=1758, avg=916.93, stdev=1189.55 lat (msec): min=75, max=1758, avg=916.93, stdev=1189.55 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 77], 10.00th=[ 77], 20.00th=[ 77], | 30.00th=[ 77], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 1754], | 70.00th=[ 1754], 80.00th=[ 1754], 90.00th=[ 1754], 95.00th=[ 1754], | 99.00th=[ 1754], 99.50th=[ 1754], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 2048, max= 2048, per=48.24%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3205KiB/s (3282kB/s)(10.0MiB/3195msec) clat (msec): min=23, max=995, avg=132.43, stdev=303.69 lat (msec): min=23, max=995, avg=132.51, stdev=303.67 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 32], | 70.00th=[ 45], 80.00th=[ 45], 90.00th=[ 73], 95.00th=[ 995], | 99.00th=[ 995], 99.50th=[ 995], 99.90th=[ 995], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=66.67%, 100=16.67%, 1000=8.33% cpu : usr=0.03%, sys=3.16%, ctx=53, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25117: Fri Sep 20 15:24:12 2024 read: IOPS=1, BW=2010KiB/s (2059kB/s)(6144KiB/3056msec) clat (msec): min=16, max=719, avg=156.81, stdev=276.40 lat (msec): min=16, max=719, avg=156.81, stdev=276.40 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 46], 50.00th=[ 46], 60.00th=[ 66], | 70.00th=[ 72], 80.00th=[ 72], 90.00th=[ 718], 95.00th=[ 718], | 99.00th=[ 718], 99.50th=[ 718], 99.90th=[ 718], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 2043, max= 4096, per=60.28%, avg=2558.75, stdev=1024.84, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 write: IOPS=1, BW=2010KiB/s (2059kB/s)(6144KiB/3056msec) clat (msec): min=31, max=1759, avg=351.31, stdev=692.74 lat (msec): min=31, max=1759, avg=351.40, stdev=692.74 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 33], 60.00th=[ 63], | 70.00th=[ 190], 80.00th=[ 190], 90.00th=[ 1754], 95.00th=[ 1754], | 99.00th=[ 1754], 99.50th=[ 1754], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 2043, max= 6144, per=51.67%, avg=3070.75, stdev=2048.83, samples=4 iops : min= 1, max= 6, avg= 2.75, stdev= 2.22, samples=4 lat (msec) : 20=8.33%, 50=41.67%, 100=25.00%, 250=8.33%, 750=8.33% cpu : usr=0.03%, sys=2.45%, ctx=51, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25119: Fri Sep 20 15:24:12 2024 read: IOPS=1, BW=1689KiB/s (1730kB/s)(5120KiB/3031msec) clat (usec): min=11852, max=65888, avg=26587.98, stdev=23203.44 lat (usec): min=11853, max=65890, avg=26589.03, stdev=23203.63 clat percentiles (usec): | 1.00th=[11863], 5.00th=[11863], 10.00th=[11863], 20.00th=[11863], | 30.00th=[12780], 40.00th=[12780], 50.00th=[12780], 60.00th=[12780], | 70.00th=[29754], 80.00th=[29754], 90.00th=[65799], 95.00th=[65799], | 99.00th=[65799], 99.50th=[65799], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2365KiB/s (2422kB/s)(7168KiB/3031msec) clat (msec): min=36, max=2546, avg=412.88, stdev=941.35 lat (msec): min=36, max=2546, avg=412.97, stdev=941.35 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 39], | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 42], 60.00th=[ 70], | 70.00th=[ 70], 80.00th=[ 118], 90.00th=[ 2534], 95.00th=[ 2534], | 99.00th=[ 2534], 99.50th=[ 2534], 99.90th=[ 2534], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 4096, max=10240, per=100.00%, avg=7168.00, stdev=4344.46, samples=2 iops : min= 4, max= 10, avg= 7.00, stdev= 4.24, samples=2 lat (msec) : 20=25.00%, 50=41.67%, 100=16.67%, 250=8.33% cpu : usr=0.00%, sys=3.37%, ctx=44, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4245KiB/s (4347kB/s), 641KiB/s-2010KiB/s (656kB/s-2059kB/s), io=20.0MiB (20.0MB), run=3031-4824msec WRITE: bw=5944KiB/s (6086kB/s), 1061KiB/s-3205KiB/s (1087kB/s-3282kB/s), io=28.0MiB (29.4MB), run=3031-4824msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25131: Fri Sep 20 15:24:16 2024 read: IOPS=8, BW=33.8MiB/s (35.4MB/s)(12.0MiB/355msec) clat (msec): min=61, max=195, avg=112.51, stdev=72.73 lat (msec): min=61, max=195, avg=112.52, stdev=72.73 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 63], | 30.00th=[ 63], 40.00th=[ 81], 50.00th=[ 81], 60.00th=[ 81], | 70.00th=[ 197], 80.00th=[ 197], 90.00th=[ 197], 95.00th=[ 197], | 99.00th=[ 197], 99.50th=[ 197], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] lat (msec) : 100=66.67%, 250=33.33% cpu : usr=0.00%, sys=17.46%, ctx=53, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25133: Fri Sep 20 15:24:16 2024 read: IOPS=0, BW=1676KiB/s (1716kB/s)(4096KiB/2444msec) clat (nsec): min=333115k, max=333115k, avg=333114809.00, stdev= 0.00 lat (nsec): min=333116k, max=333116k, avg=333116064.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 334], 10.00th=[ 334], 20.00th=[ 334], | 30.00th=[ 334], 40.00th=[ 334], 50.00th=[ 334], 60.00th=[ 334], | 70.00th=[ 334], 80.00th=[ 334], 90.00th=[ 334], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 8126, max= 8126, per=100.00%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3352KiB/s (3432kB/s)(8192KiB/2444msec) clat (msec): min=198, max=1902, avg=1050.22, stdev=1204.85 lat (msec): min=198, max=1902, avg=1050.41, stdev=1204.98 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 199], 10.00th=[ 199], 20.00th=[ 199], | 30.00th=[ 199], 40.00th=[ 199], 50.00th=[ 199], 60.00th=[ 1905], | 70.00th=[ 1905], 80.00th=[ 1905], 90.00th=[ 1905], 95.00th=[ 1905], | 99.00th=[ 1905], 99.50th=[ 1905], 99.90th=[ 1905], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 8192, max= 8192, per=79.56%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.04%, sys=3.68%, ctx=140, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25135: Fri Sep 20 15:24:16 2024 write: IOPS=0, BW=3862KiB/s (3954kB/s)(12.0MiB/3182msec) clat (msec): min=204, max=1908, avg=1044.74, stdev=852.48 lat (msec): min=204, max=1909, avg=1045.03, stdev=852.40 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 205], 10.00th=[ 205], 20.00th=[ 205], | 30.00th=[ 205], 40.00th=[ 1020], 50.00th=[ 1020], 60.00th=[ 1020], | 70.00th=[ 1905], 80.00th=[ 1905], 90.00th=[ 1905], 95.00th=[ 1905], | 99.00th=[ 1905], 99.50th=[ 1905], 99.90th=[ 1905], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 8192, max= 8192, per=79.56%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.06%, sys=4.31%, ctx=35, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25137: Fri Sep 20 15:24:16 2024 write: IOPS=1, BW=5423KiB/s (5553kB/s)(12.0MiB/2266msec) clat (msec): min=110, max=1927, avg=752.09, stdev=1019.53 lat (msec): min=110, max=1927, avg=752.35, stdev=1019.54 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 220], 50.00th=[ 220], 60.00th=[ 220], | 70.00th=[ 1921], 80.00th=[ 1921], 90.00th=[ 1921], 95.00th=[ 1921], | 99.00th=[ 1921], 99.50th=[ 1921], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 8192, max= 8192, per=79.56%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=4.50%, ctx=88, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6704KiB/s (6865kB/s), 1676KiB/s-33.8MiB/s (1716kB/s-35.4MB/s), io=16.0MiB (16.8MB), run=355-2444msec WRITE: bw=10.1MiB/s (10.5MB/s), 3352KiB/s-5423KiB/s (3432kB/s-5553kB/s), io=32.0MiB (33.6MB), run=2266-3182msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25132: Fri Sep 20 15:24:17 2024 read: IOPS=0, BW=3991KiB/s (4087kB/s)(12.0MiB/3079msec) clat (msec): min=158, max=2080, avg=1024.05, stdev=975.36 lat (msec): min=158, max=2080, avg=1024.05, stdev=975.36 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 835], 50.00th=[ 835], 60.00th=[ 835], | 70.00th=[ 2089], 80.00th=[ 2089], 90.00th=[ 2089], 95.00th=[ 2089], | 99.00th=[ 2089], 99.50th=[ 2089], 99.90th=[ 2089], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 8126, max= 8192, per=100.00%, avg=8159.00, stdev=46.67, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.06%, sys=47.71%, ctx=952, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25134: Fri Sep 20 15:24:17 2024 read: IOPS=0, BW=1737KiB/s (1779kB/s)(4096KiB/2358msec) clat (nsec): min=142333k, max=142333k, avg=142332898.00, stdev= 0.00 lat (nsec): min=142334k, max=142334k, avg=142334073.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 142], 10.00th=[ 142], 20.00th=[ 142], | 30.00th=[ 142], 40.00th=[ 142], 50.00th=[ 142], 60.00th=[ 142], | 70.00th=[ 142], 80.00th=[ 142], 90.00th=[ 142], 95.00th=[ 142], | 99.00th=[ 142], 99.50th=[ 142], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 8126, max= 8126, per=100.00%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3474KiB/s (3558kB/s)(8192KiB/2358msec) clat (msec): min=318, max=1881, avg=1099.94, stdev=1105.67 lat (msec): min=318, max=1882, avg=1100.13, stdev=1105.77 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 317], 10.00th=[ 317], 20.00th=[ 317], | 30.00th=[ 317], 40.00th=[ 317], 50.00th=[ 317], 60.00th=[ 1888], | 70.00th=[ 1888], 80.00th=[ 1888], 90.00th=[ 1888], 95.00th=[ 1888], | 99.00th=[ 1888], 99.50th=[ 1888], 99.90th=[ 1888], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 8126, max= 8126, per=81.15%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.04%, sys=8.95%, ctx=108, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25136: Fri Sep 20 15:24:17 2024 write: IOPS=0, BW=3756KiB/s (3846kB/s)(12.0MiB/3272msec) clat (msec): min=301, max=1985, avg=1085.66, stdev=847.53 lat (msec): min=302, max=1985, avg=1085.89, stdev=847.54 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 969], 50.00th=[ 969], 60.00th=[ 969], | 70.00th=[ 1989], 80.00th=[ 1989], 90.00th=[ 1989], 95.00th=[ 1989], | 99.00th=[ 1989], 99.50th=[ 1989], 99.90th=[ 1989], 99.95th=[ 1989], | 99.99th=[ 1989] bw ( KiB/s): min= 7968, max= 8192, per=80.69%, avg=8080.00, stdev=158.39, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.03%, sys=9.29%, ctx=42, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25138: Fri Sep 20 15:24:17 2024 write: IOPS=2, BW=9660KiB/s (9892kB/s)(12.0MiB/1272msec) clat (msec): min=81, max=1035, avg=418.36, stdev=534.82 lat (msec): min=82, max=1035, avg=418.63, stdev=534.81 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 83], 10.00th=[ 83], 20.00th=[ 83], | 30.00th=[ 83], 40.00th=[ 138], 50.00th=[ 138], 60.00th=[ 138], | 70.00th=[ 1036], 80.00th=[ 1036], 90.00th=[ 1036], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 8175, max= 8175, per=81.64%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=17.94%, ctx=66, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5321KiB/s (5449kB/s), 1737KiB/s-3991KiB/s (1779kB/s-4087kB/s), io=16.0MiB (16.8MB), run=2358-3079msec WRITE: bw=9.78MiB/s (10.3MB/s), 3474KiB/s-9660KiB/s (3558kB/s-9892kB/s), io=32.0MiB (33.6MB), run=1272-3272msec