== sanity test 398b: DIO and buffer IO race ============== 13:35:31 (1736793331) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.39233 s, 36.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2785: Mon Jan 13 13:35:46 2025 read: IOPS=126, BW=508KiB/s (520kB/s)(5992KiB/11796msec) clat (usec): min=50, max=165091, avg=3627.86, stdev=5068.02 lat (usec): min=50, max=165091, avg=3628.60, stdev=5068.00 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 72], 10.00th=[ 86], 20.00th=[ 1958], | 30.00th=[ 2278], 40.00th=[ 2606], 50.00th=[ 2966], 60.00th=[ 3425], | 70.00th=[ 4146], 80.00th=[ 5211], 90.00th=[ 6718], 95.00th=[ 8094], | 99.00th=[ 14746], 99.50th=[ 17433], 99.90th=[ 37487], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 320, max= 688, per=24.76%, avg=502.96, stdev=79.11, samples=23 iops : min= 80, max= 172, avg=125.65, stdev=19.73, samples=23 write: IOPS=133, BW=534KiB/s (547kB/s)(6296KiB/11796msec) clat (usec): min=466, max=53538, avg=3980.04, stdev=3828.04 lat (usec): min=466, max=53538, avg=3981.37, stdev=3827.79 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 562], 20.00th=[ 668], | 30.00th=[ 955], 40.00th=[ 2278], 50.00th=[ 3326], 60.00th=[ 4228], | 70.00th=[ 5211], 80.00th=[ 6521], 90.00th=[ 8455], 95.00th=[10945], | 99.00th=[16057], 99.50th=[17695], 99.90th=[40633], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 376, max= 750, per=24.93%, avg=530.17, stdev=95.65, samples=23 iops : min= 94, max= 187, avg=132.43, stdev=23.87, samples=23 lat (usec) : 100=5.70%, 250=1.76%, 500=0.85%, 750=11.13%, 1000=4.07% lat (msec) : 2=6.77%, 4=32.75%, 10=32.68%, 20=3.97%, 50=0.26% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.22%, sys=23.24%, ctx=5866, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2786: Mon Jan 13 13:35:46 2025 read: IOPS=125, BW=503KiB/s (515kB/s)(5940KiB/11819msec) clat (usec): min=50, max=39229, avg=3564.42, stdev=2947.90 lat (usec): min=50, max=39230, avg=3565.02, stdev=2947.91 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 71], 10.00th=[ 85], 20.00th=[ 1991], | 30.00th=[ 2311], 40.00th=[ 2638], 50.00th=[ 2966], 60.00th=[ 3425], | 70.00th=[ 4293], 80.00th=[ 5342], 90.00th=[ 6718], 95.00th=[ 8160], | 99.00th=[12256], 99.50th=[17695], 99.90th=[38536], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 256, max= 688, per=24.46%, avg=496.78, stdev=100.09, samples=23 iops : min= 64, max= 172, avg=124.09, stdev=25.01, samples=23 write: IOPS=134, BW=537KiB/s (550kB/s)(6348KiB/11819msec) clat (usec): min=403, max=167574, avg=4048.73, stdev=5586.02 lat (usec): min=404, max=167575, avg=4049.72, stdev=5586.04 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 570], 20.00th=[ 676], | 30.00th=[ 955], 40.00th=[ 2147], 50.00th=[ 3228], 60.00th=[ 4047], | 70.00th=[ 5014], 80.00th=[ 6456], 90.00th=[ 8455], 95.00th=[ 10945], | 99.00th=[ 16188], 99.50th=[ 23200], 99.90th=[ 36439], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 336, max= 648, per=24.62%, avg=523.61, stdev=76.03, samples=23 iops : min= 84, max= 162, avg=130.78, stdev=19.04, samples=23 lat (usec) : 100=5.99%, 250=1.40%, 500=0.68%, 750=11.72%, 1000=3.58% lat (msec) : 2=6.77%, 4=32.91%, 10=32.71%, 20=3.74%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=23.75%, ctx=5763, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2787: Mon Jan 13 13:35:46 2025 read: IOPS=127, BW=511KiB/s (524kB/s)(6028KiB/11787msec) clat (usec): min=52, max=53160, avg=3584.96, stdev=3018.40 lat (usec): min=52, max=53161, avg=3585.98, stdev=3018.24 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 72], 10.00th=[ 88], 20.00th=[ 1975], | 30.00th=[ 2278], 40.00th=[ 2606], 50.00th=[ 2966], 60.00th=[ 3523], | 70.00th=[ 4293], 80.00th=[ 5473], 90.00th=[ 6980], 95.00th=[ 8029], | 99.00th=[12256], 99.50th=[16319], 99.90th=[33424], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 216, max= 736, per=24.94%, avg=506.57, stdev=114.49, samples=23 iops : min= 54, max= 184, avg=126.52, stdev=28.67, samples=23 write: IOPS=132, BW=531KiB/s (544kB/s)(6260KiB/11787msec) clat (usec): min=455, max=200990, avg=4016.28, stdev=6187.11 lat (usec): min=460, max=200992, avg=4017.57, stdev=6186.94 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 529], 10.00th=[ 570], 20.00th=[ 685], | 30.00th=[ 947], 40.00th=[ 2278], 50.00th=[ 3195], 60.00th=[ 4047], | 70.00th=[ 4948], 80.00th=[ 6259], 90.00th=[ 8586], 95.00th=[ 10421], | 99.00th=[ 15795], 99.50th=[ 20317], 99.90th=[ 39584], 99.95th=[200279], | 99.99th=[200279] bw ( KiB/s): min= 312, max= 760, per=24.63%, avg=523.87, stdev=96.96, samples=23 iops : min= 78, max= 190, avg=130.87, stdev=24.25, samples=23 lat (usec) : 100=6.18%, 250=1.27%, 500=0.88%, 750=11.39%, 1000=3.58% lat (msec) : 2=6.58%, 4=33.72%, 10=32.39%, 20=3.61%, 50=0.33% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.23%, sys=23.28%, ctx=5904, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2788: Mon Jan 13 13:35:46 2025 read: IOPS=128, BW=516KiB/s (528kB/s)(6052KiB/11734msec) clat (usec): min=52, max=40775, avg=3495.25, stdev=2934.33 lat (usec): min=53, max=40776, avg=3495.88, stdev=2934.34 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 75], 10.00th=[ 86], 20.00th=[ 1958], | 30.00th=[ 2278], 40.00th=[ 2573], 50.00th=[ 2900], 60.00th=[ 3326], | 70.00th=[ 4228], 80.00th=[ 5276], 90.00th=[ 6587], 95.00th=[ 7898], | 99.00th=[11994], 99.50th=[17695], 99.90th=[33424], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 208, max= 648, per=25.28%, avg=513.39, stdev=103.68, samples=23 iops : min= 52, max= 162, avg=128.35, stdev=25.92, samples=23 write: IOPS=132, BW=531KiB/s (544kB/s)(6236KiB/11734msec) clat (usec): min=444, max=184904, avg=4070.45, stdev=6034.85 lat (usec): min=444, max=184906, avg=4071.52, stdev=6034.76 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 652], | 30.00th=[ 922], 40.00th=[ 2024], 50.00th=[ 3195], 60.00th=[ 4080], | 70.00th=[ 4948], 80.00th=[ 6325], 90.00th=[ 8586], 95.00th=[ 11338], | 99.00th=[ 18744], 99.50th=[ 23200], 99.90th=[ 34341], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 232, max= 736, per=24.84%, avg=528.35, stdev=114.55, samples=23 iops : min= 58, max= 184, avg=132.09, stdev=28.64, samples=23 lat (usec) : 100=5.96%, 250=1.60%, 500=1.24%, 750=11.10%, 1000=3.68% lat (msec) : 2=7.03%, 4=32.91%, 10=31.97%, 20=4.00%, 50=0.49% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=23.36%, ctx=5705, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2032KiB/s (2080kB/s), 503KiB/s-516KiB/s (515kB/s-528kB/s), io=23.4MiB (24.6MB), run=11734-11819msec WRITE: bw=2127KiB/s (2178kB/s), 531KiB/s-537KiB/s (544kB/s-550kB/s), io=24.6MiB (25.7MB), run=11734-11819msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2781: Mon Jan 13 13:36:07 2025 read: IOPS=46, BW=186KiB/s (190kB/s)(5992KiB/32243msec) clat (usec): min=1656, max=171457, avg=3474.08, stdev=5237.24 lat (usec): min=1656, max=171459, avg=3474.85, stdev=5237.26 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1876], 10.00th=[ 1926], 20.00th=[ 2024], | 30.00th=[ 2114], 40.00th=[ 2212], 50.00th=[ 2343], 60.00th=[ 2606], | 70.00th=[ 2900], 80.00th=[ 3556], 90.00th=[ 6259], 95.00th=[ 8455], | 99.00th=[ 15795], 99.50th=[ 19792], 99.90th=[ 44303], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 56, max= 312, per=25.01%, avg=185.55, stdev=67.79, samples=64 iops : min= 14, max= 78, avg=46.33, stdev=16.98, samples=64 write: IOPS=48, BW=195KiB/s (200kB/s)(6296KiB/32243msec) clat (usec): min=9171, max=61100, avg=17110.10, stdev=5934.44 lat (usec): min=9171, max=61101, avg=17111.08, stdev=5934.48 clat percentiles (usec): | 1.00th=[ 9765], 5.00th=[10552], 10.00th=[10945], 20.00th=[11863], | 30.00th=[13435], 40.00th=[15008], 50.00th=[16319], 60.00th=[17171], | 70.00th=[18482], 80.00th=[20317], 90.00th=[24773], 95.00th=[28443], | 99.00th=[38011], 99.50th=[42730], 99.90th=[56886], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 64, max= 264, per=25.09%, avg=194.92, stdev=52.49, samples=64 iops : min= 16, max= 66, avg=48.67, stdev=13.15, samples=64 lat (msec) : 2=8.69%, 4=31.58%, 10=7.68%, 20=40.89%, 50=11.07% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.12%, sys=6.30%, ctx=3506, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2782: Mon Jan 13 13:36:07 2025 read: IOPS=45, BW=184KiB/s (188kB/s)(5940KiB/32343msec) clat (usec): min=1605, max=176035, avg=3568.39, stdev=5519.47 lat (usec): min=1605, max=176038, avg=3569.15, stdev=5519.51 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1860], 10.00th=[ 1942], 20.00th=[ 2040], | 30.00th=[ 2147], 40.00th=[ 2245], 50.00th=[ 2376], 60.00th=[ 2606], | 70.00th=[ 2933], 80.00th=[ 3556], 90.00th=[ 6456], 95.00th=[ 9110], | 99.00th=[ 15926], 99.50th=[ 20841], 99.90th=[ 54789], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 16, max= 336, per=24.77%, avg=183.80, stdev=73.97, samples=64 iops : min= 4, max= 84, avg=45.89, stdev=18.51, samples=64 write: IOPS=49, BW=196KiB/s (201kB/s)(6348KiB/32343msec) clat (usec): min=9281, max=56592, avg=16973.33, stdev=5976.07 lat (usec): min=9282, max=56593, avg=16974.24, stdev=5976.06 clat percentiles (usec): | 1.00th=[ 9896], 5.00th=[10421], 10.00th=[10945], 20.00th=[11994], | 30.00th=[13173], 40.00th=[14746], 50.00th=[16057], 60.00th=[17171], | 70.00th=[18220], 80.00th=[20317], 90.00th=[24773], 95.00th=[28967], | 99.00th=[37487], 99.50th=[41157], 99.90th=[51119], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 72, max= 264, per=25.15%, avg=195.42, stdev=54.64, samples=64 iops : min= 18, max= 66, avg=48.80, stdev=13.70, samples=64 lat (msec) : 2=8.04%, 4=31.54%, 10=7.55%, 20=41.73%, 50=10.94% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.12%, sys=6.30%, ctx=3513, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2783: Mon Jan 13 13:36:07 2025 read: IOPS=46, BW=188KiB/s (192kB/s)(6028KiB/32113msec) clat (usec): min=1649, max=28473, avg=3474.80, stdev=2661.98 lat (usec): min=1650, max=28474, avg=3475.57, stdev=2661.97 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1876], 10.00th=[ 1942], 20.00th=[ 2040], | 30.00th=[ 2147], 40.00th=[ 2245], 50.00th=[ 2409], 60.00th=[ 2704], | 70.00th=[ 2999], 80.00th=[ 3982], 90.00th=[ 6980], 95.00th=[ 8848], | 99.00th=[13435], 99.50th=[16319], 99.90th=[23987], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 56, max= 344, per=25.29%, avg=187.67, stdev=62.68, samples=64 iops : min= 14, max= 86, avg=46.86, stdev=15.69, samples=64 write: IOPS=48, BW=195KiB/s (200kB/s)(6260KiB/32113msec) clat (msec): min=9, max=199, avg=17.10, stdev= 7.54 lat (msec): min=9, max=199, avg=17.10, stdev= 7.54 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 17], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 26], 95.00th=[ 30], | 99.00th=[ 37], 99.50th=[ 40], 99.90th=[ 62], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 64, max= 264, per=25.07%, avg=194.80, stdev=55.57, samples=64 iops : min= 16, max= 66, avg=48.64, stdev=13.93, samples=64 lat (msec) : 2=7.94%, 4=31.32%, 10=9.15%, 20=40.69%, 50=10.84% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.12%, sys=6.42%, ctx=3450, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2784: Mon Jan 13 13:36:07 2025 read: IOPS=46, BW=188KiB/s (192kB/s)(6052KiB/32235msec) clat (usec): min=1602, max=54896, avg=3428.55, stdev=2953.08 lat (usec): min=1602, max=54896, avg=3429.29, stdev=2953.08 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1860], 10.00th=[ 1926], 20.00th=[ 2024], | 30.00th=[ 2114], 40.00th=[ 2212], 50.00th=[ 2376], 60.00th=[ 2638], | 70.00th=[ 2966], 80.00th=[ 3654], 90.00th=[ 6587], 95.00th=[ 9110], | 99.00th=[15533], 99.50th=[17433], 99.90th=[23462], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 56, max= 336, per=25.27%, avg=187.53, stdev=70.63, samples=64 iops : min= 14, max= 84, avg=46.81, stdev=17.69, samples=64 write: IOPS=48, BW=193KiB/s (198kB/s)(6236KiB/32235msec) clat (msec): min=9, max=186, avg=17.28, stdev= 7.36 lat (msec): min=9, max=186, avg=17.28, stdev= 7.36 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 25], 95.00th=[ 29], | 99.00th=[ 41], 99.50th=[ 50], 99.90th=[ 60], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 72, max= 264, per=24.88%, avg=193.28, stdev=52.49, samples=64 iops : min= 18, max= 66, avg=48.25, stdev=13.16, samples=64 lat (msec) : 2=8.20%, 4=31.93%, 10=8.27%, 20=39.97%, 50=11.36% lat (msec) : 100=0.23%, 250=0.03% cpu : usr=0.13%, sys=6.29%, ctx=3489, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=742KiB/s (760kB/s), 184KiB/s-188KiB/s (188kB/s-192kB/s), io=23.4MiB (24.6MB), run=32113-32343msec WRITE: bw=777KiB/s (796kB/s), 193KiB/s-196KiB/s (198kB/s-201kB/s), io=24.6MiB (25.7MB), run=32113-32343msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2828: Mon Jan 13 13:36:10 2025 read: IOPS=146, BW=2338KiB/s (2394kB/s)(5920KiB/2532msec) clat (usec): min=59, max=24008, avg=3141.64, stdev=2797.05 lat (usec): min=59, max=24008, avg=3142.37, stdev=2796.97 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 73], 10.00th=[ 77], 20.00th=[ 90], | 30.00th=[ 1598], 40.00th=[ 2540], 50.00th=[ 2900], 60.00th=[ 3425], | 70.00th=[ 4293], 80.00th=[ 5276], 90.00th=[ 6456], 95.00th=[ 7439], | 99.00th=[11863], 99.50th=[15270], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 1536, max= 3200, per=24.98%, avg=2322.20, stdev=632.17, samples=5 iops : min= 96, max= 200, avg=145.00, stdev=39.51, samples=5 write: IOPS=157, BW=2515KiB/s (2575kB/s)(6368KiB/2532msec) clat (usec): min=627, max=19833, avg=3378.11, stdev=3236.09 lat (usec): min=628, max=19834, avg=3379.06, stdev=3236.12 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 766], 10.00th=[ 816], 20.00th=[ 922], | 30.00th=[ 1045], 40.00th=[ 1287], 50.00th=[ 1663], 60.00th=[ 2933], | 70.00th=[ 4293], 80.00th=[ 6128], 90.00th=[ 8029], 95.00th=[ 9634], | 99.00th=[14091], 99.50th=[17433], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 1760, max= 3520, per=25.70%, avg=2514.20, stdev=689.56, samples=5 iops : min= 110, max= 220, avg=157.00, stdev=43.10, samples=5 lat (usec) : 100=10.42%, 250=3.26%, 500=0.26%, 750=1.30%, 1000=13.02% lat (msec) : 2=14.19%, 4=25.13%, 10=29.82%, 20=2.47%, 50=0.13% cpu : usr=0.32%, sys=29.36%, ctx=1613, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2829: Mon Jan 13 13:36:10 2025 read: IOPS=139, BW=2231KiB/s (2284kB/s)(5744KiB/2575msec) clat (usec): min=56, max=19529, avg=2996.83, stdev=2955.71 lat (usec): min=56, max=19529, avg=2998.64, stdev=2956.91 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 71], 10.00th=[ 75], 20.00th=[ 86], | 30.00th=[ 155], 40.00th=[ 2409], 50.00th=[ 2704], 60.00th=[ 3261], | 70.00th=[ 4080], 80.00th=[ 5014], 90.00th=[ 6652], 95.00th=[ 7570], | 99.00th=[13698], 99.50th=[18744], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 1408, max= 2752, per=23.73%, avg=2206.80, stdev=597.82, samples=5 iops : min= 88, max= 172, avg=137.80, stdev=37.23, samples=5 write: IOPS=158, BW=2541KiB/s (2602kB/s)(6544KiB/2575msec) clat (usec): min=646, max=40639, avg=3428.48, stdev=4203.78 lat (usec): min=647, max=40641, avg=3431.20, stdev=4204.52 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 750], 10.00th=[ 816], 20.00th=[ 906], | 30.00th=[ 1029], 40.00th=[ 1221], 50.00th=[ 1598], 60.00th=[ 2999], | 70.00th=[ 4113], 80.00th=[ 5800], 90.00th=[ 7308], 95.00th=[ 9110], | 99.00th=[19268], 99.50th=[27657], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 1504, max= 3168, per=25.57%, avg=2501.20, stdev=608.22, samples=5 iops : min= 94, max= 198, avg=156.20, stdev=37.98, samples=5 lat (usec) : 100=11.72%, 250=3.78%, 500=0.26%, 750=3.12%, 1000=12.11% lat (msec) : 2=14.32%, 4=24.22%, 10=27.34%, 20=2.60%, 50=0.52% cpu : usr=0.27%, sys=27.47%, ctx=1597, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2830: Mon Jan 13 13:36:10 2025 read: IOPS=151, BW=2429KiB/s (2487kB/s)(6256KiB/2576msec) clat (usec): min=57, max=30408, avg=3018.12, stdev=3198.15 lat (usec): min=57, max=30409, avg=3020.06, stdev=3198.14 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 67], 10.00th=[ 72], 20.00th=[ 83], | 30.00th=[ 99], 40.00th=[ 2180], 50.00th=[ 2769], 60.00th=[ 3359], | 70.00th=[ 4490], 80.00th=[ 5342], 90.00th=[ 6652], 95.00th=[ 7832], | 99.00th=[11600], 99.50th=[22938], 99.90th=[30278], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 1216, max= 3040, per=26.27%, avg=2442.40, stdev=708.27, samples=5 iops : min= 76, max= 190, avg=152.40, stdev=44.16, samples=5 write: IOPS=146, BW=2342KiB/s (2398kB/s)(6032KiB/2576msec) clat (usec): min=655, max=39365, avg=3476.49, stdev=3849.35 lat (usec): min=656, max=39367, avg=3477.39, stdev=3849.47 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 742], 10.00th=[ 799], 20.00th=[ 930], | 30.00th=[ 1057], 40.00th=[ 1254], 50.00th=[ 1893], 60.00th=[ 3228], | 70.00th=[ 4228], 80.00th=[ 5997], 90.00th=[ 7504], 95.00th=[ 9634], | 99.00th=[16057], 99.50th=[33817], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 1440, max= 2848, per=23.79%, avg=2327.20, stdev=609.70, samples=5 iops : min= 90, max= 178, avg=145.20, stdev=37.92, samples=5 lat (usec) : 100=15.49%, 250=3.12%, 500=0.26%, 750=3.65%, 1000=10.03% lat (msec) : 2=12.63%, 4=20.70%, 10=31.64%, 20=1.95%, 50=0.52% cpu : usr=0.27%, sys=27.50%, ctx=1537, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2831: Mon Jan 13 13:36:10 2025 read: IOPS=148, BW=2374KiB/s (2431kB/s)(6032KiB/2541msec) clat (usec): min=58, max=35428, avg=3126.91, stdev=3341.30 lat (usec): min=58, max=35429, avg=3127.51, stdev=3341.31 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 69], 10.00th=[ 74], 20.00th=[ 87], | 30.00th=[ 157], 40.00th=[ 2540], 50.00th=[ 2933], 60.00th=[ 3425], | 70.00th=[ 4293], 80.00th=[ 5276], 90.00th=[ 6259], 95.00th=[ 7504], | 99.00th=[10683], 99.50th=[30802], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 1440, max= 3040, per=24.97%, avg=2322.00, stdev=584.84, samples=5 iops : min= 90, max= 190, avg=145.00, stdev=36.48, samples=5 write: IOPS=153, BW=2462KiB/s (2521kB/s)(6256KiB/2541msec) clat (usec): min=636, max=29837, avg=3210.44, stdev=3561.77 lat (usec): min=637, max=29839, avg=3211.36, stdev=3561.82 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 742], 10.00th=[ 799], 20.00th=[ 889], | 30.00th=[ 988], 40.00th=[ 1156], 50.00th=[ 1500], 60.00th=[ 2704], | 70.00th=[ 3785], 80.00th=[ 5080], 90.00th=[ 7635], 95.00th=[ 9503], | 99.00th=[15926], 99.50th=[26608], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 1664, max= 2912, per=24.92%, avg=2437.20, stdev=534.03, samples=5 iops : min= 104, max= 182, avg=152.20, stdev=33.26, samples=5 lat (usec) : 100=11.98%, 250=3.78%, 500=0.26%, 750=2.99%, 1000=12.63% lat (msec) : 2=13.28%, 4=25.00%, 10=27.86%, 20=1.56%, 50=0.65% cpu : usr=0.04%, sys=28.27%, ctx=1578, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9298KiB/s (9521kB/s), 2231KiB/s-2429KiB/s (2284kB/s-2487kB/s), io=23.4MiB (24.5MB), run=2532-2576msec WRITE: bw=9783KiB/s (10.0MB/s), 2342KiB/s-2541KiB/s (2398kB/s-2602kB/s), io=24.6MiB (25.8MB), run=2532-2576msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2824: Mon Jan 13 13:36:16 2025 read: IOPS=43, BW=697KiB/s (714kB/s)(5920KiB/8488msec) clat (usec): min=2084, max=20997, avg=4042.08, stdev=2634.04 lat (usec): min=2084, max=20998, avg=4042.98, stdev=2634.02 clat percentiles (usec): | 1.00th=[ 2147], 5.00th=[ 2409], 10.00th=[ 2540], 20.00th=[ 2671], | 30.00th=[ 2802], 40.00th=[ 2966], 50.00th=[ 3130], 60.00th=[ 3294], | 70.00th=[ 3589], 80.00th=[ 4555], 90.00th=[ 6849], 95.00th=[ 9503], | 99.00th=[16581], 99.50th=[18744], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 448, max= 1216, per=24.80%, avg=679.88, stdev=235.97, samples=16 iops : min= 28, max= 76, avg=42.44, stdev=14.79, samples=16 write: IOPS=46, BW=750KiB/s (768kB/s)(6368KiB/8488msec) clat (usec): min=10341, max=46242, avg=17399.21, stdev=5304.36 lat (usec): min=10343, max=46243, avg=17401.03, stdev=5304.31 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11469], 10.00th=[11863], 20.00th=[12780], | 30.00th=[13960], 40.00th=[15270], 50.00th=[16712], 60.00th=[17957], | 70.00th=[19006], 80.00th=[20579], 90.00th=[23462], 95.00th=[27132], | 99.00th=[38011], 99.50th=[44827], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 480, max= 992, per=25.52%, avg=735.88, stdev=164.58, samples=16 iops : min= 30, max= 62, avg=45.94, stdev=10.36, samples=16 lat (msec) : 4=37.11%, 10=8.85%, 20=42.06%, 50=11.98% cpu : usr=0.19%, sys=8.44%, ctx=926, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2825: Mon Jan 13 13:36:16 2025 read: IOPS=41, BW=657KiB/s (673kB/s)(5744KiB/8737msec) clat (usec): min=1907, max=25781, avg=4098.79, stdev=2850.66 lat (usec): min=1908, max=25782, avg=4099.74, stdev=2850.63 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2311], 10.00th=[ 2474], 20.00th=[ 2704], | 30.00th=[ 2835], 40.00th=[ 2966], 50.00th=[ 3130], 60.00th=[ 3294], | 70.00th=[ 3621], 80.00th=[ 4424], 90.00th=[ 6915], 95.00th=[ 9896], | 99.00th=[16057], 99.50th=[18744], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 320, max= 1120, per=24.03%, avg=658.71, stdev=216.45, samples=17 iops : min= 20, max= 70, avg=41.12, stdev=13.53, samples=17 write: IOPS=46, BW=749KiB/s (767kB/s)(6544KiB/8737msec) clat (usec): min=9582, max=62636, avg=17721.93, stdev=5800.16 lat (usec): min=9582, max=62638, avg=17723.81, stdev=5800.08 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11469], 10.00th=[11994], 20.00th=[13304], | 30.00th=[14615], 40.00th=[15664], 50.00th=[16712], 60.00th=[17957], | 70.00th=[19006], 80.00th=[20579], 90.00th=[24249], 95.00th=[27919], | 99.00th=[38536], 99.50th=[42730], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 448, max= 992, per=25.84%, avg=745.29, stdev=159.26, samples=17 iops : min= 28, max= 62, avg=46.53, stdev=10.03, samples=17 lat (msec) : 2=0.26%, 4=35.55%, 10=8.98%, 20=42.97%, 50=11.98% lat (msec) : 100=0.26% cpu : usr=0.05%, sys=8.40%, ctx=901, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2826: Mon Jan 13 13:36:16 2025 read: IOPS=47, BW=756KiB/s (775kB/s)(6256KiB/8270msec) clat (usec): min=1943, max=27385, avg=3878.60, stdev=2585.68 lat (usec): min=1944, max=27386, avg=3879.57, stdev=2585.67 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2343], 10.00th=[ 2474], 20.00th=[ 2638], | 30.00th=[ 2802], 40.00th=[ 2933], 50.00th=[ 3097], 60.00th=[ 3228], | 70.00th=[ 3425], 80.00th=[ 3884], 90.00th=[ 6456], 95.00th=[ 8586], | 99.00th=[15139], 99.50th=[16188], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 384, max= 1277, per=27.70%, avg=759.37, stdev=285.66, samples=16 iops : min= 24, max= 79, avg=47.25, stdev=17.68, samples=16 write: IOPS=45, BW=729KiB/s (747kB/s)(6032KiB/8270msec) clat (usec): min=9369, max=42276, avg=17710.57, stdev=5116.12 lat (usec): min=9370, max=42278, avg=17712.44, stdev=5116.39 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11469], 10.00th=[12125], 20.00th=[13173], | 30.00th=[14353], 40.00th=[15795], 50.00th=[16909], 60.00th=[18220], | 70.00th=[19530], 80.00th=[20841], 90.00th=[23987], 95.00th=[27657], | 99.00th=[34866], 99.50th=[40109], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 448, max= 928, per=24.95%, avg=719.50, stdev=143.50, samples=16 iops : min= 28, max= 58, avg=44.75, stdev= 8.94, samples=16 lat (msec) : 2=0.13%, 4=41.15%, 10=7.81%, 20=37.63%, 50=13.28% cpu : usr=0.19%, sys=8.68%, ctx=933, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2827: Mon Jan 13 13:36:16 2025 read: IOPS=44, BW=708KiB/s (725kB/s)(6032KiB/8519msec) clat (usec): min=1997, max=19421, avg=4102.84, stdev=2956.38 lat (usec): min=1998, max=19422, avg=4104.02, stdev=2956.26 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2311], 10.00th=[ 2507], 20.00th=[ 2671], | 30.00th=[ 2802], 40.00th=[ 2933], 50.00th=[ 3097], 60.00th=[ 3261], | 70.00th=[ 3490], 80.00th=[ 4146], 90.00th=[ 6456], 95.00th=[11731], | 99.00th=[17433], 99.50th=[19268], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 319, max= 1120, per=25.61%, avg=702.06, stdev=203.28, samples=17 iops : min= 19, max= 70, avg=43.82, stdev=12.82, samples=17 write: IOPS=45, BW=734KiB/s (752kB/s)(6256KiB/8519msec) clat (usec): min=9334, max=48560, avg=17612.48, stdev=5450.75 lat (usec): min=9335, max=48562, avg=17614.27, stdev=5450.68 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11600], 10.00th=[12256], 20.00th=[12911], | 30.00th=[13829], 40.00th=[15139], 50.00th=[16909], 60.00th=[17957], | 70.00th=[19268], 80.00th=[21103], 90.00th=[24511], 95.00th=[27132], | 99.00th=[38536], 99.50th=[44827], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 448, max= 928, per=25.45%, avg=734.00, stdev=165.83, samples=17 iops : min= 28, max= 58, avg=45.82, stdev=10.43, samples=17 lat (msec) : 2=0.13%, 4=38.41%, 10=7.68%, 20=40.89%, 50=12.89% cpu : usr=0.14%, sys=8.45%, ctx=910, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2741KiB/s (2807kB/s), 657KiB/s-756KiB/s (673kB/s-775kB/s), io=23.4MiB (24.5MB), run=8270-8737msec WRITE: bw=2884KiB/s (2954kB/s), 729KiB/s-750KiB/s (747kB/s-768kB/s), io=24.6MiB (25.8MB), run=8270-8737msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2843: Mon Jan 13 13:36:18 2025 read: IOPS=5, BW=5804KiB/s (5943kB/s)(7168KiB/1235msec) clat (usec): min=455, max=740401, avg=129049.16, stdev=269852.38 lat (usec): min=456, max=740402, avg=129050.12, stdev=269852.26 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 457], 10.00th=[ 457], 20.00th=[ 30016], | 30.00th=[ 30802], 40.00th=[ 30802], 50.00th=[ 32637], 60.00th=[ 32900], | 70.00th=[ 32900], 80.00th=[ 36439], 90.00th=[742392], 95.00th=[742392], | 99.00th=[742392], 99.50th=[742392], 99.90th=[742392], 99.95th=[742392], | 99.99th=[742392] bw ( KiB/s): min= 4096, max= 6144, per=33.45%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=4, BW=4146KiB/s (4245kB/s)(5120KiB/1235msec) clat (usec): min=44451, max=94562, avg=64860.38, stdev=20712.33 lat (usec): min=44503, max=94625, avg=64922.42, stdev=20722.72 clat percentiles (usec): | 1.00th=[44303], 5.00th=[44303], 10.00th=[44303], 20.00th=[44303], | 30.00th=[49021], 40.00th=[49021], 50.00th=[59507], 60.00th=[59507], | 70.00th=[77071], 80.00th=[77071], 90.00th=[94897], 95.00th=[94897], | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 4096, max= 4096, per=19.12%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=58.33%, 100=25.00%, 750=8.33% cpu : usr=0.00%, sys=23.10%, ctx=960, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2844: Mon Jan 13 13:36:18 2025 read: IOPS=1, BW=1531KiB/s (1567kB/s)(2048KiB/1338msec) clat (msec): min=27, max=700, avg=363.72, stdev=475.83 lat (msec): min=27, max=700, avg=363.72, stdev=475.83 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 28], 60.00th=[ 701], | 70.00th=[ 701], 80.00th=[ 701], 90.00th=[ 701], 95.00th=[ 701], | 99.00th=[ 701], 99.50th=[ 701], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 2043, max= 2043, per=13.35%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7653KiB/s (7837kB/s)(10.0MiB/1338msec) clat (msec): min=21, max=139, avg=56.01, stdev=40.60 lat (msec): min=21, max=139, avg=56.08, stdev=40.60 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 30], 40.00th=[ 34], 50.00th=[ 38], 60.00th=[ 42], | 70.00th=[ 70], 80.00th=[ 70], 90.00th=[ 115], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 8175, max= 8175, per=38.15%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=66.67%, 100=8.33%, 250=16.67%, 750=8.33% cpu : usr=0.00%, sys=25.88%, ctx=585, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2845: Mon Jan 13 13:36:18 2025 read: IOPS=4, BW=4880KiB/s (4997kB/s)(6144KiB/1259msec) clat (usec): min=328, max=51836, avg=27298.53, stdev=22431.47 lat (usec): min=329, max=51837, avg=27299.31, stdev=22431.46 clat percentiles (usec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 2343], | 30.00th=[ 2343], 40.00th=[29230], 50.00th=[29230], 60.00th=[29230], | 70.00th=[51119], 80.00th=[51119], 90.00th=[51643], 95.00th=[51643], | 99.00th=[51643], 99.50th=[51643], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 4096, max= 4096, per=26.76%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4880KiB/s (4997kB/s)(6144KiB/1259msec) clat (msec): min=24, max=849, avg=174.18, stdev=331.42 lat (msec): min=24, max=849, avg=174.22, stdev=331.42 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 26], | 30.00th=[ 26], 40.00th=[ 30], 50.00th=[ 30], 60.00th=[ 35], | 70.00th=[ 82], 80.00th=[ 82], 90.00th=[ 852], 95.00th=[ 852], | 99.00th=[ 852], 99.50th=[ 852], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 2048, max= 4096, per=14.34%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (usec) : 500=8.33% lat (msec) : 4=8.33%, 50=50.00%, 100=25.00%, 1000=8.33% cpu : usr=0.00%, sys=11.61%, ctx=517, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2847: Mon Jan 13 13:36:18 2025 read: IOPS=3, BW=3911KiB/s (4005kB/s)(5120KiB/1309msec) clat (usec): min=484, max=85864, avg=25684.05, stdev=35688.21 lat (usec): min=485, max=85865, avg=25684.97, stdev=35688.29 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 486], 10.00th=[ 486], 20.00th=[ 486], | 30.00th=[ 5080], 40.00th=[ 5080], 50.00th=[ 6063], 60.00th=[ 6063], | 70.00th=[31065], 80.00th=[31065], 90.00th=[85459], 95.00th=[85459], | 99.00th=[85459], 99.50th=[85459], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] write: IOPS=5, BW=5476KiB/s (5607kB/s)(7168KiB/1309msec) clat (msec): min=35, max=801, avg=163.86, stdev=281.96 lat (msec): min=35, max=801, avg=163.93, stdev=281.96 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 46], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 50], 60.00th=[ 64], | 70.00th=[ 64], 80.00th=[ 106], 90.00th=[ 802], 95.00th=[ 802], | 99.00th=[ 802], 99.50th=[ 802], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 2048, max= 6131, per=19.08%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (usec) : 500=8.33% lat (msec) : 10=16.67%, 50=41.67%, 100=16.67%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=14.83%, ctx=211, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=14.9MiB/s (15.7MB/s), 1531KiB/s-5804KiB/s (1567kB/s-5943kB/s), io=20.0MiB (20.0MB), run=1235-1338msec WRITE: bw=20.9MiB/s (21.9MB/s), 4146KiB/s-7653KiB/s (4245kB/s-7837kB/s), io=28.0MiB (29.4MB), run=1235-1338msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2848: Mon Jan 13 13:36:19 2025 read: IOPS=4, BW=5051KiB/s (5173kB/s)(7168KiB/1419msec) clat (msec): min=23, max=842, avg=153.89, stdev=304.21 lat (msec): min=23, max=842, avg=153.89, stdev=304.21 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 26], | 30.00th=[ 30], 40.00th=[ 30], 50.00th=[ 41], 60.00th=[ 54], | 70.00th=[ 54], 80.00th=[ 63], 90.00th=[ 844], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 2048, max= 4096, per=22.02%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 write: IOPS=3, BW=3608KiB/s (3695kB/s)(5120KiB/1419msec) clat (msec): min=26, max=121, avg=61.73, stdev=40.30 lat (msec): min=26, max=121, avg=61.79, stdev=40.31 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 38], 40.00th=[ 38], 50.00th=[ 39], 60.00th=[ 39], | 70.00th=[ 85], 80.00th=[ 85], 90.00th=[ 123], 95.00th=[ 123], | 99.00th=[ 123], 99.50th=[ 123], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 2048, max= 2048, per=10.49%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=25.00%, 250=8.33%, 1000=8.33% cpu : usr=0.14%, sys=5.85%, ctx=40, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2849: Mon Jan 13 13:36:19 2025 read: IOPS=1, BW=1395KiB/s (1429kB/s)(2048KiB/1468msec) clat (msec): min=65, max=659, avg=362.41, stdev=420.14 lat (msec): min=65, max=659, avg=362.41, stdev=420.14 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 66], 10.00th=[ 66], 20.00th=[ 66], | 30.00th=[ 66], 40.00th=[ 66], 50.00th=[ 66], 60.00th=[ 659], | 70.00th=[ 659], 80.00th=[ 659], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 2043, max= 2043, per=14.65%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6975KiB/s (7143kB/s)(10.0MiB/1468msec) clat (msec): min=22, max=170, avg=65.81, stdev=45.44 lat (msec): min=22, max=170, avg=65.87, stdev=45.44 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 35], 50.00th=[ 39], 60.00th=[ 69], | 70.00th=[ 87], 80.00th=[ 87], 90.00th=[ 91], 95.00th=[ 171], | 99.00th=[ 171], 99.50th=[ 171], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 4087, max= 4087, per=20.93%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=41.67%, 250=8.33%, 750=8.33% cpu : usr=0.07%, sys=7.84%, ctx=79, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2850: Mon Jan 13 13:36:19 2025 read: IOPS=4, BW=4367KiB/s (4472kB/s)(6144KiB/1407msec) clat (usec): min=16857, max=68110, avg=32803.09, stdev=19576.12 lat (usec): min=16859, max=68111, avg=32804.11, stdev=19576.03 clat percentiles (usec): | 1.00th=[16909], 5.00th=[16909], 10.00th=[16909], 20.00th=[20579], | 30.00th=[20579], 40.00th=[23200], 50.00th=[23200], 60.00th=[24511], | 70.00th=[43254], 80.00th=[43254], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] write: IOPS=4, BW=4367KiB/s (4472kB/s)(6144KiB/1407msec) clat (msec): min=23, max=921, avg=187.54, stdev=360.20 lat (msec): min=23, max=921, avg=187.59, stdev=360.20 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 50], | 70.00th=[ 72], 80.00th=[ 72], 90.00th=[ 919], 95.00th=[ 919], | 99.00th=[ 919], 99.50th=[ 919], 99.90th=[ 919], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 4087, max= 4087, per=20.93%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=66.67%, 100=16.67%, 1000=8.33% cpu : usr=0.00%, sys=7.54%, ctx=58, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2851: Mon Jan 13 13:36:19 2025 read: IOPS=3, BW=3611KiB/s (3697kB/s)(5120KiB/1418msec) clat (usec): min=21848, max=56072, avg=31300.78, stdev=14220.56 lat (usec): min=21849, max=56072, avg=31301.78, stdev=14220.51 clat percentiles (usec): | 1.00th=[21890], 5.00th=[21890], 10.00th=[21890], 20.00th=[21890], | 30.00th=[21890], 40.00th=[21890], 50.00th=[27657], 60.00th=[27657], | 70.00th=[28967], 80.00th=[28967], 90.00th=[55837], 95.00th=[55837], | 99.00th=[55837], 99.50th=[55837], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] write: IOPS=4, BW=5055KiB/s (5176kB/s)(7168KiB/1418msec) clat (msec): min=25, max=850, avg=166.37, stdev=302.65 lat (msec): min=25, max=850, avg=166.44, stdev=302.69 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 29], | 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 55], 60.00th=[ 82], | 70.00th=[ 82], 80.00th=[ 91], 90.00th=[ 852], 95.00th=[ 852], | 99.00th=[ 852], 99.50th=[ 852], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 2048, max= 2048, per=10.49%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 50=58.33%, 100=33.33%, 1000=8.33% cpu : usr=0.14%, sys=6.56%, ctx=67, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.6MiB/s (14.3MB/s), 1395KiB/s-5051KiB/s (1429kB/s-5173kB/s), io=20.0MiB (20.0MB), run=1407-1468msec WRITE: bw=19.1MiB/s (20.0MB/s), 3608KiB/s-6975KiB/s (3695kB/s-7143kB/s), io=28.0MiB (29.4MB), run=1407-1468msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2858: Mon Jan 13 13:36:21 2025 read: IOPS=2, BW=11.2MiB/s (11.8MB/s)(12.0MiB/1068msec) clat (msec): min=36, max=817, avg=352.18, stdev=411.36 lat (msec): min=36, max=817, avg=352.18, stdev=411.36 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 203], 50.00th=[ 203], 60.00th=[ 203], | 70.00th=[ 818], 80.00th=[ 818], 90.00th=[ 818], 95.00th=[ 818], | 99.00th=[ 818], 99.50th=[ 818], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 8175, max= 8175, per=69.36%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33%, 1000=33.33% cpu : usr=0.09%, sys=21.65%, ctx=1558, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2859: Mon Jan 13 13:36:21 2025 read: IOPS=0, BW=2947KiB/s (3017kB/s)(4096KiB/1390msec) clat (nsec): min=113851k, max=113851k, avg=113850687.00, stdev= 0.00 lat (nsec): min=113853k, max=113853k, avg=113853287.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 114], 10.00th=[ 114], 20.00th=[ 114], | 30.00th=[ 114], 40.00th=[ 114], 50.00th=[ 114], 60.00th=[ 114], | 70.00th=[ 114], 80.00th=[ 114], 90.00th=[ 114], 95.00th=[ 114], | 99.00th=[ 114], 99.50th=[ 114], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 8175, max= 8175, per=69.36%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5894KiB/s (6035kB/s)(8192KiB/1390msec) clat (msec): min=297, max=971, avg=634.36, stdev=476.61 lat (msec): min=297, max=971, avg=634.62, stdev=476.61 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 296], 10.00th=[ 296], 20.00th=[ 296], | 30.00th=[ 296], 40.00th=[ 296], 50.00th=[ 296], 60.00th=[ 969], | 70.00th=[ 969], 80.00th=[ 969], 90.00th=[ 969], 95.00th=[ 969], | 99.00th=[ 969], 99.50th=[ 969], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=14.18%, ctx=596, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2860: Mon Jan 13 13:36:21 2025 write: IOPS=2, BW=9791KiB/s (10.0MB/s)(12.0MiB/1255msec) clat (msec): min=83, max=882, avg=409.19, stdev=419.57 lat (msec): min=83, max=883, avg=409.55, stdev=419.73 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 84], 10.00th=[ 84], 20.00th=[ 84], | 30.00th=[ 84], 40.00th=[ 262], 50.00th=[ 262], 60.00th=[ 262], | 70.00th=[ 885], 80.00th=[ 885], 90.00th=[ 885], 95.00th=[ 885], | 99.00th=[ 885], 99.50th=[ 885], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 8192, max= 8192, per=34.75%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=18.18%, ctx=52, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2861: Mon Jan 13 13:36:21 2025 write: IOPS=2, BW=9886KiB/s (10.1MB/s)(12.0MiB/1243msec) clat (msec): min=138, max=864, avg=407.05, stdev=397.98 lat (msec): min=138, max=864, avg=407.38, stdev=398.05 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 138], 10.00th=[ 138], 20.00th=[ 138], | 30.00th=[ 138], 40.00th=[ 220], 50.00th=[ 220], 60.00th=[ 220], | 70.00th=[ 869], 80.00th=[ 869], 90.00th=[ 869], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 8192, max= 8192, per=34.75%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.00%, sys=19.48%, ctx=58, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.5MiB/s (12.1MB/s), 2947KiB/s-11.2MiB/s (3017kB/s-11.8MB/s), io=16.0MiB (16.8MB), run=1068-1390msec WRITE: bw=23.0MiB/s (24.1MB/s), 5894KiB/s-9886KiB/s (6035kB/s-10.1MB/s), io=32.0MiB (33.6MB), run=1243-1390msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2862: Mon Jan 13 13:36:21 2025 read: IOPS=2, BW=9467KiB/s (9694kB/s)(12.0MiB/1298msec) clat (msec): min=80, max=947, avg=429.71, stdev=457.62 lat (msec): min=80, max=947, avg=429.71, stdev=457.62 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 81], 10.00th=[ 81], 20.00th=[ 81], | 30.00th=[ 81], 40.00th=[ 262], 50.00th=[ 262], 60.00th=[ 262], | 70.00th=[ 944], 80.00th=[ 944], 90.00th=[ 944], 95.00th=[ 944], | 99.00th=[ 944], 99.50th=[ 944], 99.90th=[ 944], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 8192, max= 8192, per=71.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.08%, sys=4.09%, ctx=32, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2863: Mon Jan 13 13:36:21 2025 read: IOPS=0, BW=2876KiB/s (2945kB/s)(4096KiB/1424msec) clat (nsec): min=981092k, max=981092k, avg=981091725.00, stdev= 0.00 lat (nsec): min=981093k, max=981093k, avg=981093073.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 978], 5.00th=[ 978], 10.00th=[ 978], 20.00th=[ 978], | 30.00th=[ 978], 40.00th=[ 978], 50.00th=[ 978], 60.00th=[ 978], | 70.00th=[ 978], 80.00th=[ 978], 90.00th=[ 978], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] write: IOPS=1, BW=5753KiB/s (5891kB/s)(8192KiB/1424msec) clat (msec): min=132, max=287, avg=209.62, stdev=109.47 lat (msec): min=132, max=287, avg=209.84, stdev=109.58 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 133], | 30.00th=[ 133], 40.00th=[ 133], 50.00th=[ 133], 60.00th=[ 288], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.14%, sys=5.55%, ctx=27, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2864: Mon Jan 13 13:36:21 2025 write: IOPS=1, BW=8058KiB/s (8251kB/s)(12.0MiB/1525msec) clat (msec): min=38, max=1087, avg=505.14, stdev=534.43 lat (msec): min=38, max=1088, avg=505.38, stdev=534.45 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 388], 50.00th=[ 388], 60.00th=[ 388], | 70.00th=[ 1083], 80.00th=[ 1083], 90.00th=[ 1083], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] lat (msec) : 50=33.33%, 500=33.33% cpu : usr=0.07%, sys=5.91%, ctx=190, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2865: Mon Jan 13 13:36:21 2025 write: IOPS=2, BW=10.5MiB/s (10.0MB/s)(12.0MiB/1144msec) clat (msec): min=115, max=781, avg=376.20, stdev=355.97 lat (msec): min=116, max=782, avg=376.75, stdev=356.26 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 116], 10.00th=[ 116], 20.00th=[ 116], | 30.00th=[ 116], 40.00th=[ 232], 50.00th=[ 232], 60.00th=[ 232], | 70.00th=[ 785], 80.00th=[ 785], 90.00th=[ 785], 95.00th=[ 785], | 99.00th=[ 785], 99.50th=[ 785], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 8192, max= 8192, per=38.13%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.09%, sys=8.40%, ctx=45, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.2MiB/s (11.8MB/s), 2876KiB/s-9467KiB/s (2945kB/s-9694kB/s), io=16.0MiB (16.8MB), run=1298-1424msec WRITE: bw=20.0MiB/s (22.0MB/s), 5753KiB/s-10.5MiB/s (5891kB/s-10.0MB/s), io=32.0MiB (33.6MB), run=1144-1525msec