== sanity test 398b: DIO and buffer IO race ============== 19:05:22 (1746140722) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.77993 s, 28.3 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22430: Thu May 1 19:06:04 2025 read: IOPS=38, BW=154KiB/s (158kB/s)(5992KiB/38822msec) clat (usec): min=72, max=238257, avg=14417.44, stdev=10963.80 lat (usec): min=73, max=238259, avg=14418.41, stdev=10963.82 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 135], 10.00th=[ 5473], 20.00th=[ 7504], | 30.00th=[ 8848], 40.00th=[ 10290], 50.00th=[ 12518], 60.00th=[ 14877], | 70.00th=[ 17171], 80.00th=[ 20317], 90.00th=[ 26084], 95.00th=[ 31065], | 99.00th=[ 46924], 99.50th=[ 53740], 99.90th=[ 74974], 99.95th=[238027], | 99.99th=[238027] bw ( KiB/s): min= 40, max= 288, per=24.76%, avg=153.00, stdev=40.10, samples=77 iops : min= 10, max= 72, avg=37.99, stdev=10.06, samples=77 write: IOPS=40, BW=162KiB/s (166kB/s)(6296KiB/38822msec) clat (usec): min=472, max=81710, avg=10814.42, stdev=11516.68 lat (usec): min=472, max=81711, avg=10820.77, stdev=11515.18 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 742], 10.00th=[ 783], 20.00th=[ 955], | 30.00th=[ 1844], 40.00th=[ 4178], 50.00th=[ 7963], 60.00th=[10814], | 70.00th=[13960], 80.00th=[18744], 90.00th=[26608], 95.00th=[32637], | 99.00th=[53216], 99.50th=[63701], 99.90th=[76022], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 24, max= 312, per=24.75%, avg=160.16, stdev=50.69, samples=77 iops : min= 6, max= 78, avg=39.78, stdev=12.73, samples=77 lat (usec) : 100=0.75%, 250=2.08%, 500=0.16%, 750=2.93%, 1000=7.81% lat (msec) : 2=5.14%, 4=4.65%, 10=24.97%, 20=32.32%, 50=18.23% lat (msec) : 100=0.91%, 250=0.03% cpu : usr=0.12%, sys=11.73%, ctx=5109, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22431: Thu May 1 19:06:04 2025 read: IOPS=38, BW=154KiB/s (158kB/s)(5940KiB/38569msec) clat (usec): min=60, max=80258, avg=14191.00, stdev=9522.74 lat (usec): min=60, max=80259, avg=14192.00, stdev=9522.73 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 121], 10.00th=[ 5145], 20.00th=[ 7177], | 30.00th=[ 8979], 40.00th=[10552], 50.00th=[12256], 60.00th=[14615], | 70.00th=[17171], 80.00th=[20317], 90.00th=[25560], 95.00th=[31589], | 99.00th=[50070], 99.50th=[55313], 99.90th=[69731], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 23, max= 280, per=24.81%, avg=153.32, stdev=40.57, samples=77 iops : min= 5, max= 70, avg=37.95, stdev=10.22, samples=77 write: IOPS=41, BW=165KiB/s (169kB/s)(6348KiB/38569msec) clat (usec): min=480, max=300129, avg=10948.59, stdev=13635.43 lat (usec): min=481, max=300131, avg=10950.47, stdev=13635.66 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 938], | 30.00th=[ 1745], 40.00th=[ 3523], 50.00th=[ 7308], 60.00th=[ 10552], | 70.00th=[ 13698], 80.00th=[ 19268], 90.00th=[ 27395], 95.00th=[ 33424], | 99.00th=[ 50594], 99.50th=[ 58459], 99.90th=[ 69731], 99.95th=[299893], | 99.99th=[299893] bw ( KiB/s): min= 87, max= 294, per=25.25%, avg=163.39, stdev=48.29, samples=77 iops : min= 21, max= 73, avg=40.47, stdev=12.12, samples=77 lat (usec) : 100=1.20%, 250=2.05%, 500=0.13%, 750=3.42%, 1000=7.85% lat (msec) : 2=5.34%, 4=5.21%, 10=22.75%, 20=32.39%, 50=18.68% lat (msec) : 100=0.94%, 500=0.03% cpu : usr=0.16%, sys=11.53%, ctx=4958, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22432: Thu May 1 19:06:04 2025 read: IOPS=40, BW=160KiB/s (164kB/s)(6028KiB/37587msec) clat (usec): min=62, max=95225, avg=14117.67, stdev=9210.37 lat (usec): min=63, max=95227, avg=14118.72, stdev=9210.39 clat percentiles (usec): | 1.00th=[ 96], 5.00th=[ 126], 10.00th=[ 5276], 20.00th=[ 7439], | 30.00th=[ 8979], 40.00th=[10552], 50.00th=[12387], 60.00th=[14353], | 70.00th=[16909], 80.00th=[20317], 90.00th=[25297], 95.00th=[29754], | 99.00th=[45351], 99.50th=[55313], 99.90th=[76022], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 39, max= 271, per=25.91%, avg=160.13, stdev=43.19, samples=75 iops : min= 9, max= 67, avg=39.73, stdev=10.83, samples=75 write: IOPS=41, BW=167KiB/s (171kB/s)(6260KiB/37587msec) clat (usec): min=473, max=223882, avg=10307.80, stdev=12146.19 lat (usec): min=474, max=223885, avg=10310.80, stdev=12147.89 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 922], | 30.00th=[ 1663], 40.00th=[ 3458], 50.00th=[ 7504], 60.00th=[ 9896], | 70.00th=[ 13173], 80.00th=[ 17695], 90.00th=[ 25035], 95.00th=[ 31589], | 99.00th=[ 47449], 99.50th=[ 57410], 99.90th=[ 76022], 99.95th=[223347], | 99.99th=[223347] bw ( KiB/s): min= 47, max= 279, per=25.66%, avg=166.00, stdev=51.03, samples=75 iops : min= 11, max= 69, avg=41.20, stdev=12.80, samples=75 lat (usec) : 100=0.94%, 250=1.89%, 500=0.13%, 750=3.58%, 1000=7.81% lat (msec) : 2=5.34%, 4=4.92%, 10=24.15%, 20=32.78%, 50=17.68% lat (msec) : 100=0.75%, 250=0.03% cpu : usr=0.24%, sys=11.45%, ctx=5060, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22433: Thu May 1 19:06:04 2025 read: IOPS=39, BW=157KiB/s (160kB/s)(6052KiB/38628msec) clat (usec): min=77, max=70962, avg=14167.95, stdev=9041.24 lat (usec): min=77, max=70963, avg=14169.33, stdev=9041.10 clat percentiles (usec): | 1.00th=[ 96], 5.00th=[ 182], 10.00th=[ 5538], 20.00th=[ 7635], | 30.00th=[ 8979], 40.00th=[10552], 50.00th=[12256], 60.00th=[14353], | 70.00th=[16909], 80.00th=[20317], 90.00th=[25035], 95.00th=[31065], | 99.00th=[46400], 99.50th=[52167], 99.90th=[57934], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 40, max= 255, per=25.30%, avg=156.32, stdev=37.09, samples=77 iops : min= 10, max= 63, avg=38.82, stdev= 9.26, samples=77 write: IOPS=40, BW=161KiB/s (165kB/s)(6236KiB/38628msec) clat (usec): min=480, max=202204, avg=10894.50, stdev=12677.03 lat (usec): min=481, max=202206, avg=10898.05, stdev=12676.12 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 930], | 30.00th=[ 1778], 40.00th=[ 3687], 50.00th=[ 7504], 60.00th=[ 10421], | 70.00th=[ 13829], 80.00th=[ 18744], 90.00th=[ 27132], 95.00th=[ 33424], | 99.00th=[ 50594], 99.50th=[ 62653], 99.90th=[ 96994], 99.95th=[202376], | 99.99th=[202376] bw ( KiB/s): min= 56, max= 320, per=24.82%, avg=160.56, stdev=47.21, samples=77 iops : min= 14, max= 80, avg=39.88, stdev=11.84, samples=77 lat (usec) : 100=0.88%, 250=1.69%, 500=0.10%, 750=2.83%, 1000=8.20% lat (msec) : 2=5.18%, 4=5.21%, 10=24.02%, 20=32.49%, 50=18.55% lat (msec) : 100=0.81%, 250=0.03% cpu : usr=0.13%, sys=11.73%, ctx=5170, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=619KiB/s (633kB/s), 154KiB/s-160KiB/s (158kB/s-164kB/s), io=23.4MiB (24.6MB), run=37587-38822msec WRITE: bw=648KiB/s (663kB/s), 161KiB/s-167KiB/s (165kB/s-171kB/s), io=24.6MiB (25.7MB), run=37587-38822msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22426: Thu May 1 19:06:19 2025 read: IOPS=27, BW=111KiB/s (114kB/s)(5992KiB/53784msec) clat (msec): min=3, max=310, avg=13.76, stdev=15.00 lat (msec): min=3, max=310, avg=13.76, stdev=15.00 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 19], 90.00th=[ 30], 95.00th=[ 39], | 99.00th=[ 72], 99.50th=[ 81], 99.90th=[ 112], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 23, max= 288, per=24.83%, avg=110.76, stdev=64.21, samples=107 iops : min= 5, max= 72, avg=27.50, stdev=16.09, samples=107 write: IOPS=29, BW=117KiB/s (120kB/s)(6296KiB/53784msec) clat (msec): min=4, max=126, avg=20.98, stdev=14.39 lat (msec): min=4, max=126, avg=20.99, stdev=14.39 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 11], | 30.00th=[ 13], 40.00th=[ 14], 50.00th=[ 16], 60.00th=[ 19], | 70.00th=[ 23], 80.00th=[ 30], 90.00th=[ 41], 95.00th=[ 51], | 99.00th=[ 80], 99.50th=[ 86], 99.90th=[ 107], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 24, max= 256, per=24.90%, avg=116.29, stdev=62.58, samples=107 iops : min= 6, max= 64, avg=28.88, stdev=15.67, samples=107 lat (msec) : 4=1.46%, 10=34.21%, 20=37.21%, 50=23.31%, 100=3.61% lat (msec) : 250=0.16%, 500=0.03% cpu : usr=0.10%, sys=5.82%, ctx=3365, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22427: Thu May 1 19:06:19 2025 read: IOPS=27, BW=111KiB/s (113kB/s)(5940KiB/53725msec) clat (msec): min=2, max=275, avg=13.24, stdev=13.67 lat (msec): min=2, max=275, avg=13.25, stdev=13.67 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 11], | 70.00th=[ 14], 80.00th=[ 18], 90.00th=[ 29], 95.00th=[ 37], | 99.00th=[ 65], 99.50th=[ 74], 99.90th=[ 111], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 8, max= 320, per=24.68%, avg=110.07, stdev=63.59, samples=107 iops : min= 2, max= 80, avg=27.31, stdev=15.94, samples=107 write: IOPS=29, BW=118KiB/s (121kB/s)(6348KiB/53725msec) clat (msec): min=4, max=114, avg=21.37, stdev=14.32 lat (msec): min=4, max=114, avg=21.37, stdev=14.33 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 12], | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 16], 60.00th=[ 19], | 70.00th=[ 24], 80.00th=[ 31], 90.00th=[ 40], 95.00th=[ 50], | 99.00th=[ 79], 99.50th=[ 88], 99.90th=[ 94], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 24, max= 280, per=25.12%, avg=117.32, stdev=62.35, samples=107 iops : min= 6, max= 70, avg=29.12, stdev=15.61, samples=107 lat (msec) : 4=1.56%, 10=32.78%, 20=37.96%, 50=24.06%, 100=3.55% lat (msec) : 250=0.07%, 500=0.03% cpu : usr=0.08%, sys=5.86%, ctx=3404, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22428: Thu May 1 19:06:19 2025 read: IOPS=28, BW=112KiB/s (115kB/s)(6028KiB/53780msec) clat (usec): min=3105, max=90408, avg=13702.03, stdev=12549.03 lat (usec): min=3106, max=90410, avg=13703.11, stdev=12549.05 clat percentiles (usec): | 1.00th=[ 3458], 5.00th=[ 4228], 10.00th=[ 4752], 20.00th=[ 5800], | 30.00th=[ 6587], 40.00th=[ 7570], 50.00th=[ 8717], 60.00th=[10290], | 70.00th=[12911], 80.00th=[19268], 90.00th=[32113], 95.00th=[42730], | 99.00th=[58983], 99.50th=[67634], 99.90th=[83362], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 24, max= 256, per=24.98%, avg=111.42, stdev=62.80, samples=107 iops : min= 6, max= 64, avg=27.61, stdev=15.70, samples=107 write: IOPS=29, BW=116KiB/s (119kB/s)(6260KiB/53780msec) clat (msec): min=5, max=228, avg=21.04, stdev=14.73 lat (msec): min=5, max=228, avg=21.04, stdev=14.73 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 12], | 30.00th=[ 13], 40.00th=[ 14], 50.00th=[ 16], 60.00th=[ 20], | 70.00th=[ 24], 80.00th=[ 30], 90.00th=[ 39], 95.00th=[ 48], | 99.00th=[ 78], 99.50th=[ 86], 99.90th=[ 110], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 31, max= 264, per=24.76%, avg=115.61, stdev=63.66, samples=107 iops : min= 7, max= 66, avg=28.65, stdev=15.94, samples=107 lat (msec) : 4=1.63%, 10=33.46%, 20=36.65%, 50=24.80%, 100=3.39% lat (msec) : 250=0.07% cpu : usr=0.17%, sys=5.74%, ctx=3381, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22429: Thu May 1 19:06:19 2025 read: IOPS=28, BW=113KiB/s (115kB/s)(6052KiB/53766msec) clat (msec): min=2, max=105, avg=13.19, stdev=12.42 lat (msec): min=2, max=105, avg=13.19, stdev=12.42 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 13], 80.00th=[ 18], 90.00th=[ 30], 95.00th=[ 41], | 99.00th=[ 61], 99.50th=[ 69], 99.90th=[ 101], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 23, max= 320, per=25.00%, avg=111.50, stdev=63.17, samples=107 iops : min= 5, max= 80, avg=27.57, stdev=15.88, samples=107 write: IOPS=28, BW=116KiB/s (119kB/s)(6236KiB/53766msec) clat (msec): min=5, max=286, avg=21.60, stdev=15.76 lat (msec): min=5, max=286, avg=21.60, stdev=15.76 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 12], | 30.00th=[ 13], 40.00th=[ 14], 50.00th=[ 16], 60.00th=[ 19], | 70.00th=[ 24], 80.00th=[ 32], 90.00th=[ 41], 95.00th=[ 52], | 99.00th=[ 73], 99.50th=[ 83], 99.90th=[ 122], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 31, max= 280, per=24.66%, avg=115.17, stdev=62.58, samples=107 iops : min= 7, max= 70, avg=28.49, stdev=15.72, samples=107 lat (msec) : 4=1.56%, 10=34.28%, 20=36.72%, 50=23.47%, 100=3.81% lat (msec) : 250=0.13%, 500=0.03% cpu : usr=0.13%, sys=5.83%, ctx=3379, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=446KiB/s (457kB/s), 111KiB/s-113KiB/s (113kB/s-115kB/s), io=23.4MiB (24.6MB), run=53725-53784msec WRITE: bw=467KiB/s (479kB/s), 116KiB/s-118KiB/s (119kB/s-121kB/s), io=24.6MiB (25.7MB), run=53725-53784msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22498: Thu May 1 19:06:31 2025 read: IOPS=37, BW=594KiB/s (608kB/s)(5920KiB/9972msec) clat (usec): min=68, max=53097, avg=13727.80, stdev=9468.88 lat (usec): min=69, max=53097, avg=13736.34, stdev=9470.76 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 111], 10.00th=[ 125], 20.00th=[ 6587], | 30.00th=[ 8717], 40.00th=[10814], 50.00th=[12911], 60.00th=[15139], | 70.00th=[17171], 80.00th=[20841], 90.00th=[25297], 95.00th=[30802], | 99.00th=[43254], 99.50th=[50070], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 382, max= 1021, per=24.64%, avg=583.21, stdev=156.09, samples=19 iops : min= 23, max= 63, avg=36.05, stdev= 9.65, samples=19 write: IOPS=39, BW=639KiB/s (654kB/s)(6368KiB/9972msec) clat (usec): min=856, max=68917, avg=12033.48, stdev=13072.67 lat (usec): min=857, max=68918, avg=12038.65, stdev=13069.83 clat percentiles (usec): | 1.00th=[ 947], 5.00th=[ 1074], 10.00th=[ 1172], 20.00th=[ 1352], | 30.00th=[ 2311], 40.00th=[ 3818], 50.00th=[ 7373], 60.00th=[10814], | 70.00th=[15270], 80.00th=[21103], 90.00th=[30278], 95.00th=[38011], | 99.00th=[56886], 99.50th=[64226], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 384, max= 1017, per=25.10%, avg=625.11, stdev=180.08, samples=19 iops : min= 24, max= 63, avg=38.68, stdev=11.14, samples=19 lat (usec) : 100=1.04%, 250=5.73%, 500=0.13%, 1000=1.04% lat (msec) : 2=13.54%, 4=7.03%, 10=18.49%, 20=31.38%, 50=20.44% lat (msec) : 100=1.17% cpu : usr=0.25%, sys=11.87%, ctx=1269, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22499: Thu May 1 19:06:31 2025 read: IOPS=35, BW=568KiB/s (581kB/s)(5744KiB/10117msec) clat (usec): min=81, max=76728, avg=15348.68, stdev=12853.97 lat (usec): min=82, max=76729, avg=15349.80, stdev=12854.07 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 100], 10.00th=[ 116], 20.00th=[ 2147], | 30.00th=[ 8979], 40.00th=[11076], 50.00th=[13435], 60.00th=[15795], | 70.00th=[19792], 80.00th=[23200], 90.00th=[32375], 95.00th=[40109], | 99.00th=[58459], 99.50th=[65799], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 287, max= 992, per=23.92%, avg=566.15, stdev=171.32, samples=20 iops : min= 17, max= 62, avg=35.10, stdev=10.69, samples=20 write: IOPS=40, BW=647KiB/s (662kB/s)(6544KiB/10117msec) clat (usec): min=843, max=78042, avg=11175.80, stdev=12839.60 lat (usec): min=844, max=78044, avg=11178.50, stdev=12839.73 clat percentiles (usec): | 1.00th=[ 963], 5.00th=[ 1057], 10.00th=[ 1139], 20.00th=[ 1287], | 30.00th=[ 2245], 40.00th=[ 3818], 50.00th=[ 6718], 60.00th=[10159], | 70.00th=[13435], 80.00th=[18744], 90.00th=[27919], 95.00th=[35390], | 99.00th=[65274], 99.50th=[70779], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 352, max= 956, per=25.83%, avg=643.10, stdev=159.24, samples=20 iops : min= 22, max= 59, avg=39.90, stdev= 9.85, samples=20 lat (usec) : 100=2.21%, 250=5.99%, 500=0.52%, 750=0.26%, 1000=1.17% lat (msec) : 2=13.41%, 4=7.94%, 10=16.41%, 20=28.91%, 50=21.09% lat (msec) : 100=2.08% cpu : usr=0.18%, sys=12.38%, ctx=1221, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22501: Thu May 1 19:06:31 2025 read: IOPS=39, BW=632KiB/s (648kB/s)(6256KiB/9892msec) clat (usec): min=87, max=68197, avg=14104.44, stdev=11171.12 lat (usec): min=88, max=68198, avg=14105.40, stdev=11171.14 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 100], 10.00th=[ 110], 20.00th=[ 1582], | 30.00th=[ 8586], 40.00th=[10945], 50.00th=[12911], 60.00th=[15139], | 70.00th=[18220], 80.00th=[22414], 90.00th=[28443], 95.00th=[34341], | 99.00th=[44827], 99.50th=[67634], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 256, max= 1021, per=26.81%, avg=634.58, stdev=201.10, samples=19 iops : min= 16, max= 63, avg=39.16, stdev=12.40, samples=19 write: IOPS=38, BW=610KiB/s (624kB/s)(6032KiB/9892msec) clat (usec): min=703, max=69081, avg=11131.29, stdev=11629.12 lat (usec): min=704, max=69083, avg=11133.70, stdev=11630.02 clat percentiles (usec): | 1.00th=[ 971], 5.00th=[ 1106], 10.00th=[ 1188], 20.00th=[ 1516], | 30.00th=[ 2540], 40.00th=[ 3982], 50.00th=[ 7373], 60.00th=[10552], | 70.00th=[13566], 80.00th=[19792], 90.00th=[27395], 95.00th=[34341], | 99.00th=[51119], 99.50th=[64226], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 287, max= 1245, per=23.80%, avg=592.53, stdev=256.39, samples=19 iops : min= 17, max= 77, avg=36.53, stdev=15.96, samples=19 lat (usec) : 100=2.73%, 250=6.64%, 500=0.52%, 750=0.13%, 1000=0.65% lat (msec) : 2=12.63%, 4=6.90%, 10=16.80%, 20=30.08%, 50=22.14% lat (msec) : 100=0.78% cpu : usr=0.14%, sys=12.29%, ctx=1277, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22502: Thu May 1 19:06:31 2025 read: IOPS=38, BW=618KiB/s (633kB/s)(6032KiB/9761msec) clat (usec): min=57, max=67093, avg=14080.87, stdev=10848.05 lat (usec): min=57, max=67094, avg=14081.84, stdev=10848.04 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 105], 10.00th=[ 116], 20.00th=[ 5080], | 30.00th=[ 8717], 40.00th=[10814], 50.00th=[12780], 60.00th=[15401], | 70.00th=[17957], 80.00th=[21365], 90.00th=[27657], 95.00th=[33817], | 99.00th=[51643], 99.50th=[56886], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 256, max= 832, per=25.86%, avg=612.11, stdev=135.21, samples=19 iops : min= 16, max= 52, avg=38.00, stdev= 8.56, samples=19 write: IOPS=40, BW=641KiB/s (656kB/s)(6256KiB/9761msec) clat (usec): min=859, max=101793, avg=10984.29, stdev=13153.20 lat (usec): min=860, max=101795, avg=10985.79, stdev=13153.21 clat percentiles (usec): | 1.00th=[ 971], 5.00th=[ 1090], 10.00th=[ 1156], 20.00th=[ 1319], | 30.00th=[ 1844], 40.00th=[ 3130], 50.00th=[ 5407], 60.00th=[ 9634], | 70.00th=[ 12780], 80.00th=[ 19268], 90.00th=[ 28705], 95.00th=[ 35390], | 99.00th=[ 62653], 99.50th=[ 82314], 99.90th=[102237], 99.95th=[102237], | 99.99th=[102237] bw ( KiB/s): min= 352, max= 1248, per=25.73%, avg=640.58, stdev=193.89, samples=19 iops : min= 22, max= 78, avg=39.79, stdev=12.12, samples=19 lat (usec) : 100=2.08%, 250=6.38%, 500=0.26%, 1000=0.65% lat (msec) : 2=15.76%, 4=6.90%, 10=16.93%, 20=28.78%, 50=20.96% lat (msec) : 100=1.17%, 250=0.13% cpu : usr=0.06%, sys=12.16%, ctx=1240, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2368KiB/s (2424kB/s), 568KiB/s-632KiB/s (581kB/s-648kB/s), io=23.4MiB (24.5MB), run=9761-10117msec WRITE: bw=2491KiB/s (2551kB/s), 610KiB/s-647KiB/s (624kB/s-662kB/s), io=24.6MiB (25.8MB), run=9761-10117msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22495: Thu May 1 19:06:36 2025 read: IOPS=24, BW=387KiB/s (396kB/s)(5920KiB/15308msec) clat (usec): min=3186, max=97237, avg=15817.97, stdev=15434.96 lat (usec): min=3187, max=97238, avg=15820.54, stdev=15434.57 clat percentiles (usec): | 1.00th=[ 3884], 5.00th=[ 4490], 10.00th=[ 5145], 20.00th=[ 6194], | 30.00th=[ 7439], 40.00th=[ 8455], 50.00th=[ 9896], 60.00th=[11731], | 70.00th=[13960], 80.00th=[20055], 90.00th=[40109], 95.00th=[52167], | 99.00th=[66323], 99.50th=[88605], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 160, max= 862, per=24.10%, avg=376.90, stdev=202.54, samples=30 iops : min= 10, max= 53, avg=23.17, stdev=12.72, samples=30 write: IOPS=25, BW=416KiB/s (426kB/s)(6368KiB/15308msec) clat (msec): min=6, max=113, avg=23.37, stdev=15.03 lat (msec): min=6, max=113, avg=23.37, stdev=15.03 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 21], | 70.00th=[ 27], 80.00th=[ 33], 90.00th=[ 43], 95.00th=[ 54], | 99.00th=[ 78], 99.50th=[ 104], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 96, max= 800, per=24.97%, avg=410.97, stdev=230.53, samples=30 iops : min= 6, max= 50, avg=25.30, stdev=14.46, samples=30 lat (msec) : 4=1.17%, 10=26.43%, 20=40.36%, 50=26.17%, 100=5.47% lat (msec) : 250=0.39% cpu : usr=0.09%, sys=5.59%, ctx=869, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22496: Thu May 1 19:06:36 2025 read: IOPS=23, BW=384KiB/s (393kB/s)(5744KiB/14972msec) clat (usec): min=3804, max=97026, avg=14305.57, stdev=11947.50 lat (usec): min=3805, max=97027, avg=14308.17, stdev=11946.91 clat percentiles (usec): | 1.00th=[ 3916], 5.00th=[ 4752], 10.00th=[ 5538], 20.00th=[ 6915], | 30.00th=[ 7832], 40.00th=[ 8979], 50.00th=[10159], 60.00th=[11863], | 70.00th=[14091], 80.00th=[18482], 90.00th=[30278], 95.00th=[40109], | 99.00th=[58983], 99.50th=[67634], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 126, max= 768, per=23.93%, avg=374.31, stdev=192.68, samples=29 iops : min= 7, max= 48, avg=22.93, stdev=12.16, samples=29 write: IOPS=27, BW=437KiB/s (448kB/s)(6544KiB/14972msec) clat (msec): min=6, max=101, avg=23.93, stdev=15.49 lat (msec): min=6, max=101, avg=23.93, stdev=15.49 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 22], | 70.00th=[ 25], 80.00th=[ 33], 90.00th=[ 46], 95.00th=[ 60], | 99.00th=[ 79], 99.50th=[ 88], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 191, max= 992, per=25.56%, avg=420.66, stdev=194.42, samples=29 iops : min= 11, max= 62, avg=25.83, stdev=12.33, samples=29 lat (msec) : 4=0.52%, 10=25.39%, 20=41.80%, 50=27.47%, 100=4.69% lat (msec) : 250=0.13% cpu : usr=0.00%, sys=5.87%, ctx=872, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22497: Thu May 1 19:06:36 2025 read: IOPS=25, BW=409KiB/s (419kB/s)(6256KiB/15281msec) clat (usec): min=3887, max=98930, avg=14649.33, stdev=13795.54 lat (usec): min=3888, max=99750, avg=14652.66, stdev=13808.35 clat percentiles (usec): | 1.00th=[ 4228], 5.00th=[ 4948], 10.00th=[ 5735], 20.00th=[ 6587], | 30.00th=[ 7635], 40.00th=[ 8586], 50.00th=[ 9765], 60.00th=[11076], | 70.00th=[13173], 80.00th=[19268], 90.00th=[29754], 95.00th=[44303], | 99.00th=[80217], 99.50th=[88605], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 159, max= 1088, per=25.92%, avg=405.47, stdev=246.71, samples=30 iops : min= 9, max= 68, avg=24.93, stdev=15.50, samples=30 write: IOPS=24, BW=395KiB/s (404kB/s)(6032KiB/15281msec) clat (msec): min=5, max=102, avg=24.92, stdev=16.48 lat (msec): min=5, max=102, avg=24.93, stdev=16.48 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 23], | 70.00th=[ 27], 80.00th=[ 36], 90.00th=[ 49], 95.00th=[ 64], | 99.00th=[ 80], 99.50th=[ 88], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 160, max= 896, per=23.41%, avg=385.27, stdev=204.99, samples=30 iops : min= 10, max= 56, avg=23.67, stdev=12.91, samples=30 lat (msec) : 4=0.13%, 10=28.78%, 20=38.80%, 50=26.04%, 100=6.12% lat (msec) : 250=0.13% cpu : usr=0.05%, sys=5.79%, ctx=872, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22500: Thu May 1 19:06:36 2025 read: IOPS=25, BW=401KiB/s (411kB/s)(6032KiB/15042msec) clat (usec): min=3037, max=78008, avg=15291.71, stdev=13109.35 lat (usec): min=3037, max=78009, avg=15292.82, stdev=13109.33 clat percentiles (usec): | 1.00th=[ 3884], 5.00th=[ 4817], 10.00th=[ 5538], 20.00th=[ 6718], | 30.00th=[ 7635], 40.00th=[ 8848], 50.00th=[10290], 60.00th=[12125], | 70.00th=[15139], 80.00th=[20579], 90.00th=[33817], 95.00th=[44827], | 99.00th=[66847], 99.50th=[74974], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 128, max= 894, per=25.41%, avg=397.47, stdev=212.41, samples=30 iops : min= 8, max= 55, avg=24.67, stdev=13.22, samples=30 write: IOPS=25, BW=416KiB/s (426kB/s)(6256KiB/15042msec) clat (msec): min=4, max=100, avg=23.46, stdev=15.07 lat (msec): min=4, max=100, avg=23.46, stdev=15.07 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 21], | 70.00th=[ 26], 80.00th=[ 33], 90.00th=[ 45], 95.00th=[ 56], | 99.00th=[ 78], 99.50th=[ 94], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 160, max= 832, per=25.25%, avg=415.53, stdev=215.01, samples=30 iops : min= 10, max= 52, avg=25.80, stdev=13.36, samples=30 lat (msec) : 4=0.52%, 10=25.78%, 20=42.32%, 50=26.04%, 100=5.21% lat (msec) : 250=0.13% cpu : usr=0.06%, sys=5.75%, ctx=863, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1565KiB/s (1602kB/s), 384KiB/s-409KiB/s (393kB/s-419kB/s), io=23.4MiB (24.5MB), run=14972-15308msec WRITE: bw=1646KiB/s (1686kB/s), 395KiB/s-437KiB/s (404kB/s-448kB/s), io=24.6MiB (25.8MB), run=14972-15308msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22521: Thu May 1 19:06:40 2025 read: IOPS=2, BW=2826KiB/s (2894kB/s)(7168KiB/2536msec) clat (usec): min=513, max=1312.6k, avg=269937.96, stdev=465838.27 lat (usec): min=515, max=1312.6k, avg=269939.27, stdev=465838.64 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 515], 10.00th=[ 515], | 20.00th=[ 72877], 30.00th=[ 76022], 40.00th=[ 76022], | 50.00th=[ 81265], 60.00th=[ 96994], 70.00th=[ 96994], | 80.00th=[ 248513], 90.00th=[1317012], 95.00th=[1317012], | 99.00th=[1317012], 99.50th=[1317012], 99.90th=[1317012], | 99.95th=[1317012], 99.99th=[1317012] bw ( KiB/s): min= 4096, max= 6131, per=68.02%, avg=5113.50, stdev=1438.96, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=1, BW=2019KiB/s (2067kB/s)(5120KiB/2536msec) clat (msec): min=48, max=186, avg=101.47, stdev=61.63 lat (msec): min=48, max=186, avg=101.52, stdev=61.63 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 49], 10.00th=[ 49], 20.00th=[ 49], | 30.00th=[ 55], 40.00th=[ 55], 50.00th=[ 71], 60.00th=[ 71], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 186], 95.00th=[ 186], | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 6131, max= 6131, per=58.25%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=8.33%, 100=50.00%, 250=25.00% cpu : usr=0.04%, sys=14.99%, ctx=463, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22522: Thu May 1 19:06:40 2025 read: IOPS=0, BW=752KiB/s (770kB/s)(2048KiB/2724msec) clat (usec): min=91217, max=95687, avg=93452.54, stdev=3160.64 lat (usec): min=91219, max=95688, avg=93453.98, stdev=3159.64 clat percentiles (usec): | 1.00th=[90702], 5.00th=[90702], 10.00th=[90702], 20.00th=[90702], | 30.00th=[90702], 40.00th=[90702], 50.00th=[90702], 60.00th=[95945], | 70.00th=[95945], 80.00th=[95945], 90.00th=[95945], 95.00th=[95945], | 99.00th=[95945], 99.50th=[95945], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 1875, max= 2048, per=26.09%, avg=1961.50, stdev=122.33, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3759KiB/s (3849kB/s)(10.0MiB/2724msec) clat (msec): min=29, max=1256, avg=247.65, stdev=358.57 lat (msec): min=30, max=1256, avg=247.72, stdev=358.57 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 79], | 30.00th=[ 110], 40.00th=[ 131], 50.00th=[ 136], 60.00th=[ 165], | 70.00th=[ 186], 80.00th=[ 186], 90.00th=[ 213], 95.00th=[ 1250], | 99.00th=[ 1250], 99.50th=[ 1250], 99.90th=[ 1250], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 4096, max= 6144, per=50.25%, avg=5288.67, stdev=1064.86, samples=3 iops : min= 4, max= 6, avg= 5.00, stdev= 1.00, samples=3 lat (msec) : 50=8.33%, 100=25.00%, 250=58.33% cpu : usr=0.00%, sys=9.18%, ctx=149, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22524: Thu May 1 19:06:40 2025 read: IOPS=2, BW=2837KiB/s (2905kB/s)(6144KiB/2166msec) clat (usec): min=454, max=95195, avg=41172.35, stdev=38752.64 lat (usec): min=455, max=95197, avg=41173.70, stdev=38752.87 clat percentiles (usec): | 1.00th=[ 453], 5.00th=[ 453], 10.00th=[ 453], 20.00th=[ 594], | 30.00th=[ 594], 40.00th=[30802], 50.00th=[30802], 60.00th=[44827], | 70.00th=[74974], 80.00th=[74974], 90.00th=[94897], 95.00th=[94897], | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2837KiB/s (2905kB/s)(6144KiB/2166msec) clat (msec): min=56, max=1289, avg=298.57, stdev=486.31 lat (msec): min=56, max=1289, avg=298.64, stdev=486.33 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 72], | 30.00th=[ 72], 40.00th=[ 112], 50.00th=[ 112], 60.00th=[ 115], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 1284], 95.00th=[ 1284], | 99.00th=[ 1284], 99.50th=[ 1284], 99.90th=[ 1284], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 4087, max= 6131, per=48.54%, avg=5109.00, stdev=1445.33, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=16.67%, 100=33.33%, 250=25.00% cpu : usr=0.09%, sys=9.52%, ctx=182, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22527: Thu May 1 19:06:40 2025 read: IOPS=2, BW=2229KiB/s (2282kB/s)(5120KiB/2297msec) clat (usec): min=591, max=130088, avg=43694.14, stdev=55900.76 lat (usec): min=593, max=130090, avg=43695.47, stdev=55900.99 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 594], 10.00th=[ 594], 20.00th=[ 594], | 30.00th=[ 3720], 40.00th=[ 3720], 50.00th=[ 13960], 60.00th=[ 13960], | 70.00th=[ 69731], 80.00th=[ 69731], 90.00th=[130548], 95.00th=[130548], | 99.00th=[130548], 99.50th=[130548], 99.90th=[130548], 99.95th=[130548], | 99.99th=[130548] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3121KiB/s (3195kB/s)(7168KiB/2297msec) clat (msec): min=53, max=1383, avg=293.81, stdev=483.89 lat (msec): min=54, max=1383, avg=293.89, stdev=483.88 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 54], 10.00th=[ 54], 20.00th=[ 64], | 30.00th=[ 70], 40.00th=[ 70], 50.00th=[ 116], 60.00th=[ 167], | 70.00th=[ 167], 80.00th=[ 203], 90.00th=[ 1385], 95.00th=[ 1385], | 99.00th=[ 1385], 99.50th=[ 1385], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 2048, max= 6131, per=38.86%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (usec) : 750=8.33% lat (msec) : 4=8.33%, 20=8.33%, 100=33.33%, 250=33.33% cpu : usr=0.17%, sys=9.97%, ctx=83, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7518KiB/s (7699kB/s), 752KiB/s-2837KiB/s (770kB/s-2905kB/s), io=20.0MiB (20.0MB), run=2166-2724msec WRITE: bw=10.3MiB/s (10.8MB/s), 2019KiB/s-3759KiB/s (2067kB/s-3849kB/s), io=28.0MiB (29.4MB), run=2166-2724msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22523: Thu May 1 19:06:40 2025 read: IOPS=2, BW=2507KiB/s (2567kB/s)(7168KiB/2859msec) clat (msec): min=31, max=1345, avg=294.35, stdev=468.91 lat (msec): min=31, max=1345, avg=294.35, stdev=468.91 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 44], | 30.00th=[ 110], 40.00th=[ 110], 50.00th=[ 113], 60.00th=[ 205], | 70.00th=[ 205], 80.00th=[ 215], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 4015, max= 6144, per=72.57%, avg=5079.50, stdev=1505.43, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 write: IOPS=1, BW=1791KiB/s (1834kB/s)(5120KiB/2859msec) clat (msec): min=55, max=173, avg=113.80, stdev=43.59 lat (msec): min=55, max=173, avg=114.10, stdev=43.54 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 56], 10.00th=[ 56], 20.00th=[ 56], | 30.00th=[ 101], 40.00th=[ 101], 50.00th=[ 107], 60.00th=[ 107], | 70.00th=[ 133], 80.00th=[ 133], 90.00th=[ 174], 95.00th=[ 174], | 99.00th=[ 174], 99.50th=[ 174], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 2048, max= 4015, per=30.94%, avg=3031.50, stdev=1390.88, samples=2 iops : min= 2, max= 3, avg= 2.50, stdev= 0.71, samples=2 lat (msec) : 50=16.67%, 100=8.33%, 250=66.67% cpu : usr=0.07%, sys=4.65%, ctx=84, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22525: Thu May 1 19:06:40 2025 read: IOPS=0, BW=700KiB/s (717kB/s)(2048KiB/2926msec) clat (msec): min=40, max=143, avg=92.00, stdev=73.10 lat (msec): min=40, max=143, avg=92.00, stdev=73.10 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 41], 60.00th=[ 144], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 2043, max= 2043, per=29.19%, avg=2043.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3500KiB/s (3584kB/s)(10.0MiB/2926msec) clat (msec): min=34, max=1378, avg=268.27, stdev=397.98 lat (msec): min=34, max=1378, avg=268.38, stdev=397.99 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 70], | 30.00th=[ 86], 40.00th=[ 90], 50.00th=[ 150], 60.00th=[ 167], | 70.00th=[ 203], 80.00th=[ 203], 90.00th=[ 300], 95.00th=[ 1385], | 99.00th=[ 1385], 99.50th=[ 1385], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 4087, max= 6131, per=52.14%, avg=5109.00, stdev=1445.33, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 lat (msec) : 50=16.67%, 100=25.00%, 250=41.67%, 500=8.33% cpu : usr=0.00%, sys=4.79%, ctx=56, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22526: Thu May 1 19:06:40 2025 read: IOPS=2, BW=2165KiB/s (2217kB/s)(6144KiB/2838msec) clat (msec): min=27, max=268, avg=81.42, stdev=92.37 lat (msec): min=27, max=268, avg=81.42, stdev=92.37 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 35], | 30.00th=[ 35], 40.00th=[ 48], 50.00th=[ 48], 60.00th=[ 50], | 70.00th=[ 62], 80.00th=[ 62], 90.00th=[ 268], 95.00th=[ 268], | 99.00th=[ 268], 99.50th=[ 268], 99.90th=[ 268], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 2043, max= 2048, per=29.23%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2165KiB/s (2217kB/s)(6144KiB/2838msec) clat (msec): min=39, max=1319, avg=369.38, stdev=488.44 lat (msec): min=40, max=1319, avg=369.44, stdev=488.44 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 89], | 30.00th=[ 89], 40.00th=[ 102], 50.00th=[ 102], 60.00th=[ 215], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 1318], 95.00th=[ 1318], | 99.00th=[ 1318], 99.50th=[ 1318], 99.90th=[ 1318], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 2048, max= 4096, per=31.35%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (msec) : 50=41.67%, 100=16.67%, 250=16.67%, 500=16.67% cpu : usr=0.07%, sys=5.08%, ctx=72, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22528: Thu May 1 19:06:40 2025 read: IOPS=1, BW=1831KiB/s (1874kB/s)(5120KiB/2797msec) clat (msec): min=49, max=137, avg=96.02, stdev=41.27 lat (msec): min=49, max=137, avg=96.02, stdev=41.27 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 51], 10.00th=[ 51], 20.00th=[ 51], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 104], 60.00th=[ 104], | 70.00th=[ 133], 80.00th=[ 133], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 8062, max= 8062, per=100.00%, avg=8062.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2563KiB/s (2624kB/s)(7168KiB/2797msec) clat (msec): min=48, max=1255, avg=309.43, stdev=424.64 lat (msec): min=48, max=1255, avg=309.51, stdev=424.67 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 49], 10.00th=[ 49], 20.00th=[ 57], | 30.00th=[ 153], 40.00th=[ 153], 50.00th=[ 176], 60.00th=[ 211], | 70.00th=[ 211], 80.00th=[ 271], 90.00th=[ 1250], 95.00th=[ 1250], | 99.00th=[ 1250], 99.50th=[ 1250], 99.90th=[ 1250], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 2015, max= 6131, per=34.68%, avg=3398.00, stdev=2366.90, samples=3 iops : min= 1, max= 5, avg= 2.67, stdev= 2.08, samples=3 lat (msec) : 50=16.67%, 100=16.67%, 250=50.00%, 500=8.33% cpu : usr=0.00%, sys=4.18%, ctx=69, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6999KiB/s (7167kB/s), 700KiB/s-2507KiB/s (717kB/s-2567kB/s), io=20.0MiB (20.0MB), run=2797-2926msec WRITE: bw=9799KiB/s (10.0MB/s), 1791KiB/s-3500KiB/s (1834kB/s-3584kB/s), io=28.0MiB (29.4MB), run=2797-2926msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22536: Thu May 1 19:06:44 2025 read: IOPS=2, BW=11.7MiB/s (12.2MB/s)(12.0MiB/1029msec) clat (msec): min=246, max=411, avg=335.85, stdev=83.41 lat (msec): min=246, max=411, avg=335.85, stdev=83.41 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 247], 10.00th=[ 247], 20.00th=[ 247], | 30.00th=[ 247], 40.00th=[ 351], 50.00th=[ 351], 60.00th=[ 351], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 8175, max= 8192, per=93.21%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=12.35%, ctx=311, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22537: Thu May 1 19:06:44 2025 read: IOPS=0, BW=2195KiB/s (2248kB/s)(4096KiB/1866msec) clat (nsec): min=269237k, max=269237k, avg=269236585.00, stdev= 0.00 lat (nsec): min=269239k, max=269239k, avg=269239341.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 271], 10.00th=[ 271], 20.00th=[ 271], | 30.00th=[ 271], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 8192, max= 8192, per=93.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4390KiB/s (4496kB/s)(8192KiB/1866msec) clat (msec): min=694, max=767, avg=731.05, stdev=51.62 lat (msec): min=694, max=767, avg=731.35, stdev=51.59 clat percentiles (msec): | 1.00th=[ 693], 5.00th=[ 693], 10.00th=[ 693], 20.00th=[ 693], | 30.00th=[ 693], 40.00th=[ 693], 50.00th=[ 693], 60.00th=[ 768], | 70.00th=[ 768], 80.00th=[ 768], 90.00th=[ 768], 95.00th=[ 768], | 99.00th=[ 768], 99.50th=[ 768], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 5580, max= 5580, per=31.78%, avg=5580.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=12.65%, ctx=171, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22538: Thu May 1 19:06:44 2025 write: IOPS=1, BW=7571KiB/s (7753kB/s)(12.0MiB/1623msec) clat (msec): min=312, max=671, avg=520.32, stdev=186.16 lat (msec): min=312, max=671, avg=520.58, stdev=186.11 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 313], 10.00th=[ 313], 20.00th=[ 313], | 30.00th=[ 313], 40.00th=[ 575], 50.00th=[ 575], 60.00th=[ 575], | 70.00th=[ 676], 80.00th=[ 676], 90.00th=[ 676], 95.00th=[ 676], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 8175, max= 8175, per=46.55%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.12%, sys=18.13%, ctx=69, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22539: Thu May 1 19:06:44 2025 write: IOPS=2, BW=10.1MiB/s (10.6MB/s)(12.0MiB/1186msec) clat (msec): min=264, max=545, avg=388.55, stdev=143.35 lat (msec): min=264, max=545, avg=388.84, stdev=143.35 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 266], 10.00th=[ 266], 20.00th=[ 266], | 30.00th=[ 266], 40.00th=[ 355], 50.00th=[ 355], 60.00th=[ 355], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 8143, max= 8192, per=46.51%, avg=8167.50, stdev=34.65, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.08%, sys=22.70%, ctx=72, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8780KiB/s (8991kB/s), 2195KiB/s-11.7MiB/s (2248kB/s-12.2MB/s), io=16.0MiB (16.8MB), run=1029-1866msec WRITE: bw=17.1MiB/s (17.0MB/s), 4390KiB/s-10.1MiB/s (4496kB/s-10.6MB/s), io=32.0MiB (33.6MB), run=1186-1866msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22540: Thu May 1 19:06:44 2025 read: IOPS=1, BW=6686KiB/s (6846kB/s)(12.0MiB/1838msec) clat (msec): min=449, max=675, avg=592.64, stdev=124.63 lat (msec): min=449, max=675, avg=592.64, stdev=124.63 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 451], 10.00th=[ 451], 20.00th=[ 451], | 30.00th=[ 451], 40.00th=[ 651], 50.00th=[ 651], 60.00th=[ 651], | 70.00th=[ 676], 80.00th=[ 676], 90.00th=[ 676], 95.00th=[ 676], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 8110, max= 8192, per=98.56%, avg=8151.00, stdev=57.98, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.27%, sys=5.61%, ctx=60, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22541: Thu May 1 19:06:44 2025 read: IOPS=0, BW=2068KiB/s (2117kB/s)(4096KiB/1981msec) clat (nsec): min=667622k, max=667622k, avg=667622150.00, stdev= 0.00 lat (nsec): min=667625k, max=667625k, avg=667625022.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 667], 5.00th=[ 667], 10.00th=[ 667], 20.00th=[ 667], | 30.00th=[ 667], 40.00th=[ 667], 50.00th=[ 667], 60.00th=[ 667], | 70.00th=[ 667], 80.00th=[ 667], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 8110, max= 8110, per=98.07%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4135KiB/s (4235kB/s)(8192KiB/1981msec) clat (msec): min=590, max=620, avg=605.34, stdev=20.83 lat (msec): min=590, max=620, avg=605.66, stdev=20.87 clat percentiles (msec): | 1.00th=[ 592], 5.00th=[ 592], 10.00th=[ 592], 20.00th=[ 592], | 30.00th=[ 592], 40.00th=[ 592], 50.00th=[ 592], 60.00th=[ 617], | 70.00th=[ 617], 80.00th=[ 617], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 8159, max= 8159, per=54.95%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 750=100.00% cpu : usr=0.25%, sys=4.55%, ctx=38, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22542: Thu May 1 19:06:44 2025 write: IOPS=1, BW=6631KiB/s (6791kB/s)(12.0MiB/1853msec) clat (msec): min=332, max=760, avg=588.06, stdev=226.00 lat (msec): min=332, max=760, avg=588.34, stdev=225.98 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 334], 10.00th=[ 334], 20.00th=[ 334], | 30.00th=[ 334], 40.00th=[ 676], 50.00th=[ 676], 60.00th=[ 676], | 70.00th=[ 760], 80.00th=[ 760], 90.00th=[ 760], 95.00th=[ 760], | 99.00th=[ 760], 99.50th=[ 760], 99.90th=[ 760], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 8192, max= 8192, per=55.18%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.22%, sys=3.78%, ctx=37, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22543: Thu May 1 19:06:44 2025 write: IOPS=1, BW=5568KiB/s (5701kB/s)(12.0MiB/2207msec) clat (msec): min=171, max=1064, avg=680.23, stdev=459.04 lat (msec): min=172, max=1064, avg=680.46, stdev=459.11 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 802], 50.00th=[ 802], 60.00th=[ 802], | 70.00th=[ 1062], 80.00th=[ 1062], 90.00th=[ 1062], 95.00th=[ 1062], | 99.00th=[ 1062], 99.50th=[ 1062], 99.90th=[ 1062], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 8192, max= 8192, per=55.18%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.23%, sys=7.34%, ctx=104, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8271KiB/s (8469kB/s), 2068KiB/s-6686KiB/s (2117kB/s-6846kB/s), io=16.0MiB (16.8MB), run=1838-1981msec WRITE: bw=14.5MiB/s (15.2MB/s), 4135KiB/s-6631KiB/s (4235kB/s-6791kB/s), io=32.0MiB (33.6MB), run=1853-2207msec