== sanity test 398b: DIO and buffer IO race ============== 14:09:00 (1744135740) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.0669 s, 47.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11147: Tue Apr 8 14:09:16 2025 read: IOPS=114, BW=458KiB/s (469kB/s)(5992KiB/13092msec) clat (usec): min=54, max=157925, avg=3932.88, stdev=5098.57 lat (usec): min=54, max=157925, avg=3933.98, stdev=5098.79 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 77], 10.00th=[ 94], 20.00th=[ 2008], | 30.00th=[ 2442], 40.00th=[ 2802], 50.00th=[ 3195], 60.00th=[ 3752], | 70.00th=[ 4752], 80.00th=[ 6063], 90.00th=[ 7635], 95.00th=[ 8979], | 99.00th=[ 12649], 99.50th=[ 16188], 99.90th=[ 56886], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 336, max= 688, per=25.29%, avg=457.04, stdev=82.84, samples=26 iops : min= 84, max= 172, avg=114.15, stdev=20.75, samples=26 write: IOPS=120, BW=481KiB/s (492kB/s)(6296KiB/13092msec) clat (usec): min=472, max=44130, avg=4509.25, stdev=4364.64 lat (usec): min=473, max=44130, avg=4512.16, stdev=4363.71 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 562], 10.00th=[ 594], 20.00th=[ 734], | 30.00th=[ 971], 40.00th=[ 2073], 50.00th=[ 3556], 60.00th=[ 4686], | 70.00th=[ 5866], 80.00th=[ 7635], 90.00th=[10290], 95.00th=[12256], | 99.00th=[19268], 99.50th=[22414], 99.90th=[38011], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 375, max= 760, per=25.37%, avg=480.08, stdev=95.04, samples=26 iops : min= 93, max= 190, avg=119.92, stdev=23.81, samples=26 lat (usec) : 100=5.31%, 250=2.99%, 500=0.33%, 750=10.77%, 1000=4.92% lat (msec) : 2=5.57%, 4=28.48%, 10=35.09%, 20=6.02%, 50=0.46% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.24%, sys=24.09%, ctx=5695, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11148: Tue Apr 8 14:09:16 2025 read: IOPS=112, BW=452KiB/s (462kB/s)(5940KiB/13155msec) clat (usec): min=54, max=45012, avg=3901.15, stdev=3466.63 lat (usec): min=54, max=45013, avg=3901.84, stdev=3466.63 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 77], 10.00th=[ 91], 20.00th=[ 1926], | 30.00th=[ 2409], 40.00th=[ 2802], 50.00th=[ 3195], 60.00th=[ 3752], | 70.00th=[ 4686], 80.00th=[ 6063], 90.00th=[ 7701], 95.00th=[ 8979], | 99.00th=[15795], 99.50th=[20317], 99.90th=[33162], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 272, max= 552, per=24.85%, avg=449.08, stdev=73.66, samples=26 iops : min= 68, max= 138, avg=112.15, stdev=18.47, samples=26 write: IOPS=120, BW=483KiB/s (494kB/s)(6348KiB/13155msec) clat (usec): min=455, max=156132, avg=4577.79, stdev=5747.06 lat (usec): min=455, max=156133, avg=4578.74, stdev=5747.34 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 562], 10.00th=[ 603], 20.00th=[ 717], | 30.00th=[ 979], 40.00th=[ 2114], 50.00th=[ 3556], 60.00th=[ 4621], | 70.00th=[ 5866], 80.00th=[ 7504], 90.00th=[ 10159], 95.00th=[ 12780], | 99.00th=[ 18744], 99.50th=[ 21627], 99.90th=[ 43779], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 336, max= 720, per=25.16%, avg=476.12, stdev=84.11, samples=26 iops : min= 84, max= 180, avg=118.92, stdev=21.04, samples=26 lat (usec) : 100=5.86%, 250=2.80%, 500=0.33%, 750=10.87%, 1000=4.82% lat (msec) : 2=5.73%, 4=27.70%, 10=34.96%, 20=6.35%, 50=0.55% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=23.75%, ctx=5743, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11149: Tue Apr 8 14:09:16 2025 read: IOPS=113, BW=456KiB/s (467kB/s)(6028KiB/13228msec) clat (usec): min=55, max=53819, avg=3854.47, stdev=3544.25 lat (usec): min=55, max=53819, avg=3855.34, stdev=3544.20 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 74], 10.00th=[ 90], 20.00th=[ 1958], | 30.00th=[ 2442], 40.00th=[ 2802], 50.00th=[ 3195], 60.00th=[ 3752], | 70.00th=[ 4686], 80.00th=[ 5866], 90.00th=[ 7570], 95.00th=[ 8979], | 99.00th=[12780], 99.50th=[19530], 99.90th=[44303], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 192, max= 616, per=24.97%, avg=451.23, stdev=96.08, samples=26 iops : min= 48, max= 154, avg=112.69, stdev=24.08, samples=26 write: IOPS=118, BW=473KiB/s (485kB/s)(6260KiB/13228msec) clat (usec): min=442, max=213254, avg=4675.86, stdev=6799.68 lat (usec): min=442, max=213255, avg=4676.90, stdev=6799.94 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 562], 10.00th=[ 603], 20.00th=[ 725], | 30.00th=[ 1004], 40.00th=[ 2245], 50.00th=[ 3720], 60.00th=[ 4686], | 70.00th=[ 5997], 80.00th=[ 7701], 90.00th=[ 9765], 95.00th=[ 12518], | 99.00th=[ 18482], 99.50th=[ 20579], 99.90th=[ 48497], 99.95th=[212861], | 99.99th=[212861] bw ( KiB/s): min= 256, max= 632, per=24.70%, avg=467.23, stdev=86.10, samples=26 iops : min= 64, max= 158, avg=116.69, stdev=21.58, samples=26 lat (usec) : 100=5.76%, 250=2.96%, 500=0.46%, 750=10.64%, 1000=4.46% lat (msec) : 2=5.37%, 4=28.45%, 10=35.22%, 20=6.18%, 50=0.42% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.21%, sys=23.64%, ctx=5730, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11150: Tue Apr 8 14:09:16 2025 read: IOPS=113, BW=456KiB/s (467kB/s)(6052KiB/13284msec) clat (usec): min=52, max=58305, avg=3764.52, stdev=3490.20 lat (usec): min=52, max=58305, avg=3765.17, stdev=3490.20 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 74], 10.00th=[ 87], 20.00th=[ 163], | 30.00th=[ 2343], 40.00th=[ 2704], 50.00th=[ 3064], 60.00th=[ 3654], | 70.00th=[ 4555], 80.00th=[ 6063], 90.00th=[ 7701], 95.00th=[ 9110], | 99.00th=[13173], 99.50th=[18482], 99.90th=[37487], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 152, max= 616, per=24.73%, avg=446.88, stdev=97.33, samples=26 iops : min= 38, max= 154, avg=111.62, stdev=24.35, samples=26 write: IOPS=117, BW=469KiB/s (481kB/s)(6236KiB/13284msec) clat (usec): min=484, max=228589, avg=4803.93, stdev=7202.95 lat (usec): min=485, max=228590, avg=4804.72, stdev=7202.95 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 553], 10.00th=[ 611], 20.00th=[ 750], | 30.00th=[ 1037], 40.00th=[ 2409], 50.00th=[ 3687], 60.00th=[ 4686], | 70.00th=[ 6128], 80.00th=[ 7832], 90.00th=[ 10683], 95.00th=[ 13173], | 99.00th=[ 18744], 99.50th=[ 24249], 99.90th=[ 33817], 99.95th=[227541], | 99.99th=[227541] bw ( KiB/s): min= 192, max= 608, per=24.41%, avg=461.92, stdev=90.71, samples=26 iops : min= 48, max= 152, avg=115.38, stdev=22.70, samples=26 lat (usec) : 100=6.64%, 250=3.52%, 500=0.36%, 750=10.09%, 1000=4.62% lat (msec) : 2=5.60%, 4=28.03%, 10=33.37%, 20=7.16%, 50=0.55% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.18%, sys=23.11%, ctx=5630, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1808KiB/s (1851kB/s), 452KiB/s-458KiB/s (462kB/s-469kB/s), io=23.4MiB (24.6MB), run=13092-13284msec WRITE: bw=1893KiB/s (1938kB/s), 469KiB/s-483KiB/s (481kB/s-494kB/s), io=24.6MiB (25.7MB), run=13092-13284msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11151: Tue Apr 8 14:09:37 2025 read: IOPS=43, BW=175KiB/s (179kB/s)(5992KiB/34314msec) clat (usec): min=1684, max=159102, avg=3589.30, stdev=5189.98 lat (usec): min=1684, max=159104, avg=3590.03, stdev=5190.03 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1860], 10.00th=[ 1909], 20.00th=[ 1991], | 30.00th=[ 2089], 40.00th=[ 2180], 50.00th=[ 2278], 60.00th=[ 2442], | 70.00th=[ 2835], 80.00th=[ 3785], 90.00th=[ 7242], 95.00th=[ 9896], | 99.00th=[ 16057], 99.50th=[ 20055], 99.90th=[ 56361], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 56, max= 296, per=24.96%, avg=174.25, stdev=69.30, samples=68 iops : min= 14, max= 74, avg=43.49, stdev=17.34, samples=68 write: IOPS=45, BW=183KiB/s (188kB/s)(6296KiB/34314msec) clat (usec): min=8428, max=60480, avg=18326.39, stdev=6971.62 lat (usec): min=8429, max=60481, avg=18327.28, stdev=6971.66 clat percentiles (usec): | 1.00th=[ 9765], 5.00th=[10945], 10.00th=[11600], 20.00th=[12649], | 30.00th=[13698], 40.00th=[15401], 50.00th=[16909], 60.00th=[18220], | 70.00th=[19530], 80.00th=[22414], 90.00th=[27657], 95.00th=[32375], | 99.00th=[41157], 99.50th=[44827], 99.90th=[59507], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min= 64, max= 255, per=25.08%, avg=183.07, stdev=56.37, samples=68 iops : min= 16, max= 63, avg=45.69, stdev=14.10, samples=68 lat (msec) : 2=9.83%, 4=29.59%, 10=8.07%, 20=38.12%, 50=14.16% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.10%, sys=5.81%, ctx=3517, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11152: Tue Apr 8 14:09:37 2025 read: IOPS=43, BW=173KiB/s (177kB/s)(5940KiB/34400msec) clat (usec): min=1589, max=199989, avg=3653.25, stdev=6165.46 lat (usec): min=1590, max=199990, avg=3654.01, stdev=6165.47 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1844], 10.00th=[ 1909], 20.00th=[ 1991], | 30.00th=[ 2073], 40.00th=[ 2147], 50.00th=[ 2245], 60.00th=[ 2409], | 70.00th=[ 2737], 80.00th=[ 3720], 90.00th=[ 7635], 95.00th=[ 10421], | 99.00th=[ 17433], 99.50th=[ 25560], 99.90th=[ 53216], 99.95th=[200279], | 99.99th=[200279] bw ( KiB/s): min= 16, max= 336, per=24.76%, avg=172.81, stdev=74.66, samples=68 iops : min= 4, max= 84, avg=43.19, stdev=18.68, samples=68 write: IOPS=46, BW=185KiB/s (189kB/s)(6348KiB/34400msec) clat (usec): min=8633, max=64555, avg=18197.23, stdev=6946.81 lat (usec): min=8634, max=64556, avg=18198.13, stdev=6946.80 clat percentiles (usec): | 1.00th=[ 9503], 5.00th=[10683], 10.00th=[11338], 20.00th=[12518], | 30.00th=[13698], 40.00th=[15401], 50.00th=[16909], 60.00th=[17957], | 70.00th=[19530], 80.00th=[22152], 90.00th=[28443], 95.00th=[32375], | 99.00th=[40633], 99.50th=[43254], 99.90th=[57410], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 64, max= 256, per=25.19%, avg=183.87, stdev=57.39, samples=68 iops : min= 16, max= 64, avg=45.96, stdev=14.36, samples=68 lat (msec) : 2=9.77%, 4=29.65%, 10=7.36%, 20=39.10%, 50=13.93% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.08%, sys=5.81%, ctx=3475, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11153: Tue Apr 8 14:09:37 2025 read: IOPS=44, BW=176KiB/s (181kB/s)(6028KiB/34165msec) clat (usec): min=1606, max=53283, avg=3677.01, stdev=3469.52 lat (usec): min=1606, max=53284, avg=3677.79, stdev=3469.52 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1860], 10.00th=[ 1926], 20.00th=[ 2024], | 30.00th=[ 2114], 40.00th=[ 2212], 50.00th=[ 2278], 60.00th=[ 2507], | 70.00th=[ 2900], 80.00th=[ 4424], 90.00th=[ 8029], 95.00th=[10421], | 99.00th=[17171], 99.50th=[20579], 99.90th=[34341], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 56, max= 360, per=25.28%, avg=176.49, stdev=67.49, samples=68 iops : min= 14, max= 90, avg=44.04, stdev=16.86, samples=68 write: IOPS=45, BW=183KiB/s (188kB/s)(6260KiB/34165msec) clat (msec): min=8, max=212, avg=18.23, stdev= 8.48 lat (msec): min=8, max=212, avg=18.23, stdev= 8.48 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 20], 80.00th=[ 22], 90.00th=[ 28], 95.00th=[ 33], | 99.00th=[ 41], 99.50th=[ 46], 99.90th=[ 65], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 64, max= 272, per=25.08%, avg=183.07, stdev=59.48, samples=68 iops : min= 16, max= 68, avg=45.69, stdev=14.89, samples=68 lat (msec) : 2=8.33%, 4=30.08%, 10=8.69%, 20=39.06%, 50=13.64% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.12%, sys=5.84%, ctx=3459, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11154: Tue Apr 8 14:09:37 2025 read: IOPS=44, BW=178KiB/s (182kB/s)(6052KiB/33987msec) clat (usec): min=1543, max=53554, avg=3577.95, stdev=3468.13 lat (usec): min=1544, max=53554, avg=3578.69, stdev=3468.16 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1844], 10.00th=[ 1909], 20.00th=[ 2008], | 30.00th=[ 2089], 40.00th=[ 2180], 50.00th=[ 2278], 60.00th=[ 2409], | 70.00th=[ 2769], 80.00th=[ 4015], 90.00th=[ 7439], 95.00th=[10028], | 99.00th=[16909], 99.50th=[20579], 99.90th=[36439], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 56, max= 360, per=25.44%, avg=177.57, stdev=71.27, samples=67 iops : min= 14, max= 90, avg=44.31, stdev=17.86, samples=67 write: IOPS=45, BW=183KiB/s (188kB/s)(6236KiB/33987msec) clat (msec): min=8, max=225, avg=18.27, stdev= 8.60 lat (msec): min=8, max=225, avg=18.27, stdev= 8.60 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 19], | 70.00th=[ 20], 80.00th=[ 22], 90.00th=[ 29], 95.00th=[ 33], | 99.00th=[ 40], 99.50th=[ 46], 99.90th=[ 58], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 56, max= 256, per=25.11%, avg=183.30, stdev=57.87, samples=67 iops : min= 14, max= 64, avg=45.75, stdev=14.49, samples=67 lat (msec) : 2=9.28%, 4=30.05%, 10=8.59%, 20=38.05%, 50=13.80% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.10%, sys=5.93%, ctx=3468, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=698KiB/s (715kB/s), 173KiB/s-178KiB/s (177kB/s-182kB/s), io=23.4MiB (24.6MB), run=33987-34400msec WRITE: bw=731KiB/s (748kB/s), 183KiB/s-185KiB/s (188kB/s-189kB/s), io=24.6MiB (25.7MB), run=33987-34400msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11198: Tue Apr 8 14:09:41 2025 read: IOPS=134, BW=2151KiB/s (2203kB/s)(5920KiB/2752msec) clat (usec): min=59, max=27619, avg=3367.67, stdev=3188.54 lat (usec): min=59, max=27620, avg=3368.30, stdev=3188.54 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 76], 10.00th=[ 87], 20.00th=[ 120], | 30.00th=[ 2245], 40.00th=[ 2737], 50.00th=[ 3130], 60.00th=[ 3621], | 70.00th=[ 4293], 80.00th=[ 5276], 90.00th=[ 7046], 95.00th=[ 8160], | 99.00th=[10945], 99.50th=[27132], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 1856, max= 2368, per=24.05%, avg=2092.00, stdev=215.00, samples=5 iops : min= 116, max= 148, avg=130.60, stdev=13.59, samples=5 write: IOPS=144, BW=2314KiB/s (2369kB/s)(6368KiB/2752msec) clat (usec): min=671, max=34244, avg=3733.84, stdev=3816.01 lat (usec): min=671, max=34245, avg=3734.75, stdev=3816.01 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 783], 10.00th=[ 873], 20.00th=[ 1057], | 30.00th=[ 1205], 40.00th=[ 1418], 50.00th=[ 2245], 60.00th=[ 3195], | 70.00th=[ 4424], 80.00th=[ 6783], 90.00th=[ 8586], 95.00th=[10814], | 99.00th=[14353], 99.50th=[25822], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 1984, max= 2816, per=25.23%, avg=2309.40, stdev=345.28, samples=5 iops : min= 124, max= 176, avg=144.20, stdev=21.73, samples=5 lat (usec) : 100=6.38%, 250=7.68%, 750=1.17%, 1000=7.55% lat (msec) : 2=16.41%, 4=27.73%, 10=28.91%, 20=3.52%, 50=0.65% cpu : usr=0.04%, sys=29.30%, ctx=1447, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11199: Tue Apr 8 14:09:41 2025 read: IOPS=134, BW=2146KiB/s (2198kB/s)(5744KiB/2676msec) clat (usec): min=61, max=17481, avg=3209.94, stdev=2739.63 lat (usec): min=61, max=17482, avg=3211.83, stdev=2740.28 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 73], 10.00th=[ 85], 20.00th=[ 116], | 30.00th=[ 269], 40.00th=[ 2671], 50.00th=[ 3097], 60.00th=[ 3556], | 70.00th=[ 4178], 80.00th=[ 5342], 90.00th=[ 6849], 95.00th=[ 8029], | 99.00th=[10814], 99.50th=[15533], 99.90th=[17433], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 1664, max= 2464, per=24.79%, avg=2156.80, stdev=320.48, samples=5 iops : min= 104, max= 154, avg=134.80, stdev=20.03, samples=5 write: IOPS=152, BW=2445KiB/s (2504kB/s)(6544KiB/2676msec) clat (usec): min=665, max=34881, avg=3547.06, stdev=3787.69 lat (usec): min=666, max=34882, avg=3548.05, stdev=3787.72 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 783], 10.00th=[ 848], 20.00th=[ 1004], | 30.00th=[ 1156], 40.00th=[ 1319], 50.00th=[ 1975], 60.00th=[ 3294], | 70.00th=[ 4178], 80.00th=[ 5342], 90.00th=[ 8291], 95.00th=[10421], | 99.00th=[15926], 99.50th=[23725], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 1888, max= 2784, per=26.64%, avg=2438.40, stdev=348.64, samples=5 iops : min= 118, max= 174, avg=152.40, stdev=21.79, samples=5 lat (usec) : 100=5.86%, 250=8.07%, 500=0.39%, 750=1.56%, 1000=8.85% lat (msec) : 2=16.54%, 4=26.69%, 10=28.12%, 20=3.52%, 50=0.39% cpu : usr=0.30%, sys=29.20%, ctx=1456, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11200: Tue Apr 8 14:09:41 2025 read: IOPS=142, BW=2272KiB/s (2327kB/s)(6256KiB/2753msec) clat (usec): min=59, max=52177, avg=3407.63, stdev=3981.34 lat (usec): min=60, max=52177, avg=3408.26, stdev=3981.35 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 72], 10.00th=[ 84], 20.00th=[ 114], | 30.00th=[ 174], 40.00th=[ 2704], 50.00th=[ 3130], 60.00th=[ 3523], | 70.00th=[ 4424], 80.00th=[ 5604], 90.00th=[ 6980], 95.00th=[ 8029], | 99.00th=[16188], 99.50th=[25822], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 1405, max= 2656, per=25.52%, avg=2220.20, stdev=491.06, samples=5 iops : min= 87, max= 166, avg=138.60, stdev=31.03, samples=5 write: IOPS=136, BW=2191KiB/s (2244kB/s)(6032KiB/2753msec) clat (usec): min=657, max=27527, avg=3603.48, stdev=4078.17 lat (usec): min=657, max=27529, avg=3604.42, stdev=4078.18 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 791], 10.00th=[ 865], 20.00th=[ 1020], | 30.00th=[ 1123], 40.00th=[ 1303], 50.00th=[ 1893], 60.00th=[ 3163], | 70.00th=[ 4228], 80.00th=[ 5342], 90.00th=[ 7898], 95.00th=[10159], | 99.00th=[25035], 99.50th=[26608], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 1724, max= 2528, per=23.35%, avg=2136.80, stdev=292.42, samples=5 iops : min= 107, max= 158, avg=133.40, stdev=18.54, samples=5 lat (usec) : 100=7.81%, 250=8.07%, 500=0.39%, 750=1.56%, 1000=8.33% lat (msec) : 2=15.62%, 4=25.39%, 10=29.43%, 20=2.08%, 50=1.17% lat (msec) : 100=0.13% cpu : usr=0.18%, sys=28.12%, ctx=1382, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11201: Tue Apr 8 14:09:41 2025 read: IOPS=146, BW=2338KiB/s (2394kB/s)(6032KiB/2580msec) clat (usec): min=61, max=18874, avg=3158.80, stdev=2952.87 lat (usec): min=61, max=18874, avg=3159.40, stdev=2952.86 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 72], 10.00th=[ 80], 20.00th=[ 103], | 30.00th=[ 130], 40.00th=[ 2606], 50.00th=[ 3097], 60.00th=[ 3556], | 70.00th=[ 4178], 80.00th=[ 5276], 90.00th=[ 6718], 95.00th=[ 7963], | 99.00th=[13960], 99.50th=[17695], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 1728, max= 2746, per=26.60%, avg=2314.60, stdev=427.20, samples=5 iops : min= 108, max= 171, avg=144.40, stdev=26.58, samples=5 write: IOPS=151, BW=2425KiB/s (2483kB/s)(6256KiB/2580msec) clat (usec): min=687, max=29784, avg=3366.10, stdev=3825.96 lat (usec): min=688, max=29785, avg=3367.13, stdev=3825.96 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 791], 10.00th=[ 840], 20.00th=[ 971], | 30.00th=[ 1106], 40.00th=[ 1319], 50.00th=[ 1565], 60.00th=[ 2606], | 70.00th=[ 3720], 80.00th=[ 5407], 90.00th=[ 8094], 95.00th=[10028], | 99.00th=[18220], 99.50th=[29230], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 1856, max= 3200, per=26.62%, avg=2436.40, stdev=510.94, samples=5 iops : min= 116, max= 200, avg=152.00, stdev=32.04, samples=5 lat (usec) : 100=9.51%, 250=6.64%, 500=0.13%, 750=1.17%, 1000=10.81% lat (msec) : 2=17.32%, 4=24.61%, 10=25.91%, 20=3.52%, 50=0.39% cpu : usr=0.23%, sys=28.93%, ctx=1389, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8700KiB/s (8909kB/s), 2146KiB/s-2338KiB/s (2198kB/s-2394kB/s), io=23.4MiB (24.5MB), run=2580-2753msec WRITE: bw=9154KiB/s (9373kB/s), 2191KiB/s-2445KiB/s (2244kB/s-2504kB/s), io=24.6MiB (25.8MB), run=2580-2753msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11194: Tue Apr 8 14:09:47 2025 read: IOPS=41, BW=663KiB/s (679kB/s)(5920KiB/8933msec) clat (usec): min=1910, max=33135, avg=3947.39, stdev=3679.18 lat (usec): min=1911, max=33135, avg=3948.18, stdev=3679.17 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2089], 10.00th=[ 2147], 20.00th=[ 2278], | 30.00th=[ 2343], 40.00th=[ 2474], 50.00th=[ 2606], 60.00th=[ 2868], | 70.00th=[ 3392], 80.00th=[ 4178], 90.00th=[ 7439], 95.00th=[10552], | 99.00th=[20841], 99.50th=[22938], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 320, max= 1184, per=24.84%, avg=656.88, stdev=227.80, samples=17 iops : min= 20, max= 74, avg=41.00, stdev=14.32, samples=17 write: IOPS=44, BW=713KiB/s (730kB/s)(6368KiB/8933msec) clat (usec): min=10296, max=45211, avg=18600.77, stdev=5876.96 lat (usec): min=10297, max=45212, avg=18602.14, stdev=5877.00 clat percentiles (usec): | 1.00th=[10945], 5.00th=[11994], 10.00th=[12649], 20.00th=[13698], | 30.00th=[14746], 40.00th=[16188], 50.00th=[17957], 60.00th=[19006], | 70.00th=[20055], 80.00th=[21627], 90.00th=[26870], 95.00th=[30016], | 99.00th=[39060], 99.50th=[43254], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 384, max= 928, per=25.30%, avg=703.94, stdev=184.26, samples=17 iops : min= 24, max= 58, avg=43.94, stdev=11.60, samples=17 lat (msec) : 2=1.04%, 4=36.59%, 10=7.68%, 20=37.89%, 50=16.80% cpu : usr=0.09%, sys=6.62%, ctx=951, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11195: Tue Apr 8 14:09:47 2025 read: IOPS=39, BW=635KiB/s (651kB/s)(5744KiB/9039msec) clat (usec): min=1791, max=31788, avg=3904.43, stdev=3323.62 lat (usec): min=1792, max=31789, avg=3905.12, stdev=3323.63 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2008], 10.00th=[ 2073], 20.00th=[ 2180], | 30.00th=[ 2311], 40.00th=[ 2442], 50.00th=[ 2573], 60.00th=[ 2900], | 70.00th=[ 3523], 80.00th=[ 4359], 90.00th=[ 8160], 95.00th=[10552], | 99.00th=[15664], 99.50th=[19006], 99.90th=[31851], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 288, max= 1056, per=23.85%, avg=630.53, stdev=226.26, samples=17 iops : min= 18, max= 66, avg=39.35, stdev=14.19, samples=17 write: IOPS=45, BW=724KiB/s (741kB/s)(6544KiB/9039msec) clat (usec): min=9863, max=58650, avg=18607.85, stdev=6262.99 lat (usec): min=9864, max=58651, avg=18609.21, stdev=6263.05 clat percentiles (usec): | 1.00th=[10814], 5.00th=[11863], 10.00th=[12387], 20.00th=[13435], | 30.00th=[14615], 40.00th=[15926], 50.00th=[17957], 60.00th=[19006], | 70.00th=[20055], 80.00th=[21890], 90.00th=[25560], 95.00th=[31327], | 99.00th=[41157], 99.50th=[43254], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 415, max= 896, per=25.71%, avg=715.24, stdev=178.97, samples=17 iops : min= 25, max= 56, avg=44.65, stdev=11.29, samples=17 lat (msec) : 2=2.08%, 4=33.72%, 10=8.20%, 20=39.06%, 50=16.80% lat (msec) : 100=0.13% cpu : usr=0.04%, sys=6.28%, ctx=916, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11196: Tue Apr 8 14:09:47 2025 read: IOPS=45, BW=724KiB/s (742kB/s)(6256KiB/8636msec) clat (usec): min=1805, max=21982, avg=3717.08, stdev=3339.25 lat (usec): min=1805, max=21982, avg=3717.89, stdev=3339.23 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2040], 10.00th=[ 2089], 20.00th=[ 2180], | 30.00th=[ 2278], 40.00th=[ 2376], 50.00th=[ 2474], 60.00th=[ 2638], | 70.00th=[ 2999], 80.00th=[ 3752], 90.00th=[ 6718], 95.00th=[11731], | 99.00th=[20841], 99.50th=[21890], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 320, max= 1280, per=27.69%, avg=732.12, stdev=322.22, samples=17 iops : min= 20, max= 80, avg=45.71, stdev=20.16, samples=17 write: IOPS=43, BW=698KiB/s (715kB/s)(6032KiB/8636msec) clat (usec): min=10077, max=58066, avg=18933.08, stdev=6645.51 lat (usec): min=10079, max=58067, avg=18934.36, stdev=6645.55 clat percentiles (usec): | 1.00th=[11207], 5.00th=[11863], 10.00th=[12256], 20.00th=[13435], | 30.00th=[14746], 40.00th=[16581], 50.00th=[17957], 60.00th=[19006], | 70.00th=[20317], 80.00th=[22152], 90.00th=[26870], 95.00th=[31851], | 99.00th=[41681], 99.50th=[51643], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 416, max= 928, per=24.96%, avg=694.47, stdev=179.20, samples=17 iops : min= 26, max= 58, avg=43.35, stdev=11.22, samples=17 lat (msec) : 2=1.69%, 4=39.71%, 10=6.25%, 20=36.07%, 50=16.02% lat (msec) : 100=0.26% cpu : usr=0.12%, sys=6.71%, ctx=925, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11197: Tue Apr 8 14:09:47 2025 read: IOPS=41, BW=666KiB/s (682kB/s)(6032KiB/9057msec) clat (usec): min=1842, max=54284, avg=4070.32, stdev=4263.30 lat (usec): min=1842, max=54284, avg=4071.03, stdev=4263.31 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2040], 10.00th=[ 2114], 20.00th=[ 2245], | 30.00th=[ 2343], 40.00th=[ 2442], 50.00th=[ 2606], 60.00th=[ 2802], | 70.00th=[ 3425], 80.00th=[ 4424], 90.00th=[ 7898], 95.00th=[12518], | 99.00th=[20055], 99.50th=[24249], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 224, max= 1120, per=24.94%, avg=659.39, stdev=254.46, samples=18 iops : min= 14, max= 70, avg=41.11, stdev=15.97, samples=18 write: IOPS=43, BW=691KiB/s (707kB/s)(6256KiB/9057msec) clat (usec): min=9609, max=49425, avg=19196.88, stdev=6392.23 lat (usec): min=9609, max=49427, avg=19198.22, stdev=6392.26 clat percentiles (usec): | 1.00th=[10945], 5.00th=[11994], 10.00th=[12649], 20.00th=[13829], | 30.00th=[15270], 40.00th=[17171], 50.00th=[18482], 60.00th=[19530], | 70.00th=[20579], 80.00th=[22152], 90.00th=[27395], 95.00th=[32375], | 99.00th=[44303], 99.50th=[49021], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 352, max= 896, per=24.79%, avg=689.56, stdev=183.25, samples=18 iops : min= 22, max= 56, avg=43.00, stdev=11.39, samples=18 lat (msec) : 2=1.82%, 4=35.55%, 10=8.07%, 20=36.07%, 50=18.36% lat (msec) : 100=0.13% cpu : usr=0.12%, sys=6.29%, ctx=928, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2645KiB/s (2708kB/s), 635KiB/s-724KiB/s (651kB/s-742kB/s), io=23.4MiB (24.5MB), run=8636-9057msec WRITE: bw=2782KiB/s (2849kB/s), 691KiB/s-724KiB/s (707kB/s-741kB/s), io=24.6MiB (25.8MB), run=8636-9057msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11217: Tue Apr 8 14:09:49 2025 read: IOPS=21, BW=21.7MiB/s (22.7MB/s)(7168KiB/323msec) clat (usec): min=394, max=39351, avg=21137.34, stdev=19436.28 lat (usec): min=395, max=39353, avg=21138.51, stdev=19436.13 clat percentiles (usec): | 1.00th=[ 396], 5.00th=[ 396], 10.00th=[ 396], 20.00th=[ 506], | 30.00th=[ 537], 40.00th=[ 537], 50.00th=[32900], 60.00th=[35914], | 70.00th=[35914], 80.00th=[38536], 90.00th=[39584], 95.00th=[39584], | 99.00th=[39584], 99.50th=[39584], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] write: IOPS=15, BW=15.5MiB/s (16.2MB/s)(5120KiB/323msec) clat (usec): min=17291, max=42407, avg=32696.41, stdev=10165.25 lat (usec): min=17310, max=42465, avg=32731.70, stdev=10170.05 clat percentiles (usec): | 1.00th=[17171], 5.00th=[17171], 10.00th=[17171], 20.00th=[17171], | 30.00th=[28443], 40.00th=[28443], 50.00th=[34866], 60.00th=[34866], | 70.00th=[40633], 80.00th=[40633], 90.00th=[42206], 95.00th=[42206], | 99.00th=[42206], 99.50th=[42206], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] lat (usec) : 500=8.33%, 750=16.67% lat (msec) : 20=8.33%, 50=66.67% cpu : usr=0.00%, sys=58.07%, ctx=658, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11219: Tue Apr 8 14:09:49 2025 read: IOPS=2, BW=2972KiB/s (3044kB/s)(2048KiB/689msec) clat (usec): min=39806, max=43325, avg=41565.70, stdev=2488.22 lat (usec): min=39807, max=43325, avg=41566.65, stdev=2487.73 clat percentiles (usec): | 1.00th=[39584], 5.00th=[39584], 10.00th=[39584], 20.00th=[39584], | 30.00th=[39584], 40.00th=[39584], 50.00th=[39584], 60.00th=[43254], | 70.00th=[43254], 80.00th=[43254], 90.00th=[43254], 95.00th=[43254], | 99.00th=[43254], 99.50th=[43254], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 3637, max= 3637, per=12.77%, avg=3637.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=14, BW=14.5MiB/s (15.2MB/s)(10.0MiB/689msec) clat (usec): min=31995, max=88374, avg=56930.58, stdev=22534.30 lat (usec): min=32060, max=88436, avg=56986.06, stdev=22537.70 clat percentiles (usec): | 1.00th=[32113], 5.00th=[32113], 10.00th=[32113], 20.00th=[32113], | 30.00th=[36439], 40.00th=[42730], 50.00th=[43779], 60.00th=[59507], | 70.00th=[76022], 80.00th=[76022], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min=10912, max=10912, per=27.36%, avg=10912.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=41.67% cpu : usr=0.00%, sys=34.45%, ctx=294, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11221: Tue Apr 8 14:09:49 2025 read: IOPS=12, BW=12.4MiB/s (13.0MB/s)(6144KiB/483msec) clat (usec): min=317, max=47027, avg=19282.48, stdev=21288.33 lat (usec): min=318, max=47028, avg=19283.29, stdev=21288.27 clat percentiles (usec): | 1.00th=[ 318], 5.00th=[ 318], 10.00th=[ 318], 20.00th=[ 343], | 30.00th=[ 343], 40.00th=[ 424], 50.00th=[ 424], 60.00th=[33162], | 70.00th=[34341], 80.00th=[34341], 90.00th=[46924], 95.00th=[46924], | 99.00th=[46924], 99.50th=[46924], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] write: IOPS=12, BW=12.4MiB/s (13.0MB/s)(6144KiB/483msec) clat (msec): min=15, max=156, avg=58.14, stdev=49.93 lat (msec): min=15, max=156, avg=58.19, stdev=49.94 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 43], 50.00th=[ 43], 60.00th=[ 47], | 70.00th=[ 57], 80.00th=[ 57], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] lat (usec) : 500=25.00% lat (msec) : 20=8.33%, 50=50.00%, 100=8.33%, 250=8.33% cpu : usr=0.00%, sys=30.50%, ctx=186, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11222: Tue Apr 8 14:09:49 2025 read: IOPS=6, BW=7121KiB/s (7292kB/s)(5120KiB/719msec) clat (usec): min=354, max=33942, avg=26481.11, stdev=14636.71 lat (usec): min=355, max=33943, avg=26481.98, stdev=14636.84 clat percentiles (usec): | 1.00th=[ 355], 5.00th=[ 355], 10.00th=[ 355], 20.00th=[ 355], | 30.00th=[31327], 40.00th=[31327], 50.00th=[33162], 60.00th=[33162], | 70.00th=[33424], 80.00th=[33424], 90.00th=[33817], 95.00th=[33817], | 99.00th=[33817], 99.50th=[33817], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 4096, max= 4096, per=14.38%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9969KiB/s (10.2MB/s)(7168KiB/719msec) clat (msec): min=45, max=130, avg=80.10, stdev=27.94 lat (msec): min=45, max=131, avg=80.24, stdev=28.10 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 61], | 30.00th=[ 67], 40.00th=[ 67], 50.00th=[ 77], 60.00th=[ 84], | 70.00th=[ 84], 80.00th=[ 99], 90.00th=[ 131], 95.00th=[ 131], | 99.00th=[ 131], 99.50th=[ 131], 99.90th=[ 131], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 8192, max= 8192, per=20.54%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=41.67%, 100=41.67%, 250=8.33% cpu : usr=0.28%, sys=29.11%, ctx=788, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=27.8MiB/s (29.2MB/s), 2972KiB/s-21.7MiB/s (3044kB/s-22.7MB/s), io=20.0MiB (20.0MB), run=323-719msec WRITE: bw=38.9MiB/s (40.8MB/s), 9969KiB/s-15.5MiB/s (10.2MB/s-16.2MB/s), io=28.0MiB (29.4MB), run=323-719msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11215: Tue Apr 8 14:09:49 2025 read: IOPS=9, BW=9752KiB/s (9986kB/s)(7168KiB/735msec) clat (msec): min=24, max=103, avg=47.40, stdev=27.75 lat (msec): min=24, max=103, avg=47.40, stdev=27.75 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 30], 40.00th=[ 30], 50.00th=[ 40], 60.00th=[ 53], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 104], 95.00th=[ 104], | 99.00th=[ 104], 99.50th=[ 104], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min=10240, max=10240, per=37.05%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6966KiB/s (7133kB/s)(5120KiB/735msec) clat (usec): min=45005, max=97901, avg=78406.88, stdev=19856.49 lat (usec): min=45020, max=97945, avg=78452.74, stdev=19870.41 clat percentiles (usec): | 1.00th=[44827], 5.00th=[44827], 10.00th=[44827], 20.00th=[44827], | 30.00th=[80217], 40.00th=[80217], 50.00th=[84411], 60.00th=[84411], | 70.00th=[85459], 80.00th=[85459], 90.00th=[98042], 95.00th=[98042], | 99.00th=[98042], 99.50th=[98042], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 4096, max= 4096, per=10.59%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=50.00%, 250=8.33% cpu : usr=0.00%, sys=12.53%, ctx=38, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11216: Tue Apr 8 14:09:49 2025 read: IOPS=2, BW=2999KiB/s (3071kB/s)(2048KiB/683msec) clat (usec): min=15210, max=51090, avg=33150.58, stdev=25370.59 lat (usec): min=15213, max=51090, avg=33152.09, stdev=25369.44 clat percentiles (usec): | 1.00th=[15270], 5.00th=[15270], 10.00th=[15270], 20.00th=[15270], | 30.00th=[15270], 40.00th=[15270], 50.00th=[15270], 60.00th=[51119], | 70.00th=[51119], 80.00th=[51119], 90.00th=[51119], 95.00th=[51119], | 99.00th=[51119], 99.50th=[51119], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 4096, max= 4096, per=14.82%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=14, BW=14.6MiB/s (15.4MB/s)(10.0MiB/683msec) clat (msec): min=28, max=138, avg=60.41, stdev=33.83 lat (msec): min=28, max=138, avg=60.47, stdev=33.82 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 32], | 30.00th=[ 37], 40.00th=[ 40], 50.00th=[ 44], 60.00th=[ 61], | 70.00th=[ 78], 80.00th=[ 78], 90.00th=[ 89], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min=12288, max=12288, per=31.76%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=41.67%, 250=8.33% cpu : usr=0.29%, sys=12.61%, ctx=39, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11218: Tue Apr 8 14:09:49 2025 read: IOPS=8, BW=8740KiB/s (8949kB/s)(6144KiB/703msec) clat (usec): min=30580, max=75378, avg=54733.03, stdev=17857.92 lat (usec): min=30581, max=75379, avg=54733.89, stdev=17858.11 clat percentiles (usec): | 1.00th=[30540], 5.00th=[30540], 10.00th=[30540], 20.00th=[40633], | 30.00th=[40633], 40.00th=[52691], 50.00th=[52691], 60.00th=[54789], | 70.00th=[73925], 80.00th=[73925], 90.00th=[74974], 95.00th=[74974], | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 8192, max= 8192, per=29.64%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8740KiB/s (8949kB/s)(6144KiB/703msec) clat (msec): min=37, max=114, avg=61.08, stdev=33.80 lat (msec): min=37, max=114, avg=61.13, stdev=33.81 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 40], | 30.00th=[ 40], 40.00th=[ 42], 50.00th=[ 42], 60.00th=[ 42], | 70.00th=[ 93], 80.00th=[ 93], 90.00th=[ 115], 95.00th=[ 115], | 99.00th=[ 115], 99.50th=[ 115], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 8192, max= 8192, per=21.17%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=41.67%, 250=8.33% cpu : usr=0.14%, sys=12.11%, ctx=50, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11220: Tue Apr 8 14:09:49 2025 read: IOPS=6, BW=6910KiB/s (7075kB/s)(5120KiB/741msec) clat (usec): min=22384, max=71319, avg=42560.56, stdev=24297.42 lat (usec): min=22385, max=71320, avg=42561.29, stdev=24297.37 clat percentiles (usec): | 1.00th=[22414], 5.00th=[22414], 10.00th=[22414], 20.00th=[22414], | 30.00th=[25822], 40.00th=[25822], 50.00th=[26608], 60.00th=[26608], | 70.00th=[66847], 80.00th=[66847], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 6131, max= 6131, per=22.18%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9673KiB/s (9906kB/s)(7168KiB/741msec) clat (msec): min=33, max=122, avg=72.67, stdev=28.18 lat (msec): min=33, max=122, avg=72.72, stdev=28.18 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 54], | 30.00th=[ 61], 40.00th=[ 61], 50.00th=[ 73], 60.00th=[ 84], | 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 123], 95.00th=[ 123], | 99.00th=[ 123], 99.50th=[ 123], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 8175, max= 8175, per=21.13%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=58.33%, 250=8.33% cpu : usr=0.00%, sys=9.86%, ctx=45, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=26.0MiB/s (28.3MB/s), 2999KiB/s-9752KiB/s (3071kB/s-9986kB/s), io=20.0MiB (20.0MB), run=683-741msec WRITE: bw=37.8MiB/s (39.6MB/s), 6966KiB/s-14.6MiB/s (7133kB/s-15.4MB/s), io=28.0MiB (29.4MB), run=683-741msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11229: Tue Apr 8 14:09:50 2025 read: IOPS=6, BW=24.8MiB/s (25.0MB/s)(12.0MiB/484msec) clat (msec): min=133, max=173, avg=155.27, stdev=20.48 lat (msec): min=133, max=173, avg=155.27, stdev=20.48 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 134], 10.00th=[ 134], 20.00th=[ 134], | 30.00th=[ 134], 40.00th=[ 159], 50.00th=[ 159], 60.00th=[ 159], | 70.00th=[ 174], 80.00th=[ 174], 90.00th=[ 174], 95.00th=[ 174], | 99.00th=[ 174], 99.50th=[ 174], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=12.84%, ctx=44, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11230: Tue Apr 8 14:09:50 2025 read: IOPS=1, BW=6628KiB/s (6787kB/s)(4096KiB/618msec) clat (nsec): min=60685k, max=60685k, avg=60685257.00, stdev= 0.00 lat (nsec): min=60686k, max=60686k, avg=60686252.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[60556], 5.00th=[60556], 10.00th=[60556], 20.00th=[60556], | 30.00th=[60556], 40.00th=[60556], 50.00th=[60556], 60.00th=[60556], | 70.00th=[60556], 80.00th=[60556], 90.00th=[60556], 95.00th=[60556], | 99.00th=[60556], 99.50th=[60556], 99.90th=[60556], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min= 8126, max= 8126, per=30.65%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=3, BW=12.9MiB/s (13.6MB/s)(8192KiB/618msec) clat (msec): min=216, max=330, avg=273.80, stdev=80.69 lat (msec): min=217, max=331, avg=274.07, stdev=80.68 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 218], 10.00th=[ 218], 20.00th=[ 218], | 30.00th=[ 218], 40.00th=[ 218], 50.00th=[ 218], 60.00th=[ 330], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 8126, max= 8126, per=16.47%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=9.89%, ctx=149, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11232: Tue Apr 8 14:09:50 2025 write: IOPS=4, BW=18.1MiB/s (18.9MB/s)(12.0MiB/664msec) clat (msec): min=106, max=326, avg=216.77, stdev=110.11 lat (msec): min=106, max=326, avg=217.07, stdev=110.11 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 218], 50.00th=[ 218], 60.00th=[ 218], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 8192, max= 8192, per=16.60%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=13.57%, ctx=50, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11234: Tue Apr 8 14:09:50 2025 write: IOPS=6, BW=24.6MiB/s (25.8MB/s)(12.0MiB/488msec) clat (msec): min=80, max=252, avg=143.95, stdev=94.32 lat (msec): min=81, max=252, avg=144.25, stdev=94.33 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 82], 10.00th=[ 82], 20.00th=[ 82], | 30.00th=[ 82], 40.00th=[ 99], 50.00th=[ 99], 60.00th=[ 99], | 70.00th=[ 253], 80.00th=[ 253], 90.00th=[ 253], 95.00th=[ 253], | 99.00th=[ 253], 99.50th=[ 253], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 253] lat (msec) : 100=66.67%, 500=33.33% cpu : usr=0.21%, sys=19.71%, ctx=81, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.9MiB/s (27.1MB/s), 6628KiB/s-24.8MiB/s (6787kB/s-25.0MB/s), io=16.0MiB (16.8MB), run=484-618msec WRITE: bw=48.2MiB/s (50.5MB/s), 12.9MiB/s-24.6MiB/s (13.6MB/s-25.8MB/s), io=32.0MiB (33.6MB), run=488-664msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11231: Tue Apr 8 14:09:51 2025 read: IOPS=7, BW=31.6MiB/s (33.1MB/s)(12.0MiB/380msec) clat (msec): min=81, max=180, avg=123.78, stdev=51.00 lat (msec): min=81, max=180, avg=123.78, stdev=51.00 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 82], 10.00th=[ 82], 20.00th=[ 82], | 30.00th=[ 82], 40.00th=[ 110], 50.00th=[ 110], 60.00th=[ 110], | 70.00th=[ 182], 80.00th=[ 182], 90.00th=[ 182], 95.00th=[ 182], | 99.00th=[ 182], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.26%, sys=31.93%, ctx=1014, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11233: Tue Apr 8 14:09:51 2025 read: IOPS=1, BW=5185KiB/s (5309kB/s)(4096KiB/790msec) clat (nsec): min=84874k, max=84874k, avg=84873651.00, stdev= 0.00 lat (nsec): min=84875k, max=84875k, avg=84874744.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[84411], 5.00th=[84411], 10.00th=[84411], 20.00th=[84411], | 30.00th=[84411], 40.00th=[84411], 50.00th=[84411], 60.00th=[84411], | 70.00th=[84411], 80.00th=[84411], 90.00th=[84411], 95.00th=[84411], | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 8192, max= 8192, per=39.50%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.1MiB/s (10.6MB/s)(8192KiB/790msec) clat (msec): min=283, max=393, avg=338.49, stdev=77.16 lat (msec): min=284, max=393, avg=338.88, stdev=77.34 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 284], 10.00th=[ 284], 20.00th=[ 284], | 30.00th=[ 284], 40.00th=[ 284], 50.00th=[ 284], 60.00th=[ 393], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 8192, max= 8192, per=20.73%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.00%, sys=20.15%, ctx=595, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11235: Tue Apr 8 14:09:51 2025 write: IOPS=3, BW=14.5MiB/s (15.2MB/s)(12.0MiB/829msec) clat (msec): min=122, max=363, avg=270.12, stdev=129.07 lat (msec): min=123, max=364, avg=270.34, stdev=129.14 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 124], 10.00th=[ 124], 20.00th=[ 124], | 30.00th=[ 124], 40.00th=[ 326], 50.00th=[ 326], 60.00th=[ 326], | 70.00th=[ 363], 80.00th=[ 363], 90.00th=[ 363], 95.00th=[ 363], | 99.00th=[ 363], 99.50th=[ 363], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 8192, max= 8192, per=20.73%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=33.94%, ctx=40, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11236: Tue Apr 8 14:09:51 2025 write: IOPS=5, BW=20.5MiB/s (21.5MB/s)(12.0MiB/584msec) clat (msec): min=106, max=263, avg=190.04, stdev=78.99 lat (msec): min=106, max=263, avg=190.18, stdev=79.00 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 201], 50.00th=[ 201], 60.00th=[ 201], | 70.00th=[ 264], 80.00th=[ 264], 90.00th=[ 264], 95.00th=[ 264], | 99.00th=[ 264], 99.50th=[ 264], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min=16384, max=16384, per=41.45%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=49.91%, ctx=72, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=20.3MiB/s (21.2MB/s), 5185KiB/s-31.6MiB/s (5309kB/s-33.1MB/s), io=16.0MiB (16.8MB), run=380-790msec WRITE: bw=38.6MiB/s (40.5MB/s), 10.1MiB/s-20.5MiB/s (10.6MB/s-21.5MB/s), io=32.0MiB (33.6MB), run=584-829msec