== sanity test 398b: DIO and buffer IO race ============== 13:14:34 (1722791674) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.03032 s, 48.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23055: Sun Aug 4 13:14:49 2024 read: IOPS=123, BW=492KiB/s (504kB/s)(5992KiB/12174msec) clat (usec): min=46, max=174477, avg=3817.70, stdev=5281.61 lat (usec): min=46, max=174477, avg=3819.05, stdev=5282.31 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 62], 10.00th=[ 73], 20.00th=[ 1975], | 30.00th=[ 2343], 40.00th=[ 2638], 50.00th=[ 3032], 60.00th=[ 3654], | 70.00th=[ 4686], 80.00th=[ 5932], 90.00th=[ 7504], 95.00th=[ 8848], | 99.00th=[ 11338], 99.50th=[ 13960], 99.90th=[ 39584], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 272, max= 686, per=25.24%, avg=488.96, stdev=104.17, samples=24 iops : min= 68, max= 171, avg=122.17, stdev=25.98, samples=24 write: IOPS=129, BW=517KiB/s (530kB/s)(6296KiB/12174msec) clat (usec): min=433, max=50237, avg=4020.55, stdev=3980.30 lat (usec): min=433, max=50238, avg=4021.30, stdev=3980.32 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 627], | 30.00th=[ 914], 40.00th=[ 1942], 50.00th=[ 3261], 60.00th=[ 4047], | 70.00th=[ 5276], 80.00th=[ 6718], 90.00th=[ 8848], 95.00th=[10814], | 99.00th=[16712], 99.50th=[19792], 99.90th=[35390], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 336, max= 680, per=25.30%, avg=513.13, stdev=101.28, samples=24 iops : min= 84, max= 170, avg=128.21, stdev=25.33, samples=24 lat (usec) : 50=0.26%, 100=7.13%, 250=0.98%, 500=0.72%, 750=12.60% lat (usec) : 1000=3.19% lat (msec) : 2=5.86%, 4=30.86%, 10=33.79%, 20=4.30%, 50=0.26% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.16%, sys=25.11%, ctx=6480, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23056: Sun Aug 4 13:14:49 2024 read: IOPS=121, BW=487KiB/s (499kB/s)(5940KiB/12185msec) clat (usec): min=46, max=36192, avg=3823.50, stdev=3087.50 lat (usec): min=46, max=36193, avg=3824.34, stdev=3087.63 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 61], 10.00th=[ 74], 20.00th=[ 1975], | 30.00th=[ 2376], 40.00th=[ 2704], 50.00th=[ 3163], 60.00th=[ 3851], | 70.00th=[ 4817], 80.00th=[ 5932], 90.00th=[ 7439], 95.00th=[ 8848], | 99.00th=[12780], 99.50th=[16057], 99.90th=[30540], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 320, max= 608, per=25.07%, avg=485.54, stdev=88.32, samples=24 iops : min= 80, max= 152, avg=121.29, stdev=22.16, samples=24 write: IOPS=130, BW=521KiB/s (533kB/s)(6348KiB/12185msec) clat (usec): min=431, max=172920, avg=4020.33, stdev=5684.65 lat (usec): min=432, max=172921, avg=4021.21, stdev=5684.67 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 652], | 30.00th=[ 930], 40.00th=[ 2180], 50.00th=[ 3163], 60.00th=[ 3949], | 70.00th=[ 4948], 80.00th=[ 6325], 90.00th=[ 8848], 95.00th=[ 10814], | 99.00th=[ 17171], 99.50th=[ 20317], 99.90th=[ 42206], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 360, max= 768, per=25.49%, avg=516.88, stdev=102.64, samples=24 iops : min= 90, max= 192, avg=129.12, stdev=25.75, samples=24 lat (usec) : 50=0.26%, 100=6.45%, 250=1.01%, 500=1.11%, 750=12.53% lat (usec) : 1000=2.77% lat (msec) : 2=6.05%, 4=31.15%, 10=33.66%, 20=4.56%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=25.43%, ctx=6432, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23057: Sun Aug 4 13:14:49 2024 read: IOPS=121, BW=486KiB/s (498kB/s)(6028KiB/12391msec) clat (usec): min=47, max=30531, avg=3722.54, stdev=3009.02 lat (usec): min=47, max=30532, avg=3723.41, stdev=3009.04 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 61], 10.00th=[ 73], 20.00th=[ 1909], | 30.00th=[ 2343], 40.00th=[ 2638], 50.00th=[ 3097], 60.00th=[ 3752], | 70.00th=[ 4621], 80.00th=[ 5800], 90.00th=[ 7504], 95.00th=[ 8717], | 99.00th=[12649], 99.50th=[16712], 99.90th=[25035], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 160, max= 688, per=24.38%, avg=472.25, stdev=104.06, samples=24 iops : min= 40, max= 172, avg=118.00, stdev=26.06, samples=24 write: IOPS=126, BW=505KiB/s (517kB/s)(6260KiB/12391msec) clat (usec): min=474, max=213362, avg=4253.46, stdev=6615.92 lat (usec): min=475, max=213364, avg=4254.28, stdev=6615.95 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 644], | 30.00th=[ 930], 40.00th=[ 2278], 50.00th=[ 3425], 60.00th=[ 4228], | 70.00th=[ 5407], 80.00th=[ 6849], 90.00th=[ 8979], 95.00th=[ 10945], | 99.00th=[ 16581], 99.50th=[ 21365], 99.90th=[ 41681], 99.95th=[212861], | 99.99th=[212861] bw ( KiB/s): min= 192, max= 704, per=24.11%, avg=488.92, stdev=108.20, samples=24 iops : min= 48, max= 176, avg=122.17, stdev=27.10, samples=24 lat (usec) : 50=0.23%, 100=7.45%, 250=1.04%, 500=0.78%, 750=12.08% lat (usec) : 1000=3.09% lat (msec) : 2=5.50%, 4=29.43%, 10=35.77%, 20=4.07%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=24.62%, ctx=6568, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23058: Sun Aug 4 13:14:49 2024 read: IOPS=126, BW=506KiB/s (518kB/s)(6052KiB/11972msec) clat (usec): min=48, max=45261, avg=3660.18, stdev=3065.24 lat (usec): min=48, max=45262, avg=3662.15, stdev=3065.30 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 63], 10.00th=[ 75], 20.00th=[ 1942], | 30.00th=[ 2311], 40.00th=[ 2638], 50.00th=[ 3064], 60.00th=[ 3654], | 70.00th=[ 4555], 80.00th=[ 5604], 90.00th=[ 7046], 95.00th=[ 8586], | 99.00th=[12518], 99.50th=[16450], 99.90th=[33162], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 304, max= 656, per=25.99%, avg=503.48, stdev=97.43, samples=23 iops : min= 76, max= 164, avg=125.78, stdev=24.40, samples=23 write: IOPS=130, BW=521KiB/s (533kB/s)(6236KiB/11972msec) clat (usec): min=432, max=168181, avg=4044.93, stdev=5619.45 lat (usec): min=433, max=168181, avg=4045.68, stdev=5619.42 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 652], | 30.00th=[ 898], 40.00th=[ 1778], 50.00th=[ 3228], 60.00th=[ 4047], | 70.00th=[ 5080], 80.00th=[ 6587], 90.00th=[ 8848], 95.00th=[ 11076], | 99.00th=[ 16581], 99.50th=[ 18744], 99.90th=[ 41157], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 367, max= 736, per=25.67%, avg=520.52, stdev=85.84, samples=23 iops : min= 91, max= 184, avg=130.04, stdev=21.53, samples=23 lat (usec) : 50=0.42%, 100=6.90%, 250=1.01%, 500=0.94%, 750=12.21% lat (usec) : 1000=3.29% lat (msec) : 2=6.74%, 4=30.50%, 10=33.11%, 20=4.49%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=25.27%, ctx=6274, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1938KiB/s (1984kB/s), 486KiB/s-506KiB/s (498kB/s-518kB/s), io=23.4MiB (24.6MB), run=11972-12391msec WRITE: bw=2029KiB/s (2078kB/s), 505KiB/s-521KiB/s (517kB/s-533kB/s), io=24.6MiB (25.7MB), run=11972-12391msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23059: Sun Aug 4 13:15:16 2024 read: IOPS=38, BW=153KiB/s (157kB/s)(5992KiB/39079msec) clat (usec): min=1618, max=176260, avg=3367.70, stdev=5293.33 lat (usec): min=1618, max=176262, avg=3368.43, stdev=5293.35 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1860], 10.00th=[ 1926], 20.00th=[ 2024], | 30.00th=[ 2114], 40.00th=[ 2180], 50.00th=[ 2311], 60.00th=[ 2442], | 70.00th=[ 2704], 80.00th=[ 3228], 90.00th=[ 6063], 95.00th=[ 8291], | 99.00th=[ 14615], 99.50th=[ 22152], 99.90th=[ 39584], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 40, max= 272, per=25.08%, avg=153.24, stdev=57.03, samples=78 iops : min= 10, max= 68, avg=38.24, stdev=14.28, samples=78 write: IOPS=40, BW=161KiB/s (165kB/s)(6296KiB/39079msec) clat (usec): min=11528, max=54084, avg=21549.62, stdev=6575.50 lat (usec): min=11529, max=54085, avg=21550.52, stdev=6575.50 clat percentiles (usec): | 1.00th=[12649], 5.00th=[13435], 10.00th=[14091], 20.00th=[15270], | 30.00th=[17433], 40.00th=[19530], 50.00th=[21103], 60.00th=[22414], | 70.00th=[23725], 80.00th=[25560], 90.00th=[30016], 95.00th=[34866], | 99.00th=[42730], 99.50th=[44827], 99.90th=[52167], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 48, max= 208, per=25.13%, avg=160.83, stdev=36.93, samples=78 iops : min= 12, max= 52, avg=40.14, stdev= 9.24, samples=78 lat (msec) : 2=8.04%, 4=32.94%, 10=6.22%, 20=23.27%, 50=29.39% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.09%, sys=4.94%, ctx=3446, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23060: Sun Aug 4 13:15:16 2024 read: IOPS=37, BW=151KiB/s (155kB/s)(5940KiB/39258msec) clat (usec): min=1589, max=170859, avg=3430.45, stdev=5307.16 lat (usec): min=1590, max=170861, avg=3431.14, stdev=5307.21 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1876], 10.00th=[ 1942], 20.00th=[ 2008], | 30.00th=[ 2114], 40.00th=[ 2212], 50.00th=[ 2311], 60.00th=[ 2442], | 70.00th=[ 2671], 80.00th=[ 3326], 90.00th=[ 6128], 95.00th=[ 8717], | 99.00th=[ 15795], 99.50th=[ 21103], 99.90th=[ 41681], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 16, max= 296, per=24.77%, avg=151.32, stdev=58.78, samples=78 iops : min= 4, max= 74, avg=37.78, stdev=14.72, samples=78 write: IOPS=40, BW=162KiB/s (166kB/s)(6348KiB/39258msec) clat (usec): min=11237, max=52691, avg=21454.70, stdev=6428.43 lat (usec): min=11238, max=52692, avg=21455.55, stdev=6428.46 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13435], 10.00th=[14091], 20.00th=[15533], | 30.00th=[17433], 40.00th=[19530], 50.00th=[20841], 60.00th=[22152], | 70.00th=[23462], 80.00th=[25035], 90.00th=[29492], 95.00th=[34341], | 99.00th=[43254], 99.50th=[45876], 99.90th=[52167], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 56, max= 208, per=25.20%, avg=161.27, stdev=36.77, samples=78 iops : min= 14, max= 52, avg=40.27, stdev= 9.19, samples=78 lat (msec) : 2=8.27%, 4=32.10%, 10=5.99%, 20=24.48%, 50=29.04% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.10%, sys=4.72%, ctx=3425, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23061: Sun Aug 4 13:15:16 2024 read: IOPS=38, BW=154KiB/s (158kB/s)(6028KiB/39109msec) clat (usec): min=1616, max=44787, avg=3401.48, stdev=3372.93 lat (usec): min=1617, max=44787, avg=3402.25, stdev=3372.93 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1876], 10.00th=[ 1942], 20.00th=[ 2040], | 30.00th=[ 2147], 40.00th=[ 2245], 50.00th=[ 2343], 60.00th=[ 2474], | 70.00th=[ 2704], 80.00th=[ 3294], 90.00th=[ 6587], 95.00th=[ 8979], | 99.00th=[15795], 99.50th=[25035], 99.90th=[38536], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 40, max= 312, per=25.20%, avg=153.97, stdev=55.47, samples=78 iops : min= 10, max= 78, avg=38.44, stdev=13.89, samples=78 write: IOPS=40, BW=160KiB/s (164kB/s)(6260KiB/39109msec) clat (msec): min=10, max=210, avg=21.64, stdev= 8.22 lat (msec): min=10, max=210, avg=21.64, stdev= 8.22 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 26], 90.00th=[ 31], 95.00th=[ 35], | 99.00th=[ 44], 99.50th=[ 47], 99.90th=[ 54], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 40, max= 224, per=24.97%, avg=159.82, stdev=39.96, samples=78 iops : min= 10, max= 56, avg=39.90, stdev=10.01, samples=78 lat (msec) : 2=7.55%, 4=33.92%, 10=5.66%, 20=23.57%, 50=29.13% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.08%, sys=4.85%, ctx=3449, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23062: Sun Aug 4 13:15:16 2024 read: IOPS=38, BW=154KiB/s (158kB/s)(6052KiB/39178msec) clat (usec): min=1651, max=41592, avg=3336.63, stdev=3245.18 lat (usec): min=1651, max=41593, avg=3337.40, stdev=3245.18 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1893], 10.00th=[ 1958], 20.00th=[ 2057], | 30.00th=[ 2147], 40.00th=[ 2212], 50.00th=[ 2343], 60.00th=[ 2474], | 70.00th=[ 2704], 80.00th=[ 3228], 90.00th=[ 5997], 95.00th=[ 8979], | 99.00th=[15533], 99.50th=[22938], 99.90th=[40109], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 48, max= 280, per=25.31%, avg=154.62, stdev=56.26, samples=78 iops : min= 12, max= 70, avg=38.62, stdev=14.08, samples=78 write: IOPS=39, BW=159KiB/s (163kB/s)(6236KiB/39178msec) clat (msec): min=11, max=179, avg=21.81, stdev= 7.53 lat (msec): min=11, max=179, avg=21.82, stdev= 7.53 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 26], 90.00th=[ 31], 95.00th=[ 35], | 99.00th=[ 43], 99.50th=[ 46], 99.90th=[ 54], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 56, max= 200, per=24.83%, avg=158.92, stdev=36.11, samples=78 iops : min= 14, max= 50, avg=39.69, stdev= 9.03, samples=78 lat (msec) : 2=7.36%, 4=34.24%, 10=5.73%, 20=22.40%, 50=30.14% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.09%, sys=4.78%, ctx=3418, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=612KiB/s (626kB/s), 151KiB/s-154KiB/s (155kB/s-158kB/s), io=23.4MiB (24.6MB), run=39079-39258msec WRITE: bw=640KiB/s (656kB/s), 159KiB/s-162KiB/s (163kB/s-166kB/s), io=24.6MiB (25.7MB), run=39079-39258msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23104: Sun Aug 4 13:15:22 2024 read: IOPS=81, BW=1311KiB/s (1343kB/s)(5920KiB/4515msec) clat (usec): min=52, max=23562, avg=8802.85, stdev=5444.02 lat (usec): min=53, max=23563, avg=8803.78, stdev=5444.04 clat percentiles (usec): | 1.00th=[ 54], 5.00th=[ 63], 10.00th=[ 69], 20.00th=[ 118], | 30.00th=[ 7898], 40.00th=[ 8848], 50.00th=[ 9765], 60.00th=[10552], | 70.00th=[11863], 80.00th=[12911], 90.00th=[14746], 95.00th=[16188], | 99.00th=[21103], 99.50th=[22414], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 800, max= 1472, per=23.98%, avg=1272.00, stdev=204.36, samples=8 iops : min= 50, max= 92, avg=79.50, stdev=12.77, samples=8 write: IOPS=88, BW=1410KiB/s (1444kB/s)(6368KiB/4515msec) clat (usec): min=644, max=36598, avg=3069.29, stdev=3480.37 lat (usec): min=644, max=36600, avg=3070.26, stdev=3480.43 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 734], 10.00th=[ 775], 20.00th=[ 824], | 30.00th=[ 906], 40.00th=[ 1045], 50.00th=[ 1696], 60.00th=[ 2737], | 70.00th=[ 3458], 80.00th=[ 5145], 90.00th=[ 6980], 95.00th=[ 8979], | 99.00th=[17171], 99.50th=[22152], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 864, max= 2112, per=24.94%, avg=1392.00, stdev=436.76, samples=8 iops : min= 54, max= 132, avg=87.00, stdev=27.30, samples=8 lat (usec) : 100=9.24%, 250=1.43%, 500=0.13%, 750=3.52%, 1000=16.15% lat (msec) : 2=6.90%, 4=12.63%, 10=25.91%, 20=23.05%, 50=1.04% cpu : usr=0.07%, sys=21.91%, ctx=1993, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23105: Sun Aug 4 13:15:22 2024 read: IOPS=81, BW=1302KiB/s (1334kB/s)(5744KiB/4410msec) clat (usec): min=52, max=31025, avg=8996.46, stdev=5622.32 lat (usec): min=53, max=31026, avg=8997.07, stdev=5622.35 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 98], | 30.00th=[ 8225], 40.00th=[ 9110], 50.00th=[10028], 60.00th=[10814], | 70.00th=[11863], 80.00th=[12649], 90.00th=[14484], 95.00th=[16188], | 99.00th=[24773], 99.50th=[30016], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 800, max= 1564, per=23.90%, avg=1267.50, stdev=227.42, samples=8 iops : min= 50, max= 97, avg=79.12, stdev=14.08, samples=8 write: IOPS=92, BW=1484KiB/s (1520kB/s)(6544KiB/4410msec) clat (usec): min=627, max=22565, avg=2844.16, stdev=2961.63 lat (usec): min=628, max=22566, avg=2845.12, stdev=2961.64 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 725], 10.00th=[ 758], 20.00th=[ 807], | 30.00th=[ 873], 40.00th=[ 1045], 50.00th=[ 1565], 60.00th=[ 2540], | 70.00th=[ 3228], 80.00th=[ 4555], 90.00th=[ 6325], 95.00th=[ 8586], | 99.00th=[12387], 99.50th=[20055], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 1024, max= 1920, per=25.58%, avg=1427.50, stdev=328.64, samples=8 iops : min= 64, max= 120, avg=89.12, stdev=20.48, samples=8 lat (usec) : 100=9.37%, 250=0.78%, 750=4.82%, 1000=15.62% lat (msec) : 2=7.81%, 4=12.63%, 10=24.09%, 20=23.57%, 50=1.30% cpu : usr=0.14%, sys=22.36%, ctx=1960, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23106: Sun Aug 4 13:15:22 2024 read: IOPS=88, BW=1419KiB/s (1453kB/s)(6256KiB/4409msec) clat (usec): min=49, max=34149, avg=8283.19, stdev=5807.83 lat (usec): min=50, max=34149, avg=8283.86, stdev=5807.91 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 80], | 30.00th=[ 7373], 40.00th=[ 8717], 50.00th=[ 9634], 60.00th=[10552], | 70.00th=[11338], 80.00th=[12387], 90.00th=[14222], 95.00th=[16581], | 99.00th=[22414], 99.50th=[23725], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 768, max= 1792, per=25.57%, avg=1356.00, stdev=352.60, samples=8 iops : min= 48, max= 112, avg=84.75, stdev=22.04, samples=8 write: IOPS=85, BW=1368KiB/s (1401kB/s)(6032KiB/4409msec) clat (usec): min=608, max=26705, avg=3030.02, stdev=3595.31 lat (usec): min=609, max=26705, avg=3031.45, stdev=3595.10 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 709], 10.00th=[ 742], 20.00th=[ 799], | 30.00th=[ 840], 40.00th=[ 947], 50.00th=[ 1516], 60.00th=[ 2573], | 70.00th=[ 3294], 80.00th=[ 4555], 90.00th=[ 6718], 95.00th=[ 8979], | 99.00th=[21103], 99.50th=[24511], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 832, max= 1888, per=23.87%, avg=1332.00, stdev=340.35, samples=8 iops : min= 52, max= 118, avg=83.25, stdev=21.27, samples=8 lat (usec) : 50=0.13%, 100=12.11%, 250=1.69%, 750=5.86%, 1000=15.23% lat (msec) : 2=5.21%, 4=11.20%, 10=22.79%, 20=24.22%, 50=1.56% cpu : usr=0.18%, sys=21.53%, ctx=1903, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23107: Sun Aug 4 13:15:22 2024 read: IOPS=87, BW=1402KiB/s (1436kB/s)(6032KiB/4301msec) clat (usec): min=51, max=26479, avg=8312.60, stdev=5481.34 lat (usec): min=51, max=26480, avg=8313.21, stdev=5481.36 clat percentiles (usec): | 1.00th=[ 54], 5.00th=[ 61], 10.00th=[ 66], 20.00th=[ 81], | 30.00th=[ 7767], 40.00th=[ 8979], 50.00th=[ 9503], 60.00th=[10421], | 70.00th=[11338], 80.00th=[12387], 90.00th=[13698], 95.00th=[14877], | 99.00th=[21365], 99.50th=[26346], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 928, max= 1760, per=25.19%, avg=1336.00, stdev=241.14, samples=8 iops : min= 58, max= 110, avg=83.50, stdev=15.07, samples=8 write: IOPS=90, BW=1455KiB/s (1489kB/s)(6256KiB/4301msec) clat (usec): min=612, max=25799, avg=2739.61, stdev=3188.61 lat (usec): min=613, max=25800, avg=2740.66, stdev=3188.58 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 717], 10.00th=[ 750], 20.00th=[ 799], | 30.00th=[ 840], 40.00th=[ 898], 50.00th=[ 1090], 60.00th=[ 2278], | 70.00th=[ 3032], 80.00th=[ 4555], 90.00th=[ 6521], 95.00th=[ 8094], | 99.00th=[17957], 99.50th=[22938], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 1120, max= 1856, per=25.30%, avg=1412.00, stdev=285.93, samples=8 iops : min= 70, max= 116, avg=88.25, stdev=17.87, samples=8 lat (usec) : 100=11.33%, 250=1.17%, 500=0.13%, 750=4.95%, 1000=18.62% lat (msec) : 2=6.64%, 4=9.64%, 10=23.96%, 20=22.66%, 50=0.91% cpu : usr=0.23%, sys=22.02%, ctx=1838, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5305KiB/s (5432kB/s), 1302KiB/s-1419KiB/s (1334kB/s-1453kB/s), io=23.4MiB (24.5MB), run=4301-4515msec WRITE: bw=5581KiB/s (5715kB/s), 1368KiB/s-1484KiB/s (1401kB/s-1520kB/s), io=24.6MiB (25.8MB), run=4301-4515msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23109: Sun Aug 4 13:15:27 2024 read: IOPS=37, BW=599KiB/s (613kB/s)(5920KiB/9888msec) clat (usec): min=1872, max=29324, avg=4072.76, stdev=3995.64 lat (usec): min=1872, max=29325, avg=4073.51, stdev=3995.64 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2040], 10.00th=[ 2114], 20.00th=[ 2212], | 30.00th=[ 2343], 40.00th=[ 2442], 50.00th=[ 2573], 60.00th=[ 2802], | 70.00th=[ 3163], 80.00th=[ 4146], 90.00th=[ 7767], 95.00th=[12649], | 99.00th=[22938], 99.50th=[28443], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 351, max= 864, per=24.61%, avg=587.63, stdev=163.33, samples=19 iops : min= 21, max= 54, avg=36.63, stdev=10.29, samples=19 write: IOPS=40, BW=644KiB/s (659kB/s)(6368KiB/9888msec) clat (usec): min=11840, max=49044, avg=20851.29, stdev=6176.23 lat (usec): min=11841, max=49046, avg=20852.65, stdev=6176.30 clat percentiles (usec): | 1.00th=[11994], 5.00th=[13042], 10.00th=[13566], 20.00th=[14746], | 30.00th=[16909], 40.00th=[18482], 50.00th=[20317], 60.00th=[21627], | 70.00th=[22938], 80.00th=[25035], 90.00th=[29492], 95.00th=[32375], | 99.00th=[40109], 99.50th=[42206], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 320, max= 832, per=25.33%, avg=636.42, stdev=168.93, samples=19 iops : min= 20, max= 52, avg=39.68, stdev=10.55, samples=19 lat (msec) : 2=1.17%, 4=36.85%, 10=6.25%, 20=27.86%, 50=27.86% cpu : usr=0.02%, sys=5.87%, ctx=881, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23110: Sun Aug 4 13:15:27 2024 read: IOPS=35, BW=573KiB/s (587kB/s)(5744KiB/10027msec) clat (usec): min=1891, max=22686, avg=3673.91, stdev=2943.85 lat (usec): min=1891, max=22687, avg=3674.62, stdev=2943.83 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2057], 10.00th=[ 2147], 20.00th=[ 2278], | 30.00th=[ 2343], 40.00th=[ 2442], 50.00th=[ 2540], 60.00th=[ 2737], | 70.00th=[ 3064], 80.00th=[ 4113], 90.00th=[ 6587], 95.00th=[10421], | 99.00th=[17433], 99.50th=[22676], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 192, max= 1056, per=24.04%, avg=574.10, stdev=224.85, samples=20 iops : min= 12, max= 66, avg=35.75, stdev=14.01, samples=20 write: IOPS=40, BW=653KiB/s (668kB/s)(6544KiB/10027msec) clat (usec): min=11851, max=65246, avg=21110.37, stdev=6568.16 lat (usec): min=11852, max=65249, avg=21111.68, stdev=6568.23 clat percentiles (usec): | 1.00th=[12125], 5.00th=[13173], 10.00th=[13829], 20.00th=[15401], | 30.00th=[16909], 40.00th=[19006], 50.00th=[20317], 60.00th=[21890], | 70.00th=[23200], 80.00th=[25035], 90.00th=[29230], 95.00th=[33817], | 99.00th=[40633], 99.50th=[44827], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 384, max= 832, per=25.90%, avg=650.90, stdev=139.67, samples=20 iops : min= 24, max= 52, avg=40.55, stdev= 8.68, samples=20 lat (msec) : 2=1.30%, 4=35.81%, 10=7.03%, 20=26.82%, 50=28.91% lat (msec) : 100=0.13% cpu : usr=0.05%, sys=5.68%, ctx=886, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23111: Sun Aug 4 13:15:27 2024 read: IOPS=40, BW=649KiB/s (664kB/s)(6256KiB/9643msec) clat (usec): min=1858, max=23515, avg=4081.62, stdev=3512.96 lat (usec): min=1859, max=23516, avg=4082.35, stdev=3512.96 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2073], 10.00th=[ 2114], 20.00th=[ 2245], | 30.00th=[ 2343], 40.00th=[ 2507], 50.00th=[ 2638], 60.00th=[ 2868], | 70.00th=[ 3294], 80.00th=[ 4883], 90.00th=[ 8455], 95.00th=[12649], | 99.00th=[19006], 99.50th=[20579], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 352, max= 1280, per=27.43%, avg=655.05, stdev=258.77, samples=19 iops : min= 22, max= 80, avg=40.89, stdev=16.19, samples=19 write: IOPS=39, BW=626KiB/s (641kB/s)(6032KiB/9643msec) clat (usec): min=11641, max=72533, avg=21301.83, stdev=6476.15 lat (usec): min=11642, max=72535, avg=21303.17, stdev=6476.26 clat percentiles (usec): | 1.00th=[11863], 5.00th=[13042], 10.00th=[13566], 20.00th=[15795], | 30.00th=[17695], 40.00th=[19530], 50.00th=[20579], 60.00th=[22152], | 70.00th=[23462], 80.00th=[25822], 90.00th=[29492], 95.00th=[32900], | 99.00th=[38011], 99.50th=[44303], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 416, max= 832, per=24.73%, avg=621.42, stdev=135.97, samples=19 iops : min= 26, max= 52, avg=38.79, stdev= 8.57, samples=19 lat (msec) : 2=0.39%, 4=38.67%, 10=7.94%, 20=25.39%, 50=27.47% lat (msec) : 100=0.13% cpu : usr=0.05%, sys=5.85%, ctx=881, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23112: Sun Aug 4 13:15:27 2024 read: IOPS=38, BW=609KiB/s (623kB/s)(6032KiB/9910msec) clat (usec): min=1831, max=22829, avg=4084.41, stdev=3456.23 lat (usec): min=1832, max=22829, avg=4085.08, stdev=3456.23 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2073], 10.00th=[ 2147], 20.00th=[ 2278], | 30.00th=[ 2343], 40.00th=[ 2474], 50.00th=[ 2606], 60.00th=[ 2868], | 70.00th=[ 3490], 80.00th=[ 5342], 90.00th=[ 7898], 95.00th=[11469], | 99.00th=[19530], 99.50th=[22414], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 256, max= 832, per=25.10%, avg=599.42, stdev=170.67, samples=19 iops : min= 16, max= 52, avg=37.37, stdev=10.75, samples=19 write: IOPS=39, BW=631KiB/s (646kB/s)(6256KiB/9910msec) clat (usec): min=11095, max=58204, avg=21179.60, stdev=6175.94 lat (usec): min=11097, max=58207, avg=21180.93, stdev=6176.07 clat percentiles (usec): | 1.00th=[12125], 5.00th=[13042], 10.00th=[13829], 20.00th=[15795], | 30.00th=[17957], 40.00th=[19530], 50.00th=[20579], 60.00th=[21627], | 70.00th=[23200], 80.00th=[25035], 90.00th=[28443], 95.00th=[32113], | 99.00th=[42730], 99.50th=[49021], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 352, max= 864, per=24.92%, avg=626.32, stdev=146.14, samples=19 iops : min= 22, max= 54, avg=39.05, stdev= 9.20, samples=19 lat (msec) : 2=0.91%, 4=34.90%, 10=9.90%, 20=24.87%, 50=29.30% lat (msec) : 100=0.13% cpu : usr=0.10%, sys=5.66%, ctx=867, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2389KiB/s (2446kB/s), 573KiB/s-649KiB/s (587kB/s-664kB/s), io=23.4MiB (24.5MB), run=9643-10027msec WRITE: bw=2513KiB/s (2574kB/s), 626KiB/s-653KiB/s (641kB/s-668kB/s), io=24.6MiB (25.8MB), run=9643-10027msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23128: Sun Aug 4 13:15:32 2024 read: IOPS=2, BW=2227KiB/s (2281kB/s)(7168KiB/3218msec) clat (usec): min=484, max=899133, avg=419282.11, stdev=393336.58 lat (usec): min=484, max=899134, avg=419283.48, stdev=393336.03 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 486], 10.00th=[ 486], 20.00th=[ 799], | 30.00th=[ 60031], 40.00th=[ 60031], 50.00th=[530580], 60.00th=[616563], | 70.00th=[616563], 80.00th=[826278], 90.00th=[901776], 95.00th=[901776], | 99.00th=[901776], 99.50th=[901776], 99.90th=[901776], 99.95th=[901776], | 99.99th=[901776] bw ( KiB/s): min= 2048, max= 4096, per=65.58%, avg=3069.75, stdev=1179.82, samples=4 iops : min= 2, max= 4, avg= 2.75, stdev= 0.96, samples=4 write: IOPS=1, BW=1591KiB/s (1629kB/s)(5120KiB/3218msec) clat (usec): min=28773, max=78511, avg=47560.16, stdev=19918.96 lat (usec): min=28864, max=78580, avg=47632.77, stdev=19907.70 clat percentiles (usec): | 1.00th=[28705], 5.00th=[28705], 10.00th=[28705], 20.00th=[28705], | 30.00th=[33817], 40.00th=[33817], 50.00th=[41681], 60.00th=[41681], | 70.00th=[54789], 80.00th=[54789], 90.00th=[78119], 95.00th=[78119], | 99.00th=[78119], 99.50th=[78119], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 2048, max= 6144, per=52.09%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 lat (usec) : 500=8.33%, 1000=8.33% lat (msec) : 50=25.00%, 100=25.00%, 750=16.67%, 1000=16.67% cpu : usr=0.09%, sys=18.81%, ctx=1236, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23130: Sun Aug 4 13:15:32 2024 read: IOPS=0, BW=627KiB/s (642kB/s)(2048KiB/3268msec) clat (msec): min=45, max=789, avg=417.58, stdev=525.61 lat (msec): min=45, max=789, avg=417.58, stdev=525.61 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 46], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 46], 60.00th=[ 793], | 70.00th=[ 793], 80.00th=[ 793], 90.00th=[ 793], 95.00th=[ 793], | 99.00th=[ 793], 99.50th=[ 793], 99.90th=[ 793], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 2048, max= 2048, per=43.75%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3133KiB/s (3209kB/s)(10.0MiB/3268msec) clat (msec): min=14, max=932, avg=239.15, stdev=333.08 lat (msec): min=14, max=932, avg=239.20, stdev=333.07 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 21], | 30.00th=[ 29], 40.00th=[ 29], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 592], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 2048, max= 6144, per=62.51%, avg=4096.00, stdev=1672.18, samples=4 iops : min= 2, max= 6, avg= 4.00, stdev= 1.63, samples=4 lat (msec) : 20=8.33%, 50=33.33%, 100=25.00%, 750=16.67%, 1000=16.67% cpu : usr=0.03%, sys=8.51%, ctx=109, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23132: Sun Aug 4 13:15:32 2024 read: IOPS=1, BW=1893KiB/s (1939kB/s)(6144KiB/3245msec) clat (usec): min=584, max=905980, avg=365807.04, stdev=398149.88 lat (usec): min=586, max=905981, avg=365808.33, stdev=398149.76 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 586], 10.00th=[ 586], 20.00th=[ 3654], | 30.00th=[ 3654], 40.00th=[ 41157], 50.00th=[ 41157], 60.00th=[616563], | 70.00th=[624952], 80.00th=[624952], 90.00th=[910164], 95.00th=[910164], | 99.00th=[910164], 99.50th=[910164], 99.90th=[910164], 99.95th=[910164], | 99.99th=[910164] bw ( KiB/s): min= 2043, max= 4096, per=58.30%, avg=2729.00, stdev=1183.86, samples=3 iops : min= 1, max= 4, avg= 2.33, stdev= 1.53, samples=3 write: IOPS=1, BW=1893KiB/s (1939kB/s)(6144KiB/3245msec) clat (msec): min=21, max=901, avg=173.87, stdev=356.32 lat (msec): min=22, max=901, avg=173.92, stdev=356.31 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 24], | 30.00th=[ 24], 40.00th=[ 29], 50.00th=[ 29], 60.00th=[ 31], | 70.00th=[ 39], 80.00th=[ 39], 90.00th=[ 902], 95.00th=[ 902], | 99.00th=[ 902], 99.50th=[ 902], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 2048, max= 6131, per=52.02%, avg=3409.00, stdev=2357.32, samples=3 iops : min= 2, max= 5, avg= 3.00, stdev= 1.73, samples=3 lat (usec) : 750=8.33% lat (msec) : 4=8.33%, 50=50.00%, 750=16.67%, 1000=16.67% cpu : usr=0.00%, sys=13.69%, ctx=650, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23133: Sun Aug 4 13:15:32 2024 read: IOPS=1, BW=1170KiB/s (1198kB/s)(5120KiB/4375msec) clat (usec): min=310, max=901727, avg=291604.18, stdev=388118.10 lat (usec): min=311, max=901729, avg=291605.21, stdev=388118.46 clat percentiles (usec): | 1.00th=[ 310], 5.00th=[ 310], 10.00th=[ 310], 20.00th=[ 310], | 30.00th=[ 955], 40.00th=[ 955], 50.00th=[103285], 60.00th=[103285], | 70.00th=[450888], 80.00th=[450888], 90.00th=[901776], 95.00th=[901776], | 99.00th=[901776], 99.50th=[901776], 99.90th=[901776], 99.95th=[901776], | 99.99th=[901776] bw ( KiB/s): min= 2048, max= 6131, per=87.36%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=1, BW=1638KiB/s (1678kB/s)(7168KiB/4375msec) clat (msec): min=26, max=1855, avg=412.17, stdev=677.45 lat (msec): min=26, max=1856, avg=412.24, stdev=677.45 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 40], | 30.00th=[ 45], 40.00th=[ 45], 50.00th=[ 117], 60.00th=[ 120], | 70.00th=[ 120], 80.00th=[ 684], 90.00th=[ 1854], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 2048, max= 6144, per=54.66%, avg=3581.75, stdev=1960.03, samples=4 iops : min= 2, max= 6, avg= 3.25, stdev= 1.89, samples=4 lat (usec) : 500=8.33%, 1000=8.33% lat (msec) : 50=25.00%, 250=25.00%, 500=8.33%, 750=8.33%, 1000=8.33% cpu : usr=0.02%, sys=8.96%, ctx=671, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4681KiB/s (4793kB/s), 627KiB/s-2227KiB/s (642kB/s-2281kB/s), io=20.0MiB (20.0MB), run=3218-4375msec WRITE: bw=6554KiB/s (6711kB/s), 1591KiB/s-3133KiB/s (1629kB/s-3209kB/s), io=28.0MiB (29.4MB), run=3218-4375msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23126: Sun Aug 4 13:15:33 2024 read: IOPS=2, BW=2077KiB/s (2127kB/s)(7168KiB/3451msec) clat (msec): min=18, max=923, avg=286.57, stdev=432.44 lat (msec): min=18, max=923, avg=286.58, stdev=432.44 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 28], | 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 37], 60.00th=[ 54], | 70.00th=[ 54], 80.00th=[ 919], 90.00th=[ 927], 95.00th=[ 927], | 99.00th=[ 927], 99.50th=[ 927], 99.90th=[ 927], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 2048, max= 3482, per=52.99%, avg=2406.50, stdev=717.00, samples=4 iops : min= 2, max= 3, avg= 2.25, stdev= 0.50, samples=4 write: IOPS=1, BW=1484KiB/s (1519kB/s)(5120KiB/3451msec) clat (msec): min=60, max=696, avg=279.18, stdev=298.43 lat (msec): min=60, max=696, avg=279.25, stdev=298.44 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 61], 10.00th=[ 61], 20.00th=[ 61], | 30.00th=[ 71], 40.00th=[ 71], 50.00th=[ 71], 60.00th=[ 71], | 70.00th=[ 498], 80.00th=[ 498], 90.00th=[ 701], 95.00th=[ 701], | 99.00th=[ 701], 99.50th=[ 701], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 5224, max= 5224, per=82.18%, avg=5224.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=25.00%, 100=33.33%, 500=8.33%, 750=8.33% lat (msec) : 1000=16.67% cpu : usr=0.00%, sys=3.42%, ctx=208, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23127: Sun Aug 4 13:15:33 2024 read: IOPS=0, BW=454KiB/s (465kB/s)(2048KiB/4510msec) clat (msec): min=78, max=409, avg=243.84, stdev=233.76 lat (msec): min=78, max=409, avg=243.85, stdev=233.76 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 79], 10.00th=[ 79], 20.00th=[ 79], | 30.00th=[ 79], 40.00th=[ 79], 50.00th=[ 79], 60.00th=[ 409], | 70.00th=[ 409], 80.00th=[ 409], 90.00th=[ 409], 95.00th=[ 409], | 99.00th=[ 409], 99.50th=[ 409], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 1735, max= 2048, per=41.65%, avg=1891.50, stdev=221.32, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2271KiB/s (2325kB/s)(10.0MiB/4510msec) clat (msec): min=25, max=984, avg=397.97, stdev=421.37 lat (msec): min=25, max=984, avg=398.05, stdev=421.37 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 48], | 30.00th=[ 57], 40.00th=[ 106], 50.00th=[ 111], 60.00th=[ 114], | 70.00th=[ 802], 80.00th=[ 802], 90.00th=[ 978], 95.00th=[ 986], | 99.00th=[ 986], 99.50th=[ 986], 99.90th=[ 986], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 2043, max= 5206, per=58.29%, avg=3705.50, stdev=1319.10, samples=4 iops : min= 1, max= 5, avg= 3.25, stdev= 1.71, samples=4 lat (msec) : 50=16.67%, 100=16.67%, 250=25.00%, 500=8.33%, 1000=33.33% cpu : usr=0.00%, sys=3.61%, ctx=58, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23129: Sun Aug 4 13:15:33 2024 read: IOPS=1, BW=1849KiB/s (1894kB/s)(6144KiB/3322msec) clat (msec): min=13, max=623, avg=195.60, stdev=257.66 lat (msec): min=13, max=623, avg=195.61, stdev=257.66 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 20], | 30.00th=[ 20], 40.00th=[ 27], 50.00th=[ 27], 60.00th=[ 87], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 7148, max= 7148, per=100.00%, avg=7148.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1849KiB/s (1894kB/s)(6144KiB/3322msec) clat (msec): min=30, max=1833, avg=355.29, stdev=724.83 lat (msec): min=30, max=1833, avg=355.35, stdev=724.83 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 44], 50.00th=[ 44], 60.00th=[ 84], | 70.00th=[ 111], 80.00th=[ 111], 90.00th=[ 1838], 95.00th=[ 1838], | 99.00th=[ 1838], 99.50th=[ 1838], 99.90th=[ 1838], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 1783, max= 4096, per=49.57%, avg=3151.00, stdev=1213.13, samples=3 iops : min= 1, max= 4, avg= 2.67, stdev= 1.53, samples=3 lat (msec) : 20=16.67%, 50=33.33%, 100=16.67%, 250=8.33%, 500=8.33% lat (msec) : 750=8.33% cpu : usr=0.03%, sys=3.58%, ctx=48, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23131: Sun Aug 4 13:15:33 2024 read: IOPS=2, BW=2400KiB/s (2458kB/s)(5120KiB/2133msec) clat (usec): min=13850, max=68628, avg=31163.93, stdev=24711.61 lat (usec): min=13850, max=68630, avg=31165.06, stdev=24712.31 clat percentiles (usec): | 1.00th=[13829], 5.00th=[13829], 10.00th=[13829], 20.00th=[13829], | 30.00th=[14091], 40.00th=[14091], 50.00th=[14746], 60.00th=[14746], | 70.00th=[44303], 80.00th=[44303], 90.00th=[68682], 95.00th=[68682], | 99.00th=[68682], 99.50th=[68682], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3361KiB/s (3441kB/s)(7168KiB/2133msec) clat (msec): min=24, max=902, avg=280.80, stdev=407.67 lat (msec): min=24, max=902, avg=280.86, stdev=407.64 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 34], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 55], 60.00th=[ 65], | 70.00th=[ 65], 80.00th=[ 852], 90.00th=[ 902], 95.00th=[ 902], | 99.00th=[ 902], 99.50th=[ 902], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 2043, max= 8192, per=64.38%, avg=4092.67, stdev=3550.13, samples=3 iops : min= 1, max= 8, avg= 3.33, stdev= 4.04, samples=3 lat (msec) : 20=25.00%, 50=33.33%, 100=25.00%, 1000=16.67% cpu : usr=0.05%, sys=4.88%, ctx=34, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4541KiB/s (4650kB/s), 454KiB/s-2400KiB/s (465kB/s-2458kB/s), io=20.0MiB (20.0MB), run=2133-4510msec WRITE: bw=6357KiB/s (6510kB/s), 1484KiB/s-3361KiB/s (1519kB/s-3441kB/s), io=28.0MiB (29.4MB), run=2133-4510msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23141: Sun Aug 4 13:15:37 2024 read: IOPS=0, BW=3821KiB/s (3913kB/s)(12.0MiB/3216msec) clat (msec): min=166, max=1828, avg=1071.10, stdev=841.11 lat (msec): min=166, max=1828, avg=1071.10, stdev=841.11 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 167], | 30.00th=[ 167], 40.00th=[ 1217], 50.00th=[ 1217], 60.00th=[ 1217], | 70.00th=[ 1838], 80.00th=[ 1838], 90.00th=[ 1838], 95.00th=[ 1838], | 99.00th=[ 1838], 99.50th=[ 1838], 99.90th=[ 1838], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=41.87%, ctx=1136, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23142: Sun Aug 4 13:15:37 2024 read: IOPS=0, BW=1222KiB/s (1251kB/s)(4096KiB/3353msec) clat (nsec): min=148789k, max=148789k, avg=148789082.00, stdev= 0.00 lat (nsec): min=148790k, max=148790k, avg=148789980.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 148], 10.00th=[ 148], 20.00th=[ 148], | 30.00th=[ 148], 40.00th=[ 148], 50.00th=[ 148], 60.00th=[ 148], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2443KiB/s (2502kB/s)(8192KiB/3353msec) clat (msec): min=370, max=2814, avg=1592.23, stdev=1727.88 lat (msec): min=370, max=2814, avg=1592.46, stdev=1727.90 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 372], 10.00th=[ 372], 20.00th=[ 372], | 30.00th=[ 372], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 2802], | 70.00th=[ 2802], 80.00th=[ 2802], 90.00th=[ 2802], 95.00th=[ 2802], | 99.00th=[ 2802], 99.50th=[ 2802], 99.90th=[ 2802], 99.95th=[ 2802], | 99.99th=[ 2802] bw ( KiB/s): min= 8192, max= 8192, per=83.83%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=7.07%, ctx=145, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23143: Sun Aug 4 13:15:37 2024 write: IOPS=1, BW=4975KiB/s (5094kB/s)(12.0MiB/2470msec) clat (msec): min=281, max=1854, avg=816.70, stdev=898.58 lat (msec): min=281, max=1854, avg=816.96, stdev=898.60 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 284], 10.00th=[ 284], 20.00th=[ 284], | 30.00th=[ 284], 40.00th=[ 317], 50.00th=[ 317], 60.00th=[ 317], | 70.00th=[ 1854], 80.00th=[ 1854], 90.00th=[ 1854], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 8192, max= 8192, per=83.83%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=10.73%, ctx=177, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23144: Sun Aug 4 13:15:37 2024 write: IOPS=1, BW=7115KiB/s (7286kB/s)(12.0MiB/1727msec) clat (msec): min=158, max=1353, avg=570.95, stdev=678.30 lat (msec): min=158, max=1354, avg=571.22, stdev=678.28 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 201], 50.00th=[ 201], 60.00th=[ 201], | 70.00th=[ 1351], 80.00th=[ 1351], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 8192, max= 8192, per=83.83%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=13.38%, ctx=45, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4886KiB/s (5004kB/s), 1222KiB/s-3821KiB/s (1251kB/s-3913kB/s), io=16.0MiB (16.8MB), run=3216-3353msec WRITE: bw=9773KiB/s (10.0MB/s), 2443KiB/s-7115KiB/s (2502kB/s-7286kB/s), io=32.0MiB (33.6MB), run=1727-3353msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23147: Sun Aug 4 13:15:37 2024 read: IOPS=0, BW=3783KiB/s (3874kB/s)(12.0MiB/3248msec) clat (msec): min=97, max=2747, avg=1079.71, stdev=1452.12 lat (msec): min=97, max=2747, avg=1079.71, stdev=1452.12 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 99], 10.00th=[ 99], 20.00th=[ 99], | 30.00th=[ 99], 40.00th=[ 393], 50.00th=[ 393], 60.00th=[ 393], | 70.00th=[ 2735], 80.00th=[ 2735], 90.00th=[ 2735], 95.00th=[ 2735], | 99.00th=[ 2735], 99.50th=[ 2735], 99.90th=[ 2735], 99.95th=[ 2735], | 99.99th=[ 2735] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.03%, sys=2.53%, ctx=31, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23148: Sun Aug 4 13:15:37 2024 read: IOPS=0, BW=1222KiB/s (1252kB/s)(4096KiB/3351msec) clat (nsec): min=292930k, max=292930k, avg=292929522.00, stdev= 0.00 lat (nsec): min=292932k, max=292932k, avg=292931889.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 292], 10.00th=[ 292], 20.00th=[ 292], | 30.00th=[ 292], 40.00th=[ 292], 50.00th=[ 292], 60.00th=[ 292], | 70.00th=[ 292], 80.00th=[ 292], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2445KiB/s (2503kB/s)(8192KiB/3351msec) clat (msec): min=1249, max=1775, avg=1512.40, stdev=372.01 lat (msec): min=1249, max=1775, avg=1512.63, stdev=372.12 clat percentiles (msec): | 1.00th=[ 1250], 5.00th=[ 1250], 10.00th=[ 1250], 20.00th=[ 1250], | 30.00th=[ 1250], 40.00th=[ 1250], 50.00th=[ 1250], 60.00th=[ 1770], | 70.00th=[ 1770], 80.00th=[ 1770], 90.00th=[ 1770], 95.00th=[ 1770], | 99.00th=[ 1770], 99.50th=[ 1770], 99.90th=[ 1770], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 8192, max= 8192, per=85.78%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33% cpu : usr=0.03%, sys=3.43%, ctx=173, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23149: Sun Aug 4 13:15:37 2024 write: IOPS=0, BW=3858KiB/s (3951kB/s)(12.0MiB/3185msec) clat (msec): min=356, max=2008, avg=1056.60, stdev=854.25 lat (msec): min=356, max=2008, avg=1056.81, stdev=854.32 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 359], | 30.00th=[ 359], 40.00th=[ 802], 50.00th=[ 802], 60.00th=[ 802], | 70.00th=[ 2005], 80.00th=[ 2005], 90.00th=[ 2005], 95.00th=[ 2005], | 99.00th=[ 2005], 99.50th=[ 2005], 99.90th=[ 2005], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 8192, max= 8192, per=85.78%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=3.49%, ctx=52, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23150: Sun Aug 4 13:15:37 2024 write: IOPS=0, BW=3581KiB/s (3667kB/s)(12.0MiB/3431msec) clat (msec): min=92, max=3002, avg=1139.66, stdev=1617.31 lat (msec): min=92, max=3002, avg=1139.91, stdev=1617.32 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 93], 10.00th=[ 93], 20.00th=[ 93], | 30.00th=[ 93], 40.00th=[ 326], 50.00th=[ 326], 60.00th=[ 326], | 70.00th=[ 3004], 80.00th=[ 3004], 90.00th=[ 3004], 95.00th=[ 3004], | 99.00th=[ 3004], 99.50th=[ 3004], 99.90th=[ 3004], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 8192, max= 8192, per=85.78%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.09%, sys=3.18%, ctx=197, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4889KiB/s (5007kB/s), 1222KiB/s-3783KiB/s (1252kB/s-3874kB/s), io=16.0MiB (16.8MB), run=3248-3351msec WRITE: bw=9551KiB/s (9780kB/s), 2445KiB/s-3858KiB/s (2503kB/s-3951kB/s), io=32.0MiB (33.6MB), run=3185-3431msec