== sanity test 398b: DIO and buffer IO race ============== 00:54:14 (1735106054) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.12124 s, 44.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4388: Wed Dec 25 00:54:29 2024 read: IOPS=124, BW=499KiB/s (511kB/s)(5992KiB/12008msec) clat (usec): min=49, max=164000, avg=3772.75, stdev=4965.87 lat (usec): min=49, max=164001, avg=3774.89, stdev=4965.66 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 62], 10.00th=[ 85], 20.00th=[ 2147], | 30.00th=[ 2442], 40.00th=[ 2737], 50.00th=[ 3130], 60.00th=[ 3654], | 70.00th=[ 4424], 80.00th=[ 5407], 90.00th=[ 6849], 95.00th=[ 8094], | 99.00th=[ 11731], 99.50th=[ 17171], 99.90th=[ 32900], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 296, max= 726, per=25.36%, avg=499.17, stdev=93.86, samples=24 iops : min= 74, max= 181, avg=124.71, stdev=23.44, samples=24 write: IOPS=131, BW=524KiB/s (537kB/s)(6296KiB/12008msec) clat (usec): min=422, max=37570, avg=3963.17, stdev=3754.36 lat (usec): min=422, max=37571, avg=3963.93, stdev=3754.31 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 562], 20.00th=[ 676], | 30.00th=[ 1057], 40.00th=[ 2343], 50.00th=[ 3261], 60.00th=[ 4146], | 70.00th=[ 5145], 80.00th=[ 6456], 90.00th=[ 8356], 95.00th=[10683], | 99.00th=[16712], 99.50th=[19792], 99.90th=[36439], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 360, max= 720, per=25.43%, avg=523.79, stdev=90.43, samples=24 iops : min= 90, max= 180, avg=130.88, stdev=22.59, samples=24 lat (usec) : 50=0.10%, 100=5.60%, 250=0.81%, 500=0.75%, 750=11.07% lat (usec) : 1000=3.48% lat (msec) : 2=5.40%, 4=34.21%, 10=34.57%, 20=3.65%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=23.85%, ctx=6220, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4390: Wed Dec 25 00:54:29 2024 read: IOPS=123, BW=492KiB/s (504kB/s)(5940KiB/12072msec) clat (usec): min=46, max=45543, avg=3712.02, stdev=2906.54 lat (usec): min=46, max=45544, avg=3712.90, stdev=2906.47 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 64], 10.00th=[ 76], 20.00th=[ 2073], | 30.00th=[ 2442], 40.00th=[ 2769], 50.00th=[ 3064], 60.00th=[ 3720], | 70.00th=[ 4424], 80.00th=[ 5538], 90.00th=[ 7046], 95.00th=[ 8356], | 99.00th=[13042], 99.50th=[14877], 99.90th=[29754], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 328, max= 664, per=25.06%, avg=493.08, stdev=86.43, samples=24 iops : min= 82, max= 166, avg=123.17, stdev=21.62, samples=24 write: IOPS=131, BW=526KiB/s (538kB/s)(6348KiB/12072msec) clat (usec): min=422, max=161155, avg=4060.40, stdev=5654.57 lat (usec): min=423, max=161175, avg=4062.44, stdev=5655.51 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 644], | 30.00th=[ 865], 40.00th=[ 1827], 50.00th=[ 3097], 60.00th=[ 3949], | 70.00th=[ 5014], 80.00th=[ 6325], 90.00th=[ 8979], 95.00th=[ 11469], | 99.00th=[ 19006], 99.50th=[ 23725], 99.90th=[ 38536], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 368, max= 768, per=25.51%, avg=525.42, stdev=99.50, samples=24 iops : min= 92, max= 192, avg=131.25, stdev=24.92, samples=24 lat (usec) : 50=0.10%, 100=6.02%, 250=0.98%, 500=0.72%, 750=12.66% lat (usec) : 1000=3.97% lat (msec) : 2=5.21%, 4=32.49%, 10=32.78%, 20=4.52%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=23.28%, ctx=6129, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4392: Wed Dec 25 00:54:29 2024 read: IOPS=124, BW=498KiB/s (510kB/s)(6028KiB/12099msec) clat (usec): min=48, max=41586, avg=3640.60, stdev=3085.87 lat (usec): min=48, max=41586, avg=3641.29, stdev=3085.85 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 61], 10.00th=[ 74], 20.00th=[ 1991], | 30.00th=[ 2343], 40.00th=[ 2671], 50.00th=[ 3032], 60.00th=[ 3523], | 70.00th=[ 4424], 80.00th=[ 5604], 90.00th=[ 6849], 95.00th=[ 8291], | 99.00th=[10814], 99.50th=[19268], 99.90th=[36963], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 64, max= 622, per=25.30%, avg=497.83, stdev=120.85, samples=24 iops : min= 16, max= 155, avg=124.42, stdev=30.19, samples=24 write: IOPS=129, BW=517KiB/s (530kB/s)(6260KiB/12099msec) clat (usec): min=469, max=191062, avg=4151.55, stdev=6176.03 lat (usec): min=470, max=191064, avg=4152.49, stdev=6176.03 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 660], | 30.00th=[ 971], 40.00th=[ 2114], 50.00th=[ 3261], 60.00th=[ 4080], | 70.00th=[ 5080], 80.00th=[ 6587], 90.00th=[ 9110], 95.00th=[ 11076], | 99.00th=[ 18482], 99.50th=[ 22676], 99.90th=[ 36439], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 80, max= 680, per=25.07%, avg=516.50, stdev=119.87, samples=24 iops : min= 20, max= 170, avg=129.08, stdev=29.95, samples=24 lat (usec) : 50=0.23%, 100=6.74%, 250=0.62%, 500=0.78%, 750=11.43% lat (usec) : 1000=3.58% lat (msec) : 2=6.61%, 4=32.32%, 10=33.14%, 20=4.00%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=23.27%, ctx=6239, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4394: Wed Dec 25 00:54:29 2024 read: IOPS=124, BW=496KiB/s (508kB/s)(6052KiB/12199msec) clat (usec): min=46, max=34631, avg=3755.18, stdev=3082.91 lat (usec): min=46, max=34631, avg=3755.97, stdev=3082.87 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 63], 10.00th=[ 81], 20.00th=[ 2089], | 30.00th=[ 2409], 40.00th=[ 2737], 50.00th=[ 3195], 60.00th=[ 3687], | 70.00th=[ 4424], 80.00th=[ 5407], 90.00th=[ 6980], 95.00th=[ 8717], | 99.00th=[13698], 99.50th=[22414], 99.90th=[31065], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 232, max= 704, per=25.05%, avg=492.92, stdev=112.38, samples=24 iops : min= 58, max= 176, avg=123.21, stdev=28.09, samples=24 write: IOPS=127, BW=511KiB/s (523kB/s)(6236KiB/12199msec) clat (usec): min=419, max=185471, avg=4104.82, stdev=6145.23 lat (usec): min=419, max=185473, avg=4105.57, stdev=6145.24 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 644], | 30.00th=[ 865], 40.00th=[ 1909], 50.00th=[ 3097], 60.00th=[ 3982], | 70.00th=[ 5145], 80.00th=[ 6521], 90.00th=[ 8586], 95.00th=[ 11207], | 99.00th=[ 18220], 99.50th=[ 24773], 99.90th=[ 38536], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 272, max= 704, per=24.62%, avg=507.25, stdev=102.78, samples=24 iops : min= 68, max= 176, avg=126.79, stdev=25.69, samples=24 lat (usec) : 50=0.33%, 100=5.89%, 250=0.78%, 500=0.75%, 750=12.21% lat (usec) : 1000=3.91% lat (msec) : 2=5.70%, 4=32.58%, 10=33.14%, 20=4.00%, 50=0.68% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=23.60%, ctx=6099, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1968KiB/s (2016kB/s), 492KiB/s-499KiB/s (504kB/s-511kB/s), io=23.4MiB (24.6MB), run=12008-12199msec WRITE: bw=2061KiB/s (2110kB/s), 511KiB/s-526KiB/s (523kB/s-538kB/s), io=24.6MiB (25.7MB), run=12008-12199msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4387: Wed Dec 25 00:54:50 2024 read: IOPS=45, BW=182KiB/s (187kB/s)(5992KiB/32866msec) clat (usec): min=1562, max=166220, avg=3338.43, stdev=5167.21 lat (usec): min=1562, max=166222, avg=3339.16, stdev=5167.23 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1795], 10.00th=[ 1860], 20.00th=[ 1942], | 30.00th=[ 2008], 40.00th=[ 2089], 50.00th=[ 2180], 60.00th=[ 2278], | 70.00th=[ 2540], 80.00th=[ 3359], 90.00th=[ 6325], 95.00th=[ 8717], | 99.00th=[ 15401], 99.50th=[ 21103], 99.90th=[ 41157], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 64, max= 344, per=25.10%, avg=182.22, stdev=68.33, samples=65 iops : min= 16, max= 86, avg=45.51, stdev=17.10, samples=65 write: IOPS=47, BW=192KiB/s (196kB/s)(6296KiB/32866msec) clat (usec): min=9156, max=66302, avg=17631.07, stdev=6430.07 lat (usec): min=9156, max=66303, avg=17631.99, stdev=6430.10 clat percentiles (usec): | 1.00th=[ 9896], 5.00th=[10552], 10.00th=[11207], 20.00th=[12125], | 30.00th=[13698], 40.00th=[15270], 50.00th=[16450], 60.00th=[17433], | 70.00th=[18744], 80.00th=[21103], 90.00th=[26346], 95.00th=[30802], | 99.00th=[39584], 99.50th=[41157], 99.90th=[53216], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 72, max= 264, per=25.09%, avg=190.71, stdev=55.98, samples=65 iops : min= 18, max= 66, avg=47.63, stdev=14.00, samples=65 lat (msec) : 2=14.06%, 4=26.27%, 10=7.13%, 20=40.14%, 50=12.27% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.08%, sys=5.52%, ctx=3439, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4389: Wed Dec 25 00:54:50 2024 read: IOPS=44, BW=180KiB/s (184kB/s)(5940KiB/33043msec) clat (usec): min=1538, max=167582, avg=3405.51, stdev=5279.23 lat (usec): min=1539, max=167584, avg=3406.25, stdev=5279.30 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1811], 10.00th=[ 1876], 20.00th=[ 1958], | 30.00th=[ 2024], 40.00th=[ 2089], 50.00th=[ 2180], 60.00th=[ 2278], | 70.00th=[ 2507], 80.00th=[ 3261], 90.00th=[ 6587], 95.00th=[ 9634], | 99.00th=[ 18220], 99.50th=[ 21365], 99.90th=[ 40633], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 16, max= 368, per=24.80%, avg=180.02, stdev=76.89, samples=65 iops : min= 4, max= 92, avg=44.88, stdev=19.22, samples=65 write: IOPS=48, BW=192KiB/s (197kB/s)(6348KiB/33043msec) clat (usec): min=9262, max=62687, avg=17561.47, stdev=6488.15 lat (usec): min=9263, max=62688, avg=17562.32, stdev=6488.18 clat percentiles (usec): | 1.00th=[ 9896], 5.00th=[10552], 10.00th=[11076], 20.00th=[11994], | 30.00th=[13435], 40.00th=[15139], 50.00th=[16450], 60.00th=[17695], | 70.00th=[19006], 80.00th=[21103], 90.00th=[26346], 95.00th=[31327], | 99.00th=[38536], 99.50th=[42730], 99.90th=[61604], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 72, max= 272, per=25.08%, avg=190.60, stdev=55.94, samples=65 iops : min= 18, max= 68, avg=47.52, stdev=14.00, samples=65 lat (msec) : 2=13.35%, 4=26.99%, 10=6.51%, 20=40.56%, 50=12.47% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.11%, sys=5.48%, ctx=3423, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4391: Wed Dec 25 00:54:50 2024 read: IOPS=45, BW=184KiB/s (188kB/s)(6028KiB/32826msec) clat (usec): min=1594, max=25990, avg=3308.13, stdev=2870.85 lat (usec): min=1595, max=25990, avg=3308.82, stdev=2870.88 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1811], 10.00th=[ 1876], 20.00th=[ 1958], | 30.00th=[ 2024], 40.00th=[ 2114], 50.00th=[ 2180], 60.00th=[ 2311], | 70.00th=[ 2573], 80.00th=[ 3458], 90.00th=[ 6652], 95.00th=[ 9241], | 99.00th=[16188], 99.50th=[17957], 99.90th=[25822], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 72, max= 327, per=25.32%, avg=183.85, stdev=61.96, samples=65 iops : min= 18, max= 81, avg=45.94, stdev=15.48, samples=65 write: IOPS=47, BW=191KiB/s (195kB/s)(6260KiB/32826msec) clat (msec): min=8, max=183, avg=17.72, stdev= 7.55 lat (msec): min=8, max=183, avg=17.72, stdev= 7.55 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 20], 80.00th=[ 21], 90.00th=[ 27], 95.00th=[ 31], | 99.00th=[ 40], 99.50th=[ 44], 99.90th=[ 58], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 80, max= 264, per=25.03%, avg=190.25, stdev=55.98, samples=65 iops : min= 20, max= 66, avg=47.54, stdev=14.00, samples=65 lat (msec) : 2=12.66%, 4=27.28%, 10=7.81%, 20=40.14%, 50=11.98% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.09%, sys=5.57%, ctx=3433, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4393: Wed Dec 25 00:54:50 2024 read: IOPS=46, BW=185KiB/s (189kB/s)(6052KiB/32706msec) clat (usec): min=1645, max=32132, avg=3187.51, stdev=2658.07 lat (usec): min=1646, max=32133, avg=3188.21, stdev=2658.10 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1811], 10.00th=[ 1860], 20.00th=[ 1942], | 30.00th=[ 2008], 40.00th=[ 2089], 50.00th=[ 2147], 60.00th=[ 2278], | 70.00th=[ 2540], 80.00th=[ 3326], 90.00th=[ 6456], 95.00th=[ 8848], | 99.00th=[13829], 99.50th=[17171], 99.90th=[22676], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 48, max= 352, per=25.40%, avg=184.37, stdev=74.61, samples=65 iops : min= 12, max= 88, avg=46.00, stdev=18.67, samples=65 write: IOPS=47, BW=191KiB/s (195kB/s)(6236KiB/32706msec) clat (msec): min=8, max=184, avg=17.81, stdev= 7.74 lat (msec): min=8, max=184, avg=17.81, stdev= 7.74 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 20], 80.00th=[ 21], 90.00th=[ 27], 95.00th=[ 31], | 99.00th=[ 41], 99.50th=[ 45], 99.90th=[ 67], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 64, max= 264, per=25.02%, avg=190.15, stdev=54.83, samples=65 iops : min= 16, max= 66, avg=47.45, stdev=13.75, samples=65 lat (msec) : 2=14.10%, 4=26.89%, 10=7.19%, 20=39.97%, 50=11.69% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.13%, sys=5.51%, ctx=3423, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=727KiB/s (744kB/s), 180KiB/s-185KiB/s (184kB/s-189kB/s), io=23.4MiB (24.6MB), run=32706-33043msec WRITE: bw=761KiB/s (779kB/s), 191KiB/s-192KiB/s (195kB/s-197kB/s), io=24.6MiB (25.7MB), run=32706-33043msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4432: Wed Dec 25 00:54:55 2024 read: IOPS=79, BW=1267KiB/s (1298kB/s)(5920KiB/4671msec) clat (usec): min=53, max=32778, avg=9140.77, stdev=5240.77 lat (usec): min=53, max=32779, avg=9141.41, stdev=5240.79 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 64], 10.00th=[ 76], 20.00th=[ 7046], | 30.00th=[ 8291], 40.00th=[ 9372], 50.00th=[ 9896], 60.00th=[10814], | 70.00th=[11731], 80.00th=[12518], 90.00th=[14222], 95.00th=[16319], | 99.00th=[22676], 99.50th=[24773], 99.90th=[32900], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 864, max= 1440, per=24.27%, avg=1244.44, stdev=172.66, samples=9 iops : min= 54, max= 90, avg=77.78, stdev=10.79, samples=9 write: IOPS=85, BW=1363KiB/s (1396kB/s)(6368KiB/4671msec) clat (usec): min=616, max=28987, avg=3184.60, stdev=3567.20 lat (usec): min=617, max=28989, avg=3185.79, stdev=3567.16 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 717], 10.00th=[ 766], 20.00th=[ 840], | 30.00th=[ 971], 40.00th=[ 1254], 50.00th=[ 2278], 60.00th=[ 2900], | 70.00th=[ 3425], 80.00th=[ 4883], 90.00th=[ 6652], 95.00th=[ 8586], | 99.00th=[21365], 99.50th=[28705], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 896, max= 2016, per=25.25%, avg=1361.78, stdev=351.31, samples=9 iops : min= 56, max= 126, avg=85.11, stdev=21.96, samples=9 lat (usec) : 100=7.68%, 250=1.30%, 500=0.13%, 750=3.65%, 1000=12.63% lat (msec) : 2=8.20%, 4=14.71%, 10=25.65%, 20=24.61%, 50=1.43% cpu : usr=0.13%, sys=22.25%, ctx=2127, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4433: Wed Dec 25 00:54:55 2024 read: IOPS=78, BW=1261KiB/s (1291kB/s)(5744KiB/4556msec) clat (usec): min=53, max=24463, avg=9033.84, stdev=4603.62 lat (usec): min=54, max=24465, avg=9034.50, stdev=4603.54 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 67], 10.00th=[ 77], 20.00th=[ 7046], | 30.00th=[ 8356], 40.00th=[ 9110], 50.00th=[10028], 60.00th=[10683], | 70.00th=[11207], 80.00th=[12387], 90.00th=[13698], 95.00th=[14877], | 99.00th=[18482], 99.50th=[20579], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 832, max= 1472, per=24.41%, avg=1251.56, stdev=182.04, samples=9 iops : min= 52, max= 92, avg=78.22, stdev=11.38, samples=9 write: IOPS=89, BW=1436KiB/s (1471kB/s)(6544KiB/4556msec) clat (usec): min=642, max=25147, avg=3177.60, stdev=3410.32 lat (usec): min=642, max=25148, avg=3179.10, stdev=3410.97 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 725], 10.00th=[ 766], 20.00th=[ 832], | 30.00th=[ 938], 40.00th=[ 1172], 50.00th=[ 2212], 60.00th=[ 2704], | 70.00th=[ 3687], 80.00th=[ 5080], 90.00th=[ 6652], 95.00th=[ 9110], | 99.00th=[17957], 99.50th=[20841], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 1056, max= 1792, per=26.50%, avg=1429.33, stdev=245.80, samples=9 iops : min= 66, max= 112, avg=89.33, stdev=15.36, samples=9 lat (usec) : 100=6.25%, 250=1.17%, 750=4.69%, 1000=13.54% lat (msec) : 2=7.55%, 4=13.28%, 10=27.73%, 20=25.00%, 50=0.78% cpu : usr=0.11%, sys=22.20%, ctx=2184, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4434: Wed Dec 25 00:54:55 2024 read: IOPS=85, BW=1373KiB/s (1406kB/s)(6256KiB/4556msec) clat (usec): min=50, max=28459, avg=8953.51, stdev=5004.25 lat (usec): min=51, max=28460, avg=8954.21, stdev=5004.23 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 64], 10.00th=[ 79], 20.00th=[ 6718], | 30.00th=[ 8586], 40.00th=[ 9241], 50.00th=[ 9765], 60.00th=[10683], | 70.00th=[11338], 80.00th=[12387], 90.00th=[13829], 95.00th=[15008], | 99.00th=[21890], 99.50th=[23725], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 832, max= 1696, per=26.49%, avg=1358.22, stdev=236.84, samples=9 iops : min= 52, max= 106, avg=84.89, stdev=14.80, samples=9 write: IOPS=82, BW=1324KiB/s (1356kB/s)(6032KiB/4556msec) clat (usec): min=623, max=26841, avg=2713.95, stdev=3128.82 lat (usec): min=624, max=26842, avg=2714.97, stdev=3128.85 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 701], 10.00th=[ 742], 20.00th=[ 791], | 30.00th=[ 857], 40.00th=[ 996], 50.00th=[ 1369], 60.00th=[ 2376], | 70.00th=[ 3032], 80.00th=[ 3982], 90.00th=[ 5735], 95.00th=[ 8160], | 99.00th=[18220], 99.50th=[19792], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 1056, max= 1568, per=23.86%, avg=1287.11, stdev=175.84, samples=9 iops : min= 66, max= 98, avg=80.44, stdev=10.99, samples=9 lat (usec) : 100=7.68%, 250=1.69%, 750=6.12%, 1000=13.80% lat (msec) : 2=6.77%, 4=12.89%, 10=25.00%, 20=25.00%, 50=1.04% cpu : usr=0.18%, sys=21.98%, ctx=2137, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4435: Wed Dec 25 00:54:55 2024 read: IOPS=84, BW=1352KiB/s (1385kB/s)(6032KiB/4461msec) clat (usec): min=52, max=39444, avg=8737.74, stdev=5381.63 lat (usec): min=52, max=39445, avg=8738.44, stdev=5381.83 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 63], 10.00th=[ 73], 20.00th=[ 131], | 30.00th=[ 8094], 40.00th=[ 9110], 50.00th=[ 9896], 60.00th=[10421], | 70.00th=[11076], 80.00th=[11994], 90.00th=[14222], 95.00th=[16450], | 99.00th=[21365], 99.50th=[23200], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 896, max= 1504, per=25.51%, avg=1308.00, stdev=214.62, samples=8 iops : min= 56, max= 94, avg=81.75, stdev=13.41, samples=8 write: IOPS=87, BW=1402KiB/s (1436kB/s)(6256KiB/4461msec) clat (usec): min=627, max=29569, avg=2883.09, stdev=3399.01 lat (usec): min=628, max=29570, avg=2884.35, stdev=3398.95 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 709], 10.00th=[ 766], 20.00th=[ 824], | 30.00th=[ 906], 40.00th=[ 1090], 50.00th=[ 1418], 60.00th=[ 2474], | 70.00th=[ 3130], 80.00th=[ 4490], 90.00th=[ 6063], 95.00th=[ 8586], | 99.00th=[16450], 99.50th=[22414], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 1056, max= 1664, per=25.29%, avg=1364.00, stdev=194.23, samples=8 iops : min= 66, max= 104, avg=85.25, stdev=12.14, samples=8 lat (usec) : 100=9.37%, 250=1.04%, 500=0.13%, 750=4.30%, 1000=13.93% lat (msec) : 2=10.03%, 4=11.85%, 10=24.09%, 20=23.83%, 50=1.43% cpu : usr=0.13%, sys=22.11%, ctx=2109, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5128KiB/s (5251kB/s), 1261KiB/s-1373KiB/s (1291kB/s-1406kB/s), io=23.4MiB (24.5MB), run=4461-4671msec WRITE: bw=5395KiB/s (5524kB/s), 1324KiB/s-1436KiB/s (1356kB/s-1471kB/s), io=24.6MiB (25.8MB), run=4461-4671msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4436: Wed Dec 25 00:54:59 2024 read: IOPS=42, BW=678KiB/s (695kB/s)(5920KiB/8726msec) clat (usec): min=1877, max=21043, avg=4352.65, stdev=3225.58 lat (usec): min=1877, max=21043, avg=4353.57, stdev=3225.52 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2073], 10.00th=[ 2147], 20.00th=[ 2376], | 30.00th=[ 2606], 40.00th=[ 2900], 50.00th=[ 3228], 60.00th=[ 3490], | 70.00th=[ 3949], 80.00th=[ 5342], 90.00th=[ 9372], 95.00th=[11994], | 99.00th=[17433], 99.50th=[20841], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 416, max= 1120, per=24.50%, avg=662.47, stdev=191.94, samples=17 iops : min= 26, max= 70, avg=41.35, stdev=12.00, samples=17 write: IOPS=45, BW=730KiB/s (747kB/s)(6368KiB/8726msec) clat (usec): min=9956, max=46830, avg=17609.04, stdev=5702.70 lat (usec): min=9957, max=46832, avg=17610.84, stdev=5702.58 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11076], 10.00th=[11731], 20.00th=[12649], | 30.00th=[13698], 40.00th=[15270], 50.00th=[16712], 60.00th=[17695], | 70.00th=[19530], 80.00th=[21627], 90.00th=[24773], 95.00th=[27919], | 99.00th=[38011], 99.50th=[40109], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 384, max= 1024, per=25.47%, avg=724.59, stdev=197.11, samples=17 iops : min= 24, max= 64, avg=45.24, stdev=12.27, samples=17 lat (msec) : 2=1.17%, 4=33.07%, 10=10.03%, 20=40.89%, 50=14.84% cpu : usr=0.16%, sys=7.81%, ctx=891, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4437: Wed Dec 25 00:54:59 2024 read: IOPS=40, BW=649KiB/s (664kB/s)(5744KiB/8855msec) clat (usec): min=1867, max=23503, avg=4357.58, stdev=3538.15 lat (usec): min=1868, max=23504, avg=4358.44, stdev=3538.15 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2114], 10.00th=[ 2180], 20.00th=[ 2376], | 30.00th=[ 2573], 40.00th=[ 2900], 50.00th=[ 3228], 60.00th=[ 3523], | 70.00th=[ 3916], 80.00th=[ 5014], 90.00th=[ 7504], 95.00th=[13435], | 99.00th=[19792], 99.50th=[20841], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 288, max= 1120, per=24.00%, avg=649.06, stdev=249.07, samples=17 iops : min= 18, max= 70, avg=40.41, stdev=15.56, samples=17 write: IOPS=46, BW=739KiB/s (757kB/s)(6544KiB/8855msec) clat (usec): min=9834, max=66072, avg=17739.53, stdev=5713.22 lat (usec): min=9835, max=66075, avg=17741.21, stdev=5713.17 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11469], 10.00th=[12125], 20.00th=[13042], | 30.00th=[14222], 40.00th=[15401], 50.00th=[16581], 60.00th=[17957], | 70.00th=[19268], 80.00th=[21365], 90.00th=[25035], 95.00th=[28181], | 99.00th=[34866], 99.50th=[36439], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 448, max= 960, per=25.73%, avg=731.88, stdev=162.38, samples=17 iops : min= 28, max= 60, avg=45.59, stdev=10.17, samples=17 lat (msec) : 2=1.04%, 4=32.29%, 10=10.16%, 20=42.45%, 50=13.93% lat (msec) : 100=0.13% cpu : usr=0.10%, sys=7.65%, ctx=903, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4438: Wed Dec 25 00:54:59 2024 read: IOPS=45, BW=728KiB/s (746kB/s)(6256KiB/8590msec) clat (usec): min=1852, max=32492, avg=4512.50, stdev=4021.41 lat (usec): min=1852, max=32493, avg=4513.45, stdev=4021.37 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2089], 10.00th=[ 2212], 20.00th=[ 2474], | 30.00th=[ 2704], 40.00th=[ 2999], 50.00th=[ 3326], 60.00th=[ 3589], | 70.00th=[ 3949], 80.00th=[ 4752], 90.00th=[ 8029], 95.00th=[12518], | 99.00th=[23200], 99.50th=[30016], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 384, max= 1440, per=27.14%, avg=734.00, stdev=296.18, samples=17 iops : min= 24, max= 90, avg=45.82, stdev=18.50, samples=17 write: IOPS=43, BW=702KiB/s (719kB/s)(6032KiB/8590msec) clat (usec): min=9046, max=48113, avg=17778.38, stdev=5740.19 lat (usec): min=9047, max=48116, avg=17780.53, stdev=5739.84 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11207], 10.00th=[11731], 20.00th=[12780], | 30.00th=[14091], 40.00th=[15533], 50.00th=[16712], 60.00th=[18220], | 70.00th=[19530], 80.00th=[21890], 90.00th=[25035], 95.00th=[28705], | 99.00th=[39060], 99.50th=[40633], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 352, max= 990, per=24.48%, avg=696.35, stdev=183.11, samples=17 iops : min= 22, max= 61, avg=43.47, stdev=11.36, samples=17 lat (msec) : 2=0.65%, 4=35.68%, 10=11.07%, 20=38.41%, 50=14.19% cpu : usr=0.14%, sys=7.85%, ctx=934, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4439: Wed Dec 25 00:54:59 2024 read: IOPS=43, BW=692KiB/s (709kB/s)(6032KiB/8714msec) clat (usec): min=1847, max=29556, avg=4085.90, stdev=3094.70 lat (usec): min=1847, max=29557, avg=4086.73, stdev=3094.69 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2057], 10.00th=[ 2180], 20.00th=[ 2409], | 30.00th=[ 2606], 40.00th=[ 2802], 50.00th=[ 3097], 60.00th=[ 3392], | 70.00th=[ 3720], 80.00th=[ 4752], 90.00th=[ 7570], 95.00th=[ 9765], | 99.00th=[18482], 99.50th=[25035], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 320, max= 896, per=25.13%, avg=679.53, stdev=181.91, samples=17 iops : min= 20, max= 56, avg=42.47, stdev=11.37, samples=17 write: IOPS=44, BW=718KiB/s (735kB/s)(6256KiB/8714msec) clat (usec): min=9829, max=41684, avg=18032.23, stdev=5709.62 lat (usec): min=9830, max=41685, avg=18034.40, stdev=5708.87 clat percentiles (usec): | 1.00th=[10028], 5.00th=[10945], 10.00th=[11731], 20.00th=[12911], | 30.00th=[14353], 40.00th=[15664], 50.00th=[16909], 60.00th=[18220], | 70.00th=[20055], 80.00th=[22414], 90.00th=[26346], 95.00th=[29230], | 99.00th=[35914], 99.50th=[38011], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 352, max= 1024, per=25.21%, avg=717.18, stdev=185.92, samples=17 iops : min= 22, max= 64, avg=44.82, stdev=11.62, samples=17 lat (msec) : 2=1.17%, 4=35.68%, 10=10.42%, 20=37.11%, 50=15.62% cpu : usr=0.05%, sys=7.71%, ctx=903, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2705KiB/s (2770kB/s), 649KiB/s-728KiB/s (664kB/s-746kB/s), io=23.4MiB (24.5MB), run=8590-8855msec WRITE: bw=2846KiB/s (2914kB/s), 702KiB/s-739KiB/s (719kB/s-757kB/s), io=24.6MiB (25.8MB), run=8590-8855msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4452: Wed Dec 25 00:55:03 2024 read: IOPS=3, BW=3528KiB/s (3612kB/s)(7168KiB/2032msec) clat (usec): min=337, max=713860, avg=251509.34, stdev=300660.21 lat (usec): min=338, max=713861, avg=251510.44, stdev=300659.98 clat percentiles (usec): | 1.00th=[ 338], 5.00th=[ 338], 10.00th=[ 338], 20.00th=[ 635], | 30.00th=[ 29492], 40.00th=[ 29492], 50.00th=[ 90702], 60.00th=[325059], | 70.00th=[325059], 80.00th=[599786], 90.00th=[717226], 95.00th=[717226], | 99.00th=[717226], 99.50th=[717226], 99.90th=[717226], 99.95th=[717226], | 99.99th=[717226] bw ( KiB/s): min= 2043, max= 4096, per=34.40%, avg=3070.75, stdev=1183.86, samples=4 iops : min= 1, max= 4, avg= 2.75, stdev= 1.50, samples=4 write: IOPS=2, BW=2520KiB/s (2580kB/s)(5120KiB/2032msec) clat (msec): min=17, max=134, avg=50.80, stdev=47.23 lat (msec): min=17, max=134, avg=50.87, stdev=47.25 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 34], 60.00th=[ 34], | 70.00th=[ 40], 80.00th=[ 40], 90.00th=[ 134], 95.00th=[ 134], | 99.00th=[ 134], 99.50th=[ 134], 99.90th=[ 134], 99.95th=[ 134], | 99.99th=[ 134] bw ( KiB/s): min= 2043, max= 6144, per=32.75%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 20=8.33%, 50=33.33%, 100=8.33%, 250=8.33%, 500=8.33% lat (msec) : 750=16.67% cpu : usr=0.00%, sys=21.71%, ctx=739, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4453: Wed Dec 25 00:55:03 2024 read: IOPS=0, BW=893KiB/s (914kB/s)(2048KiB/2294msec) clat (msec): min=52, max=675, avg=363.66, stdev=440.66 lat (msec): min=52, max=675, avg=363.66, stdev=440.66 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 53], 50.00th=[ 53], 60.00th=[ 676], | 70.00th=[ 676], 80.00th=[ 676], 90.00th=[ 676], 95.00th=[ 676], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 2048, max= 2048, per=22.94%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4464KiB/s (4571kB/s)(10.0MiB/2294msec) clat (msec): min=23, max=841, avg=151.20, stdev=252.22 lat (msec): min=23, max=841, avg=151.26, stdev=252.21 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 24], | 30.00th=[ 24], 40.00th=[ 35], 50.00th=[ 54], 60.00th=[ 58], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 247], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 4096, max= 6144, per=43.70%, avg=5461.33, stdev=1182.41, samples=3 iops : min= 4, max= 6, avg= 5.33, stdev= 1.15, samples=3 lat (msec) : 50=33.33%, 100=33.33%, 250=16.67%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=17.36%, ctx=334, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4454: Wed Dec 25 00:55:03 2024 read: IOPS=2, BW=2926KiB/s (2996kB/s)(6144KiB/2100msec) clat (usec): min=316, max=148297, avg=46071.30, stdev=55629.61 lat (usec): min=317, max=148298, avg=46072.14, stdev=55629.75 clat percentiles (usec): | 1.00th=[ 318], 5.00th=[ 318], 10.00th=[ 318], 20.00th=[ 562], | 30.00th=[ 562], 40.00th=[ 21890], 50.00th=[ 21890], 60.00th=[ 42730], | 70.00th=[ 62653], 80.00th=[ 62653], 90.00th=[147850], 95.00th=[147850], | 99.00th=[147850], 99.50th=[147850], 99.90th=[147850], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 2048, max= 8192, per=57.35%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=2, BW=2926KiB/s (2996kB/s)(6144KiB/2100msec) clat (msec): min=24, max=808, avg=294.84, stdev=316.47 lat (msec): min=25, max=808, avg=294.90, stdev=316.47 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 97], 50.00th=[ 97], 60.00th=[ 259], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 810], 95.00th=[ 810], | 99.00th=[ 810], 99.50th=[ 810], 99.90th=[ 810], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 2043, max= 4096, per=24.57%, avg=3070.75, stdev=1183.86, samples=4 iops : min= 1, max= 4, avg= 2.75, stdev= 1.50, samples=4 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=33.33%, 100=16.67%, 250=8.33%, 500=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=11.58%, ctx=180, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4455: Wed Dec 25 00:55:03 2024 read: IOPS=2, BW=2440KiB/s (2499kB/s)(5120KiB/2098msec) clat (usec): min=316, max=686555, avg=195365.52, stdev=291112.16 lat (usec): min=316, max=686556, avg=195366.81, stdev=291111.98 clat percentiles (usec): | 1.00th=[ 318], 5.00th=[ 318], 10.00th=[ 318], 20.00th=[ 318], | 30.00th=[ 474], 40.00th=[ 474], 50.00th=[ 53740], 60.00th=[ 53740], | 70.00th=[235930], 80.00th=[235930], 90.00th=[683672], 95.00th=[683672], | 99.00th=[683672], 99.50th=[683672], 99.90th=[683672], 99.95th=[683672], | 99.99th=[683672] bw ( KiB/s): min= 2048, max= 6144, per=45.88%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=3, BW=3417KiB/s (3499kB/s)(7168KiB/2098msec) clat (msec): min=24, max=738, avg=156.37, stdev=257.94 lat (msec): min=24, max=738, avg=156.43, stdev=257.94 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 40], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 65], 60.00th=[ 84], | 70.00th=[ 84], 80.00th=[ 102], 90.00th=[ 743], 95.00th=[ 743], | 99.00th=[ 743], 99.50th=[ 743], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 2048, max= 6144, per=38.20%, avg=4774.33, stdev=2361.08, samples=3 iops : min= 2, max= 6, avg= 4.33, stdev= 2.08, samples=3 lat (usec) : 500=16.67% lat (msec) : 50=25.00%, 100=25.00%, 250=16.67%, 750=16.67% cpu : usr=0.00%, sys=18.50%, ctx=445, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8928KiB/s (9142kB/s), 893KiB/s-3528KiB/s (914kB/s-3612kB/s), io=20.0MiB (20.0MB), run=2032-2294msec WRITE: bw=12.2MiB/s (12.8MB/s), 2520KiB/s-4464KiB/s (2580kB/s-4571kB/s), io=28.0MiB (29.4MB), run=2032-2294msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4456: Wed Dec 25 00:55:03 2024 read: IOPS=3, BW=3451KiB/s (3534kB/s)(7168KiB/2077msec) clat (msec): min=12, max=772, avg=245.32, stdev=299.35 lat (msec): min=12, max=772, avg=245.32, stdev=299.35 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 90], 60.00th=[ 255], | 70.00th=[ 255], 80.00th=[ 535], 90.00th=[ 776], 95.00th=[ 776], | 99.00th=[ 776], 99.50th=[ 776], 99.90th=[ 776], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 2043, max= 4096, per=34.19%, avg=3070.75, stdev=1183.86, samples=4 iops : min= 1, max= 4, avg= 2.75, stdev= 1.50, samples=4 write: IOPS=2, BW=2465KiB/s (2524kB/s)(5120KiB/2077msec) clat (usec): min=35946, max=73732, avg=58602.77, stdev=17879.93 lat (usec): min=36013, max=73761, avg=58663.67, stdev=17873.71 clat percentiles (usec): | 1.00th=[35914], 5.00th=[35914], 10.00th=[35914], 20.00th=[35914], | 30.00th=[42730], 40.00th=[42730], 50.00th=[69731], 60.00th=[69731], | 70.00th=[70779], 80.00th=[70779], 90.00th=[73925], 95.00th=[73925], | 99.00th=[73925], 99.50th=[73925], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 8192, max= 8192, per=65.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=25.00%, 100=33.33%, 500=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.05%, sys=4.58%, ctx=48, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4457: Wed Dec 25 00:55:03 2024 read: IOPS=0, BW=920KiB/s (943kB/s)(2048KiB/2225msec) clat (usec): min=14692, max=18835, avg=16764.09, stdev=2929.60 lat (usec): min=14694, max=18838, avg=16766.17, stdev=2930.05 clat percentiles (usec): | 1.00th=[14746], 5.00th=[14746], 10.00th=[14746], 20.00th=[14746], | 30.00th=[14746], 40.00th=[14746], 50.00th=[14746], 60.00th=[18744], | 70.00th=[18744], 80.00th=[18744], 90.00th=[18744], 95.00th=[18744], | 99.00th=[18744], 99.50th=[18744], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 2048, max= 2048, per=22.80%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4602KiB/s (4713kB/s)(10.0MiB/2225msec) clat (msec): min=24, max=786, avg=215.78, stdev=263.14 lat (msec): min=24, max=787, avg=215.84, stdev=263.15 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 36], | 30.00th=[ 44], 40.00th=[ 57], 50.00th=[ 60], 60.00th=[ 80], | 70.00th=[ 262], 80.00th=[ 262], 90.00th=[ 575], 95.00th=[ 785], | 99.00th=[ 785], 99.50th=[ 785], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 2048, max= 8192, per=32.57%, avg=4096.00, stdev=2896.31, samples=4 iops : min= 2, max= 8, avg= 4.00, stdev= 2.83, samples=4 lat (msec) : 20=16.67%, 50=25.00%, 100=25.00%, 250=8.33%, 500=8.33% lat (msec) : 750=8.33%, 1000=8.33% cpu : usr=0.04%, sys=5.26%, ctx=60, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4458: Wed Dec 25 00:55:03 2024 read: IOPS=2, BW=2695KiB/s (2759kB/s)(6144KiB/2280msec) clat (msec): min=12, max=262, avg=74.32, stdev=94.76 lat (msec): min=12, max=262, avg=74.32, stdev=94.76 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 26], | 30.00th=[ 26], 40.00th=[ 26], 50.00th=[ 26], 60.00th=[ 47], | 70.00th=[ 74], 80.00th=[ 74], 90.00th=[ 264], 95.00th=[ 264], | 99.00th=[ 264], 99.50th=[ 264], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2695KiB/s (2759kB/s)(6144KiB/2280msec) clat (msec): min=27, max=825, avg=296.38, stdev=320.75 lat (msec): min=27, max=825, avg=296.43, stdev=320.75 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 44], | 30.00th=[ 44], 40.00th=[ 94], 50.00th=[ 94], 60.00th=[ 257], | 70.00th=[ 531], 80.00th=[ 531], 90.00th=[ 827], 95.00th=[ 827], | 99.00th=[ 827], 99.50th=[ 827], 99.90th=[ 827], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 4087, max= 6144, per=40.68%, avg=5115.50, stdev=1454.52, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 lat (msec) : 20=8.33%, 50=41.67%, 100=16.67%, 500=16.67%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.04%, sys=3.77%, ctx=51, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4459: Wed Dec 25 00:55:03 2024 read: IOPS=2, BW=2446KiB/s (2505kB/s)(5120KiB/2093msec) clat (msec): min=13, max=621, avg=148.33, stdev=264.83 lat (msec): min=13, max=621, avg=148.33, stdev=264.83 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 14], | 30.00th=[ 25], 40.00th=[ 25], 50.00th=[ 33], 60.00th=[ 33], | 70.00th=[ 51], 80.00th=[ 51], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 8192, max= 8192, per=91.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3425KiB/s (3507kB/s)(7168KiB/2093msec) clat (msec): min=42, max=837, avg=192.10, stdev=286.20 lat (msec): min=42, max=837, avg=192.22, stdev=286.19 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 64], | 30.00th=[ 69], 40.00th=[ 69], 50.00th=[ 81], 60.00th=[ 120], | 70.00th=[ 120], 80.00th=[ 136], 90.00th=[ 835], 95.00th=[ 835], | 99.00th=[ 835], 99.50th=[ 835], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 2043, max=10240, per=37.99%, avg=4777.00, stdev=4731.10, samples=3 iops : min= 1, max= 10, avg= 4.33, stdev= 4.93, samples=3 lat (msec) : 20=8.33%, 50=25.00%, 100=33.33%, 250=16.67%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=4.11%, ctx=60, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8982KiB/s (9198kB/s), 920KiB/s-3451KiB/s (943kB/s-3534kB/s), io=20.0MiB (20.0MB), run=2077-2280msec WRITE: bw=12.3MiB/s (12.9MB/s), 2465KiB/s-4602KiB/s (2524kB/s-4713kB/s), io=28.0MiB (29.4MB), run=2077-2280msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4466: Wed Dec 25 00:55:09 2024 read: IOPS=0, BW=2369KiB/s (2426kB/s)(12.0MiB/5186msec) clat (msec): min=203, max=3213, avg=1724.06, stdev=1505.15 lat (msec): min=203, max=3213, avg=1724.07, stdev=1505.15 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 205], 10.00th=[ 205], 20.00th=[ 205], | 30.00th=[ 205], 40.00th=[ 1754], 50.00th=[ 1754], 60.00th=[ 1754], | 70.00th=[ 3205], 80.00th=[ 3205], 90.00th=[ 3205], 95.00th=[ 3205], | 99.00th=[ 3205], 99.50th=[ 3205], 99.90th=[ 3205], 99.95th=[ 3205], | 99.99th=[ 3205] bw ( KiB/s): min= 5595, max= 5603, per=100.00%, avg=5599.00, stdev= 5.66, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=1.56%, ctx=36, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4468: Wed Dec 25 00:55:09 2024 read: IOPS=0, BW=1103KiB/s (1129kB/s)(4096KiB/3715msec) clat (nsec): min=190295k, max=190295k, avg=190295438.00, stdev= 0.00 lat (nsec): min=190298k, max=190298k, avg=190297663.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 190], 10.00th=[ 190], 20.00th=[ 190], | 30.00th=[ 190], 40.00th=[ 190], 50.00th=[ 190], 60.00th=[ 190], | 70.00th=[ 190], 80.00th=[ 190], 90.00th=[ 190], 95.00th=[ 190], | 99.00th=[ 190], 99.50th=[ 190], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2205KiB/s (2258kB/s)(8192KiB/3715msec) clat (msec): min=203, max=3304, avg=1754.11, stdev=2192.64 lat (msec): min=203, max=3304, avg=1754.38, stdev=2192.66 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 205], 10.00th=[ 205], 20.00th=[ 205], | 30.00th=[ 205], 40.00th=[ 205], 50.00th=[ 205], 60.00th=[ 3306], | 70.00th=[ 3306], 80.00th=[ 3306], 90.00th=[ 3306], 95.00th=[ 3306], | 99.00th=[ 3306], 99.50th=[ 3306], 99.90th=[ 3306], 99.95th=[ 3306], | 99.99th=[ 3306] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=2.96%, ctx=36, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4469: Wed Dec 25 00:55:09 2024 write: IOPS=0, BW=2341KiB/s (2397kB/s)(12.0MiB/5250msec) clat (msec): min=261, max=3205, avg=1739.54, stdev=1471.97 lat (msec): min=261, max=3205, avg=1739.77, stdev=1472.02 clat percentiles (msec): | 1.00th=[ 262], 5.00th=[ 262], 10.00th=[ 262], 20.00th=[ 262], | 30.00th=[ 262], 40.00th=[ 1754], 50.00th=[ 1754], 60.00th=[ 1754], | 70.00th=[ 3205], 80.00th=[ 3205], 90.00th=[ 3205], 95.00th=[ 3205], | 99.00th=[ 3205], 99.50th=[ 3205], 99.90th=[ 3205], 99.95th=[ 3205], | 99.99th=[ 3205] bw ( KiB/s): min= 5527, max= 5535, per=88.62%, avg=5531.00, stdev= 5.66, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=1.96%, ctx=38, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4472: Wed Dec 25 00:55:09 2024 write: IOPS=6, BW=25.2MiB/s (26.4MB/s)(12.0MiB/477msec) clat (msec): min=100, max=259, avg=154.48, stdev=90.78 lat (msec): min=100, max=259, avg=154.76, stdev=90.77 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 101], 10.00th=[ 101], 20.00th=[ 101], | 30.00th=[ 101], 40.00th=[ 105], 50.00th=[ 105], 60.00th=[ 105], | 70.00th=[ 259], 80.00th=[ 259], 90.00th=[ 259], 95.00th=[ 259], | 99.00th=[ 259], 99.50th=[ 259], 99.90th=[ 259], 99.95th=[ 259], | 99.99th=[ 259] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=15.13%, ctx=99, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3159KiB/s (3235kB/s), 1103KiB/s-2369KiB/s (1129kB/s-2426kB/s), io=16.0MiB (16.8MB), run=3715-5186msec WRITE: bw=6242KiB/s (6391kB/s), 2205KiB/s-25.2MiB/s (2258kB/s-26.4MB/s), io=32.0MiB (33.6MB), run=477-5250msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4467: Wed Dec 25 00:55:09 2024 read: IOPS=0, BW=2382KiB/s (2439kB/s)(12.0MiB/5159msec) clat (msec): min=141, max=3223, avg=1715.35, stdev=1542.15 lat (msec): min=141, max=3223, avg=1715.35, stdev=1542.15 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 142], 10.00th=[ 142], 20.00th=[ 142], | 30.00th=[ 142], 40.00th=[ 1787], 50.00th=[ 1787], 60.00th=[ 1787], | 70.00th=[ 3239], 80.00th=[ 3239], 90.00th=[ 3239], 95.00th=[ 3239], | 99.00th=[ 3239], 99.50th=[ 3239], 99.90th=[ 3239], 99.95th=[ 3239], | 99.99th=[ 3239] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.04%, sys=39.69%, ctx=1502, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4470: Wed Dec 25 00:55:09 2024 read: IOPS=0, BW=1124KiB/s (1151kB/s)(4096KiB/3645msec) clat (nsec): min=180061k, max=180061k, avg=180061483.00, stdev= 0.00 lat (nsec): min=180064k, max=180064k, avg=180063658.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 180], 10.00th=[ 180], 20.00th=[ 180], | 30.00th=[ 180], 40.00th=[ 180], 50.00th=[ 180], 60.00th=[ 180], | 70.00th=[ 180], 80.00th=[ 180], 90.00th=[ 180], 95.00th=[ 180], | 99.00th=[ 180], 99.50th=[ 180], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2247KiB/s (2301kB/s)(8192KiB/3645msec) clat (msec): min=135, max=3292, avg=1713.84, stdev=2231.89 lat (msec): min=135, max=3292, avg=1714.15, stdev=2231.87 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 3306], | 70.00th=[ 3306], 80.00th=[ 3306], 90.00th=[ 3306], 95.00th=[ 3306], | 99.00th=[ 3306], 99.50th=[ 3306], 99.90th=[ 3306], 99.95th=[ 3306], | 99.99th=[ 3306] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=8.81%, ctx=264, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4471: Wed Dec 25 00:55:09 2024 write: IOPS=0, BW=2317KiB/s (2372kB/s)(12.0MiB/5304msec) clat (msec): min=365, max=3418, avg=1763.79, stdev=1542.97 lat (msec): min=365, max=3418, avg=1763.97, stdev=1542.89 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 368], 10.00th=[ 368], 20.00th=[ 368], | 30.00th=[ 368], 40.00th=[ 1502], 50.00th=[ 1502], 60.00th=[ 1502], | 70.00th=[ 3406], 80.00th=[ 3406], 90.00th=[ 3406], 95.00th=[ 3406], | 99.00th=[ 3406], 99.50th=[ 3406], 99.90th=[ 3406], 99.95th=[ 3406], | 99.99th=[ 3406] bw ( KiB/s): min= 5712, max= 8192, per=100.00%, avg=6952.00, stdev=1753.62, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33% cpu : usr=0.02%, sys=5.15%, ctx=53, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4473: Wed Dec 25 00:55:09 2024 write: IOPS=0, BW=3427KiB/s (3509kB/s)(12.0MiB/3586msec) clat (msec): min=189, max=3176, avg=1189.30, stdev=1720.92 lat (msec): min=189, max=3176, avg=1189.57, stdev=1720.94 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 190], 10.00th=[ 190], 20.00th=[ 190], | 30.00th=[ 190], 40.00th=[ 203], 50.00th=[ 203], 60.00th=[ 203], | 70.00th=[ 3171], 80.00th=[ 3171], 90.00th=[ 3171], 95.00th=[ 3171], | 99.00th=[ 3171], 99.50th=[ 3171], 99.90th=[ 3171], 99.95th=[ 3171], | 99.99th=[ 3171] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=6.56%, ctx=89, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3176KiB/s (3252kB/s), 1124KiB/s-2382KiB/s (1151kB/s-2439kB/s), io=16.0MiB (16.8MB), run=3645-5159msec WRITE: bw=6178KiB/s (6326kB/s), 2247KiB/s-3427KiB/s (2301kB/s-3509kB/s), io=32.0MiB (33.6MB), run=3586-5304msec