== sanity test 398b: DIO and buffer IO race ============== 15:51:59 (1733172719) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.50102 s, 33.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12650: Mon Dec 2 15:52:17 2024 read: IOPS=98, BW=394KiB/s (404kB/s)(5992KiB/15205msec) clat (usec): min=49, max=199117, avg=5870.52, stdev=5973.28 lat (usec): min=49, max=199119, avg=5872.93, stdev=5973.53 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 2376], 10.00th=[ 2900], 20.00th=[ 3425], | 30.00th=[ 3949], 40.00th=[ 4490], 50.00th=[ 5145], 60.00th=[ 5997], | 70.00th=[ 6718], 80.00th=[ 7570], 90.00th=[ 9241], 95.00th=[ 11338], | 99.00th=[ 17957], 99.50th=[ 19268], 99.90th=[ 32637], 99.95th=[198181], | 99.99th=[198181] bw ( KiB/s): min= 176, max= 512, per=25.19%, avg=393.50, stdev=65.52, samples=30 iops : min= 44, max= 128, avg=98.30, stdev=16.37, samples=30 write: IOPS=103, BW=414KiB/s (424kB/s)(6296KiB/15205msec) clat (usec): min=457, max=28199, avg=4021.13, stdev=3771.24 lat (usec): min=458, max=28200, avg=4022.86, stdev=3770.67 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 644], 10.00th=[ 701], 20.00th=[ 758], | 30.00th=[ 898], 40.00th=[ 1680], 50.00th=[ 3163], 60.00th=[ 3949], | 70.00th=[ 5080], 80.00th=[ 6980], 90.00th=[ 9110], 95.00th=[11076], | 99.00th=[16581], 99.50th=[19268], 99.90th=[24511], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 200, max= 576, per=25.18%, avg=411.63, stdev=87.84, samples=30 iops : min= 50, max= 144, avg=102.83, stdev=21.94, samples=30 lat (usec) : 50=0.03%, 100=1.50%, 250=0.59%, 500=0.13%, 750=9.34% lat (usec) : 1000=7.62% lat (msec) : 2=4.04%, 4=23.21%, 10=46.00%, 20=7.16%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=22.95%, ctx=6100, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12651: Mon Dec 2 15:52:17 2024 read: IOPS=96, BW=387KiB/s (397kB/s)(5940KiB/15339msec) clat (usec): min=45, max=206961, avg=5761.13, stdev=6116.17 lat (usec): min=45, max=206963, avg=5764.35, stdev=6116.75 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 159], 10.00th=[ 2802], 20.00th=[ 3425], | 30.00th=[ 3851], 40.00th=[ 4359], 50.00th=[ 5014], 60.00th=[ 5735], | 70.00th=[ 6652], 80.00th=[ 7767], 90.00th=[ 9241], 95.00th=[ 11207], | 99.00th=[ 16909], 99.50th=[ 19268], 99.90th=[ 26346], 99.95th=[206570], | 99.99th=[206570] bw ( KiB/s): min= 112, max= 520, per=24.57%, avg=383.83, stdev=75.32, samples=30 iops : min= 28, max= 130, avg=95.83, stdev=18.78, samples=30 write: IOPS=103, BW=414KiB/s (424kB/s)(6348KiB/15339msec) clat (usec): min=378, max=45381, avg=4231.50, stdev=3862.62 lat (usec): min=379, max=45382, avg=4232.95, stdev=3862.20 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 644], 10.00th=[ 693], 20.00th=[ 775], | 30.00th=[ 979], 40.00th=[ 2474], 50.00th=[ 3458], 60.00th=[ 4228], | 70.00th=[ 5407], 80.00th=[ 7111], 90.00th=[ 9634], 95.00th=[11338], | 99.00th=[15533], 99.50th=[19006], 99.90th=[26608], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 200, max= 558, per=24.81%, avg=405.67, stdev=69.56, samples=30 iops : min= 50, max= 139, avg=101.30, stdev=17.35, samples=30 lat (usec) : 50=0.10%, 100=1.63%, 250=0.81%, 500=0.26%, 750=8.95% lat (usec) : 1000=6.54% lat (msec) : 2=4.10%, 4=23.37%, 10=46.29%, 20=7.55%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=22.67%, ctx=5943, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12652: Mon Dec 2 15:52:17 2024 read: IOPS=98, BW=392KiB/s (402kB/s)(6028KiB/15372msec) clat (usec): min=49, max=33613, avg=5629.95, stdev=3247.10 lat (usec): min=49, max=33614, avg=5632.32, stdev=3248.26 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 111], 10.00th=[ 2802], 20.00th=[ 3359], | 30.00th=[ 3884], 40.00th=[ 4490], 50.00th=[ 5211], 60.00th=[ 5866], | 70.00th=[ 6718], 80.00th=[ 7701], 90.00th=[ 9241], 95.00th=[10814], | 99.00th=[17957], 99.50th=[20841], 99.90th=[23987], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 152, max= 528, per=24.82%, avg=387.63, stdev=68.28, samples=30 iops : min= 38, max= 132, avg=96.83, stdev=17.11, samples=30 write: IOPS=101, BW=407KiB/s (417kB/s)(6260KiB/15372msec) clat (usec): min=483, max=201734, avg=4351.39, stdev=6293.04 lat (usec): min=484, max=201737, avg=4352.71, stdev=6292.89 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 644], 10.00th=[ 693], 20.00th=[ 775], | 30.00th=[ 947], 40.00th=[ 2409], 50.00th=[ 3425], 60.00th=[ 4293], | 70.00th=[ 5473], 80.00th=[ 7177], 90.00th=[ 9503], 95.00th=[ 11863], | 99.00th=[ 15926], 99.50th=[ 19268], 99.90th=[ 35914], 99.95th=[202376], | 99.99th=[202376] bw ( KiB/s): min= 176, max= 568, per=24.46%, avg=399.90, stdev=81.21, samples=30 iops : min= 44, max= 142, avg=99.90, stdev=20.26, samples=30 lat (usec) : 50=0.03%, 100=2.15%, 250=0.85%, 500=0.20%, 750=8.50% lat (usec) : 1000=7.36% lat (msec) : 2=3.84%, 4=21.94%, 10=47.40%, 20=7.29%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=22.56%, ctx=6075, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12653: Mon Dec 2 15:52:17 2024 read: IOPS=98, BW=396KiB/s (405kB/s)(6052KiB/15290msec) clat (usec): min=50, max=31740, avg=5679.05, stdev=3063.37 lat (usec): min=50, max=31740, avg=5680.00, stdev=3063.43 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 2474], 10.00th=[ 2999], 20.00th=[ 3458], | 30.00th=[ 3949], 40.00th=[ 4424], 50.00th=[ 5145], 60.00th=[ 5866], | 70.00th=[ 6652], 80.00th=[ 7635], 90.00th=[ 8979], 95.00th=[10814], | 99.00th=[16188], 99.50th=[17171], 99.90th=[29492], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 208, max= 496, per=25.16%, avg=392.93, stdev=65.52, samples=30 iops : min= 52, max= 124, avg=98.13, stdev=16.33, samples=30 write: IOPS=101, BW=408KiB/s (418kB/s)(6236KiB/15290msec) clat (usec): min=461, max=215980, avg=4265.97, stdev=6569.01 lat (usec): min=462, max=215981, avg=4267.19, stdev=6569.11 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 652], 10.00th=[ 701], 20.00th=[ 775], | 30.00th=[ 930], 40.00th=[ 2008], 50.00th=[ 3326], 60.00th=[ 4178], | 70.00th=[ 5211], 80.00th=[ 6915], 90.00th=[ 9503], 95.00th=[ 11600], | 99.00th=[ 16319], 99.50th=[ 18744], 99.90th=[ 27395], 99.95th=[214959], | 99.99th=[214959] bw ( KiB/s): min= 224, max= 528, per=24.75%, avg=404.63, stdev=72.09, samples=30 iops : min= 56, max= 132, avg=101.07, stdev=18.00, samples=30 lat (usec) : 100=1.40%, 250=0.39%, 500=0.29%, 750=8.43%, 1000=7.55% lat (msec) : 2=4.07%, 4=22.72%, 10=47.27%, 20=7.49%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=22.81%, ctx=6013, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1562KiB/s (1600kB/s), 387KiB/s-396KiB/s (397kB/s-405kB/s), io=23.4MiB (24.6MB), run=15205-15372msec WRITE: bw=1635KiB/s (1675kB/s), 407KiB/s-414KiB/s (417kB/s-424kB/s), io=24.6MiB (25.7MB), run=15205-15372msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12645: Mon Dec 2 15:52:22 2024 read: IOPS=75, BW=304KiB/s (311kB/s)(5992KiB/19731msec) clat (usec): min=1613, max=225128, avg=5152.36, stdev=6967.92 lat (usec): min=1614, max=225130, avg=5153.11, stdev=6967.96 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2008], 10.00th=[ 2114], 20.00th=[ 2343], | 30.00th=[ 2573], 40.00th=[ 2868], 50.00th=[ 3359], 60.00th=[ 3949], | 70.00th=[ 5014], 80.00th=[ 6915], 90.00th=[ 11338], 95.00th=[ 13435], | 99.00th=[ 19006], 99.50th=[ 24249], 99.90th=[ 41681], 99.95th=[225444], | 99.99th=[225444] bw ( KiB/s): min= 96, max= 744, per=24.67%, avg=299.97, stdev=159.99, samples=39 iops : min= 24, max= 186, avg=74.92, stdev=40.01, samples=39 write: IOPS=79, BW=319KiB/s (327kB/s)(6296KiB/19731msec) clat (usec): min=2582, max=34713, avg=7582.69, stdev=4926.16 lat (usec): min=2583, max=34713, avg=7583.73, stdev=4926.17 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3261], 10.00th=[ 3490], 20.00th=[ 3884], | 30.00th=[ 4359], 40.00th=[ 4817], 50.00th=[ 5538], 60.00th=[ 6849], | 70.00th=[ 8848], 80.00th=[10945], 90.00th=[14353], 95.00th=[17171], | 99.00th=[26084], 99.50th=[30278], 99.90th=[34341], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 64, max= 680, per=24.75%, avg=315.36, stdev=161.82, samples=39 iops : min= 16, max= 170, avg=78.77, stdev=40.47, samples=39 lat (msec) : 2=2.38%, 4=38.48%, 10=40.89%, 20=16.44%, 50=1.79% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=10.71%, ctx=3567, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12646: Mon Dec 2 15:52:22 2024 read: IOPS=75, BW=303KiB/s (310kB/s)(5940KiB/19633msec) clat (usec): min=1683, max=220620, avg=5096.87, stdev=6863.14 lat (usec): min=1683, max=220621, avg=5097.63, stdev=6863.16 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2040], 10.00th=[ 2180], 20.00th=[ 2376], | 30.00th=[ 2606], 40.00th=[ 2868], 50.00th=[ 3261], 60.00th=[ 3851], | 70.00th=[ 4686], 80.00th=[ 6718], 90.00th=[ 11207], 95.00th=[ 14353], | 99.00th=[ 19530], 99.50th=[ 21365], 99.90th=[ 27657], 99.95th=[221250], | 99.99th=[221250] bw ( KiB/s): min= 48, max= 688, per=24.79%, avg=301.41, stdev=162.34, samples=39 iops : min= 12, max= 172, avg=75.26, stdev=40.63, samples=39 write: IOPS=80, BW=323KiB/s (331kB/s)(6348KiB/19633msec) clat (usec): min=2279, max=54251, avg=7552.33, stdev=4880.13 lat (usec): min=2280, max=54252, avg=7553.30, stdev=4880.21 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3261], 10.00th=[ 3556], 20.00th=[ 3949], | 30.00th=[ 4293], 40.00th=[ 4948], 50.00th=[ 5604], 60.00th=[ 6849], | 70.00th=[ 8717], 80.00th=[11076], 90.00th=[14222], 95.00th=[16450], | 99.00th=[25560], 99.50th=[30802], 99.90th=[34866], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 104, max= 664, per=25.17%, avg=320.69, stdev=154.72, samples=39 iops : min= 26, max= 166, avg=80.08, stdev=38.74, samples=39 lat (msec) : 2=1.92%, 4=39.13%, 10=40.20%, 20=17.19%, 50=1.50% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.12%, sys=10.82%, ctx=3580, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12647: Mon Dec 2 15:52:22 2024 read: IOPS=76, BW=306KiB/s (313kB/s)(6028KiB/19724msec) clat (usec): min=1751, max=28358, avg=4972.74, stdev=3948.17 lat (usec): min=1752, max=28359, avg=4973.66, stdev=3948.30 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2040], 10.00th=[ 2147], 20.00th=[ 2343], | 30.00th=[ 2573], 40.00th=[ 2835], 50.00th=[ 3261], 60.00th=[ 3916], | 70.00th=[ 4948], 80.00th=[ 7046], 90.00th=[11076], 95.00th=[13698], | 99.00th=[18744], 99.50th=[21103], 99.90th=[27395], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 96, max= 696, per=24.91%, avg=302.90, stdev=154.94, samples=39 iops : min= 24, max= 174, avg=75.67, stdev=38.77, samples=39 write: IOPS=79, BW=317KiB/s (325kB/s)(6260KiB/19724msec) clat (msec): min=2, max=222, avg= 7.77, stdev= 7.36 lat (msec): min=2, max=222, avg= 7.77, stdev= 7.36 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 12], 90.00th=[ 15], 95.00th=[ 18], | 99.00th=[ 27], 99.50th=[ 31], 99.90th=[ 39], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 88, max= 696, per=24.61%, avg=313.56, stdev=165.08, samples=39 iops : min= 22, max= 174, avg=78.33, stdev=41.29, samples=39 lat (msec) : 2=1.56%, 4=39.94%, 10=39.78%, 20=17.02%, 50=1.66% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=10.59%, ctx=3585, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12648: Mon Dec 2 15:52:22 2024 read: IOPS=76, BW=307KiB/s (315kB/s)(6052KiB/19703msec) clat (usec): min=1723, max=26441, avg=5048.65, stdev=4013.66 lat (usec): min=1724, max=26442, avg=5049.46, stdev=4013.64 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2024], 10.00th=[ 2147], 20.00th=[ 2343], | 30.00th=[ 2606], 40.00th=[ 2900], 50.00th=[ 3294], 60.00th=[ 3884], | 70.00th=[ 5014], 80.00th=[ 7308], 90.00th=[11207], 95.00th=[13960], | 99.00th=[19268], 99.50th=[20841], 99.90th=[24511], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 96, max= 800, per=25.01%, avg=304.08, stdev=161.48, samples=39 iops : min= 24, max= 200, avg=75.95, stdev=40.38, samples=39 write: IOPS=79, BW=317KiB/s (324kB/s)(6236KiB/19703msec) clat (msec): min=2, max=222, avg= 7.69, stdev= 7.19 lat (msec): min=2, max=222, avg= 7.69, stdev= 7.19 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 8], | 70.00th=[ 9], 80.00th=[ 12], 90.00th=[ 15], 95.00th=[ 17], | 99.00th=[ 25], 99.50th=[ 30], 99.90th=[ 35], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 104, max= 720, per=24.59%, avg=313.31, stdev=160.22, samples=39 iops : min= 26, max= 180, avg=78.26, stdev=40.06, samples=39 lat (msec) : 2=1.99%, 4=39.94%, 10=39.65%, 20=17.12%, 50=1.27% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=10.65%, ctx=3594, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1217KiB/s (1246kB/s), 303KiB/s-307KiB/s (310kB/s-315kB/s), io=23.4MiB (24.6MB), run=19633-19731msec WRITE: bw=1274KiB/s (1305kB/s), 317KiB/s-323KiB/s (324kB/s-331kB/s), io=24.6MiB (25.7MB), run=19633-19731msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12680: Mon Dec 2 15:52:30 2024 read: IOPS=49, BW=796KiB/s (816kB/s)(5920KiB/7433msec) clat (usec): min=62, max=37337, avg=14211.98, stdev=4995.97 lat (usec): min=62, max=37338, avg=14212.78, stdev=4995.98 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 652], 10.00th=[10159], 20.00th=[11731], | 30.00th=[12649], 40.00th=[13435], 50.00th=[14484], 60.00th=[15270], | 70.00th=[16057], 80.00th=[17171], 90.00th=[18744], 95.00th=[21103], | 99.00th=[30540], 99.50th=[33424], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 672, max= 992, per=24.69%, avg=781.57, stdev=104.16, samples=14 iops : min= 42, max= 62, avg=48.79, stdev= 6.57, samples=14 write: IOPS=53, BW=857KiB/s (877kB/s)(6368KiB/7433msec) clat (usec): min=767, max=27817, avg=5347.33, stdev=4609.47 lat (usec): min=769, max=27818, avg=5348.79, stdev=4609.47 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 971], 10.00th=[ 1045], 20.00th=[ 1237], | 30.00th=[ 2966], 40.00th=[ 3458], 50.00th=[ 3982], 60.00th=[ 4883], | 70.00th=[ 6325], 80.00th=[ 7635], 90.00th=[10552], 95.00th=[16319], | 99.00th=[23200], 99.50th=[24511], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 640, max= 1149, per=25.45%, avg=847.79, stdev=154.67, samples=14 iops : min= 40, max= 71, avg=52.93, stdev= 9.55, samples=14 lat (usec) : 100=1.43%, 250=0.78%, 750=0.26%, 1000=3.39% lat (msec) : 2=8.85%, 4=14.45%, 10=20.96%, 20=45.83%, 50=4.04% cpu : usr=0.23%, sys=18.65%, ctx=2655, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12682: Mon Dec 2 15:52:30 2024 read: IOPS=48, BW=773KiB/s (791kB/s)(5744KiB/7433msec) clat (usec): min=75, max=33156, avg=14586.77, stdev=4397.48 lat (usec): min=75, max=33157, avg=14587.58, stdev=4397.46 clat percentiles (usec): | 1.00th=[ 98], 5.00th=[ 9765], 10.00th=[10683], 20.00th=[11863], | 30.00th=[12911], 40.00th=[13566], 50.00th=[14353], 60.00th=[15270], | 70.00th=[16450], 80.00th=[17433], 90.00th=[19530], 95.00th=[21365], | 99.00th=[26608], 99.50th=[27919], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 544, max= 896, per=24.03%, avg=760.71, stdev=108.71, samples=14 iops : min= 34, max= 56, avg=47.36, stdev= 6.76, samples=14 write: IOPS=55, BW=880KiB/s (902kB/s)(6544KiB/7433msec) clat (usec): min=860, max=40423, avg=5246.35, stdev=5176.85 lat (usec): min=861, max=40424, avg=5248.80, stdev=5176.07 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 1037], 10.00th=[ 1090], 20.00th=[ 1221], | 30.00th=[ 2573], 40.00th=[ 3294], 50.00th=[ 3752], 60.00th=[ 4555], | 70.00th=[ 5800], 80.00th=[ 7177], 90.00th=[ 9765], 95.00th=[16450], | 99.00th=[23987], 99.50th=[28443], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 576, max= 1184, per=25.65%, avg=854.43, stdev=188.64, samples=14 iops : min= 36, max= 74, avg=53.21, stdev=11.88, samples=14 lat (usec) : 100=0.52%, 250=0.78%, 1000=1.56% lat (msec) : 2=12.63%, 4=15.76%, 10=19.79%, 20=43.75%, 50=5.21% cpu : usr=0.17%, sys=18.47%, ctx=2654, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12684: Mon Dec 2 15:52:30 2024 read: IOPS=51, BW=827KiB/s (847kB/s)(6256KiB/7564msec) clat (usec): min=61, max=34302, avg=14203.21, stdev=5315.83 lat (usec): min=62, max=34303, avg=14205.46, stdev=5312.28 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 112], 10.00th=[ 9896], 20.00th=[11600], | 30.00th=[12649], 40.00th=[13566], 50.00th=[14353], 60.00th=[15270], | 70.00th=[16057], 80.00th=[17433], 90.00th=[19792], 95.00th=[22414], | 99.00th=[31589], 99.50th=[34341], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 672, max= 1056, per=25.54%, avg=808.57, stdev=106.41, samples=14 iops : min= 42, max= 66, avg=50.29, stdev= 6.81, samples=14 write: IOPS=49, BW=797KiB/s (817kB/s)(6032KiB/7564msec) clat (usec): min=797, max=41581, avg=5285.57, stdev=5289.11 lat (usec): min=798, max=41582, avg=5288.70, stdev=5288.47 clat percentiles (usec): | 1.00th=[ 857], 5.00th=[ 971], 10.00th=[ 1037], 20.00th=[ 1156], | 30.00th=[ 2704], 40.00th=[ 3261], 50.00th=[ 3785], 60.00th=[ 4359], | 70.00th=[ 5538], 80.00th=[ 7832], 90.00th=[10945], 95.00th=[17171], | 99.00th=[25560], 99.50th=[37487], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 576, max= 1152, per=23.11%, avg=769.64, stdev=183.45, samples=14 iops : min= 36, max= 72, avg=47.86, stdev=11.37, samples=14 lat (usec) : 100=1.95%, 250=1.30%, 1000=3.26% lat (msec) : 2=10.03%, 4=13.28%, 10=18.88%, 20=45.57%, 50=5.73% cpu : usr=0.17%, sys=18.60%, ctx=2695, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12686: Mon Dec 2 15:52:30 2024 read: IOPS=51, BW=826KiB/s (846kB/s)(6032KiB/7301msec) clat (usec): min=72, max=33128, avg=14428.35, stdev=4974.91 lat (usec): min=72, max=33128, avg=14429.19, stdev=4974.92 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 3818], 10.00th=[10290], 20.00th=[11600], | 30.00th=[12911], 40.00th=[13566], 50.00th=[14222], 60.00th=[15139], | 70.00th=[16188], 80.00th=[17433], 90.00th=[19530], 95.00th=[22152], | 99.00th=[29230], 99.50th=[32637], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 640, max= 1056, per=25.49%, avg=806.86, stdev=105.89, samples=14 iops : min= 40, max= 66, avg=50.43, stdev= 6.62, samples=14 write: IOPS=53, BW=857KiB/s (877kB/s)(6256KiB/7301msec) clat (usec): min=783, max=35369, avg=4620.80, stdev=4546.56 lat (usec): min=784, max=35370, avg=4622.41, stdev=4546.55 clat percentiles (usec): | 1.00th=[ 832], 5.00th=[ 979], 10.00th=[ 1057], 20.00th=[ 1139], | 30.00th=[ 1500], 40.00th=[ 3032], 50.00th=[ 3425], 60.00th=[ 3982], | 70.00th=[ 4752], 80.00th=[ 6849], 90.00th=[ 9765], 95.00th=[13698], | 99.00th=[23200], 99.50th=[29230], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 544, max= 1120, per=25.32%, avg=843.43, stdev=148.83, samples=14 iops : min= 34, max= 70, avg=52.71, stdev= 9.30, samples=14 lat (usec) : 100=0.91%, 250=1.17%, 500=0.26%, 1000=2.99% lat (msec) : 2=14.06%, 4=14.06%, 10=16.54%, 20=45.31%, 50=4.69% cpu : usr=0.14%, sys=19.05%, ctx=2621, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3167KiB/s (3243kB/s), 773KiB/s-827KiB/s (791kB/s-847kB/s), io=23.4MiB (24.5MB), run=7301-7564msec WRITE: bw=3332KiB/s (3412kB/s), 797KiB/s-880KiB/s (817kB/s-902kB/s), io=24.6MiB (25.8MB), run=7301-7564msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12679: Mon Dec 2 15:52:30 2024 read: IOPS=49, BW=791KiB/s (810kB/s)(5920KiB/7481msec) clat (usec): min=2310, max=37426, avg=7517.08, stdev=5826.19 lat (usec): min=2311, max=37427, avg=7517.94, stdev=5826.22 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2769], 10.00th=[ 2933], 20.00th=[ 3359], | 30.00th=[ 3687], 40.00th=[ 4228], 50.00th=[ 5014], 60.00th=[ 6259], | 70.00th=[ 8586], 80.00th=[11600], 90.00th=[15139], 95.00th=[19268], | 99.00th=[30802], 99.50th=[33817], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 510, max= 1056, per=24.83%, avg=779.43, stdev=144.92, samples=14 iops : min= 31, max= 66, avg=48.50, stdev= 9.10, samples=14 write: IOPS=53, BW=851KiB/s (872kB/s)(6368KiB/7481msec) clat (usec): min=3287, max=40669, avg=11717.57, stdev=6241.80 lat (usec): min=3289, max=40670, avg=11719.22, stdev=6241.72 clat percentiles (usec): | 1.00th=[ 3589], 5.00th=[ 4948], 10.00th=[ 5473], 20.00th=[ 6521], | 30.00th=[ 7701], 40.00th=[ 8848], 50.00th=[10028], 60.00th=[11600], | 70.00th=[13304], 80.00th=[15533], 90.00th=[20579], 95.00th=[24511], | 99.00th=[31589], 99.50th=[37487], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 672, max= 1056, per=25.41%, avg=839.21, stdev=136.07, samples=14 iops : min= 42, max= 66, avg=52.21, stdev= 8.60, samples=14 lat (msec) : 4=18.75%, 10=42.71%, 20=31.38%, 50=7.16% cpu : usr=0.09%, sys=9.39%, ctx=1086, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12681: Mon Dec 2 15:52:30 2024 read: IOPS=52, BW=842KiB/s (862kB/s)(5744KiB/6825msec) clat (usec): min=2050, max=36505, avg=6724.33, stdev=4932.12 lat (usec): min=2051, max=36506, avg=6725.15, stdev=4932.16 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2769], 10.00th=[ 2999], 20.00th=[ 3425], | 30.00th=[ 3785], 40.00th=[ 4293], 50.00th=[ 4752], 60.00th=[ 5800], | 70.00th=[ 7111], 80.00th=[ 9634], 90.00th=[13829], 95.00th=[16057], | 99.00th=[24773], 99.50th=[36439], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 672, max= 1152, per=27.04%, avg=848.92, stdev=141.57, samples=13 iops : min= 42, max= 72, avg=52.92, stdev= 8.81, samples=13 write: IOPS=59, BW=959KiB/s (982kB/s)(6544KiB/6825msec) clat (usec): min=3585, max=48703, avg=10713.81, stdev=6103.62 lat (usec): min=3587, max=48707, avg=10715.90, stdev=6103.53 clat percentiles (usec): | 1.00th=[ 3949], 5.00th=[ 4490], 10.00th=[ 5014], 20.00th=[ 5735], | 30.00th=[ 6587], 40.00th=[ 7832], 50.00th=[ 8848], 60.00th=[10290], | 70.00th=[12387], 80.00th=[14746], 90.00th=[19530], 95.00th=[22676], | 99.00th=[29230], 99.50th=[32637], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 768, max= 1088, per=28.76%, avg=949.85, stdev=105.10, samples=13 iops : min= 48, max= 68, avg=59.23, stdev= 6.70, samples=13 lat (msec) : 4=17.84%, 10=50.52%, 20=25.39%, 50=6.25% cpu : usr=0.04%, sys=9.06%, ctx=930, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12683: Mon Dec 2 15:52:30 2024 read: IOPS=51, BW=820KiB/s (840kB/s)(6256KiB/7629msec) clat (usec): min=2241, max=41550, avg=7676.99, stdev=5624.42 lat (usec): min=2242, max=41550, avg=7677.79, stdev=5624.44 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2802], 10.00th=[ 3097], 20.00th=[ 3359], | 30.00th=[ 3851], 40.00th=[ 4424], 50.00th=[ 5342], 60.00th=[ 6849], | 70.00th=[ 9372], 80.00th=[11469], 90.00th=[15008], 95.00th=[19268], | 99.00th=[30016], 99.50th=[31851], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 640, max= 1037, per=25.80%, avg=809.93, stdev=118.29, samples=15 iops : min= 40, max= 64, avg=50.40, stdev= 7.29, samples=15 write: IOPS=49, BW=791KiB/s (810kB/s)(6032KiB/7629msec) clat (usec): min=3459, max=47253, avg=12235.57, stdev=6579.86 lat (usec): min=3461, max=47255, avg=12237.25, stdev=6579.95 clat percentiles (usec): | 1.00th=[ 4047], 5.00th=[ 4817], 10.00th=[ 5473], 20.00th=[ 6718], | 30.00th=[ 7832], 40.00th=[ 9634], 50.00th=[10814], 60.00th=[12649], | 70.00th=[14091], 80.00th=[16188], 90.00th=[21103], 95.00th=[26346], | 99.00th=[34341], 99.50th=[39584], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 512, max= 992, per=23.40%, avg=772.93, stdev=150.64, samples=15 iops : min= 32, max= 62, avg=48.13, stdev= 9.52, samples=15 lat (msec) : 4=16.54%, 10=41.80%, 20=33.85%, 50=7.81% cpu : usr=0.20%, sys=9.62%, ctx=1148, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12685: Mon Dec 2 15:52:30 2024 read: IOPS=57, BW=924KiB/s (946kB/s)(6032KiB/6531msec) clat (usec): min=2270, max=36623, avg=6960.17, stdev=4846.92 lat (usec): min=2272, max=36624, avg=6961.12, stdev=4847.11 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2737], 10.00th=[ 2933], 20.00th=[ 3261], | 30.00th=[ 3687], 40.00th=[ 4293], 50.00th=[ 4948], 60.00th=[ 6390], | 70.00th=[ 8160], 80.00th=[10421], 90.00th=[13435], 95.00th=[16712], | 99.00th=[22938], 99.50th=[28443], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 640, max= 1088, per=29.24%, avg=917.69, stdev=125.79, samples=13 iops : min= 40, max= 68, avg=57.15, stdev= 7.97, samples=13 write: IOPS=59, BW=958KiB/s (981kB/s)(6256KiB/6531msec) clat (usec): min=3672, max=31829, avg=9884.85, stdev=5743.90 lat (usec): min=3674, max=31830, avg=9886.75, stdev=5744.14 clat percentiles (usec): | 1.00th=[ 3851], 5.00th=[ 4424], 10.00th=[ 4817], 20.00th=[ 5407], | 30.00th=[ 6063], 40.00th=[ 6521], 50.00th=[ 7701], 60.00th=[ 9503], | 70.00th=[10945], 80.00th=[14222], 90.00th=[19268], 95.00th=[21890], | 99.00th=[28967], 99.50th=[30016], 99.90th=[31851], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 736, max= 1120, per=28.98%, avg=957.08, stdev=102.77, samples=13 iops : min= 46, max= 70, avg=59.62, stdev= 6.59, samples=13 lat (msec) : 4=18.36%, 10=52.60%, 20=23.57%, 50=5.47% cpu : usr=0.25%, sys=9.45%, ctx=921, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3140KiB/s (3215kB/s), 791KiB/s-924KiB/s (810kB/s-946kB/s), io=23.4MiB (24.5MB), run=6531-7629msec WRITE: bw=3303KiB/s (3382kB/s), 791KiB/s-959KiB/s (810kB/s-982kB/s), io=24.6MiB (25.8MB), run=6531-7629msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12698: Mon Dec 2 15:52:34 2024 read: IOPS=2, BW=2337KiB/s (2393kB/s)(7168KiB/3067msec) clat (msec): min=10, max=642, avg=406.92, stdev=253.62 lat (msec): min=10, max=642, avg=406.92, stdev=253.62 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 88], | 30.00th=[ 460], 40.00th=[ 460], 50.00th=[ 514], 60.00th=[ 514], | 70.00th=[ 514], 80.00th=[ 617], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 1906, max= 4096, per=35.13%, avg=2345.60, stdev=978.51, samples=5 iops : min= 1, max= 4, avg= 1.60, stdev= 1.34, samples=5 write: IOPS=1, BW=1669KiB/s (1709kB/s)(5120KiB/3067msec) clat (usec): min=30874, max=51359, avg=40006.60, stdev=8302.96 lat (usec): min=30904, max=51394, avg=40062.99, stdev=8314.23 clat percentiles (usec): | 1.00th=[30802], 5.00th=[30802], 10.00th=[30802], 20.00th=[30802], | 30.00th=[34341], 40.00th=[34341], 50.00th=[38536], 60.00th=[38536], | 70.00th=[45351], 80.00th=[45351], 90.00th=[51119], 95.00th=[51119], | 99.00th=[51119], 99.50th=[51119], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 1910, max= 5720, per=34.02%, avg=3180.00, stdev=2199.70, samples=3 iops : min= 1, max= 5, avg= 2.33, stdev= 2.31, samples=3 lat (msec) : 20=8.33%, 50=33.33%, 100=16.67%, 500=8.33%, 750=33.33% cpu : usr=0.23%, sys=24.66%, ctx=1267, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12699: Mon Dec 2 15:52:34 2024 read: IOPS=1, BW=1314KiB/s (1345kB/s)(2048KiB/1559msec) clat (msec): min=76, max=183, avg=130.11, stdev=76.07 lat (msec): min=76, max=183, avg=130.11, stdev=76.07 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 77], 10.00th=[ 77], 20.00th=[ 77], | 30.00th=[ 77], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 184], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 1906, max= 2043, per=29.57%, avg=1974.50, stdev=96.87, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=6, BW=6568KiB/s (6726kB/s)(10.0MiB/1559msec) clat (msec): min=35, max=764, avg=122.90, stdev=225.52 lat (msec): min=35, max=764, avg=122.98, stdev=225.52 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 42], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 53], | 70.00th=[ 66], 80.00th=[ 66], 90.00th=[ 70], 95.00th=[ 768], | 99.00th=[ 768], 99.50th=[ 768], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 7627, max=10219, per=95.45%, avg=8923.00, stdev=1832.82, samples=2 iops : min= 7, max= 9, avg= 8.00, stdev= 1.41, samples=2 lat (msec) : 50=41.67%, 100=41.67%, 250=8.33%, 1000=8.33% cpu : usr=0.06%, sys=24.97%, ctx=127, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12701: Mon Dec 2 15:52:34 2024 read: IOPS=10, BW=10.5MiB/s (10.0MB/s)(6144KiB/572msec) clat (usec): min=476, max=97240, avg=38653.10, stdev=40366.60 lat (usec): min=478, max=97241, avg=38654.47, stdev=40366.36 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 478], 10.00th=[ 478], 20.00th=[ 5407], | 30.00th=[ 5407], 40.00th=[ 5800], 50.00th=[ 5800], 60.00th=[57934], | 70.00th=[65274], 80.00th=[65274], 90.00th=[96994], 95.00th=[96994], | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min=10219, max=10219, per=100.00%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.5MiB/s (10.0MB/s)(6144KiB/572msec) clat (usec): min=32592, max=55458, avg=44685.58, stdev=9504.67 lat (usec): min=32659, max=55527, avg=44739.70, stdev=9502.91 clat percentiles (usec): | 1.00th=[32637], 5.00th=[32637], 10.00th=[32637], 20.00th=[38011], | 30.00th=[38011], 40.00th=[39060], 50.00th=[39060], 60.00th=[49021], | 70.00th=[54264], 80.00th=[54264], 90.00th=[55313], 95.00th=[55313], | 99.00th=[55313], 99.50th=[55313], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min=10219, max=10219, per=100.00%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 10=16.67%, 50=33.33%, 100=41.67% cpu : usr=0.00%, sys=36.43%, ctx=96, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12703: Mon Dec 2 15:52:34 2024 read: IOPS=3, BW=3732KiB/s (3821kB/s)(5120KiB/1372msec) clat (usec): min=447, max=120941, avg=39640.97, stdev=55784.60 lat (usec): min=449, max=120942, avg=39642.37, stdev=55784.50 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 449], 10.00th=[ 449], 20.00th=[ 449], | 30.00th=[ 652], 40.00th=[ 652], 50.00th=[ 816], 60.00th=[ 816], | 70.00th=[ 74974], 80.00th=[ 74974], 90.00th=[121111], 95.00th=[121111], | 99.00th=[121111], 99.50th=[121111], 99.90th=[121111], 99.95th=[121111], | 99.99th=[121111] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5224KiB/s (5350kB/s)(7168KiB/1372msec) clat (msec): min=34, max=791, avg=163.25, stdev=277.81 lat (msec): min=34, max=791, avg=163.30, stdev=277.82 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 40], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 69], 60.00th=[ 79], | 70.00th=[ 79], 80.00th=[ 85], 90.00th=[ 793], 95.00th=[ 793], | 99.00th=[ 793], 99.50th=[ 793], 99.90th=[ 793], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 2048, max= 8192, per=54.77%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (usec) : 500=8.33%, 750=8.33%, 1000=8.33% lat (msec) : 50=25.00%, 100=33.33%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=15.54%, ctx=173, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6678KiB/s (6838kB/s), 1314KiB/s-10.5MiB/s (1345kB/s-10.0MB/s), io=20.0MiB (20.0MB), run=572-3067msec WRITE: bw=9349KiB/s (9573kB/s), 1669KiB/s-10.5MiB/s (1709kB/s-10.0MB/s), io=28.0MiB (29.4MB), run=572-3067msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12700: Mon Dec 2 15:52:34 2024 read: IOPS=2, BW=2285KiB/s (2340kB/s)(7168KiB/3137msec) clat (msec): min=17, max=748, avg=262.32, stdev=286.74 lat (msec): min=17, max=748, avg=262.32, stdev=286.74 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 19], | 30.00th=[ 60], 40.00th=[ 60], 50.00th=[ 100], 60.00th=[ 422], | 70.00th=[ 422], 80.00th=[ 477], 90.00th=[ 751], 95.00th=[ 751], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 1928, max= 4096, per=36.92%, avg=2410.40, stdev=944.13, samples=5 iops : min= 1, max= 4, avg= 2.00, stdev= 1.22, samples=5 write: IOPS=1, BW=1632KiB/s (1671kB/s)(5120KiB/3137msec) clat (msec): min=45, max=574, avg=257.74, stdev=269.88 lat (msec): min=45, max=574, avg=257.82, stdev=269.87 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 46], | 30.00th=[ 66], 40.00th=[ 66], 50.00th=[ 73], 60.00th=[ 73], | 70.00th=[ 531], 80.00th=[ 531], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 1928, max= 2048, per=21.44%, avg=1959.00, stdev=59.36, samples=4 iops : min= 1, max= 2, avg= 1.25, stdev= 0.50, samples=4 lat (msec) : 20=16.67%, 50=8.33%, 100=33.33%, 500=16.67%, 750=25.00% cpu : usr=0.03%, sys=4.59%, ctx=58, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12702: Mon Dec 2 15:52:34 2024 read: IOPS=1, BW=1275KiB/s (1306kB/s)(2048KiB/1606msec) clat (usec): min=46138, max=98485, avg=72311.60, stdev=37014.99 lat (usec): min=46138, max=98486, avg=72312.72, stdev=37015.26 clat percentiles (usec): | 1.00th=[46400], 5.00th=[46400], 10.00th=[46400], 20.00th=[46400], | 30.00th=[46400], 40.00th=[46400], 50.00th=[46400], 60.00th=[98042], | 70.00th=[98042], 80.00th=[98042], 90.00th=[98042], 95.00th=[98042], | 99.00th=[98042], 99.50th=[98042], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 2043, max= 2048, per=31.33%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=6, BW=6376KiB/s (6529kB/s)(10.0MiB/1606msec) clat (msec): min=45, max=765, avg=135.07, stdev=222.89 lat (msec): min=45, max=765, avg=135.21, stdev=222.87 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 47], | 30.00th=[ 49], 40.00th=[ 55], 50.00th=[ 57], 60.00th=[ 64], | 70.00th=[ 75], 80.00th=[ 75], 90.00th=[ 128], 95.00th=[ 768], | 99.00th=[ 768], 99.50th=[ 768], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 2048, max= 8175, per=59.47%, avg=5435.33, stdev=3114.42, samples=3 iops : min= 2, max= 7, avg= 4.67, stdev= 2.52, samples=3 lat (msec) : 50=33.33%, 100=50.00%, 250=8.33%, 1000=8.33% cpu : usr=0.19%, sys=10.59%, ctx=134, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12704: Mon Dec 2 15:52:34 2024 read: IOPS=4, BW=4504KiB/s (4613kB/s)(6144KiB/1364msec) clat (usec): min=17098, max=77492, avg=45866.34, stdev=24007.13 lat (usec): min=17099, max=77493, avg=45867.80, stdev=24006.79 clat percentiles (usec): | 1.00th=[17171], 5.00th=[17171], 10.00th=[17171], 20.00th=[18220], | 30.00th=[18220], 40.00th=[46924], 50.00th=[46924], 60.00th=[56886], | 70.00th=[58983], 80.00th=[58983], 90.00th=[77071], 95.00th=[77071], | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 4096, max= 4096, per=62.75%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4504KiB/s (4613kB/s)(6144KiB/1364msec) clat (msec): min=23, max=567, avg=163.57, stdev=207.04 lat (msec): min=23, max=567, avg=163.66, stdev=207.04 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 78], 50.00th=[ 78], 60.00th=[ 87], | 70.00th=[ 194], 80.00th=[ 194], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 2048, max= 6144, per=44.82%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=16.67%, 50=25.00%, 100=41.67%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=9.32%, ctx=125, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12705: Mon Dec 2 15:52:34 2024 read: IOPS=3, BW=3331KiB/s (3411kB/s)(5120KiB/1537msec) clat (usec): min=18441, max=70025, avg=41008.65, stdev=24950.62 lat (usec): min=18442, max=70026, avg=41010.18, stdev=24950.57 clat percentiles (usec): | 1.00th=[18482], 5.00th=[18482], 10.00th=[18482], 20.00th=[18482], | 30.00th=[19792], 40.00th=[19792], 50.00th=[31327], 60.00th=[31327], | 70.00th=[65274], 80.00th=[65274], 90.00th=[69731], 95.00th=[69731], | 99.00th=[69731], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 2048, max= 6144, per=62.75%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=4, BW=4664KiB/s (4776kB/s)(7168KiB/1537msec) clat (msec): min=35, max=857, avg=174.70, stdev=301.64 lat (msec): min=35, max=857, avg=174.79, stdev=301.63 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 42], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 64], 60.00th=[ 70], | 70.00th=[ 70], 80.00th=[ 99], 90.00th=[ 860], 95.00th=[ 860], | 99.00th=[ 860], 99.50th=[ 860], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 2048, max= 8192, per=52.26%, avg=4775.67, stdev=3129.36, samples=3 iops : min= 2, max= 8, avg= 4.33, stdev= 3.21, samples=3 lat (msec) : 20=16.67%, 50=25.00%, 100=50.00%, 1000=8.33% cpu : usr=0.00%, sys=10.48%, ctx=93, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6529KiB/s (6685kB/s), 1275KiB/s-4504KiB/s (1306kB/s-4613kB/s), io=20.0MiB (20.0MB), run=1364-3137msec WRITE: bw=9140KiB/s (9359kB/s), 1632KiB/s-6376KiB/s (1671kB/s-6529kB/s), io=28.0MiB (29.4MB), run=1364-3137msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12718: Mon Dec 2 15:52:39 2024 read: IOPS=1, BW=4140KiB/s (4240kB/s)(12.0MiB/2968msec) clat (msec): min=171, max=1578, avg=975.03, stdev=724.54 lat (msec): min=171, max=1578, avg=975.03, stdev=724.54 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 1183], 50.00th=[ 1183], 60.00th=[ 1183], | 70.00th=[ 1586], 80.00th=[ 1586], 90.00th=[ 1586], 95.00th=[ 1586], | 99.00th=[ 1586], 99.50th=[ 1586], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 7684, max= 8192, per=100.00%, avg=7938.00, stdev=359.21, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.13%, sys=47.76%, ctx=988, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12719: Mon Dec 2 15:52:39 2024 read: IOPS=0, BW=1697KiB/s (1738kB/s)(4096KiB/2413msec) clat (nsec): min=154219k, max=154219k, avg=154219134.00, stdev= 0.00 lat (nsec): min=154221k, max=154221k, avg=154220901.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 155], | 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3395KiB/s (3476kB/s)(8192KiB/2413msec) clat (msec): min=384, max=1849, avg=1116.66, stdev=1036.09 lat (msec): min=384, max=1849, avg=1117.00, stdev=1036.09 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 384], 10.00th=[ 384], 20.00th=[ 384], | 30.00th=[ 384], 40.00th=[ 384], 50.00th=[ 384], 60.00th=[ 1854], | 70.00th=[ 1854], 80.00th=[ 1854], 90.00th=[ 1854], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 8192, max= 8192, per=75.18%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.04%, sys=15.67%, ctx=97, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12720: Mon Dec 2 15:52:39 2024 write: IOPS=0, BW=4086KiB/s (4185kB/s)(12.0MiB/3007msec) clat (msec): min=271, max=1637, avg=990.41, stdev=685.58 lat (msec): min=272, max=1637, avg=990.75, stdev=685.43 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 271], 10.00th=[ 271], 20.00th=[ 271], | 30.00th=[ 271], 40.00th=[ 1062], 50.00th=[ 1062], 60.00th=[ 1062], | 70.00th=[ 1636], 80.00th=[ 1636], 90.00th=[ 1636], 95.00th=[ 1636], | 99.00th=[ 1636], 99.50th=[ 1636], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 8126, max= 8192, per=74.87%, avg=8159.00, stdev=46.67, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=11.64%, ctx=36, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12721: Mon Dec 2 15:52:39 2024 write: IOPS=1, BW=6269KiB/s (6420kB/s)(12.0MiB/1960msec) clat (msec): min=150, max=1550, avg=645.64, stdev=784.87 lat (msec): min=150, max=1550, avg=645.87, stdev=784.89 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 150], 10.00th=[ 150], 20.00th=[ 150], | 30.00th=[ 150], 40.00th=[ 236], 50.00th=[ 236], 60.00th=[ 236], | 70.00th=[ 1552], 80.00th=[ 1552], 90.00th=[ 1552], 95.00th=[ 1552], | 99.00th=[ 1552], 99.50th=[ 1552], 99.90th=[ 1552], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 8192, max= 8192, per=75.18%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.05%, sys=18.94%, ctx=45, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5520KiB/s (5653kB/s), 1697KiB/s-4140KiB/s (1738kB/s-4240kB/s), io=16.0MiB (16.8MB), run=2413-2968msec WRITE: bw=10.6MiB/s (11.2MB/s), 3395KiB/s-6269KiB/s (3476kB/s-6420kB/s), io=32.0MiB (33.6MB), run=1960-3007msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12714: Mon Dec 2 15:52:39 2024 read: IOPS=0, BW=3946KiB/s (4041kB/s)(12.0MiB/3114msec) clat (msec): min=417, max=1821, avg=1033.53, stdev=717.72 lat (msec): min=417, max=1821, avg=1033.53, stdev=717.72 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 418], 10.00th=[ 418], 20.00th=[ 418], | 30.00th=[ 418], 40.00th=[ 860], 50.00th=[ 860], 60.00th=[ 860], | 70.00th=[ 1821], 80.00th=[ 1821], 90.00th=[ 1821], 95.00th=[ 1821], | 99.00th=[ 1821], 99.50th=[ 1821], 99.90th=[ 1821], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 5483, max= 8192, per=100.00%, avg=6837.50, stdev=1915.55, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=3.69%, ctx=60, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12715: Mon Dec 2 15:52:39 2024 read: IOPS=0, BW=1557KiB/s (1595kB/s)(4096KiB/2630msec) clat (nsec): min=60549k, max=60549k, avg=60548670.00, stdev= 0.00 lat (nsec): min=60551k, max=60551k, avg=60550810.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[60556], 5.00th=[60556], 10.00th=[60556], 20.00th=[60556], | 30.00th=[60556], 40.00th=[60556], 50.00th=[60556], 60.00th=[60556], | 70.00th=[60556], 80.00th=[60556], 90.00th=[60556], 95.00th=[60556], | 99.00th=[60556], 99.50th=[60556], 99.90th=[60556], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3115KiB/s (3190kB/s)(8192KiB/2630msec) clat (msec): min=606, max=1948, avg=1277.58, stdev=948.36 lat (msec): min=607, max=1948, avg=1277.95, stdev=948.47 clat percentiles (msec): | 1.00th=[ 609], 5.00th=[ 609], 10.00th=[ 609], 20.00th=[ 609], | 30.00th=[ 609], 40.00th=[ 609], 50.00th=[ 609], 60.00th=[ 1955], | 70.00th=[ 1955], 80.00th=[ 1955], 90.00th=[ 1955], 95.00th=[ 1955], | 99.00th=[ 1955], 99.50th=[ 1955], 99.90th=[ 1955], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7728, max= 7728, per=75.80%, avg=7728.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.00%, sys=6.69%, ctx=184, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12716: Mon Dec 2 15:52:39 2024 write: IOPS=0, BW=3823KiB/s (3915kB/s)(12.0MiB/3214msec) clat (msec): min=538, max=1774, avg=1058.52, stdev=640.85 lat (msec): min=539, max=1774, avg=1058.87, stdev=640.83 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 542], 10.00th=[ 542], 20.00th=[ 542], | 30.00th=[ 542], 40.00th=[ 860], 50.00th=[ 860], 60.00th=[ 860], | 70.00th=[ 1770], 80.00th=[ 1770], 90.00th=[ 1770], 95.00th=[ 1770], | 99.00th=[ 1770], 99.50th=[ 1770], 99.90th=[ 1770], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 7380, max= 7380, per=72.39%, avg=7380.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 750=33.33%, 1000=33.33% cpu : usr=0.03%, sys=2.68%, ctx=41, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12717: Mon Dec 2 15:52:39 2024 write: IOPS=4, BW=16.4MiB/s (17.2MB/s)(12.0MiB/733msec) clat (msec): min=73, max=499, avg=240.75, stdev=227.08 lat (msec): min=73, max=499, avg=241.10, stdev=227.06 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 73], 10.00th=[ 73], 20.00th=[ 73], | 30.00th=[ 73], 40.00th=[ 150], 50.00th=[ 150], 60.00th=[ 150], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.41%, sys=10.25%, ctx=54, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5261KiB/s (5388kB/s), 1557KiB/s-3946KiB/s (1595kB/s-4041kB/s), io=16.0MiB (16.8MB), run=2630-3114msec WRITE: bw=9.96MiB/s (10.4MB/s), 3115KiB/s-16.4MiB/s (3190kB/s-17.2MB/s), io=32.0MiB (33.6MB), run=733-3214msec