== sanity test 398b: DIO and buffer IO race ============== 00:57:57 (1735106277) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.21057 s, 41.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6094: Wed Dec 25 00:58:12 2024 read: IOPS=122, BW=491KiB/s (503kB/s)(5992KiB/12205msec) clat (usec): min=47, max=169819, avg=3714.52, stdev=5212.74 lat (usec): min=48, max=169820, avg=3715.31, stdev=5212.79 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 60], 10.00th=[ 72], 20.00th=[ 1844], | 30.00th=[ 2278], 40.00th=[ 2638], 50.00th=[ 3032], 60.00th=[ 3589], | 70.00th=[ 4490], 80.00th=[ 5669], 90.00th=[ 7373], 95.00th=[ 8848], | 99.00th=[ 11863], 99.50th=[ 13566], 99.90th=[ 33424], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 232, max= 664, per=24.90%, avg=486.12, stdev=90.80, samples=24 iops : min= 58, max= 166, avg=121.46, stdev=22.69, samples=24 write: IOPS=128, BW=516KiB/s (528kB/s)(6296KiB/12205msec) clat (usec): min=430, max=39062, avg=4143.83, stdev=4077.28 lat (usec): min=431, max=39063, avg=4144.63, stdev=4077.27 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 519], 10.00th=[ 553], 20.00th=[ 668], | 30.00th=[ 938], 40.00th=[ 2024], 50.00th=[ 3261], 60.00th=[ 4178], | 70.00th=[ 5276], 80.00th=[ 6718], 90.00th=[ 9372], 95.00th=[12125], | 99.00th=[17433], 99.50th=[20579], 99.90th=[33817], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 288, max= 680, per=24.96%, avg=510.12, stdev=99.95, samples=24 iops : min= 72, max= 170, avg=127.46, stdev=24.98, samples=24 lat (usec) : 50=0.29%, 100=7.75%, 250=1.17%, 500=1.17%, 750=11.36% lat (usec) : 1000=3.65% lat (msec) : 2=6.15%, 4=30.08%, 10=33.01%, 20=5.01%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.11%, sys=24.65%, ctx=5893, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6095: Wed Dec 25 00:58:12 2024 read: IOPS=120, BW=483KiB/s (495kB/s)(5940KiB/12298msec) clat (usec): min=46, max=27920, avg=3764.49, stdev=3044.96 lat (usec): min=46, max=27921, avg=3765.38, stdev=3044.92 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 61], 10.00th=[ 73], 20.00th=[ 1975], | 30.00th=[ 2376], 40.00th=[ 2737], 50.00th=[ 3163], 60.00th=[ 3654], | 70.00th=[ 4555], 80.00th=[ 5866], 90.00th=[ 7373], 95.00th=[ 8717], | 99.00th=[15139], 99.50th=[18220], 99.90th=[26084], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 248, max= 680, per=24.77%, avg=483.50, stdev=102.76, samples=24 iops : min= 62, max= 170, avg=120.83, stdev=25.65, samples=24 write: IOPS=129, BW=516KiB/s (529kB/s)(6348KiB/12298msec) clat (usec): min=461, max=198370, avg=4150.65, stdev=6117.67 lat (usec): min=461, max=198371, avg=4151.83, stdev=6117.77 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 635], | 30.00th=[ 938], 40.00th=[ 1844], 50.00th=[ 3228], 60.00th=[ 4293], | 70.00th=[ 5407], 80.00th=[ 6783], 90.00th=[ 9110], 95.00th=[ 11338], | 99.00th=[ 15926], 99.50th=[ 18482], 99.90th=[ 24773], 99.95th=[198181], | 99.99th=[198181] bw ( KiB/s): min= 288, max= 766, per=24.94%, avg=509.83, stdev=106.37, samples=24 iops : min= 72, max= 191, avg=127.42, stdev=26.54, samples=24 lat (usec) : 50=0.42%, 100=6.54%, 250=1.20%, 500=0.85%, 750=12.21% lat (usec) : 1000=3.32% lat (msec) : 2=6.45%, 4=29.65%, 10=33.82%, 20=5.21%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=24.71%, ctx=5972, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6096: Wed Dec 25 00:58:12 2024 read: IOPS=122, BW=490KiB/s (502kB/s)(6028KiB/12292msec) clat (usec): min=48, max=199588, avg=3827.91, stdev=5877.30 lat (usec): min=48, max=199589, avg=3828.64, stdev=5877.28 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 60], 10.00th=[ 73], 20.00th=[ 1876], | 30.00th=[ 2343], 40.00th=[ 2737], 50.00th=[ 3163], 60.00th=[ 3720], | 70.00th=[ 4555], 80.00th=[ 5866], 90.00th=[ 7439], 95.00th=[ 8717], | 99.00th=[ 11994], 99.50th=[ 15533], 99.90th=[ 40109], 99.95th=[200279], | 99.99th=[200279] bw ( KiB/s): min= 216, max= 664, per=24.96%, avg=487.25, stdev=107.49, samples=24 iops : min= 54, max= 166, avg=121.79, stdev=26.84, samples=24 write: IOPS=127, BW=509KiB/s (521kB/s)(6260KiB/12292msec) clat (usec): min=423, max=38674, avg=4090.32, stdev=3965.85 lat (usec): min=424, max=38675, avg=4091.90, stdev=3967.56 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 519], 10.00th=[ 545], 20.00th=[ 627], | 30.00th=[ 889], 40.00th=[ 1795], 50.00th=[ 3163], 60.00th=[ 4146], | 70.00th=[ 5407], 80.00th=[ 6980], 90.00th=[ 9241], 95.00th=[11731], | 99.00th=[16909], 99.50th=[18482], 99.90th=[36439], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 312, max= 616, per=24.67%, avg=504.25, stdev=86.61, samples=24 iops : min= 78, max= 154, avg=126.04, stdev=21.63, samples=24 lat (usec) : 50=0.33%, 100=7.55%, 250=1.27%, 500=1.37%, 750=11.59% lat (usec) : 1000=3.74% lat (msec) : 2=6.05%, 4=29.20%, 10=33.59%, 20=4.98%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=24.30%, ctx=5843, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6097: Wed Dec 25 00:58:12 2024 read: IOPS=123, BW=496KiB/s (507kB/s)(6052KiB/12213msec) clat (usec): min=46, max=167276, avg=3852.88, stdev=5462.17 lat (usec): min=46, max=167276, avg=3853.95, stdev=5462.18 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 61], 10.00th=[ 72], 20.00th=[ 1876], | 30.00th=[ 2343], 40.00th=[ 2737], 50.00th=[ 3130], 60.00th=[ 3687], | 70.00th=[ 4490], 80.00th=[ 5669], 90.00th=[ 7242], 95.00th=[ 8848], | 99.00th=[ 14222], 99.50th=[ 20841], 99.90th=[ 49021], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 272, max= 662, per=25.25%, avg=492.83, stdev=101.13, samples=24 iops : min= 68, max= 165, avg=123.17, stdev=25.23, samples=24 write: IOPS=127, BW=511KiB/s (523kB/s)(6236KiB/12213msec) clat (usec): min=468, max=48977, avg=4018.92, stdev=3836.55 lat (usec): min=469, max=48978, avg=4020.13, stdev=3836.32 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 676], | 30.00th=[ 996], 40.00th=[ 2114], 50.00th=[ 3294], 60.00th=[ 4178], | 70.00th=[ 5145], 80.00th=[ 6521], 90.00th=[ 9110], 95.00th=[10945], | 99.00th=[16057], 99.50th=[19268], 99.90th=[27657], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 344, max= 720, per=24.81%, avg=507.21, stdev=96.58, samples=24 iops : min= 86, max= 180, avg=126.75, stdev=24.14, samples=24 lat (usec) : 50=0.49%, 100=7.26%, 250=1.14%, 500=1.07%, 750=10.61% lat (usec) : 1000=4.00% lat (msec) : 2=6.09%, 4=30.50%, 10=33.72%, 20=4.65%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=24.56%, ctx=5862, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1953KiB/s (1999kB/s), 483KiB/s-496KiB/s (495kB/s-507kB/s), io=23.4MiB (24.6MB), run=12205-12298msec WRITE: bw=2044KiB/s (2093kB/s), 509KiB/s-516KiB/s (521kB/s-529kB/s), io=24.6MiB (25.7MB), run=12205-12298msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6090: Wed Dec 25 00:58:33 2024 read: IOPS=45, BW=183KiB/s (187kB/s)(5992KiB/32775msec) clat (usec): min=1472, max=203410, avg=3310.11, stdev=5845.20 lat (usec): min=1472, max=203412, avg=3310.81, stdev=5845.22 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1745], 10.00th=[ 1811], 20.00th=[ 1876], | 30.00th=[ 1942], 40.00th=[ 2024], 50.00th=[ 2114], 60.00th=[ 2245], | 70.00th=[ 2540], 80.00th=[ 3621], 90.00th=[ 6587], 95.00th=[ 8717], | 99.00th=[ 12780], 99.50th=[ 14615], 99.90th=[ 42730], 99.95th=[202376], | 99.99th=[202376] bw ( KiB/s): min= 71, max= 336, per=25.16%, avg=182.65, stdev=70.34, samples=65 iops : min= 17, max= 84, avg=45.57, stdev=17.60, samples=65 write: IOPS=48, BW=192KiB/s (197kB/s)(6296KiB/32775msec) clat (usec): min=8457, max=51051, avg=17592.63, stdev=6329.18 lat (usec): min=8457, max=51052, avg=17593.47, stdev=6329.20 clat percentiles (usec): | 1.00th=[ 9634], 5.00th=[10683], 10.00th=[11076], 20.00th=[12125], | 30.00th=[13435], 40.00th=[15139], 50.00th=[16581], 60.00th=[17695], | 70.00th=[18744], 80.00th=[21103], 90.00th=[26608], 95.00th=[31065], | 99.00th=[38536], 99.50th=[41681], 99.90th=[49546], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 55, max= 264, per=25.18%, avg=191.63, stdev=56.36, samples=65 iops : min= 13, max= 66, avg=47.82, stdev=14.15, samples=65 lat (msec) : 2=18.49%, 4=21.45%, 10=8.33%, 20=39.78%, 50=11.88% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.09%, sys=5.58%, ctx=3422, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6091: Wed Dec 25 00:58:33 2024 read: IOPS=44, BW=180KiB/s (184kB/s)(5940KiB/33029msec) clat (usec): min=1444, max=170680, avg=3339.17, stdev=5453.03 lat (usec): min=1445, max=170681, avg=3339.85, stdev=5453.04 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1745], 10.00th=[ 1811], 20.00th=[ 1876], | 30.00th=[ 1942], 40.00th=[ 2008], 50.00th=[ 2114], 60.00th=[ 2245], | 70.00th=[ 2540], 80.00th=[ 3359], 90.00th=[ 6783], 95.00th=[ 9241], | 99.00th=[ 14353], 99.50th=[ 18744], 99.90th=[ 65274], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 16, max= 336, per=24.80%, avg=180.08, stdev=73.34, samples=65 iops : min= 4, max= 84, avg=44.94, stdev=18.30, samples=65 write: IOPS=48, BW=192KiB/s (197kB/s)(6348KiB/33029msec) clat (usec): min=8774, max=48831, avg=17605.75, stdev=6338.00 lat (usec): min=8775, max=48832, avg=17606.54, stdev=6338.01 clat percentiles (usec): | 1.00th=[ 9634], 5.00th=[10552], 10.00th=[11076], 20.00th=[12256], | 30.00th=[13698], 40.00th=[15270], 50.00th=[16581], 60.00th=[17695], | 70.00th=[19006], 80.00th=[20841], 90.00th=[25822], 95.00th=[30278], | 99.00th=[39584], 99.50th=[43254], 99.90th=[47449], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 72, max= 272, per=25.10%, avg=191.03, stdev=54.34, samples=65 iops : min= 18, max= 68, avg=47.68, stdev=13.61, samples=65 lat (msec) : 2=18.85%, 4=21.29%, 10=7.19%, 20=40.10%, 50=12.50% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.09%, sys=5.41%, ctx=3379, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6092: Wed Dec 25 00:58:33 2024 read: IOPS=45, BW=183KiB/s (188kB/s)(6028KiB/32896msec) clat (usec): min=1513, max=31638, avg=3336.03, stdev=2824.36 lat (usec): min=1513, max=31638, avg=3336.77, stdev=2824.40 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1762], 10.00th=[ 1811], 20.00th=[ 1893], | 30.00th=[ 1958], 40.00th=[ 2040], 50.00th=[ 2147], 60.00th=[ 2278], | 70.00th=[ 2704], 80.00th=[ 3949], 90.00th=[ 7177], 95.00th=[ 9503], | 99.00th=[13698], 99.50th=[16712], 99.90th=[29230], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 56, max= 336, per=25.23%, avg=183.18, stdev=64.44, samples=65 iops : min= 14, max= 84, avg=45.74, stdev=16.14, samples=65 write: IOPS=47, BW=190KiB/s (195kB/s)(6260KiB/32896msec) clat (msec): min=9, max=181, avg=17.72, stdev= 7.49 lat (msec): min=9, max=181, avg=17.72, stdev= 7.49 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 27], 95.00th=[ 31], | 99.00th=[ 40], 99.50th=[ 44], 99.90th=[ 75], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 64, max= 256, per=24.93%, avg=189.71, stdev=55.20, samples=65 iops : min= 16, max= 64, avg=47.37, stdev=13.82, samples=65 lat (msec) : 2=17.97%, 4=21.42%, 10=8.04%, 20=40.95%, 50=11.49% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.11%, sys=5.44%, ctx=3408, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6093: Wed Dec 25 00:58:33 2024 read: IOPS=46, BW=185KiB/s (189kB/s)(6052KiB/32793msec) clat (usec): min=1560, max=42783, avg=3351.95, stdev=3203.16 lat (usec): min=1561, max=42783, avg=3352.62, stdev=3203.16 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1745], 10.00th=[ 1795], 20.00th=[ 1876], | 30.00th=[ 1958], 40.00th=[ 2057], 50.00th=[ 2147], 60.00th=[ 2311], | 70.00th=[ 2671], 80.00th=[ 3752], 90.00th=[ 6915], 95.00th=[ 9503], | 99.00th=[16319], 99.50th=[21103], 99.90th=[32113], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 32, max= 352, per=25.32%, avg=183.82, stdev=72.91, samples=65 iops : min= 8, max= 88, avg=45.91, stdev=18.22, samples=65 write: IOPS=47, BW=190KiB/s (195kB/s)(6236KiB/32793msec) clat (msec): min=9, max=192, avg=17.70, stdev= 7.79 lat (msec): min=9, max=192, avg=17.70, stdev= 7.79 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 20], 80.00th=[ 21], 90.00th=[ 27], 95.00th=[ 31], | 99.00th=[ 41], 99.50th=[ 44], 99.90th=[ 75], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 56, max= 256, per=24.95%, avg=189.85, stdev=55.07, samples=65 iops : min= 14, max= 64, avg=47.42, stdev=13.77, samples=65 lat (msec) : 2=16.99%, 4=23.05%, 10=7.45%, 20=40.17%, 50=12.17% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.09%, sys=5.44%, ctx=3417, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=727KiB/s (744kB/s), 180KiB/s-185KiB/s (184kB/s-189kB/s), io=23.4MiB (24.6MB), run=32775-33029msec WRITE: bw=761KiB/s (779kB/s), 190KiB/s-192KiB/s (195kB/s-197kB/s), io=24.6MiB (25.7MB), run=32775-33029msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6139: Wed Dec 25 00:58:38 2024 read: IOPS=81, BW=1307KiB/s (1339kB/s)(5920KiB/4529msec) clat (usec): min=52, max=44471, avg=9255.38, stdev=5602.21 lat (usec): min=52, max=44472, avg=9255.99, stdev=5602.23 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 64], 10.00th=[ 76], 20.00th=[ 7308], | 30.00th=[ 8455], 40.00th=[ 9110], 50.00th=[ 9765], 60.00th=[10290], | 70.00th=[10945], 80.00th=[11863], 90.00th=[13698], 95.00th=[16319], | 99.00th=[28443], 99.50th=[41681], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 1024, max= 1728, per=26.24%, avg=1304.89, stdev=215.72, samples=9 iops : min= 64, max= 108, avg=81.56, stdev=13.48, samples=9 write: IOPS=87, BW=1406KiB/s (1440kB/s)(6368KiB/4529msec) clat (usec): min=641, max=21115, avg=2744.62, stdev=2668.44 lat (usec): min=642, max=21116, avg=2745.72, stdev=2668.38 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 725], 10.00th=[ 766], 20.00th=[ 816], | 30.00th=[ 873], 40.00th=[ 996], 50.00th=[ 1762], 60.00th=[ 2606], | 70.00th=[ 3294], 80.00th=[ 4555], 90.00th=[ 6194], 95.00th=[ 7177], | 99.00th=[13173], 99.50th=[15926], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 1056, max= 1792, per=26.84%, avg=1404.44, stdev=268.42, samples=9 iops : min= 66, max= 112, avg=87.78, stdev=16.78, samples=9 lat (usec) : 100=6.90%, 250=0.91%, 500=0.13%, 750=3.78%, 1000=17.06% lat (msec) : 2=5.60%, 4=13.02%, 10=29.43%, 20=21.87%, 50=1.30% cpu : usr=0.09%, sys=23.34%, ctx=2186, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6141: Wed Dec 25 00:58:38 2024 read: IOPS=79, BW=1265KiB/s (1295kB/s)(5744KiB/4541msec) clat (usec): min=52, max=27679, avg=8403.81, stdev=4962.05 lat (usec): min=53, max=27679, avg=8404.37, stdev=4962.05 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 64], 10.00th=[ 71], 20.00th=[ 141], | 30.00th=[ 8029], 40.00th=[ 8848], 50.00th=[ 9503], 60.00th=[10290], | 70.00th=[11076], 80.00th=[11863], 90.00th=[13173], 95.00th=[14615], | 99.00th=[17957], 99.50th=[22676], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 352, max= 1792, per=25.17%, avg=1251.56, stdev=402.06, samples=9 iops : min= 22, max= 112, avg=78.22, stdev=25.13, samples=9 write: IOPS=90, BW=1441KiB/s (1476kB/s)(6544KiB/4541msec) clat (usec): min=639, max=67640, avg=3554.42, stdev=5543.26 lat (usec): min=640, max=67642, avg=3555.58, stdev=5543.35 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 725], 10.00th=[ 766], 20.00th=[ 816], | 30.00th=[ 889], 40.00th=[ 1139], 50.00th=[ 2311], 60.00th=[ 2737], | 70.00th=[ 3228], 80.00th=[ 4752], 90.00th=[ 6652], 95.00th=[12649], | 99.00th=[23987], 99.50th=[30802], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 608, max= 1920, per=27.32%, avg=1429.33, stdev=430.52, samples=9 iops : min= 38, max= 120, avg=89.33, stdev=26.91, samples=9 lat (usec) : 100=8.07%, 250=1.69%, 750=3.78%, 1000=15.62% lat (msec) : 2=6.12%, 4=15.49%, 10=25.65%, 20=21.87%, 50=1.56% lat (msec) : 100=0.13% cpu : usr=0.20%, sys=21.45%, ctx=2052, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6142: Wed Dec 25 00:58:38 2024 read: IOPS=81, BW=1299KiB/s (1330kB/s)(6256KiB/4816msec) clat (usec): min=52, max=28781, avg=8766.75, stdev=5123.61 lat (usec): min=52, max=28782, avg=8767.33, stdev=5123.61 clat percentiles (usec): | 1.00th=[ 54], 5.00th=[ 64], 10.00th=[ 77], 20.00th=[ 4293], | 30.00th=[ 8029], 40.00th=[ 8979], 50.00th=[ 9765], 60.00th=[10421], | 70.00th=[11207], 80.00th=[12125], 90.00th=[13829], 95.00th=[15533], | 99.00th=[23725], 99.50th=[25560], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 416, max= 1888, per=25.30%, avg=1258.33, stdev=394.74, samples=9 iops : min= 26, max= 118, avg=78.56, stdev=24.64, samples=9 write: IOPS=78, BW=1252KiB/s (1283kB/s)(6032KiB/4816msec) clat (usec): min=651, max=83517, avg=3505.77, stdev=6105.80 lat (usec): min=652, max=83519, avg=3506.80, stdev=6105.84 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 709], 10.00th=[ 750], 20.00th=[ 799], | 30.00th=[ 840], 40.00th=[ 947], 50.00th=[ 1565], 60.00th=[ 2573], | 70.00th=[ 3163], 80.00th=[ 4752], 90.00th=[ 6915], 95.00th=[12649], | 99.00th=[23725], 99.50th=[38536], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 512, max= 1632, per=22.96%, avg=1201.44, stdev=330.48, samples=9 iops : min= 32, max= 102, avg=75.00, stdev=20.71, samples=9 lat (usec) : 100=8.85%, 250=1.04%, 500=0.13%, 750=5.34%, 1000=15.36% lat (msec) : 2=4.95%, 4=11.59%, 10=26.17%, 20=24.74%, 50=1.69% lat (msec) : 100=0.13% cpu : usr=0.12%, sys=20.77%, ctx=2133, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6143: Wed Dec 25 00:58:38 2024 read: IOPS=81, BW=1306KiB/s (1337kB/s)(6032KiB/4619msec) clat (usec): min=51, max=52593, avg=8725.02, stdev=5283.76 lat (usec): min=51, max=52593, avg=8726.34, stdev=5284.30 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 63], 10.00th=[ 72], 20.00th=[ 5932], | 30.00th=[ 8160], 40.00th=[ 8979], 50.00th=[ 9634], 60.00th=[10290], | 70.00th=[11076], 80.00th=[11863], 90.00th=[13566], 95.00th=[14746], | 99.00th=[20317], 99.50th=[21365], 99.90th=[52691], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 448, max= 1504, per=25.81%, avg=1283.56, stdev=324.94, samples=9 iops : min= 28, max= 94, avg=80.22, stdev=20.31, samples=9 write: IOPS=84, BW=1354KiB/s (1387kB/s)(6256KiB/4619msec) clat (usec): min=631, max=78441, avg=3200.37, stdev=5375.69 lat (usec): min=631, max=78443, avg=3201.35, stdev=5375.74 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 734], 10.00th=[ 775], 20.00th=[ 807], | 30.00th=[ 857], 40.00th=[ 963], 50.00th=[ 1369], 60.00th=[ 2606], | 70.00th=[ 3294], 80.00th=[ 4490], 90.00th=[ 6390], 95.00th=[ 8586], | 99.00th=[21365], 99.50th=[33424], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 512, max= 1824, per=25.69%, avg=1344.00, stdev=404.46, samples=9 iops : min= 32, max= 114, avg=84.00, stdev=25.28, samples=9 lat (usec) : 100=8.07%, 250=1.30%, 500=0.13%, 750=3.26%, 1000=18.49% lat (msec) : 2=5.99%, 4=11.59%, 10=26.43%, 20=23.57%, 50=0.91% lat (msec) : 100=0.26% cpu : usr=0.19%, sys=21.20%, ctx=2048, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4973KiB/s (5093kB/s), 1265KiB/s-1307KiB/s (1295kB/s-1339kB/s), io=23.4MiB (24.5MB), run=4529-4816msec WRITE: bw=5233KiB/s (5358kB/s), 1252KiB/s-1441KiB/s (1283kB/s-1476kB/s), io=24.6MiB (25.8MB), run=4529-4816msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6136: Wed Dec 25 00:58:43 2024 read: IOPS=41, BW=663KiB/s (678kB/s)(5920KiB/8935msec) clat (usec): min=1920, max=40217, avg=4227.89, stdev=4308.40 lat (usec): min=1921, max=40219, avg=4228.60, stdev=4308.43 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2057], 10.00th=[ 2089], 20.00th=[ 2180], | 30.00th=[ 2278], 40.00th=[ 2409], 50.00th=[ 2606], 60.00th=[ 2868], | 70.00th=[ 3326], 80.00th=[ 4883], 90.00th=[ 8979], 95.00th=[12256], | 99.00th=[24249], 99.50th=[34341], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 448, max= 1088, per=24.80%, avg=656.94, stdev=203.37, samples=17 iops : min= 28, max= 68, avg=41.06, stdev=12.71, samples=17 write: IOPS=44, BW=713KiB/s (730kB/s)(6368KiB/8935msec) clat (usec): min=9974, max=39578, avg=18346.52, stdev=5251.48 lat (usec): min=9976, max=39579, avg=18348.20, stdev=5251.55 clat percentiles (usec): | 1.00th=[10945], 5.00th=[11731], 10.00th=[12256], 20.00th=[13566], | 30.00th=[15008], 40.00th=[16450], 50.00th=[17433], 60.00th=[19268], | 70.00th=[20317], 80.00th=[21890], 90.00th=[24249], 95.00th=[28443], | 99.00th=[37487], 99.50th=[38536], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 480, max= 928, per=25.26%, avg=704.00, stdev=150.94, samples=17 iops : min= 30, max= 58, avg=44.00, stdev= 9.43, samples=17 lat (msec) : 2=0.78%, 4=35.42%, 10=8.20%, 20=37.89%, 50=17.71% cpu : usr=0.09%, sys=6.06%, ctx=895, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6137: Wed Dec 25 00:58:43 2024 read: IOPS=39, BW=635KiB/s (651kB/s)(5744KiB/9041msec) clat (usec): min=1850, max=25766, avg=3915.88, stdev=3332.03 lat (usec): min=1851, max=25767, avg=3916.55, stdev=3332.02 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2040], 10.00th=[ 2089], 20.00th=[ 2212], | 30.00th=[ 2311], 40.00th=[ 2474], 50.00th=[ 2671], 60.00th=[ 2868], | 70.00th=[ 3195], 80.00th=[ 4686], 90.00th=[ 7832], 95.00th=[10814], | 99.00th=[20841], 99.50th=[23200], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 288, max= 992, per=23.80%, avg=630.59, stdev=216.67, samples=17 iops : min= 18, max= 62, avg=39.41, stdev=13.54, samples=17 write: IOPS=45, BW=724KiB/s (741kB/s)(6544KiB/9041msec) clat (usec): min=10614, max=90447, avg=18548.67, stdev=6142.80 lat (usec): min=10616, max=90448, avg=18550.18, stdev=6142.94 clat percentiles (usec): | 1.00th=[11076], 5.00th=[11731], 10.00th=[12649], 20.00th=[13829], | 30.00th=[15139], 40.00th=[16581], 50.00th=[17957], 60.00th=[19006], | 70.00th=[20317], 80.00th=[21890], 90.00th=[25035], 95.00th=[27919], | 99.00th=[34341], 99.50th=[36963], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 480, max= 960, per=25.60%, avg=713.41, stdev=132.79, samples=17 iops : min= 30, max= 60, avg=44.59, stdev= 8.30, samples=17 lat (msec) : 2=1.43%, 4=34.11%, 10=8.72%, 20=37.63%, 50=17.97% lat (msec) : 100=0.13% cpu : usr=0.17%, sys=5.90%, ctx=887, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6138: Wed Dec 25 00:58:43 2024 read: IOPS=44, BW=720KiB/s (737kB/s)(6256KiB/8689msec) clat (usec): min=1745, max=24557, avg=3972.59, stdev=3209.88 lat (usec): min=1746, max=24558, avg=3973.29, stdev=3209.89 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2024], 10.00th=[ 2089], 20.00th=[ 2212], | 30.00th=[ 2311], 40.00th=[ 2474], 50.00th=[ 2671], 60.00th=[ 2966], | 70.00th=[ 3621], 80.00th=[ 4883], 90.00th=[ 7308], 95.00th=[11076], | 99.00th=[18482], 99.50th=[21103], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 352, max= 1344, per=27.56%, avg=730.18, stdev=270.33, samples=17 iops : min= 22, max= 84, avg=45.53, stdev=17.00, samples=17 write: IOPS=43, BW=694KiB/s (711kB/s)(6032KiB/8689msec) clat (usec): min=10807, max=87600, avg=18885.14, stdev=6595.34 lat (usec): min=10808, max=87602, avg=18886.79, stdev=6595.36 clat percentiles (usec): | 1.00th=[11076], 5.00th=[11863], 10.00th=[12387], 20.00th=[14091], | 30.00th=[15270], 40.00th=[16581], 50.00th=[17957], 60.00th=[19006], | 70.00th=[20317], 80.00th=[22152], 90.00th=[25560], 95.00th=[30540], | 99.00th=[39584], 99.50th=[45876], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 510, max= 864, per=24.78%, avg=690.59, stdev=135.92, samples=17 iops : min= 31, max= 54, avg=43.06, stdev= 8.55, samples=17 lat (msec) : 2=1.43%, 4=36.07%, 10=9.90%, 20=36.33%, 50=16.15% lat (msec) : 100=0.13% cpu : usr=0.15%, sys=5.99%, ctx=887, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6140: Wed Dec 25 00:58:43 2024 read: IOPS=42, BW=685KiB/s (702kB/s)(6032KiB/8803msec) clat (usec): min=1797, max=31366, avg=3959.91, stdev=3381.56 lat (usec): min=1798, max=31366, avg=3960.64, stdev=3381.52 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2057], 10.00th=[ 2114], 20.00th=[ 2212], | 30.00th=[ 2278], 40.00th=[ 2376], 50.00th=[ 2606], 60.00th=[ 2835], | 70.00th=[ 3326], 80.00th=[ 4883], 90.00th=[ 8094], 95.00th=[11600], | 99.00th=[18220], 99.50th=[21627], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 384, max= 992, per=25.51%, avg=675.65, stdev=186.02, samples=17 iops : min= 24, max= 62, avg=42.18, stdev=11.54, samples=17 write: IOPS=44, BW=711KiB/s (728kB/s)(6256KiB/8803msec) clat (usec): min=10652, max=69264, avg=18581.59, stdev=5872.68 lat (usec): min=10654, max=69266, avg=18583.28, stdev=5872.82 clat percentiles (usec): | 1.00th=[10814], 5.00th=[11731], 10.00th=[12125], 20.00th=[13829], | 30.00th=[15401], 40.00th=[16909], 50.00th=[18220], 60.00th=[19006], | 70.00th=[20317], 80.00th=[21890], 90.00th=[24511], 95.00th=[27919], | 99.00th=[42730], 99.50th=[46400], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 512, max= 928, per=25.32%, avg=705.76, stdev=155.66, samples=17 iops : min= 32, max= 58, avg=44.06, stdev= 9.70, samples=17 lat (msec) : 2=0.65%, 4=36.85%, 10=7.81%, 20=38.15%, 50=16.41% lat (msec) : 100=0.13% cpu : usr=0.14%, sys=6.01%, ctx=894, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2649KiB/s (2713kB/s), 635KiB/s-720KiB/s (651kB/s-737kB/s), io=23.4MiB (24.5MB), run=8689-9041msec WRITE: bw=2787KiB/s (2854kB/s), 694KiB/s-724KiB/s (711kB/s-741kB/s), io=24.6MiB (25.8MB), run=8689-9041msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6156: Wed Dec 25 00:58:47 2024 read: IOPS=1, BW=2011KiB/s (2059kB/s)(7168KiB/3564msec) clat (msec): min=17, max=1179, avg=301.37, stdev=430.61 lat (msec): min=17, max=1179, avg=301.37, stdev=430.61 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 31], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 78], 60.00th=[ 211], | 70.00th=[ 211], 80.00th=[ 550], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 2048, max= 3549, per=42.17%, avg=2423.25, stdev=750.50, samples=4 iops : min= 2, max= 3, avg= 2.25, stdev= 0.50, samples=4 write: IOPS=1, BW=1437KiB/s (1471kB/s)(5120KiB/3564msec) clat (msec): min=24, max=659, avg=285.32, stdev=321.40 lat (msec): min=24, max=659, avg=285.38, stdev=321.38 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 26], 40.00th=[ 26], 50.00th=[ 107], 60.00th=[ 107], | 70.00th=[ 609], 80.00th=[ 609], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 1774, max= 3543, per=33.05%, avg=2658.50, stdev=1250.87, samples=2 iops : min= 1, max= 3, avg= 2.00, stdev= 1.41, samples=2 lat (msec) : 20=8.33%, 50=33.33%, 100=8.33%, 250=16.67%, 750=25.00% cpu : usr=0.00%, sys=3.82%, ctx=55, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6157: Wed Dec 25 00:58:47 2024 read: IOPS=0, BW=839KiB/s (859kB/s)(2048KiB/2441msec) clat (usec): min=12714, max=17375, avg=15045.20, stdev=3295.66 lat (usec): min=12716, max=17376, avg=15046.76, stdev=3295.29 clat percentiles (usec): | 1.00th=[12780], 5.00th=[12780], 10.00th=[12780], 20.00th=[12780], | 30.00th=[12780], 40.00th=[12780], 50.00th=[12780], 60.00th=[17433], | 70.00th=[17433], 80.00th=[17433], 90.00th=[17433], 95.00th=[17433], | 99.00th=[17433], 99.50th=[17433], 99.90th=[17433], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 2048, max= 2048, per=35.64%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4195KiB/s (4296kB/s)(10.0MiB/2441msec) clat (msec): min=23, max=1083, avg=239.86, stdev=393.16 lat (msec): min=23, max=1083, avg=239.95, stdev=393.17 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 31], | 30.00th=[ 35], 40.00th=[ 51], 50.00th=[ 64], 60.00th=[ 67], | 70.00th=[ 105], 80.00th=[ 105], 90.00th=[ 877], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min=14336, max=14336, per=100.00%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=25.00%, 100=33.33%, 250=8.33%, 1000=8.33% cpu : usr=0.04%, sys=4.26%, ctx=51, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6159: Wed Dec 25 00:58:47 2024 read: IOPS=2, BW=2483KiB/s (2543kB/s)(6144KiB/2474msec) clat (msec): min=20, max=895, avg=192.92, stdev=344.88 lat (msec): min=20, max=895, avg=192.92, stdev=344.88 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 38], | 30.00th=[ 38], 40.00th=[ 61], 50.00th=[ 61], 60.00th=[ 63], | 70.00th=[ 81], 80.00th=[ 81], 90.00th=[ 894], 95.00th=[ 894], | 99.00th=[ 894], 99.50th=[ 894], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 6144, max= 6144, per=100.00%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2483KiB/s (2543kB/s)(6144KiB/2474msec) clat (msec): min=31, max=1075, avg=217.69, stdev=420.37 lat (msec): min=31, max=1075, avg=217.80, stdev=420.34 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 35], | 30.00th=[ 35], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 50], | 70.00th=[ 78], 80.00th=[ 78], 90.00th=[ 1083], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 4096, max= 4096, per=50.92%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 lat (msec) : 50=50.00%, 100=33.33%, 1000=8.33% cpu : usr=0.04%, sys=3.36%, ctx=32, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6161: Wed Dec 25 00:58:47 2024 read: IOPS=2, BW=2786KiB/s (2852kB/s)(5120KiB/1838msec) clat (msec): min=13, max=431, avg=104.36, stdev=183.81 lat (msec): min=13, max=431, avg=104.36, stdev=183.81 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 14], | 30.00th=[ 14], 40.00th=[ 14], 50.00th=[ 14], 60.00th=[ 14], | 70.00th=[ 51], 80.00th=[ 51], 90.00th=[ 430], 95.00th=[ 430], | 99.00th=[ 430], 99.50th=[ 430], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3900KiB/s (3993kB/s)(7168KiB/1838msec) clat (msec): min=25, max=1039, avg=185.99, stdev=377.00 lat (msec): min=25, max=1040, avg=186.08, stdev=376.99 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 26], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 47], 60.00th=[ 65], | 70.00th=[ 65], 80.00th=[ 70], 90.00th=[ 1036], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 2043, max=12288, per=89.08%, avg=7165.50, stdev=7244.31, samples=2 iops : min= 1, max= 12, avg= 6.50, stdev= 7.78, samples=2 lat (msec) : 20=25.00%, 50=33.33%, 100=25.00%, 500=8.33% cpu : usr=0.00%, sys=4.74%, ctx=45, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5746KiB/s (5884kB/s), 839KiB/s-2786KiB/s (859kB/s-2852kB/s), io=20.0MiB (20.0MB), run=1838-3564msec WRITE: bw=8045KiB/s (8238kB/s), 1437KiB/s-4195KiB/s (1471kB/s-4296kB/s), io=28.0MiB (29.4MB), run=1838-3564msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6158: Wed Dec 25 00:58:47 2024 read: IOPS=2, BW=2258KiB/s (2312kB/s)(7168KiB/3175msec) clat (usec): min=414, max=932817, avg=258905.09, stdev=412652.04 lat (usec): min=415, max=932818, avg=258906.38, stdev=412652.51 clat percentiles (usec): | 1.00th=[ 416], 5.00th=[ 416], 10.00th=[ 416], 20.00th=[ 441], | 30.00th=[ 3818], 40.00th=[ 3818], 50.00th=[ 9503], 60.00th=[ 82314], | 70.00th=[ 82314], 80.00th=[784335], 90.00th=[935330], 95.00th=[935330], | 99.00th=[935330], 99.50th=[935330], 99.90th=[935330], 99.95th=[935330], | 99.99th=[935330] bw ( KiB/s): min= 2048, max= 4096, per=46.33%, avg=2560.00, stdev=1024.00, samples=4 iops : min= 2, max= 4, avg= 2.50, stdev= 1.00, samples=4 write: IOPS=1, BW=1613KiB/s (1651kB/s)(5120KiB/3175msec) clat (msec): min=15, max=709, avg=270.10, stdev=348.38 lat (msec): min=15, max=709, avg=270.14, stdev=348.39 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 17], 50.00th=[ 22], 60.00th=[ 22], | 70.00th=[ 592], 80.00th=[ 592], 90.00th=[ 709], 95.00th=[ 709], | 99.00th=[ 709], 99.50th=[ 709], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 6144, max= 6144, per=79.42%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 4=8.33%, 10=8.33%, 20=16.67%, 50=8.33%, 100=8.33% lat (msec) : 750=16.67%, 1000=16.67% cpu : usr=0.00%, sys=12.82%, ctx=646, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6160: Wed Dec 25 00:58:47 2024 read: IOPS=0, BW=553KiB/s (566kB/s)(2048KiB/3706msec) clat (msec): min=93, max=666, avg=380.05, stdev=404.99 lat (msec): min=93, max=666, avg=380.05, stdev=404.99 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 94], 10.00th=[ 94], 20.00th=[ 94], | 30.00th=[ 94], 40.00th=[ 94], 50.00th=[ 94], 60.00th=[ 667], | 70.00th=[ 667], 80.00th=[ 667], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 1865, max= 2048, per=35.41%, avg=1956.50, stdev=129.40, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2763KiB/s (2829kB/s)(10.0MiB/3706msec) clat (msec): min=22, max=1024, avg=292.69, stdev=391.05 lat (msec): min=22, max=1024, avg=292.75, stdev=391.04 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 36], | 30.00th=[ 50], 40.00th=[ 65], 50.00th=[ 100], 60.00th=[ 180], | 70.00th=[ 230], 80.00th=[ 230], 90.00th=[ 1020], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 4087, max= 6144, per=68.19%, avg=5275.33, stdev=1065.11, samples=3 iops : min= 3, max= 6, avg= 4.67, stdev= 1.53, samples=3 lat (msec) : 50=25.00%, 100=25.00%, 250=25.00%, 750=8.33% cpu : usr=0.03%, sys=25.26%, ctx=237, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6162: Wed Dec 25 00:58:47 2024 read: IOPS=2, BW=2390KiB/s (2447kB/s)(6144KiB/2571msec) clat (msec): min=15, max=879, avg=204.37, stdev=335.55 lat (msec): min=15, max=879, avg=204.37, stdev=335.55 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 53], 50.00th=[ 53], 60.00th=[ 70], | 70.00th=[ 176], 80.00th=[ 176], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 4096, max= 6144, per=92.65%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2390KiB/s (2447kB/s)(6144KiB/2571msec) clat (msec): min=20, max=1129, avg=219.54, stdev=446.12 lat (msec): min=20, max=1129, avg=219.75, stdev=446.05 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 24], | 30.00th=[ 24], 40.00th=[ 29], 50.00th=[ 29], 60.00th=[ 53], | 70.00th=[ 63], 80.00th=[ 63], 90.00th=[ 1133], 95.00th=[ 1133], | 99.00th=[ 1133], 99.50th=[ 1133], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 4096, max= 4096, per=52.95%, avg=4096.00, stdev= 0.00, samples=3 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=3 lat (msec) : 20=8.33%, 50=33.33%, 100=33.33%, 250=8.33%, 1000=8.33% cpu : usr=0.04%, sys=16.26%, ctx=413, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6163: Wed Dec 25 00:58:47 2024 read: IOPS=1, BW=1653KiB/s (1692kB/s)(5120KiB/3098msec) clat (usec): min=499, max=837735, avg=194626.68, stdev=361245.80 lat (usec): min=500, max=837736, avg=194627.81, stdev=361245.57 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 498], 10.00th=[ 498], 20.00th=[ 498], | 30.00th=[ 717], 40.00th=[ 717], 50.00th=[ 50070], 60.00th=[ 50070], | 70.00th=[ 84411], 80.00th=[ 84411], 90.00th=[834667], 95.00th=[834667], | 99.00th=[834667], 99.50th=[834667], 99.90th=[834667], 99.95th=[834667], | 99.99th=[834667] bw ( KiB/s): min= 2048, max= 6144, per=74.12%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=2, BW=2314KiB/s (2369kB/s)(7168KiB/3098msec) clat (msec): min=24, max=1103, avg=300.03, stdev=431.56 lat (msec): min=24, max=1103, avg=300.09, stdev=431.58 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 31], | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 70], 60.00th=[ 121], | 70.00th=[ 121], 80.00th=[ 709], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 2048, max= 6144, per=52.95%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=33.33%, 100=16.67%, 250=8.33%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=11.75%, ctx=377, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5526KiB/s (5659kB/s), 553KiB/s-2390KiB/s (566kB/s-2447kB/s), io=20.0MiB (20.0MB), run=2571-3706msec WRITE: bw=7737KiB/s (7922kB/s), 1613KiB/s-2763KiB/s (1651kB/s-2829kB/s), io=28.0MiB (29.4MB), run=2571-3706msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6172: Wed Dec 25 00:58:49 2024 read: IOPS=4, BW=19.1MiB/s (20.0MB/s)(12.0MiB/629msec) clat (msec): min=114, max=330, avg=193.32, stdev=119.18 lat (msec): min=114, max=330, avg=193.32, stdev=119.18 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 114], 10.00th=[ 114], 20.00th=[ 114], | 30.00th=[ 114], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 136], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.16%, sys=45.70%, ctx=307, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6173: Wed Dec 25 00:58:49 2024 read: IOPS=1, BW=5603KiB/s (5738kB/s)(4096KiB/731msec) clat (nsec): min=132404k, max=132404k, avg=132404403.00, stdev= 0.00 lat (nsec): min=132406k, max=132406k, avg=132405506.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 133], | 30.00th=[ 133], 40.00th=[ 133], 50.00th=[ 133], 60.00th=[ 133], | 70.00th=[ 133], 80.00th=[ 133], 90.00th=[ 133], 95.00th=[ 133], | 99.00th=[ 133], 99.50th=[ 133], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] write: IOPS=2, BW=10.9MiB/s (11.5MB/s)(8192KiB/731msec) clat (msec): min=159, max=387, avg=273.76, stdev=161.17 lat (msec): min=159, max=387, avg=273.88, stdev=161.16 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 388], | 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=31.51%, ctx=103, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6174: Wed Dec 25 00:58:49 2024 write: IOPS=4, BW=18.8MiB/s (19.7MB/s)(12.0MiB/639msec) clat (msec): min=175, max=245, avg=207.78, stdev=35.42 lat (msec): min=175, max=245, avg=208.05, stdev=35.37 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 176], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 203], 50.00th=[ 203], 60.00th=[ 203], | 70.00th=[ 247], 80.00th=[ 247], 90.00th=[ 247], 95.00th=[ 247], | 99.00th=[ 247], 99.50th=[ 247], 99.90th=[ 247], 99.95th=[ 247], | 99.99th=[ 247] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=33.07%, ctx=62, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6175: Wed Dec 25 00:58:49 2024 write: IOPS=6, BW=25.9MiB/s (27.2MB/s)(12.0MiB/463msec) clat (msec): min=97, max=172, avg=139.75, stdev=38.64 lat (msec): min=97, max=172, avg=140.08, stdev=38.65 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 97], 10.00th=[ 97], 20.00th=[ 97], | 30.00th=[ 97], 40.00th=[ 150], 50.00th=[ 150], 60.00th=[ 150], | 70.00th=[ 174], 80.00th=[ 174], 90.00th=[ 174], 95.00th=[ 174], | 99.00th=[ 174], 99.50th=[ 174], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=48.48%, ctx=115, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=21.9MiB/s (22.0MB/s), 5603KiB/s-19.1MiB/s (5738kB/s-20.0MB/s), io=16.0MiB (16.8MB), run=629-731msec WRITE: bw=43.8MiB/s (45.9MB/s), 10.9MiB/s-25.9MiB/s (11.5MB/s-27.2MB/s), io=32.0MiB (33.6MB), run=463-731msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6176: Wed Dec 25 00:58:49 2024 read: IOPS=3, BW=14.0MiB/s (15.7MB/s)(12.0MiB/802msec) clat (msec): min=76, max=434, avg=260.99, stdev=179.45 lat (msec): min=76, max=434, avg=260.99, stdev=179.45 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 77], 10.00th=[ 77], 20.00th=[ 77], | 30.00th=[ 77], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 435], 80.00th=[ 435], 90.00th=[ 435], 95.00th=[ 435], | 99.00th=[ 435], 99.50th=[ 435], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 8192, max= 8192, per=40.10%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.00%, sys=7.48%, ctx=35, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6177: Wed Dec 25 00:58:49 2024 read: IOPS=1, BW=5361KiB/s (5490kB/s)(4096KiB/764msec) clat (nsec): min=293635k, max=293635k, avg=293634594.00, stdev= 0.00 lat (nsec): min=293636k, max=293636k, avg=293635656.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 296], 10.00th=[ 296], 20.00th=[ 296], | 30.00th=[ 296], 40.00th=[ 296], 50.00th=[ 296], 60.00th=[ 296], | 70.00th=[ 296], 80.00th=[ 296], 90.00th=[ 296], 95.00th=[ 296], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 8192, max= 8192, per=40.10%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.5MiB/s (10.0MB/s)(8192KiB/764msec) clat (msec): min=60, max=398, avg=229.43, stdev=238.69 lat (msec): min=60, max=398, avg=229.69, stdev=238.56 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 61], 10.00th=[ 61], 20.00th=[ 61], | 30.00th=[ 61], 40.00th=[ 61], 50.00th=[ 61], 60.00th=[ 397], | 70.00th=[ 397], 80.00th=[ 397], 90.00th=[ 397], 95.00th=[ 397], | 99.00th=[ 397], 99.50th=[ 397], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.26%, sys=12.06%, ctx=92, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6178: Wed Dec 25 00:58:49 2024 write: IOPS=3, BW=15.4MiB/s (16.1MB/s)(12.0MiB/780msec) clat (msec): min=183, max=330, avg=254.80, stdev=73.66 lat (msec): min=183, max=330, avg=255.04, stdev=73.59 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 251], 50.00th=[ 251], 60.00th=[ 251], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 8192, max= 8192, per=19.50%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.13%, sys=11.04%, ctx=106, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6179: Wed Dec 25 00:58:49 2024 write: IOPS=5, BW=22.6MiB/s (23.7MB/s)(12.0MiB/531msec) clat (msec): min=77, max=240, avg=172.65, stdev=85.11 lat (msec): min=77, max=241, avg=172.94, stdev=85.14 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 78], 10.00th=[ 78], 20.00th=[ 78], | 30.00th=[ 78], 40.00th=[ 201], 50.00th=[ 201], 60.00th=[ 201], | 70.00th=[ 241], 80.00th=[ 241], 90.00th=[ 241], 95.00th=[ 241], | 99.00th=[ 241], 99.50th=[ 241], 99.90th=[ 241], 99.95th=[ 241], | 99.99th=[ 241] bw ( KiB/s): min=16351, max=16351, per=38.92%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=22.22%, ctx=107, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.9MiB/s (20.9MB/s), 5361KiB/s-14.0MiB/s (5490kB/s-15.7MB/s), io=16.0MiB (16.8MB), run=764-802msec WRITE: bw=41.0MiB/s (43.0MB/s), 10.5MiB/s-22.6MiB/s (10.0MB/s-23.7MB/s), io=32.0MiB (33.6MB), run=531-780msec