== sanity test 398b: DIO and buffer IO race ============== 08:09:12 (1712146152) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.06152 s, 47.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=16281: Wed Apr 3 08:09:26 2024 read: IOPS=128, BW=515KiB/s (528kB/s)(5992KiB/11628msec) clat (usec): min=47, max=151235, avg=4377.18, stdev=4436.87 lat (usec): min=48, max=151236, avg=4377.90, stdev=4437.00 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 1696], 10.00th=[ 2147], 20.00th=[ 2507], | 30.00th=[ 2868], 40.00th=[ 3261], 50.00th=[ 3851], 60.00th=[ 4490], | 70.00th=[ 5211], 80.00th=[ 5997], 90.00th=[ 7111], 95.00th=[ 8356], | 99.00th=[ 11994], 99.50th=[ 13566], 99.90th=[ 16909], 99.95th=[152044], | 99.99th=[152044] bw ( KiB/s): min= 272, max= 712, per=25.68%, avg=513.52, stdev=94.78, samples=23 iops : min= 68, max= 178, avg=128.30, stdev=23.70, samples=23 write: IOPS=135, BW=541KiB/s (554kB/s)(6296KiB/11628msec) clat (usec): min=455, max=18309, avg=3169.27, stdev=2723.93 lat (usec): min=456, max=18310, avg=3172.00, stdev=2725.98 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 506], 10.00th=[ 529], 20.00th=[ 594], | 30.00th=[ 783], 40.00th=[ 1991], 50.00th=[ 2638], 60.00th=[ 3294], | 70.00th=[ 4080], 80.00th=[ 5276], 90.00th=[ 7046], 95.00th=[ 8455], | 99.00th=[11338], 99.50th=[12387], 99.90th=[16057], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 344, max= 712, per=25.67%, avg=537.48, stdev=87.11, samples=23 iops : min= 86, max= 178, avg=134.30, stdev=21.76, samples=23 lat (usec) : 50=0.07%, 100=1.76%, 250=0.33%, 500=2.38%, 750=12.50% lat (usec) : 1000=2.73% lat (msec) : 2=4.52%, 4=36.46%, 10=36.91%, 20=2.31%, 250=0.03% cpu : usr=0.24%, sys=23.25%, ctx=5829, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16282: Wed Apr 3 08:09:26 2024 read: IOPS=124, BW=498KiB/s (510kB/s)(5940KiB/11929msec) clat (usec): min=54, max=20912, avg=4385.96, stdev=2484.32 lat (usec): min=55, max=20912, avg=4387.06, stdev=2484.13 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 1811], 10.00th=[ 2114], 20.00th=[ 2540], | 30.00th=[ 2900], 40.00th=[ 3294], 50.00th=[ 3851], 60.00th=[ 4490], | 70.00th=[ 5211], 80.00th=[ 6063], 90.00th=[ 7373], 95.00th=[ 8848], | 99.00th=[12649], 99.50th=[14091], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 264, max= 624, per=24.45%, avg=488.96, stdev=77.50, samples=23 iops : min= 66, max= 156, avg=122.22, stdev=19.37, samples=23 write: IOPS=133, BW=532KiB/s (545kB/s)(6348KiB/11929msec) clat (usec): min=388, max=148879, avg=3364.84, stdev=4631.56 lat (usec): min=389, max=148880, avg=3365.56, stdev=4631.57 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 502], 10.00th=[ 529], 20.00th=[ 603], | 30.00th=[ 791], 40.00th=[ 2073], 50.00th=[ 2704], 60.00th=[ 3326], | 70.00th=[ 4178], 80.00th=[ 5604], 90.00th=[ 7177], 95.00th=[ 8717], | 99.00th=[ 12256], 99.50th=[ 13173], 99.90th=[ 18482], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 336, max= 656, per=24.71%, avg=517.48, stdev=82.27, samples=23 iops : min= 84, max= 164, avg=129.35, stdev=20.54, samples=23 lat (usec) : 100=1.53%, 250=0.29%, 500=2.51%, 750=12.53%, 1000=2.38% lat (msec) : 2=5.01%, 4=36.52%, 10=35.81%, 20=3.32%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=22.63%, ctx=5758, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16283: Wed Apr 3 08:09:26 2024 read: IOPS=125, BW=502KiB/s (514kB/s)(6028KiB/12001msec) clat (usec): min=45, max=24439, avg=4429.87, stdev=2451.56 lat (usec): min=45, max=24439, avg=4430.48, stdev=2451.56 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 1844], 10.00th=[ 2212], 20.00th=[ 2638], | 30.00th=[ 3032], 40.00th=[ 3490], 50.00th=[ 4015], 60.00th=[ 4621], | 70.00th=[ 5211], 80.00th=[ 5932], 90.00th=[ 7242], 95.00th=[ 8455], | 99.00th=[12387], 99.50th=[15664], 99.90th=[21103], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 159, max= 656, per=24.76%, avg=495.26, stdev=99.84, samples=23 iops : min= 39, max= 164, avg=123.78, stdev=25.07, samples=23 write: IOPS=130, BW=522KiB/s (534kB/s)(6260KiB/12001msec) clat (usec): min=405, max=163991, avg=3350.85, stdev=5060.87 lat (usec): min=406, max=163992, avg=3353.40, stdev=5060.96 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 502], 10.00th=[ 529], 20.00th=[ 594], | 30.00th=[ 807], 40.00th=[ 1663], 50.00th=[ 2606], 60.00th=[ 3195], | 70.00th=[ 4047], 80.00th=[ 5342], 90.00th=[ 7439], 95.00th=[ 9503], | 99.00th=[ 12780], 99.50th=[ 15139], 99.90th=[ 26084], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 199, max= 608, per=24.43%, avg=511.61, stdev=91.89, samples=23 iops : min= 49, max= 152, avg=127.87, stdev=23.09, samples=23 lat (usec) : 50=0.03%, 100=1.46%, 250=0.39%, 500=2.41%, 750=12.27% lat (usec) : 1000=3.09% lat (msec) : 2=5.05%, 4=34.93%, 10=37.11%, 20=3.09%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=22.80%, ctx=5732, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16285: Wed Apr 3 08:09:26 2024 read: IOPS=126, BW=508KiB/s (520kB/s)(6052KiB/11923msec) clat (usec): min=45, max=159243, avg=4459.96, stdev=4734.49 lat (usec): min=45, max=159244, avg=4460.90, stdev=4734.48 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 130], 10.00th=[ 2089], 20.00th=[ 2540], | 30.00th=[ 2900], 40.00th=[ 3326], 50.00th=[ 3884], 60.00th=[ 4555], | 70.00th=[ 5145], 80.00th=[ 5866], 90.00th=[ 7242], 95.00th=[ 9241], | 99.00th=[ 13435], 99.50th=[ 16057], 99.90th=[ 20317], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 248, max= 638, per=24.87%, avg=497.30, stdev=83.67, samples=23 iops : min= 62, max= 159, avg=124.30, stdev=20.88, samples=23 write: IOPS=130, BW=523KiB/s (536kB/s)(6236KiB/11923msec) clat (usec): min=380, max=23590, avg=3267.31, stdev=2982.18 lat (usec): min=380, max=23590, avg=3269.09, stdev=2982.14 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 506], 10.00th=[ 529], 20.00th=[ 586], | 30.00th=[ 783], 40.00th=[ 1778], 50.00th=[ 2638], 60.00th=[ 3261], | 70.00th=[ 4228], 80.00th=[ 5342], 90.00th=[ 7373], 95.00th=[ 8979], | 99.00th=[12911], 99.50th=[15926], 99.90th=[18220], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 336, max= 768, per=24.56%, avg=514.35, stdev=89.21, samples=23 iops : min= 84, max= 192, avg=128.57, stdev=22.30, samples=23 lat (usec) : 50=0.16%, 100=2.08%, 250=0.36%, 500=2.21%, 750=12.43% lat (usec) : 1000=2.93% lat (msec) : 2=4.88%, 4=34.96%, 10=36.46%, 20=3.42%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=22.60%, ctx=5630, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2001KiB/s (2049kB/s), 498KiB/s-515KiB/s (510kB/s-528kB/s), io=23.4MiB (24.6MB), run=11628-12001msec WRITE: bw=2095KiB/s (2145kB/s), 522KiB/s-541KiB/s (534kB/s-554kB/s), io=24.6MiB (25.7MB), run=11628-12001msec rand-rw: (groupid=0, jobs=1): err= 0: pid=16284: Wed Apr 3 08:09:30 2024 read: IOPS=99, BW=397KiB/s (407kB/s)(5992KiB/15087msec) clat (usec): min=1512, max=153718, avg=4070.69, stdev=4825.29 lat (usec): min=1513, max=153721, avg=4071.41, stdev=4825.29 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1778], 10.00th=[ 1860], 20.00th=[ 1991], | 30.00th=[ 2180], 40.00th=[ 2409], 50.00th=[ 2769], 60.00th=[ 3261], | 70.00th=[ 4113], 80.00th=[ 5604], 90.00th=[ 8291], 95.00th=[ 10159], | 99.00th=[ 14222], 99.50th=[ 17957], 99.90th=[ 21627], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 176, max= 896, per=25.55%, avg=395.30, stdev=184.45, samples=30 iops : min= 44, max= 224, avg=98.70, stdev=46.17, samples=30 write: IOPS=104, BW=417KiB/s (427kB/s)(6296KiB/15087msec) clat (usec): min=2027, max=27002, avg=5666.81, stdev=3546.85 lat (usec): min=2028, max=27003, avg=5667.57, stdev=3546.88 clat percentiles (usec): | 1.00th=[ 2343], 5.00th=[ 2606], 10.00th=[ 2802], 20.00th=[ 3064], | 30.00th=[ 3294], 40.00th=[ 3621], 50.00th=[ 4113], 60.00th=[ 5145], | 70.00th=[ 6652], 80.00th=[ 8029], 90.00th=[10421], 95.00th=[12387], | 99.00th=[19268], 99.50th=[22152], 99.90th=[25822], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 184, max= 864, per=25.62%, avg=415.03, stdev=185.49, samples=30 iops : min= 46, max= 216, avg=103.63, stdev=46.44, samples=30 lat (msec) : 2=9.93%, 4=48.21%, 10=33.46%, 20=7.88%, 50=0.49% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=11.12%, ctx=3464, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16286: Wed Apr 3 08:09:30 2024 read: IOPS=98, BW=394KiB/s (403kB/s)(5940KiB/15091msec) clat (usec): min=1502, max=159711, avg=3962.47, stdev=4937.91 lat (usec): min=1503, max=159713, avg=3963.10, stdev=4937.93 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1795], 10.00th=[ 1860], 20.00th=[ 2008], | 30.00th=[ 2147], 40.00th=[ 2376], 50.00th=[ 2671], 60.00th=[ 3064], | 70.00th=[ 3818], 80.00th=[ 5211], 90.00th=[ 8094], 95.00th=[ 10159], | 99.00th=[ 14615], 99.50th=[ 16581], 99.90th=[ 20055], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 111, max= 792, per=25.36%, avg=392.33, stdev=188.29, samples=30 iops : min= 27, max= 198, avg=98.00, stdev=47.07, samples=30 write: IOPS=105, BW=421KiB/s (431kB/s)(6348KiB/15091msec) clat (usec): min=2038, max=33114, avg=5756.77, stdev=3606.01 lat (usec): min=2039, max=33115, avg=5757.64, stdev=3606.09 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2606], 10.00th=[ 2835], 20.00th=[ 3064], | 30.00th=[ 3359], 40.00th=[ 3687], 50.00th=[ 4293], 60.00th=[ 5407], | 70.00th=[ 6849], 80.00th=[ 8291], 90.00th=[10290], 95.00th=[12387], | 99.00th=[19006], 99.50th=[23725], 99.90th=[26870], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 191, max= 800, per=25.78%, avg=417.67, stdev=182.19, samples=30 iops : min= 47, max= 200, avg=104.33, stdev=45.53, samples=30 lat (msec) : 2=9.34%, 4=49.28%, 10=32.98%, 20=7.94%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=11.17%, ctx=3414, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16287: Wed Apr 3 08:09:30 2024 read: IOPS=97, BW=389KiB/s (398kB/s)(6028KiB/15515msec) clat (usec): min=1523, max=22269, avg=4145.50, stdev=3153.95 lat (usec): min=1524, max=22270, avg=4146.16, stdev=3153.94 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1778], 10.00th=[ 1860], 20.00th=[ 2008], | 30.00th=[ 2180], 40.00th=[ 2442], 50.00th=[ 2769], 60.00th=[ 3195], | 70.00th=[ 4146], 80.00th=[ 6063], 90.00th=[ 8979], 95.00th=[10814], | 99.00th=[15139], 99.50th=[18744], 99.90th=[21103], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 112, max= 848, per=25.08%, avg=387.97, stdev=192.45, samples=30 iops : min= 28, max= 212, avg=96.97, stdev=48.11, samples=30 write: IOPS=100, BW=403KiB/s (413kB/s)(6260KiB/15515msec) clat (msec): min=2, max=162, avg= 5.88, stdev= 5.35 lat (msec): min=2, max=162, avg= 5.88, stdev= 5.35 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 20], 99.50th=[ 22], 99.90th=[ 26], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 152, max= 848, per=24.72%, avg=400.50, stdev=192.93, samples=30 iops : min= 38, max= 212, avg=100.10, stdev=48.26, samples=30 lat (msec) : 2=9.54%, 4=48.34%, 10=31.90%, 20=9.67%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=10.93%, ctx=3483, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16288: Wed Apr 3 08:09:30 2024 read: IOPS=99, BW=400KiB/s (409kB/s)(6052KiB/15140msec) clat (usec): min=1489, max=18422, avg=4006.12, stdev=2993.44 lat (usec): min=1489, max=18423, avg=4006.76, stdev=2993.42 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1778], 10.00th=[ 1860], 20.00th=[ 1991], | 30.00th=[ 2147], 40.00th=[ 2376], 50.00th=[ 2704], 60.00th=[ 3163], | 70.00th=[ 4015], 80.00th=[ 5669], 90.00th=[ 8717], 95.00th=[10814], | 99.00th=[14615], 99.50th=[16319], 99.90th=[18482], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 119, max= 862, per=25.60%, avg=396.03, stdev=188.28, samples=30 iops : min= 29, max= 215, avg=98.90, stdev=47.06, samples=30 write: IOPS=102, BW=412KiB/s (422kB/s)(6236KiB/15140msec) clat (msec): min=2, max=165, avg= 5.78, stdev= 5.37 lat (msec): min=2, max=165, avg= 5.78, stdev= 5.37 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 19], 99.50th=[ 21], 99.90th=[ 33], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 159, max= 816, per=25.27%, avg=409.37, stdev=186.58, samples=30 iops : min= 39, max= 204, avg=102.23, stdev=46.65, samples=30 lat (msec) : 2=10.25%, 4=48.08%, 10=32.81%, 20=8.56%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=11.25%, ctx=3466, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1548KiB/s (1585kB/s), 389KiB/s-400KiB/s (398kB/s-409kB/s), io=23.4MiB (24.6MB), run=15087-15515msec WRITE: bw=1620KiB/s (1659kB/s), 403KiB/s-421KiB/s (413kB/s-431kB/s), io=24.6MiB (25.7MB), run=15087-15515msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=16316: Wed Apr 3 08:09:36 2024 read: IOPS=74, BW=1194KiB/s (1223kB/s)(5920KiB/4957msec) clat (usec): min=2100, max=22866, avg=5229.71, stdev=3524.05 lat (usec): min=2101, max=22866, avg=5230.31, stdev=3524.05 clat percentiles (usec): | 1.00th=[ 2147], 5.00th=[ 2245], 10.00th=[ 2343], 20.00th=[ 2769], | 30.00th=[ 2999], 40.00th=[ 3261], 50.00th=[ 3916], 60.00th=[ 4555], | 70.00th=[ 5604], 80.00th=[ 7046], 90.00th=[10552], 95.00th=[12387], | 99.00th=[18220], 99.50th=[20841], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 928, max= 1344, per=25.10%, avg=1154.56, stdev=139.01, samples=9 iops : min= 58, max= 84, avg=71.89, stdev= 8.71, samples=9 write: IOPS=80, BW=1285KiB/s (1315kB/s)(6368KiB/4957msec) clat (usec): min=2741, max=24525, avg=7576.39, stdev=4478.49 lat (usec): min=2742, max=24526, avg=7577.60, stdev=4478.49 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3458], 10.00th=[ 3654], 20.00th=[ 3949], | 30.00th=[ 4424], 40.00th=[ 5080], 50.00th=[ 5866], 60.00th=[ 6980], | 70.00th=[ 8586], 80.00th=[11076], 90.00th=[14615], 95.00th=[17695], | 99.00th=[20579], 99.50th=[22676], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 1056, max= 1408, per=26.00%, avg=1257.67, stdev=111.69, samples=9 iops : min= 66, max= 88, avg=78.33, stdev= 6.91, samples=9 lat (msec) : 4=35.42%, 10=46.61%, 20=16.80%, 50=1.17% cpu : usr=0.06%, sys=10.53%, ctx=886, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16317: Wed Apr 3 08:09:36 2024 read: IOPS=68, BW=1103KiB/s (1129kB/s)(5744KiB/5208msec) clat (usec): min=1826, max=21105, avg=5587.17, stdev=3771.14 lat (usec): min=1827, max=21106, avg=5587.78, stdev=3771.15 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2245], 10.00th=[ 2409], 20.00th=[ 2704], | 30.00th=[ 2999], 40.00th=[ 3326], 50.00th=[ 3884], 60.00th=[ 4948], | 70.00th=[ 6587], 80.00th=[ 8717], 90.00th=[11600], 95.00th=[13566], | 99.00th=[18220], 99.50th=[21103], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 896, max= 1376, per=24.00%, avg=1103.80, stdev=158.55, samples=10 iops : min= 56, max= 86, avg=68.90, stdev= 9.98, samples=10 write: IOPS=78, BW=1257KiB/s (1287kB/s)(6544KiB/5208msec) clat (usec): min=2721, max=30026, avg=7813.52, stdev=4441.26 lat (usec): min=2722, max=30027, avg=7814.86, stdev=4441.58 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3294], 10.00th=[ 3621], 20.00th=[ 4146], | 30.00th=[ 4752], 40.00th=[ 5604], 50.00th=[ 6587], 60.00th=[ 7701], | 70.00th=[ 8979], 80.00th=[10945], 90.00th=[14091], 95.00th=[16450], | 99.00th=[22152], 99.50th=[25822], 99.90th=[30016], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 1088, max= 1408, per=25.59%, avg=1238.10, stdev=106.99, samples=10 iops : min= 68, max= 88, avg=77.30, stdev= 6.77, samples=10 lat (msec) : 2=0.13%, 4=33.59%, 10=47.27%, 20=17.45%, 50=1.56% cpu : usr=0.17%, sys=9.99%, ctx=900, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16318: Wed Apr 3 08:09:36 2024 read: IOPS=78, BW=1250KiB/s (1280kB/s)(6256KiB/5003msec) clat (usec): min=2008, max=31145, avg=5471.16, stdev=4124.22 lat (usec): min=2008, max=31146, avg=5471.83, stdev=4124.26 clat percentiles (usec): | 1.00th=[ 2089], 5.00th=[ 2278], 10.00th=[ 2442], 20.00th=[ 2671], | 30.00th=[ 2966], 40.00th=[ 3359], 50.00th=[ 3818], 60.00th=[ 4686], | 70.00th=[ 5932], 80.00th=[ 7635], 90.00th=[10552], 95.00th=[13304], | 99.00th=[26084], 99.50th=[27132], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 958, max= 1536, per=27.20%, avg=1251.00, stdev=231.75, samples=10 iops : min= 59, max= 96, avg=78.10, stdev=14.61, samples=10 write: IOPS=75, BW=1206KiB/s (1235kB/s)(6032KiB/5003msec) clat (usec): min=2880, max=41240, avg=7578.20, stdev=4894.88 lat (usec): min=2882, max=41240, avg=7579.38, stdev=4894.88 clat percentiles (usec): | 1.00th=[ 3195], 5.00th=[ 3458], 10.00th=[ 3621], 20.00th=[ 3982], | 30.00th=[ 4424], 40.00th=[ 4948], 50.00th=[ 6128], 60.00th=[ 7046], | 70.00th=[ 8225], 80.00th=[10028], 90.00th=[14091], 95.00th=[17957], | 99.00th=[24249], 99.50th=[27657], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 896, max= 1440, per=24.86%, avg=1202.90, stdev=158.20, samples=10 iops : min= 56, max= 90, avg=75.10, stdev= 9.85, samples=10 lat (msec) : 4=37.50%, 10=47.14%, 20=12.89%, 50=2.47% cpu : usr=0.18%, sys=10.44%, ctx=888, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16319: Wed Apr 3 08:09:36 2024 read: IOPS=74, BW=1186KiB/s (1215kB/s)(6032KiB/5085msec) clat (usec): min=2018, max=22111, avg=5353.09, stdev=3535.67 lat (usec): min=2018, max=22111, avg=5353.75, stdev=3535.65 clat percentiles (usec): | 1.00th=[ 2073], 5.00th=[ 2278], 10.00th=[ 2409], 20.00th=[ 2737], | 30.00th=[ 2966], 40.00th=[ 3425], 50.00th=[ 3818], 60.00th=[ 4686], | 70.00th=[ 5932], 80.00th=[ 8160], 90.00th=[10945], 95.00th=[11994], | 99.00th=[17171], 99.50th=[21365], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 864, max= 1536, per=25.67%, avg=1180.50, stdev=204.61, samples=10 iops : min= 54, max= 96, avg=73.70, stdev=12.72, samples=10 write: IOPS=76, BW=1230KiB/s (1260kB/s)(6256KiB/5085msec) clat (usec): min=2850, max=44677, avg=7828.42, stdev=5077.60 lat (usec): min=2851, max=44678, avg=7829.62, stdev=5077.59 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3425], 10.00th=[ 3589], 20.00th=[ 4047], | 30.00th=[ 4359], 40.00th=[ 4817], 50.00th=[ 5932], 60.00th=[ 7177], | 70.00th=[ 9110], 80.00th=[11338], 90.00th=[14484], 95.00th=[17695], | 99.00th=[23462], 99.50th=[27919], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 1056, max= 1405, per=25.46%, avg=1231.70, stdev=117.56, samples=10 iops : min= 66, max= 87, avg=76.90, stdev= 7.22, samples=10 lat (msec) : 4=36.07%, 10=43.75%, 20=18.23%, 50=1.95% cpu : usr=0.22%, sys=10.27%, ctx=902, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4599KiB/s (4709kB/s), 1103KiB/s-1250KiB/s (1129kB/s-1280kB/s), io=23.4MiB (24.5MB), run=4957-5208msec WRITE: bw=4839KiB/s (4955kB/s), 1206KiB/s-1285KiB/s (1235kB/s-1315kB/s), io=24.6MiB (25.8MB), run=4957-5208msec rand-rw: (groupid=0, jobs=1): err= 0: pid=16320: Wed Apr 3 08:09:36 2024 read: IOPS=68, BW=1099KiB/s (1126kB/s)(5920KiB/5385msec) clat (usec): min=47, max=27570, avg=10565.28, stdev=4435.12 lat (usec): min=47, max=27571, avg=10565.97, stdev=4435.16 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 71], 10.00th=[ 6128], 20.00th=[ 8455], | 30.00th=[ 9634], 40.00th=[10421], 50.00th=[10945], 60.00th=[11469], | 70.00th=[12387], 80.00th=[13435], 90.00th=[15139], 95.00th=[16450], | 99.00th=[20841], 99.50th=[25822], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 896, max= 1216, per=23.67%, avg=1027.00, stdev=88.79, samples=10 iops : min= 56, max= 76, avg=64.10, stdev= 5.67, samples=10 write: IOPS=73, BW=1183KiB/s (1211kB/s)(6368KiB/5385msec) clat (usec): min=639, max=27777, avg=3691.46, stdev=3971.45 lat (usec): min=639, max=27778, avg=3692.42, stdev=3971.45 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 750], 10.00th=[ 791], 20.00th=[ 889], | 30.00th=[ 1270], 40.00th=[ 2245], 50.00th=[ 2606], 60.00th=[ 2966], | 70.00th=[ 3589], 80.00th=[ 4948], 90.00th=[ 8291], 95.00th=[12387], | 99.00th=[22152], 99.50th=[24511], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 832, max= 1504, per=24.59%, avg=1122.90, stdev=205.33, samples=10 iops : min= 52, max= 94, avg=70.10, stdev=12.81, samples=10 lat (usec) : 50=0.26%, 100=3.52%, 250=0.65%, 750=2.86%, 1000=10.29% lat (msec) : 2=5.60%, 4=19.40%, 10=22.14%, 20=34.24%, 50=1.04% cpu : usr=0.13%, sys=19.47%, ctx=2435, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16321: Wed Apr 3 08:09:36 2024 read: IOPS=66, BW=1068KiB/s (1093kB/s)(5744KiB/5380msec) clat (usec): min=57, max=34739, avg=10826.23, stdev=4032.55 lat (usec): min=57, max=34740, avg=10826.83, stdev=4032.56 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 121], 10.00th=[ 6980], 20.00th=[ 8586], | 30.00th=[ 9372], 40.00th=[10290], 50.00th=[10814], 60.00th=[11600], | 70.00th=[12387], 80.00th=[13435], 90.00th=[15008], 95.00th=[16712], | 99.00th=[20055], 99.50th=[27657], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 798, max= 1248, per=23.45%, avg=1017.40, stdev=122.76, samples=10 iops : min= 49, max= 78, avg=63.50, stdev= 7.85, samples=10 write: IOPS=76, BW=1216KiB/s (1246kB/s)(6544KiB/5380msec) clat (usec): min=629, max=23348, avg=3630.88, stdev=3524.88 lat (usec): min=630, max=23349, avg=3631.91, stdev=3524.88 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 758], 10.00th=[ 791], 20.00th=[ 898], | 30.00th=[ 1319], 40.00th=[ 2409], 50.00th=[ 2737], 60.00th=[ 3064], | 70.00th=[ 3556], 80.00th=[ 4948], 90.00th=[ 8291], 95.00th=[11469], | 99.00th=[16712], 99.50th=[17695], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 864, max= 1664, per=25.08%, avg=1145.30, stdev=275.34, samples=10 iops : min= 54, max= 104, avg=71.50, stdev=17.25, samples=10 lat (usec) : 100=1.95%, 250=0.39%, 750=2.21%, 1000=10.55% lat (msec) : 2=4.69%, 4=22.40%, 10=24.22%, 20=32.81%, 50=0.78% cpu : usr=0.11%, sys=19.50%, ctx=2485, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16322: Wed Apr 3 08:09:36 2024 read: IOPS=70, BW=1134KiB/s (1161kB/s)(6256KiB/5519msec) clat (usec): min=52, max=26362, avg=10440.66, stdev=4349.44 lat (usec): min=52, max=26362, avg=10441.26, stdev=4349.45 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 84], 10.00th=[ 6587], 20.00th=[ 8455], | 30.00th=[ 9372], 40.00th=[10290], 50.00th=[10945], 60.00th=[11600], | 70.00th=[12256], 80.00th=[13173], 90.00th=[14877], 95.00th=[16188], | 99.00th=[20841], 99.50th=[23200], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 736, max= 1184, per=24.04%, avg=1042.90, stdev=147.01, samples=10 iops : min= 46, max= 74, avg=65.10, stdev= 9.15, samples=10 write: IOPS=68, BW=1093KiB/s (1119kB/s)(6032KiB/5519msec) clat (usec): min=639, max=21863, avg=3788.80, stdev=3749.07 lat (usec): min=640, max=21865, avg=3789.81, stdev=3749.06 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 734], 10.00th=[ 799], 20.00th=[ 955], | 30.00th=[ 1303], 40.00th=[ 2311], 50.00th=[ 2737], 60.00th=[ 3163], | 70.00th=[ 3982], 80.00th=[ 5407], 90.00th=[ 8225], 95.00th=[12780], | 99.00th=[19792], 99.50th=[21627], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 704, max= 1408, per=22.63%, avg=1033.40, stdev=215.72, samples=10 iops : min= 44, max= 88, avg=64.50, stdev=13.59, samples=10 lat (usec) : 100=3.52%, 250=1.30%, 500=0.13%, 750=2.99%, 1000=8.33% lat (msec) : 2=5.60%, 4=17.71%, 10=25.13%, 20=34.24%, 50=1.04% cpu : usr=0.14%, sys=18.96%, ctx=2514, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16323: Wed Apr 3 08:09:36 2024 read: IOPS=69, BW=1110KiB/s (1137kB/s)(6032KiB/5434msec) clat (usec): min=45, max=29313, avg=10545.07, stdev=4382.43 lat (usec): min=45, max=29314, avg=10545.68, stdev=4382.44 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 75], 10.00th=[ 5932], 20.00th=[ 8291], | 30.00th=[ 9372], 40.00th=[10028], 50.00th=[10945], 60.00th=[11731], | 70.00th=[12649], 80.00th=[13829], 90.00th=[15139], 95.00th=[16450], | 99.00th=[20055], 99.50th=[22414], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 832, max= 1152, per=23.66%, avg=1026.70, stdev=106.00, samples=10 iops : min= 52, max= 72, avg=64.00, stdev= 6.65, samples=10 write: IOPS=71, BW=1151KiB/s (1179kB/s)(6256KiB/5434msec) clat (usec): min=625, max=32126, avg=3715.81, stdev=3845.15 lat (usec): min=626, max=32127, avg=3716.80, stdev=3845.16 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 750], 10.00th=[ 791], 20.00th=[ 938], | 30.00th=[ 1303], 40.00th=[ 2376], 50.00th=[ 2704], 60.00th=[ 3097], | 70.00th=[ 3720], 80.00th=[ 5145], 90.00th=[ 7504], 95.00th=[12125], | 99.00th=[21365], 99.50th=[23725], 99.90th=[32113], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 768, max= 1472, per=23.40%, avg=1068.30, stdev=199.23, samples=10 iops : min= 48, max= 92, avg=66.60, stdev=12.52, samples=10 lat (usec) : 50=0.39%, 100=3.52%, 250=0.52%, 750=2.47%, 1000=9.37% lat (msec) : 2=5.08%, 4=19.79%, 10=25.65%, 20=32.16%, 50=1.04% cpu : usr=0.09%, sys=19.44%, ctx=2516, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4340KiB/s (4444kB/s), 1068KiB/s-1134KiB/s (1093kB/s-1161kB/s), io=23.4MiB (24.5MB), run=5380-5519msec WRITE: bw=4566KiB/s (4676kB/s), 1093KiB/s-1216KiB/s (1119kB/s-1246kB/s), io=24.6MiB (25.8MB), run=5380-5519msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=16337: Wed Apr 3 08:09:40 2024 read: IOPS=2, BW=2210KiB/s (2263kB/s)(7168KiB/3244msec) clat (usec): min=330, max=863231, avg=438608.31, stdev=401246.34 lat (usec): min=330, max=863232, avg=438611.93, stdev=401243.67 clat percentiles (usec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 465], | 30.00th=[ 65274], 40.00th=[ 65274], 50.00th=[574620], 60.00th=[725615], | 70.00th=[725615], 80.00th=[843056], 90.00th=[859833], 95.00th=[859833], | 99.00th=[859833], 99.50th=[859833], 99.90th=[859833], 99.95th=[859833], | 99.99th=[859833] bw ( KiB/s): min= 1796, max= 6131, per=47.77%, avg=3004.50, stdev=2087.65, samples=4 iops : min= 1, max= 5, avg= 2.25, stdev= 1.89, samples=4 write: IOPS=1, BW=1578KiB/s (1616kB/s)(5120KiB/3244msec) clat (usec): min=17298, max=44466, avg=30555.88, stdev=12357.34 lat (usec): min=17394, max=44607, avg=30631.15, stdev=12379.17 clat percentiles (usec): | 1.00th=[17171], 5.00th=[17171], 10.00th=[17171], 20.00th=[17171], | 30.00th=[23200], 40.00th=[23200], 50.00th=[24773], 60.00th=[24773], | 70.00th=[43254], 80.00th=[43254], 90.00th=[44303], 95.00th=[44303], | 99.00th=[44303], 99.50th=[44303], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 1796, max= 8175, per=56.62%, avg=4985.50, stdev=4510.63, samples=2 iops : min= 1, max= 7, avg= 4.00, stdev= 4.24, samples=2 lat (usec) : 500=16.67% lat (msec) : 20=8.33%, 50=33.33%, 100=8.33%, 750=16.67%, 1000=16.67% cpu : usr=0.12%, sys=17.64%, ctx=1223, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16338: Wed Apr 3 08:09:40 2024 read: IOPS=1, BW=1291KiB/s (1322kB/s)(2048KiB/1586msec) clat (msec): min=70, max=670, avg=370.38, stdev=423.94 lat (msec): min=70, max=670, avg=370.38, stdev=423.94 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 71], 10.00th=[ 71], 20.00th=[ 71], | 30.00th=[ 71], 40.00th=[ 71], 50.00th=[ 71], 60.00th=[ 667], | 70.00th=[ 667], 80.00th=[ 667], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 2048, max= 2048, per=32.56%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=6, BW=6456KiB/s (6611kB/s)(10.0MiB/1586msec) clat (msec): min=24, max=502, avg=83.41, stdev=147.98 lat (msec): min=24, max=502, avg=83.46, stdev=147.97 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 28], 40.00th=[ 29], 50.00th=[ 32], 60.00th=[ 35], | 70.00th=[ 50], 80.00th=[ 50], 90.00th=[ 78], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 2048, max=10240, per=62.03%, avg=5461.33, stdev=4263.25, samples=3 iops : min= 2, max= 10, avg= 5.33, stdev= 4.16, samples=3 lat (msec) : 50=66.67%, 100=16.67%, 750=16.67% cpu : usr=0.00%, sys=21.39%, ctx=322, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16339: Wed Apr 3 08:09:40 2024 read: IOPS=1, BW=1887KiB/s (1932kB/s)(6144KiB/3256msec) clat (usec): min=318, max=848753, avg=143156.97, stdev=345686.12 lat (usec): min=318, max=848754, avg=143157.80, stdev=345686.19 clat percentiles (usec): | 1.00th=[ 318], 5.00th=[ 318], 10.00th=[ 318], 20.00th=[ 367], | 30.00th=[ 367], 40.00th=[ 400], 50.00th=[ 400], 60.00th=[ 461], | 70.00th=[ 8586], 80.00th=[ 8586], 90.00th=[851444], 95.00th=[851444], | 99.00th=[851444], 99.50th=[851444], 99.90th=[851444], 99.95th=[851444], | 99.99th=[851444] bw ( KiB/s): min= 9126, max= 9126, per=100.00%, avg=9126.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1887KiB/s (1932kB/s)(6144KiB/3256msec) clat (msec): min=19, max=1463, avg=397.46, stdev=598.63 lat (msec): min=19, max=1464, avg=397.60, stdev=598.57 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 41], 50.00th=[ 41], 60.00th=[ 62], | 70.00th=[ 768], 80.00th=[ 768], 90.00th=[ 1469], 95.00th=[ 1469], | 99.00th=[ 1469], 99.50th=[ 1469], 99.90th=[ 1469], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 1825, max= 5475, per=43.14%, avg=3798.67, stdev=1843.08, samples=3 iops : min= 1, max= 5, avg= 3.33, stdev= 2.08, samples=3 lat (usec) : 500=33.33% lat (msec) : 10=8.33%, 20=8.33%, 50=16.67%, 100=8.33%, 1000=16.67% cpu : usr=0.00%, sys=8.76%, ctx=333, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16340: Wed Apr 3 08:09:40 2024 read: IOPS=2, BW=2801KiB/s (2868kB/s)(5120KiB/1828msec) clat (usec): min=267, max=656, avg=398.10, stdev=170.43 lat (usec): min=267, max=657, avg=398.78, stdev=170.70 clat percentiles (usec): | 1.00th=[ 269], 5.00th=[ 269], 10.00th=[ 269], 20.00th=[ 269], | 30.00th=[ 289], 40.00th=[ 289], 50.00th=[ 289], 60.00th=[ 289], | 70.00th=[ 490], 80.00th=[ 490], 90.00th=[ 660], 95.00th=[ 660], | 99.00th=[ 660], 99.50th=[ 660], 99.90th=[ 660], 99.95th=[ 660], | 99.99th=[ 660] write: IOPS=3, BW=3921KiB/s (4015kB/s)(7168KiB/1828msec) clat (msec): min=19, max=1488, avg=257.99, stdev=543.25 lat (msec): min=19, max=1488, avg=258.04, stdev=543.26 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 24], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 57], 60.00th=[ 67], | 70.00th=[ 67], 80.00th=[ 109], 90.00th=[ 1485], 95.00th=[ 1485], | 99.00th=[ 1485], 99.50th=[ 1485], 99.90th=[ 1485], 99.95th=[ 1485], | 99.99th=[ 1485] bw ( KiB/s): min= 2048, max= 2048, per=23.26%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 500=33.33%, 750=8.33% lat (msec) : 20=8.33%, 50=16.67%, 100=16.67%, 250=8.33% cpu : usr=0.11%, sys=8.37%, ctx=39, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6290KiB/s (6441kB/s), 1291KiB/s-2801KiB/s (1322kB/s-2868kB/s), io=20.0MiB (20.0MB), run=1586-3256msec WRITE: bw=8806KiB/s (9017kB/s), 1578KiB/s-6456KiB/s (1616kB/s-6611kB/s), io=28.0MiB (29.4MB), run=1586-3256msec rand-rw: (groupid=0, jobs=1): err= 0: pid=16332: Wed Apr 3 08:09:40 2024 read: IOPS=16, BW=16.4MiB/s (17.1MB/s)(7168KiB/428msec) clat (usec): min=20431, max=73665, avg=37254.34, stdev=19076.85 lat (usec): min=20432, max=73668, avg=37255.55, stdev=19077.19 clat percentiles (usec): | 1.00th=[20317], 5.00th=[20317], 10.00th=[20317], 20.00th=[25560], | 30.00th=[26870], 40.00th=[26870], 50.00th=[29492], 60.00th=[32113], | 70.00th=[32113], 80.00th=[52691], 90.00th=[73925], 95.00th=[73925], | 99.00th=[73925], 99.50th=[73925], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] write: IOPS=11, BW=11.7MiB/s (12.2MB/s)(5120KiB/428msec) clat (usec): min=22049, max=44423, avg=31200.37, stdev=10565.53 lat (usec): min=22075, max=44463, avg=31252.33, stdev=10557.87 clat percentiles (usec): | 1.00th=[22152], 5.00th=[22152], 10.00th=[22152], 20.00th=[22152], | 30.00th=[22938], 40.00th=[22938], 50.00th=[25822], 60.00th=[25822], | 70.00th=[40633], 80.00th=[40633], 90.00th=[44303], 95.00th=[44303], | 99.00th=[44303], 99.50th=[44303], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] lat (msec) : 50=83.33%, 100=16.67% cpu : usr=0.00%, sys=29.27%, ctx=67, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16334: Wed Apr 3 08:09:40 2024 read: IOPS=1, BW=1060KiB/s (1085kB/s)(2048KiB/1932msec) clat (msec): min=79, max=1413, avg=746.75, stdev=943.32 lat (msec): min=79, max=1413, avg=746.75, stdev=943.32 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 81], 10.00th=[ 81], 20.00th=[ 81], | 30.00th=[ 81], 40.00th=[ 81], 50.00th=[ 81], 60.00th=[ 1418], | 70.00th=[ 1418], 80.00th=[ 1418], 90.00th=[ 1418], 95.00th=[ 1418], | 99.00th=[ 1418], 99.50th=[ 1418], 99.90th=[ 1418], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 2048, max= 2048, per=33.39%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5300KiB/s (5427kB/s)(10.0MiB/1932msec) clat (msec): min=20, max=100, avg=42.67, stdev=26.79 lat (msec): min=20, max=100, avg=42.76, stdev=26.79 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 28], 50.00th=[ 29], 60.00th=[ 30], | 70.00th=[ 67], 80.00th=[ 67], 90.00th=[ 70], 95.00th=[ 102], | 99.00th=[ 102], 99.50th=[ 102], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=25.00%, 250=8.33% cpu : usr=0.10%, sys=7.15%, ctx=60, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16335: Wed Apr 3 08:09:40 2024 read: IOPS=1, BW=1840KiB/s (1884kB/s)(6144KiB/3339msec) clat (msec): min=20, max=793, avg=170.91, stdev=305.71 lat (msec): min=20, max=793, avg=170.91, stdev=305.71 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 42], 50.00th=[ 42], 60.00th=[ 63], | 70.00th=[ 83], 80.00th=[ 83], 90.00th=[ 793], 95.00th=[ 793], | 99.00th=[ 793], 99.50th=[ 793], 99.90th=[ 793], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 9411, max= 9411, per=100.00%, avg=9411.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1840KiB/s (1884kB/s)(6144KiB/3339msec) clat (msec): min=29, max=1461, avg=384.44, stdev=575.97 lat (msec): min=29, max=1461, avg=384.49, stdev=575.98 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 65], 50.00th=[ 65], 60.00th=[ 82], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 1469], 95.00th=[ 1469], | 99.00th=[ 1469], 99.50th=[ 1469], 99.90th=[ 1469], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 1878, max= 5647, per=45.11%, avg=3873.67, stdev=1894.31, samples=3 iops : min= 1, max= 5, avg= 3.33, stdev= 2.08, samples=3 lat (msec) : 50=41.67%, 100=33.33%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=6.53%, ctx=86, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16336: Wed Apr 3 08:09:40 2024 read: IOPS=4, BW=5000KiB/s (5120kB/s)(5120KiB/1024msec) clat (msec): min=18, max=662, avg=153.92, stdev=284.27 lat (msec): min=18, max=662, avg=153.92, stdev=284.27 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 20], | 30.00th=[ 23], 40.00th=[ 23], 50.00th=[ 24], 60.00th=[ 24], | 70.00th=[ 43], 80.00th=[ 43], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=6, BW=7000KiB/s (7168kB/s)(7168KiB/1024msec) clat (usec): min=20656, max=83892, avg=35342.31, stdev=22978.98 lat (usec): min=20736, max=83928, avg=35399.35, stdev=22976.27 clat percentiles (usec): | 1.00th=[20579], 5.00th=[20579], 10.00th=[20579], 20.00th=[21103], | 30.00th=[21890], 40.00th=[21890], 50.00th=[22414], 60.00th=[35390], | 70.00th=[35390], 80.00th=[41681], 90.00th=[84411], 95.00th=[84411], | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min=14307, max=14307, per=100.00%, avg=14307.00, stdev= 0.00, samples=1 iops : min= 13, max= 13, avg=13.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=75.00%, 100=8.33%, 750=8.33% cpu : usr=0.00%, sys=10.75%, ctx=45, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6134KiB/s (6281kB/s), 1060KiB/s-16.4MiB/s (1085kB/s-17.1MB/s), io=20.0MiB (20.0MB), run=428-3339msec WRITE: bw=8587KiB/s (8793kB/s), 1840KiB/s-11.7MiB/s (1884kB/s-12.2MB/s), io=28.0MiB (29.4MB), run=428-3339msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=16348: Wed Apr 3 08:09:43 2024 read: IOPS=1, BW=7144KiB/s (7316kB/s)(12.0MiB/1720msec) clat (msec): min=195, max=770, avg=573.08, stdev=326.91 lat (msec): min=195, max=770, avg=573.08, stdev=326.91 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 197], 10.00th=[ 197], 20.00th=[ 197], | 30.00th=[ 197], 40.00th=[ 751], 50.00th=[ 751], 60.00th=[ 751], | 70.00th=[ 768], 80.00th=[ 768], 90.00th=[ 768], 95.00th=[ 768], | 99.00th=[ 768], 99.50th=[ 768], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 5490, max= 5490, per=63.70%, avg=5490.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 1000=66.67% cpu : usr=0.00%, sys=23.44%, ctx=671, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16349: Wed Apr 3 08:09:43 2024 read: IOPS=0, BW=2155KiB/s (2206kB/s)(4096KiB/1901msec) clat (nsec): min=154377k, max=154377k, avg=154377130.00, stdev= 0.00 lat (nsec): min=154379k, max=154379k, avg=154379000.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 155], | 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 8192, max= 8192, per=95.06%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4309KiB/s (4413kB/s)(8192KiB/1901msec) clat (msec): min=175, max=1566, avg=871.10, stdev=984.02 lat (msec): min=175, max=1567, avg=871.37, stdev=984.04 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 176], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 176], 50.00th=[ 176], 60.00th=[ 1569], | 70.00th=[ 1569], 80.00th=[ 1569], 90.00th=[ 1569], 95.00th=[ 1569], | 99.00th=[ 1569], 99.50th=[ 1569], 99.90th=[ 1569], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 8192, max= 8192, per=47.53%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.05%, sys=15.53%, ctx=122, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16350: Wed Apr 3 08:09:43 2024 write: IOPS=1, BW=6857KiB/s (7022kB/s)(12.0MiB/1792msec) clat (msec): min=186, max=887, avg=593.48, stdev=364.06 lat (msec): min=186, max=887, avg=593.76, stdev=364.04 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 186], 10.00th=[ 186], 20.00th=[ 186], | 30.00th=[ 186], 40.00th=[ 709], 50.00th=[ 709], 60.00th=[ 709], | 70.00th=[ 885], 80.00th=[ 885], 90.00th=[ 885], 95.00th=[ 885], | 99.00th=[ 885], 99.50th=[ 885], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 8000, max= 8015, per=46.46%, avg=8007.50, stdev=10.61, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.06%, sys=14.74%, ctx=39, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16352: Wed Apr 3 08:09:43 2024 write: IOPS=5, BW=23.1MiB/s (24.2MB/s)(12.0MiB/519msec) clat (msec): min=96, max=254, avg=158.69, stdev=83.86 lat (msec): min=97, max=254, avg=158.92, stdev=83.79 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 97], 10.00th=[ 97], 20.00th=[ 97], | 30.00th=[ 97], 40.00th=[ 126], 50.00th=[ 126], 60.00th=[ 126], | 70.00th=[ 255], 80.00th=[ 255], 90.00th=[ 255], 95.00th=[ 255], | 99.00th=[ 255], 99.50th=[ 255], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min=16384, max=16384, per=95.05%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=50.58%, ctx=43, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8619KiB/s (8825kB/s), 2155KiB/s-7144KiB/s (2206kB/s-7316kB/s), io=16.0MiB (16.8MB), run=1720-1901msec WRITE: bw=16.8MiB/s (17.7MB/s), 4309KiB/s-23.1MiB/s (4413kB/s-24.2MB/s), io=32.0MiB (33.6MB), run=519-1901msec rand-rw: (groupid=0, jobs=1): err= 0: pid=16351: Wed Apr 3 08:09:43 2024 read: IOPS=1, BW=6387KiB/s (6540kB/s)(12.0MiB/1924msec) clat (msec): min=167, max=1478, avg=640.61, stdev=727.97 lat (msec): min=167, max=1478, avg=640.61, stdev=727.97 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 167], | 30.00th=[ 167], 40.00th=[ 275], 50.00th=[ 275], 60.00th=[ 275], | 70.00th=[ 1485], 80.00th=[ 1485], 90.00th=[ 1485], 95.00th=[ 1485], | 99.00th=[ 1485], 99.50th=[ 1485], 99.90th=[ 1485], 99.95th=[ 1485], | 99.99th=[ 1485] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=7.12%, ctx=70, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16353: Wed Apr 3 08:09:43 2024 read: IOPS=0, BW=1997KiB/s (2045kB/s)(4096KiB/2051msec) clat (nsec): min=376156k, max=376156k, avg=376155730.00, stdev= 0.00 lat (nsec): min=376156k, max=376156k, avg=376156440.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 376], 10.00th=[ 376], 20.00th=[ 376], | 30.00th=[ 376], 40.00th=[ 376], 50.00th=[ 376], 60.00th=[ 376], | 70.00th=[ 376], 80.00th=[ 376], 90.00th=[ 376], 95.00th=[ 376], | 99.00th=[ 376], 99.50th=[ 376], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3994KiB/s (4090kB/s)(8192KiB/2051msec) clat (msec): min=110, max=1562, avg=836.67, stdev=1026.99 lat (msec): min=110, max=1562, avg=836.80, stdev=1026.99 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 111], 50.00th=[ 111], 60.00th=[ 1569], | 70.00th=[ 1569], 80.00th=[ 1569], 90.00th=[ 1569], 95.00th=[ 1569], | 99.00th=[ 1569], 99.50th=[ 1569], 99.90th=[ 1569], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 8192, max= 8192, per=51.28%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=5.80%, ctx=41, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16354: Wed Apr 3 08:09:43 2024 write: IOPS=1, BW=6012KiB/s (6156kB/s)(12.0MiB/2044msec) clat (msec): min=52, max=1657, avg=680.99, stdev=857.46 lat (msec): min=52, max=1658, avg=681.16, stdev=857.57 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 334], 50.00th=[ 334], 60.00th=[ 334], | 70.00th=[ 1653], 80.00th=[ 1653], 90.00th=[ 1653], 95.00th=[ 1653], | 99.00th=[ 1653], 99.50th=[ 1653], 99.90th=[ 1653], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 8078, max= 8192, per=50.92%, avg=8135.00, stdev=80.61, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=5.97%, ctx=55, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16355: Wed Apr 3 08:09:43 2024 write: IOPS=2, BW=9526KiB/s (9754kB/s)(12.0MiB/1290msec) clat (msec): min=106, max=951, avg=429.07, stdev=456.71 lat (msec): min=106, max=951, avg=429.29, stdev=456.75 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 230], 50.00th=[ 230], 60.00th=[ 230], | 70.00th=[ 953], 80.00th=[ 953], 90.00th=[ 953], 95.00th=[ 953], | 99.00th=[ 953], 99.50th=[ 953], 99.90th=[ 953], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 8192, max= 8192, per=51.28%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.00%, sys=12.49%, ctx=178, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7988KiB/s (8180kB/s), 1997KiB/s-6387KiB/s (2045kB/s-6540kB/s), io=16.0MiB (16.8MB), run=1924-2051msec WRITE: bw=15.6MiB/s (16.4MB/s), 3994KiB/s-9526KiB/s (4090kB/s-9754kB/s), io=32.0MiB (33.6MB), run=1290-2051msec