== sanity test 398b: DIO and buffer IO race ============== 01:25:24 (1735107924) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.26065 s, 39.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25888: Wed Dec 25 01:25:40 2024 read: IOPS=110, BW=443KiB/s (453kB/s)(5992KiB/13531msec) clat (usec): min=52, max=194073, avg=5090.17, stdev=5612.68 lat (usec): min=52, max=194073, avg=5091.26, stdev=5612.57 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 163], 10.00th=[ 2409], 20.00th=[ 2900], | 30.00th=[ 3359], 40.00th=[ 3916], 50.00th=[ 4555], 60.00th=[ 5276], | 70.00th=[ 6063], 80.00th=[ 6849], 90.00th=[ 8291], 95.00th=[ 9765], | 99.00th=[ 15401], 99.50th=[ 17433], 99.90th=[ 22676], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 216, max= 552, per=25.01%, avg=434.69, stdev=76.54, samples=26 iops : min= 54, max= 138, avg=108.62, stdev=19.17, samples=26 write: IOPS=116, BW=465KiB/s (476kB/s)(6296KiB/13531msec) clat (usec): min=491, max=19565, avg=3699.89, stdev=3103.75 lat (usec): min=492, max=19566, avg=3700.93, stdev=3103.61 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 562], 10.00th=[ 619], 20.00th=[ 750], | 30.00th=[ 1057], 40.00th=[ 2311], 50.00th=[ 3064], 60.00th=[ 3785], | 70.00th=[ 4883], 80.00th=[ 6325], 90.00th=[ 8094], 95.00th=[ 9634], | 99.00th=[13435], 99.50th=[14877], 99.90th=[19268], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 248, max= 576, per=25.05%, avg=455.92, stdev=66.13, samples=26 iops : min= 62, max= 144, avg=113.92, stdev=16.56, samples=26 lat (usec) : 100=1.73%, 250=0.85%, 500=0.07%, 750=10.19%, 1000=4.69% lat (msec) : 2=4.75%, 4=30.08%, 10=43.46%, 20=4.07%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=23.84%, ctx=6177, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25889: Wed Dec 25 01:25:40 2024 read: IOPS=107, BW=430KiB/s (440kB/s)(5940KiB/13811msec) clat (usec): min=55, max=28982, avg=4933.70, stdev=2655.78 lat (usec): min=56, max=28983, avg=4934.68, stdev=2656.15 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 2024], 10.00th=[ 2474], 20.00th=[ 2900], | 30.00th=[ 3359], 40.00th=[ 3851], 50.00th=[ 4490], 60.00th=[ 5080], | 70.00th=[ 5932], 80.00th=[ 6783], 90.00th=[ 8094], 95.00th=[ 9372], | 99.00th=[13698], 99.50th=[15664], 99.90th=[19006], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 216, max= 561, per=24.34%, avg=423.04, stdev=69.41, samples=27 iops : min= 54, max= 140, avg=105.67, stdev=17.34, samples=27 write: IOPS=114, BW=460KiB/s (471kB/s)(6348KiB/13811msec) clat (usec): min=472, max=203494, avg=4034.30, stdev=6018.61 lat (usec): min=473, max=203495, avg=4035.72, stdev=6019.05 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 586], 10.00th=[ 644], 20.00th=[ 816], | 30.00th=[ 1156], 40.00th=[ 2442], 50.00th=[ 3163], 60.00th=[ 3851], | 70.00th=[ 4948], 80.00th=[ 6521], 90.00th=[ 8717], 95.00th=[ 10552], | 99.00th=[ 14222], 99.50th=[ 16712], 99.90th=[ 21627], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 288, max= 592, per=24.61%, avg=447.96, stdev=62.03, samples=27 iops : min= 72, max= 148, avg=111.89, stdev=15.44, samples=27 lat (usec) : 100=1.37%, 250=0.49%, 500=0.16%, 750=8.82%, 1000=4.92% lat (msec) : 2=5.14%, 4=31.51%, 10=42.45%, 20=5.01%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=23.66%, ctx=6327, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25890: Wed Dec 25 01:25:40 2024 read: IOPS=111, BW=448KiB/s (459kB/s)(6028KiB/13459msec) clat (usec): min=56, max=23551, avg=4863.74, stdev=2554.57 lat (usec): min=56, max=23552, avg=4864.79, stdev=2554.63 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 1942], 10.00th=[ 2442], 20.00th=[ 2933], | 30.00th=[ 3392], 40.00th=[ 3884], 50.00th=[ 4490], 60.00th=[ 5080], | 70.00th=[ 5800], 80.00th=[ 6652], 90.00th=[ 7898], 95.00th=[ 9241], | 99.00th=[13566], 99.50th=[15139], 99.90th=[18744], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 152, max= 560, per=25.43%, avg=442.04, stdev=72.74, samples=26 iops : min= 38, max= 140, avg=110.42, stdev=18.18, samples=26 write: IOPS=116, BW=465KiB/s (476kB/s)(6260KiB/13459msec) clat (usec): min=455, max=209317, avg=3865.31, stdev=6074.63 lat (usec): min=455, max=209318, avg=3867.20, stdev=6075.58 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 570], 10.00th=[ 635], 20.00th=[ 791], | 30.00th=[ 1020], 40.00th=[ 2278], 50.00th=[ 3163], 60.00th=[ 3851], | 70.00th=[ 4883], 80.00th=[ 6259], 90.00th=[ 7963], 95.00th=[ 10028], | 99.00th=[ 13829], 99.50th=[ 15139], 99.90th=[ 19530], 99.95th=[208667], | 99.99th=[208667] bw ( KiB/s): min= 176, max= 640, per=25.06%, avg=456.12, stdev=99.18, samples=26 iops : min= 44, max= 160, avg=113.96, stdev=24.76, samples=26 lat (usec) : 100=1.69%, 250=0.52%, 500=0.10%, 750=8.95%, 1000=5.92% lat (msec) : 2=4.85%, 4=29.75%, 10=43.85%, 20=4.30%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=24.23%, ctx=6287, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25891: Wed Dec 25 01:25:40 2024 read: IOPS=110, BW=443KiB/s (453kB/s)(6052KiB/13669msec) clat (usec): min=58, max=21448, avg=4904.44, stdev=2710.71 lat (usec): min=58, max=21449, avg=4906.13, stdev=2710.14 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 2008], 10.00th=[ 2409], 20.00th=[ 2900], | 30.00th=[ 3294], 40.00th=[ 3752], 50.00th=[ 4293], 60.00th=[ 5080], | 70.00th=[ 5866], 80.00th=[ 6783], 90.00th=[ 8094], 95.00th=[ 9634], | 99.00th=[14615], 99.50th=[17433], 99.90th=[20317], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 207, max= 544, per=25.20%, avg=437.93, stdev=78.07, samples=27 iops : min= 51, max= 136, avg=109.30, stdev=19.59, samples=27 write: IOPS=114, BW=456KiB/s (467kB/s)(6236KiB/13669msec) clat (usec): min=442, max=207989, avg=3953.08, stdev=6148.27 lat (usec): min=443, max=207991, avg=3955.39, stdev=6148.73 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 570], 10.00th=[ 627], 20.00th=[ 783], | 30.00th=[ 1029], 40.00th=[ 2278], 50.00th=[ 3130], 60.00th=[ 3851], | 70.00th=[ 4817], 80.00th=[ 6587], 90.00th=[ 8586], 95.00th=[ 10290], | 99.00th=[ 14091], 99.50th=[ 15795], 99.90th=[ 21890], 99.95th=[208667], | 99.99th=[208667] bw ( KiB/s): min= 239, max= 608, per=24.73%, avg=450.04, stdev=77.52, samples=27 iops : min= 59, max= 152, avg=112.33, stdev=19.50, samples=27 lat (usec) : 100=1.07%, 250=0.88%, 500=0.26%, 750=9.34%, 1000=5.34% lat (msec) : 2=5.14%, 4=31.05%, 10=41.99%, 20=4.79%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=23.47%, ctx=6277, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1739KiB/s (1780kB/s), 430KiB/s-448KiB/s (440kB/s-459kB/s), io=23.4MiB (24.6MB), run=13459-13811msec WRITE: bw=1820KiB/s (1864kB/s), 456KiB/s-465KiB/s (467kB/s-476kB/s), io=24.6MiB (25.7MB), run=13459-13811msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25892: Wed Dec 25 01:25:43 2024 read: IOPS=87, BW=350KiB/s (358kB/s)(5992KiB/17123msec) clat (usec): min=1580, max=217747, avg=4726.63, stdev=6592.11 lat (usec): min=1580, max=217748, avg=4727.32, stdev=6592.14 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1860], 10.00th=[ 1958], 20.00th=[ 2147], | 30.00th=[ 2311], 40.00th=[ 2573], 50.00th=[ 2999], 60.00th=[ 3687], | 70.00th=[ 4555], 80.00th=[ 6521], 90.00th=[ 10028], 95.00th=[ 13042], | 99.00th=[ 17695], 99.50th=[ 19530], 99.90th=[ 25035], 99.95th=[217056], | 99.99th=[217056] bw ( KiB/s): min= 136, max= 768, per=24.87%, avg=348.68, stdev=175.97, samples=34 iops : min= 34, max= 192, avg=87.15, stdev=44.01, samples=34 write: IOPS=91, BW=368KiB/s (377kB/s)(6296KiB/17123msec) clat (usec): min=2362, max=33252, avg=6339.89, stdev=3902.02 lat (usec): min=2362, max=33252, avg=6340.73, stdev=3902.12 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2835], 10.00th=[ 3032], 20.00th=[ 3326], | 30.00th=[ 3556], 40.00th=[ 3949], 50.00th=[ 4752], 60.00th=[ 5866], | 70.00th=[ 7570], 80.00th=[ 9241], 90.00th=[12125], 95.00th=[14091], | 99.00th=[18744], 99.50th=[21365], 99.90th=[25822], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 152, max= 800, per=24.92%, avg=365.82, stdev=177.78, samples=34 iops : min= 38, max= 200, avg=91.44, stdev=44.45, samples=34 lat (msec) : 2=5.73%, 4=46.16%, 10=34.60%, 20=12.96%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=11.39%, ctx=3658, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25893: Wed Dec 25 01:25:43 2024 read: IOPS=87, BW=350KiB/s (358kB/s)(5940KiB/16973msec) clat (usec): min=1560, max=185867, avg=4690.63, stdev=5952.17 lat (usec): min=1560, max=185869, avg=4691.36, stdev=5952.18 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1876], 10.00th=[ 1958], 20.00th=[ 2114], | 30.00th=[ 2311], 40.00th=[ 2540], 50.00th=[ 2999], 60.00th=[ 3589], | 70.00th=[ 4555], 80.00th=[ 6521], 90.00th=[ 9896], 95.00th=[ 12780], | 99.00th=[ 18220], 99.50th=[ 20055], 99.90th=[ 23725], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 112, max= 800, per=24.57%, avg=344.45, stdev=179.87, samples=33 iops : min= 28, max= 200, avg=86.09, stdev=44.99, samples=33 write: IOPS=93, BW=374KiB/s (383kB/s)(6348KiB/16973msec) clat (usec): min=2275, max=29450, avg=6252.27, stdev=3813.85 lat (usec): min=2276, max=29451, avg=6253.17, stdev=3813.92 clat percentiles (usec): | 1.00th=[ 2507], 5.00th=[ 2868], 10.00th=[ 3064], 20.00th=[ 3294], | 30.00th=[ 3556], 40.00th=[ 3982], 50.00th=[ 4686], 60.00th=[ 5735], | 70.00th=[ 7242], 80.00th=[ 9110], 90.00th=[11863], 95.00th=[13960], | 99.00th=[18744], 99.50th=[20841], 99.90th=[26084], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 168, max= 752, per=24.72%, avg=362.85, stdev=169.30, samples=33 iops : min= 42, max= 188, avg=90.70, stdev=42.32, samples=33 lat (msec) : 2=6.02%, 4=46.16%, 10=34.60%, 20=12.63%, 50=0.55% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.41%, ctx=3739, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25894: Wed Dec 25 01:25:43 2024 read: IOPS=88, BW=356KiB/s (365kB/s)(6028KiB/16933msec) clat (usec): min=1497, max=24953, avg=4500.16, stdev=3504.32 lat (usec): min=1498, max=24953, avg=4500.85, stdev=3504.33 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1876], 10.00th=[ 1975], 20.00th=[ 2147], | 30.00th=[ 2343], 40.00th=[ 2606], 50.00th=[ 3032], 60.00th=[ 3621], | 70.00th=[ 4555], 80.00th=[ 6390], 90.00th=[ 9896], 95.00th=[11994], | 99.00th=[17171], 99.50th=[19006], 99.90th=[24511], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 120, max= 816, per=24.87%, avg=348.67, stdev=175.35, samples=33 iops : min= 30, max= 204, avg=87.06, stdev=43.89, samples=33 write: IOPS=92, BW=370KiB/s (379kB/s)(6260KiB/16933msec) clat (msec): min=2, max=208, avg= 6.45, stdev= 6.46 lat (msec): min=2, max=208, avg= 6.45, stdev= 6.46 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 15], | 99.00th=[ 20], 99.50th=[ 24], 99.90th=[ 30], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 168, max= 816, per=24.56%, avg=360.55, stdev=175.38, samples=33 iops : min= 42, max= 204, avg=90.03, stdev=43.90, samples=33 lat (msec) : 2=5.89%, 4=46.55%, 10=34.18%, 20=12.79%, 50=0.55% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=11.46%, ctx=3711, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25895: Wed Dec 25 01:25:43 2024 read: IOPS=89, BW=357KiB/s (365kB/s)(6052KiB/16964msec) clat (usec): min=1523, max=22776, avg=4540.86, stdev=3514.58 lat (usec): min=1523, max=22777, avg=4541.52, stdev=3514.59 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1876], 10.00th=[ 1975], 20.00th=[ 2114], | 30.00th=[ 2311], 40.00th=[ 2671], 50.00th=[ 3064], 60.00th=[ 3654], | 70.00th=[ 4555], 80.00th=[ 6325], 90.00th=[ 9634], 95.00th=[12256], | 99.00th=[17957], 99.50th=[18744], 99.90th=[21365], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 128, max= 752, per=24.79%, avg=347.55, stdev=169.08, samples=33 iops : min= 32, max= 188, avg=86.85, stdev=42.28, samples=33 write: IOPS=91, BW=368KiB/s (376kB/s)(6236KiB/16964msec) clat (msec): min=2, max=208, avg= 6.43, stdev= 6.40 lat (msec): min=2, max=208, avg= 6.43, stdev= 6.40 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 20], 99.50th=[ 25], 99.90th=[ 34], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 160, max= 832, per=24.50%, avg=359.67, stdev=173.84, samples=33 iops : min= 40, max= 208, avg=89.88, stdev=43.49, samples=33 lat (msec) : 2=6.05%, 4=46.32%, 10=34.64%, 20=12.53%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=11.49%, ctx=3676, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1402KiB/s (1436kB/s), 350KiB/s-357KiB/s (358kB/s-365kB/s), io=23.4MiB (24.6MB), run=16933-17123msec WRITE: bw=1468KiB/s (1503kB/s), 368KiB/s-374KiB/s (376kB/s-383kB/s), io=24.6MiB (25.7MB), run=16933-17123msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25922: Wed Dec 25 01:25:51 2024 read: IOPS=64, BW=1033KiB/s (1058kB/s)(5920KiB/5729msec) clat (usec): min=1944, max=26884, avg=5777.19, stdev=4211.19 lat (usec): min=1944, max=26885, avg=5777.84, stdev=4211.20 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2409], 10.00th=[ 2507], 20.00th=[ 2769], | 30.00th=[ 3064], 40.00th=[ 3458], 50.00th=[ 3720], 60.00th=[ 4359], | 70.00th=[ 6652], 80.00th=[ 9241], 90.00th=[11863], 95.00th=[14222], | 99.00th=[20317], 99.50th=[24511], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 608, max= 1472, per=26.64%, avg=1015.00, stdev=263.44, samples=11 iops : min= 38, max= 92, avg=63.36, stdev=16.45, samples=11 write: IOPS=69, BW=1112KiB/s (1138kB/s)(6368KiB/5729msec) clat (usec): min=2628, max=99554, avg=8989.23, stdev=7084.49 lat (usec): min=2629, max=99556, avg=8990.54, stdev=7084.61 clat percentiles (usec): | 1.00th=[ 2999], 5.00th=[ 3621], 10.00th=[ 3916], 20.00th=[ 4228], | 30.00th=[ 4883], 40.00th=[ 5866], 50.00th=[ 6783], 60.00th=[ 8356], | 70.00th=[10159], 80.00th=[12125], 90.00th=[17433], 95.00th=[20055], | 99.00th=[26346], 99.50th=[34341], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 864, max= 1440, per=27.21%, avg=1090.64, stdev=172.01, samples=11 iops : min= 54, max= 90, avg=68.09, stdev=10.76, samples=11 lat (msec) : 2=0.39%, 4=32.68%, 10=42.97%, 20=21.09%, 50=2.73% lat (msec) : 100=0.13% cpu : usr=0.16%, sys=9.58%, ctx=928, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25923: Wed Dec 25 01:25:51 2024 read: IOPS=57, BW=914KiB/s (936kB/s)(5744KiB/6286msec) clat (usec): min=1946, max=22777, avg=6551.54, stdev=4603.82 lat (usec): min=1947, max=22777, avg=6552.29, stdev=4603.79 clat percentiles (usec): | 1.00th=[ 2057], 5.00th=[ 2278], 10.00th=[ 2507], 20.00th=[ 2835], | 30.00th=[ 3195], 40.00th=[ 3621], 50.00th=[ 4424], 60.00th=[ 6128], | 70.00th=[ 8225], 80.00th=[10945], 90.00th=[13304], 95.00th=[16188], | 99.00th=[21103], 99.50th=[22414], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 608, max= 1212, per=23.71%, avg=903.17, stdev=177.00, samples=12 iops : min= 38, max= 75, avg=56.17, stdev=11.04, samples=12 write: IOPS=65, BW=1041KiB/s (1066kB/s)(6544KiB/6286msec) clat (msec): min=2, max=105, avg= 9.49, stdev= 6.83 lat (msec): min=2, max=105, avg= 9.49, stdev= 6.83 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 14], 90.00th=[ 16], 95.00th=[ 18], | 99.00th=[ 26], 99.50th=[ 28], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 702, max= 1370, per=25.39%, avg=1017.58, stdev=202.42, samples=12 iops : min= 43, max= 85, avg=63.33, stdev=12.66, samples=12 lat (msec) : 2=0.26%, 4=26.04%, 10=42.06%, 20=29.56%, 50=1.95% lat (msec) : 250=0.13% cpu : usr=0.21%, sys=10.37%, ctx=1089, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25924: Wed Dec 25 01:25:51 2024 read: IOPS=70, BW=1123KiB/s (1149kB/s)(6256KiB/5573msec) clat (usec): min=1999, max=30530, avg=5763.31, stdev=4290.30 lat (usec): min=2000, max=30531, avg=5763.97, stdev=4290.30 clat percentiles (usec): | 1.00th=[ 2089], 5.00th=[ 2376], 10.00th=[ 2540], 20.00th=[ 2868], | 30.00th=[ 3130], 40.00th=[ 3490], 50.00th=[ 3982], 60.00th=[ 4621], | 70.00th=[ 6063], 80.00th=[ 8455], 90.00th=[11600], 95.00th=[14353], | 99.00th=[21103], 99.50th=[27132], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 800, max= 1472, per=29.61%, avg=1128.18, stdev=216.25, samples=11 iops : min= 50, max= 92, avg=70.36, stdev=13.55, samples=11 write: IOPS=67, BW=1082KiB/s (1108kB/s)(6032KiB/5573msec) clat (usec): min=2883, max=99322, avg=8648.03, stdev=6973.84 lat (usec): min=2884, max=99324, avg=8649.27, stdev=6973.82 clat percentiles (usec): | 1.00th=[ 3163], 5.00th=[ 3425], 10.00th=[ 3752], 20.00th=[ 4113], | 30.00th=[ 4621], 40.00th=[ 5473], 50.00th=[ 6456], 60.00th=[ 7701], | 70.00th=[ 9765], 80.00th=[12911], 90.00th=[16450], 95.00th=[19268], | 99.00th=[25035], 99.50th=[28705], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 768, max= 1373, per=26.70%, avg=1070.09, stdev=178.56, samples=11 iops : min= 48, max= 85, avg=66.73, stdev=11.08, samples=11 lat (msec) : 2=0.13%, 4=34.37%, 10=43.10%, 20=19.92%, 50=2.34% lat (msec) : 100=0.13% cpu : usr=0.14%, sys=9.98%, ctx=952, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25925: Wed Dec 25 01:25:51 2024 read: IOPS=67, BW=1073KiB/s (1099kB/s)(6032KiB/5621msec) clat (usec): min=2004, max=50551, avg=6130.63, stdev=4893.55 lat (usec): min=2005, max=50552, avg=6131.28, stdev=4893.56 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2245], 10.00th=[ 2442], 20.00th=[ 2704], | 30.00th=[ 3064], 40.00th=[ 3490], 50.00th=[ 4293], 60.00th=[ 5211], | 70.00th=[ 7177], 80.00th=[ 9503], 90.00th=[12387], 95.00th=[14877], | 99.00th=[22414], 99.50th=[26346], 99.90th=[50594], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 894, max= 1312, per=27.70%, avg=1055.36, stdev=153.14, samples=11 iops : min= 55, max= 82, avg=65.73, stdev= 9.73, samples=11 write: IOPS=69, BW=1113KiB/s (1140kB/s)(6256KiB/5621msec) clat (usec): min=2827, max=28709, avg=8292.01, stdev=5109.78 lat (usec): min=2828, max=28709, avg=8293.37, stdev=5109.61 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3621], 10.00th=[ 3851], 20.00th=[ 4293], | 30.00th=[ 4752], 40.00th=[ 5342], 50.00th=[ 6194], 60.00th=[ 7701], | 70.00th=[ 9503], 80.00th=[11994], 90.00th=[16319], 95.00th=[19006], | 99.00th=[23987], 99.50th=[28443], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 960, max= 1216, per=27.63%, avg=1107.55, stdev=100.26, samples=11 iops : min= 60, max= 76, avg=69.00, stdev= 6.23, samples=11 lat (msec) : 4=30.08%, 10=47.66%, 20=19.92%, 50=2.21%, 100=0.13% cpu : usr=0.12%, sys=9.86%, ctx=936, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3810KiB/s (3902kB/s), 914KiB/s-1123KiB/s (936kB/s-1149kB/s), io=23.4MiB (24.5MB), run=5573-6286msec WRITE: bw=4009KiB/s (4105kB/s), 1041KiB/s-1113KiB/s (1066kB/s-1140kB/s), io=24.6MiB (25.8MB), run=5573-6286msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25926: Wed Dec 25 01:25:51 2024 read: IOPS=58, BW=942KiB/s (965kB/s)(5920KiB/6284msec) clat (usec): min=52, max=82800, avg=12078.48, stdev=6031.83 lat (usec): min=52, max=82800, avg=12079.14, stdev=6031.85 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 101], 10.00th=[ 7832], 20.00th=[ 9372], | 30.00th=[10290], 40.00th=[11207], 50.00th=[11863], 60.00th=[12911], | 70.00th=[13829], 80.00th=[15270], 90.00th=[17171], 95.00th=[19268], | 99.00th=[23725], 99.50th=[33817], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 704, max= 1280, per=23.88%, avg=904.00, stdev=170.49, samples=12 iops : min= 44, max= 80, avg=56.50, stdev=10.66, samples=12 write: IOPS=63, BW=1013KiB/s (1038kB/s)(6368KiB/6284msec) clat (usec): min=666, max=29207, avg=4381.35, stdev=4325.82 lat (usec): min=667, max=29208, avg=4382.39, stdev=4325.84 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 848], 10.00th=[ 955], 20.00th=[ 1221], | 30.00th=[ 1860], 40.00th=[ 2704], 50.00th=[ 3195], 60.00th=[ 3687], | 70.00th=[ 4555], 80.00th=[ 5669], 90.00th=[10290], 95.00th=[15533], | 99.00th=[20579], 99.50th=[22676], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 576, max= 1376, per=24.63%, avg=981.33, stdev=245.35, samples=12 iops : min= 36, max= 86, avg=61.33, stdev=15.33, samples=12 lat (usec) : 100=2.34%, 250=1.17%, 750=0.26%, 1000=5.99% lat (msec) : 2=9.64%, 4=17.97%, 10=21.35%, 20=38.67%, 50=2.47% lat (msec) : 100=0.13% cpu : usr=0.08%, sys=21.87%, ctx=2649, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25927: Wed Dec 25 01:25:51 2024 read: IOPS=58, BW=936KiB/s (958kB/s)(5744KiB/6139msec) clat (usec): min=51, max=26911, avg=11677.64, stdev=4395.68 lat (usec): min=52, max=26911, avg=11678.70, stdev=4395.65 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 106], 10.00th=[ 8094], 20.00th=[ 9634], | 30.00th=[10421], 40.00th=[11207], 50.00th=[11994], 60.00th=[12649], | 70.00th=[13566], 80.00th=[14615], 90.00th=[16319], 95.00th=[17695], | 99.00th=[21103], 99.50th=[25035], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 640, max= 1184, per=24.16%, avg=914.67, stdev=152.83, samples=12 iops : min= 40, max= 74, avg=57.17, stdev= 9.55, samples=12 write: IOPS=66, BW=1066KiB/s (1092kB/s)(6544KiB/6139msec) clat (usec): min=697, max=93253, avg=4711.84, stdev=6118.53 lat (usec): min=697, max=93255, avg=4712.99, stdev=6118.65 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 816], 10.00th=[ 889], 20.00th=[ 1074], | 30.00th=[ 1844], 40.00th=[ 2671], 50.00th=[ 3228], 60.00th=[ 3949], | 70.00th=[ 4883], 80.00th=[ 6652], 90.00th=[10290], 95.00th=[14484], | 99.00th=[19792], 99.50th=[25035], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 704, max= 1376, per=25.70%, avg=1024.00, stdev=227.91, samples=12 iops : min= 44, max= 86, avg=64.00, stdev=14.24, samples=12 lat (usec) : 100=1.95%, 250=1.17%, 500=0.26%, 750=0.65%, 1000=8.07% lat (msec) : 2=7.68%, 4=16.28%, 10=23.83%, 20=38.67%, 50=1.30% lat (msec) : 100=0.13% cpu : usr=0.07%, sys=20.80%, ctx=2662, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25928: Wed Dec 25 01:25:51 2024 read: IOPS=61, BW=990KiB/s (1014kB/s)(6256KiB/6320msec) clat (usec): min=59, max=31310, avg=11553.36, stdev=4504.78 lat (usec): min=59, max=31311, avg=11553.99, stdev=4504.80 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 119], 10.00th=[ 7373], 20.00th=[ 9372], | 30.00th=[10290], 40.00th=[11207], 50.00th=[11863], 60.00th=[12518], | 70.00th=[13435], 80.00th=[14353], 90.00th=[16057], 95.00th=[17957], | 99.00th=[21890], 99.50th=[22414], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 576, max= 1536, per=25.43%, avg=962.67, stdev=231.34, samples=12 iops : min= 36, max= 96, avg=60.17, stdev=14.46, samples=12 write: IOPS=59, BW=954KiB/s (977kB/s)(6032KiB/6320msec) clat (usec): min=752, max=96557, avg=4565.10, stdev=6391.67 lat (usec): min=752, max=96558, avg=4566.17, stdev=6391.69 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 832], 10.00th=[ 898], 20.00th=[ 1106], | 30.00th=[ 1500], 40.00th=[ 2540], 50.00th=[ 2933], 60.00th=[ 3589], | 70.00th=[ 4686], 80.00th=[ 6456], 90.00th=[ 9634], 95.00th=[14746], | 99.00th=[21890], 99.50th=[27657], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 672, max= 1248, per=22.76%, avg=906.67, stdev=162.79, samples=12 iops : min= 42, max= 78, avg=56.67, stdev=10.17, samples=12 lat (usec) : 100=2.08%, 250=1.56%, 1000=7.68% lat (msec) : 2=8.98%, 4=14.97%, 10=22.66%, 20=39.97%, 50=1.95% lat (msec) : 100=0.13% cpu : usr=0.14%, sys=20.35%, ctx=2741, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25929: Wed Dec 25 01:25:51 2024 read: IOPS=59, BW=954KiB/s (977kB/s)(6032KiB/6325msec) clat (usec): min=60, max=29865, avg=11992.04, stdev=4202.02 lat (usec): min=60, max=29866, avg=11992.70, stdev=4202.02 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 4293], 10.00th=[ 8094], 20.00th=[ 9634], | 30.00th=[10421], 40.00th=[11469], 50.00th=[12256], 60.00th=[12911], | 70.00th=[13698], 80.00th=[14484], 90.00th=[16057], 95.00th=[17695], | 99.00th=[23987], 99.50th=[29230], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 672, max= 1216, per=24.30%, avg=920.00, stdev=147.04, samples=12 iops : min= 42, max= 76, avg=57.50, stdev= 9.19, samples=12 write: IOPS=61, BW=989KiB/s (1013kB/s)(6256KiB/6325msec) clat (usec): min=632, max=90889, avg=4388.81, stdev=5871.81 lat (usec): min=633, max=90891, avg=4389.97, stdev=5871.81 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 824], 10.00th=[ 906], 20.00th=[ 1090], | 30.00th=[ 1500], 40.00th=[ 2540], 50.00th=[ 3064], 60.00th=[ 3621], | 70.00th=[ 4621], 80.00th=[ 5866], 90.00th=[ 9634], 95.00th=[13304], | 99.00th=[20579], 99.50th=[22676], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 736, max= 1344, per=24.30%, avg=968.00, stdev=163.81, samples=12 iops : min= 46, max= 84, avg=60.50, stdev=10.24, samples=12 lat (usec) : 100=0.91%, 250=1.43%, 750=0.13%, 1000=7.81% lat (msec) : 2=9.11%, 4=15.76%, 10=22.53%, 20=40.49%, 50=1.69% lat (msec) : 100=0.13% cpu : usr=0.14%, sys=20.60%, ctx=2785, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3787KiB/s (3878kB/s), 936KiB/s-990KiB/s (958kB/s-1014kB/s), io=23.4MiB (24.5MB), run=6139-6325msec WRITE: bw=3984KiB/s (4080kB/s), 954KiB/s-1066KiB/s (977kB/s-1092kB/s), io=24.6MiB (25.8MB), run=6139-6325msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25940: Wed Dec 25 01:25:58 2024 read: IOPS=1, BW=1235KiB/s (1265kB/s)(7168KiB/5803msec) clat (usec): min=452, max=778158, avg=413142.59, stdev=334293.55 lat (usec): min=453, max=778159, avg=413143.78, stdev=334293.71 clat percentiles (usec): | 1.00th=[ 453], 5.00th=[ 453], 10.00th=[ 453], 20.00th=[ 82314], | 30.00th=[106431], 40.00th=[106431], 50.00th=[599786], 60.00th=[616563], | 70.00th=[616563], 80.00th=[708838], 90.00th=[775947], 95.00th=[775947], | 99.00th=[775947], 99.50th=[775947], 99.90th=[775947], 99.95th=[775947], | 99.99th=[775947] bw ( KiB/s): min= 1822, max= 4096, per=68.36%, avg=2412.40, stdev=946.24, samples=5 iops : min= 1, max= 4, avg= 2.20, stdev= 1.10, samples=5 write: IOPS=0, BW=882KiB/s (903kB/s)(5120KiB/5803msec) clat (msec): min=42, max=1774, avg=576.61, stdev=773.66 lat (msec): min=42, max=1774, avg=576.67, stdev=773.67 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 45], 50.00th=[ 74], 60.00th=[ 74], | 70.00th=[ 944], 80.00th=[ 944], 90.00th=[ 1770], 95.00th=[ 1770], | 99.00th=[ 1770], 99.50th=[ 1770], 99.90th=[ 1770], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 1822, max= 5456, per=62.93%, avg=3108.67, stdev=2035.99, samples=3 iops : min= 1, max= 5, avg= 2.67, stdev= 2.08, samples=3 lat (usec) : 500=8.33% lat (msec) : 50=16.67%, 100=16.67%, 250=8.33%, 750=25.00%, 1000=16.67% cpu : usr=0.02%, sys=11.98%, ctx=1268, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25941: Wed Dec 25 01:25:58 2024 read: IOPS=0, BW=664KiB/s (680kB/s)(2048KiB/3086msec) clat (msec): min=104, max=374, avg=239.59, stdev=190.74 lat (msec): min=104, max=374, avg=239.59, stdev=190.74 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 105], 10.00th=[ 105], 20.00th=[ 105], | 30.00th=[ 105], 40.00th=[ 105], 50.00th=[ 105], 60.00th=[ 376], | 70.00th=[ 376], 80.00th=[ 376], 90.00th=[ 376], 95.00th=[ 376], | 99.00th=[ 376], 99.50th=[ 376], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 2048, max= 2048, per=58.03%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3318KiB/s (3398kB/s)(10.0MiB/3086msec) clat (msec): min=26, max=1699, avg=259.27, stdev=529.35 lat (msec): min=26, max=1699, avg=259.33, stdev=529.36 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 30], | 30.00th=[ 35], 40.00th=[ 40], 50.00th=[ 44], 60.00th=[ 48], | 70.00th=[ 78], 80.00th=[ 78], 90.00th=[ 535], 95.00th=[ 1703], | 99.00th=[ 1703], 99.50th=[ 1703], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 2048, max=10240, per=100.00%, avg=6144.00, stdev=4096.00, samples=3 iops : min= 2, max= 10, avg= 6.00, stdev= 4.00, samples=3 lat (msec) : 50=50.00%, 100=16.67%, 250=8.33%, 500=8.33%, 750=8.33% cpu : usr=0.03%, sys=11.12%, ctx=281, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25943: Wed Dec 25 01:25:58 2024 read: IOPS=1, BW=1555KiB/s (1592kB/s)(6144KiB/3951msec) clat (usec): min=488, max=850644, avg=367079.02, stdev=397015.64 lat (usec): min=489, max=850644, avg=367079.98, stdev=397015.65 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 490], 10.00th=[ 490], 20.00th=[ 39584], | 30.00th=[ 39584], 40.00th=[ 65799], 50.00th=[ 65799], 60.00th=[408945], | 70.00th=[834667], 80.00th=[834667], 90.00th=[851444], 95.00th=[851444], | 99.00th=[851444], 99.50th=[851444], 99.90th=[851444], 99.95th=[851444], | 99.99th=[851444] bw ( KiB/s): min= 2048, max= 6144, per=96.72%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 write: IOPS=1, BW=1555KiB/s (1592kB/s)(6144KiB/3951msec) clat (msec): min=25, max=889, avg=286.77, stdev=377.37 lat (msec): min=25, max=890, avg=286.84, stdev=377.37 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 51], | 30.00th=[ 51], 40.00th=[ 52], 50.00th=[ 52], 60.00th=[ 70], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 894], 95.00th=[ 894], | 99.00th=[ 894], 99.50th=[ 894], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 2048, max= 4096, per=62.19%, avg=3072.00, stdev=1182.41, samples=4 iops : min= 2, max= 4, avg= 3.00, stdev= 1.15, samples=4 lat (usec) : 500=8.33% lat (msec) : 50=16.67%, 100=33.33%, 500=8.33%, 750=8.33%, 1000=25.00% cpu : usr=0.00%, sys=13.80%, ctx=896, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25945: Wed Dec 25 01:25:58 2024 read: IOPS=1, BW=1296KiB/s (1327kB/s)(5120KiB/3950msec) clat (usec): min=513, max=869866, avg=581797.23, stdev=380160.46 lat (usec): min=515, max=869869, avg=581799.02, stdev=380160.82 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 515], 10.00th=[ 515], 20.00th=[ 515], | 30.00th=[387974], 40.00th=[387974], 50.00th=[826278], 60.00th=[826278], | 70.00th=[826278], 80.00th=[826278], 90.00th=[868221], 95.00th=[868221], | 99.00th=[868221], 99.50th=[868221], 99.90th=[868221], 99.95th=[868221], | 99.99th=[868221] bw ( KiB/s): min= 2043, max= 2048, per=58.00%, avg=2046.75, stdev= 2.50, samples=4 iops : min= 1, max= 2, avg= 1.75, stdev= 0.50, samples=4 write: IOPS=1, BW=1815KiB/s (1858kB/s)(7168KiB/3950msec) clat (msec): min=27, max=644, avg=136.76, stdev=225.06 lat (msec): min=27, max=644, avg=136.83, stdev=225.06 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 29], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 50], 60.00th=[ 78], | 70.00th=[ 78], 80.00th=[ 85], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 2048, max= 6144, per=96.73%, avg=4778.67, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 4.67, stdev= 2.31, samples=3 lat (usec) : 750=8.33% lat (msec) : 50=33.33%, 100=16.67%, 500=8.33%, 750=8.33%, 1000=25.00% cpu : usr=0.05%, sys=15.57%, ctx=1083, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3529KiB/s (3614kB/s), 664KiB/s-1555KiB/s (680kB/s-1592kB/s), io=20.0MiB (20.0MB), run=3086-5803msec WRITE: bw=4941KiB/s (5059kB/s), 882KiB/s-3318KiB/s (903kB/s-3398kB/s), io=28.0MiB (29.4MB), run=3086-5803msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25942: Wed Dec 25 01:25:58 2024 read: IOPS=1, BW=1224KiB/s (1253kB/s)(7168KiB/5856msec) clat (msec): min=16, max=964, avg=414.49, stdev=386.93 lat (msec): min=16, max=964, avg=414.50, stdev=386.93 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 23], | 30.00th=[ 27], 40.00th=[ 27], 50.00th=[ 592], 60.00th=[ 625], | 70.00th=[ 625], 80.00th=[ 659], 90.00th=[ 961], 95.00th=[ 961], | 99.00th=[ 961], 99.50th=[ 961], 99.90th=[ 961], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 1851, max= 3696, per=66.83%, avg=2337.20, stdev=764.29, samples=5 iops : min= 1, max= 3, avg= 1.80, stdev= 0.84, samples=5 write: IOPS=0, BW=874KiB/s (895kB/s)(5120KiB/5856msec) clat (msec): min=59, max=1793, avg=571.19, stdev=762.48 lat (msec): min=59, max=1793, avg=571.25, stdev=762.47 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 59], | 30.00th=[ 61], 40.00th=[ 61], 50.00th=[ 92], 60.00th=[ 92], | 70.00th=[ 852], 80.00th=[ 852], 90.00th=[ 1787], 95.00th=[ 1787], | 99.00th=[ 1787], 99.50th=[ 1787], 99.90th=[ 1787], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 2011, max= 5545, per=77.17%, avg=3778.00, stdev=2498.92, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 lat (msec) : 20=8.33%, 50=16.67%, 100=25.00%, 750=25.00%, 1000=16.67% cpu : usr=0.02%, sys=2.44%, ctx=73, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25944: Wed Dec 25 01:25:58 2024 read: IOPS=0, BW=655KiB/s (671kB/s)(2048KiB/3125msec) clat (msec): min=67, max=511, avg=289.42, stdev=314.22 lat (msec): min=67, max=511, avg=289.43, stdev=314.23 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 68], | 30.00th=[ 68], 40.00th=[ 68], 50.00th=[ 68], 60.00th=[ 510], | 70.00th=[ 510], 80.00th=[ 510], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 2048, max= 2048, per=58.56%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3277KiB/s (3355kB/s)(10.0MiB/3125msec) clat (msec): min=23, max=1241, avg=248.03, stdev=439.53 lat (msec): min=23, max=1241, avg=248.09, stdev=439.53 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 35], | 30.00th=[ 36], 40.00th=[ 37], 50.00th=[ 40], 60.00th=[ 48], | 70.00th=[ 66], 80.00th=[ 66], 90.00th=[ 894], 95.00th=[ 1250], | 99.00th=[ 1250], 99.50th=[ 1250], 99.90th=[ 1250], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 2048, max=10240, per=100.00%, avg=6144.00, stdev=4096.00, samples=3 iops : min= 2, max= 10, avg= 6.00, stdev= 4.00, samples=3 lat (msec) : 50=50.00%, 100=25.00%, 750=8.33%, 1000=8.33% cpu : usr=0.16%, sys=5.38%, ctx=54, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25946: Wed Dec 25 01:25:58 2024 read: IOPS=1, BW=1503KiB/s (1539kB/s)(6144KiB/4088msec) clat (msec): min=12, max=952, avg=237.83, stdev=369.83 lat (msec): min=12, max=952, avg=237.83, stdev=369.83 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 17], | 30.00th=[ 17], 40.00th=[ 24], 50.00th=[ 24], 60.00th=[ 97], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 953], 95.00th=[ 953], | 99.00th=[ 953], 99.50th=[ 953], 99.90th=[ 953], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 2048, max= 4096, per=97.61%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 write: IOPS=1, BW=1503KiB/s (1539kB/s)(6144KiB/4088msec) clat (msec): min=22, max=907, avg=440.36, stdev=425.19 lat (msec): min=22, max=907, avg=440.43, stdev=425.20 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 67], | 30.00th=[ 67], 40.00th=[ 89], 50.00th=[ 89], 60.00th=[ 684], | 70.00th=[ 877], 80.00th=[ 877], 90.00th=[ 911], 95.00th=[ 911], | 99.00th=[ 911], 99.50th=[ 911], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 2048, max= 4096, per=62.75%, avg=3072.00, stdev=1182.41, samples=4 iops : min= 2, max= 4, avg= 3.00, stdev= 1.15, samples=4 lat (msec) : 20=16.67%, 50=16.67%, 100=25.00%, 500=8.33%, 750=8.33% lat (msec) : 1000=25.00% cpu : usr=0.00%, sys=3.43%, ctx=163, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25947: Wed Dec 25 01:25:58 2024 read: IOPS=1, BW=1192KiB/s (1220kB/s)(5120KiB/4296msec) clat (msec): min=14, max=1145, avg=448.65, stdev=521.16 lat (msec): min=14, max=1145, avg=448.65, stdev=521.16 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15], | 30.00th=[ 94], 40.00th=[ 94], 50.00th=[ 121], 60.00th=[ 121], | 70.00th=[ 869], 80.00th=[ 869], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 2048, max= 4087, per=78.00%, avg=2727.67, stdev=1177.22, samples=3 iops : min= 2, max= 3, avg= 2.33, stdev= 0.58, samples=3 write: IOPS=1, BW=1669KiB/s (1709kB/s)(7168KiB/4296msec) clat (msec): min=18, max=952, avg=280.48, stdev=376.12 lat (msec): min=19, max=952, avg=280.56, stdev=376.11 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 53], | 30.00th=[ 66], 40.00th=[ 66], 50.00th=[ 93], 60.00th=[ 99], | 70.00th=[ 99], 80.00th=[ 684], 90.00th=[ 953], 95.00th=[ 953], | 99.00th=[ 953], 99.50th=[ 953], 99.90th=[ 953], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 2048, max= 6144, per=83.66%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=16.67%, 100=41.67%, 250=8.33%, 750=8.33%, 1000=16.67% cpu : usr=0.07%, sys=3.59%, ctx=77, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3497KiB/s (3581kB/s), 655KiB/s-1503KiB/s (671kB/s-1539kB/s), io=20.0MiB (20.0MB), run=3125-5856msec WRITE: bw=4896KiB/s (5014kB/s), 874KiB/s-3277KiB/s (895kB/s-3355kB/s), io=28.0MiB (29.4MB), run=3125-5856msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25962: Wed Dec 25 01:26:02 2024 read: IOPS=0, BW=3712KiB/s (3801kB/s)(12.0MiB/3310msec) clat (msec): min=102, max=1764, avg=1096.94, stdev=878.07 lat (msec): min=102, max=1764, avg=1096.94, stdev=878.07 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 103], 10.00th=[ 103], 20.00th=[ 103], | 30.00th=[ 103], 40.00th=[ 1418], 50.00th=[ 1418], 60.00th=[ 1418], | 70.00th=[ 1770], 80.00th=[ 1770], 90.00th=[ 1770], 95.00th=[ 1770], | 99.00th=[ 1770], 99.50th=[ 1770], 99.90th=[ 1770], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.06%, sys=32.64%, ctx=1569, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25963: Wed Dec 25 01:26:02 2024 read: IOPS=0, BW=1149KiB/s (1176kB/s)(4096KiB/3566msec) clat (nsec): min=96874k, max=96874k, avg=96874263.00, stdev= 0.00 lat (nsec): min=96876k, max=96876k, avg=96875517.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[96994], 5.00th=[96994], 10.00th=[96994], 20.00th=[96994], | 30.00th=[96994], 40.00th=[96994], 50.00th=[96994], 60.00th=[96994], | 70.00th=[96994], 80.00th=[96994], 90.00th=[96994], 95.00th=[96994], | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2297KiB/s (2352kB/s)(8192KiB/3566msec) clat (msec): min=238, max=3220, avg=1729.17, stdev=2108.77 lat (msec): min=238, max=3220, avg=1729.45, stdev=2108.79 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 239], 10.00th=[ 239], 20.00th=[ 239], | 30.00th=[ 239], 40.00th=[ 239], 50.00th=[ 239], 60.00th=[ 3205], | 70.00th=[ 3205], 80.00th=[ 3205], 90.00th=[ 3205], 95.00th=[ 3205], | 99.00th=[ 3205], 99.50th=[ 3205], 99.90th=[ 3205], 99.95th=[ 3205], | 99.99th=[ 3205] bw ( KiB/s): min= 8192, max= 8192, per=90.33%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.03%, sys=5.55%, ctx=171, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25964: Wed Dec 25 01:26:02 2024 write: IOPS=0, BW=3497KiB/s (3581kB/s)(12.0MiB/3514msec) clat (msec): min=91, max=3232, avg=1167.28, stdev=1789.39 lat (msec): min=91, max=3233, avg=1167.54, stdev=1789.40 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 91], 10.00th=[ 91], 20.00th=[ 91], | 30.00th=[ 91], 40.00th=[ 178], 50.00th=[ 178], 60.00th=[ 178], | 70.00th=[ 3239], 80.00th=[ 3239], 90.00th=[ 3239], 95.00th=[ 3239], | 99.00th=[ 3239], 99.50th=[ 3239], 99.90th=[ 3239], 99.95th=[ 3239], | 99.99th=[ 3239] bw ( KiB/s): min= 8192, max= 8192, per=90.33%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=7.74%, ctx=18, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25965: Wed Dec 25 01:26:02 2024 write: IOPS=0, BW=3401KiB/s (3483kB/s)(12.0MiB/3613msec) clat (msec): min=95, max=3248, avg=1196.85, stdev=1778.52 lat (msec): min=95, max=3248, avg=1197.81, stdev=1777.98 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 96], 10.00th=[ 96], 20.00th=[ 96], | 30.00th=[ 96], 40.00th=[ 247], 50.00th=[ 247], 60.00th=[ 247], | 70.00th=[ 3239], 80.00th=[ 3239], 90.00th=[ 3239], 95.00th=[ 3239], | 99.00th=[ 3239], 99.50th=[ 3239], 99.90th=[ 3239], 99.95th=[ 3239], | 99.99th=[ 3239] bw ( KiB/s): min= 8192, max= 8192, per=90.33%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.06%, sys=7.09%, ctx=21, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4595KiB/s (4705kB/s), 1149KiB/s-3712KiB/s (1176kB/s-3801kB/s), io=16.0MiB (16.8MB), run=3310-3566msec WRITE: bw=9069KiB/s (9287kB/s), 2297KiB/s-3497KiB/s (2352kB/s-3581kB/s), io=32.0MiB (33.6MB), run=3514-3613msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25957: Wed Dec 25 01:26:02 2024 read: IOPS=0, BW=3629KiB/s (3716kB/s)(12.0MiB/3386msec) clat (msec): min=86, max=1756, avg=1126.88, stdev=907.90 lat (msec): min=86, max=1756, avg=1126.88, stdev=907.90 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 87], 10.00th=[ 87], 20.00th=[ 87], | 30.00th=[ 87], 40.00th=[ 1536], 50.00th=[ 1536], 60.00th=[ 1536], | 70.00th=[ 1754], 80.00th=[ 1754], 90.00th=[ 1754], 95.00th=[ 1754], | 99.00th=[ 1754], 99.50th=[ 1754], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 7185, max= 7185, per=100.00%, avg=7185.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 100=33.33% cpu : usr=0.00%, sys=2.33%, ctx=52, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25958: Wed Dec 25 01:26:02 2024 read: IOPS=0, BW=1199KiB/s (1228kB/s)(4096KiB/3416msec) clat (nsec): min=52312k, max=52312k, avg=52311750.00, stdev= 0.00 lat (nsec): min=52314k, max=52314k, avg=52313710.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[52167], 5.00th=[52167], 10.00th=[52167], 20.00th=[52167], | 30.00th=[52167], 40.00th=[52167], 50.00th=[52167], 60.00th=[52167], | 70.00th=[52167], 80.00th=[52167], 90.00th=[52167], 95.00th=[52167], | 99.00th=[52167], 99.50th=[52167], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 7135, max= 7135, per=100.00%, avg=7135.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2398KiB/s (2456kB/s)(8192KiB/3416msec) clat (msec): min=168, max=3190, avg=1679.48, stdev=2136.32 lat (msec): min=169, max=3190, avg=1679.76, stdev=2136.32 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 169], 10.00th=[ 169], 20.00th=[ 169], | 30.00th=[ 169], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 3205], | 70.00th=[ 3205], 80.00th=[ 3205], 90.00th=[ 3205], 95.00th=[ 3205], | 99.00th=[ 3205], 99.50th=[ 3205], 99.90th=[ 3205], 99.95th=[ 3205], | 99.99th=[ 3205] bw ( KiB/s): min= 7135, max= 7135, per=82.98%, avg=7135.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.03%, sys=2.20%, ctx=33, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25959: Wed Dec 25 01:26:02 2024 write: IOPS=0, BW=3224KiB/s (3302kB/s)(12.0MiB/3811msec) clat (msec): min=34, max=3344, avg=1261.64, stdev=1812.97 lat (msec): min=34, max=3344, avg=1261.89, stdev=1812.87 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 35], | 30.00th=[ 35], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 405], | 70.00th=[ 3339], 80.00th=[ 3339], 90.00th=[ 3339], 95.00th=[ 3339], | 99.00th=[ 3339], 99.50th=[ 3339], 99.90th=[ 3339], 99.95th=[ 3339], | 99.99th=[ 3339] bw ( KiB/s): min= 8192, max= 8192, per=95.28%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 500=33.33% cpu : usr=0.03%, sys=3.83%, ctx=69, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25960: Wed Dec 25 01:26:02 2024 write: IOPS=6, BW=25.9MiB/s (27.2MB/s)(12.0MiB/463msec) clat (msec): min=51, max=234, avg=140.51, stdev=91.43 lat (msec): min=52, max=234, avg=140.76, stdev=91.40 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 52], | 30.00th=[ 52], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 136], | 70.00th=[ 234], 80.00th=[ 234], 90.00th=[ 234], 95.00th=[ 234], | 99.00th=[ 234], 99.50th=[ 234], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 234] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.43%, sys=13.20%, ctx=38, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4796KiB/s (4911kB/s), 1199KiB/s-3629KiB/s (1228kB/s-3716kB/s), io=16.0MiB (16.8MB), run=3386-3416msec WRITE: bw=8598KiB/s (8805kB/s), 2398KiB/s-25.9MiB/s (2456kB/s-27.2MB/s), io=32.0MiB (33.6MB), run=463-3811msec