== sanity test 398b: DIO and buffer IO race ============== 15:45:20 (1726861520) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.2942 s, 38.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10392: Fri Sep 20 15:45:34 2024 read: IOPS=133, BW=535KiB/s (548kB/s)(5992KiB/11200msec) clat (usec): min=48, max=165223, avg=3561.67, stdev=5156.84 lat (usec): min=48, max=165224, avg=3562.53, stdev=5157.13 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 59], 10.00th=[ 69], 20.00th=[ 1680], | 30.00th=[ 2114], 40.00th=[ 2442], 50.00th=[ 2868], 60.00th=[ 3458], | 70.00th=[ 4293], 80.00th=[ 5538], 90.00th=[ 7046], 95.00th=[ 8455], | 99.00th=[ 13042], 99.50th=[ 15139], 99.90th=[ 42730], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 248, max= 822, per=24.64%, avg=528.14, stdev=117.73, samples=22 iops : min= 62, max= 205, avg=131.95, stdev=29.39, samples=22 write: IOPS=140, BW=562KiB/s (576kB/s)(6296KiB/11200msec) clat (usec): min=426, max=27946, avg=3650.14, stdev=3460.50 lat (usec): min=427, max=27947, avg=3650.93, stdev=3460.46 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 523], 10.00th=[ 553], 20.00th=[ 627], | 30.00th=[ 857], 40.00th=[ 1811], 50.00th=[ 2933], 60.00th=[ 3818], | 70.00th=[ 4621], 80.00th=[ 5997], 90.00th=[ 8094], 95.00th=[10683], | 99.00th=[14615], 99.50th=[18744], 99.90th=[27657], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 296, max= 854, per=24.75%, avg=555.41, stdev=126.65, samples=22 iops : min= 74, max= 213, avg=138.77, stdev=31.64, samples=22 lat (usec) : 50=0.42%, 100=8.01%, 250=0.81%, 500=0.94%, 750=13.44% lat (usec) : 1000=2.70% lat (msec) : 2=7.94%, 4=30.66%, 10=30.86%, 20=3.91%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=24.83%, ctx=6286, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10394: Fri Sep 20 15:45:34 2024 read: IOPS=133, BW=533KiB/s (546kB/s)(5940KiB/11141msec) clat (usec): min=47, max=25651, avg=3332.70, stdev=2659.63 lat (usec): min=47, max=25651, avg=3333.35, stdev=2659.62 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 59], 10.00th=[ 70], 20.00th=[ 1713], | 30.00th=[ 2114], 40.00th=[ 2442], 50.00th=[ 2835], 60.00th=[ 3359], | 70.00th=[ 4178], 80.00th=[ 5211], 90.00th=[ 6587], 95.00th=[ 7832], | 99.00th=[11994], 99.50th=[14746], 99.90th=[22676], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 248, max= 728, per=24.82%, avg=531.91, stdev=109.92, samples=22 iops : min= 62, max= 182, avg=132.86, stdev=27.47, samples=22 write: IOPS=142, BW=570KiB/s (583kB/s)(6348KiB/11141msec) clat (usec): min=418, max=162105, avg=3827.96, stdev=5257.25 lat (usec): min=418, max=162106, avg=3828.64, stdev=5257.24 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 519], 10.00th=[ 545], 20.00th=[ 619], | 30.00th=[ 898], 40.00th=[ 1926], 50.00th=[ 2966], 60.00th=[ 3851], | 70.00th=[ 4817], 80.00th=[ 6325], 90.00th=[ 8586], 95.00th=[ 10421], | 99.00th=[ 14091], 99.50th=[ 16057], 99.90th=[ 26870], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 288, max= 776, per=25.15%, avg=564.27, stdev=112.36, samples=22 iops : min= 72, max= 194, avg=140.95, stdev=28.11, samples=22 lat (usec) : 50=0.62%, 100=7.68%, 250=0.81%, 500=0.94%, 750=12.86% lat (usec) : 1000=2.99% lat (msec) : 2=8.04%, 4=31.58%, 10=30.53%, 20=3.71%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=25.27%, ctx=6214, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10396: Fri Sep 20 15:45:34 2024 read: IOPS=135, BW=542KiB/s (555kB/s)(6028KiB/11126msec) clat (usec): min=47, max=34150, avg=3284.11, stdev=2754.84 lat (usec): min=47, max=34151, avg=3284.73, stdev=2754.89 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 58], 10.00th=[ 66], 20.00th=[ 206], | 30.00th=[ 2057], 40.00th=[ 2442], 50.00th=[ 2802], 60.00th=[ 3326], | 70.00th=[ 4080], 80.00th=[ 5276], 90.00th=[ 6652], 95.00th=[ 7898], | 99.00th=[11469], 99.50th=[13435], 99.90th=[27132], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 216, max= 808, per=25.09%, avg=537.73, stdev=133.36, samples=22 iops : min= 54, max= 202, avg=134.32, stdev=33.37, samples=22 write: IOPS=140, BW=563KiB/s (576kB/s)(6260KiB/11126msec) clat (usec): min=449, max=164930, avg=3871.55, stdev=5479.87 lat (usec): min=449, max=164931, avg=3872.36, stdev=5479.91 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 619], | 30.00th=[ 848], 40.00th=[ 2089], 50.00th=[ 3064], 60.00th=[ 3785], | 70.00th=[ 4817], 80.00th=[ 6194], 90.00th=[ 8455], 95.00th=[ 10159], | 99.00th=[ 15401], 99.50th=[ 22938], 99.90th=[ 36963], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 312, max= 776, per=24.85%, avg=557.68, stdev=102.28, samples=22 iops : min= 78, max= 194, avg=139.32, stdev=25.62, samples=22 lat (usec) : 50=0.81%, 100=8.33%, 250=0.78%, 500=0.88%, 750=13.05% lat (usec) : 1000=2.90% lat (msec) : 2=7.52%, 4=31.18%, 10=30.96%, 20=3.19%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=25.02%, ctx=6287, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10398: Fri Sep 20 15:45:34 2024 read: IOPS=135, BW=543KiB/s (556kB/s)(6052KiB/11149msec) clat (usec): min=46, max=169791, avg=3498.06, stdev=5181.36 lat (usec): min=46, max=169792, avg=3498.62, stdev=5181.38 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 59], 10.00th=[ 69], 20.00th=[ 1156], | 30.00th=[ 2089], 40.00th=[ 2442], 50.00th=[ 2737], 60.00th=[ 3294], | 70.00th=[ 4178], 80.00th=[ 5473], 90.00th=[ 6849], 95.00th=[ 8225], | 99.00th=[ 13173], 99.50th=[ 18744], 99.90th=[ 31589], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 255, max= 800, per=25.17%, avg=539.50, stdev=131.69, samples=22 iops : min= 63, max= 200, avg=134.73, stdev=33.02, samples=22 write: IOPS=139, BW=559KiB/s (573kB/s)(6236KiB/11149msec) clat (usec): min=453, max=25683, avg=3681.24, stdev=3387.37 lat (usec): min=453, max=25683, avg=3682.82, stdev=3386.89 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 644], | 30.00th=[ 947], 40.00th=[ 1958], 50.00th=[ 2966], 60.00th=[ 3654], | 70.00th=[ 4686], 80.00th=[ 5997], 90.00th=[ 8291], 95.00th=[10421], | 99.00th=[14746], 99.50th=[16909], 99.90th=[24773], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 335, max= 912, per=24.77%, avg=555.91, stdev=130.91, samples=22 iops : min= 83, max= 228, avg=138.82, stdev=32.86, samples=22 lat (usec) : 50=0.46%, 100=8.46%, 250=0.68%, 500=0.88%, 750=11.72% lat (usec) : 1000=3.45% lat (msec) : 2=8.46%, 4=31.84%, 10=29.95%, 20=3.81%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=25.15%, ctx=6307, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2144KiB/s (2195kB/s), 533KiB/s-543KiB/s (546kB/s-556kB/s), io=23.4MiB (24.6MB), run=11126-11200msec WRITE: bw=2245KiB/s (2299kB/s), 559KiB/s-570KiB/s (573kB/s-583kB/s), io=24.6MiB (25.7MB), run=11126-11200msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10391: Fri Sep 20 15:45:59 2024 read: IOPS=41, BW=165KiB/s (169kB/s)(5992KiB/36250msec) clat (usec): min=1631, max=167759, avg=4936.24, stdev=4908.94 lat (usec): min=1632, max=167760, avg=4937.44, stdev=4908.89 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2474], 10.00th=[ 2769], 20.00th=[ 2999], | 30.00th=[ 3163], 40.00th=[ 3425], 50.00th=[ 4293], 60.00th=[ 5342], | 70.00th=[ 5669], 80.00th=[ 6063], 90.00th=[ 7046], 95.00th=[ 8848], | 99.00th=[ 15664], 99.50th=[ 20317], 99.90th=[ 29230], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 72, max= 280, per=24.93%, avg=165.01, stdev=42.68, samples=72 iops : min= 18, max= 70, avg=41.18, stdev=10.67, samples=72 write: IOPS=43, BW=174KiB/s (178kB/s)(6296KiB/36250msec) clat (usec): min=9616, max=57737, avg=18249.22, stdev=4620.46 lat (usec): min=9617, max=57738, avg=18250.52, stdev=4620.38 clat percentiles (usec): | 1.00th=[11469], 5.00th=[12518], 10.00th=[13042], 20.00th=[14222], | 30.00th=[15401], 40.00th=[16319], 50.00th=[17433], 60.00th=[18744], | 70.00th=[20317], 80.00th=[21627], 90.00th=[23725], 95.00th=[26608], | 99.00th=[33162], 99.50th=[35390], 99.90th=[50594], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 48, max= 232, per=25.01%, avg=173.35, stdev=28.30, samples=72 iops : min= 12, max= 58, avg=43.26, stdev= 7.09, samples=72 lat (msec) : 2=0.59%, 4=22.59%, 10=23.99%, 20=36.17%, 50=16.57% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.15%, sys=7.83%, ctx=3852, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10393: Fri Sep 20 15:45:59 2024 read: IOPS=41, BW=164KiB/s (168kB/s)(5940KiB/36197msec) clat (usec): min=1800, max=170031, avg=4934.58, stdev=4918.16 lat (usec): min=1801, max=170034, avg=4935.66, stdev=4918.20 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 6], 80.00th=[ 6], 90.00th=[ 7], 95.00th=[ 9], | 99.00th=[ 14], 99.50th=[ 16], 99.90th=[ 44], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 16, max= 248, per=24.83%, avg=164.35, stdev=50.59, samples=72 iops : min= 4, max= 62, avg=41.01, stdev=12.63, samples=72 write: IOPS=43, BW=175KiB/s (180kB/s)(6348KiB/36197msec) clat (usec): min=9436, max=57628, avg=18107.63, stdev=4684.71 lat (usec): min=9437, max=57628, avg=18109.02, stdev=4684.44 clat percentiles (usec): | 1.00th=[11338], 5.00th=[12518], 10.00th=[13173], 20.00th=[14091], | 30.00th=[15270], 40.00th=[16057], 50.00th=[17171], 60.00th=[18744], | 70.00th=[20055], 80.00th=[21365], 90.00th=[23725], 95.00th=[26084], | 99.00th=[32637], 99.50th=[36439], 99.90th=[45351], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 56, max= 240, per=25.25%, avg=175.01, stdev=27.77, samples=72 iops : min= 14, max= 60, avg=43.68, stdev= 6.95, samples=72 lat (msec) : 2=0.39%, 4=21.65%, 10=24.90%, 20=37.04%, 50=15.95% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.14%, sys=7.94%, ctx=3898, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10395: Fri Sep 20 15:45:59 2024 read: IOPS=42, BW=169KiB/s (173kB/s)(6028KiB/35741msec) clat (usec): min=1787, max=39924, avg=4856.32, stdev=2575.36 lat (usec): min=1787, max=39924, avg=4857.37, stdev=2575.34 clat percentiles (usec): | 1.00th=[ 2057], 5.00th=[ 2540], 10.00th=[ 2737], 20.00th=[ 2999], | 30.00th=[ 3195], 40.00th=[ 3458], 50.00th=[ 4359], 60.00th=[ 5342], | 70.00th=[ 5604], 80.00th=[ 6063], 90.00th=[ 6980], 95.00th=[ 8586], | 99.00th=[14091], 99.50th=[16909], 99.90th=[27919], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 56, max= 264, per=25.51%, avg=168.86, stdev=43.16, samples=71 iops : min= 14, max= 66, avg=42.18, stdev=10.80, samples=71 write: IOPS=43, BW=175KiB/s (179kB/s)(6260KiB/35741msec) clat (msec): min=9, max=187, avg=18.08, stdev= 6.30 lat (msec): min=9, max=187, avg=18.08, stdev= 6.30 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 15], | 30.00th=[ 16], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 20], 80.00th=[ 22], 90.00th=[ 24], 95.00th=[ 26], | 99.00th=[ 34], 99.50th=[ 38], 99.90th=[ 51], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 64, max= 232, per=25.23%, avg=174.83, stdev=28.71, samples=71 iops : min= 16, max= 58, avg=43.68, stdev= 7.17, samples=71 lat (msec) : 2=0.33%, 4=22.53%, 10=24.58%, 20=37.66%, 50=14.84% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.13%, sys=7.92%, ctx=3870, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10397: Fri Sep 20 15:45:59 2024 read: IOPS=41, BW=167KiB/s (171kB/s)(6052KiB/36167msec) clat (usec): min=1656, max=43951, avg=4814.70, stdev=2636.74 lat (usec): min=1657, max=43952, avg=4815.83, stdev=2636.66 clat percentiles (usec): | 1.00th=[ 2089], 5.00th=[ 2540], 10.00th=[ 2769], 20.00th=[ 2999], | 30.00th=[ 3195], 40.00th=[ 3458], 50.00th=[ 4424], 60.00th=[ 5342], | 70.00th=[ 5538], 80.00th=[ 5997], 90.00th=[ 6915], 95.00th=[ 8455], | 99.00th=[13435], 99.50th=[16188], 99.90th=[39584], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 40, max= 272, per=25.30%, avg=167.46, stdev=48.28, samples=72 iops : min= 10, max= 68, avg=41.79, stdev=12.06, samples=72 write: IOPS=43, BW=172KiB/s (177kB/s)(6236KiB/36167msec) clat (msec): min=9, max=209, avg=18.44, stdev= 6.68 lat (msec): min=9, max=209, avg=18.44, stdev= 6.68 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 15], | 30.00th=[ 16], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 21], 80.00th=[ 22], 90.00th=[ 24], 95.00th=[ 27], | 99.00th=[ 34], 99.50th=[ 41], 99.90th=[ 51], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 56, max= 216, per=24.85%, avg=172.24, stdev=26.99, samples=72 iops : min= 14, max= 54, avg=42.99, stdev= 6.75, samples=72 lat (msec) : 2=0.39%, 4=22.40%, 10=25.13%, 20=35.25%, 50=16.76% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.11%, sys=7.93%, ctx=3878, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=662KiB/s (678kB/s), 164KiB/s-169KiB/s (168kB/s-173kB/s), io=23.4MiB (24.6MB), run=35741-36250msec WRITE: bw=694KiB/s (710kB/s), 172KiB/s-175KiB/s (177kB/s-180kB/s), io=24.6MiB (25.7MB), run=35741-36250msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10438: Fri Sep 20 15:46:04 2024 read: IOPS=86, BW=1381KiB/s (1414kB/s)(5920KiB/4287msec) clat (usec): min=52, max=22828, avg=8506.10, stdev=4678.26 lat (usec): min=52, max=22829, avg=8507.02, stdev=4678.29 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 63], 10.00th=[ 74], 20.00th=[ 6390], | 30.00th=[ 7898], 40.00th=[ 8586], 50.00th=[ 9241], 60.00th=[ 9896], | 70.00th=[10945], 80.00th=[11863], 90.00th=[13435], 95.00th=[14615], | 99.00th=[18744], 99.50th=[19268], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 864, max= 1625, per=23.99%, avg=1339.12, stdev=237.30, samples=8 iops : min= 54, max= 101, avg=83.62, stdev=14.74, samples=8 write: IOPS=92, BW=1485KiB/s (1521kB/s)(6368KiB/4287msec) clat (usec): min=656, max=24096, avg=2752.03, stdev=2921.78 lat (usec): min=657, max=24096, avg=2753.24, stdev=2921.82 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 709], 10.00th=[ 742], 20.00th=[ 807], | 30.00th=[ 857], 40.00th=[ 971], 50.00th=[ 2114], 60.00th=[ 2573], | 70.00th=[ 3130], 80.00th=[ 4228], 90.00th=[ 5866], 95.00th=[ 7111], | 99.00th=[16188], 99.50th=[21890], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 1024, max= 2112, per=24.91%, avg=1463.00, stdev=347.91, samples=8 iops : min= 64, max= 132, avg=91.38, stdev=21.67, samples=8 lat (usec) : 100=7.42%, 250=1.43%, 750=5.47%, 1000=15.89% lat (msec) : 2=4.04%, 4=15.23%, 10=29.95%, 20=20.18%, 50=0.39% cpu : usr=0.19%, sys=23.05%, ctx=2089, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10440: Fri Sep 20 15:46:04 2024 read: IOPS=84, BW=1348KiB/s (1381kB/s)(5744KiB/4260msec) clat (usec): min=50, max=24238, avg=8255.62, stdev=4623.29 lat (usec): min=50, max=24239, avg=8257.89, stdev=4624.94 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 62], 10.00th=[ 74], 20.00th=[ 4817], | 30.00th=[ 7767], 40.00th=[ 8586], 50.00th=[ 9110], 60.00th=[ 9634], | 70.00th=[10683], 80.00th=[11469], 90.00th=[12780], 95.00th=[14484], | 99.00th=[18744], 99.50th=[21627], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 576, max= 1824, per=23.64%, avg=1320.00, stdev=385.05, samples=8 iops : min= 36, max= 114, avg=82.50, stdev=24.07, samples=8 write: IOPS=96, BW=1536KiB/s (1573kB/s)(6544KiB/4260msec) clat (usec): min=622, max=37193, avg=3021.09, stdev=4083.69 lat (usec): min=623, max=37195, avg=3021.98, stdev=4083.77 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 717], 10.00th=[ 750], 20.00th=[ 791], | 30.00th=[ 848], 40.00th=[ 938], 50.00th=[ 1598], 60.00th=[ 2474], | 70.00th=[ 3032], 80.00th=[ 4424], 90.00th=[ 5932], 95.00th=[ 9765], | 99.00th=[23725], 99.50th=[24773], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 864, max= 2048, per=25.13%, avg=1476.00, stdev=360.19, samples=8 iops : min= 54, max= 128, avg=92.25, stdev=22.51, samples=8 lat (usec) : 100=7.68%, 250=0.91%, 750=5.86%, 1000=17.97% lat (msec) : 2=3.65%, 4=14.58%, 10=29.95%, 20=18.36%, 50=1.04% cpu : usr=0.09%, sys=22.19%, ctx=2053, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10442: Fri Sep 20 15:46:04 2024 read: IOPS=91, BW=1458KiB/s (1493kB/s)(6256KiB/4290msec) clat (usec): min=49, max=26966, avg=7506.18, stdev=5219.17 lat (usec): min=49, max=26967, avg=7506.88, stdev=5219.08 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 59], 10.00th=[ 62], 20.00th=[ 78], | 30.00th=[ 6849], 40.00th=[ 7963], 50.00th=[ 8979], 60.00th=[ 9765], | 70.00th=[10552], 80.00th=[11469], 90.00th=[12911], 95.00th=[14353], | 99.00th=[19268], 99.50th=[20841], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 416, max= 2080, per=24.50%, avg=1368.00, stdev=484.02, samples=8 iops : min= 26, max= 130, avg=85.50, stdev=30.25, samples=8 write: IOPS=87, BW=1406KiB/s (1440kB/s)(6032KiB/4290msec) clat (usec): min=636, max=36530, avg=3455.76, stdev=4823.78 lat (usec): min=637, max=36531, avg=3456.76, stdev=4823.78 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 709], 10.00th=[ 742], 20.00th=[ 816], | 30.00th=[ 865], 40.00th=[ 988], 50.00th=[ 2212], 60.00th=[ 2802], | 70.00th=[ 3490], 80.00th=[ 4883], 90.00th=[ 6456], 95.00th=[13304], | 99.00th=[27657], 99.50th=[34341], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 544, max= 2176, per=22.74%, avg=1336.00, stdev=482.51, samples=8 iops : min= 34, max= 136, avg=83.50, stdev=30.16, samples=8 lat (usec) : 50=0.13%, 100=13.15%, 250=0.91%, 500=0.26%, 750=5.21% lat (usec) : 1000=14.84% lat (msec) : 2=4.04%, 4=11.98%, 10=28.39%, 20=19.66%, 50=1.43% cpu : usr=0.23%, sys=21.38%, ctx=2042, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10444: Fri Sep 20 15:46:04 2024 read: IOPS=90, BW=1446KiB/s (1481kB/s)(6032KiB/4171msec) clat (usec): min=49, max=54178, avg=7915.51, stdev=5169.28 lat (usec): min=50, max=54178, avg=7916.06, stdev=5169.29 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 61], 10.00th=[ 64], 20.00th=[ 91], | 30.00th=[ 7308], 40.00th=[ 8160], 50.00th=[ 8979], 60.00th=[ 9503], | 70.00th=[10421], 80.00th=[11207], 90.00th=[12911], 95.00th=[14091], | 99.00th=[16057], 99.50th=[16450], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 704, max= 2071, per=25.19%, avg=1406.50, stdev=380.40, samples=8 iops : min= 44, max= 129, avg=87.75, stdev=23.72, samples=8 write: IOPS=93, BW=1500KiB/s (1536kB/s)(6256KiB/4171msec) clat (usec): min=640, max=34904, avg=2984.17, stdev=3894.41 lat (usec): min=641, max=34905, avg=2985.03, stdev=3894.49 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 709], 10.00th=[ 742], 20.00th=[ 791], | 30.00th=[ 840], 40.00th=[ 922], 50.00th=[ 1450], 60.00th=[ 2409], | 70.00th=[ 2999], 80.00th=[ 4146], 90.00th=[ 5997], 95.00th=[ 9634], | 99.00th=[20841], 99.50th=[24511], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 800, max= 1888, per=25.10%, avg=1474.62, stdev=391.43, samples=8 iops : min= 50, max= 118, avg=92.00, stdev=24.50, samples=8 lat (usec) : 50=0.13%, 100=10.16%, 250=0.52%, 750=6.38%, 1000=16.02% lat (msec) : 2=4.30%, 4=13.80%, 10=29.56%, 20=18.49%, 50=0.52% lat (msec) : 100=0.13% cpu : usr=0.02%, sys=22.45%, ctx=2108, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5583KiB/s (5717kB/s), 1348KiB/s-1458KiB/s (1381kB/s-1493kB/s), io=23.4MiB (24.5MB), run=4171-4290msec WRITE: bw=5874KiB/s (6015kB/s), 1406KiB/s-1536KiB/s (1440kB/s-1573kB/s), io=24.6MiB (25.8MB), run=4171-4290msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10437: Fri Sep 20 15:46:09 2024 read: IOPS=40, BW=646KiB/s (661kB/s)(5920KiB/9166msec) clat (usec): min=1907, max=22460, avg=4271.29, stdev=3038.70 lat (usec): min=1908, max=22461, avg=4272.15, stdev=3038.66 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2147], 10.00th=[ 2376], 20.00th=[ 2704], | 30.00th=[ 2999], 40.00th=[ 3326], 50.00th=[ 3458], 60.00th=[ 3589], | 70.00th=[ 3818], 80.00th=[ 4490], 90.00th=[ 6521], 95.00th=[10290], | 99.00th=[20055], 99.50th=[22152], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 352, max= 1088, per=24.81%, avg=638.06, stdev=208.78, samples=18 iops : min= 22, max= 68, avg=39.78, stdev=13.12, samples=18 write: IOPS=43, BW=695KiB/s (711kB/s)(6368KiB/9166msec) clat (usec): min=10332, max=55479, avg=18823.99, stdev=5322.48 lat (usec): min=10333, max=55480, avg=18825.37, stdev=5322.44 clat percentiles (usec): | 1.00th=[11076], 5.00th=[12780], 10.00th=[13435], 20.00th=[14222], | 30.00th=[15139], 40.00th=[16909], 50.00th=[17957], 60.00th=[19268], | 70.00th=[20841], 80.00th=[22414], 90.00th=[25035], 95.00th=[27132], | 99.00th=[37487], 99.50th=[45876], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 544, max= 800, per=25.68%, avg=694.89, stdev=90.11, samples=18 iops : min= 34, max= 50, avg=43.33, stdev= 5.52, samples=18 lat (msec) : 2=0.91%, 4=34.77%, 10=9.90%, 20=35.81%, 50=18.49% lat (msec) : 100=0.13% cpu : usr=0.05%, sys=7.77%, ctx=899, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10439: Fri Sep 20 15:46:09 2024 read: IOPS=38, BW=617KiB/s (632kB/s)(5744KiB/9310msec) clat (usec): min=1805, max=21030, avg=4349.26, stdev=2876.49 lat (usec): min=1806, max=21031, avg=4350.14, stdev=2876.45 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2147], 10.00th=[ 2311], 20.00th=[ 2769], | 30.00th=[ 3097], 40.00th=[ 3392], 50.00th=[ 3556], 60.00th=[ 3687], | 70.00th=[ 3982], 80.00th=[ 4686], 90.00th=[ 7373], 95.00th=[10552], | 99.00th=[17695], 99.50th=[20841], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 320, max= 992, per=24.19%, avg=622.22, stdev=182.42, samples=18 iops : min= 20, max= 62, avg=38.89, stdev=11.40, samples=18 write: IOPS=43, BW=703KiB/s (720kB/s)(6544KiB/9310msec) clat (usec): min=10373, max=38621, avg=18849.10, stdev=4698.63 lat (usec): min=10374, max=38622, avg=18850.53, stdev=4698.54 clat percentiles (usec): | 1.00th=[11207], 5.00th=[12911], 10.00th=[13566], 20.00th=[14615], | 30.00th=[15533], 40.00th=[17433], 50.00th=[18482], 60.00th=[19530], | 70.00th=[20841], 80.00th=[22414], 90.00th=[24511], 95.00th=[26608], | 99.00th=[33817], 99.50th=[36963], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 544, max= 832, per=25.88%, avg=700.44, stdev=78.30, samples=18 iops : min= 34, max= 52, avg=43.78, stdev= 4.89, samples=18 lat (msec) : 2=0.78%, 4=32.16%, 10=11.20%, 20=36.20%, 50=19.66% cpu : usr=0.14%, sys=7.59%, ctx=887, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10441: Fri Sep 20 15:46:09 2024 read: IOPS=43, BW=698KiB/s (715kB/s)(6256KiB/8957msec) clat (usec): min=1884, max=22333, avg=4308.67, stdev=2882.22 lat (usec): min=1885, max=22333, avg=4309.65, stdev=2882.11 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2212], 10.00th=[ 2409], 20.00th=[ 2802], | 30.00th=[ 3064], 40.00th=[ 3326], 50.00th=[ 3523], 60.00th=[ 3687], | 70.00th=[ 3916], 80.00th=[ 4555], 90.00th=[ 7046], 95.00th=[11076], | 99.00th=[16909], 99.50th=[19792], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 384, max= 1248, per=27.66%, avg=711.29, stdev=220.20, samples=17 iops : min= 24, max= 78, avg=44.35, stdev=13.80, samples=17 write: IOPS=42, BW=673KiB/s (690kB/s)(6032KiB/8957msec) clat (usec): min=10669, max=53156, avg=19219.26, stdev=5109.17 lat (usec): min=10670, max=53157, avg=19220.72, stdev=5109.07 clat percentiles (usec): | 1.00th=[11600], 5.00th=[12911], 10.00th=[13566], 20.00th=[14615], | 30.00th=[16057], 40.00th=[17433], 50.00th=[18744], 60.00th=[20055], | 70.00th=[21103], 80.00th=[22676], 90.00th=[25297], 95.00th=[28181], | 99.00th=[36439], 99.50th=[38011], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 544, max= 832, per=24.55%, avg=664.24, stdev=87.04, samples=17 iops : min= 34, max= 52, avg=41.41, stdev= 5.34, samples=17 lat (msec) : 2=0.78%, 4=36.20%, 10=10.81%, 20=32.81%, 50=19.27% lat (msec) : 100=0.13% cpu : usr=0.13%, sys=7.86%, ctx=888, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10443: Fri Sep 20 15:46:09 2024 read: IOPS=41, BW=662KiB/s (678kB/s)(6032KiB/9106msec) clat (usec): min=1855, max=25067, avg=4473.87, stdev=3328.80 lat (usec): min=1855, max=25067, avg=4474.79, stdev=3328.73 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2147], 10.00th=[ 2343], 20.00th=[ 2802], | 30.00th=[ 3032], 40.00th=[ 3261], 50.00th=[ 3425], 60.00th=[ 3589], | 70.00th=[ 3818], 80.00th=[ 4817], 90.00th=[ 7701], 95.00th=[11731], | 99.00th=[19268], 99.50th=[22414], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 288, max= 1024, per=25.57%, avg=657.67, stdev=181.38, samples=18 iops : min= 18, max= 64, avg=41.06, stdev=11.32, samples=18 write: IOPS=42, BW=687KiB/s (704kB/s)(6256KiB/9106msec) clat (usec): min=10749, max=40905, avg=18934.04, stdev=5005.29 lat (usec): min=10750, max=40906, avg=18935.43, stdev=5005.28 clat percentiles (usec): | 1.00th=[11338], 5.00th=[12387], 10.00th=[13435], 20.00th=[14615], | 30.00th=[15664], 40.00th=[16909], 50.00th=[18482], 60.00th=[19792], | 70.00th=[21103], 80.00th=[22676], 90.00th=[24773], 95.00th=[27919], | 99.00th=[35390], 99.50th=[37487], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 576, max= 832, per=25.36%, avg=686.11, stdev=69.74, samples=18 iops : min= 36, max= 52, avg=42.83, stdev= 4.42, samples=18 lat (msec) : 2=1.30%, 4=33.59%, 10=10.42%, 20=34.37%, 50=20.31% cpu : usr=0.11%, sys=7.62%, ctx=905, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2573KiB/s (2634kB/s), 617KiB/s-698KiB/s (632kB/s-715kB/s), io=23.4MiB (24.5MB), run=8957-9310msec WRITE: bw=2707KiB/s (2772kB/s), 673KiB/s-703KiB/s (690kB/s-720kB/s), io=24.6MiB (25.8MB), run=8957-9310msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10459: Fri Sep 20 15:46:16 2024 read: IOPS=1, BW=1308KiB/s (1339kB/s)(7168KiB/5480msec) clat (usec): min=609, max=1211.3k, avg=572615.67, stdev=524795.62 lat (usec): min=610, max=1211.3k, avg=572616.97, stdev=524795.26 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 611], 10.00th=[ 611], | 20.00th=[ 742], 30.00th=[ 72877], 40.00th=[ 72877], | 50.00th=[ 859833], 60.00th=[ 901776], 70.00th=[ 901776], | 80.00th=[ 952108], 90.00th=[1216349], 95.00th=[1216349], | 99.00th=[1216349], 99.50th=[1216349], 99.90th=[1216349], | 99.95th=[1216349], 99.99th=[1216349] bw ( KiB/s): min= 2048, max= 3494, per=64.48%, avg=2409.50, stdev=723.00, samples=4 iops : min= 2, max= 3, avg= 2.25, stdev= 0.50, samples=4 write: IOPS=0, BW=934KiB/s (957kB/s)(5120KiB/5480msec) clat (msec): min=31, max=1306, avg=292.27, stdev=566.90 lat (msec): min=31, max=1306, avg=292.32, stdev=566.92 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 41], 60.00th=[ 41], | 70.00th=[ 47], 80.00th=[ 47], 90.00th=[ 1301], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 1750, max= 5242, per=66.82%, avg=3496.00, stdev=2469.22, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 lat (usec) : 750=16.67% lat (msec) : 50=33.33%, 100=8.33%, 1000=25.00% cpu : usr=0.02%, sys=12.89%, ctx=1198, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10461: Fri Sep 20 15:46:16 2024 read: IOPS=0, BW=456KiB/s (467kB/s)(2048KiB/4493msec) clat (msec): min=58, max=231, avg=145.05, stdev=122.06 lat (msec): min=58, max=231, avg=145.05, stdev=122.06 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 59], | 30.00th=[ 59], 40.00th=[ 59], 50.00th=[ 59], 60.00th=[ 232], | 70.00th=[ 232], 80.00th=[ 232], 90.00th=[ 232], 95.00th=[ 232], | 99.00th=[ 232], 99.50th=[ 232], 99.90th=[ 232], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 2043, max= 2048, per=54.74%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2279KiB/s (2334kB/s)(10.0MiB/4493msec) clat (msec): min=14, max=1279, avg=417.41, stdev=503.12 lat (msec): min=14, max=1279, avg=417.45, stdev=503.12 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 18], | 30.00th=[ 22], 40.00th=[ 39], 50.00th=[ 103], 60.00th=[ 127], | 70.00th=[ 969], 80.00th=[ 969], 90.00th=[ 1053], 95.00th=[ 1284], | 99.00th=[ 1284], 99.50th=[ 1284], 99.90th=[ 1284], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 1796, max= 6144, per=77.04%, avg=4030.75, stdev=1776.45, samples=4 iops : min= 1, max= 6, avg= 3.50, stdev= 2.08, samples=4 lat (msec) : 20=16.67%, 50=16.67%, 100=8.33%, 250=25.00%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=8.41%, ctx=144, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10463: Fri Sep 20 15:46:16 2024 read: IOPS=1, BW=1395KiB/s (1429kB/s)(6144KiB/4404msec) clat (msec): min=50, max=1211, avg=442.29, stdev=485.44 lat (msec): min=50, max=1211, avg=442.29, stdev=485.44 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 51], 10.00th=[ 51], 20.00th=[ 69], | 30.00th=[ 69], 40.00th=[ 182], 50.00th=[ 182], 60.00th=[ 262], | 70.00th=[ 877], 80.00th=[ 877], 90.00th=[ 1217], 95.00th=[ 1217], | 99.00th=[ 1217], 99.50th=[ 1217], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 2048, max= 4096, per=91.12%, avg=3405.00, stdev=1175.26, samples=3 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=3 write: IOPS=1, BW=1395KiB/s (1429kB/s)(6144KiB/4404msec) clat (msec): min=20, max=1052, avg=289.20, stdev=434.38 lat (msec): min=20, max=1052, avg=289.23, stdev=434.38 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 28], 50.00th=[ 28], 60.00th=[ 35], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 1053], 95.00th=[ 1053], | 99.00th=[ 1053], 99.50th=[ 1053], 99.90th=[ 1053], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 1790, max= 4096, per=57.42%, avg=3004.25, stdev=1264.61, samples=4 iops : min= 1, max= 4, avg= 2.50, stdev= 1.73, samples=4 lat (msec) : 50=33.33%, 100=16.67%, 250=8.33%, 500=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.07%, sys=15.49%, ctx=725, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10464: Fri Sep 20 15:46:16 2024 read: IOPS=1, BW=1692KiB/s (1733kB/s)(5120KiB/3026msec) clat (msec): min=3, max=655, avg=145.84, stdev=285.43 lat (msec): min=3, max=655, avg=145.84, stdev=285.43 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 22], 60.00th=[ 22], | 70.00th=[ 44], 80.00th=[ 44], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 2048, max= 6144, per=100.00%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=2, BW=2369KiB/s (2426kB/s)(7168KiB/3026msec) clat (msec): min=15, max=1027, avg=321.57, stdev=465.34 lat (msec): min=15, max=1027, avg=321.61, stdev=465.34 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16], | 30.00th=[ 16], 40.00th=[ 16], 50.00th=[ 43], 60.00th=[ 167], | 70.00th=[ 167], 80.00th=[ 969], 90.00th=[ 1028], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 2043, max= 4096, per=48.91%, avg=2558.75, stdev=1024.84, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 lat (msec) : 4=8.33%, 10=8.33%, 20=25.00%, 50=25.00%, 250=8.33% lat (msec) : 750=8.33%, 1000=8.33% cpu : usr=0.03%, sys=27.14%, ctx=147, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3737KiB/s (3827kB/s), 456KiB/s-1692KiB/s (467kB/s-1733kB/s), io=20.0MiB (20.0MB), run=3026-5480msec WRITE: bw=5232KiB/s (5358kB/s), 934KiB/s-2369KiB/s (957kB/s-2426kB/s), io=28.0MiB (29.4MB), run=3026-5480msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10457: Fri Sep 20 15:46:16 2024 read: IOPS=1, BW=1286KiB/s (1317kB/s)(7168KiB/5572msec) clat (msec): min=15, max=1175, avg=561.02, stdev=514.00 lat (msec): min=15, max=1175, avg=561.05, stdev=514.02 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 37], | 30.00th=[ 54], 40.00th=[ 54], 50.00th=[ 735], 60.00th=[ 818], | 70.00th=[ 818], 80.00th=[ 1083], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 1793, max= 3586, per=64.46%, avg=2368.75, stdev=820.35, samples=4 iops : min= 1, max= 3, avg= 2.00, stdev= 0.82, samples=4 write: IOPS=0, BW=919KiB/s (941kB/s)(5120KiB/5572msec) clat (msec): min=43, max=1344, avg=323.02, stdev=571.46 lat (msec): min=43, max=1344, avg=323.08, stdev=571.46 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 44], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 70], 60.00th=[ 70], | 70.00th=[ 102], 80.00th=[ 102], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 1793, max= 3586, per=46.47%, avg=2390.67, stdev=1035.19, samples=3 iops : min= 1, max= 3, avg= 1.67, stdev= 1.15, samples=3 lat (msec) : 20=8.33%, 50=16.67%, 100=25.00%, 250=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=2.33%, ctx=52, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10458: Fri Sep 20 15:46:16 2024 read: IOPS=0, BW=1004KiB/s (1029kB/s)(2048KiB/2039msec) clat (usec): min=15795, max=49635, avg=32715.34, stdev=23928.37 lat (usec): min=15796, max=49635, avg=32716.28, stdev=23927.88 clat percentiles (usec): | 1.00th=[15795], 5.00th=[15795], 10.00th=[15795], 20.00th=[15795], | 30.00th=[15795], 40.00th=[15795], 50.00th=[15795], 60.00th=[49546], | 70.00th=[49546], 80.00th=[49546], 90.00th=[49546], 95.00th=[49546], | 99.00th=[49546], 99.50th=[49546], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 2048, max= 2048, per=55.73%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=5022KiB/s (5143kB/s)(10.0MiB/2039msec) clat (msec): min=32, max=1102, avg=195.86, stdev=352.10 lat (msec): min=32, max=1102, avg=195.91, stdev=352.10 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 34], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 47], 60.00th=[ 48], | 70.00th=[ 59], 80.00th=[ 59], 90.00th=[ 518], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 2048, max=16384, per=100.00%, avg=9216.00, stdev=10137.08, samples=2 iops : min= 2, max= 16, avg= 9.00, stdev= 9.90, samples=2 lat (msec) : 20=8.33%, 50=66.67%, 100=8.33%, 750=8.33% cpu : usr=0.00%, sys=5.15%, ctx=25, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10460: Fri Sep 20 15:46:16 2024 read: IOPS=1, BW=1358KiB/s (1391kB/s)(6144KiB/4523msec) clat (msec): min=16, max=1170, avg=473.80, stdev=518.23 lat (msec): min=16, max=1170, avg=473.80, stdev=518.23 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 34], 50.00th=[ 34], 60.00th=[ 651], | 70.00th=[ 944], 80.00th=[ 944], 90.00th=[ 1167], 95.00th=[ 1167], | 99.00th=[ 1167], 99.50th=[ 1167], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 1777, max= 4096, per=69.39%, avg=2550.00, stdev=1338.88, samples=3 iops : min= 1, max= 4, avg= 2.00, stdev= 1.73, samples=3 write: IOPS=1, BW=1358KiB/s (1391kB/s)(6144KiB/4523msec) clat (msec): min=42, max=1012, avg=269.60, stdev=373.17 lat (msec): min=42, max=1012, avg=269.65, stdev=373.18 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 58], | 30.00th=[ 58], 40.00th=[ 99], 50.00th=[ 99], 60.00th=[ 134], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 1011], 95.00th=[ 1011], | 99.00th=[ 1011], 99.50th=[ 1011], 99.90th=[ 1011], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 1777, max= 4096, per=64.59%, avg=3323.00, stdev=1338.88, samples=3 iops : min= 1, max= 4, avg= 3.00, stdev= 1.73, samples=3 lat (msec) : 20=8.33%, 50=25.00%, 100=16.67%, 250=8.33%, 500=8.33% lat (msec) : 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=2.26%, ctx=58, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10462: Fri Sep 20 15:46:16 2024 read: IOPS=2, BW=2557KiB/s (2619kB/s)(5120KiB/2002msec) clat (msec): min=13, max=470, avg=108.07, stdev=202.75 lat (msec): min=13, max=470, avg=108.07, stdev=202.75 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 15], 50.00th=[ 15], 60.00th=[ 15], | 70.00th=[ 28], 80.00th=[ 28], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3580KiB/s (3666kB/s)(7168KiB/2002msec) clat (msec): min=33, max=1039, avg=201.29, stdev=371.48 lat (msec): min=33, max=1039, avg=201.35, stdev=371.50 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 35], | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 57], 60.00th=[ 66], | 70.00th=[ 66], 80.00th=[ 142], 90.00th=[ 1036], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 2048, max=10240, per=92.88%, avg=4778.67, stdev=4729.65, samples=3 iops : min= 2, max= 10, avg= 4.67, stdev= 4.62, samples=3 lat (msec) : 20=25.00%, 50=33.33%, 100=16.67%, 250=8.33%, 500=8.33% cpu : usr=0.05%, sys=3.80%, ctx=33, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3676KiB/s (3764kB/s), 1004KiB/s-2557KiB/s (1029kB/s-2619kB/s), io=20.0MiB (20.0MB), run=2002-5572msec WRITE: bw=5146KiB/s (5269kB/s), 919KiB/s-5022KiB/s (941kB/s-5143kB/s), io=28.0MiB (29.4MB), run=2002-5572msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10474: Fri Sep 20 15:46:19 2024 read: IOPS=1, BW=6200KiB/s (6349kB/s)(12.0MiB/1982msec) clat (msec): min=121, max=1125, avg=658.44, stdev=505.39 lat (msec): min=121, max=1125, avg=658.44, stdev=505.39 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 123], 10.00th=[ 123], 20.00th=[ 123], | 30.00th=[ 123], 40.00th=[ 726], 50.00th=[ 726], 60.00th=[ 726], | 70.00th=[ 1133], 80.00th=[ 1133], 90.00th=[ 1133], 95.00th=[ 1133], | 99.00th=[ 1133], 99.50th=[ 1133], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 8192, max= 8192, per=99.10%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=3.28%, ctx=22, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10476: Fri Sep 20 15:46:19 2024 read: IOPS=0, BW=2411KiB/s (2469kB/s)(4096KiB/1699msec) clat (nsec): min=96643k, max=96643k, avg=96642819.00, stdev= 0.00 lat (nsec): min=96644k, max=96644k, avg=96643927.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[96994], 5.00th=[96994], 10.00th=[96994], 20.00th=[96994], | 30.00th=[96994], 40.00th=[96994], 50.00th=[96994], 60.00th=[96994], | 70.00th=[96994], 80.00th=[96994], 90.00th=[96994], 95.00th=[96994], | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 8192, max= 8192, per=99.10%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4822KiB/s (4937kB/s)(8192KiB/1699msec) clat (msec): min=304, max=1289, avg=796.84, stdev=696.76 lat (msec): min=304, max=1289, avg=797.03, stdev=696.83 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 305], 10.00th=[ 305], 20.00th=[ 305], | 30.00th=[ 305], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 1284], | 70.00th=[ 1284], 80.00th=[ 1284], 90.00th=[ 1284], 95.00th=[ 1284], | 99.00th=[ 1284], 99.50th=[ 1284], 99.90th=[ 1284], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 8192, max= 8192, per=51.58%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.06%, sys=5.12%, ctx=231, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10478: Fri Sep 20 15:46:19 2024 write: IOPS=1, BW=5956KiB/s (6099kB/s)(12.0MiB/2063msec) clat (msec): min=172, max=1051, avg=683.88, stdev=456.91 lat (msec): min=172, max=1052, avg=684.11, stdev=456.90 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 827], 50.00th=[ 827], 60.00th=[ 827], | 70.00th=[ 1053], 80.00th=[ 1053], 90.00th=[ 1053], 95.00th=[ 1053], | 99.00th=[ 1053], 99.50th=[ 1053], 99.90th=[ 1053], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 8192, max= 8192, per=51.58%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=5.19%, ctx=13, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10480: Fri Sep 20 15:46:19 2024 write: IOPS=1, BW=6896KiB/s (7061kB/s)(12.0MiB/1782msec) clat (msec): min=117, max=1501, avg=585.19, stdev=793.29 lat (msec): min=117, max=1501, avg=585.48, stdev=793.25 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 117], 10.00th=[ 117], 20.00th=[ 117], | 30.00th=[ 117], 40.00th=[ 138], 50.00th=[ 138], 60.00th=[ 138], | 70.00th=[ 1502], 80.00th=[ 1502], 90.00th=[ 1502], 95.00th=[ 1502], | 99.00th=[ 1502], 99.50th=[ 1502], 99.90th=[ 1502], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 8192, max= 8192, per=51.58%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=6.68%, ctx=39, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8266KiB/s (8465kB/s), 2411KiB/s-6200KiB/s (2469kB/s-6349kB/s), io=16.0MiB (16.8MB), run=1699-1982msec WRITE: bw=15.5MiB/s (16.3MB/s), 4822KiB/s-6896KiB/s (4937kB/s-7061kB/s), io=32.0MiB (33.6MB), run=1699-2063msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10475: Fri Sep 20 15:46:19 2024 read: IOPS=1, BW=6276KiB/s (6426kB/s)(12.0MiB/1958msec) clat (msec): min=81, max=1099, avg=648.37, stdev=518.96 lat (msec): min=81, max=1099, avg=648.37, stdev=518.96 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 82], 10.00th=[ 82], 20.00th=[ 82], | 30.00th=[ 82], 40.00th=[ 768], 50.00th=[ 768], 60.00th=[ 768], | 70.00th=[ 1099], 80.00th=[ 1099], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 8192, max= 8192, per=97.91%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 100=33.33%, 1000=33.33% cpu : usr=0.00%, sys=50.84%, ctx=875, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10477: Fri Sep 20 15:46:19 2024 read: IOPS=0, BW=2747KiB/s (2813kB/s)(4096KiB/1491msec) clat (nsec): min=54265k, max=54265k, avg=54265396.00, stdev= 0.00 lat (nsec): min=54266k, max=54266k, avg=54266234.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[54264], 5.00th=[54264], 10.00th=[54264], 20.00th=[54264], | 30.00th=[54264], 40.00th=[54264], 50.00th=[54264], 60.00th=[54264], | 70.00th=[54264], 80.00th=[54264], 90.00th=[54264], 95.00th=[54264], | 99.00th=[54264], 99.50th=[54264], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 8192, max= 8192, per=97.91%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5494KiB/s (5626kB/s)(8192KiB/1491msec) clat (msec): min=220, max=1195, avg=708.01, stdev=689.79 lat (msec): min=220, max=1195, avg=708.22, stdev=689.81 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 222], 10.00th=[ 222], 20.00th=[ 222], | 30.00th=[ 222], 40.00th=[ 222], 50.00th=[ 222], 60.00th=[ 1200], | 70.00th=[ 1200], 80.00th=[ 1200], 90.00th=[ 1200], 95.00th=[ 1200], | 99.00th=[ 1200], 99.50th=[ 1200], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 8192, max= 8192, per=54.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=8.72%, ctx=49, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10479: Fri Sep 20 15:46:19 2024 write: IOPS=1, BW=5673KiB/s (5809kB/s)(12.0MiB/2166msec) clat (msec): min=256, max=1154, avg=716.57, stdev=449.68 lat (msec): min=256, max=1155, avg=716.86, stdev=449.66 clat percentiles (msec): | 1.00th=[ 257], 5.00th=[ 257], 10.00th=[ 257], 20.00th=[ 257], | 30.00th=[ 257], 40.00th=[ 743], 50.00th=[ 743], 60.00th=[ 743], | 70.00th=[ 1150], 80.00th=[ 1150], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 8031, max= 8192, per=53.62%, avg=8111.50, stdev=113.84, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.05%, sys=11.36%, ctx=36, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10481: Fri Sep 20 15:46:19 2024 write: IOPS=2, BW=8672KiB/s (8880kB/s)(12.0MiB/1417msec) clat (msec): min=67, max=1123, avg=463.42, stdev=575.67 lat (msec): min=67, max=1124, avg=463.62, stdev=575.70 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 68], | 30.00th=[ 68], 40.00th=[ 199], 50.00th=[ 199], 60.00th=[ 199], | 70.00th=[ 1116], 80.00th=[ 1116], 90.00th=[ 1116], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 8192, max= 8192, per=54.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=18.57%, ctx=33, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8368KiB/s (8569kB/s), 2747KiB/s-6276KiB/s (2813kB/s-6426kB/s), io=16.0MiB (16.8MB), run=1491-1958msec WRITE: bw=14.8MiB/s (15.5MB/s), 5494KiB/s-8672KiB/s (5626kB/s-8880kB/s), io=32.0MiB (33.6MB), run=1417-2166msec