== sanity test 398b: DIO and buffer IO race ============== 04:05:57 (1734685557) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.16494 s, 43.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6723: Fri Dec 20 04:06:18 2024 read: IOPS=82, BW=328KiB/s (336kB/s)(5992KiB/18241msec) clat (usec): min=46, max=221395, avg=5416.92, stdev=7274.25 lat (usec): min=47, max=221397, avg=5418.37, stdev=7274.14 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 81], 10.00th=[ 95], 20.00th=[ 184], | 30.00th=[ 3130], 40.00th=[ 3687], 50.00th=[ 4424], 60.00th=[ 5145], | 70.00th=[ 6521], 80.00th=[ 8356], 90.00th=[ 11207], 95.00th=[ 13435], | 99.00th=[ 22152], 99.50th=[ 23725], 99.90th=[ 50594], 99.95th=[221250], | 99.99th=[221250] bw ( KiB/s): min= 184, max= 537, per=24.52%, avg=322.72, stdev=76.76, samples=36 iops : min= 46, max= 134, avg=80.53, stdev=19.15, samples=36 write: IOPS=86, BW=345KiB/s (353kB/s)(6296KiB/18241msec) clat (usec): min=429, max=135296, avg=6371.98, stdev=7245.25 lat (usec): min=429, max=135297, avg=6374.05, stdev=7244.85 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 660], 10.00th=[ 709], 20.00th=[ 799], | 30.00th=[ 1090], 40.00th=[ 2638], 50.00th=[ 4817], 60.00th=[ 6390], | 70.00th=[ 8029], 80.00th=[ 10421], 90.00th=[ 14877], 95.00th=[ 18744], | 99.00th=[ 28705], 99.50th=[ 34341], 99.90th=[ 53740], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 239, max= 560, per=24.59%, avg=338.89, stdev=79.37, samples=36 iops : min= 59, max= 140, avg=84.56, stdev=19.82, samples=36 lat (usec) : 50=0.13%, 100=5.60%, 250=4.23%, 500=0.49%, 750=6.97% lat (usec) : 1000=7.52% lat (msec) : 2=4.62%, 4=14.78%, 10=37.73%, 20=15.27%, 50=2.54% lat (msec) : 100=0.07%, 250=0.07% cpu : usr=0.16%, sys=21.88%, ctx=5656, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6724: Fri Dec 20 04:06:18 2024 read: IOPS=82, BW=330KiB/s (338kB/s)(5940KiB/17975msec) clat (usec): min=42, max=58108, avg=5205.16, stdev=5234.99 lat (usec): min=42, max=58108, avg=5206.47, stdev=5235.08 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 80], 10.00th=[ 88], 20.00th=[ 125], | 30.00th=[ 3032], 40.00th=[ 3589], 50.00th=[ 4228], 60.00th=[ 4948], | 70.00th=[ 6063], 80.00th=[ 8225], 90.00th=[11338], 95.00th=[13304], | 99.00th=[19792], 99.50th=[30016], 99.90th=[57410], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 215, max= 496, per=24.80%, avg=326.37, stdev=73.91, samples=35 iops : min= 53, max= 124, avg=81.40, stdev=18.46, samples=35 write: IOPS=88, BW=353KiB/s (362kB/s)(6348KiB/17975msec) clat (usec): min=427, max=217104, avg=6395.46, stdev=8415.37 lat (usec): min=428, max=217104, avg=6397.10, stdev=8415.21 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 652], 10.00th=[ 709], 20.00th=[ 791], | 30.00th=[ 1057], 40.00th=[ 2606], 50.00th=[ 4621], 60.00th=[ 6128], | 70.00th=[ 7963], 80.00th=[ 10683], 90.00th=[ 14615], 95.00th=[ 18744], | 99.00th=[ 27919], 99.50th=[ 35390], 99.90th=[ 53740], 99.95th=[217056], | 99.99th=[217056] bw ( KiB/s): min= 232, max= 592, per=24.93%, avg=343.51, stdev=73.69, samples=35 iops : min= 58, max= 148, avg=85.69, stdev=18.50, samples=35 lat (usec) : 50=0.23%, 100=6.77%, 250=3.71%, 500=0.72%, 750=7.49% lat (usec) : 1000=7.10% lat (msec) : 2=4.85%, 4=15.23%, 10=35.97%, 20=15.40%, 50=2.34% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.21%, sys=21.80%, ctx=5594, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6725: Fri Dec 20 04:06:18 2024 read: IOPS=83, BW=336KiB/s (344kB/s)(6028KiB/17954msec) clat (usec): min=40, max=35334, avg=5138.99, stdev=4385.33 lat (usec): min=40, max=35334, avg=5140.99, stdev=4385.45 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 78], 10.00th=[ 89], 20.00th=[ 124], | 30.00th=[ 3032], 40.00th=[ 3687], 50.00th=[ 4359], 60.00th=[ 5276], | 70.00th=[ 6587], 80.00th=[ 8291], 90.00th=[10945], 95.00th=[13304], | 99.00th=[18220], 99.50th=[20579], 99.90th=[34866], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 160, max= 504, per=25.10%, avg=330.31, stdev=79.55, samples=35 iops : min= 40, max= 126, avg=82.43, stdev=19.88, samples=35 write: IOPS=87, BW=349KiB/s (357kB/s)(6260KiB/17954msec) clat (usec): min=426, max=240810, avg=6462.43, stdev=9344.67 lat (usec): min=427, max=240813, avg=6463.69, stdev=9344.60 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 660], 10.00th=[ 717], 20.00th=[ 816], | 30.00th=[ 1139], 40.00th=[ 2474], 50.00th=[ 4621], 60.00th=[ 6194], | 70.00th=[ 7963], 80.00th=[ 10290], 90.00th=[ 14877], 95.00th=[ 19006], | 99.00th=[ 29492], 99.50th=[ 36439], 99.90th=[114820], 99.95th=[240124], | 99.99th=[240124] bw ( KiB/s): min= 200, max= 552, per=24.73%, avg=340.83, stdev=68.75, samples=35 iops : min= 50, max= 138, avg=85.06, stdev=17.16, samples=35 lat (usec) : 50=0.13%, 100=6.58%, 250=4.36%, 500=0.42%, 750=7.00% lat (usec) : 1000=6.61% lat (msec) : 2=5.27%, 4=15.10%, 10=37.34%, 20=14.78%, 50=2.25% lat (msec) : 100=0.10%, 250=0.07% cpu : usr=0.18%, sys=22.25%, ctx=5563, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6726: Fri Dec 20 04:06:18 2024 read: IOPS=83, BW=335KiB/s (343kB/s)(6052KiB/18051msec) clat (usec): min=43, max=78985, avg=5102.47, stdev=5279.07 lat (usec): min=44, max=78985, avg=5103.54, stdev=5278.94 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 80], 10.00th=[ 88], 20.00th=[ 117], | 30.00th=[ 2966], 40.00th=[ 3654], 50.00th=[ 4293], 60.00th=[ 5080], | 70.00th=[ 6325], 80.00th=[ 8291], 90.00th=[10814], 95.00th=[12911], | 99.00th=[17695], 99.50th=[20841], 99.90th=[78119], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 184, max= 512, per=25.30%, avg=333.00, stdev=78.08, samples=36 iops : min= 46, max= 128, avg=83.17, stdev=19.53, samples=36 write: IOPS=86, BW=345KiB/s (354kB/s)(6236KiB/18051msec) clat (usec): min=457, max=219339, avg=6566.68, stdev=8590.78 lat (usec): min=458, max=219340, avg=6568.32, stdev=8590.77 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 676], 10.00th=[ 725], 20.00th=[ 816], | 30.00th=[ 1139], 40.00th=[ 2900], 50.00th=[ 5080], 60.00th=[ 6521], | 70.00th=[ 8291], 80.00th=[ 10552], 90.00th=[ 14877], 95.00th=[ 18744], | 99.00th=[ 30802], 99.50th=[ 34866], 99.90th=[ 73925], 99.95th=[219153], | 99.99th=[219153] bw ( KiB/s): min= 232, max= 512, per=24.94%, avg=343.67, stdev=67.06, samples=36 iops : min= 58, max= 128, avg=85.83, stdev=16.79, samples=36 lat (usec) : 50=0.10%, 100=6.84%, 250=4.75%, 500=0.42%, 750=6.97% lat (usec) : 1000=7.19% lat (msec) : 2=4.82%, 4=13.57%, 10=38.02%, 20=15.01%, 50=2.12% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.20%, sys=21.77%, ctx=5658, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1316KiB/s (1348kB/s), 328KiB/s-336KiB/s (336kB/s-344kB/s), io=23.4MiB (24.6MB), run=17954-18241msec WRITE: bw=1378KiB/s (1411kB/s), 345KiB/s-353KiB/s (353kB/s-362kB/s), io=24.6MiB (25.7MB), run=17954-18241msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6727: Fri Dec 20 04:06:54 2024 read: IOPS=27, BW=110KiB/s (113kB/s)(5992KiB/54284msec) clat (usec): min=1640, max=242474, avg=6048.52, stdev=7589.99 lat (usec): min=1640, max=242476, avg=6050.01, stdev=7590.01 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 11], 95.00th=[ 15], | 99.00th=[ 22], 99.50th=[ 30], 99.90th=[ 65], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min= 15, max= 264, per=25.04%, avg=110.41, stdev=52.51, samples=108 iops : min= 3, max= 66, avg=27.46, stdev=13.18, samples=108 write: IOPS=28, BW=116KiB/s (119kB/s)(6296KiB/54284msec) clat (msec): min=10, max=115, avg=28.67, stdev=12.49 lat (msec): min=10, max=115, avg=28.67, stdev=12.49 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 17], 20.00th=[ 19], | 30.00th=[ 22], 40.00th=[ 24], 50.00th=[ 26], 60.00th=[ 28], | 70.00th=[ 32], 80.00th=[ 38], 90.00th=[ 46], 95.00th=[ 53], | 99.00th=[ 72], 99.50th=[ 80], 99.90th=[ 114], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 39, max= 208, per=25.11%, avg=115.77, stdev=39.38, samples=108 iops : min= 9, max= 52, avg=28.80, stdev= 9.91, samples=108 lat (msec) : 2=0.33%, 4=17.84%, 10=24.97%, 20=17.58%, 50=36.10% lat (msec) : 100=3.06%, 250=0.13% cpu : usr=0.13%, sys=4.54%, ctx=3990, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6728: Fri Dec 20 04:06:54 2024 read: IOPS=27, BW=109KiB/s (112kB/s)(5940KiB/54418msec) clat (usec): min=1539, max=219741, avg=6057.63, stdev=7342.44 lat (usec): min=1539, max=219742, avg=6058.88, stdev=7342.42 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2245], 10.00th=[ 2474], 20.00th=[ 2835], | 30.00th=[ 3490], 40.00th=[ 4228], 50.00th=[ 4817], 60.00th=[ 5407], | 70.00th=[ 6128], 80.00th=[ 7308], 90.00th=[ 10421], 95.00th=[ 14615], | 99.00th=[ 22938], 99.50th=[ 30802], 99.90th=[ 71828], 99.95th=[219153], | 99.99th=[219153] bw ( KiB/s): min= 8, max= 271, per=24.79%, avg=109.34, stdev=53.58, samples=108 iops : min= 2, max= 67, avg=27.21, stdev=13.38, samples=108 write: IOPS=29, BW=117KiB/s (119kB/s)(6348KiB/54418msec) clat (msec): min=11, max=150, avg=28.56, stdev=13.03 lat (msec): min=11, max=150, avg=28.57, stdev=13.03 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 17], 20.00th=[ 19], | 30.00th=[ 21], 40.00th=[ 23], 50.00th=[ 26], 60.00th=[ 28], | 70.00th=[ 32], 80.00th=[ 37], 90.00th=[ 47], 95.00th=[ 55], | 99.00th=[ 75], 99.50th=[ 78], 99.90th=[ 118], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 47, max= 200, per=25.24%, avg=116.34, stdev=40.02, samples=108 iops : min= 11, max= 50, avg=28.96, stdev=10.03, samples=108 lat (msec) : 2=0.55%, 4=17.29%, 10=25.10%, 20=18.10%, 50=35.12% lat (msec) : 100=3.74%, 250=0.10% cpu : usr=0.12%, sys=4.52%, ctx=4002, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6729: Fri Dec 20 04:06:54 2024 read: IOPS=28, BW=113KiB/s (115kB/s)(6028KiB/53550msec) clat (usec): min=1743, max=64241, avg=6011.65, stdev=4789.25 lat (usec): min=1743, max=64241, avg=6013.14, stdev=4789.12 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2343], 10.00th=[ 2507], 20.00th=[ 2966], | 30.00th=[ 3589], 40.00th=[ 4359], 50.00th=[ 4883], 60.00th=[ 5473], | 70.00th=[ 6128], 80.00th=[ 7439], 90.00th=[10814], 95.00th=[14353], | 99.00th=[23987], 99.50th=[29754], 99.90th=[58983], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 23, max= 304, per=25.56%, avg=112.72, stdev=54.02, samples=106 iops : min= 5, max= 76, avg=28.09, stdev=13.53, samples=106 write: IOPS=29, BW=117KiB/s (120kB/s)(6260KiB/53550msec) clat (msec): min=10, max=238, avg=28.37, stdev=13.65 lat (msec): min=10, max=238, avg=28.37, stdev=13.65 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 19], | 30.00th=[ 21], 40.00th=[ 23], 50.00th=[ 26], 60.00th=[ 29], | 70.00th=[ 32], 80.00th=[ 37], 90.00th=[ 45], 95.00th=[ 53], | 99.00th=[ 70], 99.50th=[ 79], 99.90th=[ 150], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 40, max= 216, per=25.37%, avg=116.94, stdev=41.41, samples=106 iops : min= 10, max= 54, avg=29.15, stdev=10.38, samples=106 lat (msec) : 2=0.36%, 4=16.96%, 10=25.94%, 20=19.17%, 50=34.24% lat (msec) : 100=3.22%, 250=0.10% cpu : usr=0.13%, sys=4.59%, ctx=4047, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6730: Fri Dec 20 04:06:54 2024 read: IOPS=28, BW=114KiB/s (116kB/s)(6052KiB/53299msec) clat (usec): min=1553, max=52586, avg=5875.01, stdev=4270.91 lat (usec): min=1554, max=52586, avg=5876.19, stdev=4270.92 clat percentiles (usec): | 1.00th=[ 2057], 5.00th=[ 2343], 10.00th=[ 2507], 20.00th=[ 2900], | 30.00th=[ 3523], 40.00th=[ 4293], 50.00th=[ 4883], 60.00th=[ 5473], | 70.00th=[ 6194], 80.00th=[ 7177], 90.00th=[10159], 95.00th=[14353], | 99.00th=[22676], 99.50th=[26084], 99.90th=[47973], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 8, max= 272, per=25.79%, avg=113.74, stdev=54.13, samples=106 iops : min= 2, max= 68, avg=28.38, stdev=13.59, samples=106 write: IOPS=29, BW=117KiB/s (120kB/s)(6236KiB/53299msec) clat (msec): min=11, max=244, avg=28.42, stdev=13.79 lat (msec): min=11, max=244, avg=28.43, stdev=13.79 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 17], 20.00th=[ 19], | 30.00th=[ 21], 40.00th=[ 23], 50.00th=[ 26], 60.00th=[ 28], | 70.00th=[ 31], 80.00th=[ 37], 90.00th=[ 45], 95.00th=[ 54], | 99.00th=[ 72], 99.50th=[ 79], 99.90th=[ 150], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 55, max= 216, per=25.36%, avg=116.91, stdev=39.66, samples=106 iops : min= 13, max= 54, avg=29.17, stdev= 9.96, samples=106 lat (msec) : 2=0.39%, 4=17.02%, 10=26.60%, 20=18.26%, 50=34.15% lat (msec) : 100=3.48%, 250=0.10% cpu : usr=0.14%, sys=4.59%, ctx=4026, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=441KiB/s (452kB/s), 109KiB/s-114KiB/s (112kB/s-116kB/s), io=23.4MiB (24.6MB), run=53299-54418msec WRITE: bw=462KiB/s (473kB/s), 116KiB/s-117KiB/s (119kB/s-120kB/s), io=24.6MiB (25.7MB), run=53299-54418msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6788: Fri Dec 20 04:07:02 2024 read: IOPS=57, BW=916KiB/s (938kB/s)(5920KiB/6462msec) clat (usec): min=51, max=163308, avg=12894.74, stdev=11869.32 lat (usec): min=51, max=163309, avg=12896.17, stdev=11869.69 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 84], 10.00th=[ 89], 20.00th=[ 110], | 30.00th=[ 10683], 40.00th=[ 12911], 50.00th=[ 14353], 60.00th=[ 15664], | 70.00th=[ 16909], 80.00th=[ 19268], 90.00th=[ 21365], 95.00th=[ 23725], | 99.00th=[ 42730], 99.50th=[ 58459], 99.90th=[162530], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 544, max= 1184, per=24.11%, avg=866.67, stdev=175.51, samples=12 iops : min= 34, max= 74, avg=54.17, stdev=10.97, samples=12 write: IOPS=61, BW=985KiB/s (1009kB/s)(6368KiB/6462msec) clat (usec): min=691, max=41444, avg=4199.33, stdev=4881.94 lat (usec): min=694, max=41445, avg=4201.25, stdev=4881.85 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 922], 10.00th=[ 988], 20.00th=[ 1074], | 30.00th=[ 1123], 40.00th=[ 1237], 50.00th=[ 1680], 60.00th=[ 3687], | 70.00th=[ 4948], 80.00th=[ 6980], 90.00th=[10290], 95.00th=[11994], | 99.00th=[25297], 99.50th=[32113], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 576, max= 1408, per=24.97%, avg=944.00, stdev=319.85, samples=12 iops : min= 36, max= 88, avg=59.00, stdev=19.99, samples=12 lat (usec) : 100=7.55%, 250=5.21%, 750=0.65%, 1000=5.08% lat (msec) : 2=21.48%, 4=5.73%, 10=14.45%, 20=31.64%, 50=7.94% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.26%, sys=21.30%, ctx=1888, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6790: Fri Dec 20 04:07:02 2024 read: IOPS=55, BW=886KiB/s (907kB/s)(5744KiB/6482msec) clat (usec): min=50, max=39392, avg=12672.99, stdev=8239.73 lat (usec): min=50, max=39393, avg=12673.96, stdev=8239.73 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 83], 10.00th=[ 91], 20.00th=[ 119], | 30.00th=[11600], 40.00th=[13698], 50.00th=[14877], 60.00th=[16319], | 70.00th=[17171], 80.00th=[18744], 90.00th=[21890], 95.00th=[23462], | 99.00th=[26608], 99.50th=[38011], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 128, max= 1088, per=23.88%, avg=858.17, stdev=265.59, samples=12 iops : min= 8, max= 68, avg=53.42, stdev=16.61, samples=12 write: IOPS=63, BW=1010KiB/s (1034kB/s)(6544KiB/6482msec) clat (usec): min=643, max=239914, avg=4586.52, stdev=12767.05 lat (usec): min=645, max=239916, avg=4588.29, stdev=12767.03 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 906], 10.00th=[ 955], 20.00th=[ 1045], | 30.00th=[ 1123], 40.00th=[ 1221], 50.00th=[ 1516], 60.00th=[ 3359], | 70.00th=[ 4686], 80.00th=[ 6456], 90.00th=[ 9241], 95.00th=[ 11994], | 99.00th=[ 30016], 99.50th=[ 34866], 99.90th=[240124], 99.95th=[240124], | 99.99th=[240124] bw ( KiB/s): min= 352, max= 1600, per=25.66%, avg=970.08, stdev=337.06, samples=12 iops : min= 22, max= 100, avg=60.42, stdev=21.10, samples=12 lat (usec) : 100=7.55%, 250=4.17%, 750=0.39%, 1000=6.77% lat (msec) : 2=23.57%, 4=4.30%, 10=15.23%, 20=30.08%, 50=7.81% lat (msec) : 250=0.13% cpu : usr=0.17%, sys=19.09%, ctx=1863, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6792: Fri Dec 20 04:07:02 2024 read: IOPS=58, BW=939KiB/s (961kB/s)(6256KiB/6664msec) clat (usec): min=52, max=48039, avg=12385.39, stdev=8248.16 lat (usec): min=52, max=48041, avg=12386.56, stdev=8248.41 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 82], 10.00th=[ 89], 20.00th=[ 113], | 30.00th=[11469], 40.00th=[13173], 50.00th=[14222], 60.00th=[15664], | 70.00th=[16712], 80.00th=[18482], 90.00th=[21103], 95.00th=[23462], | 99.00th=[29492], 99.50th=[40633], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 384, max= 1440, per=25.95%, avg=932.77, stdev=260.35, samples=13 iops : min= 24, max= 90, avg=58.23, stdev=16.32, samples=13 write: IOPS=56, BW=905KiB/s (927kB/s)(6032KiB/6664msec) clat (usec): min=721, max=196176, avg=4663.70, stdev=11142.71 lat (usec): min=721, max=196179, avg=4666.93, stdev=11142.39 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 898], 10.00th=[ 963], 20.00th=[ 1045], | 30.00th=[ 1139], 40.00th=[ 1254], 50.00th=[ 1549], 60.00th=[ 3589], | 70.00th=[ 4948], 80.00th=[ 6718], 90.00th=[ 9110], 95.00th=[ 13042], | 99.00th=[ 26346], 99.50th=[ 41157], 99.90th=[196084], 99.95th=[196084], | 99.99th=[196084] bw ( KiB/s): min= 448, max= 1632, per=23.24%, avg=878.62, stdev=325.28, samples=13 iops : min= 28, max= 102, avg=54.85, stdev=20.37, samples=13 lat (usec) : 100=7.94%, 250=5.08%, 750=0.26%, 1000=6.64% lat (msec) : 2=19.92%, 4=4.56%, 10=14.71%, 20=32.94%, 50=7.81% lat (msec) : 250=0.13% cpu : usr=0.20%, sys=18.67%, ctx=1991, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6794: Fri Dec 20 04:07:02 2024 read: IOPS=57, BW=914KiB/s (936kB/s)(6032KiB/6599msec) clat (usec): min=56, max=56202, avg=12630.29, stdev=8279.07 lat (usec): min=57, max=56203, avg=12631.21, stdev=8279.10 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 85], 10.00th=[ 94], 20.00th=[ 114], | 30.00th=[11338], 40.00th=[13304], 50.00th=[14484], 60.00th=[15795], | 70.00th=[17171], 80.00th=[18482], 90.00th=[21365], 95.00th=[24511], | 99.00th=[27657], 99.50th=[28705], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 448, max= 1280, per=24.85%, avg=893.08, stdev=207.70, samples=13 iops : min= 28, max= 80, avg=55.62, stdev=13.04, samples=13 write: IOPS=59, BW=948KiB/s (971kB/s)(6256KiB/6599msec) clat (usec): min=732, max=162539, avg=4519.93, stdev=9575.42 lat (usec): min=733, max=162543, avg=4521.46, stdev=9575.56 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 938], 10.00th=[ 988], 20.00th=[ 1057], | 30.00th=[ 1123], 40.00th=[ 1237], 50.00th=[ 1467], 60.00th=[ 3195], | 70.00th=[ 4359], 80.00th=[ 6718], 90.00th=[ 9896], 95.00th=[ 14091], | 99.00th=[ 32375], 99.50th=[ 42730], 99.90th=[162530], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 512, max= 1440, per=24.79%, avg=937.31, stdev=295.24, samples=13 iops : min= 32, max= 90, avg=58.38, stdev=18.43, samples=13 lat (usec) : 100=6.64%, 250=5.34%, 500=0.13%, 750=0.26%, 1000=5.86% lat (msec) : 2=22.14%, 4=5.34%, 10=13.41%, 20=32.68%, 50=7.94% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.14%, sys=19.08%, ctx=1889, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3594KiB/s (3680kB/s), 886KiB/s-939KiB/s (907kB/s-961kB/s), io=23.4MiB (24.5MB), run=6462-6664msec WRITE: bw=3782KiB/s (3872kB/s), 905KiB/s-1010KiB/s (927kB/s-1034kB/s), io=24.6MiB (25.8MB), run=6462-6664msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6787: Fri Dec 20 04:07:10 2024 read: IOPS=24, BW=399KiB/s (409kB/s)(5920KiB/14830msec) clat (msec): min=2, max=176, avg= 6.43, stdev=10.38 lat (msec): min=2, max=176, avg= 6.43, stdev=10.38 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 5], | 70.00th=[ 5], 80.00th=[ 8], 90.00th=[ 12], 95.00th=[ 19], | 99.00th=[ 33], 99.50th=[ 39], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 191, max= 672, per=24.53%, avg=387.14, stdev=136.46, samples=29 iops : min= 11, max= 42, avg=24.03, stdev= 8.60, samples=29 write: IOPS=26, BW=429KiB/s (440kB/s)(6368KiB/14830msec) clat (usec): min=14897, max=78629, avg=31218.51, stdev=11516.38 lat (usec): min=14900, max=78631, avg=31221.00, stdev=11516.45 clat percentiles (usec): | 1.00th=[15401], 5.00th=[16909], 10.00th=[18744], 20.00th=[21103], | 30.00th=[23987], 40.00th=[26346], 50.00th=[29230], 60.00th=[31851], | 70.00th=[34341], 80.00th=[39060], 90.00th=[47973], 95.00th=[55837], | 99.00th=[66847], 99.50th=[69731], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 191, max= 608, per=25.44%, avg=422.38, stdev=119.83, samples=29 iops : min= 11, max= 38, avg=26.24, stdev= 7.55, samples=29 lat (msec) : 4=26.04%, 10=15.89%, 20=11.07%, 50=42.45%, 100=4.43% lat (msec) : 250=0.13% cpu : usr=0.08%, sys=4.77%, ctx=902, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6789: Fri Dec 20 04:07:10 2024 read: IOPS=23, BW=379KiB/s (388kB/s)(5744KiB/15173msec) clat (usec): min=2215, max=62333, avg=5584.77, stdev=5383.31 lat (usec): min=2216, max=62334, avg=5585.98, stdev=5383.23 clat percentiles (usec): | 1.00th=[ 2311], 5.00th=[ 2606], 10.00th=[ 2802], 20.00th=[ 3064], | 30.00th=[ 3261], 40.00th=[ 3490], 50.00th=[ 3752], 60.00th=[ 4113], | 70.00th=[ 4621], 80.00th=[ 5932], 90.00th=[11731], 95.00th=[17433], | 99.00th=[24511], 99.50th=[27919], 99.90th=[62129], 99.95th=[62129], | 99.99th=[62129] bw ( KiB/s): min= 63, max= 768, per=23.77%, avg=375.17, stdev=172.04, samples=30 iops : min= 3, max= 48, avg=23.27, stdev=10.73, samples=30 write: IOPS=26, BW=431KiB/s (442kB/s)(6544KiB/15173msec) clat (msec): min=11, max=255, avg=32.10, stdev=16.35 lat (msec): min=11, max=255, avg=32.11, stdev=16.35 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 17], 10.00th=[ 19], 20.00th=[ 21], | 30.00th=[ 24], 40.00th=[ 27], 50.00th=[ 31], 60.00th=[ 34], | 70.00th=[ 37], 80.00th=[ 40], 90.00th=[ 48], 95.00th=[ 56], | 99.00th=[ 70], 99.50th=[ 79], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 159, max= 702, per=25.56%, avg=424.30, stdev=121.41, samples=30 iops : min= 9, max= 43, avg=26.33, stdev= 7.62, samples=30 lat (msec) : 4=26.43%, 10=14.71%, 20=13.15%, 50=41.28%, 100=4.30% lat (msec) : 500=0.13% cpu : usr=0.15%, sys=4.63%, ctx=886, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6791: Fri Dec 20 04:07:10 2024 read: IOPS=26, BW=426KiB/s (436kB/s)(6256KiB/14688msec) clat (usec): min=2222, max=70476, avg=5757.33, stdev=5674.90 lat (usec): min=2223, max=70477, avg=5760.16, stdev=5674.53 clat percentiles (usec): | 1.00th=[ 2474], 5.00th=[ 2769], 10.00th=[ 2900], 20.00th=[ 3130], | 30.00th=[ 3392], 40.00th=[ 3621], 50.00th=[ 3949], 60.00th=[ 4293], | 70.00th=[ 4752], 80.00th=[ 6783], 90.00th=[10421], 95.00th=[16319], | 99.00th=[29230], 99.50th=[40633], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 160, max= 928, per=27.19%, avg=429.07, stdev=203.13, samples=29 iops : min= 10, max= 58, avg=26.72, stdev=12.67, samples=29 write: IOPS=25, BW=411KiB/s (421kB/s)(6032KiB/14688msec) clat (msec): min=15, max=208, avg=32.86, stdev=14.68 lat (msec): min=15, max=208, avg=32.86, stdev=14.68 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 18], 10.00th=[ 20], 20.00th=[ 23], | 30.00th=[ 26], 40.00th=[ 28], 50.00th=[ 31], 60.00th=[ 34], | 70.00th=[ 37], 80.00th=[ 42], 90.00th=[ 49], 95.00th=[ 55], | 99.00th=[ 67], 99.50th=[ 83], 99.90th=[ 209], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 192, max= 608, per=24.45%, avg=405.90, stdev=113.36, samples=29 iops : min= 12, max= 38, avg=25.28, stdev= 7.05, samples=29 lat (msec) : 4=26.17%, 10=19.27%, 20=10.03%, 50=40.36%, 100=4.04% lat (msec) : 250=0.13% cpu : usr=0.08%, sys=4.90%, ctx=884, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6793: Fri Dec 20 04:07:10 2024 read: IOPS=25, BW=404KiB/s (414kB/s)(6032KiB/14924msec) clat (usec): min=2245, max=34972, avg=5812.36, stdev=5030.95 lat (usec): min=2246, max=34973, avg=5813.49, stdev=5030.89 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2671], 10.00th=[ 2835], 20.00th=[ 3097], | 30.00th=[ 3326], 40.00th=[ 3589], 50.00th=[ 3916], 60.00th=[ 4228], | 70.00th=[ 4883], 80.00th=[ 6915], 90.00th=[12387], 95.00th=[17433], | 99.00th=[27395], 99.50th=[27657], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 127, max= 640, per=24.95%, avg=393.69, stdev=150.55, samples=29 iops : min= 7, max= 40, avg=24.45, stdev= 9.48, samples=29 write: IOPS=26, BW=419KiB/s (429kB/s)(6256KiB/14924msec) clat (msec): min=12, max=200, avg=32.37, stdev=14.66 lat (msec): min=12, max=200, avg=32.37, stdev=14.66 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 17], 10.00th=[ 19], 20.00th=[ 23], | 30.00th=[ 26], 40.00th=[ 28], 50.00th=[ 30], 60.00th=[ 33], | 70.00th=[ 36], 80.00th=[ 41], 90.00th=[ 48], 95.00th=[ 56], | 99.00th=[ 71], 99.50th=[ 97], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 159, max= 640, per=24.78%, avg=411.34, stdev=121.41, samples=29 iops : min= 9, max= 40, avg=25.55, stdev= 7.62, samples=29 lat (msec) : 4=25.78%, 10=17.32%, 20=11.07%, 50=41.80%, 100=3.91% lat (msec) : 250=0.13% cpu : usr=0.21%, sys=4.78%, ctx=909, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1579KiB/s (1616kB/s), 379KiB/s-426KiB/s (388kB/s-436kB/s), io=23.4MiB (24.5MB), run=14688-15173msec WRITE: bw=1661KiB/s (1701kB/s), 411KiB/s-431KiB/s (421kB/s-442kB/s), io=24.6MiB (25.8MB), run=14688-15173msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6812: Fri Dec 20 04:07:14 2024 read: IOPS=2, BW=2388KiB/s (2445kB/s)(7168KiB/3002msec) clat (msec): min=5, max=685, avg=376.51, stdev=280.85 lat (msec): min=5, max=685, avg=376.51, stdev=280.85 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 92], | 30.00th=[ 182], 40.00th=[ 182], 50.00th=[ 439], 60.00th=[ 609], | 70.00th=[ 609], 80.00th=[ 625], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 1744, max= 4087, per=43.84%, avg=2981.50, stdev=1259.66, samples=4 iops : min= 1, max= 3, avg= 2.25, stdev= 0.96, samples=4 write: IOPS=1, BW=1706KiB/s (1746kB/s)(5120KiB/3002msec) clat (msec): min=26, max=146, avg=70.92, stdev=48.00 lat (msec): min=26, max=146, avg=71.01, stdev=48.02 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 73], 60.00th=[ 73], | 70.00th=[ 78], 80.00th=[ 78], 90.00th=[ 146], 95.00th=[ 146], | 99.00th=[ 146], 99.50th=[ 146], 99.90th=[ 146], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 1744, max= 6131, per=34.74%, avg=3307.67, stdev=2449.80, samples=3 iops : min= 1, max= 5, avg= 2.67, stdev= 2.08, samples=3 lat (msec) : 10=8.33%, 50=16.67%, 100=25.00%, 250=16.67%, 500=8.33% lat (msec) : 750=25.00% cpu : usr=0.07%, sys=25.96%, ctx=1060, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6813: Fri Dec 20 04:07:14 2024 read: IOPS=0, BW=881KiB/s (902kB/s)(2048KiB/2324msec) clat (msec): min=87, max=691, avg=389.77, stdev=427.42 lat (msec): min=87, max=691, avg=389.77, stdev=427.42 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 88], 10.00th=[ 88], 20.00th=[ 88], | 30.00th=[ 88], 40.00th=[ 88], 50.00th=[ 88], 60.00th=[ 693], | 70.00th=[ 693], 80.00th=[ 693], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 2048, max= 2048, per=30.11%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4406KiB/s (4512kB/s)(10.0MiB/2324msec) clat (msec): min=24, max=603, avg=153.11, stdev=194.82 lat (msec): min=24, max=603, avg=153.20, stdev=194.82 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 28], | 30.00th=[ 29], 40.00th=[ 30], 50.00th=[ 39], 60.00th=[ 54], | 70.00th=[ 279], 80.00th=[ 279], 90.00th=[ 342], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 8192, max=10240, per=96.79%, avg=9216.00, stdev=1448.15, samples=2 iops : min= 8, max= 10, avg= 9.00, stdev= 1.41, samples=2 lat (msec) : 50=41.67%, 100=16.67%, 250=8.33%, 500=16.67%, 750=16.67% cpu : usr=0.04%, sys=18.47%, ctx=370, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6814: Fri Dec 20 04:07:14 2024 read: IOPS=1, BW=2041KiB/s (2089kB/s)(6144KiB/3011msec) clat (usec): min=527, max=607131, avg=206163.28, stdev=302356.48 lat (usec): min=529, max=607132, avg=206164.63, stdev=302356.36 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 529], 10.00th=[ 529], 20.00th=[ 562], | 30.00th=[ 562], 40.00th=[ 6063], 50.00th=[ 6063], 60.00th=[ 38011], | 70.00th=[583009], 80.00th=[583009], 90.00th=[608175], 95.00th=[608175], | 99.00th=[608175], 99.50th=[608175], 99.90th=[608175], 99.95th=[608175], | 99.99th=[608175] bw ( KiB/s): min= 4096, max= 6144, per=75.28%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=1, BW=2041KiB/s (2089kB/s)(6144KiB/3011msec) clat (msec): min=27, max=935, avg=286.43, stdev=338.85 lat (msec): min=27, max=935, avg=286.48, stdev=338.85 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 30], | 30.00th=[ 30], 40.00th=[ 148], 50.00th=[ 148], 60.00th=[ 275], | 70.00th=[ 305], 80.00th=[ 305], 90.00th=[ 936], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 2043, max= 4096, per=32.25%, avg=3070.75, stdev=1183.86, samples=4 iops : min= 1, max= 4, avg= 2.75, stdev= 1.50, samples=4 lat (usec) : 750=16.67% lat (msec) : 10=8.33%, 50=25.00%, 250=8.33%, 500=16.67%, 750=16.67% lat (msec) : 1000=8.33% cpu : usr=0.03%, sys=10.90%, ctx=559, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6815: Fri Dec 20 04:07:14 2024 read: IOPS=1, BW=1700KiB/s (1741kB/s)(5120KiB/3011msec) clat (msec): min=5, max=653, avg=278.24, stdev=308.49 lat (msec): min=5, max=653, avg=278.24, stdev=308.49 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 6], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 112], 60.00th=[ 112], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 651], 95.00th=[ 651], | 99.00th=[ 651], 99.50th=[ 651], 99.90th=[ 651], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 2048, max= 6144, per=60.23%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=2, BW=2381KiB/s (2438kB/s)(7168KiB/3011msec) clat (msec): min=20, max=1094, avg=225.59, stdev=393.22 lat (msec): min=20, max=1094, avg=225.66, stdev=393.21 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 27], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 51], 60.00th=[ 90], | 70.00th=[ 90], 80.00th=[ 271], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 2048, max= 6144, per=37.62%, avg=3581.75, stdev=1960.03, samples=4 iops : min= 2, max= 6, avg= 3.25, stdev= 1.89, samples=4 lat (msec) : 10=8.33%, 50=33.33%, 100=16.67%, 250=8.33%, 500=8.33% lat (msec) : 750=16.67% cpu : usr=0.03%, sys=12.46%, ctx=650, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6802KiB/s (6965kB/s), 881KiB/s-2388KiB/s (902kB/s-2445kB/s), io=20.0MiB (20.0MB), run=2324-3011msec WRITE: bw=9522KiB/s (9751kB/s), 1706KiB/s-4406KiB/s (1746kB/s-4512kB/s), io=28.0MiB (29.4MB), run=2324-3011msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6816: Fri Dec 20 04:07:14 2024 read: IOPS=2, BW=2314KiB/s (2370kB/s)(7168KiB/3097msec) clat (msec): min=15, max=684, avg=270.78, stdev=276.41 lat (msec): min=15, max=684, avg=270.78, stdev=276.41 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 20], | 30.00th=[ 42], 40.00th=[ 42], 50.00th=[ 180], 60.00th=[ 401], | 70.00th=[ 401], 80.00th=[ 558], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 2048, max= 4096, per=53.07%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 write: IOPS=1, BW=1653KiB/s (1693kB/s)(5120KiB/3097msec) clat (msec): min=49, max=834, avg=225.73, stdev=341.53 lat (msec): min=49, max=834, avg=225.80, stdev=341.53 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 50], | 30.00th=[ 61], 40.00th=[ 61], 50.00th=[ 67], 60.00th=[ 67], | 70.00th=[ 118], 80.00th=[ 118], 90.00th=[ 835], 95.00th=[ 835], | 99.00th=[ 835], 99.50th=[ 835], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 2048, max= 4096, per=30.32%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 lat (msec) : 20=16.67%, 50=16.67%, 100=16.67%, 250=16.67%, 500=8.33% lat (msec) : 750=16.67%, 1000=8.33% cpu : usr=0.03%, sys=4.23%, ctx=49, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6817: Fri Dec 20 04:07:14 2024 read: IOPS=0, BW=671KiB/s (688kB/s)(2048KiB/3050msec) clat (msec): min=255, max=734, avg=495.25, stdev=338.99 lat (msec): min=255, max=734, avg=495.25, stdev=338.99 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 255], 10.00th=[ 255], 20.00th=[ 255], | 30.00th=[ 255], 40.00th=[ 255], 50.00th=[ 255], 60.00th=[ 735], | 70.00th=[ 735], 80.00th=[ 735], 90.00th=[ 735], 95.00th=[ 735], | 99.00th=[ 735], 99.50th=[ 735], 99.90th=[ 735], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 2035, max= 2048, per=31.74%, avg=2041.50, stdev= 9.19, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3357KiB/s (3438kB/s)(10.0MiB/3050msec) clat (msec): min=28, max=672, avg=200.94, stdev=244.28 lat (msec): min=29, max=672, avg=201.02, stdev=244.28 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 44], | 30.00th=[ 50], 40.00th=[ 51], 50.00th=[ 77], 60.00th=[ 104], | 70.00th=[ 247], 80.00th=[ 247], 90.00th=[ 625], 95.00th=[ 676], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 4096, max= 6144, per=51.17%, avg=4608.00, stdev=1024.00, samples=4 iops : min= 4, max= 6, avg= 4.50, stdev= 1.00, samples=4 lat (msec) : 50=25.00%, 100=16.67%, 250=25.00%, 500=8.33%, 750=25.00% cpu : usr=0.03%, sys=4.43%, ctx=281, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6818: Fri Dec 20 04:07:14 2024 read: IOPS=1, BW=1987KiB/s (2035kB/s)(6144KiB/3092msec) clat (msec): min=15, max=682, avg=177.72, stdev=267.10 lat (msec): min=15, max=682, avg=177.72, stdev=267.10 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 17], 40.00th=[ 27], 50.00th=[ 27], 60.00th=[ 49], | 70.00th=[ 279], 80.00th=[ 279], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 2027, max= 4096, per=39.72%, avg=2554.75, stdev=1027.55, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 write: IOPS=1, BW=1987KiB/s (2035kB/s)(6144KiB/3092msec) clat (msec): min=30, max=937, avg=336.47, stdev=353.09 lat (msec): min=30, max=937, avg=336.57, stdev=353.11 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 60], | 30.00th=[ 60], 40.00th=[ 125], 50.00th=[ 125], 60.00th=[ 313], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 936], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 2027, max= 4096, per=28.37%, avg=2554.75, stdev=1027.55, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 lat (msec) : 20=16.67%, 50=25.00%, 100=8.33%, 250=8.33%, 500=16.67% lat (msec) : 750=16.67%, 1000=8.33% cpu : usr=0.06%, sys=2.88%, ctx=37, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6819: Fri Dec 20 04:07:14 2024 read: IOPS=1, BW=1608KiB/s (1647kB/s)(5120KiB/3184msec) clat (msec): min=17, max=707, avg=302.40, stdev=335.27 lat (msec): min=17, max=707, avg=302.40, stdev=335.27 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 148], 60.00th=[ 148], | 70.00th=[ 617], 80.00th=[ 617], 90.00th=[ 709], 95.00th=[ 709], | 99.00th=[ 709], 99.50th=[ 709], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 6144, max= 6144, per=95.52%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2251KiB/s (2305kB/s)(7168KiB/3184msec) clat (msec): min=31, max=931, avg=233.46, stdev=329.15 lat (msec): min=31, max=931, avg=233.54, stdev=329.15 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 35], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 279], | 70.00th=[ 279], 80.00th=[ 288], 90.00th=[ 936], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 2048, max= 4079, per=30.26%, avg=2725.00, stdev=1172.60, samples=3 iops : min= 2, max= 3, avg= 2.33, stdev= 0.58, samples=3 lat (msec) : 20=16.67%, 50=33.33%, 250=8.33%, 500=16.67%, 750=16.67% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=3.02%, ctx=44, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6432KiB/s (6587kB/s), 671KiB/s-2314KiB/s (688kB/s-2370kB/s), io=20.0MiB (20.0MB), run=3050-3184msec WRITE: bw=9005KiB/s (9221kB/s), 1653KiB/s-3357KiB/s (1693kB/s-3438kB/s), io=28.0MiB (29.4MB), run=3050-3184msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6828: Fri Dec 20 04:07:20 2024 read: IOPS=0, BW=3487KiB/s (3571kB/s)(12.0MiB/3524msec) clat (msec): min=177, max=2259, avg=1162.77, stdev=1045.79 lat (msec): min=177, max=2259, avg=1162.77, stdev=1045.79 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 178], 10.00th=[ 178], 20.00th=[ 178], | 30.00th=[ 178], 40.00th=[ 1053], 50.00th=[ 1053], 60.00th=[ 1053], | 70.00th=[ 2265], 80.00th=[ 2265], 90.00th=[ 2265], 95.00th=[ 2265], | 99.00th=[ 2265], 99.50th=[ 2265], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.09%, sys=42.32%, ctx=1006, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6830: Fri Dec 20 04:07:20 2024 read: IOPS=0, BW=1013KiB/s (1037kB/s)(4096KiB/4044msec) clat (nsec): min=165613k, max=165613k, avg=165613321.00, stdev= 0.00 lat (nsec): min=165616k, max=165616k, avg=165615974.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 165], 10.00th=[ 165], 20.00th=[ 165], | 30.00th=[ 165], 40.00th=[ 165], 50.00th=[ 165], 60.00th=[ 165], | 70.00th=[ 165], 80.00th=[ 165], 90.00th=[ 165], 95.00th=[ 165], | 99.00th=[ 165], 99.50th=[ 165], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2026KiB/s (2074kB/s)(8192KiB/4044msec) clat (msec): min=402, max=3434, avg=1918.55, stdev=2143.37 lat (msec): min=403, max=3434, avg=1918.82, stdev=2143.35 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 405], 10.00th=[ 405], 20.00th=[ 405], | 30.00th=[ 405], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 3440], | 70.00th=[ 3440], 80.00th=[ 3440], 90.00th=[ 3440], 95.00th=[ 3440], | 99.00th=[ 3440], 99.50th=[ 3440], 99.90th=[ 3440], 99.95th=[ 3440], | 99.99th=[ 3440] lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.05%, sys=6.28%, ctx=121, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6833: Fri Dec 20 04:07:20 2024 write: IOPS=0, BW=2895KiB/s (2965kB/s)(12.0MiB/4244msec) clat (msec): min=352, max=2380, avg=1408.66, stdev=1016.56 lat (msec): min=353, max=2380, avg=1408.95, stdev=1016.52 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 355], 10.00th=[ 355], 20.00th=[ 355], | 30.00th=[ 355], 40.00th=[ 1485], 50.00th=[ 1485], 60.00th=[ 1485], | 70.00th=[ 2366], 80.00th=[ 2366], 90.00th=[ 2366], 95.00th=[ 2366], | 99.00th=[ 2366], 99.50th=[ 2366], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33% cpu : usr=0.02%, sys=7.24%, ctx=27, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6834: Fri Dec 20 04:07:20 2024 write: IOPS=0, BW=3379KiB/s (3460kB/s)(12.0MiB/3637msec) clat (msec): min=263, max=2416, avg=1207.15, stdev=1101.09 lat (msec): min=263, max=2417, avg=1207.51, stdev=1101.07 clat percentiles (msec): | 1.00th=[ 264], 5.00th=[ 264], 10.00th=[ 264], 20.00th=[ 264], | 30.00th=[ 264], 40.00th=[ 944], 50.00th=[ 944], 60.00th=[ 944], | 70.00th=[ 2433], 80.00th=[ 2433], 90.00th=[ 2433], 95.00th=[ 2433], | 99.00th=[ 2433], 99.50th=[ 2433], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.03%, sys=8.33%, ctx=54, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4051KiB/s (4149kB/s), 1013KiB/s-3487KiB/s (1037kB/s-3571kB/s), io=16.0MiB (16.8MB), run=3524-4044msec WRITE: bw=7721KiB/s (7906kB/s), 2026KiB/s-3379KiB/s (2074kB/s-3460kB/s), io=32.0MiB (33.6MB), run=3637-4244msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6829: Fri Dec 20 04:07:20 2024 read: IOPS=0, BW=3088KiB/s (3162kB/s)(12.0MiB/3979msec) clat (msec): min=249, max=2240, avg=1322.90, stdev=1004.65 lat (msec): min=249, max=2240, avg=1322.91, stdev=1004.65 clat percentiles (msec): | 1.00th=[ 251], 5.00th=[ 251], 10.00th=[ 251], 20.00th=[ 251], | 30.00th=[ 251], 40.00th=[ 1485], 50.00th=[ 1485], 60.00th=[ 1485], | 70.00th=[ 2232], 80.00th=[ 2232], 90.00th=[ 2232], 95.00th=[ 2232], | 99.00th=[ 2232], 99.50th=[ 2232], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 5490, max= 5490, per=100.00%, avg=5490.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.08%, sys=1.63%, ctx=33, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6831: Fri Dec 20 04:07:20 2024 read: IOPS=0, BW=898KiB/s (919kB/s)(4096KiB/4563msec) clat (nsec): min=163909k, max=163909k, avg=163909077.00, stdev= 0.00 lat (nsec): min=163912k, max=163912k, avg=163911775.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 165], 10.00th=[ 165], 20.00th=[ 165], | 30.00th=[ 165], 40.00th=[ 165], 50.00th=[ 165], 60.00th=[ 165], | 70.00th=[ 165], 80.00th=[ 165], 90.00th=[ 165], 95.00th=[ 165], | 99.00th=[ 165], 99.50th=[ 165], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1795KiB/s (1838kB/s)(8192KiB/4563msec) clat (msec): min=406, max=3977, avg=2191.74, stdev=2525.10 lat (msec): min=406, max=3977, avg=2192.02, stdev=2525.09 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 405], 10.00th=[ 405], 20.00th=[ 405], | 30.00th=[ 405], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 3977], | 70.00th=[ 3977], 80.00th=[ 3977], 90.00th=[ 3977], 95.00th=[ 3977], | 99.00th=[ 3977], 99.50th=[ 3977], 99.90th=[ 3977], 99.95th=[ 3977], | 99.99th=[ 3977] bw ( KiB/s): min= 7728, max= 7728, per=100.00%, avg=7728.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.09%, sys=2.26%, ctx=94, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6832: Fri Dec 20 04:07:20 2024 write: IOPS=1, BW=7262KiB/s (7437kB/s)(12.0MiB/1692msec) clat (msec): min=61, max=1534, avg=556.19, stdev=847.63 lat (msec): min=61, max=1535, avg=556.59, stdev=847.53 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 62], 10.00th=[ 62], 20.00th=[ 62], | 30.00th=[ 62], 40.00th=[ 72], 50.00th=[ 72], 60.00th=[ 72], | 70.00th=[ 1536], 80.00th=[ 1536], 90.00th=[ 1536], 95.00th=[ 1536], | 99.00th=[ 1536], 99.50th=[ 1536], 99.90th=[ 1536], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=66.67% cpu : usr=0.12%, sys=5.68%, ctx=41, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6835: Fri Dec 20 04:07:20 2024 write: IOPS=6, BW=25.5MiB/s (26.8MB/s)(12.0MiB/470msec) clat (msec): min=60, max=264, avg=152.99, stdev=103.21 lat (msec): min=61, max=264, avg=153.33, stdev=102.99 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 61], 10.00th=[ 61], 20.00th=[ 61], | 30.00th=[ 61], 40.00th=[ 134], 50.00th=[ 134], 60.00th=[ 134], | 70.00th=[ 266], 80.00th=[ 266], 90.00th=[ 266], 95.00th=[ 266], | 99.00th=[ 266], 99.50th=[ 266], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.21%, sys=19.62%, ctx=33, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3591KiB/s (3677kB/s), 898KiB/s-3088KiB/s (919kB/s-3162kB/s), io=16.0MiB (16.8MB), run=3979-4563msec WRITE: bw=7181KiB/s (7354kB/s), 1795KiB/s-25.5MiB/s (1838kB/s-26.8MB/s), io=32.0MiB (33.6MB), run=470-4563msec