== sanity test 398b: DIO and buffer IO race ============== 10:24:44 (1724768684) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.04855 s, 48.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23136: Tue Aug 27 10:25:00 2024 read: IOPS=108, BW=435KiB/s (445kB/s)(5992KiB/13783msec) clat (usec): min=47, max=184680, avg=4274.79, stdev=5750.17 lat (usec): min=47, max=184681, avg=4275.84, stdev=5750.09 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 72], 10.00th=[ 90], 20.00th=[ 2245], | 30.00th=[ 2704], 40.00th=[ 3097], 50.00th=[ 3523], 60.00th=[ 4146], | 70.00th=[ 5080], 80.00th=[ 6259], 90.00th=[ 7898], 95.00th=[ 9503], | 99.00th=[ 14746], 99.50th=[ 21890], 99.90th=[ 42730], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 232, max= 552, per=24.87%, avg=427.70, stdev=73.72, samples=27 iops : min= 58, max= 138, avg=106.81, stdev=18.45, samples=27 write: IOPS=114, BW=457KiB/s (468kB/s)(6296KiB/13783msec) clat (usec): min=428, max=59856, avg=4626.86, stdev=4426.31 lat (usec): min=428, max=59856, avg=4628.45, stdev=4425.86 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 537], 10.00th=[ 578], 20.00th=[ 750], | 30.00th=[ 1037], 40.00th=[ 2769], 50.00th=[ 3884], 60.00th=[ 4948], | 70.00th=[ 5997], 80.00th=[ 7504], 90.00th=[10159], 95.00th=[12518], | 99.00th=[17695], 99.50th=[24249], 99.90th=[38536], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 256, max= 640, per=25.00%, avg=450.19, stdev=90.47, samples=27 iops : min= 64, max= 160, avg=112.44, stdev=22.68, samples=27 lat (usec) : 50=0.26%, 100=5.50%, 250=2.05%, 500=0.72%, 750=9.57% lat (usec) : 1000=5.05% lat (msec) : 2=4.10%, 4=27.28%, 10=38.22%, 20=6.61%, 50=0.59% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.22%, sys=23.31%, ctx=6175, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23137: Tue Aug 27 10:25:00 2024 read: IOPS=108, BW=433KiB/s (444kB/s)(5940KiB/13710msec) clat (usec): min=48, max=37126, avg=4100.52, stdev=3265.34 lat (usec): min=48, max=37126, avg=4101.31, stdev=3265.39 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 63], 10.00th=[ 91], 20.00th=[ 2212], | 30.00th=[ 2638], 40.00th=[ 3032], 50.00th=[ 3589], 60.00th=[ 4146], | 70.00th=[ 4948], 80.00th=[ 6194], 90.00th=[ 7963], 95.00th=[ 9372], | 99.00th=[13435], 99.50th=[17171], 99.90th=[30802], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 240, max= 608, per=25.08%, avg=431.33, stdev=89.41, samples=27 iops : min= 60, max= 152, avg=107.78, stdev=22.39, samples=27 write: IOPS=115, BW=463KiB/s (474kB/s)(6348KiB/13710msec) clat (usec): min=412, max=165566, avg=4743.61, stdev=6024.62 lat (usec): min=413, max=165567, avg=4744.58, stdev=6024.56 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 545], 10.00th=[ 594], 20.00th=[ 791], | 30.00th=[ 1106], 40.00th=[ 2638], 50.00th=[ 3785], 60.00th=[ 4752], | 70.00th=[ 6063], 80.00th=[ 7504], 90.00th=[ 10159], 95.00th=[ 12518], | 99.00th=[ 19530], 99.50th=[ 23462], 99.90th=[ 54264], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 288, max= 624, per=25.38%, avg=457.11, stdev=89.64, samples=27 iops : min= 72, max= 156, avg=114.22, stdev=22.42, samples=27 lat (usec) : 50=0.13%, 100=5.83%, 250=1.89%, 500=0.62%, 750=9.21% lat (usec) : 1000=4.62% lat (msec) : 2=5.21%, 4=27.31%, 10=37.73%, 20=6.80%, 50=0.59% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.13%, sys=23.74%, ctx=6180, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23138: Tue Aug 27 10:25:00 2024 read: IOPS=108, BW=432KiB/s (442kB/s)(6028KiB/13953msec) clat (usec): min=47, max=43855, avg=4201.54, stdev=3341.07 lat (usec): min=47, max=43856, avg=4202.71, stdev=3341.33 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 70], 10.00th=[ 90], 20.00th=[ 2245], | 30.00th=[ 2638], 40.00th=[ 3064], 50.00th=[ 3523], 60.00th=[ 4228], | 70.00th=[ 5211], 80.00th=[ 6325], 90.00th=[ 8160], 95.00th=[ 9765], | 99.00th=[14484], 99.50th=[17171], 99.90th=[31851], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 184, max= 600, per=24.53%, avg=421.93, stdev=90.50, samples=27 iops : min= 46, max= 150, avg=105.48, stdev=22.62, samples=27 write: IOPS=112, BW=449KiB/s (459kB/s)(6260KiB/13953msec) clat (usec): min=437, max=171101, avg=4808.69, stdev=6357.54 lat (usec): min=438, max=171103, avg=4809.58, stdev=6357.51 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 545], 10.00th=[ 603], 20.00th=[ 799], | 30.00th=[ 1045], 40.00th=[ 2507], 50.00th=[ 3752], 60.00th=[ 4686], | 70.00th=[ 5866], 80.00th=[ 7701], 90.00th=[ 10552], 95.00th=[ 13042], | 99.00th=[ 20579], 99.50th=[ 27657], 99.90th=[ 62653], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 248, max= 600, per=24.25%, avg=436.74, stdev=89.81, samples=27 iops : min= 62, max= 150, avg=109.19, stdev=22.45, samples=27 lat (usec) : 50=0.20%, 100=5.50%, 250=1.86%, 500=0.62%, 750=8.40% lat (usec) : 1000=5.99% lat (msec) : 2=4.85%, 4=27.38%, 10=37.34%, 20=7.19%, 50=0.62% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.18%, sys=23.37%, ctx=6247, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23139: Tue Aug 27 10:25:00 2024 read: IOPS=110, BW=442KiB/s (453kB/s)(6052KiB/13684msec) clat (usec): min=46, max=36309, avg=4143.02, stdev=3060.45 lat (usec): min=47, max=36309, avg=4143.74, stdev=3060.49 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 71], 10.00th=[ 99], 20.00th=[ 2245], | 30.00th=[ 2704], 40.00th=[ 3097], 50.00th=[ 3490], 60.00th=[ 4146], | 70.00th=[ 5080], 80.00th=[ 6259], 90.00th=[ 7832], 95.00th=[ 9372], | 99.00th=[13173], 99.50th=[18220], 99.90th=[22414], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 88, max= 576, per=25.61%, avg=440.48, stdev=99.64, samples=27 iops : min= 22, max= 144, avg=109.96, stdev=24.88, samples=27 write: IOPS=113, BW=456KiB/s (467kB/s)(6236KiB/13684msec) clat (usec): min=467, max=170516, avg=4695.63, stdev=6309.95 lat (usec): min=467, max=170517, avg=4697.04, stdev=6309.62 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 545], 10.00th=[ 586], 20.00th=[ 758], | 30.00th=[ 1037], 40.00th=[ 2540], 50.00th=[ 3687], 60.00th=[ 4555], | 70.00th=[ 5669], 80.00th=[ 7439], 90.00th=[ 10290], 95.00th=[ 12518], | 99.00th=[ 20317], 99.50th=[ 28967], 99.90th=[ 57934], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 104, max= 640, per=25.30%, avg=455.63, stdev=100.80, samples=27 iops : min= 26, max= 160, avg=113.74, stdev=25.17, samples=27 lat (usec) : 50=0.23%, 100=4.82%, 250=1.89%, 500=0.59%, 750=9.47% lat (usec) : 1000=4.98% lat (msec) : 2=4.43%, 4=29.33%, 10=37.30%, 20=6.18%, 50=0.72% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.12%, sys=23.87%, ctx=6066, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1721KiB/s (1762kB/s), 432KiB/s-442KiB/s (442kB/s-453kB/s), io=23.4MiB (24.6MB), run=13684-13953msec WRITE: bw=1802KiB/s (1845kB/s), 449KiB/s-463KiB/s (459kB/s-474kB/s), io=24.6MiB (25.7MB), run=13684-13953msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23141: Tue Aug 27 10:25:22 2024 read: IOPS=41, BW=167KiB/s (171kB/s)(5992KiB/35821msec) clat (usec): min=1680, max=212574, avg=3839.22, stdev=6238.24 lat (usec): min=1681, max=212575, avg=3840.11, stdev=6238.25 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1893], 10.00th=[ 1975], 20.00th=[ 2089], | 30.00th=[ 2180], 40.00th=[ 2278], 50.00th=[ 2442], 60.00th=[ 2606], | 70.00th=[ 3064], 80.00th=[ 4752], 90.00th=[ 7570], 95.00th=[ 10028], | 99.00th=[ 16909], 99.50th=[ 18482], 99.90th=[ 43779], 99.95th=[212861], | 99.99th=[212861] bw ( KiB/s): min= 56, max= 304, per=24.90%, avg=166.86, stdev=64.42, samples=71 iops : min= 14, max= 76, avg=41.62, stdev=16.16, samples=71 write: IOPS=43, BW=176KiB/s (180kB/s)(6296KiB/35821msec) clat (usec): min=8810, max=65455, avg=19063.99, stdev=7171.69 lat (usec): min=8811, max=65456, avg=19065.01, stdev=7171.74 clat percentiles (usec): | 1.00th=[ 9765], 5.00th=[11076], 10.00th=[11863], 20.00th=[13304], | 30.00th=[14615], 40.00th=[16057], 50.00th=[17695], 60.00th=[19006], | 70.00th=[20841], 80.00th=[23200], 90.00th=[28705], 95.00th=[32637], | 99.00th=[44827], 99.50th=[49021], 99.90th=[56886], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 64, max= 240, per=24.96%, avg=174.97, stdev=51.44, samples=71 iops : min= 16, max= 60, avg=43.65, stdev=12.92, samples=71 lat (msec) : 2=5.99%, 4=31.32%, 10=9.86%, 20=35.32%, 50=17.25% lat (msec) : 100=0.23%, 250=0.03% cpu : usr=0.10%, sys=5.69%, ctx=3549, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23142: Tue Aug 27 10:25:22 2024 read: IOPS=41, BW=166KiB/s (170kB/s)(5940KiB/35749msec) clat (usec): min=1567, max=165084, avg=3750.58, stdev=5081.46 lat (usec): min=1568, max=165086, avg=3751.57, stdev=5081.46 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1893], 10.00th=[ 1975], 20.00th=[ 2089], | 30.00th=[ 2212], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2638], | 70.00th=[ 3032], 80.00th=[ 4752], 90.00th=[ 7242], 95.00th=[ 9503], | 99.00th=[ 15139], 99.50th=[ 18220], 99.90th=[ 35914], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 16, max= 304, per=24.79%, avg=166.11, stdev=70.13, samples=71 iops : min= 4, max= 76, avg=41.46, stdev=17.59, samples=71 write: IOPS=44, BW=178KiB/s (182kB/s)(6348KiB/35749msec) clat (usec): min=8505, max=58146, avg=18974.94, stdev=7253.95 lat (usec): min=8506, max=58146, avg=18975.87, stdev=7253.95 clat percentiles (usec): | 1.00th=[10028], 5.00th=[11076], 10.00th=[11863], 20.00th=[13173], | 30.00th=[14353], 40.00th=[15795], 50.00th=[17433], 60.00th=[19006], | 70.00th=[20841], 80.00th=[23200], 90.00th=[28443], 95.00th=[33817], | 99.00th=[44827], 99.50th=[50594], 99.90th=[56886], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 80, max= 248, per=25.24%, avg=176.93, stdev=50.49, samples=71 iops : min= 20, max= 62, avg=44.17, stdev=12.68, samples=71 lat (msec) : 2=5.73%, 4=31.41%, 10=9.57%, 20=35.38%, 50=17.58% lat (msec) : 100=0.29%, 250=0.03% cpu : usr=0.07%, sys=5.72%, ctx=3545, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23143: Tue Aug 27 10:25:22 2024 read: IOPS=42, BW=169KiB/s (173kB/s)(6028KiB/35597msec) clat (usec): min=1708, max=56906, avg=4003.09, stdev=3857.61 lat (usec): min=1709, max=56907, avg=4003.87, stdev=3857.61 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1942], 10.00th=[ 2008], 20.00th=[ 2114], | 30.00th=[ 2245], 40.00th=[ 2343], 50.00th=[ 2474], 60.00th=[ 2737], | 70.00th=[ 3392], 80.00th=[ 5145], 90.00th=[ 8225], 95.00th=[10683], | 99.00th=[17695], 99.50th=[22414], 99.90th=[43779], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 56, max= 312, per=25.30%, avg=169.49, stdev=62.02, samples=71 iops : min= 14, max= 78, avg=42.31, stdev=15.54, samples=71 write: IOPS=43, BW=176KiB/s (180kB/s)(6260KiB/35597msec) clat (msec): min=8, max=170, avg=18.85, stdev= 8.01 lat (msec): min=8, max=170, avg=18.85, stdev= 8.01 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 28], 95.00th=[ 34], | 99.00th=[ 43], 99.50th=[ 46], 99.90th=[ 79], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 80, max= 240, per=25.06%, avg=175.69, stdev=53.15, samples=71 iops : min= 20, max= 60, avg=43.86, stdev=13.35, samples=71 lat (msec) : 2=4.43%, 4=32.26%, 10=10.09%, 20=36.23%, 50=16.83% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.15%, sys=5.76%, ctx=3598, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23144: Tue Aug 27 10:25:22 2024 read: IOPS=42, BW=170KiB/s (174kB/s)(6052KiB/35607msec) clat (usec): min=1681, max=38301, avg=3808.91, stdev=3421.06 lat (usec): min=1682, max=38301, avg=3809.65, stdev=3421.06 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1893], 10.00th=[ 1975], 20.00th=[ 2089], | 30.00th=[ 2180], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2638], | 70.00th=[ 3130], 80.00th=[ 5014], 90.00th=[ 7504], 95.00th=[10290], | 99.00th=[17433], 99.50th=[23987], 99.90th=[36963], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 56, max= 328, per=25.36%, avg=169.90, stdev=68.73, samples=71 iops : min= 14, max= 82, avg=42.38, stdev=17.24, samples=71 write: IOPS=43, BW=175KiB/s (179kB/s)(6236KiB/35607msec) clat (msec): min=8, max=170, avg=19.10, stdev= 8.21 lat (msec): min=8, max=170, avg=19.10, stdev= 8.21 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 21], 80.00th=[ 24], 90.00th=[ 29], 95.00th=[ 34], | 99.00th=[ 44], 99.50th=[ 47], 99.90th=[ 72], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 80, max= 240, per=24.94%, avg=174.86, stdev=52.50, samples=71 iops : min= 20, max= 60, avg=43.62, stdev=13.19, samples=71 lat (msec) : 2=6.12%, 4=31.02%, 10=9.96%, 20=35.81%, 50=16.86% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.10%, sys=5.73%, ctx=3590, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=670KiB/s (686kB/s), 166KiB/s-170KiB/s (170kB/s-174kB/s), io=23.4MiB (24.6MB), run=35597-35821msec WRITE: bw=702KiB/s (719kB/s), 175KiB/s-178KiB/s (179kB/s-182kB/s), io=24.6MiB (25.7MB), run=35597-35821msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23188: Tue Aug 27 10:25:28 2024 read: IOPS=70, BW=1126KiB/s (1153kB/s)(5920KiB/5259msec) clat (usec): min=51, max=36551, avg=10480.73, stdev=5611.27 lat (usec): min=52, max=36552, avg=10481.44, stdev=5611.29 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 67], 10.00th=[ 95], 20.00th=[ 8356], | 30.00th=[ 9503], 40.00th=[10421], 50.00th=[11207], 60.00th=[11994], | 70.00th=[12518], 80.00th=[13698], 90.00th=[15664], 95.00th=[17957], | 99.00th=[27132], 99.50th=[31589], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 926, max= 1472, per=24.85%, avg=1116.60, stdev=177.73, samples=10 iops : min= 57, max= 92, avg=69.70, stdev=11.22, samples=10 write: IOPS=75, BW=1211KiB/s (1240kB/s)(6368KiB/5259msec) clat (usec): min=672, max=29650, avg=3429.61, stdev=3866.80 lat (usec): min=673, max=29652, avg=3430.98, stdev=3866.82 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 783], 10.00th=[ 816], 20.00th=[ 930], | 30.00th=[ 1090], 40.00th=[ 1303], 50.00th=[ 1975], 60.00th=[ 2835], | 70.00th=[ 3818], 80.00th=[ 5342], 90.00th=[ 7242], 95.00th=[10159], | 99.00th=[21365], 99.50th=[25560], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 864, max= 1632, per=25.58%, avg=1209.30, stdev=247.36, samples=10 iops : min= 54, max= 102, avg=75.50, stdev=15.47, samples=10 lat (usec) : 100=5.21%, 250=2.08%, 500=0.13%, 750=1.69%, 1000=10.94% lat (msec) : 2=13.41%, 4=11.33%, 10=22.01%, 20=31.12%, 50=2.08% cpu : usr=0.06%, sys=22.40%, ctx=2249, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23190: Tue Aug 27 10:25:28 2024 read: IOPS=67, BW=1078KiB/s (1104kB/s)(5744KiB/5330msec) clat (usec): min=53, max=28019, avg=9935.76, stdev=5997.74 lat (usec): min=53, max=28020, avg=9936.47, stdev=5997.76 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 65], 10.00th=[ 84], 20.00th=[ 139], | 30.00th=[ 9110], 40.00th=[ 9896], 50.00th=[11076], 60.00th=[11863], | 70.00th=[12911], 80.00th=[14091], 90.00th=[16057], 95.00th=[18482], | 99.00th=[23987], 99.50th=[26346], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 448, max= 1312, per=23.50%, avg=1055.80, stdev=260.46, samples=10 iops : min= 28, max= 82, avg=65.90, stdev=16.31, samples=10 write: IOPS=76, BW=1228KiB/s (1257kB/s)(6544KiB/5330msec) clat (usec): min=661, max=63634, avg=4219.38, stdev=5692.43 lat (usec): min=662, max=63635, avg=4220.56, stdev=5692.43 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 775], 10.00th=[ 824], 20.00th=[ 971], | 30.00th=[ 1139], 40.00th=[ 1450], 50.00th=[ 2442], 60.00th=[ 3294], | 70.00th=[ 4228], 80.00th=[ 5866], 90.00th=[ 8979], 95.00th=[15795], | 99.00th=[24249], 99.50th=[27132], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 704, max= 1632, per=24.91%, avg=1177.40, stdev=274.20, samples=10 iops : min= 44, max= 102, avg=73.50, stdev=17.25, samples=10 lat (usec) : 100=7.81%, 250=1.95%, 500=0.13%, 750=1.43%, 1000=10.29% lat (msec) : 2=12.11%, 4=12.89%, 10=21.09%, 20=29.17%, 50=2.99% lat (msec) : 100=0.13% cpu : usr=0.08%, sys=21.54%, ctx=2159, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23192: Tue Aug 27 10:25:28 2024 read: IOPS=73, BW=1177KiB/s (1205kB/s)(6256KiB/5317msec) clat (usec): min=59, max=28064, avg=10224.61, stdev=5450.57 lat (usec): min=60, max=28065, avg=10225.35, stdev=5450.50 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 76], 10.00th=[ 99], 20.00th=[ 8029], | 30.00th=[ 9372], 40.00th=[10290], 50.00th=[11076], 60.00th=[11863], | 70.00th=[12649], 80.00th=[13698], 90.00th=[15533], 95.00th=[18482], | 99.00th=[24249], 99.50th=[25297], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 800, max= 1408, per=25.98%, avg=1167.40, stdev=184.68, samples=10 iops : min= 50, max= 88, avg=72.80, stdev=11.49, samples=10 write: IOPS=70, BW=1134KiB/s (1162kB/s)(6032KiB/5317msec) clat (usec): min=678, max=64622, avg=3385.54, stdev=4598.17 lat (usec): min=680, max=64623, avg=3387.34, stdev=4598.36 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 766], 10.00th=[ 816], 20.00th=[ 938], | 30.00th=[ 1123], 40.00th=[ 1401], 50.00th=[ 2311], 60.00th=[ 3097], | 70.00th=[ 3654], 80.00th=[ 4948], 90.00th=[ 7046], 95.00th=[ 8586], | 99.00th=[18482], 99.50th=[31065], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 734, max= 1664, per=23.41%, avg=1106.70, stdev=274.78, samples=10 iops : min= 45, max= 104, avg=69.00, stdev=17.29, samples=10 lat (usec) : 100=5.47%, 250=2.86%, 750=1.56%, 1000=9.51% lat (msec) : 2=13.02%, 4=13.02%, 10=20.70%, 20=31.77%, 50=1.95% lat (msec) : 100=0.13% cpu : usr=0.13%, sys=21.76%, ctx=2279, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23194: Tue Aug 27 10:25:28 2024 read: IOPS=71, BW=1148KiB/s (1176kB/s)(6032KiB/5253msec) clat (usec): min=51, max=30424, avg=9844.58, stdev=5776.03 lat (usec): min=52, max=30425, avg=9845.28, stdev=5776.03 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 73], 10.00th=[ 92], 20.00th=[ 6390], | 30.00th=[ 9110], 40.00th=[ 9896], 50.00th=[10683], 60.00th=[11469], | 70.00th=[12518], 80.00th=[13698], 90.00th=[15401], 95.00th=[16909], | 99.00th=[29230], 99.50th=[30016], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 640, max= 1376, per=24.93%, avg=1120.00, stdev=194.36, samples=10 iops : min= 40, max= 86, avg=70.00, stdev=12.15, samples=10 write: IOPS=74, BW=1191KiB/s (1220kB/s)(6256KiB/5253msec) clat (usec): min=644, max=53656, avg=3777.63, stdev=4436.72 lat (usec): min=645, max=53658, avg=3778.88, stdev=4436.73 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 783], 10.00th=[ 832], 20.00th=[ 938], | 30.00th=[ 1205], 40.00th=[ 1483], 50.00th=[ 2638], 60.00th=[ 3326], | 70.00th=[ 4178], 80.00th=[ 6063], 90.00th=[ 7767], 95.00th=[10028], | 99.00th=[18744], 99.50th=[30016], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 768, max= 1760, per=24.98%, avg=1180.80, stdev=274.63, samples=10 iops : min= 48, max= 110, avg=73.80, stdev=17.16, samples=10 lat (usec) : 100=6.38%, 250=2.99%, 500=0.13%, 750=1.56%, 1000=10.29% lat (msec) : 2=11.07%, 4=11.98%, 10=23.70%, 20=30.47%, 50=1.30% lat (msec) : 100=0.13% cpu : usr=0.23%, sys=22.16%, ctx=2313, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4494KiB/s (4602kB/s), 1078KiB/s-1177KiB/s (1104kB/s-1205kB/s), io=23.4MiB (24.5MB), run=5253-5330msec WRITE: bw=4728KiB/s (4841kB/s), 1134KiB/s-1228KiB/s (1162kB/s-1257kB/s), io=24.6MiB (25.8MB), run=5253-5330msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23187: Tue Aug 27 10:25:32 2024 read: IOPS=40, BW=651KiB/s (667kB/s)(5920KiB/9089msec) clat (usec): min=1815, max=23850, avg=4296.68, stdev=3786.21 lat (usec): min=1816, max=23851, avg=4297.59, stdev=3786.36 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2024], 10.00th=[ 2114], 20.00th=[ 2212], | 30.00th=[ 2343], 40.00th=[ 2442], 50.00th=[ 2737], 60.00th=[ 3032], | 70.00th=[ 3621], 80.00th=[ 5407], 90.00th=[ 9110], 95.00th=[12780], | 99.00th=[21890], 99.50th=[23725], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 384, max= 1216, per=23.93%, avg=628.71, stdev=227.10, samples=17 iops : min= 24, max= 76, avg=39.29, stdev=14.19, samples=17 write: IOPS=43, BW=701KiB/s (717kB/s)(6368KiB/9089msec) clat (usec): min=8721, max=49134, avg=18758.59, stdev=7055.74 lat (usec): min=8722, max=49135, avg=18760.13, stdev=7055.72 clat percentiles (usec): | 1.00th=[ 9372], 5.00th=[10552], 10.00th=[11207], 20.00th=[12649], | 30.00th=[14353], 40.00th=[16188], 50.00th=[17433], 60.00th=[19006], | 70.00th=[20317], 80.00th=[23200], 90.00th=[29492], 95.00th=[32637], | 99.00th=[42206], 99.50th=[49021], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 480, max= 1056, per=24.72%, avg=683.29, stdev=214.64, samples=17 iops : min= 30, max= 66, avg=42.71, stdev=13.42, samples=17 lat (msec) : 2=1.95%, 4=33.46%, 10=10.16%, 20=36.85%, 50=17.58% cpu : usr=0.18%, sys=6.27%, ctx=897, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23189: Tue Aug 27 10:25:32 2024 read: IOPS=39, BW=630KiB/s (645kB/s)(5744KiB/9117msec) clat (usec): min=1749, max=30819, avg=4539.95, stdev=4399.32 lat (usec): min=1750, max=30820, avg=4540.70, stdev=4399.32 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2073], 10.00th=[ 2147], 20.00th=[ 2278], | 30.00th=[ 2409], 40.00th=[ 2540], 50.00th=[ 2704], 60.00th=[ 2999], | 70.00th=[ 3884], 80.00th=[ 5800], 90.00th=[ 9503], 95.00th=[13173], | 99.00th=[23725], 99.50th=[23725], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 288, max= 1152, per=23.82%, avg=625.72, stdev=266.27, samples=18 iops : min= 18, max= 72, avg=39.06, stdev=16.71, samples=18 write: IOPS=44, BW=718KiB/s (735kB/s)(6544KiB/9117msec) clat (usec): min=8719, max=66216, avg=18274.85, stdev=6890.01 lat (usec): min=8720, max=66217, avg=18276.30, stdev=6890.11 clat percentiles (usec): | 1.00th=[ 9634], 5.00th=[10290], 10.00th=[11207], 20.00th=[12518], | 30.00th=[13829], 40.00th=[15401], 50.00th=[16712], 60.00th=[18220], | 70.00th=[20841], 80.00th=[23462], 90.00th=[26608], 95.00th=[31065], | 99.00th=[40633], 99.50th=[40633], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 448, max= 1056, per=25.72%, avg=711.00, stdev=211.64, samples=18 iops : min= 28, max= 66, avg=44.39, stdev=13.26, samples=18 lat (msec) : 2=1.30%, 4=32.03%, 10=10.81%, 20=37.11%, 50=18.62% lat (msec) : 100=0.13% cpu : usr=0.18%, sys=6.18%, ctx=912, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23191: Tue Aug 27 10:25:32 2024 read: IOPS=43, BW=697KiB/s (714kB/s)(6256KiB/8976msec) clat (usec): min=1841, max=35482, avg=4190.73, stdev=3862.87 lat (usec): min=1841, max=35483, avg=4191.47, stdev=3862.89 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2040], 10.00th=[ 2114], 20.00th=[ 2245], | 30.00th=[ 2311], 40.00th=[ 2474], 50.00th=[ 2671], 60.00th=[ 2933], | 70.00th=[ 3621], 80.00th=[ 5145], 90.00th=[ 8455], 95.00th=[13042], | 99.00th=[19792], 99.50th=[25560], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 320, max= 1504, per=26.50%, avg=696.24, stdev=328.10, samples=17 iops : min= 20, max= 94, avg=43.41, stdev=20.46, samples=17 write: IOPS=42, BW=672KiB/s (688kB/s)(6032KiB/8976msec) clat (usec): min=9082, max=52882, avg=19337.05, stdev=7361.48 lat (usec): min=9083, max=52884, avg=19338.50, stdev=7361.59 clat percentiles (usec): | 1.00th=[ 9372], 5.00th=[10683], 10.00th=[11338], 20.00th=[12911], | 30.00th=[14746], 40.00th=[16188], 50.00th=[17433], 60.00th=[19006], | 70.00th=[21365], 80.00th=[25560], 90.00th=[30278], 95.00th=[33424], | 99.00th=[39584], 99.50th=[48497], 99.90th=[52691], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 415, max= 960, per=23.63%, avg=653.00, stdev=203.26, samples=17 iops : min= 25, max= 60, avg=40.71, stdev=12.70, samples=17 lat (msec) : 2=1.69%, 4=35.94%, 10=10.42%, 20=34.90%, 50=16.93% lat (msec) : 100=0.13% cpu : usr=0.13%, sys=6.53%, ctx=915, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23193: Tue Aug 27 10:25:32 2024 read: IOPS=41, BW=671KiB/s (688kB/s)(6032KiB/8984msec) clat (usec): min=1843, max=46208, avg=4565.62, stdev=4659.98 lat (usec): min=1843, max=46209, avg=4566.42, stdev=4659.99 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2057], 10.00th=[ 2147], 20.00th=[ 2245], | 30.00th=[ 2343], 40.00th=[ 2507], 50.00th=[ 2769], 60.00th=[ 2999], | 70.00th=[ 3785], 80.00th=[ 5473], 90.00th=[ 9765], 95.00th=[14877], | 99.00th=[23462], 99.50th=[23987], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 416, max= 1184, per=24.57%, avg=645.41, stdev=199.02, samples=17 iops : min= 26, max= 74, avg=40.24, stdev=12.47, samples=17 write: IOPS=43, BW=696KiB/s (713kB/s)(6256KiB/8984msec) clat (usec): min=8600, max=75512, avg=18497.65, stdev=7084.09 lat (usec): min=8602, max=75515, avg=18499.17, stdev=7084.13 clat percentiles (usec): | 1.00th=[ 9896], 5.00th=[10552], 10.00th=[11207], 20.00th=[12518], | 30.00th=[14353], 40.00th=[15926], 50.00th=[16909], 60.00th=[18220], | 70.00th=[20579], 80.00th=[23462], 90.00th=[27395], 95.00th=[31065], | 99.00th=[40633], 99.50th=[52691], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 448, max= 1056, per=24.64%, avg=681.18, stdev=215.42, samples=17 iops : min= 28, max= 66, avg=42.47, stdev=13.44, samples=17 lat (msec) : 2=1.43%, 4=33.72%, 10=10.16%, 20=37.76%, 50=16.67% lat (msec) : 100=0.26% cpu : usr=0.12%, sys=6.48%, ctx=910, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2627KiB/s (2690kB/s), 630KiB/s-697KiB/s (645kB/s-714kB/s), io=23.4MiB (24.5MB), run=8976-9117msec WRITE: bw=2764KiB/s (2830kB/s), 672KiB/s-718KiB/s (688kB/s-735kB/s), io=24.6MiB (25.8MB), run=8976-9117msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23207: Tue Aug 27 10:25:36 2024 read: IOPS=1, BW=2024KiB/s (2072kB/s)(7168KiB/3542msec) clat (msec): min=45, max=689, avg=445.16, stdev=194.02 lat (msec): min=45, max=689, avg=445.16, stdev=194.02 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 456], | 30.00th=[ 460], 40.00th=[ 460], 50.00th=[ 481], 60.00th=[ 485], | 70.00th=[ 485], 80.00th=[ 502], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 1673, max= 2048, per=34.31%, avg=1983.83, stdev=152.30, samples=6 iops : min= 1, max= 2, avg= 1.50, stdev= 0.55, samples=6 write: IOPS=1, BW=1446KiB/s (1480kB/s)(5120KiB/3542msec) clat (msec): min=25, max=175, avg=76.99, stdev=65.10 lat (msec): min=25, max=175, avg=77.06, stdev=65.10 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 26], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 37], 60.00th=[ 37], | 70.00th=[ 113], 80.00th=[ 113], 90.00th=[ 176], 95.00th=[ 176], | 99.00th=[ 176], 99.50th=[ 176], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 2043, max= 4087, per=39.02%, avg=3158.67, stdev=1034.80, samples=3 iops : min= 1, max= 3, avg= 2.33, stdev= 1.15, samples=3 lat (msec) : 50=33.33%, 250=16.67%, 500=41.67%, 750=8.33% cpu : usr=0.00%, sys=19.97%, ctx=1827, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23208: Tue Aug 27 10:25:36 2024 read: IOPS=0, BW=1007KiB/s (1032kB/s)(2048KiB/2033msec) clat (usec): min=419, max=40807, avg=20613.22, stdev=28558.51 lat (usec): min=420, max=40809, avg=20614.57, stdev=28559.37 clat percentiles (usec): | 1.00th=[ 420], 5.00th=[ 420], 10.00th=[ 420], 20.00th=[ 420], | 30.00th=[ 420], 40.00th=[ 420], 50.00th=[ 420], 60.00th=[40633], | 70.00th=[40633], 80.00th=[40633], 90.00th=[40633], 95.00th=[40633], | 99.00th=[40633], 99.50th=[40633], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 2048, max= 2048, per=35.42%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=5037KiB/s (5158kB/s)(10.0MiB/2033msec) clat (msec): min=18, max=1130, avg=198.57, stdev=370.86 lat (msec): min=18, max=1130, avg=198.62, stdev=370.86 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 24], | 30.00th=[ 26], 40.00th=[ 28], 50.00th=[ 34], 60.00th=[ 34], | 70.00th=[ 62], 80.00th=[ 62], 90.00th=[ 584], 95.00th=[ 1133], | 99.00th=[ 1133], 99.50th=[ 1133], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 16, max= 16, avg=16.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 20=8.33%, 50=58.33%, 100=8.33%, 750=8.33% cpu : usr=0.00%, sys=12.40%, ctx=105, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23209: Tue Aug 27 10:25:36 2024 read: IOPS=3, BW=3095KiB/s (3169kB/s)(6144KiB/1985msec) clat (usec): min=429, max=507043, avg=195946.93, stdev=225073.87 lat (usec): min=430, max=507044, avg=195948.09, stdev=225073.74 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 429], 10.00th=[ 429], 20.00th=[ 56361], | 30.00th=[ 56361], 40.00th=[ 64750], 50.00th=[ 64750], 60.00th=[ 87557], | 70.00th=[459277], 80.00th=[459277], 90.00th=[505414], 95.00th=[505414], | 99.00th=[505414], 99.50th=[505414], 99.90th=[505414], 99.95th=[505414], | 99.99th=[505414] bw ( KiB/s): min= 2048, max= 8192, per=88.55%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=3, BW=3095KiB/s (3169kB/s)(6144KiB/1985msec) clat (msec): min=15, max=630, avg=131.51, stdev=244.89 lat (msec): min=15, max=630, avg=131.56, stdev=244.90 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 24], | 30.00th=[ 24], 40.00th=[ 28], 50.00th=[ 28], 60.00th=[ 33], | 70.00th=[ 60], 80.00th=[ 60], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 4096, max= 6144, per=63.26%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (usec) : 500=8.33% lat (msec) : 20=8.33%, 50=25.00%, 100=33.33%, 500=8.33%, 750=16.67% cpu : usr=0.05%, sys=16.48%, ctx=447, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23210: Tue Aug 27 10:25:36 2024 read: IOPS=1, BW=2025KiB/s (2074kB/s)(5120KiB/2528msec) clat (msec): min=67, max=489, avg=325.31, stdev=209.65 lat (msec): min=67, max=489, avg=325.31, stdev=209.65 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 68], | 30.00th=[ 127], 40.00th=[ 127], 50.00th=[ 460], 60.00th=[ 460], | 70.00th=[ 485], 80.00th=[ 485], 90.00th=[ 489], 95.00th=[ 489], | 99.00th=[ 489], 99.50th=[ 489], 99.90th=[ 489], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 2048, max= 4096, per=47.23%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 write: IOPS=2, BW=2835KiB/s (2903kB/s)(7168KiB/2528msec) clat (msec): min=22, max=553, avg=123.54, stdev=192.20 lat (msec): min=23, max=553, avg=123.60, stdev=192.20 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 34], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 45], 60.00th=[ 47], | 70.00th=[ 47], 80.00th=[ 122], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 2048, max= 6144, per=44.28%, avg=3584.00, stdev=1960.81, samples=4 iops : min= 2, max= 6, avg= 3.50, stdev= 1.91, samples=4 lat (msec) : 50=41.67%, 100=8.33%, 250=16.67%, 500=25.00%, 750=8.33% cpu : usr=0.00%, sys=17.69%, ctx=959, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5782KiB/s (5921kB/s), 1007KiB/s-3095KiB/s (1032kB/s-3169kB/s), io=20.0MiB (20.0MB), run=1985-3542msec WRITE: bw=8095KiB/s (8289kB/s), 1446KiB/s-5037KiB/s (1480kB/s-5158kB/s), io=28.0MiB (29.4MB), run=1985-3542msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23211: Tue Aug 27 10:25:36 2024 read: IOPS=1, BW=2032KiB/s (2081kB/s)(7168KiB/3528msec) clat (msec): min=11, max=597, avg=363.94, stdev=244.60 lat (msec): min=11, max=597, avg=363.94, stdev=244.60 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 33], | 30.00th=[ 409], 40.00th=[ 409], 50.00th=[ 426], 60.00th=[ 489], | 70.00th=[ 489], 80.00th=[ 592], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 1689, max= 2048, per=33.21%, avg=1927.50, stdev=184.75, samples=6 iops : min= 1, max= 2, avg= 1.50, stdev= 0.55, samples=6 write: IOPS=1, BW=1451KiB/s (1486kB/s)(5120KiB/3528msec) clat (msec): min=23, max=518, avg=187.03, stdev=202.14 lat (msec): min=23, max=518, avg=187.08, stdev=202.15 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 24], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 159], 60.00th=[ 159], | 70.00th=[ 209], 80.00th=[ 209], 90.00th=[ 518], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 1689, max= 3379, per=29.19%, avg=2372.00, stdev=890.37, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 lat (msec) : 20=8.33%, 50=25.00%, 250=16.67%, 500=25.00%, 750=25.00% cpu : usr=0.03%, sys=2.72%, ctx=67, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23212: Tue Aug 27 10:25:36 2024 read: IOPS=0, BW=670KiB/s (686kB/s)(2048KiB/3056msec) clat (usec): min=23933, max=71619, avg=47776.39, stdev=33718.89 lat (usec): min=23935, max=71619, avg=47777.65, stdev=33718.10 clat percentiles (usec): | 1.00th=[23987], 5.00th=[23987], 10.00th=[23987], 20.00th=[23987], | 30.00th=[23987], 40.00th=[23987], 50.00th=[23987], 60.00th=[71828], | 70.00th=[71828], 80.00th=[71828], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 2048, max= 2048, per=35.29%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3351KiB/s (3431kB/s)(10.0MiB/3056msec) clat (msec): min=26, max=621, avg=294.44, stdev=269.18 lat (msec): min=26, max=623, avg=294.66, stdev=269.39 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 36], | 30.00th=[ 43], 40.00th=[ 51], 50.00th=[ 61], 60.00th=[ 477], | 70.00th=[ 506], 80.00th=[ 506], 90.00th=[ 617], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 2039, max= 8192, per=50.38%, avg=4093.75, stdev=2898.43, samples=4 iops : min= 1, max= 8, avg= 3.75, stdev= 3.10, samples=4 lat (msec) : 50=33.33%, 100=25.00%, 500=8.33%, 750=33.33% cpu : usr=0.03%, sys=3.63%, ctx=209, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23213: Tue Aug 27 10:25:36 2024 read: IOPS=3, BW=3077KiB/s (3150kB/s)(6144KiB/1997msec) clat (msec): min=12, max=471, avg=183.04, stdev=215.14 lat (msec): min=12, max=471, avg=183.04, stdev=215.14 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 19], | 30.00th=[ 19], 40.00th=[ 66], 50.00th=[ 66], 60.00th=[ 86], | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 2043, max= 6144, per=70.53%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 write: IOPS=3, BW=3077KiB/s (3150kB/s)(6144KiB/1997msec) clat (msec): min=22, max=622, avg=142.61, stdev=236.93 lat (msec): min=22, max=622, avg=142.66, stdev=236.94 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 37], | 70.00th=[ 109], 80.00th=[ 109], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 2043, max= 4096, per=41.98%, avg=3411.67, stdev=1185.30, samples=3 iops : min= 1, max= 4, avg= 3.00, stdev= 1.73, samples=3 lat (msec) : 20=16.67%, 50=33.33%, 100=16.67%, 250=8.33%, 500=16.67% lat (msec) : 750=8.33% cpu : usr=0.05%, sys=4.81%, ctx=299, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23214: Tue Aug 27 10:25:36 2024 read: IOPS=1, BW=1965KiB/s (2013kB/s)(5120KiB/2605msec) clat (msec): min=11, max=505, avg=196.17, stdev=249.82 lat (msec): min=11, max=505, avg=196.17, stdev=249.82 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 12], | 30.00th=[ 17], 40.00th=[ 17], 50.00th=[ 17], 60.00th=[ 17], | 70.00th=[ 430], 80.00th=[ 430], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 2043, max= 6144, per=70.53%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 write: IOPS=2, BW=2752KiB/s (2818kB/s)(7168KiB/2605msec) clat (msec): min=29, max=634, avg=226.64, stdev=247.95 lat (msec): min=29, max=634, avg=226.70, stdev=247.95 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 37], | 30.00th=[ 82], 40.00th=[ 82], 50.00th=[ 112], 60.00th=[ 167], | 70.00th=[ 167], 80.00th=[ 527], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 2048, max= 6144, per=50.41%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (msec) : 20=25.00%, 50=16.67%, 100=8.33%, 250=16.67%, 500=8.33% lat (msec) : 750=25.00% cpu : usr=0.04%, sys=4.49%, ctx=69, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5805KiB/s (5944kB/s), 670KiB/s-3077KiB/s (686kB/s-3150kB/s), io=20.0MiB (20.0MB), run=1997-3528msec WRITE: bw=8127KiB/s (8322kB/s), 1451KiB/s-3351KiB/s (1486kB/s-3431kB/s), io=28.0MiB (29.4MB), run=1997-3528msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23222: Tue Aug 27 10:25:40 2024 read: IOPS=0, BW=3706KiB/s (3795kB/s)(12.0MiB/3316msec) clat (msec): min=211, max=2122, avg=1094.81, stdev=963.51 lat (msec): min=211, max=2122, avg=1094.81, stdev=963.51 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 211], 10.00th=[ 211], 20.00th=[ 211], | 30.00th=[ 211], 40.00th=[ 953], 50.00th=[ 953], 60.00th=[ 953], | 70.00th=[ 2123], 80.00th=[ 2123], 90.00th=[ 2123], 95.00th=[ 2123], | 99.00th=[ 2123], 99.50th=[ 2123], 99.90th=[ 2123], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 7393, max= 7393, per=100.00%, avg=7393.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.03%, sys=30.02%, ctx=1523, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23223: Tue Aug 27 10:25:40 2024 read: IOPS=0, BW=1519KiB/s (1555kB/s)(4096KiB/2697msec) clat (nsec): min=207646k, max=207646k, avg=207646303.00, stdev= 0.00 lat (nsec): min=207648k, max=207648k, avg=207647659.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 209], 10.00th=[ 209], 20.00th=[ 209], | 30.00th=[ 209], 40.00th=[ 209], 50.00th=[ 209], 60.00th=[ 209], | 70.00th=[ 209], 80.00th=[ 209], 90.00th=[ 209], 95.00th=[ 209], | 99.00th=[ 209], 99.50th=[ 209], 99.90th=[ 209], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 7406, max= 7406, per=100.00%, avg=7406.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3037KiB/s (3110kB/s)(8192KiB/2697msec) clat (msec): min=131, max=2326, avg=1229.10, stdev=1552.19 lat (msec): min=131, max=2326, avg=1229.34, stdev=1552.25 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 132], 10.00th=[ 132], 20.00th=[ 132], | 30.00th=[ 132], 40.00th=[ 132], 50.00th=[ 132], 60.00th=[ 2333], | 70.00th=[ 2333], 80.00th=[ 2333], 90.00th=[ 2333], 95.00th=[ 2333], | 99.00th=[ 2333], 99.50th=[ 2333], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7406, max= 7406, per=60.96%, avg=7406.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=9.35%, ctx=92, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23225: Tue Aug 27 10:25:40 2024 write: IOPS=6, BW=25.7MiB/s (26.9MB/s)(12.0MiB/467msec) clat (msec): min=135, max=180, avg=151.37, stdev=25.37 lat (msec): min=136, max=180, avg=151.73, stdev=25.37 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 138], 50.00th=[ 138], 60.00th=[ 138], | 70.00th=[ 182], 80.00th=[ 182], 90.00th=[ 182], 95.00th=[ 182], | 99.00th=[ 182], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=54.29%, ctx=55, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23226: Tue Aug 27 10:25:40 2024 write: IOPS=2, BW=8847KiB/s (9059kB/s)(12.0MiB/1389msec) clat (msec): min=97, max=1138, avg=453.49, stdev=593.15 lat (msec): min=98, max=1138, avg=453.81, stdev=593.13 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 99], 10.00th=[ 99], 20.00th=[ 99], | 30.00th=[ 99], 40.00th=[ 125], 50.00th=[ 125], 60.00th=[ 125], | 70.00th=[ 1133], 80.00th=[ 1133], 90.00th=[ 1133], 95.00th=[ 1133], | 99.00th=[ 1133], 99.50th=[ 1133], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 8192, max= 8192, per=67.43%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=18.44%, ctx=68, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4941KiB/s (5059kB/s), 1519KiB/s-3706KiB/s (1555kB/s-3795kB/s), io=16.0MiB (16.8MB), run=2697-3316msec WRITE: bw=11.9MiB/s (12.4MB/s), 3037KiB/s-25.7MiB/s (3110kB/s-26.9MB/s), io=32.0MiB (33.6MB), run=467-2697msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23227: Tue Aug 27 10:25:41 2024 read: IOPS=0, BW=3749KiB/s (3839kB/s)(12.0MiB/3278msec) clat (msec): min=278, max=1905, avg=1084.95, stdev=813.47 lat (msec): min=278, max=1905, avg=1084.95, stdev=813.47 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 279], 10.00th=[ 279], 20.00th=[ 279], | 30.00th=[ 279], 40.00th=[ 1070], 50.00th=[ 1070], 60.00th=[ 1070], | 70.00th=[ 1905], 80.00th=[ 1905], 90.00th=[ 1905], 95.00th=[ 1905], | 99.00th=[ 1905], 99.50th=[ 1905], 99.90th=[ 1905], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 7757, max= 8192, per=100.00%, avg=7974.50, stdev=307.59, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33% cpu : usr=0.03%, sys=1.98%, ctx=40, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23228: Tue Aug 27 10:25:41 2024 read: IOPS=0, BW=1420KiB/s (1454kB/s)(4096KiB/2884msec) clat (nsec): min=256596k, max=256596k, avg=256596457.00, stdev= 0.00 lat (nsec): min=256599k, max=256599k, avg=256598551.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 257], 5.00th=[ 257], 10.00th=[ 257], 20.00th=[ 257], | 30.00th=[ 257], 40.00th=[ 257], 50.00th=[ 257], 60.00th=[ 257], | 70.00th=[ 257], 80.00th=[ 257], 90.00th=[ 257], 95.00th=[ 257], | 99.00th=[ 257], 99.50th=[ 257], 99.90th=[ 257], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2840KiB/s (2909kB/s)(8192KiB/2884msec) clat (msec): min=169, max=2417, avg=1293.47, stdev=1589.30 lat (msec): min=169, max=2417, avg=1293.77, stdev=1589.29 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 169], 10.00th=[ 169], 20.00th=[ 169], | 30.00th=[ 169], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 2433], | 70.00th=[ 2433], 80.00th=[ 2433], 90.00th=[ 2433], 95.00th=[ 2433], | 99.00th=[ 2433], 99.50th=[ 2433], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 8192, max= 8192, per=85.63%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.07%, sys=3.61%, ctx=189, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23229: Tue Aug 27 10:25:41 2024 write: IOPS=0, BW=3588KiB/s (3674kB/s)(12.0MiB/3425msec) clat (msec): min=276, max=2044, avg=1138.31, stdev=884.64 lat (msec): min=276, max=2044, avg=1138.63, stdev=884.56 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 275], 10.00th=[ 275], 20.00th=[ 275], | 30.00th=[ 275], 40.00th=[ 1099], 50.00th=[ 1099], 60.00th=[ 1099], | 70.00th=[ 2039], 80.00th=[ 2039], 90.00th=[ 2039], 95.00th=[ 2039], | 99.00th=[ 2039], 99.50th=[ 2039], 99.90th=[ 2039], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 8192, max= 8192, per=85.63%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33% cpu : usr=0.06%, sys=3.04%, ctx=61, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23230: Tue Aug 27 10:25:41 2024 write: IOPS=2, BW=8709KiB/s (8918kB/s)(12.0MiB/1411msec) clat (msec): min=124, max=1067, avg=459.63, stdev=527.48 lat (msec): min=125, max=1067, avg=459.91, stdev=527.48 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 126], 10.00th=[ 126], 20.00th=[ 126], | 30.00th=[ 126], 40.00th=[ 186], 50.00th=[ 186], 60.00th=[ 186], | 70.00th=[ 1070], 80.00th=[ 1070], 90.00th=[ 1070], 95.00th=[ 1070], | 99.00th=[ 1070], 99.50th=[ 1070], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 8192, max= 8192, per=85.63%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.07%, sys=7.66%, ctx=213, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4998KiB/s (5118kB/s), 1420KiB/s-3749KiB/s (1454kB/s-3839kB/s), io=16.0MiB (16.8MB), run=2884-3278msec WRITE: bw=9567KiB/s (9797kB/s), 2840KiB/s-8709KiB/s (2909kB/s-8918kB/s), io=32.0MiB (33.6MB), run=1411-3425msec