== sanity test 398b: DIO and buffer IO race ============== 19:20:01 (1717024801) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.49733 s, 33.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24540: Wed May 29 19:20:19 2024 read: IOPS=99, BW=397KiB/s (406kB/s)(5992KiB/15105msec) clat (usec): min=42, max=219861, avg=5861.44, stdev=6354.53 lat (usec): min=43, max=219863, avg=5862.20, stdev=6354.57 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 2376], 10.00th=[ 2933], 20.00th=[ 3458], | 30.00th=[ 3916], 40.00th=[ 4490], 50.00th=[ 5145], 60.00th=[ 5932], | 70.00th=[ 6718], 80.00th=[ 7767], 90.00th=[ 9241], 95.00th=[ 11076], | 99.00th=[ 17695], 99.50th=[ 19792], 99.90th=[ 26084], 99.95th=[219153], | 99.99th=[219153] bw ( KiB/s): min= 216, max= 496, per=25.32%, avg=395.57, stdev=54.44, samples=30 iops : min= 54, max= 124, avg=98.77, stdev=13.58, samples=30 write: IOPS=104, BW=417KiB/s (427kB/s)(6296KiB/15105msec) clat (usec): min=453, max=20485, avg=3983.29, stdev=3559.98 lat (usec): min=454, max=20486, avg=3984.41, stdev=3559.92 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 652], 10.00th=[ 709], 20.00th=[ 791], | 30.00th=[ 963], 40.00th=[ 2212], 50.00th=[ 3228], 60.00th=[ 4015], | 70.00th=[ 4948], 80.00th=[ 6652], 90.00th=[ 9110], 95.00th=[11207], | 99.00th=[15664], 99.50th=[17171], 99.90th=[20317], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 240, max= 574, per=25.38%, avg=415.00, stdev=78.23, samples=30 iops : min= 60, max= 143, avg=103.63, stdev=19.57, samples=30 lat (usec) : 50=0.03%, 100=1.53%, 250=0.42%, 500=0.29%, 750=7.45% lat (usec) : 1000=8.14% lat (msec) : 2=4.36%, 4=23.73%, 10=46.65%, 20=7.06%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=23.58%, ctx=6059, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24541: Wed May 29 19:20:19 2024 read: IOPS=96, BW=386KiB/s (396kB/s)(5940KiB/15371msec) clat (usec): min=51, max=26203, avg=5721.30, stdev=3191.75 lat (usec): min=51, max=26203, avg=5722.34, stdev=3191.66 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 2376], 10.00th=[ 2868], 20.00th=[ 3392], | 30.00th=[ 3851], 40.00th=[ 4424], 50.00th=[ 5080], 60.00th=[ 5866], | 70.00th=[ 6718], 80.00th=[ 7701], 90.00th=[ 9503], 95.00th=[11338], | 99.00th=[17433], 99.50th=[20055], 99.90th=[25035], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 200, max= 528, per=24.36%, avg=380.43, stdev=63.26, samples=30 iops : min= 50, max= 132, avg=95.03, stdev=15.84, samples=30 write: IOPS=103, BW=413KiB/s (423kB/s)(6348KiB/15371msec) clat (usec): min=451, max=204840, avg=4296.25, stdev=6349.85 lat (usec): min=451, max=204841, avg=4297.24, stdev=6349.88 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 652], 10.00th=[ 717], 20.00th=[ 791], | 30.00th=[ 979], 40.00th=[ 2507], 50.00th=[ 3359], 60.00th=[ 4146], | 70.00th=[ 5145], 80.00th=[ 6915], 90.00th=[ 9503], 95.00th=[ 11863], | 99.00th=[ 16450], 99.50th=[ 20841], 99.90th=[ 38011], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 256, max= 488, per=24.52%, avg=400.97, stdev=54.93, samples=30 iops : min= 64, max= 122, avg=100.17, stdev=13.76, samples=30 lat (usec) : 100=1.43%, 250=0.52%, 500=0.23%, 750=7.58%, 1000=8.20% lat (msec) : 2=3.94%, 4=24.12%, 10=45.74%, 20=7.71%, 50=0.49% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=22.83%, ctx=6201, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24542: Wed May 29 19:20:19 2024 read: IOPS=99, BW=397KiB/s (406kB/s)(6028KiB/15201msec) clat (usec): min=44, max=33176, avg=5648.49, stdev=3096.65 lat (usec): min=44, max=33177, avg=5649.34, stdev=3096.61 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 2343], 10.00th=[ 2868], 20.00th=[ 3392], | 30.00th=[ 3884], 40.00th=[ 4424], 50.00th=[ 5080], 60.00th=[ 5866], | 70.00th=[ 6718], 80.00th=[ 7635], 90.00th=[ 9241], 95.00th=[10683], | 99.00th=[16909], 99.50th=[18482], 99.90th=[22152], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 168, max= 504, per=25.24%, avg=394.23, stdev=64.37, samples=30 iops : min= 42, max= 126, avg=98.43, stdev=16.12, samples=30 write: IOPS=102, BW=412KiB/s (422kB/s)(6260KiB/15201msec) clat (usec): min=471, max=213591, avg=4236.17, stdev=6505.31 lat (usec): min=472, max=213592, avg=4237.60, stdev=6505.54 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 619], 10.00th=[ 685], 20.00th=[ 758], | 30.00th=[ 898], 40.00th=[ 1975], 50.00th=[ 3326], 60.00th=[ 4080], | 70.00th=[ 5211], 80.00th=[ 6849], 90.00th=[ 9765], 95.00th=[ 11338], | 99.00th=[ 16057], 99.50th=[ 18482], 99.90th=[ 25035], 99.95th=[212861], | 99.99th=[212861] bw ( KiB/s): min= 208, max= 536, per=25.01%, avg=408.90, stdev=72.43, samples=30 iops : min= 52, max= 134, avg=102.10, stdev=18.08, samples=30 lat (usec) : 50=0.03%, 100=1.50%, 250=0.59%, 500=0.26%, 750=9.54% lat (usec) : 1000=7.16% lat (msec) : 2=3.52%, 4=23.40%, 10=45.80%, 20=7.88%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=22.96%, ctx=6082, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24543: Wed May 29 19:20:19 2024 read: IOPS=99, BW=398KiB/s (408kB/s)(6052KiB/15195msec) clat (usec): min=52, max=26301, avg=5630.04, stdev=3057.87 lat (usec): min=52, max=26302, avg=5631.91, stdev=3056.78 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 2245], 10.00th=[ 2900], 20.00th=[ 3359], | 30.00th=[ 3851], 40.00th=[ 4424], 50.00th=[ 5080], 60.00th=[ 5866], | 70.00th=[ 6652], 80.00th=[ 7635], 90.00th=[ 9241], 95.00th=[11076], | 99.00th=[16581], 99.50th=[18744], 99.90th=[23200], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 200, max= 512, per=25.22%, avg=393.93, stdev=58.93, samples=30 iops : min= 50, max= 128, avg=98.33, stdev=14.73, samples=30 write: IOPS=102, BW=410KiB/s (420kB/s)(6236KiB/15195msec) clat (usec): min=444, max=215754, avg=4251.67, stdev=6667.50 lat (usec): min=444, max=215755, avg=4252.93, stdev=6667.35 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 644], 10.00th=[ 701], 20.00th=[ 775], | 30.00th=[ 898], 40.00th=[ 1663], 50.00th=[ 3228], 60.00th=[ 4015], | 70.00th=[ 5080], 80.00th=[ 7046], 90.00th=[ 9634], 95.00th=[ 11863], | 99.00th=[ 16909], 99.50th=[ 19530], 99.90th=[ 34866], 99.95th=[214959], | 99.99th=[214959] bw ( KiB/s): min= 216, max= 576, per=24.78%, avg=405.13, stdev=74.86, samples=30 iops : min= 54, max= 144, avg=101.13, stdev=18.75, samples=30 lat (usec) : 100=1.46%, 250=0.75%, 500=0.23%, 750=7.81%, 1000=8.82% lat (msec) : 2=4.23%, 4=23.18%, 10=45.38%, 20=7.75%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=22.93%, ctx=6140, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1562KiB/s (1600kB/s), 386KiB/s-398KiB/s (396kB/s-408kB/s), io=23.4MiB (24.6MB), run=15105-15371msec WRITE: bw=1636KiB/s (1675kB/s), 410KiB/s-417KiB/s (420kB/s-427kB/s), io=24.6MiB (25.7MB), run=15105-15371msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24536: Wed May 29 19:20:24 2024 read: IOPS=77, BW=309KiB/s (316kB/s)(5992KiB/19416msec) clat (usec): min=1593, max=247176, avg=5223.10, stdev=7497.94 lat (usec): min=1593, max=247177, avg=5223.96, stdev=7497.96 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 2024], 10.00th=[ 2147], 20.00th=[ 2311], | 30.00th=[ 2540], 40.00th=[ 2868], 50.00th=[ 3326], 60.00th=[ 3949], | 70.00th=[ 4948], 80.00th=[ 7111], 90.00th=[ 11338], 95.00th=[ 14222], | 99.00th=[ 20841], 99.50th=[ 23725], 99.90th=[ 30802], 99.95th=[246416], | 99.99th=[246416] bw ( KiB/s): min= 112, max= 678, per=24.65%, avg=302.97, stdev=154.45, samples=38 iops : min= 28, max= 169, avg=75.63, stdev=38.64, samples=38 write: IOPS=81, BW=324KiB/s (332kB/s)(6296KiB/19416msec) clat (usec): min=2439, max=37239, avg=7325.15, stdev=4574.31 lat (usec): min=2440, max=37240, avg=7326.37, stdev=4574.14 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 3163], 10.00th=[ 3458], 20.00th=[ 3818], | 30.00th=[ 4228], 40.00th=[ 4752], 50.00th=[ 5538], 60.00th=[ 6652], | 70.00th=[ 8586], 80.00th=[10814], 90.00th=[13960], 95.00th=[16319], | 99.00th=[23987], 99.50th=[26084], 99.90th=[30016], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 104, max= 736, per=24.65%, avg=317.29, stdev=161.42, samples=38 iops : min= 26, max= 184, avg=79.21, stdev=40.36, samples=38 lat (msec) : 2=2.21%, 4=40.20%, 10=39.58%, 20=16.47%, 50=1.50% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=10.99%, ctx=3640, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24537: Wed May 29 19:20:24 2024 read: IOPS=76, BW=306KiB/s (313kB/s)(5940KiB/19419msec) clat (usec): min=1577, max=207451, avg=5044.38, stdev=6636.05 lat (usec): min=1578, max=207452, avg=5045.19, stdev=6636.08 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 2008], 10.00th=[ 2114], 20.00th=[ 2311], | 30.00th=[ 2507], 40.00th=[ 2835], 50.00th=[ 3228], 60.00th=[ 3785], | 70.00th=[ 4686], 80.00th=[ 6652], 90.00th=[ 10552], 95.00th=[ 13698], | 99.00th=[ 21365], 99.50th=[ 22414], 99.90th=[ 28705], 99.95th=[206570], | 99.99th=[206570] bw ( KiB/s): min= 96, max= 718, per=24.57%, avg=301.97, stdev=165.78, samples=38 iops : min= 24, max= 179, avg=75.42, stdev=41.44, samples=38 write: IOPS=81, BW=327KiB/s (335kB/s)(6348KiB/19419msec) clat (usec): min=2483, max=43592, avg=7476.38, stdev=4754.92 lat (usec): min=2484, max=43598, avg=7477.45, stdev=4755.24 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3163], 10.00th=[ 3458], 20.00th=[ 3851], | 30.00th=[ 4228], 40.00th=[ 4817], 50.00th=[ 5669], 60.00th=[ 6980], | 70.00th=[ 8717], 80.00th=[10814], 90.00th=[13829], 95.00th=[16450], | 99.00th=[25822], 99.50th=[27657], 99.90th=[34341], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 152, max= 720, per=24.76%, avg=318.61, stdev=154.74, samples=38 iops : min= 38, max= 180, avg=79.58, stdev=38.68, samples=38 lat (msec) : 2=2.34%, 4=40.27%, 10=39.55%, 20=15.95%, 50=1.86% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=10.96%, ctx=3681, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24538: Wed May 29 19:20:24 2024 read: IOPS=77, BW=309KiB/s (316kB/s)(6028KiB/19531msec) clat (usec): min=1617, max=27476, avg=5027.17, stdev=4074.24 lat (usec): min=1618, max=27476, avg=5027.95, stdev=4074.27 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2040], 10.00th=[ 2180], 20.00th=[ 2343], | 30.00th=[ 2573], 40.00th=[ 2868], 50.00th=[ 3294], 60.00th=[ 3851], | 70.00th=[ 4883], 80.00th=[ 6980], 90.00th=[11600], 95.00th=[14091], | 99.00th=[19792], 99.50th=[21627], 99.90th=[24249], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 104, max= 672, per=25.09%, avg=308.41, stdev=156.72, samples=39 iops : min= 26, max= 168, avg=77.05, stdev=39.18, samples=39 write: IOPS=80, BW=321KiB/s (328kB/s)(6260KiB/19531msec) clat (msec): min=2, max=212, avg= 7.60, stdev= 7.14 lat (msec): min=2, max=212, avg= 7.60, stdev= 7.14 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 14], 95.00th=[ 17], | 99.00th=[ 27], 99.50th=[ 31], 99.90th=[ 37], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 128, max= 712, per=24.86%, avg=319.90, stdev=169.85, samples=39 iops : min= 32, max= 178, avg=79.92, stdev=42.46, samples=39 lat (msec) : 2=1.79%, 4=41.60%, 10=38.51%, 20=16.21%, 50=1.86% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=10.79%, ctx=3635, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24539: Wed May 29 19:20:24 2024 read: IOPS=77, BW=310KiB/s (318kB/s)(6052KiB/19506msec) clat (usec): min=1679, max=24360, avg=4984.34, stdev=4000.22 lat (usec): min=1680, max=24360, avg=4985.48, stdev=4000.06 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 2024], 10.00th=[ 2114], 20.00th=[ 2311], | 30.00th=[ 2507], 40.00th=[ 2802], 50.00th=[ 3261], 60.00th=[ 3851], | 70.00th=[ 4948], 80.00th=[ 7177], 90.00th=[10945], 95.00th=[13829], | 99.00th=[19530], 99.50th=[21365], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 112, max= 696, per=25.23%, avg=310.03, stdev=163.03, samples=39 iops : min= 28, max= 174, avg=77.44, stdev=40.76, samples=39 write: IOPS=79, BW=320KiB/s (327kB/s)(6236KiB/19506msec) clat (msec): min=2, max=212, avg= 7.63, stdev= 7.17 lat (msec): min=2, max=212, avg= 7.63, stdev= 7.17 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 14], 95.00th=[ 17], | 99.00th=[ 26], 99.50th=[ 34], 99.90th=[ 38], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 128, max= 734, per=24.82%, avg=319.46, stdev=165.13, samples=39 iops : min= 32, max= 183, avg=79.79, stdev=41.28, samples=39 lat (msec) : 2=2.12%, 4=40.43%, 10=39.29%, 20=16.37%, 50=1.76% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=10.89%, ctx=3710, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1229KiB/s (1259kB/s), 306KiB/s-310KiB/s (313kB/s-318kB/s), io=23.4MiB (24.6MB), run=19416-19531msec WRITE: bw=1287KiB/s (1318kB/s), 320KiB/s-327KiB/s (327kB/s-335kB/s), io=24.6MiB (25.7MB), run=19416-19531msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24574: Wed May 29 19:20:32 2024 read: IOPS=50, BW=815KiB/s (835kB/s)(5920KiB/7260msec) clat (usec): min=2054, max=50514, avg=7384.52, stdev=6003.08 lat (usec): min=2055, max=50515, avg=7385.33, stdev=6003.06 clat percentiles (usec): | 1.00th=[ 2278], 5.00th=[ 2606], 10.00th=[ 2802], 20.00th=[ 3097], | 30.00th=[ 3621], 40.00th=[ 4015], 50.00th=[ 4948], 60.00th=[ 6128], | 70.00th=[ 8586], 80.00th=[11338], 90.00th=[15008], 95.00th=[18744], | 99.00th=[29492], 99.50th=[40109], 99.90th=[50594], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 544, max= 1088, per=25.31%, avg=815.43, stdev=159.36, samples=14 iops : min= 34, max= 68, avg=50.71, stdev=10.06, samples=14 write: IOPS=54, BW=877KiB/s (898kB/s)(6368KiB/7260msec) clat (usec): min=3059, max=66928, avg=11348.32, stdev=6941.37 lat (usec): min=3061, max=66929, avg=11350.11, stdev=6941.32 clat percentiles (usec): | 1.00th=[ 3425], 5.00th=[ 4228], 10.00th=[ 4686], 20.00th=[ 5604], | 30.00th=[ 6587], 40.00th=[ 8291], 50.00th=[ 9634], 60.00th=[10945], | 70.00th=[13304], 80.00th=[15926], 90.00th=[20317], 95.00th=[23987], | 99.00th=[32637], 99.50th=[35914], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 544, max= 1184, per=26.08%, avg=884.00, stdev=192.75, samples=14 iops : min= 34, max= 74, avg=55.00, stdev=12.10, samples=14 lat (msec) : 4=20.18%, 10=43.10%, 20=29.56%, 50=6.90%, 100=0.26% cpu : usr=0.15%, sys=9.66%, ctx=1113, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24575: Wed May 29 19:20:32 2024 read: IOPS=49, BW=788KiB/s (807kB/s)(5744KiB/7292msec) clat (usec): min=2003, max=37457, avg=7629.32, stdev=6105.41 lat (usec): min=2004, max=37458, avg=7630.14, stdev=6105.42 clat percentiles (usec): | 1.00th=[ 2278], 5.00th=[ 2442], 10.00th=[ 2638], 20.00th=[ 3130], | 30.00th=[ 3490], 40.00th=[ 3916], 50.00th=[ 4752], 60.00th=[ 6915], | 70.00th=[ 9634], 80.00th=[11994], 90.00th=[15533], 95.00th=[20841], | 99.00th=[29492], 99.50th=[30802], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 638, max= 896, per=24.46%, avg=788.14, stdev=85.69, samples=14 iops : min= 39, max= 56, avg=49.07, stdev= 5.37, samples=14 write: IOPS=56, BW=897KiB/s (919kB/s)(6544KiB/7292msec) clat (usec): min=3376, max=83652, avg=11107.70, stdev=7357.51 lat (usec): min=3377, max=83654, avg=11109.26, stdev=7357.50 clat percentiles (usec): | 1.00th=[ 3589], 5.00th=[ 4113], 10.00th=[ 4621], 20.00th=[ 5800], | 30.00th=[ 6980], 40.00th=[ 8094], 50.00th=[ 9241], 60.00th=[10683], | 70.00th=[12518], 80.00th=[15008], 90.00th=[19530], 95.00th=[22676], | 99.00th=[38011], 99.50th=[41681], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 640, max= 1085, per=26.35%, avg=893.21, stdev=139.32, samples=14 iops : min= 40, max= 67, avg=55.64, stdev= 8.71, samples=14 lat (msec) : 4=21.09%, 10=42.32%, 20=29.43%, 50=7.03%, 100=0.13% cpu : usr=0.16%, sys=9.49%, ctx=1106, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24576: Wed May 29 19:20:32 2024 read: IOPS=54, BW=878KiB/s (899kB/s)(6256KiB/7123msec) clat (usec): min=2140, max=53580, avg=7689.40, stdev=6061.07 lat (usec): min=2141, max=53581, avg=7690.37, stdev=6061.01 clat percentiles (usec): | 1.00th=[ 2245], 5.00th=[ 2573], 10.00th=[ 2835], 20.00th=[ 3228], | 30.00th=[ 3687], 40.00th=[ 4228], 50.00th=[ 5145], 60.00th=[ 6718], | 70.00th=[ 9503], 80.00th=[11863], 90.00th=[15270], 95.00th=[19268], | 99.00th=[29230], 99.50th=[31065], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 640, max= 1248, per=27.60%, avg=889.14, stdev=200.89, samples=14 iops : min= 40, max= 78, avg=55.57, stdev=12.56, samples=14 write: IOPS=52, BW=847KiB/s (867kB/s)(6032KiB/7123msec) clat (usec): min=2865, max=44270, avg=10890.76, stdev=6338.36 lat (usec): min=2866, max=44271, avg=10893.74, stdev=6339.01 clat percentiles (usec): | 1.00th=[ 3458], 5.00th=[ 4424], 10.00th=[ 4883], 20.00th=[ 5669], | 30.00th=[ 6718], 40.00th=[ 7767], 50.00th=[ 9241], 60.00th=[10683], | 70.00th=[12518], 80.00th=[15008], 90.00th=[19006], 95.00th=[22414], | 99.00th=[35390], 99.50th=[35914], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 576, max= 1088, per=24.88%, avg=843.43, stdev=157.57, samples=14 iops : min= 36, max= 68, avg=52.71, stdev= 9.85, samples=14 lat (msec) : 4=19.01%, 10=44.53%, 20=29.95%, 50=6.38%, 100=0.13% cpu : usr=0.25%, sys=9.46%, ctx=1114, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24577: Wed May 29 19:20:32 2024 read: IOPS=50, BW=812KiB/s (831kB/s)(6032KiB/7433msec) clat (usec): min=1639, max=47586, avg=7925.85, stdev=6775.13 lat (usec): min=1640, max=47587, avg=7926.72, stdev=6775.12 clat percentiles (usec): | 1.00th=[ 2245], 5.00th=[ 2606], 10.00th=[ 2802], 20.00th=[ 3130], | 30.00th=[ 3589], 40.00th=[ 4146], 50.00th=[ 5014], 60.00th=[ 6652], | 70.00th=[ 8979], 80.00th=[11863], 90.00th=[17433], 95.00th=[20317], | 99.00th=[35390], 99.50th=[39584], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 544, max= 1053, per=24.60%, avg=792.50, stdev=132.20, samples=14 iops : min= 34, max= 65, avg=49.29, stdev= 8.02, samples=14 write: IOPS=52, BW=842KiB/s (862kB/s)(6256KiB/7433msec) clat (usec): min=3246, max=60230, avg=11341.72, stdev=6899.12 lat (usec): min=3247, max=60231, avg=11343.33, stdev=6899.17 clat percentiles (usec): | 1.00th=[ 3720], 5.00th=[ 4293], 10.00th=[ 4883], 20.00th=[ 6063], | 30.00th=[ 6915], 40.00th=[ 8356], 50.00th=[ 9634], 60.00th=[11076], | 70.00th=[12911], 80.00th=[15270], 90.00th=[19792], 95.00th=[24511], | 99.00th=[35914], 99.50th=[42206], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 480, max= 1088, per=24.59%, avg=833.64, stdev=179.05, samples=14 iops : min= 30, max= 68, avg=51.86, stdev=11.24, samples=14 lat (msec) : 2=0.13%, 4=19.92%, 10=42.58%, 20=29.95%, 50=7.29% lat (msec) : 100=0.13% cpu : usr=0.20%, sys=9.46%, ctx=1163, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3222KiB/s (3300kB/s), 788KiB/s-878KiB/s (807kB/s-899kB/s), io=23.4MiB (24.5MB), run=7123-7433msec WRITE: bw=3390KiB/s (3472kB/s), 842KiB/s-897KiB/s (862kB/s-919kB/s), io=24.6MiB (25.8MB), run=7123-7433msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24570: Wed May 29 19:20:32 2024 read: IOPS=51, BW=817KiB/s (837kB/s)(5920KiB/7244msec) clat (usec): min=56, max=39331, avg=13625.11, stdev=4679.55 lat (usec): min=57, max=39331, avg=13625.88, stdev=4679.57 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 7701], 10.00th=[ 9503], 20.00th=[10945], | 30.00th=[12125], 40.00th=[12780], 50.00th=[13435], 60.00th=[14222], | 70.00th=[15270], 80.00th=[16319], 90.00th=[17695], 95.00th=[21365], | 99.00th=[26346], 99.50th=[31327], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 606, max= 1024, per=25.38%, avg=813.57, stdev=98.92, samples=14 iops : min= 37, max= 64, avg=50.79, stdev= 6.33, samples=14 write: IOPS=54, BW=879KiB/s (900kB/s)(6368KiB/7244msec) clat (usec): min=717, max=65945, avg=5507.25, stdev=6080.62 lat (usec): min=718, max=65948, avg=5508.67, stdev=6080.63 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 1037], 10.00th=[ 1123], 20.00th=[ 1418], | 30.00th=[ 2573], 40.00th=[ 3064], 50.00th=[ 3556], 60.00th=[ 4293], | 70.00th=[ 5735], 80.00th=[ 7832], 90.00th=[13304], 95.00th=[16909], | 99.00th=[27657], 99.50th=[43254], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 512, max= 1216, per=26.23%, avg=884.43, stdev=217.68, samples=14 iops : min= 32, max= 76, avg=55.21, stdev=13.63, samples=14 lat (usec) : 100=0.65%, 250=1.17%, 750=0.13%, 1000=2.21% lat (msec) : 2=10.16%, 4=16.80%, 10=18.75%, 20=46.09%, 50=3.91% lat (msec) : 100=0.13% cpu : usr=0.19%, sys=19.81%, ctx=2775, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24571: Wed May 29 19:20:32 2024 read: IOPS=49, BW=790KiB/s (809kB/s)(5744KiB/7275msec) clat (usec): min=78, max=31392, avg=13721.83, stdev=4461.16 lat (usec): min=79, max=31393, avg=13722.87, stdev=4461.46 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 8586], 10.00th=[ 9765], 20.00th=[10945], | 30.00th=[11994], 40.00th=[12518], 50.00th=[13435], 60.00th=[14091], | 70.00th=[15139], 80.00th=[16712], 90.00th=[19006], 95.00th=[21103], | 99.00th=[27657], 99.50th=[30016], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 638, max= 960, per=24.73%, avg=792.71, stdev=97.69, samples=14 iops : min= 39, max= 60, avg=49.36, stdev= 6.26, samples=14 write: IOPS=56, BW=900KiB/s (921kB/s)(6544KiB/7275msec) clat (usec): min=703, max=60592, avg=5713.67, stdev=6741.77 lat (usec): min=704, max=60594, avg=5717.76, stdev=6742.64 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 955], 10.00th=[ 1037], 20.00th=[ 1303], | 30.00th=[ 2507], 40.00th=[ 3130], 50.00th=[ 3687], 60.00th=[ 4555], | 70.00th=[ 5866], 80.00th=[ 7373], 90.00th=[11600], 95.00th=[19006], | 99.00th=[31851], 99.50th=[42206], 99.90th=[60556], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min= 670, max= 1056, per=26.49%, avg=893.29, stdev=137.83, samples=14 iops : min= 41, max= 66, avg=55.64, stdev= 8.65, samples=14 lat (usec) : 100=1.04%, 250=0.26%, 750=0.39%, 1000=4.04% lat (msec) : 2=8.85%, 4=15.62%, 10=22.14%, 20=42.32%, 50=5.21% lat (msec) : 100=0.13% cpu : usr=0.08%, sys=19.32%, ctx=2748, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24572: Wed May 29 19:20:32 2024 read: IOPS=52, BW=837KiB/s (857kB/s)(6256KiB/7472msec) clat (usec): min=95, max=55830, avg=13673.65, stdev=5185.24 lat (usec): min=96, max=55832, avg=13674.45, stdev=5185.28 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 7373], 10.00th=[ 9503], 20.00th=[10683], | 30.00th=[11600], 40.00th=[12256], 50.00th=[13173], 60.00th=[14222], | 70.00th=[15270], 80.00th=[16450], 90.00th=[19006], 95.00th=[20841], | 99.00th=[32113], 99.50th=[35390], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 606, max= 1024, per=25.53%, avg=818.14, stdev=134.09, samples=14 iops : min= 37, max= 64, avg=51.07, stdev= 8.49, samples=14 write: IOPS=50, BW=807KiB/s (827kB/s)(6032KiB/7472msec) clat (usec): min=699, max=50176, avg=5613.08, stdev=6099.90 lat (usec): min=700, max=50177, avg=5614.97, stdev=6100.45 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 1012], 10.00th=[ 1090], 20.00th=[ 1303], | 30.00th=[ 2573], 40.00th=[ 2999], 50.00th=[ 3589], 60.00th=[ 4228], | 70.00th=[ 5604], 80.00th=[ 7898], 90.00th=[14353], 95.00th=[18482], | 99.00th=[27657], 99.50th=[43254], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 448, max= 1120, per=23.04%, avg=777.00, stdev=199.97, samples=14 iops : min= 28, max= 70, avg=48.50, stdev=12.57, samples=14 lat (usec) : 100=0.13%, 250=1.04%, 500=0.26%, 750=0.52%, 1000=2.08% lat (msec) : 2=9.77%, 4=15.89%, 10=19.14%, 20=46.22%, 50=4.69% lat (msec) : 100=0.26% cpu : usr=0.08%, sys=19.81%, ctx=2849, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24573: Wed May 29 19:20:32 2024 read: IOPS=50, BW=810KiB/s (830kB/s)(6032KiB/7446msec) clat (usec): min=83, max=42508, avg=13838.13, stdev=4966.74 lat (usec): min=84, max=42509, avg=13838.91, stdev=4966.74 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 7898], 10.00th=[ 9634], 20.00th=[11076], | 30.00th=[11863], 40.00th=[12649], 50.00th=[13304], 60.00th=[14222], | 70.00th=[15270], 80.00th=[16450], 90.00th=[19268], 95.00th=[21890], | 99.00th=[31589], 99.50th=[35914], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 574, max= 992, per=25.03%, avg=802.14, stdev=122.28, samples=14 iops : min= 35, max= 62, avg=50.07, stdev= 7.77, samples=14 write: IOPS=52, BW=840KiB/s (860kB/s)(6256KiB/7446msec) clat (usec): min=721, max=45088, avg=5677.68, stdev=5467.89 lat (usec): min=722, max=45089, avg=5679.10, stdev=5467.90 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 922], 10.00th=[ 1090], 20.00th=[ 1483], | 30.00th=[ 2835], 40.00th=[ 3130], 50.00th=[ 3720], 60.00th=[ 4752], | 70.00th=[ 6259], 80.00th=[ 8160], 90.00th=[13566], 95.00th=[16319], | 99.00th=[23200], 99.50th=[40109], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 576, max= 1216, per=25.01%, avg=843.29, stdev=174.85, samples=14 iops : min= 36, max= 76, avg=52.64, stdev=11.02, samples=14 lat (usec) : 100=0.78%, 250=0.52%, 750=0.52%, 1000=3.52% lat (msec) : 2=7.29%, 4=16.41%, 10=20.18%, 20=46.09%, 50=4.69% cpu : usr=0.11%, sys=20.32%, ctx=2850, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3206KiB/s (3283kB/s), 790KiB/s-837KiB/s (809kB/s-857kB/s), io=23.4MiB (24.5MB), run=7244-7472msec WRITE: bw=3373KiB/s (3454kB/s), 807KiB/s-900KiB/s (827kB/s-921kB/s), io=24.6MiB (25.8MB), run=7244-7472msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24589: Wed May 29 19:20:36 2024 read: IOPS=2, BW=2788KiB/s (2855kB/s)(7168KiB/2571msec) clat (usec): min=419, max=1000.7k, avg=238860.34, stdev=369711.40 lat (usec): min=419, max=1000.7k, avg=238861.95, stdev=369711.47 clat percentiles (usec): | 1.00th=[ 420], 5.00th=[ 420], 10.00th=[ 420], | 20.00th=[ 478], 30.00th=[ 5014], 40.00th=[ 5014], | 50.00th=[ 56886], 60.00th=[ 183501], 70.00th=[ 183501], | 80.00th=[ 425722], 90.00th=[1002439], 95.00th=[1002439], | 99.00th=[1002439], 99.50th=[1002439], 99.90th=[1002439], | 99.95th=[1002439], 99.99th=[1002439] bw ( KiB/s): min= 4096, max= 6144, per=64.28%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=1, BW=1991KiB/s (2039kB/s)(5120KiB/2571msec) clat (msec): min=17, max=517, avg=178.38, stdev=198.86 lat (msec): min=17, max=517, avg=178.44, stdev=198.87 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 63], 40.00th=[ 63], 50.00th=[ 120], 60.00th=[ 120], | 70.00th=[ 176], 80.00th=[ 176], 90.00th=[ 518], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 1969, max= 2048, per=18.19%, avg=2028.25, stdev=39.50, samples=4 iops : min= 1, max= 2, avg= 1.75, stdev= 0.50, samples=4 lat (usec) : 500=16.67% lat (msec) : 10=8.33%, 20=8.33%, 100=16.67%, 250=25.00%, 500=8.33% lat (msec) : 750=8.33% cpu : usr=0.04%, sys=42.26%, ctx=454, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24591: Wed May 29 19:20:36 2024 read: IOPS=1, BW=1107KiB/s (1134kB/s)(2048KiB/1850msec) clat (usec): min=495, max=65062, avg=32778.83, stdev=45655.72 lat (usec): min=496, max=65064, avg=32780.73, stdev=45656.62 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 494], 10.00th=[ 494], 20.00th=[ 494], | 30.00th=[ 494], 40.00th=[ 494], 50.00th=[ 494], 60.00th=[65274], | 70.00th=[65274], 80.00th=[65274], 90.00th=[65274], 95.00th=[65274], | 99.00th=[65274], 99.50th=[65274], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 2043, max= 2048, per=25.68%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=5, BW=5535KiB/s (5668kB/s)(10.0MiB/1850msec) clat (msec): min=24, max=906, avg=178.03, stdev=270.22 lat (msec): min=24, max=906, avg=178.09, stdev=270.20 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 26], | 30.00th=[ 26], 40.00th=[ 32], 50.00th=[ 51], 60.00th=[ 87], | 70.00th=[ 226], 80.00th=[ 226], 90.00th=[ 264], 95.00th=[ 911], | 99.00th=[ 911], 99.50th=[ 911], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 2043, max= 8192, per=42.84%, avg=4777.00, stdev=3130.55, samples=3 iops : min= 1, max= 8, avg= 4.33, stdev= 3.51, samples=3 lat (usec) : 500=8.33% lat (msec) : 50=41.67%, 100=16.67%, 250=16.67%, 500=8.33%, 1000=8.33% cpu : usr=0.00%, sys=16.39%, ctx=63, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24593: Wed May 29 19:20:36 2024 read: IOPS=2, BW=2494KiB/s (2553kB/s)(6144KiB/2464msec) clat (usec): min=496, max=474553, avg=176264.89, stdev=209498.83 lat (usec): min=497, max=474555, avg=176266.38, stdev=209498.90 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 498], 10.00th=[ 498], 20.00th=[ 24511], | 30.00th=[ 24511], 40.00th=[ 48497], 50.00th=[ 48497], 60.00th=[100140], | 70.00th=[408945], 80.00th=[408945], 90.00th=[476054], 95.00th=[476054], | 99.00th=[476054], 99.50th=[476054], 99.90th=[476054], 99.95th=[476054], | 99.99th=[476054] bw ( KiB/s): min= 4096, max= 4096, per=51.42%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2494KiB/s (2553kB/s)(6144KiB/2464msec) clat (msec): min=25, max=929, avg=234.10, stdev=355.76 lat (msec): min=25, max=929, avg=234.16, stdev=355.76 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 35], | 30.00th=[ 35], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 70], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 927], 95.00th=[ 927], | 99.00th=[ 927], 99.50th=[ 927], 99.90th=[ 927], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 2048, max= 6144, per=30.61%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 lat (usec) : 500=8.33% lat (msec) : 50=41.67%, 100=16.67%, 500=25.00%, 1000=8.33% cpu : usr=0.00%, sys=16.00%, ctx=737, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24595: Wed May 29 19:20:36 2024 read: IOPS=2, BW=2075KiB/s (2125kB/s)(5120KiB/2467msec) clat (usec): min=475, max=381434, avg=169897.43, stdev=156246.40 lat (usec): min=476, max=381435, avg=169898.87, stdev=156246.37 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 478], 10.00th=[ 478], 20.00th=[ 478], | 30.00th=[ 45876], 40.00th=[ 45876], 50.00th=[158335], 60.00th=[158335], | 70.00th=[263193], 80.00th=[263193], 90.00th=[379585], 95.00th=[379585], | 99.00th=[379585], 99.50th=[379585], 99.90th=[379585], 99.95th=[379585], | 99.99th=[379585] bw ( KiB/s): min= 4087, max= 4096, per=51.37%, avg=4091.50, stdev= 6.36, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2906KiB/s (2975kB/s)(7168KiB/2467msec) clat (msec): min=72, max=541, avg=230.62, stdev=174.80 lat (msec): min=72, max=541, avg=230.69, stdev=174.79 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 73], 10.00th=[ 73], 20.00th=[ 73], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 144], 60.00th=[ 284], | 70.00th=[ 284], 80.00th=[ 363], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 2043, max= 6131, per=27.51%, avg=3067.50, stdev=2042.33, samples=4 iops : min= 1, max= 5, avg= 2.50, stdev= 1.73, samples=4 lat (usec) : 500=8.33% lat (msec) : 50=8.33%, 100=16.67%, 250=25.00%, 500=33.33%, 750=8.33% cpu : usr=0.00%, sys=27.09%, ctx=306, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7966KiB/s (8157kB/s), 1107KiB/s-2788KiB/s (1134kB/s-2855kB/s), io=20.0MiB (20.0MB), run=1850-2571msec WRITE: bw=10.9MiB/s (11.4MB/s), 1991KiB/s-5535KiB/s (2039kB/s-5668kB/s), io=28.0MiB (29.4MB), run=1850-2571msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24590: Wed May 29 19:20:36 2024 read: IOPS=2, BW=2692KiB/s (2756kB/s)(7168KiB/2663msec) clat (msec): min=20, max=961, avg=185.64, stdev=347.61 lat (msec): min=20, max=961, avg=185.64, stdev=347.61 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 29], 40.00th=[ 29], 50.00th=[ 29], 60.00th=[ 44], | 70.00th=[ 44], 80.00th=[ 194], 90.00th=[ 961], 95.00th=[ 961], | 99.00th=[ 961], 99.50th=[ 961], 99.90th=[ 961], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 4055, max= 6131, per=66.23%, avg=5093.00, stdev=1467.95, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 write: IOPS=1, BW=1923KiB/s (1969kB/s)(5120KiB/2663msec) clat (msec): min=59, max=696, avg=270.67, stdev=256.60 lat (msec): min=59, max=696, avg=270.75, stdev=256.61 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 60], 10.00th=[ 60], 20.00th=[ 60], | 30.00th=[ 132], 40.00th=[ 132], 50.00th=[ 146], 60.00th=[ 146], | 70.00th=[ 321], 80.00th=[ 321], 90.00th=[ 701], 95.00th=[ 701], | 99.00th=[ 701], 99.50th=[ 701], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 1825, max= 3657, per=23.30%, avg=2508.33, stdev=1000.73, samples=3 iops : min= 1, max= 3, avg= 1.67, stdev= 1.15, samples=3 lat (msec) : 50=41.67%, 100=8.33%, 250=25.00%, 500=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=5.67%, ctx=86, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24592: Wed May 29 19:20:36 2024 read: IOPS=0, BW=772KiB/s (790kB/s)(2048KiB/2654msec) clat (msec): min=73, max=269, avg=171.89, stdev=138.56 lat (msec): min=73, max=269, avg=171.89, stdev=138.56 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 74], 20.00th=[ 74], | 30.00th=[ 74], 40.00th=[ 74], 50.00th=[ 74], 60.00th=[ 271], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 1917, max= 2043, per=25.75%, avg=1980.00, stdev=89.10, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3858KiB/s (3951kB/s)(10.0MiB/2654msec) clat (msec): min=14, max=1264, avg=230.73, stdev=414.42 lat (msec): min=14, max=1264, avg=230.80, stdev=414.43 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 19], | 30.00th=[ 26], 40.00th=[ 27], 50.00th=[ 40], 60.00th=[ 51], | 70.00th=[ 140], 80.00th=[ 140], 90.00th=[ 667], 95.00th=[ 1267], | 99.00th=[ 1267], 99.50th=[ 1267], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 4096, max= 7670, per=54.64%, avg=5883.00, stdev=2527.20, samples=2 iops : min= 4, max= 7, avg= 5.50, stdev= 2.12, samples=2 lat (msec) : 20=16.67%, 50=25.00%, 100=25.00%, 250=8.33%, 500=8.33% lat (msec) : 750=8.33% cpu : usr=0.08%, sys=4.26%, ctx=112, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24594: Wed May 29 19:20:36 2024 read: IOPS=2, BW=2424KiB/s (2482kB/s)(6144KiB/2535msec) clat (msec): min=17, max=519, avg=173.08, stdev=225.92 lat (msec): min=17, max=519, avg=173.08, stdev=225.92 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 23], | 30.00th=[ 23], 40.00th=[ 34], 50.00th=[ 34], 60.00th=[ 45], | 70.00th=[ 401], 80.00th=[ 401], 90.00th=[ 518], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 4055, max= 4096, per=53.00%, avg=4075.50, stdev=28.99, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2424KiB/s (2482kB/s)(6144KiB/2535msec) clat (msec): min=25, max=971, avg=248.90, stdev=365.13 lat (msec): min=25, max=971, avg=248.99, stdev=365.16 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 45], | 30.00th=[ 45], 40.00th=[ 64], 50.00th=[ 64], 60.00th=[ 115], | 70.00th=[ 275], 80.00th=[ 275], 90.00th=[ 969], 95.00th=[ 969], | 99.00th=[ 969], 99.50th=[ 969], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 2027, max= 6144, per=31.64%, avg=3406.33, stdev=2370.91, samples=3 iops : min= 1, max= 6, avg= 3.00, stdev= 2.65, samples=3 lat (msec) : 20=8.33%, 50=41.67%, 100=8.33%, 250=8.33%, 500=16.67% lat (msec) : 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=3.99%, ctx=57, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24597: Wed May 29 19:20:36 2024 read: IOPS=2, BW=2721KiB/s (2786kB/s)(5120KiB/1882msec) clat (msec): min=20, max=259, avg=81.19, stdev=101.79 lat (msec): min=20, max=259, avg=81.19, stdev=101.79 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 22], 40.00th=[ 22], 50.00th=[ 32], 60.00th=[ 32], | 70.00th=[ 73], 80.00th=[ 73], 90.00th=[ 259], 95.00th=[ 259], | 99.00th=[ 259], 99.50th=[ 259], 99.90th=[ 259], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3809KiB/s (3900kB/s)(7168KiB/1882msec) clat (msec): min=37, max=685, avg=209.08, stdev=222.64 lat (msec): min=38, max=685, avg=209.19, stdev=222.61 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 87], | 30.00th=[ 99], 40.00th=[ 99], 50.00th=[ 132], 60.00th=[ 150], | 70.00th=[ 150], 80.00th=[ 275], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 2027, max= 6131, per=31.60%, avg=3402.00, stdev=2363.41, samples=3 iops : min= 1, max= 5, avg= 2.67, stdev= 2.08, samples=3 lat (msec) : 50=33.33%, 100=25.00%, 250=16.67%, 500=16.67%, 750=8.33% cpu : usr=0.00%, sys=6.86%, ctx=68, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7691KiB/s (7875kB/s), 772KiB/s-2721KiB/s (790kB/s-2786kB/s), io=20.0MiB (20.0MB), run=1882-2663msec WRITE: bw=10.5MiB/s (11.0MB/s), 1923KiB/s-3858KiB/s (1969kB/s-3951kB/s), io=28.0MiB (29.4MB), run=1882-2663msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24605: Wed May 29 19:20:38 2024 read: IOPS=4, BW=17.3MiB/s (18.2MB/s)(12.0MiB/692msec) clat (msec): min=47, max=567, avg=222.26, stdev=299.08 lat (msec): min=47, max=567, avg=222.26, stdev=299.08 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 52], 50.00th=[ 52], 60.00th=[ 52], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=33.33%, 750=33.33% cpu : usr=0.00%, sys=12.45%, ctx=40, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24606: Wed May 29 19:20:38 2024 read: IOPS=0, BW=4080KiB/s (4178kB/s)(4096KiB/1004msec) clat (nsec): min=266054k, max=266054k, avg=266053609.00, stdev= 0.00 lat (nsec): min=266055k, max=266055k, avg=266054986.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 266], 10.00th=[ 266], 20.00th=[ 266], | 30.00th=[ 266], 40.00th=[ 266], 50.00th=[ 266], 60.00th=[ 266], | 70.00th=[ 266], 80.00th=[ 266], 90.00th=[ 266], 95.00th=[ 266], | 99.00th=[ 266], 99.50th=[ 266], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 8143, max= 8143, per=49.90%, avg=8143.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=8159KiB/s (8355kB/s)(8192KiB/1004msec) clat (msec): min=50, max=685, avg=368.04, stdev=449.14 lat (msec): min=50, max=686, avg=368.40, stdev=449.27 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 51], 10.00th=[ 51], 20.00th=[ 51], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 684], | 70.00th=[ 684], 80.00th=[ 684], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] lat (msec) : 100=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=11.96%, ctx=38, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24607: Wed May 29 19:20:38 2024 write: IOPS=2, BW=11.9MiB/s (12.4MB/s)(12.0MiB/1012msec) clat (msec): min=220, max=554, avg=335.40, stdev=189.87 lat (msec): min=220, max=554, avg=335.72, stdev=189.91 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 222], 10.00th=[ 222], 20.00th=[ 222], | 30.00th=[ 222], 40.00th=[ 232], 50.00th=[ 232], 60.00th=[ 232], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min=16351, max=16351, per=50.50%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=10.78%, ctx=57, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24609: Wed May 29 19:20:38 2024 write: IOPS=4, BW=16.9MiB/s (17.7MB/s)(12.0MiB/709msec) clat (msec): min=33, max=392, avg=234.57, stdev=183.46 lat (msec): min=33, max=393, avg=234.90, stdev=183.50 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34], | 30.00th=[ 34], 40.00th=[ 279], 50.00th=[ 279], 60.00th=[ 279], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 8031, max= 8031, per=24.80%, avg=8031.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 500=66.67% cpu : usr=0.28%, sys=20.48%, ctx=71, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.9MiB/s (16.7MB/s), 4080KiB/s-17.3MiB/s (4178kB/s-18.2MB/s), io=16.0MiB (16.8MB), run=692-1004msec WRITE: bw=31.6MiB/s (33.2MB/s), 8159KiB/s-16.9MiB/s (8355kB/s-17.7MB/s), io=32.0MiB (33.6MB), run=709-1012msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24608: Wed May 29 19:20:38 2024 read: IOPS=4, BW=19.1MiB/s (20.0MB/s)(12.0MiB/629msec) clat (msec): min=120, max=320, avg=209.41, stdev=101.73 lat (msec): min=120, max=320, avg=209.42, stdev=101.73 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 121], 10.00th=[ 121], 20.00th=[ 121], | 30.00th=[ 121], 40.00th=[ 188], 50.00th=[ 188], 60.00th=[ 188], | 70.00th=[ 321], 80.00th=[ 321], 90.00th=[ 321], 95.00th=[ 321], | 99.00th=[ 321], 99.50th=[ 321], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min=15784, max=15784, per=83.53%, avg=15784.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=51.43%, ctx=280, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24610: Wed May 29 19:20:38 2024 read: IOPS=1, BW=4724KiB/s (4838kB/s)(4096KiB/867msec) clat (nsec): min=171288k, max=171288k, avg=171288254.00, stdev= 0.00 lat (nsec): min=171290k, max=171290k, avg=171289798.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 171], 50.00th=[ 171], 60.00th=[ 171], | 70.00th=[ 171], 80.00th=[ 171], 90.00th=[ 171], 95.00th=[ 171], | 99.00th=[ 171], 99.50th=[ 171], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 8126, max= 8126, per=43.00%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9449KiB/s (9675kB/s)(8192KiB/867msec) clat (msec): min=96, max=596, avg=346.44, stdev=352.97 lat (msec): min=97, max=596, avg=346.69, stdev=352.98 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 97], 10.00th=[ 97], 20.00th=[ 97], | 30.00th=[ 97], 40.00th=[ 97], 50.00th=[ 97], 60.00th=[ 600], | 70.00th=[ 600], 80.00th=[ 600], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] lat (msec) : 100=33.33%, 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=33.26%, ctx=114, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24611: Wed May 29 19:20:38 2024 write: IOPS=2, BW=11.1MiB/s (11.6MB/s)(12.0MiB/1083msec) clat (msec): min=183, max=538, avg=355.43, stdev=178.17 lat (msec): min=183, max=539, avg=355.82, stdev=178.13 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 347], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 8175, max= 8175, per=27.02%, avg=8175.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.09%, sys=29.02%, ctx=34, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24612: Wed May 29 19:20:38 2024 write: IOPS=6, BW=25.5MiB/s (26.7MB/s)(12.0MiB/471msec) clat (msec): min=126, max=183, avg=154.51, stdev=28.72 lat (msec): min=126, max=189, avg=156.53, stdev=31.37 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 127], 10.00th=[ 127], 20.00th=[ 127], | 30.00th=[ 127], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 155], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] lat (msec) : 250=100.00% cpu : usr=0.21%, sys=61.06%, ctx=45, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.5MiB/s (19.3MB/s), 4724KiB/s-19.1MiB/s (4838kB/s-20.0MB/s), io=16.0MiB (16.8MB), run=629-867msec WRITE: bw=29.5MiB/s (30.0MB/s), 9449KiB/s-25.5MiB/s (9675kB/s-26.7MB/s), io=32.0MiB (33.6MB), run=471-1083msec