== sanity test 398b: DIO and buffer IO race ============== 23:05:25 (1723259125) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.24956 s, 40.3 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20989: Fri Aug 9 23:05:44 2024 read: IOPS=94, BW=379KiB/s (388kB/s)(5992KiB/15819msec) clat (usec): min=47, max=194594, avg=4502.18, stdev=6606.96 lat (usec): min=47, max=194595, avg=4503.64, stdev=6607.50 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 61], 10.00th=[ 73], 20.00th=[ 98], | 30.00th=[ 2147], 40.00th=[ 3064], 50.00th=[ 3720], 60.00th=[ 4490], | 70.00th=[ 5604], 80.00th=[ 7308], 90.00th=[ 10028], 95.00th=[ 11994], | 99.00th=[ 15795], 99.50th=[ 17957], 99.90th=[ 81265], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 232, max= 542, per=24.50%, avg=371.61, stdev=73.53, samples=31 iops : min= 58, max= 135, avg=92.74, stdev=18.36, samples=31 write: IOPS=99, BW=398KiB/s (408kB/s)(6296KiB/15819msec) clat (usec): min=399, max=48844, avg=5702.72, stdev=5633.49 lat (usec): min=400, max=48845, avg=5703.81, stdev=5633.35 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 545], 10.00th=[ 611], 20.00th=[ 824], | 30.00th=[ 1418], 40.00th=[ 2966], 50.00th=[ 4424], 60.00th=[ 5538], | 70.00th=[ 7177], 80.00th=[ 9503], 90.00th=[13042], 95.00th=[16909], | 99.00th=[24511], 99.50th=[27919], 99.90th=[47973], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 280, max= 558, per=24.59%, avg=390.71, stdev=65.12, samples=31 iops : min= 70, max= 139, avg=97.52, stdev=16.27, samples=31 lat (usec) : 50=0.46%, 100=9.70%, 250=3.48%, 500=0.72%, 750=8.14% lat (usec) : 1000=4.92% lat (msec) : 2=4.43%, 4=18.03%, 10=35.94%, 20=12.92%, 50=1.20% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.19%, sys=24.14%, ctx=6175, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20990: Fri Aug 9 23:05:44 2024 read: IOPS=94, BW=377KiB/s (386kB/s)(5940KiB/15744msec) clat (usec): min=47, max=59841, avg=4372.63, stdev=4359.38 lat (usec): min=47, max=59841, avg=4374.70, stdev=4359.48 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 60], 10.00th=[ 73], 20.00th=[ 98], | 30.00th=[ 2040], 40.00th=[ 2933], 50.00th=[ 3720], 60.00th=[ 4490], | 70.00th=[ 5473], 80.00th=[ 7242], 90.00th=[10028], 95.00th=[12387], | 99.00th=[15664], 99.50th=[19792], 99.90th=[31851], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 232, max= 512, per=24.87%, avg=377.32, stdev=64.49, samples=31 iops : min= 58, max= 128, avg=94.16, stdev=16.11, samples=31 write: IOPS=100, BW=403KiB/s (413kB/s)(6348KiB/15744msec) clat (usec): min=457, max=191674, avg=5768.61, stdev=7470.73 lat (usec): min=458, max=191675, avg=5769.90, stdev=7470.53 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 553], 10.00th=[ 619], 20.00th=[ 807], | 30.00th=[ 1221], 40.00th=[ 2966], 50.00th=[ 4490], 60.00th=[ 5800], | 70.00th=[ 7242], 80.00th=[ 9241], 90.00th=[ 12780], 95.00th=[ 16319], | 99.00th=[ 23725], 99.50th=[ 29230], 99.90th=[ 95945], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 320, max= 576, per=25.14%, avg=399.52, stdev=58.73, samples=31 iops : min= 80, max= 144, avg=99.71, stdev=14.69, samples=31 lat (usec) : 50=0.39%, 100=9.67%, 250=3.71%, 500=0.75%, 750=8.69% lat (usec) : 1000=4.95% lat (msec) : 2=4.59%, 4=16.86%, 10=36.33%, 20=12.79%, 50=1.17% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.17%, sys=24.33%, ctx=6258, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20991: Fri Aug 9 23:05:44 2024 read: IOPS=96, BW=385KiB/s (395kB/s)(6028KiB/15637msec) clat (usec): min=47, max=191854, avg=4464.00, stdev=6270.06 lat (usec): min=47, max=191855, avg=4465.17, stdev=6270.15 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 62], 10.00th=[ 74], 20.00th=[ 98], | 30.00th=[ 2278], 40.00th=[ 3097], 50.00th=[ 3785], 60.00th=[ 4621], | 70.00th=[ 5473], 80.00th=[ 7242], 90.00th=[ 9765], 95.00th=[ 11731], | 99.00th=[ 16057], 99.50th=[ 19268], 99.90th=[ 32637], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 271, max= 552, per=25.33%, avg=384.29, stdev=77.75, samples=31 iops : min= 67, max= 138, avg=95.90, stdev=19.46, samples=31 write: IOPS=100, BW=400KiB/s (410kB/s)(6260KiB/15637msec) clat (usec): min=451, max=79082, avg=5630.56, stdev=5892.29 lat (usec): min=452, max=79084, avg=5631.54, stdev=5892.20 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 545], 10.00th=[ 611], 20.00th=[ 791], | 30.00th=[ 1057], 40.00th=[ 2802], 50.00th=[ 4490], 60.00th=[ 5538], | 70.00th=[ 6849], 80.00th=[ 9241], 90.00th=[13435], 95.00th=[16450], | 99.00th=[23987], 99.50th=[26870], 99.90th=[49021], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 304, max= 528, per=25.04%, avg=397.94, stdev=58.14, samples=31 iops : min= 76, max= 132, avg=99.32, stdev=14.48, samples=31 lat (usec) : 50=0.59%, 100=9.77%, 250=3.19%, 500=0.78%, 750=8.59% lat (usec) : 1000=5.76% lat (msec) : 2=4.04%, 4=17.19%, 10=36.72%, 20=11.82%, 50=1.50% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.15%, sys=24.06%, ctx=6109, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20992: Fri Aug 9 23:05:44 2024 read: IOPS=96, BW=387KiB/s (397kB/s)(6052KiB/15628msec) clat (usec): min=47, max=24810, avg=4280.26, stdev=3986.46 lat (usec): min=47, max=24811, avg=4281.22, stdev=3986.84 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 60], 10.00th=[ 72], 20.00th=[ 95], | 30.00th=[ 2114], 40.00th=[ 3032], 50.00th=[ 3720], 60.00th=[ 4490], | 70.00th=[ 5342], 80.00th=[ 7111], 90.00th=[10159], 95.00th=[12125], | 99.00th=[16188], 99.50th=[18744], 99.90th=[23987], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 240, max= 510, per=25.35%, avg=384.52, stdev=78.08, samples=31 iops : min= 60, max= 127, avg=95.97, stdev=19.51, samples=31 write: IOPS=99, BW=399KiB/s (409kB/s)(6236KiB/15628msec) clat (usec): min=463, max=192043, avg=5805.25, stdev=7386.64 lat (usec): min=463, max=192044, avg=5806.51, stdev=7386.39 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 553], 10.00th=[ 619], 20.00th=[ 816], | 30.00th=[ 1287], 40.00th=[ 2835], 50.00th=[ 4490], 60.00th=[ 5604], | 70.00th=[ 7111], 80.00th=[ 9372], 90.00th=[ 13304], 95.00th=[ 16909], | 99.00th=[ 23987], 99.50th=[ 26870], 99.90th=[ 65799], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 264, max= 550, per=24.98%, avg=396.90, stdev=60.50, samples=31 iops : min= 66, max= 137, avg=99.06, stdev=15.10, samples=31 lat (usec) : 50=0.65%, 100=9.83%, 250=3.71%, 500=0.78%, 750=8.30% lat (usec) : 1000=4.95% lat (msec) : 2=4.52%, 4=17.90%, 10=35.12%, 20=12.63%, 50=1.53% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.13%, sys=23.98%, ctx=6206, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1518KiB/s (1554kB/s), 377KiB/s-387KiB/s (386kB/s-397kB/s), io=23.4MiB (24.6MB), run=15628-15819msec WRITE: bw=1589KiB/s (1627kB/s), 398KiB/s-403KiB/s (408kB/s-413kB/s), io=24.6MiB (25.7MB), run=15628-15819msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20993: Fri Aug 9 23:06:17 2024 read: IOPS=30, BW=122KiB/s (125kB/s)(5992KiB/49023msec) clat (usec): min=1732, max=194425, avg=4239.14, stdev=6520.38 lat (usec): min=1733, max=194427, avg=4240.06, stdev=6520.38 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1942], 10.00th=[ 2040], 20.00th=[ 2180], | 30.00th=[ 2311], 40.00th=[ 2442], 50.00th=[ 2573], 60.00th=[ 2835], | 70.00th=[ 3130], 80.00th=[ 4178], 90.00th=[ 9110], 95.00th=[ 13173], | 99.00th=[ 21627], 99.50th=[ 25297], 99.90th=[ 57934], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 24, max= 224, per=25.16%, avg=122.26, stdev=53.09, samples=98 iops : min= 6, max= 56, avg=30.54, stdev=13.28, samples=98 write: IOPS=32, BW=128KiB/s (132kB/s)(6296KiB/49023msec) clat (usec): min=12770, max=83271, avg=27053.55, stdev=10896.60 lat (usec): min=12771, max=83273, avg=27054.56, stdev=10896.62 clat percentiles (usec): | 1.00th=[14222], 5.00th=[15533], 10.00th=[16319], 20.00th=[17957], | 30.00th=[20317], 40.00th=[22938], 50.00th=[25035], 60.00th=[26870], | 70.00th=[28705], 80.00th=[32375], 90.00th=[42206], 95.00th=[51643], | 99.00th=[64226], 99.50th=[68682], 99.90th=[79168], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 40, max= 184, per=25.22%, avg=128.38, stdev=40.70, samples=98 iops : min= 10, max= 46, avg=32.07, stdev=10.19, samples=98 lat (msec) : 2=3.74%, 4=34.77%, 10=6.09%, 20=18.55%, 50=33.82% lat (msec) : 100=2.99%, 250=0.03% cpu : usr=0.09%, sys=4.15%, ctx=3361, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20994: Fri Aug 9 23:06:17 2024 read: IOPS=30, BW=120KiB/s (123kB/s)(5940KiB/49389msec) clat (usec): min=1685, max=190085, avg=4172.75, stdev=6317.21 lat (usec): min=1685, max=190086, avg=4173.60, stdev=6317.21 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1958], 10.00th=[ 2057], 20.00th=[ 2180], | 30.00th=[ 2311], 40.00th=[ 2442], 50.00th=[ 2573], 60.00th=[ 2769], | 70.00th=[ 3064], 80.00th=[ 4293], 90.00th=[ 8717], 95.00th=[ 12518], | 99.00th=[ 19268], 99.50th=[ 23725], 99.90th=[ 55313], 99.95th=[189793], | 99.99th=[189793] bw ( KiB/s): min= 16, max= 272, per=24.70%, avg=120.04, stdev=57.97, samples=98 iops : min= 4, max= 68, avg=29.98, stdev=14.51, samples=98 write: IOPS=32, BW=129KiB/s (132kB/s)(6348KiB/49389msec) clat (usec): min=12963, max=94874, avg=27158.99, stdev=11141.24 lat (usec): min=12964, max=94875, avg=27160.06, stdev=11141.19 clat percentiles (usec): | 1.00th=[14353], 5.00th=[15533], 10.00th=[16450], 20.00th=[18220], | 30.00th=[20317], 40.00th=[22938], 50.00th=[25035], 60.00th=[26608], | 70.00th=[28705], 80.00th=[32113], 90.00th=[44303], 95.00th=[51119], | 99.00th=[66847], 99.50th=[70779], 99.90th=[86508], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 48, max= 184, per=25.09%, avg=127.71, stdev=40.59, samples=98 iops : min= 12, max= 46, avg=31.90, stdev=10.16, samples=98 lat (msec) : 2=3.45%, 4=34.60%, 10=6.45%, 20=18.03%, 50=34.54% lat (msec) : 100=2.90%, 250=0.03% cpu : usr=0.06%, sys=4.09%, ctx=3383, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20995: Fri Aug 9 23:06:17 2024 read: IOPS=30, BW=123KiB/s (126kB/s)(6028KiB/48897msec) clat (usec): min=1719, max=35816, avg=4316.83, stdev=4072.25 lat (usec): min=1719, max=35817, avg=4317.72, stdev=4072.27 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1975], 10.00th=[ 2073], 20.00th=[ 2180], | 30.00th=[ 2343], 40.00th=[ 2474], 50.00th=[ 2638], 60.00th=[ 2868], | 70.00th=[ 3326], 80.00th=[ 5211], 90.00th=[10028], 95.00th=[13042], | 99.00th=[20317], 99.50th=[25297], 99.90th=[30278], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 32, max= 280, per=25.33%, avg=123.10, stdev=49.49, samples=97 iops : min= 8, max= 70, avg=30.69, stdev=12.36, samples=97 write: IOPS=32, BW=128KiB/s (131kB/s)(6260KiB/48897msec) clat (msec): min=12, max=200, avg=27.03, stdev=11.06 lat (msec): min=12, max=200, avg=27.03, stdev=11.06 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 19], | 30.00th=[ 21], 40.00th=[ 24], 50.00th=[ 26], 60.00th=[ 27], | 70.00th=[ 29], 80.00th=[ 33], 90.00th=[ 42], 95.00th=[ 50], | 99.00th=[ 61], 99.50th=[ 65], 99.90th=[ 81], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 48, max= 184, per=25.06%, avg=127.56, stdev=39.69, samples=97 iops : min= 12, max= 46, avg=31.80, stdev= 9.95, samples=97 lat (msec) : 2=2.99%, 4=34.15%, 10=7.00%, 20=18.33%, 50=35.19% lat (msec) : 100=2.31%, 250=0.03% cpu : usr=0.09%, sys=4.15%, ctx=3427, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20996: Fri Aug 9 23:06:17 2024 read: IOPS=30, BW=123KiB/s (126kB/s)(6052KiB/49176msec) clat (usec): min=1650, max=57852, avg=4008.65, stdev=3974.87 lat (usec): min=1650, max=57853, avg=4009.52, stdev=3974.85 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 1942], 10.00th=[ 2040], 20.00th=[ 2180], | 30.00th=[ 2311], 40.00th=[ 2409], 50.00th=[ 2573], 60.00th=[ 2769], | 70.00th=[ 3130], 80.00th=[ 4178], 90.00th=[ 8717], 95.00th=[12387], | 99.00th=[18744], 99.50th=[22152], 99.90th=[42206], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 8, max= 248, per=25.30%, avg=122.95, stdev=53.85, samples=98 iops : min= 2, max= 62, avg=30.62, stdev=13.44, samples=98 write: IOPS=31, BW=127KiB/s (130kB/s)(6236KiB/49176msec) clat (msec): min=12, max=177, avg=27.58, stdev=11.75 lat (msec): min=12, max=177, avg=27.58, stdev=11.75 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 19], | 30.00th=[ 21], 40.00th=[ 24], 50.00th=[ 26], 60.00th=[ 28], | 70.00th=[ 29], 80.00th=[ 34], 90.00th=[ 44], 95.00th=[ 52], | 99.00th=[ 68], 99.50th=[ 72], 99.90th=[ 95], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 48, max= 176, per=24.81%, avg=126.30, stdev=39.53, samples=98 iops : min= 12, max= 44, avg=31.46, stdev= 9.88, samples=98 lat (msec) : 2=3.81%, 4=35.19%, 10=6.35%, 20=17.15%, 50=34.64% lat (msec) : 100=2.83%, 250=0.03% cpu : usr=0.09%, sys=4.11%, ctx=3398, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=486KiB/s (498kB/s), 120KiB/s-123KiB/s (123kB/s-126kB/s), io=23.4MiB (24.6MB), run=48897-49389msec WRITE: bw=509KiB/s (521kB/s), 127KiB/s-129KiB/s (130kB/s-132kB/s), io=24.6MiB (25.7MB), run=48897-49389msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21045: Fri Aug 9 23:06:23 2024 read: IOPS=72, BW=1156KiB/s (1184kB/s)(5920KiB/5120msec) clat (usec): min=49, max=30821, avg=9806.04, stdev=6613.00 lat (usec): min=50, max=30822, avg=9806.72, stdev=6613.01 clat percentiles (usec): | 1.00th=[ 54], 5.00th=[ 63], 10.00th=[ 75], 20.00th=[ 99], | 30.00th=[ 8455], 40.00th=[10290], 50.00th=[11076], 60.00th=[12387], | 70.00th=[13435], 80.00th=[14877], 90.00th=[16581], 95.00th=[18220], | 99.00th=[26084], 99.50th=[30016], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 734, max= 1548, per=24.30%, avg=1136.70, stdev=233.25, samples=10 iops : min= 45, max= 96, avg=70.80, stdev=14.51, samples=10 write: IOPS=77, BW=1244KiB/s (1274kB/s)(6368KiB/5120msec) clat (usec): min=665, max=41606, avg=3600.71, stdev=4747.12 lat (usec): min=665, max=41607, avg=3601.77, stdev=4747.12 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 742], 10.00th=[ 807], 20.00th=[ 930], | 30.00th=[ 1074], 40.00th=[ 1205], 50.00th=[ 1418], 60.00th=[ 2737], | 70.00th=[ 3785], 80.00th=[ 5473], 90.00th=[ 7963], 95.00th=[11469], | 99.00th=[26084], 99.50th=[35914], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 576, max= 2048, per=25.03%, avg=1231.50, stdev=401.17, samples=10 iops : min= 36, max= 128, avg=76.70, stdev=25.17, samples=10 lat (usec) : 50=0.13%, 100=9.77%, 250=2.73%, 500=0.13%, 750=2.99% lat (usec) : 1000=10.03% lat (msec) : 2=16.54%, 4=7.68%, 10=15.89%, 20=31.90%, 50=2.21% cpu : usr=0.10%, sys=23.27%, ctx=2077, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21047: Fri Aug 9 23:06:23 2024 read: IOPS=72, BW=1157KiB/s (1184kB/s)(5744KiB/4966msec) clat (usec): min=50, max=41413, avg=9891.07, stdev=6925.37 lat (usec): min=51, max=41413, avg=9891.91, stdev=6925.48 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 63], 10.00th=[ 77], 20.00th=[ 108], | 30.00th=[ 8586], 40.00th=[10159], 50.00th=[11469], 60.00th=[12518], | 70.00th=[13698], 80.00th=[14746], 90.00th=[16581], 95.00th=[17957], | 99.00th=[30540], 99.50th=[36439], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 800, max= 1600, per=24.85%, avg=1162.33, stdev=274.99, samples=9 iops : min= 50, max= 100, avg=72.56, stdev=17.11, samples=9 write: IOPS=82, BW=1318KiB/s (1349kB/s)(6544KiB/4966msec) clat (usec): min=627, max=27376, avg=3388.96, stdev=3504.48 lat (usec): min=628, max=27377, avg=3390.14, stdev=3504.56 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 742], 10.00th=[ 824], 20.00th=[ 938], | 30.00th=[ 1123], 40.00th=[ 1287], 50.00th=[ 1663], 60.00th=[ 3163], | 70.00th=[ 4293], 80.00th=[ 5211], 90.00th=[ 7635], 95.00th=[ 9896], | 99.00th=[17695], 99.50th=[19006], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 960, max= 1948, per=26.51%, avg=1304.44, stdev=302.00, samples=9 iops : min= 60, max= 121, avg=81.44, stdev=18.68, samples=9 lat (usec) : 100=7.68%, 250=4.30%, 750=3.12%, 1000=9.64% lat (msec) : 2=15.76%, 4=8.20%, 10=19.14%, 20=30.60%, 50=1.56% cpu : usr=0.16%, sys=24.09%, ctx=2111, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21049: Fri Aug 9 23:06:23 2024 read: IOPS=76, BW=1226KiB/s (1255kB/s)(6256KiB/5104msec) clat (usec): min=49, max=23891, avg=9962.06, stdev=6431.07 lat (usec): min=49, max=23891, avg=9962.78, stdev=6431.04 clat percentiles (usec): | 1.00th=[ 54], 5.00th=[ 62], 10.00th=[ 65], 20.00th=[ 105], | 30.00th=[ 9241], 40.00th=[10552], 50.00th=[11600], 60.00th=[12518], | 70.00th=[13566], 80.00th=[14877], 90.00th=[16909], 95.00th=[18744], | 99.00th=[22676], 99.50th=[23200], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 896, max= 1756, per=26.12%, avg=1222.00, stdev=288.08, samples=10 iops : min= 56, max= 109, avg=76.30, stdev=17.85, samples=10 write: IOPS=73, BW=1182KiB/s (1210kB/s)(6032KiB/5104msec) clat (usec): min=632, max=23354, avg=3082.07, stdev=3000.80 lat (usec): min=633, max=23355, avg=3083.23, stdev=3000.84 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 734], 10.00th=[ 799], 20.00th=[ 898], | 30.00th=[ 1057], 40.00th=[ 1188], 50.00th=[ 1369], 60.00th=[ 2802], | 70.00th=[ 3720], 80.00th=[ 5604], 90.00th=[ 7570], 95.00th=[ 8586], | 99.00th=[12518], 99.50th=[15139], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 672, max= 1692, per=23.66%, avg=1164.40, stdev=302.13, samples=10 iops : min= 42, max= 105, avg=72.70, stdev=18.74, samples=10 lat (usec) : 50=0.13%, 100=9.24%, 250=2.86%, 500=0.52%, 750=2.86% lat (usec) : 1000=10.16% lat (msec) : 2=14.97%, 4=8.07%, 10=16.67%, 20=32.42%, 50=2.08% cpu : usr=0.18%, sys=23.36%, ctx=2154, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21051: Fri Aug 9 23:06:23 2024 read: IOPS=74, BW=1192KiB/s (1220kB/s)(6032KiB/5062msec) clat (usec): min=50, max=26822, avg=9498.69, stdev=6670.40 lat (usec): min=51, max=26823, avg=9499.40, stdev=6670.32 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 62], 10.00th=[ 73], 20.00th=[ 98], | 30.00th=[ 8225], 40.00th=[10290], 50.00th=[11338], 60.00th=[12518], | 70.00th=[13435], 80.00th=[14484], 90.00th=[16712], 95.00th=[18744], | 99.00th=[23462], 99.50th=[25297], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 864, max= 1405, per=24.96%, avg=1167.70, stdev=171.03, samples=10 iops : min= 54, max= 87, avg=72.90, stdev=10.57, samples=10 write: IOPS=77, BW=1236KiB/s (1266kB/s)(6256KiB/5062msec) clat (usec): min=643, max=40177, avg=3751.66, stdev=4657.10 lat (usec): min=644, max=40178, avg=3753.84, stdev=4656.62 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 906], | 30.00th=[ 1074], 40.00th=[ 1287], 50.00th=[ 1778], 60.00th=[ 3294], | 70.00th=[ 4228], 80.00th=[ 5735], 90.00th=[ 7570], 95.00th=[10552], | 99.00th=[25035], 99.50th=[34866], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 928, max= 1728, per=24.77%, avg=1218.90, stdev=290.10, samples=10 iops : min= 58, max= 108, avg=76.10, stdev=18.17, samples=10 lat (usec) : 100=10.42%, 250=3.65%, 750=2.99%, 1000=9.90% lat (msec) : 2=13.28%, 4=8.07%, 10=18.36%, 20=30.47%, 50=2.86% cpu : usr=0.12%, sys=23.08%, ctx=2133, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4678KiB/s (4790kB/s), 1156KiB/s-1226KiB/s (1184kB/s-1255kB/s), io=23.4MiB (24.5MB), run=4966-5120msec WRITE: bw=4922KiB/s (5040kB/s), 1182KiB/s-1318KiB/s (1210kB/s-1349kB/s), io=24.6MiB (25.8MB), run=4966-5120msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21046: Fri Aug 9 23:06:30 2024 read: IOPS=30, BW=495KiB/s (507kB/s)(5920KiB/11948msec) clat (usec): min=1940, max=26600, avg=4614.63, stdev=4114.45 lat (usec): min=1941, max=26601, avg=4615.46, stdev=4114.43 clat percentiles (usec): | 1.00th=[ 2089], 5.00th=[ 2212], 10.00th=[ 2311], 20.00th=[ 2507], | 30.00th=[ 2671], 40.00th=[ 2835], 50.00th=[ 2999], 60.00th=[ 3326], | 70.00th=[ 3752], 80.00th=[ 5014], 90.00th=[ 8979], 95.00th=[14615], | 99.00th=[22676], 99.50th=[25297], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 319, max= 896, per=24.70%, avg=485.43, stdev=132.58, samples=23 iops : min= 19, max= 56, avg=30.22, stdev= 8.35, samples=23 write: IOPS=33, BW=533KiB/s (546kB/s)(6368KiB/11948msec) clat (usec): min=13344, max=55261, avg=25689.15, stdev=7621.90 lat (usec): min=13346, max=55262, avg=25690.73, stdev=7621.91 clat percentiles (usec): | 1.00th=[14484], 5.00th=[15926], 10.00th=[16581], 20.00th=[18482], | 30.00th=[21365], 40.00th=[23200], 50.00th=[24511], 60.00th=[26608], | 70.00th=[28443], 80.00th=[30802], 90.00th=[36439], 95.00th=[40633], | 99.00th=[50594], 99.50th=[52691], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 320, max= 704, per=25.69%, avg=531.26, stdev=126.40, samples=23 iops : min= 20, max= 44, avg=33.09, stdev= 7.92, samples=23 lat (msec) : 2=0.13%, 4=35.29%, 10=8.33%, 20=16.41%, 50=39.19% lat (msec) : 100=0.65% cpu : usr=0.08%, sys=5.19%, ctx=868, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21048: Fri Aug 9 23:06:30 2024 read: IOPS=29, BW=471KiB/s (483kB/s)(5744KiB/12184msec) clat (usec): min=1992, max=33362, avg=4508.47, stdev=4200.05 lat (usec): min=1992, max=33363, avg=4509.34, stdev=4200.02 clat percentiles (usec): | 1.00th=[ 2057], 5.00th=[ 2212], 10.00th=[ 2343], 20.00th=[ 2507], | 30.00th=[ 2638], 40.00th=[ 2737], 50.00th=[ 2900], 60.00th=[ 3130], | 70.00th=[ 3621], 80.00th=[ 5276], 90.00th=[ 8848], 95.00th=[13435], | 99.00th=[22676], 99.50th=[31327], 99.90th=[33424], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 128, max= 832, per=23.87%, avg=469.08, stdev=191.61, samples=24 iops : min= 8, max= 52, avg=29.17, stdev=11.97, samples=24 write: IOPS=33, BW=537KiB/s (550kB/s)(6544KiB/12184msec) clat (usec): min=13166, max=63745, avg=25683.33, stdev=8035.20 lat (usec): min=13167, max=63747, avg=25684.96, stdev=8035.18 clat percentiles (usec): | 1.00th=[13698], 5.00th=[15270], 10.00th=[16188], 20.00th=[17957], | 30.00th=[21103], 40.00th=[23462], 50.00th=[25035], 60.00th=[26870], | 70.00th=[28181], 80.00th=[31065], 90.00th=[35390], 95.00th=[41157], | 99.00th=[51643], 99.50th=[54264], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 352, max= 704, per=25.58%, avg=529.08, stdev=126.15, samples=24 iops : min= 22, max= 44, avg=32.92, stdev= 7.88, samples=24 lat (msec) : 2=0.13%, 4=34.77%, 10=7.81%, 20=17.97%, 50=38.54% lat (msec) : 100=0.78% cpu : usr=0.09%, sys=5.18%, ctx=883, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21050: Fri Aug 9 23:06:30 2024 read: IOPS=33, BW=540KiB/s (553kB/s)(6256KiB/11587msec) clat (usec): min=2002, max=26927, avg=4499.83, stdev=3790.20 lat (usec): min=2003, max=26927, avg=4500.65, stdev=3790.22 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2245], 10.00th=[ 2343], 20.00th=[ 2540], | 30.00th=[ 2671], 40.00th=[ 2835], 50.00th=[ 2999], 60.00th=[ 3228], | 70.00th=[ 3851], 80.00th=[ 5604], 90.00th=[ 8291], 95.00th=[13042], | 99.00th=[20841], 99.50th=[24249], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 160, max= 1088, per=27.67%, avg=543.78, stdev=227.35, samples=23 iops : min= 10, max= 68, avg=33.87, stdev=14.27, samples=23 write: IOPS=32, BW=521KiB/s (533kB/s)(6032KiB/11587msec) clat (usec): min=13521, max=69061, avg=25864.50, stdev=8095.39 lat (usec): min=13523, max=69062, avg=25866.14, stdev=8095.36 clat percentiles (usec): | 1.00th=[14615], 5.00th=[15795], 10.00th=[16581], 20.00th=[18744], | 30.00th=[20841], 40.00th=[22676], 50.00th=[24511], 60.00th=[26870], | 70.00th=[28705], 80.00th=[31589], 90.00th=[36439], 95.00th=[41681], | 99.00th=[45876], 99.50th=[66323], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 351, max= 734, per=25.08%, avg=518.74, stdev=123.00, samples=23 iops : min= 21, max= 45, avg=32.30, stdev= 7.65, samples=23 lat (msec) : 4=35.81%, 10=11.20%, 20=16.41%, 50=36.20%, 100=0.39% cpu : usr=0.16%, sys=5.28%, ctx=879, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21052: Fri Aug 9 23:06:30 2024 read: IOPS=31, BW=504KiB/s (516kB/s)(6032KiB/11973msec) clat (usec): min=1794, max=33588, avg=4629.97, stdev=4195.20 lat (usec): min=1794, max=33588, avg=4630.80, stdev=4195.19 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2245], 10.00th=[ 2376], 20.00th=[ 2540], | 30.00th=[ 2704], 40.00th=[ 2802], 50.00th=[ 2966], 60.00th=[ 3228], | 70.00th=[ 3785], 80.00th=[ 5735], 90.00th=[ 9110], 95.00th=[13304], | 99.00th=[21627], 99.50th=[29230], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 224, max= 864, per=25.21%, avg=495.30, stdev=186.31, samples=23 iops : min= 14, max= 54, avg=30.96, stdev=11.64, samples=23 write: IOPS=32, BW=523KiB/s (535kB/s)(6256KiB/11973msec) clat (usec): min=13566, max=54942, avg=25960.65, stdev=7632.75 lat (usec): min=13568, max=54944, avg=25962.24, stdev=7632.84 clat percentiles (usec): | 1.00th=[13829], 5.00th=[15664], 10.00th=[17171], 20.00th=[19006], | 30.00th=[20841], 40.00th=[23462], 50.00th=[25297], 60.00th=[27132], | 70.00th=[28967], 80.00th=[31327], 90.00th=[36439], 95.00th=[40633], | 99.00th=[51643], 99.50th=[54789], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 288, max= 704, per=25.03%, avg=517.57, stdev=125.67, samples=23 iops : min= 18, max= 44, avg=32.35, stdev= 7.85, samples=23 lat (msec) : 2=0.52%, 4=34.90%, 10=9.24%, 20=16.67%, 50=38.15% lat (msec) : 100=0.52% cpu : usr=0.08%, sys=5.25%, ctx=891, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1966KiB/s (2013kB/s), 471KiB/s-540KiB/s (483kB/s-553kB/s), io=23.4MiB (24.5MB), run=11587-12184msec WRITE: bw=2068KiB/s (2118kB/s), 521KiB/s-537KiB/s (533kB/s-550kB/s), io=24.6MiB (25.8MB), run=11587-12184msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21071: Fri Aug 9 23:06:33 2024 read: IOPS=6, BW=6731KiB/s (6892kB/s)(7168KiB/1065msec) clat (usec): min=15534, max=40944, avg=23734.45, stdev=10055.25 lat (usec): min=15536, max=40945, avg=23736.01, stdev=10055.11 clat percentiles (usec): | 1.00th=[15533], 5.00th=[15533], 10.00th=[15533], 20.00th=[16057], | 30.00th=[16581], 40.00th=[16581], 50.00th=[17695], 60.00th=[25822], | 70.00th=[25822], 80.00th=[33424], 90.00th=[41157], 95.00th=[41157], | 99.00th=[41157], 99.50th=[41157], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 2048, max=10240, per=57.87%, avg=6144.00, stdev=5792.62, samples=2 iops : min= 2, max= 10, avg= 6.00, stdev= 5.66, samples=2 write: IOPS=4, BW=4808KiB/s (4923kB/s)(5120KiB/1065msec) clat (msec): min=53, max=637, avg=174.50, stdev=258.99 lat (msec): min=54, max=637, avg=174.69, stdev=258.91 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 54], 10.00th=[ 54], 20.00th=[ 54], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 62], 60.00th=[ 62], | 70.00th=[ 63], 80.00th=[ 63], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 2048, max= 6144, per=27.56%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=33.33%, 50=25.00%, 100=33.33%, 750=8.33% cpu : usr=0.09%, sys=9.30%, ctx=48, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21072: Fri Aug 9 23:06:33 2024 read: IOPS=1, BW=1062KiB/s (1087kB/s)(2048KiB/1929msec) clat (usec): min=23668, max=83835, avg=53751.77, stdev=42544.88 lat (usec): min=23668, max=83836, avg=53752.89, stdev=42545.19 clat percentiles (usec): | 1.00th=[23725], 5.00th=[23725], 10.00th=[23725], 20.00th=[23725], | 30.00th=[23725], 40.00th=[23725], 50.00th=[23725], 60.00th=[83362], | 70.00th=[83362], 80.00th=[83362], 90.00th=[83362], 95.00th=[83362], | 99.00th=[83362], 99.50th=[83362], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 2048, max= 2048, per=19.29%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5308KiB/s (5436kB/s)(10.0MiB/1929msec) clat (msec): min=30, max=653, avg=178.80, stdev=233.10 lat (msec): min=30, max=653, avg=178.88, stdev=233.09 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 41], | 30.00th=[ 49], 40.00th=[ 57], 50.00th=[ 66], 60.00th=[ 77], | 70.00th=[ 167], 80.00th=[ 167], 90.00th=[ 575], 95.00th=[ 651], | 99.00th=[ 651], 99.50th=[ 651], 99.90th=[ 651], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 2048, max= 6144, per=27.56%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (msec) : 50=33.33%, 100=41.67%, 250=8.33%, 750=16.67% cpu : usr=0.05%, sys=6.95%, ctx=228, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21073: Fri Aug 9 23:06:33 2024 read: IOPS=3, BW=3346KiB/s (3427kB/s)(6144KiB/1836msec) clat (msec): min=14, max=567, avg=112.41, stdev=222.95 lat (msec): min=14, max=567, avg=112.41, stdev=222.95 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15], | 30.00th=[ 15], 40.00th=[ 19], 50.00th=[ 19], 60.00th=[ 20], | 70.00th=[ 41], 80.00th=[ 41], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 4096, max= 4096, per=38.58%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3346KiB/s (3427kB/s)(6144KiB/1836msec) clat (msec): min=43, max=691, avg=186.26, stdev=250.94 lat (msec): min=43, max=691, avg=186.34, stdev=250.94 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 46], | 30.00th=[ 46], 40.00th=[ 79], 50.00th=[ 79], 60.00th=[ 105], | 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 2048, max= 4096, per=22.97%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 lat (msec) : 20=33.33%, 50=25.00%, 100=8.33%, 250=16.67%, 750=16.67% cpu : usr=0.00%, sys=6.59%, ctx=106, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21074: Fri Aug 9 23:06:33 2024 read: IOPS=3, BW=3263KiB/s (3342kB/s)(5120KiB/1569msec) clat (msec): min=12, max=480, avg=120.14, stdev=202.35 lat (msec): min=12, max=480, avg=120.14, stdev=202.35 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 13], | 30.00th=[ 17], 40.00th=[ 17], 50.00th=[ 27], 60.00th=[ 27], | 70.00th=[ 65], 80.00th=[ 65], 90.00th=[ 481], 95.00th=[ 481], | 99.00th=[ 481], 99.50th=[ 481], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 8192, max= 8192, per=77.17%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4569KiB/s (4678kB/s)(7168KiB/1569msec) clat (msec): min=36, max=507, avg=137.44, stdev=167.85 lat (msec): min=36, max=507, avg=137.52, stdev=167.85 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 53], | 30.00th=[ 66], 40.00th=[ 66], 50.00th=[ 68], 60.00th=[ 74], | 70.00th=[ 74], 80.00th=[ 159], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 2048, max= 8192, per=32.15%, avg=4778.67, stdev=3128.37, samples=3 iops : min= 2, max= 8, avg= 4.67, stdev= 3.06, samples=3 lat (msec) : 20=16.67%, 50=16.67%, 100=41.67%, 250=8.33%, 500=8.33% lat (msec) : 750=8.33% cpu : usr=0.13%, sys=7.08%, ctx=56, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.4MiB/s (10.9MB/s), 1062KiB/s-6731KiB/s (1087kB/s-6892kB/s), io=20.0MiB (20.0MB), run=1065-1929msec WRITE: bw=14.5MiB/s (15.2MB/s), 3346KiB/s-5308KiB/s (3427kB/s-5436kB/s), io=28.0MiB (29.4MB), run=1065-1929msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21075: Fri Aug 9 23:06:34 2024 read: IOPS=3, BW=3074KiB/s (3148kB/s)(7168KiB/2332msec) clat (usec): min=402, max=745545, avg=296220.31, stdev=313447.78 lat (usec): min=403, max=745547, avg=296221.54, stdev=313447.80 clat percentiles (usec): | 1.00th=[ 404], 5.00th=[ 404], 10.00th=[ 404], 20.00th=[ 627], | 30.00th=[ 72877], 40.00th=[ 72877], 50.00th=[152044], 60.00th=[476054], | 70.00th=[476054], 80.00th=[624952], 90.00th=[742392], 95.00th=[742392], | 99.00th=[742392], 99.50th=[742392], 99.90th=[742392], 99.95th=[742392], | 99.99th=[742392] bw ( KiB/s): min= 1695, max= 3390, per=26.14%, avg=2295.25, stdev=748.56, samples=4 iops : min= 1, max= 3, avg= 2.00, stdev= 0.82, samples=4 write: IOPS=2, BW=2196KiB/s (2248kB/s)(5120KiB/2332msec) clat (usec): min=24981, max=68914, avg=41865.40, stdev=19855.12 lat (usec): min=25049, max=68931, avg=41921.58, stdev=19839.14 clat percentiles (usec): | 1.00th=[25035], 5.00th=[25035], 10.00th=[25035], 20.00th=[25035], | 30.00th=[28705], 40.00th=[28705], 50.00th=[29492], 60.00th=[29492], | 70.00th=[57410], 80.00th=[57410], 90.00th=[68682], 95.00th=[68682], | 99.00th=[68682], 99.50th=[68682], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 1695, max= 5086, per=27.58%, avg=3390.50, stdev=2397.80, samples=2 iops : min= 1, max= 4, avg= 2.50, stdev= 2.12, samples=2 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=25.00%, 100=25.00%, 250=8.33%, 500=8.33%, 750=16.67% cpu : usr=0.04%, sys=25.87%, ctx=849, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21076: Fri Aug 9 23:06:34 2024 read: IOPS=1, BW=1061KiB/s (1087kB/s)(2048KiB/1930msec) clat (msec): min=94, max=602, avg=348.90, stdev=359.24 lat (msec): min=94, max=602, avg=348.90, stdev=359.24 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 95], 10.00th=[ 95], 20.00th=[ 95], | 30.00th=[ 95], 40.00th=[ 95], 50.00th=[ 95], 60.00th=[ 600], | 70.00th=[ 600], 80.00th=[ 600], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 1681, max= 2048, per=21.23%, avg=1864.50, stdev=259.51, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=5, BW=5306KiB/s (5433kB/s)(10.0MiB/1930msec) clat (msec): min=22, max=579, avg=118.31, stdev=170.72 lat (msec): min=22, max=579, avg=118.38, stdev=170.72 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 39], | 30.00th=[ 49], 40.00th=[ 52], 50.00th=[ 54], 60.00th=[ 55], | 70.00th=[ 62], 80.00th=[ 62], 90.00th=[ 215], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 4096, max= 6144, per=41.64%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 50=25.00%, 100=50.00%, 250=8.33%, 750=16.67% cpu : usr=0.00%, sys=18.46%, ctx=374, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21077: Fri Aug 9 23:06:34 2024 read: IOPS=3, BW=3642KiB/s (3729kB/s)(6144KiB/1687msec) clat (usec): min=418, max=592960, avg=134446.24, stdev=233723.41 lat (usec): min=419, max=592961, avg=134447.36, stdev=233723.37 clat percentiles (usec): | 1.00th=[ 420], 5.00th=[ 420], 10.00th=[ 420], 20.00th=[ 553], | 30.00th=[ 553], 40.00th=[ 11338], 50.00th=[ 11338], 60.00th=[ 31589], | 70.00th=[170918], 80.00th=[170918], 90.00th=[591397], 95.00th=[591397], | 99.00th=[591397], 99.50th=[591397], 99.90th=[591397], 99.95th=[591397], | 99.99th=[591397] bw ( KiB/s): min= 2048, max= 6119, per=46.50%, avg=4083.50, stdev=2878.63, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=3, BW=3642KiB/s (3729kB/s)(6144KiB/1687msec) clat (msec): min=17, max=668, avg=143.72, stdev=257.85 lat (msec): min=17, max=668, avg=143.78, stdev=257.87 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 24], | 30.00th=[ 24], 40.00th=[ 34], 50.00th=[ 34], 60.00th=[ 43], | 70.00th=[ 78], 80.00th=[ 78], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 2048, max= 4096, per=27.72%, avg=3407.67, stdev=1177.54, samples=3 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=3 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 20=16.67%, 50=33.33%, 100=8.33%, 250=8.33%, 750=16.67% cpu : usr=0.06%, sys=21.23%, ctx=391, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21078: Fri Aug 9 23:06:34 2024 read: IOPS=3, BW=3429KiB/s (3512kB/s)(5120KiB/1493msec) clat (usec): min=483, max=520398, avg=221026.24, stdev=245450.46 lat (usec): min=485, max=520398, avg=221027.48, stdev=245450.27 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 482], 10.00th=[ 482], 20.00th=[ 482], | 30.00th=[ 58459], 40.00th=[ 58459], 50.00th=[ 71828], 60.00th=[ 71828], | 70.00th=[455082], 80.00th=[455082], 90.00th=[522191], 95.00th=[522191], | 99.00th=[522191], 99.50th=[522191], 99.90th=[522191], 99.95th=[522191], | 99.99th=[522191] bw ( KiB/s): min= 2048, max= 6144, per=46.64%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=4, BW=4801KiB/s (4916kB/s)(7168KiB/1493msec) clat (usec): min=32615, max=55410, avg=45173.58, stdev=7237.94 lat (usec): min=32682, max=55477, avg=45236.91, stdev=7234.71 clat percentiles (usec): | 1.00th=[32637], 5.00th=[32637], 10.00th=[32637], 20.00th=[41681], | 30.00th=[42206], 40.00th=[42206], 50.00th=[47449], 60.00th=[48497], | 70.00th=[48497], 80.00th=[49021], 90.00th=[55313], 95.00th=[55313], | 99.00th=[55313], 99.50th=[55313], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 6144, max= 8192, per=58.30%, avg=7168.00, stdev=1448.15, samples=2 iops : min= 6, max= 8, avg= 7.00, stdev= 1.41, samples=2 lat (usec) : 500=8.33% lat (msec) : 50=50.00%, 100=25.00%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=23.19%, ctx=606, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8782KiB/s (8993kB/s), 1061KiB/s-3642KiB/s (1087kB/s-3729kB/s), io=20.0MiB (20.0MB), run=1493-2332msec WRITE: bw=12.0MiB/s (12.6MB/s), 2196KiB/s-5306KiB/s (2248kB/s-5433kB/s), io=28.0MiB (29.4MB), run=1493-2332msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21086: Fri Aug 9 23:06:36 2024 read: IOPS=6, BW=26.8MiB/s (28.1MB/s)(12.0MiB/448msec) clat (usec): min=1773, max=196188, avg=127666.42, stdev=109168.96 lat (usec): min=1774, max=196189, avg=127667.41, stdev=109169.27 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1778], 10.00th=[ 1778], 20.00th=[ 1778], | 30.00th=[ 1778], 40.00th=[185598], 50.00th=[185598], 60.00th=[185598], | 70.00th=[196084], 80.00th=[196084], 90.00th=[196084], 95.00th=[196084], | 99.00th=[196084], 99.50th=[196084], 99.90th=[196084], 99.95th=[196084], | 99.99th=[196084] lat (msec) : 2=33.33%, 250=66.67% cpu : usr=0.00%, sys=52.57%, ctx=214, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21087: Fri Aug 9 23:06:36 2024 read: IOPS=1, BW=4876KiB/s (4993kB/s)(4096KiB/840msec) clat (nsec): min=182723k, max=182723k, avg=182723257.00, stdev= 0.00 lat (nsec): min=182724k, max=182724k, avg=182724351.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 184], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 8192, max= 8192, per=42.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9752KiB/s (9986kB/s)(8192KiB/840msec) clat (msec): min=196, max=452, avg=324.35, stdev=181.43 lat (msec): min=196, max=452, avg=324.56, stdev=181.54 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 197], 10.00th=[ 197], 20.00th=[ 197], | 30.00th=[ 197], 40.00th=[ 197], 50.00th=[ 197], 60.00th=[ 451], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 8192, max= 8192, per=21.40%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=28.25%, ctx=223, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21088: Fri Aug 9 23:06:36 2024 write: IOPS=3, BW=14.0MiB/s (14.7MB/s)(12.0MiB/856msec) clat (msec): min=91, max=374, avg=276.44, stdev=160.66 lat (msec): min=91, max=375, avg=276.66, stdev=160.63 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 91], 10.00th=[ 91], 20.00th=[ 91], | 30.00th=[ 91], 40.00th=[ 363], 50.00th=[ 363], 60.00th=[ 363], | 70.00th=[ 376], 80.00th=[ 376], 90.00th=[ 376], 95.00th=[ 376], | 99.00th=[ 376], 99.50th=[ 376], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 8192, max= 8192, per=21.40%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.00%, sys=31.23%, ctx=26, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21091: Fri Aug 9 23:06:36 2024 write: IOPS=4, BW=19.5MiB/s (20.4MB/s)(12.0MiB/616msec) clat (msec): min=131, max=245, avg=195.41, stdev=58.19 lat (msec): min=132, max=245, avg=195.73, stdev=58.10 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 132], 10.00th=[ 132], 20.00th=[ 132], | 30.00th=[ 132], 40.00th=[ 209], 50.00th=[ 209], 60.00th=[ 209], | 70.00th=[ 247], 80.00th=[ 247], 90.00th=[ 247], 95.00th=[ 247], | 99.00th=[ 247], 99.50th=[ 247], 99.90th=[ 247], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min=16384, max=16384, per=42.80%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=47.97%, ctx=90, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.0MiB/s (19.0MB/s), 4876KiB/s-26.8MiB/s (4993kB/s-28.1MB/s), io=16.0MiB (16.8MB), run=448-840msec WRITE: bw=37.4MiB/s (39.2MB/s), 9752KiB/s-19.5MiB/s (9986kB/s-20.4MB/s), io=32.0MiB (33.6MB), run=616-856msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21089: Fri Aug 9 23:06:36 2024 read: IOPS=4, BW=18.4MiB/s (19.3MB/s)(12.0MiB/651msec) clat (msec): min=78, max=352, avg=209.05, stdev=137.60 lat (msec): min=78, max=352, avg=209.05, stdev=137.60 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 79], 10.00th=[ 79], 20.00th=[ 79], | 30.00th=[ 79], 40.00th=[ 197], 50.00th=[ 197], 60.00th=[ 197], | 70.00th=[ 355], 80.00th=[ 355], 90.00th=[ 355], 95.00th=[ 355], | 99.00th=[ 355], 99.50th=[ 355], 99.90th=[ 355], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 8192, max= 8192, per=35.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=9.85%, ctx=27, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21090: Fri Aug 9 23:06:36 2024 read: IOPS=1, BW=5818KiB/s (5958kB/s)(4096KiB/704msec) clat (nsec): min=63432k, max=63432k, avg=63432002.00, stdev= 0.00 lat (nsec): min=63434k, max=63434k, avg=63433800.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[63177], 5.00th=[63177], 10.00th=[63177], 20.00th=[63177], | 30.00th=[63177], 40.00th=[63177], 50.00th=[63177], 60.00th=[63177], | 70.00th=[63177], 80.00th=[63177], 90.00th=[63177], 95.00th=[63177], | 99.00th=[63177], 99.50th=[63177], 99.90th=[63177], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 8192, max= 8192, per=35.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=11.4MiB/s (11.9MB/s)(8192KiB/704msec) clat (msec): min=193, max=404, avg=299.43, stdev=149.18 lat (msec): min=194, max=405, avg=299.70, stdev=149.20 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 194], 10.00th=[ 194], 20.00th=[ 194], | 30.00th=[ 194], 40.00th=[ 194], 50.00th=[ 194], 60.00th=[ 405], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 8192, max= 8192, per=24.83%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.14%, sys=12.80%, ctx=56, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21092: Fri Aug 9 23:06:36 2024 write: IOPS=3, BW=12.1MiB/s (12.7MB/s)(12.0MiB/993msec) clat (msec): min=46, max=484, avg=315.06, stdev=235.55 lat (msec): min=46, max=484, avg=315.31, stdev=235.55 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 46], | 30.00th=[ 46], 40.00th=[ 414], 50.00th=[ 414], 60.00th=[ 414], | 70.00th=[ 485], 80.00th=[ 485], 90.00th=[ 485], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 8175, max= 8175, per=24.77%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 500=66.67% cpu : usr=0.10%, sys=11.29%, ctx=57, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21093: Fri Aug 9 23:06:36 2024 write: IOPS=4, BW=16.4MiB/s (17.2MB/s)(12.0MiB/731msec) clat (msec): min=200, max=278, avg=235.38, stdev=39.72 lat (msec): min=201, max=278, avg=235.70, stdev=39.65 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 201], 10.00th=[ 201], 20.00th=[ 201], | 30.00th=[ 201], 40.00th=[ 228], 50.00th=[ 228], 60.00th=[ 228], | 70.00th=[ 279], 80.00th=[ 279], 90.00th=[ 279], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 8192, max= 8192, per=24.83%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.14%, sys=18.36%, ctx=47, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.7MiB/s (23.8MB/s), 5818KiB/s-18.4MiB/s (5958kB/s-19.3MB/s), io=16.0MiB (16.8MB), run=651-704msec WRITE: bw=32.2MiB/s (33.8MB/s), 11.4MiB/s-16.4MiB/s (11.9MB/s-17.2MB/s), io=32.0MiB (33.6MB), run=704-993msec