== sanity test 398b: DIO and buffer IO race ============== 05:26:43 (1727774803) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.10066 s, 45.7 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=18703: Tue Oct 1 05:27:01 2024 read: IOPS=97, BW=392KiB/s (401kB/s)(5992KiB/15302msec) clat (usec): min=48, max=182943, avg=4723.74, stdev=6067.30 lat (usec): min=48, max=182943, avg=4725.79, stdev=6066.99 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 76], 10.00th=[ 94], 20.00th=[ 2278], | 30.00th=[ 2900], 40.00th=[ 3392], 50.00th=[ 3916], 60.00th=[ 4490], | 70.00th=[ 5538], 80.00th=[ 7242], 90.00th=[ 9372], 95.00th=[ 10814], | 99.00th=[ 14484], 99.50th=[ 23987], 99.90th=[ 54789], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 232, max= 552, per=24.92%, avg=387.33, stdev=76.60, samples=30 iops : min= 58, max= 138, avg=96.77, stdev=19.09, samples=30 write: IOPS=102, BW=411KiB/s (421kB/s)(6296KiB/15302msec) clat (usec): min=496, max=65581, avg=5152.18, stdev=5097.50 lat (usec): min=497, max=65581, avg=5153.59, stdev=5097.48 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 619], 10.00th=[ 685], 20.00th=[ 865], | 30.00th=[ 1254], 40.00th=[ 2606], 50.00th=[ 4113], 60.00th=[ 5342], | 70.00th=[ 6587], 80.00th=[ 8717], 90.00th=[11469], 95.00th=[13829], | 99.00th=[20579], 99.50th=[24773], 99.90th=[52691], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 272, max= 528, per=25.00%, avg=406.77, stdev=73.27, samples=30 iops : min= 68, max= 132, avg=101.63, stdev=18.25, samples=30 lat (usec) : 50=0.03%, 100=5.70%, 250=3.26%, 500=0.07%, 750=6.84% lat (usec) : 1000=5.92% lat (msec) : 2=6.09%, 4=22.66%, 10=38.22%, 20=10.38%, 50=0.72% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.21%, sys=24.39%, ctx=6391, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18704: Tue Oct 1 05:27:01 2024 read: IOPS=96, BW=385KiB/s (394kB/s)(5940KiB/15422msec) clat (usec): min=49, max=58270, avg=4674.61, stdev=3891.48 lat (usec): min=49, max=58271, avg=4675.71, stdev=3891.28 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 73], 10.00th=[ 98], 20.00th=[ 2343], | 30.00th=[ 2966], 40.00th=[ 3392], 50.00th=[ 3884], 60.00th=[ 4686], | 70.00th=[ 5866], 80.00th=[ 7242], 90.00th=[ 9241], 95.00th=[10945], | 99.00th=[15139], 99.50th=[19530], 99.90th=[42730], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 256, max= 544, per=24.43%, avg=379.70, stdev=68.20, samples=30 iops : min= 64, max= 136, avg=94.90, stdev=17.05, samples=30 write: IOPS=102, BW=412KiB/s (421kB/s)(6348KiB/15422msec) clat (usec): min=493, max=180365, avg=5273.88, stdev=6644.56 lat (usec): min=493, max=180366, avg=5275.14, stdev=6644.51 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 611], 10.00th=[ 685], 20.00th=[ 873], | 30.00th=[ 1237], 40.00th=[ 2442], 50.00th=[ 4080], 60.00th=[ 5342], | 70.00th=[ 6587], 80.00th=[ 8455], 90.00th=[ 11469], 95.00th=[ 14615], | 99.00th=[ 22152], 99.50th=[ 29754], 99.90th=[ 39584], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 303, max= 552, per=24.55%, avg=399.43, stdev=60.87, samples=30 iops : min= 75, max= 138, avg=99.83, stdev=15.26, samples=30 lat (usec) : 50=0.03%, 100=5.01%, 250=3.26%, 500=0.33%, 750=7.10% lat (usec) : 1000=6.05% lat (msec) : 2=6.38%, 4=22.36%, 10=38.28%, 20=10.29%, 50=0.85% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.27%, sys=24.33%, ctx=6475, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18705: Tue Oct 1 05:27:01 2024 read: IOPS=97, BW=390KiB/s (400kB/s)(6028KiB/15446msec) clat (usec): min=49, max=65556, avg=4773.88, stdev=4632.77 lat (usec): min=49, max=65557, avg=4775.16, stdev=4632.93 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 74], 10.00th=[ 97], 20.00th=[ 2278], | 30.00th=[ 2933], 40.00th=[ 3425], 50.00th=[ 3949], 60.00th=[ 4686], | 70.00th=[ 5800], 80.00th=[ 7242], 90.00th=[ 9241], 95.00th=[10945], | 99.00th=[16712], 99.50th=[27132], 99.90th=[63177], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 240, max= 512, per=24.59%, avg=382.10, stdev=74.03, samples=30 iops : min= 60, max= 128, avg=95.50, stdev=18.49, samples=30 write: IOPS=101, BW=405KiB/s (415kB/s)(6260KiB/15446msec) clat (usec): min=489, max=182408, avg=5199.90, stdev=6436.82 lat (usec): min=490, max=182409, avg=5201.09, stdev=6436.93 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 603], 10.00th=[ 685], 20.00th=[ 873], | 30.00th=[ 1221], 40.00th=[ 2933], 50.00th=[ 4293], 60.00th=[ 5342], | 70.00th=[ 6652], 80.00th=[ 8291], 90.00th=[ 11207], 95.00th=[ 13960], | 99.00th=[ 21890], 99.50th=[ 23987], 99.90th=[ 39060], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 279, max= 520, per=24.30%, avg=395.43, stdev=61.38, samples=30 iops : min= 69, max= 130, avg=98.83, stdev=15.39, samples=30 lat (usec) : 50=0.16%, 100=5.14%, 250=3.39%, 500=0.36%, 750=7.00% lat (usec) : 1000=6.28% lat (msec) : 2=4.88%, 4=22.23%, 10=40.04%, 20=9.54%, 50=0.85% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.23%, sys=24.06%, ctx=6429, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18707: Tue Oct 1 05:27:01 2024 read: IOPS=99, BW=398KiB/s (408kB/s)(6052KiB/15197msec) clat (usec): min=46, max=199162, avg=4631.51, stdev=6101.40 lat (usec): min=46, max=199162, avg=4632.41, stdev=6101.42 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 75], 10.00th=[ 99], 20.00th=[ 2311], | 30.00th=[ 2868], 40.00th=[ 3326], 50.00th=[ 3884], 60.00th=[ 4555], | 70.00th=[ 5604], 80.00th=[ 7177], 90.00th=[ 9110], 95.00th=[ 10421], | 99.00th=[ 14222], 99.50th=[ 19792], 99.90th=[ 32637], 99.95th=[198181], | 99.99th=[198181] bw ( KiB/s): min= 264, max= 510, per=25.46%, avg=395.60, stdev=61.79, samples=30 iops : min= 66, max= 127, avg=98.83, stdev=15.39, samples=30 write: IOPS=102, BW=410KiB/s (420kB/s)(6236KiB/15197msec) clat (usec): min=508, max=48380, avg=5176.64, stdev=4959.59 lat (usec): min=508, max=48381, avg=5178.19, stdev=4959.66 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 619], 10.00th=[ 701], 20.00th=[ 889], | 30.00th=[ 1254], 40.00th=[ 2704], 50.00th=[ 4228], 60.00th=[ 5407], | 70.00th=[ 6718], 80.00th=[ 8586], 90.00th=[11600], 95.00th=[14353], | 99.00th=[20317], 99.50th=[26608], 99.90th=[41157], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 288, max= 536, per=25.05%, avg=407.57, stdev=64.49, samples=30 iops : min= 72, max= 134, avg=101.83, stdev=16.09, samples=30 lat (usec) : 50=0.13%, 100=4.95%, 250=3.87%, 500=0.13%, 750=6.32% lat (usec) : 1000=6.87% lat (msec) : 2=5.76%, 4=21.81%, 10=39.75%, 20=9.64%, 50=0.75% lat (msec) : 250=0.03% cpu : usr=0.27%, sys=24.37%, ctx=6289, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1555KiB/s (1592kB/s), 385KiB/s-398KiB/s (394kB/s-408kB/s), io=23.4MiB (24.6MB), run=15197-15446msec WRITE: bw=1628KiB/s (1667kB/s), 405KiB/s-412KiB/s (415kB/s-421kB/s), io=24.6MiB (25.7MB), run=15197-15446msec rand-rw: (groupid=0, jobs=1): err= 0: pid=18706: Tue Oct 1 05:27:24 2024 read: IOPS=39, BW=158KiB/s (162kB/s)(5992KiB/37866msec) clat (usec): min=1725, max=184179, avg=4231.18, stdev=5883.10 lat (usec): min=1726, max=184180, avg=4232.32, stdev=5883.18 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 2024], 10.00th=[ 2114], 20.00th=[ 2278], | 30.00th=[ 2376], 40.00th=[ 2507], 50.00th=[ 2704], 60.00th=[ 2966], | 70.00th=[ 3458], 80.00th=[ 4883], 90.00th=[ 8717], 95.00th=[ 12125], | 99.00th=[ 17957], 99.50th=[ 20055], 99.90th=[ 52691], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 56, max= 304, per=25.16%, avg=157.99, stdev=64.24, samples=75 iops : min= 14, max= 76, avg=39.43, stdev=16.06, samples=75 write: IOPS=41, BW=166KiB/s (170kB/s)(6296KiB/37866msec) clat (usec): min=9501, max=82180, avg=19959.98, stdev=7821.13 lat (usec): min=9502, max=82181, avg=19961.27, stdev=7821.16 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11731], 10.00th=[12518], 20.00th=[13698], | 30.00th=[15270], 40.00th=[16712], 50.00th=[18220], 60.00th=[19792], | 70.00th=[21365], 80.00th=[24249], 90.00th=[30016], 95.00th=[34866], | 99.00th=[46400], 99.50th=[53216], 99.90th=[81265], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 48, max= 264, per=25.16%, avg=165.56, stdev=51.46, samples=75 iops : min= 12, max= 66, avg=41.32, stdev=12.86, samples=75 lat (msec) : 2=2.08%, 4=34.60%, 10=8.56%, 20=34.28%, 50=20.15% lat (msec) : 100=0.29%, 250=0.03% cpu : usr=0.15%, sys=5.86%, ctx=3499, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18708: Tue Oct 1 05:27:24 2024 read: IOPS=38, BW=156KiB/s (159kB/s)(5940KiB/38194msec) clat (usec): min=1647, max=200390, avg=4240.23, stdev=6361.86 lat (usec): min=1648, max=200391, avg=4241.22, stdev=6361.86 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1958], 10.00th=[ 2073], 20.00th=[ 2212], | 30.00th=[ 2376], 40.00th=[ 2474], 50.00th=[ 2638], 60.00th=[ 2868], | 70.00th=[ 3425], 80.00th=[ 4883], 90.00th=[ 8848], 95.00th=[ 11994], | 99.00th=[ 19268], 99.50th=[ 22152], 99.90th=[ 65274], 99.95th=[200279], | 99.99th=[200279] bw ( KiB/s): min= 16, max= 320, per=24.72%, avg=155.25, stdev=71.87, samples=76 iops : min= 4, max= 80, avg=38.80, stdev=17.97, samples=76 write: IOPS=41, BW=166KiB/s (170kB/s)(6348KiB/38194msec) clat (usec): min=8821, max=81129, avg=20028.53, stdev=7932.56 lat (usec): min=8822, max=81131, avg=20029.90, stdev=7932.49 clat percentiles (usec): | 1.00th=[10552], 5.00th=[11731], 10.00th=[12387], 20.00th=[13566], | 30.00th=[15008], 40.00th=[16712], 50.00th=[18482], 60.00th=[20055], | 70.00th=[21627], 80.00th=[24511], 90.00th=[31327], 95.00th=[35914], | 99.00th=[46400], 99.50th=[52167], 99.90th=[77071], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 56, max= 256, per=25.15%, avg=165.46, stdev=52.11, samples=76 iops : min= 14, max= 64, avg=41.36, stdev=13.01, samples=76 lat (msec) : 2=3.09%, 4=33.20%, 10=8.30%, 20=34.51%, 50=20.51% lat (msec) : 100=0.36%, 250=0.03% cpu : usr=0.17%, sys=5.77%, ctx=3543, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18709: Tue Oct 1 05:27:24 2024 read: IOPS=40, BW=160KiB/s (164kB/s)(6028KiB/37660msec) clat (usec): min=1632, max=77301, avg=4330.83, stdev=4130.34 lat (usec): min=1633, max=77302, avg=4331.93, stdev=4130.33 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2024], 10.00th=[ 2114], 20.00th=[ 2278], | 30.00th=[ 2409], 40.00th=[ 2573], 50.00th=[ 2769], 60.00th=[ 3097], | 70.00th=[ 3785], 80.00th=[ 5342], 90.00th=[ 9241], 95.00th=[12125], | 99.00th=[17433], 99.50th=[25035], 99.90th=[37487], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 48, max= 296, per=25.48%, avg=160.03, stdev=59.66, samples=75 iops : min= 12, max= 74, avg=39.95, stdev=14.91, samples=75 write: IOPS=41, BW=166KiB/s (170kB/s)(6260KiB/37660msec) clat (msec): min=9, max=216, avg=19.82, stdev= 8.93 lat (msec): min=9, max=216, avg=19.82, stdev= 8.93 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 20], | 70.00th=[ 22], 80.00th=[ 24], 90.00th=[ 31], 95.00th=[ 35], | 99.00th=[ 44], 99.50th=[ 47], 99.90th=[ 83], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 64, max= 264, per=25.21%, avg=165.89, stdev=53.30, samples=75 iops : min= 16, max= 66, avg=41.41, stdev=13.34, samples=75 lat (msec) : 2=2.02%, 4=33.63%, 10=9.64%, 20=34.90%, 50=19.66% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.15%, sys=5.93%, ctx=3545, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18710: Tue Oct 1 05:27:24 2024 read: IOPS=39, BW=160KiB/s (163kB/s)(6052KiB/37917msec) clat (usec): min=1681, max=73481, avg=4273.47, stdev=4105.51 lat (usec): min=1682, max=73483, avg=4274.55, stdev=4105.46 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2024], 10.00th=[ 2114], 20.00th=[ 2278], | 30.00th=[ 2409], 40.00th=[ 2540], 50.00th=[ 2737], 60.00th=[ 3032], | 70.00th=[ 3621], 80.00th=[ 5145], 90.00th=[ 9110], 95.00th=[12125], | 99.00th=[18220], 99.50th=[22152], 99.90th=[52691], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 40, max= 304, per=25.33%, avg=159.05, stdev=64.95, samples=75 iops : min= 10, max= 76, avg=39.69, stdev=16.25, samples=75 write: IOPS=41, BW=164KiB/s (168kB/s)(6236KiB/37917msec) clat (msec): min=9, max=215, avg=20.10, stdev= 9.31 lat (msec): min=9, max=215, avg=20.10, stdev= 9.31 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 20], | 70.00th=[ 22], 80.00th=[ 25], 90.00th=[ 31], 95.00th=[ 36], | 99.00th=[ 47], 99.50th=[ 58], 99.90th=[ 73], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 56, max= 256, per=24.93%, avg=164.07, stdev=52.50, samples=75 iops : min= 14, max= 64, avg=40.95, stdev=13.12, samples=75 lat (msec) : 2=2.12%, 4=33.72%, 10=9.67%, 20=34.41%, 50=19.76% lat (msec) : 100=0.29%, 250=0.03% cpu : usr=0.16%, sys=5.86%, ctx=3527, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=629KiB/s (644kB/s), 156KiB/s-160KiB/s (159kB/s-164kB/s), io=23.4MiB (24.6MB), run=37660-38194msec WRITE: bw=658KiB/s (674kB/s), 164KiB/s-166KiB/s (168kB/s-170kB/s), io=24.6MiB (25.7MB), run=37660-38194msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=18751: Tue Oct 1 05:27:31 2024 read: IOPS=67, BW=1074KiB/s (1100kB/s)(5920KiB/5510msec) clat (usec): min=53, max=41426, avg=10613.31, stdev=6384.64 lat (usec): min=53, max=41427, avg=10614.20, stdev=6384.64 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 75], 10.00th=[ 96], 20.00th=[ 159], | 30.00th=[ 9634], 40.00th=[10945], 50.00th=[11994], 60.00th=[13042], | 70.00th=[13960], 80.00th=[15401], 90.00th=[16909], 95.00th=[18744], | 99.00th=[22938], 99.50th=[25560], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 832, max= 1341, per=25.77%, avg=1073.18, stdev=154.51, samples=11 iops : min= 52, max= 83, avg=67.00, stdev= 9.52, samples=11 write: IOPS=72, BW=1156KiB/s (1183kB/s)(6368KiB/5510msec) clat (usec): min=665, max=29609, avg=3899.34, stdev=4164.59 lat (usec): min=666, max=29609, avg=3900.96, stdev=4164.47 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 816], 10.00th=[ 889], 20.00th=[ 1020], | 30.00th=[ 1172], 40.00th=[ 1385], 50.00th=[ 2474], 60.00th=[ 3621], | 70.00th=[ 4293], 80.00th=[ 5800], 90.00th=[ 8979], 95.00th=[11469], | 99.00th=[21365], 99.50th=[28967], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 894, max= 1600, per=26.36%, avg=1154.73, stdev=220.76, samples=11 iops : min= 55, max= 100, avg=72.09, stdev=13.90, samples=11 lat (usec) : 100=5.21%, 250=4.82%, 500=0.13%, 750=1.04%, 1000=9.11% lat (msec) : 2=14.45%, 4=10.55%, 10=17.97%, 20=34.37%, 50=2.34% cpu : usr=0.07%, sys=22.45%, ctx=2079, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18752: Tue Oct 1 05:27:31 2024 read: IOPS=65, BW=1045KiB/s (1070kB/s)(5744KiB/5498msec) clat (usec): min=57, max=75610, avg=11208.25, stdev=7324.02 lat (usec): min=58, max=75610, avg=11209.27, stdev=7324.09 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 80], 10.00th=[ 97], 20.00th=[ 176], | 30.00th=[10421], 40.00th=[11600], 50.00th=[12125], 60.00th=[13173], | 70.00th=[14353], 80.00th=[15533], 90.00th=[17695], 95.00th=[19792], | 99.00th=[28181], 99.50th=[30278], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 704, max= 1248, per=25.12%, avg=1046.20, stdev=159.80, samples=10 iops : min= 44, max= 78, avg=65.30, stdev=10.04, samples=10 write: IOPS=74, BW=1190KiB/s (1219kB/s)(6544KiB/5498msec) clat (usec): min=663, max=30294, avg=3552.75, stdev=3821.22 lat (usec): min=664, max=30295, avg=3554.16, stdev=3821.25 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 807], 10.00th=[ 857], 20.00th=[ 1037], | 30.00th=[ 1205], 40.00th=[ 1352], 50.00th=[ 1827], 60.00th=[ 3195], | 70.00th=[ 3884], 80.00th=[ 5407], 90.00th=[ 8225], 95.00th=[ 9896], | 99.00th=[18482], 99.50th=[21627], 99.90th=[30278], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 832, max= 1600, per=26.80%, avg=1174.00, stdev=246.73, samples=10 iops : min= 52, max= 100, avg=73.30, stdev=15.29, samples=10 lat (usec) : 100=5.08%, 250=4.43%, 750=1.17%, 1000=8.07% lat (msec) : 2=18.36%, 4=10.55%, 10=15.76%, 20=33.98%, 50=2.47% lat (msec) : 100=0.13% cpu : usr=0.15%, sys=23.81%, ctx=2066, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18753: Tue Oct 1 05:27:31 2024 read: IOPS=67, BW=1088KiB/s (1114kB/s)(6256KiB/5752msec) clat (usec): min=56, max=55940, avg=11064.21, stdev=6966.42 lat (usec): min=57, max=55940, avg=11065.55, stdev=6966.67 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 82], 10.00th=[ 100], 20.00th=[ 3032], | 30.00th=[ 9372], 40.00th=[10814], 50.00th=[12125], 60.00th=[13304], | 70.00th=[14353], 80.00th=[15664], 90.00th=[17957], 95.00th=[20055], | 99.00th=[28181], 99.50th=[43779], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 704, max= 1405, per=25.90%, avg=1078.45, stdev=194.50, samples=11 iops : min= 44, max= 87, avg=67.18, stdev=11.97, samples=11 write: IOPS=65, BW=1049KiB/s (1074kB/s)(6032KiB/5752msec) clat (usec): min=636, max=28369, avg=3708.36, stdev=3905.03 lat (usec): min=636, max=28370, avg=3709.72, stdev=3905.01 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 807], 10.00th=[ 881], 20.00th=[ 1037], | 30.00th=[ 1205], 40.00th=[ 1418], 50.00th=[ 2311], 60.00th=[ 3359], | 70.00th=[ 4047], 80.00th=[ 5538], 90.00th=[ 8160], 95.00th=[11338], | 99.00th=[17433], 99.50th=[26870], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 448, max= 1440, per=23.16%, avg=1014.73, stdev=299.63, samples=11 iops : min= 28, max= 90, avg=63.18, stdev=18.92, samples=11 lat (usec) : 100=4.95%, 250=5.08%, 500=0.13%, 750=1.04%, 1000=8.33% lat (msec) : 2=14.32%, 4=10.42%, 10=18.49%, 20=34.37%, 50=2.73% lat (msec) : 100=0.13% cpu : usr=0.19%, sys=21.39%, ctx=2148, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18754: Tue Oct 1 05:27:31 2024 read: IOPS=69, BW=1117KiB/s (1144kB/s)(6032KiB/5401msec) clat (usec): min=54, max=35815, avg=10311.52, stdev=6207.25 lat (usec): min=55, max=35815, avg=10314.63, stdev=6208.18 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 77], 10.00th=[ 98], 20.00th=[ 151], | 30.00th=[ 9634], 40.00th=[10814], 50.00th=[11731], 60.00th=[12911], | 70.00th=[13960], 80.00th=[15139], 90.00th=[16581], 95.00th=[18482], | 99.00th=[20841], 99.50th=[22414], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 800, max= 1248, per=26.20%, avg=1090.90, stdev=145.84, samples=10 iops : min= 50, max= 78, avg=68.10, stdev= 9.12, samples=10 write: IOPS=72, BW=1158KiB/s (1186kB/s)(6256KiB/5401msec) clat (usec): min=639, max=77422, avg=3836.12, stdev=5408.11 lat (usec): min=640, max=77425, avg=3837.77, stdev=5408.03 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 824], 10.00th=[ 873], 20.00th=[ 1037], | 30.00th=[ 1221], 40.00th=[ 1352], 50.00th=[ 2008], 60.00th=[ 3294], | 70.00th=[ 4293], 80.00th=[ 5669], 90.00th=[ 8029], 95.00th=[10683], | 99.00th=[25035], 99.50th=[30278], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 800, max= 1440, per=26.29%, avg=1151.70, stdev=176.57, samples=10 iops : min= 50, max= 90, avg=71.90, stdev=11.04, samples=10 lat (usec) : 100=5.47%, 250=5.47%, 500=0.13%, 750=1.04%, 1000=8.20% lat (msec) : 2=16.15%, 4=8.07%, 10=20.05%, 20=33.72%, 50=1.56% lat (msec) : 100=0.13% cpu : usr=0.22%, sys=22.48%, ctx=2044, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4164KiB/s (4264kB/s), 1045KiB/s-1117KiB/s (1070kB/s-1144kB/s), io=23.4MiB (24.5MB), run=5401-5752msec WRITE: bw=4381KiB/s (4486kB/s), 1049KiB/s-1190KiB/s (1074kB/s-1219kB/s), io=24.6MiB (25.8MB), run=5401-5752msec rand-rw: (groupid=0, jobs=1): err= 0: pid=18755: Tue Oct 1 05:27:35 2024 read: IOPS=37, BW=592KiB/s (607kB/s)(5920KiB/9992msec) clat (usec): min=1879, max=111980, avg=5105.04, stdev=7329.73 lat (usec): min=1879, max=111981, avg=5105.96, stdev=7329.75 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2147], 10.00th=[ 2278], 20.00th=[ 2409], | 30.00th=[ 2606], 40.00th=[ 2737], 50.00th=[ 2900], 60.00th=[ 3195], | 70.00th=[ 3982], 80.00th=[ 5604], 90.00th=[ 9765], 95.00th=[ 15139], | 99.00th=[ 31327], 99.50th=[ 31851], 99.90th=[111674], 99.95th=[111674], | 99.99th=[111674] bw ( KiB/s): min= 320, max= 1088, per=24.26%, avg=574.16, stdev=214.63, samples=19 iops : min= 20, max= 68, avg=35.79, stdev=13.48, samples=19 write: IOPS=39, BW=637KiB/s (653kB/s)(6368KiB/9992msec) clat (usec): min=9866, max=52299, avg=20300.18, stdev=7155.80 lat (usec): min=9868, max=52303, avg=20301.82, stdev=7155.91 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11731], 10.00th=[12387], 20.00th=[14091], | 30.00th=[15926], 40.00th=[17433], 50.00th=[19006], 60.00th=[20317], | 70.00th=[22414], 80.00th=[25822], 90.00th=[30540], 95.00th=[34341], | 99.00th=[42206], 99.50th=[51119], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 384, max= 960, per=24.95%, avg=621.21, stdev=186.83, samples=19 iops : min= 24, max= 60, avg=38.74, stdev=11.73, samples=19 lat (msec) : 2=0.52%, 4=33.33%, 10=9.77%, 20=33.07%, 50=22.92% lat (msec) : 100=0.26%, 250=0.13% cpu : usr=0.13%, sys=6.47%, ctx=901, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18756: Tue Oct 1 05:27:35 2024 read: IOPS=35, BW=573KiB/s (587kB/s)(5744KiB/10017msec) clat (usec): min=1854, max=62400, avg=4566.40, stdev=4755.61 lat (usec): min=1855, max=62402, avg=4567.36, stdev=4755.64 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2180], 10.00th=[ 2278], 20.00th=[ 2442], | 30.00th=[ 2573], 40.00th=[ 2737], 50.00th=[ 2999], 60.00th=[ 3261], | 70.00th=[ 3982], 80.00th=[ 5735], 90.00th=[ 8848], 95.00th=[12256], | 99.00th=[19268], 99.50th=[29492], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 159, max= 960, per=24.24%, avg=573.85, stdev=225.94, samples=20 iops : min= 9, max= 60, avg=35.55, stdev=14.25, samples=20 write: IOPS=40, BW=653KiB/s (669kB/s)(6544KiB/10017msec) clat (usec): min=8790, max=55641, avg=20338.89, stdev=7106.61 lat (usec): min=8791, max=55642, avg=20340.77, stdev=7106.65 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11469], 10.00th=[12518], 20.00th=[14353], | 30.00th=[16319], 40.00th=[17695], 50.00th=[19006], 60.00th=[20579], | 70.00th=[22414], 80.00th=[25035], 90.00th=[30016], 95.00th=[34341], | 99.00th=[43779], 99.50th=[46924], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 351, max= 992, per=26.13%, avg=650.55, stdev=170.48, samples=20 iops : min= 21, max= 62, avg=40.35, stdev=10.76, samples=20 lat (msec) : 2=0.39%, 4=32.42%, 10=10.94%, 20=32.55%, 50=23.31% lat (msec) : 100=0.39% cpu : usr=0.17%, sys=6.31%, ctx=904, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18757: Tue Oct 1 05:27:35 2024 read: IOPS=40, BW=642KiB/s (658kB/s)(6256KiB/9737msec) clat (usec): min=1885, max=25106, avg=4815.05, stdev=4240.59 lat (usec): min=1886, max=25107, avg=4816.38, stdev=4240.70 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2114], 10.00th=[ 2278], 20.00th=[ 2442], | 30.00th=[ 2540], 40.00th=[ 2704], 50.00th=[ 2999], 60.00th=[ 3458], | 70.00th=[ 4228], 80.00th=[ 6325], 90.00th=[10159], 95.00th=[15139], | 99.00th=[21365], 99.50th=[23462], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 351, max= 1216, per=27.23%, avg=644.42, stdev=270.25, samples=19 iops : min= 21, max= 76, avg=39.95, stdev=16.89, samples=19 write: IOPS=38, BW=619KiB/s (634kB/s)(6032KiB/9737msec) clat (msec): min=10, max=111, avg=20.76, stdev= 7.84 lat (msec): min=10, max=111, avg=20.76, stdev= 7.84 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 20], 60.00th=[ 22], | 70.00th=[ 23], 80.00th=[ 25], 90.00th=[ 29], 95.00th=[ 34], | 99.00th=[ 41], 99.50th=[ 48], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 351, max= 864, per=24.60%, avg=612.42, stdev=160.45, samples=19 iops : min= 21, max= 54, avg=37.95, stdev=10.12, samples=19 lat (msec) : 2=0.65%, 4=33.85%, 10=11.07%, 20=29.69%, 50=24.61% lat (msec) : 250=0.13% cpu : usr=0.13%, sys=6.72%, ctx=894, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18758: Tue Oct 1 05:27:35 2024 read: IOPS=37, BW=596KiB/s (611kB/s)(6032KiB/10117msec) clat (usec): min=1959, max=31553, avg=4951.36, stdev=4831.15 lat (usec): min=1959, max=31554, avg=4952.47, stdev=4831.48 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2147], 10.00th=[ 2245], 20.00th=[ 2409], | 30.00th=[ 2573], 40.00th=[ 2737], 50.00th=[ 2933], 60.00th=[ 3195], | 70.00th=[ 4178], 80.00th=[ 6521], 90.00th=[10421], 95.00th=[15139], | 99.00th=[29492], 99.50th=[31327], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 287, max= 928, per=24.93%, avg=590.15, stdev=180.75, samples=20 iops : min= 17, max= 58, avg=36.70, stdev=11.48, samples=20 write: IOPS=38, BW=618KiB/s (633kB/s)(6256KiB/10117msec) clat (usec): min=10203, max=57085, avg=20920.51, stdev=7545.06 lat (usec): min=10204, max=57089, avg=20922.27, stdev=7545.16 clat percentiles (usec): | 1.00th=[10683], 5.00th=[12125], 10.00th=[13173], 20.00th=[14615], | 30.00th=[16581], 40.00th=[18220], 50.00th=[19792], 60.00th=[21103], | 70.00th=[22676], 80.00th=[24773], 90.00th=[31589], 95.00th=[35914], | 99.00th=[47449], 99.50th=[51643], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 384, max= 896, per=24.59%, avg=612.40, stdev=180.03, samples=20 iops : min= 24, max= 56, avg=38.10, stdev=11.31, samples=20 lat (msec) : 2=0.52%, 4=33.20%, 10=10.29%, 20=30.60%, 50=25.13% lat (msec) : 100=0.26% cpu : usr=0.15%, sys=6.43%, ctx=894, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2368KiB/s (2424kB/s), 573KiB/s-642KiB/s (587kB/s-658kB/s), io=23.4MiB (24.5MB), run=9737-10117msec WRITE: bw=2491KiB/s (2551kB/s), 618KiB/s-653KiB/s (633kB/s-669kB/s), io=24.6MiB (25.8MB), run=9737-10117msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=18782: Tue Oct 1 05:27:38 2024 read: IOPS=3, BW=3222KiB/s (3299kB/s)(7168KiB/2225msec) clat (usec): min=359, max=770144, avg=249366.06, stdev=314811.57 lat (usec): min=359, max=770146, avg=249367.26, stdev=314811.73 clat percentiles (usec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 3851], | 30.00th=[ 4015], 40.00th=[ 4015], 50.00th=[ 35914], 60.00th=[434111], | 70.00th=[434111], 80.00th=[497026], 90.00th=[767558], 95.00th=[767558], | 99.00th=[767558], 99.50th=[767558], 99.90th=[767558], 99.95th=[767558], | 99.99th=[767558] bw ( KiB/s): min= 1721, max= 8192, per=43.30%, avg=3985.33, stdev=3646.64, samples=3 iops : min= 1, max= 8, avg= 3.33, stdev= 4.04, samples=3 write: IOPS=2, BW=2301KiB/s (2356kB/s)(5120KiB/2225msec) clat (msec): min=16, max=332, avg=89.34, stdev=136.24 lat (msec): min=16, max=332, avg=89.40, stdev=136.23 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 17], | 30.00th=[ 22], 40.00th=[ 22], 50.00th=[ 28], 60.00th=[ 28], | 70.00th=[ 49], 80.00th=[ 49], 90.00th=[ 334], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 1721, max= 6131, per=25.61%, avg=3300.00, stdev=2457.16, samples=3 iops : min= 1, max= 5, avg= 2.67, stdev= 2.08, samples=3 lat (usec) : 500=8.33% lat (msec) : 4=8.33%, 10=8.33%, 20=8.33%, 50=33.33%, 500=25.00% lat (msec) : 1000=8.33% cpu : usr=0.09%, sys=46.81%, ctx=584, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18783: Tue Oct 1 05:27:38 2024 read: IOPS=1, BW=1317KiB/s (1349kB/s)(2048KiB/1555msec) clat (usec): min=14967, max=73293, avg=44130.71, stdev=41242.57 lat (usec): min=14968, max=73295, avg=44132.09, stdev=41243.48 clat percentiles (usec): | 1.00th=[15008], 5.00th=[15008], 10.00th=[15008], 20.00th=[15008], | 30.00th=[15008], 40.00th=[15008], 50.00th=[15008], 60.00th=[72877], | 70.00th=[72877], 80.00th=[72877], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 2048, max= 2048, per=22.25%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=6, BW=6585KiB/s (6743kB/s)(10.0MiB/1555msec) clat (msec): min=17, max=1045, avg=146.00, stdev=317.78 lat (msec): min=18, max=1045, avg=146.18, stdev=317.77 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 22], | 30.00th=[ 28], 40.00th=[ 34], 50.00th=[ 34], 60.00th=[ 37], | 70.00th=[ 71], 80.00th=[ 71], 90.00th=[ 131], 95.00th=[ 1045], | 99.00th=[ 1045], 99.50th=[ 1045], 99.90th=[ 1045], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 16, max= 16, avg=16.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=50.00%, 100=16.67%, 250=8.33% cpu : usr=0.00%, sys=16.73%, ctx=82, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18784: Tue Oct 1 05:27:38 2024 read: IOPS=2, BW=2770KiB/s (2837kB/s)(6144KiB/2218msec) clat (usec): min=570, max=415039, avg=126208.76, stdev=152392.96 lat (usec): min=572, max=415040, avg=126210.20, stdev=152392.46 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 570], 10.00th=[ 570], 20.00th=[ 30278], | 30.00th=[ 30278], 40.00th=[ 60556], 50.00th=[ 60556], 60.00th=[ 84411], | 70.00th=[166724], 80.00th=[166724], 90.00th=[413139], 95.00th=[413139], | 99.00th=[413139], 99.50th=[413139], 99.90th=[413139], 99.95th=[413139], | 99.99th=[413139] bw ( KiB/s): min= 4096, max= 6144, per=55.63%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2770KiB/s (2837kB/s)(6144KiB/2218msec) clat (msec): min=25, max=1012, avg=242.13, stdev=381.79 lat (msec): min=25, max=1012, avg=242.18, stdev=381.79 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 60], 50.00th=[ 60], 60.00th=[ 118], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 1011], 95.00th=[ 1011], | 99.00th=[ 1011], 99.50th=[ 1011], 99.90th=[ 1011], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 2048, max= 6144, per=31.76%, avg=4093.00, stdev=2048.01, samples=3 iops : min= 2, max= 6, avg= 3.67, stdev= 2.08, samples=3 lat (usec) : 750=8.33% lat (msec) : 50=16.67%, 100=33.33%, 250=25.00%, 500=8.33% cpu : usr=0.05%, sys=13.49%, ctx=347, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18785: Tue Oct 1 05:27:38 2024 read: IOPS=2, BW=2309KiB/s (2365kB/s)(5120KiB/2217msec) clat (usec): min=673, max=475931, avg=169766.73, stdev=187080.40 lat (usec): min=675, max=475931, avg=169767.97, stdev=187080.16 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 676], 10.00th=[ 676], 20.00th=[ 676], | 30.00th=[ 77071], 40.00th=[ 77071], 50.00th=[ 84411], 60.00th=[ 84411], | 70.00th=[210764], 80.00th=[210764], 90.00th=[476054], 95.00th=[476054], | 99.00th=[476054], 99.50th=[476054], 99.90th=[476054], 99.95th=[476054], | 99.99th=[476054] bw ( KiB/s): min= 2031, max= 6144, per=44.41%, avg=4087.50, stdev=2908.33, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 write: IOPS=3, BW=3233KiB/s (3311kB/s)(7168KiB/2217msec) clat (msec): min=25, max=956, avg=193.04, stdev=338.78 lat (msec): min=25, max=956, avg=193.10, stdev=338.78 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 39], | 30.00th=[ 54], 40.00th=[ 54], 50.00th=[ 67], 60.00th=[ 68], | 70.00th=[ 68], 80.00th=[ 144], 90.00th=[ 961], 95.00th=[ 961], | 99.00th=[ 961], 99.50th=[ 961], 99.90th=[ 961], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 2048, max= 6144, per=36.96%, avg=4762.33, stdev=2350.81, samples=3 iops : min= 2, max= 6, avg= 4.33, stdev= 2.08, samples=3 lat (usec) : 750=8.33% lat (msec) : 50=16.67%, 100=41.67%, 250=16.67%, 500=8.33%, 1000=8.33% cpu : usr=0.00%, sys=11.82%, ctx=357, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9204KiB/s (9425kB/s), 1317KiB/s-3222KiB/s (1349kB/s-3299kB/s), io=20.0MiB (20.0MB), run=1555-2225msec WRITE: bw=12.6MiB/s (13.2MB/s), 2301KiB/s-6585KiB/s (2356kB/s-6743kB/s), io=28.0MiB (29.4MB), run=1555-2225msec rand-rw: (groupid=0, jobs=1): err= 0: pid=18778: Tue Oct 1 05:27:38 2024 read: IOPS=3, BW=3114KiB/s (3189kB/s)(7168KiB/2302msec) clat (msec): min=14, max=695, avg=150.64, stdev=250.92 lat (msec): min=14, max=695, avg=150.64, stdev=250.92 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15], | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 36], 60.00th=[ 58], | 70.00th=[ 58], 80.00th=[ 218], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 2048, max= 6144, per=46.04%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 write: IOPS=2, BW=2224KiB/s (2278kB/s)(5120KiB/2302msec) clat (msec): min=32, max=565, avg=242.24, stdev=239.42 lat (msec): min=32, max=565, avg=242.32, stdev=239.42 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33], | 30.00th=[ 71], 40.00th=[ 71], 50.00th=[ 114], 60.00th=[ 114], | 70.00th=[ 430], 80.00th=[ 430], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 2048, max= 4096, per=21.92%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 lat (msec) : 20=25.00%, 50=16.67%, 100=16.67%, 250=16.67%, 500=8.33% lat (msec) : 750=16.67% cpu : usr=0.04%, sys=4.65%, ctx=48, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18779: Tue Oct 1 05:27:38 2024 read: IOPS=0, BW=900KiB/s (921kB/s)(2048KiB/2276msec) clat (usec): min=16669, max=65815, avg=41242.54, stdev=34750.96 lat (usec): min=16670, max=65816, avg=41243.55, stdev=34750.93 clat percentiles (usec): | 1.00th=[16712], 5.00th=[16712], 10.00th=[16712], 20.00th=[16712], | 30.00th=[16712], 40.00th=[16712], 50.00th=[16712], 60.00th=[65799], | 70.00th=[65799], 80.00th=[65799], 90.00th=[65799], 95.00th=[65799], | 99.00th=[65799], 99.50th=[65799], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 2048, max= 2048, per=23.02%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4499KiB/s (4607kB/s)(10.0MiB/2276msec) clat (msec): min=27, max=1081, avg=215.53, stdev=343.63 lat (msec): min=27, max=1081, avg=215.60, stdev=343.63 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 36], | 30.00th=[ 37], 40.00th=[ 53], 50.00th=[ 62], 60.00th=[ 87], | 70.00th=[ 118], 80.00th=[ 118], 90.00th=[ 567], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 6131, max=12288, per=73.94%, avg=9209.50, stdev=4353.66, samples=2 iops : min= 5, max= 12, avg= 8.50, stdev= 4.95, samples=2 lat (msec) : 20=8.33%, 50=25.00%, 100=41.67%, 250=8.33%, 750=8.33% cpu : usr=0.09%, sys=4.84%, ctx=67, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18780: Tue Oct 1 05:27:38 2024 read: IOPS=2, BW=2714KiB/s (2779kB/s)(6144KiB/2264msec) clat (msec): min=22, max=471, avg=142.34, stdev=173.23 lat (msec): min=22, max=471, avg=142.34, stdev=173.22 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 24], | 30.00th=[ 24], 40.00th=[ 66], 50.00th=[ 66], 60.00th=[ 75], | 70.00th=[ 197], 80.00th=[ 197], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 4096, max= 6144, per=57.55%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2714KiB/s (2779kB/s)(6144KiB/2264msec) clat (msec): min=22, max=980, avg=231.85, stdev=369.91 lat (msec): min=22, max=980, avg=231.93, stdev=369.91 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 87], 50.00th=[ 87], 60.00th=[ 92], | 70.00th=[ 163], 80.00th=[ 163], 90.00th=[ 978], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 4096, max= 4096, per=32.89%, avg=4096.00, stdev= 0.00, samples=3 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=3 lat (msec) : 50=33.33%, 100=33.33%, 250=16.67%, 500=8.33%, 1000=8.33% cpu : usr=0.00%, sys=4.24%, ctx=166, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18781: Tue Oct 1 05:27:38 2024 read: IOPS=2, BW=2250KiB/s (2304kB/s)(5120KiB/2276msec) clat (usec): min=17194, max=42558, avg=27251.91, stdev=12491.32 lat (usec): min=17197, max=42559, avg=27253.56, stdev=12490.74 clat percentiles (usec): | 1.00th=[17171], 5.00th=[17171], 10.00th=[17171], 20.00th=[17171], | 30.00th=[17957], 40.00th=[17957], 50.00th=[19530], 60.00th=[19530], | 70.00th=[39060], 80.00th=[39060], 90.00th=[42730], 95.00th=[42730], | 99.00th=[42730], 99.50th=[42730], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 8175, max= 8175, per=91.90%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3149KiB/s (3225kB/s)(7168KiB/2276msec) clat (msec): min=27, max=1056, avg=300.31, stdev=364.43 lat (msec): min=27, max=1056, avg=300.38, stdev=364.43 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 43], | 30.00th=[ 103], 40.00th=[ 103], 50.00th=[ 174], 60.00th=[ 241], | 70.00th=[ 241], 80.00th=[ 460], 90.00th=[ 1053], 95.00th=[ 1053], | 99.00th=[ 1053], 99.50th=[ 1053], 99.90th=[ 1053], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 2048, max= 6131, per=32.78%, avg=4083.33, stdev=2041.53, samples=3 iops : min= 2, max= 5, avg= 3.33, stdev= 1.53, samples=3 lat (msec) : 20=25.00%, 50=33.33%, 250=25.00%, 500=8.33% cpu : usr=0.09%, sys=5.32%, ctx=98, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8897KiB/s (9110kB/s), 900KiB/s-3114KiB/s (921kB/s-3189kB/s), io=20.0MiB (20.0MB), run=2264-2302msec WRITE: bw=12.2MiB/s (12.8MB/s), 2224KiB/s-4499KiB/s (2278kB/s-4607kB/s), io=28.0MiB (29.4MB), run=2264-2302msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=18792: Tue Oct 1 05:27:41 2024 read: IOPS=2, BW=9706KiB/s (9939kB/s)(12.0MiB/1266msec) clat (msec): min=31, max=850, avg=415.63, stdev=411.88 lat (msec): min=31, max=850, avg=415.63, stdev=411.88 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 368], 50.00th=[ 368], 60.00th=[ 368], | 70.00th=[ 852], 80.00th=[ 852], 90.00th=[ 852], 95.00th=[ 852], | 99.00th=[ 852], 99.50th=[ 852], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 8175, max= 8175, per=74.80%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=28.46%, ctx=478, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18793: Tue Oct 1 05:27:41 2024 read: IOPS=0, BW=2732KiB/s (2798kB/s)(4096KiB/1499msec) clat (nsec): min=292146k, max=292146k, avg=292146383.00, stdev= 0.00 lat (nsec): min=292148k, max=292148k, avg=292148395.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 292], 10.00th=[ 292], 20.00th=[ 292], | 30.00th=[ 292], 40.00th=[ 292], 50.00th=[ 292], 60.00th=[ 292], | 70.00th=[ 292], 80.00th=[ 292], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 8192, max= 8192, per=74.96%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5465KiB/s (5596kB/s)(8192KiB/1499msec) clat (msec): min=582, max=612, avg=597.59, stdev=21.75 lat (msec): min=582, max=613, avg=597.81, stdev=21.84 clat percentiles (msec): | 1.00th=[ 584], 5.00th=[ 584], 10.00th=[ 584], 20.00th=[ 584], | 30.00th=[ 584], 40.00th=[ 584], 50.00th=[ 584], 60.00th=[ 617], | 70.00th=[ 617], 80.00th=[ 617], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 8192, max= 8192, per=37.48%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.00%, sys=22.03%, ctx=123, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18796: Tue Oct 1 05:27:41 2024 write: IOPS=2, BW=9029KiB/s (9245kB/s)(12.0MiB/1361msec) clat (msec): min=95, max=894, avg=450.57, stdev=406.69 lat (msec): min=95, max=894, avg=450.89, stdev=406.69 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 95], 10.00th=[ 95], 20.00th=[ 95], | 30.00th=[ 95], 40.00th=[ 363], 50.00th=[ 363], 60.00th=[ 363], | 70.00th=[ 894], 80.00th=[ 894], 90.00th=[ 894], 95.00th=[ 894], | 99.00th=[ 894], 99.50th=[ 894], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 8192, max= 8192, per=37.48%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=17.65%, ctx=132, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18797: Tue Oct 1 05:27:41 2024 write: IOPS=2, BW=10.4MiB/s (10.9MB/s)(12.0MiB/1157msec) clat (msec): min=122, max=882, avg=376.69, stdev=438.20 lat (msec): min=122, max=882, avg=376.97, stdev=438.19 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 123], 10.00th=[ 123], 20.00th=[ 123], | 30.00th=[ 123], 40.00th=[ 126], 50.00th=[ 126], 60.00th=[ 126], | 70.00th=[ 885], 80.00th=[ 885], 90.00th=[ 885], 95.00th=[ 885], | 99.00th=[ 885], 99.50th=[ 885], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min=16384, max=16384, per=74.95%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.00%, sys=25.52%, ctx=68, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.7MiB/s (11.2MB/s), 2732KiB/s-9706KiB/s (2798kB/s-9939kB/s), io=16.0MiB (16.8MB), run=1266-1499msec WRITE: bw=21.3MiB/s (22.4MB/s), 5465KiB/s-10.4MiB/s (5596kB/s-10.9MB/s), io=32.0MiB (33.6MB), run=1157-1499msec rand-rw: (groupid=0, jobs=1): err= 0: pid=18794: Tue Oct 1 05:27:41 2024 read: IOPS=2, BW=8557KiB/s (8762kB/s)(12.0MiB/1436msec) clat (msec): min=61, max=893, avg=475.93, stdev=415.78 lat (msec): min=61, max=893, avg=475.93, stdev=415.78 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 62], 10.00th=[ 62], 20.00th=[ 62], | 30.00th=[ 62], 40.00th=[ 472], 50.00th=[ 472], 60.00th=[ 472], | 70.00th=[ 894], 80.00th=[ 894], 90.00th=[ 894], 95.00th=[ 894], | 99.00th=[ 894], 99.50th=[ 894], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 8192, max= 8192, per=80.40%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=5.16%, ctx=31, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18795: Tue Oct 1 05:27:41 2024 read: IOPS=0, BW=2547KiB/s (2608kB/s)(4096KiB/1608msec) clat (nsec): min=911086k, max=911086k, avg=911085821.00, stdev= 0.00 lat (nsec): min=911089k, max=911089k, avg=911088646.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 911], 5.00th=[ 911], 10.00th=[ 911], 20.00th=[ 911], | 30.00th=[ 911], 40.00th=[ 911], 50.00th=[ 911], 60.00th=[ 911], | 70.00th=[ 911], 80.00th=[ 911], 90.00th=[ 911], 95.00th=[ 911], | 99.00th=[ 911], 99.50th=[ 911], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 8192, max= 8192, per=80.40%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5095KiB/s (5217kB/s)(8192KiB/1608msec) clat (msec): min=115, max=562, avg=338.80, stdev=316.34 lat (msec): min=115, max=562, avg=339.00, stdev=316.42 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 115], 10.00th=[ 115], 20.00th=[ 115], | 30.00th=[ 115], 40.00th=[ 115], 50.00th=[ 115], 60.00th=[ 567], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] lat (msec) : 250=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.06%, sys=5.10%, ctx=75, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18798: Tue Oct 1 05:27:41 2024 write: IOPS=1, BW=7169KiB/s (7341kB/s)(12.0MiB/1714msec) clat (msec): min=47, max=951, avg=554.52, stdev=461.72 lat (msec): min=48, max=951, avg=554.78, stdev=461.71 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 667], 50.00th=[ 667], 60.00th=[ 667], | 70.00th=[ 953], 80.00th=[ 953], 90.00th=[ 953], 95.00th=[ 953], | 99.00th=[ 953], 99.50th=[ 953], 99.90th=[ 953], 99.95th=[ 953], | 99.99th=[ 953] lat (msec) : 50=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.06%, sys=6.48%, ctx=40, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18799: Tue Oct 1 05:27:41 2024 write: IOPS=2, BW=8581KiB/s (8787kB/s)(12.0MiB/1432msec) clat (msec): min=147, max=999, avg=470.32, stdev=461.68 lat (msec): min=148, max=999, avg=470.59, stdev=461.68 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 148], 10.00th=[ 148], 20.00th=[ 148], | 30.00th=[ 148], 40.00th=[ 264], 50.00th=[ 264], 60.00th=[ 264], | 70.00th=[ 1003], 80.00th=[ 1003], 90.00th=[ 1003], 95.00th=[ 1003], | 99.00th=[ 1003], 99.50th=[ 1003], 99.90th=[ 1003], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 8192, max= 8192, per=42.85%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=9.99%, ctx=93, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9.95MiB/s (10.4MB/s), 2547KiB/s-8557KiB/s (2608kB/s-8762kB/s), io=16.0MiB (16.8MB), run=1436-1608msec WRITE: bw=18.7MiB/s (19.6MB/s), 5095KiB/s-8581KiB/s (5217kB/s-8787kB/s), io=32.0MiB (33.6MB), run=1432-1714msec