== sanity test 398b: DIO and buffer IO race ============== 13:59:28 (1714413568) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.48085 s, 34.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9510: Mon Apr 29 13:59:46 2024 read: IOPS=98, BW=396KiB/s (405kB/s)(5992KiB/15145msec) clat (usec): min=43, max=233655, avg=5800.45, stdev=6807.27 lat (usec): min=44, max=233656, avg=5801.93, stdev=6807.41 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 116], 10.00th=[ 2671], 20.00th=[ 3294], | 30.00th=[ 3818], 40.00th=[ 4359], 50.00th=[ 5080], 60.00th=[ 5866], | 70.00th=[ 6783], 80.00th=[ 7767], 90.00th=[ 9372], 95.00th=[ 11076], | 99.00th=[ 17695], 99.50th=[ 20579], 99.90th=[ 40633], 99.95th=[233833], | 99.99th=[233833] bw ( KiB/s): min= 184, max= 574, per=24.94%, avg=391.80, stdev=74.21, samples=30 iops : min= 46, max= 143, avg=97.83, stdev=18.54, samples=30 write: IOPS=103, BW=416KiB/s (426kB/s)(6296KiB/15145msec) clat (usec): min=409, max=25131, avg=4053.44, stdev=3838.10 lat (usec): min=409, max=25132, avg=4055.27, stdev=3837.67 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 603], 10.00th=[ 685], 20.00th=[ 783], | 30.00th=[ 906], 40.00th=[ 1598], 50.00th=[ 3097], 60.00th=[ 3982], | 70.00th=[ 5145], 80.00th=[ 6915], 90.00th=[ 9372], 95.00th=[11600], | 99.00th=[16909], 99.50th=[20317], 99.90th=[22414], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 216, max= 624, per=24.94%, avg=410.20, stdev=87.42, samples=30 iops : min= 54, max= 156, avg=102.43, stdev=21.87, samples=30 lat (usec) : 50=0.03%, 100=2.08%, 250=0.65%, 500=0.75%, 750=7.29% lat (usec) : 1000=8.85% lat (msec) : 2=4.72%, 4=23.18%, 10=44.37%, 20=7.49%, 50=0.55% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=23.10%, ctx=6232, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9511: Mon Apr 29 13:59:46 2024 read: IOPS=97, BW=392KiB/s (401kB/s)(5940KiB/15163msec) clat (usec): min=49, max=28140, avg=5577.46, stdev=3208.43 lat (usec): min=50, max=28140, avg=5578.21, stdev=3208.41 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 105], 10.00th=[ 2638], 20.00th=[ 3261], | 30.00th=[ 3720], 40.00th=[ 4359], 50.00th=[ 5014], 60.00th=[ 5800], | 70.00th=[ 6718], 80.00th=[ 7898], 90.00th=[ 9241], 95.00th=[11076], | 99.00th=[15926], 99.50th=[19530], 99.90th=[23987], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 184, max= 504, per=24.74%, avg=388.70, stdev=69.07, samples=30 iops : min= 46, max= 126, avg=97.10, stdev=17.28, samples=30 write: IOPS=104, BW=419KiB/s (429kB/s)(6348KiB/15163msec) clat (usec): min=436, max=202138, avg=4289.67, stdev=6304.26 lat (usec): min=437, max=202138, avg=4291.24, stdev=6304.43 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 619], 10.00th=[ 693], 20.00th=[ 807], | 30.00th=[ 996], 40.00th=[ 1795], 50.00th=[ 3359], 60.00th=[ 4178], | 70.00th=[ 5342], 80.00th=[ 7046], 90.00th=[ 9503], 95.00th=[ 11600], | 99.00th=[ 15401], 99.50th=[ 18220], 99.90th=[ 41681], 99.95th=[202376], | 99.99th=[202376] bw ( KiB/s): min= 232, max= 520, per=25.04%, avg=411.90, stdev=75.35, samples=30 iops : min= 58, max= 130, avg=102.90, stdev=18.86, samples=30 lat (usec) : 50=0.07%, 100=2.21%, 250=0.52%, 500=0.62%, 750=6.87% lat (usec) : 1000=8.30% lat (msec) : 2=5.37%, 4=22.95%, 10=44.82%, 20=7.85%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=23.00%, ctx=6329, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9512: Mon Apr 29 13:59:46 2024 read: IOPS=98, BW=394KiB/s (404kB/s)(6028KiB/15282msec) clat (usec): min=47, max=27720, avg=5528.59, stdev=3284.42 lat (usec): min=48, max=27721, avg=5530.06, stdev=3284.57 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 117], 10.00th=[ 2638], 20.00th=[ 3228], | 30.00th=[ 3720], 40.00th=[ 4228], 50.00th=[ 4948], 60.00th=[ 5800], | 70.00th=[ 6587], 80.00th=[ 7635], 90.00th=[ 9241], 95.00th=[10814], | 99.00th=[16450], 99.50th=[22414], 99.90th=[26084], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 192, max= 568, per=24.88%, avg=390.80, stdev=82.97, samples=30 iops : min= 48, max= 142, avg=97.63, stdev=20.75, samples=30 write: IOPS=102, BW=410KiB/s (419kB/s)(6260KiB/15282msec) clat (usec): min=419, max=212205, avg=4385.23, stdev=6496.98 lat (usec): min=419, max=212207, avg=4387.14, stdev=6496.81 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 594], 10.00th=[ 717], 20.00th=[ 807], | 30.00th=[ 979], 40.00th=[ 2180], 50.00th=[ 3556], 60.00th=[ 4359], | 70.00th=[ 5604], 80.00th=[ 7242], 90.00th=[ 9372], 95.00th=[ 11863], | 99.00th=[ 16319], 99.50th=[ 19006], 99.90th=[ 23200], 99.95th=[212861], | 99.99th=[212861] bw ( KiB/s): min= 256, max= 528, per=24.60%, avg=404.70, stdev=77.76, samples=30 iops : min= 64, max= 132, avg=101.10, stdev=19.43, samples=30 lat (usec) : 50=0.07%, 100=1.82%, 250=0.94%, 500=0.59%, 750=6.22% lat (usec) : 1000=8.85% lat (msec) : 2=4.56%, 4=23.21%, 10=46.09%, 20=7.16%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=23.19%, ctx=6268, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9513: Mon Apr 29 13:59:46 2024 read: IOPS=100, BW=401KiB/s (410kB/s)(6052KiB/15100msec) clat (usec): min=43, max=205520, avg=5587.82, stdev=6051.07 lat (usec): min=43, max=205521, avg=5589.81, stdev=6051.36 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 338], 10.00th=[ 2704], 20.00th=[ 3195], | 30.00th=[ 3654], 40.00th=[ 4146], 50.00th=[ 4817], 60.00th=[ 5538], | 70.00th=[ 6390], 80.00th=[ 7504], 90.00th=[ 8979], 95.00th=[ 11076], | 99.00th=[ 17171], 99.50th=[ 19268], 99.90th=[ 31589], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 224, max= 502, per=25.33%, avg=397.93, stdev=62.18, samples=30 iops : min= 56, max= 125, avg=99.37, stdev=15.56, samples=30 write: IOPS=103, BW=413KiB/s (423kB/s)(6236KiB/15100msec) clat (usec): min=387, max=29595, avg=4210.54, stdev=3921.11 lat (usec): min=388, max=29597, avg=4212.10, stdev=3921.48 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 611], 10.00th=[ 693], 20.00th=[ 791], | 30.00th=[ 971], 40.00th=[ 1926], 50.00th=[ 3326], 60.00th=[ 4228], | 70.00th=[ 5407], 80.00th=[ 7046], 90.00th=[ 9503], 95.00th=[11863], | 99.00th=[17695], 99.50th=[19530], 99.90th=[26870], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 248, max= 584, per=25.00%, avg=411.27, stdev=82.16, samples=30 iops : min= 62, max= 146, avg=102.70, stdev=20.55, samples=30 lat (usec) : 50=0.07%, 100=1.89%, 250=0.49%, 500=0.62%, 750=6.97% lat (usec) : 1000=8.14% lat (msec) : 2=4.95%, 4=24.32%, 10=44.79%, 20=7.39%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=22.91%, ctx=6176, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1571KiB/s (1609kB/s), 392KiB/s-401KiB/s (401kB/s-410kB/s), io=23.4MiB (24.6MB), run=15100-15282msec WRITE: bw=1645KiB/s (1685kB/s), 410KiB/s-419KiB/s (419kB/s-429kB/s), io=24.6MiB (25.7MB), run=15100-15282msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9514: Mon Apr 29 13:59:50 2024 read: IOPS=78, BW=314KiB/s (321kB/s)(5992KiB/19085msec) clat (usec): min=1478, max=242477, avg=5071.61, stdev=7450.24 lat (usec): min=1478, max=242479, avg=5072.35, stdev=7450.30 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1860], 10.00th=[ 1975], 20.00th=[ 2180], | 30.00th=[ 2376], 40.00th=[ 2704], 50.00th=[ 3195], 60.00th=[ 3851], | 70.00th=[ 4883], 80.00th=[ 7046], 90.00th=[ 11207], 95.00th=[ 14091], | 99.00th=[ 19792], 99.50th=[ 21890], 99.90th=[ 51119], 99.95th=[242222], | 99.99th=[242222] bw ( KiB/s): min= 111, max= 768, per=24.79%, avg=311.92, stdev=185.44, samples=38 iops : min= 27, max= 192, avg=77.79, stdev=46.44, samples=38 write: IOPS=82, BW=330KiB/s (338kB/s)(6296KiB/19085msec) clat (usec): min=2065, max=37970, avg=7259.48, stdev=5002.12 lat (usec): min=2065, max=37971, avg=7260.38, stdev=5002.20 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2868], 10.00th=[ 3130], 20.00th=[ 3425], | 30.00th=[ 3752], 40.00th=[ 4228], 50.00th=[ 5276], 60.00th=[ 6718], | 70.00th=[ 8586], 80.00th=[10814], 90.00th=[14091], 95.00th=[16712], | 99.00th=[26346], 99.50th=[29492], 99.90th=[35390], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 135, max= 776, per=24.87%, avg=327.50, stdev=187.32, samples=38 iops : min= 33, max= 194, avg=81.68, stdev=46.90, samples=38 lat (msec) : 2=5.66%, 4=43.10%, 10=33.53%, 20=15.98%, 50=1.66% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.20%, sys=10.68%, ctx=3715, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9515: Mon Apr 29 13:59:50 2024 read: IOPS=77, BW=312KiB/s (319kB/s)(5940KiB/19060msec) clat (usec): min=1522, max=207102, avg=5079.30, stdev=6734.59 lat (usec): min=1522, max=207103, avg=5080.09, stdev=6734.58 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1876], 10.00th=[ 1991], 20.00th=[ 2147], | 30.00th=[ 2376], 40.00th=[ 2737], 50.00th=[ 3261], 60.00th=[ 3851], | 70.00th=[ 4948], 80.00th=[ 6915], 90.00th=[ 11338], 95.00th=[ 14091], | 99.00th=[ 19006], 99.50th=[ 20317], 99.90th=[ 52167], 99.95th=[206570], | 99.99th=[206570] bw ( KiB/s): min= 80, max= 720, per=24.70%, avg=310.68, stdev=184.21, samples=38 iops : min= 20, max= 180, avg=77.63, stdev=46.06, samples=38 write: IOPS=83, BW=333KiB/s (341kB/s)(6348KiB/19060msec) clat (usec): min=2323, max=36818, avg=7218.45, stdev=4852.36 lat (usec): min=2324, max=36819, avg=7219.40, stdev=4852.38 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2900], 10.00th=[ 3130], 20.00th=[ 3425], | 30.00th=[ 3785], 40.00th=[ 4359], 50.00th=[ 5276], 60.00th=[ 6718], | 70.00th=[ 8717], 80.00th=[10683], 90.00th=[13829], 95.00th=[16909], | 99.00th=[24511], 99.50th=[26608], 99.90th=[33424], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 144, max= 728, per=25.16%, avg=331.32, stdev=183.13, samples=38 iops : min= 36, max= 182, avg=82.79, stdev=45.81, samples=38 lat (msec) : 2=5.18%, 4=42.58%, 10=34.41%, 20=16.24%, 50=1.53% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.14%, sys=10.81%, ctx=3769, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9516: Mon Apr 29 13:59:50 2024 read: IOPS=79, BW=317KiB/s (324kB/s)(6028KiB/19045msec) clat (usec): min=1495, max=28556, avg=4987.01, stdev=4050.28 lat (usec): min=1495, max=28557, avg=4987.74, stdev=4050.30 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1893], 10.00th=[ 1991], 20.00th=[ 2180], | 30.00th=[ 2376], 40.00th=[ 2737], 50.00th=[ 3228], 60.00th=[ 3851], | 70.00th=[ 5211], 80.00th=[ 7439], 90.00th=[11076], 95.00th=[14222], | 99.00th=[18744], 99.50th=[19792], 99.90th=[27132], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 104, max= 776, per=25.09%, avg=315.61, stdev=180.48, samples=38 iops : min= 26, max= 194, avg=78.79, stdev=45.15, samples=38 write: IOPS=82, BW=329KiB/s (337kB/s)(6260KiB/19045msec) clat (msec): min=2, max=215, avg= 7.33, stdev= 7.16 lat (msec): min=2, max=215, avg= 7.33, stdev= 7.16 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 14], 95.00th=[ 17], | 99.00th=[ 24], 99.50th=[ 30], 99.90th=[ 40], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 128, max= 792, per=24.84%, avg=327.18, stdev=187.60, samples=38 iops : min= 32, max= 198, avg=81.68, stdev=46.93, samples=38 lat (msec) : 2=4.95%, 4=42.74%, 10=34.64%, 20=16.37%, 50=1.27% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=10.86%, ctx=3702, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9517: Mon Apr 29 13:59:50 2024 read: IOPS=79, BW=319KiB/s (326kB/s)(6052KiB/18983msec) clat (usec): min=1484, max=31051, avg=4852.73, stdev=4091.98 lat (usec): min=1484, max=31052, avg=4853.59, stdev=4092.09 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1893], 10.00th=[ 1991], 20.00th=[ 2180], | 30.00th=[ 2376], 40.00th=[ 2671], 50.00th=[ 3163], 60.00th=[ 3818], | 70.00th=[ 4752], 80.00th=[ 6783], 90.00th=[10552], 95.00th=[14091], | 99.00th=[20317], 99.50th=[21890], 99.90th=[28443], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 112, max= 728, per=24.57%, avg=309.08, stdev=172.41, samples=37 iops : min= 28, max= 182, avg=77.19, stdev=43.14, samples=37 write: IOPS=82, BW=329KiB/s (336kB/s)(6236KiB/18983msec) clat (msec): min=2, max=215, avg= 7.43, stdev= 7.26 lat (msec): min=2, max=215, avg= 7.43, stdev= 7.26 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 15], 95.00th=[ 17], | 99.00th=[ 26], 99.50th=[ 30], 99.90th=[ 40], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 128, max= 784, per=24.19%, avg=318.59, stdev=176.99, samples=37 iops : min= 32, max= 196, avg=79.57, stdev=44.30, samples=37 lat (msec) : 2=5.31%, 4=42.97%, 10=34.18%, 20=15.82%, 50=1.69% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=10.77%, ctx=3664, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1258KiB/s (1288kB/s), 312KiB/s-319KiB/s (319kB/s-326kB/s), io=23.4MiB (24.6MB), run=18983-19085msec WRITE: bw=1317KiB/s (1349kB/s), 329KiB/s-333KiB/s (336kB/s-341kB/s), io=24.6MiB (25.7MB), run=18983-19085msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9548: Mon Apr 29 13:59:57 2024 read: IOPS=55, BW=891KiB/s (912kB/s)(5920KiB/6644msec) clat (usec): min=55, max=102013, avg=12932.50, stdev=5956.11 lat (usec): min=55, max=102014, avg=12933.27, stdev=5956.10 clat percentiles (usec): | 1.00th=[ 97], 5.00th=[ 8225], 10.00th=[ 9372], 20.00th=[ 10159], | 30.00th=[ 10945], 40.00th=[ 11731], 50.00th=[ 12518], 60.00th=[ 13566], | 70.00th=[ 14484], 80.00th=[ 15139], 90.00th=[ 16319], 95.00th=[ 18220], | 99.00th=[ 25560], 99.50th=[ 30802], 99.90th=[102237], 99.95th=[102237], | 99.99th=[102237] bw ( KiB/s): min= 672, max= 1024, per=25.10%, avg=880.92, stdev=94.66, samples=13 iops : min= 42, max= 64, avg=54.92, stdev= 5.88, samples=13 write: IOPS=59, BW=958KiB/s (981kB/s)(6368KiB/6644msec) clat (usec): min=747, max=30563, avg=4648.18, stdev=4466.13 lat (usec): min=748, max=30565, avg=4650.06, stdev=4467.37 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 914], 10.00th=[ 1004], 20.00th=[ 1270], | 30.00th=[ 2278], 40.00th=[ 2933], 50.00th=[ 3458], 60.00th=[ 3916], | 70.00th=[ 5014], 80.00th=[ 6587], 90.00th=[ 9896], 95.00th=[14484], | 99.00th=[24249], 99.50th=[28443], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 479, max= 1312, per=25.99%, avg=959.77, stdev=241.78, samples=13 iops : min= 29, max= 82, avg=59.85, stdev=15.28, samples=13 lat (usec) : 100=0.65%, 250=0.39%, 750=0.13%, 1000=4.82% lat (msec) : 2=9.37%, 4=17.84%, 10=22.01%, 20=42.58%, 50=2.08% lat (msec) : 250=0.13% cpu : usr=0.18%, sys=20.32%, ctx=2645, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9549: Mon Apr 29 13:59:57 2024 read: IOPS=52, BW=844KiB/s (864kB/s)(5744KiB/6809msec) clat (usec): min=56, max=31325, avg=12699.30, stdev=3899.00 lat (usec): min=56, max=31325, avg=12700.07, stdev=3899.01 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 7767], 10.00th=[ 8848], 20.00th=[10159], | 30.00th=[10945], 40.00th=[11731], 50.00th=[12518], 60.00th=[13304], | 70.00th=[14353], 80.00th=[15401], 90.00th=[16450], 95.00th=[18482], | 99.00th=[25822], 99.50th=[29754], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 480, max= 1024, per=23.35%, avg=819.54, stdev=140.26, samples=13 iops : min= 30, max= 64, avg=51.15, stdev= 8.70, samples=13 write: IOPS=60, BW=961KiB/s (984kB/s)(6544KiB/6809msec) clat (usec): min=744, max=142249, avg=5480.29, stdev=8308.13 lat (usec): min=746, max=142250, avg=5482.05, stdev=8307.98 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 906], 10.00th=[ 1037], 20.00th=[ 1254], | 30.00th=[ 2671], 40.00th=[ 3163], 50.00th=[ 3556], 60.00th=[ 4293], | 70.00th=[ 5538], 80.00th=[ 7373], 90.00th=[ 12125], 95.00th=[ 17171], | 99.00th=[ 22414], 99.50th=[ 25822], 99.90th=[141558], 99.95th=[141558], | 99.99th=[141558] bw ( KiB/s): min= 640, max= 1312, per=24.92%, avg=920.46, stdev=192.03, samples=13 iops : min= 40, max= 82, avg=57.46, stdev=11.99, samples=13 lat (usec) : 100=0.78%, 250=0.26%, 750=0.39%, 1000=4.17% lat (msec) : 2=8.33%, 4=17.97%, 10=22.92%, 20=42.45%, 50=2.60% lat (msec) : 250=0.13% cpu : usr=0.18%, sys=18.79%, ctx=2642, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9550: Mon Apr 29 13:59:57 2024 read: IOPS=57, BW=917KiB/s (939kB/s)(6256KiB/6823msec) clat (usec): min=55, max=24019, avg=12600.28, stdev=4060.69 lat (usec): min=55, max=24020, avg=12601.07, stdev=4060.69 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 7046], 10.00th=[ 8848], 20.00th=[10290], | 30.00th=[11338], 40.00th=[11994], 50.00th=[12518], 60.00th=[13304], | 70.00th=[14222], 80.00th=[15533], 90.00th=[17171], 95.00th=[19006], | 99.00th=[22414], 99.50th=[23987], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 608, max= 1184, per=25.53%, avg=896.00, stdev=125.98, samples=13 iops : min= 38, max= 74, avg=56.00, stdev= 7.87, samples=13 write: IOPS=55, BW=884KiB/s (905kB/s)(6032KiB/6823msec) clat (usec): min=590, max=142535, avg=5006.74, stdev=8270.03 lat (usec): min=591, max=142536, avg=5008.34, stdev=8269.98 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 865], 10.00th=[ 988], 20.00th=[ 1172], | 30.00th=[ 2180], 40.00th=[ 2835], 50.00th=[ 3359], 60.00th=[ 3949], | 70.00th=[ 5407], 80.00th=[ 7111], 90.00th=[ 10159], 95.00th=[ 15533], | 99.00th=[ 19792], 99.50th=[ 27919], 99.90th=[141558], 99.95th=[141558], | 99.99th=[141558] bw ( KiB/s): min= 576, max= 1056, per=22.93%, avg=846.77, stdev=181.31, samples=13 iops : min= 36, max= 66, avg=52.92, stdev=11.33, samples=13 lat (usec) : 100=1.56%, 250=0.65%, 750=0.78%, 1000=4.82% lat (msec) : 2=8.72%, 4=15.62%, 10=20.83%, 20=44.92%, 50=1.95% lat (msec) : 250=0.13% cpu : usr=0.10%, sys=18.72%, ctx=2682, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9551: Mon Apr 29 13:59:57 2024 read: IOPS=56, BW=898KiB/s (919kB/s)(6032KiB/6718msec) clat (usec): min=58, max=26620, avg=12215.46, stdev=4114.74 lat (usec): min=58, max=26620, avg=12216.50, stdev=4114.72 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 186], 10.00th=[ 8586], 20.00th=[ 9896], | 30.00th=[10814], 40.00th=[11600], 50.00th=[12387], 60.00th=[13173], | 70.00th=[13960], 80.00th=[15008], 90.00th=[16909], 95.00th=[18220], | 99.00th=[22152], 99.50th=[24511], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 576, max= 1088, per=25.03%, avg=878.62, stdev=127.72, samples=13 iops : min= 36, max= 68, avg=54.85, stdev= 7.98, samples=13 write: IOPS=58, BW=931KiB/s (954kB/s)(6256KiB/6718msec) clat (usec): min=677, max=137475, avg=5373.05, stdev=8240.95 lat (usec): min=679, max=137477, avg=5374.51, stdev=8240.94 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 873], 10.00th=[ 1004], 20.00th=[ 1221], | 30.00th=[ 2573], 40.00th=[ 3130], 50.00th=[ 3687], 60.00th=[ 4359], | 70.00th=[ 5735], 80.00th=[ 7373], 90.00th=[ 10683], 95.00th=[ 15270], | 99.00th=[ 26608], 99.50th=[ 34866], 99.90th=[137364], 99.95th=[137364], | 99.99th=[137364] bw ( KiB/s): min= 544, max= 1248, per=24.92%, avg=920.46, stdev=210.33, samples=13 iops : min= 34, max= 78, avg=57.46, stdev=13.16, samples=13 lat (usec) : 100=1.17%, 250=1.30%, 750=0.78%, 1000=4.30% lat (msec) : 2=7.81%, 4=16.15%, 10=24.61%, 20=41.28%, 50=2.47% lat (msec) : 250=0.13% cpu : usr=0.18%, sys=18.36%, ctx=2621, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3510KiB/s (3595kB/s), 844KiB/s-917KiB/s (864kB/s-939kB/s), io=23.4MiB (24.5MB), run=6644-6823msec WRITE: bw=3693KiB/s (3782kB/s), 884KiB/s-961KiB/s (905kB/s-984kB/s), io=24.6MiB (25.8MB), run=6644-6823msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9544: Mon Apr 29 13:59:57 2024 read: IOPS=59, BW=954KiB/s (977kB/s)(5920KiB/6204msec) clat (usec): min=2022, max=96305, avg=6286.10, stdev=6155.30 lat (usec): min=2023, max=96306, avg=6286.95, stdev=6155.28 clat percentiles (usec): | 1.00th=[ 2180], 5.00th=[ 2474], 10.00th=[ 2638], 20.00th=[ 2999], | 30.00th=[ 3359], 40.00th=[ 3818], 50.00th=[ 4424], 60.00th=[ 5342], | 70.00th=[ 6783], 80.00th=[ 9241], 90.00th=[12387], 95.00th=[14615], | 99.00th=[20055], 99.50th=[23725], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 672, max= 1344, per=27.20%, avg=946.67, stdev=191.74, samples=12 iops : min= 42, max= 84, avg=59.17, stdev=11.98, samples=12 write: IOPS=64, BW=1026KiB/s (1051kB/s)(6368KiB/6204msec) clat (usec): min=3024, max=49306, avg=9720.17, stdev=6082.98 lat (usec): min=3026, max=49307, avg=9721.87, stdev=6082.85 clat percentiles (usec): | 1.00th=[ 3294], 5.00th=[ 3949], 10.00th=[ 4293], 20.00th=[ 4817], | 30.00th=[ 5538], 40.00th=[ 6325], 50.00th=[ 7504], 60.00th=[ 9110], | 70.00th=[11469], 80.00th=[14222], 90.00th=[18482], 95.00th=[21103], | 99.00th=[29230], 99.50th=[33817], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 896, max= 1184, per=28.11%, avg=1029.33, stdev=101.96, samples=12 iops : min= 56, max= 74, avg=64.33, stdev= 6.37, samples=12 lat (msec) : 4=24.22%, 10=47.79%, 20=24.35%, 50=3.52%, 100=0.13% cpu : usr=0.16%, sys=9.70%, ctx=951, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9545: Mon Apr 29 13:59:57 2024 read: IOPS=56, BW=912KiB/s (934kB/s)(5744KiB/6299msec) clat (usec): min=2071, max=24852, avg=6617.33, stdev=4517.12 lat (usec): min=2072, max=24853, avg=6618.17, stdev=4517.12 clat percentiles (usec): | 1.00th=[ 2212], 5.00th=[ 2442], 10.00th=[ 2638], 20.00th=[ 2966], | 30.00th=[ 3425], 40.00th=[ 3949], 50.00th=[ 4621], 60.00th=[ 6194], | 70.00th=[ 7963], 80.00th=[10421], 90.00th=[13304], 95.00th=[15926], | 99.00th=[20841], 99.50th=[21890], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 512, max= 1280, per=26.35%, avg=917.08, stdev=244.82, samples=12 iops : min= 32, max= 80, avg=57.25, stdev=15.26, samples=12 write: IOPS=64, BW=1039KiB/s (1064kB/s)(6544KiB/6299msec) clat (msec): min=2, max=143, avg= 9.57, stdev= 8.80 lat (msec): min=2, max=143, avg= 9.57, stdev= 8.80 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 18], 95.00th=[ 21], | 99.00th=[ 28], 99.50th=[ 31], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 800, max= 1216, per=27.96%, avg=1023.75, stdev=129.90, samples=12 iops : min= 50, max= 76, avg=63.92, stdev= 8.05, samples=12 lat (msec) : 4=24.22%, 10=47.79%, 20=24.61%, 50=3.12%, 100=0.13% lat (msec) : 250=0.13% cpu : usr=0.17%, sys=9.51%, ctx=943, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9546: Mon Apr 29 13:59:57 2024 read: IOPS=56, BW=909KiB/s (931kB/s)(6256KiB/6881msec) clat (usec): min=1854, max=34428, avg=6978.28, stdev=4998.72 lat (usec): min=1855, max=34429, avg=6979.11, stdev=4998.73 clat percentiles (usec): | 1.00th=[ 2089], 5.00th=[ 2409], 10.00th=[ 2671], 20.00th=[ 2999], | 30.00th=[ 3425], 40.00th=[ 4015], 50.00th=[ 4948], 60.00th=[ 6521], | 70.00th=[ 8717], 80.00th=[11076], 90.00th=[13960], 95.00th=[16450], | 99.00th=[21365], 99.50th=[33424], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 576, max= 1344, per=25.53%, avg=888.62, stdev=180.15, samples=13 iops : min= 36, max= 84, avg=55.54, stdev=11.26, samples=13 write: IOPS=54, BW=877KiB/s (898kB/s)(6032KiB/6881msec) clat (msec): min=2, max=135, avg=10.99, stdev= 8.50 lat (msec): min=2, max=135, avg=10.99, stdev= 8.50 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 13], 80.00th=[ 15], 90.00th=[ 19], 95.00th=[ 22], | 99.00th=[ 32], 99.50th=[ 34], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 576, max= 1088, per=22.85%, avg=836.92, stdev=173.24, samples=13 iops : min= 36, max= 68, avg=52.31, stdev=10.83, samples=13 lat (msec) : 2=0.13%, 4=22.92%, 10=40.76%, 20=31.90%, 50=4.17% lat (msec) : 250=0.13% cpu : usr=0.15%, sys=10.09%, ctx=1163, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9547: Mon Apr 29 13:59:57 2024 read: IOPS=55, BW=892KiB/s (914kB/s)(6032KiB/6760msec) clat (usec): min=2045, max=29742, avg=6925.35, stdev=4766.05 lat (usec): min=2045, max=29743, avg=6926.20, stdev=4766.05 clat percentiles (usec): | 1.00th=[ 2245], 5.00th=[ 2573], 10.00th=[ 2737], 20.00th=[ 3097], | 30.00th=[ 3458], 40.00th=[ 4080], 50.00th=[ 4752], 60.00th=[ 6456], | 70.00th=[ 8586], 80.00th=[10814], 90.00th=[13829], 95.00th=[16581], | 99.00th=[20841], 99.50th=[27919], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 544, max= 1024, per=24.97%, avg=868.92, stdev=159.92, samples=13 iops : min= 34, max= 64, avg=54.31, stdev= 9.99, samples=13 write: IOPS=57, BW=925KiB/s (948kB/s)(6256KiB/6760msec) clat (msec): min=3, max=143, avg=10.59, stdev= 8.67 lat (msec): min=3, max=143, avg=10.59, stdev= 8.67 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 15], 90.00th=[ 18], 95.00th=[ 21], | 99.00th=[ 32], 99.50th=[ 35], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 640, max= 1152, per=24.80%, avg=908.31, stdev=160.29, samples=13 iops : min= 40, max= 72, avg=56.77, stdev=10.02, samples=13 lat (msec) : 4=22.01%, 10=42.84%, 20=31.77%, 50=3.26%, 250=0.13% cpu : usr=0.24%, sys=9.91%, ctx=1087, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3481KiB/s (3564kB/s), 892KiB/s-954KiB/s (914kB/s-977kB/s), io=23.4MiB (24.5MB), run=6204-6881msec WRITE: bw=3662KiB/s (3750kB/s), 877KiB/s-1039KiB/s (898kB/s-1064kB/s), io=24.6MiB (25.8MB), run=6204-6881msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9561: Mon Apr 29 14:00:01 2024 read: IOPS=2, BW=3033KiB/s (3106kB/s)(7168KiB/2363msec) clat (usec): min=621, max=686439, avg=310223.12, stdev=274784.31 lat (usec): min=623, max=686441, avg=310224.70, stdev=274784.35 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 619], 10.00th=[ 619], 20.00th=[ 40633], | 30.00th=[ 55313], 40.00th=[ 55313], 50.00th=[392168], 60.00th=[484443], | 70.00th=[484443], 80.00th=[513803], 90.00th=[683672], 95.00th=[683672], | 99.00th=[683672], 99.50th=[683672], 99.90th=[683672], 99.95th=[683672], | 99.99th=[683672] bw ( KiB/s): min= 1747, max= 4096, per=27.81%, avg=2410.25, stdev=1132.67, samples=4 iops : min= 1, max= 4, avg= 2.00, stdev= 1.41, samples=4 write: IOPS=2, BW=2167KiB/s (2219kB/s)(5120KiB/2363msec) clat (usec): min=28424, max=66624, avg=37625.47, stdev=16265.28 lat (usec): min=28484, max=66727, avg=37690.26, stdev=16285.60 clat percentiles (usec): | 1.00th=[28443], 5.00th=[28443], 10.00th=[28443], 20.00th=[28443], | 30.00th=[30278], 40.00th=[30278], 50.00th=[30540], 60.00th=[30540], | 70.00th=[32113], 80.00th=[32113], 90.00th=[66847], 95.00th=[66847], | 99.00th=[66847], 99.50th=[66847], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 2048, max= 5251, per=30.08%, avg=3649.50, stdev=2264.86, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (usec) : 750=8.33% lat (msec) : 50=41.67%, 100=16.67%, 500=16.67%, 750=16.67% cpu : usr=0.04%, sys=21.59%, ctx=1156, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9562: Mon Apr 29 14:00:01 2024 read: IOPS=0, BW=877KiB/s (898kB/s)(2048KiB/2336msec) clat (msec): min=57, max=527, avg=292.34, stdev=332.33 lat (msec): min=57, max=527, avg=292.34, stdev=332.33 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 58], 10.00th=[ 58], 20.00th=[ 58], | 30.00th=[ 58], 40.00th=[ 58], 50.00th=[ 58], 60.00th=[ 527], | 70.00th=[ 527], 80.00th=[ 527], 90.00th=[ 527], 95.00th=[ 527], | 99.00th=[ 527], 99.50th=[ 527], 99.90th=[ 527], 99.95th=[ 527], | 99.99th=[ 527] bw ( KiB/s): min= 2048, max= 2048, per=23.63%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4384KiB/s (4489kB/s)(10.0MiB/2336msec) clat (msec): min=25, max=533, avg=174.43, stdev=197.85 lat (msec): min=25, max=533, avg=174.49, stdev=197.85 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 47], | 30.00th=[ 50], 40.00th=[ 51], 50.00th=[ 53], 60.00th=[ 55], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 422], 95.00th=[ 535], | 99.00th=[ 535], 99.50th=[ 535], 99.90th=[ 535], 99.95th=[ 535], | 99.99th=[ 535] bw ( KiB/s): min= 3482, max=10240, per=48.95%, avg=5939.33, stdev=3737.12, samples=3 iops : min= 3, max= 10, avg= 5.67, stdev= 3.79, samples=3 lat (msec) : 50=33.33%, 100=33.33%, 500=16.67%, 750=16.67% cpu : usr=0.00%, sys=14.09%, ctx=360, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9563: Mon Apr 29 14:00:01 2024 read: IOPS=3, BW=3279KiB/s (3357kB/s)(6144KiB/1874msec) clat (usec): min=509, max=541189, avg=192817.70, stdev=257227.11 lat (usec): min=511, max=541190, avg=192819.15, stdev=257226.81 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 510], 10.00th=[ 510], 20.00th=[ 594], | 30.00th=[ 594], 40.00th=[ 33817], 50.00th=[ 33817], 60.00th=[ 77071], | 70.00th=[505414], 80.00th=[505414], 90.00th=[541066], 95.00th=[541066], | 99.00th=[541066], 99.50th=[541066], 99.90th=[541066], 99.95th=[541066], | 99.99th=[541066] bw ( KiB/s): min= 1747, max= 6144, per=38.23%, avg=3313.00, stdev=2456.33, samples=3 iops : min= 1, max= 6, avg= 3.00, stdev= 2.65, samples=3 write: IOPS=3, BW=3279KiB/s (3357kB/s)(6144KiB/1874msec) clat (msec): min=33, max=499, avg=118.90, stdev=186.74 lat (msec): min=33, max=499, avg=118.96, stdev=186.73 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 45], 50.00th=[ 45], 60.00th=[ 47], | 70.00th=[ 52], 80.00th=[ 52], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 2048, max= 8192, per=42.20%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (usec) : 750=16.67% lat (msec) : 50=41.67%, 100=16.67%, 500=8.33%, 750=16.67% cpu : usr=0.00%, sys=16.28%, ctx=627, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9564: Mon Apr 29 14:00:01 2024 read: IOPS=3, BW=3497KiB/s (3581kB/s)(5120KiB/1464msec) clat (usec): min=646, max=548987, avg=207145.49, stdev=266640.56 lat (usec): min=648, max=548988, avg=207146.85, stdev=266640.45 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 644], 10.00th=[ 644], 20.00th=[ 644], | 30.00th=[ 676], 40.00th=[ 676], 50.00th=[ 43254], 60.00th=[ 43254], | 70.00th=[442500], 80.00th=[442500], 90.00th=[549454], 95.00th=[549454], | 99.00th=[549454], 99.50th=[549454], 99.90th=[549454], 99.95th=[549454], | 99.99th=[549454] bw ( KiB/s): min= 2048, max= 2048, per=23.63%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4896KiB/s (5014kB/s)(7168KiB/1464msec) clat (msec): min=25, max=111, avg=60.60, stdev=35.45 lat (msec): min=25, max=111, avg=60.71, stdev=35.44 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 34], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 47], 60.00th=[ 63], | 70.00th=[ 63], 80.00th=[ 108], 90.00th=[ 112], 95.00th=[ 112], | 99.00th=[ 112], 99.50th=[ 112], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 8192, max= 8192, per=67.52%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 50=41.67%, 100=8.33%, 250=16.67%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=22.42%, ctx=603, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8667KiB/s (8875kB/s), 877KiB/s-3497KiB/s (898kB/s-3581kB/s), io=20.0MiB (20.0MB), run=1464-2363msec WRITE: bw=11.8MiB/s (12.4MB/s), 2167KiB/s-4896KiB/s (2219kB/s-5014kB/s), io=28.0MiB (29.4MB), run=1464-2363msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9566: Mon Apr 29 14:00:01 2024 read: IOPS=2, BW=3041KiB/s (3114kB/s)(7168KiB/2357msec) clat (msec): min=19, max=564, avg=243.74, stdev=253.83 lat (msec): min=19, max=564, avg=243.74, stdev=253.83 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 23], | 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 97], 60.00th=[ 485], | 70.00th=[ 485], 80.00th=[ 489], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 1787, max= 4096, per=27.96%, avg=2429.50, stdev=1117.79, samples=4 iops : min= 1, max= 4, avg= 2.00, stdev= 1.41, samples=4 write: IOPS=2, BW=2172KiB/s (2224kB/s)(5120KiB/2357msec) clat (msec): min=19, max=378, avg=123.86, stdev=146.38 lat (msec): min=20, max=378, avg=123.92, stdev=146.38 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 58], 60.00th=[ 58], | 70.00th=[ 112], 80.00th=[ 112], 90.00th=[ 380], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 1787, max= 2048, per=15.76%, avg=1917.50, stdev=184.55, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 20=16.67%, 50=16.67%, 100=25.00%, 250=8.33%, 500=25.00% lat (msec) : 750=8.33% cpu : usr=0.00%, sys=5.43%, ctx=55, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9567: Mon Apr 29 14:00:01 2024 read: IOPS=0, BW=878KiB/s (899kB/s)(2048KiB/2333msec) clat (usec): min=20800, max=79153, avg=49976.86, stdev=41261.58 lat (usec): min=20801, max=79154, avg=49978.19, stdev=41261.57 clat percentiles (usec): | 1.00th=[20841], 5.00th=[20841], 10.00th=[20841], 20.00th=[20841], | 30.00th=[20841], 40.00th=[20841], 50.00th=[20841], 60.00th=[79168], | 70.00th=[79168], 80.00th=[79168], 90.00th=[79168], 95.00th=[79168], | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 1796, max= 2048, per=22.12%, avg=1922.00, stdev=178.19, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4389KiB/s (4495kB/s)(10.0MiB/2333msec) clat (msec): min=27, max=902, avg=220.49, stdev=334.92 lat (msec): min=27, max=902, avg=220.56, stdev=334.93 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 30], | 30.00th=[ 57], 40.00th=[ 66], 50.00th=[ 72], 60.00th=[ 73], | 70.00th=[ 93], 80.00th=[ 93], 90.00th=[ 802], 95.00th=[ 902], | 99.00th=[ 902], 99.50th=[ 902], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 5389, max= 8192, per=55.82%, avg=6790.50, stdev=1982.02, samples=2 iops : min= 5, max= 8, avg= 6.50, stdev= 2.12, samples=2 lat (msec) : 50=25.00%, 100=58.33%, 1000=16.67% cpu : usr=0.00%, sys=5.79%, ctx=55, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9568: Mon Apr 29 14:00:01 2024 read: IOPS=2, BW=2668KiB/s (2732kB/s)(6144KiB/2303msec) clat (msec): min=20, max=867, avg=179.60, stdev=337.25 lat (msec): min=20, max=867, avg=179.60, stdev=337.25 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 48], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 54], 80.00th=[ 54], 90.00th=[ 869], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 4096, max= 6144, per=58.93%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2668KiB/s (2732kB/s)(6144KiB/2303msec) clat (msec): min=36, max=931, avg=199.79, stdev=358.97 lat (msec): min=36, max=931, avg=199.84, stdev=358.98 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 43], | 30.00th=[ 43], 40.00th=[ 52], 50.00th=[ 52], 60.00th=[ 54], | 70.00th=[ 83], 80.00th=[ 83], 90.00th=[ 936], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 2048, max= 8192, per=42.09%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (msec) : 50=41.67%, 100=41.67%, 1000=16.67% cpu : usr=0.00%, sys=7.34%, ctx=114, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9569: Mon Apr 29 14:00:01 2024 read: IOPS=3, BW=3568KiB/s (3654kB/s)(5120KiB/1435msec) clat (msec): min=17, max=473, avg=191.99, stdev=238.09 lat (msec): min=17, max=473, avg=191.99, stdev=238.09 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 20], 60.00th=[ 20], | 70.00th=[ 430], 80.00th=[ 430], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 2043, max= 2048, per=23.54%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4995KiB/s (5115kB/s)(7168KiB/1435msec) clat (msec): min=22, max=115, avg=64.71, stdev=33.89 lat (msec): min=22, max=115, avg=64.79, stdev=33.88 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 26], | 30.00th=[ 52], 40.00th=[ 52], 50.00th=[ 70], 60.00th=[ 81], | 70.00th=[ 81], 80.00th=[ 87], 90.00th=[ 116], 95.00th=[ 116], | 99.00th=[ 116], 99.50th=[ 116], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 8192, max= 8192, per=67.35%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=16.67%, 100=33.33%, 250=8.33%, 500=16.67% cpu : usr=0.07%, sys=8.93%, ctx=50, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8689KiB/s (8898kB/s), 878KiB/s-3568KiB/s (899kB/s-3654kB/s), io=20.0MiB (20.0MB), run=1435-2357msec WRITE: bw=11.9MiB/s (12.5MB/s), 2172KiB/s-4995KiB/s (2224kB/s-5115kB/s), io=28.0MiB (29.4MB), run=1435-2357msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9581: Mon Apr 29 14:00:06 2024 read: IOPS=0, BW=3451KiB/s (3534kB/s)(12.0MiB/3561msec) clat (msec): min=84, max=2214, avg=1185.35, stdev=1066.80 lat (msec): min=84, max=2214, avg=1185.36, stdev=1066.80 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 85], 10.00th=[ 85], 20.00th=[ 85], | 30.00th=[ 85], 40.00th=[ 1250], 50.00th=[ 1250], 60.00th=[ 1250], | 70.00th=[ 2232], 80.00th=[ 2232], 90.00th=[ 2232], 95.00th=[ 2232], | 99.00th=[ 2232], 99.50th=[ 2232], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 100=33.33% cpu : usr=0.00%, sys=50.22%, ctx=1254, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9582: Mon Apr 29 14:00:06 2024 read: IOPS=0, BW=1035KiB/s (1060kB/s)(4096KiB/3957msec) clat (nsec): min=215946k, max=215946k, avg=215946161.00, stdev= 0.00 lat (nsec): min=215948k, max=215948k, avg=215948121.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 215], 10.00th=[ 215], 20.00th=[ 215], | 30.00th=[ 215], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 215], | 70.00th=[ 215], 80.00th=[ 215], 90.00th=[ 215], 95.00th=[ 215], | 99.00th=[ 215], 99.50th=[ 215], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2070KiB/s (2120kB/s)(8192KiB/3957msec) clat (msec): min=736, max=3002, avg=1869.82, stdev=1602.31 lat (msec): min=736, max=3003, avg=1870.07, stdev=1602.44 clat percentiles (msec): | 1.00th=[ 735], 5.00th=[ 735], 10.00th=[ 735], 20.00th=[ 735], | 30.00th=[ 735], 40.00th=[ 735], 50.00th=[ 735], 60.00th=[ 3004], | 70.00th=[ 3004], 80.00th=[ 3004], 90.00th=[ 3004], 95.00th=[ 3004], | 99.00th=[ 3004], 99.50th=[ 3004], 99.90th=[ 3004], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 8192, max= 8192, per=99.61%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=8.67%, ctx=181, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9583: Mon Apr 29 14:00:06 2024 write: IOPS=0, BW=3163KiB/s (3239kB/s)(12.0MiB/3885msec) clat (msec): min=485, max=1848, avg=1293.19, stdev=715.68 lat (msec): min=485, max=1849, avg=1293.49, stdev=715.72 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 485], 10.00th=[ 485], 20.00th=[ 485], | 30.00th=[ 485], 40.00th=[ 1552], 50.00th=[ 1552], 60.00th=[ 1552], | 70.00th=[ 1854], 80.00th=[ 1854], 90.00th=[ 1854], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 8192, max= 8192, per=99.61%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33% cpu : usr=0.08%, sys=8.96%, ctx=401, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9584: Mon Apr 29 14:00:06 2024 write: IOPS=0, BW=3084KiB/s (3158kB/s)(12.0MiB/3984msec) clat (msec): min=107, max=3125, avg=1327.22, stdev=1589.80 lat (msec): min=108, max=3125, avg=1327.58, stdev=1589.83 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 108], 10.00th=[ 108], 20.00th=[ 108], | 30.00th=[ 108], 40.00th=[ 751], 50.00th=[ 751], 60.00th=[ 751], | 70.00th=[ 3138], 80.00th=[ 3138], 90.00th=[ 3138], 95.00th=[ 3138], | 99.00th=[ 3138], 99.50th=[ 3138], 99.90th=[ 3138], 99.95th=[ 3138], | 99.99th=[ 3138] bw ( KiB/s): min= 8192, max= 8192, per=99.61%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=7.53%, ctx=29, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4141KiB/s (4240kB/s), 1035KiB/s-3451KiB/s (1060kB/s-3534kB/s), io=16.0MiB (16.8MB), run=3561-3957msec WRITE: bw=8225KiB/s (8422kB/s), 2070KiB/s-3163KiB/s (2120kB/s-3239kB/s), io=32.0MiB (33.6MB), run=3885-3984msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9576: Mon Apr 29 14:00:06 2024 read: IOPS=3, BW=13.5MiB/s (14.1MB/s)(12.0MiB/890msec) clat (msec): min=43, max=787, avg=296.36, stdev=425.42 lat (msec): min=43, max=787, avg=296.36, stdev=425.42 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 44], | 30.00th=[ 44], 40.00th=[ 58], 50.00th=[ 58], 60.00th=[ 58], | 70.00th=[ 785], 80.00th=[ 785], 90.00th=[ 785], 95.00th=[ 785], | 99.00th=[ 785], 99.50th=[ 785], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=33.33%, 1000=33.33% cpu : usr=0.00%, sys=8.32%, ctx=50, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9577: Mon Apr 29 14:00:06 2024 read: IOPS=0, BW=962KiB/s (985kB/s)(4096KiB/4259msec) clat (nsec): min=284590k, max=284590k, avg=284590078.00, stdev= 0.00 lat (nsec): min=284593k, max=284593k, avg=284593268.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 284], 10.00th=[ 284], 20.00th=[ 284], | 30.00th=[ 284], 40.00th=[ 284], 50.00th=[ 284], 60.00th=[ 284], | 70.00th=[ 284], 80.00th=[ 284], 90.00th=[ 284], 95.00th=[ 284], | 99.00th=[ 284], 99.50th=[ 284], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1923KiB/s (1970kB/s)(8192KiB/4259msec) clat (msec): min=822, max=3140, avg=1981.95, stdev=1639.09 lat (msec): min=823, max=3141, avg=1982.29, stdev=1639.07 clat percentiles (msec): | 1.00th=[ 827], 5.00th=[ 827], 10.00th=[ 827], 20.00th=[ 827], | 30.00th=[ 827], 40.00th=[ 827], 50.00th=[ 827], 60.00th=[ 3138], | 70.00th=[ 3138], 80.00th=[ 3138], 90.00th=[ 3138], 95.00th=[ 3138], | 99.00th=[ 3138], 99.50th=[ 3138], 99.90th=[ 3138], 99.95th=[ 3138], | 99.99th=[ 3138] bw ( KiB/s): min= 7366, max= 7366, per=97.47%, avg=7366.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=3.48%, ctx=249, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9578: Mon Apr 29 14:00:06 2024 write: IOPS=0, BW=2834KiB/s (2902kB/s)(12.0MiB/4336msec) clat (msec): min=93, max=3575, avg=1441.34, stdev=1869.77 lat (msec): min=93, max=3576, avg=1441.66, stdev=1869.79 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 94], 10.00th=[ 94], 20.00th=[ 94], | 30.00th=[ 94], 40.00th=[ 659], 50.00th=[ 659], 60.00th=[ 659], | 70.00th=[ 3574], 80.00th=[ 3574], 90.00th=[ 3574], 95.00th=[ 3574], | 99.00th=[ 3574], 99.50th=[ 3574], 99.90th=[ 3574], 99.95th=[ 3574], | 99.99th=[ 3574] bw ( KiB/s): min= 7876, max= 7876, per=100.00%, avg=7876.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.00%, sys=3.81%, ctx=41, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9579: Mon Apr 29 14:00:06 2024 write: IOPS=3, BW=15.7MiB/s (16.4MB/s)(12.0MiB/765msec) clat (msec): min=45, max=589, avg=250.62, stdev=296.05 lat (msec): min=45, max=590, avg=250.89, stdev=295.93 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 45], | 30.00th=[ 45], 40.00th=[ 117], 50.00th=[ 117], 60.00th=[ 117], | 70.00th=[ 592], 80.00th=[ 592], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min=16351, max=16351, per=100.00%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=9.42%, ctx=31, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3847KiB/s (3939kB/s), 962KiB/s-13.5MiB/s (985kB/s-14.1MB/s), io=16.0MiB (16.8MB), run=890-4259msec WRITE: bw=7557KiB/s (7739kB/s), 1923KiB/s-15.7MiB/s (1970kB/s-16.4MB/s), io=32.0MiB (33.6MB), run=765-4336msec