== sanity test 398b: DIO and buffer IO race ============== 07:09:11 (1726312151) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.21632 s, 41.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9739: Sat Sep 14 07:09:26 2024 read: IOPS=124, BW=499KiB/s (511kB/s)(5992KiB/12011msec) clat (usec): min=53, max=37878, avg=4442.98, stdev=2797.27 lat (usec): min=54, max=37879, avg=4443.98, stdev=2797.70 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 114], 10.00th=[ 2114], 20.00th=[ 2507], | 30.00th=[ 2900], 40.00th=[ 3326], 50.00th=[ 3884], 60.00th=[ 4555], | 70.00th=[ 5276], 80.00th=[ 6063], 90.00th=[ 7570], 95.00th=[ 8979], | 99.00th=[13829], 99.50th=[15270], 99.90th=[25822], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 240, max= 712, per=24.91%, avg=498.00, stdev=78.50, samples=24 iops : min= 60, max= 178, avg=124.50, stdev=19.62, samples=24 write: IOPS=131, BW=524KiB/s (537kB/s)(6296KiB/12011msec) clat (usec): min=419, max=131866, avg=3356.10, stdev=4334.00 lat (usec): min=420, max=131867, avg=3357.86, stdev=4334.17 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 619], | 30.00th=[ 775], 40.00th=[ 1958], 50.00th=[ 2704], 60.00th=[ 3359], | 70.00th=[ 4178], 80.00th=[ 5473], 90.00th=[ 7439], 95.00th=[ 8979], | 99.00th=[ 12649], 99.50th=[ 13698], 99.90th=[ 18744], 99.95th=[131597], | 99.99th=[131597] bw ( KiB/s): min= 296, max= 752, per=24.99%, avg=523.00, stdev=94.56, samples=24 iops : min= 74, max= 188, avg=130.75, stdev=23.64, samples=24 lat (usec) : 100=2.38%, 250=0.13%, 500=0.68%, 750=14.32%, 1000=2.31% lat (msec) : 2=4.72%, 4=35.94%, 10=36.17%, 20=3.22%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=22.91%, ctx=5810, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9741: Sat Sep 14 07:09:26 2024 read: IOPS=125, BW=500KiB/s (512kB/s)(5940KiB/11876msec) clat (usec): min=47, max=22172, avg=4300.41, stdev=2377.97 lat (usec): min=47, max=22172, avg=4301.37, stdev=2377.98 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 104], 10.00th=[ 2073], 20.00th=[ 2540], | 30.00th=[ 2933], 40.00th=[ 3359], 50.00th=[ 3916], 60.00th=[ 4490], | 70.00th=[ 5211], 80.00th=[ 5997], 90.00th=[ 7111], 95.00th=[ 8455], | 99.00th=[12125], 99.50th=[13698], 99.90th=[16581], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 263, max= 624, per=24.95%, avg=498.74, stdev=71.44, samples=23 iops : min= 65, max= 156, avg=124.65, stdev=17.97, samples=23 write: IOPS=133, BW=535KiB/s (547kB/s)(6348KiB/11876msec) clat (usec): min=398, max=168953, avg=3415.28, stdev=5174.59 lat (usec): min=398, max=168954, avg=3416.14, stdev=5174.60 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 611], | 30.00th=[ 750], 40.00th=[ 1844], 50.00th=[ 2737], 60.00th=[ 3326], | 70.00th=[ 4113], 80.00th=[ 5604], 90.00th=[ 7570], 95.00th=[ 9241], | 99.00th=[ 12780], 99.50th=[ 14877], 99.90th=[ 38011], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 335, max= 648, per=25.09%, avg=525.17, stdev=78.22, samples=23 iops : min= 83, max= 162, avg=131.26, stdev=19.64, samples=23 lat (usec) : 50=0.07%, 100=2.31%, 250=0.20%, 500=0.88%, 750=14.91% lat (usec) : 1000=2.57% lat (msec) : 2=4.62%, 4=34.67%, 10=36.85%, 20=2.80%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=22.76%, ctx=5808, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9743: Sat Sep 14 07:09:26 2024 read: IOPS=126, BW=507KiB/s (519kB/s)(6028KiB/11899msec) clat (usec): min=49, max=29365, avg=4302.75, stdev=2392.25 lat (usec): min=49, max=29366, avg=4303.78, stdev=2392.58 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 594], 10.00th=[ 2180], 20.00th=[ 2540], | 30.00th=[ 2933], 40.00th=[ 3326], 50.00th=[ 3851], 60.00th=[ 4490], | 70.00th=[ 5145], 80.00th=[ 5997], 90.00th=[ 7177], 95.00th=[ 8291], | 99.00th=[12125], 99.50th=[13698], 99.90th=[19006], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 240, max= 648, per=25.03%, avg=500.43, stdev=78.74, samples=23 iops : min= 60, max= 162, avg=125.09, stdev=19.68, samples=23 write: IOPS=131, BW=526KiB/s (539kB/s)(6260KiB/11899msec) clat (usec): min=438, max=175136, avg=3414.28, stdev=5261.52 lat (usec): min=438, max=175137, avg=3415.09, stdev=5261.49 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 603], | 30.00th=[ 758], 40.00th=[ 1582], 50.00th=[ 2638], 60.00th=[ 3326], | 70.00th=[ 4293], 80.00th=[ 5604], 90.00th=[ 7570], 95.00th=[ 9110], | 99.00th=[ 12256], 99.50th=[ 14353], 99.90th=[ 22414], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 320, max= 632, per=24.71%, avg=517.13, stdev=77.79, samples=23 iops : min= 80, max= 158, avg=129.26, stdev=19.44, samples=23 lat (usec) : 50=0.07%, 100=2.12%, 250=0.23%, 500=0.65%, 750=14.65% lat (usec) : 1000=2.96% lat (msec) : 2=3.78%, 4=35.71%, 10=37.01%, 20=2.73%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=22.49%, ctx=5779, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9745: Sat Sep 14 07:09:26 2024 read: IOPS=126, BW=505KiB/s (517kB/s)(6052KiB/11983msec) clat (usec): min=48, max=23108, avg=4327.91, stdev=2455.14 lat (usec): min=48, max=23108, avg=4328.76, stdev=2455.08 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 113], 10.00th=[ 2114], 20.00th=[ 2507], | 30.00th=[ 2900], 40.00th=[ 3359], 50.00th=[ 3851], 60.00th=[ 4555], | 70.00th=[ 5211], 80.00th=[ 5997], 90.00th=[ 7308], 95.00th=[ 8455], | 99.00th=[12518], 99.50th=[14091], 99.90th=[21103], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 216, max= 672, per=24.93%, avg=498.35, stdev=86.26, samples=23 iops : min= 54, max= 168, avg=124.57, stdev=21.54, samples=23 write: IOPS=130, BW=520KiB/s (533kB/s)(6236KiB/11983msec) clat (usec): min=404, max=165394, avg=3434.35, stdev=5119.63 lat (usec): min=404, max=165396, avg=3435.27, stdev=5119.56 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 619], | 30.00th=[ 750], 40.00th=[ 1582], 50.00th=[ 2638], 60.00th=[ 3326], | 70.00th=[ 4293], 80.00th=[ 5735], 90.00th=[ 7635], 95.00th=[ 8979], | 99.00th=[ 13042], 99.50th=[ 14877], 99.90th=[ 34866], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 248, max= 872, per=24.64%, avg=515.78, stdev=105.30, samples=23 iops : min= 62, max= 218, avg=128.91, stdev=26.34, samples=23 lat (usec) : 50=0.10%, 100=2.18%, 250=0.46%, 500=0.78%, 750=14.26% lat (usec) : 1000=2.60% lat (msec) : 2=4.59%, 4=35.19%, 10=37.04%, 20=2.67%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=22.30%, ctx=5826, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1999KiB/s (2047kB/s), 499KiB/s-507KiB/s (511kB/s-519kB/s), io=23.4MiB (24.6MB), run=11876-12011msec WRITE: bw=2093KiB/s (2143kB/s), 520KiB/s-535KiB/s (533kB/s-547kB/s), io=24.6MiB (25.7MB), run=11876-12011msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9740: Sat Sep 14 07:09:30 2024 read: IOPS=95, BW=382KiB/s (391kB/s)(5992KiB/15705msec) clat (usec): min=1489, max=203863, avg=4442.70, stdev=6094.99 lat (usec): min=1489, max=203864, avg=4443.33, stdev=6094.99 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1860], 10.00th=[ 1958], 20.00th=[ 2147], | 30.00th=[ 2343], 40.00th=[ 2638], 50.00th=[ 2999], 60.00th=[ 3556], | 70.00th=[ 4359], 80.00th=[ 5932], 90.00th=[ 9110], 95.00th=[ 11469], | 99.00th=[ 16909], 99.50th=[ 20055], 99.90th=[ 26346], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 136, max= 798, per=25.25%, avg=380.77, stdev=158.03, samples=31 iops : min= 34, max= 199, avg=95.16, stdev=39.45, samples=31 write: IOPS=100, BW=401KiB/s (411kB/s)(6296KiB/15705msec) clat (usec): min=2083, max=26222, avg=5704.31, stdev=3200.56 lat (usec): min=2084, max=26223, avg=5705.06, stdev=3200.55 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2835], 10.00th=[ 2999], 20.00th=[ 3228], | 30.00th=[ 3523], 40.00th=[ 3916], 50.00th=[ 4490], 60.00th=[ 5342], | 70.00th=[ 6521], 80.00th=[ 7898], 90.00th=[10028], 95.00th=[12125], | 99.00th=[16712], 99.50th=[19268], 99.90th=[23725], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 144, max= 760, per=25.24%, avg=398.58, stdev=158.15, samples=31 iops : min= 36, max= 190, avg=99.61, stdev=39.49, samples=31 lat (msec) : 2=6.22%, 4=47.07%, 10=37.79%, 20=8.46%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=10.98%, ctx=3693, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9742: Sat Sep 14 07:09:30 2024 read: IOPS=94, BW=376KiB/s (385kB/s)(5940KiB/15789msec) clat (usec): min=1625, max=172017, avg=4299.39, stdev=5320.74 lat (usec): min=1626, max=172018, avg=4300.12, stdev=5320.70 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1876], 10.00th=[ 1975], 20.00th=[ 2147], | 30.00th=[ 2311], 40.00th=[ 2606], 50.00th=[ 2933], 60.00th=[ 3458], | 70.00th=[ 4228], 80.00th=[ 5538], 90.00th=[ 8717], 95.00th=[ 11076], | 99.00th=[ 15795], 99.50th=[ 16712], 99.90th=[ 23462], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 128, max= 784, per=24.92%, avg=375.74, stdev=168.64, samples=31 iops : min= 32, max= 196, avg=93.94, stdev=42.16, samples=31 write: IOPS=100, BW=402KiB/s (412kB/s)(6348KiB/15789msec) clat (usec): min=2243, max=42206, avg=5883.01, stdev=3549.45 lat (usec): min=2244, max=42208, avg=5883.79, stdev=3549.42 clat percentiles (usec): | 1.00th=[ 2507], 5.00th=[ 2835], 10.00th=[ 3032], 20.00th=[ 3294], | 30.00th=[ 3621], 40.00th=[ 3982], 50.00th=[ 4490], 60.00th=[ 5473], | 70.00th=[ 6587], 80.00th=[ 8160], 90.00th=[10552], 95.00th=[12649], | 99.00th=[17433], 99.50th=[21627], 99.90th=[28443], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 200, max= 768, per=25.20%, avg=397.94, stdev=154.99, samples=31 iops : min= 50, max= 192, avg=99.48, stdev=38.75, samples=31 lat (msec) : 2=5.66%, 4=47.79%, 10=36.75%, 20=9.34%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=11.05%, ctx=3682, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9744: Sat Sep 14 07:09:30 2024 read: IOPS=94, BW=379KiB/s (388kB/s)(6028KiB/15913msec) clat (usec): min=1521, max=26522, avg=4407.67, stdev=3297.73 lat (usec): min=1522, max=26522, avg=4408.32, stdev=3297.74 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1844], 10.00th=[ 1975], 20.00th=[ 2147], | 30.00th=[ 2376], 40.00th=[ 2638], 50.00th=[ 3064], 60.00th=[ 3687], | 70.00th=[ 4490], 80.00th=[ 6128], 90.00th=[ 9110], 95.00th=[11731], | 99.00th=[16188], 99.50th=[18744], 99.90th=[22152], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 160, max= 832, per=25.01%, avg=377.23, stdev=157.90, samples=31 iops : min= 40, max= 208, avg=94.29, stdev=39.46, samples=31 write: IOPS=98, BW=393KiB/s (403kB/s)(6260KiB/15913msec) clat (msec): min=2, max=175, avg= 5.88, stdev= 5.43 lat (msec): min=2, max=175, avg= 5.88, stdev= 5.43 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 18], 99.50th=[ 22], 99.90th=[ 29], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 200, max= 840, per=24.71%, avg=390.13, stdev=165.72, samples=31 iops : min= 50, max= 210, avg=97.52, stdev=41.42, samples=31 lat (msec) : 2=5.60%, 4=46.52%, 10=38.12%, 20=9.28%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=10.90%, ctx=3710, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9746: Sat Sep 14 07:09:30 2024 read: IOPS=95, BW=383KiB/s (392kB/s)(6052KiB/15811msec) clat (usec): min=1601, max=26110, avg=4171.95, stdev=2976.63 lat (usec): min=1602, max=26111, avg=4172.58, stdev=2976.64 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1844], 10.00th=[ 1958], 20.00th=[ 2147], | 30.00th=[ 2343], 40.00th=[ 2638], 50.00th=[ 3032], 60.00th=[ 3556], | 70.00th=[ 4293], 80.00th=[ 5538], 90.00th=[ 8455], 95.00th=[10683], | 99.00th=[14484], 99.50th=[17695], 99.90th=[22152], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 144, max= 760, per=25.27%, avg=381.10, stdev=159.91, samples=31 iops : min= 36, max= 190, avg=95.23, stdev=40.00, samples=31 write: IOPS=98, BW=394KiB/s (404kB/s)(6236KiB/15811msec) clat (msec): min=2, max=175, avg= 6.05, stdev= 5.52 lat (msec): min=2, max=175, avg= 6.05, stdev= 5.52 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 18], 99.50th=[ 22], 99.90th=[ 35], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 160, max= 760, per=24.82%, avg=391.94, stdev=160.07, samples=31 iops : min= 40, max= 190, avg=97.94, stdev=40.05, samples=31 lat (msec) : 2=6.18%, 4=46.84%, 10=37.47%, 20=9.05%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=10.99%, ctx=3669, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1509KiB/s (1545kB/s), 376KiB/s-383KiB/s (385kB/s-392kB/s), io=23.4MiB (24.6MB), run=15705-15913msec WRITE: bw=1580KiB/s (1618kB/s), 393KiB/s-402KiB/s (403kB/s-412kB/s), io=24.6MiB (25.7MB), run=15705-15913msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9772: Sat Sep 14 07:09:36 2024 read: IOPS=71, BW=1140KiB/s (1168kB/s)(5920KiB/5191msec) clat (usec): min=1785, max=21445, avg=5565.03, stdev=3837.57 lat (usec): min=1786, max=21445, avg=5565.68, stdev=3837.53 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2114], 10.00th=[ 2311], 20.00th=[ 2507], | 30.00th=[ 2835], 40.00th=[ 3261], 50.00th=[ 3818], 60.00th=[ 5080], | 70.00th=[ 6718], 80.00th=[ 8717], 90.00th=[11207], 95.00th=[13698], | 99.00th=[17695], 99.50th=[18220], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 768, max= 1536, per=25.11%, avg=1116.80, stdev=255.76, samples=10 iops : min= 48, max= 96, avg=69.80, stdev=15.98, samples=10 write: IOPS=76, BW=1227KiB/s (1256kB/s)(6368KiB/5191msec) clat (usec): min=2592, max=25508, avg=7754.92, stdev=4876.54 lat (usec): min=2593, max=25509, avg=7756.13, stdev=4876.59 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3195], 10.00th=[ 3490], 20.00th=[ 3818], | 30.00th=[ 4293], 40.00th=[ 5145], 50.00th=[ 6194], 60.00th=[ 7242], | 70.00th=[ 8717], 80.00th=[11076], 90.00th=[14746], 95.00th=[19006], | 99.00th=[23987], 99.50th=[24773], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 896, max= 1504, per=26.06%, avg=1219.20, stdev=178.14, samples=10 iops : min= 56, max= 94, avg=76.20, stdev=11.13, samples=10 lat (msec) : 2=1.43%, 4=35.29%, 10=43.75%, 20=17.32%, 50=2.21% cpu : usr=0.15%, sys=9.54%, ctx=935, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9773: Sat Sep 14 07:09:36 2024 read: IOPS=66, BW=1066KiB/s (1092kB/s)(5744KiB/5386msec) clat (usec): min=1757, max=32180, avg=5782.42, stdev=4288.55 lat (usec): min=1758, max=32181, avg=5783.06, stdev=4288.53 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2180], 10.00th=[ 2278], 20.00th=[ 2573], | 30.00th=[ 2966], 40.00th=[ 3359], 50.00th=[ 3949], 60.00th=[ 5080], | 70.00th=[ 6390], 80.00th=[ 8979], 90.00th=[11338], 95.00th=[14222], | 99.00th=[19792], 99.50th=[21890], 99.90th=[32113], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 736, max= 1440, per=23.80%, avg=1058.60, stdev=203.25, samples=10 iops : min= 46, max= 90, avg=66.00, stdev=12.57, samples=10 write: IOPS=75, BW=1215KiB/s (1244kB/s)(6544KiB/5386msec) clat (usec): min=2841, max=26196, avg=8024.92, stdev=5062.36 lat (usec): min=2841, max=26196, avg=8026.10, stdev=5062.40 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3163], 10.00th=[ 3490], 20.00th=[ 3884], | 30.00th=[ 4424], 40.00th=[ 5014], 50.00th=[ 6194], 60.00th=[ 7767], | 70.00th=[ 9503], 80.00th=[11863], 90.00th=[15008], 95.00th=[19268], | 99.00th=[24249], 99.50th=[25297], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 928, max= 1405, per=25.23%, avg=1180.30, stdev=149.45, samples=10 iops : min= 58, max= 87, avg=73.60, stdev= 9.34, samples=10 lat (msec) : 2=0.52%, 4=35.03%, 10=41.02%, 20=20.44%, 50=2.99% cpu : usr=0.04%, sys=9.40%, ctx=914, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9774: Sat Sep 14 07:09:36 2024 read: IOPS=77, BW=1240KiB/s (1269kB/s)(6256KiB/5047msec) clat (usec): min=1847, max=24603, avg=5562.60, stdev=3765.67 lat (usec): min=1847, max=24603, avg=5563.21, stdev=3765.69 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2114], 10.00th=[ 2278], 20.00th=[ 2606], | 30.00th=[ 2900], 40.00th=[ 3392], 50.00th=[ 4178], 60.00th=[ 4948], | 70.00th=[ 6521], 80.00th=[ 8717], 90.00th=[11207], 95.00th=[12780], | 99.00th=[17171], 99.50th=[22938], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 800, max= 1504, per=28.05%, avg=1247.20, stdev=259.26, samples=10 iops : min= 50, max= 94, avg=77.70, stdev=16.30, samples=10 write: IOPS=74, BW=1195KiB/s (1224kB/s)(6032KiB/5047msec) clat (usec): min=2458, max=25582, avg=7582.55, stdev=4795.55 lat (usec): min=2459, max=25583, avg=7583.90, stdev=4795.82 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3130], 10.00th=[ 3392], 20.00th=[ 3720], | 30.00th=[ 4178], 40.00th=[ 4752], 50.00th=[ 5669], 60.00th=[ 7242], | 70.00th=[ 8717], 80.00th=[11338], 90.00th=[14222], 95.00th=[18744], | 99.00th=[21890], 99.50th=[22414], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 1024, max= 1376, per=25.50%, avg=1192.70, stdev=104.77, samples=10 iops : min= 64, max= 86, avg=74.30, stdev= 6.62, samples=10 lat (msec) : 2=0.65%, 4=37.11%, 10=42.71%, 20=17.58%, 50=1.95% cpu : usr=0.12%, sys=9.93%, ctx=942, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9775: Sat Sep 14 07:09:36 2024 read: IOPS=71, BW=1138KiB/s (1165kB/s)(6032KiB/5300msec) clat (usec): min=1876, max=32520, avg=5669.10, stdev=4057.16 lat (usec): min=1877, max=32520, avg=5669.72, stdev=4057.19 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2147], 10.00th=[ 2245], 20.00th=[ 2606], | 30.00th=[ 2966], 40.00th=[ 3261], 50.00th=[ 4015], 60.00th=[ 5211], | 70.00th=[ 6718], 80.00th=[ 8717], 90.00th=[11076], 95.00th=[13566], | 99.00th=[21103], 99.50th=[21890], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 704, max= 1312, per=24.95%, avg=1109.50, stdev=196.45, samples=10 iops : min= 44, max= 82, avg=69.10, stdev=12.22, samples=10 write: IOPS=73, BW=1180KiB/s (1209kB/s)(6256KiB/5300msec) clat (usec): min=2729, max=27961, avg=7996.21, stdev=5142.30 lat (usec): min=2730, max=27963, avg=7997.51, stdev=5142.26 clat percentiles (usec): | 1.00th=[ 2802], 5.00th=[ 3163], 10.00th=[ 3359], 20.00th=[ 3752], | 30.00th=[ 4293], 40.00th=[ 5211], 50.00th=[ 6325], 60.00th=[ 7504], | 70.00th=[ 9372], 80.00th=[11731], 90.00th=[15401], 95.00th=[19268], | 99.00th=[23987], 99.50th=[25035], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 1056, max= 1344, per=25.09%, avg=1173.50, stdev=98.17, samples=10 iops : min= 66, max= 84, avg=73.10, stdev= 6.24, samples=10 lat (msec) : 2=0.78%, 4=36.85%, 10=41.28%, 20=18.49%, 50=2.60% cpu : usr=0.11%, sys=9.45%, ctx=933, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4447KiB/s (4554kB/s), 1066KiB/s-1240KiB/s (1092kB/s-1269kB/s), io=23.4MiB (24.5MB), run=5047-5386msec WRITE: bw=4679KiB/s (4791kB/s), 1180KiB/s-1227KiB/s (1209kB/s-1256kB/s), io=24.6MiB (25.8MB), run=5047-5386msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9777: Sat Sep 14 07:09:36 2024 read: IOPS=67, BW=1076KiB/s (1102kB/s)(5920KiB/5502msec) clat (usec): min=50, max=33692, avg=10781.58, stdev=4529.70 lat (usec): min=50, max=33693, avg=10782.20, stdev=4529.70 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 70], 10.00th=[ 6980], 20.00th=[ 8455], | 30.00th=[ 9241], 40.00th=[10290], 50.00th=[10945], 60.00th=[11469], | 70.00th=[12649], 80.00th=[13698], 90.00th=[15401], 95.00th=[17433], | 99.00th=[22676], 99.50th=[25297], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 734, max= 1184, per=23.01%, avg=988.60, stdev=132.32, samples=10 iops : min= 45, max= 74, avg=61.70, stdev= 8.46, samples=10 write: IOPS=72, BW=1157KiB/s (1185kB/s)(6368KiB/5502msec) clat (usec): min=636, max=22044, avg=3670.06, stdev=3499.66 lat (usec): min=637, max=22045, avg=3671.05, stdev=3499.69 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 807], 10.00th=[ 840], 20.00th=[ 955], | 30.00th=[ 1565], 40.00th=[ 2311], 50.00th=[ 2737], 60.00th=[ 3097], | 70.00th=[ 3818], 80.00th=[ 5407], 90.00th=[ 7767], 95.00th=[11469], | 99.00th=[17695], 99.50th=[21103], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 768, max= 1408, per=24.14%, avg=1091.00, stdev=242.78, samples=10 iops : min= 48, max= 88, avg=68.10, stdev=15.29, samples=10 lat (usec) : 100=3.52%, 250=0.26%, 750=0.78%, 1000=10.42% lat (msec) : 2=5.47%, 4=20.57%, 10=25.39%, 20=32.03%, 50=1.56% cpu : usr=0.18%, sys=19.03%, ctx=2468, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9778: Sat Sep 14 07:09:36 2024 read: IOPS=65, BW=1041KiB/s (1066kB/s)(5744KiB/5520msec) clat (usec): min=55, max=26212, avg=10891.91, stdev=3683.44 lat (usec): min=55, max=26214, avg=10892.54, stdev=3683.53 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 5014], 10.00th=[ 7242], 20.00th=[ 8848], | 30.00th=[ 9634], 40.00th=[10421], 50.00th=[11076], 60.00th=[11600], | 70.00th=[12256], 80.00th=[13173], 90.00th=[15270], 95.00th=[16581], | 99.00th=[21103], 99.50th=[22152], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 734, max= 1469, per=24.03%, avg=1032.27, stdev=192.74, samples=11 iops : min= 45, max= 91, avg=64.36, stdev=12.00, samples=11 write: IOPS=74, BW=1186KiB/s (1214kB/s)(6544KiB/5520msec) clat (usec): min=630, max=22845, avg=3890.54, stdev=3657.87 lat (usec): min=631, max=22846, avg=3891.50, stdev=3657.89 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 799], 10.00th=[ 840], 20.00th=[ 1074], | 30.00th=[ 2147], 40.00th=[ 2474], 50.00th=[ 2769], 60.00th=[ 3326], | 70.00th=[ 4113], 80.00th=[ 5145], 90.00th=[ 7504], 95.00th=[12387], | 99.00th=[18482], 99.50th=[21103], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 926, max= 1852, per=26.18%, avg=1183.45, stdev=295.68, samples=11 iops : min= 57, max= 115, avg=73.82, stdev=18.39, samples=11 lat (usec) : 100=1.95%, 250=0.26%, 750=1.04%, 1000=8.20% lat (msec) : 2=5.21%, 4=22.14%, 10=27.08%, 20=33.07%, 50=1.04% cpu : usr=0.11%, sys=19.37%, ctx=2554, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9779: Sat Sep 14 07:09:36 2024 read: IOPS=70, BW=1128KiB/s (1156kB/s)(6256KiB/5544msec) clat (usec): min=50, max=27308, avg=10425.44, stdev=4174.01 lat (usec): min=51, max=27309, avg=10426.02, stdev=4174.03 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 77], 10.00th=[ 6652], 20.00th=[ 8356], | 30.00th=[ 9241], 40.00th=[10028], 50.00th=[10683], 60.00th=[11338], | 70.00th=[12125], 80.00th=[13173], 90.00th=[14615], 95.00th=[16319], | 99.00th=[20841], 99.50th=[24511], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 736, max= 2016, per=26.14%, avg=1122.91, stdev=316.61, samples=11 iops : min= 46, max= 126, avg=70.18, stdev=19.79, samples=11 write: IOPS=68, BW=1088KiB/s (1114kB/s)(6032KiB/5544msec) clat (usec): min=587, max=24609, avg=3706.17, stdev=3611.51 lat (usec): min=588, max=24610, avg=3707.88, stdev=3611.45 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 783], 10.00th=[ 832], 20.00th=[ 930], | 30.00th=[ 1270], 40.00th=[ 2376], 50.00th=[ 2737], 60.00th=[ 3294], | 70.00th=[ 3982], 80.00th=[ 5080], 90.00th=[ 8094], 95.00th=[11994], | 99.00th=[18744], 99.50th=[20055], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 768, max= 1536, per=23.62%, avg=1067.64, stdev=213.36, samples=11 iops : min= 48, max= 96, avg=66.73, stdev=13.33, samples=11 lat (usec) : 100=3.52%, 250=0.26%, 500=0.13%, 750=1.82%, 1000=10.29% lat (msec) : 2=5.21%, 4=17.45%, 10=27.34%, 20=33.20%, 50=0.78% cpu : usr=0.18%, sys=19.30%, ctx=2503, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9780: Sat Sep 14 07:09:36 2024 read: IOPS=67, BW=1082KiB/s (1108kB/s)(6032KiB/5575msec) clat (usec): min=57, max=28344, avg=10415.30, stdev=4060.61 lat (usec): min=57, max=28344, avg=10415.89, stdev=4060.62 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 82], 10.00th=[ 6718], 20.00th=[ 8455], | 30.00th=[ 9503], 40.00th=[10159], 50.00th=[10683], 60.00th=[11338], | 70.00th=[12125], 80.00th=[12911], 90.00th=[14353], 95.00th=[16188], | 99.00th=[20317], 99.50th=[23987], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 704, max= 1724, per=24.43%, avg=1049.64, stdev=249.64, samples=11 iops : min= 44, max= 107, avg=65.45, stdev=15.42, samples=11 write: IOPS=70, BW=1122KiB/s (1149kB/s)(6256KiB/5575msec) clat (usec): min=608, max=41093, avg=4060.03, stdev=4637.23 lat (usec): min=608, max=41095, avg=4061.05, stdev=4637.23 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 766], 10.00th=[ 816], 20.00th=[ 898], | 30.00th=[ 1172], 40.00th=[ 2180], 50.00th=[ 2638], 60.00th=[ 3163], | 70.00th=[ 3949], 80.00th=[ 5407], 90.00th=[ 9765], 95.00th=[13829], | 99.00th=[20317], 99.50th=[23987], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 768, max= 1628, per=24.38%, avg=1101.91, stdev=242.13, samples=11 iops : min= 48, max= 101, avg=68.73, stdev=14.97, samples=11 lat (usec) : 100=2.86%, 250=0.65%, 500=0.13%, 750=1.95%, 1000=11.33% lat (msec) : 2=5.60%, 4=16.80%, 10=25.00%, 20=34.37%, 50=1.30% cpu : usr=0.18%, sys=18.78%, ctx=2455, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4296KiB/s (4399kB/s), 1041KiB/s-1128KiB/s (1066kB/s-1156kB/s), io=23.4MiB (24.5MB), run=5502-5575msec WRITE: bw=4520KiB/s (4629kB/s), 1088KiB/s-1186KiB/s (1114kB/s-1214kB/s), io=24.6MiB (25.8MB), run=5502-5575msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9791: Sat Sep 14 07:09:42 2024 read: IOPS=1, BW=1499KiB/s (1535kB/s)(7168KiB/4781msec) clat (usec): min=497, max=927870, avg=626357.00, stdev=383679.75 lat (usec): min=497, max=927872, avg=626358.19, stdev=383679.67 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 498], 10.00th=[ 498], 20.00th=[149947], | 30.00th=[767558], 40.00th=[767558], 50.00th=[784335], 60.00th=[851444], | 70.00th=[851444], 80.00th=[910164], 90.00th=[926942], 95.00th=[926942], | 99.00th=[926942], 99.50th=[926942], 99.90th=[926942], 99.95th=[926942], | 99.99th=[926942] bw ( KiB/s): min= 1910, max= 4096, per=54.84%, avg=2348.80, stdev=976.72, samples=5 iops : min= 1, max= 4, avg= 1.60, stdev= 1.34, samples=5 write: IOPS=1, BW=1071KiB/s (1097kB/s)(5120KiB/4781msec) clat (msec): min=32, max=171, avg=72.37, stdev=56.98 lat (msec): min=32, max=171, avg=72.53, stdev=57.12 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33], | 30.00th=[ 45], 40.00th=[ 45], 50.00th=[ 46], 60.00th=[ 46], | 70.00th=[ 69], 80.00th=[ 69], 90.00th=[ 171], 95.00th=[ 171], | 99.00th=[ 171], 99.50th=[ 171], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 1910, max= 3828, per=39.88%, avg=2391.50, stdev=957.67, samples=4 iops : min= 1, max= 3, avg= 1.50, stdev= 1.00, samples=4 lat (usec) : 500=8.33% lat (msec) : 50=25.00%, 100=8.33%, 250=16.67%, 1000=41.67% cpu : usr=0.00%, sys=18.95%, ctx=1616, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9793: Sat Sep 14 07:09:42 2024 read: IOPS=0, BW=650KiB/s (666kB/s)(2048KiB/3150msec) clat (msec): min=85, max=898, avg=492.05, stdev=575.22 lat (msec): min=85, max=898, avg=492.05, stdev=575.22 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 86], 10.00th=[ 86], 20.00th=[ 86], | 30.00th=[ 86], 40.00th=[ 86], 50.00th=[ 86], 60.00th=[ 902], | 70.00th=[ 902], 80.00th=[ 902], 90.00th=[ 902], 95.00th=[ 902], | 99.00th=[ 902], 99.50th=[ 902], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 1906, max= 2048, per=46.16%, avg=1977.00, stdev=100.41, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3251KiB/s (3329kB/s)(10.0MiB/3150msec) clat (msec): min=23, max=886, avg=204.08, stdev=323.71 lat (msec): min=23, max=886, avg=204.15, stdev=323.69 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 26], | 30.00th=[ 27], 40.00th=[ 30], 50.00th=[ 47], 60.00th=[ 50], | 70.00th=[ 174], 80.00th=[ 174], 90.00th=[ 735], 95.00th=[ 885], | 99.00th=[ 885], 99.50th=[ 885], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 3813, max= 6144, per=76.54%, avg=4590.00, stdev=1345.80, samples=3 iops : min= 3, max= 6, avg= 4.00, stdev= 1.73, samples=3 lat (msec) : 50=58.33%, 100=8.33%, 250=8.33%, 750=8.33%, 1000=16.67% cpu : usr=0.00%, sys=14.26%, ctx=389, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9795: Sat Sep 14 07:09:42 2024 read: IOPS=5, BW=5948KiB/s (6090kB/s)(6144KiB/1033msec) clat (usec): min=424, max=648901, avg=119728.22, stdev=260597.65 lat (usec): min=426, max=648903, avg=119729.28, stdev=260597.69 clat percentiles (usec): | 1.00th=[ 424], 5.00th=[ 424], 10.00th=[ 424], 20.00th=[ 461], | 30.00th=[ 461], 40.00th=[ 570], 50.00th=[ 570], 60.00th=[ 1004], | 70.00th=[ 66847], 80.00th=[ 66847], 90.00th=[650118], 95.00th=[650118], | 99.00th=[650118], 99.50th=[650118], 99.90th=[650118], 99.95th=[650118], | 99.99th=[650118] bw ( KiB/s): min= 4096, max= 5785, per=100.00%, avg=4940.50, stdev=1194.30, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=5, BW=5948KiB/s (6090kB/s)(6144KiB/1033msec) clat (usec): min=30157, max=67268, avg=47227.31, stdev=14638.48 lat (usec): min=30182, max=67344, avg=47282.83, stdev=14672.04 clat percentiles (usec): | 1.00th=[30278], 5.00th=[30278], 10.00th=[30278], 20.00th=[35390], | 30.00th=[35390], 40.00th=[38536], 50.00th=[38536], 60.00th=[53216], | 70.00th=[58983], 80.00th=[58983], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 3856, max= 8192, per=100.00%, avg=6024.00, stdev=3066.02, samples=2 iops : min= 3, max= 8, avg= 5.50, stdev= 3.54, samples=2 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 2=8.33%, 50=25.00%, 100=33.33%, 750=8.33% cpu : usr=0.00%, sys=27.91%, ctx=312, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9796: Sat Sep 14 07:09:42 2024 read: IOPS=5, BW=5262KiB/s (5388kB/s)(5120KiB/973msec) clat (usec): min=556, max=542979, avg=119211.57, stdev=237528.07 lat (usec): min=557, max=542979, avg=119213.07, stdev=237527.64 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 553], 10.00th=[ 553], 20.00th=[ 553], | 30.00th=[ 725], 40.00th=[ 725], 50.00th=[ 9110], 60.00th=[ 9110], | 70.00th=[ 42730], 80.00th=[ 42730], 90.00th=[541066], 95.00th=[541066], | 99.00th=[541066], 99.50th=[541066], 99.90th=[541066], 99.95th=[541066], | 99.99th=[541066] bw ( KiB/s): min= 2048, max= 2048, per=47.82%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7367KiB/s (7544kB/s)(7168KiB/973msec) clat (usec): min=31629, max=75626, avg=50555.39, stdev=17919.93 lat (usec): min=31663, max=75700, avg=50603.07, stdev=17933.27 clat percentiles (usec): | 1.00th=[31589], 5.00th=[31589], 10.00th=[31589], 20.00th=[35914], | 30.00th=[41157], 40.00th=[41157], 50.00th=[44827], 60.00th=[49546], | 70.00th=[49546], 80.00th=[74974], 90.00th=[76022], 95.00th=[76022], | 99.00th=[76022], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 10=8.33%, 50=50.00%, 100=16.67%, 750=8.33% cpu : usr=0.10%, sys=32.00%, ctx=295, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4284KiB/s (4386kB/s), 650KiB/s-5948KiB/s (666kB/s-6090kB/s), io=20.0MiB (20.0MB), run=973-4781msec WRITE: bw=5997KiB/s (6141kB/s), 1071KiB/s-7367KiB/s (1097kB/s-7544kB/s), io=28.0MiB (29.4MB), run=973-4781msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9790: Sat Sep 14 07:09:42 2024 read: IOPS=1, BW=1498KiB/s (1534kB/s)(7168KiB/4786msec) clat (msec): min=14, max=924, avg=469.93, stdev=416.94 lat (msec): min=14, max=924, avg=469.93, stdev=416.94 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 25], | 30.00th=[ 74], 40.00th=[ 74], 50.00th=[ 634], 60.00th=[ 718], | 70.00th=[ 718], 80.00th=[ 902], 90.00th=[ 927], 95.00th=[ 927], | 99.00th=[ 927], 99.50th=[ 927], 99.90th=[ 927], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 1976, max= 4096, per=56.46%, avg=2416.00, stdev=939.63, samples=5 iops : min= 1, max= 4, avg= 1.80, stdev= 1.30, samples=5 write: IOPS=1, BW=1070KiB/s (1095kB/s)(5120KiB/4786msec) clat (msec): min=63, max=922, avg=265.12, stdev=370.35 lat (msec): min=63, max=922, avg=265.21, stdev=370.32 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 64], | 30.00th=[ 67], 40.00th=[ 67], 50.00th=[ 94], 60.00th=[ 94], | 70.00th=[ 180], 80.00th=[ 180], 90.00th=[ 919], 95.00th=[ 919], | 99.00th=[ 919], 99.50th=[ 919], 99.90th=[ 919], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 1980, max= 3961, per=41.61%, avg=2492.25, stdev=979.69, samples=4 iops : min= 1, max= 3, avg= 1.75, stdev= 0.96, samples=4 lat (msec) : 20=8.33%, 50=8.33%, 100=33.33%, 250=8.33%, 750=16.67% lat (msec) : 1000=25.00% cpu : usr=0.02%, sys=3.87%, ctx=79, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9792: Sat Sep 14 07:09:42 2024 read: IOPS=0, BW=513KiB/s (525kB/s)(2048KiB/3993msec) clat (msec): min=32, max=875, avg=454.42, stdev=596.04 lat (msec): min=32, max=875, avg=454.42, stdev=596.04 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 33], 60.00th=[ 877], | 70.00th=[ 877], 80.00th=[ 877], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 2048, max= 2048, per=47.86%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2564KiB/s (2626kB/s)(10.0MiB/3993msec) clat (msec): min=22, max=869, avg=291.39, stdev=361.04 lat (msec): min=22, max=869, avg=291.45, stdev=361.03 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 28], | 30.00th=[ 39], 40.00th=[ 44], 50.00th=[ 52], 60.00th=[ 88], | 70.00th=[ 709], 80.00th=[ 709], 90.00th=[ 835], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 3961, max= 6144, per=75.83%, avg=4542.25, stdev=1069.63, samples=4 iops : min= 3, max= 6, avg= 4.00, stdev= 1.41, samples=4 lat (msec) : 50=41.67%, 100=16.67%, 250=8.33%, 750=8.33%, 1000=25.00% cpu : usr=0.05%, sys=4.43%, ctx=105, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9794: Sat Sep 14 07:09:42 2024 read: IOPS=1, BW=2029KiB/s (2078kB/s)(6144KiB/3028msec) clat (msec): min=12, max=903, avg=290.41, stdev=409.72 lat (msec): min=12, max=903, avg=290.41, stdev=409.72 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 34], 50.00th=[ 34], 60.00th=[ 48], | 70.00th=[ 726], 80.00th=[ 726], 90.00th=[ 902], 95.00th=[ 902], | 99.00th=[ 902], 99.50th=[ 902], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 1954, max= 4096, per=79.04%, avg=3382.00, stdev=1236.68, samples=3 iops : min= 1, max= 4, avg= 3.00, stdev= 1.73, samples=3 write: IOPS=1, BW=2029KiB/s (2078kB/s)(6144KiB/3028msec) clat (msec): min=14, max=978, avg=208.34, stdev=378.48 lat (msec): min=14, max=978, avg=208.38, stdev=378.48 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 43], | 30.00th=[ 43], 40.00th=[ 48], 50.00th=[ 48], 60.00th=[ 67], | 70.00th=[ 101], 80.00th=[ 101], 90.00th=[ 978], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 1954, max= 8192, per=67.86%, avg=4064.67, stdev=3574.68, samples=3 iops : min= 1, max= 8, avg= 3.67, stdev= 3.79, samples=3 lat (msec) : 20=16.67%, 50=41.67%, 100=8.33%, 250=8.33%, 750=8.33% lat (msec) : 1000=16.67% cpu : usr=0.00%, sys=2.87%, ctx=204, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9797: Sat Sep 14 07:09:42 2024 read: IOPS=2, BW=2525KiB/s (2585kB/s)(5120KiB/2028msec) clat (msec): min=15, max=600, avg=136.42, stdev=259.70 lat (msec): min=15, max=600, avg=136.43, stdev=259.70 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16], | 30.00th=[ 16], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 17], | 70.00th=[ 34], 80.00th=[ 34], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 2048, max= 6144, per=95.72%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=3, BW=3535KiB/s (3619kB/s)(7168KiB/2028msec) clat (msec): min=15, max=912, avg=174.14, stdev=327.24 lat (msec): min=15, max=912, avg=174.20, stdev=327.23 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 69], 60.00th=[ 74], | 70.00th=[ 74], 80.00th=[ 97], 90.00th=[ 911], 95.00th=[ 911], | 99.00th=[ 911], 99.50th=[ 911], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 2048, max= 8192, per=79.73%, avg=4775.67, stdev=3129.36, samples=3 iops : min= 2, max= 8, avg= 4.33, stdev= 3.21, samples=3 lat (msec) : 20=41.67%, 50=16.67%, 100=25.00%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=6.71%, ctx=73, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4279KiB/s (4382kB/s), 513KiB/s-2525KiB/s (525kB/s-2585kB/s), io=20.0MiB (20.0MB), run=2028-4786msec WRITE: bw=5991KiB/s (6135kB/s), 1070KiB/s-3535KiB/s (1095kB/s-3619kB/s), io=28.0MiB (29.4MB), run=2028-4786msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9806: Sat Sep 14 07:09:45 2024 read: IOPS=2, BW=11.7MiB/s (12.3MB/s)(12.0MiB/1022msec) clat (msec): min=55, max=737, avg=333.73, stdev=358.00 lat (msec): min=55, max=737, avg=333.73, stdev=358.00 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 56], 10.00th=[ 56], 20.00th=[ 56], | 30.00th=[ 56], 40.00th=[ 209], 50.00th=[ 209], 60.00th=[ 209], | 70.00th=[ 735], 80.00th=[ 735], 90.00th=[ 735], 95.00th=[ 735], | 99.00th=[ 735], 99.50th=[ 735], 99.90th=[ 735], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 8192, max= 8192, per=79.95%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 100=33.33%, 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=67.29%, ctx=135, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9807: Sat Sep 14 07:09:45 2024 read: IOPS=0, BW=2562KiB/s (2623kB/s)(4096KiB/1599msec) clat (nsec): min=82914k, max=82914k, avg=82914363.00, stdev= 0.00 lat (nsec): min=82917k, max=82917k, avg=82917261.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[83362], 5.00th=[83362], 10.00th=[83362], 20.00th=[83362], | 30.00th=[83362], 40.00th=[83362], 50.00th=[83362], 60.00th=[83362], | 70.00th=[83362], 80.00th=[83362], 90.00th=[83362], 95.00th=[83362], | 99.00th=[83362], 99.50th=[83362], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 8192, max= 8192, per=79.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5123KiB/s (5246kB/s)(8192KiB/1599msec) clat (msec): min=475, max=1028, avg=752.05, stdev=390.86 lat (msec): min=475, max=1028, avg=752.28, stdev=390.85 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 477], 10.00th=[ 477], 20.00th=[ 477], | 30.00th=[ 477], 40.00th=[ 477], 50.00th=[ 477], 60.00th=[ 1028], | 70.00th=[ 1028], 80.00th=[ 1028], 90.00th=[ 1028], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 6977, max= 6977, per=34.43%, avg=6977.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=12.64%, ctx=217, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9808: Sat Sep 14 07:09:45 2024 write: IOPS=2, BW=8783KiB/s (8994kB/s)(12.0MiB/1399msec) clat (msec): min=140, max=977, avg=460.80, stdev=451.78 lat (msec): min=140, max=977, avg=461.09, stdev=451.79 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 142], 10.00th=[ 142], 20.00th=[ 142], | 30.00th=[ 142], 40.00th=[ 266], 50.00th=[ 266], 60.00th=[ 266], | 70.00th=[ 978], 80.00th=[ 978], 90.00th=[ 978], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 8192, max= 8192, per=40.43%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=22.17%, ctx=38, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9809: Sat Sep 14 07:09:45 2024 write: IOPS=1, BW=7599KiB/s (7782kB/s)(12.0MiB/1617msec) clat (msec): min=110, max=1025, avg=534.91, stdev=461.33 lat (msec): min=110, max=1026, avg=535.14, stdev=461.34 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 468], 50.00th=[ 468], 60.00th=[ 468], | 70.00th=[ 1028], 80.00th=[ 1028], 90.00th=[ 1028], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 8192, max= 8192, per=40.43%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=17.57%, ctx=48, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.0MiB/s (10.5MB/s), 2562KiB/s-11.7MiB/s (2623kB/s-12.3MB/s), io=16.0MiB (16.8MB), run=1022-1599msec WRITE: bw=19.8MiB/s (20.8MB/s), 5123KiB/s-8783KiB/s (5246kB/s-8994kB/s), io=32.0MiB (33.6MB), run=1399-1617msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9811: Sat Sep 14 07:09:45 2024 read: IOPS=2, BW=8690KiB/s (8899kB/s)(12.0MiB/1414msec) clat (msec): min=163, max=922, avg=462.45, stdev=404.62 lat (msec): min=163, max=922, avg=462.45, stdev=404.62 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 165], 10.00th=[ 165], 20.00th=[ 165], | 30.00th=[ 165], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 927], 80.00th=[ 927], 90.00th=[ 927], 95.00th=[ 927], | 99.00th=[ 927], 99.50th=[ 927], 99.90th=[ 927], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 8175, max= 8175, per=80.59%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=6.58%, ctx=71, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9812: Sat Sep 14 07:09:45 2024 read: IOPS=0, BW=2536KiB/s (2597kB/s)(4096KiB/1615msec) clat (nsec): min=324554k, max=324554k, avg=324553691.00, stdev= 0.00 lat (nsec): min=324555k, max=324555k, avg=324554765.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 326], 10.00th=[ 326], 20.00th=[ 326], | 30.00th=[ 326], 40.00th=[ 326], 50.00th=[ 326], 60.00th=[ 326], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 8175, max= 8175, per=80.59%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5072KiB/s (5194kB/s)(8192KiB/1615msec) clat (msec): min=398, max=877, avg=637.89, stdev=338.97 lat (msec): min=398, max=877, avg=638.11, stdev=338.87 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 397], 10.00th=[ 397], 20.00th=[ 397], | 30.00th=[ 397], 40.00th=[ 397], 50.00th=[ 397], 60.00th=[ 877], | 70.00th=[ 877], 80.00th=[ 877], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 8175, max= 8175, per=42.59%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 1000=33.33% cpu : usr=0.06%, sys=3.59%, ctx=27, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9813: Sat Sep 14 07:09:45 2024 write: IOPS=2, BW=8365KiB/s (8566kB/s)(12.0MiB/1469msec) clat (msec): min=176, max=936, avg=483.16, stdev=401.01 lat (msec): min=176, max=937, avg=483.37, stdev=401.06 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 178], 10.00th=[ 178], 20.00th=[ 178], | 30.00th=[ 178], 40.00th=[ 338], 50.00th=[ 338], 60.00th=[ 338], | 70.00th=[ 936], 80.00th=[ 936], 90.00th=[ 936], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 8192, max= 8192, per=42.68%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=4.09%, ctx=32, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9814: Sat Sep 14 07:09:45 2024 write: IOPS=1, BW=7199KiB/s (7371kB/s)(12.0MiB/1707msec) clat (msec): min=63, max=1162, avg=559.13, stdev=557.55 lat (msec): min=63, max=1162, avg=559.43, stdev=557.61 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 64], | 30.00th=[ 64], 40.00th=[ 451], 50.00th=[ 451], 60.00th=[ 451], | 70.00th=[ 1167], 80.00th=[ 1167], 90.00th=[ 1167], 95.00th=[ 1167], | 99.00th=[ 1167], 99.50th=[ 1167], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 8175, max= 8175, per=42.59%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=8.50%, ctx=62, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9.91MiB/s (10.4MB/s), 2536KiB/s-8690KiB/s (2597kB/s-8899kB/s), io=16.0MiB (16.8MB), run=1414-1615msec WRITE: bw=18.7MiB/s (19.7MB/s), 5072KiB/s-8365KiB/s (5194kB/s-8566kB/s), io=32.0MiB (33.6MB), run=1469-1707msec