== sanity test 398b: DIO and buffer IO race ============== 16:52:40 (1730580760) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.09309 s, 46.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28843: Sat Nov 2 16:52:55 2024 read: IOPS=124, BW=496KiB/s (508kB/s)(5992KiB/12072msec) clat (usec): min=47, max=193321, avg=3706.85, stdev=5713.35 lat (usec): min=48, max=193322, avg=3707.61, stdev=5713.27 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 59], 10.00th=[ 69], 20.00th=[ 1713], | 30.00th=[ 2245], 40.00th=[ 2573], 50.00th=[ 3032], 60.00th=[ 3621], | 70.00th=[ 4490], 80.00th=[ 5800], 90.00th=[ 7177], 95.00th=[ 8455], | 99.00th=[ 13566], 99.50th=[ 15664], 99.90th=[ 32113], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 224, max= 680, per=24.30%, avg=483.39, stdev=92.46, samples=23 iops : min= 56, max= 170, avg=120.83, stdev=23.11, samples=23 write: IOPS=130, BW=522KiB/s (534kB/s)(6296KiB/12072msec) clat (usec): min=418, max=63664, avg=4067.58, stdev=4388.47 lat (usec): min=418, max=63665, avg=4068.25, stdev=4388.46 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 619], | 30.00th=[ 857], 40.00th=[ 1942], 50.00th=[ 3294], 60.00th=[ 4293], | 70.00th=[ 5342], 80.00th=[ 6521], 90.00th=[ 8717], 95.00th=[10814], | 99.00th=[18744], 99.50th=[21103], 99.90th=[57410], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 256, max= 648, per=24.39%, avg=507.78, stdev=88.75, samples=23 iops : min= 64, max= 162, avg=126.91, stdev=22.21, samples=23 lat (usec) : 50=0.29%, 100=8.56%, 250=0.65%, 500=0.91%, 750=13.05% lat (usec) : 1000=2.99% lat (msec) : 2=6.12%, 4=28.06%, 10=34.57%, 20=4.30%, 50=0.39% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.17%, sys=24.05%, ctx=6253, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28844: Sat Nov 2 16:52:55 2024 read: IOPS=125, BW=502KiB/s (514kB/s)(5940KiB/11843msec) clat (usec): min=48, max=18681, avg=3547.71, stdev=2742.43 lat (usec): min=48, max=18681, avg=3548.57, stdev=2742.11 clat percentiles (usec): | 1.00th=[ 54], 5.00th=[ 59], 10.00th=[ 68], 20.00th=[ 1123], | 30.00th=[ 2343], 40.00th=[ 2704], 50.00th=[ 3064], 60.00th=[ 3589], | 70.00th=[ 4424], 80.00th=[ 5538], 90.00th=[ 7177], 95.00th=[ 8455], | 99.00th=[11731], 99.50th=[14222], 99.90th=[18482], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 264, max= 736, per=25.11%, avg=499.39, stdev=110.15, samples=23 iops : min= 66, max= 184, avg=124.78, stdev=27.58, samples=23 write: IOPS=134, BW=536KiB/s (549kB/s)(6348KiB/11843msec) clat (usec): min=456, max=213518, avg=4066.65, stdev=6511.50 lat (usec): min=456, max=213520, avg=4067.89, stdev=6511.63 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 519], 10.00th=[ 545], 20.00th=[ 619], | 30.00th=[ 840], 40.00th=[ 1811], 50.00th=[ 3195], 60.00th=[ 4015], | 70.00th=[ 4948], 80.00th=[ 6521], 90.00th=[ 8979], 95.00th=[ 11076], | 99.00th=[ 17433], 99.50th=[ 21627], 99.90th=[ 35914], 99.95th=[212861], | 99.99th=[212861] bw ( KiB/s): min= 344, max= 864, per=25.32%, avg=527.17, stdev=107.28, samples=23 iops : min= 86, max= 216, avg=131.74, stdev=26.83, samples=23 lat (usec) : 50=0.23%, 100=8.40%, 250=0.81%, 500=1.17%, 750=12.92% lat (usec) : 1000=3.45% lat (msec) : 2=5.01%, 4=30.60%, 10=32.71%, 20=4.33%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=23.98%, ctx=6132, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28845: Sat Nov 2 16:52:55 2024 read: IOPS=126, BW=506KiB/s (518kB/s)(6028KiB/11906msec) clat (usec): min=47, max=22587, avg=3620.67, stdev=2812.58 lat (usec): min=48, max=22587, avg=3621.37, stdev=2812.54 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 59], 10.00th=[ 69], 20.00th=[ 1844], | 30.00th=[ 2278], 40.00th=[ 2638], 50.00th=[ 3064], 60.00th=[ 3720], | 70.00th=[ 4621], 80.00th=[ 5669], 90.00th=[ 7177], 95.00th=[ 8586], | 99.00th=[12387], 99.50th=[14615], 99.90th=[20055], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 280, max= 752, per=25.35%, avg=504.30, stdev=113.64, samples=23 iops : min= 70, max= 188, avg=126.04, stdev=28.43, samples=23 write: IOPS=131, BW=526KiB/s (538kB/s)(6260KiB/11906msec) clat (usec): min=439, max=216803, avg=4042.48, stdev=6691.61 lat (usec): min=440, max=216804, avg=4043.81, stdev=6691.42 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 519], 10.00th=[ 537], 20.00th=[ 619], | 30.00th=[ 824], 40.00th=[ 1647], 50.00th=[ 3130], 60.00th=[ 4015], | 70.00th=[ 4948], 80.00th=[ 6521], 90.00th=[ 8717], 95.00th=[ 10945], | 99.00th=[ 17171], 99.50th=[ 21365], 99.90th=[ 41157], 99.95th=[217056], | 99.99th=[217056] bw ( KiB/s): min= 336, max= 696, per=25.01%, avg=520.61, stdev=97.77, samples=23 iops : min= 84, max= 174, avg=130.13, stdev=24.42, samples=23 lat (usec) : 50=0.42%, 100=8.14%, 250=0.72%, 500=1.14%, 750=13.15% lat (usec) : 1000=3.03% lat (msec) : 2=5.44%, 4=29.79%, 10=33.59%, 20=4.26%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.10%, sys=24.12%, ctx=6346, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28846: Sat Nov 2 16:52:55 2024 read: IOPS=127, BW=511KiB/s (524kB/s)(6052KiB/11834msec) clat (usec): min=47, max=180655, avg=3522.55, stdev=5219.66 lat (usec): min=47, max=180656, avg=3523.32, stdev=5219.77 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 59], 10.00th=[ 69], 20.00th=[ 1795], | 30.00th=[ 2278], 40.00th=[ 2573], 50.00th=[ 2999], 60.00th=[ 3490], | 70.00th=[ 4293], 80.00th=[ 5342], 90.00th=[ 6980], 95.00th=[ 8029], | 99.00th=[ 10814], 99.50th=[ 12125], 99.90th=[ 16712], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 288, max= 824, per=25.58%, avg=508.83, stdev=102.38, samples=23 iops : min= 72, max= 206, avg=127.17, stdev=25.61, samples=23 write: IOPS=131, BW=527KiB/s (540kB/s)(6236KiB/11834msec) clat (usec): min=422, max=58721, avg=4096.59, stdev=4347.04 lat (usec): min=422, max=58722, avg=4097.81, stdev=4346.70 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 619], | 30.00th=[ 791], 40.00th=[ 1827], 50.00th=[ 3163], 60.00th=[ 4113], | 70.00th=[ 5145], 80.00th=[ 6652], 90.00th=[ 9110], 95.00th=[11994], | 99.00th=[18482], 99.50th=[23462], 99.90th=[37487], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 368, max= 744, per=25.21%, avg=524.78, stdev=98.72, samples=23 iops : min= 92, max= 186, avg=131.17, stdev=24.67, samples=23 lat (usec) : 50=0.36%, 100=8.43%, 250=0.78%, 500=1.04%, 750=13.54% lat (usec) : 1000=2.70% lat (msec) : 2=5.05%, 4=30.76%, 10=32.36%, 20=4.59%, 50=0.33% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.15%, sys=23.54%, ctx=6344, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1989KiB/s (2037kB/s), 496KiB/s-511KiB/s (508kB/s-524kB/s), io=23.4MiB (24.6MB), run=11834-12072msec WRITE: bw=2083KiB/s (2132kB/s), 522KiB/s-536KiB/s (534kB/s-549kB/s), io=24.6MiB (25.7MB), run=11834-12072msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28847: Sat Nov 2 16:53:17 2024 read: IOPS=43, BW=173KiB/s (177kB/s)(5992KiB/34589msec) clat (usec): min=1494, max=224847, avg=3588.25, stdev=6420.24 lat (usec): min=1495, max=224849, avg=3588.94, stdev=6420.28 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1811], 10.00th=[ 1876], 20.00th=[ 1942], | 30.00th=[ 2008], 40.00th=[ 2114], 50.00th=[ 2245], 60.00th=[ 2573], | 70.00th=[ 3326], 80.00th=[ 3916], 90.00th=[ 6718], 95.00th=[ 9503], | 99.00th=[ 15926], 99.50th=[ 19530], 99.90th=[ 32900], 99.95th=[225444], | 99.99th=[225444] bw ( KiB/s): min= 48, max= 312, per=25.16%, avg=173.09, stdev=67.04, samples=69 iops : min= 12, max= 78, avg=43.17, stdev=16.79, samples=69 write: IOPS=45, BW=182KiB/s (186kB/s)(6296KiB/34589msec) clat (usec): min=9960, max=61930, avg=18488.41, stdev=6673.81 lat (usec): min=9961, max=61931, avg=18489.20, stdev=6673.82 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11207], 10.00th=[11731], 20.00th=[12780], | 30.00th=[13960], 40.00th=[15795], 50.00th=[17433], 60.00th=[18744], | 70.00th=[20055], 80.00th=[21890], 90.00th=[28181], 95.00th=[32637], | 99.00th=[41157], 99.50th=[43779], 99.90th=[50594], 99.95th=[62129], | 99.99th=[62129] bw ( KiB/s): min= 40, max= 248, per=25.23%, avg=181.67, stdev=51.67, samples=69 iops : min= 10, max= 62, avg=45.32, stdev=12.96, samples=69 lat (msec) : 2=14.32%, 4=25.07%, 10=7.49%, 20=37.76%, 50=15.27% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.10%, sys=4.88%, ctx=3580, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28848: Sat Nov 2 16:53:17 2024 read: IOPS=42, BW=170KiB/s (174kB/s)(5940KiB/34881msec) clat (usec): min=1475, max=186638, avg=3561.08, stdev=5936.54 lat (usec): min=1475, max=186640, avg=3561.73, stdev=5936.57 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1778], 10.00th=[ 1844], 20.00th=[ 1926], | 30.00th=[ 1991], 40.00th=[ 2089], 50.00th=[ 2212], 60.00th=[ 2540], | 70.00th=[ 3261], 80.00th=[ 3818], 90.00th=[ 6521], 95.00th=[ 9241], | 99.00th=[ 16909], 99.50th=[ 21103], 99.90th=[ 69731], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 16, max= 328, per=24.79%, avg=170.57, stdev=71.84, samples=69 iops : min= 4, max= 82, avg=42.57, stdev=17.96, samples=69 write: IOPS=45, BW=182KiB/s (186kB/s)(6348KiB/34881msec) clat (usec): min=9711, max=56997, avg=18576.75, stdev=6583.35 lat (usec): min=9712, max=56999, avg=18577.46, stdev=6583.37 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11338], 10.00th=[11863], 20.00th=[12780], | 30.00th=[14353], 40.00th=[16188], 50.00th=[17695], 60.00th=[18744], | 70.00th=[19792], 80.00th=[21627], 90.00th=[28181], 95.00th=[32375], | 99.00th=[40633], 99.50th=[43779], 99.90th=[55837], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 48, max= 240, per=25.22%, avg=181.58, stdev=50.44, samples=69 iops : min= 12, max= 60, avg=45.32, stdev=12.61, samples=69 lat (msec) : 2=14.78%, 4=24.67%, 10=6.97%, 20=38.80%, 50=14.62% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.11%, sys=4.73%, ctx=3536, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28849: Sat Nov 2 16:53:17 2024 read: IOPS=43, BW=175KiB/s (179kB/s)(6028KiB/34444msec) clat (usec): min=1513, max=69755, avg=3514.34, stdev=3258.71 lat (usec): min=1513, max=69755, avg=3514.97, stdev=3258.72 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1811], 10.00th=[ 1860], 20.00th=[ 1942], | 30.00th=[ 2008], 40.00th=[ 2114], 50.00th=[ 2278], 60.00th=[ 2737], | 70.00th=[ 3458], 80.00th=[ 4015], 90.00th=[ 7177], 95.00th=[ 9372], | 99.00th=[14091], 99.50th=[16909], 99.90th=[36963], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 40, max= 336, per=25.45%, avg=175.09, stdev=65.77, samples=68 iops : min= 10, max= 84, avg=43.71, stdev=16.41, samples=68 write: IOPS=45, BW=182KiB/s (186kB/s)(6260KiB/34444msec) clat (msec): min=10, max=214, avg=18.55, stdev= 8.19 lat (msec): min=10, max=214, avg=18.55, stdev= 8.20 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 20], 80.00th=[ 22], 90.00th=[ 28], 95.00th=[ 32], | 99.00th=[ 41], 99.50th=[ 44], 99.90th=[ 67], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 56, max= 248, per=25.15%, avg=181.09, stdev=52.45, samples=68 iops : min= 14, max= 62, avg=45.21, stdev=13.13, samples=68 lat (msec) : 2=14.03%, 4=25.16%, 10=7.94%, 20=38.22%, 50=14.49% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.09%, sys=4.86%, ctx=3558, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28850: Sat Nov 2 16:53:17 2024 read: IOPS=43, BW=175KiB/s (179kB/s)(6052KiB/34535msec) clat (usec): min=1414, max=47776, avg=3550.18, stdev=3310.18 lat (usec): min=1415, max=47776, avg=3550.81, stdev=3310.17 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1778], 10.00th=[ 1860], 20.00th=[ 1942], | 30.00th=[ 2008], 40.00th=[ 2114], 50.00th=[ 2245], 60.00th=[ 2606], | 70.00th=[ 3359], 80.00th=[ 3884], 90.00th=[ 6915], 95.00th=[10159], | 99.00th=[15533], 99.50th=[23725], 99.90th=[32637], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 32, max= 336, per=25.47%, avg=175.22, stdev=65.70, samples=69 iops : min= 8, max= 84, avg=43.74, stdev=16.43, samples=69 write: IOPS=45, BW=181KiB/s (185kB/s)(6236KiB/34535msec) clat (msec): min=10, max=197, avg=18.63, stdev= 8.08 lat (msec): min=10, max=197, avg=18.63, stdev= 8.08 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 21], 80.00th=[ 22], 90.00th=[ 28], 95.00th=[ 33], | 99.00th=[ 41], 99.50th=[ 48], 99.90th=[ 84], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 48, max= 256, per=25.06%, avg=180.43, stdev=51.64, samples=69 iops : min= 12, max= 64, avg=45.04, stdev=12.89, samples=69 lat (msec) : 2=14.00%, 4=25.81%, 10=6.84%, 20=38.18%, 50=14.94% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.10%, sys=4.80%, ctx=3578, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=688KiB/s (705kB/s), 170KiB/s-175KiB/s (174kB/s-179kB/s), io=23.4MiB (24.6MB), run=34444-34881msec WRITE: bw=721KiB/s (738kB/s), 181KiB/s-182KiB/s (185kB/s-186kB/s), io=24.6MiB (25.7MB), run=34444-34881msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28886: Sat Nov 2 16:53:23 2024 read: IOPS=87, BW=1401KiB/s (1435kB/s)(5920KiB/4225msec) clat (usec): min=50, max=25996, avg=8519.23, stdev=4814.40 lat (usec): min=50, max=25997, avg=8519.80, stdev=4814.41 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 61], 10.00th=[ 72], 20.00th=[ 5866], | 30.00th=[ 7898], 40.00th=[ 8717], 50.00th=[ 9372], 60.00th=[10159], | 70.00th=[10814], 80.00th=[11863], 90.00th=[13304], 95.00th=[15008], | 99.00th=[19792], 99.50th=[22938], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 960, max= 1600, per=26.13%, avg=1380.00, stdev=183.78, samples=8 iops : min= 60, max= 100, avg=86.25, stdev=11.49, samples=8 write: IOPS=94, BW=1507KiB/s (1543kB/s)(6368KiB/4225msec) clat (usec): min=611, max=16549, avg=2512.84, stdev=2467.24 lat (usec): min=611, max=16550, avg=2513.83, stdev=2467.28 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 693], 10.00th=[ 742], 20.00th=[ 775], | 30.00th=[ 857], 40.00th=[ 955], 50.00th=[ 1385], 60.00th=[ 2409], | 70.00th=[ 2900], 80.00th=[ 3687], 90.00th=[ 5669], 95.00th=[ 6915], | 99.00th=[14353], 99.50th=[15139], 99.90th=[16581], 99.95th=[16581], | 99.99th=[16581] bw ( KiB/s): min= 1152, max= 1952, per=26.78%, avg=1488.00, stdev=277.92, samples=8 iops : min= 72, max= 122, avg=93.00, stdev=17.37, samples=8 lat (usec) : 100=8.33%, 250=0.91%, 500=0.13%, 750=6.51%, 1000=15.89% lat (msec) : 2=4.95%, 4=15.36%, 10=26.82%, 20=20.70%, 50=0.39% cpu : usr=0.07%, sys=22.35%, ctx=2123, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28887: Sat Nov 2 16:53:23 2024 read: IOPS=79, BW=1267KiB/s (1297kB/s)(5744KiB/4535msec) clat (usec): min=50, max=32221, avg=8091.89, stdev=4869.65 lat (usec): min=51, max=32221, avg=8092.61, stdev=4869.71 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 61], 10.00th=[ 65], 20.00th=[ 127], | 30.00th=[ 7701], 40.00th=[ 8455], 50.00th=[ 9110], 60.00th=[ 9634], | 70.00th=[10552], 80.00th=[11469], 90.00th=[12780], 95.00th=[14222], | 99.00th=[19006], 99.50th=[24511], 99.90th=[32113], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 288, max= 1600, per=22.04%, avg=1164.00, stdev=425.88, samples=8 iops : min= 18, max= 100, avg=72.75, stdev=26.62, samples=8 write: IOPS=90, BW=1443KiB/s (1478kB/s)(6544KiB/4535msec) clat (usec): min=601, max=41584, avg=3770.29, stdev=5406.14 lat (usec): min=601, max=41586, avg=3771.15, stdev=5406.21 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 701], 10.00th=[ 734], 20.00th=[ 791], | 30.00th=[ 873], 40.00th=[ 1057], 50.00th=[ 2245], 60.00th=[ 2638], | 70.00th=[ 3556], 80.00th=[ 5211], 90.00th=[ 7308], 95.00th=[14091], | 99.00th=[27919], 99.50th=[31065], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 512, max= 1696, per=22.75%, avg=1264.00, stdev=397.11, samples=8 iops : min= 32, max= 106, avg=79.00, stdev=24.82, samples=8 lat (usec) : 100=8.85%, 250=0.91%, 750=6.77%, 1000=13.02% lat (msec) : 2=5.08%, 4=14.19%, 10=30.47%, 20=18.36%, 50=2.34% cpu : usr=0.11%, sys=20.93%, ctx=2049, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28888: Sat Nov 2 16:53:23 2024 read: IOPS=92, BW=1475KiB/s (1510kB/s)(6256KiB/4242msec) clat (usec): min=50, max=23621, avg=8185.79, stdev=4989.21 lat (usec): min=50, max=23623, avg=8186.39, stdev=4989.24 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 61], 10.00th=[ 64], 20.00th=[ 128], | 30.00th=[ 7504], 40.00th=[ 8586], 50.00th=[ 9110], 60.00th=[ 9896], | 70.00th=[10683], 80.00th=[11600], 90.00th=[13173], 95.00th=[15008], | 99.00th=[21627], 99.50th=[23462], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 832, max= 1920, per=27.11%, avg=1431.62, stdev=340.73, samples=8 iops : min= 52, max= 120, avg=89.38, stdev=21.31, samples=8 write: IOPS=88, BW=1422KiB/s (1456kB/s)(6032KiB/4242msec) clat (usec): min=624, max=17628, avg=2725.13, stdev=2846.50 lat (usec): min=625, max=17628, avg=2726.57, stdev=2846.17 clat percentiles (usec): | 1.00th=[ 635], 5.00th=[ 693], 10.00th=[ 725], 20.00th=[ 783], | 30.00th=[ 832], 40.00th=[ 922], 50.00th=[ 1401], 60.00th=[ 2507], | 70.00th=[ 2868], 80.00th=[ 4359], 90.00th=[ 5932], 95.00th=[ 8586], | 99.00th=[14353], 99.50th=[17171], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 1024, max= 1888, per=24.47%, avg=1359.62, stdev=296.16, samples=8 iops : min= 64, max= 118, avg=84.88, stdev=18.62, samples=8 lat (usec) : 100=9.64%, 250=1.43%, 500=0.13%, 750=6.90%, 1000=14.84% lat (msec) : 2=3.91%, 4=12.24%, 10=29.56%, 20=20.83%, 50=0.52% cpu : usr=0.07%, sys=22.26%, ctx=2148, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28890: Sat Nov 2 16:53:23 2024 read: IOPS=87, BW=1394KiB/s (1428kB/s)(6032KiB/4326msec) clat (usec): min=52, max=30682, avg=8488.54, stdev=4900.96 lat (usec): min=52, max=30683, avg=8489.29, stdev=4900.99 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 61], 10.00th=[ 65], 20.00th=[ 6456], | 30.00th=[ 8029], 40.00th=[ 8848], 50.00th=[ 9372], 60.00th=[10028], | 70.00th=[10683], 80.00th=[11600], 90.00th=[12780], 95.00th=[15139], | 99.00th=[19530], 99.50th=[27132], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 960, max= 1536, per=25.68%, avg=1356.00, stdev=185.76, samples=8 iops : min= 60, max= 96, avg=84.75, stdev=11.61, samples=8 write: IOPS=90, BW=1446KiB/s (1481kB/s)(6256KiB/4326msec) clat (usec): min=613, max=30258, avg=2664.79, stdev=2945.85 lat (usec): min=614, max=30261, avg=2665.69, stdev=2945.93 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 701], 10.00th=[ 750], 20.00th=[ 783], | 30.00th=[ 832], 40.00th=[ 914], 50.00th=[ 1074], 60.00th=[ 2376], | 70.00th=[ 3097], 80.00th=[ 4555], 90.00th=[ 5866], 95.00th=[ 7308], | 99.00th=[15008], 99.50th=[17433], 99.90th=[30278], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 1184, max= 1952, per=25.63%, avg=1424.00, stdev=292.79, samples=8 iops : min= 74, max= 122, avg=89.00, stdev=18.30, samples=8 lat (usec) : 100=8.85%, 250=0.65%, 750=5.47%, 1000=18.23% lat (msec) : 2=4.69%, 4=10.81%, 10=30.60%, 20=20.18%, 50=0.52% cpu : usr=0.14%, sys=21.78%, ctx=2162, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5282KiB/s (5408kB/s), 1267KiB/s-1475KiB/s (1297kB/s-1510kB/s), io=23.4MiB (24.5MB), run=4225-4535msec WRITE: bw=5557KiB/s (5690kB/s), 1422KiB/s-1507KiB/s (1456kB/s-1543kB/s), io=24.6MiB (25.8MB), run=4225-4535msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28889: Sat Nov 2 16:53:27 2024 read: IOPS=41, BW=670KiB/s (686kB/s)(5920KiB/8839msec) clat (usec): min=1819, max=26404, avg=3685.31, stdev=3391.57 lat (usec): min=1820, max=26405, avg=3685.99, stdev=3391.58 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 1991], 10.00th=[ 2024], 20.00th=[ 2089], | 30.00th=[ 2180], 40.00th=[ 2278], 50.00th=[ 2409], 60.00th=[ 2606], | 70.00th=[ 2966], 80.00th=[ 3785], 90.00th=[ 7373], 95.00th=[10814], | 99.00th=[20579], 99.50th=[22414], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 448, max= 1248, per=24.49%, avg=656.59, stdev=201.09, samples=17 iops : min= 28, max= 78, avg=40.88, stdev=12.54, samples=17 write: IOPS=45, BW=720KiB/s (738kB/s)(6368KiB/8839msec) clat (usec): min=10461, max=45151, avg=18547.04, stdev=5451.02 lat (usec): min=10462, max=45153, avg=18548.11, stdev=5451.06 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11600], 10.00th=[12125], 20.00th=[13566], | 30.00th=[15139], 40.00th=[17171], 50.00th=[18220], 60.00th=[19268], | 70.00th=[20317], 80.00th=[21627], 90.00th=[25822], 95.00th=[28967], | 99.00th=[34341], 99.50th=[44827], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 416, max= 896, per=25.14%, avg=709.29, stdev=171.78, samples=17 iops : min= 26, max= 56, avg=44.18, stdev=10.79, samples=17 lat (msec) : 2=2.60%, 4=36.46%, 10=6.25%, 20=37.50%, 50=17.19% cpu : usr=0.09%, sys=5.70%, ctx=884, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28891: Sat Nov 2 16:53:27 2024 read: IOPS=40, BW=644KiB/s (659kB/s)(5744KiB/8925msec) clat (usec): min=1769, max=27761, avg=3547.64, stdev=3156.21 lat (usec): min=1769, max=27762, avg=3548.31, stdev=3156.20 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 1975], 10.00th=[ 2040], 20.00th=[ 2114], | 30.00th=[ 2180], 40.00th=[ 2278], 50.00th=[ 2376], 60.00th=[ 2540], | 70.00th=[ 2868], 80.00th=[ 3851], 90.00th=[ 6652], 95.00th=[10421], | 99.00th=[19268], 99.50th=[22414], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 256, max= 1120, per=24.22%, avg=649.29, stdev=258.67, samples=17 iops : min= 16, max= 70, avg=40.53, stdev=16.16, samples=17 write: IOPS=45, BW=733KiB/s (751kB/s)(6544KiB/8925msec) clat (usec): min=9981, max=54918, avg=18649.08, stdev=5765.78 lat (usec): min=9982, max=54920, avg=18650.17, stdev=5765.84 clat percentiles (usec): | 1.00th=[10945], 5.00th=[11600], 10.00th=[12256], 20.00th=[13829], | 30.00th=[15139], 40.00th=[16581], 50.00th=[17957], 60.00th=[19006], | 70.00th=[20317], 80.00th=[22152], 90.00th=[25822], 95.00th=[29230], | 99.00th=[36439], 99.50th=[37487], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 480, max= 896, per=25.69%, avg=724.59, stdev=152.05, samples=17 iops : min= 30, max= 56, avg=45.24, stdev= 9.44, samples=17 lat (msec) : 2=2.60%, 4=35.55%, 10=6.25%, 20=38.02%, 50=17.45% lat (msec) : 100=0.13% cpu : usr=0.02%, sys=5.57%, ctx=882, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28892: Sat Nov 2 16:53:27 2024 read: IOPS=45, BW=722KiB/s (740kB/s)(6256KiB/8661msec) clat (usec): min=1873, max=21984, avg=3791.69, stdev=3081.45 lat (usec): min=1874, max=21985, avg=3792.31, stdev=3081.45 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2008], 10.00th=[ 2040], 20.00th=[ 2114], | 30.00th=[ 2180], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2671], | 70.00th=[ 3130], 80.00th=[ 4817], 90.00th=[ 8225], 95.00th=[10814], | 99.00th=[17695], 99.50th=[19006], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 352, max= 1344, per=27.24%, avg=730.35, stdev=278.79, samples=17 iops : min= 22, max= 84, avg=45.65, stdev=17.42, samples=17 write: IOPS=43, BW=696KiB/s (713kB/s)(6032KiB/8661msec) clat (usec): min=10576, max=45834, avg=18773.16, stdev=5649.00 lat (usec): min=10577, max=45835, avg=18774.16, stdev=5649.04 clat percentiles (usec): | 1.00th=[11076], 5.00th=[11600], 10.00th=[12387], 20.00th=[14091], | 30.00th=[15401], 40.00th=[16712], 50.00th=[17957], 60.00th=[19006], | 70.00th=[20317], 80.00th=[22414], 90.00th=[26608], 95.00th=[29230], | 99.00th=[36963], 99.50th=[43779], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 448, max= 896, per=24.49%, avg=690.82, stdev=153.49, samples=17 iops : min= 28, max= 56, avg=43.18, stdev= 9.59, samples=17 lat (msec) : 2=2.21%, 4=37.11%, 10=8.72%, 20=35.03%, 50=16.93% cpu : usr=0.08%, sys=5.90%, ctx=898, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28893: Sat Nov 2 16:53:27 2024 read: IOPS=42, BW=675KiB/s (692kB/s)(6032KiB/8932msec) clat (usec): min=1838, max=26339, avg=3889.24, stdev=3487.08 lat (usec): min=1838, max=26339, avg=3889.93, stdev=3487.04 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 1958], 10.00th=[ 2024], 20.00th=[ 2114], | 30.00th=[ 2180], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2671], | 70.00th=[ 3130], 80.00th=[ 4555], 90.00th=[ 8455], 95.00th=[12387], | 99.00th=[17695], 99.50th=[25560], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 320, max= 1120, per=24.92%, avg=668.00, stdev=214.03, samples=17 iops : min= 20, max= 70, avg=41.65, stdev=13.38, samples=17 write: IOPS=43, BW=700KiB/s (717kB/s)(6256KiB/8932msec) clat (usec): min=9817, max=68369, avg=18870.45, stdev=6199.34 lat (usec): min=9818, max=68372, avg=18871.45, stdev=6199.39 clat percentiles (usec): | 1.00th=[10552], 5.00th=[11338], 10.00th=[11994], 20.00th=[13435], | 30.00th=[15270], 40.00th=[16909], 50.00th=[18220], 60.00th=[19268], | 70.00th=[20579], 80.00th=[22676], 90.00th=[27132], 95.00th=[30278], | 99.00th=[35390], 99.50th=[46400], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 416, max= 958, per=24.61%, avg=694.35, stdev=187.61, samples=17 iops : min= 26, max= 59, avg=43.29, stdev=11.61, samples=17 lat (msec) : 2=3.52%, 4=33.72%, 10=8.59%, 20=37.37%, 50=16.67% lat (msec) : 100=0.13% cpu : usr=0.07%, sys=5.63%, ctx=885, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2682KiB/s (2746kB/s), 644KiB/s-722KiB/s (659kB/s-740kB/s), io=23.4MiB (24.5MB), run=8661-8932msec WRITE: bw=2821KiB/s (2889kB/s), 696KiB/s-733KiB/s (713kB/s-751kB/s), io=24.6MiB (25.8MB), run=8661-8932msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28909: Sat Nov 2 16:53:33 2024 read: IOPS=1, BW=1595KiB/s (1634kB/s)(7168KiB/4493msec) clat (msec): min=28, max=1107, avg=584.68, stdev=478.70 lat (msec): min=28, max=1107, avg=584.68, stdev=478.70 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 64], | 30.00th=[ 174], 40.00th=[ 174], 50.00th=[ 810], 60.00th=[ 844], | 70.00th=[ 844], 80.00th=[ 1070], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 1793, max= 4087, per=51.63%, avg=2353.40, stdev=977.21, samples=5 iops : min= 1, max= 3, avg= 1.60, stdev= 0.89, samples=5 write: IOPS=1, BW=1140KiB/s (1167kB/s)(5120KiB/4493msec) clat (msec): min=52, max=121, avg=75.84, stdev=29.17 lat (msec): min=52, max=121, avg=75.88, stdev=29.15 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 53], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 61], 60.00th=[ 61], | 70.00th=[ 89], 80.00th=[ 89], 90.00th=[ 122], 95.00th=[ 122], | 99.00th=[ 122], 99.50th=[ 122], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 2043, max= 3592, per=48.17%, avg=3073.67, stdev=892.59, samples=3 iops : min= 1, max= 3, avg= 2.33, stdev= 1.15, samples=3 lat (msec) : 50=8.33%, 100=41.67%, 250=16.67%, 1000=16.67% cpu : usr=0.00%, sys=2.63%, ctx=46, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28911: Sat Nov 2 16:53:33 2024 read: IOPS=0, BW=606KiB/s (620kB/s)(2048KiB/3380msec) clat (usec): min=15002, max=21269, avg=18135.75, stdev=4431.21 lat (usec): min=15003, max=21269, avg=18136.72, stdev=4430.80 clat percentiles (usec): | 1.00th=[15008], 5.00th=[15008], 10.00th=[15008], 20.00th=[15008], | 30.00th=[15008], 40.00th=[15008], 50.00th=[15008], 60.00th=[21365], | 70.00th=[21365], 80.00th=[21365], 90.00th=[21365], 95.00th=[21365], | 99.00th=[21365], 99.50th=[21365], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 2048, max= 2048, per=44.93%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=3030KiB/s (3102kB/s)(10.0MiB/3380msec) clat (msec): min=32, max=1788, avg=330.57, stdev=602.87 lat (msec): min=33, max=1788, avg=330.63, stdev=602.86 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33], | 30.00th=[ 34], 40.00th=[ 38], 50.00th=[ 63], 60.00th=[ 78], | 70.00th=[ 94], 80.00th=[ 94], 90.00th=[ 1070], 95.00th=[ 1787], | 99.00th=[ 1787], 99.50th=[ 1787], 99.90th=[ 1787], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 2048, max=14336, per=100.00%, avg=8192.00, stdev=8688.93, samples=2 iops : min= 2, max= 14, avg= 8.00, stdev= 8.49, samples=2 lat (msec) : 20=8.33%, 50=41.67%, 100=33.33% cpu : usr=0.00%, sys=3.34%, ctx=53, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28913: Sat Nov 2 16:53:33 2024 read: IOPS=1, BW=1413KiB/s (1447kB/s)(6144KiB/4348msec) clat (msec): min=19, max=872, avg=304.90, stdev=386.00 lat (msec): min=19, max=872, avg=304.90, stdev=386.00 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 44], | 30.00th=[ 44], 40.00th=[ 52], 50.00th=[ 52], 60.00th=[ 121], | 70.00th=[ 726], 80.00th=[ 726], 90.00th=[ 869], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 2048, max= 4096, per=74.89%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 write: IOPS=1, BW=1413KiB/s (1447kB/s)(6144KiB/4348msec) clat (msec): min=35, max=1185, avg=418.05, stdev=561.63 lat (msec): min=35, max=1185, avg=418.11, stdev=561.64 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 43], | 30.00th=[ 43], 40.00th=[ 48], 50.00th=[ 48], 60.00th=[ 100], | 70.00th=[ 1099], 80.00th=[ 1099], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 2048, max= 4096, per=48.14%, avg=3072.00, stdev=1182.41, samples=4 iops : min= 2, max= 4, avg= 3.00, stdev= 1.15, samples=4 lat (msec) : 20=8.33%, 50=33.33%, 100=16.67%, 250=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.05%, sys=2.16%, ctx=37, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28915: Sat Nov 2 16:53:33 2024 read: IOPS=1, BW=1184KiB/s (1212kB/s)(5120KiB/4326msec) clat (msec): min=20, max=965, avg=356.81, stdev=463.63 lat (msec): min=20, max=965, avg=356.81, stdev=463.63 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 23], 40.00th=[ 23], 50.00th=[ 26], 60.00th=[ 26], | 70.00th=[ 751], 80.00th=[ 751], 90.00th=[ 969], 95.00th=[ 969], | 99.00th=[ 969], 99.50th=[ 969], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 2048, max= 6144, per=89.86%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=1, BW=1657KiB/s (1697kB/s)(7168KiB/4326msec) clat (msec): min=29, max=1163, avg=354.17, stdev=436.06 lat (msec): min=29, max=1163, avg=354.24, stdev=436.06 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 75], | 30.00th=[ 92], 40.00th=[ 92], 50.00th=[ 110], 60.00th=[ 253], | 70.00th=[ 253], 80.00th=[ 760], 90.00th=[ 1167], 95.00th=[ 1167], | 99.00th=[ 1167], 99.50th=[ 1167], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 2043, max= 6144, per=56.11%, avg=3580.50, stdev=1961.34, samples=4 iops : min= 1, max= 6, avg= 3.00, stdev= 2.16, samples=4 lat (msec) : 50=33.33%, 100=16.67%, 250=8.33%, 500=8.33%, 750=8.33% lat (msec) : 1000=16.67% cpu : usr=0.00%, sys=2.98%, ctx=63, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4558KiB/s (4668kB/s), 606KiB/s-1595KiB/s (620kB/s-1634kB/s), io=20.0MiB (20.0MB), run=3380-4493msec WRITE: bw=6381KiB/s (6535kB/s), 1140KiB/s-3030KiB/s (1167kB/s-3102kB/s), io=28.0MiB (29.4MB), run=3380-4493msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28910: Sat Nov 2 16:53:33 2024 read: IOPS=1, BW=1603KiB/s (1641kB/s)(7168KiB/4472msec) clat (usec): min=453, max=1015.7k, avg=602572.80, stdev=456748.96 lat (usec): min=454, max=1015.7k, avg=602573.94, stdev=456748.76 clat percentiles (usec): | 1.00th=[ 453], 5.00th=[ 453], 10.00th=[ 453], | 20.00th=[ 66847], 30.00th=[ 308282], 40.00th=[ 308282], | 50.00th=[ 926942], 60.00th=[ 935330], 70.00th=[ 935330], | 80.00th=[ 968885], 90.00th=[1019216], 95.00th=[1019216], | 99.00th=[1019216], 99.50th=[1019216], 99.90th=[1019216], | 99.95th=[1019216], 99.99th=[1019216] bw ( KiB/s): min= 1790, max= 4096, per=64.00%, avg=2878.50, stdev=1132.69, samples=4 iops : min= 1, max= 4, avg= 2.50, stdev= 1.29, samples=4 write: IOPS=1, BW=1145KiB/s (1172kB/s)(5120KiB/4472msec) clat (usec): min=20110, max=86808, avg=48961.93, stdev=26622.31 lat (usec): min=20135, max=86835, avg=49011.34, stdev=26616.12 clat percentiles (usec): | 1.00th=[20055], 5.00th=[20055], 10.00th=[20055], 20.00th=[20055], | 30.00th=[34341], 40.00th=[34341], 50.00th=[38536], 60.00th=[38536], | 70.00th=[64750], 80.00th=[64750], 90.00th=[86508], 95.00th=[86508], | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 2048, max= 7160, per=73.11%, avg=4604.00, stdev=3614.73, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (usec) : 500=8.33% lat (msec) : 50=25.00%, 100=25.00%, 500=8.33%, 1000=25.00% cpu : usr=0.00%, sys=19.75%, ctx=1270, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28912: Sat Nov 2 16:53:33 2024 read: IOPS=0, BW=450KiB/s (461kB/s)(2048KiB/4553msec) clat (msec): min=38, max=929, avg=483.92, stdev=630.49 lat (msec): min=38, max=929, avg=483.92, stdev=630.49 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 927], | 70.00th=[ 927], 80.00th=[ 927], 90.00th=[ 927], 95.00th=[ 927], | 99.00th=[ 927], 99.50th=[ 927], 99.90th=[ 927], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 1793, max= 2048, per=42.70%, avg=1920.50, stdev=180.31, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2249KiB/s (2303kB/s)(10.0MiB/4553msec) clat (msec): min=35, max=1036, avg=352.30, stdev=385.69 lat (msec): min=36, max=1036, avg=352.36, stdev=385.69 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 37], | 30.00th=[ 45], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 309], | 70.00th=[ 676], 80.00th=[ 676], 90.00th=[ 911], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 1790, max=10240, per=82.63%, avg=5203.33, stdev=4452.76, samples=3 iops : min= 1, max= 10, avg= 4.67, stdev= 4.73, samples=3 lat (msec) : 50=33.33%, 100=16.67%, 500=16.67%, 750=8.33%, 1000=16.67% cpu : usr=0.02%, sys=8.88%, ctx=374, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28914: Sat Nov 2 16:53:33 2024 read: IOPS=1, BW=1477KiB/s (1512kB/s)(6144KiB/4160msec) clat (usec): min=360, max=954264, avg=474137.22, stdev=493066.50 lat (usec): min=361, max=954265, avg=474138.11, stdev=493066.56 clat percentiles (usec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 18220], | 30.00th=[ 18220], 40.00th=[ 55837], 50.00th=[ 55837], 60.00th=[884999], | 70.00th=[935330], 80.00th=[935330], 90.00th=[952108], 95.00th=[952108], | 99.00th=[952108], 99.50th=[952108], 99.90th=[952108], 99.95th=[952108], | 99.99th=[952108] bw ( KiB/s): min= 1822, max= 5466, per=67.53%, avg=3037.67, stdev=2103.00, samples=3 iops : min= 1, max= 5, avg= 2.33, stdev= 2.31, samples=3 write: IOPS=1, BW=1477KiB/s (1512kB/s)(6144KiB/4160msec) clat (msec): min=19, max=1104, avg=210.89, stdev=437.95 lat (msec): min=19, max=1104, avg=210.94, stdev=437.94 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 24], | 30.00th=[ 24], 40.00th=[ 30], 50.00th=[ 30], 60.00th=[ 31], | 70.00th=[ 58], 80.00th=[ 58], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 1822, max= 4087, per=45.17%, avg=2844.50, stdev=1192.74, samples=4 iops : min= 1, max= 3, avg= 2.00, stdev= 1.15, samples=4 lat (usec) : 500=8.33% lat (msec) : 20=16.67%, 50=25.00%, 100=16.67%, 1000=25.00% cpu : usr=0.02%, sys=12.98%, ctx=908, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28916: Sat Nov 2 16:53:33 2024 read: IOPS=3, BW=3810KiB/s (3901kB/s)(5120KiB/1344msec) clat (usec): min=328, max=106716, avg=22251.30, stdev=47235.86 lat (usec): min=329, max=106717, avg=22251.83, stdev=47235.79 clat percentiles (usec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 330], | 30.00th=[ 343], 40.00th=[ 343], 50.00th=[ 457], 60.00th=[ 457], | 70.00th=[ 3425], 80.00th=[ 3425], 90.00th=[106431], 95.00th=[106431], | 99.00th=[106431], 99.50th=[106431], 99.90th=[106431], 99.95th=[106431], | 99.99th=[106431] write: IOPS=5, BW=5333KiB/s (5461kB/s)(7168KiB/1344msec) clat (msec): min=15, max=1039, avg=170.01, stdev=383.57 lat (msec): min=15, max=1039, avg=170.06, stdev=383.57 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16], | 30.00th=[ 25], 40.00th=[ 25], 50.00th=[ 28], 60.00th=[ 29], | 70.00th=[ 29], 80.00th=[ 40], 90.00th=[ 1036], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 2048, max= 2048, per=32.52%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 500=25.00% lat (msec) : 4=8.33%, 20=16.67%, 50=33.33%, 250=8.33% cpu : usr=0.07%, sys=13.33%, ctx=92, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4498KiB/s (4606kB/s), 450KiB/s-3810KiB/s (461kB/s-3901kB/s), io=20.0MiB (20.0MB), run=1344-4553msec WRITE: bw=6297KiB/s (6449kB/s), 1145KiB/s-5333KiB/s (1172kB/s-5461kB/s), io=28.0MiB (29.4MB), run=1344-4553msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28925: Sat Nov 2 16:53:37 2024 read: IOPS=0, BW=3531KiB/s (3616kB/s)(12.0MiB/3480msec) clat (msec): min=209, max=1756, avg=1156.15, stdev=829.55 lat (msec): min=209, max=1756, avg=1156.15, stdev=829.55 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 209], 10.00th=[ 209], 20.00th=[ 209], | 30.00th=[ 209], 40.00th=[ 1502], 50.00th=[ 1502], 60.00th=[ 1502], | 70.00th=[ 1754], 80.00th=[ 1754], 90.00th=[ 1754], 95.00th=[ 1754], | 99.00th=[ 1754], 99.50th=[ 1754], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 8062, max= 8078, per=100.00%, avg=8070.00, stdev=11.31, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=2.27%, ctx=30, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28926: Sat Nov 2 16:53:37 2024 read: IOPS=0, BW=1722KiB/s (1764kB/s)(4096KiB/2378msec) clat (nsec): min=182194k, max=182194k, avg=182193677.00, stdev= 0.00 lat (nsec): min=182196k, max=182196k, avg=182195556.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 182], 10.00th=[ 182], 20.00th=[ 182], | 30.00th=[ 182], 40.00th=[ 182], 50.00th=[ 182], 60.00th=[ 182], | 70.00th=[ 182], 80.00th=[ 182], 90.00th=[ 182], 95.00th=[ 182], | 99.00th=[ 182], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 7937, max= 7937, per=100.00%, avg=7937.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3445KiB/s (3528kB/s)(8192KiB/2378msec) clat (msec): min=136, max=2050, avg=1093.62, stdev=1353.92 lat (msec): min=136, max=2051, avg=1093.89, stdev=1353.96 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 2056], | 70.00th=[ 2056], 80.00th=[ 2056], 90.00th=[ 2056], 95.00th=[ 2056], | 99.00th=[ 2056], 99.50th=[ 2056], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7937, max= 7937, per=57.60%, avg=7937.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.04%, sys=4.04%, ctx=38, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28928: Sat Nov 2 16:53:37 2024 write: IOPS=6, BW=27.4MiB/s (28.7MB/s)(12.0MiB/438msec) clat (msec): min=69, max=231, avg=142.33, stdev=82.11 lat (msec): min=69, max=231, avg=142.64, stdev=82.09 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 70], 10.00th=[ 70], 20.00th=[ 70], | 30.00th=[ 70], 40.00th=[ 127], 50.00th=[ 127], 60.00th=[ 127], | 70.00th=[ 232], 80.00th=[ 232], 90.00th=[ 232], 95.00th=[ 232], | 99.00th=[ 232], 99.50th=[ 232], 99.90th=[ 232], 99.95th=[ 232], | 99.99th=[ 232] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.23%, sys=13.04%, ctx=29, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28930: Sat Nov 2 16:53:37 2024 write: IOPS=5, BW=23.5MiB/s (24.7MB/s)(12.0MiB/510msec) clat (msec): min=61, max=333, avg=159.10, stdev=151.66 lat (msec): min=61, max=333, avg=159.36, stdev=151.55 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 62], 10.00th=[ 62], 20.00th=[ 62], | 30.00th=[ 62], 40.00th=[ 83], 50.00th=[ 83], 60.00th=[ 83], | 70.00th=[ 334], 80.00th=[ 334], 90.00th=[ 334], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=66.67%, 500=33.33% cpu : usr=0.00%, sys=14.93%, ctx=61, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4708KiB/s (4821kB/s), 1722KiB/s-3531KiB/s (1764kB/s-3616kB/s), io=16.0MiB (16.8MB), run=2378-3480msec WRITE: bw=13.5MiB/s (14.1MB/s), 3445KiB/s-27.4MiB/s (3528kB/s-28.7MB/s), io=32.0MiB (33.6MB), run=438-2378msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28927: Sat Nov 2 16:53:37 2024 read: IOPS=0, BW=3553KiB/s (3639kB/s)(12.0MiB/3458msec) clat (msec): min=153, max=1766, avg=1147.07, stdev=869.03 lat (msec): min=153, max=1766, avg=1147.07, stdev=869.03 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 1519], 50.00th=[ 1519], 60.00th=[ 1519], | 70.00th=[ 1770], 80.00th=[ 1770], 90.00th=[ 1770], 95.00th=[ 1770], | 99.00th=[ 1770], 99.50th=[ 1770], 99.90th=[ 1770], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=46.34%, ctx=1070, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28929: Sat Nov 2 16:53:37 2024 read: IOPS=0, BW=1954KiB/s (2001kB/s)(4096KiB/2096msec) clat (nsec): min=133927k, max=133927k, avg=133927463.00, stdev= 0.00 lat (nsec): min=133929k, max=133929k, avg=133928588.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 134], 10.00th=[ 134], 20.00th=[ 134], | 30.00th=[ 134], 40.00th=[ 134], 50.00th=[ 134], 60.00th=[ 134], | 70.00th=[ 134], 80.00th=[ 134], 90.00th=[ 134], 95.00th=[ 134], | 99.00th=[ 134], 99.50th=[ 134], 99.90th=[ 134], 99.95th=[ 134], | 99.99th=[ 134] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3908KiB/s (4002kB/s)(8192KiB/2096msec) clat (msec): min=78, max=1855, avg=966.79, stdev=1256.57 lat (msec): min=78, max=1855, avg=966.97, stdev=1256.67 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 79], 10.00th=[ 79], 20.00th=[ 79], | 30.00th=[ 79], 40.00th=[ 79], 50.00th=[ 79], 60.00th=[ 1854], | 70.00th=[ 1854], 80.00th=[ 1854], 90.00th=[ 1854], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 8192, max= 8192, per=90.53%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=9.36%, ctx=123, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28931: Sat Nov 2 16:53:37 2024 write: IOPS=0, BW=3394KiB/s (3475kB/s)(12.0MiB/3621msec) clat (msec): min=352, max=1903, avg=1200.72, stdev=785.47 lat (msec): min=353, max=1903, avg=1200.99, stdev=785.48 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 355], 10.00th=[ 355], 20.00th=[ 355], | 30.00th=[ 355], 40.00th=[ 1351], 50.00th=[ 1351], 60.00th=[ 1351], | 70.00th=[ 1905], 80.00th=[ 1905], 90.00th=[ 1905], 95.00th=[ 1905], | 99.00th=[ 1905], 99.50th=[ 1905], 99.90th=[ 1905], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 5618, max= 8192, per=76.31%, avg=6905.00, stdev=1820.09, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=7.71%, ctx=54, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28932: Sat Nov 2 16:53:37 2024 write: IOPS=1, BW=6585KiB/s (6743kB/s)(12.0MiB/1866msec) clat (msec): min=124, max=1602, avg=618.40, stdev=852.36 lat (msec): min=124, max=1602, avg=618.64, stdev=852.40 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 125], 10.00th=[ 125], 20.00th=[ 125], | 30.00th=[ 125], 40.00th=[ 129], 50.00th=[ 129], 60.00th=[ 129], | 70.00th=[ 1603], 80.00th=[ 1603], 90.00th=[ 1603], 95.00th=[ 1603], | 99.00th=[ 1603], 99.50th=[ 1603], 99.90th=[ 1603], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min= 8192, max= 8192, per=90.53%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=15.71%, ctx=65, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4738KiB/s (4852kB/s), 1954KiB/s-3553KiB/s (2001kB/s-3639kB/s), io=16.0MiB (16.8MB), run=2096-3458msec WRITE: bw=9049KiB/s (9267kB/s), 3394KiB/s-6585KiB/s (3475kB/s-6743kB/s), io=32.0MiB (33.6MB), run=1866-3621msec