== sanity test 398b: DIO and buffer IO race ============== 03:36:27 (1746084987) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.45431 s, 34.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21491: Thu May 1 03:36:44 2025 read: IOPS=105, BW=423KiB/s (433kB/s)(5992KiB/14178msec) clat (usec): min=54, max=208915, avg=5224.15, stdev=6039.34 lat (usec): min=54, max=208915, avg=5225.75, stdev=6039.51 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 127], 10.00th=[ 2245], 20.00th=[ 2933], | 30.00th=[ 3359], 40.00th=[ 3916], 50.00th=[ 4555], 60.00th=[ 5473], | 70.00th=[ 6325], 80.00th=[ 7177], 90.00th=[ 8586], 95.00th=[ 9896], | 99.00th=[ 16188], 99.50th=[ 18482], 99.90th=[ 22938], 99.95th=[208667], | 99.99th=[208667] bw ( KiB/s): min= 216, max= 600, per=24.69%, avg=417.93, stdev=78.24, samples=28 iops : min= 54, max= 150, avg=104.43, stdev=19.55, samples=28 write: IOPS=111, BW=444KiB/s (455kB/s)(6296KiB/14178msec) clat (usec): min=401, max=22214, avg=3992.76, stdev=3511.30 lat (usec): min=401, max=22215, avg=3994.26, stdev=3510.78 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 603], 10.00th=[ 676], 20.00th=[ 816], | 30.00th=[ 1012], 40.00th=[ 2180], 50.00th=[ 3294], 60.00th=[ 4015], | 70.00th=[ 5145], 80.00th=[ 6718], 90.00th=[ 8848], 95.00th=[11076], | 99.00th=[14746], 99.50th=[16581], 99.90th=[21103], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 256, max= 680, per=24.78%, avg=439.36, stdev=84.67, samples=28 iops : min= 64, max= 170, avg=109.79, stdev=21.17, samples=28 lat (usec) : 100=1.24%, 250=1.86%, 500=0.68%, 750=7.39%, 1000=7.19% lat (msec) : 2=5.44%, 4=27.44%, 10=42.71%, 20=5.86%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=23.03%, ctx=5675, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21493: Thu May 1 03:36:44 2025 read: IOPS=105, BW=422KiB/s (432kB/s)(5940KiB/14087msec) clat (usec): min=54, max=25748, avg=4947.91, stdev=2847.89 lat (usec): min=54, max=25749, avg=4950.01, stdev=2846.63 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 121], 10.00th=[ 2343], 20.00th=[ 2900], | 30.00th=[ 3359], 40.00th=[ 3785], 50.00th=[ 4490], 60.00th=[ 5145], | 70.00th=[ 5866], 80.00th=[ 6915], 90.00th=[ 8455], 95.00th=[10159], | 99.00th=[14353], 99.50th=[16057], 99.90th=[20055], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 208, max= 560, per=24.82%, avg=420.18, stdev=79.62, samples=28 iops : min= 52, max= 140, avg=104.96, stdev=19.90, samples=28 write: IOPS=112, BW=451KiB/s (461kB/s)(6348KiB/14087msec) clat (usec): min=437, max=193515, avg=4201.85, stdev=5970.46 lat (usec): min=438, max=193516, avg=4203.23, stdev=5970.51 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 611], 10.00th=[ 685], 20.00th=[ 816], | 30.00th=[ 996], 40.00th=[ 2245], 50.00th=[ 3326], 60.00th=[ 4146], | 70.00th=[ 5276], 80.00th=[ 6915], 90.00th=[ 9372], 95.00th=[ 10945], | 99.00th=[ 15401], 99.50th=[ 16909], 99.90th=[ 25297], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 264, max= 640, per=25.21%, avg=447.04, stdev=90.18, samples=28 iops : min= 66, max= 160, avg=111.68, stdev=22.57, samples=28 lat (usec) : 100=1.43%, 250=1.63%, 500=0.62%, 750=6.84%, 1000=8.27% lat (msec) : 2=4.79%, 4=27.25%, 10=42.51%, 20=6.48%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=22.82%, ctx=5781, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21495: Thu May 1 03:36:44 2025 read: IOPS=106, BW=427KiB/s (437kB/s)(6028KiB/14131msec) clat (usec): min=54, max=23687, avg=4987.00, stdev=2759.25 lat (usec): min=54, max=23688, avg=4991.08, stdev=2758.31 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 190], 10.00th=[ 2376], 20.00th=[ 2933], | 30.00th=[ 3294], 40.00th=[ 3851], 50.00th=[ 4490], 60.00th=[ 5211], | 70.00th=[ 6063], 80.00th=[ 6980], 90.00th=[ 8455], 95.00th=[ 9765], | 99.00th=[13960], 99.50th=[15401], 99.90th=[22676], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 200, max= 648, per=25.09%, avg=424.79, stdev=91.49, samples=28 iops : min= 50, max= 162, avg=106.14, stdev=22.90, samples=28 write: IOPS=110, BW=443KiB/s (454kB/s)(6260KiB/14131msec) clat (usec): min=459, max=199856, avg=4174.09, stdev=6145.47 lat (usec): min=460, max=199857, avg=4175.52, stdev=6145.95 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 619], 10.00th=[ 709], 20.00th=[ 832], | 30.00th=[ 1004], 40.00th=[ 1975], 50.00th=[ 3294], 60.00th=[ 4080], | 70.00th=[ 5145], 80.00th=[ 6849], 90.00th=[ 9110], 95.00th=[ 10814], | 99.00th=[ 15795], 99.50th=[ 19530], 99.90th=[ 25822], 99.95th=[200279], | 99.99th=[200279] bw ( KiB/s): min= 272, max= 584, per=24.81%, avg=439.93, stdev=76.99, samples=28 iops : min= 68, max= 146, avg=109.93, stdev=19.25, samples=28 lat (usec) : 100=0.91%, 250=1.63%, 500=0.26%, 750=6.84%, 1000=8.33% lat (msec) : 2=5.57%, 4=27.64%, 10=43.03%, 20=5.57%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=22.96%, ctx=5871, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21496: Thu May 1 03:36:44 2025 read: IOPS=106, BW=428KiB/s (438kB/s)(6052KiB/14152msec) clat (usec): min=48, max=23204, avg=5132.67, stdev=3011.58 lat (usec): min=48, max=23206, avg=5133.39, stdev=3011.60 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 137], 10.00th=[ 2376], 20.00th=[ 2900], | 30.00th=[ 3359], 40.00th=[ 3916], 50.00th=[ 4621], 60.00th=[ 5342], | 70.00th=[ 6128], 80.00th=[ 7242], 90.00th=[ 8717], 95.00th=[10552], | 99.00th=[16057], 99.50th=[17957], 99.90th=[21103], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 208, max= 584, per=24.97%, avg=422.79, stdev=73.43, samples=28 iops : min= 52, max= 146, avg=105.64, stdev=18.37, samples=28 write: IOPS=110, BW=441KiB/s (451kB/s)(6236KiB/14152msec) clat (usec): min=462, max=187896, avg=4041.38, stdev=5857.20 lat (usec): min=463, max=187910, avg=4042.29, stdev=5857.45 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 594], 10.00th=[ 668], 20.00th=[ 791], | 30.00th=[ 938], 40.00th=[ 1532], 50.00th=[ 3097], 60.00th=[ 3982], | 70.00th=[ 5276], 80.00th=[ 6915], 90.00th=[ 9110], 95.00th=[ 11076], | 99.00th=[ 14484], 99.50th=[ 17433], 99.90th=[ 21365], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 240, max= 632, per=24.59%, avg=435.93, stdev=79.37, samples=28 iops : min= 60, max= 158, avg=108.93, stdev=19.85, samples=28 lat (usec) : 50=0.03%, 100=1.14%, 250=1.60%, 500=0.52%, 750=8.20% lat (usec) : 1000=7.68% lat (msec) : 2=5.40%, 4=26.53%, 10=42.35%, 20=6.41%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.31%, sys=22.80%, ctx=5722, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1694KiB/s (1734kB/s), 422KiB/s-428KiB/s (432kB/s-438kB/s), io=23.4MiB (24.6MB), run=14087-14178msec WRITE: bw=1773KiB/s (1816kB/s), 441KiB/s-451KiB/s (451kB/s-461kB/s), io=24.6MiB (25.7MB), run=14087-14178msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21489: Thu May 1 03:36:48 2025 read: IOPS=84, BW=339KiB/s (347kB/s)(5992KiB/17695msec) clat (usec): min=1470, max=208713, avg=4624.20, stdev=6436.01 lat (usec): min=1470, max=208715, avg=4625.12, stdev=6436.12 clat percentiles (usec): | 1.00th=[ 1582], 5.00th=[ 1762], 10.00th=[ 1860], 20.00th=[ 2008], | 30.00th=[ 2180], 40.00th=[ 2474], 50.00th=[ 2966], 60.00th=[ 3589], | 70.00th=[ 4555], 80.00th=[ 6390], 90.00th=[ 9896], 95.00th=[ 12649], | 99.00th=[ 18744], 99.50th=[ 20579], 99.90th=[ 28705], 99.95th=[208667], | 99.99th=[208667] bw ( KiB/s): min= 112, max= 824, per=24.58%, avg=333.29, stdev=197.83, samples=35 iops : min= 28, max= 206, avg=83.20, stdev=49.48, samples=35 write: IOPS=88, BW=356KiB/s (364kB/s)(6296KiB/17695msec) clat (usec): min=2069, max=33636, avg=6797.00, stdev=4602.57 lat (usec): min=2069, max=33637, avg=6797.95, stdev=4602.62 clat percentiles (usec): | 1.00th=[ 2343], 5.00th=[ 2671], 10.00th=[ 2900], 20.00th=[ 3163], | 30.00th=[ 3425], 40.00th=[ 3884], 50.00th=[ 4883], 60.00th=[ 6390], | 70.00th=[ 8455], 80.00th=[10421], 90.00th=[13304], 95.00th=[15664], | 99.00th=[22152], 99.50th=[24249], 99.90th=[32900], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 128, max= 824, per=24.66%, avg=350.20, stdev=202.90, samples=35 iops : min= 32, max= 206, avg=87.43, stdev=50.73, samples=35 lat (msec) : 2=9.18%, 4=43.55%, 10=31.12%, 20=15.01%, 50=1.11% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=11.29%, ctx=3594, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21490: Thu May 1 03:36:48 2025 read: IOPS=84, BW=336KiB/s (344kB/s)(5940KiB/17667msec) clat (usec): min=1417, max=196763, avg=4792.20, stdev=6376.41 lat (usec): min=1418, max=196766, avg=4792.90, stdev=6376.48 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 1762], 10.00th=[ 1860], 20.00th=[ 1991], | 30.00th=[ 2147], 40.00th=[ 2442], 50.00th=[ 3032], 60.00th=[ 3752], | 70.00th=[ 4817], 80.00th=[ 6718], 90.00th=[ 10421], 95.00th=[ 13304], | 99.00th=[ 20055], 99.50th=[ 22676], 99.90th=[ 30278], 99.95th=[196084], | 99.99th=[196084] bw ( KiB/s): min= 96, max= 808, per=24.60%, avg=333.51, stdev=203.87, samples=35 iops : min= 24, max= 202, avg=83.26, stdev=50.99, samples=35 write: IOPS=89, BW=359KiB/s (368kB/s)(6348KiB/17667msec) clat (usec): min=2154, max=33158, avg=6605.78, stdev=4403.68 lat (usec): min=2155, max=33158, avg=6606.65, stdev=4403.74 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2671], 10.00th=[ 2835], 20.00th=[ 3130], | 30.00th=[ 3392], 40.00th=[ 3851], 50.00th=[ 4817], 60.00th=[ 6128], | 70.00th=[ 8160], 80.00th=[10028], 90.00th=[12911], 95.00th=[15401], | 99.00th=[21103], 99.50th=[24249], 99.90th=[29230], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 152, max= 816, per=24.89%, avg=353.40, stdev=200.48, samples=35 iops : min= 38, max= 204, avg=88.23, stdev=50.13, samples=35 lat (msec) : 2=9.80%, 4=42.29%, 10=32.26%, 20=14.55%, 50=1.07% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=11.09%, ctx=3534, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21492: Thu May 1 03:36:48 2025 read: IOPS=85, BW=342KiB/s (351kB/s)(6028KiB/17601msec) clat (usec): min=1462, max=25681, avg=4591.42, stdev=3774.23 lat (usec): min=1462, max=25681, avg=4592.16, stdev=3774.32 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1762], 10.00th=[ 1844], 20.00th=[ 2008], | 30.00th=[ 2212], 40.00th=[ 2573], 50.00th=[ 3195], 60.00th=[ 3752], | 70.00th=[ 4752], 80.00th=[ 6390], 90.00th=[10028], 95.00th=[12780], | 99.00th=[18744], 99.50th=[22152], 99.90th=[25035], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 112, max= 766, per=25.10%, avg=340.37, stdev=192.43, samples=35 iops : min= 28, max= 191, avg=84.97, stdev=48.13, samples=35 write: IOPS=88, BW=356KiB/s (364kB/s)(6260KiB/17601msec) clat (msec): min=2, max=198, avg= 6.78, stdev= 6.57 lat (msec): min=2, max=198, avg= 6.78, stdev= 6.57 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 11], 90.00th=[ 14], 95.00th=[ 16], | 99.00th=[ 22], 99.50th=[ 27], 99.90th=[ 34], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 136, max= 848, per=24.82%, avg=352.49, stdev=205.73, samples=35 iops : min= 34, max= 212, avg=88.00, stdev=51.45, samples=35 lat (msec) : 2=9.70%, 4=42.35%, 10=32.26%, 20=14.71%, 50=0.94% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=11.25%, ctx=3579, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21494: Thu May 1 03:36:48 2025 read: IOPS=85, BW=343KiB/s (351kB/s)(6052KiB/17654msec) clat (usec): min=1464, max=30096, avg=4683.30, stdev=3923.76 lat (usec): min=1465, max=30097, avg=4684.11, stdev=3923.77 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 1745], 10.00th=[ 1844], 20.00th=[ 2024], | 30.00th=[ 2212], 40.00th=[ 2540], 50.00th=[ 3163], 60.00th=[ 3785], | 70.00th=[ 4686], 80.00th=[ 6783], 90.00th=[10552], 95.00th=[13435], | 99.00th=[17433], 99.50th=[23200], 99.90th=[28181], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 112, max= 846, per=24.95%, avg=338.31, stdev=193.31, samples=35 iops : min= 28, max= 211, avg=84.46, stdev=48.34, samples=35 write: IOPS=88, BW=353KiB/s (362kB/s)(6236KiB/17654msec) clat (usec): min=1970, max=198528, avg=6735.39, stdev=6504.69 lat (usec): min=1970, max=198529, avg=6736.33, stdev=6504.75 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 13], 95.00th=[ 16], | 99.00th=[ 21], 99.50th=[ 25], 99.90th=[ 30], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 144, max= 784, per=24.58%, avg=349.06, stdev=198.71, samples=35 iops : min= 36, max= 196, avg=87.14, stdev=49.69, samples=35 lat (msec) : 2=9.28%, 4=42.25%, 10=32.68%, 20=14.75%, 50=1.01% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=11.17%, ctx=3604, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1357KiB/s (1390kB/s), 336KiB/s-343KiB/s (344kB/s-351kB/s), io=23.4MiB (24.6MB), run=17601-17695msec WRITE: bw=1421KiB/s (1455kB/s), 353KiB/s-359KiB/s (362kB/s-368kB/s), io=24.6MiB (25.7MB), run=17601-17695msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21524: Thu May 1 03:36:52 2025 read: IOPS=110, BW=1771KiB/s (1814kB/s)(5920KiB/3342msec) clat (usec): min=53, max=24806, avg=4581.81, stdev=2755.92 lat (usec): min=53, max=24807, avg=4582.48, stdev=2755.93 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 94], 10.00th=[ 141], 20.00th=[ 2802], | 30.00th=[ 3326], 40.00th=[ 3752], 50.00th=[ 4228], 60.00th=[ 4752], | 70.00th=[ 5604], 80.00th=[ 6521], 90.00th=[ 7898], 95.00th=[ 9372], | 99.00th=[12256], 99.50th=[13960], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 1472, max= 1920, per=25.40%, avg=1717.33, stdev=149.64, samples=6 iops : min= 92, max= 120, avg=107.33, stdev= 9.35, samples=6 write: IOPS=119, BW=1905KiB/s (1951kB/s)(6368KiB/3342msec) clat (usec): min=639, max=22327, avg=4089.54, stdev=3298.18 lat (usec): min=640, max=22328, avg=4092.22, stdev=3298.84 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 799], 10.00th=[ 898], 20.00th=[ 1106], | 30.00th=[ 1614], 40.00th=[ 2737], 50.00th=[ 3458], 60.00th=[ 4113], | 70.00th=[ 4948], 80.00th=[ 6521], 90.00th=[ 8586], 95.00th=[10421], | 99.00th=[15139], 99.50th=[16909], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 1696, max= 2048, per=26.31%, avg=1872.00, stdev=138.38, samples=6 iops : min= 106, max= 128, avg=117.00, stdev= 8.65, samples=6 lat (usec) : 100=2.73%, 250=2.21%, 500=0.13%, 750=1.17%, 1000=5.73% lat (msec) : 2=11.46%, 4=28.91%, 10=43.49%, 20=3.91%, 50=0.26% cpu : usr=0.33%, sys=25.38%, ctx=1622, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21525: Thu May 1 03:36:52 2025 read: IOPS=101, BW=1622KiB/s (1661kB/s)(5744KiB/3542msec) clat (usec): min=60, max=20339, avg=5005.76, stdev=3139.85 lat (usec): min=61, max=20340, avg=5006.41, stdev=3139.86 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 97], 10.00th=[ 126], 20.00th=[ 2900], | 30.00th=[ 3458], 40.00th=[ 3982], 50.00th=[ 4686], 60.00th=[ 5342], | 70.00th=[ 5997], 80.00th=[ 6915], 90.00th=[ 8717], 95.00th=[10814], | 99.00th=[15139], 99.50th=[17695], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 1344, max= 1824, per=24.00%, avg=1622.86, stdev=156.46, samples=7 iops : min= 84, max= 114, avg=101.43, stdev= 9.78, samples=7 write: IOPS=115, BW=1848KiB/s (1892kB/s)(6544KiB/3542msec) clat (usec): min=637, max=21939, avg=4232.82, stdev=3417.58 lat (usec): min=638, max=21939, avg=4235.99, stdev=3421.46 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 799], 10.00th=[ 881], 20.00th=[ 1090], | 30.00th=[ 1500], 40.00th=[ 2606], 50.00th=[ 3261], 60.00th=[ 4146], | 70.00th=[ 5276], 80.00th=[ 6980], 90.00th=[ 9503], 95.00th=[10552], | 99.00th=[13960], 99.50th=[14746], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 1504, max= 2336, per=25.96%, avg=1846.86, stdev=291.95, samples=7 iops : min= 94, max= 146, avg=115.43, stdev=18.25, samples=7 lat (usec) : 100=2.73%, 250=2.21%, 500=0.13%, 750=0.91%, 1000=7.29% lat (msec) : 2=9.64%, 4=26.95%, 10=43.49%, 20=6.38%, 50=0.26% cpu : usr=0.20%, sys=24.85%, ctx=1691, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21526: Thu May 1 03:36:52 2025 read: IOPS=110, BW=1771KiB/s (1813kB/s)(6256KiB/3533msec) clat (usec): min=62, max=19967, avg=4722.95, stdev=2930.97 lat (usec): min=62, max=19968, avg=4723.64, stdev=2930.98 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 96], 10.00th=[ 119], 20.00th=[ 2868], | 30.00th=[ 3359], 40.00th=[ 3818], 50.00th=[ 4359], 60.00th=[ 5080], | 70.00th=[ 5669], 80.00th=[ 6718], 90.00th=[ 8094], 95.00th=[10159], | 99.00th=[14484], 99.50th=[17957], 99.90th=[20055], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 1472, max= 2240, per=26.37%, avg=1782.86, stdev=252.45, samples=7 iops : min= 92, max= 140, avg=111.43, stdev=15.78, samples=7 write: IOPS=106, BW=1707KiB/s (1748kB/s)(6032KiB/3533msec) clat (usec): min=662, max=24150, avg=4405.48, stdev=3557.79 lat (usec): min=663, max=24151, avg=4406.54, stdev=3557.86 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 840], 10.00th=[ 963], 20.00th=[ 1156], | 30.00th=[ 1729], 40.00th=[ 2835], 50.00th=[ 3621], 60.00th=[ 4555], | 70.00th=[ 5604], 80.00th=[ 6915], 90.00th=[ 9110], 95.00th=[11338], | 99.00th=[15664], 99.50th=[17957], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 1376, max= 1952, per=23.78%, avg=1691.43, stdev=240.48, samples=7 iops : min= 86, max= 122, avg=105.71, stdev=15.03, samples=7 lat (usec) : 100=3.26%, 250=2.73%, 750=1.04%, 1000=4.95% lat (msec) : 2=10.68%, 4=26.04%, 10=44.27%, 20=6.90%, 50=0.13% cpu : usr=0.23%, sys=24.55%, ctx=1554, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21527: Thu May 1 03:36:52 2025 read: IOPS=109, BW=1752KiB/s (1795kB/s)(6032KiB/3442msec) clat (usec): min=61, max=18351, avg=4763.96, stdev=2872.89 lat (usec): min=61, max=18352, avg=4764.65, stdev=2872.89 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 94], 10.00th=[ 119], 20.00th=[ 2900], | 30.00th=[ 3392], 40.00th=[ 3785], 50.00th=[ 4424], 60.00th=[ 5080], | 70.00th=[ 5997], 80.00th=[ 6783], 90.00th=[ 8356], 95.00th=[10028], | 99.00th=[13829], 99.50th=[14615], 99.90th=[18482], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 1472, max= 1856, per=24.21%, avg=1637.33, stdev=142.27, samples=6 iops : min= 92, max= 116, avg=102.33, stdev= 8.89, samples=6 write: IOPS=113, BW=1818KiB/s (1861kB/s)(6256KiB/3442msec) clat (usec): min=552, max=18043, avg=4133.82, stdev=3546.83 lat (usec): min=553, max=18044, avg=4134.91, stdev=3546.84 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 807], 10.00th=[ 914], 20.00th=[ 1074], | 30.00th=[ 1237], 40.00th=[ 2245], 50.00th=[ 3130], 60.00th=[ 4047], | 70.00th=[ 5080], 80.00th=[ 6783], 90.00th=[ 9503], 95.00th=[11994], | 99.00th=[15926], 99.50th=[16712], 99.90th=[17957], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 1632, max= 1952, per=24.59%, avg=1749.33, stdev=119.16, samples=6 iops : min= 102, max= 122, avg=109.33, stdev= 7.45, samples=6 lat (usec) : 100=3.78%, 250=1.82%, 750=1.69%, 1000=6.64% lat (msec) : 2=11.59%, 4=26.30%, 10=41.28%, 20=6.90% cpu : usr=0.17%, sys=25.02%, ctx=1561, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6762KiB/s (6925kB/s), 1622KiB/s-1771KiB/s (1661kB/s-1814kB/s), io=23.4MiB (24.5MB), run=3342-3542msec WRITE: bw=7115KiB/s (7285kB/s), 1707KiB/s-1905KiB/s (1748kB/s-1951kB/s), io=24.6MiB (25.8MB), run=3342-3542msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21528: Thu May 1 03:36:53 2025 read: IOPS=83, BW=1338KiB/s (1371kB/s)(5920KiB/4423msec) clat (usec): min=1671, max=21770, avg=4650.37, stdev=3702.37 lat (usec): min=1672, max=21770, avg=4651.12, stdev=3702.37 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1909], 10.00th=[ 2040], 20.00th=[ 2245], | 30.00th=[ 2442], 40.00th=[ 2671], 50.00th=[ 2999], 60.00th=[ 3458], | 70.00th=[ 4555], 80.00th=[ 6194], 90.00th=[10945], 95.00th=[12911], | 99.00th=[17171], 99.50th=[19792], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 766, max= 2208, per=21.85%, avg=1179.38, stdev=455.67, samples=8 iops : min= 47, max= 138, avg=73.50, stdev=28.64, samples=8 write: IOPS=89, BW=1440KiB/s (1474kB/s)(6368KiB/4423msec) clat (usec): min=2307, max=23334, avg=6736.58, stdev=4058.13 lat (usec): min=2308, max=23336, avg=6737.92, stdev=4058.19 clat percentiles (usec): | 1.00th=[ 2507], 5.00th=[ 2900], 10.00th=[ 3130], 20.00th=[ 3556], | 30.00th=[ 3949], 40.00th=[ 4359], 50.00th=[ 5145], 60.00th=[ 6325], | 70.00th=[ 7767], 80.00th=[10159], 90.00th=[12911], 95.00th=[15008], | 99.00th=[19792], 99.50th=[22414], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 960, max= 2176, per=22.53%, avg=1279.12, stdev=427.61, samples=8 iops : min= 60, max= 136, avg=79.75, stdev=26.68, samples=8 lat (msec) : 2=4.04%, 4=43.88%, 10=35.55%, 20=16.02%, 50=0.52% cpu : usr=0.23%, sys=11.87%, ctx=913, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21529: Thu May 1 03:36:53 2025 read: IOPS=80, BW=1294KiB/s (1325kB/s)(5744KiB/4438msec) clat (usec): min=1660, max=23248, avg=4659.47, stdev=3683.82 lat (usec): min=1660, max=23249, avg=4660.15, stdev=3683.83 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2040], 10.00th=[ 2147], 20.00th=[ 2343], | 30.00th=[ 2540], 40.00th=[ 2835], 50.00th=[ 3163], 60.00th=[ 3654], | 70.00th=[ 4555], 80.00th=[ 5735], 90.00th=[10028], 95.00th=[13698], | 99.00th=[18220], 99.50th=[21365], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 798, max= 2208, per=21.86%, avg=1179.75, stdev=439.49, samples=8 iops : min= 49, max= 138, avg=73.62, stdev=27.58, samples=8 write: IOPS=92, BW=1475KiB/s (1510kB/s)(6544KiB/4438msec) clat (usec): min=2325, max=25905, avg=6664.90, stdev=4149.79 lat (usec): min=2326, max=25906, avg=6666.23, stdev=4149.84 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 2966], 10.00th=[ 3163], 20.00th=[ 3458], | 30.00th=[ 3851], 40.00th=[ 4293], 50.00th=[ 4948], 60.00th=[ 5997], | 70.00th=[ 7439], 80.00th=[ 9896], 90.00th=[12518], 95.00th=[15139], | 99.00th=[20055], 99.50th=[21365], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 1024, max= 2144, per=22.75%, avg=1291.62, stdev=371.82, samples=8 iops : min= 64, max= 134, avg=80.63, stdev=23.31, samples=8 lat (msec) : 2=1.82%, 4=45.96%, 10=37.24%, 20=13.93%, 50=1.04% cpu : usr=0.11%, sys=11.88%, ctx=937, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21530: Thu May 1 03:36:53 2025 read: IOPS=88, BW=1411KiB/s (1445kB/s)(6256KiB/4433msec) clat (usec): min=1611, max=23796, avg=4600.81, stdev=3529.02 lat (usec): min=1612, max=23797, avg=4601.51, stdev=3529.03 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1876], 10.00th=[ 2024], 20.00th=[ 2212], | 30.00th=[ 2409], 40.00th=[ 2671], 50.00th=[ 3097], 60.00th=[ 3589], | 70.00th=[ 4752], 80.00th=[ 6587], 90.00th=[ 9896], 95.00th=[12125], | 99.00th=[17171], 99.50th=[19792], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 800, max= 2112, per=22.53%, avg=1216.00, stdev=453.19, samples=8 iops : min= 50, max= 132, avg=76.00, stdev=28.32, samples=8 write: IOPS=85, BW=1361KiB/s (1393kB/s)(6032KiB/4433msec) clat (usec): min=2360, max=24053, avg=6845.38, stdev=4253.39 lat (usec): min=2361, max=24054, avg=6846.68, stdev=4253.52 clat percentiles (usec): | 1.00th=[ 2507], 5.00th=[ 2900], 10.00th=[ 3130], 20.00th=[ 3458], | 30.00th=[ 3851], 40.00th=[ 4359], 50.00th=[ 5276], 60.00th=[ 6587], | 70.00th=[ 7963], 80.00th=[ 9765], 90.00th=[12649], 95.00th=[16057], | 99.00th=[21627], 99.50th=[23987], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 960, max= 2304, per=21.35%, avg=1212.00, stdev=453.01, samples=8 iops : min= 60, max= 144, avg=75.75, stdev=28.31, samples=8 lat (msec) : 2=4.30%, 4=44.79%, 10=36.72%, 20=13.41%, 50=0.78% cpu : usr=0.18%, sys=11.73%, ctx=896, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21531: Thu May 1 03:36:53 2025 read: IOPS=85, BW=1370KiB/s (1403kB/s)(6032KiB/4403msec) clat (usec): min=1687, max=23907, avg=4763.54, stdev=3788.10 lat (usec): min=1687, max=23908, avg=4764.36, stdev=3788.25 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1958], 10.00th=[ 2073], 20.00th=[ 2278], | 30.00th=[ 2540], 40.00th=[ 2769], 50.00th=[ 3163], 60.00th=[ 3752], | 70.00th=[ 4621], 80.00th=[ 6456], 90.00th=[10945], 95.00th=[13173], | 99.00th=[17171], 99.50th=[23725], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 928, max= 1984, per=22.23%, avg=1199.75, stdev=356.53, samples=8 iops : min= 58, max= 124, avg=74.87, stdev=22.37, samples=8 write: IOPS=88, BW=1421KiB/s (1455kB/s)(6256KiB/4403msec) clat (usec): min=2207, max=26514, avg=6534.57, stdev=3912.29 lat (usec): min=2208, max=26515, avg=6535.88, stdev=3912.32 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2966], 10.00th=[ 3195], 20.00th=[ 3556], | 30.00th=[ 3916], 40.00th=[ 4359], 50.00th=[ 5145], 60.00th=[ 6128], | 70.00th=[ 7439], 80.00th=[ 8848], 90.00th=[12518], 95.00th=[15008], | 99.00th=[20055], 99.50th=[23725], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 1053, max= 2336, per=22.68%, avg=1287.62, stdev=442.56, samples=8 iops : min= 65, max= 146, avg=80.38, stdev=27.72, samples=8 lat (msec) : 2=3.65%, 4=44.40%, 10=37.63%, 20=13.54%, 50=0.78% cpu : usr=0.23%, sys=11.95%, ctx=933, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5397KiB/s (5527kB/s), 1294KiB/s-1411KiB/s (1325kB/s-1445kB/s), io=23.4MiB (24.5MB), run=4403-4438msec WRITE: bw=5678KiB/s (5815kB/s), 1361KiB/s-1475KiB/s (1393kB/s-1510kB/s), io=24.6MiB (25.8MB), run=4403-4438msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21540: Thu May 1 03:36:55 2025 read: IOPS=14, BW=14.9MiB/s (15.6MB/s)(7168KiB/470msec) clat (usec): min=428, max=38239, avg=24129.93, stdev=12827.01 lat (usec): min=429, max=38241, avg=24131.42, stdev=12827.14 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 429], 10.00th=[ 429], 20.00th=[15926], | 30.00th=[21365], 40.00th=[21365], 50.00th=[29230], 60.00th=[30016], | 70.00th=[30016], 80.00th=[33817], 90.00th=[38011], 95.00th=[38011], | 99.00th=[38011], 99.50th=[38011], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] write: IOPS=10, BW=10.6MiB/s (11.2MB/s)(5120KiB/470msec) clat (usec): min=25046, max=76831, avg=52634.75, stdev=21594.11 lat (usec): min=25078, max=76864, avg=52677.62, stdev=21591.89 clat percentiles (usec): | 1.00th=[25035], 5.00th=[25035], 10.00th=[25035], 20.00th=[25035], | 30.00th=[35914], 40.00th=[35914], 50.00th=[58459], 60.00th=[58459], | 70.00th=[66847], 80.00th=[66847], 90.00th=[77071], 95.00th=[77071], | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] lat (usec) : 500=8.33% lat (msec) : 20=8.33%, 50=58.33%, 100=25.00% cpu : usr=0.00%, sys=42.00%, ctx=780, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21541: Thu May 1 03:36:55 2025 read: IOPS=2, BW=2660KiB/s (2724kB/s)(2048KiB/770msec) clat (usec): min=27890, max=31662, avg=29776.45, stdev=2667.24 lat (usec): min=27892, max=31663, avg=29777.95, stdev=2665.95 clat percentiles (usec): | 1.00th=[27919], 5.00th=[27919], 10.00th=[27919], 20.00th=[27919], | 30.00th=[27919], 40.00th=[27919], 50.00th=[27919], 60.00th=[31589], | 70.00th=[31589], 80.00th=[31589], 90.00th=[31589], 95.00th=[31589], | 99.00th=[31589], 99.50th=[31589], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 4096, max= 4096, per=15.40%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.0MiB/s (13.6MB/s)(10.0MiB/770msec) clat (msec): min=28, max=104, avg=66.86, stdev=25.75 lat (msec): min=28, max=104, avg=66.91, stdev=25.74 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 40], | 30.00th=[ 46], 40.00th=[ 55], 50.00th=[ 66], 60.00th=[ 69], | 70.00th=[ 90], 80.00th=[ 90], 90.00th=[ 102], 95.00th=[ 106], | 99.00th=[ 106], 99.50th=[ 106], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min=10240, max=10240, per=27.50%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=41.67%, 250=16.67% cpu : usr=0.00%, sys=36.28%, ctx=88, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21542: Thu May 1 03:36:55 2025 read: IOPS=13, BW=13.5MiB/s (14.2MB/s)(6144KiB/444msec) clat (usec): min=3858, max=34452, avg=26722.90, stdev=11515.34 lat (usec): min=3859, max=34453, avg=26723.82, stdev=11515.50 clat percentiles (usec): | 1.00th=[ 3851], 5.00th=[ 3851], 10.00th=[ 3851], 20.00th=[27657], | 30.00th=[27657], 40.00th=[28705], 50.00th=[28705], 60.00th=[31851], | 70.00th=[33817], 80.00th=[33817], 90.00th=[34341], 95.00th=[34341], | 99.00th=[34341], 99.50th=[34341], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] write: IOPS=13, BW=13.5MiB/s (14.2MB/s)(6144KiB/444msec) clat (usec): min=28670, max=93140, avg=45916.89, stdev=24249.14 lat (usec): min=28688, max=93212, avg=45989.85, stdev=24241.51 clat percentiles (usec): | 1.00th=[28705], 5.00th=[28705], 10.00th=[28705], 20.00th=[30802], | 30.00th=[30802], 40.00th=[32375], 50.00th=[32375], 60.00th=[44303], | 70.00th=[46400], 80.00th=[46400], 90.00th=[92799], 95.00th=[92799], | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] lat (msec) : 4=8.33%, 50=83.33%, 100=8.33% cpu : usr=0.00%, sys=47.63%, ctx=505, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21543: Thu May 1 03:36:55 2025 read: IOPS=9, BW=9790KiB/s (10.0MB/s)(5120KiB/523msec) clat (usec): min=412, max=35316, avg=13394.82, stdev=17787.22 lat (usec): min=413, max=35316, avg=13395.79, stdev=17787.22 clat percentiles (usec): | 1.00th=[ 412], 5.00th=[ 412], 10.00th=[ 412], 20.00th=[ 412], | 30.00th=[ 441], 40.00th=[ 441], 50.00th=[ 562], 60.00th=[ 562], | 70.00th=[30278], 80.00th=[30278], 90.00th=[35390], 95.00th=[35390], | 99.00th=[35390], 99.50th=[35390], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 8192, max= 8192, per=30.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.4MiB/s (14.0MB/s)(7168KiB/523msec) clat (msec): min=28, max=130, avg=57.59, stdev=35.31 lat (msec): min=28, max=130, avg=57.67, stdev=35.37 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 36], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 44], 60.00th=[ 48], | 70.00th=[ 48], 80.00th=[ 75], 90.00th=[ 131], 95.00th=[ 131], | 99.00th=[ 131], 99.50th=[ 131], 99.90th=[ 131], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min=12288, max=12288, per=33.00%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 50=58.33%, 100=8.33%, 250=8.33% cpu : usr=0.00%, sys=35.25%, ctx=334, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.0MiB/s (27.2MB/s), 2660KiB/s-14.9MiB/s (2724kB/s-15.6MB/s), io=20.0MiB (20.0MB), run=444-770msec WRITE: bw=36.4MiB/s (38.1MB/s), 10.6MiB/s-13.5MiB/s (11.2MB/s-14.2MB/s), io=28.0MiB (29.4MB), run=444-770msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21545: Thu May 1 03:36:55 2025 read: IOPS=11, BW=11.6MiB/s (12.1MB/s)(7168KiB/606msec) clat (msec): min=10, max=118, avg=46.25, stdev=42.66 lat (msec): min=10, max=118, avg=46.25, stdev=42.66 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 11], | 30.00th=[ 17], 40.00th=[ 17], 50.00th=[ 20], 60.00th=[ 71], | 70.00th=[ 71], 80.00th=[ 79], 90.00th=[ 118], 95.00th=[ 118], | 99.00th=[ 118], 99.50th=[ 118], 99.90th=[ 118], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min=10240, max=10240, per=38.90%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8449KiB/s (8652kB/s)(5120KiB/606msec) clat (usec): min=14414, max=61835, avg=35474.50, stdev=20877.27 lat (usec): min=14480, max=61892, avg=35538.11, stdev=20870.55 clat percentiles (usec): | 1.00th=[14353], 5.00th=[14353], 10.00th=[14353], 20.00th=[14353], | 30.00th=[15401], 40.00th=[15401], 50.00th=[36439], 60.00th=[36439], | 70.00th=[49546], 80.00th=[49546], 90.00th=[61604], 95.00th=[61604], | 99.00th=[61604], 99.50th=[61604], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 2048, max= 2048, per=5.56%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=50.00%, 50=16.67%, 100=25.00%, 250=8.33% cpu : usr=0.17%, sys=19.01%, ctx=53, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21546: Thu May 1 03:36:55 2025 read: IOPS=2, BW=2632KiB/s (2696kB/s)(2048KiB/778msec) clat (usec): min=10390, max=29479, avg=19934.87, stdev=13497.56 lat (usec): min=10391, max=29481, avg=19936.60, stdev=13498.53 clat percentiles (usec): | 1.00th=[10421], 5.00th=[10421], 10.00th=[10421], 20.00th=[10421], | 30.00th=[10421], 40.00th=[10421], 50.00th=[10421], 60.00th=[29492], | 70.00th=[29492], 80.00th=[29492], 90.00th=[29492], 95.00th=[29492], | 99.00th=[29492], 99.50th=[29492], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 4087, max= 4087, per=15.53%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.9MiB/s (13.5MB/s)(10.0MiB/778msec) clat (msec): min=21, max=126, avg=64.14, stdev=29.64 lat (msec): min=21, max=126, avg=64.21, stdev=29.64 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 37], | 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 65], 60.00th=[ 74], | 70.00th=[ 77], 80.00th=[ 77], 90.00th=[ 81], 95.00th=[ 127], | 99.00th=[ 127], 99.50th=[ 127], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min=10219, max=10219, per=27.73%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=41.67%, 250=8.33% cpu : usr=0.13%, sys=20.85%, ctx=86, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21547: Thu May 1 03:36:55 2025 read: IOPS=8, BW=8440KiB/s (8642kB/s)(6144KiB/728msec) clat (msec): min=12, max=140, avg=56.60, stdev=52.63 lat (msec): min=12, max=140, avg=56.60, stdev=52.63 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 16], 60.00th=[ 65], | 70.00th=[ 93], 80.00th=[ 93], 90.00th=[ 142], 95.00th=[ 142], | 99.00th=[ 142], 99.50th=[ 142], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 4096, max= 4096, per=15.56%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8440KiB/s (8642kB/s)(6144KiB/728msec) clat (msec): min=12, max=135, avg=63.10, stdev=53.48 lat (msec): min=12, max=135, avg=63.15, stdev=53.48 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 16], | 30.00th=[ 16], 40.00th=[ 20], 50.00th=[ 20], 60.00th=[ 97], | 70.00th=[ 99], 80.00th=[ 99], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 6144, max= 6144, per=16.67%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=50.00%, 100=33.33%, 250=16.67% cpu : usr=0.00%, sys=13.48%, ctx=59, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21548: Thu May 1 03:36:55 2025 read: IOPS=6, BW=6966KiB/s (7133kB/s)(5120KiB/735msec) clat (usec): min=10741, max=59362, avg=29553.72, stdev=20741.39 lat (usec): min=10742, max=59364, avg=29554.66, stdev=20741.78 clat percentiles (usec): | 1.00th=[10683], 5.00th=[10683], 10.00th=[10683], 20.00th=[10683], | 30.00th=[13304], 40.00th=[13304], 50.00th=[22152], 60.00th=[22152], | 70.00th=[42206], 80.00th=[42206], 90.00th=[59507], 95.00th=[59507], | 99.00th=[59507], 99.50th=[59507], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 4096, max= 4096, per=15.56%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9752KiB/s (9986kB/s)(7168KiB/735msec) clat (msec): min=15, max=127, avg=67.56, stdev=37.58 lat (msec): min=15, max=127, avg=67.62, stdev=37.58 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 30], | 30.00th=[ 59], 40.00th=[ 59], 50.00th=[ 74], 60.00th=[ 82], | 70.00th=[ 82], 80.00th=[ 87], 90.00th=[ 128], 95.00th=[ 128], | 99.00th=[ 128], 99.50th=[ 128], 99.90th=[ 128], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 8192, max= 8192, per=22.23%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=25.00%, 100=41.67%, 250=8.33% cpu : usr=0.14%, sys=12.13%, ctx=94, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.7MiB/s (26.0MB/s), 2632KiB/s-11.6MiB/s (2696kB/s-12.1MB/s), io=20.0MiB (20.0MB), run=606-778msec WRITE: bw=35.0MiB/s (37.7MB/s), 8440KiB/s-12.9MiB/s (8642kB/s-13.5MB/s), io=28.0MiB (29.4MB), run=606-778msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21554: Thu May 1 03:36:56 2025 read: IOPS=4, BW=17.8MiB/s (18.6MB/s)(12.0MiB/676msec) clat (msec): min=122, max=335, avg=213.62, stdev=109.85 lat (msec): min=122, max=335, avg=213.63, stdev=109.85 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 123], 10.00th=[ 123], 20.00th=[ 123], | 30.00th=[ 123], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 184], | 70.00th=[ 334], 80.00th=[ 334], 90.00th=[ 334], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.15%, sys=13.78%, ctx=60, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21555: Thu May 1 03:36:56 2025 read: IOPS=7, BW=31.5MiB/s (33.0MB/s)(4096KiB/127msec) clat (nsec): min=42423k, max=42423k, avg=42422785.00, stdev= 0.00 lat (nsec): min=42426k, max=42426k, avg=42425509.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[42206], 5.00th=[42206], 10.00th=[42206], 20.00th=[42206], | 30.00th=[42206], 40.00th=[42206], 50.00th=[42206], 60.00th=[42206], | 70.00th=[42206], 80.00th=[42206], 90.00th=[42206], 95.00th=[42206], | 99.00th=[42206], 99.50th=[42206], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] write: IOPS=15, BW=62.0MiB/s (66.1MB/s)(8192KiB/127msec) clat (usec): min=37322, max=40284, avg=38803.87, stdev=2094.32 lat (usec): min=37521, max=40547, avg=39034.48, stdev=2139.32 clat percentiles (usec): | 1.00th=[37487], 5.00th=[37487], 10.00th=[37487], 20.00th=[37487], | 30.00th=[37487], 40.00th=[37487], 50.00th=[37487], 60.00th=[40109], | 70.00th=[40109], 80.00th=[40109], 90.00th=[40109], 95.00th=[40109], | 99.00th=[40109], 99.50th=[40109], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] lat (msec) : 50=100.00% cpu : usr=0.00%, sys=46.83%, ctx=51, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21557: Thu May 1 03:36:56 2025 write: IOPS=7, BW=28.2MiB/s (29.5MB/s)(12.0MiB/426msec) clat (msec): min=58, max=258, avg=137.26, stdev=106.75 lat (msec): min=59, max=259, avg=137.63, stdev=106.67 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 59], | 30.00th=[ 59], 40.00th=[ 94], 50.00th=[ 94], 60.00th=[ 94], | 70.00th=[ 259], 80.00th=[ 259], 90.00th=[ 259], 95.00th=[ 259], | 99.00th=[ 259], 99.50th=[ 259], 99.90th=[ 259], 99.95th=[ 259], | 99.99th=[ 259] lat (msec) : 100=66.67%, 500=33.33% cpu : usr=0.00%, sys=17.18%, ctx=224, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21559: Thu May 1 03:36:56 2025 write: IOPS=6, BW=25.9MiB/s (27.2MB/s)(12.0MiB/463msec) clat (msec): min=117, max=189, avg=141.88, stdev=41.50 lat (msec): min=118, max=190, avg=142.65, stdev=41.07 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 117], 10.00th=[ 117], 20.00th=[ 117], | 30.00th=[ 117], 40.00th=[ 120], 50.00th=[ 120], 60.00th=[ 120], | 70.00th=[ 190], 80.00th=[ 190], 90.00th=[ 190], 95.00th=[ 190], | 99.00th=[ 190], 99.50th=[ 190], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] lat (msec) : 250=100.00% cpu : usr=0.43%, sys=18.18%, ctx=55, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=23.7MiB/s (24.8MB/s), 17.8MiB/s-31.5MiB/s (18.6MB/s-33.0MB/s), io=16.0MiB (16.8MB), run=127-676msec WRITE: bw=69.1MiB/s (72.5MB/s), 25.9MiB/s-62.0MiB/s (27.2MB/s-66.1MB/s), io=32.0MiB (33.6MB), run=127-463msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21556: Thu May 1 03:36:57 2025 read: IOPS=6, BW=25.1MiB/s (26.3MB/s)(12.0MiB/478msec) clat (msec): min=133, max=176, avg=151.65, stdev=21.93 lat (msec): min=133, max=176, avg=151.65, stdev=21.93 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 134], 10.00th=[ 134], 20.00th=[ 134], | 30.00th=[ 134], 40.00th=[ 146], 50.00th=[ 146], 60.00th=[ 146], | 70.00th=[ 176], 80.00th=[ 176], 90.00th=[ 176], 95.00th=[ 176], | 99.00th=[ 176], 99.50th=[ 176], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=41.51%, ctx=1570, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21558: Thu May 1 03:36:57 2025 read: IOPS=1, BW=5737KiB/s (5874kB/s)(4096KiB/714msec) clat (nsec): min=92244k, max=92244k, avg=92244247.00, stdev= 0.00 lat (nsec): min=92245k, max=92245k, avg=92245272.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[91751], 5.00th=[91751], 10.00th=[91751], 20.00th=[91751], | 30.00th=[91751], 40.00th=[91751], 50.00th=[91751], 60.00th=[91751], | 70.00th=[91751], 80.00th=[91751], 90.00th=[91751], 95.00th=[91751], | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 8192, max= 8192, per=35.70%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=11.2MiB/s (11.7MB/s)(8192KiB/714msec) clat (msec): min=154, max=406, avg=280.55, stdev=178.55 lat (msec): min=154, max=407, avg=280.79, stdev=178.54 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 405], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=26.51%, ctx=605, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21560: Thu May 1 03:36:57 2025 write: IOPS=3, BW=15.1MiB/s (15.8MB/s)(12.0MiB/795msec) clat (msec): min=188, max=308, avg=245.70, stdev=60.34 lat (msec): min=188, max=308, avg=245.97, stdev=60.35 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 188], 10.00th=[ 188], 20.00th=[ 188], | 30.00th=[ 188], 40.00th=[ 241], 50.00th=[ 241], 60.00th=[ 241], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 8192, max= 8192, per=19.88%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=40.30%, ctx=60, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21561: Thu May 1 03:36:57 2025 write: IOPS=4, BW=19.3MiB/s (20.2MB/s)(12.0MiB/623msec) clat (msec): min=124, max=238, avg=186.73, stdev=57.86 lat (msec): min=124, max=238, avg=186.98, stdev=57.86 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 125], 10.00th=[ 125], 20.00th=[ 125], | 30.00th=[ 125], 40.00th=[ 199], 50.00th=[ 199], 60.00th=[ 199], | 70.00th=[ 239], 80.00th=[ 239], 90.00th=[ 239], 95.00th=[ 239], | 99.00th=[ 239], 99.50th=[ 239], 99.90th=[ 239], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min=16384, max=16384, per=39.75%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=50.32%, ctx=68, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.4MiB/s (23.5MB/s), 5737KiB/s-25.1MiB/s (5874kB/s-26.3MB/s), io=16.0MiB (16.8MB), run=478-714msec WRITE: bw=40.3MiB/s (42.2MB/s), 11.2MiB/s-19.3MiB/s (11.7MB/s-20.2MB/s), io=32.0MiB (33.6MB), run=623-795msec