== sanity test 398b: DIO and buffer IO race ============== 03:32:27 (1746084747) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.35584 s, 37.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23706: Thu May 1 03:32:44 2025 read: IOPS=106, BW=427KiB/s (437kB/s)(5992KiB/14033msec) clat (usec): min=50, max=190378, avg=5092.17, stdev=5635.12 lat (usec): min=50, max=190379, avg=5093.73, stdev=5634.89 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 118], 10.00th=[ 2376], 20.00th=[ 2933], | 30.00th=[ 3359], 40.00th=[ 3851], 50.00th=[ 4555], 60.00th=[ 5276], | 70.00th=[ 5997], 80.00th=[ 6915], 90.00th=[ 8225], 95.00th=[ 9765], | 99.00th=[ 16188], 99.50th=[ 18744], 99.90th=[ 43779], 99.95th=[189793], | 99.99th=[189793] bw ( KiB/s): min= 248, max= 552, per=25.34%, avg=426.43, stdev=69.54, samples=28 iops : min= 62, max= 138, avg=106.54, stdev=17.39, samples=28 write: IOPS=112, BW=449KiB/s (459kB/s)(6296KiB/14033msec) clat (usec): min=444, max=25609, avg=4029.62, stdev=3617.42 lat (usec): min=445, max=25610, avg=4030.65, stdev=3617.33 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 611], 10.00th=[ 685], 20.00th=[ 766], | 30.00th=[ 922], 40.00th=[ 2057], 50.00th=[ 3261], 60.00th=[ 4113], | 70.00th=[ 5080], 80.00th=[ 6915], 90.00th=[ 9503], 95.00th=[11207], | 99.00th=[14746], 99.50th=[17433], 99.90th=[22152], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 296, max= 616, per=25.39%, avg=447.29, stdev=75.15, samples=28 iops : min= 74, max= 154, avg=111.75, stdev=18.79, samples=28 lat (usec) : 100=1.63%, 250=1.30%, 500=0.36%, 750=9.21%, 1000=7.10% lat (msec) : 2=4.17%, 4=27.28%, 10=43.20%, 20=5.50%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=22.41%, ctx=5596, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23707: Thu May 1 03:32:44 2025 read: IOPS=106, BW=426KiB/s (437kB/s)(5940KiB/13931msec) clat (usec): min=58, max=23624, avg=5043.50, stdev=2881.06 lat (usec): min=58, max=23625, avg=5044.37, stdev=2880.99 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 124], 10.00th=[ 2442], 20.00th=[ 2900], | 30.00th=[ 3359], 40.00th=[ 3818], 50.00th=[ 4424], 60.00th=[ 5276], | 70.00th=[ 6194], 80.00th=[ 7177], 90.00th=[ 8455], 95.00th=[ 9896], | 99.00th=[14615], 99.50th=[16450], 99.90th=[23200], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 272, max= 528, per=25.15%, avg=423.33, stdev=56.58, samples=27 iops : min= 68, max= 132, avg=105.78, stdev=14.15, samples=27 write: IOPS=113, BW=456KiB/s (467kB/s)(6348KiB/13931msec) clat (usec): min=435, max=190546, avg=4018.38, stdev=5908.14 lat (usec): min=436, max=190547, avg=4019.43, stdev=5908.20 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 594], 10.00th=[ 660], 20.00th=[ 742], | 30.00th=[ 873], 40.00th=[ 1631], 50.00th=[ 3228], 60.00th=[ 3982], | 70.00th=[ 5014], 80.00th=[ 6652], 90.00th=[ 8979], 95.00th=[ 10683], | 99.00th=[ 16057], 99.50th=[ 17433], 99.90th=[ 25297], 99.95th=[189793], | 99.99th=[189793] bw ( KiB/s): min= 352, max= 560, per=25.32%, avg=446.11, stdev=66.61, samples=27 iops : min= 88, max= 140, avg=111.48, stdev=16.63, samples=27 lat (usec) : 100=1.37%, 250=1.43%, 500=0.68%, 750=10.12%, 1000=6.87% lat (msec) : 2=4.17%, 4=26.89%, 10=42.48%, 20=5.73%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=22.49%, ctx=5494, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23708: Thu May 1 03:32:44 2025 read: IOPS=105, BW=423KiB/s (433kB/s)(6028KiB/14262msec) clat (usec): min=50, max=36526, avg=5192.96, stdev=3203.86 lat (usec): min=50, max=36528, avg=5196.25, stdev=3205.06 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 120], 10.00th=[ 2442], 20.00th=[ 2966], | 30.00th=[ 3392], 40.00th=[ 3916], 50.00th=[ 4555], 60.00th=[ 5342], | 70.00th=[ 6194], 80.00th=[ 7308], 90.00th=[ 8848], 95.00th=[10290], | 99.00th=[15533], 99.50th=[18744], 99.90th=[34341], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 240, max= 552, per=24.86%, avg=418.43, stdev=63.66, samples=28 iops : min= 60, max= 138, avg=104.54, stdev=15.92, samples=28 write: IOPS=109, BW=439KiB/s (449kB/s)(6260KiB/14262msec) clat (usec): min=426, max=187518, avg=4060.70, stdev=5827.90 lat (usec): min=427, max=187521, avg=4062.59, stdev=5828.26 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 627], 10.00th=[ 693], 20.00th=[ 783], | 30.00th=[ 947], 40.00th=[ 1909], 50.00th=[ 3228], 60.00th=[ 4015], | 70.00th=[ 5145], 80.00th=[ 6587], 90.00th=[ 8848], 95.00th=[ 10945], | 99.00th=[ 15139], 99.50th=[ 16450], 99.90th=[ 22938], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 304, max= 600, per=24.56%, avg=432.71, stdev=72.67, samples=28 iops : min= 76, max= 150, avg=108.11, stdev=18.20, samples=28 lat (usec) : 100=1.37%, 250=1.43%, 500=0.33%, 750=8.20%, 1000=7.75% lat (msec) : 2=4.46%, 4=27.51%, 10=42.48%, 20=6.18%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=22.13%, ctx=5645, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23709: Thu May 1 03:32:44 2025 read: IOPS=106, BW=427KiB/s (437kB/s)(6052KiB/14173msec) clat (usec): min=55, max=23183, avg=5139.19, stdev=2980.53 lat (usec): min=56, max=23184, avg=5141.14, stdev=2979.85 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 125], 10.00th=[ 2442], 20.00th=[ 2966], | 30.00th=[ 3326], 40.00th=[ 3851], 50.00th=[ 4490], 60.00th=[ 5342], | 70.00th=[ 6194], 80.00th=[ 7308], 90.00th=[ 8717], 95.00th=[10290], | 99.00th=[15664], 99.50th=[17433], 99.90th=[21103], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 240, max= 560, per=25.18%, avg=423.82, stdev=61.60, samples=28 iops : min= 60, max= 140, avg=105.82, stdev=15.39, samples=28 write: IOPS=109, BW=440KiB/s (451kB/s)(6236KiB/14173msec) clat (usec): min=443, max=184601, avg=4050.93, stdev=5885.48 lat (usec): min=443, max=184602, avg=4052.54, stdev=5885.38 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 603], 10.00th=[ 668], 20.00th=[ 758], | 30.00th=[ 873], 40.00th=[ 1549], 50.00th=[ 3097], 60.00th=[ 3916], | 70.00th=[ 4883], 80.00th=[ 6783], 90.00th=[ 9372], 95.00th=[ 11338], | 99.00th=[ 14877], 99.50th=[ 17433], 99.90th=[ 30016], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 296, max= 608, per=24.78%, avg=436.64, stdev=75.96, samples=28 iops : min= 74, max= 152, avg=109.04, stdev=19.02, samples=28 lat (usec) : 100=1.60%, 250=1.01%, 500=0.42%, 750=9.24%, 1000=7.71% lat (msec) : 2=4.52%, 4=27.80%, 10=41.08%, 20=6.32%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=21.97%, ctx=5549, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1684KiB/s (1724kB/s), 423KiB/s-427KiB/s (433kB/s-437kB/s), io=23.4MiB (24.6MB), run=13931-14262msec WRITE: bw=1763KiB/s (1805kB/s), 439KiB/s-456KiB/s (449kB/s-467kB/s), io=24.6MiB (25.7MB), run=13931-14262msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23702: Thu May 1 03:32:49 2025 read: IOPS=80, BW=320KiB/s (328kB/s)(5992KiB/18710msec) clat (usec): min=1518, max=194765, avg=4874.26, stdev=6259.12 lat (usec): min=1519, max=194766, avg=4875.14, stdev=6259.10 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1909], 10.00th=[ 2040], 20.00th=[ 2245], | 30.00th=[ 2442], 40.00th=[ 2737], 50.00th=[ 3163], 60.00th=[ 3687], | 70.00th=[ 4555], 80.00th=[ 6456], 90.00th=[ 10421], 95.00th=[ 13566], | 99.00th=[ 19006], 99.50th=[ 21365], 99.90th=[ 28181], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 152, max= 712, per=24.58%, avg=315.35, stdev=164.37, samples=37 iops : min= 38, max= 178, avg=78.76, stdev=41.13, samples=37 write: IOPS=84, BW=337KiB/s (345kB/s)(6296KiB/18710msec) clat (usec): min=2278, max=48919, avg=7205.23, stdev=4640.54 lat (usec): min=2279, max=48920, avg=7206.30, stdev=4640.52 clat percentiles (usec): | 1.00th=[ 2507], 5.00th=[ 2933], 10.00th=[ 3261], 20.00th=[ 3687], | 30.00th=[ 4080], 40.00th=[ 4686], 50.00th=[ 5473], 60.00th=[ 6783], | 70.00th=[ 8455], 80.00th=[10421], 90.00th=[13304], 95.00th=[15926], | 99.00th=[23200], 99.50th=[26608], 99.90th=[38011], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 160, max= 736, per=24.66%, avg=331.14, stdev=169.38, samples=37 iops : min= 40, max= 184, avg=82.70, stdev=42.38, samples=37 lat (msec) : 2=4.00%, 4=41.96%, 10=37.27%, 20=15.27%, 50=1.46% lat (msec) : 250=0.03% cpu : usr=0.27%, sys=10.71%, ctx=3597, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23703: Thu May 1 03:32:49 2025 read: IOPS=79, BW=319KiB/s (327kB/s)(5940KiB/18601msec) clat (usec): min=1541, max=188964, avg=4816.80, stdev=6091.37 lat (usec): min=1542, max=188965, avg=4817.60, stdev=6091.38 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1909], 10.00th=[ 2024], 20.00th=[ 2180], | 30.00th=[ 2474], 40.00th=[ 2769], 50.00th=[ 3195], 60.00th=[ 3687], | 70.00th=[ 4621], 80.00th=[ 6456], 90.00th=[ 10421], 95.00th=[ 13173], | 99.00th=[ 18482], 99.50th=[ 21890], 99.90th=[ 33817], 99.95th=[189793], | 99.99th=[189793] bw ( KiB/s): min= 128, max= 712, per=24.76%, avg=317.73, stdev=168.90, samples=37 iops : min= 32, max= 178, avg=79.35, stdev=42.28, samples=37 write: IOPS=85, BW=341KiB/s (349kB/s)(6348KiB/18601msec) clat (usec): min=2281, max=49073, avg=7168.81, stdev=4635.41 lat (usec): min=2282, max=49074, avg=7169.92, stdev=4635.42 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2966], 10.00th=[ 3261], 20.00th=[ 3621], | 30.00th=[ 4047], 40.00th=[ 4752], 50.00th=[ 5407], 60.00th=[ 6521], | 70.00th=[ 8225], 80.00th=[10421], 90.00th=[13566], 95.00th=[16319], | 99.00th=[23200], 99.50th=[25297], 99.90th=[37487], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 176, max= 752, per=25.17%, avg=338.05, stdev=163.00, samples=37 iops : min= 44, max= 188, avg=84.43, stdev=40.78, samples=37 lat (msec) : 2=4.23%, 4=41.60%, 10=37.60%, 20=15.20%, 50=1.33% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=10.62%, ctx=3569, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23704: Thu May 1 03:32:49 2025 read: IOPS=81, BW=326KiB/s (334kB/s)(6028KiB/18473msec) clat (usec): min=1601, max=28039, avg=4685.01, stdev=3585.32 lat (usec): min=1601, max=28039, avg=4686.40, stdev=3585.06 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1926], 10.00th=[ 2024], 20.00th=[ 2278], | 30.00th=[ 2507], 40.00th=[ 2835], 50.00th=[ 3261], 60.00th=[ 3818], | 70.00th=[ 4686], 80.00th=[ 6521], 90.00th=[10028], 95.00th=[12518], | 99.00th=[17695], 99.50th=[19792], 99.90th=[22152], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 120, max= 736, per=24.71%, avg=317.00, stdev=151.98, samples=36 iops : min= 30, max= 184, avg=79.17, stdev=38.04, samples=36 write: IOPS=84, BW=339KiB/s (347kB/s)(6260KiB/18473msec) clat (msec): min=2, max=196, avg= 7.25, stdev= 6.56 lat (msec): min=2, max=196, avg= 7.25, stdev= 6.56 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 14], 95.00th=[ 16], | 99.00th=[ 25], 99.50th=[ 31], 99.90th=[ 41], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 168, max= 704, per=24.43%, avg=328.11, stdev=155.72, samples=36 iops : min= 42, max= 176, avg=81.94, stdev=38.96, samples=36 lat (msec) : 2=4.10%, 4=40.40%, 10=39.39%, 20=15.10%, 50=0.98% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=10.99%, ctx=3541, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23705: Thu May 1 03:32:49 2025 read: IOPS=81, BW=326KiB/s (334kB/s)(6052KiB/18567msec) clat (usec): min=1598, max=27981, avg=4798.57, stdev=3882.89 lat (usec): min=1598, max=27982, avg=4799.39, stdev=3882.91 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1876], 10.00th=[ 2040], 20.00th=[ 2245], | 30.00th=[ 2507], 40.00th=[ 2835], 50.00th=[ 3228], 60.00th=[ 3752], | 70.00th=[ 4752], 80.00th=[ 6718], 90.00th=[10159], 95.00th=[13566], | 99.00th=[20317], 99.50th=[21890], 99.90th=[23200], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 151, max= 688, per=25.23%, avg=323.65, stdev=161.91, samples=37 iops : min= 37, max= 172, avg=80.76, stdev=40.56, samples=37 write: IOPS=83, BW=336KiB/s (344kB/s)(6236KiB/18567msec) clat (msec): min=2, max=188, avg= 7.20, stdev= 6.45 lat (msec): min=2, max=188, avg= 7.20, stdev= 6.45 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 14], 95.00th=[ 16], | 99.00th=[ 23], 99.50th=[ 29], 99.90th=[ 41], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 183, max= 752, per=24.87%, avg=334.03, stdev=166.59, samples=37 iops : min= 45, max= 188, avg=83.35, stdev=41.71, samples=37 lat (msec) : 2=3.94%, 4=41.44%, 10=38.70%, 20=14.42%, 50=1.46% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=10.77%, ctx=3527, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1283KiB/s (1314kB/s), 319KiB/s-326KiB/s (327kB/s-334kB/s), io=23.4MiB (24.6MB), run=18473-18710msec WRITE: bw=1344KiB/s (1376kB/s), 336KiB/s-341KiB/s (344kB/s-349kB/s), io=24.6MiB (25.7MB), run=18473-18710msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23737: Thu May 1 03:32:53 2025 read: IOPS=115, BW=1851KiB/s (1895kB/s)(5920KiB/3199msec) clat (usec): min=51, max=18783, avg=4606.12, stdev=2820.23 lat (usec): min=51, max=18783, avg=4606.78, stdev=2820.24 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 97], 10.00th=[ 137], 20.00th=[ 2737], | 30.00th=[ 3261], 40.00th=[ 3818], 50.00th=[ 4424], 60.00th=[ 4883], | 70.00th=[ 5473], 80.00th=[ 6259], 90.00th=[ 7570], 95.00th=[ 9241], | 99.00th=[14615], 99.50th=[17695], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 1568, max= 2267, per=26.28%, avg=1849.83, stdev=285.05, samples=6 iops : min= 98, max= 141, avg=115.50, stdev=17.62, samples=6 write: IOPS=124, BW=1991KiB/s (2038kB/s)(6368KiB/3199msec) clat (usec): min=649, max=22520, avg=3716.11, stdev=3235.31 lat (usec): min=650, max=22521, avg=3717.13, stdev=3235.29 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 775], 10.00th=[ 881], 20.00th=[ 996], | 30.00th=[ 1172], 40.00th=[ 1795], 50.00th=[ 2737], 60.00th=[ 3654], | 70.00th=[ 4555], 80.00th=[ 5997], 90.00th=[ 8094], 95.00th=[10290], | 99.00th=[13566], 99.50th=[18744], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 1728, max= 2267, per=26.77%, avg=1983.17, stdev=219.48, samples=6 iops : min= 108, max= 141, avg=123.83, stdev=13.54, samples=6 lat (usec) : 100=2.99%, 250=2.21%, 750=1.56%, 1000=8.85% lat (msec) : 2=11.07%, 4=27.86%, 10=40.62%, 20=4.69%, 50=0.13% cpu : usr=0.34%, sys=25.61%, ctx=1570, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23739: Thu May 1 03:32:53 2025 read: IOPS=106, BW=1712KiB/s (1753kB/s)(5744KiB/3356msec) clat (usec): min=60, max=18030, avg=4762.86, stdev=3005.45 lat (usec): min=61, max=18031, avg=4767.28, stdev=3006.63 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 87], 10.00th=[ 111], 20.00th=[ 2769], | 30.00th=[ 3294], 40.00th=[ 4080], 50.00th=[ 4621], 60.00th=[ 5211], | 70.00th=[ 5800], 80.00th=[ 6718], 90.00th=[ 8291], 95.00th=[10028], | 99.00th=[14746], 99.50th=[16319], 99.90th=[17957], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 1344, max= 1920, per=23.93%, avg=1684.67, stdev=209.48, samples=6 iops : min= 84, max= 120, avg=105.17, stdev=13.00, samples=6 write: IOPS=121, BW=1950KiB/s (1997kB/s)(6544KiB/3356msec) clat (usec): min=614, max=18982, avg=3959.25, stdev=3139.68 lat (usec): min=615, max=18983, avg=3961.22, stdev=3139.59 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 807], 10.00th=[ 898], 20.00th=[ 1057], | 30.00th=[ 1319], 40.00th=[ 2507], 50.00th=[ 3130], 60.00th=[ 3949], | 70.00th=[ 5407], 80.00th=[ 6456], 90.00th=[ 8586], 95.00th=[ 9896], | 99.00th=[13042], 99.50th=[13566], 99.90th=[19006], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 1568, max= 2336, per=25.62%, avg=1898.00, stdev=311.79, samples=6 iops : min= 98, max= 146, avg=118.50, stdev=19.57, samples=6 lat (usec) : 100=3.91%, 250=1.95%, 500=0.13%, 750=1.56%, 1000=7.42% lat (msec) : 2=10.55%, 4=25.26%, 10=44.27%, 20=4.95% cpu : usr=0.18%, sys=25.13%, ctx=1583, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23741: Thu May 1 03:32:53 2025 read: IOPS=114, BW=1839KiB/s (1883kB/s)(6256KiB/3402msec) clat (usec): min=62, max=19317, avg=4873.86, stdev=2854.75 lat (usec): min=63, max=19317, avg=4876.32, stdev=2854.47 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 103], 10.00th=[ 2147], 20.00th=[ 2900], | 30.00th=[ 3458], 40.00th=[ 3982], 50.00th=[ 4555], 60.00th=[ 5211], | 70.00th=[ 5866], 80.00th=[ 6915], 90.00th=[ 8225], 95.00th=[ 9503], | 99.00th=[15795], 99.50th=[16909], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 1472, max= 2112, per=25.30%, avg=1781.33, stdev=222.32, samples=6 iops : min= 92, max= 132, avg=111.33, stdev=13.89, samples=6 write: IOPS=110, BW=1773KiB/s (1816kB/s)(6032KiB/3402msec) clat (usec): min=671, max=14892, avg=3887.02, stdev=3014.13 lat (usec): min=671, max=14893, avg=3887.96, stdev=3014.12 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 816], 10.00th=[ 898], 20.00th=[ 1057], | 30.00th=[ 1287], 40.00th=[ 2311], 50.00th=[ 3097], 60.00th=[ 4113], | 70.00th=[ 5276], 80.00th=[ 6587], 90.00th=[ 8225], 95.00th=[ 9634], | 99.00th=[13435], 99.50th=[14877], 99.90th=[14877], 99.95th=[14877], | 99.99th=[14877] bw ( KiB/s): min= 1600, max= 1856, per=23.76%, avg=1760.00, stdev=94.93, samples=6 iops : min= 100, max= 116, avg=110.00, stdev= 5.93, samples=6 lat (usec) : 100=2.21%, 250=2.21%, 500=0.13%, 750=1.04%, 1000=7.03% lat (msec) : 2=10.81%, 4=26.17%, 10=46.74%, 20=3.65% cpu : usr=0.09%, sys=24.79%, ctx=1665, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23743: Thu May 1 03:32:53 2025 read: IOPS=114, BW=1827KiB/s (1871kB/s)(6032KiB/3302msec) clat (usec): min=59, max=21903, avg=4701.58, stdev=2938.53 lat (usec): min=59, max=21904, avg=4702.96, stdev=2938.90 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 93], 10.00th=[ 127], 20.00th=[ 2769], | 30.00th=[ 3294], 40.00th=[ 3621], 50.00th=[ 4228], 60.00th=[ 4883], | 70.00th=[ 5932], 80.00th=[ 6783], 90.00th=[ 8586], 95.00th=[ 9765], | 99.00th=[14091], 99.50th=[16909], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 1628, max= 2048, per=25.75%, avg=1812.67, stdev=167.36, samples=6 iops : min= 101, max= 128, avg=113.17, stdev=10.63, samples=6 write: IOPS=118, BW=1895KiB/s (1940kB/s)(6256KiB/3302msec) clat (usec): min=555, max=15196, avg=3820.07, stdev=3100.53 lat (usec): min=555, max=15197, avg=3821.04, stdev=3100.54 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 799], 10.00th=[ 906], 20.00th=[ 1020], | 30.00th=[ 1221], 40.00th=[ 1778], 50.00th=[ 3032], 60.00th=[ 3916], | 70.00th=[ 5145], 80.00th=[ 6521], 90.00th=[ 8094], 95.00th=[10028], | 99.00th=[13698], 99.50th=[14877], 99.90th=[15139], 99.95th=[15139], | 99.99th=[15139] bw ( KiB/s): min= 1760, max= 2043, per=25.77%, avg=1908.50, stdev=99.15, samples=6 iops : min= 110, max= 127, avg=119.17, stdev= 6.01, samples=6 lat (usec) : 100=3.52%, 250=1.69%, 500=0.13%, 750=1.43%, 1000=7.29% lat (msec) : 2=12.76%, 4=27.08%, 10=41.28%, 20=4.69%, 50=0.13% cpu : usr=0.21%, sys=24.75%, ctx=1661, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7041KiB/s (7210kB/s), 1712KiB/s-1851KiB/s (1753kB/s-1895kB/s), io=23.4MiB (24.5MB), run=3199-3402msec WRITE: bw=7407KiB/s (7585kB/s), 1773KiB/s-1991KiB/s (1816kB/s-2038kB/s), io=24.6MiB (25.8MB), run=3199-3402msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23738: Thu May 1 03:32:54 2025 read: IOPS=83, BW=1329KiB/s (1361kB/s)(5920KiB/4454msec) clat (usec): min=1766, max=23096, avg=4666.16, stdev=3485.86 lat (usec): min=1767, max=23096, avg=4668.78, stdev=3485.19 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2147], 10.00th=[ 2311], 20.00th=[ 2474], | 30.00th=[ 2638], 40.00th=[ 2835], 50.00th=[ 3228], 60.00th=[ 3687], | 70.00th=[ 4424], 80.00th=[ 6456], 90.00th=[ 9503], 95.00th=[12387], | 99.00th=[18744], 99.50th=[22152], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 864, max= 2240, per=22.47%, avg=1207.62, stdev=450.61, samples=8 iops : min= 54, max= 140, avg=75.38, stdev=28.19, samples=8 write: IOPS=89, BW=1430KiB/s (1464kB/s)(6368KiB/4454msec) clat (usec): min=2597, max=25260, avg=6743.31, stdev=3880.96 lat (usec): min=2598, max=25261, avg=6744.83, stdev=3880.96 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3228], 10.00th=[ 3556], 20.00th=[ 3785], | 30.00th=[ 4178], 40.00th=[ 4555], 50.00th=[ 5014], 60.00th=[ 5997], | 70.00th=[ 7635], 80.00th=[ 9110], 90.00th=[12780], 95.00th=[15008], | 99.00th=[19268], 99.50th=[21365], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 960, max= 2272, per=23.41%, avg=1323.50, stdev=432.63, samples=8 iops : min= 60, max= 142, avg=82.62, stdev=26.97, samples=8 lat (msec) : 2=0.65%, 4=44.01%, 10=41.80%, 20=12.89%, 50=0.65% cpu : usr=0.36%, sys=11.63%, ctx=916, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23740: Thu May 1 03:32:54 2025 read: IOPS=80, BW=1289KiB/s (1320kB/s)(5744KiB/4457msec) clat (usec): min=1837, max=27249, avg=4852.36, stdev=3827.19 lat (usec): min=1838, max=27250, avg=4853.09, stdev=3827.17 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2114], 10.00th=[ 2245], 20.00th=[ 2442], | 30.00th=[ 2638], 40.00th=[ 2802], 50.00th=[ 3163], 60.00th=[ 3654], | 70.00th=[ 4490], 80.00th=[ 6849], 90.00th=[10814], 95.00th=[13042], | 99.00th=[20579], 99.50th=[24511], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 800, max= 2176, per=22.33%, avg=1200.00, stdev=436.09, samples=8 iops : min= 50, max= 136, avg=75.00, stdev=27.26, samples=8 write: IOPS=91, BW=1468KiB/s (1503kB/s)(6544KiB/4457msec) clat (usec): min=2613, max=26868, avg=6609.88, stdev=3775.90 lat (usec): min=2614, max=26869, avg=6611.43, stdev=3775.84 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3359], 10.00th=[ 3589], 20.00th=[ 3851], | 30.00th=[ 4080], 40.00th=[ 4424], 50.00th=[ 4883], 60.00th=[ 5669], | 70.00th=[ 7504], 80.00th=[ 9372], 90.00th=[12256], 95.00th=[13698], | 99.00th=[18744], 99.50th=[20841], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 928, max= 2368, per=23.98%, avg=1356.00, stdev=448.63, samples=8 iops : min= 58, max= 148, avg=84.75, stdev=28.04, samples=8 lat (msec) : 2=0.39%, 4=44.27%, 10=39.84%, 20=14.58%, 50=0.91% cpu : usr=0.11%, sys=11.71%, ctx=965, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23742: Thu May 1 03:32:54 2025 read: IOPS=88, BW=1419KiB/s (1453kB/s)(6256KiB/4409msec) clat (usec): min=1853, max=21653, avg=4538.08, stdev=3427.47 lat (usec): min=1854, max=21654, avg=4538.84, stdev=3427.46 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2147], 10.00th=[ 2278], 20.00th=[ 2409], | 30.00th=[ 2606], 40.00th=[ 2802], 50.00th=[ 3064], 60.00th=[ 3425], | 70.00th=[ 4178], 80.00th=[ 5997], 90.00th=[ 9896], 95.00th=[11600], | 99.00th=[18482], 99.50th=[21365], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 894, max= 2176, per=24.11%, avg=1295.75, stdev=456.66, samples=8 iops : min= 55, max= 136, avg=80.88, stdev=28.65, samples=8 write: IOPS=85, BW=1368KiB/s (1401kB/s)(6032KiB/4409msec) clat (usec): min=2613, max=26717, avg=6841.41, stdev=4000.44 lat (usec): min=2614, max=26719, avg=6842.73, stdev=4000.40 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3228], 10.00th=[ 3556], 20.00th=[ 3916], | 30.00th=[ 4228], 40.00th=[ 4555], 50.00th=[ 5145], 60.00th=[ 6194], | 70.00th=[ 7767], 80.00th=[ 9634], 90.00th=[12387], 95.00th=[14746], | 99.00th=[20841], 99.50th=[25560], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 896, max= 2400, per=22.77%, avg=1287.62, stdev=465.26, samples=8 iops : min= 56, max= 150, avg=80.38, stdev=29.09, samples=8 lat (msec) : 2=0.78%, 4=45.31%, 10=39.71%, 20=13.28%, 50=0.91% cpu : usr=0.32%, sys=11.73%, ctx=886, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23744: Thu May 1 03:32:54 2025 read: IOPS=85, BW=1372KiB/s (1404kB/s)(6032KiB/4398msec) clat (usec): min=1894, max=27381, avg=4718.86, stdev=3782.09 lat (usec): min=1894, max=27382, avg=4719.57, stdev=3782.09 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2147], 10.00th=[ 2311], 20.00th=[ 2409], | 30.00th=[ 2638], 40.00th=[ 2835], 50.00th=[ 3163], 60.00th=[ 3556], | 70.00th=[ 4228], 80.00th=[ 5800], 90.00th=[10683], 95.00th=[13304], | 99.00th=[20317], 99.50th=[20841], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 896, max= 2112, per=23.74%, avg=1275.62, stdev=379.33, samples=8 iops : min= 56, max= 132, avg=79.63, stdev=23.75, samples=8 write: IOPS=88, BW=1422KiB/s (1457kB/s)(6256KiB/4398msec) clat (usec): min=2804, max=27085, avg=6555.72, stdev=3638.15 lat (usec): min=2806, max=27088, avg=6557.19, stdev=3638.06 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 3425], 10.00th=[ 3654], 20.00th=[ 3949], | 30.00th=[ 4228], 40.00th=[ 4686], 50.00th=[ 5145], 60.00th=[ 5997], | 70.00th=[ 7111], 80.00th=[ 8586], 90.00th=[11338], 95.00th=[14222], | 99.00th=[21103], 99.50th=[23200], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 896, max= 2304, per=23.48%, avg=1327.62, stdev=439.24, samples=8 iops : min= 56, max= 144, avg=82.88, stdev=27.49, samples=8 lat (msec) : 2=0.39%, 4=44.40%, 10=42.58%, 20=11.59%, 50=1.04% cpu : usr=0.25%, sys=11.69%, ctx=924, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5374KiB/s (5503kB/s), 1289KiB/s-1419KiB/s (1320kB/s-1453kB/s), io=23.4MiB (24.5MB), run=4398-4457msec WRITE: bw=5654KiB/s (5790kB/s), 1368KiB/s-1468KiB/s (1401kB/s-1503kB/s), io=24.6MiB (25.8MB), run=4398-4457msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23753: Thu May 1 03:32:56 2025 read: IOPS=11, BW=11.1MiB/s (11.7MB/s)(7168KiB/629msec) clat (usec): min=10733, max=98940, avg=29362.79, stdev=32882.72 lat (usec): min=10735, max=98941, avg=29363.98, stdev=32882.63 clat percentiles (usec): | 1.00th=[10683], 5.00th=[10683], 10.00th=[10683], 20.00th=[10945], | 30.00th=[11994], 40.00th=[11994], 50.00th=[11994], 60.00th=[17171], | 70.00th=[17171], 80.00th=[43779], 90.00th=[99091], 95.00th=[99091], | 99.00th=[99091], 99.50th=[99091], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min=10219, max=10219, per=37.82%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=7, BW=8140KiB/s (8335kB/s)(5120KiB/629msec) clat (msec): min=26, max=105, avg=78.54, stdev=30.97 lat (msec): min=26, max=106, avg=78.60, stdev=30.97 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 79], 40.00th=[ 79], 50.00th=[ 90], 60.00th=[ 90], | 70.00th=[ 93], 80.00th=[ 93], 90.00th=[ 107], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 8175, max= 8175, per=21.61%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=41.67%, 50=16.67%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=15.61%, ctx=68, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23754: Thu May 1 03:32:56 2025 read: IOPS=2, BW=2702KiB/s (2767kB/s)(2048KiB/758msec) clat (usec): min=12042, max=78796, avg=45419.36, stdev=47202.27 lat (usec): min=12044, max=78797, avg=45421.25, stdev=47201.68 clat percentiles (usec): | 1.00th=[11994], 5.00th=[11994], 10.00th=[11994], 20.00th=[11994], | 30.00th=[11994], 40.00th=[11994], 50.00th=[11994], 60.00th=[79168], | 70.00th=[79168], 80.00th=[79168], 90.00th=[79168], 95.00th=[79168], | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 4096, max= 4096, per=15.16%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.2MiB/s (13.8MB/s)(10.0MiB/758msec) clat (msec): min=15, max=189, avg=66.14, stdev=49.31 lat (msec): min=15, max=189, avg=66.24, stdev=49.30 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 29], | 30.00th=[ 35], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 51], | 70.00th=[ 81], 80.00th=[ 81], 90.00th=[ 88], 95.00th=[ 190], | 99.00th=[ 190], 99.50th=[ 190], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min=10240, max=10240, per=27.07%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=33.33%, 100=41.67%, 250=8.33% cpu : usr=0.00%, sys=17.04%, ctx=76, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23755: Thu May 1 03:32:56 2025 read: IOPS=9, BW=9926KiB/s (10.2MB/s)(6144KiB/619msec) clat (usec): min=11406, max=42564, avg=27889.02, stdev=13194.53 lat (usec): min=11407, max=42565, avg=27890.49, stdev=13194.45 clat percentiles (usec): | 1.00th=[11469], 5.00th=[11469], 10.00th=[11469], 20.00th=[16712], | 30.00th=[16712], 40.00th=[22152], 50.00th=[22152], 60.00th=[32375], | 70.00th=[42206], 80.00th=[42206], 90.00th=[42730], 95.00th=[42730], | 99.00th=[42730], 99.50th=[42730], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 6144, max= 6144, per=22.74%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9926KiB/s (10.2MB/s)(6144KiB/619msec) clat (msec): min=12, max=191, avg=70.09, stdev=70.31 lat (msec): min=12, max=192, avg=70.17, stdev=70.31 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 16], | 30.00th=[ 16], 40.00th=[ 29], 50.00th=[ 29], 60.00th=[ 61], | 70.00th=[ 112], 80.00th=[ 112], 90.00th=[ 192], 95.00th=[ 192], | 99.00th=[ 192], 99.50th=[ 192], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 8192, max= 8192, per=21.66%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=41.67%, 100=8.33%, 250=16.67% cpu : usr=0.32%, sys=12.14%, ctx=40, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23757: Thu May 1 03:32:56 2025 read: IOPS=6, BW=7014KiB/s (7182kB/s)(5120KiB/730msec) clat (usec): min=13239, max=75485, avg=43148.18, stdev=29944.43 lat (usec): min=13239, max=75486, avg=43149.44, stdev=29944.43 clat percentiles (usec): | 1.00th=[13304], 5.00th=[13304], 10.00th=[13304], 20.00th=[13304], | 30.00th=[18482], 40.00th=[18482], 50.00th=[34341], 60.00th=[34341], | 70.00th=[73925], 80.00th=[73925], 90.00th=[74974], 95.00th=[74974], | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 4087, max= 4087, per=15.13%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9819KiB/s (10.1MB/s)(7168KiB/730msec) clat (msec): min=13, max=122, avg=69.98, stdev=37.59 lat (msec): min=13, max=122, avg=70.06, stdev=37.60 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 41], | 30.00th=[ 58], 40.00th=[ 58], 50.00th=[ 67], 60.00th=[ 88], | 70.00th=[ 88], 80.00th=[ 104], 90.00th=[ 123], 95.00th=[ 123], | 99.00th=[ 123], 99.50th=[ 123], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 8175, max= 8175, per=21.61%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=16.67%, 100=41.67%, 250=16.67% cpu : usr=0.00%, sys=12.47%, ctx=53, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=26.4MiB/s (27.7MB/s), 2702KiB/s-11.1MiB/s (2767kB/s-11.7MB/s), io=20.0MiB (20.0MB), run=619-758msec WRITE: bw=36.9MiB/s (38.7MB/s), 8140KiB/s-13.2MiB/s (8335kB/s-13.8MB/s), io=28.0MiB (29.4MB), run=619-758msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23756: Thu May 1 03:32:56 2025 read: IOPS=12, BW=12.5MiB/s (13.1MB/s)(7168KiB/560msec) clat (usec): min=413, max=35204, avg=22046.32, stdev=15269.73 lat (usec): min=413, max=35205, avg=22047.54, stdev=15270.01 clat percentiles (usec): | 1.00th=[ 412], 5.00th=[ 412], 10.00th=[ 412], 20.00th=[ 420], | 30.00th=[24249], 40.00th=[24249], 50.00th=[27919], 60.00th=[30802], | 70.00th=[30802], 80.00th=[35390], 90.00th=[35390], 95.00th=[35390], | 99.00th=[35390], 99.50th=[35390], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min=12263, max=12263, per=44.73%, avg=12263.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 write: IOPS=8, BW=9143KiB/s (9362kB/s)(5120KiB/560msec) clat (msec): min=33, max=185, avg=70.75, stdev=64.39 lat (msec): min=33, max=185, avg=70.80, stdev=64.38 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 44], 60.00th=[ 44], | 70.00th=[ 55], 80.00th=[ 55], 90.00th=[ 186], 95.00th=[ 186], | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 8175, max= 8175, per=21.30%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 50=66.67%, 100=8.33%, 250=8.33% cpu : usr=0.00%, sys=33.21%, ctx=877, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23758: Thu May 1 03:32:56 2025 read: IOPS=2, BW=2889KiB/s (2958kB/s)(2048KiB/709msec) clat (usec): min=538, max=34040, avg=17289.24, stdev=23689.75 lat (usec): min=539, max=34042, avg=17291.11, stdev=23690.66 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 537], 10.00th=[ 537], 20.00th=[ 537], | 30.00th=[ 537], 40.00th=[ 537], 50.00th=[ 537], 60.00th=[33817], | 70.00th=[33817], 80.00th=[33817], 90.00th=[33817], 95.00th=[33817], | 99.00th=[33817], 99.50th=[33817], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 4055, max= 4055, per=14.79%, avg=4055.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=14, BW=14.1MiB/s (14.8MB/s)(10.0MiB/709msec) clat (msec): min=29, max=105, avg=62.26, stdev=27.61 lat (msec): min=30, max=105, avg=62.32, stdev=27.61 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 38], | 30.00th=[ 41], 40.00th=[ 44], 50.00th=[ 47], 60.00th=[ 64], | 70.00th=[ 90], 80.00th=[ 90], 90.00th=[ 101], 95.00th=[ 106], | 99.00th=[ 106], 99.50th=[ 106], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min=12166, max=12166, per=31.70%, avg=12166.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=50.00%, 100=25.00%, 250=16.67% cpu : usr=0.14%, sys=40.82%, ctx=282, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23759: Thu May 1 03:32:56 2025 read: IOPS=14, BW=14.4MiB/s (15.1MB/s)(6144KiB/417msec) clat (usec): min=556, max=42218, avg=16854.46, stdev=18504.54 lat (usec): min=557, max=42219, avg=16855.60, stdev=18504.64 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 553], 10.00th=[ 553], 20.00th=[ 603], | 30.00th=[ 603], 40.00th=[ 635], 50.00th=[ 635], 60.00th=[27657], | 70.00th=[29492], 80.00th=[29492], 90.00th=[42206], 95.00th=[42206], | 99.00th=[42206], 99.50th=[42206], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] write: IOPS=14, BW=14.4MiB/s (15.1MB/s)(6144KiB/417msec) clat (usec): min=31805, max=95730, avg=50561.29, stdev=23382.65 lat (usec): min=31823, max=95805, avg=50607.39, stdev=23393.08 clat percentiles (usec): | 1.00th=[31851], 5.00th=[31851], 10.00th=[31851], 20.00th=[36439], | 30.00th=[36439], 40.00th=[41157], 50.00th=[41157], 60.00th=[44827], | 70.00th=[53740], 80.00th=[53740], 90.00th=[95945], 95.00th=[95945], | 99.00th=[95945], 99.50th=[95945], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] lat (usec) : 750=25.00% lat (msec) : 50=58.33%, 100=16.67% cpu : usr=0.00%, sys=51.20%, ctx=212, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23760: Thu May 1 03:32:56 2025 read: IOPS=6, BW=6854KiB/s (7019kB/s)(5120KiB/747msec) clat (usec): min=332, max=63386, avg=30100.73, stdev=22460.15 lat (usec): min=333, max=63387, avg=30101.90, stdev=22460.15 clat percentiles (usec): | 1.00th=[ 334], 5.00th=[ 334], 10.00th=[ 334], 20.00th=[ 334], | 30.00th=[25822], 40.00th=[25822], 50.00th=[28967], 60.00th=[28967], | 70.00th=[31851], 80.00th=[31851], 90.00th=[63177], 95.00th=[63177], | 99.00th=[63177], 99.50th=[63177], 99.90th=[63177], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 2048, max= 2048, per=7.47%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9596KiB/s (9826kB/s)(7168KiB/747msec) clat (msec): min=32, max=191, avg=76.95, stdev=60.18 lat (msec): min=32, max=191, avg=77.03, stdev=60.19 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 59], 60.00th=[ 65], | 70.00th=[ 65], 80.00th=[ 125], 90.00th=[ 192], 95.00th=[ 192], | 99.00th=[ 192], 99.50th=[ 192], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 8192, max= 8192, per=21.34%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=50.00%, 100=25.00%, 250=16.67% cpu : usr=0.00%, sys=30.43%, ctx=617, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=26.8MiB/s (28.1MB/s), 2889KiB/s-14.4MiB/s (2958kB/s-15.1MB/s), io=20.0MiB (20.0MB), run=417-747msec WRITE: bw=37.5MiB/s (39.3MB/s), 9143KiB/s-14.4MiB/s (9362kB/s-15.1MB/s), io=28.0MiB (29.4MB), run=417-747msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23767: Thu May 1 03:32:58 2025 read: IOPS=4, BW=16.7MiB/s (17.5MB/s)(12.0MiB/720msec) clat (msec): min=128, max=333, avg=232.29, stdev=102.59 lat (msec): min=128, max=333, avg=232.29, stdev=102.59 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 129], 10.00th=[ 129], 20.00th=[ 129], | 30.00th=[ 129], 40.00th=[ 236], 50.00th=[ 236], 60.00th=[ 236], | 70.00th=[ 334], 80.00th=[ 334], 90.00th=[ 334], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 8192, max= 8192, per=48.90%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=23.09%, ctx=1894, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23768: Thu May 1 03:32:58 2025 read: IOPS=1, BW=4188KiB/s (4289kB/s)(4096KiB/978msec) clat (nsec): min=542833k, max=542833k, avg=542832857.00, stdev= 0.00 lat (nsec): min=542834k, max=542834k, avg=542834487.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 542], 10.00th=[ 542], 20.00th=[ 542], | 30.00th=[ 542], 40.00th=[ 542], 50.00th=[ 542], 60.00th=[ 542], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] write: IOPS=2, BW=8376KiB/s (8577kB/s)(8192KiB/978msec) clat (msec): min=182, max=225, avg=204.15, stdev=30.16 lat (msec): min=183, max=225, avg=204.36, stdev=30.10 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 226], | 70.00th=[ 226], 80.00th=[ 226], 90.00th=[ 226], 95.00th=[ 226], | 99.00th=[ 226], 99.50th=[ 226], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 226] lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.20%, sys=28.86%, ctx=939, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23769: Thu May 1 03:32:58 2025 write: IOPS=2, BW=11.2MiB/s (11.7MB/s)(12.0MiB/1072msec) clat (msec): min=112, max=690, avg=345.60, stdev=304.49 lat (msec): min=112, max=690, avg=345.85, stdev=304.48 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 113], 10.00th=[ 113], 20.00th=[ 113], | 30.00th=[ 113], 40.00th=[ 234], 50.00th=[ 234], 60.00th=[ 234], | 70.00th=[ 693], 80.00th=[ 693], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min=15875, max=15875, per=51.94%, avg=15875.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.09%, sys=26.24%, ctx=106, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23770: Thu May 1 03:32:58 2025 write: IOPS=2, BW=11.6MiB/s (12.1MB/s)(12.0MiB/1036msec) clat (msec): min=102, max=465, avg=337.34, stdev=203.81 lat (msec): min=102, max=466, avg=337.67, stdev=203.69 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 103], 10.00th=[ 103], 20.00th=[ 103], | 30.00th=[ 103], 40.00th=[ 443], 50.00th=[ 443], 60.00th=[ 443], | 70.00th=[ 468], 80.00th=[ 468], 90.00th=[ 468], 95.00th=[ 468], | 99.00th=[ 468], 99.50th=[ 468], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 8192, max= 8192, per=26.80%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=25.80%, ctx=132, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.4MiB/s (17.2MB/s), 4188KiB/s-16.7MiB/s (4289kB/s-17.5MB/s), io=16.0MiB (16.8MB), run=720-978msec WRITE: bw=29.9MiB/s (31.3MB/s), 8376KiB/s-11.6MiB/s (8577kB/s-12.1MB/s), io=32.0MiB (33.6MB), run=978-1072msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23771: Thu May 1 03:32:58 2025 read: IOPS=3, BW=12.3MiB/s (12.9MB/s)(12.0MiB/973msec) clat (msec): min=155, max=537, avg=311.27, stdev=200.51 lat (msec): min=155, max=537, avg=311.27, stdev=200.51 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 241], 50.00th=[ 241], 60.00th=[ 241], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=5.66%, ctx=42, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23772: Thu May 1 03:32:58 2025 read: IOPS=0, BW=3430KiB/s (3513kB/s)(4096KiB/1194msec) clat (nsec): min=810060k, max=810060k, avg=810060225.00, stdev= 0.00 lat (nsec): min=810061k, max=810061k, avg=810061380.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 810], 5.00th=[ 810], 10.00th=[ 810], 20.00th=[ 810], | 30.00th=[ 810], 40.00th=[ 810], 50.00th=[ 810], 60.00th=[ 810], | 70.00th=[ 810], 80.00th=[ 810], 90.00th=[ 810], 95.00th=[ 810], | 99.00th=[ 810], 99.50th=[ 810], 99.90th=[ 810], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 8094, max= 8094, per=58.99%, avg=8094.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6861KiB/s (7026kB/s)(8192KiB/1194msec) clat (msec): min=62, max=308, avg=185.35, stdev=174.22 lat (msec): min=62, max=308, avg=185.50, stdev=174.20 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 63], | 30.00th=[ 63], 40.00th=[ 63], 50.00th=[ 63], 60.00th=[ 309], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] lat (msec) : 100=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=9.14%, ctx=102, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23773: Thu May 1 03:32:58 2025 write: IOPS=3, BW=13.7MiB/s (14.4MB/s)(12.0MiB/874msec) clat (msec): min=158, max=485, avg=280.01, stdev=178.92 lat (msec): min=158, max=485, avg=280.33, stdev=178.99 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 197], 50.00th=[ 197], 60.00th=[ 197], | 70.00th=[ 485], 80.00th=[ 485], 90.00th=[ 485], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.23%, sys=11.56%, ctx=65, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23774: Thu May 1 03:32:58 2025 write: IOPS=2, BW=11.3MiB/s (11.8MB/s)(12.0MiB/1062msec) clat (msec): min=198, max=424, avg=346.81, stdev=128.77 lat (msec): min=198, max=424, avg=347.10, stdev=128.80 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 199], 10.00th=[ 199], 20.00th=[ 199], | 30.00th=[ 199], 40.00th=[ 418], 50.00th=[ 418], 60.00th=[ 418], | 70.00th=[ 426], 80.00th=[ 426], 90.00th=[ 426], 95.00th=[ 426], | 99.00th=[ 426], 99.50th=[ 426], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 8192, max= 8192, per=29.85%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.09%, sys=15.93%, ctx=110, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.4MiB/s (14.1MB/s), 3430KiB/s-12.3MiB/s (3513kB/s-12.9MB/s), io=16.0MiB (16.8MB), run=973-1194msec WRITE: bw=26.8MiB/s (28.1MB/s), 6861KiB/s-13.7MiB/s (7026kB/s-14.4MB/s), io=32.0MiB (33.6MB), run=874-1194msec