== sanity test 398b: DIO and buffer IO race ============== 06:45:02 (1747219502) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.29238 s, 38.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32052: Wed May 14 06:45:18 2025 read: IOPS=112, BW=449KiB/s (460kB/s)(5992KiB/13342msec) clat (usec): min=65, max=171798, avg=5007.86, stdev=5073.99 lat (usec): min=65, max=171799, avg=5008.64, stdev=5073.98 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 2073], 10.00th=[ 2376], 20.00th=[ 2868], | 30.00th=[ 3261], 40.00th=[ 3720], 50.00th=[ 4424], 60.00th=[ 5145], | 70.00th=[ 5866], 80.00th=[ 6718], 90.00th=[ 8160], 95.00th=[ 9372], | 99.00th=[ 14615], 99.50th=[ 16057], 99.90th=[ 24249], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 256, max= 552, per=25.00%, avg=443.27, stdev=61.48, samples=26 iops : min= 64, max= 138, avg=110.73, stdev=15.38, samples=26 write: IOPS=117, BW=472KiB/s (483kB/s)(6296KiB/13342msec) clat (usec): min=420, max=25815, avg=3663.61, stdev=3214.33 lat (usec): min=421, max=25816, avg=3664.97, stdev=3214.22 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 578], 10.00th=[ 619], 20.00th=[ 750], | 30.00th=[ 971], 40.00th=[ 2245], 50.00th=[ 2999], 60.00th=[ 3687], | 70.00th=[ 4686], 80.00th=[ 6063], 90.00th=[ 8094], 95.00th=[10028], | 99.00th=[13566], 99.50th=[15139], 99.90th=[25035], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 304, max= 592, per=25.07%, avg=465.38, stdev=69.03, samples=26 iops : min= 76, max= 148, avg=116.27, stdev=17.28, samples=26 lat (usec) : 100=0.78%, 250=0.85%, 500=0.16%, 750=10.16%, 1000=5.57% lat (msec) : 2=4.43%, 4=32.32%, 10=41.24%, 20=4.26%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=22.94%, ctx=5750, majf=0, minf=37 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32053: Wed May 14 06:45:18 2025 read: IOPS=109, BW=439KiB/s (449kB/s)(5940KiB/13538msec) clat (usec): min=53, max=24938, avg=4863.22, stdev=2668.49 lat (usec): min=53, max=24939, avg=4865.75, stdev=2672.07 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 2024], 10.00th=[ 2376], 20.00th=[ 2835], | 30.00th=[ 3228], 40.00th=[ 3720], 50.00th=[ 4359], 60.00th=[ 5014], | 70.00th=[ 5735], 80.00th=[ 6849], 90.00th=[ 8094], 95.00th=[ 9372], | 99.00th=[13698], 99.50th=[16319], 99.90th=[18482], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 280, max= 560, per=24.61%, avg=436.33, stdev=56.50, samples=27 iops : min= 70, max= 140, avg=109.00, stdev=14.10, samples=27 write: IOPS=117, BW=469KiB/s (480kB/s)(6348KiB/13538msec) clat (usec): min=446, max=177075, avg=3930.36, stdev=5483.90 lat (usec): min=446, max=177076, avg=3932.34, stdev=5483.37 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 570], 10.00th=[ 635], 20.00th=[ 791], | 30.00th=[ 1029], 40.00th=[ 2311], 50.00th=[ 3097], 60.00th=[ 3785], | 70.00th=[ 4817], 80.00th=[ 6521], 90.00th=[ 8586], 95.00th=[ 10159], | 99.00th=[ 13960], 99.50th=[ 17433], 99.90th=[ 23725], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 352, max= 664, per=25.04%, avg=464.74, stdev=68.04, samples=27 iops : min= 88, max= 166, avg=116.11, stdev=17.02, samples=27 lat (usec) : 100=1.11%, 250=0.81%, 500=0.33%, 750=9.28%, 1000=5.57% lat (msec) : 2=5.14%, 4=31.80%, 10=41.41%, 20=4.43%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=22.86%, ctx=5923, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32054: Wed May 14 06:45:18 2025 read: IOPS=111, BW=448KiB/s (458kB/s)(6028KiB/13463msec) clat (usec): min=65, max=21329, avg=4938.05, stdev=2626.03 lat (usec): min=66, max=21330, avg=4940.38, stdev=2628.85 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 2073], 10.00th=[ 2474], 20.00th=[ 2966], | 30.00th=[ 3392], 40.00th=[ 3884], 50.00th=[ 4424], 60.00th=[ 5014], | 70.00th=[ 5866], 80.00th=[ 6718], 90.00th=[ 8094], 95.00th=[ 9503], | 99.00th=[14222], 99.50th=[15795], 99.90th=[17695], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 240, max= 520, per=24.98%, avg=442.96, stdev=60.49, samples=26 iops : min= 60, max= 130, avg=110.65, stdev=15.12, samples=26 write: IOPS=116, BW=465KiB/s (476kB/s)(6260KiB/13463msec) clat (usec): min=422, max=176868, avg=3797.18, stdev=5398.70 lat (usec): min=423, max=176868, avg=3798.29, stdev=5398.87 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 578], 10.00th=[ 627], 20.00th=[ 775], | 30.00th=[ 1012], 40.00th=[ 2278], 50.00th=[ 2966], 60.00th=[ 3687], | 70.00th=[ 4752], 80.00th=[ 6128], 90.00th=[ 8356], 95.00th=[ 10159], | 99.00th=[ 13042], 99.50th=[ 15139], 99.90th=[ 20841], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 312, max= 638, per=24.69%, avg=458.27, stdev=75.09, samples=26 iops : min= 78, max= 159, avg=114.50, stdev=18.72, samples=26 lat (usec) : 100=0.98%, 250=0.72%, 500=0.20%, 750=9.47%, 1000=5.57% lat (msec) : 2=4.72%, 4=31.58%, 10=41.99%, 20=4.69%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=22.83%, ctx=5956, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32055: Wed May 14 06:45:18 2025 read: IOPS=111, BW=447KiB/s (458kB/s)(6052KiB/13541msec) clat (usec): min=67, max=180329, avg=4966.66, stdev=5231.63 lat (usec): min=67, max=180330, avg=4967.45, stdev=5231.66 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 2040], 10.00th=[ 2409], 20.00th=[ 2900], | 30.00th=[ 3294], 40.00th=[ 3818], 50.00th=[ 4424], 60.00th=[ 5080], | 70.00th=[ 5735], 80.00th=[ 6587], 90.00th=[ 7832], 95.00th=[ 9110], | 99.00th=[ 14353], 99.50th=[ 14877], 99.90th=[ 32900], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 248, max= 664, per=25.19%, avg=446.67, stdev=72.94, samples=27 iops : min= 62, max= 166, avg=111.59, stdev=18.22, samples=27 write: IOPS=115, BW=461KiB/s (472kB/s)(6236KiB/13541msec) clat (usec): min=487, max=21360, avg=3811.37, stdev=3316.93 lat (usec): min=487, max=21361, avg=3812.68, stdev=3316.71 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 578], 10.00th=[ 627], 20.00th=[ 758], | 30.00th=[ 955], 40.00th=[ 2311], 50.00th=[ 3195], 60.00th=[ 3851], | 70.00th=[ 4948], 80.00th=[ 6194], 90.00th=[ 8291], 95.00th=[10159], | 99.00th=[14484], 99.50th=[16188], 99.90th=[20317], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 328, max= 656, per=24.74%, avg=459.15, stdev=81.04, samples=27 iops : min= 82, max= 164, avg=114.70, stdev=20.28, samples=27 lat (usec) : 100=0.94%, 250=0.98%, 500=0.07%, 750=9.80%, 1000=5.89% lat (msec) : 2=4.07%, 4=30.89%, 10=43.00%, 20=4.17%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=22.80%, ctx=5881, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1773KiB/s (1816kB/s), 439KiB/s-449KiB/s (449kB/s-460kB/s), io=23.4MiB (24.6MB), run=13342-13541msec WRITE: bw=1857KiB/s (1901kB/s), 461KiB/s-472KiB/s (472kB/s-483kB/s), io=24.6MiB (25.7MB), run=13342-13541msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32057: Wed May 14 06:45:22 2025 read: IOPS=87, BW=350KiB/s (359kB/s)(5992KiB/17101msec) clat (usec): min=1705, max=180455, avg=4764.28, stdev=5750.91 lat (usec): min=1706, max=180456, avg=4764.97, stdev=5750.92 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1958], 10.00th=[ 2057], 20.00th=[ 2212], | 30.00th=[ 2409], 40.00th=[ 2704], 50.00th=[ 3130], 60.00th=[ 3687], | 70.00th=[ 4883], 80.00th=[ 6718], 90.00th=[ 10028], 95.00th=[ 11994], | 99.00th=[ 17695], 99.50th=[ 20317], 99.90th=[ 25297], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 160, max= 776, per=24.91%, avg=349.29, stdev=166.20, samples=34 iops : min= 40, max= 194, avg=87.26, stdev=41.55, samples=34 write: IOPS=92, BW=368KiB/s (377kB/s)(6296KiB/17101msec) clat (usec): min=2309, max=34457, avg=6300.76, stdev=3891.99 lat (usec): min=2309, max=34458, avg=6301.68, stdev=3891.98 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2966], 10.00th=[ 3163], 20.00th=[ 3425], | 30.00th=[ 3720], 40.00th=[ 4047], 50.00th=[ 4621], 60.00th=[ 5538], | 70.00th=[ 7242], 80.00th=[ 9110], 90.00th=[11600], 95.00th=[13829], | 99.00th=[19792], 99.50th=[23462], 99.90th=[29754], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 184, max= 728, per=24.96%, avg=366.47, stdev=166.73, samples=34 iops : min= 46, max= 182, avg=91.56, stdev=41.67, samples=34 lat (msec) : 2=3.42%, 4=47.17%, 10=36.26%, 20=12.43%, 50=0.68% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=12.09%, ctx=3659, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32058: Wed May 14 06:45:22 2025 read: IOPS=86, BW=347KiB/s (356kB/s)(5940KiB/17108msec) clat (usec): min=1600, max=180555, avg=4580.58, stdev=5664.77 lat (usec): min=1600, max=180556, avg=4581.32, stdev=5664.78 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1942], 10.00th=[ 2040], 20.00th=[ 2212], | 30.00th=[ 2409], 40.00th=[ 2638], 50.00th=[ 2999], 60.00th=[ 3556], | 70.00th=[ 4490], 80.00th=[ 6128], 90.00th=[ 9765], 95.00th=[ 12387], | 99.00th=[ 16712], 99.50th=[ 17171], 99.90th=[ 22938], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 151, max= 752, per=24.71%, avg=346.50, stdev=167.29, samples=34 iops : min= 37, max= 188, avg=86.56, stdev=41.87, samples=34 write: IOPS=92, BW=371KiB/s (380kB/s)(6348KiB/17108msec) clat (usec): min=2297, max=28019, avg=6472.15, stdev=3923.37 lat (usec): min=2298, max=28020, avg=6473.11, stdev=3923.41 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 2999], 10.00th=[ 3163], 20.00th=[ 3425], | 30.00th=[ 3720], 40.00th=[ 4080], 50.00th=[ 4752], 60.00th=[ 5932], | 70.00th=[ 7701], 80.00th=[ 9634], 90.00th=[11600], 95.00th=[13698], | 99.00th=[20841], 99.50th=[23462], 99.90th=[26084], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 215, max= 744, per=25.14%, avg=369.09, stdev=163.69, samples=34 iops : min= 53, max= 186, avg=92.21, stdev=40.97, samples=34 lat (msec) : 2=3.74%, 4=47.69%, 10=34.57%, 20=13.28%, 50=0.68% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=12.11%, ctx=3640, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32059: Wed May 14 06:45:22 2025 read: IOPS=88, BW=355KiB/s (363kB/s)(6028KiB/16998msec) clat (usec): min=1581, max=24505, avg=4520.12, stdev=3376.60 lat (usec): min=1582, max=24506, avg=4520.87, stdev=3376.61 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1975], 10.00th=[ 2089], 20.00th=[ 2245], | 30.00th=[ 2442], 40.00th=[ 2704], 50.00th=[ 3032], 60.00th=[ 3654], | 70.00th=[ 4621], 80.00th=[ 6325], 90.00th=[ 9503], 95.00th=[11994], | 99.00th=[16909], 99.50th=[19268], 99.90th=[21103], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 159, max= 736, per=24.78%, avg=347.45, stdev=156.44, samples=33 iops : min= 39, max= 184, avg=86.76, stdev=39.14, samples=33 write: IOPS=92, BW=368KiB/s (377kB/s)(6260KiB/16998msec) clat (msec): min=2, max=161, avg= 6.47, stdev= 5.53 lat (msec): min=2, max=161, avg= 6.47, stdev= 5.53 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 23], 99.50th=[ 24], 99.90th=[ 29], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 183, max= 744, per=24.48%, avg=359.33, stdev=163.75, samples=33 iops : min= 45, max= 186, avg=89.73, stdev=40.96, samples=33 lat (msec) : 2=2.99%, 4=47.72%, 10=36.82%, 20=11.69%, 50=0.75% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=12.14%, ctx=3653, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32060: Wed May 14 06:45:22 2025 read: IOPS=88, BW=354KiB/s (362kB/s)(6052KiB/17117msec) clat (usec): min=1570, max=25529, avg=4518.86, stdev=3425.74 lat (usec): min=1571, max=25529, avg=4519.85, stdev=3425.68 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1958], 10.00th=[ 2073], 20.00th=[ 2245], | 30.00th=[ 2474], 40.00th=[ 2704], 50.00th=[ 3032], 60.00th=[ 3589], | 70.00th=[ 4490], 80.00th=[ 6325], 90.00th=[ 9765], 95.00th=[11994], | 99.00th=[17433], 99.50th=[19530], 99.90th=[25035], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 176, max= 712, per=25.03%, avg=350.88, stdev=159.83, samples=34 iops : min= 44, max= 178, avg=87.59, stdev=40.02, samples=34 write: IOPS=91, BW=364KiB/s (373kB/s)(6236KiB/17117msec) clat (msec): min=2, max=159, avg= 6.56, stdev= 5.54 lat (msec): min=2, max=159, avg= 6.56, stdev= 5.54 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 21], 99.50th=[ 26], 99.90th=[ 35], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 192, max= 688, per=24.67%, avg=362.18, stdev=163.22, samples=34 iops : min= 48, max= 172, avg=90.41, stdev=40.87, samples=34 lat (msec) : 2=3.06%, 4=48.27%, 10=35.35%, 20=12.57%, 50=0.72% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=12.24%, ctx=3663, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1403KiB/s (1436kB/s), 347KiB/s-355KiB/s (356kB/s-363kB/s), io=23.4MiB (24.6MB), run=16998-17117msec WRITE: bw=1469KiB/s (1504kB/s), 364KiB/s-371KiB/s (373kB/s-380kB/s), io=24.6MiB (25.7MB), run=16998-17117msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32086: Wed May 14 06:45:27 2025 read: IOPS=99, BW=1593KiB/s (1631kB/s)(5920KiB/3717msec) clat (usec): min=71, max=15663, avg=5186.73, stdev=2953.18 lat (usec): min=72, max=15663, avg=5188.52, stdev=2954.42 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 95], 10.00th=[ 2311], 20.00th=[ 3130], | 30.00th=[ 3654], 40.00th=[ 4080], 50.00th=[ 4752], 60.00th=[ 5473], | 70.00th=[ 6325], 80.00th=[ 7373], 90.00th=[ 9110], 95.00th=[10683], | 99.00th=[13698], 99.50th=[15533], 99.90th=[15664], 99.95th=[15664], | 99.99th=[15664] bw ( KiB/s): min= 1344, max= 1948, per=25.08%, avg=1576.57, stdev=239.65, samples=7 iops : min= 84, max= 121, avg=98.43, stdev=14.79, samples=7 write: IOPS=107, BW=1713KiB/s (1754kB/s)(6368KiB/3717msec) clat (usec): min=708, max=24700, avg=4385.03, stdev=3758.23 lat (usec): min=709, max=24701, avg=4390.25, stdev=3755.79 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 832], 10.00th=[ 906], 20.00th=[ 1090], | 30.00th=[ 1483], 40.00th=[ 2606], 50.00th=[ 3326], 60.00th=[ 4113], | 70.00th=[ 5604], 80.00th=[ 7308], 90.00th=[ 9503], 95.00th=[12125], | 99.00th=[15664], 99.50th=[19006], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 1088, max= 2016, per=25.70%, avg=1700.00, stdev=336.48, samples=7 iops : min= 68, max= 126, avg=106.14, stdev=20.96, samples=7 lat (usec) : 100=2.60%, 250=1.69%, 750=0.52%, 1000=7.16% lat (msec) : 2=10.55%, 4=26.43%, 10=43.36%, 20=7.55%, 50=0.13% cpu : usr=0.19%, sys=24.87%, ctx=1631, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32087: Wed May 14 06:45:27 2025 read: IOPS=94, BW=1508KiB/s (1544kB/s)(5744KiB/3810msec) clat (usec): min=72, max=21693, avg=5526.36, stdev=3374.71 lat (usec): min=72, max=21694, avg=5529.28, stdev=3375.81 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 99], 10.00th=[ 2507], 20.00th=[ 3163], | 30.00th=[ 3785], 40.00th=[ 4555], 50.00th=[ 5080], 60.00th=[ 5735], | 70.00th=[ 6521], 80.00th=[ 7439], 90.00th=[ 9110], 95.00th=[11469], | 99.00th=[17695], 99.50th=[20317], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 1213, max= 1664, per=23.40%, avg=1471.00, stdev=149.25, samples=7 iops : min= 75, max= 104, avg=91.71, stdev= 9.46, samples=7 write: IOPS=107, BW=1718KiB/s (1759kB/s)(6544KiB/3810msec) clat (usec): min=676, max=20716, avg=4429.94, stdev=3636.21 lat (usec): min=677, max=20718, avg=4431.03, stdev=3636.27 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 889], 10.00th=[ 979], 20.00th=[ 1205], | 30.00th=[ 1762], 40.00th=[ 2704], 50.00th=[ 3425], 60.00th=[ 4359], | 70.00th=[ 5342], 80.00th=[ 6980], 90.00th=[ 9372], 95.00th=[11731], | 99.00th=[16057], 99.50th=[17957], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 1469, max= 2016, per=25.14%, avg=1663.00, stdev=193.17, samples=7 iops : min= 91, max= 126, avg=103.71, stdev=12.18, samples=7 lat (usec) : 100=2.34%, 250=1.04%, 750=0.52%, 1000=5.47% lat (msec) : 2=11.72%, 4=24.22%, 10=46.87%, 20=7.42%, 50=0.39% cpu : usr=0.03%, sys=24.38%, ctx=1759, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32088: Wed May 14 06:45:27 2025 read: IOPS=106, BW=1711KiB/s (1752kB/s)(6256KiB/3657msec) clat (usec): min=68, max=22155, avg=5243.51, stdev=3355.60 lat (usec): min=69, max=22156, avg=5244.24, stdev=3355.61 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 98], 10.00th=[ 2311], 20.00th=[ 2999], | 30.00th=[ 3490], 40.00th=[ 3949], 50.00th=[ 4686], 60.00th=[ 5473], | 70.00th=[ 6325], 80.00th=[ 7504], 90.00th=[ 8848], 95.00th=[11207], | 99.00th=[16581], 99.50th=[20055], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 1280, max= 2368, per=27.27%, avg=1714.29, stdev=353.38, samples=7 iops : min= 80, max= 148, avg=107.14, stdev=22.09, samples=7 write: IOPS=103, BW=1649KiB/s (1689kB/s)(6032KiB/3657msec) clat (usec): min=720, max=18887, avg=4141.49, stdev=3457.14 lat (usec): min=721, max=18888, avg=4142.87, stdev=3457.19 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 865], 10.00th=[ 914], 20.00th=[ 1090], | 30.00th=[ 1336], 40.00th=[ 2008], 50.00th=[ 3228], 60.00th=[ 4080], | 70.00th=[ 5276], 80.00th=[ 7111], 90.00th=[ 9241], 95.00th=[11207], | 99.00th=[15401], 99.50th=[15795], 99.90th=[19006], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 1088, max= 2112, per=24.26%, avg=1604.57, stdev=303.26, samples=7 iops : min= 68, max= 132, avg=100.29, stdev=18.95, samples=7 lat (usec) : 100=2.60%, 250=2.08%, 750=0.39%, 1000=7.29% lat (msec) : 2=11.72%, 4=26.17%, 10=42.45%, 20=7.16%, 50=0.13% cpu : usr=0.16%, sys=24.70%, ctx=1626, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32089: Wed May 14 06:45:27 2025 read: IOPS=104, BW=1671KiB/s (1711kB/s)(6032KiB/3609msec) clat (usec): min=61, max=19800, avg=5211.74, stdev=3045.03 lat (usec): min=62, max=19800, avg=5212.43, stdev=3045.03 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 112], 10.00th=[ 2474], 20.00th=[ 3228], | 30.00th=[ 3490], 40.00th=[ 4080], 50.00th=[ 4686], 60.00th=[ 5538], | 70.00th=[ 6128], 80.00th=[ 7177], 90.00th=[ 8848], 95.00th=[10552], | 99.00th=[16188], 99.50th=[18482], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 1472, max= 1792, per=26.25%, avg=1650.29, stdev=109.08, samples=7 iops : min= 92, max= 112, avg=103.14, stdev= 6.82, samples=7 write: IOPS=108, BW=1733KiB/s (1775kB/s)(6256KiB/3609msec) clat (usec): min=737, max=19483, avg=4098.87, stdev=3388.89 lat (usec): min=738, max=19483, avg=4100.18, stdev=3388.96 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 873], 10.00th=[ 922], 20.00th=[ 1057], | 30.00th=[ 1401], 40.00th=[ 2278], 50.00th=[ 3195], 60.00th=[ 3949], | 70.00th=[ 4948], 80.00th=[ 6915], 90.00th=[ 8848], 95.00th=[11338], | 99.00th=[14222], 99.50th=[16712], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 1440, max= 2112, per=26.33%, avg=1741.71, stdev=208.09, samples=7 iops : min= 90, max= 132, avg=108.86, stdev=13.01, samples=7 lat (usec) : 100=2.08%, 250=1.82%, 750=0.13%, 1000=8.59% lat (msec) : 2=10.16%, 4=27.21%, 10=42.97%, 20=7.03% cpu : usr=0.22%, sys=25.00%, ctx=1585, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6287KiB/s (6437kB/s), 1508KiB/s-1711KiB/s (1544kB/s-1752kB/s), io=23.4MiB (24.5MB), run=3609-3810msec WRITE: bw=6614KiB/s (6773kB/s), 1649KiB/s-1733KiB/s (1689kB/s-1775kB/s), io=24.6MiB (25.8MB), run=3609-3810msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32090: Wed May 14 06:45:28 2025 read: IOPS=79, BW=1271KiB/s (1302kB/s)(5920KiB/4656msec) clat (usec): min=1960, max=25928, avg=4864.20, stdev=3697.44 lat (usec): min=1961, max=25928, avg=4864.96, stdev=3697.43 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2180], 10.00th=[ 2311], 20.00th=[ 2507], | 30.00th=[ 2704], 40.00th=[ 3032], 50.00th=[ 3326], 60.00th=[ 3785], | 70.00th=[ 4490], 80.00th=[ 6063], 90.00th=[10552], 95.00th=[13173], | 99.00th=[18220], 99.50th=[18220], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 832, max= 2496, per=24.44%, avg=1240.89, stdev=552.35, samples=9 iops : min= 52, max= 156, avg=77.56, stdev=34.52, samples=9 write: IOPS=85, BW=1368KiB/s (1401kB/s)(6368KiB/4656msec) clat (usec): min=2553, max=28858, avg=7032.87, stdev=4326.92 lat (usec): min=2555, max=28860, avg=7034.26, stdev=4326.90 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3163], 10.00th=[ 3359], 20.00th=[ 3687], | 30.00th=[ 3982], 40.00th=[ 4555], 50.00th=[ 5407], 60.00th=[ 6783], | 70.00th=[ 8094], 80.00th=[10290], 90.00th=[12780], 95.00th=[16450], | 99.00th=[22938], 99.50th=[26084], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 864, max= 2688, per=24.95%, avg=1333.33, stdev=585.70, samples=9 iops : min= 54, max= 168, avg=83.33, stdev=36.61, samples=9 lat (msec) : 2=0.13%, 4=45.96%, 10=37.50%, 20=15.62%, 50=0.78% cpu : usr=0.26%, sys=12.37%, ctx=933, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32091: Wed May 14 06:45:28 2025 read: IOPS=76, BW=1218KiB/s (1247kB/s)(5744KiB/4716msec) clat (usec): min=1931, max=25569, avg=4982.05, stdev=3912.12 lat (usec): min=1931, max=25570, avg=4982.87, stdev=3912.06 clat percentiles (usec): | 1.00th=[ 2057], 5.00th=[ 2147], 10.00th=[ 2311], 20.00th=[ 2474], | 30.00th=[ 2704], 40.00th=[ 2999], 50.00th=[ 3359], 60.00th=[ 3916], | 70.00th=[ 4555], 80.00th=[ 6521], 90.00th=[10814], 95.00th=[15008], | 99.00th=[18220], 99.50th=[20579], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 800, max= 2272, per=23.23%, avg=1179.78, stdev=500.29, samples=9 iops : min= 50, max= 142, avg=73.56, stdev=31.22, samples=9 write: IOPS=86, BW=1388KiB/s (1421kB/s)(6544KiB/4716msec) clat (usec): min=2481, max=28005, avg=7087.18, stdev=4338.36 lat (usec): min=2482, max=28007, avg=7088.82, stdev=4338.24 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3032], 10.00th=[ 3294], 20.00th=[ 3687], | 30.00th=[ 4015], 40.00th=[ 4490], 50.00th=[ 5211], 60.00th=[ 6652], | 70.00th=[ 8455], 80.00th=[10421], 90.00th=[13566], 95.00th=[15664], | 99.00th=[20055], 99.50th=[24773], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 928, max= 2816, per=24.74%, avg=1321.89, stdev=592.77, samples=9 iops : min= 58, max= 176, avg=82.44, stdev=37.04, samples=9 lat (msec) : 2=0.13%, 4=44.01%, 10=38.54%, 20=16.54%, 50=0.78% cpu : usr=0.23%, sys=11.71%, ctx=921, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32092: Wed May 14 06:45:28 2025 read: IOPS=83, BW=1344KiB/s (1376kB/s)(6256KiB/4656msec) clat (usec): min=1884, max=25406, avg=4997.47, stdev=3712.05 lat (usec): min=1884, max=25407, avg=4998.17, stdev=3712.05 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2180], 10.00th=[ 2278], 20.00th=[ 2474], | 30.00th=[ 2704], 40.00th=[ 2900], 50.00th=[ 3326], 60.00th=[ 3916], | 70.00th=[ 4752], 80.00th=[ 7635], 90.00th=[10945], 95.00th=[13042], | 99.00th=[17695], 99.50th=[21103], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 768, max= 2848, per=26.11%, avg=1325.89, stdev=646.75, samples=9 iops : min= 48, max= 178, avg=82.78, stdev=40.47, samples=9 write: IOPS=80, BW=1296KiB/s (1327kB/s)(6032KiB/4656msec) clat (usec): min=2597, max=27004, avg=7006.89, stdev=4303.16 lat (usec): min=2599, max=27005, avg=7008.25, stdev=4303.25 clat percentiles (usec): | 1.00th=[ 2802], 5.00th=[ 3097], 10.00th=[ 3326], 20.00th=[ 3621], | 30.00th=[ 4015], 40.00th=[ 4359], 50.00th=[ 5211], 60.00th=[ 6980], | 70.00th=[ 8029], 80.00th=[ 9896], 90.00th=[12911], 95.00th=[15664], | 99.00th=[21890], 99.50th=[26608], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 896, max= 2432, per=23.28%, avg=1244.11, stdev=485.76, samples=9 iops : min= 56, max= 152, avg=77.67, stdev=30.38, samples=9 lat (msec) : 2=0.65%, 4=45.57%, 10=37.50%, 20=15.23%, 50=1.04% cpu : usr=0.19%, sys=12.35%, ctx=903, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32093: Wed May 14 06:45:28 2025 read: IOPS=82, BW=1319KiB/s (1351kB/s)(6032KiB/4572msec) clat (usec): min=1931, max=21403, avg=5025.24, stdev=3876.30 lat (usec): min=1932, max=21404, avg=5025.96, stdev=3876.29 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2180], 10.00th=[ 2278], 20.00th=[ 2442], | 30.00th=[ 2671], 40.00th=[ 2966], 50.00th=[ 3392], 60.00th=[ 3884], | 70.00th=[ 5014], 80.00th=[ 6783], 90.00th=[11338], 95.00th=[13829], | 99.00th=[20317], 99.50th=[21103], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 864, max= 2624, per=25.41%, avg=1290.33, stdev=538.27, samples=9 iops : min= 54, max= 164, avg=80.56, stdev=33.69, samples=9 write: IOPS=85, BW=1368KiB/s (1401kB/s)(6256KiB/4572msec) clat (usec): min=2689, max=33184, avg=6817.96, stdev=4119.17 lat (usec): min=2690, max=33185, avg=6819.32, stdev=4119.18 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3163], 10.00th=[ 3425], 20.00th=[ 3785], | 30.00th=[ 4015], 40.00th=[ 4490], 50.00th=[ 5080], 60.00th=[ 6259], | 70.00th=[ 7963], 80.00th=[ 9896], 90.00th=[12649], 95.00th=[15008], | 99.00th=[19792], 99.50th=[25560], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 832, max= 2464, per=25.35%, avg=1354.33, stdev=506.97, samples=9 iops : min= 52, max= 154, avg=84.56, stdev=31.75, samples=9 lat (msec) : 2=0.13%, 4=44.92%, 10=39.06%, 20=14.97%, 50=0.91% cpu : usr=0.15%, sys=12.62%, ctx=924, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5079KiB/s (5201kB/s), 1218KiB/s-1344KiB/s (1247kB/s-1376kB/s), io=23.4MiB (24.5MB), run=4572-4716msec WRITE: bw=5344KiB/s (5472kB/s), 1296KiB/s-1388KiB/s (1327kB/s-1421kB/s), io=24.6MiB (25.8MB), run=4572-4716msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32105: Wed May 14 06:45:29 2025 read: IOPS=20, BW=20.9MiB/s (21.9MB/s)(7168KiB/335msec) clat (usec): min=939, max=31738, avg=20490.92, stdev=12812.24 lat (usec): min=940, max=31739, avg=20492.18, stdev=12812.30 clat percentiles (usec): | 1.00th=[ 938], 5.00th=[ 938], 10.00th=[ 938], 20.00th=[ 3294], | 30.00th=[24773], 40.00th=[24773], 50.00th=[25035], 60.00th=[28705], | 70.00th=[28705], 80.00th=[29230], 90.00th=[31851], 95.00th=[31851], | 99.00th=[31851], 99.50th=[31851], 99.90th=[31851], 99.95th=[31851], | 99.99th=[31851] write: IOPS=14, BW=14.9MiB/s (15.7MB/s)(5120KiB/335msec) clat (usec): min=23675, max=52338, avg=36910.45, stdev=13385.11 lat (usec): min=23745, max=52417, avg=36956.84, stdev=13385.99 clat percentiles (usec): | 1.00th=[23725], 5.00th=[23725], 10.00th=[23725], 20.00th=[23725], | 30.00th=[28705], 40.00th=[28705], 50.00th=[29492], 60.00th=[29492], | 70.00th=[50594], 80.00th=[50594], 90.00th=[52167], 95.00th=[52167], | 99.00th=[52167], 99.50th=[52167], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] lat (usec) : 1000=8.33% lat (msec) : 4=8.33%, 50=66.67%, 100=16.67% cpu : usr=0.00%, sys=53.59%, ctx=946, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32106: Wed May 14 06:45:29 2025 read: IOPS=3, BW=3325KiB/s (3404kB/s)(2048KiB/616msec) clat (usec): min=32137, max=55667, avg=43902.34, stdev=16638.58 lat (usec): min=32138, max=55668, avg=43903.40, stdev=16638.67 clat percentiles (usec): | 1.00th=[32113], 5.00th=[32113], 10.00th=[32113], 20.00th=[32113], | 30.00th=[32113], 40.00th=[32113], 50.00th=[32113], 60.00th=[55837], | 70.00th=[55837], 80.00th=[55837], 90.00th=[55837], 95.00th=[55837], | 99.00th=[55837], 99.50th=[55837], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 4096, max= 4096, per=12.32%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=16, BW=16.2MiB/s (17.0MB/s)(10.0MiB/616msec) clat (msec): min=19, max=102, avg=41.94, stdev=25.52 lat (msec): min=19, max=102, avg=42.00, stdev=25.53 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 21], | 30.00th=[ 25], 40.00th=[ 27], 50.00th=[ 32], 60.00th=[ 34], | 70.00th=[ 51], 80.00th=[ 51], 90.00th=[ 61], 95.00th=[ 104], | 99.00th=[ 104], 99.50th=[ 104], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min=14336, max=14336, per=30.80%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=58.33%, 100=25.00%, 250=8.33% cpu : usr=0.16%, sys=40.98%, ctx=566, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32107: Wed May 14 06:45:29 2025 read: IOPS=11, BW=11.2MiB/s (11.7MB/s)(6144KiB/536msec) clat (usec): min=542, max=50150, avg=31944.94, stdev=17524.73 lat (usec): min=544, max=50152, avg=31946.35, stdev=17524.73 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 545], 10.00th=[ 545], 20.00th=[24773], | 30.00th=[24773], 40.00th=[34866], 50.00th=[34866], 60.00th=[40109], | 70.00th=[41681], 80.00th=[41681], 90.00th=[50070], 95.00th=[50070], | 99.00th=[50070], 99.50th=[50070], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min=10178, max=10178, per=30.61%, avg=10178.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.2MiB/s (11.7MB/s)(6144KiB/536msec) clat (msec): min=22, max=143, avg=51.43, stdev=45.86 lat (msec): min=22, max=143, avg=51.48, stdev=45.84 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 26], | 30.00th=[ 26], 40.00th=[ 34], 50.00th=[ 34], 60.00th=[ 40], | 70.00th=[ 44], 80.00th=[ 44], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min=12214, max=12214, per=26.24%, avg=12214.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=75.00%, 100=8.33%, 250=8.33% cpu : usr=0.19%, sys=41.31%, ctx=481, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32108: Wed May 14 06:45:29 2025 read: IOPS=8, BW=9209KiB/s (9430kB/s)(5120KiB/556msec) clat (usec): min=367, max=53660, avg=11093.97, stdev=23795.64 lat (usec): min=368, max=53674, avg=11097.22, stdev=23801.56 clat percentiles (usec): | 1.00th=[ 367], 5.00th=[ 367], 10.00th=[ 367], 20.00th=[ 367], | 30.00th=[ 453], 40.00th=[ 453], 50.00th=[ 469], 60.00th=[ 469], | 70.00th=[ 523], 80.00th=[ 523], 90.00th=[53740], 95.00th=[53740], | 99.00th=[53740], 99.50th=[53740], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 8192, max= 8192, per=24.64%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.6MiB/s (13.2MB/s)(7168KiB/556msec) clat (msec): min=25, max=110, avg=63.24, stdev=29.93 lat (msec): min=25, max=110, avg=63.30, stdev=29.94 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 34], | 30.00th=[ 52], 40.00th=[ 52], 50.00th=[ 60], 60.00th=[ 73], | 70.00th=[ 73], 80.00th=[ 89], 90.00th=[ 111], 95.00th=[ 111], | 99.00th=[ 111], 99.50th=[ 111], 99.90th=[ 111], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min=14336, max=14336, per=30.80%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (usec) : 500=25.00%, 750=8.33% lat (msec) : 50=16.67%, 100=41.67%, 250=8.33% cpu : usr=0.00%, sys=30.45%, ctx=185, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=32.5MiB/s (34.0MB/s), 3325KiB/s-20.9MiB/s (3404kB/s-21.9MB/s), io=20.0MiB (20.0MB), run=335-616msec WRITE: bw=45.5MiB/s (47.7MB/s), 11.2MiB/s-16.2MiB/s (11.7MB/s-17.0MB/s), io=28.0MiB (29.4MB), run=335-616msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32109: Wed May 14 06:45:29 2025 read: IOPS=9, BW=9.91MiB/s (10.4MB/s)(7168KiB/706msec) clat (msec): min=22, max=124, avg=55.36, stdev=42.61 lat (msec): min=22, max=124, avg=55.36, stdev=42.61 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 24], | 30.00th=[ 30], 40.00th=[ 30], 50.00th=[ 34], 60.00th=[ 46], | 70.00th=[ 46], 80.00th=[ 109], 90.00th=[ 125], 95.00th=[ 125], | 99.00th=[ 125], 99.50th=[ 125], 99.90th=[ 125], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min=10240, max=10240, per=38.55%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7252KiB/s (7426kB/s)(5120KiB/706msec) clat (msec): min=17, max=132, avg=61.47, stdev=49.23 lat (msec): min=17, max=132, avg=61.57, stdev=49.19 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 48], 60.00th=[ 48], | 70.00th=[ 90], 80.00th=[ 90], 90.00th=[ 133], 95.00th=[ 133], | 99.00th=[ 133], 99.50th=[ 133], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 2048, max= 2048, per=5.51%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=58.33%, 100=8.33%, 250=25.00% cpu : usr=0.14%, sys=17.30%, ctx=76, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32110: Wed May 14 06:45:29 2025 read: IOPS=2, BW=2745KiB/s (2811kB/s)(2048KiB/746msec) clat (msec): min=64, max=113, avg=89.15, stdev=34.74 lat (msec): min=64, max=113, avg=89.15, stdev=34.74 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 65], 10.00th=[ 65], 20.00th=[ 65], | 30.00th=[ 65], 40.00th=[ 65], 50.00th=[ 65], 60.00th=[ 114], | 70.00th=[ 114], 80.00th=[ 114], 90.00th=[ 114], 95.00th=[ 114], | 99.00th=[ 114], 99.50th=[ 114], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 2035, max= 2035, per=7.66%, avg=2035.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.4MiB/s (14.1MB/s)(10.0MiB/746msec) clat (msec): min=19, max=148, avg=46.47, stdev=40.70 lat (msec): min=19, max=148, avg=46.56, stdev=40.71 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 20], | 30.00th=[ 23], 40.00th=[ 24], 50.00th=[ 25], 60.00th=[ 32], | 70.00th=[ 63], 80.00th=[ 63], 90.00th=[ 77], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 6107, max= 6107, per=16.42%, avg=6107.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=41.67%, 100=25.00%, 250=16.67% cpu : usr=0.40%, sys=15.84%, ctx=62, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32111: Wed May 14 06:45:29 2025 read: IOPS=7, BW=7969KiB/s (8160kB/s)(6144KiB/771msec) clat (usec): min=17120, max=61675, avg=33317.47, stdev=15363.62 lat (usec): min=17122, max=61676, avg=33318.56, stdev=15363.28 clat percentiles (usec): | 1.00th=[17171], 5.00th=[17171], 10.00th=[17171], 20.00th=[23987], | 30.00th=[23987], 40.00th=[28967], 50.00th=[28967], 60.00th=[32113], | 70.00th=[35914], 80.00th=[35914], 90.00th=[61604], 95.00th=[61604], | 99.00th=[61604], 99.50th=[61604], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] write: IOPS=7, BW=7969KiB/s (8160kB/s)(6144KiB/771msec) clat (msec): min=16, max=187, avg=78.01, stdev=66.63 lat (msec): min=16, max=187, avg=78.09, stdev=66.62 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 33], 50.00th=[ 33], 60.00th=[ 86], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 188], 95.00th=[ 188], | 99.00th=[ 188], 99.50th=[ 188], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 6144, max= 6144, per=16.52%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=50.00%, 100=16.67%, 250=16.67% cpu : usr=0.13%, sys=15.19%, ctx=48, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32112: Wed May 14 06:45:29 2025 read: IOPS=6, BW=6966KiB/s (7133kB/s)(5120KiB/735msec) clat (usec): min=13387, max=58235, avg=34363.86, stdev=21703.31 lat (usec): min=13388, max=58236, avg=34365.04, stdev=21703.23 clat percentiles (usec): | 1.00th=[13435], 5.00th=[13435], 10.00th=[13435], 20.00th=[13435], | 30.00th=[14484], 40.00th=[14484], 50.00th=[30016], 60.00th=[30016], | 70.00th=[55837], 80.00th=[55837], 90.00th=[58459], 95.00th=[58459], | 99.00th=[58459], 99.50th=[58459], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] write: IOPS=9, BW=9752KiB/s (9986kB/s)(7168KiB/735msec) clat (msec): min=16, max=132, avg=66.68, stdev=56.87 lat (msec): min=16, max=132, avg=66.76, stdev=56.88 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 20], | 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 30], 60.00th=[ 123], | 70.00th=[ 123], 80.00th=[ 128], 90.00th=[ 133], 95.00th=[ 133], | 99.00th=[ 133], 99.50th=[ 133], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 6144, max= 6144, per=16.52%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=25.00%, 100=16.67%, 250=25.00% cpu : usr=0.14%, sys=17.17%, ctx=80, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.9MiB/s (27.2MB/s), 2745KiB/s-9.91MiB/s (2811kB/s-10.4MB/s), io=20.0MiB (20.0MB), run=706-771msec WRITE: bw=36.3MiB/s (38.1MB/s), 7252KiB/s-13.4MiB/s (7426kB/s-14.1MB/s), io=28.0MiB (29.4MB), run=706-771msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32123: Wed May 14 06:45:31 2025 read: IOPS=11, BW=46.9MiB/s (49.2MB/s)(12.0MiB/256msec) clat (msec): min=13, max=127, avg=82.10, stdev=60.66 lat (msec): min=13, max=127, avg=82.10, stdev=60.66 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 14], | 30.00th=[ 14], 40.00th=[ 106], 50.00th=[ 106], 60.00th=[ 106], | 70.00th=[ 128], 80.00th=[ 128], 90.00th=[ 128], 95.00th=[ 128], | 99.00th=[ 128], 99.50th=[ 128], 99.90th=[ 128], 99.95th=[ 128], | 99.99th=[ 128] lat (msec) : 20=33.33%, 250=66.67% cpu : usr=0.00%, sys=45.88%, ctx=1819, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32124: Wed May 14 06:45:31 2025 read: IOPS=1, BW=4888KiB/s (5005kB/s)(4096KiB/838msec) clat (nsec): min=91375k, max=91375k, avg=91374706.00, stdev= 0.00 lat (nsec): min=91377k, max=91377k, avg=91376919.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[91751], 5.00th=[91751], 10.00th=[91751], 20.00th=[91751], | 30.00th=[91751], 40.00th=[91751], 50.00th=[91751], 60.00th=[91751], | 70.00th=[91751], 80.00th=[91751], 90.00th=[91751], 95.00th=[91751], | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 8192, max= 8192, per=41.90%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9776KiB/s (10.0MB/s)(8192KiB/838msec) clat (msec): min=341, max=391, avg=366.58, stdev=35.27 lat (msec): min=341, max=391, avg=366.88, stdev=35.29 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 342], 10.00th=[ 342], 20.00th=[ 342], | 30.00th=[ 342], 40.00th=[ 342], 50.00th=[ 342], 60.00th=[ 393], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 8192, max= 8192, per=20.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.00%, sys=26.40%, ctx=875, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32125: Wed May 14 06:45:31 2025 write: IOPS=3, BW=15.4MiB/s (16.1MB/s)(12.0MiB/781msec) clat (msec): min=159, max=373, avg=256.98, stdev=108.00 lat (msec): min=159, max=373, avg=257.30, stdev=108.00 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 239], 50.00th=[ 239], 60.00th=[ 239], | 70.00th=[ 372], 80.00th=[ 372], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min=16384, max=16384, per=41.90%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=38.33%, ctx=67, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32126: Wed May 14 06:45:31 2025 write: IOPS=4, BW=19.7MiB/s (20.6MB/s)(12.0MiB/610msec) clat (msec): min=163, max=217, avg=199.24, stdev=31.36 lat (msec): min=163, max=218, avg=199.55, stdev=31.34 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 163], 10.00th=[ 163], 20.00th=[ 163], | 30.00th=[ 163], 40.00th=[ 218], 50.00th=[ 218], 60.00th=[ 218], | 70.00th=[ 218], 80.00th=[ 218], 90.00th=[ 218], 95.00th=[ 218], | 99.00th=[ 218], 99.50th=[ 218], 99.90th=[ 218], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min=16384, max=16384, per=41.90%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.33%, sys=57.31%, ctx=19, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.1MiB/s (20.0MB/s), 4888KiB/s-46.9MiB/s (5005kB/s-49.2MB/s), io=16.0MiB (16.8MB), run=256-838msec WRITE: bw=38.2MiB/s (40.0MB/s), 9776KiB/s-19.7MiB/s (10.0MB/s-20.6MB/s), io=32.0MiB (33.6MB), run=610-838msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32119: Wed May 14 06:45:31 2025 read: IOPS=4, BW=18.4MiB/s (19.3MB/s)(12.0MiB/652msec) clat (msec): min=52, max=470, avg=215.25, stdev=223.80 lat (msec): min=52, max=470, avg=215.25, stdev=223.80 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 124], 50.00th=[ 124], 60.00th=[ 124], | 70.00th=[ 472], 80.00th=[ 472], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min=16384, max=16384, per=77.50%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=10.14%, ctx=48, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32120: Wed May 14 06:45:31 2025 read: IOPS=1, BW=5285KiB/s (5412kB/s)(4096KiB/775msec) clat (nsec): min=93002k, max=93002k, avg=93002124.00, stdev= 0.00 lat (nsec): min=93004k, max=93004k, avg=93004249.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[92799], 5.00th=[92799], 10.00th=[92799], 20.00th=[92799], | 30.00th=[92799], 40.00th=[92799], 50.00th=[92799], 60.00th=[92799], | 70.00th=[92799], 80.00th=[92799], 90.00th=[92799], 95.00th=[92799], | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 7571, max= 7571, per=35.81%, avg=7571.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.3MiB/s (10.8MB/s)(8192KiB/775msec) clat (usec): min=333439, max=334697, avg=334068.40, stdev=889.57 lat (usec): min=333743, max=334981, avg=334362.49, stdev=874.89 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 334], 10.00th=[ 334], 20.00th=[ 334], | 30.00th=[ 334], 40.00th=[ 334], 50.00th=[ 334], 60.00th=[ 334], | 70.00th=[ 334], 80.00th=[ 334], 90.00th=[ 334], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 7571, max= 7571, per=19.69%, avg=7571.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.00%, sys=12.66%, ctx=59, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32121: Wed May 14 06:45:31 2025 write: IOPS=3, BW=14.1MiB/s (14.8MB/s)(12.0MiB/852msec) clat (msec): min=241, max=355, avg=281.04, stdev=64.47 lat (msec): min=241, max=355, avg=281.32, stdev=64.49 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 241], 10.00th=[ 241], 20.00th=[ 241], | 30.00th=[ 241], 40.00th=[ 247], 50.00th=[ 247], 60.00th=[ 247], | 70.00th=[ 355], 80.00th=[ 355], 90.00th=[ 355], 95.00th=[ 355], | 99.00th=[ 355], 99.50th=[ 355], 99.90th=[ 355], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min=10966, max=10966, per=28.51%, avg=10966.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.12%, sys=20.80%, ctx=107, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32122: Wed May 14 06:45:31 2025 write: IOPS=7, BW=29.8MiB/s (31.2MB/s)(12.0MiB/403msec) clat (msec): min=125, max=137, avg=131.60, stdev= 6.13 lat (msec): min=125, max=138, avg=131.91, stdev= 6.12 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 126], 10.00th=[ 126], 20.00th=[ 126], | 30.00th=[ 126], 40.00th=[ 132], 50.00th=[ 132], 60.00th=[ 132], | 70.00th=[ 138], 80.00th=[ 138], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] lat (msec) : 250=100.00% cpu : usr=0.25%, sys=22.64%, ctx=55, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=20.6MiB/s (21.6MB/s), 5285KiB/s-18.4MiB/s (5412kB/s-19.3MB/s), io=16.0MiB (16.8MB), run=652-775msec WRITE: bw=37.6MiB/s (39.4MB/s), 10.3MiB/s-29.8MiB/s (10.8MB/s-31.2MB/s), io=32.0MiB (33.6MB), run=403-852msec