== sanity test 398b: DIO and buffer IO race ============== 13:50:05 (1747849805) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.69057 s, 29.8 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11085: Wed May 21 13:50:35 2025 read: IOPS=55, BW=222KiB/s (228kB/s)(5992KiB/26956msec) clat (usec): min=56, max=238835, avg=9995.13, stdev=8681.49 lat (usec): min=57, max=238837, avg=9996.72, stdev=8681.40 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 133], 10.00th=[ 4113], 20.00th=[ 5276], | 30.00th=[ 6325], 40.00th=[ 7504], 50.00th=[ 8717], 60.00th=[ 10028], | 70.00th=[ 11600], 80.00th=[ 13960], 90.00th=[ 17171], 95.00th=[ 20841], | 99.00th=[ 34866], 99.50th=[ 41681], 99.90th=[ 52167], 99.95th=[238027], | 99.99th=[238027] bw ( KiB/s): min= 103, max= 320, per=24.47%, avg=217.81, stdev=52.79, samples=53 iops : min= 25, max= 80, avg=54.13, stdev=13.20, samples=53 write: IOPS=58, BW=234KiB/s (239kB/s)(6296KiB/26956msec) clat (usec): min=497, max=68121, avg=7506.77, stdev=7682.14 lat (usec): min=498, max=68121, avg=7508.12, stdev=7682.32 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 725], 10.00th=[ 775], 20.00th=[ 922], | 30.00th=[ 1319], 40.00th=[ 2835], 50.00th=[ 5604], 60.00th=[ 7439], | 70.00th=[10028], 80.00th=[13304], 90.00th=[17957], 95.00th=[21627], | 99.00th=[36439], 99.50th=[39584], 99.90th=[44827], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 95, max= 375, per=24.65%, avg=229.45, stdev=65.68, samples=53 iops : min= 23, max= 93, avg=57.04, stdev=16.46, samples=53 lat (usec) : 100=1.11%, 250=1.73%, 500=0.20%, 750=3.81%, 1000=8.07% lat (msec) : 2=7.10%, 4=4.39%, 10=38.61%, 20=28.78%, 50=6.12% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.29%, sys=15.28%, ctx=5391, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11086: Wed May 21 13:50:35 2025 read: IOPS=55, BW=221KiB/s (226kB/s)(5940KiB/26862msec) clat (usec): min=65, max=49459, avg=9850.54, stdev=6005.49 lat (usec): min=66, max=49460, avg=9851.47, stdev=6005.53 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 196], 10.00th=[ 3982], 20.00th=[ 5407], | 30.00th=[ 6456], 40.00th=[ 7504], 50.00th=[ 8586], 60.00th=[10028], | 70.00th=[11731], 80.00th=[13960], 90.00th=[17433], 95.00th=[21103], | 99.00th=[30016], 99.50th=[33162], 99.90th=[42730], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 104, max= 359, per=24.63%, avg=219.17, stdev=56.41, samples=53 iops : min= 26, max= 89, avg=54.58, stdev=14.13, samples=53 write: IOPS=59, BW=236KiB/s (242kB/s)(6348KiB/26862msec) clat (usec): min=440, max=249835, avg=7626.95, stdev=9927.27 lat (usec): min=440, max=249835, avg=7628.77, stdev=9927.28 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 734], 10.00th=[ 775], 20.00th=[ 906], | 30.00th=[ 1336], 40.00th=[ 2671], 50.00th=[ 5604], 60.00th=[ 7373], | 70.00th=[ 9634], 80.00th=[ 13173], 90.00th=[ 17695], 95.00th=[ 21890], | 99.00th=[ 32113], 99.50th=[ 41157], 99.90th=[ 67634], 99.95th=[250610], | 99.99th=[250610] bw ( KiB/s): min= 95, max= 360, per=24.89%, avg=231.70, stdev=61.61, samples=53 iops : min= 23, max= 90, avg=57.72, stdev=15.40, samples=53 lat (usec) : 100=1.14%, 250=1.33%, 500=0.13%, 750=4.10%, 1000=8.37% lat (msec) : 2=6.58%, 4=5.92%, 10=38.05%, 20=27.73%, 50=6.48% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.20%, sys=15.47%, ctx=5285, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11087: Wed May 21 13:50:35 2025 read: IOPS=55, BW=223KiB/s (229kB/s)(6028KiB/26977msec) clat (usec): min=60, max=60049, avg=9826.30, stdev=6212.60 lat (usec): min=60, max=60050, avg=9828.70, stdev=6213.12 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 123], 10.00th=[ 4113], 20.00th=[ 5342], | 30.00th=[ 6390], 40.00th=[ 7570], 50.00th=[ 8848], 60.00th=[10159], | 70.00th=[11731], 80.00th=[13829], 90.00th=[16909], 95.00th=[20055], | 99.00th=[31065], 99.50th=[38011], 99.90th=[54264], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 55, max= 368, per=24.73%, avg=220.11, stdev=56.94, samples=53 iops : min= 13, max= 92, avg=54.72, stdev=14.35, samples=53 write: IOPS=58, BW=232KiB/s (238kB/s)(6260KiB/26977msec) clat (usec): min=440, max=240203, avg=7663.58, stdev=9838.91 lat (usec): min=441, max=240206, avg=7665.21, stdev=9839.23 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 725], 10.00th=[ 775], 20.00th=[ 922], | 30.00th=[ 1450], 40.00th=[ 2802], 50.00th=[ 5538], 60.00th=[ 7242], | 70.00th=[ 9503], 80.00th=[ 12911], 90.00th=[ 18482], 95.00th=[ 22676], | 99.00th=[ 33817], 99.50th=[ 45876], 99.90th=[ 57410], 99.95th=[240124], | 99.99th=[240124] bw ( KiB/s): min= 79, max= 360, per=24.44%, avg=227.51, stdev=62.62, samples=53 iops : min= 19, max= 90, avg=56.57, stdev=15.70, samples=53 lat (usec) : 100=1.17%, 250=1.60%, 500=0.26%, 750=3.52%, 1000=8.11% lat (msec) : 2=6.61%, 4=5.79%, 10=38.25%, 20=28.16%, 50=6.32% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.16%, sys=15.22%, ctx=5352, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11088: Wed May 21 13:50:35 2025 read: IOPS=56, BW=224KiB/s (230kB/s)(6052KiB/26964msec) clat (usec): min=43, max=259032, avg=10147.33, stdev=8941.99 lat (usec): min=44, max=259033, avg=10148.63, stdev=8942.30 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 131], 10.00th=[ 4047], 20.00th=[ 5407], | 30.00th=[ 6390], 40.00th=[ 7504], 50.00th=[ 8717], 60.00th=[ 10159], | 70.00th=[ 11994], 80.00th=[ 14222], 90.00th=[ 18220], 95.00th=[ 21103], | 99.00th=[ 31065], 99.50th=[ 36439], 99.90th=[ 51119], 99.95th=[258999], | 99.99th=[258999] bw ( KiB/s): min= 87, max= 360, per=24.86%, avg=221.25, stdev=58.82, samples=53 iops : min= 21, max= 90, avg=55.08, stdev=14.84, samples=53 write: IOPS=57, BW=231KiB/s (237kB/s)(6236KiB/26964msec) clat (usec): min=482, max=49782, avg=7364.51, stdev=7617.05 lat (usec): min=483, max=49782, avg=7366.13, stdev=7617.10 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 717], 10.00th=[ 775], 20.00th=[ 914], | 30.00th=[ 1369], 40.00th=[ 2638], 50.00th=[ 5342], 60.00th=[ 7046], | 70.00th=[ 9372], 80.00th=[13173], 90.00th=[17957], 95.00th=[22414], | 99.00th=[33817], 99.50th=[38536], 99.90th=[46400], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 152, max= 352, per=24.49%, avg=228.04, stdev=56.26, samples=53 iops : min= 38, max= 88, avg=56.77, stdev=14.12, samples=53 lat (usec) : 50=0.07%, 100=1.04%, 250=1.73%, 500=0.03%, 750=4.20% lat (usec) : 1000=7.88% lat (msec) : 2=6.93%, 4=5.50%, 10=38.12%, 20=27.44%, 50=7.00% lat (msec) : 100=0.03%, 500=0.03% cpu : usr=0.18%, sys=15.13%, ctx=5237, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=890KiB/s (911kB/s), 221KiB/s-224KiB/s (226kB/s-230kB/s), io=23.4MiB (24.6MB), run=26862-26977msec WRITE: bw=932KiB/s (954kB/s), 231KiB/s-236KiB/s (237kB/s-242kB/s), io=24.6MiB (25.7MB), run=26862-26977msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11081: Wed May 21 13:50:43 2025 read: IOPS=42, BW=170KiB/s (174kB/s)(5992KiB/35185msec) clat (msec): min=2, max=259, avg= 9.07, stdev=10.80 lat (msec): min=2, max=259, avg= 9.07, stdev=10.80 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 13], 90.00th=[ 21], 95.00th=[ 28], | 99.00th=[ 44], 99.50th=[ 53], 99.90th=[ 85], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 48, max= 431, per=24.89%, avg=168.99, stdev=110.41, samples=70 iops : min= 12, max= 107, avg=41.96, stdev=27.68, samples=70 write: IOPS=44, BW=179KiB/s (183kB/s)(6296KiB/35185msec) clat (usec): min=3656, max=93676, avg=13632.22, stdev=9961.61 lat (usec): min=3657, max=93677, avg=13633.39, stdev=9961.61 clat percentiles (usec): | 1.00th=[ 4293], 5.00th=[ 5145], 10.00th=[ 5735], 20.00th=[ 6325], | 30.00th=[ 6980], 40.00th=[ 7963], 50.00th=[ 9634], 60.00th=[12387], | 70.00th=[15795], 80.00th=[20055], 90.00th=[26870], 95.00th=[32637], | 99.00th=[48497], 99.50th=[56886], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 32, max= 448, per=24.96%, avg=177.44, stdev=112.87, samples=70 iops : min= 8, max= 112, avg=44.07, stdev=28.32, samples=70 lat (msec) : 4=16.18%, 10=46.26%, 20=22.01%, 50=14.84%, 100=0.68% lat (msec) : 500=0.03% cpu : usr=0.15%, sys=7.86%, ctx=3605, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11082: Wed May 21 13:50:43 2025 read: IOPS=42, BW=168KiB/s (172kB/s)(5940KiB/35316msec) clat (msec): min=2, max=266, avg= 8.94, stdev=10.79 lat (msec): min=2, max=266, avg= 8.94, stdev=10.79 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 13], 90.00th=[ 20], 95.00th=[ 28], | 99.00th=[ 42], 99.50th=[ 48], 99.90th=[ 79], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 16, max= 472, per=24.62%, avg=167.14, stdev=115.63, samples=70 iops : min= 4, max= 118, avg=41.66, stdev=28.88, samples=70 write: IOPS=44, BW=180KiB/s (184kB/s)(6348KiB/35316msec) clat (usec): min=3724, max=85084, avg=13800.62, stdev=10276.65 lat (usec): min=3726, max=85085, avg=13802.20, stdev=10277.05 clat percentiles (usec): | 1.00th=[ 4293], 5.00th=[ 5080], 10.00th=[ 5604], 20.00th=[ 6194], | 30.00th=[ 6849], 40.00th=[ 7832], 50.00th=[ 9241], 60.00th=[11994], | 70.00th=[16319], 80.00th=[21365], 90.00th=[28181], 95.00th=[34341], | 99.00th=[49546], 99.50th=[56886], 99.90th=[71828], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 56, max= 440, per=24.91%, avg=177.09, stdev=114.30, samples=70 iops : min= 14, max= 110, avg=44.14, stdev=28.58, samples=70 lat (msec) : 4=16.34%, 10=47.01%, 20=20.35%, 50=15.72%, 100=0.55% lat (msec) : 500=0.03% cpu : usr=0.20%, sys=7.75%, ctx=3580, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11083: Wed May 21 13:50:43 2025 read: IOPS=42, BW=171KiB/s (175kB/s)(6028KiB/35288msec) clat (usec): min=2277, max=83425, avg=8997.01, stdev=8836.29 lat (usec): min=2278, max=83426, avg=8998.06, stdev=8836.22 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2835], 10.00th=[ 3097], 20.00th=[ 3425], | 30.00th=[ 3818], 40.00th=[ 4424], 50.00th=[ 5276], 60.00th=[ 6587], | 70.00th=[ 8848], 80.00th=[12911], 90.00th=[21365], 95.00th=[26608], | 99.00th=[40633], 99.50th=[48497], 99.90th=[77071], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 40, max= 448, per=24.90%, avg=169.07, stdev=107.94, samples=70 iops : min= 10, max= 112, avg=42.04, stdev=27.04, samples=70 write: IOPS=44, BW=177KiB/s (182kB/s)(6260KiB/35288msec) clat (msec): min=3, max=271, avg=13.80, stdev=11.90 lat (msec): min=3, max=271, avg=13.80, stdev=11.90 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 10], 60.00th=[ 13], | 70.00th=[ 17], 80.00th=[ 21], 90.00th=[ 27], 95.00th=[ 33], | 99.00th=[ 47], 99.50th=[ 59], 99.90th=[ 95], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 40, max= 432, per=24.63%, avg=175.13, stdev=116.02, samples=70 iops : min= 10, max= 108, avg=43.56, stdev=29.06, samples=70 lat (msec) : 4=16.73%, 10=46.03%, 20=20.74%, 50=15.98%, 100=0.49% lat (msec) : 500=0.03% cpu : usr=0.18%, sys=7.70%, ctx=3608, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11084: Wed May 21 13:50:43 2025 read: IOPS=42, BW=172KiB/s (176kB/s)(6052KiB/35192msec) clat (usec): min=2252, max=87990, avg=8918.14, stdev=9072.34 lat (usec): min=2253, max=87991, avg=8919.60, stdev=9072.32 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2868], 10.00th=[ 3064], 20.00th=[ 3425], | 30.00th=[ 3851], 40.00th=[ 4424], 50.00th=[ 5211], 60.00th=[ 6521], | 70.00th=[ 8455], 80.00th=[12649], 90.00th=[20317], 95.00th=[27919], | 99.00th=[42730], 99.50th=[52167], 99.90th=[86508], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 24, max= 456, per=25.07%, avg=170.21, stdev=114.41, samples=70 iops : min= 6, max= 114, avg=42.33, stdev=28.71, samples=70 write: IOPS=44, BW=177KiB/s (181kB/s)(6236KiB/35192msec) clat (msec): min=3, max=247, avg=13.82, stdev=11.45 lat (msec): min=3, max=247, avg=13.82, stdev=11.45 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 10], 60.00th=[ 13], | 70.00th=[ 17], 80.00th=[ 21], 90.00th=[ 28], 95.00th=[ 33], | 99.00th=[ 45], 99.50th=[ 55], 99.90th=[ 94], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min= 47, max= 456, per=24.76%, avg=176.04, stdev=114.90, samples=70 iops : min= 11, max= 114, avg=43.79, stdev=28.83, samples=70 lat (msec) : 4=16.41%, 10=46.71%, 20=20.70%, 50=15.59%, 100=0.55% lat (msec) : 250=0.03% cpu : usr=0.09%, sys=7.91%, ctx=3570, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=680KiB/s (696kB/s), 168KiB/s-172KiB/s (172kB/s-176kB/s), io=23.4MiB (24.6MB), run=35185-35316msec WRITE: bw=712KiB/s (729kB/s), 177KiB/s-180KiB/s (181kB/s-184kB/s), io=24.6MiB (25.7MB), run=35185-35316msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11142: Wed May 21 13:50:49 2025 read: IOPS=76, BW=1229KiB/s (1258kB/s)(5920KiB/4817msec) clat (usec): min=64, max=59727, avg=7200.89, stdev=5376.57 lat (usec): min=65, max=59729, avg=7201.76, stdev=5376.57 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 106], 10.00th=[ 122], 20.00th=[ 3720], | 30.00th=[ 4817], 40.00th=[ 5669], 50.00th=[ 6652], 60.00th=[ 8094], | 70.00th=[ 9372], 80.00th=[10421], 90.00th=[12256], 95.00th=[15533], | 99.00th=[23200], 99.50th=[25822], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 896, max= 1469, per=24.83%, avg=1187.22, stdev=184.95, samples=9 iops : min= 56, max= 91, avg=74.11, stdev=11.41, samples=9 write: IOPS=82, BW=1322KiB/s (1354kB/s)(6368KiB/4817msec) clat (usec): min=670, max=35466, avg=5365.76, stdev=5318.15 lat (usec): min=671, max=35467, avg=5369.52, stdev=5316.48 clat percentiles (usec): | 1.00th=[ 889], 5.00th=[ 1020], 10.00th=[ 1090], 20.00th=[ 1205], | 30.00th=[ 1369], 40.00th=[ 1778], 50.00th=[ 3654], 60.00th=[ 5014], | 70.00th=[ 6325], 80.00th=[ 8979], 90.00th=[12780], 95.00th=[15533], | 99.00th=[21890], 99.50th=[34341], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 768, max= 1664, per=25.65%, avg=1290.33, stdev=287.29, samples=9 iops : min= 48, max= 104, avg=80.56, stdev=18.01, samples=9 lat (usec) : 100=1.69%, 250=5.34%, 500=0.13%, 750=0.13%, 1000=1.43% lat (msec) : 2=20.44%, 4=8.72%, 10=40.76%, 20=19.79%, 50=1.43% lat (msec) : 100=0.13% cpu : usr=0.37%, sys=20.39%, ctx=1369, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11143: Wed May 21 13:50:49 2025 read: IOPS=74, BW=1190KiB/s (1219kB/s)(5744KiB/4825msec) clat (usec): min=61, max=44524, avg=7135.20, stdev=5375.95 lat (usec): min=62, max=44525, avg=7136.54, stdev=5376.46 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 97], 10.00th=[ 117], 20.00th=[ 4015], | 30.00th=[ 4817], 40.00th=[ 5669], 50.00th=[ 6456], 60.00th=[ 7570], | 70.00th=[ 8717], 80.00th=[10290], 90.00th=[12518], 95.00th=[15664], | 99.00th=[28967], 99.50th=[30278], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 864, max= 1472, per=24.83%, avg=1186.89, stdev=201.42, samples=9 iops : min= 54, max= 92, avg=74.00, stdev=12.60, samples=9 write: IOPS=84, BW=1356KiB/s (1389kB/s)(6544KiB/4825msec) clat (usec): min=664, max=31380, avg=5436.28, stdev=5737.22 lat (usec): min=665, max=31381, avg=5437.57, stdev=5737.21 clat percentiles (usec): | 1.00th=[ 889], 5.00th=[ 988], 10.00th=[ 1074], 20.00th=[ 1172], | 30.00th=[ 1303], 40.00th=[ 1778], 50.00th=[ 3130], 60.00th=[ 4686], | 70.00th=[ 6259], 80.00th=[ 8979], 90.00th=[13698], 95.00th=[19006], | 99.00th=[25560], 99.50th=[26084], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 864, max= 1756, per=26.42%, avg=1329.00, stdev=242.76, samples=9 iops : min= 54, max= 109, avg=82.89, stdev=14.98, samples=9 lat (usec) : 100=2.99%, 250=4.30%, 500=0.13%, 750=0.13%, 1000=2.73% lat (msec) : 2=19.27%, 4=8.33%, 10=42.58%, 20=16.80%, 50=2.73% cpu : usr=0.19%, sys=20.67%, ctx=1315, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11144: Wed May 21 13:50:49 2025 read: IOPS=78, BW=1249KiB/s (1279kB/s)(6256KiB/5009msec) clat (usec): min=66, max=28552, avg=7216.92, stdev=4519.50 lat (usec): min=66, max=28553, avg=7217.72, stdev=4519.53 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 102], 10.00th=[ 137], 20.00th=[ 4146], | 30.00th=[ 5080], 40.00th=[ 5800], 50.00th=[ 6915], 60.00th=[ 7963], | 70.00th=[ 8979], 80.00th=[10421], 90.00th=[12780], 95.00th=[14484], | 99.00th=[22676], 99.50th=[26608], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 928, max= 1632, per=26.10%, avg=1247.70, stdev=223.53, samples=10 iops : min= 58, max= 102, avg=77.90, stdev=14.05, samples=10 write: IOPS=75, BW=1204KiB/s (1233kB/s)(6032KiB/5009msec) clat (usec): min=820, max=38868, avg=5719.26, stdev=5734.71 lat (usec): min=821, max=38870, avg=5720.86, stdev=5734.48 clat percentiles (usec): | 1.00th=[ 955], 5.00th=[ 1057], 10.00th=[ 1123], 20.00th=[ 1254], | 30.00th=[ 1500], 40.00th=[ 2057], 50.00th=[ 3982], 60.00th=[ 5407], | 70.00th=[ 6783], 80.00th=[ 9110], 90.00th=[13173], 95.00th=[17433], | 99.00th=[27132], 99.50th=[34866], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 864, max= 1536, per=23.73%, avg=1193.40, stdev=232.51, samples=10 iops : min= 54, max= 96, avg=74.50, stdev=14.65, samples=10 lat (usec) : 100=2.34%, 250=3.65%, 500=0.13%, 750=0.26%, 1000=1.30% lat (msec) : 2=17.84%, 4=8.85%, 10=45.57%, 20=17.97%, 50=2.08% cpu : usr=0.14%, sys=20.71%, ctx=1440, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11145: Wed May 21 13:50:49 2025 read: IOPS=75, BW=1206KiB/s (1235kB/s)(6032KiB/5003msec) clat (usec): min=61, max=32334, avg=6986.74, stdev=4990.94 lat (usec): min=62, max=32334, avg=6988.47, stdev=4991.15 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 99], 10.00th=[ 112], 20.00th=[ 3785], | 30.00th=[ 4817], 40.00th=[ 5604], 50.00th=[ 6456], 60.00th=[ 7504], | 70.00th=[ 8586], 80.00th=[10028], 90.00th=[12780], 95.00th=[15008], | 99.00th=[26346], 99.50th=[27919], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 864, max= 1472, per=24.97%, avg=1193.60, stdev=174.00, samples=10 iops : min= 54, max= 92, avg=74.60, stdev=10.88, samples=10 write: IOPS=78, BW=1250KiB/s (1280kB/s)(6256KiB/5003msec) clat (usec): min=735, max=37489, avg=5943.33, stdev=5931.27 lat (usec): min=736, max=37489, avg=5944.63, stdev=5931.26 clat percentiles (usec): | 1.00th=[ 816], 5.00th=[ 996], 10.00th=[ 1090], 20.00th=[ 1205], | 30.00th=[ 1352], 40.00th=[ 1729], 50.00th=[ 3818], 60.00th=[ 5604], | 70.00th=[ 7373], 80.00th=[10683], 90.00th=[14746], 95.00th=[17695], | 99.00th=[23725], 99.50th=[25560], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 768, max= 1664, per=24.87%, avg=1251.20, stdev=261.91, samples=10 iops : min= 48, max= 104, avg=78.20, stdev=16.37, samples=10 lat (usec) : 100=2.86%, 250=4.69%, 500=0.13%, 750=0.13%, 1000=2.60% lat (msec) : 2=19.14%, 4=6.64%, 10=42.45%, 20=18.36%, 50=2.99% cpu : usr=0.12%, sys=19.91%, ctx=1377, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4782KiB/s (4897kB/s), 1190KiB/s-1249KiB/s (1219kB/s-1279kB/s), io=23.4MiB (24.5MB), run=4817-5009msec WRITE: bw=5031KiB/s (5152kB/s), 1204KiB/s-1356KiB/s (1233kB/s-1389kB/s), io=24.6MiB (25.8MB), run=4817-5009msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11146: Wed May 21 13:50:52 2025 read: IOPS=50, BW=804KiB/s (823kB/s)(5920KiB/7366msec) clat (usec): min=2640, max=40374, avg=7731.99, stdev=6165.12 lat (usec): min=2641, max=40375, avg=7732.95, stdev=6165.10 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3261], 10.00th=[ 3556], 20.00th=[ 3916], | 30.00th=[ 4359], 40.00th=[ 4817], 50.00th=[ 5342], 60.00th=[ 6128], | 70.00th=[ 7373], 80.00th=[ 9241], 90.00th=[16319], 95.00th=[22414], | 99.00th=[33817], 99.50th=[36439], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 447, max= 1248, per=23.65%, avg=765.50, stdev=268.58, samples=14 iops : min= 27, max= 78, avg=47.71, stdev=16.89, samples=14 write: IOPS=54, BW=865KiB/s (885kB/s)(6368KiB/7366msec) clat (usec): min=4493, max=38678, avg=11237.35, stdev=6195.56 lat (usec): min=4494, max=38680, avg=11239.80, stdev=6195.40 clat percentiles (usec): | 1.00th=[ 4752], 5.00th=[ 5800], 10.00th=[ 6390], 20.00th=[ 6915], | 30.00th=[ 7439], 40.00th=[ 8029], 50.00th=[ 8848], 60.00th=[ 9896], | 70.00th=[11863], 80.00th=[14746], 90.00th=[20579], 95.00th=[23987], | 99.00th=[35390], 99.50th=[38536], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 448, max= 1440, per=24.55%, avg=836.29, stdev=332.04, samples=14 iops : min= 28, max= 90, avg=52.14, stdev=20.85, samples=14 lat (msec) : 4=10.68%, 10=60.29%, 20=20.18%, 50=8.85% cpu : usr=0.00%, sys=10.12%, ctx=899, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11147: Wed May 21 13:50:52 2025 read: IOPS=48, BW=777KiB/s (796kB/s)(5744KiB/7389msec) clat (usec): min=2651, max=37466, avg=7292.72, stdev=5948.45 lat (usec): min=2652, max=37467, avg=7293.85, stdev=5948.44 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3130], 10.00th=[ 3490], 20.00th=[ 3818], | 30.00th=[ 4228], 40.00th=[ 4555], 50.00th=[ 5014], 60.00th=[ 5538], | 70.00th=[ 6325], 80.00th=[ 9372], 90.00th=[15401], 95.00th=[21103], | 99.00th=[31327], 99.50th=[35914], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 319, max= 1312, per=23.28%, avg=753.71, stdev=317.06, samples=14 iops : min= 19, max= 82, avg=46.86, stdev=19.83, samples=14 write: IOPS=55, BW=886KiB/s (907kB/s)(6544KiB/7389msec) clat (usec): min=3307, max=42827, avg=11479.07, stdev=6436.81 lat (usec): min=3309, max=42829, avg=11480.89, stdev=6436.78 clat percentiles (usec): | 1.00th=[ 4948], 5.00th=[ 5669], 10.00th=[ 6128], 20.00th=[ 6915], | 30.00th=[ 7635], 40.00th=[ 8455], 50.00th=[ 9241], 60.00th=[10159], | 70.00th=[12125], 80.00th=[15270], 90.00th=[21103], 95.00th=[24773], | 99.00th=[37487], 99.50th=[38536], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 480, max= 1405, per=24.67%, avg=840.43, stdev=305.53, samples=14 iops : min= 30, max= 87, avg=52.29, stdev=18.91, samples=14 lat (msec) : 4=11.46%, 10=57.94%, 20=21.87%, 50=8.72% cpu : usr=0.18%, sys=9.73%, ctx=905, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11148: Wed May 21 13:50:52 2025 read: IOPS=52, BW=846KiB/s (866kB/s)(6256KiB/7398msec) clat (usec): min=2593, max=39294, avg=7150.33, stdev=5455.77 lat (usec): min=2594, max=39295, avg=7151.33, stdev=5455.74 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 3294], 10.00th=[ 3556], 20.00th=[ 3851], | 30.00th=[ 4146], 40.00th=[ 4686], 50.00th=[ 5211], 60.00th=[ 5800], | 70.00th=[ 7046], 80.00th=[ 8848], 90.00th=[12387], 95.00th=[19268], | 99.00th=[29754], 99.50th=[36439], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 416, max= 1628, per=25.04%, avg=810.57, stdev=363.70, samples=14 iops : min= 26, max= 101, avg=50.43, stdev=22.45, samples=14 write: IOPS=50, BW=815KiB/s (835kB/s)(6032KiB/7398msec) clat (usec): min=3836, max=42193, avg=11993.27, stdev=6694.95 lat (usec): min=3838, max=42195, avg=11995.19, stdev=6695.05 clat percentiles (usec): | 1.00th=[ 4817], 5.00th=[ 5669], 10.00th=[ 6325], 20.00th=[ 7046], | 30.00th=[ 7767], 40.00th=[ 8455], 50.00th=[ 9503], 60.00th=[10814], | 70.00th=[13304], 80.00th=[17171], 90.00th=[21627], 95.00th=[26084], | 99.00th=[36963], 99.50th=[39584], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 415, max= 1277, per=22.73%, avg=774.14, stdev=271.25, samples=14 iops : min= 25, max= 79, avg=48.14, stdev=16.78, samples=14 lat (msec) : 4=13.15%, 10=55.60%, 20=23.05%, 50=8.20% cpu : usr=0.20%, sys=9.46%, ctx=921, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11149: Wed May 21 13:50:52 2025 read: IOPS=51, BW=818KiB/s (838kB/s)(6032KiB/7374msec) clat (usec): min=2815, max=40258, avg=7729.81, stdev=6261.74 lat (usec): min=2816, max=40259, avg=7731.03, stdev=6261.53 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3130], 10.00th=[ 3425], 20.00th=[ 3851], | 30.00th=[ 4293], 40.00th=[ 4752], 50.00th=[ 5276], 60.00th=[ 5932], | 70.00th=[ 7177], 80.00th=[10028], 90.00th=[17433], 95.00th=[22152], | 99.00th=[33817], 99.50th=[37487], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 415, max= 1469, per=23.98%, avg=776.36, stdev=294.92, samples=14 iops : min= 25, max= 91, avg=48.21, stdev=18.34, samples=14 write: IOPS=53, BW=848KiB/s (869kB/s)(6256KiB/7374msec) clat (usec): min=4280, max=42860, avg=11195.09, stdev=6133.53 lat (usec): min=4281, max=42861, avg=11199.68, stdev=6132.13 clat percentiles (usec): | 1.00th=[ 4621], 5.00th=[ 5473], 10.00th=[ 6063], 20.00th=[ 6980], | 30.00th=[ 7504], 40.00th=[ 8225], 50.00th=[ 8979], 60.00th=[10290], | 70.00th=[11863], 80.00th=[14877], 90.00th=[18744], 95.00th=[22676], | 99.00th=[35914], 99.50th=[42730], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 479, max= 1405, per=24.00%, avg=817.50, stdev=328.00, samples=14 iops : min= 29, max= 87, avg=50.79, stdev=20.47, samples=14 lat (msec) : 4=11.72%, 10=57.16%, 20=23.83%, 50=7.29% cpu : usr=0.05%, sys=9.79%, ctx=894, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3238KiB/s (3315kB/s), 777KiB/s-846KiB/s (796kB/s-866kB/s), io=23.4MiB (24.5MB), run=7366-7398msec WRITE: bw=3406KiB/s (3488kB/s), 815KiB/s-886KiB/s (835kB/s-907kB/s), io=24.6MiB (25.8MB), run=7366-7398msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11159: Wed May 21 13:50:53 2025 read: IOPS=10, BW=10.4MiB/s (10.9MB/s)(7168KiB/674msec) clat (usec): min=474, max=120255, avg=37724.53, stdev=42405.86 lat (usec): min=476, max=120256, avg=37725.72, stdev=42405.60 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 474], 10.00th=[ 474], 20.00th=[ 523], | 30.00th=[ 15795], 40.00th=[ 15795], 50.00th=[ 21365], 60.00th=[ 51119], | 70.00th=[ 51119], 80.00th=[ 54264], 90.00th=[120062], 95.00th=[120062], | 99.00th=[120062], 99.50th=[120062], 99.90th=[120062], 99.95th=[120062], | 99.99th=[120062] bw ( KiB/s): min=10240, max=10240, per=33.70%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7596KiB/s (7779kB/s)(5120KiB/674msec) clat (msec): min=27, max=142, avg=70.54, stdev=48.07 lat (msec): min=27, max=143, avg=70.61, stdev=48.07 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 28], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 52], 60.00th=[ 52], | 70.00th=[ 95], 80.00th=[ 95], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 4096, max= 4096, per=9.63%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 20=8.33%, 50=25.00%, 100=33.33%, 250=16.67% cpu : usr=0.15%, sys=23.03%, ctx=299, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11160: Wed May 21 13:50:53 2025 read: IOPS=3, BW=3346KiB/s (3427kB/s)(2048KiB/612msec) clat (usec): min=41884, max=54245, avg=48065.26, stdev=8740.59 lat (usec): min=41887, max=54246, avg=48066.78, stdev=8739.33 clat percentiles (usec): | 1.00th=[41681], 5.00th=[41681], 10.00th=[41681], 20.00th=[41681], | 30.00th=[41681], 40.00th=[41681], 50.00th=[41681], 60.00th=[54264], | 70.00th=[54264], 80.00th=[54264], 90.00th=[54264], 95.00th=[54264], | 99.00th=[54264], 99.50th=[54264], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 4096, max= 4096, per=13.48%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=16, BW=16.3MiB/s (17.1MB/s)(10.0MiB/612msec) clat (msec): min=24, max=112, avg=50.65, stdev=25.40 lat (msec): min=24, max=113, avg=50.82, stdev=25.68 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 32], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 41], 60.00th=[ 49], | 70.00th=[ 62], 80.00th=[ 62], 90.00th=[ 65], 95.00th=[ 113], | 99.00th=[ 113], 99.50th=[ 113], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min=16384, max=16384, per=38.51%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 16, max= 16, avg=16.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=33.33%, 250=8.33% cpu : usr=0.33%, sys=46.64%, ctx=422, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11161: Wed May 21 13:50:53 2025 read: IOPS=10, BW=10.4MiB/s (10.9MB/s)(6144KiB/576msec) clat (usec): min=501, max=76237, avg=21997.78, stdev=33822.05 lat (usec): min=502, max=76238, avg=21999.13, stdev=33821.79 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 502], 10.00th=[ 502], 20.00th=[ 506], | 30.00th=[ 506], 40.00th=[ 529], 50.00th=[ 529], 60.00th=[ 1221], | 70.00th=[53216], 80.00th=[53216], 90.00th=[76022], 95.00th=[76022], | 99.00th=[76022], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 8192, max= 8192, per=26.96%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.4MiB/s (10.9MB/s)(6144KiB/576msec) clat (msec): min=24, max=111, avg=68.06, stdev=40.69 lat (msec): min=24, max=112, avg=68.14, stdev=40.71 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 94], | 70.00th=[ 108], 80.00th=[ 108], 90.00th=[ 112], 95.00th=[ 112], | 99.00th=[ 112], 99.50th=[ 112], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 8192, max= 8192, per=19.26%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 750=25.00% lat (msec) : 2=8.33%, 50=25.00%, 100=25.00%, 250=16.67% cpu : usr=0.00%, sys=25.39%, ctx=342, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11162: Wed May 21 13:50:53 2025 read: IOPS=9, BW=9377KiB/s (9602kB/s)(5120KiB/546msec) clat (usec): min=453, max=49761, avg=20326.85, stdev=21635.43 lat (usec): min=455, max=49763, avg=20328.08, stdev=21635.52 clat percentiles (usec): | 1.00th=[ 453], 5.00th=[ 453], 10.00th=[ 453], 20.00th=[ 453], | 30.00th=[ 490], 40.00th=[ 490], 50.00th=[16450], 60.00th=[16450], | 70.00th=[34341], 80.00th=[34341], 90.00th=[49546], 95.00th=[49546], | 99.00th=[49546], 99.50th=[49546], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 8192, max= 8192, per=26.96%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.8MiB/s (13.4MB/s)(7168KiB/546msec) clat (msec): min=25, max=103, avg=55.83, stdev=27.38 lat (msec): min=25, max=103, avg=55.89, stdev=27.39 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 34], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 58], 60.00th=[ 58], | 70.00th=[ 58], 80.00th=[ 78], 90.00th=[ 104], 95.00th=[ 104], | 99.00th=[ 104], 99.50th=[ 104], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min=12288, max=12288, per=28.89%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 20=8.33%, 50=41.67%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=30.64%, ctx=281, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=29.7MiB/s (31.1MB/s), 3346KiB/s-10.4MiB/s (3427kB/s-10.9MB/s), io=20.0MiB (20.0MB), run=546-674msec WRITE: bw=41.5MiB/s (43.6MB/s), 7596KiB/s-16.3MiB/s (7779kB/s-17.1MB/s), io=28.0MiB (29.4MB), run=546-674msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11164: Wed May 21 13:50:54 2025 read: IOPS=7, BW=7741KiB/s (7927kB/s)(7168KiB/926msec) clat (msec): min=21, max=174, avg=81.49, stdev=62.66 lat (msec): min=21, max=174, avg=81.49, stdev=62.66 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 26], | 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 56], 60.00th=[ 122], | 70.00th=[ 122], 80.00th=[ 142], 90.00th=[ 176], 95.00th=[ 176], | 99.00th=[ 176], 99.50th=[ 176], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 6144, max= 6144, per=27.81%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5529KiB/s (5662kB/s)(5120KiB/926msec) clat (usec): min=25858, max=62319, avg=44990.73, stdev=15859.59 lat (usec): min=25936, max=62384, avg=45063.94, stdev=15858.43 clat percentiles (usec): | 1.00th=[25822], 5.00th=[25822], 10.00th=[25822], 20.00th=[25822], | 30.00th=[31065], 40.00th=[31065], 50.00th=[49546], 60.00th=[49546], | 70.00th=[56361], 80.00th=[56361], 90.00th=[62129], 95.00th=[62129], | 99.00th=[62129], 99.50th=[62129], 99.90th=[62129], 99.95th=[62129], | 99.99th=[62129] lat (msec) : 50=50.00%, 100=25.00%, 250=25.00% cpu : usr=0.11%, sys=13.51%, ctx=80, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11165: Wed May 21 13:50:54 2025 read: IOPS=2, BW=2263KiB/s (2317kB/s)(2048KiB/905msec) clat (msec): min=35, max=127, avg=81.44, stdev=65.67 lat (msec): min=35, max=127, avg=81.45, stdev=65.67 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 35], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 35], 60.00th=[ 128], | 70.00th=[ 128], 80.00th=[ 128], 90.00th=[ 128], 95.00th=[ 128], | 99.00th=[ 128], 99.50th=[ 128], 99.90th=[ 128], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 2043, max= 2043, per=9.25%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.0MiB/s (11.6MB/s)(10.0MiB/905msec) clat (msec): min=38, max=171, avg=68.43, stdev=41.16 lat (msec): min=38, max=171, avg=68.51, stdev=41.15 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 40], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 45], 60.00th=[ 53], | 70.00th=[ 83], 80.00th=[ 83], 90.00th=[ 91], 95.00th=[ 171], | 99.00th=[ 171], 99.50th=[ 171], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 6131, max= 6131, per=19.82%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=33.33%, 250=16.67% cpu : usr=0.00%, sys=12.39%, ctx=62, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11166: Wed May 21 13:50:54 2025 read: IOPS=6, BW=6628KiB/s (6787kB/s)(6144KiB/927msec) clat (usec): min=21933, max=80756, avg=42781.72, stdev=21639.37 lat (usec): min=21935, max=80757, avg=42783.21, stdev=21639.14 clat percentiles (usec): | 1.00th=[21890], 5.00th=[21890], 10.00th=[21890], 20.00th=[26346], | 30.00th=[26346], 40.00th=[31851], 50.00th=[31851], 60.00th=[44303], | 70.00th=[51119], 80.00th=[51119], 90.00th=[81265], 95.00th=[81265], | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 2048, max= 2048, per=9.27%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6628KiB/s (6787kB/s)(6144KiB/927msec) clat (msec): min=36, max=179, avg=90.96, stdev=57.85 lat (msec): min=36, max=179, avg=91.04, stdev=57.85 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 68], 50.00th=[ 68], 60.00th=[ 70], | 70.00th=[ 146], 80.00th=[ 146], 90.00th=[ 180], 95.00th=[ 180], | 99.00th=[ 180], 99.50th=[ 180], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 6144, max= 6144, per=19.86%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=33.33%, 250=16.67% cpu : usr=0.00%, sys=9.50%, ctx=66, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11167: Wed May 21 13:50:54 2025 read: IOPS=5, BW=5651KiB/s (5787kB/s)(5120KiB/906msec) clat (msec): min=21, max=110, avg=55.27, stdev=37.66 lat (msec): min=21, max=110, avg=55.28, stdev=37.66 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 27], 40.00th=[ 27], 50.00th=[ 42], 60.00th=[ 42], | 70.00th=[ 77], 80.00th=[ 77], 90.00th=[ 111], 95.00th=[ 111], | 99.00th=[ 111], 99.50th=[ 111], 99.90th=[ 111], 99.95th=[ 111], | 99.99th=[ 111] write: IOPS=7, BW=7912KiB/s (8102kB/s)(7168KiB/906msec) clat (msec): min=37, max=125, avg=75.87, stdev=36.53 lat (msec): min=37, max=125, avg=75.94, stdev=36.52 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 41], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 75], 60.00th=[ 102], | 70.00th=[ 102], 80.00th=[ 109], 90.00th=[ 126], 95.00th=[ 126], | 99.00th=[ 126], 99.50th=[ 126], 99.90th=[ 126], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 6144, max= 6144, per=19.86%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=16.67%, 250=33.33% cpu : usr=0.00%, sys=14.81%, ctx=96, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=21.6MiB/s (22.6MB/s), 2263KiB/s-7741KiB/s (2317kB/s-7927kB/s), io=20.0MiB (20.0MB), run=905-927msec WRITE: bw=30.2MiB/s (31.7MB/s), 5529KiB/s-11.0MiB/s (5662kB/s-11.6MB/s), io=28.0MiB (29.4MB), run=905-927msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11175: Wed May 21 13:50:56 2025 read: IOPS=6, BW=27.3MiB/s (28.7MB/s)(12.0MiB/439msec) clat (msec): min=68, max=177, avg=141.01, stdev=62.69 lat (msec): min=68, max=177, avg=141.02, stdev=62.69 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 69], 10.00th=[ 69], 20.00th=[ 69], | 30.00th=[ 69], 40.00th=[ 178], 50.00th=[ 178], 60.00th=[ 178], | 70.00th=[ 178], 80.00th=[ 178], 90.00th=[ 178], 95.00th=[ 178], | 99.00th=[ 178], 99.50th=[ 178], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.91%, sys=32.65%, ctx=1457, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11177: Wed May 21 13:50:56 2025 read: IOPS=1, BW=5278KiB/s (5405kB/s)(4096KiB/776msec) clat (nsec): min=104068k, max=104068k, avg=104068246.00, stdev= 0.00 lat (nsec): min=104072k, max=104072k, avg=104071531.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 105], 10.00th=[ 105], 20.00th=[ 105], | 30.00th=[ 105], 40.00th=[ 105], 50.00th=[ 105], 60.00th=[ 105], | 70.00th=[ 105], 80.00th=[ 105], 90.00th=[ 105], 95.00th=[ 105], | 99.00th=[ 105], 99.50th=[ 105], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 8192, max= 8192, per=38.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.3MiB/s (10.8MB/s)(8192KiB/776msec) clat (msec): min=230, max=406, avg=318.58, stdev=124.83 lat (msec): min=230, max=407, avg=318.96, stdev=124.57 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 230], 10.00th=[ 230], 20.00th=[ 230], | 30.00th=[ 230], 40.00th=[ 230], 50.00th=[ 230], 60.00th=[ 405], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.13%, sys=26.32%, ctx=328, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11179: Wed May 21 13:50:56 2025 write: IOPS=3, BW=13.2MiB/s (13.8MB/s)(12.0MiB/911msec) clat (msec): min=169, max=339, avg=269.59, stdev=88.75 lat (msec): min=170, max=339, avg=269.87, stdev=88.71 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 169], 10.00th=[ 169], 20.00th=[ 169], | 30.00th=[ 169], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 8192, max= 8192, per=22.78%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=39.56%, ctx=40, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11181: Wed May 21 13:50:56 2025 write: IOPS=3, BW=14.0MiB/s (15.7MB/s)(12.0MiB/802msec) clat (msec): min=127, max=376, avg=256.11, stdev=124.35 lat (msec): min=128, max=376, avg=256.49, stdev=124.19 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 129], 10.00th=[ 129], 20.00th=[ 129], | 30.00th=[ 129], 40.00th=[ 264], 50.00th=[ 264], 60.00th=[ 264], | 70.00th=[ 376], 80.00th=[ 376], 90.00th=[ 376], 95.00th=[ 376], | 99.00th=[ 376], 99.50th=[ 376], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 8192, max= 8192, per=22.78%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=38.45%, ctx=51, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=20.6MiB/s (21.6MB/s), 5278KiB/s-27.3MiB/s (5405kB/s-28.7MB/s), io=16.0MiB (16.8MB), run=439-776msec WRITE: bw=35.1MiB/s (36.8MB/s), 10.3MiB/s-14.0MiB/s (10.8MB/s-15.7MB/s), io=32.0MiB (33.6MB), run=776-911msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11174: Wed May 21 13:50:56 2025 read: IOPS=3, BW=14.0MiB/s (15.7MB/s)(12.0MiB/801msec) clat (msec): min=69, max=550, avg=263.21, stdev=253.76 lat (msec): min=69, max=550, avg=263.22, stdev=253.76 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 70], 10.00th=[ 70], 20.00th=[ 70], | 30.00th=[ 70], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 169], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min=16384, max=16384, per=98.90%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=10.99%, ctx=92, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11176: Wed May 21 13:50:56 2025 read: IOPS=1, BW=4142KiB/s (4241kB/s)(4096KiB/989msec) clat (nsec): min=203525k, max=203525k, avg=203525205.00, stdev= 0.00 lat (nsec): min=203528k, max=203528k, avg=203527995.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 205], 10.00th=[ 205], 20.00th=[ 205], | 30.00th=[ 205], 40.00th=[ 205], 50.00th=[ 205], 60.00th=[ 205], | 70.00th=[ 205], 80.00th=[ 205], 90.00th=[ 205], 95.00th=[ 205], | 99.00th=[ 205], 99.50th=[ 205], 99.90th=[ 205], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 8192, max= 8192, per=49.45%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=8283KiB/s (8482kB/s)(8192KiB/989msec) clat (msec): min=233, max=460, avg=346.81, stdev=160.78 lat (msec): min=233, max=460, avg=346.98, stdev=160.80 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 234], 10.00th=[ 234], 20.00th=[ 234], | 30.00th=[ 234], 40.00th=[ 234], 50.00th=[ 234], 60.00th=[ 460], | 70.00th=[ 460], 80.00th=[ 460], 90.00th=[ 460], 95.00th=[ 460], | 99.00th=[ 460], 99.50th=[ 460], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.10%, sys=6.88%, ctx=34, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11178: Wed May 21 13:50:56 2025 write: IOPS=2, BW=9.90MiB/s (10.4MB/s)(12.0MiB/1212msec) clat (msec): min=172, max=516, avg=391.10, stdev=190.16 lat (msec): min=172, max=517, avg=391.41, stdev=190.18 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 485], 50.00th=[ 485], 60.00th=[ 485], | 70.00th=[ 518], 80.00th=[ 518], 90.00th=[ 518], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 8175, max= 8175, per=30.24%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.08%, sys=16.93%, ctx=161, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11180: Wed May 21 13:50:56 2025 write: IOPS=2, BW=11.4MiB/s (11.9MB/s)(12.0MiB/1057msec) clat (msec): min=169, max=665, avg=342.67, stdev=280.01 lat (msec): min=170, max=666, avg=342.98, stdev=280.02 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 169], 10.00th=[ 169], 20.00th=[ 169], | 30.00th=[ 169], 40.00th=[ 192], 50.00th=[ 192], 60.00th=[ 192], | 70.00th=[ 667], 80.00th=[ 667], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min=16384, max=16384, per=60.60%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.19%, sys=14.20%, ctx=51, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.2MiB/s (16.0MB/s), 4142KiB/s-14.0MiB/s (4241kB/s-15.7MB/s), io=16.0MiB (16.8MB), run=801-989msec WRITE: bw=26.4MiB/s (27.7MB/s), 8283KiB/s-11.4MiB/s (8482kB/s-11.9MB/s), io=32.0MiB (33.6MB), run=989-1212msec