== sanity test 398b: DIO and buffer IO race ============== 09:34:02 (1742909642) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.16819 s, 43.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6780: Tue Mar 25 09:34:18 2025 read: IOPS=109, BW=437KiB/s (448kB/s)(5992KiB/13711msec) clat (usec): min=52, max=159397, avg=3986.05, stdev=5275.36 lat (usec): min=53, max=159397, avg=3988.07, stdev=5275.63 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 72], 10.00th=[ 89], 20.00th=[ 529], | 30.00th=[ 2409], 40.00th=[ 2769], 50.00th=[ 3228], 60.00th=[ 3818], | 70.00th=[ 4686], 80.00th=[ 6194], 90.00th=[ 8094], 95.00th=[ 9372], | 99.00th=[ 12911], 99.50th=[ 15008], 99.90th=[ 58983], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 280, max= 614, per=24.70%, avg=430.33, stdev=68.40, samples=27 iops : min= 70, max= 153, avg=107.48, stdev=17.08, samples=27 write: IOPS=114, BW=459KiB/s (470kB/s)(6296KiB/13711msec) clat (usec): min=475, max=61495, avg=4872.43, stdev=4683.67 lat (usec): min=476, max=61496, avg=4873.58, stdev=4683.69 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 553], 10.00th=[ 603], 20.00th=[ 783], | 30.00th=[ 1123], 40.00th=[ 2769], 50.00th=[ 3982], 60.00th=[ 5014], | 70.00th=[ 6390], 80.00th=[ 8094], 90.00th=[10552], 95.00th=[13173], | 99.00th=[19006], 99.50th=[25035], 99.90th=[34341], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 336, max= 576, per=24.81%, avg=452.56, stdev=63.01, samples=27 iops : min= 84, max= 144, avg=113.04, stdev=15.74, samples=27 lat (usec) : 100=5.99%, 250=3.61%, 500=0.33%, 750=8.98%, 1000=5.40% lat (msec) : 2=5.08%, 4=27.21%, 10=35.55%, 20=7.29%, 50=0.46% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.20%, sys=23.16%, ctx=5688, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6781: Tue Mar 25 09:34:18 2025 read: IOPS=107, BW=431KiB/s (442kB/s)(5940KiB/13777msec) clat (usec): min=52, max=149062, avg=4100.29, stdev=5280.11 lat (usec): min=52, max=149064, avg=4101.14, stdev=5280.08 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 76], 10.00th=[ 94], 20.00th=[ 1090], | 30.00th=[ 2442], 40.00th=[ 2900], 50.00th=[ 3228], 60.00th=[ 3851], | 70.00th=[ 4817], 80.00th=[ 6325], 90.00th=[ 8094], 95.00th=[ 9634], | 99.00th=[ 15533], 99.50th=[ 26084], 99.90th=[ 41157], 99.95th=[149947], | 99.99th=[149947] bw ( KiB/s): min= 248, max= 600, per=24.51%, avg=426.93, stdev=80.87, samples=27 iops : min= 62, max= 150, avg=106.70, stdev=20.27, samples=27 write: IOPS=115, BW=461KiB/s (472kB/s)(6348KiB/13777msec) clat (usec): min=452, max=49057, avg=4797.43, stdev=4479.43 lat (usec): min=453, max=49058, avg=4799.56, stdev=4479.62 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 553], 10.00th=[ 594], 20.00th=[ 783], | 30.00th=[ 1123], 40.00th=[ 2573], 50.00th=[ 4047], 60.00th=[ 5014], | 70.00th=[ 6259], 80.00th=[ 8029], 90.00th=[10552], 95.00th=[12780], | 99.00th=[19268], 99.50th=[22938], 99.90th=[28705], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 288, max= 576, per=24.77%, avg=451.81, stdev=63.01, samples=27 iops : min= 72, max= 144, avg=112.93, stdev=15.76, samples=27 lat (usec) : 100=5.57%, 250=3.81%, 500=0.46%, 750=9.34%, 1000=5.24% lat (msec) : 2=4.98%, 4=26.69%, 10=35.84%, 20=7.16%, 50=0.88% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=22.82%, ctx=5628, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6782: Tue Mar 25 09:34:18 2025 read: IOPS=110, BW=441KiB/s (452kB/s)(6028KiB/13667msec) clat (usec): min=52, max=36944, avg=3881.14, stdev=3292.81 lat (usec): min=52, max=36944, avg=3882.07, stdev=3292.86 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 77], 10.00th=[ 92], 20.00th=[ 326], | 30.00th=[ 2409], 40.00th=[ 2802], 50.00th=[ 3261], 60.00th=[ 3720], | 70.00th=[ 4686], 80.00th=[ 6194], 90.00th=[ 8094], 95.00th=[ 9503], | 99.00th=[13435], 99.50th=[18220], 99.90th=[27395], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 248, max= 592, per=25.13%, avg=437.81, stdev=85.58, samples=27 iops : min= 62, max= 148, avg=109.41, stdev=21.35, samples=27 write: IOPS=114, BW=458KiB/s (469kB/s)(6260KiB/13667msec) clat (usec): min=470, max=175414, avg=4956.98, stdev=6294.00 lat (usec): min=470, max=175416, avg=4957.90, stdev=6294.24 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 562], 10.00th=[ 619], 20.00th=[ 816], | 30.00th=[ 1123], 40.00th=[ 2507], 50.00th=[ 3884], 60.00th=[ 5080], | 70.00th=[ 6325], 80.00th=[ 7963], 90.00th=[ 10945], 95.00th=[ 13304], | 99.00th=[ 20055], 99.50th=[ 23725], 99.90th=[ 44827], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 304, max= 576, per=24.90%, avg=454.11, stdev=76.88, samples=27 iops : min= 76, max= 144, avg=113.48, stdev=19.19, samples=27 lat (usec) : 100=5.63%, 250=4.10%, 500=0.42%, 750=8.59%, 1000=5.27% lat (msec) : 2=5.37%, 4=27.99%, 10=34.21%, 20=7.68%, 50=0.68% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=23.34%, ctx=5587, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6783: Tue Mar 25 09:34:18 2025 read: IOPS=111, BW=448KiB/s (459kB/s)(6052KiB/13513msec) clat (usec): min=54, max=38856, avg=3871.85, stdev=3202.04 lat (usec): min=54, max=38856, avg=3872.58, stdev=3202.03 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 78], 10.00th=[ 97], 20.00th=[ 1893], | 30.00th=[ 2409], 40.00th=[ 2802], 50.00th=[ 3228], 60.00th=[ 3818], | 70.00th=[ 4817], 80.00th=[ 6063], 90.00th=[ 7963], 95.00th=[ 9372], | 99.00th=[12256], 99.50th=[15401], 99.90th=[33817], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 288, max= 616, per=25.71%, avg=447.89, stdev=71.42, samples=27 iops : min= 72, max= 154, avg=111.89, stdev=17.86, samples=27 write: IOPS=115, BW=461KiB/s (473kB/s)(6236KiB/13513msec) clat (usec): min=446, max=169891, avg=4867.50, stdev=6281.33 lat (usec): min=447, max=169893, avg=4868.62, stdev=6281.29 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 553], 10.00th=[ 594], 20.00th=[ 750], | 30.00th=[ 955], 40.00th=[ 2311], 50.00th=[ 3752], 60.00th=[ 4752], | 70.00th=[ 6128], 80.00th=[ 7898], 90.00th=[ 11076], 95.00th=[ 13435], | 99.00th=[ 21103], 99.50th=[ 24249], 99.90th=[ 36963], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 335, max= 638, per=25.28%, avg=461.15, stdev=75.07, samples=27 iops : min= 83, max= 159, avg=115.22, stdev=18.76, samples=27 lat (usec) : 100=5.08%, 250=4.23%, 500=0.46%, 750=9.96%, 1000=5.47% lat (msec) : 2=4.65%, 4=27.05%, 10=34.77%, 20=7.62%, 50=0.68% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=23.13%, ctx=5577, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1743KiB/s (1785kB/s), 431KiB/s-448KiB/s (442kB/s-459kB/s), io=23.4MiB (24.6MB), run=13513-13777msec WRITE: bw=1825KiB/s (1869kB/s), 458KiB/s-461KiB/s (469kB/s-473kB/s), io=24.6MiB (25.7MB), run=13513-13777msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6776: Tue Mar 25 09:34:40 2025 read: IOPS=41, BW=166KiB/s (170kB/s)(5992KiB/36023msec) clat (usec): min=1694, max=162864, avg=3808.99, stdev=5165.08 lat (usec): min=1695, max=162865, avg=3809.84, stdev=5165.07 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1909], 10.00th=[ 1991], 20.00th=[ 2089], | 30.00th=[ 2180], 40.00th=[ 2278], 50.00th=[ 2442], 60.00th=[ 2606], | 70.00th=[ 3032], 80.00th=[ 4228], 90.00th=[ 8094], 95.00th=[ 10683], | 99.00th=[ 15139], 99.50th=[ 20841], 99.90th=[ 32375], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 56, max= 312, per=25.02%, avg=166.37, stdev=68.94, samples=72 iops : min= 14, max= 78, avg=41.54, stdev=17.27, samples=72 write: IOPS=43, BW=175KiB/s (179kB/s)(6296KiB/36023msec) clat (usec): min=9353, max=65736, avg=19208.30, stdev=7106.31 lat (usec): min=9354, max=65737, avg=19209.33, stdev=7106.29 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11600], 10.00th=[12125], 20.00th=[13304], | 30.00th=[14746], 40.00th=[16450], 50.00th=[17695], 60.00th=[19006], | 70.00th=[20579], 80.00th=[23200], 90.00th=[29492], 95.00th=[33817], | 99.00th=[42730], 99.50th=[46400], 99.90th=[58983], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 63, max= 248, per=25.09%, avg=174.60, stdev=52.59, samples=72 iops : min= 15, max= 62, avg=43.60, stdev=13.18, samples=72 lat (msec) : 2=5.34%, 4=32.94%, 10=7.58%, 20=36.88%, 50=17.02% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.11%, sys=5.69%, ctx=3470, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6777: Tue Mar 25 09:34:40 2025 read: IOPS=41, BW=165KiB/s (169kB/s)(5940KiB/36093msec) clat (usec): min=1654, max=159376, avg=3842.87, stdev=5602.81 lat (usec): min=1655, max=159377, avg=3843.71, stdev=5602.81 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1909], 10.00th=[ 1975], 20.00th=[ 2089], | 30.00th=[ 2180], 40.00th=[ 2311], 50.00th=[ 2409], 60.00th=[ 2573], | 70.00th=[ 2900], 80.00th=[ 4015], 90.00th=[ 7635], 95.00th=[ 10552], | 99.00th=[ 19268], 99.50th=[ 25297], 99.90th=[ 58459], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 23, max= 328, per=24.74%, avg=164.50, stdev=71.64, samples=72 iops : min= 5, max= 82, avg=41.00, stdev=17.99, samples=72 write: IOPS=43, BW=176KiB/s (180kB/s)(6348KiB/36093msec) clat (usec): min=9143, max=74255, avg=19093.28, stdev=7298.61 lat (usec): min=9144, max=74256, avg=19094.28, stdev=7298.63 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11469], 10.00th=[11994], 20.00th=[13173], | 30.00th=[14484], 40.00th=[16188], 50.00th=[17695], 60.00th=[19006], | 70.00th=[20317], 80.00th=[22676], 90.00th=[29492], 95.00th=[34341], | 99.00th=[43254], 99.50th=[49021], 99.90th=[58983], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 79, max= 248, per=25.20%, avg=175.39, stdev=53.20, samples=72 iops : min= 19, max= 62, avg=43.72, stdev=13.35, samples=72 lat (msec) : 2=5.63%, 4=32.94%, 10=7.13%, 20=37.50%, 50=16.47% lat (msec) : 100=0.29%, 250=0.03% cpu : usr=0.07%, sys=5.75%, ctx=3497, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6778: Tue Mar 25 09:34:40 2025 read: IOPS=42, BW=168KiB/s (172kB/s)(6028KiB/35860msec) clat (usec): min=1730, max=51096, avg=3961.01, stdev=3754.86 lat (usec): min=1731, max=51097, avg=3961.84, stdev=3754.83 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1926], 10.00th=[ 1991], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2671], | 70.00th=[ 3163], 80.00th=[ 4752], 90.00th=[ 8848], 95.00th=[11207], | 99.00th=[17695], 99.50th=[21890], 99.90th=[37487], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 56, max= 336, per=25.29%, avg=168.18, stdev=62.16, samples=71 iops : min= 14, max= 84, avg=41.93, stdev=15.56, samples=71 write: IOPS=43, BW=175KiB/s (179kB/s)(6260KiB/35860msec) clat (msec): min=9, max=173, avg=19.05, stdev= 7.94 lat (msec): min=9, max=173, avg=19.05, stdev= 7.94 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 29], 95.00th=[ 34], | 99.00th=[ 44], 99.50th=[ 46], 99.90th=[ 52], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 80, max= 247, per=25.07%, avg=174.49, stdev=55.10, samples=71 iops : min= 20, max= 61, avg=43.51, stdev=13.76, samples=71 lat (msec) : 2=4.98%, 4=32.29%, 10=8.24%, 20=38.22%, 50=16.18% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.10%, sys=5.81%, ctx=3446, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6779: Tue Mar 25 09:34:40 2025 read: IOPS=42, BW=168KiB/s (172kB/s)(6052KiB/36023msec) clat (usec): min=1601, max=58487, avg=3757.22, stdev=3761.38 lat (usec): min=1601, max=58487, avg=3758.05, stdev=3761.36 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1909], 10.00th=[ 1991], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2638], | 70.00th=[ 2966], 80.00th=[ 4047], 90.00th=[ 7701], 95.00th=[10552], | 99.00th=[17171], 99.50th=[22414], 99.90th=[54789], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 40, max= 352, per=25.24%, avg=167.88, stdev=69.71, samples=72 iops : min= 10, max= 88, avg=41.88, stdev=17.43, samples=72 write: IOPS=43, BW=173KiB/s (177kB/s)(6236KiB/36023msec) clat (msec): min=9, max=173, avg=19.41, stdev= 8.39 lat (msec): min=9, max=173, avg=19.41, stdev= 8.39 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 21], 80.00th=[ 24], 90.00th=[ 31], 95.00th=[ 35], | 99.00th=[ 44], 99.50th=[ 50], 99.90th=[ 75], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 72, max= 248, per=24.84%, avg=172.87, stdev=53.98, samples=72 iops : min= 18, max= 62, avg=43.13, stdev=13.51, samples=72 lat (msec) : 2=5.08%, 4=34.28%, 10=7.36%, 20=36.23%, 50=16.76% lat (msec) : 100=0.26%, 250=0.03% cpu : usr=0.13%, sys=5.72%, ctx=3466, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=665KiB/s (681kB/s), 165KiB/s-168KiB/s (169kB/s-172kB/s), io=23.4MiB (24.6MB), run=35860-36093msec WRITE: bw=697KiB/s (713kB/s), 173KiB/s-176KiB/s (177kB/s-180kB/s), io=24.6MiB (25.7MB), run=35860-36093msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6819: Tue Mar 25 09:34:44 2025 read: IOPS=128, BW=2063KiB/s (2113kB/s)(5920KiB/2869msec) clat (usec): min=58, max=21084, avg=3656.46, stdev=2875.02 lat (usec): min=58, max=21085, avg=3658.77, stdev=2877.92 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 82], 10.00th=[ 99], 20.00th=[ 129], | 30.00th=[ 2474], 40.00th=[ 2966], 50.00th=[ 3556], 60.00th=[ 4146], | 70.00th=[ 4752], 80.00th=[ 5604], 90.00th=[ 7308], 95.00th=[ 8586], | 99.00th=[11863], 99.50th=[14877], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 1888, max= 2464, per=25.33%, avg=2084.60, stdev=230.67, samples=5 iops : min= 118, max= 154, avg=130.20, stdev=14.39, samples=5 write: IOPS=138, BW=2220KiB/s (2273kB/s)(6368KiB/2869msec) clat (usec): min=671, max=35631, avg=3749.07, stdev=3628.89 lat (usec): min=672, max=35632, avg=3750.34, stdev=3628.75 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 832], 10.00th=[ 914], 20.00th=[ 1090], | 30.00th=[ 1270], 40.00th=[ 1516], 50.00th=[ 2212], 60.00th=[ 3687], | 70.00th=[ 4621], 80.00th=[ 6194], 90.00th=[ 8225], 95.00th=[10028], | 99.00th=[15926], 99.50th=[20317], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 2144, max= 2592, per=26.14%, avg=2263.80, stdev=187.49, samples=5 iops : min= 134, max= 162, avg=141.40, stdev=11.74, samples=5 lat (usec) : 100=4.82%, 250=6.90%, 500=0.26%, 750=0.52%, 1000=7.68% lat (msec) : 2=17.19%, 4=23.18%, 10=36.07%, 20=2.99%, 50=0.39% cpu : usr=0.35%, sys=30.33%, ctx=1560, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6820: Tue Mar 25 09:34:44 2025 read: IOPS=123, BW=1976KiB/s (2023kB/s)(5744KiB/2907msec) clat (usec): min=54, max=24133, avg=3791.74, stdev=3543.58 lat (usec): min=54, max=24134, avg=3792.97, stdev=3543.05 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 73], 10.00th=[ 90], 20.00th=[ 116], | 30.00th=[ 2507], 40.00th=[ 2933], 50.00th=[ 3359], 60.00th=[ 3884], | 70.00th=[ 4555], 80.00th=[ 5669], 90.00th=[ 7635], 95.00th=[10421], | 99.00th=[17695], 99.50th=[23462], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 1760, max= 2144, per=24.24%, avg=1995.00, stdev=144.71, samples=5 iops : min= 110, max= 134, avg=124.60, stdev= 8.99, samples=5 write: IOPS=140, BW=2251KiB/s (2305kB/s)(6544KiB/2907msec) clat (usec): min=708, max=30566, avg=3747.17, stdev=3691.70 lat (usec): min=709, max=30567, avg=3748.17, stdev=3691.71 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 824], 10.00th=[ 881], 20.00th=[ 1012], | 30.00th=[ 1172], 40.00th=[ 1434], 50.00th=[ 2180], 60.00th=[ 3490], | 70.00th=[ 4817], 80.00th=[ 6390], 90.00th=[ 8455], 95.00th=[10290], | 99.00th=[13960], 99.50th=[19792], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 2016, max= 2432, per=26.14%, avg=2263.60, stdev=190.33, samples=5 iops : min= 126, max= 152, avg=141.40, stdev=11.87, samples=5 lat (usec) : 100=5.86%, 250=5.86%, 500=0.39%, 750=0.39%, 1000=9.37% lat (msec) : 2=16.67%, 4=24.87%, 10=31.25%, 20=4.82%, 50=0.52% cpu : usr=0.24%, sys=29.04%, ctx=1539, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6821: Tue Mar 25 09:34:44 2025 read: IOPS=134, BW=2150KiB/s (2201kB/s)(6256KiB/2910msec) clat (usec): min=57, max=39775, avg=3872.15, stdev=3509.66 lat (usec): min=57, max=39776, avg=3872.84, stdev=3509.67 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 73], 10.00th=[ 91], 20.00th=[ 133], | 30.00th=[ 2540], 40.00th=[ 2966], 50.00th=[ 3458], 60.00th=[ 4113], | 70.00th=[ 5014], 80.00th=[ 6194], 90.00th=[ 7635], 95.00th=[ 8848], | 99.00th=[13435], 99.50th=[19268], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 1856, max= 2208, per=25.02%, avg=2059.20, stdev=145.28, samples=5 iops : min= 116, max= 138, avg=128.60, stdev= 9.15, samples=5 write: IOPS=129, BW=2073KiB/s (2123kB/s)(6032KiB/2910msec) clat (usec): min=664, max=21524, avg=3648.16, stdev=3677.15 lat (usec): min=665, max=21526, avg=3649.22, stdev=3677.20 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 783], 10.00th=[ 832], 20.00th=[ 979], | 30.00th=[ 1188], 40.00th=[ 1385], 50.00th=[ 1860], 60.00th=[ 3294], | 70.00th=[ 4293], 80.00th=[ 5932], 90.00th=[ 8291], 95.00th=[10945], | 99.00th=[18744], 99.50th=[19792], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 1632, max= 2517, per=23.55%, avg=2039.40, stdev=374.70, samples=5 iops : min= 102, max= 157, avg=127.40, stdev=23.32, samples=5 lat (usec) : 100=5.99%, 250=5.86%, 500=0.78%, 750=1.56%, 1000=8.85% lat (msec) : 2=15.10%, 4=24.87%, 10=32.16%, 20=4.56%, 50=0.26% cpu : usr=0.28%, sys=28.46%, ctx=1626, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6822: Tue Mar 25 09:34:44 2025 read: IOPS=133, BW=2137KiB/s (2189kB/s)(6032KiB/2822msec) clat (usec): min=59, max=26271, avg=3674.35, stdev=3403.03 lat (usec): min=59, max=26441, avg=3675.48, stdev=3406.05 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 73], 10.00th=[ 85], 20.00th=[ 114], | 30.00th=[ 2057], 40.00th=[ 2999], 50.00th=[ 3458], 60.00th=[ 4047], | 70.00th=[ 4752], 80.00th=[ 5800], 90.00th=[ 7701], 95.00th=[ 8717], | 99.00th=[16581], 99.50th=[22152], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 1856, max= 2336, per=25.87%, avg=2129.40, stdev=194.18, samples=5 iops : min= 116, max= 146, avg=133.00, stdev=12.21, samples=5 write: IOPS=138, BW=2217KiB/s (2270kB/s)(6256KiB/2822msec) clat (usec): min=680, max=33583, avg=3583.50, stdev=3770.59 lat (usec): min=681, max=33584, avg=3584.58, stdev=3770.58 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 783], 10.00th=[ 840], 20.00th=[ 1004], | 30.00th=[ 1139], 40.00th=[ 1287], 50.00th=[ 1811], 60.00th=[ 3195], | 70.00th=[ 4555], 80.00th=[ 6128], 90.00th=[ 8717], 95.00th=[10028], | 99.00th=[15008], 99.50th=[31327], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 1984, max= 2592, per=25.92%, avg=2244.60, stdev=229.07, samples=5 iops : min= 124, max= 162, avg=140.20, stdev=14.32, samples=5 lat (usec) : 100=6.64%, 250=7.42%, 500=0.13%, 750=1.17%, 1000=9.11% lat (msec) : 2=16.80%, 4=22.01%, 10=32.68%, 20=3.39%, 50=0.65% cpu : usr=0.25%, sys=28.54%, ctx=1452, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8231KiB/s (8428kB/s), 1976KiB/s-2150KiB/s (2023kB/s-2201kB/s), io=23.4MiB (24.5MB), run=2822-2910msec WRITE: bw=8660KiB/s (8868kB/s), 2073KiB/s-2251KiB/s (2123kB/s-2305kB/s), io=24.6MiB (25.8MB), run=2822-2910msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6824: Tue Mar 25 09:34:50 2025 read: IOPS=41, BW=658KiB/s (674kB/s)(5920KiB/8992msec) clat (usec): min=1717, max=40179, avg=4382.80, stdev=4341.16 lat (usec): min=1717, max=40180, avg=4383.62, stdev=4341.17 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2040], 10.00th=[ 2147], 20.00th=[ 2311], | 30.00th=[ 2507], 40.00th=[ 2737], 50.00th=[ 3130], 60.00th=[ 3720], | 70.00th=[ 4080], 80.00th=[ 4555], 90.00th=[ 6718], 95.00th=[12518], | 99.00th=[26084], 99.50th=[34341], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 384, max= 1184, per=24.45%, avg=647.24, stdev=234.96, samples=17 iops : min= 24, max= 74, avg=40.24, stdev=14.87, samples=17 write: IOPS=44, BW=708KiB/s (725kB/s)(6368KiB/8992msec) clat (usec): min=9834, max=59150, avg=18402.18, stdev=6488.58 lat (usec): min=9836, max=59151, avg=18403.70, stdev=6488.63 clat percentiles (usec): | 1.00th=[10159], 5.00th=[11600], 10.00th=[12256], 20.00th=[13173], | 30.00th=[14484], 40.00th=[15795], 50.00th=[17171], 60.00th=[18220], | 70.00th=[19530], 80.00th=[22414], 90.00th=[26608], 95.00th=[31851], | 99.00th=[40633], 99.50th=[52691], 99.90th=[58983], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 383, max= 928, per=25.06%, avg=698.00, stdev=194.24, samples=17 iops : min= 23, max= 58, avg=43.41, stdev=12.32, samples=17 lat (msec) : 2=1.69%, 4=30.86%, 10=13.02%, 20=38.80%, 50=15.36% lat (msec) : 100=0.26% cpu : usr=0.04%, sys=6.84%, ctx=1019, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6825: Tue Mar 25 09:34:50 2025 read: IOPS=39, BW=635KiB/s (650kB/s)(5744KiB/9047msec) clat (usec): min=1824, max=23825, avg=4219.24, stdev=3156.58 lat (usec): min=1825, max=23826, avg=4220.06, stdev=3156.56 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 1991], 10.00th=[ 2114], 20.00th=[ 2278], | 30.00th=[ 2507], 40.00th=[ 2769], 50.00th=[ 3359], 60.00th=[ 3851], | 70.00th=[ 4228], 80.00th=[ 4621], 90.00th=[ 7701], 95.00th=[11469], | 99.00th=[17695], 99.50th=[19006], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 288, max= 1152, per=23.82%, avg=630.41, stdev=227.65, samples=17 iops : min= 18, max= 72, avg=39.29, stdev=14.27, samples=17 write: IOPS=45, BW=723KiB/s (741kB/s)(6544KiB/9047msec) clat (usec): min=9518, max=57566, avg=18285.95, stdev=7025.39 lat (usec): min=9520, max=57568, avg=18287.49, stdev=7025.46 clat percentiles (usec): | 1.00th=[10159], 5.00th=[11076], 10.00th=[11731], 20.00th=[12780], | 30.00th=[13698], 40.00th=[15270], 50.00th=[16909], 60.00th=[18220], | 70.00th=[19792], 80.00th=[21627], 90.00th=[27919], 95.00th=[32375], | 99.00th=[44303], 99.50th=[49021], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 384, max= 928, per=25.34%, avg=705.65, stdev=200.71, samples=17 iops : min= 24, max= 58, avg=44.00, stdev=12.44, samples=17 lat (msec) : 2=2.34%, 4=28.39%, 10=13.28%, 20=40.89%, 50=14.84% lat (msec) : 100=0.26% cpu : usr=0.14%, sys=6.51%, ctx=983, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6826: Tue Mar 25 09:34:50 2025 read: IOPS=45, BW=723KiB/s (740kB/s)(6256KiB/8653msec) clat (usec): min=1731, max=39897, avg=4072.66, stdev=3653.79 lat (usec): min=1732, max=39897, avg=4073.47, stdev=3653.79 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 1975], 10.00th=[ 2114], 20.00th=[ 2278], | 30.00th=[ 2409], 40.00th=[ 2573], 50.00th=[ 2900], 60.00th=[ 3523], | 70.00th=[ 4047], 80.00th=[ 4555], 90.00th=[ 6390], 95.00th=[11076], | 99.00th=[18744], 99.50th=[32375], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 288, max= 1248, per=27.59%, avg=730.18, stdev=320.61, samples=17 iops : min= 18, max= 78, avg=45.53, stdev=20.05, samples=17 write: IOPS=43, BW=697KiB/s (714kB/s)(6032KiB/8653msec) clat (usec): min=9921, max=50653, avg=18684.22, stdev=6800.54 lat (usec): min=9923, max=50655, avg=18685.78, stdev=6800.56 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11863], 10.00th=[12387], 20.00th=[13435], | 30.00th=[14484], 40.00th=[15664], 50.00th=[17171], 60.00th=[18482], | 70.00th=[19530], 80.00th=[21890], 90.00th=[27657], 95.00th=[33817], | 99.00th=[43254], 99.50th=[50070], 99.90th=[50594], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 415, max= 992, per=24.87%, avg=692.53, stdev=179.60, samples=17 iops : min= 25, max= 62, avg=43.18, stdev=11.29, samples=17 lat (msec) : 2=3.12%, 4=31.90%, 10=13.15%, 20=37.50%, 50=14.19% lat (msec) : 100=0.13% cpu : usr=0.16%, sys=6.76%, ctx=1004, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6827: Tue Mar 25 09:34:50 2025 read: IOPS=42, BW=686KiB/s (702kB/s)(6032KiB/8795msec) clat (usec): min=1747, max=40184, avg=4271.84, stdev=3790.34 lat (usec): min=1748, max=40184, avg=4272.71, stdev=3790.33 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 2024], 10.00th=[ 2114], 20.00th=[ 2311], | 30.00th=[ 2507], 40.00th=[ 2737], 50.00th=[ 3294], 60.00th=[ 3687], | 70.00th=[ 4047], 80.00th=[ 4490], 90.00th=[ 7177], 95.00th=[11863], | 99.00th=[20579], 99.50th=[20841], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 288, max= 1088, per=25.60%, avg=677.65, stdev=212.94, samples=17 iops : min= 18, max= 68, avg=42.35, stdev=13.31, samples=17 write: IOPS=44, BW=711KiB/s (728kB/s)(6256KiB/8795msec) clat (usec): min=9695, max=48573, avg=18198.30, stdev=6042.57 lat (usec): min=9696, max=48574, avg=18199.91, stdev=6042.60 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11469], 10.00th=[12256], 20.00th=[13304], | 30.00th=[14222], 40.00th=[15533], 50.00th=[16712], 60.00th=[18482], | 70.00th=[20055], 80.00th=[21890], 90.00th=[25822], 95.00th=[29492], | 99.00th=[42206], 99.50th=[45876], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 416, max= 928, per=25.41%, avg=707.76, stdev=186.89, samples=17 iops : min= 26, max= 58, avg=44.24, stdev=11.68, samples=17 lat (msec) : 2=1.95%, 4=31.25%, 10=13.28%, 20=37.24%, 50=16.28% cpu : usr=0.14%, sys=6.68%, ctx=985, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2648KiB/s (2711kB/s), 635KiB/s-723KiB/s (650kB/s-740kB/s), io=23.4MiB (24.5MB), run=8653-9047msec WRITE: bw=2785KiB/s (2852kB/s), 697KiB/s-723KiB/s (714kB/s-741kB/s), io=24.6MiB (25.8MB), run=8653-9047msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6846: Tue Mar 25 09:34:52 2025 read: IOPS=7, BW=7666KiB/s (7850kB/s)(7168KiB/935msec) clat (usec): min=14891, max=57177, avg=35404.26, stdev=19955.00 lat (usec): min=14893, max=57178, avg=35405.57, stdev=19954.85 clat percentiles (usec): | 1.00th=[14877], 5.00th=[14877], 10.00th=[14877], 20.00th=[15270], | 30.00th=[22152], 40.00th=[22152], 50.00th=[26870], 60.00th=[54789], | 70.00th=[54789], 80.00th=[56886], 90.00th=[57410], 95.00th=[57410], | 99.00th=[57410], 99.50th=[57410], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 6144, max= 6144, per=33.30%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5476KiB/s (5607kB/s)(5120KiB/935msec) clat (msec): min=24, max=463, avg=133.89, stdev=185.41 lat (msec): min=24, max=463, avg=133.95, stdev=185.40 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 61], 60.00th=[ 61], | 70.00th=[ 80], 80.00th=[ 80], 90.00th=[ 464], 95.00th=[ 464], | 99.00th=[ 464], 99.50th=[ 464], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] lat (msec) : 20=16.67%, 50=33.33%, 100=41.67%, 500=8.33% cpu : usr=0.00%, sys=8.14%, ctx=54, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6847: Tue Mar 25 09:34:52 2025 read: IOPS=1, BW=1900KiB/s (1945kB/s)(2048KiB/1078msec) clat (usec): min=13445, max=14394, avg=13919.96, stdev=671.54 lat (usec): min=13445, max=14397, avg=13921.44, stdev=672.63 clat percentiles (usec): | 1.00th=[13435], 5.00th=[13435], 10.00th=[13435], 20.00th=[13435], | 30.00th=[13435], 40.00th=[13435], 50.00th=[13435], 60.00th=[14353], | 70.00th=[14353], 80.00th=[14353], 90.00th=[14353], 95.00th=[14353], | 99.00th=[14353], 99.50th=[14353], 99.90th=[14353], 99.95th=[14353], | 99.99th=[14353] bw ( KiB/s): min= 2048, max= 2048, per=11.10%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=9, BW=9499KiB/s (9727kB/s)(10.0MiB/1078msec) clat (msec): min=27, max=505, avg=103.84, stdev=143.85 lat (msec): min=27, max=505, avg=103.90, stdev=143.86 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 29], | 30.00th=[ 36], 40.00th=[ 41], 50.00th=[ 52], 60.00th=[ 66], | 70.00th=[ 88], 80.00th=[ 88], 90.00th=[ 114], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 6144, max= 8192, per=27.75%, avg=7168.00, stdev=1448.15, samples=2 iops : min= 6, max= 8, avg= 7.00, stdev= 1.41, samples=2 lat (msec) : 20=16.67%, 50=33.33%, 100=33.33%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=11.14%, ctx=64, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6848: Tue Mar 25 09:34:52 2025 read: IOPS=5, BW=5535KiB/s (5668kB/s)(6144KiB/1110msec) clat (msec): min=14, max=436, avg=118.59, stdev=161.83 lat (msec): min=14, max=436, avg=118.59, stdev=161.83 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15], | 30.00th=[ 15], 40.00th=[ 54], 50.00th=[ 54], 60.00th=[ 56], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 439], 95.00th=[ 439], | 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 4096, max= 4096, per=22.20%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5535KiB/s (5668kB/s)(6144KiB/1110msec) clat (usec): min=32309, max=80407, avg=56057.22, stdev=19991.60 lat (usec): min=32351, max=80445, avg=56109.89, stdev=19988.42 clat percentiles (usec): | 1.00th=[32375], 5.00th=[32375], 10.00th=[32375], 20.00th=[34341], | 30.00th=[34341], 40.00th=[53216], 50.00th=[53216], 60.00th=[62129], | 70.00th=[73925], 80.00th=[73925], 90.00th=[80217], 95.00th=[80217], | 99.00th=[80217], 99.50th=[80217], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 2048, max= 8192, per=19.82%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (msec) : 20=16.67%, 50=16.67%, 100=50.00%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=9.65%, ctx=84, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6849: Tue Mar 25 09:34:52 2025 read: IOPS=4, BW=4667KiB/s (4779kB/s)(5120KiB/1097msec) clat (usec): min=13761, max=84485, avg=41178.04, stdev=27642.11 lat (usec): min=13762, max=84485, avg=41179.26, stdev=27641.82 clat percentiles (usec): | 1.00th=[13698], 5.00th=[13698], 10.00th=[13698], 20.00th=[13698], | 30.00th=[20841], 40.00th=[20841], 50.00th=[41157], 60.00th=[41157], | 70.00th=[45351], 80.00th=[45351], 90.00th=[84411], 95.00th=[84411], | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 8192, max= 8192, per=44.40%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6534KiB/s (6691kB/s)(7168KiB/1097msec) clat (msec): min=29, max=499, avg=125.71, stdev=169.94 lat (msec): min=29, max=499, avg=125.80, stdev=169.93 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 34], | 30.00th=[ 38], 40.00th=[ 38], 50.00th=[ 43], 60.00th=[ 99], | 70.00th=[ 99], 80.00th=[ 140], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 4096, max= 8192, per=23.79%, avg=6144.00, stdev=2896.31, samples=2 iops : min= 4, max= 8, avg= 6.00, stdev= 2.83, samples=2 lat (msec) : 20=8.33%, 50=58.33%, 100=16.67%, 250=8.33%, 500=8.33% cpu : usr=0.09%, sys=7.85%, ctx=46, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.0MiB/s (18.9MB/s), 1900KiB/s-7666KiB/s (1945kB/s-7850kB/s), io=20.0MiB (20.0MB), run=935-1110msec WRITE: bw=25.2MiB/s (26.5MB/s), 5476KiB/s-9499KiB/s (5607kB/s-9727kB/s), io=28.0MiB (29.4MB), run=935-1110msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6850: Tue Mar 25 09:34:52 2025 read: IOPS=7, BW=7601KiB/s (7784kB/s)(7168KiB/943msec) clat (usec): min=615, max=49627, avg=30000.36, stdev=14699.28 lat (usec): min=616, max=49628, avg=30001.29, stdev=14699.17 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 619], 10.00th=[ 619], 20.00th=[27919], | 30.00th=[31589], 40.00th=[31589], 50.00th=[32113], 60.00th=[32900], | 70.00th=[32900], 80.00th=[35390], 90.00th=[49546], 95.00th=[49546], | 99.00th=[49546], 99.50th=[49546], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 6144, max= 6144, per=34.86%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5429KiB/s (5560kB/s)(5120KiB/943msec) clat (msec): min=26, max=470, avg=136.95, stdev=187.34 lat (msec): min=26, max=470, avg=136.99, stdev=187.32 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 48], 40.00th=[ 48], 50.00th=[ 57], 60.00th=[ 57], | 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] lat (usec) : 750=8.33% lat (msec) : 50=66.67%, 100=16.67%, 500=8.33% cpu : usr=0.00%, sys=21.87%, ctx=1008, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6851: Tue Mar 25 09:34:52 2025 read: IOPS=1, BW=1762KiB/s (1805kB/s)(2048KiB/1162msec) clat (msec): min=28, max=496, avg=262.55, stdev=331.44 lat (msec): min=28, max=496, avg=262.55, stdev=331.44 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 29], 50.00th=[ 29], 60.00th=[ 498], | 70.00th=[ 498], 80.00th=[ 498], 90.00th=[ 498], 95.00th=[ 498], | 99.00th=[ 498], 99.50th=[ 498], 99.90th=[ 498], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 4031, max= 4031, per=22.87%, avg=4031.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8812KiB/s (9024kB/s)(10.0MiB/1162msec) clat (msec): min=19, max=231, avg=62.28, stdev=63.26 lat (msec): min=19, max=231, avg=62.34, stdev=63.25 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 26], | 30.00th=[ 27], 40.00th=[ 30], 50.00th=[ 31], 60.00th=[ 44], | 70.00th=[ 70], 80.00th=[ 70], 90.00th=[ 80], 95.00th=[ 232], | 99.00th=[ 232], 99.50th=[ 232], 99.90th=[ 232], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min=12094, max=12094, per=49.02%, avg=12094.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=25.00%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=25.93%, ctx=499, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6852: Tue Mar 25 09:34:52 2025 read: IOPS=9, BW=9615KiB/s (9846kB/s)(6144KiB/639msec) clat (msec): min=17, max=182, avg=52.93, stdev=63.92 lat (msec): min=17, max=182, avg=52.93, stdev=63.92 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 25], | 30.00th=[ 25], 40.00th=[ 28], 50.00th=[ 28], 60.00th=[ 32], | 70.00th=[ 33], 80.00th=[ 33], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 6144, max= 6144, per=34.86%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9615KiB/s (9846kB/s)(6144KiB/639msec) clat (usec): min=17587, max=72179, avg=48579.55, stdev=19963.77 lat (usec): min=17624, max=72238, avg=48831.46, stdev=19866.89 clat percentiles (usec): | 1.00th=[17695], 5.00th=[17695], 10.00th=[17695], 20.00th=[39584], | 30.00th=[39584], 40.00th=[39584], 50.00th=[39584], 60.00th=[61080], | 70.00th=[61080], 80.00th=[61080], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 8192, max= 8192, per=33.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=50.00%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=35.42%, ctx=825, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6853: Tue Mar 25 09:34:52 2025 read: IOPS=4, BW=4835KiB/s (4951kB/s)(5120KiB/1059msec) clat (usec): min=597, max=45807, avg=21588.97, stdev=20213.54 lat (usec): min=598, max=45807, avg=21590.04, stdev=20213.48 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 594], 10.00th=[ 594], 20.00th=[ 594], | 30.00th=[ 627], 40.00th=[ 627], 50.00th=[28181], 60.00th=[28181], | 70.00th=[32637], 80.00th=[32637], 90.00th=[45876], 95.00th=[45876], | 99.00th=[45876], 99.50th=[45876], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 8175, max= 8175, per=46.39%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6769KiB/s (6931kB/s)(7168KiB/1059msec) clat (msec): min=25, max=560, avg=128.07, stdev=192.62 lat (msec): min=25, max=560, avg=128.13, stdev=192.62 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 32], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 54], 60.00th=[ 72], | 70.00th=[ 72], 80.00th=[ 107], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 4096, max= 8175, per=24.87%, avg=6135.50, stdev=2884.29, samples=2 iops : min= 4, max= 7, avg= 5.50, stdev= 2.12, samples=2 lat (usec) : 750=16.67% lat (msec) : 50=50.00%, 100=16.67%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=18.90%, ctx=460, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.2MiB/s (18.0MB/s), 1762KiB/s-9615KiB/s (1805kB/s-9846kB/s), io=20.0MiB (20.0MB), run=639-1162msec WRITE: bw=24.1MiB/s (25.3MB/s), 5429KiB/s-9615KiB/s (5560kB/s-9846kB/s), io=28.0MiB (29.4MB), run=639-1162msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6860: Tue Mar 25 09:34:54 2025 read: IOPS=4, BW=18.4MiB/s (19.3MB/s)(12.0MiB/653msec) clat (msec): min=131, max=244, avg=205.55, stdev=63.97 lat (msec): min=131, max=244, avg=205.55, stdev=63.97 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 132], 10.00th=[ 132], 20.00th=[ 132], | 30.00th=[ 132], 40.00th=[ 241], 50.00th=[ 241], 60.00th=[ 241], | 70.00th=[ 245], 80.00th=[ 245], 90.00th=[ 245], 95.00th=[ 245], | 99.00th=[ 245], 99.50th=[ 245], 99.90th=[ 245], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 8192, max= 8192, per=32.65%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=11.35%, ctx=39, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6861: Tue Mar 25 09:34:54 2025 read: IOPS=1, BW=7123KiB/s (7294kB/s)(4096KiB/575msec) clat (nsec): min=42745k, max=42745k, avg=42744672.00, stdev= 0.00 lat (nsec): min=42747k, max=42747k, avg=42746832.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[42730], 5.00th=[42730], 10.00th=[42730], 20.00th=[42730], | 30.00th=[42730], 40.00th=[42730], 50.00th=[42730], 60.00th=[42730], | 70.00th=[42730], 80.00th=[42730], 90.00th=[42730], 95.00th=[42730], | 99.00th=[42730], 99.50th=[42730], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 8175, max= 8175, per=32.58%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=3, BW=13.9MiB/s (14.6MB/s)(8192KiB/575msec) clat (msec): min=217, max=273, avg=245.34, stdev=39.67 lat (msec): min=217, max=273, avg=245.53, stdev=39.60 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 218], 10.00th=[ 218], 20.00th=[ 218], | 30.00th=[ 218], 40.00th=[ 218], 50.00th=[ 218], 60.00th=[ 275], | 70.00th=[ 275], 80.00th=[ 275], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 8175, max= 8175, per=14.35%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.17%, sys=12.20%, ctx=67, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6862: Tue Mar 25 09:34:54 2025 write: IOPS=5, BW=21.1MiB/s (22.2MB/s)(12.0MiB/568msec) clat (msec): min=111, max=203, avg=169.70, stdev=50.44 lat (msec): min=111, max=203, avg=169.90, stdev=50.49 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 194], 50.00th=[ 194], 60.00th=[ 194], | 70.00th=[ 203], 80.00th=[ 203], 90.00th=[ 203], 95.00th=[ 203], | 99.00th=[ 203], 99.50th=[ 203], 99.90th=[ 203], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min=16253, max=16253, per=28.52%, avg=16253.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=25.57%, ctx=79, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6863: Tue Mar 25 09:34:54 2025 write: IOPS=5, BW=21.2MiB/s (22.2MB/s)(12.0MiB/566msec) clat (msec): min=130, max=205, avg=178.43, stdev=41.88 lat (msec): min=130, max=205, avg=178.66, stdev=41.86 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 131], 10.00th=[ 131], 20.00th=[ 131], | 30.00th=[ 131], 40.00th=[ 201], 50.00th=[ 201], 60.00th=[ 201], | 70.00th=[ 205], 80.00th=[ 205], 90.00th=[ 205], 95.00th=[ 205], | 99.00th=[ 205], 99.50th=[ 205], 99.90th=[ 205], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min=16351, max=16351, per=28.69%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=20.53%, ctx=110, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=24.5MiB/s (25.7MB/s), 7123KiB/s-18.4MiB/s (7294kB/s-19.3MB/s), io=16.0MiB (16.8MB), run=575-653msec WRITE: bw=55.7MiB/s (58.4MB/s), 13.9MiB/s-21.2MiB/s (14.6MB/s-22.2MB/s), io=32.0MiB (33.6MB), run=566-575msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6864: Tue Mar 25 09:34:54 2025 read: IOPS=8, BW=32.5MiB/s (34.1MB/s)(12.0MiB/369msec) clat (msec): min=70, max=173, avg=116.31, stdev=52.45 lat (msec): min=70, max=173, avg=116.31, stdev=52.45 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 71], 10.00th=[ 71], 20.00th=[ 71], | 30.00th=[ 71], 40.00th=[ 106], 50.00th=[ 106], 60.00th=[ 106], | 70.00th=[ 174], 80.00th=[ 174], 90.00th=[ 174], 95.00th=[ 174], | 99.00th=[ 174], 99.50th=[ 174], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=34.24%, ctx=1246, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6865: Tue Mar 25 09:34:54 2025 read: IOPS=1, BW=5558KiB/s (5691kB/s)(4096KiB/737msec) clat (nsec): min=126150k, max=126150k, avg=126149719.00, stdev= 0.00 lat (nsec): min=126151k, max=126151k, avg=126151459.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 127], 10.00th=[ 127], 20.00th=[ 127], | 30.00th=[ 127], 40.00th=[ 127], 50.00th=[ 127], 60.00th=[ 127], | 70.00th=[ 127], 80.00th=[ 127], 90.00th=[ 127], 95.00th=[ 127], | 99.00th=[ 127], 99.50th=[ 127], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 7831, max= 7831, per=35.23%, avg=7831.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.9MiB/s (11.4MB/s)(8192KiB/737msec) clat (msec): min=288, max=306, avg=297.56, stdev=12.36 lat (msec): min=288, max=306, avg=297.75, stdev=12.47 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 288], 10.00th=[ 288], 20.00th=[ 288], | 30.00th=[ 288], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 309], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 7831, max= 7831, per=17.78%, avg=7831.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=25.14%, ctx=770, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6866: Tue Mar 25 09:34:54 2025 write: IOPS=4, BW=16.1MiB/s (16.9MB/s)(12.0MiB/744msec) clat (msec): min=99, max=339, avg=245.96, stdev=128.03 lat (msec): min=100, max=339, avg=246.24, stdev=128.03 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 101], 10.00th=[ 101], 20.00th=[ 101], | 30.00th=[ 101], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 7892, max= 7892, per=17.92%, avg=7892.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.00%, sys=35.13%, ctx=63, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6867: Tue Mar 25 09:34:54 2025 write: IOPS=6, BW=27.7MiB/s (29.1MB/s)(12.0MiB/433msec) clat (msec): min=110, max=194, avg=141.58, stdev=45.63 lat (msec): min=111, max=194, avg=141.83, stdev=45.53 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 121], 50.00th=[ 121], 60.00th=[ 121], | 70.00th=[ 194], 80.00th=[ 194], 90.00th=[ 194], 95.00th=[ 194], | 99.00th=[ 194], 99.50th=[ 194], 99.90th=[ 194], 99.95th=[ 194], | 99.99th=[ 194] lat (msec) : 250=100.00% cpu : usr=0.23%, sys=52.78%, ctx=59, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=21.7MiB/s (22.8MB/s), 5558KiB/s-32.5MiB/s (5691kB/s-34.1MB/s), io=16.0MiB (16.8MB), run=369-737msec WRITE: bw=43.0MiB/s (45.1MB/s), 10.9MiB/s-27.7MiB/s (11.4MB/s-29.1MB/s), io=32.0MiB (33.6MB), run=433-744msec