== sanity test 398b: DIO and buffer IO race ============== 21:25:05 (1743211505) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.12807 s, 44.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12366: Fri Mar 28 21:25:20 2025 read: IOPS=125, BW=504KiB/s (516kB/s)(5992KiB/11892msec) clat (usec): min=56, max=29487, avg=3272.69, stdev=2734.24 lat (usec): min=56, max=29488, avg=3273.78, stdev=2733.84 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 69], 10.00th=[ 75], 20.00th=[ 139], | 30.00th=[ 2040], 40.00th=[ 2343], 50.00th=[ 2737], 60.00th=[ 3195], | 70.00th=[ 4047], 80.00th=[ 5276], 90.00th=[ 6915], 95.00th=[ 8356], | 99.00th=[11076], 99.50th=[12256], 99.90th=[20841], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 248, max= 774, per=24.99%, avg=499.39, stdev=110.37, samples=23 iops : min= 62, max= 193, avg=124.70, stdev=27.47, samples=23 write: IOPS=132, BW=529KiB/s (542kB/s)(6296KiB/11892msec) clat (usec): min=473, max=157772, avg=4341.59, stdev=5706.60 lat (usec): min=473, max=157773, avg=4342.50, stdev=5706.60 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 619], | 30.00th=[ 1004], 40.00th=[ 2024], 50.00th=[ 3326], 60.00th=[ 4293], | 70.00th=[ 5538], 80.00th=[ 7111], 90.00th=[ 9503], 95.00th=[ 11994], | 99.00th=[ 17957], 99.50th=[ 20841], 99.90th=[ 52691], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 296, max= 784, per=24.95%, avg=522.00, stdev=95.72, samples=23 iops : min= 74, max= 196, avg=130.35, stdev=23.92, samples=23 lat (usec) : 100=9.44%, 250=0.62%, 500=0.78%, 750=12.70%, 1000=2.15% lat (msec) : 2=8.69%, 4=28.78%, 10=31.41%, 20=5.01%, 50=0.36% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.14%, sys=22.36%, ctx=5774, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12367: Fri Mar 28 21:25:20 2025 read: IOPS=124, BW=497KiB/s (509kB/s)(5940KiB/11951msec) clat (usec): min=52, max=22101, avg=3260.35, stdev=2792.90 lat (usec): min=52, max=22101, avg=3261.43, stdev=2792.42 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 69], 10.00th=[ 76], 20.00th=[ 102], | 30.00th=[ 2073], 40.00th=[ 2376], 50.00th=[ 2737], 60.00th=[ 3163], | 70.00th=[ 3949], 80.00th=[ 5211], 90.00th=[ 6915], 95.00th=[ 8225], | 99.00th=[12518], 99.50th=[14222], 99.90th=[20841], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 224, max= 712, per=24.59%, avg=491.39, stdev=122.41, samples=23 iops : min= 56, max= 178, avg=122.70, stdev=30.59, samples=23 write: IOPS=132, BW=531KiB/s (544kB/s)(6348KiB/11951msec) clat (usec): min=451, max=165067, avg=4383.70, stdev=5930.89 lat (usec): min=452, max=165068, avg=4384.65, stdev=5930.73 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 619], | 30.00th=[ 873], 40.00th=[ 2212], 50.00th=[ 3458], 60.00th=[ 4359], | 70.00th=[ 5407], 80.00th=[ 6980], 90.00th=[ 9503], 95.00th=[ 11731], | 99.00th=[ 19006], 99.50th=[ 24773], 99.90th=[ 49546], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 288, max= 854, per=24.79%, avg=518.52, stdev=112.13, samples=23 iops : min= 72, max= 213, avg=129.48, stdev=27.98, samples=23 lat (usec) : 100=9.51%, 250=1.20%, 500=0.78%, 750=13.28%, 1000=2.70% lat (msec) : 2=6.35%, 4=29.30%, 10=31.32%, 20=5.01%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=22.38%, ctx=5751, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12368: Fri Mar 28 21:25:20 2025 read: IOPS=125, BW=502KiB/s (514kB/s)(6028KiB/12017msec) clat (usec): min=54, max=166957, avg=3432.76, stdev=5206.83 lat (usec): min=55, max=166958, avg=3433.43, stdev=5206.85 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 68], 10.00th=[ 74], 20.00th=[ 103], | 30.00th=[ 2040], 40.00th=[ 2343], 50.00th=[ 2704], 60.00th=[ 3228], | 70.00th=[ 4080], 80.00th=[ 5407], 90.00th=[ 6849], 95.00th=[ 8586], | 99.00th=[ 12256], 99.50th=[ 19268], 99.90th=[ 43254], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 184, max= 710, per=24.58%, avg=491.17, stdev=113.68, samples=23 iops : min= 46, max= 177, avg=122.65, stdev=28.42, samples=23 write: IOPS=130, BW=521KiB/s (533kB/s)(6260KiB/12017msec) clat (usec): min=426, max=35323, avg=4274.42, stdev=4292.28 lat (usec): min=427, max=35324, avg=4275.11, stdev=4292.30 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 594], | 30.00th=[ 824], 40.00th=[ 2114], 50.00th=[ 3425], 60.00th=[ 4293], | 70.00th=[ 5276], 80.00th=[ 7111], 90.00th=[ 9896], 95.00th=[12256], | 99.00th=[18220], 99.50th=[21365], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 240, max= 710, per=24.27%, avg=507.83, stdev=101.39, samples=23 iops : min= 60, max= 177, avg=126.83, stdev=25.30, samples=23 lat (usec) : 100=9.70%, 250=0.94%, 500=1.14%, 750=13.77%, 1000=2.18% lat (msec) : 2=6.45%, 4=29.07%, 10=30.70%, 20=5.50%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=22.05%, ctx=5675, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12369: Fri Mar 28 21:25:20 2025 read: IOPS=130, BW=523KiB/s (535kB/s)(6052KiB/11575msec) clat (usec): min=54, max=34967, avg=3254.10, stdev=2821.86 lat (usec): min=54, max=34967, avg=3256.05, stdev=2821.22 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 69], 10.00th=[ 78], 20.00th=[ 116], | 30.00th=[ 2057], 40.00th=[ 2376], 50.00th=[ 2704], 60.00th=[ 3163], | 70.00th=[ 3884], 80.00th=[ 5145], 90.00th=[ 6718], 95.00th=[ 8356], | 99.00th=[11863], 99.50th=[13435], 99.90th=[22152], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 128, max= 800, per=26.07%, avg=520.87, stdev=132.25, samples=23 iops : min= 32, max= 200, avg=130.17, stdev=33.07, samples=23 write: IOPS=134, BW=539KiB/s (552kB/s)(6236KiB/11575msec) clat (usec): min=425, max=205014, avg=4166.57, stdev=6544.41 lat (usec): min=425, max=205016, avg=4167.65, stdev=6544.27 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 603], | 30.00th=[ 807], 40.00th=[ 1729], 50.00th=[ 3228], 60.00th=[ 4080], | 70.00th=[ 5145], 80.00th=[ 6521], 90.00th=[ 9241], 95.00th=[ 12125], | 99.00th=[ 18220], 99.50th=[ 21890], 99.90th=[ 35390], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 160, max= 808, per=25.70%, avg=537.61, stdev=130.48, samples=23 iops : min= 40, max= 202, avg=134.35, stdev=32.64, samples=23 lat (usec) : 100=9.47%, 250=1.01%, 500=0.85%, 750=13.83%, 1000=2.44% lat (msec) : 2=7.19%, 4=30.37%, 10=29.65%, 20=4.72%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=22.63%, ctx=5526, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1998KiB/s (2046kB/s), 497KiB/s-523KiB/s (509kB/s-535kB/s), io=23.4MiB (24.6MB), run=11575-12017msec WRITE: bw=2092KiB/s (2142kB/s), 521KiB/s-539KiB/s (533kB/s-552kB/s), io=24.6MiB (25.7MB), run=11575-12017msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12371: Fri Mar 28 21:25:44 2025 read: IOPS=42, BW=169KiB/s (173kB/s)(5992KiB/35418msec) clat (usec): min=1549, max=171240, avg=3655.11, stdev=5423.99 lat (usec): min=1550, max=171242, avg=3655.76, stdev=5424.00 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1811], 10.00th=[ 1893], 20.00th=[ 1975], | 30.00th=[ 2057], 40.00th=[ 2180], 50.00th=[ 2311], 60.00th=[ 2638], | 70.00th=[ 3326], 80.00th=[ 3982], 90.00th=[ 6980], 95.00th=[ 10290], | 99.00th=[ 15139], 99.50th=[ 18220], 99.90th=[ 58983], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 64, max= 296, per=25.11%, avg=169.23, stdev=63.63, samples=70 iops : min= 16, max= 74, avg=42.29, stdev=15.93, samples=70 write: IOPS=44, BW=178KiB/s (182kB/s)(6296KiB/35418msec) clat (usec): min=8661, max=62728, avg=18956.61, stdev=6859.12 lat (usec): min=8661, max=62729, avg=18957.42, stdev=6859.09 clat percentiles (usec): | 1.00th=[10159], 5.00th=[11338], 10.00th=[11863], 20.00th=[13042], | 30.00th=[14484], 40.00th=[16188], 50.00th=[17957], 60.00th=[19268], | 70.00th=[20579], 80.00th=[22938], 90.00th=[28705], 95.00th=[33162], | 99.00th=[41681], 99.50th=[44303], 99.90th=[54264], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 55, max= 240, per=25.14%, avg=177.23, stdev=49.62, samples=70 iops : min= 13, max= 60, avg=44.29, stdev=12.44, samples=70 lat (msec) : 2=11.72%, 4=27.38%, 10=7.52%, 20=35.77%, 50=17.48% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.08%, sys=4.87%, ctx=3568, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12372: Fri Mar 28 21:25:44 2025 read: IOPS=41, BW=167KiB/s (171kB/s)(5940KiB/35564msec) clat (usec): min=1523, max=168260, avg=3668.78, stdev=5530.74 lat (usec): min=1524, max=168262, avg=3669.41, stdev=5530.75 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1795], 10.00th=[ 1876], 20.00th=[ 1975], | 30.00th=[ 2057], 40.00th=[ 2180], 50.00th=[ 2376], 60.00th=[ 2704], | 70.00th=[ 3359], 80.00th=[ 3884], 90.00th=[ 6783], 95.00th=[ 9503], | 99.00th=[ 16188], 99.50th=[ 23725], 99.90th=[ 58459], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 16, max= 320, per=24.77%, avg=166.93, stdev=70.24, samples=71 iops : min= 4, max= 80, avg=41.69, stdev=17.55, samples=71 write: IOPS=44, BW=178KiB/s (183kB/s)(6348KiB/35564msec) clat (usec): min=9276, max=54168, avg=18910.46, stdev=6969.38 lat (usec): min=9276, max=54169, avg=18911.23, stdev=6969.41 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11338], 10.00th=[11863], 20.00th=[12911], | 30.00th=[14353], 40.00th=[15926], 50.00th=[17695], 60.00th=[19006], | 70.00th=[20317], 80.00th=[22676], 90.00th=[28967], 95.00th=[33424], | 99.00th=[41157], 99.50th=[45876], 99.90th=[52691], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 72, max= 232, per=25.29%, avg=178.31, stdev=50.10, samples=71 iops : min= 18, max= 58, avg=44.54, stdev=12.52, samples=71 lat (msec) : 2=11.49%, 4=27.83%, 10=6.93%, 20=36.62%, 50=16.96% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.08%, sys=4.84%, ctx=3553, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12373: Fri Mar 28 21:25:44 2025 read: IOPS=42, BW=169KiB/s (173kB/s)(6028KiB/35616msec) clat (usec): min=1520, max=39412, avg=3579.11, stdev=3145.76 lat (usec): min=1521, max=39412, avg=3579.71, stdev=3145.76 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1811], 10.00th=[ 1876], 20.00th=[ 1975], | 30.00th=[ 2057], 40.00th=[ 2180], 50.00th=[ 2442], 60.00th=[ 2933], | 70.00th=[ 3490], 80.00th=[ 4113], 90.00th=[ 6652], 95.00th=[ 9241], | 99.00th=[16581], 99.50th=[19530], 99.90th=[39060], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 47, max= 304, per=25.10%, avg=169.17, stdev=61.93, samples=71 iops : min= 11, max= 76, avg=42.24, stdev=15.53, samples=71 write: IOPS=43, BW=176KiB/s (180kB/s)(6260KiB/35616msec) clat (msec): min=8, max=186, avg=19.24, stdev= 8.26 lat (msec): min=8, max=186, avg=19.24, stdev= 8.26 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 29], 95.00th=[ 36], | 99.00th=[ 44], 99.50th=[ 46], 99.90th=[ 63], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 55, max= 240, per=24.91%, avg=175.59, stdev=49.99, samples=71 iops : min= 13, max= 60, avg=43.85, stdev=12.53, samples=71 lat (msec) : 2=11.46%, 4=27.31%, 10=8.72%, 20=34.86%, 50=17.55% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.04%, sys=4.87%, ctx=3602, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12374: Fri Mar 28 21:25:44 2025 read: IOPS=42, BW=170KiB/s (175kB/s)(6052KiB/35513msec) clat (usec): min=1511, max=40162, avg=3553.14, stdev=3104.57 lat (usec): min=1511, max=40163, avg=3553.76, stdev=3104.57 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1811], 10.00th=[ 1893], 20.00th=[ 1975], | 30.00th=[ 2057], 40.00th=[ 2180], 50.00th=[ 2376], 60.00th=[ 2737], | 70.00th=[ 3392], 80.00th=[ 3916], 90.00th=[ 6783], 95.00th=[10028], | 99.00th=[15008], 99.50th=[19268], 99.90th=[35390], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 16, max= 384, per=25.29%, avg=170.45, stdev=68.81, samples=71 iops : min= 4, max= 96, avg=42.59, stdev=17.20, samples=71 write: IOPS=43, BW=176KiB/s (180kB/s)(6236KiB/35513msec) clat (msec): min=9, max=204, avg=19.26, stdev= 8.62 lat (msec): min=9, max=204, avg=19.27, stdev= 8.62 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 30], 95.00th=[ 34], | 99.00th=[ 46], 99.50th=[ 52], 99.90th=[ 64], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 56, max= 232, per=24.90%, avg=175.52, stdev=49.89, samples=71 iops : min= 14, max= 58, avg=43.86, stdev=12.46, samples=71 lat (msec) : 2=11.30%, 4=28.42%, 10=7.32%, 20=36.00%, 50=16.70% lat (msec) : 100=0.23%, 250=0.03% cpu : usr=0.10%, sys=4.82%, ctx=3577, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=674KiB/s (690kB/s), 167KiB/s-170KiB/s (171kB/s-175kB/s), io=23.4MiB (24.6MB), run=35418-35616msec WRITE: bw=706KiB/s (723kB/s), 176KiB/s-178KiB/s (180kB/s-183kB/s), io=24.6MiB (25.7MB), run=35418-35616msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12418: Fri Mar 28 21:25:47 2025 read: IOPS=156, BW=2504KiB/s (2564kB/s)(5920KiB/2364msec) clat (usec): min=56, max=16216, avg=2813.09, stdev=2377.12 lat (usec): min=57, max=16216, avg=2815.31, stdev=2377.00 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 72], 10.00th=[ 75], 20.00th=[ 90], | 30.00th=[ 2073], 40.00th=[ 2442], 50.00th=[ 2704], 60.00th=[ 3064], | 70.00th=[ 3621], 80.00th=[ 4424], 90.00th=[ 5669], 95.00th=[ 6652], | 99.00th=[12518], 99.50th=[14615], 99.90th=[16188], 99.95th=[16188], | 99.99th=[16188] bw ( KiB/s): min= 1664, max= 2976, per=25.26%, avg=2534.50, stdev=592.21, samples=4 iops : min= 104, max= 186, avg=158.25, stdev=36.92, samples=4 write: IOPS=168, BW=2694KiB/s (2758kB/s)(6368KiB/2364msec) clat (usec): min=620, max=27092, avg=3057.50, stdev=2993.18 lat (usec): min=621, max=27094, avg=3058.30, stdev=2993.20 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 750], 10.00th=[ 799], 20.00th=[ 857], | 30.00th=[ 922], 40.00th=[ 1106], 50.00th=[ 1631], 60.00th=[ 2769], | 70.00th=[ 3851], 80.00th=[ 5211], 90.00th=[ 6980], 95.00th=[ 8717], | 99.00th=[13566], 99.50th=[15008], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 1856, max= 3864, per=26.20%, avg=2766.00, stdev=829.49, samples=4 iops : min= 116, max= 241, avg=172.75, stdev=51.62, samples=4 lat (usec) : 100=11.07%, 250=2.21%, 500=0.39%, 750=2.34%, 1000=16.02% lat (msec) : 2=9.64%, 4=31.12%, 10=25.26%, 20=1.82%, 50=0.13% cpu : usr=0.04%, sys=27.97%, ctx=1602, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12420: Fri Mar 28 21:25:47 2025 read: IOPS=150, BW=2406KiB/s (2464kB/s)(5744KiB/2387msec) clat (usec): min=61, max=21380, avg=2746.67, stdev=2520.38 lat (usec): min=61, max=21381, avg=2747.19, stdev=2520.40 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 71], 10.00th=[ 76], 20.00th=[ 87], | 30.00th=[ 204], 40.00th=[ 2376], 50.00th=[ 2671], 60.00th=[ 2966], | 70.00th=[ 3687], 80.00th=[ 4555], 90.00th=[ 5473], 95.00th=[ 6456], | 99.00th=[ 9896], 99.50th=[17957], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 1664, max= 2720, per=23.76%, avg=2384.00, stdev=493.33, samples=4 iops : min= 104, max= 170, avg=149.00, stdev=30.83, samples=4 write: IOPS=171, BW=2742KiB/s (2807kB/s)(6544KiB/2387msec) clat (usec): min=670, max=31112, avg=3209.87, stdev=3925.68 lat (usec): min=671, max=31114, avg=3210.72, stdev=3925.73 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 758], 10.00th=[ 799], 20.00th=[ 865], | 30.00th=[ 947], 40.00th=[ 1172], 50.00th=[ 1647], 60.00th=[ 2835], | 70.00th=[ 3621], 80.00th=[ 5014], 90.00th=[ 6783], 95.00th=[ 8455], | 99.00th=[26084], 99.50th=[28443], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 1888, max= 3200, per=25.08%, avg=2648.00, stdev=549.54, samples=4 iops : min= 118, max= 200, avg=165.50, stdev=34.35, samples=4 lat (usec) : 100=11.59%, 250=2.47%, 500=0.13%, 750=2.21%, 1000=14.97% lat (msec) : 2=11.33%, 4=31.25%, 10=23.96%, 20=1.17%, 50=0.91% cpu : usr=0.17%, sys=27.52%, ctx=1570, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12422: Fri Mar 28 21:25:47 2025 read: IOPS=170, BW=2725KiB/s (2790kB/s)(6256KiB/2296msec) clat (usec): min=58, max=18679, avg=2818.84, stdev=2432.90 lat (usec): min=59, max=18679, avg=2819.52, stdev=2433.02 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 72], 10.00th=[ 76], 20.00th=[ 93], | 30.00th=[ 2073], 40.00th=[ 2343], 50.00th=[ 2671], 60.00th=[ 3130], | 70.00th=[ 3785], 80.00th=[ 4490], 90.00th=[ 5538], 95.00th=[ 6456], | 99.00th=[12911], 99.50th=[14746], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 1568, max= 3168, per=26.63%, avg=2672.00, stdev=748.87, samples=4 iops : min= 98, max= 198, avg=167.00, stdev=46.80, samples=4 write: IOPS=164, BW=2627KiB/s (2690kB/s)(6032KiB/2296msec) clat (usec): min=625, max=32552, avg=3024.88, stdev=3459.77 lat (usec): min=626, max=32554, avg=3025.71, stdev=3459.82 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 750], 10.00th=[ 807], 20.00th=[ 881], | 30.00th=[ 955], 40.00th=[ 1057], 50.00th=[ 1516], 60.00th=[ 2704], | 70.00th=[ 3556], 80.00th=[ 4817], 90.00th=[ 6849], 95.00th=[ 8094], | 99.00th=[17171], 99.50th=[31065], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 1952, max= 3008, per=24.93%, avg=2632.00, stdev=483.45, samples=4 iops : min= 122, max= 188, avg=164.50, stdev=30.22, samples=4 lat (usec) : 100=11.33%, 250=2.47%, 500=0.52%, 750=2.47%, 1000=15.10% lat (msec) : 2=9.11%, 4=32.03%, 10=25.00%, 20=1.56%, 50=0.39% cpu : usr=0.26%, sys=28.41%, ctx=1723, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12423: Fri Mar 28 21:25:47 2025 read: IOPS=168, BW=2695KiB/s (2760kB/s)(6032KiB/2238msec) clat (usec): min=59, max=23376, avg=2868.73, stdev=2685.27 lat (usec): min=60, max=23377, avg=2869.30, stdev=2685.30 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 72], 10.00th=[ 75], 20.00th=[ 86], | 30.00th=[ 174], 40.00th=[ 2376], 50.00th=[ 2737], 60.00th=[ 3228], | 70.00th=[ 3818], 80.00th=[ 4424], 90.00th=[ 5604], 95.00th=[ 7177], | 99.00th=[12649], 99.50th=[17957], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 2240, max= 3129, per=26.36%, avg=2644.75, stdev=372.05, samples=4 iops : min= 140, max= 195, avg=165.00, stdev=23.08, samples=4 write: IOPS=174, BW=2795KiB/s (2862kB/s)(6256KiB/2238msec) clat (usec): min=650, max=16423, avg=2909.60, stdev=2701.94 lat (usec): min=651, max=16423, avg=2911.54, stdev=2703.54 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 766], 10.00th=[ 816], 20.00th=[ 881], | 30.00th=[ 996], 40.00th=[ 1221], 50.00th=[ 1713], 60.00th=[ 2737], | 70.00th=[ 3392], 80.00th=[ 4817], 90.00th=[ 6390], 95.00th=[ 8160], | 99.00th=[13960], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450], | 99.99th=[16450] bw ( KiB/s): min= 2400, max= 3072, per=26.34%, avg=2781.00, stdev=281.96, samples=4 iops : min= 150, max= 192, avg=173.50, stdev=17.56, samples=4 lat (usec) : 100=12.37%, 250=2.60%, 500=0.13%, 750=2.21%, 1000=13.28% lat (msec) : 2=11.72%, 4=30.86%, 10=24.87%, 20=1.82%, 50=0.13% cpu : usr=0.31%, sys=29.82%, ctx=1636, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9.80MiB/s (10.3MB/s), 2406KiB/s-2725KiB/s (2464kB/s-2790kB/s), io=23.4MiB (24.5MB), run=2238-2387msec WRITE: bw=10.3MiB/s (10.8MB/s), 2627KiB/s-2795KiB/s (2690kB/s-2862kB/s), io=24.6MiB (25.8MB), run=2238-2387msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12416: Fri Mar 28 21:25:54 2025 read: IOPS=41, BW=661KiB/s (677kB/s)(5920KiB/8956msec) clat (usec): min=1797, max=25596, avg=3925.20, stdev=2857.30 lat (usec): min=1797, max=25597, avg=3925.79, stdev=2857.30 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 1991], 10.00th=[ 2040], 20.00th=[ 2147], | 30.00th=[ 2245], 40.00th=[ 2409], 50.00th=[ 2638], 60.00th=[ 3490], | 70.00th=[ 4490], 80.00th=[ 5538], 90.00th=[ 6194], 95.00th=[ 7832], | 99.00th=[17957], 99.50th=[19792], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 352, max= 1152, per=25.25%, avg=656.94, stdev=204.31, samples=17 iops : min= 22, max= 72, avg=41.06, stdev=12.77, samples=17 write: IOPS=44, BW=711KiB/s (728kB/s)(6368KiB/8956msec) clat (usec): min=10175, max=44006, avg=18591.42, stdev=5868.86 lat (usec): min=10177, max=44007, avg=18592.77, stdev=5869.07 clat percentiles (usec): | 1.00th=[10552], 5.00th=[11600], 10.00th=[12256], 20.00th=[13173], | 30.00th=[14222], 40.00th=[16319], 50.00th=[17957], 60.00th=[19530], | 70.00th=[20579], 80.00th=[22414], 90.00th=[26608], 95.00th=[30540], | 99.00th=[38536], 99.50th=[42730], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 384, max= 928, per=25.71%, avg=704.00, stdev=163.95, samples=17 iops : min= 24, max= 58, avg=44.00, stdev=10.25, samples=17 lat (msec) : 2=3.12%, 4=28.91%, 10=13.93%, 20=34.64%, 50=19.40% cpu : usr=0.09%, sys=6.15%, ctx=989, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12417: Fri Mar 28 21:25:54 2025 read: IOPS=39, BW=624KiB/s (639kB/s)(5744KiB/9202msec) clat (usec): min=1736, max=20257, avg=4221.66, stdev=2959.23 lat (usec): min=1737, max=20258, avg=4222.27, stdev=2959.24 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2040], 10.00th=[ 2114], 20.00th=[ 2212], | 30.00th=[ 2311], 40.00th=[ 2507], 50.00th=[ 2769], 60.00th=[ 3949], | 70.00th=[ 5080], 80.00th=[ 5735], 90.00th=[ 6980], 95.00th=[10290], | 99.00th=[16188], 99.50th=[17695], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 288, max= 1184, per=24.03%, avg=625.28, stdev=213.56, samples=18 iops : min= 18, max= 74, avg=39.00, stdev=13.41, samples=18 write: IOPS=44, BW=711KiB/s (728kB/s)(6544KiB/9202msec) clat (usec): min=9906, max=46564, avg=18682.75, stdev=5774.68 lat (usec): min=9907, max=46565, avg=18684.08, stdev=5774.69 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11994], 10.00th=[12518], 20.00th=[13566], | 30.00th=[14877], 40.00th=[16188], 50.00th=[18220], 60.00th=[19792], | 70.00th=[20579], 80.00th=[21627], 90.00th=[25822], 95.00th=[30540], | 99.00th=[38011], 99.50th=[42206], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 479, max= 864, per=26.00%, avg=711.83, stdev=142.91, samples=18 iops : min= 29, max= 54, avg=44.39, stdev= 8.98, samples=18 lat (msec) : 2=1.56%, 4=27.08%, 10=15.76%, 20=35.55%, 50=20.05% cpu : usr=0.05%, sys=6.19%, ctx=1034, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12419: Fri Mar 28 21:25:54 2025 read: IOPS=44, BW=717KiB/s (734kB/s)(6256KiB/8727msec) clat (usec): min=1819, max=20190, avg=3907.06, stdev=2627.42 lat (usec): min=1819, max=20191, avg=3907.74, stdev=2627.40 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2008], 10.00th=[ 2089], 20.00th=[ 2180], | 30.00th=[ 2245], 40.00th=[ 2376], 50.00th=[ 2638], 60.00th=[ 3294], | 70.00th=[ 4621], 80.00th=[ 5538], 90.00th=[ 6390], 95.00th=[ 9110], | 99.00th=[15926], 99.50th=[16909], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 351, max= 1277, per=27.84%, avg=724.47, stdev=309.98, samples=17 iops : min= 21, max= 79, avg=45.18, stdev=19.36, samples=17 write: IOPS=43, BW=691KiB/s (708kB/s)(6032KiB/8727msec) clat (usec): min=10084, max=42885, avg=19059.12, stdev=6041.87 lat (usec): min=10085, max=42886, avg=19060.44, stdev=6041.92 clat percentiles (usec): | 1.00th=[10814], 5.00th=[12125], 10.00th=[12780], 20.00th=[13829], | 30.00th=[15008], 40.00th=[16319], 50.00th=[17957], 60.00th=[19792], | 70.00th=[21103], 80.00th=[23200], 90.00th=[27919], 95.00th=[30802], | 99.00th=[40109], 99.50th=[41681], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 448, max= 864, per=25.09%, avg=686.88, stdev=140.87, samples=17 iops : min= 28, max= 54, avg=42.82, stdev= 8.84, samples=17 lat (msec) : 2=1.56%, 4=31.77%, 10=15.49%, 20=32.55%, 50=18.62% cpu : usr=0.06%, sys=6.22%, ctx=1039, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12421: Fri Mar 28 21:25:54 2025 read: IOPS=42, BW=676KiB/s (692kB/s)(6032KiB/8929msec) clat (usec): min=1848, max=17291, avg=3835.08, stdev=2483.95 lat (usec): min=1848, max=17292, avg=3835.68, stdev=2483.95 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2008], 10.00th=[ 2057], 20.00th=[ 2180], | 30.00th=[ 2278], 40.00th=[ 2442], 50.00th=[ 2671], 60.00th=[ 3326], | 70.00th=[ 4686], 80.00th=[ 5538], 90.00th=[ 6259], 95.00th=[ 7701], | 99.00th=[15926], 99.50th=[17171], 99.90th=[17171], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 288, max= 1024, per=25.75%, avg=669.94, stdev=194.20, samples=17 iops : min= 18, max= 64, avg=41.76, stdev=12.22, samples=17 write: IOPS=43, BW=701KiB/s (717kB/s)(6256KiB/8929msec) clat (usec): min=9794, max=45640, avg=18864.84, stdev=6034.49 lat (usec): min=9795, max=45641, avg=18866.32, stdev=6034.55 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11863], 10.00th=[12649], 20.00th=[13566], | 30.00th=[15008], 40.00th=[16188], 50.00th=[17957], 60.00th=[19530], | 70.00th=[20841], 80.00th=[22414], 90.00th=[26084], 95.00th=[31851], | 99.00th=[39584], 99.50th=[43254], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 351, max= 864, per=25.43%, avg=696.29, stdev=148.72, samples=17 iops : min= 21, max= 54, avg=43.41, stdev= 9.42, samples=17 lat (msec) : 2=2.08%, 4=30.34%, 10=15.10%, 20=33.98%, 50=18.49% cpu : usr=0.06%, sys=6.01%, ctx=1036, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2603KiB/s (2665kB/s), 624KiB/s-717KiB/s (639kB/s-734kB/s), io=23.4MiB (24.5MB), run=8727-9202msec WRITE: bw=2739KiB/s (2804kB/s), 691KiB/s-711KiB/s (708kB/s-728kB/s), io=24.6MiB (25.8MB), run=8727-9202msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12437: Fri Mar 28 21:25:56 2025 read: IOPS=5, BW=5217KiB/s (5342kB/s)(7168KiB/1374msec) clat (msec): min=29, max=917, avg=166.30, stdev=331.59 lat (msec): min=29, max=917, avg=166.30, stdev=331.59 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 45], 60.00th=[ 50], | 70.00th=[ 50], 80.00th=[ 61], 90.00th=[ 919], 95.00th=[ 919], | 99.00th=[ 919], 99.50th=[ 919], 99.90th=[ 919], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 2043, max= 2048, per=13.95%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3726KiB/s (3816kB/s)(5120KiB/1374msec) clat (usec): min=35357, max=45900, avg=39733.21, stdev=4082.27 lat (usec): min=35413, max=45917, avg=39769.79, stdev=4066.52 clat percentiles (usec): | 1.00th=[35390], 5.00th=[35390], 10.00th=[35390], 20.00th=[35390], | 30.00th=[36963], 40.00th=[36963], 50.00th=[39060], 60.00th=[39060], | 70.00th=[41157], 80.00th=[41157], 90.00th=[45876], 95.00th=[45876], | 99.00th=[45876], 99.50th=[45876], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] lat (msec) : 50=83.33%, 100=8.33%, 1000=8.33% cpu : usr=0.00%, sys=25.05%, ctx=740, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12438: Fri Mar 28 21:25:56 2025 read: IOPS=1, BW=1466KiB/s (1501kB/s)(2048KiB/1397msec) clat (usec): min=34879, max=39554, avg=37217.03, stdev=3305.64 lat (usec): min=34880, max=39555, avg=37217.71, stdev=3305.91 clat percentiles (usec): | 1.00th=[34866], 5.00th=[34866], 10.00th=[34866], 20.00th=[34866], | 30.00th=[34866], 40.00th=[34866], 50.00th=[34866], 60.00th=[39584], | 70.00th=[39584], 80.00th=[39584], 90.00th=[39584], 95.00th=[39584], | 99.00th=[39584], 99.50th=[39584], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 2048, max= 2048, per=13.97%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7330KiB/s (7506kB/s)(10.0MiB/1397msec) clat (msec): min=19, max=1007, avg=130.03, stdev=308.40 lat (msec): min=19, max=1007, avg=130.08, stdev=308.39 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 20], | 30.00th=[ 27], 40.00th=[ 28], 50.00th=[ 29], 60.00th=[ 30], | 70.00th=[ 44], 80.00th=[ 44], 90.00th=[ 60], 95.00th=[ 1011], | 99.00th=[ 1011], 99.50th=[ 1011], 99.90th=[ 1011], 99.95th=[ 1011], | 99.99th=[ 1011] lat (msec) : 20=16.67%, 50=66.67%, 100=8.33% cpu : usr=0.00%, sys=14.04%, ctx=487, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12440: Fri Mar 28 21:25:56 2025 read: IOPS=4, BW=4455KiB/s (4562kB/s)(6144KiB/1379msec) clat (usec): min=271, max=28345, avg=14834.25, stdev=13292.02 lat (usec): min=271, max=28346, avg=14834.90, stdev=13292.07 clat percentiles (usec): | 1.00th=[ 273], 5.00th=[ 273], 10.00th=[ 273], 20.00th=[ 2343], | 30.00th=[ 2343], 40.00th=[ 5932], 50.00th=[ 5932], 60.00th=[25822], | 70.00th=[26346], 80.00th=[26346], 90.00th=[28443], 95.00th=[28443], | 99.00th=[28443], 99.50th=[28443], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] write: IOPS=4, BW=4455KiB/s (4562kB/s)(6144KiB/1379msec) clat (msec): min=15, max=1058, avg=213.78, stdev=415.07 lat (msec): min=15, max=1058, avg=213.82, stdev=415.06 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 23], | 30.00th=[ 23], 40.00th=[ 42], 50.00th=[ 42], 60.00th=[ 44], | 70.00th=[ 101], 80.00th=[ 101], 90.00th=[ 1062], 95.00th=[ 1062], | 99.00th=[ 1062], 99.50th=[ 1062], 99.90th=[ 1062], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 4096, max= 4096, per=19.96%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 4=8.33%, 10=8.33%, 20=8.33%, 50=50.00%, 250=8.33% cpu : usr=0.00%, sys=9.51%, ctx=341, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12442: Fri Mar 28 21:25:56 2025 read: IOPS=3, BW=3801KiB/s (3892kB/s)(5120KiB/1347msec) clat (usec): min=385, max=23917, avg=10438.00, stdev=11435.50 lat (usec): min=386, max=23918, avg=10438.58, stdev=11435.49 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 388], 10.00th=[ 388], 20.00th=[ 388], | 30.00th=[ 2040], 40.00th=[ 2040], 50.00th=[ 4047], 60.00th=[ 4047], | 70.00th=[21890], 80.00th=[21890], 90.00th=[23987], 95.00th=[23987], | 99.00th=[23987], 99.50th=[23987], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] write: IOPS=5, BW=5321KiB/s (5449kB/s)(7168KiB/1347msec) clat (msec): min=18, max=1042, avg=183.22, stdev=379.05 lat (msec): min=18, max=1042, avg=183.26, stdev=379.05 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 32], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 42], 60.00th=[ 43], | 70.00th=[ 43], 80.00th=[ 73], 90.00th=[ 1045], 95.00th=[ 1045], | 99.00th=[ 1045], 99.50th=[ 1045], 99.90th=[ 1045], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 2048, max= 2048, per=9.98%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 4=8.33%, 10=8.33%, 20=8.33%, 50=50.00%, 100=8.33% cpu : usr=0.00%, sys=12.11%, ctx=63, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=14.3MiB/s (15.0MB/s), 1466KiB/s-5217KiB/s (1501kB/s-5342kB/s), io=20.0MiB (20.0MB), run=1347-1397msec WRITE: bw=20.0MiB/s (21.0MB/s), 3726KiB/s-7330KiB/s (3816kB/s-7506kB/s), io=28.0MiB (29.4MB), run=1347-1397msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12439: Fri Mar 28 21:25:56 2025 read: IOPS=4, BW=4441KiB/s (4548kB/s)(7168KiB/1614msec) clat (msec): min=16, max=895, avg=187.47, stdev=324.93 lat (msec): min=16, max=895, avg=187.47, stdev=324.93 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 18], | 30.00th=[ 23], 40.00th=[ 23], 50.00th=[ 26], 60.00th=[ 67], | 70.00th=[ 67], 80.00th=[ 271], 90.00th=[ 894], 95.00th=[ 894], | 99.00th=[ 894], 99.50th=[ 894], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 2048, max= 8192, per=40.35%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=3, BW=3172KiB/s (3248kB/s)(5120KiB/1614msec) clat (usec): min=22903, max=71383, avg=40240.54, stdev=18797.64 lat (usec): min=22972, max=71448, avg=40315.45, stdev=18794.43 clat percentiles (usec): | 1.00th=[22938], 5.00th=[22938], 10.00th=[22938], 20.00th=[22938], | 30.00th=[29492], 40.00th=[29492], 50.00th=[35390], 60.00th=[35390], | 70.00th=[42206], 80.00th=[42206], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 4096, max= 4096, per=23.06%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=50.00%, 100=16.67%, 500=8.33%, 1000=8.33% cpu : usr=0.06%, sys=4.46%, ctx=48, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12441: Fri Mar 28 21:25:56 2025 read: IOPS=1, BW=1284KiB/s (1315kB/s)(2048KiB/1595msec) clat (usec): min=13119, max=84204, avg=48661.84, stdev=50264.46 lat (usec): min=13120, max=84205, avg=48662.67, stdev=50264.63 clat percentiles (usec): | 1.00th=[13173], 5.00th=[13173], 10.00th=[13173], 20.00th=[13173], | 30.00th=[13173], 40.00th=[13173], 50.00th=[13173], 60.00th=[84411], | 70.00th=[84411], 80.00th=[84411], 90.00th=[84411], 95.00th=[84411], | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 2048, max= 2048, per=16.14%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=6, BW=6420KiB/s (6574kB/s)(10.0MiB/1595msec) clat (msec): min=27, max=1054, avg=148.27, stdev=320.29 lat (msec): min=27, max=1055, avg=148.34, stdev=320.28 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 31], | 30.00th=[ 32], 40.00th=[ 33], 50.00th=[ 37], 60.00th=[ 42], | 70.00th=[ 46], 80.00th=[ 46], 90.00th=[ 140], 95.00th=[ 1053], | 99.00th=[ 1053], 99.50th=[ 1053], 99.90th=[ 1053], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min=14336, max=14336, per=80.70%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=66.67%, 100=8.33%, 250=8.33% cpu : usr=0.06%, sys=5.02%, ctx=52, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12443: Fri Mar 28 21:25:56 2025 read: IOPS=3, BW=3936KiB/s (4030kB/s)(6144KiB/1561msec) clat (usec): min=15222, max=40984, avg=26184.12, stdev=11020.55 lat (usec): min=15223, max=40985, avg=26184.85, stdev=11020.66 clat percentiles (usec): | 1.00th=[15270], 5.00th=[15270], 10.00th=[15270], 20.00th=[16188], | 30.00th=[16188], 40.00th=[20317], 50.00th=[20317], 60.00th=[26608], | 70.00th=[37487], 80.00th=[37487], 90.00th=[41157], 95.00th=[41157], | 99.00th=[41157], 99.50th=[41157], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min=10240, max=10240, per=80.71%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3936KiB/s (4030kB/s)(6144KiB/1561msec) clat (msec): min=35, max=1087, avg=222.19, stdev=424.18 lat (msec): min=35, max=1087, avg=222.28, stdev=424.25 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 43], 50.00th=[ 43], 60.00th=[ 44], | 70.00th=[ 89], 80.00th=[ 89], 90.00th=[ 1083], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 4096, max= 6144, per=28.82%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 20=16.67%, 50=66.67%, 100=8.33% cpu : usr=0.06%, sys=6.03%, ctx=55, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12444: Fri Mar 28 21:25:56 2025 read: IOPS=3, BW=3206KiB/s (3283kB/s)(5120KiB/1597msec) clat (usec): min=11953, max=68205, avg=25122.10, stdev=24306.92 lat (usec): min=11954, max=68205, avg=25122.87, stdev=24307.00 clat percentiles (usec): | 1.00th=[11994], 5.00th=[11994], 10.00th=[11994], 20.00th=[11994], | 30.00th=[12125], 40.00th=[12125], 50.00th=[13435], 60.00th=[13435], | 70.00th=[20055], 80.00th=[20055], 90.00th=[68682], 95.00th=[68682], | 99.00th=[68682], 99.50th=[68682], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 8192, max= 8192, per=64.56%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4488KiB/s (4596kB/s)(7168KiB/1597msec) clat (msec): min=22, max=1075, avg=204.24, stdev=385.19 lat (msec): min=22, max=1075, avg=204.30, stdev=385.20 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 41], | 30.00th=[ 45], 40.00th=[ 45], 50.00th=[ 65], 60.00th=[ 77], | 70.00th=[ 77], 80.00th=[ 107], 90.00th=[ 1083], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 2043, max= 6144, per=23.04%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (msec) : 20=33.33%, 50=25.00%, 100=25.00%, 250=8.33% cpu : usr=0.06%, sys=5.14%, ctx=44, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.4MiB/s (12.0MB/s), 1284KiB/s-4441KiB/s (1315kB/s-4548kB/s), io=20.0MiB (20.0MB), run=1561-1614msec WRITE: bw=17.3MiB/s (18.2MB/s), 3172KiB/s-6420KiB/s (3248kB/s-6574kB/s), io=28.0MiB (29.4MB), run=1561-1614msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12450: Fri Mar 28 21:25:59 2025 read: IOPS=2, BW=9668KiB/s (9900kB/s)(12.0MiB/1271msec) clat (msec): min=41, max=1051, avg=419.52, stdev=550.65 lat (msec): min=41, max=1051, avg=419.52, stdev=550.65 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 167], | 70.00th=[ 1053], 80.00th=[ 1053], 90.00th=[ 1053], 95.00th=[ 1053], | 99.00th=[ 1053], 99.50th=[ 1053], 99.90th=[ 1053], 99.95th=[ 1053], | 99.99th=[ 1053] lat (msec) : 50=33.33%, 250=33.33% cpu : usr=0.08%, sys=23.70%, ctx=1851, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12451: Fri Mar 28 21:25:59 2025 read: IOPS=0, BW=2743KiB/s (2809kB/s)(4096KiB/1493msec) clat (nsec): min=89495k, max=89495k, avg=89494551.00, stdev= 0.00 lat (nsec): min=89496k, max=89496k, avg=89495773.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[89654], 5.00th=[89654], 10.00th=[89654], 20.00th=[89654], | 30.00th=[89654], 40.00th=[89654], 50.00th=[89654], 60.00th=[89654], | 70.00th=[89654], 80.00th=[89654], 90.00th=[89654], 95.00th=[89654], | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 8192, max= 8192, per=74.66%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5487KiB/s (5619kB/s)(8192KiB/1493msec) clat (msec): min=204, max=1180, avg=692.27, stdev=690.29 lat (msec): min=204, max=1180, avg=692.68, stdev=690.43 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 205], 10.00th=[ 205], 20.00th=[ 205], | 30.00th=[ 205], 40.00th=[ 205], 50.00th=[ 205], 60.00th=[ 1183], | 70.00th=[ 1183], 80.00th=[ 1183], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=14.34%, ctx=549, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12452: Fri Mar 28 21:25:59 2025 write: IOPS=1, BW=7107KiB/s (7278kB/s)(12.0MiB/1729msec) clat (msec): min=113, max=1306, avg=562.91, stdev=649.03 lat (msec): min=113, max=1307, avg=563.23, stdev=649.05 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 113], 10.00th=[ 113], 20.00th=[ 113], | 30.00th=[ 113], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 1301], 80.00th=[ 1301], 90.00th=[ 1301], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 8175, max= 8175, per=43.14%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=14.06%, ctx=82, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12453: Fri Mar 28 21:25:59 2025 write: IOPS=1, BW=7892KiB/s (8082kB/s)(12.0MiB/1557msec) clat (msec): min=83, max=1339, avg=515.50, stdev=713.79 lat (msec): min=83, max=1339, avg=515.72, stdev=713.72 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 84], 10.00th=[ 84], 20.00th=[ 84], | 30.00th=[ 84], 40.00th=[ 124], 50.00th=[ 124], 60.00th=[ 124], | 70.00th=[ 1334], 80.00th=[ 1334], 90.00th=[ 1334], 95.00th=[ 1334], | 99.00th=[ 1334], 99.50th=[ 1334], 99.90th=[ 1334], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min=16384, max=16384, per=86.45%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.06%, sys=15.17%, ctx=69, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.7MiB/s (11.2MB/s), 2743KiB/s-9668KiB/s (2809kB/s-9900kB/s), io=16.0MiB (16.8MB), run=1271-1493msec WRITE: bw=18.5MiB/s (19.4MB/s), 5487KiB/s-7892KiB/s (5619kB/s-8082kB/s), io=32.0MiB (33.6MB), run=1493-1729msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12455: Fri Mar 28 21:25:59 2025 read: IOPS=1, BW=7074KiB/s (7244kB/s)(12.0MiB/1737msec) clat (msec): min=112, max=1338, avg=569.84, stdev=669.76 lat (msec): min=112, max=1338, avg=569.84, stdev=669.76 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 259], 50.00th=[ 259], 60.00th=[ 259], | 70.00th=[ 1334], 80.00th=[ 1334], 90.00th=[ 1334], 95.00th=[ 1334], | 99.00th=[ 1334], 99.50th=[ 1334], 99.90th=[ 1334], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 7861, max= 7861, per=83.34%, avg=7861.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.06%, sys=3.57%, ctx=35, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12456: Fri Mar 28 21:25:59 2025 read: IOPS=0, BW=2707KiB/s (2772kB/s)(4096KiB/1513msec) clat (nsec): min=1016.0M, max=1016.0M, avg=1016017949.00, stdev= 0.00 lat (nsec): min=1016.0M, max=1016.0M, avg=1016020044.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1020], 5.00th=[ 1020], 10.00th=[ 1020], 20.00th=[ 1020], | 30.00th=[ 1020], 40.00th=[ 1020], 50.00th=[ 1020], 60.00th=[ 1020], | 70.00th=[ 1020], 80.00th=[ 1020], 90.00th=[ 1020], 95.00th=[ 1020], | 99.00th=[ 1020], 99.50th=[ 1020], 99.90th=[ 1020], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 8175, max= 8175, per=86.67%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5414KiB/s (5544kB/s)(8192KiB/1513msec) clat (msec): min=181, max=290, avg=236.36, stdev=77.14 lat (msec): min=181, max=291, avg=236.65, stdev=77.36 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 182], 10.00th=[ 182], 20.00th=[ 182], | 30.00th=[ 182], 40.00th=[ 182], 50.00th=[ 182], 60.00th=[ 292], | 70.00th=[ 292], 80.00th=[ 292], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 8175, max= 8175, per=39.09%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.07%, sys=6.75%, ctx=309, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12457: Fri Mar 28 21:25:59 2025 write: IOPS=2, BW=8672KiB/s (8880kB/s)(12.0MiB/1417msec) clat (msec): min=56, max=1189, avg=457.96, stdev=634.68 lat (msec): min=56, max=1190, avg=458.28, stdev=634.73 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 57], | 30.00th=[ 57], 40.00th=[ 128], 50.00th=[ 128], 60.00th=[ 128], | 70.00th=[ 1183], 80.00th=[ 1183], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.07%, sys=6.57%, ctx=45, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12458: Fri Mar 28 21:25:59 2025 write: IOPS=1, BW=7842KiB/s (8030kB/s)(12.0MiB/1567msec) clat (msec): min=255, max=1009, avg=514.51, stdev=428.48 lat (msec): min=256, max=1009, avg=514.94, stdev=428.44 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 255], 10.00th=[ 255], 20.00th=[ 255], | 30.00th=[ 255], 40.00th=[ 279], 50.00th=[ 279], 60.00th=[ 279], | 70.00th=[ 1011], 80.00th=[ 1011], 90.00th=[ 1011], 95.00th=[ 1011], | 99.00th=[ 1011], 99.50th=[ 1011], 99.90th=[ 1011], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min=16384, max=16384, per=78.35%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.13%, sys=8.11%, ctx=56, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9432KiB/s (9659kB/s), 2707KiB/s-7074KiB/s (2772kB/s-7244kB/s), io=16.0MiB (16.8MB), run=1513-1737msec WRITE: bw=20.4MiB/s (21.4MB/s), 5414KiB/s-8672KiB/s (5544kB/s-8880kB/s), io=32.0MiB (33.6MB), run=1417-1567msec