== sanity test 398b: DIO and buffer IO race ============== 19:20:10 (1746141610) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.01237 s, 25.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=862: Thu May 1 19:20:52 2025 read: IOPS=40, BW=161KiB/s (165kB/s)(5992KiB/37290msec) clat (usec): min=59, max=113070, avg=10711.46, stdev=10182.29 lat (usec): min=60, max=113071, avg=10713.17, stdev=10181.87 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 93], 10.00th=[ 104], 20.00th=[ 133], | 30.00th=[ 5145], 40.00th=[ 7308], 50.00th=[ 9110], 60.00th=[ 11469], | 70.00th=[ 13960], 80.00th=[ 17433], 90.00th=[ 23200], 95.00th=[ 28181], | 99.00th=[ 41157], 99.50th=[ 47449], 99.90th=[ 96994], 99.95th=[112722], | 99.99th=[112722] bw ( KiB/s): min= 23, max= 279, per=25.16%, avg=160.28, stdev=46.54, samples=74 iops : min= 5, max= 69, avg=39.74, stdev=11.61, samples=74 write: IOPS=42, BW=169KiB/s (173kB/s)(6296KiB/37290msec) clat (usec): min=543, max=397057, avg=13353.36, stdev=16628.64 lat (usec): min=544, max=397059, avg=13355.26, stdev=16629.84 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 725], 10.00th=[ 783], 20.00th=[ 963], | 30.00th=[ 2278], 40.00th=[ 5669], 50.00th=[ 9896], 60.00th=[ 13566], | 70.00th=[ 17171], 80.00th=[ 23200], 90.00th=[ 31589], 95.00th=[ 39584], | 99.00th=[ 57410], 99.50th=[ 64750], 99.90th=[103285], 99.95th=[396362], | 99.99th=[396362] bw ( KiB/s): min= 63, max= 287, per=25.60%, avg=170.73, stdev=47.41, samples=73 iops : min= 15, max= 71, avg=42.36, stdev=11.81, samples=73 lat (usec) : 100=3.94%, 250=7.65%, 500=0.52%, 750=3.58%, 1000=7.29% lat (msec) : 2=4.52%, 4=4.04%, 10=20.44%, 20=28.48%, 50=18.39% lat (msec) : 100=1.04%, 250=0.07%, 500=0.03% cpu : usr=0.12%, sys=12.18%, ctx=4898, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=864: Thu May 1 19:20:52 2025 read: IOPS=39, BW=159KiB/s (162kB/s)(5940KiB/37448msec) clat (usec): min=47, max=112202, avg=10135.31, stdev=9929.11 lat (usec): min=47, max=112203, avg=10136.98, stdev=9929.52 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 91], 10.00th=[ 101], 20.00th=[ 127], | 30.00th=[ 3785], 40.00th=[ 7111], 50.00th=[ 8717], 60.00th=[ 10683], | 70.00th=[ 13042], 80.00th=[ 16909], 90.00th=[ 22938], 95.00th=[ 27657], | 99.00th=[ 41681], 99.50th=[ 45876], 99.90th=[ 86508], 99.95th=[112722], | 99.99th=[112722] bw ( KiB/s): min= 7, max= 303, per=24.81%, avg=158.05, stdev=53.63, samples=74 iops : min= 1, max= 75, avg=39.28, stdev=13.44, samples=74 write: IOPS=42, BW=170KiB/s (174kB/s)(6348KiB/37448msec) clat (usec): min=439, max=379011, avg=13967.03, stdev=16423.65 lat (usec): min=439, max=379013, avg=13968.77, stdev=16423.52 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 734], 10.00th=[ 799], 20.00th=[ 1205], | 30.00th=[ 2802], 40.00th=[ 6652], 50.00th=[ 10552], 60.00th=[ 14222], | 70.00th=[ 18482], 80.00th=[ 23987], 90.00th=[ 32637], 95.00th=[ 39584], | 99.00th=[ 60031], 99.50th=[ 68682], 99.90th=[ 96994], 99.95th=[379585], | 99.99th=[379585] bw ( KiB/s): min= 87, max= 264, per=25.58%, avg=170.64, stdev=45.85, samples=73 iops : min= 21, max= 66, avg=42.44, stdev=11.52, samples=73 lat (usec) : 50=0.03%, 100=4.49%, 250=8.37%, 500=0.49%, 750=3.32% lat (usec) : 1000=5.96% lat (msec) : 2=4.59%, 4=5.14%, 10=20.64%, 20=26.01%, 50=19.86% lat (msec) : 100=1.04%, 250=0.03%, 500=0.03% cpu : usr=0.20%, sys=11.97%, ctx=4888, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=865: Thu May 1 19:20:52 2025 read: IOPS=39, BW=160KiB/s (164kB/s)(6028KiB/37683msec) clat (usec): min=52, max=85115, avg=10417.70, stdev=10026.01 lat (usec): min=53, max=85117, avg=10420.33, stdev=10025.40 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 91], 10.00th=[ 99], 20.00th=[ 121], | 30.00th=[ 1975], 40.00th=[ 6915], 50.00th=[ 8979], 60.00th=[11207], | 70.00th=[13566], 80.00th=[17695], 90.00th=[24249], 95.00th=[28967], | 99.00th=[40109], 99.50th=[44827], 99.90th=[65274], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 40, max= 272, per=25.30%, avg=161.16, stdev=46.42, samples=74 iops : min= 10, max= 68, avg=40.12, stdev=11.64, samples=74 write: IOPS=41, BW=166KiB/s (170kB/s)(6260KiB/37683msec) clat (usec): min=498, max=459327, avg=13875.01, stdev=18507.72 lat (usec): min=499, max=459330, avg=13877.32, stdev=18507.75 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 734], 10.00th=[ 775], 20.00th=[ 1029], | 30.00th=[ 2376], 40.00th=[ 5538], 50.00th=[ 9896], 60.00th=[ 13960], | 70.00th=[ 17957], 80.00th=[ 23462], 90.00th=[ 31851], 95.00th=[ 39060], | 99.00th=[ 65274], 99.50th=[ 80217], 99.90th=[141558], 99.95th=[459277], | 99.99th=[459277] bw ( KiB/s): min= 48, max= 360, per=25.01%, avg=166.78, stdev=52.94, samples=74 iops : min= 12, max= 90, avg=41.53, stdev=13.29, samples=74 lat (usec) : 100=5.27%, 250=8.30%, 500=0.46%, 750=4.10%, 1000=6.32% lat (msec) : 2=4.23%, 4=4.88%, 10=19.47%, 20=25.78%, 50=19.95% lat (msec) : 100=1.07%, 250=0.13%, 500=0.03% cpu : usr=0.17%, sys=11.69%, ctx=4805, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=866: Thu May 1 19:20:52 2025 read: IOPS=40, BW=161KiB/s (165kB/s)(6052KiB/37514msec) clat (usec): min=63, max=61194, avg=10280.41, stdev=9780.36 lat (usec): min=63, max=61194, avg=10281.95, stdev=9780.17 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 90], 10.00th=[ 99], 20.00th=[ 124], | 30.00th=[ 3326], 40.00th=[ 6783], 50.00th=[ 8717], 60.00th=[10814], | 70.00th=[13698], 80.00th=[17433], 90.00th=[23725], 95.00th=[28967], | 99.00th=[41157], 99.50th=[44827], 99.90th=[51119], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 23, max= 280, per=25.58%, avg=162.96, stdev=48.34, samples=74 iops : min= 5, max= 70, avg=40.47, stdev=12.15, samples=74 write: IOPS=41, BW=166KiB/s (170kB/s)(6236KiB/37514msec) clat (usec): min=482, max=473627, avg=13924.85, stdev=18454.94 lat (usec): min=483, max=473630, avg=13927.13, stdev=18456.41 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 1090], | 30.00th=[ 2311], 40.00th=[ 5538], 50.00th=[ 10552], 60.00th=[ 14615], | 70.00th=[ 17957], 80.00th=[ 23462], 90.00th=[ 32113], 95.00th=[ 39584], | 99.00th=[ 64750], 99.50th=[ 83362], 99.90th=[110625], 99.95th=[471860], | 99.99th=[471860] bw ( KiB/s): min= 31, max= 288, per=25.15%, avg=167.73, stdev=51.44, samples=74 iops : min= 7, max= 72, avg=41.66, stdev=12.92, samples=74 lat (usec) : 100=5.24%, 250=7.91%, 500=0.62%, 750=3.58%, 1000=6.64% lat (msec) : 2=4.49%, 4=4.65%, 10=19.14%, 20=27.18%, 50=19.17% lat (msec) : 100=1.27%, 250=0.07%, 500=0.03% cpu : usr=0.14%, sys=11.86%, ctx=4804, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=637KiB/s (653kB/s), 159KiB/s-161KiB/s (162kB/s-165kB/s), io=23.4MiB (24.6MB), run=37290-37683msec WRITE: bw=667KiB/s (683kB/s), 166KiB/s-170KiB/s (170kB/s-174kB/s), io=24.6MiB (25.7MB), run=37290-37683msec rand-rw: (groupid=0, jobs=1): err= 0: pid=867: Thu May 1 19:22:32 2025 read: IOPS=10, BW=43.7KiB/s (44.8kB/s)(5992KiB/137110msec) clat (msec): min=2, max=393, avg=13.29, stdev=14.75 lat (msec): min=2, max=393, avg=13.30, stdev=14.75 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 11], | 70.00th=[ 15], 80.00th=[ 20], 90.00th=[ 26], 95.00th=[ 33], | 99.00th=[ 54], 99.50th=[ 82], 99.90th=[ 122], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 7, max= 128, per=25.75%, avg=44.81, stdev=26.49, samples=265 iops : min= 1, max= 32, avg=10.93, stdev= 6.70, samples=265 write: IOPS=11, BW=45.9KiB/s (47.0kB/s)(6296KiB/137110msec) clat (msec): min=18, max=286, avg=74.32, stdev=29.30 lat (msec): min=18, max=286, avg=74.32, stdev=29.30 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 36], 10.00th=[ 43], 20.00th=[ 51], | 30.00th=[ 57], 40.00th=[ 64], 50.00th=[ 69], 60.00th=[ 78], | 70.00th=[ 85], 80.00th=[ 95], 90.00th=[ 110], 95.00th=[ 129], | 99.00th=[ 169], 99.50th=[ 182], 99.90th=[ 245], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 7, max= 120, per=25.11%, avg=45.70, stdev=15.78, samples=273 iops : min= 1, max= 30, avg=11.15, stdev= 3.99, samples=273 lat (msec) : 4=1.89%, 10=25.59%, 20=11.75%, 50=18.42%, 100=33.95% lat (msec) : 250=8.33%, 500=0.07% cpu : usr=0.04%, sys=2.77%, ctx=3840, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=868: Thu May 1 19:22:32 2025 read: IOPS=10, BW=43.2KiB/s (44.2kB/s)(5940KiB/137478msec) clat (msec): min=2, max=370, avg=13.53, stdev=15.16 lat (msec): min=2, max=370, avg=13.53, stdev=15.16 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 15], 80.00th=[ 21], 90.00th=[ 27], 95.00th=[ 32], | 99.00th=[ 58], 99.50th=[ 85], 99.90th=[ 163], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 7, max= 152, per=25.43%, avg=44.25, stdev=25.73, samples=266 iops : min= 1, max= 38, avg=10.85, stdev= 6.51, samples=266 write: IOPS=11, BW=46.2KiB/s (47.3kB/s)(6348KiB/137478msec) clat (msec): min=15, max=240, avg=73.81, stdev=28.48 lat (msec): min=15, max=240, avg=73.81, stdev=28.48 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 37], 10.00th=[ 43], 20.00th=[ 51], | 30.00th=[ 57], 40.00th=[ 63], 50.00th=[ 70], 60.00th=[ 77], | 70.00th=[ 85], 80.00th=[ 94], 90.00th=[ 110], 95.00th=[ 128], | 99.00th=[ 165], 99.50th=[ 178], 99.90th=[ 222], 99.95th=[ 241], | 99.99th=[ 241] bw ( KiB/s): min= 15, max= 128, per=25.23%, avg=45.93, stdev=15.56, samples=273 iops : min= 3, max= 32, avg=11.27, stdev= 3.99, samples=273 lat (msec) : 4=1.60%, 10=24.87%, 20=11.98%, 50=19.37%, 100=34.15% lat (msec) : 250=8.01%, 500=0.03% cpu : usr=0.07%, sys=2.72%, ctx=3856, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=869: Thu May 1 19:22:32 2025 read: IOPS=10, BW=43.0KiB/s (45.1kB/s)(6028KiB/137015msec) clat (msec): min=2, max=162, avg=13.79, stdev=11.99 lat (msec): min=2, max=162, avg=13.79, stdev=11.99 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 28], 95.00th=[ 34], | 99.00th=[ 55], 99.50th=[ 73], 99.90th=[ 140], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 7, max= 176, per=25.54%, avg=44.44, stdev=25.79, samples=268 iops : min= 1, max= 44, avg=10.84, stdev= 6.47, samples=268 write: IOPS=11, BW=45.7KiB/s (46.8kB/s)(6260KiB/137015msec) clat (msec): min=18, max=480, avg=74.14, stdev=30.53 lat (msec): min=18, max=480, avg=74.14, stdev=30.53 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 36], 10.00th=[ 42], 20.00th=[ 50], | 30.00th=[ 56], 40.00th=[ 63], 50.00th=[ 71], 60.00th=[ 79], | 70.00th=[ 86], 80.00th=[ 94], 90.00th=[ 111], 95.00th=[ 128], | 99.00th=[ 167], 99.50th=[ 180], 99.90th=[ 266], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 15, max= 119, per=24.96%, avg=45.43, stdev=16.00, samples=272 iops : min= 3, max= 29, avg=11.08, stdev= 4.09, samples=272 lat (msec) : 4=1.99%, 10=23.70%, 20=12.76%, 50=20.35%, 100=33.27% lat (msec) : 250=7.88%, 500=0.07% cpu : usr=0.06%, sys=2.78%, ctx=3904, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=870: Thu May 1 19:22:32 2025 read: IOPS=11, BW=44.4KiB/s (45.4kB/s)(6052KiB/136420msec) clat (msec): min=2, max=126, avg=12.95, stdev=10.30 lat (msec): min=2, max=126, avg=12.95, stdev=10.30 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 15], 80.00th=[ 21], 90.00th=[ 26], 95.00th=[ 32], | 99.00th=[ 51], 99.50th=[ 65], 99.90th=[ 85], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 7, max= 143, per=25.88%, avg=45.03, stdev=25.69, samples=266 iops : min= 1, max= 35, avg=10.99, stdev= 6.47, samples=266 write: IOPS=11, BW=45.7KiB/s (46.8kB/s)(6236KiB/136420msec) clat (msec): min=19, max=483, avg=74.80, stdev=30.69 lat (msec): min=19, max=483, avg=74.80, stdev=30.69 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 39], 10.00th=[ 44], 20.00th=[ 51], | 30.00th=[ 58], 40.00th=[ 64], 50.00th=[ 69], 60.00th=[ 77], | 70.00th=[ 85], 80.00th=[ 96], 90.00th=[ 115], 95.00th=[ 131], | 99.00th=[ 167], 99.50th=[ 180], 99.90th=[ 247], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 15, max= 119, per=24.99%, avg=45.48, stdev=15.12, samples=271 iops : min= 3, max= 29, avg=11.10, stdev= 3.85, samples=271 lat (msec) : 4=2.08%, 10=25.16%, 20=12.30%, 50=19.14%, 100=32.85% lat (msec) : 250=8.43%, 500=0.03% cpu : usr=0.02%, sys=2.84%, ctx=3880, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=175KiB/s (179kB/s), 43.2KiB/s-44.4KiB/s (44.2kB/s-45.4kB/s), io=23.4MiB (24.6MB), run=136420-137478msec WRITE: bw=183KiB/s (187kB/s), 45.7KiB/s-46.2KiB/s (46.8kB/s-47.3kB/s), io=24.6MiB (25.7MB), run=136420-137478msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1002: Thu May 1 19:22:43 2025 read: IOPS=42, BW=684KiB/s (701kB/s)(5920KiB/8652msec) clat (usec): min=64, max=105585, avg=10706.19, stdev=12431.38 lat (usec): min=64, max=105586, avg=10707.33, stdev=12431.32 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 92], 10.00th=[ 103], 20.00th=[ 114], | 30.00th=[ 131], 40.00th=[ 6718], 50.00th=[ 9241], 60.00th=[ 12125], | 70.00th=[ 13960], 80.00th=[ 17433], 90.00th=[ 25035], 95.00th=[ 29754], | 99.00th=[ 57934], 99.50th=[ 82314], 99.90th=[105382], 99.95th=[105382], | 99.99th=[105382] bw ( KiB/s): min= 159, max= 1152, per=26.26%, avg=674.06, stdev=245.81, samples=17 iops : min= 9, max= 72, avg=41.71, stdev=15.52, samples=17 write: IOPS=46, BW=736KiB/s (754kB/s)(6368KiB/8652msec) clat (usec): min=855, max=110837, avg=11275.35, stdev=15742.91 lat (usec): min=857, max=110840, avg=11278.57, stdev=15742.02 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 988], 10.00th=[ 1074], 20.00th=[ 1205], | 30.00th=[ 1729], 40.00th=[ 2900], 50.00th=[ 4621], 60.00th=[ 8979], | 70.00th=[ 13304], 80.00th=[ 19268], 90.00th=[ 26608], 95.00th=[ 40633], | 99.00th=[ 91751], 99.50th=[108528], 99.90th=[110625], 99.95th=[110625], | 99.99th=[110625] bw ( KiB/s): min= 63, max= 1088, per=27.20%, avg=734.47, stdev=257.50, samples=17 iops : min= 3, max= 68, avg=45.47, stdev=16.26, samples=17 lat (usec) : 100=3.65%, 250=13.28%, 500=0.39%, 750=0.26%, 1000=2.86% lat (msec) : 2=14.71%, 4=7.81%, 10=14.58%, 20=24.87%, 50=15.76% lat (msec) : 100=1.30%, 250=0.52% cpu : usr=0.15%, sys=12.35%, ctx=1076, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1003: Thu May 1 19:22:43 2025 read: IOPS=38, BW=616KiB/s (630kB/s)(5744KiB/9330msec) clat (usec): min=67, max=88143, avg=9988.59, stdev=11047.93 lat (usec): min=68, max=88144, avg=9989.67, stdev=11048.00 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 92], 10.00th=[ 99], 20.00th=[ 115], | 30.00th=[ 135], 40.00th=[ 2835], 50.00th=[ 8717], 60.00th=[11207], | 70.00th=[14615], 80.00th=[17695], 90.00th=[23462], 95.00th=[28181], | 99.00th=[50594], 99.50th=[56886], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 31, max= 1152, per=23.48%, avg=602.72, stdev=324.67, samples=18 iops : min= 1, max= 72, avg=37.39, stdev=20.40, samples=18 write: IOPS=43, BW=701KiB/s (718kB/s)(6544KiB/9330msec) clat (usec): min=832, max=143633, avg=13918.15, stdev=21007.20 lat (usec): min=833, max=143636, avg=13919.82, stdev=21007.31 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 996], 10.00th=[ 1057], 20.00th=[ 1270], | 30.00th=[ 1680], 40.00th=[ 2802], 50.00th=[ 5669], 60.00th=[ 9503], | 70.00th=[ 15401], 80.00th=[ 23462], 90.00th=[ 32637], 95.00th=[ 46924], | 99.00th=[108528], 99.50th=[135267], 99.90th=[143655], 99.95th=[143655], | 99.99th=[143655] bw ( KiB/s): min= 127, max= 1056, per=25.01%, avg=675.22, stdev=270.26, samples=18 iops : min= 7, max= 66, avg=41.94, stdev=16.97, samples=18 lat (usec) : 100=4.95%, 250=12.37%, 500=0.39%, 750=0.13%, 1000=3.26% lat (msec) : 2=14.84%, 4=6.77%, 10=14.97%, 20=22.53%, 50=16.93% lat (msec) : 100=1.95%, 250=0.91% cpu : usr=0.03%, sys=11.66%, ctx=1082, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1004: Thu May 1 19:22:43 2025 read: IOPS=45, BW=726KiB/s (743kB/s)(6256KiB/8619msec) clat (usec): min=73, max=171204, avg=11612.01, stdev=15110.90 lat (usec): min=73, max=171205, avg=11614.04, stdev=15111.00 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 97], 10.00th=[ 108], 20.00th=[ 131], | 30.00th=[ 408], 40.00th=[ 7504], 50.00th=[ 9634], 60.00th=[ 11731], | 70.00th=[ 14746], 80.00th=[ 19006], 90.00th=[ 23987], 95.00th=[ 30540], | 99.00th=[ 80217], 99.50th=[106431], 99.90th=[170918], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 352, max= 1088, per=28.59%, avg=733.88, stdev=204.62, samples=17 iops : min= 22, max= 68, avg=45.41, stdev=12.70, samples=17 write: IOPS=43, BW=700KiB/s (717kB/s)(6032KiB/8619msec) clat (usec): min=812, max=77118, avg=10500.70, stdev=11773.73 lat (usec): min=813, max=77119, avg=10503.75, stdev=11772.79 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 1045], 10.00th=[ 1123], 20.00th=[ 1418], | 30.00th=[ 2278], 40.00th=[ 3326], 50.00th=[ 5407], 60.00th=[ 9372], | 70.00th=[13304], 80.00th=[18744], 90.00th=[26346], 95.00th=[31851], | 99.00th=[55837], 99.50th=[65274], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 447, max= 1085, per=25.94%, avg=700.35, stdev=175.66, samples=17 iops : min= 27, max= 67, avg=43.29, stdev=11.09, samples=17 lat (usec) : 100=3.26%, 250=11.33%, 500=0.78%, 750=0.52%, 1000=2.34% lat (msec) : 2=12.50%, 4=8.33%, 10=17.45%, 20=25.39%, 50=16.54% lat (msec) : 100=1.30%, 250=0.26% cpu : usr=0.17%, sys=12.91%, ctx=1118, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1005: Thu May 1 19:22:43 2025 read: IOPS=43, BW=694KiB/s (710kB/s)(6032KiB/8695msec) clat (usec): min=74, max=111758, avg=10987.12, stdev=12003.95 lat (usec): min=75, max=111760, avg=10990.47, stdev=12004.30 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 97], 10.00th=[ 110], 20.00th=[ 128], | 30.00th=[ 265], 40.00th=[ 6783], 50.00th=[ 9503], 60.00th=[ 11994], | 70.00th=[ 14484], 80.00th=[ 18220], 90.00th=[ 24773], 95.00th=[ 30016], | 99.00th=[ 55313], 99.50th=[ 79168], 99.90th=[111674], 99.95th=[111674], | 99.99th=[111674] bw ( KiB/s): min= 510, max= 1024, per=28.63%, avg=734.81, stdev=155.11, samples=16 iops : min= 31, max= 64, avg=45.69, stdev= 9.86, samples=16 write: IOPS=44, BW=719KiB/s (737kB/s)(6256KiB/8695msec) clat (usec): min=863, max=131985, avg=11006.50, stdev=15493.72 lat (usec): min=864, max=131987, avg=11008.21, stdev=15493.80 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 1045], 10.00th=[ 1123], 20.00th=[ 1319], | 30.00th=[ 1926], 40.00th=[ 3032], 50.00th=[ 4752], 60.00th=[ 8094], | 70.00th=[ 12911], 80.00th=[ 18220], 90.00th=[ 28181], 95.00th=[ 34341], | 99.00th=[ 70779], 99.50th=[131597], 99.90th=[131597], 99.95th=[131597], | 99.99th=[131597] bw ( KiB/s): min= 95, max= 1344, per=26.59%, avg=717.94, stdev=268.63, samples=17 iops : min= 5, max= 84, avg=44.59, stdev=17.01, samples=17 lat (usec) : 100=2.99%, 250=11.59%, 500=0.78%, 750=0.26%, 1000=1.30% lat (msec) : 2=14.84%, 4=8.46%, 10=17.58%, 20=24.48%, 50=15.89% lat (msec) : 100=1.43%, 250=0.39% cpu : usr=0.30%, sys=12.61%, ctx=1087, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2567KiB/s (2629kB/s), 616KiB/s-726KiB/s (630kB/s-743kB/s), io=23.4MiB (24.5MB), run=8619-9330msec WRITE: bw=2701KiB/s (2766kB/s), 700KiB/s-736KiB/s (717kB/s-754kB/s), io=24.6MiB (25.8MB), run=8619-9330msec rand-rw: (groupid=0, jobs=1): err= 0: pid=998: Thu May 1 19:23:11 2025 read: IOPS=10, BW=162KiB/s (165kB/s)(5920KiB/36637msec) clat (msec): min=3, max=112, avg=12.25, stdev=11.31 lat (msec): min=3, max=112, avg=12.26, stdev=11.31 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 11], 80.00th=[ 16], 90.00th=[ 24], 95.00th=[ 32], | 99.00th=[ 64], 99.50th=[ 92], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 31, max= 512, per=25.96%, avg=165.38, stdev=89.74, samples=71 iops : min= 1, max= 32, avg=10.01, stdev= 5.74, samples=71 write: IOPS=10, BW=174KiB/s (178kB/s)(6368KiB/36637msec) clat (msec): min=30, max=205, avg=80.24, stdev=29.95 lat (msec): min=30, max=205, avg=80.24, stdev=29.95 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 45], 10.00th=[ 48], 20.00th=[ 56], | 30.00th=[ 61], 40.00th=[ 69], 50.00th=[ 74], 60.00th=[ 82], | 70.00th=[ 92], 80.00th=[ 102], 90.00th=[ 121], 95.00th=[ 142], | 99.00th=[ 186], 99.50th=[ 203], 99.90th=[ 205], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 31, max= 256, per=25.83%, avg=173.07, stdev=51.31, samples=73 iops : min= 1, max= 16, avg=10.49, stdev= 3.19, samples=73 lat (msec) : 4=0.13%, 10=30.21%, 20=11.46%, 50=12.11%, 100=35.03% lat (msec) : 250=11.07% cpu : usr=0.04%, sys=2.63%, ctx=899, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=999: Thu May 1 19:23:11 2025 read: IOPS=9, BW=153KiB/s (156kB/s)(5744KiB/37599msec) clat (msec): min=3, max=111, avg=14.06, stdev=14.58 lat (msec): min=3, max=111, avg=14.06, stdev=14.57 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 17], 90.00th=[ 30], 95.00th=[ 45], | 99.00th=[ 90], 99.50th=[ 104], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 31, max= 447, per=24.78%, avg=157.83, stdev=87.40, samples=72 iops : min= 1, max= 27, avg= 9.44, stdev= 5.54, samples=72 write: IOPS=10, BW=174KiB/s (178kB/s)(6544KiB/37599msec) clat (msec): min=35, max=207, avg=79.49, stdev=28.49 lat (msec): min=35, max=207, avg=79.49, stdev=28.49 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 45], 10.00th=[ 50], 20.00th=[ 57], | 30.00th=[ 62], 40.00th=[ 69], 50.00th=[ 74], 60.00th=[ 81], | 70.00th=[ 88], 80.00th=[ 99], 90.00th=[ 115], 95.00th=[ 142], | 99.00th=[ 178], 99.50th=[ 182], 99.90th=[ 209], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 63, max= 256, per=25.66%, avg=171.89, stdev=49.95, samples=75 iops : min= 3, max= 16, avg=10.32, stdev= 3.17, samples=75 lat (msec) : 4=0.26%, 10=26.56%, 20=13.15%, 50=10.55%, 100=39.58% lat (msec) : 250=9.90% cpu : usr=0.00%, sys=2.56%, ctx=884, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1000: Thu May 1 19:23:11 2025 read: IOPS=11, BW=176KiB/s (180kB/s)(6256KiB/35537msec) clat (usec): min=3639, max=97558, avg=11756.35, stdev=10634.70 lat (usec): min=3641, max=97559, avg=11758.07, stdev=10634.51 clat percentiles (usec): | 1.00th=[ 4047], 5.00th=[ 5145], 10.00th=[ 5604], 20.00th=[ 6259], | 30.00th=[ 7111], 40.00th=[ 7635], 50.00th=[ 8356], 60.00th=[ 9765], | 70.00th=[10945], 80.00th=[13042], 90.00th=[21627], 95.00th=[27919], | 99.00th=[62653], 99.50th=[81265], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 31, max= 480, per=27.99%, avg=178.27, stdev=112.24, samples=70 iops : min= 1, max= 30, avg=10.77, stdev= 7.17, samples=70 write: IOPS=10, BW=170KiB/s (174kB/s)(6032KiB/35537msec) clat (msec): min=35, max=221, avg=81.68, stdev=30.30 lat (msec): min=35, max=221, avg=81.68, stdev=30.30 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 44], 10.00th=[ 50], 20.00th=[ 57], | 30.00th=[ 64], 40.00th=[ 70], 50.00th=[ 75], 60.00th=[ 83], | 70.00th=[ 93], 80.00th=[ 104], 90.00th=[ 115], 95.00th=[ 144], | 99.00th=[ 188], 99.50th=[ 201], 99.90th=[ 222], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 64, max= 256, per=25.21%, avg=168.92, stdev=47.30, samples=71 iops : min= 4, max= 16, avg=10.18, stdev= 2.99, samples=71 lat (msec) : 4=0.39%, 10=30.99%, 20=13.93%, 50=9.90%, 100=33.20% lat (msec) : 250=11.59% cpu : usr=0.02%, sys=2.63%, ctx=899, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1001: Thu May 1 19:23:11 2025 read: IOPS=10, BW=162KiB/s (166kB/s)(6032KiB/37133msec) clat (msec): min=3, max=123, avg=12.77, stdev=13.25 lat (msec): min=3, max=123, avg=12.77, stdev=13.25 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 22], 95.00th=[ 39], | 99.00th=[ 70], 99.50th=[ 101], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 31, max= 416, per=25.59%, avg=163.03, stdev=94.58, samples=73 iops : min= 1, max= 26, avg= 9.86, stdev= 5.89, samples=73 write: IOPS=10, BW=168KiB/s (173kB/s)(6256KiB/37133msec) clat (msec): min=35, max=207, avg=82.19, stdev=30.05 lat (msec): min=35, max=207, avg=82.19, stdev=30.05 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 50], 20.00th=[ 58], | 30.00th=[ 65], 40.00th=[ 71], 50.00th=[ 77], 60.00th=[ 83], | 70.00th=[ 93], 80.00th=[ 101], 90.00th=[ 122], 95.00th=[ 142], | 99.00th=[ 201], 99.50th=[ 207], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 64, max= 254, per=25.04%, avg=167.74, stdev=47.11, samples=74 iops : min= 4, max= 15, avg=10.16, stdev= 2.98, samples=74 lat (msec) : 4=0.13%, 10=30.73%, 20=12.76%, 50=9.37%, 100=36.20% lat (msec) : 250=10.81% cpu : usr=0.02%, sys=2.62%, ctx=906, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=637KiB/s (652kB/s), 153KiB/s-176KiB/s (156kB/s-180kB/s), io=23.4MiB (24.5MB), run=35537-37599msec WRITE: bw=670KiB/s (686kB/s), 168KiB/s-174KiB/s (173kB/s-178kB/s), io=24.6MiB (25.8MB), run=35537-37599msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1050: Thu May 1 19:23:14 2025 read: IOPS=5, BW=5622KiB/s (5757kB/s)(7168KiB/1275msec) clat (usec): min=797, max=380337, avg=103843.61, stdev=133421.93 lat (usec): min=798, max=380339, avg=103845.45, stdev=133421.85 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 799], 10.00th=[ 799], 20.00th=[ 865], | 30.00th=[ 11076], 40.00th=[ 11076], 50.00th=[100140], 60.00th=[107480], | 70.00th=[107480], 80.00th=[126354], 90.00th=[379585], 95.00th=[379585], | 99.00th=[379585], 99.50th=[379585], 99.90th=[379585], 99.95th=[379585], | 99.99th=[379585] bw ( KiB/s): min= 4047, max= 6131, per=40.58%, avg=5089.00, stdev=1473.61, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 write: IOPS=3, BW=4016KiB/s (4112kB/s)(5120KiB/1275msec) clat (msec): min=39, max=164, avg=98.15, stdev=51.12 lat (msec): min=39, max=165, avg=98.22, stdev=51.12 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 40], | 30.00th=[ 67], 40.00th=[ 67], 50.00th=[ 85], 60.00th=[ 85], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 165], 95.00th=[ 165], | 99.00th=[ 165], 99.50th=[ 165], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 4087, max= 4087, per=23.28%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (usec) : 1000=16.67% lat (msec) : 20=8.33%, 50=8.33%, 100=16.67%, 250=41.67%, 500=8.33% cpu : usr=0.00%, sys=15.23%, ctx=219, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1051: Thu May 1 19:23:14 2025 read: IOPS=1, BW=1254KiB/s (1284kB/s)(2048KiB/1633msec) clat (msec): min=115, max=143, avg=129.69, stdev=19.48 lat (msec): min=115, max=143, avg=129.69, stdev=19.48 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 116], 10.00th=[ 116], 20.00th=[ 116], | 30.00th=[ 116], 40.00th=[ 116], 50.00th=[ 116], 60.00th=[ 144], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 1988, max= 2043, per=16.07%, avg=2015.50, stdev=38.89, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=6, BW=6271KiB/s (6421kB/s)(10.0MiB/1633msec) clat (msec): min=34, max=263, avg=133.14, stdev=77.81 lat (msec): min=34, max=263, avg=133.22, stdev=77.81 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 41], | 30.00th=[ 90], 40.00th=[ 104], 50.00th=[ 112], 60.00th=[ 116], | 70.00th=[ 176], 80.00th=[ 176], 90.00th=[ 249], 95.00th=[ 264], | 99.00th=[ 264], 99.50th=[ 264], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 1988, max= 8175, per=32.68%, avg=5737.33, stdev=3295.47, samples=3 iops : min= 1, max= 7, avg= 4.67, stdev= 3.21, samples=3 lat (msec) : 50=16.67%, 100=8.33%, 250=66.67%, 500=8.33% cpu : usr=0.00%, sys=15.20%, ctx=116, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1052: Thu May 1 19:23:14 2025 read: IOPS=5, BW=5471KiB/s (5602kB/s)(6144KiB/1123msec) clat (usec): min=492, max=128874, avg=38985.06, stdev=57616.01 lat (usec): min=493, max=128875, avg=38986.46, stdev=57616.13 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 494], 10.00th=[ 494], 20.00th=[ 529], | 30.00th=[ 529], 40.00th=[ 603], 50.00th=[ 603], 60.00th=[ 8356], | 70.00th=[ 94897], 80.00th=[ 94897], 90.00th=[128451], 95.00th=[128451], | 99.00th=[128451], 99.50th=[128451], 99.90th=[128451], 99.95th=[128451], | 99.99th=[128451] bw ( KiB/s): min= 4087, max= 4087, per=32.59%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5471KiB/s (5602kB/s)(6144KiB/1123msec) clat (msec): min=44, max=369, avg=133.82, stdev=124.10 lat (msec): min=44, max=369, avg=133.90, stdev=124.09 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 49], | 30.00th=[ 49], 40.00th=[ 63], 50.00th=[ 63], 60.00th=[ 115], | 70.00th=[ 165], 80.00th=[ 165], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 4087, max= 4096, per=23.30%, avg=4091.50, stdev= 6.36, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 lat (usec) : 500=8.33%, 750=16.67% lat (msec) : 10=8.33%, 50=16.67%, 100=16.67%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=15.86%, ctx=186, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1053: Thu May 1 19:23:14 2025 read: IOPS=4, BW=4710KiB/s (4823kB/s)(5120KiB/1087msec) clat (usec): min=445, max=229743, avg=60949.24, stdev=99510.79 lat (usec): min=446, max=229745, avg=60950.74, stdev=99511.01 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 445], 10.00th=[ 445], 20.00th=[ 445], | 30.00th=[ 465], 40.00th=[ 465], 50.00th=[ 603], 60.00th=[ 603], | 70.00th=[ 73925], 80.00th=[ 73925], 90.00th=[229639], 95.00th=[229639], | 99.00th=[229639], 99.50th=[229639], 99.90th=[229639], 99.95th=[229639], | 99.99th=[229639] bw ( KiB/s): min= 8192, max= 8192, per=65.32%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6594KiB/s (6753kB/s)(7168KiB/1087msec) clat (msec): min=29, max=162, avg=93.91, stdev=44.54 lat (msec): min=29, max=162, avg=93.98, stdev=44.56 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 57], | 30.00th=[ 80], 40.00th=[ 80], 50.00th=[ 91], 60.00th=[ 114], | 70.00th=[ 114], 80.00th=[ 126], 90.00th=[ 163], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 4096, max= 5988, per=28.72%, avg=5042.00, stdev=1337.85, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 50=8.33%, 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=18.23%, ctx=103, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.2MiB/s (12.8MB/s), 1254KiB/s-5622KiB/s (1284kB/s-5757kB/s), io=20.0MiB (20.0MB), run=1087-1633msec WRITE: bw=17.1MiB/s (17.0MB/s), 4016KiB/s-6594KiB/s (4112kB/s-6753kB/s), io=28.0MiB (29.4MB), run=1087-1633msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1054: Thu May 1 19:23:15 2025 read: IOPS=4, BW=4267KiB/s (4369kB/s)(7168KiB/1680msec) clat (msec): min=61, max=366, avg=137.04, stdev=114.60 lat (msec): min=61, max=366, avg=137.42, stdev=114.38 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 62], 10.00th=[ 62], 20.00th=[ 65], | 30.00th=[ 79], 40.00th=[ 79], 50.00th=[ 80], 60.00th=[ 94], | 70.00th=[ 94], 80.00th=[ 215], 90.00th=[ 368], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 2027, max= 4096, per=32.81%, avg=3403.33, stdev=1191.95, samples=3 iops : min= 1, max= 4, avg= 2.67, stdev= 1.53, samples=3 write: IOPS=2, BW=3048KiB/s (3121kB/s)(5120KiB/1680msec) clat (msec): min=80, max=244, avg=117.07, stdev=71.64 lat (msec): min=80, max=244, avg=117.14, stdev=71.64 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 81], 10.00th=[ 81], 20.00th=[ 81], | 30.00th=[ 82], 40.00th=[ 82], 50.00th=[ 82], 60.00th=[ 82], | 70.00th=[ 99], 80.00th=[ 99], 90.00th=[ 245], 95.00th=[ 245], | 99.00th=[ 245], 99.50th=[ 245], 99.90th=[ 245], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 2048, max= 6131, per=28.16%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (msec) : 100=75.00%, 250=16.67%, 500=8.33% cpu : usr=0.06%, sys=4.47%, ctx=43, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1055: Thu May 1 19:23:15 2025 read: IOPS=1, BW=1037KiB/s (1062kB/s)(2048KiB/1974msec) clat (msec): min=63, max=198, avg=130.59, stdev=95.52 lat (msec): min=63, max=198, avg=130.59, stdev=95.52 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 64], | 30.00th=[ 64], 40.00th=[ 64], 50.00th=[ 64], 60.00th=[ 199], | 70.00th=[ 199], 80.00th=[ 199], 90.00th=[ 199], 95.00th=[ 199], | 99.00th=[ 199], 99.50th=[ 199], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 2027, max= 2043, per=19.62%, avg=2035.00, stdev=11.31, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5187KiB/s (5312kB/s)(10.0MiB/1974msec) clat (msec): min=78, max=285, avg=150.38, stdev=58.15 lat (msec): min=78, max=285, avg=150.46, stdev=58.14 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 80], 10.00th=[ 80], 20.00th=[ 110], | 30.00th=[ 117], 40.00th=[ 120], 50.00th=[ 126], 60.00th=[ 148], | 70.00th=[ 176], 80.00th=[ 176], 90.00th=[ 197], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 6131, max= 6144, per=42.26%, avg=6137.50, stdev= 9.19, samples=2 iops : min= 5, max= 6, avg= 5.50, stdev= 0.71, samples=2 lat (msec) : 100=16.67%, 250=75.00%, 500=8.33% cpu : usr=0.00%, sys=3.24%, ctx=35, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1056: Thu May 1 19:23:15 2025 read: IOPS=3, BW=3188KiB/s (3265kB/s)(6144KiB/1927msec) clat (msec): min=21, max=148, avg=68.21, stdev=48.41 lat (msec): min=21, max=148, avg=68.22, stdev=48.41 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 38], 50.00th=[ 38], 60.00th=[ 73], | 70.00th=[ 97], 80.00th=[ 97], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 8175, max= 8175, per=78.80%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3188KiB/s (3265kB/s)(6144KiB/1927msec) clat (msec): min=105, max=457, avg=239.00, stdev=126.28 lat (msec): min=105, max=458, avg=239.28, stdev=126.68 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 106], 10.00th=[ 106], 20.00th=[ 146], | 30.00th=[ 146], 40.00th=[ 209], 50.00th=[ 209], 60.00th=[ 213], | 70.00th=[ 305], 80.00th=[ 305], 90.00th=[ 460], 95.00th=[ 460], | 99.00th=[ 460], 99.50th=[ 460], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 4087, max= 4096, per=28.17%, avg=4091.50, stdev= 6.36, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 lat (msec) : 50=25.00%, 100=16.67%, 250=41.67%, 500=16.67% cpu : usr=0.00%, sys=3.27%, ctx=46, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1057: Thu May 1 19:23:15 2025 read: IOPS=2, BW=2631KiB/s (2694kB/s)(5120KiB/1946msec) clat (msec): min=19, max=134, avg=79.17, stdev=55.47 lat (msec): min=19, max=134, avg=79.18, stdev=55.47 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 20], | 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 91], 60.00th=[ 91], | 70.00th=[ 129], 80.00th=[ 129], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 8192, max= 8192, per=78.97%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3683KiB/s (3772kB/s)(7168KiB/1946msec) clat (msec): min=70, max=371, avg=193.19, stdev=114.08 lat (msec): min=70, max=371, avg=193.27, stdev=114.08 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 71], 10.00th=[ 71], 20.00th=[ 80], | 30.00th=[ 102], 40.00th=[ 102], 50.00th=[ 211], 60.00th=[ 243], | 70.00th=[ 243], 80.00th=[ 275], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 2019, max= 4096, per=18.73%, avg=2721.00, stdev=1190.87, samples=3 iops : min= 1, max= 4, avg= 2.33, stdev= 1.53, samples=3 lat (msec) : 20=8.33%, 50=8.33%, 100=25.00%, 250=41.67%, 500=16.67% cpu : usr=0.00%, sys=3.44%, ctx=44, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.1MiB/s (10.6MB/s), 1037KiB/s-4267KiB/s (1062kB/s-4369kB/s), io=20.0MiB (20.0MB), run=1680-1974msec WRITE: bw=14.2MiB/s (14.9MB/s), 3048KiB/s-5187KiB/s (3121kB/s-5312kB/s), io=28.0MiB (29.4MB), run=1680-1974msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1064: Thu May 1 19:23:19 2025 read: IOPS=1, BW=6689KiB/s (6850kB/s)(12.0MiB/1837msec) clat (msec): min=457, max=712, avg=559.17, stdev=135.09 lat (msec): min=457, max=712, avg=559.17, stdev=135.09 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 456], 10.00th=[ 456], 20.00th=[ 456], | 30.00th=[ 456], 40.00th=[ 510], 50.00th=[ 510], 60.00th=[ 510], | 70.00th=[ 709], 80.00th=[ 709], 90.00th=[ 709], 95.00th=[ 709], | 99.00th=[ 709], 99.50th=[ 709], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 8143, max= 8175, per=100.00%, avg=8159.00, stdev=22.63, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.27%, sys=11.87%, ctx=679, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1065: Thu May 1 19:23:19 2025 read: IOPS=0, BW=1961KiB/s (2008kB/s)(4096KiB/2089msec) clat (nsec): min=921293k, max=921293k, avg=921292870.00, stdev= 0.00 lat (nsec): min=921296k, max=921296k, avg=921295543.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 919], 5.00th=[ 919], 10.00th=[ 919], 20.00th=[ 919], | 30.00th=[ 919], 40.00th=[ 919], 50.00th=[ 919], 60.00th=[ 919], | 70.00th=[ 919], 80.00th=[ 919], 90.00th=[ 919], 95.00th=[ 919], | 99.00th=[ 919], 99.50th=[ 919], 99.90th=[ 919], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3921KiB/s (4016kB/s)(8192KiB/2089msec) clat (msec): min=386, max=727, avg=557.31, stdev=241.03 lat (msec): min=387, max=728, avg=557.61, stdev=241.01 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 388], 10.00th=[ 388], 20.00th=[ 388], | 30.00th=[ 388], 40.00th=[ 388], 50.00th=[ 388], 60.00th=[ 726], | 70.00th=[ 726], 80.00th=[ 726], 90.00th=[ 726], 95.00th=[ 726], | 99.00th=[ 726], 99.50th=[ 726], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 8175, max= 8175, per=52.12%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.19%, sys=12.60%, ctx=330, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1066: Thu May 1 19:23:19 2025 write: IOPS=1, BW=6250KiB/s (6400kB/s)(12.0MiB/1966msec) clat (msec): min=445, max=847, avg=590.98, stdev=222.54 lat (msec): min=446, max=847, avg=591.30, stdev=222.49 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 447], 10.00th=[ 447], 20.00th=[ 447], | 30.00th=[ 447], 40.00th=[ 481], 50.00th=[ 481], 60.00th=[ 481], | 70.00th=[ 844], 80.00th=[ 844], 90.00th=[ 844], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 1000=33.33% cpu : usr=0.15%, sys=13.18%, ctx=114, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1067: Thu May 1 19:23:19 2025 write: IOPS=1, BW=6982KiB/s (7149kB/s)(12.0MiB/1760msec) clat (msec): min=196, max=1162, avg=582.55, stdev=511.56 lat (msec): min=197, max=1163, avg=583.03, stdev=511.43 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 197], 10.00th=[ 197], 20.00th=[ 197], | 30.00th=[ 197], 40.00th=[ 388], 50.00th=[ 388], 60.00th=[ 388], | 70.00th=[ 1167], 80.00th=[ 1167], 90.00th=[ 1167], 95.00th=[ 1167], | 99.00th=[ 1167], 99.50th=[ 1167], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 8192, max= 8192, per=52.23%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=14.61%, ctx=128, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7843KiB/s (8031kB/s), 1961KiB/s-6689KiB/s (2008kB/s-6850kB/s), io=16.0MiB (16.8MB), run=1837-2089msec WRITE: bw=15.3MiB/s (16.1MB/s), 3921KiB/s-6982KiB/s (4016kB/s-7149kB/s), io=32.0MiB (33.6MB), run=1760-2089msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1069: Thu May 1 19:23:19 2025 read: IOPS=1, BW=6156KiB/s (6304kB/s)(12.0MiB/1996msec) clat (msec): min=406, max=885, avg=596.32, stdev=254.28 lat (msec): min=406, max=885, avg=596.33, stdev=254.28 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 405], 10.00th=[ 405], 20.00th=[ 405], | 30.00th=[ 405], 40.00th=[ 498], 50.00th=[ 498], 60.00th=[ 498], | 70.00th=[ 885], 80.00th=[ 885], 90.00th=[ 885], 95.00th=[ 885], | 99.00th=[ 885], 99.50th=[ 885], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 1000=33.33% cpu : usr=0.05%, sys=1.50%, ctx=29, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1070: Thu May 1 19:23:19 2025 read: IOPS=0, BW=1897KiB/s (1943kB/s)(4096KiB/2159msec) clat (nsec): min=862824k, max=862824k, avg=862823810.00, stdev= 0.00 lat (nsec): min=862826k, max=862826k, avg=862826192.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 860], 5.00th=[ 860], 10.00th=[ 860], 20.00th=[ 860], | 30.00th=[ 860], 40.00th=[ 860], 50.00th=[ 860], 60.00th=[ 860], | 70.00th=[ 860], 80.00th=[ 860], 90.00th=[ 860], 95.00th=[ 860], | 99.00th=[ 860], 99.50th=[ 860], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3794KiB/s (3885kB/s)(8192KiB/2159msec) clat (msec): min=215, max=953, avg=584.38, stdev=522.15 lat (msec): min=217, max=953, avg=585.68, stdev=520.71 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 215], 10.00th=[ 215], 20.00th=[ 215], | 30.00th=[ 215], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 953], | 70.00th=[ 953], 80.00th=[ 953], 90.00th=[ 953], 95.00th=[ 953], | 99.00th=[ 953], 99.50th=[ 953], 99.90th=[ 953], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 8175, max= 8175, per=57.78%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 1000=66.67% cpu : usr=0.14%, sys=4.12%, ctx=56, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1071: Thu May 1 19:23:19 2025 write: IOPS=1, BW=5306KiB/s (5433kB/s)(12.0MiB/2316msec) clat (msec): min=95, max=1115, avg=702.97, stdev=537.49 lat (msec): min=96, max=1116, avg=703.45, stdev=537.17 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 95], 10.00th=[ 95], 20.00th=[ 95], | 30.00th=[ 95], 40.00th=[ 902], 50.00th=[ 902], 60.00th=[ 902], | 70.00th=[ 1116], 80.00th=[ 1116], 90.00th=[ 1116], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 8192, max= 8192, per=57.90%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 1000=33.33% cpu : usr=0.30%, sys=1.68%, ctx=85, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1072: Thu May 1 19:23:19 2025 write: IOPS=1, BW=5331KiB/s (5459kB/s)(12.0MiB/2305msec) clat (msec): min=200, max=1211, avg=712.79, stdev=505.45 lat (msec): min=200, max=1211, avg=713.09, stdev=505.61 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 201], 10.00th=[ 201], 20.00th=[ 201], | 30.00th=[ 201], 40.00th=[ 726], 50.00th=[ 726], 60.00th=[ 726], | 70.00th=[ 1217], 80.00th=[ 1217], 90.00th=[ 1217], 95.00th=[ 1217], | 99.00th=[ 1217], 99.50th=[ 1217], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 8175, max= 8175, per=57.78%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.30%, sys=2.34%, ctx=40, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7589KiB/s (7771kB/s), 1897KiB/s-6156KiB/s (1943kB/s-6304kB/s), io=16.0MiB (16.8MB), run=1996-2159msec WRITE: bw=13.8MiB/s (14.5MB/s), 3794KiB/s-5331KiB/s (3885kB/s-5459kB/s), io=32.0MiB (33.6MB), run=2159-2316msec