== sanity test 398b: DIO and buffer IO race ============== 21:39:51 (1746149991) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 4.12116 s, 12.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2280: Thu May 1 21:41:15 2025 read: IOPS=20, BW=81.6KiB/s (83.6kB/s)(5992KiB/73407msec) clat (usec): min=65, max=149770, avg=19648.05, stdev=19238.97 lat (usec): min=66, max=149771, avg=19649.83, stdev=19238.70 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 103], 10.00th=[ 118], 20.00th=[ 149], | 30.00th=[ 1975], 40.00th=[ 13304], 50.00th=[ 17433], 60.00th=[ 21627], | 70.00th=[ 26084], 80.00th=[ 32637], 90.00th=[ 43779], 95.00th=[ 55313], | 99.00th=[ 80217], 99.50th=[103285], 99.90th=[131597], 99.95th=[149947], | 99.99th=[149947] bw ( KiB/s): min= 23, max= 173, per=25.63%, avg=81.75, stdev=30.72, samples=145 iops : min= 5, max= 43, avg=19.96, stdev= 7.70, samples=145 write: IOPS=21, BW=85.8KiB/s (87.8kB/s)(6296KiB/73407msec) clat (usec): min=574, max=654487, avg=27691.64, stdev=33840.70 lat (usec): min=575, max=654488, avg=27706.63, stdev=33833.80 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 906], 10.00th=[ 1074], 20.00th=[ 1975], | 30.00th=[ 5080], 40.00th=[ 10683], 50.00th=[ 19530], 60.00th=[ 27132], | 70.00th=[ 35390], 80.00th=[ 46400], 90.00th=[ 66847], 95.00th=[ 85459], | 99.00th=[137364], 99.50th=[164627], 99.90th=[261096], 99.95th=[658506], | 99.99th=[658506] bw ( KiB/s): min= 8, max= 175, per=25.69%, avg=85.82, stdev=27.87, samples=145 iops : min= 2, max= 43, avg=20.99, stdev= 6.95, samples=145 lat (usec) : 100=2.05%, 250=10.03%, 500=1.56%, 750=0.55%, 1000=3.87% lat (msec) : 2=6.90%, 4=4.04%, 10=7.23%, 20=17.25%, 50=34.24% lat (msec) : 100=10.51%, 250=1.69%, 500=0.03%, 750=0.03% cpu : usr=0.16%, sys=8.25%, ctx=4925, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2281: Thu May 1 21:41:15 2025 read: IOPS=19, BW=78.0KiB/s (80.8kB/s)(5940KiB/75233msec) clat (usec): min=73, max=135493, avg=19652.85, stdev=19322.23 lat (usec): min=74, max=135494, avg=19658.54, stdev=19321.89 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 102], 10.00th=[ 117], 20.00th=[ 143], | 30.00th=[ 660], 40.00th=[ 13304], 50.00th=[ 16909], 60.00th=[ 21103], | 70.00th=[ 26346], 80.00th=[ 33162], 90.00th=[ 45876], 95.00th=[ 55837], | 99.00th=[ 79168], 99.50th=[ 88605], 99.90th=[123208], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 7, max= 160, per=24.67%, avg=78.70, stdev=31.40, samples=149 iops : min= 1, max= 40, avg=19.23, stdev= 7.88, samples=149 write: IOPS=21, BW=84.4KiB/s (86.4kB/s)(6348KiB/75233msec) clat (usec): min=698, max=675146, avg=28783.34, stdev=34961.83 lat (usec): min=698, max=675147, avg=28792.57, stdev=34965.65 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 906], 10.00th=[ 1057], 20.00th=[ 1713], | 30.00th=[ 5211], 40.00th=[ 11600], 50.00th=[ 19530], 60.00th=[ 28443], | 70.00th=[ 36963], 80.00th=[ 49021], 90.00th=[ 69731], 95.00th=[ 86508], | 99.00th=[143655], 99.50th=[168821], 99.90th=[252707], 99.95th=[675283], | 99.99th=[675283] bw ( KiB/s): min= 8, max= 167, per=24.95%, avg=83.33, stdev=30.14, samples=150 iops : min= 2, max= 41, avg=20.39, stdev= 7.54, samples=150 lat (usec) : 100=2.18%, 250=11.07%, 500=1.11%, 750=0.49%, 1000=4.04% lat (msec) : 2=7.16%, 4=3.16%, 10=7.29%, 20=17.58%, 50=32.03% lat (msec) : 100=12.04%, 250=1.79%, 500=0.03%, 750=0.03% cpu : usr=0.15%, sys=8.13%, ctx=4797, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2282: Thu May 1 21:41:15 2025 read: IOPS=20, BW=80.4KiB/s (82.4kB/s)(6028KiB/74941msec) clat (usec): min=75, max=677172, avg=21238.09, stdev=27042.17 lat (usec): min=75, max=677174, avg=21241.61, stdev=27041.00 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 102], 10.00th=[ 116], 20.00th=[ 147], | 30.00th=[ 2573], 40.00th=[ 13566], 50.00th=[ 17957], 60.00th=[ 21890], | 70.00th=[ 27395], 80.00th=[ 35914], 90.00th=[ 47449], 95.00th=[ 58459], | 99.00th=[ 83362], 99.50th=[116917], 99.90th=[244319], 99.95th=[675283], | 99.99th=[675283] bw ( KiB/s): min= 15, max= 167, per=25.01%, avg=79.78, stdev=30.20, samples=149 iops : min= 3, max= 41, avg=19.54, stdev= 7.52, samples=149 write: IOPS=20, BW=83.5KiB/s (85.5kB/s)(6260KiB/74941msec) clat (usec): min=625, max=205546, avg=27204.81, stdev=29135.83 lat (usec): min=626, max=205548, avg=27207.18, stdev=29135.85 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 938], 10.00th=[ 1074], 20.00th=[ 1614], | 30.00th=[ 5145], 40.00th=[ 10814], 50.00th=[ 20841], 60.00th=[ 27657], | 70.00th=[ 35390], 80.00th=[ 46400], 90.00th=[ 65799], 95.00th=[ 80217], | 99.00th=[126354], 99.50th=[149947], 99.90th=[198181], 99.95th=[206570], | 99.99th=[206570] bw ( KiB/s): min= 8, max= 200, per=24.76%, avg=82.70, stdev=29.35, samples=149 iops : min= 2, max= 50, avg=20.28, stdev= 7.32, samples=149 lat (usec) : 100=2.25%, 250=9.96%, 500=1.66%, 750=0.55%, 1000=3.39% lat (msec) : 2=7.78%, 4=3.35%, 10=7.39%, 20=15.79%, 50=34.83% lat (msec) : 100=11.33%, 250=1.69%, 750=0.03% cpu : usr=0.10%, sys=8.24%, ctx=4887, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2283: Thu May 1 21:41:15 2025 read: IOPS=20, BW=80.0KiB/s (82.9kB/s)(6052KiB/74743msec) clat (usec): min=71, max=226709, avg=20010.87, stdev=20726.51 lat (usec): min=71, max=226711, avg=20012.75, stdev=20726.21 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 102], 10.00th=[ 119], 20.00th=[ 147], | 30.00th=[ 2212], 40.00th=[ 12780], 50.00th=[ 16581], 60.00th=[ 20579], | 70.00th=[ 25822], 80.00th=[ 33817], 90.00th=[ 47449], 95.00th=[ 58459], | 99.00th=[ 79168], 99.50th=[ 86508], 99.90th=[198181], 99.95th=[227541], | 99.99th=[227541] bw ( KiB/s): min= 7, max= 176, per=25.50%, avg=81.35, stdev=32.45, samples=147 iops : min= 1, max= 44, avg=19.95, stdev= 8.14, samples=147 write: IOPS=20, BW=83.4KiB/s (85.4kB/s)(6236KiB/74743msec) clat (usec): min=548, max=664121, avg=28267.93, stdev=33850.03 lat (usec): min=548, max=664124, avg=28275.33, stdev=33863.38 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 889], 10.00th=[ 1045], 20.00th=[ 2024], | 30.00th=[ 5014], 40.00th=[ 11076], 50.00th=[ 19006], 60.00th=[ 27132], | 70.00th=[ 36963], 80.00th=[ 49546], 90.00th=[ 68682], 95.00th=[ 85459], | 99.00th=[129500], 99.50th=[149947], 99.90th=[206570], 99.95th=[666895], | 99.99th=[666895] bw ( KiB/s): min= 8, max= 152, per=24.90%, avg=83.18, stdev=27.47, samples=148 iops : min= 2, max= 38, avg=20.41, stdev= 6.90, samples=148 lat (usec) : 100=2.21%, 250=9.86%, 500=1.69%, 750=0.52%, 1000=3.97% lat (msec) : 2=6.48%, 4=4.10%, 10=7.71%, 20=18.29%, 50=30.86% lat (msec) : 100=12.70%, 250=1.56%, 750=0.03% cpu : usr=0.17%, sys=8.03%, ctx=4905, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=319KiB/s (327kB/s), 78.0KiB/s-81.6KiB/s (80.8kB/s-83.6kB/s), io=23.4MiB (24.6MB), run=73407-75233msec WRITE: bw=334KiB/s (342kB/s), 83.4KiB/s-85.8KiB/s (85.4kB/s-87.8kB/s), io=24.6MiB (25.7MB), run=73407-75233msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2276: Thu May 1 21:44:21 2025 read: IOPS=5, BW=23.1KiB/s (23.7kB/s)(5992KiB/259270msec) clat (msec): min=4, max=676, avg=22.05, stdev=25.01 lat (msec): min=4, max=676, avg=22.05, stdev=25.01 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 13], 40.00th=[ 14], 50.00th=[ 17], 60.00th=[ 20], | 70.00th=[ 22], 80.00th=[ 27], 90.00th=[ 41], 95.00th=[ 60], | 99.00th=[ 106], 99.50th=[ 116], 99.90th=[ 190], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 7, max= 103, per=28.63%, avg=26.34, stdev=17.90, samples=446 iops : min= 1, max= 25, avg= 6.21, stdev= 4.52, samples=446 write: IOPS=6, BW=24.3KiB/s (24.9kB/s)(6296KiB/259270msec) clat (msec): min=47, max=652, avg=143.41, stdev=68.72 lat (msec): min=47, max=652, avg=143.41, stdev=68.72 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 69], 10.00th=[ 80], 20.00th=[ 92], | 30.00th=[ 104], 40.00th=[ 115], 50.00th=[ 127], 60.00th=[ 140], | 70.00th=[ 157], 80.00th=[ 180], 90.00th=[ 239], 95.00th=[ 292], | 99.00th=[ 380], 99.50th=[ 401], 99.90th=[ 550], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 7, max= 56, per=24.92%, avg=23.93, stdev= 9.64, samples=514 iops : min= 1, max= 14, avg= 5.60, stdev= 2.50, samples=514 lat (msec) : 10=8.17%, 20=23.01%, 50=14.19%, 100=16.67%, 250=33.66% lat (msec) : 500=4.20%, 750=0.10% cpu : usr=0.03%, sys=1.87%, ctx=3850, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2277: Thu May 1 21:44:21 2025 read: IOPS=5, BW=22.8KiB/s (23.3kB/s)(5940KiB/260998msec) clat (msec): min=3, max=706, avg=22.94, stdev=26.96 lat (msec): min=3, max=706, avg=22.94, stdev=26.96 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 20], | 70.00th=[ 23], 80.00th=[ 28], 90.00th=[ 43], 95.00th=[ 63], | 99.00th=[ 115], 99.50th=[ 142], 99.90th=[ 228], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 7, max= 112, per=28.49%, avg=26.21, stdev=17.48, samples=444 iops : min= 1, max= 28, avg= 6.22, stdev= 4.41, samples=444 write: IOPS=6, BW=24.3KiB/s (24.9kB/s)(6348KiB/260998msec) clat (msec): min=41, max=512, avg=142.75, stdev=68.38 lat (msec): min=41, max=512, avg=142.75, stdev=68.38 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 69], 10.00th=[ 78], 20.00th=[ 90], | 30.00th=[ 101], 40.00th=[ 113], 50.00th=[ 125], 60.00th=[ 140], | 70.00th=[ 157], 80.00th=[ 184], 90.00th=[ 236], 95.00th=[ 288], | 99.00th=[ 372], 99.50th=[ 401], 99.90th=[ 468], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 7, max= 56, per=24.99%, avg=23.99, stdev= 9.51, samples=516 iops : min= 1, max= 14, avg= 5.64, stdev= 2.48, samples=516 lat (msec) : 4=0.03%, 10=7.81%, 20=22.59%, 50=14.32%, 100=18.13% lat (msec) : 250=32.71%, 500=4.33%, 750=0.07% cpu : usr=0.04%, sys=1.86%, ctx=3900, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2278: Thu May 1 21:44:21 2025 read: IOPS=5, BW=23.4KiB/s (23.9kB/s)(6028KiB/258025msec) clat (msec): min=4, max=218, avg=22.46, stdev=18.22 lat (msec): min=4, max=218, avg=22.46, stdev=18.22 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 18], 60.00th=[ 21], | 70.00th=[ 24], 80.00th=[ 29], 90.00th=[ 44], 95.00th=[ 62], | 99.00th=[ 90], 99.50th=[ 108], 99.90th=[ 155], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 7, max= 120, per=28.60%, avg=26.31, stdev=18.25, samples=448 iops : min= 1, max= 30, avg= 6.19, stdev= 4.62, samples=448 write: IOPS=6, BW=24.3KiB/s (24.8kB/s)(6260KiB/258025msec) clat (msec): min=50, max=653, avg=142.85, stdev=70.73 lat (msec): min=50, max=653, avg=142.86, stdev=70.73 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 69], 10.00th=[ 79], 20.00th=[ 91], | 30.00th=[ 102], 40.00th=[ 113], 50.00th=[ 125], 60.00th=[ 140], | 70.00th=[ 157], 80.00th=[ 180], 90.00th=[ 232], 95.00th=[ 292], | 99.00th=[ 409], 99.50th=[ 439], 99.90th=[ 567], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 7, max= 48, per=25.05%, avg=24.05, stdev= 9.52, samples=508 iops : min= 1, max= 12, avg= 5.62, stdev= 2.48, samples=508 lat (msec) : 10=7.94%, 20=21.52%, 50=15.79%, 100=17.77%, 250=32.81% lat (msec) : 500=4.07%, 750=0.10% cpu : usr=0.05%, sys=1.86%, ctx=3848, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2279: Thu May 1 21:44:21 2025 read: IOPS=5, BW=23.5KiB/s (24.1kB/s)(6052KiB/257390msec) clat (msec): min=4, max=235, avg=22.46, stdev=21.68 lat (msec): min=4, max=235, avg=22.47, stdev=21.68 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 19], | 70.00th=[ 23], 80.00th=[ 27], 90.00th=[ 40], 95.00th=[ 62], | 99.00th=[ 118], 99.50th=[ 148], 99.90th=[ 230], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 7, max= 120, per=29.00%, avg=26.68, stdev=18.15, samples=445 iops : min= 1, max= 30, avg= 6.29, stdev= 4.58, samples=445 write: IOPS=6, BW=24.2KiB/s (24.8kB/s)(6236KiB/257390msec) clat (msec): min=47, max=874, avg=142.97, stdev=70.34 lat (msec): min=47, max=874, avg=142.97, stdev=70.34 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 70], 10.00th=[ 78], 20.00th=[ 91], | 30.00th=[ 102], 40.00th=[ 114], 50.00th=[ 126], 60.00th=[ 140], | 70.00th=[ 159], 80.00th=[ 182], 90.00th=[ 232], 95.00th=[ 284], | 99.00th=[ 393], 99.50th=[ 439], 99.90th=[ 575], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 7, max= 48, per=24.81%, avg=23.82, stdev= 9.77, samples=512 iops : min= 1, max= 12, avg= 5.57, stdev= 2.56, samples=512 lat (msec) : 10=8.37%, 20=23.18%, 50=14.26%, 100=16.96%, 250=33.33% lat (msec) : 500=3.81%, 750=0.07%, 1000=0.03% cpu : usr=0.03%, sys=1.89%, ctx=3896, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=92.0KiB/s (94.2kB/s), 22.8KiB/s-23.5KiB/s (23.3kB/s-24.1kB/s), io=23.4MiB (24.6MB), run=257390-260998msec WRITE: bw=96.3KiB/s (98.6kB/s), 24.2KiB/s-24.3KiB/s (24.8kB/s-24.9kB/s), io=24.6MiB (25.7MB), run=257390-260998msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2503: Thu May 1 21:44:40 2025 read: IOPS=25, BW=410KiB/s (420kB/s)(5920KiB/14435msec) clat (usec): min=68, max=96319, avg=17741.00, stdev=17203.70 lat (usec): min=68, max=96321, avg=17742.43, stdev=17203.74 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 106], 10.00th=[ 122], 20.00th=[ 147], | 30.00th=[ 326], 40.00th=[11338], 50.00th=[16319], 60.00th=[20841], | 70.00th=[24511], 80.00th=[30278], 90.00th=[41157], 95.00th=[50070], | 99.00th=[71828], 99.50th=[87557], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 190, max= 800, per=28.60%, avg=410.07, stdev=161.09, samples=28 iops : min= 11, max= 50, avg=25.04, stdev=10.17, samples=28 write: IOPS=27, BW=441KiB/s (452kB/s)(6368KiB/14435msec) clat (usec): min=785, max=253125, avg=19665.96, stdev=26221.54 lat (usec): min=787, max=253126, avg=19668.35, stdev=26221.50 clat percentiles (usec): | 1.00th=[ 1123], 5.00th=[ 1254], 10.00th=[ 1418], 20.00th=[ 1811], | 30.00th=[ 3654], 40.00th=[ 5342], 50.00th=[ 9110], 60.00th=[ 16057], | 70.00th=[ 21890], 80.00th=[ 33162], 90.00th=[ 51119], 95.00th=[ 70779], | 99.00th=[121111], 99.50th=[152044], 99.90th=[252707], 99.95th=[252707], | 99.99th=[252707] bw ( KiB/s): min= 158, max= 892, per=29.49%, avg=444.96, stdev=173.42, samples=28 iops : min= 9, max= 55, avg=27.21, stdev=10.92, samples=28 lat (usec) : 100=1.30%, 250=11.85%, 500=2.60%, 750=0.39%, 1000=0.26% lat (msec) : 2=11.72%, 4=5.21%, 10=12.37%, 20=17.58%, 50=28.78% lat (msec) : 100=7.16%, 250=0.65%, 500=0.13% cpu : usr=0.08%, sys=10.11%, ctx=1136, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2504: Thu May 1 21:44:40 2025 read: IOPS=21, BW=344KiB/s (352kB/s)(5744KiB/16697msec) clat (usec): min=83, max=145523, avg=16067.46, stdev=18715.04 lat (usec): min=84, max=145524, avg=16068.84, stdev=18714.98 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 103], 10.00th=[ 117], 20.00th=[ 135], | 30.00th=[ 172], 40.00th=[ 889], 50.00th=[ 13042], 60.00th=[ 18482], | 70.00th=[ 23200], 80.00th=[ 30016], 90.00th=[ 41157], 95.00th=[ 51119], | 99.00th=[ 74974], 99.50th=[ 88605], 99.90th=[145753], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 31, max= 670, per=25.70%, avg=368.53, stdev=156.60, samples=30 iops : min= 1, max= 41, avg=22.37, stdev= 9.85, samples=30 write: IOPS=24, BW=392KiB/s (401kB/s)(6544KiB/16697msec) clat (usec): min=1030, max=416872, avg=26130.58, stdev=46516.24 lat (usec): min=1035, max=416875, avg=26135.29, stdev=46516.58 clat percentiles (usec): | 1.00th=[ 1156], 5.00th=[ 1319], 10.00th=[ 1467], 20.00th=[ 2057], | 30.00th=[ 4359], 40.00th=[ 6849], 50.00th=[ 10683], 60.00th=[ 18482], | 70.00th=[ 25822], 80.00th=[ 38011], 90.00th=[ 57410], 95.00th=[ 74974], | 99.00th=[278922], 99.50th=[299893], 99.90th=[417334], 99.95th=[417334], | 99.99th=[417334] bw ( KiB/s): min= 31, max= 885, per=25.22%, avg=380.58, stdev=219.19, samples=33 iops : min= 1, max= 55, avg=23.12, stdev=13.79, samples=33 lat (usec) : 100=1.69%, 250=13.54%, 500=2.86%, 750=0.13%, 1000=0.91% lat (msec) : 2=11.07%, 4=5.34%, 10=11.72%, 20=15.89%, 50=27.34% lat (msec) : 100=7.55%, 250=1.30%, 500=0.65% cpu : usr=0.09%, sys=8.83%, ctx=1098, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2505: Thu May 1 21:44:40 2025 read: IOPS=27, BW=434KiB/s (445kB/s)(6256KiB/14400msec) clat (usec): min=57, max=127478, avg=16168.55, stdev=19319.85 lat (usec): min=57, max=127479, avg=16175.25, stdev=19322.83 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 101], 10.00th=[ 112], 20.00th=[ 133], | 30.00th=[ 151], 40.00th=[ 330], 50.00th=[ 13173], 60.00th=[ 19268], | 70.00th=[ 24249], 80.00th=[ 29492], 90.00th=[ 41157], 95.00th=[ 47449], | 99.00th=[ 88605], 99.50th=[117965], 99.90th=[127402], 99.95th=[127402], | 99.99th=[127402] bw ( KiB/s): min= 222, max= 1024, per=31.96%, avg=458.33, stdev=205.49, samples=27 iops : min= 13, max= 64, avg=28.26, stdev=12.88, samples=27 write: IOPS=26, BW=419KiB/s (429kB/s)(6032KiB/14400msec) clat (usec): min=977, max=162877, avg=20266.18, stdev=24572.03 lat (usec): min=980, max=162879, avg=20278.37, stdev=24579.52 clat percentiles (usec): | 1.00th=[ 1029], 5.00th=[ 1205], 10.00th=[ 1369], 20.00th=[ 1909], | 30.00th=[ 3884], 40.00th=[ 6128], 50.00th=[ 10552], 60.00th=[ 15926], | 70.00th=[ 24773], 80.00th=[ 35914], 90.00th=[ 49546], 95.00th=[ 66847], | 99.00th=[116917], 99.50th=[135267], 99.90th=[162530], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 31, max= 800, per=27.70%, avg=417.93, stdev=180.34, samples=28 iops : min= 1, max= 50, avg=25.71, stdev=11.40, samples=28 lat (usec) : 100=2.47%, 250=17.19%, 500=1.56%, 750=0.26%, 1000=0.65% lat (msec) : 2=10.68%, 4=5.21%, 10=9.64%, 20=16.15%, 50=29.30% lat (msec) : 100=5.60%, 250=1.30% cpu : usr=0.13%, sys=9.44%, ctx=995, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2506: Thu May 1 21:44:40 2025 read: IOPS=26, BW=416KiB/s (426kB/s)(6032KiB/14491msec) clat (usec): min=81, max=148661, avg=19035.90, stdev=19406.67 lat (usec): min=82, max=148663, avg=19037.34, stdev=19406.71 clat percentiles (usec): | 1.00th=[ 97], 5.00th=[ 113], 10.00th=[ 127], 20.00th=[ 151], | 30.00th=[ 383], 40.00th=[ 13304], 50.00th=[ 17433], 60.00th=[ 21627], | 70.00th=[ 25560], 80.00th=[ 32900], 90.00th=[ 42206], 95.00th=[ 51119], | 99.00th=[ 83362], 99.50th=[139461], 99.90th=[147850], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 95, max= 763, per=28.63%, avg=410.61, stdev=145.29, samples=28 iops : min= 5, max= 47, avg=25.04, stdev= 9.11, samples=28 write: IOPS=26, BW=432KiB/s (442kB/s)(6256KiB/14491msec) clat (usec): min=1037, max=250518, avg=18451.28, stdev=25395.05 lat (usec): min=1038, max=250522, avg=18470.83, stdev=25411.59 clat percentiles (usec): | 1.00th=[ 1074], 5.00th=[ 1319], 10.00th=[ 1450], 20.00th=[ 1893], | 30.00th=[ 2966], 40.00th=[ 5473], 50.00th=[ 9372], 60.00th=[ 14484], | 70.00th=[ 21890], 80.00th=[ 30278], 90.00th=[ 43779], 95.00th=[ 65274], | 99.00th=[132645], 99.50th=[152044], 99.90th=[250610], 99.95th=[250610], | 99.99th=[250610] bw ( KiB/s): min= 191, max= 928, per=28.73%, avg=433.46, stdev=174.70, samples=28 iops : min= 11, max= 58, avg=26.46, stdev=11.05, samples=28 lat (usec) : 100=0.78%, 250=12.37%, 500=1.82%, 750=0.13%, 1000=0.26% lat (msec) : 2=11.98%, 4=6.51%, 10=9.64%, 20=19.01%, 50=30.47% lat (msec) : 100=5.99%, 250=0.91%, 500=0.13% cpu : usr=0.08%, sys=9.88%, ctx=1069, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1435KiB/s (1469kB/s), 344KiB/s-434KiB/s (352kB/s-445kB/s), io=23.4MiB (24.5MB), run=14400-16697msec WRITE: bw=1509KiB/s (1545kB/s), 392KiB/s-441KiB/s (401kB/s-452kB/s), io=24.6MiB (25.8MB), run=14400-16697msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2507: Thu May 1 21:45:28 2025 read: IOPS=5, BW=91.5KiB/s (93.7kB/s)(5920KiB/64691msec) clat (msec): min=5, max=254, avg=20.04, stdev=21.95 lat (msec): min=5, max=254, avg=20.05, stdev=21.95 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 13], 50.00th=[ 14], 60.00th=[ 16], | 70.00th=[ 18], 80.00th=[ 22], 90.00th=[ 33], 95.00th=[ 59], | 99.00th=[ 112], 99.50th=[ 167], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 31, max= 415, per=27.35%, avg=100.11, stdev=69.09, samples=117 iops : min= 1, max= 25, avg= 5.83, stdev= 4.33, samples=117 write: IOPS=6, BW=98.4KiB/s (101kB/s)(6368KiB/64691msec) clat (msec): min=56, max=436, avg=143.42, stdev=59.19 lat (msec): min=56, max=436, avg=143.43, stdev=59.19 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 75], 10.00th=[ 82], 20.00th=[ 100], | 30.00th=[ 109], 40.00th=[ 121], 50.00th=[ 132], 60.00th=[ 142], | 70.00th=[ 159], 80.00th=[ 182], 90.00th=[ 215], 95.00th=[ 257], | 99.00th=[ 372], 99.50th=[ 388], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 31, max= 189, per=25.38%, avg=97.72, stdev=35.35, samples=129 iops : min= 1, max= 11, avg= 5.68, stdev= 2.31, samples=129 lat (msec) : 10=7.42%, 20=29.43%, 50=8.07%, 100=13.54%, 250=38.28% lat (msec) : 500=3.26% cpu : usr=0.03%, sys=1.90%, ctx=940, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2508: Thu May 1 21:45:28 2025 read: IOPS=5, BW=87.9KiB/s (89.0kB/s)(5744KiB/65381msec) clat (msec): min=5, max=300, avg=20.95, stdev=25.09 lat (msec): min=5, max=300, avg=20.96, stdev=25.09 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 14], 50.00th=[ 15], 60.00th=[ 17], | 70.00th=[ 19], 80.00th=[ 22], 90.00th=[ 34], 95.00th=[ 59], | 99.00th=[ 131], 99.50th=[ 142], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 31, max= 416, per=28.31%, avg=103.61, stdev=67.73, samples=109 iops : min= 1, max= 26, avg= 6.11, stdev= 4.27, samples=109 write: IOPS=6, BW=100KiB/s (102kB/s)(6544KiB/65381msec) clat (msec): min=36, max=403, avg=141.11, stdev=56.69 lat (msec): min=36, max=403, avg=141.12, stdev=56.69 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 71], 10.00th=[ 80], 20.00th=[ 96], | 30.00th=[ 108], 40.00th=[ 118], 50.00th=[ 129], 60.00th=[ 144], | 70.00th=[ 159], 80.00th=[ 180], 90.00th=[ 211], 95.00th=[ 249], | 99.00th=[ 330], 99.50th=[ 355], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 31, max= 220, per=25.65%, avg=98.77, stdev=36.00, samples=130 iops : min= 1, max= 13, avg= 5.83, stdev= 2.34, samples=130 lat (msec) : 10=8.33%, 20=26.56%, 50=9.11%, 100=13.67%, 250=39.71% lat (msec) : 500=2.60% cpu : usr=0.03%, sys=1.90%, ctx=902, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2509: Thu May 1 21:45:28 2025 read: IOPS=6, BW=99.9KiB/s (102kB/s)(6256KiB/62640msec) clat (msec): min=4, max=297, avg=20.38, stdev=23.61 lat (msec): min=4, max=297, avg=20.39, stdev=23.61 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 9], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 13], 50.00th=[ 14], 60.00th=[ 16], | 70.00th=[ 19], 80.00th=[ 23], 90.00th=[ 34], 95.00th=[ 54], | 99.00th=[ 121], 99.50th=[ 186], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 31, max= 416, per=29.34%, avg=107.37, stdev=75.77, samples=116 iops : min= 1, max= 26, avg= 6.28, stdev= 4.80, samples=116 write: IOPS=6, BW=96.3KiB/s (98.6kB/s)(6032KiB/62640msec) clat (msec): min=53, max=487, avg=144.77, stdev=63.37 lat (msec): min=53, max=487, avg=144.78, stdev=63.37 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 73], 10.00th=[ 85], 20.00th=[ 99], | 30.00th=[ 109], 40.00th=[ 123], 50.00th=[ 131], 60.00th=[ 144], | 70.00th=[ 161], 80.00th=[ 176], 90.00th=[ 224], 95.00th=[ 275], | 99.00th=[ 384], 99.50th=[ 485], 99.90th=[ 489], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 31, max= 160, per=25.00%, avg=96.26, stdev=33.38, samples=124 iops : min= 1, max= 10, avg= 5.56, stdev= 2.24, samples=124 lat (msec) : 10=8.98%, 20=28.39%, 50=10.68%, 100=12.50%, 250=36.07% lat (msec) : 500=3.39% cpu : usr=0.04%, sys=2.09%, ctx=948, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2510: Thu May 1 21:45:28 2025 read: IOPS=5, BW=93.4KiB/s (95.7kB/s)(6032KiB/64551msec) clat (msec): min=5, max=250, avg=20.27, stdev=21.04 lat (msec): min=5, max=250, avg=20.27, stdev=21.04 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 13], 50.00th=[ 15], 60.00th=[ 16], | 70.00th=[ 19], 80.00th=[ 24], 90.00th=[ 34], 95.00th=[ 51], | 99.00th=[ 113], 99.50th=[ 128], 99.90th=[ 251], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 31, max= 384, per=29.10%, avg=106.51, stdev=67.29, samples=112 iops : min= 1, max= 24, avg= 6.10, stdev= 4.22, samples=112 write: IOPS=6, BW=96.9KiB/s (99.2kB/s)(6256KiB/64551msec) clat (msec): min=58, max=419, avg=144.54, stdev=57.25 lat (msec): min=58, max=419, avg=144.55, stdev=57.25 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 73], 10.00th=[ 84], 20.00th=[ 96], | 30.00th=[ 111], 40.00th=[ 121], 50.00th=[ 133], 60.00th=[ 150], | 70.00th=[ 161], 80.00th=[ 180], 90.00th=[ 224], 95.00th=[ 259], | 99.00th=[ 338], 99.50th=[ 359], 99.90th=[ 422], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 31, max= 192, per=25.23%, avg=97.14, stdev=34.17, samples=128 iops : min= 1, max= 12, avg= 5.53, stdev= 2.22, samples=128 lat (msec) : 10=7.16%, 20=29.04%, 50=10.42%, 100=13.41%, 250=36.59% lat (msec) : 500=3.39% cpu : usr=0.05%, sys=1.88%, ctx=943, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=366KiB/s (375kB/s), 87.9KiB/s-99.9KiB/s (89.0kB/s-102kB/s), io=23.4MiB (24.5MB), run=62640-65381msec WRITE: bw=385KiB/s (395kB/s), 96.3KiB/s-100KiB/s (98.6kB/s-102kB/s), io=24.6MiB (25.8MB), run=62640-65381msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2572: Thu May 1 21:45:33 2025 read: IOPS=3, BW=3391KiB/s (3472kB/s)(7168KiB/2114msec) clat (msec): min=79, max=159, avg=112.17, stdev=34.15 lat (msec): min=79, max=159, avg=112.17, stdev=34.15 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 80], 10.00th=[ 80], 20.00th=[ 81], | 30.00th=[ 92], 40.00th=[ 92], 50.00th=[ 92], 60.00th=[ 138], | 70.00th=[ 138], 80.00th=[ 146], 90.00th=[ 161], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 2043, max= 6131, per=60.30%, avg=4079.00, stdev=2044.05, samples=3 iops : min= 1, max= 5, avg= 3.00, stdev= 2.00, samples=3 write: IOPS=2, BW=2422KiB/s (2480kB/s)(5120KiB/2114msec) clat (msec): min=203, max=307, avg=249.24, stdev=51.92 lat (msec): min=203, max=307, avg=249.43, stdev=51.90 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 203], 10.00th=[ 203], 20.00th=[ 203], | 30.00th=[ 203], 40.00th=[ 203], 50.00th=[ 230], 60.00th=[ 230], | 70.00th=[ 305], 80.00th=[ 305], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 2031, max= 4096, per=35.94%, avg=3404.67, stdev=1189.64, samples=3 iops : min= 1, max= 4, avg= 2.67, stdev= 1.53, samples=3 lat (msec) : 100=33.33%, 250=50.00%, 500=16.67% cpu : usr=0.09%, sys=1.75%, ctx=32, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2573: Thu May 1 21:45:33 2025 read: IOPS=0, BW=688KiB/s (705kB/s)(2048KiB/2975msec) clat (msec): min=56, max=189, avg=122.98, stdev=94.56 lat (msec): min=56, max=189, avg=122.98, stdev=94.56 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 57], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 57], 60.00th=[ 190], | 70.00th=[ 190], 80.00th=[ 190], 90.00th=[ 190], 95.00th=[ 190], | 99.00th=[ 190], 99.50th=[ 190], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 2039, max= 2048, per=30.21%, avg=2043.50, stdev= 6.36, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3442KiB/s (3525kB/s)(10.0MiB/2975msec) clat (msec): min=167, max=375, avg=245.94, stdev=63.80 lat (msec): min=167, max=376, avg=246.23, stdev=63.83 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 188], | 30.00th=[ 203], 40.00th=[ 211], 50.00th=[ 222], 60.00th=[ 230], | 70.00th=[ 284], 80.00th=[ 284], 90.00th=[ 313], 95.00th=[ 376], | 99.00th=[ 376], 99.50th=[ 376], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 2043, max= 4096, per=37.76%, avg=3576.25, stdev=1022.19, samples=4 iops : min= 1, max= 4, avg= 2.75, stdev= 1.26, samples=4 lat (msec) : 100=8.33%, 250=58.33%, 500=33.33% cpu : usr=0.13%, sys=4.24%, ctx=59, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2574: Thu May 1 21:45:33 2025 read: IOPS=1, BW=2030KiB/s (2078kB/s)(6144KiB/3027msec) clat (msec): min=29, max=733, avg=182.91, stdev=271.53 lat (msec): min=29, max=733, avg=182.91, stdev=271.53 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 73], 50.00th=[ 73], 60.00th=[ 99], | 70.00th=[ 116], 80.00th=[ 116], 90.00th=[ 735], 95.00th=[ 735], | 99.00th=[ 735], 99.50th=[ 735], 99.90th=[ 735], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 2039, max= 4096, per=37.79%, avg=2556.50, stdev=1026.34, samples=4 iops : min= 1, max= 4, avg= 2.00, stdev= 1.41, samples=4 write: IOPS=1, BW=2030KiB/s (2078kB/s)(6144KiB/3027msec) clat (msec): min=161, max=423, avg=288.23, stdev=117.33 lat (msec): min=161, max=423, avg=288.33, stdev=117.34 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 376], | 70.00th=[ 380], 80.00th=[ 380], 90.00th=[ 422], 95.00th=[ 422], | 99.00th=[ 422], 99.50th=[ 422], 99.90th=[ 422], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 2043, max= 4096, per=32.37%, avg=3066.50, stdev=1178.97, samples=4 iops : min= 1, max= 4, avg= 2.50, stdev= 1.29, samples=4 lat (msec) : 50=16.67%, 100=16.67%, 250=33.33%, 500=25.00%, 750=8.33% cpu : usr=0.10%, sys=3.54%, ctx=61, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2575: Thu May 1 21:45:33 2025 read: IOPS=1, BW=1736KiB/s (1778kB/s)(5120KiB/2949msec) clat (msec): min=79, max=235, avg=131.82, stdev=61.98 lat (msec): min=79, max=235, avg=131.83, stdev=61.98 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 80], 10.00th=[ 80], 20.00th=[ 80], | 30.00th=[ 89], 40.00th=[ 89], 50.00th=[ 126], 60.00th=[ 126], | 70.00th=[ 131], 80.00th=[ 131], 90.00th=[ 236], 95.00th=[ 236], | 99.00th=[ 236], 99.50th=[ 236], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 2048, max= 6131, per=60.45%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=2, BW=2431KiB/s (2489kB/s)(7168KiB/2949msec) clat (msec): min=194, max=518, avg=324.24, stdev=138.75 lat (msec): min=194, max=518, avg=324.38, stdev=138.72 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 194], 10.00th=[ 194], 20.00th=[ 194], | 30.00th=[ 218], 40.00th=[ 218], 50.00th=[ 259], 60.00th=[ 435], | 70.00th=[ 435], 80.00th=[ 451], 90.00th=[ 518], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 2039, max= 2048, per=21.58%, avg=2044.20, stdev= 3.83, samples=5 iops : min= 1, max= 2, avg= 1.40, stdev= 0.55, samples=5 lat (msec) : 100=16.67%, 250=50.00%, 500=25.00%, 750=8.33% cpu : usr=0.00%, sys=3.09%, ctx=36, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6766KiB/s (6928kB/s), 688KiB/s-3391KiB/s (705kB/s-3472kB/s), io=20.0MiB (20.0MB), run=2114-3027msec WRITE: bw=9472KiB/s (9699kB/s), 2030KiB/s-3442KiB/s (2078kB/s-3525kB/s), io=28.0MiB (29.4MB), run=2114-3027msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2576: Thu May 1 21:45:33 2025 read: IOPS=3, BW=3232KiB/s (3309kB/s)(7168KiB/2218msec) clat (usec): min=1630, max=127648, avg=80199.80, stdev=53662.30 lat (usec): min=1631, max=127649, avg=80201.58, stdev=53662.67 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1631], 10.00th=[ 1631], 20.00th=[ 3326], | 30.00th=[104334], 40.00th=[104334], 50.00th=[105382], 60.00th=[108528], | 70.00th=[108528], 80.00th=[110625], 90.00th=[127402], 95.00th=[127402], | 99.00th=[127402], 99.50th=[127402], 99.90th=[127402], 99.95th=[127402], | 99.99th=[127402] bw ( KiB/s): min= 2035, max= 6144, per=44.98%, avg=3067.50, stdev=2051.01, samples=4 iops : min= 1, max= 6, avg= 2.50, stdev= 2.38, samples=4 write: IOPS=2, BW=2308KiB/s (2364kB/s)(5120KiB/2218msec) clat (msec): min=152, max=472, avg=317.85, stdev=127.48 lat (msec): min=153, max=472, avg=317.97, stdev=127.42 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 153], 10.00th=[ 153], 20.00th=[ 153], | 30.00th=[ 245], 40.00th=[ 245], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 409], 80.00th=[ 409], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 2035, max= 4096, per=28.54%, avg=2724.67, stdev=1187.62, samples=3 iops : min= 1, max= 4, avg= 2.00, stdev= 1.73, samples=3 lat (msec) : 2=8.33%, 4=8.33%, 250=58.33%, 500=25.00% cpu : usr=0.00%, sys=12.95%, ctx=341, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2577: Thu May 1 21:45:33 2025 read: IOPS=0, BW=685KiB/s (701kB/s)(2048KiB/2991msec) clat (msec): min=178, max=244, avg=211.66, stdev=46.47 lat (msec): min=178, max=244, avg=211.67, stdev=46.47 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 180], 10.00th=[ 180], 20.00th=[ 180], | 30.00th=[ 180], 40.00th=[ 180], 50.00th=[ 180], 60.00th=[ 245], | 70.00th=[ 245], 80.00th=[ 245], 90.00th=[ 245], 95.00th=[ 245], | 99.00th=[ 245], 99.50th=[ 245], 99.90th=[ 245], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 2015, max= 2048, per=29.79%, avg=2031.50, stdev=23.33, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3424KiB/s (3506kB/s)(10.0MiB/2991msec) clat (msec): min=132, max=372, avg=234.57, stdev=88.04 lat (msec): min=132, max=373, avg=234.73, stdev=88.07 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 150], | 30.00th=[ 155], 40.00th=[ 182], 50.00th=[ 201], 60.00th=[ 241], | 70.00th=[ 305], 80.00th=[ 305], 90.00th=[ 368], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 2027, max= 6047, per=37.21%, avg=3552.25, stdev=1923.44, samples=4 iops : min= 1, max= 5, avg= 2.75, stdev= 1.71, samples=4 lat (msec) : 250=75.00%, 500=25.00% cpu : usr=0.00%, sys=11.54%, ctx=245, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2578: Thu May 1 21:45:33 2025 read: IOPS=3, BW=3227KiB/s (3304kB/s)(6144KiB/1904msec) clat (msec): min=18, max=209, avg=125.94, stdev=65.45 lat (msec): min=18, max=209, avg=125.94, stdev=65.45 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 105], | 30.00th=[ 105], 40.00th=[ 107], 50.00th=[ 107], 60.00th=[ 153], | 70.00th=[ 165], 80.00th=[ 165], 90.00th=[ 209], 95.00th=[ 209], | 99.00th=[ 209], 99.50th=[ 209], 99.90th=[ 209], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 4079, max= 4096, per=59.94%, avg=4087.50, stdev=12.02, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3227KiB/s (3304kB/s)(6144KiB/1904msec) clat (msec): min=57, max=516, avg=173.25, stdev=173.61 lat (msec): min=57, max=516, avg=173.37, stdev=173.60 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 58], 10.00th=[ 58], 20.00th=[ 71], | 30.00th=[ 71], 40.00th=[ 83], 50.00th=[ 83], 60.00th=[ 155], | 70.00th=[ 159], 80.00th=[ 159], 90.00th=[ 518], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 2039, max= 4096, per=35.72%, avg=3410.33, stdev=1187.61, samples=3 iops : min= 1, max= 4, avg= 3.00, stdev= 1.73, samples=3 lat (msec) : 20=8.33%, 100=25.00%, 250=58.33%, 750=8.33% cpu : usr=0.00%, sys=14.29%, ctx=214, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2579: Thu May 1 21:45:33 2025 read: IOPS=1, BW=1705KiB/s (1746kB/s)(5120KiB/3003msec) clat (usec): min=918, max=270813, avg=129605.80, stdev=96312.27 lat (usec): min=921, max=270816, avg=129607.93, stdev=96312.26 clat percentiles (usec): | 1.00th=[ 922], 5.00th=[ 922], 10.00th=[ 922], 20.00th=[ 922], | 30.00th=[106431], 40.00th=[106431], 50.00th=[132645], 60.00th=[132645], | 70.00th=[137364], 80.00th=[137364], 90.00th=[270533], 95.00th=[270533], | 99.00th=[270533], 99.50th=[270533], 99.90th=[270533], 99.95th=[270533], | 99.99th=[270533] bw ( KiB/s): min= 2048, max= 6107, per=59.80%, avg=4077.50, stdev=2870.15, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=2, BW=2387KiB/s (2444kB/s)(7168KiB/3003msec) clat (msec): min=82, max=563, avg=290.91, stdev=170.96 lat (msec): min=82, max=563, avg=291.03, stdev=170.98 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 83], 10.00th=[ 83], 20.00th=[ 161], | 30.00th=[ 194], 40.00th=[ 194], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 292], 80.00th=[ 468], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 2027, max= 2048, per=21.38%, avg=2041.20, stdev= 9.73, samples=5 iops : min= 1, max= 2, avg= 1.60, stdev= 0.55, samples=5 lat (usec) : 1000=8.33% lat (msec) : 100=8.33%, 250=41.67%, 500=33.33%, 750=8.33% cpu : usr=0.00%, sys=9.83%, ctx=173, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6820KiB/s (6984kB/s), 685KiB/s-3232KiB/s (701kB/s-3309kB/s), io=20.0MiB (20.0MB), run=1904-3003msec WRITE: bw=9548KiB/s (9777kB/s), 2308KiB/s-3424KiB/s (2364kB/s-3506kB/s), io=28.0MiB (29.4MB), run=1904-3003msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2589: Thu May 1 21:45:39 2025 read: IOPS=1, BW=7797KiB/s (7984kB/s)(12.0MiB/1576msec) clat (msec): min=409, max=521, avg=480.91, stdev=61.85 lat (msec): min=409, max=521, avg=480.91, stdev=61.85 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 409], 10.00th=[ 409], 20.00th=[ 409], | 30.00th=[ 409], 40.00th=[ 510], 50.00th=[ 510], 60.00th=[ 510], | 70.00th=[ 523], 80.00th=[ 523], 90.00th=[ 523], 95.00th=[ 523], | 99.00th=[ 523], 99.50th=[ 523], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 8159, max= 8192, per=100.00%, avg=8175.50, stdev=23.33, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.19%, sys=17.08%, ctx=283, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2592: Thu May 1 21:45:39 2025 read: IOPS=0, BW=1918KiB/s (1964kB/s)(4096KiB/2136msec) clat (nsec): min=402990k, max=402990k, avg=402990099.00, stdev= 0.00 lat (nsec): min=402993k, max=402993k, avg=402993252.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 405], 10.00th=[ 405], 20.00th=[ 405], | 30.00th=[ 405], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 405], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3835KiB/s (3927kB/s)(8192KiB/2136msec) clat (msec): min=450, max=1035, avg=742.81, stdev=413.96 lat (msec): min=450, max=1036, avg=743.32, stdev=414.12 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 451], 10.00th=[ 451], 20.00th=[ 451], | 30.00th=[ 451], 40.00th=[ 451], 50.00th=[ 451], 60.00th=[ 1036], | 70.00th=[ 1036], 80.00th=[ 1036], 90.00th=[ 1036], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 8159, max= 8159, per=65.66%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=12.93%, ctx=121, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2594: Thu May 1 21:45:39 2025 write: IOPS=1, BW=4776KiB/s (4890kB/s)(12.0MiB/2573msec) clat (msec): min=431, max=1190, avg=766.03, stdev=387.49 lat (msec): min=432, max=1191, avg=766.83, stdev=387.27 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 430], 10.00th=[ 430], 20.00th=[ 430], | 30.00th=[ 430], 40.00th=[ 676], 50.00th=[ 676], 60.00th=[ 676], | 70.00th=[ 1183], 80.00th=[ 1183], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 8192, max= 8192, per=65.93%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=11.55%, ctx=87, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2595: Thu May 1 21:45:39 2025 write: IOPS=1, BW=4660KiB/s (4772kB/s)(12.0MiB/2637msec) clat (msec): min=754, max=903, avg=833.00, stdev=75.03 lat (msec): min=754, max=903, avg=833.49, stdev=74.86 clat percentiles (msec): | 1.00th=[ 751], 5.00th=[ 751], 10.00th=[ 751], 20.00th=[ 751], | 30.00th=[ 751], 40.00th=[ 844], 50.00th=[ 844], 60.00th=[ 844], | 70.00th=[ 902], 80.00th=[ 902], 90.00th=[ 902], 95.00th=[ 902], | 99.00th=[ 902], 99.50th=[ 902], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 8126, max= 8175, per=65.59%, avg=8150.50, stdev=34.65, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 1000=100.00% cpu : usr=0.11%, sys=15.82%, ctx=105, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7670KiB/s (7855kB/s), 1918KiB/s-7797KiB/s (1964kB/s-7984kB/s), io=16.0MiB (16.8MB), run=1576-2136msec WRITE: bw=12.1MiB/s (12.7MB/s), 3835KiB/s-4776KiB/s (3927kB/s-4890kB/s), io=32.0MiB (33.6MB), run=2136-2637msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2588: Thu May 1 21:45:39 2025 read: IOPS=1, BW=5354KiB/s (5483kB/s)(12.0MiB/2295msec) clat (msec): min=498, max=1062, avg=693.96, stdev=319.30 lat (msec): min=498, max=1062, avg=693.96, stdev=319.30 clat percentiles (msec): | 1.00th=[ 498], 5.00th=[ 498], 10.00th=[ 498], 20.00th=[ 498], | 30.00th=[ 498], 40.00th=[ 523], 50.00th=[ 523], 60.00th=[ 523], | 70.00th=[ 1062], 80.00th=[ 1062], 90.00th=[ 1062], 95.00th=[ 1062], | 99.00th=[ 1062], 99.50th=[ 1062], 99.90th=[ 1062], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.13%, sys=1.26%, ctx=29, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2590: Thu May 1 21:45:39 2025 read: IOPS=0, BW=1451KiB/s (1486kB/s)(4096KiB/2822msec) clat (nsec): min=130381k, max=130381k, avg=130381459.00, stdev= 0.00 lat (nsec): min=130383k, max=130383k, avg=130382867.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 131], 10.00th=[ 131], 20.00th=[ 131], | 30.00th=[ 131], 40.00th=[ 131], 50.00th=[ 131], 60.00th=[ 131], | 70.00th=[ 131], 80.00th=[ 131], 90.00th=[ 131], 95.00th=[ 131], | 99.00th=[ 131], 99.50th=[ 131], 99.90th=[ 131], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2903KiB/s (2973kB/s)(8192KiB/2822msec) clat (msec): min=696, max=1821, avg=1259.41, stdev=795.57 lat (msec): min=697, max=1822, avg=1259.95, stdev=795.61 clat percentiles (msec): | 1.00th=[ 701], 5.00th=[ 701], 10.00th=[ 701], 20.00th=[ 701], | 30.00th=[ 701], 40.00th=[ 701], 50.00th=[ 701], 60.00th=[ 1821], | 70.00th=[ 1821], 80.00th=[ 1821], 90.00th=[ 1821], 95.00th=[ 1821], | 99.00th=[ 1821], 99.50th=[ 1821], 99.90th=[ 1821], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 5910, max= 5910, per=53.12%, avg=5910.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=3.40%, ctx=33, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2591: Thu May 1 21:45:39 2025 write: IOPS=1, BW=4172KiB/s (4273kB/s)(12.0MiB/2945msec) clat (msec): min=468, max=1748, avg=902.27, stdev=733.11 lat (msec): min=469, max=1749, avg=902.94, stdev=733.08 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 468], 10.00th=[ 468], 20.00th=[ 468], | 30.00th=[ 468], 40.00th=[ 489], 50.00th=[ 489], 60.00th=[ 489], | 70.00th=[ 1754], 80.00th=[ 1754], 90.00th=[ 1754], 95.00th=[ 1754], | 99.00th=[ 1754], 99.50th=[ 1754], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 6068, max= 8175, per=64.01%, avg=7121.50, stdev=1489.87, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67% cpu : usr=0.14%, sys=5.71%, ctx=46, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2593: Thu May 1 21:45:39 2025 write: IOPS=1, BW=5103KiB/s (5225kB/s)(12.0MiB/2408msec) clat (msec): min=252, max=1586, avg=741.28, stdev=734.87 lat (msec): min=253, max=1587, avg=741.80, stdev=734.91 clat percentiles (msec): | 1.00th=[ 253], 5.00th=[ 253], 10.00th=[ 253], 20.00th=[ 253], | 30.00th=[ 253], 40.00th=[ 384], 50.00th=[ 384], 60.00th=[ 384], | 70.00th=[ 1586], 80.00th=[ 1586], 90.00th=[ 1586], 95.00th=[ 1586], | 99.00th=[ 1586], 99.50th=[ 1586], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 8159, max= 8192, per=73.48%, avg=8175.50, stdev=23.33, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=2.83%, ctx=39, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5806KiB/s (5945kB/s), 1451KiB/s-5354KiB/s (1486kB/s-5483kB/s), io=16.0MiB (16.8MB), run=2295-2822msec WRITE: bw=10.9MiB/s (11.4MB/s), 2903KiB/s-5103KiB/s (2973kB/s-5225kB/s), io=32.0MiB (33.6MB), run=2408-2945msec