== sanity test 398b: DIO and buffer IO race ============== 00:28:03 (1746160083) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.71833 s, 18.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25852: Fri May 2 00:29:16 2025 read: IOPS=22, BW=90.6KiB/s (92.8kB/s)(5992KiB/66114msec) clat (usec): min=88, max=234759, avg=24894.46, stdev=17844.83 lat (usec): min=89, max=234762, avg=24898.96, stdev=17844.47 clat percentiles (usec): | 1.00th=[ 106], 5.00th=[ 265], 10.00th=[ 9372], 20.00th=[ 12780], | 30.00th=[ 15664], 40.00th=[ 18220], 50.00th=[ 21103], 60.00th=[ 25297], | 70.00th=[ 30016], 80.00th=[ 35390], 90.00th=[ 43254], 95.00th=[ 52691], | 99.00th=[ 87557], 99.50th=[101188], 99.90th=[202376], 99.95th=[233833], | 99.99th=[233833] bw ( KiB/s): min= 16, max= 144, per=24.90%, avg=89.65, stdev=24.86, samples=132 iops : min= 4, max= 36, avg=22.02, stdev= 6.26, samples=132 write: IOPS=23, BW=95.2KiB/s (97.5kB/s)(6296KiB/66114msec) clat (usec): min=566, max=617851, avg=18136.80, stdev=24594.26 lat (usec): min=567, max=617852, avg=18148.04, stdev=24591.09 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 873], 10.00th=[ 988], 20.00th=[ 1319], | 30.00th=[ 2933], 40.00th=[ 5735], 50.00th=[ 11863], 60.00th=[ 17171], | 70.00th=[ 23200], 80.00th=[ 31327], 90.00th=[ 45876], 95.00th=[ 55313], | 99.00th=[ 83362], 99.50th=[ 87557], 99.90th=[158335], 99.95th=[616563], | 99.99th=[616563] bw ( KiB/s): min= 7, max= 189, per=25.16%, avg=94.85, stdev=30.61, samples=131 iops : min= 1, max= 47, avg=23.32, stdev= 7.67, samples=131 lat (usec) : 100=0.23%, 250=2.18%, 500=0.23%, 750=0.46%, 1000=5.01% lat (msec) : 2=7.58%, 4=5.05%, 10=9.18%, 20=25.91%, 50=37.57% lat (msec) : 100=6.18%, 250=0.39%, 750=0.03% cpu : usr=0.13%, sys=8.61%, ctx=5138, majf=0, minf=37 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25853: Fri May 2 00:29:16 2025 read: IOPS=22, BW=89.0KiB/s (92.1kB/s)(5940KiB/66013msec) clat (usec): min=56, max=142558, avg=24461.69, stdev=15497.58 lat (usec): min=57, max=142559, avg=24463.43, stdev=15498.16 clat percentiles (usec): | 1.00th=[ 102], 5.00th=[ 145], 10.00th=[ 9372], 20.00th=[ 13173], | 30.00th=[ 15926], 40.00th=[ 18220], 50.00th=[ 21365], 60.00th=[ 25822], | 70.00th=[ 30016], 80.00th=[ 35390], 90.00th=[ 42730], 95.00th=[ 51119], | 99.00th=[ 73925], 99.50th=[ 82314], 99.90th=[132645], 99.95th=[141558], | 99.99th=[141558] bw ( KiB/s): min= 7, max= 152, per=24.79%, avg=89.26, stdev=25.50, samples=130 iops : min= 1, max= 38, avg=21.91, stdev= 6.42, samples=130 write: IOPS=24, BW=96.2KiB/s (98.5kB/s)(6348KiB/66013msec) clat (usec): min=660, max=635412, avg=18557.41, stdev=25603.90 lat (usec): min=661, max=635413, avg=18563.32, stdev=25602.61 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 857], 10.00th=[ 971], 20.00th=[ 1352], | 30.00th=[ 3097], 40.00th=[ 6521], 50.00th=[ 12649], 60.00th=[ 17957], | 70.00th=[ 22414], 80.00th=[ 32375], 90.00th=[ 45351], 95.00th=[ 55837], | 99.00th=[ 91751], 99.50th=[116917], 99.90th=[193987], 99.95th=[633340], | 99.99th=[633340] bw ( KiB/s): min= 7, max= 192, per=24.99%, avg=94.21, stdev=32.84, samples=131 iops : min= 1, max= 48, avg=23.15, stdev= 8.24, samples=131 lat (usec) : 100=0.39%, 250=2.25%, 500=0.20%, 750=0.36%, 1000=5.34% lat (msec) : 2=7.26%, 4=4.85%, 10=7.91%, 20=27.18%, 50=38.02% lat (msec) : 100=5.73%, 250=0.49%, 750=0.03% cpu : usr=0.11%, sys=8.72%, ctx=5177, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25854: Fri May 2 00:29:16 2025 read: IOPS=22, BW=90.6KiB/s (92.8kB/s)(6028KiB/66551msec) clat (usec): min=81, max=644549, avg=25011.48, stdev=22834.27 lat (usec): min=81, max=644550, avg=25013.12, stdev=22834.19 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 255], 10.00th=[ 9110], 20.00th=[ 12911], | 30.00th=[ 15926], 40.00th=[ 18482], 50.00th=[ 21890], 60.00th=[ 25560], | 70.00th=[ 29754], 80.00th=[ 34866], 90.00th=[ 42206], 95.00th=[ 52691], | 99.00th=[ 86508], 99.50th=[ 99091], 99.90th=[164627], 99.95th=[641729], | 99.99th=[641729] bw ( KiB/s): min= 15, max= 168, per=25.07%, avg=90.27, stdev=26.09, samples=132 iops : min= 3, max= 42, avg=22.17, stdev= 6.52, samples=132 write: IOPS=23, BW=94.1KiB/s (96.3kB/s)(6260KiB/66551msec) clat (usec): min=691, max=126240, avg=18284.70, stdev=19662.45 lat (usec): min=691, max=126241, avg=18286.28, stdev=19662.52 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 857], 10.00th=[ 979], 20.00th=[ 1352], | 30.00th=[ 3195], 40.00th=[ 6521], 50.00th=[ 13435], 60.00th=[ 17957], | 70.00th=[ 23462], 80.00th=[ 32113], 90.00th=[ 45351], 95.00th=[ 55313], | 99.00th=[ 91751], 99.50th=[101188], 99.90th=[124257], 99.95th=[126354], | 99.99th=[126354] bw ( KiB/s): min= 7, max= 189, per=24.66%, avg=92.97, stdev=31.34, samples=133 iops : min= 1, max= 47, avg=22.84, stdev= 7.88, samples=133 lat (usec) : 100=0.29%, 250=2.15%, 500=0.49%, 750=0.36%, 1000=4.98% lat (msec) : 2=7.49%, 4=4.69%, 10=8.24%, 20=25.88%, 50=39.00% lat (msec) : 100=5.96%, 250=0.46%, 750=0.03% cpu : usr=0.11%, sys=8.68%, ctx=5208, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25855: Fri May 2 00:29:16 2025 read: IOPS=23, BW=95.1KiB/s (97.3kB/s)(6052KiB/63660msec) clat (usec): min=90, max=122704, avg=23619.35, stdev=14928.90 lat (usec): min=91, max=122706, avg=23620.82, stdev=14929.01 clat percentiles (usec): | 1.00th=[ 106], 5.00th=[ 219], 10.00th=[ 9241], 20.00th=[ 12387], | 30.00th=[ 15139], 40.00th=[ 17957], 50.00th=[ 21103], 60.00th=[ 23987], | 70.00th=[ 28443], 80.00th=[ 34341], 90.00th=[ 42730], 95.00th=[ 48497], | 99.00th=[ 71828], 99.50th=[ 85459], 99.90th=[113771], 99.95th=[123208], | 99.99th=[123208] bw ( KiB/s): min= 40, max= 168, per=26.61%, avg=95.80, stdev=24.85, samples=125 iops : min= 10, max= 42, avg=23.50, stdev= 6.23, samples=125 write: IOPS=24, BW=97.0KiB/s (100kB/s)(6236KiB/63660msec) clat (usec): min=624, max=671637, avg=17740.44, stdev=25002.59 lat (usec): min=625, max=671639, avg=17753.28, stdev=25010.03 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 865], 10.00th=[ 1004], 20.00th=[ 1385], | 30.00th=[ 3228], 40.00th=[ 6325], 50.00th=[ 12518], 60.00th=[ 17433], | 70.00th=[ 22676], 80.00th=[ 30802], 90.00th=[ 41157], 95.00th=[ 53740], | 99.00th=[ 84411], 99.50th=[103285], 99.90th=[141558], 99.95th=[675283], | 99.99th=[675283] bw ( KiB/s): min= 16, max= 207, per=25.90%, avg=97.64, stdev=32.97, samples=126 iops : min= 4, max= 51, avg=23.96, stdev= 8.25, samples=126 lat (usec) : 100=0.23%, 250=2.25%, 500=0.33%, 750=0.42%, 1000=4.62% lat (msec) : 2=7.55%, 4=4.79%, 10=8.98%, 20=26.79%, 50=38.83% lat (msec) : 100=4.75%, 250=0.42%, 750=0.03% cpu : usr=0.10%, sys=8.84%, ctx=5122, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=361KiB/s (369kB/s), 89.0KiB/s-95.1KiB/s (92.1kB/s-97.3kB/s), io=23.4MiB (24.6MB), run=63660-66551msec WRITE: bw=378KiB/s (387kB/s), 94.1KiB/s-97.0KiB/s (96.3kB/s-100kB/s), io=24.6MiB (25.7MB), run=63660-66551msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25856: Fri May 2 00:29:42 2025 read: IOPS=16, BW=65.1KiB/s (66.6kB/s)(5992KiB/92106msec) clat (msec): min=3, max=667, avg=23.00, stdev=26.13 lat (msec): min=3, max=667, avg=23.00, stdev=26.13 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 12], 40.00th=[ 14], 50.00th=[ 16], 60.00th=[ 19], | 70.00th=[ 23], 80.00th=[ 32], 90.00th=[ 49], 95.00th=[ 67], | 99.00th=[ 102], 99.50th=[ 114], 99.90th=[ 192], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 8, max= 200, per=25.09%, avg=64.98, stdev=39.36, samples=183 iops : min= 2, max= 50, avg=15.92, stdev= 9.88, samples=183 write: IOPS=17, BW=68.4KiB/s (69.0kB/s)(6296KiB/92106msec) clat (msec): min=7, max=159, avg=36.44, stdev=23.80 lat (msec): min=7, max=159, avg=36.44, stdev=23.80 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 14], 10.00th=[ 16], 20.00th=[ 18], | 30.00th=[ 21], 40.00th=[ 24], 50.00th=[ 28], 60.00th=[ 34], | 70.00th=[ 43], 80.00th=[ 55], 90.00th=[ 71], 95.00th=[ 83], | 99.00th=[ 124], 99.50th=[ 131], 99.90th=[ 159], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 15, max= 168, per=25.28%, avg=68.52, stdev=39.34, samples=182 iops : min= 3, max= 42, avg=16.81, stdev= 9.87, samples=182 lat (msec) : 4=0.13%, 10=11.69%, 20=34.15%, 50=37.14%, 100=15.27% lat (msec) : 250=1.60%, 750=0.03% cpu : usr=0.04%, sys=4.21%, ctx=3507, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25857: Fri May 2 00:29:42 2025 read: IOPS=16, BW=64.2KiB/s (65.8kB/s)(5940KiB/92494msec) clat (msec): min=3, max=559, avg=23.34, stdev=25.86 lat (msec): min=3, max=559, avg=23.34, stdev=25.86 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 13], 50.00th=[ 16], 60.00th=[ 18], | 70.00th=[ 23], 80.00th=[ 32], 90.00th=[ 51], 95.00th=[ 70], | 99.00th=[ 109], 99.50th=[ 117], 99.90th=[ 184], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 7, max= 192, per=24.61%, avg=63.73, stdev=41.50, samples=183 iops : min= 1, max= 48, avg=15.54, stdev=10.44, samples=183 write: IOPS=17, BW=68.6KiB/s (70.3kB/s)(6348KiB/92494msec) clat (msec): min=6, max=310, avg=36.29, stdev=25.31 lat (msec): min=6, max=310, avg=36.29, stdev=25.31 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 14], 10.00th=[ 16], 20.00th=[ 18], | 30.00th=[ 21], 40.00th=[ 24], 50.00th=[ 28], 60.00th=[ 33], | 70.00th=[ 41], 80.00th=[ 53], 90.00th=[ 71], 95.00th=[ 84], | 99.00th=[ 127], 99.50th=[ 155], 99.90th=[ 201], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 7, max= 192, per=25.03%, avg=67.82, stdev=40.40, samples=183 iops : min= 1, max= 48, avg=16.56, stdev=10.17, samples=183 lat (msec) : 4=0.07%, 10=12.50%, 20=32.88%, 50=38.09%, 100=14.45% lat (msec) : 250=1.95%, 500=0.03%, 750=0.03% cpu : usr=0.07%, sys=4.18%, ctx=3524, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25858: Fri May 2 00:29:42 2025 read: IOPS=16, BW=65.4KiB/s (66.0kB/s)(6028KiB/92158msec) clat (msec): min=3, max=192, avg=22.59, stdev=22.05 lat (msec): min=3, max=192, avg=22.60, stdev=22.05 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 13], 50.00th=[ 15], 60.00th=[ 18], | 70.00th=[ 22], 80.00th=[ 31], 90.00th=[ 52], 95.00th=[ 68], | 99.00th=[ 109], 99.50th=[ 132], 99.90th=[ 184], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 7, max= 183, per=25.13%, avg=65.09, stdev=42.01, samples=183 iops : min= 1, max= 45, avg=15.86, stdev=10.55, samples=183 write: IOPS=16, BW=67.9KiB/s (69.6kB/s)(6260KiB/92158msec) clat (msec): min=6, max=623, avg=36.99, stdev=29.35 lat (msec): min=6, max=623, avg=36.99, stdev=29.35 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 14], 10.00th=[ 16], 20.00th=[ 18], | 30.00th=[ 21], 40.00th=[ 24], 50.00th=[ 28], 60.00th=[ 34], | 70.00th=[ 43], 80.00th=[ 55], 90.00th=[ 71], 95.00th=[ 86], | 99.00th=[ 124], 99.50th=[ 140], 99.90th=[ 321], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 7, max= 168, per=24.94%, avg=67.60, stdev=40.97, samples=183 iops : min= 1, max= 42, avg=16.49, stdev=10.30, samples=183 lat (msec) : 4=0.13%, 10=12.83%, 20=34.44%, 50=36.07%, 100=14.55% lat (msec) : 250=1.92%, 500=0.03%, 750=0.03% cpu : usr=0.10%, sys=4.07%, ctx=3515, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25859: Fri May 2 00:29:42 2025 read: IOPS=16, BW=65.5KiB/s (67.1kB/s)(6052KiB/92379msec) clat (msec): min=4, max=172, avg=23.00, stdev=20.79 lat (msec): min=4, max=172, avg=23.00, stdev=20.79 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 13], 50.00th=[ 15], 60.00th=[ 19], | 70.00th=[ 24], 80.00th=[ 34], 90.00th=[ 52], 95.00th=[ 68], | 99.00th=[ 101], 99.50th=[ 121], 99.90th=[ 165], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 7, max= 200, per=25.20%, avg=65.28, stdev=42.18, samples=182 iops : min= 1, max= 50, avg=15.91, stdev=10.61, samples=182 write: IOPS=16, BW=67.5KiB/s (69.1kB/s)(6236KiB/92379msec) clat (msec): min=6, max=675, avg=36.84, stdev=30.53 lat (msec): min=6, max=675, avg=36.84, stdev=30.53 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 14], 10.00th=[ 16], 20.00th=[ 18], | 30.00th=[ 21], 40.00th=[ 24], 50.00th=[ 28], 60.00th=[ 33], | 70.00th=[ 42], 80.00th=[ 54], 90.00th=[ 70], 95.00th=[ 83], | 99.00th=[ 134], 99.50th=[ 157], 99.90th=[ 317], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 7, max= 191, per=24.72%, avg=66.99, stdev=39.92, samples=183 iops : min= 1, max= 47, avg=16.34, stdev=10.04, samples=183 lat (msec) : 10=12.57%, 20=33.17%, 50=37.40%, 100=14.94%, 250=1.86% lat (msec) : 500=0.03%, 750=0.03% cpu : usr=0.06%, sys=4.19%, ctx=3525, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=260KiB/s (266kB/s), 64.2KiB/s-65.5KiB/s (65.8kB/s-67.1kB/s), io=23.4MiB (24.6MB), run=92106-92494msec WRITE: bw=272KiB/s (278kB/s), 67.5KiB/s-68.6KiB/s (69.1kB/s-70.3kB/s), io=24.6MiB (25.7MB), run=92106-92494msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25954: Fri May 2 00:29:57 2025 read: IOPS=28, BW=462KiB/s (473kB/s)(5920KiB/12807msec) clat (usec): min=71, max=90028, avg=18592.21, stdev=12826.25 lat (usec): min=71, max=90029, avg=18593.80, stdev=12826.16 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[ 121], 10.00th=[ 147], 20.00th=[10028], | 30.00th=[12518], 40.00th=[14484], 50.00th=[16581], 60.00th=[19268], | 70.00th=[22938], 80.00th=[26608], 90.00th=[34866], 95.00th=[42206], | 99.00th=[62129], 99.50th=[86508], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 287, max= 669, per=25.17%, avg=464.40, stdev=112.37, samples=25 iops : min= 17, max= 41, avg=28.60, stdev= 7.09, samples=25 write: IOPS=31, BW=497KiB/s (509kB/s)(6368KiB/12807msec) clat (usec): min=772, max=92991, avg=14750.92, stdev=15148.33 lat (usec): min=773, max=92993, avg=14754.59, stdev=15148.36 clat percentiles (usec): | 1.00th=[ 1045], 5.00th=[ 1319], 10.00th=[ 1500], 20.00th=[ 1975], | 30.00th=[ 3556], 40.00th=[ 6194], 50.00th=[10290], 60.00th=[12911], | 70.00th=[18482], 80.00th=[25560], 90.00th=[36963], 95.00th=[43779], | 99.00th=[70779], 99.50th=[81265], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 284, max= 721, per=25.83%, avg=501.40, stdev=114.96, samples=25 iops : min= 17, max= 45, avg=30.92, stdev= 7.34, samples=25 lat (usec) : 100=0.91%, 250=4.04%, 500=0.91%, 750=0.13%, 1000=0.39% lat (msec) : 2=10.42%, 4=5.99%, 10=12.11%, 20=32.29%, 50=30.73% lat (msec) : 100=2.08% cpu : usr=0.07%, sys=11.65%, ctx=1409, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25955: Fri May 2 00:29:57 2025 read: IOPS=27, BW=442KiB/s (453kB/s)(5744KiB/12981msec) clat (usec): min=91, max=79055, avg=18215.40, stdev=13411.98 lat (usec): min=92, max=79056, avg=18217.08, stdev=13411.86 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 118], 10.00th=[ 145], 20.00th=[ 8225], | 30.00th=[11338], 40.00th=[13829], 50.00th=[16581], 60.00th=[19268], | 70.00th=[22414], 80.00th=[27395], 90.00th=[35914], 95.00th=[43254], | 99.00th=[65799], 99.50th=[71828], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 63, max= 640, per=23.17%, avg=427.44, stdev=131.74, samples=25 iops : min= 3, max= 40, avg=26.28, stdev= 8.35, samples=25 write: IOPS=31, BW=504KiB/s (516kB/s)(6544KiB/12981msec) clat (usec): min=848, max=104861, avg=15370.71, stdev=16044.15 lat (usec): min=850, max=104862, avg=15372.62, stdev=16044.14 clat percentiles (usec): | 1.00th=[ 1057], 5.00th=[ 1287], 10.00th=[ 1500], 20.00th=[ 2311], | 30.00th=[ 3294], 40.00th=[ 5145], 50.00th=[ 10421], 60.00th=[ 15008], | 70.00th=[ 19792], 80.00th=[ 27395], 90.00th=[ 36963], 95.00th=[ 47449], | 99.00th=[ 72877], 99.50th=[ 76022], 99.90th=[105382], 99.95th=[105382], | 99.99th=[105382] bw ( KiB/s): min= 223, max= 736, per=24.85%, avg=482.32, stdev=142.35, samples=25 iops : min= 13, max= 46, avg=29.72, stdev= 9.04, samples=25 lat (usec) : 100=0.39%, 250=5.34%, 500=1.04%, 750=0.13%, 1000=0.39% lat (msec) : 2=9.77%, 4=8.72%, 10=11.98%, 20=28.78%, 50=30.60% lat (msec) : 100=2.73%, 250=0.13% cpu : usr=0.07%, sys=11.57%, ctx=1413, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25956: Fri May 2 00:29:57 2025 read: IOPS=30, BW=485KiB/s (497kB/s)(6256KiB/12894msec) clat (usec): min=86, max=72703, avg=18424.31, stdev=12616.07 lat (usec): min=87, max=72704, avg=18425.38, stdev=12616.08 clat percentiles (usec): | 1.00th=[ 100], 5.00th=[ 119], 10.00th=[ 137], 20.00th=[ 9503], | 30.00th=[12256], 40.00th=[14746], 50.00th=[16909], 60.00th=[20055], | 70.00th=[23462], 80.00th=[27132], 90.00th=[34341], 95.00th=[42730], | 99.00th=[53216], 99.50th=[61604], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 256, max= 670, per=26.16%, avg=482.64, stdev=99.80, samples=25 iops : min= 16, max= 41, avg=29.76, stdev= 6.05, samples=25 write: IOPS=29, BW=468KiB/s (479kB/s)(6032KiB/12894msec) clat (usec): min=907, max=154530, avg=14763.16, stdev=16584.20 lat (usec): min=908, max=154532, avg=14768.88, stdev=16583.15 clat percentiles (usec): | 1.00th=[ 1020], 5.00th=[ 1237], 10.00th=[ 1549], 20.00th=[ 2212], | 30.00th=[ 3458], 40.00th=[ 5669], 50.00th=[ 10028], 60.00th=[ 13173], | 70.00th=[ 17695], 80.00th=[ 24249], 90.00th=[ 38011], 95.00th=[ 44827], | 99.00th=[ 65274], 99.50th=[ 95945], 99.90th=[154141], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 220, max= 798, per=23.29%, avg=452.04, stdev=160.41, samples=25 iops : min= 13, max= 49, avg=27.84, stdev= 9.95, samples=25 lat (usec) : 100=0.39%, 250=6.25%, 500=0.78%, 750=0.13%, 1000=0.52% lat (msec) : 2=8.85%, 4=7.16%, 10=11.59%, 20=30.47%, 50=31.38% lat (msec) : 100=2.34%, 250=0.13% cpu : usr=0.17%, sys=11.33%, ctx=1358, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25957: Fri May 2 00:29:57 2025 read: IOPS=29, BW=475KiB/s (486kB/s)(6032KiB/12701msec) clat (usec): min=74, max=129880, avg=18180.45, stdev=14236.33 lat (usec): min=74, max=129881, avg=18182.76, stdev=14236.37 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 109], 10.00th=[ 127], 20.00th=[ 7504], | 30.00th=[ 11076], 40.00th=[ 14353], 50.00th=[ 17171], 60.00th=[ 20579], | 70.00th=[ 23200], 80.00th=[ 27919], 90.00th=[ 33817], 95.00th=[ 39060], | 99.00th=[ 62653], 99.50th=[102237], 99.90th=[129500], 99.95th=[129500], | 99.99th=[129500] bw ( KiB/s): min= 288, max= 672, per=25.29%, avg=466.60, stdev=93.22, samples=25 iops : min= 18, max= 42, avg=28.60, stdev= 5.79, samples=25 write: IOPS=30, BW=493KiB/s (504kB/s)(6256KiB/12701msec) clat (usec): min=670, max=74222, avg=14473.66, stdev=14851.27 lat (usec): min=671, max=74223, avg=14475.92, stdev=14851.41 clat percentiles (usec): | 1.00th=[ 988], 5.00th=[ 1319], 10.00th=[ 1467], 20.00th=[ 2114], | 30.00th=[ 3294], 40.00th=[ 5145], 50.00th=[ 8356], 60.00th=[13829], | 70.00th=[18744], 80.00th=[26084], 90.00th=[37487], 95.00th=[45351], | 99.00th=[58983], 99.50th=[72877], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 284, max= 798, per=25.09%, avg=487.04, stdev=130.34, samples=25 iops : min= 17, max= 49, avg=29.88, stdev= 8.07, samples=25 lat (usec) : 100=1.43%, 250=5.21%, 500=0.65%, 750=0.39%, 1000=0.52% lat (msec) : 2=9.37%, 4=8.33%, 10=13.54%, 20=26.56%, 50=31.90% lat (msec) : 100=1.82%, 250=0.26% cpu : usr=0.09%, sys=11.50%, ctx=1314, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1845KiB/s (1889kB/s), 442KiB/s-485KiB/s (453kB/s-497kB/s), io=23.4MiB (24.5MB), run=12701-12981msec WRITE: bw=1941KiB/s (1988kB/s), 468KiB/s-504KiB/s (479kB/s-516kB/s), io=24.6MiB (25.8MB), run=12701-12981msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25958: Fri May 2 00:30:03 2025 read: IOPS=19, BW=305KiB/s (312kB/s)(5920KiB/19441msec) clat (msec): min=3, max=113, avg=19.69, stdev=16.55 lat (msec): min=3, max=113, avg=19.70, stdev=16.55 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 16], | 70.00th=[ 20], 80.00th=[ 27], 90.00th=[ 43], 95.00th=[ 58], | 99.00th=[ 80], 99.50th=[ 103], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 127, max= 768, per=24.10%, avg=294.97, stdev=150.85, samples=38 iops : min= 7, max= 48, avg=18.08, stdev= 9.57, samples=38 write: IOPS=20, BW=328KiB/s (335kB/s)(6368KiB/19441msec) clat (msec): min=8, max=115, avg=29.96, stdev=18.09 lat (msec): min=8, max=115, avg=29.97, stdev=18.09 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 21], 50.00th=[ 24], 60.00th=[ 29], | 70.00th=[ 34], 80.00th=[ 42], 90.00th=[ 56], 95.00th=[ 66], | 99.00th=[ 106], 99.50th=[ 114], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 64, max= 640, per=25.06%, avg=322.71, stdev=154.79, samples=38 iops : min= 4, max= 40, avg=19.82, stdev= 9.76, samples=38 lat (msec) : 4=0.13%, 10=12.89%, 20=40.89%, 50=35.16%, 100=10.16% lat (msec) : 250=0.78% cpu : usr=0.14%, sys=4.76%, ctx=892, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25959: Fri May 2 00:30:03 2025 read: IOPS=18, BW=294KiB/s (301kB/s)(5744KiB/19560msec) clat (msec): min=5, max=107, avg=18.67, stdev=16.78 lat (msec): min=5, max=107, avg=18.67, stdev=16.78 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 12], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 23], 90.00th=[ 44], 95.00th=[ 56], | 99.00th=[ 93], 99.50th=[ 107], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 63, max= 640, per=23.81%, avg=291.46, stdev=171.86, samples=39 iops : min= 3, max= 40, avg=17.82, stdev=10.81, samples=39 write: IOPS=20, BW=335KiB/s (343kB/s)(6544KiB/19560msec) clat (msec): min=6, max=202, avg=31.25, stdev=20.40 lat (msec): min=6, max=202, avg=31.26, stdev=20.42 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 22], 50.00th=[ 25], 60.00th=[ 29], | 70.00th=[ 36], 80.00th=[ 45], 90.00th=[ 58], 95.00th=[ 69], | 99.00th=[ 94], 99.50th=[ 97], 99.90th=[ 203], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 159, max= 764, per=25.87%, avg=333.26, stdev=159.21, samples=39 iops : min= 9, max= 47, avg=20.44, stdev=10.07, samples=39 lat (msec) : 10=15.36%, 20=38.54%, 50=34.64%, 100=10.94%, 250=0.52% cpu : usr=0.04%, sys=4.84%, ctx=883, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25960: Fri May 2 00:30:03 2025 read: IOPS=20, BW=322KiB/s (330kB/s)(6256KiB/19418msec) clat (msec): min=4, max=131, avg=19.64, stdev=17.85 lat (msec): min=4, max=131, avg=19.64, stdev=17.85 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 12], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 19], 80.00th=[ 26], 90.00th=[ 46], 95.00th=[ 59], | 99.00th=[ 87], 99.50th=[ 92], 99.90th=[ 132], 99.95th=[ 132], | 99.99th=[ 132] bw ( KiB/s): min= 127, max= 1056, per=26.06%, avg=318.97, stdev=194.90, samples=38 iops : min= 7, max= 66, avg=19.34, stdev=12.37, samples=38 write: IOPS=19, BW=311KiB/s (318kB/s)(6032KiB/19418msec) clat (msec): min=8, max=108, avg=30.69, stdev=18.23 lat (msec): min=8, max=108, avg=30.69, stdev=18.23 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 14], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 20], 40.00th=[ 22], 50.00th=[ 24], 60.00th=[ 29], | 70.00th=[ 33], 80.00th=[ 43], 90.00th=[ 58], 95.00th=[ 68], | 99.00th=[ 101], 99.50th=[ 107], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 127, max= 640, per=23.26%, avg=299.55, stdev=131.36, samples=38 iops : min= 7, max= 40, avg=18.13, stdev= 8.36, samples=38 lat (msec) : 10=16.67%, 20=36.20%, 50=36.33%, 100=10.16%, 250=0.65% cpu : usr=0.06%, sys=4.91%, ctx=917, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25961: Fri May 2 00:30:03 2025 read: IOPS=19, BW=308KiB/s (316kB/s)(6032KiB/19554msec) clat (msec): min=3, max=105, avg=20.64, stdev=18.11 lat (msec): min=3, max=105, avg=20.64, stdev=18.11 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 13], 50.00th=[ 14], 60.00th=[ 16], | 70.00th=[ 21], 80.00th=[ 29], 90.00th=[ 46], 95.00th=[ 65], | 99.00th=[ 95], 99.50th=[ 96], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 96, max= 736, per=24.93%, avg=305.15, stdev=157.13, samples=39 iops : min= 6, max= 46, avg=18.77, stdev= 9.77, samples=39 write: IOPS=19, BW=320KiB/s (328kB/s)(6256KiB/19554msec) clat (msec): min=6, max=204, avg=29.59, stdev=18.86 lat (msec): min=6, max=204, avg=29.61, stdev=18.86 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 13], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 20], 40.00th=[ 22], 50.00th=[ 24], 60.00th=[ 28], | 70.00th=[ 33], 80.00th=[ 41], 90.00th=[ 53], 95.00th=[ 66], | 99.00th=[ 91], 99.50th=[ 116], 99.90th=[ 205], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 64, max= 672, per=24.78%, avg=319.13, stdev=166.23, samples=39 iops : min= 4, max= 42, avg=19.64, stdev=10.39, samples=39 lat (msec) : 4=0.13%, 10=13.67%, 20=38.41%, 50=37.76%, 100=9.51% lat (msec) : 250=0.52% cpu : usr=0.09%, sys=4.96%, ctx=895, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1225KiB/s (1254kB/s), 294KiB/s-322KiB/s (301kB/s-330kB/s), io=23.4MiB (24.5MB), run=19418-19560msec WRITE: bw=1288KiB/s (1319kB/s), 311KiB/s-335KiB/s (318kB/s-343kB/s), io=24.6MiB (25.8MB), run=19418-19560msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25988: Fri May 2 00:30:08 2025 read: IOPS=2, BW=3024KiB/s (3097kB/s)(7168KiB/2370msec) clat (usec): min=461, max=200251, avg=116795.88, stdev=70130.38 lat (usec): min=462, max=200254, avg=116797.54, stdev=70130.71 clat percentiles (usec): | 1.00th=[ 461], 5.00th=[ 461], 10.00th=[ 461], 20.00th=[ 53740], | 30.00th=[112722], 40.00th=[112722], 50.00th=[129500], 60.00th=[137364], | 70.00th=[137364], 80.00th=[183501], 90.00th=[200279], 95.00th=[200279], | 99.00th=[200279], 99.50th=[200279], 99.90th=[200279], 99.95th=[200279], | 99.99th=[200279] bw ( KiB/s): min= 2039, max= 8110, per=47.93%, avg=4062.67, stdev=3505.09, samples=3 iops : min= 1, max= 7, avg= 3.00, stdev= 3.46, samples=3 write: IOPS=2, BW=2160KiB/s (2212kB/s)(5120KiB/2370msec) clat (msec): min=33, max=571, avg=251.09, stdev=208.15 lat (msec): min=33, max=571, avg=251.15, stdev=208.17 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34], | 30.00th=[ 146], 40.00th=[ 146], 50.00th=[ 174], 60.00th=[ 174], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 2027, max= 4000, per=22.66%, avg=2688.67, stdev=1135.66, samples=3 iops : min= 1, max= 3, avg= 1.67, stdev= 1.15, samples=3 lat (usec) : 500=8.33% lat (msec) : 50=8.33%, 100=8.33%, 250=58.33%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=11.02%, ctx=263, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25989: Fri May 2 00:30:08 2025 read: IOPS=0, BW=848KiB/s (868kB/s)(2048KiB/2416msec) clat (msec): min=19, max=203, avg=111.71, stdev=130.37 lat (msec): min=19, max=203, avg=111.71, stdev=130.37 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 20], | 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 20], 60.00th=[ 205], | 70.00th=[ 205], 80.00th=[ 205], 90.00th=[ 205], 95.00th=[ 205], | 99.00th=[ 205], 99.50th=[ 205], 99.90th=[ 205], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 1988, max= 2027, per=23.68%, avg=2007.50, stdev=27.58, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4238KiB/s (4340kB/s)(10.0MiB/2416msec) clat (msec): min=98, max=514, avg=211.86, stdev=127.76 lat (msec): min=98, max=514, avg=211.97, stdev=127.75 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 100], 10.00th=[ 100], 20.00th=[ 116], | 30.00th=[ 120], 40.00th=[ 126], 50.00th=[ 153], 60.00th=[ 211], | 70.00th=[ 243], 80.00th=[ 243], 90.00th=[ 326], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 1924, max= 5965, per=29.47%, avg=3496.75, stdev=1914.06, samples=4 iops : min= 1, max= 5, avg= 2.50, stdev= 1.91, samples=4 lat (msec) : 20=8.33%, 100=8.33%, 250=66.67%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=12.84%, ctx=112, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25990: Fri May 2 00:30:08 2025 read: IOPS=2, BW=2725KiB/s (2790kB/s)(6144KiB/2255msec) clat (usec): min=562, max=211349, avg=135628.29, stdev=75043.57 lat (usec): min=564, max=211351, avg=135629.86, stdev=75043.60 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 562], 10.00th=[ 562], 20.00th=[128451], | 30.00th=[128451], 40.00th=[131597], 50.00th=[131597], 60.00th=[141558], | 70.00th=[200279], 80.00th=[200279], 90.00th=[210764], 95.00th=[210764], | 99.00th=[210764], 99.50th=[210764], 99.90th=[210764], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 3930, max= 4096, per=47.35%, avg=4013.00, stdev=117.38, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2725KiB/s (2790kB/s)(6144KiB/2255msec) clat (msec): min=103, max=583, avg=218.44, stdev=182.63 lat (msec): min=103, max=583, avg=218.52, stdev=182.63 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 118], | 30.00th=[ 118], 40.00th=[ 146], 50.00th=[ 146], 60.00th=[ 150], | 70.00th=[ 211], 80.00th=[ 211], 90.00th=[ 584], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 1965, max= 4063, per=21.29%, avg=2526.75, stdev=1024.79, samples=4 iops : min= 1, max= 3, avg= 1.75, stdev= 0.96, samples=4 lat (usec) : 750=8.33% lat (msec) : 250=83.33%, 750=8.33% cpu : usr=0.00%, sys=13.98%, ctx=200, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25991: Fri May 2 00:30:08 2025 read: IOPS=2, BW=2263KiB/s (2318kB/s)(5120KiB/2262msec) clat (usec): min=540, max=249413, avg=98657.76, stdev=105778.23 lat (usec): min=542, max=249414, avg=98659.41, stdev=105777.80 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 537], 10.00th=[ 537], 20.00th=[ 537], | 30.00th=[ 668], 40.00th=[ 668], 50.00th=[ 91751], 60.00th=[ 91751], | 70.00th=[152044], 80.00th=[152044], 90.00th=[248513], 95.00th=[248513], | 99.00th=[248513], 99.50th=[248513], 99.90th=[248513], 99.95th=[248513], | 99.99th=[248513] bw ( KiB/s): min= 8159, max= 8159, per=96.26%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3169KiB/s (3245kB/s)(7168KiB/2262msec) clat (msec): min=91, max=624, avg=209.45, stdev=193.91 lat (msec): min=91, max=624, avg=209.56, stdev=193.92 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 92], 10.00th=[ 92], 20.00th=[ 99], | 30.00th=[ 106], 40.00th=[ 106], 50.00th=[ 126], 60.00th=[ 142], | 70.00th=[ 142], 80.00th=[ 279], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 2039, max= 5988, per=33.82%, avg=4013.50, stdev=2792.36, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 lat (usec) : 750=16.67% lat (msec) : 100=25.00%, 250=41.67%, 500=8.33%, 750=8.33% cpu : usr=0.04%, sys=13.22%, ctx=208, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8477KiB/s (8680kB/s), 848KiB/s-3024KiB/s (868kB/s-3097kB/s), io=20.0MiB (20.0MB), run=2255-2416msec WRITE: bw=11.6MiB/s (12.2MB/s), 2160KiB/s-4238KiB/s (2212kB/s-4340kB/s), io=28.0MiB (29.4MB), run=2255-2416msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25983: Fri May 2 00:30:08 2025 read: IOPS=2, BW=2621KiB/s (2684kB/s)(7168KiB/2735msec) clat (msec): min=44, max=401, avg=157.92, stdev=119.96 lat (msec): min=44, max=401, avg=157.93, stdev=119.96 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 74], | 30.00th=[ 108], 40.00th=[ 108], 50.00th=[ 112], 60.00th=[ 159], | 70.00th=[ 159], 80.00th=[ 207], 90.00th=[ 401], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 2007, max= 6023, per=55.20%, avg=4023.00, stdev=2008.05, samples=3 iops : min= 1, max= 5, avg= 3.00, stdev= 2.00, samples=3 write: IOPS=1, BW=1872KiB/s (1917kB/s)(5120KiB/2735msec) clat (msec): min=210, max=596, avg=304.22, stdev=164.38 lat (msec): min=210, max=596, avg=304.30, stdev=164.38 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 211], 10.00th=[ 211], 20.00th=[ 211], | 30.00th=[ 220], 40.00th=[ 220], 50.00th=[ 234], 60.00th=[ 234], | 70.00th=[ 262], 80.00th=[ 262], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 2007, max= 4071, per=24.83%, avg=2533.00, stdev=1025.40, samples=4 iops : min= 1, max= 3, avg= 1.50, stdev= 1.00, samples=4 lat (msec) : 50=8.33%, 100=8.33%, 250=58.33%, 500=16.67%, 750=8.33% cpu : usr=0.00%, sys=2.27%, ctx=40, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25984: Fri May 2 00:30:08 2025 read: IOPS=0, BW=729KiB/s (746kB/s)(2048KiB/2810msec) clat (msec): min=112, max=602, avg=357.64, stdev=346.77 lat (msec): min=112, max=602, avg=357.64, stdev=346.77 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 113], 10.00th=[ 113], 20.00th=[ 113], | 30.00th=[ 113], 40.00th=[ 113], 50.00th=[ 113], 60.00th=[ 600], | 70.00th=[ 600], 80.00th=[ 600], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 1980, max= 2039, per=27.57%, avg=2009.50, stdev=41.72, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3644KiB/s (3732kB/s)(10.0MiB/2810msec) clat (msec): min=33, max=713, avg=206.99, stdev=205.80 lat (msec): min=33, max=713, avg=207.07, stdev=205.80 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 39], | 30.00th=[ 80], 40.00th=[ 112], 50.00th=[ 122], 60.00th=[ 155], | 70.00th=[ 268], 80.00th=[ 268], 90.00th=[ 368], 95.00th=[ 718], | 99.00th=[ 718], 99.50th=[ 718], 99.90th=[ 718], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 2011, max= 5941, per=34.44%, avg=3513.75, stdev=1878.93, samples=4 iops : min= 1, max= 5, avg= 2.75, stdev= 1.71, samples=4 lat (msec) : 50=16.67%, 100=8.33%, 250=41.67%, 500=16.67%, 750=16.67% cpu : usr=0.04%, sys=4.81%, ctx=59, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25985: Fri May 2 00:30:08 2025 read: IOPS=2, BW=2321KiB/s (2377kB/s)(6144KiB/2647msec) clat (msec): min=30, max=358, avg=237.12, stdev=111.55 lat (msec): min=30, max=358, avg=237.12, stdev=111.55 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 230], | 30.00th=[ 230], 40.00th=[ 234], 50.00th=[ 234], 60.00th=[ 279], | 70.00th=[ 292], 80.00th=[ 292], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 2027, max= 4055, per=34.88%, avg=2542.00, stdev=1008.69, samples=4 iops : min= 1, max= 3, avg= 1.50, stdev= 1.00, samples=4 write: IOPS=2, BW=2321KiB/s (2377kB/s)(6144KiB/2647msec) clat (msec): min=109, max=437, avg=193.49, stdev=127.46 lat (msec): min=109, max=437, avg=193.56, stdev=127.46 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 110], 10.00th=[ 110], 20.00th=[ 110], | 30.00th=[ 110], 40.00th=[ 113], 50.00th=[ 113], 60.00th=[ 178], | 70.00th=[ 215], 80.00th=[ 215], 90.00th=[ 439], 95.00th=[ 439], | 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 2027, max= 4055, per=23.90%, avg=2439.00, stdev=903.41, samples=5 iops : min= 1, max= 3, avg= 1.40, stdev= 0.89, samples=5 lat (msec) : 50=8.33%, 250=58.33%, 500=33.33% cpu : usr=0.00%, sys=2.80%, ctx=44, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25986: Fri May 2 00:30:08 2025 read: IOPS=1, BW=1997KiB/s (2045kB/s)(5120KiB/2564msec) clat (msec): min=83, max=358, avg=206.69, stdev=126.52 lat (msec): min=83, max=358, avg=206.69, stdev=126.52 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 84], 10.00th=[ 84], 20.00th=[ 84], | 30.00th=[ 128], 40.00th=[ 128], 50.00th=[ 138], 60.00th=[ 138], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 3900, max= 4087, per=54.80%, avg=3993.50, stdev=132.23, samples=2 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2796KiB/s (2863kB/s)(7168KiB/2564msec) clat (msec): min=96, max=416, avg=202.62, stdev=114.09 lat (msec): min=96, max=416, avg=202.72, stdev=114.05 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 96], 10.00th=[ 96], 20.00th=[ 129], | 30.00th=[ 133], 40.00th=[ 133], 50.00th=[ 140], 60.00th=[ 224], | 70.00th=[ 224], 80.00th=[ 284], 90.00th=[ 418], 95.00th=[ 418], | 99.00th=[ 418], 99.50th=[ 418], 99.90th=[ 418], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 2043, max= 6131, per=35.06%, avg=3577.25, stdev=1955.68, samples=4 iops : min= 1, max= 5, avg= 2.75, stdev= 1.71, samples=4 lat (msec) : 100=16.67%, 250=50.00%, 500=33.33% cpu : usr=0.16%, sys=4.14%, ctx=58, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7288KiB/s (7463kB/s), 729KiB/s-2621KiB/s (746kB/s-2684kB/s), io=20.0MiB (20.0MB), run=2564-2810msec WRITE: bw=9.96MiB/s (10.4MB/s), 1872KiB/s-3644KiB/s (1917kB/s-3732kB/s), io=28.0MiB (29.4MB), run=2564-2810msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=26000: Fri May 2 00:30:14 2025 read: IOPS=1, BW=8143KiB/s (8339kB/s)(12.0MiB/1509msec) clat (msec): min=395, max=526, avg=470.13, stdev=67.40 lat (msec): min=395, max=526, avg=470.14, stdev=67.40 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 397], 10.00th=[ 397], 20.00th=[ 397], | 30.00th=[ 397], 40.00th=[ 489], 50.00th=[ 489], 60.00th=[ 489], | 70.00th=[ 527], 80.00th=[ 527], 90.00th=[ 527], 95.00th=[ 527], | 99.00th=[ 527], 99.50th=[ 527], 99.90th=[ 527], 99.95th=[ 527], | 99.99th=[ 527] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.27%, sys=14.32%, ctx=545, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26002: Fri May 2 00:30:14 2025 read: IOPS=0, BW=1297KiB/s (1328kB/s)(4096KiB/3159msec) clat (nsec): min=439203k, max=439203k, avg=439202747.00, stdev= 0.00 lat (nsec): min=439204k, max=439204k, avg=439204241.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 439], 10.00th=[ 439], 20.00th=[ 439], | 30.00th=[ 439], 40.00th=[ 439], 50.00th=[ 439], 60.00th=[ 439], | 70.00th=[ 439], 80.00th=[ 439], 90.00th=[ 439], 95.00th=[ 439], | 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2593KiB/s (2655kB/s)(8192KiB/3159msec) clat (msec): min=1231, max=1417, avg=1324.79, stdev=131.42 lat (msec): min=1233, max=1418, avg=1325.83, stdev=130.39 clat percentiles (msec): | 1.00th=[ 1234], 5.00th=[ 1234], 10.00th=[ 1234], 20.00th=[ 1234], | 30.00th=[ 1234], 40.00th=[ 1234], 50.00th=[ 1234], 60.00th=[ 1418], | 70.00th=[ 1418], 80.00th=[ 1418], 90.00th=[ 1418], 95.00th=[ 1418], | 99.00th=[ 1418], 99.50th=[ 1418], 99.90th=[ 1418], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 8192, max= 8192, per=78.98%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=12.10%, ctx=175, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26004: Fri May 2 00:30:14 2025 write: IOPS=0, BW=3992KiB/s (4088kB/s)(12.0MiB/3078msec) clat (msec): min=594, max=1703, avg=986.49, stdev=621.49 lat (msec): min=594, max=1703, avg=986.89, stdev=621.48 clat percentiles (msec): | 1.00th=[ 592], 5.00th=[ 592], 10.00th=[ 592], 20.00th=[ 592], | 30.00th=[ 592], 40.00th=[ 659], 50.00th=[ 659], 60.00th=[ 659], | 70.00th=[ 1703], 80.00th=[ 1703], 90.00th=[ 1703], 95.00th=[ 1703], | 99.00th=[ 1703], 99.50th=[ 1703], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7968, max= 8175, per=77.82%, avg=8071.50, stdev=146.37, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 750=66.67% cpu : usr=0.03%, sys=13.13%, ctx=100, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26006: Fri May 2 00:30:14 2025 write: IOPS=1, BW=5198KiB/s (5323kB/s)(12.0MiB/2364msec) clat (msec): min=512, max=1166, avg=756.98, stdev=357.06 lat (msec): min=512, max=1167, avg=757.28, stdev=357.06 clat percentiles (msec): | 1.00th=[ 514], 5.00th=[ 514], 10.00th=[ 514], 20.00th=[ 514], | 30.00th=[ 514], 40.00th=[ 592], 50.00th=[ 592], 60.00th=[ 592], | 70.00th=[ 1167], 80.00th=[ 1167], 90.00th=[ 1167], 95.00th=[ 1167], | 99.00th=[ 1167], 99.50th=[ 1167], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 8175, max= 8192, per=78.90%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 750=66.67% cpu : usr=0.00%, sys=17.35%, ctx=101, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5186KiB/s (5311kB/s), 1297KiB/s-8143KiB/s (1328kB/s-8339kB/s), io=16.0MiB (16.8MB), run=1509-3159msec WRITE: bw=10.1MiB/s (10.6MB/s), 2593KiB/s-5198KiB/s (2655kB/s-5323kB/s), io=32.0MiB (33.6MB), run=2364-3159msec rand-rw: (groupid=0, jobs=1): err= 0: pid=26001: Fri May 2 00:30:15 2025 read: IOPS=0, BW=3760KiB/s (3850kB/s)(12.0MiB/3268msec) clat (msec): min=857, max=1335, avg=1048.40, stdev=253.34 lat (msec): min=857, max=1335, avg=1048.40, stdev=253.34 clat percentiles (msec): | 1.00th=[ 860], 5.00th=[ 860], 10.00th=[ 860], 20.00th=[ 860], | 30.00th=[ 860], 40.00th=[ 953], 50.00th=[ 953], 60.00th=[ 953], | 70.00th=[ 1334], 80.00th=[ 1334], 90.00th=[ 1334], 95.00th=[ 1334], | 99.00th=[ 1334], 99.50th=[ 1334], 99.90th=[ 1334], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 8143, max= 8192, per=100.00%, avg=8167.50, stdev=34.65, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 1000=66.67% cpu : usr=0.00%, sys=3.83%, ctx=39, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26003: Fri May 2 00:30:15 2025 read: IOPS=0, BW=1209KiB/s (1238kB/s)(4096KiB/3388msec) clat (nsec): min=931867k, max=931867k, avg=931866712.00, stdev= 0.00 lat (nsec): min=931869k, max=931869k, avg=931869266.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 936], 5.00th=[ 936], 10.00th=[ 936], 20.00th=[ 936], | 30.00th=[ 936], 40.00th=[ 936], 50.00th=[ 936], 60.00th=[ 936], | 70.00th=[ 936], 80.00th=[ 936], 90.00th=[ 936], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2418KiB/s (2476kB/s)(8192KiB/3388msec) clat (msec): min=915, max=1301, avg=1108.57, stdev=273.48 lat (msec): min=915, max=1302, avg=1109.01, stdev=273.33 clat percentiles (msec): | 1.00th=[ 919], 5.00th=[ 919], 10.00th=[ 919], 20.00th=[ 919], | 30.00th=[ 919], 40.00th=[ 919], 50.00th=[ 919], 60.00th=[ 1301], | 70.00th=[ 1301], 80.00th=[ 1301], 90.00th=[ 1301], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 8192, max= 8192, per=85.78%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 1000=66.67% cpu : usr=0.00%, sys=3.40%, ctx=44, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26005: Fri May 2 00:30:15 2025 write: IOPS=0, BW=3581KiB/s (3667kB/s)(12.0MiB/3431msec) clat (msec): min=730, max=1342, avg=1112.04, stdev=332.71 lat (msec): min=730, max=1347, avg=1113.80, stdev=334.19 clat percentiles (msec): | 1.00th=[ 735], 5.00th=[ 735], 10.00th=[ 735], 20.00th=[ 735], | 30.00th=[ 735], 40.00th=[ 1267], 50.00th=[ 1267], 60.00th=[ 1267], | 70.00th=[ 1351], 80.00th=[ 1351], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 8047, max= 8078, per=84.42%, avg=8062.50, stdev=21.92, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 750=33.33% cpu : usr=0.09%, sys=2.57%, ctx=62, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26007: Fri May 2 00:30:15 2025 write: IOPS=1, BW=4139KiB/s (4238kB/s)(12.0MiB/2969msec) clat (msec): min=429, max=1354, avg=935.88, stdev=468.62 lat (msec): min=429, max=1354, avg=936.21, stdev=468.63 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 430], 10.00th=[ 430], 20.00th=[ 430], | 30.00th=[ 430], 40.00th=[ 1028], 50.00th=[ 1028], 60.00th=[ 1028], | 70.00th=[ 1351], 80.00th=[ 1351], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 8175, max= 8175, per=85.60%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33% cpu : usr=0.03%, sys=5.22%, ctx=61, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4836KiB/s (4952kB/s), 1209KiB/s-3760KiB/s (1238kB/s-3850kB/s), io=16.0MiB (16.8MB), run=3268-3388msec WRITE: bw=9551KiB/s (9780kB/s), 2418KiB/s-4139KiB/s (2476kB/s-4238kB/s), io=32.0MiB (33.6MB), run=2969-3431msec