== sanity test 398b: DIO and buffer IO race ============== 03:11:05 (1746083465) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.10384 s, 23.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2080: Thu May 1 03:12:02 2025 read: IOPS=28, BW=114KiB/s (117kB/s)(5992KiB/52410msec) clat (usec): min=46, max=367254, avg=15725.27, stdev=18655.66 lat (usec): min=47, max=367255, avg=15726.34, stdev=18655.69 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 95], 10.00th=[ 108], 20.00th=[ 159], | 30.00th=[ 7767], 40.00th=[ 10290], 50.00th=[ 12911], 60.00th=[ 15795], | 70.00th=[ 19268], 80.00th=[ 24249], 90.00th=[ 32900], 95.00th=[ 42206], | 99.00th=[ 66323], 99.50th=[ 85459], 99.90th=[270533], 99.95th=[367002], | 99.99th=[367002] bw ( KiB/s): min= 8, max= 240, per=24.99%, avg=112.96, stdev=40.66, samples=104 iops : min= 2, max= 60, avg=27.90, stdev=10.20, samples=104 write: IOPS=30, BW=120KiB/s (123kB/s)(6296KiB/52410msec) clat (usec): min=601, max=225464, avg=18150.77, stdev=20771.34 lat (usec): min=602, max=225464, avg=18158.93, stdev=20767.67 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 725], 10.00th=[ 775], 20.00th=[ 1045], | 30.00th=[ 3097], 40.00th=[ 7373], 50.00th=[ 12911], 60.00th=[ 18220], | 70.00th=[ 23725], 80.00th=[ 30802], 90.00th=[ 43254], 95.00th=[ 56361], | 99.00th=[ 86508], 99.50th=[103285], 99.90th=[212861], 99.95th=[225444], | 99.99th=[225444] bw ( KiB/s): min= 24, max= 232, per=25.24%, avg=119.41, stdev=43.30, samples=103 iops : min= 6, max= 58, avg=29.51, stdev=10.90, samples=103 lat (usec) : 50=0.03%, 100=3.55%, 250=6.54%, 500=0.55%, 750=4.04% lat (usec) : 1000=6.15% lat (msec) : 2=3.52%, 4=4.07%, 10=13.15%, 20=26.27%, 50=27.83% lat (msec) : 100=3.81%, 250=0.42%, 500=0.07% cpu : usr=0.11%, sys=8.89%, ctx=4855, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2083: Thu May 1 03:12:02 2025 read: IOPS=27, BW=112KiB/s (115kB/s)(5940KiB/53083msec) clat (usec): min=70, max=299540, avg=15961.05, stdev=17619.29 lat (usec): min=71, max=299543, avg=15962.19, stdev=17619.31 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 96], 10.00th=[ 109], 20.00th=[ 139], | 30.00th=[ 7635], 40.00th=[ 10683], 50.00th=[ 13435], 60.00th=[ 15926], | 70.00th=[ 20317], 80.00th=[ 24773], 90.00th=[ 33424], 95.00th=[ 43254], | 99.00th=[ 65799], 99.50th=[ 99091], 99.90th=[219153], 99.95th=[299893], | 99.99th=[299893] bw ( KiB/s): min= 16, max= 252, per=24.81%, avg=112.13, stdev=44.71, samples=105 iops : min= 4, max= 63, avg=27.72, stdev=11.12, samples=105 write: IOPS=29, BW=120KiB/s (122kB/s)(6348KiB/53083msec) clat (usec): min=619, max=149523, avg=18315.74, stdev=19498.81 lat (usec): min=619, max=149523, avg=18317.77, stdev=19498.49 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 742], 10.00th=[ 783], 20.00th=[ 1319], | 30.00th=[ 3130], 40.00th=[ 6915], 50.00th=[ 12387], 60.00th=[ 19006], | 70.00th=[ 24511], 80.00th=[ 31851], 90.00th=[ 44827], 95.00th=[ 58459], | 99.00th=[ 81265], 99.50th=[ 87557], 99.90th=[137364], 99.95th=[149947], | 99.99th=[149947] bw ( KiB/s): min= 31, max= 276, per=25.29%, avg=119.63, stdev=44.57, samples=105 iops : min= 7, max= 69, avg=29.60, stdev=11.14, samples=105 lat (usec) : 100=3.48%, 250=7.06%, 500=0.59%, 750=3.39%, 1000=5.57% lat (msec) : 2=4.00%, 4=4.95%, 10=11.85%, 20=24.71%, 50=29.23% lat (msec) : 100=4.82%, 250=0.33%, 500=0.03% cpu : usr=0.11%, sys=8.91%, ctx=4758, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2085: Thu May 1 03:12:02 2025 read: IOPS=28, BW=114KiB/s (117kB/s)(6028KiB/52881msec) clat (usec): min=68, max=163399, avg=15222.84, stdev=14456.85 lat (usec): min=69, max=163401, avg=15224.37, stdev=14457.09 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 96], 10.00th=[ 108], 20.00th=[ 145], | 30.00th=[ 7963], 40.00th=[ 10421], 50.00th=[ 12911], 60.00th=[ 15926], | 70.00th=[ 19006], 80.00th=[ 24511], 90.00th=[ 32637], 95.00th=[ 39584], | 99.00th=[ 64750], 99.50th=[ 73925], 99.90th=[127402], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 32, max= 224, per=25.20%, avg=113.89, stdev=41.28, samples=104 iops : min= 8, max= 56, avg=28.06, stdev=10.38, samples=104 write: IOPS=29, BW=118KiB/s (121kB/s)(6260KiB/52881msec) clat (usec): min=533, max=394619, avg=18946.27, stdev=22983.29 lat (usec): min=535, max=394620, avg=18948.06, stdev=22982.94 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 725], 10.00th=[ 766], 20.00th=[ 955], | 30.00th=[ 2868], 40.00th=[ 7111], 50.00th=[ 13566], 60.00th=[ 18482], | 70.00th=[ 24249], 80.00th=[ 32113], 90.00th=[ 45351], 95.00th=[ 58459], | 99.00th=[ 94897], 99.50th=[116917], 99.90th=[181404], 99.95th=[396362], | 99.99th=[396362] bw ( KiB/s): min= 48, max= 216, per=24.93%, avg=117.94, stdev=36.75, samples=104 iops : min= 12, max= 54, avg=29.07, stdev= 9.18, samples=104 lat (usec) : 100=3.55%, 250=6.97%, 500=0.23%, 750=3.97%, 1000=6.71% lat (msec) : 2=3.16%, 4=4.10%, 10=12.21%, 20=26.33%, 50=27.96% lat (msec) : 100=4.20%, 250=0.59%, 500=0.03% cpu : usr=0.12%, sys=9.03%, ctx=4852, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2087: Thu May 1 03:12:02 2025 read: IOPS=28, BW=116KiB/s (119kB/s)(6052KiB/52205msec) clat (usec): min=73, max=164151, avg=15017.68, stdev=13295.07 lat (usec): min=74, max=164152, avg=15018.75, stdev=13295.07 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 95], 10.00th=[ 109], 20.00th=[ 149], | 30.00th=[ 8291], 40.00th=[ 10814], 50.00th=[ 13304], 60.00th=[ 15533], | 70.00th=[ 19268], 80.00th=[ 23987], 90.00th=[ 32113], 95.00th=[ 38536], | 99.00th=[ 53216], 99.50th=[ 56886], 99.90th=[124257], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 15, max= 231, per=25.80%, avg=116.62, stdev=42.71, samples=103 iops : min= 3, max= 57, avg=28.86, stdev=10.69, samples=103 write: IOPS=29, BW=119KiB/s (122kB/s)(6236KiB/52205msec) clat (usec): min=535, max=436182, avg=18707.24, stdev=23887.40 lat (usec): min=535, max=436185, avg=18710.58, stdev=23886.87 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 742], 10.00th=[ 783], 20.00th=[ 1123], | 30.00th=[ 2999], 40.00th=[ 7373], 50.00th=[ 13435], 60.00th=[ 18744], | 70.00th=[ 23987], 80.00th=[ 31065], 90.00th=[ 43779], 95.00th=[ 55313], | 99.00th=[ 83362], 99.50th=[135267], 99.90th=[265290], 99.95th=[434111], | 99.99th=[434111] bw ( KiB/s): min= 48, max= 263, per=25.35%, avg=119.90, stdev=41.95, samples=103 iops : min= 12, max= 65, avg=29.68, stdev=10.45, samples=103 lat (usec) : 100=3.42%, 250=7.06%, 500=0.13%, 750=3.32%, 1000=6.25% lat (msec) : 2=4.00%, 4=3.94%, 10=12.14%, 20=26.53%, 50=29.13% lat (msec) : 100=3.65%, 250=0.36%, 500=0.07% cpu : usr=0.12%, sys=8.96%, ctx=4756, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=452KiB/s (463kB/s), 112KiB/s-116KiB/s (115kB/s-119kB/s), io=23.4MiB (24.6MB), run=52205-53083msec WRITE: bw=474KiB/s (485kB/s), 118KiB/s-120KiB/s (121kB/s-123kB/s), io=24.6MiB (25.7MB), run=52205-53083msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2081: Thu May 1 03:13:47 2025 read: IOPS=9, BW=37.0KiB/s (38.9kB/s)(5992KiB/157783msec) clat (msec): min=2, max=371, avg=13.93, stdev=16.84 lat (msec): min=2, max=371, avg=13.93, stdev=16.84 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 17], 90.00th=[ 29], 95.00th=[ 43], | 99.00th=[ 65], 99.50th=[ 73], 99.90th=[ 182], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 7, max= 136, per=25.93%, avg=39.42, stdev=25.06, samples=300 iops : min= 1, max= 34, avg= 9.52, stdev= 6.32, samples=300 write: IOPS=9, BW=39.9KiB/s (40.9kB/s)(6296KiB/157783msec) clat (msec): min=26, max=327, avg=86.80, stdev=44.44 lat (msec): min=26, max=327, avg=86.81, stdev=44.44 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 42], 10.00th=[ 46], 20.00th=[ 54], | 30.00th=[ 62], 40.00th=[ 68], 50.00th=[ 75], 60.00th=[ 83], | 70.00th=[ 94], 80.00th=[ 111], 90.00th=[ 146], 95.00th=[ 184], | 99.00th=[ 241], 99.50th=[ 275], 99.90th=[ 326], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 7, max= 87, per=24.85%, avg=39.51, stdev=16.36, samples=314 iops : min= 1, max= 21, avg= 9.54, stdev= 4.17, samples=314 lat (msec) : 4=1.11%, 10=26.37%, 20=12.92%, 50=14.71%, 100=31.67% lat (msec) : 250=12.83%, 500=0.39% cpu : usr=0.04%, sys=2.31%, ctx=3529, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2082: Thu May 1 03:13:47 2025 read: IOPS=9, BW=37.8KiB/s (38.7kB/s)(5940KiB/157318msec) clat (msec): min=2, max=320, avg=13.28, stdev=14.22 lat (msec): min=2, max=320, avg=13.28, stdev=14.22 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 17], 90.00th=[ 27], 95.00th=[ 37], | 99.00th=[ 64], 99.50th=[ 71], 99.90th=[ 112], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 7, max= 184, per=26.87%, avg=40.85, stdev=26.52, samples=288 iops : min= 1, max= 46, avg= 9.98, stdev= 6.68, samples=288 write: IOPS=10, BW=40.4KiB/s (41.3kB/s)(6348KiB/157318msec) clat (msec): min=21, max=428, avg=86.53, stdev=45.85 lat (msec): min=21, max=428, avg=86.53, stdev=45.85 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 42], 10.00th=[ 46], 20.00th=[ 53], | 30.00th=[ 61], 40.00th=[ 68], 50.00th=[ 74], 60.00th=[ 83], | 70.00th=[ 93], 80.00th=[ 110], 90.00th=[ 148], 95.00th=[ 186], | 99.00th=[ 249], 99.50th=[ 284], 99.90th=[ 334], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 7, max= 88, per=25.21%, avg=40.08, stdev=16.73, samples=313 iops : min= 1, max= 22, avg= 9.77, stdev= 4.24, samples=313 lat (msec) : 4=0.88%, 10=27.08%, 20=12.57%, 50=14.75%, 100=31.87% lat (msec) : 250=12.34%, 500=0.52% cpu : usr=0.04%, sys=2.28%, ctx=3501, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2084: Thu May 1 03:13:47 2025 read: IOPS=9, BW=38.2KiB/s (39.1kB/s)(6028KiB/157831msec) clat (msec): min=2, max=214, avg=13.86, stdev=14.23 lat (msec): min=2, max=214, avg=13.86, stdev=14.23 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 18], 90.00th=[ 29], 95.00th=[ 40], | 99.00th=[ 66], 99.50th=[ 87], 99.90th=[ 153], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 7, max= 112, per=25.84%, avg=39.28, stdev=24.41, samples=303 iops : min= 1, max= 28, avg= 9.55, stdev= 6.15, samples=303 write: IOPS=9, BW=39.7KiB/s (40.6kB/s)(6260KiB/157831msec) clat (msec): min=27, max=428, avg=87.32, stdev=45.14 lat (msec): min=27, max=428, avg=87.33, stdev=45.13 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 41], 10.00th=[ 46], 20.00th=[ 54], | 30.00th=[ 62], 40.00th=[ 69], 50.00th=[ 75], 60.00th=[ 84], | 70.00th=[ 95], 80.00th=[ 112], 90.00th=[ 146], 95.00th=[ 182], | 99.00th=[ 251], 99.50th=[ 275], 99.90th=[ 388], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 7, max= 80, per=24.74%, avg=39.33, stdev=16.10, samples=314 iops : min= 1, max= 20, avg= 9.57, stdev= 4.08, samples=314 lat (msec) : 4=0.98%, 10=26.79%, 20=12.76%, 50=14.65%, 100=31.25% lat (msec) : 250=13.05%, 500=0.52% cpu : usr=0.05%, sys=2.33%, ctx=3546, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2086: Thu May 1 03:13:47 2025 read: IOPS=9, BW=38.5KiB/s (39.5kB/s)(6052KiB/157005msec) clat (msec): min=3, max=177, avg=13.65, stdev=13.44 lat (msec): min=3, max=177, avg=13.65, stdev=13.44 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 18], 90.00th=[ 28], 95.00th=[ 42], | 99.00th=[ 70], 99.50th=[ 82], 99.90th=[ 108], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 7, max= 120, per=26.87%, avg=40.84, stdev=25.42, samples=294 iops : min= 1, max= 30, avg= 9.97, stdev= 6.37, samples=294 write: IOPS=9, BW=39.7KiB/s (40.7kB/s)(6236KiB/157005msec) clat (msec): min=21, max=465, avg=87.28, stdev=46.79 lat (msec): min=21, max=465, avg=87.28, stdev=46.79 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 43], 10.00th=[ 47], 20.00th=[ 54], | 30.00th=[ 62], 40.00th=[ 68], 50.00th=[ 74], 60.00th=[ 83], | 70.00th=[ 93], 80.00th=[ 110], 90.00th=[ 144], 95.00th=[ 186], | 99.00th=[ 262], 99.50th=[ 305], 99.90th=[ 334], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 7, max= 88, per=24.83%, avg=39.48, stdev=16.53, samples=313 iops : min= 1, max= 22, avg= 9.62, stdev= 4.20, samples=313 lat (msec) : 4=0.85%, 10=28.06%, 20=11.98%, 50=14.55%, 100=31.77% lat (msec) : 250=12.17%, 500=0.62% cpu : usr=0.04%, sys=2.28%, ctx=3536, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=152KiB/s (156kB/s), 37.8KiB/s-38.5KiB/s (38.7kB/s-39.5kB/s), io=23.4MiB (24.6MB), run=157005-157831msec WRITE: bw=159KiB/s (163kB/s), 39.7KiB/s-40.4KiB/s (40.6kB/s-41.3kB/s), io=24.6MiB (25.7MB), run=157005-157831msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2227: Thu May 1 03:13:56 2025 read: IOPS=51, BW=826KiB/s (845kB/s)(5920KiB/7171msec) clat (usec): min=76, max=37620, avg=9075.84, stdev=7695.25 lat (usec): min=76, max=37620, avg=9085.74, stdev=7686.74 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 96], 10.00th=[ 110], 20.00th=[ 125], | 30.00th=[ 4015], 40.00th=[ 7177], 50.00th=[ 8848], 60.00th=[10945], | 70.00th=[12911], 80.00th=[14484], 90.00th=[18482], 95.00th=[22938], | 99.00th=[33162], 99.50th=[37487], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 574, max= 1213, per=26.89%, avg=830.93, stdev=208.51, samples=14 iops : min= 35, max= 75, avg=51.57, stdev=12.96, samples=14 write: IOPS=55, BW=888KiB/s (909kB/s)(6368KiB/7171msec) clat (usec): min=723, max=158418, avg=9479.58, stdev=12963.60 lat (usec): min=724, max=158420, avg=9481.18, stdev=12963.75 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 1020], 10.00th=[ 1090], 20.00th=[ 1221], | 30.00th=[ 1663], 40.00th=[ 2507], 50.00th=[ 3949], 60.00th=[ 7111], | 70.00th=[ 11469], 80.00th=[ 15795], 90.00th=[ 26084], 95.00th=[ 32113], | 99.00th=[ 49546], 99.50th=[ 54789], 99.90th=[158335], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 640, max= 1184, per=27.60%, avg=897.29, stdev=179.66, samples=14 iops : min= 40, max= 74, avg=55.71, stdev=11.45, samples=14 lat (usec) : 100=2.86%, 250=9.37%, 500=0.65%, 750=0.39%, 1000=1.82% lat (msec) : 2=16.28%, 4=8.85%, 10=21.35%, 20=26.69%, 50=11.33% lat (msec) : 100=0.26%, 250=0.13% cpu : usr=0.18%, sys=14.48%, ctx=1166, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2229: Thu May 1 03:13:56 2025 read: IOPS=47, BW=761KiB/s (780kB/s)(5744KiB/7545msec) clat (usec): min=75, max=70642, avg=9112.33, stdev=9090.74 lat (usec): min=76, max=70643, avg=9113.28, stdev=9090.70 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 97], 10.00th=[ 106], 20.00th=[ 121], | 30.00th=[ 310], 40.00th=[ 6587], 50.00th=[ 8455], 60.00th=[ 9765], | 70.00th=[12256], 80.00th=[15139], 90.00th=[19530], 95.00th=[22938], | 99.00th=[42730], 99.50th=[51643], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 384, max= 1117, per=24.88%, avg=768.79, stdev=213.62, samples=14 iops : min= 24, max= 69, avg=47.64, stdev=13.25, samples=14 write: IOPS=54, BW=867KiB/s (888kB/s)(6544KiB/7545msec) clat (usec): min=894, max=115201, avg=10120.96, stdev=12572.63 lat (usec): min=895, max=115201, avg=10122.49, stdev=12572.59 clat percentiles (usec): | 1.00th=[ 988], 5.00th=[ 1074], 10.00th=[ 1123], 20.00th=[ 1401], | 30.00th=[ 2147], 40.00th=[ 3097], 50.00th=[ 5276], 60.00th=[ 8586], | 70.00th=[ 12780], 80.00th=[ 17695], 90.00th=[ 25035], 95.00th=[ 29754], | 99.00th=[ 51643], 99.50th=[ 69731], 99.90th=[114820], 99.95th=[114820], | 99.99th=[114820] bw ( KiB/s): min= 640, max= 1184, per=26.81%, avg=871.64, stdev=152.72, samples=14 iops : min= 40, max= 74, avg=54.07, stdev= 9.39, samples=14 lat (usec) : 100=2.99%, 250=10.94%, 500=0.39%, 750=0.26%, 1000=1.04% lat (msec) : 2=14.84%, 4=9.37%, 10=23.05%, 20=25.00%, 50=11.20% lat (msec) : 100=0.65%, 250=0.26% cpu : usr=0.13%, sys=14.33%, ctx=1185, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2231: Thu May 1 03:13:56 2025 read: IOPS=54, BW=865KiB/s (885kB/s)(6256KiB/7235msec) clat (usec): min=52, max=63723, avg=8979.54, stdev=10124.84 lat (usec): min=53, max=63725, avg=8980.95, stdev=10124.97 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 93], 10.00th=[ 100], 20.00th=[ 115], | 30.00th=[ 133], 40.00th=[ 4817], 50.00th=[ 7635], 60.00th=[ 9765], | 70.00th=[12387], 80.00th=[15008], 90.00th=[20841], 95.00th=[26870], | 99.00th=[54264], 99.50th=[60556], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 319, max= 1152, per=28.22%, avg=872.00, stdev=195.04, samples=14 iops : min= 19, max= 72, avg=54.07, stdev=12.36, samples=14 write: IOPS=52, BW=834KiB/s (854kB/s)(6032KiB/7235msec) clat (usec): min=848, max=95772, avg=9548.30, stdev=11990.48 lat (usec): min=849, max=95774, avg=9549.74, stdev=11990.53 clat percentiles (usec): | 1.00th=[ 906], 5.00th=[ 996], 10.00th=[ 1123], 20.00th=[ 1254], | 30.00th=[ 1614], 40.00th=[ 2573], 50.00th=[ 3982], 60.00th=[ 7308], | 70.00th=[11076], 80.00th=[17695], 90.00th=[25035], 95.00th=[33424], | 99.00th=[56361], 99.50th=[74974], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 351, max= 1152, per=25.48%, avg=828.50, stdev=235.62, samples=14 iops : min= 21, max= 72, avg=51.36, stdev=14.80, samples=14 lat (usec) : 100=4.69%, 250=13.15%, 500=0.91%, 750=0.39%, 1000=2.86% lat (msec) : 2=14.32%, 4=8.59%, 10=19.66%, 20=22.14%, 50=12.24% lat (msec) : 100=1.04% cpu : usr=0.21%, sys=13.88%, ctx=1066, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2232: Thu May 1 03:13:56 2025 read: IOPS=48, BW=778KiB/s (797kB/s)(6032KiB/7750msec) clat (usec): min=72, max=69705, avg=8982.58, stdev=9922.89 lat (usec): min=72, max=69707, avg=8984.66, stdev=9922.87 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 92], 10.00th=[ 99], 20.00th=[ 113], | 30.00th=[ 126], 40.00th=[ 2540], 50.00th=[ 7767], 60.00th=[10290], | 70.00th=[12911], 80.00th=[16188], 90.00th=[21103], 95.00th=[26608], | 99.00th=[45876], 99.50th=[54264], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 95, max= 1184, per=24.56%, avg=758.80, stdev=304.86, samples=15 iops : min= 5, max= 74, avg=47.20, stdev=19.20, samples=15 write: IOPS=50, BW=807KiB/s (827kB/s)(6256KiB/7750msec) clat (usec): min=774, max=167138, avg=10958.72, stdev=18343.88 lat (usec): min=775, max=167140, avg=10960.26, stdev=18343.88 clat percentiles (usec): | 1.00th=[ 938], 5.00th=[ 1012], 10.00th=[ 1123], 20.00th=[ 1319], | 30.00th=[ 1827], 40.00th=[ 2606], 50.00th=[ 4228], 60.00th=[ 7963], | 70.00th=[ 12387], 80.00th=[ 16581], 90.00th=[ 23725], 95.00th=[ 38536], | 99.00th=[ 98042], 99.50th=[158335], 99.90th=[166724], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 159, max= 1376, per=24.39%, avg=792.93, stdev=325.65, samples=15 iops : min= 9, max= 86, avg=49.33, stdev=20.51, samples=15 lat (usec) : 100=5.08%, 250=13.02%, 500=0.39%, 750=0.26%, 1000=1.95% lat (msec) : 2=15.10%, 4=9.51%, 10=16.93%, 20=25.26%, 50=10.81% lat (msec) : 100=1.30%, 250=0.39% cpu : usr=0.13%, sys=13.00%, ctx=1074, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3091KiB/s (3165kB/s), 761KiB/s-865KiB/s (780kB/s-885kB/s), io=23.4MiB (24.5MB), run=7171-7750msec WRITE: bw=3252KiB/s (3330kB/s), 807KiB/s-888KiB/s (827kB/s-909kB/s), io=24.6MiB (25.8MB), run=7171-7750msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2225: Thu May 1 03:14:25 2025 read: IOPS=10, BW=164KiB/s (168kB/s)(5920KiB/36084msec) clat (usec): min=3241, max=98359, avg=11316.06, stdev=9285.43 lat (usec): min=3243, max=98360, avg=11325.46, stdev=9284.37 clat percentiles (usec): | 1.00th=[ 3982], 5.00th=[ 4948], 10.00th=[ 5669], 20.00th=[ 6521], | 30.00th=[ 7242], 40.00th=[ 7898], 50.00th=[ 8586], 60.00th=[ 9372], | 70.00th=[10683], 80.00th=[13304], 90.00th=[18744], 95.00th=[27657], | 99.00th=[54264], 99.50th=[64226], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 31, max= 480, per=25.28%, avg=165.57, stdev=95.26, samples=69 iops : min= 1, max= 30, avg=10.04, stdev= 5.99, samples=69 write: IOPS=11, BW=176KiB/s (181kB/s)(6368KiB/36084msec) clat (msec): min=34, max=290, avg=79.86, stdev=27.94 lat (msec): min=34, max=290, avg=79.86, stdev=27.94 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 45], 10.00th=[ 50], 20.00th=[ 59], | 30.00th=[ 66], 40.00th=[ 71], 50.00th=[ 77], 60.00th=[ 82], | 70.00th=[ 89], 80.00th=[ 97], 90.00th=[ 111], 95.00th=[ 126], | 99.00th=[ 184], 99.50th=[ 224], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 63, max= 256, per=25.37%, avg=174.82, stdev=41.16, samples=71 iops : min= 3, max= 16, avg=10.62, stdev= 2.64, samples=71 lat (msec) : 4=0.52%, 10=31.38%, 20=12.24%, 50=8.85%, 100=37.76% lat (msec) : 250=9.11%, 500=0.13% cpu : usr=0.06%, sys=2.67%, ctx=893, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2226: Thu May 1 03:14:25 2025 read: IOPS=9, BW=157KiB/s (161kB/s)(5744KiB/36530msec) clat (msec): min=3, max=109, avg=12.44, stdev=11.92 lat (msec): min=3, max=109, avg=12.44, stdev=11.92 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 15], 90.00th=[ 23], 95.00th=[ 31], | 99.00th=[ 78], 99.50th=[ 83], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 32, max= 480, per=24.41%, avg=159.89, stdev=97.55, samples=70 iops : min= 2, max= 30, avg= 9.61, stdev= 6.01, samples=70 write: IOPS=11, BW=179KiB/s (183kB/s)(6544KiB/36530msec) clat (msec): min=22, max=227, avg=77.86, stdev=26.67 lat (msec): min=22, max=227, avg=77.86, stdev=26.67 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 43], 10.00th=[ 47], 20.00th=[ 56], | 30.00th=[ 63], 40.00th=[ 69], 50.00th=[ 75], 60.00th=[ 81], | 70.00th=[ 90], 80.00th=[ 97], 90.00th=[ 111], 95.00th=[ 124], | 99.00th=[ 155], 99.50th=[ 165], 99.90th=[ 228], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 64, max= 254, per=25.44%, avg=175.28, stdev=44.57, samples=72 iops : min= 4, max= 15, avg=10.60, stdev= 2.75, samples=72 lat (msec) : 4=0.26%, 10=28.65%, 20=12.63%, 50=10.55%, 100=38.67% lat (msec) : 250=9.24% cpu : usr=0.06%, sys=2.64%, ctx=875, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2228: Thu May 1 03:14:25 2025 read: IOPS=11, BW=183KiB/s (187kB/s)(6256KiB/34254msec) clat (usec): min=3018, max=70828, avg=11190.52, stdev=8262.69 lat (usec): min=3019, max=70829, avg=11196.66, stdev=8267.24 clat percentiles (usec): | 1.00th=[ 4015], 5.00th=[ 5211], 10.00th=[ 5932], 20.00th=[ 6915], | 30.00th=[ 7635], 40.00th=[ 8291], 50.00th=[ 9110], 60.00th=[ 9896], | 70.00th=[11076], 80.00th=[12387], 90.00th=[17433], 95.00th=[23462], | 99.00th=[52691], 99.50th=[63177], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 31, max= 448, per=28.02%, avg=183.54, stdev=102.00, samples=68 iops : min= 1, max= 28, avg=11.15, stdev= 6.47, samples=68 write: IOPS=11, BW=176KiB/s (180kB/s)(6032KiB/34254msec) clat (msec): min=36, max=194, avg=78.96, stdev=24.44 lat (msec): min=36, max=194, avg=78.97, stdev=24.44 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 46], 10.00th=[ 52], 20.00th=[ 58], | 30.00th=[ 66], 40.00th=[ 69], 50.00th=[ 75], 60.00th=[ 82], | 70.00th=[ 88], 80.00th=[ 97], 90.00th=[ 111], 95.00th=[ 124], | 99.00th=[ 163], 99.50th=[ 174], 99.90th=[ 194], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 95, max= 224, per=25.41%, avg=175.06, stdev=35.80, samples=68 iops : min= 5, max= 14, avg=10.62, stdev= 2.31, samples=68 lat (msec) : 4=0.39%, 10=30.47%, 20=16.41%, 50=7.03%, 100=37.24% lat (msec) : 250=8.46% cpu : usr=0.04%, sys=2.85%, ctx=886, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2230: Thu May 1 03:14:25 2025 read: IOPS=10, BW=170KiB/s (174kB/s)(6032KiB/35579msec) clat (msec): min=3, max=141, avg=11.98, stdev=11.24 lat (msec): min=3, max=141, avg=11.98, stdev=11.24 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 12], 80.00th=[ 14], 90.00th=[ 22], 95.00th=[ 27], | 99.00th=[ 64], 99.50th=[ 80], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 31, max= 352, per=25.60%, avg=167.65, stdev=84.48, samples=71 iops : min= 1, max= 22, avg=10.08, stdev= 5.45, samples=71 write: IOPS=10, BW=176KiB/s (180kB/s)(6256KiB/35579msec) clat (msec): min=34, max=181, avg=79.33, stdev=25.68 lat (msec): min=34, max=181, avg=79.33, stdev=25.68 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 47], 10.00th=[ 52], 20.00th=[ 58], | 30.00th=[ 65], 40.00th=[ 70], 50.00th=[ 74], 60.00th=[ 81], | 70.00th=[ 89], 80.00th=[ 99], 90.00th=[ 112], 95.00th=[ 127], | 99.00th=[ 178], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 64, max= 256, per=25.44%, avg=175.27, stdev=42.36, samples=71 iops : min= 4, max= 16, avg=10.56, stdev= 2.73, samples=71 lat (msec) : 4=0.13%, 10=28.52%, 20=15.23%, 50=8.72%, 100=38.54% lat (msec) : 250=8.85% cpu : usr=0.09%, sys=2.83%, ctx=888, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=656KiB/s (671kB/s), 157KiB/s-183KiB/s (161kB/s-187kB/s), io=23.4MiB (24.5MB), run=34254-36530msec WRITE: bw=690KiB/s (706kB/s), 176KiB/s-179KiB/s (180kB/s-183kB/s), io=24.6MiB (25.8MB), run=34254-36530msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2282: Thu May 1 03:14:28 2025 read: IOPS=5, BW=5973KiB/s (6117kB/s)(7168KiB/1200msec) clat (usec): min=442, max=74194, avg=37621.12, stdev=33998.18 lat (usec): min=443, max=74196, avg=37622.55, stdev=33998.15 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 445], 10.00th=[ 445], 20.00th=[ 553], | 30.00th=[ 4015], 40.00th=[ 4015], 50.00th=[58459], 60.00th=[61604], | 70.00th=[61604], 80.00th=[64226], 90.00th=[73925], 95.00th=[73925], | 99.00th=[73925], 99.50th=[73925], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 4096, max= 6144, per=38.90%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=4, BW=4267KiB/s (4369kB/s)(5120KiB/1200msec) clat (msec): min=58, max=269, avg=138.34, stdev=92.86 lat (msec): min=58, max=269, avg=138.39, stdev=92.85 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 59], | 30.00th=[ 63], 40.00th=[ 63], 50.00th=[ 102], 60.00th=[ 102], | 70.00th=[ 201], 80.00th=[ 201], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 2048, max= 4096, per=16.67%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 4=8.33%, 100=50.00%, 250=16.67%, 500=8.33% cpu : usr=0.00%, sys=17.01%, ctx=285, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2283: Thu May 1 03:14:28 2025 read: IOPS=1, BW=1316KiB/s (1348kB/s)(2048KiB/1556msec) clat (msec): min=69, max=144, avg=107.22, stdev=53.18 lat (msec): min=69, max=144, avg=107.22, stdev=53.18 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 70], 10.00th=[ 70], 20.00th=[ 70], | 30.00th=[ 70], 40.00th=[ 70], 50.00th=[ 70], 60.00th=[ 146], | 70.00th=[ 146], 80.00th=[ 146], 90.00th=[ 146], 95.00th=[ 146], | 99.00th=[ 146], 99.50th=[ 146], 99.90th=[ 146], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 2048, max= 2048, per=15.56%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=6, BW=6581KiB/s (6739kB/s)(10.0MiB/1556msec) clat (msec): min=42, max=262, avg=114.32, stdev=79.75 lat (msec): min=42, max=262, avg=114.39, stdev=79.75 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 60], 50.00th=[ 80], 60.00th=[ 101], | 70.00th=[ 165], 80.00th=[ 165], 90.00th=[ 226], 95.00th=[ 264], | 99.00th=[ 264], 99.50th=[ 264], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 6107, max=10240, per=44.36%, avg=8173.50, stdev=2922.47, samples=2 iops : min= 5, max= 10, avg= 7.50, stdev= 3.54, samples=2 lat (msec) : 50=25.00%, 100=33.33%, 250=33.33%, 500=8.33% cpu : usr=0.00%, sys=17.04%, ctx=123, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2284: Thu May 1 03:14:28 2025 read: IOPS=5, BW=5347KiB/s (5476kB/s)(6144KiB/1149msec) clat (usec): min=705, max=80387, avg=35347.38, stdev=38106.68 lat (usec): min=707, max=80389, avg=35348.63, stdev=38106.86 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 709], 10.00th=[ 709], 20.00th=[ 971], | 30.00th=[ 971], 40.00th=[ 1958], 50.00th=[ 1958], 60.00th=[57410], | 70.00th=[70779], 80.00th=[70779], 90.00th=[80217], 95.00th=[80217], | 99.00th=[80217], 99.50th=[80217], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 4087, max= 6144, per=38.87%, avg=5115.50, stdev=1454.52, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 write: IOPS=5, BW=5347KiB/s (5476kB/s)(6144KiB/1149msec) clat (msec): min=44, max=332, avg=148.26, stdev=102.57 lat (msec): min=44, max=332, avg=148.31, stdev=102.55 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 91], | 30.00th=[ 91], 40.00th=[ 106], 50.00th=[ 106], 60.00th=[ 124], | 70.00th=[ 194], 80.00th=[ 194], 90.00th=[ 334], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 4096, max= 6131, per=27.75%, avg=5113.50, stdev=1438.96, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 lat (usec) : 750=8.33%, 1000=8.33% lat (msec) : 2=8.33%, 50=8.33%, 100=33.33%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=16.38%, ctx=344, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2285: Thu May 1 03:14:28 2025 read: IOPS=3, BW=3815KiB/s (3907kB/s)(5120KiB/1342msec) clat (usec): min=427, max=109477, avg=31344.94, stdev=47907.33 lat (usec): min=428, max=109478, avg=31346.18, stdev=47907.59 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 429], 10.00th=[ 429], 20.00th=[ 429], | 30.00th=[ 433], 40.00th=[ 433], 50.00th=[ 486], 60.00th=[ 486], | 70.00th=[ 45876], 80.00th=[ 45876], 90.00th=[109577], 95.00th=[109577], | 99.00th=[109577], 99.50th=[109577], 99.90th=[109577], 99.95th=[109577], | 99.99th=[109577] bw ( KiB/s): min= 8192, max= 8192, per=62.24%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5341KiB/s (5469kB/s)(7168KiB/1342msec) clat (msec): min=44, max=381, avg=151.06, stdev=113.60 lat (msec): min=44, max=381, avg=151.14, stdev=113.59 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 70], | 30.00th=[ 86], 40.00th=[ 86], 50.00th=[ 140], 60.00th=[ 140], | 70.00th=[ 140], 80.00th=[ 197], 90.00th=[ 380], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 2048, max= 8192, per=27.79%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (usec) : 500=25.00% lat (msec) : 50=16.67%, 100=16.67%, 250=33.33%, 500=8.33% cpu : usr=0.00%, sys=13.93%, ctx=179, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.9MiB/s (13.5MB/s), 1316KiB/s-5973KiB/s (1348kB/s-6117kB/s), io=20.0MiB (20.0MB), run=1149-1556msec WRITE: bw=17.0MiB/s (18.9MB/s), 4267KiB/s-6581KiB/s (4369kB/s-6739kB/s), io=28.0MiB (29.4MB), run=1149-1556msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2278: Thu May 1 03:14:28 2025 read: IOPS=5, BW=5832KiB/s (5972kB/s)(7168KiB/1229msec) clat (msec): min=35, max=104, avg=61.57, stdev=26.02 lat (msec): min=35, max=104, avg=61.57, stdev=26.02 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 48], 60.00th=[ 77], | 70.00th=[ 77], 80.00th=[ 83], 90.00th=[ 105], 95.00th=[ 105], | 99.00th=[ 105], 99.50th=[ 105], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 2039, max= 8192, per=40.39%, avg=5115.50, stdev=4350.83, samples=2 iops : min= 1, max= 8, avg= 4.50, stdev= 4.95, samples=2 write: IOPS=4, BW=4166KiB/s (4266kB/s)(5120KiB/1229msec) clat (msec): min=96, max=214, avg=154.33, stdev=44.44 lat (msec): min=96, max=214, avg=154.39, stdev=44.45 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 97], 10.00th=[ 97], 20.00th=[ 97], | 30.00th=[ 134], 40.00th=[ 134], 50.00th=[ 148], 60.00th=[ 148], | 70.00th=[ 180], 80.00th=[ 180], 90.00th=[ 215], 95.00th=[ 215], | 99.00th=[ 215], 99.50th=[ 215], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 2048, max= 6119, per=23.03%, avg=4083.50, stdev=2878.63, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (msec) : 50=33.33%, 100=25.00%, 250=41.67% cpu : usr=0.00%, sys=7.90%, ctx=43, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2279: Thu May 1 03:14:28 2025 read: IOPS=1, BW=1328KiB/s (1360kB/s)(2048KiB/1542msec) clat (msec): min=44, max=183, avg=114.14, stdev=97.91 lat (msec): min=44, max=183, avg=114.15, stdev=97.91 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 45], | 30.00th=[ 45], 40.00th=[ 45], 50.00th=[ 45], 60.00th=[ 184], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 2043, max= 2048, per=16.15%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=6, BW=6641KiB/s (6800kB/s)(10.0MiB/1542msec) clat (msec): min=66, max=217, avg=115.93, stdev=58.88 lat (msec): min=66, max=217, avg=116.05, stdev=58.84 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 69], | 30.00th=[ 70], 40.00th=[ 71], 50.00th=[ 89], 60.00th=[ 96], | 70.00th=[ 165], 80.00th=[ 165], 90.00th=[ 209], 95.00th=[ 218], | 99.00th=[ 218], 99.50th=[ 218], 99.90th=[ 218], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 8192, max=10240, per=51.98%, avg=9216.00, stdev=1448.15, samples=2 iops : min= 8, max= 10, avg= 9.00, stdev= 1.41, samples=2 lat (msec) : 50=8.33%, 100=50.00%, 250=41.67% cpu : usr=0.00%, sys=7.33%, ctx=53, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2280: Thu May 1 03:14:28 2025 read: IOPS=3, BW=3814KiB/s (3905kB/s)(6144KiB/1611msec) clat (msec): min=17, max=226, avg=100.62, stdev=77.34 lat (msec): min=17, max=226, avg=100.62, stdev=77.34 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 85], 50.00th=[ 85], 60.00th=[ 107], | 70.00th=[ 140], 80.00th=[ 140], 90.00th=[ 228], 95.00th=[ 228], | 99.00th=[ 228], 99.50th=[ 228], 99.90th=[ 228], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 4096, max= 6144, per=40.43%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=3, BW=3814KiB/s (3905kB/s)(6144KiB/1611msec) clat (msec): min=84, max=251, avg=138.72, stdev=59.45 lat (msec): min=84, max=251, avg=138.79, stdev=59.45 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 85], 10.00th=[ 85], 20.00th=[ 97], | 30.00th=[ 97], 40.00th=[ 125], 50.00th=[ 125], 60.00th=[ 129], | 70.00th=[ 146], 80.00th=[ 146], 90.00th=[ 251], 95.00th=[ 251], | 99.00th=[ 251], 99.50th=[ 251], 99.90th=[ 251], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 2043, max= 6144, per=19.24%, avg=3411.67, stdev=2366.27, samples=3 iops : min= 1, max= 6, avg= 3.00, stdev= 2.65, samples=3 lat (msec) : 20=8.33%, 50=8.33%, 100=25.00%, 250=50.00%, 500=8.33% cpu : usr=0.00%, sys=6.71%, ctx=81, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2281: Thu May 1 03:14:28 2025 read: IOPS=3, BW=3166KiB/s (3242kB/s)(5120KiB/1617msec) clat (usec): min=22684, max=81035, avg=52019.24, stdev=21872.83 lat (usec): min=22686, max=81036, avg=52020.77, stdev=21872.60 clat percentiles (usec): | 1.00th=[22676], 5.00th=[22676], 10.00th=[22676], 20.00th=[22676], | 30.00th=[40633], 40.00th=[40633], 50.00th=[55837], 60.00th=[55837], | 70.00th=[60031], 80.00th=[60031], 90.00th=[81265], 95.00th=[81265], | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 8175, max= 8175, per=64.55%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4433KiB/s (4539kB/s)(7168KiB/1617msec) clat (msec): min=56, max=269, avg=163.66, stdev=74.18 lat (msec): min=56, max=269, avg=163.73, stdev=74.20 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 107], | 30.00th=[ 114], 40.00th=[ 114], 50.00th=[ 188], 60.00th=[ 188], | 70.00th=[ 188], 80.00th=[ 224], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 2048, max= 6131, per=23.06%, avg=4088.67, stdev=2041.50, samples=3 iops : min= 2, max= 5, avg= 3.33, stdev= 1.53, samples=3 lat (msec) : 50=16.67%, 100=33.33%, 250=41.67%, 500=8.33% cpu : usr=0.19%, sys=7.36%, ctx=91, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.4MiB/s (12.0MB/s), 1328KiB/s-5832KiB/s (1360kB/s-5972kB/s), io=20.0MiB (20.0MB), run=1229-1617msec WRITE: bw=17.3MiB/s (18.2MB/s), 3814KiB/s-6641KiB/s (3905kB/s-6800kB/s), io=28.0MiB (29.4MB), run=1229-1617msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2293: Thu May 1 03:14:31 2025 read: IOPS=3, BW=13.0MiB/s (13.6MB/s)(12.0MiB/923msec) clat (msec): min=230, max=250, avg=243.41, stdev=11.20 lat (msec): min=230, max=250, avg=243.41, stdev=11.20 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 230], 10.00th=[ 230], 20.00th=[ 230], | 30.00th=[ 230], 40.00th=[ 251], 50.00th=[ 251], 60.00th=[ 251], | 70.00th=[ 251], 80.00th=[ 251], 90.00th=[ 251], 95.00th=[ 251], | 99.00th=[ 251], 99.50th=[ 251], 99.90th=[ 251], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 8192, max= 8192, per=88.41%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=19.09%, ctx=1007, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2295: Thu May 1 03:14:31 2025 read: IOPS=0, BW=2317KiB/s (2372kB/s)(4096KiB/1768msec) clat (nsec): min=295347k, max=295347k, avg=295346633.00, stdev= 0.00 lat (nsec): min=295349k, max=295349k, avg=295349018.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 296], 10.00th=[ 296], 20.00th=[ 296], | 30.00th=[ 296], 40.00th=[ 296], 50.00th=[ 296], 60.00th=[ 296], | 70.00th=[ 296], 80.00th=[ 296], 90.00th=[ 296], 95.00th=[ 296], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 8192, max= 8192, per=88.41%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4633KiB/s (4745kB/s)(8192KiB/1768msec) clat (msec): min=231, max=1049, avg=640.45, stdev=577.83 lat (msec): min=232, max=1049, avg=640.72, stdev=577.85 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 232], 10.00th=[ 232], 20.00th=[ 232], | 30.00th=[ 232], 40.00th=[ 232], 50.00th=[ 232], 60.00th=[ 1053], | 70.00th=[ 1053], 80.00th=[ 1053], 90.00th=[ 1053], 95.00th=[ 1053], | 99.00th=[ 1053], 99.50th=[ 1053], 99.90th=[ 1053], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 6420, max= 6420, per=34.64%, avg=6420.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=11.77%, ctx=247, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2296: Thu May 1 03:14:31 2025 write: IOPS=1, BW=6966KiB/s (7133kB/s)(12.0MiB/1764msec) clat (msec): min=312, max=943, avg=562.20, stdev=335.59 lat (msec): min=312, max=943, avg=562.44, stdev=335.51 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 313], 10.00th=[ 313], 20.00th=[ 313], | 30.00th=[ 313], 40.00th=[ 430], 50.00th=[ 430], 60.00th=[ 430], | 70.00th=[ 944], 80.00th=[ 944], 90.00th=[ 944], 95.00th=[ 944], | 99.00th=[ 944], 99.50th=[ 944], 99.90th=[ 944], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min=12740, max=12740, per=68.74%, avg=12740.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 1000=33.33% cpu : usr=0.00%, sys=12.07%, ctx=64, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2299: Thu May 1 03:14:31 2025 write: IOPS=3, BW=12.7MiB/s (13.3MB/s)(12.0MiB/946msec) clat (msec): min=205, max=461, avg=301.02, stdev=140.09 lat (msec): min=206, max=462, avg=301.27, stdev=140.11 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 207], 10.00th=[ 207], 20.00th=[ 207], | 30.00th=[ 207], 40.00th=[ 236], 50.00th=[ 236], 60.00th=[ 236], | 70.00th=[ 464], 80.00th=[ 464], 90.00th=[ 464], 95.00th=[ 464], | 99.00th=[ 464], 99.50th=[ 464], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 8192, max= 8192, per=44.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=30.37%, ctx=93, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9267KiB/s (9489kB/s), 2317KiB/s-13.0MiB/s (2372kB/s-13.6MB/s), io=16.0MiB (16.8MB), run=923-1768msec WRITE: bw=18.1MiB/s (18.0MB/s), 4633KiB/s-12.7MiB/s (4745kB/s-13.3MB/s), io=32.0MiB (33.6MB), run=946-1768msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2292: Thu May 1 03:14:31 2025 read: IOPS=3, BW=12.5MiB/s (13.1MB/s)(12.0MiB/960msec) clat (msec): min=103, max=561, avg=305.64, stdev=233.37 lat (msec): min=103, max=561, avg=305.64, stdev=233.36 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 105], 10.00th=[ 105], 20.00th=[ 105], | 30.00th=[ 105], 40.00th=[ 253], 50.00th=[ 253], 60.00th=[ 253], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min=16318, max=16318, per=100.00%, avg=16318.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.21%, sys=6.57%, ctx=40, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2294: Thu May 1 03:14:31 2025 read: IOPS=0, BW=3828KiB/s (3920kB/s)(4096KiB/1070msec) clat (nsec): min=59224k, max=59224k, avg=59224336.00, stdev= 0.00 lat (nsec): min=59227k, max=59227k, avg=59226798.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[58983], 5.00th=[58983], 10.00th=[58983], 20.00th=[58983], | 30.00th=[58983], 40.00th=[58983], 50.00th=[58983], 60.00th=[58983], | 70.00th=[58983], 80.00th=[58983], 90.00th=[58983], 95.00th=[58983], | 99.00th=[58983], 99.50th=[58983], 99.90th=[58983], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 8192, max= 8192, per=53.50%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=7656KiB/s (7840kB/s)(8192KiB/1070msec) clat (msec): min=315, max=636, avg=475.72, stdev=227.23 lat (msec): min=315, max=636, avg=476.02, stdev=227.21 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 317], 10.00th=[ 317], 20.00th=[ 317], | 30.00th=[ 317], 40.00th=[ 317], 50.00th=[ 317], 60.00th=[ 634], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 8192, max= 8192, per=48.93%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.19%, sys=10.29%, ctx=49, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2297: Thu May 1 03:14:31 2025 write: IOPS=1, BW=6279KiB/s (6430kB/s)(12.0MiB/1957msec) clat (msec): min=175, max=1026, avg=624.07, stdev=427.49 lat (msec): min=175, max=1026, avg=624.36, stdev=427.51 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 176], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 667], 50.00th=[ 667], 60.00th=[ 667], | 70.00th=[ 1028], 80.00th=[ 1028], 90.00th=[ 1028], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min=16031, max=16031, per=95.75%, avg=16031.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.15%, sys=6.80%, ctx=61, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2298: Thu May 1 03:14:31 2025 write: IOPS=1, BW=6896KiB/s (7061kB/s)(12.0MiB/1782msec) clat (msec): min=213, max=1214, avg=556.93, stdev=569.51 lat (msec): min=213, max=1214, avg=557.19, stdev=569.53 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 213], 10.00th=[ 213], 20.00th=[ 213], | 30.00th=[ 213], 40.00th=[ 243], 50.00th=[ 243], 60.00th=[ 243], | 70.00th=[ 1217], 80.00th=[ 1217], 90.00th=[ 1217], 95.00th=[ 1217], | 99.00th=[ 1217], 99.50th=[ 1217], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 8159, max= 8159, per=48.73%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.22%, sys=5.00%, ctx=37, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=14.0MiB/s (15.7MB/s), 3828KiB/s-12.5MiB/s (3920kB/s-13.1MB/s), io=16.0MiB (16.8MB), run=960-1070msec WRITE: bw=16.4MiB/s (17.1MB/s), 6279KiB/s-7656KiB/s (6430kB/s-7840kB/s), io=32.0MiB (33.6MB), run=1070-1957msec