== sanity test 398b: DIO and buffer IO race ============== 21:27:39 (1746149259) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.47137 s, 34.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2551: Thu May 1 21:28:10 2025 read: IOPS=54, BW=217KiB/s (222kB/s)(5992KiB/27627msec) clat (usec): min=51, max=208226, avg=8180.47, stdev=8760.43 lat (usec): min=51, max=208230, avg=8182.02, stdev=8762.00 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 97], 10.00th=[ 111], 20.00th=[ 196], | 30.00th=[ 4490], 40.00th=[ 5604], 50.00th=[ 6783], 60.00th=[ 8291], | 70.00th=[ 10028], 80.00th=[ 13042], 90.00th=[ 17695], 95.00th=[ 21103], | 99.00th=[ 29230], 99.50th=[ 39060], 99.90th=[ 58459], 99.95th=[208667], | 99.99th=[208667] bw ( KiB/s): min= 63, max= 415, per=25.32%, avg=215.98, stdev=63.85, samples=55 iops : min= 15, max= 103, avg=53.80, stdev=15.99, samples=55 write: IOPS=56, BW=228KiB/s (233kB/s)(6296KiB/27627msec) clat (usec): min=464, max=100749, avg=9628.94, stdev=10596.83 lat (usec): min=465, max=100751, avg=9630.35, stdev=10596.88 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 963], | 30.00th=[ 1614], 40.00th=[ 3621], 50.00th=[ 7111], 60.00th=[ 9765], | 70.00th=[ 12649], 80.00th=[ 16319], 90.00th=[ 22676], 95.00th=[ 28443], | 99.00th=[ 46924], 99.50th=[ 61604], 99.90th=[ 86508], 99.95th=[101188], | 99.99th=[101188] bw ( KiB/s): min= 47, max= 407, per=25.46%, avg=227.35, stdev=75.32, samples=55 iops : min= 11, max= 101, avg=56.64, stdev=18.84, samples=55 lat (usec) : 100=3.26%, 250=6.77%, 500=0.62%, 750=3.55%, 1000=7.68% lat (msec) : 2=6.09%, 4=6.51%, 10=30.99%, 20=24.71%, 50=9.34% lat (msec) : 100=0.42%, 250=0.07% cpu : usr=0.23%, sys=15.41%, ctx=5084, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2553: Thu May 1 21:28:10 2025 read: IOPS=52, BW=211KiB/s (216kB/s)(5940KiB/28126msec) clat (usec): min=55, max=192219, avg=8122.98, stdev=9317.29 lat (usec): min=56, max=192220, avg=8124.13, stdev=9317.29 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 96], 10.00th=[ 110], 20.00th=[ 151], | 30.00th=[ 4490], 40.00th=[ 5604], 50.00th=[ 6521], 60.00th=[ 7898], | 70.00th=[ 9896], 80.00th=[ 12780], 90.00th=[ 16909], 95.00th=[ 21627], | 99.00th=[ 33162], 99.50th=[ 37487], 99.90th=[106431], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 64, max= 384, per=24.73%, avg=210.96, stdev=65.12, samples=56 iops : min= 16, max= 96, avg=52.41, stdev=16.32, samples=56 write: IOPS=56, BW=226KiB/s (231kB/s)(6348KiB/28126msec) clat (usec): min=437, max=239217, avg=10011.69, stdev=11626.06 lat (usec): min=438, max=239218, avg=10013.32, stdev=11626.16 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 725], 10.00th=[ 791], 20.00th=[ 971], | 30.00th=[ 1827], 40.00th=[ 4359], 50.00th=[ 7635], 60.00th=[ 10290], | 70.00th=[ 13042], 80.00th=[ 16450], 90.00th=[ 23200], 95.00th=[ 28443], | 99.00th=[ 49546], 99.50th=[ 55837], 99.90th=[ 79168], 99.95th=[240124], | 99.99th=[240124] bw ( KiB/s): min= 112, max= 440, per=25.16%, avg=224.68, stdev=63.85, samples=56 iops : min= 28, max= 110, avg=55.84, stdev=16.02, samples=56 lat (usec) : 100=3.19%, 250=7.52%, 500=0.65%, 750=3.87%, 1000=7.26% lat (msec) : 2=5.92%, 4=5.05%, 10=30.76%, 20=25.13%, 50=9.96% lat (msec) : 100=0.55%, 250=0.13% cpu : usr=0.24%, sys=15.10%, ctx=4966, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2554: Thu May 1 21:28:10 2025 read: IOPS=53, BW=215KiB/s (220kB/s)(6028KiB/28054msec) clat (usec): min=50, max=247747, avg=8165.72, stdev=9467.48 lat (usec): min=50, max=247753, avg=8167.78, stdev=9467.46 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 93], 10.00th=[ 105], 20.00th=[ 141], | 30.00th=[ 4228], 40.00th=[ 5604], 50.00th=[ 6718], 60.00th=[ 8225], | 70.00th=[ 10028], 80.00th=[ 13173], 90.00th=[ 17433], 95.00th=[ 21365], | 99.00th=[ 31589], 99.50th=[ 38536], 99.90th=[ 56886], 99.95th=[248513], | 99.99th=[248513] bw ( KiB/s): min= 96, max= 392, per=25.17%, avg=214.73, stdev=65.24, samples=56 iops : min= 24, max= 98, avg=53.52, stdev=16.40, samples=56 write: IOPS=55, BW=223KiB/s (228kB/s)(6260KiB/28054msec) clat (usec): min=460, max=133598, avg=9950.37, stdev=10962.91 lat (usec): min=461, max=133600, avg=9951.76, stdev=10963.04 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 742], 10.00th=[ 816], 20.00th=[ 1020], | 30.00th=[ 1844], 40.00th=[ 4080], 50.00th=[ 7439], 60.00th=[ 9765], | 70.00th=[ 12649], 80.00th=[ 16909], 90.00th=[ 23725], 95.00th=[ 28705], | 99.00th=[ 44303], 99.50th=[ 57410], 99.90th=[107480], 99.95th=[133694], | 99.99th=[133694] bw ( KiB/s): min= 71, max= 376, per=24.93%, avg=222.61, stdev=66.16, samples=56 iops : min= 17, max= 94, avg=55.48, stdev=16.59, samples=56 lat (usec) : 100=3.81%, 250=7.13%, 500=0.39%, 750=3.09%, 1000=7.23% lat (msec) : 2=6.28%, 4=6.18%, 10=31.15%, 20=24.06%, 50=10.22% lat (msec) : 100=0.33%, 250=0.13% cpu : usr=0.21%, sys=15.51%, ctx=5037, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2555: Thu May 1 21:28:10 2025 read: IOPS=53, BW=216KiB/s (221kB/s)(6052KiB/28033msec) clat (usec): min=50, max=115093, avg=7743.20, stdev=7610.61 lat (usec): min=51, max=115094, avg=7744.43, stdev=7610.43 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 93], 10.00th=[ 104], 20.00th=[ 135], | 30.00th=[ 4015], 40.00th=[ 5604], 50.00th=[ 6587], 60.00th=[ 7963], | 70.00th=[ 9765], 80.00th=[ 12518], 90.00th=[ 16450], 95.00th=[ 20317], | 99.00th=[ 32637], 99.50th=[ 38536], 99.90th=[ 59507], 99.95th=[114820], | 99.99th=[114820] bw ( KiB/s): min= 103, max= 416, per=25.71%, avg=219.33, stdev=63.29, samples=55 iops : min= 25, max= 104, avg=54.51, stdev=15.92, samples=55 write: IOPS=55, BW=222KiB/s (228kB/s)(6236KiB/28033msec) clat (usec): min=465, max=270354, avg=10340.46, stdev=13995.33 lat (usec): min=466, max=270357, avg=10342.19, stdev=13995.49 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 717], 10.00th=[ 791], 20.00th=[ 988], | 30.00th=[ 1631], 40.00th=[ 3851], 50.00th=[ 6980], 60.00th=[ 9765], | 70.00th=[ 12780], 80.00th=[ 16909], 90.00th=[ 24511], 95.00th=[ 31327], | 99.00th=[ 51643], 99.50th=[ 72877], 99.90th=[200279], 99.95th=[270533], | 99.99th=[270533] bw ( KiB/s): min= 16, max= 415, per=24.84%, avg=221.84, stdev=67.42, samples=56 iops : min= 4, max= 103, avg=55.14, stdev=16.86, samples=56 lat (usec) : 100=4.17%, 250=7.88%, 500=0.68%, 750=3.52%, 1000=7.16% lat (msec) : 2=6.58%, 4=5.27%, 10=30.70%, 20=23.80%, 50=9.64% lat (msec) : 100=0.49%, 250=0.10%, 500=0.03% cpu : usr=0.20%, sys=15.20%, ctx=4884, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=854KiB/s (874kB/s), 211KiB/s-217KiB/s (216kB/s-222kB/s), io=23.4MiB (24.6MB), run=27627-28126msec WRITE: bw=894KiB/s (915kB/s), 222KiB/s-228KiB/s (228kB/s-233kB/s), io=24.6MiB (25.7MB), run=27627-28126msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2548: Thu May 1 21:29:30 2025 read: IOPS=13, BW=55.9KiB/s (57.3kB/s)(5992KiB/107112msec) clat (msec): min=2, max=247, avg= 9.03, stdev= 9.30 lat (msec): min=2, max=247, avg= 9.03, stdev= 9.30 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 9], 80.00th=[ 12], 90.00th=[ 18], 95.00th=[ 24], | 99.00th=[ 38], 99.50th=[ 43], 99.90th=[ 69], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min= 8, max= 158, per=25.35%, avg=56.28, stdev=29.00, samples=212 iops : min= 2, max= 39, avg=13.89, stdev= 7.25, samples=212 write: IOPS=14, BW=58.8KiB/s (60.2kB/s)(6296KiB/107112msec) clat (msec): min=17, max=180, avg=59.34, stdev=21.76 lat (msec): min=17, max=180, avg=59.34, stdev=21.76 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 32], 10.00th=[ 35], 20.00th=[ 43], | 30.00th=[ 47], 40.00th=[ 52], 50.00th=[ 56], 60.00th=[ 61], | 70.00th=[ 68], 80.00th=[ 75], 90.00th=[ 87], 95.00th=[ 101], | 99.00th=[ 129], 99.50th=[ 153], 99.90th=[ 171], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 23, max= 112, per=25.22%, avg=58.76, stdev=16.03, samples=213 iops : min= 5, max= 28, avg=14.50, stdev= 4.05, samples=213 lat (msec) : 4=4.30%, 10=32.81%, 20=8.14%, 50=22.53%, 100=29.62% lat (msec) : 250=2.60% cpu : usr=0.07%, sys=3.16%, ctx=3464, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2549: Thu May 1 21:29:30 2025 read: IOPS=13, BW=55.2KiB/s (56.5kB/s)(5940KiB/107701msec) clat (msec): min=2, max=213, avg= 8.93, stdev= 8.81 lat (msec): min=2, max=213, avg= 8.93, stdev= 8.81 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 18], 95.00th=[ 23], | 99.00th=[ 41], 99.50th=[ 45], 99.90th=[ 79], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 7, max= 152, per=25.02%, avg=55.54, stdev=30.12, samples=212 iops : min= 1, max= 38, avg=13.67, stdev= 7.56, samples=212 write: IOPS=14, BW=58.9KiB/s (60.4kB/s)(6348KiB/107701msec) clat (msec): min=17, max=291, avg=59.39, stdev=22.26 lat (msec): min=17, max=291, avg=59.39, stdev=22.26 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 32], 10.00th=[ 35], 20.00th=[ 42], | 30.00th=[ 46], 40.00th=[ 52], 50.00th=[ 56], 60.00th=[ 62], | 70.00th=[ 68], 80.00th=[ 75], 90.00th=[ 88], 95.00th=[ 100], | 99.00th=[ 130], 99.50th=[ 144], 99.90th=[ 169], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 16, max= 119, per=25.10%, avg=58.49, stdev=16.37, samples=215 iops : min= 4, max= 29, avg=14.41, stdev= 4.12, samples=215 lat (msec) : 4=3.61%, 10=34.54%, 20=6.93%, 50=22.53%, 100=29.85% lat (msec) : 250=2.51%, 500=0.03% cpu : usr=0.11%, sys=3.05%, ctx=3468, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2550: Thu May 1 21:29:30 2025 read: IOPS=14, BW=56.8KiB/s (58.2kB/s)(6028KiB/106109msec) clat (usec): min=2469, max=78758, avg=9074.45, stdev=7293.22 lat (usec): min=2470, max=78759, avg=9076.15, stdev=7293.02 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 3687], 10.00th=[ 4113], 20.00th=[ 4752], | 30.00th=[ 5342], 40.00th=[ 5932], 50.00th=[ 6587], 60.00th=[ 7242], | 70.00th=[ 8717], 80.00th=[11469], 90.00th=[18220], 95.00th=[23462], | 99.00th=[39060], 99.50th=[44827], 99.90th=[63701], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 7, max= 168, per=25.58%, avg=56.79, stdev=30.10, samples=211 iops : min= 1, max= 42, avg=13.93, stdev= 7.55, samples=211 write: IOPS=14, BW=58.0KiB/s (60.4kB/s)(6260KiB/106109msec) clat (msec): min=16, max=342, avg=58.94, stdev=22.62 lat (msec): min=16, max=342, avg=58.95, stdev=22.62 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 32], 10.00th=[ 36], 20.00th=[ 41], | 30.00th=[ 47], 40.00th=[ 51], 50.00th=[ 55], 60.00th=[ 61], | 70.00th=[ 67], 80.00th=[ 74], 90.00th=[ 88], 95.00th=[ 100], | 99.00th=[ 129], 99.50th=[ 146], 99.90th=[ 209], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 8, max= 119, per=25.16%, avg=58.63, stdev=17.03, samples=212 iops : min= 2, max= 29, avg=14.40, stdev= 4.29, samples=212 lat (msec) : 4=3.74%, 10=33.43%, 20=8.37%, 50=22.98%, 100=29.13% lat (msec) : 250=2.31%, 500=0.03% cpu : usr=0.03%, sys=3.25%, ctx=3507, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2552: Thu May 1 21:29:30 2025 read: IOPS=14, BW=56.5KiB/s (57.8kB/s)(6052KiB/107138msec) clat (usec): min=2571, max=73201, avg=8904.72, stdev=7294.50 lat (usec): min=2573, max=73203, avg=8906.19, stdev=7294.43 clat percentiles (usec): | 1.00th=[ 3195], 5.00th=[ 3687], 10.00th=[ 4047], 20.00th=[ 4686], | 30.00th=[ 5342], 40.00th=[ 5800], 50.00th=[ 6521], 60.00th=[ 7308], | 70.00th=[ 8455], 80.00th=[10814], 90.00th=[17695], 95.00th=[22414], | 99.00th=[41157], 99.50th=[52691], 99.90th=[68682], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 7, max= 159, per=25.46%, avg=56.52, stdev=30.25, samples=213 iops : min= 1, max= 39, avg=13.92, stdev= 7.62, samples=213 write: IOPS=14, BW=58.2KiB/s (59.6kB/s)(6236KiB/107138msec) clat (msec): min=19, max=292, avg=59.96, stdev=23.42 lat (msec): min=19, max=292, avg=59.96, stdev=23.42 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 32], 10.00th=[ 35], 20.00th=[ 42], | 30.00th=[ 47], 40.00th=[ 52], 50.00th=[ 57], 60.00th=[ 62], | 70.00th=[ 69], 80.00th=[ 75], 90.00th=[ 87], 95.00th=[ 101], | 99.00th=[ 132], 99.50th=[ 155], 99.90th=[ 249], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 16, max= 104, per=24.85%, avg=57.90, stdev=15.85, samples=214 iops : min= 4, max= 26, avg=14.27, stdev= 4.00, samples=214 lat (msec) : 4=4.39%, 10=33.72%, 20=7.65%, 50=22.10%, 100=29.46% lat (msec) : 250=2.64%, 500=0.03% cpu : usr=0.10%, sys=3.11%, ctx=3507, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=223KiB/s (228kB/s), 55.2KiB/s-56.8KiB/s (56.5kB/s-58.2kB/s), io=23.4MiB (24.6MB), run=106109-107701msec WRITE: bw=233KiB/s (239kB/s), 58.2KiB/s-58.0KiB/s (59.6kB/s-60.4kB/s), io=24.6MiB (25.7MB), run=106109-107701msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2662: Thu May 1 21:29:37 2025 read: IOPS=63, BW=1024KiB/s (1048kB/s)(5920KiB/5782msec) clat (usec): min=52, max=37382, avg=6866.91, stdev=6930.01 lat (usec): min=52, max=37382, avg=6873.11, stdev=6925.62 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 90], 10.00th=[ 105], 20.00th=[ 119], | 30.00th=[ 143], 40.00th=[ 4424], 50.00th=[ 6325], 60.00th=[ 7635], | 70.00th=[ 9241], 80.00th=[11600], 90.00th=[15795], 95.00th=[20841], | 99.00th=[26608], 99.50th=[31065], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 544, max= 1408, per=25.30%, avg=1020.18, stdev=270.65, samples=11 iops : min= 34, max= 88, avg=63.45, stdev=16.89, samples=11 write: IOPS=68, BW=1101KiB/s (1128kB/s)(6368KiB/5782msec) clat (usec): min=656, max=58564, avg=8014.98, stdev=9454.31 lat (usec): min=657, max=58565, avg=8016.55, stdev=9454.31 clat percentiles (usec): | 1.00th=[ 881], 5.00th=[ 1012], 10.00th=[ 1106], 20.00th=[ 1221], | 30.00th=[ 1516], 40.00th=[ 2057], 50.00th=[ 2966], 60.00th=[ 6128], | 70.00th=[10290], 80.00th=[14353], 90.00th=[22414], 95.00th=[26870], | 99.00th=[39060], 99.50th=[56361], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 672, max= 1696, per=26.24%, avg=1113.09, stdev=320.37, samples=11 iops : min= 42, max= 106, avg=69.27, stdev=19.89, samples=11 lat (usec) : 100=3.91%, 250=11.72%, 500=1.04%, 750=0.65%, 1000=2.34% lat (msec) : 2=18.75%, 4=8.72%, 10=23.70%, 20=19.79%, 50=9.11% lat (msec) : 100=0.26% cpu : usr=0.21%, sys=17.92%, ctx=1094, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2663: Thu May 1 21:29:37 2025 read: IOPS=63, BW=1012KiB/s (1037kB/s)(5744KiB/5674msec) clat (usec): min=58, max=51095, avg=6405.63, stdev=6992.92 lat (usec): min=59, max=51096, avg=6406.62, stdev=6992.83 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 92], 10.00th=[ 100], 20.00th=[ 113], | 30.00th=[ 130], 40.00th=[ 619], 50.00th=[ 5473], 60.00th=[ 7701], | 70.00th=[ 9503], 80.00th=[11338], 90.00th=[15533], 95.00th=[19268], | 99.00th=[26346], 99.50th=[30540], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 544, max= 1312, per=25.39%, avg=1023.73, stdev=239.39, samples=11 iops : min= 34, max= 82, avg=63.91, stdev=14.94, samples=11 write: IOPS=72, BW=1153KiB/s (1181kB/s)(6544KiB/5674msec) clat (usec): min=678, max=59054, avg=8194.78, stdev=9322.07 lat (usec): min=678, max=59057, avg=8198.20, stdev=9322.82 clat percentiles (usec): | 1.00th=[ 848], 5.00th=[ 996], 10.00th=[ 1074], 20.00th=[ 1221], | 30.00th=[ 1647], 40.00th=[ 2212], 50.00th=[ 3752], 60.00th=[ 7373], | 70.00th=[10552], 80.00th=[14222], 90.00th=[21890], 95.00th=[27657], | 99.00th=[37487], 99.50th=[44827], 99.90th=[58983], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 832, max= 1564, per=27.49%, avg=1166.18, stdev=204.80, samples=11 iops : min= 52, max= 97, avg=72.82, stdev=12.66, samples=11 lat (usec) : 100=4.43%, 250=13.15%, 500=0.78%, 750=1.04%, 1000=2.73% lat (msec) : 2=17.06%, 4=8.72%, 10=22.79%, 20=21.22%, 50=7.68% lat (msec) : 100=0.39% cpu : usr=0.32%, sys=17.66%, ctx=1085, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2664: Thu May 1 21:29:37 2025 read: IOPS=69, BW=1110KiB/s (1137kB/s)(6256KiB/5635msec) clat (usec): min=63, max=58094, avg=6428.29, stdev=7326.99 lat (usec): min=64, max=58094, avg=6429.11, stdev=7326.98 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 93], 10.00th=[ 99], 20.00th=[ 113], | 30.00th=[ 126], 40.00th=[ 416], 50.00th=[ 5735], 60.00th=[ 7570], | 70.00th=[ 9503], 80.00th=[11731], 90.00th=[15270], 95.00th=[18482], | 99.00th=[27657], 99.50th=[47973], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 768, max= 1632, per=28.13%, avg=1134.09, stdev=288.98, samples=11 iops : min= 48, max= 102, avg=70.73, stdev=18.13, samples=11 write: IOPS=66, BW=1070KiB/s (1096kB/s)(6032KiB/5635msec) clat (usec): min=635, max=53987, avg=7941.69, stdev=9661.68 lat (usec): min=636, max=53989, avg=7943.19, stdev=9661.67 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 947], 10.00th=[ 1012], 20.00th=[ 1172], | 30.00th=[ 1401], 40.00th=[ 1958], 50.00th=[ 3097], 60.00th=[ 5800], | 70.00th=[ 9372], 80.00th=[14615], 90.00th=[21627], 95.00th=[26870], | 99.00th=[43254], 99.50th=[52691], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 798, max= 1728, per=25.37%, avg=1076.00, stdev=284.10, samples=11 iops : min= 49, max= 108, avg=67.09, stdev=17.89, samples=11 lat (usec) : 100=6.12%, 250=13.80%, 500=0.78%, 750=1.04%, 1000=4.17% lat (msec) : 2=15.89%, 4=7.42%, 10=22.66%, 20=20.18%, 50=7.55% lat (msec) : 100=0.39% cpu : usr=0.16%, sys=17.13%, ctx=1014, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2665: Thu May 1 21:29:37 2025 read: IOPS=63, BW=1015KiB/s (1040kB/s)(6032KiB/5940msec) clat (usec): min=75, max=36486, avg=6580.36, stdev=7327.73 lat (usec): min=76, max=36487, avg=6581.25, stdev=7327.74 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 91], 10.00th=[ 98], 20.00th=[ 112], | 30.00th=[ 127], 40.00th=[ 269], 50.00th=[ 5276], 60.00th=[ 7701], | 70.00th=[ 9765], 80.00th=[12780], 90.00th=[16581], 95.00th=[20579], | 99.00th=[28443], 99.50th=[32900], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 224, max= 1472, per=24.96%, avg=1006.55, stdev=338.16, samples=11 iops : min= 14, max= 92, avg=62.91, stdev=21.14, samples=11 write: IOPS=65, BW=1053KiB/s (1078kB/s)(6256KiB/5940msec) clat (usec): min=823, max=123569, avg=8636.93, stdev=13010.75 lat (usec): min=824, max=123570, avg=8648.43, stdev=13009.44 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 979], 10.00th=[ 1057], 20.00th=[ 1205], | 30.00th=[ 1532], 40.00th=[ 2311], 50.00th=[ 3359], 60.00th=[ 6390], | 70.00th=[ 10290], 80.00th=[ 14615], 90.00th=[ 20055], 95.00th=[ 25822], | 99.00th=[ 87557], 99.50th=[103285], 99.90th=[123208], 99.95th=[123208], | 99.99th=[123208] bw ( KiB/s): min= 256, max= 1440, per=24.89%, avg=1056.00, stdev=339.86, samples=11 iops : min= 16, max= 90, avg=66.00, stdev=21.24, samples=11 lat (usec) : 100=5.86%, 250=13.54%, 500=1.04%, 750=0.65%, 1000=3.78% lat (msec) : 2=15.89%, 4=8.85%, 10=20.05%, 20=22.14%, 50=7.42% lat (msec) : 100=0.52%, 250=0.26% cpu : usr=0.08%, sys=16.91%, ctx=1049, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4032KiB/s (4129kB/s), 1012KiB/s-1110KiB/s (1037kB/s-1137kB/s), io=23.4MiB (24.5MB), run=5635-5940msec WRITE: bw=4242KiB/s (4344kB/s), 1053KiB/s-1153KiB/s (1078kB/s-1181kB/s), io=24.6MiB (25.8MB), run=5635-5940msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2666: Thu May 1 21:30:03 2025 read: IOPS=11, BW=189KiB/s (194kB/s)(5920KiB/31275msec) clat (usec): min=2406, max=76644, avg=9771.52, stdev=8024.91 lat (usec): min=2407, max=76646, avg=9773.09, stdev=8024.80 clat percentiles (usec): | 1.00th=[ 3752], 5.00th=[ 4080], 10.00th=[ 4686], 20.00th=[ 5407], | 30.00th=[ 5997], 40.00th=[ 6521], 50.00th=[ 7373], 60.00th=[ 8455], | 70.00th=[ 9896], 80.00th=[11338], 90.00th=[17171], 95.00th=[23462], | 99.00th=[49546], 99.50th=[63701], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 31, max= 640, per=25.12%, avg=191.13, stdev=113.59, samples=61 iops : min= 1, max= 40, avg=11.67, stdev= 7.06, samples=61 write: IOPS=12, BW=204KiB/s (208kB/s)(6368KiB/31275msec) clat (msec): min=26, max=151, avg=68.99, stdev=22.55 lat (msec): min=26, max=151, avg=68.99, stdev=22.55 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 36], 10.00th=[ 42], 20.00th=[ 51], | 30.00th=[ 56], 40.00th=[ 62], 50.00th=[ 67], 60.00th=[ 73], | 70.00th=[ 80], 80.00th=[ 87], 90.00th=[ 99], 95.00th=[ 111], | 99.00th=[ 138], 99.50th=[ 144], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 95, max= 351, per=25.38%, avg=203.02, stdev=52.52, samples=62 iops : min= 5, max= 21, avg=12.40, stdev= 3.32, samples=62 lat (msec) : 4=1.82%, 10=32.42%, 20=11.07%, 50=12.37%, 100=37.37% lat (msec) : 250=4.95% cpu : usr=0.07%, sys=2.83%, ctx=843, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2667: Thu May 1 21:30:03 2025 read: IOPS=11, BW=183KiB/s (187kB/s)(5744KiB/31469msec) clat (usec): min=2644, max=52460, avg=9512.57, stdev=6626.37 lat (usec): min=2646, max=52462, avg=9514.17, stdev=6626.20 clat percentiles (usec): | 1.00th=[ 3294], 5.00th=[ 4113], 10.00th=[ 4555], 20.00th=[ 5342], | 30.00th=[ 6128], 40.00th=[ 6849], 50.00th=[ 7504], 60.00th=[ 8586], | 70.00th=[10028], 80.00th=[11863], 90.00th=[16188], 95.00th=[21365], | 99.00th=[41681], 99.50th=[45876], 99.90th=[52691], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 31, max= 447, per=24.19%, avg=184.10, stdev=98.49, samples=61 iops : min= 1, max= 27, avg=11.05, stdev= 6.20, samples=61 write: IOPS=12, BW=208KiB/s (213kB/s)(6544KiB/31469msec) clat (msec): min=24, max=163, avg=68.20, stdev=24.53 lat (msec): min=24, max=163, avg=68.21, stdev=24.53 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 35], 10.00th=[ 39], 20.00th=[ 46], | 30.00th=[ 54], 40.00th=[ 60], 50.00th=[ 66], 60.00th=[ 71], | 70.00th=[ 78], 80.00th=[ 87], 90.00th=[ 105], 95.00th=[ 116], | 99.00th=[ 133], 99.50th=[ 142], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 96, max= 384, per=25.67%, avg=205.39, stdev=59.00, samples=62 iops : min= 6, max= 24, avg=12.39, stdev= 3.78, samples=62 lat (msec) : 4=1.95%, 10=30.73%, 20=11.20%, 50=16.28%, 100=33.59% lat (msec) : 250=6.25% cpu : usr=0.00%, sys=2.93%, ctx=866, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2668: Thu May 1 21:30:03 2025 read: IOPS=13, BW=211KiB/s (216kB/s)(6256KiB/29609msec) clat (usec): min=3082, max=68235, avg=9297.42, stdev=7052.87 lat (usec): min=3082, max=68237, avg=9299.17, stdev=7052.86 clat percentiles (usec): | 1.00th=[ 3458], 5.00th=[ 4047], 10.00th=[ 4490], 20.00th=[ 5276], | 30.00th=[ 5932], 40.00th=[ 6718], 50.00th=[ 7373], 60.00th=[ 8225], | 70.00th=[ 9241], 80.00th=[10683], 90.00th=[15795], 95.00th=[21890], | 99.00th=[44827], 99.50th=[49546], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 31, max= 608, per=27.82%, avg=211.69, stdev=126.42, samples=59 iops : min= 1, max= 38, avg=12.92, stdev= 7.92, samples=59 write: IOPS=12, BW=204KiB/s (209kB/s)(6032KiB/29609msec) clat (msec): min=27, max=139, avg=68.38, stdev=22.11 lat (msec): min=27, max=139, avg=68.38, stdev=22.11 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 39], 10.00th=[ 44], 20.00th=[ 49], | 30.00th=[ 54], 40.00th=[ 60], 50.00th=[ 67], 60.00th=[ 73], | 70.00th=[ 80], 80.00th=[ 86], 90.00th=[ 102], 95.00th=[ 109], | 99.00th=[ 127], 99.50th=[ 136], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 95, max= 320, per=25.31%, avg=202.47, stdev=48.20, samples=59 iops : min= 5, max= 20, avg=12.34, stdev= 3.11, samples=59 lat (msec) : 4=2.34%, 10=36.07%, 20=9.11%, 50=14.97%, 100=32.29% lat (msec) : 250=5.21% cpu : usr=0.05%, sys=2.96%, ctx=857, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2669: Thu May 1 21:30:03 2025 read: IOPS=12, BW=195KiB/s (199kB/s)(6032KiB/31002msec) clat (usec): min=3378, max=72237, avg=10049.38, stdev=8455.13 lat (usec): min=3380, max=72239, avg=10050.91, stdev=8455.09 clat percentiles (usec): | 1.00th=[ 3458], 5.00th=[ 4178], 10.00th=[ 4817], 20.00th=[ 5538], | 30.00th=[ 6259], 40.00th=[ 6783], 50.00th=[ 7701], 60.00th=[ 8586], | 70.00th=[ 9503], 80.00th=[11863], 90.00th=[16909], 95.00th=[23987], | 99.00th=[52691], 99.50th=[62653], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 31, max= 415, per=25.44%, avg=193.63, stdev=101.84, samples=62 iops : min= 1, max= 25, avg=11.74, stdev= 6.39, samples=62 write: IOPS=12, BW=202KiB/s (207kB/s)(6256KiB/31002msec) clat (msec): min=23, max=200, avg=69.52, stdev=24.45 lat (msec): min=23, max=200, avg=69.52, stdev=24.45 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 37], 10.00th=[ 41], 20.00th=[ 50], | 30.00th=[ 55], 40.00th=[ 61], 50.00th=[ 67], 60.00th=[ 73], | 70.00th=[ 80], 80.00th=[ 88], 90.00th=[ 104], 95.00th=[ 114], | 99.00th=[ 140], 99.50th=[ 161], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 95, max= 320, per=25.17%, avg=201.37, stdev=55.23, samples=62 iops : min= 5, max= 20, avg=12.23, stdev= 3.54, samples=62 lat (msec) : 4=1.82%, 10=33.98%, 20=9.37%, 50=14.45%, 100=34.77% lat (msec) : 250=5.60% cpu : usr=0.07%, sys=2.82%, ctx=861, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=761KiB/s (779kB/s), 183KiB/s-211KiB/s (187kB/s-216kB/s), io=23.4MiB (24.5MB), run=29609-31469msec WRITE: bw=801KiB/s (820kB/s), 202KiB/s-208KiB/s (207kB/s-213kB/s), io=24.6MiB (25.8MB), run=29609-31469msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2700: Thu May 1 21:30:07 2025 read: IOPS=2, BW=2339KiB/s (2395kB/s)(7168KiB/3065msec) clat (msec): min=52, max=1057, avg=228.29, stdev=370.08 lat (msec): min=52, max=1057, avg=228.29, stdev=370.08 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 53], | 30.00th=[ 59], 40.00th=[ 59], 50.00th=[ 73], 60.00th=[ 89], | 70.00th=[ 89], 80.00th=[ 215], 90.00th=[ 1062], 95.00th=[ 1062], | 99.00th=[ 1062], 99.50th=[ 1062], 99.90th=[ 1062], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 4096, max= 6144, per=76.98%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=1, BW=1670KiB/s (1711kB/s)(5120KiB/3065msec) clat (msec): min=34, max=1143, avg=275.07, stdev=486.14 lat (msec): min=34, max=1143, avg=275.15, stdev=486.13 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 35], | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 55], 60.00th=[ 55], | 70.00th=[ 104], 80.00th=[ 104], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 2043, max= 6144, per=43.96%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (msec) : 50=16.67%, 100=50.00%, 250=16.67% cpu : usr=0.00%, sys=9.60%, ctx=799, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2701: Thu May 1 21:30:07 2025 read: IOPS=0, BW=803KiB/s (822kB/s)(2048KiB/2551msec) clat (msec): min=57, max=1317, avg=687.85, stdev=891.09 lat (msec): min=57, max=1317, avg=687.86, stdev=891.09 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 58], 10.00th=[ 58], 20.00th=[ 58], | 30.00th=[ 58], 40.00th=[ 58], 50.00th=[ 58], 60.00th=[ 1318], | 70.00th=[ 1318], 80.00th=[ 1318], 90.00th=[ 1318], 95.00th=[ 1318], | 99.00th=[ 1318], 99.50th=[ 1318], 99.90th=[ 1318], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 2043, max= 2043, per=30.72%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=3, BW=4014KiB/s (4110kB/s)(10.0MiB/2551msec) clat (msec): min=63, max=257, avg=115.97, stdev=68.29 lat (msec): min=63, max=257, avg=116.04, stdev=68.30 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 66], | 30.00th=[ 71], 40.00th=[ 79], 50.00th=[ 82], 60.00th=[ 100], | 70.00th=[ 114], 80.00th=[ 114], 90.00th=[ 224], 95.00th=[ 257], | 99.00th=[ 257], 99.50th=[ 257], 99.90th=[ 257], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 2043, max= 4096, per=32.96%, avg=3069.50, stdev=1451.69, samples=2 iops : min= 1, max= 4, avg= 2.50, stdev= 2.12, samples=2 lat (msec) : 100=58.33%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=15.10%, ctx=363, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2702: Thu May 1 21:30:07 2025 read: IOPS=1, BW=1995KiB/s (2043kB/s)(6144KiB/3079msec) clat (msec): min=8, max=189, avg=96.44, stdev=65.04 lat (msec): min=8, max=189, avg=96.44, stdev=65.04 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 9], 10.00th=[ 9], 20.00th=[ 69], | 30.00th=[ 69], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 82], | 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 190], 95.00th=[ 190], | 99.00th=[ 190], 99.50th=[ 190], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 1831, max= 8126, per=74.85%, avg=4978.50, stdev=4451.24, samples=2 iops : min= 1, max= 7, avg= 4.00, stdev= 4.24, samples=2 write: IOPS=1, BW=1995KiB/s (2043kB/s)(6144KiB/3079msec) clat (msec): min=29, max=1331, avg=409.89, stdev=543.34 lat (msec): min=29, max=1331, avg=409.97, stdev=543.35 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 57], | 30.00th=[ 57], 40.00th=[ 65], 50.00th=[ 65], 60.00th=[ 153], | 70.00th=[ 827], 80.00th=[ 827], 90.00th=[ 1334], 95.00th=[ 1334], | 99.00th=[ 1334], 99.50th=[ 1334], 99.90th=[ 1334], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 1831, max= 4087, per=31.71%, avg=2953.00, stdev=1295.61, samples=4 iops : min= 1, max= 3, avg= 2.00, stdev= 1.15, samples=4 lat (msec) : 10=8.33%, 50=8.33%, 100=41.67%, 250=25.00%, 1000=8.33% cpu : usr=0.00%, sys=7.54%, ctx=418, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2703: Thu May 1 21:30:07 2025 read: IOPS=2, BW=2201KiB/s (2254kB/s)(5120KiB/2326msec) clat (usec): min=588, max=156820, avg=55164.49, stdev=63974.09 lat (usec): min=590, max=156821, avg=55165.84, stdev=63974.02 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 586], 10.00th=[ 586], 20.00th=[ 586], | 30.00th=[ 709], 40.00th=[ 709], 50.00th=[ 53216], 60.00th=[ 53216], | 70.00th=[ 64750], 80.00th=[ 64750], 90.00th=[156238], 95.00th=[156238], | 99.00th=[156238], 99.50th=[156238], 99.90th=[156238], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 8110, max= 8110, per=100.00%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3082KiB/s (3156kB/s)(7168KiB/2326msec) clat (msec): min=25, max=1275, avg=276.77, stdev=453.17 lat (msec): min=25, max=1275, avg=276.85, stdev=453.18 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 28], | 30.00th=[ 42], 40.00th=[ 42], 50.00th=[ 55], 60.00th=[ 209], | 70.00th=[ 209], 80.00th=[ 305], 90.00th=[ 1284], 95.00th=[ 1284], | 99.00th=[ 1284], 99.50th=[ 1284], 99.90th=[ 1284], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 4096, max= 8110, per=65.54%, avg=6103.00, stdev=2838.33, samples=2 iops : min= 4, max= 7, avg= 5.50, stdev= 2.12, samples=2 lat (usec) : 750=16.67% lat (msec) : 50=25.00%, 100=25.00%, 250=16.67%, 500=8.33% cpu : usr=0.00%, sys=9.03%, ctx=272, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6652KiB/s (6811kB/s), 803KiB/s-2339KiB/s (822kB/s-2395kB/s), io=20.0MiB (20.0MB), run=2326-3079msec WRITE: bw=9312KiB/s (9536kB/s), 1670KiB/s-4014KiB/s (1711kB/s-4110kB/s), io=28.0MiB (29.4MB), run=2326-3079msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2704: Thu May 1 21:30:07 2025 read: IOPS=2, BW=2249KiB/s (2303kB/s)(7168KiB/3187msec) clat (msec): min=19, max=814, avg=165.52, stdev=287.38 lat (msec): min=19, max=814, avg=165.53, stdev=287.38 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 46], | 30.00th=[ 52], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 66], | 70.00th=[ 66], 80.00th=[ 108], 90.00th=[ 818], 95.00th=[ 818], | 99.00th=[ 818], 99.50th=[ 818], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 1831, max= 6131, per=62.55%, avg=4019.33, stdev=2151.02, samples=3 iops : min= 1, max= 5, avg= 3.33, stdev= 2.08, samples=3 write: IOPS=1, BW=1607KiB/s (1645kB/s)(5120KiB/3187msec) clat (msec): min=96, max=1315, avg=389.11, stdev=520.46 lat (msec): min=96, max=1315, avg=389.17, stdev=520.46 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 96], 10.00th=[ 96], 20.00th=[ 96], | 30.00th=[ 122], 40.00th=[ 122], 50.00th=[ 182], 60.00th=[ 182], | 70.00th=[ 232], 80.00th=[ 232], 90.00th=[ 1318], 95.00th=[ 1318], | 99.00th=[ 1318], 99.50th=[ 1318], 99.90th=[ 1318], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 2043, max= 6144, per=45.50%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (msec) : 20=8.33%, 50=8.33%, 100=33.33%, 250=33.33%, 1000=8.33% cpu : usr=0.03%, sys=2.45%, ctx=62, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2705: Thu May 1 21:30:07 2025 read: IOPS=0, BW=756KiB/s (774kB/s)(2048KiB/2709msec) clat (msec): min=20, max=1323, avg=672.16, stdev=921.17 lat (msec): min=20, max=1323, avg=672.17, stdev=921.17 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 21], 60.00th=[ 1318], | 70.00th=[ 1318], 80.00th=[ 1318], 90.00th=[ 1318], 95.00th=[ 1318], | 99.00th=[ 1318], 99.50th=[ 1318], 99.90th=[ 1318], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 2043, max= 2048, per=31.83%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3780KiB/s (3871kB/s)(10.0MiB/2709msec) clat (msec): min=64, max=318, avg=133.67, stdev=83.20 lat (msec): min=64, max=318, avg=133.81, stdev=83.17 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 65], 10.00th=[ 65], 20.00th=[ 67], | 30.00th=[ 72], 40.00th=[ 95], 50.00th=[ 105], 60.00th=[ 106], | 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 241], 95.00th=[ 317], | 99.00th=[ 317], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 4079, max=10240, per=79.59%, avg=7159.50, stdev=4356.48, samples=2 iops : min= 3, max= 10, avg= 6.50, stdev= 4.95, samples=2 lat (msec) : 50=8.33%, 100=33.33%, 250=41.67%, 500=8.33% cpu : usr=0.07%, sys=2.62%, ctx=45, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2706: Thu May 1 21:30:07 2025 read: IOPS=1, BW=1948KiB/s (1995kB/s)(6144KiB/3154msec) clat (usec): min=21472, max=71654, avg=37434.86, stdev=20677.49 lat (usec): min=21473, max=71655, avg=37436.43, stdev=20677.44 clat percentiles (usec): | 1.00th=[21365], 5.00th=[21365], 10.00th=[21365], 20.00th=[21627], | 30.00th=[21627], 40.00th=[22152], 50.00th=[22152], 60.00th=[35390], | 70.00th=[52691], 80.00th=[52691], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 4071, max= 6144, per=79.48%, avg=5107.50, stdev=1465.83, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 write: IOPS=1, BW=1948KiB/s (1995kB/s)(6144KiB/3154msec) clat (msec): min=78, max=1451, avg=466.03, stdev=544.07 lat (msec): min=78, max=1451, avg=466.10, stdev=544.08 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 79], 10.00th=[ 79], 20.00th=[ 106], | 30.00th=[ 106], 40.00th=[ 197], 50.00th=[ 197], 60.00th=[ 203], | 70.00th=[ 760], 80.00th=[ 760], 90.00th=[ 1452], 95.00th=[ 1452], | 99.00th=[ 1452], 99.50th=[ 1452], 99.90th=[ 1452], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 4096, max= 6107, per=56.71%, avg=5101.50, stdev=1421.99, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 lat (msec) : 50=33.33%, 100=25.00%, 250=25.00%, 1000=8.33% cpu : usr=0.00%, sys=1.81%, ctx=47, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2707: Thu May 1 21:30:07 2025 read: IOPS=1, BW=2030KiB/s (2079kB/s)(5120KiB/2522msec) clat (msec): min=21, max=105, avg=56.33, stdev=36.10 lat (msec): min=21, max=105, avg=56.33, stdev=36.10 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 53], 60.00th=[ 53], | 70.00th=[ 79], 80.00th=[ 79], 90.00th=[ 106], 95.00th=[ 106], | 99.00th=[ 106], 99.50th=[ 106], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 2048, max= 6131, per=63.64%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=2, BW=2842KiB/s (2910kB/s)(7168KiB/2522msec) clat (msec): min=87, max=1449, avg=302.83, stdev=506.11 lat (msec): min=87, max=1449, avg=302.91, stdev=506.12 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 88], 10.00th=[ 88], 20.00th=[ 89], | 30.00th=[ 90], 40.00th=[ 90], 50.00th=[ 121], 60.00th=[ 138], | 70.00th=[ 138], 80.00th=[ 148], 90.00th=[ 1452], 95.00th=[ 1452], | 99.00th=[ 1452], 99.50th=[ 1452], 99.90th=[ 1452], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 6131, max= 8192, per=79.61%, avg=7161.50, stdev=1457.35, samples=2 iops : min= 5, max= 8, avg= 6.50, stdev= 2.12, samples=2 lat (msec) : 50=16.67%, 100=41.67%, 250=33.33% cpu : usr=0.04%, sys=3.05%, ctx=42, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6426KiB/s (6580kB/s), 756KiB/s-2249KiB/s (774kB/s-2303kB/s), io=20.0MiB (20.0MB), run=2522-3187msec WRITE: bw=8997KiB/s (9212kB/s), 1607KiB/s-3780KiB/s (1645kB/s-3871kB/s), io=28.0MiB (29.4MB), run=2522-3187msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2714: Thu May 1 21:30:10 2025 read: IOPS=4, BW=16.3MiB/s (17.1MB/s)(12.0MiB/736msec) clat (msec): min=77, max=342, avg=223.73, stdev=134.67 lat (msec): min=77, max=342, avg=223.73, stdev=134.67 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 78], 10.00th=[ 78], 20.00th=[ 78], | 30.00th=[ 78], 40.00th=[ 251], 50.00th=[ 251], 60.00th=[ 251], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 342], 95.00th=[ 342], | 99.00th=[ 342], 99.50th=[ 342], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.00%, sys=17.14%, ctx=440, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2715: Thu May 1 21:30:10 2025 read: IOPS=0, BW=3108KiB/s (3182kB/s)(4096KiB/1318msec) clat (nsec): min=236904k, max=236904k, avg=236904167.00, stdev= 0.00 lat (nsec): min=236908k, max=236908k, avg=236907637.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 236], 10.00th=[ 236], 20.00th=[ 236], | 30.00th=[ 236], 40.00th=[ 236], 50.00th=[ 236], 60.00th=[ 236], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 236], 95.00th=[ 236], | 99.00th=[ 236], 99.50th=[ 236], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 8192, max= 8192, per=65.91%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6215KiB/s (6365kB/s)(8192KiB/1318msec) clat (msec): min=480, max=499, avg=489.96, stdev=12.87 lat (msec): min=481, max=499, avg=490.35, stdev=12.83 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 481], 10.00th=[ 481], 20.00th=[ 481], | 30.00th=[ 481], 40.00th=[ 481], 50.00th=[ 481], 60.00th=[ 498], | 70.00th=[ 498], 80.00th=[ 498], 90.00th=[ 498], 95.00th=[ 498], | 99.00th=[ 498], 99.50th=[ 498], 99.90th=[ 498], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 8192, max= 8192, per=32.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.08%, sys=17.16%, ctx=254, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2716: Thu May 1 21:30:10 2025 write: IOPS=2, BW=10.8MiB/s (11.4MB/s)(12.0MiB/1108msec) clat (msec): min=183, max=500, avg=352.32, stdev=159.21 lat (msec): min=184, max=500, avg=352.78, stdev=159.17 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 372], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 7223, max= 8192, per=31.00%, avg=7707.50, stdev=685.19, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=23.13%, ctx=77, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2717: Thu May 1 21:30:10 2025 write: IOPS=3, BW=13.3MiB/s (13.9MB/s)(12.0MiB/902msec) clat (msec): min=193, max=446, avg=288.68, stdev=137.93 lat (msec): min=193, max=447, avg=289.10, stdev=137.84 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 194], 10.00th=[ 194], 20.00th=[ 194], | 30.00th=[ 194], 40.00th=[ 226], 50.00th=[ 226], 60.00th=[ 226], | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min=16384, max=16384, per=65.90%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=31.85%, ctx=81, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.1MiB/s (12.7MB/s), 3108KiB/s-16.3MiB/s (3182kB/s-17.1MB/s), io=16.0MiB (16.8MB), run=736-1318msec WRITE: bw=24.3MiB/s (25.5MB/s), 6215KiB/s-13.3MiB/s (6365kB/s-13.9MB/s), io=32.0MiB (33.6MB), run=902-1318msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2719: Thu May 1 21:30:10 2025 read: IOPS=2, BW=10.8MiB/s (11.4MB/s)(12.0MiB/1108msec) clat (msec): min=265, max=408, avg=323.46, stdev=75.14 lat (msec): min=265, max=408, avg=323.47, stdev=75.14 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 266], 10.00th=[ 266], 20.00th=[ 266], | 30.00th=[ 266], 40.00th=[ 296], 50.00th=[ 296], 60.00th=[ 296], | 70.00th=[ 409], 80.00th=[ 409], 90.00th=[ 409], 95.00th=[ 409], | 99.00th=[ 409], 99.50th=[ 409], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 8175, max= 8192, per=73.68%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=100.00% cpu : usr=0.36%, sys=1.81%, ctx=28, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2720: Thu May 1 21:30:10 2025 read: IOPS=0, BW=2777KiB/s (2844kB/s)(4096KiB/1475msec) clat (nsec): min=345698k, max=345698k, avg=345697890.00, stdev= 0.00 lat (nsec): min=345699k, max=345699k, avg=345698707.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 347], 10.00th=[ 347], 20.00th=[ 347], | 30.00th=[ 347], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 347], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 8175, max= 8175, per=73.60%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5554KiB/s (5687kB/s)(8192KiB/1475msec) clat (msec): min=323, max=685, avg=504.12, stdev=255.87 lat (msec): min=323, max=685, avg=504.44, stdev=255.90 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 326], 10.00th=[ 326], 20.00th=[ 326], | 30.00th=[ 326], 40.00th=[ 326], 50.00th=[ 326], 60.00th=[ 684], | 70.00th=[ 684], 80.00th=[ 684], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.07%, sys=3.32%, ctx=81, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2721: Thu May 1 21:30:10 2025 write: IOPS=2, BW=8956KiB/s (9171kB/s)(12.0MiB/1372msec) clat (msec): min=256, max=505, avg=404.84, stdev=131.10 lat (msec): min=256, max=506, avg=405.36, stdev=131.29 clat percentiles (msec): | 1.00th=[ 257], 5.00th=[ 257], 10.00th=[ 257], 20.00th=[ 257], | 30.00th=[ 257], 40.00th=[ 451], 50.00th=[ 451], 60.00th=[ 451], | 70.00th=[ 506], 80.00th=[ 506], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 8192, max= 8192, per=36.88%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.36%, sys=2.99%, ctx=33, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2722: Thu May 1 21:30:10 2025 write: IOPS=2, BW=10.4MiB/s (10.0MB/s)(12.0MiB/1149msec) clat (msec): min=254, max=438, avg=366.58, stdev=98.12 lat (msec): min=255, max=439, avg=368.27, stdev=98.91 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 255], 10.00th=[ 255], 20.00th=[ 255], | 30.00th=[ 255], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 405], | 70.00th=[ 439], 80.00th=[ 439], 90.00th=[ 439], 95.00th=[ 439], | 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 8192, max= 8192, per=36.88%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=100.00% cpu : usr=0.35%, sys=6.18%, ctx=49, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.8MiB/s (11.4MB/s), 2777KiB/s-10.8MiB/s (2844kB/s-11.4MB/s), io=16.0MiB (16.8MB), run=1108-1475msec WRITE: bw=21.7MiB/s (22.7MB/s), 5554KiB/s-10.4MiB/s (5687kB/s-10.0MB/s), io=32.0MiB (33.6MB), run=1149-1475msec