== sanity test 398b: DIO and buffer IO race ============== 02:03:55 (1739516635) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.44931 s, 20.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28112: Fri Feb 14 02:04:48 2025 read: IOPS=31, BW=125KiB/s (128kB/s)(5992KiB/47752msec) clat (usec): min=62, max=154077, avg=13491.50, stdev=13113.38 lat (usec): min=62, max=154078, avg=13493.62, stdev=13113.91 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 94], 10.00th=[ 106], 20.00th=[ 130], | 30.00th=[ 6259], 40.00th=[ 9241], 50.00th=[ 11600], 60.00th=[ 13960], | 70.00th=[ 17171], 80.00th=[ 21890], 90.00th=[ 30278], 95.00th=[ 37487], | 99.00th=[ 57410], 99.50th=[ 65799], 99.90th=[ 83362], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 23, max= 223, per=25.06%, avg=124.53, stdev=40.43, samples=95 iops : min= 5, max= 55, avg=30.80, stdev=10.13, samples=95 write: IOPS=32, BW=132KiB/s (135kB/s)(6296KiB/47752msec) clat (usec): min=576, max=362039, avg=17359.94, stdev=20939.59 lat (usec): min=577, max=362043, avg=17361.69, stdev=20939.89 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 1004], | 30.00th=[ 2343], 40.00th=[ 5669], 50.00th=[ 11863], 60.00th=[ 16909], | 70.00th=[ 22676], 80.00th=[ 30540], 90.00th=[ 43779], 95.00th=[ 53740], | 99.00th=[ 84411], 99.50th=[106431], 99.90th=[149947], 99.95th=[362808], | 99.99th=[362808] bw ( KiB/s): min= 39, max= 303, per=25.43%, avg=132.21, stdev=45.08, samples=94 iops : min= 9, max= 75, avg=32.72, stdev=11.34, samples=94 lat (usec) : 100=4.04%, 250=8.66%, 500=0.13%, 750=3.35%, 1000=7.19% lat (msec) : 2=4.59%, 4=4.17%, 10=12.96%, 20=25.98%, 50=25.07% lat (msec) : 100=3.52%, 250=0.33%, 500=0.03% cpu : usr=0.14%, sys=9.78%, ctx=4675, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28113: Fri Feb 14 02:04:48 2025 read: IOPS=31, BW=124KiB/s (127kB/s)(5940KiB/47857msec) clat (usec): min=59, max=172889, avg=13124.81, stdev=13301.50 lat (usec): min=59, max=172891, avg=13126.78, stdev=13300.85 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 92], 10.00th=[ 106], 20.00th=[ 130], | 30.00th=[ 5669], 40.00th=[ 8717], 50.00th=[ 11207], 60.00th=[ 13566], | 70.00th=[ 16909], 80.00th=[ 21627], 90.00th=[ 29492], 95.00th=[ 36439], | 99.00th=[ 54789], 99.50th=[ 61604], 99.90th=[121111], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 7, max= 263, per=24.71%, avg=122.79, stdev=43.82, samples=95 iops : min= 1, max= 65, avg=30.25, stdev=10.94, samples=95 write: IOPS=33, BW=133KiB/s (136kB/s)(6348KiB/47857msec) clat (usec): min=574, max=387412, avg=17728.81, stdev=22222.64 lat (usec): min=575, max=387413, avg=17730.91, stdev=22222.34 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 734], 10.00th=[ 791], 20.00th=[ 1012], | 30.00th=[ 2376], 40.00th=[ 6652], 50.00th=[ 12518], 60.00th=[ 17695], | 70.00th=[ 22414], 80.00th=[ 30278], 90.00th=[ 42730], 95.00th=[ 53216], | 99.00th=[ 85459], 99.50th=[114820], 99.90th=[225444], 99.95th=[387974], | 99.99th=[387974] bw ( KiB/s): min= 7, max= 295, per=25.12%, avg=130.61, stdev=45.57, samples=95 iops : min= 1, max= 73, avg=32.21, stdev=11.43, samples=95 lat (usec) : 100=3.81%, 250=8.72%, 500=0.42%, 750=3.71%, 1000=6.87% lat (msec) : 2=4.43%, 4=3.81%, 10=13.67%, 20=24.90%, 50=25.81% lat (msec) : 100=3.42%, 250=0.39%, 500=0.03% cpu : usr=0.14%, sys=9.90%, ctx=4677, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28114: Fri Feb 14 02:04:48 2025 read: IOPS=31, BW=125KiB/s (128kB/s)(6028KiB/48274msec) clat (usec): min=47, max=162181, avg=13280.90, stdev=13450.64 lat (usec): min=48, max=162182, avg=13284.77, stdev=13449.21 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 92], 10.00th=[ 105], 20.00th=[ 129], | 30.00th=[ 5211], 40.00th=[ 8717], 50.00th=[ 11207], 60.00th=[ 13698], | 70.00th=[ 17433], 80.00th=[ 21627], 90.00th=[ 29492], 95.00th=[ 38011], | 99.00th=[ 53740], 99.50th=[ 64750], 99.90th=[105382], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 48, max= 230, per=25.18%, avg=125.17, stdev=41.20, samples=95 iops : min= 12, max= 57, avg=30.93, stdev=10.26, samples=95 write: IOPS=32, BW=130KiB/s (133kB/s)(6260KiB/48274msec) clat (usec): min=576, max=495555, avg=17907.64, stdev=22942.99 lat (usec): min=576, max=495556, avg=17909.78, stdev=22942.99 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 734], 10.00th=[ 799], 20.00th=[ 1090], | 30.00th=[ 3163], 40.00th=[ 7308], 50.00th=[ 12780], 60.00th=[ 16909], | 70.00th=[ 22414], 80.00th=[ 29492], 90.00th=[ 42730], 95.00th=[ 55837], | 99.00th=[ 86508], 99.50th=[109577], 99.90th=[183501], 99.95th=[497026], | 99.99th=[497026] bw ( KiB/s): min= 64, max= 247, per=24.94%, avg=129.68, stdev=39.61, samples=95 iops : min= 16, max= 61, avg=32.05, stdev= 9.92, samples=95 lat (usec) : 50=0.03%, 100=3.87%, 250=9.11%, 500=0.46%, 750=3.03% lat (usec) : 1000=6.97% lat (msec) : 2=3.45%, 4=4.10%, 10=13.77%, 20=26.07%, 50=24.97% lat (msec) : 100=3.81%, 250=0.33%, 500=0.03% cpu : usr=0.13%, sys=9.59%, ctx=4720, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28115: Fri Feb 14 02:04:48 2025 read: IOPS=31, BW=126KiB/s (129kB/s)(6052KiB/48167msec) clat (usec): min=60, max=134841, avg=13056.28, stdev=13530.18 lat (usec): min=60, max=134843, avg=13057.94, stdev=13529.79 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 90], 10.00th=[ 100], 20.00th=[ 122], | 30.00th=[ 635], 40.00th=[ 8291], 50.00th=[ 11076], 60.00th=[ 13829], | 70.00th=[ 17433], 80.00th=[ 22152], 90.00th=[ 29492], 95.00th=[ 37487], | 99.00th=[ 56886], 99.50th=[ 67634], 99.90th=[135267], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 47, max= 271, per=25.62%, avg=127.33, stdev=44.48, samples=94 iops : min= 11, max= 67, avg=31.43, stdev=11.21, samples=94 write: IOPS=32, BW=129KiB/s (133kB/s)(6236KiB/48167msec) clat (usec): min=517, max=416211, avg=18072.59, stdev=22733.25 lat (usec): min=519, max=416214, avg=18074.39, stdev=22733.69 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 742], 10.00th=[ 783], 20.00th=[ 1106], | 30.00th=[ 2671], 40.00th=[ 6652], 50.00th=[ 12387], 60.00th=[ 16909], | 70.00th=[ 22938], 80.00th=[ 31065], 90.00th=[ 43254], 95.00th=[ 55313], | 99.00th=[ 86508], 99.50th=[123208], 99.90th=[189793], 99.95th=[417334], | 99.99th=[417334] bw ( KiB/s): min= 23, max= 255, per=24.94%, avg=129.69, stdev=39.79, samples=95 iops : min= 5, max= 63, avg=32.01, stdev=10.00, samples=95 lat (usec) : 100=5.08%, 250=9.08%, 500=0.52%, 750=3.58%, 1000=6.58% lat (msec) : 2=4.30%, 4=4.43%, 10=12.01%, 20=25.00%, 50=25.03% lat (msec) : 100=4.00%, 250=0.36%, 500=0.03% cpu : usr=0.15%, sys=9.72%, ctx=4605, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=497KiB/s (509kB/s), 124KiB/s-126KiB/s (127kB/s-129kB/s), io=23.4MiB (24.6MB), run=47752-48274msec WRITE: bw=521KiB/s (533kB/s), 129KiB/s-133KiB/s (133kB/s-136kB/s), io=24.6MiB (25.7MB), run=47752-48274msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28108: Fri Feb 14 02:06:54 2025 read: IOPS=8, BW=34.4KiB/s (35.3kB/s)(5992KiB/173980msec) clat (msec): min=3, max=498, avg=17.12, stdev=19.37 lat (msec): min=3, max=499, avg=17.12, stdev=19.37 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 10], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 16], | 70.00th=[ 19], 80.00th=[ 22], 90.00th=[ 30], 95.00th=[ 40], | 99.00th=[ 70], 99.50th=[ 85], 99.90th=[ 292], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 7, max= 111, per=26.23%, avg=35.93, stdev=22.01, samples=328 iops : min= 1, max= 27, avg= 8.63, stdev= 5.50, samples=328 write: IOPS=9, BW=36.2KiB/s (37.1kB/s)(6296KiB/173980msec) clat (msec): min=30, max=372, avg=94.05, stdev=39.88 lat (msec): min=30, max=372, avg=94.06, stdev=39.88 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 51], 10.00th=[ 56], 20.00th=[ 64], | 30.00th=[ 71], 40.00th=[ 78], 50.00th=[ 85], 60.00th=[ 93], | 70.00th=[ 104], 80.00th=[ 120], 90.00th=[ 144], 95.00th=[ 169], | 99.00th=[ 232], 99.50th=[ 245], 99.90th=[ 368], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 8, max= 64, per=24.90%, avg=35.86, stdev=11.61, samples=345 iops : min= 2, max= 16, avg= 8.62, stdev= 2.95, samples=345 lat (msec) : 4=0.13%, 10=15.76%, 20=21.35%, 50=12.79%, 100=32.85% lat (msec) : 250=16.86%, 500=0.26% cpu : usr=0.05%, sys=2.26%, ctx=4016, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28109: Fri Feb 14 02:06:54 2025 read: IOPS=8, BW=34.1KiB/s (34.9kB/s)(5940KiB/174050msec) clat (msec): min=3, max=323, avg=16.65, stdev=16.01 lat (msec): min=3, max=323, avg=16.65, stdev=16.01 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 16], | 70.00th=[ 18], 80.00th=[ 22], 90.00th=[ 29], 95.00th=[ 39], | 99.00th=[ 69], 99.50th=[ 80], 99.90th=[ 271], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 7, max= 136, per=25.94%, avg=35.54, stdev=21.85, samples=330 iops : min= 1, max= 34, avg= 8.63, stdev= 5.49, samples=330 write: IOPS=9, BW=36.5KiB/s (37.3kB/s)(6348KiB/174050msec) clat (msec): min=31, max=333, avg=93.89, stdev=39.12 lat (msec): min=31, max=333, avg=93.90, stdev=39.13 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 51], 10.00th=[ 57], 20.00th=[ 65], | 30.00th=[ 72], 40.00th=[ 79], 50.00th=[ 86], 60.00th=[ 92], | 70.00th=[ 103], 80.00th=[ 116], 90.00th=[ 140], 95.00th=[ 169], | 99.00th=[ 236], 99.50th=[ 279], 99.90th=[ 321], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 7, max= 64, per=25.10%, avg=36.14, stdev=11.77, samples=346 iops : min= 1, max= 16, avg= 8.78, stdev= 3.01, samples=346 lat (msec) : 4=0.13%, 10=15.27%, 20=21.81%, 50=12.27%, 100=33.72% lat (msec) : 250=16.41%, 500=0.39% cpu : usr=0.06%, sys=2.23%, ctx=4009, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28110: Fri Feb 14 02:06:54 2025 read: IOPS=8, BW=34.8KiB/s (35.6kB/s)(6028KiB/173220msec) clat (msec): min=3, max=288, avg=16.98, stdev=15.15 lat (msec): min=3, max=288, avg=16.98, stdev=15.15 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 10], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 17], | 70.00th=[ 19], 80.00th=[ 23], 90.00th=[ 30], 95.00th=[ 39], | 99.00th=[ 65], 99.50th=[ 74], 99.90th=[ 271], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 7, max= 120, per=26.27%, avg=35.99, stdev=21.28, samples=331 iops : min= 1, max= 30, avg= 8.69, stdev= 5.35, samples=331 write: IOPS=9, BW=36.1KiB/s (37.0kB/s)(6260KiB/173220msec) clat (msec): min=31, max=487, avg=94.18, stdev=40.42 lat (msec): min=31, max=487, avg=94.18, stdev=40.42 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 50], 10.00th=[ 56], 20.00th=[ 65], | 30.00th=[ 72], 40.00th=[ 78], 50.00th=[ 86], 60.00th=[ 94], | 70.00th=[ 104], 80.00th=[ 118], 90.00th=[ 142], 95.00th=[ 174], | 99.00th=[ 245], 99.50th=[ 268], 99.90th=[ 334], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 7, max= 64, per=24.87%, avg=35.81, stdev=11.87, samples=345 iops : min= 1, max= 16, avg= 8.65, stdev= 2.99, samples=345 lat (msec) : 4=0.10%, 10=15.56%, 20=20.74%, 50=14.23%, 100=32.29% lat (msec) : 250=16.60%, 500=0.49% cpu : usr=0.04%, sys=2.29%, ctx=4058, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28111: Fri Feb 14 02:06:54 2025 read: IOPS=8, BW=35.0KiB/s (35.9kB/s)(6052KiB/172708msec) clat (msec): min=3, max=126, avg=16.55, stdev=11.79 lat (msec): min=3, max=126, avg=16.55, stdev=11.79 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 10], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 17], | 70.00th=[ 19], 80.00th=[ 22], 90.00th=[ 30], 95.00th=[ 39], | 99.00th=[ 68], 99.50th=[ 73], 99.90th=[ 104], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 7, max= 127, per=26.97%, avg=36.95, stdev=21.49, samples=323 iops : min= 1, max= 31, avg= 8.87, stdev= 5.39, samples=323 write: IOPS=9, BW=36.1KiB/s (36.0kB/s)(6236KiB/172708msec) clat (msec): min=33, max=452, avg=94.56, stdev=40.04 lat (msec): min=33, max=452, avg=94.57, stdev=40.04 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 50], 10.00th=[ 57], 20.00th=[ 66], | 30.00th=[ 72], 40.00th=[ 80], 50.00th=[ 86], 60.00th=[ 94], | 70.00th=[ 104], 80.00th=[ 117], 90.00th=[ 142], 95.00th=[ 169], | 99.00th=[ 239], 99.50th=[ 296], 99.90th=[ 388], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 7, max= 63, per=24.80%, avg=35.71, stdev=11.16, samples=344 iops : min= 1, max= 15, avg= 8.55, stdev= 2.83, samples=344 lat (msec) : 4=0.20%, 10=15.72%, 20=20.57%, 50=14.23%, 100=32.49% lat (msec) : 250=16.44%, 500=0.36% cpu : usr=0.06%, sys=2.24%, ctx=3992, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=138KiB/s (141kB/s), 34.1KiB/s-35.0KiB/s (34.9kB/s-35.9kB/s), io=23.4MiB (24.6MB), run=172708-174050msec WRITE: bw=144KiB/s (148kB/s), 36.1KiB/s-36.5KiB/s (36.0kB/s-37.3kB/s), io=24.6MiB (25.7MB), run=172708-174050msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28282: Fri Feb 14 02:07:05 2025 read: IOPS=40, BW=650KiB/s (666kB/s)(5920KiB/9106msec) clat (usec): min=51, max=134975, avg=11398.95, stdev=13583.64 lat (usec): min=51, max=134976, avg=11400.06, stdev=13583.65 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 95], 10.00th=[ 100], 20.00th=[ 116], | 30.00th=[ 131], 40.00th=[ 5276], 50.00th=[ 10290], 60.00th=[ 12780], | 70.00th=[ 15664], 80.00th=[ 19792], 90.00th=[ 27395], 95.00th=[ 30540], | 99.00th=[ 47449], 99.50th=[107480], 99.90th=[135267], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 384, max= 1437, per=25.64%, avg=645.33, stdev=240.58, samples=18 iops : min= 24, max= 89, avg=40.00, stdev=14.78, samples=18 write: IOPS=43, BW=699KiB/s (716kB/s)(6368KiB/9106msec) clat (usec): min=736, max=118538, avg=11983.01, stdev=14890.22 lat (usec): min=737, max=118540, avg=11984.85, stdev=14890.40 clat percentiles (usec): | 1.00th=[ 947], 5.00th=[ 1045], 10.00th=[ 1139], 20.00th=[ 1336], | 30.00th=[ 2147], 40.00th=[ 3326], 50.00th=[ 5538], 60.00th=[ 9765], | 70.00th=[ 15533], 80.00th=[ 22152], 90.00th=[ 30016], 95.00th=[ 38011], | 99.00th=[ 74974], 99.50th=[101188], 99.90th=[119014], 99.95th=[119014], | 99.99th=[119014] bw ( KiB/s): min= 384, max= 1088, per=26.31%, avg=696.83, stdev=218.80, samples=18 iops : min= 24, max= 68, avg=43.22, stdev=13.55, samples=18 lat (usec) : 100=4.69%, 250=12.11%, 500=0.65%, 750=0.13%, 1000=2.08% lat (msec) : 2=14.06%, 4=8.07%, 10=13.28%, 20=23.44%, 50=20.05% lat (msec) : 100=0.91%, 250=0.52% cpu : usr=0.09%, sys=12.37%, ctx=1013, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28283: Fri Feb 14 02:07:05 2025 read: IOPS=38, BW=614KiB/s (629kB/s)(5744KiB/9348msec) clat (usec): min=72, max=68408, avg=10974.00, stdev=11059.38 lat (usec): min=72, max=68408, avg=10975.44, stdev=11059.05 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 93], 10.00th=[ 102], 20.00th=[ 116], | 30.00th=[ 145], 40.00th=[ 7308], 50.00th=[10290], 60.00th=[12125], | 70.00th=[14484], 80.00th=[19530], 90.00th=[27132], 95.00th=[33162], | 99.00th=[40633], 99.50th=[43254], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 128, max= 1049, per=24.37%, avg=613.28, stdev=218.79, samples=18 iops : min= 8, max= 65, avg=37.94, stdev=13.59, samples=18 write: IOPS=43, BW=700KiB/s (717kB/s)(6544KiB/9348msec) clat (usec): min=790, max=145519, avg=12476.95, stdev=16493.32 lat (usec): min=792, max=145521, avg=12478.78, stdev=16493.31 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 1020], 10.00th=[ 1090], 20.00th=[ 1205], | 30.00th=[ 1811], 40.00th=[ 3228], 50.00th=[ 5735], 60.00th=[ 9503], | 70.00th=[ 15795], 80.00th=[ 22676], 90.00th=[ 30278], 95.00th=[ 41157], | 99.00th=[ 57410], 99.50th=[113771], 99.90th=[145753], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 352, max= 990, per=26.17%, avg=693.28, stdev=163.32, samples=18 iops : min= 22, max= 61, avg=42.94, stdev=10.17, samples=18 lat (usec) : 100=4.30%, 250=10.94%, 500=0.39%, 750=0.13%, 1000=2.99% lat (msec) : 2=14.84%, 4=7.03%, 10=14.71%, 20=22.40%, 50=20.44% lat (msec) : 100=1.43%, 250=0.39% cpu : usr=0.16%, sys=11.96%, ctx=1047, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28284: Fri Feb 14 02:07:05 2025 read: IOPS=42, BW=685KiB/s (701kB/s)(6256KiB/9139msec) clat (usec): min=72, max=108030, avg=11748.39, stdev=12863.65 lat (usec): min=73, max=108031, avg=11749.80, stdev=12863.73 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 88], 10.00th=[ 97], 20.00th=[ 113], | 30.00th=[ 135], 40.00th=[ 7439], 50.00th=[ 10159], 60.00th=[ 13042], | 70.00th=[ 15533], 80.00th=[ 21365], 90.00th=[ 27395], 95.00th=[ 32637], | 99.00th=[ 44827], 99.50th=[ 90702], 99.90th=[108528], 99.95th=[108528], | 99.99th=[108528] bw ( KiB/s): min= 287, max= 992, per=27.49%, avg=691.83, stdev=171.61, samples=18 iops : min= 17, max= 62, avg=42.83, stdev=10.79, samples=18 write: IOPS=41, BW=660KiB/s (676kB/s)(6032KiB/9139msec) clat (usec): min=867, max=131916, avg=11927.25, stdev=15073.68 lat (usec): min=869, max=131919, avg=11929.04, stdev=15073.73 clat percentiles (usec): | 1.00th=[ 922], 5.00th=[ 996], 10.00th=[ 1057], 20.00th=[ 1205], | 30.00th=[ 1467], 40.00th=[ 2999], 50.00th=[ 5800], 60.00th=[ 9634], | 70.00th=[ 15008], 80.00th=[ 21627], 90.00th=[ 30802], 95.00th=[ 42206], | 99.00th=[ 59507], 99.50th=[ 90702], 99.90th=[131597], 99.95th=[131597], | 99.99th=[131597] bw ( KiB/s): min= 383, max= 1181, per=24.77%, avg=656.11, stdev=217.17, samples=18 iops : min= 23, max= 73, avg=40.61, stdev=13.59, samples=18 lat (usec) : 100=5.86%, 250=12.24%, 500=0.26%, 1000=3.12% lat (msec) : 2=14.32%, 4=5.34%, 10=13.80%, 20=22.66%, 50=21.09% lat (msec) : 100=1.04%, 250=0.26% cpu : usr=0.09%, sys=12.08%, ctx=1058, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28287: Fri Feb 14 02:07:05 2025 read: IOPS=39, BW=634KiB/s (649kB/s)(6032KiB/9513msec) clat (usec): min=73, max=83925, avg=11144.81, stdev=12095.03 lat (usec): min=74, max=83925, avg=11145.91, stdev=12095.05 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 89], 10.00th=[ 98], 20.00th=[ 113], | 30.00th=[ 131], 40.00th=[ 1500], 50.00th=[ 9503], 60.00th=[12911], | 70.00th=[16450], 80.00th=[21103], 90.00th=[26870], 95.00th=[30802], | 99.00th=[44827], 99.50th=[74974], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 383, max= 1024, per=26.14%, avg=658.06, stdev=187.92, samples=18 iops : min= 23, max= 64, avg=40.78, stdev=11.74, samples=18 write: IOPS=41, BW=658KiB/s (673kB/s)(6256KiB/9513msec) clat (usec): min=763, max=129516, avg=13017.12, stdev=17835.09 lat (usec): min=764, max=129518, avg=13018.94, stdev=17835.13 clat percentiles (usec): | 1.00th=[ 873], 5.00th=[ 996], 10.00th=[ 1106], 20.00th=[ 1303], | 30.00th=[ 1876], 40.00th=[ 3359], 50.00th=[ 5145], 60.00th=[ 9110], | 70.00th=[ 14484], 80.00th=[ 22414], 90.00th=[ 34866], 95.00th=[ 47449], | 99.00th=[ 91751], 99.50th=[116917], 99.90th=[129500], 99.95th=[129500], | 99.99th=[129500] bw ( KiB/s): min= 96, max= 990, per=24.61%, avg=651.95, stdev=225.90, samples=19 iops : min= 6, max= 61, avg=40.42, stdev=13.92, samples=19 lat (usec) : 100=5.73%, 250=12.37%, 500=0.65%, 750=0.26%, 1000=2.86% lat (msec) : 2=13.41%, 4=7.16%, 10=14.32%, 20=20.70%, 50=19.92% lat (msec) : 100=2.34%, 250=0.26% cpu : usr=0.13%, sys=11.60%, ctx=1040, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2518KiB/s (2578kB/s), 614KiB/s-685KiB/s (629kB/s-701kB/s), io=23.4MiB (24.5MB), run=9106-9513msec WRITE: bw=2649KiB/s (2713kB/s), 658KiB/s-700KiB/s (673kB/s-717kB/s), io=24.6MiB (25.8MB), run=9106-9513msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28285: Fri Feb 14 02:07:34 2025 read: IOPS=9, BW=154KiB/s (158kB/s)(5920KiB/38465msec) clat (msec): min=4, max=109, avg=13.64, stdev=13.34 lat (msec): min=4, max=109, avg=13.65, stdev=13.34 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 16], 90.00th=[ 26], 95.00th=[ 39], | 99.00th=[ 72], 99.50th=[ 100], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 31, max= 415, per=24.74%, avg=152.87, stdev=96.66, samples=75 iops : min= 1, max= 25, avg= 9.12, stdev= 6.15, samples=75 write: IOPS=10, BW=166KiB/s (170kB/s)(6368KiB/38465msec) clat (msec): min=34, max=277, avg=83.74, stdev=35.52 lat (msec): min=34, max=277, avg=83.74, stdev=35.52 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 45], 10.00th=[ 49], 20.00th=[ 56], | 30.00th=[ 64], 40.00th=[ 69], 50.00th=[ 77], 60.00th=[ 83], | 70.00th=[ 91], 80.00th=[ 103], 90.00th=[ 129], 95.00th=[ 163], | 99.00th=[ 201], 99.50th=[ 211], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 62, max= 256, per=25.12%, avg=163.25, stdev=53.01, samples=76 iops : min= 3, max= 16, avg= 9.74, stdev= 3.35, samples=76 lat (msec) : 10=27.73%, 20=13.41%, 50=11.07%, 100=36.07%, 250=11.59% lat (msec) : 500=0.13% cpu : usr=0.04%, sys=2.72%, ctx=905, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28286: Fri Feb 14 02:07:34 2025 read: IOPS=9, BW=148KiB/s (152kB/s)(5744KiB/38724msec) clat (msec): min=3, max=142, avg=14.07, stdev=15.43 lat (msec): min=3, max=142, avg=14.07, stdev=15.43 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 16], 90.00th=[ 25], 95.00th=[ 40], | 99.00th=[ 95], 99.50th=[ 112], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 31, max= 480, per=25.04%, avg=154.77, stdev=97.05, samples=73 iops : min= 1, max= 30, avg= 9.29, stdev= 6.18, samples=73 write: IOPS=10, BW=169KiB/s (173kB/s)(6544KiB/38724msec) clat (msec): min=27, max=296, avg=81.59, stdev=35.57 lat (msec): min=27, max=296, avg=81.60, stdev=35.56 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 44], 10.00th=[ 47], 20.00th=[ 54], | 30.00th=[ 63], 40.00th=[ 69], 50.00th=[ 75], 60.00th=[ 81], | 70.00th=[ 89], 80.00th=[ 102], 90.00th=[ 126], 95.00th=[ 153], | 99.00th=[ 209], 99.50th=[ 249], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 31, max= 256, per=25.79%, avg=167.64, stdev=55.89, samples=77 iops : min= 1, max= 16, avg=10.10, stdev= 3.56, samples=77 lat (msec) : 4=0.52%, 10=24.87%, 20=14.45%, 50=12.76%, 100=35.81% lat (msec) : 250=11.33%, 500=0.26% cpu : usr=0.07%, sys=2.57%, ctx=898, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28288: Fri Feb 14 02:07:34 2025 read: IOPS=10, BW=174KiB/s (178kB/s)(6256KiB/35999msec) clat (msec): min=4, max=109, avg=13.99, stdev=15.29 lat (msec): min=4, max=109, avg=14.00, stdev=15.29 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 16], 90.00th=[ 24], 95.00th=[ 42], | 99.00th=[ 99], 99.50th=[ 100], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 31, max= 416, per=28.28%, avg=174.77, stdev=109.60, samples=71 iops : min= 1, max= 26, avg=10.58, stdev= 6.95, samples=71 write: IOPS=10, BW=168KiB/s (172kB/s)(6032KiB/35999msec) clat (msec): min=31, max=272, avg=80.87, stdev=34.41 lat (msec): min=31, max=272, avg=80.87, stdev=34.41 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 46], 10.00th=[ 51], 20.00th=[ 56], | 30.00th=[ 61], 40.00th=[ 66], 50.00th=[ 72], 60.00th=[ 79], | 70.00th=[ 87], 80.00th=[ 103], 90.00th=[ 125], 95.00th=[ 142], | 99.00th=[ 239], 99.50th=[ 268], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 63, max= 256, per=25.58%, avg=166.27, stdev=51.74, samples=71 iops : min= 3, max= 16, avg=10.04, stdev= 3.38, samples=71 lat (msec) : 10=28.52%, 20=15.49%, 50=9.90%, 100=35.29%, 250=10.55% lat (msec) : 500=0.26% cpu : usr=0.01%, sys=2.87%, ctx=903, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28289: Fri Feb 14 02:07:34 2025 read: IOPS=10, BW=160KiB/s (164kB/s)(6032KiB/37618msec) clat (usec): min=3881, max=99314, avg=12499.00, stdev=11649.04 lat (usec): min=3883, max=99316, avg=12500.58, stdev=11649.05 clat percentiles (usec): | 1.00th=[ 4080], 5.00th=[ 4948], 10.00th=[ 5669], 20.00th=[ 6456], | 30.00th=[ 7111], 40.00th=[ 7898], 50.00th=[ 8717], 60.00th=[10290], | 70.00th=[11863], 80.00th=[14484], 90.00th=[22676], 95.00th=[36439], | 99.00th=[65799], 99.50th=[98042], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 31, max= 448, per=26.74%, avg=165.26, stdev=94.55, samples=72 iops : min= 1, max= 28, avg= 9.93, stdev= 6.01, samples=72 write: IOPS=10, BW=166KiB/s (170kB/s)(6256KiB/37618msec) clat (msec): min=35, max=299, avg=83.89, stdev=34.86 lat (msec): min=35, max=299, avg=83.90, stdev=34.86 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 45], 10.00th=[ 50], 20.00th=[ 56], | 30.00th=[ 64], 40.00th=[ 70], 50.00th=[ 78], 60.00th=[ 84], | 70.00th=[ 90], 80.00th=[ 107], 90.00th=[ 132], 95.00th=[ 150], | 99.00th=[ 203], 99.50th=[ 239], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 63, max= 256, per=25.53%, avg=165.92, stdev=53.51, samples=75 iops : min= 3, max= 16, avg= 9.97, stdev= 3.45, samples=75 lat (msec) : 4=0.13%, 10=28.91%, 20=14.45%, 50=10.03%, 100=34.64% lat (msec) : 250=11.72%, 500=0.13% cpu : usr=0.06%, sys=2.69%, ctx=887, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=619KiB/s (633kB/s), 148KiB/s-174KiB/s (152kB/s-178kB/s), io=23.4MiB (24.5MB), run=35999-38724msec WRITE: bw=651KiB/s (666kB/s), 166KiB/s-169KiB/s (170kB/s-173kB/s), io=24.6MiB (25.8MB), run=35999-38724msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28327: Fri Feb 14 02:07:37 2025 read: IOPS=6, BW=6158KiB/s (6306kB/s)(7168KiB/1164msec) clat (usec): min=714, max=97870, avg=59996.33, stdev=41569.44 lat (usec): min=716, max=97872, avg=59998.33, stdev=41569.30 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 717], 10.00th=[ 717], 20.00th=[ 1369], | 30.00th=[73925], 40.00th=[73925], 50.00th=[73925], 60.00th=[74974], | 70.00th=[74974], 80.00th=[96994], 90.00th=[98042], 95.00th=[98042], | 99.00th=[98042], 99.50th=[98042], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 2048, max=10240, per=61.80%, avg=6144.00, stdev=5792.62, samples=2 iops : min= 2, max= 10, avg= 6.00, stdev= 5.66, samples=2 write: IOPS=4, BW=4399KiB/s (4504kB/s)(5120KiB/1164msec) clat (msec): min=50, max=264, avg=132.23, stdev=104.31 lat (msec): min=50, max=264, avg=132.32, stdev=104.28 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 51], 10.00th=[ 51], 20.00th=[ 51], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 63], 60.00th=[ 63], | 70.00th=[ 226], 80.00th=[ 226], 90.00th=[ 266], 95.00th=[ 266], | 99.00th=[ 266], 99.50th=[ 266], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 2048, max= 6144, per=29.43%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (usec) : 750=8.33% lat (msec) : 2=8.33%, 100=66.67%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=17.45%, ctx=245, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28328: Fri Feb 14 02:07:37 2025 read: IOPS=1, BW=1553KiB/s (1590kB/s)(2048KiB/1319msec) clat (msec): min=91, max=103, avg=97.57, stdev= 9.08 lat (msec): min=91, max=103, avg=97.57, stdev= 9.08 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 91], 10.00th=[ 91], 20.00th=[ 91], | 30.00th=[ 91], 40.00th=[ 91], 50.00th=[ 91], 60.00th=[ 105], | 70.00th=[ 105], 80.00th=[ 105], 90.00th=[ 105], 95.00th=[ 105], | 99.00th=[ 105], 99.50th=[ 105], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 2039, max= 2043, per=20.53%, avg=2041.00, stdev= 2.83, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=7, BW=7763KiB/s (7950kB/s)(10.0MiB/1319msec) clat (msec): min=49, max=277, avg=109.76, stdev=71.69 lat (msec): min=49, max=278, avg=109.90, stdev=71.81 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 51], 10.00th=[ 51], 20.00th=[ 59], | 30.00th=[ 63], 40.00th=[ 66], 50.00th=[ 72], 60.00th=[ 92], | 70.00th=[ 153], 80.00th=[ 153], 90.00th=[ 171], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 4087, max=10199, per=51.32%, avg=7143.00, stdev=4321.84, samples=2 iops : min= 3, max= 9, avg= 6.00, stdev= 4.24, samples=2 lat (msec) : 50=8.33%, 100=58.33%, 250=25.00%, 500=8.33% cpu : usr=0.08%, sys=18.21%, ctx=119, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28329: Fri Feb 14 02:07:37 2025 read: IOPS=4, BW=4379KiB/s (4484kB/s)(6144KiB/1403msec) clat (msec): min=3, max=133, avg=66.45, stdev=55.07 lat (msec): min=3, max=133, avg=66.45, stdev=55.07 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 16], | 30.00th=[ 16], 40.00th=[ 34], 50.00th=[ 34], 60.00th=[ 105], | 70.00th=[ 107], 80.00th=[ 107], 90.00th=[ 134], 95.00th=[ 134], | 99.00th=[ 134], 99.50th=[ 134], 99.90th=[ 134], 99.95th=[ 134], | 99.99th=[ 134] bw ( KiB/s): min= 4096, max= 4096, per=41.20%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4379KiB/s (4484kB/s)(6144KiB/1403msec) clat (msec): min=62, max=274, avg=156.81, stdev=92.93 lat (msec): min=62, max=274, avg=156.89, stdev=92.92 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 66], | 30.00th=[ 66], 40.00th=[ 96], 50.00th=[ 96], 60.00th=[ 213], | 70.00th=[ 230], 80.00th=[ 230], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 2039, max= 6144, per=29.40%, avg=4091.50, stdev=2902.67, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (msec) : 4=8.33%, 20=8.33%, 50=8.33%, 100=25.00%, 250=41.67% lat (msec) : 500=8.33% cpu : usr=0.00%, sys=14.55%, ctx=217, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28332: Fri Feb 14 02:07:37 2025 read: IOPS=2, BW=2485KiB/s (2545kB/s)(5120KiB/2060msec) clat (msec): min=35, max=123, avg=81.12, stdev=32.21 lat (msec): min=35, max=123, avg=81.13, stdev=32.21 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 68], 40.00th=[ 68], 50.00th=[ 89], 60.00th=[ 89], | 70.00th=[ 91], 80.00th=[ 91], 90.00th=[ 124], 95.00th=[ 124], | 99.00th=[ 124], 99.50th=[ 124], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 8192, max= 8192, per=82.41%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3480KiB/s (3563kB/s)(7168KiB/2060msec) clat (msec): min=102, max=404, avg=219.49, stdev=105.21 lat (msec): min=102, max=404, avg=219.58, stdev=105.20 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 153], | 30.00th=[ 171], 40.00th=[ 171], 50.00th=[ 180], 60.00th=[ 209], | 70.00th=[ 209], 80.00th=[ 321], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 2043, max= 6144, per=34.29%, avg=4772.67, stdev=2363.97, samples=3 iops : min= 1, max= 6, avg= 4.00, stdev= 2.65, samples=3 lat (msec) : 50=8.33%, 100=25.00%, 250=50.00%, 500=16.67% cpu : usr=0.00%, sys=11.56%, ctx=133, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9942KiB/s (10.2MB/s), 1553KiB/s-6158KiB/s (1590kB/s-6306kB/s), io=20.0MiB (20.0MB), run=1164-2060msec WRITE: bw=13.6MiB/s (14.3MB/s), 3480KiB/s-7763KiB/s (3563kB/s-7950kB/s), io=28.0MiB (29.4MB), run=1164-2060msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28330: Fri Feb 14 02:07:38 2025 read: IOPS=3, BW=3451KiB/s (3534kB/s)(7168KiB/2077msec) clat (msec): min=50, max=260, avg=130.32, stdev=80.46 lat (msec): min=50, max=260, avg=130.33, stdev=80.46 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 51], 10.00th=[ 51], 20.00th=[ 64], | 30.00th=[ 73], 40.00th=[ 73], 50.00th=[ 95], 60.00th=[ 176], | 70.00th=[ 176], 80.00th=[ 197], 90.00th=[ 262], 95.00th=[ 262], | 99.00th=[ 262], 99.50th=[ 262], 99.90th=[ 262], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 1914, max= 4087, per=34.10%, avg=3031.75, stdev=1219.57, samples=4 iops : min= 1, max= 3, avg= 2.00, stdev= 1.15, samples=4 write: IOPS=2, BW=2465KiB/s (2524kB/s)(5120KiB/2077msec) clat (msec): min=96, max=350, avg=198.87, stdev=116.62 lat (msec): min=96, max=350, avg=198.96, stdev=116.62 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 97], 10.00th=[ 97], 20.00th=[ 97], | 30.00th=[ 110], 40.00th=[ 110], 50.00th=[ 142], 60.00th=[ 142], | 70.00th=[ 296], 80.00th=[ 296], 90.00th=[ 351], 95.00th=[ 351], | 99.00th=[ 351], 99.50th=[ 351], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 2043, max= 7656, per=38.95%, avg=4849.50, stdev=3968.99, samples=2 iops : min= 1, max= 7, avg= 4.00, stdev= 4.24, samples=2 lat (msec) : 100=41.67%, 250=33.33%, 500=25.00% cpu : usr=0.29%, sys=5.64%, ctx=47, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28331: Fri Feb 14 02:07:38 2025 read: IOPS=0, BW=889KiB/s (911kB/s)(2048KiB/2303msec) clat (usec): min=49928, max=63679, avg=56803.75, stdev=9723.61 lat (usec): min=49929, max=63682, avg=56806.00, stdev=9724.59 clat percentiles (usec): | 1.00th=[50070], 5.00th=[50070], 10.00th=[50070], 20.00th=[50070], | 30.00th=[50070], 40.00th=[50070], 50.00th=[50070], 60.00th=[63701], | 70.00th=[63701], 80.00th=[63701], 90.00th=[63701], 95.00th=[63701], | 99.00th=[63701], 99.50th=[63701], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 1885, max= 2043, per=22.09%, avg=1964.00, stdev=111.72, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4446KiB/s (4553kB/s)(10.0MiB/2303msec) clat (msec): min=66, max=386, avg=200.38, stdev=97.25 lat (msec): min=66, max=386, avg=200.46, stdev=97.25 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 127], | 30.00th=[ 134], 40.00th=[ 144], 50.00th=[ 184], 60.00th=[ 186], | 70.00th=[ 262], 80.00th=[ 262], 90.00th=[ 321], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 4079, max= 5657, per=37.04%, avg=4610.67, stdev=906.19, samples=3 iops : min= 3, max= 5, avg= 4.00, stdev= 1.00, samples=3 lat (msec) : 50=8.33%, 100=16.67%, 250=50.00%, 500=25.00% cpu : usr=0.00%, sys=6.04%, ctx=71, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28333: Fri Feb 14 02:07:38 2025 read: IOPS=2, BW=2843KiB/s (2911kB/s)(6144KiB/2161msec) clat (msec): min=48, max=131, avg=91.43, stdev=35.81 lat (msec): min=48, max=131, avg=91.44, stdev=35.81 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 83], 50.00th=[ 83], 60.00th=[ 116], | 70.00th=[ 120], 80.00th=[ 120], 90.00th=[ 132], 95.00th=[ 132], | 99.00th=[ 132], 99.50th=[ 132], 99.90th=[ 132], 99.95th=[ 132], | 99.99th=[ 132] bw ( KiB/s): min= 3696, max= 6131, per=55.26%, avg=4913.50, stdev=1721.81, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2843KiB/s (2911kB/s)(6144KiB/2161msec) clat (msec): min=127, max=499, avg=264.46, stdev=128.55 lat (msec): min=127, max=499, avg=264.53, stdev=128.55 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 128], 10.00th=[ 128], 20.00th=[ 178], | 30.00th=[ 178], 40.00th=[ 247], 50.00th=[ 247], 60.00th=[ 251], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 498], 95.00th=[ 498], | 99.00th=[ 498], 99.50th=[ 498], 99.90th=[ 498], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 1848, max= 4087, per=20.13%, avg=2506.50, stdev=1057.77, samples=4 iops : min= 1, max= 3, avg= 1.75, stdev= 0.96, samples=4 lat (msec) : 50=8.33%, 100=16.67%, 250=50.00%, 500=25.00% cpu : usr=0.00%, sys=4.77%, ctx=47, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28334: Fri Feb 14 02:07:38 2025 read: IOPS=2, BW=2956KiB/s (3027kB/s)(5120KiB/1732msec) clat (msec): min=43, max=132, avg=84.82, stdev=39.77 lat (msec): min=43, max=132, avg=84.82, stdev=39.77 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 44], | 30.00th=[ 52], 40.00th=[ 52], 50.00th=[ 79], 60.00th=[ 79], | 70.00th=[ 120], 80.00th=[ 120], 90.00th=[ 133], 95.00th=[ 133], | 99.00th=[ 133], 99.50th=[ 133], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 2048, max= 6131, per=45.99%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=4, BW=4139KiB/s (4238kB/s)(7168KiB/1732msec) clat (msec): min=115, max=253, avg=184.17, stdev=62.76 lat (msec): min=115, max=253, avg=184.25, stdev=62.77 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 115], 10.00th=[ 115], 20.00th=[ 118], | 30.00th=[ 127], 40.00th=[ 127], 50.00th=[ 197], 60.00th=[ 241], | 70.00th=[ 241], 80.00th=[ 241], 90.00th=[ 253], 95.00th=[ 253], | 99.00th=[ 253], 99.50th=[ 253], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 2043, max= 4096, per=27.38%, avg=3408.67, stdev=1182.71, samples=3 iops : min= 1, max= 4, avg= 2.67, stdev= 1.53, samples=3 lat (msec) : 50=8.33%, 100=16.67%, 250=66.67%, 500=8.33% cpu : usr=0.00%, sys=7.39%, ctx=51, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8893KiB/s (9106kB/s), 889KiB/s-3451KiB/s (911kB/s-3534kB/s), io=20.0MiB (20.0MB), run=1732-2303msec WRITE: bw=12.2MiB/s (12.7MB/s), 2465KiB/s-4446KiB/s (2524kB/s-4553kB/s), io=28.0MiB (29.4MB), run=1732-2303msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28342: Fri Feb 14 02:07:41 2025 read: IOPS=3, BW=14.2MiB/s (14.9MB/s)(12.0MiB/844msec) clat (msec): min=59, max=427, avg=256.61, stdev=185.51 lat (msec): min=59, max=427, avg=256.62, stdev=185.51 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 60], 10.00th=[ 60], 20.00th=[ 60], | 30.00th=[ 60], 40.00th=[ 284], 50.00th=[ 284], 60.00th=[ 284], | 70.00th=[ 426], 80.00th=[ 426], 90.00th=[ 426], 95.00th=[ 426], | 99.00th=[ 426], 99.50th=[ 426], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 8159, max= 8159, per=72.66%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.12%, sys=13.52%, ctx=411, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28343: Fri Feb 14 02:07:41 2025 read: IOPS=0, BW=2807KiB/s (2875kB/s)(4096KiB/1459msec) clat (nsec): min=193368k, max=193368k, avg=193368306.00, stdev= 0.00 lat (nsec): min=193371k, max=193371k, avg=193370503.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 194], 10.00th=[ 194], 20.00th=[ 194], | 30.00th=[ 194], 40.00th=[ 194], 50.00th=[ 194], 60.00th=[ 194], | 70.00th=[ 194], 80.00th=[ 194], 90.00th=[ 194], 95.00th=[ 194], | 99.00th=[ 194], 99.50th=[ 194], 99.90th=[ 194], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 8192, max= 8192, per=72.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5615KiB/s (5750kB/s)(8192KiB/1459msec) clat (msec): min=279, max=732, avg=506.28, stdev=320.31 lat (msec): min=280, max=733, avg=506.62, stdev=320.38 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 279], 10.00th=[ 279], 20.00th=[ 279], | 30.00th=[ 279], 40.00th=[ 279], 50.00th=[ 279], 60.00th=[ 735], | 70.00th=[ 735], 80.00th=[ 735], 90.00th=[ 735], 95.00th=[ 735], | 99.00th=[ 735], 99.50th=[ 735], 99.90th=[ 735], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 8110, max= 8110, per=36.11%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=18.18%, ctx=374, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28345: Fri Feb 14 02:07:41 2025 write: IOPS=2, BW=9745KiB/s (9979kB/s)(12.0MiB/1261msec) clat (msec): min=207, max=521, avg=403.84, stdev=170.83 lat (msec): min=208, max=522, avg=404.25, stdev=170.77 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 209], 10.00th=[ 209], 20.00th=[ 209], | 30.00th=[ 209], 40.00th=[ 481], 50.00th=[ 481], 60.00th=[ 481], | 70.00th=[ 523], 80.00th=[ 523], 90.00th=[ 523], 95.00th=[ 523], | 99.00th=[ 523], 99.50th=[ 523], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 8110, max= 8192, per=36.29%, avg=8151.00, stdev=57.98, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.16%, sys=19.21%, ctx=87, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28347: Fri Feb 14 02:07:41 2025 write: IOPS=2, BW=11.7MiB/s (12.3MB/s)(12.0MiB/1025msec) clat (msec): min=222, max=532, avg=326.38, stdev=178.27 lat (msec): min=223, max=532, avg=328.74, stdev=176.75 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 224], 10.00th=[ 224], 20.00th=[ 224], | 30.00th=[ 224], 40.00th=[ 224], 50.00th=[ 224], 60.00th=[ 224], | 70.00th=[ 531], 80.00th=[ 531], 90.00th=[ 531], 95.00th=[ 531], | 99.00th=[ 531], 99.50th=[ 531], 99.90th=[ 531], 99.95th=[ 531], | 99.99th=[ 531] bw ( KiB/s): min=16351, max=16351, per=72.80%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=29.59%, ctx=98, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.0MiB/s (11.5MB/s), 2807KiB/s-14.2MiB/s (2875kB/s-14.9MB/s), io=16.0MiB (16.8MB), run=844-1459msec WRITE: bw=21.9MiB/s (22.0MB/s), 5615KiB/s-11.7MiB/s (5750kB/s-12.3MB/s), io=32.0MiB (33.6MB), run=1025-1459msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28344: Fri Feb 14 02:07:41 2025 read: IOPS=2, BW=9323KiB/s (9547kB/s)(12.0MiB/1318msec) clat (msec): min=333, max=467, avg=414.64, stdev=71.36 lat (msec): min=333, max=467, avg=414.64, stdev=71.36 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 334], 10.00th=[ 334], 20.00th=[ 334], | 30.00th=[ 334], 40.00th=[ 443], 50.00th=[ 443], 60.00th=[ 443], | 70.00th=[ 468], 80.00th=[ 468], 90.00th=[ 468], 95.00th=[ 468], | 99.00th=[ 468], 99.50th=[ 468], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 8192, max= 8192, per=89.15%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=100.00% cpu : usr=0.30%, sys=6.15%, ctx=47, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28346: Fri Feb 14 02:07:41 2025 read: IOPS=0, BW=2297KiB/s (2352kB/s)(4096KiB/1783msec) clat (nsec): min=299413k, max=299413k, avg=299412927.00, stdev= 0.00 lat (nsec): min=299414k, max=299414k, avg=299414430.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 8192, max= 8192, per=89.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4595KiB/s (4705kB/s)(8192KiB/1783msec) clat (msec): min=504, max=766, avg=635.20, stdev=185.27 lat (msec): min=504, max=766, avg=635.57, stdev=185.24 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 506], 10.00th=[ 506], 20.00th=[ 506], | 30.00th=[ 506], 40.00th=[ 506], 50.00th=[ 506], 60.00th=[ 768], | 70.00th=[ 768], 80.00th=[ 768], 90.00th=[ 768], 95.00th=[ 768], | 99.00th=[ 768], 99.50th=[ 768], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 8192, max= 8192, per=44.58%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.11%, sys=9.03%, ctx=67, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28348: Fri Feb 14 02:07:41 2025 write: IOPS=1, BW=7928KiB/s (8118kB/s)(12.0MiB/1550msec) clat (msec): min=289, max=536, avg=449.59, stdev=139.21 lat (msec): min=289, max=537, avg=449.89, stdev=139.17 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 288], 10.00th=[ 288], 20.00th=[ 288], | 30.00th=[ 288], 40.00th=[ 523], 50.00th=[ 523], 60.00th=[ 523], | 70.00th=[ 535], 80.00th=[ 535], 90.00th=[ 535], 95.00th=[ 535], | 99.00th=[ 535], 99.50th=[ 535], 99.90th=[ 535], 99.95th=[ 535], | 99.99th=[ 535] bw ( KiB/s): min= 8159, max= 8192, per=44.49%, avg=8175.50, stdev=23.33, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.32%, sys=6.58%, ctx=58, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28349: Fri Feb 14 02:07:41 2025 write: IOPS=1, BW=7782KiB/s (7969kB/s)(12.0MiB/1579msec) clat (msec): min=382, max=619, avg=508.99, stdev=119.16 lat (msec): min=383, max=619, avg=509.28, stdev=119.16 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 384], 10.00th=[ 384], 20.00th=[ 384], | 30.00th=[ 384], 40.00th=[ 527], 50.00th=[ 527], 60.00th=[ 527], | 70.00th=[ 617], 80.00th=[ 617], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 8159, max= 8175, per=44.44%, avg=8167.00, stdev=11.31, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.00%, sys=6.46%, ctx=46, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9189KiB/s (9410kB/s), 2297KiB/s-9323KiB/s (2352kB/s-9547kB/s), io=16.0MiB (16.8MB), run=1318-1783msec WRITE: bw=17.9MiB/s (18.8MB/s), 4595KiB/s-7928KiB/s (4705kB/s-8118kB/s), io=32.0MiB (33.6MB), run=1550-1783msec