== sanity test 398b: DIO and buffer IO race ============== 03:28:31 (1746084511) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.55021 s, 32.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31329: Thu May 1 03:29:00 2025 read: IOPS=58, BW=233KiB/s (238kB/s)(5992KiB/25730msec) clat (usec): min=57, max=67183, avg=7239.88, stdev=6761.70 lat (usec): min=58, max=67185, avg=7240.83, stdev=6761.72 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 91], 10.00th=[ 100], 20.00th=[ 133], | 30.00th=[ 3752], 40.00th=[ 5080], 50.00th=[ 6325], 60.00th=[ 7504], | 70.00th=[ 9241], 80.00th=[11863], 90.00th=[15926], 95.00th=[19006], | 99.00th=[27132], 99.50th=[32637], 99.90th=[55313], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 40, max= 360, per=24.87%, avg=231.76, stdev=59.93, samples=51 iops : min= 10, max= 90, avg=57.69, stdev=14.97, samples=51 write: IOPS=61, BW=245KiB/s (251kB/s)(6296KiB/25730msec) clat (usec): min=509, max=238166, avg=9343.31, stdev=10289.83 lat (usec): min=510, max=238168, avg=9344.62, stdev=10289.94 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 725], 10.00th=[ 783], 20.00th=[ 1029], | 30.00th=[ 1942], 40.00th=[ 5014], 50.00th=[ 7635], 60.00th=[ 9896], | 70.00th=[ 12518], 80.00th=[ 15926], 90.00th=[ 20841], 95.00th=[ 25035], | 99.00th=[ 35390], 99.50th=[ 44303], 99.90th=[ 54789], 99.95th=[238027], | 99.99th=[238027] bw ( KiB/s): min= 32, max= 400, per=24.95%, avg=243.49, stdev=60.98, samples=51 iops : min= 8, max= 100, avg=60.63, stdev=15.24, samples=51 lat (usec) : 100=4.85%, 250=6.64%, 500=0.68%, 750=4.00%, 1000=6.41% lat (msec) : 2=5.99%, 4=6.22%, 10=31.84%, 20=25.33%, 50=7.85% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.12%, sys=16.50%, ctx=5127, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31330: Thu May 1 03:29:00 2025 read: IOPS=57, BW=231KiB/s (236kB/s)(5940KiB/25743msec) clat (usec): min=47, max=59742, avg=7056.22, stdev=6451.08 lat (usec): min=47, max=59743, avg=7057.27, stdev=6451.10 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 91], 10.00th=[ 102], 20.00th=[ 127], | 30.00th=[ 3916], 40.00th=[ 5080], 50.00th=[ 6063], 60.00th=[ 7308], | 70.00th=[ 8848], 80.00th=[11731], 90.00th=[15533], 95.00th=[18744], | 99.00th=[24773], 99.50th=[30540], 99.90th=[52691], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 8, max= 367, per=24.68%, avg=230.00, stdev=68.97, samples=51 iops : min= 2, max= 91, avg=57.31, stdev=17.23, samples=51 write: IOPS=61, BW=247KiB/s (253kB/s)(6348KiB/25743msec) clat (usec): min=454, max=252477, avg=9507.25, stdev=11247.23 lat (usec): min=456, max=252477, avg=9509.04, stdev=11247.16 clat percentiles (usec): | 1.00th=[ 578], 5.00th=[ 717], 10.00th=[ 766], 20.00th=[ 979], | 30.00th=[ 2024], 40.00th=[ 5014], 50.00th=[ 7504], 60.00th=[ 9503], | 70.00th=[ 11994], 80.00th=[ 15795], 90.00th=[ 21365], 95.00th=[ 26608], | 99.00th=[ 40633], 99.50th=[ 52167], 99.90th=[ 86508], 99.95th=[252707], | 99.99th=[252707] bw ( KiB/s): min= 40, max= 360, per=25.07%, avg=244.69, stdev=62.25, samples=51 iops : min= 10, max= 90, avg=60.98, stdev=15.60, samples=51 lat (usec) : 50=0.07%, 100=4.30%, 250=7.62%, 500=0.68%, 750=4.39% lat (usec) : 1000=6.28% lat (msec) : 2=4.88%, 4=6.09%, 10=33.98%, 20=23.80%, 50=7.55% lat (msec) : 100=0.33%, 500=0.03% cpu : usr=0.19%, sys=16.19%, ctx=5061, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31331: Thu May 1 03:29:00 2025 read: IOPS=58, BW=235KiB/s (240kB/s)(6028KiB/25671msec) clat (usec): min=50, max=41563, avg=7069.66, stdev=6458.19 lat (usec): min=50, max=41564, avg=7071.00, stdev=6458.21 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 91], 10.00th=[ 99], 20.00th=[ 123], | 30.00th=[ 3392], 40.00th=[ 5080], 50.00th=[ 6259], 60.00th=[ 7504], | 70.00th=[ 9241], 80.00th=[11731], 90.00th=[15926], 95.00th=[19268], | 99.00th=[25822], 99.50th=[30540], 99.90th=[40109], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 119, max= 399, per=25.64%, avg=239.00, stdev=60.04, samples=50 iops : min= 29, max= 99, avg=59.48, stdev=14.99, samples=50 write: IOPS=60, BW=244KiB/s (250kB/s)(6260KiB/25671msec) clat (usec): min=459, max=297695, avg=9484.64, stdev=12249.91 lat (usec): min=460, max=297696, avg=9486.11, stdev=12249.98 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 709], 10.00th=[ 750], 20.00th=[ 947], | 30.00th=[ 1647], 40.00th=[ 4359], 50.00th=[ 7308], 60.00th=[ 9372], | 70.00th=[ 12125], 80.00th=[ 15795], 90.00th=[ 21103], 95.00th=[ 26608], | 99.00th=[ 42730], 99.50th=[ 55313], 99.90th=[100140], 99.95th=[295699], | 99.99th=[295699] bw ( KiB/s): min= 8, max= 392, per=24.93%, avg=243.31, stdev=66.42, samples=51 iops : min= 2, max= 98, avg=60.57, stdev=16.64, samples=51 lat (usec) : 100=5.08%, 250=7.71%, 500=0.78%, 750=4.65%, 1000=6.38% lat (msec) : 2=5.73%, 4=5.70%, 10=31.90%, 20=24.22%, 50=7.45% lat (msec) : 100=0.36%, 500=0.03% cpu : usr=0.21%, sys=16.30%, ctx=4894, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31332: Thu May 1 03:29:00 2025 read: IOPS=59, BW=239KiB/s (245kB/s)(6052KiB/25318msec) clat (usec): min=53, max=250862, avg=7309.77, stdev=9248.66 lat (usec): min=53, max=252507, avg=7313.58, stdev=9276.54 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 96], 10.00th=[ 105], 20.00th=[ 129], | 30.00th=[ 4015], 40.00th=[ 5211], 50.00th=[ 6194], 60.00th=[ 7242], | 70.00th=[ 8979], 80.00th=[ 11731], 90.00th=[ 15008], 95.00th=[ 18744], | 99.00th=[ 27132], 99.50th=[ 33162], 99.90th=[ 92799], 99.95th=[250610], | 99.99th=[250610] bw ( KiB/s): min= 56, max= 368, per=25.64%, avg=238.94, stdev=61.11, samples=50 iops : min= 14, max= 92, avg=59.56, stdev=15.25, samples=50 write: IOPS=61, BW=246KiB/s (252kB/s)(6236KiB/25318msec) clat (usec): min=487, max=103262, avg=9027.49, stdev=9422.98 lat (usec): min=487, max=103263, avg=9029.05, stdev=9423.07 clat percentiles (usec): | 1.00th=[ 635], 5.00th=[ 725], 10.00th=[ 766], 20.00th=[ 971], | 30.00th=[ 1729], 40.00th=[ 4080], 50.00th=[ 6980], 60.00th=[ 9110], | 70.00th=[ 11600], 80.00th=[ 15270], 90.00th=[ 21365], 95.00th=[ 27657], | 99.00th=[ 37487], 99.50th=[ 42206], 99.90th=[ 98042], 99.95th=[103285], | 99.99th=[103285] bw ( KiB/s): min= 80, max= 376, per=25.27%, avg=246.64, stdev=59.21, samples=50 iops : min= 20, max= 94, avg=61.48, stdev=14.86, samples=50 lat (usec) : 100=3.84%, 250=7.94%, 500=0.52%, 750=4.10%, 1000=6.35% lat (msec) : 2=5.83%, 4=6.18%, 10=34.34%, 20=23.14%, 50=7.49% lat (msec) : 100=0.20%, 250=0.03%, 500=0.03% cpu : usr=0.14%, sys=16.47%, ctx=4969, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=933KiB/s (955kB/s), 231KiB/s-239KiB/s (236kB/s-245kB/s), io=23.4MiB (24.6MB), run=25318-25743msec WRITE: bw=977KiB/s (1000kB/s), 244KiB/s-247KiB/s (250kB/s-253kB/s), io=24.6MiB (25.7MB), run=25318-25743msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31333: Thu May 1 03:30:17 2025 read: IOPS=14, BW=58.6KiB/s (60.1kB/s)(5992KiB/102165msec) clat (msec): min=2, max=249, avg=10.02, stdev= 9.74 lat (msec): min=2, max=249, avg=10.02, stdev= 9.74 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 12], 80.00th=[ 15], 90.00th=[ 19], 95.00th=[ 22], | 99.00th=[ 37], 99.50th=[ 45], 99.90th=[ 120], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min= 7, max= 144, per=25.06%, avg=58.65, stdev=28.95, samples=203 iops : min= 1, max= 36, avg=14.44, stdev= 7.25, samples=203 write: IOPS=15, BW=61.6KiB/s (63.1kB/s)(6296KiB/102165msec) clat (msec): min=21, max=149, avg=55.22, stdev=17.59 lat (msec): min=21, max=149, avg=55.22, stdev=17.59 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 32], 10.00th=[ 36], 20.00th=[ 41], | 30.00th=[ 45], 40.00th=[ 49], 50.00th=[ 53], 60.00th=[ 58], | 70.00th=[ 63], 80.00th=[ 69], 90.00th=[ 79], 95.00th=[ 88], | 99.00th=[ 110], 99.50th=[ 120], 99.90th=[ 146], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 31, max= 96, per=25.13%, avg=61.56, stdev=14.14, samples=203 iops : min= 7, max= 24, avg=15.17, stdev= 3.57, samples=203 lat (msec) : 4=4.04%, 10=27.41%, 20=13.80%, 50=25.62%, 100=28.09% lat (msec) : 250=1.04% cpu : usr=0.05%, sys=3.59%, ctx=3866, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31334: Thu May 1 03:30:17 2025 read: IOPS=14, BW=58.0KiB/s (59.4kB/s)(5940KiB/102339msec) clat (msec): min=2, max=258, avg= 9.93, stdev= 9.45 lat (msec): min=2, max=258, avg= 9.93, stdev= 9.45 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 12], 80.00th=[ 15], 90.00th=[ 19], 95.00th=[ 22], | 99.00th=[ 35], 99.50th=[ 45], 99.90th=[ 74], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 7, max= 160, per=24.94%, avg=58.36, stdev=28.75, samples=202 iops : min= 1, max= 40, avg=14.40, stdev= 7.23, samples=202 write: IOPS=15, BW=62.0KiB/s (63.5kB/s)(6348KiB/102339msec) clat (msec): min=21, max=148, avg=55.08, stdev=17.79 lat (msec): min=21, max=148, avg=55.09, stdev=17.79 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 32], 10.00th=[ 36], 20.00th=[ 41], | 30.00th=[ 45], 40.00th=[ 49], 50.00th=[ 53], 60.00th=[ 57], | 70.00th=[ 62], 80.00th=[ 68], 90.00th=[ 79], 95.00th=[ 89], | 99.00th=[ 110], 99.50th=[ 117], 99.90th=[ 146], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 8, max= 96, per=25.17%, avg=61.67, stdev=14.29, samples=204 iops : min= 2, max= 24, avg=15.23, stdev= 3.60, samples=204 lat (msec) : 4=3.42%, 10=28.84%, 20=12.43%, 50=26.30%, 100=27.60% lat (msec) : 250=1.37%, 500=0.03% cpu : usr=0.09%, sys=3.54%, ctx=3914, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31335: Thu May 1 03:30:17 2025 read: IOPS=14, BW=58.0KiB/s (60.4kB/s)(6028KiB/102217msec) clat (msec): min=2, max=110, avg= 9.88, stdev= 7.43 lat (msec): min=2, max=110, avg= 9.88, stdev= 7.43 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 12], 80.00th=[ 15], 90.00th=[ 20], 95.00th=[ 23], | 99.00th=[ 35], 99.50th=[ 41], 99.90th=[ 85], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 7, max= 160, per=25.31%, avg=59.22, stdev=28.57, samples=202 iops : min= 1, max= 40, avg=14.59, stdev= 7.19, samples=202 write: IOPS=15, BW=61.2KiB/s (62.7kB/s)(6260KiB/102217msec) clat (msec): min=21, max=294, avg=55.68, stdev=19.28 lat (msec): min=21, max=294, avg=55.68, stdev=19.28 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 32], 10.00th=[ 35], 20.00th=[ 41], | 30.00th=[ 45], 40.00th=[ 49], 50.00th=[ 53], 60.00th=[ 57], | 70.00th=[ 63], 80.00th=[ 70], 90.00th=[ 82], 95.00th=[ 91], | 99.00th=[ 114], 99.50th=[ 122], 99.90th=[ 148], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 7, max= 104, per=24.83%, avg=60.84, stdev=14.39, samples=204 iops : min= 1, max= 26, avg=14.99, stdev= 3.58, samples=204 lat (msec) : 4=4.49%, 10=27.77%, 20=12.66%, 50=26.56%, 100=27.18% lat (msec) : 250=1.30%, 500=0.03% cpu : usr=0.07%, sys=3.51%, ctx=3843, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31336: Thu May 1 03:30:17 2025 read: IOPS=14, BW=59.8KiB/s (61.2kB/s)(6052KiB/101251msec) clat (msec): min=2, max=110, avg= 9.80, stdev= 7.45 lat (msec): min=2, max=111, avg= 9.80, stdev= 7.45 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 12], 80.00th=[ 15], 90.00th=[ 19], 95.00th=[ 23], | 99.00th=[ 35], 99.50th=[ 40], 99.90th=[ 84], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 8, max= 152, per=25.86%, avg=60.52, stdev=27.59, samples=199 iops : min= 2, max= 38, avg=14.96, stdev= 6.90, samples=199 write: IOPS=15, BW=61.6KiB/s (63.1kB/s)(6236KiB/101251msec) clat (msec): min=21, max=316, avg=55.31, stdev=19.06 lat (msec): min=21, max=316, avg=55.31, stdev=19.06 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 32], 10.00th=[ 35], 20.00th=[ 41], | 30.00th=[ 45], 40.00th=[ 49], 50.00th=[ 53], 60.00th=[ 57], | 70.00th=[ 63], 80.00th=[ 69], 90.00th=[ 80], 95.00th=[ 88], | 99.00th=[ 115], 99.50th=[ 121], 99.90th=[ 169], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 8, max= 95, per=25.03%, avg=61.32, stdev=14.34, samples=202 iops : min= 2, max= 23, avg=15.16, stdev= 3.60, samples=202 lat (msec) : 4=3.68%, 10=28.45%, 20=13.25%, 50=26.24%, 100=27.28% lat (msec) : 250=1.07%, 500=0.03% cpu : usr=0.03%, sys=3.61%, ctx=3893, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=235KiB/s (240kB/s), 58.0KiB/s-59.8KiB/s (59.4kB/s-61.2kB/s), io=23.4MiB (24.6MB), run=101251-102339msec WRITE: bw=246KiB/s (252kB/s), 61.2KiB/s-62.0KiB/s (62.7kB/s-63.5kB/s), io=24.6MiB (25.7MB), run=101251-102339msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31429: Thu May 1 03:30:24 2025 read: IOPS=57, BW=926KiB/s (949kB/s)(5920KiB/6390msec) clat (usec): min=53, max=78837, avg=7988.91, stdev=8915.89 lat (usec): min=53, max=78837, avg=7989.94, stdev=8915.91 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 94], 10.00th=[ 101], 20.00th=[ 119], | 30.00th=[ 229], 40.00th=[ 5932], 50.00th=[ 7111], 60.00th=[ 8160], | 70.00th=[10159], 80.00th=[12780], 90.00th=[16909], 95.00th=[20317], | 99.00th=[43254], 99.50th=[70779], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 544, max= 1440, per=24.35%, avg=912.75, stdev=247.05, samples=12 iops : min= 34, max= 90, avg=56.75, stdev=15.40, samples=12 write: IOPS=62, BW=997KiB/s (1020kB/s)(6368KiB/6390msec) clat (usec): min=633, max=95038, avg=8421.93, stdev=10628.09 lat (usec): min=633, max=95040, avg=8424.49, stdev=10629.78 clat percentiles (usec): | 1.00th=[ 848], 5.00th=[ 1074], 10.00th=[ 1123], 20.00th=[ 1270], | 30.00th=[ 1631], 40.00th=[ 2278], 50.00th=[ 3982], 60.00th=[ 7111], | 70.00th=[10159], 80.00th=[13829], 90.00th=[20317], 95.00th=[27132], | 99.00th=[52691], 99.50th=[72877], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 544, max= 1408, per=25.03%, avg=986.83, stdev=281.11, samples=12 iops : min= 34, max= 88, avg=61.33, stdev=17.59, samples=12 lat (usec) : 100=4.43%, 250=10.16%, 500=0.39%, 750=0.13%, 1000=1.30% lat (msec) : 2=18.62%, 4=6.64%, 10=27.60%, 20=22.66%, 50=7.03% lat (msec) : 100=1.04% cpu : usr=0.05%, sys=16.87%, ctx=1184, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31431: Thu May 1 03:30:24 2025 read: IOPS=60, BW=974KiB/s (998kB/s)(5744KiB/5895msec) clat (usec): min=72, max=80290, avg=7842.45, stdev=8317.27 lat (usec): min=73, max=80292, avg=7853.32, stdev=8348.20 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 97], 10.00th=[ 102], 20.00th=[ 123], | 30.00th=[ 1975], 40.00th=[ 5604], 50.00th=[ 7308], 60.00th=[ 8455], | 70.00th=[10159], 80.00th=[12256], 90.00th=[15401], 95.00th=[19792], | 99.00th=[45876], 99.50th=[50070], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 702, max= 1504, per=26.67%, avg=999.55, stdev=276.93, samples=11 iops : min= 43, max= 94, avg=62.09, stdev=17.53, samples=11 write: IOPS=69, BW=1110KiB/s (1137kB/s)(6544KiB/5895msec) clat (usec): min=907, max=76003, avg=7373.46, stdev=8376.50 lat (usec): min=908, max=76004, avg=7376.15, stdev=8376.52 clat percentiles (usec): | 1.00th=[ 938], 5.00th=[ 1045], 10.00th=[ 1106], 20.00th=[ 1319], | 30.00th=[ 1729], 40.00th=[ 2474], 50.00th=[ 3621], 60.00th=[ 6128], | 70.00th=[ 8979], 80.00th=[13304], 90.00th=[19268], 95.00th=[22676], | 99.00th=[33817], 99.50th=[40109], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 798, max= 1341, per=28.30%, avg=1115.73, stdev=189.39, samples=11 iops : min= 49, max= 83, avg=69.36, stdev=12.07, samples=11 lat (usec) : 100=3.78%, 250=8.85%, 500=0.26%, 750=0.52%, 1000=1.69% lat (msec) : 2=17.32%, 4=9.64%, 10=29.17%, 20=22.40%, 50=5.99% lat (msec) : 100=0.39% cpu : usr=0.12%, sys=18.04%, ctx=1144, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31434: Thu May 1 03:30:24 2025 read: IOPS=63, BW=1014KiB/s (1038kB/s)(6256KiB/6171msec) clat (usec): min=65, max=92370, avg=7959.64, stdev=8604.15 lat (usec): min=66, max=92371, avg=7961.13, stdev=8604.36 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 92], 10.00th=[ 101], 20.00th=[ 123], | 30.00th=[ 1139], 40.00th=[ 5473], 50.00th=[ 7439], 60.00th=[ 8848], | 70.00th=[10814], 80.00th=[12649], 90.00th=[16909], 95.00th=[19530], | 99.00th=[28181], 99.50th=[77071], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 704, max= 1376, per=27.58%, avg=1033.75, stdev=210.77, samples=12 iops : min= 44, max= 86, avg=64.33, stdev=13.34, samples=12 write: IOPS=61, BW=977KiB/s (1001kB/s)(6032KiB/6171msec) clat (usec): min=675, max=75237, avg=8002.39, stdev=9833.11 lat (usec): min=676, max=75239, avg=8003.87, stdev=9833.16 clat percentiles (usec): | 1.00th=[ 857], 5.00th=[ 971], 10.00th=[ 1106], 20.00th=[ 1237], | 30.00th=[ 1565], 40.00th=[ 2343], 50.00th=[ 4228], 60.00th=[ 6521], | 70.00th=[10028], 80.00th=[14222], 90.00th=[18744], 95.00th=[22676], | 99.00th=[54789], 99.50th=[73925], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 640, max= 1341, per=24.86%, avg=980.25, stdev=171.51, samples=12 iops : min= 40, max= 83, avg=61.00, stdev=10.55, samples=12 lat (usec) : 100=4.95%, 250=9.51%, 500=0.39%, 750=0.65%, 1000=2.60% lat (msec) : 2=15.23%, 4=7.55%, 10=27.34%, 20=25.39%, 50=5.60% lat (msec) : 100=0.78% cpu : usr=0.03%, sys=17.07%, ctx=1209, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31435: Thu May 1 03:30:24 2025 read: IOPS=59, BW=950KiB/s (973kB/s)(6032KiB/6351msec) clat (usec): min=71, max=83467, avg=8627.32, stdev=8725.74 lat (usec): min=72, max=83467, avg=8628.27, stdev=8725.77 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 94], 10.00th=[ 105], 20.00th=[ 126], | 30.00th=[ 4490], 40.00th=[ 6521], 50.00th=[ 7701], 60.00th=[ 8979], | 70.00th=[10421], 80.00th=[13435], 90.00th=[19006], 95.00th=[22676], | 99.00th=[39060], 99.50th=[66323], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 704, max= 1179, per=25.28%, avg=947.58, stdev=168.24, samples=12 iops : min= 44, max= 73, avg=59.00, stdev=10.53, samples=12 write: IOPS=61, BW=985KiB/s (1009kB/s)(6256KiB/6351msec) clat (usec): min=669, max=62853, avg=7539.15, stdev=8657.04 lat (usec): min=672, max=62855, avg=7540.73, stdev=8657.03 clat percentiles (usec): | 1.00th=[ 848], 5.00th=[ 988], 10.00th=[ 1090], 20.00th=[ 1221], | 30.00th=[ 1516], 40.00th=[ 2040], 50.00th=[ 3556], 60.00th=[ 6521], | 70.00th=[ 9372], 80.00th=[13960], 90.00th=[18744], 95.00th=[23200], | 99.00th=[43254], 99.50th=[54264], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 638, max= 1593, per=25.04%, avg=987.33, stdev=265.97, samples=12 iops : min= 39, max= 99, avg=61.50, stdev=16.63, samples=12 lat (usec) : 100=4.17%, 250=8.33%, 500=0.52%, 750=0.13%, 1000=2.73% lat (msec) : 2=17.84%, 4=6.64%, 10=28.52%, 20=22.40%, 50=8.20% lat (msec) : 100=0.52% cpu : usr=0.06%, sys=16.55%, ctx=1177, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3748KiB/s (3838kB/s), 926KiB/s-1014KiB/s (949kB/s-1038kB/s), io=23.4MiB (24.5MB), run=5895-6390msec WRITE: bw=3944KiB/s (4038kB/s), 977KiB/s-1110KiB/s (1001kB/s-1137kB/s), io=24.6MiB (25.8MB), run=5895-6390msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31430: Thu May 1 03:30:43 2025 read: IOPS=14, BW=238KiB/s (244kB/s)(5920KiB/24840msec) clat (usec): min=2604, max=99616, avg=8464.32, stdev=8731.77 lat (usec): min=2605, max=99617, avg=8466.04, stdev=8731.66 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 16], 95.00th=[ 22], | 99.00th=[ 44], 99.50th=[ 82], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 64, max= 542, per=24.21%, avg=233.43, stdev=114.42, samples=49 iops : min= 4, max= 33, avg=14.33, stdev= 7.10, samples=49 write: IOPS=16, BW=256KiB/s (263kB/s)(6368KiB/24840msec) clat (msec): min=20, max=132, avg=54.09, stdev=21.37 lat (msec): min=20, max=132, avg=54.09, stdev=21.37 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 36], | 30.00th=[ 41], 40.00th=[ 45], 50.00th=[ 50], 60.00th=[ 55], | 70.00th=[ 63], 80.00th=[ 70], 90.00th=[ 84], 95.00th=[ 99], | 99.00th=[ 123], 99.50th=[ 125], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 95, max= 448, per=25.09%, avg=254.37, stdev=83.35, samples=49 iops : min= 5, max= 28, avg=15.63, stdev= 5.26, samples=49 lat (msec) : 4=5.47%, 10=34.37%, 20=5.60%, 50=28.52%, 100=23.70% lat (msec) : 250=2.34% cpu : usr=0.03%, sys=3.71%, ctx=907, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31432: Thu May 1 03:30:43 2025 read: IOPS=14, BW=231KiB/s (237kB/s)(5744KiB/24813msec) clat (usec): min=2754, max=90642, avg=8878.15, stdev=9841.13 lat (usec): min=2755, max=90644, avg=8879.63, stdev=9840.98 clat percentiles (usec): | 1.00th=[ 3097], 5.00th=[ 3523], 10.00th=[ 3851], 20.00th=[ 4359], | 30.00th=[ 4883], 40.00th=[ 5473], 50.00th=[ 6063], 60.00th=[ 6783], | 70.00th=[ 7635], 80.00th=[ 9241], 90.00th=[14484], 95.00th=[25560], | 99.00th=[61080], 99.50th=[64750], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 31, max= 512, per=23.79%, avg=229.35, stdev=133.76, samples=49 iops : min= 1, max= 32, avg=13.90, stdev= 8.41, samples=49 write: IOPS=16, BW=264KiB/s (270kB/s)(6544KiB/24813msec) clat (msec): min=20, max=139, avg=52.71, stdev=21.70 lat (msec): min=20, max=139, avg=52.71, stdev=21.70 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 33], | 30.00th=[ 39], 40.00th=[ 45], 50.00th=[ 50], 60.00th=[ 54], | 70.00th=[ 62], 80.00th=[ 70], 90.00th=[ 84], 95.00th=[ 94], | 99.00th=[ 117], 99.50th=[ 125], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 127, max= 479, per=25.71%, avg=260.69, stdev=91.57, samples=49 iops : min= 7, max= 29, avg=15.86, stdev= 5.82, samples=49 lat (msec) : 4=6.12%, 10=32.29%, 20=5.21%, 50=30.47%, 100=24.35% lat (msec) : 250=1.56% cpu : usr=0.06%, sys=3.67%, ctx=911, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31433: Thu May 1 03:30:43 2025 read: IOPS=16, BW=261KiB/s (268kB/s)(6256KiB/23943msec) clat (usec): min=2996, max=90446, avg=8667.11, stdev=8722.55 lat (usec): min=2997, max=90447, avg=8668.53, stdev=8722.51 clat percentiles (usec): | 1.00th=[ 3392], 5.00th=[ 3818], 10.00th=[ 4146], 20.00th=[ 4621], | 30.00th=[ 5145], 40.00th=[ 5604], 50.00th=[ 6063], 60.00th=[ 6915], | 70.00th=[ 7767], 80.00th=[ 9241], 90.00th=[13698], 95.00th=[26608], | 99.00th=[52167], 99.50th=[73925], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 63, max= 702, per=26.95%, avg=259.77, stdev=141.72, samples=47 iops : min= 3, max= 43, avg=15.96, stdev= 8.86, samples=47 write: IOPS=15, BW=252KiB/s (258kB/s)(6032KiB/23943msec) clat (msec): min=17, max=117, avg=54.44, stdev=21.29 lat (msec): min=17, max=117, avg=54.45, stdev=21.29 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 26], 10.00th=[ 31], 20.00th=[ 37], | 30.00th=[ 42], 40.00th=[ 46], 50.00th=[ 52], 60.00th=[ 57], | 70.00th=[ 64], 80.00th=[ 71], 90.00th=[ 82], 95.00th=[ 97], | 99.00th=[ 117], 99.50th=[ 117], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 96, max= 510, per=24.41%, avg=247.51, stdev=78.46, samples=47 iops : min= 6, max= 31, avg=15.19, stdev= 4.94, samples=47 lat (msec) : 4=3.65%, 10=38.41%, 20=6.12%, 50=26.04%, 100=23.70% lat (msec) : 250=2.08% cpu : usr=0.11%, sys=3.72%, ctx=900, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31436: Thu May 1 03:30:43 2025 read: IOPS=15, BW=247KiB/s (253kB/s)(6032KiB/24385msec) clat (usec): min=2913, max=73802, avg=8707.18, stdev=8499.61 lat (usec): min=2914, max=73803, avg=8709.24, stdev=8499.31 clat percentiles (usec): | 1.00th=[ 3097], 5.00th=[ 3490], 10.00th=[ 3916], 20.00th=[ 4555], | 30.00th=[ 5014], 40.00th=[ 5604], 50.00th=[ 6194], 60.00th=[ 6915], | 70.00th=[ 7767], 80.00th=[ 9765], 90.00th=[15008], 95.00th=[23725], | 99.00th=[52167], 99.50th=[66323], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 31, max= 606, per=26.03%, avg=250.89, stdev=129.15, samples=47 iops : min= 1, max= 37, avg=15.38, stdev= 8.09, samples=47 write: IOPS=16, BW=257KiB/s (263kB/s)(6256KiB/24385msec) clat (msec): min=18, max=166, avg=53.51, stdev=21.53 lat (msec): min=18, max=166, avg=53.52, stdev=21.53 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 27], 10.00th=[ 31], 20.00th=[ 35], | 30.00th=[ 40], 40.00th=[ 46], 50.00th=[ 50], 60.00th=[ 55], | 70.00th=[ 63], 80.00th=[ 70], 90.00th=[ 80], 95.00th=[ 100], | 99.00th=[ 117], 99.50th=[ 138], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 95, max= 448, per=25.08%, avg=254.29, stdev=88.11, samples=48 iops : min= 5, max= 28, avg=15.58, stdev= 5.58, samples=48 lat (msec) : 4=5.34%, 10=34.24%, 20=6.38%, 50=28.91%, 100=22.79% lat (msec) : 250=2.34% cpu : usr=0.03%, sys=3.76%, ctx=924, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=964KiB/s (987kB/s), 231KiB/s-261KiB/s (237kB/s-268kB/s), io=23.4MiB (24.5MB), run=23943-24840msec WRITE: bw=1014KiB/s (1039kB/s), 252KiB/s-264KiB/s (258kB/s-270kB/s), io=24.6MiB (25.8MB), run=23943-24840msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31470: Thu May 1 03:30:45 2025 read: IOPS=9, BW=9673KiB/s (9906kB/s)(7168KiB/741msec) clat (usec): min=419, max=68117, avg=45523.29, stdev=21919.15 lat (usec): min=422, max=68118, avg=45525.07, stdev=21918.95 clat percentiles (usec): | 1.00th=[ 420], 5.00th=[ 420], 10.00th=[ 420], 20.00th=[38011], | 30.00th=[49546], 40.00th=[49546], 50.00th=[50594], 60.00th=[52691], | 70.00th=[52691], 80.00th=[58983], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min=10240, max=10240, per=49.90%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6910KiB/s (7075kB/s)(5120KiB/741msec) clat (msec): min=44, max=120, avg=80.96, stdev=29.40 lat (msec): min=44, max=120, avg=81.02, stdev=29.40 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 45], | 30.00th=[ 66], 40.00th=[ 66], 50.00th=[ 75], 60.00th=[ 75], | 70.00th=[ 99], 80.00th=[ 99], 90.00th=[ 122], 95.00th=[ 122], | 99.00th=[ 122], 99.50th=[ 122], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 6144, max= 6144, per=21.39%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=25.00%, 100=58.33%, 250=8.33% cpu : usr=0.14%, sys=29.32%, ctx=397, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31471: Thu May 1 03:30:45 2025 read: IOPS=2, BW=2656KiB/s (2720kB/s)(2048KiB/771msec) clat (usec): min=57393, max=72232, avg=64813.09, stdev=10492.97 lat (usec): min=57395, max=72234, avg=64815.04, stdev=10492.37 clat percentiles (usec): | 1.00th=[57410], 5.00th=[57410], 10.00th=[57410], 20.00th=[57410], | 30.00th=[57410], 40.00th=[57410], 50.00th=[57410], 60.00th=[71828], | 70.00th=[71828], 80.00th=[71828], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 4087, max= 4087, per=19.92%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.0MiB/s (13.6MB/s)(10.0MiB/771msec) clat (msec): min=21, max=121, avg=59.23, stdev=40.43 lat (msec): min=21, max=121, avg=59.31, stdev=40.44 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 26], 40.00th=[ 29], 50.00th=[ 39], 60.00th=[ 54], | 70.00th=[ 102], 80.00th=[ 102], 90.00th=[ 120], 95.00th=[ 123], | 99.00th=[ 123], 99.50th=[ 123], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min=10219, max=10219, per=35.57%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=33.33%, 250=25.00% cpu : usr=0.00%, sys=36.10%, ctx=304, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31472: Thu May 1 03:30:45 2025 read: IOPS=6, BW=6156KiB/s (6304kB/s)(6144KiB/998msec) clat (usec): min=26880, max=93431, avg=56464.67, stdev=28849.41 lat (usec): min=26881, max=93432, avg=56466.36, stdev=28849.49 clat percentiles (usec): | 1.00th=[26870], 5.00th=[26870], 10.00th=[26870], 20.00th=[35914], | 30.00th=[35914], 40.00th=[40633], 50.00th=[40633], 60.00th=[51119], | 70.00th=[90702], 80.00th=[90702], 90.00th=[93848], 95.00th=[93848], | 99.00th=[93848], 99.50th=[93848], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] write: IOPS=6, BW=6156KiB/s (6304kB/s)(6144KiB/998msec) clat (msec): min=36, max=292, avg=88.38, stdev=100.36 lat (msec): min=36, max=292, avg=88.58, stdev=100.29 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 55], | 70.00th=[ 61], 80.00th=[ 61], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 4096, max= 4096, per=14.26%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=41.67%, 500=8.33% cpu : usr=0.00%, sys=23.27%, ctx=566, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31473: Thu May 1 03:30:45 2025 read: IOPS=5, BW=5187KiB/s (5312kB/s)(5120KiB/987msec) clat (usec): min=52101, max=73668, avg=64459.36, stdev=7866.64 lat (usec): min=52103, max=73669, avg=64460.95, stdev=7866.67 clat percentiles (usec): | 1.00th=[52167], 5.00th=[52167], 10.00th=[52167], 20.00th=[52167], | 30.00th=[63177], 40.00th=[63177], 50.00th=[66323], 60.00th=[66323], | 70.00th=[66847], 80.00th=[66847], 90.00th=[73925], 95.00th=[73925], | 99.00th=[73925], 99.50th=[73925], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] write: IOPS=7, BW=7262KiB/s (7437kB/s)(7168KiB/987msec) clat (msec): min=29, max=185, avg=84.04, stdev=56.51 lat (msec): min=29, max=185, avg=84.10, stdev=56.50 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 32], | 30.00th=[ 63], 40.00th=[ 63], 50.00th=[ 71], 60.00th=[ 77], | 70.00th=[ 77], 80.00th=[ 134], 90.00th=[ 186], 95.00th=[ 186], | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 8192, max= 8192, per=28.51%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=16.67%, 100=66.67%, 250=16.67% cpu : usr=0.00%, sys=25.56%, ctx=265, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=20.0MiB/s (21.0MB/s), 2656KiB/s-9673KiB/s (2720kB/s-9906kB/s), io=20.0MiB (20.0MB), run=741-998msec WRITE: bw=28.1MiB/s (29.4MB/s), 6156KiB/s-12.0MiB/s (6304kB/s-13.6MB/s), io=28.0MiB (29.4MB), run=741-998msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31466: Thu May 1 03:30:45 2025 read: IOPS=6, BW=6179KiB/s (6328kB/s)(7168KiB/1160msec) clat (msec): min=23, max=291, avg=77.62, stdev=95.21 lat (msec): min=23, max=291, avg=77.62, stdev=95.21 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 34], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 46], 60.00th=[ 56], | 70.00th=[ 56], 80.00th=[ 59], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 5456, max= 6144, per=36.36%, avg=5800.00, stdev=486.49, samples=2 iops : min= 5, max= 6, avg= 5.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4414KiB/s (4520kB/s)(5120KiB/1160msec) clat (msec): min=73, max=134, avg=102.12, stdev=26.00 lat (msec): min=73, max=135, avg=102.25, stdev=26.08 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 73], 10.00th=[ 73], 20.00th=[ 73], | 30.00th=[ 81], 40.00th=[ 81], 50.00th=[ 102], 60.00th=[ 102], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 7275, max= 7275, per=32.58%, avg=7275.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=33.33%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=9.15%, ctx=59, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31467: Thu May 1 03:30:45 2025 read: IOPS=1, BW=1595KiB/s (1633kB/s)(2048KiB/1284msec) clat (usec): min=43380, max=51946, avg=47663.44, stdev=6057.33 lat (usec): min=43381, max=51948, avg=47665.17, stdev=6057.91 clat percentiles (usec): | 1.00th=[43254], 5.00th=[43254], 10.00th=[43254], 20.00th=[43254], | 30.00th=[43254], 40.00th=[43254], 50.00th=[43254], 60.00th=[52167], | 70.00th=[52167], 80.00th=[52167], 90.00th=[52167], 95.00th=[52167], | 99.00th=[52167], 99.50th=[52167], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 1783, max= 2048, per=12.01%, avg=1915.50, stdev=187.38, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=7, BW=7975KiB/s (8166kB/s)(10.0MiB/1284msec) clat (msec): min=45, max=199, avg=108.27, stdev=57.32 lat (msec): min=45, max=199, avg=108.35, stdev=57.33 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 51], | 30.00th=[ 62], 40.00th=[ 64], 50.00th=[ 86], 60.00th=[ 105], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 190], 95.00th=[ 201], | 99.00th=[ 201], 99.50th=[ 201], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 4096, max= 7135, per=25.15%, avg=5615.50, stdev=2148.90, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 50=16.67%, 100=41.67%, 250=41.67% cpu : usr=0.00%, sys=9.43%, ctx=55, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31468: Thu May 1 03:30:45 2025 read: IOPS=5, BW=6138KiB/s (6285kB/s)(6144KiB/1001msec) clat (msec): min=31, max=107, avg=67.54, stdev=31.78 lat (msec): min=31, max=107, avg=67.54, stdev=31.78 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 53], 50.00th=[ 53], 60.00th=[ 71], | 70.00th=[ 103], 80.00th=[ 103], 90.00th=[ 108], 95.00th=[ 108], | 99.00th=[ 108], 99.50th=[ 108], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 4096, max= 6144, per=32.10%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=5, BW=6138KiB/s (6285kB/s)(6144KiB/1001msec) clat (msec): min=62, max=152, avg=95.30, stdev=34.18 lat (msec): min=62, max=152, avg=95.39, stdev=34.18 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 69], | 30.00th=[ 69], 40.00th=[ 78], 50.00th=[ 78], 60.00th=[ 94], | 70.00th=[ 118], 80.00th=[ 118], 90.00th=[ 153], 95.00th=[ 153], | 99.00th=[ 153], 99.50th=[ 153], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 6144, max= 6144, per=27.51%, avg=6144.00, stdev= 0.00, samples=2 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=2 lat (msec) : 50=16.67%, 100=50.00%, 250=33.33% cpu : usr=0.00%, sys=7.20%, ctx=30, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31469: Thu May 1 03:30:45 2025 read: IOPS=5, BW=5740KiB/s (5878kB/s)(5120KiB/892msec) clat (usec): min=40186, max=72578, avg=53896.25, stdev=16909.47 lat (usec): min=40187, max=72579, avg=53897.65, stdev=16908.89 clat percentiles (usec): | 1.00th=[40109], 5.00th=[40109], 10.00th=[40109], 20.00th=[40109], | 30.00th=[40633], 40.00th=[40633], 50.00th=[43779], 60.00th=[43779], | 70.00th=[71828], 80.00th=[71828], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 2048, max= 2048, per=12.84%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=7, BW=8036KiB/s (8229kB/s)(7168KiB/892msec) clat (msec): min=46, max=118, avg=84.24, stdev=23.93 lat (msec): min=47, max=118, avg=84.34, stdev=23.93 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 67], | 30.00th=[ 79], 40.00th=[ 79], 50.00th=[ 85], 60.00th=[ 90], | 70.00th=[ 90], 80.00th=[ 107], 90.00th=[ 120], 95.00th=[ 120], | 99.00th=[ 120], 99.50th=[ 120], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 8192, max= 8192, per=36.69%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=50.00%, 250=16.67% cpu : usr=0.00%, sys=10.89%, ctx=35, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.6MiB/s (16.3MB/s), 1595KiB/s-6179KiB/s (1633kB/s-6328kB/s), io=20.0MiB (20.0MB), run=892-1284msec WRITE: bw=21.8MiB/s (22.9MB/s), 4414KiB/s-8036KiB/s (4520kB/s-8229kB/s), io=28.0MiB (29.4MB), run=892-1284msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31480: Thu May 1 03:30:47 2025 read: IOPS=5, BW=21.7MiB/s (22.8MB/s)(12.0MiB/552msec) clat (msec): min=147, max=186, avg=167.66, stdev=19.60 lat (msec): min=147, max=186, avg=167.66, stdev=19.60 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 148], 10.00th=[ 148], 20.00th=[ 148], | 30.00th=[ 148], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 169], | 70.00th=[ 186], 80.00th=[ 186], 90.00th=[ 186], 95.00th=[ 186], | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min=16351, max=16351, per=100.00%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=33.94%, ctx=1934, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31482: Thu May 1 03:30:47 2025 read: IOPS=0, BW=3468KiB/s (3551kB/s)(4096KiB/1181msec) clat (nsec): min=227580k, max=227580k, avg=227580251.00, stdev= 0.00 lat (nsec): min=227583k, max=227583k, avg=227583232.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 228], 10.00th=[ 228], 20.00th=[ 228], | 30.00th=[ 228], 40.00th=[ 228], 50.00th=[ 228], 60.00th=[ 228], | 70.00th=[ 228], 80.00th=[ 228], 90.00th=[ 228], 95.00th=[ 228], | 99.00th=[ 228], 99.50th=[ 228], 99.90th=[ 228], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 8175, max= 8175, per=58.93%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6936KiB/s (7103kB/s)(8192KiB/1181msec) clat (msec): min=360, max=560, avg=460.83, stdev=141.62 lat (msec): min=360, max=561, avg=461.19, stdev=141.71 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 359], | 30.00th=[ 359], 40.00th=[ 359], 50.00th=[ 359], 60.00th=[ 558], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 8094, max= 8094, per=29.17%, avg=8094.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.08%, sys=22.12%, ctx=762, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31485: Thu May 1 03:30:47 2025 write: IOPS=2, BW=10.8MiB/s (11.4MB/s)(12.0MiB/1106msec) clat (msec): min=123, max=476, avg=343.48, stdev=192.16 lat (msec): min=123, max=476, avg=343.84, stdev=192.17 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 124], 10.00th=[ 124], 20.00th=[ 124], | 30.00th=[ 124], 40.00th=[ 430], 50.00th=[ 430], 60.00th=[ 430], | 70.00th=[ 477], 80.00th=[ 477], 90.00th=[ 477], 95.00th=[ 477], | 99.00th=[ 477], 99.50th=[ 477], 99.90th=[ 477], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min=16094, max=16094, per=58.01%, avg=16094.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.18%, sys=25.70%, ctx=94, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31486: Thu May 1 03:30:47 2025 write: IOPS=4, BW=17.0MiB/s (18.9MB/s)(12.0MiB/667msec) clat (msec): min=186, max=241, avg=214.42, stdev=27.44 lat (msec): min=186, max=241, avg=214.77, stdev=27.45 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 186], 10.00th=[ 186], 20.00th=[ 186], | 30.00th=[ 186], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 215], | 70.00th=[ 243], 80.00th=[ 243], 90.00th=[ 243], 95.00th=[ 243], | 99.00th=[ 243], 99.50th=[ 243], 99.90th=[ 243], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min=16318, max=16318, per=58.81%, avg=16318.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=40.24%, ctx=57, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.5MiB/s (14.2MB/s), 3468KiB/s-21.7MiB/s (3551kB/s-22.8MB/s), io=16.0MiB (16.8MB), run=552-1181msec WRITE: bw=27.1MiB/s (28.4MB/s), 6936KiB/s-17.0MiB/s (7103kB/s-18.9MB/s), io=32.0MiB (33.6MB), run=667-1181msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31481: Thu May 1 03:30:48 2025 read: IOPS=3, BW=13.5MiB/s (14.2MB/s)(12.0MiB/888msec) clat (msec): min=233, max=390, avg=286.92, stdev=89.79 lat (msec): min=233, max=390, avg=286.92, stdev=89.79 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 234], 10.00th=[ 234], 20.00th=[ 234], | 30.00th=[ 234], 40.00th=[ 236], 50.00th=[ 236], 60.00th=[ 236], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.11%, sys=7.55%, ctx=29, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31483: Thu May 1 03:30:48 2025 read: IOPS=0, BW=3141KiB/s (3216kB/s)(4096KiB/1304msec) clat (nsec): min=93379k, max=93379k, avg=93378826.00, stdev= 0.00 lat (nsec): min=93382k, max=93382k, avg=93381548.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[93848], 5.00th=[93848], 10.00th=[93848], 20.00th=[93848], | 30.00th=[93848], 40.00th=[93848], 50.00th=[93848], 60.00th=[93848], | 70.00th=[93848], 80.00th=[93848], 90.00th=[93848], 95.00th=[93848], | 99.00th=[93848], 99.50th=[93848], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 8175, max= 8175, per=65.07%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6282KiB/s (6433kB/s)(8192KiB/1304msec) clat (msec): min=560, max=630, avg=595.26, stdev=49.17 lat (msec): min=560, max=630, avg=595.63, stdev=49.16 clat percentiles (msec): | 1.00th=[ 558], 5.00th=[ 558], 10.00th=[ 558], 20.00th=[ 558], | 30.00th=[ 558], 40.00th=[ 558], 50.00th=[ 558], 60.00th=[ 634], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 8192, max= 8192, per=32.60%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=66.67% cpu : usr=0.00%, sys=8.98%, ctx=75, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31484: Thu May 1 03:30:48 2025 write: IOPS=3, BW=12.3MiB/s (12.9MB/s)(12.0MiB/974msec) clat (msec): min=302, max=307, avg=304.95, stdev= 2.51 lat (msec): min=302, max=307, avg=305.28, stdev= 2.55 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 305], 10.00th=[ 305], 20.00th=[ 305], | 30.00th=[ 305], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 305], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 8192, max= 8192, per=32.60%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=100.00% cpu : usr=0.21%, sys=7.09%, ctx=43, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31487: Thu May 1 03:30:48 2025 write: IOPS=3, BW=12.4MiB/s (13.0MB/s)(12.0MiB/965msec) clat (msec): min=134, max=465, avg=310.42, stdev=166.73 lat (msec): min=134, max=465, avg=310.60, stdev=166.72 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 330], 50.00th=[ 330], 60.00th=[ 330], | 70.00th=[ 468], 80.00th=[ 468], 90.00th=[ 468], 95.00th=[ 468], | 99.00th=[ 468], 99.50th=[ 468], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 8175, max= 8175, per=32.53%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.10%, sys=10.58%, ctx=42, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.3MiB/s (12.9MB/s), 3141KiB/s-13.5MiB/s (3216kB/s-14.2MB/s), io=16.0MiB (16.8MB), run=888-1304msec WRITE: bw=24.5MiB/s (25.7MB/s), 6282KiB/s-12.4MiB/s (6433kB/s-13.0MB/s), io=32.0MiB (33.6MB), run=965-1304msec