== sanity test 398b: DIO and buffer IO race ============== 20:47:06 (1746146826) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 3.51496 s, 14.3 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32244: Thu May 1 20:48:15 2025 read: IOPS=24, BW=98.8KiB/s (101kB/s)(5992KiB/60672msec) clat (usec): min=68, max=546133, avg=18104.48, stdev=20748.43 lat (usec): min=69, max=546135, avg=18107.25, stdev=20749.21 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 108], 10.00th=[ 122], 20.00th=[ 938], | 30.00th=[ 9765], 40.00th=[ 13042], 50.00th=[ 15795], 60.00th=[ 18744], | 70.00th=[ 22938], 80.00th=[ 27919], 90.00th=[ 35914], 95.00th=[ 44827], | 99.00th=[ 67634], 99.50th=[ 80217], 99.90th=[198181], 99.95th=[549454], | 99.99th=[549454] bw ( KiB/s): min= 16, max= 199, per=25.20%, avg=99.03, stdev=35.53, samples=120 iops : min= 4, max= 49, avg=24.38, stdev= 8.80, samples=120 write: IOPS=25, BW=104KiB/s (106kB/s)(6296KiB/60672msec) clat (usec): min=471, max=365987, avg=21078.78, stdev=25033.44 lat (usec): min=472, max=365989, avg=21082.41, stdev=25039.16 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 857], 10.00th=[ 1012], 20.00th=[ 1614], | 30.00th=[ 4080], 40.00th=[ 8717], 50.00th=[ 14877], 60.00th=[ 21103], | 70.00th=[ 27395], 80.00th=[ 35914], 90.00th=[ 47973], 95.00th=[ 61080], | 99.00th=[ 99091], 99.50th=[123208], 99.90th=[333448], 99.95th=[367002], | 99.99th=[367002] bw ( KiB/s): min= 15, max= 176, per=25.28%, avg=103.89, stdev=33.76, samples=120 iops : min= 3, max= 44, avg=25.60, stdev= 8.46, samples=120 lat (usec) : 100=1.43%, 250=6.84%, 500=0.98%, 750=0.98%, 1000=4.43% lat (msec) : 2=6.77%, 4=4.39%, 10=10.68%, 20=24.41%, 50=33.14% lat (msec) : 100=5.34%, 250=0.49%, 500=0.10%, 750=0.03% cpu : usr=0.12%, sys=9.24%, ctx=5137, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32246: Thu May 1 20:48:15 2025 read: IOPS=24, BW=97.4KiB/s (99.7kB/s)(5940KiB/61003msec) clat (usec): min=54, max=310999, avg=17473.94, stdev=17453.92 lat (usec): min=54, max=310999, avg=17475.60, stdev=17454.00 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 104], 10.00th=[ 121], 20.00th=[ 334], | 30.00th=[ 9372], 40.00th=[ 12256], 50.00th=[ 15008], 60.00th=[ 18482], | 70.00th=[ 22414], 80.00th=[ 28181], 90.00th=[ 35914], 95.00th=[ 42730], | 99.00th=[ 64750], 99.50th=[ 73925], 99.90th=[214959], 99.95th=[312476], | 99.99th=[312476] bw ( KiB/s): min= 7, max= 200, per=24.51%, avg=96.31, stdev=37.16, samples=121 iops : min= 1, max= 50, avg=23.74, stdev= 9.30, samples=121 write: IOPS=26, BW=104KiB/s (107kB/s)(6348KiB/61003msec) clat (usec): min=606, max=516051, avg=21843.32, stdev=29297.89 lat (usec): min=608, max=516053, avg=21845.18, stdev=29297.84 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 857], 10.00th=[ 1037], 20.00th=[ 1549], | 30.00th=[ 3884], 40.00th=[ 9241], 50.00th=[ 16057], 60.00th=[ 21890], | 70.00th=[ 27657], 80.00th=[ 35390], 90.00th=[ 48497], 95.00th=[ 63701], | 99.00th=[109577], 99.50th=[158335], 99.90th=[371196], 99.95th=[517997], | 99.99th=[517997] bw ( KiB/s): min= 39, max= 217, per=25.16%, avg=103.42, stdev=34.18, samples=120 iops : min= 9, max= 54, avg=25.52, stdev= 8.52, samples=120 lat (usec) : 100=2.02%, 250=7.00%, 500=1.07%, 750=0.75%, 1000=4.20% lat (msec) : 2=7.45%, 4=4.43%, 10=10.09%, 20=23.31%, 50=33.43% lat (msec) : 100=5.44%, 250=0.62%, 500=0.16%, 750=0.03% cpu : usr=0.15%, sys=9.04%, ctx=5173, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32248: Thu May 1 20:48:15 2025 read: IOPS=24, BW=98.7KiB/s (101kB/s)(6028KiB/61058msec) clat (usec): min=56, max=534762, avg=18749.85, stdev=22390.98 lat (usec): min=57, max=534763, avg=18754.40, stdev=22391.96 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 105], 10.00th=[ 122], 20.00th=[ 1106], | 30.00th=[ 9896], 40.00th=[ 12911], 50.00th=[ 15795], 60.00th=[ 19530], | 70.00th=[ 23462], 80.00th=[ 28967], 90.00th=[ 37487], 95.00th=[ 46400], | 99.00th=[ 68682], 99.50th=[ 84411], 99.90th=[308282], 99.95th=[534774], | 99.99th=[534774] bw ( KiB/s): min= 7, max= 199, per=24.92%, avg=97.95, stdev=36.49, samples=122 iops : min= 1, max= 49, avg=24.06, stdev= 9.11, samples=122 write: IOPS=25, BW=103KiB/s (105kB/s)(6260KiB/61058msec) clat (usec): min=552, max=144915, avg=20729.79, stdev=21280.26 lat (usec): min=553, max=144916, avg=20732.04, stdev=21280.01 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 873], 10.00th=[ 1057], 20.00th=[ 2089], | 30.00th=[ 4424], 40.00th=[ 9634], 50.00th=[ 15926], 60.00th=[ 20841], | 70.00th=[ 27132], 80.00th=[ 34866], 90.00th=[ 49021], 95.00th=[ 61604], | 99.00th=[ 94897], 99.50th=[114820], 99.90th=[139461], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 15, max= 195, per=24.72%, avg=101.61, stdev=33.18, samples=122 iops : min= 3, max= 48, avg=24.97, stdev= 8.30, samples=122 lat (usec) : 100=1.33%, 250=6.84%, 500=1.24%, 750=0.72%, 1000=4.04% lat (msec) : 2=6.05%, 4=5.08%, 10=10.48%, 20=23.76%, 50=33.72% lat (msec) : 100=6.12%, 250=0.55%, 500=0.03%, 750=0.03% cpu : usr=0.12%, sys=9.27%, ctx=5250, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32250: Thu May 1 20:48:15 2025 read: IOPS=24, BW=99.9KiB/s (102kB/s)(6052KiB/60560msec) clat (usec): min=77, max=224202, avg=18111.80, stdev=16375.78 lat (usec): min=78, max=224202, avg=18114.12, stdev=16377.20 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 106], 10.00th=[ 125], 20.00th=[ 848], | 30.00th=[ 9896], 40.00th=[ 12780], 50.00th=[ 15664], 60.00th=[ 19006], | 70.00th=[ 23200], 80.00th=[ 28705], 90.00th=[ 36963], 95.00th=[ 45876], | 99.00th=[ 64226], 99.50th=[ 80217], 99.90th=[145753], 99.95th=[223347], | 99.99th=[223347] bw ( KiB/s): min= 16, max= 208, per=25.46%, avg=100.08, stdev=33.60, samples=120 iops : min= 4, max= 52, avg=24.61, stdev= 8.43, samples=120 write: IOPS=25, BW=103KiB/s (105kB/s)(6236KiB/60560msec) clat (usec): min=641, max=539192, avg=21028.74, stdev=26570.29 lat (usec): min=642, max=539192, avg=21033.53, stdev=26568.14 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 857], 10.00th=[ 1029], 20.00th=[ 1844], | 30.00th=[ 3752], 40.00th=[ 8029], 50.00th=[ 14484], 60.00th=[ 20579], | 70.00th=[ 27657], 80.00th=[ 35914], 90.00th=[ 50070], 95.00th=[ 62653], | 99.00th=[ 93848], 99.50th=[117965], 99.90th=[304088], 99.95th=[541066], | 99.99th=[541066] bw ( KiB/s): min= 8, max= 190, per=24.84%, avg=102.08, stdev=37.22, samples=121 iops : min= 2, max= 47, avg=25.12, stdev= 9.26, samples=121 lat (usec) : 100=1.60%, 250=6.22%, 500=1.66%, 750=0.88%, 1000=4.20% lat (msec) : 2=6.48%, 4=5.24%, 10=10.38%, 20=24.06%, 50=32.55% lat (msec) : 100=6.18%, 250=0.49%, 500=0.03%, 750=0.03% cpu : usr=0.08%, sys=9.46%, ctx=5242, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=393KiB/s (403kB/s), 97.4KiB/s-99.9KiB/s (99.7kB/s-102kB/s), io=23.4MiB (24.6MB), run=60560-61058msec WRITE: bw=412KiB/s (422kB/s), 103KiB/s-104KiB/s (105kB/s-107kB/s), io=24.6MiB (25.7MB), run=60560-61058msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32243: Thu May 1 20:50:38 2025 read: IOPS=7, BW=29.5KiB/s (30.2kB/s)(5992KiB/203051msec) clat (msec): min=2, max=552, avg=16.89, stdev=21.20 lat (msec): min=2, max=552, avg=16.89, stdev=21.20 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 34], 95.00th=[ 47], | 99.00th=[ 91], 99.50th=[ 106], 99.90th=[ 211], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 7, max= 120, per=27.06%, avg=31.66, stdev=20.80, samples=372 iops : min= 1, max= 30, avg= 7.60, stdev= 5.23, samples=372 write: IOPS=7, BW=31.0KiB/s (31.8kB/s)(6296KiB/203051msec) clat (msec): min=28, max=403, avg=112.68, stdev=52.21 lat (msec): min=28, max=403, avg=112.69, stdev=52.20 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 56], 10.00th=[ 62], 20.00th=[ 73], | 30.00th=[ 82], 40.00th=[ 90], 50.00th=[ 100], 60.00th=[ 112], | 70.00th=[ 127], 80.00th=[ 144], 90.00th=[ 180], 95.00th=[ 215], | 99.00th=[ 313], 99.50th=[ 338], 99.90th=[ 376], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 7, max= 64, per=25.15%, avg=30.68, stdev=11.77, samples=403 iops : min= 1, max= 16, avg= 7.36, stdev= 3.03, samples=403 lat (msec) : 4=0.16%, 10=18.78%, 20=19.89%, 50=8.98%, 100=26.63% lat (msec) : 250=24.22%, 500=1.30%, 750=0.03% cpu : usr=0.04%, sys=2.13%, ctx=3610, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32245: Thu May 1 20:50:38 2025 read: IOPS=7, BW=29.0KiB/s (29.7kB/s)(5940KiB/204784msec) clat (msec): min=3, max=571, avg=17.33, stdev=22.75 lat (msec): min=3, max=571, avg=17.34, stdev=22.75 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 35], 95.00th=[ 48], | 99.00th=[ 82], 99.50th=[ 101], 99.90th=[ 266], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 7, max= 96, per=27.21%, avg=31.84, stdev=20.05, samples=366 iops : min= 1, max= 24, avg= 7.59, stdev= 5.04, samples=366 write: IOPS=7, BW=30.0KiB/s (31.7kB/s)(6348KiB/204784msec) clat (msec): min=34, max=369, avg=112.58, stdev=50.04 lat (msec): min=34, max=369, avg=112.58, stdev=50.04 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 55], 10.00th=[ 63], 20.00th=[ 73], | 30.00th=[ 83], 40.00th=[ 92], 50.00th=[ 102], 60.00th=[ 112], | 70.00th=[ 126], 80.00th=[ 144], 90.00th=[ 178], 95.00th=[ 215], | 99.00th=[ 292], 99.50th=[ 317], 99.90th=[ 363], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 7, max= 64, per=25.07%, avg=30.59, stdev=11.50, samples=406 iops : min= 1, max= 16, avg= 7.28, stdev= 2.97, samples=406 lat (msec) : 4=0.13%, 10=17.71%, 20=20.44%, 50=9.34%, 100=25.75% lat (msec) : 250=25.49%, 500=1.11%, 750=0.03% cpu : usr=0.04%, sys=2.11%, ctx=3613, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32247: Thu May 1 20:50:38 2025 read: IOPS=7, BW=29.8KiB/s (30.5kB/s)(6028KiB/202374msec) clat (msec): min=2, max=261, avg=16.77, stdev=17.07 lat (msec): min=2, max=261, avg=16.77, stdev=17.07 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 16], 80.00th=[ 20], 90.00th=[ 35], 95.00th=[ 49], | 99.00th=[ 80], 99.50th=[ 94], 99.90th=[ 197], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 7, max= 120, per=26.64%, avg=31.17, stdev=21.33, samples=380 iops : min= 1, max= 30, avg= 7.41, stdev= 5.39, samples=380 write: IOPS=7, BW=30.9KiB/s (31.7kB/s)(6260KiB/202374msec) clat (msec): min=31, max=745, avg=112.92, stdev=52.83 lat (msec): min=31, max=745, avg=112.93, stdev=52.83 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 54], 10.00th=[ 61], 20.00th=[ 73], | 30.00th=[ 84], 40.00th=[ 92], 50.00th=[ 103], 60.00th=[ 112], | 70.00th=[ 127], 80.00th=[ 146], 90.00th=[ 178], 95.00th=[ 215], | 99.00th=[ 279], 99.50th=[ 313], 99.90th=[ 418], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 7, max= 64, per=25.03%, avg=30.54, stdev=11.95, samples=402 iops : min= 1, max= 16, avg= 7.25, stdev= 3.07, samples=402 lat (msec) : 4=0.23%, 10=18.39%, 20=20.74%, 50=9.05%, 100=25.23% lat (msec) : 250=25.26%, 500=1.07%, 750=0.03% cpu : usr=0.05%, sys=2.09%, ctx=3538, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32249: Thu May 1 20:50:38 2025 read: IOPS=7, BW=30.0KiB/s (30.7kB/s)(6052KiB/201574msec) clat (msec): min=3, max=210, avg=16.41, stdev=16.91 lat (msec): min=3, max=210, avg=16.41, stdev=16.91 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 16], 80.00th=[ 20], 90.00th=[ 32], 95.00th=[ 44], | 99.00th=[ 84], 99.50th=[ 105], 99.90th=[ 197], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 7, max= 128, per=28.39%, avg=33.22, stdev=20.59, samples=359 iops : min= 1, max= 32, avg= 7.93, stdev= 5.17, samples=359 write: IOPS=7, BW=30.9KiB/s (31.7kB/s)(6236KiB/201574msec) clat (msec): min=36, max=737, avg=113.12, stdev=53.93 lat (msec): min=36, max=737, avg=113.12, stdev=53.93 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 55], 10.00th=[ 62], 20.00th=[ 72], | 30.00th=[ 83], 40.00th=[ 92], 50.00th=[ 102], 60.00th=[ 112], | 70.00th=[ 127], 80.00th=[ 146], 90.00th=[ 178], 95.00th=[ 213], | 99.00th=[ 321], 99.50th=[ 342], 99.90th=[ 401], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 7, max= 64, per=25.05%, avg=30.57, stdev=11.93, samples=401 iops : min= 1, max= 16, avg= 7.25, stdev= 3.06, samples=401 lat (msec) : 4=0.03%, 10=19.17%, 20=20.64%, 50=8.98%, 100=24.80% lat (msec) : 250=25.13%, 500=1.20%, 750=0.03% cpu : usr=0.03%, sys=2.13%, ctx=3583, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=117KiB/s (120kB/s), 29.0KiB/s-30.0KiB/s (29.7kB/s-30.7kB/s), io=23.4MiB (24.6MB), run=201574-204784msec WRITE: bw=123KiB/s (126kB/s), 30.9KiB/s-31.0KiB/s (31.7kB/s-31.8kB/s), io=24.6MiB (25.7MB), run=201574-204784msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32428: Thu May 1 20:50:51 2025 read: IOPS=35, BW=561KiB/s (575kB/s)(5920KiB/10549msec) clat (usec): min=73, max=106270, avg=11632.81, stdev=14340.05 lat (usec): min=73, max=106271, avg=11633.87, stdev=14340.09 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 98], 10.00th=[ 106], 20.00th=[ 120], | 30.00th=[ 139], 40.00th=[ 367], 50.00th=[ 8586], 60.00th=[ 12780], | 70.00th=[ 17433], 80.00th=[ 21627], 90.00th=[ 28967], 95.00th=[ 35390], | 99.00th=[ 66323], 99.50th=[ 96994], 99.90th=[106431], 99.95th=[106431], | 99.99th=[106431] bw ( KiB/s): min= 319, max= 928, per=25.36%, avg=559.10, stdev=168.26, samples=21 iops : min= 19, max= 58, avg=34.48, stdev=10.64, samples=21 write: IOPS=37, BW=604KiB/s (618kB/s)(6368KiB/10549msec) clat (usec): min=722, max=228416, avg=15441.87, stdev=19438.13 lat (usec): min=723, max=228417, avg=15443.61, stdev=19438.12 clat percentiles (usec): | 1.00th=[ 996], 5.00th=[ 1237], 10.00th=[ 1418], 20.00th=[ 1844], | 30.00th=[ 2802], 40.00th=[ 4555], 50.00th=[ 7439], 60.00th=[ 12780], | 70.00th=[ 20055], 80.00th=[ 27395], 90.00th=[ 39584], 95.00th=[ 48497], | 99.00th=[ 80217], 99.50th=[ 83362], 99.90th=[227541], 99.95th=[227541], | 99.99th=[227541] bw ( KiB/s): min= 256, max= 1179, per=25.87%, avg=600.10, stdev=221.52, samples=21 iops : min= 16, max= 73, avg=37.05, stdev=13.78, samples=21 lat (usec) : 100=2.99%, 250=14.32%, 500=2.60%, 750=0.91%, 1000=1.17% lat (msec) : 2=11.85%, 4=7.16%, 10=12.63%, 20=19.27%, 50=23.70% lat (msec) : 100=3.12%, 250=0.26% cpu : usr=0.09%, sys=11.82%, ctx=1121, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32429: Thu May 1 20:50:51 2025 read: IOPS=33, BW=529KiB/s (542kB/s)(5744KiB/10862msec) clat (usec): min=71, max=77785, avg=11740.89, stdev=13048.89 lat (usec): min=72, max=77786, avg=11743.64, stdev=13049.03 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 102], 10.00th=[ 111], 20.00th=[ 128], | 30.00th=[ 143], 40.00th=[ 725], 50.00th=[10421], 60.00th=[14091], | 70.00th=[17433], 80.00th=[21103], 90.00th=[26870], 95.00th=[35390], | 99.00th=[56361], 99.50th=[61604], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 158, max= 924, per=23.64%, avg=521.24, stdev=213.78, samples=21 iops : min= 9, max= 57, avg=32.24, stdev=13.48, samples=21 write: IOPS=37, BW=602KiB/s (617kB/s)(6544KiB/10862msec) clat (usec): min=960, max=192690, avg=15872.87, stdev=20866.52 lat (usec): min=961, max=192691, avg=15875.61, stdev=20865.80 clat percentiles (usec): | 1.00th=[ 1074], 5.00th=[ 1205], 10.00th=[ 1385], 20.00th=[ 1778], | 30.00th=[ 2933], 40.00th=[ 4686], 50.00th=[ 7570], 60.00th=[ 13698], | 70.00th=[ 19006], 80.00th=[ 26870], 90.00th=[ 39060], 95.00th=[ 51119], | 99.00th=[ 86508], 99.50th=[122160], 99.90th=[191890], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 288, max= 896, per=25.08%, avg=581.95, stdev=178.08, samples=21 iops : min= 18, max= 56, avg=36.05, stdev=11.13, samples=21 lat (usec) : 100=1.95%, 250=14.06%, 500=2.34%, 750=0.65%, 1000=0.39% lat (msec) : 2=12.63%, 4=7.81%, 10=11.46%, 20=24.09%, 50=20.83% lat (msec) : 100=3.39%, 250=0.39% cpu : usr=0.06%, sys=11.89%, ctx=1123, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32430: Thu May 1 20:50:51 2025 read: IOPS=36, BW=591KiB/s (605kB/s)(6256KiB/10590msec) clat (usec): min=78, max=83621, avg=12909.39, stdev=13683.20 lat (usec): min=79, max=83622, avg=12912.92, stdev=13685.49 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 99], 10.00th=[ 106], 20.00th=[ 125], | 30.00th=[ 153], 40.00th=[ 1106], 50.00th=[11469], 60.00th=[14746], | 70.00th=[18744], 80.00th=[24511], 90.00th=[31589], 95.00th=[35914], | 99.00th=[54789], 99.50th=[62129], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 284, max= 1216, per=26.95%, avg=594.14, stdev=231.79, samples=21 iops : min= 17, max= 76, avg=36.76, stdev=14.60, samples=21 write: IOPS=35, BW=570KiB/s (583kB/s)(6032KiB/10590msec) clat (usec): min=929, max=165408, avg=14356.03, stdev=19245.85 lat (usec): min=930, max=165412, avg=14359.28, stdev=19245.06 clat percentiles (usec): | 1.00th=[ 1057], 5.00th=[ 1156], 10.00th=[ 1319], 20.00th=[ 1680], | 30.00th=[ 2540], 40.00th=[ 4080], 50.00th=[ 5997], 60.00th=[ 10683], | 70.00th=[ 17433], 80.00th=[ 25822], 90.00th=[ 35914], 95.00th=[ 46400], | 99.00th=[106431], 99.50th=[110625], 99.90th=[164627], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 284, max= 864, per=24.43%, avg=566.67, stdev=170.54, samples=21 iops : min= 17, max= 54, avg=35.05, stdev=10.72, samples=21 lat (usec) : 100=3.26%, 250=13.67%, 500=2.34%, 750=0.39%, 1000=0.78% lat (msec) : 2=12.76%, 4=6.64%, 10=12.89%, 20=19.53%, 50=24.87% lat (msec) : 100=2.21%, 250=0.65% cpu : usr=0.14%, sys=11.54%, ctx=1164, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32431: Thu May 1 20:50:51 2025 read: IOPS=35, BW=564KiB/s (577kB/s)(6032KiB/10699msec) clat (usec): min=79, max=152557, avg=13132.68, stdev=15919.67 lat (usec): min=79, max=152559, avg=13133.85, stdev=15919.65 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 102], 10.00th=[ 110], 20.00th=[ 121], | 30.00th=[ 147], 40.00th=[ 1156], 50.00th=[ 11207], 60.00th=[ 14484], | 70.00th=[ 19006], 80.00th=[ 23200], 90.00th=[ 31327], 95.00th=[ 40109], | 99.00th=[ 61604], 99.50th=[ 84411], 99.90th=[152044], 99.95th=[152044], | 99.99th=[152044] bw ( KiB/s): min= 255, max= 896, per=25.41%, avg=560.33, stdev=151.15, samples=21 iops : min= 15, max= 56, avg=34.52, stdev= 9.69, samples=21 write: IOPS=36, BW=585KiB/s (599kB/s)(6256KiB/10699msec) clat (usec): min=971, max=156466, avg=14548.57, stdev=17615.71 lat (usec): min=974, max=156467, avg=14551.41, stdev=17617.43 clat percentiles (usec): | 1.00th=[ 1020], 5.00th=[ 1156], 10.00th=[ 1352], 20.00th=[ 1909], | 30.00th=[ 2966], 40.00th=[ 4817], 50.00th=[ 7570], 60.00th=[ 11731], | 70.00th=[ 17171], 80.00th=[ 26346], 90.00th=[ 38536], 95.00th=[ 46400], | 99.00th=[ 80217], 99.50th=[ 84411], 99.90th=[156238], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 314, max= 1088, per=25.00%, avg=579.95, stdev=191.39, samples=21 iops : min= 19, max= 68, avg=35.76, stdev=12.07, samples=21 lat (usec) : 100=2.34%, 250=13.93%, 500=1.95%, 750=0.52%, 1000=1.04% lat (msec) : 2=11.46%, 4=7.81%, 10=12.50%, 20=22.27%, 50=23.18% lat (msec) : 100=2.73%, 250=0.26% cpu : usr=0.15%, sys=11.75%, ctx=1154, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2205KiB/s (2258kB/s), 529KiB/s-591KiB/s (542kB/s-605kB/s), io=23.4MiB (24.5MB), run=10549-10862msec WRITE: bw=2320KiB/s (2376kB/s), 570KiB/s-604KiB/s (583kB/s-618kB/s), io=24.6MiB (25.8MB), run=10549-10862msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32433: Thu May 1 20:51:35 2025 read: IOPS=6, BW=109KiB/s (112kB/s)(5920KiB/54237msec) clat (msec): min=4, max=128, avg=17.32, stdev=15.55 lat (msec): min=4, max=128, avg=17.32, stdev=15.55 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 17], 80.00th=[ 20], 90.00th=[ 29], 95.00th=[ 50], | 99.00th=[ 83], 99.50th=[ 120], 99.90th=[ 129], 99.95th=[ 129], | 99.99th=[ 129] bw ( KiB/s): min= 31, max= 351, per=26.25%, avg=114.43, stdev=71.19, samples=102 iops : min= 1, max= 21, avg= 6.68, stdev= 4.47, samples=102 write: IOPS=7, BW=117KiB/s (120kB/s)(6368KiB/54237msec) clat (msec): min=53, max=287, avg=119.53, stdev=40.49 lat (msec): min=53, max=287, avg=119.54, stdev=40.49 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 68], 10.00th=[ 75], 20.00th=[ 85], | 30.00th=[ 94], 40.00th=[ 103], 50.00th=[ 112], 60.00th=[ 123], | 70.00th=[ 134], 80.00th=[ 150], 90.00th=[ 171], 95.00th=[ 197], | 99.00th=[ 262], 99.50th=[ 284], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 31, max= 192, per=25.58%, avg=117.43, stdev=30.63, samples=107 iops : min= 1, max= 12, avg= 6.87, stdev= 2.06, samples=107 lat (msec) : 10=11.98%, 20=27.86%, 50=6.12%, 100=20.70%, 250=32.81% lat (msec) : 500=0.52% cpu : usr=0.03%, sys=2.13%, ctx=922, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32434: Thu May 1 20:51:35 2025 read: IOPS=6, BW=105KiB/s (107kB/s)(5744KiB/54862msec) clat (msec): min=4, max=125, avg=16.51, stdev=14.39 lat (msec): min=4, max=125, avg=16.51, stdev=14.39 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 12], 50.00th=[ 13], 60.00th=[ 14], | 70.00th=[ 16], 80.00th=[ 19], 90.00th=[ 30], 95.00th=[ 48], | 99.00th=[ 96], 99.50th=[ 110], 99.90th=[ 126], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 31, max= 416, per=26.94%, avg=117.48, stdev=81.16, samples=96 iops : min= 1, max= 26, avg= 7.04, stdev= 5.06, samples=96 write: IOPS=7, BW=119KiB/s (122kB/s)(6544KiB/54862msec) clat (msec): min=49, max=377, avg=118.82, stdev=42.48 lat (msec): min=49, max=377, avg=118.82, stdev=42.48 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 66], 10.00th=[ 74], 20.00th=[ 84], | 30.00th=[ 94], 40.00th=[ 103], 50.00th=[ 111], 60.00th=[ 125], | 70.00th=[ 134], 80.00th=[ 150], 90.00th=[ 169], 95.00th=[ 186], | 99.00th=[ 275], 99.50th=[ 309], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 31, max= 224, per=25.71%, avg=118.00, stdev=34.43, samples=109 iops : min= 1, max= 14, avg= 7.06, stdev= 2.23, samples=109 lat (msec) : 10=14.19%, 20=24.35%, 50=6.51%, 100=20.70%, 250=33.46% lat (msec) : 500=0.78% cpu : usr=0.04%, sys=2.02%, ctx=902, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32435: Thu May 1 20:51:35 2025 read: IOPS=7, BW=119KiB/s (122kB/s)(6256KiB/52433msec) clat (msec): min=5, max=138, avg=16.55, stdev=14.67 lat (msec): min=5, max=138, avg=16.55, stdev=14.67 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 17], 80.00th=[ 20], 90.00th=[ 28], 95.00th=[ 40], | 99.00th=[ 77], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 31, max= 351, per=28.30%, avg=123.40, stdev=82.81, samples=101 iops : min= 1, max= 21, avg= 7.32, stdev= 5.31, samples=101 write: IOPS=7, BW=115KiB/s (118kB/s)(6032KiB/52433msec) clat (msec): min=50, max=298, avg=121.61, stdev=41.48 lat (msec): min=50, max=298, avg=121.62, stdev=41.49 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 65], 10.00th=[ 74], 20.00th=[ 87], | 30.00th=[ 96], 40.00th=[ 108], 50.00th=[ 117], 60.00th=[ 125], | 70.00th=[ 136], 80.00th=[ 150], 90.00th=[ 174], 95.00th=[ 203], | 99.00th=[ 255], 99.50th=[ 271], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 31, max= 222, per=24.89%, avg=114.26, stdev=31.37, samples=104 iops : min= 1, max= 13, avg= 6.75, stdev= 2.05, samples=104 lat (msec) : 10=13.41%, 20=28.52%, 50=7.29%, 100=17.71%, 250=32.55% lat (msec) : 500=0.52% cpu : usr=0.02%, sys=2.21%, ctx=937, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32436: Thu May 1 20:51:35 2025 read: IOPS=6, BW=110KiB/s (113kB/s)(6032KiB/54880msec) clat (msec): min=4, max=143, avg=16.86, stdev=16.02 lat (msec): min=4, max=143, avg=16.87, stdev=16.02 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 16], 80.00th=[ 20], 90.00th=[ 29], 95.00th=[ 52], | 99.00th=[ 93], 99.50th=[ 107], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 31, max= 350, per=28.54%, avg=124.45, stdev=79.81, samples=95 iops : min= 1, max= 21, avg= 7.37, stdev= 5.04, samples=95 write: IOPS=7, BW=114KiB/s (117kB/s)(6256KiB/54880msec) clat (msec): min=53, max=302, avg=123.22, stdev=42.02 lat (msec): min=53, max=302, avg=123.22, stdev=42.01 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 70], 10.00th=[ 78], 20.00th=[ 88], | 30.00th=[ 97], 40.00th=[ 107], 50.00th=[ 118], 60.00th=[ 128], | 70.00th=[ 138], 80.00th=[ 150], 90.00th=[ 171], 95.00th=[ 203], | 99.00th=[ 264], 99.50th=[ 300], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 32, max= 192, per=24.87%, avg=114.14, stdev=29.35, samples=108 iops : min= 2, max= 12, avg= 6.71, stdev= 1.88, samples=108 lat (msec) : 10=15.36%, 20=25.00%, 50=6.12%, 100=19.79%, 250=33.07% lat (msec) : 500=0.65% cpu : usr=0.04%, sys=2.05%, ctx=903, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=436KiB/s (447kB/s), 105KiB/s-119KiB/s (107kB/s-122kB/s), io=23.4MiB (24.5MB), run=52433-54880msec WRITE: bw=459KiB/s (470kB/s), 114KiB/s-119KiB/s (117kB/s-122kB/s), io=24.6MiB (25.8MB), run=52433-54880msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32489: Thu May 1 20:51:43 2025 read: IOPS=1, BW=1927KiB/s (1974kB/s)(7168KiB/3719msec) clat (usec): min=671, max=2165.3k, avg=409396.78, stdev=776373.21 lat (usec): min=672, max=2165.3k, avg=409398.56, stdev=776373.12 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 668], 10.00th=[ 668], | 20.00th=[ 114820], 30.00th=[ 116917], 40.00th=[ 116917], | 50.00th=[ 130548], 60.00th=[ 154141], 70.00th=[ 154141], | 80.00th=[ 183501], 90.00th=[2164261], 95.00th=[2164261], | 99.00th=[2164261], 99.50th=[2164261], 99.90th=[2164261], | 99.95th=[2164261], 99.99th=[2164261] bw ( KiB/s): min= 4096, max= 6144, per=100.00%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=1, BW=1377KiB/s (1410kB/s)(5120KiB/3719msec) clat (msec): min=77, max=319, avg=148.26, stdev=98.94 lat (msec): min=77, max=319, avg=148.39, stdev=98.90 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 78], 10.00th=[ 78], 20.00th=[ 78], | 30.00th=[ 86], 40.00th=[ 86], 50.00th=[ 121], 60.00th=[ 121], | 70.00th=[ 140], 80.00th=[ 140], 90.00th=[ 321], 95.00th=[ 321], | 99.00th=[ 321], 99.50th=[ 321], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 2048, max= 6131, per=76.70%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (usec) : 750=8.33% lat (msec) : 100=16.67%, 250=58.33%, 500=8.33% cpu : usr=0.11%, sys=12.02%, ctx=588, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32490: Thu May 1 20:51:43 2025 read: IOPS=0, BW=483KiB/s (494kB/s)(2048KiB/4242msec) clat (usec): min=120023, max=120495, avg=120259.65, stdev=333.36 lat (usec): min=120026, max=120496, avg=120261.47, stdev=332.74 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 121], 10.00th=[ 121], 20.00th=[ 121], | 30.00th=[ 121], 40.00th=[ 121], 50.00th=[ 121], 60.00th=[ 121], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 121], 95.00th=[ 121], | 99.00th=[ 121], 99.50th=[ 121], 99.90th=[ 121], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 2027, max= 2043, per=53.44%, avg=2035.00, stdev=11.31, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2414KiB/s (2472kB/s)(10.0MiB/4242msec) clat (msec): min=45, max=2217, avg=361.50, stdev=658.99 lat (msec): min=45, max=2217, avg=361.60, stdev=658.98 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 55], | 30.00th=[ 69], 40.00th=[ 81], 50.00th=[ 176], 60.00th=[ 182], | 70.00th=[ 257], 80.00th=[ 257], 90.00th=[ 338], 95.00th=[ 2232], | 99.00th=[ 2232], 99.50th=[ 2232], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 4087, max= 6107, per=98.33%, avg=5243.00, stdev=1041.18, samples=3 iops : min= 3, max= 5, avg= 4.33, stdev= 1.15, samples=3 lat (msec) : 50=8.33%, 100=25.00%, 250=41.67%, 500=16.67% cpu : usr=0.00%, sys=7.10%, ctx=230, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32491: Thu May 1 20:51:43 2025 read: IOPS=1, BW=1603KiB/s (1642kB/s)(6144KiB/3832msec) clat (usec): min=489, max=139981, avg=54457.61, stdev=62990.13 lat (usec): min=490, max=139982, avg=54459.21, stdev=62990.36 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 490], 10.00th=[ 490], 20.00th=[ 510], | 30.00th=[ 510], 40.00th=[ 586], 50.00th=[ 586], 60.00th=[ 71828], | 70.00th=[113771], 80.00th=[113771], 90.00th=[139461], 95.00th=[139461], | 99.00th=[139461], 99.50th=[139461], 99.90th=[139461], 99.95th=[139461], | 99.99th=[139461] bw ( KiB/s): min= 2043, max= 6011, per=100.00%, avg=4027.00, stdev=2805.80, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 write: IOPS=1, BW=1603KiB/s (1642kB/s)(6144KiB/3832msec) clat (msec): min=91, max=2413, avg=543.10, stdev=919.86 lat (msec): min=91, max=2413, avg=543.14, stdev=919.88 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 92], 10.00th=[ 92], 20.00th=[ 101], | 30.00th=[ 101], 40.00th=[ 125], 50.00th=[ 125], 60.00th=[ 232], | 70.00th=[ 296], 80.00th=[ 296], 90.00th=[ 2400], 95.00th=[ 2400], | 99.00th=[ 2400], 99.50th=[ 2400], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 2003, max= 4071, per=50.74%, avg=2705.67, stdev=1182.58, samples=3 iops : min= 1, max= 3, avg= 1.67, stdev= 1.15, samples=3 lat (usec) : 500=8.33%, 750=16.67% lat (msec) : 100=16.67%, 250=41.67%, 500=8.33% cpu : usr=0.03%, sys=6.73%, ctx=104, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32492: Thu May 1 20:51:43 2025 read: IOPS=0, BW=952KiB/s (975kB/s)(5120KiB/5377msec) clat (msec): min=91, max=159, avg=130.46, stdev=26.37 lat (msec): min=91, max=159, avg=130.46, stdev=26.37 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 92], 10.00th=[ 92], 20.00th=[ 92], | 30.00th=[ 118], 40.00th=[ 118], 50.00th=[ 140], 60.00th=[ 140], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 161], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1333KiB/s (1365kB/s)(7168KiB/5377msec) clat (msec): min=158, max=2486, avg=625.47, stdev=832.98 lat (msec): min=158, max=2486, avg=625.54, stdev=832.98 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 178], | 30.00th=[ 300], 40.00th=[ 300], 50.00th=[ 326], 60.00th=[ 338], | 70.00th=[ 338], 80.00th=[ 592], 90.00th=[ 2500], 95.00th=[ 2500], | 99.00th=[ 2500], 99.50th=[ 2500], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 2015, max= 4071, per=45.79%, avg=2441.60, stdev=910.95, samples=5 iops : min= 1, max= 3, avg= 1.60, stdev= 0.89, samples=5 lat (msec) : 100=8.33%, 250=50.00%, 500=25.00%, 750=8.33% cpu : usr=0.00%, sys=6.64%, ctx=163, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3809KiB/s (3900kB/s), 483KiB/s-1927KiB/s (494kB/s-1974kB/s), io=20.0MiB (20.0MB), run=3719-5377msec WRITE: bw=5332KiB/s (5460kB/s), 1333KiB/s-2414KiB/s (1365kB/s-2472kB/s), io=28.0MiB (29.4MB), run=3719-5377msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32493: Thu May 1 20:51:43 2025 read: IOPS=1, BW=1611KiB/s (1649kB/s)(7168KiB/4450msec) clat (msec): min=25, max=2126, avg=396.08, stdev=765.33 lat (msec): min=25, max=2126, avg=396.08, stdev=765.33 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 52], | 30.00th=[ 81], 40.00th=[ 81], 50.00th=[ 157], 60.00th=[ 161], | 70.00th=[ 161], 80.00th=[ 171], 90.00th=[ 2123], 95.00th=[ 2123], | 99.00th=[ 2123], 99.50th=[ 2123], 99.90th=[ 2123], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 4087, max= 6107, per=100.00%, avg=5097.00, stdev=1428.36, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 write: IOPS=1, BW=1151KiB/s (1178kB/s)(5120KiB/4450msec) clat (msec): min=204, max=428, avg=312.40, stdev=79.20 lat (msec): min=204, max=428, avg=312.48, stdev=79.19 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 205], 10.00th=[ 205], 20.00th=[ 205], | 30.00th=[ 305], 40.00th=[ 305], 50.00th=[ 313], 60.00th=[ 313], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 430], 95.00th=[ 430], | 99.00th=[ 430], 99.50th=[ 430], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 2035, max= 4063, per=51.02%, avg=2715.33, stdev=1167.13, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 lat (msec) : 50=8.33%, 100=16.67%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=1.46%, ctx=43, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32494: Thu May 1 20:51:43 2025 read: IOPS=0, BW=405KiB/s (414kB/s)(2048KiB/5061msec) clat (msec): min=134, max=231, avg=182.96, stdev=68.08 lat (msec): min=134, max=231, avg=182.96, stdev=68.08 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 232], | 70.00th=[ 232], 80.00th=[ 232], 90.00th=[ 232], 95.00th=[ 232], | 99.00th=[ 232], 99.50th=[ 232], 99.90th=[ 232], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 2048, max= 2048, per=53.88%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=1, BW=2023KiB/s (2072kB/s)(10.0MiB/5061msec) clat (msec): min=87, max=2505, avg=444.91, stdev=738.31 lat (msec): min=87, max=2505, avg=444.99, stdev=738.31 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 88], 10.00th=[ 88], 20.00th=[ 121], | 30.00th=[ 133], 40.00th=[ 140], 50.00th=[ 155], 60.00th=[ 163], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 575], 95.00th=[ 2500], | 99.00th=[ 2500], 99.50th=[ 2500], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 2031, max= 6023, per=68.71%, avg=3657.00, stdev=1673.94, samples=5 iops : min= 1, max= 5, avg= 3.00, stdev= 1.58, samples=5 lat (msec) : 100=8.33%, 250=66.67%, 500=8.33%, 750=8.33% cpu : usr=0.02%, sys=1.92%, ctx=40, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32495: Thu May 1 20:51:43 2025 read: IOPS=1, BW=1141KiB/s (1168kB/s)(6144KiB/5387msec) clat (msec): min=53, max=785, avg=278.04, stdev=286.46 lat (msec): min=53, max=785, avg=278.05, stdev=286.46 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 54], 10.00th=[ 54], 20.00th=[ 79], | 30.00th=[ 79], 40.00th=[ 106], 50.00th=[ 106], 60.00th=[ 203], | 70.00th=[ 443], 80.00th=[ 443], 90.00th=[ 785], 95.00th=[ 785], | 99.00th=[ 785], 99.50th=[ 785], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 2043, max= 6144, per=100.00%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 write: IOPS=1, BW=1141KiB/s (1168kB/s)(6144KiB/5387msec) clat (msec): min=116, max=2363, avg=534.04, stdev=897.82 lat (msec): min=116, max=2363, avg=534.12, stdev=897.80 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 117], 10.00th=[ 117], 20.00th=[ 122], | 30.00th=[ 122], 40.00th=[ 148], 50.00th=[ 148], 60.00th=[ 205], | 70.00th=[ 251], 80.00th=[ 251], 90.00th=[ 2366], 95.00th=[ 2366], | 99.00th=[ 2366], 99.50th=[ 2366], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 2043, max= 4071, per=47.96%, avg=2552.50, stdev=1012.34, samples=4 iops : min= 1, max= 3, avg= 2.00, stdev= 0.82, samples=4 lat (msec) : 100=16.67%, 250=50.00%, 500=16.67%, 1000=8.33% cpu : usr=0.02%, sys=1.80%, ctx=47, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32496: Thu May 1 20:51:43 2025 read: IOPS=0, BW=1002KiB/s (1026kB/s)(5120KiB/5111msec) clat (msec): min=34, max=355, avg=175.75, stdev=131.26 lat (msec): min=34, max=355, avg=175.75, stdev=131.26 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 35], | 30.00th=[ 68], 40.00th=[ 68], 50.00th=[ 174], 60.00th=[ 174], | 70.00th=[ 247], 80.00th=[ 247], 90.00th=[ 355], 95.00th=[ 355], | 99.00th=[ 355], 99.50th=[ 355], 99.90th=[ 355], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 4071, max= 4096, per=100.00%, avg=4083.50, stdev=17.68, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 write: IOPS=1, BW=1402KiB/s (1436kB/s)(7168KiB/5111msec) clat (msec): min=132, max=2375, avg=588.08, stdev=799.90 lat (msec): min=132, max=2375, avg=588.17, stdev=799.89 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 136], | 30.00th=[ 236], 40.00th=[ 236], 50.00th=[ 334], 60.00th=[ 397], | 70.00th=[ 397], 80.00th=[ 510], 90.00th=[ 2366], 95.00th=[ 2366], | 99.00th=[ 2366], 99.50th=[ 2366], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 2035, max= 4096, per=46.10%, avg=2453.20, stdev=918.37, samples=5 iops : min= 1, max= 4, avg= 2.00, stdev= 1.22, samples=5 lat (msec) : 50=8.33%, 100=8.33%, 250=41.67%, 500=25.00%, 750=8.33% cpu : usr=0.10%, sys=1.29%, ctx=39, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3802KiB/s (3893kB/s), 405KiB/s-1611KiB/s (414kB/s-1649kB/s), io=20.0MiB (20.0MB), run=4450-5387msec WRITE: bw=5322KiB/s (5450kB/s), 1141KiB/s-2023KiB/s (1168kB/s-2072kB/s), io=28.0MiB (29.4MB), run=4450-5387msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32508: Thu May 1 20:51:48 2025 read: IOPS=1, BW=5710KiB/s (5847kB/s)(12.0MiB/2152msec) clat (msec): min=403, max=944, avg=690.80, stdev=272.10 lat (msec): min=403, max=944, avg=690.81, stdev=272.10 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 405], 10.00th=[ 405], 20.00th=[ 405], | 30.00th=[ 405], 40.00th=[ 726], 50.00th=[ 726], 60.00th=[ 726], | 70.00th=[ 944], 80.00th=[ 944], 90.00th=[ 944], 95.00th=[ 944], | 99.00th=[ 944], 99.50th=[ 944], 99.90th=[ 944], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 8078, max= 8175, per=100.00%, avg=8126.50, stdev=68.59, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=11.30%, ctx=477, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32511: Thu May 1 20:51:48 2025 read: IOPS=0, BW=1875KiB/s (1920kB/s)(4096KiB/2184msec) clat (nsec): min=297821k, max=297821k, avg=297821206.00, stdev= 0.00 lat (nsec): min=297824k, max=297824k, avg=297824098.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3751KiB/s (3841kB/s)(8192KiB/2184msec) clat (msec): min=421, max=1281, avg=851.17, stdev=608.13 lat (msec): min=421, max=1281, avg=851.45, stdev=608.16 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 422], 10.00th=[ 422], 20.00th=[ 422], | 30.00th=[ 422], 40.00th=[ 422], 50.00th=[ 422], 60.00th=[ 1284], | 70.00th=[ 1284], 80.00th=[ 1284], 90.00th=[ 1284], 95.00th=[ 1284], | 99.00th=[ 1284], 99.50th=[ 1284], 99.90th=[ 1284], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 8192, max= 8192, per=56.35%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=10.86%, ctx=101, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32513: Thu May 1 20:51:48 2025 write: IOPS=1, BW=5452KiB/s (5582kB/s)(12.0MiB/2254msec) clat (msec): min=487, max=988, avg=729.03, stdev=250.60 lat (msec): min=488, max=988, avg=729.32, stdev=250.58 clat percentiles (msec): | 1.00th=[ 489], 5.00th=[ 489], 10.00th=[ 489], 20.00th=[ 489], | 30.00th=[ 489], 40.00th=[ 709], 50.00th=[ 709], 60.00th=[ 709], | 70.00th=[ 986], 80.00th=[ 986], 90.00th=[ 986], 95.00th=[ 986], | 99.00th=[ 986], 99.50th=[ 986], 99.90th=[ 986], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 8094, max= 8192, per=56.02%, avg=8143.00, stdev=69.30, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=14.43%, ctx=104, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32514: Thu May 1 20:51:48 2025 write: IOPS=1, BW=6175KiB/s (6323kB/s)(12.0MiB/1990msec) clat (msec): min=352, max=824, avg=555.38, stdev=242.53 lat (msec): min=353, max=824, avg=555.73, stdev=242.52 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 355], 10.00th=[ 355], 20.00th=[ 355], | 30.00th=[ 355], 40.00th=[ 489], 50.00th=[ 489], 60.00th=[ 489], | 70.00th=[ 827], 80.00th=[ 827], 90.00th=[ 827], 95.00th=[ 827], | 99.00th=[ 827], 99.50th=[ 827], 99.90th=[ 827], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 8110, max= 8143, per=55.90%, avg=8126.50, stdev=23.33, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67%, 1000=33.33% cpu : usr=0.00%, sys=15.49%, ctx=78, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7502KiB/s (7682kB/s), 1875KiB/s-5710KiB/s (1920kB/s-5847kB/s), io=16.0MiB (16.8MB), run=2152-2184msec WRITE: bw=14.2MiB/s (14.9MB/s), 3751KiB/s-6175KiB/s (3841kB/s-6323kB/s), io=32.0MiB (33.6MB), run=1990-2254msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32507: Thu May 1 20:51:49 2025 read: IOPS=1, BW=4857KiB/s (4973kB/s)(12.0MiB/2530msec) clat (msec): min=543, max=951, avg=779.70, stdev=211.74 lat (msec): min=543, max=951, avg=779.70, stdev=211.74 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 542], 10.00th=[ 542], 20.00th=[ 542], | 30.00th=[ 542], 40.00th=[ 844], 50.00th=[ 844], 60.00th=[ 844], | 70.00th=[ 953], 80.00th=[ 953], 90.00th=[ 953], 95.00th=[ 953], | 99.00th=[ 953], 99.50th=[ 953], 99.90th=[ 953], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 7984, max= 8094, per=100.00%, avg=8039.00, stdev=77.78, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 750=33.33%, 1000=66.67% cpu : usr=0.08%, sys=1.11%, ctx=18, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32509: Thu May 1 20:51:49 2025 read: IOPS=0, BW=1482KiB/s (1518kB/s)(4096KiB/2763msec) clat (nsec): min=887530k, max=887530k, avg=887530061.00, stdev= 0.00 lat (nsec): min=887534k, max=887534k, avg=887533506.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 885], 5.00th=[ 885], 10.00th=[ 885], 20.00th=[ 885], | 30.00th=[ 885], 40.00th=[ 885], 50.00th=[ 885], 60.00th=[ 885], | 70.00th=[ 885], 80.00th=[ 885], 90.00th=[ 885], 95.00th=[ 885], | 99.00th=[ 885], 99.50th=[ 885], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2965KiB/s (3036kB/s)(8192KiB/2763msec) clat (msec): min=402, max=1341, avg=871.62, stdev=663.84 lat (msec): min=402, max=1341, avg=871.90, stdev=663.85 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 401], 10.00th=[ 401], 20.00th=[ 401], | 30.00th=[ 401], 40.00th=[ 401], 50.00th=[ 401], 60.00th=[ 1334], | 70.00th=[ 1334], 80.00th=[ 1334], 90.00th=[ 1334], 95.00th=[ 1334], | 99.00th=[ 1334], 99.50th=[ 1334], 99.90th=[ 1334], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 8175, max= 8175, per=77.94%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.04%, sys=1.52%, ctx=25, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32510: Thu May 1 20:51:49 2025 write: IOPS=0, BW=3933KiB/s (4028kB/s)(12.0MiB/3124msec) clat (msec): min=211, max=1519, avg=923.80, stdev=661.78 lat (msec): min=211, max=1519, avg=924.20, stdev=661.66 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 211], 10.00th=[ 211], 20.00th=[ 211], | 30.00th=[ 211], 40.00th=[ 1045], 50.00th=[ 1045], 60.00th=[ 1045], | 70.00th=[ 1519], 80.00th=[ 1519], 90.00th=[ 1519], 95.00th=[ 1519], | 99.00th=[ 1519], 99.50th=[ 1519], 99.90th=[ 1519], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 6692, max= 8175, per=70.87%, avg=7433.50, stdev=1048.64, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=1.89%, ctx=34, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32512: Thu May 1 20:51:49 2025 write: IOPS=1, BW=6865KiB/s (7030kB/s)(12.0MiB/1790msec) clat (msec): min=280, max=1007, avg=548.91, stdev=398.67 lat (msec): min=281, max=1007, avg=549.49, stdev=398.40 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 279], 10.00th=[ 279], 20.00th=[ 279], | 30.00th=[ 279], 40.00th=[ 359], 50.00th=[ 359], 60.00th=[ 359], | 70.00th=[ 1011], 80.00th=[ 1011], 90.00th=[ 1011], 95.00th=[ 1011], | 99.00th=[ 1011], 99.50th=[ 1011], 99.90th=[ 1011], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 8175, max= 8192, per=78.02%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67% cpu : usr=0.11%, sys=3.58%, ctx=24, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5930KiB/s (6072kB/s), 1482KiB/s-4857KiB/s (1518kB/s-4973kB/s), io=16.0MiB (16.8MB), run=2530-2763msec WRITE: bw=10.2MiB/s (10.7MB/s), 2965KiB/s-6865KiB/s (3036kB/s-7030kB/s), io=32.0MiB (33.6MB), run=1790-3124msec