== sanity test 398b: DIO and buffer IO race ============== 23:50:03 (1746157803) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.10022 s, 24.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31392: Thu May 1 23:50:53 2025 read: IOPS=33, BW=133KiB/s (136kB/s)(5992KiB/45072msec) clat (usec): min=70, max=322608, avg=13645.49, stdev=15548.45 lat (usec): min=70, max=322609, avg=13647.48, stdev=15549.16 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 96], 10.00th=[ 106], 20.00th=[ 137], | 30.00th=[ 6521], 40.00th=[ 9110], 50.00th=[ 11338], 60.00th=[ 13829], | 70.00th=[ 17433], 80.00th=[ 22152], 90.00th=[ 28967], 95.00th=[ 35914], | 99.00th=[ 53740], 99.50th=[ 66847], 99.90th=[204473], 99.95th=[320865], | 99.99th=[320865] bw ( KiB/s): min= 7, max= 232, per=25.29%, avg=132.03, stdev=43.61, samples=89 iops : min= 1, max= 58, avg=32.70, stdev=10.95, samples=89 write: IOPS=34, BW=140KiB/s (143kB/s)(6296KiB/45072msec) clat (usec): min=611, max=229271, avg=15456.11, stdev=16883.98 lat (usec): min=612, max=229271, avg=15457.85, stdev=16883.79 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 750], 10.00th=[ 799], 20.00th=[ 1020], | 30.00th=[ 2474], 40.00th=[ 6128], 50.00th=[ 11338], 60.00th=[ 15926], | 70.00th=[ 20317], 80.00th=[ 27132], 90.00th=[ 36439], 95.00th=[ 46400], | 99.00th=[ 65274], 99.50th=[ 74974], 99.90th=[137364], 99.95th=[229639], | 99.99th=[229639] bw ( KiB/s): min= 48, max= 272, per=25.55%, avg=139.78, stdev=42.52, samples=88 iops : min= 12, max= 68, avg=34.65, stdev=10.67, samples=88 lat (usec) : 100=3.19%, 250=8.20%, 500=0.26%, 750=2.70%, 1000=7.62% lat (msec) : 2=4.36%, 4=4.17%, 10=14.78%, 20=27.18%, 50=24.80% lat (msec) : 100=2.54%, 250=0.16%, 500=0.03% cpu : usr=0.11%, sys=10.47%, ctx=4851, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31393: Thu May 1 23:50:53 2025 read: IOPS=32, BW=129KiB/s (132kB/s)(5940KiB/45957msec) clat (usec): min=49, max=263793, avg=12324.65, stdev=13298.91 lat (usec): min=50, max=263795, avg=12327.95, stdev=13297.58 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 92], 10.00th=[ 103], 20.00th=[ 129], | 30.00th=[ 5276], 40.00th=[ 8455], 50.00th=[ 10683], 60.00th=[ 12780], | 70.00th=[ 15795], 80.00th=[ 19792], 90.00th=[ 28443], 95.00th=[ 33162], | 99.00th=[ 49021], 99.50th=[ 53740], 99.90th=[121111], 99.95th=[263193], | 99.99th=[263193] bw ( KiB/s): min= 7, max= 248, per=24.96%, avg=130.30, stdev=47.52, samples=89 iops : min= 1, max= 62, avg=32.33, stdev=11.88, samples=89 write: IOPS=34, BW=138KiB/s (141kB/s)(6348KiB/45957msec) clat (usec): min=556, max=379758, avg=17163.66, stdev=21883.90 lat (usec): min=557, max=379760, avg=17166.09, stdev=21884.28 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 758], 10.00th=[ 799], 20.00th=[ 1254], | 30.00th=[ 2835], 40.00th=[ 6652], 50.00th=[ 11994], 60.00th=[ 17171], | 70.00th=[ 21890], 80.00th=[ 28443], 90.00th=[ 40633], 95.00th=[ 50070], | 99.00th=[ 95945], 99.50th=[129500], 99.90th=[196084], 99.95th=[379585], | 99.99th=[379585] bw ( KiB/s): min= 16, max= 232, per=24.95%, avg=136.49, stdev=42.82, samples=90 iops : min= 4, max= 58, avg=33.88, stdev=10.70, samples=90 lat (usec) : 50=0.03%, 100=4.13%, 250=8.33%, 500=0.42%, 750=2.60% lat (usec) : 1000=6.77% lat (msec) : 2=5.21%, 4=4.39%, 10=14.49%, 20=27.02%, 50=23.60% lat (msec) : 100=2.47%, 250=0.46%, 500=0.07% cpu : usr=0.09%, sys=10.12%, ctx=4799, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31394: Thu May 1 23:50:53 2025 read: IOPS=33, BW=133KiB/s (137kB/s)(6028KiB/45199msec) clat (usec): min=56, max=321591, avg=13335.98, stdev=14874.92 lat (usec): min=56, max=321593, avg=13339.14, stdev=14873.51 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 93], 10.00th=[ 103], 20.00th=[ 133], | 30.00th=[ 5997], 40.00th=[ 9110], 50.00th=[ 11469], 60.00th=[ 13960], | 70.00th=[ 17433], 80.00th=[ 21627], 90.00th=[ 28705], 95.00th=[ 34866], | 99.00th=[ 52167], 99.50th=[ 57934], 99.90th=[183501], 99.95th=[320865], | 99.99th=[320865] bw ( KiB/s): min= 7, max= 256, per=25.38%, avg=132.50, stdev=44.14, samples=90 iops : min= 1, max= 64, avg=32.78, stdev=11.07, samples=90 write: IOPS=34, BW=138KiB/s (142kB/s)(6260KiB/45199msec) clat (usec): min=501, max=197783, avg=15848.38, stdev=17455.98 lat (usec): min=501, max=197785, avg=15851.20, stdev=17457.37 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 742], 10.00th=[ 799], 20.00th=[ 1123], | 30.00th=[ 2737], 40.00th=[ 6325], 50.00th=[ 11863], 60.00th=[ 16319], | 70.00th=[ 20841], 80.00th=[ 26870], 90.00th=[ 37487], 95.00th=[ 46400], | 99.00th=[ 77071], 99.50th=[ 93848], 99.90th=[149947], 99.95th=[198181], | 99.99th=[198181] bw ( KiB/s): min= 23, max= 223, per=25.10%, avg=137.30, stdev=41.33, samples=90 iops : min= 5, max= 55, avg=33.98, stdev=10.41, samples=90 lat (usec) : 100=4.13%, 250=7.78%, 500=0.36%, 750=3.22%, 1000=6.71% lat (msec) : 2=4.07%, 4=4.62%, 10=14.03%, 20=27.83%, 50=24.64% lat (msec) : 100=2.34%, 250=0.23%, 500=0.03% cpu : usr=0.12%, sys=10.22%, ctx=4811, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31395: Thu May 1 23:50:53 2025 read: IOPS=33, BW=136KiB/s (139kB/s)(6052KiB/44618msec) clat (usec): min=68, max=196648, avg=13503.20, stdev=13085.68 lat (usec): min=68, max=196649, avg=13504.40, stdev=13085.67 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 97], 10.00th=[ 110], 20.00th=[ 157], | 30.00th=[ 6915], 40.00th=[ 9503], 50.00th=[ 11731], 60.00th=[ 14353], | 70.00th=[ 16909], 80.00th=[ 21627], 90.00th=[ 28443], 95.00th=[ 34866], | 99.00th=[ 47973], 99.50th=[ 60031], 99.90th=[164627], 99.95th=[196084], | 99.99th=[196084] bw ( KiB/s): min= 23, max= 216, per=26.14%, avg=136.45, stdev=40.32, samples=88 iops : min= 5, max= 54, avg=33.70, stdev=10.06, samples=88 write: IOPS=34, BW=140KiB/s (143kB/s)(6236KiB/44618msec) clat (usec): min=565, max=364504, avg=15323.06, stdev=18172.20 lat (usec): min=567, max=364507, avg=15325.34, stdev=18173.13 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 734], 10.00th=[ 791], 20.00th=[ 971], | 30.00th=[ 2343], 40.00th=[ 5735], 50.00th=[ 10945], 60.00th=[ 15664], | 70.00th=[ 20317], 80.00th=[ 26608], 90.00th=[ 35914], 95.00th=[ 46400], | 99.00th=[ 67634], 99.50th=[ 82314], 99.90th=[ 95945], 99.95th=[362808], | 99.99th=[362808] bw ( KiB/s): min= 40, max= 254, per=25.68%, avg=140.47, stdev=41.14, samples=88 iops : min= 10, max= 63, avg=34.70, stdev=10.32, samples=88 lat (usec) : 100=3.35%, 250=6.97%, 500=0.36%, 750=3.32%, 1000=7.42% lat (msec) : 2=4.59%, 4=3.68%, 10=15.27%, 20=28.09%, 50=24.64% lat (msec) : 100=2.21%, 250=0.07%, 500=0.03% cpu : usr=0.10%, sys=10.55%, ctx=4862, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=522KiB/s (535kB/s), 129KiB/s-136KiB/s (132kB/s-139kB/s), io=23.4MiB (24.6MB), run=44618-45957msec WRITE: bw=547KiB/s (560kB/s), 138KiB/s-140KiB/s (141kB/s-143kB/s), io=24.6MiB (25.7MB), run=44618-45957msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31396: Thu May 1 23:52:31 2025 read: IOPS=10, BW=41.6KiB/s (42.6kB/s)(5992KiB/144115msec) clat (msec): min=2, max=328, avg=12.45, stdev=14.16 lat (msec): min=2, max=328, avg=12.46, stdev=14.16 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 16], 90.00th=[ 27], 95.00th=[ 36], | 99.00th=[ 54], 99.50th=[ 74], 99.90th=[ 144], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 7, max= 127, per=25.52%, avg=42.36, stdev=27.01, samples=280 iops : min= 1, max= 31, avg=10.28, stdev= 6.79, samples=280 write: IOPS=10, BW=43.7KiB/s (44.7kB/s)(6296KiB/144115msec) clat (msec): min=24, max=269, avg=79.53, stdev=36.49 lat (msec): min=24, max=269, avg=79.54, stdev=36.49 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 40], 10.00th=[ 43], 20.00th=[ 51], | 30.00th=[ 58], 40.00th=[ 65], 50.00th=[ 71], 60.00th=[ 79], | 70.00th=[ 88], 80.00th=[ 104], 90.00th=[ 128], 95.00th=[ 155], | 99.00th=[ 207], 99.50th=[ 239], 99.90th=[ 264], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 8, max= 87, per=24.93%, avg=43.39, stdev=15.77, samples=287 iops : min= 2, max= 21, avg=10.54, stdev= 3.99, samples=287 lat (msec) : 4=0.98%, 10=29.98%, 20=10.61%, 50=16.44%, 100=30.76% lat (msec) : 250=11.13%, 500=0.10% cpu : usr=0.03%, sys=2.35%, ctx=3446, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31397: Thu May 1 23:52:31 2025 read: IOPS=10, BW=41.3KiB/s (42.3kB/s)(5940KiB/143849msec) clat (msec): min=2, max=369, avg=12.48, stdev=15.22 lat (msec): min=2, max=369, avg=12.48, stdev=15.22 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 11], 80.00th=[ 15], 90.00th=[ 26], 95.00th=[ 36], | 99.00th=[ 61], 99.50th=[ 83], 99.90th=[ 133], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 7, max= 120, per=25.88%, avg=42.96, stdev=26.23, samples=274 iops : min= 1, max= 30, avg=10.51, stdev= 6.59, samples=274 write: IOPS=11, BW=44.1KiB/s (45.2kB/s)(6348KiB/143849msec) clat (msec): min=24, max=287, avg=78.79, stdev=36.90 lat (msec): min=24, max=287, avg=78.80, stdev=36.90 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 40], 10.00th=[ 44], 20.00th=[ 52], | 30.00th=[ 57], 40.00th=[ 63], 50.00th=[ 69], 60.00th=[ 77], | 70.00th=[ 87], 80.00th=[ 101], 90.00th=[ 128], 95.00th=[ 155], | 99.00th=[ 213], 99.50th=[ 232], 99.90th=[ 257], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 7, max= 87, per=25.26%, avg=43.95, stdev=16.32, samples=286 iops : min= 1, max= 21, avg=10.76, stdev= 4.15, samples=286 lat (msec) : 4=0.88%, 10=30.11%, 20=10.58%, 50=15.23%, 100=32.62% lat (msec) : 250=10.45%, 500=0.13% cpu : usr=0.03%, sys=2.34%, ctx=3444, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31398: Thu May 1 23:52:31 2025 read: IOPS=10, BW=41.9KiB/s (42.9kB/s)(6028KiB/143934msec) clat (msec): min=2, max=142, avg=13.14, stdev=13.15 lat (msec): min=2, max=142, avg=13.15, stdev=13.16 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 18], 90.00th=[ 27], 95.00th=[ 38], | 99.00th=[ 65], 99.50th=[ 89], 99.90th=[ 118], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 7, max= 159, per=26.02%, avg=43.20, stdev=26.29, samples=275 iops : min= 1, max= 39, avg=10.45, stdev= 6.58, samples=275 write: IOPS=10, BW=43.5KiB/s (44.5kB/s)(6260KiB/143934msec) clat (msec): min=23, max=471, avg=79.14, stdev=38.07 lat (msec): min=23, max=471, avg=79.14, stdev=38.07 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 39], 10.00th=[ 43], 20.00th=[ 51], | 30.00th=[ 58], 40.00th=[ 65], 50.00th=[ 71], 60.00th=[ 79], | 70.00th=[ 88], 80.00th=[ 100], 90.00th=[ 125], 95.00th=[ 153], | 99.00th=[ 205], 99.50th=[ 239], 99.90th=[ 414], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 7, max= 96, per=24.77%, avg=43.10, stdev=16.24, samples=286 iops : min= 1, max= 24, avg=10.42, stdev= 4.12, samples=286 lat (msec) : 4=0.75%, 10=29.36%, 20=11.10%, 50=16.70%, 100=31.97% lat (msec) : 250=9.93%, 500=0.20% cpu : usr=0.03%, sys=2.36%, ctx=3434, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31399: Thu May 1 23:52:31 2025 read: IOPS=10, BW=42.1KiB/s (43.2kB/s)(6052KiB/143591msec) clat (msec): min=3, max=115, avg=12.54, stdev=11.87 lat (msec): min=3, max=115, avg=12.54, stdev=11.87 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 16], 90.00th=[ 27], 95.00th=[ 36], | 99.00th=[ 63], 99.50th=[ 74], 99.90th=[ 101], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 7, max= 136, per=26.51%, avg=44.01, stdev=26.43, samples=272 iops : min= 1, max= 34, avg=10.66, stdev= 6.65, samples=272 write: IOPS=10, BW=43.4KiB/s (44.5kB/s)(6236KiB/143591msec) clat (msec): min=18, max=479, avg=79.71, stdev=38.54 lat (msec): min=18, max=479, avg=79.71, stdev=38.54 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 39], 10.00th=[ 44], 20.00th=[ 51], | 30.00th=[ 58], 40.00th=[ 65], 50.00th=[ 71], 60.00th=[ 79], | 70.00th=[ 88], 80.00th=[ 102], 90.00th=[ 129], 95.00th=[ 155], | 99.00th=[ 205], 99.50th=[ 236], 99.90th=[ 414], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 7, max= 88, per=24.77%, avg=43.09, stdev=16.02, samples=286 iops : min= 1, max= 22, avg=10.43, stdev= 4.07, samples=286 lat (msec) : 4=0.68%, 10=30.57%, 20=10.61%, 50=15.82%, 100=31.61% lat (msec) : 250=10.51%, 500=0.20% cpu : usr=0.05%, sys=2.32%, ctx=3449, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=167KiB/s (171kB/s), 41.3KiB/s-42.1KiB/s (42.3kB/s-43.2kB/s), io=23.4MiB (24.6MB), run=143591-144115msec WRITE: bw=174KiB/s (179kB/s), 43.4KiB/s-44.1KiB/s (44.5kB/s-45.2kB/s), io=24.6MiB (25.7MB), run=143591-144115msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31529: Thu May 1 23:52:41 2025 read: IOPS=44, BW=716KiB/s (733kB/s)(5920KiB/8273msec) clat (usec): min=73, max=100770, avg=9402.09, stdev=11574.73 lat (usec): min=74, max=100771, avg=9405.26, stdev=11574.67 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 93], 10.00th=[ 103], 20.00th=[ 120], | 30.00th=[ 151], 40.00th=[ 5276], 50.00th=[ 7242], 60.00th=[ 9634], | 70.00th=[ 12518], 80.00th=[ 15795], 90.00th=[ 19792], 95.00th=[ 28181], | 99.00th=[ 52691], 99.50th=[ 86508], 99.90th=[101188], 99.95th=[101188], | 99.99th=[101188] bw ( KiB/s): min= 192, max= 1408, per=24.54%, avg=710.44, stdev=370.65, samples=16 iops : min= 12, max= 88, avg=44.19, stdev=23.25, samples=16 write: IOPS=48, BW=770KiB/s (788kB/s)(6368KiB/8273msec) clat (usec): min=811, max=160052, avg=11436.92, stdev=17711.43 lat (usec): min=813, max=160055, avg=11438.72, stdev=17711.61 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 1057], 10.00th=[ 1139], 20.00th=[ 1369], | 30.00th=[ 1778], 40.00th=[ 2311], 50.00th=[ 3621], 60.00th=[ 7767], | 70.00th=[ 13173], 80.00th=[ 18220], 90.00th=[ 27395], 95.00th=[ 39060], | 99.00th=[102237], 99.50th=[108528], 99.90th=[160433], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 128, max= 1568, per=25.37%, avg=772.69, stdev=443.34, samples=16 iops : min= 8, max= 98, avg=48.12, stdev=27.78, samples=16 lat (usec) : 100=3.78%, 250=11.98%, 500=0.78%, 750=0.13%, 1000=1.69% lat (msec) : 2=16.54%, 4=9.64%, 10=18.88%, 20=22.53%, 50=11.33% lat (msec) : 100=2.08%, 250=0.65% cpu : usr=0.12%, sys=12.94%, ctx=1167, majf=0, minf=37 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31530: Thu May 1 23:52:41 2025 read: IOPS=46, BW=751KiB/s (769kB/s)(5744KiB/7650msec) clat (usec): min=70, max=62840, avg=9821.85, stdev=9256.45 lat (usec): min=71, max=62842, avg=9831.15, stdev=9257.68 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 100], 10.00th=[ 111], 20.00th=[ 135], | 30.00th=[ 5342], 40.00th=[ 7242], 50.00th=[ 8586], 60.00th=[10421], | 70.00th=[12780], 80.00th=[15533], 90.00th=[20841], 95.00th=[26608], | 99.00th=[40109], 99.50th=[55837], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 186, max= 1277, per=25.96%, avg=751.60, stdev=355.05, samples=15 iops : min= 11, max= 79, avg=46.53, stdev=22.23, samples=15 write: IOPS=53, BW=855KiB/s (876kB/s)(6544KiB/7650msec) clat (usec): min=704, max=266794, avg=9938.33, stdev=19153.52 lat (usec): min=706, max=266796, avg=9940.24, stdev=19153.54 clat percentiles (usec): | 1.00th=[ 947], 5.00th=[ 1057], 10.00th=[ 1139], 20.00th=[ 1385], | 30.00th=[ 1926], 40.00th=[ 2573], 50.00th=[ 3556], 60.00th=[ 6390], | 70.00th=[ 10552], 80.00th=[ 16057], 90.00th=[ 23987], 95.00th=[ 33817], | 99.00th=[ 51119], 99.50th=[ 90702], 99.90th=[267387], 99.95th=[267387], | 99.99th=[267387] bw ( KiB/s): min= 217, max= 1600, per=28.10%, avg=855.93, stdev=364.22, samples=15 iops : min= 13, max= 100, avg=53.07, stdev=22.85, samples=15 lat (usec) : 100=2.21%, 250=9.90%, 500=0.52%, 750=0.26%, 1000=1.43% lat (msec) : 2=14.97%, 4=11.46%, 10=22.66%, 20=24.74%, 50=10.81% lat (msec) : 100=0.78%, 250=0.13%, 500=0.13% cpu : usr=0.34%, sys=14.11%, ctx=1148, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31531: Thu May 1 23:52:41 2025 read: IOPS=48, BW=781KiB/s (799kB/s)(6256KiB/8013msec) clat (usec): min=72, max=57329, avg=8972.14, stdev=8983.08 lat (usec): min=73, max=57330, avg=8973.23, stdev=8983.11 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 94], 10.00th=[ 101], 20.00th=[ 123], | 30.00th=[ 174], 40.00th=[ 5866], 50.00th=[ 7898], 60.00th=[10159], | 70.00th=[12518], 80.00th=[16057], 90.00th=[20579], 95.00th=[25822], | 99.00th=[36963], 99.50th=[43779], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 224, max= 1376, per=26.94%, avg=779.81, stdev=341.03, samples=16 iops : min= 14, max= 86, avg=48.38, stdev=21.32, samples=16 write: IOPS=47, BW=753KiB/s (771kB/s)(6032KiB/8013msec) clat (usec): min=869, max=124345, avg=11690.66, stdev=16472.58 lat (usec): min=871, max=124348, avg=11694.51, stdev=16472.55 clat percentiles (usec): | 1.00th=[ 938], 5.00th=[ 1029], 10.00th=[ 1106], 20.00th=[ 1336], | 30.00th=[ 1942], 40.00th=[ 3195], 50.00th=[ 5407], 60.00th=[ 8586], | 70.00th=[ 13042], 80.00th=[ 18220], 90.00th=[ 30278], 95.00th=[ 41681], | 99.00th=[ 98042], 99.50th=[112722], 99.90th=[124257], 99.95th=[124257], | 99.99th=[124257] bw ( KiB/s): min= 254, max= 1280, per=24.62%, avg=749.88, stdev=338.11, samples=16 iops : min= 15, max= 80, avg=46.50, stdev=21.23, samples=16 lat (usec) : 100=4.82%, 250=11.20%, 500=0.65%, 750=0.91%, 1000=1.30% lat (msec) : 2=13.93%, 4=7.03%, 10=21.48%, 20=24.61%, 50=12.50% lat (msec) : 100=1.17%, 250=0.39% cpu : usr=0.27%, sys=13.11%, ctx=1184, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31532: Thu May 1 23:52:41 2025 read: IOPS=46, BW=750KiB/s (768kB/s)(6032KiB/8041msec) clat (usec): min=71, max=103924, avg=9961.15, stdev=12425.93 lat (usec): min=72, max=103925, avg=9964.21, stdev=12425.67 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 95], 10.00th=[ 102], 20.00th=[ 117], | 30.00th=[ 149], 40.00th=[ 6128], 50.00th=[ 8291], 60.00th=[ 10421], | 70.00th=[ 13042], 80.00th=[ 16319], 90.00th=[ 22152], 95.00th=[ 26346], | 99.00th=[ 72877], 99.50th=[100140], 99.90th=[104334], 99.95th=[104334], | 99.99th=[104334] bw ( KiB/s): min= 192, max= 1184, per=25.58%, avg=740.62, stdev=318.27, samples=16 iops : min= 12, max= 74, avg=46.00, stdev=20.00, samples=16 write: IOPS=48, BW=778KiB/s (797kB/s)(6256KiB/8041msec) clat (usec): min=786, max=124049, avg=10292.14, stdev=15069.03 lat (usec): min=788, max=124051, avg=10293.81, stdev=15069.07 clat percentiles (usec): | 1.00th=[ 840], 5.00th=[ 1004], 10.00th=[ 1106], 20.00th=[ 1221], | 30.00th=[ 1631], 40.00th=[ 2474], 50.00th=[ 3621], 60.00th=[ 6587], | 70.00th=[ 11469], 80.00th=[ 16581], 90.00th=[ 26084], 95.00th=[ 36963], | 99.00th=[ 87557], 99.50th=[ 94897], 99.90th=[124257], 99.95th=[124257], | 99.99th=[124257] bw ( KiB/s): min= 223, max= 1373, per=25.34%, avg=771.94, stdev=365.90, samples=16 iops : min= 13, max= 85, avg=47.94, stdev=23.00, samples=16 lat (usec) : 100=4.04%, 250=11.85%, 500=0.91%, 750=0.13%, 1000=2.60% lat (msec) : 2=15.89%, 4=8.72%, 10=17.97%, 20=23.70%, 50=12.50% lat (msec) : 100=1.43%, 250=0.26% cpu : usr=0.16%, sys=12.96%, ctx=1129, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2895KiB/s (2965kB/s), 716KiB/s-781KiB/s (733kB/s-799kB/s), io=23.4MiB (24.5MB), run=7650-8273msec WRITE: bw=3046KiB/s (3119kB/s), 753KiB/s-855KiB/s (771kB/s-876kB/s), io=24.6MiB (25.8MB), run=7650-8273msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31525: Thu May 1 23:53:07 2025 read: IOPS=10, BW=173KiB/s (177kB/s)(5920KiB/34285msec) clat (msec): min=3, max=105, avg=11.73, stdev=10.83 lat (msec): min=3, max=105, avg=11.73, stdev=10.83 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 21], 95.00th=[ 29], | 99.00th=[ 66], 99.50th=[ 77], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 31, max= 576, per=25.72%, avg=177.95, stdev=96.30, samples=66 iops : min= 1, max= 36, avg=10.70, stdev= 6.09, samples=66 write: IOPS=11, BW=186KiB/s (190kB/s)(6368KiB/34285msec) clat (msec): min=32, max=261, avg=75.04, stdev=31.56 lat (msec): min=32, max=261, avg=75.04, stdev=31.56 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 41], 10.00th=[ 45], 20.00th=[ 51], | 30.00th=[ 58], 40.00th=[ 63], 50.00th=[ 70], 60.00th=[ 77], | 70.00th=[ 83], 80.00th=[ 92], 90.00th=[ 108], 95.00th=[ 138], | 99.00th=[ 215], 99.50th=[ 226], 99.90th=[ 262], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 63, max= 256, per=25.47%, avg=185.40, stdev=54.80, samples=68 iops : min= 3, max= 16, avg=11.16, stdev= 3.49, samples=68 lat (msec) : 4=0.52%, 10=30.99%, 20=11.46%, 50=14.58%, 100=35.42% lat (msec) : 250=6.90%, 500=0.13% cpu : usr=0.04%, sys=2.73%, ctx=908, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31526: Thu May 1 23:53:07 2025 read: IOPS=10, BW=166KiB/s (170kB/s)(5744KiB/34572msec) clat (msec): min=4, max=161, avg=12.70, stdev=13.65 lat (msec): min=4, max=161, avg=12.71, stdev=13.65 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 15], 90.00th=[ 25], 95.00th=[ 36], | 99.00th=[ 72], 99.50th=[ 104], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 31, max= 416, per=25.38%, avg=175.63, stdev=91.23, samples=64 iops : min= 1, max= 26, avg=10.64, stdev= 5.74, samples=64 write: IOPS=11, BW=189KiB/s (194kB/s)(6544KiB/34572msec) clat (msec): min=30, max=258, avg=72.92, stdev=29.95 lat (msec): min=30, max=258, avg=72.93, stdev=29.95 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 42], 10.00th=[ 45], 20.00th=[ 51], | 30.00th=[ 56], 40.00th=[ 62], 50.00th=[ 66], 60.00th=[ 73], | 70.00th=[ 81], 80.00th=[ 91], 90.00th=[ 107], 95.00th=[ 126], | 99.00th=[ 201], 99.50th=[ 207], 99.90th=[ 259], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 63, max= 288, per=25.86%, avg=188.28, stdev=54.49, samples=68 iops : min= 3, max= 18, avg=11.40, stdev= 3.47, samples=68 lat (msec) : 10=28.78%, 20=11.72%, 50=16.02%, 100=36.72%, 250=6.64% lat (msec) : 500=0.13% cpu : usr=0.02%, sys=2.72%, ctx=901, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31527: Thu May 1 23:53:07 2025 read: IOPS=11, BW=190KiB/s (194kB/s)(6256KiB/33000msec) clat (msec): min=3, max=208, avg=12.31, stdev=14.02 lat (msec): min=3, max=208, avg=12.31, stdev=14.02 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 15], 90.00th=[ 25], 95.00th=[ 33], | 99.00th=[ 59], 99.50th=[ 77], 99.90th=[ 209], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 32, max= 447, per=29.03%, avg=200.90, stdev=104.34, samples=62 iops : min= 2, max= 27, avg=12.23, stdev= 6.56, samples=62 write: IOPS=11, BW=183KiB/s (187kB/s)(6032KiB/33000msec) clat (msec): min=29, max=273, avg=74.69, stdev=30.07 lat (msec): min=29, max=273, avg=74.69, stdev=30.07 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 43], 10.00th=[ 46], 20.00th=[ 52], | 30.00th=[ 57], 40.00th=[ 65], 50.00th=[ 70], 60.00th=[ 75], | 70.00th=[ 84], 80.00th=[ 91], 90.00th=[ 105], 95.00th=[ 125], | 99.00th=[ 213], 99.50th=[ 232], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 63, max= 288, per=25.04%, avg=182.28, stdev=54.46, samples=65 iops : min= 3, max= 18, avg=11.06, stdev= 3.43, samples=65 lat (msec) : 4=0.39%, 10=33.98%, 20=8.72%, 50=15.62%, 100=34.51% lat (msec) : 250=6.64%, 500=0.13% cpu : usr=0.02%, sys=2.86%, ctx=905, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31528: Thu May 1 23:53:07 2025 read: IOPS=11, BW=176KiB/s (180kB/s)(6032KiB/34229msec) clat (msec): min=3, max=206, avg=13.00, stdev=16.41 lat (msec): min=3, max=206, avg=13.00, stdev=16.41 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 11], 80.00th=[ 15], 90.00th=[ 23], 95.00th=[ 32], | 99.00th=[ 77], 99.50th=[ 144], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 32, max= 447, per=25.68%, avg=177.67, stdev=95.59, samples=67 iops : min= 2, max= 27, avg=10.70, stdev= 6.05, samples=67 write: IOPS=11, BW=183KiB/s (187kB/s)(6256KiB/34229msec) clat (msec): min=28, max=275, avg=74.84, stdev=28.49 lat (msec): min=28, max=275, avg=74.84, stdev=28.49 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 41], 10.00th=[ 47], 20.00th=[ 53], | 30.00th=[ 60], 40.00th=[ 65], 50.00th=[ 70], 60.00th=[ 78], | 70.00th=[ 84], 80.00th=[ 92], 90.00th=[ 105], 95.00th=[ 123], | 99.00th=[ 186], 99.50th=[ 243], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 31, max= 318, per=25.08%, avg=182.59, stdev=55.80, samples=68 iops : min= 1, max= 19, avg=11.00, stdev= 3.59, samples=68 lat (msec) : 4=0.52%, 10=30.60%, 20=11.07%, 50=13.28%, 100=37.50% lat (msec) : 250=6.90%, 500=0.13% cpu : usr=0.09%, sys=2.68%, ctx=944, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=693KiB/s (709kB/s), 166KiB/s-190KiB/s (170kB/s-194kB/s), io=23.4MiB (24.5MB), run=33000-34572msec WRITE: bw=729KiB/s (746kB/s), 183KiB/s-189KiB/s (187kB/s-194kB/s), io=24.6MiB (25.8MB), run=33000-34572msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31575: Thu May 1 23:53:10 2025 read: IOPS=6, BW=6919KiB/s (7085kB/s)(7168KiB/1036msec) clat (usec): min=541, max=176551, avg=72315.73, stdev=60108.10 lat (usec): min=542, max=176554, avg=72318.95, stdev=60108.79 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 545], 10.00th=[ 545], 20.00th=[ 685], | 30.00th=[ 77071], 40.00th=[ 77071], 50.00th=[ 78119], 60.00th=[ 81265], | 70.00th=[ 81265], 80.00th=[ 92799], 90.00th=[177210], 95.00th=[177210], | 99.00th=[177210], 99.50th=[177210], 99.90th=[177210], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 6095, max= 6095, per=41.43%, avg=6095.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4942KiB/s (5061kB/s)(5120KiB/1036msec) clat (msec): min=59, max=152, avg=102.04, stdev=34.76 lat (msec): min=60, max=152, avg=102.10, stdev=34.79 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 61], 10.00th=[ 61], 20.00th=[ 61], | 30.00th=[ 83], 40.00th=[ 83], 50.00th=[ 103], 60.00th=[ 103], | 70.00th=[ 114], 80.00th=[ 114], 90.00th=[ 153], 95.00th=[ 153], | 99.00th=[ 153], 99.50th=[ 153], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 2031, max= 2031, per=9.86%, avg=2031.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 100=50.00%, 250=33.33% cpu : usr=0.00%, sys=19.71%, ctx=238, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31576: Thu May 1 23:53:10 2025 read: IOPS=1, BW=1471KiB/s (1507kB/s)(2048KiB/1392msec) clat (usec): min=87534, max=98878, avg=93206.84, stdev=8021.42 lat (usec): min=87536, max=98881, avg=93208.85, stdev=8022.24 clat percentiles (usec): | 1.00th=[87557], 5.00th=[87557], 10.00th=[87557], 20.00th=[87557], | 30.00th=[87557], 40.00th=[87557], 50.00th=[87557], 60.00th=[99091], | 70.00th=[99091], 80.00th=[99091], 90.00th=[99091], 95.00th=[99091], | 99.00th=[99091], 99.50th=[99091], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 1723, max= 2031, per=12.76%, avg=1877.00, stdev=217.79, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=7, BW=7356KiB/s (7533kB/s)(10.0MiB/1392msec) clat (msec): min=31, max=221, avg=107.75, stdev=71.93 lat (msec): min=31, max=222, avg=107.83, stdev=71.92 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 43], | 30.00th=[ 50], 40.00th=[ 66], 50.00th=[ 68], 60.00th=[ 79], | 70.00th=[ 197], 80.00th=[ 197], 90.00th=[ 197], 95.00th=[ 222], | 99.00th=[ 222], 99.50th=[ 222], 99.90th=[ 222], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 2031, max= 6895, per=21.67%, avg=4463.00, stdev=3439.37, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (msec) : 50=25.00%, 100=41.67%, 250=33.33% cpu : usr=0.00%, sys=19.77%, ctx=288, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31577: Thu May 1 23:53:10 2025 read: IOPS=6, BW=6321KiB/s (6473kB/s)(6144KiB/972msec) clat (usec): min=518, max=89670, avg=39783.52, stdev=37735.93 lat (usec): min=520, max=89671, avg=39785.03, stdev=37735.98 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 519], 10.00th=[ 519], 20.00th=[ 529], | 30.00th=[ 529], 40.00th=[22152], 50.00th=[22152], 60.00th=[54264], | 70.00th=[71828], 80.00th=[71828], 90.00th=[89654], 95.00th=[89654], | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] write: IOPS=6, BW=6321KiB/s (6473kB/s)(6144KiB/972msec) clat (msec): min=30, max=368, avg=110.41, stdev=128.24 lat (msec): min=30, max=368, avg=110.47, stdev=128.23 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 40], | 30.00th=[ 40], 40.00th=[ 69], 50.00th=[ 69], 60.00th=[ 70], | 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 368], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 4071, max= 4071, per=19.77%, avg=4071.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 50=25.00%, 100=50.00%, 500=8.33% cpu : usr=0.00%, sys=20.80%, ctx=295, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31578: Thu May 1 23:53:10 2025 read: IOPS=5, BW=5251KiB/s (5377kB/s)(5120KiB/975msec) clat (usec): min=496, max=88458, avg=46752.20, stdev=43581.23 lat (usec): min=496, max=88460, avg=46753.39, stdev=43581.70 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 498], 10.00th=[ 498], 20.00th=[ 498], | 30.00th=[ 635], 40.00th=[ 635], 50.00th=[59507], 60.00th=[59507], | 70.00th=[84411], 80.00th=[84411], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] write: IOPS=7, BW=7352KiB/s (7528kB/s)(7168KiB/975msec) clat (msec): min=32, max=289, avg=90.28, stdev=90.60 lat (msec): min=32, max=289, avg=90.34, stdev=90.61 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 36], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 57], 60.00th=[ 85], | 70.00th=[ 85], 80.00th=[ 87], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 6059, max= 6059, per=29.42%, avg=6059.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=25.00%, 100=50.00%, 500=8.33% cpu : usr=0.00%, sys=19.71%, ctx=392, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=14.4MiB/s (15.1MB/s), 1471KiB/s-6919KiB/s (1507kB/s-7085kB/s), io=20.0MiB (20.0MB), run=972-1392msec WRITE: bw=20.1MiB/s (21.1MB/s), 4942KiB/s-7356KiB/s (5061kB/s-7533kB/s), io=28.0MiB (29.4MB), run=972-1392msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31580: Thu May 1 23:53:10 2025 read: IOPS=4, BW=4798KiB/s (4913kB/s)(7168KiB/1494msec) clat (msec): min=41, max=384, avg=129.10, stdev=130.85 lat (msec): min=41, max=384, avg=129.10, stdev=130.85 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 52], 40.00th=[ 52], 50.00th=[ 57], 60.00th=[ 97], | 70.00th=[ 97], 80.00th=[ 230], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 2019, max= 6144, per=31.71%, avg=4081.50, stdev=2916.82, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 write: IOPS=3, BW=3427KiB/s (3509kB/s)(5120KiB/1494msec) clat (msec): min=61, max=125, avg=100.55, stdev=25.77 lat (msec): min=61, max=127, avg=100.98, stdev=26.20 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 63], | 30.00th=[ 92], 40.00th=[ 92], 50.00th=[ 103], 60.00th=[ 103], | 70.00th=[ 122], 80.00th=[ 122], 90.00th=[ 127], 95.00th=[ 127], | 99.00th=[ 127], 99.50th=[ 127], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 2048, max= 2048, per=11.36%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=16.67%, 100=41.67%, 250=33.33%, 500=8.33% cpu : usr=0.13%, sys=3.68%, ctx=36, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31581: Thu May 1 23:53:10 2025 read: IOPS=1, BW=1287KiB/s (1318kB/s)(2048KiB/1591msec) clat (usec): min=34661, max=42992, avg=38826.98, stdev=5891.05 lat (usec): min=34663, max=42995, avg=38829.05, stdev=5891.63 clat percentiles (usec): | 1.00th=[34866], 5.00th=[34866], 10.00th=[34866], 20.00th=[34866], | 30.00th=[34866], 40.00th=[34866], 50.00th=[34866], 60.00th=[43254], | 70.00th=[43254], 80.00th=[43254], 90.00th=[43254], 95.00th=[43254], | 99.00th=[43254], 99.50th=[43254], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 1796, max= 2048, per=14.93%, avg=1922.00, stdev=178.19, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=6, BW=6436KiB/s (6591kB/s)(10.0MiB/1591msec) clat (msec): min=52, max=274, avg=126.60, stdev=69.38 lat (msec): min=52, max=274, avg=126.69, stdev=69.37 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 59], | 30.00th=[ 85], 40.00th=[ 101], 50.00th=[ 108], 60.00th=[ 112], | 70.00th=[ 142], 80.00th=[ 142], 90.00th=[ 218], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 7185, max=10219, per=48.29%, avg=8702.00, stdev=2145.36, samples=2 iops : min= 7, max= 9, avg= 8.00, stdev= 1.41, samples=2 lat (msec) : 50=16.67%, 100=33.33%, 250=41.67%, 500=8.33% cpu : usr=0.00%, sys=4.72%, ctx=46, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31582: Thu May 1 23:53:10 2025 read: IOPS=4, BW=4333KiB/s (4437kB/s)(6144KiB/1418msec) clat (usec): min=21158, max=80190, avg=51388.14, stdev=23611.27 lat (usec): min=21160, max=80192, avg=51389.57, stdev=23611.44 clat percentiles (usec): | 1.00th=[21103], 5.00th=[21103], 10.00th=[21103], 20.00th=[30278], | 30.00th=[30278], 40.00th=[46400], 50.00th=[46400], 60.00th=[55313], | 70.00th=[74974], 80.00th=[74974], 90.00th=[80217], 95.00th=[80217], | 99.00th=[80217], 99.50th=[80217], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 4096, max= 4096, per=31.82%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4333KiB/s (4437kB/s)(6144KiB/1418msec) clat (msec): min=89, max=263, avg=146.76, stdev=74.38 lat (msec): min=89, max=263, avg=146.84, stdev=74.38 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 89], 10.00th=[ 89], 20.00th=[ 97], | 30.00th=[ 97], 40.00th=[ 103], 50.00th=[ 103], 60.00th=[ 111], | 70.00th=[ 218], 80.00th=[ 218], 90.00th=[ 264], 95.00th=[ 264], | 99.00th=[ 264], 99.50th=[ 264], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 2048, max= 4096, per=17.05%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (msec) : 50=25.00%, 100=41.67%, 250=25.00%, 500=8.33% cpu : usr=0.49%, sys=1.76%, ctx=27, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31583: Thu May 1 23:53:10 2025 read: IOPS=3, BW=3502KiB/s (3586kB/s)(5120KiB/1462msec) clat (msec): min=28, max=112, avg=53.90, stdev=34.96 lat (msec): min=28, max=112, avg=53.90, stdev=34.96 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 29], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 41], 60.00th=[ 41], | 70.00th=[ 58], 80.00th=[ 58], 90.00th=[ 113], 95.00th=[ 113], | 99.00th=[ 113], 99.50th=[ 113], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] write: IOPS=4, BW=4903KiB/s (5021kB/s)(7168KiB/1462msec) clat (msec): min=63, max=252, avg=136.27, stdev=63.28 lat (msec): min=63, max=252, avg=136.35, stdev=63.28 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 100], | 30.00th=[ 105], 40.00th=[ 105], 50.00th=[ 120], 60.00th=[ 130], | 70.00th=[ 130], 80.00th=[ 186], 90.00th=[ 253], 95.00th=[ 253], | 99.00th=[ 253], 99.50th=[ 253], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 2043, max= 6144, per=22.72%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (msec) : 50=25.00%, 100=25.00%, 250=41.67%, 500=8.33% cpu : usr=0.00%, sys=4.11%, ctx=29, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.6MiB/s (13.2MB/s), 1287KiB/s-4798KiB/s (1318kB/s-4913kB/s), io=20.0MiB (20.0MB), run=1418-1591msec WRITE: bw=17.6MiB/s (18.5MB/s), 3427KiB/s-6436KiB/s (3509kB/s-6591kB/s), io=28.0MiB (29.4MB), run=1418-1591msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31589: Thu May 1 23:53:13 2025 read: IOPS=2, BW=9533KiB/s (9762kB/s)(12.0MiB/1289msec) clat (msec): min=333, max=510, avg=414.03, stdev=89.39 lat (msec): min=333, max=510, avg=414.03, stdev=89.39 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 334], 10.00th=[ 334], 20.00th=[ 334], | 30.00th=[ 334], 40.00th=[ 397], 50.00th=[ 397], 60.00th=[ 397], | 70.00th=[ 510], 80.00th=[ 510], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 8192, max= 8192, per=74.06%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.16%, sys=1.86%, ctx=24, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31590: Thu May 1 23:53:13 2025 read: IOPS=0, BW=2766KiB/s (2832kB/s)(4096KiB/1481msec) clat (nsec): min=406496k, max=406496k, avg=406495524.00, stdev= 0.00 lat (nsec): min=406499k, max=406499k, avg=406499259.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 405], 10.00th=[ 405], 20.00th=[ 405], | 30.00th=[ 405], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 405], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 8126, max= 8126, per=73.46%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5531KiB/s (5664kB/s)(8192KiB/1481msec) clat (msec): min=159, max=864, avg=512.04, stdev=498.61 lat (msec): min=159, max=864, avg=512.31, stdev=498.63 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 869], | 70.00th=[ 869], 80.00th=[ 869], 90.00th=[ 869], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.20%, sys=4.12%, ctx=135, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31591: Thu May 1 23:53:13 2025 write: IOPS=2, BW=10.7MiB/s (11.2MB/s)(12.0MiB/1126msec) clat (msec): min=162, max=701, avg=364.59, stdev=293.99 lat (msec): min=162, max=702, avg=365.15, stdev=293.85 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 163], 10.00th=[ 163], 20.00th=[ 163], | 30.00th=[ 163], 40.00th=[ 230], 50.00th=[ 230], 60.00th=[ 230], | 70.00th=[ 701], 80.00th=[ 701], 90.00th=[ 701], 95.00th=[ 701], | 99.00th=[ 701], 99.50th=[ 701], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min=16384, max=16384, per=74.05%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.09%, sys=3.64%, ctx=33, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31593: Thu May 1 23:53:13 2025 write: IOPS=2, BW=11.7MiB/s (12.3MB/s)(12.0MiB/1026msec) clat (msec): min=124, max=471, avg=305.02, stdev=173.94 lat (msec): min=125, max=471, avg=307.46, stdev=173.71 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 125], 10.00th=[ 125], 20.00th=[ 125], | 30.00th=[ 125], 40.00th=[ 321], 50.00th=[ 321], 60.00th=[ 321], | 70.00th=[ 472], 80.00th=[ 472], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 8175, max= 8192, per=36.99%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=5.37%, ctx=64, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.8MiB/s (11.3MB/s), 2766KiB/s-9533KiB/s (2832kB/s-9762kB/s), io=16.0MiB (16.8MB), run=1289-1481msec WRITE: bw=21.6MiB/s (22.7MB/s), 5531KiB/s-11.7MiB/s (5664kB/s-12.3MB/s), io=32.0MiB (33.6MB), run=1026-1481msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31594: Thu May 1 23:53:13 2025 read: IOPS=3, BW=14.0MiB/s (14.7MB/s)(12.0MiB/856msec) clat (msec): min=143, max=345, avg=277.35, stdev=116.30 lat (msec): min=143, max=345, avg=277.35, stdev=116.30 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 342], 50.00th=[ 342], 60.00th=[ 342], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 8143, max= 8143, per=62.53%, avg=8143.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.35%, sys=18.83%, ctx=575, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31595: Thu May 1 23:53:13 2025 read: IOPS=0, BW=3256KiB/s (3334kB/s)(4096KiB/1258msec) clat (nsec): min=289271k, max=289271k, avg=289271368.00, stdev= 0.00 lat (nsec): min=289274k, max=289274k, avg=289273795.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 288], 10.00th=[ 288], 20.00th=[ 288], | 30.00th=[ 288], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 288], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 8192, max= 8192, per=62.90%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6512KiB/s (6668kB/s)(8192KiB/1258msec) clat (msec): min=410, max=512, avg=461.76, stdev=71.90 lat (msec): min=411, max=514, avg=462.76, stdev=72.89 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 409], 10.00th=[ 409], 20.00th=[ 409], | 30.00th=[ 409], 40.00th=[ 409], 50.00th=[ 409], 60.00th=[ 514], | 70.00th=[ 514], 80.00th=[ 514], 90.00th=[ 514], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 8192, max= 8192, per=36.48%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.00%, sys=15.91%, ctx=324, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31596: Thu May 1 23:53:13 2025 write: IOPS=2, BW=8422KiB/s (8624kB/s)(12.0MiB/1459msec) clat (msec): min=361, max=589, avg=445.12, stdev=125.32 lat (msec): min=362, max=589, avg=445.40, stdev=125.41 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 363], | 30.00th=[ 363], 40.00th=[ 384], 50.00th=[ 384], 60.00th=[ 384], | 70.00th=[ 592], 80.00th=[ 592], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 8192, max= 8192, per=36.48%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.00%, sys=19.00%, ctx=86, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31597: Thu May 1 23:53:13 2025 write: IOPS=2, BW=9714KiB/s (9947kB/s)(12.0MiB/1265msec) clat (msec): min=285, max=530, avg=371.84, stdev=137.22 lat (msec): min=285, max=535, avg=373.94, stdev=140.29 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 284], 10.00th=[ 284], 20.00th=[ 284], | 30.00th=[ 284], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 531], 80.00th=[ 531], 90.00th=[ 531], 95.00th=[ 531], | 99.00th=[ 531], 99.50th=[ 531], 99.90th=[ 531], 99.95th=[ 531], | 99.99th=[ 531] bw ( KiB/s): min= 8192, max= 8192, per=36.48%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.16%, sys=23.73%, ctx=106, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.7MiB/s (13.3MB/s), 3256KiB/s-14.0MiB/s (3334kB/s-14.7MB/s), io=16.0MiB (16.8MB), run=856-1258msec WRITE: bw=21.9MiB/s (22.0MB/s), 6512KiB/s-9714KiB/s (6668kB/s-9947kB/s), io=32.0MiB (33.6MB), run=1258-1459msec