== sanity test 398b: DIO and buffer IO race ============== 23:19:25 (1746155965) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.34539 s, 21.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=436: Thu May 1 23:20:18 2025 read: IOPS=31, BW=126KiB/s (129kB/s)(5992KiB/47692msec) clat (usec): min=69, max=433820, avg=13542.07, stdev=16531.20 lat (usec): min=69, max=433822, avg=13543.37, stdev=16531.23 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 96], 10.00th=[ 106], 20.00th=[ 133], | 30.00th=[ 5014], 40.00th=[ 8848], 50.00th=[ 11731], 60.00th=[ 14484], | 70.00th=[ 17695], 80.00th=[ 22414], 90.00th=[ 29492], 95.00th=[ 36439], | 99.00th=[ 49021], 99.50th=[ 66323], 99.90th=[ 96994], 99.95th=[434111], | 99.99th=[434111] bw ( KiB/s): min= 8, max= 215, per=24.75%, avg=124.47, stdev=42.60, samples=95 iops : min= 2, max= 53, avg=30.75, stdev=10.66, samples=95 write: IOPS=33, BW=132KiB/s (135kB/s)(6296KiB/47692msec) clat (usec): min=623, max=154250, avg=17177.13, stdev=18116.56 lat (usec): min=624, max=154251, avg=17180.16, stdev=18116.14 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 766], 10.00th=[ 824], 20.00th=[ 1369], | 30.00th=[ 2966], 40.00th=[ 6849], 50.00th=[ 12649], 60.00th=[ 17433], | 70.00th=[ 22676], 80.00th=[ 29754], 90.00th=[ 42206], 95.00th=[ 52691], | 99.00th=[ 74974], 99.50th=[ 92799], 99.90th=[133694], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 31, max= 254, per=25.11%, avg=132.31, stdev=41.08, samples=94 iops : min= 7, max= 63, avg=32.70, stdev=10.25, samples=94 lat (usec) : 100=3.35%, 250=9.24%, 500=0.52%, 750=2.25%, 1000=6.77% lat (msec) : 2=4.07%, 4=5.57%, 10=12.70%, 20=25.42%, 50=26.50% lat (msec) : 100=3.45%, 250=0.13%, 500=0.03% cpu : usr=0.16%, sys=10.44%, ctx=4698, majf=0, minf=37 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=439: Thu May 1 23:20:18 2025 read: IOPS=31, BW=125KiB/s (128kB/s)(5940KiB/47386msec) clat (usec): min=51, max=104264, avg=13229.39, stdev=12860.83 lat (usec): min=52, max=104265, avg=13232.45, stdev=12860.66 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 97], 10.00th=[ 106], 20.00th=[ 133], | 30.00th=[ 4883], 40.00th=[ 9110], 50.00th=[ 11207], 60.00th=[ 13698], | 70.00th=[ 17171], 80.00th=[ 21890], 90.00th=[ 30278], 95.00th=[ 36963], | 99.00th=[ 51119], 99.50th=[ 60031], 99.90th=[ 99091], 99.95th=[104334], | 99.99th=[104334] bw ( KiB/s): min= 7, max= 239, per=24.74%, avg=124.44, stdev=42.16, samples=94 iops : min= 1, max= 59, avg=30.72, stdev=10.57, samples=94 write: IOPS=33, BW=134KiB/s (137kB/s)(6348KiB/47386msec) clat (usec): min=560, max=524523, avg=17262.73, stdev=21974.49 lat (usec): min=560, max=524524, avg=17269.51, stdev=21972.74 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 766], 10.00th=[ 824], 20.00th=[ 1287], | 30.00th=[ 3064], 40.00th=[ 7242], 50.00th=[ 12387], 60.00th=[ 16909], | 70.00th=[ 22414], 80.00th=[ 29754], 90.00th=[ 40633], 95.00th=[ 52167], | 99.00th=[ 76022], 99.50th=[ 86508], 99.90th=[152044], 99.95th=[526386], | 99.99th=[526386] bw ( KiB/s): min= 24, max= 263, per=25.13%, avg=132.46, stdev=43.71, samples=94 iops : min= 6, max= 65, avg=32.73, stdev=10.93, samples=94 lat (usec) : 100=3.19%, 250=9.73%, 500=0.33%, 750=2.34%, 1000=7.00% lat (msec) : 2=4.52%, 4=4.39%, 10=13.44%, 20=25.85%, 50=25.85% lat (msec) : 100=3.16%, 250=0.16%, 750=0.03% cpu : usr=0.19%, sys=10.21%, ctx=4678, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=440: Thu May 1 23:20:18 2025 read: IOPS=31, BW=127KiB/s (130kB/s)(6028KiB/47390msec) clat (usec): min=69, max=445561, avg=13690.39, stdev=16641.63 lat (usec): min=70, max=445562, avg=13691.79, stdev=16641.74 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 97], 10.00th=[ 110], 20.00th=[ 135], | 30.00th=[ 5997], 40.00th=[ 9503], 50.00th=[ 11600], 60.00th=[ 14353], | 70.00th=[ 17695], 80.00th=[ 22414], 90.00th=[ 30540], 95.00th=[ 36439], | 99.00th=[ 52167], 99.50th=[ 60031], 99.90th=[ 87557], 99.95th=[446694], | 99.99th=[446694] bw ( KiB/s): min= 8, max= 247, per=25.05%, avg=126.02, stdev=40.60, samples=94 iops : min= 2, max= 61, avg=31.05, stdev=10.11, samples=94 write: IOPS=33, BW=132KiB/s (135kB/s)(6260KiB/47390msec) clat (usec): min=479, max=170175, avg=16881.27, stdev=17903.19 lat (usec): min=479, max=170178, avg=16883.36, stdev=17904.27 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 758], 10.00th=[ 816], 20.00th=[ 1139], | 30.00th=[ 3032], 40.00th=[ 7111], 50.00th=[ 12649], 60.00th=[ 17433], | 70.00th=[ 22152], 80.00th=[ 28967], 90.00th=[ 41157], 95.00th=[ 50594], | 99.00th=[ 78119], 99.50th=[ 89654], 99.90th=[147850], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 24, max= 238, per=24.78%, avg=130.61, stdev=41.50, samples=94 iops : min= 6, max= 59, avg=32.20, stdev=10.46, samples=94 lat (usec) : 100=3.16%, 250=9.08%, 500=0.39%, 750=2.70%, 1000=7.62% lat (msec) : 2=3.68%, 4=4.56%, 10=12.04%, 20=27.47%, 50=26.17% lat (msec) : 100=2.93%, 250=0.16%, 500=0.03% cpu : usr=0.13%, sys=10.31%, ctx=4701, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=441: Thu May 1 23:20:18 2025 read: IOPS=32, BW=131KiB/s (134kB/s)(6052KiB/46373msec) clat (usec): min=67, max=529501, avg=13279.92, stdev=17943.77 lat (usec): min=67, max=529504, avg=13281.75, stdev=17943.92 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 96], 10.00th=[ 109], 20.00th=[ 137], | 30.00th=[ 5669], 40.00th=[ 8979], 50.00th=[ 11076], 60.00th=[ 13566], | 70.00th=[ 16712], 80.00th=[ 22414], 90.00th=[ 29492], 95.00th=[ 34866], | 99.00th=[ 49546], 99.50th=[ 60031], 99.90th=[ 89654], 99.95th=[530580], | 99.99th=[530580] bw ( KiB/s): min= 8, max= 231, per=25.74%, avg=129.47, stdev=46.45, samples=92 iops : min= 2, max= 57, avg=32.03, stdev=11.63, samples=92 write: IOPS=33, BW=134KiB/s (138kB/s)(6236KiB/46373msec) clat (usec): min=471, max=171324, avg=16633.97, stdev=18404.24 lat (usec): min=472, max=171325, avg=16638.39, stdev=18404.62 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 750], 10.00th=[ 807], 20.00th=[ 1029], | 30.00th=[ 2573], 40.00th=[ 5997], 50.00th=[ 11863], 60.00th=[ 16581], | 70.00th=[ 21890], 80.00th=[ 28705], 90.00th=[ 41157], 95.00th=[ 51119], | 99.00th=[ 73925], 99.50th=[ 88605], 99.90th=[162530], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 24, max= 263, per=25.39%, avg=133.82, stdev=42.14, samples=92 iops : min= 6, max= 65, avg=33.11, stdev=10.50, samples=92 lat (usec) : 100=3.09%, 250=9.15%, 500=0.46%, 750=2.86%, 1000=7.78% lat (msec) : 2=4.46%, 4=4.13%, 10=13.48%, 20=25.78%, 50=25.55% lat (msec) : 100=3.09%, 250=0.13%, 750=0.03% cpu : usr=0.12%, sys=10.33%, ctx=4655, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=503KiB/s (516kB/s), 125KiB/s-131KiB/s (128kB/s-134kB/s), io=23.4MiB (24.6MB), run=46373-47692msec WRITE: bw=527KiB/s (540kB/s), 132KiB/s-134KiB/s (135kB/s-138kB/s), io=24.6MiB (25.7MB), run=46373-47692msec rand-rw: (groupid=0, jobs=1): err= 0: pid=437: Thu May 1 23:22:04 2025 read: IOPS=9, BW=39.1KiB/s (40.0kB/s)(5992KiB/153292msec) clat (msec): min=2, max=471, avg=14.22, stdev=17.33 lat (msec): min=2, max=471, avg=14.22, stdev=17.33 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 11], 60.00th=[ 13], | 70.00th=[ 15], 80.00th=[ 19], 90.00th=[ 27], 95.00th=[ 36], | 99.00th=[ 65], 99.50th=[ 70], 99.90th=[ 249], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 7, max= 167, per=25.67%, avg=40.05, stdev=25.89, samples=295 iops : min= 1, max= 41, avg= 9.69, stdev= 6.53, samples=295 write: IOPS=10, BW=41.1KiB/s (42.1kB/s)(6296KiB/153292msec) clat (msec): min=20, max=310, avg=83.64, stdev=41.53 lat (msec): min=20, max=310, avg=83.64, stdev=41.53 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 40], 10.00th=[ 45], 20.00th=[ 52], | 30.00th=[ 59], 40.00th=[ 66], 50.00th=[ 73], 60.00th=[ 82], | 70.00th=[ 92], 80.00th=[ 109], 90.00th=[ 138], 95.00th=[ 171], | 99.00th=[ 228], 99.50th=[ 257], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 7, max= 88, per=25.02%, avg=40.78, stdev=16.33, samples=304 iops : min= 1, max= 22, avg= 9.87, stdev= 4.18, samples=304 lat (msec) : 4=1.50%, 10=21.45%, 20=17.51%, 50=16.31%, 100=30.83% lat (msec) : 250=12.11%, 500=0.29% cpu : usr=0.08%, sys=2.33%, ctx=3751, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=442: Thu May 1 23:22:04 2025 read: IOPS=9, BW=38.7KiB/s (39.6kB/s)(5940KiB/153581msec) clat (msec): min=2, max=553, avg=14.21, stdev=19.25 lat (msec): min=2, max=553, avg=14.21, stdev=19.25 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 11], 60.00th=[ 12], | 70.00th=[ 14], 80.00th=[ 18], 90.00th=[ 26], 95.00th=[ 35], | 99.00th=[ 68], 99.50th=[ 94], 99.90th=[ 215], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 7, max= 143, per=25.69%, avg=40.08, stdev=26.73, samples=292 iops : min= 1, max= 35, avg= 9.62, stdev= 6.73, samples=292 write: IOPS=10, BW=41.3KiB/s (42.3kB/s)(6348KiB/153581msec) clat (msec): min=23, max=284, avg=83.26, stdev=41.27 lat (msec): min=23, max=284, avg=83.26, stdev=41.27 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 39], 10.00th=[ 44], 20.00th=[ 53], | 30.00th=[ 60], 40.00th=[ 66], 50.00th=[ 72], 60.00th=[ 81], | 70.00th=[ 92], 80.00th=[ 107], 90.00th=[ 140], 95.00th=[ 171], | 99.00th=[ 232], 99.50th=[ 251], 99.90th=[ 264], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 7, max= 111, per=25.18%, avg=41.04, stdev=16.78, samples=305 iops : min= 1, max= 27, avg= 9.86, stdev= 4.27, samples=305 lat (msec) : 4=0.78%, 10=23.24%, 20=16.73%, 50=15.53%, 100=30.92% lat (msec) : 250=12.50%, 500=0.26%, 750=0.03% cpu : usr=0.02%, sys=2.37%, ctx=3781, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=443: Thu May 1 23:22:04 2025 read: IOPS=9, BW=39.4KiB/s (40.4kB/s)(6028KiB/152849msec) clat (msec): min=3, max=247, avg=14.63, stdev=14.07 lat (msec): min=3, max=247, avg=14.64, stdev=14.07 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 13], | 70.00th=[ 15], 80.00th=[ 19], 90.00th=[ 29], 95.00th=[ 39], | 99.00th=[ 66], 99.50th=[ 81], 99.90th=[ 165], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min= 7, max= 152, per=25.73%, avg=40.14, stdev=24.82, samples=296 iops : min= 1, max= 38, avg= 9.69, stdev= 6.27, samples=296 write: IOPS=10, BW=40.0KiB/s (41.9kB/s)(6260KiB/152849msec) clat (msec): min=22, max=481, avg=83.25, stdev=41.20 lat (msec): min=22, max=481, avg=83.25, stdev=41.20 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 40], 10.00th=[ 45], 20.00th=[ 53], | 30.00th=[ 59], 40.00th=[ 66], 50.00th=[ 73], 60.00th=[ 83], | 70.00th=[ 92], 80.00th=[ 106], 90.00th=[ 138], 95.00th=[ 167], | 99.00th=[ 228], 99.50th=[ 243], 99.90th=[ 334], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 7, max= 96, per=24.88%, avg=40.56, stdev=16.27, samples=304 iops : min= 1, max= 24, avg= 9.80, stdev= 4.17, samples=304 lat (msec) : 4=0.62%, 10=21.61%, 20=18.29%, 50=15.62%, 100=32.00% lat (msec) : 250=11.65%, 500=0.20% cpu : usr=0.05%, sys=2.42%, ctx=3770, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=444: Thu May 1 23:22:04 2025 read: IOPS=9, BW=39.7KiB/s (40.7kB/s)(6052KiB/152424msec) clat (msec): min=3, max=214, avg=13.93, stdev=13.16 lat (msec): min=3, max=214, avg=13.93, stdev=13.16 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 11], 60.00th=[ 12], | 70.00th=[ 15], 80.00th=[ 18], 90.00th=[ 27], 95.00th=[ 39], | 99.00th=[ 59], 99.50th=[ 74], 99.90th=[ 163], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 7, max= 136, per=27.15%, avg=42.36, stdev=24.69, samples=282 iops : min= 1, max= 34, avg=10.27, stdev= 6.22, samples=282 write: IOPS=10, BW=40.9KiB/s (41.9kB/s)(6236KiB/152424msec) clat (msec): min=24, max=600, avg=83.94, stdev=42.74 lat (msec): min=24, max=600, avg=83.94, stdev=42.74 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 41], 10.00th=[ 46], 20.00th=[ 54], | 30.00th=[ 61], 40.00th=[ 67], 50.00th=[ 73], 60.00th=[ 82], | 70.00th=[ 91], 80.00th=[ 105], 90.00th=[ 140], 95.00th=[ 171], | 99.00th=[ 228], 99.50th=[ 262], 99.90th=[ 321], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 7, max= 96, per=24.92%, avg=40.63, stdev=15.84, samples=302 iops : min= 1, max= 24, avg= 9.83, stdev= 4.03, samples=302 lat (msec) : 4=0.98%, 10=23.37%, 20=16.76%, 50=15.23%, 100=31.74% lat (msec) : 250=11.65%, 500=0.23%, 750=0.03% cpu : usr=0.07%, sys=2.42%, ctx=3775, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=156KiB/s (160kB/s), 38.7KiB/s-39.7KiB/s (39.6kB/s-40.7kB/s), io=23.4MiB (24.6MB), run=152424-153581msec WRITE: bw=164KiB/s (168kB/s), 40.9KiB/s-41.3KiB/s (41.9kB/s-42.3kB/s), io=24.6MiB (25.7MB), run=152424-153581msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=587: Thu May 1 23:22:11 2025 read: IOPS=62, BW=999KiB/s (1023kB/s)(5920KiB/5926msec) clat (usec): min=54, max=99509, avg=7531.20, stdev=9332.87 lat (usec): min=55, max=99509, avg=7532.20, stdev=9332.87 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 91], 10.00th=[ 100], 20.00th=[ 115], | 30.00th=[ 137], 40.00th=[ 4424], 50.00th=[ 6325], 60.00th=[ 7898], | 70.00th=[ 9634], 80.00th=[12387], 90.00th=[16712], 95.00th=[23987], | 99.00th=[38011], 99.50th=[54264], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 608, max= 1408, per=26.14%, avg=1008.64, stdev=293.53, samples=11 iops : min= 38, max= 88, avg=62.82, stdev=18.49, samples=11 write: IOPS=67, BW=1075KiB/s (1100kB/s)(6368KiB/5926msec) clat (usec): min=826, max=62635, avg=7651.64, stdev=8629.66 lat (usec): min=829, max=62638, avg=7657.23, stdev=8633.31 clat percentiles (usec): | 1.00th=[ 955], 5.00th=[ 1037], 10.00th=[ 1123], 20.00th=[ 1319], | 30.00th=[ 1893], 40.00th=[ 2704], 50.00th=[ 4015], 60.00th=[ 6259], | 70.00th=[ 9372], 80.00th=[12911], 90.00th=[19006], 95.00th=[24249], | 99.00th=[39584], 99.50th=[49021], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 640, max= 1408, per=26.64%, avg=1081.36, stdev=215.92, samples=11 iops : min= 40, max= 88, avg=67.36, stdev=13.62, samples=11 lat (usec) : 100=4.43%, 250=11.98%, 500=0.91%, 750=0.65%, 1000=1.30% lat (msec) : 2=15.89%, 4=9.37%, 10=27.60%, 20=19.92%, 50=7.55% lat (msec) : 100=0.39% cpu : usr=0.25%, sys=17.42%, ctx=1097, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=588: Thu May 1 23:22:11 2025 read: IOPS=61, BW=981KiB/s (1005kB/s)(5744KiB/5855msec) clat (usec): min=60, max=65279, avg=7039.62, stdev=7657.51 lat (usec): min=60, max=65280, avg=7040.60, stdev=7657.63 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 93], 10.00th=[ 100], 20.00th=[ 114], | 30.00th=[ 139], 40.00th=[ 4178], 50.00th=[ 6587], 60.00th=[ 8160], | 70.00th=[ 9634], 80.00th=[11994], 90.00th=[16909], 95.00th=[20055], | 99.00th=[32113], 99.50th=[34341], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 702, max= 1216, per=25.71%, avg=991.82, stdev=189.08, samples=11 iops : min= 43, max= 76, avg=61.91, stdev=11.95, samples=11 write: IOPS=69, BW=1118KiB/s (1145kB/s)(6544KiB/5855msec) clat (usec): min=628, max=67222, avg=8055.95, stdev=9257.14 lat (usec): min=629, max=67223, avg=8057.61, stdev=9257.08 clat percentiles (usec): | 1.00th=[ 889], 5.00th=[ 1012], 10.00th=[ 1106], 20.00th=[ 1254], | 30.00th=[ 1582], 40.00th=[ 2245], 50.00th=[ 3818], 60.00th=[ 6652], | 70.00th=[10552], 80.00th=[14877], 90.00th=[21103], 95.00th=[26084], | 99.00th=[36963], 99.50th=[52167], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 862, max= 1376, per=27.37%, avg=1111.09, stdev=172.67, samples=11 iops : min= 53, max= 86, avg=69.36, stdev=10.92, samples=11 lat (usec) : 100=4.69%, 250=11.72%, 500=0.52%, 750=0.78%, 1000=2.47% lat (msec) : 2=18.10%, 4=7.03%, 10=24.87%, 20=21.48%, 50=7.81% lat (msec) : 100=0.52% cpu : usr=0.34%, sys=16.64%, ctx=1081, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=589: Thu May 1 23:22:11 2025 read: IOPS=67, BW=1075KiB/s (1101kB/s)(6256KiB/5819msec) clat (usec): min=71, max=121614, avg=7895.70, stdev=9529.43 lat (usec): min=71, max=121615, avg=7896.73, stdev=9529.39 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 94], 10.00th=[ 102], 20.00th=[ 124], | 30.00th=[ 334], 40.00th=[ 5800], 50.00th=[ 7308], 60.00th=[ 8717], | 70.00th=[ 10159], 80.00th=[ 12911], 90.00th=[ 16581], 95.00th=[ 20055], | 99.00th=[ 35914], 99.50th=[ 66847], 99.90th=[121111], 99.95th=[121111], | 99.99th=[121111] bw ( KiB/s): min= 798, max= 1632, per=28.47%, avg=1098.27, stdev=260.97, samples=11 iops : min= 49, max= 102, avg=68.27, stdev=16.42, samples=11 write: IOPS=64, BW=1037KiB/s (1061kB/s)(6032KiB/5819msec) clat (usec): min=836, max=61461, avg=7042.73, stdev=8356.81 lat (usec): min=837, max=61463, avg=7044.40, stdev=8356.65 clat percentiles (usec): | 1.00th=[ 955], 5.00th=[ 1045], 10.00th=[ 1106], 20.00th=[ 1237], | 30.00th=[ 1565], 40.00th=[ 2114], 50.00th=[ 3392], 60.00th=[ 5080], | 70.00th=[ 8029], 80.00th=[11863], 90.00th=[18744], 95.00th=[23987], | 99.00th=[34866], 99.50th=[50070], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 670, max= 1309, per=25.26%, avg=1025.45, stdev=205.75, samples=11 iops : min= 41, max= 81, avg=63.73, stdev=12.85, samples=11 lat (usec) : 100=4.30%, 250=10.55%, 500=0.65%, 750=1.17%, 1000=1.30% lat (msec) : 2=18.62%, 4=8.46%, 10=26.17%, 20=21.61%, 50=6.64% lat (msec) : 100=0.39%, 250=0.13% cpu : usr=0.10%, sys=18.08%, ctx=1130, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=590: Thu May 1 23:22:11 2025 read: IOPS=60, BW=972KiB/s (995kB/s)(6032KiB/6207msec) clat (usec): min=62, max=89225, avg=7186.72, stdev=8992.00 lat (usec): min=63, max=89226, avg=7187.67, stdev=8992.14 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 91], 10.00th=[ 98], 20.00th=[ 112], | 30.00th=[ 126], 40.00th=[ 482], 50.00th=[ 6325], 60.00th=[ 7898], | 70.00th=[ 9765], 80.00th=[12649], 90.00th=[16712], 95.00th=[20055], | 99.00th=[41681], 99.50th=[65799], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 160, max= 1248, per=24.51%, avg=945.50, stdev=302.75, samples=12 iops : min= 10, max= 78, avg=58.83, stdev=18.76, samples=12 write: IOPS=62, BW=1008KiB/s (1032kB/s)(6256KiB/6207msec) clat (usec): min=765, max=120069, avg=8528.06, stdev=12486.23 lat (usec): min=766, max=120072, avg=8530.12, stdev=12486.00 clat percentiles (usec): | 1.00th=[ 889], 5.00th=[ 996], 10.00th=[ 1074], 20.00th=[ 1237], | 30.00th=[ 1549], 40.00th=[ 2245], 50.00th=[ 3425], 60.00th=[ 6325], | 70.00th=[ 9503], 80.00th=[ 12911], 90.00th=[ 20317], 95.00th=[ 29754], | 99.00th=[ 65799], 99.50th=[ 90702], 99.90th=[120062], 99.95th=[120062], | 99.99th=[120062] bw ( KiB/s): min= 224, max= 1344, per=24.22%, avg=982.92, stdev=324.78, samples=12 iops : min= 14, max= 84, avg=61.17, stdev=20.22, samples=12 lat (usec) : 100=5.47%, 250=13.28%, 500=0.91%, 1000=2.86% lat (msec) : 2=17.06%, 4=8.20%, 10=23.70%, 20=20.83%, 50=6.77% lat (msec) : 100=0.78%, 250=0.13% cpu : usr=0.18%, sys=16.11%, ctx=1102, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3859KiB/s (3951kB/s), 972KiB/s-1075KiB/s (995kB/s-1101kB/s), io=23.4MiB (24.5MB), run=5819-6207msec WRITE: bw=4060KiB/s (4157kB/s), 1008KiB/s-1118KiB/s (1032kB/s-1145kB/s), io=24.6MiB (25.8MB), run=5819-6207msec rand-rw: (groupid=0, jobs=1): err= 0: pid=591: Thu May 1 23:22:34 2025 read: IOPS=12, BW=200KiB/s (205kB/s)(5920KiB/29544msec) clat (usec): min=3162, max=56705, avg=9580.70, stdev=8166.99 lat (usec): min=3163, max=56706, avg=9582.88, stdev=8166.82 clat percentiles (usec): | 1.00th=[ 3523], 5.00th=[ 3949], 10.00th=[ 4490], 20.00th=[ 5080], | 30.00th=[ 5473], 40.00th=[ 6194], 50.00th=[ 7242], 60.00th=[ 8225], | 70.00th=[ 9110], 80.00th=[10552], 90.00th=[15926], 95.00th=[26084], | 99.00th=[50594], 99.50th=[56361], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 63, max= 702, per=25.03%, avg=202.46, stdev=109.59, samples=57 iops : min= 3, max= 43, avg=12.35, stdev= 6.86, samples=57 write: IOPS=13, BW=216KiB/s (221kB/s)(6368KiB/29544msec) clat (msec): min=26, max=142, avg=64.76, stdev=23.25 lat (msec): min=26, max=142, avg=64.76, stdev=23.25 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 36], 10.00th=[ 40], 20.00th=[ 45], | 30.00th=[ 49], 40.00th=[ 57], 50.00th=[ 63], 60.00th=[ 68], | 70.00th=[ 74], 80.00th=[ 83], 90.00th=[ 96], 95.00th=[ 108], | 99.00th=[ 142], 99.50th=[ 142], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 63, max= 352, per=25.32%, avg=215.71, stdev=64.66, samples=58 iops : min= 3, max= 22, avg=13.17, stdev= 4.20, samples=58 lat (msec) : 4=2.73%, 10=34.24%, 20=7.42%, 50=20.05%, 100=31.64% lat (msec) : 250=3.91% cpu : usr=0.05%, sys=3.04%, ctx=876, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=592: Thu May 1 23:22:34 2025 read: IOPS=12, BW=194KiB/s (199kB/s)(5744KiB/29573msec) clat (usec): min=3274, max=80163, avg=9719.55, stdev=9389.06 lat (usec): min=3275, max=80165, avg=9721.05, stdev=9388.98 clat percentiles (usec): | 1.00th=[ 3523], 5.00th=[ 3884], 10.00th=[ 4293], 20.00th=[ 5014], | 30.00th=[ 5669], 40.00th=[ 6325], 50.00th=[ 6980], 60.00th=[ 7832], | 70.00th=[ 9110], 80.00th=[11076], 90.00th=[15795], 95.00th=[24511], | 99.00th=[52691], 99.50th=[72877], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 31, max= 512, per=24.36%, avg=197.11, stdev=114.22, samples=57 iops : min= 1, max= 32, avg=12.02, stdev= 7.33, samples=57 write: IOPS=13, BW=221KiB/s (227kB/s)(6544KiB/29573msec) clat (msec): min=22, max=169, avg=63.28, stdev=22.55 lat (msec): min=22, max=169, avg=63.28, stdev=22.55 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 34], 10.00th=[ 38], 20.00th=[ 45], | 30.00th=[ 50], 40.00th=[ 55], 50.00th=[ 59], 60.00th=[ 67], | 70.00th=[ 72], 80.00th=[ 81], 90.00th=[ 94], 95.00th=[ 104], | 99.00th=[ 136], 99.50th=[ 142], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 63, max= 384, per=25.88%, avg=220.50, stdev=66.18, samples=58 iops : min= 3, max= 24, avg=13.48, stdev= 4.33, samples=58 lat (msec) : 4=2.73%, 10=32.16%, 20=8.46%, 50=19.01%, 100=34.24% lat (msec) : 250=3.39% cpu : usr=0.02%, sys=3.07%, ctx=882, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=593: Thu May 1 23:22:34 2025 read: IOPS=14, BW=228KiB/s (234kB/s)(6256KiB/27398msec) clat (usec): min=2956, max=72942, avg=9459.84, stdev=8513.61 lat (usec): min=2958, max=72943, avg=9462.73, stdev=8513.16 clat percentiles (usec): | 1.00th=[ 3326], 5.00th=[ 3949], 10.00th=[ 4293], 20.00th=[ 5080], | 30.00th=[ 5538], 40.00th=[ 6325], 50.00th=[ 6980], 60.00th=[ 7570], | 70.00th=[ 8717], 80.00th=[10159], 90.00th=[16581], 95.00th=[28705], | 99.00th=[49021], 99.50th=[53740], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 31, max= 704, per=29.07%, avg=235.15, stdev=132.67, samples=53 iops : min= 1, max= 44, avg=14.42, stdev= 8.41, samples=53 write: IOPS=13, BW=220KiB/s (225kB/s)(6032KiB/27398msec) clat (msec): min=22, max=160, avg=62.27, stdev=21.91 lat (msec): min=22, max=160, avg=62.27, stdev=21.91 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 34], 10.00th=[ 37], 20.00th=[ 44], | 30.00th=[ 49], 40.00th=[ 54], 50.00th=[ 60], 60.00th=[ 65], | 70.00th=[ 71], 80.00th=[ 80], 90.00th=[ 92], 95.00th=[ 102], | 99.00th=[ 133], 99.50th=[ 138], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 64, max= 384, per=25.83%, avg=220.09, stdev=62.03, samples=54 iops : min= 4, max= 24, avg=13.48, stdev= 3.89, samples=54 lat (msec) : 4=3.12%, 10=37.11%, 20=6.51%, 50=20.18%, 100=30.34% lat (msec) : 250=2.73% cpu : usr=0.05%, sys=3.15%, ctx=865, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=594: Thu May 1 23:22:34 2025 read: IOPS=13, BW=210KiB/s (215kB/s)(6032KiB/28688msec) clat (usec): min=3199, max=83517, avg=9918.05, stdev=9188.07 lat (usec): min=3200, max=83616, avg=9919.73, stdev=9190.09 clat percentiles (usec): | 1.00th=[ 3654], 5.00th=[ 4047], 10.00th=[ 4490], 20.00th=[ 5080], | 30.00th=[ 5735], 40.00th=[ 6325], 50.00th=[ 7177], 60.00th=[ 8029], | 70.00th=[ 9110], 80.00th=[10945], 90.00th=[17433], 95.00th=[28181], | 99.00th=[47973], 99.50th=[67634], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 31, max= 576, per=25.69%, avg=207.82, stdev=111.80, samples=57 iops : min= 1, max= 36, avg=12.60, stdev= 7.12, samples=57 write: IOPS=13, BW=218KiB/s (223kB/s)(6256KiB/28688msec) clat (msec): min=23, max=206, avg=63.70, stdev=23.11 lat (msec): min=23, max=206, avg=63.70, stdev=23.11 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 33], 10.00th=[ 40], 20.00th=[ 45], | 30.00th=[ 51], 40.00th=[ 56], 50.00th=[ 60], 60.00th=[ 65], | 70.00th=[ 71], 80.00th=[ 81], 90.00th=[ 93], 95.00th=[ 109], | 99.00th=[ 142], 99.50th=[ 150], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 127, max= 383, per=25.58%, avg=217.93, stdev=59.08, samples=57 iops : min= 7, max= 23, avg=13.23, stdev= 3.81, samples=57 lat (msec) : 4=2.08%, 10=34.90%, 20=7.81%, 50=19.01%, 100=32.55% lat (msec) : 250=3.65% cpu : usr=0.05%, sys=3.01%, ctx=867, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=810KiB/s (829kB/s), 194KiB/s-228KiB/s (199kB/s-234kB/s), io=23.4MiB (24.5MB), run=27398-29573msec WRITE: bw=852KiB/s (873kB/s), 216KiB/s-221KiB/s (221kB/s-227kB/s), io=24.6MiB (25.8MB), run=27398-29573msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=630: Thu May 1 23:22:38 2025 read: IOPS=6, BW=7000KiB/s (7168kB/s)(7168KiB/1024msec) clat (usec): min=483, max=116400, avg=64651.87, stdev=48887.57 lat (usec): min=484, max=116401, avg=64653.67, stdev=48887.79 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 482], 10.00th=[ 482], 20.00th=[ 494], | 30.00th=[ 50594], 40.00th=[ 50594], 50.00th=[ 83362], 60.00th=[ 88605], | 70.00th=[ 88605], 80.00th=[112722], 90.00th=[116917], 95.00th=[116917], | 99.00th=[116917], 99.50th=[116917], 99.90th=[116917], 99.95th=[116917], | 99.99th=[116917] bw ( KiB/s): min= 6131, max= 6131, per=49.97%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 write: IOPS=4, BW=5000KiB/s (5120kB/s)(5120KiB/1024msec) clat (msec): min=50, max=158, avg=88.61, stdev=45.37 lat (msec): min=51, max=158, avg=88.68, stdev=45.38 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 52], | 30.00th=[ 62], 40.00th=[ 62], 50.00th=[ 63], 60.00th=[ 63], | 70.00th=[ 111], 80.00th=[ 111], 90.00th=[ 159], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] lat (usec) : 500=16.67% lat (msec) : 100=50.00%, 250=33.33% cpu : usr=0.10%, sys=15.72%, ctx=244, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=631: Thu May 1 23:22:38 2025 read: IOPS=2, BW=2075KiB/s (2125kB/s)(2048KiB/987msec) clat (msec): min=95, max=124, avg=109.70, stdev=20.33 lat (msec): min=95, max=124, avg=109.70, stdev=20.33 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 95], 10.00th=[ 95], 20.00th=[ 95], | 30.00th=[ 95], 40.00th=[ 95], 50.00th=[ 95], 60.00th=[ 125], | 70.00th=[ 125], 80.00th=[ 125], 90.00th=[ 125], 95.00th=[ 125], | 99.00th=[ 125], 99.50th=[ 125], 99.90th=[ 125], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 2011, max= 2011, per=16.39%, avg=2011.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.1MiB/s (10.6MB/s)(10.0MiB/987msec) clat (msec): min=31, max=111, avg=72.17, stdev=24.80 lat (msec): min=31, max=111, avg=72.25, stdev=24.81 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 42], | 30.00th=[ 58], 40.00th=[ 58], 50.00th=[ 73], 60.00th=[ 83], | 70.00th=[ 89], 80.00th=[ 89], 90.00th=[ 93], 95.00th=[ 112], | 99.00th=[ 112], 99.50th=[ 112], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 8047, max= 8047, per=46.84%, avg=8047.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=16.67%, 100=66.67%, 250=16.67% cpu : usr=0.00%, sys=25.15%, ctx=248, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=632: Thu May 1 23:22:38 2025 read: IOPS=3, BW=3681KiB/s (3770kB/s)(6144KiB/1669msec) clat (usec): min=627, max=94958, avg=71444.09, stdev=35560.94 lat (usec): min=628, max=94960, avg=71445.65, stdev=35560.93 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 627], 10.00th=[ 627], 20.00th=[77071], | 30.00th=[77071], 40.00th=[78119], 50.00th=[78119], 60.00th=[83362], | 70.00th=[94897], 80.00th=[94897], 90.00th=[94897], 95.00th=[94897], | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 4079, max= 5408, per=38.66%, avg=4743.50, stdev=939.74, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 write: IOPS=3, BW=3681KiB/s (3770kB/s)(6144KiB/1669msec) clat (msec): min=55, max=299, avg=171.10, stdev=92.62 lat (msec): min=56, max=299, avg=171.17, stdev=92.62 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 56], 10.00th=[ 56], 20.00th=[ 110], | 30.00th=[ 110], 40.00th=[ 112], 50.00th=[ 112], 60.00th=[ 224], | 70.00th=[ 228], 80.00th=[ 228], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 2043, max= 4079, per=18.87%, avg=3242.33, stdev=1065.35, samples=3 iops : min= 1, max= 3, avg= 2.33, stdev= 1.15, samples=3 lat (usec) : 750=8.33% lat (msec) : 100=50.00%, 250=33.33%, 500=8.33% cpu : usr=0.12%, sys=15.35%, ctx=383, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=633: Thu May 1 23:22:38 2025 read: IOPS=4, BW=4613KiB/s (4723kB/s)(5120KiB/1110msec) clat (usec): min=612, max=72332, avg=36202.36, stdev=33109.89 lat (usec): min=614, max=72332, avg=36203.90, stdev=33109.60 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 611], 10.00th=[ 611], 20.00th=[ 611], | 30.00th=[ 9110], 40.00th=[ 9110], 50.00th=[30540], 60.00th=[30540], | 70.00th=[68682], 80.00th=[68682], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 8175, max= 8175, per=66.63%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6458KiB/s (6613kB/s)(7168KiB/1110msec) clat (msec): min=38, max=299, avg=118.33, stdev=89.94 lat (msec): min=38, max=299, avg=118.42, stdev=89.93 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 53], | 30.00th=[ 65], 40.00th=[ 65], 50.00th=[ 90], 60.00th=[ 140], | 70.00th=[ 140], 80.00th=[ 144], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 2043, max= 8062, per=29.41%, avg=5052.50, stdev=4256.08, samples=2 iops : min= 1, max= 7, avg= 4.00, stdev= 4.24, samples=2 lat (usec) : 750=8.33% lat (msec) : 10=8.33%, 50=16.67%, 100=41.67%, 250=16.67%, 500=8.33% cpu : usr=0.18%, sys=18.49%, ctx=243, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.0MiB/s (12.6MB/s), 2075KiB/s-7000KiB/s (2125kB/s-7168kB/s), io=20.0MiB (20.0MB), run=987-1669msec WRITE: bw=16.8MiB/s (17.6MB/s), 3681KiB/s-10.1MiB/s (3770kB/s-10.6MB/s), io=28.0MiB (29.4MB), run=987-1669msec rand-rw: (groupid=0, jobs=1): err= 0: pid=626: Thu May 1 23:22:38 2025 read: IOPS=3, BW=4045KiB/s (4142kB/s)(7168KiB/1772msec) clat (msec): min=27, max=273, avg=98.26, stdev=86.07 lat (msec): min=27, max=273, avg=98.26, stdev=86.07 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 33], | 30.00th=[ 52], 40.00th=[ 52], 50.00th=[ 82], 60.00th=[ 85], | 70.00th=[ 85], 80.00th=[ 138], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 4087, max= 6095, per=47.43%, avg=5091.00, stdev=1419.87, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2889KiB/s (2959kB/s)(5120KiB/1772msec) clat (msec): min=108, max=272, avg=175.22, stdev=70.13 lat (msec): min=108, max=272, avg=175.29, stdev=70.14 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 109], 10.00th=[ 109], 20.00th=[ 109], | 30.00th=[ 134], 40.00th=[ 134], 50.00th=[ 136], 60.00th=[ 136], | 70.00th=[ 226], 80.00th=[ 226], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 2019, max= 6131, per=27.12%, avg=4075.00, stdev=2907.62, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 lat (msec) : 50=16.67%, 100=25.00%, 250=41.67%, 500=16.67% cpu : usr=0.56%, sys=2.94%, ctx=33, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=627: Thu May 1 23:22:38 2025 read: IOPS=1, BW=1073KiB/s (1099kB/s)(2048KiB/1908msec) clat (msec): min=57, max=126, avg=91.95, stdev=49.28 lat (msec): min=57, max=126, avg=91.95, stdev=49.28 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 57], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 57], 60.00th=[ 127], | 70.00th=[ 127], 80.00th=[ 127], 90.00th=[ 127], 95.00th=[ 127], | 99.00th=[ 127], 99.50th=[ 127], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 2027, max= 2048, per=18.98%, avg=2037.50, stdev=14.85, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=5, BW=5367KiB/s (5496kB/s)(10.0MiB/1908msec) clat (msec): min=48, max=295, avg=166.34, stdev=92.77 lat (msec): min=48, max=295, avg=166.53, stdev=92.85 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 74], | 30.00th=[ 86], 40.00th=[ 92], 50.00th=[ 126], 60.00th=[ 188], | 70.00th=[ 268], 80.00th=[ 268], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 2027, max= 8192, per=27.21%, avg=4089.00, stdev=3553.32, samples=3 iops : min= 1, max= 8, avg= 3.67, stdev= 3.79, samples=3 lat (msec) : 50=8.33%, 100=33.33%, 250=33.33%, 500=25.00% cpu : usr=0.00%, sys=2.94%, ctx=41, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=628: Thu May 1 23:22:38 2025 read: IOPS=3, BW=3842KiB/s (3935kB/s)(6144KiB/1599msec) clat (msec): min=36, max=113, avg=60.21, stdev=27.31 lat (msec): min=36, max=113, avg=60.21, stdev=27.31 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 53], | 70.00th=[ 62], 80.00th=[ 62], 90.00th=[ 114], 95.00th=[ 114], | 99.00th=[ 114], 99.50th=[ 114], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 4096, max= 6047, per=47.25%, avg=5071.50, stdev=1379.57, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3842KiB/s (3935kB/s)(6144KiB/1599msec) clat (msec): min=104, max=308, avg=200.26, stdev=68.91 lat (msec): min=104, max=308, avg=200.36, stdev=68.91 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 105], 10.00th=[ 105], 20.00th=[ 163], | 30.00th=[ 163], 40.00th=[ 194], 50.00th=[ 194], 60.00th=[ 197], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 4031, max= 4096, per=27.09%, avg=4071.33, stdev=35.22, samples=3 iops : min= 3, max= 4, avg= 3.33, stdev= 0.58, samples=3 lat (msec) : 50=16.67%, 100=25.00%, 250=50.00%, 500=8.33% cpu : usr=0.19%, sys=2.57%, ctx=32, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=629: Thu May 1 23:22:38 2025 read: IOPS=2, BW=2891KiB/s (2960kB/s)(5120KiB/1771msec) clat (msec): min=24, max=188, avg=73.43, stdev=67.42 lat (msec): min=24, max=188, avg=73.43, stdev=67.42 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 26], 40.00th=[ 26], 50.00th=[ 64], 60.00th=[ 64], | 70.00th=[ 66], 80.00th=[ 66], 90.00th=[ 188], 95.00th=[ 188], | 99.00th=[ 188], 99.50th=[ 188], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 7984, max= 7984, per=74.39%, avg=7984.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=3, BW=4047KiB/s (4145kB/s)(7168KiB/1771msec) clat (msec): min=92, max=284, avg=168.52, stdev=86.23 lat (msec): min=93, max=284, avg=168.60, stdev=86.23 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 93], 10.00th=[ 93], 20.00th=[ 97], | 30.00th=[ 110], 40.00th=[ 110], 50.00th=[ 111], 60.00th=[ 207], | 70.00th=[ 207], 80.00th=[ 279], 90.00th=[ 284], 95.00th=[ 284], | 99.00th=[ 284], 99.50th=[ 284], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 1996, max= 4087, per=18.03%, avg=2708.67, stdev=1193.90, samples=3 iops : min= 1, max= 3, avg= 1.67, stdev= 1.15, samples=3 lat (msec) : 50=16.67%, 100=33.33%, 250=33.33%, 500=16.67% cpu : usr=0.00%, sys=3.67%, ctx=43, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.5MiB/s (10.0MB/s), 1073KiB/s-4045KiB/s (1099kB/s-4142kB/s), io=20.0MiB (20.0MB), run=1599-1908msec WRITE: bw=14.7MiB/s (15.4MB/s), 2889KiB/s-5367KiB/s (2959kB/s-5496kB/s), io=28.0MiB (29.4MB), run=1599-1908msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=645: Thu May 1 23:22:41 2025 read: IOPS=2, BW=10.5MiB/s (10.0MB/s)(12.0MiB/1147msec) clat (msec): min=248, max=454, avg=332.28, stdev=108.07 lat (msec): min=248, max=454, avg=332.28, stdev=108.07 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 249], 10.00th=[ 249], 20.00th=[ 249], | 30.00th=[ 249], 40.00th=[ 296], 50.00th=[ 296], 60.00th=[ 296], | 70.00th=[ 456], 80.00th=[ 456], 90.00th=[ 456], 95.00th=[ 456], | 99.00th=[ 456], 99.50th=[ 456], 99.90th=[ 456], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.26%, sys=14.49%, ctx=579, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=646: Thu May 1 23:22:41 2025 read: IOPS=0, BW=2612KiB/s (2675kB/s)(4096KiB/1568msec) clat (nsec): min=349062k, max=349062k, avg=349062048.00, stdev= 0.00 lat (nsec): min=349065k, max=349065k, avg=349065330.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 351], 10.00th=[ 351], 20.00th=[ 351], | 30.00th=[ 351], 40.00th=[ 351], 50.00th=[ 351], 60.00th=[ 351], | 70.00th=[ 351], 80.00th=[ 351], 90.00th=[ 351], 95.00th=[ 351], | 99.00th=[ 351], 99.50th=[ 351], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 8192, max= 8192, per=78.41%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5224KiB/s (5350kB/s)(8192KiB/1568msec) clat (msec): min=370, max=594, avg=482.92, stdev=158.29 lat (msec): min=371, max=597, avg=484.55, stdev=160.10 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 372], 10.00th=[ 372], 20.00th=[ 372], | 30.00th=[ 372], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 592], | 70.00th=[ 592], 80.00th=[ 592], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.00%, sys=13.98%, ctx=166, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=647: Thu May 1 23:22:41 2025 write: IOPS=1, BW=7511KiB/s (7691kB/s)(12.0MiB/1636msec) clat (msec): min=291, max=681, avg=527.97, stdev=208.09 lat (msec): min=291, max=682, avg=528.57, stdev=207.90 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 292], 10.00th=[ 292], 20.00th=[ 292], | 30.00th=[ 292], 40.00th=[ 609], 50.00th=[ 609], 60.00th=[ 609], | 70.00th=[ 684], 80.00th=[ 684], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 8110, max= 8110, per=40.49%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.24%, sys=17.00%, ctx=82, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=648: Thu May 1 23:22:41 2025 write: IOPS=2, BW=10.0MiB/s (11.5MB/s)(12.0MiB/1095msec) clat (msec): min=181, max=607, avg=326.44, stdev=243.47 lat (msec): min=181, max=607, avg=326.77, stdev=243.46 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 182], 10.00th=[ 182], 20.00th=[ 182], | 30.00th=[ 182], 40.00th=[ 190], 50.00th=[ 190], 60.00th=[ 190], | 70.00th=[ 609], 80.00th=[ 609], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 8192, max= 8192, per=40.90%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.09%, sys=25.14%, ctx=61, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.2MiB/s (10.7MB/s), 2612KiB/s-10.5MiB/s (2675kB/s-10.0MB/s), io=16.0MiB (16.8MB), run=1147-1568msec WRITE: bw=19.6MiB/s (20.5MB/s), 5224KiB/s-10.0MiB/s (5350kB/s-11.5MB/s), io=32.0MiB (33.6MB), run=1095-1636msec rand-rw: (groupid=0, jobs=1): err= 0: pid=641: Thu May 1 23:22:41 2025 read: IOPS=6, BW=25.5MiB/s (26.8MB/s)(12.0MiB/470msec) clat (msec): min=70, max=301, avg=150.63, stdev=130.84 lat (msec): min=70, max=301, avg=150.63, stdev=130.84 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 71], 10.00th=[ 71], 20.00th=[ 71], | 30.00th=[ 71], 40.00th=[ 81], 50.00th=[ 81], 60.00th=[ 81], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] lat (msec) : 100=66.67%, 500=33.33% cpu : usr=0.21%, sys=2.99%, ctx=10, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=642: Thu May 1 23:22:41 2025 read: IOPS=0, BW=2123KiB/s (2174kB/s)(4096KiB/1929msec) clat (nsec): min=802267k, max=802267k, avg=802266640.00, stdev= 0.00 lat (nsec): min=802269k, max=802269k, avg=802269050.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 802], 5.00th=[ 802], 10.00th=[ 802], 20.00th=[ 802], | 30.00th=[ 802], 40.00th=[ 802], 50.00th=[ 802], 60.00th=[ 802], | 70.00th=[ 802], 80.00th=[ 802], 90.00th=[ 802], 95.00th=[ 802], | 99.00th=[ 802], 99.50th=[ 802], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 7543, max= 7543, per=88.81%, avg=7543.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4247KiB/s (4349kB/s)(8192KiB/1929msec) clat (msec): min=171, max=895, avg=533.33, stdev=512.33 lat (msec): min=171, max=895, avg=533.70, stdev=512.32 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 171], 50.00th=[ 171], 60.00th=[ 894], | 70.00th=[ 894], 80.00th=[ 894], 90.00th=[ 894], 95.00th=[ 894], | 99.00th=[ 894], 99.50th=[ 894], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] lat (msec) : 250=33.33%, 1000=66.67% cpu : usr=0.05%, sys=4.36%, ctx=60, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=643: Thu May 1 23:22:41 2025 write: IOPS=3, BW=12.8MiB/s (13.4MB/s)(12.0MiB/939msec) clat (msec): min=265, max=363, avg=299.15, stdev=55.98 lat (msec): min=266, max=364, avg=299.56, stdev=55.88 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 266], 10.00th=[ 266], 20.00th=[ 266], | 30.00th=[ 266], 40.00th=[ 268], 50.00th=[ 268], 60.00th=[ 268], | 70.00th=[ 363], 80.00th=[ 363], 90.00th=[ 363], 95.00th=[ 363], | 99.00th=[ 363], 99.50th=[ 363], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 8110, max= 8110, per=47.74%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=100.00% cpu : usr=0.32%, sys=2.66%, ctx=10, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=644: Thu May 1 23:22:41 2025 write: IOPS=1, BW=7074KiB/s (7244kB/s)(12.0MiB/1737msec) clat (msec): min=322, max=798, avg=511.18, stdev=252.76 lat (msec): min=322, max=798, avg=511.47, stdev=252.82 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 321], 10.00th=[ 321], 20.00th=[ 321], | 30.00th=[ 321], 40.00th=[ 414], 50.00th=[ 414], 60.00th=[ 414], | 70.00th=[ 802], 80.00th=[ 802], 90.00th=[ 802], 95.00th=[ 802], | 99.00th=[ 802], 99.50th=[ 802], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 8175, max= 8192, per=48.18%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67%, 1000=33.33% cpu : usr=0.00%, sys=4.32%, ctx=59, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8494KiB/s (8697kB/s), 2123KiB/s-25.5MiB/s (2174kB/s-26.8MB/s), io=16.0MiB (16.8MB), run=470-1929msec WRITE: bw=16.6MiB/s (17.4MB/s), 4247KiB/s-12.8MiB/s (4349kB/s-13.4MB/s), io=32.0MiB (33.6MB), run=939-1929msec