== sanity test 398b: DIO and buffer IO race ============== 21:43:17 (1746150197) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.15841 s, 23.3 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23643: Thu May 1 21:44:04 2025 read: IOPS=35, BW=143KiB/s (146kB/s)(5992KiB/41919msec) clat (usec): min=71, max=279226, avg=15864.23, stdev=12802.33 lat (usec): min=71, max=279228, avg=15865.63, stdev=12802.43 clat percentiles (usec): | 1.00th=[ 96], 5.00th=[ 129], 10.00th=[ 5407], 20.00th=[ 8029], | 30.00th=[ 9765], 40.00th=[ 11600], 50.00th=[ 13829], 60.00th=[ 16319], | 70.00th=[ 19006], 80.00th=[ 22152], 90.00th=[ 27919], 95.00th=[ 33817], | 99.00th=[ 54264], 99.50th=[ 66847], 99.90th=[105382], 99.95th=[278922], | 99.99th=[278922] bw ( KiB/s): min= 48, max= 232, per=24.79%, avg=141.80, stdev=37.65, samples=83 iops : min= 12, max= 58, avg=35.12, stdev= 9.43, samples=83 write: IOPS=37, BW=150KiB/s (154kB/s)(6296KiB/41919msec) clat (usec): min=610, max=95275, avg=11437.78, stdev=12388.55 lat (usec): min=610, max=95277, avg=11440.32, stdev=12387.72 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 766], 10.00th=[ 840], 20.00th=[ 1106], | 30.00th=[ 2057], 40.00th=[ 3949], 50.00th=[ 8160], 60.00th=[10945], | 70.00th=[14222], 80.00th=[20055], 90.00th=[28967], 95.00th=[35390], | 99.00th=[55313], 99.50th=[63177], 99.90th=[83362], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 32, max= 239, per=24.85%, avg=148.86, stdev=40.77, samples=83 iops : min= 8, max= 59, avg=36.90, stdev=10.17, samples=83 lat (usec) : 100=0.91%, 250=2.18%, 500=0.16%, 750=1.86%, 1000=7.58% lat (msec) : 2=5.99%, 4=5.70%, 10=20.18%, 20=32.42%, 50=21.68% lat (msec) : 100=1.24%, 250=0.07%, 500=0.03% cpu : usr=0.23%, sys=11.05%, ctx=5100, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23644: Thu May 1 21:44:04 2025 read: IOPS=35, BW=142KiB/s (145kB/s)(5940KiB/41869msec) clat (usec): min=74, max=73192, avg=15721.96, stdev=10324.19 lat (usec): min=74, max=73193, avg=15724.34, stdev=10324.09 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 135], 10.00th=[ 5997], 20.00th=[ 8160], | 30.00th=[ 9896], 40.00th=[11731], 50.00th=[13829], 60.00th=[15926], | 70.00th=[18744], 80.00th=[22414], 90.00th=[28705], 95.00th=[34341], | 99.00th=[51643], 99.50th=[60031], 99.90th=[69731], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 24, max= 223, per=24.62%, avg=140.83, stdev=37.28, samples=83 iops : min= 6, max= 55, avg=34.78, stdev= 9.34, samples=83 write: IOPS=37, BW=152KiB/s (155kB/s)(6348KiB/41869msec) clat (usec): min=642, max=293628, avg=11584.22, stdev=14145.15 lat (usec): min=643, max=293628, avg=11586.77, stdev=14145.10 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 766], 10.00th=[ 824], 20.00th=[ 1012], | 30.00th=[ 2024], 40.00th=[ 4293], 50.00th=[ 8160], 60.00th=[ 10945], | 70.00th=[ 14484], 80.00th=[ 20317], 90.00th=[ 28443], 95.00th=[ 35914], | 99.00th=[ 51643], 99.50th=[ 58459], 99.90th=[106431], 99.95th=[295699], | 99.99th=[295699] bw ( KiB/s): min= 64, max= 253, per=24.99%, avg=149.67, stdev=47.45, samples=83 iops : min= 16, max= 63, avg=36.99, stdev=11.87, samples=83 lat (usec) : 100=0.88%, 250=2.18%, 500=0.20%, 750=1.89%, 1000=8.33% lat (msec) : 2=5.24%, 4=4.85%, 10=20.67%, 20=32.55%, 50=22.14% lat (msec) : 100=1.01%, 250=0.03%, 500=0.03% cpu : usr=0.15%, sys=11.10%, ctx=5055, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23645: Thu May 1 21:44:04 2025 read: IOPS=36, BW=145KiB/s (148kB/s)(6028KiB/41680msec) clat (usec): min=73, max=91148, avg=15488.39, stdev=10547.44 lat (usec): min=73, max=91149, avg=15490.51, stdev=10547.24 clat percentiles (usec): | 1.00th=[ 96], 5.00th=[ 124], 10.00th=[ 4948], 20.00th=[ 7898], | 30.00th=[ 9896], 40.00th=[11600], 50.00th=[13829], 60.00th=[16057], | 70.00th=[18744], 80.00th=[22152], 90.00th=[27919], 95.00th=[34341], | 99.00th=[51119], 99.50th=[55837], 99.90th=[78119], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 47, max= 232, per=25.17%, avg=143.98, stdev=35.86, samples=83 iops : min= 11, max= 58, avg=35.57, stdev= 9.04, samples=83 write: IOPS=37, BW=150KiB/s (154kB/s)(6260KiB/41680msec) clat (usec): min=621, max=267445, avg=11635.06, stdev=13775.77 lat (usec): min=621, max=267452, avg=11637.68, stdev=13776.50 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 775], 10.00th=[ 824], 20.00th=[ 1020], | 30.00th=[ 2089], 40.00th=[ 3884], 50.00th=[ 7570], 60.00th=[ 11207], | 70.00th=[ 15008], 80.00th=[ 20841], 90.00th=[ 28443], 95.00th=[ 35914], | 99.00th=[ 51643], 99.50th=[ 59507], 99.90th=[ 90702], 99.95th=[267387], | 99.99th=[267387] bw ( KiB/s): min= 63, max= 248, per=24.92%, avg=149.28, stdev=42.89, samples=83 iops : min= 15, max= 62, avg=36.88, stdev=10.78, samples=83 lat (usec) : 100=0.85%, 250=2.80%, 500=0.16%, 750=1.60%, 1000=8.53% lat (msec) : 2=5.27%, 4=5.76%, 10=19.17%, 20=32.19%, 50=22.53% lat (msec) : 100=1.11%, 500=0.03% cpu : usr=0.19%, sys=11.21%, ctx=5132, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23646: Thu May 1 21:44:04 2025 read: IOPS=36, BW=146KiB/s (150kB/s)(6052KiB/41314msec) clat (usec): min=51, max=296887, avg=15335.36, stdev=12484.78 lat (usec): min=51, max=296889, avg=15338.08, stdev=12484.22 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 125], 10.00th=[ 5473], 20.00th=[ 7898], | 30.00th=[ 9634], 40.00th=[ 11207], 50.00th=[ 13173], 60.00th=[ 15533], | 70.00th=[ 18220], 80.00th=[ 21627], 90.00th=[ 27132], 95.00th=[ 32637], | 99.00th=[ 50594], 99.50th=[ 58983], 99.90th=[ 92799], 99.95th=[295699], | 99.99th=[295699] bw ( KiB/s): min= 47, max= 232, per=25.49%, avg=145.79, stdev=36.00, samples=82 iops : min= 11, max= 58, avg=36.10, stdev= 9.03, samples=82 write: IOPS=37, BW=151KiB/s (155kB/s)(6236KiB/41314msec) clat (usec): min=537, max=114780, avg=11521.13, stdev=12738.49 lat (usec): min=538, max=114782, avg=11525.05, stdev=12737.88 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 766], 10.00th=[ 816], 20.00th=[ 1012], | 30.00th=[ 1811], 40.00th=[ 3523], 50.00th=[ 8029], 60.00th=[ 11076], | 70.00th=[ 14746], 80.00th=[ 20579], 90.00th=[ 28443], 95.00th=[ 35914], | 99.00th=[ 54789], 99.50th=[ 61080], 99.90th=[ 84411], 99.95th=[114820], | 99.99th=[114820] bw ( KiB/s): min= 55, max= 279, per=25.09%, avg=150.28, stdev=44.38, samples=82 iops : min= 13, max= 69, avg=37.21, stdev=11.13, samples=82 lat (usec) : 100=0.94%, 250=2.60%, 500=0.07%, 750=1.95%, 1000=8.20% lat (msec) : 2=6.15%, 4=5.01%, 10=19.60%, 20=32.55%, 50=21.55% lat (msec) : 100=1.30%, 250=0.03%, 500=0.03% cpu : usr=0.21%, sys=11.02%, ctx=5033, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=573KiB/s (587kB/s), 142KiB/s-146KiB/s (145kB/s-150kB/s), io=23.4MiB (24.6MB), run=41314-41919msec WRITE: bw=600KiB/s (614kB/s), 150KiB/s-152KiB/s (154kB/s-155kB/s), io=24.6MiB (25.7MB), run=41314-41919msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23639: Thu May 1 21:44:27 2025 read: IOPS=23, BW=92.3KiB/s (94.6kB/s)(5992KiB/64894msec) clat (msec): min=3, max=281, avg=15.73, stdev=14.55 lat (msec): min=3, max=281, avg=15.73, stdev=14.55 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 13], | 70.00th=[ 16], 80.00th=[ 20], 90.00th=[ 32], 95.00th=[ 43], | 99.00th=[ 72], 99.50th=[ 85], 99.90th=[ 107], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 24, max= 200, per=25.09%, avg=91.59, stdev=39.90, samples=129 iops : min= 6, max= 50, avg=22.57, stdev= 9.99, samples=129 write: IOPS=24, BW=97.0KiB/s (99.3kB/s)(6296KiB/64894msec) clat (msec): min=5, max=139, avg=26.15, stdev=14.25 lat (msec): min=5, max=139, avg=26.15, stdev=14.26 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 25], | 70.00th=[ 29], 80.00th=[ 35], 90.00th=[ 44], 95.00th=[ 54], | 99.00th=[ 74], 99.50th=[ 96], 99.90th=[ 127], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 8, max= 192, per=25.11%, avg=96.18, stdev=36.91, samples=129 iops : min= 2, max= 48, avg=23.72, stdev= 9.27, samples=129 lat (msec) : 4=0.10%, 10=19.89%, 20=40.01%, 50=35.42%, 100=4.30% lat (msec) : 250=0.26%, 500=0.03% cpu : usr=0.11%, sys=4.95%, ctx=3392, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23640: Thu May 1 21:44:27 2025 read: IOPS=22, BW=90.5KiB/s (92.7kB/s)(5940KiB/65618msec) clat (msec): min=3, max=300, avg=16.33, stdev=15.37 lat (msec): min=3, max=300, avg=16.33, stdev=15.37 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 16], 80.00th=[ 22], 90.00th=[ 33], 95.00th=[ 45], | 99.00th=[ 71], 99.50th=[ 91], 99.90th=[ 144], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 8, max= 207, per=24.61%, avg=89.82, stdev=41.36, samples=131 iops : min= 2, max= 51, avg=22.02, stdev=10.36, samples=131 write: IOPS=24, BW=96.7KiB/s (99.1kB/s)(6348KiB/65618msec) clat (msec): min=6, max=125, avg=25.98, stdev=14.39 lat (msec): min=6, max=125, avg=25.98, stdev=14.39 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 25], | 70.00th=[ 28], 80.00th=[ 35], 90.00th=[ 46], 95.00th=[ 55], | 99.00th=[ 79], 99.50th=[ 90], 99.90th=[ 121], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 31, max= 190, per=25.06%, avg=95.96, stdev=39.29, samples=131 iops : min= 7, max= 47, avg=23.56, stdev= 9.86, samples=131 lat (msec) : 4=0.23%, 10=19.50%, 20=39.97%, 50=34.99%, 100=5.01% lat (msec) : 250=0.26%, 500=0.03% cpu : usr=0.14%, sys=4.92%, ctx=3412, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23641: Thu May 1 21:44:27 2025 read: IOPS=23, BW=92.2KiB/s (94.4kB/s)(6028KiB/65355msec) clat (msec): min=3, max=140, avg=15.81, stdev=13.39 lat (msec): min=3, max=140, avg=15.81, stdev=13.39 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 32], 95.00th=[ 42], | 99.00th=[ 71], 99.50th=[ 85], 99.90th=[ 121], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 15, max= 215, per=25.09%, avg=91.57, stdev=39.58, samples=130 iops : min= 3, max= 53, avg=22.57, stdev= 9.92, samples=130 write: IOPS=23, BW=95.8KiB/s (98.1kB/s)(6260KiB/65355msec) clat (msec): min=5, max=270, avg=26.41, stdev=15.43 lat (msec): min=5, max=270, avg=26.41, stdev=15.43 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 17], | 30.00th=[ 18], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 26], | 70.00th=[ 29], 80.00th=[ 35], 90.00th=[ 45], 95.00th=[ 54], | 99.00th=[ 80], 99.50th=[ 97], 99.90th=[ 120], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 24, max= 192, per=24.81%, avg=95.01, stdev=37.79, samples=130 iops : min= 6, max= 48, avg=23.43, stdev= 9.50, samples=130 lat (msec) : 4=0.20%, 10=20.35%, 20=38.67%, 50=35.90%, 100=4.62% lat (msec) : 250=0.23%, 500=0.03% cpu : usr=0.12%, sys=4.86%, ctx=3439, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23642: Thu May 1 21:44:27 2025 read: IOPS=23, BW=92.7KiB/s (94.9kB/s)(6052KiB/65316msec) clat (msec): min=3, max=122, avg=15.80, stdev=13.05 lat (msec): min=3, max=122, avg=15.80, stdev=13.05 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 30], 95.00th=[ 42], | 99.00th=[ 70], 99.50th=[ 88], 99.90th=[ 116], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 8, max= 216, per=25.35%, avg=92.53, stdev=39.44, samples=129 iops : min= 2, max= 54, avg=22.81, stdev= 9.97, samples=129 write: IOPS=23, BW=95.5KiB/s (97.8kB/s)(6236KiB/65316msec) clat (msec): min=6, max=270, avg=26.43, stdev=15.53 lat (msec): min=6, max=270, avg=26.43, stdev=15.53 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 25], | 70.00th=[ 29], 80.00th=[ 35], 90.00th=[ 46], 95.00th=[ 55], | 99.00th=[ 82], 99.50th=[ 95], 99.90th=[ 117], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 31, max= 184, per=24.73%, avg=94.70, stdev=36.90, samples=130 iops : min= 7, max= 46, avg=23.35, stdev= 9.30, samples=130 lat (msec) : 4=0.03%, 10=19.99%, 20=39.78%, 50=35.22%, 100=4.65% lat (msec) : 250=0.29%, 500=0.03% cpu : usr=0.15%, sys=4.94%, ctx=3395, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=366KiB/s (375kB/s), 90.5KiB/s-92.7KiB/s (92.7kB/s-94.9kB/s), io=23.4MiB (24.6MB), run=64894-65618msec WRITE: bw=383KiB/s (392kB/s), 95.5KiB/s-97.0KiB/s (97.8kB/s-99.3kB/s), io=24.6MiB (25.7MB), run=64894-65618msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23720: Thu May 1 21:44:41 2025 read: IOPS=32, BW=516KiB/s (528kB/s)(5920KiB/11482msec) clat (usec): min=73, max=83440, avg=17301.78, stdev=12597.86 lat (usec): min=74, max=83441, avg=17307.46, stdev=12598.42 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 119], 10.00th=[ 151], 20.00th=[ 7701], | 30.00th=[10814], 40.00th=[13566], 50.00th=[15795], 60.00th=[18220], | 70.00th=[20841], 80.00th=[24511], 90.00th=[31589], 95.00th=[40633], | 99.00th=[63701], 99.50th=[70779], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 256, max= 734, per=25.50%, avg=503.18, stdev=121.59, samples=22 iops : min= 16, max= 45, avg=31.09, stdev= 7.48, samples=22 write: IOPS=34, BW=555KiB/s (568kB/s)(6368KiB/11482msec) clat (usec): min=908, max=76530, avg=12582.43, stdev=13563.22 lat (usec): min=909, max=76531, avg=12584.50, stdev=13563.15 clat percentiles (usec): | 1.00th=[ 996], 5.00th=[ 1156], 10.00th=[ 1205], 20.00th=[ 1467], | 30.00th=[ 2278], 40.00th=[ 3720], 50.00th=[ 7767], 60.00th=[11600], | 70.00th=[16909], 80.00th=[23462], 90.00th=[30540], 95.00th=[40109], | 99.00th=[60556], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 252, max= 830, per=25.99%, avg=539.59, stdev=171.10, samples=22 iops : min= 15, max= 51, avg=33.36, stdev=10.61, samples=22 lat (usec) : 100=0.39%, 250=5.34%, 750=0.13%, 1000=0.78% lat (msec) : 2=13.41%, 4=7.42%, 10=13.93%, 20=29.69%, 50=26.69% lat (msec) : 100=2.21% cpu : usr=0.17%, sys=11.01%, ctx=1350, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23721: Thu May 1 21:44:41 2025 read: IOPS=29, BW=473KiB/s (485kB/s)(5744KiB/12138msec) clat (usec): min=92, max=97178, avg=15864.25, stdev=11901.85 lat (usec): min=92, max=97180, avg=15865.46, stdev=11901.91 clat percentiles (usec): | 1.00th=[ 98], 5.00th=[ 113], 10.00th=[ 127], 20.00th=[ 6587], | 30.00th=[ 9634], 40.00th=[12387], 50.00th=[15139], 60.00th=[17695], | 70.00th=[20055], 80.00th=[24511], 90.00th=[28967], 95.00th=[33424], | 99.00th=[58459], 99.50th=[67634], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 96, max= 768, per=23.64%, avg=466.50, stdev=160.37, samples=24 iops : min= 6, max= 48, avg=28.67, stdev=10.04, samples=24 write: IOPS=33, BW=539KiB/s (552kB/s)(6544KiB/12138msec) clat (usec): min=749, max=110626, avg=15221.79, stdev=16777.75 lat (usec): min=750, max=110628, avg=15226.12, stdev=16776.04 clat percentiles (usec): | 1.00th=[ 1057], 5.00th=[ 1139], 10.00th=[ 1237], 20.00th=[ 1631], | 30.00th=[ 3359], 40.00th=[ 6063], 50.00th=[ 9634], 60.00th=[ 12649], | 70.00th=[ 17171], 80.00th=[ 28181], 90.00th=[ 39060], 95.00th=[ 46924], | 99.00th=[ 69731], 99.50th=[ 73925], 99.90th=[110625], 99.95th=[110625], | 99.99th=[110625] bw ( KiB/s): min= 256, max= 1085, per=25.51%, avg=529.54, stdev=217.17, samples=24 iops : min= 16, max= 67, avg=32.63, stdev=13.51, samples=24 lat (usec) : 100=0.65%, 250=6.12%, 500=0.13%, 750=0.52%, 1000=0.39% lat (msec) : 2=12.37%, 4=5.99%, 10=16.02%, 20=29.56%, 50=25.52% lat (msec) : 100=2.47%, 250=0.26% cpu : usr=0.19%, sys=10.86%, ctx=1321, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23722: Thu May 1 21:44:41 2025 read: IOPS=32, BW=525KiB/s (537kB/s)(6256KiB/11923msec) clat (usec): min=86, max=72311, avg=17329.95, stdev=11363.13 lat (usec): min=86, max=72312, avg=17331.22, stdev=11363.09 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[ 121], 10.00th=[ 149], 20.00th=[ 8979], | 30.00th=[11600], 40.00th=[13435], 50.00th=[15664], 60.00th=[18744], | 70.00th=[21365], 80.00th=[25822], 90.00th=[31851], 95.00th=[36439], | 99.00th=[54789], 99.50th=[58459], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 320, max= 768, per=26.32%, avg=519.26, stdev=136.42, samples=23 iops : min= 20, max= 48, avg=32.17, stdev= 8.65, samples=23 write: IOPS=31, BW=506KiB/s (518kB/s)(6032KiB/11923msec) clat (usec): min=795, max=84011, avg=13517.97, stdev=13506.76 lat (usec): min=796, max=84012, avg=13520.06, stdev=13506.76 clat percentiles (usec): | 1.00th=[ 1057], 5.00th=[ 1205], 10.00th=[ 1287], 20.00th=[ 2040], | 30.00th=[ 3294], 40.00th=[ 6456], 50.00th=[10290], 60.00th=[12911], | 70.00th=[17171], 80.00th=[21890], 90.00th=[31851], 95.00th=[43254], | 99.00th=[57410], 99.50th=[65274], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 256, max= 704, per=23.41%, avg=485.91, stdev=123.16, samples=23 iops : min= 16, max= 44, avg=30.09, stdev= 7.75, samples=23 lat (usec) : 100=0.91%, 250=4.69%, 500=0.13%, 750=0.13%, 1000=0.13% lat (msec) : 2=9.64%, 4=6.64%, 10=13.41%, 20=35.29%, 50=26.95% lat (msec) : 100=2.08% cpu : usr=0.20%, sys=10.88%, ctx=1345, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23723: Thu May 1 21:44:41 2025 read: IOPS=32, BW=518KiB/s (530kB/s)(6032KiB/11648msec) clat (usec): min=84, max=91262, avg=16545.08, stdev=12494.77 lat (usec): min=85, max=91264, avg=16546.38, stdev=12494.83 clat percentiles (usec): | 1.00th=[ 97], 5.00th=[ 114], 10.00th=[ 133], 20.00th=[ 7898], | 30.00th=[10421], 40.00th=[12780], 50.00th=[14746], 60.00th=[17433], | 70.00th=[20317], 80.00th=[24249], 90.00th=[32113], 95.00th=[39060], | 99.00th=[52691], 99.50th=[89654], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 224, max= 704, per=25.95%, avg=512.09, stdev=126.26, samples=23 iops : min= 14, max= 44, avg=31.61, stdev= 7.95, samples=23 write: IOPS=33, BW=537KiB/s (550kB/s)(6256KiB/11648msec) clat (usec): min=908, max=93703, avg=13300.53, stdev=14126.51 lat (usec): min=909, max=93703, avg=13302.50, stdev=14126.46 clat percentiles (usec): | 1.00th=[ 1020], 5.00th=[ 1156], 10.00th=[ 1221], 20.00th=[ 1598], | 30.00th=[ 2737], 40.00th=[ 4555], 50.00th=[ 7963], 60.00th=[12387], | 70.00th=[17695], 80.00th=[24249], 90.00th=[34341], 95.00th=[40109], | 99.00th=[57934], 99.50th=[76022], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 319, max= 960, per=25.54%, avg=530.22, stdev=167.11, samples=23 iops : min= 19, max= 60, avg=32.74, stdev=10.61, samples=23 lat (usec) : 100=0.52%, 250=6.38%, 750=0.26%, 1000=0.39% lat (msec) : 2=11.46%, 4=7.42%, 10=15.10%, 20=30.08%, 50=26.56% lat (msec) : 100=1.82% cpu : usr=0.11%, sys=11.39%, ctx=1309, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1973KiB/s (2021kB/s), 473KiB/s-525KiB/s (485kB/s-537kB/s), io=23.4MiB (24.5MB), run=11482-12138msec WRITE: bw=2076KiB/s (2126kB/s), 506KiB/s-555KiB/s (518kB/s-568kB/s), io=24.6MiB (25.8MB), run=11482-12138msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23724: Thu May 1 21:44:47 2025 read: IOPS=20, BW=335KiB/s (344kB/s)(5920KiB/17647msec) clat (usec): min=3754, max=91944, avg=17589.04, stdev=14665.31 lat (usec): min=3755, max=91944, avg=17590.47, stdev=14665.29 clat percentiles (usec): | 1.00th=[ 4359], 5.00th=[ 5735], 10.00th=[ 6718], 20.00th=[ 8160], | 30.00th=[ 9241], 40.00th=[10290], 50.00th=[11731], 60.00th=[14091], | 70.00th=[17433], 80.00th=[23725], 90.00th=[38011], 95.00th=[51119], | 99.00th=[68682], 99.50th=[81265], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 128, max= 862, per=24.92%, avg=333.17, stdev=156.15, samples=35 iops : min= 8, max= 53, avg=20.49, stdev= 9.66, samples=35 write: IOPS=22, BW=361KiB/s (370kB/s)(6368KiB/17647msec) clat (msec): min=6, max=129, avg=27.61, stdev=17.06 lat (msec): min=6, max=129, avg=27.62, stdev=17.06 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 26], | 70.00th=[ 31], 80.00th=[ 39], 90.00th=[ 50], 95.00th=[ 64], | 99.00th=[ 95], 99.50th=[ 113], 99.90th=[ 130], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 95, max= 704, per=25.62%, avg=360.54, stdev=175.28, samples=35 iops : min= 5, max= 44, avg=22.20, stdev=10.93, samples=35 lat (msec) : 4=0.13%, 10=18.75%, 20=38.67%, 50=34.64%, 100=7.55% lat (msec) : 250=0.26% cpu : usr=0.01%, sys=4.77%, ctx=845, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23725: Thu May 1 21:44:47 2025 read: IOPS=20, BW=323KiB/s (331kB/s)(5744KiB/17790msec) clat (msec): min=3, max=124, avg=16.39, stdev=14.86 lat (msec): min=3, max=124, avg=16.39, stdev=14.86 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 33], 95.00th=[ 50], | 99.00th=[ 78], 99.50th=[ 100], 99.90th=[ 126], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 31, max= 734, per=24.01%, avg=320.97, stdev=188.74, samples=35 iops : min= 1, max= 45, avg=19.51, stdev=11.85, samples=35 write: IOPS=22, BW=368KiB/s (377kB/s)(6544KiB/17790msec) clat (msec): min=7, max=113, avg=29.00, stdev=17.38 lat (msec): min=7, max=113, avg=29.00, stdev=17.38 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 19], 40.00th=[ 22], 50.00th=[ 24], 60.00th=[ 28], | 70.00th=[ 33], 80.00th=[ 42], 90.00th=[ 53], 95.00th=[ 64], | 99.00th=[ 93], 99.50th=[ 104], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 159, max= 736, per=26.00%, avg=365.80, stdev=153.97, samples=35 iops : min= 9, max= 46, avg=22.31, stdev= 9.75, samples=35 lat (msec) : 4=0.39%, 10=19.92%, 20=35.81%, 50=35.68%, 100=7.68% lat (msec) : 250=0.52% cpu : usr=0.15%, sys=4.72%, ctx=861, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23726: Thu May 1 21:44:47 2025 read: IOPS=21, BW=351KiB/s (359kB/s)(6256KiB/17837msec) clat (msec): min=4, max=126, avg=18.51, stdev=17.84 lat (msec): min=4, max=126, avg=18.51, stdev=17.84 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 18], 80.00th=[ 26], 90.00th=[ 43], 95.00th=[ 56], | 99.00th=[ 90], 99.50th=[ 100], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 127, max= 864, per=26.28%, avg=351.37, stdev=194.66, samples=35 iops : min= 7, max= 54, avg=21.54, stdev=12.25, samples=35 write: IOPS=21, BW=338KiB/s (346kB/s)(6032KiB/17837msec) clat (msec): min=8, max=113, avg=27.82, stdev=16.43 lat (msec): min=8, max=113, avg=27.83, stdev=16.43 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 27], | 70.00th=[ 32], 80.00th=[ 39], 90.00th=[ 50], 95.00th=[ 62], | 99.00th=[ 90], 99.50th=[ 111], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 95, max= 736, per=23.67%, avg=333.06, stdev=155.69, samples=35 iops : min= 5, max= 46, avg=20.40, stdev= 9.81, samples=35 lat (msec) : 10=21.61%, 20=35.29%, 50=34.90%, 100=7.81%, 250=0.39% cpu : usr=0.10%, sys=4.66%, ctx=885, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23727: Thu May 1 21:44:47 2025 read: IOPS=21, BW=337KiB/s (345kB/s)(6032KiB/17910msec) clat (msec): min=3, max=118, avg=18.09, stdev=17.11 lat (msec): min=3, max=118, avg=18.09, stdev=17.11 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 23], 90.00th=[ 39], 95.00th=[ 57], | 99.00th=[ 93], 99.50th=[ 100], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 128, max= 766, per=24.97%, avg=333.83, stdev=167.18, samples=35 iops : min= 8, max= 47, avg=20.40, stdev=10.35, samples=35 write: IOPS=21, BW=349KiB/s (358kB/s)(6256KiB/17910msec) clat (msec): min=6, max=108, avg=27.77, stdev=16.95 lat (msec): min=6, max=108, avg=27.78, stdev=16.95 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 27], | 70.00th=[ 32], 80.00th=[ 39], 90.00th=[ 53], 95.00th=[ 64], | 99.00th=[ 89], 99.50th=[ 106], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 128, max= 736, per=24.71%, avg=347.60, stdev=181.20, samples=35 iops : min= 8, max= 46, avg=21.26, stdev=11.39, samples=35 lat (msec) : 4=0.13%, 10=20.18%, 20=38.80%, 50=31.38%, 100=9.11% lat (msec) : 250=0.39% cpu : usr=0.08%, sys=4.79%, ctx=877, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1337KiB/s (1369kB/s), 323KiB/s-351KiB/s (331kB/s-359kB/s), io=23.4MiB (24.5MB), run=17647-17910msec WRITE: bw=1407KiB/s (1441kB/s), 338KiB/s-368KiB/s (346kB/s-377kB/s), io=24.6MiB (25.8MB), run=17647-17910msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23748: Thu May 1 21:44:50 2025 read: IOPS=5, BW=5493KiB/s (5625kB/s)(7168KiB/1305msec) clat (usec): min=466, max=104773, avg=31986.00, stdev=42111.62 lat (usec): min=468, max=104775, avg=31987.30, stdev=42111.55 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 465], 10.00th=[ 465], 20.00th=[ 553], | 30.00th=[ 586], 40.00th=[ 586], 50.00th=[ 8291], 60.00th=[ 34866], | 70.00th=[ 34866], 80.00th=[ 74974], 90.00th=[104334], 95.00th=[104334], | 99.00th=[104334], 99.50th=[104334], 99.90th=[104334], 99.95th=[104334], | 99.99th=[104334] bw ( KiB/s): min= 4087, max= 6144, per=32.60%, avg=5115.50, stdev=1454.52, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 write: IOPS=3, BW=3923KiB/s (4018kB/s)(5120KiB/1305msec) clat (msec): min=46, max=264, avg=132.75, stdev=81.87 lat (msec): min=46, max=264, avg=132.82, stdev=81.85 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 100], 40.00th=[ 100], 50.00th=[ 109], 60.00th=[ 109], | 70.00th=[ 146], 80.00th=[ 146], 90.00th=[ 266], 95.00th=[ 266], | 99.00th=[ 266], 99.50th=[ 266], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 4096, max= 4096, per=18.64%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=16.67% lat (msec) : 10=8.33%, 50=16.67%, 100=16.67%, 250=25.00%, 500=8.33% cpu : usr=0.08%, sys=11.96%, ctx=69, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23749: Thu May 1 21:44:50 2025 read: IOPS=1, BW=1720KiB/s (1761kB/s)(2048KiB/1191msec) clat (usec): min=538, max=129945, avg=65242.05, stdev=91504.67 lat (usec): min=539, max=129948, avg=65243.87, stdev=91506.30 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 537], 10.00th=[ 537], 20.00th=[ 537], | 30.00th=[ 537], 40.00th=[ 537], 50.00th=[ 537], 60.00th=[129500], | 70.00th=[129500], 80.00th=[129500], 90.00th=[129500], 95.00th=[129500], | 99.00th=[129500], 99.50th=[129500], 99.90th=[129500], 99.95th=[129500], | 99.99th=[129500] bw ( KiB/s): min= 2043, max= 2048, per=13.03%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=8, BW=8598KiB/s (8804kB/s)(10.0MiB/1191msec) clat (msec): min=38, max=161, avg=103.19, stdev=49.04 lat (msec): min=38, max=161, avg=103.36, stdev=49.01 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 46], | 30.00th=[ 65], 40.00th=[ 68], 50.00th=[ 84], 60.00th=[ 110], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 159], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 6131, max=10240, per=37.26%, avg=8185.50, stdev=2905.50, samples=2 iops : min= 5, max= 10, avg= 7.50, stdev= 3.54, samples=2 lat (usec) : 750=8.33% lat (msec) : 50=16.67%, 100=25.00%, 250=50.00% cpu : usr=0.00%, sys=21.51%, ctx=282, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23750: Thu May 1 21:44:50 2025 read: IOPS=5, BW=5176KiB/s (5300kB/s)(6144KiB/1187msec) clat (usec): min=521, max=186785, avg=54719.42, stdev=72993.12 lat (usec): min=523, max=186786, avg=54720.61, stdev=72993.02 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 523], 10.00th=[ 523], 20.00th=[ 594], | 30.00th=[ 594], 40.00th=[ 2769], 50.00th=[ 2769], 60.00th=[ 57934], | 70.00th=[ 79168], 80.00th=[ 79168], 90.00th=[187696], 95.00th=[187696], | 99.00th=[187696], 99.50th=[187696], 99.90th=[187696], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 4071, max= 5988, per=32.05%, avg=5029.50, stdev=1355.52, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 write: IOPS=5, BW=5176KiB/s (5300kB/s)(6144KiB/1187msec) clat (msec): min=32, max=206, avg=127.05, stdev=72.99 lat (msec): min=32, max=206, avg=127.12, stdev=72.99 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 52], | 30.00th=[ 52], 40.00th=[ 114], 50.00th=[ 114], 60.00th=[ 176], | 70.00th=[ 186], 80.00th=[ 186], 90.00th=[ 207], 95.00th=[ 207], | 99.00th=[ 207], 99.50th=[ 207], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 3992, max= 6107, per=22.98%, avg=5049.50, stdev=1495.53, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 lat (usec) : 750=16.67% lat (msec) : 4=8.33%, 50=8.33%, 100=25.00%, 250=41.67% cpu : usr=0.00%, sys=15.35%, ctx=138, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23751: Thu May 1 21:44:50 2025 read: IOPS=4, BW=4324KiB/s (4428kB/s)(5120KiB/1184msec) clat (usec): min=591, max=167680, avg=91953.70, stdev=59936.15 lat (usec): min=593, max=167681, avg=91955.48, stdev=59936.00 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 594], 10.00th=[ 594], 20.00th=[ 594], | 30.00th=[ 86508], 40.00th=[ 86508], 50.00th=[ 98042], 60.00th=[ 98042], | 70.00th=[106431], 80.00th=[106431], 90.00th=[166724], 95.00th=[166724], | 99.00th=[166724], 99.50th=[166724], 99.90th=[166724], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 6144, max= 6144, per=39.15%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=5, BW=6054KiB/s (6199kB/s)(7168KiB/1184msec) clat (msec): min=36, max=179, avg=80.49, stdev=51.65 lat (msec): min=36, max=180, avg=80.70, stdev=51.90 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 39], | 30.00th=[ 55], 40.00th=[ 55], 50.00th=[ 65], 60.00th=[ 74], | 70.00th=[ 74], 80.00th=[ 117], 90.00th=[ 180], 95.00th=[ 180], | 99.00th=[ 180], 99.50th=[ 180], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 2048, max= 6119, per=18.59%, avg=4083.50, stdev=2878.63, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (usec) : 750=8.33% lat (msec) : 50=16.67%, 100=41.67%, 250=33.33% cpu : usr=0.00%, sys=19.70%, ctx=201, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.3MiB/s (16.1MB/s), 1720KiB/s-5493KiB/s (1761kB/s-5625kB/s), io=20.0MiB (20.0MB), run=1184-1305msec WRITE: bw=21.5MiB/s (22.5MB/s), 3923KiB/s-8598KiB/s (4018kB/s-8804kB/s), io=28.0MiB (29.4MB), run=1184-1305msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23752: Thu May 1 21:44:50 2025 read: IOPS=4, BW=4096KiB/s (4194kB/s)(7168KiB/1750msec) clat (msec): min=15, max=397, avg=124.07, stdev=143.52 lat (msec): min=15, max=397, avg=124.07, stdev=143.52 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 33], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 58], 60.00th=[ 74], | 70.00th=[ 74], 80.00th=[ 247], 90.00th=[ 397], 95.00th=[ 397], | 99.00th=[ 397], 99.50th=[ 397], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 4096, max= 6144, per=43.75%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2926KiB/s (2996kB/s)(5120KiB/1750msec) clat (msec): min=46, max=206, avg=88.54, stdev=69.10 lat (msec): min=46, max=206, avg=88.64, stdev=69.10 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 48], 40.00th=[ 48], 50.00th=[ 48], 60.00th=[ 48], | 70.00th=[ 94], 80.00th=[ 94], 90.00th=[ 207], 95.00th=[ 207], | 99.00th=[ 207], 99.50th=[ 207], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 2048, max= 2048, per=12.50%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=25.00%, 250=16.67%, 500=8.33% cpu : usr=0.00%, sys=3.66%, ctx=46, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23753: Thu May 1 21:44:50 2025 read: IOPS=1, BW=1208KiB/s (1237kB/s)(2048KiB/1695msec) clat (msec): min=56, max=109, avg=83.02, stdev=37.55 lat (msec): min=56, max=109, avg=83.02, stdev=37.55 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 57], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 57], 60.00th=[ 110], | 70.00th=[ 110], 80.00th=[ 110], 90.00th=[ 110], 95.00th=[ 110], | 99.00th=[ 110], 99.50th=[ 110], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 2039, max= 2048, per=17.46%, avg=2043.50, stdev= 6.36, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=5, BW=6041KiB/s (6186kB/s)(10.0MiB/1695msec) clat (msec): min=39, max=280, avg=108.61, stdev=89.28 lat (msec): min=39, max=280, avg=108.85, stdev=89.17 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 42], | 30.00th=[ 43], 40.00th=[ 47], 50.00th=[ 63], 60.00th=[ 66], | 70.00th=[ 199], 80.00th=[ 199], 90.00th=[ 220], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 6119, max= 8192, per=43.67%, avg=7155.50, stdev=1465.83, samples=2 iops : min= 5, max= 8, avg= 6.50, stdev= 2.12, samples=2 lat (msec) : 50=33.33%, 100=33.33%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=3.66%, ctx=38, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23754: Thu May 1 21:44:50 2025 read: IOPS=3, BW=3551KiB/s (3637kB/s)(6144KiB/1730msec) clat (msec): min=25, max=178, avg=82.69, stdev=60.99 lat (msec): min=25, max=178, avg=82.69, stdev=60.99 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 35], | 30.00th=[ 35], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 107], | 70.00th=[ 116], 80.00th=[ 116], 90.00th=[ 178], 95.00th=[ 178], | 99.00th=[ 178], 99.50th=[ 178], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 8126, max= 8126, per=69.44%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3551KiB/s (3637kB/s)(6144KiB/1730msec) clat (msec): min=17, max=438, avg=184.25, stdev=154.30 lat (msec): min=17, max=438, avg=184.34, stdev=154.28 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 60], | 30.00th=[ 60], 40.00th=[ 129], 50.00th=[ 129], 60.00th=[ 188], | 70.00th=[ 275], 80.00th=[ 275], 90.00th=[ 439], 95.00th=[ 439], | 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 4063, max= 6144, per=31.15%, avg=5103.50, stdev=1471.49, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 lat (msec) : 20=8.33%, 50=25.00%, 100=8.33%, 250=41.67%, 500=16.67% cpu : usr=0.00%, sys=5.49%, ctx=46, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23755: Thu May 1 21:44:50 2025 read: IOPS=3, BW=3073KiB/s (3147kB/s)(5120KiB/1666msec) clat (msec): min=20, max=205, avg=113.87, stdev=69.63 lat (msec): min=20, max=205, avg=113.87, stdev=69.63 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 85], 40.00th=[ 85], 50.00th=[ 108], 60.00th=[ 108], | 70.00th=[ 153], 80.00th=[ 153], 90.00th=[ 205], 95.00th=[ 205], | 99.00th=[ 205], 99.50th=[ 205], 99.90th=[ 205], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 8159, max= 8159, per=69.72%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4303KiB/s (4406kB/s)(7168KiB/1666msec) clat (msec): min=37, max=287, avg=103.09, stdev=92.60 lat (msec): min=37, max=292, avg=103.92, stdev=94.27 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 48], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 58], 60.00th=[ 71], | 70.00th=[ 71], 80.00th=[ 169], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 2039, max= 8192, per=31.22%, avg=5115.50, stdev=4350.83, samples=2 iops : min= 1, max= 8, avg= 4.50, stdev= 4.95, samples=2 lat (msec) : 50=25.00%, 100=33.33%, 250=33.33%, 500=8.33% cpu : usr=0.30%, sys=4.68%, ctx=114, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.4MiB/s (11.0MB/s), 1208KiB/s-4096KiB/s (1237kB/s-4194kB/s), io=20.0MiB (20.0MB), run=1666-1750msec WRITE: bw=16.0MiB/s (16.8MB/s), 2926KiB/s-6041KiB/s (2996kB/s-6186kB/s), io=28.0MiB (29.4MB), run=1666-1750msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23764: Thu May 1 21:44:54 2025 read: IOPS=2, BW=10.3MiB/s (10.8MB/s)(12.0MiB/1168msec) clat (msec): min=275, max=449, avg=337.17, stdev=97.68 lat (msec): min=275, max=449, avg=337.17, stdev=97.68 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 275], 10.00th=[ 275], 20.00th=[ 275], | 30.00th=[ 275], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 288], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 8175, max= 8192, per=77.42%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=100.00% cpu : usr=0.00%, sys=17.74%, ctx=705, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23765: Thu May 1 21:44:54 2025 read: IOPS=0, BW=2643KiB/s (2706kB/s)(4096KiB/1550msec) clat (nsec): min=318206k, max=318206k, avg=318205600.00, stdev= 0.00 lat (nsec): min=318209k, max=318209k, avg=318208551.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 317], 10.00th=[ 317], 20.00th=[ 317], | 30.00th=[ 317], 40.00th=[ 317], 50.00th=[ 317], 60.00th=[ 317], | 70.00th=[ 317], 80.00th=[ 317], 90.00th=[ 317], 95.00th=[ 317], | 99.00th=[ 317], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 8126, max= 8126, per=76.88%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5285KiB/s (5412kB/s)(8192KiB/1550msec) clat (msec): min=479, max=721, avg=600.18, stdev=170.94 lat (msec): min=479, max=721, avg=600.69, stdev=170.88 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 481], 10.00th=[ 481], 20.00th=[ 481], | 30.00th=[ 481], 40.00th=[ 481], 50.00th=[ 481], 60.00th=[ 718], | 70.00th=[ 718], 80.00th=[ 718], 90.00th=[ 718], 95.00th=[ 718], | 99.00th=[ 718], 99.50th=[ 718], 99.90th=[ 718], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 8192, max= 8192, per=39.90%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.19%, sys=15.88%, ctx=387, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23766: Thu May 1 21:44:54 2025 write: IOPS=1, BW=7699KiB/s (7884kB/s)(12.0MiB/1596msec) clat (msec): min=325, max=637, avg=512.59, stdev=165.02 lat (msec): min=328, max=637, avg=513.95, stdev=163.37 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 326], 10.00th=[ 326], 20.00th=[ 326], | 30.00th=[ 326], 40.00th=[ 575], 50.00th=[ 575], 60.00th=[ 575], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 8192, max= 8192, per=39.90%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.13%, sys=19.87%, ctx=69, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23767: Thu May 1 21:44:54 2025 write: IOPS=2, BW=8891KiB/s (9105kB/s)(12.0MiB/1382msec) clat (msec): min=267, max=637, avg=434.45, stdev=187.85 lat (msec): min=267, max=638, avg=434.95, stdev=187.86 clat percentiles (msec): | 1.00th=[ 268], 5.00th=[ 268], 10.00th=[ 268], 20.00th=[ 268], | 30.00th=[ 268], 40.00th=[ 397], 50.00th=[ 397], 60.00th=[ 397], | 70.00th=[ 642], 80.00th=[ 642], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 8110, max= 8110, per=39.50%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.00%, sys=20.85%, ctx=93, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.3MiB/s (10.8MB/s), 2643KiB/s-10.3MiB/s (2706kB/s-10.8MB/s), io=16.0MiB (16.8MB), run=1168-1550msec WRITE: bw=20.0MiB/s (21.0MB/s), 5285KiB/s-8891KiB/s (5412kB/s-9105kB/s), io=32.0MiB (33.6MB), run=1382-1596msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23768: Thu May 1 21:44:54 2025 read: IOPS=1, BW=5977KiB/s (6120kB/s)(12.0MiB/2056msec) clat (msec): min=215, max=974, avg=637.94, stdev=386.84 lat (msec): min=215, max=974, avg=637.94, stdev=386.84 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 215], 10.00th=[ 215], 20.00th=[ 215], | 30.00th=[ 215], 40.00th=[ 726], 50.00th=[ 726], 60.00th=[ 726], | 70.00th=[ 978], 80.00th=[ 978], 90.00th=[ 978], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=4.23%, ctx=45, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23769: Thu May 1 21:44:54 2025 read: IOPS=0, BW=2109KiB/s (2160kB/s)(4096KiB/1942msec) clat (nsec): min=654328k, max=654328k, avg=654328202.00, stdev= 0.00 lat (nsec): min=654331k, max=654331k, avg=654331112.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 659], 5.00th=[ 659], 10.00th=[ 659], 20.00th=[ 659], | 30.00th=[ 659], 40.00th=[ 659], 50.00th=[ 659], 60.00th=[ 659], | 70.00th=[ 659], 80.00th=[ 659], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4218KiB/s (4320kB/s)(8192KiB/1942msec) clat (msec): min=353, max=633, avg=493.38, stdev=198.53 lat (msec): min=358, max=635, avg=497.22, stdev=195.58 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 355], 10.00th=[ 355], 20.00th=[ 355], | 30.00th=[ 355], 40.00th=[ 355], 50.00th=[ 355], 60.00th=[ 634], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 7892, max= 7892, per=52.36%, avg=7892.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.05%, sys=2.22%, ctx=20, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23770: Thu May 1 21:44:54 2025 write: IOPS=2, BW=9974KiB/s (10.2MB/s)(12.0MiB/1232msec) clat (msec): min=262, max=633, avg=391.65, stdev=209.46 lat (msec): min=263, max=633, avg=392.04, stdev=209.50 clat percentiles (msec): | 1.00th=[ 264], 5.00th=[ 264], 10.00th=[ 264], 20.00th=[ 264], | 30.00th=[ 264], 40.00th=[ 279], 50.00th=[ 279], 60.00th=[ 279], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 8175, max= 8192, per=54.30%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.08%, sys=5.36%, ctx=169, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23771: Thu May 1 21:44:54 2025 write: IOPS=1, BW=5652KiB/s (5788kB/s)(12.0MiB/2174msec) clat (msec): min=144, max=999, avg=615.37, stdev=433.75 lat (msec): min=145, max=999, avg=615.67, stdev=433.66 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 146], 10.00th=[ 146], 20.00th=[ 146], | 30.00th=[ 146], 40.00th=[ 701], 50.00th=[ 701], 60.00th=[ 701], | 70.00th=[ 1003], 80.00th=[ 1003], 90.00th=[ 1003], 95.00th=[ 1003], | 99.00th=[ 1003], 99.50th=[ 1003], 99.90th=[ 1003], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 8175, max= 8175, per=54.24%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.05%, sys=3.45%, ctx=86, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7969KiB/s (8160kB/s), 2109KiB/s-5977KiB/s (2160kB/s-6120kB/s), io=16.0MiB (16.8MB), run=1942-2056msec WRITE: bw=14.7MiB/s (15.4MB/s), 4218KiB/s-9974KiB/s (4320kB/s-10.2MB/s), io=32.0MiB (33.6MB), run=1232-2174msec