== sanity test 398b: DIO and buffer IO race ============== 23:09:05 (1746155345) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 3.60443 s, 14.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32075: Thu May 1 23:10:15 2025 read: IOPS=24, BW=96.3KiB/s (98.6kB/s)(5992KiB/62201msec) clat (usec): min=72, max=389205, avg=17993.29, stdev=19691.34 lat (usec): min=72, max=389208, avg=17999.92, stdev=19687.18 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 99], 10.00th=[ 115], 20.00th=[ 147], | 30.00th=[ 6915], 40.00th=[ 11731], 50.00th=[ 15401], 60.00th=[ 18744], | 70.00th=[ 23200], 80.00th=[ 30016], 90.00th=[ 40633], 95.00th=[ 50070], | 99.00th=[ 68682], 99.50th=[ 73925], 99.90th=[193987], 99.95th=[387974], | 99.99th=[387974] bw ( KiB/s): min= 8, max= 200, per=25.10%, avg=96.14, stdev=33.67, samples=123 iops : min= 2, max= 50, avg=23.70, stdev= 8.49, samples=123 write: IOPS=25, BW=101KiB/s (104kB/s)(6296KiB/62201msec) clat (usec): min=675, max=230012, avg=22163.09, stdev=24340.45 lat (usec): min=676, max=230014, avg=22165.07, stdev=24340.44 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 865], 10.00th=[ 1012], 20.00th=[ 1483], | 30.00th=[ 3851], 40.00th=[ 9241], 50.00th=[ 16450], 60.00th=[ 22414], | 70.00th=[ 28705], 80.00th=[ 38536], 90.00th=[ 52167], 95.00th=[ 66847], | 99.00th=[101188], 99.50th=[137364], 99.90th=[210764], 99.95th=[229639], | 99.99th=[229639] bw ( KiB/s): min= 7, max= 216, per=25.22%, avg=101.15, stdev=32.60, samples=123 iops : min= 1, max= 54, avg=24.96, stdev= 8.23, samples=123 lat (usec) : 100=2.57%, 250=8.82%, 500=1.46%, 750=0.62%, 1000=4.95% lat (msec) : 2=7.39%, 4=4.13%, 10=8.20%, 20=21.26%, 50=32.29% lat (msec) : 100=7.62%, 250=0.65%, 500=0.03% cpu : usr=0.11%, sys=9.12%, ctx=4885, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32076: Thu May 1 23:10:15 2025 read: IOPS=23, BW=94.9KiB/s (97.2kB/s)(5940KiB/62570msec) clat (usec): min=61, max=341160, avg=17711.49, stdev=19773.48 lat (usec): min=61, max=341162, avg=17713.39, stdev=19774.04 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 104], 10.00th=[ 118], 20.00th=[ 149], | 30.00th=[ 7701], 40.00th=[ 11469], 50.00th=[ 15008], 60.00th=[ 17957], | 70.00th=[ 22676], 80.00th=[ 28705], 90.00th=[ 39060], 95.00th=[ 46924], | 99.00th=[ 71828], 99.50th=[108528], 99.90th=[191890], 99.95th=[341836], | 99.99th=[341836] bw ( KiB/s): min= 7, max= 239, per=24.98%, avg=95.68, stdev=35.09, samples=123 iops : min= 1, max= 59, avg=23.52, stdev= 8.79, samples=123 write: IOPS=25, BW=101KiB/s (104kB/s)(6348KiB/62570msec) clat (usec): min=670, max=281521, avg=22645.26, stdev=25149.94 lat (usec): min=670, max=281523, avg=22647.98, stdev=25149.60 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 865], 10.00th=[ 1004], 20.00th=[ 1680], | 30.00th=[ 4424], 40.00th=[ 9503], 50.00th=[ 17171], 60.00th=[ 23200], | 70.00th=[ 30016], 80.00th=[ 38536], 90.00th=[ 52691], 95.00th=[ 64750], | 99.00th=[103285], 99.50th=[128451], 99.90th=[238027], 99.95th=[283116], | 99.99th=[283116] bw ( KiB/s): min= 15, max= 278, per=25.27%, avg=101.31, stdev=37.33, samples=124 iops : min= 3, max= 69, avg=24.94, stdev= 9.35, samples=124 lat (usec) : 100=2.02%, 250=8.92%, 500=1.46%, 750=0.62%, 1000=4.82% lat (msec) : 2=6.67%, 4=3.84%, 10=9.37%, 20=22.88%, 50=31.45% lat (msec) : 100=7.13%, 250=0.75%, 500=0.07% cpu : usr=0.13%, sys=9.19%, ctx=4976, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32077: Thu May 1 23:10:15 2025 read: IOPS=24, BW=97.4KiB/s (99.8kB/s)(6028KiB/61869msec) clat (usec): min=71, max=542768, avg=18260.92, stdev=21764.40 lat (usec): min=71, max=542771, avg=18264.04, stdev=21764.30 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 103], 10.00th=[ 118], 20.00th=[ 176], | 30.00th=[ 9372], 40.00th=[ 12518], 50.00th=[ 15533], 60.00th=[ 18744], | 70.00th=[ 23462], 80.00th=[ 29492], 90.00th=[ 39060], 95.00th=[ 46400], | 99.00th=[ 65799], 99.50th=[ 78119], 99.90th=[189793], 99.95th=[541066], | 99.99th=[541066] bw ( KiB/s): min= 7, max= 192, per=25.19%, avg=96.47, stdev=34.60, samples=123 iops : min= 1, max= 48, avg=23.64, stdev= 8.69, samples=123 write: IOPS=25, BW=101KiB/s (104kB/s)(6260KiB/61869msec) clat (usec): min=626, max=381012, avg=21845.66, stdev=24250.93 lat (usec): min=627, max=381038, avg=21849.95, stdev=24254.70 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 840], 10.00th=[ 988], 20.00th=[ 1631], | 30.00th=[ 4015], 40.00th=[ 8094], 50.00th=[ 15270], 60.00th=[ 21627], | 70.00th=[ 28967], 80.00th=[ 38536], 90.00th=[ 53216], 95.00th=[ 69731], | 99.00th=[ 93848], 99.50th=[103285], 99.90th=[141558], 99.95th=[379585], | 99.99th=[379585] bw ( KiB/s): min= 23, max= 183, per=24.94%, avg=99.99, stdev=31.21, samples=123 iops : min= 5, max= 45, avg=24.52, stdev= 7.82, samples=123 lat (usec) : 100=2.08%, 250=8.01%, 500=1.20%, 750=1.14%, 1000=4.56% lat (msec) : 2=6.35%, 4=4.52%, 10=9.83%, 20=22.66%, 50=31.71% lat (msec) : 100=7.45%, 250=0.42%, 500=0.03%, 750=0.03% cpu : usr=0.16%, sys=9.45%, ctx=4979, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32078: Thu May 1 23:10:15 2025 read: IOPS=24, BW=97.7KiB/s (100kB/s)(6052KiB/61966msec) clat (usec): min=70, max=196602, avg=17293.39, stdev=17171.84 lat (usec): min=71, max=196603, avg=17295.02, stdev=17171.60 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 101], 10.00th=[ 114], 20.00th=[ 149], | 30.00th=[ 7701], 40.00th=[ 11994], 50.00th=[ 14746], 60.00th=[ 18220], | 70.00th=[ 22676], 80.00th=[ 28967], 90.00th=[ 38536], 95.00th=[ 45876], | 99.00th=[ 69731], 99.50th=[ 81265], 99.90th=[183501], 99.95th=[196084], | 99.99th=[196084] bw ( KiB/s): min= 32, max= 176, per=25.82%, avg=98.91, stdev=31.52, samples=120 iops : min= 8, max= 44, avg=24.36, stdev= 7.92, samples=120 write: IOPS=25, BW=101KiB/s (103kB/s)(6236KiB/61966msec) clat (usec): min=557, max=391517, avg=22727.02, stdev=26677.72 lat (usec): min=558, max=391520, avg=22732.49, stdev=26676.30 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 848], 10.00th=[ 1020], 20.00th=[ 1663], | 30.00th=[ 3916], 40.00th=[ 8717], 50.00th=[ 16188], 60.00th=[ 22938], | 70.00th=[ 30016], 80.00th=[ 38536], 90.00th=[ 53216], 95.00th=[ 66847], | 99.00th=[102237], 99.50th=[166724], 99.90th=[256902], 99.95th=[392168], | 99.99th=[392168] bw ( KiB/s): min= 16, max= 199, per=25.06%, avg=100.49, stdev=31.27, samples=122 iops : min= 4, max= 49, avg=24.75, stdev= 7.82, samples=122 lat (usec) : 100=2.25%, 250=9.24%, 500=1.43%, 750=0.85%, 1000=4.43% lat (msec) : 2=6.71%, 4=4.59%, 10=8.40%, 20=22.14%, 50=32.26% lat (msec) : 100=7.06%, 250=0.59%, 500=0.07% cpu : usr=0.11%, sys=9.16%, ctx=4979, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=384KiB/s (393kB/s), 94.9KiB/s-97.7KiB/s (97.2kB/s-100kB/s), io=23.4MiB (24.6MB), run=61869-62570msec WRITE: bw=402KiB/s (411kB/s), 101KiB/s-101KiB/s (103kB/s-104kB/s), io=24.6MiB (25.7MB), run=61869-62570msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32070: Thu May 1 23:12:52 2025 read: IOPS=6, BW=27.2KiB/s (27.9kB/s)(5992KiB/220212msec) clat (msec): min=3, max=538, avg=18.31, stdev=21.65 lat (msec): min=3, max=538, avg=18.31, stdev=21.65 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 12], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 23], 90.00th=[ 35], 95.00th=[ 52], | 99.00th=[ 91], 99.50th=[ 106], 99.90th=[ 296], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 7, max= 104, per=27.25%, avg=29.71, stdev=19.21, samples=398 iops : min= 1, max= 26, avg= 7.15, stdev= 4.84, samples=398 write: IOPS=7, BW=28.6KiB/s (29.3kB/s)(6296KiB/220212msec) clat (msec): min=39, max=385, avg=122.29, stdev=51.24 lat (msec): min=39, max=385, avg=122.30, stdev=51.24 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 62], 10.00th=[ 69], 20.00th=[ 82], | 30.00th=[ 91], 40.00th=[ 102], 50.00th=[ 113], 60.00th=[ 126], | 70.00th=[ 138], 80.00th=[ 155], 90.00th=[ 186], 95.00th=[ 222], | 99.00th=[ 300], 99.50th=[ 330], 99.90th=[ 380], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 7, max= 48, per=24.87%, avg=28.35, stdev= 9.84, samples=437 iops : min= 1, max= 12, avg= 6.81, stdev= 2.54, samples=437 lat (msec) : 4=0.07%, 10=15.10%, 20=21.78%, 50=9.73%, 100=21.55% lat (msec) : 250=30.24%, 500=1.50%, 750=0.03% cpu : usr=0.03%, sys=2.04%, ctx=3689, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32071: Thu May 1 23:12:52 2025 read: IOPS=6, BW=27.0KiB/s (27.7kB/s)(5940KiB/219910msec) clat (msec): min=4, max=467, avg=17.55, stdev=18.06 lat (msec): min=4, max=467, avg=17.55, stdev=18.06 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 12], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 23], 90.00th=[ 34], 95.00th=[ 47], | 99.00th=[ 69], 99.50th=[ 82], 99.90th=[ 148], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 7, max= 110, per=27.62%, avg=30.11, stdev=19.64, samples=388 iops : min= 1, max= 27, avg= 7.20, stdev= 4.99, samples=388 write: IOPS=7, BW=28.9KiB/s (29.6kB/s)(6348KiB/219910msec) clat (msec): min=36, max=385, avg=121.97, stdev=50.73 lat (msec): min=36, max=385, avg=121.97, stdev=50.73 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 62], 10.00th=[ 70], 20.00th=[ 82], | 30.00th=[ 91], 40.00th=[ 101], 50.00th=[ 111], 60.00th=[ 123], | 70.00th=[ 138], 80.00th=[ 157], 90.00th=[ 188], 95.00th=[ 226], | 99.00th=[ 292], 99.50th=[ 313], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 7, max= 56, per=24.97%, avg=28.47, stdev= 9.89, samples=438 iops : min= 1, max= 14, avg= 6.79, stdev= 2.55, samples=438 lat (msec) : 10=14.97%, 20=22.17%, 50=9.57%, 100=22.10%, 250=29.62% lat (msec) : 500=1.56% cpu : usr=0.04%, sys=2.01%, ctx=3627, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32072: Thu May 1 23:12:52 2025 read: IOPS=6, BW=27.4KiB/s (28.1kB/s)(6028KiB/219748msec) clat (msec): min=3, max=222, avg=18.44, stdev=17.06 lat (msec): min=3, max=222, avg=18.45, stdev=17.06 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 24], 90.00th=[ 39], 95.00th=[ 53], | 99.00th=[ 85], 99.50th=[ 96], 99.90th=[ 174], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 7, max= 96, per=27.06%, avg=29.50, stdev=19.25, samples=402 iops : min= 1, max= 24, avg= 7.03, stdev= 4.88, samples=402 write: IOPS=7, BW=28.5KiB/s (29.2kB/s)(6260KiB/219748msec) clat (msec): min=41, max=550, avg=122.46, stdev=49.42 lat (msec): min=41, max=550, avg=122.46, stdev=49.42 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 64], 10.00th=[ 70], 20.00th=[ 82], | 30.00th=[ 92], 40.00th=[ 104], 50.00th=[ 113], 60.00th=[ 124], | 70.00th=[ 138], 80.00th=[ 157], 90.00th=[ 188], 95.00th=[ 218], | 99.00th=[ 284], 99.50th=[ 292], 99.90th=[ 393], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 7, max= 56, per=24.62%, avg=28.06, stdev= 9.89, samples=438 iops : min= 1, max= 14, avg= 6.67, stdev= 2.57, samples=438 lat (msec) : 4=0.07%, 10=14.55%, 20=22.46%, 50=9.51%, 100=21.09% lat (msec) : 250=31.05%, 500=1.24%, 750=0.03% cpu : usr=0.03%, sys=2.03%, ctx=3645, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32073: Thu May 1 23:12:52 2025 read: IOPS=6, BW=27.7KiB/s (28.4kB/s)(6052KiB/218406msec) clat (msec): min=4, max=293, avg=18.00, stdev=17.39 lat (msec): min=4, max=293, avg=18.00, stdev=17.39 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 22], 90.00th=[ 35], 95.00th=[ 53], | 99.00th=[ 85], 99.50th=[ 101], 99.90th=[ 157], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 7, max= 120, per=27.76%, avg=30.26, stdev=20.03, samples=394 iops : min= 1, max= 30, avg= 7.24, stdev= 5.06, samples=394 write: IOPS=7, BW=28.6KiB/s (29.2kB/s)(6236KiB/218406msec) clat (msec): min=38, max=561, avg=122.44, stdev=51.39 lat (msec): min=38, max=561, avg=122.44, stdev=51.39 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 62], 10.00th=[ 71], 20.00th=[ 82], | 30.00th=[ 90], 40.00th=[ 102], 50.00th=[ 112], 60.00th=[ 123], | 70.00th=[ 138], 80.00th=[ 157], 90.00th=[ 188], 95.00th=[ 224], | 99.00th=[ 292], 99.50th=[ 317], 99.90th=[ 368], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 7, max= 56, per=24.71%, avg=28.17, stdev=10.04, samples=435 iops : min= 1, max= 14, avg= 6.71, stdev= 2.58, samples=435 lat (msec) : 10=16.15%, 20=21.58%, 50=9.15%, 100=21.68%, 250=29.85% lat (msec) : 500=1.56%, 750=0.03% cpu : usr=0.04%, sys=2.03%, ctx=3632, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=109KiB/s (112kB/s), 27.0KiB/s-27.7KiB/s (27.7kB/s-28.4kB/s), io=23.4MiB (24.6MB), run=218406-220212msec WRITE: bw=114KiB/s (117kB/s), 28.5KiB/s-28.9KiB/s (29.2kB/s-29.6kB/s), io=24.6MiB (25.7MB), run=218406-220212msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32268: Thu May 1 23:13:08 2025 read: IOPS=26, BW=425KiB/s (436kB/s)(5920KiB/13917msec) clat (usec): min=80, max=166960, avg=15629.54, stdev=18916.25 lat (usec): min=81, max=166961, avg=15631.26, stdev=18916.71 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 105], 10.00th=[ 113], 20.00th=[ 130], | 30.00th=[ 153], 40.00th=[ 1090], 50.00th=[ 12387], 60.00th=[ 17957], | 70.00th=[ 23462], 80.00th=[ 27919], 90.00th=[ 37487], 95.00th=[ 46924], | 99.00th=[ 85459], 99.50th=[126354], 99.90th=[166724], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 64, max= 798, per=23.99%, avg=412.81, stdev=183.79, samples=27 iops : min= 4, max= 49, avg=25.26, stdev=11.47, samples=27 write: IOPS=28, BW=458KiB/s (469kB/s)(6368KiB/13917msec) clat (usec): min=790, max=182831, avg=19888.91, stdev=26127.71 lat (usec): min=791, max=182835, avg=19890.86, stdev=26127.81 clat percentiles (usec): | 1.00th=[ 1020], 5.00th=[ 1188], 10.00th=[ 1336], 20.00th=[ 1745], | 30.00th=[ 3163], 40.00th=[ 5473], 50.00th=[ 7832], 60.00th=[ 13960], | 70.00th=[ 23200], 80.00th=[ 36439], 90.00th=[ 50594], 95.00th=[ 73925], | 99.00th=[149947], 99.50th=[164627], 99.90th=[183501], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 63, max= 864, per=24.93%, avg=451.22, stdev=204.08, samples=27 iops : min= 3, max= 54, avg=27.67, stdev=12.85, samples=27 lat (usec) : 100=1.30%, 250=15.10%, 500=2.21%, 750=0.13%, 1000=0.65% lat (msec) : 2=12.50%, 4=6.12%, 10=11.72%, 20=15.49%, 50=27.47% lat (msec) : 100=6.12%, 250=1.17% cpu : usr=0.17%, sys=9.91%, ctx=1108, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32269: Thu May 1 23:13:08 2025 read: IOPS=27, BW=433KiB/s (444kB/s)(5744KiB/13262msec) clat (usec): min=81, max=103515, avg=15491.78, stdev=16487.59 lat (usec): min=82, max=103518, avg=15494.52, stdev=16486.28 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 104], 10.00th=[ 115], 20.00th=[ 135], | 30.00th=[ 153], 40.00th=[ 2040], 50.00th=[ 13304], 60.00th=[ 17695], | 70.00th=[ 22938], 80.00th=[ 28181], 90.00th=[ 40633], 95.00th=[ 46924], | 99.00th=[ 55837], 99.50th=[ 61080], 99.90th=[103285], 99.95th=[103285], | 99.99th=[103285] bw ( KiB/s): min= 31, max= 701, per=25.01%, avg=430.46, stdev=163.34, samples=26 iops : min= 1, max= 43, avg=26.42, stdev=10.21, samples=26 write: IOPS=30, BW=493KiB/s (505kB/s)(6544KiB/13262msec) clat (usec): min=936, max=162873, avg=17972.25, stdev=21614.46 lat (usec): min=938, max=162875, avg=17974.16, stdev=21614.48 clat percentiles (usec): | 1.00th=[ 996], 5.00th=[ 1172], 10.00th=[ 1287], 20.00th=[ 1631], | 30.00th=[ 2933], 40.00th=[ 5866], 50.00th=[ 9110], 60.00th=[ 14484], | 70.00th=[ 22676], 80.00th=[ 33817], 90.00th=[ 46400], 95.00th=[ 57410], | 99.00th=[104334], 99.50th=[110625], 99.90th=[162530], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 63, max= 960, per=27.04%, avg=489.38, stdev=172.86, samples=26 iops : min= 3, max= 60, avg=30.12, stdev=10.96, samples=26 lat (usec) : 100=1.56%, 250=13.54%, 500=2.60%, 750=0.13%, 1000=0.78% lat (msec) : 2=13.15%, 4=4.95%, 10=11.72%, 20=17.32%, 50=28.52% lat (msec) : 100=4.95%, 250=0.78% cpu : usr=0.09%, sys=10.30%, ctx=1090, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32270: Thu May 1 23:13:08 2025 read: IOPS=30, BW=489KiB/s (501kB/s)(6256KiB/12791msec) clat (usec): min=82, max=133757, avg=15340.18, stdev=17163.73 lat (usec): min=82, max=133759, avg=15346.18, stdev=17160.65 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 100], 10.00th=[ 113], 20.00th=[ 137], | 30.00th=[ 326], 40.00th=[ 6849], 50.00th=[ 12911], 60.00th=[ 16712], | 70.00th=[ 20579], 80.00th=[ 26608], 90.00th=[ 36439], 95.00th=[ 46924], | 99.00th=[ 74974], 99.50th=[ 87557], 99.90th=[133694], 99.95th=[133694], | 99.99th=[133694] bw ( KiB/s): min= 287, max= 758, per=28.82%, avg=495.92, stdev=162.13, samples=25 iops : min= 17, max= 47, avg=30.40, stdev=10.07, samples=25 write: IOPS=29, BW=472KiB/s (483kB/s)(6032KiB/12791msec) clat (usec): min=925, max=156215, avg=17872.89, stdev=22892.16 lat (usec): min=927, max=156218, avg=17876.20, stdev=22891.25 clat percentiles (usec): | 1.00th=[ 1029], 5.00th=[ 1221], 10.00th=[ 1385], 20.00th=[ 1696], | 30.00th=[ 2802], 40.00th=[ 5276], 50.00th=[ 8160], 60.00th=[ 13173], | 70.00th=[ 20841], 80.00th=[ 30016], 90.00th=[ 49021], 95.00th=[ 62129], | 99.00th=[104334], 99.50th=[143655], 99.90th=[156238], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 251, max= 734, per=26.16%, avg=473.44, stdev=142.13, samples=25 iops : min= 15, max= 45, avg=29.00, stdev= 8.93, samples=25 lat (usec) : 100=2.60%, 250=11.98%, 500=2.34%, 750=0.26%, 1000=0.65% lat (msec) : 2=11.98%, 4=7.29%, 10=11.59%, 20=20.70%, 50=23.57% lat (msec) : 100=6.12%, 250=0.91% cpu : usr=0.17%, sys=10.49%, ctx=1043, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32271: Thu May 1 23:13:08 2025 read: IOPS=27, BW=435KiB/s (445kB/s)(6032KiB/13871msec) clat (usec): min=66, max=131853, avg=16510.22, stdev=19566.24 lat (usec): min=66, max=131853, avg=16511.97, stdev=19565.91 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 98], 10.00th=[ 112], 20.00th=[ 130], | 30.00th=[ 182], 40.00th=[ 4293], 50.00th=[ 13173], 60.00th=[ 17433], | 70.00th=[ 21890], 80.00th=[ 30802], 90.00th=[ 41157], 95.00th=[ 47973], | 99.00th=[100140], 99.50th=[103285], 99.90th=[131597], 99.95th=[131597], | 99.99th=[131597] bw ( KiB/s): min= 188, max= 768, per=24.85%, avg=427.70, stdev=158.28, samples=27 iops : min= 11, max= 48, avg=26.22, stdev= 9.95, samples=27 write: IOPS=28, BW=451KiB/s (462kB/s)(6256KiB/13871msec) clat (usec): min=910, max=228996, avg=18884.60, stdev=25206.33 lat (usec): min=911, max=228997, avg=18889.32, stdev=25206.74 clat percentiles (usec): | 1.00th=[ 996], 5.00th=[ 1188], 10.00th=[ 1319], 20.00th=[ 1696], | 30.00th=[ 3359], 40.00th=[ 5538], 50.00th=[ 9372], 60.00th=[ 14353], | 70.00th=[ 21890], 80.00th=[ 33817], 90.00th=[ 47973], 95.00th=[ 63701], | 99.00th=[112722], 99.50th=[183501], 99.90th=[229639], 99.95th=[229639], | 99.99th=[229639] bw ( KiB/s): min= 160, max= 800, per=24.80%, avg=448.89, stdev=133.90, samples=27 iops : min= 10, max= 50, avg=27.56, stdev= 8.44, samples=27 lat (usec) : 100=2.73%, 250=12.63%, 500=2.34%, 750=0.39%, 1000=0.65% lat (msec) : 2=12.50%, 4=5.60%, 10=11.33%, 20=18.49%, 50=26.82% lat (msec) : 100=5.34%, 250=1.17% cpu : usr=0.13%, sys=10.19%, ctx=1099, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1721KiB/s (1762kB/s), 425KiB/s-489KiB/s (436kB/s-501kB/s), io=23.4MiB (24.5MB), run=12791-13917msec WRITE: bw=1811KiB/s (1854kB/s), 451KiB/s-493KiB/s (462kB/s-505kB/s), io=24.6MiB (25.8MB), run=12791-13917msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32272: Thu May 1 23:13:54 2025 read: IOPS=6, BW=98.5KiB/s (101kB/s)(5920KiB/60119msec) clat (msec): min=5, max=133, avg=18.74, stdev=16.76 lat (msec): min=5, max=133, avg=18.74, stdev=16.76 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 13], 50.00th=[ 14], 60.00th=[ 16], | 70.00th=[ 18], 80.00th=[ 24], 90.00th=[ 32], 95.00th=[ 52], | 99.00th=[ 104], 99.50th=[ 126], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 30, max= 416, per=28.16%, avg=111.79, stdev=66.91, samples=105 iops : min= 1, max= 26, avg= 6.47, stdev= 4.27, samples=105 write: IOPS=6, BW=106KiB/s (108kB/s)(6368KiB/60119msec) clat (msec): min=48, max=443, avg=133.48, stdev=51.99 lat (msec): min=48, max=443, avg=133.48, stdev=51.99 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 68], 10.00th=[ 81], 20.00th=[ 93], | 30.00th=[ 105], 40.00th=[ 114], 50.00th=[ 127], 60.00th=[ 136], | 70.00th=[ 150], 80.00th=[ 165], 90.00th=[ 190], 95.00th=[ 230], | 99.00th=[ 317], 99.50th=[ 326], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 31, max= 192, per=25.11%, avg=104.97, stdev=33.79, samples=120 iops : min= 1, max= 12, avg= 6.04, stdev= 2.16, samples=120 lat (msec) : 10=11.72%, 20=24.48%, 50=9.37%, 100=15.49%, 250=36.98% lat (msec) : 500=1.95% cpu : usr=0.05%, sys=1.87%, ctx=899, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32273: Thu May 1 23:13:54 2025 read: IOPS=5, BW=95.4KiB/s (97.7kB/s)(5744KiB/60181msec) clat (msec): min=4, max=186, avg=18.69, stdev=19.31 lat (msec): min=4, max=186, avg=18.69, stdev=19.31 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 13], 60.00th=[ 14], | 70.00th=[ 17], 80.00th=[ 22], 90.00th=[ 37], 95.00th=[ 54], | 99.00th=[ 110], 99.50th=[ 121], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 31, max= 320, per=27.35%, avg=108.59, stdev=72.60, samples=105 iops : min= 1, max= 20, avg= 6.31, stdev= 4.57, samples=105 write: IOPS=6, BW=109KiB/s (111kB/s)(6544KiB/60181msec) clat (msec): min=42, max=371, avg=130.44, stdev=50.06 lat (msec): min=42, max=371, avg=130.44, stdev=50.06 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 72], 10.00th=[ 78], 20.00th=[ 89], | 30.00th=[ 100], 40.00th=[ 111], 50.00th=[ 122], 60.00th=[ 134], | 70.00th=[ 150], 80.00th=[ 163], 90.00th=[ 194], 95.00th=[ 228], | 99.00th=[ 305], 99.50th=[ 321], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 31, max= 192, per=26.05%, avg=108.89, stdev=34.58, samples=119 iops : min= 1, max= 12, avg= 6.33, stdev= 2.25, samples=119 lat (msec) : 10=12.63%, 20=23.57%, 50=7.81%, 100=18.49%, 250=35.81% lat (msec) : 500=1.69% cpu : usr=0.02%, sys=1.89%, ctx=906, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32274: Thu May 1 23:13:54 2025 read: IOPS=6, BW=109KiB/s (112kB/s)(6256KiB/57272msec) clat (msec): min=3, max=144, avg=18.64, stdev=18.82 lat (msec): min=3, max=144, avg=18.65, stdev=18.82 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 22], 90.00th=[ 35], 95.00th=[ 54], | 99.00th=[ 112], 99.50th=[ 138], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 31, max= 416, per=29.86%, avg=118.56, stdev=83.86, samples=105 iops : min= 1, max= 26, avg= 6.97, stdev= 5.25, samples=105 write: IOPS=6, BW=105KiB/s (108kB/s)(6032KiB/57272msec) clat (msec): min=53, max=434, avg=131.48, stdev=54.55 lat (msec): min=53, max=434, avg=131.48, stdev=54.55 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 68], 10.00th=[ 79], 20.00th=[ 89], | 30.00th=[ 100], 40.00th=[ 109], 50.00th=[ 120], 60.00th=[ 134], | 70.00th=[ 146], 80.00th=[ 167], 90.00th=[ 199], 95.00th=[ 243], | 99.00th=[ 321], 99.50th=[ 414], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 31, max= 192, per=24.98%, avg=104.43, stdev=37.03, samples=114 iops : min= 1, max= 12, avg= 6.07, stdev= 2.42, samples=114 lat (msec) : 4=0.13%, 10=15.10%, 20=24.22%, 50=8.72%, 100=17.71% lat (msec) : 250=32.03%, 500=2.08% cpu : usr=0.03%, sys=1.96%, ctx=939, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32275: Thu May 1 23:13:54 2025 read: IOPS=6, BW=101KiB/s (103kB/s)(6032KiB/59680msec) clat (msec): min=5, max=163, avg=18.05, stdev=18.48 lat (msec): min=5, max=163, avg=18.06, stdev=18.48 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 14], | 70.00th=[ 17], 80.00th=[ 22], 90.00th=[ 35], 95.00th=[ 50], | 99.00th=[ 107], 99.50th=[ 155], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 31, max= 318, per=27.70%, avg=109.96, stdev=73.49, samples=108 iops : min= 1, max= 19, avg= 6.48, stdev= 4.67, samples=108 write: IOPS=6, BW=105KiB/s (107kB/s)(6256KiB/59680msec) clat (msec): min=52, max=380, avg=134.54, stdev=51.35 lat (msec): min=52, max=380, avg=134.55, stdev=51.35 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 72], 10.00th=[ 80], 20.00th=[ 92], | 30.00th=[ 103], 40.00th=[ 113], 50.00th=[ 126], 60.00th=[ 138], | 70.00th=[ 155], 80.00th=[ 174], 90.00th=[ 197], 95.00th=[ 228], | 99.00th=[ 313], 99.50th=[ 321], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 31, max= 192, per=24.98%, avg=104.40, stdev=34.63, samples=119 iops : min= 1, max= 12, avg= 6.13, stdev= 2.29, samples=119 lat (msec) : 10=15.23%, 20=22.92%, 50=8.59%, 100=15.10%, 250=36.33% lat (msec) : 500=1.82% cpu : usr=0.04%, sys=1.91%, ctx=911, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=398KiB/s (408kB/s), 95.4KiB/s-109KiB/s (97.7kB/s-112kB/s), io=23.4MiB (24.5MB), run=57272-60181msec WRITE: bw=419KiB/s (429kB/s), 105KiB/s-109KiB/s (107kB/s-111kB/s), io=24.6MiB (25.8MB), run=57272-60181msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32330: Thu May 1 23:13:59 2025 read: IOPS=4, BW=4474KiB/s (4582kB/s)(7168KiB/1602msec) clat (usec): min=919, max=155066, avg=98104.81, stdev=67580.52 lat (usec): min=920, max=155068, avg=98106.78, stdev=67580.91 clat percentiles (usec): | 1.00th=[ 922], 5.00th=[ 922], 10.00th=[ 922], 20.00th=[ 1172], | 30.00th=[122160], 40.00th=[122160], 50.00th=[125305], 60.00th=[127402], | 70.00th=[127402], 80.00th=[154141], 90.00th=[154141], 95.00th=[154141], | 99.00th=[154141], 99.50th=[154141], 99.90th=[154141], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 2027, max= 6131, per=55.03%, avg=4079.00, stdev=2052.00, samples=3 iops : min= 1, max= 5, avg= 3.00, stdev= 2.00, samples=3 write: IOPS=3, BW=3196KiB/s (3273kB/s)(5120KiB/1602msec) clat (msec): min=103, max=266, avg=166.96, stdev=65.45 lat (msec): min=103, max=266, avg=167.04, stdev=65.41 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 104], | 30.00th=[ 120], 40.00th=[ 120], 50.00th=[ 153], 60.00th=[ 153], | 70.00th=[ 194], 80.00th=[ 194], 90.00th=[ 266], 95.00th=[ 266], | 99.00th=[ 266], 99.50th=[ 266], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 2039, max= 6083, per=39.13%, avg=4061.00, stdev=2859.54, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 lat (usec) : 1000=8.33% lat (msec) : 2=8.33%, 250=75.00%, 500=8.33% cpu : usr=0.06%, sys=16.30%, ctx=307, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32331: Thu May 1 23:13:59 2025 read: IOPS=0, BW=839KiB/s (859kB/s)(2048KiB/2440msec) clat (msec): min=4, max=163, avg=84.15, stdev=112.00 lat (msec): min=4, max=163, avg=84.15, stdev=112.00 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 163], | 70.00th=[ 163], 80.00th=[ 163], 90.00th=[ 163], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 2039, max= 2048, per=27.57%, avg=2043.50, stdev= 6.36, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4197KiB/s (4297kB/s)(10.0MiB/2440msec) clat (msec): min=74, max=523, avg=220.77, stdev=151.16 lat (msec): min=74, max=523, avg=220.86, stdev=151.17 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 74], 20.00th=[ 94], | 30.00th=[ 96], 40.00th=[ 106], 50.00th=[ 167], 60.00th=[ 203], | 70.00th=[ 351], 80.00th=[ 351], 90.00th=[ 384], 95.00th=[ 523], | 99.00th=[ 523], 99.50th=[ 523], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 2048, max= 6144, per=45.97%, avg=4770.33, stdev=2357.64, samples=3 iops : min= 2, max= 6, avg= 4.33, stdev= 2.08, samples=3 lat (msec) : 10=8.33%, 100=25.00%, 250=41.67%, 500=16.67%, 750=8.33% cpu : usr=0.08%, sys=14.15%, ctx=109, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32332: Thu May 1 23:13:59 2025 read: IOPS=2, BW=2511KiB/s (2571kB/s)(6144KiB/2447msec) clat (usec): min=572, max=165538, avg=74007.64, stdev=81524.03 lat (usec): min=573, max=165540, avg=74008.94, stdev=81524.13 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 570], 10.00th=[ 570], 20.00th=[ 586], | 30.00th=[ 586], 40.00th=[ 1020], 50.00th=[ 1020], 60.00th=[122160], | 70.00th=[154141], 80.00th=[154141], 90.00th=[164627], 95.00th=[164627], | 99.00th=[164627], 99.50th=[164627], 99.90th=[164627], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 2043, max= 6083, per=54.82%, avg=4063.00, stdev=2856.71, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 write: IOPS=2, BW=2511KiB/s (2571kB/s)(6144KiB/2447msec) clat (msec): min=131, max=390, avg=285.75, stdev=89.16 lat (msec): min=131, max=390, avg=285.83, stdev=89.17 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 132], 10.00th=[ 132], 20.00th=[ 257], | 30.00th=[ 257], 40.00th=[ 275], 50.00th=[ 275], 60.00th=[ 330], | 70.00th=[ 334], 80.00th=[ 334], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 2027, max= 4096, per=26.23%, avg=2722.00, stdev=1189.95, samples=3 iops : min= 1, max= 4, avg= 2.00, stdev= 1.73, samples=3 lat (usec) : 750=16.67% lat (msec) : 2=8.33%, 250=33.33%, 500=41.67% cpu : usr=0.00%, sys=10.83%, ctx=241, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32333: Thu May 1 23:13:59 2025 read: IOPS=1, BW=1853KiB/s (1898kB/s)(5120KiB/2763msec) clat (usec): min=589, max=146273, avg=87812.26, stdev=79024.13 lat (usec): min=591, max=146275, avg=87813.90, stdev=79024.22 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 586], 10.00th=[ 586], 20.00th=[ 586], | 30.00th=[ 1909], 40.00th=[ 1909], 50.00th=[143655], 60.00th=[143655], | 70.00th=[145753], 80.00th=[145753], 90.00th=[145753], 95.00th=[145753], | 99.00th=[145753], 99.50th=[145753], 99.90th=[145753], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2594KiB/s (2657kB/s)(7168KiB/2763msec) clat (msec): min=147, max=436, avg=284.20, stdev=130.59 lat (msec): min=148, max=441, avg=284.99, stdev=131.53 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 148], 10.00th=[ 148], 20.00th=[ 150], | 30.00th=[ 153], 40.00th=[ 153], 50.00th=[ 326], 60.00th=[ 359], | 70.00th=[ 359], 80.00th=[ 418], 90.00th=[ 435], 95.00th=[ 435], | 99.00th=[ 435], 99.50th=[ 435], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 2039, max= 4087, per=23.64%, avg=2453.00, stdev=913.44, samples=5 iops : min= 1, max= 3, avg= 1.80, stdev= 0.84, samples=5 lat (usec) : 750=8.33% lat (msec) : 2=8.33%, 250=50.00%, 500=33.33% cpu : usr=0.00%, sys=9.92%, ctx=135, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7412KiB/s (7590kB/s), 839KiB/s-4474KiB/s (859kB/s-4582kB/s), io=20.0MiB (20.0MB), run=1602-2763msec WRITE: bw=10.1MiB/s (10.6MB/s), 2511KiB/s-4197KiB/s (2571kB/s-4297kB/s), io=28.0MiB (29.4MB), run=1602-2763msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32334: Thu May 1 23:14:00 2025 read: IOPS=3, BW=3473KiB/s (3556kB/s)(7168KiB/2064msec) clat (msec): min=46, max=167, avg=105.56, stdev=39.36 lat (msec): min=46, max=167, avg=105.56, stdev=39.36 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 70], | 30.00th=[ 97], 40.00th=[ 97], 50.00th=[ 116], 60.00th=[ 120], | 70.00th=[ 120], 80.00th=[ 123], 90.00th=[ 169], 95.00th=[ 169], | 99.00th=[ 169], 99.50th=[ 169], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 2031, max= 6131, per=49.42%, avg=3060.00, stdev=2047.34, samples=4 iops : min= 1, max= 5, avg= 2.00, stdev= 2.00, samples=4 write: IOPS=2, BW=2481KiB/s (2540kB/s)(5120KiB/2064msec) clat (msec): min=142, max=356, avg=246.72, stdev=103.21 lat (msec): min=143, max=357, avg=246.86, stdev=103.27 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 155], 50.00th=[ 230], 60.00th=[ 230], | 70.00th=[ 351], 80.00th=[ 351], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 2031, max= 4071, per=31.31%, avg=2715.00, stdev=1174.35, samples=3 iops : min= 1, max= 3, avg= 1.67, stdev= 1.15, samples=3 lat (msec) : 50=8.33%, 100=16.67%, 250=58.33%, 500=16.67% cpu : usr=0.05%, sys=4.36%, ctx=35, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32335: Thu May 1 23:14:00 2025 read: IOPS=0, BW=619KiB/s (634kB/s)(2048KiB/3307msec) clat (usec): min=37993, max=79338, avg=58666.22, stdev=29235.20 lat (usec): min=37996, max=79340, avg=58668.56, stdev=29234.22 clat percentiles (usec): | 1.00th=[38011], 5.00th=[38011], 10.00th=[38011], 20.00th=[38011], | 30.00th=[38011], 40.00th=[38011], 50.00th=[38011], 60.00th=[79168], | 70.00th=[79168], 80.00th=[79168], 90.00th=[79168], 95.00th=[79168], | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 2000, max= 2048, per=32.69%, avg=2024.00, stdev=33.94, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3096KiB/s (3171kB/s)(10.0MiB/3307msec) clat (msec): min=112, max=937, avg=296.80, stdev=262.13 lat (msec): min=112, max=937, avg=296.92, stdev=262.12 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 117], | 30.00th=[ 134], 40.00th=[ 148], 50.00th=[ 205], 60.00th=[ 230], | 70.00th=[ 279], 80.00th=[ 279], 90.00th=[ 575], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 2031, max= 6095, per=40.84%, avg=3541.25, stdev=1938.19, samples=4 iops : min= 1, max= 5, avg= 2.50, stdev= 1.91, samples=4 lat (msec) : 50=8.33%, 100=8.33%, 250=58.33%, 500=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=4.45%, ctx=55, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32336: Thu May 1 23:14:00 2025 read: IOPS=2, BW=2056KiB/s (2106kB/s)(6144KiB/2988msec) clat (msec): min=43, max=629, avg=202.10, stdev=218.18 lat (msec): min=43, max=629, avg=202.10, stdev=218.18 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 146], 50.00th=[ 146], 60.00th=[ 146], | 70.00th=[ 201], 80.00th=[ 201], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 2043, max= 4071, per=43.94%, avg=2720.67, stdev=1169.43, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 write: IOPS=2, BW=2056KiB/s (2106kB/s)(6144KiB/2988msec) clat (msec): min=189, max=346, avg=260.11, stdev=62.47 lat (msec): min=189, max=346, avg=260.20, stdev=62.48 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 190], 10.00th=[ 190], 20.00th=[ 199], | 30.00th=[ 199], 40.00th=[ 239], 50.00th=[ 239], 60.00th=[ 288], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 2031, max= 2048, per=23.52%, avg=2039.25, stdev= 7.68, samples=4 iops : min= 1, max= 2, avg= 1.25, stdev= 0.50, samples=4 lat (msec) : 50=16.67%, 250=50.00%, 500=25.00%, 750=8.33% cpu : usr=0.00%, sys=2.81%, ctx=44, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32337: Thu May 1 23:14:00 2025 read: IOPS=1, BW=1962KiB/s (2009kB/s)(5120KiB/2610msec) clat (msec): min=63, max=110, avg=84.93, stdev=22.82 lat (msec): min=63, max=110, avg=84.93, stdev=22.82 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 64], | 30.00th=[ 65], 40.00th=[ 65], 50.00th=[ 80], 60.00th=[ 80], | 70.00th=[ 107], 80.00th=[ 107], 90.00th=[ 111], 95.00th=[ 111], | 99.00th=[ 111], 99.50th=[ 111], 99.90th=[ 111], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 2048, max= 6059, per=65.46%, avg=4053.50, stdev=2836.21, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=2, BW=2746KiB/s (2812kB/s)(7168KiB/2610msec) clat (msec): min=163, max=382, avg=292.84, stdev=70.54 lat (msec): min=164, max=382, avg=293.13, stdev=70.30 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 163], 10.00th=[ 163], 20.00th=[ 243], | 30.00th=[ 305], 40.00th=[ 305], 50.00th=[ 309], 60.00th=[ 321], | 70.00th=[ 321], 80.00th=[ 326], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 2043, max= 6131, per=41.27%, avg=3578.25, stdev=1957.77, samples=4 iops : min= 1, max= 5, avg= 2.75, stdev= 2.06, samples=4 lat (msec) : 100=25.00%, 250=33.33%, 500=41.67% cpu : usr=0.04%, sys=2.18%, ctx=29, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6193KiB/s (6342kB/s), 619KiB/s-3473KiB/s (634kB/s-3556kB/s), io=20.0MiB (20.0MB), run=2064-3307msec WRITE: bw=8670KiB/s (8878kB/s), 2056KiB/s-3096KiB/s (2106kB/s-3171kB/s), io=28.0MiB (29.4MB), run=2064-3307msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32349: Thu May 1 23:14:05 2025 read: IOPS=1, BW=6450KiB/s (6605kB/s)(12.0MiB/1905msec) clat (msec): min=458, max=725, avg=551.92, stdev=150.11 lat (msec): min=458, max=725, avg=551.92, stdev=150.12 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 460], 10.00th=[ 460], 20.00th=[ 460], | 30.00th=[ 460], 40.00th=[ 472], 50.00th=[ 472], 60.00th=[ 472], | 70.00th=[ 726], 80.00th=[ 726], 90.00th=[ 726], 95.00th=[ 726], | 99.00th=[ 726], 99.50th=[ 726], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 6770, max= 8159, per=100.00%, avg=7464.50, stdev=982.17, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.11%, sys=11.40%, ctx=459, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32351: Thu May 1 23:14:05 2025 read: IOPS=0, BW=1426KiB/s (1460kB/s)(4096KiB/2872msec) clat (nsec): min=640888k, max=640888k, avg=640888494.00, stdev= 0.00 lat (nsec): min=640892k, max=640892k, avg=640891773.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 642], 5.00th=[ 642], 10.00th=[ 642], 20.00th=[ 642], | 30.00th=[ 642], 40.00th=[ 642], 50.00th=[ 642], 60.00th=[ 642], | 70.00th=[ 642], 80.00th=[ 642], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2852KiB/s (2921kB/s)(8192KiB/2872msec) clat (msec): min=963, max=968, avg=965.90, stdev= 3.60 lat (msec): min=963, max=968, avg=966.17, stdev= 3.62 clat percentiles (msec): | 1.00th=[ 961], 5.00th=[ 961], 10.00th=[ 961], 20.00th=[ 961], | 30.00th=[ 961], 40.00th=[ 961], 50.00th=[ 961], 60.00th=[ 969], | 70.00th=[ 969], 80.00th=[ 969], 90.00th=[ 969], 95.00th=[ 969], | 99.00th=[ 969], 99.50th=[ 969], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 7380, max= 7380, per=64.69%, avg=7380.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 750=33.33%, 1000=66.67% cpu : usr=0.00%, sys=9.20%, ctx=138, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32352: Thu May 1 23:14:05 2025 write: IOPS=1, BW=5067KiB/s (5189kB/s)(12.0MiB/2425msec) clat (msec): min=618, max=894, avg=712.26, stdev=158.20 lat (msec): min=618, max=895, avg=713.17, stdev=158.11 clat percentiles (msec): | 1.00th=[ 617], 5.00th=[ 617], 10.00th=[ 617], 20.00th=[ 617], | 30.00th=[ 617], 40.00th=[ 625], 50.00th=[ 625], 60.00th=[ 625], | 70.00th=[ 894], 80.00th=[ 894], 90.00th=[ 894], 95.00th=[ 894], | 99.00th=[ 894], 99.50th=[ 894], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 7460, max= 7571, per=65.87%, avg=7515.50, stdev=78.49, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 750=66.67%, 1000=33.33% cpu : usr=0.00%, sys=13.90%, ctx=139, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32353: Thu May 1 23:14:05 2025 write: IOPS=2, BW=8382KiB/s (8583kB/s)(12.0MiB/1466msec) clat (msec): min=286, max=666, avg=440.77, stdev=199.98 lat (msec): min=287, max=668, avg=442.03, stdev=200.46 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 288], 10.00th=[ 288], 20.00th=[ 288], | 30.00th=[ 288], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 372], | 70.00th=[ 667], 80.00th=[ 667], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min=16189, max=16189, per=100.00%, avg=16189.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.00%, sys=23.41%, ctx=79, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5705KiB/s (5842kB/s), 1426KiB/s-6450KiB/s (1460kB/s-6605kB/s), io=16.0MiB (16.8MB), run=1905-2872msec WRITE: bw=11.1MiB/s (11.7MB/s), 2852KiB/s-8382KiB/s (2921kB/s-8583kB/s), io=32.0MiB (33.6MB), run=1466-2872msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32346: Thu May 1 23:14:05 2025 read: IOPS=3, BW=12.4MiB/s (13.0MB/s)(12.0MiB/967msec) clat (msec): min=119, max=691, avg=313.53, stdev=326.94 lat (msec): min=119, max=691, avg=313.53, stdev=326.94 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 131], 50.00th=[ 131], 60.00th=[ 131], | 70.00th=[ 693], 80.00th=[ 693], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=1.66%, ctx=8, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32347: Thu May 1 23:14:05 2025 read: IOPS=0, BW=1774KiB/s (1817kB/s)(4096KiB/2309msec) clat (nsec): min=214815k, max=214815k, avg=214814814.00, stdev= 0.00 lat (nsec): min=214818k, max=214818k, avg=214817529.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 215], 10.00th=[ 215], 20.00th=[ 215], | 30.00th=[ 215], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 215], | 70.00th=[ 215], 80.00th=[ 215], 90.00th=[ 215], 95.00th=[ 215], | 99.00th=[ 215], 99.50th=[ 215], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3548KiB/s (3633kB/s)(8192KiB/2309msec) clat (msec): min=985, max=1017, avg=1001.30, stdev=23.02 lat (msec): min=985, max=1018, avg=1001.98, stdev=23.38 clat percentiles (msec): | 1.00th=[ 986], 5.00th=[ 986], 10.00th=[ 986], 20.00th=[ 986], | 30.00th=[ 986], 40.00th=[ 986], 50.00th=[ 986], 60.00th=[ 1020], | 70.00th=[ 1020], 80.00th=[ 1020], 90.00th=[ 1020], 95.00th=[ 1020], | 99.00th=[ 1020], 99.50th=[ 1020], 99.90th=[ 1020], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 8192, max= 8192, per=73.43%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.09%, sys=3.29%, ctx=18, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32348: Thu May 1 23:14:05 2025 write: IOPS=1, BW=4327KiB/s (4431kB/s)(12.0MiB/2840msec) clat (msec): min=501, max=1706, avg=931.59, stdev=672.14 lat (msec): min=501, max=1706, avg=932.00, stdev=672.02 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 502], 10.00th=[ 502], 20.00th=[ 502], | 30.00th=[ 502], 40.00th=[ 592], 50.00th=[ 592], 60.00th=[ 592], | 70.00th=[ 1703], 80.00th=[ 1703], 90.00th=[ 1703], 95.00th=[ 1703], | 99.00th=[ 1703], 99.50th=[ 1703], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 8175, max= 8175, per=73.28%, avg=8175.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 750=66.67% cpu : usr=0.00%, sys=4.02%, ctx=84, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32350: Thu May 1 23:14:05 2025 write: IOPS=1, BW=4184KiB/s (4284kB/s)(12.0MiB/2937msec) clat (msec): min=299, max=1333, avg=929.73, stdev=553.25 lat (msec): min=299, max=1334, avg=933.60, stdev=555.21 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 1150], 50.00th=[ 1150], 60.00th=[ 1150], | 70.00th=[ 1334], 80.00th=[ 1334], 90.00th=[ 1334], 95.00th=[ 1334], | 99.00th=[ 1334], 99.50th=[ 1334], 99.90th=[ 1334], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 8159, max= 8159, per=73.14%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=3.27%, ctx=50, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7096KiB/s (7266kB/s), 1774KiB/s-12.4MiB/s (1817kB/s-13.0MB/s), io=16.0MiB (16.8MB), run=967-2309msec WRITE: bw=10.9MiB/s (11.4MB/s), 3548KiB/s-4327KiB/s (3633kB/s-4431kB/s), io=32.0MiB (33.6MB), run=2309-2937msec