== sanity test 398b: DIO and buffer IO race ============== 10:23:54 (1743517434) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.1591 s, 43.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29826: Tue Apr 1 10:24:09 2025 read: IOPS=120, BW=482KiB/s (494kB/s)(5992KiB/12431msec) clat (usec): min=64, max=25196, avg=4511.70, stdev=2600.00 lat (usec): min=64, max=25197, avg=4514.10, stdev=2599.36 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 128], 10.00th=[ 2114], 20.00th=[ 2540], | 30.00th=[ 2966], 40.00th=[ 3425], 50.00th=[ 4015], 60.00th=[ 4686], | 70.00th=[ 5473], 80.00th=[ 6390], 90.00th=[ 7767], 95.00th=[ 8848], | 99.00th=[13042], 99.50th=[13829], 99.90th=[18220], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 296, max= 600, per=25.07%, avg=476.88, stdev=60.90, samples=24 iops : min= 74, max= 150, avg=119.17, stdev=15.20, samples=24 write: IOPS=126, BW=506KiB/s (519kB/s)(6296KiB/12431msec) clat (usec): min=437, max=160285, avg=3554.10, stdev=4983.32 lat (usec): min=437, max=160286, avg=3555.38, stdev=4983.38 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 545], 10.00th=[ 578], 20.00th=[ 685], | 30.00th=[ 881], 40.00th=[ 1876], 50.00th=[ 2769], 60.00th=[ 3490], | 70.00th=[ 4555], 80.00th=[ 5735], 90.00th=[ 7832], 95.00th=[ 9503], | 99.00th=[ 12780], 99.50th=[ 15664], 99.90th=[ 19268], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 360, max= 678, per=25.05%, avg=498.83, stdev=72.58, samples=24 iops : min= 90, max= 169, avg=124.67, stdev=18.09, samples=24 lat (usec) : 100=1.60%, 250=1.33%, 500=0.23%, 750=12.24%, 1000=4.69% lat (msec) : 2=4.36%, 4=33.01%, 10=38.87%, 20=3.61%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=22.80%, ctx=5745, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29827: Tue Apr 1 10:24:09 2025 read: IOPS=118, BW=474KiB/s (485kB/s)(5940KiB/12534msec) clat (usec): min=54, max=23903, avg=4553.22, stdev=2621.93 lat (usec): min=54, max=23904, avg=4553.85, stdev=2621.92 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 1811], 10.00th=[ 2180], 20.00th=[ 2671], | 30.00th=[ 3064], 40.00th=[ 3589], 50.00th=[ 4113], 60.00th=[ 4686], | 70.00th=[ 5407], 80.00th=[ 6128], 90.00th=[ 7439], 95.00th=[ 8848], | 99.00th=[14877], 99.50th=[16450], 99.90th=[23725], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 320, max= 624, per=24.91%, avg=473.84, stdev=58.66, samples=25 iops : min= 80, max= 156, avg=118.40, stdev=14.69, samples=25 write: IOPS=126, BW=506KiB/s (519kB/s)(6348KiB/12534msec) clat (usec): min=442, max=165360, avg=3588.63, stdev=5073.99 lat (usec): min=442, max=165361, avg=3589.40, stdev=5074.00 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 545], 10.00th=[ 578], 20.00th=[ 668], | 30.00th=[ 881], 40.00th=[ 1958], 50.00th=[ 2868], 60.00th=[ 3556], | 70.00th=[ 4490], 80.00th=[ 6063], 90.00th=[ 7767], 95.00th=[ 9372], | 99.00th=[ 13173], 99.50th=[ 14353], 99.90th=[ 21103], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 360, max= 736, per=25.35%, avg=504.80, stdev=82.44, samples=25 iops : min= 90, max= 184, avg=126.16, stdev=20.56, samples=25 lat (usec) : 100=1.66%, 250=0.49%, 500=0.16%, 750=12.50%, 1000=4.43% lat (msec) : 2=4.85%, 4=32.91%, 10=39.42%, 20=3.42%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=22.42%, ctx=5786, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29828: Tue Apr 1 10:24:09 2025 read: IOPS=119, BW=478KiB/s (489kB/s)(6028KiB/12621msec) clat (usec): min=59, max=158295, avg=4688.43, stdev=4739.86 lat (usec): min=59, max=158296, avg=4689.09, stdev=4739.89 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 139], 10.00th=[ 2147], 20.00th=[ 2573], | 30.00th=[ 2999], 40.00th=[ 3589], 50.00th=[ 4146], 60.00th=[ 4817], | 70.00th=[ 5538], 80.00th=[ 6390], 90.00th=[ 7701], 95.00th=[ 9110], | 99.00th=[ 13042], 99.50th=[ 14877], 99.90th=[ 24249], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 304, max= 584, per=24.92%, avg=474.04, stdev=66.84, samples=25 iops : min= 76, max= 146, avg=118.44, stdev=16.66, samples=25 write: IOPS=123, BW=496KiB/s (508kB/s)(6260KiB/12621msec) clat (usec): min=440, max=20644, avg=3503.57, stdev=3004.79 lat (usec): min=441, max=20645, avg=3504.36, stdev=3004.79 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 553], 10.00th=[ 594], 20.00th=[ 709], | 30.00th=[ 938], 40.00th=[ 2114], 50.00th=[ 2966], 60.00th=[ 3654], | 70.00th=[ 4424], 80.00th=[ 5997], 90.00th=[ 7701], 95.00th=[ 9372], | 99.00th=[12387], 99.50th=[14222], 99.90th=[18744], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 360, max= 640, per=24.70%, avg=491.68, stdev=63.04, samples=25 iops : min= 90, max= 160, avg=122.84, stdev=15.78, samples=25 lat (usec) : 100=1.82%, 250=0.72%, 500=0.26%, 750=11.49%, 1000=4.33% lat (msec) : 2=5.27%, 4=32.16%, 10=40.30%, 20=3.52%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=22.79%, ctx=5805, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29830: Tue Apr 1 10:24:09 2025 read: IOPS=120, BW=482KiB/s (494kB/s)(6052KiB/12556msec) clat (usec): min=51, max=164552, avg=4615.66, stdev=4864.48 lat (usec): min=52, max=164553, avg=4616.50, stdev=4864.54 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 469], 10.00th=[ 2180], 20.00th=[ 2573], | 30.00th=[ 2966], 40.00th=[ 3425], 50.00th=[ 4015], 60.00th=[ 4686], | 70.00th=[ 5407], 80.00th=[ 6194], 90.00th=[ 7635], 95.00th=[ 8979], | 99.00th=[ 13829], 99.50th=[ 15533], 99.90th=[ 22152], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 296, max= 640, per=25.30%, avg=481.12, stdev=67.87, samples=25 iops : min= 74, max= 160, avg=120.24, stdev=16.93, samples=25 write: IOPS=124, BW=497KiB/s (509kB/s)(6236KiB/12556msec) clat (usec): min=431, max=19356, avg=3528.65, stdev=3074.74 lat (usec): min=432, max=19357, avg=3529.37, stdev=3074.74 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 545], 10.00th=[ 578], 20.00th=[ 668], | 30.00th=[ 889], 40.00th=[ 2057], 50.00th=[ 2868], 60.00th=[ 3589], | 70.00th=[ 4555], 80.00th=[ 5997], 90.00th=[ 7832], 95.00th=[ 9503], | 99.00th=[12911], 99.50th=[15270], 99.90th=[18220], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 376, max= 680, per=24.87%, avg=495.24, stdev=71.38, samples=25 iops : min= 94, max= 170, avg=123.76, stdev=17.87, samples=25 lat (usec) : 100=1.79%, 250=0.65%, 500=0.29%, 750=11.98%, 1000=4.30% lat (msec) : 2=4.56%, 4=33.89%, 10=38.67%, 20=3.78%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=22.38%, ctx=5682, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1903KiB/s (1948kB/s), 474KiB/s-482KiB/s (485kB/s-494kB/s), io=23.4MiB (24.6MB), run=12431-12621msec WRITE: bw=1992KiB/s (2040kB/s), 496KiB/s-506KiB/s (508kB/s-519kB/s), io=24.6MiB (25.7MB), run=12431-12621msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29829: Tue Apr 1 10:24:14 2025 read: IOPS=87, BW=351KiB/s (360kB/s)(5992KiB/17055msec) clat (usec): min=1549, max=168500, avg=4676.96, stdev=5285.11 lat (usec): min=1550, max=168502, avg=4677.68, stdev=5285.16 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1975], 10.00th=[ 2114], 20.00th=[ 2343], | 30.00th=[ 2573], 40.00th=[ 2868], 50.00th=[ 3359], 60.00th=[ 4015], | 70.00th=[ 4817], 80.00th=[ 6194], 90.00th=[ 9503], 95.00th=[ 11338], | 99.00th=[ 16188], 99.50th=[ 18482], 99.90th=[ 24773], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 151, max= 688, per=24.92%, avg=350.18, stdev=136.97, samples=34 iops : min= 37, max= 172, avg=87.41, stdev=34.31, samples=34 write: IOPS=92, BW=369KiB/s (378kB/s)(6296KiB/17055msec) clat (usec): min=2268, max=31256, avg=6336.00, stdev=3637.18 lat (usec): min=2269, max=31257, avg=6336.83, stdev=3637.14 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 3032], 10.00th=[ 3261], 20.00th=[ 3589], | 30.00th=[ 3982], 40.00th=[ 4490], 50.00th=[ 5080], 60.00th=[ 5866], | 70.00th=[ 7046], 80.00th=[ 8848], 90.00th=[11076], 95.00th=[13566], | 99.00th=[19792], 99.50th=[24511], 99.90th=[29230], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 167, max= 656, per=25.04%, avg=368.29, stdev=134.44, samples=34 iops : min= 41, max= 164, avg=91.94, stdev=33.71, samples=34 lat (msec) : 2=2.73%, 4=42.06%, 10=44.08%, 20=10.48%, 50=0.62% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=11.16%, ctx=3660, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29831: Tue Apr 1 10:24:14 2025 read: IOPS=87, BW=350KiB/s (358kB/s)(5940KiB/16974msec) clat (usec): min=1633, max=165088, avg=4556.03, stdev=5158.74 lat (usec): min=1634, max=165090, avg=4556.74, stdev=5158.76 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1975], 10.00th=[ 2114], 20.00th=[ 2343], | 30.00th=[ 2573], 40.00th=[ 2835], 50.00th=[ 3228], 60.00th=[ 3818], | 70.00th=[ 4686], 80.00th=[ 5932], 90.00th=[ 9110], 95.00th=[ 11207], | 99.00th=[ 15533], 99.50th=[ 17695], 99.90th=[ 20579], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 111, max= 752, per=24.79%, avg=348.30, stdev=149.98, samples=33 iops : min= 27, max= 188, avg=87.03, stdev=37.54, samples=33 write: IOPS=93, BW=374KiB/s (383kB/s)(6348KiB/16974msec) clat (usec): min=2239, max=29275, avg=6387.70, stdev=3729.76 lat (usec): min=2239, max=29275, avg=6388.51, stdev=3729.76 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2999], 10.00th=[ 3228], 20.00th=[ 3654], | 30.00th=[ 4047], 40.00th=[ 4490], 50.00th=[ 5080], 60.00th=[ 5866], | 70.00th=[ 7111], 80.00th=[ 8586], 90.00th=[11338], 95.00th=[13304], | 99.00th=[22152], 99.50th=[24511], 99.90th=[27919], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 191, max= 656, per=25.06%, avg=368.67, stdev=128.54, samples=33 iops : min= 47, max= 164, avg=92.12, stdev=32.18, samples=33 lat (msec) : 2=2.77%, 4=42.25%, 10=44.24%, 20=10.06%, 50=0.65% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=11.22%, ctx=3662, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29832: Tue Apr 1 10:24:14 2025 read: IOPS=88, BW=353KiB/s (361kB/s)(6028KiB/17079msec) clat (usec): min=1545, max=24011, avg=4635.52, stdev=3241.85 lat (usec): min=1546, max=24011, avg=4636.26, stdev=3241.81 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1958], 10.00th=[ 2114], 20.00th=[ 2343], | 30.00th=[ 2606], 40.00th=[ 2933], 50.00th=[ 3425], 60.00th=[ 3982], | 70.00th=[ 4948], 80.00th=[ 6325], 90.00th=[ 9372], 95.00th=[11600], | 99.00th=[16450], 99.50th=[18220], 99.90th=[21890], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 160, max= 728, per=25.10%, avg=352.65, stdev=135.61, samples=34 iops : min= 40, max= 182, avg=88.12, stdev=33.93, samples=34 write: IOPS=91, BW=367KiB/s (375kB/s)(6260KiB/17079msec) clat (msec): min=2, max=173, avg= 6.41, stdev= 5.56 lat (msec): min=2, max=173, avg= 6.41, stdev= 5.56 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 21], 99.50th=[ 24], 99.90th=[ 29], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 176, max= 672, per=24.82%, avg=365.12, stdev=138.33, samples=34 iops : min= 44, max= 168, avg=91.24, stdev=34.61, samples=34 lat (msec) : 2=3.12%, 4=41.57%, 10=43.42%, 20=11.26%, 50=0.59% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=11.39%, ctx=3706, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29833: Tue Apr 1 10:24:14 2025 read: IOPS=88, BW=355KiB/s (363kB/s)(6052KiB/17050msec) clat (usec): min=1594, max=25180, avg=4551.80, stdev=3291.82 lat (usec): min=1594, max=25181, avg=4552.44, stdev=3291.81 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1975], 10.00th=[ 2114], 20.00th=[ 2311], | 30.00th=[ 2573], 40.00th=[ 2868], 50.00th=[ 3294], 60.00th=[ 3851], | 70.00th=[ 4621], 80.00th=[ 5997], 90.00th=[ 9372], 95.00th=[11469], | 99.00th=[16188], 99.50th=[19792], 99.90th=[25035], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 159, max= 680, per=25.23%, avg=354.53, stdev=137.81, samples=34 iops : min= 39, max= 170, avg=88.59, stdev=34.50, samples=34 write: IOPS=91, BW=366KiB/s (375kB/s)(6236KiB/17050msec) clat (msec): min=2, max=150, avg= 6.46, stdev= 5.13 lat (msec): min=2, max=150, avg= 6.46, stdev= 5.13 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 20], 99.50th=[ 24], 99.90th=[ 28], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 191, max= 656, per=24.77%, avg=364.41, stdev=136.07, samples=34 iops : min= 47, max= 164, avg=91.06, stdev=34.06, samples=34 lat (msec) : 2=2.96%, 4=42.94%, 10=42.64%, 20=10.81%, 50=0.62% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=11.10%, ctx=3655, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1406KiB/s (1440kB/s), 350KiB/s-355KiB/s (358kB/s-363kB/s), io=23.4MiB (24.6MB), run=16974-17079msec WRITE: bw=1472KiB/s (1507kB/s), 366KiB/s-374KiB/s (375kB/s-383kB/s), io=24.6MiB (25.7MB), run=16974-17079msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29862: Tue Apr 1 10:24:18 2025 read: IOPS=116, BW=1870KiB/s (1915kB/s)(5920KiB/3166msec) clat (usec): min=57, max=18826, avg=4376.50, stdev=2673.10 lat (usec): min=57, max=18826, avg=4377.07, stdev=2673.10 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 80], 10.00th=[ 121], 20.00th=[ 2704], | 30.00th=[ 3064], 40.00th=[ 3589], 50.00th=[ 4080], 60.00th=[ 4555], | 70.00th=[ 5211], 80.00th=[ 6194], 90.00th=[ 7373], 95.00th=[ 8717], | 99.00th=[13566], 99.50th=[17957], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 1408, max= 2176, per=25.56%, avg=1849.83, stdev=275.73, samples=6 iops : min= 88, max= 136, avg=115.50, stdev=17.16, samples=6 write: IOPS=125, BW=2011KiB/s (2060kB/s)(6368KiB/3166msec) clat (usec): min=700, max=17260, avg=3698.76, stdev=3213.87 lat (usec): min=701, max=17261, avg=3699.63, stdev=3213.92 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 775], 10.00th=[ 799], 20.00th=[ 848], | 30.00th=[ 1004], 40.00th=[ 1745], 50.00th=[ 2999], 60.00th=[ 3687], | 70.00th=[ 4817], 80.00th=[ 6063], 90.00th=[ 8455], 95.00th=[10814], | 99.00th=[13435], 99.50th=[16057], 99.90th=[17171], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 1664, max= 2368, per=26.05%, avg=1983.17, stdev=253.22, samples=6 iops : min= 104, max= 148, avg=123.83, stdev=15.78, samples=6 lat (usec) : 100=4.17%, 250=1.17%, 750=1.69%, 1000=13.80% lat (msec) : 2=6.38%, 4=29.43%, 10=38.41%, 20=4.95% cpu : usr=0.03%, sys=24.83%, ctx=1445, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29864: Tue Apr 1 10:24:18 2025 read: IOPS=114, BW=1835KiB/s (1879kB/s)(5744KiB/3131msec) clat (usec): min=57, max=12799, avg=4255.66, stdev=2475.59 lat (usec): min=57, max=12800, avg=4256.23, stdev=2475.60 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 80], 10.00th=[ 97], 20.00th=[ 2671], | 30.00th=[ 3130], 40.00th=[ 3523], 50.00th=[ 3949], 60.00th=[ 4621], | 70.00th=[ 5276], 80.00th=[ 6259], 90.00th=[ 7439], 95.00th=[ 8717], | 99.00th=[11338], 99.50th=[12387], 99.90th=[12780], 99.95th=[12780], | 99.99th=[12780] bw ( KiB/s): min= 1536, max= 2304, per=25.43%, avg=1840.00, stdev=285.32, samples=6 iops : min= 96, max= 144, avg=115.00, stdev=17.83, samples=6 write: IOPS=130, BW=2090KiB/s (2140kB/s)(6544KiB/3131msec) clat (usec): min=626, max=18688, avg=3871.18, stdev=3309.34 lat (usec): min=626, max=18688, avg=3872.33, stdev=3310.32 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 791], 10.00th=[ 807], 20.00th=[ 881], | 30.00th=[ 1156], 40.00th=[ 2147], 50.00th=[ 3032], 60.00th=[ 3851], | 70.00th=[ 5014], 80.00th=[ 6456], 90.00th=[ 8848], 95.00th=[10552], | 99.00th=[14877], 99.50th=[15926], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 1792, max= 2464, per=27.39%, avg=2085.33, stdev=247.39, samples=6 iops : min= 112, max= 154, avg=130.33, stdev=15.46, samples=6 lat (usec) : 100=4.95%, 250=0.78%, 750=1.69%, 1000=12.11% lat (msec) : 2=7.55%, 4=29.82%, 10=39.19%, 20=3.91% cpu : usr=0.13%, sys=25.02%, ctx=1456, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29866: Tue Apr 1 10:24:18 2025 read: IOPS=118, BW=1890KiB/s (1935kB/s)(6256KiB/3310msec) clat (usec): min=62, max=19597, avg=4526.15, stdev=2811.59 lat (usec): min=62, max=19597, avg=4526.79, stdev=2811.68 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 83], 10.00th=[ 265], 20.00th=[ 2737], | 30.00th=[ 3064], 40.00th=[ 3589], 50.00th=[ 4047], 60.00th=[ 4752], | 70.00th=[ 5473], 80.00th=[ 6521], 90.00th=[ 7767], 95.00th=[ 9110], | 99.00th=[13435], 99.50th=[17171], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 1248, max= 2304, per=24.99%, avg=1808.00, stdev=376.87, samples=6 iops : min= 78, max= 144, avg=113.00, stdev=23.55, samples=6 write: IOPS=113, BW=1822KiB/s (1866kB/s)(6032KiB/3310msec) clat (usec): min=686, max=21470, avg=3899.70, stdev=3419.46 lat (usec): min=687, max=21472, avg=3900.57, stdev=3419.47 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 791], 10.00th=[ 824], 20.00th=[ 889], | 30.00th=[ 1205], 40.00th=[ 2474], 50.00th=[ 3064], 60.00th=[ 3654], | 70.00th=[ 4490], 80.00th=[ 6325], 90.00th=[ 8717], 95.00th=[10552], | 99.00th=[15401], 99.50th=[20055], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 1408, max= 2112, per=23.33%, avg=1776.00, stdev=301.38, samples=6 iops : min= 88, max= 132, avg=111.00, stdev=18.84, samples=6 lat (usec) : 100=4.04%, 250=1.04%, 500=0.26%, 750=1.17%, 1000=11.59% lat (msec) : 2=5.60%, 4=32.55%, 10=38.93%, 20=4.56%, 50=0.26% cpu : usr=0.18%, sys=24.06%, ctx=1557, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29868: Tue Apr 1 10:24:18 2025 read: IOPS=119, BW=1906KiB/s (1952kB/s)(6032KiB/3164msec) clat (usec): min=63, max=15320, avg=4601.00, stdev=2698.94 lat (usec): min=63, max=15320, avg=4601.61, stdev=2699.01 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 80], 10.00th=[ 1991], 20.00th=[ 2737], | 30.00th=[ 3130], 40.00th=[ 3687], 50.00th=[ 4293], 60.00th=[ 4883], | 70.00th=[ 5473], 80.00th=[ 6587], 90.00th=[ 7767], 95.00th=[ 9372], | 99.00th=[13435], 99.50th=[14746], 99.90th=[15270], 99.95th=[15270], | 99.99th=[15270] bw ( KiB/s): min= 1472, max= 2112, per=26.09%, avg=1888.00, stdev=234.28, samples=6 iops : min= 92, max= 132, avg=118.00, stdev=14.64, samples=6 write: IOPS=123, BW=1977KiB/s (2025kB/s)(6256KiB/3164msec) clat (usec): min=664, max=18298, avg=3531.03, stdev=3199.40 lat (usec): min=665, max=18320, avg=3533.38, stdev=3199.06 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 758], 10.00th=[ 807], 20.00th=[ 865], | 30.00th=[ 1090], 40.00th=[ 1532], 50.00th=[ 2409], 60.00th=[ 3261], | 70.00th=[ 4228], 80.00th=[ 5997], 90.00th=[ 8225], 95.00th=[10421], | 99.00th=[13173], 99.50th=[13566], 99.90th=[18220], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 1696, max= 2560, per=25.85%, avg=1968.00, stdev=321.76, samples=6 iops : min= 106, max= 160, avg=123.00, stdev=20.11, samples=6 lat (usec) : 100=3.78%, 250=0.78%, 750=2.21%, 1000=11.46% lat (msec) : 2=10.42%, 4=27.34%, 10=38.80%, 20=5.21% cpu : usr=0.16%, sys=24.44%, ctx=1564, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7236KiB/s (7410kB/s), 1835KiB/s-1906KiB/s (1879kB/s-1952kB/s), io=23.4MiB (24.5MB), run=3131-3310msec WRITE: bw=7613KiB/s (7796kB/s), 1822KiB/s-2090KiB/s (1866kB/s-2140kB/s), io=24.6MiB (25.8MB), run=3131-3310msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29861: Tue Apr 1 10:24:19 2025 read: IOPS=85, BW=1372KiB/s (1405kB/s)(5920KiB/4316msec) clat (usec): min=1890, max=18417, avg=4556.55, stdev=3441.46 lat (usec): min=1891, max=18418, avg=4557.32, stdev=3441.35 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2089], 10.00th=[ 2180], 20.00th=[ 2343], | 30.00th=[ 2507], 40.00th=[ 2704], 50.00th=[ 3032], 60.00th=[ 3425], | 70.00th=[ 4228], 80.00th=[ 5997], 90.00th=[10683], 95.00th=[12780], | 99.00th=[16712], 99.50th=[17957], 99.90th=[18482], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 896, max= 2496, per=23.07%, avg=1280.00, stdev=525.81, samples=8 iops : min= 56, max= 156, avg=80.00, stdev=32.86, samples=8 write: IOPS=92, BW=1475KiB/s (1511kB/s)(6368KiB/4316msec) clat (usec): min=2769, max=24625, avg=6384.25, stdev=3535.08 lat (usec): min=2770, max=24627, avg=6385.38, stdev=3535.10 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3195], 10.00th=[ 3425], 20.00th=[ 3720], | 30.00th=[ 4015], 40.00th=[ 4359], 50.00th=[ 4883], 60.00th=[ 5604], | 70.00th=[ 7373], 80.00th=[ 8979], 90.00th=[11731], 95.00th=[13042], | 99.00th=[19006], 99.50th=[23987], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 992, max= 2496, per=23.84%, avg=1392.00, stdev=522.74, samples=8 iops : min= 62, max= 156, avg=87.00, stdev=32.67, samples=8 lat (msec) : 2=0.91%, 4=47.14%, 10=37.63%, 20=13.93%, 50=0.39% cpu : usr=0.23%, sys=11.68%, ctx=884, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29863: Tue Apr 1 10:24:19 2025 read: IOPS=83, BW=1338KiB/s (1370kB/s)(5744KiB/4292msec) clat (usec): min=1820, max=21140, avg=4366.70, stdev=3230.09 lat (usec): min=1820, max=21141, avg=4367.29, stdev=3230.10 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2057], 10.00th=[ 2180], 20.00th=[ 2311], | 30.00th=[ 2442], 40.00th=[ 2638], 50.00th=[ 2933], 60.00th=[ 3392], | 70.00th=[ 4178], 80.00th=[ 5800], 90.00th=[ 9896], 95.00th=[11863], | 99.00th=[15664], 99.50th=[19006], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 864, max= 2304, per=22.99%, avg=1275.75, stdev=498.70, samples=8 iops : min= 54, max= 144, avg=79.62, stdev=31.25, samples=8 write: IOPS=95, BW=1525KiB/s (1561kB/s)(6544KiB/4292msec) clat (usec): min=2506, max=22776, avg=6630.20, stdev=3860.05 lat (usec): min=2507, max=22777, avg=6631.18, stdev=3860.07 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3294], 10.00th=[ 3523], 20.00th=[ 3785], | 30.00th=[ 4015], 40.00th=[ 4293], 50.00th=[ 4817], 60.00th=[ 5735], | 70.00th=[ 7701], 80.00th=[ 9765], 90.00th=[12518], 95.00th=[13960], | 99.00th=[20317], 99.50th=[22676], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 1088, max= 2656, per=24.59%, avg=1435.63, stdev=538.81, samples=8 iops : min= 68, max= 166, avg=89.62, stdev=33.74, samples=8 lat (msec) : 2=1.56%, 4=45.83%, 10=37.89%, 20=13.93%, 50=0.78% cpu : usr=0.14%, sys=11.54%, ctx=927, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29865: Tue Apr 1 10:24:19 2025 read: IOPS=91, BW=1459KiB/s (1494kB/s)(6256KiB/4288msec) clat (usec): min=1843, max=20692, avg=4287.51, stdev=3313.76 lat (usec): min=1844, max=20693, avg=4288.31, stdev=3313.63 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2073], 10.00th=[ 2180], 20.00th=[ 2311], | 30.00th=[ 2474], 40.00th=[ 2638], 50.00th=[ 2900], 60.00th=[ 3195], | 70.00th=[ 3884], 80.00th=[ 5407], 90.00th=[ 8979], 95.00th=[11863], | 99.00th=[17695], 99.50th=[18744], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 800, max= 2560, per=24.58%, avg=1364.00, stdev=633.55, samples=8 iops : min= 50, max= 160, avg=85.25, stdev=39.60, samples=8 write: IOPS=87, BW=1407KiB/s (1440kB/s)(6032KiB/4288msec) clat (usec): min=2515, max=27684, avg=6821.98, stdev=4396.90 lat (usec): min=2515, max=27685, avg=6822.94, stdev=4396.91 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 3163], 10.00th=[ 3425], 20.00th=[ 3687], | 30.00th=[ 3916], 40.00th=[ 4146], 50.00th=[ 4817], 60.00th=[ 5866], | 70.00th=[ 7570], 80.00th=[10290], 90.00th=[13304], 95.00th=[15795], | 99.00th=[23200], 99.50th=[24773], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 896, max= 2592, per=22.88%, avg=1336.00, stdev=547.02, samples=8 iops : min= 56, max= 162, avg=83.50, stdev=34.19, samples=8 lat (msec) : 2=1.30%, 4=51.56%, 10=32.29%, 20=13.67%, 50=1.17% cpu : usr=0.30%, sys=11.57%, ctx=902, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29867: Tue Apr 1 10:24:19 2025 read: IOPS=88, BW=1413KiB/s (1447kB/s)(6032KiB/4269msec) clat (usec): min=1876, max=19815, avg=4479.73, stdev=3428.74 lat (usec): min=1876, max=19816, avg=4480.32, stdev=3428.74 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2073], 10.00th=[ 2180], 20.00th=[ 2311], | 30.00th=[ 2442], 40.00th=[ 2638], 50.00th=[ 2966], 60.00th=[ 3359], | 70.00th=[ 4113], 80.00th=[ 5997], 90.00th=[10421], 95.00th=[12911], | 99.00th=[16319], 99.50th=[18220], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 928, max= 2336, per=23.85%, avg=1323.62, stdev=470.22, samples=8 iops : min= 58, max= 146, avg=82.63, stdev=29.46, samples=8 write: IOPS=91, BW=1465KiB/s (1501kB/s)(6256KiB/4269msec) clat (usec): min=2397, max=26526, avg=6450.68, stdev=3737.09 lat (usec): min=2398, max=26527, avg=6451.69, stdev=3737.12 clat percentiles (usec): | 1.00th=[ 2802], 5.00th=[ 3195], 10.00th=[ 3425], 20.00th=[ 3752], | 30.00th=[ 3949], 40.00th=[ 4293], 50.00th=[ 4752], 60.00th=[ 5800], | 70.00th=[ 7373], 80.00th=[ 9110], 90.00th=[11469], 95.00th=[13698], | 99.00th=[20317], 99.50th=[24773], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 928, max= 2656, per=23.97%, avg=1399.62, stdev=575.84, samples=8 iops : min= 58, max= 166, avg=87.38, stdev=36.04, samples=8 lat (msec) : 2=1.04%, 4=49.22%, 10=36.72%, 20=12.50%, 50=0.52% cpu : usr=0.02%, sys=11.81%, ctx=908, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5550KiB/s (5683kB/s), 1338KiB/s-1459KiB/s (1370kB/s-1494kB/s), io=23.4MiB (24.5MB), run=4269-4316msec WRITE: bw=5839KiB/s (5979kB/s), 1407KiB/s-1525KiB/s (1440kB/s-1561kB/s), io=24.6MiB (25.8MB), run=4269-4316msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29877: Tue Apr 1 10:24:21 2025 read: IOPS=6, BW=6619KiB/s (6777kB/s)(7168KiB/1083msec) clat (usec): min=469, max=57531, avg=24147.70, stdev=19479.22 lat (usec): min=470, max=57533, avg=24148.76, stdev=19479.50 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 469], 10.00th=[ 469], 20.00th=[ 652], | 30.00th=[26870], 40.00th=[26870], 50.00th=[27132], 60.00th=[27657], | 70.00th=[27657], 80.00th=[28705], 90.00th=[57410], 95.00th=[57410], | 99.00th=[57410], 99.50th=[57410], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 8677, max= 8677, per=52.71%, avg=8677.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4728KiB/s (4841kB/s)(5120KiB/1083msec) clat (msec): min=24, max=572, avg=173.41, stdev=226.30 lat (msec): min=24, max=572, avg=173.45, stdev=226.31 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 105], 60.00th=[ 105], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 3471, max= 3471, per=15.06%, avg=3471.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=50.00%, 100=8.33%, 250=16.67%, 750=8.33% cpu : usr=0.00%, sys=19.50%, ctx=325, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29878: Tue Apr 1 10:24:21 2025 read: IOPS=1, BW=1646KiB/s (1686kB/s)(2048KiB/1244msec) clat (usec): min=55984, max=92041, avg=74013.11, stdev=25496.38 lat (usec): min=55985, max=92042, avg=74013.91, stdev=25496.05 clat percentiles (usec): | 1.00th=[55837], 5.00th=[55837], 10.00th=[55837], 20.00th=[55837], | 30.00th=[55837], 40.00th=[55837], 50.00th=[55837], 60.00th=[91751], | 70.00th=[91751], 80.00th=[91751], 90.00th=[91751], 95.00th=[91751], | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 1765, max= 1768, per=10.73%, avg=1766.50, stdev= 2.12, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=8, BW=8232KiB/s (8429kB/s)(10.0MiB/1244msec) clat (msec): min=24, max=603, avg=102.05, stdev=178.60 lat (msec): min=24, max=603, avg=102.10, stdev=178.61 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 27], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 41], 60.00th=[ 44], | 70.00th=[ 47], 80.00th=[ 47], 90.00th=[ 127], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 3531, max= 5305, per=19.17%, avg=4418.00, stdev=1254.41, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 lat (msec) : 50=66.67%, 100=16.67%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=23.65%, ctx=406, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29879: Tue Apr 1 10:24:21 2025 read: IOPS=6, BW=7111KiB/s (7282kB/s)(6144KiB/864msec) clat (usec): min=623, max=509237, avg=94349.40, stdev=203681.16 lat (usec): min=624, max=509238, avg=94350.66, stdev=203680.86 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 627], 10.00th=[ 627], 20.00th=[ 799], | 30.00th=[ 799], 40.00th=[ 1020], 50.00th=[ 1020], 60.00th=[ 22938], | 70.00th=[ 31327], 80.00th=[ 31327], 90.00th=[509608], 95.00th=[509608], | 99.00th=[509608], 99.50th=[509608], 99.90th=[509608], 99.95th=[509608], | 99.99th=[509608] bw ( KiB/s): min= 4087, max= 4087, per=24.83%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=6, BW=7111KiB/s (7282kB/s)(6144KiB/864msec) clat (msec): min=23, max=109, avg=46.92, stdev=31.60 lat (msec): min=23, max=109, avg=46.97, stdev=31.63 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 35], 50.00th=[ 35], 60.00th=[ 36], | 70.00th=[ 50], 80.00th=[ 50], 90.00th=[ 110], 95.00th=[ 110], | 99.00th=[ 110], 99.50th=[ 110], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 8175, max= 8175, per=35.47%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33%, 1000=8.33% lat (msec) : 2=8.33%, 50=58.33%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=28.16%, ctx=524, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29880: Tue Apr 1 10:24:21 2025 read: IOPS=11, BW=11.3MiB/s (11.8MB/s)(5120KiB/443msec) clat (usec): min=731, max=35167, avg=17436.89, stdev=15830.02 lat (usec): min=732, max=35168, avg=17437.69, stdev=15830.03 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 734], 10.00th=[ 734], 20.00th=[ 734], | 30.00th=[ 1680], 40.00th=[ 1680], 50.00th=[19792], 60.00th=[19792], | 70.00th=[30016], 80.00th=[30016], 90.00th=[35390], 95.00th=[35390], | 99.00th=[35390], 99.50th=[35390], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] write: IOPS=15, BW=15.8MiB/s (16.6MB/s)(7168KiB/443msec) clat (usec): min=19235, max=93273, avg=46618.11, stdev=30977.41 lat (usec): min=19264, max=93304, avg=46663.30, stdev=30968.36 clat percentiles (usec): | 1.00th=[19268], 5.00th=[19268], 10.00th=[19268], 20.00th=[25822], | 30.00th=[28181], 40.00th=[28181], 50.00th=[29230], 60.00th=[42730], | 70.00th=[42730], 80.00th=[88605], 90.00th=[92799], 95.00th=[92799], | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] lat (usec) : 750=8.33% lat (msec) : 2=8.33%, 20=16.67%, 50=50.00%, 100=16.67% cpu : usr=0.00%, sys=43.89%, ctx=388, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.1MiB/s (16.9MB/s), 1646KiB/s-11.3MiB/s (1686kB/s-11.8MB/s), io=20.0MiB (20.0MB), run=443-1244msec WRITE: bw=22.5MiB/s (23.6MB/s), 4728KiB/s-15.8MiB/s (4841kB/s-16.6MB/s), io=28.0MiB (29.4MB), run=443-1244msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29881: Tue Apr 1 10:24:21 2025 read: IOPS=5, BW=5493KiB/s (5625kB/s)(7168KiB/1305msec) clat (msec): min=15, max=154, avg=66.53, stdev=56.58 lat (msec): min=15, max=154, avg=66.53, stdev=56.58 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 21], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 43], 60.00th=[ 67], | 70.00th=[ 67], 80.00th=[ 136], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 6144, max= 6144, per=39.15%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3923KiB/s (4018kB/s)(5120KiB/1305msec) clat (msec): min=25, max=644, avg=161.41, stdev=270.20 lat (msec): min=25, max=644, avg=161.46, stdev=270.19 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 26], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 45], 60.00th=[ 45], | 70.00th=[ 58], 80.00th=[ 58], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 2048, max= 2048, per=9.32%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=16.67%, 250=16.67%, 750=8.33% cpu : usr=0.08%, sys=7.59%, ctx=65, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29882: Tue Apr 1 10:24:21 2025 read: IOPS=1, BW=1680KiB/s (1720kB/s)(2048KiB/1219msec) clat (msec): min=110, max=135, avg=122.74, stdev=17.85 lat (msec): min=110, max=135, avg=122.74, stdev=17.85 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 111], 50.00th=[ 111], 60.00th=[ 136], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 2048, max= 2048, per=13.05%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8400KiB/s (8602kB/s)(10.0MiB/1219msec) clat (msec): min=23, max=556, avg=96.29, stdev=162.69 lat (msec): min=23, max=556, avg=96.37, stdev=162.69 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 27], | 30.00th=[ 32], 40.00th=[ 34], 50.00th=[ 44], 60.00th=[ 50], | 70.00th=[ 58], 80.00th=[ 58], 90.00th=[ 88], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 4096, max= 6144, per=23.30%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 50=50.00%, 100=25.00%, 250=16.67%, 750=8.33% cpu : usr=0.16%, sys=9.03%, ctx=45, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29883: Tue Apr 1 10:24:21 2025 read: IOPS=4, BW=4896KiB/s (5013kB/s)(6144KiB/1255msec) clat (msec): min=15, max=654, avg=142.42, stdev=252.30 lat (msec): min=15, max=655, avg=142.42, stdev=252.30 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16], | 30.00th=[ 16], 40.00th=[ 42], 50.00th=[ 42], 60.00th=[ 47], | 70.00th=[ 82], 80.00th=[ 82], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 4096, max= 4096, per=26.10%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4896KiB/s (5013kB/s)(6144KiB/1255msec) clat (msec): min=23, max=111, avg=58.75, stdev=32.16 lat (msec): min=23, max=111, avg=58.79, stdev=32.16 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 41], 50.00th=[ 41], 60.00th=[ 66], | 70.00th=[ 75], 80.00th=[ 75], 90.00th=[ 112], 95.00th=[ 112], | 99.00th=[ 112], 99.50th=[ 112], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 8192, max= 8192, per=37.29%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=41.67%, 100=25.00%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=9.17%, ctx=155, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29884: Tue Apr 1 10:24:21 2025 read: IOPS=3, BW=4044KiB/s (4141kB/s)(5120KiB/1266msec) clat (usec): min=14855, max=88158, avg=43599.76, stdev=31274.48 lat (usec): min=14856, max=88159, avg=43600.58, stdev=31274.56 clat percentiles (usec): | 1.00th=[14877], 5.00th=[14877], 10.00th=[14877], 20.00th=[14877], | 30.00th=[15401], 40.00th=[15401], 50.00th=[39060], 60.00th=[39060], | 70.00th=[60556], 80.00th=[60556], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] write: IOPS=5, BW=5662KiB/s (5798kB/s)(7168KiB/1266msec) clat (msec): min=32, max=573, avg=144.06, stdev=193.07 lat (msec): min=32, max=573, avg=144.13, stdev=193.08 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 39], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 82], 60.00th=[ 117], | 70.00th=[ 117], 80.00th=[ 128], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 4096, max= 4096, per=18.64%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 lat (msec) : 20=16.67%, 50=33.33%, 100=25.00%, 250=16.67%, 750=8.33% cpu : usr=0.24%, sys=9.72%, ctx=55, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.3MiB/s (16.1MB/s), 1680KiB/s-5493KiB/s (1720kB/s-5625kB/s), io=20.0MiB (20.0MB), run=1219-1305msec WRITE: bw=21.5MiB/s (22.5MB/s), 3923KiB/s-8400KiB/s (4018kB/s-8602kB/s), io=28.0MiB (29.4MB), run=1219-1305msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29895: Tue Apr 1 10:24:23 2025 read: IOPS=13, BW=54.5MiB/s (57.2MB/s)(12.0MiB/220msec) clat (msec): min=2, max=110, avg=70.96, stdev=59.55 lat (msec): min=2, max=110, avg=70.96, stdev=59.55 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 3], 40.00th=[ 101], 50.00th=[ 101], 60.00th=[ 101], | 70.00th=[ 111], 80.00th=[ 111], 90.00th=[ 111], 95.00th=[ 111], | 99.00th=[ 111], 99.50th=[ 111], 99.90th=[ 111], 99.95th=[ 111], | 99.99th=[ 111] lat (msec) : 4=33.33%, 250=66.67% cpu : usr=0.46%, sys=39.73%, ctx=1005, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29896: Tue Apr 1 10:24:23 2025 read: IOPS=1, BW=6596KiB/s (6754kB/s)(4096KiB/621msec) clat (nsec): min=103459k, max=103459k, avg=103459127.00, stdev= 0.00 lat (nsec): min=103461k, max=103461k, avg=103461314.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 104], | 30.00th=[ 104], 40.00th=[ 104], 50.00th=[ 104], 60.00th=[ 104], | 70.00th=[ 104], 80.00th=[ 104], 90.00th=[ 104], 95.00th=[ 104], | 99.00th=[ 104], 99.50th=[ 104], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 8175, max= 8175, per=30.99%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=3, BW=12.9MiB/s (13.5MB/s)(8192KiB/621msec) clat (msec): min=106, max=385, avg=245.83, stdev=197.09 lat (msec): min=106, max=385, avg=246.08, stdev=197.05 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 107], 50.00th=[ 107], 60.00th=[ 384], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=35.32%, ctx=534, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29897: Tue Apr 1 10:24:23 2025 write: IOPS=5, BW=21.3MiB/s (22.3MB/s)(12.0MiB/563msec) clat (msec): min=103, max=327, avg=185.52, stdev=123.35 lat (msec): min=103, max=327, avg=185.79, stdev=123.36 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 104], | 30.00th=[ 104], 40.00th=[ 127], 50.00th=[ 127], 60.00th=[ 127], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min=16221, max=16221, per=30.74%, avg=16221.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.18%, sys=39.79%, ctx=31, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29898: Tue Apr 1 10:24:23 2025 write: IOPS=5, BW=23.4MiB/s (24.5MB/s)(12.0MiB/513msec) clat (msec): min=96, max=213, avg=161.30, stdev=59.17 lat (msec): min=97, max=213, avg=161.55, stdev=59.19 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 97], 10.00th=[ 97], 20.00th=[ 97], | 30.00th=[ 97], 40.00th=[ 174], 50.00th=[ 174], 60.00th=[ 174], | 70.00th=[ 213], 80.00th=[ 213], 90.00th=[ 213], 95.00th=[ 213], | 99.00th=[ 213], 99.50th=[ 213], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min=16384, max=16384, per=31.05%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.20%, sys=54.10%, ctx=38, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.8MiB/s (27.0MB/s), 6596KiB/s-54.5MiB/s (6754kB/s-57.2MB/s), io=16.0MiB (16.8MB), run=220-621msec WRITE: bw=51.5MiB/s (54.0MB/s), 12.9MiB/s-23.4MiB/s (13.5MB/s-24.5MB/s), io=32.0MiB (33.6MB), run=513-621msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29890: Tue Apr 1 10:24:23 2025 read: IOPS=6, BW=27.9MiB/s (29.3MB/s)(12.0MiB/430msec) clat (msec): min=46, max=322, avg=139.94, stdev=158.34 lat (msec): min=46, max=322, avg=139.94, stdev=158.34 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 51], | 70.00th=[ 321], 80.00th=[ 321], 90.00th=[ 321], 95.00th=[ 321], | 99.00th=[ 321], 99.50th=[ 321], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] lat (msec) : 50=33.33%, 100=33.33%, 500=33.33% cpu : usr=0.23%, sys=12.82%, ctx=45, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29891: Tue Apr 1 10:24:23 2025 read: IOPS=1, BW=7199KiB/s (7371kB/s)(4096KiB/569msec) clat (nsec): min=71119k, max=71119k, avg=71119027.00, stdev= 0.00 lat (nsec): min=71121k, max=71121k, avg=71121245.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[70779], 5.00th=[70779], 10.00th=[70779], 20.00th=[70779], | 30.00th=[70779], 40.00th=[70779], 50.00th=[70779], 60.00th=[70779], | 70.00th=[70779], 80.00th=[70779], 90.00th=[70779], 95.00th=[70779], | 99.00th=[70779], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 7406, max= 7406, per=25.72%, avg=7406.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=3, BW=14.1MiB/s (14.7MB/s)(8192KiB/569msec) clat (msec): min=138, max=351, avg=245.09, stdev=150.14 lat (msec): min=139, max=351, avg=245.23, stdev=150.15 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 140], 10.00th=[ 140], 20.00th=[ 140], | 30.00th=[ 140], 40.00th=[ 140], 50.00th=[ 140], 60.00th=[ 351], | 70.00th=[ 351], 80.00th=[ 351], 90.00th=[ 351], 95.00th=[ 351], | 99.00th=[ 351], 99.50th=[ 351], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 7406, max= 7406, per=17.45%, avg=7406.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=10.04%, ctx=30, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29892: Tue Apr 1 10:24:23 2025 write: IOPS=3, BW=15.5MiB/s (16.3MB/s)(12.0MiB/772msec) clat (msec): min=184, max=326, avg=251.42, stdev=71.32 lat (msec): min=184, max=326, avg=251.66, stdev=71.39 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 245], 50.00th=[ 245], 60.00th=[ 245], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min=15031, max=15031, per=35.41%, avg=15031.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.13%, sys=19.58%, ctx=66, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29894: Tue Apr 1 10:24:23 2025 write: IOPS=6, BW=27.2MiB/s (28.5MB/s)(12.0MiB/441msec) clat (msec): min=45, max=255, avg=137.01, stdev=107.17 lat (msec): min=46, max=255, avg=137.29, stdev=107.16 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 46], | 30.00th=[ 46], 40.00th=[ 110], 50.00th=[ 110], 60.00th=[ 110], | 70.00th=[ 255], 80.00th=[ 255], 90.00th=[ 255], 95.00th=[ 255], | 99.00th=[ 255], 99.50th=[ 255], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] lat (msec) : 50=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=13.41%, ctx=43, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=28.1MiB/s (29.5MB/s), 7199KiB/s-27.9MiB/s (7371kB/s-29.3MB/s), io=16.0MiB (16.8MB), run=430-569msec WRITE: bw=41.5MiB/s (43.5MB/s), 14.1MiB/s-27.2MiB/s (14.7MB/s-28.5MB/s), io=32.0MiB (33.6MB), run=441-772msec