== sanity test 398b: DIO and buffer IO race ============== 08:23:22 (1744028602) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.21257 s, 41.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32216: Mon Apr 7 08:23:37 2025 read: IOPS=125, BW=504KiB/s (516kB/s)(5992KiB/11891msec) clat (usec): min=57, max=150939, avg=4437.24, stdev=4492.85 lat (usec): min=58, max=150941, avg=4437.84, stdev=4492.87 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 239], 10.00th=[ 2114], 20.00th=[ 2507], | 30.00th=[ 2868], 40.00th=[ 3326], 50.00th=[ 3949], 60.00th=[ 4555], | 70.00th=[ 5276], 80.00th=[ 6128], 90.00th=[ 7177], 95.00th=[ 8586], | 99.00th=[ 12518], 99.50th=[ 13698], 99.90th=[ 19792], 99.95th=[149947], | 99.99th=[149947] bw ( KiB/s): min= 272, max= 616, per=25.21%, avg=500.00, stdev=72.73, samples=23 iops : min= 68, max= 154, avg=124.96, stdev=18.16, samples=23 write: IOPS=132, BW=529KiB/s (542kB/s)(6296KiB/11891msec) clat (usec): min=429, max=20607, avg=3307.14, stdev=2901.24 lat (usec): min=429, max=20608, avg=3307.99, stdev=2901.18 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 545], 10.00th=[ 586], 20.00th=[ 668], | 30.00th=[ 865], 40.00th=[ 1909], 50.00th=[ 2704], 60.00th=[ 3294], | 70.00th=[ 4178], 80.00th=[ 5538], 90.00th=[ 7308], 95.00th=[ 9110], | 99.00th=[12387], 99.50th=[13829], 99.90th=[18220], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 320, max= 616, per=25.24%, avg=524.04, stdev=67.93, samples=23 iops : min= 80, max= 154, avg=130.96, stdev=17.00, samples=23 lat (usec) : 100=1.66%, 250=0.78%, 500=0.26%, 750=12.86%, 1000=4.23% lat (msec) : 2=4.79%, 4=35.19%, 10=37.21%, 20=2.96%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=22.85%, ctx=5684, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32217: Mon Apr 7 08:23:37 2025 read: IOPS=123, BW=495KiB/s (507kB/s)(5940KiB/12004msec) clat (usec): min=56, max=19413, avg=4315.99, stdev=2354.41 lat (usec): min=56, max=19413, avg=4317.36, stdev=2353.50 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 157], 10.00th=[ 2114], 20.00th=[ 2507], | 30.00th=[ 2933], 40.00th=[ 3392], 50.00th=[ 3949], 60.00th=[ 4555], | 70.00th=[ 5145], 80.00th=[ 5932], 90.00th=[ 7111], 95.00th=[ 8586], | 99.00th=[11863], 99.50th=[13304], 99.90th=[17433], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 271, max= 662, per=24.92%, avg=494.25, stdev=85.21, samples=24 iops : min= 67, max= 165, avg=123.42, stdev=21.33, samples=24 write: IOPS=132, BW=529KiB/s (542kB/s)(6348KiB/12004msec) clat (usec): min=476, max=154105, avg=3498.51, stdev=4873.57 lat (usec): min=476, max=154106, avg=3499.46, stdev=4873.54 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 545], 10.00th=[ 578], 20.00th=[ 652], | 30.00th=[ 848], 40.00th=[ 1598], 50.00th=[ 2704], 60.00th=[ 3458], | 70.00th=[ 4359], 80.00th=[ 5866], 90.00th=[ 7767], 95.00th=[ 9241], | 99.00th=[ 13304], 99.50th=[ 14484], 99.90th=[ 21627], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 351, max= 736, per=25.44%, avg=528.21, stdev=77.87, samples=24 iops : min= 87, max= 184, avg=131.92, stdev=19.48, samples=24 lat (usec) : 100=1.95%, 250=0.59%, 500=0.29%, 750=13.09%, 1000=4.69% lat (msec) : 2=5.05%, 4=33.76%, 10=37.34%, 20=3.16%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=22.89%, ctx=5706, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32218: Mon Apr 7 08:23:37 2025 read: IOPS=124, BW=499KiB/s (511kB/s)(6028KiB/12069msec) clat (usec): min=52, max=156442, avg=4426.33, stdev=4660.58 lat (usec): min=52, max=156443, avg=4427.13, stdev=4660.54 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 106], 10.00th=[ 2024], 20.00th=[ 2442], | 30.00th=[ 2835], 40.00th=[ 3294], 50.00th=[ 3851], 60.00th=[ 4555], | 70.00th=[ 5211], 80.00th=[ 6259], 90.00th=[ 7504], 95.00th=[ 8586], | 99.00th=[ 12911], 99.50th=[ 14615], 99.90th=[ 23462], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 304, max= 680, per=25.07%, avg=497.17, stdev=83.59, samples=24 iops : min= 76, max= 170, avg=124.21, stdev=20.91, samples=24 write: IOPS=129, BW=519KiB/s (531kB/s)(6260KiB/12069msec) clat (usec): min=430, max=22976, avg=3424.42, stdev=3076.99 lat (usec): min=431, max=22977, avg=3425.59, stdev=3076.64 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 545], 10.00th=[ 578], 20.00th=[ 644], | 30.00th=[ 848], 40.00th=[ 1827], 50.00th=[ 2835], 60.00th=[ 3490], | 70.00th=[ 4359], 80.00th=[ 5669], 90.00th=[ 7635], 95.00th=[ 9372], | 99.00th=[12649], 99.50th=[14615], 99.90th=[21365], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 352, max= 672, per=24.80%, avg=514.75, stdev=84.84, samples=24 iops : min= 88, max= 168, avg=128.62, stdev=21.19, samples=24 lat (usec) : 100=2.31%, 250=0.68%, 500=0.55%, 750=13.25%, 1000=3.26% lat (msec) : 2=5.47%, 4=33.92%, 10=37.21%, 20=3.19%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=22.70%, ctx=5722, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32219: Mon Apr 7 08:23:37 2025 read: IOPS=124, BW=500KiB/s (512kB/s)(6052KiB/12105msec) clat (usec): min=53, max=19810, avg=4330.61, stdev=2290.67 lat (usec): min=53, max=19811, avg=4333.02, stdev=2290.51 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 1680], 10.00th=[ 2147], 20.00th=[ 2573], | 30.00th=[ 2999], 40.00th=[ 3425], 50.00th=[ 3982], 60.00th=[ 4490], | 70.00th=[ 5211], 80.00th=[ 5932], 90.00th=[ 7111], 95.00th=[ 8455], | 99.00th=[11338], 99.50th=[13566], 99.90th=[15795], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 256, max= 632, per=25.06%, avg=496.92, stdev=78.60, samples=24 iops : min= 64, max= 158, avg=124.21, stdev=19.65, samples=24 write: IOPS=128, BW=515KiB/s (528kB/s)(6236KiB/12105msec) clat (usec): min=408, max=150871, avg=3529.58, stdev=4761.70 lat (usec): min=408, max=150872, avg=3530.29, stdev=4761.72 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 545], 10.00th=[ 586], 20.00th=[ 685], | 30.00th=[ 898], 40.00th=[ 2212], 50.00th=[ 2900], 60.00th=[ 3425], | 70.00th=[ 4293], 80.00th=[ 5735], 90.00th=[ 7767], 95.00th=[ 9241], | 99.00th=[ 12387], 99.50th=[ 15664], 99.90th=[ 21890], 99.95th=[149947], | 99.99th=[149947] bw ( KiB/s): min= 336, max= 712, per=24.71%, avg=512.92, stdev=87.55, samples=24 iops : min= 84, max= 178, avg=128.21, stdev=21.89, samples=24 lat (usec) : 100=1.86%, 250=0.55%, 500=0.39%, 750=12.08%, 1000=3.74% lat (msec) : 2=4.39%, 4=35.45%, 10=38.80%, 20=2.67%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=22.75%, ctx=5736, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1984KiB/s (2031kB/s), 495KiB/s-504KiB/s (507kB/s-516kB/s), io=23.4MiB (24.6MB), run=11891-12105msec WRITE: bw=2077KiB/s (2127kB/s), 515KiB/s-529KiB/s (528kB/s-542kB/s), io=24.6MiB (25.7MB), run=11891-12105msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32211: Mon Apr 7 08:23:40 2025 read: IOPS=94, BW=379KiB/s (388kB/s)(5992KiB/15816msec) clat (usec): min=1580, max=155883, avg=4298.16, stdev=5066.57 lat (usec): min=1581, max=155884, avg=4298.80, stdev=5066.58 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1844], 10.00th=[ 1942], 20.00th=[ 2073], | 30.00th=[ 2245], 40.00th=[ 2507], 50.00th=[ 2835], 60.00th=[ 3392], | 70.00th=[ 4228], 80.00th=[ 5800], 90.00th=[ 8979], 95.00th=[ 11338], | 99.00th=[ 15664], 99.50th=[ 18220], 99.90th=[ 24511], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 152, max= 808, per=24.69%, avg=374.77, stdev=179.35, samples=31 iops : min= 38, max= 202, avg=93.61, stdev=44.80, samples=31 write: IOPS=99, BW=398KiB/s (408kB/s)(6296KiB/15816msec) clat (usec): min=2101, max=26113, avg=5916.10, stdev=3551.16 lat (usec): min=2102, max=26114, avg=5916.92, stdev=3551.13 clat percentiles (usec): | 1.00th=[ 2474], 5.00th=[ 2769], 10.00th=[ 2966], 20.00th=[ 3228], | 30.00th=[ 3425], 40.00th=[ 3818], 50.00th=[ 4424], 60.00th=[ 5604], | 70.00th=[ 6915], 80.00th=[ 8586], 90.00th=[10683], 95.00th=[12911], | 99.00th=[18482], 99.50th=[20841], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 168, max= 824, per=24.79%, avg=393.87, stdev=177.93, samples=31 iops : min= 42, max= 206, avg=98.39, stdev=44.44, samples=31 lat (msec) : 2=7.00%, 4=48.89%, 10=33.85%, 20=9.70%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=11.49%, ctx=3693, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32212: Mon Apr 7 08:23:40 2025 read: IOPS=93, BW=376KiB/s (385kB/s)(5940KiB/15808msec) clat (usec): min=1589, max=157895, avg=4340.18, stdev=5123.68 lat (usec): min=1590, max=157897, avg=4340.85, stdev=5123.68 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1860], 10.00th=[ 1942], 20.00th=[ 2073], | 30.00th=[ 2245], 40.00th=[ 2507], 50.00th=[ 2835], 60.00th=[ 3523], | 70.00th=[ 4228], 80.00th=[ 5932], 90.00th=[ 9110], 95.00th=[ 11207], | 99.00th=[ 16057], 99.50th=[ 19006], 99.90th=[ 24511], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 128, max= 744, per=24.66%, avg=374.29, stdev=185.33, samples=31 iops : min= 32, max= 186, avg=93.48, stdev=46.35, samples=31 write: IOPS=100, BW=402KiB/s (411kB/s)(6348KiB/15808msec) clat (usec): min=2189, max=28622, avg=5858.93, stdev=3561.94 lat (usec): min=2190, max=28622, avg=5859.72, stdev=3562.00 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2769], 10.00th=[ 2966], 20.00th=[ 3195], | 30.00th=[ 3392], 40.00th=[ 3785], 50.00th=[ 4424], 60.00th=[ 5342], | 70.00th=[ 6718], 80.00th=[ 8291], 90.00th=[10945], 95.00th=[12911], | 99.00th=[17695], 99.50th=[21627], 99.90th=[26608], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 200, max= 808, per=24.89%, avg=395.45, stdev=172.96, samples=31 iops : min= 50, max= 202, avg=98.77, stdev=43.27, samples=31 lat (msec) : 2=7.55%, 4=47.40%, 10=34.67%, 20=9.93%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=11.41%, ctx=3783, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32213: Mon Apr 7 08:23:40 2025 read: IOPS=95, BW=382KiB/s (391kB/s)(6028KiB/15799msec) clat (usec): min=1522, max=22966, avg=4190.83, stdev=3147.74 lat (usec): min=1523, max=22967, avg=4191.56, stdev=3147.73 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1860], 10.00th=[ 1958], 20.00th=[ 2089], | 30.00th=[ 2278], 40.00th=[ 2507], 50.00th=[ 2868], 60.00th=[ 3392], | 70.00th=[ 4113], 80.00th=[ 5997], 90.00th=[ 8717], 95.00th=[10945], | 99.00th=[16450], 99.50th=[17957], 99.90th=[22676], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 168, max= 768, per=24.95%, avg=378.77, stdev=173.83, samples=31 iops : min= 42, max= 192, avg=94.65, stdev=43.49, samples=31 write: IOPS=99, BW=396KiB/s (406kB/s)(6260KiB/15799msec) clat (msec): min=2, max=161, avg= 6.02, stdev= 5.26 lat (msec): min=2, max=161, avg= 6.02, stdev= 5.26 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 19], 99.50th=[ 21], 99.90th=[ 28], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 216, max= 800, per=24.71%, avg=392.71, stdev=183.82, samples=31 iops : min= 54, max= 200, avg=98.13, stdev=45.97, samples=31 lat (msec) : 2=6.32%, 4=48.99%, 10=34.15%, 20=10.12%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=11.72%, ctx=3718, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32214: Mon Apr 7 08:23:40 2025 read: IOPS=95, BW=384KiB/s (393kB/s)(6052KiB/15779msec) clat (usec): min=1524, max=24671, avg=4331.62, stdev=3443.86 lat (usec): min=1524, max=24672, avg=4332.27, stdev=3443.85 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1844], 10.00th=[ 1942], 20.00th=[ 2089], | 30.00th=[ 2245], 40.00th=[ 2507], 50.00th=[ 2900], 60.00th=[ 3425], | 70.00th=[ 4228], 80.00th=[ 5932], 90.00th=[ 9372], 95.00th=[11600], | 99.00th=[17695], 99.50th=[20579], 99.90th=[23725], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 168, max= 854, per=25.05%, avg=380.19, stdev=177.41, samples=31 iops : min= 42, max= 213, avg=94.97, stdev=44.30, samples=31 write: IOPS=98, BW=395KiB/s (405kB/s)(6236KiB/15779msec) clat (msec): min=2, max=159, avg= 5.88, stdev= 5.14 lat (msec): min=2, max=159, avg= 5.88, stdev= 5.14 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 18], 99.50th=[ 22], 99.90th=[ 29], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 192, max= 782, per=24.66%, avg=391.81, stdev=178.91, samples=31 iops : min= 48, max= 195, avg=97.87, stdev=44.69, samples=31 lat (msec) : 2=6.67%, 4=48.57%, 10=34.90%, 20=9.28%, 50=0.55% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=11.40%, ctx=3707, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1518KiB/s (1555kB/s), 376KiB/s-384KiB/s (385kB/s-393kB/s), io=23.4MiB (24.6MB), run=15779-15816msec WRITE: bw=1590KiB/s (1628kB/s), 395KiB/s-402KiB/s (405kB/s-411kB/s), io=24.6MiB (25.7MB), run=15779-15816msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32250: Mon Apr 7 08:23:45 2025 read: IOPS=109, BW=1752KiB/s (1794kB/s)(5920KiB/3379msec) clat (usec): min=60, max=15086, avg=4674.04, stdev=2760.27 lat (usec): min=61, max=15087, avg=4676.53, stdev=2762.85 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 84], 10.00th=[ 151], 20.00th=[ 2737], | 30.00th=[ 3228], 40.00th=[ 3818], 50.00th=[ 4490], 60.00th=[ 5145], | 70.00th=[ 5800], 80.00th=[ 6521], 90.00th=[ 7832], 95.00th=[ 9372], | 99.00th=[14222], 99.50th=[14877], 99.90th=[15139], 99.95th=[15139], | 99.99th=[15139] bw ( KiB/s): min= 1120, max= 2016, per=23.84%, avg=1690.00, stdev=311.18, samples=6 iops : min= 70, max= 126, avg=105.50, stdev=19.45, samples=6 write: IOPS=117, BW=1885KiB/s (1930kB/s)(6368KiB/3379msec) clat (usec): min=693, max=20289, avg=4059.47, stdev=3435.11 lat (usec): min=694, max=20290, avg=4060.40, stdev=3435.18 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 799], 10.00th=[ 832], 20.00th=[ 955], | 30.00th=[ 1336], 40.00th=[ 2442], 50.00th=[ 3163], 60.00th=[ 4047], | 70.00th=[ 5080], 80.00th=[ 6652], 90.00th=[ 8455], 95.00th=[10683], | 99.00th=[17171], 99.50th=[19268], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 1408, max= 2400, per=24.67%, avg=1839.33, stdev=425.40, samples=6 iops : min= 88, max= 150, avg=114.83, stdev=26.69, samples=6 lat (usec) : 100=4.04%, 250=0.78%, 500=0.26%, 750=0.91%, 1000=11.33% lat (msec) : 2=6.90%, 4=26.82%, 10=43.62%, 20=5.21%, 50=0.13% cpu : usr=0.15%, sys=24.27%, ctx=1728, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32251: Mon Apr 7 08:23:45 2025 read: IOPS=107, BW=1720KiB/s (1762kB/s)(5744KiB/3339msec) clat (usec): min=66, max=18771, avg=4673.36, stdev=2855.03 lat (usec): min=67, max=18772, avg=4678.16, stdev=2853.93 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 85], 10.00th=[ 2147], 20.00th=[ 2769], | 30.00th=[ 3228], 40.00th=[ 3589], 50.00th=[ 4146], 60.00th=[ 4883], | 70.00th=[ 5604], 80.00th=[ 6456], 90.00th=[ 8094], 95.00th=[ 9896], | 99.00th=[14484], 99.50th=[16057], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 1472, max= 2016, per=23.85%, avg=1690.67, stdev=215.54, samples=6 iops : min= 92, max= 126, avg=105.67, stdev=13.47, samples=6 write: IOPS=122, BW=1960KiB/s (2007kB/s)(6544KiB/3339msec) clat (usec): min=664, max=17918, avg=3983.54, stdev=3219.73 lat (usec): min=665, max=17919, avg=3984.73, stdev=3219.43 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 832], 10.00th=[ 881], 20.00th=[ 1057], | 30.00th=[ 1434], 40.00th=[ 2606], 50.00th=[ 3064], 60.00th=[ 3851], | 70.00th=[ 5211], 80.00th=[ 6456], 90.00th=[ 8291], 95.00th=[10945], | 99.00th=[13829], 99.50th=[15008], 99.90th=[17957], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 1632, max= 2272, per=25.53%, avg=1904.00, stdev=226.05, samples=6 iops : min= 102, max= 142, avg=119.00, stdev=14.13, samples=6 lat (usec) : 100=2.99%, 250=1.04%, 500=0.26%, 750=0.91%, 1000=8.33% lat (msec) : 2=10.55%, 4=30.21%, 10=40.23%, 20=5.47% cpu : usr=0.15%, sys=25.34%, ctx=1662, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32253: Mon Apr 7 08:23:45 2025 read: IOPS=118, BW=1897KiB/s (1942kB/s)(6256KiB/3298msec) clat (usec): min=63, max=20125, avg=4561.58, stdev=3004.82 lat (usec): min=64, max=20126, avg=4562.28, stdev=3004.92 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 81], 10.00th=[ 135], 20.00th=[ 2638], | 30.00th=[ 3261], 40.00th=[ 3785], 50.00th=[ 4293], 60.00th=[ 4686], | 70.00th=[ 5342], 80.00th=[ 6194], 90.00th=[ 7701], 95.00th=[ 9634], | 99.00th=[17433], 99.50th=[17695], 99.90th=[20055], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 1504, max= 2272, per=26.11%, avg=1850.67, stdev=334.65, samples=6 iops : min= 94, max= 142, avg=115.67, stdev=20.92, samples=6 write: IOPS=114, BW=1829KiB/s (1873kB/s)(6032KiB/3298msec) clat (usec): min=631, max=16062, avg=3971.13, stdev=3122.54 lat (usec): min=631, max=16063, avg=3972.01, stdev=3122.54 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 791], 10.00th=[ 873], 20.00th=[ 971], | 30.00th=[ 1336], 40.00th=[ 2442], 50.00th=[ 3130], 60.00th=[ 3949], | 70.00th=[ 5604], 80.00th=[ 6587], 90.00th=[ 8586], 95.00th=[ 9896], | 99.00th=[14091], 99.50th=[15664], 99.90th=[16057], 99.95th=[16057], | 99.99th=[16057] bw ( KiB/s): min= 1472, max= 2080, per=23.89%, avg=1781.33, stdev=213.87, samples=6 iops : min= 92, max= 130, avg=111.33, stdev=13.37, samples=6 lat (usec) : 100=4.04%, 250=1.69%, 500=0.13%, 750=1.17%, 1000=9.64% lat (msec) : 2=7.42%, 4=28.26%, 10=43.10%, 20=4.43%, 50=0.13% cpu : usr=0.06%, sys=24.66%, ctx=1678, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32255: Mon Apr 7 08:23:45 2025 read: IOPS=112, BW=1801KiB/s (1844kB/s)(6032KiB/3350msec) clat (usec): min=68, max=18603, avg=4589.42, stdev=2635.33 lat (usec): min=69, max=18604, avg=4590.03, stdev=2635.33 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 87], 10.00th=[ 1516], 20.00th=[ 2769], | 30.00th=[ 3326], 40.00th=[ 3687], 50.00th=[ 4424], 60.00th=[ 4948], | 70.00th=[ 5669], 80.00th=[ 6456], 90.00th=[ 7504], 95.00th=[ 8455], | 99.00th=[13960], 99.50th=[16057], 99.90th=[18482], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 1472, max= 2048, per=24.83%, avg=1760.00, stdev=222.62, samples=6 iops : min= 92, max= 128, avg=110.00, stdev=13.91, samples=6 write: IOPS=116, BW=1867KiB/s (1912kB/s)(6256KiB/3350msec) clat (usec): min=686, max=16565, avg=4012.76, stdev=3108.33 lat (usec): min=687, max=16566, avg=4014.12, stdev=3108.21 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 816], 10.00th=[ 873], 20.00th=[ 1139], | 30.00th=[ 1549], 40.00th=[ 2442], 50.00th=[ 3130], 60.00th=[ 3982], | 70.00th=[ 5211], 80.00th=[ 6783], 90.00th=[ 8717], 95.00th=[ 9765], | 99.00th=[13829], 99.50th=[15664], 99.90th=[16581], 99.95th=[16581], | 99.99th=[16581] bw ( KiB/s): min= 1568, max= 2208, per=24.75%, avg=1845.33, stdev=226.88, samples=6 iops : min= 98, max= 138, avg=115.33, stdev=14.18, samples=6 lat (usec) : 100=3.39%, 250=1.17%, 500=0.26%, 750=0.39%, 1000=7.81% lat (msec) : 2=9.51%, 4=30.08%, 10=43.49%, 20=3.91% cpu : usr=0.12%, sys=24.90%, ctx=1697, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7088KiB/s (7259kB/s), 1720KiB/s-1897KiB/s (1762kB/s-1942kB/s), io=23.4MiB (24.5MB), run=3298-3379msec WRITE: bw=7458KiB/s (7637kB/s), 1829KiB/s-1960KiB/s (1873kB/s-2007kB/s), io=24.6MiB (25.8MB), run=3298-3379msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32252: Mon Apr 7 08:23:46 2025 read: IOPS=87, BW=1397KiB/s (1430kB/s)(5920KiB/4238msec) clat (usec): min=1984, max=18424, avg=4409.42, stdev=3002.70 lat (usec): min=1985, max=18424, avg=4410.04, stdev=3002.69 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2147], 10.00th=[ 2245], 20.00th=[ 2442], | 30.00th=[ 2606], 40.00th=[ 2835], 50.00th=[ 3064], 60.00th=[ 3490], | 70.00th=[ 4293], 80.00th=[ 5669], 90.00th=[ 9241], 95.00th=[11600], | 99.00th=[14877], 99.50th=[15926], 99.90th=[18482], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 990, max= 2304, per=24.18%, avg=1323.75, stdev=434.60, samples=8 iops : min= 61, max= 144, avg=82.62, stdev=27.26, samples=8 write: IOPS=93, BW=1503KiB/s (1539kB/s)(6368KiB/4238msec) clat (usec): min=2454, max=32529, avg=6492.11, stdev=3790.48 lat (usec): min=2455, max=32530, avg=6493.23, stdev=3790.43 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 3195], 10.00th=[ 3458], 20.00th=[ 3720], | 30.00th=[ 4047], 40.00th=[ 4424], 50.00th=[ 5080], 60.00th=[ 6128], | 70.00th=[ 7308], 80.00th=[ 8717], 90.00th=[11338], 95.00th=[13304], | 99.00th=[21365], 99.50th=[26346], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 1152, max= 2528, per=25.06%, avg=1443.62, stdev=450.10, samples=8 iops : min= 72, max= 158, avg=90.12, stdev=28.13, samples=8 lat (msec) : 2=0.26%, 4=46.74%, 10=41.54%, 20=10.81%, 50=0.65% cpu : usr=0.14%, sys=12.41%, ctx=964, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32254: Mon Apr 7 08:23:46 2025 read: IOPS=83, BW=1329KiB/s (1361kB/s)(5744KiB/4322msec) clat (usec): min=1938, max=27176, avg=4580.62, stdev=3695.25 lat (usec): min=1938, max=27177, avg=4581.26, stdev=3695.22 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2147], 10.00th=[ 2245], 20.00th=[ 2442], | 30.00th=[ 2606], 40.00th=[ 2802], 50.00th=[ 3032], 60.00th=[ 3326], | 70.00th=[ 4293], 80.00th=[ 5932], 90.00th=[ 9765], 95.00th=[13173], | 99.00th=[19530], 99.50th=[23725], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 704, max= 2208, per=22.94%, avg=1255.62, stdev=442.52, samples=8 iops : min= 44, max= 138, avg=78.38, stdev=27.62, samples=8 write: IOPS=94, BW=1514KiB/s (1550kB/s)(6544KiB/4322msec) clat (usec): min=2613, max=24236, avg=6458.43, stdev=3797.77 lat (usec): min=2614, max=24238, avg=6459.61, stdev=3797.76 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3064], 10.00th=[ 3294], 20.00th=[ 3589], | 30.00th=[ 3851], 40.00th=[ 4178], 50.00th=[ 4817], 60.00th=[ 6063], | 70.00th=[ 7439], 80.00th=[ 9372], 90.00th=[12125], 95.00th=[14222], | 99.00th=[19006], 99.50th=[21890], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 1117, max= 2528, per=24.58%, avg=1415.62, stdev=464.24, samples=8 iops : min= 69, max= 158, avg=88.38, stdev=29.09, samples=8 lat (msec) : 2=0.39%, 4=50.52%, 10=36.07%, 20=12.11%, 50=0.91% cpu : usr=0.21%, sys=12.22%, ctx=957, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32256: Mon Apr 7 08:23:46 2025 read: IOPS=92, BW=1482KiB/s (1518kB/s)(6256KiB/4221msec) clat (usec): min=1857, max=18614, avg=4575.83, stdev=3247.81 lat (usec): min=1858, max=18615, avg=4576.45, stdev=3247.79 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2114], 10.00th=[ 2278], 20.00th=[ 2474], | 30.00th=[ 2606], 40.00th=[ 2802], 50.00th=[ 3064], 60.00th=[ 3621], | 70.00th=[ 4621], 80.00th=[ 6128], 90.00th=[ 9241], 95.00th=[12387], | 99.00th=[16188], 99.50th=[18220], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 992, max= 2624, per=26.29%, avg=1439.12, stdev=517.57, samples=8 iops : min= 62, max= 164, avg=89.75, stdev=32.33, samples=8 write: IOPS=89, BW=1429KiB/s (1463kB/s)(6032KiB/4221msec) clat (usec): min=2722, max=32374, avg=6424.91, stdev=3828.75 lat (usec): min=2723, max=32375, avg=6426.06, stdev=3828.69 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3261], 10.00th=[ 3458], 20.00th=[ 3752], | 30.00th=[ 3949], 40.00th=[ 4293], 50.00th=[ 5014], 60.00th=[ 5932], | 70.00th=[ 6915], 80.00th=[ 8586], 90.00th=[11731], 95.00th=[14615], | 99.00th=[20841], 99.50th=[25560], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 1024, max= 2272, per=23.60%, avg=1359.25, stdev=392.82, samples=8 iops : min= 64, max= 142, avg=84.75, stdev=24.70, samples=8 lat (msec) : 2=0.78%, 4=47.66%, 10=39.97%, 20=11.07%, 50=0.52% cpu : usr=0.19%, sys=12.56%, ctx=983, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32257: Mon Apr 7 08:23:46 2025 read: IOPS=86, BW=1379KiB/s (1412kB/s)(6032KiB/4375msec) clat (usec): min=1946, max=32660, avg=4799.52, stdev=3849.86 lat (usec): min=1946, max=32661, avg=4800.16, stdev=3849.85 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2114], 10.00th=[ 2245], 20.00th=[ 2409], | 30.00th=[ 2540], 40.00th=[ 2737], 50.00th=[ 2933], 60.00th=[ 3556], | 70.00th=[ 4817], 80.00th=[ 6849], 90.00th=[10552], 95.00th=[12518], | 99.00th=[18744], 99.50th=[21627], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 864, max= 2368, per=23.38%, avg=1279.62, stdev=470.27, samples=8 iops : min= 54, max= 148, avg=79.88, stdev=29.39, samples=8 write: IOPS=89, BW=1430KiB/s (1464kB/s)(6256KiB/4375msec) clat (usec): min=2665, max=25300, avg=6435.76, stdev=3853.14 lat (usec): min=2666, max=25301, avg=6436.89, stdev=3853.07 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3097], 10.00th=[ 3359], 20.00th=[ 3687], | 30.00th=[ 3884], 40.00th=[ 4228], 50.00th=[ 4752], 60.00th=[ 5800], | 70.00th=[ 7504], 80.00th=[ 8979], 90.00th=[11469], 95.00th=[13435], | 99.00th=[21365], 99.50th=[24773], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 928, max= 2464, per=23.05%, avg=1327.63, stdev=521.87, samples=8 iops : min= 58, max= 154, avg=82.88, stdev=32.61, samples=8 lat (msec) : 2=0.39%, 4=48.18%, 10=38.02%, 20=12.24%, 50=1.17% cpu : usr=0.02%, sys=12.44%, ctx=972, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5475KiB/s (5606kB/s), 1329KiB/s-1482KiB/s (1361kB/s-1518kB/s), io=23.4MiB (24.5MB), run=4221-4375msec WRITE: bw=5760KiB/s (5898kB/s), 1429KiB/s-1514KiB/s (1463kB/s-1550kB/s), io=24.6MiB (25.8MB), run=4221-4375msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32265: Mon Apr 7 08:23:47 2025 read: IOPS=11, BW=11.4MiB/s (11.9MB/s)(7168KiB/616msec) clat (usec): min=513, max=115649, avg=36207.50, stdev=38918.19 lat (usec): min=514, max=115650, avg=36208.89, stdev=38918.05 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 515], 10.00th=[ 515], 20.00th=[ 594], | 30.00th=[ 21365], 40.00th=[ 21365], 50.00th=[ 38011], 60.00th=[ 38536], | 70.00th=[ 38536], 80.00th=[ 38536], 90.00th=[115868], 95.00th=[115868], | 99.00th=[115868], 99.50th=[115868], 99.90th=[115868], 99.95th=[115868], | 99.99th=[115868] bw ( KiB/s): min=10178, max=10178, per=30.61%, avg=10178.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8312KiB/s (8511kB/s)(5120KiB/616msec) clat (msec): min=32, max=118, avg=63.29, stdev=38.03 lat (msec): min=32, max=118, avg=63.35, stdev=38.02 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 43], 60.00th=[ 43], | 70.00th=[ 88], 80.00th=[ 88], 90.00th=[ 118], 95.00th=[ 118], | 99.00th=[ 118], 99.50th=[ 118], 99.90th=[ 118], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 6107, max= 6107, per=13.12%, avg=6107.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 50=58.33%, 100=8.33%, 250=16.67% cpu : usr=0.00%, sys=29.43%, ctx=514, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32266: Mon Apr 7 08:23:47 2025 read: IOPS=3, BW=3357KiB/s (3438kB/s)(2048KiB/610msec) clat (usec): min=30381, max=44940, avg=37661.03, stdev=10295.31 lat (usec): min=30381, max=44942, avg=37662.05, stdev=10295.73 clat percentiles (usec): | 1.00th=[30278], 5.00th=[30278], 10.00th=[30278], 20.00th=[30278], | 30.00th=[30278], 40.00th=[30278], 50.00th=[30278], 60.00th=[44827], | 70.00th=[44827], 80.00th=[44827], 90.00th=[44827], 95.00th=[44827], | 99.00th=[44827], 99.50th=[44827], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 4096, max= 4096, per=12.32%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=16, BW=16.4MiB/s (17.2MB/s)(10.0MiB/610msec) clat (usec): min=22311, max=88292, avg=49118.04, stdev=25009.37 lat (usec): min=22367, max=88451, avg=49202.48, stdev=25030.64 clat percentiles (usec): | 1.00th=[22414], 5.00th=[22414], 10.00th=[22414], 20.00th=[26608], | 30.00th=[28705], 40.00th=[38536], 50.00th=[39060], 60.00th=[39060], | 70.00th=[80217], 80.00th=[80217], 90.00th=[83362], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min=16384, max=16384, per=35.20%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 16, max= 16, avg=16.00, stdev= 0.00, samples=1 lat (msec) : 50=75.00%, 100=25.00% cpu : usr=0.00%, sys=46.96%, ctx=295, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32267: Mon Apr 7 08:23:47 2025 read: IOPS=11, BW=11.9MiB/s (12.5MB/s)(6144KiB/503msec) clat (usec): min=543, max=93751, avg=31844.46, stdev=34913.70 lat (usec): min=543, max=93751, avg=31845.50, stdev=34913.64 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 545], 10.00th=[ 545], 20.00th=[ 619], | 30.00th=[ 619], 40.00th=[19268], 50.00th=[19268], 60.00th=[32900], | 70.00th=[43779], 80.00th=[43779], 90.00th=[93848], 95.00th=[93848], | 99.00th=[93848], 99.50th=[93848], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min=10240, max=10240, per=30.80%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.9MiB/s (12.5MB/s)(6144KiB/503msec) clat (usec): min=31083, max=84265, avg=44847.35, stdev=19909.68 lat (usec): min=31175, max=84344, avg=44906.81, stdev=19911.87 clat percentiles (usec): | 1.00th=[31065], 5.00th=[31065], 10.00th=[31065], 20.00th=[32637], | 30.00th=[32637], 40.00th=[35914], 50.00th=[35914], 60.00th=[41157], | 70.00th=[43779], 80.00th=[43779], 90.00th=[84411], 95.00th=[84411], | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min=12288, max=12288, per=26.40%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 20=8.33%, 50=58.33%, 100=16.67% cpu : usr=0.00%, sys=34.86%, ctx=171, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32268: Mon Apr 7 08:23:47 2025 read: IOPS=8, BW=8605KiB/s (8812kB/s)(5120KiB/595msec) clat (usec): min=5518, max=65880, avg=29771.67, stdev=22646.29 lat (usec): min=5520, max=65880, avg=29772.90, stdev=22645.87 clat percentiles (usec): | 1.00th=[ 5538], 5.00th=[ 5538], 10.00th=[ 5538], 20.00th=[ 5538], | 30.00th=[19006], 40.00th=[19006], 50.00th=[24511], 60.00th=[24511], | 70.00th=[33817], 80.00th=[33817], 90.00th=[65799], 95.00th=[65799], | 99.00th=[65799], 99.50th=[65799], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 8126, max= 8126, per=24.44%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.8MiB/s (12.3MB/s)(7168KiB/595msec) clat (msec): min=40, max=123, avg=62.75, stdev=29.46 lat (msec): min=40, max=123, avg=62.82, stdev=29.46 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 51], 60.00th=[ 56], | 70.00th=[ 56], 80.00th=[ 77], 90.00th=[ 125], 95.00th=[ 125], | 99.00th=[ 125], 99.50th=[ 125], 99.90th=[ 125], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min=10158, max=10158, per=21.82%, avg=10158.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 10=8.33%, 20=8.33%, 50=41.67%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=33.50%, ctx=407, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=32.5MiB/s (34.0MB/s), 3357KiB/s-11.9MiB/s (3438kB/s-12.5MB/s), io=20.0MiB (20.0MB), run=503-616msec WRITE: bw=45.5MiB/s (47.7MB/s), 8312KiB/s-16.4MiB/s (8511kB/s-17.2MB/s), io=28.0MiB (29.4MB), run=503-616msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32270: Mon Apr 7 08:23:47 2025 read: IOPS=10, BW=10.7MiB/s (11.2MB/s)(7168KiB/655msec) clat (msec): min=15, max=113, avg=39.06, stdev=34.81 lat (msec): min=15, max=113, avg=39.07, stdev=34.81 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 23], 40.00th=[ 23], 50.00th=[ 26], 60.00th=[ 29], | 70.00th=[ 29], 80.00th=[ 53], 90.00th=[ 113], 95.00th=[ 113], | 99.00th=[ 113], 99.50th=[ 113], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min=10240, max=10240, per=34.30%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7817KiB/s (8004kB/s)(5120KiB/655msec) clat (usec): min=32543, max=81198, avg=53187.60, stdev=19015.11 lat (usec): min=32644, max=81274, avg=53273.62, stdev=19010.12 clat percentiles (usec): | 1.00th=[32637], 5.00th=[32637], 10.00th=[32637], 20.00th=[32637], | 30.00th=[43254], 40.00th=[43254], 50.00th=[46400], 60.00th=[46400], | 70.00th=[62653], 80.00th=[62653], 90.00th=[81265], 95.00th=[81265], | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 4096, max= 4096, per=9.80%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=50.00%, 100=25.00%, 250=8.33% cpu : usr=0.15%, sys=13.91%, ctx=65, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32271: Mon Apr 7 08:23:47 2025 read: IOPS=2, BW=3048KiB/s (3121kB/s)(2048KiB/672msec) clat (usec): min=12026, max=14147, avg=13086.73, stdev=1499.65 lat (usec): min=12027, max=14149, avg=13088.69, stdev=1500.50 clat percentiles (usec): | 1.00th=[11994], 5.00th=[11994], 10.00th=[11994], 20.00th=[11994], | 30.00th=[11994], 40.00th=[11994], 50.00th=[11994], 60.00th=[14091], | 70.00th=[14091], 80.00th=[14091], 90.00th=[14091], 95.00th=[14091], | 99.00th=[14091], 99.50th=[14091], 99.90th=[14091], 99.95th=[14091], | 99.99th=[14091] bw ( KiB/s): min= 4087, max= 4087, per=13.69%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=14, BW=14.9MiB/s (15.6MB/s)(10.0MiB/672msec) clat (msec): min=20, max=106, avg=55.61, stdev=25.30 lat (msec): min=20, max=106, avg=55.68, stdev=25.30 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 35], | 30.00th=[ 39], 40.00th=[ 45], 50.00th=[ 46], 60.00th=[ 56], | 70.00th=[ 77], 80.00th=[ 77], 90.00th=[ 79], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min=12263, max=12263, per=29.34%, avg=12263.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=41.67%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=21.01%, ctx=94, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32272: Mon Apr 7 08:23:47 2025 read: IOPS=9, BW=9752KiB/s (9986kB/s)(6144KiB/630msec) clat (usec): min=13334, max=97667, avg=40253.35, stdev=31142.19 lat (usec): min=13336, max=97668, avg=40254.83, stdev=31142.03 clat percentiles (usec): | 1.00th=[13304], 5.00th=[13304], 10.00th=[13304], 20.00th=[16909], | 30.00th=[16909], 40.00th=[30016], 50.00th=[30016], 60.00th=[32637], | 70.00th=[51119], 80.00th=[51119], 90.00th=[98042], 95.00th=[98042], | 99.00th=[98042], 99.50th=[98042], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 6144, max= 6144, per=20.58%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9752KiB/s (9986kB/s)(6144KiB/630msec) clat (msec): min=19, max=109, avg=59.48, stdev=38.92 lat (msec): min=19, max=109, avg=59.62, stdev=38.95 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 72], | 70.00th=[ 99], 80.00th=[ 99], 90.00th=[ 110], 95.00th=[ 110], | 99.00th=[ 110], 99.50th=[ 110], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 8192, max= 8192, per=19.60%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=33.33%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=17.65%, ctx=99, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32273: Mon Apr 7 08:23:47 2025 read: IOPS=7, BW=7464KiB/s (7643kB/s)(5120KiB/686msec) clat (msec): min=15, max=146, avg=57.68, stdev=53.11 lat (msec): min=15, max=146, avg=57.68, stdev=53.11 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16], | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 54], 60.00th=[ 54], | 70.00th=[ 54], 80.00th=[ 54], 90.00th=[ 146], 95.00th=[ 146], | 99.00th=[ 146], 99.50th=[ 146], 99.90th=[ 146], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 2048, max= 2048, per=6.86%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.2MiB/s (10.7MB/s)(7168KiB/686msec) clat (msec): min=15, max=106, avg=44.15, stdev=35.71 lat (msec): min=15, max=106, avg=44.24, stdev=35.71 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 19], | 30.00th=[ 22], 40.00th=[ 22], 50.00th=[ 23], 60.00th=[ 45], | 70.00th=[ 45], 80.00th=[ 81], 90.00th=[ 107], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 8192, max= 8192, per=19.60%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=25.00%, 100=25.00%, 250=16.67% cpu : usr=0.29%, sys=16.35%, ctx=79, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=29.2MiB/s (30.6MB/s), 3048KiB/s-10.7MiB/s (3121kB/s-11.2MB/s), io=20.0MiB (20.0MB), run=630-686msec WRITE: bw=40.8MiB/s (42.8MB/s), 7817KiB/s-14.9MiB/s (8004kB/s-15.6MB/s), io=28.0MiB (29.4MB), run=630-686msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32279: Mon Apr 7 08:23:49 2025 read: IOPS=9, BW=37.2MiB/s (38.0MB/s)(12.0MiB/323msec) clat (usec): min=1789, max=205377, avg=102908.78, stdev=101800.96 lat (usec): min=1790, max=205378, avg=102910.14, stdev=101800.99 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1795], 10.00th=[ 1795], 20.00th=[ 1795], | 30.00th=[ 1795], 40.00th=[101188], 50.00th=[101188], 60.00th=[101188], | 70.00th=[204473], 80.00th=[204473], 90.00th=[204473], 95.00th=[204473], | 99.00th=[204473], 99.50th=[204473], 99.90th=[204473], 99.95th=[204473], | 99.99th=[204473] lat (msec) : 2=33.33%, 250=66.67% cpu : usr=0.00%, sys=31.68%, ctx=809, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32280: Mon Apr 7 08:23:49 2025 read: IOPS=1, BW=5603KiB/s (5738kB/s)(4096KiB/731msec) clat (nsec): min=101960k, max=101960k, avg=101960187.00, stdev= 0.00 lat (nsec): min=101962k, max=101962k, avg=101962052.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 103], 10.00th=[ 103], 20.00th=[ 103], | 30.00th=[ 103], 40.00th=[ 103], 50.00th=[ 103], 60.00th=[ 103], | 70.00th=[ 103], 80.00th=[ 103], 90.00th=[ 103], 95.00th=[ 103], | 99.00th=[ 103], 99.50th=[ 103], 99.90th=[ 103], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min= 8192, max= 8192, per=36.55%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.9MiB/s (11.5MB/s)(8192KiB/731msec) clat (msec): min=217, max=346, avg=281.79, stdev=91.38 lat (msec): min=217, max=346, avg=282.10, stdev=91.34 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 218], 10.00th=[ 218], 20.00th=[ 218], | 30.00th=[ 218], 40.00th=[ 218], 50.00th=[ 218], 60.00th=[ 347], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 8192, max= 8192, per=18.28%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=26.58%, ctx=387, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32281: Mon Apr 7 08:23:49 2025 write: IOPS=4, BW=19.3MiB/s (20.2MB/s)(12.0MiB/623msec) clat (msec): min=203, max=205, avg=204.50, stdev= 1.05 lat (msec): min=204, max=206, avg=204.81, stdev= 1.09 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 205], 10.00th=[ 205], 20.00th=[ 205], | 30.00th=[ 205], 40.00th=[ 205], 50.00th=[ 205], 60.00th=[ 205], | 70.00th=[ 207], 80.00th=[ 207], 90.00th=[ 207], 95.00th=[ 207], | 99.00th=[ 207], 99.50th=[ 207], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min=16384, max=16384, per=36.55%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=42.60%, ctx=64, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32282: Mon Apr 7 08:23:49 2025 write: IOPS=4, BW=19.9MiB/s (20.8MB/s)(12.0MiB/604msec) clat (msec): min=127, max=263, avg=192.97, stdev=68.12 lat (msec): min=127, max=263, avg=193.27, stdev=68.24 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 128], 10.00th=[ 128], 20.00th=[ 128], | 30.00th=[ 128], 40.00th=[ 188], 50.00th=[ 188], 60.00th=[ 188], | 70.00th=[ 264], 80.00th=[ 264], 90.00th=[ 264], 95.00th=[ 264], | 99.00th=[ 264], 99.50th=[ 264], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min=16351, max=16351, per=36.48%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.33%, sys=48.76%, ctx=57, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=21.9MiB/s (22.0MB/s), 5603KiB/s-37.2MiB/s (5738kB/s-38.0MB/s), io=16.0MiB (16.8MB), run=323-731msec WRITE: bw=43.8MiB/s (45.9MB/s), 10.9MiB/s-19.9MiB/s (11.5MB/s-20.8MB/s), io=32.0MiB (33.6MB), run=604-731msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32283: Mon Apr 7 08:23:49 2025 read: IOPS=4, BW=16.5MiB/s (17.3MB/s)(12.0MiB/728msec) clat (msec): min=132, max=316, avg=232.50, stdev=93.16 lat (msec): min=132, max=316, avg=232.51, stdev=93.16 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 133], | 30.00th=[ 133], 40.00th=[ 249], 50.00th=[ 249], 60.00th=[ 249], | 70.00th=[ 317], 80.00th=[ 317], 90.00th=[ 317], 95.00th=[ 317], | 99.00th=[ 317], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 8192, max= 8192, per=42.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=14.86%, ctx=60, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32284: Mon Apr 7 08:23:49 2025 read: IOPS=1, BW=4859KiB/s (4975kB/s)(4096KiB/843msec) clat (nsec): min=285292k, max=285292k, avg=285292082.00, stdev= 0.00 lat (nsec): min=285294k, max=285294k, avg=285294301.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 288], 10.00th=[ 288], 20.00th=[ 288], | 30.00th=[ 288], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 288], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 7742, max= 7742, per=39.84%, avg=7742.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9718KiB/s (9951kB/s)(8192KiB/843msec) clat (msec): min=150, max=375, avg=262.98, stdev=159.20 lat (msec): min=150, max=375, avg=263.32, stdev=159.16 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 150], 10.00th=[ 150], 20.00th=[ 150], | 30.00th=[ 150], 40.00th=[ 150], 50.00th=[ 150], 60.00th=[ 376], | 70.00th=[ 376], 80.00th=[ 376], 90.00th=[ 376], 95.00th=[ 376], | 99.00th=[ 376], 99.50th=[ 376], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 7742, max= 7742, per=19.92%, avg=7742.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.12%, sys=13.78%, ctx=60, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32285: Mon Apr 7 08:23:49 2025 write: IOPS=3, BW=14.4MiB/s (15.1MB/s)(12.0MiB/836msec) clat (msec): min=39, max=458, avg=270.73, stdev=212.60 lat (msec): min=40, max=458, avg=270.96, stdev=212.56 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 313], 50.00th=[ 313], 60.00th=[ 313], | 70.00th=[ 460], 80.00th=[ 460], 90.00th=[ 460], 95.00th=[ 460], | 99.00th=[ 460], 99.50th=[ 460], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 7907, max= 7907, per=20.34%, avg=7907.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 500=66.67% cpu : usr=0.12%, sys=11.98%, ctx=164, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32286: Mon Apr 7 08:23:49 2025 write: IOPS=7, BW=29.7MiB/s (31.1MB/s)(12.0MiB/404msec) clat (msec): min=59, max=189, avg=125.19, stdev=65.18 lat (msec): min=59, max=189, avg=125.57, stdev=65.11 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 60], 10.00th=[ 60], 20.00th=[ 60], | 30.00th=[ 60], 40.00th=[ 127], 50.00th=[ 127], 60.00th=[ 127], | 70.00th=[ 190], 80.00th=[ 190], 90.00th=[ 190], 95.00th=[ 190], | 99.00th=[ 190], 99.50th=[ 190], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=19.35%, ctx=47, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.0MiB/s (19.9MB/s), 4859KiB/s-16.5MiB/s (4975kB/s-17.3MB/s), io=16.0MiB (16.8MB), run=728-843msec WRITE: bw=37.0MiB/s (39.8MB/s), 9718KiB/s-29.7MiB/s (9951kB/s-31.1MB/s), io=32.0MiB (33.6MB), run=404-843msec