== sanity test 398b: DIO and buffer IO race ============== 09:54:09 (1741701249) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.47108 s, 34.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29569: Tue Mar 11 09:54:27 2025 read: IOPS=98, BW=395KiB/s (405kB/s)(5992KiB/15153msec) clat (usec): min=56, max=36371, avg=5547.37, stdev=3481.35 lat (usec): min=57, max=36371, avg=5549.28, stdev=3482.58 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 141], 10.00th=[ 2507], 20.00th=[ 3130], | 30.00th=[ 3654], 40.00th=[ 4178], 50.00th=[ 4817], 60.00th=[ 5669], | 70.00th=[ 6587], 80.00th=[ 7570], 90.00th=[ 9503], 95.00th=[11469], | 99.00th=[18220], 99.50th=[22152], 99.90th=[29230], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 160, max= 536, per=25.02%, avg=392.00, stdev=75.77, samples=30 iops : min= 40, max= 134, avg=97.80, stdev=18.93, samples=30 write: IOPS=103, BW=415KiB/s (425kB/s)(6296KiB/15153msec) clat (usec): min=416, max=209781, avg=4291.76, stdev=6462.60 lat (usec): min=416, max=209783, avg=4293.14, stdev=6462.86 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 594], 10.00th=[ 676], 20.00th=[ 766], | 30.00th=[ 906], 40.00th=[ 2024], 50.00th=[ 3326], 60.00th=[ 4228], | 70.00th=[ 5342], 80.00th=[ 7111], 90.00th=[ 9503], 95.00th=[ 11731], | 99.00th=[ 17433], 99.50th=[ 19530], 99.90th=[ 27132], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 176, max= 584, per=25.03%, avg=410.67, stdev=79.65, samples=30 iops : min= 44, max= 146, avg=102.47, stdev=19.97, samples=30 lat (usec) : 100=1.37%, 250=1.20%, 500=0.81%, 750=8.63%, 1000=7.49% lat (msec) : 2=4.13%, 4=23.83%, 10=44.47%, 20=7.52%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=20.98%, ctx=5658, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29570: Tue Mar 11 09:54:27 2025 read: IOPS=97, BW=389KiB/s (398kB/s)(5940KiB/15267msec) clat (usec): min=45, max=31041, avg=5595.83, stdev=3512.87 lat (usec): min=45, max=31042, avg=5597.36, stdev=3512.71 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 1876], 10.00th=[ 2507], 20.00th=[ 3032], | 30.00th=[ 3589], 40.00th=[ 4113], 50.00th=[ 4752], 60.00th=[ 5604], | 70.00th=[ 6587], 80.00th=[ 7898], 90.00th=[ 9765], 95.00th=[11731], | 99.00th=[18220], 99.50th=[20841], 99.90th=[26084], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 176, max= 544, per=24.60%, avg=385.53, stdev=77.97, samples=30 iops : min= 44, max= 136, avg=96.33, stdev=19.53, samples=30 write: IOPS=103, BW=416KiB/s (426kB/s)(6348KiB/15267msec) clat (usec): min=439, max=216261, avg=4331.90, stdev=6638.92 lat (usec): min=440, max=216262, avg=4333.42, stdev=6639.40 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 594], 10.00th=[ 668], 20.00th=[ 766], | 30.00th=[ 938], 40.00th=[ 1926], 50.00th=[ 3359], 60.00th=[ 4146], | 70.00th=[ 5473], 80.00th=[ 7046], 90.00th=[ 9503], 95.00th=[ 11994], | 99.00th=[ 17171], 99.50th=[ 21365], 99.90th=[ 30540], 99.95th=[217056], | 99.99th=[217056] bw ( KiB/s): min= 224, max= 528, per=24.79%, avg=406.87, stdev=84.44, samples=30 iops : min= 56, max= 132, avg=101.67, stdev=21.15, samples=30 lat (usec) : 50=0.07%, 100=1.40%, 250=0.65%, 500=0.68%, 750=8.92% lat (usec) : 1000=6.67% lat (msec) : 2=5.05%, 4=25.46%, 10=42.19%, 20=8.30%, 50=0.59% lat (msec) : 250=0.03% cpu : usr=0.28%, sys=21.22%, ctx=5594, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29571: Tue Mar 11 09:54:27 2025 read: IOPS=98, BW=394KiB/s (403kB/s)(6028KiB/15316msec) clat (usec): min=57, max=219695, avg=5731.16, stdev=6482.57 lat (usec): min=58, max=219695, avg=5733.02, stdev=6483.29 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 122], 10.00th=[ 2573], 20.00th=[ 3130], | 30.00th=[ 3654], 40.00th=[ 4178], 50.00th=[ 4948], 60.00th=[ 5866], | 70.00th=[ 6652], 80.00th=[ 7701], 90.00th=[ 9634], 95.00th=[ 11338], | 99.00th=[ 17433], 99.50th=[ 22676], 99.90th=[ 25035], 99.95th=[219153], | 99.99th=[219153] bw ( KiB/s): min= 168, max= 552, per=24.91%, avg=390.33, stdev=85.54, samples=30 iops : min= 42, max= 138, avg=97.53, stdev=21.39, samples=30 write: IOPS=102, BW=409KiB/s (419kB/s)(6260KiB/15316msec) clat (usec): min=373, max=29453, avg=4216.61, stdev=4029.54 lat (usec): min=373, max=29454, avg=4217.62, stdev=4029.51 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 603], 10.00th=[ 685], 20.00th=[ 766], | 30.00th=[ 922], 40.00th=[ 1680], 50.00th=[ 3261], 60.00th=[ 4228], | 70.00th=[ 5473], 80.00th=[ 7177], 90.00th=[ 9765], 95.00th=[12256], | 99.00th=[16909], 99.50th=[20317], 99.90th=[27657], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 208, max= 512, per=24.58%, avg=403.40, stdev=69.17, samples=30 iops : min= 52, max= 128, avg=100.80, stdev=17.35, samples=30 lat (usec) : 100=1.37%, 250=1.30%, 500=0.68%, 750=8.63%, 1000=7.32% lat (msec) : 2=4.62%, 4=24.32%, 10=42.87%, 20=8.27%, 50=0.59% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=20.73%, ctx=5615, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29572: Tue Mar 11 09:54:27 2025 read: IOPS=99, BW=397KiB/s (406kB/s)(6052KiB/15259msec) clat (usec): min=56, max=34715, avg=5482.38, stdev=3323.64 lat (usec): min=57, max=34716, avg=5483.28, stdev=3323.68 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 210], 10.00th=[ 2474], 20.00th=[ 3097], | 30.00th=[ 3589], 40.00th=[ 4113], 50.00th=[ 4817], 60.00th=[ 5604], | 70.00th=[ 6587], 80.00th=[ 7701], 90.00th=[ 9372], 95.00th=[11600], | 99.00th=[15926], 99.50th=[18482], 99.90th=[29754], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 216, max= 544, per=25.06%, avg=392.77, stdev=75.53, samples=30 iops : min= 54, max= 136, avg=98.17, stdev=18.90, samples=30 write: IOPS=102, BW=409KiB/s (418kB/s)(6236KiB/15259msec) clat (usec): min=419, max=216620, avg=4415.36, stdev=6705.73 lat (usec): min=419, max=216621, avg=4416.56, stdev=6705.72 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 594], 10.00th=[ 676], 20.00th=[ 775], | 30.00th=[ 938], 40.00th=[ 1926], 50.00th=[ 3490], 60.00th=[ 4359], | 70.00th=[ 5538], 80.00th=[ 7308], 90.00th=[ 9765], 95.00th=[ 12125], | 99.00th=[ 16712], 99.50th=[ 20317], 99.90th=[ 35390], 99.95th=[217056], | 99.99th=[217056] bw ( KiB/s): min= 240, max= 584, per=24.67%, avg=404.77, stdev=90.16, samples=30 iops : min= 60, max= 146, avg=101.17, stdev=22.55, samples=30 lat (usec) : 100=1.17%, 250=1.33%, 500=0.94%, 750=8.37%, 1000=7.13% lat (msec) : 2=4.62%, 4=23.96%, 10=43.91%, 20=8.07%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.30%, sys=20.99%, ctx=5704, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1568KiB/s (1605kB/s), 389KiB/s-397KiB/s (398kB/s-406kB/s), io=23.4MiB (24.6MB), run=15153-15316msec WRITE: bw=1641KiB/s (1681kB/s), 409KiB/s-416KiB/s (418kB/s-426kB/s), io=24.6MiB (25.7MB), run=15153-15316msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29573: Tue Mar 11 09:54:31 2025 read: IOPS=77, BW=312KiB/s (319kB/s)(5992KiB/19223msec) clat (usec): min=1511, max=223730, avg=5075.49, stdev=7187.31 lat (usec): min=1511, max=223732, avg=5076.44, stdev=7187.32 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1827], 10.00th=[ 1958], 20.00th=[ 2114], | 30.00th=[ 2376], 40.00th=[ 2671], 50.00th=[ 3195], 60.00th=[ 3851], | 70.00th=[ 4752], 80.00th=[ 6587], 90.00th=[ 11076], 95.00th=[ 14746], | 99.00th=[ 24249], 99.50th=[ 27132], 99.90th=[ 35390], 99.95th=[223347], | 99.99th=[223347] bw ( KiB/s): min= 119, max= 776, per=24.74%, avg=307.76, stdev=187.65, samples=38 iops : min= 29, max= 194, avg=76.79, stdev=46.96, samples=38 write: IOPS=81, BW=328KiB/s (335kB/s)(6296KiB/19223msec) clat (usec): min=2127, max=45426, avg=7342.62, stdev=5213.56 lat (usec): min=2127, max=45427, avg=7343.57, stdev=5213.64 clat percentiles (usec): | 1.00th=[ 2474], 5.00th=[ 2769], 10.00th=[ 2999], 20.00th=[ 3359], | 30.00th=[ 3752], 40.00th=[ 4293], 50.00th=[ 5211], 60.00th=[ 6783], | 70.00th=[ 8717], 80.00th=[11338], 90.00th=[14222], 95.00th=[17171], | 99.00th=[26084], 99.50th=[30016], 99.90th=[38536], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 136, max= 758, per=24.80%, avg=322.92, stdev=189.23, samples=38 iops : min= 34, max= 189, avg=80.58, stdev=47.35, samples=38 lat (msec) : 2=6.35%, 4=41.96%, 10=33.17%, 20=16.11%, 50=2.38% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=10.67%, ctx=3646, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29574: Tue Mar 11 09:54:31 2025 read: IOPS=76, BW=308KiB/s (315kB/s)(5940KiB/19295msec) clat (usec): min=1361, max=221052, avg=5118.89, stdev=7233.13 lat (usec): min=1362, max=221053, avg=5119.80, stdev=7233.14 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1811], 10.00th=[ 1926], 20.00th=[ 2089], | 30.00th=[ 2278], 40.00th=[ 2606], 50.00th=[ 3064], 60.00th=[ 3720], | 70.00th=[ 4752], 80.00th=[ 6849], 90.00th=[ 11469], 95.00th=[ 15270], | 99.00th=[ 22414], 99.50th=[ 24773], 99.90th=[ 36439], 99.95th=[221250], | 99.99th=[221250] bw ( KiB/s): min= 96, max= 752, per=24.42%, avg=303.79, stdev=194.36, samples=38 iops : min= 24, max= 188, avg=75.82, stdev=48.63, samples=38 write: IOPS=82, BW=329KiB/s (337kB/s)(6348KiB/19295msec) clat (usec): min=2111, max=46134, avg=7332.58, stdev=5248.53 lat (usec): min=2112, max=46135, avg=7333.58, stdev=5248.64 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2769], 10.00th=[ 3032], 20.00th=[ 3392], | 30.00th=[ 3752], 40.00th=[ 4228], 50.00th=[ 5211], 60.00th=[ 6652], | 70.00th=[ 8717], 80.00th=[11338], 90.00th=[14091], 95.00th=[17171], | 99.00th=[27657], 99.50th=[29492], 99.90th=[39060], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 143, max= 752, per=24.66%, avg=321.05, stdev=187.24, samples=38 iops : min= 35, max= 188, avg=80.13, stdev=46.85, samples=38 lat (msec) : 2=6.64%, 4=42.32%, 10=32.29%, 20=16.37%, 50=2.34% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=10.53%, ctx=3652, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29575: Tue Mar 11 09:54:31 2025 read: IOPS=78, BW=313KiB/s (321kB/s)(6028KiB/19241msec) clat (usec): min=1440, max=35535, avg=5037.18, stdev=4468.06 lat (usec): min=1440, max=35536, avg=5037.96, stdev=4468.15 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1827], 10.00th=[ 1942], 20.00th=[ 2114], | 30.00th=[ 2376], 40.00th=[ 2704], 50.00th=[ 3195], 60.00th=[ 3884], | 70.00th=[ 5080], 80.00th=[ 7046], 90.00th=[11469], 95.00th=[14484], | 99.00th=[21365], 99.50th=[24249], 99.90th=[33424], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 120, max= 704, per=24.85%, avg=309.18, stdev=178.39, samples=38 iops : min= 30, max= 176, avg=77.24, stdev=44.64, samples=38 write: IOPS=81, BW=325KiB/s (333kB/s)(6260KiB/19241msec) clat (msec): min=2, max=202, avg= 7.39, stdev= 7.04 lat (msec): min=2, max=202, avg= 7.39, stdev= 7.04 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 12], 90.00th=[ 14], 95.00th=[ 18], | 99.00th=[ 27], 99.50th=[ 29], 99.90th=[ 36], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 104, max= 816, per=24.62%, avg=320.55, stdev=192.80, samples=38 iops : min= 26, max= 204, avg=80.08, stdev=48.23, samples=38 lat (msec) : 2=6.32%, 4=41.73%, 10=33.24%, 20=16.44%, 50=2.25% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=10.62%, ctx=3638, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29576: Tue Mar 11 09:54:31 2025 read: IOPS=78, BW=314KiB/s (322kB/s)(6052KiB/19257msec) clat (usec): min=1412, max=35297, avg=4984.43, stdev=4401.68 lat (usec): min=1412, max=35298, avg=4985.24, stdev=4401.70 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1844], 10.00th=[ 1942], 20.00th=[ 2114], | 30.00th=[ 2311], 40.00th=[ 2638], 50.00th=[ 3130], 60.00th=[ 3785], | 70.00th=[ 4948], 80.00th=[ 6915], 90.00th=[11469], 95.00th=[14222], | 99.00th=[21103], 99.50th=[23987], 99.90th=[33817], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 104, max= 776, per=24.86%, avg=309.21, stdev=183.25, samples=38 iops : min= 26, max= 194, avg=77.13, stdev=45.85, samples=38 write: IOPS=80, BW=324KiB/s (332kB/s)(6236KiB/19257msec) clat (usec): min=1994, max=198390, avg=7463.47, stdev=7050.94 lat (usec): min=1994, max=198392, avg=7464.45, stdev=7051.00 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 12], 90.00th=[ 15], 95.00th=[ 18], | 99.00th=[ 27], 99.50th=[ 31], 99.90th=[ 47], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 159, max= 736, per=24.48%, avg=318.68, stdev=185.25, samples=38 iops : min= 39, max= 184, avg=79.50, stdev=46.36, samples=38 lat (msec) : 2=6.90%, 4=41.15%, 10=32.88%, 20=17.12%, 50=1.92% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=10.63%, ctx=3617, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1244KiB/s (1274kB/s), 308KiB/s-314KiB/s (315kB/s-322kB/s), io=23.4MiB (24.6MB), run=19223-19295msec WRITE: bw=1303KiB/s (1334kB/s), 324KiB/s-329KiB/s (332kB/s-337kB/s), io=24.6MiB (25.7MB), run=19223-19295msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29611: Tue Mar 11 09:54:36 2025 read: IOPS=104, BW=1667KiB/s (1707kB/s)(5920KiB/3551msec) clat (usec): min=61, max=22671, avg=5032.01, stdev=2870.66 lat (usec): min=61, max=22672, avg=5032.68, stdev=2870.66 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 125], 10.00th=[ 2442], 20.00th=[ 3097], | 30.00th=[ 3458], 40.00th=[ 4015], 50.00th=[ 4621], 60.00th=[ 5276], | 70.00th=[ 6128], 80.00th=[ 6849], 90.00th=[ 7635], 95.00th=[ 9896], | 99.00th=[16057], 99.50th=[17171], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 1312, max= 1888, per=24.82%, avg=1664.00, stdev=226.27, samples=7 iops : min= 82, max= 118, avg=104.00, stdev=14.14, samples=7 write: IOPS=112, BW=1793KiB/s (1836kB/s)(6368KiB/3551msec) clat (usec): min=637, max=18623, avg=4171.16, stdev=3427.48 lat (usec): min=638, max=18624, avg=4172.13, stdev=3427.49 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 783], 10.00th=[ 906], 20.00th=[ 1057], | 30.00th=[ 1303], 40.00th=[ 2114], 50.00th=[ 3359], 60.00th=[ 4293], | 70.00th=[ 5538], 80.00th=[ 7177], 90.00th=[ 8717], 95.00th=[10290], | 99.00th=[16909], 99.50th=[18220], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 1568, max= 2112, per=25.47%, avg=1796.57, stdev=187.89, samples=7 iops : min= 98, max= 132, avg=112.29, stdev=11.74, samples=7 lat (usec) : 100=1.43%, 250=1.56%, 750=2.08%, 1000=6.64% lat (msec) : 2=11.72%, 4=25.13%, 10=46.22%, 20=5.08%, 50=0.13% cpu : usr=0.08%, sys=24.48%, ctx=1629, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29612: Tue Mar 11 09:54:36 2025 read: IOPS=100, BW=1608KiB/s (1647kB/s)(5744KiB/3572msec) clat (usec): min=48, max=19302, avg=5121.13, stdev=3168.20 lat (usec): min=48, max=19303, avg=5123.09, stdev=3167.16 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 86], 10.00th=[ 112], 20.00th=[ 2835], | 30.00th=[ 3523], 40.00th=[ 4228], 50.00th=[ 4948], 60.00th=[ 5669], | 70.00th=[ 6390], 80.00th=[ 7308], 90.00th=[ 9372], 95.00th=[10421], | 99.00th=[15664], 99.50th=[16188], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 1408, max= 1980, per=23.30%, avg=1562.00, stdev=211.08, samples=6 iops : min= 88, max= 123, avg=97.50, stdev=12.90, samples=6 write: IOPS=114, BW=1832KiB/s (1876kB/s)(6544KiB/3572msec) clat (usec): min=585, max=17980, avg=4122.11, stdev=3359.67 lat (usec): min=585, max=17981, avg=4123.23, stdev=3359.58 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 750], 10.00th=[ 857], 20.00th=[ 1074], | 30.00th=[ 1467], 40.00th=[ 2474], 50.00th=[ 3326], 60.00th=[ 4228], | 70.00th=[ 5145], 80.00th=[ 6587], 90.00th=[ 8979], 95.00th=[10814], | 99.00th=[14877], 99.50th=[15926], 99.90th=[17957], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 1536, max= 1888, per=23.96%, avg=1690.00, stdev=145.10, samples=6 iops : min= 96, max= 118, avg=105.50, stdev= 9.07, samples=6 lat (usec) : 50=0.13%, 100=4.04%, 250=1.43%, 750=2.73%, 1000=5.73% lat (msec) : 2=10.68%, 4=22.79%, 10=45.83%, 20=6.64% cpu : usr=0.08%, sys=23.83%, ctx=1590, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29613: Tue Mar 11 09:54:36 2025 read: IOPS=111, BW=1782KiB/s (1825kB/s)(6256KiB/3511msec) clat (usec): min=51, max=20768, avg=4989.54, stdev=3168.10 lat (usec): min=52, max=20769, avg=4991.68, stdev=3170.27 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 98], 10.00th=[ 215], 20.00th=[ 2900], | 30.00th=[ 3425], 40.00th=[ 4047], 50.00th=[ 4752], 60.00th=[ 5211], | 70.00th=[ 5866], 80.00th=[ 6587], 90.00th=[ 8848], 95.00th=[10945], | 99.00th=[15533], 99.50th=[20317], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 1248, max= 2048, per=25.77%, avg=1728.00, stdev=307.60, samples=6 iops : min= 78, max= 128, avg=108.00, stdev=19.22, samples=6 write: IOPS=107, BW=1718KiB/s (1759kB/s)(6032KiB/3511msec) clat (usec): min=599, max=17749, avg=4109.94, stdev=3359.46 lat (usec): min=600, max=17750, avg=4110.93, stdev=3359.45 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 775], 10.00th=[ 873], 20.00th=[ 1057], | 30.00th=[ 1270], 40.00th=[ 2311], 50.00th=[ 3261], 60.00th=[ 4293], | 70.00th=[ 5407], 80.00th=[ 6718], 90.00th=[ 8848], 95.00th=[11338], | 99.00th=[13173], 99.50th=[16909], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 1344, max= 2208, per=24.35%, avg=1717.33, stdev=335.42, samples=6 iops : min= 84, max= 138, avg=107.33, stdev=20.96, samples=6 lat (usec) : 100=3.12%, 250=2.08%, 750=1.82%, 1000=6.12% lat (msec) : 2=10.81%, 4=23.96%, 10=45.05%, 20=6.77%, 50=0.26% cpu : usr=0.20%, sys=24.39%, ctx=1599, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29614: Tue Mar 11 09:54:36 2025 read: IOPS=107, BW=1727KiB/s (1769kB/s)(6032KiB/3492msec) clat (usec): min=59, max=21219, avg=4952.52, stdev=3297.83 lat (usec): min=59, max=21220, avg=4953.17, stdev=3297.80 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 87], 10.00th=[ 126], 20.00th=[ 2737], | 30.00th=[ 3294], 40.00th=[ 3851], 50.00th=[ 4490], 60.00th=[ 5342], | 70.00th=[ 5866], 80.00th=[ 6718], 90.00th=[ 8717], 95.00th=[10945], | 99.00th=[16909], 99.50th=[18482], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 1408, max= 1952, per=24.82%, avg=1664.00, stdev=223.54, samples=6 iops : min= 88, max= 122, avg=104.00, stdev=13.97, samples=6 write: IOPS=111, BW=1792KiB/s (1835kB/s)(6256KiB/3492msec) clat (usec): min=594, max=18214, avg=4023.08, stdev=3162.57 lat (usec): min=595, max=18215, avg=4024.03, stdev=3162.59 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 807], 10.00th=[ 898], 20.00th=[ 1090], | 30.00th=[ 1434], 40.00th=[ 2343], 50.00th=[ 3326], 60.00th=[ 4015], | 70.00th=[ 5145], 80.00th=[ 6390], 90.00th=[ 9110], 95.00th=[10290], | 99.00th=[12518], 99.50th=[14353], 99.90th=[18220], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 1632, max= 1920, per=24.95%, avg=1760.00, stdev=107.09, samples=6 iops : min= 102, max= 120, avg=110.00, stdev= 6.69, samples=6 lat (usec) : 100=3.52%, 250=1.82%, 750=0.91%, 1000=6.90% lat (msec) : 2=10.68%, 4=27.86%, 10=41.93%, 20=6.25%, 50=0.13% cpu : usr=0.09%, sys=24.12%, ctx=1610, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6705KiB/s (6866kB/s), 1608KiB/s-1782KiB/s (1647kB/s-1825kB/s), io=23.4MiB (24.5MB), run=3492-3572msec WRITE: bw=7055KiB/s (7224kB/s), 1718KiB/s-1832KiB/s (1759kB/s-1876kB/s), io=24.6MiB (25.8MB), run=3492-3572msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29607: Tue Mar 11 09:54:37 2025 read: IOPS=84, BW=1353KiB/s (1385kB/s)(5920KiB/4377msec) clat (usec): min=1558, max=20600, avg=4735.30, stdev=3697.51 lat (usec): min=1559, max=20600, avg=4735.95, stdev=3697.55 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1811], 10.00th=[ 1958], 20.00th=[ 2212], | 30.00th=[ 2442], 40.00th=[ 2835], 50.00th=[ 3261], 60.00th=[ 3949], | 70.00th=[ 4686], 80.00th=[ 6587], 90.00th=[10552], 95.00th=[13435], | 99.00th=[18220], 99.50th=[20055], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 864, max= 2304, per=22.40%, avg=1203.75, stdev=468.58, samples=8 iops : min= 54, max= 144, avg=75.12, stdev=29.36, samples=8 write: IOPS=90, BW=1455KiB/s (1490kB/s)(6368KiB/4377msec) clat (usec): min=2141, max=23735, avg=6530.51, stdev=3955.55 lat (usec): min=2142, max=23736, avg=6531.78, stdev=3955.65 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2802], 10.00th=[ 2933], 20.00th=[ 3294], | 30.00th=[ 3687], 40.00th=[ 4424], 50.00th=[ 5145], 60.00th=[ 6390], | 70.00th=[ 7504], 80.00th=[ 9372], 90.00th=[12125], 95.00th=[14484], | 99.00th=[20055], 99.50th=[23725], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 896, max= 2464, per=23.34%, avg=1319.62, stdev=483.97, samples=8 iops : min= 56, max= 154, avg=82.38, stdev=30.24, samples=8 lat (msec) : 2=5.60%, 4=41.93%, 10=38.54%, 20=13.28%, 50=0.65% cpu : usr=0.25%, sys=11.81%, ctx=914, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29608: Tue Mar 11 09:54:37 2025 read: IOPS=80, BW=1289KiB/s (1320kB/s)(5744KiB/4456msec) clat (usec): min=1594, max=24056, avg=4853.31, stdev=3732.10 lat (usec): min=1594, max=24056, avg=4853.96, stdev=3732.12 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1860], 10.00th=[ 1991], 20.00th=[ 2212], | 30.00th=[ 2474], 40.00th=[ 2769], 50.00th=[ 3228], 60.00th=[ 3982], | 70.00th=[ 5342], 80.00th=[ 7111], 90.00th=[10683], 95.00th=[12911], | 99.00th=[18744], 99.50th=[22152], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 704, max= 2496, per=21.88%, avg=1176.00, stdev=565.17, samples=8 iops : min= 44, max= 156, avg=73.50, stdev=35.32, samples=8 write: IOPS=91, BW=1469KiB/s (1504kB/s)(6544KiB/4456msec) clat (usec): min=2168, max=23696, avg=6607.07, stdev=4214.27 lat (usec): min=2169, max=23698, avg=6608.30, stdev=4214.42 clat percentiles (usec): | 1.00th=[ 2343], 5.00th=[ 2737], 10.00th=[ 2900], 20.00th=[ 3261], | 30.00th=[ 3654], 40.00th=[ 4178], 50.00th=[ 5014], 60.00th=[ 5997], | 70.00th=[ 7701], 80.00th=[ 9765], 90.00th=[12911], 95.00th=[15664], | 99.00th=[20055], 99.50th=[22152], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 992, max= 2304, per=22.42%, avg=1268.00, stdev=431.00, samples=8 iops : min= 62, max= 144, avg=79.25, stdev=26.94, samples=8 lat (msec) : 2=4.69%, 4=43.49%, 10=36.72%, 20=14.19%, 50=0.91% cpu : usr=0.13%, sys=11.60%, ctx=900, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29609: Tue Mar 11 09:54:37 2025 read: IOPS=89, BW=1432KiB/s (1466kB/s)(6256KiB/4369msec) clat (usec): min=1660, max=19579, avg=4857.55, stdev=3803.80 lat (usec): min=1660, max=19580, avg=4858.28, stdev=3803.81 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1926], 10.00th=[ 2024], 20.00th=[ 2245], | 30.00th=[ 2474], 40.00th=[ 2802], 50.00th=[ 3228], 60.00th=[ 3818], | 70.00th=[ 4817], 80.00th=[ 6652], 90.00th=[10683], 95.00th=[14222], | 99.00th=[16581], 99.50th=[19268], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 960, max= 2272, per=23.67%, avg=1272.00, stdev=429.24, samples=8 iops : min= 60, max= 142, avg=79.50, stdev=26.83, samples=8 write: IOPS=86, BW=1381KiB/s (1414kB/s)(6032KiB/4369msec) clat (usec): min=2192, max=24463, avg=6503.12, stdev=3987.07 lat (usec): min=2193, max=24465, avg=6504.52, stdev=3987.02 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2835], 10.00th=[ 3097], 20.00th=[ 3326], | 30.00th=[ 3687], 40.00th=[ 4228], 50.00th=[ 5080], 60.00th=[ 6128], | 70.00th=[ 7767], 80.00th=[ 9503], 90.00th=[11994], 95.00th=[14222], | 99.00th=[22152], 99.50th=[23725], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 800, max= 2496, per=22.42%, avg=1268.00, stdev=534.35, samples=8 iops : min= 50, max= 156, avg=79.25, stdev=33.40, samples=8 lat (msec) : 2=4.43%, 4=44.14%, 10=36.20%, 20=14.58%, 50=0.65% cpu : usr=0.16%, sys=12.02%, ctx=929, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29610: Tue Mar 11 09:54:37 2025 read: IOPS=84, BW=1355KiB/s (1387kB/s)(6032KiB/4453msec) clat (usec): min=1640, max=22638, avg=4744.00, stdev=3797.20 lat (usec): min=1641, max=22638, avg=4744.65, stdev=3797.22 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1811], 10.00th=[ 1975], 20.00th=[ 2147], | 30.00th=[ 2474], 40.00th=[ 2769], 50.00th=[ 3195], 60.00th=[ 3884], | 70.00th=[ 4752], 80.00th=[ 6718], 90.00th=[ 9896], 95.00th=[13304], | 99.00th=[21365], 99.50th=[22152], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 896, max= 2144, per=21.73%, avg=1168.00, stdev=406.57, samples=8 iops : min= 56, max= 134, avg=73.00, stdev=25.41, samples=8 write: IOPS=87, BW=1405KiB/s (1439kB/s)(6256KiB/4453msec) clat (usec): min=2247, max=27034, avg=6739.86, stdev=4554.02 lat (usec): min=2248, max=27035, avg=6741.11, stdev=4554.05 clat percentiles (usec): | 1.00th=[ 2507], 5.00th=[ 2802], 10.00th=[ 2999], 20.00th=[ 3294], | 30.00th=[ 3785], 40.00th=[ 4228], 50.00th=[ 4883], 60.00th=[ 6063], | 70.00th=[ 7570], 80.00th=[ 9765], 90.00th=[13042], 95.00th=[16909], | 99.00th=[23462], 99.50th=[24249], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 960, max= 2528, per=22.28%, avg=1260.00, stdev=531.33, samples=8 iops : min= 60, max= 158, avg=78.75, stdev=33.21, samples=8 lat (msec) : 2=5.60%, 4=43.23%, 10=36.85%, 20=12.63%, 50=1.69% cpu : usr=0.16%, sys=11.66%, ctx=952, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5375KiB/s (5504kB/s), 1289KiB/s-1432KiB/s (1320kB/s-1466kB/s), io=23.4MiB (24.5MB), run=4369-4456msec WRITE: bw=5655KiB/s (5791kB/s), 1381KiB/s-1469KiB/s (1414kB/s-1504kB/s), io=24.6MiB (25.8MB), run=4369-4456msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29623: Tue Mar 11 09:54:39 2025 read: IOPS=12, BW=12.8MiB/s (13.4MB/s)(7168KiB/546msec) clat (usec): min=557, max=51376, avg=32577.24, stdev=16329.57 lat (usec): min=558, max=51377, avg=32578.53, stdev=16329.48 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 562], 10.00th=[ 562], 20.00th=[29230], | 30.00th=[31065], 40.00th=[31065], 50.00th=[31851], 60.00th=[38536], | 70.00th=[38536], 80.00th=[45876], 90.00th=[51119], 95.00th=[51119], | 99.00th=[51119], 99.50th=[51119], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min=12263, max=12263, per=73.41%, avg=12263.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9377KiB/s (9602kB/s)(5120KiB/546msec) clat (usec): min=30416, max=96856, avg=61198.05, stdev=30297.73 lat (usec): min=30441, max=96921, avg=61249.16, stdev=30300.79 clat percentiles (usec): | 1.00th=[30540], 5.00th=[30540], 10.00th=[30540], 20.00th=[30540], | 30.00th=[30802], 40.00th=[30802], 50.00th=[63701], 60.00th=[63701], | 70.00th=[84411], 80.00th=[84411], 90.00th=[96994], 95.00th=[96994], | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 8175, max= 8175, per=34.96%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=58.33%, 100=33.33% cpu : usr=0.00%, sys=41.83%, ctx=543, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29624: Tue Mar 11 09:54:39 2025 read: IOPS=2, BW=2790KiB/s (2857kB/s)(2048KiB/734msec) clat (usec): min=29316, max=40519, avg=34918.12, stdev=7922.04 lat (usec): min=29317, max=40521, avg=34919.56, stdev=7922.29 clat percentiles (usec): | 1.00th=[29230], 5.00th=[29230], 10.00th=[29230], 20.00th=[29230], | 30.00th=[29230], 40.00th=[29230], 50.00th=[29230], 60.00th=[40633], | 70.00th=[40633], 80.00th=[40633], 90.00th=[40633], 95.00th=[40633], | 99.00th=[40633], 99.50th=[40633], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 2048, max= 2048, per=12.26%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.6MiB/s (14.3MB/s)(10.0MiB/734msec) clat (msec): min=19, max=131, avg=57.50, stdev=31.35 lat (msec): min=19, max=131, avg=57.56, stdev=31.35 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 31], | 30.00th=[ 40], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 55], | 70.00th=[ 64], 80.00th=[ 64], 90.00th=[ 81], 95.00th=[ 132], | 99.00th=[ 132], 99.50th=[ 132], 99.90th=[ 132], 99.95th=[ 132], | 99.99th=[ 132] bw ( KiB/s): min=10240, max=10240, per=43.79%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=33.33%, 250=8.33% cpu : usr=0.14%, sys=39.65%, ctx=78, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29625: Tue Mar 11 09:54:39 2025 read: IOPS=11, BW=11.5MiB/s (12.0MB/s)(6144KiB/524msec) clat (usec): min=493, max=51100, avg=30711.34, stdev=19602.67 lat (usec): min=493, max=51102, avg=30712.57, stdev=19602.67 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 494], 10.00th=[ 494], 20.00th=[13042], | 30.00th=[13042], 40.00th=[36963], 50.00th=[36963], 60.00th=[38536], | 70.00th=[44303], 80.00th=[44303], 90.00th=[51119], 95.00th=[51119], | 99.00th=[51119], 99.50th=[51119], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min=10240, max=10240, per=61.30%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.5MiB/s (12.0MB/s)(6144KiB/524msec) clat (usec): min=28268, max=90320, avg=53679.81, stdev=25901.15 lat (usec): min=28329, max=90330, avg=53728.82, stdev=25887.09 clat percentiles (usec): | 1.00th=[28181], 5.00th=[28181], 10.00th=[28181], 20.00th=[35390], | 30.00th=[35390], 40.00th=[36963], 50.00th=[36963], 60.00th=[50070], | 70.00th=[81265], 80.00th=[81265], 90.00th=[90702], 95.00th=[90702], | 99.00th=[90702], 99.50th=[90702], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min=12288, max=12288, per=52.54%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 20=8.33%, 50=50.00%, 100=33.33% cpu : usr=0.00%, sys=41.30%, ctx=441, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29626: Tue Mar 11 09:54:39 2025 read: IOPS=4, BW=4176KiB/s (4276kB/s)(5120KiB/1226msec) clat (msec): min=29, max=565, avg=153.18, stdev=232.43 lat (msec): min=29, max=565, avg=153.18, stdev=232.43 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 30], | 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 35], 60.00th=[ 35], | 70.00th=[ 105], 80.00th=[ 105], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 2048, max= 2048, per=12.26%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5847KiB/s (5987kB/s)(7168KiB/1226msec) clat (usec): min=32294, max=97605, avg=56659.97, stdev=28090.48 lat (usec): min=32378, max=97675, avg=56729.32, stdev=28096.68 clat percentiles (usec): | 1.00th=[32375], 5.00th=[32375], 10.00th=[32375], 20.00th=[34341], | 30.00th=[38011], 40.00th=[38011], 50.00th=[43779], 60.00th=[55313], | 70.00th=[55313], 80.00th=[94897], 90.00th=[98042], 95.00th=[98042], | 99.00th=[98042], 99.50th=[98042], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 8192, max= 8192, per=35.03%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=25.00%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=28.08%, ctx=1058, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.3MiB/s (17.1MB/s), 2790KiB/s-12.8MiB/s (2857kB/s-13.4MB/s), io=20.0MiB (20.0MB), run=524-1226msec WRITE: bw=22.8MiB/s (23.9MB/s), 5847KiB/s-13.6MiB/s (5987kB/s-14.3MB/s), io=28.0MiB (29.4MB), run=524-1226msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29627: Tue Mar 11 09:54:39 2025 read: IOPS=10, BW=10.4MiB/s (10.9MB/s)(7168KiB/676msec) clat (usec): min=20716, max=49518, avg=31755.97, stdev=10712.46 lat (usec): min=20718, max=49520, avg=31757.32, stdev=10712.52 clat percentiles (usec): | 1.00th=[20841], 5.00th=[20841], 10.00th=[20841], 20.00th=[21103], | 30.00th=[26870], 40.00th=[26870], 50.00th=[27919], 60.00th=[34866], | 70.00th=[34866], 80.00th=[41157], 90.00th=[49546], 95.00th=[49546], | 99.00th=[49546], 99.50th=[49546], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min=10178, max=10178, per=62.77%, avg=10178.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7574KiB/s (7756kB/s)(5120KiB/676msec) clat (msec): min=62, max=113, avg=89.06, stdev=22.13 lat (msec): min=62, max=113, avg=89.13, stdev=22.11 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 63], | 30.00th=[ 72], 40.00th=[ 72], 50.00th=[ 90], 60.00th=[ 90], | 70.00th=[ 108], 80.00th=[ 108], 90.00th=[ 114], 95.00th=[ 114], | 99.00th=[ 114], 99.50th=[ 114], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 6107, max= 6107, per=26.90%, avg=6107.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=25.00%, 250=16.67% cpu : usr=0.00%, sys=18.67%, ctx=77, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29628: Tue Mar 11 09:54:39 2025 read: IOPS=2, BW=2589KiB/s (2651kB/s)(2048KiB/791msec) clat (usec): min=68606, max=99408, avg=84007.69, stdev=21780.72 lat (usec): min=68607, max=99410, avg=84008.90, stdev=21780.61 clat percentiles (usec): | 1.00th=[68682], 5.00th=[68682], 10.00th=[68682], 20.00th=[68682], | 30.00th=[68682], 40.00th=[68682], 50.00th=[68682], 60.00th=[99091], | 70.00th=[99091], 80.00th=[99091], 90.00th=[99091], 95.00th=[99091], | 99.00th=[99091], 99.50th=[99091], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 2048, max= 2048, per=12.63%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.6MiB/s (13.3MB/s)(10.0MiB/791msec) clat (msec): min=19, max=117, avg=53.66, stdev=30.51 lat (msec): min=19, max=117, avg=53.72, stdev=30.51 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 21], | 30.00th=[ 28], 40.00th=[ 43], 50.00th=[ 45], 60.00th=[ 51], | 70.00th=[ 69], 80.00th=[ 69], 90.00th=[ 83], 95.00th=[ 118], | 99.00th=[ 118], 99.50th=[ 118], 99.90th=[ 118], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 8192, max= 8192, per=36.09%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=50.00%, 250=8.33% cpu : usr=0.00%, sys=20.00%, ctx=70, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29629: Tue Mar 11 09:54:39 2025 read: IOPS=4, BW=4935KiB/s (5053kB/s)(6144KiB/1245msec) clat (msec): min=14, max=513, avg=124.34, stdev=192.34 lat (msec): min=14, max=513, avg=124.34, stdev=192.34 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 67], | 70.00th=[ 85], 80.00th=[ 85], 90.00th=[ 514], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 2048, max= 4071, per=18.87%, avg=3059.50, stdev=1430.48, samples=2 iops : min= 2, max= 3, avg= 2.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4935KiB/s (5053kB/s)(6144KiB/1245msec) clat (msec): min=21, max=160, avg=71.51, stdev=49.61 lat (msec): min=21, max=160, avg=71.57, stdev=49.60 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 59], 50.00th=[ 59], 60.00th=[ 72], | 70.00th=[ 86], 80.00th=[ 86], 90.00th=[ 161], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 2048, max= 6107, per=17.96%, avg=4077.50, stdev=2870.15, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (msec) : 20=8.33%, 50=33.33%, 100=41.67%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=8.92%, ctx=53, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29630: Tue Mar 11 09:54:39 2025 read: IOPS=3, BW=4054KiB/s (4151kB/s)(5120KiB/1263msec) clat (msec): min=16, max=549, avg=138.80, stdev=231.08 lat (msec): min=16, max=549, avg=138.80, stdev=231.08 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 17], | 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 32], 60.00th=[ 32], | 70.00th=[ 80], 80.00th=[ 80], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 2048, max= 2048, per=12.63%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5675KiB/s (5812kB/s)(7168KiB/1263msec) clat (msec): min=45, max=128, avg=72.49, stdev=28.16 lat (msec): min=45, max=128, avg=72.58, stdev=28.12 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 52], | 30.00th=[ 62], 40.00th=[ 62], 50.00th=[ 65], 60.00th=[ 68], | 70.00th=[ 68], 80.00th=[ 90], 90.00th=[ 129], 95.00th=[ 129], | 99.00th=[ 129], 99.50th=[ 129], 99.90th=[ 129], 99.95th=[ 129], | 99.99th=[ 129] bw ( KiB/s): min= 8192, max= 8192, per=36.09%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=16.67%, 100=50.00%, 250=8.33%, 750=8.33% cpu : usr=0.08%, sys=8.24%, ctx=59, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.8MiB/s (16.6MB/s), 2589KiB/s-10.4MiB/s (2651kB/s-10.9MB/s), io=20.0MiB (20.0MB), run=676-1263msec WRITE: bw=22.2MiB/s (23.2MB/s), 4935KiB/s-12.6MiB/s (5053kB/s-13.3MB/s), io=28.0MiB (29.4MB), run=676-1263msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29638: Tue Mar 11 09:54:41 2025 read: IOPS=9, BW=37.0MiB/s (38.8MB/s)(12.0MiB/324msec) clat (msec): min=27, max=149, avg=101.16, stdev=65.00 lat (msec): min=27, max=149, avg=101.17, stdev=65.00 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 127], 50.00th=[ 127], 60.00th=[ 127], | 70.00th=[ 150], 80.00th=[ 150], 90.00th=[ 150], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] lat (msec) : 50=33.33%, 250=66.67% cpu : usr=0.00%, sys=43.34%, ctx=1448, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29640: Tue Mar 11 09:54:41 2025 read: IOPS=1, BW=5634KiB/s (5769kB/s)(4096KiB/727msec) clat (nsec): min=89416k, max=89416k, avg=89415532.00, stdev= 0.00 lat (nsec): min=89418k, max=89418k, avg=89418315.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[89654], 5.00th=[89654], 10.00th=[89654], 20.00th=[89654], | 30.00th=[89654], 40.00th=[89654], 50.00th=[89654], 60.00th=[89654], | 70.00th=[89654], 80.00th=[89654], 90.00th=[89654], 95.00th=[89654], | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 8192, max= 8192, per=36.35%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=11.0MiB/s (11.5MB/s)(8192KiB/727msec) clat (msec): min=176, max=432, avg=304.50, stdev=181.29 lat (msec): min=176, max=433, avg=304.78, stdev=181.38 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 178], 10.00th=[ 178], 20.00th=[ 178], | 30.00th=[ 178], 40.00th=[ 178], 50.00th=[ 178], 60.00th=[ 435], | 70.00th=[ 435], 80.00th=[ 435], 90.00th=[ 435], 95.00th=[ 435], | 99.00th=[ 435], 99.50th=[ 435], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 8192, max= 8192, per=20.48%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=33.06%, ctx=60, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29642: Tue Mar 11 09:54:41 2025 write: IOPS=3, BW=14.7MiB/s (15.4MB/s)(12.0MiB/819msec) clat (msec): min=140, max=342, avg=269.60, stdev=112.32 lat (msec): min=140, max=342, avg=269.88, stdev=112.31 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 140], 10.00th=[ 140], 20.00th=[ 140], | 30.00th=[ 140], 40.00th=[ 326], 50.00th=[ 326], 60.00th=[ 326], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 342], 95.00th=[ 342], | 99.00th=[ 342], 99.50th=[ 342], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 8175, max= 8175, per=20.43%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=38.14%, ctx=53, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29644: Tue Mar 11 09:54:41 2025 write: IOPS=4, BW=16.3MiB/s (17.1MB/s)(12.0MiB/736msec) clat (msec): min=143, max=344, avg=238.18, stdev=101.04 lat (msec): min=143, max=344, avg=238.50, stdev=101.00 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 226], 50.00th=[ 226], 60.00th=[ 226], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 8192, max= 8192, per=20.48%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=47.48%, ctx=95, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.0MiB/s (23.1MB/s), 5634KiB/s-37.0MiB/s (5769kB/s-38.8MB/s), io=16.0MiB (16.8MB), run=324-727msec WRITE: bw=39.1MiB/s (40.0MB/s), 11.0MiB/s-16.3MiB/s (11.5MB/s-17.1MB/s), io=32.0MiB (33.6MB), run=727-819msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29637: Tue Mar 11 09:54:41 2025 read: IOPS=5, BW=22.1MiB/s (23.2MB/s)(12.0MiB/542msec) clat (msec): min=71, max=266, avg=171.72, stdev=97.91 lat (msec): min=71, max=266, avg=171.73, stdev=97.91 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 72], 10.00th=[ 72], 20.00th=[ 72], | 30.00th=[ 72], 40.00th=[ 178], 50.00th=[ 178], 60.00th=[ 178], | 70.00th=[ 268], 80.00th=[ 268], 90.00th=[ 268], 95.00th=[ 268], | 99.00th=[ 268], 99.50th=[ 268], 99.90th=[ 268], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min=16384, max=16384, per=88.90%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=20.15%, ctx=80, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29639: Tue Mar 11 09:54:41 2025 read: IOPS=1, BW=4607KiB/s (4718kB/s)(4096KiB/889msec) clat (nsec): min=436927k, max=436927k, avg=436927117.00, stdev= 0.00 lat (nsec): min=436930k, max=436930k, avg=436930405.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 439], 10.00th=[ 439], 20.00th=[ 439], | 30.00th=[ 439], 40.00th=[ 439], 50.00th=[ 439], 60.00th=[ 439], | 70.00th=[ 439], 80.00th=[ 439], 90.00th=[ 439], 95.00th=[ 439], | 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 8192, max= 8192, per=44.45%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9215KiB/s (9436kB/s)(8192KiB/889msec) clat (msec): min=122, max=318, avg=220.78, stdev=138.34 lat (msec): min=123, max=318, avg=221.06, stdev=138.33 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 124], 10.00th=[ 124], 20.00th=[ 124], | 30.00th=[ 124], 40.00th=[ 124], 50.00th=[ 124], 60.00th=[ 317], | 70.00th=[ 317], 80.00th=[ 317], 90.00th=[ 317], 95.00th=[ 317], | 99.00th=[ 317], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.11%, sys=12.05%, ctx=50, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29641: Tue Mar 11 09:54:41 2025 write: IOPS=3, BW=12.3MiB/s (12.9MB/s)(12.0MiB/979msec) clat (msec): min=63, max=529, avg=316.41, stdev=235.45 lat (msec): min=63, max=529, avg=316.70, stdev=235.45 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 64], | 30.00th=[ 64], 40.00th=[ 359], 50.00th=[ 359], 60.00th=[ 359], | 70.00th=[ 531], 80.00th=[ 531], 90.00th=[ 531], 95.00th=[ 531], | 99.00th=[ 531], 99.50th=[ 531], 99.90th=[ 531], 99.95th=[ 531], | 99.99th=[ 531] lat (msec) : 100=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=11.25%, ctx=50, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29643: Tue Mar 11 09:54:41 2025 write: IOPS=4, BW=18.0MiB/s (18.9MB/s)(12.0MiB/666msec) clat (msec): min=70, max=450, avg=213.22, stdev=206.98 lat (msec): min=70, max=450, avg=213.53, stdev=206.95 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 70], 10.00th=[ 70], 20.00th=[ 70], | 30.00th=[ 70], 40.00th=[ 120], 50.00th=[ 120], 60.00th=[ 120], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min=16351, max=16351, per=48.85%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.15%, sys=12.63%, ctx=43, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.0MiB/s (18.9MB/s), 4607KiB/s-22.1MiB/s (4718kB/s-23.2MB/s), io=16.0MiB (16.8MB), run=542-889msec WRITE: bw=32.7MiB/s (34.3MB/s), 9215KiB/s-18.0MiB/s (9436kB/s-18.9MB/s), io=32.0MiB (33.6MB), run=666-979msec