== sanity test 398b: DIO and buffer IO race ============== 20:53:37 (1746147217) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.95386 s, 25.8 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21523: Thu May 1 20:54:27 2025 read: IOPS=32, BW=131KiB/s (135kB/s)(5992KiB/45587msec) clat (usec): min=76, max=88736, avg=17065.73, stdev=11659.59 lat (usec): min=76, max=88737, avg=17068.63, stdev=11661.67 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 123], 10.00th=[ 5669], 20.00th=[ 8291], | 30.00th=[10421], 40.00th=[12518], 50.00th=[14877], 60.00th=[17695], | 70.00th=[20841], 80.00th=[24773], 90.00th=[30540], 95.00th=[38536], | 99.00th=[57934], 99.50th=[63701], 99.90th=[86508], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 47, max= 271, per=25.29%, avg=130.74, stdev=38.31, samples=91 iops : min= 11, max= 67, avg=32.35, stdev= 9.60, samples=91 write: IOPS=34, BW=138KiB/s (141kB/s)(6296KiB/45587msec) clat (usec): min=523, max=271480, avg=12590.63, stdev=15144.55 lat (usec): min=523, max=271482, avg=12593.17, stdev=15146.69 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 758], 10.00th=[ 816], 20.00th=[ 947], | 30.00th=[ 1926], 40.00th=[ 4047], 50.00th=[ 8455], 60.00th=[ 12125], | 70.00th=[ 16057], 80.00th=[ 22152], 90.00th=[ 30802], 95.00th=[ 40109], | 99.00th=[ 60031], 99.50th=[ 73925], 99.90th=[ 91751], 99.95th=[270533], | 99.99th=[270533] bw ( KiB/s): min= 39, max= 232, per=25.37%, avg=137.24, stdev=36.95, samples=91 iops : min= 9, max= 58, avg=33.98, stdev= 9.34, samples=91 lat (usec) : 100=0.91%, 250=2.60%, 500=0.13%, 750=2.05%, 1000=9.18% lat (msec) : 2=4.62%, 4=4.88%, 10=17.19%, 20=30.60%, 50=25.88% lat (msec) : 100=1.92%, 500=0.03% cpu : usr=0.16%, sys=10.29%, ctx=4957, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21524: Thu May 1 20:54:27 2025 read: IOPS=31, BW=128KiB/s (131kB/s)(5940KiB/46442msec) clat (usec): min=72, max=120288, avg=16774.43, stdev=11980.74 lat (usec): min=73, max=120289, avg=16777.28, stdev=11979.10 clat percentiles (usec): | 1.00th=[ 96], 5.00th=[ 128], 10.00th=[ 5473], 20.00th=[ 8455], | 30.00th=[ 10683], 40.00th=[ 12649], 50.00th=[ 14877], 60.00th=[ 17433], | 70.00th=[ 19792], 80.00th=[ 23462], 90.00th=[ 29492], 95.00th=[ 36963], | 99.00th=[ 68682], 99.50th=[ 77071], 99.90th=[100140], 99.95th=[120062], | 99.99th=[120062] bw ( KiB/s): min= 39, max= 232, per=24.52%, avg=126.75, stdev=34.02, samples=92 iops : min= 9, max= 58, avg=31.32, stdev= 8.55, samples=92 write: IOPS=34, BW=137KiB/s (140kB/s)(6348KiB/46442msec) clat (usec): min=581, max=314705, avg=13469.49, stdev=16054.94 lat (usec): min=582, max=314706, avg=13471.22, stdev=16055.12 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 758], 10.00th=[ 824], 20.00th=[ 996], | 30.00th=[ 2073], 40.00th=[ 5014], 50.00th=[ 9634], 60.00th=[ 12911], | 70.00th=[ 17171], 80.00th=[ 24249], 90.00th=[ 31851], 95.00th=[ 40633], | 99.00th=[ 60556], 99.50th=[ 69731], 99.90th=[111674], 99.95th=[316670], | 99.99th=[316670] bw ( KiB/s): min= 64, max= 256, per=24.92%, avg=134.82, stdev=39.95, samples=92 iops : min= 16, max= 64, avg=33.34, stdev=10.00, samples=92 lat (usec) : 100=0.75%, 250=2.70%, 500=0.16%, 750=2.15%, 1000=8.37% lat (msec) : 2=4.82%, 4=4.43%, 10=16.11%, 20=32.94%, 50=25.36% lat (msec) : 100=2.12%, 250=0.07%, 500=0.03% cpu : usr=0.13%, sys=10.16%, ctx=5015, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21525: Thu May 1 20:54:27 2025 read: IOPS=32, BW=130KiB/s (133kB/s)(6028KiB/46310msec) clat (usec): min=66, max=117574, avg=17485.69, stdev=11684.69 lat (usec): min=66, max=117575, avg=17493.35, stdev=11684.28 clat percentiles (usec): | 1.00th=[ 97], 5.00th=[ 143], 10.00th=[ 6587], 20.00th=[ 8979], | 30.00th=[ 11076], 40.00th=[ 13042], 50.00th=[ 15139], 60.00th=[ 17957], | 70.00th=[ 20841], 80.00th=[ 25035], 90.00th=[ 31589], 95.00th=[ 38536], | 99.00th=[ 56886], 99.50th=[ 65799], 99.90th=[ 92799], 99.95th=[117965], | 99.99th=[117965] bw ( KiB/s): min= 32, max= 231, per=25.06%, avg=129.55, stdev=37.46, samples=92 iops : min= 8, max= 57, avg=32.03, stdev= 9.33, samples=92 write: IOPS=33, BW=135KiB/s (138kB/s)(6260KiB/46310msec) clat (usec): min=564, max=297137, avg=12606.61, stdev=15579.34 lat (usec): min=565, max=297139, avg=12608.72, stdev=15579.10 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 758], 10.00th=[ 799], 20.00th=[ 930], | 30.00th=[ 1893], 40.00th=[ 4359], 50.00th=[ 8455], 60.00th=[ 11863], | 70.00th=[ 15926], 80.00th=[ 21627], 90.00th=[ 31065], 95.00th=[ 40109], | 99.00th=[ 56886], 99.50th=[ 68682], 99.90th=[119014], 99.95th=[295699], | 99.99th=[295699] bw ( KiB/s): min= 32, max= 248, per=24.85%, avg=134.42, stdev=40.43, samples=92 iops : min= 8, max= 62, avg=33.25, stdev=10.11, samples=92 lat (usec) : 100=0.78%, 250=2.08%, 500=0.10%, 750=2.18%, 1000=9.44% lat (msec) : 2=4.26%, 4=4.36%, 10=16.41%, 20=33.04%, 50=25.33% lat (msec) : 100=1.89%, 250=0.10%, 500=0.03% cpu : usr=0.13%, sys=10.03%, ctx=5036, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21526: Thu May 1 20:54:27 2025 read: IOPS=32, BW=131KiB/s (134kB/s)(6052KiB/46344msec) clat (usec): min=82, max=315892, avg=17355.25, stdev=14216.55 lat (usec): min=83, max=315894, avg=17356.48, stdev=14216.60 clat percentiles (usec): | 1.00th=[ 97], 5.00th=[ 129], 10.00th=[ 5800], 20.00th=[ 8717], | 30.00th=[ 10683], 40.00th=[ 12649], 50.00th=[ 14615], 60.00th=[ 17433], | 70.00th=[ 20579], 80.00th=[ 24511], 90.00th=[ 31065], 95.00th=[ 38536], | 99.00th=[ 60031], 99.50th=[ 67634], 99.90th=[120062], 99.95th=[316670], | 99.99th=[316670] bw ( KiB/s): min= 56, max= 263, per=25.12%, avg=129.89, stdev=32.94, samples=92 iops : min= 14, max= 65, avg=32.17, stdev= 8.18, samples=92 write: IOPS=33, BW=135KiB/s (138kB/s)(6236KiB/46344msec) clat (usec): min=484, max=89102, avg=12774.70, stdev=13705.39 lat (usec): min=485, max=89104, avg=12779.10, stdev=13712.99 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 758], 10.00th=[ 807], 20.00th=[ 1012], | 30.00th=[ 1991], 40.00th=[ 4490], 50.00th=[ 9241], 60.00th=[12518], | 70.00th=[16450], 80.00th=[21890], 90.00th=[31851], 95.00th=[41681], | 99.00th=[58983], 99.50th=[66323], 99.90th=[85459], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 47, max= 232, per=24.78%, avg=134.05, stdev=40.90, samples=92 iops : min= 11, max= 58, avg=33.22, stdev=10.23, samples=92 lat (usec) : 100=0.72%, 250=2.28%, 500=0.20%, 750=2.21%, 1000=7.88% lat (msec) : 2=5.18%, 4=4.39%, 10=16.83%, 20=32.85%, 50=25.55% lat (msec) : 100=1.82%, 250=0.07%, 500=0.03% cpu : usr=0.16%, sys=10.25%, ctx=5083, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=517KiB/s (529kB/s), 128KiB/s-131KiB/s (131kB/s-135kB/s), io=23.4MiB (24.6MB), run=45587-46442msec WRITE: bw=541KiB/s (554kB/s), 135KiB/s-138KiB/s (138kB/s-141kB/s), io=24.6MiB (25.7MB), run=45587-46442msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21527: Thu May 1 20:54:53 2025 read: IOPS=20, BW=83.8KiB/s (85.8kB/s)(5992KiB/71499msec) clat (msec): min=3, max=309, avg=17.29, stdev=16.12 lat (msec): min=3, max=309, avg=17.29, stdev=16.12 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 14], | 70.00th=[ 17], 80.00th=[ 22], 90.00th=[ 37], 95.00th=[ 49], | 99.00th=[ 72], 99.50th=[ 89], 99.90th=[ 114], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 16, max= 208, per=24.95%, avg=83.32, stdev=41.35, samples=142 iops : min= 4, max= 52, avg=20.56, stdev=10.35, samples=142 write: IOPS=22, BW=88.1KiB/s (90.2kB/s)(6296KiB/71499msec) clat (msec): min=6, max=130, avg=28.85, stdev=16.97 lat (msec): min=6, max=130, avg=28.86, stdev=16.97 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 21], 50.00th=[ 24], 60.00th=[ 28], | 70.00th=[ 32], 80.00th=[ 40], 90.00th=[ 52], 95.00th=[ 65], | 99.00th=[ 91], 99.50th=[ 105], 99.90th=[ 129], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 8, max= 184, per=25.01%, avg=87.54, stdev=38.88, samples=142 iops : min= 2, max= 46, avg=21.61, stdev= 9.74, samples=142 lat (msec) : 4=0.23%, 10=17.81%, 20=38.35%, 50=35.71%, 100=7.49% lat (msec) : 250=0.39%, 500=0.03% cpu : usr=0.13%, sys=4.52%, ctx=3452, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21528: Thu May 1 20:54:53 2025 read: IOPS=20, BW=82.9KiB/s (84.8kB/s)(5940KiB/71686msec) clat (msec): min=3, max=323, avg=17.39, stdev=16.38 lat (msec): min=3, max=323, avg=17.39, stdev=16.38 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 17], 80.00th=[ 23], 90.00th=[ 36], 95.00th=[ 51], | 99.00th=[ 74], 99.50th=[ 80], 99.90th=[ 123], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 7, max= 199, per=24.62%, avg=82.24, stdev=42.18, samples=143 iops : min= 1, max= 49, avg=20.16, stdev=10.55, samples=143 write: IOPS=22, BW=88.6KiB/s (90.7kB/s)(6348KiB/71686msec) clat (msec): min=6, max=146, avg=28.81, stdev=16.89 lat (msec): min=6, max=146, avg=28.81, stdev=16.89 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 21], 50.00th=[ 24], 60.00th=[ 28], | 70.00th=[ 33], 80.00th=[ 40], 90.00th=[ 51], 95.00th=[ 61], | 99.00th=[ 91], 99.50th=[ 106], 99.90th=[ 132], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 15, max= 191, per=25.09%, avg=87.80, stdev=39.08, samples=143 iops : min= 3, max= 47, avg=21.55, stdev= 9.76, samples=143 lat (msec) : 4=0.20%, 10=17.81%, 20=37.66%, 50=36.52%, 100=7.29% lat (msec) : 250=0.49%, 500=0.03% cpu : usr=0.09%, sys=4.54%, ctx=3410, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21529: Thu May 1 20:54:53 2025 read: IOPS=21, BW=85.0KiB/s (87.1kB/s)(6028KiB/70903msec) clat (msec): min=3, max=130, avg=17.67, stdev=14.56 lat (msec): min=3, max=130, avg=17.68, stdev=14.56 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 24], 90.00th=[ 37], 95.00th=[ 48], | 99.00th=[ 75], 99.50th=[ 88], 99.90th=[ 103], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 16, max= 192, per=25.29%, avg=84.47, stdev=39.82, samples=141 iops : min= 4, max= 48, avg=20.72, stdev= 9.99, samples=141 write: IOPS=22, BW=88.3KiB/s (90.4kB/s)(6260KiB/70903msec) clat (msec): min=5, max=291, avg=28.15, stdev=17.80 lat (msec): min=5, max=291, avg=28.15, stdev=17.80 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 19], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 27], | 70.00th=[ 31], 80.00th=[ 39], 90.00th=[ 51], 95.00th=[ 62], | 99.00th=[ 91], 99.50th=[ 102], 99.90th=[ 148], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 24, max= 224, per=25.03%, avg=87.60, stdev=40.37, samples=141 iops : min= 6, max= 56, avg=21.50, stdev=10.13, samples=141 lat (msec) : 4=0.10%, 10=17.51%, 20=38.54%, 50=36.62%, 100=6.90% lat (msec) : 250=0.29%, 500=0.03% cpu : usr=0.12%, sys=4.53%, ctx=3406, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21530: Thu May 1 20:54:53 2025 read: IOPS=21, BW=84.5KiB/s (86.6kB/s)(6052KiB/71599msec) clat (msec): min=3, max=134, avg=17.80, stdev=15.63 lat (msec): min=3, max=134, avg=17.80, stdev=15.63 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 24], 90.00th=[ 38], 95.00th=[ 54], | 99.00th=[ 78], 99.50th=[ 90], 99.90th=[ 121], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 15, max= 207, per=25.32%, avg=84.56, stdev=40.82, samples=142 iops : min= 3, max= 51, avg=20.75, stdev=10.24, samples=142 write: IOPS=21, BW=87.1KiB/s (89.2kB/s)(6236KiB/71599msec) clat (msec): min=6, max=346, avg=28.55, stdev=17.90 lat (msec): min=6, max=346, avg=28.55, stdev=17.90 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 21], 50.00th=[ 24], 60.00th=[ 27], | 70.00th=[ 32], 80.00th=[ 39], 90.00th=[ 50], 95.00th=[ 62], | 99.00th=[ 90], 99.50th=[ 101], 99.90th=[ 106], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 23, max= 192, per=24.68%, avg=86.36, stdev=38.73, samples=143 iops : min= 5, max= 48, avg=21.20, stdev= 9.71, samples=143 lat (msec) : 4=0.13%, 10=17.77%, 20=37.53%, 50=36.85%, 100=7.36% lat (msec) : 250=0.33%, 500=0.03% cpu : usr=0.13%, sys=4.56%, ctx=3407, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=335KiB/s (343kB/s), 82.9KiB/s-85.0KiB/s (84.8kB/s-87.1kB/s), io=23.4MiB (24.6MB), run=70903-71686msec WRITE: bw=351KiB/s (359kB/s), 87.1KiB/s-88.6KiB/s (89.2kB/s-90.7kB/s), io=24.6MiB (25.7MB), run=70903-71686msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21602: Thu May 1 20:55:06 2025 read: IOPS=31, BW=500KiB/s (512kB/s)(5920KiB/11848msec) clat (usec): min=70, max=102504, avg=16350.34, stdev=13581.96 lat (usec): min=70, max=102504, avg=16354.21, stdev=13586.73 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 105], 10.00th=[ 116], 20.00th=[ 5342], | 30.00th=[ 9110], 40.00th=[ 11994], 50.00th=[ 15008], 60.00th=[ 18220], | 70.00th=[ 20317], 80.00th=[ 23462], 90.00th=[ 33424], 95.00th=[ 40109], | 99.00th=[ 58983], 99.50th=[ 74974], 99.90th=[102237], 99.95th=[102237], | 99.99th=[102237] bw ( KiB/s): min= 160, max= 864, per=24.66%, avg=492.74, stdev=162.65, samples=23 iops : min= 10, max= 54, avg=30.30, stdev=10.15, samples=23 write: IOPS=33, BW=537KiB/s (550kB/s)(6368KiB/11848msec) clat (usec): min=912, max=97012, avg=13994.29, stdev=16065.26 lat (usec): min=913, max=97013, avg=13996.19, stdev=16065.28 clat percentiles (usec): | 1.00th=[ 963], 5.00th=[ 1074], 10.00th=[ 1172], 20.00th=[ 1418], | 30.00th=[ 2573], 40.00th=[ 4686], 50.00th=[ 8291], 60.00th=[11994], | 70.00th=[16909], 80.00th=[23462], 90.00th=[34866], 95.00th=[51119], | 99.00th=[69731], 99.50th=[81265], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 96, max= 894, per=25.48%, avg=535.83, stdev=173.69, samples=23 iops : min= 6, max= 55, avg=33.00, stdev=10.91, samples=23 lat (usec) : 100=1.82%, 250=6.25%, 500=0.78%, 750=0.13%, 1000=0.65% lat (msec) : 2=12.24%, 4=7.03%, 10=15.23%, 20=28.65%, 50=23.44% lat (msec) : 100=3.65%, 250=0.13% cpu : usr=0.18%, sys=10.13%, ctx=1236, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21603: Thu May 1 20:55:06 2025 read: IOPS=29, BW=479KiB/s (491kB/s)(5744KiB/11982msec) clat (usec): min=73, max=98958, avg=18364.50, stdev=14112.14 lat (usec): min=74, max=98959, avg=18375.48, stdev=14101.20 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 109], 10.00th=[ 133], 20.00th=[ 7635], | 30.00th=[10814], 40.00th=[13698], 50.00th=[16712], 60.00th=[19530], | 70.00th=[22676], 80.00th=[28967], 90.00th=[35914], 95.00th=[42206], | 99.00th=[68682], 99.50th=[84411], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 192, max= 736, per=23.88%, avg=477.13, stdev=126.12, samples=23 iops : min= 12, max= 46, avg=29.35, stdev= 7.80, samples=23 write: IOPS=34, BW=546KiB/s (559kB/s)(6544KiB/11982msec) clat (usec): min=792, max=62737, avg=12760.72, stdev=12706.74 lat (usec): min=793, max=62739, avg=12768.30, stdev=12701.94 clat percentiles (usec): | 1.00th=[ 906], 5.00th=[ 1106], 10.00th=[ 1172], 20.00th=[ 1418], | 30.00th=[ 2868], 40.00th=[ 5538], 50.00th=[ 9896], 60.00th=[13042], | 70.00th=[15795], 80.00th=[21103], 90.00th=[32375], 95.00th=[38011], | 99.00th=[55313], 99.50th=[60031], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 255, max= 1117, per=25.40%, avg=534.13, stdev=182.87, samples=23 iops : min= 15, max= 69, avg=32.91, stdev=11.51, samples=23 lat (usec) : 100=1.04%, 250=5.60%, 500=0.13%, 1000=1.43% lat (msec) : 2=13.15%, 4=4.04%, 10=13.80%, 20=31.12%, 50=27.73% lat (msec) : 100=1.95% cpu : usr=0.18%, sys=10.63%, ctx=1312, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21605: Thu May 1 20:55:06 2025 read: IOPS=32, BW=525KiB/s (537kB/s)(6256KiB/11923msec) clat (usec): min=86, max=87934, avg=17875.15, stdev=14032.18 lat (usec): min=86, max=87935, avg=17882.27, stdev=14028.18 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 110], 10.00th=[ 121], 20.00th=[ 7635], | 30.00th=[11207], 40.00th=[13566], 50.00th=[16057], 60.00th=[18482], | 70.00th=[21627], 80.00th=[26870], 90.00th=[34866], 95.00th=[43254], | 99.00th=[71828], 99.50th=[79168], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 286, max= 864, per=26.06%, avg=520.65, stdev=153.16, samples=23 iops : min= 17, max= 54, avg=32.00, stdev= 9.74, samples=23 write: IOPS=31, BW=506KiB/s (518kB/s)(6032KiB/11923msec) clat (usec): min=888, max=129699, avg=12951.96, stdev=15530.39 lat (usec): min=891, max=129701, avg=12953.81, stdev=15530.39 clat percentiles (usec): | 1.00th=[ 1029], 5.00th=[ 1106], 10.00th=[ 1172], 20.00th=[ 1319], | 30.00th=[ 2278], 40.00th=[ 4146], 50.00th=[ 7242], 60.00th=[ 10814], | 70.00th=[ 15401], 80.00th=[ 22676], 90.00th=[ 32637], 95.00th=[ 47973], | 99.00th=[ 61604], 99.50th=[ 80217], 99.90th=[129500], 99.95th=[129500], | 99.99th=[129500] bw ( KiB/s): min= 255, max= 670, per=23.17%, avg=487.26, stdev=137.13, samples=23 iops : min= 15, max= 41, avg=29.91, stdev= 8.59, samples=23 lat (usec) : 100=1.30%, 250=6.64%, 500=0.13%, 750=0.26%, 1000=0.26% lat (msec) : 2=14.06%, 4=5.08%, 10=13.28%, 20=30.86%, 50=25.00% lat (msec) : 100=2.99%, 250=0.13% cpu : usr=0.09%, sys=10.26%, ctx=1268, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21606: Thu May 1 20:55:06 2025 read: IOPS=32, BW=514KiB/s (527kB/s)(6032KiB/11729msec) clat (usec): min=87, max=64703, avg=17104.15, stdev=11717.12 lat (usec): min=87, max=64705, avg=17105.51, stdev=11717.41 clat percentiles (usec): | 1.00th=[ 97], 5.00th=[ 113], 10.00th=[ 133], 20.00th=[ 8291], | 30.00th=[11076], 40.00th=[13829], 50.00th=[16188], 60.00th=[18482], | 70.00th=[21627], 80.00th=[24773], 90.00th=[32900], 95.00th=[36963], | 99.00th=[54264], 99.50th=[60556], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 127, max= 704, per=25.63%, avg=512.13, stdev=124.94, samples=23 iops : min= 7, max= 44, avg=31.57, stdev= 7.96, samples=23 write: IOPS=33, BW=533KiB/s (546kB/s)(6256KiB/11729msec) clat (usec): min=831, max=79000, avg=12989.17, stdev=15073.85 lat (usec): min=833, max=79001, avg=12991.13, stdev=15073.84 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 1090], 10.00th=[ 1156], 20.00th=[ 1369], | 30.00th=[ 2474], 40.00th=[ 4015], 50.00th=[ 5932], 60.00th=[10945], | 70.00th=[17171], 80.00th=[23462], 90.00th=[33162], 95.00th=[42730], | 99.00th=[66847], 99.50th=[72877], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 223, max= 768, per=25.20%, avg=529.96, stdev=148.90, samples=23 iops : min= 13, max= 48, avg=32.70, stdev= 9.43, samples=23 lat (usec) : 100=0.65%, 250=5.60%, 500=0.39%, 750=0.13%, 1000=0.78% lat (msec) : 2=13.02%, 4=6.77%, 10=15.89%, 20=27.08%, 50=26.95% lat (msec) : 100=2.73% cpu : usr=0.11%, sys=10.52%, ctx=1259, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1999KiB/s (2047kB/s), 479KiB/s-525KiB/s (491kB/s-537kB/s), io=23.4MiB (24.5MB), run=11729-11982msec WRITE: bw=2103KiB/s (2154kB/s), 506KiB/s-546KiB/s (518kB/s-559kB/s), io=24.6MiB (25.8MB), run=11729-11982msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21607: Thu May 1 20:55:13 2025 read: IOPS=20, BW=330KiB/s (338kB/s)(5920KiB/17943msec) clat (msec): min=3, max=107, avg=17.67, stdev=16.70 lat (msec): min=3, max=107, avg=17.67, stdev=16.70 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 18], 80.00th=[ 23], 90.00th=[ 37], 95.00th=[ 58], | 99.00th=[ 92], 99.50th=[ 104], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 127, max= 704, per=24.43%, avg=322.03, stdev=156.76, samples=35 iops : min= 7, max= 44, avg=19.71, stdev= 9.88, samples=35 write: IOPS=22, BW=355KiB/s (363kB/s)(6368KiB/17943msec) clat (usec): min=7707, max=97762, avg=28471.55, stdev=17584.97 lat (usec): min=7709, max=97764, avg=28477.47, stdev=17582.52 clat percentiles (usec): | 1.00th=[ 8586], 5.00th=[11731], 10.00th=[13042], 20.00th=[15533], | 30.00th=[17433], 40.00th=[19530], 50.00th=[21890], 60.00th=[25822], | 70.00th=[31065], 80.00th=[40633], 90.00th=[53740], 95.00th=[63701], | 99.00th=[94897], 99.50th=[96994], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 95, max= 734, per=25.40%, avg=352.09, stdev=180.82, samples=35 iops : min= 5, max= 45, avg=21.60, stdev=11.35, samples=35 lat (msec) : 4=0.13%, 10=19.27%, 20=39.71%, 50=31.51%, 100=9.11% lat (msec) : 250=0.26% cpu : usr=0.08%, sys=4.63%, ctx=890, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21608: Thu May 1 20:55:13 2025 read: IOPS=19, BW=317KiB/s (324kB/s)(5744KiB/18142msec) clat (usec): min=4237, max=96999, avg=17128.51, stdev=14747.18 lat (usec): min=4239, max=97000, avg=17130.16, stdev=14747.08 clat percentiles (usec): | 1.00th=[ 4883], 5.00th=[ 5932], 10.00th=[ 6521], 20.00th=[ 7570], | 30.00th=[ 8979], 40.00th=[10290], 50.00th=[11863], 60.00th=[13566], | 70.00th=[16581], 80.00th=[22152], 90.00th=[36439], 95.00th=[50594], | 99.00th=[77071], 99.50th=[79168], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 31, max= 734, per=23.88%, avg=314.75, stdev=180.89, samples=36 iops : min= 1, max= 45, avg=19.22, stdev=11.32, samples=36 write: IOPS=22, BW=361KiB/s (369kB/s)(6544KiB/18142msec) clat (msec): min=7, max=121, avg=28.71, stdev=18.05 lat (msec): min=7, max=121, avg=28.72, stdev=18.06 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 26], | 70.00th=[ 31], 80.00th=[ 41], 90.00th=[ 56], 95.00th=[ 70], | 99.00th=[ 83], 99.50th=[ 93], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 127, max= 800, per=25.85%, avg=358.22, stdev=165.08, samples=36 iops : min= 7, max= 50, avg=21.94, stdev=10.28, samples=36 lat (msec) : 10=18.49%, 20=39.06%, 50=32.94%, 100=9.37%, 250=0.13% cpu : usr=0.18%, sys=4.33%, ctx=866, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21609: Thu May 1 20:55:13 2025 read: IOPS=21, BW=346KiB/s (354kB/s)(6256KiB/18078msec) clat (usec): min=3617, max=98595, avg=17273.93, stdev=15402.42 lat (usec): min=3617, max=98596, avg=17275.39, stdev=15402.32 clat percentiles (usec): | 1.00th=[ 4424], 5.00th=[ 5932], 10.00th=[ 6390], 20.00th=[ 7439], | 30.00th=[ 8717], 40.00th=[ 9634], 50.00th=[11207], 60.00th=[13435], | 70.00th=[16319], 80.00th=[22152], 90.00th=[39060], 95.00th=[53740], | 99.00th=[68682], 99.50th=[87557], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 63, max= 896, per=26.29%, avg=346.53, stdev=207.43, samples=36 iops : min= 3, max= 56, avg=21.11, stdev=12.94, samples=36 write: IOPS=20, BW=334KiB/s (342kB/s)(6032KiB/18078msec) clat (msec): min=7, max=143, avg=29.27, stdev=18.53 lat (msec): min=7, max=143, avg=29.27, stdev=18.53 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 27], | 70.00th=[ 32], 80.00th=[ 40], 90.00th=[ 57], 95.00th=[ 72], | 99.00th=[ 90], 99.50th=[ 96], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 95, max= 766, per=23.91%, avg=331.42, stdev=166.45, samples=36 iops : min= 5, max= 47, avg=20.17, stdev=10.34, samples=36 lat (msec) : 4=0.13%, 10=22.40%, 20=36.07%, 50=32.29%, 100=8.98% lat (msec) : 250=0.13% cpu : usr=0.10%, sys=4.69%, ctx=881, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21610: Thu May 1 20:55:13 2025 read: IOPS=20, BW=332KiB/s (340kB/s)(6032KiB/18169msec) clat (msec): min=4, max=109, avg=18.41, stdev=17.82 lat (msec): min=4, max=109, avg=18.41, stdev=17.82 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 18], 80.00th=[ 24], 90.00th=[ 43], 95.00th=[ 62], | 99.00th=[ 94], 99.50th=[ 103], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 127, max= 768, per=25.51%, avg=336.26, stdev=171.46, samples=35 iops : min= 7, max= 48, avg=20.40, stdev=10.82, samples=35 write: IOPS=21, BW=344KiB/s (353kB/s)(6256KiB/18169msec) clat (msec): min=6, max=144, avg=27.94, stdev=18.51 lat (msec): min=6, max=144, avg=27.94, stdev=18.51 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 25], | 70.00th=[ 31], 80.00th=[ 39], 90.00th=[ 52], 95.00th=[ 71], | 99.00th=[ 90], 99.50th=[ 123], 99.90th=[ 146], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 95, max= 768, per=24.67%, avg=341.92, stdev=186.42, samples=36 iops : min= 5, max= 48, avg=20.75, stdev=11.81, samples=36 lat (msec) : 10=20.70%, 20=39.19%, 50=31.38%, 100=8.20%, 250=0.52% cpu : usr=0.16%, sys=4.67%, ctx=904, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1318KiB/s (1350kB/s), 317KiB/s-346KiB/s (324kB/s-354kB/s), io=23.4MiB (24.5MB), run=17943-18169msec WRITE: bw=1387KiB/s (1420kB/s), 334KiB/s-361KiB/s (342kB/s-369kB/s), io=24.6MiB (25.8MB), run=17943-18169msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21635: Thu May 1 20:55:16 2025 read: IOPS=5, BW=5306KiB/s (5433kB/s)(7168KiB/1351msec) clat (usec): min=509, max=690144, avg=120284.80, stdev=253655.53 lat (usec): min=511, max=690148, avg=120286.89, stdev=253656.14 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 510], 10.00th=[ 510], 20.00th=[ 570], | 30.00th=[ 1729], 40.00th=[ 1729], 50.00th=[ 2835], 60.00th=[ 60556], | 70.00th=[ 60556], 80.00th=[ 85459], 90.00th=[692061], 95.00th=[692061], | 99.00th=[692061], 99.50th=[692061], 99.90th=[692061], 99.95th=[692061], | 99.99th=[692061] bw ( KiB/s): min= 8159, max= 8159, per=76.10%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3790KiB/s (3881kB/s)(5120KiB/1351msec) clat (msec): min=37, max=165, avg=78.50, stdev=51.85 lat (msec): min=37, max=165, avg=78.55, stdev=51.86 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 38], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 59], 60.00th=[ 59], | 70.00th=[ 86], 80.00th=[ 86], 90.00th=[ 165], 95.00th=[ 165], | 99.00th=[ 165], 99.50th=[ 165], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 2039, max= 2039, per=13.58%, avg=2039.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 2=8.33%, 4=8.33%, 50=16.67%, 100=33.33%, 250=8.33% lat (msec) : 750=8.33% cpu : usr=0.00%, sys=16.07%, ctx=205, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21636: Thu May 1 20:55:16 2025 read: IOPS=1, BW=1072KiB/s (1098kB/s)(2048KiB/1910msec) clat (msec): min=79, max=135, avg=107.53, stdev=39.46 lat (msec): min=79, max=135, avg=107.53, stdev=39.46 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 80], 10.00th=[ 80], 20.00th=[ 80], | 30.00th=[ 80], 40.00th=[ 80], 50.00th=[ 80], 60.00th=[ 136], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 1980, max= 2039, per=18.74%, avg=2009.50, stdev=41.72, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5361KiB/s (5490kB/s)(10.0MiB/1910msec) clat (msec): min=37, max=642, avg=161.88, stdev=176.96 lat (msec): min=37, max=642, avg=162.00, stdev=176.94 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 58], | 30.00th=[ 73], 40.00th=[ 84], 50.00th=[ 91], 60.00th=[ 95], | 70.00th=[ 178], 80.00th=[ 178], 90.00th=[ 190], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 2031, max= 4079, per=22.36%, avg=3357.00, stdev=1149.86, samples=3 iops : min= 1, max= 3, avg= 2.33, stdev= 1.15, samples=3 lat (msec) : 50=8.33%, 100=50.00%, 250=33.33%, 750=8.33% cpu : usr=0.00%, sys=16.13%, ctx=225, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21637: Thu May 1 20:55:16 2025 read: IOPS=5, BW=5333KiB/s (5461kB/s)(6144KiB/1152msec) clat (usec): min=1354, max=188082, avg=71638.47, stdev=69014.40 lat (usec): min=1356, max=188084, avg=71640.10, stdev=69014.43 clat percentiles (usec): | 1.00th=[ 1352], 5.00th=[ 1352], 10.00th=[ 1352], 20.00th=[ 1958], | 30.00th=[ 1958], 40.00th=[ 66323], 50.00th=[ 66323], 60.00th=[ 83362], | 70.00th=[ 88605], 80.00th=[ 88605], 90.00th=[187696], 95.00th=[187696], | 99.00th=[187696], 99.50th=[187696], 99.90th=[187696], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 8192, max= 8192, per=76.40%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5333KiB/s (5461kB/s)(6144KiB/1152msec) clat (msec): min=55, max=154, avg=84.85, stdev=35.79 lat (msec): min=55, max=154, avg=84.93, stdev=35.80 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 56], 10.00th=[ 56], 20.00th=[ 66], | 30.00th=[ 66], 40.00th=[ 71], 50.00th=[ 71], 60.00th=[ 74], | 70.00th=[ 90], 80.00th=[ 90], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 4079, max= 4096, per=27.23%, avg=4087.50, stdev=12.02, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 lat (msec) : 2=16.67%, 100=66.67%, 250=16.67% cpu : usr=0.00%, sys=20.49%, ctx=327, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21638: Thu May 1 20:55:16 2025 read: IOPS=2, BW=3014KiB/s (3086kB/s)(5120KiB/1699msec) clat (usec): min=508, max=145874, avg=60200.35, stdev=55136.11 lat (usec): min=510, max=145877, avg=60201.98, stdev=55136.37 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 510], 10.00th=[ 510], 20.00th=[ 510], | 30.00th=[ 25560], 40.00th=[ 25560], 50.00th=[ 62129], 60.00th=[ 62129], | 70.00th=[ 66847], 80.00th=[ 66847], 90.00th=[145753], 95.00th=[145753], | 99.00th=[145753], 99.50th=[145753], 99.90th=[145753], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 8192, max= 8192, per=76.40%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4219KiB/s (4320kB/s)(7168KiB/1699msec) clat (msec): min=76, max=467, avg=171.25, stdev=135.50 lat (msec): min=76, max=467, avg=171.34, stdev=135.52 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 78], 10.00th=[ 78], 20.00th=[ 95], | 30.00th=[ 102], 40.00th=[ 102], 50.00th=[ 120], 60.00th=[ 163], | 70.00th=[ 163], 80.00th=[ 178], 90.00th=[ 468], 95.00th=[ 468], | 99.00th=[ 468], 99.50th=[ 468], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 2048, max= 6144, per=27.29%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (usec) : 750=8.33% lat (msec) : 50=8.33%, 100=33.33%, 250=41.67%, 500=8.33% cpu : usr=0.00%, sys=13.31%, ctx=301, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.5MiB/s (10.0MB/s), 1072KiB/s-5333KiB/s (1098kB/s-5461kB/s), io=20.0MiB (20.0MB), run=1152-1910msec WRITE: bw=14.7MiB/s (15.4MB/s), 3790KiB/s-5361KiB/s (3881kB/s-5490kB/s), io=28.0MiB (29.4MB), run=1152-1910msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21631: Thu May 1 20:55:16 2025 read: IOPS=8, BW=8605KiB/s (8812kB/s)(7168KiB/833msec) clat (msec): min=31, max=165, avg=80.72, stdev=55.56 lat (msec): min=31, max=165, avg=80.72, stdev=55.56 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 34], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 65], 60.00th=[ 84], | 70.00th=[ 84], 80.00th=[ 148], 90.00th=[ 165], 95.00th=[ 165], | 99.00th=[ 165], 99.50th=[ 165], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 8192, max= 8192, per=81.12%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6146KiB/s (6294kB/s)(5120KiB/833msec) clat (msec): min=20, max=126, avg=48.87, stdev=43.79 lat (msec): min=20, max=126, avg=48.99, stdev=43.81 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 27], 40.00th=[ 27], 50.00th=[ 35], 60.00th=[ 35], | 70.00th=[ 37], 80.00th=[ 37], 90.00th=[ 127], 95.00th=[ 127], | 99.00th=[ 127], 99.50th=[ 127], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 2048, max= 2048, per=14.49%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=16.67%, 250=25.00% cpu : usr=0.36%, sys=4.45%, ctx=30, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21632: Thu May 1 20:55:16 2025 read: IOPS=0, BW=1010KiB/s (1034kB/s)(2048KiB/2028msec) clat (msec): min=35, max=102, avg=69.12, stdev=47.43 lat (msec): min=35, max=102, avg=69.12, stdev=47.43 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 103], | 70.00th=[ 103], 80.00th=[ 103], 90.00th=[ 103], 95.00th=[ 103], | 99.00th=[ 103], 99.50th=[ 103], 99.90th=[ 103], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min= 2048, max= 2048, per=20.28%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=5049KiB/s (5170kB/s)(10.0MiB/2028msec) clat (msec): min=28, max=646, avg=182.92, stdev=180.86 lat (msec): min=29, max=646, avg=183.09, stdev=180.82 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 65], | 30.00th=[ 78], 40.00th=[ 91], 50.00th=[ 95], 60.00th=[ 122], | 70.00th=[ 218], 80.00th=[ 218], 90.00th=[ 275], 95.00th=[ 651], | 99.00th=[ 651], 99.50th=[ 651], 99.90th=[ 651], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 2031, max= 4096, per=24.10%, avg=3407.67, stdev=1192.23, samples=3 iops : min= 1, max= 4, avg= 3.00, stdev= 1.73, samples=3 lat (msec) : 50=16.67%, 100=33.33%, 250=33.33%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=5.08%, ctx=53, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21633: Thu May 1 20:55:16 2025 read: IOPS=3, BW=3152KiB/s (3228kB/s)(6144KiB/1949msec) clat (msec): min=60, max=676, avg=209.17, stdev=231.79 lat (msec): min=60, max=676, avg=209.18, stdev=231.79 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 61], 10.00th=[ 61], 20.00th=[ 94], | 30.00th=[ 94], 40.00th=[ 130], 50.00th=[ 130], 60.00th=[ 132], | 70.00th=[ 163], 80.00th=[ 163], 90.00th=[ 676], 95.00th=[ 676], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 2043, max= 2048, per=20.26%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3152KiB/s (3228kB/s)(6144KiB/1949msec) clat (msec): min=32, max=154, avg=82.58, stdev=47.43 lat (msec): min=32, max=154, avg=82.67, stdev=47.41 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 57], | 30.00th=[ 57], 40.00th=[ 62], 50.00th=[ 62], 60.00th=[ 63], | 70.00th=[ 129], 80.00th=[ 129], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 2043, max= 6131, per=28.91%, avg=4087.00, stdev=2890.65, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 lat (msec) : 50=8.33%, 100=41.67%, 250=41.67%, 750=8.33% cpu : usr=0.00%, sys=4.41%, ctx=46, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21634: Thu May 1 20:55:16 2025 read: IOPS=3, BW=3120KiB/s (3195kB/s)(5120KiB/1641msec) clat (msec): min=38, max=140, avg=72.22, stdev=46.51 lat (msec): min=38, max=140, avg=72.22, stdev=46.50 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 39], | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 43], 60.00th=[ 43], | 70.00th=[ 102], 80.00th=[ 102], 90.00th=[ 140], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 2003, max= 6144, per=40.34%, avg=4073.50, stdev=2928.13, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 write: IOPS=4, BW=4368KiB/s (4473kB/s)(7168KiB/1641msec) clat (msec): min=46, max=577, avg=170.49, stdev=185.57 lat (msec): min=46, max=577, avg=170.70, stdev=185.56 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 57], | 30.00th=[ 66], 40.00th=[ 66], 50.00th=[ 140], 60.00th=[ 146], | 70.00th=[ 146], 80.00th=[ 163], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 2003, max= 6131, per=28.83%, avg=4076.67, stdev=2064.07, samples=3 iops : min= 1, max= 5, avg= 3.33, stdev= 2.08, samples=3 lat (msec) : 50=33.33%, 100=16.67%, 250=41.67%, 750=8.33% cpu : usr=0.00%, sys=2.38%, ctx=33, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9.86MiB/s (10.3MB/s), 1010KiB/s-8605KiB/s (1034kB/s-8812kB/s), io=20.0MiB (20.0MB), run=833-2028msec WRITE: bw=13.8MiB/s (14.5MB/s), 3152KiB/s-6146KiB/s (3228kB/s-6294kB/s), io=28.0MiB (29.4MB), run=833-2028msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21646: Thu May 1 20:55:19 2025 read: IOPS=3, BW=13.0MiB/s (14.6MB/s)(12.0MiB/860msec) clat (msec): min=133, max=411, avg=252.03, stdev=143.66 lat (msec): min=133, max=411, avg=252.03, stdev=143.66 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 134], 10.00th=[ 134], 20.00th=[ 134], | 30.00th=[ 134], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 211], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min=16318, max=16318, per=100.00%, avg=16318.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=18.28%, ctx=688, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21648: Thu May 1 20:55:19 2025 read: IOPS=0, BW=2926KiB/s (2996kB/s)(4096KiB/1400msec) clat (nsec): min=280144k, max=280144k, avg=280144440.00, stdev= 0.00 lat (nsec): min=280147k, max=280147k, avg=280146915.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 279], 10.00th=[ 279], 20.00th=[ 279], | 30.00th=[ 279], 40.00th=[ 279], 50.00th=[ 279], 60.00th=[ 279], | 70.00th=[ 279], 80.00th=[ 279], 90.00th=[ 279], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 8175, max= 8175, per=69.86%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5851KiB/s (5992kB/s)(8192KiB/1400msec) clat (msec): min=436, max=633, avg=535.06, stdev=139.23 lat (msec): min=437, max=633, avg=535.57, stdev=139.19 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 439], 10.00th=[ 439], 20.00th=[ 439], | 30.00th=[ 439], 40.00th=[ 439], 50.00th=[ 439], 60.00th=[ 634], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 8192, max= 8192, per=36.68%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.00%, sys=19.80%, ctx=338, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21650: Thu May 1 20:55:19 2025 write: IOPS=2, BW=8376KiB/s (8577kB/s)(12.0MiB/1467msec) clat (msec): min=245, max=595, avg=457.83, stdev=186.26 lat (msec): min=247, max=596, avg=458.77, stdev=185.43 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 247], 10.00th=[ 247], 20.00th=[ 247], | 30.00th=[ 247], 40.00th=[ 531], 50.00th=[ 531], 60.00th=[ 531], | 70.00th=[ 600], 80.00th=[ 600], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 8192, max= 8192, per=36.68%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=66.67% cpu : usr=0.07%, sys=21.01%, ctx=81, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21652: Thu May 1 20:55:19 2025 write: IOPS=2, BW=9.96MiB/s (10.4MB/s)(12.0MiB/1205msec) clat (msec): min=240, max=659, avg=391.67, stdev=232.83 lat (msec): min=240, max=660, avg=392.90, stdev=232.28 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 241], 10.00th=[ 241], 20.00th=[ 241], | 30.00th=[ 241], 40.00th=[ 275], 50.00th=[ 275], 60.00th=[ 275], | 70.00th=[ 659], 80.00th=[ 659], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 8110, max= 8175, per=36.45%, avg=8142.50, stdev=45.96, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=24.09%, ctx=87, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.4MiB/s (11.0MB/s), 2926KiB/s-13.0MiB/s (2996kB/s-14.6MB/s), io=16.0MiB (16.8MB), run=860-1400msec WRITE: bw=21.8MiB/s (22.9MB/s), 5851KiB/s-9.96MiB/s (5992kB/s-10.4MB/s), io=32.0MiB (33.6MB), run=1205-1467msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21647: Thu May 1 20:55:19 2025 read: IOPS=2, BW=10.2MiB/s (10.7MB/s)(12.0MiB/1173msec) clat (msec): min=246, max=498, avg=365.41, stdev=126.99 lat (msec): min=246, max=498, avg=365.42, stdev=126.99 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 247], 10.00th=[ 247], 20.00th=[ 247], | 30.00th=[ 247], 40.00th=[ 351], 50.00th=[ 351], 60.00th=[ 351], | 70.00th=[ 498], 80.00th=[ 498], 90.00th=[ 498], 95.00th=[ 498], | 99.00th=[ 498], 99.50th=[ 498], 99.90th=[ 498], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 8192, max= 8192, per=80.70%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.26%, sys=3.75%, ctx=31, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21649: Thu May 1 20:55:19 2025 read: IOPS=0, BW=2538KiB/s (2599kB/s)(4096KiB/1614msec) clat (nsec): min=676611k, max=676611k, avg=676611317.00, stdev= 0.00 lat (nsec): min=676615k, max=676615k, avg=676614524.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 676], 5.00th=[ 676], 10.00th=[ 676], 20.00th=[ 676], | 30.00th=[ 676], 40.00th=[ 676], 50.00th=[ 676], 60.00th=[ 676], | 70.00th=[ 676], 80.00th=[ 676], 90.00th=[ 676], 95.00th=[ 676], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 8175, max= 8175, per=80.53%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5076KiB/s (5197kB/s)(8192KiB/1614msec) clat (msec): min=317, max=448, avg=382.81, stdev=92.79 lat (msec): min=317, max=448, avg=383.31, stdev=92.83 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 317], 10.00th=[ 317], 20.00th=[ 317], | 30.00th=[ 317], 40.00th=[ 317], 50.00th=[ 317], 60.00th=[ 447], | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.00%, sys=5.33%, ctx=57, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21651: Thu May 1 20:55:19 2025 write: IOPS=2, BW=9.84MiB/s (10.3MB/s)(12.0MiB/1220msec) clat (msec): min=303, max=506, avg=392.01, stdev=103.86 lat (msec): min=304, max=507, avg=392.70, stdev=103.96 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 305], 10.00th=[ 305], 20.00th=[ 305], | 30.00th=[ 305], 40.00th=[ 368], 50.00th=[ 368], 60.00th=[ 368], | 70.00th=[ 506], 80.00th=[ 506], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 8126, max= 8143, per=40.07%, avg=8134.50, stdev=12.02, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.33%, sys=3.77%, ctx=19, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21653: Thu May 1 20:55:19 2025 write: IOPS=2, BW=9745KiB/s (9979kB/s)(12.0MiB/1261msec) clat (msec): min=317, max=462, avg=378.82, stdev=74.89 lat (msec): min=317, max=462, avg=379.20, stdev=74.81 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 317], 10.00th=[ 317], 20.00th=[ 317], | 30.00th=[ 317], 40.00th=[ 359], 50.00th=[ 359], 60.00th=[ 359], | 70.00th=[ 464], 80.00th=[ 464], 90.00th=[ 464], 95.00th=[ 464], | 99.00th=[ 464], 99.50th=[ 464], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min=16189, max=16189, per=79.74%, avg=16189.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 500=100.00% cpu : usr=0.00%, sys=8.65%, ctx=36, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9.91MiB/s (10.4MB/s), 2538KiB/s-10.2MiB/s (2599kB/s-10.7MB/s), io=16.0MiB (16.8MB), run=1173-1614msec WRITE: bw=19.8MiB/s (20.8MB/s), 5076KiB/s-9.84MiB/s (5197kB/s-10.3MB/s), io=32.0MiB (33.6MB), run=1220-1614msec