== sanity test 398b: DIO and buffer IO race ============== 00:27:19 (1746160039) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.38618 s, 21.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20477: Fri May 2 00:28:10 2025 read: IOPS=33, BW=135KiB/s (138kB/s)(5992KiB/44416msec) clat (usec): min=74, max=255851, avg=16592.73, stdev=12121.11 lat (usec): min=75, max=255854, avg=16594.51, stdev=12121.35 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[ 119], 10.00th=[ 5735], 20.00th=[ 8586], | 30.00th=[ 10421], 40.00th=[ 12518], 50.00th=[ 14746], 60.00th=[ 17433], | 70.00th=[ 20317], 80.00th=[ 23987], 90.00th=[ 29492], 95.00th=[ 35390], | 99.00th=[ 49021], 99.50th=[ 56361], 99.90th=[ 88605], 99.95th=[254804], | 99.99th=[254804] bw ( KiB/s): min= 24, max= 288, per=25.26%, avg=133.61, stdev=37.86, samples=88 iops : min= 6, max= 72, avg=33.08, stdev= 9.46, samples=88 write: IOPS=35, BW=142KiB/s (145kB/s)(6296KiB/44416msec) clat (usec): min=372, max=85712, avg=12246.33, stdev=13640.60 lat (usec): min=373, max=85714, avg=12252.45, stdev=13638.13 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 742], 10.00th=[ 799], 20.00th=[ 930], | 30.00th=[ 1647], 40.00th=[ 3490], 50.00th=[ 8094], 60.00th=[11731], | 70.00th=[15795], 80.00th=[21627], 90.00th=[31327], 95.00th=[39060], | 99.00th=[62129], 99.50th=[69731], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 55, max= 288, per=25.54%, avg=141.48, stdev=43.46, samples=87 iops : min= 13, max= 72, avg=35.05, stdev=10.91, samples=87 lat (usec) : 100=1.11%, 250=2.34%, 500=0.10%, 750=2.86%, 1000=8.76% lat (msec) : 2=5.27%, 4=4.72%, 10=16.47%, 20=31.93%, 50=25.00% lat (msec) : 100=1.40%, 500=0.03% cpu : usr=0.15%, sys=10.22%, ctx=4830, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20478: Fri May 2 00:28:10 2025 read: IOPS=32, BW=131KiB/s (135kB/s)(5940KiB/45206msec) clat (usec): min=79, max=78317, avg=16304.46, stdev=10351.47 lat (usec): min=80, max=78318, avg=16306.18, stdev=10351.51 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 123], 10.00th=[ 5932], 20.00th=[ 8455], | 30.00th=[10552], 40.00th=[12387], 50.00th=[14484], 60.00th=[16909], | 70.00th=[20055], 80.00th=[23987], 90.00th=[29754], 95.00th=[34341], | 99.00th=[48497], 99.50th=[58983], 99.90th=[71828], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 8, max= 248, per=24.65%, avg=130.40, stdev=40.47, samples=90 iops : min= 2, max= 62, avg=32.24, stdev=10.14, samples=90 write: IOPS=35, BW=140KiB/s (144kB/s)(6348KiB/45206msec) clat (usec): min=558, max=332521, avg=13118.70, stdev=16551.54 lat (usec): min=559, max=332521, avg=13120.75, stdev=16551.28 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 750], 10.00th=[ 783], 20.00th=[ 963], | 30.00th=[ 1844], 40.00th=[ 3949], 50.00th=[ 8586], 60.00th=[ 12256], | 70.00th=[ 16712], 80.00th=[ 22414], 90.00th=[ 32637], 95.00th=[ 43779], | 99.00th=[ 60556], 99.50th=[ 71828], 99.90th=[116917], 99.95th=[333448], | 99.99th=[333448] bw ( KiB/s): min= 24, max= 334, per=25.07%, avg=138.90, stdev=49.00, samples=90 iops : min= 6, max= 83, avg=34.38, stdev=12.25, samples=90 lat (usec) : 100=0.75%, 250=2.64%, 500=0.13%, 750=2.77%, 1000=8.24% lat (msec) : 2=5.14%, 4=4.82%, 10=16.37%, 20=32.26%, 50=25.07% lat (msec) : 100=1.73%, 250=0.07%, 500=0.03% cpu : usr=0.15%, sys=10.31%, ctx=4907, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20480: Fri May 2 00:28:10 2025 read: IOPS=33, BW=133KiB/s (136kB/s)(6028KiB/45379msec) clat (usec): min=72, max=351677, avg=17085.13, stdev=14197.75 lat (usec): min=72, max=351679, avg=17086.51, stdev=14197.67 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 126], 10.00th=[ 5669], 20.00th=[ 8717], | 30.00th=[ 10421], 40.00th=[ 12518], 50.00th=[ 14877], 60.00th=[ 17433], | 70.00th=[ 20055], 80.00th=[ 24511], 90.00th=[ 30802], 95.00th=[ 37487], | 99.00th=[ 56361], 99.50th=[ 67634], 99.90th=[ 85459], 99.95th=[350225], | 99.99th=[350225] bw ( KiB/s): min= 16, max= 231, per=24.93%, avg=131.89, stdev=37.43, samples=90 iops : min= 4, max= 57, avg=32.61, stdev= 9.35, samples=90 write: IOPS=34, BW=138KiB/s (141kB/s)(6260KiB/45379msec) clat (usec): min=467, max=98170, avg=12419.33, stdev=13309.47 lat (usec): min=468, max=98171, avg=12421.17, stdev=13309.56 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 750], 10.00th=[ 791], 20.00th=[ 914], | 30.00th=[ 1745], 40.00th=[ 3884], 50.00th=[ 8717], 60.00th=[12125], | 70.00th=[16188], 80.00th=[21890], 90.00th=[32113], 95.00th=[38536], | 99.00th=[55837], 99.50th=[63177], 99.90th=[79168], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 24, max= 264, per=24.68%, avg=136.70, stdev=43.98, samples=90 iops : min= 6, max= 66, avg=33.81, stdev=10.95, samples=90 lat (usec) : 100=0.88%, 250=2.38%, 500=0.16%, 750=2.60%, 1000=8.89% lat (msec) : 2=4.79%, 4=4.62%, 10=16.37%, 20=32.36%, 50=25.23% lat (msec) : 100=1.69%, 500=0.03% cpu : usr=0.11%, sys=10.38%, ctx=4980, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20481: Fri May 2 00:28:10 2025 read: IOPS=33, BW=134KiB/s (137kB/s)(6052KiB/45173msec) clat (usec): min=71, max=73812, avg=17129.12, stdev=11243.99 lat (usec): min=71, max=73813, avg=17132.64, stdev=11242.59 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 126], 10.00th=[ 5800], 20.00th=[ 8586], | 30.00th=[10552], 40.00th=[12780], 50.00th=[15008], 60.00th=[17695], | 70.00th=[20841], 80.00th=[25297], 90.00th=[31851], 95.00th=[37487], | 99.00th=[55837], 99.50th=[60556], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 56, max= 271, per=25.47%, avg=134.73, stdev=40.23, samples=89 iops : min= 14, max= 67, avg=33.29, stdev=10.11, samples=89 write: IOPS=34, BW=138KiB/s (141kB/s)(6236KiB/45173msec) clat (usec): min=536, max=358253, avg=12187.09, stdev=16072.16 lat (usec): min=537, max=358257, avg=12188.43, stdev=16072.21 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 750], 10.00th=[ 799], 20.00th=[ 906], | 30.00th=[ 1942], 40.00th=[ 3687], 50.00th=[ 8094], 60.00th=[ 11731], | 70.00th=[ 15139], 80.00th=[ 20841], 90.00th=[ 30540], 95.00th=[ 38011], | 99.00th=[ 62653], 99.50th=[ 71828], 99.90th=[ 99091], 99.95th=[358613], | 99.99th=[358613] bw ( KiB/s): min= 63, max= 279, per=25.03%, avg=138.64, stdev=42.29, samples=89 iops : min= 15, max= 69, avg=34.28, stdev=10.63, samples=89 lat (usec) : 100=0.98%, 250=2.41%, 500=0.13%, 750=2.47%, 1000=9.18% lat (msec) : 2=4.07%, 4=5.40%, 10=16.63%, 20=32.23%, 50=24.67% lat (msec) : 100=1.79%, 500=0.03% cpu : usr=0.16%, sys=10.20%, ctx=5027, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=529KiB/s (542kB/s), 131KiB/s-135KiB/s (135kB/s-138kB/s), io=23.4MiB (24.6MB), run=44416-45379msec WRITE: bw=554KiB/s (567kB/s), 138KiB/s-142KiB/s (141kB/s-145kB/s), io=24.6MiB (25.7MB), run=44416-45379msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20482: Fri May 2 00:28:28 2025 read: IOPS=23, BW=95.4KiB/s (97.7kB/s)(5992KiB/62798msec) clat (msec): min=3, max=351, avg=15.44, stdev=16.63 lat (msec): min=3, max=351, avg=15.44, stdev=16.63 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 15], 80.00th=[ 21], 90.00th=[ 35], 95.00th=[ 48], | 99.00th=[ 70], 99.50th=[ 81], 99.90th=[ 112], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 8, max= 256, per=24.99%, avg=94.46, stdev=57.28, samples=125 iops : min= 2, max= 64, avg=23.28, stdev=14.41, samples=125 write: IOPS=25, BW=100KiB/s (103kB/s)(6296KiB/62798msec) clat (msec): min=4, max=127, avg=25.12, stdev=17.17 lat (msec): min=4, max=127, avg=25.12, stdev=17.18 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 19], 60.00th=[ 22], | 70.00th=[ 29], 80.00th=[ 37], 90.00th=[ 50], 95.00th=[ 61], | 99.00th=[ 88], 99.50th=[ 101], 99.90th=[ 118], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 15, max= 248, per=25.25%, avg=100.00, stdev=56.60, samples=124 iops : min= 3, max= 62, avg=24.67, stdev=14.29, samples=124 lat (msec) : 4=0.55%, 10=27.77%, 20=38.22%, 50=26.50%, 100=6.64% lat (msec) : 250=0.29%, 500=0.03% cpu : usr=0.15%, sys=4.85%, ctx=3443, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20483: Fri May 2 00:28:28 2025 read: IOPS=23, BW=93.9KiB/s (96.1kB/s)(5940KiB/63288msec) clat (msec): min=2, max=367, avg=15.94, stdev=17.84 lat (msec): min=2, max=367, avg=15.95, stdev=17.84 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 15], 80.00th=[ 22], 90.00th=[ 36], 95.00th=[ 50], | 99.00th=[ 78], 99.50th=[ 97], 99.90th=[ 118], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 7, max= 264, per=24.62%, avg=93.07, stdev=57.17, samples=126 iops : min= 1, max= 66, avg=22.89, stdev=14.41, samples=126 write: IOPS=25, BW=100KiB/s (103kB/s)(6348KiB/63288msec) clat (msec): min=6, max=140, avg=24.83, stdev=16.41 lat (msec): min=6, max=140, avg=24.84, stdev=16.41 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 23], | 70.00th=[ 29], 80.00th=[ 36], 90.00th=[ 48], 95.00th=[ 58], | 99.00th=[ 86], 99.50th=[ 97], 99.90th=[ 113], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 23, max= 240, per=25.24%, avg=99.97, stdev=55.54, samples=125 iops : min= 5, max= 60, avg=24.62, stdev=13.99, samples=125 lat (msec) : 4=0.78%, 10=26.86%, 20=38.35%, 50=27.28%, 100=6.35% lat (msec) : 250=0.36%, 500=0.03% cpu : usr=0.14%, sys=4.81%, ctx=3488, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20484: Fri May 2 00:28:28 2025 read: IOPS=23, BW=95.3KiB/s (97.6kB/s)(6028KiB/63256msec) clat (msec): min=3, max=130, avg=15.75, stdev=15.08 lat (msec): min=3, max=130, avg=15.75, stdev=15.08 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 16], 80.00th=[ 22], 90.00th=[ 36], 95.00th=[ 49], | 99.00th=[ 71], 99.50th=[ 84], 99.90th=[ 105], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 23, max= 280, per=25.22%, avg=95.34, stdev=57.38, samples=125 iops : min= 5, max= 70, avg=23.54, stdev=14.39, samples=125 write: IOPS=24, BW=98.0KiB/s (101kB/s)(6260KiB/63256msec) clat (msec): min=5, max=303, avg=25.09, stdev=18.26 lat (msec): min=5, max=303, avg=25.09, stdev=18.27 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 23], | 70.00th=[ 28], 80.00th=[ 36], 90.00th=[ 50], 95.00th=[ 60], | 99.00th=[ 86], 99.50th=[ 104], 99.90th=[ 127], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 7, max= 232, per=24.78%, avg=98.13, stdev=56.81, samples=126 iops : min= 1, max= 58, avg=24.24, stdev=14.26, samples=126 lat (msec) : 4=0.46%, 10=27.60%, 20=38.12%, 50=26.82%, 100=6.54% lat (msec) : 250=0.42%, 500=0.03% cpu : usr=0.08%, sys=4.81%, ctx=3453, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20485: Fri May 2 00:28:28 2025 read: IOPS=23, BW=95.3KiB/s (97.6kB/s)(6052KiB/63474msec) clat (msec): min=3, max=118, avg=15.71, stdev=15.17 lat (msec): min=3, max=118, avg=15.71, stdev=15.17 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 16], 80.00th=[ 22], 90.00th=[ 36], 95.00th=[ 48], | 99.00th=[ 77], 99.50th=[ 87], 99.90th=[ 106], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 16, max= 272, per=25.01%, avg=94.55, stdev=57.66, samples=125 iops : min= 4, max= 68, avg=23.28, stdev=14.42, samples=125 write: IOPS=24, BW=98.2KiB/s (101kB/s)(6236KiB/63474msec) clat (msec): min=4, max=390, avg=25.32, stdev=19.40 lat (msec): min=4, max=390, avg=25.32, stdev=19.40 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 19], 60.00th=[ 23], | 70.00th=[ 29], 80.00th=[ 38], 90.00th=[ 50], 95.00th=[ 62], | 99.00th=[ 86], 99.50th=[ 96], 99.90th=[ 140], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 31, max= 240, per=24.68%, avg=97.75, stdev=56.49, samples=125 iops : min= 7, max= 60, avg=24.08, stdev=14.13, samples=125 lat (msec) : 4=0.52%, 10=28.26%, 20=37.63%, 50=26.50%, 100=6.80% lat (msec) : 250=0.26%, 500=0.03% cpu : usr=0.12%, sys=4.78%, ctx=3499, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=378KiB/s (387kB/s), 93.9KiB/s-95.4KiB/s (96.1kB/s-97.7kB/s), io=23.4MiB (24.6MB), run=62798-63474msec WRITE: bw=396KiB/s (406kB/s), 98.2KiB/s-100KiB/s (101kB/s-103kB/s), io=24.6MiB (25.7MB), run=62798-63474msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20551: Fri May 2 00:28:39 2025 read: IOPS=38, BW=624KiB/s (639kB/s)(5920KiB/9492msec) clat (usec): min=77, max=68395, avg=13431.64, stdev=10650.50 lat (usec): min=78, max=68396, avg=13432.64, stdev=10650.51 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 104], 10.00th=[ 120], 20.00th=[ 5669], | 30.00th=[ 8455], 40.00th=[10159], 50.00th=[12387], 60.00th=[13829], | 70.00th=[16712], 80.00th=[20055], 90.00th=[25560], 95.00th=[30540], | 99.00th=[53216], 99.50th=[65274], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 318, max= 928, per=24.38%, avg=606.50, stdev=167.49, samples=18 iops : min= 19, max= 58, avg=37.56, stdev=10.58, samples=18 write: IOPS=41, BW=671KiB/s (687kB/s)(6368KiB/9492msec) clat (usec): min=810, max=57921, avg=11020.85, stdev=11823.80 lat (usec): min=811, max=57922, avg=11022.56, stdev=11823.78 clat percentiles (usec): | 1.00th=[ 1020], 5.00th=[ 1123], 10.00th=[ 1205], 20.00th=[ 1549], | 30.00th=[ 2311], 40.00th=[ 3720], 50.00th=[ 5932], 60.00th=[ 9896], | 70.00th=[14222], 80.00th=[19006], 90.00th=[28705], 95.00th=[35914], | 99.00th=[53740], 99.50th=[56886], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 256, max= 1024, per=25.13%, avg=657.78, stdev=193.01, samples=18 iops : min= 16, max= 64, avg=40.78, stdev=11.94, samples=18 lat (usec) : 100=2.08%, 250=5.73%, 500=0.26%, 750=0.13%, 1000=0.39% lat (msec) : 2=14.45%, 4=6.90%, 10=20.31%, 20=30.34%, 50=17.84% lat (msec) : 100=1.56% cpu : usr=0.18%, sys=12.85%, ctx=1251, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20552: Fri May 2 00:28:39 2025 read: IOPS=37, BW=597KiB/s (611kB/s)(5744KiB/9625msec) clat (usec): min=79, max=62470, avg=13366.18, stdev=10558.45 lat (usec): min=80, max=62471, avg=13367.23, stdev=10558.47 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 110], 10.00th=[ 121], 20.00th=[ 4555], | 30.00th=[ 7242], 40.00th=[ 9503], 50.00th=[11731], 60.00th=[15008], | 70.00th=[17171], 80.00th=[21627], 90.00th=[27395], 95.00th=[31327], | 99.00th=[46400], 99.50th=[61080], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 320, max= 864, per=23.70%, avg=589.58, stdev=160.66, samples=19 iops : min= 20, max= 54, avg=36.68, stdev=10.12, samples=19 write: IOPS=42, BW=680KiB/s (696kB/s)(6544KiB/9625msec) clat (usec): min=697, max=86217, avg=11436.55, stdev=12987.56 lat (usec): min=698, max=86219, avg=11438.25, stdev=12987.58 clat percentiles (usec): | 1.00th=[ 881], 5.00th=[ 1090], 10.00th=[ 1205], 20.00th=[ 1483], | 30.00th=[ 2474], 40.00th=[ 3785], 50.00th=[ 6063], 60.00th=[ 9110], | 70.00th=[13304], 80.00th=[21103], 90.00th=[28181], 95.00th=[36963], | 99.00th=[58459], 99.50th=[65274], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 414, max= 1073, per=25.72%, avg=673.37, stdev=197.12, samples=19 iops : min= 25, max= 67, avg=41.95, stdev=12.43, samples=19 lat (usec) : 100=0.65%, 250=6.77%, 500=0.52%, 750=0.39%, 1000=0.78% lat (msec) : 2=12.89%, 4=8.85%, 10=22.27%, 20=24.35%, 50=21.35% lat (msec) : 100=1.17% cpu : usr=0.09%, sys=12.61%, ctx=1306, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20553: Fri May 2 00:28:39 2025 read: IOPS=41, BW=658KiB/s (674kB/s)(6256KiB/9506msec) clat (usec): min=76, max=60696, avg=13676.13, stdev=9938.00 lat (usec): min=77, max=60697, avg=13677.18, stdev=9937.99 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 105], 10.00th=[ 126], 20.00th=[ 6194], | 30.00th=[ 8029], 40.00th=[10159], 50.00th=[11994], 60.00th=[14353], | 70.00th=[16909], 80.00th=[21627], 90.00th=[27657], 95.00th=[32375], | 99.00th=[40633], 99.50th=[43779], 99.90th=[60556], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min= 349, max= 1088, per=26.32%, avg=654.78, stdev=196.41, samples=18 iops : min= 21, max= 68, avg=40.44, stdev=12.57, samples=18 write: IOPS=39, BW=635KiB/s (650kB/s)(6032KiB/9506msec) clat (usec): min=796, max=107007, avg=10914.05, stdev=12210.69 lat (usec): min=797, max=107009, avg=10918.49, stdev=12215.90 clat percentiles (usec): | 1.00th=[ 922], 5.00th=[ 1106], 10.00th=[ 1188], 20.00th=[ 1483], | 30.00th=[ 2114], 40.00th=[ 3687], 50.00th=[ 5800], 60.00th=[ 10159], | 70.00th=[ 14353], 80.00th=[ 19530], 90.00th=[ 27132], 95.00th=[ 34341], | 99.00th=[ 54789], 99.50th=[ 60031], 99.90th=[107480], 99.95th=[107480], | 99.99th=[107480] bw ( KiB/s): min= 223, max= 1149, per=23.45%, avg=613.83, stdev=204.62, samples=18 iops : min= 13, max= 71, avg=37.89, stdev=12.92, samples=18 lat (usec) : 100=1.43%, 250=5.60%, 500=0.26%, 750=0.13%, 1000=1.04% lat (msec) : 2=13.02%, 4=7.29%, 10=20.70%, 20=29.30%, 50=20.57% lat (msec) : 100=0.52%, 250=0.13% cpu : usr=0.21%, sys=12.96%, ctx=1351, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20554: Fri May 2 00:28:39 2025 read: IOPS=41, BW=656KiB/s (672kB/s)(6032KiB/9194msec) clat (usec): min=77, max=97108, avg=13212.83, stdev=10757.25 lat (usec): min=77, max=97110, avg=13213.84, stdev=10757.29 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 106], 10.00th=[ 117], 20.00th=[ 5145], | 30.00th=[ 8225], 40.00th=[10290], 50.00th=[12256], 60.00th=[14091], | 70.00th=[16712], 80.00th=[19530], 90.00th=[25560], 95.00th=[31851], | 99.00th=[43254], 99.50th=[74974], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 446, max= 928, per=25.89%, avg=644.22, stdev=142.76, samples=18 iops : min= 27, max= 58, avg=39.89, stdev= 9.04, samples=18 write: IOPS=42, BW=680KiB/s (697kB/s)(6256KiB/9194msec) clat (usec): min=907, max=69967, avg=10348.69, stdev=10954.07 lat (usec): min=909, max=69968, avg=10350.46, stdev=10953.99 clat percentiles (usec): | 1.00th=[ 947], 5.00th=[ 1106], 10.00th=[ 1172], 20.00th=[ 1434], | 30.00th=[ 2147], 40.00th=[ 3720], 50.00th=[ 5473], 60.00th=[ 9110], | 70.00th=[14353], 80.00th=[19268], 90.00th=[25560], 95.00th=[32637], | 99.00th=[47449], 99.50th=[55837], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 380, max= 990, per=25.83%, avg=676.33, stdev=162.73, samples=18 iops : min= 23, max= 61, avg=41.89, stdev=10.35, samples=18 lat (usec) : 100=1.30%, 250=7.03%, 500=0.39%, 750=0.13%, 1000=1.04% lat (msec) : 2=13.41%, 4=7.42%, 10=19.66%, 20=30.34%, 50=18.75% lat (msec) : 100=0.52% cpu : usr=0.27%, sys=12.88%, ctx=1297, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2489KiB/s (2548kB/s), 597KiB/s-658KiB/s (611kB/s-674kB/s), io=23.4MiB (24.5MB), run=9194-9625msec WRITE: bw=2618KiB/s (2681kB/s), 635KiB/s-680KiB/s (650kB/s-697kB/s), io=24.6MiB (25.8MB), run=9194-9625msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20555: Fri May 2 00:28:44 2025 read: IOPS=25, BW=410KiB/s (420kB/s)(5920KiB/14444msec) clat (msec): min=3, max=121, avg=14.86, stdev=14.78 lat (msec): min=3, max=121, avg=14.86, stdev=14.78 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 8], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 14], 80.00th=[ 20], 90.00th=[ 33], 95.00th=[ 51], | 99.00th=[ 67], 99.50th=[ 90], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 160, max= 1021, per=24.03%, avg=398.39, stdev=215.13, samples=28 iops : min= 10, max= 63, avg=24.64, stdev=13.32, samples=28 write: IOPS=27, BW=441KiB/s (451kB/s)(6368KiB/14444msec) clat (msec): min=5, max=101, avg=22.29, stdev=13.85 lat (msec): min=5, max=101, avg=22.29, stdev=13.85 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 9], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 21], | 70.00th=[ 24], 80.00th=[ 31], 90.00th=[ 41], 95.00th=[ 53], | 99.00th=[ 71], 99.50th=[ 80], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 64, max= 832, per=24.61%, avg=429.25, stdev=232.67, samples=28 iops : min= 4, max= 52, avg=26.57, stdev=14.46, samples=28 lat (msec) : 4=0.91%, 10=30.34%, 20=37.76%, 50=25.65%, 100=5.08% lat (msec) : 250=0.26% cpu : usr=0.03%, sys=5.51%, ctx=899, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20556: Fri May 2 00:28:44 2025 read: IOPS=25, BW=403KiB/s (413kB/s)(5744KiB/14249msec) clat (msec): min=3, max=128, avg=13.86, stdev=14.48 lat (msec): min=3, max=128, avg=13.86, stdev=14.49 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 7], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 17], 90.00th=[ 30], 95.00th=[ 45], | 99.00th=[ 72], 99.50th=[ 87], 99.90th=[ 129], 99.95th=[ 129], | 99.99th=[ 129] bw ( KiB/s): min= 96, max= 928, per=24.11%, avg=399.71, stdev=243.79, samples=28 iops : min= 6, max= 58, avg=24.75, stdev=15.29, samples=28 write: IOPS=28, BW=459KiB/s (470kB/s)(6544KiB/14249msec) clat (usec): min=5168, max=90895, avg=22395.99, stdev=14140.79 lat (usec): min=5169, max=90898, avg=22400.83, stdev=14140.39 clat percentiles (usec): | 1.00th=[ 7635], 5.00th=[ 9110], 10.00th=[10421], 20.00th=[12125], | 30.00th=[13698], 40.00th=[15795], 50.00th=[17433], 60.00th=[20317], | 70.00th=[24249], 80.00th=[30278], 90.00th=[41157], 95.00th=[51643], | 99.00th=[70779], 99.50th=[73925], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 160, max= 928, per=25.93%, avg=452.25, stdev=221.48, samples=28 iops : min= 10, max= 58, avg=28.04, stdev=13.91, samples=28 lat (msec) : 4=0.91%, 10=31.38%, 20=38.15%, 50=24.74%, 100=4.69% lat (msec) : 250=0.13% cpu : usr=0.00%, sys=5.66%, ctx=899, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20557: Fri May 2 00:28:44 2025 read: IOPS=27, BW=436KiB/s (447kB/s)(6256KiB/14343msec) clat (msec): min=3, max=102, avg=14.28, stdev=14.05 lat (msec): min=3, max=102, avg=14.28, stdev=14.05 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 7], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 19], 90.00th=[ 31], 95.00th=[ 44], | 99.00th=[ 72], 99.50th=[ 95], 99.90th=[ 103], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min= 191, max= 1120, per=26.16%, avg=433.79, stdev=268.41, samples=28 iops : min= 11, max= 70, avg=26.75, stdev=16.89, samples=28 write: IOPS=26, BW=421KiB/s (431kB/s)(6032KiB/14343msec) clat (usec): min=6322, max=90587, avg=23131.79, stdev=15407.75 lat (usec): min=6324, max=90589, avg=23133.86, stdev=15407.72 clat percentiles (usec): | 1.00th=[ 7570], 5.00th=[ 8979], 10.00th=[10028], 20.00th=[11994], | 30.00th=[13829], 40.00th=[15401], 50.00th=[17171], 60.00th=[20317], | 70.00th=[24249], 80.00th=[33162], 90.00th=[44827], 95.00th=[58459], | 99.00th=[81265], 99.50th=[89654], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 128, max= 832, per=23.37%, avg=407.54, stdev=215.06, samples=28 iops : min= 8, max= 52, avg=25.11, stdev=13.55, samples=28 lat (msec) : 4=1.43%, 10=33.46%, 20=35.16%, 50=24.09%, 100=5.73% lat (msec) : 250=0.13% cpu : usr=0.08%, sys=5.41%, ctx=906, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20558: Fri May 2 00:28:44 2025 read: IOPS=26, BW=421KiB/s (431kB/s)(6032KiB/14332msec) clat (usec): min=3375, max=86401, avg=14502.37, stdev=13785.49 lat (usec): min=3376, max=86402, avg=14503.99, stdev=13785.33 clat percentiles (usec): | 1.00th=[ 3884], 5.00th=[ 4621], 10.00th=[ 5145], 20.00th=[ 5866], | 30.00th=[ 6652], 40.00th=[ 7832], 50.00th=[ 9110], 60.00th=[10683], | 70.00th=[13960], 80.00th=[19792], 90.00th=[31851], 95.00th=[51119], | 99.00th=[70779], 99.50th=[72877], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 191, max= 992, per=24.72%, avg=409.89, stdev=215.79, samples=28 iops : min= 11, max= 62, avg=25.32, stdev=13.54, samples=28 write: IOPS=27, BW=437KiB/s (447kB/s)(6256KiB/14332msec) clat (msec): min=5, max=100, avg=22.26, stdev=14.11 lat (msec): min=5, max=100, avg=22.26, stdev=14.11 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 21], | 70.00th=[ 24], 80.00th=[ 32], 90.00th=[ 42], 95.00th=[ 51], | 99.00th=[ 66], 99.50th=[ 100], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 159, max= 960, per=24.55%, avg=428.18, stdev=234.40, samples=28 iops : min= 9, max= 60, avg=26.46, stdev=14.74, samples=28 lat (msec) : 4=0.65%, 10=30.47%, 20=39.06%, 50=24.74%, 100=4.95% lat (msec) : 250=0.13% cpu : usr=0.13%, sys=5.55%, ctx=887, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1658KiB/s (1698kB/s), 403KiB/s-436KiB/s (413kB/s-447kB/s), io=23.4MiB (24.5MB), run=14249-14444msec WRITE: bw=1745KiB/s (1787kB/s), 421KiB/s-459KiB/s (431kB/s-470kB/s), io=24.6MiB (25.8MB), run=14249-14444msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20576: Fri May 2 00:28:47 2025 read: IOPS=6, BW=6674KiB/s (6834kB/s)(7168KiB/1074msec) clat (msec): min=14, max=130, avg=49.90, stdev=44.46 lat (msec): min=14, max=130, avg=49.90, stdev=44.46 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 21], | 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 24], 60.00th=[ 48], | 70.00th=[ 48], 80.00th=[ 91], 90.00th=[ 131], 95.00th=[ 131], | 99.00th=[ 131], 99.50th=[ 131], 99.90th=[ 131], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min=10240, max=10240, per=74.15%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4767KiB/s (4882kB/s)(5120KiB/1074msec) clat (msec): min=37, max=261, avg=140.43, stdev=86.44 lat (msec): min=37, max=261, avg=140.51, stdev=86.44 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 38], | 30.00th=[ 88], 40.00th=[ 88], 50.00th=[ 134], 60.00th=[ 134], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 262], 95.00th=[ 262], | 99.00th=[ 262], 99.50th=[ 262], 99.90th=[ 262], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 2048, max= 6131, per=21.15%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (msec) : 20=8.33%, 50=41.67%, 100=16.67%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=6.90%, ctx=44, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20578: Fri May 2 00:28:47 2025 read: IOPS=1, BW=1381KiB/s (1414kB/s)(2048KiB/1483msec) clat (usec): min=18908, max=86451, avg=52679.65, stdev=47760.26 lat (usec): min=18909, max=86453, avg=52681.75, stdev=47760.64 clat percentiles (usec): | 1.00th=[19006], 5.00th=[19006], 10.00th=[19006], 20.00th=[19006], | 30.00th=[19006], 40.00th=[19006], 50.00th=[19006], 60.00th=[86508], | 70.00th=[86508], 80.00th=[86508], 90.00th=[86508], 95.00th=[86508], | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 2015, max= 2048, per=14.71%, avg=2031.50, stdev=23.33, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=6, BW=6905KiB/s (7071kB/s)(10.0MiB/1483msec) clat (msec): min=47, max=181, avg=126.96, stdev=47.23 lat (msec): min=47, max=181, avg=127.04, stdev=47.23 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 64], | 30.00th=[ 91], 40.00th=[ 121], 50.00th=[ 131], 60.00th=[ 131], | 70.00th=[ 165], 80.00th=[ 165], 90.00th=[ 178], 95.00th=[ 182], | 99.00th=[ 182], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 2015, max= 8192, per=26.40%, avg=5103.50, stdev=4367.80, samples=2 iops : min= 1, max= 8, avg= 4.50, stdev= 4.95, samples=2 lat (msec) : 20=8.33%, 50=8.33%, 100=25.00%, 250=58.33% cpu : usr=0.20%, sys=4.25%, ctx=64, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20580: Fri May 2 00:28:47 2025 read: IOPS=4, BW=4174KiB/s (4274kB/s)(6144KiB/1472msec) clat (msec): min=30, max=222, avg=138.85, stdev=75.81 lat (msec): min=30, max=222, avg=138.86, stdev=75.81 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 72], | 30.00th=[ 72], 40.00th=[ 128], 50.00th=[ 128], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 197], 90.00th=[ 224], 95.00th=[ 224], | 99.00th=[ 224], 99.50th=[ 224], 99.90th=[ 224], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 2043, max= 2048, per=14.81%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4174KiB/s (4274kB/s)(6144KiB/1472msec) clat (msec): min=24, max=156, avg=78.22, stdev=47.29 lat (msec): min=24, max=156, avg=78.32, stdev=47.30 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 44], | 30.00th=[ 44], 40.00th=[ 59], 50.00th=[ 59], 60.00th=[ 86], | 70.00th=[ 102], 80.00th=[ 102], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 2043, max= 6144, per=21.17%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (msec) : 50=25.00%, 100=25.00%, 250=50.00% cpu : usr=0.00%, sys=3.74%, ctx=48, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20581: Fri May 2 00:28:47 2025 read: IOPS=4, BW=4193KiB/s (4294kB/s)(5120KiB/1221msec) clat (msec): min=16, max=173, avg=53.19, stdev=67.24 lat (msec): min=16, max=173, avg=53.19, stdev=67.24 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 29], 60.00th=[ 29], | 70.00th=[ 30], 80.00th=[ 30], 90.00th=[ 174], 95.00th=[ 174], | 99.00th=[ 174], 99.50th=[ 174], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 8175, max= 8175, per=59.20%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5871KiB/s (6011kB/s)(7168KiB/1221msec) clat (msec): min=35, max=298, avg=132.66, stdev=88.03 lat (msec): min=35, max=298, avg=132.76, stdev=88.01 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 39], | 30.00th=[ 128], 40.00th=[ 128], 50.00th=[ 133], 60.00th=[ 140], | 70.00th=[ 140], 80.00th=[ 159], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 2043, max= 8192, per=26.47%, avg=5117.50, stdev=4348.00, samples=2 iops : min= 1, max= 8, avg= 4.50, stdev= 4.95, samples=2 lat (msec) : 20=16.67%, 50=33.33%, 250=41.67%, 500=8.33% cpu : usr=0.00%, sys=5.25%, ctx=47, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.5MiB/s (14.1MB/s), 1381KiB/s-6674KiB/s (1414kB/s-6834kB/s), io=20.0MiB (20.0MB), run=1074-1483msec WRITE: bw=18.9MiB/s (19.8MB/s), 4174KiB/s-6905KiB/s (4274kB/s-7071kB/s), io=28.0MiB (29.4MB), run=1074-1483msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20577: Fri May 2 00:28:47 2025 read: IOPS=5, BW=5823KiB/s (5963kB/s)(7168KiB/1231msec) clat (usec): min=553, max=125653, avg=48285.34, stdev=44435.62 lat (usec): min=554, max=125654, avg=48286.79, stdev=44435.23 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 553], 10.00th=[ 553], 20.00th=[ 906], | 30.00th=[ 23462], 40.00th=[ 23462], 50.00th=[ 54789], 60.00th=[ 62129], | 70.00th=[ 62129], 80.00th=[ 70779], 90.00th=[125305], 95.00th=[125305], | 99.00th=[125305], 99.50th=[125305], 99.90th=[125305], 99.95th=[125305], | 99.99th=[125305] bw ( KiB/s): min= 2023, max= 8159, per=38.31%, avg=5091.00, stdev=4338.81, samples=2 iops : min= 1, max= 7, avg= 4.00, stdev= 4.24, samples=2 write: IOPS=4, BW=4159KiB/s (4259kB/s)(5120KiB/1231msec) clat (msec): min=71, max=263, avg=144.84, stdev=75.29 lat (msec): min=71, max=263, avg=144.91, stdev=75.32 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 72], 10.00th=[ 72], 20.00th=[ 72], | 30.00th=[ 89], 40.00th=[ 89], 50.00th=[ 142], 60.00th=[ 142], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 264], 95.00th=[ 264], | 99.00th=[ 264], 99.50th=[ 264], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 2039, max= 4047, per=16.35%, avg=3043.00, stdev=1419.87, samples=2 iops : min= 1, max= 3, avg= 2.00, stdev= 1.41, samples=2 lat (usec) : 750=8.33%, 1000=8.33% lat (msec) : 50=8.33%, 100=41.67%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=17.32%, ctx=400, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20579: Fri May 2 00:28:47 2025 read: IOPS=1, BW=1329KiB/s (1361kB/s)(2048KiB/1541msec) clat (usec): min=75637, max=79434, avg=77536.23, stdev=2685.18 lat (usec): min=75640, max=79436, avg=77538.57, stdev=2684.09 clat percentiles (usec): | 1.00th=[76022], 5.00th=[76022], 10.00th=[76022], 20.00th=[76022], | 30.00th=[76022], 40.00th=[76022], 50.00th=[76022], 60.00th=[79168], | 70.00th=[79168], 80.00th=[79168], 90.00th=[79168], 95.00th=[79168], | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 1906, max= 2043, per=14.86%, avg=1974.50, stdev=96.87, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=6, BW=6645KiB/s (6805kB/s)(10.0MiB/1541msec) clat (msec): min=40, max=248, avg=124.00, stdev=74.03 lat (msec): min=40, max=248, avg=124.08, stdev=74.03 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 46], | 30.00th=[ 54], 40.00th=[ 59], 50.00th=[ 136], 60.00th=[ 138], | 70.00th=[ 153], 80.00th=[ 153], 90.00th=[ 224], 95.00th=[ 249], | 99.00th=[ 249], 99.50th=[ 249], 99.90th=[ 249], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min= 2043, max= 8031, per=31.71%, avg=5900.33, stdev=3346.65, samples=3 iops : min= 1, max= 7, avg= 5.00, stdev= 3.46, samples=3 lat (msec) : 50=16.67%, 100=33.33%, 250=50.00% cpu : usr=0.00%, sys=18.11%, ctx=264, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20582: Fri May 2 00:28:47 2025 read: IOPS=7, BW=7169KiB/s (7341kB/s)(6144KiB/857msec) clat (usec): min=464, max=134400, avg=61453.71, stdev=67149.38 lat (usec): min=465, max=134401, avg=61455.11, stdev=67149.71 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 465], 10.00th=[ 465], 20.00th=[ 486], | 30.00th=[ 486], 40.00th=[ 586], 50.00th=[ 586], 60.00th=[111674], | 70.00th=[121111], 80.00th=[121111], 90.00th=[135267], 95.00th=[135267], | 99.00th=[135267], 99.50th=[135267], 99.90th=[135267], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 8175, max= 8175, per=61.51%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7169KiB/s (7341kB/s)(6144KiB/857msec) clat (msec): min=38, max=188, avg=75.64, stdev=56.52 lat (msec): min=38, max=188, avg=75.71, stdev=56.50 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 55], 50.00th=[ 55], 60.00th=[ 61], | 70.00th=[ 72], 80.00th=[ 72], 90.00th=[ 188], 95.00th=[ 188], | 99.00th=[ 188], 99.50th=[ 188], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 8175, max= 8175, per=43.94%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 50=16.67%, 100=25.00%, 250=33.33% cpu : usr=0.00%, sys=19.63%, ctx=215, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20583: Fri May 2 00:28:47 2025 read: IOPS=4, BW=4129KiB/s (4228kB/s)(5120KiB/1240msec) clat (usec): min=502, max=153376, avg=57976.96, stdev=63039.44 lat (usec): min=504, max=153377, avg=57983.92, stdev=63033.11 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 502], 10.00th=[ 502], 20.00th=[ 502], | 30.00th=[ 652], 40.00th=[ 652], 50.00th=[ 64750], 60.00th=[ 64750], | 70.00th=[ 70779], 80.00th=[ 70779], 90.00th=[154141], 95.00th=[154141], | 99.00th=[154141], 99.50th=[154141], 99.90th=[154141], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 8159, max= 8159, per=61.39%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5781KiB/s (5919kB/s)(7168KiB/1240msec) clat (msec): min=38, max=219, avg=116.96, stdev=63.73 lat (msec): min=38, max=219, avg=117.05, stdev=63.74 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 64], | 30.00th=[ 79], 40.00th=[ 79], 50.00th=[ 121], 60.00th=[ 126], | 70.00th=[ 126], 80.00th=[ 176], 90.00th=[ 220], 95.00th=[ 220], | 99.00th=[ 220], 99.50th=[ 220], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 4079, max= 6144, per=27.47%, avg=5111.50, stdev=1460.18, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 lat (usec) : 750=16.67% lat (msec) : 50=8.33%, 100=33.33%, 250=41.67% cpu : usr=0.00%, sys=21.47%, ctx=341, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.0MiB/s (13.6MB/s), 1329KiB/s-7169KiB/s (1361kB/s-7341kB/s), io=20.0MiB (20.0MB), run=857-1541msec WRITE: bw=18.2MiB/s (19.1MB/s), 4159KiB/s-7169KiB/s (4259kB/s-7341kB/s), io=28.0MiB (29.4MB), run=857-1541msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20593: Fri May 2 00:28:50 2025 read: IOPS=3, BW=12.0MiB/s (13.6MB/s)(12.0MiB/925msec) clat (msec): min=218, max=332, avg=291.20, stdev=63.22 lat (msec): min=218, max=332, avg=291.20, stdev=63.22 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 220], 10.00th=[ 220], 20.00th=[ 220], | 30.00th=[ 220], 40.00th=[ 321], 50.00th=[ 321], 60.00th=[ 321], | 70.00th=[ 334], 80.00th=[ 334], 90.00th=[ 334], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 8192, max= 8192, per=76.55%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=16.88%, ctx=663, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20595: Fri May 2 00:28:50 2025 read: IOPS=0, BW=2675KiB/s (2740kB/s)(4096KiB/1531msec) clat (nsec): min=292533k, max=292533k, avg=292533157.00, stdev= 0.00 lat (nsec): min=292536k, max=292536k, avg=292535935.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 292], 10.00th=[ 292], 20.00th=[ 292], | 30.00th=[ 292], 40.00th=[ 292], 50.00th=[ 292], 60.00th=[ 292], | 70.00th=[ 292], 80.00th=[ 292], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 8159, max= 8159, per=76.25%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5351KiB/s (5479kB/s)(8192KiB/1531msec) clat (msec): min=536, max=606, avg=571.47, stdev=49.39 lat (msec): min=536, max=606, avg=571.85, stdev=49.33 clat percentiles (msec): | 1.00th=[ 535], 5.00th=[ 535], 10.00th=[ 535], 20.00th=[ 535], | 30.00th=[ 535], 40.00th=[ 535], 50.00th=[ 535], 60.00th=[ 609], | 70.00th=[ 609], 80.00th=[ 609], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 8192, max= 8192, per=41.35%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.00%, sys=13.79%, ctx=414, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20597: Fri May 2 00:28:50 2025 write: IOPS=1, BW=7429KiB/s (7608kB/s)(12.0MiB/1654msec) clat (msec): min=404, max=562, avg=498.05, stdev=82.86 lat (msec): min=411, max=562, avg=500.51, stdev=79.40 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 405], 10.00th=[ 405], 20.00th=[ 405], | 30.00th=[ 405], 40.00th=[ 527], 50.00th=[ 527], 60.00th=[ 527], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 7025, max= 8175, per=38.36%, avg=7600.00, stdev=813.17, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.00%, sys=18.21%, ctx=88, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20598: Fri May 2 00:28:50 2025 write: IOPS=2, BW=8611KiB/s (8818kB/s)(12.0MiB/1427msec) clat (msec): min=310, max=613, avg=439.11, stdev=156.84 lat (msec): min=310, max=614, avg=439.34, stdev=156.91 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 309], 10.00th=[ 309], 20.00th=[ 309], | 30.00th=[ 309], 40.00th=[ 393], 50.00th=[ 393], 60.00th=[ 393], | 70.00th=[ 617], 80.00th=[ 617], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 8159, max= 8159, per=41.18%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.00%, sys=22.09%, ctx=74, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.5MiB/s (10.0MB/s), 2675KiB/s-12.0MiB/s (2740kB/s-13.6MB/s), io=16.0MiB (16.8MB), run=925-1531msec WRITE: bw=19.3MiB/s (20.3MB/s), 5351KiB/s-8611KiB/s (5479kB/s-8818kB/s), io=32.0MiB (33.6MB), run=1427-1654msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20591: Fri May 2 00:28:50 2025 read: IOPS=2, BW=8672KiB/s (8880kB/s)(12.0MiB/1417msec) clat (msec): min=303, max=719, avg=443.28, stdev=239.51 lat (msec): min=303, max=719, avg=443.28, stdev=239.51 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 305], 10.00th=[ 305], 20.00th=[ 305], | 30.00th=[ 305], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 718], 80.00th=[ 718], 90.00th=[ 718], 95.00th=[ 718], | 99.00th=[ 718], 99.50th=[ 718], 99.90th=[ 718], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 8143, max= 8192, per=88.44%, avg=8167.50, stdev=34.65, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.07%, sys=1.48%, ctx=22, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20592: Fri May 2 00:28:50 2025 read: IOPS=0, BW=2309KiB/s (2364kB/s)(4096KiB/1774msec) clat (nsec): min=307217k, max=307217k, avg=307216568.00, stdev= 0.00 lat (nsec): min=307219k, max=307219k, avg=307219413.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 309], 10.00th=[ 309], 20.00th=[ 309], | 30.00th=[ 309], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 8192, max= 8192, per=88.71%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4618KiB/s (4729kB/s)(8192KiB/1774msec) clat (msec): min=317, max=1013, avg=665.34, stdev=492.60 lat (msec): min=321, max=1013, avg=667.62, stdev=489.66 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 317], 10.00th=[ 317], 20.00th=[ 317], | 30.00th=[ 317], 40.00th=[ 317], 50.00th=[ 317], 60.00th=[ 1011], | 70.00th=[ 1011], 80.00th=[ 1011], 90.00th=[ 1011], 95.00th=[ 1011], | 99.00th=[ 1011], 99.50th=[ 1011], 99.90th=[ 1011], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 6884, max= 6884, per=37.27%, avg=6884.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.11%, sys=6.94%, ctx=71, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20594: Fri May 2 00:28:50 2025 write: IOPS=2, BW=9288KiB/s (9511kB/s)(12.0MiB/1323msec) clat (msec): min=109, max=984, avg=414.76, stdev=493.46 lat (msec): min=110, max=984, avg=415.22, stdev=493.33 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 110], 10.00th=[ 110], 20.00th=[ 110], | 30.00th=[ 110], 40.00th=[ 150], 50.00th=[ 150], 60.00th=[ 150], | 70.00th=[ 986], 80.00th=[ 986], 90.00th=[ 986], 95.00th=[ 986], | 99.00th=[ 986], 99.50th=[ 986], 99.90th=[ 986], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min=16253, max=16253, per=87.99%, avg=16253.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.00%, sys=6.20%, ctx=65, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20596: Fri May 2 00:28:50 2025 write: IOPS=2, BW=10.1MiB/s (10.6MB/s)(12.0MiB/1189msec) clat (msec): min=165, max=766, avg=378.85, stdev=336.07 lat (msec): min=165, max=766, avg=379.28, stdev=335.99 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 165], 10.00th=[ 165], 20.00th=[ 165], | 30.00th=[ 165], 40.00th=[ 205], 50.00th=[ 205], 60.00th=[ 205], | 70.00th=[ 768], 80.00th=[ 768], 90.00th=[ 768], 95.00th=[ 768], | 99.00th=[ 768], 99.50th=[ 768], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min=16351, max=16351, per=88.52%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.08%, sys=8.40%, ctx=58, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9236KiB/s (9457kB/s), 2309KiB/s-8672KiB/s (2364kB/s-8880kB/s), io=16.0MiB (16.8MB), run=1417-1774msec WRITE: bw=18.0MiB/s (18.9MB/s), 4618KiB/s-10.1MiB/s (4729kB/s-10.6MB/s), io=32.0MiB (33.6MB), run=1189-1774msec