== sanity test 398b: DIO and buffer IO race ============== 18:31:53 (1746138713) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.94787 s, 25.8 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4268: Thu May 1 18:32:31 2025 read: IOPS=45, BW=183KiB/s (187kB/s)(5992KiB/32725msec) clat (usec): min=49, max=69947, avg=8754.69, stdev=8690.83 lat (usec): min=49, max=69948, avg=8756.57, stdev=8690.43 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 91], 10.00th=[ 100], 20.00th=[ 120], | 30.00th=[ 725], 40.00th=[ 6063], 50.00th=[ 7504], 60.00th=[ 8848], | 70.00th=[10945], 80.00th=[14746], 90.00th=[20841], 95.00th=[25822], | 99.00th=[37487], 99.50th=[41681], 99.90th=[61604], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 24, max= 319, per=25.71%, avg=182.82, stdev=48.01, samples=65 iops : min= 6, max= 79, avg=45.49, stdev=12.01, samples=65 write: IOPS=48, BW=192KiB/s (197kB/s)(6296KiB/32725msec) clat (usec): min=472, max=266355, avg=12324.74, stdev=14643.48 lat (usec): min=473, max=266357, avg=12326.28, stdev=14643.78 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 742], 10.00th=[ 799], 20.00th=[ 1074], | 30.00th=[ 2180], 40.00th=[ 5473], 50.00th=[ 9241], 60.00th=[ 12387], | 70.00th=[ 15926], 80.00th=[ 20579], 90.00th=[ 28705], 95.00th=[ 34866], | 99.00th=[ 55313], 99.50th=[ 72877], 99.90th=[168821], 99.95th=[267387], | 99.99th=[267387] bw ( KiB/s): min= 8, max= 335, per=25.79%, avg=191.86, stdev=51.56, samples=65 iops : min= 2, max= 83, avg=47.75, stdev=12.90, samples=65 lat (usec) : 50=0.03%, 100=4.85%, 250=8.79%, 500=0.68%, 750=3.29% lat (usec) : 1000=6.71% lat (msec) : 2=5.44%, 4=4.75%, 10=24.22%, 20=25.10%, 50=15.27% lat (msec) : 100=0.78%, 250=0.07%, 500=0.03% cpu : usr=0.14%, sys=13.57%, ctx=4718, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4269: Thu May 1 18:32:31 2025 read: IOPS=45, BW=182KiB/s (187kB/s)(5940KiB/32600msec) clat (usec): min=61, max=77296, avg=8838.62, stdev=8681.55 lat (usec): min=62, max=77297, avg=8841.07, stdev=8681.11 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 90], 10.00th=[ 99], 20.00th=[ 120], | 30.00th=[ 1876], 40.00th=[ 5997], 50.00th=[ 7570], 60.00th=[ 9241], | 70.00th=[11469], 80.00th=[14746], 90.00th=[20317], 95.00th=[24773], | 99.00th=[34866], 99.50th=[39584], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 8, max= 311, per=25.56%, avg=181.75, stdev=56.31, samples=65 iops : min= 2, max= 77, avg=45.17, stdev=14.05, samples=65 write: IOPS=48, BW=195KiB/s (199kB/s)(6348KiB/32600msec) clat (usec): min=485, max=285099, avg=12131.52, stdev=13755.91 lat (usec): min=485, max=285100, avg=12136.60, stdev=13760.68 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 1090], | 30.00th=[ 2278], 40.00th=[ 5342], 50.00th=[ 9110], 60.00th=[ 12518], | 70.00th=[ 16450], 80.00th=[ 20579], 90.00th=[ 28181], 95.00th=[ 35914], | 99.00th=[ 50594], 99.50th=[ 59507], 99.90th=[ 73925], 99.95th=[283116], | 99.99th=[283116] bw ( KiB/s): min= 16, max= 327, per=26.06%, avg=193.88, stdev=53.57, samples=65 iops : min= 4, max= 81, avg=48.20, stdev=13.37, samples=65 lat (usec) : 100=5.14%, 250=8.07%, 500=0.49%, 750=3.42%, 1000=6.51% lat (msec) : 2=5.57%, 4=5.08%, 10=23.80%, 20=25.85%, 50=15.43% lat (msec) : 100=0.62%, 500=0.03% cpu : usr=0.19%, sys=13.57%, ctx=4748, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4270: Thu May 1 18:32:31 2025 read: IOPS=44, BW=179KiB/s (183kB/s)(6028KiB/33756msec) clat (usec): min=61, max=88912, avg=8812.21, stdev=9105.78 lat (usec): min=62, max=88912, avg=8813.88, stdev=9105.93 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 89], 10.00th=[ 97], 20.00th=[ 114], | 30.00th=[ 188], 40.00th=[ 5669], 50.00th=[ 7373], 60.00th=[ 9241], | 70.00th=[11469], 80.00th=[15008], 90.00th=[20841], 95.00th=[26346], | 99.00th=[35914], 99.50th=[41681], 99.90th=[65274], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 47, max= 383, per=25.16%, avg=178.91, stdev=62.61, samples=66 iops : min= 11, max= 95, avg=44.42, stdev=15.62, samples=66 write: IOPS=46, BW=185KiB/s (190kB/s)(6260KiB/33756msec) clat (usec): min=462, max=336050, avg=12948.42, stdev=15495.41 lat (usec): min=462, max=336051, avg=12950.23, stdev=15495.39 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 1106], | 30.00th=[ 2180], 40.00th=[ 5538], 50.00th=[ 9503], 60.00th=[ 13173], | 70.00th=[ 16909], 80.00th=[ 21627], 90.00th=[ 30540], 95.00th=[ 37487], | 99.00th=[ 57410], 99.50th=[ 70779], 99.90th=[ 98042], 99.95th=[337642], | 99.99th=[337642] bw ( KiB/s): min= 8, max= 312, per=24.56%, avg=182.76, stdev=50.45, samples=67 iops : min= 2, max= 78, avg=45.40, stdev=12.53, samples=67 lat (usec) : 100=6.22%, 250=8.72%, 500=0.81%, 750=3.52%, 1000=6.22% lat (msec) : 2=5.40%, 4=4.88%, 10=21.78%, 20=25.20%, 50=16.18% lat (msec) : 100=1.04%, 500=0.03% cpu : usr=0.20%, sys=13.37%, ctx=4751, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4271: Thu May 1 18:32:31 2025 read: IOPS=46, BW=187KiB/s (192kB/s)(6052KiB/32343msec) clat (usec): min=54, max=296402, avg=9372.31, stdev=11340.11 lat (usec): min=54, max=296403, avg=9376.82, stdev=11340.64 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 93], 10.00th=[ 103], 20.00th=[ 129], | 30.00th=[ 4293], 40.00th=[ 6325], 50.00th=[ 8160], 60.00th=[ 9896], | 70.00th=[ 11863], 80.00th=[ 14877], 90.00th=[ 20317], 95.00th=[ 24773], | 99.00th=[ 36439], 99.50th=[ 43254], 99.90th=[ 88605], 99.95th=[295699], | 99.99th=[295699] bw ( KiB/s): min= 32, max= 296, per=26.16%, avg=186.02, stdev=51.04, samples=64 iops : min= 8, max= 74, avg=46.28, stdev=12.76, samples=64 write: IOPS=48, BW=193KiB/s (197kB/s)(6236KiB/32343msec) clat (usec): min=444, max=80364, avg=11508.44, stdev=12312.14 lat (usec): min=445, max=80365, avg=11509.95, stdev=12311.90 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 734], 10.00th=[ 791], 20.00th=[ 1057], | 30.00th=[ 2024], 40.00th=[ 4293], 50.00th=[ 8586], 60.00th=[11600], | 70.00th=[15270], 80.00th=[19268], 90.00th=[27657], 95.00th=[36439], | 99.00th=[54789], 99.50th=[64750], 99.90th=[77071], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 48, max= 312, per=25.81%, avg=192.00, stdev=46.83, samples=64 iops : min= 12, max= 78, avg=47.80, stdev=11.73, samples=64 lat (usec) : 100=4.43%, 250=7.78%, 500=0.75%, 750=3.12%, 1000=7.10% lat (msec) : 2=5.70%, 4=5.37%, 10=23.63%, 20=27.51%, 50=13.70% lat (msec) : 100=0.88%, 500=0.03% cpu : usr=0.23%, sys=13.81%, ctx=4824, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=711KiB/s (728kB/s), 179KiB/s-187KiB/s (183kB/s-192kB/s), io=23.4MiB (24.6MB), run=32343-33756msec WRITE: bw=745KiB/s (763kB/s), 185KiB/s-195KiB/s (190kB/s-199kB/s), io=24.6MiB (25.7MB), run=32343-33756msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4273: Thu May 1 18:34:01 2025 read: IOPS=12, BW=48.5KiB/s (49.7kB/s)(5992KiB/123577msec) clat (msec): min=3, max=298, avg=10.95, stdev=12.35 lat (msec): min=3, max=298, avg=10.95, stdev=12.35 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 22], 95.00th=[ 29], | 99.00th=[ 46], 99.50th=[ 61], 99.90th=[ 169], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 7, max= 144, per=25.00%, avg=48.24, stdev=26.97, samples=245 iops : min= 1, max= 36, avg=11.82, stdev= 6.80, samples=245 write: IOPS=12, BW=50.9KiB/s (52.2kB/s)(6296KiB/123577msec) clat (msec): min=21, max=191, avg=67.95, stdev=24.35 lat (msec): min=21, max=191, avg=67.96, stdev=24.35 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 36], 10.00th=[ 41], 20.00th=[ 47], | 30.00th=[ 53], 40.00th=[ 59], 50.00th=[ 65], 60.00th=[ 70], | 70.00th=[ 78], 80.00th=[ 87], 90.00th=[ 101], 95.00th=[ 113], | 99.00th=[ 144], 99.50th=[ 157], 99.90th=[ 178], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 16, max= 104, per=25.13%, avg=50.77, stdev=14.38, samples=245 iops : min= 4, max= 26, avg=12.45, stdev= 3.66, samples=245 lat (msec) : 4=1.30%, 10=32.88%, 20=8.95%, 50=18.23%, 100=33.43% lat (msec) : 250=5.18%, 500=0.03% cpu : usr=0.09%, sys=2.87%, ctx=3435, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4274: Thu May 1 18:34:01 2025 read: IOPS=11, BW=47.9KiB/s (49.0kB/s)(5940KiB/124063msec) clat (msec): min=3, max=206, avg=10.81, stdev=10.49 lat (msec): min=3, max=206, avg=10.81, stdev=10.49 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 22], 95.00th=[ 28], | 99.00th=[ 46], 99.50th=[ 52], 99.90th=[ 169], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 7, max= 128, per=25.22%, avg=48.67, stdev=27.19, samples=241 iops : min= 1, max= 32, avg=11.98, stdev= 6.82, samples=241 write: IOPS=12, BW=51.2KiB/s (52.4kB/s)(6348KiB/124063msec) clat (msec): min=21, max=200, avg=67.90, stdev=24.35 lat (msec): min=21, max=200, avg=67.90, stdev=24.35 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 37], 10.00th=[ 41], 20.00th=[ 47], | 30.00th=[ 53], 40.00th=[ 59], 50.00th=[ 64], 60.00th=[ 70], | 70.00th=[ 78], 80.00th=[ 87], 90.00th=[ 101], 95.00th=[ 114], | 99.00th=[ 142], 99.50th=[ 155], 99.90th=[ 180], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 15, max= 96, per=25.17%, avg=50.83, stdev=14.14, samples=246 iops : min= 3, max= 24, avg=12.51, stdev= 3.58, samples=246 lat (msec) : 4=1.82%, 10=31.15%, 20=9.96%, 50=18.00%, 100=33.79% lat (msec) : 250=5.27% cpu : usr=0.09%, sys=2.85%, ctx=3462, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4275: Thu May 1 18:34:01 2025 read: IOPS=12, BW=48.0KiB/s (50.1kB/s)(6028KiB/123092msec) clat (msec): min=3, max=149, avg=11.01, stdev= 9.88 lat (msec): min=3, max=149, avg=11.01, stdev= 9.88 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 22], 95.00th=[ 29], | 99.00th=[ 44], 99.50th=[ 58], 99.90th=[ 140], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 7, max= 144, per=25.76%, avg=49.71, stdev=26.80, samples=240 iops : min= 1, max= 36, avg=12.19, stdev= 6.74, samples=240 write: IOPS=12, BW=50.9KiB/s (52.1kB/s)(6260KiB/123092msec) clat (msec): min=22, max=332, avg=67.94, stdev=25.58 lat (msec): min=22, max=332, avg=67.95, stdev=25.58 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 37], 10.00th=[ 41], 20.00th=[ 48], | 30.00th=[ 52], 40.00th=[ 58], 50.00th=[ 64], 60.00th=[ 70], | 70.00th=[ 78], 80.00th=[ 87], 90.00th=[ 100], 95.00th=[ 115], | 99.00th=[ 144], 99.50th=[ 157], 99.90th=[ 215], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 8, max= 88, per=24.99%, avg=50.49, stdev=15.12, samples=245 iops : min= 2, max= 22, avg=12.38, stdev= 3.83, samples=245 lat (msec) : 4=1.46%, 10=31.32%, 20=10.61%, 50=18.55%, 100=33.07% lat (msec) : 250=4.95%, 500=0.03% cpu : usr=0.07%, sys=2.92%, ctx=3497, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4276: Thu May 1 18:34:01 2025 read: IOPS=12, BW=49.3KiB/s (50.5kB/s)(6052KiB/122782msec) clat (msec): min=2, max=127, avg=10.64, stdev= 8.89 lat (msec): min=2, max=127, avg=10.64, stdev= 8.89 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 21], 95.00th=[ 27], | 99.00th=[ 45], 99.50th=[ 54], 99.90th=[ 99], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 7, max= 160, per=26.08%, avg=50.33, stdev=28.30, samples=239 iops : min= 1, max= 40, avg=12.40, stdev= 7.07, samples=239 write: IOPS=12, BW=50.8KiB/s (52.0kB/s)(6236KiB/122782msec) clat (msec): min=25, max=259, avg=68.28, stdev=24.82 lat (msec): min=25, max=259, avg=68.28, stdev=24.82 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 36], 10.00th=[ 41], 20.00th=[ 48], | 30.00th=[ 54], 40.00th=[ 60], 50.00th=[ 65], 60.00th=[ 70], | 70.00th=[ 78], 80.00th=[ 87], 90.00th=[ 102], 95.00th=[ 115], | 99.00th=[ 142], 99.50th=[ 159], 99.90th=[ 190], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 8, max= 112, per=25.02%, avg=50.53, stdev=14.56, samples=245 iops : min= 2, max= 28, avg=12.45, stdev= 3.68, samples=245 lat (msec) : 4=1.63%, 10=31.67%, 20=10.45%, 50=17.51%, 100=33.43% lat (msec) : 250=5.27%, 500=0.03% cpu : usr=0.02%, sys=2.89%, ctx=3483, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=194KiB/s (198kB/s), 47.9KiB/s-49.3KiB/s (49.0kB/s-50.5kB/s), io=23.4MiB (24.6MB), run=122782-124063msec WRITE: bw=203KiB/s (208kB/s), 50.8KiB/s-51.2KiB/s (52.0kB/s-52.4kB/s), io=24.6MiB (25.7MB), run=122782-124063msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4385: Thu May 1 18:34:09 2025 read: IOPS=56, BW=896KiB/s (918kB/s)(5920KiB/6606msec) clat (usec): min=68, max=67483, avg=8636.87, stdev=9272.85 lat (usec): min=68, max=67485, avg=8638.00, stdev=9272.94 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 96], 10.00th=[ 104], 20.00th=[ 122], | 30.00th=[ 840], 40.00th=[ 5800], 50.00th=[ 7504], 60.00th=[ 9372], | 70.00th=[11076], 80.00th=[13698], 90.00th=[19006], 95.00th=[23987], | 99.00th=[41681], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 574, max= 1504, per=25.84%, avg=897.31, stdev=311.45, samples=13 iops : min= 35, max= 94, avg=55.69, stdev=19.59, samples=13 write: IOPS=60, BW=964KiB/s (987kB/s)(6368KiB/6606msec) clat (usec): min=723, max=53953, avg=8489.14, stdev=9219.74 lat (usec): min=724, max=53955, avg=8494.96, stdev=9218.38 clat percentiles (usec): | 1.00th=[ 922], 5.00th=[ 1004], 10.00th=[ 1090], 20.00th=[ 1270], | 30.00th=[ 1713], 40.00th=[ 2507], 50.00th=[ 3982], 60.00th=[ 7701], | 70.00th=[10945], 80.00th=[15401], 90.00th=[22938], 95.00th=[27657], | 99.00th=[39060], 99.50th=[46924], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 448, max= 1600, per=26.51%, avg=968.46, stdev=334.07, samples=13 iops : min= 28, max= 100, avg=60.15, stdev=20.84, samples=13 lat (usec) : 100=2.86%, 250=10.68%, 500=0.39%, 750=0.52%, 1000=2.60% lat (msec) : 2=15.76%, 4=9.51%, 10=23.44%, 20=23.70%, 50=10.03% lat (msec) : 100=0.52% cpu : usr=0.44%, sys=15.52%, ctx=1169, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4386: Thu May 1 18:34:09 2025 read: IOPS=52, BW=845KiB/s (865kB/s)(5744KiB/6800msec) clat (usec): min=64, max=106843, avg=8717.46, stdev=11462.11 lat (usec): min=65, max=106843, avg=8718.76, stdev=11462.22 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 93], 10.00th=[ 105], 20.00th=[ 117], | 30.00th=[ 137], 40.00th=[ 5473], 50.00th=[ 7570], 60.00th=[ 9241], | 70.00th=[ 10683], 80.00th=[ 14222], 90.00th=[ 19268], 95.00th=[ 23987], | 99.00th=[ 44303], 99.50th=[102237], 99.90th=[106431], 99.95th=[106431], | 99.99th=[106431] bw ( KiB/s): min= 352, max= 1664, per=24.21%, avg=840.62, stdev=412.21, samples=13 iops : min= 22, max= 104, avg=52.15, stdev=25.75, samples=13 write: IOPS=60, BW=962KiB/s (985kB/s)(6544KiB/6800msec) clat (usec): min=820, max=67947, avg=8792.42, stdev=10195.36 lat (usec): min=822, max=67948, avg=8794.15, stdev=10195.49 clat percentiles (usec): | 1.00th=[ 857], 5.00th=[ 996], 10.00th=[ 1074], 20.00th=[ 1254], | 30.00th=[ 1516], 40.00th=[ 2114], 50.00th=[ 3654], 60.00th=[ 7242], | 70.00th=[11731], 80.00th=[16909], 90.00th=[22676], 95.00th=[27395], | 99.00th=[40633], 99.50th=[49546], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 574, max= 1536, per=25.77%, avg=941.46, stdev=350.14, samples=13 iops : min= 35, max= 96, avg=58.46, stdev=21.90, samples=13 lat (usec) : 100=3.52%, 250=11.72%, 500=0.26%, 750=0.13%, 1000=2.86% lat (msec) : 2=17.97%, 4=7.81%, 10=22.53%, 20=21.61%, 50=10.94% lat (msec) : 100=0.39%, 250=0.26% cpu : usr=0.24%, sys=15.72%, ctx=1108, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4387: Thu May 1 18:34:09 2025 read: IOPS=58, BW=942KiB/s (964kB/s)(6256KiB/6643msec) clat (usec): min=57, max=47126, avg=7884.19, stdev=7551.18 lat (usec): min=58, max=47126, avg=7886.31, stdev=7551.66 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 95], 10.00th=[ 104], 20.00th=[ 119], | 30.00th=[ 208], 40.00th=[ 5538], 50.00th=[ 7439], 60.00th=[ 9110], | 70.00th=[10683], 80.00th=[13304], 90.00th=[17171], 95.00th=[21627], | 99.00th=[33424], 99.50th=[41681], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 448, max= 1600, per=27.29%, avg=947.46, stdev=362.92, samples=13 iops : min= 28, max= 100, avg=59.15, stdev=22.63, samples=13 write: IOPS=56, BW=908KiB/s (930kB/s)(6032KiB/6643msec) clat (usec): min=662, max=119189, avg=9209.15, stdev=13310.90 lat (usec): min=665, max=119191, avg=9220.52, stdev=13313.66 clat percentiles (usec): | 1.00th=[ 824], 5.00th=[ 988], 10.00th=[ 1057], 20.00th=[ 1270], | 30.00th=[ 1778], 40.00th=[ 2540], 50.00th=[ 3949], 60.00th=[ 7046], | 70.00th=[ 10683], 80.00th=[ 15139], 90.00th=[ 23200], 95.00th=[ 29492], | 99.00th=[ 73925], 99.50th=[103285], 99.90th=[119014], 99.95th=[119014], | 99.99th=[119014] bw ( KiB/s): min= 448, max= 1760, per=24.66%, avg=900.69, stdev=328.74, samples=13 iops : min= 28, max= 110, avg=56.23, stdev=20.50, samples=13 lat (usec) : 100=3.65%, 250=12.11%, 500=0.78%, 750=0.52%, 1000=2.34% lat (msec) : 2=14.32%, 4=9.24%, 10=23.05%, 20=24.09%, 50=9.24% lat (msec) : 100=0.39%, 250=0.26% cpu : usr=0.12%, sys=16.28%, ctx=1091, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4389: Thu May 1 18:34:09 2025 read: IOPS=54, BW=874KiB/s (895kB/s)(6032KiB/6898msec) clat (usec): min=78, max=111717, avg=8246.73, stdev=10776.43 lat (usec): min=78, max=111718, avg=8247.92, stdev=10776.43 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 93], 10.00th=[ 101], 20.00th=[ 112], | 30.00th=[ 128], 40.00th=[ 1614], 50.00th=[ 7046], 60.00th=[ 9372], | 70.00th=[ 11469], 80.00th=[ 14091], 90.00th=[ 17957], 95.00th=[ 23462], | 99.00th=[ 48497], 99.50th=[ 85459], 99.90th=[111674], 99.95th=[111674], | 99.99th=[111674] bw ( KiB/s): min= 223, max= 1536, per=25.00%, avg=868.08, stdev=400.50, samples=13 iops : min= 13, max= 96, avg=53.92, stdev=25.13, samples=13 write: IOPS=56, BW=907KiB/s (929kB/s)(6256KiB/6898msec) clat (usec): min=672, max=139303, avg=9377.22, stdev=14294.30 lat (usec): min=674, max=139305, avg=9386.85, stdev=14291.20 clat percentiles (usec): | 1.00th=[ 840], 5.00th=[ 1037], 10.00th=[ 1106], 20.00th=[ 1319], | 30.00th=[ 1713], 40.00th=[ 2442], 50.00th=[ 3720], 60.00th=[ 6849], | 70.00th=[ 11207], 80.00th=[ 15008], 90.00th=[ 22414], 95.00th=[ 30278], | 99.00th=[ 83362], 99.50th=[126354], 99.90th=[139461], 99.95th=[139461], | 99.99th=[139461] bw ( KiB/s): min= 224, max= 1472, per=24.84%, avg=907.38, stdev=402.25, samples=13 iops : min= 14, max= 92, avg=56.38, stdev=25.20, samples=13 lat (usec) : 100=4.56%, 250=13.15%, 500=1.04%, 750=0.52%, 1000=1.43% lat (msec) : 2=17.06%, 4=8.33%, 10=19.40%, 20=25.13%, 50=8.07% lat (msec) : 100=0.91%, 250=0.39% cpu : usr=0.20%, sys=14.64%, ctx=1064, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3472KiB/s (3556kB/s), 845KiB/s-942KiB/s (865kB/s-964kB/s), io=23.4MiB (24.5MB), run=6606-6898msec WRITE: bw=3653KiB/s (3741kB/s), 907KiB/s-964KiB/s (929kB/s-987kB/s), io=24.6MiB (25.8MB), run=6606-6898msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4388: Thu May 1 18:34:34 2025 read: IOPS=11, BW=189KiB/s (194kB/s)(5920KiB/31280msec) clat (usec): min=3615, max=86747, avg=11514.62, stdev=9491.37 lat (usec): min=3616, max=86748, avg=11516.43, stdev=9491.52 clat percentiles (usec): | 1.00th=[ 3818], 5.00th=[ 4817], 10.00th=[ 5407], 20.00th=[ 5932], | 30.00th=[ 6521], 40.00th=[ 7308], 50.00th=[ 8160], 60.00th=[ 9372], | 70.00th=[11076], 80.00th=[14484], 90.00th=[21365], 95.00th=[31327], | 99.00th=[54264], 99.50th=[57410], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 31, max= 480, per=25.38%, avg=191.87, stdev=93.47, samples=61 iops : min= 1, max= 30, avg=11.70, stdev= 5.80, samples=61 write: IOPS=12, BW=204KiB/s (208kB/s)(6368KiB/31280msec) clat (msec): min=26, max=154, avg=67.57, stdev=20.56 lat (msec): min=26, max=154, avg=67.58, stdev=20.56 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 40], 10.00th=[ 44], 20.00th=[ 51], | 30.00th=[ 57], 40.00th=[ 62], 50.00th=[ 66], 60.00th=[ 70], | 70.00th=[ 77], 80.00th=[ 84], 90.00th=[ 91], 95.00th=[ 109], | 99.00th=[ 133], 99.50th=[ 144], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 95, max= 320, per=25.52%, avg=202.92, stdev=50.01, samples=62 iops : min= 5, max= 20, avg=12.39, stdev= 3.28, samples=62 lat (msec) : 4=0.91%, 10=29.43%, 20=12.24%, 50=15.23%, 100=38.67% lat (msec) : 250=3.52% cpu : usr=0.02%, sys=3.20%, ctx=949, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4390: Thu May 1 18:34:34 2025 read: IOPS=11, BW=181KiB/s (186kB/s)(5744KiB/31665msec) clat (usec): min=3752, max=50862, avg=10646.30, stdev=7415.78 lat (usec): min=3754, max=50863, avg=10652.19, stdev=7414.98 clat percentiles (usec): | 1.00th=[ 4080], 5.00th=[ 4555], 10.00th=[ 5080], 20.00th=[ 5735], | 30.00th=[ 6325], 40.00th=[ 7308], 50.00th=[ 8094], 60.00th=[ 8979], | 70.00th=[10552], 80.00th=[13829], 90.00th=[20055], 95.00th=[27919], | 99.00th=[40109], 99.50th=[44827], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 31, max= 480, per=25.14%, avg=190.05, stdev=95.88, samples=60 iops : min= 1, max= 30, avg=11.55, stdev= 6.00, samples=60 write: IOPS=12, BW=207KiB/s (212kB/s)(6544KiB/31665msec) clat (msec): min=24, max=179, avg=67.73, stdev=23.05 lat (msec): min=24, max=179, avg=67.74, stdev=23.05 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 38], 10.00th=[ 42], 20.00th=[ 49], | 30.00th=[ 54], 40.00th=[ 59], 50.00th=[ 64], 60.00th=[ 71], | 70.00th=[ 77], 80.00th=[ 86], 90.00th=[ 100], 95.00th=[ 112], | 99.00th=[ 144], 99.50th=[ 153], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 127, max= 320, per=25.74%, avg=204.62, stdev=49.22, samples=63 iops : min= 7, max= 20, avg=12.43, stdev= 3.08, samples=63 lat (msec) : 4=0.39%, 10=30.60%, 20=11.07%, 50=16.93%, 100=36.07% lat (msec) : 250=4.95% cpu : usr=0.05%, sys=2.99%, ctx=908, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4391: Thu May 1 18:34:34 2025 read: IOPS=13, BW=213KiB/s (218kB/s)(6256KiB/29410msec) clat (usec): min=2906, max=87301, avg=10145.18, stdev=8230.07 lat (usec): min=2907, max=87302, avg=10146.65, stdev=8230.02 clat percentiles (usec): | 1.00th=[ 3916], 5.00th=[ 4686], 10.00th=[ 5145], 20.00th=[ 5800], | 30.00th=[ 6390], 40.00th=[ 7177], 50.00th=[ 7832], 60.00th=[ 8717], | 70.00th=[ 9503], 80.00th=[12125], 90.00th=[17695], 95.00th=[22938], | 99.00th=[45351], 99.50th=[66847], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 63, max= 576, per=28.40%, avg=214.69, stdev=119.29, samples=58 iops : min= 3, max= 36, avg=13.02, stdev= 7.65, samples=58 write: IOPS=12, BW=205KiB/s (210kB/s)(6032KiB/29410msec) clat (msec): min=28, max=193, avg=67.39, stdev=23.12 lat (msec): min=28, max=193, avg=67.39, stdev=23.12 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 49], | 30.00th=[ 52], 40.00th=[ 59], 50.00th=[ 63], 60.00th=[ 70], | 70.00th=[ 77], 80.00th=[ 86], 90.00th=[ 99], 95.00th=[ 109], | 99.00th=[ 146], 99.50th=[ 155], 99.90th=[ 194], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 95, max= 320, per=25.69%, avg=204.21, stdev=47.57, samples=58 iops : min= 5, max= 20, avg=12.36, stdev= 3.00, samples=58 lat (msec) : 4=0.65%, 10=36.59%, 20=10.16%, 50=14.58%, 100=33.98% lat (msec) : 250=4.04% cpu : usr=0.05%, sys=3.29%, ctx=907, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4392: Thu May 1 18:34:34 2025 read: IOPS=12, BW=198KiB/s (202kB/s)(6032KiB/30524msec) clat (usec): min=3052, max=63456, avg=10262.41, stdev=7155.47 lat (usec): min=3054, max=63457, avg=10264.08, stdev=7155.39 clat percentiles (usec): | 1.00th=[ 3720], 5.00th=[ 4490], 10.00th=[ 5014], 20.00th=[ 5800], | 30.00th=[ 6718], 40.00th=[ 7373], 50.00th=[ 7963], 60.00th=[ 8717], | 70.00th=[10028], 80.00th=[12911], 90.00th=[18744], 95.00th=[23987], | 99.00th=[40633], 99.50th=[43254], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 31, max= 512, per=25.92%, avg=195.92, stdev=100.99, samples=61 iops : min= 1, max= 32, avg=12.00, stdev= 6.37, samples=61 write: IOPS=12, BW=205KiB/s (210kB/s)(6256KiB/30524msec) clat (msec): min=23, max=164, avg=67.93, stdev=22.49 lat (msec): min=23, max=164, avg=67.94, stdev=22.48 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 39], 10.00th=[ 43], 20.00th=[ 48], | 30.00th=[ 54], 40.00th=[ 59], 50.00th=[ 64], 60.00th=[ 70], | 70.00th=[ 79], 80.00th=[ 87], 90.00th=[ 99], 95.00th=[ 109], | 99.00th=[ 138], 99.50th=[ 155], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 96, max= 320, per=25.76%, avg=204.82, stdev=49.42, samples=61 iops : min= 6, max= 20, avg=12.56, stdev= 3.19, samples=61 lat (msec) : 4=0.91%, 10=33.59%, 20=10.16%, 50=15.36%, 100=35.29% lat (msec) : 250=4.69% cpu : usr=0.00%, sys=3.27%, ctx=907, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=756KiB/s (775kB/s), 181KiB/s-213KiB/s (186kB/s-218kB/s), io=23.4MiB (24.5MB), run=29410-31665msec WRITE: bw=796KiB/s (815kB/s), 204KiB/s-207KiB/s (208kB/s-212kB/s), io=24.6MiB (25.8MB), run=29410-31665msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4427: Thu May 1 18:34:37 2025 read: IOPS=8, BW=9062KiB/s (9279kB/s)(7168KiB/791msec) clat (usec): min=523, max=111663, avg=52051.93, stdev=39988.68 lat (usec): min=525, max=111665, avg=52053.61, stdev=39988.79 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 523], 10.00th=[ 523], 20.00th=[ 603], | 30.00th=[ 51119], 40.00th=[ 51119], 50.00th=[ 60556], 60.00th=[ 67634], | 70.00th=[ 67634], 80.00th=[ 72877], 90.00th=[111674], 95.00th=[111674], | 99.00th=[111674], 99.50th=[111674], 99.90th=[111674], 99.95th=[111674], | 99.99th=[111674] bw ( KiB/s): min=10240, max=10240, per=82.60%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6473KiB/s (6628kB/s)(5120KiB/791msec) clat (msec): min=41, max=116, avg=72.61, stdev=27.28 lat (msec): min=42, max=116, avg=72.74, stdev=27.24 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 43], | 30.00th=[ 67], 40.00th=[ 67], 50.00th=[ 67], 60.00th=[ 67], | 70.00th=[ 72], 80.00th=[ 72], 90.00th=[ 117], 95.00th=[ 117], | 99.00th=[ 117], 99.50th=[ 117], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 2048, max= 2048, per=11.80%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 50=8.33%, 100=58.33%, 250=16.67% cpu : usr=0.00%, sys=23.29%, ctx=485, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4428: Thu May 1 18:34:37 2025 read: IOPS=1, BW=1240KiB/s (1269kB/s)(2048KiB/1652msec) clat (usec): min=50084, max=72662, avg=61373.22, stdev=15964.96 lat (usec): min=50085, max=72664, avg=61374.91, stdev=15965.34 clat percentiles (usec): | 1.00th=[50070], 5.00th=[50070], 10.00th=[50070], 20.00th=[50070], | 30.00th=[50070], 40.00th=[50070], 50.00th=[50070], 60.00th=[72877], | 70.00th=[72877], 80.00th=[72877], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 1848, max= 2048, per=15.71%, avg=1948.00, stdev=141.42, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=6, BW=6199KiB/s (6347kB/s)(10.0MiB/1652msec) clat (msec): min=54, max=236, avg=137.31, stdev=54.80 lat (msec): min=55, max=236, avg=137.66, stdev=54.36 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 55], 10.00th=[ 55], 20.00th=[ 59], | 30.00th=[ 116], 40.00th=[ 125], 50.00th=[ 131], 60.00th=[ 146], | 70.00th=[ 171], 80.00th=[ 171], 90.00th=[ 186], 95.00th=[ 236], | 99.00th=[ 236], 99.50th=[ 236], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 2048, max= 7393, per=28.92%, avg=5019.00, stdev=2722.05, samples=3 iops : min= 2, max= 7, avg= 4.67, stdev= 2.52, samples=3 lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=17.93%, ctx=275, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4430: Thu May 1 18:34:37 2025 read: IOPS=6, BW=6564KiB/s (6722kB/s)(6144KiB/936msec) clat (usec): min=608, max=109639, avg=45006.05, stdev=41463.15 lat (usec): min=610, max=109641, avg=45007.66, stdev=41463.15 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 611], 10.00th=[ 611], 20.00th=[ 14746], | 30.00th=[ 14746], 40.00th=[ 32375], 50.00th=[ 32375], 60.00th=[ 32900], | 70.00th=[ 80217], 80.00th=[ 80217], 90.00th=[109577], 95.00th=[109577], | 99.00th=[109577], 99.50th=[109577], 99.90th=[109577], 99.95th=[109577], | 99.99th=[109577] bw ( KiB/s): min= 2048, max= 2048, per=16.52%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6564KiB/s (6722kB/s)(6144KiB/936msec) clat (msec): min=35, max=280, avg=107.01, stdev=92.25 lat (msec): min=35, max=280, avg=107.09, stdev=92.24 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 40], | 30.00th=[ 40], 40.00th=[ 58], 50.00th=[ 58], 60.00th=[ 101], | 70.00th=[ 129], 80.00th=[ 129], 90.00th=[ 279], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 6144, max= 6144, per=35.40%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 20=8.33%, 50=33.33%, 100=16.67%, 250=25.00%, 500=8.33% cpu : usr=0.11%, sys=18.61%, ctx=100, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4432: Thu May 1 18:34:37 2025 read: IOPS=5, BW=5614KiB/s (5749kB/s)(5120KiB/912msec) clat (usec): min=676, max=80563, avg=52336.40, stdev=31799.10 lat (usec): min=677, max=80564, avg=52337.78, stdev=31799.28 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 676], 10.00th=[ 676], 20.00th=[ 676], | 30.00th=[51119], 40.00th=[51119], 50.00th=[53216], 60.00th=[53216], | 70.00th=[76022], 80.00th=[76022], 90.00th=[80217], 95.00th=[80217], | 99.00th=[80217], 99.50th=[80217], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] write: IOPS=7, BW=7860KiB/s (8048kB/s)(7168KiB/912msec) clat (msec): min=36, max=226, avg=86.77, stdev=67.34 lat (msec): min=36, max=226, avg=86.84, stdev=67.34 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 39], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 77], 60.00th=[ 79], | 70.00th=[ 79], 80.00th=[ 111], 90.00th=[ 226], 95.00th=[ 226], | 99.00th=[ 226], 99.50th=[ 226], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 8192, max= 8192, per=47.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=25.00%, 100=50.00%, 250=16.67% cpu : usr=0.00%, sys=18.55%, ctx=130, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.1MiB/s (12.7MB/s), 1240KiB/s-9062KiB/s (1269kB/s-9279kB/s), io=20.0MiB (20.0MB), run=791-1652msec WRITE: bw=16.9MiB/s (17.8MB/s), 6199KiB/s-7860KiB/s (6347kB/s-8048kB/s), io=28.0MiB (29.4MB), run=791-1652msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4429: Thu May 1 18:34:37 2025 read: IOPS=5, BW=5325KiB/s (5453kB/s)(7168KiB/1346msec) clat (msec): min=34, max=276, avg=99.70, stdev=88.60 lat (msec): min=34, max=276, avg=99.70, stdev=88.60 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 39], | 30.00th=[ 45], 40.00th=[ 45], 50.00th=[ 64], 60.00th=[ 85], | 70.00th=[ 85], 80.00th=[ 157], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 4096, max= 6131, per=42.67%, avg=5113.50, stdev=1438.96, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3804KiB/s (3895kB/s)(5120KiB/1346msec) clat (msec): min=63, max=144, avg=111.63, stdev=32.93 lat (msec): min=63, max=144, avg=112.00, stdev=32.89 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 64], | 30.00th=[ 102], 40.00th=[ 102], 50.00th=[ 110], 60.00th=[ 110], | 70.00th=[ 140], 80.00th=[ 140], 90.00th=[ 146], 95.00th=[ 146], | 99.00th=[ 146], 99.50th=[ 146], 99.90th=[ 146], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 4096, max= 4096, per=24.41%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=25.00%, 100=25.00%, 250=41.67%, 500=8.33% cpu : usr=0.22%, sys=7.06%, ctx=56, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4431: Thu May 1 18:34:37 2025 read: IOPS=1, BW=1269KiB/s (1299kB/s)(2048KiB/1614msec) clat (msec): min=33, max=143, avg=88.90, stdev=77.80 lat (msec): min=33, max=143, avg=88.90, stdev=77.80 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34], | 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 34], 60.00th=[ 144], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 2048, max= 2048, per=17.09%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=6, BW=6344KiB/s (6497kB/s)(10.0MiB/1614msec) clat (msec): min=72, max=203, avg=141.22, stdev=44.76 lat (msec): min=72, max=203, avg=141.35, stdev=44.75 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 73], 10.00th=[ 73], 20.00th=[ 81], | 30.00th=[ 116], 40.00th=[ 127], 50.00th=[ 134], 60.00th=[ 142], | 70.00th=[ 178], 80.00th=[ 178], 90.00th=[ 192], 95.00th=[ 205], | 99.00th=[ 205], 99.50th=[ 205], 99.90th=[ 205], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 2048, max= 8192, per=32.43%, avg=5441.00, stdev=3121.91, samples=3 iops : min= 2, max= 8, avg= 5.00, stdev= 3.00, samples=3 lat (msec) : 50=8.33%, 100=16.67%, 250=75.00% cpu : usr=0.00%, sys=6.57%, ctx=53, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4433: Thu May 1 18:34:37 2025 read: IOPS=3, BW=3595KiB/s (3681kB/s)(6144KiB/1709msec) clat (msec): min=65, max=203, avg=119.19, stdev=59.36 lat (msec): min=65, max=203, avg=119.19, stdev=59.36 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 66], 10.00th=[ 66], 20.00th=[ 68], | 30.00th=[ 68], 40.00th=[ 70], 50.00th=[ 70], 60.00th=[ 144], | 70.00th=[ 165], 80.00th=[ 165], 90.00th=[ 203], 95.00th=[ 203], | 99.00th=[ 203], 99.50th=[ 203], 99.90th=[ 203], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 4087, max= 6144, per=42.69%, avg=5115.50, stdev=1454.52, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 write: IOPS=3, BW=3595KiB/s (3681kB/s)(6144KiB/1709msec) clat (msec): min=68, max=331, avg=162.35, stdev=97.36 lat (msec): min=68, max=331, avg=162.43, stdev=97.36 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 69], 10.00th=[ 69], 20.00th=[ 82], | 30.00th=[ 82], 40.00th=[ 130], 50.00th=[ 130], 60.00th=[ 153], | 70.00th=[ 211], 80.00th=[ 211], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 2043, max= 4096, per=20.34%, avg=3411.67, stdev=1185.30, samples=3 iops : min= 1, max= 4, avg= 3.00, stdev= 1.73, samples=3 lat (msec) : 100=41.67%, 250=50.00%, 500=8.33% cpu : usr=0.12%, sys=5.15%, ctx=110, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4434: Thu May 1 18:34:37 2025 read: IOPS=2, BW=3001KiB/s (3073kB/s)(5120KiB/1706msec) clat (msec): min=29, max=124, avg=70.30, stdev=39.53 lat (msec): min=29, max=124, avg=70.30, stdev=39.53 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 30], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 66], 60.00th=[ 66], | 70.00th=[ 94], 80.00th=[ 94], 90.00th=[ 125], 95.00th=[ 125], | 99.00th=[ 125], 99.50th=[ 125], 99.90th=[ 125], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 8175, max= 8175, per=68.22%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4202KiB/s (4302kB/s)(7168KiB/1706msec) clat (msec): min=74, max=299, avg=186.55, stdev=82.19 lat (msec): min=74, max=300, avg=186.66, stdev=82.23 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 74], 20.00th=[ 112], | 30.00th=[ 161], 40.00th=[ 161], 50.00th=[ 180], 60.00th=[ 201], | 70.00th=[ 201], 80.00th=[ 279], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 2048, max= 4087, per=20.25%, avg=3396.67, stdev=1168.09, samples=3 iops : min= 2, max= 3, avg= 2.67, stdev= 0.58, samples=3 lat (msec) : 50=16.67%, 100=25.00%, 250=41.67%, 500=16.67% cpu : usr=0.00%, sys=5.28%, ctx=46, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.7MiB/s (12.3MB/s), 1269KiB/s-5325KiB/s (1299kB/s-5453kB/s), io=20.0MiB (20.0MB), run=1346-1709msec WRITE: bw=16.4MiB/s (17.2MB/s), 3595KiB/s-6344KiB/s (3681kB/s-6497kB/s), io=28.0MiB (29.4MB), run=1346-1709msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4442: Thu May 1 18:34:40 2025 read: IOPS=3, BW=14.2MiB/s (14.9MB/s)(12.0MiB/845msec) clat (msec): min=156, max=360, avg=265.00, stdev=102.48 lat (msec): min=156, max=360, avg=265.00, stdev=102.48 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 279], 50.00th=[ 279], 60.00th=[ 279], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.12%, sys=20.97%, ctx=1198, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4444: Thu May 1 18:34:40 2025 read: IOPS=0, BW=3772KiB/s (3862kB/s)(4096KiB/1086msec) clat (nsec): min=185022k, max=185022k, avg=185022414.00, stdev= 0.00 lat (nsec): min=185025k, max=185025k, avg=185025249.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 186], 10.00th=[ 186], 20.00th=[ 186], | 30.00th=[ 186], 40.00th=[ 186], 50.00th=[ 186], 60.00th=[ 186], | 70.00th=[ 186], 80.00th=[ 186], 90.00th=[ 186], 95.00th=[ 186], | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 8192, max= 8192, per=54.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=7543KiB/s (7724kB/s)(8192KiB/1086msec) clat (msec): min=193, max=670, avg=431.61, stdev=337.39 lat (msec): min=193, max=670, avg=431.87, stdev=337.47 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 194], 10.00th=[ 194], 20.00th=[ 194], | 30.00th=[ 194], 40.00th=[ 194], 50.00th=[ 194], 60.00th=[ 667], | 70.00th=[ 667], 80.00th=[ 667], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 8192, max= 8192, per=31.70%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=17.70%, ctx=571, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4446: Thu May 1 18:34:40 2025 write: IOPS=2, BW=9691KiB/s (9923kB/s)(12.0MiB/1268msec) clat (msec): min=283, max=489, avg=393.81, stdev=103.62 lat (msec): min=284, max=489, avg=394.14, stdev=103.50 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 284], 10.00th=[ 284], 20.00th=[ 284], | 30.00th=[ 284], 40.00th=[ 409], 50.00th=[ 409], 60.00th=[ 409], | 70.00th=[ 489], 80.00th=[ 489], 90.00th=[ 489], 95.00th=[ 489], | 99.00th=[ 489], 99.50th=[ 489], 99.90th=[ 489], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 8192, max= 8192, per=31.70%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=100.00% cpu : usr=0.00%, sys=21.78%, ctx=81, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4449: Thu May 1 18:34:40 2025 write: IOPS=2, BW=10.7MiB/s (11.2MB/s)(12.0MiB/1124msec) clat (msec): min=105, max=611, avg=351.75, stdev=253.58 lat (msec): min=105, max=612, avg=352.14, stdev=253.54 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 106], 10.00th=[ 106], 20.00th=[ 106], | 30.00th=[ 106], 40.00th=[ 338], 50.00th=[ 338], 60.00th=[ 338], | 70.00th=[ 609], 80.00th=[ 609], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 8175, max= 8192, per=31.67%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=22.26%, ctx=69, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=14.7MiB/s (15.4MB/s), 3772KiB/s-14.2MiB/s (3862kB/s-14.9MB/s), io=16.0MiB (16.8MB), run=845-1086msec WRITE: bw=25.2MiB/s (26.5MB/s), 7543KiB/s-10.7MiB/s (7724kB/s-11.2MB/s), io=32.0MiB (33.6MB), run=1086-1268msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4443: Thu May 1 18:34:40 2025 read: IOPS=4, BW=16.0MiB/s (16.8MB/s)(12.0MiB/750msec) clat (msec): min=50, max=561, avg=227.39, stdev=289.59 lat (msec): min=50, max=561, avg=227.39, stdev=289.59 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 51], 10.00th=[ 51], 20.00th=[ 51], | 30.00th=[ 51], 40.00th=[ 71], 50.00th=[ 71], 60.00th=[ 71], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=66.67%, 750=33.33% cpu : usr=0.00%, sys=9.88%, ctx=61, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4445: Thu May 1 18:34:40 2025 read: IOPS=0, BW=2926KiB/s (2996kB/s)(4096KiB/1400msec) clat (nsec): min=232323k, max=232323k, avg=232322660.00, stdev= 0.00 lat (nsec): min=232326k, max=232326k, avg=232325615.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 232], 10.00th=[ 232], 20.00th=[ 232], | 30.00th=[ 232], 40.00th=[ 232], 50.00th=[ 232], 60.00th=[ 232], | 70.00th=[ 232], 80.00th=[ 232], 90.00th=[ 232], 95.00th=[ 232], | 99.00th=[ 232], 99.50th=[ 232], 99.90th=[ 232], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 8192, max= 8192, per=70.01%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5851KiB/s (5992kB/s)(8192KiB/1400msec) clat (msec): min=288, max=849, avg=569.04, stdev=396.93 lat (msec): min=288, max=850, avg=569.46, stdev=396.90 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 288], 10.00th=[ 288], 20.00th=[ 288], | 30.00th=[ 288], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 852], | 70.00th=[ 852], 80.00th=[ 852], 90.00th=[ 852], 95.00th=[ 852], | 99.00th=[ 852], 99.50th=[ 852], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 6023, max= 6023, per=29.61%, avg=6023.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=6.29%, ctx=47, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4448: Thu May 1 18:34:40 2025 write: IOPS=1, BW=7628KiB/s (7811kB/s)(12.0MiB/1611msec) clat (msec): min=151, max=889, avg=528.75, stdev=369.65 lat (msec): min=151, max=890, avg=529.40, stdev=369.71 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 153], 10.00th=[ 153], 20.00th=[ 153], | 30.00th=[ 153], 40.00th=[ 542], 50.00th=[ 542], 60.00th=[ 542], | 70.00th=[ 894], 80.00th=[ 894], 90.00th=[ 894], 95.00th=[ 894], | 99.00th=[ 894], 99.50th=[ 894], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 6140, max= 6140, per=30.19%, avg=6140.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.12%, sys=6.89%, ctx=52, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4450: Thu May 1 18:34:40 2025 write: IOPS=3, BW=14.4MiB/s (15.1MB/s)(12.0MiB/832msec) clat (msec): min=131, max=464, avg=251.19, stdev=185.30 lat (msec): min=131, max=465, avg=251.73, stdev=185.48 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 132], 10.00th=[ 132], 20.00th=[ 132], | 30.00th=[ 132], 40.00th=[ 159], 50.00th=[ 159], 60.00th=[ 159], | 70.00th=[ 464], 80.00th=[ 464], 90.00th=[ 464], 95.00th=[ 464], | 99.00th=[ 464], 99.50th=[ 464], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min=16384, max=16384, per=80.55%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=11.43%, ctx=52, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.4MiB/s (11.0MB/s), 2926KiB/s-16.0MiB/s (2996kB/s-16.8MB/s), io=16.0MiB (16.8MB), run=750-1400msec WRITE: bw=19.9MiB/s (20.8MB/s), 5851KiB/s-14.4MiB/s (5992kB/s-15.1MB/s), io=32.0MiB (33.6MB), run=832-1611msec