== sanity test 398b: DIO and buffer IO race ============== 19:03:53 (1746140633) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.1881 s, 23.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21426: Thu May 1 19:04:26 2025 read: IOPS=52, BW=212KiB/s (217kB/s)(5992KiB/28330msec) clat (usec): min=64, max=156444, avg=10549.45, stdev=7677.32 lat (usec): min=65, max=156445, avg=10551.15, stdev=7677.22 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 383], 10.00th=[ 4047], 20.00th=[ 5538], | 30.00th=[ 6587], 40.00th=[ 7701], 50.00th=[ 9110], 60.00th=[ 10683], | 70.00th=[ 12387], 80.00th=[ 14746], 90.00th=[ 19006], 95.00th=[ 22676], | 99.00th=[ 33424], 99.50th=[ 41157], 99.90th=[ 49546], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 120, max= 327, per=24.78%, avg=209.93, stdev=45.62, samples=56 iops : min= 30, max= 81, avg=52.27, stdev=11.30, samples=56 write: IOPS=55, BW=222KiB/s (228kB/s)(6296KiB/28330msec) clat (usec): min=466, max=50541, avg=7923.46, stdev=7979.93 lat (usec): min=466, max=50542, avg=7924.62, stdev=7980.01 clat percentiles (usec): | 1.00th=[ 635], 5.00th=[ 717], 10.00th=[ 766], 20.00th=[ 922], | 30.00th=[ 1582], 40.00th=[ 3556], 50.00th=[ 5932], 60.00th=[ 7767], | 70.00th=[10159], 80.00th=[13698], 90.00th=[18744], 95.00th=[23462], | 99.00th=[35914], 99.50th=[40109], 99.90th=[49546], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 136, max= 360, per=24.76%, avg=219.61, stdev=54.06, samples=56 iops : min= 34, max= 90, avg=54.68, stdev=13.47, samples=56 lat (usec) : 100=0.88%, 250=1.43%, 500=0.33%, 750=4.07%, 1000=7.23% lat (msec) : 2=5.99%, 4=5.79%, 10=37.43%, 20=28.61%, 50=8.17% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.12%, sys=14.91%, ctx=5336, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21427: Thu May 1 19:04:26 2025 read: IOPS=54, BW=219KiB/s (224kB/s)(5940KiB/27178msec) clat (usec): min=61, max=165602, avg=9974.93, stdev=7688.05 lat (usec): min=62, max=165604, avg=9975.84, stdev=7688.07 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[ 123], 10.00th=[ 4015], 20.00th=[ 5211], | 30.00th=[ 6194], 40.00th=[ 7373], 50.00th=[ 8586], 60.00th=[ 10290], | 70.00th=[ 11863], 80.00th=[ 13960], 90.00th=[ 17695], 95.00th=[ 20841], | 99.00th=[ 34341], 99.50th=[ 40109], 99.90th=[ 64750], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 47, max= 328, per=25.69%, avg=217.57, stdev=52.83, samples=54 iops : min= 11, max= 82, avg=54.17, stdev=13.22, samples=54 write: IOPS=58, BW=234KiB/s (239kB/s)(6348KiB/27178msec) clat (usec): min=511, max=55491, avg=7759.68, stdev=8099.61 lat (usec): min=511, max=55492, avg=7761.82, stdev=8100.14 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 734], 10.00th=[ 775], 20.00th=[ 889], | 30.00th=[ 1336], 40.00th=[ 2671], 50.00th=[ 5538], 60.00th=[ 7570], | 70.00th=[10028], 80.00th=[13435], 90.00th=[19006], 95.00th=[23725], | 99.00th=[34866], 99.50th=[41157], 99.90th=[52167], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 103, max= 384, per=26.08%, avg=231.37, stdev=61.09, samples=54 iops : min= 25, max= 96, avg=57.61, stdev=15.28, samples=54 lat (usec) : 100=1.17%, 250=1.86%, 500=0.10%, 750=3.58%, 1000=8.85% lat (msec) : 2=6.48%, 4=5.34%, 10=37.14%, 20=28.22%, 50=7.13% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.20%, sys=14.88%, ctx=4989, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21428: Thu May 1 19:04:26 2025 read: IOPS=53, BW=214KiB/s (219kB/s)(6028KiB/28136msec) clat (usec): min=58, max=49725, avg=10178.74, stdev=6616.87 lat (usec): min=59, max=49726, avg=10179.86, stdev=6616.72 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 131], 10.00th=[ 4080], 20.00th=[ 5407], | 30.00th=[ 6390], 40.00th=[ 7439], 50.00th=[ 8848], 60.00th=[10290], | 70.00th=[12125], 80.00th=[14353], 90.00th=[18744], 95.00th=[21627], | 99.00th=[34341], 99.50th=[38536], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 104, max= 344, per=25.21%, avg=213.54, stdev=49.75, samples=56 iops : min= 26, max= 86, avg=53.18, stdev=12.46, samples=56 write: IOPS=55, BW=222KiB/s (228kB/s)(6260KiB/28136msec) clat (usec): min=450, max=180497, avg=8140.39, stdev=9200.76 lat (usec): min=451, max=180498, avg=8143.01, stdev=9199.97 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 725], 10.00th=[ 783], 20.00th=[ 938], | 30.00th=[ 1549], 40.00th=[ 3326], 50.00th=[ 5997], 60.00th=[ 7898], | 70.00th=[ 10421], 80.00th=[ 14091], 90.00th=[ 19268], 95.00th=[ 24249], | 99.00th=[ 34341], 99.50th=[ 39060], 99.90th=[ 58459], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 88, max= 361, per=25.00%, avg=221.71, stdev=57.87, samples=56 iops : min= 22, max= 90, avg=55.23, stdev=14.44, samples=56 lat (usec) : 100=1.11%, 250=1.60%, 500=0.23%, 750=3.45%, 1000=7.81% lat (msec) : 2=6.09%, 4=5.92%, 10=37.21%, 20=28.16%, 50=8.33% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.21%, sys=14.65%, ctx=5209, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21429: Thu May 1 19:04:26 2025 read: IOPS=54, BW=219KiB/s (224kB/s)(6052KiB/27666msec) clat (usec): min=60, max=48535, avg=10052.43, stdev=6312.42 lat (usec): min=61, max=48536, avg=10053.69, stdev=6312.13 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 131], 10.00th=[ 4113], 20.00th=[ 5276], | 30.00th=[ 6521], 40.00th=[ 7504], 50.00th=[ 8717], 60.00th=[10290], | 70.00th=[12256], 80.00th=[14222], 90.00th=[17695], 95.00th=[21365], | 99.00th=[32113], 99.50th=[34866], 99.90th=[44303], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 80, max= 336, per=25.78%, avg=218.36, stdev=50.41, samples=55 iops : min= 20, max= 84, avg=54.35, stdev=12.68, samples=55 write: IOPS=56, BW=225KiB/s (231kB/s)(6236KiB/27666msec) clat (usec): min=477, max=177368, avg=7957.15, stdev=9336.62 lat (usec): min=478, max=177369, avg=7959.42, stdev=9335.85 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 725], 10.00th=[ 766], 20.00th=[ 906], | 30.00th=[ 1352], 40.00th=[ 2638], 50.00th=[ 5669], 60.00th=[ 7767], | 70.00th=[ 10290], 80.00th=[ 13960], 90.00th=[ 18744], 95.00th=[ 23725], | 99.00th=[ 36439], 99.50th=[ 40633], 99.90th=[ 64750], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 96, max= 384, per=25.36%, avg=224.91, stdev=60.50, samples=55 iops : min= 24, max= 96, avg=55.98, stdev=15.17, samples=55 lat (usec) : 100=1.24%, 250=1.63%, 500=0.20%, 750=3.39%, 1000=8.59% lat (msec) : 2=6.51%, 4=5.21%, 10=36.85%, 20=28.81%, 50=7.42% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.18%, sys=14.87%, ctx=5070, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=848KiB/s (868kB/s), 212KiB/s-219KiB/s (217kB/s-224kB/s), io=23.4MiB (24.6MB), run=27178-28330msec WRITE: bw=887KiB/s (909kB/s), 222KiB/s-234KiB/s (228kB/s-239kB/s), io=24.6MiB (25.7MB), run=27178-28330msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21421: Thu May 1 19:04:39 2025 read: IOPS=36, BW=145KiB/s (149kB/s)(5992KiB/41260msec) clat (msec): min=2, max=258, avg=10.36, stdev=10.94 lat (msec): min=2, max=258, avg=10.36, stdev=10.94 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 7], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 13], 90.00th=[ 23], 95.00th=[ 30], | 99.00th=[ 46], 99.50th=[ 58], 99.90th=[ 77], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 40, max= 304, per=24.85%, avg=144.35, stdev=67.71, samples=82 iops : min= 10, max= 76, avg=35.84, stdev=17.04, samples=82 write: IOPS=38, BW=153KiB/s (156kB/s)(6296KiB/41260msec) clat (usec): min=4826, max=79940, avg=16285.67, stdev=9576.06 lat (usec): min=4827, max=79941, avg=16287.08, stdev=9576.16 clat percentiles (usec): | 1.00th=[ 6194], 5.00th=[ 7242], 10.00th=[ 8094], 20.00th=[ 9241], | 30.00th=[10290], 40.00th=[11469], 50.00th=[12649], 60.00th=[14746], | 70.00th=[17695], 80.00th=[22676], 90.00th=[30278], 95.00th=[36439], | 99.00th=[49021], 99.50th=[54789], 99.90th=[64226], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 16, max= 304, per=24.89%, avg=151.57, stdev=68.81, samples=82 iops : min= 4, max= 76, avg=37.65, stdev=17.32, samples=82 lat (msec) : 4=3.91%, 10=44.73%, 20=32.98%, 50=17.51%, 100=0.85% lat (msec) : 500=0.03% cpu : usr=0.12%, sys=6.86%, ctx=3433, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21422: Thu May 1 19:04:39 2025 read: IOPS=36, BW=144KiB/s (148kB/s)(5940KiB/41231msec) clat (msec): min=2, max=271, avg=10.27, stdev=11.07 lat (msec): min=2, max=271, avg=10.27, stdev=11.07 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 10], 80.00th=[ 13], 90.00th=[ 23], 95.00th=[ 30], | 99.00th=[ 46], 99.50th=[ 52], 99.90th=[ 68], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 7, max= 312, per=24.69%, avg=143.43, stdev=69.81, samples=82 iops : min= 1, max= 78, avg=35.67, stdev=17.48, samples=82 write: IOPS=38, BW=154KiB/s (158kB/s)(6348KiB/41231msec) clat (usec): min=4480, max=80103, avg=16299.20, stdev=9546.43 lat (usec): min=4481, max=80104, avg=16300.53, stdev=9546.37 clat percentiles (usec): | 1.00th=[ 5932], 5.00th=[ 7242], 10.00th=[ 8160], 20.00th=[ 9372], | 30.00th=[10552], 40.00th=[11469], 50.00th=[12911], 60.00th=[15008], | 70.00th=[17695], 80.00th=[22414], 90.00th=[29754], 95.00th=[35914], | 99.00th=[50070], 99.50th=[55837], 99.90th=[71828], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 39, max= 304, per=25.09%, avg=152.80, stdev=68.55, samples=82 iops : min= 9, max= 76, avg=38.01, stdev=17.16, samples=82 lat (msec) : 4=3.68%, 10=44.66%, 20=33.69%, 50=17.15%, 100=0.78% lat (msec) : 500=0.03% cpu : usr=0.19%, sys=6.79%, ctx=3451, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21423: Thu May 1 19:04:39 2025 read: IOPS=36, BW=147KiB/s (150kB/s)(6028KiB/41092msec) clat (usec): min=2395, max=65983, avg=10205.91, stdev=8454.96 lat (usec): min=2396, max=65983, avg=10207.17, stdev=8455.11 clat percentiles (usec): | 1.00th=[ 3261], 5.00th=[ 3818], 10.00th=[ 4178], 20.00th=[ 4883], | 30.00th=[ 5538], 40.00th=[ 6128], 50.00th=[ 6849], 60.00th=[ 8029], | 70.00th=[10028], 80.00th=[13698], 90.00th=[21365], 95.00th=[28967], | 99.00th=[41157], 99.50th=[46924], 99.90th=[62653], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 24, max= 319, per=25.19%, avg=146.38, stdev=68.37, samples=82 iops : min= 6, max= 79, avg=36.48, stdev=17.04, samples=82 write: IOPS=38, BW=152KiB/s (156kB/s)(6260KiB/41092msec) clat (msec): min=5, max=280, avg=16.36, stdev=11.58 lat (msec): min=5, max=280, avg=16.36, stdev=11.58 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 23], 90.00th=[ 29], 95.00th=[ 36], | 99.00th=[ 50], 99.50th=[ 59], 99.90th=[ 87], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 32, max= 304, per=24.95%, avg=151.94, stdev=67.44, samples=82 iops : min= 8, max= 76, avg=37.87, stdev=16.85, samples=82 lat (msec) : 4=3.68%, 10=43.29%, 20=35.35%, 50=17.02%, 100=0.62% lat (msec) : 500=0.03% cpu : usr=0.05%, sys=6.92%, ctx=3398, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21424: Thu May 1 19:04:39 2025 read: IOPS=36, BW=148KiB/s (151kB/s)(6052KiB/41006msec) clat (usec): min=2704, max=74079, avg=10041.01, stdev=8531.38 lat (usec): min=2705, max=74079, avg=10042.15, stdev=8531.63 clat percentiles (usec): | 1.00th=[ 3163], 5.00th=[ 3687], 10.00th=[ 4080], 20.00th=[ 4817], | 30.00th=[ 5342], 40.00th=[ 6063], 50.00th=[ 6915], 60.00th=[ 8094], | 70.00th=[ 9765], 80.00th=[12780], 90.00th=[21103], 95.00th=[28443], | 99.00th=[44303], 99.50th=[51119], 99.90th=[68682], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 16, max= 360, per=25.36%, avg=147.34, stdev=71.72, samples=82 iops : min= 4, max= 90, avg=36.63, stdev=18.04, samples=82 write: IOPS=38, BW=152KiB/s (156kB/s)(6236KiB/41006msec) clat (msec): min=3, max=279, avg=16.49, stdev=11.22 lat (msec): min=3, max=279, avg=16.49, stdev=11.22 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 16], | 70.00th=[ 18], 80.00th=[ 23], 90.00th=[ 30], 95.00th=[ 36], | 99.00th=[ 48], 99.50th=[ 53], 99.90th=[ 68], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 32, max= 304, per=24.91%, avg=151.73, stdev=64.53, samples=82 iops : min= 8, max= 76, avg=37.73, stdev=16.23, samples=82 lat (msec) : 4=4.46%, 10=42.42%, 20=35.45%, 50=16.99%, 100=0.65% lat (msec) : 500=0.03% cpu : usr=0.15%, sys=6.74%, ctx=3418, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=582KiB/s (596kB/s), 144KiB/s-148KiB/s (148kB/s-151kB/s), io=23.4MiB (24.6MB), run=41006-41260msec WRITE: bw=609KiB/s (624kB/s), 152KiB/s-154KiB/s (156kB/s-158kB/s), io=24.6MiB (25.7MB), run=41006-41260msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21477: Thu May 1 19:04:48 2025 read: IOPS=44, BW=711KiB/s (729kB/s)(5920KiB/8321msec) clat (usec): min=92, max=52856, avg=12103.92, stdev=8674.81 lat (usec): min=93, max=52857, avg=12105.12, stdev=8675.32 clat percentiles (usec): | 1.00th=[ 97], 5.00th=[ 114], 10.00th=[ 155], 20.00th=[ 6194], | 30.00th=[ 7635], 40.00th=[ 8979], 50.00th=[10421], 60.00th=[12387], | 70.00th=[14615], 80.00th=[17957], 90.00th=[22676], 95.00th=[25822], | 99.00th=[47973], 99.50th=[51643], 99.90th=[52691], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 384, max= 992, per=24.42%, avg=692.94, stdev=164.41, samples=16 iops : min= 24, max= 62, avg=43.00, stdev=10.35, samples=16 write: IOPS=47, BW=765KiB/s (784kB/s)(6368KiB/8321msec) clat (usec): min=864, max=46760, avg=9190.22, stdev=9475.23 lat (usec): min=865, max=46761, avg=9193.82, stdev=9474.11 clat percentiles (usec): | 1.00th=[ 922], 5.00th=[ 1074], 10.00th=[ 1156], 20.00th=[ 1385], | 30.00th=[ 2024], 40.00th=[ 3261], 50.00th=[ 5735], 60.00th=[ 8029], | 70.00th=[11731], 80.00th=[16909], 90.00th=[23462], 95.00th=[28967], | 99.00th=[43779], 99.50th=[44827], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 384, max= 1184, per=24.95%, avg=744.88, stdev=212.40, samples=16 iops : min= 24, max= 74, avg=46.25, stdev=13.32, samples=16 lat (usec) : 100=1.04%, 250=4.04%, 500=0.39%, 1000=0.91% lat (msec) : 2=14.71%, 4=8.46%, 10=26.56%, 20=28.91%, 50=14.58% lat (msec) : 100=0.39% cpu : usr=0.17%, sys=14.30%, ctx=1280, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21478: Thu May 1 19:04:48 2025 read: IOPS=42, BW=681KiB/s (697kB/s)(5744KiB/8439msec) clat (usec): min=76, max=65967, avg=12707.15, stdev=9512.67 lat (usec): min=77, max=65969, avg=12708.13, stdev=9512.70 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 103], 10.00th=[ 122], 20.00th=[ 6390], | 30.00th=[ 8029], 40.00th=[ 9503], 50.00th=[11338], 60.00th=[13304], | 70.00th=[15139], 80.00th=[19268], 90.00th=[23987], 95.00th=[29754], | 99.00th=[48497], 99.50th=[58459], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 320, max= 832, per=23.57%, avg=668.88, stdev=136.73, samples=16 iops : min= 20, max= 52, avg=41.50, stdev= 8.46, samples=16 write: IOPS=48, BW=775KiB/s (794kB/s)(6544KiB/8439msec) clat (usec): min=671, max=49579, avg=9130.70, stdev=9235.86 lat (usec): min=672, max=49581, avg=9132.17, stdev=9235.88 clat percentiles (usec): | 1.00th=[ 947], 5.00th=[ 1057], 10.00th=[ 1172], 20.00th=[ 1434], | 30.00th=[ 2278], 40.00th=[ 3589], 50.00th=[ 5932], 60.00th=[ 8586], | 70.00th=[11600], 80.00th=[15533], 90.00th=[23725], 95.00th=[28181], | 99.00th=[40109], 99.50th=[42206], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 448, max= 1120, per=25.14%, avg=750.81, stdev=194.68, samples=16 iops : min= 28, max= 70, avg=46.62, stdev=12.23, samples=16 lat (usec) : 100=1.43%, 250=5.08%, 500=0.52%, 750=0.13%, 1000=0.91% lat (msec) : 2=13.28%, 4=8.59%, 10=24.09%, 20=30.73%, 50=14.84% lat (msec) : 100=0.39% cpu : usr=0.11%, sys=14.12%, ctx=1304, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21479: Thu May 1 19:04:48 2025 read: IOPS=46, BW=748KiB/s (766kB/s)(6256KiB/8366msec) clat (usec): min=89, max=95691, avg=12012.73, stdev=8823.91 lat (usec): min=90, max=95692, avg=12013.68, stdev=8823.88 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 109], 10.00th=[ 130], 20.00th=[ 6325], | 30.00th=[ 8029], 40.00th=[ 9372], 50.00th=[11338], 60.00th=[12649], | 70.00th=[14353], 80.00th=[16909], 90.00th=[21103], 95.00th=[24511], | 99.00th=[43779], 99.50th=[45351], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 480, max= 992, per=26.74%, avg=758.94, stdev=157.07, samples=16 iops : min= 30, max= 62, avg=47.19, stdev= 9.83, samples=16 write: IOPS=45, BW=721KiB/s (738kB/s)(6032KiB/8366msec) clat (usec): min=625, max=65928, avg=9631.65, stdev=10027.05 lat (usec): min=626, max=65930, avg=9633.15, stdev=10027.04 clat percentiles (usec): | 1.00th=[ 922], 5.00th=[ 1123], 10.00th=[ 1188], 20.00th=[ 1418], | 30.00th=[ 2245], 40.00th=[ 3359], 50.00th=[ 6325], 60.00th=[ 8586], | 70.00th=[12518], 80.00th=[17171], 90.00th=[22414], 95.00th=[28181], | 99.00th=[45876], 99.50th=[61604], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 288, max= 1088, per=23.88%, avg=712.94, stdev=204.40, samples=16 iops : min= 18, max= 68, avg=44.31, stdev=12.69, samples=16 lat (usec) : 100=1.17%, 250=4.69%, 500=0.13%, 750=0.13%, 1000=1.04% lat (msec) : 2=12.63%, 4=7.03%, 10=26.69%, 20=33.85%, 50=12.24% lat (msec) : 100=0.39% cpu : usr=0.12%, sys=14.25%, ctx=1420, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21480: Thu May 1 19:04:48 2025 read: IOPS=45, BW=727KiB/s (744kB/s)(6032KiB/8297msec) clat (usec): min=83, max=64491, avg=12389.19, stdev=9637.78 lat (usec): min=84, max=64492, avg=12390.13, stdev=9637.81 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 109], 10.00th=[ 122], 20.00th=[ 5866], | 30.00th=[ 7570], 40.00th=[ 9372], 50.00th=[11207], 60.00th=[13042], | 70.00th=[15008], 80.00th=[17957], 90.00th=[23200], 95.00th=[31851], | 99.00th=[43254], 99.50th=[60031], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 415, max= 992, per=24.91%, avg=706.94, stdev=149.43, samples=16 iops : min= 25, max= 62, avg=43.75, stdev= 9.43, samples=16 write: IOPS=47, BW=754KiB/s (772kB/s)(6256KiB/8297msec) clat (usec): min=842, max=54011, avg=8794.24, stdev=9073.20 lat (usec): min=843, max=54014, avg=8799.89, stdev=9086.74 clat percentiles (usec): | 1.00th=[ 963], 5.00th=[ 1057], 10.00th=[ 1123], 20.00th=[ 1352], | 30.00th=[ 1926], 40.00th=[ 3392], 50.00th=[ 5866], 60.00th=[ 8356], | 70.00th=[10945], 80.00th=[14615], 90.00th=[21103], 95.00th=[28443], | 99.00th=[38536], 99.50th=[42730], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 448, max= 992, per=24.68%, avg=736.88, stdev=177.79, samples=16 iops : min= 28, max= 62, avg=45.62, stdev=11.15, samples=16 lat (usec) : 100=1.04%, 250=6.25%, 500=0.13%, 750=0.13%, 1000=1.30% lat (msec) : 2=14.58%, 4=6.64%, 10=24.74%, 20=32.29%, 50=12.37% lat (msec) : 100=0.52% cpu : usr=0.06%, sys=14.14%, ctx=1362, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2838KiB/s (2906kB/s), 681KiB/s-748KiB/s (697kB/s-766kB/s), io=23.4MiB (24.5MB), run=8297-8439msec WRITE: bw=2986KiB/s (3058kB/s), 721KiB/s-775KiB/s (738kB/s-794kB/s), io=24.6MiB (25.8MB), run=8297-8439msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21481: Thu May 1 19:04:52 2025 read: IOPS=29, BW=475KiB/s (486kB/s)(5920KiB/12473msec) clat (usec): min=2770, max=84246, avg=12187.41, stdev=11264.22 lat (usec): min=2771, max=84246, avg=12188.55, stdev=11264.26 clat percentiles (usec): | 1.00th=[ 3326], 5.00th=[ 4146], 10.00th=[ 4817], 20.00th=[ 5735], | 30.00th=[ 6390], 40.00th=[ 7242], 50.00th=[ 8356], 60.00th=[ 9503], | 70.00th=[10945], 80.00th=[15401], 90.00th=[25297], 95.00th=[36963], | 99.00th=[57410], 99.50th=[73925], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 191, max= 1053, per=24.53%, avg=464.62, stdev=240.02, samples=24 iops : min= 11, max= 65, avg=28.58, stdev=15.05, samples=24 write: IOPS=31, BW=511KiB/s (523kB/s)(6368KiB/12473msec) clat (usec): min=5208, max=98809, avg=19553.87, stdev=11195.59 lat (usec): min=5209, max=98810, avg=19555.84, stdev=11195.50 clat percentiles (usec): | 1.00th=[ 5997], 5.00th=[ 8717], 10.00th=[ 9634], 20.00th=[11076], | 30.00th=[12387], 40.00th=[13698], 50.00th=[15533], 60.00th=[18220], | 70.00th=[22414], 80.00th=[27395], 90.00th=[36963], 95.00th=[41681], | 99.00th=[56361], 99.50th=[58459], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 192, max= 896, per=25.05%, avg=499.25, stdev=223.86, samples=24 iops : min= 12, max= 56, avg=30.75, stdev=14.08, samples=24 lat (msec) : 4=1.69%, 10=35.55%, 20=38.67%, 50=22.40%, 100=1.69% cpu : usr=0.15%, sys=6.02%, ctx=914, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21482: Thu May 1 19:04:52 2025 read: IOPS=28, BW=458KiB/s (469kB/s)(5744KiB/12534msec) clat (usec): min=2945, max=82820, avg=12089.16, stdev=11300.87 lat (usec): min=2946, max=82821, avg=12090.53, stdev=11300.67 clat percentiles (usec): | 1.00th=[ 3785], 5.00th=[ 4228], 10.00th=[ 4621], 20.00th=[ 5342], | 30.00th=[ 5866], 40.00th=[ 7046], 50.00th=[ 8029], 60.00th=[ 9503], | 70.00th=[11600], 80.00th=[15664], 90.00th=[26870], 95.00th=[37487], | 99.00th=[57410], 99.50th=[68682], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 192, max= 864, per=24.15%, avg=457.44, stdev=223.58, samples=25 iops : min= 12, max= 54, avg=28.28, stdev=14.02, samples=25 write: IOPS=32, BW=522KiB/s (535kB/s)(6544KiB/12534msec) clat (usec): min=6867, max=81561, avg=19929.59, stdev=12268.89 lat (usec): min=6869, max=81562, avg=19931.51, stdev=12268.92 clat percentiles (usec): | 1.00th=[ 7242], 5.00th=[ 8455], 10.00th=[ 9634], 20.00th=[11207], | 30.00th=[12780], 40.00th=[14091], 50.00th=[15795], 60.00th=[17957], | 70.00th=[21103], 80.00th=[26346], 90.00th=[34866], 95.00th=[44303], | 99.00th=[69731], 99.50th=[73925], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 287, max= 1024, per=26.10%, avg=520.24, stdev=224.83, samples=25 iops : min= 17, max= 64, avg=32.20, stdev=14.15, samples=25 lat (msec) : 4=1.04%, 10=35.03%, 20=40.10%, 50=21.22%, 100=2.60% cpu : usr=0.15%, sys=6.04%, ctx=869, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21483: Thu May 1 19:04:52 2025 read: IOPS=30, BW=495KiB/s (507kB/s)(6256KiB/12640msec) clat (usec): min=2573, max=86335, avg=12547.22, stdev=11187.41 lat (usec): min=2574, max=86337, avg=12548.37, stdev=11187.34 clat percentiles (usec): | 1.00th=[ 3720], 5.00th=[ 4178], 10.00th=[ 4752], 20.00th=[ 5538], | 30.00th=[ 6390], 40.00th=[ 7308], 50.00th=[ 8225], 60.00th=[ 9372], | 70.00th=[12256], 80.00th=[16188], 90.00th=[28181], 95.00th=[39060], | 99.00th=[50070], 99.50th=[63701], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 160, max= 1248, per=26.12%, avg=494.68, stdev=282.70, samples=25 iops : min= 10, max= 78, avg=30.60, stdev=17.73, samples=25 write: IOPS=29, BW=477KiB/s (489kB/s)(6032KiB/12640msec) clat (usec): min=5740, max=98079, avg=20283.87, stdev=12534.61 lat (usec): min=5742, max=98081, avg=20286.10, stdev=12534.31 clat percentiles (usec): | 1.00th=[ 6849], 5.00th=[ 8586], 10.00th=[ 9896], 20.00th=[11076], | 30.00th=[12256], 40.00th=[14222], 50.00th=[16319], 60.00th=[19006], | 70.00th=[23462], 80.00th=[27395], 90.00th=[35914], 95.00th=[45351], | 99.00th=[73925], 99.50th=[79168], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 191, max= 800, per=23.54%, avg=469.08, stdev=207.33, samples=25 iops : min= 11, max= 50, avg=29.00, stdev=13.02, samples=25 lat (msec) : 4=1.56%, 10=35.55%, 20=37.11%, 50=23.31%, 100=2.47% cpu : usr=0.16%, sys=5.93%, ctx=873, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21484: Thu May 1 19:04:52 2025 read: IOPS=30, BW=487KiB/s (499kB/s)(6032KiB/12377msec) clat (usec): min=2983, max=82411, avg=12804.37, stdev=11227.16 lat (usec): min=2985, max=82412, avg=12805.66, stdev=11227.02 clat percentiles (usec): | 1.00th=[ 3490], 5.00th=[ 4424], 10.00th=[ 4883], 20.00th=[ 5538], | 30.00th=[ 6259], 40.00th=[ 7111], 50.00th=[ 8291], 60.00th=[ 9896], | 70.00th=[12387], 80.00th=[17433], 90.00th=[28967], 95.00th=[38011], | 99.00th=[54264], 99.50th=[57934], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 224, max= 992, per=25.02%, avg=473.92, stdev=231.87, samples=24 iops : min= 14, max= 62, avg=29.21, stdev=14.57, samples=24 write: IOPS=31, BW=505KiB/s (518kB/s)(6256KiB/12377msec) clat (usec): min=6170, max=72036, avg=19029.57, stdev=11321.45 lat (usec): min=6172, max=72038, avg=19032.84, stdev=11321.16 clat percentiles (usec): | 1.00th=[ 7046], 5.00th=[ 8717], 10.00th=[ 9765], 20.00th=[10814], | 30.00th=[12649], 40.00th=[13960], 50.00th=[15270], 60.00th=[17171], | 70.00th=[20579], 80.00th=[25822], 90.00th=[33162], 95.00th=[40109], | 99.00th=[67634], 99.50th=[69731], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 223, max= 990, per=24.71%, avg=492.54, stdev=235.33, samples=24 iops : min= 13, max= 61, avg=30.37, stdev=14.77, samples=24 lat (msec) : 4=1.04%, 10=34.51%, 20=39.97%, 50=22.14%, 100=2.34% cpu : usr=0.00%, sys=6.29%, ctx=894, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1895KiB/s (1940kB/s), 458KiB/s-495KiB/s (469kB/s-507kB/s), io=23.4MiB (24.5MB), run=12377-12640msec WRITE: bw=1994KiB/s (2042kB/s), 477KiB/s-522KiB/s (489kB/s-535kB/s), io=24.6MiB (25.8MB), run=12377-12640msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21500: Thu May 1 19:04:55 2025 read: IOPS=5, BW=6095KiB/s (6242kB/s)(7168KiB/1176msec) clat (usec): min=485, max=171034, avg=62737.73, stdev=56996.69 lat (usec): min=487, max=171035, avg=62738.95, stdev=56996.64 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 486], 10.00th=[ 486], 20.00th=[ 2737], | 30.00th=[ 57934], 40.00th=[ 57934], 50.00th=[ 60556], 60.00th=[ 68682], | 70.00th=[ 68682], 80.00th=[ 78119], 90.00th=[170918], 95.00th=[170918], | 99.00th=[170918], 99.50th=[170918], 99.90th=[170918], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 4096, max= 6144, per=31.60%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=4, BW=4354KiB/s (4458kB/s)(5120KiB/1176msec) clat (msec): min=47, max=167, avg=111.13, stdev=54.23 lat (msec): min=47, max=168, avg=111.30, stdev=54.29 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 48], | 30.00th=[ 60], 40.00th=[ 60], 50.00th=[ 132], 60.00th=[ 132], | 70.00th=[ 150], 80.00th=[ 150], 90.00th=[ 169], 95.00th=[ 169], | 99.00th=[ 169], 99.50th=[ 169], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 6144, max= 6144, per=27.09%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 4=8.33%, 50=8.33%, 100=41.67%, 250=33.33% cpu : usr=0.09%, sys=17.79%, ctx=629, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21504: Thu May 1 19:04:55 2025 read: IOPS=1, BW=1660KiB/s (1699kB/s)(2048KiB/1234msec) clat (usec): min=514, max=76773, avg=38643.57, stdev=53923.17 lat (usec): min=515, max=76775, avg=38645.30, stdev=53924.36 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 515], 10.00th=[ 515], 20.00th=[ 515], | 30.00th=[ 515], 40.00th=[ 515], 50.00th=[ 515], 60.00th=[77071], | 70.00th=[77071], 80.00th=[77071], 90.00th=[77071], 95.00th=[77071], | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 2043, max= 2043, per=12.61%, avg=2043.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=8, BW=8298KiB/s (8497kB/s)(10.0MiB/1234msec) clat (msec): min=39, max=224, avg=97.18, stdev=63.86 lat (msec): min=39, max=224, avg=97.25, stdev=63.84 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 42], | 30.00th=[ 54], 40.00th=[ 54], 50.00th=[ 70], 60.00th=[ 81], | 70.00th=[ 110], 80.00th=[ 110], 90.00th=[ 192], 95.00th=[ 226], | 99.00th=[ 226], 99.50th=[ 226], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 4087, max= 8175, per=27.03%, avg=6131.00, stdev=2890.65, samples=2 iops : min= 3, max= 7, avg= 5.00, stdev= 2.83, samples=2 lat (usec) : 750=8.33% lat (msec) : 50=16.67%, 100=41.67%, 250=33.33% cpu : usr=0.00%, sys=19.87%, ctx=147, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21505: Thu May 1 19:04:55 2025 read: IOPS=5, BW=5699KiB/s (5836kB/s)(6144KiB/1078msec) clat (usec): min=467, max=149730, avg=62729.76, stdev=48811.77 lat (usec): min=467, max=149732, avg=62730.89, stdev=48812.10 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 469], 10.00th=[ 469], 20.00th=[ 45876], | 30.00th=[ 45876], 40.00th=[ 54789], 50.00th=[ 54789], 60.00th=[ 54789], | 70.00th=[ 70779], 80.00th=[ 70779], 90.00th=[149947], 95.00th=[149947], | 99.00th=[149947], 99.50th=[149947], 99.90th=[149947], 99.95th=[149947], | 99.99th=[149947] bw ( KiB/s): min= 4096, max= 6144, per=31.60%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=5, BW=5699KiB/s (5836kB/s)(6144KiB/1078msec) clat (msec): min=52, max=270, avg=109.13, stdev=85.04 lat (msec): min=52, max=270, avg=109.18, stdev=85.04 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 60], | 30.00th=[ 60], 40.00th=[ 61], 50.00th=[ 61], 60.00th=[ 74], | 70.00th=[ 140], 80.00th=[ 140], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 4096, max= 6144, per=22.57%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (usec) : 500=8.33% lat (msec) : 50=8.33%, 100=58.33%, 250=16.67%, 500=8.33% cpu : usr=0.19%, sys=19.41%, ctx=551, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21507: Thu May 1 19:04:55 2025 read: IOPS=3, BW=4051KiB/s (4148kB/s)(5120KiB/1264msec) clat (msec): min=48, max=105, avg=64.70, stdev=23.12 lat (msec): min=48, max=105, avg=64.70, stdev=23.12 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 48], | 30.00th=[ 56], 40.00th=[ 56], 50.00th=[ 57], 60.00th=[ 57], | 70.00th=[ 58], 80.00th=[ 58], 90.00th=[ 106], 95.00th=[ 106], | 99.00th=[ 106], 99.50th=[ 106], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 8159, max= 8159, per=50.36%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5671KiB/s (5807kB/s)(7168KiB/1264msec) clat (msec): min=29, max=215, avg=105.93, stdev=82.17 lat (msec): min=29, max=215, avg=106.06, stdev=82.11 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 32], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 59], 60.00th=[ 174], | 70.00th=[ 174], 80.00th=[ 188], 90.00th=[ 215], 95.00th=[ 215], | 99.00th=[ 215], 99.50th=[ 215], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 2048, max=10199, per=27.00%, avg=6123.50, stdev=5763.63, samples=2 iops : min= 2, max= 9, avg= 5.50, stdev= 4.95, samples=2 lat (msec) : 50=33.33%, 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=20.74%, ctx=736, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.8MiB/s (16.6MB/s), 1660KiB/s-6095KiB/s (1699kB/s-6242kB/s), io=20.0MiB (20.0MB), run=1078-1264msec WRITE: bw=22.2MiB/s (23.2MB/s), 4354KiB/s-8298KiB/s (4458kB/s-8497kB/s), io=28.0MiB (29.4MB), run=1078-1264msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21501: Thu May 1 19:04:55 2025 read: IOPS=6, BW=6788KiB/s (6951kB/s)(7168KiB/1056msec) clat (msec): min=24, max=133, avg=57.63, stdev=45.04 lat (msec): min=24, max=133, avg=57.63, stdev=45.04 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 40], 60.00th=[ 42], | 70.00th=[ 42], 80.00th=[ 111], 90.00th=[ 134], 95.00th=[ 134], | 99.00th=[ 134], 99.50th=[ 134], 99.90th=[ 134], 99.95th=[ 134], | 99.99th=[ 134] bw ( KiB/s): min=10219, max=10219, per=68.66%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4848KiB/s (4965kB/s)(5120KiB/1056msec) clat (msec): min=37, max=188, avg=109.02, stdev=57.50 lat (msec): min=37, max=188, avg=109.11, stdev=57.53 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 38], | 30.00th=[ 72], 40.00th=[ 72], 50.00th=[ 120], 60.00th=[ 120], | 70.00th=[ 129], 80.00th=[ 129], 90.00th=[ 188], 95.00th=[ 188], | 99.00th=[ 188], 99.50th=[ 188], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 2043, max= 2043, per=9.80%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=8.33%, 250=41.67% cpu : usr=0.00%, sys=3.70%, ctx=33, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21502: Thu May 1 19:04:55 2025 read: IOPS=1, BW=1488KiB/s (1524kB/s)(2048KiB/1376msec) clat (msec): min=24, max=143, avg=84.42, stdev=84.18 lat (msec): min=24, max=143, avg=84.42, stdev=84.18 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 25], 40.00th=[ 25], 50.00th=[ 25], 60.00th=[ 144], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 1851, max= 2043, per=13.08%, avg=1947.00, stdev=135.76, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=7, BW=7442KiB/s (7620kB/s)(10.0MiB/1376msec) clat (msec): min=24, max=305, avg=116.48, stdev=113.42 lat (msec): min=24, max=305, avg=116.58, stdev=113.41 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 32], | 30.00th=[ 42], 40.00th=[ 46], 50.00th=[ 51], 60.00th=[ 59], | 70.00th=[ 241], 80.00th=[ 241], 90.00th=[ 288], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 3703, max= 8175, per=28.50%, avg=5939.00, stdev=3162.18, samples=2 iops : min= 3, max= 7, avg= 5.00, stdev= 2.83, samples=2 lat (msec) : 50=41.67%, 100=25.00%, 250=16.67%, 500=16.67% cpu : usr=0.00%, sys=6.84%, ctx=61, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21503: Thu May 1 19:04:55 2025 read: IOPS=4, BW=4478KiB/s (4586kB/s)(6144KiB/1372msec) clat (msec): min=9, max=252, avg=85.36, stdev=94.37 lat (msec): min=9, max=252, avg=85.36, stdev=94.37 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 10], 10.00th=[ 10], 20.00th=[ 21], | 30.00th=[ 21], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 51], | 70.00th=[ 142], 80.00th=[ 142], 90.00th=[ 253], 95.00th=[ 253], | 99.00th=[ 253], 99.50th=[ 253], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 8175, max= 8175, per=54.93%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4478KiB/s (4586kB/s)(6144KiB/1372msec) clat (msec): min=34, max=203, avg=123.56, stdev=61.77 lat (msec): min=34, max=203, avg=123.62, stdev=61.77 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 77], | 30.00th=[ 77], 40.00th=[ 115], 50.00th=[ 115], 60.00th=[ 146], | 70.00th=[ 167], 80.00th=[ 167], 90.00th=[ 205], 95.00th=[ 205], | 99.00th=[ 205], 99.50th=[ 205], 99.90th=[ 205], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 4087, max= 4096, per=19.64%, avg=4091.50, stdev= 6.36, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 lat (msec) : 10=8.33%, 50=25.00%, 100=16.67%, 250=41.67%, 500=8.33% cpu : usr=0.00%, sys=4.74%, ctx=41, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21506: Thu May 1 19:04:55 2025 read: IOPS=5, BW=5577KiB/s (5711kB/s)(5120KiB/918msec) clat (msec): min=24, max=133, avg=69.26, stdev=50.16 lat (msec): min=24, max=133, avg=69.26, stdev=50.16 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 45], 60.00th=[ 45], | 70.00th=[ 113], 80.00th=[ 113], 90.00th=[ 134], 95.00th=[ 134], | 99.00th=[ 134], 99.50th=[ 134], 99.90th=[ 134], 99.95th=[ 134], | 99.99th=[ 134] bw ( KiB/s): min= 8192, max= 8192, per=55.04%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7808KiB/s (7996kB/s)(7168KiB/918msec) clat (msec): min=24, max=192, avg=78.62, stdev=63.43 lat (msec): min=24, max=193, avg=78.87, stdev=63.80 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 27], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 50], 60.00th=[ 103], | 70.00th=[ 103], 80.00th=[ 124], 90.00th=[ 192], 95.00th=[ 192], | 99.00th=[ 192], 99.50th=[ 192], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 8192, max= 8192, per=39.31%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 250=41.67% cpu : usr=0.00%, sys=4.25%, ctx=31, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=14.5MiB/s (15.2MB/s), 1488KiB/s-6788KiB/s (1524kB/s-6951kB/s), io=20.0MiB (20.0MB), run=918-1376msec WRITE: bw=20.3MiB/s (21.3MB/s), 4478KiB/s-7808KiB/s (4586kB/s-7996kB/s), io=28.0MiB (29.4MB), run=918-1376msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21515: Thu May 1 19:04:57 2025 read: IOPS=4, BW=18.8MiB/s (19.8MB/s)(12.0MiB/637msec) clat (msec): min=169, max=229, avg=201.72, stdev=30.13 lat (msec): min=169, max=229, avg=201.72, stdev=30.13 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 205], 50.00th=[ 205], 60.00th=[ 205], | 70.00th=[ 230], 80.00th=[ 230], 90.00th=[ 230], 95.00th=[ 230], | 99.00th=[ 230], 99.50th=[ 230], 99.90th=[ 230], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min=16253, max=16253, per=100.00%, avg=16253.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=33.65%, ctx=1165, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21516: Thu May 1 19:04:57 2025 read: IOPS=0, BW=3886KiB/s (3979kB/s)(4096KiB/1054msec) clat (nsec): min=119330k, max=119330k, avg=119330157.00, stdev= 0.00 lat (nsec): min=119331k, max=119331k, avg=119331395.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 120], 50.00th=[ 120], 60.00th=[ 120], | 70.00th=[ 120], 80.00th=[ 120], 90.00th=[ 120], 95.00th=[ 120], | 99.00th=[ 120], 99.50th=[ 120], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 8175, max= 8175, per=52.59%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=7772KiB/s (7959kB/s)(8192KiB/1054msec) clat (msec): min=320, max=560, avg=440.88, stdev=169.78 lat (msec): min=321, max=561, avg=441.12, stdev=169.63 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 321], 10.00th=[ 321], 20.00th=[ 321], | 30.00th=[ 321], 40.00th=[ 321], 50.00th=[ 321], 60.00th=[ 558], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 7922, max= 7922, per=26.35%, avg=7922.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=21.84%, ctx=541, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21517: Thu May 1 19:04:57 2025 write: IOPS=2, BW=11.0MiB/s (11.5MB/s)(12.0MiB/1090msec) clat (msec): min=176, max=452, avg=337.07, stdev=143.65 lat (msec): min=176, max=453, avg=337.31, stdev=143.61 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 178], 10.00th=[ 178], 20.00th=[ 178], | 30.00th=[ 178], 40.00th=[ 384], 50.00th=[ 384], 60.00th=[ 384], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 7543, max= 8175, per=26.14%, avg=7859.00, stdev=446.89, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.09%, sys=27.82%, ctx=61, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21518: Thu May 1 19:04:57 2025 write: IOPS=4, BW=19.2MiB/s (20.1MB/s)(12.0MiB/626msec) clat (msec): min=175, max=234, avg=202.33, stdev=30.13 lat (msec): min=175, max=234, avg=202.59, stdev=29.99 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 176], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 197], 50.00th=[ 197], 60.00th=[ 197], | 70.00th=[ 234], 80.00th=[ 234], 90.00th=[ 234], 95.00th=[ 234], | 99.00th=[ 234], 99.50th=[ 234], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min=16384, max=16384, per=54.50%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=48.16%, ctx=63, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.2MiB/s (15.9MB/s), 3886KiB/s-18.8MiB/s (3979kB/s-19.8MB/s), io=16.0MiB (16.8MB), run=637-1054msec WRITE: bw=29.4MiB/s (30.8MB/s), 7772KiB/s-19.2MiB/s (7959kB/s-20.1MB/s), io=32.0MiB (33.6MB), run=626-1090msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21519: Thu May 1 19:04:58 2025 read: IOPS=2, BW=9902KiB/s (10.1MB/s)(12.0MiB/1241msec) clat (msec): min=271, max=484, avg=392.43, stdev=109.63 lat (msec): min=271, max=484, avg=392.43, stdev=109.63 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 271], 10.00th=[ 271], 20.00th=[ 271], | 30.00th=[ 271], 40.00th=[ 422], 50.00th=[ 422], 60.00th=[ 422], | 70.00th=[ 485], 80.00th=[ 485], 90.00th=[ 485], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 8192, max= 8192, per=65.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=100.00% cpu : usr=0.24%, sys=5.97%, ctx=43, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21520: Thu May 1 19:04:58 2025 read: IOPS=0, BW=3151KiB/s (3226kB/s)(4096KiB/1300msec) clat (nsec): min=299043k, max=299043k, avg=299043042.00, stdev= 0.00 lat (nsec): min=299045k, max=299045k, avg=299044635.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 8015, max= 8015, per=63.60%, avg=8015.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6302KiB/s (6453kB/s)(8192KiB/1300msec) clat (msec): min=443, max=471, avg=457.47, stdev=19.89 lat (msec): min=444, max=471, avg=458.33, stdev=19.12 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 443], 10.00th=[ 443], 20.00th=[ 443], | 30.00th=[ 443], 40.00th=[ 443], 50.00th=[ 443], 60.00th=[ 472], | 70.00th=[ 472], 80.00th=[ 472], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 8062, max= 8062, per=32.38%, avg=8062.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=100.00% cpu : usr=0.46%, sys=9.16%, ctx=92, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21521: Thu May 1 19:04:58 2025 write: IOPS=2, BW=9337KiB/s (9561kB/s)(12.0MiB/1316msec) clat (msec): min=337, max=507, avg=409.62, stdev=88.05 lat (msec): min=337, max=507, avg=409.90, stdev=88.04 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 338], 10.00th=[ 338], 20.00th=[ 338], | 30.00th=[ 338], 40.00th=[ 384], 50.00th=[ 384], 60.00th=[ 384], | 70.00th=[ 510], 80.00th=[ 510], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 8078, max= 8110, per=32.51%, avg=8094.00, stdev=22.63, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.23%, sys=1.52%, ctx=17, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21522: Thu May 1 19:04:58 2025 write: IOPS=3, BW=12.3MiB/s (12.9MB/s)(12.0MiB/976msec) clat (msec): min=117, max=447, avg=315.91, stdev=174.79 lat (msec): min=117, max=448, avg=316.21, stdev=174.80 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 118], 10.00th=[ 118], 20.00th=[ 118], | 30.00th=[ 118], 40.00th=[ 384], 50.00th=[ 384], 60.00th=[ 384], | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 7968, max= 7968, per=32.00%, avg=7968.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.31%, sys=8.00%, ctx=51, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.3MiB/s (12.9MB/s), 3151KiB/s-9902KiB/s (3226kB/s-10.1MB/s), io=16.0MiB (16.8MB), run=1241-1300msec WRITE: bw=24.3MiB/s (25.5MB/s), 6302KiB/s-12.3MiB/s (6453kB/s-12.9MB/s), io=32.0MiB (33.6MB), run=976-1316msec