== sanity test 398b: DIO and buffer IO race ============== 09:07:45 (1740492465) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 0.958743 s, 52.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5888: Tue Feb 25 09:08:01 2025 read: IOPS=114, BW=460KiB/s (471kB/s)(5992KiB/13036msec) clat (usec): min=54, max=143078, avg=3873.87, stdev=4754.47 lat (usec): min=55, max=143079, avg=3874.92, stdev=4754.36 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 71], 10.00th=[ 83], 20.00th=[ 330], | 30.00th=[ 2343], 40.00th=[ 2769], 50.00th=[ 3261], 60.00th=[ 3785], | 70.00th=[ 4752], 80.00th=[ 6063], 90.00th=[ 7701], 95.00th=[ 8979], | 99.00th=[ 13829], 99.50th=[ 16319], 99.90th=[ 31589], 99.95th=[143655], | 99.99th=[143655] bw ( KiB/s): min= 344, max= 686, per=25.46%, avg=459.50, stdev=83.03, samples=26 iops : min= 86, max= 171, avg=114.81, stdev=20.67, samples=26 write: IOPS=120, BW=483KiB/s (495kB/s)(6296KiB/13036msec) clat (usec): min=467, max=35582, avg=4575.02, stdev=4285.96 lat (usec): min=468, max=35583, avg=4576.51, stdev=4285.33 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 537], 10.00th=[ 578], 20.00th=[ 709], | 30.00th=[ 1106], 40.00th=[ 2442], 50.00th=[ 3752], 60.00th=[ 4752], | 70.00th=[ 5932], 80.00th=[ 7570], 90.00th=[10159], 95.00th=[12649], | 99.00th=[17957], 99.50th=[22676], 99.90th=[31851], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 344, max= 640, per=25.54%, avg=482.62, stdev=73.77, samples=26 iops : min= 86, max= 160, avg=120.58, stdev=18.45, samples=26 lat (usec) : 100=7.36%, 250=2.21%, 500=0.65%, 750=10.87%, 1000=3.58% lat (msec) : 2=5.83%, 4=26.76%, 10=35.81%, 20=6.45%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=23.01%, ctx=5644, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5889: Tue Feb 25 09:08:01 2025 read: IOPS=112, BW=450KiB/s (461kB/s)(5940KiB/13192msec) clat (usec): min=51, max=158197, avg=3968.51, stdev=5252.74 lat (usec): min=52, max=158198, avg=3969.33, stdev=5252.73 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 72], 10.00th=[ 84], 20.00th=[ 200], | 30.00th=[ 2343], 40.00th=[ 2769], 50.00th=[ 3195], 60.00th=[ 3818], | 70.00th=[ 4752], 80.00th=[ 6194], 90.00th=[ 8029], 95.00th=[ 9634], | 99.00th=[ 13566], 99.50th=[ 16909], 99.90th=[ 50070], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 152, max= 640, per=24.86%, avg=448.81, stdev=99.39, samples=26 iops : min= 38, max= 160, avg=112.15, stdev=24.80, samples=26 write: IOPS=120, BW=481KiB/s (493kB/s)(6348KiB/13192msec) clat (usec): min=474, max=36902, avg=4574.74, stdev=4466.25 lat (usec): min=474, max=36903, avg=4575.95, stdev=4466.06 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 545], 10.00th=[ 570], 20.00th=[ 685], | 30.00th=[ 988], 40.00th=[ 2212], 50.00th=[ 3589], 60.00th=[ 4621], | 70.00th=[ 5866], 80.00th=[ 7570], 90.00th=[10290], 95.00th=[13173], | 99.00th=[20579], 99.50th=[23987], 99.90th=[31589], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 216, max= 686, per=25.16%, avg=475.58, stdev=93.93, samples=26 iops : min= 54, max= 171, avg=118.85, stdev=23.43, samples=26 lat (usec) : 100=7.42%, 250=2.28%, 500=0.29%, 750=12.04%, 1000=3.61% lat (msec) : 2=5.31%, 4=26.99%, 10=34.67%, 20=6.61%, 50=0.72% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.19%, sys=22.83%, ctx=5647, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5890: Tue Feb 25 09:08:01 2025 read: IOPS=115, BW=461KiB/s (472kB/s)(6028KiB/13086msec) clat (usec): min=52, max=36680, avg=3803.91, stdev=3111.62 lat (usec): min=52, max=36680, avg=3804.57, stdev=3111.63 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 72], 10.00th=[ 83], 20.00th=[ 1811], | 30.00th=[ 2442], 40.00th=[ 2802], 50.00th=[ 3195], 60.00th=[ 3785], | 70.00th=[ 4686], 80.00th=[ 5932], 90.00th=[ 7701], 95.00th=[ 9241], | 99.00th=[13698], 99.50th=[15401], 99.90th=[28443], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 344, max= 616, per=25.47%, avg=459.77, stdev=78.16, samples=26 iops : min= 86, max= 154, avg=114.85, stdev=19.54, samples=26 write: IOPS=119, BW=478KiB/s (490kB/s)(6260KiB/13086msec) clat (usec): min=460, max=148645, avg=4666.10, stdev=5963.65 lat (usec): min=460, max=148646, avg=4667.03, stdev=5963.56 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 545], 10.00th=[ 578], 20.00th=[ 701], | 30.00th=[ 996], 40.00th=[ 2089], 50.00th=[ 3752], 60.00th=[ 4817], | 70.00th=[ 6128], 80.00th=[ 7504], 90.00th=[ 9765], 95.00th=[ 12387], | 99.00th=[ 18220], 99.50th=[ 26870], 99.90th=[ 66323], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 383, max= 584, per=25.22%, avg=476.73, stdev=63.45, samples=26 iops : min= 95, max= 146, avg=119.08, stdev=15.93, samples=26 lat (usec) : 100=7.32%, 250=1.95%, 500=0.59%, 750=10.68%, 1000=4.36% lat (msec) : 2=5.70%, 4=26.99%, 10=36.07%, 20=5.83%, 50=0.42% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.21%, sys=23.20%, ctx=5527, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5891: Tue Feb 25 09:08:01 2025 read: IOPS=113, BW=455KiB/s (466kB/s)(6052KiB/13301msec) clat (usec): min=51, max=48895, avg=3756.37, stdev=3600.22 lat (usec): min=51, max=48896, avg=3758.43, stdev=3600.09 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 70], 10.00th=[ 78], 20.00th=[ 109], | 30.00th=[ 2245], 40.00th=[ 2671], 50.00th=[ 3130], 60.00th=[ 3752], | 70.00th=[ 4817], 80.00th=[ 5997], 90.00th=[ 7832], 95.00th=[ 9503], | 99.00th=[14877], 99.50th=[18744], 99.90th=[39060], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 96, max= 606, per=24.85%, avg=448.62, stdev=97.73, samples=26 iops : min= 24, max= 151, avg=112.00, stdev=24.41, samples=26 write: IOPS=117, BW=469KiB/s (480kB/s)(6236KiB/13301msec) clat (usec): min=481, max=147062, avg=4853.75, stdev=5762.12 lat (usec): min=482, max=147063, avg=4854.50, stdev=5762.11 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 537], 10.00th=[ 570], 20.00th=[ 676], | 30.00th=[ 1106], 40.00th=[ 2737], 50.00th=[ 3916], 60.00th=[ 4948], | 70.00th=[ 6063], 80.00th=[ 7832], 90.00th=[ 10683], 95.00th=[ 13435], | 99.00th=[ 20579], 99.50th=[ 22676], 99.90th=[ 35390], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 104, max= 672, per=24.45%, avg=462.15, stdev=100.59, samples=26 iops : min= 26, max= 168, avg=115.38, stdev=25.17, samples=26 lat (usec) : 100=9.21%, 250=2.34%, 500=0.75%, 750=11.30%, 1000=2.77% lat (msec) : 2=5.08%, 4=25.85%, 10=34.67%, 20=7.32%, 50=0.68% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=22.42%, ctx=5597, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1805KiB/s (1849kB/s), 450KiB/s-461KiB/s (461kB/s-472kB/s), io=23.4MiB (24.6MB), run=13036-13301msec WRITE: bw=1890KiB/s (1935kB/s), 469KiB/s-483KiB/s (480kB/s-495kB/s), io=24.6MiB (25.7MB), run=13036-13301msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5882: Tue Feb 25 09:08:27 2025 read: IOPS=38, BW=153KiB/s (157kB/s)(5992KiB/39056msec) clat (usec): min=1514, max=163044, avg=4681.12, stdev=4910.50 lat (usec): min=1515, max=163047, avg=4682.03, stdev=4910.50 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1942], 10.00th=[ 2212], 20.00th=[ 2868], | 30.00th=[ 3294], 40.00th=[ 3589], 50.00th=[ 3851], 60.00th=[ 4228], | 70.00th=[ 4686], 80.00th=[ 5342], 90.00th=[ 8029], 95.00th=[ 10290], | 99.00th=[ 14222], 99.50th=[ 16712], 99.90th=[ 32113], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 56, max= 272, per=25.17%, avg=153.53, stdev=57.76, samples=77 iops : min= 14, max= 68, avg=38.29, stdev=14.43, samples=77 write: IOPS=40, BW=161KiB/s (165kB/s)(6296KiB/39056msec) clat (usec): min=9475, max=88966, avg=20301.10, stdev=7345.24 lat (usec): min=9476, max=88967, avg=20302.20, stdev=7345.12 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11994], 10.00th=[12780], 20.00th=[14353], | 30.00th=[15795], 40.00th=[17433], 50.00th=[18744], 60.00th=[20317], | 70.00th=[22152], 80.00th=[25297], 90.00th=[29754], 95.00th=[34341], | 99.00th=[43779], 99.50th=[49021], 99.90th=[68682], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 80, max= 232, per=25.12%, avg=160.49, stdev=41.50, samples=77 iops : min= 20, max= 58, avg=40.03, stdev=10.39, samples=77 lat (msec) : 2=3.09%, 4=23.08%, 10=20.05%, 20=32.29%, 50=21.26% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.14%, sys=5.69%, ctx=4098, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5883: Tue Feb 25 09:08:27 2025 read: IOPS=37, BW=151KiB/s (155kB/s)(5940KiB/39325msec) clat (usec): min=1549, max=182184, avg=4797.39, stdev=5734.29 lat (usec): min=1549, max=182187, avg=4798.27, stdev=5734.32 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1893], 10.00th=[ 2114], 20.00th=[ 2802], | 30.00th=[ 3294], 40.00th=[ 3621], 50.00th=[ 3884], 60.00th=[ 4228], | 70.00th=[ 4686], 80.00th=[ 5669], 90.00th=[ 8094], 95.00th=[ 10290], | 99.00th=[ 15926], 99.50th=[ 22152], 99.90th=[ 70779], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 40, max= 279, per=24.80%, avg=151.26, stdev=60.37, samples=78 iops : min= 10, max= 69, avg=37.72, stdev=15.07, samples=78 write: IOPS=40, BW=161KiB/s (165kB/s)(6348KiB/39325msec) clat (usec): min=9774, max=56977, avg=20236.25, stdev=7192.91 lat (usec): min=9775, max=56978, avg=20237.23, stdev=7192.89 clat percentiles (usec): | 1.00th=[10552], 5.00th=[12125], 10.00th=[12911], 20.00th=[14222], | 30.00th=[15533], 40.00th=[16909], 50.00th=[18744], 60.00th=[20317], | 70.00th=[22414], 80.00th=[25035], 90.00th=[30016], 95.00th=[34866], | 99.00th=[43254], 99.50th=[47449], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 88, max= 224, per=25.20%, avg=161.00, stdev=41.35, samples=78 iops : min= 22, max= 56, avg=40.15, stdev=10.36, samples=78 lat (msec) : 2=3.68%, 4=22.10%, 10=20.12%, 20=32.36%, 50=21.55% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.11%, sys=5.70%, ctx=4106, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5885: Tue Feb 25 09:08:27 2025 read: IOPS=38, BW=153KiB/s (157kB/s)(6028KiB/39315msec) clat (usec): min=1435, max=70682, avg=4798.54, stdev=3517.66 lat (usec): min=1435, max=70683, avg=4799.36, stdev=3517.66 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1909], 10.00th=[ 2147], 20.00th=[ 2900], | 30.00th=[ 3359], 40.00th=[ 3654], 50.00th=[ 3982], 60.00th=[ 4293], | 70.00th=[ 4817], 80.00th=[ 5866], 90.00th=[ 8160], 95.00th=[10552], | 99.00th=[16450], 99.50th=[21627], 99.90th=[38011], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 40, max= 272, per=25.11%, avg=153.18, stdev=50.98, samples=78 iops : min= 10, max= 68, avg=38.19, stdev=12.73, samples=78 write: IOPS=39, BW=159KiB/s (163kB/s)(6260KiB/39315msec) clat (msec): min=9, max=183, avg=20.45, stdev= 8.20 lat (msec): min=9, max=183, avg=20.45, stdev= 8.20 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 19], 60.00th=[ 21], | 70.00th=[ 23], 80.00th=[ 26], 90.00th=[ 30], 95.00th=[ 35], | 99.00th=[ 43], 99.50th=[ 47], 99.90th=[ 58], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 87, max= 232, per=24.85%, avg=158.78, stdev=43.25, samples=78 iops : min= 21, max= 58, avg=39.59, stdev=10.80, samples=78 lat (msec) : 2=3.58%, 4=21.39%, 10=21.16%, 20=31.84%, 50=21.81% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.11%, sys=5.72%, ctx=4078, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5886: Tue Feb 25 09:08:27 2025 read: IOPS=38, BW=156KiB/s (160kB/s)(6052KiB/38798msec) clat (usec): min=1534, max=41633, avg=4641.28, stdev=3292.98 lat (usec): min=1534, max=41634, avg=4642.98, stdev=3292.76 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1909], 10.00th=[ 2180], 20.00th=[ 2802], | 30.00th=[ 3261], 40.00th=[ 3556], 50.00th=[ 3818], 60.00th=[ 4113], | 70.00th=[ 4555], 80.00th=[ 5342], 90.00th=[ 7832], 95.00th=[10159], | 99.00th=[17433], 99.50th=[23987], 99.90th=[38011], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 64, max= 312, per=25.44%, avg=155.18, stdev=58.44, samples=77 iops : min= 16, max= 78, avg=38.77, stdev=14.60, samples=77 write: IOPS=40, BW=161KiB/s (165kB/s)(6236KiB/38798msec) clat (msec): min=8, max=183, avg=20.33, stdev= 8.37 lat (msec): min=8, max=183, avg=20.33, stdev= 8.37 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 19], 60.00th=[ 21], | 70.00th=[ 23], 80.00th=[ 25], 90.00th=[ 30], 95.00th=[ 34], | 99.00th=[ 45], 99.50th=[ 51], 99.90th=[ 89], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 88, max= 216, per=25.08%, avg=160.27, stdev=41.64, samples=77 iops : min= 22, max= 54, avg=40.04, stdev=10.40, samples=77 lat (msec) : 2=3.52%, 4=24.25%, 10=18.85%, 20=31.80%, 50=21.32% lat (msec) : 100=0.23%, 250=0.03% cpu : usr=0.09%, sys=5.81%, ctx=4127, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=611KiB/s (625kB/s), 151KiB/s-156KiB/s (155kB/s-160kB/s), io=23.4MiB (24.6MB), run=38798-39325msec WRITE: bw=639KiB/s (655kB/s), 159KiB/s-161KiB/s (163kB/s-165kB/s), io=24.6MiB (25.7MB), run=38798-39325msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5937: Tue Feb 25 09:08:31 2025 read: IOPS=147, BW=2364KiB/s (2421kB/s)(5920KiB/2504msec) clat (usec): min=55, max=34250, avg=2880.33, stdev=3155.41 lat (usec): min=55, max=34250, avg=2881.11, stdev=3155.40 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 70], 10.00th=[ 72], 20.00th=[ 83], | 30.00th=[ 98], 40.00th=[ 2114], 50.00th=[ 2802], 60.00th=[ 3326], | 70.00th=[ 3982], 80.00th=[ 5276], 90.00th=[ 6390], 95.00th=[ 7242], | 99.00th=[10290], 99.50th=[18482], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 1472, max= 2816, per=24.74%, avg=2366.80, stdev=520.68, samples=5 iops : min= 92, max= 176, avg=147.80, stdev=32.47, samples=5 write: IOPS=158, BW=2543KiB/s (2604kB/s)(6368KiB/2504msec) clat (usec): min=641, max=39713, avg=3496.50, stdev=4190.89 lat (usec): min=641, max=39714, avg=3497.39, stdev=4191.04 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 734], 10.00th=[ 775], 20.00th=[ 832], | 30.00th=[ 906], 40.00th=[ 996], 50.00th=[ 1319], 60.00th=[ 2835], | 70.00th=[ 4146], 80.00th=[ 5669], 90.00th=[ 8848], 95.00th=[11207], | 99.00th=[18744], 99.50th=[23200], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 1760, max= 3200, per=25.24%, avg=2539.80, stdev=621.68, samples=5 iops : min= 110, max= 200, avg=158.60, stdev=38.98, samples=5 lat (usec) : 100=14.58%, 250=3.78%, 500=0.13%, 750=3.65%, 1000=17.71% lat (msec) : 2=8.07%, 4=21.48%, 10=26.17%, 20=3.91%, 50=0.52% cpu : usr=0.16%, sys=26.93%, ctx=1230, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5938: Tue Feb 25 09:08:31 2025 read: IOPS=143, BW=2295KiB/s (2350kB/s)(5744KiB/2503msec) clat (usec): min=57, max=16512, avg=2858.94, stdev=2616.23 lat (usec): min=58, max=16513, avg=2859.53, stdev=2616.18 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 71], 10.00th=[ 74], 20.00th=[ 86], | 30.00th=[ 113], 40.00th=[ 2442], 50.00th=[ 2802], 60.00th=[ 3359], | 70.00th=[ 3884], 80.00th=[ 4817], 90.00th=[ 6128], 95.00th=[ 7701], | 99.00th=[10421], 99.50th=[11994], 99.90th=[16450], 99.95th=[16450], | 99.99th=[16450] bw ( KiB/s): min= 1632, max= 2816, per=24.02%, avg=2297.60, stdev=489.19, samples=5 iops : min= 102, max= 176, avg=143.60, stdev=30.57, samples=5 write: IOPS=163, BW=2614KiB/s (2677kB/s)(6544KiB/2503msec) clat (usec): min=655, max=30467, avg=3485.96, stdev=3995.63 lat (usec): min=655, max=30468, avg=3486.79, stdev=3995.66 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 734], 10.00th=[ 775], 20.00th=[ 840], | 30.00th=[ 930], 40.00th=[ 1074], 50.00th=[ 1549], 60.00th=[ 3097], | 70.00th=[ 4178], 80.00th=[ 5800], 90.00th=[ 8291], 95.00th=[10290], | 99.00th=[16909], 99.50th=[25822], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 1824, max= 3264, per=25.95%, avg=2611.20, stdev=531.91, samples=5 iops : min= 114, max= 204, avg=163.20, stdev=33.24, samples=5 lat (usec) : 100=12.76%, 250=3.12%, 500=0.26%, 750=4.30%, 1000=14.84% lat (msec) : 2=9.64%, 4=25.52%, 10=25.91%, 20=3.12%, 50=0.52% cpu : usr=0.16%, sys=27.42%, ctx=1293, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5939: Tue Feb 25 09:08:31 2025 read: IOPS=162, BW=2601KiB/s (2664kB/s)(6256KiB/2405msec) clat (usec): min=54, max=23968, avg=2981.94, stdev=3067.85 lat (usec): min=55, max=23969, avg=2982.51, stdev=3067.86 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 68], 10.00th=[ 74], 20.00th=[ 84], | 30.00th=[ 97], 40.00th=[ 2376], 50.00th=[ 2868], 60.00th=[ 3261], | 70.00th=[ 4015], 80.00th=[ 5145], 90.00th=[ 6587], 95.00th=[ 8094], | 99.00th=[11994], 99.50th=[22676], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 1856, max= 2912, per=26.42%, avg=2526.75, stdev=492.51, samples=4 iops : min= 116, max= 182, avg=157.75, stdev=30.82, samples=4 write: IOPS=156, BW=2508KiB/s (2568kB/s)(6032KiB/2405msec) clat (usec): min=653, max=19701, avg=3252.71, stdev=3408.40 lat (usec): min=654, max=19703, avg=3253.55, stdev=3408.42 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 725], 10.00th=[ 775], 20.00th=[ 857], | 30.00th=[ 955], 40.00th=[ 1123], 50.00th=[ 1532], 60.00th=[ 2769], | 70.00th=[ 3818], 80.00th=[ 5735], 90.00th=[ 7832], 95.00th=[ 9634], | 99.00th=[17433], 99.50th=[19006], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 1948, max= 2976, per=25.11%, avg=2527.00, stdev=500.03, samples=4 iops : min= 121, max= 186, avg=157.75, stdev=31.54, samples=4 lat (usec) : 100=15.89%, 250=2.34%, 500=0.26%, 750=4.30%, 1000=12.76% lat (msec) : 2=10.29%, 4=24.48%, 10=26.56%, 20=2.86%, 50=0.26% cpu : usr=0.08%, sys=27.87%, ctx=1297, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5940: Tue Feb 25 09:08:31 2025 read: IOPS=156, BW=2502KiB/s (2562kB/s)(6032KiB/2411msec) clat (usec): min=58, max=22283, avg=2889.20, stdev=3062.82 lat (usec): min=58, max=22284, avg=2889.77, stdev=3062.82 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 70], 10.00th=[ 73], 20.00th=[ 80], | 30.00th=[ 95], 40.00th=[ 2114], 50.00th=[ 2671], 60.00th=[ 3032], | 70.00th=[ 3720], 80.00th=[ 5014], 90.00th=[ 6783], 95.00th=[ 8848], | 99.00th=[12649], 99.50th=[15270], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 1888, max= 2720, per=26.10%, avg=2496.00, stdev=406.45, samples=4 iops : min= 118, max= 170, avg=156.00, stdev=25.40, samples=4 write: IOPS=162, BW=2595KiB/s (2657kB/s)(6256KiB/2411msec) clat (usec): min=649, max=19287, avg=3238.34, stdev=3248.33 lat (usec): min=650, max=19288, avg=3239.17, stdev=3248.31 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 734], 10.00th=[ 791], 20.00th=[ 840], | 30.00th=[ 938], 40.00th=[ 1106], 50.00th=[ 1647], 60.00th=[ 2835], | 70.00th=[ 4047], 80.00th=[ 5538], 90.00th=[ 7242], 95.00th=[10028], | 99.00th=[15795], 99.50th=[16188], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 2144, max= 3264, per=26.31%, avg=2648.00, stdev=518.21, samples=4 iops : min= 134, max= 204, avg=165.50, stdev=32.39, samples=4 lat (usec) : 100=15.10%, 250=3.26%, 500=0.52%, 750=3.26%, 1000=14.71% lat (msec) : 2=9.11%, 4=25.13%, 10=25.26%, 20=3.52%, 50=0.13% cpu : usr=0.25%, sys=27.72%, ctx=1330, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9565KiB/s (9795kB/s), 2295KiB/s-2601KiB/s (2350kB/s-2664kB/s), io=23.4MiB (24.5MB), run=2405-2504msec WRITE: bw=9.83MiB/s (10.3MB/s), 2508KiB/s-2614KiB/s (2568kB/s-2677kB/s), io=24.6MiB (25.8MB), run=2405-2504msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5933: Tue Feb 25 09:08:37 2025 read: IOPS=42, BW=679KiB/s (695kB/s)(5920KiB/8721msec) clat (usec): min=1759, max=21294, avg=3623.32, stdev=2954.30 lat (usec): min=1759, max=21294, avg=3624.01, stdev=2954.29 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2008], 10.00th=[ 2089], 20.00th=[ 2180], | 30.00th=[ 2278], 40.00th=[ 2376], 50.00th=[ 2540], 60.00th=[ 2704], | 70.00th=[ 2966], 80.00th=[ 3687], 90.00th=[ 6652], 95.00th=[ 9372], | 99.00th=[18482], 99.50th=[19792], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 256, max= 1216, per=24.61%, avg=666.24, stdev=231.18, samples=17 iops : min= 16, max= 76, avg=41.59, stdev=14.49, samples=17 write: IOPS=45, BW=730KiB/s (748kB/s)(6368KiB/8721msec) clat (usec): min=10469, max=47005, avg=18422.26, stdev=6014.88 lat (usec): min=10471, max=47006, avg=18423.58, stdev=6015.00 clat percentiles (usec): | 1.00th=[11207], 5.00th=[11994], 10.00th=[12649], 20.00th=[13435], | 30.00th=[14353], 40.00th=[16188], 50.00th=[17171], 60.00th=[18482], | 70.00th=[19792], 80.00th=[21365], 90.00th=[26870], 95.00th=[31065], | 99.00th=[40109], 99.50th=[43779], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 416, max= 928, per=25.51%, avg=726.53, stdev=191.68, samples=17 iops : min= 26, max= 58, avg=45.35, stdev=12.07, samples=17 lat (msec) : 2=1.95%, 4=37.11%, 10=6.77%, 20=39.58%, 50=14.58% cpu : usr=0.13%, sys=5.81%, ctx=872, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5934: Tue Feb 25 09:08:37 2025 read: IOPS=40, BW=649KiB/s (665kB/s)(5744KiB/8847msec) clat (usec): min=1851, max=34976, avg=3540.30, stdev=3262.07 lat (usec): min=1852, max=34976, avg=3540.96, stdev=3262.07 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2024], 10.00th=[ 2114], 20.00th=[ 2212], | 30.00th=[ 2278], 40.00th=[ 2343], 50.00th=[ 2474], 60.00th=[ 2606], | 70.00th=[ 2966], 80.00th=[ 3458], 90.00th=[ 6325], 95.00th=[10028], | 99.00th=[17957], 99.50th=[24249], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 288, max= 1184, per=23.99%, avg=649.41, stdev=239.40, samples=17 iops : min= 18, max= 74, avg=40.59, stdev=14.96, samples=17 write: IOPS=46, BW=740KiB/s (757kB/s)(6544KiB/8847msec) clat (usec): min=10534, max=46233, avg=18416.85, stdev=5864.00 lat (usec): min=10534, max=46235, avg=18418.21, stdev=5863.99 clat percentiles (usec): | 1.00th=[11338], 5.00th=[11994], 10.00th=[12256], 20.00th=[13304], | 30.00th=[14615], 40.00th=[16319], 50.00th=[17433], 60.00th=[18744], | 70.00th=[19792], 80.00th=[21365], 90.00th=[26084], 95.00th=[30278], | 99.00th=[37487], 99.50th=[42206], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 448, max= 928, per=25.71%, avg=732.24, stdev=180.98, samples=17 iops : min= 28, max= 58, avg=45.76, stdev=11.31, samples=17 lat (msec) : 2=0.65%, 4=38.80%, 10=4.95%, 20=39.84%, 50=15.76% cpu : usr=0.03%, sys=6.03%, ctx=859, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5935: Tue Feb 25 09:08:37 2025 read: IOPS=46, BW=740KiB/s (757kB/s)(6256KiB/8459msec) clat (usec): min=1835, max=31301, avg=3531.96, stdev=3133.51 lat (usec): min=1836, max=31302, avg=3532.66, stdev=3133.51 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2024], 10.00th=[ 2073], 20.00th=[ 2180], | 30.00th=[ 2245], 40.00th=[ 2343], 50.00th=[ 2474], 60.00th=[ 2606], | 70.00th=[ 2802], 80.00th=[ 3621], 90.00th=[ 6063], 95.00th=[ 9765], | 99.00th=[15270], 99.50th=[31065], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 319, max= 1312, per=27.70%, avg=749.94, stdev=343.74, samples=16 iops : min= 19, max= 82, avg=46.81, stdev=21.56, samples=16 write: IOPS=44, BW=713KiB/s (730kB/s)(6032KiB/8459msec) clat (usec): min=10378, max=48745, avg=18738.28, stdev=6251.11 lat (usec): min=10379, max=48747, avg=18739.67, stdev=6251.18 clat percentiles (usec): | 1.00th=[11076], 5.00th=[11600], 10.00th=[12387], 20.00th=[13435], | 30.00th=[14746], 40.00th=[16188], 50.00th=[17695], 60.00th=[18744], | 70.00th=[20317], 80.00th=[22152], 90.00th=[27395], 95.00th=[32375], | 99.00th=[38536], 99.50th=[41681], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 384, max= 896, per=24.65%, avg=701.94, stdev=165.94, samples=16 iops : min= 24, max= 56, avg=43.81, stdev=10.46, samples=16 lat (msec) : 2=1.17%, 4=40.10%, 10=7.29%, 20=35.68%, 50=15.76% cpu : usr=0.17%, sys=6.05%, ctx=875, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5936: Tue Feb 25 09:08:37 2025 read: IOPS=43, BW=693KiB/s (709kB/s)(6032KiB/8707msec) clat (usec): min=1873, max=31492, avg=3651.68, stdev=3388.55 lat (usec): min=1874, max=31493, avg=3652.34, stdev=3388.56 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2024], 10.00th=[ 2073], 20.00th=[ 2180], | 30.00th=[ 2278], 40.00th=[ 2376], 50.00th=[ 2507], 60.00th=[ 2671], | 70.00th=[ 2835], 80.00th=[ 3326], 90.00th=[ 6783], 95.00th=[10814], | 99.00th=[17957], 99.50th=[26084], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 320, max= 1024, per=25.10%, avg=679.53, stdev=210.00, samples=17 iops : min= 20, max= 64, avg=42.47, stdev=13.12, samples=17 write: IOPS=44, BW=719KiB/s (736kB/s)(6256KiB/8707msec) clat (usec): min=10521, max=47497, avg=18649.86, stdev=5816.94 lat (usec): min=10522, max=47513, avg=18651.21, stdev=5817.16 clat percentiles (usec): | 1.00th=[10945], 5.00th=[11731], 10.00th=[12518], 20.00th=[13566], | 30.00th=[15008], 40.00th=[17171], 50.00th=[17957], 60.00th=[19268], | 70.00th=[20055], 80.00th=[21365], 90.00th=[26084], 95.00th=[29230], | 99.00th=[40633], 99.50th=[45876], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 384, max= 896, per=25.12%, avg=715.29, stdev=168.17, samples=17 iops : min= 24, max= 56, avg=44.71, stdev=10.51, samples=17 lat (msec) : 2=1.56%, 4=39.45%, 10=5.08%, 20=37.89%, 50=16.02% cpu : usr=0.16%, sys=5.88%, ctx=886, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2707KiB/s (2772kB/s), 649KiB/s-740KiB/s (665kB/s-757kB/s), io=23.4MiB (24.5MB), run=8459-8847msec WRITE: bw=2848KiB/s (2917kB/s), 713KiB/s-740KiB/s (730kB/s-757kB/s), io=24.6MiB (25.8MB), run=8459-8847msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5954: Tue Feb 25 09:08:40 2025 read: IOPS=3, BW=4059KiB/s (4156kB/s)(7168KiB/1766msec) clat (usec): min=605, max=631610, avg=111903.69, stdev=229482.26 lat (usec): min=644, max=631611, avg=111910.11, stdev=229479.31 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 603], 10.00th=[ 603], 20.00th=[ 23987], | 30.00th=[ 24773], 40.00th=[ 24773], 50.00th=[ 31851], 60.00th=[ 33424], | 70.00th=[ 33424], 80.00th=[ 36963], 90.00th=[633340], 95.00th=[633340], | 99.00th=[633340], 99.50th=[633340], 99.90th=[633340], 99.95th=[633340], | 99.99th=[633340] bw ( KiB/s): min= 2048, max= 8192, per=47.00%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=2, BW=2899KiB/s (2969kB/s)(5120KiB/1766msec) clat (msec): min=27, max=699, avg=191.74, stdev=286.56 lat (msec): min=27, max=699, avg=191.77, stdev=286.57 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 28], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 75], 60.00th=[ 75], | 70.00th=[ 124], 80.00th=[ 124], 90.00th=[ 701], 95.00th=[ 701], | 99.00th=[ 701], 99.50th=[ 701], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 4096, max= 4096, per=26.86%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=58.33%, 100=8.33%, 250=8.33%, 750=16.67% cpu : usr=0.06%, sys=17.22%, ctx=863, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5957: Tue Feb 25 09:08:40 2025 read: IOPS=1, BW=1089KiB/s (1116kB/s)(2048KiB/1880msec) clat (usec): min=29167, max=30587, avg=29877.35, stdev=1004.07 lat (usec): min=29168, max=30589, avg=29878.69, stdev=1004.80 clat percentiles (usec): | 1.00th=[29230], 5.00th=[29230], 10.00th=[29230], 20.00th=[29230], | 30.00th=[29230], 40.00th=[29230], 50.00th=[29230], 60.00th=[30540], | 70.00th=[30540], 80.00th=[30540], 90.00th=[30540], 95.00th=[30540], | 99.00th=[30540], 99.50th=[30540], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 2043, max= 2043, per=18.76%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5447KiB/s (5578kB/s)(10.0MiB/1880msec) clat (msec): min=15, max=741, avg=181.31, stdev=293.33 lat (msec): min=15, max=741, avg=181.37, stdev=293.33 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 22], | 30.00th=[ 26], 40.00th=[ 39], 50.00th=[ 42], 60.00th=[ 44], | 70.00th=[ 85], 80.00th=[ 85], 90.00th=[ 735], 95.00th=[ 743], | 99.00th=[ 743], 99.50th=[ 743], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 6144, max= 6144, per=40.29%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=58.33%, 100=16.67%, 750=16.67% cpu : usr=0.00%, sys=12.24%, ctx=555, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5958: Tue Feb 25 09:08:40 2025 read: IOPS=3, BW=3623KiB/s (3710kB/s)(6144KiB/1696msec) clat (usec): min=497, max=649049, avg=127423.19, stdev=257258.14 lat (usec): min=497, max=649050, avg=127424.15, stdev=257258.08 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 498], 10.00th=[ 498], 20.00th=[ 529], | 30.00th=[ 529], 40.00th=[ 7832], 50.00th=[ 7832], 60.00th=[ 27657], | 70.00th=[ 79168], 80.00th=[ 79168], 90.00th=[650118], 95.00th=[650118], | 99.00th=[650118], 99.50th=[650118], 99.90th=[650118], 99.95th=[650118], | 99.99th=[650118] bw ( KiB/s): min= 1753, max= 4096, per=26.85%, avg=2924.50, stdev=1656.75, samples=2 iops : min= 1, max= 4, avg= 2.50, stdev= 2.12, samples=2 write: IOPS=3, BW=3623KiB/s (3710kB/s)(6144KiB/1696msec) clat (msec): min=20, max=678, avg=145.32, stdev=261.65 lat (msec): min=20, max=678, avg=145.35, stdev=261.64 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 40], | 70.00th=[ 66], 80.00th=[ 66], 90.00th=[ 676], 95.00th=[ 676], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 4096, max= 4096, per=26.86%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 10=8.33%, 50=41.67%, 100=16.67%, 750=16.67% cpu : usr=0.00%, sys=14.34%, ctx=379, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5960: Tue Feb 25 09:08:40 2025 read: IOPS=5, BW=5156KiB/s (5280kB/s)(5120KiB/993msec) clat (usec): min=387, max=24388, avg=14658.88, stdev=13000.68 lat (usec): min=387, max=24389, avg=14659.61, stdev=13000.74 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 388], 10.00th=[ 388], 20.00th=[ 388], | 30.00th=[ 453], 40.00th=[ 453], 50.00th=[23725], 60.00th=[23725], | 70.00th=[24249], 80.00th=[24249], 90.00th=[24511], 95.00th=[24511], | 99.00th=[24511], 99.50th=[24511], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] write: IOPS=7, BW=7219KiB/s (7392kB/s)(7168KiB/993msec) clat (msec): min=15, max=688, avg=127.33, stdev=248.35 lat (msec): min=15, max=688, avg=127.39, stdev=248.34 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 21], | 30.00th=[ 23], 40.00th=[ 23], 50.00th=[ 32], 60.00th=[ 41], | 70.00th=[ 41], 80.00th=[ 72], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 2048, max= 2048, per=13.43%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 20=8.33%, 50=58.33%, 100=8.33%, 750=8.33% cpu : usr=0.10%, sys=19.86%, ctx=736, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.6MiB/s (11.2MB/s), 1089KiB/s-5156KiB/s (1116kB/s-5280kB/s), io=20.0MiB (20.0MB), run=993-1880msec WRITE: bw=14.9MiB/s (15.6MB/s), 2899KiB/s-7219KiB/s (2969kB/s-7392kB/s), io=28.0MiB (29.4MB), run=993-1880msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5955: Tue Feb 25 09:08:40 2025 read: IOPS=3, BW=3591KiB/s (3677kB/s)(7168KiB/1996msec) clat (msec): min=14, max=633, avg=142.51, stdev=229.86 lat (msec): min=14, max=633, avg=142.51, stdev=229.86 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16], | 30.00th=[ 26], 40.00th=[ 26], 50.00th=[ 32], 60.00th=[ 45], | 70.00th=[ 45], 80.00th=[ 232], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 6144, max= 6144, per=61.14%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2565KiB/s (2627kB/s)(5120KiB/1996msec) clat (msec): min=25, max=783, avg=186.24, stdev=334.11 lat (msec): min=25, max=783, avg=186.30, stdev=334.11 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 26], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 40], 60.00th=[ 40], | 70.00th=[ 51], 80.00th=[ 51], 90.00th=[ 785], 95.00th=[ 785], | 99.00th=[ 785], 99.50th=[ 785], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] lat (msec) : 20=16.67%, 50=50.00%, 100=8.33%, 250=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.05%, sys=4.06%, ctx=45, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5956: Tue Feb 25 09:08:40 2025 read: IOPS=0, BW=1005KiB/s (1029kB/s)(2048KiB/2038msec) clat (usec): min=19480, max=41335, avg=30408.17, stdev=15453.79 lat (usec): min=19482, max=41336, avg=30409.40, stdev=15453.31 clat percentiles (usec): | 1.00th=[19530], 5.00th=[19530], 10.00th=[19530], 20.00th=[19530], | 30.00th=[19530], 40.00th=[19530], 50.00th=[19530], 60.00th=[41157], | 70.00th=[41157], 80.00th=[41157], 90.00th=[41157], 95.00th=[41157], | 99.00th=[41157], 99.50th=[41157], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 2043, max= 2048, per=20.36%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=5025KiB/s (5145kB/s)(10.0MiB/2038msec) clat (msec): min=22, max=838, avg=194.52, stdev=311.05 lat (msec): min=22, max=838, avg=194.59, stdev=311.03 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 23], | 30.00th=[ 28], 40.00th=[ 37], 50.00th=[ 50], 60.00th=[ 57], | 70.00th=[ 103], 80.00th=[ 103], 90.00th=[ 726], 95.00th=[ 835], | 99.00th=[ 835], 99.50th=[ 835], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 2048, max=14336, per=58.23%, avg=8192.00, stdev=8688.93, samples=2 iops : min= 2, max= 14, avg= 8.00, stdev= 8.49, samples=2 lat (msec) : 20=8.33%, 50=50.00%, 100=16.67%, 250=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=4.71%, ctx=36, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5959: Tue Feb 25 09:08:40 2025 read: IOPS=3, BW=3195KiB/s (3272kB/s)(6144KiB/1923msec) clat (msec): min=13, max=728, avg=146.93, stdev=285.01 lat (msec): min=13, max=728, avg=146.93, stdev=285.01 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 28], 60.00th=[ 35], | 70.00th=[ 50], 80.00th=[ 50], 90.00th=[ 726], 95.00th=[ 726], | 99.00th=[ 726], 99.50th=[ 726], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 4096, max= 4096, per=40.76%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3195KiB/s (3272kB/s)(6144KiB/1923msec) clat (msec): min=25, max=729, avg=165.82, stdev=277.57 lat (msec): min=26, max=729, avg=165.91, stdev=277.58 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 66], | 70.00th=[ 100], 80.00th=[ 100], 90.00th=[ 726], 95.00th=[ 726], | 99.00th=[ 726], 99.50th=[ 726], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 2043, max= 4096, per=19.40%, avg=2729.00, stdev=1183.86, samples=3 iops : min= 1, max= 4, avg= 2.33, stdev= 1.53, samples=3 lat (msec) : 20=8.33%, 50=58.33%, 100=16.67%, 750=16.67% cpu : usr=0.00%, sys=4.42%, ctx=42, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5961: Tue Feb 25 09:08:40 2025 read: IOPS=2, BW=2604KiB/s (2667kB/s)(5120KiB/1966msec) clat (usec): min=12771, max=92174, avg=38009.75, stdev=32201.36 lat (usec): min=12772, max=92175, avg=38010.58, stdev=32201.46 clat percentiles (usec): | 1.00th=[12780], 5.00th=[12780], 10.00th=[12780], 20.00th=[12780], | 30.00th=[16057], 40.00th=[16057], 50.00th=[28443], 60.00th=[28443], | 70.00th=[40633], 80.00th=[40633], 90.00th=[91751], 95.00th=[91751], | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] write: IOPS=3, BW=3646KiB/s (3733kB/s)(7168KiB/1966msec) clat (msec): min=28, max=718, avg=243.79, stdev=323.95 lat (msec): min=28, max=718, avg=243.85, stdev=323.95 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 35], | 30.00th=[ 48], 40.00th=[ 48], 50.00th=[ 81], 60.00th=[ 82], | 70.00th=[ 82], 80.00th=[ 718], 90.00th=[ 718], 95.00th=[ 718], | 99.00th=[ 718], 99.50th=[ 718], 99.90th=[ 718], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 2048, max= 2048, per=14.56%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 20=16.67%, 50=41.67%, 100=25.00%, 750=16.67% cpu : usr=0.05%, sys=4.78%, ctx=53, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9.81MiB/s (10.3MB/s), 1005KiB/s-3591KiB/s (1029kB/s-3677kB/s), io=20.0MiB (20.0MB), run=1923-2038msec WRITE: bw=13.7MiB/s (14.4MB/s), 2565KiB/s-5025KiB/s (2627kB/s-5145kB/s), io=28.0MiB (29.4MB), run=1923-2038msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5969: Tue Feb 25 09:08:42 2025 read: IOPS=7, BW=31.4MiB/s (32.9MB/s)(12.0MiB/382msec) clat (msec): min=90, max=139, avg=122.53, stdev=28.14 lat (msec): min=90, max=139, avg=122.53, stdev=28.14 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 90], 10.00th=[ 90], 20.00th=[ 90], | 30.00th=[ 90], 40.00th=[ 138], 50.00th=[ 138], 60.00th=[ 138], | 70.00th=[ 140], 80.00th=[ 140], 90.00th=[ 140], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=38.58%, ctx=1032, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5971: Tue Feb 25 09:08:42 2025 read: IOPS=1, BW=6617KiB/s (6776kB/s)(4096KiB/619msec) clat (nsec): min=67366k, max=67366k, avg=67366273.00, stdev= 0.00 lat (nsec): min=67367k, max=67367k, avg=67367302.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[67634], 5.00th=[67634], 10.00th=[67634], 20.00th=[67634], | 30.00th=[67634], 40.00th=[67634], 50.00th=[67634], 60.00th=[67634], | 70.00th=[67634], 80.00th=[67634], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 8192, max= 8192, per=30.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=12.9MiB/s (13.6MB/s)(8192KiB/619msec) clat (msec): min=210, max=313, avg=262.00, stdev=72.71 lat (msec): min=210, max=313, avg=262.25, stdev=72.66 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 211], 10.00th=[ 211], 20.00th=[ 211], | 30.00th=[ 211], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 313], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 313], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 8192, max= 8192, per=16.33%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=27.67%, ctx=192, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5973: Tue Feb 25 09:08:42 2025 write: IOPS=4, BW=18.4MiB/s (19.3MB/s)(12.0MiB/653msec) clat (msec): min=138, max=344, avg=215.36, stdev=112.63 lat (msec): min=138, max=345, avg=215.71, stdev=112.64 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 140], 10.00th=[ 140], 20.00th=[ 140], | 30.00th=[ 140], 40.00th=[ 163], 50.00th=[ 163], 60.00th=[ 163], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min=16384, max=16384, per=32.65%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=35.74%, ctx=72, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5975: Tue Feb 25 09:08:42 2025 write: IOPS=5, BW=23.3MiB/s (24.5MB/s)(12.0MiB/514msec) clat (msec): min=104, max=279, avg=164.12, stdev=100.27 lat (msec): min=105, max=280, avg=164.41, stdev=100.25 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 105], 10.00th=[ 105], 20.00th=[ 105], | 30.00th=[ 105], 40.00th=[ 108], 50.00th=[ 108], 60.00th=[ 108], | 70.00th=[ 279], 80.00th=[ 279], 90.00th=[ 279], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min=16384, max=16384, per=32.65%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.19%, sys=48.34%, ctx=99, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.8MiB/s (27.1MB/s), 6617KiB/s-31.4MiB/s (6776kB/s-32.9MB/s), io=16.0MiB (16.8MB), run=382-619msec WRITE: bw=49.0MiB/s (51.4MB/s), 12.9MiB/s-23.3MiB/s (13.6MB/s-24.5MB/s), io=32.0MiB (33.6MB), run=514-653msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5968: Tue Feb 25 09:08:42 2025 read: IOPS=4, BW=17.7MiB/s (18.6MB/s)(12.0MiB/678msec) clat (msec): min=171, max=268, avg=219.81, stdev=48.46 lat (msec): min=171, max=268, avg=219.82, stdev=48.46 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 220], 50.00th=[ 220], 60.00th=[ 220], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min=16384, max=16384, per=74.20%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=7.68%, ctx=38, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5970: Tue Feb 25 09:08:42 2025 read: IOPS=1, BW=5520KiB/s (5653kB/s)(4096KiB/742msec) clat (nsec): min=200762k, max=200762k, avg=200762129.00, stdev= 0.00 lat (nsec): min=200764k, max=200764k, avg=200764274.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 201], 10.00th=[ 201], 20.00th=[ 201], | 30.00th=[ 201], 40.00th=[ 201], 50.00th=[ 201], 60.00th=[ 201], | 70.00th=[ 201], 80.00th=[ 201], 90.00th=[ 201], 95.00th=[ 201], | 99.00th=[ 201], 99.50th=[ 201], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 8175, max= 8175, per=37.02%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.8MiB/s (11.3MB/s)(8192KiB/742msec) clat (msec): min=252, max=264, avg=258.52, stdev= 8.91 lat (msec): min=252, max=265, avg=258.79, stdev= 8.92 clat percentiles (msec): | 1.00th=[ 253], 5.00th=[ 253], 10.00th=[ 253], 20.00th=[ 253], | 30.00th=[ 253], 40.00th=[ 253], 50.00th=[ 253], 60.00th=[ 266], | 70.00th=[ 266], 80.00th=[ 266], 90.00th=[ 266], 95.00th=[ 266], | 99.00th=[ 266], 99.50th=[ 266], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 8175, max= 8175, per=23.73%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=8.77%, ctx=61, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5972: Tue Feb 25 09:08:42 2025 write: IOPS=4, BW=16.6MiB/s (17.4MB/s)(12.0MiB/722msec) clat (msec): min=155, max=285, avg=220.14, stdev=65.19 lat (msec): min=155, max=285, avg=220.36, stdev=65.26 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 220], 50.00th=[ 220], 60.00th=[ 220], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 8192, max= 8192, per=23.78%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.14%, sys=10.12%, ctx=34, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5974: Tue Feb 25 09:08:42 2025 write: IOPS=3, BW=12.6MiB/s (13.2MB/s)(12.0MiB/951msec) clat (msec): min=125, max=428, avg=309.52, stdev=161.54 lat (msec): min=125, max=428, avg=309.84, stdev=161.41 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 126], 10.00th=[ 126], 20.00th=[ 126], | 30.00th=[ 126], 40.00th=[ 376], 50.00th=[ 376], 60.00th=[ 376], | 70.00th=[ 430], 80.00th=[ 430], 90.00th=[ 430], 95.00th=[ 430], | 99.00th=[ 430], 99.50th=[ 430], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 8175, max= 8175, per=23.73%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=14.74%, ctx=162, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=21.6MiB/s (22.6MB/s), 5520KiB/s-17.7MiB/s (5653kB/s-18.6MB/s), io=16.0MiB (16.8MB), run=678-742msec WRITE: bw=33.6MiB/s (35.3MB/s), 10.8MiB/s-16.6MiB/s (11.3MB/s-17.4MB/s), io=32.0MiB (33.6MB), run=722-951msec