== sanity test 398b: DIO and buffer IO race ============== 01:48:53 (1744177733) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.10023 s, 45.7 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10444: Wed Apr 9 01:49:09 2025 read: IOPS=116, BW=468KiB/s (479kB/s)(5992KiB/12805msec) clat (usec): min=52, max=158742, avg=3775.13, stdev=5127.16 lat (usec): min=52, max=158742, avg=3775.92, stdev=5127.14 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 70], 10.00th=[ 79], 20.00th=[ 217], | 30.00th=[ 2212], 40.00th=[ 2573], 50.00th=[ 2933], 60.00th=[ 3458], | 70.00th=[ 4424], 80.00th=[ 5932], 90.00th=[ 7832], 95.00th=[ 9372], | 99.00th=[ 13960], 99.50th=[ 16057], 99.90th=[ 40633], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 304, max= 704, per=24.89%, avg=466.16, stdev=89.54, samples=25 iops : min= 76, max= 176, avg=116.48, stdev=22.34, samples=25 write: IOPS=122, BW=492KiB/s (503kB/s)(6296KiB/12805msec) clat (usec): min=432, max=36282, avg=4465.29, stdev=4341.57 lat (usec): min=432, max=36282, avg=4466.56, stdev=4341.99 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 529], 10.00th=[ 562], 20.00th=[ 652], | 30.00th=[ 930], 40.00th=[ 2212], 50.00th=[ 3490], 60.00th=[ 4424], | 70.00th=[ 5735], 80.00th=[ 7373], 90.00th=[10421], 95.00th=[13304], | 99.00th=[19268], 99.50th=[21365], 99.90th=[30278], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 352, max= 926, per=24.89%, avg=488.00, stdev=111.14, samples=25 iops : min= 88, max= 231, avg=121.92, stdev=27.71, samples=25 lat (usec) : 100=7.78%, 250=2.08%, 500=0.91%, 750=11.95%, 1000=3.32% lat (msec) : 2=6.09%, 4=28.42%, 10=31.64%, 20=7.39%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=22.91%, ctx=5592, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10445: Wed Apr 9 01:49:09 2025 read: IOPS=116, BW=465KiB/s (476kB/s)(5940KiB/12787msec) clat (usec): min=52, max=26951, avg=3531.54, stdev=3175.79 lat (usec): min=52, max=26952, avg=3532.21, stdev=3175.78 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 69], 10.00th=[ 79], 20.00th=[ 113], | 30.00th=[ 2073], 40.00th=[ 2474], 50.00th=[ 2802], 60.00th=[ 3326], | 70.00th=[ 4113], 80.00th=[ 5800], 90.00th=[ 8094], 95.00th=[ 9503], | 99.00th=[14222], 99.50th=[15401], 99.90th=[24511], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 248, max= 712, per=24.75%, avg=463.64, stdev=100.53, samples=25 iops : min= 62, max= 178, avg=115.88, stdev=25.16, samples=25 write: IOPS=124, BW=496KiB/s (508kB/s)(6348KiB/12787msec) clat (usec): min=466, max=157593, avg=4676.88, stdev=5829.59 lat (usec): min=467, max=157593, avg=4677.80, stdev=5829.56 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 529], 10.00th=[ 570], 20.00th=[ 701], | 30.00th=[ 1074], 40.00th=[ 2409], 50.00th=[ 3687], 60.00th=[ 4686], | 70.00th=[ 5800], 80.00th=[ 7635], 90.00th=[ 10421], 95.00th=[ 12649], | 99.00th=[ 19268], 99.50th=[ 22152], 99.90th=[ 40109], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 288, max= 800, per=24.93%, avg=488.92, stdev=110.06, samples=25 iops : min= 72, max= 200, avg=122.20, stdev=27.51, samples=25 lat (usec) : 100=8.37%, 250=2.83%, 500=0.98%, 750=10.51%, 1000=3.58% lat (msec) : 2=6.64%, 4=27.64%, 10=31.71%, 20=7.19%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=22.98%, ctx=5637, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10447: Wed Apr 9 01:49:09 2025 read: IOPS=117, BW=470KiB/s (482kB/s)(6028KiB/12816msec) clat (usec): min=51, max=156759, avg=3748.47, stdev=5178.96 lat (usec): min=51, max=156760, avg=3749.24, stdev=5179.00 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 70], 10.00th=[ 78], 20.00th=[ 116], | 30.00th=[ 2089], 40.00th=[ 2474], 50.00th=[ 2868], 60.00th=[ 3523], | 70.00th=[ 4490], 80.00th=[ 6194], 90.00th=[ 7963], 95.00th=[ 9503], | 99.00th=[ 13042], 99.50th=[ 16581], 99.90th=[ 43254], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 335, max= 712, per=24.82%, avg=464.88, stdev=104.49, samples=25 iops : min= 83, max= 178, avg=116.16, stdev=26.15, samples=25 write: IOPS=122, BW=488KiB/s (500kB/s)(6260KiB/12816msec) clat (usec): min=468, max=42079, avg=4499.79, stdev=4401.29 lat (usec): min=469, max=42080, avg=4500.66, stdev=4401.31 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 562], 20.00th=[ 668], | 30.00th=[ 988], 40.00th=[ 2089], 50.00th=[ 3654], 60.00th=[ 4490], | 70.00th=[ 5735], 80.00th=[ 7439], 90.00th=[10290], 95.00th=[12911], | 99.00th=[18482], 99.50th=[22676], 99.90th=[35390], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 360, max= 720, per=24.47%, avg=479.92, stdev=88.82, samples=25 iops : min= 90, max= 180, avg=119.92, stdev=22.25, samples=25 lat (usec) : 100=8.85%, 250=2.21%, 500=0.85%, 750=11.56%, 1000=3.32% lat (msec) : 2=6.51%, 4=25.94%, 10=33.14%, 20=7.13%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=22.61%, ctx=5583, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10448: Wed Apr 9 01:49:09 2025 read: IOPS=122, BW=489KiB/s (501kB/s)(6052KiB/12380msec) clat (usec): min=53, max=26527, avg=3451.58, stdev=3019.52 lat (usec): min=53, max=26527, avg=3452.23, stdev=3019.53 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 71], 10.00th=[ 80], 20.00th=[ 121], | 30.00th=[ 2073], 40.00th=[ 2474], 50.00th=[ 2868], 60.00th=[ 3326], | 70.00th=[ 4178], 80.00th=[ 5604], 90.00th=[ 7635], 95.00th=[ 9110], | 99.00th=[12387], 99.50th=[14353], 99.90th=[25297], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 288, max= 768, per=25.93%, avg=485.58, stdev=106.46, samples=24 iops : min= 72, max= 192, avg=121.33, stdev=26.64, samples=24 write: IOPS=125, BW=504KiB/s (516kB/s)(6236KiB/12380msec) clat (usec): min=463, max=157863, avg=4510.64, stdev=5808.72 lat (usec): min=463, max=157864, avg=4512.65, stdev=5808.64 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 578], 20.00th=[ 660], | 30.00th=[ 865], 40.00th=[ 1696], 50.00th=[ 3523], 60.00th=[ 4490], | 70.00th=[ 5669], 80.00th=[ 7570], 90.00th=[ 10552], 95.00th=[ 12911], | 99.00th=[ 17171], 99.50th=[ 19006], 99.90th=[ 46400], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 368, max= 776, per=25.53%, avg=500.58, stdev=97.55, samples=24 iops : min= 92, max= 194, avg=125.08, stdev=24.41, samples=24 lat (usec) : 100=8.63%, 250=2.64%, 500=0.85%, 750=11.91%, 1000=4.23% lat (msec) : 2=6.67%, 4=26.46%, 10=31.32%, 20=7.06%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=23.24%, ctx=5313, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1874KiB/s (1919kB/s), 465KiB/s-489KiB/s (476kB/s-501kB/s), io=23.4MiB (24.6MB), run=12380-12816msec WRITE: bw=1962KiB/s (2009kB/s), 488KiB/s-504KiB/s (500kB/s-516kB/s), io=24.6MiB (25.7MB), run=12380-12816msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10449: Wed Apr 9 01:49:30 2025 read: IOPS=43, BW=176KiB/s (180kB/s)(5992KiB/34087msec) clat (usec): min=1658, max=160893, avg=3529.52, stdev=5235.91 lat (usec): min=1659, max=160896, avg=3530.25, stdev=5235.96 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1795], 10.00th=[ 1860], 20.00th=[ 1926], | 30.00th=[ 1991], 40.00th=[ 2073], 50.00th=[ 2180], 60.00th=[ 2311], | 70.00th=[ 2671], 80.00th=[ 3621], 90.00th=[ 7111], 95.00th=[ 10159], | 99.00th=[ 17957], 99.50th=[ 20579], 99.90th=[ 39060], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 48, max= 320, per=24.81%, avg=174.66, stdev=74.62, samples=67 iops : min= 12, max= 80, avg=43.55, stdev=18.72, samples=67 write: IOPS=46, BW=185KiB/s (189kB/s)(6296KiB/34087msec) clat (usec): min=9599, max=62100, avg=18234.88, stdev=7667.63 lat (usec): min=9600, max=62101, avg=18235.73, stdev=7667.69 clat percentiles (usec): | 1.00th=[10159], 5.00th=[10814], 10.00th=[11207], 20.00th=[12125], | 30.00th=[13435], 40.00th=[15139], 50.00th=[16581], 60.00th=[17695], | 70.00th=[18744], 80.00th=[21627], 90.00th=[29754], 95.00th=[34866], | 99.00th=[44827], 99.50th=[49546], 99.90th=[56886], 99.95th=[62129], | 99.99th=[62129] bw ( KiB/s): min= 64, max= 256, per=24.90%, avg=183.49, stdev=62.29, samples=67 iops : min= 16, max= 64, avg=45.76, stdev=15.61, samples=67 lat (msec) : 2=15.14%, 4=24.48%, 10=6.80%, 20=41.41%, 50=11.98% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.10%, sys=5.46%, ctx=3422, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10450: Wed Apr 9 01:49:30 2025 read: IOPS=43, BW=174KiB/s (178kB/s)(5940KiB/34081msec) clat (usec): min=1671, max=166553, avg=3478.58, stdev=5469.54 lat (usec): min=1671, max=166554, avg=3479.26, stdev=5469.56 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1811], 10.00th=[ 1844], 20.00th=[ 1926], | 30.00th=[ 1975], 40.00th=[ 2057], 50.00th=[ 2147], 60.00th=[ 2311], | 70.00th=[ 2573], 80.00th=[ 3556], 90.00th=[ 6849], 95.00th=[ 9765], | 99.00th=[ 16909], 99.50th=[ 22414], 99.90th=[ 52691], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 16, max= 344, per=24.80%, avg=174.58, stdev=75.62, samples=67 iops : min= 4, max= 86, avg=43.57, stdev=18.91, samples=67 write: IOPS=46, BW=186KiB/s (191kB/s)(6348KiB/34081msec) clat (usec): min=9495, max=65216, avg=18157.73, stdev=7325.55 lat (usec): min=9496, max=65217, avg=18158.53, stdev=7325.57 clat percentiles (usec): | 1.00th=[10159], 5.00th=[10683], 10.00th=[11207], 20.00th=[12125], | 30.00th=[13829], 40.00th=[15270], 50.00th=[16581], 60.00th=[17695], | 70.00th=[19006], 80.00th=[21627], 90.00th=[28967], 95.00th=[32900], | 99.00th=[43779], 99.50th=[47973], 99.90th=[53216], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 88, max= 256, per=25.08%, avg=184.85, stdev=60.06, samples=67 iops : min= 22, max= 64, avg=46.13, stdev=15.03, samples=67 lat (msec) : 2=15.82%, 4=23.83%, 10=6.71%, 20=40.85%, 50=12.60% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.09%, sys=5.51%, ctx=3427, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10451: Wed Apr 9 01:49:30 2025 read: IOPS=44, BW=178KiB/s (182kB/s)(6028KiB/33859msec) clat (usec): min=1589, max=55738, avg=3553.26, stdev=3697.41 lat (usec): min=1589, max=55739, avg=3553.99, stdev=3697.57 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1795], 10.00th=[ 1844], 20.00th=[ 1926], | 30.00th=[ 1991], 40.00th=[ 2089], 50.00th=[ 2180], 60.00th=[ 2343], | 70.00th=[ 2704], 80.00th=[ 4113], 90.00th=[ 7635], 95.00th=[10683], | 99.00th=[16319], 99.50th=[25035], 99.90th=[41681], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 56, max= 360, per=25.25%, avg=177.73, stdev=71.05, samples=67 iops : min= 14, max= 90, avg=44.39, stdev=17.79, samples=67 write: IOPS=46, BW=185KiB/s (189kB/s)(6260KiB/33859msec) clat (msec): min=9, max=169, avg=18.14, stdev= 8.28 lat (msec): min=9, max=169, avg=18.14, stdev= 8.28 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 20], 80.00th=[ 22], 90.00th=[ 28], 95.00th=[ 34], | 99.00th=[ 45], 99.50th=[ 50], 99.90th=[ 71], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 72, max= 256, per=24.97%, avg=184.06, stdev=62.18, samples=67 iops : min= 18, max= 64, avg=45.97, stdev=15.60, samples=67 lat (msec) : 2=15.14%, 4=23.86%, 10=7.94%, 20=40.46%, 50=12.43% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.08%, sys=5.42%, ctx=3438, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10452: Wed Apr 9 01:49:30 2025 read: IOPS=44, BW=179KiB/s (183kB/s)(6052KiB/33868msec) clat (usec): min=1585, max=41621, avg=3472.35, stdev=3318.00 lat (usec): min=1586, max=41622, avg=3473.02, stdev=3318.03 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1795], 10.00th=[ 1860], 20.00th=[ 1926], | 30.00th=[ 1991], 40.00th=[ 2089], 50.00th=[ 2180], 60.00th=[ 2311], | 70.00th=[ 2638], 80.00th=[ 3752], 90.00th=[ 7439], 95.00th=[10552], | 99.00th=[16909], 99.50th=[20317], 99.90th=[38011], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 40, max= 344, per=25.23%, avg=177.60, stdev=72.99, samples=67 iops : min= 10, max= 86, avg=44.34, stdev=18.24, samples=67 write: IOPS=46, BW=184KiB/s (189kB/s)(6236KiB/33868msec) clat (msec): min=9, max=192, avg=18.29, stdev= 8.60 lat (msec): min=9, max=192, avg=18.29, stdev= 8.60 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 20], 80.00th=[ 22], 90.00th=[ 29], 95.00th=[ 35], | 99.00th=[ 43], 99.50th=[ 46], 99.90th=[ 71], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 64, max= 256, per=24.87%, avg=183.33, stdev=61.26, samples=67 iops : min= 16, max= 64, avg=45.78, stdev=15.32, samples=67 lat (msec) : 2=14.97%, 4=25.00%, 10=7.03%, 20=39.81%, 50=13.05% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.12%, sys=5.44%, ctx=3439, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=704KiB/s (721kB/s), 174KiB/s-179KiB/s (178kB/s-183kB/s), io=23.4MiB (24.6MB), run=33859-34087msec WRITE: bw=738KiB/s (755kB/s), 184KiB/s-186KiB/s (189kB/s-191kB/s), io=24.6MiB (25.7MB), run=33859-34087msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10495: Wed Apr 9 01:49:34 2025 read: IOPS=148, BW=2371KiB/s (2428kB/s)(5920KiB/2497msec) clat (usec): min=59, max=14895, avg=3077.52, stdev=2456.77 lat (usec): min=59, max=14896, avg=3078.14, stdev=2456.74 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 73], 10.00th=[ 78], 20.00th=[ 99], | 30.00th=[ 2245], 40.00th=[ 2606], 50.00th=[ 2868], 60.00th=[ 3326], | 70.00th=[ 3982], 80.00th=[ 4686], 90.00th=[ 5932], 95.00th=[ 7177], | 99.00th=[12780], 99.50th=[14746], 99.90th=[14877], 99.95th=[14877], | 99.99th=[14877] bw ( KiB/s): min= 1824, max= 2656, per=26.09%, avg=2368.00, stdev=392.79, samples=4 iops : min= 114, max= 166, avg=148.00, stdev=24.55, samples=4 write: IOPS=159, BW=2550KiB/s (2611kB/s)(6368KiB/2497msec) clat (usec): min=670, max=21349, avg=3124.86, stdev=3178.16 lat (usec): min=671, max=21350, avg=3125.71, stdev=3178.15 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 791], 10.00th=[ 824], 20.00th=[ 881], | 30.00th=[ 979], 40.00th=[ 1188], 50.00th=[ 1762], 60.00th=[ 2704], | 70.00th=[ 3392], 80.00th=[ 5211], 90.00th=[ 7177], 95.00th=[ 9372], | 99.00th=[15401], 99.50th=[21103], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 1984, max= 3200, per=27.06%, avg=2584.00, stdev=580.35, samples=4 iops : min= 124, max= 200, avg=161.50, stdev=36.27, samples=4 lat (usec) : 100=9.64%, 250=1.82%, 500=0.52%, 750=1.04%, 1000=15.10% lat (msec) : 2=11.07%, 4=32.55%, 10=25.13%, 20=2.86%, 50=0.26% cpu : usr=0.12%, sys=28.73%, ctx=1521, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10496: Wed Apr 9 01:49:34 2025 read: IOPS=136, BW=2177KiB/s (2229kB/s)(5744KiB/2639msec) clat (usec): min=60, max=23501, avg=3025.38, stdev=2898.98 lat (usec): min=60, max=23501, avg=3027.60, stdev=2899.28 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 74], 10.00th=[ 78], 20.00th=[ 91], | 30.00th=[ 343], 40.00th=[ 2409], 50.00th=[ 2835], 60.00th=[ 3425], | 70.00th=[ 4015], 80.00th=[ 4752], 90.00th=[ 5866], 95.00th=[ 6849], | 99.00th=[13566], 99.50th=[21627], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 544, max= 2976, per=23.62%, avg=2144.00, stdev=930.48, samples=5 iops : min= 34, max= 186, avg=134.00, stdev=58.15, samples=5 write: IOPS=154, BW=2480KiB/s (2539kB/s)(6544KiB/2639msec) clat (usec): min=671, max=34187, avg=3516.00, stdev=4248.95 lat (usec): min=672, max=34189, avg=3516.84, stdev=4248.98 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 742], 10.00th=[ 775], 20.00th=[ 857], | 30.00th=[ 922], 40.00th=[ 1106], 50.00th=[ 1483], 60.00th=[ 2802], | 70.00th=[ 4015], 80.00th=[ 5932], 90.00th=[ 8029], 95.00th=[10814], | 99.00th=[22676], 99.50th=[23200], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 864, max= 3232, per=25.27%, avg=2412.80, stdev=938.04, samples=5 iops : min= 54, max= 202, avg=150.80, stdev=58.63, samples=5 lat (usec) : 100=11.46%, 250=2.34%, 500=0.39%, 750=3.52%, 1000=15.36% lat (msec) : 2=10.68%, 4=25.78%, 10=26.56%, 20=2.73%, 50=1.17% cpu : usr=0.27%, sys=26.50%, ctx=1362, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10497: Wed Apr 9 01:49:34 2025 read: IOPS=160, BW=2561KiB/s (2622kB/s)(6256KiB/2443msec) clat (usec): min=58, max=17457, avg=3121.72, stdev=2493.40 lat (usec): min=59, max=17458, avg=3122.60, stdev=2493.35 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 73], 10.00th=[ 79], 20.00th=[ 98], | 30.00th=[ 2180], 40.00th=[ 2573], 50.00th=[ 2966], 60.00th=[ 3425], | 70.00th=[ 4015], 80.00th=[ 4948], 90.00th=[ 6259], 95.00th=[ 7570], | 99.00th=[10683], 99.50th=[12649], 99.90th=[17433], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 1856, max= 2784, per=26.88%, avg=2440.00, stdev=416.72, samples=4 iops : min= 116, max= 174, avg=152.50, stdev=26.04, samples=4 write: IOPS=154, BW=2469KiB/s (2528kB/s)(6032KiB/2443msec) clat (usec): min=642, max=32242, avg=3188.55, stdev=3331.35 lat (usec): min=642, max=32242, avg=3189.50, stdev=3331.33 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 758], 10.00th=[ 799], 20.00th=[ 857], | 30.00th=[ 963], 40.00th=[ 1188], 50.00th=[ 1942], 60.00th=[ 2966], | 70.00th=[ 3982], 80.00th=[ 5473], 90.00th=[ 6783], 95.00th=[ 8455], | 99.00th=[15533], 99.50th=[25297], 99.90th=[32113], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 2080, max= 2752, per=25.47%, avg=2432.00, stdev=299.05, samples=4 iops : min= 130, max= 172, avg=152.00, stdev=18.69, samples=4 lat (usec) : 100=10.68%, 250=1.95%, 500=0.39%, 750=1.69%, 1000=14.58% lat (msec) : 2=9.11%, 4=31.51%, 10=28.26%, 20=1.56%, 50=0.26% cpu : usr=0.25%, sys=29.28%, ctx=1422, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10498: Wed Apr 9 01:49:34 2025 read: IOPS=148, BW=2379KiB/s (2437kB/s)(6032KiB/2535msec) clat (usec): min=61, max=16376, avg=2985.39, stdev=2578.89 lat (usec): min=61, max=16377, avg=2985.95, stdev=2578.90 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 72], 10.00th=[ 76], 20.00th=[ 91], | 30.00th=[ 1909], 40.00th=[ 2540], 50.00th=[ 2868], 60.00th=[ 3359], | 70.00th=[ 4015], 80.00th=[ 4883], 90.00th=[ 5932], 95.00th=[ 7111], | 99.00th=[11994], 99.50th=[16319], 99.90th=[16319], 99.95th=[16319], | 99.99th=[16319] bw ( KiB/s): min= 1088, max= 2976, per=25.81%, avg=2342.40, stdev=764.46, samples=5 iops : min= 68, max= 186, avg=146.40, stdev=47.78, samples=5 write: IOPS=154, BW=2468KiB/s (2527kB/s)(6256KiB/2535msec) clat (usec): min=644, max=29170, avg=3405.12, stdev=3906.23 lat (usec): min=645, max=29172, avg=3405.98, stdev=3906.27 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 766], 10.00th=[ 799], 20.00th=[ 857], | 30.00th=[ 971], 40.00th=[ 1139], 50.00th=[ 1729], 60.00th=[ 2671], | 70.00th=[ 3949], 80.00th=[ 5669], 90.00th=[ 7373], 95.00th=[10814], | 99.00th=[21627], 99.50th=[25297], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 1376, max= 3008, per=25.60%, avg=2444.80, stdev=625.68, samples=5 iops : min= 86, max= 188, avg=152.80, stdev=39.10, samples=5 lat (usec) : 100=11.46%, 250=2.73%, 500=0.39%, 750=2.21%, 1000=14.71% lat (msec) : 2=10.16%, 4=28.78%, 10=25.52%, 20=3.39%, 50=0.65% cpu : usr=0.16%, sys=27.31%, ctx=1506, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9076KiB/s (9294kB/s), 2177KiB/s-2561KiB/s (2229kB/s-2622kB/s), io=23.4MiB (24.5MB), run=2443-2639msec WRITE: bw=9549KiB/s (9778kB/s), 2468KiB/s-2550KiB/s (2527kB/s-2611kB/s), io=24.6MiB (25.8MB), run=2443-2639msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10491: Wed Apr 9 01:49:40 2025 read: IOPS=44, BW=709KiB/s (726kB/s)(5920KiB/8350msec) clat (usec): min=1765, max=28148, avg=3793.27, stdev=3304.41 lat (usec): min=1766, max=28149, avg=3793.98, stdev=3304.42 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2040], 10.00th=[ 2114], 20.00th=[ 2180], | 30.00th=[ 2311], 40.00th=[ 2409], 50.00th=[ 2507], 60.00th=[ 2704], | 70.00th=[ 3359], 80.00th=[ 4883], 90.00th=[ 6718], 95.00th=[ 9372], | 99.00th=[20841], 99.50th=[27395], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 448, max= 1216, per=25.09%, avg=697.87, stdev=223.05, samples=16 iops : min= 28, max= 76, avg=43.56, stdev=13.97, samples=16 write: IOPS=47, BW=763KiB/s (781kB/s)(6368KiB/8350msec) clat (usec): min=10258, max=50331, avg=17424.26, stdev=5100.08 lat (usec): min=10259, max=50333, avg=17425.66, stdev=5100.09 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11469], 10.00th=[11994], 20.00th=[12911], | 30.00th=[13960], 40.00th=[15401], 50.00th=[16909], 60.00th=[18220], | 70.00th=[19006], 80.00th=[20317], 90.00th=[23987], 95.00th=[27919], | 99.00th=[33424], 99.50th=[35914], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 479, max= 928, per=25.89%, avg=757.94, stdev=166.48, samples=16 iops : min= 29, max= 58, avg=47.31, stdev=10.51, samples=16 lat (msec) : 2=1.43%, 4=34.77%, 10=10.29%, 20=41.54%, 50=11.85% lat (msec) : 100=0.13% cpu : usr=0.11%, sys=6.58%, ctx=923, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10492: Wed Apr 9 01:49:40 2025 read: IOPS=41, BW=667KiB/s (683kB/s)(5744KiB/8608msec) clat (usec): min=1873, max=24698, avg=3796.20, stdev=3298.88 lat (usec): min=1874, max=24699, avg=3796.89, stdev=3298.89 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2024], 10.00th=[ 2089], 20.00th=[ 2180], | 30.00th=[ 2278], 40.00th=[ 2343], 50.00th=[ 2474], 60.00th=[ 2769], | 70.00th=[ 3261], 80.00th=[ 4621], 90.00th=[ 6915], 95.00th=[10159], | 99.00th=[20317], 99.50th=[23987], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 288, max= 1152, per=24.01%, avg=667.88, stdev=247.02, samples=16 iops : min= 18, max= 72, avg=41.69, stdev=15.45, samples=16 write: IOPS=47, BW=760KiB/s (778kB/s)(6544KiB/8608msec) clat (usec): min=9870, max=42533, avg=17554.97, stdev=5356.27 lat (usec): min=9871, max=42534, avg=17556.38, stdev=5356.32 clat percentiles (usec): | 1.00th=[10552], 5.00th=[11207], 10.00th=[11994], 20.00th=[12911], | 30.00th=[13960], 40.00th=[15795], 50.00th=[16909], 60.00th=[17957], | 70.00th=[19268], 80.00th=[20579], 90.00th=[23725], 95.00th=[27395], | 99.00th=[38011], 99.50th=[39060], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 479, max= 960, per=25.55%, avg=747.94, stdev=182.95, samples=16 iops : min= 29, max= 60, avg=46.69, stdev=11.53, samples=16 lat (msec) : 2=1.56%, 4=34.77%, 10=8.20%, 20=42.32%, 50=13.15% cpu : usr=0.12%, sys=6.75%, ctx=921, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10493: Wed Apr 9 01:49:40 2025 read: IOPS=48, BW=772KiB/s (790kB/s)(6256KiB/8104msec) clat (usec): min=1840, max=28182, avg=3556.50, stdev=2863.11 lat (usec): min=1840, max=28183, avg=3557.18, stdev=2863.12 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2008], 10.00th=[ 2089], 20.00th=[ 2180], | 30.00th=[ 2278], 40.00th=[ 2376], 50.00th=[ 2507], 60.00th=[ 2704], | 70.00th=[ 3064], 80.00th=[ 4424], 90.00th=[ 5932], 95.00th=[ 8717], | 99.00th=[15139], 99.50th=[27919], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 383, max= 1312, per=28.03%, avg=779.69, stdev=311.29, samples=16 iops : min= 23, max= 82, avg=48.56, stdev=19.59, samples=16 write: IOPS=46, BW=744KiB/s (762kB/s)(6032KiB/8104msec) clat (usec): min=10066, max=42350, avg=17611.08, stdev=5539.51 lat (usec): min=10068, max=42352, avg=17612.61, stdev=5539.51 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11338], 10.00th=[11863], 20.00th=[13042], | 30.00th=[14091], 40.00th=[15270], 50.00th=[16581], 60.00th=[17957], | 70.00th=[19006], 80.00th=[20841], 90.00th=[25035], 95.00th=[29754], | 99.00th=[35914], 99.50th=[40109], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 480, max= 928, per=25.27%, avg=739.62, stdev=163.36, samples=16 iops : min= 30, max= 58, avg=46.06, stdev=10.32, samples=16 lat (msec) : 2=2.08%, 4=37.24%, 10=9.77%, 20=39.32%, 50=11.59% cpu : usr=0.04%, sys=6.97%, ctx=974, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10494: Wed Apr 9 01:49:40 2025 read: IOPS=45, BW=730KiB/s (748kB/s)(6032KiB/8263msec) clat (usec): min=1818, max=15988, avg=3503.42, stdev=2350.77 lat (usec): min=1818, max=15989, avg=3504.36, stdev=2350.64 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2024], 10.00th=[ 2089], 20.00th=[ 2180], | 30.00th=[ 2278], 40.00th=[ 2343], 50.00th=[ 2474], 60.00th=[ 2671], | 70.00th=[ 3163], 80.00th=[ 4424], 90.00th=[ 6194], 95.00th=[ 8848], | 99.00th=[12387], 99.50th=[15926], 99.90th=[15926], 99.95th=[15926], | 99.99th=[15926] bw ( KiB/s): min= 448, max= 1152, per=25.95%, avg=721.88, stdev=193.53, samples=16 iops : min= 28, max= 72, avg=45.06, stdev=12.15, samples=16 write: IOPS=47, BW=757KiB/s (775kB/s)(6256KiB/8263msec) clat (usec): min=10421, max=36201, avg=17703.16, stdev=4877.35 lat (usec): min=10423, max=36203, avg=17704.62, stdev=4877.30 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11469], 10.00th=[12125], 20.00th=[13173], | 30.00th=[14615], 40.00th=[16057], 50.00th=[17171], 60.00th=[18482], | 70.00th=[19268], 80.00th=[20579], 90.00th=[23200], 95.00th=[27657], | 99.00th=[34866], 99.50th=[35390], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 512, max= 928, per=25.76%, avg=753.88, stdev=149.36, samples=16 iops : min= 32, max= 58, avg=47.06, stdev= 9.42, samples=16 lat (msec) : 2=1.56%, 4=35.94%, 10=9.51%, 20=40.62%, 50=12.37% cpu : usr=0.13%, sys=6.57%, ctx=899, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2783KiB/s (2849kB/s), 667KiB/s-772KiB/s (683kB/s-790kB/s), io=23.4MiB (24.5MB), run=8104-8608msec WRITE: bw=2928KiB/s (2998kB/s), 744KiB/s-763KiB/s (762kB/s-781kB/s), io=24.6MiB (25.8MB), run=8104-8608msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10512: Wed Apr 9 01:49:42 2025 read: IOPS=5, BW=6111KiB/s (6257kB/s)(7168KiB/1173msec) clat (usec): min=421, max=778783, avg=139694.41, stdev=282298.07 lat (usec): min=422, max=778785, avg=139696.12, stdev=282298.40 clat percentiles (usec): | 1.00th=[ 424], 5.00th=[ 424], 10.00th=[ 424], 20.00th=[ 30016], | 30.00th=[ 30540], 40.00th=[ 30540], 50.00th=[ 40109], 60.00th=[ 47973], | 70.00th=[ 47973], 80.00th=[ 50070], 90.00th=[775947], 95.00th=[775947], | 99.00th=[775947], 99.50th=[775947], 99.90th=[775947], 99.95th=[775947], | 99.99th=[775947] bw ( KiB/s): min=10240, max=10240, per=73.35%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4365KiB/s (4470kB/s)(5120KiB/1173msec) clat (usec): min=24228, max=49450, avg=37336.15, stdev=12102.13 lat (usec): min=24245, max=49520, avg=37379.89, stdev=12104.68 clat percentiles (usec): | 1.00th=[24249], 5.00th=[24249], 10.00th=[24249], 20.00th=[24249], | 30.00th=[25822], 40.00th=[25822], 50.00th=[38011], 60.00th=[38011], | 70.00th=[49021], 80.00th=[49021], 90.00th=[49546], 95.00th=[49546], | 99.00th=[49546], 99.50th=[49546], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 2048, max= 2048, per=10.48%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=75.00%, 100=8.33%, 1000=8.33% cpu : usr=0.09%, sys=29.10%, ctx=1085, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10513: Wed Apr 9 01:49:42 2025 read: IOPS=1, BW=1414KiB/s (1448kB/s)(2048KiB/1448msec) clat (usec): min=24020, max=35781, avg=29901.02, stdev=8316.96 lat (usec): min=24020, max=35783, avg=29902.13, stdev=8317.27 clat percentiles (usec): | 1.00th=[23987], 5.00th=[23987], 10.00th=[23987], 20.00th=[23987], | 30.00th=[23987], 40.00th=[23987], 50.00th=[23987], 60.00th=[35914], | 70.00th=[35914], 80.00th=[35914], 90.00th=[35914], 95.00th=[35914], | 99.00th=[35914], 99.50th=[35914], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 2048, max= 2048, per=14.67%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=6, BW=7072KiB/s (7242kB/s)(10.0MiB/1448msec) clat (msec): min=29, max=842, avg=133.19, stdev=249.78 lat (msec): min=29, max=842, avg=133.25, stdev=249.77 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 38], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 46], 60.00th=[ 52], | 70.00th=[ 82], 80.00th=[ 82], 90.00th=[ 91], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 4096, max= 4096, per=20.96%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=33.33%, 1000=8.33% cpu : usr=0.00%, sys=15.96%, ctx=556, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10514: Wed Apr 9 01:49:42 2025 read: IOPS=4, BW=4188KiB/s (4289kB/s)(6144KiB/1467msec) clat (usec): min=22116, max=61980, avg=38682.02, stdev=14698.27 lat (usec): min=22116, max=61981, avg=38682.89, stdev=14698.37 clat percentiles (usec): | 1.00th=[22152], 5.00th=[22152], 10.00th=[22152], 20.00th=[26608], | 30.00th=[26608], 40.00th=[32637], 50.00th=[32637], 60.00th=[41157], | 70.00th=[47449], 80.00th=[47449], 90.00th=[62129], 95.00th=[62129], | 99.00th=[62129], 99.50th=[62129], 99.90th=[62129], 99.95th=[62129], | 99.99th=[62129] write: IOPS=4, BW=4188KiB/s (4289kB/s)(6144KiB/1467msec) clat (msec): min=36, max=878, avg=198.00, stdev=334.52 lat (msec): min=36, max=878, avg=198.03, stdev=334.51 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 45], 50.00th=[ 45], 60.00th=[ 95], | 70.00th=[ 96], 80.00th=[ 96], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 4096, max= 4096, per=20.96%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=66.67%, 100=25.00%, 1000=8.33% cpu : usr=0.00%, sys=15.48%, ctx=495, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10515: Wed Apr 9 01:49:42 2025 read: IOPS=4, BW=4267KiB/s (4369kB/s)(5120KiB/1200msec) clat (usec): min=476, max=53003, avg=27248.69, stdev=21653.43 lat (usec): min=477, max=53003, avg=27249.34, stdev=21653.40 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 478], 10.00th=[ 478], 20.00th=[ 478], | 30.00th=[ 9896], 40.00th=[ 9896], 50.00th=[32900], 60.00th=[32900], | 70.00th=[40109], 80.00th=[40109], 90.00th=[53216], 95.00th=[53216], | 99.00th=[53216], 99.50th=[53216], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 2048, max= 2048, per=14.67%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5973KiB/s (6117kB/s)(7168KiB/1200msec) clat (msec): min=15, max=854, avg=145.90, stdev=312.60 lat (msec): min=15, max=854, avg=145.96, stdev=312.60 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 24], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 30], 60.00th=[ 35], | 70.00th=[ 35], 80.00th=[ 36], 90.00th=[ 852], 95.00th=[ 852], | 99.00th=[ 852], 99.50th=[ 852], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 2048, max= 6144, per=20.96%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (usec) : 500=8.33% lat (msec) : 10=8.33%, 20=8.33%, 50=58.33%, 100=8.33%, 1000=8.33% cpu : usr=0.00%, sys=13.51%, ctx=283, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.6MiB/s (14.3MB/s), 1414KiB/s-6111KiB/s (1448kB/s-6257kB/s), io=20.0MiB (20.0MB), run=1173-1467msec WRITE: bw=19.1MiB/s (20.0MB/s), 4188KiB/s-7072KiB/s (4289kB/s-7242kB/s), io=28.0MiB (29.4MB), run=1173-1467msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10516: Wed Apr 9 01:49:42 2025 read: IOPS=4, BW=4571KiB/s (4681kB/s)(7168KiB/1568msec) clat (msec): min=14, max=731, avg=180.11, stdev=267.29 lat (msec): min=14, max=731, avg=180.11, stdev=267.29 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 18], | 30.00th=[ 23], 40.00th=[ 23], 50.00th=[ 26], 60.00th=[ 130], | 70.00th=[ 130], 80.00th=[ 321], 90.00th=[ 735], 95.00th=[ 735], | 99.00th=[ 735], 99.50th=[ 735], 99.90th=[ 735], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 2048, max= 2048, per=15.68%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3265KiB/s (3344kB/s)(5120KiB/1568msec) clat (usec): min=24914, max=90057, avg=48006.91, stdev=31990.00 lat (usec): min=24995, max=90121, avg=48069.27, stdev=31990.02 clat percentiles (usec): | 1.00th=[25035], 5.00th=[25035], 10.00th=[25035], 20.00th=[25035], | 30.00th=[25035], 40.00th=[25035], 50.00th=[25035], 60.00th=[25035], | 70.00th=[74974], 80.00th=[74974], 90.00th=[89654], 95.00th=[89654], | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] lat (msec) : 20=16.67%, 50=41.67%, 100=16.67%, 250=8.33%, 500=8.33% lat (msec) : 750=8.33% cpu : usr=0.00%, sys=5.23%, ctx=41, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10517: Wed Apr 9 01:49:42 2025 read: IOPS=1, BW=1320KiB/s (1351kB/s)(2048KiB/1552msec) clat (usec): min=21282, max=62020, avg=41651.72, stdev=28805.93 lat (usec): min=21283, max=62021, avg=41652.97, stdev=28806.10 clat percentiles (usec): | 1.00th=[21365], 5.00th=[21365], 10.00th=[21365], 20.00th=[21365], | 30.00th=[21365], 40.00th=[21365], 50.00th=[21365], 60.00th=[62129], | 70.00th=[62129], 80.00th=[62129], 90.00th=[62129], 95.00th=[62129], | 99.00th=[62129], 99.50th=[62129], 99.90th=[62129], 99.95th=[62129], | 99.99th=[62129] bw ( KiB/s): min= 2048, max= 2048, per=15.68%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=6, BW=6598KiB/s (6756kB/s)(10.0MiB/1552msec) clat (msec): min=28, max=887, avg=141.57, stdev=264.03 lat (msec): min=28, max=887, avg=141.64, stdev=264.03 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 35], | 30.00th=[ 41], 40.00th=[ 45], 50.00th=[ 46], 60.00th=[ 57], | 70.00th=[ 73], 80.00th=[ 73], 90.00th=[ 144], 95.00th=[ 885], | 99.00th=[ 885], 99.50th=[ 885], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min=16384, max=16384, per=89.60%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 16, max= 16, avg=16.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=33.33%, 250=8.33%, 1000=8.33% cpu : usr=0.06%, sys=7.61%, ctx=61, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10518: Wed Apr 9 01:49:42 2025 read: IOPS=4, BW=4279KiB/s (4381kB/s)(6144KiB/1436msec) clat (usec): min=32649, max=62668, avg=42734.99, stdev=11550.10 lat (usec): min=32650, max=62670, avg=42735.95, stdev=11550.51 clat percentiles (usec): | 1.00th=[32637], 5.00th=[32637], 10.00th=[32637], 20.00th=[34341], | 30.00th=[34341], 40.00th=[34341], 50.00th=[34341], 60.00th=[44303], | 70.00th=[47973], 80.00th=[47973], 90.00th=[62653], 95.00th=[62653], | 99.00th=[62653], 99.50th=[62653], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] write: IOPS=4, BW=4279KiB/s (4381kB/s)(6144KiB/1436msec) clat (msec): min=25, max=876, avg=194.97, stdev=335.35 lat (msec): min=25, max=876, avg=195.02, stdev=335.35 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 38], 50.00th=[ 38], 60.00th=[ 99], | 70.00th=[ 102], 80.00th=[ 102], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 4096, max= 4096, per=22.40%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=66.67%, 100=16.67%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=4.74%, ctx=46, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10519: Wed Apr 9 01:49:42 2025 read: IOPS=3, BW=3445KiB/s (3528kB/s)(5120KiB/1486msec) clat (usec): min=12497, max=39431, avg=21750.90, stdev=11325.73 lat (usec): min=12498, max=39431, avg=21751.80, stdev=11325.53 clat percentiles (usec): | 1.00th=[12518], 5.00th=[12518], 10.00th=[12518], 20.00th=[12518], | 30.00th=[12780], 40.00th=[12780], 50.00th=[17957], 60.00th=[17957], | 70.00th=[26084], 80.00th=[26084], 90.00th=[39584], 95.00th=[39584], | 99.00th=[39584], 99.50th=[39584], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] write: IOPS=4, BW=4824KiB/s (4939kB/s)(7168KiB/1486msec) clat (msec): min=24, max=859, avg=187.59, stdev=299.55 lat (msec): min=24, max=859, avg=187.65, stdev=299.53 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 26], | 30.00th=[ 64], 40.00th=[ 64], 50.00th=[ 89], 60.00th=[ 97], | 70.00th=[ 97], 80.00th=[ 155], 90.00th=[ 860], 95.00th=[ 860], | 99.00th=[ 860], 99.50th=[ 860], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 2039, max= 2048, per=11.18%, avg=2043.50, stdev= 6.36, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 20=25.00%, 50=33.33%, 100=25.00%, 250=8.33%, 1000=8.33% cpu : usr=0.07%, sys=6.60%, ctx=53, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.8MiB/s (13.4MB/s), 1320KiB/s-4571KiB/s (1351kB/s-4681kB/s), io=20.0MiB (20.0MB), run=1436-1568msec WRITE: bw=17.9MiB/s (18.7MB/s), 3265KiB/s-6598KiB/s (3344kB/s-6756kB/s), io=28.0MiB (29.4MB), run=1436-1568msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10530: Wed Apr 9 01:49:44 2025 read: IOPS=7, BW=30.9MiB/s (32.4MB/s)(12.0MiB/388msec) clat (msec): min=91, max=138, avg=115.98, stdev=23.48 lat (msec): min=91, max=138, avg=115.98, stdev=23.48 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 92], 10.00th=[ 92], 20.00th=[ 92], | 30.00th=[ 92], 40.00th=[ 118], 50.00th=[ 118], 60.00th=[ 118], | 70.00th=[ 140], 80.00th=[ 140], 90.00th=[ 140], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=36.43%, ctx=2086, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10531: Wed Apr 9 01:49:44 2025 read: IOPS=1, BW=6543KiB/s (6700kB/s)(4096KiB/626msec) clat (nsec): min=103358k, max=103358k, avg=103357640.00, stdev= 0.00 lat (nsec): min=103359k, max=103359k, avg=103358918.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 104], | 30.00th=[ 104], 40.00th=[ 104], 50.00th=[ 104], 60.00th=[ 104], | 70.00th=[ 104], 80.00th=[ 104], 90.00th=[ 104], 95.00th=[ 104], | 99.00th=[ 104], 99.50th=[ 104], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 8192, max= 8192, per=31.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=12.8MiB/s (13.4MB/s)(8192KiB/626msec) clat (msec): min=109, max=402, avg=255.71, stdev=207.40 lat (msec): min=109, max=402, avg=255.91, stdev=207.29 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 110], 10.00th=[ 110], 20.00th=[ 110], | 30.00th=[ 110], 40.00th=[ 110], 50.00th=[ 110], 60.00th=[ 401], | 70.00th=[ 401], 80.00th=[ 401], 90.00th=[ 401], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=29.12%, ctx=787, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10532: Wed Apr 9 01:49:44 2025 write: IOPS=5, BW=20.0MiB/s (21.0MB/s)(12.0MiB/599msec) clat (msec): min=185, max=207, avg=195.07, stdev=11.17 lat (msec): min=185, max=208, avg=195.55, stdev=11.49 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 186], 10.00th=[ 186], 20.00th=[ 186], | 30.00th=[ 186], 40.00th=[ 192], 50.00th=[ 192], 60.00th=[ 192], | 70.00th=[ 207], 80.00th=[ 207], 90.00th=[ 207], 95.00th=[ 207], | 99.00th=[ 207], 99.50th=[ 207], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min=16384, max=16384, per=31.30%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.17%, sys=31.10%, ctx=34, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10533: Wed Apr 9 01:49:44 2025 write: IOPS=5, BW=22.5MiB/s (23.6MB/s)(12.0MiB/533msec) clat (msec): min=138, max=210, avg=173.55, stdev=36.25 lat (msec): min=138, max=210, avg=173.83, stdev=36.27 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 138], 10.00th=[ 138], 20.00th=[ 138], | 30.00th=[ 138], 40.00th=[ 171], 50.00th=[ 171], 60.00th=[ 171], | 70.00th=[ 211], 80.00th=[ 211], 90.00th=[ 211], 95.00th=[ 211], | 99.00th=[ 211], 99.50th=[ 211], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min=16384, max=16384, per=31.30%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=43.23%, ctx=62, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.6MiB/s (26.8MB/s), 6543KiB/s-30.9MiB/s (6700kB/s-32.4MB/s), io=16.0MiB (16.8MB), run=388-626msec WRITE: bw=51.1MiB/s (53.6MB/s), 12.8MiB/s-22.5MiB/s (13.4MB/s-23.6MB/s), io=32.0MiB (33.6MB), run=533-626msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10526: Wed Apr 9 01:49:44 2025 read: IOPS=6, BW=26.1MiB/s (27.4MB/s)(12.0MiB/460msec) clat (msec): min=54, max=278, avg=144.89, stdev=118.31 lat (msec): min=54, max=278, avg=144.89, stdev=118.31 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 55], 10.00th=[ 55], 20.00th=[ 55], | 30.00th=[ 55], 40.00th=[ 103], 50.00th=[ 103], 60.00th=[ 103], | 70.00th=[ 279], 80.00th=[ 279], 90.00th=[ 279], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=12.64%, ctx=54, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10527: Wed Apr 9 01:49:44 2025 read: IOPS=1, BW=7074KiB/s (7244kB/s)(4096KiB/579msec) clat (nsec): min=119775k, max=119775k, avg=119774846.00, stdev= 0.00 lat (nsec): min=119778k, max=119778k, avg=119777635.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 121], 10.00th=[ 121], 20.00th=[ 121], | 30.00th=[ 121], 40.00th=[ 121], 50.00th=[ 121], 60.00th=[ 121], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 121], 95.00th=[ 121], | 99.00th=[ 121], 99.50th=[ 121], 99.90th=[ 121], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 8192, max= 8192, per=28.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=13.8MiB/s (14.5MB/s)(8192KiB/579msec) clat (msec): min=213, max=236, avg=224.90, stdev=16.36 lat (msec): min=213, max=236, avg=225.24, stdev=16.38 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 213], 10.00th=[ 213], 20.00th=[ 213], | 30.00th=[ 213], 40.00th=[ 213], 50.00th=[ 213], 60.00th=[ 236], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 236], 95.00th=[ 236], | 99.00th=[ 236], 99.50th=[ 236], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 8192, max= 8192, per=17.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.35%, sys=14.36%, ctx=111, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10528: Wed Apr 9 01:49:44 2025 write: IOPS=4, BW=17.6MiB/s (18.5MB/s)(12.0MiB/680msec) clat (msec): min=137, max=383, avg=222.83, stdev=139.26 lat (msec): min=137, max=383, avg=223.15, stdev=139.32 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 138], 10.00th=[ 138], 20.00th=[ 138], | 30.00th=[ 138], 40.00th=[ 148], 50.00th=[ 148], 60.00th=[ 148], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min=14921, max=14921, per=30.96%, avg=14921.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=11.63%, ctx=98, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10529: Wed Apr 9 01:49:44 2025 write: IOPS=5, BW=23.6MiB/s (24.8MB/s)(12.0MiB/508msec) clat (msec): min=42, max=407, avg=167.63, stdev=208.02 lat (msec): min=42, max=408, avg=167.92, stdev=208.06 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 43], | 30.00th=[ 43], 40.00th=[ 53], 50.00th=[ 53], 60.00th=[ 53], | 70.00th=[ 409], 80.00th=[ 409], 90.00th=[ 409], 95.00th=[ 409], | 99.00th=[ 409], 99.50th=[ 409], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min=16384, max=16384, per=34.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=33.33%, 500=33.33% cpu : usr=0.39%, sys=13.21%, ctx=141, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=27.6MiB/s (28.0MB/s), 7074KiB/s-26.1MiB/s (7244kB/s-27.4MB/s), io=16.0MiB (16.8MB), run=460-579msec WRITE: bw=47.1MiB/s (49.3MB/s), 13.8MiB/s-23.6MiB/s (14.5MB/s-24.8MB/s), io=32.0MiB (33.6MB), run=508-680msec