== sanity test 398b: DIO and buffer IO race ============== 14:35:51 (1744137351) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.34637 s, 37.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29672: Tue Apr 8 14:36:07 2025 read: IOPS=107, BW=429KiB/s (439kB/s)(5992KiB/13975msec) clat (usec): min=53, max=200607, avg=5209.70, stdev=5846.74 lat (usec): min=54, max=200608, avg=5210.55, stdev=5846.81 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 180], 10.00th=[ 2409], 20.00th=[ 2900], | 30.00th=[ 3359], 40.00th=[ 3851], 50.00th=[ 4424], 60.00th=[ 5342], | 70.00th=[ 6128], 80.00th=[ 7046], 90.00th=[ 8586], 95.00th=[ 10552], | 99.00th=[ 15401], 99.50th=[ 17433], 99.90th=[ 26084], 99.95th=[200279], | 99.99th=[200279] bw ( KiB/s): min= 208, max= 616, per=24.80%, avg=421.81, stdev=72.83, samples=27 iops : min= 52, max= 154, avg=105.37, stdev=18.23, samples=27 write: IOPS=112, BW=451KiB/s (461kB/s)(6296KiB/13975msec) clat (usec): min=495, max=30555, avg=3864.70, stdev=3482.56 lat (usec): min=496, max=30556, avg=3866.06, stdev=3483.01 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 586], 10.00th=[ 652], 20.00th=[ 807], | 30.00th=[ 988], 40.00th=[ 1942], 50.00th=[ 3130], 60.00th=[ 3884], | 70.00th=[ 4817], 80.00th=[ 6456], 90.00th=[ 8586], 95.00th=[10814], | 99.00th=[15139], 99.50th=[16450], 99.90th=[22152], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 232, max= 576, per=24.75%, avg=440.78, stdev=75.62, samples=27 iops : min= 58, max= 144, avg=110.11, stdev=18.93, samples=27 lat (usec) : 100=1.27%, 250=1.24%, 500=0.07%, 750=8.53%, 1000=7.06% lat (msec) : 2=5.34%, 4=28.74%, 10=41.70%, 20=5.89%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.30%, sys=22.44%, ctx=5826, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29673: Tue Apr 8 14:36:07 2025 read: IOPS=106, BW=424KiB/s (434kB/s)(5940KiB/14001msec) clat (usec): min=53, max=30528, avg=5075.84, stdev=2861.95 lat (usec): min=53, max=30529, avg=5076.59, stdev=2861.95 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 2008], 10.00th=[ 2507], 20.00th=[ 2966], | 30.00th=[ 3458], 40.00th=[ 3949], 50.00th=[ 4555], 60.00th=[ 5211], | 70.00th=[ 5932], 80.00th=[ 6915], 90.00th=[ 8356], 95.00th=[ 9765], | 99.00th=[15008], 99.50th=[16909], 99.90th=[25822], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 191, max= 624, per=24.60%, avg=418.44, stdev=78.64, samples=27 iops : min= 47, max= 156, avg=104.44, stdev=19.73, samples=27 write: IOPS=113, BW=453KiB/s (464kB/s)(6348KiB/14001msec) clat (usec): min=462, max=200844, avg=4018.06, stdev=5966.65 lat (usec): min=463, max=200846, avg=4019.16, stdev=5966.57 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 578], 10.00th=[ 635], 20.00th=[ 816], | 30.00th=[ 1012], 40.00th=[ 2442], 50.00th=[ 3261], 60.00th=[ 3949], | 70.00th=[ 4883], 80.00th=[ 6456], 90.00th=[ 8717], 95.00th=[ 10552], | 99.00th=[ 14353], 99.50th=[ 16319], 99.90th=[ 20317], 99.95th=[200279], | 99.99th=[200279] bw ( KiB/s): min= 255, max= 584, per=24.76%, avg=440.93, stdev=73.28, samples=27 iops : min= 63, max= 146, avg=110.07, stdev=18.43, samples=27 lat (usec) : 100=0.98%, 250=1.01%, 500=0.16%, 750=8.40%, 1000=6.80% lat (msec) : 2=4.52%, 4=29.23%, 10=43.49%, 20=5.18%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=22.54%, ctx=5682, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29674: Tue Apr 8 14:36:07 2025 read: IOPS=106, BW=428KiB/s (438kB/s)(6028KiB/14096msec) clat (usec): min=58, max=28496, avg=5141.41, stdev=2997.51 lat (usec): min=58, max=28496, avg=5142.71, stdev=2997.21 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 293], 10.00th=[ 2442], 20.00th=[ 2999], | 30.00th=[ 3458], 40.00th=[ 3982], 50.00th=[ 4555], 60.00th=[ 5276], | 70.00th=[ 6128], 80.00th=[ 7111], 90.00th=[ 8586], 95.00th=[10159], | 99.00th=[15270], 99.50th=[19792], 99.90th=[25297], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 152, max= 616, per=25.00%, avg=425.18, stdev=83.69, samples=28 iops : min= 38, max= 154, avg=106.14, stdev=20.92, samples=28 write: IOPS=111, BW=444KiB/s (455kB/s)(6260KiB/14096msec) clat (usec): min=432, max=199371, avg=4000.66, stdev=6062.77 lat (usec): min=433, max=199372, avg=4001.94, stdev=6062.92 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 578], 10.00th=[ 627], 20.00th=[ 791], | 30.00th=[ 930], 40.00th=[ 2180], 50.00th=[ 3130], 60.00th=[ 3884], | 70.00th=[ 4817], 80.00th=[ 6390], 90.00th=[ 8848], 95.00th=[ 10814], | 99.00th=[ 15401], 99.50th=[ 17171], 99.90th=[ 23725], 99.95th=[200279], | 99.99th=[200279] bw ( KiB/s): min= 176, max= 646, per=24.74%, avg=440.57, stdev=100.89, samples=28 iops : min= 44, max= 161, avg=110.00, stdev=25.21, samples=28 lat (usec) : 100=1.43%, 250=1.01%, 500=0.20%, 750=9.02%, 1000=7.39% lat (msec) : 2=3.74%, 4=28.61%, 10=42.64%, 20=5.70%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=22.08%, ctx=5644, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29675: Tue Apr 8 14:36:07 2025 read: IOPS=107, BW=429KiB/s (439kB/s)(6052KiB/14110msec) clat (usec): min=63, max=24392, avg=5062.78, stdev=2767.26 lat (usec): min=64, max=24393, avg=5063.95, stdev=2767.68 clat percentiles (usec): | 1.00th=[ 96], 5.00th=[ 2114], 10.00th=[ 2540], 20.00th=[ 2966], | 30.00th=[ 3359], 40.00th=[ 3916], 50.00th=[ 4555], 60.00th=[ 5145], | 70.00th=[ 5997], 80.00th=[ 7046], 90.00th=[ 8455], 95.00th=[ 9896], | 99.00th=[14615], 99.50th=[15926], 99.90th=[22414], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 240, max= 574, per=25.08%, avg=426.64, stdev=69.71, samples=28 iops : min= 60, max= 143, avg=106.54, stdev=17.38, samples=28 write: IOPS=110, BW=442KiB/s (453kB/s)(6236KiB/14110msec) clat (usec): min=418, max=209967, avg=4083.94, stdev=6251.02 lat (usec): min=419, max=209969, avg=4085.45, stdev=6251.25 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 578], 10.00th=[ 644], 20.00th=[ 807], | 30.00th=[ 996], 40.00th=[ 2278], 50.00th=[ 3261], 60.00th=[ 4015], | 70.00th=[ 5145], 80.00th=[ 6915], 90.00th=[ 8848], 95.00th=[ 10552], | 99.00th=[ 14353], 99.50th=[ 15664], 99.90th=[ 23725], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 296, max= 662, per=24.74%, avg=440.61, stdev=80.03, samples=28 iops : min= 74, max= 165, avg=110.04, stdev=19.99, samples=28 lat (usec) : 100=0.65%, 250=1.30%, 500=0.16%, 750=8.50%, 1000=6.74% lat (msec) : 2=4.59%, 4=28.58%, 10=43.98%, 20=5.34%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=22.11%, ctx=5863, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1702KiB/s (1743kB/s), 424KiB/s-429KiB/s (434kB/s-439kB/s), io=23.4MiB (24.6MB), run=13975-14110msec WRITE: bw=1782KiB/s (1824kB/s), 442KiB/s-453KiB/s (453kB/s-464kB/s), io=24.6MiB (25.7MB), run=13975-14110msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29676: Tue Apr 8 14:36:11 2025 read: IOPS=84, BW=337KiB/s (345kB/s)(5992KiB/17767msec) clat (usec): min=1688, max=203797, avg=4843.43, stdev=6346.07 lat (usec): min=1689, max=203800, avg=4844.17, stdev=6346.12 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1909], 10.00th=[ 2024], 20.00th=[ 2147], | 30.00th=[ 2343], 40.00th=[ 2638], 50.00th=[ 3130], 60.00th=[ 3720], | 70.00th=[ 4817], 80.00th=[ 7046], 90.00th=[ 10290], 95.00th=[ 12911], | 99.00th=[ 18744], 99.50th=[ 20841], 99.90th=[ 28705], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 120, max= 816, per=24.69%, avg=332.06, stdev=177.68, samples=35 iops : min= 30, max= 204, avg=82.97, stdev=44.44, samples=35 write: IOPS=88, BW=354KiB/s (363kB/s)(6296KiB/17767msec) clat (usec): min=2321, max=37496, avg=6631.60, stdev=4394.96 lat (usec): min=2321, max=37497, avg=6632.50, stdev=4395.03 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2933], 10.00th=[ 3130], 20.00th=[ 3425], | 30.00th=[ 3687], 40.00th=[ 4047], 50.00th=[ 4752], 60.00th=[ 5932], | 70.00th=[ 7767], 80.00th=[ 9634], 90.00th=[12256], 95.00th=[14746], | 99.00th=[23987], 99.50th=[27132], 99.90th=[32113], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 136, max= 768, per=24.77%, avg=348.97, stdev=181.62, samples=35 iops : min= 34, max= 192, avg=87.20, stdev=45.42, samples=35 lat (msec) : 2=4.20%, 4=47.27%, 10=33.46%, 20=13.90%, 50=1.14% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=11.74%, ctx=3558, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29677: Tue Apr 8 14:36:11 2025 read: IOPS=83, BW=333KiB/s (341kB/s)(5940KiB/17840msec) clat (usec): min=1619, max=208038, avg=4784.99, stdev=6434.84 lat (usec): min=1620, max=208041, avg=4785.72, stdev=6434.90 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1893], 10.00th=[ 1991], 20.00th=[ 2147], | 30.00th=[ 2343], 40.00th=[ 2606], 50.00th=[ 3097], 60.00th=[ 3720], | 70.00th=[ 4555], 80.00th=[ 6456], 90.00th=[ 10683], 95.00th=[ 12911], | 99.00th=[ 17695], 99.50th=[ 20579], 99.90th=[ 26870], 99.95th=[208667], | 99.99th=[208667] bw ( KiB/s): min= 80, max= 752, per=24.41%, avg=328.29, stdev=186.78, samples=35 iops : min= 20, max= 188, avg=82.00, stdev=46.67, samples=35 write: IOPS=88, BW=356KiB/s (364kB/s)(6348KiB/17840msec) clat (usec): min=2336, max=37433, avg=6707.96, stdev=4457.92 lat (usec): min=2337, max=37434, avg=6709.04, stdev=4458.00 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2900], 10.00th=[ 3130], 20.00th=[ 3392], | 30.00th=[ 3687], 40.00th=[ 4047], 50.00th=[ 4948], 60.00th=[ 6194], | 70.00th=[ 7767], 80.00th=[ 9634], 90.00th=[12780], 95.00th=[15401], | 99.00th=[22414], 99.50th=[26084], 99.90th=[36439], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 136, max= 768, per=24.63%, avg=347.03, stdev=175.56, samples=35 iops : min= 34, max= 192, avg=86.69, stdev=43.87, samples=35 lat (msec) : 2=5.01%, 4=46.03%, 10=33.69%, 20=14.03%, 50=1.20% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=11.49%, ctx=3632, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29678: Tue Apr 8 14:36:11 2025 read: IOPS=84, BW=340KiB/s (348kB/s)(6028KiB/17746msec) clat (usec): min=1613, max=33257, avg=4758.42, stdev=3938.67 lat (usec): min=1614, max=33257, avg=4759.16, stdev=3938.69 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1926], 10.00th=[ 2024], 20.00th=[ 2180], | 30.00th=[ 2376], 40.00th=[ 2638], 50.00th=[ 3097], 60.00th=[ 3687], | 70.00th=[ 4817], 80.00th=[ 6783], 90.00th=[10159], 95.00th=[12649], | 99.00th=[18220], 99.50th=[23987], 99.90th=[31327], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 111, max= 718, per=24.93%, avg=335.26, stdev=174.70, samples=35 iops : min= 27, max= 179, avg=83.66, stdev=43.71, samples=35 write: IOPS=88, BW=353KiB/s (361kB/s)(6260KiB/17746msec) clat (msec): min=2, max=209, avg= 6.70, stdev= 6.60 lat (msec): min=2, max=209, avg= 6.70, stdev= 6.60 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 16], | 99.00th=[ 21], 99.50th=[ 26], 99.90th=[ 31], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 143, max= 752, per=24.67%, avg=347.60, stdev=181.67, samples=35 iops : min= 35, max= 188, avg=86.74, stdev=45.43, samples=35 lat (msec) : 2=4.39%, 4=46.09%, 10=35.35%, 20=13.15%, 50=0.98% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=11.60%, ctx=3563, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29679: Tue Apr 8 14:36:11 2025 read: IOPS=85, BW=341KiB/s (349kB/s)(6052KiB/17736msec) clat (usec): min=1607, max=36438, avg=4640.38, stdev=3732.21 lat (usec): min=1608, max=36439, avg=4641.17, stdev=3732.25 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1909], 10.00th=[ 2008], 20.00th=[ 2180], | 30.00th=[ 2376], 40.00th=[ 2671], 50.00th=[ 3064], 60.00th=[ 3621], | 70.00th=[ 4686], 80.00th=[ 6521], 90.00th=[10159], 95.00th=[12387], | 99.00th=[17695], 99.50th=[21103], 99.90th=[27657], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 112, max= 776, per=25.04%, avg=336.83, stdev=178.70, samples=35 iops : min= 28, max= 194, avg=84.14, stdev=44.72, samples=35 write: IOPS=87, BW=352KiB/s (360kB/s)(6236KiB/17736msec) clat (msec): min=2, max=221, avg= 6.82, stdev= 6.95 lat (msec): min=2, max=221, avg= 6.82, stdev= 6.95 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 16], | 99.00th=[ 24], 99.50th=[ 27], 99.90th=[ 37], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 144, max= 752, per=24.62%, avg=346.89, stdev=179.64, samples=35 iops : min= 36, max= 188, avg=86.66, stdev=44.94, samples=35 lat (msec) : 2=4.92%, 4=46.42%, 10=33.82%, 20=13.74%, 50=1.07% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=11.56%, ctx=3577, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1346KiB/s (1378kB/s), 333KiB/s-341KiB/s (341kB/s-349kB/s), io=23.4MiB (24.6MB), run=17736-17840msec WRITE: bw=1409KiB/s (1443kB/s), 352KiB/s-356KiB/s (360kB/s-364kB/s), io=24.6MiB (25.7MB), run=17736-17840msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29706: Tue Apr 8 14:36:16 2025 read: IOPS=111, BW=1780KiB/s (1823kB/s)(5920KiB/3326msec) clat (usec): min=61, max=35234, avg=4846.28, stdev=3538.09 lat (usec): min=61, max=35235, avg=4846.96, stdev=3538.11 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 83], 10.00th=[ 113], 20.00th=[ 2704], | 30.00th=[ 3228], 40.00th=[ 3949], 50.00th=[ 4621], 60.00th=[ 5276], | 70.00th=[ 5932], 80.00th=[ 6980], 90.00th=[ 8356], 95.00th=[10290], | 99.00th=[16909], 99.50th=[18482], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 1568, max= 2208, per=25.63%, avg=1722.67, stdev=253.12, samples=6 iops : min= 98, max= 138, avg=107.67, stdev=15.82, samples=6 write: IOPS=119, BW=1915KiB/s (1961kB/s)(6368KiB/3326msec) clat (usec): min=702, max=17352, avg=3769.83, stdev=3300.89 lat (usec): min=704, max=17353, avg=3770.89, stdev=3300.89 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 816], 10.00th=[ 873], 20.00th=[ 971], | 30.00th=[ 1270], 40.00th=[ 1500], 50.00th=[ 2737], 60.00th=[ 3720], | 70.00th=[ 4686], 80.00th=[ 5997], 90.00th=[ 8717], 95.00th=[10683], | 99.00th=[14222], 99.50th=[16712], 99.90th=[17433], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 1248, max= 2432, per=26.48%, avg=1872.00, stdev=429.68, samples=6 iops : min= 78, max= 152, avg=117.00, stdev=26.86, samples=6 lat (usec) : 100=3.91%, 250=3.39%, 500=0.26%, 750=0.65%, 1000=10.29% lat (msec) : 2=11.98%, 4=22.01%, 10=41.28%, 20=6.12%, 50=0.13% cpu : usr=0.18%, sys=26.38%, ctx=1514, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29707: Tue Apr 8 14:36:16 2025 read: IOPS=102, BW=1635KiB/s (1674kB/s)(5744KiB/3514msec) clat (usec): min=67, max=18153, avg=4828.23, stdev=2843.80 lat (usec): min=67, max=18153, avg=4829.14, stdev=2843.75 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 111], 10.00th=[ 133], 20.00th=[ 2868], | 30.00th=[ 3392], 40.00th=[ 4047], 50.00th=[ 4555], 60.00th=[ 5407], | 70.00th=[ 5932], 80.00th=[ 6980], 90.00th=[ 8356], 95.00th=[ 9241], | 99.00th=[13173], 99.50th=[14484], 99.90th=[18220], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 1216, max= 1856, per=24.13%, avg=1621.33, stdev=221.39, samples=6 iops : min= 76, max= 116, avg=101.33, stdev=13.84, samples=6 write: IOPS=116, BW=1862KiB/s (1907kB/s)(6544KiB/3514msec) clat (usec): min=680, max=28460, avg=4291.20, stdev=3556.62 lat (usec): min=680, max=28461, avg=4292.45, stdev=3556.69 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 848], 10.00th=[ 914], 20.00th=[ 1123], | 30.00th=[ 1385], 40.00th=[ 2540], 50.00th=[ 3425], 60.00th=[ 4490], | 70.00th=[ 5538], 80.00th=[ 7242], 90.00th=[ 8717], 95.00th=[11338], | 99.00th=[14746], 99.50th=[17171], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 1600, max= 1984, per=25.95%, avg=1834.67, stdev=141.19, samples=6 iops : min= 100, max= 124, avg=114.67, stdev= 8.82, samples=6 lat (usec) : 100=1.95%, 250=3.39%, 500=0.13%, 750=0.65%, 1000=7.81% lat (msec) : 2=11.07%, 4=22.92%, 10=46.74%, 20=5.21%, 50=0.13% cpu : usr=0.17%, sys=25.45%, ctx=1590, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29708: Tue Apr 8 14:36:16 2025 read: IOPS=109, BW=1755KiB/s (1797kB/s)(6256KiB/3564msec) clat (usec): min=60, max=17821, avg=4945.60, stdev=2925.77 lat (usec): min=61, max=17822, avg=4946.35, stdev=2925.76 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 86], 10.00th=[ 137], 20.00th=[ 2933], | 30.00th=[ 3392], 40.00th=[ 3949], 50.00th=[ 4621], 60.00th=[ 5407], | 70.00th=[ 6194], 80.00th=[ 7242], 90.00th=[ 8455], 95.00th=[ 9896], | 99.00th=[14353], 99.50th=[16450], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 1344, max= 2336, per=26.04%, avg=1749.71, stdev=379.65, samples=7 iops : min= 84, max= 146, avg=109.14, stdev=23.78, samples=7 write: IOPS=105, BW=1692KiB/s (1733kB/s)(6032KiB/3564msec) clat (usec): min=692, max=27795, avg=4269.18, stdev=3496.39 lat (usec): min=693, max=27796, avg=4270.19, stdev=3496.39 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 848], 10.00th=[ 914], 20.00th=[ 1172], | 30.00th=[ 1532], 40.00th=[ 2573], 50.00th=[ 3556], 60.00th=[ 4293], | 70.00th=[ 5407], 80.00th=[ 7046], 90.00th=[ 8979], 95.00th=[10945], | 99.00th=[15008], 99.50th=[15664], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 1149, max= 1948, per=23.59%, avg=1667.57, stdev=260.52, samples=7 iops : min= 71, max= 121, avg=104.00, stdev=16.42, samples=7 lat (usec) : 100=3.12%, 250=2.34%, 500=0.13%, 750=0.91%, 1000=5.99% lat (msec) : 2=10.81%, 4=25.52%, 10=45.18%, 20=5.86%, 50=0.13% cpu : usr=0.20%, sys=25.32%, ctx=1672, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29709: Tue Apr 8 14:36:16 2025 read: IOPS=108, BW=1728KiB/s (1770kB/s)(6032KiB/3490msec) clat (usec): min=68, max=18578, avg=4988.61, stdev=2874.23 lat (usec): min=68, max=18578, avg=4989.26, stdev=2874.23 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 85], 10.00th=[ 135], 20.00th=[ 3097], | 30.00th=[ 3523], 40.00th=[ 4080], 50.00th=[ 4817], 60.00th=[ 5473], | 70.00th=[ 6325], 80.00th=[ 7177], 90.00th=[ 8586], 95.00th=[ 9765], | 99.00th=[12518], 99.50th=[17171], 99.90th=[18482], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 1568, max= 2011, per=25.38%, avg=1705.83, stdev=170.73, samples=6 iops : min= 98, max= 125, avg=106.50, stdev=10.43, samples=6 write: IOPS=112, BW=1793KiB/s (1836kB/s)(6256KiB/3490msec) clat (usec): min=649, max=17800, avg=4073.82, stdev=3198.25 lat (usec): min=650, max=17801, avg=4074.85, stdev=3198.25 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 865], 10.00th=[ 955], 20.00th=[ 1156], | 30.00th=[ 1418], 40.00th=[ 2245], 50.00th=[ 3163], 60.00th=[ 4228], | 70.00th=[ 5473], 80.00th=[ 6652], 90.00th=[ 8717], 95.00th=[10290], | 99.00th=[13960], 99.50th=[14877], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 1440, max= 2112, per=25.04%, avg=1770.00, stdev=242.09, samples=6 iops : min= 90, max= 132, avg=110.50, stdev=15.20, samples=6 lat (usec) : 100=2.86%, 250=2.47%, 750=0.26%, 1000=6.51% lat (msec) : 2=12.37%, 4=24.22%, 10=46.09%, 20=5.21% cpu : usr=0.32%, sys=25.91%, ctx=1628, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6721KiB/s (6882kB/s), 1635KiB/s-1780KiB/s (1674kB/s-1823kB/s), io=23.4MiB (24.5MB), run=3326-3564msec WRITE: bw=7071KiB/s (7240kB/s), 1692KiB/s-1915KiB/s (1733kB/s-1961kB/s), io=24.6MiB (25.8MB), run=3326-3564msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29711: Tue Apr 8 14:36:17 2025 read: IOPS=80, BW=1285KiB/s (1316kB/s)(5920KiB/4606msec) clat (usec): min=1902, max=32484, avg=4969.59, stdev=3845.22 lat (usec): min=1902, max=32485, avg=4970.32, stdev=3845.22 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2180], 10.00th=[ 2343], 20.00th=[ 2507], | 30.00th=[ 2769], 40.00th=[ 2966], 50.00th=[ 3294], 60.00th=[ 3752], | 70.00th=[ 4883], 80.00th=[ 6718], 90.00th=[10421], 95.00th=[13042], | 99.00th=[17695], 99.50th=[21365], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 734, max= 2240, per=24.42%, avg=1261.78, stdev=556.54, samples=9 iops : min= 45, max= 140, avg=78.67, stdev=34.97, samples=9 write: IOPS=86, BW=1383KiB/s (1416kB/s)(6368KiB/4606msec) clat (usec): min=2933, max=23661, avg=6923.99, stdev=4035.71 lat (usec): min=2934, max=23663, avg=6925.37, stdev=4035.75 clat percentiles (usec): | 1.00th=[ 2999], 5.00th=[ 3392], 10.00th=[ 3654], 20.00th=[ 3916], | 30.00th=[ 4146], 40.00th=[ 4555], 50.00th=[ 5145], 60.00th=[ 6390], | 70.00th=[ 7898], 80.00th=[ 9634], 90.00th=[13566], 95.00th=[15008], | 99.00th=[21627], 99.50th=[23725], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 928, max= 2464, per=25.24%, avg=1371.78, stdev=562.80, samples=9 iops : min= 58, max= 154, avg=85.56, stdev=35.23, samples=9 lat (msec) : 2=0.26%, 4=42.06%, 10=42.84%, 20=13.67%, 50=1.17% cpu : usr=0.26%, sys=12.60%, ctx=920, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29712: Tue Apr 8 14:36:17 2025 read: IOPS=77, BW=1239KiB/s (1269kB/s)(5744KiB/4635msec) clat (usec): min=1975, max=27093, avg=4926.86, stdev=3771.39 lat (usec): min=1976, max=27093, avg=4927.57, stdev=3771.38 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2278], 10.00th=[ 2409], 20.00th=[ 2540], | 30.00th=[ 2737], 40.00th=[ 2933], 50.00th=[ 3294], 60.00th=[ 3720], | 70.00th=[ 4686], 80.00th=[ 6587], 90.00th=[11731], 95.00th=[13829], | 99.00th=[16581], 99.50th=[17433], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 704, max= 2144, per=23.73%, avg=1225.89, stdev=496.07, samples=9 iops : min= 44, max= 134, avg=76.44, stdev=30.88, samples=9 write: IOPS=88, BW=1412KiB/s (1446kB/s)(6544KiB/4635msec) clat (usec): min=2847, max=32532, avg=6936.27, stdev=4064.87 lat (usec): min=2848, max=32534, avg=6937.66, stdev=4064.84 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3425], 10.00th=[ 3654], 20.00th=[ 4015], | 30.00th=[ 4293], 40.00th=[ 4621], 50.00th=[ 5211], 60.00th=[ 6652], | 70.00th=[ 7898], 80.00th=[ 9372], 90.00th=[12387], 95.00th=[14746], | 99.00th=[21890], 99.50th=[23725], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 928, max= 2618, per=25.36%, avg=1378.56, stdev=554.43, samples=9 iops : min= 58, max= 163, avg=86.00, stdev=34.51, samples=9 lat (msec) : 2=0.13%, 4=40.36%, 10=44.53%, 20=14.19%, 50=0.78% cpu : usr=0.13%, sys=12.30%, ctx=915, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29713: Tue Apr 8 14:36:17 2025 read: IOPS=84, BW=1356KiB/s (1389kB/s)(6256KiB/4613msec) clat (usec): min=1830, max=27210, avg=5032.17, stdev=4009.68 lat (usec): min=1831, max=27211, avg=5032.84, stdev=4009.68 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2212], 10.00th=[ 2311], 20.00th=[ 2507], | 30.00th=[ 2704], 40.00th=[ 2933], 50.00th=[ 3294], 60.00th=[ 3720], | 70.00th=[ 4621], 80.00th=[ 7177], 90.00th=[11338], 95.00th=[14222], | 99.00th=[22152], 99.50th=[23987], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 800, max= 2752, per=26.15%, avg=1351.11, stdev=651.26, samples=9 iops : min= 50, max= 172, avg=84.44, stdev=40.70, samples=9 write: IOPS=81, BW=1308KiB/s (1339kB/s)(6032KiB/4613msec) clat (usec): min=2818, max=24561, avg=6879.62, stdev=3958.20 lat (usec): min=2819, max=24562, avg=6880.93, stdev=3958.17 clat percentiles (usec): | 1.00th=[ 2999], 5.00th=[ 3425], 10.00th=[ 3621], 20.00th=[ 3884], | 30.00th=[ 4178], 40.00th=[ 4490], 50.00th=[ 5145], 60.00th=[ 6194], | 70.00th=[ 7701], 80.00th=[10028], 90.00th=[12256], 95.00th=[14877], | 99.00th=[21627], 99.50th=[23725], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 960, max= 2176, per=23.55%, avg=1280.00, stdev=502.66, samples=9 iops : min= 60, max= 136, avg=80.00, stdev=31.42, samples=9 lat (msec) : 2=0.26%, 4=44.53%, 10=38.67%, 20=15.49%, 50=1.04% cpu : usr=0.26%, sys=12.21%, ctx=937, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29714: Tue Apr 8 14:36:17 2025 read: IOPS=83, BW=1328KiB/s (1360kB/s)(6032KiB/4542msec) clat (usec): min=2025, max=24271, avg=4857.01, stdev=3543.51 lat (usec): min=2026, max=24272, avg=4857.80, stdev=3543.54 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2245], 10.00th=[ 2343], 20.00th=[ 2540], | 30.00th=[ 2704], 40.00th=[ 2999], 50.00th=[ 3359], 60.00th=[ 3851], | 70.00th=[ 4555], 80.00th=[ 6652], 90.00th=[10683], 95.00th=[13173], | 99.00th=[16188], 99.50th=[20841], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 896, max= 2560, per=25.46%, avg=1315.56, stdev=577.87, samples=9 iops : min= 56, max= 160, avg=82.22, stdev=36.12, samples=9 write: IOPS=86, BW=1377KiB/s (1410kB/s)(6256KiB/4542msec) clat (usec): min=2629, max=32437, avg=6835.84, stdev=4243.59 lat (usec): min=2630, max=32439, avg=6837.15, stdev=4243.60 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3228], 10.00th=[ 3523], 20.00th=[ 3851], | 30.00th=[ 4146], 40.00th=[ 4424], 50.00th=[ 4948], 60.00th=[ 5997], | 70.00th=[ 7701], 80.00th=[ 9503], 90.00th=[12518], 95.00th=[15795], | 99.00th=[20579], 99.50th=[28443], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 928, max= 2336, per=25.25%, avg=1372.44, stdev=551.35, samples=9 iops : min= 58, max= 146, avg=85.78, stdev=34.46, samples=9 lat (msec) : 4=43.36%, 10=42.84%, 20=12.89%, 50=0.91% cpu : usr=0.24%, sys=12.53%, ctx=923, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5168KiB/s (5292kB/s), 1239KiB/s-1356KiB/s (1269kB/s-1389kB/s), io=23.4MiB (24.5MB), run=4542-4635msec WRITE: bw=5437KiB/s (5567kB/s), 1308KiB/s-1412KiB/s (1339kB/s-1446kB/s), io=24.6MiB (25.8MB), run=4542-4635msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29724: Tue Apr 8 14:36:18 2025 read: IOPS=16, BW=16.1MiB/s (16.9MB/s)(7168KiB/435msec) clat (usec): min=416, max=41949, avg=23800.98, stdev=16509.55 lat (usec): min=417, max=41950, avg=23802.29, stdev=16509.80 clat percentiles (usec): | 1.00th=[ 416], 5.00th=[ 416], 10.00th=[ 416], 20.00th=[ 578], | 30.00th=[28705], 40.00th=[28705], 50.00th=[30016], 60.00th=[30540], | 70.00th=[30540], 80.00th=[34341], 90.00th=[42206], 95.00th=[42206], | 99.00th=[42206], 99.50th=[42206], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] write: IOPS=11, BW=11.5MiB/s (12.1MB/s)(5120KiB/435msec) clat (usec): min=36017, max=68523, avg=50849.70, stdev=11819.23 lat (usec): min=36036, max=68583, avg=50898.85, stdev=11833.48 clat percentiles (usec): | 1.00th=[35914], 5.00th=[35914], 10.00th=[35914], 20.00th=[35914], | 30.00th=[47449], 40.00th=[47449], 50.00th=[48497], 60.00th=[48497], | 70.00th=[53740], 80.00th=[53740], 90.00th=[68682], 95.00th=[68682], | 99.00th=[68682], 99.50th=[68682], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=66.67%, 100=16.67% cpu : usr=0.23%, sys=47.00%, ctx=964, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29726: Tue Apr 8 14:36:18 2025 read: IOPS=3, BW=3821KiB/s (3913kB/s)(2048KiB/536msec) clat (usec): min=613, max=55960, avg=28287.26, stdev=39136.44 lat (usec): min=615, max=55962, avg=28288.97, stdev=39136.24 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 611], 10.00th=[ 611], 20.00th=[ 611], | 30.00th=[ 611], 40.00th=[ 611], 50.00th=[ 611], 60.00th=[55837], | 70.00th=[55837], 80.00th=[55837], 90.00th=[55837], 95.00th=[55837], | 99.00th=[55837], 99.50th=[55837], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] write: IOPS=18, BW=18.7MiB/s (19.6MB/s)(10.0MiB/536msec) clat (usec): min=23160, max=57373, avg=42731.43, stdev=10487.12 lat (usec): min=23230, max=57440, avg=42793.60, stdev=10490.53 clat percentiles (usec): | 1.00th=[23200], 5.00th=[23200], 10.00th=[23200], 20.00th=[35390], | 30.00th=[36439], 40.00th=[37487], 50.00th=[40633], 60.00th=[43779], | 70.00th=[51119], 80.00th=[51119], 90.00th=[55837], 95.00th=[57410], | 99.00th=[57410], 99.50th=[57410], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] lat (usec) : 750=8.33% lat (msec) : 50=58.33%, 100=33.33% cpu : usr=0.19%, sys=43.36%, ctx=78, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29727: Tue Apr 8 14:36:18 2025 read: IOPS=11, BW=11.4MiB/s (11.0MB/s)(6144KiB/526msec) clat (usec): min=424, max=44669, avg=11759.60, stdev=18678.19 lat (usec): min=425, max=44670, avg=11760.88, stdev=18677.90 clat percentiles (usec): | 1.00th=[ 424], 5.00th=[ 424], 10.00th=[ 424], 20.00th=[ 441], | 30.00th=[ 441], 40.00th=[ 474], 50.00th=[ 474], 60.00th=[ 515], | 70.00th=[23987], 80.00th=[23987], 90.00th=[44827], 95.00th=[44827], | 99.00th=[44827], 99.50th=[44827], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min=10118, max=10118, per=26.48%, avg=10118.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.4MiB/s (11.0MB/s)(6144KiB/526msec) clat (msec): min=28, max=147, avg=64.68, stdev=45.14 lat (msec): min=28, max=147, avg=64.72, stdev=45.15 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 41], 50.00th=[ 41], 60.00th=[ 50], | 70.00th=[ 85], 80.00th=[ 85], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min=10118, max=10118, per=18.91%, avg=10118.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (usec) : 500=25.00%, 750=8.33% lat (msec) : 50=50.00%, 100=8.33%, 250=8.33% cpu : usr=0.00%, sys=32.19%, ctx=489, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29729: Tue Apr 8 14:36:18 2025 read: IOPS=10, BW=10.2MiB/s (10.7MB/s)(5120KiB/488msec) clat (usec): min=443, max=40846, avg=18358.16, stdev=18396.14 lat (usec): min=444, max=40846, avg=18359.21, stdev=18395.79 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 445], 10.00th=[ 445], 20.00th=[ 445], | 30.00th=[ 578], 40.00th=[ 578], 50.00th=[17171], 60.00th=[17171], | 70.00th=[32900], 80.00th=[32900], 90.00th=[40633], 95.00th=[40633], | 99.00th=[40633], 99.50th=[40633], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] write: IOPS=14, BW=14.3MiB/s (15.0MB/s)(7168KiB/488msec) clat (msec): min=35, max=102, avg=54.25, stdev=25.46 lat (msec): min=35, max=102, avg=54.31, stdev=25.47 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 39], 60.00th=[ 64], | 70.00th=[ 64], 80.00th=[ 67], 90.00th=[ 104], 95.00th=[ 104], | 99.00th=[ 104], 99.50th=[ 104], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 20=8.33%, 50=50.00%, 100=16.67%, 250=8.33% cpu : usr=0.00%, sys=32.24%, ctx=272, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=37.3MiB/s (39.1MB/s), 3821KiB/s-16.1MiB/s (3913kB/s-16.9MB/s), io=20.0MiB (20.0MB), run=435-536msec WRITE: bw=52.2MiB/s (54.8MB/s), 11.4MiB/s-18.7MiB/s (11.0MB/s-19.6MB/s), io=28.0MiB (29.4MB), run=435-536msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29725: Tue Apr 8 14:36:18 2025 read: IOPS=9, BW=9570KiB/s (9800kB/s)(7168KiB/749msec) clat (msec): min=15, max=123, avg=62.96, stdev=46.98 lat (msec): min=15, max=123, avg=62.96, stdev=46.98 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 21], | 30.00th=[ 29], 40.00th=[ 29], 50.00th=[ 39], 60.00th=[ 106], | 70.00th=[ 106], 80.00th=[ 109], 90.00th=[ 124], 95.00th=[ 124], | 99.00th=[ 124], 99.50th=[ 124], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 8192, max= 8192, per=30.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6836KiB/s (7000kB/s)(5120KiB/749msec) clat (usec): min=21626, max=67786, avg=42232.55, stdev=22646.26 lat (usec): min=21707, max=67875, avg=42304.53, stdev=22646.06 clat percentiles (usec): | 1.00th=[21627], 5.00th=[21627], 10.00th=[21627], 20.00th=[21627], | 30.00th=[26346], 40.00th=[26346], 50.00th=[29492], 60.00th=[29492], | 70.00th=[65799], 80.00th=[65799], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 2048, max= 2048, per=5.36%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=16.67%, 250=25.00% cpu : usr=0.00%, sys=14.04%, ctx=69, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29728: Tue Apr 8 14:36:18 2025 read: IOPS=2, BW=2817KiB/s (2885kB/s)(2048KiB/727msec) clat (usec): min=28839, max=66498, avg=47668.94, stdev=26629.21 lat (usec): min=28840, max=66500, avg=47670.40, stdev=26629.46 clat percentiles (usec): | 1.00th=[28967], 5.00th=[28967], 10.00th=[28967], 20.00th=[28967], | 30.00th=[28967], 40.00th=[28967], 50.00th=[28967], 60.00th=[66323], | 70.00th=[66323], 80.00th=[66323], 90.00th=[66323], 95.00th=[66323], | 99.00th=[66323], 99.50th=[66323], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 2048, max= 2048, per=7.50%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.8MiB/s (14.4MB/s)(10.0MiB/727msec) clat (msec): min=21, max=153, avg=57.67, stdev=41.60 lat (msec): min=21, max=153, avg=57.73, stdev=41.61 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 29], 40.00th=[ 33], 50.00th=[ 39], 60.00th=[ 47], | 70.00th=[ 77], 80.00th=[ 77], 90.00th=[ 96], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 8192, max= 8192, per=21.43%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=17.49%, ctx=74, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29730: Tue Apr 8 14:36:18 2025 read: IOPS=8, BW=8474KiB/s (8678kB/s)(6144KiB/725msec) clat (msec): min=15, max=132, avg=56.29, stdev=42.47 lat (msec): min=15, max=132, avg=56.30, stdev=42.47 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 44], | 70.00th=[ 78], 80.00th=[ 78], 90.00th=[ 133], 95.00th=[ 133], | 99.00th=[ 133], 99.50th=[ 133], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 4096, max= 4096, per=15.00%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8474KiB/s (8678kB/s)(6144KiB/725msec) clat (msec): min=28, max=105, avg=63.35, stdev=31.71 lat (msec): min=28, max=105, avg=63.42, stdev=31.71 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 30], | 30.00th=[ 30], 40.00th=[ 53], 50.00th=[ 53], 60.00th=[ 77], | 70.00th=[ 88], 80.00th=[ 88], 90.00th=[ 106], 95.00th=[ 106], | 99.00th=[ 106], 99.50th=[ 106], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 6144, max= 6144, per=16.07%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=33.33%, 250=16.67% cpu : usr=0.28%, sys=12.55%, ctx=53, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29731: Tue Apr 8 14:36:18 2025 read: IOPS=6, BW=6827KiB/s (6991kB/s)(5120KiB/750msec) clat (usec): min=18666, max=66579, avg=43286.92, stdev=22667.36 lat (usec): min=18667, max=66581, avg=43288.26, stdev=22667.58 clat percentiles (usec): | 1.00th=[18744], 5.00th=[18744], 10.00th=[18744], 20.00th=[18744], | 30.00th=[19792], 40.00th=[19792], 50.00th=[50594], 60.00th=[50594], | 70.00th=[60556], 80.00th=[60556], 90.00th=[66323], 95.00th=[66323], | 99.00th=[66323], 99.50th=[66323], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] write: IOPS=9, BW=9557KiB/s (9787kB/s)(7168KiB/750msec) clat (msec): min=20, max=121, avg=68.35, stdev=47.19 lat (msec): min=20, max=121, avg=68.48, stdev=47.27 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 24], | 30.00th=[ 25], 40.00th=[ 25], 50.00th=[ 61], 60.00th=[ 109], | 70.00th=[ 109], 80.00th=[ 120], 90.00th=[ 123], 95.00th=[ 123], | 99.00th=[ 123], 99.50th=[ 123], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 8192, max= 8192, per=21.43%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=25.00%, 100=33.33%, 250=25.00% cpu : usr=0.13%, sys=16.56%, ctx=161, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=26.7MiB/s (27.0MB/s), 2817KiB/s-9570KiB/s (2885kB/s-9800kB/s), io=20.0MiB (20.0MB), run=725-750msec WRITE: bw=37.3MiB/s (39.1MB/s), 6836KiB/s-13.8MiB/s (7000kB/s-14.4MB/s), io=28.0MiB (29.4MB), run=725-750msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29741: Tue Apr 8 14:36:20 2025 read: IOPS=5, BW=22.6MiB/s (23.7MB/s)(12.0MiB/532msec) clat (msec): min=131, max=185, avg=158.35, stdev=27.07 lat (msec): min=131, max=185, avg=158.35, stdev=27.07 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 132], 10.00th=[ 132], 20.00th=[ 132], | 30.00th=[ 132], 40.00th=[ 159], 50.00th=[ 159], 60.00th=[ 159], | 70.00th=[ 186], 80.00th=[ 186], 90.00th=[ 186], 95.00th=[ 186], | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min=16384, max=16384, per=89.10%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=36.91%, ctx=2280, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29742: Tue Apr 8 14:36:20 2025 read: IOPS=1, BW=4597KiB/s (4707kB/s)(4096KiB/891msec) clat (nsec): min=114972k, max=114972k, avg=114971948.00, stdev= 0.00 lat (nsec): min=114974k, max=114974k, avg=114973563.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 115], 10.00th=[ 115], 20.00th=[ 115], | 30.00th=[ 115], 40.00th=[ 115], 50.00th=[ 115], 60.00th=[ 115], | 70.00th=[ 115], 80.00th=[ 115], 90.00th=[ 115], 95.00th=[ 115], | 99.00th=[ 115], 99.50th=[ 115], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 8192, max= 8192, per=44.55%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9194KiB/s (9415kB/s)(8192KiB/891msec) clat (msec): min=161, max=547, avg=354.48, stdev=273.62 lat (msec): min=161, max=548, avg=354.69, stdev=273.73 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 550], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.11%, sys=28.09%, ctx=947, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29743: Tue Apr 8 14:36:20 2025 write: IOPS=3, BW=13.8MiB/s (14.5MB/s)(12.0MiB/869msec) clat (msec): min=122, max=508, avg=280.58, stdev=201.98 lat (msec): min=122, max=508, avg=280.86, stdev=201.98 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 123], 10.00th=[ 123], 20.00th=[ 123], | 30.00th=[ 123], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 211], | 70.00th=[ 510], 80.00th=[ 510], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min=16384, max=16384, per=44.55%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=33.87%, ctx=31, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29744: Tue Apr 8 14:36:20 2025 write: IOPS=4, BW=18.5MiB/s (19.4MB/s)(12.0MiB/649msec) clat (msec): min=113, max=298, avg=198.81, stdev=93.18 lat (msec): min=113, max=298, avg=199.12, stdev=93.18 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 114], 10.00th=[ 114], 20.00th=[ 114], | 30.00th=[ 114], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 184], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 8192, max= 8192, per=22.28%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=55.09%, ctx=41, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.0MiB/s (18.8MB/s), 4597KiB/s-22.6MiB/s (4707kB/s-23.7MB/s), io=16.0MiB (16.8MB), run=532-891msec WRITE: bw=35.9MiB/s (37.7MB/s), 9194KiB/s-18.5MiB/s (9415kB/s-19.4MB/s), io=32.0MiB (33.6MB), run=649-891msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29737: Tue Apr 8 14:36:20 2025 read: IOPS=4, BW=19.9MiB/s (20.8MB/s)(12.0MiB/604msec) clat (msec): min=45, max=409, avg=185.90, stdev=195.74 lat (msec): min=45, max=409, avg=185.91, stdev=195.74 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 46], | 30.00th=[ 46], 40.00th=[ 104], 50.00th=[ 104], 60.00th=[ 104], | 70.00th=[ 409], 80.00th=[ 409], 90.00th=[ 409], 95.00th=[ 409], | 99.00th=[ 409], 99.50th=[ 409], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min=16384, max=16384, per=82.40%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=11.77%, ctx=33, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29738: Tue Apr 8 14:36:20 2025 read: IOPS=1, BW=4971KiB/s (5090kB/s)(4096KiB/824msec) clat (nsec): min=90649k, max=90649k, avg=90648795.00, stdev= 0.00 lat (nsec): min=90650k, max=90650k, avg=90649710.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[90702], 5.00th=[90702], 10.00th=[90702], 20.00th=[90702], | 30.00th=[90702], 40.00th=[90702], 50.00th=[90702], 60.00th=[90702], | 70.00th=[90702], 80.00th=[90702], 90.00th=[90702], 95.00th=[90702], | 99.00th=[90702], 99.50th=[90702], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 8192, max= 8192, per=41.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9942KiB/s (10.2MB/s)(8192KiB/824msec) clat (msec): min=338, max=369, avg=354.01, stdev=21.90 lat (msec): min=338, max=369, avg=354.29, stdev=21.89 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 338], 10.00th=[ 338], 20.00th=[ 338], | 30.00th=[ 338], 40.00th=[ 338], 50.00th=[ 338], 60.00th=[ 372], | 70.00th=[ 372], 80.00th=[ 372], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 8192, max= 8192, per=25.33%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.12%, sys=15.19%, ctx=74, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29739: Tue Apr 8 14:36:20 2025 write: IOPS=2, BW=11.8MiB/s (12.4MB/s)(12.0MiB/1013msec) clat (msec): min=229, max=469, avg=335.28, stdev=122.74 lat (msec): min=229, max=470, avg=335.63, stdev=122.74 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 230], 10.00th=[ 230], 20.00th=[ 230], | 30.00th=[ 230], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 472], 80.00th=[ 472], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 8175, max= 8175, per=25.27%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=13.04%, ctx=40, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29740: Tue Apr 8 14:36:20 2025 write: IOPS=5, BW=21.5MiB/s (22.6MB/s)(12.0MiB/558msec) clat (msec): min=47, max=449, avg=183.36, stdev=230.18 lat (msec): min=47, max=449, avg=183.65, stdev=230.15 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 54], 50.00th=[ 54], 60.00th=[ 54], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min=16384, max=16384, per=50.65%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=33.33%, 500=33.33% cpu : usr=0.18%, sys=20.83%, ctx=60, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.4MiB/s (20.4MB/s), 4971KiB/s-19.9MiB/s (5090kB/s-20.8MB/s), io=16.0MiB (16.8MB), run=604-824msec WRITE: bw=31.6MiB/s (33.1MB/s), 9942KiB/s-21.5MiB/s (10.2MB/s-22.6MB/s), io=32.0MiB (33.6MB), run=558-1013msec