== sanity test 398b: DIO and buffer IO race ============== 10:02:34 (1743516154) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.11087 s, 45.3 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9218: Tue Apr 1 10:02:49 2025 read: IOPS=113, BW=454KiB/s (465kB/s)(5992KiB/13194msec) clat (usec): min=56, max=162652, avg=4031.93, stdev=5181.28 lat (usec): min=56, max=162653, avg=4033.04, stdev=5181.07 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 80], 10.00th=[ 102], 20.00th=[ 2089], | 30.00th=[ 2507], 40.00th=[ 2900], 50.00th=[ 3294], 60.00th=[ 3818], | 70.00th=[ 4817], 80.00th=[ 6063], 90.00th=[ 7832], 95.00th=[ 9241], | 99.00th=[ 13304], 99.50th=[ 14353], 99.90th=[ 50594], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 327, max= 640, per=25.04%, avg=450.92, stdev=83.62, samples=26 iops : min= 81, max= 160, avg=112.62, stdev=20.97, samples=26 write: IOPS=119, BW=477KiB/s (489kB/s)(6296KiB/13194msec) clat (usec): min=427, max=41717, avg=4490.54, stdev=4310.85 lat (usec): min=427, max=41719, avg=4491.30, stdev=4310.84 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 570], 10.00th=[ 619], 20.00th=[ 783], | 30.00th=[ 1045], 40.00th=[ 2278], 50.00th=[ 3589], 60.00th=[ 4555], | 70.00th=[ 5735], 80.00th=[ 7242], 90.00th=[10159], 95.00th=[12518], | 99.00th=[19530], 99.50th=[23987], 99.90th=[30016], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 320, max= 632, per=25.10%, avg=473.35, stdev=74.85, samples=26 iops : min= 80, max= 158, avg=118.23, stdev=18.75, samples=26 lat (usec) : 100=4.59%, 250=3.39%, 500=0.39%, 750=9.05%, 1000=5.63% lat (msec) : 2=5.66%, 4=29.17%, 10=35.22%, 20=6.25%, 50=0.59% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.21%, sys=23.87%, ctx=5945, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9219: Tue Apr 1 10:02:49 2025 read: IOPS=111, BW=446KiB/s (457kB/s)(5940KiB/13308msec) clat (usec): min=54, max=38094, avg=3986.18, stdev=3138.14 lat (usec): min=54, max=38095, avg=3987.11, stdev=3138.08 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 84], 10.00th=[ 111], 20.00th=[ 2147], | 30.00th=[ 2606], 40.00th=[ 3032], 50.00th=[ 3359], 60.00th=[ 3884], | 70.00th=[ 4817], 80.00th=[ 5997], 90.00th=[ 7635], 95.00th=[ 9110], | 99.00th=[13042], 99.50th=[16581], 99.90th=[33424], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 320, max= 600, per=24.54%, avg=442.00, stdev=76.51, samples=26 iops : min= 80, max= 150, avg=110.42, stdev=19.13, samples=26 write: IOPS=119, BW=477KiB/s (488kB/s)(6348KiB/13308msec) clat (usec): min=480, max=161423, avg=4602.79, stdev=5838.49 lat (usec): min=481, max=161424, avg=4604.14, stdev=5839.49 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 570], 10.00th=[ 635], 20.00th=[ 799], | 30.00th=[ 1037], 40.00th=[ 2180], 50.00th=[ 3654], 60.00th=[ 4555], | 70.00th=[ 5866], 80.00th=[ 7504], 90.00th=[ 10159], 95.00th=[ 12125], | 99.00th=[ 18482], 99.50th=[ 22676], 99.90th=[ 36963], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 320, max= 696, per=24.76%, avg=466.92, stdev=85.35, samples=26 iops : min= 80, max= 174, avg=116.65, stdev=21.37, samples=26 lat (usec) : 100=4.10%, 250=3.29%, 500=0.29%, 750=8.89%, 1000=6.28% lat (msec) : 2=5.99%, 4=29.17%, 10=34.93%, 20=6.51%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=23.69%, ctx=5810, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9220: Tue Apr 1 10:02:49 2025 read: IOPS=113, BW=454KiB/s (465kB/s)(6028KiB/13282msec) clat (usec): min=55, max=22805, avg=3865.94, stdev=2881.95 lat (usec): min=55, max=22806, avg=3866.71, stdev=2882.18 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 82], 10.00th=[ 104], 20.00th=[ 2114], | 30.00th=[ 2540], 40.00th=[ 2868], 50.00th=[ 3261], 60.00th=[ 3884], | 70.00th=[ 4817], 80.00th=[ 5932], 90.00th=[ 7767], 95.00th=[ 9110], | 99.00th=[11994], 99.50th=[15270], 99.90th=[19530], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 216, max= 600, per=24.95%, avg=449.42, stdev=83.53, samples=26 iops : min= 54, max= 150, avg=112.27, stdev=20.90, samples=26 write: IOPS=117, BW=471KiB/s (483kB/s)(6260KiB/13282msec) clat (usec): min=475, max=191607, avg=4708.19, stdev=6660.37 lat (usec): min=476, max=191608, avg=4710.62, stdev=6659.50 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 562], 10.00th=[ 627], 20.00th=[ 799], | 30.00th=[ 1057], 40.00th=[ 2057], 50.00th=[ 3556], 60.00th=[ 4555], | 70.00th=[ 5735], 80.00th=[ 7504], 90.00th=[ 10552], 95.00th=[ 12649], | 99.00th=[ 21627], 99.50th=[ 27657], 99.90th=[ 52691], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 312, max= 576, per=24.66%, avg=465.12, stdev=57.73, samples=26 iops : min= 78, max= 144, avg=116.19, stdev=14.51, samples=26 lat (usec) : 100=4.49%, 250=3.81%, 500=0.23%, 750=8.82%, 1000=5.79% lat (msec) : 2=6.32%, 4=28.52%, 10=34.83%, 20=6.54%, 50=0.59% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.20%, sys=23.55%, ctx=5887, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9221: Tue Apr 1 10:02:49 2025 read: IOPS=113, BW=454KiB/s (465kB/s)(6052KiB/13327msec) clat (usec): min=53, max=165068, avg=4037.33, stdev=5137.05 lat (usec): min=53, max=165069, avg=4038.05, stdev=5137.03 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 83], 10.00th=[ 105], 20.00th=[ 2114], | 30.00th=[ 2507], 40.00th=[ 2868], 50.00th=[ 3228], 60.00th=[ 3851], | 70.00th=[ 4883], 80.00th=[ 6325], 90.00th=[ 7635], 95.00th=[ 9110], | 99.00th=[ 14746], 99.50th=[ 17171], 99.90th=[ 25822], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 280, max= 640, per=24.71%, avg=445.12, stdev=85.43, samples=26 iops : min= 70, max= 160, avg=111.19, stdev=21.37, samples=26 write: IOPS=116, BW=468KiB/s (479kB/s)(6236KiB/13327msec) clat (usec): min=463, max=51536, avg=4576.07, stdev=4469.20 lat (usec): min=464, max=51537, avg=4576.96, stdev=4469.13 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 570], 10.00th=[ 627], 20.00th=[ 783], | 30.00th=[ 1037], 40.00th=[ 2474], 50.00th=[ 3720], 60.00th=[ 4621], | 70.00th=[ 5735], 80.00th=[ 7504], 90.00th=[10290], 95.00th=[12649], | 99.00th=[17433], 99.50th=[24249], 99.90th=[40633], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 328, max= 608, per=24.40%, avg=460.12, stdev=76.93, samples=26 iops : min= 82, max= 152, avg=114.96, stdev=19.24, samples=26 lat (usec) : 100=4.43%, 250=3.65%, 500=0.26%, 750=9.18%, 1000=5.89% lat (msec) : 2=4.82%, 4=29.04%, 10=35.87%, 20=6.28%, 50=0.52% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.18%, sys=23.74%, ctx=6129, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1802KiB/s (1845kB/s), 446KiB/s-454KiB/s (457kB/s-465kB/s), io=23.4MiB (24.6MB), run=13194-13327msec WRITE: bw=1886KiB/s (1932kB/s), 468KiB/s-477KiB/s (479kB/s-489kB/s), io=24.6MiB (25.7MB), run=13194-13327msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9223: Tue Apr 1 10:03:10 2025 read: IOPS=45, BW=180KiB/s (185kB/s)(5992KiB/33231msec) clat (usec): min=1578, max=166353, avg=3890.09, stdev=5237.42 lat (usec): min=1579, max=166354, avg=3890.81, stdev=5237.43 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1844], 10.00th=[ 1926], 20.00th=[ 2024], | 30.00th=[ 2147], 40.00th=[ 2245], 50.00th=[ 2474], 60.00th=[ 2966], | 70.00th=[ 3752], 80.00th=[ 4948], 90.00th=[ 7439], 95.00th=[ 9765], | 99.00th=[ 15795], 99.50th=[ 21365], 99.90th=[ 45351], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 56, max= 344, per=25.17%, avg=179.97, stdev=69.08, samples=66 iops : min= 14, max= 86, avg=44.97, stdev=17.29, samples=66 write: IOPS=47, BW=189KiB/s (194kB/s)(6296KiB/33231msec) clat (usec): min=8406, max=57434, avg=17375.99, stdev=6549.37 lat (usec): min=8407, max=57434, avg=17376.99, stdev=6549.23 clat percentiles (usec): | 1.00th=[ 9241], 5.00th=[10290], 10.00th=[10814], 20.00th=[11863], | 30.00th=[13304], 40.00th=[14615], 50.00th=[15926], 60.00th=[17171], | 70.00th=[18744], 80.00th=[21365], 90.00th=[27132], 95.00th=[31327], | 99.00th=[36963], 99.50th=[39060], 99.90th=[50594], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 72, max= 272, per=25.24%, avg=189.06, stdev=58.18, samples=66 iops : min= 18, max= 68, avg=47.24, stdev=14.58, samples=66 lat (msec) : 2=8.24%, 4=27.60%, 10=12.70%, 20=39.29%, 50=12.08% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.10%, sys=6.15%, ctx=3709, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9224: Tue Apr 1 10:03:10 2025 read: IOPS=44, BW=177KiB/s (181kB/s)(5940KiB/33549msec) clat (usec): min=1584, max=164538, avg=3960.00, stdev=5351.86 lat (usec): min=1584, max=164539, avg=3960.78, stdev=5351.94 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1860], 10.00th=[ 1942], 20.00th=[ 2057], | 30.00th=[ 2180], 40.00th=[ 2311], 50.00th=[ 2606], 60.00th=[ 3261], | 70.00th=[ 3752], 80.00th=[ 4686], 90.00th=[ 7308], 95.00th=[ 9634], | 99.00th=[ 15926], 99.50th=[ 23462], 99.90th=[ 44827], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 48, max= 328, per=24.81%, avg=177.39, stdev=74.50, samples=66 iops : min= 12, max= 82, avg=44.30, stdev=18.68, samples=66 write: IOPS=47, BW=189KiB/s (194kB/s)(6348KiB/33549msec) clat (usec): min=8392, max=71304, avg=17400.80, stdev=6806.97 lat (usec): min=8393, max=71304, avg=17401.72, stdev=6807.01 clat percentiles (usec): | 1.00th=[ 8848], 5.00th=[10028], 10.00th=[10814], 20.00th=[11994], | 30.00th=[13042], 40.00th=[14615], 50.00th=[15926], 60.00th=[17171], | 70.00th=[18744], 80.00th=[21627], 90.00th=[27132], 95.00th=[31589], | 99.00th=[38536], 99.50th=[44827], 99.90th=[57934], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 96, max= 256, per=25.08%, avg=187.82, stdev=55.98, samples=66 iops : min= 24, max= 64, avg=46.91, stdev=14.05, samples=66 lat (msec) : 2=7.32%, 4=28.16%, 10=13.25%, 20=38.41%, 50=12.70% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.13%, sys=6.05%, ctx=3698, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9225: Tue Apr 1 10:03:10 2025 read: IOPS=45, BW=181KiB/s (186kB/s)(6028KiB/33252msec) clat (usec): min=1652, max=45218, avg=3968.55, stdev=3441.09 lat (usec): min=1653, max=45218, avg=3969.30, stdev=3441.08 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1844], 10.00th=[ 1926], 20.00th=[ 2040], | 30.00th=[ 2147], 40.00th=[ 2311], 50.00th=[ 2638], 60.00th=[ 3359], | 70.00th=[ 3851], 80.00th=[ 5276], 90.00th=[ 7832], 95.00th=[ 9896], | 99.00th=[15401], 99.50th=[23200], 99.90th=[39584], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 64, max= 368, per=25.32%, avg=181.06, stdev=67.76, samples=66 iops : min= 16, max= 92, avg=45.15, stdev=16.92, samples=66 write: IOPS=47, BW=188KiB/s (193kB/s)(6260KiB/33252msec) clat (msec): min=7, max=190, avg=17.39, stdev= 7.91 lat (msec): min=7, max=190, avg=17.39, stdev= 7.91 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 22], 90.00th=[ 27], 95.00th=[ 31], | 99.00th=[ 40], 99.50th=[ 47], 99.90th=[ 64], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 79, max= 264, per=25.06%, avg=187.73, stdev=59.47, samples=66 iops : min= 19, max= 66, avg=46.82, stdev=14.84, samples=66 lat (msec) : 2=8.04%, 4=27.21%, 10=13.93%, 20=38.48%, 50=12.17% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.10%, sys=6.21%, ctx=3728, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9226: Tue Apr 1 10:03:10 2025 read: IOPS=45, BW=183KiB/s (187kB/s)(6052KiB/33098msec) clat (usec): min=1547, max=37176, avg=3800.50, stdev=3124.57 lat (usec): min=1547, max=37177, avg=3801.26, stdev=3124.55 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1860], 10.00th=[ 1926], 20.00th=[ 2040], | 30.00th=[ 2147], 40.00th=[ 2245], 50.00th=[ 2474], 60.00th=[ 3097], | 70.00th=[ 3785], 80.00th=[ 4948], 90.00th=[ 7504], 95.00th=[ 9765], | 99.00th=[15664], 99.50th=[20055], 99.90th=[32900], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 64, max= 400, per=25.55%, avg=182.70, stdev=73.69, samples=66 iops : min= 16, max= 100, avg=45.61, stdev=18.43, samples=66 write: IOPS=47, BW=188KiB/s (193kB/s)(6236KiB/33098msec) clat (msec): min=8, max=152, avg=17.49, stdev= 7.55 lat (msec): min=8, max=152, avg=17.49, stdev= 7.55 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 15], 50.00th=[ 16], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 22], 90.00th=[ 27], 95.00th=[ 31], | 99.00th=[ 40], 99.50th=[ 47], 99.90th=[ 64], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 88, max= 264, per=25.10%, avg=188.03, stdev=57.36, samples=66 iops : min= 22, max= 66, avg=46.94, stdev=14.38, samples=66 lat (msec) : 2=8.14%, 4=27.86%, 10=13.12%, 20=38.25%, 50=12.47% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.10%, sys=6.19%, ctx=3713, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=716KiB/s (733kB/s), 177KiB/s-183KiB/s (181kB/s-187kB/s), io=23.4MiB (24.6MB), run=33098-33549msec WRITE: bw=749KiB/s (767kB/s), 188KiB/s-189KiB/s (193kB/s-194kB/s), io=24.6MiB (25.7MB), run=33098-33549msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9269: Tue Apr 1 10:03:14 2025 read: IOPS=122, BW=1962KiB/s (2009kB/s)(5920KiB/3018msec) clat (usec): min=59, max=17751, avg=3424.69, stdev=2848.71 lat (usec): min=60, max=17753, avg=3425.51, stdev=2848.71 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 75], 10.00th=[ 89], 20.00th=[ 119], | 30.00th=[ 2278], 40.00th=[ 2835], 50.00th=[ 3163], 60.00th=[ 3687], | 70.00th=[ 4490], 80.00th=[ 5669], 90.00th=[ 7242], 95.00th=[ 8586], | 99.00th=[11076], 99.50th=[11731], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 512, max= 2400, per=24.70%, avg=1960.33, stdev=713.89, samples=6 iops : min= 32, max= 150, avg=122.33, stdev=44.55, samples=6 write: IOPS=131, BW=2110KiB/s (2161kB/s)(6368KiB/3018msec) clat (usec): min=691, max=41780, avg=4321.12, stdev=5151.42 lat (usec): min=692, max=41782, avg=4323.32, stdev=5151.00 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 816], 10.00th=[ 914], 20.00th=[ 1074], | 30.00th=[ 1287], 40.00th=[ 1549], 50.00th=[ 2343], 60.00th=[ 3523], | 70.00th=[ 4817], 80.00th=[ 6718], 90.00th=[ 9241], 95.00th=[14091], | 99.00th=[26608], 99.50th=[35390], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 544, max= 2996, per=25.26%, avg=2109.17, stdev=821.08, samples=6 iops : min= 34, max= 187, avg=131.67, stdev=51.24, samples=6 lat (usec) : 100=6.38%, 250=6.77%, 500=0.65%, 750=0.65%, 1000=7.55% lat (msec) : 2=16.54%, 4=26.04%, 10=29.95%, 20=4.30%, 50=1.17% cpu : usr=0.13%, sys=27.68%, ctx=1582, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9271: Tue Apr 1 10:03:14 2025 read: IOPS=127, BW=2042KiB/s (2091kB/s)(5744KiB/2813msec) clat (usec): min=60, max=31563, avg=3630.98, stdev=3037.05 lat (usec): min=61, max=31563, avg=3631.66, stdev=3037.15 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 83], 10.00th=[ 100], 20.00th=[ 184], | 30.00th=[ 2540], 40.00th=[ 2999], 50.00th=[ 3392], 60.00th=[ 3916], | 70.00th=[ 4555], 80.00th=[ 5538], 90.00th=[ 6718], 95.00th=[ 8094], | 99.00th=[13960], 99.50th=[15008], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 1760, max= 2517, per=26.02%, avg=2065.00, stdev=293.02, samples=5 iops : min= 110, max= 157, avg=129.00, stdev=18.19, samples=5 write: IOPS=145, BW=2326KiB/s (2382kB/s)(6544KiB/2813msec) clat (usec): min=681, max=21106, avg=3529.51, stdev=3329.86 lat (usec): min=682, max=21107, avg=3531.42, stdev=3331.84 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 816], 10.00th=[ 889], 20.00th=[ 1004], | 30.00th=[ 1172], 40.00th=[ 1336], 50.00th=[ 1860], 60.00th=[ 3261], | 70.00th=[ 4490], 80.00th=[ 6063], 90.00th=[ 8586], 95.00th=[ 9896], | 99.00th=[12780], 99.50th=[19268], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 2048, max= 2592, per=27.96%, avg=2334.00, stdev=206.58, samples=5 iops : min= 128, max= 162, avg=145.80, stdev=12.93, samples=5 lat (usec) : 100=4.69%, 250=5.34%, 500=0.52%, 750=0.39%, 1000=10.29% lat (msec) : 2=16.41%, 4=26.82%, 10=32.42%, 20=2.73%, 50=0.39% cpu : usr=0.11%, sys=29.59%, ctx=1556, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9273: Tue Apr 1 10:03:14 2025 read: IOPS=136, BW=2178KiB/s (2230kB/s)(6256KiB/2873msec) clat (usec): min=59, max=32972, avg=3754.00, stdev=3495.76 lat (usec): min=59, max=32972, avg=3754.63, stdev=3495.78 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 74], 10.00th=[ 83], 20.00th=[ 116], | 30.00th=[ 2376], 40.00th=[ 2933], 50.00th=[ 3425], 60.00th=[ 4047], | 70.00th=[ 4752], 80.00th=[ 5669], 90.00th=[ 7046], 95.00th=[ 8979], | 99.00th=[16450], 99.50th=[23987], 99.90th=[32900], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 1824, max= 2522, per=28.04%, avg=2225.00, stdev=279.75, samples=5 iops : min= 114, max= 157, avg=138.80, stdev=17.20, samples=5 write: IOPS=131, BW=2100KiB/s (2150kB/s)(6032KiB/2873msec) clat (usec): min=703, max=24490, avg=3676.41, stdev=3376.04 lat (usec): min=704, max=24491, avg=3677.51, stdev=3375.97 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 832], 10.00th=[ 889], 20.00th=[ 1074], | 30.00th=[ 1205], 40.00th=[ 1500], 50.00th=[ 2212], 60.00th=[ 3458], | 70.00th=[ 4686], 80.00th=[ 6718], 90.00th=[ 8225], 95.00th=[ 9765], | 99.00th=[15008], 99.50th=[21890], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 1820, max= 2496, per=25.73%, avg=2148.60, stdev=247.03, samples=5 iops : min= 113, max= 156, avg=134.00, stdev=15.76, samples=5 lat (usec) : 100=7.68%, 250=4.95%, 500=0.13%, 750=0.78%, 1000=7.42% lat (msec) : 2=15.49%, 4=26.04%, 10=33.72%, 20=3.26%, 50=0.52% cpu : usr=0.24%, sys=28.73%, ctx=1660, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9275: Tue Apr 1 10:03:14 2025 read: IOPS=128, BW=2050KiB/s (2100kB/s)(6032KiB/2942msec) clat (usec): min=61, max=25470, avg=3470.29, stdev=3553.54 lat (usec): min=61, max=25470, avg=3471.31, stdev=3553.22 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 76], 10.00th=[ 86], 20.00th=[ 115], | 30.00th=[ 223], 40.00th=[ 2704], 50.00th=[ 3163], 60.00th=[ 3687], | 70.00th=[ 4555], 80.00th=[ 5538], 90.00th=[ 6915], 95.00th=[ 8455], | 99.00th=[20579], 99.50th=[22938], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 1216, max= 2272, per=25.00%, avg=1984.00, stdev=438.18, samples=5 iops : min= 76, max= 142, avg=124.00, stdev=27.39, samples=5 write: IOPS=132, BW=2126KiB/s (2177kB/s)(6256KiB/2942msec) clat (usec): min=661, max=39446, avg=3957.40, stdev=4743.20 lat (usec): min=662, max=39450, avg=3958.55, stdev=4743.17 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 816], 10.00th=[ 889], 20.00th=[ 1020], | 30.00th=[ 1221], 40.00th=[ 1418], 50.00th=[ 1893], 60.00th=[ 3097], | 70.00th=[ 4293], 80.00th=[ 5866], 90.00th=[ 8717], 95.00th=[12387], | 99.00th=[24249], 99.50th=[32375], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 1568, max= 2528, per=25.14%, avg=2099.20, stdev=365.98, samples=5 iops : min= 98, max= 158, avg=131.20, stdev=22.87, samples=5 lat (usec) : 100=7.68%, 250=7.16%, 500=0.39%, 750=1.04%, 1000=8.46% lat (msec) : 2=17.32%, 4=23.44%, 10=28.91%, 20=3.91%, 50=1.69% cpu : usr=0.27%, sys=26.56%, ctx=1570, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7936KiB/s (8127kB/s), 1962KiB/s-2178KiB/s (2009kB/s-2230kB/s), io=23.4MiB (24.5MB), run=2813-3018msec WRITE: bw=8350KiB/s (8550kB/s), 2100KiB/s-2326KiB/s (2150kB/s-2382kB/s), io=24.6MiB (25.8MB), run=2813-3018msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9268: Tue Apr 1 10:03:19 2025 read: IOPS=45, BW=725KiB/s (742kB/s)(5920KiB/8166msec) clat (usec): min=1870, max=42509, avg=3835.67, stdev=3948.27 lat (usec): min=1871, max=42509, avg=3836.57, stdev=3948.20 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2008], 10.00th=[ 2089], 20.00th=[ 2180], | 30.00th=[ 2278], 40.00th=[ 2343], 50.00th=[ 2442], 60.00th=[ 2638], | 70.00th=[ 3064], 80.00th=[ 4047], 90.00th=[ 6587], 95.00th=[12387], | 99.00th=[19006], 99.50th=[29492], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 448, max= 1248, per=24.75%, avg=713.88, stdev=245.67, samples=16 iops : min= 28, max= 78, avg=44.56, stdev=15.40, samples=16 write: IOPS=48, BW=780KiB/s (799kB/s)(6368KiB/8166msec) clat (usec): min=9148, max=57017, avg=16924.13, stdev=6196.64 lat (usec): min=9149, max=57018, avg=16925.44, stdev=6196.68 clat percentiles (usec): | 1.00th=[ 9634], 5.00th=[10552], 10.00th=[10945], 20.00th=[12125], | 30.00th=[13173], 40.00th=[14484], 50.00th=[15533], 60.00th=[16712], | 70.00th=[17957], 80.00th=[19530], 90.00th=[26084], 95.00th=[30802], | 99.00th=[36439], 99.50th=[39584], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 384, max= 992, per=25.50%, avg=773.94, stdev=227.94, samples=16 iops : min= 24, max= 62, avg=48.31, stdev=14.34, samples=16 lat (msec) : 2=2.21%, 4=36.20%, 10=7.55%, 20=44.40%, 50=9.51% lat (msec) : 100=0.13% cpu : usr=0.11%, sys=7.04%, ctx=903, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9270: Tue Apr 1 10:03:19 2025 read: IOPS=43, BW=692KiB/s (708kB/s)(5744KiB/8303msec) clat (usec): min=1822, max=29444, avg=3642.56, stdev=3131.98 lat (usec): min=1823, max=29445, avg=3643.33, stdev=3131.97 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2024], 10.00th=[ 2089], 20.00th=[ 2147], | 30.00th=[ 2245], 40.00th=[ 2343], 50.00th=[ 2442], 60.00th=[ 2671], | 70.00th=[ 3032], 80.00th=[ 3818], 90.00th=[ 7242], 95.00th=[10028], | 99.00th=[17695], 99.50th=[19530], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 255, max= 1152, per=23.91%, avg=689.62, stdev=249.11, samples=16 iops : min= 15, max= 72, avg=42.94, stdev=15.63, samples=16 write: IOPS=49, BW=788KiB/s (807kB/s)(6544KiB/8303msec) clat (usec): min=9090, max=56019, avg=16895.66, stdev=6081.62 lat (usec): min=9091, max=56020, avg=16897.07, stdev=6081.63 clat percentiles (usec): | 1.00th=[ 9765], 5.00th=[10290], 10.00th=[11076], 20.00th=[11863], | 30.00th=[13173], 40.00th=[14877], 50.00th=[15926], 60.00th=[16909], | 70.00th=[17957], 80.00th=[19792], 90.00th=[24511], 95.00th=[29230], | 99.00th=[38011], 99.50th=[39060], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 415, max= 1024, per=25.43%, avg=771.69, stdev=213.62, samples=16 iops : min= 25, max= 64, avg=48.06, stdev=13.46, samples=16 lat (msec) : 2=1.43%, 4=36.46%, 10=7.55%, 20=43.88%, 50=10.55% lat (msec) : 100=0.13% cpu : usr=0.12%, sys=6.89%, ctx=898, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9272: Tue Apr 1 10:03:19 2025 read: IOPS=48, BW=782KiB/s (800kB/s)(6256KiB/8004msec) clat (usec): min=1878, max=20596, avg=3641.94, stdev=2898.82 lat (usec): min=1879, max=20597, avg=3642.74, stdev=2898.81 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2008], 10.00th=[ 2089], 20.00th=[ 2212], | 30.00th=[ 2311], 40.00th=[ 2376], 50.00th=[ 2507], 60.00th=[ 2737], | 70.00th=[ 3097], 80.00th=[ 3949], 90.00th=[ 6718], 95.00th=[ 9896], | 99.00th=[17433], 99.50th=[18744], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 352, max= 1376, per=27.12%, avg=782.00, stdev=337.44, samples=16 iops : min= 22, max= 86, avg=48.88, stdev=21.09, samples=16 write: IOPS=47, BW=754KiB/s (772kB/s)(6032KiB/8004msec) clat (usec): min=8503, max=56013, avg=17278.30, stdev=6235.47 lat (usec): min=8504, max=56015, avg=17279.77, stdev=6235.60 clat percentiles (usec): | 1.00th=[ 9634], 5.00th=[10552], 10.00th=[11076], 20.00th=[12256], | 30.00th=[13698], 40.00th=[15008], 50.00th=[16057], 60.00th=[17433], | 70.00th=[18744], 80.00th=[20317], 90.00th=[24511], 95.00th=[30802], | 99.00th=[39060], 99.50th=[44827], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 480, max= 992, per=24.78%, avg=752.00, stdev=197.26, samples=16 iops : min= 30, max= 62, avg=47.00, stdev=12.33, samples=16 lat (msec) : 2=1.69%, 4=39.06%, 10=8.98%, 20=39.84%, 50=10.29% lat (msec) : 100=0.13% cpu : usr=0.14%, sys=7.33%, ctx=916, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9274: Tue Apr 1 10:03:19 2025 read: IOPS=45, BW=730KiB/s (747kB/s)(6032KiB/8268msec) clat (usec): min=1917, max=20565, avg=3671.62, stdev=3065.46 lat (usec): min=1918, max=20566, avg=3672.34, stdev=3065.46 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2040], 10.00th=[ 2114], 20.00th=[ 2180], | 30.00th=[ 2278], 40.00th=[ 2409], 50.00th=[ 2507], 60.00th=[ 2671], | 70.00th=[ 2999], 80.00th=[ 3752], 90.00th=[ 6718], 95.00th=[10552], | 99.00th=[18220], 99.50th=[18482], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 416, max= 1184, per=24.82%, avg=715.69, stdev=242.30, samples=16 iops : min= 26, max= 74, avg=44.62, stdev=15.13, samples=16 write: IOPS=47, BW=757KiB/s (775kB/s)(6256KiB/8268msec) clat (usec): min=9397, max=47590, avg=17401.26, stdev=6160.65 lat (usec): min=9398, max=47592, avg=17402.66, stdev=6160.74 clat percentiles (usec): | 1.00th=[ 9765], 5.00th=[10552], 10.00th=[11207], 20.00th=[12387], | 30.00th=[13566], 40.00th=[15270], 50.00th=[16319], 60.00th=[17433], | 70.00th=[18482], 80.00th=[20841], 90.00th=[25297], 95.00th=[29492], | 99.00th=[44303], 99.50th=[47449], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 414, max= 960, per=24.64%, avg=747.75, stdev=203.43, samples=16 iops : min= 25, max= 60, avg=46.63, stdev=12.75, samples=16 lat (msec) : 2=1.56%, 4=38.28%, 10=7.16%, 20=40.89%, 50=12.11% cpu : usr=0.04%, sys=7.17%, ctx=912, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2885KiB/s (2954kB/s), 692KiB/s-782KiB/s (708kB/s-800kB/s), io=23.4MiB (24.5MB), run=8004-8303msec WRITE: bw=3035KiB/s (3108kB/s), 754KiB/s-788KiB/s (772kB/s-807kB/s), io=24.6MiB (25.8MB), run=8004-8303msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9289: Tue Apr 1 10:03:21 2025 read: IOPS=7, BW=7226KiB/s (7399kB/s)(7168KiB/992msec) clat (usec): min=481, max=443656, avg=81961.84, stdev=160230.32 lat (usec): min=482, max=443657, avg=81963.07, stdev=160230.34 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 482], 10.00th=[ 482], 20.00th=[ 611], | 30.00th=[ 26608], 40.00th=[ 26608], 50.00th=[ 29492], 60.00th=[ 35914], | 70.00th=[ 35914], 80.00th=[ 36963], 90.00th=[442500], 95.00th=[442500], | 99.00th=[442500], 99.50th=[442500], 99.90th=[442500], 99.95th=[442500], | 99.99th=[442500] bw ( KiB/s): min= 4096, max= 4096, per=33.02%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5161KiB/s (5285kB/s)(5120KiB/992msec) clat (msec): min=39, max=128, avg=81.48, stdev=32.84 lat (msec): min=39, max=129, avg=81.52, stdev=32.83 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 40], | 30.00th=[ 66], 40.00th=[ 66], 50.00th=[ 85], 60.00th=[ 85], | 70.00th=[ 89], 80.00th=[ 89], 90.00th=[ 129], 95.00th=[ 129], | 99.00th=[ 129], 99.50th=[ 129], 99.90th=[ 129], 99.95th=[ 129], | 99.99th=[ 129] lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=41.67%, 100=25.00%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=24.72%, ctx=1092, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9291: Tue Apr 1 10:03:21 2025 read: IOPS=1, BW=1882KiB/s (1928kB/s)(2048KiB/1088msec) clat (usec): min=410, max=565693, avg=283051.95, stdev=399715.16 lat (usec): min=412, max=565694, avg=283053.22, stdev=399714.96 clat percentiles (usec): | 1.00th=[ 412], 5.00th=[ 412], 10.00th=[ 412], 20.00th=[ 412], | 30.00th=[ 412], 40.00th=[ 412], 50.00th=[ 412], 60.00th=[566232], | 70.00th=[566232], 80.00th=[566232], 90.00th=[566232], 95.00th=[566232], | 99.00th=[566232], 99.50th=[566232], 99.90th=[566232], 99.95th=[566232], | 99.99th=[566232] bw ( KiB/s): min= 4096, max= 4096, per=33.02%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9412KiB/s (9638kB/s)(10.0MiB/1088msec) clat (msec): min=25, max=121, avg=50.91, stdev=28.20 lat (msec): min=25, max=121, avg=50.97, stdev=28.20 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 27], | 30.00th=[ 29], 40.00th=[ 39], 50.00th=[ 40], 60.00th=[ 54], | 70.00th=[ 59], 80.00th=[ 59], 90.00th=[ 62], 95.00th=[ 122], | 99.00th=[ 122], 99.50th=[ 122], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min=14336, max=14336, per=82.55%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=41.67%, 100=33.33%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=24.56%, ctx=370, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9293: Tue Apr 1 10:03:21 2025 read: IOPS=3, BW=3721KiB/s (3811kB/s)(6144KiB/1651msec) clat (usec): min=442, max=459307, avg=96291.40, stdev=180297.27 lat (usec): min=443, max=459308, avg=96292.43, stdev=180297.47 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 445], 10.00th=[ 445], 20.00th=[ 545], | 30.00th=[ 545], 40.00th=[ 611], 50.00th=[ 611], 60.00th=[ 44827], | 70.00th=[ 71828], 80.00th=[ 71828], 90.00th=[459277], 95.00th=[459277], | 99.00th=[459277], 99.50th=[459277], 99.90th=[459277], 99.95th=[459277], | 99.99th=[459277] bw ( KiB/s): min= 4096, max= 6144, per=41.28%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=3, BW=3721KiB/s (3811kB/s)(6144KiB/1651msec) clat (msec): min=46, max=612, avg=177.06, stdev=216.42 lat (msec): min=46, max=612, avg=177.10, stdev=216.44 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 61], | 30.00th=[ 61], 40.00th=[ 92], 50.00th=[ 92], 60.00th=[ 100], | 70.00th=[ 153], 80.00th=[ 153], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 2048, max= 6144, per=23.59%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (usec) : 500=8.33%, 750=16.67% lat (msec) : 50=16.67%, 100=33.33%, 250=8.33%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=10.79%, ctx=270, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9295: Tue Apr 1 10:03:21 2025 read: IOPS=3, BW=3249KiB/s (3327kB/s)(5120KiB/1576msec) clat (usec): min=411, max=471425, avg=116354.21, stdev=199213.46 lat (usec): min=412, max=471426, avg=116355.03, stdev=199213.34 clat percentiles (usec): | 1.00th=[ 412], 5.00th=[ 412], 10.00th=[ 412], 20.00th=[ 412], | 30.00th=[ 30278], 40.00th=[ 30278], 50.00th=[ 32900], 60.00th=[ 32900], | 70.00th=[ 46924], 80.00th=[ 46924], 90.00th=[471860], 95.00th=[471860], | 99.00th=[471860], 99.50th=[471860], 99.90th=[471860], 99.95th=[471860], | 99.99th=[471860] bw ( KiB/s): min= 8175, max= 8175, per=65.91%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4548KiB/s (4657kB/s)(7168KiB/1576msec) clat (msec): min=27, max=603, avg=136.97, stdev=209.08 lat (msec): min=27, max=603, avg=137.03, stdev=209.07 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 34], | 30.00th=[ 38], 40.00th=[ 38], 50.00th=[ 46], 60.00th=[ 78], | 70.00th=[ 78], 80.00th=[ 136], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 6144, max= 8175, per=41.23%, avg=7159.50, stdev=1436.13, samples=2 iops : min= 6, max= 7, avg= 6.50, stdev= 0.71, samples=2 lat (usec) : 500=8.33% lat (msec) : 50=58.33%, 100=8.33%, 250=8.33%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=16.95%, ctx=751, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.1MiB/s (12.7MB/s), 1882KiB/s-7226KiB/s (1928kB/s-7399kB/s), io=20.0MiB (20.0MB), run=992-1651msec WRITE: bw=16.0MiB/s (17.8MB/s), 3721KiB/s-9412KiB/s (3811kB/s-9638kB/s), io=28.0MiB (29.4MB), run=992-1651msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9290: Tue Apr 1 10:03:21 2025 read: IOPS=6, BW=6377KiB/s (6530kB/s)(7168KiB/1124msec) clat (msec): min=14, max=451, avg=90.41, stdev=159.64 lat (msec): min=14, max=451, avg=90.41, stdev=159.64 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 24], | 30.00th=[ 29], 40.00th=[ 29], 50.00th=[ 33], 60.00th=[ 39], | 70.00th=[ 39], 80.00th=[ 45], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 2048, max=10219, per=50.98%, avg=6133.50, stdev=5777.77, samples=2 iops : min= 2, max= 9, avg= 5.50, stdev= 4.95, samples=2 write: IOPS=4, BW=4555KiB/s (4664kB/s)(5120KiB/1124msec) clat (msec): min=39, max=154, avg=91.65, stdev=45.86 lat (msec): min=39, max=154, avg=91.72, stdev=45.86 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 40], | 30.00th=[ 56], 40.00th=[ 56], 50.00th=[ 104], 60.00th=[ 104], | 70.00th=[ 106], 80.00th=[ 106], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 8175, max= 8175, per=48.53%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=8.33%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=8.82%, ctx=162, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9292: Tue Apr 1 10:03:21 2025 read: IOPS=1, BW=1242KiB/s (1272kB/s)(2048KiB/1649msec) clat (msec): min=36, max=537, avg=287.00, stdev=353.98 lat (msec): min=36, max=537, avg=287.00, stdev=353.98 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 37], 60.00th=[ 542], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 2039, max= 2043, per=16.96%, avg=2041.00, stdev= 2.83, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=6, BW=6210KiB/s (6359kB/s)(10.0MiB/1649msec) clat (msec): min=25, max=508, avg=102.02, stdev=146.42 lat (msec): min=25, max=508, avg=102.09, stdev=146.42 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 26], | 30.00th=[ 33], 40.00th=[ 40], 50.00th=[ 41], 60.00th=[ 59], | 70.00th=[ 103], 80.00th=[ 103], 90.00th=[ 124], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 6131, max=10199, per=48.47%, avg=8165.00, stdev=2876.51, samples=2 iops : min= 5, max= 9, avg= 7.00, stdev= 2.83, samples=2 lat (msec) : 50=50.00%, 100=16.67%, 250=16.67%, 750=16.67% cpu : usr=0.06%, sys=8.25%, ctx=224, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9294: Tue Apr 1 10:03:21 2025 read: IOPS=3, BW=3672KiB/s (3761kB/s)(6144KiB/1673msec) clat (msec): min=21, max=506, avg=115.61, stdev=191.87 lat (msec): min=21, max=506, avg=115.61, stdev=191.87 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 37], 50.00th=[ 37], 60.00th=[ 38], | 70.00th=[ 65], 80.00th=[ 65], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 2048, max= 6144, per=34.04%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=3, BW=3672KiB/s (3761kB/s)(6144KiB/1673msec) clat (msec): min=27, max=608, avg=161.19, stdev=222.10 lat (msec): min=27, max=608, avg=161.25, stdev=222.10 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 90], 50.00th=[ 90], 60.00th=[ 99], | 70.00th=[ 111], 80.00th=[ 111], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 2048, max= 6144, per=20.26%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 lat (msec) : 50=50.00%, 100=25.00%, 250=8.33%, 750=16.67% cpu : usr=0.00%, sys=5.20%, ctx=54, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9296: Tue Apr 1 10:03:21 2025 read: IOPS=2, BW=3008KiB/s (3080kB/s)(5120KiB/1702msec) clat (msec): min=13, max=124, avg=54.94, stdev=49.55 lat (msec): min=13, max=124, avg=54.94, stdev=49.55 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 14], | 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 26], 60.00th=[ 26], | 70.00th=[ 90], 80.00th=[ 90], 90.00th=[ 125], 95.00th=[ 125], | 99.00th=[ 125], 99.50th=[ 125], 99.90th=[ 125], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 4096, max= 4096, per=34.04%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4212KiB/s (4313kB/s)(7168KiB/1702msec) clat (msec): min=22, max=586, avg=199.77, stdev=263.73 lat (msec): min=22, max=586, avg=199.85, stdev=263.74 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 29], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 45], 60.00th=[ 90], | 70.00th=[ 90], 80.00th=[ 584], 90.00th=[ 584], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 8192, max= 8192, per=48.63%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=16.67%, 250=8.33%, 750=16.67% cpu : usr=0.12%, sys=5.76%, ctx=44, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.8MiB/s (12.3MB/s), 1242KiB/s-6377KiB/s (1272kB/s-6530kB/s), io=20.0MiB (20.0MB), run=1124-1702msec WRITE: bw=16.5MiB/s (17.2MB/s), 3672KiB/s-6210KiB/s (3761kB/s-6359kB/s), io=28.0MiB (29.4MB), run=1124-1702msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9307: Tue Apr 1 10:03:23 2025 read: IOPS=10, BW=42.3MiB/s (44.3MB/s)(12.0MiB/284msec) clat (usec): min=1531, max=177260, avg=89096.92, stdev=87866.24 lat (usec): min=1531, max=177262, avg=89098.08, stdev=87866.74 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1532], 10.00th=[ 1532], 20.00th=[ 1532], | 30.00th=[ 1532], 40.00th=[ 88605], 50.00th=[ 88605], 60.00th=[ 88605], | 70.00th=[177210], 80.00th=[177210], 90.00th=[177210], 95.00th=[177210], | 99.00th=[177210], 99.50th=[177210], 99.90th=[177210], 99.95th=[177210], | 99.99th=[177210] lat (msec) : 2=33.33%, 100=33.33%, 250=33.33% cpu : usr=0.35%, sys=37.81%, ctx=1158, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9308: Tue Apr 1 10:03:23 2025 read: IOPS=1, BW=5178KiB/s (5303kB/s)(4096KiB/791msec) clat (nsec): min=129043k, max=129043k, avg=129043131.00, stdev= 0.00 lat (nsec): min=129045k, max=129045k, avg=129044869.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 130], 10.00th=[ 130], 20.00th=[ 130], | 30.00th=[ 130], 40.00th=[ 130], 50.00th=[ 130], 60.00th=[ 130], | 70.00th=[ 130], 80.00th=[ 130], 90.00th=[ 130], 95.00th=[ 130], | 99.00th=[ 130], 99.50th=[ 130], 99.90th=[ 130], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 8159, max= 8159, per=39.39%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.1MiB/s (10.6MB/s)(8192KiB/791msec) clat (msec): min=235, max=411, avg=323.87, stdev=124.62 lat (msec): min=235, max=412, avg=324.05, stdev=124.54 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 236], 10.00th=[ 236], 20.00th=[ 236], | 30.00th=[ 236], 40.00th=[ 236], 50.00th=[ 236], 60.00th=[ 414], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 8159, max= 8159, per=19.70%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=24.30%, ctx=879, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9309: Tue Apr 1 10:03:23 2025 write: IOPS=5, BW=21.9MiB/s (22.9MB/s)(12.0MiB/549msec) clat (msec): min=72, max=262, avg=177.10, stdev=96.83 lat (msec): min=72, max=263, avg=177.38, stdev=96.84 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 72], 10.00th=[ 72], 20.00th=[ 72], | 30.00th=[ 72], 40.00th=[ 197], 50.00th=[ 197], 60.00th=[ 197], | 70.00th=[ 264], 80.00th=[ 264], 90.00th=[ 264], 95.00th=[ 264], | 99.00th=[ 264], 99.50th=[ 264], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min=16384, max=16384, per=39.55%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=46.53%, ctx=69, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9310: Tue Apr 1 10:03:23 2025 write: IOPS=4, BW=16.6MiB/s (17.4MB/s)(12.0MiB/722msec) clat (msec): min=104, max=340, avg=235.00, stdev=120.20 lat (msec): min=104, max=340, avg=235.28, stdev=120.21 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 105], 10.00th=[ 105], 20.00th=[ 105], | 30.00th=[ 105], 40.00th=[ 262], 50.00th=[ 262], 60.00th=[ 262], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 342], 95.00th=[ 342], | 99.00th=[ 342], 99.50th=[ 342], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 8192, max= 8192, per=19.78%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=35.92%, ctx=101, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=20.2MiB/s (21.2MB/s), 5178KiB/s-42.3MiB/s (5303kB/s-44.3MB/s), io=16.0MiB (16.8MB), run=284-791msec WRITE: bw=40.5MiB/s (42.4MB/s), 10.1MiB/s-21.9MiB/s (10.6MB/s-22.9MB/s), io=32.0MiB (33.6MB), run=549-791msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9302: Tue Apr 1 10:03:23 2025 read: IOPS=4, BW=19.8MiB/s (20.8MB/s)(12.0MiB/605msec) clat (msec): min=79, max=271, avg=194.80, stdev=101.60 lat (msec): min=79, max=271, avg=194.81, stdev=101.60 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 80], 10.00th=[ 80], 20.00th=[ 80], | 30.00th=[ 80], 40.00th=[ 232], 50.00th=[ 232], 60.00th=[ 232], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 8000, max= 8000, per=29.54%, avg=8000.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=11.42%, ctx=180, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9304: Tue Apr 1 10:03:23 2025 read: IOPS=2, BW=9330KiB/s (9554kB/s)(4096KiB/439msec) clat (nsec): min=52110k, max=52110k, avg=52110125.00, stdev= 0.00 lat (nsec): min=52113k, max=52113k, avg=52112793.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[52167], 5.00th=[52167], 10.00th=[52167], 20.00th=[52167], | 30.00th=[52167], 40.00th=[52167], 50.00th=[52167], 60.00th=[52167], | 70.00th=[52167], 80.00th=[52167], 90.00th=[52167], 95.00th=[52167], | 99.00th=[52167], 99.50th=[52167], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] write: IOPS=4, BW=18.2MiB/s (19.1MB/s)(8192KiB/439msec) clat (msec): min=154, max=206, avg=180.78, stdev=36.96 lat (msec): min=154, max=207, avg=181.00, stdev=36.87 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 207], | 70.00th=[ 207], 80.00th=[ 207], 90.00th=[ 207], 95.00th=[ 207], | 99.00th=[ 207], 99.50th=[ 207], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.46%, sys=20.78%, ctx=63, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9305: Tue Apr 1 10:03:23 2025 write: IOPS=3, BW=13.7MiB/s (14.3MB/s)(12.0MiB/879msec) clat (msec): min=55, max=519, avg=285.72, stdev=231.62 lat (msec): min=56, max=519, avg=286.01, stdev=231.62 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 56], 10.00th=[ 56], 20.00th=[ 56], | 30.00th=[ 56], 40.00th=[ 284], 50.00th=[ 284], 60.00th=[ 284], | 70.00th=[ 518], 80.00th=[ 518], 90.00th=[ 518], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 8094, max= 8094, per=21.71%, avg=8094.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.23%, sys=9.91%, ctx=83, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9306: Tue Apr 1 10:03:23 2025 write: IOPS=6, BW=27.6MiB/s (28.9MB/s)(12.0MiB/435msec) clat (msec): min=88, max=211, avg=136.28, stdev=66.02 lat (msec): min=88, max=211, avg=136.52, stdev=65.94 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 88], 10.00th=[ 88], 20.00th=[ 88], | 30.00th=[ 88], 40.00th=[ 110], 50.00th=[ 110], 60.00th=[ 110], | 70.00th=[ 211], 80.00th=[ 211], 90.00th=[ 211], 95.00th=[ 211], | 99.00th=[ 211], 99.50th=[ 211], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=15.90%, ctx=57, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=26.4MiB/s (27.7MB/s), 9330KiB/s-19.8MiB/s (9554kB/s-20.8MB/s), io=16.0MiB (16.8MB), run=439-605msec WRITE: bw=36.4MiB/s (38.2MB/s), 13.7MiB/s-27.6MiB/s (14.3MB/s-28.9MB/s), io=32.0MiB (33.6MB), run=435-879msec