== sanity test 398b: DIO and buffer IO race ============== 16:04:22 (1734123862) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.11538 s, 45.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20935: Fri Dec 13 16:04:36 2024 read: IOPS=130, BW=521KiB/s (534kB/s)(5992KiB/11494msec) clat (usec): min=48, max=34705, avg=4185.40, stdev=2467.40 lat (usec): min=49, max=34706, avg=4186.62, stdev=2466.98 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 1811], 10.00th=[ 2089], 20.00th=[ 2442], | 30.00th=[ 2802], 40.00th=[ 3261], 50.00th=[ 3752], 60.00th=[ 4359], | 70.00th=[ 4948], 80.00th=[ 5669], 90.00th=[ 6849], 95.00th=[ 8029], | 99.00th=[12256], 99.50th=[13960], 99.90th=[30802], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 255, max= 608, per=25.02%, avg=509.91, stdev=72.53, samples=22 iops : min= 63, max= 152, avg=127.36, stdev=18.30, samples=22 write: IOPS=136, BW=548KiB/s (561kB/s)(6296KiB/11494msec) clat (usec): min=412, max=135264, avg=3266.65, stdev=4328.56 lat (usec): min=413, max=135265, avg=3268.51, stdev=4328.66 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 519], 10.00th=[ 537], 20.00th=[ 594], | 30.00th=[ 783], 40.00th=[ 2008], 50.00th=[ 2638], 60.00th=[ 3195], | 70.00th=[ 4080], 80.00th=[ 5407], 90.00th=[ 7177], 95.00th=[ 8455], | 99.00th=[ 11994], 99.50th=[ 13566], 99.90th=[ 22152], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 303, max= 752, per=25.15%, avg=536.45, stdev=86.37, samples=22 iops : min= 75, max= 188, avg=134.00, stdev=21.72, samples=22 lat (usec) : 50=0.07%, 100=1.92%, 250=0.16%, 500=1.04%, 750=13.90% lat (usec) : 1000=2.12% lat (msec) : 2=5.08%, 4=37.76%, 10=35.48%, 20=2.34%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.11%, sys=23.27%, ctx=6129, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20936: Fri Dec 13 16:04:36 2024 read: IOPS=127, BW=509KiB/s (521kB/s)(5940KiB/11676msec) clat (usec): min=50, max=16608, avg=4239.48, stdev=2273.36 lat (usec): min=51, max=16609, avg=4240.08, stdev=2273.34 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 1811], 10.00th=[ 2114], 20.00th=[ 2474], | 30.00th=[ 2868], 40.00th=[ 3294], 50.00th=[ 3818], 60.00th=[ 4424], | 70.00th=[ 5080], 80.00th=[ 5800], 90.00th=[ 7177], 95.00th=[ 8455], | 99.00th=[11600], 99.50th=[12518], 99.90th=[16450], 99.95th=[16581], | 99.99th=[16581] bw ( KiB/s): min= 288, max= 640, per=24.79%, avg=505.22, stdev=64.76, samples=23 iops : min= 72, max= 160, avg=126.26, stdev=16.18, samples=23 write: IOPS=135, BW=544KiB/s (557kB/s)(6348KiB/11676msec) clat (usec): min=433, max=166886, avg=3338.78, stdev=5043.84 lat (usec): min=433, max=166887, avg=3339.45, stdev=5043.85 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 519], 10.00th=[ 537], 20.00th=[ 603], | 30.00th=[ 775], 40.00th=[ 1893], 50.00th=[ 2606], 60.00th=[ 3228], | 70.00th=[ 4178], 80.00th=[ 5407], 90.00th=[ 7242], 95.00th=[ 8586], | 99.00th=[ 12911], 99.50th=[ 14877], 99.90th=[ 30802], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 352, max= 632, per=25.02%, avg=533.78, stdev=71.36, samples=23 iops : min= 88, max= 158, avg=133.39, stdev=17.88, samples=23 lat (usec) : 100=1.89%, 250=0.13%, 500=0.91%, 750=14.23%, 1000=2.28% lat (msec) : 2=4.98%, 4=36.59%, 10=36.59%, 20=2.34%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=22.90%, ctx=6031, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20937: Fri Dec 13 16:04:36 2024 read: IOPS=127, BW=512KiB/s (524kB/s)(6028KiB/11782msec) clat (usec): min=52, max=138375, avg=4400.72, stdev=4264.33 lat (usec): min=52, max=138376, avg=4401.78, stdev=4264.12 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 1713], 10.00th=[ 2073], 20.00th=[ 2442], | 30.00th=[ 2868], 40.00th=[ 3359], 50.00th=[ 3982], 60.00th=[ 4555], | 70.00th=[ 5145], 80.00th=[ 5866], 90.00th=[ 7177], 95.00th=[ 8586], | 99.00th=[ 11863], 99.50th=[ 15533], 99.90th=[ 36963], 99.95th=[137364], | 99.99th=[137364] bw ( KiB/s): min= 272, max= 624, per=24.81%, avg=505.61, stdev=74.35, samples=23 iops : min= 68, max= 156, avg=126.35, stdev=18.60, samples=23 write: IOPS=132, BW=531KiB/s (544kB/s)(6260KiB/11782msec) clat (usec): min=408, max=20189, avg=3238.64, stdev=2889.02 lat (usec): min=408, max=20189, avg=3240.06, stdev=2890.32 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 519], 10.00th=[ 545], 20.00th=[ 594], | 30.00th=[ 807], 40.00th=[ 2040], 50.00th=[ 2606], 60.00th=[ 3163], | 70.00th=[ 4047], 80.00th=[ 5407], 90.00th=[ 7373], 95.00th=[ 8848], | 99.00th=[12387], 99.50th=[13435], 99.90th=[19792], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 336, max= 672, per=24.45%, avg=521.57, stdev=77.72, samples=23 iops : min= 84, max= 168, avg=130.35, stdev=19.41, samples=23 lat (usec) : 100=2.02%, 250=0.16%, 500=1.17%, 750=13.09%, 1000=2.73% lat (msec) : 2=5.08%, 4=35.90%, 10=37.14%, 20=2.60%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=23.16%, ctx=6130, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20938: Fri Dec 13 16:04:36 2024 read: IOPS=131, BW=527KiB/s (539kB/s)(6052KiB/11491msec) clat (usec): min=55, max=172334, avg=4341.53, stdev=4842.60 lat (usec): min=56, max=172335, avg=4342.14, stdev=4842.60 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 1876], 10.00th=[ 2147], 20.00th=[ 2507], | 30.00th=[ 2868], 40.00th=[ 3326], 50.00th=[ 3818], 60.00th=[ 4490], | 70.00th=[ 5080], 80.00th=[ 5866], 90.00th=[ 6783], 95.00th=[ 7898], | 99.00th=[ 11600], 99.50th=[ 14091], 99.90th=[ 20841], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 279, max= 632, per=25.38%, avg=517.23, stdev=82.78, samples=22 iops : min= 69, max= 158, avg=129.18, stdev=20.81, samples=22 write: IOPS=135, BW=543KiB/s (556kB/s)(6236KiB/11491msec) clat (usec): min=412, max=22864, avg=3106.16, stdev=2752.93 lat (usec): min=413, max=22865, avg=3107.11, stdev=2752.68 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 515], 10.00th=[ 537], 20.00th=[ 578], | 30.00th=[ 709], 40.00th=[ 1844], 50.00th=[ 2573], 60.00th=[ 3064], | 70.00th=[ 4080], 80.00th=[ 5276], 90.00th=[ 6915], 95.00th=[ 8586], | 99.00th=[11600], 99.50th=[12911], 99.90th=[14746], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 343, max= 704, per=24.89%, avg=530.95, stdev=78.09, samples=22 iops : min= 85, max= 176, avg=132.64, stdev=19.59, samples=22 lat (usec) : 100=1.40%, 250=0.16%, 500=1.04%, 750=14.81%, 1000=2.02% lat (msec) : 2=4.92%, 4=37.01%, 10=36.39%, 20=2.15%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=23.17%, ctx=5982, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2038KiB/s (2087kB/s), 509KiB/s-527KiB/s (521kB/s-539kB/s), io=23.4MiB (24.6MB), run=11491-11782msec WRITE: bw=2134KiB/s (2185kB/s), 531KiB/s-548KiB/s (544kB/s-561kB/s), io=24.6MiB (25.7MB), run=11491-11782msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20939: Fri Dec 13 16:04:42 2024 read: IOPS=83, BW=333KiB/s (341kB/s)(5992KiB/18009msec) clat (usec): min=1568, max=187815, avg=4917.37, stdev=5525.64 lat (usec): min=1568, max=187816, avg=4918.13, stdev=5525.66 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1975], 10.00th=[ 2180], 20.00th=[ 2540], | 30.00th=[ 2999], 40.00th=[ 3589], 50.00th=[ 3949], 60.00th=[ 4293], | 70.00th=[ 5145], 80.00th=[ 6783], 90.00th=[ 9241], 95.00th=[ 11076], | 99.00th=[ 14746], 99.50th=[ 16319], 99.90th=[ 18744], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 136, max= 576, per=24.70%, avg=329.31, stdev=82.61, samples=35 iops : min= 34, max= 144, avg=82.29, stdev=20.65, samples=35 write: IOPS=87, BW=350KiB/s (358kB/s)(6296KiB/18009msec) clat (usec): min=2148, max=24964, avg=6704.69, stdev=3012.34 lat (usec): min=2166, max=24965, avg=6705.79, stdev=3012.29 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 3097], 10.00th=[ 3458], 20.00th=[ 4047], | 30.00th=[ 4817], 40.00th=[ 5538], 50.00th=[ 6259], 60.00th=[ 6783], | 70.00th=[ 7504], 80.00th=[ 8848], 90.00th=[10683], 95.00th=[12125], | 99.00th=[16909], 99.50th=[18482], 99.90th=[21627], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 160, max= 624, per=24.87%, avg=346.91, stdev=76.61, samples=35 iops : min= 40, max= 156, avg=86.69, stdev=19.17, samples=35 lat (msec) : 2=2.73%, 4=32.49%, 10=54.43%, 20=10.22%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=11.42%, ctx=3680, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20940: Fri Dec 13 16:04:42 2024 read: IOPS=83, BW=333KiB/s (341kB/s)(5940KiB/17851msec) clat (usec): min=1534, max=171680, avg=4748.77, stdev=5183.07 lat (usec): min=1534, max=171682, avg=4749.65, stdev=5183.08 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1991], 10.00th=[ 2147], 20.00th=[ 2507], | 30.00th=[ 2933], 40.00th=[ 3425], 50.00th=[ 3851], 60.00th=[ 4178], | 70.00th=[ 4686], 80.00th=[ 6128], 90.00th=[ 8586], 95.00th=[ 10683], | 99.00th=[ 15926], 99.50th=[ 17171], 99.90th=[ 21890], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 120, max= 632, per=24.87%, avg=331.49, stdev=90.40, samples=35 iops : min= 30, max= 158, avg=82.74, stdev=22.61, samples=35 write: IOPS=88, BW=356KiB/s (364kB/s)(6348KiB/17851msec) clat (usec): min=2148, max=37267, avg=6758.91, stdev=3186.96 lat (usec): min=2149, max=37268, avg=6759.83, stdev=3186.97 clat percentiles (usec): | 1.00th=[ 2507], 5.00th=[ 3097], 10.00th=[ 3458], 20.00th=[ 4047], | 30.00th=[ 4883], 40.00th=[ 5669], 50.00th=[ 6259], 60.00th=[ 6718], | 70.00th=[ 7504], 80.00th=[ 8848], 90.00th=[10814], 95.00th=[12518], | 99.00th=[17171], 99.50th=[20055], 99.90th=[27657], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 200, max= 632, per=25.12%, avg=350.46, stdev=73.95, samples=35 iops : min= 50, max= 158, avg=87.49, stdev=18.53, samples=35 lat (msec) : 2=2.64%, 4=34.02%, 10=52.96%, 20=10.03%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=11.34%, ctx=3633, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20941: Fri Dec 13 16:04:42 2024 read: IOPS=84, BW=340KiB/s (348kB/s)(6028KiB/17741msec) clat (usec): min=1584, max=26620, avg=4786.98, stdev=2909.92 lat (usec): min=1584, max=26620, avg=4787.83, stdev=2909.89 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1991], 10.00th=[ 2180], 20.00th=[ 2507], | 30.00th=[ 2999], 40.00th=[ 3589], 50.00th=[ 3916], 60.00th=[ 4293], | 70.00th=[ 5014], 80.00th=[ 6521], 90.00th=[ 8979], 95.00th=[11207], | 99.00th=[13960], 99.50th=[15926], 99.90th=[22152], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 168, max= 592, per=25.33%, avg=337.71, stdev=74.15, samples=35 iops : min= 42, max= 148, avg=84.34, stdev=18.57, samples=35 write: IOPS=88, BW=353KiB/s (361kB/s)(6260KiB/17741msec) clat (msec): min=2, max=182, avg= 6.69, stdev= 5.37 lat (msec): min=2, max=182, avg= 6.69, stdev= 5.37 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 12], | 99.00th=[ 18], 99.50th=[ 21], 99.90th=[ 28], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 216, max= 696, per=25.13%, avg=350.51, stdev=88.09, samples=35 iops : min= 54, max= 174, avg=87.54, stdev=22.02, samples=35 lat (msec) : 2=2.57%, 4=33.46%, 10=54.17%, 20=9.44%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=11.74%, ctx=3688, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20942: Fri Dec 13 16:04:42 2024 read: IOPS=85, BW=340KiB/s (349kB/s)(6052KiB/17778msec) clat (usec): min=1576, max=19282, avg=4721.76, stdev=2820.19 lat (usec): min=1576, max=19283, avg=4722.59, stdev=2820.20 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1958], 10.00th=[ 2147], 20.00th=[ 2507], | 30.00th=[ 2900], 40.00th=[ 3556], 50.00th=[ 3884], 60.00th=[ 4178], | 70.00th=[ 4948], 80.00th=[ 6456], 90.00th=[ 8979], 95.00th=[10814], | 99.00th=[14222], 99.50th=[15139], 99.90th=[17695], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 176, max= 600, per=25.29%, avg=337.14, stdev=78.55, samples=35 iops : min= 44, max= 150, avg=84.11, stdev=19.63, samples=35 write: IOPS=87, BW=351KiB/s (359kB/s)(6236KiB/17778msec) clat (msec): min=2, max=149, avg= 6.76, stdev= 4.81 lat (msec): min=2, max=149, avg= 6.76, stdev= 4.81 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 18], 99.50th=[ 21], 99.90th=[ 31], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 192, max= 632, per=24.87%, avg=346.97, stdev=76.65, samples=35 iops : min= 48, max= 158, avg=86.57, stdev=19.21, samples=35 lat (msec) : 2=2.93%, 4=34.96%, 10=51.69%, 20=10.06%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=11.46%, ctx=3612, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1333KiB/s (1365kB/s), 333KiB/s-340KiB/s (341kB/s-349kB/s), io=23.4MiB (24.6MB), run=17741-18009msec WRITE: bw=1396KiB/s (1429kB/s), 350KiB/s-356KiB/s (358kB/s-364kB/s), io=24.6MiB (25.7MB), run=17741-18009msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20975: Fri Dec 13 16:04:50 2024 read: IOPS=62, BW=995KiB/s (1019kB/s)(5920KiB/5947msec) clat (usec): min=57, max=237990, avg=11065.67, stdev=12585.55 lat (usec): min=58, max=237991, avg=11066.27, stdev=12585.60 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 6718], 10.00th=[ 7504], 20.00th=[ 8160], | 30.00th=[ 8979], 40.00th=[ 9503], 50.00th=[ 9896], 60.00th=[ 10552], | 70.00th=[ 11469], 80.00th=[ 12125], 90.00th=[ 14222], 95.00th=[ 16319], | 99.00th=[ 25822], 99.50th=[ 53740], 99.90th=[238027], 99.95th=[238027], | 99.99th=[238027] bw ( KiB/s): min= 319, max= 1248, per=26.04%, avg=980.27, stdev=259.54, samples=11 iops : min= 19, max= 78, avg=61.18, stdev=16.46, samples=11 write: IOPS=66, BW=1071KiB/s (1096kB/s)(6368KiB/5947msec) clat (usec): min=626, max=83988, avg=4608.56, stdev=6186.40 lat (usec): min=627, max=83989, avg=4609.57, stdev=6186.38 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 791], 10.00th=[ 816], 20.00th=[ 1172], | 30.00th=[ 2040], 40.00th=[ 2409], 50.00th=[ 2900], 60.00th=[ 3818], | 70.00th=[ 4817], 80.00th=[ 5997], 90.00th=[ 9110], 95.00th=[13829], | 99.00th=[23462], 99.50th=[47973], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 223, max= 1408, per=26.88%, avg=1064.64, stdev=318.05, samples=11 iops : min= 13, max= 88, avg=66.45, stdev=20.13, samples=11 lat (usec) : 100=1.17%, 250=0.39%, 500=0.13%, 750=0.26%, 1000=9.11% lat (msec) : 2=5.60%, 4=17.32%, 10=38.15%, 20=25.26%, 50=2.21% lat (msec) : 100=0.26%, 250=0.13% cpu : usr=0.08%, sys=21.95%, ctx=2744, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20976: Fri Dec 13 16:04:50 2024 read: IOPS=59, BW=948KiB/s (971kB/s)(5744KiB/6058msec) clat (usec): min=60, max=31693, avg=10415.98, stdev=3967.14 lat (usec): min=61, max=31694, avg=10416.57, stdev=3967.18 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 6325], 10.00th=[ 7111], 20.00th=[ 8094], | 30.00th=[ 8586], 40.00th=[ 9372], 50.00th=[ 9896], 60.00th=[10552], | 70.00th=[11338], 80.00th=[12518], 90.00th=[14615], 95.00th=[17695], | 99.00th=[23200], 99.50th=[31065], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 32, max= 1184, per=25.07%, avg=943.67, stdev=313.52, samples=12 iops : min= 2, max= 74, avg=58.83, stdev=19.66, samples=12 write: IOPS=67, BW=1080KiB/s (1106kB/s)(6544KiB/6058msec) clat (usec): min=593, max=313305, avg=5481.29, stdev=16419.98 lat (usec): min=594, max=313308, avg=5482.32, stdev=16420.06 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 783], 10.00th=[ 816], 20.00th=[ 1156], | 30.00th=[ 2040], 40.00th=[ 2474], 50.00th=[ 3032], 60.00th=[ 3949], | 70.00th=[ 4752], 80.00th=[ 6063], 90.00th=[ 11469], 95.00th=[ 14615], | 99.00th=[ 28705], 99.50th=[ 42730], 99.90th=[312476], 99.95th=[312476], | 99.99th=[312476] bw ( KiB/s): min= 160, max= 1696, per=27.19%, avg=1077.00, stdev=402.94, samples=12 iops : min= 10, max= 106, avg=67.17, stdev=25.26, samples=12 lat (usec) : 100=1.04%, 250=0.26%, 500=0.13%, 750=0.78%, 1000=8.33% lat (msec) : 2=6.64%, 4=16.54%, 10=37.24%, 20=26.69%, 50=2.08% lat (msec) : 100=0.13%, 500=0.13% cpu : usr=0.12%, sys=17.34%, ctx=2615, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20977: Fri Dec 13 16:04:50 2024 read: IOPS=61, BW=983KiB/s (1007kB/s)(6256KiB/6362msec) clat (usec): min=62, max=47000, avg=10907.90, stdev=4174.53 lat (usec): min=63, max=47001, avg=10908.55, stdev=4174.55 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 6849], 10.00th=[ 7701], 20.00th=[ 8586], | 30.00th=[ 9110], 40.00th=[ 9765], 50.00th=[10290], 60.00th=[10814], | 70.00th=[11469], 80.00th=[12387], 90.00th=[14877], 95.00th=[17957], | 99.00th=[27395], 99.50th=[32637], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 128, max= 1216, per=26.27%, avg=988.92, stdev=308.69, samples=12 iops : min= 8, max= 76, avg=61.67, stdev=19.35, samples=12 write: IOPS=59, BW=948KiB/s (971kB/s)(6032KiB/6362msec) clat (usec): min=737, max=304159, avg=5336.42, stdev=16161.39 lat (usec): min=738, max=304161, avg=5337.62, stdev=16161.40 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 791], 10.00th=[ 824], 20.00th=[ 1303], | 30.00th=[ 2147], 40.00th=[ 2474], 50.00th=[ 2769], 60.00th=[ 3785], | 70.00th=[ 4686], 80.00th=[ 6259], 90.00th=[ 10945], 95.00th=[ 14091], | 99.00th=[ 24511], 99.50th=[ 39584], 99.90th=[304088], 99.95th=[304088], | 99.99th=[304088] bw ( KiB/s): min= 160, max= 1280, per=23.49%, avg=930.25, stdev=291.40, samples=12 iops : min= 10, max= 80, avg=58.00, stdev=18.18, samples=12 lat (usec) : 100=0.52%, 500=0.39%, 750=0.13%, 1000=8.72% lat (msec) : 2=4.04%, 4=17.45%, 10=34.90%, 20=31.25%, 50=2.47% lat (msec) : 500=0.13% cpu : usr=0.11%, sys=17.97%, ctx=2855, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20978: Fri Dec 13 16:04:50 2024 read: IOPS=59, BW=949KiB/s (972kB/s)(6032KiB/6353msec) clat (usec): min=65, max=31530, avg=10611.90, stdev=4016.29 lat (usec): min=65, max=31531, avg=10616.81, stdev=4015.29 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 6849], 10.00th=[ 7439], 20.00th=[ 7963], | 30.00th=[ 8586], 40.00th=[ 9372], 50.00th=[10028], 60.00th=[10683], | 70.00th=[11469], 80.00th=[12649], 90.00th=[14746], 95.00th=[17171], | 99.00th=[28181], 99.50th=[30540], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 766, max= 1216, per=27.49%, avg=1034.91, stdev=136.82, samples=11 iops : min= 47, max= 76, avg=64.45, stdev= 8.65, samples=11 write: IOPS=61, BW=985KiB/s (1008kB/s)(6256KiB/6353msec) clat (usec): min=724, max=304261, avg=5774.55, stdev=16461.37 lat (usec): min=725, max=304263, avg=5775.50, stdev=16461.42 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 791], 10.00th=[ 824], 20.00th=[ 1303], | 30.00th=[ 2147], 40.00th=[ 2409], 50.00th=[ 2900], 60.00th=[ 3752], | 70.00th=[ 5211], 80.00th=[ 6783], 90.00th=[ 11600], 95.00th=[ 15008], | 99.00th=[ 36963], 99.50th=[ 72877], 99.90th=[304088], 99.95th=[304088], | 99.99th=[304088] bw ( KiB/s): min= 96, max= 1344, per=25.09%, avg=994.00, stdev=329.01, samples=12 iops : min= 6, max= 84, avg=61.92, stdev=20.59, samples=12 lat (usec) : 100=0.91%, 750=0.65%, 1000=8.20% lat (msec) : 2=3.78%, 4=18.88%, 10=36.20%, 20=28.78%, 50=2.34% lat (msec) : 100=0.13%, 500=0.13% cpu : usr=0.09%, sys=18.21%, ctx=2725, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3765KiB/s (3855kB/s), 948KiB/s-995KiB/s (971kB/s-1019kB/s), io=23.4MiB (24.5MB), run=5947-6362msec WRITE: bw=3961KiB/s (4056kB/s), 948KiB/s-1080KiB/s (971kB/s-1106kB/s), io=24.6MiB (25.8MB), run=5947-6362msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20971: Fri Dec 13 16:04:50 2024 read: IOPS=61, BW=991KiB/s (1014kB/s)(5920KiB/5976msec) clat (usec): min=1699, max=326159, avg=6942.28, stdev=17471.27 lat (usec): min=1700, max=326161, avg=6942.87, stdev=17471.34 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1975], 10.00th=[ 2089], 20.00th=[ 2409], | 30.00th=[ 2737], 40.00th=[ 3228], 50.00th=[ 3818], 60.00th=[ 5145], | 70.00th=[ 6915], 80.00th=[ 9372], 90.00th=[ 11994], 95.00th=[ 16188], | 99.00th=[ 26608], 99.50th=[ 44827], 99.90th=[325059], 99.95th=[325059], | 99.99th=[325059] bw ( KiB/s): min= 96, max= 1280, per=26.13%, avg=971.36, stdev=314.49, samples=11 iops : min= 6, max= 80, avg=60.64, stdev=19.63, samples=11 write: IOPS=66, BW=1066KiB/s (1091kB/s)(6368KiB/5976msec) clat (usec): min=2576, max=51495, avg=8344.37, stdev=5413.74 lat (usec): min=2577, max=51497, avg=8345.59, stdev=5413.73 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3163], 10.00th=[ 3425], 20.00th=[ 4359], | 30.00th=[ 5211], 40.00th=[ 5932], 50.00th=[ 6980], 60.00th=[ 8225], | 70.00th=[ 9241], 80.00th=[11076], 90.00th=[15139], 95.00th=[17957], | 99.00th=[23200], 99.50th=[49546], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 32, max= 1472, per=27.14%, avg=1061.64, stdev=382.12, samples=11 iops : min= 2, max= 92, avg=66.27, stdev=23.90, samples=11 lat (msec) : 2=3.39%, 4=29.56%, 10=45.18%, 20=19.40%, 50=2.21% lat (msec) : 100=0.13%, 500=0.13% cpu : usr=0.07%, sys=9.05%, ctx=1144, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20972: Fri Dec 13 16:04:50 2024 read: IOPS=58, BW=936KiB/s (958kB/s)(5744KiB/6137msec) clat (usec): min=1676, max=43067, avg=6225.12, stdev=5455.14 lat (usec): min=1677, max=43067, avg=6225.72, stdev=5455.15 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1926], 10.00th=[ 2057], 20.00th=[ 2311], | 30.00th=[ 2671], 40.00th=[ 3064], 50.00th=[ 4015], 60.00th=[ 5604], | 70.00th=[ 7635], 80.00th=[ 9372], 90.00th=[12911], 95.00th=[16188], | 99.00th=[27395], 99.50th=[30540], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 32, max= 1312, per=25.10%, avg=933.17, stdev=325.99, samples=12 iops : min= 2, max= 82, avg=58.25, stdev=20.37, samples=12 write: IOPS=66, BW=1066KiB/s (1092kB/s)(6544KiB/6137msec) clat (msec): min=2, max=373, avg= 9.50, stdev=19.06 lat (msec): min=2, max=373, avg= 9.50, stdev=19.06 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 13], 90.00th=[ 17], 95.00th=[ 19], | 99.00th=[ 29], 99.50th=[ 34], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 160, max= 1632, per=27.06%, avg=1058.42, stdev=382.19, samples=12 iops : min= 10, max= 102, avg=66.08, stdev=23.90, samples=12 lat (msec) : 2=3.39%, 4=31.12%, 10=41.93%, 20=19.92%, 50=3.39% lat (msec) : 100=0.13%, 500=0.13% cpu : usr=0.15%, sys=8.43%, ctx=1181, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20973: Fri Dec 13 16:04:50 2024 read: IOPS=60, BW=971KiB/s (994kB/s)(6256KiB/6442msec) clat (usec): min=1678, max=73259, avg=6599.16, stdev=6246.20 lat (usec): min=1678, max=73261, avg=6599.81, stdev=6246.21 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1926], 10.00th=[ 2073], 20.00th=[ 2376], | 30.00th=[ 2802], 40.00th=[ 3425], 50.00th=[ 4146], 60.00th=[ 5866], | 70.00th=[ 8094], 80.00th=[10028], 90.00th=[12911], 95.00th=[17171], | 99.00th=[26608], 99.50th=[31065], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 64, max= 1184, per=26.46%, avg=983.75, stdev=319.42, samples=12 iops : min= 4, max= 74, avg=61.42, stdev=19.96, samples=12 write: IOPS=58, BW=936KiB/s (959kB/s)(6032KiB/6442msec) clat (msec): min=2, max=350, avg=10.10, stdev=18.52 lat (msec): min=2, max=350, avg=10.10, stdev=18.52 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 17], 95.00th=[ 20], | 99.00th=[ 37], 99.50th=[ 46], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 96, max= 1277, per=23.59%, avg=922.42, stdev=318.77, samples=12 iops : min= 6, max= 79, avg=57.58, stdev=19.84, samples=12 lat (msec) : 2=3.52%, 4=28.91%, 10=40.76%, 20=23.18%, 50=3.39% lat (msec) : 100=0.13%, 500=0.13% cpu : usr=0.11%, sys=8.74%, ctx=1246, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20974: Fri Dec 13 16:04:50 2024 read: IOPS=59, BW=947KiB/s (970kB/s)(6032KiB/6371msec) clat (usec): min=1593, max=48131, avg=6422.84, stdev=5455.49 lat (usec): min=1594, max=48132, avg=6423.48, stdev=5455.50 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1958], 10.00th=[ 2114], 20.00th=[ 2474], | 30.00th=[ 2999], 40.00th=[ 3523], 50.00th=[ 4555], 60.00th=[ 5866], | 70.00th=[ 7570], 80.00th=[ 9503], 90.00th=[12518], 95.00th=[16450], | 99.00th=[28967], 99.50th=[31851], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 704, max= 1248, per=27.77%, avg=1032.45, stdev=164.88, samples=11 iops : min= 44, max= 78, avg=64.45, stdev=10.25, samples=11 write: IOPS=61, BW=982KiB/s (1006kB/s)(6256KiB/6371msec) clat (msec): min=2, max=325, avg= 9.92, stdev=17.49 lat (msec): min=2, max=325, avg= 9.93, stdev=17.49 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 13], 90.00th=[ 17], 95.00th=[ 21], | 99.00th=[ 48], 99.50th=[ 83], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 96, max= 1408, per=25.15%, avg=983.75, stdev=332.53, samples=12 iops : min= 6, max= 88, avg=61.42, stdev=20.77, samples=12 lat (msec) : 2=2.86%, 4=26.95%, 10=46.48%, 20=19.79%, 50=3.52% lat (msec) : 100=0.26%, 500=0.13% cpu : usr=0.11%, sys=8.62%, ctx=1253, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3718KiB/s (3807kB/s), 936KiB/s-991KiB/s (958kB/s-1014kB/s), io=23.4MiB (24.5MB), run=5976-6442msec WRITE: bw=3912KiB/s (4006kB/s), 936KiB/s-1066KiB/s (959kB/s-1092kB/s), io=24.6MiB (25.8MB), run=5976-6442msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20989: Fri Dec 13 16:04:54 2024 read: IOPS=2, BW=2731KiB/s (2796kB/s)(7168KiB/2625msec) clat (usec): min=510, max=935507, avg=329053.40, stdev=424695.92 lat (usec): min=511, max=935509, avg=329054.37, stdev=424696.22 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 510], 10.00th=[ 510], 20.00th=[ 840], | 30.00th=[ 24249], 40.00th=[ 24249], 50.00th=[ 86508], 60.00th=[337642], | 70.00th=[337642], 80.00th=[918553], 90.00th=[935330], 95.00th=[935330], | 99.00th=[935330], 99.50th=[935330], 99.90th=[935330], 99.95th=[935330], | 99.99th=[935330] bw ( KiB/s): min= 2048, max= 4096, per=38.51%, avg=2560.00, stdev=1024.00, samples=4 iops : min= 2, max= 4, avg= 2.50, stdev= 1.00, samples=4 write: IOPS=1, BW=1950KiB/s (1997kB/s)(5120KiB/2625msec) clat (msec): min=19, max=119, avg=55.04, stdev=37.92 lat (msec): min=19, max=119, avg=55.13, stdev=37.89 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 20], | 30.00th=[ 45], 40.00th=[ 45], 50.00th=[ 46], 60.00th=[ 46], | 70.00th=[ 47], 80.00th=[ 47], 90.00th=[ 121], 95.00th=[ 121], | 99.00th=[ 121], 99.50th=[ 121], 99.90th=[ 121], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 4096, max= 4096, per=44.01%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33%, 1000=8.33% lat (msec) : 20=8.33%, 50=33.33%, 100=8.33%, 250=8.33%, 500=8.33% lat (msec) : 1000=16.67% cpu : usr=0.00%, sys=25.84%, ctx=749, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20990: Fri Dec 13 16:04:54 2024 read: IOPS=0, BW=665KiB/s (681kB/s)(2048KiB/3081msec) clat (msec): min=53, max=1324, avg=688.83, stdev=898.82 lat (msec): min=53, max=1324, avg=688.83, stdev=898.82 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 54], 10.00th=[ 54], 20.00th=[ 54], | 30.00th=[ 54], 40.00th=[ 54], 50.00th=[ 54], 60.00th=[ 1318], | 70.00th=[ 1318], 80.00th=[ 1318], 90.00th=[ 1318], 95.00th=[ 1318], | 99.00th=[ 1318], 99.50th=[ 1318], 99.90th=[ 1318], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 2048, max= 2048, per=30.81%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3324KiB/s (3403kB/s)(10.0MiB/3081msec) clat (msec): min=18, max=1070, avg=169.56, stdev=322.61 lat (msec): min=18, max=1070, avg=169.62, stdev=322.61 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 28], | 30.00th=[ 30], 40.00th=[ 36], 50.00th=[ 39], 60.00th=[ 41], | 70.00th=[ 171], 80.00th=[ 171], 90.00th=[ 192], 95.00th=[ 1070], | 99.00th=[ 1070], 99.50th=[ 1070], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min=10219, max=10219, per=100.00%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=16.67%, 250=16.67% cpu : usr=0.00%, sys=28.47%, ctx=473, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20991: Fri Dec 13 16:04:54 2024 read: IOPS=2, BW=2365KiB/s (2422kB/s)(6144KiB/2598msec) clat (usec): min=336, max=859885, avg=173898.30, stdev=339456.23 lat (usec): min=337, max=859886, avg=173899.32, stdev=339456.00 clat percentiles (usec): | 1.00th=[ 338], 5.00th=[ 338], 10.00th=[ 338], 20.00th=[ 424], | 30.00th=[ 424], 40.00th=[ 523], 50.00th=[ 523], 60.00th=[ 63177], | 70.00th=[119014], 80.00th=[119014], 90.00th=[859833], 95.00th=[859833], | 99.00th=[859833], 99.50th=[859833], 99.90th=[859833], 99.95th=[859833], | 99.99th=[859833] bw ( KiB/s): min= 2048, max= 6144, per=61.62%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=2, BW=2365KiB/s (2422kB/s)(6144KiB/2598msec) clat (msec): min=32, max=1041, avg=256.23, stdev=398.04 lat (msec): min=32, max=1041, avg=256.30, stdev=398.05 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 38], 50.00th=[ 38], 60.00th=[ 91], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 1045], 95.00th=[ 1045], | 99.00th=[ 1045], 99.50th=[ 1045], 99.90th=[ 1045], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 2048, max= 4096, per=36.68%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 50=25.00%, 100=16.67%, 250=8.33%, 500=8.33%, 1000=8.33% cpu : usr=0.00%, sys=9.82%, ctx=404, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20992: Fri Dec 13 16:04:54 2024 read: IOPS=1, BW=2002KiB/s (2050kB/s)(5120KiB/2557msec) clat (usec): min=463, max=856100, avg=225685.68, stdev=362842.08 lat (usec): min=464, max=856101, avg=225686.73, stdev=362841.93 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 465], 10.00th=[ 465], 20.00th=[ 465], | 30.00th=[ 482], 40.00th=[ 482], 50.00th=[ 59507], 60.00th=[ 59507], | 70.00th=[210764], 80.00th=[210764], 90.00th=[859833], 95.00th=[859833], | 99.00th=[859833], 99.50th=[859833], 99.90th=[859833], 99.95th=[859833], | 99.99th=[859833] bw ( KiB/s): min= 2048, max= 6144, per=61.62%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=2, BW=2803KiB/s (2871kB/s)(7168KiB/2557msec) clat (msec): min=23, max=1030, avg=199.14, stdev=367.57 lat (msec): min=23, max=1030, avg=199.19, stdev=367.58 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 33], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 72], 60.00th=[ 73], | 70.00th=[ 73], 80.00th=[ 106], 90.00th=[ 1028], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 2048, max= 6144, per=44.01%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (usec) : 500=16.67% lat (msec) : 50=16.67%, 100=33.33%, 250=16.67%, 1000=8.33% cpu : usr=0.00%, sys=18.04%, ctx=380, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6647KiB/s (6807kB/s), 665KiB/s-2731KiB/s (681kB/s-2796kB/s), io=20.0MiB (20.0MB), run=2557-3081msec WRITE: bw=9306KiB/s (9529kB/s), 1950KiB/s-3324KiB/s (1997kB/s-3403kB/s), io=28.0MiB (29.4MB), run=2557-3081msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20993: Fri Dec 13 16:04:54 2024 read: IOPS=5, BW=5294KiB/s (5421kB/s)(7168KiB/1354msec) clat (msec): min=12, max=928, avg=175.38, stdev=333.77 lat (msec): min=12, max=928, avg=175.38, stdev=333.77 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 78], 60.00th=[ 83], | 70.00th=[ 83], 80.00th=[ 94], 90.00th=[ 927], 95.00th=[ 927], | 99.00th=[ 927], 99.50th=[ 927], 99.90th=[ 927], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 4096, max= 4096, per=62.82%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3781KiB/s (3872kB/s)(5120KiB/1354msec) clat (usec): min=13733, max=41197, avg=23330.17, stdev=11795.66 lat (usec): min=13792, max=41226, avg=23370.64, stdev=11792.78 clat percentiles (usec): | 1.00th=[13698], 5.00th=[13698], 10.00th=[13698], 20.00th=[13698], | 30.00th=[14615], 40.00th=[14615], 50.00th=[17695], 60.00th=[17695], | 70.00th=[29492], 80.00th=[29492], 90.00th=[41157], 95.00th=[41157], | 99.00th=[41157], 99.50th=[41157], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] lat (msec) : 20=41.67%, 50=25.00%, 100=25.00%, 1000=8.33% cpu : usr=0.00%, sys=5.17%, ctx=30, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20994: Fri Dec 13 16:04:54 2024 read: IOPS=0, BW=652KiB/s (668kB/s)(2048KiB/3141msec) clat (msec): min=31, max=1194, avg=613.26, stdev=822.48 lat (msec): min=31, max=1194, avg=613.26, stdev=822.48 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 1200], | 70.00th=[ 1200], 80.00th=[ 1200], 90.00th=[ 1200], 95.00th=[ 1200], | 99.00th=[ 1200], 99.50th=[ 1200], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 1706, max= 2048, per=28.79%, avg=1877.00, stdev=241.83, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3260KiB/s (3338kB/s)(10.0MiB/3141msec) clat (msec): min=20, max=1037, avg=182.67, stdev=308.22 lat (msec): min=20, max=1038, avg=182.72, stdev=308.23 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 23], 40.00th=[ 34], 50.00th=[ 65], 60.00th=[ 95], | 70.00th=[ 171], 80.00th=[ 171], 90.00th=[ 203], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 6826, max=10240, per=93.48%, avg=8533.00, stdev=2414.06, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (msec) : 50=41.67%, 100=16.67%, 250=25.00% cpu : usr=0.00%, sys=4.68%, ctx=76, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20995: Fri Dec 13 16:04:54 2024 read: IOPS=1, BW=1981KiB/s (2029kB/s)(6144KiB/3101msec) clat (msec): min=13, max=968, avg=216.41, stdev=372.89 lat (msec): min=13, max=968, avg=216.41, stdev=372.89 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 63], | 70.00th=[ 174], 80.00th=[ 174], 90.00th=[ 969], 95.00th=[ 969], | 99.00th=[ 969], 99.50th=[ 969], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 2048, max= 6144, per=62.82%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=1, BW=1981KiB/s (2029kB/s)(6144KiB/3101msec) clat (msec): min=38, max=1055, avg=285.48, stdev=393.73 lat (msec): min=38, max=1055, avg=285.51, stdev=393.73 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 43], | 30.00th=[ 43], 40.00th=[ 45], 50.00th=[ 45], 60.00th=[ 226], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 1053], 95.00th=[ 1053], | 99.00th=[ 1053], 99.50th=[ 1053], 99.90th=[ 1053], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 4096, max= 4096, per=44.87%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 lat (msec) : 20=8.33%, 50=41.67%, 100=8.33%, 250=16.67%, 500=8.33% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=4.94%, ctx=118, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20996: Fri Dec 13 16:04:54 2024 read: IOPS=1, BW=1748KiB/s (1790kB/s)(5120KiB/2929msec) clat (msec): min=13, max=764, avg=209.92, stdev=319.87 lat (msec): min=13, max=764, avg=209.92, stdev=319.87 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 14], | 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 45], 60.00th=[ 45], | 70.00th=[ 207], 80.00th=[ 207], 90.00th=[ 768], 95.00th=[ 768], | 99.00th=[ 768], 99.50th=[ 768], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 2048, max= 6144, per=62.82%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=2, BW=2447KiB/s (2506kB/s)(7168KiB/2929msec) clat (msec): min=24, max=1091, avg=259.69, stdev=380.44 lat (msec): min=24, max=1091, avg=259.76, stdev=380.41 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 30], | 30.00th=[ 72], 40.00th=[ 72], 50.00th=[ 108], 60.00th=[ 176], | 70.00th=[ 176], 80.00th=[ 317], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 2043, max= 6144, per=44.85%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (msec) : 20=16.67%, 50=25.00%, 100=8.33%, 250=25.00%, 500=8.33% lat (msec) : 1000=8.33% cpu : usr=0.07%, sys=3.62%, ctx=138, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6520KiB/s (6677kB/s), 652KiB/s-5294KiB/s (668kB/s-5421kB/s), io=20.0MiB (20.0MB), run=1354-3141msec WRITE: bw=9128KiB/s (9347kB/s), 1981KiB/s-3781KiB/s (2029kB/s-3872kB/s), io=28.0MiB (29.4MB), run=1354-3141msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21004: Fri Dec 13 16:04:59 2024 read: IOPS=0, BW=3156KiB/s (3231kB/s)(12.0MiB/3894msec) clat (msec): min=185, max=3430, avg=1283.02, stdev=1859.96 lat (msec): min=185, max=3430, avg=1283.02, stdev=1859.96 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 186], 10.00th=[ 186], 20.00th=[ 186], | 30.00th=[ 186], 40.00th=[ 234], 50.00th=[ 234], 60.00th=[ 234], | 70.00th=[ 3440], 80.00th=[ 3440], 90.00th=[ 3440], 95.00th=[ 3440], | 99.00th=[ 3440], 99.50th=[ 3440], 99.90th=[ 3440], 99.95th=[ 3440], | 99.99th=[ 3440] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=31.54%, ctx=1237, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21005: Fri Dec 13 16:04:59 2024 read: IOPS=0, BW=965KiB/s (988kB/s)(4096KiB/4245msec) clat (nsec): min=217527k, max=217527k, avg=217526781.00, stdev= 0.00 lat (nsec): min=217529k, max=217529k, avg=217528787.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 218], 10.00th=[ 218], 20.00th=[ 218], | 30.00th=[ 218], 40.00th=[ 218], 50.00th=[ 218], 60.00th=[ 218], | 70.00th=[ 218], 80.00th=[ 218], 90.00th=[ 218], 95.00th=[ 218], | 99.00th=[ 218], 99.50th=[ 218], 99.90th=[ 218], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1930KiB/s (1976kB/s)(8192KiB/4245msec) clat (msec): min=289, max=3723, avg=2006.48, stdev=2428.00 lat (msec): min=289, max=3723, avg=2006.69, stdev=2428.04 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 292], 10.00th=[ 292], 20.00th=[ 292], | 30.00th=[ 292], 40.00th=[ 292], 50.00th=[ 292], 60.00th=[ 3708], | 70.00th=[ 3708], 80.00th=[ 3708], 90.00th=[ 3708], 95.00th=[ 3708], | 99.00th=[ 3708], 99.50th=[ 3708], 99.90th=[ 3708], 99.95th=[ 3708], | 99.99th=[ 3708] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=7.75%, ctx=247, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21006: Fri Dec 13 16:04:59 2024 write: IOPS=0, BW=3183KiB/s (3260kB/s)(12.0MiB/3860msec) clat (msec): min=222, max=3386, avg=1280.70, stdev=1823.51 lat (msec): min=223, max=3386, avg=1281.06, stdev=1823.46 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 224], 10.00th=[ 224], 20.00th=[ 224], | 30.00th=[ 224], 40.00th=[ 234], 50.00th=[ 234], 60.00th=[ 234], | 70.00th=[ 3373], 80.00th=[ 3373], 90.00th=[ 3373], 95.00th=[ 3373], | 99.00th=[ 3373], 99.50th=[ 3373], 99.90th=[ 3373], 99.95th=[ 3373], | 99.99th=[ 3373] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.16%, sys=9.82%, ctx=502, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21007: Fri Dec 13 16:04:59 2024 write: IOPS=0, BW=3173KiB/s (3249kB/s)(12.0MiB/3873msec) clat (msec): min=112, max=3478, avg=1275.07, stdev=1909.29 lat (msec): min=112, max=3478, avg=1275.32, stdev=1909.29 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 113], 10.00th=[ 113], 20.00th=[ 113], | 30.00th=[ 113], 40.00th=[ 234], 50.00th=[ 234], 60.00th=[ 234], | 70.00th=[ 3473], 80.00th=[ 3473], 90.00th=[ 3473], 95.00th=[ 3473], | 99.00th=[ 3473], 99.50th=[ 3473], 99.90th=[ 3473], 99.95th=[ 3473], | 99.99th=[ 3473] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.03%, sys=7.54%, ctx=31, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3860KiB/s (3952kB/s), 965KiB/s-3156KiB/s (988kB/s-3231kB/s), io=16.0MiB (16.8MB), run=3894-4245msec WRITE: bw=7719KiB/s (7904kB/s), 1930KiB/s-3183KiB/s (1976kB/s-3260kB/s), io=32.0MiB (33.6MB), run=3860-4245msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21009: Fri Dec 13 16:04:59 2024 read: IOPS=0, BW=2993KiB/s (3065kB/s)(12.0MiB/4105msec) clat (msec): min=108, max=3581, avg=1365.33, stdev=1924.69 lat (msec): min=108, max=3581, avg=1365.34, stdev=1924.69 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 109], 10.00th=[ 109], 20.00th=[ 109], | 30.00th=[ 109], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 405], | 70.00th=[ 3574], 80.00th=[ 3574], 90.00th=[ 3574], 95.00th=[ 3574], | 99.00th=[ 3574], 99.50th=[ 3574], 99.90th=[ 3574], 99.95th=[ 3574], | 99.99th=[ 3574] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=1.83%, ctx=34, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21010: Fri Dec 13 16:04:59 2024 read: IOPS=0, BW=963KiB/s (986kB/s)(4096KiB/4252msec) clat (nsec): min=390528k, max=390528k, avg=390528084.00, stdev= 0.00 lat (nsec): min=390529k, max=390529k, avg=390528977.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 393], 10.00th=[ 393], 20.00th=[ 393], | 30.00th=[ 393], 40.00th=[ 393], 50.00th=[ 393], 60.00th=[ 393], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1927KiB/s (1973kB/s)(8192KiB/4252msec) clat (msec): min=299, max=3554, avg=1927.05, stdev=2301.29 lat (msec): min=300, max=3554, avg=1927.37, stdev=2301.15 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 3540], | 70.00th=[ 3540], 80.00th=[ 3540], 90.00th=[ 3540], 95.00th=[ 3540], | 99.00th=[ 3540], 99.50th=[ 3540], 99.90th=[ 3540], 99.95th=[ 3540], | 99.99th=[ 3540] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=1.34%, ctx=24, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21011: Fri Dec 13 16:04:59 2024 write: IOPS=0, BW=3051KiB/s (3124kB/s)(12.0MiB/4028msec) clat (msec): min=55, max=3605, avg=1338.18, stdev=1968.89 lat (msec): min=56, max=3605, avg=1338.42, stdev=1968.89 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 56], 10.00th=[ 56], 20.00th=[ 56], | 30.00th=[ 56], 40.00th=[ 355], 50.00th=[ 355], 60.00th=[ 355], | 70.00th=[ 3608], 80.00th=[ 3608], 90.00th=[ 3608], 95.00th=[ 3608], | 99.00th=[ 3608], 99.50th=[ 3608], 99.90th=[ 3608], 99.95th=[ 3608], | 99.99th=[ 3608] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.05%, sys=4.40%, ctx=59, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21012: Fri Dec 13 16:04:59 2024 write: IOPS=0, BW=2858KiB/s (2926kB/s)(12.0MiB/4300msec) clat (msec): min=31, max=3694, avg=1428.56, stdev=1980.14 lat (msec): min=31, max=3694, avg=1428.83, stdev=1980.18 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 558], 50.00th=[ 558], 60.00th=[ 558], | 70.00th=[ 3708], 80.00th=[ 3708], 90.00th=[ 3708], 95.00th=[ 3708], | 99.00th=[ 3708], 99.50th=[ 3708], 99.90th=[ 3708], 99.95th=[ 3708], | 99.99th=[ 3708] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 750=33.33% cpu : usr=0.07%, sys=2.54%, ctx=155, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3853KiB/s (3946kB/s), 963KiB/s-2993KiB/s (986kB/s-3065kB/s), io=16.0MiB (16.8MB), run=4105-4252msec WRITE: bw=7620KiB/s (7803kB/s), 1927KiB/s-3051KiB/s (1973kB/s-3124kB/s), io=32.0MiB (33.6MB), run=4028-4300msec