== sanity test 398b: DIO and buffer IO race ============== 01:18:17 (1735107497) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.21477 s, 41.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24989: Wed Dec 25 01:18:31 2024 read: IOPS=124, BW=499KiB/s (511kB/s)(5992KiB/12006msec) clat (usec): min=54, max=32523, avg=4453.79, stdev=2679.58 lat (usec): min=54, max=32523, avg=4454.96, stdev=2679.95 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 110], 10.00th=[ 2114], 20.00th=[ 2573], | 30.00th=[ 2999], 40.00th=[ 3425], 50.00th=[ 4015], 60.00th=[ 4752], | 70.00th=[ 5342], 80.00th=[ 6063], 90.00th=[ 7242], 95.00th=[ 8717], | 99.00th=[13698], 99.50th=[15795], 99.90th=[26870], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 263, max= 712, per=24.91%, avg=498.21, stdev=90.13, samples=24 iops : min= 65, max= 178, avg=124.50, stdev=22.62, samples=24 write: IOPS=131, BW=524KiB/s (537kB/s)(6296KiB/12006msec) clat (usec): min=436, max=139510, avg=3354.36, stdev=4541.03 lat (usec): min=437, max=139512, avg=3355.83, stdev=4541.61 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 635], | 30.00th=[ 799], 40.00th=[ 1696], 50.00th=[ 2540], 60.00th=[ 3261], | 70.00th=[ 4228], 80.00th=[ 5473], 90.00th=[ 7373], 95.00th=[ 9241], | 99.00th=[ 12911], 99.50th=[ 14484], 99.90th=[ 24511], 99.95th=[139461], | 99.99th=[139461] bw ( KiB/s): min= 311, max= 720, per=25.00%, avg=523.25, stdev=95.06, samples=24 iops : min= 77, max= 180, avg=130.75, stdev=23.85, samples=24 lat (usec) : 100=2.34%, 250=0.42%, 500=0.36%, 750=14.13%, 1000=3.22% lat (msec) : 2=5.05%, 4=33.37%, 10=37.79%, 20=3.12%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.27%, sys=23.15%, ctx=5909, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24990: Wed Dec 25 01:18:31 2024 read: IOPS=124, BW=499KiB/s (511kB/s)(5940KiB/11894msec) clat (usec): min=48, max=32169, avg=4440.71, stdev=2661.09 lat (usec): min=49, max=32170, avg=4441.57, stdev=2660.88 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 105], 10.00th=[ 2147], 20.00th=[ 2573], | 30.00th=[ 3032], 40.00th=[ 3458], 50.00th=[ 4047], 60.00th=[ 4555], | 70.00th=[ 5276], 80.00th=[ 6063], 90.00th=[ 7308], 95.00th=[ 8455], | 99.00th=[13304], 99.50th=[15401], 99.90th=[25560], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 304, max= 648, per=24.72%, avg=494.43, stdev=76.89, samples=23 iops : min= 76, max= 162, avg=123.52, stdev=19.24, samples=23 write: IOPS=133, BW=534KiB/s (547kB/s)(6348KiB/11894msec) clat (usec): min=476, max=184155, avg=3302.51, stdev=5362.41 lat (usec): min=476, max=184156, avg=3303.23, stdev=5362.40 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 627], | 30.00th=[ 791], 40.00th=[ 1680], 50.00th=[ 2638], 60.00th=[ 3130], | 70.00th=[ 3982], 80.00th=[ 5342], 90.00th=[ 7242], 95.00th=[ 8848], | 99.00th=[ 12256], 99.50th=[ 13698], 99.90th=[ 20317], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 352, max= 752, per=24.88%, avg=520.78, stdev=80.87, samples=23 iops : min= 88, max= 188, avg=130.13, stdev=20.19, samples=23 lat (usec) : 50=0.03%, 100=2.28%, 250=0.39%, 500=0.23%, 750=14.52% lat (usec) : 1000=2.96% lat (msec) : 2=4.82%, 4=34.80%, 10=36.85%, 20=2.90%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=23.12%, ctx=5968, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24991: Wed Dec 25 01:18:31 2024 read: IOPS=125, BW=503KiB/s (515kB/s)(6028KiB/11991msec) clat (usec): min=54, max=183254, avg=4426.59, stdev=5217.59 lat (usec): min=55, max=183255, avg=4428.10, stdev=5218.00 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 338], 10.00th=[ 2147], 20.00th=[ 2606], | 30.00th=[ 2966], 40.00th=[ 3425], 50.00th=[ 3916], 60.00th=[ 4424], | 70.00th=[ 5145], 80.00th=[ 5866], 90.00th=[ 6980], 95.00th=[ 8225], | 99.00th=[ 12387], 99.50th=[ 17695], 99.90th=[ 25297], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 312, max= 696, per=24.89%, avg=497.74, stdev=82.12, samples=23 iops : min= 78, max= 174, avg=124.43, stdev=20.53, samples=23 write: IOPS=130, BW=522KiB/s (535kB/s)(6260KiB/11991msec) clat (usec): min=425, max=20716, avg=3362.03, stdev=2871.19 lat (usec): min=426, max=20717, avg=3362.96, stdev=2871.18 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 644], | 30.00th=[ 865], 40.00th=[ 2114], 50.00th=[ 2737], 60.00th=[ 3425], | 70.00th=[ 4490], 80.00th=[ 5669], 90.00th=[ 7373], 95.00th=[ 8848], | 99.00th=[11731], 99.50th=[13698], 99.90th=[18744], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 312, max= 688, per=24.60%, avg=514.78, stdev=84.10, samples=23 iops : min= 78, max= 172, avg=128.70, stdev=21.02, samples=23 lat (usec) : 100=2.25%, 250=0.20%, 500=0.62%, 750=12.99%, 1000=3.06% lat (msec) : 2=4.36%, 4=35.19%, 10=38.87%, 20=2.31%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=23.23%, ctx=5887, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24992: Wed Dec 25 01:18:31 2024 read: IOPS=126, BW=505KiB/s (517kB/s)(6052KiB/11976msec) clat (usec): min=51, max=186886, avg=4469.91, stdev=5261.95 lat (usec): min=51, max=186887, avg=4470.53, stdev=5261.95 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 133], 10.00th=[ 2180], 20.00th=[ 2540], | 30.00th=[ 2966], 40.00th=[ 3359], 50.00th=[ 3884], 60.00th=[ 4621], | 70.00th=[ 5342], 80.00th=[ 6063], 90.00th=[ 7308], 95.00th=[ 8291], | 99.00th=[ 12387], 99.50th=[ 15401], 99.90th=[ 18220], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 303, max= 576, per=24.85%, avg=497.04, stdev=65.84, samples=23 iops : min= 75, max= 144, avg=124.13, stdev=16.52, samples=23 write: IOPS=130, BW=521KiB/s (533kB/s)(6236KiB/11976msec) clat (usec): min=435, max=16250, avg=3303.74, stdev=2985.96 lat (usec): min=435, max=16251, avg=3305.00, stdev=2985.61 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 545], 10.00th=[ 562], 20.00th=[ 619], | 30.00th=[ 750], 40.00th=[ 1745], 50.00th=[ 2606], 60.00th=[ 3261], | 70.00th=[ 4178], 80.00th=[ 5604], 90.00th=[ 7701], 95.00th=[ 9241], | 99.00th=[13435], 99.50th=[14877], 99.90th=[15795], 99.95th=[16188], | 99.99th=[16188] bw ( KiB/s): min= 383, max= 758, per=24.51%, avg=513.04, stdev=77.56, samples=23 iops : min= 95, max= 189, avg=128.13, stdev=19.39, samples=23 lat (usec) : 100=2.15%, 250=0.39%, 500=0.26%, 750=14.94%, 1000=2.90% lat (msec) : 2=4.20%, 4=35.16%, 10=37.34%, 20=2.64%, 250=0.03% cpu : usr=0.20%, sys=23.16%, ctx=5947, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2000KiB/s (2048kB/s), 499KiB/s-505KiB/s (511kB/s-517kB/s), io=23.4MiB (24.6MB), run=11894-12006msec WRITE: bw=2094KiB/s (2144kB/s), 521KiB/s-534KiB/s (533kB/s-547kB/s), io=24.6MiB (25.7MB), run=11894-12006msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24993: Wed Dec 25 01:18:35 2024 read: IOPS=96, BW=384KiB/s (393kB/s)(5992KiB/15599msec) clat (usec): min=1531, max=203657, avg=4234.19, stdev=5979.07 lat (usec): min=1532, max=203658, avg=4234.86, stdev=5979.05 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1893], 10.00th=[ 1975], 20.00th=[ 2147], | 30.00th=[ 2311], 40.00th=[ 2507], 50.00th=[ 2769], 60.00th=[ 3261], | 70.00th=[ 4146], 80.00th=[ 5473], 90.00th=[ 8455], 95.00th=[ 10683], | 99.00th=[ 15664], 99.50th=[ 17695], 99.90th=[ 25822], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 176, max= 712, per=24.95%, avg=383.19, stdev=155.39, samples=31 iops : min= 44, max= 178, avg=95.77, stdev=38.87, samples=31 write: IOPS=100, BW=404KiB/s (413kB/s)(6296KiB/15599msec) clat (usec): min=2137, max=25229, avg=5846.02, stdev=3366.32 lat (usec): min=2137, max=25230, avg=5846.86, stdev=3366.41 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2868], 10.00th=[ 3064], 20.00th=[ 3359], | 30.00th=[ 3589], 40.00th=[ 3851], 50.00th=[ 4424], 60.00th=[ 5342], | 70.00th=[ 6783], 80.00th=[ 8356], 90.00th=[10552], 95.00th=[12387], | 99.00th=[17695], 99.50th=[20055], 99.90th=[24773], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 184, max= 760, per=25.00%, avg=402.03, stdev=160.70, samples=31 iops : min= 46, max= 190, avg=100.48, stdev=40.18, samples=31 lat (msec) : 2=5.40%, 4=50.39%, 10=34.93%, 20=8.89%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=11.32%, ctx=3705, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24994: Wed Dec 25 01:18:35 2024 read: IOPS=95, BW=380KiB/s (389kB/s)(5940KiB/15625msec) clat (usec): min=1579, max=189686, avg=4168.25, stdev=5636.19 lat (usec): min=1580, max=189687, avg=4168.95, stdev=5636.17 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1876], 10.00th=[ 1975], 20.00th=[ 2147], | 30.00th=[ 2311], 40.00th=[ 2474], 50.00th=[ 2769], 60.00th=[ 3228], | 70.00th=[ 3982], 80.00th=[ 5538], 90.00th=[ 8356], 95.00th=[ 10683], | 99.00th=[ 14877], 99.50th=[ 16581], 99.90th=[ 18744], 99.95th=[189793], | 99.99th=[189793] bw ( KiB/s): min= 176, max= 712, per=24.68%, avg=379.06, stdev=159.99, samples=31 iops : min= 44, max= 178, avg=94.74, stdev=40.01, samples=31 write: IOPS=101, BW=406KiB/s (416kB/s)(6348KiB/15625msec) clat (usec): min=2383, max=33397, avg=5921.75, stdev=3492.00 lat (usec): min=2383, max=33397, avg=5922.67, stdev=3491.92 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2900], 10.00th=[ 3130], 20.00th=[ 3359], | 30.00th=[ 3621], 40.00th=[ 3949], 50.00th=[ 4490], 60.00th=[ 5342], | 70.00th=[ 6783], 80.00th=[ 8455], 90.00th=[10683], 95.00th=[12518], | 99.00th=[18220], 99.50th=[22938], 99.90th=[26084], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 224, max= 704, per=25.10%, avg=403.58, stdev=155.20, samples=31 iops : min= 56, max= 176, avg=100.87, stdev=38.80, samples=31 lat (msec) : 2=5.57%, 4=49.58%, 10=35.38%, 20=9.11%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.25%, ctx=3706, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24995: Wed Dec 25 01:18:35 2024 read: IOPS=96, BW=387KiB/s (396kB/s)(6028KiB/15588msec) clat (usec): min=1620, max=20408, avg=4149.22, stdev=3100.43 lat (usec): min=1621, max=20409, avg=4149.93, stdev=3100.44 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1876], 10.00th=[ 1975], 20.00th=[ 2147], | 30.00th=[ 2311], 40.00th=[ 2507], 50.00th=[ 2769], 60.00th=[ 3228], | 70.00th=[ 4113], 80.00th=[ 5735], 90.00th=[ 8848], 95.00th=[10945], | 99.00th=[15270], 99.50th=[17433], 99.90th=[20055], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 167, max= 776, per=25.11%, avg=385.71, stdev=156.86, samples=31 iops : min= 41, max= 194, avg=96.35, stdev=39.27, samples=31 write: IOPS=100, BW=402KiB/s (411kB/s)(6260KiB/15588msec) clat (msec): min=2, max=174, avg= 5.93, stdev= 5.46 lat (msec): min=2, max=174, avg= 5.93, stdev= 5.46 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 19], 99.50th=[ 24], 99.90th=[ 27], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 215, max= 760, per=24.87%, avg=399.90, stdev=160.60, samples=31 iops : min= 53, max= 190, avg=99.90, stdev=40.20, samples=31 lat (msec) : 2=5.63%, 4=50.00%, 10=35.25%, 20=8.66%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.30%, ctx=3640, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24996: Wed Dec 25 01:18:35 2024 read: IOPS=96, BW=388KiB/s (397kB/s)(6052KiB/15612msec) clat (usec): min=1605, max=25165, avg=4221.48, stdev=3088.86 lat (usec): min=1606, max=25165, avg=4222.25, stdev=3088.79 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1926], 10.00th=[ 2008], 20.00th=[ 2180], | 30.00th=[ 2343], 40.00th=[ 2540], 50.00th=[ 2868], 60.00th=[ 3392], | 70.00th=[ 4424], 80.00th=[ 6063], 90.00th=[ 8455], 95.00th=[10683], | 99.00th=[15795], 99.50th=[17957], 99.90th=[22676], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 183, max= 752, per=25.06%, avg=384.90, stdev=153.02, samples=31 iops : min= 45, max= 188, avg=96.13, stdev=38.34, samples=31 write: IOPS=99, BW=399KiB/s (409kB/s)(6236KiB/15612msec) clat (msec): min=2, max=163, avg= 5.88, stdev= 5.18 lat (msec): min=2, max=163, avg= 5.88, stdev= 5.18 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 12], | 99.00th=[ 18], 99.50th=[ 23], 99.90th=[ 29], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 199, max= 696, per=24.72%, avg=397.55, stdev=161.78, samples=31 iops : min= 49, max= 174, avg=99.29, stdev=40.49, samples=31 lat (msec) : 2=4.75%, 4=49.80%, 10=36.43%, 20=8.63%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=11.34%, ctx=3688, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1537KiB/s (1574kB/s), 380KiB/s-388KiB/s (389kB/s-397kB/s), io=23.4MiB (24.6MB), run=15588-15625msec WRITE: bw=1609KiB/s (1648kB/s), 399KiB/s-406KiB/s (409kB/s-416kB/s), io=24.6MiB (25.7MB), run=15588-15625msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25027: Wed Dec 25 01:18:43 2024 read: IOPS=55, BW=880KiB/s (901kB/s)(5920KiB/6726msec) clat (usec): min=62, max=191719, avg=12198.91, stdev=13105.70 lat (usec): min=63, max=191720, avg=12201.94, stdev=13106.06 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 6849], 10.00th=[ 7963], 20.00th=[ 8848], | 30.00th=[ 9634], 40.00th=[ 10159], 50.00th=[ 10945], 60.00th=[ 11731], | 70.00th=[ 12518], 80.00th=[ 13698], 90.00th=[ 15926], 95.00th=[ 18744], | 99.00th=[ 31327], 99.50th=[166724], 99.90th=[191890], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 672, max= 1085, per=24.77%, avg=868.54, stdev=141.59, samples=13 iops : min= 42, max= 67, avg=54.15, stdev= 8.82, samples=13 write: IOPS=59, BW=947KiB/s (969kB/s)(6368KiB/6726msec) clat (usec): min=678, max=69435, avg=5526.40, stdev=6183.54 lat (usec): min=679, max=69436, avg=5527.41, stdev=6183.52 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 848], 10.00th=[ 947], 20.00th=[ 1975], | 30.00th=[ 2474], 40.00th=[ 2868], 50.00th=[ 3359], 60.00th=[ 4555], | 70.00th=[ 5800], 80.00th=[ 7767], 90.00th=[12518], 95.00th=[16319], | 99.00th=[31589], 99.50th=[42206], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 480, max= 1344, per=25.75%, avg=949.77, stdev=233.33, samples=13 iops : min= 30, max= 84, avg=59.23, stdev=14.56, samples=13 lat (usec) : 100=1.30%, 250=0.39%, 750=0.13%, 1000=5.73% lat (msec) : 2=4.56%, 4=18.62%, 10=31.12%, 20=35.29%, 50=2.47% lat (msec) : 100=0.13%, 250=0.26% cpu : usr=0.13%, sys=22.65%, ctx=2812, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25029: Wed Dec 25 01:18:43 2024 read: IOPS=53, BW=854KiB/s (874kB/s)(5744KiB/6728msec) clat (usec): min=61, max=27781, avg=11218.20, stdev=3531.73 lat (usec): min=62, max=27782, avg=11223.82, stdev=3535.09 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 7111], 10.00th=[ 8160], 20.00th=[ 8848], | 30.00th=[ 9896], 40.00th=[10683], 50.00th=[11207], 60.00th=[11994], | 70.00th=[12518], 80.00th=[13304], 90.00th=[15008], 95.00th=[16319], | 99.00th=[21890], 99.50th=[23462], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 512, max= 1120, per=24.57%, avg=861.38, stdev=155.79, samples=13 iops : min= 32, max= 70, avg=53.77, stdev= 9.68, samples=13 write: IOPS=60, BW=973KiB/s (996kB/s)(6544KiB/6728msec) clat (usec): min=778, max=202240, avg=6535.74, stdev=14939.79 lat (usec): min=779, max=202241, avg=6536.73, stdev=14939.83 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 848], 10.00th=[ 938], 20.00th=[ 1418], | 30.00th=[ 2245], 40.00th=[ 2704], 50.00th=[ 3228], 60.00th=[ 4146], | 70.00th=[ 6128], 80.00th=[ 8848], 90.00th=[ 13304], 95.00th=[ 17171], | 99.00th=[ 35390], 99.50th=[ 46924], 99.90th=[202376], 99.95th=[202376], | 99.99th=[202376] bw ( KiB/s): min= 384, max= 1376, per=26.09%, avg=962.31, stdev=274.49, samples=13 iops : min= 24, max= 86, avg=60.08, stdev=17.16, samples=13 lat (usec) : 100=0.91%, 250=0.52%, 750=0.13%, 1000=7.29% lat (msec) : 2=5.47%, 4=19.01%, 10=25.26%, 20=39.32%, 50=1.82% lat (msec) : 250=0.26% cpu : usr=0.16%, sys=17.45%, ctx=2644, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25031: Wed Dec 25 01:18:43 2024 read: IOPS=57, BW=916KiB/s (938kB/s)(6256KiB/6830msec) clat (usec): min=65, max=158999, avg=11597.77, stdev=8246.96 lat (usec): min=66, max=159000, avg=11599.34, stdev=8247.23 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 7570], 10.00th=[ 8225], 20.00th=[ 9241], | 30.00th=[ 9765], 40.00th=[ 10421], 50.00th=[ 10945], 60.00th=[ 11731], | 70.00th=[ 12387], 80.00th=[ 13173], 90.00th=[ 14877], 95.00th=[ 15926], | 99.00th=[ 25822], 99.50th=[ 30802], 99.90th=[158335], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 512, max= 1248, per=26.05%, avg=913.23, stdev=230.24, samples=13 iops : min= 32, max= 78, avg=57.08, stdev=14.39, samples=13 write: IOPS=55, BW=883KiB/s (904kB/s)(6032KiB/6830msec) clat (usec): min=773, max=47994, avg=5599.54, stdev=6188.75 lat (usec): min=774, max=47995, avg=5600.69, stdev=6188.70 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 832], 10.00th=[ 922], 20.00th=[ 1926], | 30.00th=[ 2474], 40.00th=[ 2835], 50.00th=[ 3326], 60.00th=[ 4555], | 70.00th=[ 5997], 80.00th=[ 7898], 90.00th=[13042], 95.00th=[16057], | 99.00th=[34341], 99.50th=[47973], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 576, max= 1088, per=23.29%, avg=859.08, stdev=169.75, samples=13 iops : min= 36, max= 68, avg=53.69, stdev=10.61, samples=13 lat (usec) : 100=0.91%, 250=0.52%, 1000=5.60% lat (msec) : 2=5.08%, 4=17.06%, 10=30.60%, 20=37.50%, 50=2.60% lat (msec) : 250=0.13% cpu : usr=0.09%, sys=18.54%, ctx=2789, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25033: Wed Dec 25 01:18:43 2024 read: IOPS=55, BW=891KiB/s (912kB/s)(6032KiB/6773msec) clat (usec): min=60, max=55632, avg=11471.13, stdev=4162.98 lat (usec): min=61, max=55632, avg=11473.37, stdev=4162.05 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 7242], 10.00th=[ 8160], 20.00th=[ 8979], | 30.00th=[ 9896], 40.00th=[10552], 50.00th=[11207], 60.00th=[11863], | 70.00th=[12780], 80.00th=[13698], 90.00th=[14877], 95.00th=[16319], | 99.00th=[23462], 99.50th=[32375], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 510, max= 1120, per=25.06%, avg=878.62, stdev=202.53, samples=13 iops : min= 31, max= 70, avg=54.85, stdev=12.79, samples=13 write: IOPS=57, BW=924KiB/s (946kB/s)(6256KiB/6773msec) clat (usec): min=770, max=244897, avg=5787.23, stdev=13211.81 lat (usec): min=771, max=244898, avg=5789.48, stdev=13211.72 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 816], 10.00th=[ 848], 20.00th=[ 1287], | 30.00th=[ 2409], 40.00th=[ 2868], 50.00th=[ 3458], 60.00th=[ 4490], | 70.00th=[ 5735], 80.00th=[ 7308], 90.00th=[ 11994], 95.00th=[ 15664], | 99.00th=[ 31065], 99.50th=[ 42730], 99.90th=[244319], 99.95th=[244319], | 99.99th=[244319] bw ( KiB/s): min= 576, max= 1344, per=24.95%, avg=920.38, stdev=208.15, samples=13 iops : min= 36, max= 84, avg=57.46, stdev=12.98, samples=13 lat (usec) : 100=0.91%, 250=0.13%, 1000=8.85% lat (msec) : 2=2.73%, 4=17.45%, 10=29.56%, 20=38.28%, 50=1.82% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.12%, sys=18.50%, ctx=2815, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3507KiB/s (3591kB/s), 854KiB/s-916KiB/s (874kB/s-938kB/s), io=23.4MiB (24.5MB), run=6726-6830msec WRITE: bw=3690KiB/s (3778kB/s), 883KiB/s-973KiB/s (904kB/s-996kB/s), io=24.6MiB (25.8MB), run=6726-6830msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25026: Wed Dec 25 01:18:43 2024 read: IOPS=54, BW=878KiB/s (899kB/s)(5920KiB/6741msec) clat (usec): min=1886, max=193850, avg=7513.80, stdev=14338.48 lat (usec): min=1886, max=193851, avg=7514.43, stdev=14338.49 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2114], 10.00th=[ 2278], 20.00th=[ 2573], | 30.00th=[ 2900], 40.00th=[ 3294], 50.00th=[ 3851], 60.00th=[ 4948], | 70.00th=[ 7308], 80.00th=[ 10028], 90.00th=[ 13829], 95.00th=[ 20055], | 99.00th=[ 46924], 99.50th=[168821], 99.90th=[193987], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 640, max= 1088, per=24.94%, avg=868.77, stdev=136.97, samples=13 iops : min= 40, max= 68, avg=54.23, stdev= 8.58, samples=13 write: IOPS=59, BW=945KiB/s (967kB/s)(6368KiB/6741msec) clat (usec): min=2683, max=65766, avg=9916.97, stdev=6382.53 lat (usec): min=2684, max=65767, avg=9918.20, stdev=6382.54 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3523], 10.00th=[ 3884], 20.00th=[ 5145], | 30.00th=[ 5932], 40.00th=[ 7373], 50.00th=[ 8586], 60.00th=[ 9765], | 70.00th=[10945], 80.00th=[13435], 90.00th=[17695], 95.00th=[22414], | 99.00th=[32375], 99.50th=[36963], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 608, max= 1309, per=25.93%, avg=949.92, stdev=208.75, samples=13 iops : min= 38, max= 81, avg=59.31, stdev=12.93, samples=13 lat (msec) : 2=0.65%, 4=30.21%, 10=39.58%, 20=23.44%, 50=5.60% lat (msec) : 100=0.26%, 250=0.26% cpu : usr=0.18%, sys=8.86%, ctx=1189, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25028: Wed Dec 25 01:18:43 2024 read: IOPS=53, BW=851KiB/s (871kB/s)(5744KiB/6751msec) clat (usec): min=1755, max=196430, avg=7024.38, stdev=11267.92 lat (usec): min=1756, max=196430, avg=7025.03, stdev=11267.91 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2073], 10.00th=[ 2212], 20.00th=[ 2540], | 30.00th=[ 2966], 40.00th=[ 3523], 50.00th=[ 4293], 60.00th=[ 5735], | 70.00th=[ 7963], 80.00th=[ 9896], 90.00th=[ 14222], 95.00th=[ 18220], | 99.00th=[ 26346], 99.50th=[ 30540], 99.90th=[196084], 99.95th=[196084], | 99.99th=[196084] bw ( KiB/s): min= 448, max= 1184, per=24.66%, avg=858.92, stdev=191.01, samples=13 iops : min= 28, max= 74, avg=53.62, stdev=11.93, samples=13 write: IOPS=60, BW=969KiB/s (993kB/s)(6544KiB/6751msec) clat (usec): min=2563, max=56355, avg=9885.48, stdev=7426.04 lat (usec): min=2563, max=56356, avg=9886.66, stdev=7426.07 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3359], 10.00th=[ 3654], 20.00th=[ 4621], | 30.00th=[ 5735], 40.00th=[ 6849], 50.00th=[ 7963], 60.00th=[ 9372], | 70.00th=[10814], 80.00th=[13435], 90.00th=[17957], 95.00th=[22414], | 99.00th=[44303], 99.50th=[52691], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 480, max= 1376, per=26.26%, avg=962.31, stdev=255.15, samples=13 iops : min= 30, max= 86, avg=60.08, stdev=15.95, samples=13 lat (msec) : 2=1.30%, 4=28.39%, 10=42.45%, 20=22.66%, 50=4.69% lat (msec) : 100=0.39%, 250=0.13% cpu : usr=0.10%, sys=9.23%, ctx=1204, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25030: Wed Dec 25 01:18:43 2024 read: IOPS=56, BW=910KiB/s (932kB/s)(6256KiB/6876msec) clat (usec): min=1838, max=158050, avg=7240.03, stdev=9618.98 lat (usec): min=1839, max=158051, avg=7240.69, stdev=9618.99 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 13], 95.00th=[ 17], | 99.00th=[ 36], 99.50th=[ 53], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 480, max= 1312, per=26.14%, avg=910.54, stdev=245.42, samples=13 iops : min= 30, max= 82, avg=56.85, stdev=15.27, samples=13 write: IOPS=54, BW=877KiB/s (898kB/s)(6032KiB/6876msec) clat (msec): min=2, max=171, avg=10.65, stdev=10.74 lat (msec): min=2, max=171, avg=10.65, stdev=10.74 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 11], | 70.00th=[ 12], 80.00th=[ 14], 90.00th=[ 19], 95.00th=[ 23], | 99.00th=[ 46], 99.50th=[ 59], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 640, max= 992, per=23.38%, avg=856.46, stdev=128.59, samples=13 iops : min= 40, max= 62, avg=53.46, stdev= 7.96, samples=13 lat (msec) : 2=0.39%, 4=28.39%, 10=40.10%, 20=26.04%, 50=4.30% lat (msec) : 100=0.52%, 250=0.26% cpu : usr=0.09%, sys=8.87%, ctx=1247, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25032: Wed Dec 25 01:18:43 2024 read: IOPS=55, BW=884KiB/s (905kB/s)(6032KiB/6826msec) clat (usec): min=1791, max=177050, avg=7466.41, stdev=10312.24 lat (usec): min=1792, max=177051, avg=7467.05, stdev=10312.24 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 13], 95.00th=[ 17], | 99.00th=[ 32], 99.50th=[ 45], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 480, max= 1120, per=25.08%, avg=873.69, stdev=207.86, samples=13 iops : min= 30, max= 70, avg=54.54, stdev=12.95, samples=13 write: IOPS=57, BW=916KiB/s (938kB/s)(6256KiB/6826msec) clat (usec): min=2829, max=75734, avg=9787.54, stdev=6418.06 lat (usec): min=2831, max=75736, avg=9788.79, stdev=6418.10 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3523], 10.00th=[ 3851], 20.00th=[ 5014], | 30.00th=[ 6063], 40.00th=[ 7308], 50.00th=[ 8586], 60.00th=[ 9634], | 70.00th=[11207], 80.00th=[13829], 90.00th=[16581], 95.00th=[20055], | 99.00th=[33817], 99.50th=[41681], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 544, max= 1184, per=24.85%, avg=910.62, stdev=191.79, samples=13 iops : min= 34, max= 74, avg=56.85, stdev=11.97, samples=13 lat (msec) : 2=0.39%, 4=25.65%, 10=43.23%, 20=27.08%, 50=3.39% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.06%, sys=9.48%, ctx=1268, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3483KiB/s (3567kB/s), 851KiB/s-910KiB/s (871kB/s-932kB/s), io=23.4MiB (24.5MB), run=6741-6876msec WRITE: bw=3665KiB/s (3753kB/s), 877KiB/s-969KiB/s (898kB/s-993kB/s), io=24.6MiB (25.8MB), run=6741-6876msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25044: Wed Dec 25 01:18:46 2024 read: IOPS=2, BW=3023KiB/s (3096kB/s)(7168KiB/2371msec) clat (msec): min=18, max=783, avg=206.37, stdev=277.37 lat (msec): min=18, max=783, avg=206.37, stdev=277.37 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 20], | 30.00th=[ 64], 40.00th=[ 64], 50.00th=[ 75], 60.00th=[ 148], | 70.00th=[ 148], 80.00th=[ 338], 90.00th=[ 785], 95.00th=[ 785], | 99.00th=[ 785], 99.50th=[ 785], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 3368, max= 6119, per=54.92%, avg=4743.50, stdev=1945.25, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2159KiB/s (2211kB/s)(5120KiB/2371msec) clat (msec): min=24, max=713, avg=176.80, stdev=299.96 lat (msec): min=24, max=713, avg=176.88, stdev=299.95 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 53], 60.00th=[ 53], | 70.00th=[ 53], 80.00th=[ 53], 90.00th=[ 709], 95.00th=[ 709], | 99.00th=[ 709], 99.50th=[ 709], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 2048, max= 3368, per=22.39%, avg=2708.00, stdev=933.38, samples=2 iops : min= 2, max= 3, avg= 2.50, stdev= 0.71, samples=2 lat (msec) : 20=16.67%, 50=16.67%, 100=33.33%, 250=8.33%, 500=8.33% lat (msec) : 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=4.51%, ctx=71, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25045: Wed Dec 25 01:18:46 2024 read: IOPS=0, BW=867KiB/s (888kB/s)(2048KiB/2361msec) clat (usec): min=15875, max=32478, avg=24177.23, stdev=11740.24 lat (usec): min=15876, max=32480, avg=24178.49, stdev=11740.42 clat percentiles (usec): | 1.00th=[15926], 5.00th=[15926], 10.00th=[15926], 20.00th=[15926], | 30.00th=[15926], 40.00th=[15926], 50.00th=[15926], 60.00th=[32375], | 70.00th=[32375], 80.00th=[32375], 90.00th=[32375], 95.00th=[32375], | 99.00th=[32375], 99.50th=[32375], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 2035, max= 2048, per=23.64%, avg=2041.50, stdev= 9.19, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4337KiB/s (4441kB/s)(10.0MiB/2361msec) clat (msec): min=17, max=795, avg=225.13, stdev=278.80 lat (msec): min=17, max=795, avg=225.20, stdev=278.79 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 77], 60.00th=[ 88], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 659], 95.00th=[ 793], | 99.00th=[ 793], 99.50th=[ 793], 99.90th=[ 793], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 1681, max= 8143, per=33.01%, avg=3992.00, stdev=2964.31, samples=4 iops : min= 1, max= 7, avg= 3.50, stdev= 2.65, samples=4 lat (msec) : 20=16.67%, 50=33.33%, 100=16.67%, 250=8.33%, 500=8.33% lat (msec) : 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=5.59%, ctx=77, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25046: Wed Dec 25 01:18:46 2024 read: IOPS=2, BW=2723KiB/s (2789kB/s)(6144KiB/2256msec) clat (msec): min=18, max=705, avg=158.10, stdev=269.09 lat (msec): min=18, max=705, avg=158.11, stdev=269.09 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 59], 50.00th=[ 59], 60.00th=[ 64], | 70.00th=[ 74], 80.00th=[ 74], 90.00th=[ 709], 95.00th=[ 709], | 99.00th=[ 709], 99.50th=[ 709], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 2039, max= 3368, per=28.77%, avg=2485.00, stdev=764.71, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 write: IOPS=2, BW=2723KiB/s (2789kB/s)(6144KiB/2256msec) clat (msec): min=17, max=787, avg=209.88, stdev=305.00 lat (msec): min=17, max=787, avg=209.94, stdev=305.00 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 19], | 30.00th=[ 19], 40.00th=[ 25], 50.00th=[ 25], 60.00th=[ 97], | 70.00th=[ 317], 80.00th=[ 317], 90.00th=[ 785], 95.00th=[ 785], | 99.00th=[ 785], 99.50th=[ 785], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 3368, max= 6119, per=39.23%, avg=4743.50, stdev=1945.25, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 lat (msec) : 20=25.00%, 50=16.67%, 100=33.33%, 500=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=4.57%, ctx=55, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25047: Wed Dec 25 01:18:46 2024 read: IOPS=3, BW=3434KiB/s (3516kB/s)(5120KiB/1491msec) clat (usec): min=12884, max=77385, avg=32223.65, stdev=25707.10 lat (usec): min=12885, max=77386, avg=32224.83, stdev=25706.88 clat percentiles (usec): | 1.00th=[12911], 5.00th=[12911], 10.00th=[12911], 20.00th=[12911], | 30.00th=[21365], 40.00th=[21365], 50.00th=[24773], 60.00th=[24773], | 70.00th=[24773], 80.00th=[24773], 90.00th=[77071], 95.00th=[77071], | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 4071, max= 4087, per=47.23%, avg=4079.00, stdev=11.31, samples=2 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4808KiB/s (4923kB/s)(7168KiB/1491msec) clat (msec): min=21, max=737, avg=187.98, stdev=263.94 lat (msec): min=22, max=737, avg=188.05, stdev=263.93 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 33], | 30.00th=[ 53], 40.00th=[ 53], 50.00th=[ 55], 60.00th=[ 94], | 70.00th=[ 94], 80.00th=[ 326], 90.00th=[ 735], 95.00th=[ 735], | 99.00th=[ 735], 99.50th=[ 735], 99.90th=[ 735], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 2043, max= 8143, per=42.12%, avg=5093.00, stdev=4313.35, samples=2 iops : min= 1, max= 7, avg= 4.00, stdev= 4.24, samples=2 lat (msec) : 20=8.33%, 50=41.67%, 100=33.33%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=5.77%, ctx=113, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8638KiB/s (8845kB/s), 867KiB/s-3434KiB/s (888kB/s-3516kB/s), io=20.0MiB (20.0MB), run=1491-2371msec WRITE: bw=11.8MiB/s (12.4MB/s), 2159KiB/s-4808KiB/s (2211kB/s-4923kB/s), io=28.0MiB (29.4MB), run=1491-2371msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25048: Wed Dec 25 01:18:47 2024 read: IOPS=4, BW=4971KiB/s (5090kB/s)(7168KiB/1442msec) clat (usec): min=471, max=926611, avg=172090.21, stdev=340130.03 lat (usec): min=473, max=926613, avg=172091.88, stdev=340130.13 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 474], 10.00th=[ 474], 20.00th=[ 515], | 30.00th=[ 627], 40.00th=[ 627], 50.00th=[ 4015], 60.00th=[ 82314], | 70.00th=[ 82314], 80.00th=[189793], 90.00th=[926942], 95.00th=[926942], | 99.00th=[926942], 99.50th=[926942], 99.90th=[926942], 99.95th=[926942], | 99.99th=[926942] bw ( KiB/s): min= 6144, max= 6144, per=89.50%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3551KiB/s (3636kB/s)(5120KiB/1442msec) clat (usec): min=24959, max=63836, avg=45815.66, stdev=17710.73 lat (usec): min=24976, max=63847, avg=45845.08, stdev=17706.74 clat percentiles (usec): | 1.00th=[25035], 5.00th=[25035], 10.00th=[25035], 20.00th=[25035], | 30.00th=[31065], 40.00th=[31065], 50.00th=[46924], 60.00th=[46924], | 70.00th=[62129], 80.00th=[62129], 90.00th=[63701], 95.00th=[63701], | 99.00th=[63701], 99.50th=[63701], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 2048, max= 2048, per=21.31%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=16.67% lat (msec) : 10=8.33%, 50=25.00%, 100=25.00%, 250=8.33%, 1000=8.33% cpu : usr=0.07%, sys=72.24%, ctx=269, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25049: Wed Dec 25 01:18:47 2024 read: IOPS=0, BW=859KiB/s (880kB/s)(2048KiB/2384msec) clat (msec): min=95, max=207, avg=151.76, stdev=79.13 lat (msec): min=95, max=207, avg=151.76, stdev=79.13 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 96], 10.00th=[ 96], 20.00th=[ 96], | 30.00th=[ 96], 40.00th=[ 96], 50.00th=[ 96], 60.00th=[ 209], | 70.00th=[ 209], 80.00th=[ 209], 90.00th=[ 209], 95.00th=[ 209], | 99.00th=[ 209], 99.50th=[ 209], 99.90th=[ 209], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 2048, max= 2048, per=29.83%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4295KiB/s (4398kB/s)(10.0MiB/2384msec) clat (msec): min=28, max=743, avg=200.31, stdev=275.64 lat (msec): min=28, max=743, avg=200.37, stdev=275.63 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 45], 50.00th=[ 50], 60.00th=[ 71], | 70.00th=[ 205], 80.00th=[ 205], 90.00th=[ 684], 95.00th=[ 743], | 99.00th=[ 743], 99.50th=[ 743], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 2048, max= 6144, per=42.62%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (msec) : 50=41.67%, 100=16.67%, 250=25.00%, 750=16.67% cpu : usr=0.04%, sys=16.83%, ctx=207, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25050: Wed Dec 25 01:18:47 2024 read: IOPS=2, BW=2060KiB/s (2109kB/s)(6144KiB/2983msec) clat (usec): min=572, max=543973, avg=283666.18, stdev=239581.22 lat (usec): min=572, max=543975, avg=283667.54, stdev=239581.90 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 570], 10.00th=[ 570], 20.00th=[ 68682], | 30.00th=[ 68682], 40.00th=[141558], 50.00th=[141558], 60.00th=[463471], | 70.00th=[484443], 80.00th=[484443], 90.00th=[541066], 95.00th=[541066], | 99.00th=[541066], 99.50th=[541066], 99.90th=[541066], 99.95th=[541066], | 99.99th=[541066] bw ( KiB/s): min= 1689, max= 4096, per=38.03%, avg=2611.00, stdev=1298.51, samples=3 iops : min= 1, max= 4, avg= 2.33, stdev= 1.53, samples=3 write: IOPS=2, BW=2060KiB/s (2109kB/s)(6144KiB/2983msec) clat (msec): min=24, max=729, avg=200.69, stdev=281.34 lat (msec): min=24, max=729, avg=200.76, stdev=281.34 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 67], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 726], 95.00th=[ 726], | 99.00th=[ 726], 99.50th=[ 726], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 1689, max= 6144, per=34.27%, avg=3293.67, stdev=2474.98, samples=3 iops : min= 1, max= 6, avg= 3.00, stdev= 2.65, samples=3 lat (usec) : 750=8.33% lat (msec) : 50=25.00%, 100=16.67%, 250=8.33%, 500=25.00%, 750=16.67% cpu : usr=0.07%, sys=19.28%, ctx=982, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25051: Wed Dec 25 01:18:47 2024 read: IOPS=2, BW=2622KiB/s (2685kB/s)(5120KiB/1953msec) clat (usec): min=361, max=469256, avg=124528.58, stdev=197915.66 lat (usec): min=361, max=469257, avg=124529.54, stdev=197915.75 clat percentiles (usec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 363], | 30.00th=[ 412], 40.00th=[ 412], 50.00th=[ 41681], 60.00th=[ 41681], | 70.00th=[110625], 80.00th=[110625], 90.00th=[467665], 95.00th=[467665], | 99.00th=[467665], 99.50th=[467665], 99.90th=[467665], 99.95th=[467665], | 99.99th=[467665] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3670KiB/s (3758kB/s)(7168KiB/1953msec) clat (msec): min=25, max=726, avg=184.46, stdev=257.83 lat (msec): min=25, max=726, avg=184.52, stdev=257.82 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 33], | 30.00th=[ 48], 40.00th=[ 48], 50.00th=[ 65], 60.00th=[ 90], | 70.00th=[ 90], 80.00th=[ 305], 90.00th=[ 726], 95.00th=[ 726], | 99.00th=[ 726], 99.50th=[ 726], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 2048, max= 8192, per=49.72%, avg=4778.67, stdev=3128.37, samples=3 iops : min= 2, max= 8, avg= 4.67, stdev= 3.06, samples=3 lat (usec) : 500=16.67% lat (msec) : 50=33.33%, 100=16.67%, 250=8.33%, 500=16.67%, 750=8.33% cpu : usr=0.00%, sys=16.65%, ctx=452, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6866KiB/s (7030kB/s), 859KiB/s-4971KiB/s (880kB/s-5090kB/s), io=20.0MiB (20.0MB), run=1442-2983msec WRITE: bw=9612KiB/s (9842kB/s), 2060KiB/s-4295KiB/s (2109kB/s-4398kB/s), io=28.0MiB (29.4MB), run=1442-2983msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25059: Wed Dec 25 01:18:51 2024 read: IOPS=0, BW=3802KiB/s (3893kB/s)(12.0MiB/3232msec) clat (msec): min=138, max=2252, avg=1066.25, stdev=1080.30 lat (msec): min=138, max=2252, avg=1066.26, stdev=1080.30 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 140], 10.00th=[ 140], 20.00th=[ 140], | 30.00th=[ 140], 40.00th=[ 810], 50.00th=[ 810], 60.00th=[ 810], | 70.00th=[ 2265], 80.00th=[ 2265], 90.00th=[ 2265], 95.00th=[ 2265], | 99.00th=[ 2265], 99.50th=[ 2265], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.03%, sys=31.75%, ctx=1366, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25062: Wed Dec 25 01:18:51 2024 read: IOPS=0, BW=2296KiB/s (2351kB/s)(4096KiB/1784msec) clat (nsec): min=143278k, max=143278k, avg=143277823.00, stdev= 0.00 lat (nsec): min=143281k, max=143281k, avg=143280853.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 144], 50.00th=[ 144], 60.00th=[ 144], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4592KiB/s (4702kB/s)(8192KiB/1784msec) clat (msec): min=668, max=918, avg=793.35, stdev=176.29 lat (msec): min=669, max=918, avg=793.76, stdev=176.17 clat percentiles (msec): | 1.00th=[ 667], 5.00th=[ 667], 10.00th=[ 667], 20.00th=[ 667], | 30.00th=[ 667], 40.00th=[ 667], 50.00th=[ 667], 60.00th=[ 919], | 70.00th=[ 919], 80.00th=[ 919], 90.00th=[ 919], 95.00th=[ 919], | 99.00th=[ 919], 99.50th=[ 919], 99.90th=[ 919], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 8192, max= 8192, per=84.78%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=13.35%, ctx=206, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25063: Wed Dec 25 01:18:51 2024 write: IOPS=0, BW=3624KiB/s (3711kB/s)(12.0MiB/3391msec) clat (msec): min=796, max=1746, avg=1127.33, stdev=537.04 lat (msec): min=796, max=1747, avg=1127.78, stdev=536.95 clat percentiles (msec): | 1.00th=[ 793], 5.00th=[ 793], 10.00th=[ 793], 20.00th=[ 793], | 30.00th=[ 793], 40.00th=[ 835], 50.00th=[ 835], 60.00th=[ 835], | 70.00th=[ 1754], 80.00th=[ 1754], 90.00th=[ 1754], 95.00th=[ 1754], | 99.00th=[ 1754], 99.50th=[ 1754], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 5777, max= 8175, per=72.19%, avg=6976.00, stdev=1695.64, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 1000=66.67% cpu : usr=0.00%, sys=9.32%, ctx=58, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25065: Wed Dec 25 01:18:51 2024 write: IOPS=1, BW=7042KiB/s (7211kB/s)(12.0MiB/1745msec) clat (msec): min=208, max=817, avg=571.51, stdev=321.03 lat (msec): min=208, max=818, avg=571.86, stdev=321.08 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 209], 10.00th=[ 209], 20.00th=[ 209], | 30.00th=[ 209], 40.00th=[ 693], 50.00th=[ 693], 60.00th=[ 693], | 70.00th=[ 818], 80.00th=[ 818], 90.00th=[ 818], 95.00th=[ 818], | 99.00th=[ 818], 99.50th=[ 818], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 8192, max= 8192, per=84.78%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.06%, sys=17.89%, ctx=43, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5069KiB/s (5191kB/s), 2296KiB/s-3802KiB/s (2351kB/s-3893kB/s), io=16.0MiB (16.8MB), run=1784-3232msec WRITE: bw=9663KiB/s (9895kB/s), 3624KiB/s-7042KiB/s (3711kB/s-7211kB/s), io=32.0MiB (33.6MB), run=1745-3391msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25060: Wed Dec 25 01:18:52 2024 read: IOPS=0, BW=3756KiB/s (3846kB/s)(12.0MiB/3272msec) clat (msec): min=157, max=2252, avg=1072.68, stdev=1072.59 lat (msec): min=157, max=2252, avg=1072.68, stdev=1072.58 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 810], 50.00th=[ 810], 60.00th=[ 810], | 70.00th=[ 2265], 80.00th=[ 2265], 90.00th=[ 2265], 95.00th=[ 2265], | 99.00th=[ 2265], 99.50th=[ 2265], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=2.32%, ctx=35, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25061: Wed Dec 25 01:18:52 2024 read: IOPS=0, BW=2059KiB/s (2109kB/s)(4096KiB/1989msec) clat (nsec): min=191636k, max=191636k, avg=191635972.00, stdev= 0.00 lat (nsec): min=191637k, max=191637k, avg=191637252.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 192], 10.00th=[ 192], 20.00th=[ 192], | 30.00th=[ 192], 40.00th=[ 192], 50.00th=[ 192], 60.00th=[ 192], | 70.00th=[ 192], 80.00th=[ 192], 90.00th=[ 192], 95.00th=[ 192], | 99.00th=[ 192], 99.50th=[ 192], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4119KiB/s (4218kB/s)(8192KiB/1989msec) clat (msec): min=504, max=1265, avg=884.66, stdev=538.32 lat (msec): min=504, max=1265, avg=884.98, stdev=538.32 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 506], 10.00th=[ 506], 20.00th=[ 506], | 30.00th=[ 506], 40.00th=[ 506], 50.00th=[ 506], 60.00th=[ 1267], | 70.00th=[ 1267], 80.00th=[ 1267], 90.00th=[ 1267], 95.00th=[ 1267], | 99.00th=[ 1267], 99.50th=[ 1267], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 8192, max= 8192, per=87.31%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.05%, sys=3.72%, ctx=44, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25064: Wed Dec 25 01:18:52 2024 write: IOPS=0, BW=3519KiB/s (3603kB/s)(12.0MiB/3492msec) clat (msec): min=916, max=1572, avg=1154.77, stdev=363.20 lat (msec): min=916, max=1573, avg=1155.11, stdev=363.16 clat percentiles (msec): | 1.00th=[ 919], 5.00th=[ 919], 10.00th=[ 919], 20.00th=[ 919], | 30.00th=[ 919], 40.00th=[ 978], 50.00th=[ 978], 60.00th=[ 978], | 70.00th=[ 1569], 80.00th=[ 1569], 90.00th=[ 1569], 95.00th=[ 1569], | 99.00th=[ 1569], 99.50th=[ 1569], 99.90th=[ 1569], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 5902, max= 8192, per=75.10%, avg=7047.00, stdev=1619.27, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 1000=66.67% cpu : usr=0.03%, sys=5.93%, ctx=88, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25066: Wed Dec 25 01:18:52 2024 write: IOPS=1, BW=7443KiB/s (7621kB/s)(12.0MiB/1651msec) clat (msec): min=81, max=1450, avg=546.57, stdev=783.30 lat (msec): min=81, max=1451, avg=546.82, stdev=783.18 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 82], 10.00th=[ 82], 20.00th=[ 82], | 30.00th=[ 82], 40.00th=[ 108], 50.00th=[ 108], 60.00th=[ 108], | 70.00th=[ 1452], 80.00th=[ 1452], 90.00th=[ 1452], 95.00th=[ 1452], | 99.00th=[ 1452], 99.50th=[ 1452], 99.90th=[ 1452], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.18%, sys=7.82%, ctx=82, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5007KiB/s (5128kB/s), 2059KiB/s-3756KiB/s (2109kB/s-3846kB/s), io=16.0MiB (16.8MB), run=1989-3272msec WRITE: bw=9384KiB/s (9609kB/s), 3519KiB/s-7443KiB/s (3603kB/s-7621kB/s), io=32.0MiB (33.6MB), run=1651-3492msec