== sanity test 398b: DIO and buffer IO race ============== 19:56:44 (1718323004) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.1351 s, 44.3 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=19038: Thu Jun 13 19:56:59 2024 read: IOPS=119, BW=479KiB/s (490kB/s)(5992KiB/12519msec) clat (usec): min=47, max=196433, avg=4173.23, stdev=5975.46 lat (usec): min=48, max=196435, avg=4175.20, stdev=5975.34 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 61], 10.00th=[ 75], 20.00th=[ 1811], | 30.00th=[ 2507], 40.00th=[ 2966], 50.00th=[ 3392], 60.00th=[ 4047], | 70.00th=[ 5145], 80.00th=[ 6390], 90.00th=[ 8160], 95.00th=[ 9634], | 99.00th=[ 15401], 99.50th=[ 21627], 99.90th=[ 26084], 99.95th=[196084], | 99.99th=[196084] bw ( KiB/s): min= 200, max= 752, per=25.21%, avg=478.96, stdev=111.60, samples=25 iops : min= 50, max= 188, avg=119.68, stdev=27.92, samples=25 write: IOPS=125, BW=503KiB/s (515kB/s)(6296KiB/12519msec) clat (usec): min=422, max=32410, avg=3922.46, stdev=3836.33 lat (usec): min=422, max=32411, avg=3924.95, stdev=3836.04 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 553], 10.00th=[ 603], 20.00th=[ 717], | 30.00th=[ 898], 40.00th=[ 1352], 50.00th=[ 2933], 60.00th=[ 4015], | 70.00th=[ 5211], 80.00th=[ 6587], 90.00th=[ 9241], 95.00th=[11207], | 99.00th=[16909], 99.50th=[19268], 99.90th=[22676], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 232, max= 760, per=25.25%, avg=502.24, stdev=115.66, samples=25 iops : min= 58, max= 190, avg=125.52, stdev=28.88, samples=25 lat (usec) : 50=0.29%, 100=7.13%, 250=1.89%, 500=0.62%, 750=10.97% lat (usec) : 1000=5.89% lat (msec) : 2=7.26%, 4=25.29%, 10=34.83%, 20=5.31%, 50=0.49% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=26.41%, ctx=6277, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19039: Thu Jun 13 19:56:59 2024 read: IOPS=119, BW=478KiB/s (489kB/s)(5940KiB/12434msec) clat (usec): min=48, max=21231, avg=4026.13, stdev=3104.93 lat (usec): min=49, max=21232, avg=4027.10, stdev=3104.70 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 67], 10.00th=[ 90], 20.00th=[ 2089], | 30.00th=[ 2573], 40.00th=[ 2933], 50.00th=[ 3294], 60.00th=[ 3982], | 70.00th=[ 4948], 80.00th=[ 6259], 90.00th=[ 8291], 95.00th=[ 9765], | 99.00th=[14222], 99.50th=[17171], 99.90th=[19530], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 256, max= 624, per=25.15%, avg=477.88, stdev=106.64, samples=24 iops : min= 64, max= 156, avg=119.42, stdev=26.64, samples=24 write: IOPS=127, BW=511KiB/s (523kB/s)(6348KiB/12434msec) clat (usec): min=434, max=167520, avg=4008.51, stdev=5930.64 lat (usec): min=435, max=167521, avg=4009.83, stdev=5930.96 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 570], 10.00th=[ 619], 20.00th=[ 742], | 30.00th=[ 889], 40.00th=[ 1172], 50.00th=[ 2278], 60.00th=[ 3884], | 70.00th=[ 5080], 80.00th=[ 6718], 90.00th=[ 9241], 95.00th=[ 11338], | 99.00th=[ 18482], 99.50th=[ 26870], 99.90th=[ 46400], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 296, max= 694, per=25.31%, avg=503.50, stdev=98.84, samples=24 iops : min= 74, max= 173, avg=125.83, stdev=24.67, samples=24 lat (usec) : 50=0.16%, 100=6.05%, 250=2.08%, 500=0.49%, 750=10.32% lat (usec) : 1000=7.68% lat (msec) : 2=7.19%, 4=26.69%, 10=33.14%, 20=5.79%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=26.33%, ctx=6435, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19040: Thu Jun 13 19:56:59 2024 read: IOPS=119, BW=477KiB/s (488kB/s)(6028KiB/12637msec) clat (usec): min=48, max=52233, avg=4078.40, stdev=3687.67 lat (usec): min=48, max=52233, avg=4079.28, stdev=3687.75 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 60], 10.00th=[ 72], 20.00th=[ 121], | 30.00th=[ 2409], 40.00th=[ 2835], 50.00th=[ 3326], 60.00th=[ 4178], | 70.00th=[ 5342], 80.00th=[ 6783], 90.00th=[ 8225], 95.00th=[ 9896], | 99.00th=[14091], 99.50th=[17171], 99.90th=[40633], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 160, max= 702, per=24.90%, avg=473.16, stdev=106.79, samples=25 iops : min= 40, max= 175, avg=118.24, stdev=26.65, samples=25 write: IOPS=123, BW=495KiB/s (507kB/s)(6260KiB/12637msec) clat (usec): min=416, max=188041, avg=4087.07, stdev=6275.68 lat (usec): min=417, max=188043, avg=4088.31, stdev=6275.69 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 553], 10.00th=[ 603], 20.00th=[ 734], | 30.00th=[ 898], 40.00th=[ 1205], 50.00th=[ 2835], 60.00th=[ 3949], | 70.00th=[ 5080], 80.00th=[ 6718], 90.00th=[ 9372], 95.00th=[ 11469], | 99.00th=[ 18220], 99.50th=[ 26608], 99.90th=[ 40633], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 192, max= 656, per=24.67%, avg=490.76, stdev=116.06, samples=25 iops : min= 48, max= 164, avg=122.64, stdev=29.01, samples=25 lat (usec) : 50=0.29%, 100=8.24%, 250=1.99%, 500=0.78%, 750=10.09% lat (usec) : 1000=6.67% lat (msec) : 2=7.29%, 4=23.93%, 10=34.41%, 20=5.73%, 50=0.52% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.20%, sys=26.04%, ctx=6365, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19041: Thu Jun 13 19:56:59 2024 read: IOPS=119, BW=480KiB/s (491kB/s)(6052KiB/12610msec) clat (usec): min=47, max=29756, avg=3899.61, stdev=3297.02 lat (usec): min=47, max=29757, avg=3900.28, stdev=3297.01 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 60], 10.00th=[ 73], 20.00th=[ 206], | 30.00th=[ 2442], 40.00th=[ 2868], 50.00th=[ 3261], 60.00th=[ 3785], | 70.00th=[ 4752], 80.00th=[ 6259], 90.00th=[ 8029], 95.00th=[ 9503], | 99.00th=[13829], 99.50th=[16909], 99.90th=[27919], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 280, max= 672, per=25.06%, avg=476.08, stdev=93.46, samples=25 iops : min= 70, max= 168, avg=119.00, stdev=23.36, samples=25 write: IOPS=123, BW=495KiB/s (506kB/s)(6236KiB/12610msec) clat (usec): min=434, max=169822, avg=4243.16, stdev=6059.95 lat (usec): min=435, max=169823, avg=4244.53, stdev=6059.73 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 553], 10.00th=[ 603], 20.00th=[ 717], | 30.00th=[ 906], 40.00th=[ 1303], 50.00th=[ 2900], 60.00th=[ 4228], | 70.00th=[ 5342], 80.00th=[ 7111], 90.00th=[ 9634], 95.00th=[ 11994], | 99.00th=[ 18220], 99.50th=[ 27132], 99.90th=[ 42206], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 328, max= 784, per=24.74%, avg=492.08, stdev=98.19, samples=25 iops : min= 82, max= 196, avg=123.00, stdev=24.55, samples=25 lat (usec) : 50=0.33%, 100=7.78%, 250=1.82%, 500=0.59%, 750=10.74% lat (usec) : 1000=6.12% lat (msec) : 2=6.84%, 4=26.01%, 10=33.07%, 20=6.02%, 50=0.65% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=25.40%, ctx=6433, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1900KiB/s (1946kB/s), 477KiB/s-480KiB/s (488kB/s-491kB/s), io=23.4MiB (24.6MB), run=12434-12637msec WRITE: bw=1989KiB/s (2037kB/s), 495KiB/s-511KiB/s (506kB/s-523kB/s), io=24.6MiB (25.7MB), run=12434-12637msec rand-rw: (groupid=0, jobs=1): err= 0: pid=19034: Thu Jun 13 19:57:27 2024 read: IOPS=37, BW=150KiB/s (154kB/s)(5992KiB/39872msec) clat (usec): min=1565, max=215931, avg=4254.24, stdev=6189.98 lat (usec): min=1565, max=215932, avg=4255.00, stdev=6190.00 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1860], 10.00th=[ 1942], 20.00th=[ 2114], | 30.00th=[ 2311], 40.00th=[ 2737], 50.00th=[ 3621], 60.00th=[ 3916], | 70.00th=[ 4293], 80.00th=[ 5014], 90.00th=[ 7308], 95.00th=[ 9372], | 99.00th=[ 16188], 99.50th=[ 17957], 99.90th=[ 35390], 99.95th=[214959], | 99.99th=[214959] bw ( KiB/s): min= 48, max= 264, per=24.99%, avg=149.95, stdev=53.67, samples=79 iops : min= 12, max= 66, avg=37.39, stdev=13.48, samples=79 write: IOPS=39, BW=158KiB/s (162kB/s)(6296KiB/39872msec) clat (usec): min=10969, max=62099, avg=21211.91, stdev=7527.19 lat (usec): min=10970, max=62099, avg=21212.82, stdev=7527.20 clat percentiles (usec): | 1.00th=[11731], 5.00th=[12911], 10.00th=[13566], 20.00th=[15008], | 30.00th=[16581], 40.00th=[18220], 50.00th=[19792], 60.00th=[21103], | 70.00th=[22938], 80.00th=[25297], 90.00th=[32113], 95.00th=[37487], | 99.00th=[46400], 99.50th=[47973], 99.90th=[57410], 99.95th=[62129], | 99.99th=[62129] bw ( KiB/s): min= 48, max= 216, per=25.09%, avg=157.54, stdev=42.42, samples=79 iops : min= 12, max= 54, avg=39.29, stdev=10.66, samples=79 lat (msec) : 2=6.25%, 4=24.06%, 10=16.37%, 20=28.68%, 50=24.45% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.08%, sys=5.14%, ctx=3902, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19035: Thu Jun 13 19:57:27 2024 read: IOPS=37, BW=149KiB/s (152kB/s)(5940KiB/39998msec) clat (usec): min=1407, max=172007, avg=4406.66, stdev=5547.07 lat (usec): min=1407, max=172008, avg=4407.39, stdev=5547.08 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1876], 10.00th=[ 1991], 20.00th=[ 2147], | 30.00th=[ 2343], 40.00th=[ 2737], 50.00th=[ 3621], 60.00th=[ 3982], | 70.00th=[ 4293], 80.00th=[ 5014], 90.00th=[ 7832], 95.00th=[ 11600], | 99.00th=[ 17433], 99.50th=[ 20317], 99.90th=[ 51119], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 16, max= 288, per=24.70%, avg=148.22, stdev=58.75, samples=79 iops : min= 4, max= 72, avg=36.95, stdev=14.75, samples=79 write: IOPS=39, BW=159KiB/s (163kB/s)(6348KiB/39998msec) clat (usec): min=10804, max=72338, avg=21009.81, stdev=7420.47 lat (usec): min=10805, max=72339, avg=21010.78, stdev=7420.46 clat percentiles (usec): | 1.00th=[11863], 5.00th=[12780], 10.00th=[13566], 20.00th=[15270], | 30.00th=[16319], 40.00th=[17957], 50.00th=[19530], 60.00th=[21103], | 70.00th=[22676], 80.00th=[25035], 90.00th=[30802], 95.00th=[36963], | 99.00th=[46400], 99.50th=[49546], 99.90th=[53740], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 48, max= 216, per=25.10%, avg=157.63, stdev=42.74, samples=79 iops : min= 12, max= 54, avg=39.30, stdev=10.73, samples=79 lat (msec) : 2=5.27%, 4=23.99%, 10=15.95%, 20=30.11%, 50=24.41% lat (msec) : 100=0.23%, 250=0.03% cpu : usr=0.09%, sys=5.18%, ctx=3926, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19036: Thu Jun 13 19:57:27 2024 read: IOPS=37, BW=151KiB/s (155kB/s)(6028KiB/39949msec) clat (usec): min=1481, max=33127, avg=4325.71, stdev=3197.36 lat (usec): min=1482, max=33128, avg=4326.48, stdev=3197.36 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1893], 10.00th=[ 1991], 20.00th=[ 2180], | 30.00th=[ 2409], 40.00th=[ 3064], 50.00th=[ 3687], 60.00th=[ 4015], | 70.00th=[ 4424], 80.00th=[ 5276], 90.00th=[ 7373], 95.00th=[10683], | 99.00th=[17695], 99.50th=[21890], 99.90th=[27919], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 40, max= 304, per=25.11%, avg=150.66, stdev=54.35, samples=79 iops : min= 10, max= 76, avg=37.57, stdev=13.65, samples=79 write: IOPS=39, BW=157KiB/s (160kB/s)(6260KiB/39949msec) clat (msec): min=11, max=185, avg=21.29, stdev= 8.52 lat (msec): min=11, max=185, avg=21.29, stdev= 8.52 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 20], 60.00th=[ 22], | 70.00th=[ 23], 80.00th=[ 26], 90.00th=[ 32], 95.00th=[ 37], | 99.00th=[ 48], 99.50th=[ 52], 99.90th=[ 69], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 48, max= 216, per=24.84%, avg=156.03, stdev=42.31, samples=79 iops : min= 12, max= 54, avg=38.91, stdev=10.63, samples=79 lat (msec) : 2=5.14%, 4=24.12%, 10=17.06%, 20=29.07%, 50=24.28% lat (msec) : 100=0.29%, 250=0.03% cpu : usr=0.06%, sys=5.26%, ctx=3943, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19037: Thu Jun 13 19:57:27 2024 read: IOPS=38, BW=153KiB/s (157kB/s)(6052KiB/39502msec) clat (usec): min=1540, max=35335, avg=4277.37, stdev=3329.22 lat (usec): min=1541, max=35336, avg=4278.13, stdev=3329.25 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1876], 10.00th=[ 1975], 20.00th=[ 2147], | 30.00th=[ 2376], 40.00th=[ 2933], 50.00th=[ 3654], 60.00th=[ 3949], | 70.00th=[ 4359], 80.00th=[ 5080], 90.00th=[ 7242], 95.00th=[10552], | 99.00th=[17171], 99.50th=[25560], 99.90th=[34341], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 40, max= 304, per=25.51%, avg=153.09, stdev=61.32, samples=79 iops : min= 10, max= 76, avg=38.18, stdev=15.37, samples=79 write: IOPS=39, BW=158KiB/s (162kB/s)(6236KiB/39502msec) clat (msec): min=10, max=194, avg=21.11, stdev= 8.63 lat (msec): min=10, max=194, avg=21.12, stdev= 8.63 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 22], | 70.00th=[ 23], 80.00th=[ 26], 90.00th=[ 32], 95.00th=[ 37], | 99.00th=[ 47], 99.50th=[ 50], 99.90th=[ 67], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 56, max= 240, per=25.10%, avg=157.65, stdev=43.09, samples=79 iops : min= 14, max= 60, avg=39.32, stdev=10.84, samples=79 lat (msec) : 2=5.44%, 4=24.71%, 10=16.24%, 20=29.00%, 50=24.38% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.11%, sys=5.17%, ctx=3949, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=600KiB/s (615kB/s), 149KiB/s-153KiB/s (152kB/s-157kB/s), io=23.4MiB (24.6MB), run=39502-39998msec WRITE: bw=629KiB/s (644kB/s), 157KiB/s-159KiB/s (160kB/s-163kB/s), io=24.6MiB (25.7MB), run=39502-39998msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=19095: Thu Jun 13 19:57:33 2024 read: IOPS=64, BW=1033KiB/s (1057kB/s)(5920KiB/5733msec) clat (usec): min=52, max=34007, avg=11207.76, stdev=5885.25 lat (usec): min=52, max=34008, avg=11208.48, stdev=5885.26 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 76], 10.00th=[ 103], 20.00th=[ 8979], | 30.00th=[10028], 40.00th=[11076], 50.00th=[11731], 60.00th=[12649], | 70.00th=[13698], 80.00th=[15008], 90.00th=[16909], 95.00th=[19268], | 99.00th=[28967], 99.50th=[30802], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 768, max= 1280, per=24.45%, avg=1012.18, stdev=157.05, samples=11 iops : min= 48, max= 80, avg=63.18, stdev= 9.87, samples=11 write: IOPS=69, BW=1111KiB/s (1137kB/s)(6368KiB/5733msec) clat (usec): min=667, max=32938, avg=3914.30, stdev=3956.57 lat (usec): min=668, max=32940, avg=3915.52, stdev=3956.59 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 807], 10.00th=[ 848], 20.00th=[ 988], | 30.00th=[ 1237], 40.00th=[ 1467], 50.00th=[ 2737], 60.00th=[ 3490], | 70.00th=[ 4621], 80.00th=[ 6194], 90.00th=[ 8979], 95.00th=[11076], | 99.00th=[18220], 99.50th=[24511], 99.90th=[32900], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 736, max= 1408, per=24.90%, avg=1084.82, stdev=225.40, samples=11 iops : min= 46, max= 88, avg=67.73, stdev=14.04, samples=11 lat (usec) : 100=4.04%, 250=3.26%, 750=0.91%, 1000=9.51% lat (msec) : 2=12.89%, 4=10.68%, 10=21.35%, 20=34.90%, 50=2.47% cpu : usr=0.16%, sys=23.46%, ctx=2422, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19096: Thu Jun 13 19:57:33 2024 read: IOPS=62, BW=1002KiB/s (1026kB/s)(5744KiB/5732msec) clat (usec): min=54, max=30552, avg=10532.67, stdev=6357.10 lat (usec): min=54, max=30552, avg=10533.58, stdev=6357.14 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 67], 10.00th=[ 90], 20.00th=[ 139], | 30.00th=[ 9765], 40.00th=[11076], 50.00th=[12125], 60.00th=[12911], | 70.00th=[13829], 80.00th=[15008], 90.00th=[16581], 95.00th=[19268], | 99.00th=[23200], 99.50th=[28967], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 352, max= 1344, per=23.60%, avg=977.18, stdev=278.62, samples=11 iops : min= 22, max= 84, avg=61.00, stdev=17.38, samples=11 write: IOPS=71, BW=1142KiB/s (1169kB/s)(6544KiB/5732msec) clat (usec): min=630, max=41268, avg=4703.34, stdev=5818.69 lat (usec): min=631, max=41270, avg=4704.51, stdev=5818.73 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 807], 10.00th=[ 873], 20.00th=[ 1090], | 30.00th=[ 1254], 40.00th=[ 1565], 50.00th=[ 2802], 60.00th=[ 3589], | 70.00th=[ 4752], 80.00th=[ 6390], 90.00th=[10552], 95.00th=[18744], | 99.00th=[27919], 99.50th=[29492], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 608, max= 1692, per=25.44%, avg=1108.00, stdev=308.26, samples=11 iops : min= 38, max= 105, avg=69.18, stdev=19.12, samples=11 lat (usec) : 100=5.99%, 250=4.43%, 750=1.17%, 1000=7.03% lat (msec) : 2=14.45%, 4=10.81%, 10=18.10%, 20=34.11%, 50=3.91% cpu : usr=0.14%, sys=21.65%, ctx=2210, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19097: Thu Jun 13 19:57:33 2024 read: IOPS=67, BW=1081KiB/s (1107kB/s)(6256KiB/5785msec) clat (usec): min=52, max=33834, avg=10907.96, stdev=6095.73 lat (usec): min=52, max=33835, avg=10908.89, stdev=6095.90 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 75], 10.00th=[ 96], 20.00th=[ 8160], | 30.00th=[10290], 40.00th=[11207], 50.00th=[12125], 60.00th=[12780], | 70.00th=[13698], 80.00th=[14615], 90.00th=[17433], 95.00th=[19792], | 99.00th=[26608], 99.50th=[31589], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 736, max= 1536, per=25.29%, avg=1047.00, stdev=227.86, samples=11 iops : min= 46, max= 96, avg=65.36, stdev=14.17, samples=11 write: IOPS=65, BW=1043KiB/s (1068kB/s)(6032KiB/5785msec) clat (usec): min=657, max=28988, avg=3961.10, stdev=4065.32 lat (usec): min=657, max=28990, avg=3962.28, stdev=4065.35 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 807], 10.00th=[ 848], 20.00th=[ 1004], | 30.00th=[ 1270], 40.00th=[ 1532], 50.00th=[ 2900], 60.00th=[ 3818], | 70.00th=[ 4686], 80.00th=[ 6128], 90.00th=[ 8094], 95.00th=[10814], | 99.00th=[21103], 99.50th=[28443], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 512, max= 1437, per=22.70%, avg=988.82, stdev=271.20, samples=11 iops : min= 32, max= 89, avg=61.73, stdev=16.82, samples=11 lat (usec) : 100=6.25%, 250=2.99%, 500=0.13%, 750=1.56%, 1000=8.33% lat (msec) : 2=11.59%, 4=9.51%, 10=20.44%, 20=36.59%, 50=2.60% cpu : usr=0.10%, sys=22.18%, ctx=2452, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19098: Thu Jun 13 19:57:33 2024 read: IOPS=67, BW=1078KiB/s (1104kB/s)(6032KiB/5593msec) clat (usec): min=51, max=42133, avg=10316.25, stdev=6703.54 lat (usec): min=52, max=42134, avg=10317.97, stdev=6704.20 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 69], 10.00th=[ 81], 20.00th=[ 128], | 30.00th=[ 9503], 40.00th=[10683], 50.00th=[11469], 60.00th=[12387], | 70.00th=[13698], 80.00th=[15139], 90.00th=[16712], 95.00th=[19530], | 99.00th=[30016], 99.50th=[42206], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 512, max= 1600, per=25.50%, avg=1055.73, stdev=264.09, samples=11 iops : min= 32, max= 100, avg=65.91, stdev=16.46, samples=11 write: IOPS=69, BW=1119KiB/s (1145kB/s)(6256KiB/5593msec) clat (usec): min=656, max=33173, avg=4283.33, stdev=5038.99 lat (usec): min=657, max=33175, avg=4284.70, stdev=5039.05 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 807], 10.00th=[ 898], 20.00th=[ 1029], | 30.00th=[ 1237], 40.00th=[ 1401], 50.00th=[ 2573], 60.00th=[ 3556], | 70.00th=[ 4686], 80.00th=[ 6521], 90.00th=[ 9110], 95.00th=[15008], | 99.00th=[27657], 99.50th=[31851], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 640, max= 1600, per=25.44%, avg=1108.09, stdev=289.22, samples=11 iops : min= 40, max= 100, avg=69.18, stdev=18.09, samples=11 lat (usec) : 100=7.81%, 250=3.39%, 750=1.04%, 1000=8.33% lat (msec) : 2=14.97%, 4=8.72%, 10=19.14%, 20=33.59%, 50=2.99% cpu : usr=0.29%, sys=22.01%, ctx=2296, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4140KiB/s (4240kB/s), 1002KiB/s-1081KiB/s (1026kB/s-1107kB/s), io=23.4MiB (24.5MB), run=5593-5785msec WRITE: bw=4356KiB/s (4461kB/s), 1043KiB/s-1142KiB/s (1068kB/s-1169kB/s), io=24.6MiB (25.8MB), run=5593-5785msec rand-rw: (groupid=0, jobs=1): err= 0: pid=19089: Thu Jun 13 19:57:38 2024 read: IOPS=34, BW=554KiB/s (567kB/s)(5920KiB/10692msec) clat (usec): min=1840, max=34938, avg=4492.54, stdev=4051.69 lat (usec): min=1841, max=34939, avg=4493.49, stdev=4052.03 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2089], 10.00th=[ 2212], 20.00th=[ 2343], | 30.00th=[ 2409], 40.00th=[ 2638], 50.00th=[ 2868], 60.00th=[ 3294], | 70.00th=[ 4047], 80.00th=[ 5800], 90.00th=[ 8979], 95.00th=[11994], | 99.00th=[22676], 99.50th=[27919], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 256, max= 992, per=24.42%, avg=545.81, stdev=177.53, samples=21 iops : min= 16, max= 62, avg=34.10, stdev=11.07, samples=21 write: IOPS=37, BW=596KiB/s (610kB/s)(6368KiB/10692msec) clat (usec): min=10813, max=58634, avg=22668.80, stdev=7477.96 lat (usec): min=10814, max=58636, avg=22670.31, stdev=7478.02 clat percentiles (usec): | 1.00th=[11863], 5.00th=[13435], 10.00th=[14222], 20.00th=[16057], | 30.00th=[17695], 40.00th=[20055], 50.00th=[21627], 60.00th=[23200], | 70.00th=[25035], 80.00th=[28443], 90.00th=[32375], 95.00th=[35914], | 99.00th=[49546], 99.50th=[52167], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 384, max= 864, per=25.15%, avg=591.24, stdev=163.05, samples=21 iops : min= 24, max= 54, avg=36.95, stdev=10.19, samples=21 lat (msec) : 2=0.65%, 4=32.81%, 10=11.33%, 20=22.92%, 50=31.90% lat (msec) : 100=0.39% cpu : usr=0.14%, sys=5.77%, ctx=864, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19090: Thu Jun 13 19:57:38 2024 read: IOPS=33, BW=536KiB/s (549kB/s)(5744KiB/10716msec) clat (usec): min=1904, max=28874, avg=4318.39, stdev=3770.67 lat (usec): min=1904, max=28874, avg=4319.13, stdev=3770.69 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2073], 10.00th=[ 2147], 20.00th=[ 2311], | 30.00th=[ 2474], 40.00th=[ 2606], 50.00th=[ 2802], 60.00th=[ 3097], | 70.00th=[ 3589], 80.00th=[ 5538], 90.00th=[ 9110], 95.00th=[12911], | 99.00th=[19268], 99.50th=[28443], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 224, max= 1024, per=23.80%, avg=531.90, stdev=222.81, samples=21 iops : min= 14, max= 64, avg=33.19, stdev=13.87, samples=21 write: IOPS=38, BW=611KiB/s (625kB/s)(6544KiB/10716msec) clat (usec): min=11159, max=57260, avg=22391.57, stdev=7378.17 lat (usec): min=11160, max=57261, avg=22393.45, stdev=7378.90 clat percentiles (usec): | 1.00th=[11863], 5.00th=[13042], 10.00th=[13698], 20.00th=[15664], | 30.00th=[18220], 40.00th=[19530], 50.00th=[20841], 60.00th=[22938], | 70.00th=[24773], 80.00th=[28705], 90.00th=[32637], 95.00th=[36439], | 99.00th=[42730], 99.50th=[44827], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 416, max= 800, per=25.68%, avg=603.67, stdev=149.47, samples=21 iops : min= 26, max= 50, avg=37.67, stdev= 9.28, samples=21 lat (msec) : 2=0.65%, 4=33.98%, 10=8.07%, 20=26.56%, 50=30.60% lat (msec) : 100=0.13% cpu : usr=0.07%, sys=5.77%, ctx=864, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19091: Thu Jun 13 19:57:38 2024 read: IOPS=37, BW=600KiB/s (615kB/s)(6256KiB/10422msec) clat (usec): min=1786, max=38124, avg=4765.79, stdev=4886.07 lat (usec): min=1787, max=38124, avg=4766.56, stdev=4886.09 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2057], 10.00th=[ 2180], 20.00th=[ 2278], | 30.00th=[ 2409], 40.00th=[ 2540], 50.00th=[ 2704], 60.00th=[ 3032], | 70.00th=[ 3785], 80.00th=[ 5997], 90.00th=[10683], 95.00th=[13173], | 99.00th=[28443], 99.50th=[34866], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 288, max= 1216, per=27.05%, avg=604.65, stdev=258.32, samples=20 iops : min= 18, max= 76, avg=37.70, stdev=16.19, samples=20 write: IOPS=36, BW=579KiB/s (593kB/s)(6032KiB/10422msec) clat (usec): min=12099, max=55880, avg=22680.51, stdev=7523.74 lat (usec): min=12100, max=55881, avg=22682.03, stdev=7523.77 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13435], 10.00th=[14353], 20.00th=[16319], | 30.00th=[18482], 40.00th=[20055], 50.00th=[21365], 60.00th=[22676], | 70.00th=[24511], 80.00th=[28181], 90.00th=[33424], 95.00th=[36963], | 99.00th=[45876], 99.50th=[51643], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 351, max= 832, per=24.09%, avg=566.25, stdev=159.95, samples=20 iops : min= 21, max= 52, avg=35.30, stdev=10.07, samples=20 lat (msec) : 2=1.69%, 4=34.51%, 10=9.24%, 20=24.09%, 50=30.21% lat (msec) : 100=0.26% cpu : usr=0.06%, sys=5.94%, ctx=883, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19092: Thu Jun 13 19:57:38 2024 read: IOPS=35, BW=573KiB/s (586kB/s)(6032KiB/10532msec) clat (usec): min=1898, max=29089, avg=4207.16, stdev=3379.22 lat (usec): min=1898, max=29090, avg=4208.11, stdev=3379.35 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2089], 10.00th=[ 2180], 20.00th=[ 2311], | 30.00th=[ 2474], 40.00th=[ 2573], 50.00th=[ 2802], 60.00th=[ 3163], | 70.00th=[ 3884], 80.00th=[ 5342], 90.00th=[ 8160], 95.00th=[11469], | 99.00th=[19268], 99.50th=[20841], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 288, max= 1056, per=25.43%, avg=568.29, stdev=189.93, samples=21 iops : min= 18, max= 66, avg=35.43, stdev=11.96, samples=21 write: IOPS=37, BW=594KiB/s (608kB/s)(6256KiB/10532msec) clat (usec): min=10723, max=51706, avg=22858.11, stdev=7241.48 lat (usec): min=10724, max=51707, avg=22859.63, stdev=7241.53 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13304], 10.00th=[14222], 20.00th=[16712], | 30.00th=[19006], 40.00th=[20579], 50.00th=[22152], 60.00th=[23462], | 70.00th=[25035], 80.00th=[27395], 90.00th=[32637], 95.00th=[36439], | 99.00th=[46400], 99.50th=[51119], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 384, max= 832, per=25.27%, avg=594.10, stdev=155.73, samples=21 iops : min= 24, max= 52, avg=37.05, stdev= 9.77, samples=21 lat (msec) : 2=0.91%, 4=34.11%, 10=10.81%, 20=21.61%, 50=32.29% lat (msec) : 100=0.26% cpu : usr=0.09%, sys=5.68%, ctx=842, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2235KiB/s (2289kB/s), 536KiB/s-600KiB/s (549kB/s-615kB/s), io=23.4MiB (24.5MB), run=10422-10716msec WRITE: bw=2352KiB/s (2408kB/s), 579KiB/s-611KiB/s (593kB/s-625kB/s), io=24.6MiB (25.8MB), run=10422-10716msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=19112: Thu Jun 13 19:57:42 2024 read: IOPS=2, BW=2915KiB/s (2985kB/s)(7168KiB/2459msec) clat (msec): min=34, max=551, avg=325.07, stdev=229.42 lat (msec): min=34, max=551, avg=325.07, stdev=229.42 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 40], | 30.00th=[ 188], 40.00th=[ 188], 50.00th=[ 472], 60.00th=[ 489], | 70.00th=[ 489], 80.00th=[ 502], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 1726, max= 5180, per=33.39%, avg=2749.25, stdev=1627.49, samples=4 iops : min= 1, max= 5, avg= 2.25, stdev= 1.89, samples=4 write: IOPS=2, BW=2082KiB/s (2132kB/s)(5120KiB/2459msec) clat (usec): min=19570, max=64829, avg=36288.17, stdev=19378.46 lat (usec): min=19612, max=64893, avg=36345.68, stdev=19382.64 clat percentiles (usec): | 1.00th=[19530], 5.00th=[19530], 10.00th=[19530], 20.00th=[19530], | 30.00th=[22938], 40.00th=[22938], 50.00th=[26346], 60.00th=[26346], | 70.00th=[47973], 80.00th=[47973], 90.00th=[64750], 95.00th=[64750], | 99.00th=[64750], 99.50th=[64750], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 1726, max= 6907, per=37.44%, avg=4316.50, stdev=3663.52, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (msec) : 20=8.33%, 50=41.67%, 100=8.33%, 250=8.33%, 500=16.67% lat (msec) : 750=16.67% cpu : usr=0.04%, sys=24.69%, ctx=1211, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19113: Thu Jun 13 19:57:42 2024 read: IOPS=1, BW=1608KiB/s (1646kB/s)(2048KiB/1274msec) clat (usec): min=429, max=34752, avg=17591.30, stdev=24270.09 lat (usec): min=430, max=34754, avg=17592.75, stdev=24270.48 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 429], 10.00th=[ 429], 20.00th=[ 429], | 30.00th=[ 429], 40.00th=[ 429], 50.00th=[ 429], 60.00th=[34866], | 70.00th=[34866], 80.00th=[34866], 90.00th=[34866], 95.00th=[34866], | 99.00th=[34866], 99.50th=[34866], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 2048, max= 2048, per=24.87%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=7, BW=8038KiB/s (8231kB/s)(10.0MiB/1274msec) clat (msec): min=15, max=606, avg=122.70, stdev=210.19 lat (msec): min=15, max=606, avg=122.76, stdev=210.17 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 22], | 30.00th=[ 26], 40.00th=[ 27], 50.00th=[ 27], 60.00th=[ 27], | 70.00th=[ 33], 80.00th=[ 33], 90.00th=[ 418], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min=10240, max=10240, per=88.83%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 20=8.33%, 50=66.67%, 500=8.33%, 750=8.33% cpu : usr=0.16%, sys=18.70%, ctx=49, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19114: Thu Jun 13 19:57:42 2024 read: IOPS=2, BW=2470KiB/s (2530kB/s)(6144KiB/2487msec) clat (usec): min=324, max=483083, avg=172361.51, stdev=240222.83 lat (usec): min=325, max=483084, avg=172362.44, stdev=240222.90 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 326], 10.00th=[ 326], 20.00th=[ 424], | 30.00th=[ 424], 40.00th=[ 644], 50.00th=[ 644], 60.00th=[ 71828], | 70.00th=[476054], 80.00th=[476054], 90.00th=[484443], 95.00th=[484443], | 99.00th=[484443], 99.50th=[484443], 99.90th=[484443], 99.95th=[484443], | 99.99th=[484443] bw ( KiB/s): min= 2048, max= 8175, per=62.08%, avg=5111.50, stdev=4332.44, samples=2 iops : min= 2, max= 7, avg= 4.50, stdev= 3.54, samples=2 write: IOPS=2, BW=2470KiB/s (2530kB/s)(6144KiB/2487msec) clat (msec): min=21, max=1060, avg=240.15, stdev=405.79 lat (msec): min=21, max=1060, avg=240.19, stdev=405.80 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 34], | 30.00th=[ 34], 40.00th=[ 37], 50.00th=[ 37], 60.00th=[ 125], | 70.00th=[ 165], 80.00th=[ 165], 90.00th=[ 1062], 95.00th=[ 1062], | 99.00th=[ 1062], 99.50th=[ 1062], 99.90th=[ 1062], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 4096, max= 6131, per=44.36%, avg=5113.50, stdev=1438.96, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 50=25.00%, 100=8.33%, 250=16.67%, 500=16.67% cpu : usr=0.04%, sys=11.95%, ctx=611, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19115: Thu Jun 13 19:57:42 2024 read: IOPS=2, BW=2064KiB/s (2113kB/s)(5120KiB/2481msec) clat (usec): min=295, max=436806, avg=112818.17, stdev=189106.10 lat (usec): min=296, max=436807, avg=112819.01, stdev=189106.25 clat percentiles (usec): | 1.00th=[ 297], 5.00th=[ 297], 10.00th=[ 297], 20.00th=[ 297], | 30.00th=[ 469], 40.00th=[ 469], 50.00th=[ 490], 60.00th=[ 490], | 70.00th=[126354], 80.00th=[126354], 90.00th=[438305], 95.00th=[438305], | 99.00th=[438305], 99.50th=[438305], 99.90th=[438305], 99.95th=[438305], | 99.99th=[438305] bw ( KiB/s): min= 8175, max= 8175, per=99.28%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2889KiB/s (2958kB/s)(7168KiB/2481msec) clat (msec): min=33, max=1078, avg=272.87, stdev=400.14 lat (msec): min=33, max=1078, avg=273.00, stdev=400.21 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 38], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 74], 60.00th=[ 96], | 70.00th=[ 96], 80.00th=[ 550], 90.00th=[ 1083], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 2048, max=10219, per=53.21%, avg=6133.50, stdev=5777.77, samples=2 iops : min= 2, max= 9, avg= 5.50, stdev= 4.95, samples=2 lat (usec) : 500=25.00% lat (msec) : 50=25.00%, 100=16.67%, 250=8.33%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=9.72%, ctx=338, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8235KiB/s (8432kB/s), 1608KiB/s-2915KiB/s (1646kB/s-2985kB/s), io=20.0MiB (20.0MB), run=1274-2487msec WRITE: bw=11.3MiB/s (11.8MB/s), 2082KiB/s-8038KiB/s (2132kB/s-8231kB/s), io=28.0MiB (29.4MB), run=1274-2487msec rand-rw: (groupid=0, jobs=1): err= 0: pid=19116: Thu Jun 13 19:57:42 2024 read: IOPS=2, BW=2818KiB/s (2885kB/s)(7168KiB/2544msec) clat (msec): min=15, max=549, avg=255.14, stdev=244.47 lat (msec): min=15, max=549, avg=255.14, stdev=244.47 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 33], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 155], 60.00th=[ 481], | 70.00th=[ 481], 80.00th=[ 506], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 2048, max= 6144, per=42.59%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 write: IOPS=1, BW=2013KiB/s (2061kB/s)(5120KiB/2544msec) clat (msec): min=33, max=530, avg=150.98, stdev=212.68 lat (msec): min=33, max=530, avg=151.04, stdev=212.68 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34], | 30.00th=[ 59], 40.00th=[ 59], 50.00th=[ 61], 60.00th=[ 61], | 70.00th=[ 72], 80.00th=[ 72], 90.00th=[ 531], 95.00th=[ 531], | 99.00th=[ 531], 99.50th=[ 531], 99.90th=[ 531], 99.95th=[ 531], | 99.99th=[ 531] bw ( KiB/s): min= 6144, max= 6144, per=54.75%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=16.67%, 100=33.33%, 250=8.33%, 500=8.33% lat (msec) : 750=25.00% cpu : usr=0.00%, sys=4.40%, ctx=46, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19117: Thu Jun 13 19:57:42 2024 read: IOPS=0, BW=817KiB/s (837kB/s)(2048KiB/2506msec) clat (msec): min=49, max=464, avg=256.89, stdev=293.56 lat (msec): min=49, max=464, avg=256.89, stdev=293.56 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 464], | 70.00th=[ 464], 80.00th=[ 464], 90.00th=[ 464], 95.00th=[ 464], | 99.00th=[ 464], 99.50th=[ 464], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 2048, max= 2048, per=25.55%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=4086KiB/s (4184kB/s)(10.0MiB/2506msec) clat (msec): min=25, max=639, avg=199.08, stdev=237.43 lat (msec): min=25, max=639, avg=199.15, stdev=237.43 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 27], | 30.00th=[ 35], 40.00th=[ 47], 50.00th=[ 79], 60.00th=[ 83], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 514], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 4096, max=10240, per=54.75%, avg=6144.00, stdev=3547.24, samples=3 iops : min= 4, max= 10, avg= 6.00, stdev= 3.46, samples=3 lat (msec) : 50=41.67%, 100=25.00%, 500=16.67%, 750=16.67% cpu : usr=0.04%, sys=4.67%, ctx=46, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19118: Thu Jun 13 19:57:42 2024 read: IOPS=2, BW=2405KiB/s (2462kB/s)(6144KiB/2555msec) clat (msec): min=13, max=518, avg=113.51, stdev=199.38 lat (msec): min=13, max=518, avg=113.51, stdev=199.38 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 17], | 30.00th=[ 17], 40.00th=[ 21], 50.00th=[ 21], 60.00th=[ 46], | 70.00th=[ 67], 80.00th=[ 67], 90.00th=[ 518], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 2048, max= 6131, per=42.53%, avg=3409.00, stdev=2357.32, samples=3 iops : min= 2, max= 5, avg= 3.00, stdev= 1.73, samples=3 write: IOPS=2, BW=2405KiB/s (2462kB/s)(6144KiB/2555msec) clat (msec): min=30, max=1070, avg=311.90, stdev=406.96 lat (msec): min=30, max=1070, avg=311.96, stdev=406.96 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 45], | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 46], 60.00th=[ 220], | 70.00th=[ 464], 80.00th=[ 464], 90.00th=[ 1070], 95.00th=[ 1070], | 99.00th=[ 1070], 99.50th=[ 1070], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 2048, max= 4096, per=30.39%, avg=3410.33, stdev=1179.82, samples=3 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=3 lat (msec) : 20=16.67%, 50=41.67%, 100=8.33%, 250=8.33%, 500=8.33% lat (msec) : 750=8.33% cpu : usr=0.00%, sys=3.88%, ctx=43, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19119: Thu Jun 13 19:57:42 2024 read: IOPS=2, BW=2057KiB/s (2106kB/s)(5120KiB/2489msec) clat (msec): min=15, max=409, avg=174.82, stdev=203.48 lat (msec): min=15, max=409, avg=174.82, stdev=203.48 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 47], 60.00th=[ 47], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 409], 95.00th=[ 409], | 99.00th=[ 409], 99.50th=[ 409], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 4096, max= 4096, per=51.10%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2880KiB/s (2949kB/s)(7168KiB/2489msec) clat (msec): min=31, max=1041, avg=230.42, stdev=363.33 lat (msec): min=31, max=1041, avg=230.54, stdev=363.31 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 34], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 105], 60.00th=[ 163], | 70.00th=[ 163], 80.00th=[ 194], 90.00th=[ 1045], 95.00th=[ 1045], | 99.00th=[ 1045], 99.50th=[ 1045], 99.90th=[ 1045], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 2048, max= 6144, per=36.50%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (msec) : 20=16.67%, 50=33.33%, 250=25.00%, 500=16.67% cpu : usr=0.00%, sys=3.86%, ctx=43, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8016KiB/s (8208kB/s), 817KiB/s-2818KiB/s (837kB/s-2885kB/s), io=20.0MiB (20.0MB), run=2489-2555msec WRITE: bw=10.0MiB/s (11.5MB/s), 2013KiB/s-4086KiB/s (2061kB/s-4184kB/s), io=28.0MiB (29.4MB), run=2489-2555msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=19131: Thu Jun 13 19:57:45 2024 read: IOPS=1, BW=6962KiB/s (7129kB/s)(12.0MiB/1765msec) clat (msec): min=20, max=1575, avg=588.11, stdev=858.64 lat (msec): min=20, max=1575, avg=588.11, stdev=858.64 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 167], | 70.00th=[ 1569], 80.00th=[ 1569], 90.00th=[ 1569], 95.00th=[ 1569], | 99.00th=[ 1569], 99.50th=[ 1569], 99.90th=[ 1569], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33% cpu : usr=0.11%, sys=24.26%, ctx=875, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19132: Thu Jun 13 19:57:45 2024 read: IOPS=0, BW=1968KiB/s (2016kB/s)(4096KiB/2081msec) clat (nsec): min=172551k, max=172551k, avg=172551322.00, stdev= 0.00 lat (nsec): min=172552k, max=172552k, avg=172552137.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 174], 50.00th=[ 174], 60.00th=[ 174], | 70.00th=[ 174], 80.00th=[ 174], 90.00th=[ 174], 95.00th=[ 174], | 99.00th=[ 174], 99.50th=[ 174], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3937KiB/s (4031kB/s)(8192KiB/2081msec) clat (msec): min=293, max=1613, avg=953.66, stdev=933.74 lat (msec): min=293, max=1614, avg=953.97, stdev=933.62 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 292], 10.00th=[ 292], 20.00th=[ 292], | 30.00th=[ 292], 40.00th=[ 292], 50.00th=[ 292], 60.00th=[ 1620], | 70.00th=[ 1620], 80.00th=[ 1620], 90.00th=[ 1620], 95.00th=[ 1620], | 99.00th=[ 1620], 99.50th=[ 1620], 99.90th=[ 1620], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 8192, max= 8192, per=52.03%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=11.29%, ctx=185, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19133: Thu Jun 13 19:57:45 2024 write: IOPS=1, BW=5939KiB/s (6082kB/s)(12.0MiB/2069msec) clat (msec): min=86, max=1641, avg=689.23, stdev=834.20 lat (msec): min=86, max=1641, avg=689.46, stdev=834.23 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 87], 10.00th=[ 87], 20.00th=[ 87], | 30.00th=[ 87], 40.00th=[ 342], 50.00th=[ 342], 60.00th=[ 342], | 70.00th=[ 1636], 80.00th=[ 1636], 90.00th=[ 1636], 95.00th=[ 1636], | 99.00th=[ 1636], 99.50th=[ 1636], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 7984, max= 8192, per=51.37%, avg=8088.00, stdev=147.08, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=12.38%, ctx=30, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19134: Thu Jun 13 19:57:45 2024 write: IOPS=1, BW=6614KiB/s (6772kB/s)(12.0MiB/1858msec) clat (msec): min=131, max=1545, avg=614.26, stdev=806.61 lat (msec): min=131, max=1545, avg=614.47, stdev=806.65 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 132], 10.00th=[ 132], 20.00th=[ 132], | 30.00th=[ 132], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 167], | 70.00th=[ 1552], 80.00th=[ 1552], 90.00th=[ 1552], 95.00th=[ 1552], | 99.00th=[ 1552], 99.50th=[ 1552], 99.90th=[ 1552], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 8192, max= 8192, per=52.03%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=14.70%, ctx=116, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7873KiB/s (8062kB/s), 1968KiB/s-6962KiB/s (2016kB/s-7129kB/s), io=16.0MiB (16.8MB), run=1765-2081msec WRITE: bw=15.4MiB/s (16.1MB/s), 3937KiB/s-6614KiB/s (4031kB/s-6772kB/s), io=32.0MiB (33.6MB), run=1858-2081msec rand-rw: (groupid=0, jobs=1): err= 0: pid=19127: Thu Jun 13 19:57:45 2024 read: IOPS=2, BW=10.1MiB/s (10.5MB/s)(12.0MiB/1193msec) clat (msec): min=36, max=1092, avg=395.16, stdev=604.40 lat (msec): min=36, max=1092, avg=395.16, stdev=604.40 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 57], 50.00th=[ 57], 60.00th=[ 57], | 70.00th=[ 1099], 80.00th=[ 1099], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=33.33% cpu : usr=0.00%, sys=4.95%, ctx=49, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19128: Thu Jun 13 19:57:45 2024 read: IOPS=0, BW=1888KiB/s (1934kB/s)(4096KiB/2169msec) clat (nsec): min=244514k, max=244514k, avg=244513729.00, stdev= 0.00 lat (nsec): min=244515k, max=244515k, avg=244514985.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 245], 10.00th=[ 245], 20.00th=[ 245], | 30.00th=[ 245], 40.00th=[ 245], 50.00th=[ 245], 60.00th=[ 245], | 70.00th=[ 245], 80.00th=[ 245], 90.00th=[ 245], 95.00th=[ 245], | 99.00th=[ 245], 99.50th=[ 245], 99.90th=[ 245], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3777KiB/s (3868kB/s)(8192KiB/2169msec) clat (msec): min=249, max=1669, avg=959.32, stdev=1004.12 lat (msec): min=249, max=1669, avg=959.60, stdev=1004.12 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 249], 10.00th=[ 249], 20.00th=[ 249], | 30.00th=[ 249], 40.00th=[ 249], 50.00th=[ 249], 60.00th=[ 1670], | 70.00th=[ 1670], 80.00th=[ 1670], 90.00th=[ 1670], 95.00th=[ 1670], | 99.00th=[ 1670], 99.50th=[ 1670], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 7953, max= 7953, per=52.64%, avg=7953.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.05%, sys=4.94%, ctx=48, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19129: Thu Jun 13 19:57:45 2024 write: IOPS=1, BW=6181KiB/s (6329kB/s)(12.0MiB/1988msec) clat (msec): min=93, max=1677, avg=660.92, stdev=882.46 lat (msec): min=93, max=1677, avg=661.26, stdev=882.49 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 94], 10.00th=[ 94], 20.00th=[ 94], | 30.00th=[ 94], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 211], | 70.00th=[ 1670], 80.00th=[ 1670], 90.00th=[ 1670], 95.00th=[ 1670], | 99.00th=[ 1670], 99.50th=[ 1670], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 8192, max= 8192, per=54.23%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=6.59%, ctx=62, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19130: Thu Jun 13 19:57:45 2024 write: IOPS=2, BW=10.0MiB/s (10.5MB/s)(12.0MiB/1199msec) clat (msec): min=94, max=996, avg=397.33, stdev=518.65 lat (msec): min=95, max=996, avg=397.59, stdev=518.64 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 95], 10.00th=[ 95], 20.00th=[ 95], | 30.00th=[ 95], 40.00th=[ 102], 50.00th=[ 102], 60.00th=[ 102], | 70.00th=[ 995], 80.00th=[ 995], 90.00th=[ 995], 95.00th=[ 995], | 99.00th=[ 995], 99.50th=[ 995], 99.90th=[ 995], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=7.10%, ctx=118, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7554KiB/s (7735kB/s), 1888KiB/s-10.1MiB/s (1934kB/s-10.5MB/s), io=16.0MiB (16.8MB), run=1193-2169msec WRITE: bw=14.8MiB/s (15.5MB/s), 3777KiB/s-10.0MiB/s (3868kB/s-10.5MB/s), io=32.0MiB (33.6MB), run=1199-2169msec