== sanity test 398b: DIO and buffer IO race ============== 02:03:38 (1721887418) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.14497 s, 44.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7150: Thu Jul 25 02:03:53 2024 read: IOPS=130, BW=524KiB/s (537kB/s)(5992KiB/11436msec) clat (usec): min=53, max=144736, avg=4317.92, stdev=4335.79 lat (usec): min=54, max=144738, avg=4318.47, stdev=4335.83 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 1811], 10.00th=[ 2114], 20.00th=[ 2474], | 30.00th=[ 2835], 40.00th=[ 3228], 50.00th=[ 3818], 60.00th=[ 4424], | 70.00th=[ 4948], 80.00th=[ 5669], 90.00th=[ 6980], 95.00th=[ 8225], | 99.00th=[ 13042], 99.50th=[ 14877], 99.90th=[ 22938], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 248, max= 672, per=25.09%, avg=516.00, stdev=86.37, samples=22 iops : min= 62, max= 168, avg=129.00, stdev=21.59, samples=22 write: IOPS=137, BW=551KiB/s (564kB/s)(6296KiB/11436msec) clat (usec): min=422, max=16555, avg=3101.38, stdev=2761.96 lat (usec): min=423, max=16556, avg=3102.05, stdev=2761.96 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 515], 10.00th=[ 537], 20.00th=[ 570], | 30.00th=[ 717], 40.00th=[ 1811], 50.00th=[ 2507], 60.00th=[ 3097], | 70.00th=[ 3949], 80.00th=[ 5211], 90.00th=[ 7177], 95.00th=[ 8586], | 99.00th=[11338], 99.50th=[13042], 99.90th=[16581], 99.95th=[16581], | 99.99th=[16581] bw ( KiB/s): min= 296, max= 864, per=25.10%, avg=540.73, stdev=98.28, samples=22 iops : min= 74, max= 216, avg=135.18, stdev=24.57, samples=22 lat (usec) : 100=1.86%, 250=0.13%, 500=1.20%, 750=14.81%, 1000=2.12% lat (msec) : 2=4.75%, 4=37.01%, 10=35.64%, 20=2.38%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=22.48%, ctx=5747, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7152: Thu Jul 25 02:03:53 2024 read: IOPS=127, BW=512KiB/s (524kB/s)(5940KiB/11602msec) clat (usec): min=47, max=19509, avg=4207.90, stdev=2305.28 lat (usec): min=47, max=19510, avg=4208.52, stdev=2305.28 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 1811], 10.00th=[ 2147], 20.00th=[ 2474], | 30.00th=[ 2835], 40.00th=[ 3294], 50.00th=[ 3851], 60.00th=[ 4424], | 70.00th=[ 4948], 80.00th=[ 5604], 90.00th=[ 6718], 95.00th=[ 8029], | 99.00th=[13304], 99.50th=[14222], 99.90th=[15664], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 272, max= 696, per=24.74%, avg=508.91, stdev=84.45, samples=23 iops : min= 68, max= 174, avg=127.13, stdev=21.13, samples=23 write: IOPS=136, BW=547KiB/s (560kB/s)(6348KiB/11602msec) clat (usec): min=375, max=154297, avg=3327.45, stdev=4738.40 lat (usec): min=376, max=154298, avg=3328.29, stdev=4738.39 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 519], 10.00th=[ 537], 20.00th=[ 586], | 30.00th=[ 758], 40.00th=[ 2008], 50.00th=[ 2606], 60.00th=[ 3163], | 70.00th=[ 4047], 80.00th=[ 5538], 90.00th=[ 7373], 95.00th=[ 8979], | 99.00th=[ 11994], 99.50th=[ 13435], 99.90th=[ 15533], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 344, max= 752, per=25.13%, avg=541.22, stdev=85.29, samples=23 iops : min= 86, max= 188, avg=135.22, stdev=21.34, samples=23 lat (usec) : 50=0.03%, 100=1.89%, 250=0.13%, 500=1.11%, 750=14.39% lat (usec) : 1000=1.53% lat (msec) : 2=4.98%, 4=37.37%, 10=35.64%, 20=2.90%, 250=0.03% cpu : usr=0.20%, sys=22.29%, ctx=5719, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7153: Thu Jul 25 02:03:53 2024 read: IOPS=129, BW=516KiB/s (529kB/s)(6028KiB/11671msec) clat (usec): min=45, max=16027, avg=4278.80, stdev=2254.98 lat (usec): min=46, max=16027, avg=4279.58, stdev=2255.14 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 1827], 10.00th=[ 2147], 20.00th=[ 2540], | 30.00th=[ 2933], 40.00th=[ 3392], 50.00th=[ 3949], 60.00th=[ 4490], | 70.00th=[ 5145], 80.00th=[ 5866], 90.00th=[ 7046], 95.00th=[ 8291], | 99.00th=[11731], 99.50th=[13566], 99.90th=[15533], 99.95th=[16057], | 99.99th=[16057] bw ( KiB/s): min= 240, max= 672, per=24.96%, avg=513.39, stdev=84.41, samples=23 iops : min= 60, max= 168, avg=128.35, stdev=21.10, samples=23 write: IOPS=134, BW=536KiB/s (549kB/s)(6260KiB/11671msec) clat (usec): min=446, max=146991, avg=3282.40, stdev=4555.73 lat (usec): min=446, max=146993, avg=3283.62, stdev=4555.51 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 519], 10.00th=[ 537], 20.00th=[ 586], | 30.00th=[ 791], 40.00th=[ 2040], 50.00th=[ 2671], 60.00th=[ 3195], | 70.00th=[ 4015], 80.00th=[ 5276], 90.00th=[ 7242], 95.00th=[ 8455], | 99.00th=[ 11863], 99.50th=[ 13042], 99.90th=[ 15795], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 312, max= 712, per=24.67%, avg=531.48, stdev=91.88, samples=23 iops : min= 78, max= 178, avg=132.87, stdev=22.97, samples=23 lat (usec) : 50=0.13%, 100=1.99%, 250=0.16%, 500=1.04%, 750=13.83% lat (usec) : 1000=1.95% lat (msec) : 2=4.75%, 4=36.95%, 10=36.88%, 20=2.28%, 250=0.03% cpu : usr=0.23%, sys=22.22%, ctx=5860, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7154: Thu Jul 25 02:03:53 2024 read: IOPS=130, BW=521KiB/s (534kB/s)(6052KiB/11610msec) clat (usec): min=51, max=16231, avg=4201.88, stdev=2198.33 lat (usec): min=51, max=16232, avg=4202.48, stdev=2198.32 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 1844], 10.00th=[ 2114], 20.00th=[ 2507], | 30.00th=[ 2900], 40.00th=[ 3326], 50.00th=[ 3884], 60.00th=[ 4424], | 70.00th=[ 4948], 80.00th=[ 5604], 90.00th=[ 6849], 95.00th=[ 8029], | 99.00th=[12125], 99.50th=[12911], 99.90th=[15926], 99.95th=[16188], | 99.99th=[16188] bw ( KiB/s): min= 280, max= 640, per=25.22%, avg=518.83, stdev=73.52, samples=23 iops : min= 70, max= 160, avg=129.65, stdev=18.38, samples=23 write: IOPS=134, BW=537KiB/s (550kB/s)(6236KiB/11610msec) clat (usec): min=403, max=154497, avg=3324.69, stdev=4717.60 lat (usec): min=404, max=154498, avg=3325.77, stdev=4717.37 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 519], 10.00th=[ 537], 20.00th=[ 578], | 30.00th=[ 766], 40.00th=[ 2057], 50.00th=[ 2704], 60.00th=[ 3326], | 70.00th=[ 4293], 80.00th=[ 5538], 90.00th=[ 7308], 95.00th=[ 8356], | 99.00th=[ 11469], 99.50th=[ 13566], 99.90th=[ 15008], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 360, max= 688, per=24.88%, avg=535.83, stdev=85.94, samples=23 iops : min= 90, max= 172, avg=133.91, stdev=21.45, samples=23 lat (usec) : 100=1.82%, 250=0.13%, 500=0.85%, 750=14.19%, 1000=1.89% lat (msec) : 2=4.82%, 4=36.59%, 10=37.60%, 20=2.08%, 250=0.03% cpu : usr=0.17%, sys=22.38%, ctx=5655, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2057KiB/s (2107kB/s), 512KiB/s-524KiB/s (524kB/s-537kB/s), io=23.4MiB (24.6MB), run=11436-11671msec WRITE: bw=2154KiB/s (2206kB/s), 536KiB/s-551KiB/s (549kB/s-564kB/s), io=24.6MiB (25.7MB), run=11436-11671msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7147: Thu Jul 25 02:03:56 2024 read: IOPS=99, BW=400KiB/s (409kB/s)(5992KiB/14995msec) clat (usec): min=1467, max=156876, avg=4082.82, stdev=4981.21 lat (usec): min=1468, max=156878, avg=4083.56, stdev=4981.21 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 1745], 10.00th=[ 1844], 20.00th=[ 1975], | 30.00th=[ 2147], 40.00th=[ 2343], 50.00th=[ 2606], 60.00th=[ 3064], | 70.00th=[ 4015], 80.00th=[ 5669], 90.00th=[ 8586], 95.00th=[ 10683], | 99.00th=[ 14353], 99.50th=[ 17695], 99.90th=[ 23200], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 160, max= 792, per=24.42%, avg=390.48, stdev=176.81, samples=29 iops : min= 40, max= 198, avg=97.52, stdev=44.25, samples=29 write: IOPS=104, BW=420KiB/s (430kB/s)(6296KiB/14995msec) clat (usec): min=2143, max=24994, avg=5594.17, stdev=3240.44 lat (usec): min=2144, max=24994, avg=5594.92, stdev=3240.43 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2704], 10.00th=[ 2900], 20.00th=[ 3130], | 30.00th=[ 3359], 40.00th=[ 3654], 50.00th=[ 4146], 60.00th=[ 5145], | 70.00th=[ 6521], 80.00th=[ 8029], 90.00th=[10159], 95.00th=[11994], | 99.00th=[16909], 99.50th=[18744], 99.90th=[22938], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 176, max= 824, per=24.45%, avg=409.24, stdev=178.85, samples=29 iops : min= 44, max= 206, avg=102.21, stdev=44.76, samples=29 lat (msec) : 2=10.61%, 4=47.46%, 10=33.33%, 20=8.30%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=10.95%, ctx=3517, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7148: Thu Jul 25 02:03:56 2024 read: IOPS=98, BW=396KiB/s (405kB/s)(5940KiB/15009msec) clat (usec): min=1472, max=157380, avg=4025.87, stdev=4993.78 lat (usec): min=1472, max=157382, avg=4026.50, stdev=4993.81 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1762], 10.00th=[ 1827], 20.00th=[ 1975], | 30.00th=[ 2147], 40.00th=[ 2343], 50.00th=[ 2638], 60.00th=[ 3032], | 70.00th=[ 3752], 80.00th=[ 5342], 90.00th=[ 8586], 95.00th=[ 10814], | 99.00th=[ 15795], 99.50th=[ 17433], 99.90th=[ 18482], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 136, max= 800, per=24.74%, avg=395.53, stdev=185.78, samples=30 iops : min= 34, max= 200, avg=98.77, stdev=46.50, samples=30 write: IOPS=105, BW=423KiB/s (433kB/s)(6348KiB/15009msec) clat (usec): min=2194, max=24873, avg=5644.90, stdev=3260.18 lat (usec): min=2194, max=24874, avg=5645.60, stdev=3260.18 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2737], 10.00th=[ 2933], 20.00th=[ 3163], | 30.00th=[ 3392], 40.00th=[ 3720], 50.00th=[ 4228], 60.00th=[ 5145], | 70.00th=[ 6718], 80.00th=[ 8225], 90.00th=[10290], 95.00th=[11863], | 99.00th=[17433], 99.50th=[18482], 99.90th=[22938], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 208, max= 776, per=25.25%, avg=422.73, stdev=177.92, samples=30 iops : min= 52, max= 194, avg=105.57, stdev=44.50, samples=30 lat (msec) : 2=10.38%, 4=48.50%, 10=31.97%, 20=8.95%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=11.21%, ctx=3502, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7149: Thu Jul 25 02:03:56 2024 read: IOPS=101, BW=406KiB/s (416kB/s)(6028KiB/14844msec) clat (usec): min=1468, max=22381, avg=3872.04, stdev=3061.53 lat (usec): min=1468, max=22381, avg=3872.62, stdev=3061.53 clat percentiles (usec): | 1.00th=[ 1565], 5.00th=[ 1729], 10.00th=[ 1811], 20.00th=[ 1958], | 30.00th=[ 2114], 40.00th=[ 2311], 50.00th=[ 2606], 60.00th=[ 2999], | 70.00th=[ 3687], 80.00th=[ 5080], 90.00th=[ 8717], 95.00th=[10552], | 99.00th=[15139], 99.50th=[17695], 99.90th=[20841], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 152, max= 832, per=24.99%, avg=399.59, stdev=175.95, samples=29 iops : min= 38, max= 208, avg=99.79, stdev=44.04, samples=29 write: IOPS=105, BW=422KiB/s (432kB/s)(6260KiB/14844msec) clat (msec): min=2, max=160, avg= 5.71, stdev= 5.11 lat (msec): min=2, max=160, avg= 5.71, stdev= 5.11 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 19], 99.50th=[ 19], 99.90th=[ 23], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 176, max= 824, per=24.66%, avg=412.83, stdev=177.41, samples=29 iops : min= 44, max= 206, avg=103.10, stdev=44.39, samples=29 lat (msec) : 2=11.20%, 4=48.80%, 10=31.22%, 20=8.56%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=11.05%, ctx=3546, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7151: Thu Jul 25 02:03:56 2024 read: IOPS=101, BW=406KiB/s (416kB/s)(6052KiB/14889msec) clat (usec): min=1508, max=23485, avg=3997.03, stdev=3166.85 lat (usec): min=1508, max=23486, avg=3997.66, stdev=3166.84 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 1745], 10.00th=[ 1827], 20.00th=[ 1975], | 30.00th=[ 2114], 40.00th=[ 2343], 50.00th=[ 2606], 60.00th=[ 3032], | 70.00th=[ 3818], 80.00th=[ 5538], 90.00th=[ 8848], 95.00th=[10814], | 99.00th=[16057], 99.50th=[17433], 99.90th=[22414], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 200, max= 808, per=24.85%, avg=397.28, stdev=173.48, samples=29 iops : min= 50, max= 202, avg=99.21, stdev=43.37, samples=29 write: IOPS=104, BW=419KiB/s (429kB/s)(6236KiB/14889msec) clat (msec): min=2, max=157, avg= 5.62, stdev= 5.01 lat (msec): min=2, max=157, avg= 5.62, stdev= 5.01 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 17], 99.50th=[ 19], 99.90th=[ 22], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 208, max= 870, per=24.56%, avg=411.07, stdev=181.12, samples=29 iops : min= 52, max= 217, avg=102.66, stdev=45.22, samples=29 lat (msec) : 2=11.00%, 4=48.44%, 10=31.90%, 20=8.43%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=11.03%, ctx=3550, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1600KiB/s (1638kB/s), 396KiB/s-406KiB/s (405kB/s-416kB/s), io=23.4MiB (24.6MB), run=14844-15009msec WRITE: bw=1675KiB/s (1715kB/s), 419KiB/s-423KiB/s (429kB/s-433kB/s), io=24.6MiB (25.7MB), run=14844-15009msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7186: Thu Jul 25 02:04:02 2024 read: IOPS=72, BW=1159KiB/s (1187kB/s)(5920KiB/5109msec) clat (usec): min=1740, max=32685, avg=5225.79, stdev=3993.91 lat (usec): min=1741, max=32686, avg=5226.40, stdev=3993.91 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2024], 10.00th=[ 2212], 20.00th=[ 2409], | 30.00th=[ 2573], 40.00th=[ 2933], 50.00th=[ 3458], 60.00th=[ 4359], | 70.00th=[ 5866], 80.00th=[ 8094], 90.00th=[10814], 95.00th=[12256], | 99.00th=[19268], 99.50th=[23200], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 800, max= 1532, per=24.03%, avg=1126.67, stdev=260.06, samples=9 iops : min= 50, max= 95, avg=70.33, stdev=16.11, samples=9 write: IOPS=77, BW=1246KiB/s (1276kB/s)(6368KiB/5109msec) clat (usec): min=2400, max=74844, avg=7941.77, stdev=5717.30 lat (usec): min=2402, max=74845, avg=7943.01, stdev=5717.33 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3163], 10.00th=[ 3425], 20.00th=[ 3916], | 30.00th=[ 4424], 40.00th=[ 5407], 50.00th=[ 6194], 60.00th=[ 7635], | 70.00th=[ 9372], 80.00th=[11076], 90.00th=[15270], 95.00th=[17433], | 99.00th=[22938], 99.50th=[27395], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 928, max= 1440, per=24.79%, avg=1222.78, stdev=185.64, samples=9 iops : min= 58, max= 90, avg=76.33, stdev=11.58, samples=9 lat (msec) : 2=2.21%, 4=36.98%, 10=41.15%, 20=17.97%, 50=1.56% lat (msec) : 100=0.13% cpu : usr=0.18%, sys=9.30%, ctx=932, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7187: Thu Jul 25 02:04:02 2024 read: IOPS=70, BW=1125KiB/s (1152kB/s)(5744KiB/5107msec) clat (usec): min=1824, max=27602, avg=5526.20, stdev=3991.32 lat (usec): min=1825, max=27602, avg=5526.88, stdev=3991.27 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2057], 10.00th=[ 2212], 20.00th=[ 2507], | 30.00th=[ 2802], 40.00th=[ 3163], 50.00th=[ 3851], 60.00th=[ 5211], | 70.00th=[ 6325], 80.00th=[ 8356], 90.00th=[11338], 95.00th=[14353], | 99.00th=[17957], 99.50th=[23200], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 640, max= 1504, per=24.19%, avg=1133.89, stdev=255.34, samples=9 iops : min= 40, max= 94, avg=70.78, stdev=15.90, samples=9 write: IOPS=80, BW=1281KiB/s (1312kB/s)(6544KiB/5107msec) clat (usec): min=2553, max=69380, avg=7529.82, stdev=5970.58 lat (usec): min=2555, max=69381, avg=7531.11, stdev=5970.52 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 3032], 10.00th=[ 3326], 20.00th=[ 3687], | 30.00th=[ 4113], 40.00th=[ 4752], 50.00th=[ 5866], 60.00th=[ 6718], | 70.00th=[ 8586], 80.00th=[10552], 90.00th=[13566], 95.00th=[16712], | 99.00th=[27657], 99.50th=[35390], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 896, max= 1568, per=25.80%, avg=1272.56, stdev=234.17, samples=9 iops : min= 56, max= 98, avg=79.44, stdev=14.55, samples=9 lat (msec) : 2=1.43%, 4=36.85%, 10=43.88%, 20=15.89%, 50=1.69% lat (msec) : 100=0.26% cpu : usr=0.10%, sys=9.66%, ctx=931, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7188: Thu Jul 25 02:04:02 2024 read: IOPS=80, BW=1292KiB/s (1323kB/s)(6256KiB/4842msec) clat (usec): min=1826, max=49176, avg=5333.70, stdev=4694.93 lat (usec): min=1826, max=49177, avg=5334.39, stdev=4695.02 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2073], 10.00th=[ 2212], 20.00th=[ 2507], | 30.00th=[ 2704], 40.00th=[ 2966], 50.00th=[ 3425], 60.00th=[ 4293], | 70.00th=[ 5604], 80.00th=[ 7963], 90.00th=[10945], 95.00th=[13042], | 99.00th=[22676], 99.50th=[32375], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 704, max= 1824, per=27.46%, avg=1287.11, stdev=315.48, samples=9 iops : min= 44, max= 114, avg=80.44, stdev=19.72, samples=9 write: IOPS=77, BW=1246KiB/s (1276kB/s)(6032KiB/4842msec) clat (usec): min=2657, max=27193, avg=7218.24, stdev=4405.77 lat (usec): min=2658, max=27194, avg=7219.43, stdev=4405.77 clat percentiles (usec): | 1.00th=[ 2802], 5.00th=[ 3130], 10.00th=[ 3359], 20.00th=[ 3785], | 30.00th=[ 4228], 40.00th=[ 4752], 50.00th=[ 5735], 60.00th=[ 6587], | 70.00th=[ 8029], 80.00th=[10290], 90.00th=[14222], 95.00th=[16581], | 99.00th=[21365], 99.50th=[23462], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 928, max= 1536, per=24.66%, avg=1216.00, stdev=181.02, samples=9 iops : min= 58, max= 96, avg=76.00, stdev=11.31, samples=9 lat (msec) : 2=2.08%, 4=39.58%, 10=42.06%, 20=14.97%, 50=1.30% cpu : usr=0.14%, sys=10.08%, ctx=927, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7189: Thu Jul 25 02:04:02 2024 read: IOPS=74, BW=1185KiB/s (1214kB/s)(6032KiB/5090msec) clat (usec): min=1806, max=43460, avg=5691.37, stdev=4664.08 lat (usec): min=1806, max=43460, avg=5691.94, stdev=4664.10 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2073], 10.00th=[ 2278], 20.00th=[ 2540], | 30.00th=[ 2835], 40.00th=[ 3163], 50.00th=[ 3982], 60.00th=[ 4883], | 70.00th=[ 6325], 80.00th=[ 8356], 90.00th=[10945], 95.00th=[14091], | 99.00th=[22938], 99.50th=[31851], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 800, max= 1596, per=25.17%, avg=1180.20, stdev=252.65, samples=10 iops : min= 50, max= 99, avg=73.60, stdev=15.79, samples=10 write: IOPS=76, BW=1229KiB/s (1259kB/s)(6256KiB/5090msec) clat (usec): min=2456, max=33994, avg=7461.76, stdev=4721.46 lat (usec): min=2458, max=33995, avg=7463.09, stdev=4721.51 clat percentiles (usec): | 1.00th=[ 2802], 5.00th=[ 3261], 10.00th=[ 3490], 20.00th=[ 3752], | 30.00th=[ 4228], 40.00th=[ 4883], 50.00th=[ 5800], 60.00th=[ 6915], | 70.00th=[ 8455], 80.00th=[10945], 90.00th=[14615], 95.00th=[16909], | 99.00th=[22152], 99.50th=[28181], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 862, max= 1728, per=24.84%, avg=1225.10, stdev=260.07, samples=10 iops : min= 53, max= 108, avg=76.40, stdev=16.33, samples=10 lat (msec) : 2=1.17%, 4=36.20%, 10=44.66%, 20=15.89%, 50=2.08% cpu : usr=0.08%, sys=9.49%, ctx=920, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4688KiB/s (4801kB/s), 1125KiB/s-1292KiB/s (1152kB/s-1323kB/s), io=23.4MiB (24.5MB), run=4842-5109msec WRITE: bw=4932KiB/s (5051kB/s), 1229KiB/s-1281KiB/s (1259kB/s-1312kB/s), io=24.6MiB (25.8MB), run=4842-5109msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7191: Thu Jul 25 02:04:02 2024 read: IOPS=69, BW=1114KiB/s (1141kB/s)(5920KiB/5314msec) clat (usec): min=61, max=55697, avg=10604.13, stdev=4197.81 lat (usec): min=61, max=55698, avg=10604.78, stdev=4197.85 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 6521], 10.00th=[ 7570], 20.00th=[ 8717], | 30.00th=[ 9372], 40.00th=[ 9896], 50.00th=[10421], 60.00th=[10945], | 70.00th=[11600], 80.00th=[12518], 90.00th=[14091], 95.00th=[15926], | 99.00th=[20841], 99.50th=[25035], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 704, max= 1277, per=24.27%, avg=1061.50, stdev=156.51, samples=10 iops : min= 44, max= 79, avg=66.10, stdev= 9.56, samples=10 write: IOPS=74, BW=1198KiB/s (1227kB/s)(6368KiB/5314msec) clat (usec): min=636, max=21059, avg=3449.73, stdev=3360.21 lat (usec): min=637, max=21060, avg=3450.71, stdev=3360.20 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 766], 10.00th=[ 791], 20.00th=[ 848], | 30.00th=[ 1352], 40.00th=[ 2180], 50.00th=[ 2540], 60.00th=[ 2835], | 70.00th=[ 3490], 80.00th=[ 4817], 90.00th=[ 8094], 95.00th=[11338], | 99.00th=[17433], 99.50th=[20317], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 800, max= 1536, per=25.16%, avg=1157.60, stdev=222.24, samples=10 iops : min= 50, max= 96, avg=72.10, stdev=13.98, samples=10 lat (usec) : 100=1.95%, 250=0.13%, 500=0.13%, 750=1.82%, 1000=11.72% lat (msec) : 2=4.56%, 4=20.57%, 10=28.39%, 20=29.69%, 50=0.91% lat (msec) : 100=0.13% cpu : usr=0.09%, sys=20.29%, ctx=2540, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7192: Thu Jul 25 02:04:02 2024 read: IOPS=67, BW=1078KiB/s (1104kB/s)(5744KiB/5330msec) clat (usec): min=50, max=35734, avg=10310.72, stdev=4061.70 lat (usec): min=51, max=35735, avg=10311.33, stdev=4061.71 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 75], 10.00th=[ 7046], 20.00th=[ 8455], | 30.00th=[ 9110], 40.00th=[10159], 50.00th=[10552], 60.00th=[11076], | 70.00th=[11863], 80.00th=[12911], 90.00th=[14484], 95.00th=[15795], | 99.00th=[19530], 99.50th=[21365], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 510, max= 1184, per=23.48%, avg=1026.70, stdev=199.61, samples=10 iops : min= 31, max= 74, avg=64.00, stdev=12.68, samples=10 write: IOPS=76, BW=1228KiB/s (1257kB/s)(6544KiB/5330msec) clat (usec): min=597, max=30331, avg=3829.71, stdev=4018.01 lat (usec): min=598, max=30333, avg=3837.41, stdev=4019.66 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 766], 10.00th=[ 791], 20.00th=[ 922], | 30.00th=[ 1778], 40.00th=[ 2278], 50.00th=[ 2638], 60.00th=[ 3228], | 70.00th=[ 4015], 80.00th=[ 5145], 90.00th=[ 8160], 95.00th=[12780], | 99.00th=[17695], 99.50th=[23200], 99.90th=[30278], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 798, max= 1824, per=25.24%, avg=1161.20, stdev=371.78, samples=10 iops : min= 49, max= 114, avg=72.40, stdev=23.42, samples=10 lat (usec) : 100=3.39%, 250=0.13%, 750=2.08%, 1000=10.55% lat (msec) : 2=3.91%, 4=20.44%, 10=26.43%, 20=32.16%, 50=0.91% cpu : usr=0.17%, sys=19.44%, ctx=2522, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7193: Thu Jul 25 02:04:02 2024 read: IOPS=71, BW=1142KiB/s (1170kB/s)(6256KiB/5477msec) clat (usec): min=57, max=23382, avg=10346.12, stdev=3832.40 lat (usec): min=57, max=23383, avg=10348.47, stdev=3832.85 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 81], 10.00th=[ 7308], 20.00th=[ 8225], | 30.00th=[ 9110], 40.00th=[ 9896], 50.00th=[10552], 60.00th=[11207], | 70.00th=[11994], 80.00th=[12780], 90.00th=[14091], 95.00th=[15926], | 99.00th=[20841], 99.50th=[22938], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 640, max= 1280, per=24.79%, avg=1083.90, stdev=178.52, samples=10 iops : min= 40, max= 80, avg=67.50, stdev=11.10, samples=10 write: IOPS=68, BW=1101KiB/s (1128kB/s)(6032KiB/5477msec) clat (usec): min=575, max=41505, avg=3659.25, stdev=4002.80 lat (usec): min=576, max=41507, avg=3660.25, stdev=4002.79 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 758], 10.00th=[ 791], 20.00th=[ 938], | 30.00th=[ 1565], 40.00th=[ 2245], 50.00th=[ 2638], 60.00th=[ 3064], | 70.00th=[ 3785], 80.00th=[ 4883], 90.00th=[ 7111], 95.00th=[11863], | 99.00th=[20055], 99.50th=[21103], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 832, max= 1532, per=23.00%, avg=1058.30, stdev=209.92, samples=10 iops : min= 52, max= 95, avg=65.90, stdev=12.97, samples=10 lat (usec) : 100=2.99%, 250=0.39%, 750=2.21%, 1000=8.85% lat (msec) : 2=5.34%, 4=19.53%, 10=28.26%, 20=31.38%, 50=1.04% cpu : usr=0.16%, sys=19.28%, ctx=2516, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7194: Thu Jul 25 02:04:02 2024 read: IOPS=69, BW=1119KiB/s (1146kB/s)(6032KiB/5389msec) clat (usec): min=58, max=28192, avg=10283.52, stdev=3860.84 lat (usec): min=59, max=28194, avg=10284.17, stdev=3860.89 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 77], 10.00th=[ 7373], 20.00th=[ 8586], | 30.00th=[ 9241], 40.00th=[ 9765], 50.00th=[10421], 60.00th=[11076], | 70.00th=[11731], 80.00th=[12780], 90.00th=[14222], 95.00th=[15270], | 99.00th=[19268], 99.50th=[26608], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 608, max= 1344, per=24.37%, avg=1065.60, stdev=194.39, samples=10 iops : min= 38, max= 84, avg=66.60, stdev=12.15, samples=10 write: IOPS=72, BW=1161KiB/s (1189kB/s)(6256KiB/5389msec) clat (usec): min=600, max=25131, avg=3694.25, stdev=3712.32 lat (usec): min=601, max=25133, avg=3695.22, stdev=3712.44 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 766], 10.00th=[ 791], 20.00th=[ 938], | 30.00th=[ 1565], 40.00th=[ 2311], 50.00th=[ 2737], 60.00th=[ 3097], | 70.00th=[ 3687], 80.00th=[ 5080], 90.00th=[ 8356], 95.00th=[11600], | 99.00th=[20579], 99.50th=[22676], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 736, max= 1504, per=24.48%, avg=1126.40, stdev=237.46, samples=10 iops : min= 46, max= 94, avg=70.40, stdev=14.84, samples=10 lat (usec) : 100=3.26%, 250=0.13%, 750=1.95%, 1000=9.64% lat (msec) : 2=5.47%, 4=20.31%, 10=27.73%, 20=30.60%, 50=0.91% cpu : usr=0.06%, sys=19.60%, ctx=2529, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4373KiB/s (4478kB/s), 1078KiB/s-1142KiB/s (1104kB/s-1170kB/s), io=23.4MiB (24.5MB), run=5314-5477msec WRITE: bw=4601KiB/s (4711kB/s), 1101KiB/s-1228KiB/s (1128kB/s-1257kB/s), io=24.6MiB (25.8MB), run=5314-5477msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7205: Thu Jul 25 02:04:07 2024 read: IOPS=1, BW=1826KiB/s (1870kB/s)(7168KiB/3926msec) clat (usec): min=561, max=918658, avg=526873.33, stdev=472669.74 lat (usec): min=562, max=918659, avg=526874.46, stdev=472669.80 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 562], 10.00th=[ 562], 20.00th=[ 644], | 30.00th=[ 65274], 40.00th=[ 65274], 50.00th=[884999], 60.00th=[901776], | 70.00th=[901776], 80.00th=[918553], 90.00th=[918553], 95.00th=[918553], | 99.00th=[918553], 99.50th=[918553], 99.90th=[918553], 99.95th=[918553], | 99.99th=[918553] bw ( KiB/s): min= 1946, max= 6144, per=58.60%, avg=3046.50, stdev=2065.56, samples=4 iops : min= 1, max= 6, avg= 2.75, stdev= 2.22, samples=4 write: IOPS=1, BW=1304KiB/s (1335kB/s)(5120KiB/3926msec) clat (usec): min=18571, max=46032, avg=36600.03, stdev=10897.92 lat (usec): min=18593, max=46133, avg=36658.15, stdev=10913.87 clat percentiles (usec): | 1.00th=[18482], 5.00th=[18482], 10.00th=[18482], 20.00th=[18482], | 30.00th=[34866], 40.00th=[34866], 50.00th=[39584], 60.00th=[39584], | 70.00th=[43779], 80.00th=[43779], 90.00th=[45876], 95.00th=[45876], | 99.00th=[45876], 99.50th=[45876], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 1946, max= 5840, per=45.03%, avg=3278.00, stdev=2219.34, samples=3 iops : min= 1, max= 5, avg= 2.67, stdev= 2.08, samples=3 lat (usec) : 750=16.67% lat (msec) : 20=8.33%, 50=33.33%, 100=8.33%, 1000=33.33% cpu : usr=0.03%, sys=19.62%, ctx=1185, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7207: Thu Jul 25 02:04:07 2024 read: IOPS=0, BW=990KiB/s (1014kB/s)(2048KiB/2068msec) clat (msec): min=61, max=738, avg=399.69, stdev=478.80 lat (msec): min=61, max=738, avg=399.69, stdev=478.80 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 62], 10.00th=[ 62], 20.00th=[ 62], | 30.00th=[ 62], 40.00th=[ 62], 50.00th=[ 62], 60.00th=[ 743], | 70.00th=[ 743], 80.00th=[ 743], 90.00th=[ 743], 95.00th=[ 743], | 99.00th=[ 743], 99.50th=[ 743], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 2048, max= 2048, per=39.39%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4952KiB/s (5070kB/s)(10.0MiB/2068msec) clat (msec): min=24, max=870, avg=120.56, stdev=263.96 lat (msec): min=24, max=870, avg=120.61, stdev=263.94 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 28], 50.00th=[ 28], 60.00th=[ 33], | 70.00th=[ 52], 80.00th=[ 52], 90.00th=[ 67], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 6144, max=10240, per=100.00%, avg=8192.00, stdev=2896.31, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (msec) : 50=50.00%, 100=33.33%, 750=8.33%, 1000=8.33% cpu : usr=0.05%, sys=19.25%, ctx=292, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7209: Thu Jul 25 02:04:07 2024 read: IOPS=1, BW=1560KiB/s (1597kB/s)(6144KiB/3939msec) clat (usec): min=298, max=887952, avg=153606.43, stdev=359981.11 lat (usec): min=299, max=887954, avg=153607.20, stdev=359981.27 clat percentiles (usec): | 1.00th=[ 297], 5.00th=[ 297], 10.00th=[ 297], 20.00th=[ 351], | 30.00th=[ 351], 40.00th=[ 355], 50.00th=[ 355], 60.00th=[ 424], | 70.00th=[ 32375], 80.00th=[ 32375], 90.00th=[884999], 95.00th=[884999], | 99.00th=[884999], 99.50th=[884999], 99.90th=[884999], 99.95th=[884999], | 99.99th=[884999] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1560KiB/s (1597kB/s)(6144KiB/3939msec) clat (msec): min=18, max=1895, avg=493.59, stdev=772.50 lat (msec): min=18, max=1895, avg=493.64, stdev=772.50 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 43], 50.00th=[ 43], 60.00th=[ 54], | 70.00th=[ 919], 80.00th=[ 919], 90.00th=[ 1888], 95.00th=[ 1888], | 99.00th=[ 1888], 99.50th=[ 1888], 99.90th=[ 1888], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 1928, max= 6144, per=55.72%, avg=4056.00, stdev=2108.28, samples=3 iops : min= 1, max= 6, avg= 3.67, stdev= 2.52, samples=3 lat (usec) : 500=33.33% lat (msec) : 20=8.33%, 50=25.00%, 100=8.33%, 1000=16.67% cpu : usr=0.00%, sys=8.43%, ctx=336, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7211: Thu Jul 25 02:04:07 2024 read: IOPS=2, BW=2228KiB/s (2281kB/s)(5120KiB/2298msec) clat (usec): min=296, max=564, avg=400.69, stdev=118.81 lat (usec): min=296, max=565, avg=401.32, stdev=119.05 clat percentiles (usec): | 1.00th=[ 297], 5.00th=[ 297], 10.00th=[ 297], 20.00th=[ 297], | 30.00th=[ 318], 40.00th=[ 318], 50.00th=[ 338], 60.00th=[ 338], | 70.00th=[ 490], 80.00th=[ 490], 90.00th=[ 562], 95.00th=[ 562], | 99.00th=[ 562], 99.50th=[ 562], 99.90th=[ 562], 99.95th=[ 562], | 99.99th=[ 562] write: IOPS=3, BW=3119KiB/s (3194kB/s)(7168KiB/2298msec) clat (msec): min=17, max=1920, avg=316.13, stdev=707.85 lat (msec): min=17, max=1920, avg=316.19, stdev=707.86 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 24], | 30.00th=[ 48], 40.00th=[ 48], 50.00th=[ 55], 60.00th=[ 63], | 70.00th=[ 63], 80.00th=[ 87], 90.00th=[ 1921], 95.00th=[ 1921], | 99.00th=[ 1921], 99.50th=[ 1921], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 2048, max= 2048, per=28.14%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 500=33.33%, 750=8.33% lat (msec) : 20=8.33%, 50=16.67%, 100=25.00% cpu : usr=0.04%, sys=7.66%, ctx=48, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5199KiB/s (5324kB/s), 990KiB/s-2228KiB/s (1014kB/s-2281kB/s), io=20.0MiB (20.0MB), run=2068-3939msec WRITE: bw=7279KiB/s (7454kB/s), 1304KiB/s-4952KiB/s (1335kB/s-5070kB/s), io=28.0MiB (29.4MB), run=2068-3939msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7204: Thu Jul 25 02:04:07 2024 read: IOPS=16, BW=16.9MiB/s (17.8MB/s)(7168KiB/413msec) clat (usec): min=16804, max=60272, avg=34871.61, stdev=17031.50 lat (usec): min=16805, max=60273, avg=34872.44, stdev=17031.64 clat percentiles (usec): | 1.00th=[16909], 5.00th=[16909], 10.00th=[16909], 20.00th=[23725], | 30.00th=[24249], 40.00th=[24249], 50.00th=[27919], 60.00th=[33817], | 70.00th=[33817], 80.00th=[56886], 90.00th=[60031], 95.00th=[60031], | 99.00th=[60031], 99.50th=[60031], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] write: IOPS=12, BW=12.1MiB/s (12.7MB/s)(5120KiB/413msec) clat (usec): min=22401, max=44089, avg=31107.46, stdev=10750.94 lat (usec): min=22468, max=44154, avg=31177.63, stdev=10747.22 clat percentiles (usec): | 1.00th=[22414], 5.00th=[22414], 10.00th=[22414], 20.00th=[22414], | 30.00th=[22938], 40.00th=[22938], 50.00th=[24511], 60.00th=[24511], | 70.00th=[41681], 80.00th=[41681], 90.00th=[44303], 95.00th=[44303], | 99.00th=[44303], 99.50th=[44303], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] lat (msec) : 20=8.33%, 50=75.00%, 100=16.67% cpu : usr=0.00%, sys=27.12%, ctx=82, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7206: Thu Jul 25 02:04:07 2024 read: IOPS=0, BW=865KiB/s (886kB/s)(2048KiB/2368msec) clat (msec): min=41, max=1871, avg=956.69, stdev=1294.03 lat (msec): min=41, max=1871, avg=956.69, stdev=1294.03 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 42], 50.00th=[ 42], 60.00th=[ 1871], | 70.00th=[ 1871], 80.00th=[ 1871], 90.00th=[ 1871], 95.00th=[ 1871], | 99.00th=[ 1871], 99.50th=[ 1871], 99.90th=[ 1871], 99.95th=[ 1871], | 99.99th=[ 1871] bw ( KiB/s): min= 2048, max= 2048, per=39.71%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4324KiB/s (4428kB/s)(10.0MiB/2368msec) clat (usec): min=15923, max=88569, avg=40991.74, stdev=24808.64 lat (usec): min=15984, max=88605, avg=41033.31, stdev=24804.80 clat percentiles (usec): | 1.00th=[15926], 5.00th=[15926], 10.00th=[15926], 20.00th=[19530], | 30.00th=[26608], 40.00th=[28705], 50.00th=[31065], 60.00th=[31589], | 70.00th=[59507], 80.00th=[59507], 90.00th=[76022], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=50.00%, 100=25.00% cpu : usr=0.00%, sys=5.11%, ctx=57, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7208: Thu Jul 25 02:04:07 2024 read: IOPS=1, BW=1548KiB/s (1585kB/s)(6144KiB/3970msec) clat (msec): min=25, max=835, avg=184.96, stdev=319.47 lat (msec): min=25, max=835, avg=184.96, stdev=319.47 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 40], | 30.00th=[ 40], 40.00th=[ 52], 50.00th=[ 52], 60.00th=[ 62], | 70.00th=[ 97], 80.00th=[ 97], 90.00th=[ 835], 95.00th=[ 835], | 99.00th=[ 835], 99.50th=[ 835], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 1906, max= 7627, per=92.41%, avg=4766.50, stdev=4045.36, samples=2 iops : min= 1, max= 7, avg= 4.00, stdev= 4.24, samples=2 write: IOPS=1, BW=1548KiB/s (1585kB/s)(6144KiB/3970msec) clat (msec): min=29, max=1883, avg=474.00, stdev=748.25 lat (msec): min=29, max=1883, avg=474.16, stdev=748.17 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 49], | 30.00th=[ 49], 40.00th=[ 56], 50.00th=[ 56], 60.00th=[ 60], | 70.00th=[ 768], 80.00th=[ 768], 90.00th=[ 1888], 95.00th=[ 1888], | 99.00th=[ 1888], 99.50th=[ 1888], 99.90th=[ 1888], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 1906, max= 4096, per=40.57%, avg=2930.25, stdev=1188.33, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.50, samples=4 lat (msec) : 50=33.33%, 100=41.67%, 1000=16.67% cpu : usr=0.03%, sys=5.06%, ctx=63, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7210: Thu Jul 25 02:04:07 2024 read: IOPS=4, BW=4719KiB/s (4832kB/s)(5120KiB/1085msec) clat (msec): min=14, max=754, avg=168.14, stdev=327.72 lat (msec): min=14, max=754, avg=168.14, stdev=327.72 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 18], 60.00th=[ 18], | 70.00th=[ 37], 80.00th=[ 37], 90.00th=[ 751], 95.00th=[ 751], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6606KiB/s (6765kB/s)(7168KiB/1085msec) clat (usec): min=15594, max=67595, avg=31211.44, stdev=19444.91 lat (usec): min=15612, max=67725, avg=31270.70, stdev=19472.62 clat percentiles (usec): | 1.00th=[15533], 5.00th=[15533], 10.00th=[15533], 20.00th=[15664], | 30.00th=[21890], 40.00th=[21890], 50.00th=[22938], 60.00th=[26870], | 70.00th=[26870], 80.00th=[47973], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min=14336, max=14336, per=100.00%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (msec) : 20=41.67%, 50=41.67%, 100=8.33%, 1000=8.33% cpu : usr=0.18%, sys=8.95%, ctx=46, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5159KiB/s (5282kB/s), 865KiB/s-16.9MiB/s (886kB/s-17.8MB/s), io=20.0MiB (20.0MB), run=413-3970msec WRITE: bw=7222KiB/s (7395kB/s), 1548KiB/s-12.1MiB/s (1585kB/s-12.7MB/s), io=28.0MiB (29.4MB), run=413-3970msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7219: Thu Jul 25 02:04:12 2024 read: IOPS=0, BW=3969KiB/s (4064kB/s)(12.0MiB/3096msec) clat (msec): min=101, max=1945, avg=1001.23, stdev=922.77 lat (msec): min=101, max=1945, avg=1001.23, stdev=922.77 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 103], 10.00th=[ 103], 20.00th=[ 103], | 30.00th=[ 103], 40.00th=[ 953], 50.00th=[ 953], 60.00th=[ 953], | 70.00th=[ 1938], 80.00th=[ 1938], 90.00th=[ 1938], 95.00th=[ 1938], | 99.00th=[ 1938], 99.50th=[ 1938], 99.90th=[ 1938], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.03%, sys=40.65%, ctx=947, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7221: Thu Jul 25 02:04:12 2024 read: IOPS=0, BW=1549KiB/s (1586kB/s)(4096KiB/2644msec) clat (nsec): min=154155k, max=154155k, avg=154154654.00, stdev= 0.00 lat (nsec): min=154157k, max=154157k, avg=154157326.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 155], | 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3098KiB/s (3173kB/s)(8192KiB/2644msec) clat (msec): min=311, max=2137, avg=1224.23, stdev=1291.48 lat (msec): min=311, max=2137, avg=1224.52, stdev=1291.47 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 313], 10.00th=[ 313], 20.00th=[ 313], | 30.00th=[ 313], 40.00th=[ 313], 50.00th=[ 313], 60.00th=[ 2140], | 70.00th=[ 2140], 80.00th=[ 2140], 90.00th=[ 2140], 95.00th=[ 2140], | 99.00th=[ 2140], 99.50th=[ 2140], 99.90th=[ 2140], 99.95th=[ 2140], | 99.99th=[ 2140] bw ( KiB/s): min= 8192, max= 8192, per=78.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.04%, sys=9.80%, ctx=296, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7222: Thu Jul 25 02:04:12 2024 write: IOPS=0, BW=3891KiB/s (3984kB/s)(12.0MiB/3158msec) clat (msec): min=200, max=1917, avg=1040.74, stdev=859.42 lat (msec): min=200, max=1918, avg=1040.99, stdev=859.51 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 201], 10.00th=[ 201], 20.00th=[ 201], | 30.00th=[ 201], 40.00th=[ 1003], 50.00th=[ 1003], 60.00th=[ 1003], | 70.00th=[ 1921], 80.00th=[ 1921], 90.00th=[ 1921], 95.00th=[ 1921], | 99.00th=[ 1921], 99.50th=[ 1921], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 7937, max= 8192, per=77.72%, avg=8064.50, stdev=180.31, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=9.82%, ctx=49, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7223: Thu Jul 25 02:04:12 2024 write: IOPS=1, BW=5673KiB/s (5809kB/s)(12.0MiB/2166msec) clat (msec): min=132, max=1183, avg=710.27, stdev=533.08 lat (msec): min=133, max=1184, avg=710.58, stdev=533.04 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 133], | 30.00th=[ 133], 40.00th=[ 818], 50.00th=[ 818], 60.00th=[ 818], | 70.00th=[ 1183], 80.00th=[ 1183], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 8175, max= 8192, per=78.87%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=14.69%, ctx=48, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5292KiB/s (5419kB/s), 1549KiB/s-3969KiB/s (1586kB/s-4064kB/s), io=16.0MiB (16.8MB), run=2644-3096msec WRITE: bw=10.1MiB/s (10.6MB/s), 3098KiB/s-5673KiB/s (3173kB/s-5809kB/s), io=32.0MiB (33.6MB), run=2166-3158msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7224: Thu Jul 25 02:04:12 2024 read: IOPS=0, BW=3828KiB/s (3920kB/s)(12.0MiB/3210msec) clat (msec): min=260, max=2160, avg=1055.16, stdev=987.28 lat (msec): min=260, max=2160, avg=1055.17, stdev=987.28 clat percentiles (msec): | 1.00th=[ 262], 5.00th=[ 262], 10.00th=[ 262], 20.00th=[ 262], | 30.00th=[ 262], 40.00th=[ 743], 50.00th=[ 743], 60.00th=[ 743], | 70.00th=[ 2165], 80.00th=[ 2165], 90.00th=[ 2165], 95.00th=[ 2165], | 99.00th=[ 2165], 99.50th=[ 2165], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=3.93%, ctx=61, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7225: Thu Jul 25 02:04:12 2024 read: IOPS=0, BW=1498KiB/s (1534kB/s)(4096KiB/2735msec) clat (nsec): min=282280k, max=282280k, avg=282280314.00, stdev= 0.00 lat (nsec): min=282282k, max=282282k, avg=282282027.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 284], 10.00th=[ 284], 20.00th=[ 284], | 30.00th=[ 284], 40.00th=[ 284], 50.00th=[ 284], 60.00th=[ 284], | 70.00th=[ 284], 80.00th=[ 284], 90.00th=[ 284], 95.00th=[ 284], | 99.00th=[ 284], 99.50th=[ 284], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2995KiB/s (3067kB/s)(8192KiB/2735msec) clat (msec): min=219, max=2167, avg=1193.92, stdev=1377.41 lat (msec): min=220, max=2168, avg=1194.14, stdev=1377.60 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 220], 10.00th=[ 220], 20.00th=[ 220], | 30.00th=[ 220], 40.00th=[ 220], 50.00th=[ 220], 60.00th=[ 2165], | 70.00th=[ 2165], 80.00th=[ 2165], 90.00th=[ 2165], 95.00th=[ 2165], | 99.00th=[ 2165], 99.50th=[ 2165], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 8192, max= 8192, per=81.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.04%, sys=6.66%, ctx=161, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7226: Thu Jul 25 02:04:12 2024 write: IOPS=0, BW=3779KiB/s (3869kB/s)(12.0MiB/3252msec) clat (msec): min=251, max=2158, avg=1062.50, stdev=984.86 lat (msec): min=252, max=2158, avg=1062.75, stdev=984.88 clat percentiles (msec): | 1.00th=[ 253], 5.00th=[ 253], 10.00th=[ 253], 20.00th=[ 253], | 30.00th=[ 253], 40.00th=[ 776], 50.00th=[ 776], 60.00th=[ 776], | 70.00th=[ 2165], 80.00th=[ 2165], 90.00th=[ 2165], 95.00th=[ 2165], | 99.00th=[ 2165], 99.50th=[ 2165], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 8110, max= 8192, per=80.90%, avg=8151.00, stdev=57.98, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.03%, sys=2.77%, ctx=35, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7227: Thu Jul 25 02:04:12 2024 write: IOPS=1, BW=5009KiB/s (5130kB/s)(12.0MiB/2453msec) clat (msec): min=85, max=2087, avg=803.40, stdev=1114.66 lat (msec): min=85, max=2087, avg=803.67, stdev=1114.65 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 86], 10.00th=[ 86], 20.00th=[ 86], | 30.00th=[ 86], 40.00th=[ 239], 50.00th=[ 239], 60.00th=[ 239], | 70.00th=[ 2089], 80.00th=[ 2089], 90.00th=[ 2089], 95.00th=[ 2089], | 99.00th=[ 2089], 99.50th=[ 2089], 99.90th=[ 2089], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 8192, max= 8192, per=81.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.04%, sys=6.65%, ctx=110, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5104KiB/s (5227kB/s), 1498KiB/s-3828KiB/s (1534kB/s-3920kB/s), io=16.0MiB (16.8MB), run=2735-3210msec WRITE: bw=9.84MiB/s (10.3MB/s), 2995KiB/s-5009KiB/s (3067kB/s-5130kB/s), io=32.0MiB (33.6MB), run=2453-3252msec