== sanity test 398b: DIO and buffer IO race ============== 10:41:15 (1724769675) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.25153 s, 40.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6550: Tue Aug 27 10:41:31 2024 read: IOPS=110, BW=441KiB/s (452kB/s)(5992KiB/13581msec) clat (usec): min=55, max=25464, avg=4887.76, stdev=2727.42 lat (usec): min=56, max=25465, avg=4888.67, stdev=2727.75 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 109], 10.00th=[ 2311], 20.00th=[ 2868], | 30.00th=[ 3326], 40.00th=[ 3851], 50.00th=[ 4490], 60.00th=[ 5145], | 70.00th=[ 5932], 80.00th=[ 6783], 90.00th=[ 8094], 95.00th=[ 9503], | 99.00th=[13829], 99.50th=[16057], 99.90th=[19006], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 175, max= 608, per=25.34%, avg=438.93, stdev=76.51, samples=27 iops : min= 43, max= 152, avg=109.59, stdev=19.20, samples=27 write: IOPS=115, BW=464KiB/s (475kB/s)(6296KiB/13581msec) clat (usec): min=422, max=227961, avg=3929.26, stdev=6588.78 lat (usec): min=423, max=227962, avg=3930.21, stdev=6588.72 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 562], 10.00th=[ 603], 20.00th=[ 750], | 30.00th=[ 988], 40.00th=[ 2040], 50.00th=[ 2999], 60.00th=[ 3752], | 70.00th=[ 4948], 80.00th=[ 6521], 90.00th=[ 8586], 95.00th=[ 10421], | 99.00th=[ 14222], 99.50th=[ 16188], 99.90th=[ 27919], 99.95th=[227541], | 99.99th=[227541] bw ( KiB/s): min= 183, max= 592, per=25.37%, avg=459.93, stdev=85.16, samples=27 iops : min= 45, max= 148, avg=114.85, stdev=21.36, samples=27 lat (usec) : 100=2.28%, 250=0.55%, 500=0.20%, 750=10.03%, 1000=5.44% lat (msec) : 2=5.24%, 4=29.07%, 10=42.15%, 20=4.92%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=23.78%, ctx=6256, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6551: Tue Aug 27 10:41:31 2024 read: IOPS=109, BW=440KiB/s (450kB/s)(5940KiB/13509msec) clat (usec): min=54, max=16994, avg=4858.65, stdev=2585.14 lat (usec): min=55, max=16994, avg=4860.01, stdev=2586.46 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 1811], 10.00th=[ 2409], 20.00th=[ 2835], | 30.00th=[ 3228], 40.00th=[ 3785], 50.00th=[ 4424], 60.00th=[ 5145], | 70.00th=[ 5866], 80.00th=[ 6783], 90.00th=[ 8225], 95.00th=[ 9503], | 99.00th=[13173], 99.50th=[14222], 99.90th=[16581], 99.95th=[16909], | 99.99th=[16909] bw ( KiB/s): min= 207, max= 576, per=25.34%, avg=438.89, stdev=80.08, samples=27 iops : min= 51, max= 144, avg=109.63, stdev=20.09, samples=27 write: IOPS=117, BW=470KiB/s (481kB/s)(6348KiB/13509msec) clat (usec): min=439, max=256506, avg=3918.26, stdev=7147.94 lat (usec): min=440, max=256507, avg=3919.37, stdev=7148.06 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 553], 10.00th=[ 611], 20.00th=[ 750], | 30.00th=[ 963], 40.00th=[ 2040], 50.00th=[ 3097], 60.00th=[ 3818], | 70.00th=[ 4883], 80.00th=[ 6390], 90.00th=[ 8455], 95.00th=[ 10159], | 99.00th=[ 14222], 99.50th=[ 16450], 99.90th=[ 19006], 99.95th=[256902], | 99.99th=[256902] bw ( KiB/s): min= 255, max= 728, per=25.89%, avg=469.41, stdev=85.81, samples=27 iops : min= 63, max= 182, avg=117.26, stdev=21.54, samples=27 lat (usec) : 100=1.73%, 250=0.52%, 500=0.39%, 750=9.96%, 1000=5.79% lat (msec) : 2=4.95%, 4=29.98%, 10=41.96%, 20=4.69%, 500=0.03% cpu : usr=0.12%, sys=23.70%, ctx=6235, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6553: Tue Aug 27 10:41:31 2024 read: IOPS=108, BW=435KiB/s (445kB/s)(6028KiB/13859msec) clat (usec): min=49, max=258711, avg=5168.93, stdev=7129.76 lat (usec): min=49, max=258712, avg=5173.45, stdev=7130.31 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 112], 10.00th=[ 2278], 20.00th=[ 2835], | 30.00th=[ 3392], 40.00th=[ 4015], 50.00th=[ 4621], 60.00th=[ 5276], | 70.00th=[ 6063], 80.00th=[ 7046], 90.00th=[ 8291], 95.00th=[ 10028], | 99.00th=[ 13698], 99.50th=[ 15664], 99.90th=[ 30802], 99.95th=[258999], | 99.99th=[258999] bw ( KiB/s): min= 175, max= 568, per=24.79%, avg=429.41, stdev=74.76, samples=27 iops : min= 43, max= 142, avg=107.22, stdev=18.80, samples=27 write: IOPS=112, BW=452KiB/s (463kB/s)(6260KiB/13859msec) clat (usec): min=405, max=27893, avg=3824.63, stdev=3346.20 lat (usec): min=406, max=27894, avg=3825.69, stdev=3346.00 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 553], 10.00th=[ 627], 20.00th=[ 799], | 30.00th=[ 1004], 40.00th=[ 2278], 50.00th=[ 3130], 60.00th=[ 3982], | 70.00th=[ 4948], 80.00th=[ 6456], 90.00th=[ 8586], 95.00th=[10028], | 99.00th=[13698], 99.50th=[16057], 99.90th=[20579], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 215, max= 574, per=24.47%, avg=443.59, stdev=80.05, samples=27 iops : min= 53, max= 143, avg=110.78, stdev=20.08, samples=27 lat (usec) : 50=0.03%, 100=2.02%, 250=1.01%, 500=0.36%, 750=8.66% lat (usec) : 1000=6.25% lat (msec) : 2=4.95%, 4=27.05%, 10=44.56%, 20=4.88%, 50=0.20% lat (msec) : 500=0.03% cpu : usr=0.21%, sys=23.68%, ctx=6229, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6555: Tue Aug 27 10:41:31 2024 read: IOPS=111, BW=446KiB/s (457kB/s)(6052KiB/13562msec) clat (usec): min=48, max=259619, avg=5140.43, stdev=7041.98 lat (usec): min=49, max=259620, avg=5141.65, stdev=7041.89 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 2057], 10.00th=[ 2474], 20.00th=[ 2933], | 30.00th=[ 3359], 40.00th=[ 3949], 50.00th=[ 4621], 60.00th=[ 5276], | 70.00th=[ 5997], 80.00th=[ 6849], 90.00th=[ 8094], 95.00th=[ 9503], | 99.00th=[ 13042], 99.50th=[ 15795], 99.90th=[ 20841], 99.95th=[258999], | 99.99th=[258999] bw ( KiB/s): min= 191, max= 584, per=25.77%, avg=446.26, stdev=77.69, samples=27 iops : min= 47, max= 146, avg=111.44, stdev=19.52, samples=27 write: IOPS=114, BW=460KiB/s (471kB/s)(6236KiB/13562msec) clat (usec): min=447, max=30755, avg=3663.34, stdev=3226.79 lat (usec): min=448, max=30756, avg=3665.36, stdev=3227.95 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 553], 10.00th=[ 611], 20.00th=[ 783], | 30.00th=[ 971], 40.00th=[ 2073], 50.00th=[ 3032], 60.00th=[ 3720], | 70.00th=[ 4621], 80.00th=[ 6128], 90.00th=[ 8160], 95.00th=[ 9765], | 99.00th=[13698], 99.50th=[16581], 99.90th=[18482], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 207, max= 664, per=25.30%, avg=458.70, stdev=84.37, samples=27 iops : min= 51, max= 166, avg=114.56, stdev=21.16, samples=27 lat (usec) : 50=0.03%, 100=1.46%, 250=0.46%, 500=0.39%, 750=9.05% lat (usec) : 1000=6.09% lat (msec) : 2=4.85%, 4=30.34%, 10=43.33%, 20=3.87%, 50=0.10% lat (msec) : 500=0.03% cpu : usr=0.21%, sys=23.93%, ctx=6276, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1733KiB/s (1774kB/s), 435KiB/s-446KiB/s (445kB/s-457kB/s), io=23.4MiB (24.6MB), run=13509-13859msec WRITE: bw=1814KiB/s (1858kB/s), 452KiB/s-470KiB/s (463kB/s-481kB/s), io=24.6MiB (25.7MB), run=13509-13859msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6552: Tue Aug 27 10:41:35 2024 read: IOPS=85, BW=344KiB/s (352kB/s)(5992KiB/17438msec) clat (usec): min=1576, max=261085, avg=4869.71, stdev=7556.75 lat (usec): min=1576, max=261086, avg=4870.42, stdev=7556.77 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1926], 10.00th=[ 2024], 20.00th=[ 2212], | 30.00th=[ 2474], 40.00th=[ 2835], 50.00th=[ 3294], 60.00th=[ 3851], | 70.00th=[ 4686], 80.00th=[ 6390], 90.00th=[ 10159], 95.00th=[ 12780], | 99.00th=[ 17433], 99.50th=[ 20841], 99.90th=[ 33424], 99.95th=[261096], | 99.99th=[261096] bw ( KiB/s): min= 96, max= 744, per=24.80%, avg=338.97, stdev=152.34, samples=34 iops : min= 24, max= 186, avg=84.71, stdev=38.08, samples=34 write: IOPS=90, BW=361KiB/s (370kB/s)(6296KiB/17438msec) clat (usec): min=2209, max=34197, avg=6399.17, stdev=3857.07 lat (usec): min=2209, max=34198, avg=6400.23, stdev=3856.99 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2933], 10.00th=[ 3163], 20.00th=[ 3458], | 30.00th=[ 3818], 40.00th=[ 4359], 50.00th=[ 5014], 60.00th=[ 5997], | 70.00th=[ 7373], 80.00th=[ 9110], 90.00th=[11469], 95.00th=[13304], | 99.00th=[20055], 99.50th=[23987], 99.90th=[33162], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 80, max= 744, per=24.89%, avg=356.15, stdev=155.37, samples=34 iops : min= 20, max= 186, avg=89.00, stdev=38.83, samples=34 lat (msec) : 2=4.13%, 4=44.14%, 10=38.54%, 20=12.43%, 50=0.72% lat (msec) : 500=0.03% cpu : usr=0.21%, sys=11.00%, ctx=3666, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6554: Tue Aug 27 10:41:35 2024 read: IOPS=84, BW=340KiB/s (348kB/s)(5940KiB/17481msec) clat (usec): min=1575, max=264835, avg=4792.12, stdev=7647.96 lat (usec): min=1576, max=264836, avg=4792.92, stdev=7648.10 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1909], 10.00th=[ 2040], 20.00th=[ 2212], | 30.00th=[ 2442], 40.00th=[ 2802], 50.00th=[ 3195], 60.00th=[ 3752], | 70.00th=[ 4752], 80.00th=[ 6325], 90.00th=[ 9765], 95.00th=[ 12780], | 99.00th=[ 18744], 99.50th=[ 21627], 99.90th=[ 33424], 99.95th=[265290], | 99.99th=[265290] bw ( KiB/s): min= 72, max= 744, per=24.69%, avg=337.47, stdev=162.19, samples=34 iops : min= 18, max= 186, avg=84.26, stdev=40.56, samples=34 write: IOPS=90, BW=363KiB/s (372kB/s)(6348KiB/17481msec) clat (usec): min=2267, max=33056, avg=6489.82, stdev=3867.86 lat (usec): min=2267, max=33057, avg=6490.73, stdev=3867.86 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2966], 10.00th=[ 3195], 20.00th=[ 3523], | 30.00th=[ 3818], 40.00th=[ 4359], 50.00th=[ 5145], 60.00th=[ 5997], | 70.00th=[ 7570], 80.00th=[ 9241], 90.00th=[11600], 95.00th=[13829], | 99.00th=[20579], 99.50th=[22152], 99.90th=[29230], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 128, max= 702, per=24.88%, avg=356.06, stdev=151.22, samples=34 iops : min= 32, max= 175, avg=88.91, stdev=37.79, samples=34 lat (msec) : 2=3.91%, 4=44.08%, 10=39.36%, 20=11.69%, 50=0.94% lat (msec) : 500=0.03% cpu : usr=0.17%, sys=11.06%, ctx=3756, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6556: Tue Aug 27 10:41:35 2024 read: IOPS=86, BW=345KiB/s (353kB/s)(6028KiB/17484msec) clat (usec): min=1602, max=22775, avg=4695.04, stdev=3428.36 lat (usec): min=1603, max=22776, avg=4696.04, stdev=3428.21 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1909], 10.00th=[ 2057], 20.00th=[ 2278], | 30.00th=[ 2540], 40.00th=[ 2933], 50.00th=[ 3425], 60.00th=[ 3949], | 70.00th=[ 4817], 80.00th=[ 6521], 90.00th=[ 9765], 95.00th=[12125], | 99.00th=[17433], 99.50th=[19268], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 104, max= 728, per=24.91%, avg=340.53, stdev=152.73, samples=34 iops : min= 26, max= 182, avg=85.03, stdev=38.17, samples=34 write: IOPS=89, BW=358KiB/s (367kB/s)(6260KiB/17484msec) clat (msec): min=2, max=267, avg= 6.61, stdev= 7.67 lat (msec): min=2, max=267, avg= 6.61, stdev= 7.67 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 15], | 99.00th=[ 21], 99.50th=[ 24], 99.90th=[ 35], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 112, max= 784, per=24.55%, avg=351.35, stdev=154.17, samples=34 iops : min= 28, max= 196, avg=87.74, stdev=38.56, samples=34 lat (msec) : 2=3.84%, 4=43.75%, 10=39.62%, 20=11.98%, 50=0.78% lat (msec) : 500=0.03% cpu : usr=0.19%, sys=11.29%, ctx=3715, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6557: Tue Aug 27 10:41:35 2024 read: IOPS=86, BW=345KiB/s (353kB/s)(6052KiB/17560msec) clat (usec): min=1481, max=33128, avg=4595.31, stdev=3489.34 lat (usec): min=1481, max=33128, avg=4596.08, stdev=3489.38 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1909], 10.00th=[ 2024], 20.00th=[ 2212], | 30.00th=[ 2507], 40.00th=[ 2835], 50.00th=[ 3294], 60.00th=[ 3916], | 70.00th=[ 4555], 80.00th=[ 6063], 90.00th=[ 9503], 95.00th=[12256], | 99.00th=[16581], 99.50th=[20841], 99.90th=[25297], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 96, max= 752, per=25.14%, avg=343.71, stdev=155.66, samples=35 iops : min= 24, max= 188, avg=85.89, stdev=38.94, samples=35 write: IOPS=88, BW=355KiB/s (364kB/s)(6236KiB/17560msec) clat (msec): min=2, max=247, avg= 6.75, stdev= 7.22 lat (msec): min=2, max=247, avg= 6.75, stdev= 7.22 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 21], 99.50th=[ 24], 99.90th=[ 30], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min= 104, max= 728, per=24.72%, avg=353.77, stdev=153.79, samples=35 iops : min= 26, max= 182, avg=88.40, stdev=38.47, samples=35 lat (msec) : 2=4.36%, 4=42.32%, 10=40.07%, 20=12.43%, 50=0.78% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=10.91%, ctx=3781, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1367KiB/s (1400kB/s), 340KiB/s-345KiB/s (348kB/s-353kB/s), io=23.4MiB (24.6MB), run=17438-17560msec WRITE: bw=1432KiB/s (1466kB/s), 355KiB/s-363KiB/s (364kB/s-372kB/s), io=24.6MiB (25.7MB), run=17438-17560msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6583: Tue Aug 27 10:41:42 2024 read: IOPS=57, BW=927KiB/s (950kB/s)(5920KiB/6384msec) clat (usec): min=59, max=52107, avg=11670.64, stdev=4511.78 lat (usec): min=60, max=52108, avg=11671.41, stdev=4511.78 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 7046], 10.00th=[ 8160], 20.00th=[ 9372], | 30.00th=[ 9896], 40.00th=[10683], 50.00th=[11338], 60.00th=[12125], | 70.00th=[13173], 80.00th=[13960], 90.00th=[15270], 95.00th=[16450], | 99.00th=[28443], 99.50th=[34866], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 800, max= 1149, per=25.32%, avg=914.25, stdev=114.71, samples=12 iops : min= 50, max= 71, avg=57.00, stdev= 6.99, samples=12 write: IOPS=62, BW=997KiB/s (1021kB/s)(6368KiB/6384msec) clat (usec): min=756, max=41207, avg=5120.20, stdev=4907.37 lat (usec): min=758, max=41209, avg=5121.46, stdev=4907.35 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 840], 10.00th=[ 988], 20.00th=[ 1614], | 30.00th=[ 2409], 40.00th=[ 2966], 50.00th=[ 3589], 60.00th=[ 4555], | 70.00th=[ 5800], 80.00th=[ 7308], 90.00th=[10683], 95.00th=[15664], | 99.00th=[22414], 99.50th=[39584], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 798, max= 1184, per=26.03%, avg=989.00, stdev=105.65, samples=12 iops : min= 49, max= 74, avg=61.67, stdev= 6.80, samples=12 lat (usec) : 100=1.30%, 250=0.26%, 1000=5.21% lat (msec) : 2=5.60%, 4=18.49%, 10=29.82%, 20=37.63%, 50=1.56% lat (msec) : 100=0.13% cpu : usr=0.14%, sys=20.12%, ctx=2756, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6584: Tue Aug 27 10:41:42 2024 read: IOPS=54, BW=872KiB/s (893kB/s)(5744KiB/6586msec) clat (usec): min=60, max=50461, avg=11909.30, stdev=4505.40 lat (usec): min=60, max=50461, avg=11910.07, stdev=4505.34 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 7832], 10.00th=[ 8356], 20.00th=[ 9372], | 30.00th=[10159], 40.00th=[10814], 50.00th=[11469], 60.00th=[11863], | 70.00th=[12780], 80.00th=[13698], 90.00th=[16057], 95.00th=[19006], | 99.00th=[31851], 99.50th=[36963], 99.90th=[50594], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 640, max= 992, per=23.99%, avg=866.15, stdev=94.63, samples=13 iops : min= 40, max= 62, avg=54.00, stdev= 5.93, samples=13 write: IOPS=62, BW=994KiB/s (1017kB/s)(6544KiB/6586msec) clat (usec): min=740, max=47947, avg=5613.96, stdev=6132.31 lat (usec): min=741, max=47948, avg=5615.52, stdev=6132.15 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 832], 10.00th=[ 979], 20.00th=[ 1336], | 30.00th=[ 2343], 40.00th=[ 2802], 50.00th=[ 3490], 60.00th=[ 4490], | 70.00th=[ 5997], 80.00th=[ 8094], 90.00th=[13304], 95.00th=[17171], | 99.00th=[27132], 99.50th=[39060], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 736, max= 1376, per=25.91%, avg=984.23, stdev=186.53, samples=13 iops : min= 46, max= 86, avg=61.38, stdev=11.64, samples=13 lat (usec) : 100=0.52%, 250=0.39%, 750=0.13%, 1000=5.60% lat (msec) : 2=7.68%, 4=17.06%, 10=26.43%, 20=38.41%, 50=3.65% lat (msec) : 100=0.13% cpu : usr=0.11%, sys=19.76%, ctx=2745, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6585: Tue Aug 27 10:41:42 2024 read: IOPS=58, BW=943KiB/s (966kB/s)(6256KiB/6632msec) clat (usec): min=60, max=35738, avg=11644.28, stdev=3707.20 lat (usec): min=60, max=35739, avg=11644.99, stdev=3707.20 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 7570], 10.00th=[ 8455], 20.00th=[ 9241], | 30.00th=[10028], 40.00th=[10814], 50.00th=[11338], 60.00th=[11994], | 70.00th=[12649], 80.00th=[13960], 90.00th=[15270], 95.00th=[16909], | 99.00th=[27657], 99.50th=[30016], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 672, max= 1117, per=25.82%, avg=932.31, stdev=137.71, samples=13 iops : min= 42, max= 69, avg=58.08, stdev= 8.50, samples=13 write: IOPS=56, BW=910KiB/s (931kB/s)(6032KiB/6632msec) clat (usec): min=744, max=61600, avg=5451.35, stdev=5877.62 lat (usec): min=745, max=61601, avg=5452.54, stdev=5877.61 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 824], 10.00th=[ 930], 20.00th=[ 1434], | 30.00th=[ 2474], 40.00th=[ 2999], 50.00th=[ 3687], 60.00th=[ 4752], | 70.00th=[ 5932], 80.00th=[ 7504], 90.00th=[11600], 95.00th=[15795], | 99.00th=[30540], 99.50th=[42206], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 608, max= 1184, per=23.12%, avg=878.23, stdev=184.75, samples=13 iops : min= 38, max= 74, avg=54.69, stdev=11.49, samples=13 lat (usec) : 100=0.65%, 250=0.26%, 750=0.13%, 1000=5.99% lat (msec) : 2=5.08%, 4=14.84%, 10=30.34%, 20=40.76%, 50=1.82% lat (msec) : 100=0.13% cpu : usr=0.14%, sys=19.59%, ctx=2864, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6586: Tue Aug 27 10:41:42 2024 read: IOPS=59, BW=944KiB/s (967kB/s)(6032KiB/6387msec) clat (usec): min=58, max=31702, avg=11568.56, stdev=4062.48 lat (usec): min=59, max=31703, avg=11569.27, stdev=4062.51 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 7111], 10.00th=[ 8225], 20.00th=[ 9110], | 30.00th=[ 9765], 40.00th=[10552], 50.00th=[11207], 60.00th=[11994], | 70.00th=[12649], 80.00th=[13698], 90.00th=[15795], 95.00th=[18220], | 99.00th=[26084], 99.50th=[28705], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 832, max= 1024, per=25.84%, avg=933.00, stdev=52.59, samples=12 iops : min= 52, max= 64, avg=58.17, stdev= 3.33, samples=12 write: IOPS=61, BW=979KiB/s (1003kB/s)(6256KiB/6387msec) clat (usec): min=775, max=32910, avg=5126.87, stdev=5199.96 lat (usec): min=776, max=32910, avg=5129.28, stdev=5199.64 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 832], 10.00th=[ 938], 20.00th=[ 1336], | 30.00th=[ 2311], 40.00th=[ 2737], 50.00th=[ 3195], 60.00th=[ 4178], | 70.00th=[ 5342], 80.00th=[ 7635], 90.00th=[12125], 95.00th=[16057], | 99.00th=[29754], 99.50th=[30540], 99.90th=[32900], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 672, max= 1309, per=25.89%, avg=983.58, stdev=196.03, samples=12 iops : min= 42, max= 81, avg=61.33, stdev=12.18, samples=12 lat (usec) : 100=1.30%, 250=0.26%, 1000=5.86% lat (msec) : 2=7.16%, 4=16.93%, 10=29.17%, 20=36.85%, 50=2.47% cpu : usr=0.09%, sys=19.72%, ctx=2761, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3612KiB/s (3698kB/s), 872KiB/s-944KiB/s (893kB/s-967kB/s), io=23.4MiB (24.5MB), run=6384-6632msec WRITE: bw=3800KiB/s (3891kB/s), 910KiB/s-997KiB/s (931kB/s-1021kB/s), io=24.6MiB (25.8MB), run=6384-6632msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6588: Tue Aug 27 10:41:42 2024 read: IOPS=58, BW=933KiB/s (955kB/s)(5920KiB/6348msec) clat (usec): min=1913, max=45365, avg=6714.81, stdev=5784.58 lat (usec): min=1914, max=45366, avg=6715.55, stdev=5784.58 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2212], 10.00th=[ 2409], 20.00th=[ 2737], | 30.00th=[ 3032], 40.00th=[ 3425], 50.00th=[ 4293], 60.00th=[ 5604], | 70.00th=[ 8225], 80.00th=[10028], 90.00th=[13304], 95.00th=[16712], | 99.00th=[30278], 99.50th=[35914], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 736, max= 1216, per=25.26%, avg=914.33, stdev=142.07, samples=12 iops : min= 46, max= 76, avg=57.00, stdev= 8.87, samples=12 write: IOPS=62, BW=1003KiB/s (1027kB/s)(6368KiB/6348msec) clat (usec): min=2684, max=66100, avg=9648.87, stdev=6442.92 lat (usec): min=2685, max=66101, avg=9650.04, stdev=6442.89 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3490], 10.00th=[ 3949], 20.00th=[ 4621], | 30.00th=[ 5735], 40.00th=[ 6849], 50.00th=[ 8029], 60.00th=[ 9503], | 70.00th=[11076], 80.00th=[13435], 90.00th=[17433], 95.00th=[22676], | 99.00th=[28181], 99.50th=[42730], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 704, max= 1277, per=25.98%, avg=988.92, stdev=159.48, samples=12 iops : min= 44, max= 79, avg=61.67, stdev= 9.84, samples=12 lat (msec) : 2=0.39%, 4=27.73%, 10=43.36%, 20=23.18%, 50=5.21% lat (msec) : 100=0.13% cpu : usr=0.14%, sys=9.86%, ctx=1164, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6589: Tue Aug 27 10:41:42 2024 read: IOPS=54, BW=873KiB/s (894kB/s)(5744KiB/6579msec) clat (usec): min=1846, max=62537, avg=6929.85, stdev=5834.28 lat (usec): min=1847, max=62538, avg=6930.59, stdev=5834.26 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2180], 10.00th=[ 2376], 20.00th=[ 2704], | 30.00th=[ 3130], 40.00th=[ 3654], 50.00th=[ 4883], 60.00th=[ 6390], | 70.00th=[ 8979], 80.00th=[10814], 90.00th=[13960], 95.00th=[16319], | 99.00th=[27919], 99.50th=[36439], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 704, max= 1056, per=23.64%, avg=855.67, stdev=98.19, samples=12 iops : min= 44, max= 66, avg=53.33, stdev= 6.01, samples=12 write: IOPS=62, BW=995KiB/s (1019kB/s)(6544KiB/6579msec) clat (usec): min=2739, max=51380, avg=9904.72, stdev=6989.95 lat (usec): min=2740, max=51381, avg=9905.95, stdev=6989.95 clat percentiles (usec): | 1.00th=[ 3097], 5.00th=[ 3523], 10.00th=[ 3752], 20.00th=[ 4555], | 30.00th=[ 5932], 40.00th=[ 6980], 50.00th=[ 8455], 60.00th=[ 9634], | 70.00th=[11207], 80.00th=[13042], 90.00th=[16909], 95.00th=[21627], | 99.00th=[43779], 99.50th=[45876], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 672, max= 1216, per=25.42%, avg=967.67, stdev=169.62, samples=12 iops : min= 42, max= 76, avg=60.33, stdev=10.71, samples=12 lat (msec) : 2=0.39%, 4=26.82%, 10=41.93%, 20=26.95%, 50=3.65% lat (msec) : 100=0.26% cpu : usr=0.14%, sys=9.70%, ctx=1208, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6590: Tue Aug 27 10:41:42 2024 read: IOPS=59, BW=945KiB/s (968kB/s)(6256KiB/6618msec) clat (usec): min=1881, max=36626, avg=6795.04, stdev=5148.83 lat (usec): min=1882, max=36627, avg=6795.75, stdev=5148.83 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2212], 10.00th=[ 2409], 20.00th=[ 2737], | 30.00th=[ 3130], 40.00th=[ 3556], 50.00th=[ 4490], 60.00th=[ 6259], | 70.00th=[ 8848], 80.00th=[11076], 90.00th=[13304], 95.00th=[16319], | 99.00th=[24249], 99.50th=[28443], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 704, max= 1248, per=25.64%, avg=928.00, stdev=184.08, samples=12 iops : min= 44, max= 78, avg=58.00, stdev=11.50, samples=12 write: IOPS=56, BW=911KiB/s (933kB/s)(6032KiB/6618msec) clat (usec): min=2539, max=96425, avg=10429.54, stdev=7394.53 lat (usec): min=2540, max=96426, avg=10430.86, stdev=7394.44 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 3458], 10.00th=[ 3949], 20.00th=[ 5080], | 30.00th=[ 6718], 40.00th=[ 7570], 50.00th=[ 8979], 60.00th=[10290], | 70.00th=[11863], 80.00th=[14746], 90.00th=[18220], 95.00th=[22414], | 99.00th=[31851], 99.50th=[42730], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 672, max= 1216, per=23.54%, avg=896.00, stdev=182.56, samples=12 iops : min= 42, max= 76, avg=56.00, stdev=11.41, samples=12 lat (msec) : 2=0.26%, 4=27.73%, 10=39.06%, 20=27.86%, 50=4.95% lat (msec) : 100=0.13% cpu : usr=0.14%, sys=9.66%, ctx=1221, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6591: Tue Aug 27 10:41:42 2024 read: IOPS=59, BW=951KiB/s (973kB/s)(6032KiB/6345msec) clat (usec): min=1652, max=43991, avg=6715.54, stdev=5503.72 lat (usec): min=1653, max=43992, avg=6716.31, stdev=5503.70 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2180], 10.00th=[ 2409], 20.00th=[ 2769], | 30.00th=[ 3064], 40.00th=[ 3621], 50.00th=[ 4424], 60.00th=[ 6063], | 70.00th=[ 8225], 80.00th=[10421], 90.00th=[13042], 95.00th=[15926], | 99.00th=[26870], 99.50th=[36963], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 704, max= 1088, per=25.94%, avg=938.67, stdev=109.44, samples=12 iops : min= 44, max= 68, avg=58.67, stdev= 6.84, samples=12 write: IOPS=61, BW=986KiB/s (1010kB/s)(6256KiB/6345msec) clat (usec): min=2616, max=64217, avg=9631.12, stdev=6558.45 lat (usec): min=2617, max=64218, avg=9632.52, stdev=6558.39 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3490], 10.00th=[ 3916], 20.00th=[ 4621], | 30.00th=[ 5735], 40.00th=[ 6783], 50.00th=[ 8029], 60.00th=[ 9503], | 70.00th=[10814], 80.00th=[13042], 90.00th=[17433], 95.00th=[20841], | 99.00th=[36963], 99.50th=[39584], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 768, max= 1440, per=25.85%, avg=984.00, stdev=197.79, samples=12 iops : min= 48, max= 90, avg=61.50, stdev=12.36, samples=12 lat (msec) : 2=1.04%, 4=26.95%, 10=43.10%, 20=24.61%, 50=4.17% lat (msec) : 100=0.13% cpu : usr=0.14%, sys=9.71%, ctx=1156, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3619KiB/s (3706kB/s), 873KiB/s-951KiB/s (894kB/s-973kB/s), io=23.4MiB (24.5MB), run=6345-6618msec WRITE: bw=3808KiB/s (3899kB/s), 911KiB/s-1003KiB/s (933kB/s-1027kB/s), io=24.6MiB (25.8MB), run=6345-6618msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6602: Tue Aug 27 10:41:46 2024 read: IOPS=6, BW=6814KiB/s (6977kB/s)(7168KiB/1052msec) clat (usec): min=264, max=629734, avg=122253.34, stdev=233276.59 lat (usec): min=264, max=629735, avg=122254.34, stdev=233276.52 clat percentiles (usec): | 1.00th=[ 265], 5.00th=[ 265], 10.00th=[ 265], 20.00th=[ 416], | 30.00th=[ 424], 40.00th=[ 424], 50.00th=[ 433], 60.00th=[ 43779], | 70.00th=[ 43779], 80.00th=[181404], 90.00th=[633340], 95.00th=[633340], | 99.00th=[633340], 99.50th=[633340], 99.90th=[633340], 99.95th=[633340], | 99.99th=[633340] bw ( KiB/s): min= 2048, max= 8159, per=68.93%, avg=5103.50, stdev=4321.13, samples=2 iops : min= 2, max= 7, avg= 4.50, stdev= 3.54, samples=2 write: IOPS=4, BW=4867KiB/s (4984kB/s)(5120KiB/1052msec) clat (usec): min=17516, max=69866, avg=29504.27, stdev=22648.57 lat (usec): min=17541, max=69941, avg=29553.29, stdev=22663.38 clat percentiles (usec): | 1.00th=[17433], 5.00th=[17433], 10.00th=[17433], 20.00th=[17433], | 30.00th=[17433], 40.00th=[17433], 50.00th=[20579], 60.00th=[20579], | 70.00th=[22152], 80.00th=[22152], 90.00th=[69731], 95.00th=[69731], | 99.00th=[69731], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 4079, max= 4079, per=39.35%, avg=4079.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (usec) : 500=33.33% lat (msec) : 20=16.67%, 50=25.00%, 100=8.33%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=32.92%, ctx=383, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6603: Tue Aug 27 10:41:46 2024 read: IOPS=1, BW=1950KiB/s (1997kB/s)(2048KiB/1050msec) clat (msec): min=42, max=329, avg=185.91, stdev=203.31 lat (msec): min=42, max=329, avg=185.91, stdev=203.31 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 43], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 43], 60.00th=[ 330], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 2043, max= 2048, per=27.63%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=9, BW=9752KiB/s (9986kB/s)(10.0MiB/1050msec) clat (msec): min=17, max=162, avg=62.41, stdev=41.38 lat (msec): min=17, max=162, avg=62.48, stdev=41.37 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 41], 50.00th=[ 48], 60.00th=[ 55], | 70.00th=[ 72], 80.00th=[ 72], 90.00th=[ 96], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 8175, max=10240, per=88.83%, avg=9207.50, stdev=1460.18, samples=2 iops : min= 7, max= 10, avg= 8.50, stdev= 2.12, samples=2 lat (msec) : 20=8.33%, 50=41.67%, 100=33.33%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=32.79%, ctx=231, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6604: Tue Aug 27 10:41:46 2024 read: IOPS=2, BW=2221KiB/s (2275kB/s)(6144KiB/2766msec) clat (usec): min=428, max=813203, avg=275406.86, stdev=386015.24 lat (usec): min=429, max=813203, avg=275407.92, stdev=386015.29 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 429], 10.00th=[ 429], 20.00th=[ 502], | 30.00th=[ 502], 40.00th=[ 594], 50.00th=[ 594], 60.00th=[112722], | 70.00th=[725615], 80.00th=[725615], 90.00th=[809501], 95.00th=[809501], | 99.00th=[809501], 99.50th=[809501], 99.90th=[809501], 99.95th=[809501], | 99.99th=[809501] bw ( KiB/s): min= 1750, max= 4087, per=42.01%, avg=3110.33, stdev=1214.82, samples=3 iops : min= 1, max= 3, avg= 2.33, stdev= 1.15, samples=3 write: IOPS=2, BW=2221KiB/s (2275kB/s)(6144KiB/2766msec) clat (msec): min=21, max=873, avg=176.68, stdev=341.81 lat (msec): min=21, max=873, avg=176.74, stdev=341.82 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 34], 50.00th=[ 34], 60.00th=[ 35], | 70.00th=[ 70], 80.00th=[ 70], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 1750, max= 4087, per=27.43%, avg=2843.50, stdev=1126.35, samples=4 iops : min= 1, max= 3, avg= 2.00, stdev= 1.15, samples=4 lat (usec) : 500=8.33%, 750=16.67% lat (msec) : 50=33.33%, 100=8.33%, 250=8.33%, 750=8.33%, 1000=16.67% cpu : usr=0.04%, sys=18.66%, ctx=663, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6605: Tue Aug 27 10:41:46 2024 read: IOPS=4, BW=5079KiB/s (5201kB/s)(5120KiB/1008msec) clat (usec): min=403, max=542709, avg=144388.42, stdev=229651.46 lat (usec): min=404, max=542710, avg=144389.46, stdev=229651.17 clat percentiles (usec): | 1.00th=[ 404], 5.00th=[ 404], 10.00th=[ 404], 20.00th=[ 404], | 30.00th=[ 445], 40.00th=[ 445], 50.00th=[ 40633], 60.00th=[ 40633], | 70.00th=[137364], 80.00th=[137364], 90.00th=[541066], 95.00th=[541066], | 99.00th=[541066], 99.50th=[541066], 99.90th=[541066], 99.95th=[541066], | 99.99th=[541066] bw ( KiB/s): min= 2048, max= 6144, per=55.32%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=6, BW=7111KiB/s (7282kB/s)(7168KiB/1008msec) clat (usec): min=18700, max=79170, avg=38629.65, stdev=20779.52 lat (usec): min=18762, max=79232, avg=38689.43, stdev=20779.84 clat percentiles (usec): | 1.00th=[18744], 5.00th=[18744], 10.00th=[18744], 20.00th=[19006], | 30.00th=[30016], 40.00th=[30016], 50.00th=[34866], 60.00th=[42206], | 70.00th=[42206], 80.00th=[46400], 90.00th=[79168], 95.00th=[79168], | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 4096, max= 8192, per=59.28%, avg=6144.00, stdev=2896.31, samples=2 iops : min= 4, max= 8, avg= 6.00, stdev= 2.83, samples=2 lat (usec) : 500=16.67% lat (msec) : 20=16.67%, 50=41.67%, 100=8.33%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=48.61%, ctx=389, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7404KiB/s (7582kB/s), 1950KiB/s-6814KiB/s (1997kB/s-6977kB/s), io=20.0MiB (20.0MB), run=1008-2766msec WRITE: bw=10.1MiB/s (10.6MB/s), 2221KiB/s-9752KiB/s (2275kB/s-9986kB/s), io=28.0MiB (29.4MB), run=1008-2766msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6606: Tue Aug 27 10:41:46 2024 read: IOPS=3, BW=3311KiB/s (3390kB/s)(7168KiB/2165msec) clat (msec): min=12, max=674, avg=159.60, stdev=235.65 lat (msec): min=12, max=674, avg=159.60, stdev=235.65 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 75], 60.00th=[ 125], | 70.00th=[ 125], 80.00th=[ 186], 90.00th=[ 676], 95.00th=[ 676], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 2048, max= 4096, per=46.75%, avg=3405.00, stdev=1175.26, samples=3 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=3 write: IOPS=2, BW=2365KiB/s (2422kB/s)(5120KiB/2165msec) clat (msec): min=19, max=795, avg=202.27, stdev=333.95 lat (msec): min=20, max=795, avg=202.32, stdev=333.96 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 22], 40.00th=[ 22], 50.00th=[ 57], 60.00th=[ 57], | 70.00th=[ 117], 80.00th=[ 117], 90.00th=[ 793], 95.00th=[ 793], | 99.00th=[ 793], 99.50th=[ 793], 99.90th=[ 793], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 6107, max= 6107, per=59.90%, avg=6107.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=16.67%, 100=16.67%, 250=25.00%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=6.52%, ctx=304, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6607: Tue Aug 27 10:41:46 2024 read: IOPS=1, BW=1684KiB/s (1725kB/s)(2048KiB/1216msec) clat (msec): min=14, max=770, avg=392.31, stdev=534.44 lat (msec): min=14, max=770, avg=392.31, stdev=534.44 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15], | 30.00th=[ 15], 40.00th=[ 15], 50.00th=[ 15], 60.00th=[ 768], | 70.00th=[ 768], 80.00th=[ 768], 90.00th=[ 768], 95.00th=[ 768], | 99.00th=[ 768], 99.50th=[ 768], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 2048, max= 2048, per=28.12%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8421KiB/s (8623kB/s)(10.0MiB/1216msec) clat (usec): min=13538, max=86795, avg=38957.57, stdev=22432.00 lat (usec): min=13589, max=86858, avg=39020.98, stdev=22433.64 clat percentiles (usec): | 1.00th=[13566], 5.00th=[13566], 10.00th=[13566], 20.00th=[16581], | 30.00th=[18482], 40.00th=[23987], 50.00th=[36963], 60.00th=[42730], | 70.00th=[50070], 80.00th=[50070], 90.00th=[55313], 95.00th=[86508], | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=41.67%, 100=16.67%, 1000=8.33% cpu : usr=0.00%, sys=8.48%, ctx=80, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6608: Tue Aug 27 10:41:46 2024 read: IOPS=2, BW=2220KiB/s (2274kB/s)(6144KiB/2767msec) clat (msec): min=18, max=737, avg=248.11, stdev=333.71 lat (msec): min=18, max=737, avg=248.11, stdev=333.71 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 61], | 70.00th=[ 617], 80.00th=[ 617], 90.00th=[ 735], 95.00th=[ 735], | 99.00th=[ 735], 99.50th=[ 735], 99.90th=[ 735], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 2048, max= 8192, per=70.30%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=2, BW=2220KiB/s (2274kB/s)(6144KiB/2767msec) clat (msec): min=16, max=893, avg=206.38, stdev=338.99 lat (msec): min=16, max=893, avg=206.44, stdev=338.99 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 54], 50.00th=[ 54], 60.00th=[ 100], | 70.00th=[ 127], 80.00th=[ 127], 90.00th=[ 894], 95.00th=[ 894], | 99.00th=[ 894], 99.50th=[ 894], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 2048, max= 6144, per=40.17%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (msec) : 20=16.67%, 50=25.00%, 100=25.00%, 250=8.33%, 750=16.67% lat (msec) : 1000=8.33% cpu : usr=0.04%, sys=3.40%, ctx=50, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6609: Tue Aug 27 10:41:46 2024 read: IOPS=1, BW=1821KiB/s (1864kB/s)(5120KiB/2812msec) clat (usec): min=13822, max=24218, avg=17104.74, stdev=4221.66 lat (usec): min=13823, max=24219, avg=17105.95, stdev=4221.56 clat percentiles (usec): | 1.00th=[13829], 5.00th=[13829], 10.00th=[13829], 20.00th=[13829], | 30.00th=[14615], 40.00th=[14615], 50.00th=[15270], 60.00th=[15270], | 70.00th=[17695], 80.00th=[17695], 90.00th=[24249], 95.00th=[24249], | 99.00th=[24249], 99.50th=[24249], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2549KiB/s (2610kB/s)(7168KiB/2812msec) clat (msec): min=13, max=1477, avg=383.86, stdev=574.91 lat (msec): min=13, max=1477, avg=383.93, stdev=574.91 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 48], | 30.00th=[ 54], 40.00th=[ 54], 50.00th=[ 99], 60.00th=[ 99], | 70.00th=[ 99], 80.00th=[ 902], 90.00th=[ 1485], 95.00th=[ 1485], | 99.00th=[ 1485], 99.50th=[ 1485], 99.90th=[ 1485], 99.95th=[ 1485], | 99.99th=[ 1485] bw ( KiB/s): min= 2048, max=10240, per=60.26%, avg=6144.00, stdev=5792.62, samples=2 iops : min= 2, max= 10, avg= 6.00, stdev= 5.66, samples=2 lat (msec) : 20=41.67%, 50=16.67%, 100=25.00%, 1000=8.33% cpu : usr=0.11%, sys=3.31%, ctx=73, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7283KiB/s (7458kB/s), 1684KiB/s-3311KiB/s (1725kB/s-3390kB/s), io=20.0MiB (20.0MB), run=1216-2812msec WRITE: bw=9.96MiB/s (10.4MB/s), 2220KiB/s-8421KiB/s (2274kB/s-8623kB/s), io=28.0MiB (29.4MB), run=1216-2812msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6616: Tue Aug 27 10:41:50 2024 read: IOPS=1, BW=5788KiB/s (5927kB/s)(12.0MiB/2123msec) clat (msec): min=106, max=1369, avg=692.19, stdev=636.28 lat (msec): min=106, max=1369, avg=692.19, stdev=636.28 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 600], 50.00th=[ 600], 60.00th=[ 600], | 70.00th=[ 1368], 80.00th=[ 1368], 90.00th=[ 1368], 95.00th=[ 1368], | 99.00th=[ 1368], 99.50th=[ 1368], 99.90th=[ 1368], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min= 7037, max= 8047, per=100.00%, avg=7542.00, stdev=714.18, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=62.82%, ctx=557, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6617: Tue Aug 27 10:41:50 2024 read: IOPS=0, BW=1583KiB/s (1621kB/s)(4096KiB/2587msec) clat (nsec): min=106761k, max=106761k, avg=106761003.00, stdev= 0.00 lat (nsec): min=106763k, max=106763k, avg=106763196.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 107], 50.00th=[ 107], 60.00th=[ 107], | 70.00th=[ 107], 80.00th=[ 107], 90.00th=[ 107], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3167KiB/s (3243kB/s)(8192KiB/2587msec) clat (msec): min=438, max=2008, avg=1223.32, stdev=1110.41 lat (msec): min=438, max=2008, avg=1223.59, stdev=1110.42 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 439], 10.00th=[ 439], 20.00th=[ 439], | 30.00th=[ 439], 40.00th=[ 439], 50.00th=[ 439], 60.00th=[ 2005], | 70.00th=[ 2005], 80.00th=[ 2005], 90.00th=[ 2005], 95.00th=[ 2005], | 99.00th=[ 2005], 99.50th=[ 2005], 99.90th=[ 2005], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7198, max= 7198, per=56.83%, avg=7198.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=10.56%, ctx=65, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6618: Tue Aug 27 10:41:50 2024 write: IOPS=1, BW=5397KiB/s (5526kB/s)(12.0MiB/2277msec) clat (msec): min=218, max=1535, avg=755.44, stdev=691.08 lat (msec): min=219, max=1535, avg=755.74, stdev=691.06 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 220], 10.00th=[ 220], 20.00th=[ 220], | 30.00th=[ 220], 40.00th=[ 514], 50.00th=[ 514], 60.00th=[ 514], | 70.00th=[ 1536], 80.00th=[ 1536], 90.00th=[ 1536], 95.00th=[ 1536], | 99.00th=[ 1536], 99.50th=[ 1536], 99.90th=[ 1536], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 6954, max= 8159, per=59.66%, avg=7556.50, stdev=852.06, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=14.54%, ctx=44, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6619: Tue Aug 27 10:41:50 2024 write: IOPS=4, BW=17.8MiB/s (18.7MB/s)(12.0MiB/674msec) clat (msec): min=169, max=285, avg=210.08, stdev=65.66 lat (msec): min=169, max=285, avg=210.23, stdev=65.64 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 169], 10.00th=[ 169], 20.00th=[ 169], | 30.00th=[ 169], 40.00th=[ 176], 50.00th=[ 176], 60.00th=[ 176], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min=16157, max=16157, per=100.00%, avg=16157.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=43.68%, ctx=54, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6333KiB/s (6485kB/s), 1583KiB/s-5788KiB/s (1621kB/s-5927kB/s), io=16.0MiB (16.8MB), run=2123-2587msec WRITE: bw=12.4MiB/s (12.0MB/s), 3167KiB/s-17.8MiB/s (3243kB/s-18.7MB/s), io=32.0MiB (33.6MB), run=674-2587msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6621: Tue Aug 27 10:41:50 2024 read: IOPS=1, BW=4834KiB/s (4950kB/s)(12.0MiB/2542msec) clat (msec): min=97, max=1912, avg=832.64, stdev=955.34 lat (msec): min=97, max=1912, avg=832.64, stdev=955.34 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 99], 10.00th=[ 99], 20.00th=[ 99], | 30.00th=[ 99], 40.00th=[ 489], 50.00th=[ 489], 60.00th=[ 489], | 70.00th=[ 1905], 80.00th=[ 1905], 90.00th=[ 1905], 95.00th=[ 1905], | 99.00th=[ 1905], 99.50th=[ 1905], 99.90th=[ 1905], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=3.11%, ctx=50, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6622: Tue Aug 27 10:41:50 2024 read: IOPS=0, BW=1546KiB/s (1583kB/s)(4096KiB/2649msec) clat (nsec): min=427090k, max=427090k, avg=427090104.00, stdev= 0.00 lat (nsec): min=427091k, max=427091k, avg=427091006.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 426], 10.00th=[ 426], 20.00th=[ 426], | 30.00th=[ 426], 40.00th=[ 426], 50.00th=[ 426], 60.00th=[ 426], | 70.00th=[ 426], 80.00th=[ 426], 90.00th=[ 426], 95.00th=[ 426], | 99.00th=[ 426], 99.50th=[ 426], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3092KiB/s (3167kB/s)(8192KiB/2649msec) clat (msec): min=265, max=1944, avg=1105.10, stdev=1187.58 lat (msec): min=265, max=1945, avg=1105.29, stdev=1187.67 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 266], 10.00th=[ 266], 20.00th=[ 266], | 30.00th=[ 266], 40.00th=[ 266], 50.00th=[ 266], 60.00th=[ 1938], | 70.00th=[ 1938], 80.00th=[ 1938], 90.00th=[ 1938], 95.00th=[ 1938], | 99.00th=[ 1938], 99.50th=[ 1938], 99.90th=[ 1938], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 8192, max= 8192, per=66.23%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=5.85%, ctx=156, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6623: Tue Aug 27 10:41:50 2024 write: IOPS=1, BW=4927KiB/s (5045kB/s)(12.0MiB/2494msec) clat (msec): min=118, max=1914, avg=825.44, stdev=956.98 lat (msec): min=118, max=1914, avg=825.67, stdev=956.90 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 443], 50.00th=[ 443], 60.00th=[ 443], | 70.00th=[ 1921], 80.00th=[ 1921], 90.00th=[ 1921], 95.00th=[ 1921], | 99.00th=[ 1921], 99.50th=[ 1921], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 8192, max= 8192, per=66.23%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.04%, sys=3.33%, ctx=57, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6624: Tue Aug 27 10:41:50 2024 write: IOPS=1, BW=5101KiB/s (5223kB/s)(12.0MiB/2409msec) clat (msec): min=76, max=1924, avg=796.03, stdev=989.71 lat (msec): min=76, max=1924, avg=796.30, stdev=989.71 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 77], 10.00th=[ 77], 20.00th=[ 77], | 30.00th=[ 77], 40.00th=[ 388], 50.00th=[ 388], 60.00th=[ 388], | 70.00th=[ 1921], 80.00th=[ 1921], 90.00th=[ 1921], 95.00th=[ 1921], | 99.00th=[ 1921], 99.50th=[ 1921], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 8192, max= 8192, per=66.23%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=5.61%, ctx=113, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6185KiB/s (6333kB/s), 1546KiB/s-4834KiB/s (1583kB/s-4950kB/s), io=16.0MiB (16.8MB), run=2542-2649msec WRITE: bw=12.1MiB/s (12.7MB/s), 3092KiB/s-5101KiB/s (3167kB/s-5223kB/s), io=32.0MiB (33.6MB), run=2409-2649msec