== sanity test 398b: DIO and buffer IO race ============== 10:41:08 (1724769668) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.24657 s, 40.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9443: Tue Aug 27 10:41:24 2024 read: IOPS=109, BW=436KiB/s (447kB/s)(5992KiB/13737msec) clat (usec): min=56, max=47910, avg=5061.24, stdev=2916.43 lat (usec): min=57, max=47912, avg=5062.69, stdev=2917.62 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 2057], 10.00th=[ 2507], 20.00th=[ 3032], | 30.00th=[ 3523], 40.00th=[ 3949], 50.00th=[ 4621], 60.00th=[ 5276], | 70.00th=[ 5997], 80.00th=[ 6915], 90.00th=[ 8029], 95.00th=[ 9765], | 99.00th=[14746], 99.50th=[15664], 99.90th=[28443], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 208, max= 640, per=25.33%, avg=435.74, stdev=78.47, samples=27 iops : min= 52, max= 160, avg=108.85, stdev=19.60, samples=27 write: IOPS=114, BW=458KiB/s (469kB/s)(6296KiB/13737msec) clat (usec): min=432, max=133178, avg=3872.43, stdev=4683.85 lat (usec): min=433, max=133180, avg=3873.42, stdev=4683.98 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 562], 10.00th=[ 619], 20.00th=[ 783], | 30.00th=[ 963], 40.00th=[ 2057], 50.00th=[ 3097], 60.00th=[ 3851], | 70.00th=[ 4883], 80.00th=[ 6325], 90.00th=[ 8717], 95.00th=[ 10421], | 99.00th=[ 14091], 99.50th=[ 15926], 99.90th=[ 25560], 99.95th=[133694], | 99.99th=[133694] bw ( KiB/s): min= 240, max= 608, per=25.33%, avg=456.19, stdev=85.96, samples=27 iops : min= 60, max= 152, avg=113.96, stdev=21.52, samples=27 lat (usec) : 100=1.50%, 250=0.46%, 500=0.29%, 750=8.92%, 1000=7.03% lat (msec) : 2=4.36%, 4=29.07%, 10=42.94%, 20=5.21%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=24.48%, ctx=6296, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9444: Tue Aug 27 10:41:24 2024 read: IOPS=107, BW=431KiB/s (442kB/s)(5940KiB/13773msec) clat (usec): min=55, max=164285, avg=5094.22, stdev=4964.71 lat (usec): min=55, max=164286, avg=5095.04, stdev=4964.74 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 603], 10.00th=[ 2474], 20.00th=[ 2966], | 30.00th=[ 3425], 40.00th=[ 3916], 50.00th=[ 4424], 60.00th=[ 5080], | 70.00th=[ 5932], 80.00th=[ 6980], 90.00th=[ 8225], 95.00th=[ 9896], | 99.00th=[ 14353], 99.50th=[ 15926], 99.90th=[ 20579], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 128, max= 568, per=24.96%, avg=429.30, stdev=84.53, samples=27 iops : min= 32, max= 142, avg=107.30, stdev=21.12, samples=27 write: IOPS=115, BW=461KiB/s (472kB/s)(6348KiB/13773msec) clat (usec): min=485, max=86282, avg=3864.09, stdev=3937.67 lat (usec): min=486, max=86283, avg=3864.87, stdev=3937.67 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 553], 10.00th=[ 611], 20.00th=[ 791], | 30.00th=[ 979], 40.00th=[ 2180], 50.00th=[ 3032], 60.00th=[ 3818], | 70.00th=[ 4817], 80.00th=[ 6521], 90.00th=[ 8586], 95.00th=[10552], | 99.00th=[13960], 99.50th=[15139], 99.90th=[22152], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 208, max= 576, per=25.12%, avg=452.37, stdev=70.46, samples=27 iops : min= 52, max= 144, avg=113.07, stdev=17.60, samples=27 lat (usec) : 100=1.76%, 250=0.62%, 500=0.42%, 750=9.18%, 1000=6.22% lat (msec) : 2=4.85%, 4=29.10%, 10=42.38%, 20=5.31%, 50=0.10% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.14%, sys=23.69%, ctx=6357, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9445: Tue Aug 27 10:41:24 2024 read: IOPS=107, BW=432KiB/s (442kB/s)(6028KiB/13954msec) clat (usec): min=48, max=29017, avg=5056.44, stdev=2939.84 lat (usec): min=48, max=29017, avg=5058.00, stdev=2939.77 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 105], 10.00th=[ 2409], 20.00th=[ 2966], | 30.00th=[ 3425], 40.00th=[ 3982], 50.00th=[ 4555], 60.00th=[ 5276], | 70.00th=[ 6128], 80.00th=[ 6980], 90.00th=[ 8455], 95.00th=[ 9765], | 99.00th=[15008], 99.50th=[18220], 99.90th=[25560], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 151, max= 590, per=24.52%, avg=421.74, stdev=75.75, samples=27 iops : min= 37, max= 147, avg=105.33, stdev=19.00, samples=27 write: IOPS=112, BW=449KiB/s (459kB/s)(6260KiB/13954msec) clat (usec): min=436, max=164942, avg=3999.49, stdev=5597.00 lat (usec): min=437, max=164944, avg=4001.01, stdev=5597.32 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 562], 10.00th=[ 619], 20.00th=[ 766], | 30.00th=[ 971], 40.00th=[ 2212], 50.00th=[ 3130], 60.00th=[ 3818], | 70.00th=[ 4752], 80.00th=[ 6652], 90.00th=[ 8979], 95.00th=[ 10421], | 99.00th=[ 14484], 99.50th=[ 16057], 99.90th=[ 71828], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 175, max= 568, per=24.24%, avg=436.52, stdev=85.15, samples=27 iops : min= 43, max= 142, avg=109.04, stdev=21.35, samples=27 lat (usec) : 50=0.03%, 100=2.15%, 250=0.98%, 500=0.36%, 750=9.47% lat (usec) : 1000=5.92% lat (msec) : 2=4.36%, 4=28.22%, 10=43.33%, 20=4.92%, 50=0.20% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.19%, sys=23.79%, ctx=6277, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9446: Tue Aug 27 10:41:24 2024 read: IOPS=108, BW=435KiB/s (446kB/s)(6052KiB/13899msec) clat (usec): min=46, max=23480, avg=5100.85, stdev=2920.12 lat (usec): min=47, max=23481, avg=5102.34, stdev=2918.93 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 118], 10.00th=[ 2442], 20.00th=[ 2933], | 30.00th=[ 3458], 40.00th=[ 3982], 50.00th=[ 4621], 60.00th=[ 5407], | 70.00th=[ 6128], 80.00th=[ 7046], 90.00th=[ 8455], 95.00th=[ 9765], | 99.00th=[16450], 99.50th=[18744], 99.90th=[21627], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 175, max= 600, per=24.87%, avg=427.78, stdev=76.62, samples=27 iops : min= 43, max= 150, avg=106.89, stdev=19.26, samples=27 write: IOPS=112, BW=449KiB/s (459kB/s)(6236KiB/13899msec) clat (usec): min=409, max=160855, avg=3911.80, stdev=5550.40 lat (usec): min=410, max=160857, avg=3913.84, stdev=5549.95 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 553], 10.00th=[ 611], 20.00th=[ 775], | 30.00th=[ 979], 40.00th=[ 2180], 50.00th=[ 3130], 60.00th=[ 3884], | 70.00th=[ 4883], 80.00th=[ 6194], 90.00th=[ 8356], 95.00th=[ 10159], | 99.00th=[ 14222], 99.50th=[ 17957], 99.90th=[ 82314], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 191, max= 688, per=24.61%, avg=443.19, stdev=96.53, samples=27 iops : min= 47, max= 172, avg=110.74, stdev=24.22, samples=27 lat (usec) : 50=0.07%, 100=1.95%, 250=0.68%, 500=0.26%, 750=9.15% lat (usec) : 1000=6.15% lat (msec) : 2=4.30%, 4=28.42%, 10=44.14%, 20=4.59%, 50=0.23% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.24%, sys=23.64%, ctx=6324, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1721KiB/s (1762kB/s), 431KiB/s-436KiB/s (442kB/s-447kB/s), io=23.4MiB (24.6MB), run=13737-13954msec WRITE: bw=1802KiB/s (1845kB/s), 449KiB/s-461KiB/s (459kB/s-472kB/s), io=24.6MiB (25.7MB), run=13737-13954msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9439: Tue Aug 27 10:41:28 2024 read: IOPS=85, BW=343KiB/s (351kB/s)(5992KiB/17479msec) clat (usec): min=1565, max=213589, avg=4783.07, stdev=6484.01 lat (usec): min=1566, max=213591, avg=4783.80, stdev=6484.03 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1876], 10.00th=[ 1975], 20.00th=[ 2147], | 30.00th=[ 2343], 40.00th=[ 2638], 50.00th=[ 3163], 60.00th=[ 3851], | 70.00th=[ 4686], 80.00th=[ 6652], 90.00th=[ 9896], 95.00th=[ 12780], | 99.00th=[ 17171], 99.50th=[ 20579], 99.90th=[ 28967], 99.95th=[212861], | 99.99th=[212861] bw ( KiB/s): min= 128, max= 784, per=24.58%, avg=337.50, stdev=170.32, samples=34 iops : min= 32, max= 196, avg=84.29, stdev=42.57, samples=34 write: IOPS=90, BW=360KiB/s (369kB/s)(6296KiB/17479msec) clat (usec): min=2317, max=32151, avg=6508.01, stdev=4125.32 lat (usec): min=2318, max=32152, avg=6508.94, stdev=4125.26 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2835], 10.00th=[ 3032], 20.00th=[ 3326], | 30.00th=[ 3621], 40.00th=[ 4146], 50.00th=[ 4948], 60.00th=[ 6128], | 70.00th=[ 7504], 80.00th=[ 9241], 90.00th=[12125], 95.00th=[15139], | 99.00th=[20579], 99.50th=[22938], 99.90th=[29230], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 136, max= 800, per=24.57%, avg=353.26, stdev=174.44, samples=34 iops : min= 34, max= 200, avg=88.24, stdev=43.61, samples=34 lat (msec) : 2=5.24%, 4=44.60%, 10=36.85%, 20=12.47%, 50=0.81% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.17%, ctx=3723, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9440: Tue Aug 27 10:41:28 2024 read: IOPS=84, BW=340KiB/s (348kB/s)(5940KiB/17480msec) clat (usec): min=1634, max=176299, avg=4685.92, stdev=5653.09 lat (usec): min=1635, max=176301, avg=4686.68, stdev=5653.12 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1893], 10.00th=[ 2008], 20.00th=[ 2147], | 30.00th=[ 2343], 40.00th=[ 2638], 50.00th=[ 3097], 60.00th=[ 3720], | 70.00th=[ 4752], 80.00th=[ 6652], 90.00th=[ 9896], 95.00th=[ 12125], | 99.00th=[ 16909], 99.50th=[ 19530], 99.90th=[ 26084], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 96, max= 744, per=24.51%, avg=336.56, stdev=178.10, samples=34 iops : min= 24, max= 186, avg=84.06, stdev=44.52, samples=34 write: IOPS=90, BW=363KiB/s (372kB/s)(6348KiB/17480msec) clat (usec): min=2300, max=48936, avg=6590.52, stdev=4293.72 lat (usec): min=2301, max=48937, avg=6591.34, stdev=4293.74 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2868], 10.00th=[ 3064], 20.00th=[ 3359], | 30.00th=[ 3621], 40.00th=[ 4178], 50.00th=[ 4948], 60.00th=[ 5997], | 70.00th=[ 7701], 80.00th=[ 9634], 90.00th=[12518], 95.00th=[15008], | 99.00th=[20317], 99.50th=[25035], 99.90th=[32637], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 152, max= 776, per=24.66%, avg=354.68, stdev=168.65, samples=34 iops : min= 38, max= 194, avg=88.59, stdev=42.17, samples=34 lat (msec) : 2=4.65%, 4=44.92%, 10=36.30%, 20=13.28%, 50=0.81% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=11.32%, ctx=3734, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9441: Tue Aug 27 10:41:28 2024 read: IOPS=88, BW=352KiB/s (360kB/s)(6028KiB/17123msec) clat (usec): min=1525, max=21304, avg=4572.13, stdev=3477.33 lat (usec): min=1526, max=21305, avg=4572.86, stdev=3477.30 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1876], 10.00th=[ 1975], 20.00th=[ 2147], | 30.00th=[ 2376], 40.00th=[ 2704], 50.00th=[ 3163], 60.00th=[ 3720], | 70.00th=[ 4752], 80.00th=[ 6456], 90.00th=[ 9896], 95.00th=[12518], | 99.00th=[16450], 99.50th=[17957], 99.90th=[20841], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 120, max= 824, per=25.59%, avg=351.38, stdev=172.44, samples=34 iops : min= 30, max= 206, avg=87.74, stdev=43.15, samples=34 write: IOPS=91, BW=366KiB/s (374kB/s)(6260KiB/17123msec) clat (msec): min=2, max=179, avg= 6.50, stdev= 5.93 lat (msec): min=2, max=179, avg= 6.50, stdev= 5.93 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 21], 99.50th=[ 25], 99.90th=[ 34], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 160, max= 832, per=25.32%, avg=364.09, stdev=172.84, samples=34 iops : min= 40, max= 208, avg=90.91, stdev=43.27, samples=34 lat (msec) : 2=5.89%, 4=43.82%, 10=37.40%, 20=12.21%, 50=0.65% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=11.42%, ctx=3644, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9442: Tue Aug 27 10:41:28 2024 read: IOPS=87, BW=349KiB/s (357kB/s)(6052KiB/17343msec) clat (usec): min=1584, max=26512, avg=4632.81, stdev=3582.13 lat (usec): min=1584, max=26551, avg=4633.54, stdev=3582.30 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1876], 10.00th=[ 1991], 20.00th=[ 2180], | 30.00th=[ 2409], 40.00th=[ 2704], 50.00th=[ 3130], 60.00th=[ 3752], | 70.00th=[ 4621], 80.00th=[ 6652], 90.00th=[10028], 95.00th=[12518], | 99.00th=[16909], 99.50th=[19792], 99.90th=[23725], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 176, max= 808, per=25.18%, avg=345.71, stdev=169.28, samples=34 iops : min= 44, max= 202, avg=86.29, stdev=42.37, samples=34 write: IOPS=89, BW=360KiB/s (368kB/s)(6236KiB/17343msec) clat (msec): min=2, max=179, avg= 6.59, stdev= 5.96 lat (msec): min=2, max=179, avg= 6.59, stdev= 5.96 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 15], | 99.00th=[ 20], 99.50th=[ 23], 99.90th=[ 45], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 192, max= 744, per=24.78%, avg=356.29, stdev=171.27, samples=34 iops : min= 48, max= 186, avg=88.94, stdev=42.89, samples=34 lat (msec) : 2=5.34%, 4=44.79%, 10=36.30%, 20=12.92%, 50=0.62% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=11.27%, ctx=3684, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1374KiB/s (1407kB/s), 340KiB/s-352KiB/s (348kB/s-360kB/s), io=23.4MiB (24.6MB), run=17123-17480msec WRITE: bw=1438KiB/s (1473kB/s), 360KiB/s-366KiB/s (368kB/s-374kB/s), io=24.6MiB (25.7MB), run=17123-17480msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9477: Tue Aug 27 10:41:35 2024 read: IOPS=59, BW=946KiB/s (969kB/s)(5920KiB/6259msec) clat (usec): min=59, max=32168, avg=12240.57, stdev=3877.79 lat (usec): min=60, max=32169, avg=12241.98, stdev=3878.02 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 7111], 10.00th=[ 8586], 20.00th=[ 9765], | 30.00th=[10552], 40.00th=[11469], 50.00th=[12125], 60.00th=[12911], | 70.00th=[13829], 80.00th=[14746], 90.00th=[16319], 95.00th=[18482], | 99.00th=[23725], 99.50th=[24249], 99.90th=[32113], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 704, max= 1024, per=25.36%, avg=930.67, stdev=92.00, samples=12 iops : min= 44, max= 64, avg=58.17, stdev= 5.75, samples=12 write: IOPS=63, BW=1017KiB/s (1042kB/s)(6368KiB/6259msec) clat (usec): min=768, max=23211, avg=4215.10, stdev=3598.78 lat (usec): min=769, max=23213, avg=4216.34, stdev=3598.79 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 848], 10.00th=[ 947], 20.00th=[ 1287], | 30.00th=[ 2343], 40.00th=[ 2704], 50.00th=[ 3195], 60.00th=[ 3785], | 70.00th=[ 4686], 80.00th=[ 5932], 90.00th=[ 9241], 95.00th=[11994], | 99.00th=[17171], 99.50th=[22938], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 768, max= 1440, per=25.89%, avg=1000.00, stdev=170.49, samples=12 iops : min= 48, max= 90, avg=62.50, stdev=10.66, samples=12 lat (usec) : 100=1.04%, 250=0.13%, 750=0.13%, 1000=6.64% lat (msec) : 2=7.68%, 4=18.62%, 10=24.09%, 20=39.84%, 50=1.82% cpu : usr=0.14%, sys=20.92%, ctx=2693, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9479: Tue Aug 27 10:41:35 2024 read: IOPS=57, BW=928KiB/s (950kB/s)(5744KiB/6190msec) clat (usec): min=61, max=24627, avg=12117.25, stdev=4362.65 lat (usec): min=61, max=24628, avg=12118.11, stdev=4362.60 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 100], 10.00th=[ 8356], 20.00th=[ 9896], | 30.00th=[10814], 40.00th=[11469], 50.00th=[12518], 60.00th=[13173], | 70.00th=[14091], 80.00th=[15008], 90.00th=[17171], 95.00th=[17957], | 99.00th=[21627], 99.50th=[22938], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 766, max= 1088, per=25.06%, avg=919.83, stdev=103.40, samples=12 iops : min= 47, max= 68, avg=57.42, stdev= 6.58, samples=12 write: IOPS=66, BW=1057KiB/s (1083kB/s)(6544KiB/6190msec) clat (usec): min=681, max=26925, avg=4467.21, stdev=3829.96 lat (usec): min=682, max=26926, avg=4468.85, stdev=3829.77 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 816], 10.00th=[ 947], 20.00th=[ 1303], | 30.00th=[ 2507], 40.00th=[ 2933], 50.00th=[ 3458], 60.00th=[ 3982], | 70.00th=[ 5080], 80.00th=[ 6521], 90.00th=[ 9372], 95.00th=[12911], | 99.00th=[17171], 99.50th=[19268], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 832, max= 1440, per=26.99%, avg=1042.42, stdev=190.79, samples=12 iops : min= 52, max= 90, avg=65.08, stdev=11.94, samples=12 lat (usec) : 100=2.21%, 250=0.78%, 750=0.78%, 1000=5.60% lat (msec) : 2=8.20%, 4=17.71%, 10=22.92%, 20=40.23%, 50=1.56% cpu : usr=0.06%, sys=20.71%, ctx=2638, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9480: Tue Aug 27 10:41:35 2024 read: IOPS=59, BW=959KiB/s (982kB/s)(6256KiB/6525msec) clat (usec): min=59, max=22539, avg=11738.23, stdev=4078.20 lat (usec): min=60, max=22540, avg=11738.95, stdev=4078.23 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 99], 10.00th=[ 7832], 20.00th=[ 9634], | 30.00th=[10552], 40.00th=[11207], 50.00th=[11994], 60.00th=[12649], | 70.00th=[13566], 80.00th=[14746], 90.00th=[16188], 95.00th=[17695], | 99.00th=[19530], 99.50th=[21365], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 736, max= 1440, per=26.02%, avg=955.08, stdev=190.59, samples=13 iops : min= 46, max= 90, avg=59.69, stdev=11.91, samples=13 write: IOPS=57, BW=924KiB/s (947kB/s)(6032KiB/6525msec) clat (usec): min=774, max=25215, avg=5057.31, stdev=4450.58 lat (usec): min=775, max=25216, avg=5058.59, stdev=4450.54 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 914], 10.00th=[ 1029], 20.00th=[ 1418], | 30.00th=[ 2704], 40.00th=[ 2999], 50.00th=[ 3589], 60.00th=[ 4621], | 70.00th=[ 5407], 80.00th=[ 7373], 90.00th=[11076], 95.00th=[14877], | 99.00th=[22676], 99.50th=[24511], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 576, max= 1088, per=23.46%, avg=905.85, stdev=158.06, samples=13 iops : min= 36, max= 68, avg=56.62, stdev= 9.88, samples=13 lat (usec) : 100=2.60%, 250=0.39%, 1000=4.17% lat (msec) : 2=6.90%, 4=15.89%, 10=24.35%, 20=44.66%, 50=1.04% cpu : usr=0.09%, sys=20.39%, ctx=2754, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9481: Tue Aug 27 10:41:35 2024 read: IOPS=58, BW=943KiB/s (966kB/s)(6032KiB/6395msec) clat (usec): min=58, max=24416, avg=12052.21, stdev=3989.25 lat (usec): min=58, max=24417, avg=12053.33, stdev=3989.16 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 219], 10.00th=[ 8094], 20.00th=[ 9896], | 30.00th=[10814], 40.00th=[11600], 50.00th=[12256], 60.00th=[13042], | 70.00th=[14091], 80.00th=[15008], 90.00th=[16450], 95.00th=[17695], | 99.00th=[20055], 99.50th=[21103], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 640, max= 1120, per=24.63%, avg=904.00, stdev=135.16, samples=12 iops : min= 40, max= 70, avg=56.50, stdev= 8.45, samples=12 write: IOPS=61, BW=978KiB/s (1002kB/s)(6256KiB/6395msec) clat (usec): min=618, max=36456, avg=4621.25, stdev=4442.21 lat (usec): min=619, max=36458, avg=4624.22, stdev=4443.84 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 816], 10.00th=[ 906], 20.00th=[ 1303], | 30.00th=[ 2442], 40.00th=[ 2966], 50.00th=[ 3392], 60.00th=[ 3884], | 70.00th=[ 5014], 80.00th=[ 6390], 90.00th=[ 9241], 95.00th=[13960], | 99.00th=[20841], 99.50th=[27919], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 736, max= 1248, per=24.72%, avg=954.67, stdev=159.61, samples=12 iops : min= 46, max= 78, avg=59.67, stdev= 9.98, samples=12 lat (usec) : 100=1.43%, 250=1.04%, 750=0.78%, 1000=5.47% lat (msec) : 2=7.29%, 4=17.97%, 10=22.66%, 20=41.93%, 50=1.43% cpu : usr=0.08%, sys=20.50%, ctx=2637, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3671KiB/s (3759kB/s), 928KiB/s-959KiB/s (950kB/s-982kB/s), io=23.4MiB (24.5MB), run=6190-6525msec WRITE: bw=3862KiB/s (3955kB/s), 924KiB/s-1057KiB/s (947kB/s-1083kB/s), io=24.6MiB (25.8MB), run=6190-6525msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9474: Tue Aug 27 10:41:35 2024 read: IOPS=64, BW=1036KiB/s (1060kB/s)(5920KiB/5717msec) clat (usec): min=1953, max=31843, avg=6163.81, stdev=4736.33 lat (usec): min=1954, max=31843, avg=6164.53, stdev=4736.34 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2409], 10.00th=[ 2573], 20.00th=[ 2933], | 30.00th=[ 3261], 40.00th=[ 3654], 50.00th=[ 4146], 60.00th=[ 4948], | 70.00th=[ 6259], 80.00th=[ 9503], 90.00th=[12911], 95.00th=[15664], | 99.00th=[23200], 99.50th=[24511], 99.90th=[31851], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 800, max= 1440, per=28.05%, avg=1018.18, stdev=191.37, samples=11 iops : min= 50, max= 90, avg=63.64, stdev=11.96, samples=11 write: IOPS=69, BW=1114KiB/s (1141kB/s)(6368KiB/5717msec) clat (usec): min=2689, max=28573, avg=8510.03, stdev=5148.45 lat (usec): min=2690, max=28574, avg=8511.32, stdev=5148.46 clat percentiles (usec): | 1.00th=[ 3064], 5.00th=[ 3556], 10.00th=[ 3818], 20.00th=[ 4359], | 30.00th=[ 4883], 40.00th=[ 5604], 50.00th=[ 6849], 60.00th=[ 7898], | 70.00th=[ 9634], 80.00th=[12518], 90.00th=[16909], 95.00th=[19268], | 99.00th=[26608], 99.50th=[28181], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 928, max= 1344, per=29.17%, avg=1114.18, stdev=113.42, samples=11 iops : min= 58, max= 84, avg=69.64, stdev= 7.09, samples=11 lat (msec) : 2=0.13%, 4=29.56%, 10=46.61%, 20=20.96%, 50=2.73% cpu : usr=0.16%, sys=10.16%, ctx=927, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9475: Tue Aug 27 10:41:35 2024 read: IOPS=57, BW=919KiB/s (941kB/s)(5744KiB/6248msec) clat (usec): min=1924, max=22642, avg=6501.68, stdev=4529.63 lat (usec): min=1925, max=22642, avg=6502.38, stdev=4529.63 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2311], 10.00th=[ 2474], 20.00th=[ 2868], | 30.00th=[ 3359], 40.00th=[ 3884], 50.00th=[ 4555], 60.00th=[ 5997], | 70.00th=[ 8029], 80.00th=[10159], 90.00th=[13304], 95.00th=[16057], | 99.00th=[21627], 99.50th=[21890], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 704, max= 1164, per=25.22%, avg=915.33, stdev=152.90, samples=12 iops : min= 44, max= 72, avg=57.00, stdev= 9.53, samples=12 write: IOPS=65, BW=1047KiB/s (1073kB/s)(6544KiB/6248msec) clat (usec): min=2840, max=30677, avg=9526.58, stdev=5116.79 lat (usec): min=2842, max=30678, avg=9528.02, stdev=5116.70 clat percentiles (usec): | 1.00th=[ 3195], 5.00th=[ 3687], 10.00th=[ 4047], 20.00th=[ 5014], | 30.00th=[ 5997], 40.00th=[ 7177], 50.00th=[ 8586], 60.00th=[ 9634], | 70.00th=[11338], 80.00th=[12911], 90.00th=[16909], 95.00th=[20317], | 99.00th=[25035], 99.50th=[27657], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 800, max= 1280, per=26.87%, avg=1026.08, stdev=174.35, samples=12 iops : min= 50, max= 80, avg=63.92, stdev=10.92, samples=12 lat (msec) : 2=0.39%, 4=24.61%, 10=45.31%, 20=26.04%, 50=3.65% cpu : usr=0.19%, sys=10.44%, ctx=1083, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9476: Tue Aug 27 10:41:35 2024 read: IOPS=59, BW=948KiB/s (971kB/s)(6256KiB/6598msec) clat (usec): min=2037, max=37146, avg=7049.28, stdev=5291.10 lat (usec): min=2038, max=37147, avg=7050.02, stdev=5291.09 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2343], 10.00th=[ 2507], 20.00th=[ 2868], | 30.00th=[ 3359], 40.00th=[ 3982], 50.00th=[ 4883], 60.00th=[ 6718], | 70.00th=[ 8717], 80.00th=[10683], 90.00th=[13566], 95.00th=[16909], | 99.00th=[28181], 99.50th=[32900], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 704, max= 1138, per=25.63%, avg=930.31, stdev=150.53, samples=13 iops : min= 44, max= 71, avg=58.00, stdev= 9.42, samples=13 write: IOPS=57, BW=914KiB/s (936kB/s)(6032KiB/6598msec) clat (usec): min=2594, max=37817, avg=10081.47, stdev=5332.78 lat (usec): min=2595, max=37819, avg=10082.93, stdev=5332.66 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3490], 10.00th=[ 4146], 20.00th=[ 5342], | 30.00th=[ 6587], 40.00th=[ 7832], 50.00th=[ 9372], 60.00th=[10552], | 70.00th=[11863], 80.00th=[13960], 90.00th=[17171], 95.00th=[20317], | 99.00th=[28443], 99.50th=[30540], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 640, max= 1184, per=23.48%, avg=896.62, stdev=158.05, samples=13 iops : min= 40, max= 74, avg=55.85, stdev= 9.91, samples=13 lat (msec) : 4=25.00%, 10=40.76%, 20=30.60%, 50=3.65% cpu : usr=0.09%, sys=10.70%, ctx=1189, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9478: Tue Aug 27 10:41:35 2024 read: IOPS=67, BW=1081KiB/s (1107kB/s)(6032KiB/5580msec) clat (usec): min=1892, max=25844, avg=5718.69, stdev=4128.60 lat (usec): min=1893, max=25845, avg=5719.48, stdev=4128.59 clat percentiles (usec): | 1.00th=[ 2073], 5.00th=[ 2442], 10.00th=[ 2606], 20.00th=[ 2868], | 30.00th=[ 3097], 40.00th=[ 3425], 50.00th=[ 3916], 60.00th=[ 4621], | 70.00th=[ 5800], 80.00th=[ 8225], 90.00th=[12256], 95.00th=[14484], | 99.00th=[20579], 99.50th=[23200], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 832, max= 1376, per=29.57%, avg=1073.27, stdev=165.48, samples=11 iops : min= 52, max= 86, avg=67.00, stdev=10.44, samples=11 write: IOPS=70, BW=1121KiB/s (1148kB/s)(6256KiB/5580msec) clat (usec): min=2777, max=30188, avg=8671.12, stdev=5295.99 lat (usec): min=2778, max=30190, avg=8672.41, stdev=5295.95 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 3687], 10.00th=[ 3949], 20.00th=[ 4490], | 30.00th=[ 4948], 40.00th=[ 5800], 50.00th=[ 6980], 60.00th=[ 8225], | 70.00th=[ 9634], 80.00th=[12256], 90.00th=[16450], 95.00th=[20317], | 99.00th=[26870], 99.50th=[29492], 99.90th=[30278], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 896, max= 1280, per=29.24%, avg=1116.82, stdev=121.13, samples=11 iops : min= 56, max= 80, avg=69.73, stdev= 7.62, samples=11 lat (msec) : 2=0.26%, 4=30.08%, 10=47.92%, 20=18.10%, 50=3.65% cpu : usr=0.18%, sys=10.20%, ctx=936, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3630KiB/s (3717kB/s), 919KiB/s-1081KiB/s (941kB/s-1107kB/s), io=23.4MiB (24.5MB), run=5580-6598msec WRITE: bw=3819KiB/s (3911kB/s), 914KiB/s-1121KiB/s (936kB/s-1148kB/s), io=24.6MiB (25.8MB), run=5580-6598msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9492: Tue Aug 27 10:41:39 2024 read: IOPS=2, BW=2422KiB/s (2480kB/s)(7168KiB/2960msec) clat (msec): min=59, max=562, avg=379.93, stdev=226.89 lat (msec): min=59, max=562, avg=379.93, stdev=226.89 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 61], 10.00th=[ 61], 20.00th=[ 66], | 30.00th=[ 372], 40.00th=[ 372], 50.00th=[ 481], 60.00th=[ 558], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 1802, max= 4047, per=34.59%, avg=2392.60, stdev=930.51, samples=5 iops : min= 1, max= 3, avg= 1.60, stdev= 0.89, samples=5 write: IOPS=1, BW=1730KiB/s (1771kB/s)(5120KiB/2960msec) clat (msec): min=21, max=150, avg=57.52, stdev=53.53 lat (msec): min=21, max=150, avg=57.60, stdev=53.52 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 42], 60.00th=[ 42], | 70.00th=[ 51], 80.00th=[ 51], 90.00th=[ 150], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 1802, max= 6071, per=34.06%, avg=3298.67, stdev=2403.45, samples=3 iops : min= 1, max= 5, avg= 2.33, stdev= 2.31, samples=3 lat (msec) : 50=25.00%, 100=25.00%, 250=8.33%, 500=16.67%, 750=25.00% cpu : usr=0.03%, sys=28.08%, ctx=1370, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9493: Tue Aug 27 10:41:39 2024 read: IOPS=1, BW=1604KiB/s (1642kB/s)(2048KiB/1277msec) clat (msec): min=105, max=268, avg=187.48, stdev=115.25 lat (msec): min=105, max=268, avg=187.48, stdev=115.25 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 107], 50.00th=[ 107], 60.00th=[ 271], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 2048, max= 2048, per=29.60%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=7, BW=8019KiB/s (8211kB/s)(10.0MiB/1277msec) clat (msec): min=30, max=464, avg=88.57, stdev=133.16 lat (msec): min=30, max=464, avg=88.63, stdev=133.16 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 37], 50.00th=[ 38], 60.00th=[ 43], | 70.00th=[ 68], 80.00th=[ 68], 90.00th=[ 78], 95.00th=[ 464], | 99.00th=[ 464], 99.50th=[ 464], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 6144, max=10240, per=84.58%, avg=8192.00, stdev=2896.31, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (msec) : 50=50.00%, 100=25.00%, 250=8.33%, 500=16.67% cpu : usr=0.00%, sys=31.03%, ctx=293, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9494: Tue Aug 27 10:41:39 2024 read: IOPS=2, BW=2760KiB/s (2826kB/s)(6144KiB/2226msec) clat (usec): min=525, max=591373, avg=233199.43, stdev=258307.24 lat (usec): min=526, max=591374, avg=233200.75, stdev=258307.11 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 529], 10.00th=[ 529], 20.00th=[ 42206], | 30.00th=[ 42206], 40.00th=[ 67634], 50.00th=[ 67634], 60.00th=[175113], | 70.00th=[522191], 80.00th=[522191], 90.00th=[591397], 95.00th=[591397], | 99.00th=[591397], 99.50th=[591397], 99.90th=[591397], 99.95th=[591397], | 99.99th=[591397] bw ( KiB/s): min= 2048, max= 6144, per=49.34%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 write: IOPS=2, BW=2760KiB/s (2826kB/s)(6144KiB/2226msec) clat (msec): min=27, max=513, avg=135.60, stdev=187.42 lat (msec): min=27, max=513, avg=135.66, stdev=187.42 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 65], 50.00th=[ 65], 60.00th=[ 86], | 70.00th=[ 96], 80.00th=[ 96], 90.00th=[ 514], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 2048, max= 4096, per=31.72%, avg=3072.00, stdev=1182.41, samples=4 iops : min= 2, max= 4, avg= 3.00, stdev= 1.15, samples=4 lat (usec) : 750=8.33% lat (msec) : 50=25.00%, 100=33.33%, 250=8.33%, 750=25.00% cpu : usr=0.00%, sys=21.30%, ctx=665, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9496: Tue Aug 27 10:41:39 2024 read: IOPS=2, BW=2289KiB/s (2344kB/s)(5120KiB/2237msec) clat (usec): min=479, max=507577, avg=209742.19, stdev=230531.84 lat (usec): min=480, max=507578, avg=209743.43, stdev=230531.79 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 482], 10.00th=[ 482], 20.00th=[ 482], | 30.00th=[ 515], 40.00th=[ 515], 50.00th=[149947], 60.00th=[149947], | 70.00th=[392168], 80.00th=[392168], 90.00th=[509608], 95.00th=[509608], | 99.00th=[509608], 99.50th=[509608], 99.90th=[509608], 99.95th=[509608], | 99.99th=[509608] bw ( KiB/s): min= 2048, max= 4096, per=39.47%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 write: IOPS=3, BW=3204KiB/s (3281kB/s)(7168KiB/2237msec) clat (msec): min=35, max=530, avg=162.29, stdev=176.13 lat (msec): min=35, max=530, avg=162.35, stdev=176.13 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 62], | 30.00th=[ 65], 40.00th=[ 65], 50.00th=[ 67], 60.00th=[ 142], | 70.00th=[ 142], 80.00th=[ 236], 90.00th=[ 531], 95.00th=[ 531], | 99.00th=[ 531], 99.50th=[ 531], 99.90th=[ 531], 99.95th=[ 531], | 99.99th=[ 531] bw ( KiB/s): min= 2048, max= 6144, per=49.34%, avg=4778.67, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 4.67, stdev= 2.31, samples=3 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=8.33%, 100=25.00%, 250=25.00%, 500=8.33%, 750=16.67% cpu : usr=0.00%, sys=16.50%, ctx=560, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6919KiB/s (7085kB/s), 1604KiB/s-2760KiB/s (1642kB/s-2826kB/s), io=20.0MiB (20.0MB), run=1277-2960msec WRITE: bw=9686KiB/s (9919kB/s), 1730KiB/s-8019KiB/s (1771kB/s-8211kB/s), io=28.0MiB (29.4MB), run=1277-2960msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9495: Tue Aug 27 10:41:39 2024 read: IOPS=2, BW=2380KiB/s (2437kB/s)(7168KiB/3012msec) clat (msec): min=14, max=561, avg=239.72, stdev=267.39 lat (msec): min=14, max=561, avg=239.72, stdev=267.39 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 16], 40.00th=[ 16], 50.00th=[ 66], 60.00th=[ 451], | 70.00th=[ 451], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 1787, max= 4063, per=35.28%, avg=2398.80, stdev=937.16, samples=5 iops : min= 1, max= 3, avg= 2.00, stdev= 0.71, samples=5 write: IOPS=1, BW=1700KiB/s (1741kB/s)(5120KiB/3012msec) clat (msec): min=20, max=593, avg=263.83, stdev=272.38 lat (msec): min=20, max=593, avg=263.87, stdev=272.38 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 52], 40.00th=[ 52], 50.00th=[ 133], 60.00th=[ 133], | 70.00th=[ 523], 80.00th=[ 523], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 2031, max= 3574, per=26.80%, avg=2551.00, stdev=885.98, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 lat (msec) : 20=25.00%, 50=8.33%, 100=16.67%, 250=8.33%, 500=8.33% lat (msec) : 750=33.33% cpu : usr=0.00%, sys=4.18%, ctx=63, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9497: Tue Aug 27 10:41:39 2024 read: IOPS=1, BW=1174KiB/s (1202kB/s)(2048KiB/1744msec) clat (msec): min=46, max=397, avg=222.39, stdev=248.12 lat (msec): min=46, max=397, avg=222.39, stdev=248.12 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 397], | 70.00th=[ 397], 80.00th=[ 397], 90.00th=[ 397], 95.00th=[ 397], | 99.00th=[ 397], 99.50th=[ 397], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 2048, max= 2048, per=30.12%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5872KiB/s (6012kB/s)(10.0MiB/1744msec) clat (msec): min=29, max=742, avg=121.81, stdev=218.98 lat (msec): min=29, max=742, avg=121.87, stdev=218.98 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 38], | 30.00th=[ 42], 40.00th=[ 43], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 70], 80.00th=[ 70], 90.00th=[ 101], 95.00th=[ 743], | 99.00th=[ 743], 99.50th=[ 743], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 6144, max=10240, per=86.06%, avg=8192.00, stdev=2896.31, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (msec) : 50=58.33%, 100=25.00%, 500=8.33%, 750=8.33% cpu : usr=0.06%, sys=9.47%, ctx=98, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9498: Tue Aug 27 10:41:39 2024 read: IOPS=2, BW=2755KiB/s (2821kB/s)(6144KiB/2230msec) clat (msec): min=16, max=444, avg=177.17, stdev=204.39 lat (msec): min=16, max=444, avg=177.17, stdev=204.39 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 20], | 30.00th=[ 20], 40.00th=[ 72], 50.00th=[ 72], 60.00th=[ 78], | 70.00th=[ 435], 80.00th=[ 435], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 2048, max= 6144, per=50.20%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 write: IOPS=2, BW=2755KiB/s (2821kB/s)(6144KiB/2230msec) clat (msec): min=17, max=570, avg=182.79, stdev=225.74 lat (msec): min=17, max=570, avg=183.03, stdev=225.70 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 23], | 30.00th=[ 23], 40.00th=[ 46], 50.00th=[ 46], 60.00th=[ 99], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 4087, max= 4096, per=43.00%, avg=4093.00, stdev= 5.20, samples=3 iops : min= 3, max= 4, avg= 3.67, stdev= 0.58, samples=3 lat (msec) : 20=25.00%, 50=16.67%, 100=25.00%, 500=25.00%, 750=8.33% cpu : usr=0.00%, sys=6.91%, ctx=79, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9499: Tue Aug 27 10:41:39 2024 read: IOPS=2, BW=2280KiB/s (2334kB/s)(5120KiB/2246msec) clat (msec): min=12, max=567, avg=200.08, stdev=238.92 lat (msec): min=12, max=567, avg=200.08, stdev=238.93 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 13], | 30.00th=[ 29], 40.00th=[ 29], 50.00th=[ 77], 60.00th=[ 77], | 70.00th=[ 317], 80.00th=[ 317], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 2048, max= 6144, per=60.24%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=3, BW=3191KiB/s (3268kB/s)(7168KiB/2246msec) clat (msec): min=22, max=590, avg=165.35, stdev=205.96 lat (msec): min=23, max=590, avg=165.43, stdev=205.95 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 40], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 61], 60.00th=[ 131], | 70.00th=[ 131], 80.00th=[ 271], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 2048, max= 6144, per=50.20%, avg=4778.67, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 4.67, stdev= 2.31, samples=3 lat (msec) : 20=8.33%, 50=33.33%, 100=16.67%, 250=8.33%, 500=16.67% lat (msec) : 750=16.67% cpu : usr=0.00%, sys=6.10%, ctx=94, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6799KiB/s (6963kB/s), 1174KiB/s-2755KiB/s (1202kB/s-2821kB/s), io=20.0MiB (20.0MB), run=1744-3012msec WRITE: bw=9519KiB/s (9748kB/s), 1700KiB/s-5872KiB/s (1741kB/s-6012kB/s), io=28.0MiB (29.4MB), run=1744-3012msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9507: Tue Aug 27 10:41:43 2024 read: IOPS=7, BW=29.9MiB/s (31.4MB/s)(12.0MiB/401msec) clat (msec): min=36, max=292, avg=129.74, stdev=141.40 lat (msec): min=36, max=292, avg=129.74, stdev=141.40 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 61], 50.00th=[ 61], 60.00th=[ 61], | 70.00th=[ 292], 80.00th=[ 292], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] lat (msec) : 50=33.33%, 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=19.75%, ctx=53, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9508: Tue Aug 27 10:41:43 2024 read: IOPS=0, BW=1439KiB/s (1473kB/s)(4096KiB/2847msec) clat (nsec): min=253790k, max=253790k, avg=253789972.00, stdev= 0.00 lat (nsec): min=253792k, max=253792k, avg=253792065.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 255], 10.00th=[ 255], 20.00th=[ 255], | 30.00th=[ 255], 40.00th=[ 255], 50.00th=[ 255], 60.00th=[ 255], | 70.00th=[ 255], 80.00th=[ 255], 90.00th=[ 255], 95.00th=[ 255], | 99.00th=[ 255], 99.50th=[ 255], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2877KiB/s (2946kB/s)(8192KiB/2847msec) clat (msec): min=401, max=2172, avg=1286.99, stdev=1252.72 lat (msec): min=401, max=2173, avg=1287.23, stdev=1252.74 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 401], 10.00th=[ 401], 20.00th=[ 401], | 30.00th=[ 401], 40.00th=[ 401], 50.00th=[ 401], 60.00th=[ 2165], | 70.00th=[ 2165], 80.00th=[ 2165], 90.00th=[ 2165], 95.00th=[ 2165], | 99.00th=[ 2165], 99.50th=[ 2165], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 8175, max= 8175, per=71.03%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=4.60%, ctx=31, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9510: Tue Aug 27 10:41:43 2024 write: IOPS=4, BW=16.2MiB/s (17.0MB/s)(12.0MiB/739msec) clat (msec): min=77, max=447, avg=243.71, stdev=187.81 lat (msec): min=77, max=447, avg=244.03, stdev=187.80 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 78], 10.00th=[ 78], 20.00th=[ 78], | 30.00th=[ 78], 40.00th=[ 207], 50.00th=[ 207], 60.00th=[ 207], | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min=16318, max=16318, per=100.00%, avg=16318.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.14%, sys=12.87%, ctx=48, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9512: Tue Aug 27 10:41:43 2024 write: IOPS=5, BW=20.5MiB/s (21.5MB/s)(12.0MiB/586msec) clat (msec): min=37, max=443, avg=188.80, stdev=221.66 lat (msec): min=37, max=443, avg=189.09, stdev=221.64 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 87], 50.00th=[ 87], 60.00th=[ 87], | 70.00th=[ 443], 80.00th=[ 443], 90.00th=[ 443], 95.00th=[ 443], | 99.00th=[ 443], 99.50th=[ 443], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=33.33%, 500=33.33% cpu : usr=0.17%, sys=18.46%, ctx=55, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5755KiB/s (5893kB/s), 1439KiB/s-29.9MiB/s (1473kB/s-31.4MB/s), io=16.0MiB (16.8MB), run=401-2847msec WRITE: bw=11.2MiB/s (11.8MB/s), 2877KiB/s-20.5MiB/s (2946kB/s-21.5MB/s), io=32.0MiB (33.6MB), run=586-2847msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9509: Tue Aug 27 10:41:43 2024 read: IOPS=1, BW=4431KiB/s (4538kB/s)(12.0MiB/2773msec) clat (msec): min=177, max=2196, avg=901.93, stdev=1124.15 lat (msec): min=177, max=2196, avg=901.93, stdev=1124.15 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 178], 10.00th=[ 178], 20.00th=[ 178], | 30.00th=[ 178], 40.00th=[ 334], 50.00th=[ 334], 60.00th=[ 334], | 70.00th=[ 2198], 80.00th=[ 2198], 90.00th=[ 2198], 95.00th=[ 2198], | 99.00th=[ 2198], 99.50th=[ 2198], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.04%, sys=42.24%, ctx=1142, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9511: Tue Aug 27 10:41:43 2024 read: IOPS=0, BW=1578KiB/s (1616kB/s)(4096KiB/2596msec) clat (nsec): min=222407k, max=222407k, avg=222407134.00, stdev= 0.00 lat (nsec): min=222410k, max=222410k, avg=222409531.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 224], 10.00th=[ 224], 20.00th=[ 224], | 30.00th=[ 224], 40.00th=[ 224], 50.00th=[ 224], 60.00th=[ 224], | 70.00th=[ 224], 80.00th=[ 224], 90.00th=[ 224], 95.00th=[ 224], | 99.00th=[ 224], 99.50th=[ 224], 99.90th=[ 224], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3156KiB/s (3231kB/s)(8192KiB/2596msec) clat (msec): min=252, max=2095, avg=1174.17, stdev=1303.11 lat (msec): min=253, max=2095, avg=1174.47, stdev=1303.04 clat percentiles (msec): | 1.00th=[ 253], 5.00th=[ 253], 10.00th=[ 253], 20.00th=[ 253], | 30.00th=[ 253], 40.00th=[ 253], 50.00th=[ 253], 60.00th=[ 2089], | 70.00th=[ 2089], 80.00th=[ 2089], 90.00th=[ 2089], 95.00th=[ 2089], | 99.00th=[ 2089], 99.50th=[ 2089], 99.90th=[ 2089], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 8192, max= 8192, per=71.83%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.04%, sys=12.22%, ctx=77, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9513: Tue Aug 27 10:41:43 2024 write: IOPS=1, BW=4277KiB/s (4380kB/s)(12.0MiB/2873msec) clat (msec): min=363, max=2082, avg=939.89, stdev=989.76 lat (msec): min=364, max=2083, avg=940.15, stdev=989.76 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 363], | 30.00th=[ 363], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 372], | 70.00th=[ 2089], 80.00th=[ 2089], 90.00th=[ 2089], 95.00th=[ 2089], | 99.00th=[ 2089], 99.50th=[ 2089], 99.90th=[ 2089], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 8192, max= 8192, per=71.83%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=9.71%, ctx=27, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9514: Tue Aug 27 10:41:43 2024 write: IOPS=1, BW=4834KiB/s (4950kB/s)(12.0MiB/2542msec) clat (msec): min=109, max=2168, avg=836.28, stdev=1155.52 lat (msec): min=110, max=2168, avg=836.78, stdev=1155.32 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 110], 10.00th=[ 110], 20.00th=[ 110], | 30.00th=[ 110], 40.00th=[ 230], 50.00th=[ 230], 60.00th=[ 230], | 70.00th=[ 2165], 80.00th=[ 2165], 90.00th=[ 2165], 95.00th=[ 2165], | 99.00th=[ 2165], 99.50th=[ 2165], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 8175, max= 8192, per=71.75%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=66.67% cpu : usr=0.04%, sys=13.11%, ctx=466, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5908KiB/s (6050kB/s), 1578KiB/s-4431KiB/s (1616kB/s-4538kB/s), io=16.0MiB (16.8MB), run=2596-2773msec WRITE: bw=11.1MiB/s (11.7MB/s), 3156KiB/s-4834KiB/s (3231kB/s-4950kB/s), io=32.0MiB (33.6MB), run=2542-2873msec