== sanity test 398b: DIO and buffer IO race ============== 03:38:54 (1734683934) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.62107 s, 31.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17563: Fri Dec 20 03:39:11 2024 read: IOPS=107, BW=429KiB/s (439kB/s)(5992KiB/13982msec) clat (usec): min=49, max=89591, avg=5118.27, stdev=3644.73 lat (usec): min=49, max=89592, avg=5119.18, stdev=3644.73 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 1729], 10.00th=[ 2376], 20.00th=[ 2900], | 30.00th=[ 3392], 40.00th=[ 3916], 50.00th=[ 4490], 60.00th=[ 5211], | 70.00th=[ 5997], 80.00th=[ 7111], 90.00th=[ 8586], 95.00th=[10028], | 99.00th=[15401], 99.50th=[17695], 99.90th=[30802], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 112, max= 640, per=24.62%, avg=420.04, stdev=92.95, samples=27 iops : min= 28, max= 160, avg=104.93, stdev=23.25, samples=27 write: IOPS=112, BW=450KiB/s (461kB/s)(6296KiB/13982msec) clat (usec): min=417, max=202964, avg=3957.65, stdev=6073.18 lat (usec): min=418, max=202966, avg=3959.25, stdev=6072.87 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 578], 10.00th=[ 668], 20.00th=[ 775], | 30.00th=[ 963], 40.00th=[ 2114], 50.00th=[ 3130], 60.00th=[ 3851], | 70.00th=[ 4817], 80.00th=[ 6456], 90.00th=[ 8848], 95.00th=[ 10552], | 99.00th=[ 14877], 99.50th=[ 16909], 99.90th=[ 24511], 99.95th=[202376], | 99.99th=[202376] bw ( KiB/s): min= 128, max= 640, per=24.63%, avg=439.85, stdev=98.78, samples=27 iops : min= 32, max= 160, avg=109.89, stdev=24.67, samples=27 lat (usec) : 50=0.10%, 100=1.56%, 250=0.62%, 500=1.07%, 750=8.37% lat (usec) : 1000=6.54% lat (msec) : 2=4.62%, 4=29.26%, 10=42.32%, 20=5.24%, 50=0.23% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.16%, sys=22.92%, ctx=6108, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17564: Fri Dec 20 03:39:11 2024 read: IOPS=105, BW=422KiB/s (432kB/s)(5940KiB/14074msec) clat (usec): min=44, max=20167, avg=5043.37, stdev=2755.45 lat (usec): min=44, max=20167, avg=5044.17, stdev=2755.49 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 1860], 10.00th=[ 2409], 20.00th=[ 2966], | 30.00th=[ 3359], 40.00th=[ 3851], 50.00th=[ 4490], 60.00th=[ 5211], | 70.00th=[ 5997], 80.00th=[ 7111], 90.00th=[ 8586], 95.00th=[10159], | 99.00th=[13698], 99.50th=[16188], 99.90th=[19268], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 80, max= 576, per=24.59%, avg=419.57, stdev=96.34, samples=28 iops : min= 20, max= 144, avg=104.79, stdev=24.11, samples=28 write: IOPS=112, BW=451KiB/s (462kB/s)(6348KiB/14074msec) clat (usec): min=421, max=243815, avg=4094.35, stdev=7221.24 lat (usec): min=422, max=243816, avg=4095.43, stdev=7221.14 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 553], 10.00th=[ 644], 20.00th=[ 750], | 30.00th=[ 906], 40.00th=[ 2212], 50.00th=[ 3261], 60.00th=[ 4047], | 70.00th=[ 5014], 80.00th=[ 6587], 90.00th=[ 8586], 95.00th=[ 10683], | 99.00th=[ 14222], 99.50th=[ 17171], 99.90th=[ 85459], 99.95th=[244319], | 99.99th=[244319] bw ( KiB/s): min= 144, max= 616, per=24.98%, avg=446.14, stdev=93.31, samples=28 iops : min= 36, max= 154, avg=111.43, stdev=23.37, samples=28 lat (usec) : 50=0.03%, 100=1.60%, 250=0.39%, 500=1.04%, 750=9.34% lat (usec) : 1000=6.22% lat (msec) : 2=4.23%, 4=28.32%, 10=42.94%, 20=5.73%, 50=0.10% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.25%, sys=22.94%, ctx=6086, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17566: Fri Dec 20 03:39:11 2024 read: IOPS=107, BW=431KiB/s (441kB/s)(6028KiB/13990msec) clat (usec): min=41, max=26425, avg=5010.63, stdev=2892.53 lat (usec): min=41, max=26426, avg=5011.50, stdev=2892.54 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 502], 10.00th=[ 2376], 20.00th=[ 2900], | 30.00th=[ 3326], 40.00th=[ 3851], 50.00th=[ 4490], 60.00th=[ 5211], | 70.00th=[ 5932], 80.00th=[ 7046], 90.00th=[ 8455], 95.00th=[10028], | 99.00th=[14877], 99.50th=[17695], 99.90th=[25035], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 96, max= 624, per=24.68%, avg=421.00, stdev=98.98, samples=27 iops : min= 24, max= 156, avg=105.22, stdev=24.76, samples=27 write: IOPS=111, BW=447KiB/s (458kB/s)(6260KiB/13990msec) clat (usec): min=373, max=254799, avg=4060.00, stdev=7428.59 lat (usec): min=374, max=254800, avg=4061.09, stdev=7428.60 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 570], 10.00th=[ 644], 20.00th=[ 750], | 30.00th=[ 898], 40.00th=[ 2024], 50.00th=[ 3163], 60.00th=[ 3949], | 70.00th=[ 5014], 80.00th=[ 6521], 90.00th=[ 8979], 95.00th=[ 10421], | 99.00th=[ 14615], 99.50th=[ 17695], 99.90th=[ 73925], 99.95th=[254804], | 99.99th=[254804] bw ( KiB/s): min= 112, max= 560, per=24.33%, avg=434.59, stdev=97.13, samples=27 iops : min= 28, max= 140, avg=108.63, stdev=24.26, samples=27 lat (usec) : 50=0.20%, 100=1.69%, 250=0.49%, 500=1.20%, 750=8.92% lat (usec) : 1000=6.51% lat (msec) : 2=4.23%, 4=28.52%, 10=42.81%, 20=5.21%, 50=0.16% lat (msec) : 100=0.03%, 500=0.03% cpu : usr=0.20%, sys=22.81%, ctx=6079, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17568: Fri Dec 20 03:39:11 2024 read: IOPS=108, BW=435KiB/s (445kB/s)(6052KiB/13917msec) clat (usec): min=47, max=249401, avg=5150.30, stdev=6851.04 lat (usec): min=48, max=249402, avg=5151.54, stdev=6850.97 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 1844], 10.00th=[ 2343], 20.00th=[ 2868], | 30.00th=[ 3326], 40.00th=[ 3851], 50.00th=[ 4424], 60.00th=[ 5276], | 70.00th=[ 6063], 80.00th=[ 7046], 90.00th=[ 8356], 95.00th=[ 9634], | 99.00th=[ 13435], 99.50th=[ 16319], 99.90th=[ 24511], 99.95th=[248513], | 99.99th=[248513] bw ( KiB/s): min= 112, max= 568, per=25.18%, avg=429.56, stdev=90.59, samples=27 iops : min= 28, max= 142, avg=107.33, stdev=22.64, samples=27 write: IOPS=112, BW=448KiB/s (459kB/s)(6236KiB/13917msec) clat (usec): min=440, max=65832, avg=3867.39, stdev=3796.62 lat (usec): min=440, max=65833, avg=3868.60, stdev=3796.77 clat percentiles (usec): | 1.00th=[ 461], 5.00th=[ 537], 10.00th=[ 635], 20.00th=[ 742], | 30.00th=[ 914], 40.00th=[ 1811], 50.00th=[ 3064], 60.00th=[ 4080], | 70.00th=[ 4948], 80.00th=[ 6456], 90.00th=[ 8586], 95.00th=[10421], | 99.00th=[14091], 99.50th=[17957], 99.90th=[26608], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 144, max= 640, per=24.83%, avg=443.48, stdev=91.76, samples=27 iops : min= 36, max= 160, avg=110.81, stdev=22.94, samples=27 lat (usec) : 50=0.13%, 100=1.50%, 250=0.52%, 500=1.53%, 750=8.79% lat (usec) : 1000=6.35% lat (msec) : 2=4.49%, 4=27.93%, 10=43.72%, 20=4.85%, 50=0.13% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.26%, sys=22.84%, ctx=6136, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1706KiB/s (1747kB/s), 422KiB/s-435KiB/s (432kB/s-445kB/s), io=23.4MiB (24.6MB), run=13917-14074msec WRITE: bw=1786KiB/s (1829kB/s), 447KiB/s-451KiB/s (458kB/s-462kB/s), io=24.6MiB (25.7MB), run=13917-14074msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17565: Fri Dec 20 03:39:14 2024 read: IOPS=85, BW=341KiB/s (349kB/s)(5992KiB/17583msec) clat (usec): min=1313, max=326722, avg=4800.33, stdev=9111.45 lat (usec): min=1313, max=326723, avg=4801.21, stdev=9111.44 clat percentiles (usec): | 1.00th=[ 1516], 5.00th=[ 1713], 10.00th=[ 1811], 20.00th=[ 2024], | 30.00th=[ 2278], 40.00th=[ 2638], 50.00th=[ 3195], 60.00th=[ 3752], | 70.00th=[ 4686], 80.00th=[ 6587], 90.00th=[ 9765], 95.00th=[ 13042], | 99.00th=[ 17171], 99.50th=[ 20317], 99.90th=[ 33817], 99.95th=[325059], | 99.99th=[325059] bw ( KiB/s): min= 80, max= 816, per=24.97%, avg=339.37, stdev=189.07, samples=35 iops : min= 20, max= 204, avg=84.80, stdev=47.29, samples=35 write: IOPS=89, BW=358KiB/s (367kB/s)(6296KiB/17583msec) clat (usec): min=2003, max=30818, avg=6550.12, stdev=4283.55 lat (usec): min=2003, max=30819, avg=6551.14, stdev=4283.52 clat percentiles (usec): | 1.00th=[ 2311], 5.00th=[ 2606], 10.00th=[ 2802], 20.00th=[ 3130], | 30.00th=[ 3458], 40.00th=[ 4047], 50.00th=[ 4883], 60.00th=[ 6128], | 70.00th=[ 7898], 80.00th=[ 9765], 90.00th=[12649], 95.00th=[15139], | 99.00th=[21103], 99.50th=[23987], 99.90th=[29492], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 48, max= 888, per=25.04%, avg=356.29, stdev=198.36, samples=35 iops : min= 12, max= 222, avg=89.03, stdev=49.62, samples=35 lat (msec) : 2=9.47%, 4=41.54%, 10=34.57%, 20=13.48%, 50=0.91% lat (msec) : 500=0.03% cpu : usr=0.23%, sys=10.53%, ctx=3726, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17567: Fri Dec 20 03:39:14 2024 read: IOPS=84, BW=337KiB/s (345kB/s)(5940KiB/17650msec) clat (usec): min=1351, max=222788, avg=4697.36, stdev=6818.32 lat (usec): min=1352, max=222791, avg=4698.06, stdev=6818.40 clat percentiles (usec): | 1.00th=[ 1516], 5.00th=[ 1680], 10.00th=[ 1795], 20.00th=[ 2008], | 30.00th=[ 2245], 40.00th=[ 2638], 50.00th=[ 3163], 60.00th=[ 3687], | 70.00th=[ 4424], 80.00th=[ 6194], 90.00th=[ 9896], 95.00th=[ 13304], | 99.00th=[ 18220], 99.50th=[ 21627], 99.90th=[ 33817], 99.95th=[223347], | 99.99th=[223347] bw ( KiB/s): min= 16, max= 784, per=24.61%, avg=334.40, stdev=196.41, samples=35 iops : min= 4, max= 196, avg=83.46, stdev=49.13, samples=35 write: IOPS=89, BW=360KiB/s (368kB/s)(6348KiB/17650msec) clat (usec): min=1996, max=95117, avg=6664.86, stdev=5024.05 lat (usec): min=1997, max=95119, avg=6665.81, stdev=5024.20 clat percentiles (usec): | 1.00th=[ 2278], 5.00th=[ 2606], 10.00th=[ 2835], 20.00th=[ 3130], | 30.00th=[ 3490], 40.00th=[ 4113], 50.00th=[ 4948], 60.00th=[ 6063], | 70.00th=[ 7963], 80.00th=[ 9765], 90.00th=[12518], 95.00th=[15139], | 99.00th=[22938], 99.50th=[26346], 99.90th=[37487], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 56, max= 838, per=24.95%, avg=354.97, stdev=189.70, samples=35 iops : min= 14, max= 209, avg=88.60, stdev=47.46, samples=35 lat (msec) : 2=9.70%, 4=41.28%, 10=34.47%, 20=13.31%, 50=1.17% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.14%, sys=10.54%, ctx=3708, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17569: Fri Dec 20 03:39:14 2024 read: IOPS=85, BW=343KiB/s (351kB/s)(6028KiB/17584msec) clat (usec): min=1294, max=26944, avg=4594.79, stdev=3642.88 lat (usec): min=1294, max=26945, avg=4595.51, stdev=3642.92 clat percentiles (usec): | 1.00th=[ 1565], 5.00th=[ 1713], 10.00th=[ 1844], 20.00th=[ 2040], | 30.00th=[ 2311], 40.00th=[ 2769], 50.00th=[ 3228], 60.00th=[ 3785], | 70.00th=[ 4621], 80.00th=[ 6390], 90.00th=[10159], 95.00th=[12649], | 99.00th=[17957], 99.50th=[20055], 99.90th=[22938], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 56, max= 848, per=25.12%, avg=341.43, stdev=188.36, samples=35 iops : min= 14, max= 212, avg=85.31, stdev=47.12, samples=35 write: IOPS=89, BW=356KiB/s (365kB/s)(6260KiB/17584msec) clat (msec): min=2, max=252, avg= 6.76, stdev= 7.75 lat (msec): min=2, max=252, avg= 6.76, stdev= 7.75 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 11], 90.00th=[ 13], 95.00th=[ 16], | 99.00th=[ 22], 99.50th=[ 25], 99.90th=[ 73], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 64, max= 792, per=24.88%, avg=354.00, stdev=195.29, samples=35 iops : min= 16, max= 198, avg=88.46, stdev=48.85, samples=35 lat (msec) : 2=9.15%, 4=42.32%, 10=33.37%, 20=14.16%, 50=0.94% lat (msec) : 100=0.03%, 500=0.03% cpu : usr=0.19%, sys=10.73%, ctx=3698, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17570: Fri Dec 20 03:39:14 2024 read: IOPS=85, BW=343KiB/s (351kB/s)(6052KiB/17658msec) clat (usec): min=1191, max=27413, avg=4603.21, stdev=3814.64 lat (usec): min=1192, max=27414, avg=4603.94, stdev=3814.72 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1680], 10.00th=[ 1811], 20.00th=[ 2008], | 30.00th=[ 2245], 40.00th=[ 2606], 50.00th=[ 3097], 60.00th=[ 3654], | 70.00th=[ 4490], 80.00th=[ 6587], 90.00th=[10290], 95.00th=[13173], | 99.00th=[18220], 99.50th=[20841], 99.90th=[26346], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 56, max= 832, per=24.94%, avg=338.91, stdev=193.39, samples=35 iops : min= 14, max= 208, avg=84.69, stdev=48.37, samples=35 write: IOPS=88, BW=353KiB/s (362kB/s)(6236KiB/17658msec) clat (usec): min=1993, max=232304, avg=6800.54, stdev=7527.92 lat (usec): min=1994, max=232307, avg=6801.45, stdev=7527.98 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 11], 90.00th=[ 13], 95.00th=[ 16], | 99.00th=[ 23], 99.50th=[ 26], 99.90th=[ 86], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 72, max= 824, per=24.54%, avg=349.20, stdev=196.77, samples=35 iops : min= 18, max= 206, avg=87.26, stdev=49.22, samples=35 lat (msec) : 2=9.90%, 4=41.73%, 10=32.88%, 20=14.13%, 50=1.30% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.12%, sys=10.64%, ctx=3753, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1360KiB/s (1392kB/s), 337KiB/s-343KiB/s (345kB/s-351kB/s), io=23.4MiB (24.6MB), run=17583-17658msec WRITE: bw=1424KiB/s (1458kB/s), 353KiB/s-360KiB/s (362kB/s-368kB/s), io=24.6MiB (25.7MB), run=17583-17658msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17598: Fri Dec 20 03:39:22 2024 read: IOPS=51, BW=831KiB/s (850kB/s)(5920KiB/7128msec) clat (usec): min=105, max=50663, avg=13196.97, stdev=4147.43 lat (usec): min=106, max=50664, avg=13201.02, stdev=4147.11 clat percentiles (usec): | 1.00th=[ 125], 5.00th=[ 8455], 10.00th=[ 9372], 20.00th=[10683], | 30.00th=[11469], 40.00th=[12125], 50.00th=[12780], 60.00th=[13435], | 70.00th=[14353], 80.00th=[15401], 90.00th=[17433], 95.00th=[19530], | 99.00th=[26870], 99.50th=[34341], 99.90th=[50594], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 576, max= 1024, per=24.21%, avg=813.57, stdev=125.31, samples=14 iops : min= 36, max= 64, avg=50.79, stdev= 7.83, samples=14 write: IOPS=55, BW=893KiB/s (915kB/s)(6368KiB/7128msec) clat (usec): min=717, max=64230, avg=5542.62, stdev=5901.03 lat (usec): min=718, max=64259, avg=5551.11, stdev=5918.57 clat percentiles (usec): | 1.00th=[ 816], 5.00th=[ 922], 10.00th=[ 1090], 20.00th=[ 1598], | 30.00th=[ 2900], 40.00th=[ 3294], 50.00th=[ 3752], 60.00th=[ 4621], | 70.00th=[ 5866], 80.00th=[ 7439], 90.00th=[12256], 95.00th=[16712], | 99.00th=[25035], 99.50th=[41681], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 480, max= 1216, per=25.02%, avg=884.43, stdev=195.09, samples=14 iops : min= 30, max= 76, avg=55.21, stdev=12.19, samples=14 lat (usec) : 250=0.52%, 500=0.13%, 750=0.13%, 1000=3.91% lat (msec) : 2=7.42%, 4=16.15%, 10=24.48%, 20=44.27%, 50=2.73% lat (msec) : 100=0.26% cpu : usr=0.11%, sys=19.49%, ctx=2792, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17599: Fri Dec 20 03:39:22 2024 read: IOPS=52, BW=847KiB/s (867kB/s)(5744KiB/6785msec) clat (usec): min=60, max=49291, avg=12793.64, stdev=4263.58 lat (usec): min=61, max=49292, avg=12795.81, stdev=4263.38 clat percentiles (usec): | 1.00th=[ 99], 5.00th=[ 8586], 10.00th=[ 9241], 20.00th=[10290], | 30.00th=[11207], 40.00th=[11863], 50.00th=[12518], 60.00th=[13042], | 70.00th=[14091], 80.00th=[15270], 90.00th=[17433], 95.00th=[19792], | 99.00th=[23462], 99.50th=[24249], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 640, max= 992, per=24.98%, avg=839.23, stdev=120.08, samples=13 iops : min= 40, max= 62, avg=52.38, stdev= 7.61, samples=13 write: IOPS=60, BW=964KiB/s (988kB/s)(6544KiB/6785msec) clat (usec): min=681, max=41852, avg=5285.23, stdev=5837.53 lat (usec): min=682, max=41854, avg=5286.59, stdev=5837.57 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 898], 10.00th=[ 1004], 20.00th=[ 1237], | 30.00th=[ 2212], 40.00th=[ 2835], 50.00th=[ 3326], 60.00th=[ 4047], | 70.00th=[ 5407], 80.00th=[ 7046], 90.00th=[13435], 95.00th=[17695], | 99.00th=[25560], 99.50th=[40109], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 576, max= 1312, per=26.66%, avg=942.54, stdev=193.90, samples=13 iops : min= 36, max= 82, avg=58.85, stdev=12.08, samples=13 lat (usec) : 100=0.65%, 250=0.78%, 750=0.13%, 1000=5.08% lat (msec) : 2=10.29%, 4=16.28%, 10=20.57%, 20=42.58%, 50=3.65% cpu : usr=0.09%, sys=19.60%, ctx=2717, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17600: Fri Dec 20 03:39:22 2024 read: IOPS=55, BW=893KiB/s (914kB/s)(6256KiB/7008msec) clat (usec): min=62, max=74934, avg=12956.03, stdev=5102.95 lat (usec): min=63, max=74935, avg=12956.86, stdev=5102.98 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 8455], 10.00th=[ 9110], 20.00th=[10290], | 30.00th=[11338], 40.00th=[11994], 50.00th=[12649], 60.00th=[13304], | 70.00th=[14091], 80.00th=[15008], 90.00th=[16909], 95.00th=[18744], | 99.00th=[29754], 99.50th=[37487], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 704, max= 1152, per=26.53%, avg=891.29, stdev=125.50, samples=14 iops : min= 44, max= 72, avg=55.64, stdev= 7.88, samples=14 write: IOPS=53, BW=861KiB/s (881kB/s)(6032KiB/7008msec) clat (usec): min=691, max=64080, avg=5088.87, stdev=5507.23 lat (usec): min=692, max=64081, avg=5090.32, stdev=5507.23 clat percentiles (usec): | 1.00th=[ 807], 5.00th=[ 979], 10.00th=[ 1074], 20.00th=[ 1369], | 30.00th=[ 2409], 40.00th=[ 2999], 50.00th=[ 3490], 60.00th=[ 4228], | 70.00th=[ 5473], 80.00th=[ 7111], 90.00th=[11207], 95.00th=[14615], | 99.00th=[26084], 99.50th=[28443], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 544, max= 1120, per=24.11%, avg=852.43, stdev=178.70, samples=14 iops : min= 34, max= 70, avg=53.21, stdev=11.19, samples=14 lat (usec) : 100=0.52%, 250=0.52%, 500=0.13%, 750=0.13%, 1000=2.47% lat (msec) : 2=10.42%, 4=15.49%, 10=22.40%, 20=45.44%, 50=2.21% lat (msec) : 100=0.26% cpu : usr=0.10%, sys=19.98%, ctx=2762, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17601: Fri Dec 20 03:39:22 2024 read: IOPS=53, BW=863KiB/s (884kB/s)(6032KiB/6988msec) clat (usec): min=60, max=38283, avg=12977.90, stdev=4323.97 lat (usec): min=60, max=38284, avg=12978.76, stdev=4324.00 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 7963], 10.00th=[ 9110], 20.00th=[10683], | 30.00th=[11338], 40.00th=[12256], 50.00th=[12780], 60.00th=[13435], | 70.00th=[14091], 80.00th=[15270], 90.00th=[17171], 95.00th=[19530], | 99.00th=[33424], 99.50th=[34341], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 704, max= 1088, per=25.27%, avg=849.08, stdev=114.42, samples=13 iops : min= 44, max= 68, avg=53.00, stdev= 7.19, samples=13 write: IOPS=55, BW=895KiB/s (917kB/s)(6256KiB/6988msec) clat (usec): min=765, max=39128, avg=5314.89, stdev=5138.83 lat (usec): min=766, max=39130, avg=5324.01, stdev=5134.97 clat percentiles (usec): | 1.00th=[ 816], 5.00th=[ 979], 10.00th=[ 1139], 20.00th=[ 1582], | 30.00th=[ 2704], 40.00th=[ 3097], 50.00th=[ 3621], 60.00th=[ 4228], | 70.00th=[ 5669], 80.00th=[ 7111], 90.00th=[11994], 95.00th=[15926], | 99.00th=[24511], 99.50th=[31851], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 704, max= 1120, per=25.27%, avg=893.38, stdev=141.62, samples=13 iops : min= 44, max= 70, avg=55.77, stdev= 8.87, samples=13 lat (usec) : 100=0.65%, 250=0.78%, 1000=2.86% lat (msec) : 2=8.59%, 4=17.84%, 10=20.31%, 20=45.96%, 50=2.99% cpu : usr=0.17%, sys=20.12%, ctx=2849, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3360KiB/s (3441kB/s), 831KiB/s-893KiB/s (850kB/s-914kB/s), io=23.4MiB (24.5MB), run=6785-7128msec WRITE: bw=3535KiB/s (3620kB/s), 861KiB/s-964KiB/s (881kB/s-988kB/s), io=24.6MiB (25.8MB), run=6785-7128msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17602: Fri Dec 20 03:39:22 2024 read: IOPS=51, BW=831KiB/s (851kB/s)(5920KiB/7126msec) clat (usec): min=1926, max=112231, avg=7518.25, stdev=7838.80 lat (usec): min=1927, max=112233, avg=7519.24, stdev=7838.74 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 9], 80.00th=[ 12], 90.00th=[ 16], 95.00th=[ 20], | 99.00th=[ 26], 99.50th=[ 40], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 640, max= 1024, per=24.13%, avg=811.14, stdev=109.67, samples=14 iops : min= 40, max= 64, avg=50.57, stdev= 6.78, samples=14 write: IOPS=55, BW=894KiB/s (915kB/s)(6368KiB/7126msec) clat (usec): min=3028, max=70840, avg=10861.97, stdev=6212.79 lat (usec): min=3029, max=70841, avg=10863.49, stdev=6212.77 clat percentiles (usec): | 1.00th=[ 3359], 5.00th=[ 3916], 10.00th=[ 4621], 20.00th=[ 6063], | 30.00th=[ 7504], 40.00th=[ 8586], 50.00th=[ 9765], 60.00th=[11076], | 70.00th=[12256], 80.00th=[14222], 90.00th=[18482], 95.00th=[20841], | 99.00th=[30016], 99.50th=[35914], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 512, max= 1152, per=25.01%, avg=884.21, stdev=189.53, samples=14 iops : min= 32, max= 72, avg=55.14, stdev=11.78, samples=14 lat (msec) : 2=0.13%, 4=22.01%, 10=40.49%, 20=31.90%, 50=5.21% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.06%, sys=10.37%, ctx=1204, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17603: Fri Dec 20 03:39:22 2024 read: IOPS=52, BW=841KiB/s (861kB/s)(5744KiB/6831msec) clat (usec): min=2123, max=51950, avg=7068.79, stdev=5436.57 lat (usec): min=2124, max=51951, avg=7069.65, stdev=5436.61 clat percentiles (usec): | 1.00th=[ 2245], 5.00th=[ 2540], 10.00th=[ 2737], 20.00th=[ 2966], | 30.00th=[ 3392], 40.00th=[ 3851], 50.00th=[ 4621], 60.00th=[ 6456], | 70.00th=[ 8979], 80.00th=[11338], 90.00th=[13960], 95.00th=[16450], | 99.00th=[23725], 99.50th=[27132], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 606, max= 1088, per=24.52%, avg=824.00, stdev=141.22, samples=13 iops : min= 37, max= 68, avg=51.23, stdev= 9.02, samples=13 write: IOPS=59, BW=958KiB/s (981kB/s)(6544KiB/6831msec) clat (usec): min=2987, max=51755, avg=10418.88, stdev=6401.48 lat (usec): min=2988, max=51756, avg=10420.41, stdev=6401.54 clat percentiles (usec): | 1.00th=[ 3261], 5.00th=[ 3752], 10.00th=[ 4293], 20.00th=[ 5014], | 30.00th=[ 6390], 40.00th=[ 7832], 50.00th=[ 8848], 60.00th=[10421], | 70.00th=[12125], 80.00th=[14353], 90.00th=[18744], 95.00th=[21365], | 99.00th=[29492], 99.50th=[42206], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 576, max= 1181, per=26.50%, avg=937.08, stdev=155.39, samples=13 iops : min= 36, max= 73, avg=58.31, stdev= 9.62, samples=13 lat (msec) : 4=23.70%, 10=41.67%, 20=29.56%, 50=4.69%, 100=0.39% cpu : usr=0.18%, sys=9.84%, ctx=1097, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17604: Fri Dec 20 03:39:22 2024 read: IOPS=64, BW=1026KiB/s (1050kB/s)(6256KiB/6100msec) clat (usec): min=1957, max=53084, avg=6262.34, stdev=4848.65 lat (usec): min=1957, max=53085, avg=6263.13, stdev=4848.65 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2507], 10.00th=[ 2704], 20.00th=[ 2999], | 30.00th=[ 3261], 40.00th=[ 3621], 50.00th=[ 4293], 60.00th=[ 5080], | 70.00th=[ 6915], 80.00th=[ 9896], 90.00th=[13042], 95.00th=[15139], | 99.00th=[19792], 99.50th=[22414], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 704, max= 1309, per=31.01%, avg=1042.17, stdev=158.69, samples=12 iops : min= 44, max= 81, avg=65.00, stdev= 9.73, samples=12 write: IOPS=61, BW=989KiB/s (1013kB/s)(6032KiB/6100msec) clat (usec): min=2900, max=49870, avg=9529.64, stdev=6623.79 lat (usec): min=2901, max=49872, avg=9531.11, stdev=6623.88 clat percentiles (usec): | 1.00th=[ 3228], 5.00th=[ 3818], 10.00th=[ 4146], 20.00th=[ 4555], | 30.00th=[ 5276], 40.00th=[ 6128], 50.00th=[ 7242], 60.00th=[ 8356], | 70.00th=[10552], 80.00th=[14353], 90.00th=[18220], 95.00th=[23462], | 99.00th=[32900], 99.50th=[39060], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 766, max= 1152, per=28.12%, avg=994.33, stdev=135.10, samples=12 iops : min= 47, max= 72, avg=62.00, stdev= 8.62, samples=12 lat (msec) : 2=0.13%, 4=27.34%, 10=47.01%, 20=21.61%, 50=3.78% lat (msec) : 100=0.13% cpu : usr=0.21%, sys=9.97%, ctx=971, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17605: Fri Dec 20 03:39:22 2024 read: IOPS=54, BW=864KiB/s (885kB/s)(6032KiB/6980msec) clat (usec): min=2047, max=28806, avg=7263.26, stdev=5148.99 lat (usec): min=2048, max=28806, avg=7264.06, stdev=5149.00 clat percentiles (usec): | 1.00th=[ 2180], 5.00th=[ 2474], 10.00th=[ 2737], 20.00th=[ 3032], | 30.00th=[ 3458], 40.00th=[ 3949], 50.00th=[ 4817], 60.00th=[ 6783], | 70.00th=[ 9372], 80.00th=[11863], 90.00th=[14353], 95.00th=[17957], | 99.00th=[21365], 99.50th=[27657], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 672, max= 1085, per=24.81%, avg=833.85, stdev=131.26, samples=13 iops : min= 42, max= 67, avg=51.92, stdev= 7.93, samples=13 write: IOPS=56, BW=896KiB/s (918kB/s)(6256KiB/6980msec) clat (usec): min=3020, max=64046, avg=10698.11, stdev=6686.80 lat (usec): min=3022, max=64048, avg=10699.74, stdev=6686.77 clat percentiles (usec): | 1.00th=[ 3195], 5.00th=[ 3949], 10.00th=[ 4293], 20.00th=[ 5211], | 30.00th=[ 6718], 40.00th=[ 7832], 50.00th=[ 9110], 60.00th=[10552], | 70.00th=[12387], 80.00th=[15139], 90.00th=[18744], 95.00th=[22676], | 99.00th=[36439], 99.50th=[39060], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 704, max= 1120, per=24.84%, avg=878.31, stdev=141.43, samples=13 iops : min= 44, max= 70, avg=54.69, stdev= 9.00, samples=13 lat (msec) : 4=22.92%, 10=40.89%, 20=30.73%, 50=5.34%, 100=0.13% cpu : usr=0.23%, sys=9.97%, ctx=1154, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3361KiB/s (3442kB/s), 831KiB/s-1026KiB/s (851kB/s-1050kB/s), io=23.4MiB (24.5MB), run=6100-7126msec WRITE: bw=3536KiB/s (3621kB/s), 894KiB/s-989KiB/s (915kB/s-1013kB/s), io=24.6MiB (25.8MB), run=6100-7126msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17618: Fri Dec 20 03:39:26 2024 read: IOPS=15, BW=15.1MiB/s (15.9MB/s)(7168KiB/463msec) clat (msec): min=14, max=134, avg=38.65, stdev=43.03 lat (msec): min=14, max=134, avg=38.66, stdev=43.03 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 20], | 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 21], 60.00th=[ 22], | 70.00th=[ 22], 80.00th=[ 42], 90.00th=[ 134], 95.00th=[ 134], | 99.00th=[ 134], 99.50th=[ 134], 99.90th=[ 134], 99.95th=[ 134], | 99.99th=[ 134] write: IOPS=10, BW=10.8MiB/s (11.3MB/s)(5120KiB/463msec) clat (usec): min=19176, max=68420, avg=36919.83, stdev=21609.87 lat (usec): min=19267, max=68504, avg=37009.79, stdev=21615.08 clat percentiles (usec): | 1.00th=[19268], 5.00th=[19268], 10.00th=[19268], 20.00th=[19268], | 30.00th=[21103], 40.00th=[21103], 50.00th=[25560], 60.00th=[25560], | 70.00th=[50594], 80.00th=[50594], 90.00th=[68682], 95.00th=[68682], | 99.00th=[68682], 99.50th=[68682], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] lat (msec) : 20=33.33%, 50=41.67%, 100=16.67%, 250=8.33% cpu : usr=0.43%, sys=22.08%, ctx=65, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17620: Fri Dec 20 03:39:26 2024 read: IOPS=1, BW=1028KiB/s (1053kB/s)(2048KiB/1992msec) clat (msec): min=32, max=872, avg=452.59, stdev=594.29 lat (msec): min=32, max=872, avg=452.59, stdev=594.29 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 33], 60.00th=[ 877], | 70.00th=[ 877], 80.00th=[ 877], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 2048, max= 2048, per=21.06%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5141KiB/s (5264kB/s)(10.0MiB/1992msec) clat (msec): min=20, max=420, avg=91.08, stdev=117.46 lat (msec): min=20, max=421, avg=91.15, stdev=117.46 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 32], | 30.00th=[ 50], 40.00th=[ 54], 50.00th=[ 56], 60.00th=[ 61], | 70.00th=[ 68], 80.00th=[ 68], 90.00th=[ 91], 95.00th=[ 422], | 99.00th=[ 422], 99.50th=[ 422], 99.90th=[ 422], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 2048, max= 8192, per=35.10%, avg=4778.67, stdev=3128.37, samples=3 iops : min= 2, max= 8, avg= 4.67, stdev= 3.06, samples=3 lat (msec) : 50=33.33%, 100=50.00%, 500=8.33%, 1000=8.33% cpu : usr=0.05%, sys=10.75%, ctx=115, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17621: Fri Dec 20 03:39:26 2024 read: IOPS=2, BW=2917KiB/s (2987kB/s)(6144KiB/2106msec) clat (msec): min=10, max=370, avg=90.59, stdev=139.98 lat (msec): min=10, max=370, avg=90.59, stdev=139.98 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 14], | 30.00th=[ 14], 40.00th=[ 18], 50.00th=[ 18], 60.00th=[ 45], | 70.00th=[ 87], 80.00th=[ 87], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 4087, max= 4096, per=42.08%, avg=4091.50, stdev= 6.36, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2917KiB/s (2987kB/s)(6144KiB/2106msec) clat (msec): min=26, max=1104, avg=238.41, stdev=424.95 lat (msec): min=27, max=1104, avg=238.47, stdev=424.95 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 62], | 30.00th=[ 62], 40.00th=[ 69], 50.00th=[ 69], 60.00th=[ 81], | 70.00th=[ 89], 80.00th=[ 89], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 4087, max= 4096, per=30.05%, avg=4091.50, stdev= 6.36, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 lat (msec) : 20=25.00%, 50=16.67%, 100=41.67%, 500=8.33% cpu : usr=0.00%, sys=5.70%, ctx=66, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17624: Fri Dec 20 03:39:26 2024 read: IOPS=2, BW=2444KiB/s (2503kB/s)(5120KiB/2095msec) clat (msec): min=11, max=379, avg=88.64, stdev=162.72 lat (msec): min=11, max=379, avg=88.64, stdev=162.72 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 12], | 30.00th=[ 12], 40.00th=[ 12], 50.00th=[ 16], 60.00th=[ 16], | 70.00th=[ 26], 80.00th=[ 26], 90.00th=[ 380], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 6144, max= 6144, per=63.18%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3421KiB/s (3504kB/s)(7168KiB/2095msec) clat (msec): min=15, max=1103, avg=220.25, stdev=396.50 lat (msec): min=15, max=1103, avg=220.33, stdev=396.50 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 33], 60.00th=[ 161], | 70.00th=[ 161], 80.00th=[ 192], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 2048, max= 4096, per=20.06%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 lat (msec) : 20=41.67%, 50=25.00%, 250=16.67%, 500=8.33% cpu : usr=0.00%, sys=6.02%, ctx=114, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9725KiB/s (9958kB/s), 1028KiB/s-15.1MiB/s (1053kB/s-15.9MB/s), io=20.0MiB (20.0MB), run=463-2106msec WRITE: bw=13.3MiB/s (13.9MB/s), 2917KiB/s-10.8MiB/s (2987kB/s-11.3MB/s), io=28.0MiB (29.4MB), run=463-2106msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17617: Fri Dec 20 03:39:26 2024 read: IOPS=2, BW=2900KiB/s (2969kB/s)(7168KiB/2472msec) clat (usec): min=471, max=672609, avg=297414.11, stdev=251487.79 lat (usec): min=473, max=672610, avg=297415.36, stdev=251487.82 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 474], 10.00th=[ 474], 20.00th=[ 11731], | 30.00th=[141558], 40.00th=[141558], 50.00th=[408945], 60.00th=[413139], | 70.00th=[413139], 80.00th=[434111], 90.00th=[675283], 95.00th=[675283], | 99.00th=[675283], 99.50th=[675283], 99.90th=[675283], 99.95th=[675283], | 99.99th=[675283] bw ( KiB/s): min= 2043, max= 3968, per=30.50%, avg=2526.75, stdev=960.84, samples=4 iops : min= 1, max= 3, avg= 2.00, stdev= 0.82, samples=4 write: IOPS=2, BW=2071KiB/s (2121kB/s)(5120KiB/2472msec) clat (msec): min=18, max=154, avg=55.71, stdev=56.62 lat (msec): min=18, max=154, avg=55.77, stdev=56.63 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 26], 40.00th=[ 26], 50.00th=[ 29], 60.00th=[ 29], | 70.00th=[ 52], 80.00th=[ 52], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 5953, max= 5953, per=51.33%, avg=5953.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 20=16.67%, 50=16.67%, 100=8.33%, 250=16.67%, 500=25.00% lat (msec) : 750=8.33% cpu : usr=0.16%, sys=19.30%, ctx=1199, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17619: Fri Dec 20 03:39:26 2024 read: IOPS=1, BW=1412KiB/s (1446kB/s)(2048KiB/1450msec) clat (msec): min=65, max=480, avg=272.80, stdev=293.27 lat (msec): min=65, max=480, avg=272.80, stdev=293.27 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 66], 10.00th=[ 66], 20.00th=[ 66], | 30.00th=[ 66], 40.00th=[ 66], 50.00th=[ 66], 60.00th=[ 481], | 70.00th=[ 481], 80.00th=[ 481], 90.00th=[ 481], 95.00th=[ 481], | 99.00th=[ 481], 99.50th=[ 481], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 2043, max= 2048, per=24.69%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=6, BW=7062KiB/s (7232kB/s)(10.0MiB/1450msec) clat (msec): min=29, max=368, avg=88.68, stdev=106.33 lat (msec): min=29, max=368, avg=88.75, stdev=106.32 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 32], | 30.00th=[ 35], 40.00th=[ 38], 50.00th=[ 39], 60.00th=[ 45], | 70.00th=[ 75], 80.00th=[ 75], 90.00th=[ 165], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 4096, max=10219, per=61.71%, avg=7157.50, stdev=4329.61, samples=2 iops : min= 4, max= 9, avg= 6.50, stdev= 3.54, samples=2 lat (msec) : 50=50.00%, 100=25.00%, 250=8.33%, 500=16.67% cpu : usr=0.00%, sys=28.02%, ctx=317, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17622: Fri Dec 20 03:39:26 2024 read: IOPS=3, BW=4061KiB/s (4158kB/s)(6144KiB/1513msec) clat (usec): min=519, max=652564, avg=134644.47, stdev=256535.92 lat (usec): min=520, max=652565, avg=134645.69, stdev=256535.58 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 519], 10.00th=[ 519], 20.00th=[ 523], | 30.00th=[ 523], 40.00th=[ 537], 50.00th=[ 537], 60.00th=[ 67634], | 70.00th=[ 86508], 80.00th=[ 86508], 90.00th=[650118], 95.00th=[650118], | 99.00th=[650118], 99.50th=[650118], 99.90th=[650118], 99.95th=[650118], | 99.99th=[650118] bw ( KiB/s): min= 4096, max= 6144, per=61.81%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=3, BW=4061KiB/s (4158kB/s)(6144KiB/1513msec) clat (msec): min=31, max=451, avg=108.22, stdev=168.47 lat (msec): min=31, max=451, avg=108.29, stdev=168.44 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 34], | 30.00th=[ 34], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 43], | 70.00th=[ 56], 80.00th=[ 56], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 2048, max= 8192, per=44.15%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (usec) : 750=25.00% lat (msec) : 50=33.33%, 100=25.00%, 500=8.33%, 750=8.33% cpu : usr=0.13%, sys=21.43%, ctx=380, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17623: Fri Dec 20 03:39:26 2024 read: IOPS=2, BW=2658KiB/s (2722kB/s)(5120KiB/1926msec) clat (usec): min=491, max=622938, avg=302274.16, stdev=262633.04 lat (usec): min=492, max=622940, avg=302275.30, stdev=262633.15 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 490], 10.00th=[ 490], 20.00th=[ 490], | 30.00th=[ 63177], 40.00th=[ 63177], 50.00th=[396362], 60.00th=[396362], | 70.00th=[429917], 80.00th=[429917], 90.00th=[624952], 95.00th=[624952], | 99.00th=[624952], 99.50th=[624952], 99.90th=[624952], 99.95th=[624952], | 99.99th=[624952] bw ( KiB/s): min= 2048, max= 6144, per=49.44%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=3, BW=3722KiB/s (3811kB/s)(7168KiB/1926msec) clat (msec): min=22, max=148, avg=53.21, stdev=43.74 lat (msec): min=22, max=148, avg=53.27, stdev=43.74 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 25], | 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 43], 60.00th=[ 46], | 70.00th=[ 46], 80.00th=[ 59], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 2048, max= 8192, per=44.15%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (usec) : 500=8.33% lat (msec) : 50=41.67%, 100=16.67%, 250=8.33%, 500=16.67%, 750=8.33% cpu : usr=0.00%, sys=20.21%, ctx=941, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8285KiB/s (8484kB/s), 1412KiB/s-4061KiB/s (1446kB/s-4158kB/s), io=20.0MiB (20.0MB), run=1450-2472msec WRITE: bw=11.3MiB/s (11.9MB/s), 2071KiB/s-7062KiB/s (2121kB/s-7232kB/s), io=28.0MiB (29.4MB), run=1450-2472msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17630: Fri Dec 20 03:39:29 2024 read: IOPS=1, BW=5357KiB/s (5485kB/s)(12.0MiB/2294msec) clat (msec): min=153, max=1396, avg=755.46, stdev=622.85 lat (msec): min=153, max=1396, avg=755.46, stdev=622.85 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 718], 50.00th=[ 718], 60.00th=[ 718], | 70.00th=[ 1401], 80.00th=[ 1401], 90.00th=[ 1401], 95.00th=[ 1401], | 99.00th=[ 1401], 99.50th=[ 1401], 99.90th=[ 1401], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=44.31%, ctx=804, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17631: Fri Dec 20 03:39:29 2024 read: IOPS=0, BW=1677KiB/s (1718kB/s)(4096KiB/2442msec) clat (nsec): min=154187k, max=154187k, avg=154187144.00, stdev= 0.00 lat (nsec): min=154190k, max=154190k, avg=154189659.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 155], | 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3355KiB/s (3435kB/s)(8192KiB/2442msec) clat (msec): min=108, max=2158, avg=1133.19, stdev=1449.82 lat (msec): min=108, max=2158, avg=1133.45, stdev=1449.76 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 109], 10.00th=[ 109], 20.00th=[ 109], | 30.00th=[ 109], 40.00th=[ 109], 50.00th=[ 109], 60.00th=[ 2165], | 70.00th=[ 2165], 80.00th=[ 2165], 90.00th=[ 2165], 95.00th=[ 2165], | 99.00th=[ 2165], 99.50th=[ 2165], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] lat (msec) : 250=66.67% cpu : usr=0.04%, sys=12.09%, ctx=119, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17632: Fri Dec 20 03:39:29 2024 write: IOPS=1, BW=5080KiB/s (5202kB/s)(12.0MiB/2419msec) clat (msec): min=230, max=1260, avg=798.86, stdev=523.55 lat (msec): min=230, max=1260, avg=799.20, stdev=523.41 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 230], 10.00th=[ 230], 20.00th=[ 230], | 30.00th=[ 230], 40.00th=[ 902], 50.00th=[ 902], 60.00th=[ 902], | 70.00th=[ 1267], 80.00th=[ 1267], 90.00th=[ 1267], 95.00th=[ 1267], | 99.00th=[ 1267], 99.50th=[ 1267], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 8192, max= 8192, per=61.05%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.04%, sys=12.20%, ctx=35, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17633: Fri Dec 20 03:39:29 2024 write: IOPS=1, BW=6942KiB/s (7109kB/s)(12.0MiB/1770msec) clat (msec): min=114, max=1396, avg=578.70, stdev=710.32 lat (msec): min=114, max=1396, avg=579.06, stdev=710.25 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 115], 10.00th=[ 115], 20.00th=[ 115], | 30.00th=[ 115], 40.00th=[ 226], 50.00th=[ 226], 60.00th=[ 226], | 70.00th=[ 1401], 80.00th=[ 1401], 90.00th=[ 1401], 95.00th=[ 1401], | 99.00th=[ 1401], 99.50th=[ 1401], 99.90th=[ 1401], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 8192, max= 8192, per=61.05%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.11%, sys=17.75%, ctx=34, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6709KiB/s (6870kB/s), 1677KiB/s-5357KiB/s (1718kB/s-5485kB/s), io=16.0MiB (16.8MB), run=2294-2442msec WRITE: bw=13.1MiB/s (13.7MB/s), 3355KiB/s-6942KiB/s (3435kB/s-7109kB/s), io=32.0MiB (33.6MB), run=1770-2442msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17635: Fri Dec 20 03:39:29 2024 read: IOPS=1, BW=4815KiB/s (4931kB/s)(12.0MiB/2552msec) clat (msec): min=657, max=1037, avg=841.04, stdev=190.02 lat (msec): min=657, max=1037, avg=841.04, stdev=190.02 clat percentiles (msec): | 1.00th=[ 659], 5.00th=[ 659], 10.00th=[ 659], 20.00th=[ 659], | 30.00th=[ 659], 40.00th=[ 827], 50.00th=[ 827], 60.00th=[ 827], | 70.00th=[ 1036], 80.00th=[ 1036], 90.00th=[ 1036], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=4.63%, ctx=71, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17636: Fri Dec 20 03:39:29 2024 read: IOPS=0, BW=1590KiB/s (1628kB/s)(4096KiB/2576msec) clat (nsec): min=1103.0M, max=1103.0M, avg=1103045030.00, stdev= 0.00 lat (nsec): min=1103.0M, max=1103.0M, avg=1103046841.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1099], 5.00th=[ 1099], 10.00th=[ 1099], 20.00th=[ 1099], | 30.00th=[ 1099], 40.00th=[ 1099], 50.00th=[ 1099], 60.00th=[ 1099], | 70.00th=[ 1099], 80.00th=[ 1099], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3180KiB/s (3256kB/s)(8192KiB/2576msec) clat (msec): min=74, max=1380, avg=727.28, stdev=923.32 lat (msec): min=74, max=1380, avg=727.55, stdev=923.32 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 74], 20.00th=[ 74], | 30.00th=[ 74], 40.00th=[ 74], 50.00th=[ 74], 60.00th=[ 1385], | 70.00th=[ 1385], 80.00th=[ 1385], 90.00th=[ 1385], 95.00th=[ 1385], | 99.00th=[ 1385], 99.50th=[ 1385], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 7447, max= 7447, per=58.66%, avg=7447.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33% cpu : usr=0.04%, sys=5.17%, ctx=72, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17637: Fri Dec 20 03:39:29 2024 write: IOPS=1, BW=4761KiB/s (4875kB/s)(12.0MiB/2581msec) clat (msec): min=627, max=1074, avg=853.51, stdev=223.61 lat (msec): min=627, max=1074, avg=853.81, stdev=223.64 clat percentiles (msec): | 1.00th=[ 625], 5.00th=[ 625], 10.00th=[ 625], 20.00th=[ 625], | 30.00th=[ 625], 40.00th=[ 860], 50.00th=[ 860], 60.00th=[ 860], | 70.00th=[ 1083], 80.00th=[ 1083], 90.00th=[ 1083], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7380, max= 8192, per=61.33%, avg=7786.00, stdev=574.17, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 750=33.33%, 1000=33.33% cpu : usr=0.08%, sys=2.52%, ctx=28, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17638: Fri Dec 20 03:39:29 2024 write: IOPS=1, BW=6147KiB/s (6295kB/s)(12.0MiB/1999msec) clat (msec): min=71, max=1071, avg=663.03, stdev=524.40 lat (msec): min=71, max=1071, avg=663.62, stdev=524.55 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 72], 10.00th=[ 72], 20.00th=[ 72], | 30.00th=[ 72], 40.00th=[ 844], 50.00th=[ 844], 60.00th=[ 844], | 70.00th=[ 1070], 80.00th=[ 1070], 90.00th=[ 1070], 95.00th=[ 1070], | 99.00th=[ 1070], 99.50th=[ 1070], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 8192, max= 8192, per=64.53%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 1000=33.33% cpu : usr=0.05%, sys=5.36%, ctx=105, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6360KiB/s (6513kB/s), 1590KiB/s-4815KiB/s (1628kB/s-4931kB/s), io=16.0MiB (16.8MB), run=2552-2576msec WRITE: bw=12.4MiB/s (13.0MB/s), 3180KiB/s-6147KiB/s (3256kB/s-6295kB/s), io=32.0MiB (33.6MB), run=1999-2581msec