== sanity test 398b: DIO and buffer IO race ============== 01:02:13 (1735106533) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.18555 s, 42.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4097: Wed Dec 25 01:02:29 2024 read: IOPS=111, BW=446KiB/s (457kB/s)(5992KiB/13433msec) clat (usec): min=49, max=176564, avg=4294.03, stdev=5705.43 lat (usec): min=49, max=176565, avg=4295.65, stdev=5706.24 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 69], 10.00th=[ 90], 20.00th=[ 2311], | 30.00th=[ 2737], 40.00th=[ 3097], 50.00th=[ 3589], 60.00th=[ 4178], | 70.00th=[ 4948], 80.00th=[ 6259], 90.00th=[ 7767], 95.00th=[ 9241], | 99.00th=[ 13960], 99.50th=[ 17433], 99.90th=[ 67634], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 296, max= 584, per=24.79%, avg=438.50, stdev=77.79, samples=26 iops : min= 74, max= 146, avg=109.50, stdev=19.43, samples=26 write: IOPS=117, BW=469KiB/s (480kB/s)(6296KiB/13433msec) clat (usec): min=477, max=51702, avg=4391.68, stdev=4289.16 lat (usec): min=478, max=51702, avg=4392.46, stdev=4289.18 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 570], 10.00th=[ 619], 20.00th=[ 766], | 30.00th=[ 1090], 40.00th=[ 2212], 50.00th=[ 3589], 60.00th=[ 4490], | 70.00th=[ 5669], 80.00th=[ 7177], 90.00th=[ 9896], 95.00th=[11994], | 99.00th=[18220], 99.50th=[20317], 99.90th=[49546], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 295, max= 664, per=24.74%, avg=458.19, stdev=85.99, samples=26 iops : min= 73, max= 166, avg=114.42, stdev=21.53, samples=26 lat (usec) : 50=0.20%, 100=5.44%, 250=1.33%, 500=0.29%, 750=9.70% lat (usec) : 1000=4.92% lat (msec) : 2=5.76%, 4=28.39%, 10=37.04%, 20=6.45%, 50=0.36% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.12%, sys=24.91%, ctx=6325, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4098: Wed Dec 25 01:02:29 2024 read: IOPS=111, BW=446KiB/s (457kB/s)(5940KiB/13321msec) clat (usec): min=47, max=27451, avg=4061.84, stdev=3001.53 lat (usec): min=47, max=27452, avg=4062.64, stdev=3001.52 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 71], 10.00th=[ 90], 20.00th=[ 2245], | 30.00th=[ 2671], 40.00th=[ 2999], 50.00th=[ 3458], 60.00th=[ 4113], | 70.00th=[ 5014], 80.00th=[ 6128], 90.00th=[ 7767], 95.00th=[ 9241], | 99.00th=[13304], 99.50th=[16450], 99.90th=[25297], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 256, max= 648, per=25.03%, avg=442.69, stdev=97.32, samples=26 iops : min= 64, max= 162, avg=110.62, stdev=24.37, samples=26 write: IOPS=119, BW=477KiB/s (488kB/s)(6348KiB/13321msec) clat (usec): min=469, max=176175, avg=4539.38, stdev=6165.70 lat (usec): min=469, max=176176, avg=4540.38, stdev=6165.81 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 553], 10.00th=[ 603], 20.00th=[ 717], | 30.00th=[ 979], 40.00th=[ 1975], 50.00th=[ 3523], 60.00th=[ 4490], | 70.00th=[ 5800], 80.00th=[ 7308], 90.00th=[ 10028], 95.00th=[ 12911], | 99.00th=[ 17957], 99.50th=[ 20317], 99.90th=[ 64750], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 296, max= 632, per=25.28%, avg=468.15, stdev=95.93, samples=26 iops : min= 74, max= 158, avg=117.00, stdev=23.95, samples=26 lat (usec) : 50=0.10%, 100=5.70%, 250=1.40%, 500=0.39%, 750=10.71% lat (usec) : 1000=4.79% lat (msec) : 2=5.37%, 4=28.39%, 10=36.13%, 20=6.48%, 50=0.49% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.17%, sys=24.84%, ctx=6244, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4099: Wed Dec 25 01:02:29 2024 read: IOPS=111, BW=444KiB/s (455kB/s)(6028KiB/13572msec) clat (usec): min=48, max=37559, avg=4123.46, stdev=3203.57 lat (usec): min=48, max=37560, avg=4124.19, stdev=3203.53 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 65], 10.00th=[ 85], 20.00th=[ 2147], | 30.00th=[ 2606], 40.00th=[ 2966], 50.00th=[ 3458], 60.00th=[ 4178], | 70.00th=[ 5145], 80.00th=[ 6325], 90.00th=[ 8160], 95.00th=[ 9634], | 99.00th=[15008], 99.50th=[16712], 99.90th=[26084], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 184, max= 608, per=24.27%, avg=429.38, stdev=96.16, samples=26 iops : min= 46, max= 152, avg=107.27, stdev=24.08, samples=26 write: IOPS=115, BW=461KiB/s (472kB/s)(6260KiB/13572msec) clat (usec): min=467, max=218817, avg=4640.14, stdev=7101.58 lat (usec): min=467, max=218819, avg=4641.45, stdev=7101.58 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 562], 10.00th=[ 603], 20.00th=[ 725], | 30.00th=[ 963], 40.00th=[ 2278], 50.00th=[ 3556], 60.00th=[ 4555], | 70.00th=[ 5735], 80.00th=[ 7373], 90.00th=[ 9896], 95.00th=[ 12387], | 99.00th=[ 21627], 99.50th=[ 27657], 99.90th=[ 55837], 99.95th=[219153], | 99.99th=[219153] bw ( KiB/s): min= 248, max= 608, per=24.08%, avg=445.96, stdev=80.40, samples=26 iops : min= 62, max= 152, avg=111.42, stdev=20.12, samples=26 lat (usec) : 50=0.13%, 100=6.05%, 250=1.56%, 500=0.46%, 750=11.04% lat (usec) : 1000=4.52% lat (msec) : 2=4.52%, 4=27.86%, 10=36.98%, 20=6.18%, 50=0.62% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.16%, sys=23.93%, ctx=6119, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4100: Wed Dec 25 01:02:29 2024 read: IOPS=112, BW=451KiB/s (462kB/s)(6052KiB/13409msec) clat (usec): min=47, max=188186, avg=4280.62, stdev=5752.09 lat (usec): min=48, max=188187, avg=4281.35, stdev=5752.11 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 67], 10.00th=[ 90], 20.00th=[ 2180], | 30.00th=[ 2606], 40.00th=[ 2999], 50.00th=[ 3490], 60.00th=[ 4047], | 70.00th=[ 5145], 80.00th=[ 6456], 90.00th=[ 8029], 95.00th=[ 9634], | 99.00th=[ 14877], 99.50th=[ 18220], 99.90th=[ 39584], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 288, max= 622, per=25.23%, avg=446.35, stdev=93.31, samples=26 iops : min= 72, max= 155, avg=111.54, stdev=23.29, samples=26 write: IOPS=116, BW=465KiB/s (476kB/s)(6236KiB/13409msec) clat (usec): min=459, max=42358, avg=4392.88, stdev=4346.26 lat (usec): min=460, max=42359, avg=4394.37, stdev=4346.23 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 570], 10.00th=[ 619], 20.00th=[ 750], | 30.00th=[ 1037], 40.00th=[ 2180], 50.00th=[ 3556], 60.00th=[ 4490], | 70.00th=[ 5538], 80.00th=[ 7177], 90.00th=[ 9896], 95.00th=[12387], | 99.00th=[19006], 99.50th=[22152], 99.90th=[41157], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 344, max= 680, per=24.87%, avg=460.50, stdev=93.62, samples=26 iops : min= 86, max= 170, avg=115.08, stdev=23.41, samples=26 lat (usec) : 50=0.16%, 100=5.70%, 250=1.60%, 500=0.55%, 750=9.77% lat (usec) : 1000=4.95% lat (msec) : 2=5.50%, 4=29.30%, 10=35.55%, 20=6.38%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=24.26%, ctx=6325, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1769KiB/s (1812kB/s), 444KiB/s-451KiB/s (455kB/s-462kB/s), io=23.4MiB (24.6MB), run=13321-13572msec WRITE: bw=1852KiB/s (1897kB/s), 461KiB/s-477KiB/s (472kB/s-488kB/s), io=24.6MiB (25.7MB), run=13321-13572msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4092: Wed Dec 25 01:02:51 2024 read: IOPS=42, BW=172KiB/s (176kB/s)(5992KiB/34922msec) clat (usec): min=1669, max=180278, avg=3755.99, stdev=5705.89 lat (usec): min=1669, max=180281, avg=3756.70, stdev=5705.94 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1876], 10.00th=[ 1926], 20.00th=[ 2024], | 30.00th=[ 2114], 40.00th=[ 2212], 50.00th=[ 2311], 60.00th=[ 2540], | 70.00th=[ 2933], 80.00th=[ 4359], 90.00th=[ 7242], 95.00th=[ 10028], | 99.00th=[ 18744], 99.50th=[ 24511], 99.90th=[ 50594], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 55, max= 352, per=25.16%, avg=171.62, stdev=70.01, samples=69 iops : min= 13, max= 88, avg=42.84, stdev=17.53, samples=69 write: IOPS=45, BW=180KiB/s (185kB/s)(6296KiB/34922msec) clat (usec): min=8622, max=83029, avg=18540.32, stdev=7128.79 lat (usec): min=8623, max=83029, avg=18541.22, stdev=7128.81 clat percentiles (usec): | 1.00th=[10028], 5.00th=[11076], 10.00th=[11731], 20.00th=[12780], | 30.00th=[14091], 40.00th=[15533], 50.00th=[17171], 60.00th=[18482], | 70.00th=[20055], 80.00th=[22414], 90.00th=[28443], 95.00th=[32637], | 99.00th=[41681], 99.50th=[47449], 99.90th=[57934], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 56, max= 256, per=25.17%, avg=179.74, stdev=54.81, samples=69 iops : min= 14, max= 64, avg=44.87, stdev=13.77, samples=69 lat (msec) : 2=8.56%, 4=29.46%, 10=8.76%, 20=37.57%, 50=15.40% lat (msec) : 100=0.23%, 250=0.03% cpu : usr=0.10%, sys=5.71%, ctx=3590, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4093: Wed Dec 25 01:02:51 2024 read: IOPS=42, BW=169KiB/s (173kB/s)(5940KiB/35104msec) clat (usec): min=1636, max=192666, avg=3700.55, stdev=5893.44 lat (usec): min=1636, max=192668, avg=3701.35, stdev=5893.49 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1876], 10.00th=[ 1942], 20.00th=[ 2024], | 30.00th=[ 2114], 40.00th=[ 2212], 50.00th=[ 2343], 60.00th=[ 2474], | 70.00th=[ 2933], 80.00th=[ 4293], 90.00th=[ 6849], 95.00th=[ 9503], | 99.00th=[ 17171], 99.50th=[ 23725], 99.90th=[ 41681], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 16, max= 311, per=24.87%, avg=169.58, stdev=69.51, samples=69 iops : min= 4, max= 77, avg=42.28, stdev=17.29, samples=69 write: IOPS=45, BW=181KiB/s (185kB/s)(6348KiB/35104msec) clat (usec): min=9143, max=83374, avg=18585.61, stdev=7044.88 lat (usec): min=9144, max=83375, avg=18586.54, stdev=7044.88 clat percentiles (usec): | 1.00th=[10159], 5.00th=[11207], 10.00th=[11863], 20.00th=[12911], | 30.00th=[14091], 40.00th=[15664], 50.00th=[17433], 60.00th=[18744], | 70.00th=[20055], 80.00th=[22152], 90.00th=[28181], 95.00th=[33162], | 99.00th=[42206], 99.50th=[44303], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 72, max= 256, per=25.20%, avg=179.90, stdev=52.63, samples=69 iops : min= 18, max= 64, avg=44.86, stdev=13.10, samples=69 lat (msec) : 2=8.07%, 4=29.85%, 10=8.56%, 20=37.83%, 50=15.56% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.07%, sys=5.67%, ctx=3613, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4094: Wed Dec 25 01:02:51 2024 read: IOPS=42, BW=171KiB/s (175kB/s)(6028KiB/35194msec) clat (usec): min=1691, max=63034, avg=3891.10, stdev=4009.69 lat (usec): min=1692, max=63035, avg=3891.84, stdev=4009.69 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1876], 10.00th=[ 1942], 20.00th=[ 2024], | 30.00th=[ 2114], 40.00th=[ 2212], 50.00th=[ 2376], 60.00th=[ 2606], | 70.00th=[ 3130], 80.00th=[ 5211], 90.00th=[ 7767], 95.00th=[10683], | 99.00th=[18744], 99.50th=[25035], 99.90th=[42730], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 48, max= 296, per=25.09%, avg=171.09, stdev=62.27, samples=70 iops : min= 12, max= 74, avg=42.66, stdev=15.54, samples=70 write: IOPS=44, BW=178KiB/s (182kB/s)(6260KiB/35194msec) clat (msec): min=8, max=221, avg=18.67, stdev= 8.52 lat (msec): min=8, max=221, avg=18.67, stdev= 8.52 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 29], 95.00th=[ 32], | 99.00th=[ 42], 99.50th=[ 46], 99.90th=[ 84], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 56, max= 256, per=24.84%, avg=177.34, stdev=56.19, samples=70 iops : min= 14, max= 64, avg=44.23, stdev=14.00, samples=70 lat (msec) : 2=8.53%, 4=28.32%, 10=10.12%, 20=36.91%, 50=15.89% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.08%, sys=5.76%, ctx=3589, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4095: Wed Dec 25 01:02:51 2024 read: IOPS=43, BW=175KiB/s (179kB/s)(6052KiB/34636msec) clat (usec): min=1646, max=80635, avg=3699.52, stdev=3845.93 lat (usec): min=1646, max=80636, avg=3700.35, stdev=3845.96 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1860], 10.00th=[ 1926], 20.00th=[ 2008], | 30.00th=[ 2089], 40.00th=[ 2180], 50.00th=[ 2311], 60.00th=[ 2573], | 70.00th=[ 2999], 80.00th=[ 4621], 90.00th=[ 7111], 95.00th=[10159], | 99.00th=[19006], 99.50th=[21627], 99.90th=[42206], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 40, max= 360, per=25.61%, avg=174.65, stdev=70.48, samples=69 iops : min= 10, max= 90, avg=43.52, stdev=17.56, samples=69 write: IOPS=45, BW=180KiB/s (184kB/s)(6236KiB/34636msec) clat (msec): min=9, max=205, avg=18.55, stdev= 8.31 lat (msec): min=9, max=205, avg=18.55, stdev= 8.31 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 28], 95.00th=[ 32], | 99.00th=[ 42], 99.50th=[ 50], 99.90th=[ 85], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 56, max= 248, per=25.16%, avg=179.64, stdev=54.16, samples=69 iops : min= 14, max= 62, avg=44.77, stdev=13.49, samples=69 lat (msec) : 2=9.28%, 4=28.84%, 10=9.24%, 20=37.17%, 50=15.20% lat (msec) : 100=0.23%, 250=0.03% cpu : usr=0.14%, sys=5.73%, ctx=3574, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=682KiB/s (699kB/s), 169KiB/s-175KiB/s (173kB/s-179kB/s), io=23.4MiB (24.6MB), run=34636-35194msec WRITE: bw=714KiB/s (731kB/s), 178KiB/s-181KiB/s (182kB/s-185kB/s), io=24.6MiB (25.7MB), run=34636-35194msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4139: Wed Dec 25 01:02:57 2024 read: IOPS=70, BW=1132KiB/s (1159kB/s)(5920KiB/5229msec) clat (usec): min=50, max=29229, avg=10118.10, stdev=5635.16 lat (usec): min=50, max=29230, avg=10118.79, stdev=5635.17 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 76], 10.00th=[ 98], 20.00th=[ 7439], | 30.00th=[ 9241], 40.00th=[10028], 50.00th=[11076], 60.00th=[11863], | 70.00th=[12911], 80.00th=[14091], 90.00th=[15926], 95.00th=[18220], | 99.00th=[21627], 99.50th=[25035], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 960, max= 1312, per=24.92%, avg=1113.60, stdev=111.67, samples=10 iops : min= 60, max= 82, avg=69.60, stdev= 6.98, samples=10 write: IOPS=76, BW=1218KiB/s (1247kB/s)(6368KiB/5229msec) clat (usec): min=660, max=31203, avg=3666.54, stdev=4141.81 lat (usec): min=661, max=31204, avg=3667.72, stdev=4141.82 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 799], 10.00th=[ 857], 20.00th=[ 947], | 30.00th=[ 1106], 40.00th=[ 1369], 50.00th=[ 2278], 60.00th=[ 3392], | 70.00th=[ 4080], 80.00th=[ 5866], 90.00th=[ 7635], 95.00th=[ 9634], | 99.00th=[23462], 99.50th=[29492], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 896, max= 1472, per=25.38%, avg=1193.60, stdev=224.28, samples=10 iops : min= 56, max= 92, avg=74.60, stdev=14.02, samples=10 lat (usec) : 100=5.34%, 250=3.65%, 750=1.04%, 1000=11.59% lat (msec) : 2=12.50%, 4=10.55%, 10=23.83%, 20=29.56%, 50=1.95% cpu : usr=0.19%, sys=22.46%, ctx=2147, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4140: Wed Dec 25 01:02:57 2024 read: IOPS=69, BW=1114KiB/s (1141kB/s)(5744KiB/5154msec) clat (usec): min=50, max=28224, avg=10287.25, stdev=5936.13 lat (usec): min=50, max=28225, avg=10288.10, stdev=5936.18 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 70], 10.00th=[ 96], 20.00th=[ 635], | 30.00th=[ 9503], 40.00th=[10683], 50.00th=[11469], 60.00th=[12387], | 70.00th=[13304], 80.00th=[14353], 90.00th=[16712], 95.00th=[18744], | 99.00th=[22414], 99.50th=[26870], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 832, max= 1373, per=24.91%, avg=1113.30, stdev=184.77, samples=10 iops : min= 52, max= 85, avg=69.50, stdev=11.42, samples=10 write: IOPS=79, BW=1270KiB/s (1300kB/s)(6544KiB/5154msec) clat (usec): min=670, max=22994, avg=3483.17, stdev=3855.84 lat (usec): min=670, max=22995, avg=3484.30, stdev=3855.87 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 791], 10.00th=[ 857], 20.00th=[ 955], | 30.00th=[ 1106], 40.00th=[ 1270], 50.00th=[ 1696], 60.00th=[ 3064], | 70.00th=[ 3687], 80.00th=[ 5342], 90.00th=[ 7963], 95.00th=[11600], | 99.00th=[19792], 99.50th=[21103], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 896, max= 1536, per=26.60%, avg=1250.90, stdev=189.50, samples=10 iops : min= 56, max= 96, avg=78.10, stdev=11.74, samples=10 lat (usec) : 100=6.12%, 250=3.12%, 750=1.17%, 1000=11.98% lat (msec) : 2=14.45%, 4=11.72%, 10=17.32%, 20=32.03%, 50=2.08% cpu : usr=0.25%, sys=23.02%, ctx=2101, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4141: Wed Dec 25 01:02:57 2024 read: IOPS=72, BW=1167KiB/s (1195kB/s)(6256KiB/5359msec) clat (usec): min=52, max=40944, avg=10364.10, stdev=6157.49 lat (usec): min=53, max=40945, avg=10364.83, stdev=6157.48 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 76], 10.00th=[ 96], 20.00th=[ 7373], | 30.00th=[ 9241], 40.00th=[10290], 50.00th=[11338], 60.00th=[12256], | 70.00th=[13173], 80.00th=[14353], 90.00th=[16319], 95.00th=[19006], | 99.00th=[26346], 99.50th=[30016], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 896, max= 1600, per=25.99%, avg=1161.60, stdev=203.53, samples=10 iops : min= 56, max= 100, avg=72.60, stdev=12.72, samples=10 write: IOPS=70, BW=1126KiB/s (1153kB/s)(6032KiB/5359msec) clat (usec): min=615, max=26670, avg=3388.97, stdev=3737.24 lat (usec): min=615, max=26671, avg=3390.19, stdev=3737.33 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 799], 10.00th=[ 848], 20.00th=[ 988], | 30.00th=[ 1123], 40.00th=[ 1336], 50.00th=[ 1795], 60.00th=[ 2999], | 70.00th=[ 3818], 80.00th=[ 5080], 90.00th=[ 7767], 95.00th=[ 9503], | 99.00th=[21890], 99.50th=[26346], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 704, max= 1440, per=23.41%, avg=1100.80, stdev=253.86, samples=10 iops : min= 44, max= 90, avg=68.80, stdev=15.87, samples=10 lat (usec) : 100=6.38%, 250=3.12%, 750=0.78%, 1000=9.64% lat (msec) : 2=15.76%, 4=9.77%, 10=19.27%, 20=32.42%, 50=2.86% cpu : usr=0.04%, sys=22.66%, ctx=2145, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4143: Wed Dec 25 01:02:57 2024 read: IOPS=73, BW=1172KiB/s (1200kB/s)(6032KiB/5146msec) clat (usec): min=52, max=25228, avg=10070.69, stdev=5985.37 lat (usec): min=52, max=25229, avg=10072.11, stdev=5985.95 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 66], 10.00th=[ 78], 20.00th=[ 135], | 30.00th=[ 9372], 40.00th=[10683], 50.00th=[11469], 60.00th=[12256], | 70.00th=[13566], 80.00th=[14615], 90.00th=[15926], 95.00th=[17695], | 99.00th=[22676], 99.50th=[24511], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 958, max= 1696, per=25.92%, avg=1158.20, stdev=218.70, samples=10 iops : min= 59, max= 106, avg=72.30, stdev=13.76, samples=10 write: IOPS=75, BW=1216KiB/s (1245kB/s)(6256KiB/5146msec) clat (usec): min=647, max=26707, avg=3412.23, stdev=3470.24 lat (usec): min=648, max=26709, avg=3413.64, stdev=3470.33 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 783], 10.00th=[ 848], 20.00th=[ 955], | 30.00th=[ 1057], 40.00th=[ 1287], 50.00th=[ 2057], 60.00th=[ 3261], | 70.00th=[ 4080], 80.00th=[ 5276], 90.00th=[ 7504], 95.00th=[10028], | 99.00th=[19006], 99.50th=[21627], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 960, max= 1696, per=25.92%, avg=1218.90, stdev=239.60, samples=10 iops : min= 60, max= 106, avg=76.10, stdev=14.96, samples=10 lat (usec) : 100=7.42%, 250=3.39%, 750=1.30%, 1000=11.72% lat (msec) : 2=12.37%, 4=10.16%, 10=18.75%, 20=33.59%, 50=1.30% cpu : usr=0.19%, sys=22.70%, ctx=2065, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4469KiB/s (4577kB/s), 1114KiB/s-1172KiB/s (1141kB/s-1200kB/s), io=23.4MiB (24.5MB), run=5146-5359msec WRITE: bw=4702KiB/s (4815kB/s), 1126KiB/s-1270KiB/s (1153kB/s-1300kB/s), io=24.6MiB (25.8MB), run=5146-5359msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4142: Wed Dec 25 01:03:02 2024 read: IOPS=37, BW=597KiB/s (612kB/s)(5920KiB/9908msec) clat (usec): min=1946, max=25174, avg=4333.36, stdev=3508.20 lat (usec): min=1947, max=25175, avg=4334.11, stdev=3508.22 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2147], 10.00th=[ 2212], 20.00th=[ 2343], | 30.00th=[ 2474], 40.00th=[ 2606], 50.00th=[ 2835], 60.00th=[ 3195], | 70.00th=[ 3720], 80.00th=[ 5473], 90.00th=[ 8586], 95.00th=[13435], | 99.00th=[16909], 99.50th=[18220], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 288, max= 1120, per=24.58%, avg=587.74, stdev=215.28, samples=19 iops : min= 18, max= 70, avg=36.68, stdev=13.48, samples=19 write: IOPS=40, BW=643KiB/s (658kB/s)(6368KiB/9908msec) clat (usec): min=11432, max=45688, avg=20677.58, stdev=6464.50 lat (usec): min=11433, max=45689, avg=20679.18, stdev=6464.45 clat percentiles (usec): | 1.00th=[11731], 5.00th=[12780], 10.00th=[13435], 20.00th=[14877], | 30.00th=[16450], 40.00th=[17957], 50.00th=[19792], 60.00th=[21103], | 70.00th=[22676], 80.00th=[25297], 90.00th=[29492], 95.00th=[33162], | 99.00th=[42730], 99.50th=[44303], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 416, max= 832, per=25.11%, avg=631.47, stdev=151.79, samples=19 iops : min= 26, max= 52, avg=39.42, stdev= 9.44, samples=19 lat (msec) : 2=0.39%, 4=34.37%, 10=8.98%, 20=30.73%, 50=25.52% cpu : usr=0.13%, sys=5.92%, ctx=925, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4144: Wed Dec 25 01:03:02 2024 read: IOPS=35, BW=573KiB/s (587kB/s)(5744KiB/10016msec) clat (usec): min=1826, max=22622, avg=4164.65, stdev=3434.51 lat (usec): min=1827, max=22622, avg=4165.41, stdev=3434.53 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2114], 10.00th=[ 2212], 20.00th=[ 2311], | 30.00th=[ 2442], 40.00th=[ 2573], 50.00th=[ 2835], 60.00th=[ 3064], | 70.00th=[ 3458], 80.00th=[ 5014], 90.00th=[ 8455], 95.00th=[12911], | 99.00th=[17957], 99.50th=[19530], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 224, max= 1117, per=23.58%, avg=563.84, stdev=223.65, samples=19 iops : min= 14, max= 69, avg=35.11, stdev=13.87, samples=19 write: IOPS=40, BW=653KiB/s (669kB/s)(6544KiB/10016msec) clat (usec): min=10610, max=50923, avg=20742.91, stdev=6770.83 lat (usec): min=10610, max=50924, avg=20744.51, stdev=6770.82 clat percentiles (usec): | 1.00th=[11600], 5.00th=[12518], 10.00th=[13304], 20.00th=[15008], | 30.00th=[16909], 40.00th=[18220], 50.00th=[19792], 60.00th=[21103], | 70.00th=[22676], 80.00th=[24773], 90.00th=[29230], 95.00th=[33817], | 99.00th=[44303], 99.50th=[49546], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 448, max= 864, per=25.37%, avg=638.05, stdev=144.91, samples=19 iops : min= 28, max= 54, avg=39.74, stdev= 9.11, samples=19 lat (msec) : 2=0.26%, 4=35.03%, 10=7.29%, 20=31.64%, 50=25.65% lat (msec) : 100=0.13% cpu : usr=0.14%, sys=5.88%, ctx=919, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4145: Wed Dec 25 01:03:02 2024 read: IOPS=40, BW=644KiB/s (660kB/s)(6256KiB/9710msec) clat (usec): min=1957, max=28398, avg=4517.84, stdev=4123.62 lat (usec): min=1958, max=28399, avg=4518.60, stdev=4123.66 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2180], 10.00th=[ 2245], 20.00th=[ 2343], | 30.00th=[ 2474], 40.00th=[ 2606], 50.00th=[ 2802], 60.00th=[ 3130], | 70.00th=[ 3589], 80.00th=[ 5604], 90.00th=[10159], 95.00th=[13435], | 99.00th=[23725], 99.50th=[28181], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 352, max= 1248, per=26.98%, avg=645.00, stdev=234.79, samples=19 iops : min= 22, max= 78, avg=40.26, stdev=14.73, samples=19 write: IOPS=38, BW=621KiB/s (636kB/s)(6032KiB/9710msec) clat (usec): min=11013, max=53963, avg=21024.17, stdev=6722.93 lat (usec): min=11014, max=53965, avg=21025.60, stdev=6723.05 clat percentiles (usec): | 1.00th=[11863], 5.00th=[13042], 10.00th=[13566], 20.00th=[15401], | 30.00th=[17171], 40.00th=[18744], 50.00th=[20317], 60.00th=[21627], | 70.00th=[22938], 80.00th=[25035], 90.00th=[28443], 95.00th=[33817], | 99.00th=[46400], 99.50th=[51119], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 416, max= 832, per=24.37%, avg=612.95, stdev=133.82, samples=19 iops : min= 26, max= 52, avg=38.26, stdev= 8.39, samples=19 lat (msec) : 2=0.26%, 4=36.85%, 10=8.59%, 20=28.26%, 50=25.78% lat (msec) : 100=0.26% cpu : usr=0.14%, sys=6.18%, ctx=892, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4146: Wed Dec 25 01:03:02 2024 read: IOPS=38, BW=613KiB/s (628kB/s)(6032KiB/9841msec) clat (usec): min=1932, max=32208, avg=4679.80, stdev=4333.60 lat (usec): min=1933, max=32208, avg=4680.56, stdev=4333.59 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2114], 10.00th=[ 2245], 20.00th=[ 2376], | 30.00th=[ 2507], 40.00th=[ 2638], 50.00th=[ 2900], 60.00th=[ 3195], | 70.00th=[ 3982], 80.00th=[ 5932], 90.00th=[ 9503], 95.00th=[14615], | 99.00th=[21890], 99.50th=[27657], 99.90th=[32113], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 320, max= 926, per=25.28%, avg=604.47, stdev=172.47, samples=19 iops : min= 20, max= 57, avg=37.68, stdev=10.75, samples=19 write: IOPS=39, BW=636KiB/s (651kB/s)(6256KiB/9841msec) clat (usec): min=10932, max=50486, avg=20503.09, stdev=6686.06 lat (usec): min=10934, max=50487, avg=20505.09, stdev=6686.69 clat percentiles (usec): | 1.00th=[11469], 5.00th=[12649], 10.00th=[13173], 20.00th=[14353], | 30.00th=[16057], 40.00th=[17957], 50.00th=[19268], 60.00th=[20841], | 70.00th=[22414], 80.00th=[25297], 90.00th=[28705], 95.00th=[32637], | 99.00th=[44303], 99.50th=[49546], 99.90th=[50594], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 416, max= 832, per=25.11%, avg=631.42, stdev=168.86, samples=19 iops : min= 26, max= 52, avg=39.37, stdev=10.55, samples=19 lat (msec) : 2=0.39%, 4=33.98%, 10=10.16%, 20=30.60%, 50=24.74% lat (msec) : 100=0.13% cpu : usr=0.08%, sys=6.05%, ctx=909, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2391KiB/s (2449kB/s), 573KiB/s-644KiB/s (587kB/s-660kB/s), io=23.4MiB (24.5MB), run=9710-10016msec WRITE: bw=2516KiB/s (2576kB/s), 621KiB/s-653KiB/s (636kB/s-669kB/s), io=24.6MiB (25.8MB), run=9710-10016msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4163: Wed Dec 25 01:03:05 2024 read: IOPS=5, BW=5251KiB/s (5377kB/s)(7168KiB/1365msec) clat (usec): min=391, max=562294, avg=142147.29, stdev=209197.29 lat (usec): min=392, max=562295, avg=142148.26, stdev=209197.10 clat percentiles (usec): | 1.00th=[ 392], 5.00th=[ 392], 10.00th=[ 392], 20.00th=[ 408], | 30.00th=[ 433], 40.00th=[ 433], 50.00th=[ 50594], 60.00th=[110625], | 70.00th=[110625], 80.00th=[270533], 90.00th=[566232], 95.00th=[566232], | 99.00th=[566232], 99.50th=[566232], 99.90th=[566232], 99.95th=[566232], | 99.99th=[566232] bw ( KiB/s): min= 1686, max= 8192, per=43.00%, avg=4939.00, stdev=4600.44, samples=2 iops : min= 1, max= 8, avg= 4.50, stdev= 4.95, samples=2 write: IOPS=3, BW=3751KiB/s (3841kB/s)(5120KiB/1365msec) clat (msec): min=22, max=139, avg=57.08, stdev=48.11 lat (msec): min=22, max=139, avg=57.14, stdev=48.13 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 23], | 30.00th=[ 30], 40.00th=[ 30], 50.00th=[ 35], 60.00th=[ 35], | 70.00th=[ 59], 80.00th=[ 59], 90.00th=[ 140], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 2048, max= 5060, per=22.10%, avg=3554.00, stdev=2129.81, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (usec) : 500=25.00% lat (msec) : 50=25.00%, 100=16.67%, 250=16.67%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=33.36%, ctx=415, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4164: Wed Dec 25 01:03:05 2024 read: IOPS=1, BW=1149KiB/s (1176kB/s)(2048KiB/1783msec) clat (usec): min=12398, max=78641, avg=45520.43, stdev=46840.86 lat (usec): min=12399, max=78642, avg=45521.39, stdev=46840.98 clat percentiles (usec): | 1.00th=[12387], 5.00th=[12387], 10.00th=[12387], 20.00th=[12387], | 30.00th=[12387], 40.00th=[12387], 50.00th=[12387], 60.00th=[78119], | 70.00th=[78119], 80.00th=[78119], 90.00th=[78119], 95.00th=[78119], | 99.00th=[78119], 99.50th=[78119], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 1686, max= 1689, per=14.69%, avg=1687.50, stdev= 2.12, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5743KiB/s (5881kB/s)(10.0MiB/1783msec) clat (msec): min=32, max=546, avg=164.33, stdev=159.86 lat (msec): min=32, max=547, avg=164.39, stdev=159.87 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 39], | 30.00th=[ 49], 40.00th=[ 70], 50.00th=[ 75], 60.00th=[ 115], | 70.00th=[ 228], 80.00th=[ 228], 90.00th=[ 266], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 3379, max= 6759, per=31.50%, avg=5066.00, stdev=1690.01, samples=3 iops : min= 3, max= 6, avg= 4.33, stdev= 1.53, samples=3 lat (msec) : 20=8.33%, 50=25.00%, 100=25.00%, 250=25.00%, 500=8.33% lat (msec) : 750=8.33% cpu : usr=0.06%, sys=14.37%, ctx=92, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4165: Wed Dec 25 01:03:05 2024 read: IOPS=4, BW=4662KiB/s (4773kB/s)(6144KiB/1318msec) clat (usec): min=371, max=867439, avg=185163.10, stdev=340938.02 lat (usec): min=372, max=867440, avg=185164.01, stdev=340937.88 clat percentiles (usec): | 1.00th=[ 371], 5.00th=[ 371], 10.00th=[ 371], 20.00th=[ 388], | 30.00th=[ 388], 40.00th=[ 416], 50.00th=[ 416], 60.00th=[ 70779], | 70.00th=[170918], 80.00th=[170918], 90.00th=[868221], 95.00th=[868221], | 99.00th=[868221], 99.50th=[868221], 99.90th=[868221], 99.95th=[868221], | 99.99th=[868221] bw ( KiB/s): min=10240, max=10240, per=89.15%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4662KiB/s (4773kB/s)(6144KiB/1318msec) clat (usec): min=15898, max=53053, avg=32223.71, stdev=16396.73 lat (usec): min=15916, max=53072, avg=32275.97, stdev=16396.74 clat percentiles (usec): | 1.00th=[15926], 5.00th=[15926], 10.00th=[15926], 20.00th=[22152], | 30.00th=[22152], 40.00th=[23462], 50.00th=[23462], 60.00th=[25822], | 70.00th=[52691], 80.00th=[52691], 90.00th=[53216], 95.00th=[53216], | 99.00th=[53216], 99.50th=[53216], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min=12288, max=12288, per=76.42%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (usec) : 500=25.00% lat (msec) : 20=8.33%, 50=25.00%, 100=25.00%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=66.89%, ctx=214, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4166: Wed Dec 25 01:03:05 2024 read: IOPS=12, BW=12.4MiB/s (13.0MB/s)(5120KiB/402msec) clat (usec): min=416, max=82672, avg=27956.90, stdev=36421.09 lat (usec): min=417, max=82674, avg=27957.90, stdev=36421.37 clat percentiles (usec): | 1.00th=[ 416], 5.00th=[ 416], 10.00th=[ 416], 20.00th=[ 416], | 30.00th=[ 4080], 40.00th=[ 4080], 50.00th=[ 4228], 60.00th=[ 4228], | 70.00th=[48497], 80.00th=[48497], 90.00th=[82314], 95.00th=[82314], | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] write: IOPS=17, BW=17.4MiB/s (18.3MB/s)(7168KiB/402msec) clat (usec): min=21032, max=68105, avg=33715.11, stdev=16001.13 lat (usec): min=21050, max=68184, avg=33789.17, stdev=16008.20 clat percentiles (usec): | 1.00th=[21103], 5.00th=[21103], 10.00th=[21103], 20.00th=[22414], | 30.00th=[27395], 40.00th=[27395], 50.00th=[30540], 60.00th=[30802], | 70.00th=[30802], 80.00th=[35914], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] lat (usec) : 500=8.33% lat (msec) : 10=16.67%, 50=58.33%, 100=16.67% cpu : usr=0.25%, sys=41.90%, ctx=65, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.2MiB/s (11.8MB/s), 1149KiB/s-12.4MiB/s (1176kB/s-13.0MB/s), io=20.0MiB (20.0MB), run=402-1783msec WRITE: bw=15.7MiB/s (16.5MB/s), 3751KiB/s-17.4MiB/s (3841kB/s-18.3MB/s), io=28.0MiB (29.4MB), run=402-1783msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4167: Wed Dec 25 01:03:05 2024 read: IOPS=4, BW=4814KiB/s (4930kB/s)(7168KiB/1489msec) clat (msec): min=12, max=862, avg=185.37, stdev=319.00 lat (msec): min=12, max=862, avg=185.37, stdev=319.00 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 14], 50.00th=[ 15], 60.00th=[ 63], | 70.00th=[ 63], 80.00th=[ 321], 90.00th=[ 860], 95.00th=[ 860], | 99.00th=[ 860], 99.50th=[ 860], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 4096, max= 4096, per=36.78%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3439KiB/s (3521kB/s)(5120KiB/1489msec) clat (usec): min=21826, max=30161, avg=24144.56, stdev=3405.00 lat (usec): min=21906, max=30208, avg=24200.72, stdev=3397.74 clat percentiles (usec): | 1.00th=[21890], 5.00th=[21890], 10.00th=[21890], 20.00th=[21890], | 30.00th=[22676], 40.00th=[22676], 50.00th=[22938], 60.00th=[22938], | 70.00th=[23200], 80.00th=[23200], 90.00th=[30278], 95.00th=[30278], | 99.00th=[30278], 99.50th=[30278], 99.90th=[30278], 99.95th=[30278], | 99.99th=[30278] lat (msec) : 20=33.33%, 50=41.67%, 100=8.33%, 500=8.33%, 1000=8.33% cpu : usr=0.07%, sys=5.31%, ctx=30, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4168: Wed Dec 25 01:03:05 2024 read: IOPS=1, BW=1114KiB/s (1140kB/s)(2048KiB/1839msec) clat (usec): min=77713, max=87113, avg=82413.59, stdev=6646.49 lat (usec): min=77714, max=87113, avg=82414.46, stdev=6646.14 clat percentiles (usec): | 1.00th=[78119], 5.00th=[78119], 10.00th=[78119], 20.00th=[78119], | 30.00th=[78119], 40.00th=[78119], 50.00th=[78119], 60.00th=[87557], | 70.00th=[87557], 80.00th=[87557], 90.00th=[87557], 95.00th=[87557], | 99.00th=[87557], 99.50th=[87557], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 2048, max= 2048, per=18.39%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5568KiB/s (5702kB/s)(10.0MiB/1839msec) clat (msec): min=42, max=531, avg=166.47, stdev=154.60 lat (msec): min=42, max=531, avg=166.54, stdev=154.60 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 51], | 30.00th=[ 52], 40.00th=[ 75], 50.00th=[ 75], 60.00th=[ 109], | 70.00th=[ 230], 80.00th=[ 230], 90.00th=[ 279], 95.00th=[ 531], | 99.00th=[ 531], 99.50th=[ 531], 99.90th=[ 531], 99.95th=[ 531], | 99.99th=[ 531] bw ( KiB/s): min= 2023, max= 8192, per=34.98%, avg=5453.00, stdev=3142.01, samples=3 iops : min= 1, max= 8, avg= 5.00, stdev= 3.61, samples=3 lat (msec) : 50=8.33%, 100=50.00%, 250=25.00%, 500=8.33%, 750=8.33% cpu : usr=0.05%, sys=6.26%, ctx=55, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4169: Wed Dec 25 01:03:05 2024 read: IOPS=4, BW=4613KiB/s (4723kB/s)(6144KiB/1332msec) clat (usec): min=12776, max=53599, avg=32011.80, stdev=15919.83 lat (usec): min=12777, max=53599, avg=32012.74, stdev=15919.88 clat percentiles (usec): | 1.00th=[12780], 5.00th=[12780], 10.00th=[12780], 20.00th=[21627], | 30.00th=[21627], 40.00th=[24249], 50.00th=[24249], 60.00th=[31589], | 70.00th=[47973], 80.00th=[47973], 90.00th=[53740], 95.00th=[53740], | 99.00th=[53740], 99.50th=[53740], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min=10240, max=10240, per=91.95%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4613KiB/s (4723kB/s)(6144KiB/1332msec) clat (msec): min=25, max=509, avg=178.22, stdev=198.69 lat (msec): min=25, max=509, avg=178.28, stdev=198.68 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 40], | 30.00th=[ 40], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 82], | 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 4087, max= 6144, per=32.81%, avg=5115.50, stdev=1454.52, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 lat (msec) : 20=8.33%, 50=50.00%, 100=25.00%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=6.24%, ctx=36, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4170: Wed Dec 25 01:03:05 2024 read: IOPS=3, BW=3241KiB/s (3318kB/s)(5120KiB/1580msec) clat (msec): min=21, max=501, avg=176.35, stdev=210.14 lat (msec): min=21, max=501, avg=176.35, stdev=210.14 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 42], 60.00th=[ 42], | 70.00th=[ 279], 80.00th=[ 279], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 2048, max= 6144, per=36.78%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=4, BW=4537KiB/s (4646kB/s)(7168KiB/1580msec) clat (msec): min=25, max=315, avg=90.10, stdev=104.74 lat (msec): min=25, max=315, avg=90.15, stdev=104.74 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 28], | 30.00th=[ 29], 40.00th=[ 29], 50.00th=[ 36], 60.00th=[ 94], | 70.00th=[ 94], 80.00th=[ 106], 90.00th=[ 317], 95.00th=[ 317], | 99.00th=[ 317], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 4087, max= 6144, per=30.63%, avg=4775.67, stdev=1185.02, samples=3 iops : min= 3, max= 6, avg= 4.33, stdev= 1.53, samples=3 lat (msec) : 50=58.33%, 100=8.33%, 250=8.33%, 500=16.67%, 750=8.33% cpu : usr=0.06%, sys=4.56%, ctx=53, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.9MiB/s (11.4MB/s), 1114KiB/s-4814KiB/s (1140kB/s-4930kB/s), io=20.0MiB (20.0MB), run=1332-1839msec WRITE: bw=15.2MiB/s (15.0MB/s), 3439KiB/s-5568KiB/s (3521kB/s-5702kB/s), io=28.0MiB (29.4MB), run=1332-1839msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4177: Wed Dec 25 01:03:08 2024 read: IOPS=1, BW=6298KiB/s (6449kB/s)(12.0MiB/1951msec) clat (msec): min=100, max=1624, avg=640.79, stdev=853.40 lat (msec): min=100, max=1624, avg=640.79, stdev=853.40 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 101], 10.00th=[ 101], 20.00th=[ 101], | 30.00th=[ 101], 40.00th=[ 199], 50.00th=[ 199], 60.00th=[ 199], | 70.00th=[ 1620], 80.00th=[ 1620], 90.00th=[ 1620], 95.00th=[ 1620], | 99.00th=[ 1620], 99.50th=[ 1620], 99.90th=[ 1620], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=58.26%, ctx=432, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4179: Wed Dec 25 01:03:08 2024 read: IOPS=0, BW=1582KiB/s (1620kB/s)(4096KiB/2589msec) clat (nsec): min=116213k, max=116213k, avg=116212631.00, stdev= 0.00 lat (nsec): min=116214k, max=116214k, avg=116213851.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 116], 10.00th=[ 116], 20.00th=[ 116], | 30.00th=[ 116], 40.00th=[ 116], 50.00th=[ 116], 60.00th=[ 116], | 70.00th=[ 116], 80.00th=[ 116], 90.00th=[ 116], 95.00th=[ 116], | 99.00th=[ 116], 99.50th=[ 116], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3164KiB/s (3240kB/s)(8192KiB/2589msec) clat (msec): min=242, max=2213, avg=1228.02, stdev=1394.00 lat (msec): min=242, max=2214, avg=1228.29, stdev=1394.01 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 243], 10.00th=[ 243], 20.00th=[ 243], | 30.00th=[ 243], 40.00th=[ 243], 50.00th=[ 243], 60.00th=[ 2198], | 70.00th=[ 2198], 80.00th=[ 2198], 90.00th=[ 2198], 95.00th=[ 2198], | 99.00th=[ 2198], 99.50th=[ 2198], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 8192, max= 8192, per=64.73%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=9.70%, ctx=333, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4181: Wed Dec 25 01:03:08 2024 write: IOPS=1, BW=4949KiB/s (5068kB/s)(12.0MiB/2483msec) clat (msec): min=216, max=1860, avg=825.25, stdev=900.87 lat (msec): min=216, max=1860, avg=825.50, stdev=900.78 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 218], 10.00th=[ 218], 20.00th=[ 218], | 30.00th=[ 218], 40.00th=[ 401], 50.00th=[ 401], 60.00th=[ 401], | 70.00th=[ 1854], 80.00th=[ 1854], 90.00th=[ 1854], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 8192, max= 8192, per=64.73%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=12.37%, ctx=47, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4183: Wed Dec 25 01:03:08 2024 write: IOPS=1, BW=4991KiB/s (5111kB/s)(12.0MiB/2462msec) clat (msec): min=234, max=1880, avg=810.70, stdev=927.56 lat (msec): min=234, max=1880, avg=810.98, stdev=927.57 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 234], 10.00th=[ 234], 20.00th=[ 234], | 30.00th=[ 234], 40.00th=[ 317], 50.00th=[ 317], 60.00th=[ 317], | 70.00th=[ 1888], 80.00th=[ 1888], 90.00th=[ 1888], 95.00th=[ 1888], | 99.00th=[ 1888], 99.50th=[ 1888], 99.90th=[ 1888], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 8192, max= 8192, per=64.73%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=10.52%, ctx=75, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6328KiB/s (6480kB/s), 1582KiB/s-6298KiB/s (1620kB/s-6449kB/s), io=16.0MiB (16.8MB), run=1951-2589msec WRITE: bw=12.4MiB/s (12.0MB/s), 3164KiB/s-4991KiB/s (3240kB/s-5111kB/s), io=32.0MiB (33.6MB), run=2462-2589msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4178: Wed Dec 25 01:03:08 2024 read: IOPS=1, BW=4849KiB/s (4966kB/s)(12.0MiB/2534msec) clat (msec): min=104, max=2132, avg=829.14, stdev=1131.07 lat (msec): min=104, max=2132, avg=829.14, stdev=1131.07 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 105], 10.00th=[ 105], 20.00th=[ 105], | 30.00th=[ 105], 40.00th=[ 251], 50.00th=[ 251], 60.00th=[ 251], | 70.00th=[ 2140], 80.00th=[ 2140], 90.00th=[ 2140], 95.00th=[ 2140], | 99.00th=[ 2140], 99.50th=[ 2140], 99.90th=[ 2140], 99.95th=[ 2140], | 99.99th=[ 2140] bw ( KiB/s): min= 8000, max= 8015, per=100.00%, avg=8007.50, stdev=10.61, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=2.65%, ctx=23, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4180: Wed Dec 25 01:03:08 2024 read: IOPS=0, BW=1536KiB/s (1573kB/s)(4096KiB/2667msec) clat (nsec): min=255671k, max=255671k, avg=255670520.00, stdev= 0.00 lat (nsec): min=255672k, max=255672k, avg=255672161.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 255], 10.00th=[ 255], 20.00th=[ 255], | 30.00th=[ 255], 40.00th=[ 255], 50.00th=[ 255], 60.00th=[ 255], | 70.00th=[ 255], 80.00th=[ 255], 90.00th=[ 255], 95.00th=[ 255], | 99.00th=[ 255], 99.50th=[ 255], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3072KiB/s (3145kB/s)(8192KiB/2667msec) clat (msec): min=185, max=2215, avg=1200.65, stdev=1435.88 lat (msec): min=185, max=2216, avg=1200.90, stdev=1435.88 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 186], 10.00th=[ 186], 20.00th=[ 186], | 30.00th=[ 186], 40.00th=[ 186], 50.00th=[ 186], 60.00th=[ 2232], | 70.00th=[ 2232], 80.00th=[ 2232], 90.00th=[ 2232], 95.00th=[ 2232], | 99.00th=[ 2232], 99.50th=[ 2232], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 8192, max= 8192, per=66.68%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=4.05%, ctx=98, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4182: Wed Dec 25 01:03:08 2024 write: IOPS=1, BW=4763KiB/s (4877kB/s)(12.0MiB/2580msec) clat (msec): min=89, max=2140, avg=841.56, stdev=1129.98 lat (msec): min=89, max=2141, avg=841.86, stdev=1129.96 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 90], 10.00th=[ 90], 20.00th=[ 90], | 30.00th=[ 90], 40.00th=[ 296], 50.00th=[ 296], 60.00th=[ 296], | 70.00th=[ 2140], 80.00th=[ 2140], 90.00th=[ 2140], 95.00th=[ 2140], | 99.00th=[ 2140], 99.50th=[ 2140], 99.90th=[ 2140], 99.95th=[ 2140], | 99.99th=[ 2140] bw ( KiB/s): min= 7922, max= 8192, per=65.58%, avg=8057.00, stdev=190.92, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=2.52%, ctx=19, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4184: Wed Dec 25 01:03:08 2024 write: IOPS=7, BW=30.3MiB/s (31.8MB/s)(12.0MiB/396msec) clat (msec): min=71, max=196, avg=123.38, stdev=65.26 lat (msec): min=71, max=197, avg=123.59, stdev=65.39 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 72], 10.00th=[ 72], 20.00th=[ 72], | 30.00th=[ 72], 40.00th=[ 102], 50.00th=[ 102], 60.00th=[ 102], | 70.00th=[ 197], 80.00th=[ 197], 90.00th=[ 197], 95.00th=[ 197], | 99.00th=[ 197], 99.50th=[ 197], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=17.72%, ctx=56, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6143KiB/s (6291kB/s), 1536KiB/s-4849KiB/s (1573kB/s-4966kB/s), io=16.0MiB (16.8MB), run=2534-2667msec WRITE: bw=11.0MiB/s (12.6MB/s), 3072KiB/s-30.3MiB/s (3145kB/s-31.8MB/s), io=32.0MiB (33.6MB), run=396-2667msec