== sanity test 398b: DIO and buffer IO race ============== 03:51:11 (1732438271) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.13124 s, 44.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29560: Sun Nov 24 03:51:25 2024 read: IOPS=123, BW=495KiB/s (507kB/s)(5992KiB/12103msec) clat (usec): min=48, max=193998, avg=3807.48, stdev=5771.09 lat (usec): min=48, max=193999, avg=3808.69, stdev=5770.98 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 62], 10.00th=[ 74], 20.00th=[ 1958], | 30.00th=[ 2376], 40.00th=[ 2737], 50.00th=[ 3064], 60.00th=[ 3654], | 70.00th=[ 4621], 80.00th=[ 5669], 90.00th=[ 7242], 95.00th=[ 8455], | 99.00th=[ 13042], 99.50th=[ 18482], 99.90th=[ 41681], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 240, max= 816, per=24.88%, avg=493.33, stdev=118.77, samples=24 iops : min= 60, max= 204, avg=123.33, stdev=29.69, samples=24 write: IOPS=130, BW=520KiB/s (533kB/s)(6296KiB/12103msec) clat (usec): min=476, max=26989, avg=4001.08, stdev=3787.93 lat (usec): min=477, max=26990, avg=4001.88, stdev=3787.99 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 644], | 30.00th=[ 938], 40.00th=[ 2212], 50.00th=[ 3228], 60.00th=[ 4015], | 70.00th=[ 5014], 80.00th=[ 6521], 90.00th=[ 8979], 95.00th=[11076], | 99.00th=[16909], 99.50th=[22938], 99.90th=[25297], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 296, max= 792, per=24.83%, avg=515.67, stdev=105.05, samples=24 iops : min= 74, max= 198, avg=128.92, stdev=26.26, samples=24 lat (usec) : 50=0.23%, 100=7.26%, 250=1.20%, 500=0.65%, 750=12.11% lat (usec) : 1000=3.39% lat (msec) : 2=5.14%, 4=31.97%, 10=33.14%, 20=4.46%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=24.25%, ctx=5593, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29562: Sun Nov 24 03:51:25 2024 read: IOPS=124, BW=497KiB/s (508kB/s)(5940KiB/11963msec) clat (usec): min=47, max=48307, avg=3687.89, stdev=3169.77 lat (usec): min=47, max=48307, avg=3688.66, stdev=3169.72 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 62], 10.00th=[ 74], 20.00th=[ 2040], | 30.00th=[ 2442], 40.00th=[ 2769], 50.00th=[ 3130], 60.00th=[ 3687], | 70.00th=[ 4490], 80.00th=[ 5473], 90.00th=[ 7046], 95.00th=[ 8455], | 99.00th=[11863], 99.50th=[14484], 99.90th=[40109], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 231, max= 664, per=24.80%, avg=491.70, stdev=95.90, samples=23 iops : min= 57, max= 166, avg=122.87, stdev=24.05, samples=23 write: IOPS=132, BW=531KiB/s (543kB/s)(6348KiB/11963msec) clat (usec): min=435, max=194107, avg=4025.76, stdev=6143.84 lat (usec): min=436, max=194108, avg=4026.61, stdev=6143.81 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 635], | 30.00th=[ 889], 40.00th=[ 1795], 50.00th=[ 3163], 60.00th=[ 4047], | 70.00th=[ 5080], 80.00th=[ 6587], 90.00th=[ 8717], 95.00th=[ 10421], | 99.00th=[ 16909], 99.50th=[ 22152], 99.90th=[ 41157], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 287, max= 768, per=25.01%, avg=519.52, stdev=118.21, samples=23 iops : min= 71, max= 192, avg=129.83, stdev=29.62, samples=23 lat (usec) : 50=0.13%, 100=7.10%, 250=0.78%, 500=0.88%, 750=12.70% lat (usec) : 1000=3.42% lat (msec) : 2=5.66%, 4=31.18%, 10=33.98%, 20=3.61%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.12%, sys=24.04%, ctx=5551, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29564: Sun Nov 24 03:51:25 2024 read: IOPS=124, BW=500KiB/s (512kB/s)(6028KiB/12062msec) clat (usec): min=47, max=171482, avg=3677.74, stdev=5230.93 lat (usec): min=47, max=171482, avg=3678.51, stdev=5230.94 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 61], 10.00th=[ 72], 20.00th=[ 1876], | 30.00th=[ 2343], 40.00th=[ 2671], 50.00th=[ 2999], 60.00th=[ 3556], | 70.00th=[ 4293], 80.00th=[ 5538], 90.00th=[ 7111], 95.00th=[ 8455], | 99.00th=[ 13435], 99.50th=[ 17433], 99.90th=[ 33817], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 264, max= 648, per=24.45%, avg=484.78, stdev=98.28, samples=23 iops : min= 66, max= 162, avg=121.13, stdev=24.58, samples=23 write: IOPS=129, BW=519KiB/s (531kB/s)(6260KiB/12062msec) clat (usec): min=458, max=42633, avg=4105.37, stdev=3887.45 lat (usec): min=459, max=42634, avg=4106.25, stdev=3887.38 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 652], | 30.00th=[ 988], 40.00th=[ 2245], 50.00th=[ 3392], 60.00th=[ 4293], | 70.00th=[ 5342], 80.00th=[ 6849], 90.00th=[ 8717], 95.00th=[11076], | 99.00th=[15926], 99.50th=[20317], 99.90th=[34341], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 328, max= 688, per=24.19%, avg=502.48, stdev=95.56, samples=23 iops : min= 82, max= 172, avg=125.57, stdev=23.91, samples=23 lat (usec) : 50=0.36%, 100=7.68%, 250=1.20%, 500=0.75%, 750=11.49% lat (usec) : 1000=3.35% lat (msec) : 2=5.37%, 4=31.15%, 10=33.92%, 20=4.30%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=23.82%, ctx=5718, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29567: Sun Nov 24 03:51:25 2024 read: IOPS=126, BW=505KiB/s (517kB/s)(6052KiB/11996msec) clat (usec): min=46, max=58445, avg=3532.62, stdev=2964.72 lat (usec): min=47, max=58445, avg=3533.25, stdev=2964.73 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 61], 10.00th=[ 73], 20.00th=[ 1942], | 30.00th=[ 2376], 40.00th=[ 2737], 50.00th=[ 3097], 60.00th=[ 3556], | 70.00th=[ 4293], 80.00th=[ 5407], 90.00th=[ 6980], 95.00th=[ 8029], | 99.00th=[11076], 99.50th=[14484], 99.90th=[32375], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 176, max= 672, per=24.89%, avg=493.65, stdev=114.32, samples=23 iops : min= 44, max= 168, avg=123.30, stdev=28.57, samples=23 write: IOPS=129, BW=520KiB/s (532kB/s)(6236KiB/11996msec) clat (usec): min=439, max=196556, avg=4203.59, stdev=6617.83 lat (usec): min=440, max=196558, avg=4204.35, stdev=6617.84 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 635], | 30.00th=[ 889], 40.00th=[ 1942], 50.00th=[ 3195], 60.00th=[ 4113], | 70.00th=[ 5211], 80.00th=[ 6587], 90.00th=[ 8979], 95.00th=[ 11338], | 99.00th=[ 17695], 99.50th=[ 28967], 99.90th=[ 69731], 99.95th=[196084], | 99.99th=[196084] bw ( KiB/s): min= 200, max= 680, per=24.40%, avg=506.87, stdev=105.94, samples=23 iops : min= 50, max= 170, avg=126.61, stdev=26.51, samples=23 lat (usec) : 50=0.55%, 100=6.61%, 250=1.46%, 500=1.01%, 750=12.43% lat (usec) : 1000=3.12% lat (msec) : 2=5.83%, 4=31.41%, 10=33.11%, 20=4.00%, 50=0.36% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.18%, sys=23.93%, ctx=5618, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1984KiB/s (2032kB/s), 495KiB/s-505KiB/s (507kB/s-517kB/s), io=23.4MiB (24.6MB), run=11963-12103msec WRITE: bw=2077KiB/s (2127kB/s), 519KiB/s-531KiB/s (531kB/s-543kB/s), io=24.6MiB (25.7MB), run=11963-12103msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29561: Sun Nov 24 03:51:45 2024 read: IOPS=46, BW=186KiB/s (191kB/s)(5992KiB/32151msec) clat (usec): min=1549, max=195082, avg=3545.73, stdev=5975.34 lat (usec): min=1550, max=195083, avg=3546.48, stdev=5975.35 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1762], 10.00th=[ 1827], 20.00th=[ 1909], | 30.00th=[ 2008], 40.00th=[ 2114], 50.00th=[ 2212], 60.00th=[ 2409], | 70.00th=[ 2704], 80.00th=[ 3720], 90.00th=[ 6783], 95.00th=[ 9634], | 99.00th=[ 16450], 99.50th=[ 25560], 99.90th=[ 41681], 99.95th=[196084], | 99.99th=[196084] bw ( KiB/s): min= 56, max= 352, per=24.90%, avg=185.73, stdev=75.32, samples=64 iops : min= 14, max= 88, avg=46.33, stdev=18.77, samples=64 write: IOPS=48, BW=196KiB/s (201kB/s)(6296KiB/32151msec) clat (usec): min=8400, max=67362, avg=16987.20, stdev=6673.33 lat (usec): min=8401, max=67363, avg=16988.10, stdev=6673.35 clat percentiles (usec): | 1.00th=[ 9372], 5.00th=[10159], 10.00th=[10683], 20.00th=[11600], | 30.00th=[12780], 40.00th=[14222], 50.00th=[15533], 60.00th=[16712], | 70.00th=[18220], 80.00th=[20579], 90.00th=[26346], 95.00th=[30802], | 99.00th=[40633], 99.50th=[43779], 99.90th=[61080], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 48, max= 272, per=25.00%, avg=195.23, stdev=61.72, samples=64 iops : min= 12, max= 68, avg=48.70, stdev=15.39, samples=64 lat (msec) : 2=14.36%, 4=25.16%, 10=9.31%, 20=39.78%, 50=11.30% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.12%, sys=5.97%, ctx=3498, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29563: Sun Nov 24 03:51:45 2024 read: IOPS=46, BW=185KiB/s (189kB/s)(5940KiB/32113msec) clat (usec): min=1559, max=169264, avg=3355.34, stdev=5189.26 lat (usec): min=1560, max=169265, avg=3356.06, stdev=5189.26 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1762], 10.00th=[ 1827], 20.00th=[ 1909], | 30.00th=[ 1991], 40.00th=[ 2089], 50.00th=[ 2212], 60.00th=[ 2343], | 70.00th=[ 2606], 80.00th=[ 3556], 90.00th=[ 6390], 95.00th=[ 8848], | 99.00th=[ 15139], 99.50th=[ 17433], 99.90th=[ 38536], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 16, max= 368, per=24.82%, avg=185.19, stdev=79.89, samples=64 iops : min= 4, max= 92, avg=46.25, stdev=19.95, samples=64 write: IOPS=49, BW=198KiB/s (202kB/s)(6348KiB/32113msec) clat (usec): min=8065, max=75141, avg=17030.61, stdev=6856.34 lat (usec): min=8065, max=75142, avg=17031.56, stdev=6856.30 clat percentiles (usec): | 1.00th=[ 9372], 5.00th=[10028], 10.00th=[10552], 20.00th=[11469], | 30.00th=[12780], 40.00th=[14222], 50.00th=[15664], 60.00th=[16712], | 70.00th=[17957], 80.00th=[20579], 90.00th=[26346], 95.00th=[31065], | 99.00th=[39060], 99.50th=[46400], 99.90th=[63701], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 64, max= 272, per=25.25%, avg=197.19, stdev=61.78, samples=64 iops : min= 16, max= 68, avg=49.25, stdev=15.44, samples=64 lat (msec) : 2=15.17%, 4=24.41%, 10=9.24%, 20=39.97%, 50=10.97% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.11%, sys=5.86%, ctx=3497, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29565: Sun Nov 24 03:51:45 2024 read: IOPS=47, BW=188KiB/s (193kB/s)(6028KiB/31982msec) clat (usec): min=1527, max=34139, avg=3401.06, stdev=3011.30 lat (usec): min=1528, max=34140, avg=3401.88, stdev=3011.32 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1778], 10.00th=[ 1827], 20.00th=[ 1909], | 30.00th=[ 2008], 40.00th=[ 2114], 50.00th=[ 2212], 60.00th=[ 2409], | 70.00th=[ 2802], 80.00th=[ 4293], 90.00th=[ 6718], 95.00th=[ 8848], | 99.00th=[15533], 99.50th=[17433], 99.90th=[32900], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 39, max= 352, per=25.28%, avg=188.56, stdev=67.14, samples=63 iops : min= 9, max= 88, avg=47.03, stdev=16.76, samples=63 write: IOPS=48, BW=196KiB/s (200kB/s)(6260KiB/31982msec) clat (msec): min=8, max=204, avg=17.10, stdev= 8.15 lat (msec): min=8, max=204, avg=17.10, stdev= 8.15 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 16], 60.00th=[ 17], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 26], 95.00th=[ 31], | 99.00th=[ 41], 99.50th=[ 48], 99.90th=[ 77], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 64, max= 272, per=24.97%, avg=195.03, stdev=61.99, samples=63 iops : min= 16, max= 68, avg=48.65, stdev=15.47, samples=63 lat (msec) : 2=14.26%, 4=24.48%, 10=10.81%, 20=39.36%, 50=10.87% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.11%, sys=5.93%, ctx=3467, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29566: Sun Nov 24 03:51:45 2024 read: IOPS=47, BW=190KiB/s (194kB/s)(6052KiB/31867msec) clat (usec): min=1567, max=53022, avg=3404.12, stdev=3649.94 lat (usec): min=1568, max=53022, avg=3404.90, stdev=3649.95 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1778], 10.00th=[ 1827], 20.00th=[ 1926], | 30.00th=[ 2008], 40.00th=[ 2114], 50.00th=[ 2212], 60.00th=[ 2376], | 70.00th=[ 2638], 80.00th=[ 3556], 90.00th=[ 6587], 95.00th=[ 9110], | 99.00th=[16712], 99.50th=[27657], 99.90th=[49021], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 48, max= 376, per=25.31%, avg=188.84, stdev=75.04, samples=63 iops : min= 12, max= 94, avg=47.13, stdev=18.79, samples=63 write: IOPS=48, BW=196KiB/s (200kB/s)(6236KiB/31867msec) clat (msec): min=8, max=189, avg=17.07, stdev= 7.82 lat (msec): min=8, max=189, avg=17.07, stdev= 7.82 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 16], 60.00th=[ 17], | 70.00th=[ 18], 80.00th=[ 21], 90.00th=[ 27], 95.00th=[ 31], | 99.00th=[ 40], 99.50th=[ 43], 99.90th=[ 61], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 56, max= 264, per=24.99%, avg=195.19, stdev=62.99, samples=63 iops : min= 14, max= 66, avg=48.71, stdev=15.79, samples=63 lat (msec) : 2=14.06%, 4=26.17%, 10=8.98%, 20=39.42%, 50=11.20% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.10%, sys=5.95%, ctx=3458, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=747KiB/s (765kB/s), 185KiB/s-190KiB/s (189kB/s-194kB/s), io=23.4MiB (24.6MB), run=31867-32151msec WRITE: bw=782KiB/s (801kB/s), 196KiB/s-198KiB/s (200kB/s-202kB/s), io=24.6MiB (25.7MB), run=31867-32151msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29602: Sun Nov 24 03:51:51 2024 read: IOPS=79, BW=1279KiB/s (1310kB/s)(5920KiB/4629msec) clat (usec): min=54, max=106417, avg=9349.59, stdev=7157.89 lat (usec): min=54, max=106419, avg=9350.38, stdev=7157.74 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 63], 10.00th=[ 80], 20.00th=[ 6915], | 30.00th=[ 8586], 40.00th=[ 9503], 50.00th=[ 10028], 60.00th=[ 10683], | 70.00th=[ 11338], 80.00th=[ 12518], 90.00th=[ 13829], 95.00th=[ 15533], | 99.00th=[ 22152], 99.50th=[ 36439], 99.90th=[106431], 99.95th=[106431], | 99.99th=[106431] bw ( KiB/s): min= 800, max= 1536, per=24.96%, avg=1276.44, stdev=225.39, samples=9 iops : min= 50, max= 96, avg=79.78, stdev=14.09, samples=9 write: IOPS=85, BW=1376KiB/s (1409kB/s)(6368KiB/4629msec) clat (usec): min=609, max=21912, avg=2902.27, stdev=3202.01 lat (usec): min=609, max=21912, avg=2903.49, stdev=3202.08 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 734], 10.00th=[ 766], 20.00th=[ 807], | 30.00th=[ 914], 40.00th=[ 1090], 50.00th=[ 1500], 60.00th=[ 2540], | 70.00th=[ 3294], 80.00th=[ 4293], 90.00th=[ 6194], 95.00th=[ 8848], | 99.00th=[17957], 99.50th=[20317], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 928, max= 1888, per=25.77%, avg=1386.67, stdev=286.22, samples=9 iops : min= 58, max= 118, avg=86.67, stdev=17.89, samples=9 lat (usec) : 100=7.42%, 250=1.04%, 500=0.39%, 750=3.78%, 1000=15.76% lat (msec) : 2=7.94%, 4=12.76%, 10=25.13%, 20=24.87%, 50=0.78% lat (msec) : 250=0.13% cpu : usr=0.15%, sys=23.88%, ctx=2066, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29603: Sun Nov 24 03:51:51 2024 read: IOPS=79, BW=1265KiB/s (1295kB/s)(5744KiB/4542msec) clat (usec): min=52, max=21462, avg=8757.80, stdev=4824.95 lat (usec): min=52, max=21462, avg=8758.49, stdev=4824.98 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 64], 10.00th=[ 76], 20.00th=[ 6259], | 30.00th=[ 8455], 40.00th=[ 9110], 50.00th=[ 9765], 60.00th=[10421], | 70.00th=[11207], 80.00th=[12125], 90.00th=[13698], 95.00th=[14877], | 99.00th=[19268], 99.50th=[19530], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 576, max= 1724, per=24.74%, avg=1265.33, stdev=315.29, samples=9 iops : min= 36, max= 107, avg=79.00, stdev=19.57, samples=9 write: IOPS=90, BW=1441KiB/s (1475kB/s)(6544KiB/4542msec) clat (usec): min=604, max=170379, avg=3353.12, stdev=8844.42 lat (usec): min=605, max=170381, avg=3354.20, stdev=8844.44 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 725], 10.00th=[ 766], 20.00th=[ 824], | 30.00th=[ 881], 40.00th=[ 1045], 50.00th=[ 1958], 60.00th=[ 2737], | 70.00th=[ 3294], 80.00th=[ 4752], 90.00th=[ 6456], 95.00th=[ 8029], | 99.00th=[ 16450], 99.50th=[ 18744], 99.90th=[170918], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 864, max= 2080, per=26.89%, avg=1446.78, stdev=375.57, samples=9 iops : min= 54, max= 130, avg=90.33, stdev=23.49, samples=9 lat (usec) : 100=6.90%, 250=2.08%, 750=4.04%, 1000=16.67% lat (msec) : 2=6.25%, 4=13.15%, 10=26.69%, 20=23.83%, 50=0.26% lat (msec) : 250=0.13% cpu : usr=0.11%, sys=21.76%, ctx=2046, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29604: Sun Nov 24 03:51:51 2024 read: IOPS=83, BW=1336KiB/s (1368kB/s)(6256KiB/4683msec) clat (usec): min=51, max=21440, avg=8724.87, stdev=4878.07 lat (usec): min=52, max=21441, avg=8725.49, stdev=4878.07 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 63], 10.00th=[ 77], 20.00th=[ 5997], | 30.00th=[ 8291], 40.00th=[ 8979], 50.00th=[ 9896], 60.00th=[10552], | 70.00th=[11338], 80.00th=[12387], 90.00th=[13829], 95.00th=[14877], | 99.00th=[17957], 99.50th=[21103], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 704, max= 1888, per=25.72%, avg=1315.56, stdev=314.12, samples=9 iops : min= 44, max= 118, avg=82.22, stdev=19.63, samples=9 write: IOPS=80, BW=1288KiB/s (1319kB/s)(6032KiB/4683msec) clat (usec): min=636, max=73214, avg=3203.61, stdev=5715.95 lat (usec): min=637, max=73217, avg=3204.74, stdev=5716.03 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 734], 10.00th=[ 775], 20.00th=[ 816], | 30.00th=[ 922], 40.00th=[ 1221], 50.00th=[ 1827], 60.00th=[ 2540], | 70.00th=[ 3097], 80.00th=[ 4424], 90.00th=[ 6587], 95.00th=[ 8586], | 99.00th=[18482], 99.50th=[66847], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 768, max= 1824, per=22.99%, avg=1237.33, stdev=332.17, samples=9 iops : min= 48, max= 114, avg=77.33, stdev=20.76, samples=9 lat (usec) : 100=8.46%, 250=1.69%, 750=3.12%, 1000=13.54% lat (msec) : 2=8.33%, 4=12.89%, 10=25.78%, 20=25.65%, 50=0.26% lat (msec) : 100=0.26% cpu : usr=0.15%, sys=21.57%, ctx=2087, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29605: Sun Nov 24 03:51:51 2024 read: IOPS=82, BW=1318KiB/s (1350kB/s)(6032KiB/4577msec) clat (usec): min=50, max=34959, avg=8598.30, stdev=5213.83 lat (usec): min=51, max=34959, avg=8598.95, stdev=5213.86 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 63], 10.00th=[ 72], 20.00th=[ 122], | 30.00th=[ 8029], 40.00th=[ 8717], 50.00th=[ 9765], 60.00th=[10421], | 70.00th=[11338], 80.00th=[12387], 90.00th=[13566], 95.00th=[15270], | 99.00th=[23200], 99.50th=[25822], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 736, max= 1856, per=25.37%, avg=1297.44, stdev=285.89, samples=9 iops : min= 46, max= 116, avg=81.00, stdev=17.89, samples=9 write: IOPS=85, BW=1367KiB/s (1400kB/s)(6256KiB/4577msec) clat (usec): min=653, max=67799, avg=3244.41, stdev=5385.63 lat (usec): min=653, max=67800, avg=3245.56, stdev=5385.70 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 725], 10.00th=[ 758], 20.00th=[ 807], | 30.00th=[ 898], 40.00th=[ 1090], 50.00th=[ 1860], 60.00th=[ 2704], | 70.00th=[ 3458], 80.00th=[ 4752], 90.00th=[ 6587], 95.00th=[ 8848], | 99.00th=[14353], 99.50th=[66847], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 800, max= 1920, per=25.17%, avg=1354.33, stdev=328.23, samples=9 iops : min= 50, max= 120, avg=84.56, stdev=20.50, samples=9 lat (usec) : 100=9.11%, 250=1.43%, 750=4.30%, 1000=14.19% lat (msec) : 2=7.29%, 4=11.72%, 10=28.12%, 20=23.05%, 50=0.52% lat (msec) : 100=0.26% cpu : usr=0.13%, sys=21.68%, ctx=2088, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5115KiB/s (5237kB/s), 1265KiB/s-1336KiB/s (1295kB/s-1368kB/s), io=23.4MiB (24.5MB), run=4542-4683msec WRITE: bw=5381KiB/s (5510kB/s), 1288KiB/s-1441KiB/s (1319kB/s-1475kB/s), io=24.6MiB (25.8MB), run=4542-4683msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29606: Sun Nov 24 03:51:55 2024 read: IOPS=43, BW=689KiB/s (705kB/s)(5920KiB/8597msec) clat (usec): min=1788, max=54466, avg=4098.94, stdev=4741.14 lat (usec): min=1788, max=54468, avg=4099.69, stdev=4741.17 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 1975], 10.00th=[ 2040], 20.00th=[ 2180], | 30.00th=[ 2245], 40.00th=[ 2376], 50.00th=[ 2540], 60.00th=[ 2802], | 70.00th=[ 3392], 80.00th=[ 4490], 90.00th=[ 7570], 95.00th=[11731], | 99.00th=[26870], 99.50th=[36963], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 384, max= 1120, per=25.29%, avg=679.47, stdev=207.31, samples=17 iops : min= 24, max= 70, avg=42.41, stdev=13.02, samples=17 write: IOPS=46, BW=741KiB/s (759kB/s)(6368KiB/8597msec) clat (usec): min=9816, max=48701, avg=17630.45, stdev=5558.23 lat (usec): min=9818, max=48702, avg=17632.05, stdev=5558.26 clat percentiles (usec): | 1.00th=[10290], 5.00th=[10945], 10.00th=[11600], 20.00th=[12518], | 30.00th=[14091], 40.00th=[15270], 50.00th=[16712], 60.00th=[17957], | 70.00th=[19792], 80.00th=[21890], 90.00th=[24249], 95.00th=[27132], | 99.00th=[35390], 99.50th=[41157], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 320, max= 992, per=26.10%, avg=737.76, stdev=204.42, samples=17 iops : min= 20, max= 62, avg=46.06, stdev=12.78, samples=17 lat (msec) : 2=2.86%, 4=33.98%, 10=8.85%, 20=39.71%, 50=14.45% lat (msec) : 100=0.13% cpu : usr=0.21%, sys=6.24%, ctx=874, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29607: Sun Nov 24 03:51:55 2024 read: IOPS=40, BW=644KiB/s (660kB/s)(5744KiB/8914msec) clat (usec): min=1751, max=28724, avg=3961.69, stdev=3565.77 lat (usec): min=1751, max=28725, avg=3962.41, stdev=3565.76 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 1975], 10.00th=[ 2057], 20.00th=[ 2147], | 30.00th=[ 2278], 40.00th=[ 2376], 50.00th=[ 2540], 60.00th=[ 2704], | 70.00th=[ 3064], 80.00th=[ 4817], 90.00th=[ 8586], 95.00th=[11600], | 99.00th=[16319], 99.50th=[26870], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 192, max= 1152, per=23.47%, avg=630.53, stdev=282.84, samples=17 iops : min= 12, max= 72, avg=39.35, stdev=17.72, samples=17 write: IOPS=45, BW=734KiB/s (752kB/s)(6544KiB/8914msec) clat (msec): min=9, max=199, avg=18.25, stdev=10.51 lat (msec): min=9, max=199, avg=18.25, stdev=10.51 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 21], 80.00th=[ 22], 90.00th=[ 26], 95.00th=[ 28], | 99.00th=[ 36], 99.50th=[ 42], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 384, max= 960, per=25.16%, avg=711.41, stdev=172.57, samples=17 iops : min= 24, max= 60, avg=44.41, stdev=10.81, samples=17 lat (msec) : 2=2.60%, 4=33.59%, 10=8.46%, 20=39.19%, 50=16.02% lat (msec) : 250=0.13% cpu : usr=0.10%, sys=6.28%, ctx=884, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29608: Sun Nov 24 03:51:55 2024 read: IOPS=46, BW=740KiB/s (758kB/s)(6256KiB/8456msec) clat (usec): min=1700, max=24899, avg=3756.08, stdev=3054.16 lat (usec): min=1701, max=24900, avg=3756.97, stdev=3054.09 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 1958], 10.00th=[ 2057], 20.00th=[ 2180], | 30.00th=[ 2278], 40.00th=[ 2409], 50.00th=[ 2573], 60.00th=[ 2769], | 70.00th=[ 3228], 80.00th=[ 4490], 90.00th=[ 6849], 95.00th=[10290], | 99.00th=[16319], 99.50th=[21103], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 384, max= 1408, per=27.61%, avg=741.94, stdev=303.92, samples=16 iops : min= 24, max= 88, avg=46.31, stdev=19.06, samples=16 write: IOPS=44, BW=713KiB/s (730kB/s)(6032KiB/8456msec) clat (msec): min=10, max=114, avg=18.35, stdev= 7.61 lat (msec): min=10, max=114, avg=18.36, stdev= 7.61 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 20], 80.00th=[ 23], 90.00th=[ 27], 95.00th=[ 30], | 99.00th=[ 39], 99.50th=[ 53], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 352, max= 960, per=24.62%, avg=695.88, stdev=177.85, samples=16 iops : min= 22, max= 60, avg=43.44, stdev=11.15, samples=16 lat (msec) : 2=3.65%, 4=35.55%, 10=8.98%, 20=38.02%, 50=13.54% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.09%, sys=6.69%, ctx=894, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29609: Sun Nov 24 03:51:55 2024 read: IOPS=43, BW=689KiB/s (705kB/s)(6032KiB/8756msec) clat (usec): min=1807, max=33795, avg=4025.73, stdev=3903.07 lat (usec): min=1808, max=33796, avg=4026.67, stdev=3902.97 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 1991], 10.00th=[ 2057], 20.00th=[ 2180], | 30.00th=[ 2311], 40.00th=[ 2409], 50.00th=[ 2540], 60.00th=[ 2737], | 70.00th=[ 3261], 80.00th=[ 4490], 90.00th=[ 8586], 95.00th=[12125], | 99.00th=[19792], 99.50th=[31589], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 320, max= 1088, per=25.07%, avg=673.65, stdev=208.38, samples=17 iops : min= 20, max= 68, avg=42.00, stdev=13.00, samples=17 write: IOPS=44, BW=714KiB/s (732kB/s)(6256KiB/8756msec) clat (msec): min=9, max=109, avg=18.33, stdev= 7.53 lat (msec): min=9, max=109, avg=18.33, stdev= 7.53 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 20], 80.00th=[ 23], 90.00th=[ 27], 95.00th=[ 30], | 99.00th=[ 40], 99.50th=[ 48], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 352, max= 960, per=24.89%, avg=703.76, stdev=188.91, samples=17 iops : min= 22, max= 60, avg=43.88, stdev=11.78, samples=17 lat (msec) : 2=2.99%, 4=35.16%, 10=7.29%, 20=39.71%, 50=14.71% lat (msec) : 250=0.13% cpu : usr=0.10%, sys=6.29%, ctx=893, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2687KiB/s (2751kB/s), 644KiB/s-740KiB/s (660kB/s-758kB/s), io=23.4MiB (24.5MB), run=8456-8914msec WRITE: bw=2827KiB/s (2895kB/s), 713KiB/s-741KiB/s (730kB/s-759kB/s), io=24.6MiB (25.8MB), run=8456-8914msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29622: Sun Nov 24 03:51:59 2024 read: IOPS=2, BW=2164KiB/s (2216kB/s)(7168KiB/3312msec) clat (msec): min=42, max=807, avg=376.40, stdev=371.60 lat (msec): min=42, max=807, avg=376.40, stdev=371.60 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 47], | 30.00th=[ 69], 40.00th=[ 69], 50.00th=[ 167], 60.00th=[ 726], | 70.00th=[ 726], 80.00th=[ 776], 90.00th=[ 810], 95.00th=[ 810], | 99.00th=[ 810], 99.50th=[ 810], 99.90th=[ 810], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 2048, max= 4096, per=41.40%, avg=2560.00, stdev=1024.00, samples=4 iops : min= 2, max= 4, avg= 2.50, stdev= 1.00, samples=4 write: IOPS=1, BW=1546KiB/s (1583kB/s)(5120KiB/3312msec) clat (msec): min=46, max=317, avg=129.27, stdev=108.22 lat (msec): min=46, max=317, avg=129.32, stdev=108.23 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 71], 40.00th=[ 71], 50.00th=[ 96], 60.00th=[ 96], | 70.00th=[ 116], 80.00th=[ 116], 90.00th=[ 317], 95.00th=[ 317], | 99.00th=[ 317], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 2048, max= 4096, per=35.49%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (msec) : 50=25.00%, 100=25.00%, 250=16.67%, 500=8.33%, 750=8.33% lat (msec) : 1000=16.67% cpu : usr=0.03%, sys=3.26%, ctx=88, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29624: Sun Nov 24 03:51:59 2024 read: IOPS=0, BW=630KiB/s (645kB/s)(2048KiB/3252msec) clat (usec): min=36877, max=77553, avg=57215.59, stdev=28762.58 lat (usec): min=36878, max=77568, avg=57223.62, stdev=28771.86 clat percentiles (usec): | 1.00th=[36963], 5.00th=[36963], 10.00th=[36963], 20.00th=[36963], | 30.00th=[36963], 40.00th=[36963], 50.00th=[36963], 60.00th=[77071], | 70.00th=[77071], 80.00th=[77071], 90.00th=[77071], 95.00th=[77071], | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 2048, max= 2048, per=33.12%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3149KiB/s (3224kB/s)(10.0MiB/3252msec) clat (msec): min=24, max=1633, avg=310.58, stdev=503.99 lat (msec): min=24, max=1633, avg=310.67, stdev=503.98 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 53], | 30.00th=[ 55], 40.00th=[ 58], 50.00th=[ 72], 60.00th=[ 113], | 70.00th=[ 218], 80.00th=[ 218], 90.00th=[ 684], 95.00th=[ 1636], | 99.00th=[ 1636], 99.50th=[ 1636], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 6144, max=10240, per=94.63%, avg=8192.00, stdev=2896.31, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (msec) : 50=16.67%, 100=41.67%, 250=25.00%, 750=8.33% cpu : usr=0.00%, sys=3.51%, ctx=59, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29626: Sun Nov 24 03:51:59 2024 read: IOPS=1, BW=1872KiB/s (1917kB/s)(6144KiB/3282msec) clat (msec): min=16, max=306, avg=86.32, stdev=115.21 lat (msec): min=16, max=306, avg=86.32, stdev=115.21 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 19], | 30.00th=[ 19], 40.00th=[ 22], 50.00th=[ 22], 60.00th=[ 33], | 70.00th=[ 123], 80.00th=[ 123], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 4096, max= 4096, per=66.25%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=1, BW=1872KiB/s (1917kB/s)(6144KiB/3282msec) clat (msec): min=31, max=1630, avg=459.65, stdev=635.10 lat (msec): min=31, max=1630, avg=459.71, stdev=635.13 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 54], | 30.00th=[ 54], 40.00th=[ 97], 50.00th=[ 97], 60.00th=[ 186], | 70.00th=[ 760], 80.00th=[ 760], 90.00th=[ 1636], 95.00th=[ 1636], | 99.00th=[ 1636], 99.50th=[ 1636], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 2048, max= 4096, per=39.43%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 lat (msec) : 20=16.67%, 50=25.00%, 100=16.67%, 250=16.67%, 500=8.33% lat (msec) : 1000=8.33% cpu : usr=0.06%, sys=2.35%, ctx=35, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29628: Sun Nov 24 03:51:59 2024 read: IOPS=1, BW=1618KiB/s (1657kB/s)(5120KiB/3164msec) clat (usec): min=14096, max=72453, avg=30012.24, stdev=24307.39 lat (usec): min=14097, max=72453, avg=30013.38, stdev=24307.11 clat percentiles (usec): | 1.00th=[14091], 5.00th=[14091], 10.00th=[14091], 20.00th=[14091], | 30.00th=[14615], 40.00th=[14615], 50.00th=[22152], 60.00th=[22152], | 70.00th=[26608], 80.00th=[26608], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 2048, max= 6144, per=66.25%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=2, BW=2265KiB/s (2320kB/s)(7168KiB/3164msec) clat (msec): min=38, max=1609, avg=422.68, stdev=577.26 lat (msec): min=38, max=1609, avg=422.75, stdev=577.25 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 63], | 30.00th=[ 79], 40.00th=[ 79], 50.00th=[ 120], 60.00th=[ 326], | 70.00th=[ 326], 80.00th=[ 726], 90.00th=[ 1603], 95.00th=[ 1603], | 99.00th=[ 1603], 99.50th=[ 1603], 99.90th=[ 1603], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min= 2048, max= 6144, per=35.49%, avg=3072.00, stdev=2048.00, samples=4 iops : min= 2, max= 6, avg= 3.00, stdev= 2.00, samples=4 lat (msec) : 20=16.67%, 50=25.00%, 100=25.00%, 250=8.33%, 500=8.33% lat (msec) : 750=8.33% cpu : usr=0.06%, sys=3.13%, ctx=52, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6184KiB/s (6332kB/s), 630KiB/s-2164KiB/s (645kB/s-2216kB/s), io=20.0MiB (20.0MB), run=3164-3312msec WRITE: bw=8657KiB/s (8865kB/s), 1546KiB/s-3149KiB/s (1583kB/s-3224kB/s), io=28.0MiB (29.4MB), run=3164-3312msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29623: Sun Nov 24 03:52:00 2024 read: IOPS=2, BW=2500KiB/s (2560kB/s)(7168KiB/2867msec) clat (usec): min=412, max=813058, avg=353650.21, stdev=409906.95 lat (usec): min=413, max=813062, avg=353651.89, stdev=409907.14 clat percentiles (usec): | 1.00th=[ 412], 5.00th=[ 412], 10.00th=[ 412], 20.00th=[ 449], | 30.00th=[ 45351], 40.00th=[ 45351], 50.00th=[ 56361], 60.00th=[775947], | 70.00th=[775947], 80.00th=[784335], 90.00th=[809501], 95.00th=[809501], | 99.00th=[809501], 99.50th=[809501], 99.90th=[809501], 99.95th=[809501], | 99.99th=[809501] bw ( KiB/s): min= 2048, max= 2048, per=43.36%, avg=2048.00, stdev= 0.00, samples=3 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=3 write: IOPS=1, BW=1786KiB/s (1829kB/s)(5120KiB/2867msec) clat (msec): min=24, max=129, avg=70.58, stdev=39.05 lat (msec): min=24, max=129, avg=70.63, stdev=39.06 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 24], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 61], 60.00th=[ 61], | 70.00th=[ 83], 80.00th=[ 83], 90.00th=[ 130], 95.00th=[ 130], | 99.00th=[ 130], 99.50th=[ 130], 99.90th=[ 130], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 2048, max= 2048, per=30.97%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 50=16.67%, 100=33.33%, 250=8.33%, 1000=25.00% cpu : usr=0.03%, sys=35.07%, ctx=709, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29625: Sun Nov 24 03:52:00 2024 read: IOPS=0, BW=625KiB/s (640kB/s)(2048KiB/3278msec) clat (usec): min=88653, max=92367, avg=90510.32, stdev=2626.02 lat (usec): min=88655, max=92368, avg=90511.93, stdev=2625.48 clat percentiles (usec): | 1.00th=[88605], 5.00th=[88605], 10.00th=[88605], 20.00th=[88605], | 30.00th=[88605], 40.00th=[88605], 50.00th=[88605], 60.00th=[92799], | 70.00th=[92799], 80.00th=[92799], 90.00th=[92799], 95.00th=[92799], | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 2048, max= 2048, per=43.36%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3124KiB/s (3199kB/s)(10.0MiB/3278msec) clat (msec): min=25, max=1641, avg=305.17, stdev=504.60 lat (msec): min=25, max=1641, avg=305.23, stdev=504.59 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 49], | 30.00th=[ 53], 40.00th=[ 54], 50.00th=[ 66], 60.00th=[ 122], | 70.00th=[ 203], 80.00th=[ 203], 90.00th=[ 651], 95.00th=[ 1636], | 99.00th=[ 1636], 99.50th=[ 1636], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 6144, max=10219, per=100.00%, avg=8181.50, stdev=2881.46, samples=2 iops : min= 6, max= 9, avg= 7.50, stdev= 2.12, samples=2 lat (msec) : 50=16.67%, 100=41.67%, 250=25.00%, 750=8.33% cpu : usr=0.00%, sys=8.54%, ctx=100, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29627: Sun Nov 24 03:52:00 2024 read: IOPS=1, BW=1417KiB/s (1451kB/s)(6144KiB/4336msec) clat (usec): min=1819, max=679750, avg=257678.30, stdev=297593.84 lat (usec): min=1820, max=679751, avg=257679.40, stdev=297593.76 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1827], 10.00th=[ 1827], 20.00th=[ 6194], | 30.00th=[ 6194], 40.00th=[ 41681], 50.00th=[ 41681], 60.00th=[258999], | 70.00th=[557843], 80.00th=[557843], 90.00th=[683672], 95.00th=[683672], | 99.00th=[683672], 99.50th=[683672], 99.90th=[683672], 99.95th=[683672], | 99.99th=[683672] bw ( KiB/s): min= 1783, max= 4096, per=55.95%, avg=2642.33, stdev=1265.87, samples=3 iops : min= 1, max= 4, avg= 2.33, stdev= 1.53, samples=3 write: IOPS=1, BW=1417KiB/s (1451kB/s)(6144KiB/4336msec) clat (msec): min=21, max=1677, avg=463.36, stdev=647.82 lat (msec): min=21, max=1678, avg=463.41, stdev=647.81 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 68], | 30.00th=[ 68], 40.00th=[ 87], 50.00th=[ 87], 60.00th=[ 211], | 70.00th=[ 718], 80.00th=[ 718], 90.00th=[ 1687], 95.00th=[ 1687], | 99.00th=[ 1687], 99.50th=[ 1687], 99.90th=[ 1687], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 1783, max= 4096, per=37.72%, avg=2493.75, stdev=1075.45, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 lat (msec) : 2=8.33%, 10=8.33%, 50=16.67%, 100=16.67%, 250=8.33% lat (msec) : 500=8.33%, 750=25.00% cpu : usr=0.00%, sys=26.09%, ctx=414, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29629: Sun Nov 24 03:52:00 2024 read: IOPS=1, BW=1653KiB/s (1692kB/s)(5120KiB/3098msec) clat (usec): min=310, max=643734, avg=137818.34, stdev=283451.82 lat (usec): min=311, max=643735, avg=137819.27, stdev=283451.69 clat percentiles (usec): | 1.00th=[ 310], 5.00th=[ 310], 10.00th=[ 310], 20.00th=[ 310], | 30.00th=[ 375], 40.00th=[ 375], 50.00th=[ 461], 60.00th=[ 461], | 70.00th=[ 44303], 80.00th=[ 44303], 90.00th=[641729], 95.00th=[641729], | 99.00th=[641729], 99.50th=[641729], 99.90th=[641729], 99.95th=[641729], | 99.99th=[641729] bw ( KiB/s): min= 2048, max= 6131, per=86.59%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=2, BW=2314KiB/s (2369kB/s)(7168KiB/3098msec) clat (msec): min=63, max=1663, avg=333.33, stdev=594.29 lat (msec): min=63, max=1663, avg=333.37, stdev=594.29 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 64], | 30.00th=[ 67], 40.00th=[ 67], 50.00th=[ 74], 60.00th=[ 78], | 70.00th=[ 78], 80.00th=[ 326], 90.00th=[ 1670], 95.00th=[ 1670], | 99.00th=[ 1670], 99.50th=[ 1670], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 2048, max= 6144, per=61.90%, avg=4093.00, stdev=2048.01, samples=3 iops : min= 2, max= 6, avg= 3.67, stdev= 2.08, samples=3 lat (usec) : 500=25.00% lat (msec) : 50=8.33%, 100=41.67%, 500=8.33%, 750=8.33% cpu : usr=0.03%, sys=9.62%, ctx=334, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4723KiB/s (4837kB/s), 625KiB/s-2500KiB/s (640kB/s-2560kB/s), io=20.0MiB (20.0MB), run=2867-4336msec WRITE: bw=6613KiB/s (6771kB/s), 1417KiB/s-3124KiB/s (1451kB/s-3199kB/s), io=28.0MiB (29.4MB), run=2867-4336msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29639: Sun Nov 24 03:52:05 2024 read: IOPS=1, BW=5059KiB/s (5180kB/s)(12.0MiB/2429msec) clat (msec): min=154, max=2063, avg=804.89, stdev=1089.81 lat (msec): min=154, max=2063, avg=804.89, stdev=1089.81 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 199], 50.00th=[ 199], 60.00th=[ 199], | 70.00th=[ 2056], 80.00th=[ 2056], 90.00th=[ 2056], 95.00th=[ 2056], | 99.00th=[ 2056], 99.50th=[ 2056], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=44.98%, ctx=775, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29640: Sun Nov 24 03:52:05 2024 read: IOPS=0, BW=1354KiB/s (1387kB/s)(4096KiB/3024msec) clat (nsec): min=125082k, max=125082k, avg=125081964.00, stdev= 0.00 lat (nsec): min=125095k, max=125095k, avg=125095364.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 126], 10.00th=[ 126], 20.00th=[ 126], | 30.00th=[ 126], 40.00th=[ 126], 50.00th=[ 126], 60.00th=[ 126], | 70.00th=[ 126], 80.00th=[ 126], 90.00th=[ 126], 95.00th=[ 126], | 99.00th=[ 126], 99.50th=[ 126], 99.90th=[ 126], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2709KiB/s (2774kB/s)(8192KiB/3024msec) clat (msec): min=429, max=2425, avg=1427.79, stdev=1411.44 lat (msec): min=430, max=2426, avg=1428.10, stdev=1411.44 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 430], 10.00th=[ 430], 20.00th=[ 430], | 30.00th=[ 430], 40.00th=[ 430], 50.00th=[ 430], 60.00th=[ 2433], | 70.00th=[ 2433], 80.00th=[ 2433], 90.00th=[ 2433], 95.00th=[ 2433], | 99.00th=[ 2433], 99.50th=[ 2433], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 8192, max= 8192, per=76.33%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=8.80%, ctx=113, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29643: Sun Nov 24 03:52:05 2024 write: IOPS=0, BW=4025KiB/s (4121kB/s)(12.0MiB/3053msec) clat (msec): min=106, max=2410, avg=1008.57, stdev=1230.51 lat (msec): min=106, max=2410, avg=1008.93, stdev=1230.53 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 510], 50.00th=[ 510], 60.00th=[ 510], | 70.00th=[ 2400], 80.00th=[ 2400], 90.00th=[ 2400], 95.00th=[ 2400], | 99.00th=[ 2400], 99.50th=[ 2400], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 8192, max= 8192, per=76.33%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=8.81%, ctx=48, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29645: Sun Nov 24 03:52:05 2024 write: IOPS=1, BW=4341KiB/s (4445kB/s)(12.0MiB/2831msec) clat (msec): min=103, max=2381, avg=930.35, stdev=1261.12 lat (msec): min=103, max=2382, avg=930.68, stdev=1261.15 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 104], | 30.00th=[ 104], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 305], | 70.00th=[ 2366], 80.00th=[ 2366], 90.00th=[ 2366], 95.00th=[ 2366], | 99.00th=[ 2366], 99.50th=[ 2366], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 8175, max= 8175, per=76.17%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.07%, sys=9.89%, ctx=54, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5418KiB/s (5548kB/s), 1354KiB/s-5059KiB/s (1387kB/s-5180kB/s), io=16.0MiB (16.8MB), run=2429-3024msec WRITE: bw=10.5MiB/s (10.0MB/s), 2709KiB/s-4341KiB/s (2774kB/s-4445kB/s), io=32.0MiB (33.6MB), run=2831-3053msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29638: Sun Nov 24 03:52:05 2024 read: IOPS=1, BW=4931KiB/s (5049kB/s)(12.0MiB/2492msec) clat (msec): min=220, max=1955, avg=817.84, stdev=985.32 lat (msec): min=220, max=1955, avg=817.84, stdev=985.32 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 220], 10.00th=[ 220], 20.00th=[ 220], | 30.00th=[ 220], 40.00th=[ 279], 50.00th=[ 279], 60.00th=[ 279], | 70.00th=[ 1955], 80.00th=[ 1955], 90.00th=[ 1955], 95.00th=[ 1955], | 99.00th=[ 1955], 99.50th=[ 1955], 99.90th=[ 1955], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=3.09%, ctx=34, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29641: Sun Nov 24 03:52:05 2024 read: IOPS=0, BW=1302KiB/s (1333kB/s)(4096KiB/3146msec) clat (nsec): min=580665k, max=580665k, avg=580665262.00, stdev= 0.00 lat (nsec): min=580667k, max=580667k, avg=580667047.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 584], 5.00th=[ 584], 10.00th=[ 584], 20.00th=[ 584], | 30.00th=[ 584], 40.00th=[ 584], 50.00th=[ 584], 60.00th=[ 584], | 70.00th=[ 584], 80.00th=[ 584], 90.00th=[ 584], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2604KiB/s (2666kB/s)(8192KiB/3146msec) clat (msec): min=111, max=2436, avg=1273.83, stdev=1644.33 lat (msec): min=111, max=2436, avg=1274.15, stdev=1644.38 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 111], 50.00th=[ 111], 60.00th=[ 2433], | 70.00th=[ 2433], 80.00th=[ 2433], 90.00th=[ 2433], 95.00th=[ 2433], | 99.00th=[ 2433], 99.50th=[ 2433], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 8192, max= 8192, per=81.25%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.03%, sys=3.02%, ctx=180, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29642: Sun Nov 24 03:52:05 2024 write: IOPS=0, BW=3781KiB/s (3872kB/s)(12.0MiB/3250msec) clat (msec): min=66, max=2400, avg=1060.90, stdev=1204.70 lat (msec): min=66, max=2400, avg=1061.21, stdev=1204.59 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 67], | 30.00th=[ 67], 40.00th=[ 718], 50.00th=[ 718], 60.00th=[ 718], | 70.00th=[ 2400], 80.00th=[ 2400], 90.00th=[ 2400], 95.00th=[ 2400], | 99.00th=[ 2400], 99.50th=[ 2400], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 8192, max= 8192, per=81.25%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.00%, sys=3.66%, ctx=45, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29644: Sun Nov 24 03:52:05 2024 write: IOPS=1, BW=4154KiB/s (4254kB/s)(12.0MiB/2958msec) clat (msec): min=139, max=2366, avg=971.68, stdev=1215.68 lat (msec): min=139, max=2367, avg=972.20, stdev=1215.57 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 140], 10.00th=[ 140], 20.00th=[ 140], | 30.00th=[ 140], 40.00th=[ 409], 50.00th=[ 409], 60.00th=[ 409], | 70.00th=[ 2366], 80.00th=[ 2366], 90.00th=[ 2366], 95.00th=[ 2366], | 99.00th=[ 2366], 99.50th=[ 2366], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 8192, max= 8192, per=81.25%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=5.17%, ctx=120, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5208KiB/s (5333kB/s), 1302KiB/s-4931KiB/s (1333kB/s-5049kB/s), io=16.0MiB (16.8MB), run=2492-3146msec WRITE: bw=9.85MiB/s (10.3MB/s), 2604KiB/s-4154KiB/s (2666kB/s-4254kB/s), io=32.0MiB (33.6MB), run=2958-3250msec