== sanity test 398b: DIO and buffer IO race ============== 00:43:48 (1717562628) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 0.096662 s, 521 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25121: Wed Jun 5 00:43:56 2024 read: IOPS=231, BW=926KiB/s (948kB/s)(5992KiB/6469msec) clat (usec): min=2, max=30828, avg=1594.67, stdev=2132.83 lat (usec): min=2, max=30828, avg=1595.32, stdev=2132.81 clat percentiles (usec): | 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 13], 40.00th=[ 930], 50.00th=[ 1123], 60.00th=[ 1369], | 70.00th=[ 1778], 80.00th=[ 2540], 90.00th=[ 3982], 95.00th=[ 5080], | 99.00th=[ 7963], 99.50th=[10028], 99.90th=[28181], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 720, max= 1144, per=24.34%, avg=902.92, stdev=120.02, samples=12 iops : min= 180, max= 286, avg=225.67, stdev=30.03, samples=12 write: IOPS=243, BW=973KiB/s (997kB/s)(6296KiB/6469msec) clat (usec): min=41, max=40650, avg=2531.24, stdev=3026.23 lat (usec): min=41, max=40650, avg=2532.05, stdev=3026.14 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 65], 10.00th=[ 78], 20.00th=[ 98], | 30.00th=[ 474], 40.00th=[ 1369], 50.00th=[ 1876], 60.00th=[ 2409], | 70.00th=[ 3163], 80.00th=[ 4047], 90.00th=[ 5866], 95.00th=[ 7898], | 99.00th=[12125], 99.50th=[13960], 99.90th=[35914], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 784, max= 1200, per=24.57%, avg=954.33, stdev=148.68, samples=12 iops : min= 196, max= 300, avg=238.50, stdev=37.22, samples=12 lat (usec) : 4=1.73%, 10=10.42%, 20=2.90%, 50=0.55%, 100=10.12% lat (usec) : 250=3.84%, 500=1.20%, 750=1.95%, 1000=7.39% lat (msec) : 2=23.08%, 4=21.32%, 10=14.03%, 20=1.24%, 50=0.23% cpu : usr=0.54%, sys=14.38%, ctx=4099, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25122: Wed Jun 5 00:43:56 2024 read: IOPS=229, BW=918KiB/s (940kB/s)(5940KiB/6472msec) clat (usec): min=2, max=33062, avg=1422.27, stdev=1869.25 lat (usec): min=2, max=33063, avg=1422.97, stdev=1869.29 clat percentiles (usec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 832], 50.00th=[ 1029], 60.00th=[ 1254], | 70.00th=[ 1614], 80.00th=[ 2376], 90.00th=[ 3851], 95.00th=[ 4948], | 99.00th=[ 7111], 99.50th=[ 7767], 99.90th=[13042], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 720, max= 1200, per=24.16%, avg=896.33, stdev=135.39, samples=12 iops : min= 180, max= 300, avg=224.00, stdev=33.88, samples=12 write: IOPS=245, BW=981KiB/s (1004kB/s)(6348KiB/6472msec) clat (usec): min=42, max=31551, avg=2684.77, stdev=3224.77 lat (usec): min=42, max=31552, avg=2685.72, stdev=3224.85 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 64], 10.00th=[ 75], 20.00th=[ 102], | 30.00th=[ 799], 40.00th=[ 1516], 50.00th=[ 1958], 60.00th=[ 2474], | 70.00th=[ 3228], 80.00th=[ 4146], 90.00th=[ 6128], 95.00th=[ 8094], | 99.00th=[13304], 99.50th=[23200], 99.90th=[31065], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 806, max= 1128, per=24.64%, avg=957.00, stdev=109.22, samples=12 iops : min= 201, max= 282, avg=239.17, stdev=27.36, samples=12 lat (usec) : 4=2.31%, 10=11.69%, 20=3.58%, 50=0.59%, 100=9.70% lat (usec) : 250=2.83%, 500=1.30%, 750=2.02%, 1000=6.02% lat (msec) : 2=23.37%, 4=21.09%, 10=13.87%, 20=1.24%, 50=0.39% cpu : usr=0.48%, sys=14.46%, ctx=4068, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25124: Wed Jun 5 00:43:56 2024 read: IOPS=235, BW=942KiB/s (965kB/s)(6028KiB/6396msec) clat (usec): min=2, max=43080, avg=1557.46, stdev=2135.31 lat (usec): min=2, max=43081, avg=1558.76, stdev=2135.63 clat percentiles (usec): | 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 13], 40.00th=[ 930], 50.00th=[ 1123], 60.00th=[ 1385], | 70.00th=[ 1762], 80.00th=[ 2540], 90.00th=[ 3851], 95.00th=[ 4883], | 99.00th=[ 7308], 99.50th=[ 9503], 99.90th=[28181], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 656, max= 1256, per=25.23%, avg=936.00, stdev=165.89, samples=12 iops : min= 164, max= 314, avg=234.00, stdev=41.47, samples=12 write: IOPS=244, BW=979KiB/s (1002kB/s)(6260KiB/6396msec) clat (usec): min=43, max=41734, avg=2523.62, stdev=3101.28 lat (usec): min=44, max=41734, avg=2524.82, stdev=3101.43 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 65], 10.00th=[ 77], 20.00th=[ 101], | 30.00th=[ 537], 40.00th=[ 1450], 50.00th=[ 1860], 60.00th=[ 2376], | 70.00th=[ 3097], 80.00th=[ 4146], 90.00th=[ 5604], 95.00th=[ 7570], | 99.00th=[11469], 99.50th=[16581], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 760, max= 1160, per=24.99%, avg=970.67, stdev=114.74, samples=12 iops : min= 190, max= 290, avg=242.67, stdev=28.68, samples=12 lat (usec) : 4=1.66%, 10=10.42%, 20=2.86%, 50=0.65%, 100=9.86% lat (usec) : 250=3.55%, 500=1.50%, 750=2.02%, 1000=6.58% lat (msec) : 2=23.50%, 4=22.01%, 10=14.26%, 20=0.85%, 50=0.29% cpu : usr=0.42%, sys=14.95%, ctx=4046, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25126: Wed Jun 5 00:43:56 2024 read: IOPS=233, BW=936KiB/s (958kB/s)(6052KiB/6469msec) clat (usec): min=2, max=31430, avg=1435.48, stdev=1831.76 lat (usec): min=2, max=31430, avg=1436.22, stdev=1831.70 clat percentiles (usec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 12], 40.00th=[ 881], 50.00th=[ 1074], 60.00th=[ 1319], | 70.00th=[ 1729], 80.00th=[ 2343], 90.00th=[ 3556], 95.00th=[ 4621], | 99.00th=[ 7308], 99.50th=[ 8160], 99.90th=[20055], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 608, max= 1176, per=24.88%, avg=922.92, stdev=165.22, samples=12 iops : min= 152, max= 294, avg=230.67, stdev=41.33, samples=12 write: IOPS=240, BW=964KiB/s (987kB/s)(6236KiB/6469msec) clat (usec): min=39, max=35122, avg=2692.27, stdev=3236.30 lat (usec): min=39, max=35122, avg=2693.20, stdev=3236.16 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 63], 10.00th=[ 77], 20.00th=[ 101], | 30.00th=[ 627], 40.00th=[ 1450], 50.00th=[ 1958], 60.00th=[ 2540], | 70.00th=[ 3228], 80.00th=[ 4359], 90.00th=[ 6456], 95.00th=[ 8291], | 99.00th=[12911], 99.50th=[23725], 99.90th=[30540], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 648, max= 1192, per=24.29%, avg=943.58, stdev=164.32, samples=12 iops : min= 162, max= 298, avg=235.83, stdev=41.10, samples=12 lat (usec) : 4=1.95%, 10=11.46%, 20=3.16%, 50=0.55%, 100=9.77% lat (usec) : 250=3.29%, 500=1.37%, 750=1.99%, 1000=7.00% lat (msec) : 2=22.72%, 4=21.74%, 10=13.70%, 20=0.94%, 50=0.36% cpu : usr=0.34%, sys=14.35%, ctx=4004, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3710KiB/s (3799kB/s), 918KiB/s-942KiB/s (940kB/s-965kB/s), io=23.4MiB (24.6MB), run=6396-6472msec WRITE: bw=3884KiB/s (3978kB/s), 964KiB/s-981KiB/s (987kB/s-1004kB/s), io=24.6MiB (25.7MB), run=6396-6472msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25123: Wed Jun 5 00:44:37 2024 read: IOPS=31, BW=126KiB/s (129kB/s)(5992KiB/47444msec) clat (usec): min=647, max=43850, avg=2949.68, stdev=3157.73 lat (usec): min=648, max=43851, avg=2950.90, stdev=3158.04 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 799], 10.00th=[ 865], 20.00th=[ 988], | 30.00th=[ 1172], 40.00th=[ 1532], 50.00th=[ 1876], 60.00th=[ 2311], | 70.00th=[ 3294], 80.00th=[ 4555], 90.00th=[ 6194], 95.00th=[ 7635], | 99.00th=[15664], 99.50th=[21890], 99.90th=[36963], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 23, max= 312, per=24.95%, avg=125.99, stdev=67.07, samples=94 iops : min= 5, max= 78, avg=31.33, stdev=16.88, samples=94 write: IOPS=33, BW=133KiB/s (136kB/s)(6296KiB/47444msec) clat (usec): min=9898, max=84954, avg=27264.00, stdev=14026.03 lat (usec): min=9899, max=84956, avg=27265.21, stdev=14026.32 clat percentiles (usec): | 1.00th=[10945], 5.00th=[12125], 10.00th=[13304], 20.00th=[15795], | 30.00th=[18744], 40.00th=[20579], 50.00th=[22676], 60.00th=[25560], | 70.00th=[30802], 80.00th=[38536], 90.00th=[48497], 95.00th=[56886], | 99.00th=[70779], 99.50th=[78119], 99.90th=[81265], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 63, max= 224, per=25.05%, avg=132.29, stdev=56.72, samples=94 iops : min= 15, max= 56, avg=32.90, stdev=14.28, samples=94 lat (usec) : 750=1.17%, 1000=8.98% lat (msec) : 2=16.02%, 4=10.68%, 10=10.84%, 20=19.69%, 50=27.99% lat (msec) : 100=4.62% cpu : usr=0.15%, sys=1.28%, ctx=3763, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25125: Wed Jun 5 00:44:37 2024 read: IOPS=31, BW=125KiB/s (128kB/s)(5940KiB/47541msec) clat (usec): min=559, max=36383, avg=2910.30, stdev=3026.91 lat (usec): min=560, max=36384, avg=2911.33, stdev=3027.04 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 799], 10.00th=[ 857], 20.00th=[ 979], | 30.00th=[ 1139], 40.00th=[ 1467], 50.00th=[ 1860], 60.00th=[ 2278], | 70.00th=[ 3130], 80.00th=[ 4490], 90.00th=[ 6259], 95.00th=[ 7898], | 99.00th=[13173], 99.50th=[19792], 99.90th=[31327], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 16, max= 352, per=24.68%, avg=124.63, stdev=71.59, samples=94 iops : min= 4, max= 88, avg=31.05, stdev=17.94, samples=94 write: IOPS=33, BW=134KiB/s (137kB/s)(6348KiB/47541msec) clat (usec): min=10005, max=87895, avg=27163.43, stdev=13938.59 lat (usec): min=10006, max=87897, avg=27164.83, stdev=13939.04 clat percentiles (usec): | 1.00th=[11076], 5.00th=[12125], 10.00th=[13173], 20.00th=[15533], | 30.00th=[18220], 40.00th=[20579], 50.00th=[22676], 60.00th=[25297], | 70.00th=[30802], 80.00th=[38536], 90.00th=[49021], 95.00th=[55837], | 99.00th=[69731], 99.50th=[73925], 99.90th=[78119], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 63, max= 240, per=25.12%, avg=132.63, stdev=56.43, samples=94 iops : min= 15, max= 60, avg=33.05, stdev=14.16, samples=94 lat (usec) : 750=0.88%, 1000=10.12% lat (msec) : 2=15.14%, 4=10.81%, 10=10.09%, 20=20.02%, 50=28.22% lat (msec) : 100=4.72% cpu : usr=0.17%, sys=1.28%, ctx=3742, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25127: Wed Jun 5 00:44:37 2024 read: IOPS=31, BW=127KiB/s (130kB/s)(6028KiB/47528msec) clat (usec): min=597, max=59518, avg=2945.45, stdev=3369.57 lat (usec): min=597, max=59519, avg=2946.47, stdev=3369.74 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 791], 10.00th=[ 848], 20.00th=[ 955], | 30.00th=[ 1106], 40.00th=[ 1450], 50.00th=[ 1893], 60.00th=[ 2311], | 70.00th=[ 3163], 80.00th=[ 4555], 90.00th=[ 6194], 95.00th=[ 7963], | 99.00th=[13173], 99.50th=[21103], 99.90th=[47449], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 39, max= 312, per=25.33%, avg=127.89, stdev=63.22, samples=93 iops : min= 9, max= 78, avg=31.83, stdev=15.87, samples=93 write: IOPS=32, BW=132KiB/s (135kB/s)(6260KiB/47528msec) clat (usec): min=10315, max=81089, avg=27462.06, stdev=13888.08 lat (usec): min=10316, max=81090, avg=27463.24, stdev=13888.36 clat percentiles (usec): | 1.00th=[11076], 5.00th=[12256], 10.00th=[13173], 20.00th=[15795], | 30.00th=[19006], 40.00th=[20841], 50.00th=[22938], 60.00th=[25560], | 70.00th=[31589], 80.00th=[39060], 90.00th=[48497], 95.00th=[56361], | 99.00th=[67634], 99.50th=[72877], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 55, max= 231, per=24.80%, avg=130.96, stdev=56.68, samples=94 iops : min= 13, max= 57, avg=32.60, stdev=14.25, samples=94 lat (usec) : 750=1.33%, 1000=10.35% lat (msec) : 2=14.39%, 4=10.87%, 10=10.81%, 20=18.62%, 50=29.20% lat (msec) : 100=4.43% cpu : usr=0.19%, sys=1.25%, ctx=3686, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25128: Wed Jun 5 00:44:37 2024 read: IOPS=32, BW=128KiB/s (131kB/s)(6052KiB/47279msec) clat (usec): min=614, max=47223, avg=2981.76, stdev=3249.59 lat (usec): min=614, max=47225, avg=2982.76, stdev=3249.80 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 791], 10.00th=[ 865], 20.00th=[ 979], | 30.00th=[ 1188], 40.00th=[ 1500], 50.00th=[ 1893], 60.00th=[ 2311], | 70.00th=[ 3294], 80.00th=[ 4686], 90.00th=[ 6259], 95.00th=[ 8094], | 99.00th=[12387], 99.50th=[21365], 99.90th=[44827], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 24, max= 320, per=25.30%, avg=127.78, stdev=66.12, samples=94 iops : min= 6, max= 80, avg=31.84, stdev=16.56, samples=94 write: IOPS=32, BW=132KiB/s (135kB/s)(6236KiB/47279msec) clat (usec): min=9072, max=87480, avg=27360.80, stdev=14233.13 lat (usec): min=9073, max=87482, avg=27362.13, stdev=14233.42 clat percentiles (usec): | 1.00th=[11076], 5.00th=[12125], 10.00th=[13304], 20.00th=[15401], | 30.00th=[18482], 40.00th=[20841], 50.00th=[22414], 60.00th=[25560], | 70.00th=[31327], 80.00th=[38536], 90.00th=[49021], 95.00th=[57410], | 99.00th=[70779], 99.50th=[80217], 99.90th=[84411], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 63, max= 232, per=24.93%, avg=131.61, stdev=57.45, samples=94 iops : min= 15, max= 58, avg=32.80, stdev=14.42, samples=94 lat (usec) : 750=1.37%, 1000=9.15% lat (msec) : 2=15.76%, 4=10.58%, 10=11.13%, 20=19.27%, 50=28.19% lat (msec) : 100=4.56% cpu : usr=0.15%, sys=1.30%, ctx=3769, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=505KiB/s (517kB/s), 125KiB/s-128KiB/s (128kB/s-131kB/s), io=23.4MiB (24.6MB), run=47279-47541msec WRITE: bw=529KiB/s (541kB/s), 132KiB/s-134KiB/s (135kB/s-137kB/s), io=24.6MiB (25.7MB), run=47279-47541msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25176: Wed Jun 5 00:44:40 2024 read: IOPS=192, BW=3077KiB/s (3151kB/s)(5920KiB/1924msec) clat (usec): min=5, max=29018, avg=3449.61, stdev=3282.83 lat (usec): min=5, max=29019, avg=3450.19, stdev=3282.83 clat percentiles (usec): | 1.00th=[ 6], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 12], | 30.00th=[ 16], 40.00th=[ 2999], 50.00th=[ 3851], 60.00th=[ 4490], | 70.00th=[ 5276], 80.00th=[ 5997], 90.00th=[ 7308], 95.00th=[ 8586], | 99.00th=[ 9765], 99.50th=[10814], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 3072, max= 3417, per=26.91%, avg=3229.67, stdev=174.40, samples=3 iops : min= 192, max= 213, avg=201.67, stdev=10.60, samples=3 write: IOPS=206, BW=3310KiB/s (3389kB/s)(6368KiB/1924msec) clat (usec): min=40, max=25078, avg=1610.08, stdev=3041.42 lat (usec): min=41, max=25078, avg=1611.43, stdev=3041.66 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 60], 10.00th=[ 67], 20.00th=[ 78], | 30.00th=[ 89], 40.00th=[ 104], 50.00th=[ 131], 60.00th=[ 750], | 70.00th=[ 1663], 80.00th=[ 2704], 90.00th=[ 4293], 95.00th=[ 6259], | 99.00th=[16450], 99.50th=[23462], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 3424, max= 3704, per=28.28%, avg=3570.67, stdev=140.48, samples=3 iops : min= 214, max= 231, avg=223.00, stdev= 8.54, samples=3 lat (usec) : 10=5.34%, 20=11.07%, 50=2.73%, 100=18.88%, 250=9.11% lat (usec) : 500=1.56%, 750=0.91%, 1000=1.17% lat (msec) : 2=5.99%, 4=14.06%, 10=27.21%, 20=1.43%, 50=0.52% cpu : usr=0.52%, sys=10.19%, ctx=1527, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25177: Wed Jun 5 00:44:40 2024 read: IOPS=179, BW=2878KiB/s (2947kB/s)(5744KiB/1996msec) clat (usec): min=5, max=12187, avg=3192.67, stdev=3088.13 lat (usec): min=5, max=12187, avg=3193.25, stdev=3088.12 clat percentiles (usec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 12], | 30.00th=[ 16], 40.00th=[ 23], 50.00th=[ 3752], 60.00th=[ 4490], | 70.00th=[ 5211], 80.00th=[ 5932], 90.00th=[ 7046], 95.00th=[ 8094], | 99.00th=[10814], 99.50th=[11863], 99.90th=[12125], 99.95th=[12125], | 99.99th=[12125] bw ( KiB/s): min= 1760, max= 3800, per=25.76%, avg=3090.67, stdev=1153.23, samples=3 iops : min= 110, max= 237, avg=193.00, stdev=71.92, samples=3 write: IOPS=204, BW=3279KiB/s (3357kB/s)(6544KiB/1996msec) clat (usec): min=39, max=42304, avg=2064.30, stdev=4579.99 lat (usec): min=39, max=42304, avg=2065.27, stdev=4579.99 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 64], 10.00th=[ 70], 20.00th=[ 83], | 30.00th=[ 96], 40.00th=[ 113], 50.00th=[ 174], 60.00th=[ 1270], | 70.00th=[ 2114], 80.00th=[ 2868], 90.00th=[ 4424], 95.00th=[ 7046], | 99.00th=[25560], 99.50th=[26346], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 2080, max= 4534, per=26.33%, avg=3324.67, stdev=1227.38, samples=3 iops : min= 130, max= 283, avg=207.67, stdev=76.53, samples=3 lat (usec) : 10=5.47%, 20=12.50%, 50=2.73%, 100=16.93%, 250=10.81% lat (usec) : 500=1.69%, 750=0.52%, 1000=0.78% lat (msec) : 2=5.47%, 4=14.97%, 10=25.26%, 20=1.69%, 50=1.17% cpu : usr=0.45%, sys=9.97%, ctx=1381, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25178: Wed Jun 5 00:44:40 2024 read: IOPS=215, BW=3453KiB/s (3535kB/s)(6256KiB/1812msec) clat (usec): min=4, max=12588, avg=3215.13, stdev=2941.51 lat (usec): min=4, max=12589, avg=3215.75, stdev=2941.56 clat percentiles (usec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 12], | 30.00th=[ 15], 40.00th=[ 1123], 50.00th=[ 3720], 60.00th=[ 4424], | 70.00th=[ 5014], 80.00th=[ 5735], 90.00th=[ 6980], 95.00th=[ 8029], | 99.00th=[10552], 99.50th=[11338], 99.90th=[12649], 99.95th=[12649], | 99.99th=[12649] bw ( KiB/s): min= 2816, max= 3968, per=28.62%, avg=3434.67, stdev=580.72, samples=3 iops : min= 176, max= 248, avg=214.67, stdev=36.30, samples=3 write: IOPS=208, BW=3329KiB/s (3409kB/s)(6032KiB/1812msec) clat (usec): min=39, max=24372, avg=1456.70, stdev=2491.98 lat (usec): min=39, max=24373, avg=1457.89, stdev=2491.90 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 58], 10.00th=[ 71], 20.00th=[ 84], | 30.00th=[ 95], 40.00th=[ 104], 50.00th=[ 133], 60.00th=[ 1012], | 70.00th=[ 1663], 80.00th=[ 2507], 90.00th=[ 3916], 95.00th=[ 5407], | 99.00th=[12780], 99.50th=[14091], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 3232, max= 3744, per=26.95%, avg=3402.67, stdev=295.60, samples=3 iops : min= 202, max= 234, avg=212.67, stdev=18.48, samples=3 lat (usec) : 10=6.90%, 20=11.85%, 50=1.95%, 100=17.58%, 250=8.33% lat (usec) : 500=1.04%, 750=0.13%, 1000=1.69% lat (msec) : 2=7.29%, 4=14.45%, 10=27.21%, 20=1.43%, 50=0.13% cpu : usr=0.28%, sys=10.71%, ctx=1555, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25179: Wed Jun 5 00:44:40 2024 read: IOPS=198, BW=3178KiB/s (3254kB/s)(6032KiB/1898msec) clat (usec): min=5, max=29787, avg=3401.93, stdev=3344.44 lat (usec): min=5, max=29788, avg=3402.54, stdev=3344.39 clat percentiles (usec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 12], | 30.00th=[ 15], 40.00th=[ 72], 50.00th=[ 3884], 60.00th=[ 4490], | 70.00th=[ 5276], 80.00th=[ 6259], 90.00th=[ 7177], 95.00th=[ 8356], | 99.00th=[10552], 99.50th=[13698], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 2592, max= 4256, per=27.56%, avg=3306.67, stdev=856.46, samples=3 iops : min= 162, max= 266, avg=206.67, stdev=53.53, samples=3 write: IOPS=206, BW=3296KiB/s (3375kB/s)(6256KiB/1898msec) clat (usec): min=41, max=32544, avg=1558.62, stdev=3059.68 lat (usec): min=42, max=32546, avg=1559.73, stdev=3059.67 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 62], 10.00th=[ 69], 20.00th=[ 80], | 30.00th=[ 95], 40.00th=[ 108], 50.00th=[ 131], 60.00th=[ 979], | 70.00th=[ 1795], 80.00th=[ 2474], 90.00th=[ 4015], 95.00th=[ 5800], | 99.00th=[14615], 99.50th=[23200], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 2656, max= 4352, per=27.71%, avg=3498.67, stdev=848.05, samples=3 iops : min= 166, max= 272, avg=218.67, stdev=53.00, samples=3 lat (usec) : 10=5.60%, 20=12.24%, 50=2.08%, 100=17.32%, 250=10.55% lat (usec) : 500=1.04%, 750=0.39%, 1000=1.43% lat (msec) : 2=6.77%, 4=13.67%, 10=27.08%, 20=1.43%, 50=0.39% cpu : usr=0.16%, sys=10.65%, ctx=1450, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.7MiB/s (12.3MB/s), 2878KiB/s-3453KiB/s (2947kB/s-3535kB/s), io=23.4MiB (24.5MB), run=1812-1996msec WRITE: bw=12.3MiB/s (12.9MB/s), 3279KiB/s-3329KiB/s (3357kB/s-3409kB/s), io=24.6MiB (25.8MB), run=1812-1996msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25172: Wed Jun 5 00:44:49 2024 read: IOPS=33, BW=541KiB/s (554kB/s)(5920KiB/10936msec) clat (usec): min=654, max=15483, avg=2297.97, stdev=2004.53 lat (usec): min=655, max=15484, avg=2299.03, stdev=2004.51 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 906], 10.00th=[ 1004], 20.00th=[ 1205], | 30.00th=[ 1401], 40.00th=[ 1532], 50.00th=[ 1680], 60.00th=[ 1876], | 70.00th=[ 2114], 80.00th=[ 2769], 90.00th=[ 4047], 95.00th=[ 5932], | 99.00th=[13960], 99.50th=[14484], 99.90th=[15533], 99.95th=[15533], | 99.99th=[15533] bw ( KiB/s): min= 320, max= 928, per=24.55%, avg=531.76, stdev=175.43, samples=21 iops : min= 20, max= 58, avg=33.19, stdev=10.99, samples=21 write: IOPS=36, BW=582KiB/s (596kB/s)(6368KiB/10936msec) clat (usec): min=11229, max=68348, avg=25314.97, stdev=8123.22 lat (usec): min=11230, max=68349, avg=25317.12, stdev=8123.38 clat percentiles (usec): | 1.00th=[12256], 5.00th=[13960], 10.00th=[15139], 20.00th=[17695], | 30.00th=[20317], 40.00th=[22676], 50.00th=[25035], 60.00th=[27132], | 70.00th=[29230], 80.00th=[31851], 90.00th=[34866], 95.00th=[39060], | 99.00th=[47973], 99.50th=[54789], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 416, max= 800, per=24.94%, avg=568.29, stdev=78.96, samples=21 iops : min= 26, max= 50, avg=35.48, stdev= 4.94, samples=21 lat (usec) : 750=0.39%, 1000=4.17% lat (msec) : 2=27.34%, 4=11.07%, 10=4.56%, 20=15.62%, 50=36.46% lat (msec) : 100=0.39% cpu : usr=0.20%, sys=1.17%, ctx=856, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25173: Wed Jun 5 00:44:49 2024 read: IOPS=32, BW=520KiB/s (532kB/s)(5744KiB/11054msec) clat (usec): min=666, max=29760, avg=2588.49, stdev=3035.31 lat (usec): min=667, max=29760, avg=2589.50, stdev=3035.29 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 881], 10.00th=[ 1057], 20.00th=[ 1205], | 30.00th=[ 1401], 40.00th=[ 1549], 50.00th=[ 1729], 60.00th=[ 1975], | 70.00th=[ 2278], 80.00th=[ 2868], 90.00th=[ 4424], 95.00th=[ 7504], | 99.00th=[19530], 99.50th=[22938], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 319, max= 832, per=23.34%, avg=505.57, stdev=140.83, samples=21 iops : min= 19, max= 52, avg=31.38, stdev= 8.90, samples=21 write: IOPS=37, BW=592KiB/s (606kB/s)(6544KiB/11054msec) clat (usec): min=8137, max=64683, avg=24729.88, stdev=7965.65 lat (usec): min=8138, max=64684, avg=24731.90, stdev=7965.79 clat percentiles (usec): | 1.00th=[ 9372], 5.00th=[12256], 10.00th=[14353], 20.00th=[17433], | 30.00th=[19792], 40.00th=[22152], 50.00th=[25297], 60.00th=[26870], | 70.00th=[28967], 80.00th=[31065], 90.00th=[34866], 95.00th=[38011], | 99.00th=[42206], 99.50th=[43779], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 415, max= 864, per=24.86%, avg=566.48, stdev=94.36, samples=21 iops : min= 25, max= 54, avg=35.19, stdev= 5.93, samples=21 lat (usec) : 750=0.78%, 1000=2.47% lat (msec) : 2=25.65%, 4=12.24%, 10=5.34%, 20=16.41%, 50=36.98% lat (msec) : 100=0.13% cpu : usr=0.29%, sys=1.12%, ctx=846, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25174: Wed Jun 5 00:44:49 2024 read: IOPS=37, BW=593KiB/s (607kB/s)(6256KiB/10553msec) clat (usec): min=744, max=17176, avg=2242.34, stdev=1892.11 lat (usec): min=744, max=17176, avg=2243.46, stdev=1892.12 clat percentiles (usec): | 1.00th=[ 848], 5.00th=[ 1012], 10.00th=[ 1090], 20.00th=[ 1287], | 30.00th=[ 1450], 40.00th=[ 1582], 50.00th=[ 1762], 60.00th=[ 1909], | 70.00th=[ 2180], 80.00th=[ 2638], 90.00th=[ 3589], 95.00th=[ 4555], | 99.00th=[13829], 99.50th=[16712], 99.90th=[17171], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 288, max= 1088, per=27.50%, avg=595.57, stdev=220.02, samples=21 iops : min= 18, max= 68, avg=37.10, stdev=13.80, samples=21 write: IOPS=35, BW=572KiB/s (585kB/s)(6032KiB/10553msec) clat (usec): min=12188, max=46825, avg=25636.42, stdev=7239.59 lat (usec): min=12190, max=46827, avg=25638.30, stdev=7239.66 clat percentiles (usec): | 1.00th=[12649], 5.00th=[14615], 10.00th=[16319], 20.00th=[19006], | 30.00th=[21103], 40.00th=[22938], 50.00th=[25560], 60.00th=[26870], | 70.00th=[28967], 80.00th=[31589], 90.00th=[36439], 95.00th=[39060], | 99.00th=[43779], 99.50th=[45876], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 416, max= 736, per=25.00%, avg=569.67, stdev=69.26, samples=21 iops : min= 26, max= 46, avg=35.48, stdev= 4.46, samples=21 lat (usec) : 750=0.13%, 1000=2.34% lat (msec) : 2=29.56%, 4=14.97%, 10=3.12%, 20=12.89%, 50=36.98% cpu : usr=0.20%, sys=1.25%, ctx=849, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25175: Wed Jun 5 00:44:49 2024 read: IOPS=34, BW=558KiB/s (571kB/s)(6032KiB/10815msec) clat (usec): min=770, max=25997, avg=2349.39, stdev=2334.31 lat (usec): min=771, max=25998, avg=2350.41, stdev=2334.29 clat percentiles (usec): | 1.00th=[ 840], 5.00th=[ 988], 10.00th=[ 1074], 20.00th=[ 1270], | 30.00th=[ 1434], 40.00th=[ 1565], 50.00th=[ 1745], 60.00th=[ 1893], | 70.00th=[ 2212], 80.00th=[ 2835], 90.00th=[ 4228], 95.00th=[ 5276], | 99.00th=[11338], 99.50th=[25560], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 192, max= 960, per=25.04%, avg=542.33, stdev=181.00, samples=21 iops : min= 12, max= 60, avg=33.81, stdev=11.26, samples=21 write: IOPS=36, BW=578KiB/s (592kB/s)(6256KiB/10815msec) clat (usec): min=10744, max=52061, avg=25366.21, stdev=7376.93 lat (usec): min=10746, max=52062, avg=25368.05, stdev=7377.04 clat percentiles (usec): | 1.00th=[11600], 5.00th=[13960], 10.00th=[16319], 20.00th=[18482], | 30.00th=[21103], 40.00th=[23462], 50.00th=[25035], 60.00th=[26870], | 70.00th=[29230], 80.00th=[31327], 90.00th=[34341], 95.00th=[37487], | 99.00th=[45876], 99.50th=[50070], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 448, max= 736, per=24.93%, avg=568.24, stdev=70.14, samples=21 iops : min= 28, max= 46, avg=35.43, stdev= 4.47, samples=21 lat (usec) : 1000=2.99% lat (msec) : 2=27.73%, 4=13.02%, 10=4.56%, 20=14.45%, 50=37.11% lat (msec) : 100=0.13% cpu : usr=0.16%, sys=1.25%, ctx=856, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2167KiB/s (2219kB/s), 520KiB/s-593KiB/s (532kB/s-607kB/s), io=23.4MiB (24.5MB), run=10553-11054msec WRITE: bw=2280KiB/s (2334kB/s), 572KiB/s-592KiB/s (585kB/s-606kB/s), io=24.6MiB (25.8MB), run=10553-11054msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25193: Wed Jun 5 00:44:51 2024 read: IOPS=6, BW=6400KiB/s (6554kB/s)(7168KiB/1120msec) clat (usec): min=438, max=295726, avg=152283.33, stdev=142261.92 lat (usec): min=440, max=295727, avg=152284.67, stdev=142261.89 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 437], 10.00th=[ 437], 20.00th=[ 457], | 30.00th=[ 5800], 40.00th=[ 5800], 50.00th=[221250], 60.00th=[258999], | 70.00th=[258999], 80.00th=[283116], 90.00th=[295699], 95.00th=[295699], | 99.00th=[295699], 99.50th=[295699], 99.90th=[295699], 99.95th=[295699], | 99.99th=[295699] bw ( KiB/s): min= 3319, max= 8175, per=31.51%, avg=5747.00, stdev=3433.71, samples=2 iops : min= 3, max= 7, avg= 5.00, stdev= 2.83, samples=2 write: IOPS=4, BW=4571KiB/s (4681kB/s)(5120KiB/1120msec) clat (usec): min=707, max=27485, avg=10647.85, stdev=11313.19 lat (usec): min=809, max=27506, avg=10700.35, stdev=11281.32 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 709], 10.00th=[ 709], 20.00th=[ 709], | 30.00th=[ 3884], 40.00th=[ 3884], 50.00th=[ 4113], 60.00th=[ 4113], | 70.00th=[17171], 80.00th=[17171], 90.00th=[27395], 95.00th=[27395], | 99.00th=[27395], 99.50th=[27395], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 2043, max= 6638, per=17.00%, avg=4340.50, stdev=3249.16, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 4=8.33%, 10=16.67%, 20=8.33%, 50=8.33%, 250=8.33% lat (msec) : 500=25.00% cpu : usr=0.18%, sys=7.60%, ctx=1089, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25195: Wed Jun 5 00:44:51 2024 read: IOPS=1, BW=1824KiB/s (1867kB/s)(2048KiB/1123msec) clat (msec): min=26, max=290, avg=158.53, stdev=186.46 lat (msec): min=26, max=290, avg=158.54, stdev=186.46 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 27], 50.00th=[ 27], 60.00th=[ 292], | 70.00th=[ 292], 80.00th=[ 292], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 1651, max= 2048, per=10.14%, avg=1849.50, stdev=280.72, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=8, BW=9118KiB/s (9337kB/s)(10.0MiB/1123msec) clat (usec): min=688, max=299514, avg=80445.08, stdev=125119.58 lat (usec): min=709, max=299628, avg=80499.70, stdev=125122.14 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 693], 10.00th=[ 693], 20.00th=[ 742], | 30.00th=[ 898], 40.00th=[ 1680], 50.00th=[ 5276], 60.00th=[ 6063], | 70.00th=[212861], 80.00th=[212861], 90.00th=[265290], 95.00th=[299893], | 99.00th=[299893], 99.50th=[299893], 99.90th=[299893], 99.95th=[299893], | 99.99th=[299893] bw ( KiB/s): min= 6606, max=10240, per=32.99%, avg=8423.00, stdev=2569.63, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (usec) : 750=16.67%, 1000=8.33% lat (msec) : 2=8.33%, 10=16.67%, 20=8.33%, 50=8.33%, 250=8.33% lat (msec) : 500=25.00% cpu : usr=0.09%, sys=6.33%, ctx=269, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25196: Wed Jun 5 00:44:51 2024 read: IOPS=5, BW=5515KiB/s (5648kB/s)(6144KiB/1114msec) clat (usec): min=187, max=295922, avg=53652.45, stdev=118853.14 lat (usec): min=189, max=295923, avg=53653.76, stdev=118853.36 clat percentiles (usec): | 1.00th=[ 188], 5.00th=[ 188], 10.00th=[ 188], 20.00th=[ 367], | 30.00th=[ 367], 40.00th=[ 553], 50.00th=[ 553], 60.00th=[ 9110], | 70.00th=[ 15795], 80.00th=[ 15795], 90.00th=[295699], 95.00th=[295699], | 99.00th=[295699], 99.50th=[295699], 99.90th=[295699], 99.95th=[295699], | 99.99th=[295699] bw ( KiB/s): min= 4096, max= 4096, per=22.46%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5515KiB/s (5648kB/s)(6144KiB/1114msec) clat (usec): min=1057, max=301356, avg=131813.29, stdev=131630.65 lat (usec): min=1073, max=301373, avg=131841.47, stdev=131627.23 clat percentiles (usec): | 1.00th=[ 1057], 5.00th=[ 1057], 10.00th=[ 1057], 20.00th=[ 7177], | 30.00th=[ 7177], 40.00th=[ 40633], 50.00th=[ 40633], 60.00th=[196084], | 70.00th=[244319], 80.00th=[244319], 90.00th=[299893], 95.00th=[299893], | 99.00th=[299893], 99.50th=[299893], 99.90th=[299893], 99.95th=[299893], | 99.99th=[299893] bw ( KiB/s): min= 8192, max= 8192, per=32.09%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 250=8.33%, 500=8.33%, 750=8.33% lat (msec) : 2=8.33%, 10=16.67%, 20=8.33%, 50=8.33%, 250=16.67% lat (msec) : 500=16.67% cpu : usr=0.09%, sys=2.88%, ctx=280, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25197: Wed Jun 5 00:44:51 2024 read: IOPS=9, BW=9606KiB/s (9837kB/s)(5120KiB/533msec) clat (usec): min=255, max=191033, avg=44327.85, stdev=82386.86 lat (usec): min=255, max=191034, avg=44329.07, stdev=82387.15 clat percentiles (usec): | 1.00th=[ 255], 5.00th=[ 255], 10.00th=[ 255], 20.00th=[ 255], | 30.00th=[ 400], 40.00th=[ 400], 50.00th=[ 10945], 60.00th=[ 10945], | 70.00th=[ 19006], 80.00th=[ 19006], 90.00th=[191890], 95.00th=[191890], | 99.00th=[191890], 99.50th=[191890], 99.90th=[191890], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 8192, max= 8192, per=44.92%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.1MiB/s (13.8MB/s)(7168KiB/533msec) clat (usec): min=1471, max=197950, avg=44351.88, stdev=70551.94 lat (usec): min=1504, max=197964, avg=44388.78, stdev=70541.28 clat percentiles (usec): | 1.00th=[ 1467], 5.00th=[ 1467], 10.00th=[ 1467], 20.00th=[ 2573], | 30.00th=[ 5866], 40.00th=[ 5866], 50.00th=[ 10945], 60.00th=[ 39060], | 70.00th=[ 39060], 80.00th=[ 52167], 90.00th=[198181], 95.00th=[198181], | 99.00th=[198181], 99.50th=[198181], 99.90th=[198181], 99.95th=[198181], | 99.99th=[198181] bw ( KiB/s): min=10240, max=10240, per=40.11%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 2=8.33%, 4=8.33%, 10=8.33%, 20=25.00%, 50=8.33% lat (msec) : 100=8.33%, 250=16.67% cpu : usr=0.38%, sys=5.08%, ctx=282, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.8MiB/s (18.7MB/s), 1824KiB/s-9606KiB/s (1867kB/s-9837kB/s), io=20.0MiB (20.0MB), run=533-1123msec WRITE: bw=24.9MiB/s (26.1MB/s), 4571KiB/s-13.1MiB/s (4681kB/s-13.8MB/s), io=28.0MiB (29.4MB), run=533-1123msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25198: Wed Jun 5 00:44:51 2024 read: IOPS=5, BW=5983KiB/s (6127kB/s)(7168KiB/1198msec) clat (msec): min=3, max=377, avg=90.36, stdev=148.17 lat (msec): min=3, max=377, avg=90.36, stdev=148.17 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 7], | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 10], 60.00th=[ 16], | 70.00th=[ 16], 80.00th=[ 215], 90.00th=[ 376], 95.00th=[ 376], | 99.00th=[ 376], 99.50th=[ 376], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 2048, max= 8192, per=30.55%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=4, BW=4274KiB/s (4376kB/s)(5120KiB/1198msec) clat (msec): min=17, max=258, avg=112.75, stdev=116.71 lat (msec): min=17, max=258, avg=112.84, stdev=116.69 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 22], 40.00th=[ 22], 50.00th=[ 47], 60.00th=[ 47], | 70.00th=[ 220], 80.00th=[ 220], 90.00th=[ 259], 95.00th=[ 259], | 99.00th=[ 259], 99.50th=[ 259], 99.90th=[ 259], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 6144, max= 6144, per=26.19%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 4=8.33%, 10=25.00%, 20=16.67%, 50=16.67%, 250=16.67% lat (msec) : 500=16.67% cpu : usr=0.00%, sys=0.50%, ctx=21, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25199: Wed Jun 5 00:44:51 2024 read: IOPS=1, BW=1676KiB/s (1716kB/s)(2048KiB/1222msec) clat (usec): min=7503, max=18067, avg=12785.47, stdev=7469.33 lat (usec): min=7504, max=18068, avg=12786.64, stdev=7470.14 clat percentiles (usec): | 1.00th=[ 7504], 5.00th=[ 7504], 10.00th=[ 7504], 20.00th=[ 7504], | 30.00th=[ 7504], 40.00th=[ 7504], 50.00th=[ 7504], 60.00th=[17957], | 70.00th=[17957], 80.00th=[17957], 90.00th=[17957], 95.00th=[17957], | 99.00th=[17957], 99.50th=[17957], 99.90th=[17957], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 2048, max= 2048, per=12.22%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=8, BW=8380KiB/s (8581kB/s)(10.0MiB/1222msec) clat (msec): min=24, max=334, avg=119.34, stdev=118.97 lat (msec): min=24, max=334, avg=119.43, stdev=118.97 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 27], | 30.00th=[ 29], 40.00th=[ 30], 50.00th=[ 33], 60.00th=[ 44], | 70.00th=[ 230], 80.00th=[ 230], 90.00th=[ 234], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 6144, max= 6144, per=26.19%, avg=6144.00, stdev= 0.00, samples=2 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=2 lat (msec) : 10=8.33%, 20=8.33%, 50=50.00%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=0.66%, ctx=23, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25200: Wed Jun 5 00:44:51 2024 read: IOPS=5, BW=5306KiB/s (5433kB/s)(6144KiB/1158msec) clat (msec): min=4, max=228, avg=79.45, stdev=102.10 lat (msec): min=4, max=228, avg=79.45, stdev=102.10 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 7], | 30.00th=[ 7], 40.00th=[ 9], 50.00th=[ 9], 60.00th=[ 39], | 70.00th=[ 190], 80.00th=[ 190], 90.00th=[ 230], 95.00th=[ 230], | 99.00th=[ 230], 99.50th=[ 230], 99.90th=[ 230], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 8192, max= 8192, per=48.88%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5306KiB/s (5433kB/s)(6144KiB/1158msec) clat (msec): min=17, max=518, avg=113.10, stdev=199.14 lat (msec): min=17, max=518, avg=113.21, stdev=199.12 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 30], 50.00th=[ 30], 60.00th=[ 40], | 70.00th=[ 52], 80.00th=[ 52], 90.00th=[ 518], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 4096, max= 6144, per=21.82%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 10=25.00%, 20=8.33%, 50=33.33%, 100=8.33%, 250=16.67% lat (msec) : 750=8.33% cpu : usr=0.17%, sys=0.69%, ctx=22, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25201: Wed Jun 5 00:44:51 2024 read: IOPS=4, BW=4588KiB/s (4698kB/s)(5120KiB/1116msec) clat (msec): min=3, max=301, avg=64.60, stdev=132.58 lat (msec): min=3, max=301, avg=64.60, stdev=132.58 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 6], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 7], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 2048, max= 6131, per=24.40%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=6, BW=6423KiB/s (6577kB/s)(7168KiB/1116msec) clat (msec): min=17, max=251, avg=111.63, stdev=99.33 lat (msec): min=17, max=252, avg=111.78, stdev=99.26 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 22], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 54], 60.00th=[ 188], | 70.00th=[ 188], 80.00th=[ 205], 90.00th=[ 253], 95.00th=[ 253], | 99.00th=[ 253], 99.50th=[ 253], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 4087, max= 8192, per=26.17%, avg=6139.50, stdev=2902.67, samples=2 iops : min= 3, max= 8, avg= 5.50, stdev= 3.54, samples=2 lat (msec) : 4=8.33%, 10=25.00%, 20=8.33%, 50=16.67%, 100=8.33% lat (msec) : 250=16.67%, 500=16.67% cpu : usr=0.09%, sys=1.43%, ctx=23, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.4MiB/s (17.2MB/s), 1676KiB/s-5983KiB/s (1716kB/s-6127kB/s), io=20.0MiB (20.0MB), run=1116-1222msec WRITE: bw=22.9MiB/s (24.0MB/s), 4274KiB/s-8380KiB/s (4376kB/s-8581kB/s), io=28.0MiB (29.4MB), run=1116-1222msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25208: Wed Jun 5 00:44:53 2024 read: IOPS=2, BW=10.9MiB/s (11.5MB/s)(12.0MiB/1096msec) clat (msec): min=7, max=1039, avg=356.49, stdev=591.59 lat (msec): min=7, max=1039, avg=356.49, stdev=591.59 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 8], 10.00th=[ 8], 20.00th=[ 8], | 30.00th=[ 8], 40.00th=[ 23], 50.00th=[ 23], 60.00th=[ 23], | 70.00th=[ 1036], 80.00th=[ 1036], 90.00th=[ 1036], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 8078, max= 8078, per=58.38%, avg=8078.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 10=33.33%, 50=33.33% cpu : usr=0.64%, sys=7.31%, ctx=1068, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25209: Wed Jun 5 00:44:53 2024 read: IOPS=0, BW=3459KiB/s (3542kB/s)(4096KiB/1184msec) clat (nsec): min=17925k, max=17925k, avg=17924965.00, stdev= 0.00 lat (nsec): min=17928k, max=17928k, avg=17927564.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[17957], 5.00th=[17957], 10.00th=[17957], 20.00th=[17957], | 30.00th=[17957], 40.00th=[17957], 50.00th=[17957], 60.00th=[17957], | 70.00th=[17957], 80.00th=[17957], 90.00th=[17957], 95.00th=[17957], | 99.00th=[17957], 99.50th=[17957], 99.90th=[17957], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 7713, max= 7713, per=55.74%, avg=7713.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6919KiB/s (7085kB/s)(8192KiB/1184msec) clat (msec): min=58, max=1066, avg=562.56, stdev=712.48 lat (msec): min=58, max=1066, avg=562.77, stdev=712.54 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 59], | 30.00th=[ 59], 40.00th=[ 59], 50.00th=[ 59], 60.00th=[ 1070], | 70.00th=[ 1070], 80.00th=[ 1070], 90.00th=[ 1070], 95.00th=[ 1070], | 99.00th=[ 1070], 99.50th=[ 1070], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 7713, max= 7713, per=28.95%, avg=7713.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 100=33.33% cpu : usr=0.17%, sys=2.03%, ctx=46, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25211: Wed Jun 5 00:44:53 2024 write: IOPS=2, BW=9990KiB/s (10.2MB/s)(12.0MiB/1230msec) clat (msec): min=31, max=1070, avg=397.02, stdev=583.66 lat (msec): min=31, max=1070, avg=397.35, stdev=583.58 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 90], 50.00th=[ 90], 60.00th=[ 90], | 70.00th=[ 1070], 80.00th=[ 1070], 90.00th=[ 1070], 95.00th=[ 1070], | 99.00th=[ 1070], 99.50th=[ 1070], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 7907, max= 7907, per=29.68%, avg=7907.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=33.33% cpu : usr=0.16%, sys=1.95%, ctx=16, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25213: Wed Jun 5 00:44:53 2024 write: IOPS=49, BW=197MiB/s (206MB/s)(12.0MiB/61msec) clat (usec): min=11621, max=29643, avg=19592.20, stdev=9189.16 lat (usec): min=12064, max=29857, avg=19990.79, stdev=9053.71 clat percentiles (usec): | 1.00th=[11600], 5.00th=[11600], 10.00th=[11600], 20.00th=[11600], | 30.00th=[11600], 40.00th=[17433], 50.00th=[17433], 60.00th=[17433], | 70.00th=[29754], 80.00th=[29754], 90.00th=[29754], 95.00th=[29754], | 99.00th=[29754], 99.50th=[29754], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] lat (msec) : 20=66.67%, 50=33.33% cpu : usr=1.67%, sys=48.33%, ctx=26, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.5MiB/s (14.2MB/s), 3459KiB/s-10.9MiB/s (3542kB/s-11.5MB/s), io=16.0MiB (16.8MB), run=1096-1184msec WRITE: bw=26.0MiB/s (27.3MB/s), 6919KiB/s-197MiB/s (7085kB/s-206MB/s), io=32.0MiB (33.6MB), run=61-1230msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25210: Wed Jun 5 00:44:53 2024 read: IOPS=2, BW=9.85MiB/s (10.3MB/s)(12.0MiB/1218msec) clat (msec): min=32, max=1069, avg=393.87, stdev=585.27 lat (msec): min=32, max=1069, avg=393.87, stdev=585.27 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 80], 50.00th=[ 80], 60.00th=[ 80], | 70.00th=[ 1070], 80.00th=[ 1070], 90.00th=[ 1070], 95.00th=[ 1070], | 99.00th=[ 1070], 99.50th=[ 1070], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 8192, max= 8192, per=62.70%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=33.33% cpu : usr=0.08%, sys=0.49%, ctx=15, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25212: Wed Jun 5 00:44:53 2024 read: IOPS=0, BW=3266KiB/s (3345kB/s)(4096KiB/1254msec) clat (nsec): min=66831k, max=66831k, avg=66831182.00, stdev= 0.00 lat (nsec): min=66833k, max=66833k, avg=66833472.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[66847], 5.00th=[66847], 10.00th=[66847], 20.00th=[66847], | 30.00th=[66847], 40.00th=[66847], 50.00th=[66847], 60.00th=[66847], | 70.00th=[66847], 80.00th=[66847], 90.00th=[66847], 95.00th=[66847], | 99.00th=[66847], 99.50th=[66847], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 8192, max= 8192, per=62.70%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6533KiB/s (6689kB/s)(8192KiB/1254msec) clat (msec): min=64, max=1090, avg=577.46, stdev=725.86 lat (msec): min=64, max=1090, avg=577.81, stdev=725.73 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 65], 10.00th=[ 65], 20.00th=[ 65], | 30.00th=[ 65], 40.00th=[ 65], 50.00th=[ 65], 60.00th=[ 1099], | 70.00th=[ 1099], 80.00th=[ 1099], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 8192, max= 8192, per=31.35%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=66.67% cpu : usr=0.16%, sys=0.48%, ctx=22, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25214: Wed Jun 5 00:44:53 2024 write: IOPS=2, BW=9.78MiB/s (10.3MB/s)(12.0MiB/1227msec) clat (msec): min=54, max=1058, avg=392.97, stdev=576.41 lat (msec): min=54, max=1058, avg=393.22, stdev=576.32 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 55], 10.00th=[ 55], 20.00th=[ 55], | 30.00th=[ 55], 40.00th=[ 67], 50.00th=[ 67], 60.00th=[ 67], | 70.00th=[ 1062], 80.00th=[ 1062], 90.00th=[ 1062], 95.00th=[ 1062], | 99.00th=[ 1062], 99.50th=[ 1062], 99.90th=[ 1062], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 8175, max= 8175, per=31.29%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=66.67% cpu : usr=0.08%, sys=1.06%, ctx=19, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25215: Wed Jun 5 00:44:53 2024 write: IOPS=2, BW=10.4MiB/s (10.0MB/s)(12.0MiB/1149msec) clat (msec): min=44, max=1011, avg=370.09, stdev=555.19 lat (msec): min=47, max=1011, avg=371.38, stdev=554.33 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 45], | 30.00th=[ 45], 40.00th=[ 55], 50.00th=[ 55], 60.00th=[ 55], | 70.00th=[ 1011], 80.00th=[ 1011], 90.00th=[ 1011], 95.00th=[ 1011], | 99.00th=[ 1011], 99.50th=[ 1011], 99.90th=[ 1011], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 8192, max= 8192, per=31.35%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=33.33% cpu : usr=0.17%, sys=1.13%, ctx=20, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.8MiB/s (13.4MB/s), 3266KiB/s-9.85MiB/s (3345kB/s-10.3MB/s), io=16.0MiB (16.8MB), run=1218-1254msec WRITE: bw=25.5MiB/s (26.8MB/s), 6533KiB/s-10.4MiB/s (6689kB/s-10.0MB/s), io=32.0MiB (33.6MB), run=1149-1254msec