== sanity test 398b: DIO and buffer IO race ============== 05:05:33 (1713085533) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.15732 s, 43.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23041: Sun Apr 14 05:05:51 2024 read: IOPS=100, BW=400KiB/s (410kB/s)(5992KiB/14965msec) clat (usec): min=38, max=169290, avg=4412.58, stdev=6087.82 lat (usec): min=38, max=169291, avg=4413.41, stdev=6087.84 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 73], 10.00th=[ 90], 20.00th=[ 1795], | 30.00th=[ 2606], 40.00th=[ 3064], 50.00th=[ 3523], 60.00th=[ 4146], | 70.00th=[ 5080], 80.00th=[ 6718], 90.00th=[ 8979], 95.00th=[ 10814], | 99.00th=[ 14615], 99.50th=[ 17695], 99.90th=[ 85459], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 248, max= 592, per=25.01%, avg=396.66, stdev=80.05, samples=29 iops : min= 62, max= 148, avg=99.14, stdev=20.01, samples=29 write: IOPS=105, BW=421KiB/s (431kB/s)(6296KiB/14965msec) clat (usec): min=412, max=59705, avg=5250.98, stdev=5055.73 lat (usec): min=413, max=59705, avg=5252.15, stdev=5055.61 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 578], 10.00th=[ 652], 20.00th=[ 783], | 30.00th=[ 1303], 40.00th=[ 3261], 50.00th=[ 4424], 60.00th=[ 5473], | 70.00th=[ 6587], 80.00th=[ 8455], 90.00th=[11207], 95.00th=[14484], | 99.00th=[22152], 99.50th=[23725], 99.90th=[44827], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 296, max= 656, per=24.98%, avg=414.59, stdev=81.92, samples=29 iops : min= 74, max= 164, avg=103.62, stdev=20.46, samples=29 lat (usec) : 50=0.29%, 100=6.25%, 250=2.83%, 500=0.78%, 750=8.56% lat (usec) : 1000=4.59% lat (msec) : 2=4.33%, 4=24.41%, 10=37.57%, 20=9.51%, 50=0.75% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.29%, sys=23.38%, ctx=6209, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23042: Sun Apr 14 05:05:51 2024 read: IOPS=99, BW=398KiB/s (408kB/s)(5940KiB/14913msec) clat (usec): min=42, max=46932, avg=4192.38, stdev=3677.19 lat (usec): min=42, max=46933, avg=4193.25, stdev=3677.25 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 74], 10.00th=[ 90], 20.00th=[ 180], | 30.00th=[ 2671], 40.00th=[ 3064], 50.00th=[ 3556], 60.00th=[ 4146], | 70.00th=[ 5080], 80.00th=[ 6783], 90.00th=[ 8586], 95.00th=[ 9896], | 99.00th=[13829], 99.50th=[16909], 99.90th=[36439], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 280, max= 566, per=25.02%, avg=396.79, stdev=77.68, samples=29 iops : min= 70, max= 141, avg=99.10, stdev=19.34, samples=29 write: IOPS=106, BW=426KiB/s (436kB/s)(6348KiB/14913msec) clat (usec): min=389, max=167833, avg=5414.38, stdev=6595.86 lat (usec): min=389, max=167834, avg=5416.56, stdev=6595.33 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 586], 10.00th=[ 676], 20.00th=[ 807], | 30.00th=[ 1319], 40.00th=[ 3163], 50.00th=[ 4424], 60.00th=[ 5407], | 70.00th=[ 6652], 80.00th=[ 8717], 90.00th=[ 11731], 95.00th=[ 14222], | 99.00th=[ 23462], 99.50th=[ 28705], 99.90th=[ 55837], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 304, max= 544, per=25.27%, avg=419.45, stdev=61.22, samples=29 iops : min= 76, max= 136, avg=104.76, stdev=15.29, samples=29 lat (usec) : 50=0.10%, 100=6.35%, 250=3.55%, 500=0.75%, 750=8.07% lat (usec) : 1000=4.85% lat (msec) : 2=5.14%, 4=23.21%, 10=37.92%, 20=9.08%, 50=0.88% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.19%, sys=23.06%, ctx=6284, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23043: Sun Apr 14 05:05:51 2024 read: IOPS=100, BW=403KiB/s (413kB/s)(6028KiB/14954msec) clat (usec): min=37, max=48874, avg=4248.86, stdev=3573.53 lat (usec): min=37, max=48875, avg=4249.95, stdev=3573.28 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 74], 10.00th=[ 90], 20.00th=[ 400], | 30.00th=[ 2671], 40.00th=[ 3130], 50.00th=[ 3621], 60.00th=[ 4293], | 70.00th=[ 5276], 80.00th=[ 6718], 90.00th=[ 8717], 95.00th=[10290], | 99.00th=[14222], 99.50th=[16909], 99.90th=[32113], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 240, max= 520, per=25.31%, avg=401.34, stdev=70.68, samples=29 iops : min= 60, max= 130, avg=100.31, stdev=17.67, samples=29 write: IOPS=104, BW=419KiB/s (429kB/s)(6260KiB/14954msec) clat (usec): min=421, max=170625, avg=5407.99, stdev=6915.14 lat (usec): min=422, max=170626, avg=5409.22, stdev=6915.11 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 586], 10.00th=[ 652], 20.00th=[ 775], | 30.00th=[ 1106], 40.00th=[ 2769], 50.00th=[ 4293], 60.00th=[ 5407], | 70.00th=[ 6718], 80.00th=[ 8717], 90.00th=[ 11994], 95.00th=[ 15008], | 99.00th=[ 24511], 99.50th=[ 35914], 99.90th=[ 63177], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 272, max= 544, per=24.97%, avg=414.55, stdev=76.85, samples=29 iops : min= 68, max= 136, avg=103.62, stdev=19.19, samples=29 lat (usec) : 50=0.55%, 100=5.96%, 250=3.19%, 500=0.85%, 750=9.05% lat (usec) : 1000=4.79% lat (msec) : 2=4.82%, 4=22.40%, 10=38.12%, 20=9.51%, 50=0.72% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.17%, sys=23.47%, ctx=6162, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23044: Sun Apr 14 05:05:51 2024 read: IOPS=99, BW=400KiB/s (409kB/s)(6052KiB/15137msec) clat (usec): min=36, max=62580, avg=4298.51, stdev=3958.52 lat (usec): min=37, max=62581, avg=4301.04, stdev=3958.21 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 75], 10.00th=[ 91], 20.00th=[ 159], | 30.00th=[ 2638], 40.00th=[ 3097], 50.00th=[ 3523], 60.00th=[ 4178], | 70.00th=[ 5276], 80.00th=[ 6849], 90.00th=[ 8717], 95.00th=[10552], | 99.00th=[17171], 99.50th=[19792], 99.90th=[32375], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 176, max= 520, per=24.97%, avg=395.97, stdev=78.90, samples=30 iops : min= 44, max= 130, avg=98.97, stdev=19.70, samples=30 write: IOPS=102, BW=412KiB/s (422kB/s)(6236KiB/15137msec) clat (usec): min=431, max=196679, avg=5473.70, stdev=7218.12 lat (usec): min=432, max=196681, avg=5475.21, stdev=7218.27 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 586], 10.00th=[ 660], 20.00th=[ 791], | 30.00th=[ 1237], 40.00th=[ 3097], 50.00th=[ 4293], 60.00th=[ 5276], | 70.00th=[ 6718], 80.00th=[ 8717], 90.00th=[ 12125], 95.00th=[ 15008], | 99.00th=[ 22676], 99.50th=[ 35914], 99.90th=[ 58459], 99.95th=[196084], | 99.99th=[196084] bw ( KiB/s): min= 192, max= 608, per=24.59%, avg=408.23, stdev=89.48, samples=30 iops : min= 48, max= 152, avg=102.03, stdev=22.36, samples=30 lat (usec) : 50=0.42%, 100=6.35%, 250=3.35%, 500=0.78%, 750=8.27% lat (usec) : 1000=4.79% lat (msec) : 2=4.26%, 4=23.99%, 10=37.01%, 20=9.83%, 50=0.85% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.17%, sys=23.20%, ctx=6057, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1586KiB/s (1624kB/s), 398KiB/s-403KiB/s (408kB/s-413kB/s), io=23.4MiB (24.6MB), run=14913-15137msec WRITE: bw=1661KiB/s (1701kB/s), 412KiB/s-426KiB/s (422kB/s-436kB/s), io=24.6MiB (25.7MB), run=14913-15137msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23045: Sun Apr 14 05:06:22 2024 read: IOPS=32, BW=129KiB/s (132kB/s)(5992KiB/46353msec) clat (usec): min=1481, max=174001, avg=4323.93, stdev=5984.89 lat (usec): min=1482, max=174002, avg=4325.07, stdev=5984.88 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1942], 10.00th=[ 2073], 20.00th=[ 2245], | 30.00th=[ 2442], 40.00th=[ 2638], 50.00th=[ 2868], 60.00th=[ 3294], | 70.00th=[ 4015], 80.00th=[ 5014], 90.00th=[ 8094], 95.00th=[ 11076], | 99.00th=[ 19006], 99.50th=[ 26084], 99.90th=[ 70779], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 31, max= 263, per=25.08%, avg=129.14, stdev=57.93, samples=92 iops : min= 7, max= 65, avg=32.16, stdev=14.47, samples=92 write: IOPS=33, BW=136KiB/s (139kB/s)(6296KiB/46353msec) clat (usec): min=11576, max=63091, avg=25281.42, stdev=9072.97 lat (usec): min=11577, max=63093, avg=25282.75, stdev=9072.98 clat percentiles (usec): | 1.00th=[13435], 5.00th=[14877], 10.00th=[15926], 20.00th=[17695], | 30.00th=[19530], 40.00th=[21627], 50.00th=[23462], 60.00th=[25297], | 70.00th=[27395], 80.00th=[31065], 90.00th=[38536], 95.00th=[44827], | 99.00th=[55313], 99.50th=[56886], 99.90th=[62653], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 56, max= 200, per=25.12%, avg=135.66, stdev=37.95, samples=92 iops : min= 14, max= 50, avg=33.79, stdev= 9.46, samples=92 lat (msec) : 2=3.22%, 4=30.79%, 10=11.69%, 20=19.24%, 50=33.76% lat (msec) : 100=1.27%, 250=0.03% cpu : usr=0.13%, sys=4.48%, ctx=3501, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23046: Sun Apr 14 05:06:22 2024 read: IOPS=31, BW=128KiB/s (131kB/s)(5940KiB/46539msec) clat (usec): min=1581, max=179100, avg=4348.17, stdev=6002.26 lat (usec): min=1582, max=179103, avg=4349.26, stdev=6002.27 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1942], 10.00th=[ 2040], 20.00th=[ 2245], | 30.00th=[ 2442], 40.00th=[ 2671], 50.00th=[ 2933], 60.00th=[ 3261], | 70.00th=[ 3982], 80.00th=[ 5014], 90.00th=[ 8356], 95.00th=[ 11469], | 99.00th=[ 19268], 99.50th=[ 22676], 99.90th=[ 69731], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 16, max= 256, per=24.76%, avg=127.50, stdev=57.40, samples=92 iops : min= 4, max= 64, avg=31.76, stdev=14.33, samples=92 write: IOPS=34, BW=136KiB/s (140kB/s)(6348KiB/46539msec) clat (usec): min=11886, max=98640, avg=25205.10, stdev=9522.58 lat (usec): min=11888, max=98642, avg=25206.58, stdev=9522.56 clat percentiles (usec): | 1.00th=[13566], 5.00th=[14746], 10.00th=[15533], 20.00th=[16909], | 30.00th=[19006], 40.00th=[21365], 50.00th=[23200], 60.00th=[25297], | 70.00th=[27657], 80.00th=[30802], 90.00th=[38536], 95.00th=[44827], | 99.00th=[55837], 99.50th=[57934], 99.90th=[84411], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 56, max= 184, per=25.09%, avg=135.50, stdev=38.11, samples=92 iops : min= 14, max= 46, avg=33.76, stdev= 9.52, samples=92 lat (msec) : 2=3.87%, 4=30.14%, 10=10.81%, 20=20.80%, 50=32.98% lat (msec) : 100=1.37%, 250=0.03% cpu : usr=0.11%, sys=4.47%, ctx=3493, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23047: Sun Apr 14 05:06:22 2024 read: IOPS=32, BW=130KiB/s (133kB/s)(6028KiB/46397msec) clat (usec): min=1510, max=53817, avg=4363.61, stdev=3963.02 lat (usec): min=1510, max=53818, avg=4364.73, stdev=3963.03 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1942], 10.00th=[ 2073], 20.00th=[ 2278], | 30.00th=[ 2540], 40.00th=[ 2737], 50.00th=[ 3064], 60.00th=[ 3523], | 70.00th=[ 4113], 80.00th=[ 5080], 90.00th=[ 8225], 95.00th=[11863], | 99.00th=[19792], 99.50th=[24511], 99.90th=[45876], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 40, max= 240, per=25.22%, avg=129.86, stdev=48.16, samples=92 iops : min= 10, max= 60, avg=32.36, stdev=12.05, samples=92 write: IOPS=33, BW=135KiB/s (138kB/s)(6260KiB/46397msec) clat (msec): min=12, max=208, avg=25.39, stdev=10.40 lat (msec): min=12, max=208, avg=25.39, stdev=10.40 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 22], 50.00th=[ 24], 60.00th=[ 26], | 70.00th=[ 28], 80.00th=[ 31], 90.00th=[ 39], 95.00th=[ 46], | 99.00th=[ 55], 99.50th=[ 62], 99.90th=[ 100], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 48, max= 192, per=24.93%, avg=134.64, stdev=38.61, samples=92 iops : min= 12, max= 48, avg=33.55, stdev= 9.65, samples=92 lat (msec) : 2=3.48%, 4=29.98%, 10=11.95%, 20=19.82%, 50=33.72% lat (msec) : 100=1.01%, 250=0.03% cpu : usr=0.17%, sys=4.50%, ctx=3513, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23048: Sun Apr 14 05:06:22 2024 read: IOPS=32, BW=131KiB/s (134kB/s)(6052KiB/46311msec) clat (usec): min=1519, max=57210, avg=4347.16, stdev=4272.36 lat (usec): min=1520, max=57211, avg=4348.66, stdev=4272.43 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1909], 10.00th=[ 2073], 20.00th=[ 2245], | 30.00th=[ 2442], 40.00th=[ 2671], 50.00th=[ 2933], 60.00th=[ 3326], | 70.00th=[ 3982], 80.00th=[ 4883], 90.00th=[ 8586], 95.00th=[12387], | 99.00th=[18220], 99.50th=[24249], 99.90th=[53740], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 16, max= 248, per=25.29%, avg=130.23, stdev=52.34, samples=92 iops : min= 4, max= 62, avg=32.47, stdev=13.11, samples=92 write: IOPS=33, BW=135KiB/s (138kB/s)(6236KiB/46311msec) clat (msec): min=11, max=196, avg=25.43, stdev=10.20 lat (msec): min=11, max=196, avg=25.43, stdev=10.20 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 22], 50.00th=[ 24], 60.00th=[ 26], | 70.00th=[ 28], 80.00th=[ 31], 90.00th=[ 39], 95.00th=[ 46], | 99.00th=[ 55], 99.50th=[ 62], 99.90th=[ 85], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 55, max= 192, per=24.87%, avg=134.32, stdev=38.03, samples=92 iops : min= 13, max= 48, avg=33.49, stdev= 9.57, samples=92 lat (msec) : 2=3.78%, 4=30.89%, 10=10.55%, 20=19.92%, 50=33.59% lat (msec) : 100=1.24%, 250=0.03% cpu : usr=0.14%, sys=4.50%, ctx=3487, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=516KiB/s (528kB/s), 128KiB/s-131KiB/s (131kB/s-134kB/s), io=23.4MiB (24.6MB), run=46311-46539msec WRITE: bw=540KiB/s (553kB/s), 135KiB/s-136KiB/s (138kB/s-140kB/s), io=24.6MiB (25.7MB), run=46311-46539msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23102: Sun Apr 14 05:06:29 2024 read: IOPS=70, BW=1123KiB/s (1150kB/s)(5920KiB/5270msec) clat (usec): min=45, max=33923, avg=10411.05, stdev=6084.79 lat (usec): min=46, max=33924, avg=10411.88, stdev=6084.80 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 80], 10.00th=[ 99], 20.00th=[ 4293], | 30.00th=[ 9372], 40.00th=[10552], 50.00th=[11338], 60.00th=[12387], | 70.00th=[13435], 80.00th=[14484], 90.00th=[16319], 95.00th=[18220], | 99.00th=[26346], 99.50th=[31589], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 830, max= 1312, per=25.92%, avg=1090.80, stdev=161.92, samples=10 iops : min= 51, max= 82, avg=68.00, stdev=10.36, samples=10 write: IOPS=75, BW=1208KiB/s (1237kB/s)(6368KiB/5270msec) clat (usec): min=588, max=36788, avg=3535.23, stdev=4121.69 lat (usec): min=590, max=36789, avg=3536.77, stdev=4121.62 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 758], 10.00th=[ 873], 20.00th=[ 979], | 30.00th=[ 1074], 40.00th=[ 1172], 50.00th=[ 1614], 60.00th=[ 3097], | 70.00th=[ 4080], 80.00th=[ 5735], 90.00th=[ 7635], 95.00th=[ 9896], | 99.00th=[22152], 99.50th=[30016], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 768, max= 1632, per=26.73%, avg=1183.50, stdev=278.84, samples=10 iops : min= 48, max= 102, avg=73.80, stdev=17.43, samples=10 lat (usec) : 50=0.26%, 100=4.69%, 250=4.17%, 500=0.13%, 750=2.21% lat (usec) : 1000=10.16% lat (msec) : 2=14.58%, 4=9.37%, 10=19.40%, 20=33.07%, 50=1.95% cpu : usr=0.15%, sys=21.52%, ctx=2038, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23103: Sun Apr 14 05:06:29 2024 read: IOPS=65, BW=1046KiB/s (1072kB/s)(5744KiB/5489msec) clat (usec): min=48, max=58701, avg=9830.41, stdev=6688.49 lat (usec): min=48, max=58701, avg=9831.67, stdev=6689.48 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 80], 10.00th=[ 89], 20.00th=[ 116], | 30.00th=[ 8848], 40.00th=[10159], 50.00th=[11207], 60.00th=[11994], | 70.00th=[13173], 80.00th=[14484], 90.00th=[16188], 95.00th=[17695], | 99.00th=[22938], 99.50th=[40109], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 320, max= 1341, per=23.49%, avg=988.30, stdev=325.83, samples=10 iops : min= 20, max= 83, avg=61.60, stdev=20.29, samples=10 write: IOPS=74, BW=1192KiB/s (1221kB/s)(6544KiB/5489msec) clat (usec): min=567, max=51043, avg=4768.88, stdev=6742.44 lat (usec): min=568, max=51044, avg=4771.58, stdev=6742.77 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 766], 10.00th=[ 848], 20.00th=[ 971], | 30.00th=[ 1057], 40.00th=[ 1156], 50.00th=[ 1844], 60.00th=[ 3490], | 70.00th=[ 4490], 80.00th=[ 6652], 90.00th=[11207], 95.00th=[20317], | 99.00th=[30802], 99.50th=[38011], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 512, max= 1600, per=25.28%, avg=1119.40, stdev=391.63, samples=10 iops : min= 32, max= 100, avg=69.80, stdev=24.46, samples=10 lat (usec) : 50=0.13%, 100=6.77%, 250=4.04%, 500=0.26%, 750=2.47% lat (usec) : 1000=9.90% lat (msec) : 2=14.58%, 4=7.55%, 10=19.14%, 20=31.38%, 50=3.52% lat (msec) : 100=0.26% cpu : usr=0.22%, sys=20.23%, ctx=1977, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23104: Sun Apr 14 05:06:29 2024 read: IOPS=68, BW=1099KiB/s (1126kB/s)(6256KiB/5691msec) clat (usec): min=52, max=39275, avg=10167.69, stdev=6334.89 lat (usec): min=52, max=39276, avg=10168.65, stdev=6335.05 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 71], 10.00th=[ 94], 20.00th=[ 124], | 30.00th=[ 9110], 40.00th=[10159], 50.00th=[11207], 60.00th=[11994], | 70.00th=[13435], 80.00th=[14615], 90.00th=[16712], 95.00th=[18482], | 99.00th=[24249], 99.50th=[38536], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 416, max= 1888, per=26.12%, avg=1099.27, stdev=381.44, samples=11 iops : min= 26, max= 118, avg=68.64, stdev=23.75, samples=11 write: IOPS=66, BW=1060KiB/s (1085kB/s)(6032KiB/5691msec) clat (usec): min=579, max=43294, avg=4526.47, stdev=6437.16 lat (usec): min=580, max=43296, avg=4528.02, stdev=6437.07 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 807], 10.00th=[ 865], 20.00th=[ 963], | 30.00th=[ 1074], 40.00th=[ 1172], 50.00th=[ 1647], 60.00th=[ 3425], | 70.00th=[ 4293], 80.00th=[ 6194], 90.00th=[10814], 95.00th=[16450], | 99.00th=[34341], 99.50th=[40633], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 480, max= 1280, per=23.19%, avg=1026.73, stdev=250.99, samples=11 iops : min= 30, max= 80, avg=64.09, stdev=15.71, samples=11 lat (usec) : 100=6.25%, 250=4.82%, 750=1.17%, 1000=10.42% lat (msec) : 2=14.06%, 4=7.16%, 10=19.40%, 20=33.33%, 50=3.39% cpu : usr=0.18%, sys=19.75%, ctx=2084, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23105: Sun Apr 14 05:06:29 2024 read: IOPS=70, BW=1121KiB/s (1148kB/s)(6032KiB/5381msec) clat (usec): min=51, max=47216, avg=10035.17, stdev=6192.06 lat (usec): min=52, max=47218, avg=10035.99, stdev=6192.07 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 69], 10.00th=[ 93], 20.00th=[ 165], | 30.00th=[ 9372], 40.00th=[10290], 50.00th=[11207], 60.00th=[11994], | 70.00th=[13042], 80.00th=[14353], 90.00th=[16450], 95.00th=[17695], | 99.00th=[23987], 99.50th=[29230], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 416, max= 1312, per=25.76%, avg=1084.00, stdev=262.74, samples=10 iops : min= 26, max= 82, avg=67.50, stdev=16.44, samples=10 write: IOPS=72, BW=1163KiB/s (1191kB/s)(6256KiB/5381msec) clat (usec): min=583, max=48452, avg=4063.61, stdev=5375.47 lat (usec): min=584, max=48454, avg=4065.11, stdev=5375.48 clat percentiles (usec): | 1.00th=[ 635], 5.00th=[ 766], 10.00th=[ 840], 20.00th=[ 947], | 30.00th=[ 1057], 40.00th=[ 1172], 50.00th=[ 1582], 60.00th=[ 3425], | 70.00th=[ 4293], 80.00th=[ 6194], 90.00th=[ 8455], 95.00th=[14222], | 99.00th=[28443], 99.50th=[34341], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 480, max= 1437, per=25.85%, avg=1144.70, stdev=276.36, samples=10 iops : min= 30, max= 89, avg=71.30, stdev=17.15, samples=10 lat (usec) : 100=6.12%, 250=4.04%, 500=0.13%, 750=2.47%, 1000=10.29% lat (msec) : 2=13.93%, 4=7.42%, 10=20.83%, 20=32.29%, 50=2.47% cpu : usr=0.17%, sys=20.80%, ctx=2014, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4209KiB/s (4310kB/s), 1046KiB/s-1123KiB/s (1072kB/s-1150kB/s), io=23.4MiB (24.5MB), run=5270-5691msec WRITE: bw=4428KiB/s (4534kB/s), 1060KiB/s-1208KiB/s (1085kB/s-1237kB/s), io=24.6MiB (25.8MB), run=5270-5691msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23098: Sun Apr 14 05:06:36 2024 read: IOPS=29, BW=473KiB/s (485kB/s)(5920KiB/12509msec) clat (usec): min=2029, max=43233, avg=5149.93, stdev=4787.62 lat (usec): min=2030, max=43234, avg=5151.07, stdev=4787.53 clat percentiles (usec): | 1.00th=[ 2147], 5.00th=[ 2409], 10.00th=[ 2540], 20.00th=[ 2769], | 30.00th=[ 2966], 40.00th=[ 3195], 50.00th=[ 3490], 60.00th=[ 3785], | 70.00th=[ 4424], 80.00th=[ 6128], 90.00th=[ 9241], 95.00th=[14746], | 99.00th=[29492], 99.50th=[33162], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 287, max= 704, per=24.98%, avg=473.44, stdev=144.04, samples=25 iops : min= 17, max= 44, avg=29.48, stdev= 9.07, samples=25 write: IOPS=31, BW=509KiB/s (521kB/s)(6368KiB/12509msec) clat (usec): min=13260, max=56322, avg=26612.91, stdev=7200.39 lat (usec): min=13262, max=56323, avg=26615.01, stdev=7200.46 clat percentiles (usec): | 1.00th=[14615], 5.00th=[16057], 10.00th=[17695], 20.00th=[20317], | 30.00th=[22676], 40.00th=[24249], 50.00th=[25560], 60.00th=[27395], | 70.00th=[29492], 80.00th=[32113], 90.00th=[36439], 95.00th=[40109], | 99.00th=[49021], 99.50th=[50594], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 320, max= 736, per=25.48%, avg=508.00, stdev=103.78, samples=25 iops : min= 20, max= 46, avg=31.64, stdev= 6.49, samples=25 lat (msec) : 4=30.73%, 10=12.89%, 20=13.28%, 50=42.71%, 100=0.39% cpu : usr=0.16%, sys=5.32%, ctx=862, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23099: Sun Apr 14 05:06:36 2024 read: IOPS=28, BW=455KiB/s (466kB/s)(5744KiB/12635msec) clat (usec): min=2049, max=27446, avg=4556.15, stdev=3413.35 lat (usec): min=2050, max=27447, avg=4557.78, stdev=3413.11 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2343], 10.00th=[ 2540], 20.00th=[ 2769], | 30.00th=[ 2933], 40.00th=[ 3163], 50.00th=[ 3425], 60.00th=[ 3654], | 70.00th=[ 4146], 80.00th=[ 4948], 90.00th=[ 8356], 95.00th=[12518], | 99.00th=[19530], 99.50th=[20841], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 223, max= 736, per=23.99%, avg=454.52, stdev=142.75, samples=25 iops : min= 13, max= 46, avg=28.32, stdev= 8.97, samples=25 write: IOPS=32, BW=518KiB/s (530kB/s)(6544KiB/12635msec) clat (usec): min=12912, max=62488, avg=26864.90, stdev=7846.09 lat (usec): min=12915, max=62490, avg=26867.37, stdev=7845.81 clat percentiles (usec): | 1.00th=[14746], 5.00th=[15926], 10.00th=[17695], 20.00th=[20055], | 30.00th=[21890], 40.00th=[23987], 50.00th=[25822], 60.00th=[28181], | 70.00th=[30540], 80.00th=[32900], 90.00th=[36963], 95.00th=[41157], | 99.00th=[50594], 99.50th=[52691], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 352, max= 672, per=25.84%, avg=515.32, stdev=94.13, samples=25 iops : min= 22, max= 42, avg=32.12, stdev= 5.93, samples=25 lat (msec) : 4=30.99%, 10=12.37%, 20=13.67%, 50=42.19%, 100=0.78% cpu : usr=0.11%, sys=5.25%, ctx=847, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23100: Sun Apr 14 05:06:36 2024 read: IOPS=31, BW=505KiB/s (517kB/s)(6256KiB/12396msec) clat (usec): min=2034, max=38242, avg=5035.43, stdev=4152.85 lat (usec): min=2035, max=38244, avg=5036.53, stdev=4152.80 clat percentiles (usec): | 1.00th=[ 2278], 5.00th=[ 2474], 10.00th=[ 2606], 20.00th=[ 2868], | 30.00th=[ 3097], 40.00th=[ 3392], 50.00th=[ 3556], 60.00th=[ 4015], | 70.00th=[ 4424], 80.00th=[ 5800], 90.00th=[ 8455], 95.00th=[14484], | 99.00th=[22676], 99.50th=[24511], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 224, max= 1120, per=26.81%, avg=507.96, stdev=201.61, samples=24 iops : min= 14, max= 70, avg=31.71, stdev=12.64, samples=24 write: IOPS=30, BW=487KiB/s (498kB/s)(6032KiB/12396msec) clat (usec): min=14111, max=75356, avg=27628.87, stdev=8148.08 lat (usec): min=14113, max=75358, avg=27631.02, stdev=8148.16 clat percentiles (usec): | 1.00th=[15270], 5.00th=[16581], 10.00th=[18482], 20.00th=[21103], | 30.00th=[22938], 40.00th=[24773], 50.00th=[26608], 60.00th=[28181], | 70.00th=[30540], 80.00th=[33424], 90.00th=[37487], 95.00th=[41681], | 99.00th=[53740], 99.50th=[62653], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 320, max= 608, per=24.14%, avg=481.29, stdev=84.69, samples=24 iops : min= 20, max= 38, avg=30.04, stdev= 5.33, samples=24 lat (msec) : 4=30.08%, 10=16.15%, 20=11.20%, 50=41.93%, 100=0.65% cpu : usr=0.06%, sys=5.45%, ctx=872, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23101: Sun Apr 14 05:06:36 2024 read: IOPS=30, BW=480KiB/s (492kB/s)(6032KiB/12558msec) clat (usec): min=2152, max=33845, avg=5076.82, stdev=4382.49 lat (usec): min=2153, max=33846, avg=5078.07, stdev=4382.39 clat percentiles (usec): | 1.00th=[ 2278], 5.00th=[ 2474], 10.00th=[ 2671], 20.00th=[ 2835], | 30.00th=[ 3097], 40.00th=[ 3294], 50.00th=[ 3556], 60.00th=[ 3818], | 70.00th=[ 4293], 80.00th=[ 5604], 90.00th=[ 8848], 95.00th=[15664], | 99.00th=[24511], 99.50th=[32113], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 191, max= 800, per=25.05%, avg=474.72, stdev=167.91, samples=25 iops : min= 11, max= 50, avg=29.56, stdev=10.51, samples=25 write: IOPS=31, BW=498KiB/s (510kB/s)(6256KiB/12558msec) clat (usec): min=14343, max=51301, avg=27190.14, stdev=7259.16 lat (usec): min=14368, max=51302, avg=27192.57, stdev=7259.03 clat percentiles (usec): | 1.00th=[15139], 5.00th=[16712], 10.00th=[18220], 20.00th=[20579], | 30.00th=[22938], 40.00th=[24249], 50.00th=[26346], 60.00th=[28181], | 70.00th=[30802], 80.00th=[33424], 90.00th=[36963], 95.00th=[39584], | 99.00th=[49546], 99.50th=[50070], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 384, max= 640, per=24.96%, avg=497.72, stdev=80.47, samples=25 iops : min= 24, max= 40, avg=31.00, stdev= 5.01, samples=25 lat (msec) : 4=32.29%, 10=12.50%, 20=12.11%, 50=42.84%, 100=0.26% cpu : usr=0.07%, sys=5.49%, ctx=854, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1896KiB/s (1941kB/s), 455KiB/s-505KiB/s (466kB/s-517kB/s), io=23.4MiB (24.5MB), run=12396-12635msec WRITE: bw=1994KiB/s (2042kB/s), 487KiB/s-518KiB/s (498kB/s-530kB/s), io=24.6MiB (25.8MB), run=12396-12635msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23121: Sun Apr 14 05:06:42 2024 read: IOPS=1, BW=1866KiB/s (1911kB/s)(7168KiB/3841msec) clat (msec): min=49, max=639, avg=427.39, stdev=242.72 lat (msec): min=49, max=639, avg=427.39, stdev=242.72 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 108], | 30.00th=[ 506], 40.00th=[ 506], 50.00th=[ 535], 60.00th=[ 575], | 70.00th=[ 575], 80.00th=[ 584], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 1706, max= 4096, per=56.87%, avg=2389.20, stdev=965.55, samples=5 iops : min= 1, max= 4, avg= 2.20, stdev= 1.10, samples=5 write: IOPS=1, BW=1333KiB/s (1365kB/s)(5120KiB/3841msec) clat (msec): min=26, max=653, avg=169.36, stdev=271.62 lat (msec): min=26, max=653, avg=169.46, stdev=271.64 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 53], 60.00th=[ 53], | 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 651], 95.00th=[ 651], | 99.00th=[ 651], 99.50th=[ 651], 99.90th=[ 651], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 1706, max= 4096, per=42.07%, avg=2474.50, stdev=1092.96, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 lat (msec) : 50=25.00%, 100=16.67%, 250=8.33%, 750=50.00% cpu : usr=0.10%, sys=19.11%, ctx=1404, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23122: Sun Apr 14 05:06:42 2024 read: IOPS=0, BW=420KiB/s (430kB/s)(2048KiB/4874msec) clat (msec): min=56, max=686, avg=371.53, stdev=445.32 lat (msec): min=56, max=686, avg=371.54, stdev=445.32 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 57], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 57], 60.00th=[ 684], | 70.00th=[ 684], 80.00th=[ 684], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 1712, max= 2048, per=44.75%, avg=1880.00, stdev=237.59, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2101KiB/s (2151kB/s)(10.0MiB/4874msec) clat (msec): min=16, max=1965, avg=412.89, stdev=665.45 lat (msec): min=16, max=1965, avg=412.96, stdev=665.45 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 22], | 30.00th=[ 27], 40.00th=[ 40], 50.00th=[ 46], 60.00th=[ 85], | 70.00th=[ 667], 80.00th=[ 667], 90.00th=[ 1167], 95.00th=[ 1972], | 99.00th=[ 1972], 99.50th=[ 1972], 99.90th=[ 1972], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 1712, max= 6131, per=62.44%, avg=3672.75, stdev=1826.77, samples=4 iops : min= 1, max= 5, avg= 3.00, stdev= 1.63, samples=4 lat (msec) : 20=8.33%, 50=33.33%, 100=25.00%, 750=16.67% cpu : usr=0.00%, sys=7.88%, ctx=363, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23123: Sun Apr 14 05:06:42 2024 read: IOPS=1, BW=1282KiB/s (1313kB/s)(6144KiB/4792msec) clat (usec): min=575, max=800716, avg=493395.45, stdev=366544.54 lat (usec): min=577, max=800718, avg=493397.29, stdev=366544.78 clat percentiles (usec): | 1.00th=[ 578], 5.00th=[ 578], 10.00th=[ 578], 20.00th=[ 49546], | 30.00th=[ 49546], 40.00th=[641729], 50.00th=[641729], 60.00th=[734004], | 70.00th=[734004], 80.00th=[734004], 90.00th=[801113], 95.00th=[801113], | 99.00th=[801113], 99.50th=[801113], 99.90th=[801113], 99.95th=[801113], | 99.99th=[801113] bw ( KiB/s): min= 1706, max= 4096, per=54.85%, avg=2304.25, stdev=1194.50, samples=4 iops : min= 1, max= 4, avg= 1.75, stdev= 1.50, samples=4 write: IOPS=1, BW=1282KiB/s (1313kB/s)(6144KiB/4792msec) clat (msec): min=22, max=1123, avg=305.00, stdev=440.75 lat (msec): min=22, max=1123, avg=305.09, stdev=440.76 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 34], | 30.00th=[ 34], 40.00th=[ 44], 50.00th=[ 44], 60.00th=[ 105], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 1116], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 1706, max= 4096, per=39.45%, avg=2320.40, stdev=1006.88, samples=5 iops : min= 1, max= 4, avg= 1.80, stdev= 1.30, samples=5 lat (usec) : 750=8.33% lat (msec) : 50=33.33%, 250=8.33%, 750=33.33%, 1000=8.33% cpu : usr=0.13%, sys=11.98%, ctx=1134, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23124: Sun Apr 14 05:06:42 2024 read: IOPS=1, BW=1996KiB/s (2044kB/s)(5120KiB/2565msec) clat (usec): min=384, max=590050, avg=231311.70, stdev=298025.67 lat (usec): min=384, max=590053, avg=231312.99, stdev=298026.05 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 383], 10.00th=[ 383], 20.00th=[ 383], | 30.00th=[ 750], 40.00th=[ 750], 50.00th=[ 43254], 60.00th=[ 43254], | 70.00th=[522191], 80.00th=[522191], 90.00th=[591397], 95.00th=[591397], | 99.00th=[591397], 99.50th=[591397], 99.90th=[591397], 99.95th=[591397], | 99.99th=[591397] bw ( KiB/s): min= 2048, max= 6144, per=97.50%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=2, BW=2795KiB/s (2862kB/s)(7168KiB/2565msec) clat (msec): min=28, max=1120, avg=200.97, stdev=406.11 lat (msec): min=28, max=1120, avg=201.04, stdev=406.09 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 30], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 51], 80.00th=[ 101], 90.00th=[ 1116], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 2048, max= 6144, per=81.24%, avg=4778.67, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 4.67, stdev= 2.31, samples=3 lat (usec) : 500=8.33%, 1000=8.33% lat (msec) : 50=41.67%, 100=16.67%, 750=16.67% cpu : usr=0.16%, sys=17.20%, ctx=547, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4202KiB/s (4303kB/s), 420KiB/s-1996KiB/s (430kB/s-2044kB/s), io=20.0MiB (20.0MB), run=2565-4874msec WRITE: bw=5883KiB/s (6024kB/s), 1282KiB/s-2795KiB/s (1313kB/s-2862kB/s), io=28.0MiB (29.4MB), run=2565-4874msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23125: Sun Apr 14 05:06:42 2024 read: IOPS=1, BW=1865KiB/s (1909kB/s)(7168KiB/3844msec) clat (msec): min=54, max=647, avg=416.51, stdev=235.71 lat (msec): min=54, max=647, avg=416.52, stdev=235.71 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 55], 10.00th=[ 55], 20.00th=[ 109], | 30.00th=[ 481], 40.00th=[ 481], 50.00th=[ 502], 60.00th=[ 542], | 70.00th=[ 542], 80.00th=[ 584], 90.00th=[ 651], 95.00th=[ 651], | 99.00th=[ 651], 99.50th=[ 651], 99.90th=[ 651], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 1715, max= 4096, per=57.74%, avg=2390.00, stdev=964.42, samples=5 iops : min= 1, max= 4, avg= 2.00, stdev= 1.22, samples=5 write: IOPS=1, BW=1332KiB/s (1364kB/s)(5120KiB/3844msec) clat (msec): min=64, max=538, avg=185.00, stdev=200.50 lat (msec): min=64, max=538, avg=185.07, stdev=200.50 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 65], 10.00th=[ 65], 20.00th=[ 65], | 30.00th=[ 70], 40.00th=[ 70], 50.00th=[ 106], 60.00th=[ 106], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 1715, max= 4096, per=41.31%, avg=2393.50, stdev=1145.80, samples=4 iops : min= 1, max= 4, avg= 2.00, stdev= 1.41, samples=4 lat (msec) : 100=25.00%, 250=25.00%, 500=8.33%, 750=41.67% cpu : usr=0.00%, sys=3.23%, ctx=61, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23126: Sun Apr 14 05:06:42 2024 read: IOPS=0, BW=414KiB/s (424kB/s)(2048KiB/4948msec) clat (msec): min=62, max=672, avg=367.55, stdev=430.76 lat (msec): min=62, max=672, avg=367.55, stdev=430.76 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 64], | 30.00th=[ 64], 40.00th=[ 64], 50.00th=[ 64], 60.00th=[ 676], | 70.00th=[ 676], 80.00th=[ 676], 90.00th=[ 676], 95.00th=[ 676], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 1721, max= 2048, per=45.53%, avg=1884.50, stdev=231.22, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2070KiB/s (2119kB/s)(10.0MiB/4948msec) clat (msec): min=38, max=1409, avg=421.10, stdev=526.89 lat (msec): min=38, max=1409, avg=421.17, stdev=526.89 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 42], | 30.00th=[ 46], 40.00th=[ 55], 50.00th=[ 66], 60.00th=[ 71], | 70.00th=[ 651], 80.00th=[ 651], 90.00th=[ 1200], 95.00th=[ 1418], | 99.00th=[ 1418], 99.50th=[ 1418], 99.90th=[ 1418], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 2048, max= 5163, per=63.61%, avg=3685.75, stdev=1303.22, samples=4 iops : min= 2, max= 5, avg= 3.50, stdev= 1.29, samples=4 lat (msec) : 50=25.00%, 100=33.33%, 750=25.00% cpu : usr=0.02%, sys=2.69%, ctx=50, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23127: Sun Apr 14 05:06:42 2024 read: IOPS=1, BW=1263KiB/s (1293kB/s)(6144KiB/4866msec) clat (msec): min=18, max=1449, avg=370.35, stdev=578.22 lat (msec): min=18, max=1449, avg=370.35, stdev=578.22 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 30], | 30.00th=[ 30], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 57], | 70.00th=[ 625], 80.00th=[ 625], 90.00th=[ 1452], 95.00th=[ 1452], | 99.00th=[ 1452], 99.50th=[ 1452], 99.90th=[ 1452], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 1721, max= 4096, per=74.52%, avg=3084.33, stdev=1225.93, samples=3 iops : min= 1, max= 4, avg= 2.67, stdev= 1.53, samples=3 write: IOPS=1, BW=1263KiB/s (1293kB/s)(6144KiB/4866msec) clat (msec): min=29, max=1117, avg=440.33, stdev=427.26 lat (msec): min=29, max=1117, avg=440.42, stdev=427.25 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 68], | 30.00th=[ 68], 40.00th=[ 190], 50.00th=[ 190], 60.00th=[ 514], | 70.00th=[ 726], 80.00th=[ 726], 90.00th=[ 1116], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 2048, max= 4096, per=50.20%, avg=2908.50, stdev=1028.87, samples=4 iops : min= 2, max= 4, avg= 2.75, stdev= 0.96, samples=4 lat (msec) : 20=8.33%, 50=25.00%, 100=16.67%, 250=8.33%, 750=25.00% cpu : usr=0.00%, sys=2.63%, ctx=60, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23128: Sun Apr 14 05:06:42 2024 read: IOPS=1, BW=1833KiB/s (1877kB/s)(5120KiB/2793msec) clat (msec): min=20, max=606, avg=148.42, stdev=256.13 lat (msec): min=20, max=606, avg=148.42, stdev=256.13 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 27], 40.00th=[ 27], 50.00th=[ 40], 60.00th=[ 40], | 70.00th=[ 50], 80.00th=[ 50], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 2048, max= 4096, per=74.22%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2566KiB/s (2628kB/s)(7168KiB/2793msec) clat (msec): min=33, max=1128, avg=292.77, stdev=419.08 lat (msec): min=33, max=1128, avg=292.84, stdev=419.09 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 39], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 44], 60.00th=[ 176], | 70.00th=[ 176], 80.00th=[ 584], 90.00th=[ 1133], 95.00th=[ 1133], | 99.00th=[ 1133], 99.50th=[ 1133], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 2048, max= 4096, per=47.13%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 lat (msec) : 50=66.67%, 250=8.33%, 750=16.67% cpu : usr=0.00%, sys=4.08%, ctx=51, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4139KiB/s (4238kB/s), 414KiB/s-1865KiB/s (424kB/s-1909kB/s), io=20.0MiB (20.0MB), run=2793-4948msec WRITE: bw=5795KiB/s (5934kB/s), 1263KiB/s-2566KiB/s (1293kB/s-2628kB/s), io=28.0MiB (29.4MB), run=2793-4948msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23137: Sun Apr 14 05:06:45 2024 read: IOPS=2, BW=9.90MiB/s (10.4MB/s)(12.0MiB/1212msec) clat (msec): min=24, max=1099, avg=400.16, stdev=605.94 lat (msec): min=24, max=1099, avg=400.16, stdev=605.94 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 25], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 77], | 70.00th=[ 1099], 80.00th=[ 1099], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 8192, max= 8192, per=96.21%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=33.33% cpu : usr=0.00%, sys=72.83%, ctx=192, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23138: Sun Apr 14 05:06:45 2024 read: IOPS=0, BW=2129KiB/s (2180kB/s)(4096KiB/1924msec) clat (nsec): min=96411k, max=96411k, avg=96410700.00, stdev= 0.00 lat (nsec): min=96415k, max=96415k, avg=96414645.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[95945], 5.00th=[95945], 10.00th=[95945], 20.00th=[95945], | 30.00th=[95945], 40.00th=[95945], 50.00th=[95945], 60.00th=[95945], | 70.00th=[95945], 80.00th=[95945], 90.00th=[95945], 95.00th=[95945], | 99.00th=[95945], 99.50th=[95945], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 8192, max= 8192, per=96.21%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4258KiB/s (4360kB/s)(8192KiB/1924msec) clat (msec): min=175, max=1638, avg=907.34, stdev=1034.61 lat (msec): min=176, max=1639, avg=907.67, stdev=1034.66 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 176], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 176], 50.00th=[ 176], 60.00th=[ 1636], | 70.00th=[ 1636], 80.00th=[ 1636], 90.00th=[ 1636], 95.00th=[ 1636], | 99.00th=[ 1636], 99.50th=[ 1636], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.05%, sys=14.87%, ctx=581, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23139: Sun Apr 14 05:06:45 2024 write: IOPS=1, BW=6083KiB/s (6229kB/s)(12.0MiB/2020msec) clat (msec): min=268, max=1026, avg=665.80, stdev=380.40 lat (msec): min=269, max=1026, avg=666.26, stdev=380.19 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 271], 10.00th=[ 271], 20.00th=[ 271], | 30.00th=[ 271], 40.00th=[ 701], 50.00th=[ 701], 60.00th=[ 701], | 70.00th=[ 1028], 80.00th=[ 1028], 90.00th=[ 1028], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 8192, max= 8192, per=50.50%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=14.91%, ctx=55, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23140: Sun Apr 14 05:06:45 2024 write: IOPS=1, BW=6387KiB/s (6540kB/s)(12.0MiB/1924msec) clat (msec): min=132, max=1234, avg=634.93, stdev=557.65 lat (msec): min=132, max=1235, avg=635.38, stdev=557.92 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 133], | 30.00th=[ 133], 40.00th=[ 542], 50.00th=[ 542], 60.00th=[ 542], | 70.00th=[ 1234], 80.00th=[ 1234], 90.00th=[ 1234], 95.00th=[ 1234], | 99.00th=[ 1234], 99.50th=[ 1234], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 8192, max= 8192, per=50.50%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.05%, sys=17.00%, ctx=37, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8516KiB/s (8720kB/s), 2129KiB/s-9.90MiB/s (2180kB/s-10.4MB/s), io=16.0MiB (16.8MB), run=1212-1924msec WRITE: bw=15.8MiB/s (16.6MB/s), 4258KiB/s-6387KiB/s (4360kB/s-6540kB/s), io=32.0MiB (33.6MB), run=1924-2020msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23142: Sun Apr 14 05:06:45 2024 read: IOPS=1, BW=6003KiB/s (6147kB/s)(12.0MiB/2047msec) clat (msec): min=351, max=1157, avg=681.79, stdev=422.51 lat (msec): min=351, max=1157, avg=681.79, stdev=422.51 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 351], 10.00th=[ 351], 20.00th=[ 351], | 30.00th=[ 351], 40.00th=[ 535], 50.00th=[ 535], 60.00th=[ 535], | 70.00th=[ 1167], 80.00th=[ 1167], 90.00th=[ 1167], 95.00th=[ 1167], | 99.00th=[ 1167], 99.50th=[ 1167], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=4.74%, ctx=39, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23143: Sun Apr 14 05:06:45 2024 read: IOPS=0, BW=1817KiB/s (1861kB/s)(4096KiB/2254msec) clat (nsec): min=309917k, max=309917k, avg=309916679.00, stdev= 0.00 lat (nsec): min=309920k, max=309920k, avg=309919516.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 309], 10.00th=[ 309], 20.00th=[ 309], | 30.00th=[ 309], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3634KiB/s (3722kB/s)(8192KiB/2254msec) clat (msec): min=78, max=1861, avg=970.42, stdev=1260.86 lat (msec): min=79, max=1862, avg=970.75, stdev=1260.85 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 80], 10.00th=[ 80], 20.00th=[ 80], | 30.00th=[ 80], 40.00th=[ 80], 50.00th=[ 80], 60.00th=[ 1854], | 70.00th=[ 1854], 80.00th=[ 1854], 90.00th=[ 1854], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.04%, sys=4.93%, ctx=126, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23144: Sun Apr 14 05:06:45 2024 write: IOPS=1, BW=5821KiB/s (5961kB/s)(12.0MiB/2111msec) clat (msec): min=419, max=1183, avg=702.73, stdev=418.36 lat (msec): min=419, max=1183, avg=703.11, stdev=418.33 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 422], 10.00th=[ 422], 20.00th=[ 422], | 30.00th=[ 422], 40.00th=[ 506], 50.00th=[ 506], 60.00th=[ 506], | 70.00th=[ 1183], 80.00th=[ 1183], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 8175, max= 8192, per=57.07%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.05%, sys=5.07%, ctx=23, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23145: Sun Apr 14 05:06:45 2024 write: IOPS=1, BW=5378KiB/s (5507kB/s)(12.0MiB/2285msec) clat (msec): min=148, max=1591, avg=760.77, stdev=745.79 lat (msec): min=149, max=1591, avg=761.12, stdev=745.85 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 150], 10.00th=[ 150], 20.00th=[ 150], | 30.00th=[ 150], 40.00th=[ 542], 50.00th=[ 542], 60.00th=[ 542], | 70.00th=[ 1586], 80.00th=[ 1586], 90.00th=[ 1586], 95.00th=[ 1586], | 99.00th=[ 1586], 99.50th=[ 1586], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 8192, max= 8192, per=57.13%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.04%, sys=7.62%, ctx=130, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7269KiB/s (7443kB/s), 1817KiB/s-6003KiB/s (1861kB/s-6147kB/s), io=16.0MiB (16.8MB), run=2047-2254msec WRITE: bw=14.0MiB/s (14.7MB/s), 3634KiB/s-5821KiB/s (3722kB/s-5961kB/s), io=32.0MiB (33.6MB), run=2111-2285msec