== sanity test 398b: DIO and buffer IO race ============== 19:11:26 (1712272286) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.01092 s, 49.8 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1234: Thu Apr 4 19:11:41 2024 read: IOPS=116, BW=466KiB/s (477kB/s)(5992KiB/12857msec) clat (usec): min=43, max=161451, avg=3905.48, stdev=5151.05 lat (usec): min=43, max=161452, avg=3906.42, stdev=5150.98 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 56], 10.00th=[ 68], 20.00th=[ 1958], | 30.00th=[ 2507], 40.00th=[ 2868], 50.00th=[ 3294], 60.00th=[ 3818], | 70.00th=[ 4621], 80.00th=[ 5866], 90.00th=[ 7570], 95.00th=[ 8979], | 99.00th=[ 13173], 99.50th=[ 17695], 99.90th=[ 42206], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 328, max= 688, per=24.67%, avg=458.44, stdev=89.23, samples=25 iops : min= 82, max= 172, avg=114.52, stdev=22.34, samples=25 write: IOPS=122, BW=490KiB/s (501kB/s)(6296KiB/12857msec) clat (usec): min=391, max=59228, avg=4434.10, stdev=4618.66 lat (usec): min=392, max=59229, avg=4435.21, stdev=4618.76 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 486], 10.00th=[ 529], 20.00th=[ 709], | 30.00th=[ 947], 40.00th=[ 2343], 50.00th=[ 3589], 60.00th=[ 4424], | 70.00th=[ 5538], 80.00th=[ 7177], 90.00th=[10028], 95.00th=[13042], | 99.00th=[17171], 99.50th=[24511], 99.90th=[56361], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 336, max= 752, per=24.64%, avg=479.56, stdev=95.27, samples=25 iops : min= 84, max= 188, avg=119.80, stdev=23.85, samples=25 lat (usec) : 50=0.72%, 100=6.80%, 250=1.56%, 500=3.68%, 750=7.85% lat (usec) : 1000=4.43% lat (msec) : 2=4.65%, 4=29.10%, 10=34.77%, 20=6.02%, 50=0.33% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.17%, sys=23.28%, ctx=5730, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1235: Thu Apr 4 19:11:41 2024 read: IOPS=115, BW=462KiB/s (473kB/s)(5940KiB/12863msec) clat (usec): min=42, max=57007, avg=3731.65, stdev=3272.67 lat (usec): min=43, max=57007, avg=3732.99, stdev=3272.63 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 57], 10.00th=[ 71], 20.00th=[ 1926], | 30.00th=[ 2409], 40.00th=[ 2769], 50.00th=[ 3163], 60.00th=[ 3621], | 70.00th=[ 4555], 80.00th=[ 5800], 90.00th=[ 7373], 95.00th=[ 8717], | 99.00th=[12125], 99.50th=[16188], 99.90th=[37487], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 272, max= 726, per=24.60%, avg=457.08, stdev=100.11, samples=25 iops : min= 68, max= 181, avg=114.20, stdev=24.95, samples=25 write: IOPS=123, BW=494KiB/s (505kB/s)(6348KiB/12863msec) clat (usec): min=422, max=160043, avg=4595.72, stdev=6151.23 lat (usec): min=423, max=160044, avg=4596.96, stdev=6150.92 clat percentiles (usec): | 1.00th=[ 453], 5.00th=[ 486], 10.00th=[ 523], 20.00th=[ 668], | 30.00th=[ 938], 40.00th=[ 2507], 50.00th=[ 3654], 60.00th=[ 4490], | 70.00th=[ 5604], 80.00th=[ 7111], 90.00th=[ 10028], 95.00th=[ 12256], | 99.00th=[ 20579], 99.50th=[ 30278], 99.90th=[ 62653], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 328, max= 678, per=24.74%, avg=481.36, stdev=91.24, samples=25 iops : min= 82, max= 169, avg=120.28, stdev=22.74, samples=25 lat (usec) : 50=0.62%, 100=6.80%, 250=1.46%, 500=3.65%, 750=8.92% lat (usec) : 1000=3.65% lat (msec) : 2=4.30%, 4=29.75%, 10=34.57%, 20=5.60%, 50=0.59% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.17%, sys=23.15%, ctx=5799, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1236: Thu Apr 4 19:11:41 2024 read: IOPS=116, BW=467KiB/s (478kB/s)(6028KiB/12918msec) clat (usec): min=43, max=44313, avg=3724.23, stdev=3110.88 lat (usec): min=43, max=44313, avg=3724.85, stdev=3110.89 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 56], 10.00th=[ 68], 20.00th=[ 1827], | 30.00th=[ 2376], 40.00th=[ 2802], 50.00th=[ 3294], 60.00th=[ 3785], | 70.00th=[ 4555], 80.00th=[ 5735], 90.00th=[ 7439], 95.00th=[ 8586], | 99.00th=[11731], 99.50th=[16057], 99.90th=[31065], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 328, max= 736, per=24.62%, avg=457.48, stdev=85.56, samples=25 iops : min= 82, max= 184, avg=114.28, stdev=21.43, samples=25 write: IOPS=121, BW=485KiB/s (496kB/s)(6260KiB/12918msec) clat (usec): min=380, max=163570, avg=4650.57, stdev=6079.33 lat (usec): min=380, max=163571, avg=4651.50, stdev=6079.22 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 494], 10.00th=[ 537], 20.00th=[ 685], | 30.00th=[ 1020], 40.00th=[ 2638], 50.00th=[ 3785], 60.00th=[ 4621], | 70.00th=[ 5932], 80.00th=[ 7439], 90.00th=[ 9765], 95.00th=[ 11994], | 99.00th=[ 21627], 99.50th=[ 31327], 99.90th=[ 48497], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 368, max= 696, per=24.31%, avg=473.08, stdev=79.49, samples=25 iops : min= 92, max= 174, avg=118.20, stdev=19.84, samples=25 lat (usec) : 50=0.75%, 100=7.03%, 250=1.40%, 500=2.93%, 750=8.92% lat (usec) : 1000=3.48% lat (msec) : 2=4.95%, 4=28.16%, 10=36.62%, 20=5.01%, 50=0.72% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=23.44%, ctx=5765, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1237: Thu Apr 4 19:11:41 2024 read: IOPS=117, BW=469KiB/s (480kB/s)(6052KiB/12899msec) clat (usec): min=43, max=60762, avg=3779.50, stdev=3646.84 lat (usec): min=43, max=60763, avg=3780.21, stdev=3646.82 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 56], 10.00th=[ 68], 20.00th=[ 1647], | 30.00th=[ 2442], 40.00th=[ 2802], 50.00th=[ 3228], 60.00th=[ 3752], | 70.00th=[ 4424], 80.00th=[ 5669], 90.00th=[ 7504], 95.00th=[ 9110], | 99.00th=[14222], 99.50th=[21365], 99.90th=[52167], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min= 304, max= 648, per=24.57%, avg=456.52, stdev=94.78, samples=25 iops : min= 76, max= 162, avg=114.04, stdev=23.69, samples=25 write: IOPS=120, BW=483KiB/s (495kB/s)(6236KiB/12899msec) clat (usec): min=427, max=186158, avg=4590.18, stdev=6473.39 lat (usec): min=428, max=186159, avg=4590.94, stdev=6473.40 clat percentiles (usec): | 1.00th=[ 453], 5.00th=[ 486], 10.00th=[ 529], 20.00th=[ 701], | 30.00th=[ 1172], 40.00th=[ 2671], 50.00th=[ 3654], 60.00th=[ 4490], | 70.00th=[ 5604], 80.00th=[ 7111], 90.00th=[ 9634], 95.00th=[ 12125], | 99.00th=[ 23462], 99.50th=[ 29230], 99.90th=[ 51643], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 320, max= 712, per=24.17%, avg=470.28, stdev=79.19, samples=25 iops : min= 80, max= 178, avg=117.48, stdev=19.87, samples=25 lat (usec) : 50=0.72%, 100=7.49%, 250=1.30%, 500=3.74%, 750=7.68% lat (usec) : 1000=3.29% lat (msec) : 2=5.18%, 4=29.75%, 10=34.90%, 20=5.05%, 50=0.78% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.16%, sys=22.86%, ctx=5735, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1859KiB/s (1903kB/s), 462KiB/s-469KiB/s (473kB/s-480kB/s), io=23.4MiB (24.6MB), run=12857-12918msec WRITE: bw=1946KiB/s (1993kB/s), 483KiB/s-494KiB/s (495kB/s-505kB/s), io=24.6MiB (25.7MB), run=12857-12918msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1229: Thu Apr 4 19:12:13 2024 read: IOPS=33, BW=135KiB/s (139kB/s)(5992KiB/44263msec) clat (usec): min=1469, max=163619, avg=4103.93, stdev=5502.91 lat (usec): min=1469, max=163621, avg=4104.68, stdev=5502.93 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1811], 10.00th=[ 1893], 20.00th=[ 2024], | 30.00th=[ 2147], 40.00th=[ 2343], 50.00th=[ 2573], 60.00th=[ 3228], | 70.00th=[ 4228], 80.00th=[ 5800], 90.00th=[ 7177], 95.00th=[ 8848], | 99.00th=[ 21627], 99.50th=[ 28443], 99.90th=[ 47449], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 40, max= 256, per=25.02%, avg=135.10, stdev=50.29, samples=88 iops : min= 10, max= 64, avg=33.72, stdev=12.57, samples=88 write: IOPS=35, BW=142KiB/s (146kB/s)(6296KiB/44263msec) clat (usec): min=11785, max=76468, avg=24181.23, stdev=8885.04 lat (usec): min=11786, max=76469, avg=24182.16, stdev=8885.06 clat percentiles (usec): | 1.00th=[13435], 5.00th=[14615], 10.00th=[15270], 20.00th=[16909], | 30.00th=[18482], 40.00th=[20317], 50.00th=[22676], 60.00th=[24511], | 70.00th=[26084], 80.00th=[28705], 90.00th=[36439], 95.00th=[43254], | 99.00th=[54789], 99.50th=[57410], 99.90th=[67634], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 56, max= 192, per=25.11%, avg=142.10, stdev=37.37, samples=88 iops : min= 14, max= 48, avg=35.47, stdev= 9.34, samples=88 lat (msec) : 2=8.43%, 4=24.90%, 10=13.61%, 20=20.74%, 50=31.25% lat (msec) : 100=1.04%, 250=0.03% cpu : usr=0.09%, sys=4.43%, ctx=3916, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1230: Thu Apr 4 19:12:13 2024 read: IOPS=33, BW=134KiB/s (137kB/s)(5940KiB/44390msec) clat (usec): min=1469, max=163086, avg=4080.35, stdev=5309.79 lat (usec): min=1470, max=163087, avg=4081.13, stdev=5309.81 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1795], 10.00th=[ 1876], 20.00th=[ 2008], | 30.00th=[ 2147], 40.00th=[ 2343], 50.00th=[ 2573], 60.00th=[ 3294], | 70.00th=[ 4293], 80.00th=[ 5735], 90.00th=[ 7111], 95.00th=[ 9241], | 99.00th=[ 17695], 99.50th=[ 28443], 99.90th=[ 36439], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 16, max= 256, per=24.75%, avg=133.62, stdev=54.82, samples=88 iops : min= 4, max= 64, avg=33.33, stdev=13.69, samples=88 write: IOPS=35, BW=143KiB/s (146kB/s)(6348KiB/44390msec) clat (usec): min=11757, max=67122, avg=24117.77, stdev=8628.24 lat (usec): min=11758, max=67123, avg=24118.70, stdev=8628.25 clat percentiles (usec): | 1.00th=[13173], 5.00th=[14484], 10.00th=[15401], 20.00th=[16909], | 30.00th=[18482], 40.00th=[20317], 50.00th=[22676], 60.00th=[24511], | 70.00th=[26346], 80.00th=[28967], 90.00th=[35914], 95.00th=[42206], | 99.00th=[54264], 99.50th=[56361], 99.90th=[65799], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 72, max= 184, per=25.13%, avg=142.26, stdev=35.91, samples=88 iops : min= 18, max= 46, avg=35.49, stdev= 8.94, samples=88 lat (msec) : 2=9.18%, 4=23.21%, 10=13.93%, 20=21.35%, 50=31.58% lat (msec) : 100=0.72%, 250=0.03% cpu : usr=0.10%, sys=4.39%, ctx=3925, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1231: Thu Apr 4 19:12:13 2024 read: IOPS=34, BW=136KiB/s (139kB/s)(6028KiB/44322msec) clat (usec): min=1473, max=42456, avg=4190.24, stdev=3588.20 lat (usec): min=1473, max=42457, avg=4190.99, stdev=3588.20 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1795], 10.00th=[ 1876], 20.00th=[ 2024], | 30.00th=[ 2147], 40.00th=[ 2311], 50.00th=[ 2671], 60.00th=[ 3523], | 70.00th=[ 4883], 80.00th=[ 6063], 90.00th=[ 7635], 95.00th=[10159], | 99.00th=[17695], 99.50th=[21890], 99.90th=[40633], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 40, max= 296, per=25.16%, avg=135.88, stdev=52.40, samples=88 iops : min= 10, max= 74, avg=33.94, stdev=13.09, samples=88 write: IOPS=35, BW=141KiB/s (145kB/s)(6260KiB/44322msec) clat (msec): min=12, max=202, avg=24.25, stdev= 9.87 lat (msec): min=12, max=202, avg=24.25, stdev= 9.87 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 25], | 70.00th=[ 27], 80.00th=[ 29], 90.00th=[ 36], 95.00th=[ 44], | 99.00th=[ 55], 99.50th=[ 60], 99.90th=[ 65], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 56, max= 208, per=24.89%, avg=140.87, stdev=37.85, samples=88 iops : min= 14, max= 52, avg=35.19, stdev= 9.44, samples=88 lat (msec) : 2=9.15%, 4=22.66%, 10=14.78%, 20=21.52%, 50=30.89% lat (msec) : 100=0.98%, 250=0.03% cpu : usr=0.10%, sys=4.48%, ctx=3938, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1232: Thu Apr 4 19:12:13 2024 read: IOPS=34, BW=137KiB/s (140kB/s)(6052KiB/44231msec) clat (usec): min=1462, max=43252, avg=4039.70, stdev=3514.83 lat (usec): min=1463, max=43252, avg=4040.45, stdev=3514.86 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1762], 10.00th=[ 1860], 20.00th=[ 1991], | 30.00th=[ 2114], 40.00th=[ 2311], 50.00th=[ 2573], 60.00th=[ 3326], | 70.00th=[ 4490], 80.00th=[ 5866], 90.00th=[ 7177], 95.00th=[ 9634], | 99.00th=[16581], 99.50th=[28705], 99.90th=[34341], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 40, max= 272, per=25.38%, avg=137.07, stdev=54.05, samples=88 iops : min= 10, max= 68, avg=34.25, stdev=13.53, samples=88 write: IOPS=35, BW=141KiB/s (144kB/s)(6236KiB/44231msec) clat (msec): min=12, max=183, avg=24.42, stdev= 9.53 lat (msec): min=12, max=183, avg=24.42, stdev= 9.53 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 20], 40.00th=[ 22], 50.00th=[ 23], 60.00th=[ 25], | 70.00th=[ 27], 80.00th=[ 29], 90.00th=[ 36], 95.00th=[ 44], | 99.00th=[ 53], 99.50th=[ 59], 99.90th=[ 77], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 56, max= 199, per=24.88%, avg=140.80, stdev=36.22, samples=88 iops : min= 14, max= 49, avg=35.18, stdev= 9.05, samples=88 lat (msec) : 2=10.12%, 4=22.49%, 10=14.42%, 20=19.95%, 50=32.16% lat (msec) : 100=0.81%, 250=0.03% cpu : usr=0.09%, sys=4.41%, ctx=4004, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=541KiB/s (554kB/s), 134KiB/s-137KiB/s (137kB/s-140kB/s), io=23.4MiB (24.6MB), run=44231-44390msec WRITE: bw=566KiB/s (580kB/s), 141KiB/s-143KiB/s (144kB/s-146kB/s), io=24.6MiB (25.7MB), run=44231-44390msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1286: Thu Apr 4 19:12:19 2024 read: IOPS=73, BW=1183KiB/s (1211kB/s)(5920KiB/5006msec) clat (usec): min=46, max=141813, avg=9936.57, stdev=8565.28 lat (usec): min=46, max=141814, avg=9937.24, stdev=8565.32 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 61], 10.00th=[ 78], 20.00th=[ 7832], | 30.00th=[ 8979], 40.00th=[ 9765], 50.00th=[ 10421], 60.00th=[ 11076], | 70.00th=[ 11863], 80.00th=[ 12780], 90.00th=[ 14353], 95.00th=[ 16319], | 99.00th=[ 28705], 99.50th=[ 31327], 99.90th=[141558], 99.95th=[141558], | 99.99th=[141558] bw ( KiB/s): min= 576, max= 1600, per=25.04%, avg=1177.00, stdev=268.87, samples=10 iops : min= 36, max= 100, avg=73.40, stdev=16.79, samples=10 write: IOPS=79, BW=1272KiB/s (1303kB/s)(6368KiB/5006msec) clat (usec): min=588, max=40479, avg=3321.77, stdev=4344.63 lat (usec): min=589, max=40481, avg=3323.55, stdev=4344.30 clat percentiles (usec): | 1.00th=[ 635], 5.00th=[ 693], 10.00th=[ 734], 20.00th=[ 791], | 30.00th=[ 857], 40.00th=[ 1074], 50.00th=[ 1795], 60.00th=[ 2671], | 70.00th=[ 3458], 80.00th=[ 5145], 90.00th=[ 7177], 95.00th=[10159], | 99.00th=[23725], 99.50th=[33817], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 640, max= 1760, per=25.62%, avg=1266.70, stdev=364.13, samples=10 iops : min= 40, max= 110, avg=79.00, stdev=22.84, samples=10 lat (usec) : 50=0.13%, 100=6.38%, 250=1.30%, 500=0.13%, 750=7.03% lat (usec) : 1000=12.50% lat (msec) : 2=7.29%, 4=11.20%, 10=23.83%, 20=28.78%, 50=1.30% lat (msec) : 250=0.13% cpu : usr=0.00%, sys=23.70%, ctx=2177, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1287: Thu Apr 4 19:12:19 2024 read: IOPS=70, BW=1127KiB/s (1154kB/s)(5744KiB/5096msec) clat (usec): min=48, max=39445, avg=9196.64, stdev=5700.59 lat (usec): min=48, max=39446, avg=9197.33, stdev=5700.54 clat percentiles (usec): | 1.00th=[ 54], 5.00th=[ 61], 10.00th=[ 72], 20.00th=[ 145], | 30.00th=[ 8291], 40.00th=[ 9634], 50.00th=[10290], 60.00th=[11207], | 70.00th=[11863], 80.00th=[12780], 90.00th=[14222], 95.00th=[16450], | 99.00th=[22938], 99.50th=[37487], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 96, max= 1696, per=23.49%, avg=1103.80, stdev=438.62, samples=10 iops : min= 6, max= 106, avg=68.90, stdev=27.44, samples=10 write: IOPS=80, BW=1284KiB/s (1315kB/s)(6544KiB/5096msec) clat (usec): min=586, max=211642, avg=4366.71, stdev=11634.03 lat (usec): min=586, max=211642, avg=4368.12, stdev=11634.01 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 685], 10.00th=[ 734], 20.00th=[ 799], | 30.00th=[ 865], 40.00th=[ 1045], 50.00th=[ 2180], 60.00th=[ 2802], | 70.00th=[ 3752], 80.00th=[ 5669], 90.00th=[ 8094], 95.00th=[ 15270], | 99.00th=[ 26870], 99.50th=[ 34341], 99.90th=[210764], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 352, max= 2368, per=25.30%, avg=1250.90, stdev=527.51, samples=10 iops : min= 22, max= 148, avg=78.10, stdev=32.99, samples=10 lat (usec) : 50=0.13%, 100=7.94%, 250=1.56%, 500=0.13%, 750=6.64% lat (usec) : 1000=13.80% lat (msec) : 2=5.34%, 4=12.89%, 10=22.53%, 20=26.17%, 50=2.73% lat (msec) : 250=0.13% cpu : usr=0.18%, sys=19.69%, ctx=2021, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1288: Thu Apr 4 19:12:19 2024 read: IOPS=77, BW=1233KiB/s (1263kB/s)(6256KiB/5072msec) clat (usec): min=47, max=55644, avg=9567.36, stdev=5808.07 lat (usec): min=47, max=55645, avg=9568.02, stdev=5808.10 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 62], 10.00th=[ 73], 20.00th=[ 7504], | 30.00th=[ 8848], 40.00th=[ 9372], 50.00th=[10028], 60.00th=[10814], | 70.00th=[11863], 80.00th=[12911], 90.00th=[14091], 95.00th=[16909], | 99.00th=[25297], 99.50th=[42206], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 640, max= 1824, per=26.27%, avg=1234.90, stdev=311.18, samples=10 iops : min= 40, max= 114, avg=77.10, stdev=19.48, samples=10 write: IOPS=74, BW=1189KiB/s (1218kB/s)(6032KiB/5072msec) clat (usec): min=588, max=198679, avg=3510.64, stdev=10735.08 lat (usec): min=589, max=198680, avg=3511.93, stdev=10735.07 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 701], 10.00th=[ 725], 20.00th=[ 783], | 30.00th=[ 848], 40.00th=[ 963], 50.00th=[ 1369], 60.00th=[ 2573], | 70.00th=[ 3163], 80.00th=[ 4752], 90.00th=[ 6456], 95.00th=[ 9110], | 99.00th=[ 24249], 99.50th=[ 26608], 99.90th=[198181], 99.95th=[198181], | 99.99th=[198181] bw ( KiB/s): min= 736, max= 1760, per=23.75%, avg=1174.20, stdev=356.59, samples=10 iops : min= 46, max= 110, avg=73.30, stdev=22.39, samples=10 lat (usec) : 50=0.39%, 100=7.03%, 250=1.43%, 750=7.16%, 1000=12.89% lat (msec) : 2=6.64%, 4=11.07%, 10=25.00%, 20=26.82%, 50=1.30% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.16%, sys=20.27%, ctx=2139, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1289: Thu Apr 4 19:12:19 2024 read: IOPS=74, BW=1196KiB/s (1225kB/s)(6032KiB/5043msec) clat (usec): min=47, max=47762, avg=9672.02, stdev=6168.68 lat (usec): min=48, max=47762, avg=9672.68, stdev=6168.69 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 61], 10.00th=[ 79], 20.00th=[ 6915], | 30.00th=[ 8717], 40.00th=[ 9634], 50.00th=[10421], 60.00th=[11076], | 70.00th=[11994], 80.00th=[13042], 90.00th=[14484], 95.00th=[16319], | 99.00th=[33424], 99.50th=[44827], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 704, max= 1536, per=25.25%, avg=1186.90, stdev=244.34, samples=10 iops : min= 44, max= 96, avg=74.10, stdev=15.26, samples=10 write: IOPS=77, BW=1241KiB/s (1270kB/s)(6256KiB/5043msec) clat (usec): min=597, max=202851, avg=3551.40, stdev=10698.74 lat (usec): min=598, max=202853, avg=3553.35, stdev=10698.74 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 701], 10.00th=[ 734], 20.00th=[ 791], | 30.00th=[ 889], 40.00th=[ 1045], 50.00th=[ 1385], 60.00th=[ 2540], | 70.00th=[ 3392], 80.00th=[ 4948], 90.00th=[ 6849], 95.00th=[ 9372], | 99.00th=[ 21627], 99.50th=[ 27919], 99.90th=[202376], 99.95th=[202376], | 99.99th=[202376] bw ( KiB/s): min= 800, max= 1664, per=25.17%, avg=1244.50, stdev=278.19, samples=10 iops : min= 50, max= 104, avg=77.70, stdev=17.46, samples=10 lat (usec) : 50=0.26%, 100=6.90%, 250=1.69%, 750=6.12%, 1000=13.67% lat (msec) : 2=7.29%, 4=11.46%, 10=24.09%, 20=26.69%, 50=1.69% lat (msec) : 250=0.13% cpu : usr=0.24%, sys=20.03%, ctx=2085, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4700KiB/s (4813kB/s), 1127KiB/s-1233KiB/s (1154kB/s-1263kB/s), io=23.4MiB (24.5MB), run=5006-5096msec WRITE: bw=4945KiB/s (5064kB/s), 1189KiB/s-1284KiB/s (1218kB/s-1315kB/s), io=24.6MiB (25.8MB), run=5006-5096msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1282: Thu Apr 4 19:12:25 2024 read: IOPS=31, BW=502KiB/s (514kB/s)(5920KiB/11791msec) clat (usec): min=1826, max=142356, avg=5364.65, stdev=8080.63 lat (usec): min=1827, max=142356, avg=5365.44, stdev=8080.62 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2180], 10.00th=[ 2311], 20.00th=[ 2573], | 30.00th=[ 2802], 40.00th=[ 3195], 50.00th=[ 3884], 60.00th=[ 4228], | 70.00th=[ 4948], 80.00th=[ 6390], 90.00th=[ 8979], 95.00th=[ 13829], | 99.00th=[ 22152], 99.50th=[ 23987], 99.90th=[141558], 99.95th=[141558], | 99.99th=[141558] bw ( KiB/s): min= 352, max= 800, per=24.35%, avg=489.65, stdev=117.93, samples=23 iops : min= 22, max= 50, avg=30.57, stdev= 7.37, samples=23 write: IOPS=33, BW=540KiB/s (553kB/s)(6368KiB/11791msec) clat (usec): min=13862, max=67133, avg=24616.84, stdev=6697.61 lat (usec): min=13864, max=67135, avg=24618.93, stdev=6697.80 clat percentiles (usec): | 1.00th=[14615], 5.00th=[15926], 10.00th=[16909], 20.00th=[18482], | 30.00th=[20055], 40.00th=[21627], 50.00th=[23725], 60.00th=[25822], | 70.00th=[27395], 80.00th=[30016], 90.00th=[33817], 95.00th=[35914], | 99.00th=[42206], 99.50th=[45351], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 288, max= 672, per=25.31%, avg=535.61, stdev=104.09, samples=23 iops : min= 18, max= 42, avg=33.43, stdev= 6.58, samples=23 lat (msec) : 2=0.78%, 4=24.22%, 10=19.01%, 20=18.36%, 50=37.37% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.12%, sys=5.30%, ctx=1013, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1283: Thu Apr 4 19:12:25 2024 read: IOPS=30, BW=482KiB/s (494kB/s)(5744KiB/11909msec) clat (usec): min=1961, max=40357, avg=4988.08, stdev=4317.39 lat (usec): min=1962, max=40358, avg=4988.89, stdev=4317.41 clat percentiles (usec): | 1.00th=[ 2073], 5.00th=[ 2212], 10.00th=[ 2343], 20.00th=[ 2540], | 30.00th=[ 2769], 40.00th=[ 3064], 50.00th=[ 3556], 60.00th=[ 4359], | 70.00th=[ 5014], 80.00th=[ 6194], 90.00th=[ 8356], 95.00th=[12911], | 99.00th=[25035], 99.50th=[27657], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 160, max= 896, per=23.87%, avg=479.96, stdev=177.17, samples=23 iops : min= 10, max= 56, avg=29.96, stdev=11.12, samples=23 write: IOPS=34, BW=550KiB/s (563kB/s)(6544KiB/11909msec) clat (msec): min=13, max=211, avg=24.72, stdev=11.17 lat (msec): min=13, max=211, avg=24.72, stdev=11.17 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 19], | 30.00th=[ 21], 40.00th=[ 22], 50.00th=[ 24], 60.00th=[ 26], | 70.00th=[ 28], 80.00th=[ 29], 90.00th=[ 33], 95.00th=[ 36], | 99.00th=[ 45], 99.50th=[ 47], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 352, max= 640, per=25.64%, avg=542.57, stdev=88.73, samples=23 iops : min= 22, max= 40, avg=33.87, stdev= 5.59, samples=23 lat (msec) : 2=0.13%, 4=26.17%, 10=16.93%, 20=17.19%, 50=39.45% lat (msec) : 250=0.13% cpu : usr=0.10%, sys=5.23%, ctx=1010, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1284: Thu Apr 4 19:12:25 2024 read: IOPS=33, BW=541KiB/s (554kB/s)(6256KiB/11570msec) clat (usec): min=1866, max=27083, avg=4889.35, stdev=3088.39 lat (usec): min=1867, max=27084, avg=4890.19, stdev=3088.40 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2245], 10.00th=[ 2376], 20.00th=[ 2638], | 30.00th=[ 2900], 40.00th=[ 3392], 50.00th=[ 4113], 60.00th=[ 4621], | 70.00th=[ 5211], 80.00th=[ 6325], 90.00th=[ 8225], 95.00th=[11994], | 99.00th=[16188], 99.50th=[19530], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 224, max= 992, per=27.05%, avg=543.96, stdev=215.78, samples=23 iops : min= 14, max= 62, avg=33.96, stdev=13.52, samples=23 write: IOPS=32, BW=521KiB/s (534kB/s)(6032KiB/11570msec) clat (msec): min=14, max=199, avg=25.60, stdev=11.30 lat (msec): min=14, max=199, avg=25.60, stdev=11.30 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 20], | 30.00th=[ 21], 40.00th=[ 23], 50.00th=[ 25], 60.00th=[ 27], | 70.00th=[ 28], 80.00th=[ 31], 90.00th=[ 35], 95.00th=[ 40], | 99.00th=[ 47], 99.50th=[ 53], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 288, max= 640, per=24.52%, avg=518.91, stdev=97.92, samples=23 iops : min= 18, max= 40, avg=32.30, stdev= 6.28, samples=23 lat (msec) : 2=0.39%, 4=23.57%, 10=23.18%, 20=15.76%, 50=36.72% lat (msec) : 100=0.26%, 250=0.13% cpu : usr=0.13%, sys=5.38%, ctx=1037, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1285: Thu Apr 4 19:12:25 2024 read: IOPS=32, BW=517KiB/s (529kB/s)(6032KiB/11673msec) clat (usec): min=1811, max=27679, avg=4802.76, stdev=3309.93 lat (usec): min=1811, max=27679, avg=4803.58, stdev=3309.90 clat percentiles (usec): | 1.00th=[ 2073], 5.00th=[ 2212], 10.00th=[ 2311], 20.00th=[ 2606], | 30.00th=[ 2966], 40.00th=[ 3163], 50.00th=[ 3949], 60.00th=[ 4424], | 70.00th=[ 5080], 80.00th=[ 5866], 90.00th=[ 8717], 95.00th=[11731], | 99.00th=[18744], 99.50th=[21890], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 224, max= 768, per=25.53%, avg=513.35, stdev=150.57, samples=23 iops : min= 14, max= 48, avg=32.04, stdev= 9.43, samples=23 write: IOPS=33, BW=536KiB/s (549kB/s)(6256KiB/11673msec) clat (msec): min=12, max=210, avg=25.20, stdev=11.62 lat (msec): min=12, max=210, avg=25.20, stdev=11.62 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 19], | 30.00th=[ 21], 40.00th=[ 23], 50.00th=[ 24], 60.00th=[ 26], | 70.00th=[ 28], 80.00th=[ 30], 90.00th=[ 34], 95.00th=[ 36], | 99.00th=[ 51], 99.50th=[ 57], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 320, max= 672, per=25.25%, avg=534.22, stdev=90.81, samples=23 iops : min= 20, max= 42, avg=33.35, stdev= 5.75, samples=23 lat (msec) : 2=0.13%, 4=24.48%, 10=20.44%, 20=18.10%, 50=36.20% lat (msec) : 100=0.52%, 250=0.13% cpu : usr=0.09%, sys=5.29%, ctx=999, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2011KiB/s (2060kB/s), 482KiB/s-541KiB/s (494kB/s-554kB/s), io=23.4MiB (24.5MB), run=11570-11909msec WRITE: bw=2116KiB/s (2167kB/s), 521KiB/s-550KiB/s (534kB/s-563kB/s), io=24.6MiB (25.8MB), run=11570-11909msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1307: Thu Apr 4 19:12:30 2024 read: IOPS=2, BW=2549KiB/s (2610kB/s)(7168KiB/2812msec) clat (usec): min=449, max=772612, avg=356685.93, stdev=328749.01 lat (usec): min=451, max=772612, avg=356687.31, stdev=328748.86 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 449], 10.00th=[ 449], 20.00th=[ 15664], | 30.00th=[ 32637], 40.00th=[ 32637], 50.00th=[501220], 60.00th=[574620], | 70.00th=[574620], 80.00th=[599786], 90.00th=[775947], 95.00th=[775947], | 99.00th=[775947], 99.50th=[775947], 99.90th=[775947], 99.95th=[775947], | 99.99th=[775947] bw ( KiB/s): min= 2048, max= 3500, per=42.95%, avg=2411.00, stdev=726.00, samples=4 iops : min= 2, max= 3, avg= 2.25, stdev= 0.50, samples=4 write: IOPS=1, BW=1821KiB/s (1864kB/s)(5120KiB/2812msec) clat (msec): min=19, max=151, avg=62.74, stdev=55.12 lat (msec): min=20, max=151, avg=62.83, stdev=55.15 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 25], 40.00th=[ 25], 50.00th=[ 38], 60.00th=[ 38], | 70.00th=[ 81], 80.00th=[ 81], 90.00th=[ 153], 95.00th=[ 153], | 99.00th=[ 153], 99.50th=[ 153], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 2048, max= 5251, per=46.44%, avg=3649.50, stdev=2264.86, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (usec) : 500=8.33% lat (msec) : 20=16.67%, 50=25.00%, 100=8.33%, 250=8.33%, 750=25.00% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=34.58%, ctx=1026, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1308: Thu Apr 4 19:12:30 2024 read: IOPS=0, BW=663KiB/s (679kB/s)(2048KiB/3087msec) clat (msec): min=46, max=224, avg=135.39, stdev=126.22 lat (msec): min=46, max=224, avg=135.40, stdev=126.22 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 226], | 70.00th=[ 226], 80.00th=[ 226], 90.00th=[ 226], 95.00th=[ 226], | 99.00th=[ 226], 99.50th=[ 226], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 2048, max= 2048, per=36.48%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3317KiB/s (3397kB/s)(10.0MiB/3087msec) clat (msec): min=17, max=1220, avg=280.83, stdev=386.43 lat (msec): min=17, max=1220, avg=280.89, stdev=386.43 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 23], | 30.00th=[ 24], 40.00th=[ 27], 50.00th=[ 77], 60.00th=[ 171], | 70.00th=[ 535], 80.00th=[ 535], 90.00th=[ 535], 95.00th=[ 1217], | 99.00th=[ 1217], 99.50th=[ 1217], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 3424, max=10240, per=86.93%, avg=6832.00, stdev=4819.64, samples=2 iops : min= 3, max= 10, avg= 6.50, stdev= 4.95, samples=2 lat (msec) : 20=8.33%, 50=33.33%, 100=8.33%, 250=25.00%, 750=16.67% cpu : usr=0.00%, sys=10.11%, ctx=171, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1309: Thu Apr 4 19:12:30 2024 read: IOPS=1, BW=1684KiB/s (1725kB/s)(6144KiB/3648msec) clat (usec): min=385, max=567516, avg=263128.61, stdev=266568.49 lat (usec): min=385, max=567517, avg=263129.43, stdev=266568.64 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 388], 10.00th=[ 388], 20.00th=[ 10028], | 30.00th=[ 10028], 40.00th=[ 57934], 50.00th=[ 57934], 60.00th=[450888], | 70.00th=[492831], 80.00th=[492831], 90.00th=[566232], 95.00th=[566232], | 99.00th=[566232], 99.50th=[566232], 99.90th=[566232], 99.95th=[566232], | 99.99th=[566232] bw ( KiB/s): min= 1712, max= 4096, per=41.12%, avg=2308.75, stdev=1191.50, samples=4 iops : min= 1, max= 4, avg= 1.75, stdev= 1.50, samples=4 write: IOPS=1, BW=1684KiB/s (1725kB/s)(6144KiB/3648msec) clat (msec): min=18, max=1192, avg=343.96, stdev=500.30 lat (msec): min=18, max=1192, avg=344.02, stdev=500.28 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 25], 50.00th=[ 25], 60.00th=[ 79], | 70.00th=[ 726], 80.00th=[ 726], 90.00th=[ 1200], 95.00th=[ 1200], | 99.00th=[ 1200], 99.50th=[ 1200], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 1712, max= 4096, per=28.71%, avg=2256.60, stdev=1038.44, samples=5 iops : min= 1, max= 4, avg= 1.80, stdev= 1.30, samples=5 lat (usec) : 500=8.33% lat (msec) : 20=16.67%, 50=16.67%, 100=16.67%, 500=16.67%, 750=16.67% cpu : usr=0.00%, sys=8.50%, ctx=861, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1310: Thu Apr 4 19:12:30 2024 read: IOPS=2, BW=2224KiB/s (2278kB/s)(5120KiB/2302msec) clat (usec): min=544, max=518982, avg=121598.57, stdev=223060.05 lat (usec): min=545, max=518984, avg=121599.84, stdev=223060.38 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 545], 10.00th=[ 545], 20.00th=[ 545], | 30.00th=[ 3884], 40.00th=[ 3884], 50.00th=[ 39060], 60.00th=[ 39060], | 70.00th=[ 45351], 80.00th=[ 45351], 90.00th=[517997], 95.00th=[517997], | 99.00th=[517997], 99.50th=[517997], 99.90th=[517997], 99.95th=[517997], | 99.99th=[517997] bw ( KiB/s): min= 2048, max= 2048, per=36.48%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3114KiB/s (3189kB/s)(7168KiB/2302msec) clat (msec): min=41, max=1189, avg=241.41, stdev=420.07 lat (msec): min=41, max=1189, avg=241.48, stdev=420.07 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 58], | 30.00th=[ 65], 40.00th=[ 65], 50.00th=[ 78], 60.00th=[ 96], | 70.00th=[ 96], 80.00th=[ 163], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 2048, max= 6144, per=52.12%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (usec) : 750=8.33% lat (msec) : 4=8.33%, 50=25.00%, 100=33.33%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=10.65%, ctx=348, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5614KiB/s (5749kB/s), 663KiB/s-2549KiB/s (679kB/s-2610kB/s), io=20.0MiB (20.0MB), run=2302-3648msec WRITE: bw=7860KiB/s (8048kB/s), 1684KiB/s-3317KiB/s (1725kB/s-3397kB/s), io=28.0MiB (29.4MB), run=2302-3648msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1311: Thu Apr 4 19:12:30 2024 read: IOPS=2, BW=2159KiB/s (2211kB/s)(7168KiB/3320msec) clat (msec): min=26, max=690, avg=285.83, stdev=314.65 lat (msec): min=26, max=690, avg=285.83, stdev=314.65 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 28], | 30.00th=[ 29], 40.00th=[ 29], 50.00th=[ 58], 60.00th=[ 575], | 70.00th=[ 575], 80.00th=[ 592], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 2048, max= 3465, per=42.94%, avg=2402.25, stdev=708.50, samples=4 iops : min= 2, max= 3, avg= 2.25, stdev= 0.50, samples=4 write: IOPS=1, BW=1542KiB/s (1579kB/s)(5120KiB/3320msec) clat (msec): min=43, max=575, avg=263.49, stdev=247.88 lat (msec): min=43, max=575, avg=263.54, stdev=247.89 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 44], | 30.00th=[ 106], 40.00th=[ 106], 50.00th=[ 107], 60.00th=[ 107], | 70.00th=[ 489], 80.00th=[ 489], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 1729, max= 2048, per=23.44%, avg=1836.33, stdev=183.31, samples=3 iops : min= 1, max= 2, avg= 1.33, stdev= 0.58, samples=3 lat (msec) : 50=33.33%, 100=8.33%, 250=16.67%, 500=8.33%, 750=33.33% cpu : usr=0.00%, sys=4.37%, ctx=46, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1312: Thu Apr 4 19:12:30 2024 read: IOPS=0, BW=656KiB/s (672kB/s)(2048KiB/3123msec) clat (msec): min=111, max=130, avg=121.02, stdev=14.06 lat (msec): min=111, max=130, avg=121.02, stdev=14.06 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 111], 50.00th=[ 111], 60.00th=[ 131], | 70.00th=[ 131], 80.00th=[ 131], 90.00th=[ 131], 95.00th=[ 131], | 99.00th=[ 131], 99.50th=[ 131], 99.90th=[ 131], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 2048, max= 2048, per=36.60%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3279KiB/s (3358kB/s)(10.0MiB/3123msec) clat (msec): min=34, max=1200, avg=287.75, stdev=377.14 lat (msec): min=34, max=1200, avg=287.84, stdev=377.14 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 41], | 30.00th=[ 42], 40.00th=[ 46], 50.00th=[ 84], 60.00th=[ 165], | 70.00th=[ 514], 80.00th=[ 514], 90.00th=[ 575], 95.00th=[ 1200], | 99.00th=[ 1200], 99.50th=[ 1200], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 2048, max= 8192, per=54.74%, avg=4288.00, stdev=2685.97, samples=4 iops : min= 2, max= 8, avg= 4.00, stdev= 2.71, samples=4 lat (msec) : 50=33.33%, 100=8.33%, 250=33.33%, 750=16.67% cpu : usr=0.00%, sys=4.87%, ctx=58, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1313: Thu Apr 4 19:12:30 2024 read: IOPS=1, BW=1679KiB/s (1719kB/s)(6144KiB/3660msec) clat (msec): min=20, max=504, avg=172.50, stdev=211.13 lat (msec): min=20, max=504, avg=172.50, stdev=211.13 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 33], 50.00th=[ 33], 60.00th=[ 74], | 70.00th=[ 372], 80.00th=[ 372], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 1715, max= 4096, per=42.78%, avg=2393.50, stdev=1145.80, samples=4 iops : min= 1, max= 4, avg= 2.00, stdev= 1.41, samples=4 write: IOPS=1, BW=1679KiB/s (1719kB/s)(6144KiB/3660msec) clat (msec): min=35, max=1204, avg=437.25, stdev=460.10 lat (msec): min=35, max=1204, avg=437.31, stdev=460.13 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 140], 50.00th=[ 140], 60.00th=[ 531], | 70.00th=[ 676], 80.00th=[ 676], 90.00th=[ 1200], 95.00th=[ 1200], | 99.00th=[ 1200], 99.50th=[ 1200], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 2048, max= 4096, per=37.09%, avg=2905.50, stdev=1026.81, samples=4 iops : min= 2, max= 4, avg= 2.75, stdev= 0.96, samples=4 lat (msec) : 50=41.67%, 100=8.33%, 250=8.33%, 500=8.33%, 750=25.00% cpu : usr=0.03%, sys=3.20%, ctx=56, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1314: Thu Apr 4 19:12:30 2024 read: IOPS=2, BW=2175KiB/s (2227kB/s)(5120KiB/2354msec) clat (msec): min=19, max=533, avg=124.84, stdev=228.56 lat (msec): min=19, max=533, avg=124.84, stdev=228.56 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 20], | 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 25], 60.00th=[ 25], | 70.00th=[ 27], 80.00th=[ 27], 90.00th=[ 535], 95.00th=[ 535], | 99.00th=[ 535], 99.50th=[ 535], 99.90th=[ 535], 99.95th=[ 535], | 99.99th=[ 535] write: IOPS=2, BW=3045KiB/s (3118kB/s)(7168KiB/2354msec) clat (msec): min=40, max=1194, avg=246.64, stdev=418.87 lat (msec): min=40, max=1194, avg=246.71, stdev=418.88 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 69], | 30.00th=[ 89], 40.00th=[ 89], 50.00th=[ 101], 60.00th=[ 102], | 70.00th=[ 102], 80.00th=[ 134], 90.00th=[ 1200], 95.00th=[ 1200], | 99.00th=[ 1200], 99.50th=[ 1200], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 2048, max= 6144, per=52.29%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=16.67%, 50=25.00%, 100=25.00%, 250=16.67%, 750=8.33% cpu : usr=0.08%, sys=5.82%, ctx=45, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5596KiB/s (5730kB/s), 656KiB/s-2175KiB/s (672kB/s-2227kB/s), io=20.0MiB (20.0MB), run=2354-3660msec WRITE: bw=7834KiB/s (8022kB/s), 1542KiB/s-3279KiB/s (1579kB/s-3358kB/s), io=28.0MiB (29.4MB), run=2354-3660msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1324: Thu Apr 4 19:12:34 2024 read: IOPS=1, BW=4585KiB/s (4695kB/s)(12.0MiB/2680msec) clat (msec): min=224, max=1276, avg=891.57, stdev=580.17 lat (msec): min=224, max=1276, avg=891.57, stdev=580.17 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 224], 10.00th=[ 224], 20.00th=[ 224], | 30.00th=[ 224], 40.00th=[ 1167], 50.00th=[ 1167], 60.00th=[ 1167], | 70.00th=[ 1284], 80.00th=[ 1284], 90.00th=[ 1284], 95.00th=[ 1284], | 99.00th=[ 1284], 99.50th=[ 1284], 99.90th=[ 1284], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 7713, max= 8192, per=100.00%, avg=7952.50, stdev=338.70, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=32.59%, ctx=1241, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1326: Thu Apr 4 19:12:34 2024 read: IOPS=0, BW=2395KiB/s (2453kB/s)(4096KiB/1710msec) clat (nsec): min=182826k, max=182826k, avg=182825846.00, stdev= 0.00 lat (nsec): min=182827k, max=182827k, avg=182826968.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 184], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4791KiB/s (4906kB/s)(8192KiB/1710msec) clat (msec): min=281, max=1238, avg=759.76, stdev=676.41 lat (msec): min=281, max=1238, avg=760.06, stdev=676.38 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 284], 10.00th=[ 284], 20.00th=[ 284], | 30.00th=[ 284], 40.00th=[ 284], 50.00th=[ 284], 60.00th=[ 1234], | 70.00th=[ 1234], 80.00th=[ 1234], 90.00th=[ 1234], 95.00th=[ 1234], | 99.00th=[ 1234], 99.50th=[ 1234], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 8192, max= 8192, per=66.35%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=15.62%, ctx=91, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1328: Thu Apr 4 19:12:34 2024 write: IOPS=1, BW=4630KiB/s (4741kB/s)(12.0MiB/2654msec) clat (msec): min=270, max=1281, avg=882.30, stdev=537.99 lat (msec): min=271, max=1282, avg=882.62, stdev=537.88 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 271], 10.00th=[ 271], 20.00th=[ 271], | 30.00th=[ 271], 40.00th=[ 1099], 50.00th=[ 1099], 60.00th=[ 1099], | 70.00th=[ 1284], 80.00th=[ 1284], 90.00th=[ 1284], 95.00th=[ 1284], | 99.00th=[ 1284], 99.50th=[ 1284], 99.90th=[ 1284], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 8094, max= 8192, per=65.96%, avg=8143.00, stdev=69.30, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=9.20%, ctx=274, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1329: Thu Apr 4 19:12:34 2024 write: IOPS=1, BW=7743KiB/s (7929kB/s)(12.0MiB/1587msec) clat (msec): min=128, max=1292, avg=518.32, stdev=670.12 lat (msec): min=128, max=1292, avg=518.63, stdev=670.10 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 129], 10.00th=[ 129], 20.00th=[ 129], | 30.00th=[ 129], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 136], | 70.00th=[ 1301], 80.00th=[ 1301], 90.00th=[ 1301], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 8192, max= 8192, per=66.35%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=66.67% cpu : usr=0.06%, sys=16.65%, ctx=108, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6113KiB/s (6260kB/s), 2395KiB/s-4585KiB/s (2453kB/s-4695kB/s), io=16.0MiB (16.8MB), run=1710-2680msec WRITE: bw=12.1MiB/s (12.6MB/s), 4630KiB/s-7743KiB/s (4741kB/s-7929kB/s), io=32.0MiB (33.6MB), run=1587-2654msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1322: Thu Apr 4 19:12:34 2024 read: IOPS=1, BW=4534KiB/s (4643kB/s)(12.0MiB/2710msec) clat (msec): min=311, max=1295, avg=898.45, stdev=518.93 lat (msec): min=311, max=1295, avg=898.45, stdev=518.93 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 313], 10.00th=[ 313], 20.00th=[ 313], | 30.00th=[ 313], 40.00th=[ 1083], 50.00th=[ 1083], 60.00th=[ 1083], | 70.00th=[ 1301], 80.00th=[ 1301], 90.00th=[ 1301], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 5483, max= 8192, per=100.00%, avg=6837.50, stdev=1915.55, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=3.36%, ctx=30, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1323: Thu Apr 4 19:12:34 2024 read: IOPS=0, BW=1658KiB/s (1698kB/s)(4096KiB/2470msec) clat (nsec): min=357798k, max=357798k, avg=357798112.00, stdev= 0.00 lat (nsec): min=357799k, max=357799k, avg=357799334.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 359], | 30.00th=[ 359], 40.00th=[ 359], 50.00th=[ 359], 60.00th=[ 359], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3317KiB/s (3396kB/s)(8192KiB/2470msec) clat (msec): min=239, max=1859, avg=1049.59, stdev=1145.03 lat (msec): min=240, max=1859, avg=1049.86, stdev=1144.99 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 241], 10.00th=[ 241], 20.00th=[ 241], | 30.00th=[ 241], 40.00th=[ 241], 50.00th=[ 241], 60.00th=[ 1854], | 70.00th=[ 1854], 80.00th=[ 1854], 90.00th=[ 1854], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 5497, max= 5497, per=41.44%, avg=5497.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.08%, sys=5.63%, ctx=30, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1325: Thu Apr 4 19:12:34 2024 write: IOPS=1, BW=7666KiB/s (7850kB/s)(12.0MiB/1603msec) clat (msec): min=88, max=1365, avg=531.20, stdev=722.75 lat (msec): min=88, max=1365, avg=531.53, stdev=722.71 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 89], 10.00th=[ 89], 20.00th=[ 89], | 30.00th=[ 89], 40.00th=[ 140], 50.00th=[ 140], 60.00th=[ 140], | 70.00th=[ 1368], 80.00th=[ 1368], 90.00th=[ 1368], 95.00th=[ 1368], | 99.00th=[ 1368], 99.50th=[ 1368], 99.90th=[ 1368], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=6.87%, ctx=107, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1327: Thu Apr 4 19:12:34 2024 write: IOPS=2, BW=8192KiB/s (8389kB/s)(12.0MiB/1500msec) clat (msec): min=102, max=816, avg=496.72, stdev=363.10 lat (msec): min=102, max=817, avg=496.89, stdev=363.05 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 103], 10.00th=[ 103], 20.00th=[ 103], | 30.00th=[ 103], 40.00th=[ 575], 50.00th=[ 575], 60.00th=[ 575], | 70.00th=[ 818], 80.00th=[ 818], 90.00th=[ 818], 95.00th=[ 818], | 99.00th=[ 818], 99.50th=[ 818], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 8192, max= 8192, per=61.75%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.07%, sys=7.14%, ctx=44, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6046KiB/s (6191kB/s), 1658KiB/s-4534KiB/s (1698kB/s-4643kB/s), io=16.0MiB (16.8MB), run=2470-2710msec WRITE: bw=12.0MiB/s (13.6MB/s), 3317KiB/s-8192KiB/s (3396kB/s-8389kB/s), io=32.0MiB (33.6MB), run=1500-2470msec