== sanity test 398b: DIO and buffer IO race ============== 10:22:07 (1724768527) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.12971 s, 44.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23615: Tue Aug 27 10:22:25 2024 read: IOPS=97, BW=389KiB/s (398kB/s)(5992KiB/15416msec) clat (usec): min=47, max=172430, avg=4810.36, stdev=6030.45 lat (usec): min=47, max=172430, avg=4811.60, stdev=6030.94 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 69], 10.00th=[ 89], 20.00th=[ 437], | 30.00th=[ 2769], 40.00th=[ 3228], 50.00th=[ 3851], 60.00th=[ 4686], | 70.00th=[ 5932], 80.00th=[ 7504], 90.00th=[ 9634], 95.00th=[ 11207], | 99.00th=[ 17171], 99.50th=[ 26870], 99.90th=[ 50594], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 208, max= 654, per=24.59%, avg=382.17, stdev=84.27, samples=30 iops : min= 52, max= 163, avg=95.40, stdev=21.05, samples=30 write: IOPS=102, BW=408KiB/s (418kB/s)(6296KiB/15416msec) clat (usec): min=471, max=43597, avg=5157.55, stdev=4814.23 lat (usec): min=472, max=43597, avg=5158.71, stdev=4814.15 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 594], 10.00th=[ 668], 20.00th=[ 848], | 30.00th=[ 1188], 40.00th=[ 2704], 50.00th=[ 4359], 60.00th=[ 5276], | 70.00th=[ 6521], 80.00th=[ 8586], 90.00th=[11731], 95.00th=[14484], | 99.00th=[20841], 99.50th=[22414], 99.90th=[31327], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 240, max= 622, per=24.57%, avg=400.00, stdev=79.55, samples=30 iops : min= 60, max= 155, avg=99.87, stdev=19.84, samples=30 lat (usec) : 50=0.23%, 100=6.12%, 250=3.19%, 500=0.39%, 750=7.23% lat (usec) : 1000=6.12% lat (msec) : 2=5.83%, 4=20.54%, 10=38.64%, 20=10.77%, 50=0.88% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.20%, sys=23.80%, ctx=6310, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23616: Tue Aug 27 10:22:25 2024 read: IOPS=96, BW=385KiB/s (394kB/s)(5940KiB/15442msec) clat (usec): min=47, max=46070, avg=4620.95, stdev=3999.46 lat (usec): min=48, max=46071, avg=4622.44, stdev=3999.01 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 73], 10.00th=[ 93], 20.00th=[ 172], | 30.00th=[ 2737], 40.00th=[ 3261], 50.00th=[ 3884], 60.00th=[ 4686], | 70.00th=[ 5997], 80.00th=[ 7504], 90.00th=[ 9503], 95.00th=[10683], | 99.00th=[17171], 99.50th=[21103], 99.90th=[44827], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 232, max= 606, per=24.51%, avg=380.83, stdev=75.50, samples=30 iops : min= 58, max= 151, avg=95.07, stdev=18.81, samples=30 write: IOPS=102, BW=411KiB/s (421kB/s)(6348KiB/15442msec) clat (usec): min=470, max=169354, avg=5346.75, stdev=6399.94 lat (usec): min=471, max=169355, avg=5348.01, stdev=6399.79 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 578], 10.00th=[ 660], 20.00th=[ 865], | 30.00th=[ 1287], 40.00th=[ 2999], 50.00th=[ 4228], 60.00th=[ 5407], | 70.00th=[ 6718], 80.00th=[ 8848], 90.00th=[ 11731], 95.00th=[ 14615], | 99.00th=[ 21103], 99.50th=[ 23725], 99.90th=[ 58983], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 264, max= 560, per=24.65%, avg=401.33, stdev=70.54, samples=30 iops : min= 66, max= 140, avg=100.20, stdev=17.58, samples=30 lat (usec) : 50=0.20%, 100=5.76%, 250=4.00%, 500=0.23%, 750=7.13% lat (usec) : 1000=6.12% lat (msec) : 2=5.01%, 4=21.91%, 10=38.05%, 20=10.68%, 50=0.85% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.24%, sys=23.70%, ctx=6425, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23617: Tue Aug 27 10:22:25 2024 read: IOPS=97, BW=391KiB/s (401kB/s)(6028KiB/15409msec) clat (usec): min=47, max=167683, avg=4590.91, stdev=5584.90 lat (usec): min=47, max=167684, avg=4592.06, stdev=5584.87 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 70], 10.00th=[ 86], 20.00th=[ 388], | 30.00th=[ 2835], 40.00th=[ 3425], 50.00th=[ 3982], 60.00th=[ 4555], | 70.00th=[ 5538], 80.00th=[ 7177], 90.00th=[ 9110], 95.00th=[ 10683], | 99.00th=[ 14615], 99.50th=[ 18220], 99.90th=[ 51643], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 216, max= 568, per=24.87%, avg=386.50, stdev=79.21, samples=30 iops : min= 54, max= 142, avg=96.53, stdev=19.78, samples=30 write: IOPS=101, BW=406KiB/s (416kB/s)(6260KiB/15409msec) clat (usec): min=430, max=56684, avg=5365.03, stdev=5206.52 lat (usec): min=430, max=56685, avg=5366.15, stdev=5206.41 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 586], 10.00th=[ 676], 20.00th=[ 873], | 30.00th=[ 1270], 40.00th=[ 2900], 50.00th=[ 4424], 60.00th=[ 5473], | 70.00th=[ 6718], 80.00th=[ 8848], 90.00th=[12256], 95.00th=[15139], | 99.00th=[21890], 99.50th=[27395], 99.90th=[49546], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 264, max= 544, per=24.53%, avg=399.33, stdev=72.04, samples=30 iops : min= 66, max= 136, avg=99.73, stdev=18.03, samples=30 lat (usec) : 50=0.29%, 100=6.38%, 250=2.90%, 500=0.72%, 750=6.32% lat (usec) : 1000=6.02% lat (msec) : 2=6.09%, 4=19.95%, 10=40.23%, 20=10.25%, 50=0.75% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.14%, sys=23.85%, ctx=6437, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23620: Tue Aug 27 10:22:25 2024 read: IOPS=99, BW=397KiB/s (407kB/s)(6052KiB/15236msec) clat (usec): min=49, max=176895, avg=4593.50, stdev=5678.19 lat (usec): min=49, max=176896, avg=4594.70, stdev=5678.20 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 75], 10.00th=[ 100], 20.00th=[ 2278], | 30.00th=[ 2802], 40.00th=[ 3261], 50.00th=[ 3785], 60.00th=[ 4490], | 70.00th=[ 5604], 80.00th=[ 7111], 90.00th=[ 8979], 95.00th=[ 10814], | 99.00th=[ 14091], 99.50th=[ 20055], 99.90th=[ 42206], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 232, max= 568, per=25.52%, avg=396.57, stdev=80.88, samples=30 iops : min= 58, max= 142, avg=99.00, stdev=20.18, samples=30 write: IOPS=102, BW=409KiB/s (419kB/s)(6236KiB/15236msec) clat (usec): min=450, max=60260, avg=5256.65, stdev=5033.81 lat (usec): min=450, max=60261, avg=5258.30, stdev=5034.80 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 586], 10.00th=[ 676], 20.00th=[ 857], | 30.00th=[ 1303], 40.00th=[ 2802], 50.00th=[ 4228], 60.00th=[ 5276], | 70.00th=[ 6783], 80.00th=[ 8717], 90.00th=[11863], 95.00th=[14877], | 99.00th=[22152], 99.50th=[23987], 99.90th=[35914], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 256, max= 558, per=25.14%, avg=409.33, stdev=75.15, samples=30 iops : min= 64, max= 139, avg=102.20, stdev=18.76, samples=30 lat (usec) : 50=0.03%, 100=4.88%, 250=3.81%, 500=0.33%, 750=7.29% lat (usec) : 1000=5.50% lat (msec) : 2=5.50%, 4=23.73%, 10=37.63%, 20=10.35%, 50=0.88% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.21%, sys=24.17%, ctx=6453, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1555KiB/s (1592kB/s), 385KiB/s-397KiB/s (394kB/s-407kB/s), io=23.4MiB (24.6MB), run=15236-15442msec WRITE: bw=1628KiB/s (1667kB/s), 406KiB/s-411KiB/s (416kB/s-421kB/s), io=24.6MiB (25.7MB), run=15236-15442msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23618: Tue Aug 27 10:22:47 2024 read: IOPS=39, BW=157KiB/s (161kB/s)(5992KiB/38212msec) clat (usec): min=1683, max=172302, avg=4475.46, stdev=5699.81 lat (usec): min=1683, max=172303, avg=4476.35, stdev=5699.83 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 1958], 10.00th=[ 2057], 20.00th=[ 2180], | 30.00th=[ 2311], 40.00th=[ 2442], 50.00th=[ 2671], 60.00th=[ 3195], | 70.00th=[ 4424], 80.00th=[ 6325], 90.00th=[ 8848], 95.00th=[ 11600], | 99.00th=[ 16450], 99.50th=[ 19530], 99.90th=[ 48497], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 48, max= 360, per=24.90%, avg=156.34, stdev=65.92, samples=76 iops : min= 12, max= 90, avg=39.00, stdev=16.49, samples=76 write: IOPS=41, BW=165KiB/s (169kB/s)(6296KiB/38212msec) clat (usec): min=8822, max=79186, avg=19963.24, stdev=8172.92 lat (usec): min=8823, max=79187, avg=19964.43, stdev=8172.98 clat percentiles (usec): | 1.00th=[ 9896], 5.00th=[11207], 10.00th=[12125], 20.00th=[13435], | 30.00th=[15008], 40.00th=[16319], 50.00th=[17957], 60.00th=[19792], | 70.00th=[21627], 80.00th=[25035], 90.00th=[31327], 95.00th=[36439], | 99.00th=[45876], 99.50th=[55313], 99.90th=[68682], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 64, max= 248, per=25.01%, avg=164.29, stdev=53.21, samples=76 iops : min= 16, max= 62, avg=40.99, stdev=13.34, samples=76 lat (msec) : 2=3.35%, 4=29.04%, 10=13.44%, 20=34.21%, 50=19.60% lat (msec) : 100=0.33%, 250=0.03% cpu : usr=0.12%, sys=5.86%, ctx=3751, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23619: Tue Aug 27 10:22:47 2024 read: IOPS=38, BW=156KiB/s (159kB/s)(5940KiB/38139msec) clat (usec): min=1671, max=183489, avg=4413.10, stdev=5865.64 lat (usec): min=1672, max=183491, avg=4413.99, stdev=5865.66 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1942], 10.00th=[ 2008], 20.00th=[ 2147], | 30.00th=[ 2278], 40.00th=[ 2409], 50.00th=[ 2606], 60.00th=[ 3064], | 70.00th=[ 4424], 80.00th=[ 6194], 90.00th=[ 8979], 95.00th=[ 11994], | 99.00th=[ 16909], 99.50th=[ 19792], 99.90th=[ 52691], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 16, max= 288, per=24.77%, avg=155.57, stdev=70.63, samples=76 iops : min= 4, max= 72, avg=38.79, stdev=17.64, samples=76 write: IOPS=41, BW=166KiB/s (170kB/s)(6348KiB/38139msec) clat (usec): min=8689, max=78927, avg=19847.86, stdev=7906.39 lat (usec): min=8690, max=78928, avg=19848.97, stdev=7906.36 clat percentiles (usec): | 1.00th=[ 9896], 5.00th=[11076], 10.00th=[11994], 20.00th=[13435], | 30.00th=[14877], 40.00th=[16450], 50.00th=[17957], 60.00th=[19792], | 70.00th=[21627], 80.00th=[25035], 90.00th=[30278], 95.00th=[35914], | 99.00th=[44827], 99.50th=[52691], 99.90th=[69731], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 72, max= 256, per=25.27%, avg=166.05, stdev=50.74, samples=76 iops : min= 18, max= 64, avg=41.41, stdev=12.70, samples=76 lat (msec) : 2=4.36%, 4=27.90%, 10=12.96%, 20=34.34%, 50=20.12% lat (msec) : 100=0.29%, 250=0.03% cpu : usr=0.17%, sys=5.71%, ctx=3791, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23621: Tue Aug 27 10:22:47 2024 read: IOPS=39, BW=158KiB/s (162kB/s)(6028KiB/38073msec) clat (usec): min=1701, max=52546, avg=4745.08, stdev=4154.79 lat (usec): min=1701, max=52546, avg=4746.01, stdev=4154.81 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 1991], 10.00th=[ 2073], 20.00th=[ 2212], | 30.00th=[ 2343], 40.00th=[ 2507], 50.00th=[ 2835], 60.00th=[ 3884], | 70.00th=[ 5407], 80.00th=[ 6652], 90.00th=[ 9634], 95.00th=[12256], | 99.00th=[20055], 99.50th=[22414], 99.90th=[45351], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 48, max= 303, per=25.22%, avg=158.39, stdev=60.67, samples=75 iops : min= 12, max= 75, avg=39.51, stdev=15.16, samples=75 write: IOPS=41, BW=164KiB/s (168kB/s)(6260KiB/38073msec) clat (msec): min=8, max=177, avg=19.69, stdev= 8.74 lat (msec): min=8, max=177, avg=19.69, stdev= 8.74 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 22], 80.00th=[ 25], 90.00th=[ 31], 95.00th=[ 36], | 99.00th=[ 46], 99.50th=[ 53], 99.90th=[ 59], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 63, max= 240, per=24.94%, avg=163.83, stdev=54.89, samples=75 iops : min= 15, max= 60, avg=40.87, stdev=13.72, samples=75 lat (msec) : 2=2.70%, 4=27.34%, 10=15.46%, 20=35.68%, 50=18.49% lat (msec) : 100=0.29%, 250=0.03% cpu : usr=0.13%, sys=5.83%, ctx=3876, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23622: Tue Aug 27 10:22:47 2024 read: IOPS=40, BW=160KiB/s (164kB/s)(6052KiB/37824msec) clat (usec): min=1703, max=42792, avg=4473.36, stdev=3833.75 lat (usec): min=1703, max=42793, avg=4474.27, stdev=3833.79 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1958], 10.00th=[ 2040], 20.00th=[ 2180], | 30.00th=[ 2311], 40.00th=[ 2442], 50.00th=[ 2671], 60.00th=[ 3228], | 70.00th=[ 4883], 80.00th=[ 6390], 90.00th=[ 9241], 95.00th=[11994], | 99.00th=[18220], 99.50th=[21890], 99.90th=[41157], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 16, max= 344, per=25.41%, avg=159.60, stdev=70.58, samples=75 iops : min= 4, max= 86, avg=39.84, stdev=17.68, samples=75 write: IOPS=41, BW=165KiB/s (169kB/s)(6236KiB/37824msec) clat (msec): min=8, max=201, avg=19.86, stdev= 9.21 lat (msec): min=8, max=201, avg=19.86, stdev= 9.21 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 22], 80.00th=[ 25], 90.00th=[ 32], 95.00th=[ 37], | 99.00th=[ 48], 99.50th=[ 51], 99.90th=[ 57], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 64, max= 248, per=25.07%, avg=164.72, stdev=54.67, samples=75 iops : min= 16, max= 62, avg=41.12, stdev=13.71, samples=75 lat (msec) : 2=3.52%, 4=28.78%, 10=13.41%, 20=35.03%, 50=18.98% lat (msec) : 100=0.26%, 250=0.03% cpu : usr=0.12%, sys=5.83%, ctx=3780, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=628KiB/s (643kB/s), 156KiB/s-160KiB/s (159kB/s-164kB/s), io=23.4MiB (24.6MB), run=37824-38212msec WRITE: bw=658KiB/s (674kB/s), 164KiB/s-166KiB/s (168kB/s-170kB/s), io=24.6MiB (25.7MB), run=37824-38212msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23664: Tue Aug 27 10:22:54 2024 read: IOPS=65, BW=1043KiB/s (1068kB/s)(5920KiB/5677msec) clat (usec): min=60, max=163975, avg=11221.98, stdev=9540.59 lat (usec): min=60, max=163976, avg=11223.15, stdev=9540.11 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 79], 10.00th=[ 118], 20.00th=[ 8717], | 30.00th=[ 9765], 40.00th=[ 10814], 50.00th=[ 11600], 60.00th=[ 12256], | 70.00th=[ 12911], 80.00th=[ 14091], 90.00th=[ 15664], 95.00th=[ 16450], | 99.00th=[ 31327], 99.50th=[ 42730], 99.90th=[164627], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 640, max= 1312, per=24.83%, avg=1041.45, stdev=184.03, samples=11 iops : min= 40, max= 82, avg=65.09, stdev=11.50, samples=11 write: IOPS=70, BW=1122KiB/s (1149kB/s)(6368KiB/5677msec) clat (usec): min=655, max=28055, avg=3789.63, stdev=3999.23 lat (usec): min=657, max=28058, avg=3790.97, stdev=3999.23 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 775], 10.00th=[ 840], 20.00th=[ 988], | 30.00th=[ 1205], 40.00th=[ 1467], 50.00th=[ 2638], 60.00th=[ 3425], | 70.00th=[ 4293], 80.00th=[ 5932], 90.00th=[ 8094], 95.00th=[ 9896], | 99.00th=[22938], 99.50th=[25822], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 736, max= 1568, per=25.64%, avg=1131.64, stdev=276.12, samples=11 iops : min= 46, max= 98, avg=70.73, stdev=17.26, samples=11 lat (usec) : 100=3.91%, 250=1.95%, 500=0.26%, 750=1.30%, 1000=9.24% lat (msec) : 2=12.89%, 4=11.07%, 10=23.70%, 20=34.11%, 50=1.43% lat (msec) : 250=0.13% cpu : usr=0.14%, sys=24.22%, ctx=2314, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23667: Tue Aug 27 10:22:54 2024 read: IOPS=62, BW=1006KiB/s (1030kB/s)(5744KiB/5710msec) clat (usec): min=53, max=48890, avg=11033.85, stdev=6577.49 lat (usec): min=54, max=48891, avg=11034.59, stdev=6577.50 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 76], 10.00th=[ 91], 20.00th=[ 7963], | 30.00th=[10028], 40.00th=[10814], 50.00th=[11731], 60.00th=[12518], | 70.00th=[13829], 80.00th=[15270], 90.00th=[16909], 95.00th=[19006], | 99.00th=[27395], 99.50th=[47449], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 320, max= 1472, per=23.72%, avg=994.91, stdev=297.60, samples=11 iops : min= 20, max= 92, avg=62.18, stdev=18.60, samples=11 write: IOPS=71, BW=1146KiB/s (1174kB/s)(6544KiB/5710msec) clat (usec): min=632, max=197234, avg=4152.69, stdev=10568.84 lat (usec): min=633, max=197235, avg=4154.03, stdev=10568.84 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 758], 10.00th=[ 799], 20.00th=[ 906], | 30.00th=[ 1045], 40.00th=[ 1270], 50.00th=[ 2114], 60.00th=[ 3130], | 70.00th=[ 3982], 80.00th=[ 5276], 90.00th=[ 7898], 95.00th=[ 13566], | 99.00th=[ 22938], 99.50th=[ 30540], 99.90th=[198181], 99.95th=[198181], | 99.99th=[198181] bw ( KiB/s): min= 608, max= 1824, per=25.25%, avg=1114.18, stdev=332.19, samples=11 iops : min= 38, max= 114, avg=69.64, stdev=20.76, samples=11 lat (usec) : 100=5.60%, 250=2.34%, 500=0.13%, 750=2.60%, 1000=11.59% lat (msec) : 2=12.24%, 4=10.94%, 10=18.10%, 20=33.72%, 50=2.60% lat (msec) : 250=0.13% cpu : usr=0.14%, sys=20.77%, ctx=2172, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23669: Tue Aug 27 10:22:54 2024 read: IOPS=68, BW=1099KiB/s (1126kB/s)(6256KiB/5690msec) clat (usec): min=54, max=33197, avg=10504.54, stdev=5550.33 lat (usec): min=54, max=33198, avg=10506.49, stdev=5551.85 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 75], 10.00th=[ 102], 20.00th=[ 8455], | 30.00th=[ 9765], 40.00th=[10683], 50.00th=[11469], 60.00th=[12256], | 70.00th=[13042], 80.00th=[14091], 90.00th=[15664], 95.00th=[17957], | 99.00th=[23725], 99.50th=[23987], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 608, max= 1504, per=26.01%, avg=1090.91, stdev=250.93, samples=11 iops : min= 38, max= 94, avg=68.18, stdev=15.68, samples=11 write: IOPS=66, BW=1060KiB/s (1086kB/s)(6032KiB/5690msec) clat (usec): min=670, max=167938, avg=4120.86, stdev=9632.68 lat (usec): min=671, max=167941, avg=4122.23, stdev=9632.76 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 783], 10.00th=[ 824], 20.00th=[ 947], | 30.00th=[ 1172], 40.00th=[ 1467], 50.00th=[ 2409], 60.00th=[ 2966], | 70.00th=[ 3851], 80.00th=[ 5407], 90.00th=[ 7898], 95.00th=[ 11994], | 99.00th=[ 28705], 99.50th=[ 43254], 99.90th=[168821], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 640, max= 1440, per=23.14%, avg=1021.09, stdev=304.74, samples=11 iops : min= 40, max= 90, avg=63.82, stdev=19.05, samples=11 lat (usec) : 100=4.95%, 250=3.39%, 500=0.13%, 750=1.30%, 1000=9.90% lat (msec) : 2=10.94%, 4=13.54%, 10=17.97%, 20=35.55%, 50=2.21% lat (msec) : 250=0.13% cpu : usr=0.26%, sys=20.94%, ctx=2320, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23670: Tue Aug 27 10:22:54 2024 read: IOPS=67, BW=1080KiB/s (1106kB/s)(6032KiB/5585msec) clat (usec): min=60, max=33816, avg=10795.90, stdev=5300.43 lat (usec): min=61, max=33817, avg=10797.33, stdev=5300.94 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 84], 10.00th=[ 114], 20.00th=[ 8586], | 30.00th=[ 9765], 40.00th=[10552], 50.00th=[11469], 60.00th=[12387], | 70.00th=[13304], 80.00th=[14353], 90.00th=[15795], 95.00th=[17695], | 99.00th=[23200], 99.50th=[33424], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 574, max= 1280, per=25.38%, avg=1064.55, stdev=195.20, samples=11 iops : min= 35, max= 80, avg=66.45, stdev=12.42, samples=11 write: IOPS=70, BW=1120KiB/s (1147kB/s)(6256KiB/5585msec) clat (usec): min=655, max=157957, avg=3778.10, stdev=8805.64 lat (usec): min=656, max=157959, avg=3779.64, stdev=8805.63 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 783], 10.00th=[ 824], 20.00th=[ 947], | 30.00th=[ 1123], 40.00th=[ 1319], 50.00th=[ 2180], 60.00th=[ 2868], | 70.00th=[ 3556], 80.00th=[ 5014], 90.00th=[ 7111], 95.00th=[ 10683], | 99.00th=[ 19792], 99.50th=[ 45351], 99.90th=[158335], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 670, max= 1696, per=25.11%, avg=1108.18, stdev=288.05, samples=11 iops : min= 41, max= 106, avg=69.18, stdev=18.14, samples=11 lat (usec) : 100=4.30%, 250=2.08%, 750=1.43%, 1000=10.55% lat (msec) : 2=12.89%, 4=13.80%, 10=18.62%, 20=34.64%, 50=1.56% lat (msec) : 250=0.13% cpu : usr=0.07%, sys=21.87%, ctx=2245, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4195KiB/s (4295kB/s), 1006KiB/s-1099KiB/s (1030kB/s-1126kB/s), io=23.4MiB (24.5MB), run=5585-5710msec WRITE: bw=4413KiB/s (4519kB/s), 1060KiB/s-1146KiB/s (1086kB/s-1174kB/s), io=24.6MiB (25.8MB), run=5585-5710msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23665: Tue Aug 27 10:22:58 2024 read: IOPS=39, BW=625KiB/s (640kB/s)(5920KiB/9477msec) clat (usec): min=1951, max=141263, avg=4928.05, stdev=8414.56 lat (usec): min=1951, max=141264, avg=4928.86, stdev=8414.61 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2089], 10.00th=[ 2180], 20.00th=[ 2278], | 30.00th=[ 2409], 40.00th=[ 2540], 50.00th=[ 2737], 60.00th=[ 2999], | 70.00th=[ 3752], 80.00th=[ 5276], 90.00th=[ 10159], 95.00th=[ 14746], | 99.00th=[ 29754], 99.50th=[ 34341], 99.90th=[141558], 99.95th=[141558], | 99.99th=[141558] bw ( KiB/s): min= 256, max= 1056, per=24.53%, avg=606.11, stdev=221.11, samples=18 iops : min= 16, max= 66, avg=37.78, stdev=13.91, samples=18 write: IOPS=41, BW=672KiB/s (688kB/s)(6368KiB/9477msec) clat (usec): min=9394, max=44351, avg=19124.21, stdev=6660.76 lat (usec): min=9394, max=44353, avg=19125.71, stdev=6660.77 clat percentiles (usec): | 1.00th=[ 9765], 5.00th=[10814], 10.00th=[11600], 20.00th=[12911], | 30.00th=[14615], 40.00th=[16581], 50.00th=[17957], 60.00th=[19792], | 70.00th=[21627], 80.00th=[24249], 90.00th=[28443], 95.00th=[32113], | 99.00th=[40633], 99.50th=[43254], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 288, max= 960, per=25.16%, avg=654.06, stdev=213.55, samples=18 iops : min= 18, max= 60, avg=40.78, stdev=13.41, samples=18 lat (msec) : 2=0.52%, 4=34.24%, 10=9.24%, 20=35.42%, 50=20.44% lat (msec) : 250=0.13% cpu : usr=0.06%, sys=6.46%, ctx=896, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23666: Tue Aug 27 10:22:58 2024 read: IOPS=37, BW=593KiB/s (607kB/s)(5744KiB/9690msec) clat (usec): min=1869, max=26173, avg=4357.20, stdev=3905.06 lat (usec): min=1870, max=26174, avg=4357.99, stdev=3905.07 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2073], 10.00th=[ 2180], 20.00th=[ 2311], | 30.00th=[ 2442], 40.00th=[ 2540], 50.00th=[ 2737], 60.00th=[ 2999], | 70.00th=[ 3490], 80.00th=[ 5211], 90.00th=[ 8717], 95.00th=[13829], | 99.00th=[20841], 99.50th=[22938], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 96, max= 1184, per=23.84%, avg=589.05, stdev=277.93, samples=19 iops : min= 6, max= 74, avg=36.68, stdev=17.34, samples=19 write: IOPS=42, BW=675KiB/s (692kB/s)(6544KiB/9690msec) clat (msec): min=9, max=232, avg=19.83, stdev=12.56 lat (msec): min=9, max=232, avg=19.83, stdev=12.56 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 21], 80.00th=[ 26], 90.00th=[ 31], 95.00th=[ 34], | 99.00th=[ 41], 99.50th=[ 43], 99.90th=[ 232], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 352, max= 992, per=25.63%, avg=666.47, stdev=184.54, samples=19 iops : min= 22, max= 62, avg=41.53, stdev=11.49, samples=19 lat (msec) : 2=0.52%, 4=33.98%, 10=9.37%, 20=36.72%, 50=19.27% lat (msec) : 250=0.13% cpu : usr=0.07%, sys=6.16%, ctx=890, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23668: Tue Aug 27 10:22:58 2024 read: IOPS=41, BW=670KiB/s (686kB/s)(6256KiB/9332msec) clat (usec): min=1894, max=28077, avg=4444.82, stdev=3938.20 lat (usec): min=1895, max=28078, avg=4445.65, stdev=3938.29 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2073], 10.00th=[ 2180], 20.00th=[ 2343], | 30.00th=[ 2474], 40.00th=[ 2638], 50.00th=[ 2802], 60.00th=[ 3032], | 70.00th=[ 3687], 80.00th=[ 5997], 90.00th=[ 9634], 95.00th=[12125], | 99.00th=[21890], 99.50th=[27657], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 320, max= 1312, per=27.33%, avg=675.28, stdev=273.80, samples=18 iops : min= 20, max= 82, avg=42.06, stdev=17.18, samples=18 write: IOPS=40, BW=646KiB/s (662kB/s)(6032KiB/9332msec) clat (msec): min=8, max=186, avg=20.06, stdev=10.93 lat (msec): min=8, max=186, avg=20.06, stdev=10.93 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 21], | 70.00th=[ 22], 80.00th=[ 25], 90.00th=[ 30], 95.00th=[ 34], | 99.00th=[ 44], 99.50th=[ 46], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 352, max= 992, per=24.40%, avg=634.39, stdev=177.23, samples=18 iops : min= 22, max= 62, avg=39.50, stdev=11.18, samples=18 lat (msec) : 2=1.04%, 4=35.55%, 10=10.16%, 20=33.33%, 50=19.79% lat (msec) : 250=0.13% cpu : usr=0.13%, sys=6.28%, ctx=913, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23671: Tue Aug 27 10:22:58 2024 read: IOPS=39, BW=625KiB/s (640kB/s)(6032KiB/9656msec) clat (usec): min=1808, max=39153, avg=5000.55, stdev=4961.10 lat (usec): min=1808, max=39154, avg=5001.38, stdev=4961.11 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2089], 10.00th=[ 2180], 20.00th=[ 2278], | 30.00th=[ 2409], 40.00th=[ 2573], 50.00th=[ 2835], 60.00th=[ 3294], | 70.00th=[ 4178], 80.00th=[ 6783], 90.00th=[11731], 95.00th=[13698], | 99.00th=[27919], 99.50th=[30540], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 288, max= 990, per=25.07%, avg=619.53, stdev=212.91, samples=19 iops : min= 18, max= 61, avg=38.58, stdev=13.24, samples=19 write: IOPS=40, BW=648KiB/s (663kB/s)(6256KiB/9656msec) clat (msec): min=9, max=167, avg=19.75, stdev=10.21 lat (msec): min=9, max=167, avg=19.75, stdev=10.21 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 21], 80.00th=[ 25], 90.00th=[ 30], 95.00th=[ 33], | 99.00th=[ 44], 99.50th=[ 49], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 352, max= 992, per=24.73%, avg=643.11, stdev=194.21, samples=19 iops : min= 22, max= 62, avg=40.05, stdev=12.12, samples=19 lat (msec) : 2=1.04%, 4=32.55%, 10=9.90%, 20=37.37%, 50=19.01% lat (msec) : 250=0.13% cpu : usr=0.11%, sys=6.19%, ctx=921, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2472KiB/s (2531kB/s), 593KiB/s-670KiB/s (607kB/s-686kB/s), io=23.4MiB (24.5MB), run=9332-9690msec WRITE: bw=2601KiB/s (2663kB/s), 646KiB/s-675KiB/s (662kB/s-692kB/s), io=24.6MiB (25.8MB), run=9332-9690msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23688: Tue Aug 27 10:23:01 2024 read: IOPS=2, BW=2812KiB/s (2880kB/s)(7168KiB/2549msec) clat (msec): min=16, max=505, avg=185.43, stdev=213.75 lat (msec): min=16, max=505, avg=185.43, stdev=213.75 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 33], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 100], 60.00th=[ 111], | 70.00th=[ 111], 80.00th=[ 485], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 2039, max= 4096, per=38.21%, avg=3069.75, stdev=1185.02, samples=4 iops : min= 1, max= 4, avg= 2.75, stdev= 1.50, samples=4 write: IOPS=1, BW=2009KiB/s (2057kB/s)(5120KiB/2549msec) clat (msec): min=24, max=483, avg=247.61, stdev=223.62 lat (msec): min=24, max=483, avg=247.69, stdev=223.64 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 213], 60.00th=[ 213], | 70.00th=[ 477], 80.00th=[ 477], 90.00th=[ 485], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 2039, max= 4096, per=22.74%, avg=2557.75, stdev=1025.51, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 lat (msec) : 20=8.33%, 50=25.00%, 100=16.67%, 250=16.67%, 500=25.00% lat (msec) : 750=8.33% cpu : usr=0.00%, sys=2.83%, ctx=49, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23689: Tue Aug 27 10:23:01 2024 read: IOPS=1, BW=1031KiB/s (1056kB/s)(2048KiB/1986msec) clat (msec): min=22, max=916, avg=469.61, stdev=632.08 lat (msec): min=22, max=916, avg=469.61, stdev=632.08 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 23], | 30.00th=[ 23], 40.00th=[ 23], 50.00th=[ 23], 60.00th=[ 919], | 70.00th=[ 919], 80.00th=[ 919], 90.00th=[ 919], 95.00th=[ 919], | 99.00th=[ 919], 99.50th=[ 919], 99.90th=[ 919], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 2048, max= 2048, per=25.49%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5156KiB/s (5280kB/s)(10.0MiB/1986msec) clat (msec): min=26, max=449, avg=97.23, stdev=130.83 lat (msec): min=26, max=449, avg=97.30, stdev=130.83 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 29], | 30.00th=[ 30], 40.00th=[ 30], 50.00th=[ 31], 60.00th=[ 41], | 70.00th=[ 114], 80.00th=[ 114], 90.00th=[ 146], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 4096, max=10240, per=63.73%, avg=7168.00, stdev=4344.46, samples=2 iops : min= 4, max= 10, avg= 7.00, stdev= 4.24, samples=2 lat (msec) : 50=58.33%, 100=8.33%, 250=16.67%, 500=8.33%, 1000=8.33% cpu : usr=0.00%, sys=5.44%, ctx=43, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23690: Tue Aug 27 10:23:01 2024 read: IOPS=2, BW=2442KiB/s (2501kB/s)(6144KiB/2516msec) clat (msec): min=11, max=393, avg=110.53, stdev=142.15 lat (msec): min=11, max=393, avg=110.53, stdev=142.15 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 44], 50.00th=[ 44], 60.00th=[ 83], | 70.00th=[ 101], 80.00th=[ 101], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 2048, max= 4096, per=42.49%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 write: IOPS=2, BW=2442KiB/s (2501kB/s)(6144KiB/2516msec) clat (msec): min=25, max=1080, avg=295.08, stdev=416.78 lat (msec): min=25, max=1080, avg=295.15, stdev=416.78 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 155], | 70.00th=[ 443], 80.00th=[ 443], 90.00th=[ 1083], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 2048, max= 4096, per=22.76%, avg=2560.00, stdev=1024.00, samples=4 iops : min= 2, max= 4, avg= 2.50, stdev= 1.00, samples=4 lat (msec) : 20=8.33%, 50=41.67%, 100=16.67%, 250=8.33%, 500=16.67% cpu : usr=0.08%, sys=3.82%, ctx=72, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23691: Tue Aug 27 10:23:01 2024 read: IOPS=2, BW=2086KiB/s (2136kB/s)(5120KiB/2454msec) clat (msec): min=11, max=481, avg=184.80, stdev=237.22 lat (msec): min=11, max=481, avg=184.80, stdev=237.22 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 12], | 30.00th=[ 12], 40.00th=[ 12], 50.00th=[ 15], 60.00th=[ 15], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 481], 95.00th=[ 481], | 99.00th=[ 481], 99.50th=[ 481], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 4087, max= 4096, per=50.93%, avg=4091.50, stdev= 6.36, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2921KiB/s (2991kB/s)(7168KiB/2454msec) clat (msec): min=28, max=1071, avg=214.98, stdev=382.13 lat (msec): min=28, max=1072, avg=215.04, stdev=382.14 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 42], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 53], 60.00th=[ 65], | 70.00th=[ 65], 80.00th=[ 197], 90.00th=[ 1070], 95.00th=[ 1070], | 99.00th=[ 1070], 99.50th=[ 1070], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 2048, max= 6144, per=36.39%, avg=4093.00, stdev=2048.01, samples=3 iops : min= 2, max= 6, avg= 3.67, stdev= 2.08, samples=3 lat (msec) : 20=25.00%, 50=25.00%, 100=16.67%, 250=8.33%, 500=16.67% cpu : usr=0.04%, sys=3.79%, ctx=37, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8035KiB/s (8227kB/s), 1031KiB/s-2812KiB/s (1056kB/s-2880kB/s), io=20.0MiB (20.0MB), run=1986-2549msec WRITE: bw=10.0MiB/s (11.5MB/s), 2009KiB/s-5156KiB/s (2057kB/s-5280kB/s), io=28.0MiB (29.4MB), run=1986-2549msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23684: Tue Aug 27 10:23:02 2024 read: IOPS=2, BW=2386KiB/s (2443kB/s)(7168KiB/3004msec) clat (msec): min=3, max=514, avg=295.48, stdev=238.98 lat (msec): min=3, max=514, avg=295.49, stdev=238.98 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 48], | 30.00th=[ 73], 40.00th=[ 73], 50.00th=[ 464], 60.00th=[ 472], | 70.00th=[ 472], 80.00th=[ 493], 90.00th=[ 514], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 1692, max= 5077, per=39.85%, avg=2716.25, stdev=1582.76, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 write: IOPS=1, BW=1704KiB/s (1745kB/s)(5120KiB/3004msec) clat (msec): min=46, max=471, avg=169.51, stdev=175.76 lat (msec): min=46, max=472, avg=169.57, stdev=175.76 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 48], 40.00th=[ 48], 50.00th=[ 126], 60.00th=[ 126], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 1692, max= 3385, per=29.55%, avg=2820.67, stdev=977.45, samples=3 iops : min= 1, max= 3, avg= 2.33, stdev= 1.15, samples=3 lat (msec) : 4=8.33%, 50=25.00%, 100=8.33%, 250=16.67%, 500=33.33% lat (msec) : 750=8.33% cpu : usr=0.03%, sys=15.92%, ctx=1129, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23685: Tue Aug 27 10:23:02 2024 read: IOPS=1, BW=1707KiB/s (1748kB/s)(2048KiB/1200msec) clat (msec): min=35, max=364, avg=200.34, stdev=232.85 lat (msec): min=35, max=364, avg=200.34, stdev=232.85 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 368], | 70.00th=[ 368], 80.00th=[ 368], 90.00th=[ 368], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 2048, max= 2048, per=30.04%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=8, BW=8533KiB/s (8738kB/s)(10.0MiB/1200msec) clat (msec): min=20, max=450, avg=77.78, stdev=132.65 lat (msec): min=20, max=450, avg=77.84, stdev=132.65 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 22], | 30.00th=[ 27], 40.00th=[ 27], 50.00th=[ 29], 60.00th=[ 32], | 70.00th=[ 49], 80.00th=[ 49], 90.00th=[ 90], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 6144, max=10240, per=85.83%, avg=8192.00, stdev=2896.31, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (msec) : 50=75.00%, 100=8.33%, 500=16.67% cpu : usr=0.00%, sys=21.93%, ctx=290, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23686: Tue Aug 27 10:23:02 2024 read: IOPS=2, BW=2498KiB/s (2558kB/s)(6144KiB/2460msec) clat (usec): min=645, max=505244, avg=188561.84, stdev=238205.43 lat (usec): min=646, max=505246, avg=188562.94, stdev=238205.34 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 644], 10.00th=[ 644], 20.00th=[ 1352], | 30.00th=[ 1352], 40.00th=[ 4047], 50.00th=[ 4047], 60.00th=[152044], | 70.00th=[467665], 80.00th=[467665], 90.00th=[505414], 95.00th=[505414], | 99.00th=[505414], 99.50th=[505414], 99.90th=[505414], 99.95th=[505414], | 99.99th=[505414] bw ( KiB/s): min= 4087, max= 5044, per=66.97%, avg=4565.50, stdev=676.70, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2498KiB/s (2558kB/s)(6144KiB/2460msec) clat (msec): min=22, max=1089, avg=216.04, stdev=428.63 lat (msec): min=22, max=1090, avg=216.09, stdev=428.64 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 26], | 30.00th=[ 26], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 48], | 70.00th=[ 80], 80.00th=[ 80], 90.00th=[ 1083], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 3362, max= 4096, per=40.32%, avg=3848.33, stdev=421.20, samples=3 iops : min= 3, max= 4, avg= 3.33, stdev= 0.58, samples=3 lat (usec) : 750=8.33% lat (msec) : 2=8.33%, 10=8.33%, 50=33.33%, 100=8.33%, 250=8.33% lat (msec) : 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=16.10%, ctx=644, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23687: Tue Aug 27 10:23:02 2024 read: IOPS=3, BW=3371KiB/s (3452kB/s)(5120KiB/1519msec) clat (usec): min=483, max=523473, avg=213135.28, stdev=274240.21 lat (usec): min=484, max=523474, avg=213136.64, stdev=274240.27 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 486], 10.00th=[ 486], 20.00th=[ 486], | 30.00th=[ 635], 40.00th=[ 635], 50.00th=[ 38536], 60.00th=[ 38536], | 70.00th=[501220], 80.00th=[501220], 90.00th=[522191], 95.00th=[522191], | 99.00th=[522191], 99.50th=[522191], 99.90th=[522191], 99.95th=[522191], | 99.99th=[522191] bw ( KiB/s): min= 2048, max= 4096, per=40.06%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 write: IOPS=4, BW=4719KiB/s (4832kB/s)(7168KiB/1519msec) clat (msec): min=25, max=165, avg=62.33, stdev=48.04 lat (msec): min=25, max=165, avg=62.40, stdev=48.02 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 33], | 30.00th=[ 38], 40.00th=[ 38], 50.00th=[ 46], 60.00th=[ 64], | 70.00th=[ 64], 80.00th=[ 67], 90.00th=[ 167], 95.00th=[ 167], | 99.00th=[ 167], 99.50th=[ 167], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 4096, max= 8192, per=64.38%, avg=6144.00, stdev=2896.31, samples=2 iops : min= 4, max= 8, avg= 6.00, stdev= 2.83, samples=2 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=41.67%, 100=16.67%, 250=8.33%, 750=16.67% cpu : usr=0.20%, sys=22.79%, ctx=607, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6818KiB/s (6981kB/s), 1707KiB/s-3371KiB/s (1748kB/s-3452kB/s), io=20.0MiB (20.0MB), run=1200-3004msec WRITE: bw=9545KiB/s (9774kB/s), 1704KiB/s-8533KiB/s (1745kB/s-8738kB/s), io=28.0MiB (29.4MB), run=1200-3004msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23703: Tue Aug 27 10:23:05 2024 read: IOPS=1, BW=6522KiB/s (6679kB/s)(12.0MiB/1884msec) clat (msec): min=181, max=1434, avg=622.63, stdev=704.37 lat (msec): min=181, max=1434, avg=622.63, stdev=704.37 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 182], 10.00th=[ 182], 20.00th=[ 182], | 30.00th=[ 182], 40.00th=[ 251], 50.00th=[ 251], 60.00th=[ 251], | 70.00th=[ 1435], 80.00th=[ 1435], 90.00th=[ 1435], 95.00th=[ 1435], | 99.00th=[ 1435], 99.50th=[ 1435], 99.90th=[ 1435], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=55.92%, ctx=527, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23704: Tue Aug 27 10:23:05 2024 read: IOPS=0, BW=1818KiB/s (1862kB/s)(4096KiB/2253msec) clat (nsec): min=207092k, max=207092k, avg=207091774.00, stdev= 0.00 lat (nsec): min=207094k, max=207094k, avg=207094003.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 207], 10.00th=[ 207], 20.00th=[ 207], | 30.00th=[ 207], 40.00th=[ 207], 50.00th=[ 207], 60.00th=[ 207], | 70.00th=[ 207], 80.00th=[ 207], 90.00th=[ 207], 95.00th=[ 207], | 99.00th=[ 207], 99.50th=[ 207], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3636KiB/s (3723kB/s)(8192KiB/2253msec) clat (msec): min=758, max=1278, avg=1018.48, stdev=367.81 lat (msec): min=758, max=1278, avg=1018.69, stdev=367.71 clat percentiles (msec): | 1.00th=[ 760], 5.00th=[ 760], 10.00th=[ 760], 20.00th=[ 760], | 30.00th=[ 760], 40.00th=[ 760], 50.00th=[ 760], 60.00th=[ 1284], | 70.00th=[ 1284], 80.00th=[ 1284], 90.00th=[ 1284], 95.00th=[ 1284], | 99.00th=[ 1284], 99.50th=[ 1284], 99.90th=[ 1284], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 8192, max= 8192, per=56.55%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=12.66%, ctx=170, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23705: Tue Aug 27 10:23:05 2024 write: IOPS=1, BW=5432KiB/s (5563kB/s)(12.0MiB/2262msec) clat (msec): min=114, max=1209, avg=748.41, stdev=567.83 lat (msec): min=114, max=1210, avg=748.68, stdev=567.82 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 115], 10.00th=[ 115], 20.00th=[ 115], | 30.00th=[ 115], 40.00th=[ 919], 50.00th=[ 919], 60.00th=[ 919], | 70.00th=[ 1217], 80.00th=[ 1217], 90.00th=[ 1217], 95.00th=[ 1217], | 99.00th=[ 1217], 99.50th=[ 1217], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 5505, max= 8192, per=47.28%, avg=6848.50, stdev=1900.00, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=11.94%, ctx=56, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23706: Tue Aug 27 10:23:05 2024 write: IOPS=2, BW=10.1MiB/s (10.6MB/s)(12.0MiB/1187msec) clat (msec): min=132, max=876, avg=388.87, stdev=422.17 lat (msec): min=133, max=876, avg=389.24, stdev=422.13 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 133], | 30.00th=[ 133], 40.00th=[ 159], 50.00th=[ 159], 60.00th=[ 159], | 70.00th=[ 877], 80.00th=[ 877], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.00%, sys=25.21%, ctx=67, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7272KiB/s (7447kB/s), 1818KiB/s-6522KiB/s (1862kB/s-6679kB/s), io=16.0MiB (16.8MB), run=1884-2253msec WRITE: bw=14.1MiB/s (14.8MB/s), 3636KiB/s-10.1MiB/s (3723kB/s-10.6MB/s), io=32.0MiB (33.6MB), run=1187-2262msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23699: Tue Aug 27 10:23:05 2024 read: IOPS=1, BW=5832KiB/s (5972kB/s)(12.0MiB/2107msec) clat (msec): min=244, max=1594, avg=696.59, stdev=777.81 lat (msec): min=244, max=1594, avg=696.59, stdev=777.81 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 245], 10.00th=[ 245], 20.00th=[ 245], | 30.00th=[ 245], 40.00th=[ 251], 50.00th=[ 251], 60.00th=[ 251], | 70.00th=[ 1603], 80.00th=[ 1603], 90.00th=[ 1603], 95.00th=[ 1603], | 99.00th=[ 1603], 99.50th=[ 1603], 99.90th=[ 1603], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min= 5468, max= 5468, per=78.17%, avg=5468.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.05%, sys=3.04%, ctx=46, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23700: Tue Aug 27 10:23:05 2024 read: IOPS=0, BW=1749KiB/s (1791kB/s)(4096KiB/2342msec) clat (nsec): min=210545k, max=210545k, avg=210545325.00, stdev= 0.00 lat (nsec): min=210547k, max=210547k, avg=210546974.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 211], 10.00th=[ 211], 20.00th=[ 211], | 30.00th=[ 211], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 211], | 70.00th=[ 211], 80.00th=[ 211], 90.00th=[ 211], 95.00th=[ 211], | 99.00th=[ 211], 99.50th=[ 211], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 8110, max= 8110, per=100.00%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3498KiB/s (3582kB/s)(8192KiB/2342msec) clat (msec): min=825, max=1290, avg=1058.09, stdev=328.91 lat (msec): min=825, max=1291, avg=1058.45, stdev=328.94 clat percentiles (msec): | 1.00th=[ 827], 5.00th=[ 827], 10.00th=[ 827], 20.00th=[ 827], | 30.00th=[ 827], 40.00th=[ 827], 50.00th=[ 827], 60.00th=[ 1284], | 70.00th=[ 1284], 80.00th=[ 1284], 90.00th=[ 1284], 95.00th=[ 1284], | 99.00th=[ 1284], 99.50th=[ 1284], 99.90th=[ 1284], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 8110, max= 8110, per=57.97%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.04%, sys=4.14%, ctx=45, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23701: Tue Aug 27 10:23:05 2024 write: IOPS=9, BW=36.3MiB/s (38.0MB/s)(12.0MiB/331msec) clat (msec): min=74, max=127, avg=100.33, stdev=26.69 lat (msec): min=74, max=127, avg=100.60, stdev=26.68 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 74], 20.00th=[ 74], | 30.00th=[ 74], 40.00th=[ 100], 50.00th=[ 100], 60.00th=[ 100], | 70.00th=[ 128], 80.00th=[ 128], 90.00th=[ 128], 95.00th=[ 128], | 99.00th=[ 128], 99.50th=[ 128], 99.90th=[ 128], 99.95th=[ 128], | 99.99th=[ 128] lat (msec) : 100=66.67%, 250=33.33% cpu : usr=0.00%, sys=29.39%, ctx=53, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23702: Tue Aug 27 10:23:05 2024 write: IOPS=2, BW=9699KiB/s (9931kB/s)(12.0MiB/1267msec) clat (msec): min=64, max=1106, avg=418.46, stdev=595.68 lat (msec): min=64, max=1106, avg=418.79, stdev=595.64 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 65], 10.00th=[ 65], 20.00th=[ 65], | 30.00th=[ 65], 40.00th=[ 86], 50.00th=[ 86], 60.00th=[ 86], | 70.00th=[ 1099], 80.00th=[ 1099], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min=16351, max=16351, per=100.00%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=66.67% cpu : usr=0.00%, sys=7.74%, ctx=75, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6996KiB/s (7164kB/s), 1749KiB/s-5832KiB/s (1791kB/s-5972kB/s), io=16.0MiB (16.8MB), run=2107-2342msec WRITE: bw=13.7MiB/s (14.3MB/s), 3498KiB/s-36.3MiB/s (3582kB/s-38.0MB/s), io=32.0MiB (33.6MB), run=331-2342msec