== sanity test 398b: DIO and buffer IO race ============== 08:05:58 (1733403958) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.27755 s, 39.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30263: Thu Dec 5 08:06:16 2024 read: IOPS=98, BW=393KiB/s (402kB/s)(5992KiB/15260msec) clat (usec): min=48, max=44549, avg=4507.37, stdev=3874.87 lat (usec): min=48, max=44550, avg=4509.57, stdev=3873.93 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 74], 10.00th=[ 93], 20.00th=[ 176], | 30.00th=[ 2769], 40.00th=[ 3195], 50.00th=[ 3720], 60.00th=[ 4424], | 70.00th=[ 5538], 80.00th=[ 7177], 90.00th=[ 9503], 95.00th=[11600], | 99.00th=[15270], 99.50th=[16712], 99.90th=[34341], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 200, max= 488, per=24.78%, avg=389.00, stdev=76.89, samples=30 iops : min= 50, max= 122, avg=97.20, stdev=19.24, samples=30 write: IOPS=103, BW=413KiB/s (422kB/s)(6296KiB/15260msec) clat (usec): min=476, max=148667, avg=5340.56, stdev=6253.59 lat (usec): min=477, max=148667, avg=5341.56, stdev=6253.51 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 586], 10.00th=[ 660], 20.00th=[ 857], | 30.00th=[ 1123], 40.00th=[ 2573], 50.00th=[ 4146], 60.00th=[ 5276], | 70.00th=[ 6783], 80.00th=[ 8717], 90.00th=[ 12125], 95.00th=[ 15401], | 99.00th=[ 21365], 99.50th=[ 26608], 99.90th=[ 56361], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 224, max= 504, per=24.83%, avg=407.93, stdev=61.49, samples=30 iops : min= 56, max= 126, avg=101.93, stdev=15.37, samples=30 lat (usec) : 50=0.07%, 100=6.32%, 250=3.68%, 500=0.33%, 750=7.16% lat (usec) : 1000=6.80% lat (msec) : 2=5.11%, 4=22.30%, 10=36.10%, 20=11.26%, 50=0.81% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.21%, sys=23.97%, ctx=6296, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30264: Thu Dec 5 08:06:16 2024 read: IOPS=97, BW=389KiB/s (398kB/s)(5940KiB/15265msec) clat (usec): min=47, max=71414, avg=4343.40, stdev=3944.71 lat (usec): min=48, max=71414, avg=4344.55, stdev=3944.97 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 71], 10.00th=[ 91], 20.00th=[ 167], | 30.00th=[ 2737], 40.00th=[ 3195], 50.00th=[ 3654], 60.00th=[ 4228], | 70.00th=[ 5276], 80.00th=[ 6783], 90.00th=[ 9241], 95.00th=[10683], | 99.00th=[15139], 99.50th=[16450], 99.90th=[32113], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 216, max= 560, per=24.73%, avg=388.20, stdev=65.48, samples=30 iops : min= 54, max= 140, avg=97.00, stdev=16.39, samples=30 write: IOPS=103, BW=416KiB/s (426kB/s)(6348KiB/15265msec) clat (usec): min=475, max=181027, avg=5490.40, stdev=6878.18 lat (usec): min=476, max=181027, avg=5491.79, stdev=6878.26 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 603], 10.00th=[ 693], 20.00th=[ 889], | 30.00th=[ 1303], 40.00th=[ 2868], 50.00th=[ 4293], 60.00th=[ 5538], | 70.00th=[ 6718], 80.00th=[ 8717], 90.00th=[ 12256], 95.00th=[ 15008], | 99.00th=[ 22938], 99.50th=[ 28181], 99.90th=[ 65274], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 264, max= 624, per=25.04%, avg=411.40, stdev=80.74, samples=30 iops : min= 66, max= 156, avg=102.80, stdev=20.19, samples=30 lat (usec) : 50=0.26%, 100=6.32%, 250=3.35%, 500=0.39%, 750=6.32% lat (usec) : 1000=6.80% lat (msec) : 2=5.66%, 4=22.59%, 10=36.43%, 20=11.04%, 50=0.75% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.22%, sys=24.19%, ctx=6150, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30265: Thu Dec 5 08:06:16 2024 read: IOPS=98, BW=394KiB/s (404kB/s)(6028KiB/15292msec) clat (usec): min=47, max=183455, avg=4443.93, stdev=6024.41 lat (usec): min=47, max=183455, avg=4446.00, stdev=6023.71 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 71], 10.00th=[ 88], 20.00th=[ 119], | 30.00th=[ 2606], 40.00th=[ 3064], 50.00th=[ 3556], 60.00th=[ 4293], | 70.00th=[ 5604], 80.00th=[ 7177], 90.00th=[ 9241], 95.00th=[ 10945], | 99.00th=[ 15533], 99.50th=[ 17957], 99.90th=[ 46924], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 200, max= 568, per=24.93%, avg=391.43, stdev=81.00, samples=30 iops : min= 50, max= 142, avg=97.83, stdev=20.26, samples=30 write: IOPS=102, BW=409KiB/s (419kB/s)(6260KiB/15292msec) clat (usec): min=478, max=44943, avg=5423.36, stdev=5175.55 lat (usec): min=479, max=44944, avg=5424.84, stdev=5175.90 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 586], 10.00th=[ 668], 20.00th=[ 857], | 30.00th=[ 1123], 40.00th=[ 2802], 50.00th=[ 4293], 60.00th=[ 5604], | 70.00th=[ 7046], 80.00th=[ 9110], 90.00th=[12518], 95.00th=[15270], | 99.00th=[22152], 99.50th=[26346], 99.90th=[31851], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 272, max= 560, per=24.65%, avg=405.03, stdev=72.76, samples=30 iops : min= 68, max= 140, avg=101.23, stdev=18.19, samples=30 lat (usec) : 50=0.36%, 100=7.06%, 250=3.87%, 500=0.46%, 750=7.03% lat (usec) : 1000=7.19% lat (msec) : 2=4.62%, 4=21.32%, 10=36.17%, 20=10.90%, 50=0.98% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=23.46%, ctx=6202, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30266: Thu Dec 5 08:06:16 2024 read: IOPS=99, BW=397KiB/s (407kB/s)(6052KiB/15230msec) clat (usec): min=47, max=44105, avg=4411.84, stdev=3555.69 lat (usec): min=47, max=44106, avg=4412.71, stdev=3555.65 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 72], 10.00th=[ 91], 20.00th=[ 519], | 30.00th=[ 2737], 40.00th=[ 3261], 50.00th=[ 3752], 60.00th=[ 4490], | 70.00th=[ 5538], 80.00th=[ 7111], 90.00th=[ 8979], 95.00th=[10814], | 99.00th=[14615], 99.50th=[15795], 99.90th=[22676], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 248, max= 536, per=25.27%, avg=396.73, stdev=69.87, samples=30 iops : min= 62, max= 134, avg=99.13, stdev=17.47, samples=30 write: IOPS=102, BW=409KiB/s (419kB/s)(6236KiB/15230msec) clat (usec): min=468, max=182146, avg=5423.79, stdev=6825.97 lat (usec): min=468, max=182147, avg=5425.01, stdev=6825.91 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 578], 10.00th=[ 660], 20.00th=[ 848], | 30.00th=[ 1139], 40.00th=[ 2606], 50.00th=[ 4228], 60.00th=[ 5473], | 70.00th=[ 6915], 80.00th=[ 8979], 90.00th=[ 11994], 95.00th=[ 15139], | 99.00th=[ 21890], 99.50th=[ 23987], 99.90th=[ 66323], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 272, max= 632, per=24.97%, avg=410.33, stdev=77.31, samples=30 iops : min= 68, max= 158, avg=102.53, stdev=19.37, samples=30 lat (usec) : 50=0.20%, 100=6.25%, 250=3.29%, 500=0.26%, 750=6.80% lat (usec) : 1000=6.93% lat (msec) : 2=5.01%, 4=22.46%, 10=37.73%, 20=10.22%, 50=0.78% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.23%, sys=24.37%, ctx=6148, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1570KiB/s (1608kB/s), 389KiB/s-397KiB/s (398kB/s-407kB/s), io=23.4MiB (24.6MB), run=15230-15292msec WRITE: bw=1644KiB/s (1683kB/s), 409KiB/s-416KiB/s (419kB/s-426kB/s), io=24.6MiB (25.7MB), run=15230-15292msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30267: Thu Dec 5 08:06:40 2024 read: IOPS=38, BW=153KiB/s (157kB/s)(5992KiB/39086msec) clat (usec): min=1618, max=191809, avg=4191.74, stdev=6648.91 lat (usec): min=1619, max=191811, avg=4192.62, stdev=6648.95 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1958], 10.00th=[ 2040], 20.00th=[ 2147], | 30.00th=[ 2245], 40.00th=[ 2376], 50.00th=[ 2507], 60.00th=[ 2704], | 70.00th=[ 3064], 80.00th=[ 4359], 90.00th=[ 8717], 95.00th=[ 11863], | 99.00th=[ 19530], 99.50th=[ 36439], 99.90th=[ 60031], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 40, max= 280, per=24.88%, avg=152.75, stdev=67.37, samples=77 iops : min= 10, max= 70, avg=38.13, stdev=16.85, samples=77 write: IOPS=40, BW=161KiB/s (165kB/s)(6296KiB/39086msec) clat (usec): min=9669, max=63949, avg=20770.66, stdev=8374.41 lat (usec): min=9670, max=63950, avg=20771.75, stdev=8374.41 clat percentiles (usec): | 1.00th=[11076], 5.00th=[11994], 10.00th=[12649], 20.00th=[13960], | 30.00th=[15533], 40.00th=[17171], 50.00th=[18744], 60.00th=[20055], | 70.00th=[21890], 80.00th=[25822], 90.00th=[33817], 95.00th=[38536], | 99.00th=[48497], 99.50th=[54264], 99.90th=[62653], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 48, max= 240, per=24.87%, avg=159.92, stdev=54.71, samples=77 iops : min= 12, max= 60, avg=39.92, stdev=13.70, samples=77 lat (msec) : 2=3.45%, 4=34.64%, 10=7.13%, 20=33.53%, 50=20.74% lat (msec) : 100=0.49%, 250=0.03% cpu : usr=0.13%, sys=5.39%, ctx=3421, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30268: Thu Dec 5 08:06:40 2024 read: IOPS=38, BW=152KiB/s (156kB/s)(5940KiB/39044msec) clat (usec): min=1644, max=184258, avg=3935.48, stdev=5755.86 lat (usec): min=1644, max=184260, avg=3936.33, stdev=5755.93 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1942], 10.00th=[ 2024], 20.00th=[ 2147], | 30.00th=[ 2245], 40.00th=[ 2343], 50.00th=[ 2507], 60.00th=[ 2671], | 70.00th=[ 3032], 80.00th=[ 4490], 90.00th=[ 7832], 95.00th=[ 10945], | 99.00th=[ 17171], 99.50th=[ 19530], 99.90th=[ 36963], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 16, max= 312, per=24.78%, avg=152.13, stdev=69.71, samples=77 iops : min= 4, max= 78, avg=37.97, stdev=17.48, samples=77 write: IOPS=40, BW=163KiB/s (166kB/s)(6348KiB/39044msec) clat (usec): min=10155, max=73604, avg=20868.41, stdev=8562.95 lat (usec): min=10156, max=73606, avg=20869.46, stdev=8563.02 clat percentiles (usec): | 1.00th=[10945], 5.00th=[11863], 10.00th=[12649], 20.00th=[13829], | 30.00th=[15533], 40.00th=[17433], 50.00th=[18744], 60.00th=[20317], | 70.00th=[22152], 80.00th=[25822], 90.00th=[33162], 95.00th=[39060], | 99.00th=[49546], 99.50th=[53216], 99.90th=[71828], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 56, max= 232, per=25.16%, avg=161.79, stdev=53.39, samples=77 iops : min= 14, max= 58, avg=40.39, stdev=13.40, samples=77 lat (msec) : 2=3.71%, 4=33.69%, 10=7.81%, 20=33.17%, 50=21.13% lat (msec) : 100=0.46%, 250=0.03% cpu : usr=0.14%, sys=5.46%, ctx=3426, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30269: Thu Dec 5 08:06:40 2024 read: IOPS=38, BW=155KiB/s (159kB/s)(6028KiB/38897msec) clat (usec): min=1692, max=60517, avg=4184.10, stdev=4117.99 lat (usec): min=1693, max=60517, avg=4185.13, stdev=4118.01 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1958], 10.00th=[ 2024], 20.00th=[ 2147], | 30.00th=[ 2245], 40.00th=[ 2409], 50.00th=[ 2540], 60.00th=[ 2769], | 70.00th=[ 3359], 80.00th=[ 5080], 90.00th=[ 9503], 95.00th=[12125], | 99.00th=[17695], 99.50th=[21103], 99.90th=[59507], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min= 56, max= 280, per=25.26%, avg=155.09, stdev=59.17, samples=77 iops : min= 14, max= 70, avg=38.68, stdev=14.81, samples=77 write: IOPS=40, BW=161KiB/s (165kB/s)(6260KiB/38897msec) clat (msec): min=10, max=178, avg=20.76, stdev= 8.83 lat (msec): min=10, max=178, avg=20.76, stdev= 8.83 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 19], 60.00th=[ 21], | 70.00th=[ 23], 80.00th=[ 26], 90.00th=[ 32], 95.00th=[ 37], | 99.00th=[ 48], 99.50th=[ 53], 99.90th=[ 63], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 64, max= 248, per=24.98%, avg=160.60, stdev=54.32, samples=77 iops : min= 16, max= 62, avg=40.05, stdev=13.56, samples=77 lat (msec) : 2=4.13%, 4=32.75%, 10=7.68%, 20=32.98%, 50=22.04% lat (msec) : 100=0.39%, 250=0.03% cpu : usr=0.13%, sys=5.41%, ctx=3461, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30270: Thu Dec 5 08:06:40 2024 read: IOPS=39, BW=156KiB/s (160kB/s)(6052KiB/38743msec) clat (usec): min=1707, max=54382, avg=4027.39, stdev=4152.88 lat (usec): min=1708, max=54383, avg=4028.23, stdev=4152.89 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1958], 10.00th=[ 2024], 20.00th=[ 2147], | 30.00th=[ 2212], 40.00th=[ 2343], 50.00th=[ 2507], 60.00th=[ 2671], | 70.00th=[ 3097], 80.00th=[ 4555], 90.00th=[ 8160], 95.00th=[11994], | 99.00th=[20579], 99.50th=[23987], 99.90th=[44303], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 24, max= 384, per=25.40%, avg=155.96, stdev=71.49, samples=77 iops : min= 6, max= 96, avg=38.92, stdev=17.93, samples=77 write: IOPS=40, BW=161KiB/s (165kB/s)(6236KiB/38743msec) clat (msec): min=9, max=197, avg=20.89, stdev= 9.73 lat (msec): min=9, max=197, avg=20.89, stdev= 9.73 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 19], 60.00th=[ 21], | 70.00th=[ 23], 80.00th=[ 27], 90.00th=[ 34], 95.00th=[ 40], | 99.00th=[ 51], 99.50th=[ 54], 99.90th=[ 74], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 56, max= 240, per=25.00%, avg=160.74, stdev=55.71, samples=77 iops : min= 14, max= 60, avg=40.12, stdev=14.00, samples=77 lat (msec) : 2=4.33%, 4=33.92%, 10=7.42%, 20=32.75%, 50=21.00% lat (msec) : 100=0.55%, 250=0.03% cpu : usr=0.10%, sys=5.45%, ctx=3426, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=614KiB/s (629kB/s), 152KiB/s-156KiB/s (156kB/s-160kB/s), io=23.4MiB (24.6MB), run=38743-39086msec WRITE: bw=643KiB/s (659kB/s), 161KiB/s-163KiB/s (165kB/s-166kB/s), io=24.6MiB (25.7MB), run=38743-39086msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30315: Thu Dec 5 08:06:46 2024 read: IOPS=69, BW=1117KiB/s (1144kB/s)(5920KiB/5299msec) clat (usec): min=54, max=133780, avg=10594.35, stdev=9705.41 lat (usec): min=55, max=133781, avg=10595.08, stdev=9705.43 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 76], 10.00th=[ 94], 20.00th=[ 131], | 30.00th=[ 9634], 40.00th=[ 10814], 50.00th=[ 11731], 60.00th=[ 12518], | 70.00th=[ 13566], 80.00th=[ 14484], 90.00th=[ 16188], 95.00th=[ 17957], | 99.00th=[ 27395], 99.50th=[ 74974], 99.90th=[133694], 99.95th=[133694], | 99.99th=[133694] bw ( KiB/s): min= 448, max= 1312, per=23.88%, avg=1068.20, stdev=261.99, samples=10 iops : min= 28, max= 82, avg=66.60, stdev=16.31, samples=10 write: IOPS=75, BW=1202KiB/s (1231kB/s)(6368KiB/5299msec) clat (usec): min=655, max=25456, avg=3355.09, stdev=3353.99 lat (usec): min=656, max=25457, avg=3356.45, stdev=3353.86 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 791], 10.00th=[ 873], 20.00th=[ 1057], | 30.00th=[ 1221], 40.00th=[ 1319], 50.00th=[ 1582], 60.00th=[ 3032], | 70.00th=[ 4015], 80.00th=[ 5342], 90.00th=[ 7570], 95.00th=[ 9372], | 99.00th=[16319], 99.50th=[19006], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 512, max= 1824, per=24.87%, avg=1170.60, stdev=343.82, samples=10 iops : min= 32, max= 114, avg=73.00, stdev=21.43, samples=10 lat (usec) : 100=6.12%, 250=4.82%, 500=0.13%, 750=1.82%, 1000=7.16% lat (msec) : 2=19.01%, 4=8.46%, 10=17.32%, 20=33.98%, 50=0.78% lat (msec) : 100=0.26%, 250=0.13% cpu : usr=0.21%, sys=25.93%, ctx=2122, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30316: Thu Dec 5 08:06:46 2024 read: IOPS=69, BW=1107KiB/s (1134kB/s)(5744KiB/5188msec) clat (usec): min=51, max=58241, avg=9883.04, stdev=6811.26 lat (usec): min=51, max=58243, avg=9883.78, stdev=6811.28 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 72], 10.00th=[ 86], 20.00th=[ 112], | 30.00th=[ 9110], 40.00th=[10421], 50.00th=[11600], 60.00th=[12649], | 70.00th=[13435], 80.00th=[14615], 90.00th=[16057], 95.00th=[18220], | 99.00th=[23462], 99.50th=[31065], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 640, max= 1312, per=24.32%, avg=1088.00, stdev=189.61, samples=10 iops : min= 40, max= 82, avg=68.00, stdev=11.85, samples=10 write: IOPS=78, BW=1261KiB/s (1292kB/s)(6544KiB/5188msec) clat (usec): min=677, max=84201, avg=3979.51, stdev=6273.56 lat (usec): min=678, max=84202, avg=3981.08, stdev=6273.81 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 816], 10.00th=[ 873], 20.00th=[ 996], | 30.00th=[ 1205], 40.00th=[ 1401], 50.00th=[ 1991], 60.00th=[ 3326], | 70.00th=[ 4178], 80.00th=[ 5735], 90.00th=[ 8029], 95.00th=[ 9503], | 99.00th=[22152], 99.50th=[41681], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 896, max= 1792, per=25.98%, avg=1222.40, stdev=259.44, samples=10 iops : min= 56, max= 112, avg=76.40, stdev=16.22, samples=10 lat (usec) : 100=7.81%, 250=4.04%, 500=0.13%, 750=1.43%, 1000=9.51% lat (msec) : 2=15.89%, 4=9.77%, 10=19.27%, 20=30.47%, 50=1.30% lat (msec) : 100=0.39% cpu : usr=0.02%, sys=23.81%, ctx=2016, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30317: Thu Dec 5 08:06:46 2024 read: IOPS=73, BW=1168KiB/s (1197kB/s)(6256KiB/5354msec) clat (usec): min=50, max=97896, avg=10190.80, stdev=8064.09 lat (usec): min=51, max=97897, avg=10191.83, stdev=8064.13 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 69], 10.00th=[ 84], 20.00th=[ 119], | 30.00th=[ 8979], 40.00th=[10421], 50.00th=[11600], 60.00th=[12387], | 70.00th=[13566], 80.00th=[14746], 90.00th=[16909], 95.00th=[17957], | 99.00th=[25035], 99.50th=[57934], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 640, max= 1632, per=25.46%, avg=1138.90, stdev=275.01, samples=10 iops : min= 40, max= 102, avg=71.10, stdev=17.21, samples=10 write: IOPS=70, BW=1127KiB/s (1154kB/s)(6032KiB/5354msec) clat (usec): min=645, max=66698, avg=3577.40, stdev=4730.04 lat (usec): min=646, max=66698, avg=3578.69, stdev=4730.05 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 783], 10.00th=[ 848], 20.00th=[ 971], | 30.00th=[ 1156], 40.00th=[ 1287], 50.00th=[ 1729], 60.00th=[ 3392], | 70.00th=[ 4047], 80.00th=[ 5407], 90.00th=[ 7439], 95.00th=[10159], | 99.00th=[19792], 99.50th=[23462], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 640, max= 1536, per=23.05%, avg=1084.50, stdev=258.33, samples=10 iops : min= 40, max= 96, avg=67.70, stdev=16.12, samples=10 lat (usec) : 100=7.94%, 250=4.56%, 500=0.13%, 750=1.30%, 1000=9.64% lat (msec) : 2=14.06%, 4=8.85%, 10=19.27%, 20=32.68%, 50=1.17% lat (msec) : 100=0.39% cpu : usr=0.07%, sys=23.07%, ctx=2135, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30318: Thu Dec 5 08:06:46 2024 read: IOPS=70, BW=1136KiB/s (1163kB/s)(6032KiB/5310msec) clat (usec): min=49, max=51342, avg=9924.49, stdev=6618.28 lat (usec): min=50, max=51342, avg=9925.21, stdev=6618.28 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 63], 10.00th=[ 90], 20.00th=[ 119], | 30.00th=[ 9110], 40.00th=[10290], 50.00th=[11338], 60.00th=[12256], | 70.00th=[13173], 80.00th=[14746], 90.00th=[16319], 95.00th=[18220], | 99.00th=[24249], 99.50th=[35914], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 608, max= 1344, per=24.39%, avg=1090.90, stdev=193.29, samples=10 iops : min= 38, max= 84, avg=68.10, stdev=12.04, samples=10 write: IOPS=73, BW=1178KiB/s (1206kB/s)(6256KiB/5310msec) clat (usec): min=662, max=85278, avg=3890.72, stdev=6663.12 lat (usec): min=664, max=85279, avg=3892.32, stdev=6663.33 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 807], 10.00th=[ 906], 20.00th=[ 1057], | 30.00th=[ 1237], 40.00th=[ 1418], 50.00th=[ 2180], 60.00th=[ 3097], | 70.00th=[ 3818], 80.00th=[ 5145], 90.00th=[ 8291], 95.00th=[10683], | 99.00th=[22152], 99.50th=[78119], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 704, max= 1536, per=24.47%, avg=1151.70, stdev=228.33, samples=10 iops : min= 44, max= 96, avg=71.90, stdev=14.29, samples=10 lat (usec) : 50=0.13%, 100=7.42%, 250=4.17%, 500=0.13%, 750=1.30% lat (usec) : 1000=7.29% lat (msec) : 2=16.54%, 4=11.46%, 10=18.10%, 20=31.77%, 50=1.30% lat (msec) : 100=0.39% cpu : usr=0.19%, sys=22.94%, ctx=2129, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4474KiB/s (4581kB/s), 1107KiB/s-1168KiB/s (1134kB/s-1197kB/s), io=23.4MiB (24.5MB), run=5188-5354msec WRITE: bw=4707KiB/s (4820kB/s), 1127KiB/s-1261KiB/s (1154kB/s-1292kB/s), io=24.6MiB (25.8MB), run=5188-5354msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30319: Thu Dec 5 08:06:51 2024 read: IOPS=36, BW=585KiB/s (599kB/s)(5920KiB/10119msec) clat (usec): min=1953, max=224290, avg=4924.32, stdev=12007.53 lat (usec): min=1954, max=224292, avg=4925.22, stdev=12007.56 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2180], 10.00th=[ 2278], 20.00th=[ 2442], | 30.00th=[ 2606], 40.00th=[ 2769], 50.00th=[ 3064], 60.00th=[ 3326], | 70.00th=[ 3916], 80.00th=[ 5342], 90.00th=[ 7570], 95.00th=[ 11994], | 99.00th=[ 19530], 99.50th=[ 37487], 99.90th=[223347], 99.95th=[223347], | 99.99th=[223347] bw ( KiB/s): min= 320, max= 960, per=24.60%, avg=577.50, stdev=191.57, samples=20 iops : min= 20, max= 60, avg=36.05, stdev=11.98, samples=20 write: IOPS=39, BW=629KiB/s (644kB/s)(6368KiB/10119msec) clat (usec): min=10073, max=46739, avg=20715.21, stdev=6416.88 lat (usec): min=10074, max=46740, avg=20716.88, stdev=6416.80 clat percentiles (usec): | 1.00th=[11338], 5.00th=[12256], 10.00th=[13042], 20.00th=[14746], | 30.00th=[16909], 40.00th=[18482], 50.00th=[19792], 60.00th=[21627], | 70.00th=[23200], 80.00th=[25822], 90.00th=[28967], 95.00th=[31851], | 99.00th=[42730], 99.50th=[46400], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 192, max= 896, per=25.32%, avg=625.50, stdev=185.20, samples=20 iops : min= 12, max= 56, avg=39.05, stdev=11.54, samples=20 lat (msec) : 2=0.78%, 4=33.46%, 10=10.81%, 20=29.56%, 50=25.26% lat (msec) : 250=0.13% cpu : usr=0.05%, sys=6.36%, ctx=897, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30320: Thu Dec 5 08:06:51 2024 read: IOPS=35, BW=565KiB/s (578kB/s)(5744KiB/10169msec) clat (usec): min=1961, max=28425, avg=4312.50, stdev=3275.58 lat (usec): min=1962, max=28427, avg=4313.69, stdev=3275.42 clat percentiles (usec): | 1.00th=[ 2073], 5.00th=[ 2212], 10.00th=[ 2278], 20.00th=[ 2474], | 30.00th=[ 2671], 40.00th=[ 2835], 50.00th=[ 3064], 60.00th=[ 3294], | 70.00th=[ 3851], 80.00th=[ 5538], 90.00th=[ 8029], 95.00th=[11600], | 99.00th=[18220], 99.50th=[19268], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 64, max= 928, per=23.98%, avg=563.10, stdev=235.25, samples=20 iops : min= 4, max= 58, avg=35.15, stdev=14.71, samples=20 write: IOPS=40, BW=644KiB/s (659kB/s)(6544KiB/10169msec) clat (msec): min=10, max=235, avg=20.95, stdev=12.35 lat (msec): min=10, max=235, avg=20.95, stdev=12.35 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 20], 60.00th=[ 21], | 70.00th=[ 23], 80.00th=[ 26], 90.00th=[ 30], 95.00th=[ 33], | 99.00th=[ 40], 99.50th=[ 47], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 320, max= 896, per=25.78%, avg=636.75, stdev=163.86, samples=20 iops : min= 20, max= 56, avg=39.75, stdev=10.30, samples=20 lat (msec) : 2=0.26%, 4=33.20%, 10=9.64%, 20=30.99%, 50=25.78% lat (msec) : 250=0.13% cpu : usr=0.12%, sys=6.17%, ctx=874, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30321: Thu Dec 5 08:06:51 2024 read: IOPS=39, BW=631KiB/s (646kB/s)(6256KiB/9913msec) clat (usec): min=1912, max=27993, avg=4820.95, stdev=4121.50 lat (usec): min=1912, max=27994, avg=4821.80, stdev=4121.50 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2212], 10.00th=[ 2343], 20.00th=[ 2507], | 30.00th=[ 2704], 40.00th=[ 2868], 50.00th=[ 3130], 60.00th=[ 3458], | 70.00th=[ 4047], 80.00th=[ 6194], 90.00th=[ 9634], 95.00th=[13829], | 99.00th=[22414], 99.50th=[23462], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 224, max= 1280, per=27.18%, avg=638.11, stdev=270.34, samples=19 iops : min= 14, max= 80, avg=39.79, stdev=16.89, samples=19 write: IOPS=38, BW=608KiB/s (623kB/s)(6032KiB/9913msec) clat (msec): min=11, max=239, avg=21.25, stdev=12.78 lat (msec): min=11, max=239, avg=21.25, stdev=12.78 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 20], 60.00th=[ 22], | 70.00th=[ 23], 80.00th=[ 26], 90.00th=[ 29], 95.00th=[ 33], | 99.00th=[ 41], 99.50th=[ 44], 99.90th=[ 241], 99.95th=[ 241], | 99.99th=[ 241] bw ( KiB/s): min= 256, max= 832, per=24.27%, avg=599.42, stdev=158.44, samples=19 iops : min= 16, max= 52, avg=37.37, stdev= 9.90, samples=19 lat (msec) : 2=0.39%, 4=34.77%, 10=10.68%, 20=29.95%, 50=24.09% lat (msec) : 250=0.13% cpu : usr=0.09%, sys=6.43%, ctx=892, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30322: Thu Dec 5 08:06:51 2024 read: IOPS=36, BW=591KiB/s (606kB/s)(6032KiB/10200msec) clat (usec): min=2045, max=39379, avg=4987.85, stdev=4719.43 lat (usec): min=2045, max=39380, avg=4988.71, stdev=4719.43 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2212], 10.00th=[ 2311], 20.00th=[ 2540], | 30.00th=[ 2704], 40.00th=[ 2868], 50.00th=[ 3097], 60.00th=[ 3556], | 70.00th=[ 4359], 80.00th=[ 6390], 90.00th=[10028], 95.00th=[14746], | 99.00th=[27657], 99.50th=[30016], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 224, max= 1024, per=24.60%, avg=577.50, stdev=174.11, samples=20 iops : min= 14, max= 64, avg=36.05, stdev=10.85, samples=20 write: IOPS=38, BW=613KiB/s (628kB/s)(6256KiB/10200msec) clat (msec): min=10, max=229, avg=21.13, stdev=12.15 lat (msec): min=10, max=229, avg=21.13, stdev=12.15 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 20], 60.00th=[ 22], | 70.00th=[ 23], 80.00th=[ 25], 90.00th=[ 29], 95.00th=[ 33], | 99.00th=[ 42], 99.50th=[ 44], 99.90th=[ 230], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 224, max= 832, per=24.55%, avg=606.30, stdev=170.65, samples=20 iops : min= 14, max= 52, avg=37.85, stdev=10.62, samples=20 lat (msec) : 4=32.68%, 10=11.59%, 20=30.21%, 50=25.39%, 250=0.13% cpu : usr=0.14%, sys=6.24%, ctx=898, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2348KiB/s (2405kB/s), 565KiB/s-631KiB/s (578kB/s-646kB/s), io=23.4MiB (24.5MB), run=9913-10200msec WRITE: bw=2471KiB/s (2530kB/s), 608KiB/s-644KiB/s (623kB/s-659kB/s), io=24.6MiB (25.8MB), run=9913-10200msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30336: Thu Dec 5 08:06:54 2024 read: IOPS=4, BW=4694KiB/s (4807kB/s)(7168KiB/1527msec) clat (msec): min=13, max=447, avg=90.32, stdev=158.48 lat (msec): min=13, max=447, avg=90.32, stdev=158.48 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 17], | 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 24], 60.00th=[ 56], | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 6144, max= 6144, per=49.26%, avg=6144.00, stdev= 0.00, samples=2 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3353KiB/s (3433kB/s)(5120KiB/1527msec) clat (msec): min=30, max=595, avg=175.40, stdev=237.29 lat (msec): min=30, max=595, avg=175.46, stdev=237.29 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31], | 30.00th=[ 54], 40.00th=[ 54], 50.00th=[ 78], 60.00th=[ 78], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 2048, max= 6144, per=23.46%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=25.00%, 50=16.67%, 100=33.33%, 250=8.33%, 500=8.33% lat (msec) : 750=8.33% cpu : usr=0.13%, sys=5.90%, ctx=56, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30337: Thu Dec 5 08:06:54 2024 read: IOPS=1, BW=1247KiB/s (1277kB/s)(2048KiB/1642msec) clat (usec): min=17658, max=41538, avg=29598.66, stdev=16886.06 lat (usec): min=17660, max=41539, avg=29599.77, stdev=16885.33 clat percentiles (usec): | 1.00th=[17695], 5.00th=[17695], 10.00th=[17695], 20.00th=[17695], | 30.00th=[17695], 40.00th=[17695], 50.00th=[17695], 60.00th=[41681], | 70.00th=[41681], 80.00th=[41681], 90.00th=[41681], 95.00th=[41681], | 99.00th=[41681], 99.50th=[41681], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 2043, max= 2048, per=16.40%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=6, BW=6236KiB/s (6386kB/s)(10.0MiB/1642msec) clat (msec): min=24, max=610, avg=156.12, stdev=225.84 lat (msec): min=24, max=610, avg=156.20, stdev=225.82 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 30], | 30.00th=[ 34], 40.00th=[ 35], 50.00th=[ 48], 60.00th=[ 65], | 70.00th=[ 93], 80.00th=[ 93], 90.00th=[ 550], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 1703, max=14336, per=45.93%, avg=8019.50, stdev=8932.88, samples=2 iops : min= 1, max= 14, avg= 7.50, stdev= 9.19, samples=2 lat (msec) : 20=8.33%, 50=50.00%, 100=25.00%, 750=16.67% cpu : usr=0.00%, sys=6.33%, ctx=60, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30338: Thu Dec 5 08:06:54 2024 read: IOPS=3, BW=3974KiB/s (4070kB/s)(6144KiB/1546msec) clat (usec): min=18221, max=61592, avg=36422.31, stdev=16672.06 lat (usec): min=18223, max=61593, avg=36423.33, stdev=16671.74 clat percentiles (usec): | 1.00th=[18220], 5.00th=[18220], 10.00th=[18220], 20.00th=[18744], | 30.00th=[18744], 40.00th=[34341], 50.00th=[34341], 60.00th=[39060], | 70.00th=[46400], 80.00th=[46400], 90.00th=[61604], 95.00th=[61604], | 99.00th=[61604], 99.50th=[61604], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 2048, max= 8192, per=41.05%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=3, BW=3974KiB/s (4070kB/s)(6144KiB/1546msec) clat (msec): min=28, max=598, avg=220.20, stdev=264.45 lat (msec): min=28, max=598, avg=220.25, stdev=264.46 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 35], | 30.00th=[ 35], 40.00th=[ 42], 50.00th=[ 42], 60.00th=[ 101], | 70.00th=[ 518], 80.00th=[ 518], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 4096, max= 6144, per=29.32%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 20=16.67%, 50=50.00%, 100=16.67%, 750=16.67% cpu : usr=0.06%, sys=4.53%, ctx=34, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30339: Thu Dec 5 08:06:54 2024 read: IOPS=3, BW=3598KiB/s (3684kB/s)(5120KiB/1423msec) clat (msec): min=18, max=463, avg=111.40, stdev=197.01 lat (msec): min=18, max=463, avg=111.40, stdev=197.01 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 27], 60.00th=[ 27], | 70.00th=[ 29], 80.00th=[ 29], 90.00th=[ 464], 95.00th=[ 464], | 99.00th=[ 464], 99.50th=[ 464], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 8192, max= 8192, per=65.68%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=4, BW=5037KiB/s (5158kB/s)(7168KiB/1423msec) clat (msec): min=25, max=579, avg=118.94, stdev=204.33 lat (msec): min=25, max=579, avg=119.00, stdev=204.33 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 31], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 34], 60.00th=[ 40], | 70.00th=[ 40], 80.00th=[ 92], 90.00th=[ 584], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 2043, max=12288, per=41.04%, avg=7165.50, stdev=7244.31, samples=2 iops : min= 1, max= 12, avg= 6.50, stdev= 7.78, samples=2 lat (msec) : 20=16.67%, 50=58.33%, 100=8.33%, 500=8.33%, 750=8.33% cpu : usr=0.14%, sys=5.27%, ctx=43, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.2MiB/s (12.8MB/s), 1247KiB/s-4694KiB/s (1277kB/s-4807kB/s), io=20.0MiB (20.0MB), run=1423-1642msec WRITE: bw=17.1MiB/s (17.9MB/s), 3353KiB/s-6236KiB/s (3433kB/s-6386kB/s), io=28.0MiB (29.4MB), run=1423-1642msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30340: Thu Dec 5 08:06:54 2024 read: IOPS=3, BW=3235KiB/s (3312kB/s)(7168KiB/2216msec) clat (usec): min=526, max=666382, avg=257628.92, stdev=276078.98 lat (usec): min=527, max=666383, avg=257630.25, stdev=276079.21 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 529], 10.00th=[ 529], 20.00th=[ 570], | 30.00th=[ 54789], 40.00th=[ 54789], 50.00th=[123208], 60.00th=[459277], | 70.00th=[459277], 80.00th=[501220], 90.00th=[666895], 95.00th=[666895], | 99.00th=[666895], 99.50th=[666895], 99.90th=[666895], 99.95th=[666895], | 99.99th=[666895] bw ( KiB/s): min= 2048, max= 6144, per=33.87%, avg=3072.00, stdev=2048.00, samples=4 iops : min= 2, max= 6, avg= 3.00, stdev= 2.00, samples=4 write: IOPS=2, BW=2310KiB/s (2366kB/s)(5120KiB/2216msec) clat (msec): min=22, max=272, avg=79.00, stdev=108.80 lat (msec): min=22, max=272, avg=79.06, stdev=108.81 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 23], | 30.00th=[ 23], 40.00th=[ 23], 50.00th=[ 25], 60.00th=[ 25], | 70.00th=[ 54], 80.00th=[ 54], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min=10240, max=10240, per=80.65%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 50=25.00%, 100=16.67%, 250=8.33%, 500=25.00%, 750=8.33% cpu : usr=0.00%, sys=23.79%, ctx=941, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30341: Thu Dec 5 08:06:54 2024 read: IOPS=0, BW=907KiB/s (929kB/s)(2048KiB/2258msec) clat (msec): min=39, max=529, avg=284.47, stdev=346.30 lat (msec): min=39, max=529, avg=284.47, stdev=346.30 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 40], | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 40], 60.00th=[ 531], | 70.00th=[ 531], 80.00th=[ 531], 90.00th=[ 531], 95.00th=[ 531], | 99.00th=[ 531], 99.50th=[ 531], 99.90th=[ 531], 99.95th=[ 531], | 99.99th=[ 531] bw ( KiB/s): min= 2043, max= 2048, per=22.55%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4535KiB/s (4644kB/s)(10.0MiB/2258msec) clat (msec): min=21, max=604, avg=163.75, stdev=224.34 lat (msec): min=21, max=604, avg=163.82, stdev=224.34 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 23], | 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 46], 60.00th=[ 64], | 70.00th=[ 138], 80.00th=[ 138], 90.00th=[ 567], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 2043, max=10240, per=42.98%, avg=5456.67, stdev=4266.70, samples=3 iops : min= 1, max= 10, avg= 4.67, stdev= 4.73, samples=3 lat (msec) : 50=50.00%, 100=16.67%, 250=8.33%, 750=25.00% cpu : usr=0.00%, sys=15.94%, ctx=416, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30342: Thu Dec 5 08:06:54 2024 read: IOPS=2, BW=2897KiB/s (2966kB/s)(6144KiB/2121msec) clat (usec): min=588, max=589813, avg=207873.50, stdev=275978.49 lat (usec): min=590, max=589815, avg=207874.58, stdev=275978.49 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 586], 10.00th=[ 586], 20.00th=[ 4621], | 30.00th=[ 4621], 40.00th=[ 51643], 50.00th=[ 51643], 60.00th=[ 66323], | 70.00th=[534774], 80.00th=[534774], 90.00th=[591397], 95.00th=[591397], | 99.00th=[591397], 99.50th=[591397], 99.90th=[591397], 99.95th=[591397], | 99.99th=[591397] bw ( KiB/s): min= 4096, max= 6144, per=56.46%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2897KiB/s (2966kB/s)(6144KiB/2121msec) clat (msec): min=31, max=585, avg=137.67, stdev=220.67 lat (msec): min=31, max=586, avg=137.72, stdev=220.66 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 37], 50.00th=[ 37], 60.00th=[ 53], | 70.00th=[ 88], 80.00th=[ 88], 90.00th=[ 584], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 2048, max= 4096, per=24.18%, avg=3069.75, stdev=1179.82, samples=4 iops : min= 2, max= 4, avg= 2.75, stdev= 0.96, samples=4 lat (usec) : 750=8.33% lat (msec) : 10=8.33%, 50=25.00%, 100=33.33%, 750=25.00% cpu : usr=0.00%, sys=17.17%, ctx=674, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30343: Thu Dec 5 08:06:54 2024 read: IOPS=2, BW=2404KiB/s (2461kB/s)(5120KiB/2130msec) clat (usec): min=303, max=558951, avg=230511.68, stdev=283189.82 lat (usec): min=303, max=558952, avg=230512.74, stdev=283189.92 clat percentiles (usec): | 1.00th=[ 306], 5.00th=[ 306], 10.00th=[ 306], 20.00th=[ 306], | 30.00th=[ 3982], 40.00th=[ 3982], 50.00th=[ 70779], 60.00th=[ 70779], | 70.00th=[517997], 80.00th=[517997], 90.00th=[557843], 95.00th=[557843], | 99.00th=[557843], 99.50th=[557843], 99.90th=[557843], 99.95th=[557843], | 99.99th=[557843] bw ( KiB/s): min= 2043, max= 6144, per=45.14%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 write: IOPS=3, BW=3365KiB/s (3446kB/s)(7168KiB/2130msec) clat (msec): min=22, max=604, avg=134.57, stdev=208.88 lat (msec): min=22, max=604, avg=134.63, stdev=208.89 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 40], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 48], 60.00th=[ 89], | 70.00th=[ 89], 80.00th=[ 96], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 2048, max= 6144, per=37.60%, avg=4774.33, stdev=2361.08, samples=3 iops : min= 2, max= 6, avg= 4.33, stdev= 2.08, samples=3 lat (usec) : 500=8.33% lat (msec) : 4=8.33%, 50=33.33%, 100=25.00%, 750=25.00% cpu : usr=0.00%, sys=17.14%, ctx=682, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9070KiB/s (9288kB/s), 907KiB/s-3235KiB/s (929kB/s-3312kB/s), io=20.0MiB (20.0MB), run=2121-2258msec WRITE: bw=12.4MiB/s (13.0MB/s), 2310KiB/s-4535KiB/s (2366kB/s-4644kB/s), io=28.0MiB (29.4MB), run=2121-2258msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30350: Thu Dec 5 08:06:56 2024 read: IOPS=9, BW=37.0MiB/s (39.8MB/s)(12.0MiB/316msec) clat (msec): min=33, max=131, avg=98.26, stdev=55.72 lat (msec): min=33, max=131, avg=98.28, stdev=55.73 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34], | 30.00th=[ 34], 40.00th=[ 129], 50.00th=[ 129], 60.00th=[ 129], | 70.00th=[ 132], 80.00th=[ 132], 90.00th=[ 132], 95.00th=[ 132], | 99.00th=[ 132], 99.50th=[ 132], 99.90th=[ 132], 99.95th=[ 132], | 99.99th=[ 132] lat (msec) : 50=33.33%, 250=66.67% cpu : usr=0.95%, sys=36.83%, ctx=137, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30352: Thu Dec 5 08:06:56 2024 read: IOPS=1, BW=7354KiB/s (7530kB/s)(4096KiB/557msec) clat (nsec): min=124909k, max=124909k, avg=124908640.00, stdev= 0.00 lat (nsec): min=124910k, max=124910k, avg=124909915.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 126], 10.00th=[ 126], 20.00th=[ 126], | 30.00th=[ 126], 40.00th=[ 126], 50.00th=[ 126], 60.00th=[ 126], | 70.00th=[ 126], 80.00th=[ 126], 90.00th=[ 126], 95.00th=[ 126], | 99.00th=[ 126], 99.50th=[ 126], 99.90th=[ 126], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 8192, max= 8192, per=27.85%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=14.4MiB/s (15.1MB/s)(8192KiB/557msec) clat (msec): min=171, max=210, avg=191.21, stdev=27.94 lat (msec): min=171, max=211, avg=191.46, stdev=27.96 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 171], 50.00th=[ 171], 60.00th=[ 211], | 70.00th=[ 211], 80.00th=[ 211], 90.00th=[ 211], 95.00th=[ 211], | 99.00th=[ 211], 99.50th=[ 211], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 8192, max= 8192, per=18.18%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=45.32%, ctx=358, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30353: Thu Dec 5 08:06:56 2024 write: IOPS=4, BW=16.5MiB/s (17.3MB/s)(12.0MiB/727msec) clat (msec): min=113, max=312, avg=230.05, stdev=103.63 lat (msec): min=113, max=312, avg=230.42, stdev=103.69 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 114], 10.00th=[ 114], 20.00th=[ 114], | 30.00th=[ 114], 40.00th=[ 266], 50.00th=[ 266], 60.00th=[ 266], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 313], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 8175, max= 8175, per=18.14%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=35.95%, ctx=75, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30355: Thu Dec 5 08:06:56 2024 write: IOPS=5, BW=21.5MiB/s (22.5MB/s)(12.0MiB/559msec) clat (msec): min=119, max=268, avg=176.70, stdev=80.01 lat (msec): min=120, max=268, avg=176.99, stdev=79.88 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 121], 10.00th=[ 121], 20.00th=[ 121], | 30.00th=[ 121], 40.00th=[ 142], 50.00th=[ 142], 60.00th=[ 142], | 70.00th=[ 268], 80.00th=[ 268], 90.00th=[ 268], 95.00th=[ 268], | 99.00th=[ 268], 99.50th=[ 268], 99.90th=[ 268], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min=16384, max=16384, per=36.35%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=53.58%, ctx=85, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=28.7MiB/s (30.1MB/s), 7354KiB/s-37.0MiB/s (7530kB/s-39.8MB/s), io=16.0MiB (16.8MB), run=316-557msec WRITE: bw=44.0MiB/s (46.2MB/s), 14.4MiB/s-21.5MiB/s (15.1MB/s-22.5MB/s), io=32.0MiB (33.6MB), run=557-727msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30354: Thu Dec 5 08:06:56 2024 read: IOPS=7, BW=29.3MiB/s (30.8MB/s)(12.0MiB/409msec) clat (msec): min=57, max=179, avg=130.88, stdev=64.41 lat (msec): min=57, max=179, avg=130.88, stdev=64.41 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 58], 10.00th=[ 58], 20.00th=[ 58], | 30.00th=[ 58], 40.00th=[ 157], 50.00th=[ 157], 60.00th=[ 157], | 70.00th=[ 180], 80.00th=[ 180], 90.00th=[ 180], 95.00th=[ 180], | 99.00th=[ 180], 99.50th=[ 180], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.25%, sys=18.38%, ctx=60, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30356: Thu Dec 5 08:06:56 2024 read: IOPS=1, BW=5513KiB/s (5645kB/s)(4096KiB/743msec) clat (nsec): min=385010k, max=385010k, avg=385009770.00, stdev= 0.00 lat (nsec): min=385011k, max=385011k, avg=385010903.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 384], 10.00th=[ 384], 20.00th=[ 384], | 30.00th=[ 384], 40.00th=[ 384], 50.00th=[ 384], 60.00th=[ 384], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 8192, max= 8192, per=37.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.8MiB/s (11.3MB/s)(8192KiB/743msec) clat (msec): min=102, max=208, avg=155.36, stdev=74.86 lat (msec): min=102, max=208, avg=155.62, stdev=74.91 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 103], 10.00th=[ 103], 20.00th=[ 103], | 30.00th=[ 103], 40.00th=[ 103], 50.00th=[ 103], 60.00th=[ 209], | 70.00th=[ 209], 80.00th=[ 209], 90.00th=[ 209], 95.00th=[ 209], | 99.00th=[ 209], 99.50th=[ 209], 99.90th=[ 209], 99.95th=[ 209], | 99.99th=[ 209] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=11.31%, ctx=54, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30357: Thu Dec 5 08:06:56 2024 write: IOPS=3, BW=14.8MiB/s (15.5MB/s)(12.0MiB/810msec) clat (msec): min=93, max=542, avg=254.55, stdev=249.89 lat (msec): min=93, max=542, avg=254.80, stdev=249.96 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 93], 10.00th=[ 93], 20.00th=[ 93], | 30.00th=[ 93], 40.00th=[ 129], 50.00th=[ 129], 60.00th=[ 129], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] lat (msec) : 100=33.33%, 250=33.33%, 750=33.33% cpu : usr=0.12%, sys=12.36%, ctx=104, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30358: Thu Dec 5 08:06:56 2024 write: IOPS=4, BW=16.0MiB/s (17.8MB/s)(12.0MiB/707msec) clat (msec): min=140, max=287, avg=210.55, stdev=73.59 lat (msec): min=141, max=287, avg=210.81, stdev=73.49 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 142], 10.00th=[ 142], 20.00th=[ 142], | 30.00th=[ 142], 40.00th=[ 203], 50.00th=[ 203], 60.00th=[ 203], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min=16384, max=16384, per=40.50%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.14%, sys=15.58%, ctx=73, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=21.5MiB/s (22.6MB/s), 5513KiB/s-29.3MiB/s (5645kB/s-30.8MB/s), io=16.0MiB (16.8MB), run=409-743msec WRITE: bw=39.5MiB/s (41.4MB/s), 10.8MiB/s-16.0MiB/s (11.3MB/s-17.8MB/s), io=32.0MiB (33.6MB), run=707-810msec