== sanity test 398b: DIO and buffer IO race ============== 00:54:03 (1718859243) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.37637 s, 36.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20516: Thu Jun 20 00:54:22 2024 read: IOPS=93, BW=372KiB/s (381kB/s)(5992KiB/16102msec) clat (usec): min=49, max=216488, avg=5105.19, stdev=6962.03 lat (usec): min=49, max=216490, avg=5106.17, stdev=6962.10 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 77], 10.00th=[ 103], 20.00th=[ 2606], | 30.00th=[ 3032], 40.00th=[ 3490], 50.00th=[ 4015], 60.00th=[ 4883], | 70.00th=[ 6128], 80.00th=[ 7635], 90.00th=[ 9503], 95.00th=[ 11338], | 99.00th=[ 20317], 99.50th=[ 26346], 99.90th=[ 61080], 99.95th=[217056], | 99.99th=[217056] bw ( KiB/s): min= 184, max= 496, per=24.07%, avg=358.84, stdev=78.20, samples=31 iops : min= 46, max= 124, avg=89.61, stdev=19.54, samples=31 write: IOPS=97, BW=391KiB/s (400kB/s)(6296KiB/16102msec) clat (usec): min=489, max=56347, avg=5295.68, stdev=5393.13 lat (usec): min=490, max=56347, avg=5296.81, stdev=5393.16 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 627], 10.00th=[ 717], 20.00th=[ 873], | 30.00th=[ 1205], 40.00th=[ 2966], 50.00th=[ 4228], 60.00th=[ 5342], | 70.00th=[ 6587], 80.00th=[ 8455], 90.00th=[11863], 95.00th=[15270], | 99.00th=[21365], 99.50th=[33817], 99.90th=[50070], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 216, max= 560, per=24.21%, avg=377.94, stdev=76.40, samples=31 iops : min= 54, max= 140, avg=94.39, stdev=19.12, samples=31 lat (usec) : 50=0.03%, 100=4.43%, 250=3.03%, 500=0.20%, 750=6.28% lat (usec) : 1000=7.16% lat (msec) : 2=4.92%, 4=22.33%, 10=40.56%, 20=9.86%, 50=1.07% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.31%, sys=24.07%, ctx=6409, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20517: Thu Jun 20 00:54:22 2024 read: IOPS=93, BW=375KiB/s (384kB/s)(5940KiB/15860msec) clat (usec): min=49, max=57913, avg=4757.09, stdev=4162.77 lat (usec): min=49, max=57914, avg=4758.13, stdev=4162.76 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 83], 10.00th=[ 110], 20.00th=[ 2606], | 30.00th=[ 2999], 40.00th=[ 3392], 50.00th=[ 3884], 60.00th=[ 4555], | 70.00th=[ 5604], 80.00th=[ 7177], 90.00th=[ 8979], 95.00th=[11076], | 99.00th=[16057], 99.50th=[21103], 99.90th=[57410], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 215, max= 520, per=25.04%, avg=373.35, stdev=80.21, samples=31 iops : min= 53, max= 130, avg=93.29, stdev=20.14, samples=31 write: IOPS=100, BW=400KiB/s (410kB/s)(6348KiB/15860msec) clat (usec): min=439, max=213983, avg=5471.77, stdev=7518.18 lat (usec): min=440, max=213984, avg=5472.85, stdev=7518.14 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 627], 10.00th=[ 709], 20.00th=[ 898], | 30.00th=[ 1303], 40.00th=[ 3097], 50.00th=[ 4293], 60.00th=[ 5473], | 70.00th=[ 6587], 80.00th=[ 8455], 90.00th=[ 11469], 95.00th=[ 14615], | 99.00th=[ 25035], 99.50th=[ 32113], 99.90th=[ 62653], 99.95th=[214959], | 99.99th=[214959] bw ( KiB/s): min= 231, max= 528, per=25.22%, avg=393.74, stdev=76.85, samples=31 iops : min= 57, max= 132, avg=98.39, stdev=19.31, samples=31 lat (usec) : 50=0.03%, 100=3.84%, 250=3.03%, 500=0.13%, 750=6.25% lat (usec) : 1000=6.38% lat (msec) : 2=5.70%, 4=24.06%, 10=39.84%, 20=9.47%, 50=1.11% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.24%, sys=24.42%, ctx=6372, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20518: Thu Jun 20 00:54:22 2024 read: IOPS=94, BW=380KiB/s (389kB/s)(6028KiB/15866msec) clat (usec): min=49, max=56575, avg=4829.59, stdev=4111.69 lat (usec): min=50, max=56575, avg=4830.52, stdev=4111.76 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 83], 10.00th=[ 104], 20.00th=[ 2507], | 30.00th=[ 2966], 40.00th=[ 3392], 50.00th=[ 3949], 60.00th=[ 4621], | 70.00th=[ 5932], 80.00th=[ 7504], 90.00th=[ 9634], 95.00th=[11338], | 99.00th=[15401], 99.50th=[21890], 99.90th=[47973], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 136, max= 504, per=25.21%, avg=375.84, stdev=81.71, samples=31 iops : min= 34, max= 126, avg=93.87, stdev=20.42, samples=31 write: IOPS=98, BW=395KiB/s (404kB/s)(6260KiB/15866msec) clat (usec): min=486, max=234204, avg=5414.18, stdev=7965.99 lat (usec): min=487, max=234205, avg=5415.19, stdev=7965.99 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 619], 10.00th=[ 734], 20.00th=[ 881], | 30.00th=[ 1254], 40.00th=[ 2900], 50.00th=[ 4293], 60.00th=[ 5276], | 70.00th=[ 6456], 80.00th=[ 8586], 90.00th=[ 11469], 95.00th=[ 14484], | 99.00th=[ 23200], 99.50th=[ 35914], 99.90th=[ 60031], 99.95th=[233833], | 99.99th=[233833] bw ( KiB/s): min= 176, max= 488, per=24.92%, avg=389.03, stdev=75.66, samples=31 iops : min= 44, max= 122, avg=97.16, stdev=18.92, samples=31 lat (usec) : 50=0.03%, 100=4.46%, 250=3.26%, 500=0.20%, 750=5.66% lat (usec) : 1000=7.10% lat (msec) : 2=5.66%, 4=22.79%, 10=39.58%, 20=10.25%, 50=0.85% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.30%, sys=24.28%, ctx=6415, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20519: Thu Jun 20 00:54:22 2024 read: IOPS=95, BW=383KiB/s (392kB/s)(6052KiB/15819msec) clat (usec): min=49, max=58138, avg=4884.84, stdev=4186.51 lat (usec): min=49, max=58139, avg=4885.76, stdev=4186.49 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 77], 10.00th=[ 102], 20.00th=[ 2573], | 30.00th=[ 3032], 40.00th=[ 3490], 50.00th=[ 4113], 60.00th=[ 4817], | 70.00th=[ 5932], 80.00th=[ 7373], 90.00th=[ 9503], 95.00th=[10945], | 99.00th=[16712], 99.50th=[21890], 99.90th=[52691], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 208, max= 544, per=25.30%, avg=377.23, stdev=85.37, samples=31 iops : min= 52, max= 136, avg=94.26, stdev=21.38, samples=31 write: IOPS=98, BW=394KiB/s (404kB/s)(6236KiB/15819msec) clat (usec): min=474, max=213911, avg=5334.24, stdev=7386.58 lat (usec): min=474, max=213912, avg=5335.69, stdev=7386.75 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 619], 10.00th=[ 685], 20.00th=[ 848], | 30.00th=[ 1106], 40.00th=[ 2573], 50.00th=[ 4178], 60.00th=[ 5276], | 70.00th=[ 6587], 80.00th=[ 8586], 90.00th=[ 11731], 95.00th=[ 15008], | 99.00th=[ 23200], 99.50th=[ 31589], 99.90th=[ 45351], 99.95th=[214959], | 99.99th=[214959] bw ( KiB/s): min= 224, max= 552, per=24.94%, avg=389.35, stdev=85.27, samples=31 iops : min= 56, max= 138, avg=97.29, stdev=21.33, samples=31 lat (usec) : 50=0.07%, 100=4.65%, 250=2.86%, 500=0.20%, 750=7.03% lat (usec) : 1000=6.80% lat (msec) : 2=5.53%, 4=21.03%, 10=40.59%, 20=10.22%, 50=0.91% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.26%, sys=24.24%, ctx=6354, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1491KiB/s (1527kB/s), 372KiB/s-383KiB/s (381kB/s-392kB/s), io=23.4MiB (24.6MB), run=15819-16102msec WRITE: bw=1561KiB/s (1599kB/s), 391KiB/s-400KiB/s (400kB/s-410kB/s), io=24.6MiB (25.7MB), run=15819-16102msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20512: Thu Jun 20 00:54:57 2024 read: IOPS=29, BW=117KiB/s (120kB/s)(5992KiB/51204msec) clat (usec): min=1701, max=244364, avg=4904.57, stdev=7302.07 lat (usec): min=1702, max=244367, avg=4905.79, stdev=7302.08 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 3], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 5], | 70.00th=[ 5], 80.00th=[ 6], 90.00th=[ 9], 95.00th=[ 12], | 99.00th=[ 21], 99.50th=[ 26], 99.90th=[ 64], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 40, max= 240, per=25.03%, avg=116.91, stdev=46.29, samples=102 iops : min= 10, max= 60, avg=29.15, stdev=11.60, samples=102 write: IOPS=30, BW=123KiB/s (126kB/s)(6296KiB/51204msec) clat (usec): min=13580, max=76477, avg=27780.88, stdev=9193.99 lat (usec): min=13581, max=76479, avg=27782.33, stdev=9193.95 clat percentiles (usec): | 1.00th=[15139], 5.00th=[16712], 10.00th=[17957], 20.00th=[19530], | 30.00th=[21627], 40.00th=[24511], 50.00th=[26608], 60.00th=[28443], | 70.00th=[30540], 80.00th=[33424], 90.00th=[40109], 95.00th=[46400], | 99.00th=[56886], 99.50th=[62653], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 16, max= 168, per=25.08%, avg=122.64, stdev=31.19, samples=102 iops : min= 4, max= 42, avg=30.58, stdev= 7.81, samples=102 lat (msec) : 2=0.33%, 4=26.73%, 10=18.49%, 20=13.83%, 50=39.19% lat (msec) : 100=1.40%, 250=0.03% cpu : usr=0.12%, sys=4.69%, ctx=3886, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20513: Thu Jun 20 00:54:57 2024 read: IOPS=28, BW=116KiB/s (118kB/s)(5940KiB/51378msec) clat (usec): min=1711, max=217351, avg=4910.52, stdev=6903.29 lat (usec): min=1712, max=217353, avg=4911.65, stdev=6903.29 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2147], 10.00th=[ 2311], 20.00th=[ 2540], | 30.00th=[ 2769], 40.00th=[ 3032], 50.00th=[ 3556], 60.00th=[ 4293], | 70.00th=[ 4883], 80.00th=[ 5800], 90.00th=[ 8029], 95.00th=[ 11338], | 99.00th=[ 20579], 99.50th=[ 30802], 99.90th=[ 62653], 99.95th=[217056], | 99.99th=[217056] bw ( KiB/s): min= 8, max= 224, per=24.72%, avg=115.42, stdev=49.33, samples=102 iops : min= 2, max= 56, avg=28.77, stdev=12.34, samples=102 write: IOPS=30, BW=124KiB/s (127kB/s)(6348KiB/51378msec) clat (usec): min=13163, max=80379, avg=27699.23, stdev=9255.06 lat (usec): min=13164, max=80381, avg=27700.71, stdev=9254.98 clat percentiles (usec): | 1.00th=[15008], 5.00th=[17171], 10.00th=[18220], 20.00th=[19792], | 30.00th=[21627], 40.00th=[23725], 50.00th=[26084], 60.00th=[28181], | 70.00th=[30278], 80.00th=[33424], 90.00th=[39584], 95.00th=[45351], | 99.00th=[58459], 99.50th=[64226], 99.90th=[76022], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 32, max= 168, per=25.12%, avg=122.81, stdev=30.21, samples=102 iops : min= 8, max= 42, avg=30.63, stdev= 7.57, samples=102 lat (msec) : 2=0.91%, 4=25.78%, 10=18.59%, 20=13.28%, 50=39.68% lat (msec) : 100=1.73%, 250=0.03% cpu : usr=0.14%, sys=4.68%, ctx=3894, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20514: Thu Jun 20 00:54:57 2024 read: IOPS=29, BW=118KiB/s (121kB/s)(6028KiB/51075msec) clat (usec): min=1654, max=42360, avg=4798.31, stdev=3672.90 lat (usec): min=1655, max=42360, avg=4799.50, stdev=3672.99 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2147], 10.00th=[ 2311], 20.00th=[ 2573], | 30.00th=[ 2802], 40.00th=[ 3163], 50.00th=[ 3720], 60.00th=[ 4359], | 70.00th=[ 4948], 80.00th=[ 5932], 90.00th=[ 8455], 95.00th=[11600], | 99.00th=[19006], 99.50th=[24249], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 31, max= 256, per=25.23%, avg=117.84, stdev=47.61, samples=101 iops : min= 7, max= 64, avg=29.39, stdev=11.93, samples=101 write: IOPS=30, BW=123KiB/s (126kB/s)(6260KiB/51075msec) clat (msec): min=13, max=233, avg=27.93, stdev=10.51 lat (msec): min=13, max=233, avg=27.93, stdev=10.51 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 20], | 30.00th=[ 22], 40.00th=[ 25], 50.00th=[ 27], 60.00th=[ 29], | 70.00th=[ 31], 80.00th=[ 34], 90.00th=[ 41], 95.00th=[ 47], | 99.00th=[ 57], 99.50th=[ 64], 99.90th=[ 83], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 32, max= 176, per=24.94%, avg=121.96, stdev=30.28, samples=101 iops : min= 8, max= 44, avg=30.42, stdev= 7.63, samples=101 lat (msec) : 2=0.94%, 4=25.78%, 10=19.08%, 20=13.77%, 50=39.03% lat (msec) : 100=1.37%, 250=0.03% cpu : usr=0.11%, sys=4.72%, ctx=3887, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20515: Thu Jun 20 00:54:57 2024 read: IOPS=29, BW=120KiB/s (123kB/s)(6052KiB/50585msec) clat (usec): min=1741, max=43930, avg=4849.80, stdev=3810.47 lat (usec): min=1742, max=43931, avg=4851.03, stdev=3810.47 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2212], 10.00th=[ 2343], 20.00th=[ 2540], | 30.00th=[ 2769], 40.00th=[ 3064], 50.00th=[ 3621], 60.00th=[ 4424], | 70.00th=[ 5080], 80.00th=[ 6063], 90.00th=[ 8586], 95.00th=[12125], | 99.00th=[19268], 99.50th=[23987], 99.90th=[42206], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 8, max= 232, per=25.60%, avg=119.55, stdev=45.22, samples=101 iops : min= 2, max= 58, avg=29.79, stdev=11.36, samples=101 write: IOPS=30, BW=123KiB/s (126kB/s)(6236KiB/50585msec) clat (msec): min=13, max=245, avg=27.66, stdev=10.44 lat (msec): min=13, max=245, avg=27.66, stdev=10.44 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 20], | 30.00th=[ 22], 40.00th=[ 24], 50.00th=[ 27], 60.00th=[ 29], | 70.00th=[ 31], 80.00th=[ 34], 90.00th=[ 40], 95.00th=[ 46], | 99.00th=[ 57], 99.50th=[ 62], 99.90th=[ 78], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 39, max= 160, per=25.16%, avg=123.04, stdev=29.20, samples=101 iops : min= 9, max= 40, avg=30.66, stdev= 7.38, samples=101 lat (msec) : 2=0.62%, 4=26.01%, 10=18.91%, 20=14.58%, 50=38.61% lat (msec) : 100=1.24%, 250=0.03% cpu : usr=0.12%, sys=4.73%, ctx=3891, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=467KiB/s (479kB/s), 116KiB/s-120KiB/s (118kB/s-123kB/s), io=23.4MiB (24.6MB), run=50585-51378msec WRITE: bw=489KiB/s (501kB/s), 123KiB/s-124KiB/s (126kB/s-127kB/s), io=24.6MiB (25.7MB), run=50585-51378msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20577: Thu Jun 20 00:55:03 2024 read: IOPS=72, BW=1168KiB/s (1196kB/s)(5920KiB/5069msec) clat (usec): min=46, max=34270, avg=9942.48, stdev=6073.12 lat (usec): min=46, max=34270, avg=9943.23, stdev=6073.10 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 78], 10.00th=[ 88], 20.00th=[ 113], | 30.00th=[ 9503], 40.00th=[10552], 50.00th=[11469], 60.00th=[12256], | 70.00th=[12911], 80.00th=[14091], 90.00th=[15795], 95.00th=[17957], | 99.00th=[25822], 99.50th=[26608], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 832, max= 1504, per=25.04%, avg=1161.30, stdev=199.76, samples=10 iops : min= 52, max= 94, avg=72.50, stdev=12.54, samples=10 write: IOPS=78, BW=1256KiB/s (1286kB/s)(6368KiB/5069msec) clat (usec): min=558, max=28025, avg=3470.35, stdev=3841.92 lat (usec): min=559, max=28025, avg=3472.07, stdev=3842.32 clat percentiles (usec): | 1.00th=[ 635], 5.00th=[ 758], 10.00th=[ 840], 20.00th=[ 963], | 30.00th=[ 1074], 40.00th=[ 1188], 50.00th=[ 1565], 60.00th=[ 3032], | 70.00th=[ 3851], 80.00th=[ 5669], 90.00th=[ 7767], 95.00th=[10028], | 99.00th=[20841], 99.50th=[25297], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 864, max= 1888, per=25.71%, avg=1254.10, stdev=307.94, samples=10 iops : min= 54, max= 118, avg=78.30, stdev=19.24, samples=10 lat (usec) : 50=0.39%, 100=7.03%, 250=3.26%, 500=0.26%, 750=2.47% lat (usec) : 1000=9.51% lat (msec) : 2=14.97%, 4=10.55%, 10=17.45%, 20=32.42%, 50=1.69% cpu : usr=0.14%, sys=21.84%, ctx=1923, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20578: Thu Jun 20 00:55:03 2024 read: IOPS=70, BW=1129KiB/s (1156kB/s)(5744KiB/5089msec) clat (usec): min=44, max=47082, avg=9499.57, stdev=6659.97 lat (usec): min=45, max=47082, avg=9500.33, stdev=6659.99 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 69], 10.00th=[ 82], 20.00th=[ 99], | 30.00th=[ 8717], 40.00th=[10159], 50.00th=[11338], 60.00th=[11994], | 70.00th=[13173], 80.00th=[14353], 90.00th=[16450], 95.00th=[18220], | 99.00th=[24249], 99.50th=[25822], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 383, max= 1536, per=24.08%, avg=1116.70, stdev=303.27, samples=10 iops : min= 23, max= 96, avg=69.70, stdev=19.21, samples=10 write: IOPS=80, BW=1286KiB/s (1317kB/s)(6544KiB/5089msec) clat (usec): min=576, max=39398, avg=4084.45, stdev=5767.70 lat (usec): min=577, max=39399, avg=4085.76, stdev=5767.66 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 758], 10.00th=[ 832], 20.00th=[ 947], | 30.00th=[ 1057], 40.00th=[ 1172], 50.00th=[ 2278], 60.00th=[ 3458], | 70.00th=[ 3884], 80.00th=[ 5669], 90.00th=[ 8291], 95.00th=[13960], | 99.00th=[34341], 99.50th=[36439], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 638, max= 1824, per=25.65%, avg=1251.00, stdev=377.92, samples=10 iops : min= 39, max= 114, avg=78.10, stdev=23.78, samples=10 lat (usec) : 50=0.52%, 100=8.98%, 250=3.12%, 500=0.26%, 750=2.60% lat (usec) : 1000=11.07% lat (msec) : 2=12.50%, 4=11.46%, 10=17.84%, 20=28.78%, 50=2.86% cpu : usr=0.18%, sys=20.95%, ctx=1756, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20579: Thu Jun 20 00:55:03 2024 read: IOPS=75, BW=1215KiB/s (1244kB/s)(6256KiB/5149msec) clat (usec): min=44, max=41191, avg=9864.04, stdev=6296.24 lat (usec): min=44, max=41192, avg=9864.82, stdev=6296.17 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 70], 10.00th=[ 89], 20.00th=[ 109], | 30.00th=[ 9241], 40.00th=[10421], 50.00th=[11207], 60.00th=[12387], | 70.00th=[13304], 80.00th=[14484], 90.00th=[16188], 95.00th=[17957], | 99.00th=[22676], 99.50th=[23987], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 926, max= 1696, per=26.01%, avg=1206.20, stdev=258.91, samples=10 iops : min= 57, max= 106, avg=75.30, stdev=16.29, samples=10 write: IOPS=73, BW=1171KiB/s (1200kB/s)(6032KiB/5149msec) clat (usec): min=633, max=47237, avg=3407.74, stdev=4511.61 lat (usec): min=634, max=47239, avg=3409.07, stdev=4511.62 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 783], 10.00th=[ 857], 20.00th=[ 979], | 30.00th=[ 1074], 40.00th=[ 1156], 50.00th=[ 1450], 60.00th=[ 3195], | 70.00th=[ 3916], 80.00th=[ 5276], 90.00th=[ 7111], 95.00th=[ 9110], | 99.00th=[16188], 99.50th=[45351], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 736, max= 1952, per=23.15%, avg=1129.30, stdev=366.01, samples=10 iops : min= 46, max= 122, avg=70.50, stdev=22.85, samples=10 lat (usec) : 50=0.13%, 100=8.46%, 250=3.65%, 750=1.56%, 1000=9.37% lat (msec) : 2=14.84%, 4=9.11%, 10=18.36%, 20=32.94%, 50=1.56% cpu : usr=0.10%, sys=21.23%, ctx=2015, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20580: Thu Jun 20 00:55:03 2024 read: IOPS=72, BW=1168KiB/s (1196kB/s)(6032KiB/5165msec) clat (usec): min=46, max=24808, avg=9830.73, stdev=6235.13 lat (usec): min=46, max=24808, avg=9831.47, stdev=6235.17 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 69], 10.00th=[ 85], 20.00th=[ 110], | 30.00th=[ 8848], 40.00th=[10290], 50.00th=[11469], 60.00th=[12387], | 70.00th=[13304], 80.00th=[14615], 90.00th=[16057], 95.00th=[18220], | 99.00th=[23725], 99.50th=[24773], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 768, max= 1568, per=24.61%, avg=1141.30, stdev=238.90, samples=10 iops : min= 48, max= 98, avg=71.00, stdev=14.94, samples=10 write: IOPS=75, BW=1211KiB/s (1240kB/s)(6256KiB/5165msec) clat (usec): min=554, max=33997, avg=3710.91, stdev=4716.49 lat (usec): min=555, max=33999, avg=3712.17, stdev=4716.53 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 766], 10.00th=[ 832], 20.00th=[ 938], | 30.00th=[ 1057], 40.00th=[ 1172], 50.00th=[ 1352], 60.00th=[ 2900], | 70.00th=[ 3884], 80.00th=[ 5735], 90.00th=[ 8717], 95.00th=[12780], | 99.00th=[26084], 99.50th=[29230], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 800, max= 1440, per=24.45%, avg=1192.50, stdev=207.32, samples=10 iops : min= 50, max= 90, avg=74.20, stdev=13.13, samples=10 lat (usec) : 50=0.26%, 100=7.94%, 250=3.39%, 500=0.13%, 750=2.34% lat (usec) : 1000=11.72% lat (msec) : 2=14.32%, 4=8.20%, 10=17.19%, 20=32.16%, 50=2.34% cpu : usr=0.06%, sys=20.97%, ctx=1954, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4637KiB/s (4749kB/s), 1129KiB/s-1215KiB/s (1156kB/s-1244kB/s), io=23.4MiB (24.5MB), run=5069-5165msec WRITE: bw=4879KiB/s (4996kB/s), 1171KiB/s-1286KiB/s (1200kB/s-1317kB/s), io=24.6MiB (25.8MB), run=5069-5165msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20573: Thu Jun 20 00:55:10 2024 read: IOPS=30, BW=493KiB/s (505kB/s)(5920KiB/12011msec) clat (usec): min=1870, max=24929, avg=4540.20, stdev=3878.15 lat (usec): min=1870, max=24930, avg=4541.23, stdev=3878.17 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2212], 10.00th=[ 2376], 20.00th=[ 2474], | 30.00th=[ 2737], 40.00th=[ 2868], 50.00th=[ 3130], 60.00th=[ 3359], | 70.00th=[ 3752], 80.00th=[ 5080], 90.00th=[ 8586], 95.00th=[14877], | 99.00th=[19530], 99.50th=[24773], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 288, max= 864, per=24.41%, avg=485.43, stdev=158.59, samples=23 iops : min= 18, max= 54, avg=30.26, stdev= 9.86, samples=23 write: IOPS=33, BW=530KiB/s (543kB/s)(6368KiB/12011msec) clat (usec): min=13507, max=54534, avg=25932.83, stdev=7945.55 lat (usec): min=13509, max=54535, avg=25934.73, stdev=7945.51 clat percentiles (usec): | 1.00th=[14484], 5.00th=[15795], 10.00th=[16712], 20.00th=[18482], | 30.00th=[20579], 40.00th=[23200], 50.00th=[25035], 60.00th=[26870], | 70.00th=[28967], 80.00th=[31589], 90.00th=[36439], 95.00th=[41681], | 99.00th=[51119], 99.50th=[52691], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 383, max= 702, per=24.86%, avg=520.22, stdev=112.93, samples=23 iops : min= 23, max= 43, avg=32.43, stdev= 7.05, samples=23 lat (msec) : 2=0.65%, 4=34.51%, 10=8.85%, 20=18.10%, 50=37.24% lat (msec) : 100=0.65% cpu : usr=0.09%, sys=5.21%, ctx=899, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20574: Thu Jun 20 00:55:10 2024 read: IOPS=29, BW=477KiB/s (489kB/s)(5744KiB/12040msec) clat (usec): min=1897, max=30930, avg=4605.68, stdev=4437.36 lat (usec): min=1898, max=30931, avg=4606.66, stdev=4437.33 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2180], 10.00th=[ 2311], 20.00th=[ 2474], | 30.00th=[ 2606], 40.00th=[ 2737], 50.00th=[ 2999], 60.00th=[ 3228], | 70.00th=[ 3654], 80.00th=[ 5211], 90.00th=[ 9765], 95.00th=[13960], | 99.00th=[25297], 99.50th=[30540], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 192, max= 864, per=23.79%, avg=473.25, stdev=189.60, samples=24 iops : min= 12, max= 54, avg=29.54, stdev=11.82, samples=24 write: IOPS=33, BW=544KiB/s (557kB/s)(6544KiB/12040msec) clat (usec): min=12929, max=57997, avg=25372.00, stdev=7545.81 lat (usec): min=12930, max=57998, avg=25373.82, stdev=7545.78 clat percentiles (usec): | 1.00th=[13304], 5.00th=[15533], 10.00th=[16450], 20.00th=[18482], | 30.00th=[20841], 40.00th=[22676], 50.00th=[24511], 60.00th=[26346], | 70.00th=[28705], 80.00th=[31065], 90.00th=[34866], 95.00th=[39060], | 99.00th=[47973], 99.50th=[52691], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 384, max= 768, per=25.92%, avg=542.58, stdev=116.81, samples=24 iops : min= 24, max= 48, avg=33.88, stdev= 7.26, samples=24 lat (msec) : 2=0.52%, 4=33.85%, 10=7.94%, 20=17.71%, 50=39.45% lat (msec) : 100=0.52% cpu : usr=0.12%, sys=5.12%, ctx=882, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20575: Thu Jun 20 00:55:10 2024 read: IOPS=33, BW=535KiB/s (548kB/s)(6256KiB/11683msec) clat (usec): min=1907, max=33702, avg=4600.69, stdev=4301.55 lat (usec): min=1908, max=33703, avg=4601.88, stdev=4301.81 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2278], 10.00th=[ 2376], 20.00th=[ 2606], | 30.00th=[ 2704], 40.00th=[ 2868], 50.00th=[ 2999], 60.00th=[ 3228], | 70.00th=[ 3720], 80.00th=[ 5473], 90.00th=[ 8848], 95.00th=[12518], | 99.00th=[28967], 99.50th=[33424], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 288, max= 1120, per=27.20%, avg=541.04, stdev=224.32, samples=23 iops : min= 18, max= 70, avg=33.74, stdev=13.96, samples=23 write: IOPS=32, BW=516KiB/s (529kB/s)(6032KiB/11683msec) clat (usec): min=13379, max=59508, avg=26192.60, stdev=7839.16 lat (usec): min=13380, max=59510, avg=26194.46, stdev=7839.14 clat percentiles (usec): | 1.00th=[14746], 5.00th=[16188], 10.00th=[17171], 20.00th=[19006], | 30.00th=[20841], 40.00th=[23200], 50.00th=[25035], 60.00th=[27132], | 70.00th=[29492], 80.00th=[31589], 90.00th=[38011], 95.00th=[41157], | 99.00th=[50070], 99.50th=[53216], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 288, max= 640, per=24.52%, avg=513.26, stdev=116.36, samples=23 iops : min= 18, max= 40, avg=32.00, stdev= 7.29, samples=23 lat (msec) : 2=0.39%, 4=37.24%, 10=9.51%, 20=15.23%, 50=37.11% lat (msec) : 100=0.52% cpu : usr=0.09%, sys=5.38%, ctx=877, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20576: Thu Jun 20 00:55:10 2024 read: IOPS=31, BW=507KiB/s (520kB/s)(6032KiB/11889msec) clat (usec): min=1888, max=27813, avg=4591.36, stdev=3891.27 lat (usec): min=1889, max=27815, avg=4592.62, stdev=3891.18 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2278], 10.00th=[ 2376], 20.00th=[ 2573], | 30.00th=[ 2769], 40.00th=[ 2868], 50.00th=[ 3032], 60.00th=[ 3326], | 70.00th=[ 3851], 80.00th=[ 5080], 90.00th=[ 9372], 95.00th=[13566], | 99.00th=[23200], 99.50th=[25035], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 160, max= 864, per=25.11%, avg=499.39, stdev=174.71, samples=23 iops : min= 10, max= 54, avg=31.13, stdev=10.96, samples=23 write: IOPS=32, BW=526KiB/s (539kB/s)(6256KiB/11889msec) clat (usec): min=13851, max=58441, avg=25954.11, stdev=8049.72 lat (usec): min=13852, max=58443, avg=25956.00, stdev=8049.85 clat percentiles (usec): | 1.00th=[15139], 5.00th=[16057], 10.00th=[16909], 20.00th=[18744], | 30.00th=[20841], 40.00th=[22938], 50.00th=[24773], 60.00th=[26608], | 70.00th=[28443], 80.00th=[31851], 90.00th=[38011], 95.00th=[41681], | 99.00th=[50070], 99.50th=[55837], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 352, max= 704, per=24.92%, avg=521.61, stdev=114.64, samples=23 iops : min= 22, max= 44, avg=32.52, stdev= 7.13, samples=23 lat (msec) : 2=0.13%, 4=35.55%, 10=8.85%, 20=17.45%, 50=37.50% lat (msec) : 100=0.52% cpu : usr=0.11%, sys=5.20%, ctx=896, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1989KiB/s (2037kB/s), 477KiB/s-535KiB/s (489kB/s-548kB/s), io=23.4MiB (24.5MB), run=11683-12040msec WRITE: bw=2093KiB/s (2143kB/s), 516KiB/s-544KiB/s (529kB/s-557kB/s), io=24.6MiB (25.8MB), run=11683-12040msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20596: Thu Jun 20 00:55:13 2024 read: IOPS=3, BW=3127KiB/s (3202kB/s)(7168KiB/2292msec) clat (usec): min=438, max=473882, avg=237414.39, stdev=214432.24 lat (usec): min=439, max=473883, avg=237415.83, stdev=214432.27 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 437], 10.00th=[ 437], 20.00th=[ 36963], | 30.00th=[ 38011], 40.00th=[ 38011], 50.00th=[227541], 60.00th=[442500], | 70.00th=[442500], 80.00th=[442500], 90.00th=[471860], 95.00th=[471860], | 99.00th=[471860], 99.50th=[471860], 99.90th=[471860], 99.95th=[471860], | 99.99th=[471860] bw ( KiB/s): min= 1744, max= 4087, per=31.80%, avg=2841.75, stdev=1125.96, samples=4 iops : min= 1, max= 3, avg= 2.25, stdev= 0.96, samples=4 write: IOPS=2, BW=2234KiB/s (2287kB/s)(5120KiB/2292msec) clat (msec): min=22, max=458, avg=123.64, stdev=187.58 lat (msec): min=23, max=458, avg=124.69, stdev=187.06 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 23], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 44], 60.00th=[ 44], | 70.00th=[ 64], 80.00th=[ 64], 90.00th=[ 460], 95.00th=[ 460], | 99.00th=[ 460], 99.50th=[ 460], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 1744, max= 5233, per=24.04%, avg=3006.67, stdev=1933.85, samples=3 iops : min= 1, max= 5, avg= 2.33, stdev= 2.31, samples=3 lat (usec) : 500=8.33% lat (msec) : 50=41.67%, 100=8.33%, 250=8.33%, 500=33.33% cpu : usr=0.00%, sys=21.43%, ctx=961, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20597: Thu Jun 20 00:55:13 2024 read: IOPS=1, BW=1697KiB/s (1737kB/s)(2048KiB/1207msec) clat (usec): min=409, max=43700, avg=22054.90, stdev=30611.26 lat (usec): min=411, max=43702, avg=22056.77, stdev=30611.61 clat percentiles (usec): | 1.00th=[ 408], 5.00th=[ 408], 10.00th=[ 408], 20.00th=[ 408], | 30.00th=[ 408], 40.00th=[ 408], 50.00th=[ 408], 60.00th=[43779], | 70.00th=[43779], 80.00th=[43779], 90.00th=[43779], 95.00th=[43779], | 99.00th=[43779], 99.50th=[43779], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 2048, max= 2048, per=22.92%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=8, BW=8484KiB/s (8687kB/s)(10.0MiB/1207msec) clat (msec): min=24, max=534, avg=116.10, stdev=175.43 lat (msec): min=24, max=534, avg=116.16, stdev=175.41 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 28], 40.00th=[ 29], 50.00th=[ 30], 60.00th=[ 36], | 70.00th=[ 64], 80.00th=[ 64], 90.00th=[ 338], 95.00th=[ 535], | 99.00th=[ 535], 99.50th=[ 535], 99.90th=[ 535], 99.95th=[ 535], | 99.99th=[ 535] bw ( KiB/s): min=10240, max=10240, per=81.86%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=58.33%, 100=16.67%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=21.14%, ctx=79, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20598: Thu Jun 20 00:55:13 2024 read: IOPS=2, BW=2722KiB/s (2788kB/s)(6144KiB/2257msec) clat (usec): min=451, max=449420, avg=172868.86, stdev=206547.82 lat (usec): min=453, max=449420, avg=172870.09, stdev=206547.96 clat percentiles (usec): | 1.00th=[ 453], 5.00th=[ 453], 10.00th=[ 453], 20.00th=[ 502], | 30.00th=[ 502], 40.00th=[ 56361], 50.00th=[ 56361], 60.00th=[112722], | 70.00th=[417334], 80.00th=[417334], 90.00th=[450888], 95.00th=[450888], | 99.00th=[450888], 99.50th=[450888], 99.90th=[450888], 99.95th=[450888], | 99.99th=[450888] bw ( KiB/s): min= 2048, max= 8192, per=57.30%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=2, BW=2722KiB/s (2788kB/s)(6144KiB/2257msec) clat (msec): min=27, max=938, avg=202.01, stdev=361.46 lat (msec): min=28, max=938, avg=202.06, stdev=361.47 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 66], 50.00th=[ 66], 60.00th=[ 71], | 70.00th=[ 79], 80.00th=[ 79], 90.00th=[ 936], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 2048, max= 6144, per=32.74%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=16.67%, 100=33.33%, 250=8.33%, 500=16.67%, 1000=8.33% cpu : usr=0.04%, sys=14.49%, ctx=641, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20600: Thu Jun 20 00:55:13 2024 read: IOPS=2, BW=2267KiB/s (2322kB/s)(5120KiB/2258msec) clat (usec): min=535, max=480898, avg=210670.20, stdev=230936.43 lat (usec): min=537, max=480899, avg=210671.27, stdev=230936.30 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 537], 10.00th=[ 537], 20.00th=[ 537], | 30.00th=[ 2057], 40.00th=[ 2057], 50.00th=[141558], 60.00th=[141558], | 70.00th=[429917], 80.00th=[429917], 90.00th=[480248], 95.00th=[480248], | 99.00th=[480248], 99.50th=[480248], 99.90th=[480248], 99.95th=[480248], | 99.99th=[480248] bw ( KiB/s): min= 2048, max= 6144, per=45.84%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=3, BW=3174KiB/s (3251kB/s)(7168KiB/2258msec) clat (msec): min=26, max=931, avg=170.76, stdev=335.92 lat (msec): min=26, max=931, avg=170.83, stdev=335.92 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 44], 60.00th=[ 54], | 70.00th=[ 54], 80.00th=[ 78], 90.00th=[ 936], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 2048, max= 6144, per=38.20%, avg=4778.67, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 4.67, stdev= 2.31, samples=3 lat (usec) : 750=8.33% lat (msec) : 4=8.33%, 50=33.33%, 100=16.67%, 250=8.33%, 500=16.67% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=14.00%, ctx=623, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8935KiB/s (9150kB/s), 1697KiB/s-3127KiB/s (1737kB/s-3202kB/s), io=20.0MiB (20.0MB), run=1207-2292msec WRITE: bw=12.2MiB/s (12.8MB/s), 2234KiB/s-8484KiB/s (2287kB/s-8687kB/s), io=28.0MiB (29.4MB), run=1207-2292msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20599: Thu Jun 20 00:55:14 2024 read: IOPS=3, BW=3099KiB/s (3173kB/s)(7168KiB/2313msec) clat (msec): min=17, max=464, avg=224.15, stdev=192.34 lat (msec): min=17, max=464, avg=224.16, stdev=192.34 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 56], | 30.00th=[ 87], 40.00th=[ 87], 50.00th=[ 140], 60.00th=[ 368], | 70.00th=[ 368], 80.00th=[ 443], 90.00th=[ 464], 95.00th=[ 464], | 99.00th=[ 464], 99.50th=[ 464], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 2048, max= 4096, per=35.75%, avg=3072.00, stdev=1182.41, samples=4 iops : min= 2, max= 4, avg= 3.00, stdev= 1.15, samples=4 write: IOPS=2, BW=2214KiB/s (2267kB/s)(5120KiB/2313msec) clat (msec): min=35, max=515, avg=148.35, stdev=205.78 lat (msec): min=35, max=515, avg=148.42, stdev=205.77 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 35], | 30.00th=[ 61], 40.00th=[ 61], 50.00th=[ 65], 60.00th=[ 65], | 70.00th=[ 66], 80.00th=[ 66], 90.00th=[ 514], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 4096, max= 6144, per=42.56%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 20=8.33%, 50=8.33%, 100=41.67%, 250=8.33%, 500=25.00% lat (msec) : 750=8.33% cpu : usr=0.04%, sys=4.32%, ctx=50, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20601: Thu Jun 20 00:55:14 2024 read: IOPS=1, BW=1111KiB/s (1137kB/s)(2048KiB/1844msec) clat (msec): min=48, max=428, avg=238.70, stdev=268.88 lat (msec): min=48, max=428, avg=238.70, stdev=268.88 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 49], 10.00th=[ 49], 20.00th=[ 49], | 30.00th=[ 49], 40.00th=[ 49], 50.00th=[ 49], 60.00th=[ 430], | 70.00th=[ 430], 80.00th=[ 430], 90.00th=[ 430], 95.00th=[ 430], | 99.00th=[ 430], 99.50th=[ 430], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 2048, max= 2048, per=23.83%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5553KiB/s (5686kB/s)(10.0MiB/1844msec) clat (msec): min=30, max=564, avg=136.43, stdev=194.59 lat (msec): min=30, max=564, avg=136.50, stdev=194.59 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 35], 50.00th=[ 35], 60.00th=[ 35], | 70.00th=[ 89], 80.00th=[ 89], 90.00th=[ 435], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 4096, max=10240, per=59.58%, avg=7168.00, stdev=4344.46, samples=2 iops : min= 4, max= 10, avg= 7.00, stdev= 4.24, samples=2 lat (msec) : 50=58.33%, 100=16.67%, 500=16.67%, 750=8.33% cpu : usr=0.11%, sys=6.35%, ctx=41, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20602: Thu Jun 20 00:55:14 2024 read: IOPS=2, BW=2717KiB/s (2783kB/s)(6144KiB/2261msec) clat (msec): min=16, max=392, avg=162.55, stdev=155.50 lat (msec): min=16, max=392, avg=162.55, stdev=155.49 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 20], | 30.00th=[ 20], 40.00th=[ 82], 50.00th=[ 82], 60.00th=[ 165], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 4096, max= 6144, per=59.58%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2717KiB/s (2783kB/s)(6144KiB/2261msec) clat (msec): min=26, max=975, avg=213.93, stdev=376.15 lat (msec): min=26, max=975, avg=213.98, stdev=376.15 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 53], | 70.00th=[ 159], 80.00th=[ 159], 90.00th=[ 978], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 4096, max= 4096, per=34.05%, avg=4096.00, stdev= 0.00, samples=3 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=3 lat (msec) : 20=16.67%, 50=25.00%, 100=16.67%, 250=16.67%, 500=16.67% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=4.20%, ctx=74, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20603: Thu Jun 20 00:55:14 2024 read: IOPS=2, BW=2149KiB/s (2200kB/s)(5120KiB/2383msec) clat (msec): min=15, max=478, avg=215.97, stdev=239.68 lat (msec): min=15, max=478, avg=215.97, stdev=239.68 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16], | 30.00th=[ 53], 40.00th=[ 53], 50.00th=[ 57], 60.00th=[ 57], | 70.00th=[ 477], 80.00th=[ 477], 90.00th=[ 477], 95.00th=[ 477], | 99.00th=[ 477], 99.50th=[ 477], 99.90th=[ 477], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 2048, max= 4096, per=35.75%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 write: IOPS=2, BW=3008KiB/s (3080kB/s)(7168KiB/2383msec) clat (msec): min=25, max=979, avg=185.87, stdev=351.77 lat (msec): min=25, max=979, avg=185.93, stdev=351.77 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 27], | 30.00th=[ 29], 40.00th=[ 29], 50.00th=[ 51], 60.00th=[ 59], | 70.00th=[ 59], 80.00th=[ 133], 90.00th=[ 978], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 2048, max= 6144, per=34.05%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=8.33%, 50=33.33%, 100=25.00%, 250=8.33%, 500=16.67% lat (msec) : 1000=8.33% cpu : usr=0.04%, sys=4.16%, ctx=42, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8594KiB/s (8800kB/s), 1111KiB/s-3099KiB/s (1137kB/s-3173kB/s), io=20.0MiB (20.0MB), run=1844-2383msec WRITE: bw=11.7MiB/s (12.3MB/s), 2214KiB/s-5553KiB/s (2267kB/s-5686kB/s), io=28.0MiB (29.4MB), run=1844-2383msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20610: Thu Jun 20 00:55:17 2024 read: IOPS=1, BW=4539KiB/s (4648kB/s)(12.0MiB/2707msec) clat (msec): min=111, max=1760, avg=901.83, stdev=826.70 lat (msec): min=111, max=1760, avg=901.83, stdev=826.70 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 835], 50.00th=[ 835], 60.00th=[ 835], | 70.00th=[ 1754], 80.00th=[ 1754], 90.00th=[ 1754], 95.00th=[ 1754], | 99.00th=[ 1754], 99.50th=[ 1754], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 8126, max= 8192, per=100.00%, avg=8159.00, stdev=46.67, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=58.46%, ctx=807, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20611: Thu Jun 20 00:55:17 2024 read: IOPS=0, BW=1422KiB/s (1456kB/s)(4096KiB/2881msec) clat (nsec): min=136509k, max=136509k, avg=136509399.00, stdev= 0.00 lat (nsec): min=136512k, max=136512k, avg=136511676.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 138], 10.00th=[ 138], 20.00th=[ 138], | 30.00th=[ 138], 40.00th=[ 138], 50.00th=[ 138], 60.00th=[ 138], | 70.00th=[ 138], 80.00th=[ 138], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2843KiB/s (2912kB/s)(8192KiB/2881msec) clat (msec): min=432, max=2310, avg=1371.62, stdev=1327.61 lat (msec): min=433, max=2310, avg=1371.88, stdev=1327.62 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 435], 10.00th=[ 435], 20.00th=[ 435], | 30.00th=[ 435], 40.00th=[ 435], 50.00th=[ 435], 60.00th=[ 2299], | 70.00th=[ 2299], 80.00th=[ 2299], 90.00th=[ 2299], 95.00th=[ 2299], | 99.00th=[ 2299], 99.50th=[ 2299], 99.90th=[ 2299], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 8192, max= 8192, per=72.03%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=8.54%, ctx=102, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20612: Thu Jun 20 00:55:17 2024 write: IOPS=1, BW=5375KiB/s (5504kB/s)(12.0MiB/2286msec) clat (msec): min=282, max=1619, avg=759.90, stdev=746.11 lat (msec): min=282, max=1620, avg=760.32, stdev=746.04 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 284], 10.00th=[ 284], 20.00th=[ 284], | 30.00th=[ 284], 40.00th=[ 380], 50.00th=[ 380], 60.00th=[ 380], | 70.00th=[ 1620], 80.00th=[ 1620], 90.00th=[ 1620], 95.00th=[ 1620], | 99.00th=[ 1620], 99.50th=[ 1620], 99.90th=[ 1620], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 8192, max= 8192, per=72.03%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=10.77%, ctx=62, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20613: Thu Jun 20 00:55:17 2024 write: IOPS=1, BW=6021KiB/s (6165kB/s)(12.0MiB/2041msec) clat (msec): min=241, max=1156, avg=676.19, stdev=459.14 lat (msec): min=241, max=1156, avg=676.51, stdev=459.05 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 241], 10.00th=[ 241], 20.00th=[ 241], | 30.00th=[ 241], 40.00th=[ 634], 50.00th=[ 634], 60.00th=[ 634], | 70.00th=[ 1150], 80.00th=[ 1150], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 8175, max= 8192, per=71.96%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=15.39%, ctx=55, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5687KiB/s (5823kB/s), 1422KiB/s-4539KiB/s (1456kB/s-4648kB/s), io=16.0MiB (16.8MB), run=2707-2881msec WRITE: bw=11.1MiB/s (11.6MB/s), 2843KiB/s-6021KiB/s (2912kB/s-6165kB/s), io=32.0MiB (33.6MB), run=2041-2881msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20614: Thu Jun 20 00:55:18 2024 read: IOPS=1, BW=4357KiB/s (4462kB/s)(12.0MiB/2820msec) clat (msec): min=96, max=2244, avg=939.19, stdev=1146.39 lat (msec): min=96, max=2244, avg=939.19, stdev=1146.39 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 97], 10.00th=[ 97], 20.00th=[ 97], | 30.00th=[ 97], 40.00th=[ 477], 50.00th=[ 477], 60.00th=[ 477], | 70.00th=[ 2232], 80.00th=[ 2232], 90.00th=[ 2232], 95.00th=[ 2232], | 99.00th=[ 2232], 99.50th=[ 2232], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 8110, max= 8110, per=100.00%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=2.70%, ctx=31, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20615: Thu Jun 20 00:55:18 2024 read: IOPS=0, BW=1380KiB/s (1413kB/s)(4096KiB/2969msec) clat (nsec): min=503118k, max=503118k, avg=503118082.00, stdev= 0.00 lat (nsec): min=503121k, max=503121k, avg=503121224.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 502], 10.00th=[ 502], 20.00th=[ 502], | 30.00th=[ 502], 40.00th=[ 502], 50.00th=[ 502], 60.00th=[ 502], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2759KiB/s (2825kB/s)(8192KiB/2969msec) clat (msec): min=118, max=2317, avg=1218.35, stdev=1555.04 lat (msec): min=119, max=2318, avg=1218.63, stdev=1555.00 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 120], 50.00th=[ 120], 60.00th=[ 2333], | 70.00th=[ 2333], 80.00th=[ 2333], 90.00th=[ 2333], 95.00th=[ 2333], | 99.00th=[ 2333], 99.50th=[ 2333], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 8192, max= 8192, per=74.28%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=3.17%, ctx=41, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20616: Thu Jun 20 00:55:18 2024 write: IOPS=1, BW=4441KiB/s (4547kB/s)(12.0MiB/2767msec) clat (msec): min=425, max=1846, avg=915.03, stdev=807.03 lat (msec): min=426, max=1846, avg=915.29, stdev=807.04 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 426], 10.00th=[ 426], 20.00th=[ 426], | 30.00th=[ 426], 40.00th=[ 472], 50.00th=[ 472], 60.00th=[ 472], | 70.00th=[ 1854], 80.00th=[ 1854], 90.00th=[ 1854], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 8192, max= 8192, per=74.28%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67% cpu : usr=0.04%, sys=4.81%, ctx=94, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20617: Thu Jun 20 00:55:18 2024 write: IOPS=1, BW=4136KiB/s (4235kB/s)(12.0MiB/2971msec) clat (msec): min=71, max=2322, avg=981.26, stdev=1185.81 lat (msec): min=72, max=2322, avg=981.49, stdev=1185.84 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 72], 10.00th=[ 72], 20.00th=[ 72], | 30.00th=[ 72], 40.00th=[ 550], 50.00th=[ 550], 60.00th=[ 550], | 70.00th=[ 2333], 80.00th=[ 2333], 90.00th=[ 2333], 95.00th=[ 2333], | 99.00th=[ 2333], 99.50th=[ 2333], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 8192, max= 8192, per=74.28%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.00%, sys=4.48%, ctx=111, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5518KiB/s (5651kB/s), 1380KiB/s-4357KiB/s (1413kB/s-4462kB/s), io=16.0MiB (16.8MB), run=2820-2969msec WRITE: bw=10.8MiB/s (11.3MB/s), 2759KiB/s-4441KiB/s (2825kB/s-4547kB/s), io=32.0MiB (33.6MB), run=2767-2971msec