== sanity test 398b: DIO and buffer IO race ============== 07:15:53 (1734437753) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.18018 s, 42.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4561: Tue Dec 17 07:16:09 2024 read: IOPS=108, BW=432KiB/s (443kB/s)(5992KiB/13866msec) clat (usec): min=48, max=190787, avg=4273.46, stdev=5826.22 lat (usec): min=49, max=190788, avg=4274.42, stdev=5826.17 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 63], 10.00th=[ 84], 20.00th=[ 2057], | 30.00th=[ 2606], 40.00th=[ 3032], 50.00th=[ 3458], 60.00th=[ 4146], | 70.00th=[ 5145], 80.00th=[ 6521], 90.00th=[ 8356], 95.00th=[ 10028], | 99.00th=[ 14091], 99.50th=[ 16188], 99.90th=[ 28181], 99.95th=[189793], | 99.99th=[189793] bw ( KiB/s): min= 272, max= 568, per=24.94%, avg=426.52, stdev=70.37, samples=27 iops : min= 68, max= 142, avg=106.52, stdev=17.65, samples=27 write: IOPS=113, BW=454KiB/s (465kB/s)(6296KiB/13866msec) clat (usec): min=458, max=35430, avg=4688.43, stdev=4559.78 lat (usec): min=459, max=35431, avg=4689.60, stdev=4559.63 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 562], 10.00th=[ 603], 20.00th=[ 775], | 30.00th=[ 1106], 40.00th=[ 2311], 50.00th=[ 3621], 60.00th=[ 4686], | 70.00th=[ 5735], 80.00th=[ 7701], 90.00th=[10683], 95.00th=[13435], | 99.00th=[20579], 99.50th=[23987], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 336, max= 544, per=25.00%, avg=447.81, stdev=64.66, samples=27 iops : min= 84, max= 136, avg=111.85, stdev=16.16, samples=27 lat (usec) : 50=0.13%, 100=6.18%, 250=2.34%, 500=0.26%, 750=9.64% lat (usec) : 1000=4.62% lat (msec) : 2=5.86%, 4=26.69%, 10=35.64%, 20=7.85%, 50=0.75% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=24.13%, ctx=6297, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4562: Tue Dec 17 07:16:09 2024 read: IOPS=105, BW=423KiB/s (433kB/s)(5940KiB/14035msec) clat (usec): min=49, max=51524, avg=4209.74, stdev=3488.86 lat (usec): min=49, max=51525, avg=4210.99, stdev=3488.92 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 65], 10.00th=[ 79], 20.00th=[ 2073], | 30.00th=[ 2638], 40.00th=[ 3097], 50.00th=[ 3556], 60.00th=[ 4178], | 70.00th=[ 5211], 80.00th=[ 6587], 90.00th=[ 8455], 95.00th=[ 9634], | 99.00th=[14746], 99.50th=[16450], 99.90th=[29754], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 295, max= 632, per=24.65%, avg=421.57, stdev=93.33, samples=28 iops : min= 73, max= 158, avg=105.29, stdev=23.44, samples=28 write: IOPS=113, BW=452KiB/s (463kB/s)(6348KiB/14035msec) clat (usec): min=442, max=186030, avg=4850.77, stdev=6394.82 lat (usec): min=443, max=186030, avg=4852.27, stdev=6395.34 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 537], 10.00th=[ 586], 20.00th=[ 734], | 30.00th=[ 1106], 40.00th=[ 2671], 50.00th=[ 3851], 60.00th=[ 4883], | 70.00th=[ 6128], 80.00th=[ 7832], 90.00th=[ 10552], 95.00th=[ 13566], | 99.00th=[ 19530], 99.50th=[ 22938], 99.90th=[ 44827], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 352, max= 664, per=25.10%, avg=449.54, stdev=82.99, samples=28 iops : min= 88, max= 166, avg=112.29, stdev=20.76, samples=28 lat (usec) : 50=0.20%, 100=6.67%, 250=1.89%, 500=0.42%, 750=10.38% lat (usec) : 1000=3.97% lat (msec) : 2=5.21%, 4=25.68%, 10=37.83%, 20=7.23%, 50=0.46% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.23%, sys=23.85%, ctx=6474, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4563: Tue Dec 17 07:16:09 2024 read: IOPS=107, BW=430KiB/s (440kB/s)(6028KiB/14029msec) clat (usec): min=45, max=50038, avg=4161.91, stdev=3490.37 lat (usec): min=45, max=50039, avg=4162.95, stdev=3490.35 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 62], 10.00th=[ 75], 20.00th=[ 1762], | 30.00th=[ 2573], 40.00th=[ 2999], 50.00th=[ 3556], 60.00th=[ 4178], | 70.00th=[ 5145], 80.00th=[ 6718], 90.00th=[ 8586], 95.00th=[10028], | 99.00th=[13435], 99.50th=[16057], 99.90th=[30278], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 224, max= 616, per=25.04%, avg=428.11, stdev=88.39, samples=28 iops : min= 56, max= 154, avg=106.93, stdev=22.13, samples=28 write: IOPS=111, BW=446KiB/s (457kB/s)(6260KiB/14029msec) clat (usec): min=471, max=206323, avg=4902.19, stdev=6828.44 lat (usec): min=471, max=206325, avg=4903.16, stdev=6828.50 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 578], 20.00th=[ 783], | 30.00th=[ 1221], 40.00th=[ 2671], 50.00th=[ 3785], 60.00th=[ 4817], | 70.00th=[ 6128], 80.00th=[ 7701], 90.00th=[ 10683], 95.00th=[ 13566], | 99.00th=[ 22938], 99.50th=[ 25297], 99.90th=[ 39060], 99.95th=[206570], | 99.99th=[206570] bw ( KiB/s): min= 288, max= 624, per=24.80%, avg=444.14, stdev=85.23, samples=28 iops : min= 72, max= 156, avg=110.93, stdev=21.39, samples=28 lat (usec) : 50=0.26%, 100=7.13%, 250=2.15%, 500=0.72%, 750=8.82% lat (usec) : 1000=4.62% lat (msec) : 2=4.88%, 4=26.37%, 10=36.56%, 20=7.71%, 50=0.72% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.19%, sys=23.94%, ctx=6360, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4564: Tue Dec 17 07:16:09 2024 read: IOPS=107, BW=432KiB/s (442kB/s)(6052KiB/14012msec) clat (usec): min=47, max=39961, avg=4204.77, stdev=3564.86 lat (usec): min=47, max=39961, avg=4206.41, stdev=3564.34 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 67], 10.00th=[ 87], 20.00th=[ 2057], | 30.00th=[ 2540], 40.00th=[ 2966], 50.00th=[ 3425], 60.00th=[ 4146], | 70.00th=[ 5145], 80.00th=[ 6652], 90.00th=[ 8356], 95.00th=[10028], | 99.00th=[17171], 99.50th=[22676], 99.90th=[27132], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 224, max= 600, per=25.22%, avg=431.32, stdev=79.07, samples=28 iops : min= 56, max= 150, avg=107.79, stdev=19.78, samples=28 write: IOPS=111, BW=445KiB/s (456kB/s)(6236KiB/14012msec) clat (usec): min=443, max=218804, avg=4852.09, stdev=7043.03 lat (usec): min=444, max=218807, avg=4853.01, stdev=7043.00 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 545], 10.00th=[ 586], 20.00th=[ 791], | 30.00th=[ 1074], 40.00th=[ 2540], 50.00th=[ 3654], 60.00th=[ 4752], | 70.00th=[ 6194], 80.00th=[ 7832], 90.00th=[ 10683], 95.00th=[ 13304], | 99.00th=[ 19530], 99.50th=[ 22676], 99.90th=[ 49546], 99.95th=[219153], | 99.99th=[219153] bw ( KiB/s): min= 272, max= 608, per=24.79%, avg=443.93, stdev=78.93, samples=28 iops : min= 68, max= 152, avg=110.93, stdev=19.73, samples=28 lat (usec) : 50=0.16%, 100=6.18%, 250=2.44%, 500=0.49%, 750=9.41% lat (usec) : 1000=5.01% lat (msec) : 2=5.05%, 4=26.92%, 10=35.97%, 20=7.55%, 50=0.78% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=23.57%, ctx=6334, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1711KiB/s (1752kB/s), 423KiB/s-432KiB/s (433kB/s-443kB/s), io=23.4MiB (24.6MB), run=13866-14035msec WRITE: bw=1791KiB/s (1834kB/s), 445KiB/s-454KiB/s (456kB/s-465kB/s), io=24.6MiB (25.7MB), run=13866-14035msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4556: Tue Dec 17 07:16:30 2024 read: IOPS=43, BW=175KiB/s (179kB/s)(5992KiB/34192msec) clat (usec): min=1598, max=190991, avg=4197.67, stdev=5893.60 lat (usec): min=1599, max=190994, avg=4198.49, stdev=5893.66 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1876], 10.00th=[ 1942], 20.00th=[ 2073], | 30.00th=[ 2245], 40.00th=[ 2442], 50.00th=[ 2999], 60.00th=[ 3654], | 70.00th=[ 4047], 80.00th=[ 4817], 90.00th=[ 8094], 95.00th=[ 10552], | 99.00th=[ 15926], 99.50th=[ 23725], 99.90th=[ 44827], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 48, max= 296, per=25.15%, avg=174.79, stdev=70.59, samples=68 iops : min= 12, max= 74, avg=43.68, stdev=17.67, samples=68 write: IOPS=46, BW=184KiB/s (189kB/s)(6296KiB/34192msec) clat (usec): min=8685, max=59360, avg=17657.21, stdev=7314.38 lat (usec): min=8686, max=59361, avg=17658.18, stdev=7314.42 clat percentiles (usec): | 1.00th=[ 9372], 5.00th=[10290], 10.00th=[10945], 20.00th=[11994], | 30.00th=[13042], 40.00th=[14353], 50.00th=[15795], 60.00th=[17171], | 70.00th=[18744], 80.00th=[21365], 90.00th=[28443], 95.00th=[33817], | 99.00th=[41681], 99.50th=[45351], 99.90th=[57934], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 64, max= 264, per=25.24%, avg=183.74, stdev=59.94, samples=68 iops : min= 16, max= 66, avg=45.91, stdev=15.02, samples=68 lat (msec) : 2=7.03%, 4=26.76%, 10=13.80%, 20=39.88%, 50=12.37% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.08%, sys=5.80%, ctx=3794, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4557: Tue Dec 17 07:16:30 2024 read: IOPS=43, BW=172KiB/s (176kB/s)(5940KiB/34510msec) clat (usec): min=1478, max=190794, avg=4215.88, stdev=6143.72 lat (usec): min=1479, max=190795, avg=4216.60, stdev=6143.74 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1876], 10.00th=[ 1958], 20.00th=[ 2089], | 30.00th=[ 2245], 40.00th=[ 2409], 50.00th=[ 3032], 60.00th=[ 3654], | 70.00th=[ 4015], 80.00th=[ 4817], 90.00th=[ 7701], 95.00th=[ 10683], | 99.00th=[ 17433], 99.50th=[ 23200], 99.90th=[ 64750], 99.95th=[189793], | 99.99th=[189793] bw ( KiB/s): min= 16, max= 352, per=24.76%, avg=172.10, stdev=75.25, samples=68 iops : min= 4, max= 88, avg=43.01, stdev=18.82, samples=68 write: IOPS=45, BW=184KiB/s (188kB/s)(6348KiB/34510msec) clat (usec): min=7735, max=57008, avg=17730.75, stdev=7367.60 lat (usec): min=7735, max=57010, avg=17731.77, stdev=7367.62 clat percentiles (usec): | 1.00th=[ 8979], 5.00th=[10159], 10.00th=[10945], 20.00th=[11994], | 30.00th=[13042], 40.00th=[14222], 50.00th=[15795], 60.00th=[17171], | 70.00th=[19006], 80.00th=[21890], 90.00th=[28443], 95.00th=[33817], | 99.00th=[42730], 99.50th=[45351], 99.90th=[53216], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 80, max= 264, per=25.03%, avg=182.22, stdev=59.39, samples=68 iops : min= 20, max= 66, avg=45.54, stdev=14.87, samples=68 lat (msec) : 2=6.45%, 4=26.95%, 10=14.26%, 20=38.54%, 50=13.64% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.11%, sys=5.73%, ctx=3829, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4558: Tue Dec 17 07:16:30 2024 read: IOPS=43, BW=175KiB/s (179kB/s)(6028KiB/34401msec) clat (usec): min=1474, max=50559, avg=4251.78, stdev=3796.13 lat (usec): min=1474, max=50560, avg=4252.55, stdev=3796.13 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1860], 10.00th=[ 1942], 20.00th=[ 2114], | 30.00th=[ 2245], 40.00th=[ 2474], 50.00th=[ 3228], 60.00th=[ 3720], | 70.00th=[ 4113], 80.00th=[ 5080], 90.00th=[ 8291], 95.00th=[11076], | 99.00th=[18220], 99.50th=[25822], 99.90th=[44827], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 55, max= 336, per=25.17%, avg=174.93, stdev=64.13, samples=68 iops : min= 13, max= 84, avg=43.72, stdev=16.05, samples=68 write: IOPS=45, BW=182KiB/s (186kB/s)(6260KiB/34401msec) clat (msec): min=7, max=204, avg=17.82, stdev= 8.51 lat (msec): min=7, max=204, avg=17.82, stdev= 8.51 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 22], 90.00th=[ 28], 95.00th=[ 33], | 99.00th=[ 42], 99.50th=[ 46], 99.90th=[ 77], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 72, max= 264, per=24.85%, avg=180.93, stdev=59.56, samples=68 iops : min= 18, max= 66, avg=45.22, stdev=14.90, samples=68 lat (msec) : 2=6.74%, 4=26.50%, 10=14.32%, 20=39.55%, 50=12.70% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.08%, sys=5.69%, ctx=3729, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4559: Tue Dec 17 07:16:30 2024 read: IOPS=44, BW=177KiB/s (181kB/s)(6052KiB/34236msec) clat (usec): min=1536, max=64561, avg=4072.74, stdev=3439.60 lat (usec): min=1537, max=64562, avg=4073.51, stdev=3439.64 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1860], 10.00th=[ 1958], 20.00th=[ 2089], | 30.00th=[ 2245], 40.00th=[ 2442], 50.00th=[ 2966], 60.00th=[ 3654], | 70.00th=[ 4080], 80.00th=[ 5014], 90.00th=[ 7635], 95.00th=[10421], | 99.00th=[16319], 99.50th=[19268], 99.90th=[32637], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 23, max= 352, per=25.28%, avg=175.72, stdev=70.11, samples=68 iops : min= 5, max= 88, avg=43.81, stdev=17.57, samples=68 write: IOPS=45, BW=182KiB/s (187kB/s)(6236KiB/34236msec) clat (msec): min=8, max=217, avg=17.94, stdev= 8.73 lat (msec): min=8, max=217, avg=17.94, stdev= 8.73 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 20], 80.00th=[ 22], 90.00th=[ 28], 95.00th=[ 33], | 99.00th=[ 42], 99.50th=[ 44], 99.90th=[ 61], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 64, max= 248, per=24.93%, avg=181.49, stdev=58.03, samples=68 iops : min= 16, max= 62, avg=45.25, stdev=14.56, samples=68 lat (msec) : 2=6.67%, 4=26.92%, 10=14.65%, 20=38.25%, 50=13.31% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.15%, sys=5.67%, ctx=3749, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=696KiB/s (712kB/s), 172KiB/s-177KiB/s (176kB/s-181kB/s), io=23.4MiB (24.6MB), run=34192-34510msec WRITE: bw=728KiB/s (746kB/s), 182KiB/s-184KiB/s (186kB/s-189kB/s), io=24.6MiB (25.7MB), run=34192-34510msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4608: Tue Dec 17 07:16:35 2024 read: IOPS=77, BW=1236KiB/s (1265kB/s)(5920KiB/4791msec) clat (usec): min=53, max=20361, avg=8833.45, stdev=5344.69 lat (usec): min=53, max=20361, avg=8835.67, stdev=5345.11 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 65], 10.00th=[ 75], 20.00th=[ 104], | 30.00th=[ 8094], 40.00th=[ 9110], 50.00th=[10028], 60.00th=[11076], | 70.00th=[11863], 80.00th=[13304], 90.00th=[14353], 95.00th=[15664], | 99.00th=[19006], 99.50th=[19792], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 608, max= 1504, per=23.61%, avg=1180.44, stdev=290.85, samples=9 iops : min= 38, max= 94, avg=73.78, stdev=18.18, samples=9 write: IOPS=83, BW=1329KiB/s (1361kB/s)(6368KiB/4791msec) clat (usec): min=615, max=26475, avg=3724.05, stdev=4365.03 lat (usec): min=616, max=26476, avg=3725.08, stdev=4365.05 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 848], | 30.00th=[ 930], 40.00th=[ 1237], 50.00th=[ 1827], 60.00th=[ 3097], | 70.00th=[ 4015], 80.00th=[ 5669], 90.00th=[ 8979], 95.00th=[12387], | 99.00th=[22938], 99.50th=[24249], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 640, max= 2144, per=24.41%, avg=1283.56, stdev=446.12, samples=9 iops : min= 40, max= 134, avg=80.22, stdev=27.88, samples=9 lat (usec) : 100=9.37%, 250=1.56%, 500=0.13%, 750=3.52%, 1000=13.28% lat (msec) : 2=9.51%, 4=9.77%, 10=24.35%, 20=27.60%, 50=0.91% cpu : usr=0.19%, sys=21.32%, ctx=1993, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4609: Tue Dec 17 07:16:35 2024 read: IOPS=76, BW=1217KiB/s (1246kB/s)(5744KiB/4720msec) clat (usec): min=52, max=31176, avg=8629.55, stdev=5853.73 lat (usec): min=53, max=31177, avg=8630.18, stdev=5853.74 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 64], 10.00th=[ 68], 20.00th=[ 96], | 30.00th=[ 7898], 40.00th=[ 9241], 50.00th=[10028], 60.00th=[10814], | 70.00th=[11731], 80.00th=[13042], 90.00th=[14877], 95.00th=[16188], | 99.00th=[21890], 99.50th=[24773], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 448, max= 1920, per=24.40%, avg=1219.56, stdev=429.76, samples=9 iops : min= 28, max= 120, avg=76.22, stdev=26.86, samples=9 write: IOPS=86, BW=1386KiB/s (1420kB/s)(6544KiB/4720msec) clat (usec): min=620, max=33903, avg=3859.23, stdev=5011.89 lat (usec): min=622, max=33904, avg=3860.30, stdev=5012.07 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 734], 10.00th=[ 791], 20.00th=[ 873], | 30.00th=[ 979], 40.00th=[ 1156], 50.00th=[ 1909], 60.00th=[ 2900], | 70.00th=[ 3884], 80.00th=[ 5669], 90.00th=[ 7898], 95.00th=[14353], | 99.00th=[26346], 99.50th=[27919], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 672, max= 2496, per=25.96%, avg=1365.33, stdev=542.35, samples=9 iops : min= 42, max= 156, avg=85.33, stdev=33.90, samples=9 lat (usec) : 100=10.03%, 250=2.34%, 750=3.26%, 1000=13.41% lat (msec) : 2=10.16%, 4=11.46%, 10=22.14%, 20=25.00%, 50=2.21% cpu : usr=0.13%, sys=21.49%, ctx=1854, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4610: Tue Dec 17 07:16:35 2024 read: IOPS=83, BW=1330KiB/s (1362kB/s)(6256KiB/4703msec) clat (usec): min=51, max=20419, avg=8919.95, stdev=5508.22 lat (usec): min=51, max=20420, avg=8921.90, stdev=5508.56 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 63], 10.00th=[ 67], 20.00th=[ 100], | 30.00th=[ 8455], 40.00th=[ 9503], 50.00th=[10683], 60.00th=[11338], | 70.00th=[11994], 80.00th=[13435], 90.00th=[14877], 95.00th=[16057], | 99.00th=[19268], 99.50th=[19530], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 960, max= 1760, per=25.53%, avg=1276.11, stdev=248.76, samples=9 iops : min= 60, max= 110, avg=79.67, stdev=15.59, samples=9 write: IOPS=80, BW=1283KiB/s (1313kB/s)(6032KiB/4703msec) clat (usec): min=622, max=33874, avg=3190.86, stdev=3376.86 lat (usec): min=623, max=33875, avg=3191.86, stdev=3376.89 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 725], 10.00th=[ 783], 20.00th=[ 848], | 30.00th=[ 938], 40.00th=[ 1221], 50.00th=[ 2089], 60.00th=[ 2966], | 70.00th=[ 3752], 80.00th=[ 5342], 90.00th=[ 6980], 95.00th=[ 8717], | 99.00th=[16319], 99.50th=[18744], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 768, max= 1728, per=22.98%, avg=1208.56, stdev=282.31, samples=9 iops : min= 48, max= 108, avg=75.44, stdev=17.71, samples=9 lat (usec) : 100=10.16%, 250=1.82%, 500=0.13%, 750=3.26%, 1000=13.15% lat (msec) : 2=7.94%, 4=11.59%, 10=22.40%, 20=29.30%, 50=0.26% cpu : usr=0.23%, sys=22.03%, ctx=2018, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4611: Tue Dec 17 07:16:35 2024 read: IOPS=81, BW=1305KiB/s (1336kB/s)(6032KiB/4622msec) clat (usec): min=53, max=30818, avg=8931.41, stdev=5528.20 lat (usec): min=54, max=30818, avg=8932.03, stdev=5528.22 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 64], 10.00th=[ 68], 20.00th=[ 105], | 30.00th=[ 8291], 40.00th=[ 9372], 50.00th=[10159], 60.00th=[11207], | 70.00th=[11863], 80.00th=[13173], 90.00th=[14615], 95.00th=[16057], | 99.00th=[19530], 99.50th=[26608], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 958, max= 1664, per=25.74%, avg=1286.89, stdev=235.38, samples=9 iops : min= 59, max= 104, avg=80.33, stdev=14.87, samples=9 write: IOPS=84, BW=1354KiB/s (1386kB/s)(6256KiB/4622msec) clat (usec): min=657, max=20573, avg=3102.28, stdev=2820.31 lat (usec): min=658, max=20575, avg=3103.45, stdev=2820.24 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 766], 10.00th=[ 799], 20.00th=[ 873], | 30.00th=[ 971], 40.00th=[ 1205], 50.00th=[ 2343], 60.00th=[ 3130], | 70.00th=[ 3949], 80.00th=[ 5342], 90.00th=[ 6915], 95.00th=[ 7898], | 99.00th=[13435], 99.50th=[16909], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 926, max= 1760, per=25.75%, avg=1354.44, stdev=306.03, samples=9 iops : min= 57, max= 110, avg=84.56, stdev=19.28, samples=9 lat (usec) : 100=9.37%, 250=1.82%, 750=1.56%, 1000=14.84% lat (msec) : 2=8.59%, 4=11.20%, 10=26.04%, 20=26.04%, 50=0.52% cpu : usr=0.17%, sys=22.77%, ctx=1959, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4999KiB/s (5119kB/s), 1217KiB/s-1330KiB/s (1246kB/s-1362kB/s), io=23.4MiB (24.5MB), run=4622-4791msec WRITE: bw=5260KiB/s (5386kB/s), 1283KiB/s-1386KiB/s (1313kB/s-1420kB/s), io=24.6MiB (25.8MB), run=4622-4791msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4603: Tue Dec 17 07:16:40 2024 read: IOPS=40, BW=656KiB/s (671kB/s)(5920KiB/9030msec) clat (usec): min=1909, max=26705, avg=4106.77, stdev=3268.93 lat (usec): min=1909, max=26706, avg=4107.56, stdev=3268.92 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2114], 10.00th=[ 2212], 20.00th=[ 2343], | 30.00th=[ 2474], 40.00th=[ 2638], 50.00th=[ 2835], 60.00th=[ 3228], | 70.00th=[ 3621], 80.00th=[ 4948], 90.00th=[ 7308], 95.00th=[10683], | 99.00th=[18744], 99.50th=[21365], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 416, max= 1024, per=25.05%, avg=654.11, stdev=195.38, samples=18 iops : min= 26, max= 64, avg=40.83, stdev=12.17, samples=18 write: IOPS=44, BW=705KiB/s (722kB/s)(6368KiB/9030msec) clat (usec): min=9388, max=40767, avg=18810.39, stdev=5251.60 lat (usec): min=9390, max=40768, avg=18811.83, stdev=5251.60 clat percentiles (usec): | 1.00th=[10683], 5.00th=[12125], 10.00th=[12911], 20.00th=[13829], | 30.00th=[15139], 40.00th=[16909], 50.00th=[18220], 60.00th=[19792], | 70.00th=[21103], 80.00th=[22414], 90.00th=[26084], 95.00th=[27919], | 99.00th=[36439], 99.50th=[37487], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 512, max= 928, per=25.62%, avg=703.89, stdev=131.11, samples=18 iops : min= 32, max= 58, avg=43.94, stdev= 8.13, samples=18 lat (msec) : 2=0.39%, 4=34.90%, 10=10.16%, 20=35.03%, 50=19.53% cpu : usr=0.10%, sys=5.95%, ctx=869, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4604: Tue Dec 17 07:16:40 2024 read: IOPS=39, BW=626KiB/s (641kB/s)(5744KiB/9171msec) clat (usec): min=1958, max=24235, avg=3848.20, stdev=2970.04 lat (usec): min=1959, max=24236, avg=3848.92, stdev=2970.04 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2114], 10.00th=[ 2212], 20.00th=[ 2343], | 30.00th=[ 2474], 40.00th=[ 2606], 50.00th=[ 2769], 60.00th=[ 2999], | 70.00th=[ 3490], 80.00th=[ 4555], 90.00th=[ 6456], 95.00th=[11207], | 99.00th=[15926], 99.50th=[21103], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 320, max= 1152, per=23.83%, avg=622.11, stdev=235.39, samples=18 iops : min= 20, max= 72, avg=38.83, stdev=14.68, samples=18 write: IOPS=44, BW=714KiB/s (731kB/s)(6544KiB/9171msec) clat (usec): min=9355, max=44379, avg=18975.37, stdev=5873.43 lat (usec): min=9356, max=44380, avg=18976.90, stdev=5873.58 clat percentiles (usec): | 1.00th=[10814], 5.00th=[11600], 10.00th=[12256], 20.00th=[13698], | 30.00th=[15270], 40.00th=[16909], 50.00th=[18220], 60.00th=[19530], | 70.00th=[20841], 80.00th=[22938], 90.00th=[26608], 95.00th=[30016], | 99.00th=[38011], 99.50th=[38536], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 512, max= 960, per=25.69%, avg=705.67, stdev=143.60, samples=18 iops : min= 32, max= 60, avg=44.06, stdev= 8.92, samples=18 lat (msec) : 2=0.26%, 4=35.81%, 10=8.33%, 20=35.81%, 50=19.79% cpu : usr=0.10%, sys=6.28%, ctx=880, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4605: Tue Dec 17 07:16:40 2024 read: IOPS=44, BW=710KiB/s (727kB/s)(6256KiB/8806msec) clat (usec): min=1957, max=21922, avg=4093.50, stdev=3105.18 lat (usec): min=1957, max=21922, avg=4094.24, stdev=3105.17 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2147], 10.00th=[ 2212], 20.00th=[ 2376], | 30.00th=[ 2540], 40.00th=[ 2638], 50.00th=[ 2835], 60.00th=[ 3097], | 70.00th=[ 3720], 80.00th=[ 5014], 90.00th=[ 7898], 95.00th=[10945], | 99.00th=[17957], 99.50th=[19530], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 352, max= 1149, per=27.38%, avg=715.00, stdev=223.70, samples=17 iops : min= 22, max= 71, avg=44.59, stdev=13.92, samples=17 write: IOPS=42, BW=685KiB/s (701kB/s)(6032KiB/8806msec) clat (usec): min=10067, max=39122, avg=19078.92, stdev=5446.78 lat (usec): min=10067, max=39125, avg=19080.47, stdev=5446.83 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11994], 10.00th=[12649], 20.00th=[14091], | 30.00th=[15795], 40.00th=[17171], 50.00th=[18482], 60.00th=[19530], | 70.00th=[21365], 80.00th=[23200], 90.00th=[25560], 95.00th=[29230], | 99.00th=[35914], 99.50th=[38536], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 512, max= 896, per=24.73%, avg=679.29, stdev=126.84, samples=17 iops : min= 32, max= 56, avg=42.35, stdev= 7.97, samples=17 lat (msec) : 2=0.65%, 4=36.33%, 10=10.68%, 20=33.46%, 50=18.88% cpu : usr=0.08%, sys=6.42%, ctx=890, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4606: Tue Dec 17 07:16:40 2024 read: IOPS=41, BW=670KiB/s (686kB/s)(6032KiB/9001msec) clat (usec): min=1759, max=36905, avg=4305.04, stdev=4034.55 lat (usec): min=1760, max=36906, avg=4305.80, stdev=4034.52 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2057], 10.00th=[ 2180], 20.00th=[ 2343], | 30.00th=[ 2474], 40.00th=[ 2638], 50.00th=[ 2900], 60.00th=[ 3228], | 70.00th=[ 3916], 80.00th=[ 5342], 90.00th=[ 7373], 95.00th=[11207], | 99.00th=[23987], 99.50th=[32900], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 416, max= 1120, per=25.59%, avg=668.11, stdev=172.96, samples=18 iops : min= 26, max= 70, avg=41.61, stdev=10.88, samples=18 write: IOPS=43, BW=695KiB/s (712kB/s)(6256KiB/9001msec) clat (usec): min=9500, max=36870, avg=18821.02, stdev=5359.94 lat (usec): min=9501, max=36872, avg=18822.72, stdev=5360.05 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11600], 10.00th=[12387], 20.00th=[13960], | 30.00th=[15664], 40.00th=[16909], 50.00th=[18482], 60.00th=[19530], | 70.00th=[20841], 80.00th=[22676], 90.00th=[25297], 95.00th=[29754], | 99.00th=[35914], 99.50th=[36439], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 512, max= 864, per=25.29%, avg=694.78, stdev=133.56, samples=18 iops : min= 32, max= 54, avg=43.28, stdev= 8.41, samples=18 lat (msec) : 2=1.30%, 4=33.46%, 10=11.46%, 20=34.51%, 50=19.27% cpu : usr=0.12%, sys=6.14%, ctx=904, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2612KiB/s (2674kB/s), 626KiB/s-710KiB/s (641kB/s-727kB/s), io=23.4MiB (24.5MB), run=8806-9171msec WRITE: bw=2748KiB/s (2814kB/s), 685KiB/s-714KiB/s (701kB/s-731kB/s), io=24.6MiB (25.8MB), run=8806-9171msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4624: Tue Dec 17 07:16:43 2024 read: IOPS=3, BW=3663KiB/s (3751kB/s)(7168KiB/1957msec) clat (usec): min=623, max=794323, avg=215717.93, stdev=314043.24 lat (usec): min=624, max=794324, avg=215718.83, stdev=314043.22 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 627], 10.00th=[ 627], 20.00th=[ 20055], | 30.00th=[ 52691], 40.00th=[ 52691], 50.00th=[ 53216], 60.00th=[ 63177], | 70.00th=[ 63177], 80.00th=[526386], 90.00th=[792724], 95.00th=[792724], | 99.00th=[792724], 99.50th=[792724], 99.90th=[792724], 99.95th=[792724], | 99.99th=[792724] bw ( KiB/s): min= 4096, max= 6144, per=49.88%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2616KiB/s (2679kB/s)(5120KiB/1957msec) clat (msec): min=26, max=169, avg=72.71, stdev=61.80 lat (msec): min=26, max=169, avg=72.78, stdev=61.81 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 34], 60.00th=[ 34], | 70.00th=[ 100], 80.00th=[ 100], 90.00th=[ 169], 95.00th=[ 169], | 99.00th=[ 169], 99.50th=[ 169], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 2048, max= 5052, per=24.70%, avg=3550.00, stdev=2124.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (usec) : 750=8.33% lat (msec) : 50=33.33%, 100=33.33%, 250=8.33%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=20.04%, ctx=674, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4625: Tue Dec 17 07:16:43 2024 read: IOPS=1, BW=1027KiB/s (1051kB/s)(2048KiB/1995msec) clat (msec): min=104, max=798, avg=451.51, stdev=490.99 lat (msec): min=104, max=798, avg=451.53, stdev=491.03 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 105], 10.00th=[ 105], 20.00th=[ 105], | 30.00th=[ 105], 40.00th=[ 105], 50.00th=[ 105], 60.00th=[ 802], | 70.00th=[ 802], 80.00th=[ 802], 90.00th=[ 802], 95.00th=[ 802], | 99.00th=[ 802], 99.50th=[ 802], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 2048, max= 2048, per=19.95%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5133KiB/s (5256kB/s)(10.0MiB/1995msec) clat (msec): min=23, max=554, avg=106.59, stdev=161.27 lat (msec): min=23, max=554, avg=106.67, stdev=161.26 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 33], | 30.00th=[ 35], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 56], | 70.00th=[ 67], 80.00th=[ 67], 90.00th=[ 150], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 3373, max= 6144, per=33.11%, avg=4758.50, stdev=1959.39, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 lat (msec) : 50=41.67%, 100=25.00%, 250=16.67%, 750=8.33%, 1000=8.33% cpu : usr=0.05%, sys=21.16%, ctx=365, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4626: Tue Dec 17 07:16:43 2024 read: IOPS=6, BW=6295KiB/s (6446kB/s)(6144KiB/976msec) clat (usec): min=417, max=525024, avg=121787.62, stdev=207401.80 lat (usec): min=418, max=525025, avg=121788.39, stdev=207401.79 clat percentiles (usec): | 1.00th=[ 416], 5.00th=[ 416], 10.00th=[ 416], 20.00th=[ 482], | 30.00th=[ 482], 40.00th=[ 519], 50.00th=[ 519], 60.00th=[ 40633], | 70.00th=[164627], 80.00th=[164627], 90.00th=[526386], 95.00th=[526386], | 99.00th=[526386], 99.50th=[526386], 99.90th=[526386], 99.95th=[526386], | 99.99th=[526386] bw ( KiB/s): min= 4096, max= 4096, per=39.90%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6295KiB/s (6446kB/s)(6144KiB/976msec) clat (usec): min=21812, max=62506, avg=38607.89, stdev=13774.57 lat (usec): min=21861, max=62575, avg=38657.42, stdev=13785.29 clat percentiles (usec): | 1.00th=[21890], 5.00th=[21890], 10.00th=[21890], 20.00th=[30278], | 30.00th=[30278], 40.00th=[35390], 50.00th=[35390], 60.00th=[38536], | 70.00th=[42730], 80.00th=[42730], 90.00th=[62653], 95.00th=[62653], | 99.00th=[62653], 99.50th=[62653], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 8192, max= 8192, per=57.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 50=50.00%, 100=8.33%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=30.84%, ctx=379, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4627: Tue Dec 17 07:16:43 2024 read: IOPS=5, BW=5476KiB/s (5607kB/s)(5120KiB/935msec) clat (usec): min=521, max=514436, avg=133338.20, stdev=216372.04 lat (usec): min=522, max=514437, avg=133339.52, stdev=216371.80 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 523], 10.00th=[ 523], 20.00th=[ 523], | 30.00th=[ 701], 40.00th=[ 701], 50.00th=[ 67634], 60.00th=[ 67634], | 70.00th=[ 83362], 80.00th=[ 83362], 90.00th=[513803], 95.00th=[513803], | 99.00th=[513803], 99.50th=[513803], 99.90th=[513803], 99.95th=[513803], | 99.99th=[513803] bw ( KiB/s): min= 2048, max= 2048, per=19.95%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7666KiB/s (7850kB/s)(7168KiB/935msec) clat (usec): min=21344, max=57714, avg=35283.92, stdev=15571.20 lat (usec): min=21405, max=57757, avg=35338.42, stdev=15573.04 clat percentiles (usec): | 1.00th=[21365], 5.00th=[21365], 10.00th=[21365], 20.00th=[22414], | 30.00th=[23725], 40.00th=[23725], 50.00th=[28181], 60.00th=[38536], | 70.00th=[38536], 80.00th=[55313], 90.00th=[57934], 95.00th=[57934], | 99.00th=[57934], 99.50th=[57934], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 8192, max= 8192, per=57.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 50=41.67%, 100=33.33%, 750=8.33% cpu : usr=0.00%, sys=30.41%, ctx=389, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.0MiB/s (10.5MB/s), 1027KiB/s-6295KiB/s (1051kB/s-6446kB/s), io=20.0MiB (20.0MB), run=935-1995msec WRITE: bw=14.0MiB/s (14.7MB/s), 2616KiB/s-7666KiB/s (2679kB/s-7850kB/s), io=28.0MiB (29.4MB), run=935-1995msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4628: Tue Dec 17 07:16:43 2024 read: IOPS=6, BW=6781KiB/s (6944kB/s)(7168KiB/1057msec) clat (usec): min=20441, max=71324, avg=34499.04, stdev=19200.64 lat (usec): min=20442, max=71325, avg=34500.31, stdev=19200.66 clat percentiles (usec): | 1.00th=[20317], 5.00th=[20317], 10.00th=[20317], 20.00th=[21365], | 30.00th=[21365], 40.00th=[21365], 50.00th=[26084], 60.00th=[31327], | 70.00th=[31327], 80.00th=[49546], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4844KiB/s (4960kB/s)(5120KiB/1057msec) clat (msec): min=24, max=659, avg=155.68, stdev=281.71 lat (msec): min=25, max=659, avg=155.76, stdev=281.75 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 26], | 30.00th=[ 27], 40.00th=[ 27], 50.00th=[ 30], 60.00th=[ 30], | 70.00th=[ 39], 80.00th=[ 39], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 4096, max= 4096, per=29.22%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 lat (msec) : 50=83.33%, 100=8.33%, 750=8.33% cpu : usr=0.00%, sys=7.39%, ctx=83, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4629: Tue Dec 17 07:16:43 2024 read: IOPS=0, BW=1001KiB/s (1026kB/s)(2048KiB/2045msec) clat (msec): min=150, max=763, avg=457.05, stdev=433.08 lat (msec): min=150, max=763, avg=457.06, stdev=433.08 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 150], 10.00th=[ 150], 20.00th=[ 150], | 30.00th=[ 150], 40.00th=[ 150], 50.00th=[ 150], 60.00th=[ 760], | 70.00th=[ 760], 80.00th=[ 760], 90.00th=[ 760], 95.00th=[ 760], | 99.00th=[ 760], 99.50th=[ 760], 99.90th=[ 760], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 2048, max= 2048, per=20.45%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=4, BW=5007KiB/s (5128kB/s)(10.0MiB/2045msec) clat (msec): min=26, max=511, avg=112.01, stdev=145.72 lat (msec): min=26, max=511, avg=112.21, stdev=145.63 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 31], | 30.00th=[ 36], 40.00th=[ 40], 50.00th=[ 66], 60.00th=[ 70], | 70.00th=[ 105], 80.00th=[ 105], 90.00th=[ 150], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 2048, max= 6144, per=24.35%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 lat (msec) : 50=33.33%, 100=25.00%, 250=25.00%, 750=8.33%, 1000=8.33% cpu : usr=0.05%, sys=5.97%, ctx=90, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4630: Tue Dec 17 07:16:43 2024 read: IOPS=3, BW=3087KiB/s (3162kB/s)(6144KiB/1990msec) clat (msec): min=16, max=105, avg=43.31, stdev=31.94 lat (msec): min=16, max=105, avg=43.32, stdev=31.94 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 35], | 70.00th=[ 47], 80.00th=[ 47], 90.00th=[ 106], 95.00th=[ 106], | 99.00th=[ 106], 99.50th=[ 106], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 4096, max= 6144, per=51.13%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=3, BW=3087KiB/s (3162kB/s)(6144KiB/1990msec) clat (msec): min=27, max=749, avg=276.24, stdev=307.91 lat (msec): min=27, max=749, avg=276.29, stdev=307.91 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 57], | 30.00th=[ 57], 40.00th=[ 108], 50.00th=[ 108], 60.00th=[ 138], | 70.00th=[ 584], 80.00th=[ 584], 90.00th=[ 751], 95.00th=[ 751], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 2048, max= 6144, per=24.35%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 lat (msec) : 20=8.33%, 50=41.67%, 100=8.33%, 250=25.00%, 750=16.67% cpu : usr=0.00%, sys=5.38%, ctx=65, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4631: Tue Dec 17 07:16:43 2024 read: IOPS=2, BW=2902KiB/s (2972kB/s)(5120KiB/1764msec) clat (msec): min=12, max=645, avg=172.84, stdev=266.34 lat (msec): min=12, max=645, avg=172.84, stdev=266.34 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 13], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 65], 60.00th=[ 65], | 70.00th=[ 99], 80.00th=[ 99], 90.00th=[ 651], 95.00th=[ 651], | 99.00th=[ 651], 99.50th=[ 651], 99.90th=[ 651], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 4096, max= 4096, per=40.90%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=3, BW=4063KiB/s (4161kB/s)(7168KiB/1764msec) clat (msec): min=25, max=568, avg=127.61, stdev=199.95 lat (msec): min=25, max=568, avg=127.68, stdev=199.96 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 29], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 53], | 70.00th=[ 53], 80.00th=[ 159], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 4096, max= 6144, per=34.08%, avg=4778.67, stdev=1182.41, samples=3 iops : min= 4, max= 6, avg= 4.67, stdev= 1.15, samples=3 lat (msec) : 20=8.33%, 50=41.67%, 100=25.00%, 250=8.33%, 750=16.67% cpu : usr=0.06%, sys=4.37%, ctx=38, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9.78MiB/s (10.3MB/s), 1001KiB/s-6781KiB/s (1026kB/s-6944kB/s), io=20.0MiB (20.0MB), run=1057-2045msec WRITE: bw=13.7MiB/s (14.4MB/s), 3087KiB/s-5007KiB/s (3162kB/s-5128kB/s), io=28.0MiB (29.4MB), run=1057-2045msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4638: Tue Dec 17 07:16:46 2024 read: IOPS=6, BW=26.2MiB/s (27.5MB/s)(12.0MiB/458msec) clat (msec): min=59, max=326, avg=150.35, stdev=152.99 lat (msec): min=59, max=326, avg=150.35, stdev=152.99 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 60], 10.00th=[ 60], 20.00th=[ 60], | 30.00th=[ 60], 40.00th=[ 65], 50.00th=[ 65], 60.00th=[ 65], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] lat (msec) : 100=66.67%, 500=33.33% cpu : usr=0.00%, sys=14.00%, ctx=72, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4639: Tue Dec 17 07:16:46 2024 read: IOPS=0, BW=2473KiB/s (2533kB/s)(4096KiB/1656msec) clat (nsec): min=178373k, max=178373k, avg=178372911.00, stdev= 0.00 lat (nsec): min=178376k, max=178376k, avg=178375986.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 180], 10.00th=[ 180], 20.00th=[ 180], | 30.00th=[ 180], 40.00th=[ 180], 50.00th=[ 180], 60.00th=[ 180], | 70.00th=[ 180], 80.00th=[ 180], 90.00th=[ 180], 95.00th=[ 180], | 99.00th=[ 180], 99.50th=[ 180], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 8192, max= 8192, per=82.81%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4947KiB/s (5066kB/s)(8192KiB/1656msec) clat (msec): min=312, max=1142, avg=727.58, stdev=587.25 lat (msec): min=312, max=1143, avg=727.87, stdev=587.22 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 313], 10.00th=[ 313], 20.00th=[ 313], | 30.00th=[ 313], 40.00th=[ 313], 50.00th=[ 313], 60.00th=[ 1150], | 70.00th=[ 1150], 80.00th=[ 1150], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7585, max= 7585, per=39.84%, avg=7585.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.06%, sys=5.74%, ctx=59, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4641: Tue Dec 17 07:16:46 2024 write: IOPS=1, BW=7140KiB/s (7311kB/s)(12.0MiB/1721msec) clat (msec): min=86, max=1139, avg=569.04, stdev=531.67 lat (msec): min=87, max=1139, avg=569.30, stdev=531.66 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 87], 10.00th=[ 87], 20.00th=[ 87], | 30.00th=[ 87], 40.00th=[ 481], 50.00th=[ 481], 60.00th=[ 481], | 70.00th=[ 1133], 80.00th=[ 1133], 90.00th=[ 1133], 95.00th=[ 1133], | 99.00th=[ 1133], 99.50th=[ 1133], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 5760, max= 5769, per=30.28%, avg=5764.50, stdev= 6.36, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.12%, sys=5.35%, ctx=45, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4643: Tue Dec 17 07:16:46 2024 write: IOPS=6, BW=25.3MiB/s (26.5MB/s)(12.0MiB/475msec) clat (msec): min=70, max=293, avg=154.08, stdev=121.38 lat (msec): min=74, max=293, avg=155.58, stdev=120.10 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 71], 10.00th=[ 71], 20.00th=[ 71], | 30.00th=[ 71], 40.00th=[ 100], 50.00th=[ 100], 60.00th=[ 100], | 70.00th=[ 292], 80.00th=[ 292], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] lat (msec) : 100=66.67%, 500=33.33% cpu : usr=0.00%, sys=16.24%, ctx=60, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9894KiB/s (10.1MB/s), 2473KiB/s-26.2MiB/s (2533kB/s-27.5MB/s), io=16.0MiB (16.8MB), run=458-1656msec WRITE: bw=18.6MiB/s (19.5MB/s), 4947KiB/s-25.3MiB/s (5066kB/s-26.5MB/s), io=32.0MiB (33.6MB), run=475-1721msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4640: Tue Dec 17 07:16:46 2024 read: IOPS=2, BW=11.7MiB/s (12.3MB/s)(12.0MiB/1026msec) clat (msec): min=2, max=866, avg=337.77, stdev=463.35 lat (msec): min=2, max=866, avg=337.77, stdev=463.35 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 3], 40.00th=[ 146], 50.00th=[ 146], 60.00th=[ 146], | 70.00th=[ 869], 80.00th=[ 869], 90.00th=[ 869], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 8192, max= 8192, per=88.35%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 4=33.33%, 250=33.33%, 1000=33.33% cpu : usr=0.10%, sys=76.10%, ctx=276, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4642: Tue Dec 17 07:16:46 2024 read: IOPS=0, BW=2318KiB/s (2374kB/s)(4096KiB/1767msec) clat (nsec): min=158232k, max=158232k, avg=158232316.00, stdev= 0.00 lat (nsec): min=158234k, max=158234k, avg=158234201.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 159], 50.00th=[ 159], 60.00th=[ 159], | 70.00th=[ 159], 80.00th=[ 159], 90.00th=[ 159], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 8192, max= 8192, per=88.35%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4636KiB/s (4747kB/s)(8192KiB/1767msec) clat (msec): min=388, max=1204, avg=796.69, stdev=576.88 lat (msec): min=388, max=1204, avg=796.87, stdev=576.98 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 388], 10.00th=[ 388], 20.00th=[ 388], | 30.00th=[ 388], 40.00th=[ 388], 50.00th=[ 388], 60.00th=[ 1200], | 70.00th=[ 1200], 80.00th=[ 1200], 90.00th=[ 1200], 95.00th=[ 1200], | 99.00th=[ 1200], 99.50th=[ 1200], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 8192, max= 8192, per=44.18%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=14.95%, ctx=239, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4644: Tue Dec 17 07:16:46 2024 write: IOPS=1, BW=7497KiB/s (7677kB/s)(12.0MiB/1639msec) clat (msec): min=266, max=1042, avg=541.77, stdev=434.07 lat (msec): min=266, max=1042, avg=542.07, stdev=434.09 clat percentiles (msec): | 1.00th=[ 268], 5.00th=[ 268], 10.00th=[ 268], 20.00th=[ 268], | 30.00th=[ 268], 40.00th=[ 317], 50.00th=[ 317], 60.00th=[ 317], | 70.00th=[ 1045], 80.00th=[ 1045], 90.00th=[ 1045], 95.00th=[ 1045], | 99.00th=[ 1045], 99.50th=[ 1045], 99.90th=[ 1045], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 8192, max= 8192, per=44.18%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=13.80%, ctx=54, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4645: Tue Dec 17 07:16:46 2024 write: IOPS=2, BW=8879KiB/s (9092kB/s)(12.0MiB/1384msec) clat (msec): min=107, max=1042, avg=451.67, stdev=513.59 lat (msec): min=108, max=1042, avg=451.97, stdev=513.55 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 108], 10.00th=[ 108], 20.00th=[ 108], | 30.00th=[ 108], 40.00th=[ 205], 50.00th=[ 205], 60.00th=[ 205], | 70.00th=[ 1045], 80.00th=[ 1045], 90.00th=[ 1045], 95.00th=[ 1045], | 99.00th=[ 1045], 99.50th=[ 1045], 99.90th=[ 1045], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 8192, max= 8192, per=44.18%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=21.98%, ctx=72, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9272KiB/s (9495kB/s), 2318KiB/s-11.7MiB/s (2374kB/s-12.3MB/s), io=16.0MiB (16.8MB), run=1026-1767msec WRITE: bw=18.1MiB/s (18.0MB/s), 4636KiB/s-8879KiB/s (4747kB/s-9092kB/s), io=32.0MiB (33.6MB), run=1384-1767msec