== sanity test 398b: DIO and buffer IO race ============== 01:20:13 (1735107613) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.42382 s, 35.3 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=26847: Wed Dec 25 01:20:30 2024 read: IOPS=104, BW=419KiB/s (429kB/s)(5992KiB/14316msec) clat (usec): min=53, max=33590, avg=5261.93, stdev=3047.39 lat (usec): min=53, max=33592, avg=5262.86, stdev=3047.38 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 1745], 10.00th=[ 2638], 20.00th=[ 3130], | 30.00th=[ 3589], 40.00th=[ 4047], 50.00th=[ 4686], 60.00th=[ 5407], | 70.00th=[ 6128], 80.00th=[ 7177], 90.00th=[ 8717], 95.00th=[10421], | 99.00th=[15533], 99.50th=[20841], 99.90th=[25297], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 184, max= 528, per=25.06%, avg=416.18, stdev=64.69, samples=28 iops : min= 46, max= 132, avg=103.96, stdev=16.16, samples=28 write: IOPS=109, BW=440KiB/s (450kB/s)(6296KiB/14316msec) clat (usec): min=504, max=189027, avg=4041.73, stdev=5755.95 lat (usec): min=505, max=189028, avg=4042.65, stdev=5756.00 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 586], 10.00th=[ 644], 20.00th=[ 824], | 30.00th=[ 1012], 40.00th=[ 2376], 50.00th=[ 3261], 60.00th=[ 4047], | 70.00th=[ 5014], 80.00th=[ 6521], 90.00th=[ 8586], 95.00th=[ 10552], | 99.00th=[ 14353], 99.50th=[ 16057], 99.90th=[ 23200], 99.95th=[189793], | 99.99th=[189793] bw ( KiB/s): min= 200, max= 576, per=25.13%, avg=437.04, stdev=71.48, samples=28 iops : min= 50, max= 144, avg=109.18, stdev=17.86, samples=28 lat (usec) : 100=1.46%, 250=0.88%, 750=8.07%, 1000=7.13% lat (msec) : 2=4.23%, 4=27.73%, 10=44.40%, 20=5.66%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=23.30%, ctx=5978, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26848: Wed Dec 25 01:20:30 2024 read: IOPS=102, BW=411KiB/s (421kB/s)(5940KiB/14454msec) clat (usec): min=58, max=33160, avg=5257.95, stdev=2820.79 lat (usec): min=59, max=33161, avg=5260.67, stdev=2821.24 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 2278], 10.00th=[ 2638], 20.00th=[ 3130], | 30.00th=[ 3589], 40.00th=[ 4080], 50.00th=[ 4752], 60.00th=[ 5407], | 70.00th=[ 6194], 80.00th=[ 7111], 90.00th=[ 8455], 95.00th=[10159], | 99.00th=[15270], 99.50th=[16188], 99.90th=[20055], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 152, max= 520, per=24.50%, avg=406.96, stdev=73.27, samples=28 iops : min= 38, max= 130, avg=101.61, stdev=18.33, samples=28 write: IOPS=109, BW=439KiB/s (450kB/s)(6348KiB/14454msec) clat (usec): min=476, max=217200, avg=4139.38, stdev=6363.62 lat (usec): min=477, max=217200, avg=4140.61, stdev=6363.59 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 603], 10.00th=[ 676], 20.00th=[ 848], | 30.00th=[ 1106], 40.00th=[ 2573], 50.00th=[ 3326], 60.00th=[ 4047], | 70.00th=[ 5080], 80.00th=[ 6652], 90.00th=[ 8717], 95.00th=[ 10683], | 99.00th=[ 15533], 99.50th=[ 19006], 99.90th=[ 23200], 99.95th=[217056], | 99.99th=[217056] bw ( KiB/s): min= 216, max= 520, per=24.62%, avg=428.07, stdev=67.52, samples=28 iops : min= 54, max= 130, avg=106.89, stdev=16.82, samples=28 lat (usec) : 100=1.01%, 250=0.65%, 500=0.16%, 750=7.75%, 1000=6.84% lat (msec) : 2=4.10%, 4=28.52%, 10=45.18%, 20=5.66%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=23.56%, ctx=6022, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26849: Wed Dec 25 01:20:30 2024 read: IOPS=104, BW=418KiB/s (428kB/s)(6028KiB/14430msec) clat (usec): min=57, max=217363, avg=5289.15, stdev=6174.39 lat (usec): min=57, max=217364, avg=5290.71, stdev=6174.31 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 2040], 10.00th=[ 2540], 20.00th=[ 3097], | 30.00th=[ 3490], 40.00th=[ 3982], 50.00th=[ 4621], 60.00th=[ 5342], | 70.00th=[ 6128], 80.00th=[ 7046], 90.00th=[ 8356], 95.00th=[ 10159], | 99.00th=[ 15926], 99.50th=[ 17957], 99.90th=[ 29492], 99.95th=[217056], | 99.99th=[217056] bw ( KiB/s): min= 192, max= 520, per=24.91%, avg=413.82, stdev=65.00, samples=28 iops : min= 48, max= 130, avg=103.32, stdev=16.24, samples=28 write: IOPS=108, BW=434KiB/s (444kB/s)(6260KiB/14430msec) clat (usec): min=512, max=25913, avg=4078.66, stdev=3477.73 lat (usec): min=513, max=25914, avg=4079.51, stdev=3477.74 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 578], 10.00th=[ 644], 20.00th=[ 857], | 30.00th=[ 1139], 40.00th=[ 2507], 50.00th=[ 3261], 60.00th=[ 4047], | 70.00th=[ 5342], 80.00th=[ 6849], 90.00th=[ 9110], 95.00th=[10683], | 99.00th=[14746], 99.50th=[16057], 99.90th=[18744], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 272, max= 574, per=24.58%, avg=427.50, stdev=65.56, samples=28 iops : min= 68, max= 143, avg=106.75, stdev=16.38, samples=28 lat (usec) : 100=1.30%, 250=0.78%, 500=0.03%, 750=7.68%, 1000=6.15% lat (msec) : 2=5.01%, 4=28.97%, 10=43.98%, 20=5.89%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=23.42%, ctx=5924, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26850: Wed Dec 25 01:20:30 2024 read: IOPS=105, BW=422KiB/s (432kB/s)(6052KiB/14353msec) clat (usec): min=55, max=216344, avg=5466.89, stdev=6164.49 lat (usec): min=55, max=216345, avg=5468.97, stdev=6164.23 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 2245], 10.00th=[ 2573], 20.00th=[ 3064], | 30.00th=[ 3654], 40.00th=[ 4228], 50.00th=[ 4883], 60.00th=[ 5473], | 70.00th=[ 6259], 80.00th=[ 7242], 90.00th=[ 8717], 95.00th=[ 10552], | 99.00th=[ 15139], 99.50th=[ 17433], 99.90th=[ 25822], 99.95th=[217056], | 99.99th=[217056] bw ( KiB/s): min= 208, max= 512, per=25.21%, avg=418.75, stdev=63.03, samples=28 iops : min= 52, max= 128, avg=104.61, stdev=15.75, samples=28 write: IOPS=108, BW=434KiB/s (445kB/s)(6236KiB/14353msec) clat (usec): min=480, max=20458, avg=3853.12, stdev=3379.26 lat (usec): min=481, max=20458, avg=3854.07, stdev=3379.28 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 570], 10.00th=[ 627], 20.00th=[ 799], | 30.00th=[ 988], 40.00th=[ 2245], 50.00th=[ 3163], 60.00th=[ 3851], | 70.00th=[ 5014], 80.00th=[ 6521], 90.00th=[ 8586], 95.00th=[10421], | 99.00th=[14877], 99.50th=[16909], 99.90th=[20055], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 232, max= 550, per=24.85%, avg=432.14, stdev=66.58, samples=28 iops : min= 58, max= 137, avg=107.96, stdev=16.59, samples=28 lat (usec) : 100=1.04%, 250=0.72%, 500=0.13%, 750=8.92%, 1000=6.54% lat (msec) : 2=4.23%, 4=27.73%, 10=44.76%, 20=5.66%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=23.50%, ctx=6013, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1661KiB/s (1701kB/s), 411KiB/s-422KiB/s (421kB/s-432kB/s), io=23.4MiB (24.6MB), run=14316-14454msec WRITE: bw=1739KiB/s (1781kB/s), 434KiB/s-440KiB/s (444kB/s-450kB/s), io=24.6MiB (25.7MB), run=14316-14454msec rand-rw: (groupid=0, jobs=1): err= 0: pid=26851: Wed Dec 25 01:20:35 2024 read: IOPS=77, BW=311KiB/s (318kB/s)(5992KiB/19273msec) clat (usec): min=1707, max=250289, avg=5339.44, stdev=7351.01 lat (usec): min=1708, max=250290, avg=5340.35, stdev=7350.99 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2147], 10.00th=[ 2343], 20.00th=[ 2540], | 30.00th=[ 2835], 40.00th=[ 3130], 50.00th=[ 3589], 60.00th=[ 4424], | 70.00th=[ 5538], 80.00th=[ 7111], 90.00th=[ 10552], 95.00th=[ 13173], | 99.00th=[ 19268], 99.50th=[ 20317], 99.90th=[ 26084], 99.95th=[250610], | 99.99th=[250610] bw ( KiB/s): min= 128, max= 640, per=24.88%, avg=309.71, stdev=125.40, samples=38 iops : min= 32, max= 160, avg=77.29, stdev=31.40, samples=38 write: IOPS=81, BW=327KiB/s (335kB/s)(6296KiB/19273msec) clat (usec): min=2583, max=33774, avg=7124.91, stdev=3906.11 lat (usec): min=2584, max=33775, avg=7125.94, stdev=3906.10 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3392], 10.00th=[ 3621], 20.00th=[ 4015], | 30.00th=[ 4424], 40.00th=[ 5014], 50.00th=[ 5866], 60.00th=[ 6849], | 70.00th=[ 8160], 80.00th=[10028], 90.00th=[12649], 95.00th=[14353], | 99.00th=[19792], 99.50th=[21890], 99.90th=[31851], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 136, max= 640, per=24.90%, avg=324.45, stdev=123.31, samples=38 iops : min= 34, max= 160, avg=80.97, stdev=30.83, samples=38 lat (msec) : 2=0.65%, 4=37.11%, 10=46.32%, 20=15.10%, 50=0.78% lat (msec) : 500=0.03% cpu : usr=0.28%, sys=11.51%, ctx=3662, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26852: Wed Dec 25 01:20:35 2024 read: IOPS=78, BW=312KiB/s (320kB/s)(5940KiB/19029msec) clat (usec): min=1791, max=222253, avg=5194.16, stdev=6671.04 lat (usec): min=1792, max=222254, avg=5195.44, stdev=6670.90 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2180], 10.00th=[ 2311], 20.00th=[ 2573], | 30.00th=[ 2835], 40.00th=[ 3130], 50.00th=[ 3523], 60.00th=[ 4293], | 70.00th=[ 5211], 80.00th=[ 7046], 90.00th=[ 10421], 95.00th=[ 13173], | 99.00th=[ 18220], 99.50th=[ 19792], 99.90th=[ 25035], 99.95th=[221250], | 99.99th=[221250] bw ( KiB/s): min= 112, max= 640, per=25.05%, avg=311.89, stdev=127.22, samples=38 iops : min= 28, max= 160, avg=77.89, stdev=31.84, samples=38 write: IOPS=83, BW=334KiB/s (342kB/s)(6348KiB/19029msec) clat (usec): min=2554, max=33277, avg=7093.22, stdev=4043.44 lat (usec): min=2555, max=33278, avg=7094.32, stdev=4043.48 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3326], 10.00th=[ 3621], 20.00th=[ 3982], | 30.00th=[ 4359], 40.00th=[ 4883], 50.00th=[ 5669], 60.00th=[ 6915], | 70.00th=[ 8094], 80.00th=[ 9765], 90.00th=[12518], 95.00th=[14615], | 99.00th=[23725], 99.50th=[25822], 99.90th=[30540], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 168, max= 672, per=25.58%, avg=333.37, stdev=120.77, samples=38 iops : min= 42, max= 168, avg=83.26, stdev=30.22, samples=38 lat (msec) : 2=0.75%, 4=37.37%, 10=47.01%, 20=13.93%, 50=0.91% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=11.72%, ctx=3676, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26853: Wed Dec 25 01:20:35 2024 read: IOPS=78, BW=313KiB/s (320kB/s)(6028KiB/19284msec) clat (usec): min=1791, max=28636, avg=5113.35, stdev=3573.39 lat (usec): min=1791, max=28637, avg=5114.20, stdev=3573.38 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2212], 10.00th=[ 2343], 20.00th=[ 2573], | 30.00th=[ 2868], 40.00th=[ 3228], 50.00th=[ 3687], 60.00th=[ 4490], | 70.00th=[ 5407], 80.00th=[ 7177], 90.00th=[10290], 95.00th=[12911], | 99.00th=[17957], 99.50th=[20317], 99.90th=[25822], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 120, max= 624, per=25.00%, avg=311.26, stdev=119.33, samples=38 iops : min= 30, max= 156, avg=77.74, stdev=29.85, samples=38 write: IOPS=81, BW=325KiB/s (332kB/s)(6260KiB/19284msec) clat (msec): min=2, max=208, avg= 7.35, stdev= 6.57 lat (msec): min=2, max=208, avg= 7.35, stdev= 6.57 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 13], 95.00th=[ 16], | 99.00th=[ 24], 99.50th=[ 27], 99.90th=[ 34], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 160, max= 704, per=24.79%, avg=323.05, stdev=125.25, samples=38 iops : min= 40, max= 176, avg=80.68, stdev=31.35, samples=38 lat (msec) : 2=0.55%, 4=36.07%, 10=47.56%, 20=14.62%, 50=1.17% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=11.63%, ctx=3739, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26854: Wed Dec 25 01:20:35 2024 read: IOPS=79, BW=316KiB/s (324kB/s)(6052KiB/19128msec) clat (usec): min=1774, max=28813, avg=5109.74, stdev=3514.89 lat (usec): min=1775, max=28813, avg=5110.57, stdev=3514.89 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2180], 10.00th=[ 2311], 20.00th=[ 2573], | 30.00th=[ 2802], 40.00th=[ 3163], 50.00th=[ 3621], 60.00th=[ 4490], | 70.00th=[ 5538], 80.00th=[ 7373], 90.00th=[10290], 95.00th=[12649], | 99.00th=[16909], 99.50th=[19006], 99.90th=[25560], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 112, max= 638, per=25.37%, avg=315.84, stdev=123.06, samples=38 iops : min= 28, max= 159, avg=78.89, stdev=30.72, samples=38 write: IOPS=81, BW=326KiB/s (334kB/s)(6236KiB/19128msec) clat (msec): min=2, max=204, avg= 7.26, stdev= 6.44 lat (msec): min=2, max=204, avg= 7.26, stdev= 6.44 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 16], | 99.00th=[ 24], 99.50th=[ 27], 99.90th=[ 34], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 136, max= 616, per=24.98%, avg=325.53, stdev=115.87, samples=38 iops : min= 34, max= 154, avg=81.32, stdev=28.93, samples=38 lat (msec) : 2=0.49%, 4=36.10%, 10=48.86%, 20=13.57%, 50=0.94% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=11.60%, ctx=3697, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1245KiB/s (1275kB/s), 311KiB/s-316KiB/s (318kB/s-324kB/s), io=23.4MiB (24.6MB), run=19029-19284msec WRITE: bw=1304KiB/s (1335kB/s), 325KiB/s-334KiB/s (332kB/s-342kB/s), io=24.6MiB (25.7MB), run=19029-19284msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=26881: Wed Dec 25 01:20:42 2024 read: IOPS=60, BW=972KiB/s (995kB/s)(5920KiB/6093msec) clat (usec): min=1967, max=32511, avg=6800.58, stdev=4770.37 lat (usec): min=1967, max=32512, avg=6801.72, stdev=4770.27 clat percentiles (usec): | 1.00th=[ 2245], 5.00th=[ 2540], 10.00th=[ 2769], 20.00th=[ 3195], | 30.00th=[ 3556], 40.00th=[ 3982], 50.00th=[ 4686], 60.00th=[ 5997], | 70.00th=[ 8094], 80.00th=[10552], 90.00th=[13698], 95.00th=[16188], | 99.00th=[21627], 99.50th=[28181], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 734, max= 1248, per=24.74%, avg=959.83, stdev=184.81, samples=12 iops : min= 45, max= 78, avg=59.92, stdev=11.65, samples=12 write: IOPS=65, BW=1045KiB/s (1070kB/s)(6368KiB/6093msec) clat (usec): min=2822, max=43495, avg=8953.13, stdev=6019.23 lat (usec): min=2823, max=43496, avg=8955.02, stdev=6019.30 clat percentiles (usec): | 1.00th=[ 3097], 5.00th=[ 3687], 10.00th=[ 4015], 20.00th=[ 4621], | 30.00th=[ 5211], 40.00th=[ 5997], 50.00th=[ 6849], 60.00th=[ 8094], | 70.00th=[ 9896], 80.00th=[12387], 90.00th=[16450], 95.00th=[20579], | 99.00th=[31589], 99.50th=[42206], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 702, max= 1312, per=25.54%, avg=1042.50, stdev=185.65, samples=12 iops : min= 43, max= 82, avg=65.08, stdev=11.75, samples=12 lat (msec) : 2=0.13%, 4=24.22%, 10=50.39%, 20=21.35%, 50=3.91% cpu : usr=0.08%, sys=10.11%, ctx=927, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26882: Wed Dec 25 01:20:42 2024 read: IOPS=58, BW=936KiB/s (958kB/s)(5744KiB/6137msec) clat (usec): min=1708, max=38610, avg=6592.32, stdev=5066.43 lat (usec): min=1708, max=38611, avg=6593.16, stdev=5066.46 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2409], 10.00th=[ 2606], 20.00th=[ 2966], | 30.00th=[ 3425], 40.00th=[ 3884], 50.00th=[ 4490], 60.00th=[ 5735], | 70.00th=[ 7373], 80.00th=[ 9765], 90.00th=[13435], 95.00th=[16909], | 99.00th=[25035], 99.50th=[29230], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 640, max= 1440, per=24.05%, avg=933.08, stdev=263.92, samples=12 iops : min= 40, max= 90, avg=58.25, stdev=16.43, samples=12 write: IOPS=66, BW=1066KiB/s (1092kB/s)(6544KiB/6137msec) clat (usec): min=2634, max=37226, avg=9139.08, stdev=5979.73 lat (usec): min=2635, max=37228, avg=9140.99, stdev=5979.79 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3425], 10.00th=[ 3752], 20.00th=[ 4555], | 30.00th=[ 5276], 40.00th=[ 5997], 50.00th=[ 7177], 60.00th=[ 8160], | 70.00th=[10552], 80.00th=[13173], 90.00th=[17171], 95.00th=[21627], | 99.00th=[29754], 99.50th=[31065], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 768, max= 1568, per=25.80%, avg=1053.17, stdev=252.67, samples=12 iops : min= 48, max= 98, avg=65.75, stdev=15.86, samples=12 lat (msec) : 2=0.65%, 4=25.39%, 10=47.66%, 20=21.74%, 50=4.56% cpu : usr=0.15%, sys=9.93%, ctx=906, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26883: Wed Dec 25 01:20:42 2024 read: IOPS=63, BW=1014KiB/s (1038kB/s)(6256KiB/6172msec) clat (usec): min=1927, max=25281, avg=6545.09, stdev=4350.19 lat (usec): min=1927, max=25282, avg=6545.97, stdev=4350.22 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2376], 10.00th=[ 2737], 20.00th=[ 3195], | 30.00th=[ 3556], 40.00th=[ 4146], 50.00th=[ 4752], 60.00th=[ 5866], | 70.00th=[ 7504], 80.00th=[10421], 90.00th=[13042], 95.00th=[16057], | 99.00th=[19792], 99.50th=[20579], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 640, max= 1472, per=26.18%, avg=1015.75, stdev=292.97, samples=12 iops : min= 40, max= 92, avg=63.42, stdev=18.30, samples=12 write: IOPS=61, BW=977KiB/s (1001kB/s)(6032KiB/6172msec) clat (usec): min=2373, max=31951, avg=9511.24, stdev=6426.54 lat (usec): min=2373, max=31953, avg=9512.91, stdev=6426.62 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3359], 10.00th=[ 3884], 20.00th=[ 4686], | 30.00th=[ 5473], 40.00th=[ 6194], 50.00th=[ 7046], 60.00th=[ 8586], | 70.00th=[10552], 80.00th=[13173], 90.00th=[19530], 95.00th=[25560], | 99.00th=[29754], 99.50th=[31589], 99.90th=[31851], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 736, max= 1312, per=23.58%, avg=962.50, stdev=177.07, samples=12 iops : min= 46, max= 82, avg=60.08, stdev=11.06, samples=12 lat (msec) : 2=0.39%, 4=24.09%, 10=48.44%, 20=22.14%, 50=4.95% cpu : usr=0.15%, sys=9.92%, ctx=935, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26884: Wed Dec 25 01:20:42 2024 read: IOPS=61, BW=991KiB/s (1015kB/s)(6032KiB/6087msec) clat (usec): min=1670, max=40513, avg=6474.30, stdev=5255.81 lat (usec): min=1670, max=40514, avg=6475.18, stdev=5255.84 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2376], 10.00th=[ 2606], 20.00th=[ 2966], | 30.00th=[ 3326], 40.00th=[ 3818], 50.00th=[ 4424], 60.00th=[ 5145], | 70.00th=[ 6849], 80.00th=[ 9503], 90.00th=[12780], 95.00th=[16909], | 99.00th=[26346], 99.50th=[37487], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 768, max= 1408, per=25.49%, avg=989.17, stdev=204.03, samples=12 iops : min= 48, max= 88, avg=61.75, stdev=12.78, samples=12 write: IOPS=64, BW=1028KiB/s (1052kB/s)(6256KiB/6087msec) clat (usec): min=2466, max=42403, avg=9271.50, stdev=5904.15 lat (usec): min=2467, max=42405, avg=9273.17, stdev=5904.24 clat percentiles (usec): | 1.00th=[ 2802], 5.00th=[ 3621], 10.00th=[ 3916], 20.00th=[ 4752], | 30.00th=[ 5538], 40.00th=[ 6325], 50.00th=[ 7242], 60.00th=[ 8717], | 70.00th=[10683], 80.00th=[13173], 90.00th=[16450], 95.00th=[20579], | 99.00th=[30016], 99.50th=[34341], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 798, max= 1440, per=25.34%, avg=1034.50, stdev=206.39, samples=12 iops : min= 49, max= 90, avg=64.58, stdev=12.99, samples=12 lat (msec) : 2=0.52%, 4=25.65%, 10=47.79%, 20=21.87%, 50=4.17% cpu : usr=0.07%, sys=10.30%, ctx=936, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3881KiB/s (3974kB/s), 936KiB/s-1014KiB/s (958kB/s-1038kB/s), io=23.4MiB (24.5MB), run=6087-6172msec WRITE: bw=4083KiB/s (4181kB/s), 977KiB/s-1066KiB/s (1001kB/s-1092kB/s), io=24.6MiB (25.8MB), run=6087-6172msec rand-rw: (groupid=0, jobs=1): err= 0: pid=26885: Wed Dec 25 01:20:42 2024 read: IOPS=57, BW=917KiB/s (939kB/s)(5920KiB/6456msec) clat (usec): min=45, max=32122, avg=12596.96, stdev=6077.86 lat (usec): min=46, max=32123, avg=12597.92, stdev=6077.78 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 77], 10.00th=[ 163], 20.00th=[ 8291], | 30.00th=[10290], 40.00th=[11731], 50.00th=[13042], 60.00th=[14484], | 70.00th=[15926], 80.00th=[17171], 90.00th=[19792], 95.00th=[21365], | 99.00th=[27395], 99.50th=[29230], 99.90th=[32113], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 640, max= 1152, per=22.65%, avg=829.33, stdev=149.13, samples=12 iops : min= 40, max= 72, avg=51.83, stdev= 9.32, samples=12 write: IOPS=61, BW=986KiB/s (1010kB/s)(6368KiB/6456msec) clat (usec): min=539, max=23856, avg=4437.96, stdev=4093.70 lat (usec): min=539, max=23857, avg=4439.43, stdev=4093.76 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 766], 10.00th=[ 881], 20.00th=[ 1205], | 30.00th=[ 1975], 40.00th=[ 2704], 50.00th=[ 3458], 60.00th=[ 3916], | 70.00th=[ 4883], 80.00th=[ 6390], 90.00th=[ 9372], 95.00th=[12780], | 99.00th=[21103], 99.50th=[23200], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 480, max= 1408, per=23.60%, avg=909.33, stdev=279.46, samples=12 iops : min= 30, max= 88, avg=56.83, stdev=17.47, samples=12 lat (usec) : 50=0.26%, 100=2.86%, 250=1.95%, 750=1.95%, 1000=4.82% lat (msec) : 2=8.85%, 4=15.76%, 10=24.87%, 20=33.72%, 50=4.95% cpu : usr=0.20%, sys=20.90%, ctx=2607, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26886: Wed Dec 25 01:20:42 2024 read: IOPS=55, BW=884KiB/s (905kB/s)(5744KiB/6496msec) clat (usec): min=56, max=31504, avg=12695.88, stdev=5752.23 lat (usec): min=56, max=31506, avg=12696.72, stdev=5752.33 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 93], 10.00th=[ 5997], 20.00th=[ 9241], | 30.00th=[10552], 40.00th=[11731], 50.00th=[12649], 60.00th=[13960], | 70.00th=[15533], 80.00th=[17171], 90.00th=[19792], 95.00th=[21627], | 99.00th=[26608], 99.50th=[30802], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 576, max= 1088, per=22.05%, avg=807.42, stdev=188.56, samples=12 iops : min= 36, max= 68, avg=50.25, stdev=11.83, samples=12 write: IOPS=62, BW=1007KiB/s (1032kB/s)(6544KiB/6496msec) clat (usec): min=552, max=25503, avg=4565.86, stdev=4348.88 lat (usec): min=552, max=25505, avg=4567.24, stdev=4349.02 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 865], 10.00th=[ 979], 20.00th=[ 1172], | 30.00th=[ 1614], 40.00th=[ 2802], 50.00th=[ 3425], 60.00th=[ 4015], | 70.00th=[ 5014], 80.00th=[ 6390], 90.00th=[10290], 95.00th=[14353], | 99.00th=[20317], 99.50th=[21365], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 672, max= 1408, per=23.79%, avg=916.75, stdev=223.50, samples=12 iops : min= 42, max= 88, avg=57.08, stdev=13.99, samples=12 lat (usec) : 100=2.60%, 250=1.43%, 750=1.43%, 1000=4.43% lat (msec) : 2=11.33%, 4=14.58%, 10=24.22%, 20=35.03%, 50=4.95% cpu : usr=0.17%, sys=21.00%, ctx=2623, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26887: Wed Dec 25 01:20:42 2024 read: IOPS=59, BW=957KiB/s (980kB/s)(6256KiB/6540msec) clat (usec): min=44, max=28836, avg=12247.11, stdev=6110.01 lat (usec): min=45, max=28837, avg=12248.03, stdev=6110.18 clat percentiles (usec): | 1.00th=[ 54], 5.00th=[ 78], 10.00th=[ 112], 20.00th=[ 8291], | 30.00th=[10421], 40.00th=[11600], 50.00th=[12911], 60.00th=[14222], | 70.00th=[15533], 80.00th=[16712], 90.00th=[19268], 95.00th=[21365], | 99.00th=[25560], 99.50th=[28443], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 640, max= 2208, per=25.74%, avg=942.77, stdev=433.60, samples=13 iops : min= 40, max= 138, avg=58.92, stdev=27.10, samples=13 write: IOPS=57, BW=922KiB/s (944kB/s)(6032KiB/6540msec) clat (usec): min=566, max=35776, avg=4500.93, stdev=4794.63 lat (usec): min=566, max=35777, avg=4502.43, stdev=4794.70 clat percentiles (usec): | 1.00th=[ 635], 5.00th=[ 775], 10.00th=[ 922], 20.00th=[ 1139], | 30.00th=[ 1401], 40.00th=[ 2737], 50.00th=[ 3392], 60.00th=[ 3982], | 70.00th=[ 4555], 80.00th=[ 5800], 90.00th=[ 9372], 95.00th=[15008], | 99.00th=[22938], 99.50th=[29754], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 416, max= 1696, per=23.32%, avg=898.46, stdev=325.93, samples=13 iops : min= 26, max= 106, avg=56.15, stdev=20.37, samples=13 lat (usec) : 50=0.13%, 100=4.17%, 250=1.95%, 750=1.95%, 1000=4.69% lat (msec) : 2=11.07%, 4=12.11%, 10=22.40%, 20=36.33%, 50=5.21% cpu : usr=0.09%, sys=20.83%, ctx=2606, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26888: Wed Dec 25 01:20:42 2024 read: IOPS=57, BW=926KiB/s (949kB/s)(6032KiB/6511msec) clat (usec): min=43, max=30976, avg=12730.92, stdev=5757.55 lat (usec): min=43, max=30977, avg=12731.77, stdev=5757.67 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 77], 10.00th=[ 5997], 20.00th=[ 8717], | 30.00th=[10683], 40.00th=[11994], 50.00th=[13042], 60.00th=[14353], | 70.00th=[15664], 80.00th=[17433], 90.00th=[19268], 95.00th=[21627], | 99.00th=[24511], 99.50th=[25822], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 640, max= 1888, per=24.86%, avg=910.54, stdev=325.82, samples=13 iops : min= 40, max= 118, avg=56.85, stdev=20.32, samples=13 write: IOPS=60, BW=961KiB/s (984kB/s)(6256KiB/6511msec) clat (usec): min=544, max=31540, avg=4215.43, stdev=4366.35 lat (usec): min=545, max=31541, avg=4216.80, stdev=4366.35 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 832], 10.00th=[ 947], 20.00th=[ 1156], | 30.00th=[ 1369], 40.00th=[ 2180], 50.00th=[ 2999], 60.00th=[ 3720], | 70.00th=[ 4424], 80.00th=[ 6128], 90.00th=[ 9372], 95.00th=[13042], | 99.00th=[22676], 99.50th=[27657], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 608, max= 1824, per=24.66%, avg=950.00, stdev=333.14, samples=13 iops : min= 38, max= 114, avg=59.31, stdev=20.81, samples=13 lat (usec) : 50=0.39%, 100=2.86%, 250=1.17%, 500=0.13%, 750=1.30% lat (usec) : 1000=5.34% lat (msec) : 2=13.02%, 4=12.63%, 10=22.66%, 20=36.46%, 50=4.04% cpu : usr=0.23%, sys=20.70%, ctx=2698, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3662KiB/s (3750kB/s), 884KiB/s-957KiB/s (905kB/s-980kB/s), io=23.4MiB (24.5MB), run=6456-6540msec WRITE: bw=3853KiB/s (3946kB/s), 922KiB/s-1007KiB/s (944kB/s-1032kB/s), io=24.6MiB (25.8MB), run=6456-6540msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=26900: Wed Dec 25 01:20:46 2024 read: IOPS=3, BW=3141KiB/s (3216kB/s)(7168KiB/2282msec) clat (usec): min=430, max=615338, avg=282441.52, stdev=261542.22 lat (usec): min=430, max=615339, avg=282442.98, stdev=261542.38 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 433], 10.00th=[ 433], 20.00th=[ 594], | 30.00th=[ 62653], 40.00th=[ 62653], 50.00th=[304088], 60.00th=[497026], | 70.00th=[497026], 80.00th=[497026], 90.00th=[616563], 95.00th=[616563], | 99.00th=[616563], 99.50th=[616563], 99.90th=[616563], 99.95th=[616563], | 99.99th=[616563] bw ( KiB/s): min= 1868, max= 6144, per=34.25%, avg=3025.75, stdev=2080.52, samples=4 iops : min= 1, max= 6, avg= 2.50, stdev= 2.38, samples=4 write: IOPS=2, BW=2244KiB/s (2297kB/s)(5120KiB/2282msec) clat (usec): min=21811, max=99652, avg=41443.25, stdev=32683.53 lat (usec): min=21842, max=99732, avg=41500.71, stdev=32696.38 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 29], 60.00th=[ 29], | 70.00th=[ 31], 80.00th=[ 31], 90.00th=[ 101], 95.00th=[ 101], | 99.00th=[ 101], 99.50th=[ 101], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 1868, max= 8192, per=40.67%, avg=5030.00, stdev=4471.74, samples=2 iops : min= 1, max= 8, avg= 4.50, stdev= 4.95, samples=2 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=33.33%, 100=16.67%, 500=25.00%, 750=8.33% cpu : usr=0.09%, sys=31.78%, ctx=921, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26902: Wed Dec 25 01:20:46 2024 read: IOPS=1, BW=1508KiB/s (1544kB/s)(2048KiB/1358msec) clat (msec): min=89, max=400, avg=245.18, stdev=220.35 lat (msec): min=89, max=400, avg=245.18, stdev=220.35 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 90], 10.00th=[ 90], 20.00th=[ 90], | 30.00th=[ 90], 40.00th=[ 90], 50.00th=[ 90], 60.00th=[ 401], | 70.00th=[ 401], 80.00th=[ 401], 90.00th=[ 401], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 2048, max= 2048, per=23.18%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=7, BW=7541KiB/s (7721kB/s)(10.0MiB/1358msec) clat (msec): min=26, max=449, avg=84.90, stdev=129.93 lat (msec): min=26, max=449, avg=85.03, stdev=129.93 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 31], | 30.00th=[ 33], 40.00th=[ 34], 50.00th=[ 35], 60.00th=[ 37], | 70.00th=[ 81], 80.00th=[ 81], 90.00th=[ 86], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 6144, max=10240, per=66.23%, avg=8192.00, stdev=2896.31, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (msec) : 50=58.33%, 100=25.00%, 500=16.67% cpu : usr=0.07%, sys=25.28%, ctx=325, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26905: Wed Dec 25 01:20:46 2024 read: IOPS=2, BW=2651KiB/s (2714kB/s)(6144KiB/2318msec) clat (usec): min=372, max=554340, avg=93927.67, stdev=225570.51 lat (usec): min=373, max=554342, avg=93928.74, stdev=225570.75 clat percentiles (usec): | 1.00th=[ 375], 5.00th=[ 375], 10.00th=[ 375], 20.00th=[ 529], | 30.00th=[ 529], 40.00th=[ 586], 50.00th=[ 586], 60.00th=[ 660], | 70.00th=[ 7111], 80.00th=[ 7111], 90.00th=[557843], 95.00th=[557843], | 99.00th=[557843], 99.50th=[557843], 99.90th=[557843], 99.95th=[557843], | 99.99th=[557843] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2651KiB/s (2714kB/s)(6144KiB/2318msec) clat (msec): min=23, max=1149, avg=277.24, stdev=445.74 lat (msec): min=23, max=1149, avg=277.30, stdev=445.75 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 30], | 30.00th=[ 30], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 58], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 1892, max= 6144, per=32.69%, avg=4044.00, stdev=2126.48, samples=3 iops : min= 1, max= 6, avg= 3.67, stdev= 2.52, samples=3 lat (usec) : 500=8.33%, 750=25.00% lat (msec) : 10=8.33%, 50=25.00%, 100=8.33%, 500=8.33%, 750=8.33% cpu : usr=0.13%, sys=11.74%, ctx=355, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26906: Wed Dec 25 01:20:46 2024 read: IOPS=3, BW=3174KiB/s (3250kB/s)(5120KiB/1613msec) clat (usec): min=304, max=693, avg=458.13, stdev=175.98 lat (usec): min=305, max=695, avg=459.14, stdev=176.70 clat percentiles (usec): | 1.00th=[ 306], 5.00th=[ 306], 10.00th=[ 306], 20.00th=[ 306], | 30.00th=[ 326], 40.00th=[ 326], 50.00th=[ 367], 60.00th=[ 367], | 70.00th=[ 594], 80.00th=[ 594], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 8192, max= 8192, per=92.72%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4444KiB/s (4551kB/s)(7168KiB/1613msec) clat (msec): min=22, max=1189, avg=216.73, stdev=429.69 lat (msec): min=22, max=1189, avg=216.80, stdev=429.70 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 31], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 53], 60.00th=[ 90], | 70.00th=[ 90], 80.00th=[ 94], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 2043, max= 8192, per=41.37%, avg=5117.50, stdev=4348.00, samples=2 iops : min= 1, max= 8, avg= 4.50, stdev= 4.95, samples=2 lat (usec) : 500=25.00%, 750=16.67% lat (msec) : 50=25.00%, 100=25.00% cpu : usr=0.00%, sys=10.17%, ctx=71, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8835KiB/s (9047kB/s), 1508KiB/s-3174KiB/s (1544kB/s-3250kB/s), io=20.0MiB (20.0MB), run=1358-2318msec WRITE: bw=12.1MiB/s (12.7MB/s), 2244KiB/s-7541KiB/s (2297kB/s-7721kB/s), io=28.0MiB (29.4MB), run=1358-2318msec rand-rw: (groupid=0, jobs=1): err= 0: pid=26899: Wed Dec 25 01:20:46 2024 read: IOPS=15, BW=15.1MiB/s (15.8MB/s)(7168KiB/465msec) clat (usec): min=16192, max=76433, avg=37310.03, stdev=21214.26 lat (usec): min=16194, max=76434, avg=37311.64, stdev=21213.84 clat percentiles (usec): | 1.00th=[16188], 5.00th=[16188], 10.00th=[16188], 20.00th=[19006], | 30.00th=[21627], 40.00th=[21627], 50.00th=[39584], 60.00th=[39584], | 70.00th=[39584], 80.00th=[48497], 90.00th=[76022], 95.00th=[76022], | 99.00th=[76022], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] write: IOPS=10, BW=10.8MiB/s (11.3MB/s)(5120KiB/465msec) clat (usec): min=17625, max=47853, avg=27682.73, stdev=13049.93 lat (usec): min=17701, max=47939, avg=27756.22, stdev=13047.10 clat percentiles (usec): | 1.00th=[17695], 5.00th=[17695], 10.00th=[17695], 20.00th=[17695], | 30.00th=[19530], 40.00th=[19530], 50.00th=[19530], 60.00th=[19530], | 70.00th=[33817], 80.00th=[33817], 90.00th=[47973], 95.00th=[47973], | 99.00th=[47973], 99.50th=[47973], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] lat (msec) : 20=41.67%, 50=50.00%, 100=8.33% cpu : usr=0.43%, sys=24.57%, ctx=59, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26901: Wed Dec 25 01:20:46 2024 read: IOPS=0, BW=867KiB/s (888kB/s)(2048KiB/2361msec) clat (msec): min=35, max=1088, avg=562.27, stdev=744.83 lat (msec): min=35, max=1088, avg=562.27, stdev=744.83 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 1083], | 70.00th=[ 1083], 80.00th=[ 1083], 90.00th=[ 1083], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 1858, max= 2048, per=22.52%, avg=1953.00, stdev=134.35, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4337KiB/s (4441kB/s)(10.0MiB/2361msec) clat (msec): min=16, max=516, avg=120.63, stdev=144.29 lat (msec): min=17, max=516, avg=120.71, stdev=144.29 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 55], | 30.00th=[ 59], 40.00th=[ 65], 50.00th=[ 74], 60.00th=[ 77], | 70.00th=[ 96], 80.00th=[ 96], 90.00th=[ 167], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 1858, max=10240, per=48.51%, avg=5891.00, stdev=4199.93, samples=3 iops : min= 1, max= 10, avg= 5.33, stdev= 4.51, samples=3 lat (msec) : 20=8.33%, 50=8.33%, 100=58.33%, 250=8.33%, 750=8.33% cpu : usr=0.08%, sys=8.31%, ctx=101, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26903: Wed Dec 25 01:20:46 2024 read: IOPS=7, BW=7670KiB/s (7855kB/s)(6144KiB/801msec) clat (msec): min=16, max=380, avg=90.13, stdev=143.49 lat (msec): min=16, max=380, avg=90.13, stdev=143.49 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 21], | 30.00th=[ 21], 40.00th=[ 24], 50.00th=[ 24], 60.00th=[ 33], | 70.00th=[ 67], 80.00th=[ 67], 90.00th=[ 380], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min=10219, max=10219, per=100.00%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7670KiB/s (7855kB/s)(6144KiB/801msec) clat (usec): min=16618, max=62635, avg=38410.52, stdev=16778.07 lat (usec): min=16684, max=62721, avg=38478.76, stdev=16786.03 clat percentiles (usec): | 1.00th=[16581], 5.00th=[16581], 10.00th=[16581], 20.00th=[27395], | 30.00th=[27395], 40.00th=[29754], 50.00th=[29754], 60.00th=[46400], | 70.00th=[47449], 80.00th=[47449], 90.00th=[62653], 95.00th=[62653], | 99.00th=[62653], 99.50th=[62653], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min=12263, max=12263, per=100.00%, avg=12263.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=58.33%, 100=16.67%, 500=8.33% cpu : usr=0.00%, sys=12.00%, ctx=51, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26904: Wed Dec 25 01:20:46 2024 read: IOPS=3, BW=3938KiB/s (4033kB/s)(5120KiB/1300msec) clat (msec): min=13, max=865, avg=198.02, stdev=373.79 lat (msec): min=13, max=865, avg=198.02, stdev=373.79 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 16], 50.00th=[ 21], 60.00th=[ 21], | 70.00th=[ 75], 80.00th=[ 75], 90.00th=[ 869], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 8192, max= 8192, per=94.44%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5514KiB/s (5646kB/s)(7168KiB/1300msec) clat (usec): min=16136, max=75393, avg=34232.06, stdev=19133.21 lat (usec): min=16204, max=75491, avg=34325.37, stdev=19140.94 clat percentiles (usec): | 1.00th=[16188], 5.00th=[16188], 10.00th=[16188], 20.00th=[23725], | 30.00th=[26870], 40.00th=[26870], 50.00th=[31589], 60.00th=[32900], | 70.00th=[32900], 80.00th=[32900], 90.00th=[74974], 95.00th=[74974], | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min=14336, max=14336, per=100.00%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=50.00%, 100=16.67%, 1000=8.33% cpu : usr=0.00%, sys=7.16%, ctx=152, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8674KiB/s (8882kB/s), 867KiB/s-15.1MiB/s (888kB/s-15.8MB/s), io=20.0MiB (20.0MB), run=465-2361msec WRITE: bw=11.9MiB/s (12.4MB/s), 4337KiB/s-10.8MiB/s (4441kB/s-11.3MB/s), io=28.0MiB (29.4MB), run=465-2361msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=26914: Wed Dec 25 01:20:49 2024 read: IOPS=1, BW=6575KiB/s (6732kB/s)(12.0MiB/1869msec) clat (msec): min=297, max=1167, avg=602.43, stdev=489.64 lat (msec): min=297, max=1167, avg=602.43, stdev=489.64 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 296], 10.00th=[ 296], 20.00th=[ 296], | 30.00th=[ 296], 40.00th=[ 342], 50.00th=[ 342], 60.00th=[ 342], | 70.00th=[ 1167], 80.00th=[ 1167], 90.00th=[ 1167], 95.00th=[ 1167], | 99.00th=[ 1167], 99.50th=[ 1167], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 8192, max= 8192, per=98.81%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.05%, sys=34.64%, ctx=681, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26916: Wed Dec 25 01:20:49 2024 read: IOPS=0, BW=2073KiB/s (2123kB/s)(4096KiB/1976msec) clat (nsec): min=296773k, max=296773k, avg=296772522.00, stdev= 0.00 lat (nsec): min=296774k, max=296774k, avg=296773656.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 296], 10.00th=[ 296], 20.00th=[ 296], | 30.00th=[ 296], 40.00th=[ 296], 50.00th=[ 296], 60.00th=[ 296], | 70.00th=[ 296], 80.00th=[ 296], 90.00th=[ 296], 95.00th=[ 296], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 8192, max= 8192, per=98.81%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4146KiB/s (4245kB/s)(8192KiB/1976msec) clat (msec): min=295, max=1349, avg=822.19, stdev=745.22 lat (msec): min=295, max=1349, avg=822.49, stdev=745.17 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 296], 10.00th=[ 296], 20.00th=[ 296], | 30.00th=[ 296], 40.00th=[ 296], 50.00th=[ 296], 60.00th=[ 1351], | 70.00th=[ 1351], 80.00th=[ 1351], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 5490, max= 5490, per=33.11%, avg=5490.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=22.84%, ctx=115, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26918: Wed Dec 25 01:20:49 2024 write: IOPS=1, BW=6582KiB/s (6740kB/s)(12.0MiB/1867msec) clat (msec): min=157, max=1267, avg=614.78, stdev=579.99 lat (msec): min=157, max=1267, avg=615.09, stdev=580.03 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 422], 50.00th=[ 422], 60.00th=[ 422], | 70.00th=[ 1267], 80.00th=[ 1267], 90.00th=[ 1267], 95.00th=[ 1267], | 99.00th=[ 1267], 99.50th=[ 1267], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 5603, max= 8192, per=41.60%, avg=6897.50, stdev=1830.70, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=15.76%, ctx=58, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26920: Wed Dec 25 01:20:49 2024 write: IOPS=4, BW=17.9MiB/s (18.8MB/s)(12.0MiB/670msec) clat (msec): min=155, max=233, avg=187.83, stdev=40.74 lat (msec): min=155, max=234, avg=188.17, stdev=40.79 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 176], 50.00th=[ 176], 60.00th=[ 176], | 70.00th=[ 234], 80.00th=[ 234], 90.00th=[ 234], 95.00th=[ 234], | 99.00th=[ 234], 99.50th=[ 234], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min=16384, max=16384, per=98.81%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=49.03%, ctx=72, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8291KiB/s (8490kB/s), 2073KiB/s-6575KiB/s (2123kB/s-6732kB/s), io=16.0MiB (16.8MB), run=1869-1976msec WRITE: bw=16.2MiB/s (16.0MB/s), 4146KiB/s-17.9MiB/s (4245kB/s-18.8MB/s), io=32.0MiB (33.6MB), run=670-1976msec rand-rw: (groupid=0, jobs=1): err= 0: pid=26915: Wed Dec 25 01:20:49 2024 read: IOPS=1, BW=6357KiB/s (6510kB/s)(12.0MiB/1933msec) clat (msec): min=113, max=1269, avg=608.99, stdev=595.06 lat (msec): min=113, max=1269, avg=608.99, stdev=595.06 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 114], 10.00th=[ 114], 20.00th=[ 114], | 30.00th=[ 114], 40.00th=[ 443], 50.00th=[ 443], 60.00th=[ 443], | 70.00th=[ 1267], 80.00th=[ 1267], 90.00th=[ 1267], 95.00th=[ 1267], | 99.00th=[ 1267], 99.50th=[ 1267], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 5728, max= 5728, per=72.10%, avg=5728.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=5.38%, ctx=52, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26917: Wed Dec 25 01:20:49 2024 read: IOPS=0, BW=1986KiB/s (2034kB/s)(4096KiB/2062msec) clat (nsec): min=401925k, max=401925k, avg=401924681.00, stdev= 0.00 lat (nsec): min=401927k, max=401927k, avg=401927366.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 401], 10.00th=[ 401], 20.00th=[ 401], | 30.00th=[ 401], 40.00th=[ 401], 50.00th=[ 401], 60.00th=[ 401], | 70.00th=[ 401], 80.00th=[ 401], 90.00th=[ 401], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 8094, max= 8094, per=100.00%, avg=8094.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3973KiB/s (4068kB/s)(8192KiB/2062msec) clat (msec): min=232, max=1309, avg=771.14, stdev=761.75 lat (msec): min=232, max=1310, avg=771.43, stdev=761.76 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 232], 10.00th=[ 232], 20.00th=[ 232], | 30.00th=[ 232], 40.00th=[ 232], 50.00th=[ 232], 60.00th=[ 1318], | 70.00th=[ 1318], 80.00th=[ 1318], 90.00th=[ 1318], 95.00th=[ 1318], | 99.00th=[ 1318], 99.50th=[ 1318], 99.90th=[ 1318], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 8094, max= 8094, per=50.93%, avg=8094.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=7.13%, ctx=199, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26919: Wed Dec 25 01:20:49 2024 write: IOPS=6, BW=24.1MiB/s (25.3MB/s)(12.0MiB/497msec) clat (msec): min=95, max=204, avg=155.91, stdev=55.10 lat (msec): min=96, max=204, avg=156.24, stdev=55.01 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 96], 10.00th=[ 96], 20.00th=[ 96], | 30.00th=[ 96], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 169], | 70.00th=[ 205], 80.00th=[ 205], 90.00th=[ 205], 95.00th=[ 205], | 99.00th=[ 205], 99.50th=[ 205], 99.90th=[ 205], 99.95th=[ 205], | 99.99th=[ 205] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=22.58%, ctx=70, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26921: Wed Dec 25 01:20:49 2024 write: IOPS=1, BW=7371KiB/s (7548kB/s)(12.0MiB/1667msec) clat (msec): min=104, max=1144, avg=517.79, stdev=551.76 lat (msec): min=104, max=1144, avg=518.26, stdev=552.01 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 105], 10.00th=[ 105], 20.00th=[ 105], | 30.00th=[ 105], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 305], | 70.00th=[ 1150], 80.00th=[ 1150], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 8175, max= 8175, per=51.44%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.06%, sys=8.10%, ctx=93, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7946KiB/s (8136kB/s), 1986KiB/s-6357KiB/s (2034kB/s-6510kB/s), io=16.0MiB (16.8MB), run=1933-2062msec WRITE: bw=15.5MiB/s (16.3MB/s), 3973KiB/s-24.1MiB/s (4068kB/s-25.3MB/s), io=32.0MiB (33.6MB), run=497-2062msec