== sanity test 398b: DIO and buffer IO race ============== 02:00:42 (1721887242) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.1782 s, 42.7 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22610: Thu Jul 25 02:00:57 2024 read: IOPS=119, BW=479KiB/s (490kB/s)(5992KiB/12513msec) clat (usec): min=45, max=180998, avg=3896.17, stdev=5766.12 lat (usec): min=46, max=180999, avg=3896.85, stdev=5766.21 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 60], 10.00th=[ 70], 20.00th=[ 100], | 30.00th=[ 2212], 40.00th=[ 2606], 50.00th=[ 3064], 60.00th=[ 3687], | 70.00th=[ 4686], 80.00th=[ 6128], 90.00th=[ 7898], 95.00th=[ 9503], | 99.00th=[ 16581], 99.50th=[ 20841], 99.90th=[ 35914], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 168, max= 656, per=24.92%, avg=477.92, stdev=109.25, samples=25 iops : min= 42, max= 164, avg=119.40, stdev=27.35, samples=25 write: IOPS=125, BW=503KiB/s (515kB/s)(6296KiB/12513msec) clat (usec): min=453, max=51107, avg=4153.82, stdev=4017.92 lat (usec): min=453, max=51108, avg=4155.52, stdev=4019.44 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 570], 20.00th=[ 668], | 30.00th=[ 865], 40.00th=[ 1909], 50.00th=[ 3392], 60.00th=[ 4293], | 70.00th=[ 5473], 80.00th=[ 7046], 90.00th=[ 9372], 95.00th=[11600], | 99.00th=[16712], 99.50th=[18744], 99.90th=[32113], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 184, max= 688, per=24.97%, avg=501.64, stdev=108.40, samples=25 iops : min= 46, max= 172, avg=125.32, stdev=27.17, samples=25 lat (usec) : 50=0.36%, 100=9.37%, 250=1.14%, 500=0.72%, 750=12.34% lat (usec) : 1000=4.20% lat (msec) : 2=5.05%, 4=27.08%, 10=33.63%, 20=5.73%, 50=0.33% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.28%, sys=24.98%, ctx=6096, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22612: Thu Jul 25 02:00:57 2024 read: IOPS=120, BW=482KiB/s (494kB/s)(5940KiB/12317msec) clat (usec): min=48, max=25338, avg=3654.33, stdev=3181.48 lat (usec): min=48, max=25339, avg=3655.00, stdev=3181.45 clat percentiles (usec): | 1.00th=[ 54], 5.00th=[ 60], 10.00th=[ 69], 20.00th=[ 114], | 30.00th=[ 2245], 40.00th=[ 2638], 50.00th=[ 3064], 60.00th=[ 3621], | 70.00th=[ 4490], 80.00th=[ 5866], 90.00th=[ 7635], 95.00th=[ 9110], | 99.00th=[14484], 99.50th=[19530], 99.90th=[24773], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 184, max= 688, per=25.29%, avg=485.00, stdev=99.70, samples=24 iops : min= 46, max= 172, avg=121.04, stdev=25.01, samples=24 write: IOPS=128, BW=515KiB/s (528kB/s)(6348KiB/12317msec) clat (usec): min=426, max=201050, avg=4254.65, stdev=6333.60 lat (usec): min=426, max=201051, avg=4255.98, stdev=6333.84 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 562], 20.00th=[ 644], | 30.00th=[ 881], 40.00th=[ 1942], 50.00th=[ 3326], 60.00th=[ 4146], | 70.00th=[ 5276], 80.00th=[ 6980], 90.00th=[ 9634], 95.00th=[ 12125], | 99.00th=[ 17433], 99.50th=[ 19530], 99.90th=[ 33817], 99.95th=[200279], | 99.99th=[200279] bw ( KiB/s): min= 256, max= 752, per=25.54%, avg=513.17, stdev=98.38, samples=24 iops : min= 64, max= 188, avg=128.12, stdev=24.60, samples=24 lat (usec) : 50=0.33%, 100=8.89%, 250=1.40%, 500=0.94%, 750=13.35% lat (usec) : 1000=2.64% lat (msec) : 2=5.50%, 4=28.52%, 10=32.45%, 20=5.57%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=25.15%, ctx=5947, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22613: Thu Jul 25 02:00:57 2024 read: IOPS=121, BW=486KiB/s (497kB/s)(6028KiB/12415msec) clat (usec): min=47, max=27667, avg=3635.24, stdev=3205.89 lat (usec): min=47, max=27667, avg=3635.96, stdev=3206.06 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 61], 10.00th=[ 71], 20.00th=[ 104], | 30.00th=[ 2180], 40.00th=[ 2606], 50.00th=[ 3064], 60.00th=[ 3687], | 70.00th=[ 4555], 80.00th=[ 5866], 90.00th=[ 7635], 95.00th=[ 8979], | 99.00th=[13960], 99.50th=[17957], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 184, max= 696, per=25.16%, avg=482.54, stdev=107.48, samples=24 iops : min= 46, max= 174, avg=120.54, stdev=26.93, samples=24 write: IOPS=126, BW=504KiB/s (516kB/s)(6260KiB/12415msec) clat (usec): min=434, max=174304, avg=4346.35, stdev=5965.45 lat (usec): min=434, max=174305, avg=4347.10, stdev=5965.44 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 537], 10.00th=[ 570], 20.00th=[ 652], | 30.00th=[ 865], 40.00th=[ 2147], 50.00th=[ 3523], 60.00th=[ 4424], | 70.00th=[ 5342], 80.00th=[ 7046], 90.00th=[ 9372], 95.00th=[ 12125], | 99.00th=[ 17433], 99.50th=[ 21365], 99.90th=[ 51119], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 240, max= 688, per=24.78%, avg=497.87, stdev=101.53, samples=24 iops : min= 60, max= 172, avg=124.38, stdev=25.47, samples=24 lat (usec) : 50=0.26%, 100=9.44%, 250=1.43%, 500=0.78%, 750=12.83% lat (usec) : 1000=2.67% lat (msec) : 2=5.60%, 4=26.92%, 10=33.98%, 20=5.60%, 50=0.42% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.17%, sys=24.91%, ctx=6080, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22614: Thu Jul 25 02:00:57 2024 read: IOPS=122, BW=490KiB/s (501kB/s)(6052KiB/12362msec) clat (usec): min=46, max=177929, avg=3714.72, stdev=5593.57 lat (usec): min=46, max=177930, avg=3715.64, stdev=5593.62 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 60], 10.00th=[ 69], 20.00th=[ 100], | 30.00th=[ 2245], 40.00th=[ 2638], 50.00th=[ 3097], 60.00th=[ 3720], | 70.00th=[ 4490], 80.00th=[ 5735], 90.00th=[ 7504], 95.00th=[ 8717], | 99.00th=[ 13173], 99.50th=[ 16188], 99.90th=[ 58459], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 248, max= 688, per=25.40%, avg=487.17, stdev=94.21, samples=24 iops : min= 62, max= 172, avg=121.67, stdev=23.62, samples=24 write: IOPS=126, BW=504KiB/s (517kB/s)(6236KiB/12362msec) clat (usec): min=444, max=36323, avg=4236.48, stdev=4167.66 lat (usec): min=444, max=36324, avg=4237.48, stdev=4167.42 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 570], 20.00th=[ 660], | 30.00th=[ 947], 40.00th=[ 2180], 50.00th=[ 3359], 60.00th=[ 4293], | 70.00th=[ 5276], 80.00th=[ 6980], 90.00th=[ 9503], 95.00th=[11994], | 99.00th=[19006], 99.50th=[21627], 99.90th=[33817], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 304, max= 656, per=25.01%, avg=502.42, stdev=88.27, samples=24 iops : min= 76, max= 164, avg=125.50, stdev=22.14, samples=24 lat (usec) : 50=0.33%, 100=9.57%, 250=1.07%, 500=1.04%, 750=12.08% lat (usec) : 1000=2.67% lat (msec) : 2=5.47%, 4=28.58%, 10=33.53%, 20=5.21%, 50=0.39% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.12%, sys=24.63%, ctx=6026, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1919KiB/s (1965kB/s), 479KiB/s-490KiB/s (490kB/s-501kB/s), io=23.4MiB (24.6MB), run=12317-12513msec WRITE: bw=2009KiB/s (2057kB/s), 503KiB/s-515KiB/s (515kB/s-528kB/s), io=24.6MiB (25.7MB), run=12317-12513msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22607: Thu Jul 25 02:01:24 2024 read: IOPS=38, BW=152KiB/s (156kB/s)(5992KiB/39388msec) clat (usec): min=1644, max=210267, avg=3714.26, stdev=6044.96 lat (usec): min=1644, max=210268, avg=3715.11, stdev=6045.06 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1860], 10.00th=[ 1942], 20.00th=[ 2089], | 30.00th=[ 2212], 40.00th=[ 2376], 50.00th=[ 2540], 60.00th=[ 2769], | 70.00th=[ 3097], 80.00th=[ 4113], 90.00th=[ 7111], 95.00th=[ 9372], | 99.00th=[ 13566], 99.50th=[ 17171], 99.90th=[ 33817], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 56, max= 272, per=25.10%, avg=152.08, stdev=56.89, samples=78 iops : min= 14, max= 68, avg=37.92, stdev=14.25, samples=78 write: IOPS=39, BW=160KiB/s (164kB/s)(6296KiB/39388msec) clat (usec): min=11585, max=56932, avg=21399.48, stdev=7096.06 lat (usec): min=11585, max=56933, avg=21400.39, stdev=7096.07 clat percentiles (usec): | 1.00th=[12518], 5.00th=[13435], 10.00th=[14091], 20.00th=[15139], | 30.00th=[16581], 40.00th=[18744], 50.00th=[20579], 60.00th=[22152], | 70.00th=[23200], 80.00th=[25035], 90.00th=[31065], 95.00th=[36439], | 99.00th=[44303], 99.50th=[51119], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 40, max= 216, per=25.13%, avg=159.56, stdev=41.42, samples=78 iops : min= 10, max= 54, avg=39.79, stdev=10.35, samples=78 lat (msec) : 2=6.84%, 4=32.00%, 10=7.98%, 20=25.20%, 50=27.70% lat (msec) : 100=0.26%, 250=0.03% cpu : usr=0.09%, sys=5.56%, ctx=3514, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22608: Thu Jul 25 02:01:24 2024 read: IOPS=37, BW=150KiB/s (154kB/s)(5940KiB/39576msec) clat (usec): min=1634, max=182488, avg=3747.75, stdev=5682.58 lat (usec): min=1635, max=182491, avg=3748.57, stdev=5682.60 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1860], 10.00th=[ 1958], 20.00th=[ 2089], | 30.00th=[ 2212], 40.00th=[ 2343], 50.00th=[ 2507], 60.00th=[ 2769], | 70.00th=[ 3163], 80.00th=[ 4113], 90.00th=[ 7046], 95.00th=[ 9634], | 99.00th=[ 16581], 99.50th=[ 21103], 99.90th=[ 55313], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 16, max= 304, per=24.75%, avg=150.00, stdev=60.63, samples=79 iops : min= 4, max= 76, avg=37.44, stdev=15.16, samples=79 write: IOPS=40, BW=160KiB/s (164kB/s)(6348KiB/39576msec) clat (usec): min=11661, max=73841, avg=21342.74, stdev=7048.57 lat (usec): min=11662, max=73842, avg=21343.61, stdev=7048.53 clat percentiles (usec): | 1.00th=[12518], 5.00th=[13304], 10.00th=[13960], 20.00th=[15139], | 30.00th=[16581], 40.00th=[18482], 50.00th=[20579], 60.00th=[21890], | 70.00th=[23462], 80.00th=[25297], 90.00th=[31065], 95.00th=[34866], | 99.00th=[44827], 99.50th=[49546], 99.90th=[55313], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 40, max= 224, per=25.22%, avg=160.13, stdev=41.04, samples=79 iops : min= 10, max= 56, avg=39.97, stdev=10.27, samples=79 lat (msec) : 2=6.54%, 4=31.93%, 10=7.81%, 20=25.98%, 50=27.44% lat (msec) : 100=0.26%, 250=0.03% cpu : usr=0.08%, sys=5.59%, ctx=3526, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22609: Thu Jul 25 02:01:24 2024 read: IOPS=38, BW=152KiB/s (156kB/s)(6028KiB/39584msec) clat (usec): min=1593, max=40353, avg=3718.53, stdev=3018.34 lat (usec): min=1594, max=40354, avg=3719.42, stdev=3018.27 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1860], 10.00th=[ 1942], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2376], 50.00th=[ 2606], 60.00th=[ 2868], | 70.00th=[ 3294], 80.00th=[ 4752], 90.00th=[ 7177], 95.00th=[ 9503], | 99.00th=[14746], 99.50th=[19792], 99.90th=[30016], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 48, max= 280, per=25.15%, avg=152.39, stdev=52.31, samples=79 iops : min= 12, max= 70, avg=38.09, stdev=13.07, samples=79 write: IOPS=39, BW=158KiB/s (162kB/s)(6260KiB/39584msec) clat (msec): min=11, max=201, avg=21.62, stdev= 8.42 lat (msec): min=11, max=201, avg=21.62, stdev= 8.42 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 21], 60.00th=[ 22], | 70.00th=[ 24], 80.00th=[ 26], 90.00th=[ 32], 95.00th=[ 36], | 99.00th=[ 46], 99.50th=[ 50], 99.90th=[ 64], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 48, max= 208, per=24.88%, avg=157.96, stdev=40.44, samples=79 iops : min= 12, max= 52, avg=39.48, stdev=10.13, samples=79 lat (msec) : 2=6.25%, 4=31.41%, 10=9.47%, 20=25.13%, 50=27.54% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.10%, sys=5.54%, ctx=3590, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22611: Thu Jul 25 02:01:24 2024 read: IOPS=38, BW=154KiB/s (157kB/s)(6052KiB/39357msec) clat (usec): min=1568, max=55247, avg=3669.95, stdev=3426.11 lat (usec): min=1568, max=55248, avg=3670.72, stdev=3426.09 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1860], 10.00th=[ 1942], 20.00th=[ 2089], | 30.00th=[ 2212], 40.00th=[ 2376], 50.00th=[ 2540], 60.00th=[ 2802], | 70.00th=[ 3130], 80.00th=[ 3949], 90.00th=[ 6849], 95.00th=[ 9765], | 99.00th=[15926], 99.50th=[23725], 99.90th=[33424], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 16, max= 304, per=25.24%, avg=152.95, stdev=60.61, samples=78 iops : min= 4, max= 76, avg=38.18, stdev=15.13, samples=78 write: IOPS=39, BW=158KiB/s (162kB/s)(6236KiB/39357msec) clat (msec): min=11, max=202, avg=21.59, stdev= 8.43 lat (msec): min=11, max=202, avg=21.59, stdev= 8.43 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 21], 60.00th=[ 22], | 70.00th=[ 24], 80.00th=[ 26], 90.00th=[ 32], 95.00th=[ 37], | 99.00th=[ 46], 99.50th=[ 48], 99.90th=[ 54], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 40, max= 200, per=24.89%, avg=158.08, stdev=41.43, samples=78 iops : min= 10, max= 50, avg=39.46, stdev=10.37, samples=78 lat (msec) : 2=7.00%, 4=32.55%, 10=7.45%, 20=25.36%, 50=27.44% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.10%, sys=5.58%, ctx=3558, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=607KiB/s (621kB/s), 150KiB/s-154KiB/s (154kB/s-157kB/s), io=23.4MiB (24.6MB), run=39357-39584msec WRITE: bw=635KiB/s (650kB/s), 158KiB/s-160KiB/s (162kB/s-164kB/s), io=24.6MiB (25.7MB), run=39357-39584msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22657: Thu Jul 25 02:01:30 2024 read: IOPS=82, BW=1317KiB/s (1348kB/s)(5920KiB/4496msec) clat (usec): min=52, max=58372, avg=9103.07, stdev=6695.42 lat (usec): min=53, max=58373, avg=9105.60, stdev=6697.23 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 64], 10.00th=[ 68], 20.00th=[ 139], | 30.00th=[ 8094], 40.00th=[ 9110], 50.00th=[ 9896], 60.00th=[10683], | 70.00th=[11338], 80.00th=[12518], 90.00th=[14222], 95.00th=[15533], | 99.00th=[44827], 99.50th=[52167], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 736, max= 1600, per=25.35%, avg=1283.62, stdev=253.17, samples=8 iops : min= 46, max= 100, avg=80.12, stdev=15.77, samples=8 write: IOPS=88, BW=1416KiB/s (1450kB/s)(6368KiB/4496msec) clat (usec): min=664, max=24270, avg=2774.58, stdev=3085.16 lat (usec): min=665, max=24271, avg=2775.54, stdev=3085.16 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 725], 10.00th=[ 766], 20.00th=[ 807], | 30.00th=[ 857], 40.00th=[ 938], 50.00th=[ 1139], 60.00th=[ 2474], | 70.00th=[ 3359], 80.00th=[ 4621], 90.00th=[ 6194], 95.00th=[ 7898], | 99.00th=[17957], 99.50th=[22152], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 896, max= 2144, per=26.20%, avg=1395.62, stdev=383.21, samples=8 iops : min= 56, max= 134, avg=87.12, stdev=24.06, samples=8 lat (usec) : 100=7.94%, 250=2.21%, 750=4.30%, 1000=19.14% lat (msec) : 2=6.12%, 4=10.42%, 10=25.00%, 20=23.96%, 50=0.52% lat (msec) : 100=0.39% cpu : usr=0.07%, sys=23.36%, ctx=2122, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22659: Thu Jul 25 02:01:30 2024 read: IOPS=78, BW=1262KiB/s (1292kB/s)(5744KiB/4552msec) clat (usec): min=49, max=25732, avg=8454.85, stdev=5193.92 lat (usec): min=49, max=25732, avg=8455.44, stdev=5193.93 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 92], | 30.00th=[ 7898], 40.00th=[ 8979], 50.00th=[ 9896], 60.00th=[10552], | 70.00th=[11338], 80.00th=[12387], 90.00th=[13698], 95.00th=[15533], | 99.00th=[18482], 99.50th=[19530], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 480, max= 1600, per=24.58%, avg=1244.44, stdev=326.12, samples=9 iops : min= 30, max= 100, avg=77.78, stdev=20.38, samples=9 write: IOPS=89, BW=1438KiB/s (1472kB/s)(6544KiB/4552msec) clat (usec): min=632, max=85666, avg=3469.14, stdev=5663.35 lat (usec): min=633, max=85668, avg=3470.23, stdev=5663.34 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 717], 10.00th=[ 758], 20.00th=[ 807], | 30.00th=[ 857], 40.00th=[ 1045], 50.00th=[ 2114], 60.00th=[ 2868], | 70.00th=[ 3818], 80.00th=[ 5145], 90.00th=[ 6849], 95.00th=[ 9110], | 99.00th=[20317], 99.50th=[26608], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 800, max= 1888, per=26.36%, avg=1404.44, stdev=340.33, samples=9 iops : min= 50, max= 118, avg=87.78, stdev=21.27, samples=9 lat (usec) : 50=0.13%, 100=9.64%, 250=1.04%, 500=0.13%, 750=4.43% lat (usec) : 1000=16.15% lat (msec) : 2=5.99%, 4=12.50%, 10=25.65%, 20=23.57%, 50=0.65% lat (msec) : 100=0.13% cpu : usr=0.15%, sys=21.49%, ctx=1990, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22660: Thu Jul 25 02:01:30 2024 read: IOPS=82, BW=1323KiB/s (1354kB/s)(6256KiB/4730msec) clat (usec): min=50, max=40913, avg=8842.54, stdev=5691.59 lat (usec): min=50, max=40913, avg=8843.17, stdev=5691.63 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 64], 10.00th=[ 73], 20.00th=[ 97], | 30.00th=[ 8029], 40.00th=[ 8979], 50.00th=[ 9896], 60.00th=[10814], | 70.00th=[11731], 80.00th=[12649], 90.00th=[14484], 95.00th=[16450], | 99.00th=[22152], 99.50th=[28967], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 576, max= 1856, per=25.84%, avg=1308.11, stdev=340.94, samples=9 iops : min= 36, max= 116, avg=81.67, stdev=21.27, samples=9 write: IOPS=79, BW=1275KiB/s (1306kB/s)(6032KiB/4730msec) clat (usec): min=650, max=68290, avg=3121.79, stdev=5003.51 lat (usec): min=651, max=68292, avg=3122.72, stdev=5003.59 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 717], 10.00th=[ 742], 20.00th=[ 791], | 30.00th=[ 840], 40.00th=[ 889], 50.00th=[ 1303], 60.00th=[ 2507], | 70.00th=[ 3228], 80.00th=[ 4817], 90.00th=[ 6521], 95.00th=[ 8160], | 99.00th=[24773], 99.50th=[38536], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 704, max= 1568, per=23.09%, avg=1229.89, stdev=317.18, samples=9 iops : min= 44, max= 98, avg=76.78, stdev=19.81, samples=9 lat (usec) : 100=10.42%, 250=1.04%, 500=0.13%, 750=5.60%, 1000=16.54% lat (msec) : 2=4.56%, 4=10.42%, 10=24.61%, 20=25.26%, 50=1.30% lat (msec) : 100=0.13% cpu : usr=0.13%, sys=21.17%, ctx=2053, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22661: Thu Jul 25 02:01:30 2024 read: IOPS=84, BW=1348KiB/s (1380kB/s)(6032KiB/4475msec) clat (usec): min=51, max=42224, avg=8000.61, stdev=5583.86 lat (usec): min=51, max=42225, avg=8001.18, stdev=5583.87 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 62], 10.00th=[ 65], 20.00th=[ 80], | 30.00th=[ 7308], 40.00th=[ 8717], 50.00th=[ 9896], 60.00th=[10421], | 70.00th=[11338], 80.00th=[12125], 90.00th=[13566], 95.00th=[14484], | 99.00th=[19006], 99.50th=[20841], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 288, max= 1536, per=24.89%, avg=1260.00, stdev=403.66, samples=8 iops : min= 18, max= 96, avg=78.75, stdev=25.23, samples=8 write: IOPS=87, BW=1398KiB/s (1432kB/s)(6256KiB/4475msec) clat (usec): min=665, max=93446, avg=3489.51, stdev=6769.97 lat (usec): min=666, max=93447, avg=3490.98, stdev=6770.00 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 725], 10.00th=[ 766], 20.00th=[ 824], | 30.00th=[ 873], 40.00th=[ 979], 50.00th=[ 1237], 60.00th=[ 2573], | 70.00th=[ 3228], 80.00th=[ 4555], 90.00th=[ 6521], 95.00th=[ 8979], | 99.00th=[32113], 99.50th=[44827], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 352, max= 1760, per=24.85%, avg=1324.00, stdev=425.54, samples=8 iops : min= 22, max= 110, avg=82.75, stdev=26.60, samples=8 lat (usec) : 100=12.89%, 250=0.65%, 500=0.13%, 750=3.65%, 1000=17.97% lat (msec) : 2=6.77%, 4=11.07%, 10=21.22%, 20=23.83%, 50=1.69% lat (msec) : 100=0.13% cpu : usr=0.18%, sys=21.21%, ctx=1946, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5064KiB/s (5185kB/s), 1262KiB/s-1348KiB/s (1292kB/s-1380kB/s), io=23.4MiB (24.5MB), run=4475-4730msec WRITE: bw=5328KiB/s (5456kB/s), 1275KiB/s-1438KiB/s (1306kB/s-1472kB/s), io=24.6MiB (25.8MB), run=4475-4730msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22654: Thu Jul 25 02:01:36 2024 read: IOPS=35, BW=575KiB/s (589kB/s)(5920KiB/10297msec) clat (usec): min=1852, max=49707, avg=4490.35, stdev=5132.25 lat (usec): min=1852, max=49709, avg=4491.06, stdev=5132.28 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 1975], 10.00th=[ 2057], 20.00th=[ 2147], | 30.00th=[ 2245], 40.00th=[ 2376], 50.00th=[ 2540], 60.00th=[ 2769], | 70.00th=[ 3392], 80.00th=[ 5342], 90.00th=[ 8979], 95.00th=[12911], | 99.00th=[27919], 99.50th=[42206], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 352, max= 928, per=24.64%, avg=561.35, stdev=152.69, samples=20 iops : min= 22, max= 58, avg=34.95, stdev= 9.55, samples=20 write: IOPS=38, BW=618KiB/s (633kB/s)(6368KiB/10297msec) clat (usec): min=11419, max=56581, avg=21529.24, stdev=5999.72 lat (usec): min=11420, max=56583, avg=21530.58, stdev=5999.73 clat percentiles (usec): | 1.00th=[12780], 5.00th=[13435], 10.00th=[14091], 20.00th=[15664], | 30.00th=[17957], 40.00th=[19792], 50.00th=[21627], 60.00th=[22676], | 70.00th=[23725], 80.00th=[26084], 90.00th=[28705], 95.00th=[31327], | 99.00th=[40633], 99.50th=[41681], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 287, max= 832, per=25.62%, avg=614.15, stdev=159.35, samples=20 iops : min= 17, max= 52, avg=38.25, stdev=10.02, samples=20 lat (msec) : 2=3.12%, 4=32.42%, 10=7.94%, 20=25.52%, 50=30.86% lat (msec) : 100=0.13% cpu : usr=0.13%, sys=5.12%, ctx=891, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22655: Thu Jul 25 02:01:36 2024 read: IOPS=34, BW=546KiB/s (560kB/s)(5744KiB/10511msec) clat (usec): min=1833, max=25002, avg=3901.53, stdev=3529.82 lat (usec): min=1833, max=25002, avg=3902.21, stdev=3529.83 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 1975], 10.00th=[ 2089], 20.00th=[ 2180], | 30.00th=[ 2278], 40.00th=[ 2409], 50.00th=[ 2540], 60.00th=[ 2704], | 70.00th=[ 3064], 80.00th=[ 3884], 90.00th=[ 8586], 95.00th=[13042], | 99.00th=[18220], 99.50th=[19792], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 192, max= 1024, per=23.94%, avg=545.35, stdev=217.30, samples=20 iops : min= 12, max= 64, avg=33.95, stdev=13.56, samples=20 write: IOPS=38, BW=623KiB/s (638kB/s)(6544KiB/10511msec) clat (msec): min=12, max=115, avg=22.11, stdev= 8.21 lat (msec): min=12, max=115, avg=22.11, stdev= 8.21 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 19], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 27], 90.00th=[ 31], 95.00th=[ 35], | 99.00th=[ 44], 99.50th=[ 57], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 384, max= 830, per=25.49%, avg=610.95, stdev=140.60, samples=20 iops : min= 24, max= 51, avg=38.05, stdev= 8.74, samples=20 lat (msec) : 2=2.86%, 4=34.90%, 10=5.21%, 20=24.22%, 50=32.42% lat (msec) : 100=0.26%, 250=0.13% cpu : usr=0.05%, sys=5.27%, ctx=887, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22656: Thu Jul 25 02:01:36 2024 read: IOPS=40, BW=641KiB/s (656kB/s)(6256KiB/9764msec) clat (usec): min=1920, max=30506, avg=4068.02, stdev=3770.31 lat (usec): min=1920, max=30507, avg=4068.80, stdev=3770.34 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2040], 10.00th=[ 2114], 20.00th=[ 2180], | 30.00th=[ 2278], 40.00th=[ 2376], 50.00th=[ 2573], 60.00th=[ 2868], | 70.00th=[ 3294], 80.00th=[ 5080], 90.00th=[ 8225], 95.00th=[11469], | 99.00th=[23462], 99.50th=[29230], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 320, max= 1248, per=28.24%, avg=643.32, stdev=241.17, samples=19 iops : min= 20, max= 78, avg=40.16, stdev=15.14, samples=19 write: IOPS=38, BW=618KiB/s (633kB/s)(6032KiB/9764msec) clat (usec): min=12514, max=69009, avg=21478.65, stdev=6561.71 lat (usec): min=12515, max=69012, avg=21479.93, stdev=6561.74 clat percentiles (usec): | 1.00th=[12649], 5.00th=[13566], 10.00th=[14353], 20.00th=[15533], | 30.00th=[16909], 40.00th=[19006], 50.00th=[20579], 60.00th=[22414], | 70.00th=[24249], 80.00th=[26346], 90.00th=[30016], 95.00th=[31851], | 99.00th=[40633], 99.50th=[47973], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 352, max= 832, per=25.50%, avg=611.32, stdev=151.99, samples=19 iops : min= 22, max= 52, avg=38.16, stdev= 9.55, samples=19 lat (msec) : 2=1.56%, 4=36.72%, 10=9.24%, 20=24.74%, 50=27.60% lat (msec) : 100=0.13% cpu : usr=0.10%, sys=5.31%, ctx=861, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22658: Thu Jul 25 02:01:36 2024 read: IOPS=37, BW=594KiB/s (609kB/s)(6032KiB/10149msec) clat (usec): min=1790, max=22517, avg=4002.89, stdev=3340.42 lat (usec): min=1791, max=22517, avg=4003.52, stdev=3340.43 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 1991], 10.00th=[ 2089], 20.00th=[ 2180], | 30.00th=[ 2311], 40.00th=[ 2442], 50.00th=[ 2638], 60.00th=[ 2900], | 70.00th=[ 3458], 80.00th=[ 4686], 90.00th=[ 8717], 95.00th=[11994], | 99.00th=[17695], 99.50th=[22414], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 288, max= 1024, per=25.90%, avg=590.10, stdev=212.90, samples=20 iops : min= 18, max= 64, avg=36.75, stdev=13.31, samples=20 write: IOPS=38, BW=616KiB/s (631kB/s)(6256KiB/10149msec) clat (msec): min=12, max=147, avg=22.06, stdev= 8.77 lat (msec): min=12, max=147, avg=22.07, stdev= 8.77 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 23], | 70.00th=[ 25], 80.00th=[ 26], 90.00th=[ 30], 95.00th=[ 34], | 99.00th=[ 45], 99.50th=[ 53], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 352, max= 800, per=25.56%, avg=612.60, stdev=148.42, samples=20 iops : min= 22, max= 50, avg=38.15, stdev= 9.34, samples=20 lat (msec) : 2=2.60%, 4=34.51%, 10=7.94%, 20=23.70%, 50=30.99% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.08%, sys=5.12%, ctx=860, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2279KiB/s (2333kB/s), 546KiB/s-641KiB/s (560kB/s-656kB/s), io=23.4MiB (24.5MB), run=9764-10511msec WRITE: bw=2397KiB/s (2455kB/s), 616KiB/s-623KiB/s (631kB/s-638kB/s), io=24.6MiB (25.8MB), run=9764-10511msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22674: Thu Jul 25 02:01:39 2024 read: IOPS=5, BW=5228KiB/s (5354kB/s)(7168KiB/1371msec) clat (usec): min=13970, max=38872, avg=23148.62, stdev=9448.37 lat (usec): min=13971, max=38873, avg=23149.77, stdev=9448.72 clat percentiles (usec): | 1.00th=[13960], 5.00th=[13960], 10.00th=[13960], 20.00th=[13960], | 30.00th=[14877], 40.00th=[14877], 50.00th=[25035], 60.00th=[25560], | 70.00th=[25560], 80.00th=[29492], 90.00th=[39060], 95.00th=[39060], | 99.00th=[39060], 99.50th=[39060], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3735KiB/s (3824kB/s)(5120KiB/1371msec) clat (msec): min=32, max=905, avg=234.72, stdev=376.26 lat (msec): min=32, max=905, avg=234.77, stdev=376.25 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33], | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 87], 60.00th=[ 87], | 70.00th=[ 110], 80.00th=[ 110], 90.00th=[ 902], 95.00th=[ 902], | 99.00th=[ 902], 99.50th=[ 902], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 2048, max= 2048, per=15.54%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=50.00%, 100=8.33%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=7.01%, ctx=71, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22675: Thu Jul 25 02:01:39 2024 read: IOPS=0, BW=962KiB/s (985kB/s)(2048KiB/2130msec) clat (msec): min=19, max=875, avg=447.58, stdev=605.16 lat (msec): min=19, max=875, avg=447.58, stdev=605.16 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 20], | 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 20], 60.00th=[ 877], | 70.00th=[ 877], 80.00th=[ 877], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 2043, max= 2048, per=21.72%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4808KiB/s (4923kB/s)(10.0MiB/2130msec) clat (msec): min=32, max=579, avg=122.42, stdev=164.59 lat (msec): min=32, max=579, avg=122.52, stdev=164.59 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 34], | 30.00th=[ 39], 40.00th=[ 50], 50.00th=[ 67], 60.00th=[ 80], | 70.00th=[ 107], 80.00th=[ 107], 90.00th=[ 144], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 6131, max=10240, per=62.10%, avg=8185.50, stdev=2905.50, samples=2 iops : min= 5, max= 10, avg= 7.50, stdev= 3.54, samples=2 lat (msec) : 20=8.33%, 50=33.33%, 100=25.00%, 250=16.67%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.14%, sys=5.59%, ctx=37, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22676: Thu Jul 25 02:01:39 2024 read: IOPS=2, BW=2825KiB/s (2893kB/s)(6144KiB/2175msec) clat (msec): min=26, max=105, avg=45.31, stdev=30.64 lat (msec): min=26, max=105, avg=45.31, stdev=30.64 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 29], 50.00th=[ 29], 60.00th=[ 33], | 70.00th=[ 51], 80.00th=[ 51], 90.00th=[ 106], 95.00th=[ 106], | 99.00th=[ 106], 99.50th=[ 106], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 4096, max= 6131, per=54.31%, avg=5113.50, stdev=1438.96, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2825KiB/s (2893kB/s)(6144KiB/2175msec) clat (msec): min=33, max=901, avg=312.45, stdev=417.32 lat (msec): min=33, max=901, avg=312.54, stdev=417.28 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 40], | 30.00th=[ 40], 40.00th=[ 41], 50.00th=[ 41], 60.00th=[ 63], | 70.00th=[ 802], 80.00th=[ 802], 90.00th=[ 902], 95.00th=[ 902], | 99.00th=[ 902], 99.50th=[ 902], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 4087, max= 6144, per=38.81%, avg=5115.50, stdev=1454.52, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 lat (msec) : 50=58.33%, 100=16.67%, 250=8.33%, 1000=16.67% cpu : usr=0.05%, sys=5.24%, ctx=57, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22677: Thu Jul 25 02:01:39 2024 read: IOPS=2, BW=2555KiB/s (2616kB/s)(5120KiB/2004msec) clat (msec): min=12, max=674, avg=146.71, stdev=295.19 lat (msec): min=12, max=674, avg=146.71, stdev=295.19 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 15], 50.00th=[ 16], 60.00th=[ 16], | 70.00th=[ 17], 80.00th=[ 17], 90.00th=[ 676], 95.00th=[ 676], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 8192, max= 8192, per=87.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3577KiB/s (3663kB/s)(7168KiB/2004msec) clat (msec): min=32, max=856, avg=178.70, stdev=300.20 lat (msec): min=32, max=856, avg=178.77, stdev=300.20 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 58], 60.00th=[ 93], | 70.00th=[ 93], 80.00th=[ 113], 90.00th=[ 860], 95.00th=[ 860], | 99.00th=[ 860], 99.50th=[ 860], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 6144, max= 8192, per=54.38%, avg=7168.00, stdev=1448.15, samples=2 iops : min= 6, max= 8, avg= 7.00, stdev= 1.41, samples=2 lat (msec) : 20=33.33%, 50=16.67%, 100=25.00%, 250=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.10%, sys=4.39%, ctx=41, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9416KiB/s (9642kB/s), 962KiB/s-5228KiB/s (985kB/s-5354kB/s), io=20.0MiB (20.0MB), run=1371-2175msec WRITE: bw=12.9MiB/s (13.5MB/s), 2825KiB/s-4808KiB/s (2893kB/s-4923kB/s), io=28.0MiB (29.4MB), run=1371-2175msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22680: Thu Jul 25 02:01:40 2024 read: IOPS=1, BW=1949KiB/s (1996kB/s)(7168KiB/3678msec) clat (usec): min=389, max=865894, avg=474116.79, stdev=426404.35 lat (usec): min=389, max=865894, avg=474117.87, stdev=426404.54 clat percentiles (usec): | 1.00th=[ 392], 5.00th=[ 392], 10.00th=[ 392], 20.00th=[ 3785], | 30.00th=[ 64226], 40.00th=[ 64226], 50.00th=[692061], 60.00th=[843056], | 70.00th=[843056], 80.00th=[843056], 90.00th=[868221], 95.00th=[868221], | 99.00th=[868221], 99.50th=[868221], 99.90th=[868221], 99.95th=[868221], | 99.99th=[868221] bw ( KiB/s): min= 1709, max= 2048, per=33.13%, avg=1844.60, stdev=185.68, samples=5 iops : min= 1, max= 2, avg= 1.40, stdev= 0.55, samples=5 write: IOPS=1, BW=1392KiB/s (1425kB/s)(5120KiB/3678msec) clat (msec): min=14, max=242, avg=65.57, stdev=99.51 lat (msec): min=14, max=242, avg=65.61, stdev=99.55 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15], | 30.00th=[ 15], 40.00th=[ 15], 50.00th=[ 16], 60.00th=[ 16], | 70.00th=[ 41], 80.00th=[ 41], 90.00th=[ 243], 95.00th=[ 243], | 99.00th=[ 243], 99.50th=[ 243], 99.90th=[ 243], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min= 1709, max= 1709, per=21.92%, avg=1709.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 4=8.33%, 20=25.00%, 50=8.33%, 100=8.33%, 250=8.33% lat (msec) : 750=8.33%, 1000=25.00% cpu : usr=0.00%, sys=32.74%, ctx=857, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22681: Thu Jul 25 02:01:40 2024 read: IOPS=1, BW=1072KiB/s (1097kB/s)(2048KiB/1911msec) clat (msec): min=49, max=793, avg=421.46, stdev=526.43 lat (msec): min=49, max=793, avg=421.46, stdev=526.43 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 793], | 70.00th=[ 793], 80.00th=[ 793], 90.00th=[ 793], 95.00th=[ 793], | 99.00th=[ 793], 99.50th=[ 793], 99.90th=[ 793], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 2048, max= 2048, per=36.78%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5358KiB/s (5487kB/s)(10.0MiB/1911msec) clat (msec): min=15, max=669, avg=103.74, stdev=199.70 lat (msec): min=15, max=669, avg=103.82, stdev=199.71 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 29], | 30.00th=[ 31], 40.00th=[ 35], 50.00th=[ 37], 60.00th=[ 41], | 70.00th=[ 45], 80.00th=[ 45], 90.00th=[ 92], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 6144, max=10240, per=100.00%, avg=8192.00, stdev=2896.31, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (msec) : 20=8.33%, 50=66.67%, 100=8.33%, 750=8.33%, 1000=8.33% cpu : usr=0.05%, sys=16.44%, ctx=148, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22682: Thu Jul 25 02:01:40 2024 read: IOPS=1, BW=1974KiB/s (2022kB/s)(6144KiB/3112msec) clat (usec): min=397, max=866704, avg=309742.01, stdev=396330.66 lat (usec): min=398, max=866706, avg=309743.29, stdev=396330.75 clat percentiles (usec): | 1.00th=[ 400], 5.00th=[ 400], 10.00th=[ 400], 20.00th=[ 453], | 30.00th=[ 453], 40.00th=[ 42730], 50.00th=[ 42730], 60.00th=[191890], | 70.00th=[759170], 80.00th=[759170], 90.00th=[868221], 95.00th=[868221], | 99.00th=[868221], 99.50th=[868221], 99.90th=[868221], 99.95th=[868221], | 99.99th=[868221] bw ( KiB/s): min= 1721, max= 4096, per=55.39%, avg=3084.33, stdev=1225.93, samples=3 iops : min= 1, max= 4, avg= 2.67, stdev= 1.53, samples=3 write: IOPS=1, BW=1974KiB/s (2022kB/s)(6144KiB/3112msec) clat (msec): min=23, max=1007, avg=205.68, stdev=393.64 lat (msec): min=23, max=1007, avg=206.03, stdev=393.54 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 38], 50.00th=[ 38], 60.00th=[ 38], | 70.00th=[ 102], 80.00th=[ 102], 90.00th=[ 1011], 95.00th=[ 1011], | 99.00th=[ 1011], 99.50th=[ 1011], 99.90th=[ 1011], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 3436, max= 4096, per=49.72%, avg=3876.00, stdev=381.05, samples=3 iops : min= 3, max= 4, avg= 3.67, stdev= 0.58, samples=3 lat (usec) : 500=16.67% lat (msec) : 50=41.67%, 250=16.67%, 1000=16.67% cpu : usr=0.00%, sys=19.90%, ctx=615, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22683: Thu Jul 25 02:01:40 2024 read: IOPS=2, BW=2824KiB/s (2892kB/s)(5120KiB/1813msec) clat (usec): min=567, max=834374, avg=256353.11, stdev=365615.97 lat (usec): min=568, max=834375, avg=256354.02, stdev=365616.15 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 570], 10.00th=[ 570], 20.00th=[ 570], | 30.00th=[ 3490], 40.00th=[ 3490], 50.00th=[ 36439], 60.00th=[ 36439], | 70.00th=[408945], 80.00th=[408945], 90.00th=[834667], 95.00th=[834667], | 99.00th=[834667], 99.50th=[834667], 99.90th=[834667], 99.95th=[834667], | 99.99th=[834667] bw ( KiB/s): min= 2048, max= 6144, per=73.56%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=3, BW=3954KiB/s (4049kB/s)(7168KiB/1813msec) clat (msec): min=34, max=148, avg=74.52, stdev=37.93 lat (msec): min=34, max=150, avg=74.88, stdev=38.65 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 39], | 30.00th=[ 66], 40.00th=[ 66], 50.00th=[ 74], 60.00th=[ 78], | 70.00th=[ 78], 80.00th=[ 86], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 6144, max= 8192, per=91.96%, avg=7168.00, stdev=1448.15, samples=2 iops : min= 6, max= 8, avg= 7.00, stdev= 1.41, samples=2 lat (usec) : 750=8.33% lat (msec) : 4=8.33%, 50=25.00%, 100=33.33%, 250=8.33%, 500=8.33% lat (msec) : 1000=8.33% cpu : usr=0.11%, sys=20.14%, ctx=474, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5568KiB/s (5702kB/s), 1072KiB/s-2824KiB/s (1097kB/s-2892kB/s), io=20.0MiB (20.0MB), run=1813-3678msec WRITE: bw=7796KiB/s (7983kB/s), 1392KiB/s-5358KiB/s (1425kB/s-5487kB/s), io=28.0MiB (29.4MB), run=1813-3678msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22691: Thu Jul 25 02:01:45 2024 read: IOPS=0, BW=3759KiB/s (3849kB/s)(12.0MiB/3269msec) clat (msec): min=105, max=1619, avg=1085.35, stdev=850.12 lat (msec): min=105, max=1619, avg=1085.36, stdev=850.12 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 106], 10.00th=[ 106], 20.00th=[ 106], | 30.00th=[ 106], 40.00th=[ 1536], 50.00th=[ 1536], 60.00th=[ 1536], | 70.00th=[ 1620], 80.00th=[ 1620], 90.00th=[ 1620], 95.00th=[ 1620], | 99.00th=[ 1620], 99.50th=[ 1620], 99.90th=[ 1620], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.03%, sys=44.25%, ctx=878, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22692: Thu Jul 25 02:01:45 2024 read: IOPS=0, BW=1315KiB/s (1346kB/s)(4096KiB/3116msec) clat (nsec): min=109205k, max=109205k, avg=109204529.00, stdev= 0.00 lat (nsec): min=109206k, max=109206k, avg=109206397.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 110], 10.00th=[ 110], 20.00th=[ 110], | 30.00th=[ 110], 40.00th=[ 110], 50.00th=[ 110], 60.00th=[ 110], | 70.00th=[ 110], 80.00th=[ 110], 90.00th=[ 110], 95.00th=[ 110], | 99.00th=[ 110], 99.50th=[ 110], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 8094, max= 8094, per=100.00%, avg=8094.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2629KiB/s (2692kB/s)(8192KiB/3116msec) clat (msec): min=341, max=2627, avg=1484.40, stdev=1616.48 lat (msec): min=341, max=2627, avg=1484.64, stdev=1616.47 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 342], 10.00th=[ 342], 20.00th=[ 342], | 30.00th=[ 342], 40.00th=[ 342], 50.00th=[ 342], 60.00th=[ 2635], | 70.00th=[ 2635], 80.00th=[ 2635], 90.00th=[ 2635], 95.00th=[ 2635], | 99.00th=[ 2635], 99.50th=[ 2635], 99.90th=[ 2635], 99.95th=[ 2635], | 99.99th=[ 2635] bw ( KiB/s): min= 8094, max= 8094, per=86.87%, avg=8094.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=7.80%, ctx=339, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22693: Thu Jul 25 02:01:45 2024 write: IOPS=0, BW=3494KiB/s (3578kB/s)(12.0MiB/3517msec) clat (msec): min=289, max=2506, avg=1152.76, stdev=1187.18 lat (msec): min=289, max=2506, avg=1153.04, stdev=1187.15 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 288], 10.00th=[ 288], 20.00th=[ 288], | 30.00th=[ 288], 40.00th=[ 659], 50.00th=[ 659], 60.00th=[ 659], | 70.00th=[ 2500], 80.00th=[ 2500], 90.00th=[ 2500], 95.00th=[ 2500], | 99.00th=[ 2500], 99.50th=[ 2500], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7275, max= 8175, per=82.91%, avg=7725.00, stdev=636.40, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.03%, sys=9.10%, ctx=73, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22694: Thu Jul 25 02:01:45 2024 write: IOPS=1, BW=5829KiB/s (5969kB/s)(12.0MiB/2108msec) clat (msec): min=100, max=1752, avg=683.52, stdev=927.05 lat (msec): min=100, max=1752, avg=683.82, stdev=927.06 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 101], 10.00th=[ 101], 20.00th=[ 101], | 30.00th=[ 101], 40.00th=[ 199], 50.00th=[ 199], 60.00th=[ 199], | 70.00th=[ 1754], 80.00th=[ 1754], 90.00th=[ 1754], 95.00th=[ 1754], | 99.00th=[ 1754], 99.50th=[ 1754], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 8192, max= 8192, per=87.93%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=13.72%, ctx=74, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5012KiB/s (5132kB/s), 1315KiB/s-3759KiB/s (1346kB/s-3849kB/s), io=16.0MiB (16.8MB), run=3116-3269msec WRITE: bw=9317KiB/s (9541kB/s), 2629KiB/s-5829KiB/s (2692kB/s-5969kB/s), io=32.0MiB (33.6MB), run=2108-3517msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22695: Thu Jul 25 02:01:45 2024 read: IOPS=0, BW=3459KiB/s (3542kB/s)(12.0MiB/3552msec) clat (msec): min=337, max=2524, avg=1176.72, stdev=1179.21 lat (msec): min=337, max=2524, avg=1176.72, stdev=1179.21 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 338], 10.00th=[ 338], 20.00th=[ 338], | 30.00th=[ 338], 40.00th=[ 667], 50.00th=[ 667], 60.00th=[ 667], | 70.00th=[ 2534], 80.00th=[ 2534], 90.00th=[ 2534], 95.00th=[ 2534], | 99.00th=[ 2534], 99.50th=[ 2534], 99.90th=[ 2534], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=2.73%, ctx=23, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22696: Thu Jul 25 02:01:45 2024 read: IOPS=0, BW=1288KiB/s (1319kB/s)(4096KiB/3179msec) clat (nsec): min=368449k, max=368449k, avg=368449138.00, stdev= 0.00 lat (nsec): min=368450k, max=368450k, avg=368449945.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 368], 10.00th=[ 368], 20.00th=[ 368], | 30.00th=[ 368], 40.00th=[ 368], 50.00th=[ 368], 60.00th=[ 368], | 70.00th=[ 368], 80.00th=[ 368], 90.00th=[ 368], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2577KiB/s (2639kB/s)(8192KiB/3179msec) clat (msec): min=179, max=2625, avg=1402.13, stdev=1729.49 lat (msec): min=179, max=2625, avg=1402.41, stdev=1729.45 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 180], 10.00th=[ 180], 20.00th=[ 180], | 30.00th=[ 180], 40.00th=[ 180], 50.00th=[ 180], 60.00th=[ 2635], | 70.00th=[ 2635], 80.00th=[ 2635], 90.00th=[ 2635], 95.00th=[ 2635], | 99.00th=[ 2635], 99.50th=[ 2635], 99.90th=[ 2635], 99.95th=[ 2635], | 99.99th=[ 2635] bw ( KiB/s): min= 8192, max= 8192, per=83.63%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=3.08%, ctx=139, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22697: Thu Jul 25 02:01:45 2024 write: IOPS=0, BW=3674KiB/s (3762kB/s)(12.0MiB/3345msec) clat (msec): min=162, max=2510, avg=1108.39, stdev=1238.69 lat (msec): min=162, max=2510, avg=1108.79, stdev=1238.69 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 163], 10.00th=[ 163], 20.00th=[ 163], | 30.00th=[ 163], 40.00th=[ 651], 50.00th=[ 651], 60.00th=[ 651], | 70.00th=[ 2500], 80.00th=[ 2500], 90.00th=[ 2500], 95.00th=[ 2500], | 99.00th=[ 2500], 99.50th=[ 2500], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 8192, max= 8192, per=83.63%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.03%, sys=4.72%, ctx=63, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22698: Thu Jul 25 02:01:45 2024 write: IOPS=0, BW=4005KiB/s (4101kB/s)(12.0MiB/3068msec) clat (msec): min=128, max=2606, avg=1020.86, stdev=1377.04 lat (msec): min=128, max=2607, avg=1021.13, stdev=1377.03 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 129], 10.00th=[ 129], 20.00th=[ 129], | 30.00th=[ 129], 40.00th=[ 330], 50.00th=[ 330], 60.00th=[ 330], | 70.00th=[ 2601], 80.00th=[ 2601], 90.00th=[ 2601], 95.00th=[ 2601], | 99.00th=[ 2601], 99.50th=[ 2601], 99.90th=[ 2601], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 8192, max= 8192, per=83.63%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=3.65%, ctx=36, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4613KiB/s (4723kB/s), 1288KiB/s-3459KiB/s (1319kB/s-3542kB/s), io=16.0MiB (16.8MB), run=3179-3552msec WRITE: bw=9796KiB/s (10.0MB/s), 2577KiB/s-4005KiB/s (2639kB/s-4101kB/s), io=32.0MiB (33.6MB), run=3068-3345msec