== sanity test 398b: DIO and buffer IO race ============== 22:45:21 (1733802321) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.24057 s, 40.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=19702: Mon Dec 9 22:45:35 2024 read: IOPS=135, BW=543KiB/s (556kB/s)(5992KiB/11043msec) clat (usec): min=46, max=171741, avg=4207.11, stdev=4947.79 lat (usec): min=46, max=171742, avg=4208.82, stdev=4947.66 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 1696], 10.00th=[ 2040], 20.00th=[ 2409], | 30.00th=[ 2737], 40.00th=[ 3130], 50.00th=[ 3654], 60.00th=[ 4146], | 70.00th=[ 4752], 80.00th=[ 5604], 90.00th=[ 6915], 95.00th=[ 8160], | 99.00th=[ 12911], 99.50th=[ 14746], 99.90th=[ 27919], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 248, max= 672, per=24.83%, avg=532.10, stdev=85.88, samples=21 iops : min= 62, max= 168, avg=133.00, stdev=21.46, samples=21 write: IOPS=142, BW=570KiB/s (584kB/s)(6296KiB/11043msec) clat (usec): min=413, max=16606, avg=2955.65, stdev=2708.65 lat (usec): min=413, max=16606, avg=2957.77, stdev=2707.74 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 510], 10.00th=[ 529], 20.00th=[ 570], | 30.00th=[ 676], 40.00th=[ 1205], 50.00th=[ 2343], 60.00th=[ 2966], | 70.00th=[ 3720], 80.00th=[ 5080], 90.00th=[ 6849], 95.00th=[ 8455], | 99.00th=[11076], 99.50th=[12256], 99.90th=[15401], 99.95th=[16581], | 99.99th=[16581] bw ( KiB/s): min= 296, max= 752, per=24.93%, avg=559.52, stdev=87.14, samples=21 iops : min= 74, max= 188, avg=139.86, stdev=21.77, samples=21 lat (usec) : 50=0.07%, 100=2.12%, 250=0.07%, 500=1.63%, 750=15.72% lat (usec) : 1000=2.25% lat (msec) : 2=5.73%, 4=37.83%, 10=32.23%, 20=2.31%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=22.93%, ctx=5824, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19704: Mon Dec 9 22:45:35 2024 read: IOPS=134, BW=538KiB/s (551kB/s)(5940KiB/11038msec) clat (usec): min=46, max=193223, avg=4137.85, stdev=5392.86 lat (usec): min=46, max=193223, avg=4138.53, stdev=5392.87 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 78], 10.00th=[ 1942], 20.00th=[ 2409], | 30.00th=[ 2769], 40.00th=[ 3195], 50.00th=[ 3687], 60.00th=[ 4293], | 70.00th=[ 4817], 80.00th=[ 5538], 90.00th=[ 6521], 95.00th=[ 7767], | 99.00th=[ 11731], 99.50th=[ 14353], 99.90th=[ 20579], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 272, max= 760, per=25.03%, avg=536.45, stdev=90.84, samples=22 iops : min= 68, max= 190, avg=134.05, stdev=22.69, samples=22 write: IOPS=143, BW=575KiB/s (589kB/s)(6348KiB/11038msec) clat (usec): min=405, max=16951, avg=3028.85, stdev=2736.51 lat (usec): min=405, max=16952, avg=3029.60, stdev=2736.55 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 510], 10.00th=[ 529], 20.00th=[ 578], | 30.00th=[ 693], 40.00th=[ 1237], 50.00th=[ 2474], 60.00th=[ 3064], | 70.00th=[ 3884], 80.00th=[ 5145], 90.00th=[ 6915], 95.00th=[ 8455], | 99.00th=[11600], 99.50th=[13304], 99.90th=[16188], 99.95th=[16909], | 99.99th=[16909] bw ( KiB/s): min= 352, max= 816, per=25.50%, avg=572.14, stdev=92.05, samples=22 iops : min= 88, max= 204, avg=142.95, stdev=23.08, samples=22 lat (usec) : 50=0.13%, 100=2.54%, 250=0.23%, 500=1.79%, 750=15.10% lat (usec) : 1000=2.38% lat (msec) : 2=5.60%, 4=35.84%, 10=34.34%, 20=1.99%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=23.35%, ctx=5700, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19707: Mon Dec 9 22:45:35 2024 read: IOPS=137, BW=550KiB/s (563kB/s)(6028KiB/10957msec) clat (usec): min=54, max=21856, avg=3992.45, stdev=2239.96 lat (usec): min=54, max=21857, avg=3994.17, stdev=2240.51 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 102], 10.00th=[ 1991], 20.00th=[ 2376], | 30.00th=[ 2704], 40.00th=[ 3097], 50.00th=[ 3589], 60.00th=[ 4178], | 70.00th=[ 4752], 80.00th=[ 5473], 90.00th=[ 6587], 95.00th=[ 7767], | 99.00th=[12125], 99.50th=[13173], 99.90th=[15795], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 288, max= 632, per=25.33%, avg=542.76, stdev=81.40, samples=21 iops : min= 72, max= 158, avg=135.67, stdev=20.35, samples=21 write: IOPS=142, BW=571KiB/s (585kB/s)(6260KiB/10957msec) clat (usec): min=405, max=168269, avg=3097.12, stdev=4978.77 lat (usec): min=406, max=168270, avg=3098.30, stdev=4978.82 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 506], 10.00th=[ 523], 20.00th=[ 570], | 30.00th=[ 660], 40.00th=[ 1369], 50.00th=[ 2409], 60.00th=[ 2966], | 70.00th=[ 3851], 80.00th=[ 5211], 90.00th=[ 6849], 95.00th=[ 8356], | 99.00th=[ 11338], 99.50th=[ 13042], 99.90th=[ 16319], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 344, max= 744, per=25.05%, avg=562.19, stdev=86.56, samples=21 iops : min= 86, max= 186, avg=140.52, stdev=21.64, samples=21 lat (usec) : 100=2.41%, 250=0.20%, 500=2.21%, 750=15.23%, 1000=1.73% lat (msec) : 2=5.44%, 4=37.21%, 10=33.40%, 20=2.12%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=22.94%, ctx=5630, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19709: Mon Dec 9 22:45:35 2024 read: IOPS=135, BW=540KiB/s (553kB/s)(6052KiB/11200msec) clat (usec): min=47, max=170240, avg=4164.48, stdev=4820.67 lat (usec): min=48, max=170240, avg=4167.31, stdev=4823.48 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 1663], 10.00th=[ 2008], 20.00th=[ 2409], | 30.00th=[ 2704], 40.00th=[ 3163], 50.00th=[ 3687], 60.00th=[ 4293], | 70.00th=[ 4948], 80.00th=[ 5604], 90.00th=[ 6587], 95.00th=[ 7767], | 99.00th=[ 11863], 99.50th=[ 13435], 99.90th=[ 20841], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 264, max= 776, per=24.99%, avg=535.55, stdev=95.61, samples=22 iops : min= 66, max= 194, avg=133.86, stdev=23.89, samples=22 write: IOPS=139, BW=557KiB/s (570kB/s)(6236KiB/11200msec) clat (usec): min=414, max=23378, avg=3082.81, stdev=2826.39 lat (usec): min=414, max=23379, avg=3083.54, stdev=2826.31 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 510], 10.00th=[ 529], 20.00th=[ 578], | 30.00th=[ 717], 40.00th=[ 1811], 50.00th=[ 2474], 60.00th=[ 3097], | 70.00th=[ 3785], 80.00th=[ 5080], 90.00th=[ 6915], 95.00th=[ 8586], | 99.00th=[12387], 99.50th=[14091], 99.90th=[21627], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 344, max= 928, per=24.59%, avg=551.91, stdev=111.59, samples=22 iops : min= 86, max= 232, avg=137.95, stdev=27.89, samples=22 lat (usec) : 50=0.03%, 100=2.08%, 250=0.13%, 500=1.63%, 750=14.16% lat (usec) : 1000=2.21% lat (msec) : 2=5.76%, 4=37.96%, 10=33.56%, 20=2.34%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.10%, sys=22.94%, ctx=5775, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2144KiB/s (2195kB/s), 538KiB/s-550KiB/s (551kB/s-563kB/s), io=23.4MiB (24.6MB), run=10957-11200msec WRITE: bw=2245KiB/s (2299kB/s), 557KiB/s-575KiB/s (570kB/s-589kB/s), io=24.6MiB (25.7MB), run=10957-11200msec rand-rw: (groupid=0, jobs=1): err= 0: pid=19703: Mon Dec 9 22:45:41 2024 read: IOPS=85, BW=340KiB/s (348kB/s)(5992KiB/17617msec) clat (usec): min=1511, max=198916, avg=4759.25, stdev=5715.37 lat (usec): min=1512, max=198918, avg=4760.02, stdev=5715.37 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1975], 10.00th=[ 2212], 20.00th=[ 2671], | 30.00th=[ 3228], 40.00th=[ 3687], 50.00th=[ 3916], 60.00th=[ 4178], | 70.00th=[ 4621], 80.00th=[ 5866], 90.00th=[ 8455], 95.00th=[ 10552], | 99.00th=[ 15008], 99.50th=[ 17171], 99.90th=[ 20841], 99.95th=[198181], | 99.99th=[198181] bw ( KiB/s): min= 144, max= 616, per=25.21%, avg=339.83, stdev=75.43, samples=35 iops : min= 36, max= 154, avg=84.91, stdev=18.83, samples=35 write: IOPS=89, BW=357KiB/s (366kB/s)(6296KiB/17617msec) clat (usec): min=2303, max=27103, avg=6604.37, stdev=2827.98 lat (usec): min=2304, max=27104, avg=6605.43, stdev=2828.08 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 3163], 10.00th=[ 3490], 20.00th=[ 4146], | 30.00th=[ 5145], 40.00th=[ 5800], 50.00th=[ 6325], 60.00th=[ 6718], | 70.00th=[ 7111], 80.00th=[ 8160], 90.00th=[10159], 95.00th=[12125], | 99.00th=[16712], 99.50th=[18744], 99.90th=[22152], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 160, max= 648, per=25.32%, avg=357.20, stdev=73.42, samples=35 iops : min= 40, max= 162, avg=89.26, stdev=18.37, samples=35 lat (msec) : 2=2.67%, 4=32.52%, 10=56.45%, 20=8.11%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=12.29%, ctx=3613, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19705: Mon Dec 9 22:45:41 2024 read: IOPS=84, BW=336KiB/s (344kB/s)(5940KiB/17668msec) clat (usec): min=1617, max=166279, avg=4592.91, stdev=4908.13 lat (usec): min=1618, max=166280, avg=4593.72, stdev=4908.18 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1991], 10.00th=[ 2180], 20.00th=[ 2606], | 30.00th=[ 3195], 40.00th=[ 3654], 50.00th=[ 3916], 60.00th=[ 4146], | 70.00th=[ 4490], 80.00th=[ 5407], 90.00th=[ 7963], 95.00th=[ 10028], | 99.00th=[ 14746], 99.50th=[ 15926], 99.90th=[ 19792], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 112, max= 648, per=24.97%, avg=336.57, stdev=88.19, samples=35 iops : min= 28, max= 162, avg=84.06, stdev=22.07, samples=35 write: IOPS=89, BW=359KiB/s (368kB/s)(6348KiB/17668msec) clat (usec): min=2339, max=27179, avg=6767.02, stdev=2886.83 lat (usec): min=2340, max=27180, avg=6768.02, stdev=2886.94 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 3294], 10.00th=[ 3621], 20.00th=[ 4359], | 30.00th=[ 5276], 40.00th=[ 5997], 50.00th=[ 6521], 60.00th=[ 6849], | 70.00th=[ 7308], 80.00th=[ 8356], 90.00th=[10159], 95.00th=[11994], | 99.00th=[17433], 99.50th=[19792], 99.90th=[26084], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 168, max= 656, per=25.42%, avg=358.74, stdev=73.91, samples=35 iops : min= 42, max= 164, avg=89.60, stdev=18.45, samples=35 lat (msec) : 2=2.60%, 4=31.48%, 10=57.94%, 20=7.71%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=12.24%, ctx=3618, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19708: Mon Dec 9 22:45:41 2024 read: IOPS=85, BW=342KiB/s (351kB/s)(6028KiB/17606msec) clat (usec): min=1642, max=20915, avg=4594.88, stdev=2554.81 lat (usec): min=1642, max=20915, avg=4595.71, stdev=2554.78 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 2024], 10.00th=[ 2212], 20.00th=[ 2671], | 30.00th=[ 3294], 40.00th=[ 3720], 50.00th=[ 3949], 60.00th=[ 4228], | 70.00th=[ 4686], 80.00th=[ 5735], 90.00th=[ 8160], 95.00th=[10290], | 99.00th=[13304], 99.50th=[14877], 99.90th=[19268], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 160, max= 560, per=25.41%, avg=342.51, stdev=67.13, samples=35 iops : min= 40, max= 140, avg=85.54, stdev=16.74, samples=35 write: IOPS=88, BW=356KiB/s (364kB/s)(6260KiB/17606msec) clat (msec): min=2, max=180, avg= 6.76, stdev= 5.27 lat (msec): min=2, max=180, avg= 6.76, stdev= 5.27 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 18], 99.50th=[ 21], 99.90th=[ 28], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 176, max= 680, per=25.17%, avg=355.09, stdev=80.39, samples=35 iops : min= 44, max= 170, avg=88.69, stdev=20.12, samples=35 lat (msec) : 2=2.25%, 4=32.29%, 10=57.58%, 20=7.58%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=12.14%, ctx=3597, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19710: Mon Dec 9 22:45:41 2024 read: IOPS=84, BW=340KiB/s (348kB/s)(6052KiB/17812msec) clat (usec): min=1613, max=21598, avg=4640.57, stdev=2724.81 lat (usec): min=1613, max=21599, avg=4641.64, stdev=2724.77 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 2008], 10.00th=[ 2180], 20.00th=[ 2606], | 30.00th=[ 3195], 40.00th=[ 3687], 50.00th=[ 3916], 60.00th=[ 4146], | 70.00th=[ 4490], 80.00th=[ 5866], 90.00th=[ 8717], 95.00th=[10552], | 99.00th=[14484], 99.50th=[15139], 99.90th=[19268], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 152, max= 616, per=25.18%, avg=339.37, stdev=83.67, samples=35 iops : min= 38, max= 154, avg=84.80, stdev=20.95, samples=35 write: IOPS=87, BW=350KiB/s (359kB/s)(6236KiB/17812msec) clat (msec): min=2, max=180, avg= 6.86, stdev= 5.34 lat (msec): min=2, max=180, avg= 6.86, stdev= 5.34 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 20], 99.50th=[ 21], 99.90th=[ 26], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 168, max= 648, per=24.76%, avg=349.43, stdev=76.24, samples=35 iops : min= 42, max= 162, avg=87.31, stdev=19.05, samples=35 lat (msec) : 2=2.34%, 4=32.55%, 10=56.58%, 20=8.11%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=12.46%, ctx=3608, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1348KiB/s (1380kB/s), 336KiB/s-342KiB/s (344kB/s-351kB/s), io=23.4MiB (24.6MB), run=17606-17812msec WRITE: bw=1411KiB/s (1445kB/s), 350KiB/s-359KiB/s (359kB/s-368kB/s), io=24.6MiB (25.7MB), run=17606-17812msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=19738: Mon Dec 9 22:45:48 2024 read: IOPS=68, BW=1097KiB/s (1124kB/s)(5920KiB/5395msec) clat (usec): min=1636, max=57750, avg=5546.78, stdev=4860.20 lat (usec): min=1636, max=57751, avg=5547.48, stdev=4860.21 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1942], 10.00th=[ 2073], 20.00th=[ 2311], | 30.00th=[ 2540], 40.00th=[ 2966], 50.00th=[ 3490], 60.00th=[ 4752], | 70.00th=[ 6849], 80.00th=[ 8586], 90.00th=[10814], 95.00th=[13042], | 99.00th=[22938], 99.50th=[24773], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 832, max= 1280, per=24.78%, avg=1084.80, stdev=167.61, samples=10 iops : min= 52, max= 80, avg=67.80, stdev=10.48, samples=10 write: IOPS=73, BW=1180KiB/s (1209kB/s)(6368KiB/5395msec) clat (usec): min=2533, max=46204, avg=8368.97, stdev=4950.67 lat (usec): min=2535, max=46206, avg=8370.13, stdev=4950.90 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 3130], 10.00th=[ 3523], 20.00th=[ 4293], | 30.00th=[ 5407], 40.00th=[ 6194], 50.00th=[ 7111], 60.00th=[ 8094], | 70.00th=[ 9241], 80.00th=[11863], 90.00th=[15401], 95.00th=[17957], | 99.00th=[23200], 99.50th=[26870], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 864, max= 1600, per=25.50%, avg=1174.40, stdev=223.26, samples=10 iops : min= 54, max= 100, avg=73.40, stdev=13.95, samples=10 lat (msec) : 2=3.26%, 4=31.77%, 10=44.14%, 20=18.88%, 50=1.82% lat (msec) : 100=0.13% cpu : usr=0.06%, sys=9.68%, ctx=1177, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19740: Mon Dec 9 22:45:48 2024 read: IOPS=67, BW=1080KiB/s (1106kB/s)(5744KiB/5318msec) clat (usec): min=1642, max=24108, avg=5382.46, stdev=4189.15 lat (usec): min=1642, max=24109, avg=5383.00, stdev=4189.16 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1860], 10.00th=[ 2008], 20.00th=[ 2245], | 30.00th=[ 2507], 40.00th=[ 2900], 50.00th=[ 3359], 60.00th=[ 4293], | 70.00th=[ 6456], 80.00th=[ 9110], 90.00th=[11076], 95.00th=[13435], | 99.00th=[19530], 99.50th=[23200], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 734, max= 1309, per=24.48%, avg=1071.50, stdev=163.31, samples=10 iops : min= 45, max= 81, avg=66.80, stdev=10.28, samples=10 write: IOPS=76, BW=1231KiB/s (1260kB/s)(6544KiB/5318msec) clat (usec): min=2343, max=37290, avg=8133.23, stdev=4619.90 lat (usec): min=2344, max=37291, avg=8134.32, stdev=4619.87 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 3130], 10.00th=[ 3392], 20.00th=[ 4146], | 30.00th=[ 5276], 40.00th=[ 6325], 50.00th=[ 7177], 60.00th=[ 8455], | 70.00th=[ 9372], 80.00th=[10814], 90.00th=[13435], 95.00th=[17433], | 99.00th=[23200], 99.50th=[25297], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 926, max= 1536, per=26.33%, avg=1212.30, stdev=180.06, samples=10 iops : min= 57, max= 96, avg=75.60, stdev=11.49, samples=10 lat (msec) : 2=4.56%, 4=31.64%, 10=42.97%, 20=18.88%, 50=1.95% cpu : usr=0.11%, sys=9.91%, ctx=1155, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19742: Mon Dec 9 22:45:48 2024 read: IOPS=84, BW=1345KiB/s (1377kB/s)(6256KiB/4652msec) clat (usec): min=1656, max=22861, avg=4816.79, stdev=3552.49 lat (usec): min=1656, max=22862, avg=4817.34, stdev=3552.50 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1860], 10.00th=[ 1975], 20.00th=[ 2245], | 30.00th=[ 2507], 40.00th=[ 2835], 50.00th=[ 3261], 60.00th=[ 4015], | 70.00th=[ 5342], 80.00th=[ 7504], 90.00th=[ 9634], 95.00th=[11994], | 99.00th=[17695], 99.50th=[20317], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 736, max= 1824, per=30.87%, avg=1351.11, stdev=338.19, samples=9 iops : min= 46, max= 114, avg=84.44, stdev=21.14, samples=9 write: IOPS=81, BW=1297KiB/s (1328kB/s)(6032KiB/4652msec) clat (usec): min=2461, max=35173, avg=7255.84, stdev=4741.18 lat (usec): min=2462, max=35174, avg=7256.90, stdev=4741.16 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2999], 10.00th=[ 3326], 20.00th=[ 3720], | 30.00th=[ 4293], 40.00th=[ 4817], 50.00th=[ 5538], 60.00th=[ 6259], | 70.00th=[ 8291], 80.00th=[10159], 90.00th=[14353], 95.00th=[16450], | 99.00th=[23725], 99.50th=[31065], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 960, max= 1664, per=27.80%, avg=1280.00, stdev=218.21, samples=9 iops : min= 60, max= 104, avg=80.00, stdev=13.64, samples=9 lat (msec) : 2=5.99%, 4=37.76%, 10=42.06%, 20=13.02%, 50=1.17% cpu : usr=0.11%, sys=9.55%, ctx=928, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19743: Mon Dec 9 22:45:48 2024 read: IOPS=68, BW=1102KiB/s (1129kB/s)(6032KiB/5472msec) clat (usec): min=1682, max=22812, avg=5366.81, stdev=3834.54 lat (usec): min=1683, max=22812, avg=5367.42, stdev=3834.50 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1942], 10.00th=[ 2114], 20.00th=[ 2278], | 30.00th=[ 2573], 40.00th=[ 2868], 50.00th=[ 3621], 60.00th=[ 4752], | 70.00th=[ 6521], 80.00th=[ 8848], 90.00th=[11207], 95.00th=[13173], | 99.00th=[15533], 99.50th=[18482], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 800, max= 1312, per=24.78%, avg=1084.80, stdev=135.30, samples=10 iops : min= 50, max= 82, avg=67.80, stdev= 8.46, samples=10 write: IOPS=71, BW=1143KiB/s (1171kB/s)(6256KiB/5472msec) clat (usec): min=2453, max=31730, avg=8610.81, stdev=5288.11 lat (usec): min=2454, max=31731, avg=8612.06, stdev=5288.16 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 3130], 10.00th=[ 3359], 20.00th=[ 3949], | 30.00th=[ 4948], 40.00th=[ 5800], 50.00th=[ 7242], 60.00th=[ 8848], | 70.00th=[10421], 80.00th=[12387], 90.00th=[15533], 95.00th=[19268], | 99.00th=[26870], 99.50th=[30802], 99.90th=[31851], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 832, max= 1376, per=24.88%, avg=1145.60, stdev=156.62, samples=10 iops : min= 52, max= 86, avg=71.60, stdev= 9.79, samples=10 lat (msec) : 2=3.26%, 4=33.72%, 10=39.71%, 20=20.96%, 50=2.34% cpu : usr=0.09%, sys=9.65%, ctx=1162, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4377KiB/s (4482kB/s), 1080KiB/s-1345KiB/s (1106kB/s-1377kB/s), io=23.4MiB (24.5MB), run=4652-5472msec WRITE: bw=4605KiB/s (4716kB/s), 1143KiB/s-1297KiB/s (1171kB/s-1328kB/s), io=24.6MiB (25.8MB), run=4652-5472msec rand-rw: (groupid=0, jobs=1): err= 0: pid=19736: Mon Dec 9 22:45:48 2024 read: IOPS=68, BW=1099KiB/s (1125kB/s)(5920KiB/5387msec) clat (usec): min=59, max=38210, avg=10147.19, stdev=3641.15 lat (usec): min=59, max=38210, avg=10147.80, stdev=3641.20 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 6652], 10.00th=[ 7439], 20.00th=[ 8225], | 30.00th=[ 8717], 40.00th=[ 9241], 50.00th=[ 9896], 60.00th=[10421], | 70.00th=[11076], 80.00th=[12125], 90.00th=[13960], 95.00th=[15270], | 99.00th=[21103], 99.50th=[23462], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 798, max= 1280, per=25.34%, avg=1090.70, stdev=154.67, samples=10 iops : min= 49, max= 80, avg=68.00, stdev= 9.88, samples=10 write: IOPS=73, BW=1182KiB/s (1210kB/s)(6368KiB/5387msec) clat (usec): min=639, max=20232, avg=3928.29, stdev=3446.95 lat (usec): min=640, max=20232, avg=3930.84, stdev=3445.65 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 758], 10.00th=[ 791], 20.00th=[ 1074], | 30.00th=[ 1991], 40.00th=[ 2376], 50.00th=[ 2802], 60.00th=[ 3589], | 70.00th=[ 4555], 80.00th=[ 5604], 90.00th=[ 8586], 95.00th=[12387], | 99.00th=[15795], 99.50th=[17957], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 896, max= 1472, per=25.99%, avg=1177.10, stdev=199.91, samples=10 iops : min= 56, max= 92, avg=73.40, stdev=12.52, samples=10 lat (usec) : 100=1.43%, 250=0.39%, 500=0.13%, 750=0.65%, 1000=9.37% lat (msec) : 2=5.73%, 4=17.32%, 10=38.93%, 20=25.13%, 50=0.91% cpu : usr=0.06%, sys=19.27%, ctx=2732, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19737: Mon Dec 9 22:45:48 2024 read: IOPS=67, BW=1084KiB/s (1110kB/s)(5744KiB/5301msec) clat (usec): min=55, max=35572, avg=9990.33, stdev=3509.85 lat (usec): min=56, max=35573, avg=9993.43, stdev=3512.28 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 6521], 10.00th=[ 7308], 20.00th=[ 8029], | 30.00th=[ 8586], 40.00th=[ 9110], 50.00th=[ 9634], 60.00th=[10290], | 70.00th=[10683], 80.00th=[11600], 90.00th=[13435], 95.00th=[14746], | 99.00th=[23725], 99.50th=[27395], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 734, max= 1312, per=25.12%, avg=1081.10, stdev=161.19, samples=10 iops : min= 45, max= 82, avg=67.40, stdev=10.23, samples=10 write: IOPS=77, BW=1234KiB/s (1264kB/s)(6544KiB/5301msec) clat (usec): min=682, max=23773, avg=4078.56, stdev=3719.75 lat (usec): min=683, max=23774, avg=4079.88, stdev=3719.62 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 766], 10.00th=[ 791], 20.00th=[ 1090], | 30.00th=[ 2008], 40.00th=[ 2376], 50.00th=[ 2802], 60.00th=[ 3326], | 70.00th=[ 4555], 80.00th=[ 5735], 90.00th=[ 9896], 95.00th=[12256], | 99.00th=[16909], 99.50th=[17433], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 958, max= 1472, per=26.98%, avg=1222.00, stdev=167.86, samples=10 iops : min= 59, max= 92, avg=76.20, stdev=10.80, samples=10 lat (usec) : 100=1.04%, 250=0.26%, 750=1.43%, 1000=8.72% lat (msec) : 2=5.34%, 4=18.49%, 10=38.93%, 20=24.74%, 50=1.04% cpu : usr=0.11%, sys=19.77%, ctx=2751, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19739: Mon Dec 9 22:45:48 2024 read: IOPS=70, BW=1124KiB/s (1151kB/s)(6256KiB/5564msec) clat (usec): min=58, max=33937, avg=10225.77, stdev=3079.29 lat (usec): min=58, max=33938, avg=10227.27, stdev=3078.88 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 6849], 10.00th=[ 7504], 20.00th=[ 8160], | 30.00th=[ 8848], 40.00th=[ 9372], 50.00th=[ 9896], 60.00th=[10421], | 70.00th=[11207], 80.00th=[11994], 90.00th=[13304], 95.00th=[15401], | 99.00th=[20579], 99.50th=[21103], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 704, max= 1373, per=26.02%, avg=1119.73, stdev=182.29, samples=11 iops : min= 44, max= 85, avg=69.91, stdev=11.28, samples=11 write: IOPS=67, BW=1084KiB/s (1110kB/s)(6032KiB/5564msec) clat (usec): min=654, max=29522, avg=3993.68, stdev=3852.34 lat (usec): min=655, max=29523, avg=3995.09, stdev=3852.67 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 758], 10.00th=[ 783], 20.00th=[ 955], | 30.00th=[ 1991], 40.00th=[ 2343], 50.00th=[ 2671], 60.00th=[ 3228], | 70.00th=[ 4359], 80.00th=[ 5932], 90.00th=[ 9372], 95.00th=[12649], | 99.00th=[17957], 99.50th=[19792], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 832, max= 1344, per=23.57%, avg=1067.36, stdev=153.01, samples=11 iops : min= 52, max= 84, avg=66.64, stdev= 9.57, samples=11 lat (usec) : 100=0.52%, 750=1.69%, 1000=9.24% lat (msec) : 2=4.17%, 4=18.23%, 10=38.54%, 20=26.82%, 50=0.78% cpu : usr=0.09%, sys=18.82%, ctx=2750, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19741: Mon Dec 9 22:45:48 2024 read: IOPS=68, BW=1104KiB/s (1130kB/s)(6032KiB/5464msec) clat (usec): min=57, max=20477, avg=10013.46, stdev=3527.48 lat (usec): min=58, max=20478, avg=10014.02, stdev=3527.50 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 123], 10.00th=[ 7242], 20.00th=[ 8160], | 30.00th=[ 8717], 40.00th=[ 9372], 50.00th=[ 9765], 60.00th=[10421], | 70.00th=[11076], 80.00th=[12256], 90.00th=[13829], 95.00th=[16188], | 99.00th=[19530], 99.50th=[20317], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 896, max= 1216, per=25.35%, avg=1090.90, stdev=103.93, samples=10 iops : min= 56, max= 76, avg=68.10, stdev= 6.51, samples=10 write: IOPS=71, BW=1145KiB/s (1172kB/s)(6256KiB/5464msec) clat (usec): min=663, max=21139, avg=4272.81, stdev=3982.36 lat (usec): min=664, max=21140, avg=4274.25, stdev=3983.33 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 766], 10.00th=[ 791], 20.00th=[ 1074], | 30.00th=[ 2073], 40.00th=[ 2376], 50.00th=[ 2737], 60.00th=[ 3359], | 70.00th=[ 4490], 80.00th=[ 6718], 90.00th=[10552], 95.00th=[13435], | 99.00th=[17433], 99.50th=[20317], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 768, max= 1408, per=25.29%, avg=1145.30, stdev=166.83, samples=10 iops : min= 48, max= 88, avg=71.50, stdev=10.34, samples=10 lat (usec) : 100=2.08%, 250=0.52%, 750=1.43%, 1000=8.20% lat (msec) : 2=4.69%, 4=18.62%, 10=35.42%, 20=28.52%, 50=0.52% cpu : usr=0.05%, sys=18.82%, ctx=2724, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4305KiB/s (4408kB/s), 1084KiB/s-1124KiB/s (1110kB/s-1151kB/s), io=23.4MiB (24.5MB), run=5301-5564msec WRITE: bw=4529KiB/s (4638kB/s), 1084KiB/s-1234KiB/s (1110kB/s-1264kB/s), io=24.6MiB (25.8MB), run=5301-5564msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=19754: Mon Dec 9 22:45:51 2024 read: IOPS=11, BW=11.4MiB/s (11.9MB/s)(7168KiB/615msec) clat (usec): min=360, max=334264, avg=65449.84, stdev=125221.27 lat (usec): min=361, max=334265, avg=65450.85, stdev=125221.31 clat percentiles (usec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 412], | 30.00th=[ 445], 40.00th=[ 445], 50.00th=[ 4948], 60.00th=[ 6980], | 70.00th=[ 6980], 80.00th=[110625], 90.00th=[333448], 95.00th=[333448], | 99.00th=[333448], 99.50th=[333448], 99.90th=[333448], 99.95th=[333448], | 99.99th=[333448] bw ( KiB/s): min= 9481, max= 9481, per=85.18%, avg=9481.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8325KiB/s (8525kB/s)(5120KiB/615msec) clat (usec): min=18201, max=39388, avg=27224.20, stdev=8816.48 lat (usec): min=18219, max=39407, avg=27260.85, stdev=8802.50 clat percentiles (usec): | 1.00th=[18220], 5.00th=[18220], 10.00th=[18220], 20.00th=[18220], | 30.00th=[21627], 40.00th=[21627], 50.00th=[23725], 60.00th=[23725], | 70.00th=[33424], 80.00th=[33424], 90.00th=[39584], 95.00th=[39584], | 99.00th=[39584], 99.50th=[39584], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 3792, max= 3792, per=24.34%, avg=3792.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (usec) : 500=25.00% lat (msec) : 10=16.67%, 20=8.33%, 50=33.33%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=59.77%, ctx=171, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19756: Mon Dec 9 22:45:51 2024 read: IOPS=1, BW=1113KiB/s (1140kB/s)(2048KiB/1840msec) clat (msec): min=49, max=1025, avg=538.00, stdev=690.14 lat (msec): min=50, max=1025, avg=538.00, stdev=690.13 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 51], 10.00th=[ 51], 20.00th=[ 51], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 1028], | 70.00th=[ 1028], 80.00th=[ 1028], 90.00th=[ 1028], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 1935, max= 1935, per=17.39%, avg=1935.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5565KiB/s (5699kB/s)(10.0MiB/1840msec) clat (msec): min=27, max=122, avg=68.80, stdev=32.60 lat (msec): min=27, max=122, avg=68.86, stdev=32.58 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 51], 60.00th=[ 65], | 70.00th=[ 96], 80.00th=[ 96], 90.00th=[ 117], 95.00th=[ 123], | 99.00th=[ 123], 99.50th=[ 123], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 1939, max= 7742, per=31.06%, avg=4840.50, stdev=4103.34, samples=2 iops : min= 1, max= 7, avg= 4.00, stdev= 4.24, samples=2 lat (msec) : 50=41.67%, 100=33.33%, 250=16.67% cpu : usr=0.00%, sys=27.03%, ctx=374, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19758: Mon Dec 9 22:45:51 2024 read: IOPS=12, BW=12.9MiB/s (13.6MB/s)(6144KiB/464msec) clat (usec): min=765, max=141239, avg=40061.14, stdev=53502.68 lat (usec): min=767, max=141240, avg=40062.05, stdev=53502.61 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 766], 10.00th=[ 766], 20.00th=[ 4015], | 30.00th=[ 4015], 40.00th=[ 6783], 50.00th=[ 6783], 60.00th=[ 36963], | 70.00th=[ 50594], 80.00th=[ 50594], 90.00th=[141558], 95.00th=[141558], | 99.00th=[141558], 99.50th=[141558], 99.90th=[141558], 99.95th=[141558], | 99.99th=[141558] write: IOPS=12, BW=12.9MiB/s (13.6MB/s)(6144KiB/464msec) clat (usec): min=19977, max=60298, avg=32866.20, stdev=14770.53 lat (usec): min=19996, max=60440, avg=32922.00, stdev=14814.97 clat percentiles (usec): | 1.00th=[20055], 5.00th=[20055], 10.00th=[20055], 20.00th=[21103], | 30.00th=[21103], 40.00th=[27657], 50.00th=[27657], 60.00th=[33424], | 70.00th=[34866], 80.00th=[34866], 90.00th=[60556], 95.00th=[60556], | 99.00th=[60556], 99.50th=[60556], 99.90th=[60556], 99.95th=[60556], | 99.99th=[60556] lat (usec) : 1000=8.33% lat (msec) : 10=16.67%, 20=8.33%, 50=41.67%, 100=16.67%, 250=8.33% cpu : usr=0.43%, sys=45.14%, ctx=144, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19759: Mon Dec 9 22:45:51 2024 read: IOPS=3, BW=3202KiB/s (3279kB/s)(5120KiB/1599msec) clat (msec): min=4, max=999, avg=251.62, stdev=420.96 lat (msec): min=4, max=999, avg=251.62, stdev=420.96 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 85], 60.00th=[ 85], | 70.00th=[ 129], 80.00th=[ 129], 90.00th=[ 1003], 95.00th=[ 1003], | 99.00th=[ 1003], 99.50th=[ 1003], 99.90th=[ 1003], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 4087, max= 4096, per=36.76%, avg=4091.50, stdev= 6.36, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4483KiB/s (4590kB/s)(7168KiB/1599msec) clat (usec): min=23045, max=73613, avg=45687.51, stdev=19948.60 lat (usec): min=23056, max=73664, avg=45739.79, stdev=19963.22 clat percentiles (usec): | 1.00th=[22938], 5.00th=[22938], 10.00th=[22938], 20.00th=[29754], | 30.00th=[32637], 40.00th=[32637], 50.00th=[37487], 60.00th=[55313], | 70.00th=[55313], 80.00th=[68682], 90.00th=[73925], 95.00th=[73925], | 99.00th=[73925], 99.50th=[73925], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 6144, max= 8175, per=45.95%, avg=7159.50, stdev=1436.13, samples=2 iops : min= 6, max= 7, avg= 6.50, stdev= 0.71, samples=2 lat (msec) : 10=8.33%, 50=41.67%, 100=33.33%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=28.16%, ctx=453, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.9MiB/s (11.4MB/s), 1113KiB/s-12.9MiB/s (1140kB/s-13.6MB/s), io=20.0MiB (20.0MB), run=464-1840msec WRITE: bw=15.2MiB/s (15.0MB/s), 4483KiB/s-12.9MiB/s (4590kB/s-13.6MB/s), io=28.0MiB (29.4MB), run=464-1840msec rand-rw: (groupid=0, jobs=1): err= 0: pid=19753: Mon Dec 9 22:45:51 2024 read: IOPS=3, BW=3689KiB/s (3778kB/s)(7168KiB/1943msec) clat (msec): min=16, max=1312, avg=230.15, stdev=477.86 lat (msec): min=16, max=1312, avg=230.15, stdev=477.86 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 19], | 30.00th=[ 48], 40.00th=[ 48], 50.00th=[ 51], 60.00th=[ 74], | 70.00th=[ 74], 80.00th=[ 93], 90.00th=[ 1318], 95.00th=[ 1318], | 99.00th=[ 1318], 99.50th=[ 1318], 99.90th=[ 1318], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 2048, max= 2048, per=19.43%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2635KiB/s (2698kB/s)(5120KiB/1943msec) clat (usec): min=19605, max=25509, avg=23256.59, stdev=2186.72 lat (usec): min=19632, max=25536, avg=23295.89, stdev=2189.80 clat percentiles (usec): | 1.00th=[19530], 5.00th=[19530], 10.00th=[19530], 20.00th=[19530], | 30.00th=[23462], 40.00th=[23462], 50.00th=[23725], 60.00th=[23725], | 70.00th=[23725], 80.00th=[23725], 90.00th=[25560], 95.00th=[25560], | 99.00th=[25560], 99.50th=[25560], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] lat (msec) : 20=25.00%, 50=50.00%, 100=16.67% cpu : usr=0.00%, sys=6.02%, ctx=63, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19755: Mon Dec 9 22:45:51 2024 read: IOPS=1, BW=1058KiB/s (1083kB/s)(2048KiB/1936msec) clat (msec): min=20, max=1105, avg=563.09, stdev=767.56 lat (msec): min=20, max=1105, avg=563.09, stdev=767.56 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 21], 60.00th=[ 1099], | 70.00th=[ 1099], 80.00th=[ 1099], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 2048, max= 2048, per=19.43%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5289KiB/s (5416kB/s)(10.0MiB/1936msec) clat (msec): min=17, max=149, avg=75.76, stdev=43.70 lat (msec): min=17, max=149, avg=75.80, stdev=43.69 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 25], | 30.00th=[ 50], 40.00th=[ 53], 50.00th=[ 58], 60.00th=[ 72], | 70.00th=[ 118], 80.00th=[ 118], 90.00th=[ 121], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 2048, max= 8192, per=34.70%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (msec) : 20=8.33%, 50=25.00%, 100=33.33%, 250=25.00% cpu : usr=0.05%, sys=7.39%, ctx=55, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19757: Mon Dec 9 22:45:51 2024 read: IOPS=3, BW=3847KiB/s (3940kB/s)(6144KiB/1597msec) clat (msec): min=13, max=797, avg=161.32, stdev=313.27 lat (msec): min=13, max=797, avg=161.32, stdev=313.27 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 16], 40.00th=[ 17], 50.00th=[ 17], 60.00th=[ 31], | 70.00th=[ 94], 80.00th=[ 94], 90.00th=[ 802], 95.00th=[ 802], | 99.00th=[ 802], 99.50th=[ 802], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min=10240, max=10240, per=97.15%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3847KiB/s (3940kB/s)(6144KiB/1597msec) clat (msec): min=19, max=284, avg=90.51, stdev=100.46 lat (msec): min=19, max=284, avg=90.56, stdev=100.46 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 43], 50.00th=[ 43], 60.00th=[ 57], | 70.00th=[ 113], 80.00th=[ 113], 90.00th=[ 284], 95.00th=[ 284], | 99.00th=[ 284], 99.50th=[ 284], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 4096, max= 8192, per=41.64%, avg=6144.00, stdev=2896.31, samples=2 iops : min= 4, max= 8, avg= 6.00, stdev= 2.83, samples=2 lat (msec) : 20=33.33%, 50=25.00%, 100=16.67%, 250=8.33%, 500=8.33% lat (msec) : 1000=8.33% cpu : usr=0.06%, sys=7.83%, ctx=84, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19760: Mon Dec 9 22:45:51 2024 read: IOPS=3, BW=3162KiB/s (3238kB/s)(5120KiB/1619msec) clat (msec): min=13, max=799, avg=182.51, stdev=344.98 lat (msec): min=13, max=799, avg=182.51, stdev=344.98 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 14], | 30.00th=[ 29], 40.00th=[ 29], 50.00th=[ 29], 60.00th=[ 29], | 70.00th=[ 43], 80.00th=[ 43], 90.00th=[ 802], 95.00th=[ 802], | 99.00th=[ 802], 99.50th=[ 802], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 8192, max= 8192, per=77.72%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4427KiB/s (4534kB/s)(7168KiB/1619msec) clat (msec): min=17, max=289, avg=70.48, stdev=98.20 lat (msec): min=17, max=289, avg=70.53, stdev=98.21 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 39], 60.00th=[ 45], | 70.00th=[ 45], 80.00th=[ 67], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 2048, max=12288, per=48.58%, avg=7168.00, stdev=7240.77, samples=2 iops : min= 2, max= 12, avg= 7.00, stdev= 7.07, samples=2 lat (msec) : 20=33.33%, 50=41.67%, 100=8.33%, 500=8.33%, 1000=8.33% cpu : usr=0.12%, sys=7.11%, ctx=73, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.3MiB/s (10.8MB/s), 1058KiB/s-3847KiB/s (1083kB/s-3940kB/s), io=20.0MiB (20.0MB), run=1597-1943msec WRITE: bw=14.4MiB/s (15.1MB/s), 2635KiB/s-5289KiB/s (2698kB/s-5416kB/s), io=28.0MiB (29.4MB), run=1597-1943msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=19767: Mon Dec 9 22:45:54 2024 read: IOPS=1, BW=7190KiB/s (7363kB/s)(12.0MiB/1709msec) clat (msec): min=97, max=1353, avg=561.60, stdev=689.56 lat (msec): min=97, max=1353, avg=561.60, stdev=689.56 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 99], 10.00th=[ 99], 20.00th=[ 99], | 30.00th=[ 99], 40.00th=[ 234], 50.00th=[ 234], 60.00th=[ 234], | 70.00th=[ 1351], 80.00th=[ 1351], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 7314, max= 7314, per=77.19%, avg=7314.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=5.27%, ctx=46, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19768: Mon Dec 9 22:45:54 2024 read: IOPS=0, BW=2369KiB/s (2426kB/s)(4096KiB/1729msec) clat (nsec): min=165007k, max=165007k, avg=165006712.00, stdev= 0.00 lat (nsec): min=165009k, max=165009k, avg=165008792.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 165], 10.00th=[ 165], 20.00th=[ 165], | 30.00th=[ 165], 40.00th=[ 165], 50.00th=[ 165], 60.00th=[ 165], | 70.00th=[ 165], 80.00th=[ 165], 90.00th=[ 165], 95.00th=[ 165], | 99.00th=[ 165], 99.50th=[ 165], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 8078, max= 8078, per=85.26%, avg=8078.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4738KiB/s (4852kB/s)(8192KiB/1729msec) clat (msec): min=203, max=1337, avg=770.60, stdev=801.62 lat (msec): min=204, max=1337, avg=770.84, stdev=801.63 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 205], 10.00th=[ 205], 20.00th=[ 205], | 30.00th=[ 205], 40.00th=[ 205], 50.00th=[ 205], 60.00th=[ 1334], | 70.00th=[ 1334], 80.00th=[ 1334], 90.00th=[ 1334], 95.00th=[ 1334], | 99.00th=[ 1334], 99.50th=[ 1334], 99.90th=[ 1334], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 8078, max= 8078, per=42.63%, avg=8078.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.17%, sys=5.21%, ctx=50, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19769: Mon Dec 9 22:45:54 2024 write: IOPS=10, BW=42.7MiB/s (44.8MB/s)(12.0MiB/281msec) clat (msec): min=31, max=185, avg=88.82, stdev=83.92 lat (msec): min=31, max=185, avg=89.12, stdev=83.96 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 51], | 70.00th=[ 186], 80.00th=[ 186], 90.00th=[ 186], 95.00th=[ 186], | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] lat (msec) : 50=33.33%, 100=33.33%, 250=33.33% cpu : usr=0.71%, sys=23.57%, ctx=65, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19770: Mon Dec 9 22:45:54 2024 write: IOPS=5, BW=23.0MiB/s (24.2MB/s)(12.0MiB/521msec) clat (msec): min=52, max=335, avg=166.94, stdev=149.23 lat (msec): min=52, max=335, avg=167.25, stdev=149.26 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 113], 50.00th=[ 113], 60.00th=[ 113], | 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min=16384, max=16384, per=86.45%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=18.08%, ctx=58, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9476KiB/s (9703kB/s), 2369KiB/s-7190KiB/s (2426kB/s-7363kB/s), io=16.0MiB (16.8MB), run=1709-1729msec WRITE: bw=18.5MiB/s (19.4MB/s), 4738KiB/s-42.7MiB/s (4852kB/s-44.8MB/s), io=32.0MiB (33.6MB), run=281-1729msec rand-rw: (groupid=0, jobs=1): err= 0: pid=19772: Mon Dec 9 22:45:54 2024 read: IOPS=2, BW=9518KiB/s (9747kB/s)(12.0MiB/1291msec) clat (msec): min=31, max=1128, avg=428.04, stdev=608.19 lat (msec): min=31, max=1128, avg=428.04, stdev=608.19 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 125], 50.00th=[ 125], 60.00th=[ 125], | 70.00th=[ 1133], 80.00th=[ 1133], 90.00th=[ 1133], 95.00th=[ 1133], | 99.00th=[ 1133], 99.50th=[ 1133], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 8192, max= 8192, per=91.90%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33% cpu : usr=0.08%, sys=65.04%, ctx=218, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19773: Mon Dec 9 22:45:54 2024 read: IOPS=0, BW=2229KiB/s (2282kB/s)(4096KiB/1838msec) clat (nsec): min=130738k, max=130738k, avg=130738323.00, stdev= 0.00 lat (nsec): min=130739k, max=130739k, avg=130739178.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 131], 10.00th=[ 131], 20.00th=[ 131], | 30.00th=[ 131], 40.00th=[ 131], 50.00th=[ 131], 60.00th=[ 131], | 70.00th=[ 131], 80.00th=[ 131], 90.00th=[ 131], 95.00th=[ 131], | 99.00th=[ 131], 99.50th=[ 131], 99.90th=[ 131], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 8192, max= 8192, per=91.90%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4457KiB/s (4564kB/s)(8192KiB/1838msec) clat (msec): min=262, max=1436, avg=849.55, stdev=830.12 lat (msec): min=262, max=1436, avg=849.82, stdev=830.08 clat percentiles (msec): | 1.00th=[ 264], 5.00th=[ 264], 10.00th=[ 264], 20.00th=[ 264], | 30.00th=[ 264], 40.00th=[ 264], 50.00th=[ 264], 60.00th=[ 1435], | 70.00th=[ 1435], 80.00th=[ 1435], 90.00th=[ 1435], 95.00th=[ 1435], | 99.00th=[ 1435], 99.50th=[ 1435], 99.90th=[ 1435], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 8192, max= 8192, per=47.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.05%, sys=12.85%, ctx=96, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19774: Mon Dec 9 22:45:54 2024 write: IOPS=1, BW=6508KiB/s (6665kB/s)(12.0MiB/1888msec) clat (msec): min=87, max=1438, avg=625.56, stdev=715.83 lat (msec): min=88, max=1438, avg=625.75, stdev=715.87 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 88], 10.00th=[ 88], 20.00th=[ 88], | 30.00th=[ 88], 40.00th=[ 351], 50.00th=[ 351], 60.00th=[ 351], | 70.00th=[ 1435], 80.00th=[ 1435], 90.00th=[ 1435], 95.00th=[ 1435], | 99.00th=[ 1435], 99.50th=[ 1435], 99.90th=[ 1435], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 8192, max= 8192, per=47.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.11%, sys=15.10%, ctx=51, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19775: Mon Dec 9 22:45:54 2024 write: IOPS=5, BW=20.9MiB/s (21.9MB/s)(12.0MiB/575msec) clat (msec): min=124, max=285, avg=185.43, stdev=87.75 lat (msec): min=124, max=286, avg=185.70, stdev=87.72 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 125], 10.00th=[ 125], 20.00th=[ 125], | 30.00th=[ 125], 40.00th=[ 146], 50.00th=[ 146], 60.00th=[ 146], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min=16384, max=16384, per=94.41%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.35%, sys=49.30%, ctx=69, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8914KiB/s (9128kB/s), 2229KiB/s-9518KiB/s (2282kB/s-9747kB/s), io=16.0MiB (16.8MB), run=1291-1838msec WRITE: bw=16.9MiB/s (17.8MB/s), 4457KiB/s-20.9MiB/s (4564kB/s-21.9MB/s), io=32.0MiB (33.6MB), run=575-1888msec