== sanity test 398b: DIO and buffer IO race ============== 17:16:22 (1730582182) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.13748 s, 44.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=16939: Sat Nov 2 17:16:36 2024 read: IOPS=131, BW=526KiB/s (539kB/s)(5992KiB/11388msec) clat (usec): min=49, max=53076, avg=4201.56, stdev=2876.44 lat (usec): min=49, max=53077, avg=4202.49, stdev=2876.73 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 92], 10.00th=[ 2040], 20.00th=[ 2442], | 30.00th=[ 2769], 40.00th=[ 3228], 50.00th=[ 3752], 60.00th=[ 4228], | 70.00th=[ 4883], 80.00th=[ 5538], 90.00th=[ 7046], 95.00th=[ 8717], | 99.00th=[12649], 99.50th=[15664], 99.90th=[36439], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 224, max= 672, per=24.74%, avg=521.27, stdev=103.23, samples=22 iops : min= 56, max= 168, avg=130.27, stdev=25.79, samples=22 write: IOPS=138, BW=553KiB/s (566kB/s)(6296KiB/11388msec) clat (usec): min=409, max=124937, avg=3172.76, stdev=4073.50 lat (usec): min=409, max=124937, avg=3173.45, stdev=4073.50 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 519], 10.00th=[ 545], 20.00th=[ 603], | 30.00th=[ 750], 40.00th=[ 1893], 50.00th=[ 2606], 60.00th=[ 3130], | 70.00th=[ 3949], 80.00th=[ 5145], 90.00th=[ 7111], 95.00th=[ 8455], | 99.00th=[ 11076], 99.50th=[ 12780], 99.90th=[ 14484], 99.95th=[125305], | 99.99th=[125305] bw ( KiB/s): min= 256, max= 832, per=24.72%, avg=545.27, stdev=99.89, samples=22 iops : min= 64, max= 208, avg=136.27, stdev=24.98, samples=22 lat (usec) : 50=0.03%, 100=2.54%, 250=0.16%, 500=1.27%, 750=14.13% lat (usec) : 1000=2.54% lat (msec) : 2=4.52%, 4=38.15%, 10=33.95%, 20=2.57%, 50=0.07% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.12%, sys=23.16%, ctx=5722, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16940: Sat Nov 2 17:16:36 2024 read: IOPS=130, BW=523KiB/s (535kB/s)(5940KiB/11361msec) clat (usec): min=46, max=172095, avg=4237.20, stdev=4943.01 lat (usec): min=46, max=172097, avg=4239.55, stdev=4943.12 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 75], 10.00th=[ 2024], 20.00th=[ 2442], | 30.00th=[ 2802], 40.00th=[ 3228], 50.00th=[ 3752], 60.00th=[ 4359], | 70.00th=[ 5014], 80.00th=[ 5735], 90.00th=[ 6783], 95.00th=[ 8094], | 99.00th=[ 11994], 99.50th=[ 13698], 99.90th=[ 30278], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 136, max= 736, per=24.60%, avg=518.41, stdev=115.24, samples=22 iops : min= 34, max= 184, avg=129.55, stdev=28.81, samples=22 write: IOPS=139, BW=559KiB/s (572kB/s)(6348KiB/11361msec) clat (usec): min=424, max=53126, avg=3131.86, stdev=3188.88 lat (usec): min=424, max=53127, avg=3132.53, stdev=3188.88 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 510], 10.00th=[ 537], 20.00th=[ 586], | 30.00th=[ 701], 40.00th=[ 1385], 50.00th=[ 2442], 60.00th=[ 2999], | 70.00th=[ 3884], 80.00th=[ 5145], 90.00th=[ 7242], 95.00th=[ 8848], | 99.00th=[12911], 99.50th=[15664], 99.90th=[20317], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 216, max= 696, per=24.75%, avg=546.00, stdev=112.36, samples=22 iops : min= 54, max= 174, avg=136.45, stdev=28.05, samples=22 lat (usec) : 50=0.23%, 100=2.47%, 250=0.13%, 500=1.76%, 750=14.75% lat (usec) : 1000=2.77% lat (msec) : 2=5.18%, 4=34.96%, 10=35.42%, 20=2.18%, 50=0.10% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.24%, sys=22.76%, ctx=5614, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16941: Sat Nov 2 17:16:36 2024 read: IOPS=132, BW=529KiB/s (542kB/s)(6028KiB/11394msec) clat (usec): min=47, max=136973, avg=4260.49, stdev=4163.49 lat (usec): min=47, max=136974, avg=4261.69, stdev=4163.49 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 93], 10.00th=[ 2089], 20.00th=[ 2474], | 30.00th=[ 2835], 40.00th=[ 3261], 50.00th=[ 3818], 60.00th=[ 4359], | 70.00th=[ 5014], 80.00th=[ 5800], 90.00th=[ 6849], 95.00th=[ 8029], | 99.00th=[ 11994], 99.50th=[ 12911], 99.90th=[ 33162], 99.95th=[137364], | 99.99th=[137364] bw ( KiB/s): min= 240, max= 696, per=24.99%, avg=526.50, stdev=91.02, samples=22 iops : min= 60, max= 174, avg=131.59, stdev=22.77, samples=22 write: IOPS=137, BW=549KiB/s (563kB/s)(6260KiB/11394msec) clat (usec): min=421, max=38938, avg=3118.18, stdev=2908.84 lat (usec): min=422, max=38939, avg=3119.17, stdev=2908.84 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 515], 10.00th=[ 537], 20.00th=[ 586], | 30.00th=[ 734], 40.00th=[ 1663], 50.00th=[ 2442], 60.00th=[ 3097], | 70.00th=[ 3949], 80.00th=[ 5276], 90.00th=[ 7111], 95.00th=[ 8586], | 99.00th=[11731], 99.50th=[13698], 99.90th=[17695], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 312, max= 712, per=24.64%, avg=543.55, stdev=89.37, samples=22 iops : min= 78, max= 178, avg=135.86, stdev=22.33, samples=22 lat (usec) : 50=0.07%, 100=2.47%, 250=0.10%, 500=1.37%, 750=14.32% lat (usec) : 1000=2.60% lat (msec) : 2=4.75%, 4=36.56%, 10=35.42%, 20=2.25%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=22.97%, ctx=5710, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16943: Sat Nov 2 17:16:36 2024 read: IOPS=133, BW=533KiB/s (545kB/s)(6052KiB/11363msec) clat (usec): min=47, max=172806, avg=4183.22, stdev=4871.33 lat (usec): min=47, max=172807, avg=4183.97, stdev=4871.29 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 105], 10.00th=[ 2040], 20.00th=[ 2409], | 30.00th=[ 2769], 40.00th=[ 3163], 50.00th=[ 3752], 60.00th=[ 4293], | 70.00th=[ 4883], 80.00th=[ 5604], 90.00th=[ 6783], 95.00th=[ 7963], | 99.00th=[ 11600], 99.50th=[ 13960], 99.90th=[ 15664], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 248, max= 744, per=24.98%, avg=526.36, stdev=99.64, samples=22 iops : min= 62, max= 186, avg=131.55, stdev=24.87, samples=22 write: IOPS=137, BW=549KiB/s (562kB/s)(6236KiB/11363msec) clat (usec): min=414, max=34950, avg=3167.80, stdev=2890.51 lat (usec): min=415, max=34950, avg=3169.24, stdev=2890.14 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 515], 10.00th=[ 537], 20.00th=[ 603], | 30.00th=[ 791], 40.00th=[ 1958], 50.00th=[ 2573], 60.00th=[ 3130], | 70.00th=[ 3949], 80.00th=[ 5342], 90.00th=[ 7046], 95.00th=[ 8586], | 99.00th=[11207], 99.50th=[13566], 99.90th=[25560], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 304, max= 784, per=24.67%, avg=544.23, stdev=95.32, samples=22 iops : min= 76, max= 196, avg=136.00, stdev=23.85, samples=22 lat (usec) : 50=0.10%, 100=2.31%, 250=0.20%, 500=1.33%, 750=13.38% lat (usec) : 1000=1.95% lat (msec) : 2=5.76%, 4=37.76%, 10=35.12%, 20=1.99%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=23.50%, ctx=5808, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2107KiB/s (2158kB/s), 523KiB/s-533KiB/s (535kB/s-545kB/s), io=23.4MiB (24.6MB), run=11361-11394msec WRITE: bw=2206KiB/s (2259kB/s), 549KiB/s-559KiB/s (562kB/s-572kB/s), io=24.6MiB (25.7MB), run=11361-11394msec rand-rw: (groupid=0, jobs=1): err= 0: pid=16942: Sat Nov 2 17:16:40 2024 read: IOPS=98, BW=392KiB/s (402kB/s)(5992KiB/15267msec) clat (usec): min=1567, max=208179, avg=4173.15, stdev=6027.00 lat (usec): min=1567, max=208181, avg=4173.82, stdev=6027.04 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1778], 10.00th=[ 1860], 20.00th=[ 2008], | 30.00th=[ 2212], 40.00th=[ 2442], 50.00th=[ 2835], 60.00th=[ 3392], | 70.00th=[ 4228], 80.00th=[ 5538], 90.00th=[ 8455], 95.00th=[ 10290], | 99.00th=[ 15008], 99.50th=[ 16450], 99.90th=[ 19792], 99.95th=[208667], | 99.99th=[208667] bw ( KiB/s): min= 152, max= 782, per=24.84%, avg=389.47, stdev=160.27, samples=30 iops : min= 38, max= 195, avg=97.30, stdev=40.03, samples=30 write: IOPS=103, BW=412KiB/s (422kB/s)(6296KiB/15267msec) clat (usec): min=2207, max=28421, avg=5662.59, stdev=3237.26 lat (usec): min=2207, max=28421, avg=5663.41, stdev=3237.30 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2769], 10.00th=[ 2966], 20.00th=[ 3228], | 30.00th=[ 3556], 40.00th=[ 3851], 50.00th=[ 4490], 60.00th=[ 5276], | 70.00th=[ 6325], 80.00th=[ 7832], 90.00th=[10028], 95.00th=[11731], | 99.00th=[17433], 99.50th=[20579], 99.90th=[23987], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 160, max= 752, per=24.93%, avg=409.17, stdev=158.48, samples=30 iops : min= 40, max= 188, avg=102.23, stdev=39.57, samples=30 lat (msec) : 2=9.34%, 4=45.74%, 10=36.75%, 20=7.88%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=11.36%, ctx=3608, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16944: Sat Nov 2 17:16:40 2024 read: IOPS=96, BW=388KiB/s (397kB/s)(5940KiB/15311msec) clat (usec): min=1442, max=175652, avg=4165.19, stdev=5320.75 lat (usec): min=1443, max=175654, avg=4165.84, stdev=5320.76 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1811], 10.00th=[ 1893], 20.00th=[ 2057], | 30.00th=[ 2212], 40.00th=[ 2474], 50.00th=[ 2835], 60.00th=[ 3359], | 70.00th=[ 4228], 80.00th=[ 5538], 90.00th=[ 8586], 95.00th=[ 10552], | 99.00th=[ 13829], 99.50th=[ 16450], 99.90th=[ 24249], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 103, max= 806, per=24.70%, avg=387.27, stdev=163.19, samples=30 iops : min= 25, max= 201, avg=96.73, stdev=40.78, samples=30 write: IOPS=103, BW=415KiB/s (425kB/s)(6348KiB/15311msec) clat (usec): min=2132, max=51091, avg=5696.04, stdev=3451.93 lat (usec): min=2132, max=51092, avg=5696.77, stdev=3451.92 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2769], 10.00th=[ 2999], 20.00th=[ 3261], | 30.00th=[ 3556], 40.00th=[ 3884], 50.00th=[ 4490], 60.00th=[ 5276], | 70.00th=[ 6390], 80.00th=[ 7767], 90.00th=[10028], 95.00th=[12125], | 99.00th=[17695], 99.50th=[21365], 99.90th=[27919], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 159, max= 768, per=24.93%, avg=409.13, stdev=151.19, samples=30 iops : min= 39, max= 192, avg=102.20, stdev=37.79, samples=30 lat (msec) : 2=8.11%, 4=46.65%, 10=37.01%, 20=7.81%, 50=0.36% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.20%, sys=11.57%, ctx=3693, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16945: Sat Nov 2 17:16:40 2024 read: IOPS=98, BW=394KiB/s (404kB/s)(6028KiB/15290msec) clat (usec): min=1505, max=24089, avg=4134.17, stdev=3011.95 lat (usec): min=1506, max=24089, avg=4134.83, stdev=3011.92 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1778], 10.00th=[ 1876], 20.00th=[ 2057], | 30.00th=[ 2245], 40.00th=[ 2540], 50.00th=[ 2900], 60.00th=[ 3490], | 70.00th=[ 4359], 80.00th=[ 5604], 90.00th=[ 8717], 95.00th=[10814], | 99.00th=[14877], 99.50th=[16450], 99.90th=[23200], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 128, max= 816, per=24.95%, avg=391.27, stdev=154.93, samples=30 iops : min= 32, max= 204, avg=97.70, stdev=38.78, samples=30 write: IOPS=102, BW=409KiB/s (419kB/s)(6260KiB/15290msec) clat (msec): min=2, max=158, avg= 5.72, stdev= 5.10 lat (msec): min=2, max=158, avg= 5.72, stdev= 5.10 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 18], 99.50th=[ 22], 99.90th=[ 44], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 176, max= 824, per=24.70%, avg=405.40, stdev=164.67, samples=30 iops : min= 44, max= 206, avg=101.23, stdev=41.17, samples=30 lat (msec) : 2=8.50%, 4=45.44%, 10=37.43%, 20=8.20%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.12%, sys=11.29%, ctx=3704, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16946: Sat Nov 2 17:16:40 2024 read: IOPS=99, BW=399KiB/s (409kB/s)(6052KiB/15159msec) clat (usec): min=1553, max=21490, avg=4031.40, stdev=3001.44 lat (usec): min=1554, max=21491, avg=4032.09, stdev=3001.38 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1778], 10.00th=[ 1860], 20.00th=[ 2008], | 30.00th=[ 2180], 40.00th=[ 2409], 50.00th=[ 2737], 60.00th=[ 3326], | 70.00th=[ 4146], 80.00th=[ 5407], 90.00th=[ 8586], 95.00th=[10683], | 99.00th=[14484], 99.50th=[17433], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 144, max= 784, per=25.30%, avg=396.63, stdev=159.52, samples=30 iops : min= 36, max= 196, avg=99.07, stdev=39.89, samples=30 write: IOPS=102, BW=411KiB/s (421kB/s)(6236KiB/15159msec) clat (msec): min=2, max=147, avg= 5.74, stdev= 4.95 lat (msec): min=2, max=147, avg= 5.74, stdev= 4.95 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 11], 95.00th=[ 12], | 99.00th=[ 18], 99.50th=[ 22], 99.90th=[ 49], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 160, max= 752, per=25.00%, avg=410.23, stdev=159.50, samples=30 iops : min= 40, max= 188, avg=102.47, stdev=39.88, samples=30 lat (msec) : 2=9.73%, 4=44.95%, 10=37.08%, 20=7.78%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=11.18%, ctx=3615, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1568KiB/s (1606kB/s), 388KiB/s-399KiB/s (397kB/s-409kB/s), io=23.4MiB (24.6MB), run=15159-15311msec WRITE: bw=1642KiB/s (1681kB/s), 409KiB/s-415KiB/s (419kB/s-425kB/s), io=24.6MiB (25.7MB), run=15159-15311msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=16976: Sat Nov 2 17:16:46 2024 read: IOPS=67, BW=1084KiB/s (1110kB/s)(5920KiB/5460msec) clat (usec): min=59, max=84754, avg=10532.07, stdev=5346.55 lat (usec): min=60, max=84755, avg=10532.79, stdev=5346.59 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 102], 10.00th=[ 7570], 20.00th=[ 8586], | 30.00th=[ 9372], 40.00th=[ 9896], 50.00th=[10290], 60.00th=[10945], | 70.00th=[11600], 80.00th=[12649], 90.00th=[13960], 95.00th=[15139], | 99.00th=[20841], 99.50th=[31327], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 830, max= 1280, per=24.08%, avg=1043.00, stdev=137.11, samples=10 iops : min= 51, max= 80, avg=65.10, stdev= 8.72, samples=10 write: IOPS=72, BW=1166KiB/s (1194kB/s)(6368KiB/5460msec) clat (usec): min=715, max=20768, avg=3891.43, stdev=3380.08 lat (usec): min=715, max=20768, avg=3892.39, stdev=3380.08 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 791], 10.00th=[ 832], 20.00th=[ 1029], | 30.00th=[ 2040], 40.00th=[ 2442], 50.00th=[ 2802], 60.00th=[ 3523], | 70.00th=[ 4752], 80.00th=[ 5800], 90.00th=[ 7767], 95.00th=[10683], | 99.00th=[18482], 99.50th=[20055], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 800, max= 1472, per=24.85%, avg=1132.60, stdev=185.70, samples=10 iops : min= 50, max= 92, avg=70.70, stdev=11.70, samples=10 lat (usec) : 100=2.34%, 250=0.52%, 750=1.04%, 1000=9.11% lat (msec) : 2=4.69%, 4=18.36%, 10=33.46%, 20=29.56%, 50=0.78% lat (msec) : 100=0.13% cpu : usr=0.04%, sys=21.23%, ctx=2625, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16978: Sat Nov 2 17:16:46 2024 read: IOPS=65, BW=1054KiB/s (1079kB/s)(5744KiB/5449msec) clat (usec): min=59, max=24455, avg=10540.98, stdev=3093.65 lat (usec): min=60, max=24455, avg=10541.61, stdev=3093.64 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 6849], 10.00th=[ 7898], 20.00th=[ 8717], | 30.00th=[ 9241], 40.00th=[ 9896], 50.00th=[10552], 60.00th=[11207], | 70.00th=[11863], 80.00th=[12518], 90.00th=[13829], 95.00th=[14746], | 99.00th=[18744], 99.50th=[20055], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 736, max= 1216, per=23.71%, avg=1027.20, stdev=138.63, samples=10 iops : min= 46, max= 76, avg=64.20, stdev= 8.66, samples=10 write: IOPS=75, BW=1201KiB/s (1230kB/s)(6544KiB/5449msec) clat (usec): min=630, max=33841, avg=3810.86, stdev=3657.38 lat (usec): min=631, max=33842, avg=3811.96, stdev=3657.34 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 807], 10.00th=[ 848], 20.00th=[ 1004], | 30.00th=[ 2008], 40.00th=[ 2343], 50.00th=[ 2737], 60.00th=[ 3195], | 70.00th=[ 4146], 80.00th=[ 5211], 90.00th=[ 8455], 95.00th=[12256], | 99.00th=[15795], 99.50th=[17171], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 896, max= 1696, per=25.48%, avg=1161.60, stdev=273.22, samples=10 iops : min= 56, max= 106, avg=72.60, stdev=17.08, samples=10 lat (usec) : 100=1.17%, 250=0.39%, 750=0.52%, 1000=10.03% lat (msec) : 2=5.47%, 4=20.18%, 10=30.34%, 20=31.51%, 50=0.39% cpu : usr=0.09%, sys=20.59%, ctx=2574, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16980: Sat Nov 2 17:16:46 2024 read: IOPS=71, BW=1145KiB/s (1172kB/s)(6256KiB/5466msec) clat (usec): min=50, max=18439, avg=10265.30, stdev=3441.54 lat (usec): min=51, max=18440, avg=10265.95, stdev=3441.55 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 92], 10.00th=[ 7767], 20.00th=[ 8717], | 30.00th=[ 9372], 40.00th=[ 9896], 50.00th=[10421], 60.00th=[11076], | 70.00th=[11600], 80.00th=[12649], 90.00th=[13960], 95.00th=[15401], | 99.00th=[17433], 99.50th=[17957], 99.90th=[18482], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 736, max= 1248, per=25.25%, avg=1093.90, stdev=155.72, samples=10 iops : min= 46, max= 78, avg=68.20, stdev= 9.72, samples=10 write: IOPS=68, BW=1104KiB/s (1130kB/s)(6032KiB/5466msec) clat (usec): min=608, max=17549, avg=3574.40, stdev=3144.19 lat (usec): min=608, max=17550, avg=3575.62, stdev=3144.16 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 783], 10.00th=[ 816], 20.00th=[ 930], | 30.00th=[ 1926], 40.00th=[ 2376], 50.00th=[ 2704], 60.00th=[ 3195], | 70.00th=[ 4080], 80.00th=[ 5080], 90.00th=[ 7242], 95.00th=[10683], | 99.00th=[16188], 99.50th=[17433], 99.90th=[17433], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 768, max= 1280, per=23.44%, avg=1068.30, stdev=176.70, samples=10 iops : min= 48, max= 80, avg=66.60, stdev=11.07, samples=10 lat (usec) : 100=2.60%, 250=0.39%, 750=1.82%, 1000=9.64% lat (msec) : 2=4.04%, 4=19.01%, 10=30.21%, 20=32.29% cpu : usr=0.07%, sys=20.11%, ctx=2673, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16982: Sat Nov 2 17:16:46 2024 read: IOPS=68, BW=1091KiB/s (1117kB/s)(6032KiB/5528msec) clat (usec): min=53, max=24845, avg=10090.56, stdev=3804.40 lat (usec): min=53, max=24845, avg=10091.18, stdev=3804.41 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 79], 10.00th=[ 7439], 20.00th=[ 8455], | 30.00th=[ 9241], 40.00th=[ 9765], 50.00th=[10290], 60.00th=[11076], | 70.00th=[11600], 80.00th=[12387], 90.00th=[13829], 95.00th=[15270], | 99.00th=[19530], 99.50th=[20579], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 704, max= 1696, per=24.97%, avg=1081.73, stdev=249.68, samples=11 iops : min= 44, max= 106, avg=67.45, stdev=15.58, samples=11 write: IOPS=70, BW=1132KiB/s (1159kB/s)(6256KiB/5528msec) clat (usec): min=701, max=24269, avg=4144.69, stdev=4087.70 lat (usec): min=702, max=24270, avg=4145.93, stdev=4087.66 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 807], 10.00th=[ 832], 20.00th=[ 979], | 30.00th=[ 1926], 40.00th=[ 2376], 50.00th=[ 2704], 60.00th=[ 3294], | 70.00th=[ 4359], 80.00th=[ 6194], 90.00th=[10028], 95.00th=[12911], | 99.00th=[21103], 99.50th=[23462], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 800, max= 1824, per=24.82%, avg=1131.09, stdev=278.53, samples=11 iops : min= 50, max= 114, avg=70.55, stdev=17.47, samples=11 lat (usec) : 100=3.26%, 250=0.52%, 750=0.26%, 1000=10.68% lat (msec) : 2=4.95%, 4=18.49%, 10=29.04%, 20=31.90%, 50=0.91% cpu : usr=0.11%, sys=19.94%, ctx=2576, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4333KiB/s (4437kB/s), 1054KiB/s-1145KiB/s (1079kB/s-1172kB/s), io=23.4MiB (24.5MB), run=5449-5528msec WRITE: bw=4559KiB/s (4668kB/s), 1104KiB/s-1201KiB/s (1130kB/s-1230kB/s), io=24.6MiB (25.8MB), run=5449-5528msec rand-rw: (groupid=0, jobs=1): err= 0: pid=16975: Sat Nov 2 17:16:46 2024 read: IOPS=67, BW=1077KiB/s (1102kB/s)(5920KiB/5499msec) clat (usec): min=1836, max=50148, avg=5671.33, stdev=4869.25 lat (usec): min=1836, max=50149, avg=5671.98, stdev=4869.26 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2040], 10.00th=[ 2147], 20.00th=[ 2343], | 30.00th=[ 2638], 40.00th=[ 3032], 50.00th=[ 3621], 60.00th=[ 4883], | 70.00th=[ 6718], 80.00th=[ 8848], 90.00th=[11600], 95.00th=[13698], | 99.00th=[22414], 99.50th=[35914], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 830, max= 1248, per=24.10%, avg=1042.20, stdev=151.82, samples=10 iops : min= 51, max= 78, avg=65.00, stdev= 9.57, samples=10 write: IOPS=72, BW=1158KiB/s (1186kB/s)(6368KiB/5499msec) clat (usec): min=2307, max=24209, avg=8405.64, stdev=4179.76 lat (usec): min=2308, max=24209, avg=8406.73, stdev=4179.79 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 3195], 10.00th=[ 3654], 20.00th=[ 4817], | 30.00th=[ 5604], 40.00th=[ 6390], 50.00th=[ 7570], 60.00th=[ 8848], | 70.00th=[10159], 80.00th=[11469], 90.00th=[14615], 95.00th=[16712], | 99.00th=[20841], 99.50th=[21627], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 896, max= 1376, per=24.89%, avg=1132.30, stdev=163.05, samples=10 iops : min= 56, max= 86, avg=70.60, stdev=10.08, samples=10 lat (msec) : 2=1.95%, 4=31.77%, 10=43.75%, 20=21.22%, 50=1.17% lat (msec) : 100=0.13% cpu : usr=0.02%, sys=10.77%, ctx=1181, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16977: Sat Nov 2 17:16:46 2024 read: IOPS=76, BW=1217KiB/s (1246kB/s)(5744KiB/4720msec) clat (usec): min=1758, max=23286, avg=4787.06, stdev=3201.60 lat (usec): min=1758, max=23287, avg=4787.69, stdev=3201.58 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 2008], 10.00th=[ 2147], 20.00th=[ 2409], | 30.00th=[ 2737], 40.00th=[ 3097], 50.00th=[ 3458], 60.00th=[ 4228], | 70.00th=[ 5145], 80.00th=[ 7046], 90.00th=[ 9503], 95.00th=[11207], | 99.00th=[15926], 99.50th=[18220], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 864, max= 1504, per=28.28%, avg=1223.11, stdev=218.08, samples=9 iops : min= 54, max= 94, avg=76.44, stdev=13.63, samples=9 write: IOPS=86, BW=1386KiB/s (1420kB/s)(6544KiB/4720msec) clat (usec): min=2486, max=99669, avg=7308.63, stdev=6380.92 lat (usec): min=2488, max=99671, avg=7309.81, stdev=6380.94 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 10], 90.00th=[ 14], 95.00th=[ 17], | 99.00th=[ 23], 99.50th=[ 24], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 1120, max= 1728, per=30.01%, avg=1365.33, stdev=190.66, samples=9 iops : min= 70, max= 108, avg=85.33, stdev=11.92, samples=9 lat (msec) : 2=2.21%, 4=39.58%, 10=43.88%, 20=13.15%, 50=1.04% lat (msec) : 100=0.13% cpu : usr=0.08%, sys=10.17%, ctx=913, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16979: Sat Nov 2 17:16:46 2024 read: IOPS=70, BW=1130KiB/s (1157kB/s)(6256KiB/5538msec) clat (usec): min=1642, max=28702, avg=6051.50, stdev=4346.03 lat (usec): min=1642, max=28703, avg=6052.10, stdev=4346.04 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 2073], 10.00th=[ 2278], 20.00th=[ 2573], | 30.00th=[ 2999], 40.00th=[ 3621], 50.00th=[ 4293], 60.00th=[ 5604], | 70.00th=[ 7635], 80.00th=[ 9241], 90.00th=[11863], 95.00th=[13960], | 99.00th=[22938], 99.50th=[27132], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 704, max= 1420, per=25.33%, avg=1095.60, stdev=198.72, samples=10 iops : min= 44, max= 88, avg=68.40, stdev=12.29, samples=10 write: IOPS=68, BW=1089KiB/s (1115kB/s)(6032KiB/5538msec) clat (usec): min=2477, max=48735, avg=8250.51, stdev=4513.72 lat (usec): min=2478, max=48739, avg=8251.67, stdev=4513.83 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3163], 10.00th=[ 3589], 20.00th=[ 4293], | 30.00th=[ 5473], 40.00th=[ 6521], 50.00th=[ 7701], 60.00th=[ 8717], | 70.00th=[ 9765], 80.00th=[10814], 90.00th=[13304], 95.00th=[16712], | 99.00th=[22152], 99.50th=[23200], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 768, max= 1248, per=23.29%, avg=1059.60, stdev=168.03, samples=10 iops : min= 48, max= 78, avg=66.20, stdev=10.52, samples=10 lat (msec) : 2=1.69%, 4=29.43%, 10=47.66%, 20=20.05%, 50=1.17% cpu : usr=0.13%, sys=10.69%, ctx=1194, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16981: Sat Nov 2 17:16:46 2024 read: IOPS=83, BW=1340KiB/s (1373kB/s)(6032KiB/4500msec) clat (usec): min=1758, max=19681, avg=4698.77, stdev=3210.61 lat (usec): min=1759, max=19682, avg=4699.47, stdev=3210.56 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 1991], 10.00th=[ 2114], 20.00th=[ 2376], | 30.00th=[ 2606], 40.00th=[ 2900], 50.00th=[ 3228], 60.00th=[ 4178], | 70.00th=[ 5014], 80.00th=[ 7111], 90.00th=[ 9765], 95.00th=[11469], | 99.00th=[15139], 99.50th=[19268], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 928, max= 1664, per=30.24%, avg=1308.00, stdev=264.67, samples=8 iops : min= 58, max= 104, avg=81.75, stdev=16.54, samples=8 write: IOPS=86, BW=1390KiB/s (1424kB/s)(6256KiB/4500msec) clat (usec): min=2272, max=59619, avg=6853.24, stdev=5002.76 lat (usec): min=2273, max=59622, avg=6854.47, stdev=5002.89 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 3032], 10.00th=[ 3195], 20.00th=[ 3556], | 30.00th=[ 3916], 40.00th=[ 4424], 50.00th=[ 5080], 60.00th=[ 5932], | 70.00th=[ 7242], 80.00th=[ 9503], 90.00th=[12649], 95.00th=[16581], | 99.00th=[22152], 99.50th=[24511], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 1184, max= 1440, per=29.98%, avg=1364.00, stdev=78.27, samples=8 iops : min= 74, max= 90, avg=85.25, stdev= 4.89, samples=8 lat (msec) : 2=2.60%, 4=43.10%, 10=40.49%, 20=12.63%, 50=1.04% lat (msec) : 100=0.13% cpu : usr=0.13%, sys=10.67%, ctx=915, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4325KiB/s (4429kB/s), 1077KiB/s-1340KiB/s (1102kB/s-1373kB/s), io=23.4MiB (24.5MB), run=4500-5538msec WRITE: bw=4550KiB/s (4660kB/s), 1089KiB/s-1390KiB/s (1115kB/s-1424kB/s), io=24.6MiB (25.8MB), run=4500-5538msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=16992: Sat Nov 2 17:16:50 2024 read: IOPS=5, BW=5505KiB/s (5638kB/s)(7168KiB/1302msec) clat (usec): min=14290, max=49428, avg=25745.18, stdev=12120.32 lat (usec): min=14291, max=49429, avg=25746.38, stdev=12120.30 clat percentiles (usec): | 1.00th=[14353], 5.00th=[14353], 10.00th=[14353], 20.00th=[14746], | 30.00th=[19792], 40.00th=[19792], 50.00th=[23987], 60.00th=[26608], | 70.00th=[26608], 80.00th=[31327], 90.00th=[49546], 95.00th=[49546], | 99.00th=[49546], 99.50th=[49546], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3932KiB/s (4027kB/s)(5120KiB/1302msec) clat (msec): min=19, max=846, avg=218.90, stdev=353.35 lat (msec): min=19, max=846, avg=218.96, stdev=353.33 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 20], | 30.00th=[ 25], 40.00th=[ 25], 50.00th=[ 77], 60.00th=[ 77], | 70.00th=[ 128], 80.00th=[ 128], 90.00th=[ 844], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 4096, max= 4096, per=35.26%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=41.67%, 100=8.33%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=8.83%, ctx=64, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16993: Sat Nov 2 17:16:50 2024 read: IOPS=0, BW=830KiB/s (850kB/s)(2048KiB/2468msec) clat (usec): min=18649, max=26000, avg=22325.11, stdev=5198.42 lat (usec): min=18650, max=26002, avg=22326.26, stdev=5198.66 clat percentiles (usec): | 1.00th=[18744], 5.00th=[18744], 10.00th=[18744], 20.00th=[18744], | 30.00th=[18744], 40.00th=[18744], 50.00th=[18744], 60.00th=[26084], | 70.00th=[26084], 80.00th=[26084], 90.00th=[26084], 95.00th=[26084], | 99.00th=[26084], 99.50th=[26084], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 1851, max= 2048, per=23.49%, avg=1949.50, stdev=139.30, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4149KiB/s (4249kB/s)(10.0MiB/2468msec) clat (msec): min=26, max=927, avg=239.22, stdev=360.63 lat (msec): min=26, max=927, avg=239.27, stdev=360.61 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 53], | 30.00th=[ 65], 40.00th=[ 68], 50.00th=[ 74], 60.00th=[ 74], | 70.00th=[ 106], 80.00th=[ 106], 90.00th=[ 919], 95.00th=[ 927], | 99.00th=[ 927], 99.50th=[ 927], 99.90th=[ 927], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 5555, max= 6144, per=50.35%, avg=5849.50, stdev=416.49, samples=2 iops : min= 5, max= 6, avg= 5.50, stdev= 0.71, samples=2 lat (msec) : 20=8.33%, 50=16.67%, 100=50.00%, 250=8.33%, 1000=16.67% cpu : usr=0.04%, sys=5.67%, ctx=86, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16994: Sat Nov 2 17:16:50 2024 read: IOPS=4, BW=5036KiB/s (5157kB/s)(6144KiB/1220msec) clat (usec): min=13918, max=70407, avg=40139.63, stdev=22367.15 lat (usec): min=13919, max=70408, avg=40140.57, stdev=22367.29 clat percentiles (usec): | 1.00th=[13960], 5.00th=[13960], 10.00th=[13960], 20.00th=[15270], | 30.00th=[15270], 40.00th=[38011], 50.00th=[38011], 60.00th=[50594], | 70.00th=[52691], 80.00th=[52691], 90.00th=[70779], 95.00th=[70779], | 99.00th=[70779], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=4, BW=5036KiB/s (5157kB/s)(6144KiB/1220msec) clat (msec): min=24, max=753, avg=158.06, stdev=292.04 lat (msec): min=24, max=753, avg=158.10, stdev=292.03 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 37], 50.00th=[ 37], 60.00th=[ 48], | 70.00th=[ 55], 80.00th=[ 55], 90.00th=[ 751], 95.00th=[ 751], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min=10240, max=10240, per=88.15%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=41.67%, 100=33.33%, 1000=8.33% cpu : usr=0.00%, sys=7.71%, ctx=34, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16996: Sat Nov 2 17:16:50 2024 read: IOPS=3, BW=4044KiB/s (4141kB/s)(5120KiB/1266msec) clat (msec): min=13, max=774, avg=177.94, stdev=333.68 lat (msec): min=13, max=774, avg=177.94, stdev=333.68 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 14], | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 29], 60.00th=[ 29], | 70.00th=[ 56], 80.00th=[ 56], 90.00th=[ 776], 95.00th=[ 776], | 99.00th=[ 776], 99.50th=[ 776], 99.90th=[ 776], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 6131, max= 6131, per=73.89%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5662KiB/s (5798kB/s)(7168KiB/1266msec) clat (usec): min=27133, max=88441, avg=50448.83, stdev=23079.82 lat (usec): min=27163, max=88521, avg=50522.56, stdev=23096.16 clat percentiles (usec): | 1.00th=[27132], 5.00th=[27132], 10.00th=[27132], 20.00th=[28967], | 30.00th=[33424], 40.00th=[33424], 50.00th=[49546], 60.00th=[54789], | 70.00th=[54789], 80.00th=[70779], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 8175, max= 8175, per=70.37%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=41.67%, 100=33.33%, 1000=8.33% cpu : usr=0.00%, sys=7.98%, ctx=58, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8298KiB/s (8497kB/s), 830KiB/s-5505KiB/s (850kB/s-5638kB/s), io=20.0MiB (20.0MB), run=1220-2468msec WRITE: bw=11.3MiB/s (11.9MB/s), 3932KiB/s-5662KiB/s (4027kB/s-5798kB/s), io=28.0MiB (29.4MB), run=1220-2468msec rand-rw: (groupid=0, jobs=1): err= 0: pid=16995: Sat Nov 2 17:16:50 2024 read: IOPS=2, BW=3065KiB/s (3138kB/s)(7168KiB/2339msec) clat (usec): min=338, max=824372, avg=158148.80, stdev=303897.49 lat (usec): min=339, max=824373, avg=158149.82, stdev=303897.42 clat percentiles (usec): | 1.00th=[ 338], 5.00th=[ 338], 10.00th=[ 338], 20.00th=[ 392], | 30.00th=[ 486], 40.00th=[ 486], 50.00th=[ 21890], 60.00th=[ 42206], | 70.00th=[ 42206], 80.00th=[217056], 90.00th=[826278], 95.00th=[826278], | 99.00th=[826278], 99.50th=[826278], 99.90th=[826278], 99.95th=[826278], | 99.99th=[826278] bw ( KiB/s): min= 2048, max= 4096, per=37.05%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2189KiB/s (2242kB/s)(5120KiB/2339msec) clat (msec): min=17, max=1002, avg=228.74, stdev=433.01 lat (msec): min=17, max=1002, avg=228.77, stdev=433.04 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 27], 40.00th=[ 27], 50.00th=[ 39], 60.00th=[ 39], | 70.00th=[ 60], 80.00th=[ 60], 90.00th=[ 1003], 95.00th=[ 1003], | 99.00th=[ 1003], 99.50th=[ 1003], 99.90th=[ 1003], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 2048, max= 2048, per=17.64%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 500=25.00% lat (msec) : 20=8.33%, 50=33.33%, 100=8.33%, 250=8.33%, 1000=8.33% cpu : usr=0.09%, sys=19.85%, ctx=360, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16997: Sat Nov 2 17:16:50 2024 read: IOPS=0, BW=829KiB/s (849kB/s)(2048KiB/2470msec) clat (msec): min=48, max=194, avg=121.33, stdev=103.19 lat (msec): min=48, max=194, avg=121.33, stdev=103.19 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 49], 10.00th=[ 49], 20.00th=[ 49], | 30.00th=[ 49], 40.00th=[ 49], 50.00th=[ 49], 60.00th=[ 194], | 70.00th=[ 194], 80.00th=[ 194], 90.00th=[ 194], 95.00th=[ 194], | 99.00th=[ 194], 99.50th=[ 194], 99.90th=[ 194], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 2048, max= 2048, per=24.70%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4146KiB/s (4245kB/s)(10.0MiB/2470msec) clat (msec): min=21, max=1584, avg=214.95, stdev=482.73 lat (msec): min=21, max=1584, avg=215.01, stdev=482.73 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 27], | 30.00th=[ 44], 40.00th=[ 50], 50.00th=[ 53], 60.00th=[ 61], | 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 155], 95.00th=[ 1586], | 99.00th=[ 1586], 99.50th=[ 1586], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 4096, max= 6144, per=44.11%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 50=41.67%, 100=33.33%, 250=16.67% cpu : usr=0.04%, sys=17.42%, ctx=109, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16998: Sat Nov 2 17:16:50 2024 read: IOPS=2, BW=3015KiB/s (3087kB/s)(6144KiB/2038msec) clat (usec): min=333, max=1521.3k, avg=299994.52, stdev=599415.16 lat (usec): min=334, max=1521.3k, avg=299995.52, stdev=599415.17 clat percentiles (usec): | 1.00th=[ 334], 5.00th=[ 334], 10.00th=[ 334], | 20.00th=[ 44827], 30.00th=[ 44827], 40.00th=[ 45876], | 50.00th=[ 45876], 60.00th=[ 79168], 70.00th=[ 109577], | 80.00th=[ 109577], 90.00th=[1518339], 95.00th=[1518339], | 99.00th=[1518339], 99.50th=[1518339], 99.90th=[1518339], | 99.95th=[1518339], 99.99th=[1518339] bw ( KiB/s): min= 4096, max= 6144, per=61.75%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=2, BW=3015KiB/s (3087kB/s)(6144KiB/2038msec) clat (usec): min=18415, max=45071, avg=27118.74, stdev=10162.69 lat (usec): min=18436, max=45212, avg=27165.32, stdev=10210.76 clat percentiles (usec): | 1.00th=[18482], 5.00th=[18482], 10.00th=[18482], 20.00th=[18482], | 30.00th=[18482], 40.00th=[21890], 50.00th=[21890], 60.00th=[27919], | 70.00th=[31065], 80.00th=[31065], 90.00th=[44827], 95.00th=[44827], | 99.00th=[44827], 99.50th=[44827], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 4096, max= 8192, per=52.93%, avg=6144.00, stdev=2896.31, samples=2 iops : min= 4, max= 8, avg= 6.00, stdev= 2.83, samples=2 lat (usec) : 500=8.33% lat (msec) : 20=16.67%, 50=50.00%, 100=8.33%, 250=8.33% cpu : usr=0.05%, sys=21.40%, ctx=343, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16999: Sat Nov 2 17:16:50 2024 read: IOPS=2, BW=2488KiB/s (2548kB/s)(5120KiB/2058msec) clat (usec): min=438, max=778720, avg=335921.88, stdev=401463.85 lat (usec): min=439, max=778721, avg=335923.17, stdev=401463.71 clat percentiles (usec): | 1.00th=[ 441], 5.00th=[ 441], 10.00th=[ 441], 20.00th=[ 441], | 30.00th=[ 40109], 40.00th=[ 40109], 50.00th=[ 90702], 60.00th=[ 90702], | 70.00th=[767558], 80.00th=[767558], 90.00th=[775947], 95.00th=[775947], | 99.00th=[775947], 99.50th=[775947], 99.90th=[775947], 99.95th=[775947], | 99.99th=[775947] bw ( KiB/s): min= 2048, max= 6144, per=49.40%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=3, BW=3483KiB/s (3567kB/s)(7168KiB/2058msec) clat (usec): min=30415, max=66427, avg=49459.59, stdev=14105.26 lat (usec): min=30484, max=66461, avg=49702.73, stdev=14124.72 clat percentiles (usec): | 1.00th=[30540], 5.00th=[30540], 10.00th=[30540], 20.00th=[35914], | 30.00th=[41157], 40.00th=[41157], 50.00th=[49021], 60.00th=[58983], | 70.00th=[58983], 80.00th=[63701], 90.00th=[66323], 95.00th=[66323], | 99.00th=[66323], 99.50th=[66323], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 6144, max= 8192, per=61.75%, avg=7168.00, stdev=1448.15, samples=2 iops : min= 6, max= 8, avg= 7.00, stdev= 1.41, samples=2 lat (usec) : 500=8.33% lat (msec) : 50=41.67%, 100=33.33%, 1000=16.67% cpu : usr=0.10%, sys=29.95%, ctx=720, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8291KiB/s (8490kB/s), 829KiB/s-3065KiB/s (849kB/s-3138kB/s), io=20.0MiB (20.0MB), run=2038-2470msec WRITE: bw=11.3MiB/s (11.9MB/s), 2189KiB/s-4146KiB/s (2242kB/s-4245kB/s), io=28.0MiB (29.4MB), run=2038-2470msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17007: Sat Nov 2 17:16:52 2024 read: IOPS=4, BW=19.8MiB/s (20.8MB/s)(12.0MiB/605msec) clat (msec): min=35, max=311, avg=178.92, stdev=138.03 lat (msec): min=35, max=311, avg=178.92, stdev=138.03 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 190], 50.00th=[ 190], 60.00th=[ 190], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 313], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min=16384, max=16384, per=88.20%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.17%, sys=51.16%, ctx=179, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17009: Sat Nov 2 17:16:52 2024 read: IOPS=1, BW=4644KiB/s (4755kB/s)(4096KiB/882msec) clat (nsec): min=130123k, max=130123k, avg=130123081.00, stdev= 0.00 lat (nsec): min=130124k, max=130124k, avg=130124366.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 131], 10.00th=[ 131], 20.00th=[ 131], | 30.00th=[ 131], 40.00th=[ 131], 50.00th=[ 131], 60.00th=[ 131], | 70.00th=[ 131], 80.00th=[ 131], 90.00th=[ 131], 95.00th=[ 131], | 99.00th=[ 131], 99.50th=[ 131], 99.90th=[ 131], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 8192, max= 8192, per=44.10%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9288KiB/s (9511kB/s)(8192KiB/882msec) clat (msec): min=104, max=605, avg=354.87, stdev=354.00 lat (msec): min=104, max=605, avg=355.02, stdev=354.03 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 105], 10.00th=[ 105], 20.00th=[ 105], | 30.00th=[ 105], 40.00th=[ 105], 50.00th=[ 105], 60.00th=[ 609], | 70.00th=[ 609], 80.00th=[ 609], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=33.71%, ctx=134, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17011: Sat Nov 2 17:16:52 2024 write: IOPS=3, BW=13.5MiB/s (14.2MB/s)(12.0MiB/887msec) clat (msec): min=159, max=409, avg=292.52, stdev=125.80 lat (msec): min=159, max=409, avg=292.71, stdev=125.73 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 409], 80.00th=[ 409], 90.00th=[ 409], 95.00th=[ 409], | 99.00th=[ 409], 99.50th=[ 409], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min=16351, max=16351, per=44.26%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=26.19%, ctx=30, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17013: Sat Nov 2 17:16:52 2024 write: IOPS=5, BW=22.6MiB/s (23.7MB/s)(12.0MiB/532msec) clat (msec): min=105, max=241, avg=163.65, stdev=70.36 lat (msec): min=106, max=242, avg=163.92, stdev=70.34 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 106], 10.00th=[ 106], 20.00th=[ 106], | 30.00th=[ 106], 40.00th=[ 144], 50.00th=[ 144], 60.00th=[ 144], | 70.00th=[ 243], 80.00th=[ 243], 90.00th=[ 243], 95.00th=[ 243], | 99.00th=[ 243], 99.50th=[ 243], 99.90th=[ 243], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min=16384, max=16384, per=44.35%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=47.46%, ctx=50, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.1MiB/s (19.0MB/s), 4644KiB/s-19.8MiB/s (4755kB/s-20.8MB/s), io=16.0MiB (16.8MB), run=605-882msec WRITE: bw=36.1MiB/s (37.8MB/s), 9288KiB/s-22.6MiB/s (9511kB/s-23.7MB/s), io=32.0MiB (33.6MB), run=532-887msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17008: Sat Nov 2 17:16:52 2024 read: IOPS=2, BW=11.9MiB/s (12.4MB/s)(12.0MiB/1012msec) clat (msec): min=263, max=362, avg=314.85, stdev=49.15 lat (msec): min=263, max=362, avg=314.85, stdev=49.15 clat percentiles (msec): | 1.00th=[ 264], 5.00th=[ 264], 10.00th=[ 264], 20.00th=[ 264], | 30.00th=[ 264], 40.00th=[ 317], 50.00th=[ 317], 60.00th=[ 317], | 70.00th=[ 363], 80.00th=[ 363], 90.00th=[ 363], 95.00th=[ 363], | 99.00th=[ 363], 99.50th=[ 363], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 8192, max= 8192, per=57.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=100.00% cpu : usr=0.10%, sys=7.12%, ctx=31, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17010: Sat Nov 2 17:16:52 2024 read: IOPS=0, BW=3574KiB/s (3660kB/s)(4096KiB/1146msec) clat (nsec): min=235914k, max=235914k, avg=235914322.00, stdev= 0.00 lat (nsec): min=235917k, max=235917k, avg=235916593.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 236], 10.00th=[ 236], 20.00th=[ 236], | 30.00th=[ 236], 40.00th=[ 236], 50.00th=[ 236], 60.00th=[ 236], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 236], 95.00th=[ 236], | 99.00th=[ 236], 99.50th=[ 236], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 8175, max= 8175, per=57.18%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=7148KiB/s (7320kB/s)(8192KiB/1146msec) clat (msec): min=76, max=762, avg=419.08, stdev=484.98 lat (msec): min=76, max=762, avg=419.38, stdev=484.86 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 77], 10.00th=[ 77], 20.00th=[ 77], | 30.00th=[ 77], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 760], | 70.00th=[ 760], 80.00th=[ 760], 90.00th=[ 760], 95.00th=[ 760], | 99.00th=[ 760], 99.50th=[ 760], 99.90th=[ 760], 99.95th=[ 760], | 99.99th=[ 760] lat (msec) : 100=33.33%, 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=9.69%, ctx=191, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17012: Sat Nov 2 17:16:52 2024 write: IOPS=3, BW=12.1MiB/s (12.7MB/s)(12.0MiB/993msec) clat (msec): min=271, max=423, avg=324.31, stdev=85.88 lat (msec): min=271, max=423, avg=324.61, stdev=85.90 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 271], 10.00th=[ 271], 20.00th=[ 271], | 30.00th=[ 271], 40.00th=[ 279], 50.00th=[ 279], 60.00th=[ 279], | 70.00th=[ 422], 80.00th=[ 422], 90.00th=[ 422], 95.00th=[ 422], | 99.00th=[ 422], 99.50th=[ 422], 99.90th=[ 422], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 8192, max= 8192, per=28.65%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=100.00% cpu : usr=0.00%, sys=18.65%, ctx=92, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17014: Sat Nov 2 17:16:52 2024 write: IOPS=3, BW=15.5MiB/s (16.2MB/s)(12.0MiB/776msec) clat (msec): min=101, max=345, avg=239.35, stdev=124.78 lat (msec): min=102, max=345, avg=239.56, stdev=124.78 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 103], 10.00th=[ 103], 20.00th=[ 103], | 30.00th=[ 103], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 8192, max= 8192, per=28.65%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=16.90%, ctx=248, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.0MiB/s (14.6MB/s), 3574KiB/s-11.9MiB/s (3660kB/s-12.4MB/s), io=16.0MiB (16.8MB), run=1012-1146msec WRITE: bw=27.9MiB/s (29.3MB/s), 7148KiB/s-15.5MiB/s (7320kB/s-16.2MB/s), io=32.0MiB (33.6MB), run=776-1146msec