== sanity test 398b: DIO and buffer IO race ============== 12:55:40 (1729097740) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.21868 s, 41.3 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12249: Wed Oct 16 12:55:55 2024 read: IOPS=127, BW=509KiB/s (521kB/s)(5992KiB/11767msec) clat (usec): min=47, max=40061, avg=4273.86, stdev=2563.47 lat (usec): min=47, max=40062, avg=4274.69, stdev=2563.35 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 83], 10.00th=[ 2057], 20.00th=[ 2442], | 30.00th=[ 2835], 40.00th=[ 3359], 50.00th=[ 3884], 60.00th=[ 4490], | 70.00th=[ 5145], 80.00th=[ 5932], 90.00th=[ 7111], 95.00th=[ 8291], | 99.00th=[13042], 99.50th=[14353], 99.90th=[16057], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 248, max= 680, per=24.91%, avg=503.13, stdev=80.45, samples=23 iops : min= 62, max= 170, avg=125.70, stdev=20.13, samples=23 write: IOPS=133, BW=535KiB/s (548kB/s)(6296KiB/11767msec) clat (usec): min=422, max=160767, avg=3355.74, stdev=4935.76 lat (usec): min=423, max=160767, avg=3357.25, stdev=4935.54 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 627], | 30.00th=[ 783], 40.00th=[ 1663], 50.00th=[ 2606], 60.00th=[ 3228], | 70.00th=[ 4178], 80.00th=[ 5604], 90.00th=[ 7373], 95.00th=[ 8979], | 99.00th=[ 12125], 99.50th=[ 14615], 99.90th=[ 21365], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 296, max= 696, per=25.04%, avg=529.57, stdev=87.54, samples=23 iops : min= 74, max= 174, avg=132.30, stdev=21.92, samples=23 lat (usec) : 50=0.10%, 100=2.51%, 250=0.29%, 500=0.65%, 750=14.06% lat (usec) : 1000=3.03% lat (msec) : 2=5.21%, 4=34.28%, 10=36.88%, 20=2.90%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=22.80%, ctx=5735, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12250: Wed Oct 16 12:55:55 2024 read: IOPS=124, BW=500KiB/s (512kB/s)(5940KiB/11883msec) clat (usec): min=48, max=32245, avg=4379.14, stdev=2613.99 lat (usec): min=48, max=32245, avg=4381.98, stdev=2614.71 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 119], 10.00th=[ 2089], 20.00th=[ 2507], | 30.00th=[ 2900], 40.00th=[ 3359], 50.00th=[ 3949], 60.00th=[ 4555], | 70.00th=[ 5211], 80.00th=[ 6063], 90.00th=[ 7308], 95.00th=[ 8455], | 99.00th=[13435], 99.50th=[15795], 99.90th=[24249], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 248, max= 624, per=24.34%, avg=491.70, stdev=80.38, samples=23 iops : min= 62, max= 156, avg=122.83, stdev=20.12, samples=23 write: IOPS=133, BW=534KiB/s (547kB/s)(6348KiB/11883msec) clat (usec): min=451, max=173180, avg=3343.39, stdev=5118.50 lat (usec): min=452, max=173181, avg=3344.64, stdev=5118.83 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 603], | 30.00th=[ 742], 40.00th=[ 1991], 50.00th=[ 2704], 60.00th=[ 3359], | 70.00th=[ 4146], 80.00th=[ 5407], 90.00th=[ 7373], 95.00th=[ 8979], | 99.00th=[ 12256], 99.50th=[ 13435], 99.90th=[ 16909], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 288, max= 648, per=24.56%, avg=519.43, stdev=84.45, samples=23 iops : min= 72, max= 162, avg=129.78, stdev=21.14, samples=23 lat (usec) : 50=0.07%, 100=2.31%, 250=0.07%, 500=0.49%, 750=15.10% lat (usec) : 1000=2.80% lat (msec) : 2=4.04%, 4=35.19%, 10=37.14%, 20=2.70%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=22.71%, ctx=5790, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12251: Wed Oct 16 12:55:55 2024 read: IOPS=128, BW=514KiB/s (526kB/s)(6028KiB/11729msec) clat (usec): min=48, max=146279, avg=4430.69, stdev=4500.43 lat (usec): min=48, max=146280, avg=4432.29, stdev=4501.11 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 78], 10.00th=[ 2089], 20.00th=[ 2474], | 30.00th=[ 2868], 40.00th=[ 3326], 50.00th=[ 3884], 60.00th=[ 4555], | 70.00th=[ 5211], 80.00th=[ 5932], 90.00th=[ 7177], 95.00th=[ 8586], | 99.00th=[ 14091], 99.50th=[ 16319], 99.90th=[ 29492], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 264, max= 696, per=25.20%, avg=509.09, stdev=80.14, samples=23 iops : min= 66, max= 174, avg=127.22, stdev=20.05, samples=23 write: IOPS=133, BW=534KiB/s (547kB/s)(6260KiB/11729msec) clat (usec): min=410, max=25104, avg=3181.77, stdev=2922.79 lat (usec): min=411, max=25105, avg=3182.42, stdev=2922.80 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 603], | 30.00th=[ 717], 40.00th=[ 1369], 50.00th=[ 2540], 60.00th=[ 3195], | 70.00th=[ 4080], 80.00th=[ 5342], 90.00th=[ 7308], 95.00th=[ 8979], | 99.00th=[11731], 99.50th=[13960], 99.90th=[19268], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 320, max= 720, per=24.89%, avg=526.48, stdev=95.20, samples=23 iops : min= 80, max= 180, avg=131.57, stdev=23.82, samples=23 lat (usec) : 50=0.16%, 100=2.54%, 250=0.20%, 500=0.68%, 750=15.20% lat (usec) : 1000=2.64% lat (msec) : 2=4.39%, 4=34.83%, 10=36.46%, 20=2.77%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=23.14%, ctx=5779, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12252: Wed Oct 16 12:55:55 2024 read: IOPS=128, BW=515KiB/s (527kB/s)(6052KiB/11756msec) clat (usec): min=48, max=175886, avg=4435.16, stdev=5013.12 lat (usec): min=49, max=175886, avg=4436.29, stdev=5013.41 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 1631], 10.00th=[ 2147], 20.00th=[ 2507], | 30.00th=[ 2900], 40.00th=[ 3359], 50.00th=[ 3851], 60.00th=[ 4490], | 70.00th=[ 5145], 80.00th=[ 5997], 90.00th=[ 7177], 95.00th=[ 8455], | 99.00th=[ 12518], 99.50th=[ 14222], 99.90th=[ 18482], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 280, max= 664, per=25.21%, avg=509.26, stdev=74.66, samples=23 iops : min= 70, max= 166, avg=127.22, stdev=18.63, samples=23 write: IOPS=132, BW=530KiB/s (543kB/s)(6236KiB/11756msec) clat (usec): min=438, max=18147, avg=3187.55, stdev=2952.31 lat (usec): min=439, max=18148, avg=3190.21, stdev=2954.81 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 603], | 30.00th=[ 725], 40.00th=[ 1369], 50.00th=[ 2507], 60.00th=[ 3097], | 70.00th=[ 4080], 80.00th=[ 5604], 90.00th=[ 7242], 95.00th=[ 8717], | 99.00th=[12780], 99.50th=[15795], 99.90th=[18220], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 360, max= 816, per=24.87%, avg=525.96, stdev=91.08, samples=23 iops : min= 90, max= 204, avg=131.39, stdev=22.74, samples=23 lat (usec) : 50=0.03%, 100=2.21%, 250=0.20%, 500=0.72%, 750=15.10% lat (usec) : 1000=2.99% lat (msec) : 2=4.39%, 4=35.22%, 10=36.20%, 20=2.90%, 250=0.03% cpu : usr=0.15%, sys=22.95%, ctx=5771, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2021KiB/s (2069kB/s), 500KiB/s-515KiB/s (512kB/s-527kB/s), io=23.4MiB (24.6MB), run=11729-11883msec WRITE: bw=2116KiB/s (2166kB/s), 530KiB/s-535KiB/s (543kB/s-548kB/s), io=24.6MiB (25.7MB), run=11729-11883msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12253: Wed Oct 16 12:56:02 2024 read: IOPS=76, BW=305KiB/s (313kB/s)(5992KiB/19615msec) clat (usec): min=1504, max=197360, avg=5313.03, stdev=5832.51 lat (usec): min=1505, max=197362, avg=5313.82, stdev=5832.52 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 2147], 10.00th=[ 2442], 20.00th=[ 2933], | 30.00th=[ 3425], 40.00th=[ 3818], 50.00th=[ 4047], 60.00th=[ 4555], | 70.00th=[ 5669], 80.00th=[ 7439], 90.00th=[ 9241], 95.00th=[ 11469], | 99.00th=[ 16909], 99.50th=[ 18220], 99.90th=[ 23987], 99.95th=[198181], | 99.99th=[198181] bw ( KiB/s): min= 136, max= 504, per=24.91%, avg=304.87, stdev=77.97, samples=39 iops : min= 34, max= 126, avg=76.10, stdev=19.49, samples=39 write: IOPS=80, BW=321KiB/s (329kB/s)(6296KiB/19615msec) clat (usec): min=2339, max=32876, avg=7350.91, stdev=3227.36 lat (usec): min=2340, max=32876, avg=7351.91, stdev=3227.47 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3326], 10.00th=[ 3785], 20.00th=[ 4621], | 30.00th=[ 5407], 40.00th=[ 6194], 50.00th=[ 6783], 60.00th=[ 7373], | 70.00th=[ 8455], 80.00th=[ 9765], 90.00th=[11469], 95.00th=[12780], | 99.00th=[19006], 99.50th=[20317], 99.90th=[24249], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 152, max= 560, per=25.03%, avg=320.67, stdev=66.68, samples=39 iops : min= 38, max= 140, avg=80.05, stdev=16.72, samples=39 lat (msec) : 2=1.50%, 4=28.16%, 10=57.26%, 20=12.70%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=11.36%, ctx=3595, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12254: Wed Oct 16 12:56:02 2024 read: IOPS=76, BW=306KiB/s (314kB/s)(5940KiB/19391msec) clat (usec): min=1555, max=179158, avg=5291.63, stdev=5454.77 lat (usec): min=1555, max=179160, avg=5292.47, stdev=5454.79 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 2089], 10.00th=[ 2376], 20.00th=[ 2835], | 30.00th=[ 3425], 40.00th=[ 3785], 50.00th=[ 4080], 60.00th=[ 4555], | 70.00th=[ 5735], 80.00th=[ 7504], 90.00th=[ 9372], 95.00th=[ 10814], | 99.00th=[ 15926], 99.50th=[ 17695], 99.90th=[ 29754], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 103, max= 520, per=25.17%, avg=308.03, stdev=77.60, samples=38 iops : min= 25, max= 130, avg=76.87, stdev=19.48, samples=38 write: IOPS=81, BW=327KiB/s (335kB/s)(6348KiB/19391msec) clat (usec): min=2275, max=40096, avg=7224.63, stdev=3257.82 lat (usec): min=2276, max=40096, avg=7225.62, stdev=3257.86 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3359], 10.00th=[ 3720], 20.00th=[ 4490], | 30.00th=[ 5342], 40.00th=[ 6194], 50.00th=[ 6652], 60.00th=[ 7308], | 70.00th=[ 8160], 80.00th=[ 9503], 90.00th=[11469], 95.00th=[12780], | 99.00th=[18482], 99.50th=[21103], 99.90th=[28443], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 167, max= 536, per=25.52%, avg=326.97, stdev=69.58, samples=38 iops : min= 41, max= 134, avg=81.61, stdev=17.50, samples=38 lat (msec) : 2=1.43%, 4=28.45%, 10=57.29%, 20=12.34%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=11.44%, ctx=3647, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12255: Wed Oct 16 12:56:02 2024 read: IOPS=78, BW=313KiB/s (320kB/s)(6028KiB/19279msec) clat (usec): min=1605, max=19553, avg=5131.06, stdev=2905.73 lat (usec): min=1606, max=19553, avg=5131.92, stdev=2905.73 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2089], 10.00th=[ 2343], 20.00th=[ 2900], | 30.00th=[ 3359], 40.00th=[ 3752], 50.00th=[ 4113], 60.00th=[ 4621], | 70.00th=[ 5669], 80.00th=[ 7439], 90.00th=[ 9372], 95.00th=[11338], | 99.00th=[14091], 99.50th=[16057], 99.90th=[19006], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 136, max= 487, per=25.62%, avg=313.63, stdev=66.73, samples=38 iops : min= 34, max= 121, avg=78.37, stdev=16.63, samples=38 write: IOPS=81, BW=325KiB/s (332kB/s)(6260KiB/19279msec) clat (msec): min=2, max=165, avg= 7.32, stdev= 5.14 lat (msec): min=2, max=165, avg= 7.33, stdev= 5.14 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 9], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 13], | 99.00th=[ 18], 99.50th=[ 23], 99.90th=[ 34], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 176, max= 576, per=25.35%, avg=324.79, stdev=72.03, samples=38 iops : min= 44, max= 144, avg=81.16, stdev=17.96, samples=38 lat (msec) : 2=1.56%, 4=29.00%, 10=56.58%, 20=12.53%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=11.47%, ctx=3653, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12256: Wed Oct 16 12:56:02 2024 read: IOPS=77, BW=310KiB/s (318kB/s)(6052KiB/19494msec) clat (usec): min=1575, max=27199, avg=5135.89, stdev=3079.31 lat (usec): min=1576, max=27200, avg=5136.70, stdev=3079.32 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 2057], 10.00th=[ 2343], 20.00th=[ 2769], | 30.00th=[ 3294], 40.00th=[ 3752], 50.00th=[ 4113], 60.00th=[ 4490], | 70.00th=[ 5669], 80.00th=[ 7439], 90.00th=[ 9372], 95.00th=[11338], | 99.00th=[15401], 99.50th=[17957], 99.90th=[20317], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 152, max= 512, per=25.43%, avg=311.29, stdev=74.96, samples=38 iops : min= 38, max= 128, avg=77.76, stdev=18.72, samples=38 write: IOPS=79, BW=320KiB/s (328kB/s)(6236KiB/19494msec) clat (msec): min=2, max=153, avg= 7.46, stdev= 4.93 lat (msec): min=2, max=153, avg= 7.46, stdev= 4.93 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 9], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 19], 99.50th=[ 22], 99.90th=[ 39], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 168, max= 544, per=25.01%, avg=320.34, stdev=65.97, samples=38 iops : min= 42, max= 136, avg=80.03, stdev=16.47, samples=38 lat (msec) : 2=1.95%, 4=27.57%, 10=57.10%, 20=13.05%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.27%, ctx=3644, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1224KiB/s (1254kB/s), 305KiB/s-313KiB/s (313kB/s-320kB/s), io=23.4MiB (24.6MB), run=19279-19615msec WRITE: bw=1282KiB/s (1312kB/s), 320KiB/s-327KiB/s (328kB/s-335kB/s), io=24.6MiB (25.7MB), run=19279-19615msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12283: Wed Oct 16 12:56:08 2024 read: IOPS=71, BW=1142KiB/s (1169kB/s)(5920KiB/5186msec) clat (usec): min=1826, max=22887, avg=5501.65, stdev=3877.78 lat (usec): min=1827, max=22888, avg=5502.22, stdev=3877.80 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2089], 10.00th=[ 2245], 20.00th=[ 2474], | 30.00th=[ 2835], 40.00th=[ 3228], 50.00th=[ 3949], 60.00th=[ 4948], | 70.00th=[ 6194], 80.00th=[ 8291], 90.00th=[10814], 95.00th=[12649], | 99.00th=[21627], 99.50th=[22938], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 736, max= 1600, per=24.47%, avg=1123.00, stdev=271.46, samples=10 iops : min= 46, max= 100, avg=70.10, stdev=17.04, samples=10 write: IOPS=76, BW=1228KiB/s (1257kB/s)(6368KiB/5186msec) clat (usec): min=2585, max=71359, avg=7827.47, stdev=5478.54 lat (usec): min=2586, max=71360, avg=7828.50, stdev=5478.54 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 3294], 10.00th=[ 3458], 20.00th=[ 3949], | 30.00th=[ 4490], 40.00th=[ 5342], 50.00th=[ 6128], 60.00th=[ 7635], | 70.00th=[ 9110], 80.00th=[10814], 90.00th=[15139], 95.00th=[17171], | 99.00th=[23462], 99.50th=[25822], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 832, max= 1440, per=25.37%, avg=1225.30, stdev=202.89, samples=10 iops : min= 52, max= 90, avg=76.50, stdev=12.66, samples=10 lat (msec) : 2=1.43%, 4=34.11%, 10=45.31%, 20=17.71%, 50=1.30% lat (msec) : 100=0.13% cpu : usr=0.12%, sys=8.95%, ctx=941, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12284: Wed Oct 16 12:56:08 2024 read: IOPS=68, BW=1101KiB/s (1127kB/s)(5744KiB/5218msec) clat (usec): min=1869, max=27474, avg=5341.69, stdev=4016.88 lat (usec): min=1869, max=27475, avg=5342.26, stdev=4016.89 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2008], 10.00th=[ 2180], 20.00th=[ 2409], | 30.00th=[ 2671], 40.00th=[ 3032], 50.00th=[ 3458], 60.00th=[ 4359], | 70.00th=[ 6325], 80.00th=[ 8455], 90.00th=[11600], 95.00th=[13566], | 99.00th=[18482], 99.50th=[21103], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 672, max= 1568, per=23.98%, avg=1100.50, stdev=315.76, samples=10 iops : min= 42, max= 98, avg=68.70, stdev=19.73, samples=10 write: IOPS=78, BW=1254KiB/s (1284kB/s)(6544KiB/5218msec) clat (usec): min=2544, max=26067, avg=7964.01, stdev=4911.63 lat (usec): min=2545, max=26068, avg=7965.32, stdev=4911.71 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3261], 10.00th=[ 3458], 20.00th=[ 3884], | 30.00th=[ 4359], 40.00th=[ 5211], 50.00th=[ 6521], 60.00th=[ 7570], | 70.00th=[ 9110], 80.00th=[11469], 90.00th=[15926], 95.00th=[18744], | 99.00th=[22676], 99.50th=[23200], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 928, max= 1472, per=25.51%, avg=1231.70, stdev=137.60, samples=10 iops : min= 58, max= 92, avg=76.90, stdev= 8.60, samples=10 lat (msec) : 2=2.21%, 4=37.63%, 10=40.36%, 20=17.84%, 50=1.95% cpu : usr=0.13%, sys=9.01%, ctx=943, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12286: Wed Oct 16 12:56:08 2024 read: IOPS=78, BW=1257KiB/s (1287kB/s)(6256KiB/4978msec) clat (usec): min=1734, max=22905, avg=5235.76, stdev=3882.86 lat (usec): min=1734, max=22906, avg=5236.38, stdev=3883.01 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2114], 10.00th=[ 2245], 20.00th=[ 2474], | 30.00th=[ 2737], 40.00th=[ 2999], 50.00th=[ 3392], 60.00th=[ 4146], | 70.00th=[ 5604], 80.00th=[ 8029], 90.00th=[10945], 95.00th=[13566], | 99.00th=[19006], 99.50th=[19792], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 800, max= 1600, per=26.56%, avg=1219.22, stdev=226.89, samples=9 iops : min= 50, max= 100, avg=76.11, stdev=14.13, samples=9 write: IOPS=75, BW=1212KiB/s (1241kB/s)(6032KiB/4978msec) clat (usec): min=2440, max=27139, avg=7732.33, stdev=4815.15 lat (usec): min=2441, max=27140, avg=7733.32, stdev=4815.16 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3261], 10.00th=[ 3523], 20.00th=[ 3949], | 30.00th=[ 4359], 40.00th=[ 5014], 50.00th=[ 6194], 60.00th=[ 7373], | 70.00th=[ 8717], 80.00th=[11076], 90.00th=[14484], 95.00th=[18482], | 99.00th=[25560], 99.50th=[26084], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 1024, max= 1312, per=24.59%, avg=1187.22, stdev=115.78, samples=9 iops : min= 64, max= 82, avg=74.11, stdev= 7.22, samples=9 lat (msec) : 2=0.91%, 4=39.06%, 10=41.15%, 20=17.45%, 50=1.43% cpu : usr=0.08%, sys=9.46%, ctx=920, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12287: Wed Oct 16 12:56:08 2024 read: IOPS=76, BW=1220KiB/s (1250kB/s)(6032KiB/4943msec) clat (usec): min=1714, max=23414, avg=5063.19, stdev=3586.62 lat (usec): min=1714, max=23415, avg=5063.82, stdev=3586.63 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2008], 10.00th=[ 2089], 20.00th=[ 2376], | 30.00th=[ 2704], 40.00th=[ 3064], 50.00th=[ 3490], 60.00th=[ 3982], | 70.00th=[ 5473], 80.00th=[ 7898], 90.00th=[10683], 95.00th=[12256], | 99.00th=[16188], 99.50th=[18220], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 896, max= 1472, per=26.10%, avg=1197.89, stdev=161.00, samples=9 iops : min= 56, max= 92, avg=74.78, stdev=10.10, samples=9 write: IOPS=79, BW=1266KiB/s (1296kB/s)(6256KiB/4943msec) clat (usec): min=2624, max=26506, avg=7691.62, stdev=4344.96 lat (usec): min=2625, max=26508, avg=7692.64, stdev=4344.97 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 3294], 10.00th=[ 3621], 20.00th=[ 4015], | 30.00th=[ 4686], 40.00th=[ 5473], 50.00th=[ 6325], 60.00th=[ 7373], | 70.00th=[ 8848], 80.00th=[10945], 90.00th=[13698], 95.00th=[16581], | 99.00th=[21627], 99.50th=[23200], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 1056, max= 1472, per=26.13%, avg=1261.89, stdev=140.37, samples=9 iops : min= 66, max= 92, avg=78.78, stdev= 8.74, samples=9 lat (msec) : 2=2.34%, 4=37.37%, 10=40.89%, 20=18.23%, 50=1.17% cpu : usr=0.14%, sys=9.53%, ctx=937, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4590KiB/s (4700kB/s), 1101KiB/s-1257KiB/s (1127kB/s-1287kB/s), io=23.4MiB (24.5MB), run=4943-5218msec WRITE: bw=4829KiB/s (4945kB/s), 1212KiB/s-1266KiB/s (1241kB/s-1296kB/s), io=24.6MiB (25.8MB), run=4943-5218msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12288: Wed Oct 16 12:56:09 2024 read: IOPS=67, BW=1082KiB/s (1108kB/s)(5920KiB/5470msec) clat (usec): min=58, max=41014, avg=10595.11, stdev=4185.42 lat (usec): min=59, max=41015, avg=10597.28, stdev=4185.72 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 81], 10.00th=[ 7308], 20.00th=[ 8717], | 30.00th=[ 9372], 40.00th=[10028], 50.00th=[10552], 60.00th=[11207], | 70.00th=[11994], 80.00th=[12911], 90.00th=[14353], 95.00th=[16450], | 99.00th=[21890], 99.50th=[24773], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 704, max= 1248, per=23.60%, avg=1033.10, stdev=151.11, samples=10 iops : min= 44, max= 78, avg=64.40, stdev= 9.55, samples=10 write: IOPS=72, BW=1164KiB/s (1192kB/s)(6368KiB/5470msec) clat (usec): min=656, max=25392, avg=3733.14, stdev=3637.28 lat (usec): min=657, max=25393, avg=3734.01, stdev=3637.29 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 791], 10.00th=[ 816], 20.00th=[ 930], | 30.00th=[ 1696], 40.00th=[ 2343], 50.00th=[ 2671], 60.00th=[ 3130], | 70.00th=[ 3752], 80.00th=[ 5211], 90.00th=[ 8455], 95.00th=[11863], | 99.00th=[16581], 99.50th=[24249], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 800, max= 1536, per=24.37%, avg=1122.60, stdev=238.01, samples=10 iops : min= 50, max= 96, avg=70.00, stdev=14.94, samples=10 lat (usec) : 100=2.73%, 250=0.52%, 750=0.91%, 1000=10.81% lat (msec) : 2=4.56%, 4=21.22%, 10=25.78%, 20=32.42%, 50=1.04% cpu : usr=0.11%, sys=19.07%, ctx=2438, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12289: Wed Oct 16 12:56:09 2024 read: IOPS=65, BW=1051KiB/s (1076kB/s)(5744KiB/5465msec) clat (usec): min=53, max=27742, avg=10541.20, stdev=3752.06 lat (usec): min=54, max=27743, avg=10541.77, stdev=3752.06 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 77], 10.00th=[ 7373], 20.00th=[ 8586], | 30.00th=[ 9503], 40.00th=[10290], 50.00th=[10814], 60.00th=[11469], | 70.00th=[12125], 80.00th=[12911], 90.00th=[14484], 95.00th=[16057], | 99.00th=[17695], 99.50th=[25297], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 608, max= 1216, per=22.87%, avg=1001.10, stdev=173.48, samples=10 iops : min= 38, max= 76, avg=62.40, stdev=10.92, samples=10 write: IOPS=74, BW=1197KiB/s (1226kB/s)(6544KiB/5465msec) clat (usec): min=627, max=24651, avg=4031.46, stdev=3992.42 lat (usec): min=628, max=24652, avg=4035.33, stdev=3991.76 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 799], 10.00th=[ 824], 20.00th=[ 930], | 30.00th=[ 1270], 40.00th=[ 2278], 50.00th=[ 2671], 60.00th=[ 3294], | 70.00th=[ 4424], 80.00th=[ 6259], 90.00th=[ 9896], 95.00th=[12649], | 99.00th=[17957], 99.50th=[22414], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 864, max= 1600, per=24.58%, avg=1132.30, stdev=260.25, samples=10 iops : min= 54, max= 100, avg=70.60, stdev=16.23, samples=10 lat (usec) : 100=2.60%, 250=0.26%, 750=0.91%, 1000=11.59% lat (msec) : 2=4.95%, 4=17.58%, 10=27.73%, 20=33.72%, 50=0.65% cpu : usr=0.16%, sys=18.89%, ctx=2431, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12290: Wed Oct 16 12:56:09 2024 read: IOPS=72, BW=1154KiB/s (1182kB/s)(6256KiB/5421msec) clat (usec): min=51, max=22078, avg=10251.83, stdev=3877.36 lat (usec): min=51, max=22078, avg=10253.47, stdev=3876.86 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 77], 10.00th=[ 7177], 20.00th=[ 8717], | 30.00th=[ 9372], 40.00th=[ 9896], 50.00th=[10552], 60.00th=[11207], | 70.00th=[11863], 80.00th=[12911], 90.00th=[14222], 95.00th=[15664], | 99.00th=[17957], 99.50th=[21365], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 672, max= 1280, per=25.21%, avg=1103.70, stdev=183.53, samples=10 iops : min= 42, max= 80, avg=68.90, stdev=11.44, samples=10 write: IOPS=69, BW=1113KiB/s (1139kB/s)(6032KiB/5421msec) clat (usec): min=623, max=51458, avg=3553.48, stdev=4335.13 lat (usec): min=624, max=51459, avg=3554.38, stdev=4335.14 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 775], 10.00th=[ 807], 20.00th=[ 898], | 30.00th=[ 1123], 40.00th=[ 2114], 50.00th=[ 2409], 60.00th=[ 2769], | 70.00th=[ 3163], 80.00th=[ 4686], 90.00th=[ 7767], 95.00th=[11994], | 99.00th=[19268], 99.50th=[21365], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 800, max= 1472, per=23.20%, avg=1068.50, stdev=216.34, samples=10 iops : min= 50, max= 92, avg=66.70, stdev=13.45, samples=10 lat (usec) : 100=3.52%, 250=0.65%, 750=1.17%, 1000=11.20% lat (msec) : 2=5.34%, 4=20.05%, 10=24.74%, 20=32.81%, 50=0.39% lat (msec) : 100=0.13% cpu : usr=0.02%, sys=18.45%, ctx=2446, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12291: Wed Oct 16 12:56:09 2024 read: IOPS=69, BW=1106KiB/s (1132kB/s)(6032KiB/5455msec) clat (usec): min=58, max=23125, avg=10467.52, stdev=3945.36 lat (usec): min=58, max=23125, avg=10468.12, stdev=3945.38 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 86], 10.00th=[ 7242], 20.00th=[ 8586], | 30.00th=[ 9241], 40.00th=[ 9896], 50.00th=[10552], 60.00th=[11338], | 70.00th=[11994], 80.00th=[12911], 90.00th=[14615], 95.00th=[16319], | 99.00th=[21103], 99.50th=[22414], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 672, max= 1312, per=24.18%, avg=1058.60, stdev=181.47, samples=10 iops : min= 42, max= 82, avg=66.00, stdev=11.22, samples=10 write: IOPS=71, BW=1147KiB/s (1174kB/s)(6256KiB/5455msec) clat (usec): min=641, max=52295, avg=3650.76, stdev=4420.26 lat (usec): min=641, max=52296, avg=3651.78, stdev=4420.25 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 775], 10.00th=[ 807], 20.00th=[ 922], | 30.00th=[ 1188], 40.00th=[ 2147], 50.00th=[ 2507], 60.00th=[ 2966], | 70.00th=[ 3490], 80.00th=[ 4948], 90.00th=[ 7701], 95.00th=[11994], | 99.00th=[21103], 99.50th=[24511], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 768, max= 1536, per=24.09%, avg=1109.80, stdev=189.73, samples=10 iops : min= 48, max= 96, avg=69.20, stdev=11.89, samples=10 lat (usec) : 100=2.99%, 250=0.52%, 750=1.69%, 1000=10.29% lat (msec) : 2=7.03%, 4=19.14%, 10=25.65%, 20=31.38%, 50=1.17% lat (msec) : 100=0.13% cpu : usr=0.17%, sys=18.50%, ctx=2419, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4379KiB/s (4484kB/s), 1051KiB/s-1154KiB/s (1076kB/s-1182kB/s), io=23.4MiB (24.5MB), run=5421-5470msec WRITE: bw=4607KiB/s (4718kB/s), 1113KiB/s-1197KiB/s (1139kB/s-1226kB/s), io=24.6MiB (25.8MB), run=5421-5470msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12300: Wed Oct 16 12:56:12 2024 read: IOPS=15, BW=15.2MiB/s (15.9MB/s)(7168KiB/462msec) clat (usec): min=13859, max=59217, avg=34764.62, stdev=17930.07 lat (usec): min=13859, max=59219, avg=34767.99, stdev=17929.14 clat percentiles (usec): | 1.00th=[13829], 5.00th=[13829], 10.00th=[13829], 20.00th=[19268], | 30.00th=[25822], 40.00th=[25822], 50.00th=[26870], 60.00th=[42206], | 70.00th=[42206], 80.00th=[55837], 90.00th=[58983], 95.00th=[58983], | 99.00th=[58983], 99.50th=[58983], 99.90th=[58983], 99.95th=[58983], | 99.99th=[58983] write: IOPS=10, BW=10.8MiB/s (11.3MB/s)(5120KiB/462msec) clat (usec): min=19240, max=54752, avg=31312.28, stdev=13914.85 lat (usec): min=19303, max=54785, avg=31362.13, stdev=13902.51 clat percentiles (usec): | 1.00th=[19268], 5.00th=[19268], 10.00th=[19268], 20.00th=[19268], | 30.00th=[24249], 40.00th=[24249], 50.00th=[25822], 60.00th=[25822], | 70.00th=[32375], 80.00th=[32375], 90.00th=[54789], 95.00th=[54789], | 99.00th=[54789], 99.50th=[54789], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] lat (msec) : 20=25.00%, 50=50.00%, 100=25.00% cpu : usr=0.43%, sys=22.13%, ctx=89, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12301: Wed Oct 16 12:56:12 2024 read: IOPS=0, BW=815KiB/s (834kB/s)(2048KiB/2514msec) clat (usec): min=37639, max=45810, avg=41725.02, stdev=5778.39 lat (usec): min=37641, max=45812, avg=41727.24, stdev=5777.75 clat percentiles (usec): | 1.00th=[37487], 5.00th=[37487], 10.00th=[37487], 20.00th=[37487], | 30.00th=[37487], 40.00th=[37487], 50.00th=[37487], 60.00th=[45876], | 70.00th=[45876], 80.00th=[45876], 90.00th=[45876], 95.00th=[45876], | 99.00th=[45876], 99.50th=[45876], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 1822, max= 2048, per=23.75%, avg=1935.00, stdev=159.81, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=4073KiB/s (4171kB/s)(10.0MiB/2514msec) clat (msec): min=15, max=1085, avg=236.70, stdev=323.67 lat (msec): min=15, max=1085, avg=236.79, stdev=323.67 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 23], | 30.00th=[ 43], 40.00th=[ 106], 50.00th=[ 126], 60.00th=[ 146], | 70.00th=[ 203], 80.00th=[ 203], 90.00th=[ 456], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 3644, max= 6144, per=44.59%, avg=5084.67, stdev=1292.89, samples=3 iops : min= 3, max= 6, avg= 4.67, stdev= 1.53, samples=3 lat (msec) : 20=8.33%, 50=33.33%, 250=41.67%, 500=8.33% cpu : usr=0.04%, sys=7.68%, ctx=53, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12302: Wed Oct 16 12:56:12 2024 read: IOPS=5, BW=5180KiB/s (5305kB/s)(6144KiB/1186msec) clat (msec): min=13, max=748, avg=155.66, stdev=291.47 lat (msec): min=13, max=748, avg=155.66, stdev=291.47 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 18], | 30.00th=[ 18], 40.00th=[ 33], 50.00th=[ 33], 60.00th=[ 39], | 70.00th=[ 85], 80.00th=[ 85], 90.00th=[ 751], 95.00th=[ 751], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min=10219, max=10219, per=100.00%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5180KiB/s (5305kB/s)(6144KiB/1186msec) clat (usec): min=14461, max=46701, avg=31456.69, stdev=12430.28 lat (usec): min=14793, max=46760, avg=31559.86, stdev=12372.54 clat percentiles (usec): | 1.00th=[14484], 5.00th=[14484], 10.00th=[14484], 20.00th=[20841], | 30.00th=[20841], 40.00th=[28967], 50.00th=[28967], 60.00th=[35390], | 70.00th=[42206], 80.00th=[42206], 90.00th=[46924], 95.00th=[46924], | 99.00th=[46924], 99.50th=[46924], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min=12263, max=12263, per=100.00%, avg=12263.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=58.33%, 100=8.33%, 750=8.33% cpu : usr=0.08%, sys=7.68%, ctx=78, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12303: Wed Oct 16 12:56:12 2024 read: IOPS=4, BW=4384KiB/s (4489kB/s)(5120KiB/1168msec) clat (msec): min=17, max=726, avg=177.13, stdev=307.88 lat (msec): min=17, max=726, avg=177.13, stdev=307.88 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 17], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 42], 60.00th=[ 42], | 70.00th=[ 74], 80.00th=[ 74], 90.00th=[ 726], 95.00th=[ 726], | 99.00th=[ 726], 99.50th=[ 726], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=6137KiB/s (6284kB/s)(7168KiB/1168msec) clat (usec): min=16364, max=56224, avg=35603.80, stdev=16980.92 lat (usec): min=16397, max=56337, avg=35662.52, stdev=16985.44 clat percentiles (usec): | 1.00th=[16319], 5.00th=[16319], 10.00th=[16319], 20.00th=[16909], | 30.00th=[23987], 40.00th=[23987], 50.00th=[37487], 60.00th=[42730], | 70.00th=[42730], 80.00th=[55313], 90.00th=[56361], 95.00th=[56361], | 99.00th=[56361], 99.50th=[56361], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min=14336, max=14336, per=100.00%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=41.67%, 100=25.00%, 750=8.33% cpu : usr=0.09%, sys=10.80%, ctx=143, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8146KiB/s (8342kB/s), 815KiB/s-15.2MiB/s (834kB/s-15.9MB/s), io=20.0MiB (20.0MB), run=462-2514msec WRITE: bw=11.1MiB/s (11.7MB/s), 4073KiB/s-10.8MiB/s (4171kB/s-11.3MB/s), io=28.0MiB (29.4MB), run=462-2514msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12305: Wed Oct 16 12:56:12 2024 read: IOPS=4, BW=4933KiB/s (5052kB/s)(7168KiB/1453msec) clat (usec): min=383, max=885800, avg=138836.33, stdev=330136.96 lat (usec): min=384, max=885801, avg=138837.09, stdev=330136.93 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 383], 10.00th=[ 383], 20.00th=[ 3851], | 30.00th=[ 3916], 40.00th=[ 3916], 50.00th=[ 3949], 60.00th=[ 10028], | 70.00th=[ 10028], 80.00th=[ 63701], 90.00th=[884999], 95.00th=[884999], | 99.00th=[884999], 99.50th=[884999], 99.90th=[884999], 99.95th=[884999], | 99.99th=[884999] bw ( KiB/s): min= 4096, max= 4096, per=48.14%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3524KiB/s (3608kB/s)(5120KiB/1453msec) clat (msec): min=18, max=333, avg=86.19, stdev=138.28 lat (msec): min=18, max=333, avg=86.24, stdev=138.31 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 25], 60.00th=[ 25], | 70.00th=[ 37], 80.00th=[ 37], 90.00th=[ 334], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] lat (usec) : 500=8.33% lat (msec) : 4=25.00%, 20=25.00%, 50=16.67%, 100=8.33%, 500=8.33% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=17.01%, ctx=321, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12306: Wed Oct 16 12:56:12 2024 read: IOPS=0, BW=851KiB/s (871kB/s)(2048KiB/2407msec) clat (msec): min=59, max=547, avg=303.49, stdev=345.40 lat (msec): min=59, max=547, avg=303.49, stdev=345.40 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 60], 10.00th=[ 60], 20.00th=[ 60], | 30.00th=[ 60], 40.00th=[ 60], 50.00th=[ 60], 60.00th=[ 550], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 1969, max= 2048, per=23.61%, avg=2008.50, stdev=55.86, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4254KiB/s (4356kB/s)(10.0MiB/2407msec) clat (msec): min=17, max=1113, avg=175.96, stdev=335.85 lat (msec): min=17, max=1113, avg=176.01, stdev=335.85 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 23], | 30.00th=[ 26], 40.00th=[ 26], 50.00th=[ 32], 60.00th=[ 50], | 70.00th=[ 182], 80.00th=[ 182], 90.00th=[ 190], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 3938, max= 6144, per=39.24%, avg=4673.33, stdev=1273.63, samples=3 iops : min= 3, max= 6, avg= 4.00, stdev= 1.73, samples=3 lat (msec) : 20=8.33%, 50=41.67%, 100=8.33%, 250=25.00%, 750=8.33% cpu : usr=0.00%, sys=33.29%, ctx=60, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12307: Wed Oct 16 12:56:12 2024 read: IOPS=4, BW=4223KiB/s (4324kB/s)(6144KiB/1455msec) clat (usec): min=472, max=918289, avg=203486.26, stdev=354570.17 lat (usec): min=473, max=918290, avg=203487.38, stdev=354570.04 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 474], 10.00th=[ 474], 20.00th=[ 2606], | 30.00th=[ 2606], 40.00th=[ 68682], 50.00th=[ 68682], 60.00th=[ 81265], | 70.00th=[149947], 80.00th=[149947], 90.00th=[918553], 95.00th=[918553], | 99.00th=[918553], 99.50th=[918553], 99.90th=[918553], 99.95th=[918553], | 99.99th=[918553] bw ( KiB/s): min= 4096, max= 4096, per=48.14%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4223KiB/s (4324kB/s)(6144KiB/1455msec) clat (usec): min=22610, max=46478, avg=35093.21, stdev=8421.25 lat (usec): min=22654, max=46522, avg=35374.01, stdev=8656.80 clat percentiles (usec): | 1.00th=[22676], 5.00th=[22676], 10.00th=[22676], 20.00th=[30802], | 30.00th=[30802], 40.00th=[33162], 50.00th=[33162], 60.00th=[35390], | 70.00th=[41681], 80.00th=[41681], 90.00th=[46400], 95.00th=[46400], | 99.00th=[46400], 99.50th=[46400], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 8192, max= 8192, per=68.78%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 4=8.33%, 50=50.00%, 100=16.67%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=25.58%, ctx=454, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12308: Wed Oct 16 12:56:12 2024 read: IOPS=3, BW=3536KiB/s (3621kB/s)(5120KiB/1448msec) clat (usec): min=362, max=949776, avg=225401.95, stdev=408232.10 lat (usec): min=363, max=949777, avg=225402.91, stdev=408232.00 clat percentiles (usec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 363], | 30.00th=[ 383], 40.00th=[ 383], 50.00th=[ 49546], 60.00th=[ 49546], | 70.00th=[127402], 80.00th=[127402], 90.00th=[952108], 95.00th=[952108], | 99.00th=[952108], 99.50th=[952108], 99.90th=[952108], 99.95th=[952108], | 99.99th=[952108] bw ( KiB/s): min= 2048, max= 2048, per=24.07%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4950KiB/s (5069kB/s)(7168KiB/1448msec) clat (usec): min=18892, max=54056, avg=36757.50, stdev=10273.71 lat (usec): min=18943, max=54133, avg=38007.69, stdev=10711.98 clat percentiles (usec): | 1.00th=[19006], 5.00th=[19006], 10.00th=[19006], 20.00th=[34341], | 30.00th=[35914], 40.00th=[35914], 50.00th=[36439], 60.00th=[38536], | 70.00th=[38536], 80.00th=[39060], 90.00th=[54264], 95.00th=[54264], | 99.00th=[54264], 99.50th=[54264], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 8192, max= 8192, per=68.78%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 20=8.33%, 50=50.00%, 100=8.33%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=27.57%, ctx=389, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8509KiB/s (8713kB/s), 851KiB/s-4933KiB/s (871kB/s-5052kB/s), io=20.0MiB (20.0MB), run=1448-2407msec WRITE: bw=11.6MiB/s (12.2MB/s), 3524KiB/s-4950KiB/s (3608kB/s-5069kB/s), io=28.0MiB (29.4MB), run=1448-2407msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12316: Wed Oct 16 12:56:16 2024 read: IOPS=9, BW=36.7MiB/s (38.5MB/s)(12.0MiB/327msec) clat (msec): min=33, max=195, avg=99.53, stdev=84.70 lat (msec): min=33, max=195, avg=99.53, stdev=84.70 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34], | 30.00th=[ 34], 40.00th=[ 70], 50.00th=[ 70], 60.00th=[ 70], | 70.00th=[ 197], 80.00th=[ 197], 90.00th=[ 197], 95.00th=[ 197], | 99.00th=[ 197], 99.50th=[ 197], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] lat (msec) : 50=33.33%, 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=22.09%, ctx=65, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12317: Wed Oct 16 12:56:16 2024 read: IOPS=0, BW=1910KiB/s (1956kB/s)(4096KiB/2144msec) clat (nsec): min=65192k, max=65192k, avg=65192029.00, stdev= 0.00 lat (nsec): min=65194k, max=65194k, avg=65193804.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[65274], 5.00th=[65274], 10.00th=[65274], 20.00th=[65274], | 30.00th=[65274], 40.00th=[65274], 50.00th=[65274], 60.00th=[65274], | 70.00th=[65274], 80.00th=[65274], 90.00th=[65274], 95.00th=[65274], | 99.00th=[65274], 99.50th=[65274], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3821KiB/s (3913kB/s)(8192KiB/2144msec) clat (msec): min=181, max=1859, avg=1020.18, stdev=1186.43 lat (msec): min=181, max=1859, avg=1020.44, stdev=1186.42 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 182], 10.00th=[ 182], 20.00th=[ 182], | 30.00th=[ 182], 40.00th=[ 182], 50.00th=[ 182], 60.00th=[ 1854], | 70.00th=[ 1854], 80.00th=[ 1854], 90.00th=[ 1854], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 8192, max= 8192, per=55.70%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.05%, sys=3.83%, ctx=63, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12318: Wed Oct 16 12:56:16 2024 write: IOPS=1, BW=5515KiB/s (5648kB/s)(12.0MiB/2228msec) clat (msec): min=76, max=1976, avg=716.45, stdev=1091.30 lat (msec): min=77, max=1976, avg=716.77, stdev=1091.29 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 78], 10.00th=[ 78], 20.00th=[ 78], | 30.00th=[ 78], 40.00th=[ 96], 50.00th=[ 96], 60.00th=[ 96], | 70.00th=[ 1972], 80.00th=[ 1972], 90.00th=[ 1972], 95.00th=[ 1972], | 99.00th=[ 1972], 99.50th=[ 1972], 99.90th=[ 1972], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=66.67% cpu : usr=0.00%, sys=4.85%, ctx=61, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12319: Wed Oct 16 12:56:16 2024 write: IOPS=2, BW=9918KiB/s (10.2MB/s)(12.0MiB/1239msec) clat (msec): min=134, max=938, avg=406.71, stdev=460.68 lat (msec): min=135, max=938, avg=406.98, stdev=460.66 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 146], 50.00th=[ 146], 60.00th=[ 146], | 70.00th=[ 936], 80.00th=[ 936], 90.00th=[ 936], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.32%, sys=9.29%, ctx=82, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7642KiB/s (7825kB/s), 1910KiB/s-36.7MiB/s (1956kB/s-38.5MB/s), io=16.0MiB (16.8MB), run=327-2144msec WRITE: bw=14.4MiB/s (15.1MB/s), 3821KiB/s-9918KiB/s (3913kB/s-10.2MB/s), io=32.0MiB (33.6MB), run=1239-2228msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12320: Wed Oct 16 12:56:17 2024 read: IOPS=0, BW=3212KiB/s (3289kB/s)(12.0MiB/3826msec) clat (msec): min=10, max=3527, avg=1244.98, stdev=1978.55 lat (msec): min=10, max=3527, avg=1244.98, stdev=1978.55 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 11], | 30.00th=[ 11], 40.00th=[ 199], 50.00th=[ 199], 60.00th=[ 199], | 70.00th=[ 3540], 80.00th=[ 3540], 90.00th=[ 3540], 95.00th=[ 3540], | 99.00th=[ 3540], 99.50th=[ 3540], 99.90th=[ 3540], 99.95th=[ 3540], | 99.99th=[ 3540] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 250=33.33% cpu : usr=0.10%, sys=16.94%, ctx=1603, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12321: Wed Oct 16 12:56:17 2024 read: IOPS=0, BW=1034KiB/s (1058kB/s)(4096KiB/3963msec) clat (nsec): min=175228k, max=175228k, avg=175227800.00, stdev= 0.00 lat (nsec): min=175229k, max=175229k, avg=175228867.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 176], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 176], 50.00th=[ 176], 60.00th=[ 176], | 70.00th=[ 176], 80.00th=[ 176], 90.00th=[ 176], 95.00th=[ 176], | 99.00th=[ 176], 99.50th=[ 176], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2067KiB/s (2117kB/s)(8192KiB/3963msec) clat (msec): min=1666, max=2054, avg=1860.67, stdev=273.89 lat (msec): min=1667, max=2054, avg=1860.92, stdev=273.90 clat percentiles (msec): | 1.00th=[ 1670], 5.00th=[ 1670], 10.00th=[ 1670], 20.00th=[ 1670], | 30.00th=[ 1670], 40.00th=[ 1670], 50.00th=[ 1670], 60.00th=[ 2056], | 70.00th=[ 2056], 80.00th=[ 2056], 90.00th=[ 2056], 95.00th=[ 2056], | 99.00th=[ 2056], 99.50th=[ 2056], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33% cpu : usr=0.05%, sys=6.71%, ctx=72, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12322: Wed Oct 16 12:56:17 2024 write: IOPS=0, BW=3164KiB/s (3240kB/s)(12.0MiB/3884msec) clat (msec): min=101, max=3476, avg=1268.14, stdev=1913.22 lat (msec): min=102, max=3476, avg=1268.87, stdev=1912.90 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 102], 10.00th=[ 102], 20.00th=[ 102], | 30.00th=[ 102], 40.00th=[ 228], 50.00th=[ 228], 60.00th=[ 228], | 70.00th=[ 3473], 80.00th=[ 3473], 90.00th=[ 3473], 95.00th=[ 3473], | 99.00th=[ 3473], 99.50th=[ 3473], 99.90th=[ 3473], 99.95th=[ 3473], | 99.99th=[ 3473] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.10%, sys=10.87%, ctx=291, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12323: Wed Oct 16 12:56:17 2024 write: IOPS=0, BW=3042KiB/s (3115kB/s)(12.0MiB/4039msec) clat (msec): min=144, max=2032, avg=1318.71, stdev=1024.96 lat (msec): min=144, max=2032, avg=1319.13, stdev=1024.98 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 1787], 50.00th=[ 1787], 60.00th=[ 1787], | 70.00th=[ 2039], 80.00th=[ 2039], 90.00th=[ 2039], 95.00th=[ 2039], | 99.00th=[ 2039], 99.50th=[ 2039], 99.90th=[ 2039], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.02%, sys=9.51%, ctx=30, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4134KiB/s (4233kB/s), 1034KiB/s-3212KiB/s (1058kB/s-3289kB/s), io=16.0MiB (16.8MB), run=3826-3963msec WRITE: bw=8113KiB/s (8308kB/s), 2067KiB/s-3164KiB/s (2117kB/s-3240kB/s), io=32.0MiB (33.6MB), run=3884-4039msec