== sanity test 398b: DIO and buffer IO race ============== 06:48:32 (1726310912) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.18232 s, 42.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24533: Sat Sep 14 06:48:47 2024 read: IOPS=128, BW=512KiB/s (525kB/s)(5992KiB/11698msec) clat (usec): min=48, max=178499, avg=3550.86, stdev=5367.84 lat (usec): min=49, max=178500, avg=3551.44, stdev=5367.83 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 60], 10.00th=[ 69], 20.00th=[ 1745], | 30.00th=[ 2180], 40.00th=[ 2540], 50.00th=[ 2900], 60.00th=[ 3490], | 70.00th=[ 4228], 80.00th=[ 5407], 90.00th=[ 6849], 95.00th=[ 7963], | 99.00th=[ 11469], 99.50th=[ 14484], 99.90th=[ 40109], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 248, max= 736, per=24.69%, avg=506.61, stdev=98.19, samples=23 iops : min= 62, max= 184, avg=126.57, stdev=24.52, samples=23 write: IOPS=134, BW=538KiB/s (551kB/s)(6296KiB/11698msec) clat (usec): min=419, max=41249, avg=3992.87, stdev=3881.50 lat (usec): min=419, max=41249, avg=3993.58, stdev=3881.53 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 502], 10.00th=[ 523], 20.00th=[ 586], | 30.00th=[ 865], 40.00th=[ 2089], 50.00th=[ 3228], 60.00th=[ 4080], | 70.00th=[ 5211], 80.00th=[ 6652], 90.00th=[ 8586], 95.00th=[10945], | 99.00th=[16909], 99.50th=[20317], 99.90th=[27919], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 296, max= 720, per=24.74%, avg=531.61, stdev=99.39, samples=23 iops : min= 74, max= 180, avg=132.83, stdev=24.81, samples=23 lat (usec) : 50=0.33%, 100=8.24%, 250=0.62%, 500=2.15%, 750=12.08% lat (usec) : 1000=2.41% lat (msec) : 2=6.15%, 4=30.99%, 10=32.78%, 20=3.87%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=23.98%, ctx=5805, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24534: Sat Sep 14 06:48:47 2024 read: IOPS=127, BW=511KiB/s (524kB/s)(5940KiB/11616msec) clat (usec): min=47, max=158679, avg=3563.45, stdev=4894.10 lat (usec): min=47, max=158681, avg=3564.31, stdev=4894.14 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 59], 10.00th=[ 68], 20.00th=[ 1795], | 30.00th=[ 2212], 40.00th=[ 2507], 50.00th=[ 2900], 60.00th=[ 3458], | 70.00th=[ 4228], 80.00th=[ 5407], 90.00th=[ 6849], 95.00th=[ 8356], | 99.00th=[ 11994], 99.50th=[ 16319], 99.90th=[ 28181], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 160, max= 696, per=24.90%, avg=510.87, stdev=115.68, samples=23 iops : min= 40, max= 174, avg=127.70, stdev=28.89, samples=23 write: IOPS=136, BW=546KiB/s (560kB/s)(6348KiB/11616msec) clat (usec): min=404, max=69616, avg=3920.63, stdev=4263.74 lat (usec): min=405, max=69617, avg=3921.71, stdev=4263.56 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 502], 10.00th=[ 523], 20.00th=[ 562], | 30.00th=[ 750], 40.00th=[ 1680], 50.00th=[ 3130], 60.00th=[ 3949], | 70.00th=[ 5014], 80.00th=[ 6390], 90.00th=[ 8717], 95.00th=[11207], | 99.00th=[18744], 99.50th=[21627], 99.90th=[36963], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 216, max= 768, per=25.29%, avg=543.57, stdev=113.07, samples=23 iops : min= 54, max= 192, avg=135.87, stdev=28.24, samples=23 lat (usec) : 50=0.59%, 100=7.65%, 250=0.52%, 500=2.60%, 750=13.12% lat (usec) : 1000=2.34% lat (msec) : 2=6.38%, 4=30.53%, 10=31.71%, 20=4.17%, 50=0.33% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.15%, sys=23.27%, ctx=5712, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24535: Sat Sep 14 06:48:47 2024 read: IOPS=128, BW=516KiB/s (528kB/s)(6028KiB/11689msec) clat (usec): min=47, max=41319, avg=3462.12, stdev=3116.26 lat (usec): min=47, max=41319, avg=3463.03, stdev=3115.91 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 58], 10.00th=[ 69], 20.00th=[ 125], | 30.00th=[ 2180], 40.00th=[ 2540], 50.00th=[ 2900], 60.00th=[ 3458], | 70.00th=[ 4228], 80.00th=[ 5407], 90.00th=[ 7046], 95.00th=[ 8455], | 99.00th=[12911], 99.50th=[15270], 99.90th=[34866], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 280, max= 696, per=24.90%, avg=510.91, stdev=110.08, samples=23 iops : min= 70, max= 174, avg=127.70, stdev=27.52, samples=23 write: IOPS=133, BW=536KiB/s (548kB/s)(6260KiB/11689msec) clat (usec): min=412, max=174686, avg=4075.00, stdev=5933.11 lat (usec): min=413, max=174688, avg=4075.66, stdev=5933.13 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 502], 10.00th=[ 523], 20.00th=[ 578], | 30.00th=[ 775], 40.00th=[ 1696], 50.00th=[ 3130], 60.00th=[ 3949], | 70.00th=[ 5080], 80.00th=[ 6718], 90.00th=[ 8848], 95.00th=[ 11600], | 99.00th=[ 17957], 99.50th=[ 21890], 99.90th=[ 57410], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 336, max= 696, per=24.63%, avg=529.35, stdev=79.46, samples=23 iops : min= 84, max= 174, avg=132.30, stdev=19.88, samples=23 lat (usec) : 50=0.65%, 100=8.72%, 250=0.78%, 500=2.51%, 750=12.66% lat (usec) : 1000=2.08% lat (msec) : 2=5.92%, 4=30.76%, 10=30.96%, 20=4.52%, 50=0.36% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.15%, sys=23.42%, ctx=5640, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24536: Sat Sep 14 06:48:47 2024 read: IOPS=130, BW=521KiB/s (533kB/s)(6052KiB/11623msec) clat (usec): min=46, max=27859, avg=3468.47, stdev=2819.00 lat (usec): min=46, max=27860, avg=3469.09, stdev=2819.14 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 59], 10.00th=[ 65], 20.00th=[ 1680], | 30.00th=[ 2212], 40.00th=[ 2606], 50.00th=[ 2966], 60.00th=[ 3458], | 70.00th=[ 4228], 80.00th=[ 5473], 90.00th=[ 6980], 95.00th=[ 8225], | 99.00th=[11863], 99.50th=[14091], 99.90th=[25035], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 288, max= 704, per=25.28%, avg=518.78, stdev=95.93, samples=23 iops : min= 72, max= 176, avg=129.61, stdev=24.04, samples=23 write: IOPS=134, BW=537KiB/s (549kB/s)(6236KiB/11623msec) clat (usec): min=402, max=161840, avg=4020.28, stdev=5760.33 lat (usec): min=403, max=161842, avg=4021.65, stdev=5760.56 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 506], 10.00th=[ 529], 20.00th=[ 578], | 30.00th=[ 734], 40.00th=[ 1598], 50.00th=[ 3064], 60.00th=[ 3884], | 70.00th=[ 5014], 80.00th=[ 6652], 90.00th=[ 8979], 95.00th=[ 11076], | 99.00th=[ 17433], 99.50th=[ 22152], 99.90th=[ 57410], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 368, max= 800, per=24.90%, avg=535.13, stdev=103.94, samples=23 iops : min= 92, max= 200, avg=133.70, stdev=26.04, samples=23 lat (usec) : 50=0.55%, 100=8.30%, 250=0.72%, 500=1.82%, 750=13.70% lat (usec) : 1000=2.38% lat (msec) : 2=6.02%, 4=30.63%, 10=31.28%, 20=4.13%, 50=0.39% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.15%, sys=23.06%, ctx=5711, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2053KiB/s (2102kB/s), 511KiB/s-521KiB/s (524kB/s-533kB/s), io=23.4MiB (24.6MB), run=11616-11698msec WRITE: bw=2149KiB/s (2201kB/s), 536KiB/s-546KiB/s (548kB/s-560kB/s), io=24.6MiB (25.7MB), run=11616-11698msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24529: Sat Sep 14 06:49:09 2024 read: IOPS=44, BW=179KiB/s (184kB/s)(5992KiB/33404msec) clat (usec): min=1478, max=164454, avg=3490.77, stdev=5061.14 lat (usec): min=1478, max=164455, avg=3491.40, stdev=5061.16 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1745], 10.00th=[ 1811], 20.00th=[ 1893], | 30.00th=[ 1975], 40.00th=[ 2089], 50.00th=[ 2245], 60.00th=[ 2507], | 70.00th=[ 3326], 80.00th=[ 4015], 90.00th=[ 6521], 95.00th=[ 8979], | 99.00th=[ 15139], 99.50th=[ 19006], 99.90th=[ 44303], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 63, max= 336, per=25.34%, avg=179.42, stdev=69.64, samples=66 iops : min= 15, max= 84, avg=44.79, stdev=17.44, samples=66 write: IOPS=47, BW=188KiB/s (193kB/s)(6296KiB/33404msec) clat (usec): min=8953, max=58878, avg=17835.38, stdev=6538.27 lat (usec): min=8954, max=58878, avg=17836.33, stdev=6538.14 clat percentiles (usec): | 1.00th=[ 9765], 5.00th=[10683], 10.00th=[11338], 20.00th=[12518], | 30.00th=[13566], 40.00th=[15139], 50.00th=[16450], 60.00th=[17957], | 70.00th=[19268], 80.00th=[21365], 90.00th=[27132], 95.00th=[31327], | 99.00th=[39060], 99.50th=[44303], 99.90th=[55837], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 64, max= 256, per=25.36%, avg=187.91, stdev=53.88, samples=66 iops : min= 16, max= 64, avg=46.91, stdev=13.49, samples=66 lat (msec) : 2=16.02%, 4=22.95%, 10=9.11%, 20=38.90%, 50=12.89% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.08%, sys=5.21%, ctx=3646, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24530: Sat Sep 14 06:49:09 2024 read: IOPS=43, BW=175KiB/s (179kB/s)(5940KiB/33889msec) clat (usec): min=1453, max=165113, avg=3579.51, stdev=5299.48 lat (usec): min=1454, max=165116, avg=3580.21, stdev=5299.78 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1778], 10.00th=[ 1827], 20.00th=[ 1909], | 30.00th=[ 1991], 40.00th=[ 2114], 50.00th=[ 2245], 60.00th=[ 2638], | 70.00th=[ 3359], 80.00th=[ 4047], 90.00th=[ 6652], 95.00th=[ 9110], | 99.00th=[ 14615], 99.50th=[ 18744], 99.90th=[ 55837], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 15, max= 328, per=24.76%, avg=175.28, stdev=71.71, samples=67 iops : min= 3, max= 82, avg=43.73, stdev=17.98, samples=67 write: IOPS=46, BW=187KiB/s (192kB/s)(6348KiB/33889msec) clat (usec): min=8882, max=68294, avg=17941.01, stdev=6670.89 lat (usec): min=8883, max=68294, avg=17941.77, stdev=6670.90 clat percentiles (usec): | 1.00th=[ 9765], 5.00th=[10814], 10.00th=[11600], 20.00th=[12649], | 30.00th=[13698], 40.00th=[15270], 50.00th=[16909], 60.00th=[17957], | 70.00th=[19268], 80.00th=[21103], 90.00th=[26870], 95.00th=[31065], | 99.00th=[40633], 99.50th=[46924], 99.90th=[62653], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 71, max= 256, per=25.15%, avg=186.39, stdev=51.68, samples=67 iops : min= 17, max= 64, avg=46.51, stdev=12.95, samples=67 lat (msec) : 2=15.17%, 4=23.18%, 10=8.98%, 20=39.52%, 50=12.89% lat (msec) : 100=0.23%, 250=0.03% cpu : usr=0.08%, sys=5.09%, ctx=3740, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24531: Sat Sep 14 06:49:09 2024 read: IOPS=44, BW=179KiB/s (183kB/s)(6028KiB/33742msec) clat (usec): min=1376, max=45675, avg=3600.50, stdev=3344.59 lat (usec): min=1377, max=45675, avg=3601.16, stdev=3344.61 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1762], 10.00th=[ 1827], 20.00th=[ 1926], | 30.00th=[ 2008], 40.00th=[ 2114], 50.00th=[ 2311], 60.00th=[ 2868], | 70.00th=[ 3523], 80.00th=[ 4293], 90.00th=[ 6980], 95.00th=[ 9110], | 99.00th=[15401], 99.50th=[20317], 99.90th=[45351], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 56, max= 336, per=25.20%, avg=178.42, stdev=62.75, samples=67 iops : min= 14, max= 84, avg=44.54, stdev=15.69, samples=67 write: IOPS=46, BW=186KiB/s (190kB/s)(6260KiB/33742msec) clat (msec): min=9, max=174, avg=18.03, stdev= 7.67 lat (msec): min=9, max=174, avg=18.03, stdev= 7.67 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 20], 80.00th=[ 22], 90.00th=[ 28], 95.00th=[ 33], | 99.00th=[ 40], 99.50th=[ 44], 99.90th=[ 61], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 72, max= 248, per=24.98%, avg=185.10, stdev=54.79, samples=67 iops : min= 18, max= 62, avg=46.21, stdev=13.71, samples=67 lat (msec) : 2=14.55%, 4=23.24%, 10=10.35%, 20=38.87%, 50=12.86% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.10%, sys=5.16%, ctx=3733, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24532: Sat Sep 14 06:49:09 2024 read: IOPS=45, BW=180KiB/s (185kB/s)(6052KiB/33558msec) clat (usec): min=1377, max=43504, avg=3432.05, stdev=2912.81 lat (usec): min=1378, max=43505, avg=3432.71, stdev=2912.82 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1778], 10.00th=[ 1827], 20.00th=[ 1909], | 30.00th=[ 1975], 40.00th=[ 2089], 50.00th=[ 2245], 60.00th=[ 2606], | 70.00th=[ 3392], 80.00th=[ 4228], 90.00th=[ 6390], 95.00th=[ 8979], | 99.00th=[14222], 99.50th=[16909], 99.90th=[35914], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 55, max= 351, per=25.47%, avg=180.33, stdev=71.85, samples=67 iops : min= 13, max= 87, avg=45.01, stdev=17.96, samples=67 write: IOPS=46, BW=186KiB/s (190kB/s)(6236KiB/33558msec) clat (msec): min=8, max=173, avg=18.13, stdev= 7.75 lat (msec): min=8, max=173, avg=18.13, stdev= 7.75 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 20], 80.00th=[ 22], 90.00th=[ 28], 95.00th=[ 33], | 99.00th=[ 41], 99.50th=[ 44], 99.90th=[ 59], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 72, max= 264, per=25.04%, avg=185.58, stdev=53.60, samples=67 iops : min= 18, max= 66, avg=46.33, stdev=13.42, samples=67 lat (msec) : 2=15.85%, 4=22.59%, 10=9.64%, 20=39.06%, 50=12.73% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.07%, sys=5.19%, ctx=3739, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=709KiB/s (726kB/s), 175KiB/s-180KiB/s (179kB/s-185kB/s), io=23.4MiB (24.6MB), run=33404-33889msec WRITE: bw=742KiB/s (760kB/s), 186KiB/s-188KiB/s (190kB/s-193kB/s), io=24.6MiB (25.7MB), run=33404-33889msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24584: Sat Sep 14 06:49:14 2024 read: IOPS=85, BW=1373KiB/s (1406kB/s)(5920KiB/4311msec) clat (usec): min=49, max=47167, avg=8508.14, stdev=5842.27 lat (usec): min=49, max=47168, avg=8508.74, stdev=5842.31 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 61], 10.00th=[ 63], 20.00th=[ 77], | 30.00th=[ 7963], 40.00th=[ 8848], 50.00th=[ 9896], 60.00th=[10552], | 70.00th=[11338], 80.00th=[11994], 90.00th=[13566], 95.00th=[15533], | 99.00th=[23725], 99.50th=[35390], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 736, max= 1568, per=23.95%, avg=1312.00, stdev=273.68, samples=8 iops : min= 46, max= 98, avg=82.00, stdev=17.10, samples=8 write: IOPS=92, BW=1477KiB/s (1513kB/s)(6368KiB/4311msec) clat (usec): min=623, max=46151, avg=2876.15, stdev=3549.53 lat (usec): min=624, max=46153, avg=2877.13, stdev=3549.57 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 709], 10.00th=[ 742], 20.00th=[ 783], | 30.00th=[ 832], 40.00th=[ 922], 50.00th=[ 1205], 60.00th=[ 2704], | 70.00th=[ 3294], 80.00th=[ 4621], 90.00th=[ 6849], 95.00th=[ 8586], | 99.00th=[15926], 99.50th=[17695], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 896, max= 2336, per=25.06%, avg=1444.00, stdev=464.18, samples=8 iops : min= 56, max= 146, avg=90.25, stdev=29.01, samples=8 lat (usec) : 50=0.13%, 100=11.20%, 250=0.39%, 750=5.73%, 1000=17.19% lat (msec) : 2=5.08%, 4=11.85%, 10=23.83%, 20=23.70%, 50=0.91% cpu : usr=0.12%, sys=22.65%, ctx=1964, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24585: Sat Sep 14 06:49:14 2024 read: IOPS=83, BW=1333KiB/s (1365kB/s)(5744KiB/4308msec) clat (usec): min=49, max=32872, avg=8198.90, stdev=5401.07 lat (usec): min=49, max=32873, avg=8201.07, stdev=5398.78 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 61], 10.00th=[ 64], 20.00th=[ 80], | 30.00th=[ 7701], 40.00th=[ 8979], 50.00th=[ 9634], 60.00th=[10290], | 70.00th=[11076], 80.00th=[11994], 90.00th=[13698], 95.00th=[15008], | 99.00th=[20579], 99.50th=[21890], 99.90th=[32900], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 544, max= 1788, per=23.57%, avg=1291.12, stdev=389.82, samples=8 iops : min= 34, max= 111, avg=80.50, stdev=24.27, samples=8 write: IOPS=94, BW=1519KiB/s (1555kB/s)(6544KiB/4308msec) clat (usec): min=620, max=75028, avg=3152.98, stdev=5180.50 lat (usec): min=621, max=75030, avg=3154.05, stdev=5180.57 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 685], 10.00th=[ 734], 20.00th=[ 783], | 30.00th=[ 840], 40.00th=[ 971], 50.00th=[ 1696], 60.00th=[ 2704], | 70.00th=[ 3163], 80.00th=[ 4359], 90.00th=[ 6587], 95.00th=[ 9503], | 99.00th=[21627], 99.50th=[31065], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 864, max= 1952, per=25.32%, avg=1459.12, stdev=308.87, samples=8 iops : min= 54, max= 122, avg=91.00, stdev=19.34, samples=8 lat (usec) : 50=0.26%, 100=10.55%, 250=0.78%, 500=0.26%, 750=6.77% lat (usec) : 1000=15.49% lat (msec) : 2=5.73%, 4=14.58%, 10=22.14%, 20=22.27%, 50=1.04% lat (msec) : 100=0.13% cpu : usr=0.19%, sys=21.50%, ctx=1868, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24586: Sat Sep 14 06:49:14 2024 read: IOPS=89, BW=1431KiB/s (1465kB/s)(6256KiB/4372msec) clat (usec): min=48, max=29948, avg=7967.36, stdev=5604.69 lat (usec): min=49, max=29949, avg=7967.94, stdev=5604.71 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 58], 10.00th=[ 62], 20.00th=[ 77], | 30.00th=[ 6915], 40.00th=[ 8455], 50.00th=[ 9503], 60.00th=[10421], | 70.00th=[11338], 80.00th=[12256], 90.00th=[13960], 95.00th=[15270], | 99.00th=[21103], 99.50th=[23987], 99.90th=[30016], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 800, max= 1820, per=24.74%, avg=1355.12, stdev=341.24, samples=8 iops : min= 50, max= 113, avg=84.50, stdev=21.29, samples=8 write: IOPS=86, BW=1380KiB/s (1413kB/s)(6032KiB/4372msec) clat (usec): min=631, max=59787, avg=3110.73, stdev=4492.99 lat (usec): min=632, max=59789, avg=3111.86, stdev=4492.97 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 693], 10.00th=[ 734], 20.00th=[ 775], | 30.00th=[ 840], 40.00th=[ 963], 50.00th=[ 1352], 60.00th=[ 2540], | 70.00th=[ 3458], 80.00th=[ 4686], 90.00th=[ 6718], 95.00th=[ 9372], | 99.00th=[19268], 99.50th=[26346], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 704, max= 1728, per=22.96%, avg=1323.25, stdev=346.46, samples=8 iops : min= 44, max= 108, avg=82.50, stdev=21.67, samples=8 lat (usec) : 50=0.26%, 100=12.50%, 250=1.30%, 500=0.13%, 750=7.03% lat (usec) : 1000=14.58% lat (msec) : 2=5.60%, 4=10.29%, 10=23.57%, 20=23.83%, 50=0.78% lat (msec) : 100=0.13% cpu : usr=0.07%, sys=21.55%, ctx=1940, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24587: Sat Sep 14 06:49:14 2024 read: IOPS=86, BW=1385KiB/s (1418kB/s)(6032KiB/4356msec) clat (usec): min=48, max=23113, avg=8118.09, stdev=5181.57 lat (usec): min=49, max=23114, avg=8118.73, stdev=5181.65 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 61], 10.00th=[ 65], 20.00th=[ 84], | 30.00th=[ 7767], 40.00th=[ 8717], 50.00th=[ 9503], 60.00th=[10159], | 70.00th=[11338], 80.00th=[12256], 90.00th=[13566], 95.00th=[15008], | 99.00th=[17171], 99.50th=[19268], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 704, max= 1600, per=24.32%, avg=1332.00, stdev=293.25, samples=8 iops : min= 44, max= 100, avg=83.25, stdev=18.33, samples=8 write: IOPS=89, BW=1436KiB/s (1471kB/s)(6256KiB/4356msec) clat (usec): min=639, max=54226, avg=3082.77, stdev=4235.77 lat (usec): min=640, max=54227, avg=3083.76, stdev=4235.82 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 693], 10.00th=[ 734], 20.00th=[ 791], | 30.00th=[ 832], 40.00th=[ 938], 50.00th=[ 1680], 60.00th=[ 2606], | 70.00th=[ 3556], 80.00th=[ 4817], 90.00th=[ 6587], 95.00th=[ 8160], | 99.00th=[22152], 99.50th=[25035], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 768, max= 1824, per=24.50%, avg=1412.00, stdev=322.93, samples=8 iops : min= 48, max= 114, avg=88.25, stdev=20.18, samples=8 lat (usec) : 50=0.13%, 100=11.33%, 250=0.78%, 750=5.99%, 1000=15.76% lat (msec) : 2=4.82%, 4=11.46%, 10=28.12%, 20=20.83%, 50=0.65% lat (msec) : 100=0.13% cpu : usr=0.14%, sys=21.29%, ctx=2043, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5478KiB/s (5610kB/s), 1333KiB/s-1431KiB/s (1365kB/s-1465kB/s), io=23.4MiB (24.5MB), run=4308-4372msec WRITE: bw=5764KiB/s (5902kB/s), 1380KiB/s-1519KiB/s (1413kB/s-1555kB/s), io=24.6MiB (25.8MB), run=4308-4372msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24580: Sat Sep 14 06:49:19 2024 read: IOPS=41, BW=667KiB/s (684kB/s)(5920KiB/8869msec) clat (usec): min=1810, max=37349, avg=3809.44, stdev=3622.13 lat (usec): min=1811, max=37351, avg=3810.14, stdev=3622.16 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 1958], 10.00th=[ 2008], 20.00th=[ 2089], | 30.00th=[ 2180], 40.00th=[ 2311], 50.00th=[ 2474], 60.00th=[ 2606], | 70.00th=[ 3097], 80.00th=[ 4555], 90.00th=[ 7111], 95.00th=[10945], | 99.00th=[19006], 99.50th=[25560], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 384, max= 1120, per=24.99%, avg=656.82, stdev=214.97, samples=17 iops : min= 24, max= 70, avg=41.00, stdev=13.43, samples=17 write: IOPS=44, BW=718KiB/s (735kB/s)(6368KiB/8869msec) clat (usec): min=10430, max=45409, avg=18699.88, stdev=6220.71 lat (usec): min=10431, max=45410, avg=18701.23, stdev=6220.72 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11338], 10.00th=[11994], 20.00th=[13304], | 30.00th=[14746], 40.00th=[16450], 50.00th=[17695], 60.00th=[18744], | 70.00th=[20317], 80.00th=[22676], 90.00th=[27395], 95.00th=[32375], | 99.00th=[40633], 99.50th=[41681], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 416, max= 960, per=25.52%, avg=705.76, stdev=183.75, samples=17 iops : min= 26, max= 60, avg=44.06, stdev=11.54, samples=17 lat (msec) : 2=4.04%, 4=32.55%, 10=8.59%, 20=38.41%, 50=16.41% cpu : usr=0.05%, sys=5.76%, ctx=880, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24581: Sat Sep 14 06:49:19 2024 read: IOPS=39, BW=630KiB/s (645kB/s)(5744KiB/9113msec) clat (usec): min=1725, max=30937, avg=3684.87, stdev=3248.60 lat (usec): min=1726, max=30937, avg=3685.55, stdev=3248.60 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1958], 10.00th=[ 2008], 20.00th=[ 2089], | 30.00th=[ 2147], 40.00th=[ 2245], 50.00th=[ 2442], 60.00th=[ 2638], | 70.00th=[ 3130], 80.00th=[ 4228], 90.00th=[ 7570], 95.00th=[10421], | 99.00th=[14353], 99.50th=[26608], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 288, max= 1184, per=24.03%, avg=631.39, stdev=261.78, samples=18 iops : min= 18, max= 74, avg=39.44, stdev=16.36, samples=18 write: IOPS=44, BW=718KiB/s (735kB/s)(6544KiB/9113msec) clat (usec): min=9721, max=79901, avg=18848.63, stdev=6849.82 lat (usec): min=9722, max=79902, avg=18849.92, stdev=6849.85 clat percentiles (usec): | 1.00th=[10552], 5.00th=[11469], 10.00th=[12125], 20.00th=[13304], | 30.00th=[14746], 40.00th=[16712], 50.00th=[17695], 60.00th=[19006], | 70.00th=[20317], 80.00th=[22414], 90.00th=[26870], 95.00th=[31327], | 99.00th=[39584], 99.50th=[46400], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 448, max= 960, per=25.89%, avg=715.94, stdev=175.48, samples=18 iops : min= 28, max= 60, avg=44.72, stdev=10.94, samples=18 lat (msec) : 2=4.30%, 4=32.29%, 10=7.94%, 20=38.15%, 50=17.19% lat (msec) : 100=0.13% cpu : usr=0.09%, sys=5.65%, ctx=860, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24582: Sat Sep 14 06:49:19 2024 read: IOPS=44, BW=717KiB/s (734kB/s)(6256KiB/8722msec) clat (usec): min=1780, max=24243, avg=3957.67, stdev=3512.81 lat (usec): min=1780, max=24244, avg=3958.35, stdev=3512.81 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 1958], 10.00th=[ 2008], 20.00th=[ 2073], | 30.00th=[ 2180], 40.00th=[ 2278], 50.00th=[ 2442], 60.00th=[ 2671], | 70.00th=[ 3228], 80.00th=[ 5014], 90.00th=[ 7701], 95.00th=[12125], | 99.00th=[18482], 99.50th=[23462], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 384, max= 1280, per=27.34%, avg=718.53, stdev=266.65, samples=17 iops : min= 24, max= 80, avg=44.71, stdev=16.66, samples=17 write: IOPS=43, BW=692KiB/s (708kB/s)(6032KiB/8722msec) clat (usec): min=9798, max=56871, avg=18874.25, stdev=6293.06 lat (usec): min=9800, max=56873, avg=18875.50, stdev=6293.20 clat percentiles (usec): | 1.00th=[10814], 5.00th=[11338], 10.00th=[11863], 20.00th=[13566], | 30.00th=[15270], 40.00th=[16909], 50.00th=[17957], 60.00th=[19006], | 70.00th=[20317], 80.00th=[22676], 90.00th=[26870], 95.00th=[29754], | 99.00th=[38536], 99.50th=[55837], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 384, max= 896, per=24.76%, avg=684.71, stdev=177.85, samples=17 iops : min= 24, max= 56, avg=42.59, stdev=11.13, samples=17 lat (msec) : 2=5.08%, 4=32.68%, 10=9.11%, 20=37.37%, 50=15.49% lat (msec) : 100=0.26% cpu : usr=0.07%, sys=6.00%, ctx=879, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24583: Sat Sep 14 06:49:19 2024 read: IOPS=42, BW=688KiB/s (704kB/s)(6032KiB/8769msec) clat (usec): min=1733, max=26435, avg=3770.44, stdev=3197.23 lat (usec): min=1733, max=26436, avg=3771.10, stdev=3197.23 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 1942], 10.00th=[ 2008], 20.00th=[ 2057], | 30.00th=[ 2147], 40.00th=[ 2278], 50.00th=[ 2540], 60.00th=[ 2835], | 70.00th=[ 3261], 80.00th=[ 4555], 90.00th=[ 7111], 95.00th=[10028], | 99.00th=[17695], 99.50th=[22414], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 384, max= 1088, per=25.78%, avg=677.53, stdev=211.70, samples=17 iops : min= 24, max= 68, avg=42.29, stdev=13.22, samples=17 write: IOPS=44, BW=713KiB/s (731kB/s)(6256KiB/8769msec) clat (usec): min=10418, max=54140, avg=18584.19, stdev=5674.34 lat (usec): min=10419, max=54143, avg=18585.55, stdev=5674.56 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11207], 10.00th=[11863], 20.00th=[13435], | 30.00th=[15401], 40.00th=[16909], 50.00th=[17957], 60.00th=[19006], | 70.00th=[20579], 80.00th=[22938], 90.00th=[26084], 95.00th=[29230], | 99.00th=[34341], 99.50th=[38536], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 416, max= 960, per=25.59%, avg=707.65, stdev=178.60, samples=17 iops : min= 26, max= 60, avg=44.18, stdev=11.22, samples=17 lat (msec) : 2=4.17%, 4=33.46%, 10=8.85%, 20=36.59%, 50=16.80% lat (msec) : 100=0.13% cpu : usr=0.08%, sys=5.83%, ctx=876, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2628KiB/s (2691kB/s), 630KiB/s-717KiB/s (645kB/s-734kB/s), io=23.4MiB (24.5MB), run=8722-9113msec WRITE: bw=2765KiB/s (2832kB/s), 692KiB/s-718KiB/s (708kB/s-735kB/s), io=24.6MiB (25.8MB), run=8722-9113msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24600: Sat Sep 14 06:49:23 2024 read: IOPS=2, BW=2242KiB/s (2296kB/s)(7168KiB/3197msec) clat (usec): min=406, max=986677, avg=419356.14, stdev=397855.68 lat (usec): min=407, max=986678, avg=419357.52, stdev=397855.22 clat percentiles (usec): | 1.00th=[ 408], 5.00th=[ 408], 10.00th=[ 408], 20.00th=[ 79168], | 30.00th=[ 90702], 40.00th=[ 90702], 50.00th=[438305], 60.00th=[442500], | 70.00th=[442500], 80.00th=[893387], 90.00th=[985662], 95.00th=[985662], | 99.00th=[985662], 99.50th=[985662], 99.90th=[985662], 99.95th=[985662], | 99.99th=[985662] bw ( KiB/s): min= 1976, max= 2048, per=31.73%, avg=2032.60, stdev=31.71, samples=5 iops : min= 1, max= 2, avg= 1.60, stdev= 0.55, samples=5 write: IOPS=1, BW=1602KiB/s (1640kB/s)(5120KiB/3197msec) clat (usec): min=16637, max=98388, avg=48296.82, stdev=30647.82 lat (usec): min=16677, max=98461, avg=48359.68, stdev=30657.56 clat percentiles (usec): | 1.00th=[16581], 5.00th=[16581], 10.00th=[16581], 20.00th=[16581], | 30.00th=[34341], 40.00th=[34341], 50.00th=[41681], 60.00th=[41681], | 70.00th=[50594], 80.00th=[50594], 90.00th=[98042], 95.00th=[98042], | 99.00th=[98042], 99.50th=[98042], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 2048, max= 5930, per=44.48%, avg=3989.00, stdev=2744.99, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (usec) : 500=8.33% lat (msec) : 20=8.33%, 50=16.67%, 100=33.33%, 500=16.67%, 1000=16.67% cpu : usr=0.00%, sys=33.51%, ctx=821, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24603: Sat Sep 14 06:49:23 2024 read: IOPS=0, BW=743KiB/s (761kB/s)(2048KiB/2756msec) clat (msec): min=46, max=1145, avg=596.26, stdev=777.07 lat (msec): min=46, max=1145, avg=596.26, stdev=777.07 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 1150], | 70.00th=[ 1150], 80.00th=[ 1150], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 2048, max= 2048, per=31.97%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3716KiB/s (3805kB/s)(10.0MiB/2756msec) clat (msec): min=22, max=1108, avg=149.08, stdev=337.47 lat (msec): min=23, max=1108, avg=149.13, stdev=337.45 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 25], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 38], 60.00th=[ 44], | 70.00th=[ 51], 80.00th=[ 51], 90.00th=[ 88], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=66.67%, 100=16.67% cpu : usr=0.00%, sys=16.44%, ctx=354, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24605: Sat Sep 14 06:49:23 2024 read: IOPS=1, BW=1995KiB/s (2043kB/s)(6144KiB/3079msec) clat (usec): min=310, max=273477, avg=52384.18, stdev=109425.63 lat (usec): min=311, max=273478, avg=52385.03, stdev=109425.79 clat percentiles (usec): | 1.00th=[ 310], 5.00th=[ 310], 10.00th=[ 310], 20.00th=[ 375], | 30.00th=[ 375], 40.00th=[ 383], 50.00th=[ 383], 60.00th=[ 461], | 70.00th=[ 39060], 80.00th=[ 39060], 90.00th=[274727], 95.00th=[274727], | 99.00th=[274727], 99.50th=[274727], 99.90th=[274727], 99.95th=[274727], | 99.99th=[274727] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1995KiB/s (2043kB/s)(6144KiB/3079msec) clat (msec): min=15, max=2374, avg=448.89, stdev=944.69 lat (msec): min=15, max=2374, avg=448.93, stdev=944.69 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 17], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 75], | 70.00th=[ 163], 80.00th=[ 163], 90.00th=[ 2366], 95.00th=[ 2366], | 99.00th=[ 2366], 99.50th=[ 2366], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 4096, max= 6144, per=57.09%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (usec) : 500=33.33% lat (msec) : 20=16.67%, 50=16.67%, 100=8.33%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=10.17%, ctx=124, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24607: Sat Sep 14 06:49:23 2024 read: IOPS=1, BW=1649KiB/s (1689kB/s)(5120KiB/3105msec) clat (usec): min=261, max=125503, avg=37617.22, stdev=52423.37 lat (usec): min=262, max=125503, avg=37617.98, stdev=52423.17 clat percentiles (usec): | 1.00th=[ 262], 5.00th=[ 262], 10.00th=[ 262], 20.00th=[ 262], | 30.00th=[ 351], 40.00th=[ 351], 50.00th=[ 16909], 60.00th=[ 16909], | 70.00th=[ 44827], 80.00th=[ 44827], 90.00th=[125305], 95.00th=[125305], | 99.00th=[125305], 99.50th=[125305], 99.90th=[125305], 99.95th=[125305], | 99.99th=[125305] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2309KiB/s (2364kB/s)(7168KiB/3105msec) clat (msec): min=23, max=2402, avg=403.12, stdev=884.19 lat (msec): min=23, max=2402, avg=403.17, stdev=884.18 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 25], | 30.00th=[ 25], 40.00th=[ 25], 50.00th=[ 39], 60.00th=[ 111], | 70.00th=[ 111], 80.00th=[ 199], 90.00th=[ 2400], 95.00th=[ 2400], | 99.00th=[ 2400], 99.50th=[ 2400], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 2048, max=10240, per=68.51%, avg=6144.00, stdev=5792.62, samples=2 iops : min= 2, max= 10, avg= 6.00, stdev= 5.66, samples=2 lat (usec) : 500=16.67% lat (msec) : 20=8.33%, 50=41.67%, 250=25.00% cpu : usr=0.00%, sys=7.22%, ctx=96, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6406KiB/s (6560kB/s), 743KiB/s-2242KiB/s (761kB/s-2296kB/s), io=20.0MiB (20.0MB), run=2756-3197msec WRITE: bw=8968KiB/s (9184kB/s), 1602KiB/s-3716KiB/s (1640kB/s-3805kB/s), io=28.0MiB (29.4MB), run=2756-3197msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24601: Sat Sep 14 06:49:23 2024 read: IOPS=2, BW=2757KiB/s (2823kB/s)(7168KiB/2600msec) clat (msec): min=18, max=995, avg=221.02, stdev=359.94 lat (msec): min=18, max=995, avg=221.02, stdev=359.94 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 23], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 40], 60.00th=[ 92], | 70.00th=[ 92], 80.00th=[ 342], 90.00th=[ 995], 95.00th=[ 995], | 99.00th=[ 995], 99.50th=[ 995], 99.90th=[ 995], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 2048, max= 4096, per=56.57%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 write: IOPS=1, BW=1969KiB/s (2016kB/s)(5120KiB/2600msec) clat (msec): min=29, max=841, avg=207.45, stdev=355.19 lat (msec): min=29, max=841, avg=207.51, stdev=355.19 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 30], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 46], 60.00th=[ 46], | 70.00th=[ 88], 80.00th=[ 88], 90.00th=[ 844], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 2048, max= 4096, per=36.37%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (msec) : 20=8.33%, 50=50.00%, 100=16.67%, 500=8.33%, 1000=16.67% cpu : usr=0.00%, sys=4.35%, ctx=38, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24602: Sat Sep 14 06:49:23 2024 read: IOPS=0, BW=603KiB/s (618kB/s)(2048KiB/3394msec) clat (msec): min=20, max=1350, avg=685.26, stdev=940.59 lat (msec): min=20, max=1350, avg=685.26, stdev=940.59 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 21], 60.00th=[ 1351], | 70.00th=[ 1351], 80.00th=[ 1351], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 1787, max= 2048, per=31.78%, avg=1917.50, stdev=184.55, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=3017KiB/s (3089kB/s)(10.0MiB/3394msec) clat (msec): min=35, max=1126, avg=196.92, stdev=340.25 lat (msec): min=35, max=1126, avg=196.98, stdev=340.24 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 41], | 30.00th=[ 42], 40.00th=[ 43], 50.00th=[ 44], 60.00th=[ 68], | 70.00th=[ 120], 80.00th=[ 120], 90.00th=[ 355], 95.00th=[ 1133], | 99.00th=[ 1133], 99.50th=[ 1133], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 3574, max=10240, per=81.77%, avg=6907.00, stdev=4713.57, samples=2 iops : min= 3, max= 10, avg= 6.50, stdev= 4.95, samples=2 lat (msec) : 50=50.00%, 100=8.33%, 250=16.67%, 500=8.33% cpu : usr=0.09%, sys=3.63%, ctx=52, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24604: Sat Sep 14 06:49:23 2024 read: IOPS=1, BW=1853KiB/s (1897kB/s)(6144KiB/3316msec) clat (msec): min=14, max=260, avg=79.01, stdev=92.82 lat (msec): min=14, max=260, avg=79.01, stdev=92.82 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 17], 40.00th=[ 34], 50.00th=[ 34], 60.00th=[ 72], | 70.00th=[ 78], 80.00th=[ 78], 90.00th=[ 262], 95.00th=[ 262], | 99.00th=[ 262], 99.50th=[ 262], 99.90th=[ 262], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 3586, max= 3586, per=59.43%, avg=3586.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1853KiB/s (1897kB/s)(6144KiB/3316msec) clat (msec): min=26, max=2471, avg=459.85, stdev=986.52 lat (msec): min=26, max=2471, avg=459.90, stdev=986.53 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 54], | 70.00th=[ 138], 80.00th=[ 138], 90.00th=[ 2467], 95.00th=[ 2467], | 99.00th=[ 2467], 99.50th=[ 2467], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 3586, max= 4096, per=45.47%, avg=3841.00, stdev=360.62, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 lat (msec) : 20=16.67%, 50=33.33%, 100=25.00%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=2.11%, ctx=44, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24606: Sat Sep 14 06:49:23 2024 read: IOPS=1, BW=1932KiB/s (1978kB/s)(5120KiB/2650msec) clat (usec): min=11931, max=38519, avg=19765.52, stdev=11283.81 lat (usec): min=11931, max=38520, avg=19766.23, stdev=11283.86 clat percentiles (usec): | 1.00th=[11994], 5.00th=[11994], 10.00th=[11994], 20.00th=[11994], | 30.00th=[12649], 40.00th=[12649], 50.00th=[13435], 60.00th=[13435], | 70.00th=[22152], 80.00th=[22152], 90.00th=[38536], 95.00th=[38536], | 99.00th=[38536], 99.50th=[38536], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2705KiB/s (2770kB/s)(7168KiB/2650msec) clat (msec): min=26, max=1520, avg=361.15, stdev=573.98 lat (msec): min=26, max=1520, avg=361.23, stdev=573.98 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 37], | 30.00th=[ 45], 40.00th=[ 45], 50.00th=[ 46], 60.00th=[ 105], | 70.00th=[ 105], 80.00th=[ 751], 90.00th=[ 1519], 95.00th=[ 1519], | 99.00th=[ 1519], 99.50th=[ 1519], 99.90th=[ 1519], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 2048, max= 8192, per=60.61%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (msec) : 20=25.00%, 50=50.00%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=4.38%, ctx=49, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6034KiB/s (6179kB/s), 603KiB/s-2757KiB/s (618kB/s-2823kB/s), io=20.0MiB (20.0MB), run=2600-3394msec WRITE: bw=8448KiB/s (8651kB/s), 1853KiB/s-3017KiB/s (1897kB/s-3089kB/s), io=28.0MiB (29.4MB), run=2600-3394msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24615: Sat Sep 14 06:49:32 2024 read: IOPS=0, BW=1636KiB/s (1675kB/s)(12.0MiB/7512msec) clat (msec): min=129, max=3727, avg=2488.81, stdev=2043.85 lat (msec): min=129, max=3727, avg=2488.81, stdev=2043.85 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 130], 10.00th=[ 130], 20.00th=[ 130], | 30.00th=[ 130], 40.00th=[ 3608], 50.00th=[ 3608], 60.00th=[ 3608], | 70.00th=[ 3742], 80.00th=[ 3742], 90.00th=[ 3742], 95.00th=[ 3742], | 99.00th=[ 3742], 99.50th=[ 3742], 99.90th=[ 3742], 99.95th=[ 3742], | 99.99th=[ 3742] bw ( KiB/s): min= 5603, max= 8192, per=100.00%, avg=6897.50, stdev=1830.70, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.03%, sys=16.03%, ctx=2641, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24616: Sat Sep 14 06:49:32 2024 read: IOPS=0, BW=1037KiB/s (1062kB/s)(4096KiB/3948msec) clat (nsec): min=194809k, max=194809k, avg=194809079.00, stdev= 0.00 lat (nsec): min=194812k, max=194812k, avg=194811856.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 194], 10.00th=[ 194], 20.00th=[ 194], | 30.00th=[ 194], 40.00th=[ 194], 50.00th=[ 194], 60.00th=[ 194], | 70.00th=[ 194], 80.00th=[ 194], 90.00th=[ 194], 95.00th=[ 194], | 99.00th=[ 194], 99.50th=[ 194], 99.90th=[ 194], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2075KiB/s (2125kB/s)(8192KiB/3948msec) clat (msec): min=95, max=3608, avg=1851.64, stdev=2483.91 lat (msec): min=95, max=3608, avg=1851.93, stdev=2483.88 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 95], 10.00th=[ 95], 20.00th=[ 95], | 30.00th=[ 95], 40.00th=[ 95], 50.00th=[ 95], 60.00th=[ 3608], | 70.00th=[ 3608], 80.00th=[ 3608], 90.00th=[ 3608], 95.00th=[ 3608], | 99.00th=[ 3608], 99.50th=[ 3608], 99.90th=[ 3608], 99.95th=[ 3608], | 99.99th=[ 3608] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=6.06%, ctx=128, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24617: Sat Sep 14 06:49:32 2024 write: IOPS=0, BW=1627KiB/s (1666kB/s)(12.0MiB/7552msec) clat (msec): min=193, max=3725, avg=2510.45, stdev=2006.97 lat (msec): min=194, max=3726, avg=2510.82, stdev=2007.02 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 194], 10.00th=[ 194], 20.00th=[ 194], | 30.00th=[ 194], 40.00th=[ 3608], 50.00th=[ 3608], 60.00th=[ 3608], | 70.00th=[ 3742], 80.00th=[ 3742], 90.00th=[ 3742], 95.00th=[ 3742], | 99.00th=[ 3742], 99.50th=[ 3742], 99.90th=[ 3742], 99.95th=[ 3742], | 99.99th=[ 3742] bw ( KiB/s): min= 7861, max= 8192, per=100.00%, avg=8026.50, stdev=234.05, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.08%, sys=8.28%, ctx=1299, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24618: Sat Sep 14 06:49:32 2024 write: IOPS=2, BW=9267KiB/s (9489kB/s)(12.0MiB/1326msec) clat (msec): min=102, max=1095, avg=434.82, stdev=571.97 lat (msec): min=102, max=1095, avg=435.12, stdev=571.93 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 103], 10.00th=[ 103], 20.00th=[ 103], | 30.00th=[ 103], 40.00th=[ 107], 50.00th=[ 107], 60.00th=[ 107], | 70.00th=[ 1099], 80.00th=[ 1099], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=19.62%, ctx=75, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2181KiB/s (2233kB/s), 1037KiB/s-1636KiB/s (1062kB/s-1675kB/s), io=16.0MiB (16.8MB), run=3948-7512msec WRITE: bw=4339KiB/s (4443kB/s), 1627KiB/s-9267KiB/s (1666kB/s-9489kB/s), io=32.0MiB (33.6MB), run=1326-7552msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24619: Sat Sep 14 06:49:32 2024 read: IOPS=0, BW=1618KiB/s (1657kB/s)(12.0MiB/7593msec) clat (msec): min=1025, max=3629, avg=2504.24, stdev=1337.33 lat (msec): min=1025, max=3629, avg=2504.24, stdev=1337.33 clat percentiles (msec): | 1.00th=[ 1028], 5.00th=[ 1028], 10.00th=[ 1028], 20.00th=[ 1028], | 30.00th=[ 1028], 40.00th=[ 2869], 50.00th=[ 2869], 60.00th=[ 2869], | 70.00th=[ 3641], 80.00th=[ 3641], 90.00th=[ 3641], 95.00th=[ 3641], | 99.00th=[ 3641], 99.50th=[ 3641], 99.90th=[ 3641], 99.95th=[ 3641], | 99.99th=[ 3641] bw ( KiB/s): min= 5520, max= 5527, per=100.00%, avg=5523.50, stdev= 4.95, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 cpu : usr=0.00%, sys=1.41%, ctx=46, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24620: Sat Sep 14 06:49:32 2024 read: IOPS=0, BW=851KiB/s (872kB/s)(4096KiB/4812msec) clat (nsec): min=1066.1M, max=1066.1M, avg=1066093655.00, stdev= 0.00 lat (nsec): min=1066.1M, max=1066.1M, avg=1066094984.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1070], 5.00th=[ 1070], 10.00th=[ 1070], 20.00th=[ 1070], | 30.00th=[ 1070], 40.00th=[ 1070], 50.00th=[ 1070], 60.00th=[ 1070], | 70.00th=[ 1070], 80.00th=[ 1070], 90.00th=[ 1070], 95.00th=[ 1070], | 99.00th=[ 1070], 99.50th=[ 1070], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 5595, max= 5595, per=100.00%, avg=5595.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1702KiB/s (1743kB/s)(8192KiB/4812msec) clat (msec): min=114, max=3560, avg=1837.21, stdev=2436.79 lat (msec): min=114, max=3560, avg=1837.34, stdev=2436.79 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 114], 10.00th=[ 114], 20.00th=[ 114], | 30.00th=[ 114], 40.00th=[ 114], 50.00th=[ 114], 60.00th=[ 3574], | 70.00th=[ 3574], 80.00th=[ 3574], 90.00th=[ 3574], 95.00th=[ 3574], | 99.00th=[ 3574], 99.50th=[ 3574], 99.90th=[ 3574], 99.95th=[ 3574], | 99.99th=[ 3574] bw ( KiB/s): min= 5595, max= 5595, per=100.00%, avg=5595.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33% cpu : usr=0.02%, sys=2.33%, ctx=85, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24621: Sat Sep 14 06:49:32 2024 write: IOPS=0, BW=1605KiB/s (1643kB/s)(12.0MiB/7658msec) clat (msec): min=1317, max=3531, avg=2535.75, stdev=1123.91 lat (msec): min=1317, max=3531, avg=2536.05, stdev=1123.93 clat percentiles (msec): | 1.00th=[ 1318], 5.00th=[ 1318], 10.00th=[ 1318], 20.00th=[ 1318], | 30.00th=[ 1318], 40.00th=[ 2769], 50.00th=[ 2769], 60.00th=[ 2769], | 70.00th=[ 3540], 80.00th=[ 3540], 90.00th=[ 3540], 95.00th=[ 3540], | 99.00th=[ 3540], 99.50th=[ 3540], 99.90th=[ 3540], 99.95th=[ 3540], | 99.99th=[ 3540] bw ( KiB/s): min= 5468, max= 5468, per=100.00%, avg=5468.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 cpu : usr=0.04%, sys=1.42%, ctx=93, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24622: Sat Sep 14 06:49:32 2024 write: IOPS=2, BW=8777KiB/s (8988kB/s)(12.0MiB/1400msec) clat (msec): min=82, max=886, avg=439.58, stdev=409.72 lat (msec): min=82, max=887, avg=439.87, stdev=409.76 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 83], 10.00th=[ 83], 20.00th=[ 83], | 30.00th=[ 83], 40.00th=[ 351], 50.00th=[ 351], 60.00th=[ 351], | 70.00th=[ 885], 80.00th=[ 885], 90.00th=[ 885], 95.00th=[ 885], | 99.00th=[ 885], 99.50th=[ 885], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.07%, sys=8.43%, ctx=96, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2158KiB/s (2210kB/s), 851KiB/s-1618KiB/s (872kB/s-1657kB/s), io=16.0MiB (16.8MB), run=4812-7593msec WRITE: bw=4279KiB/s (4382kB/s), 1605KiB/s-8777KiB/s (1643kB/s-8988kB/s), io=32.0MiB (33.6MB), run=1400-7658msec