== sanity test 398b: DIO and buffer IO race ============== 13:57:22 (1736794642) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.14597 s, 43.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23094: Mon Jan 13 13:57:36 2025 read: IOPS=130, BW=520KiB/s (533kB/s)(5992KiB/11522msec) clat (usec): min=54, max=158175, avg=4300.52, stdev=4705.44 lat (usec): min=54, max=158176, avg=4301.51, stdev=4705.35 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 101], 10.00th=[ 1975], 20.00th=[ 2376], | 30.00th=[ 2737], 40.00th=[ 3195], 50.00th=[ 3785], 60.00th=[ 4424], | 70.00th=[ 5014], 80.00th=[ 5800], 90.00th=[ 7046], 95.00th=[ 8160], | 99.00th=[ 13304], 99.50th=[ 15008], 99.90th=[ 26346], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 320, max= 672, per=25.11%, avg=520.13, stdev=80.51, samples=23 iops : min= 80, max= 168, avg=129.96, stdev=20.13, samples=23 write: IOPS=136, BW=546KiB/s (560kB/s)(6296KiB/11522msec) clat (usec): min=413, max=21770, avg=3185.97, stdev=2793.29 lat (usec): min=414, max=21771, avg=3186.69, stdev=2793.29 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 510], 10.00th=[ 537], 20.00th=[ 586], | 30.00th=[ 725], 40.00th=[ 1860], 50.00th=[ 2638], 60.00th=[ 3294], | 70.00th=[ 4146], 80.00th=[ 5342], 90.00th=[ 7046], 95.00th=[ 8848], | 99.00th=[11600], 99.50th=[12911], 99.90th=[17171], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 376, max= 704, per=25.18%, avg=545.91, stdev=85.88, samples=23 iops : min= 94, max= 176, avg=136.39, stdev=21.51, samples=23 lat (usec) : 100=2.41%, 250=0.29%, 500=1.79%, 750=13.90%, 1000=1.89% lat (msec) : 2=5.89%, 4=35.38%, 10=35.71%, 20=2.60%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=21.65%, ctx=5534, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23096: Mon Jan 13 13:57:36 2025 read: IOPS=128, BW=515KiB/s (528kB/s)(5940KiB/11526msec) clat (usec): min=52, max=24926, avg=4110.64, stdev=2290.13 lat (usec): min=53, max=24927, avg=4111.78, stdev=2290.01 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 1795], 10.00th=[ 2057], 20.00th=[ 2409], | 30.00th=[ 2737], 40.00th=[ 3163], 50.00th=[ 3654], 60.00th=[ 4293], | 70.00th=[ 4948], 80.00th=[ 5604], 90.00th=[ 6718], 95.00th=[ 7963], | 99.00th=[12518], 99.50th=[13566], 99.90th=[17433], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 384, max= 616, per=24.88%, avg=515.26, stdev=62.77, samples=23 iops : min= 96, max= 154, avg=128.74, stdev=15.71, samples=23 write: IOPS=137, BW=551KiB/s (564kB/s)(6348KiB/11526msec) clat (usec): min=450, max=159615, avg=3376.93, stdev=4855.03 lat (usec): min=450, max=159616, avg=3377.84, stdev=4854.95 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 519], 10.00th=[ 537], 20.00th=[ 586], | 30.00th=[ 783], 40.00th=[ 2057], 50.00th=[ 2671], 60.00th=[ 3294], | 70.00th=[ 4359], 80.00th=[ 5538], 90.00th=[ 7242], 95.00th=[ 8586], | 99.00th=[ 12518], 99.50th=[ 14615], 99.90th=[ 20317], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 432, max= 744, per=25.35%, avg=549.70, stdev=78.70, samples=23 iops : min= 108, max= 186, avg=137.35, stdev=19.71, samples=23 lat (usec) : 100=1.69%, 250=0.26%, 500=1.27%, 750=13.83%, 1000=1.95% lat (msec) : 2=5.47%, 4=36.78%, 10=36.23%, 20=2.41%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=21.93%, ctx=5547, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23098: Mon Jan 13 13:57:36 2025 read: IOPS=132, BW=529KiB/s (542kB/s)(6028KiB/11390msec) clat (usec): min=56, max=20958, avg=4098.43, stdev=2292.76 lat (usec): min=56, max=20959, avg=4099.04, stdev=2292.79 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 1778], 10.00th=[ 2024], 20.00th=[ 2409], | 30.00th=[ 2704], 40.00th=[ 3097], 50.00th=[ 3654], 60.00th=[ 4228], | 70.00th=[ 4948], 80.00th=[ 5669], 90.00th=[ 6718], 95.00th=[ 7898], | 99.00th=[11863], 99.50th=[14091], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 328, max= 696, per=25.43%, avg=526.73, stdev=74.05, samples=22 iops : min= 82, max= 174, avg=131.59, stdev=18.57, samples=22 write: IOPS=137, BW=550KiB/s (563kB/s)(6260KiB/11390msec) clat (usec): min=395, max=160758, avg=3291.64, stdev=4906.77 lat (usec): min=395, max=160759, avg=3292.87, stdev=4907.46 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 519], 10.00th=[ 537], 20.00th=[ 578], | 30.00th=[ 766], 40.00th=[ 1762], 50.00th=[ 2638], 60.00th=[ 3228], | 70.00th=[ 4015], 80.00th=[ 5276], 90.00th=[ 7242], 95.00th=[ 8848], | 99.00th=[ 12518], 99.50th=[ 13960], 99.90th=[ 19530], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 408, max= 672, per=25.08%, avg=543.77, stdev=76.29, samples=22 iops : min= 102, max= 168, avg=135.86, stdev=19.13, samples=22 lat (usec) : 100=1.60%, 250=0.23%, 500=1.37%, 750=13.83%, 1000=2.54% lat (msec) : 2=6.09%, 4=37.17%, 10=34.64%, 20=2.44%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=21.70%, ctx=5607, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23100: Mon Jan 13 13:57:36 2025 read: IOPS=130, BW=522KiB/s (535kB/s)(6052KiB/11592msec) clat (usec): min=51, max=18377, avg=4170.16, stdev=2428.78 lat (usec): min=52, max=18378, avg=4171.00, stdev=2429.41 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 1663], 10.00th=[ 2008], 20.00th=[ 2376], | 30.00th=[ 2704], 40.00th=[ 3163], 50.00th=[ 3720], 60.00th=[ 4359], | 70.00th=[ 4948], 80.00th=[ 5669], 90.00th=[ 6915], 95.00th=[ 7963], | 99.00th=[13304], 99.50th=[15795], 99.90th=[17695], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 304, max= 694, per=25.05%, avg=518.78, stdev=86.54, samples=23 iops : min= 76, max= 173, avg=129.65, stdev=21.56, samples=23 write: IOPS=134, BW=538KiB/s (551kB/s)(6236KiB/11592msec) clat (usec): min=437, max=150387, avg=3340.08, stdev=4696.58 lat (usec): min=437, max=150389, avg=3341.11, stdev=4696.44 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 519], 10.00th=[ 537], 20.00th=[ 586], | 30.00th=[ 783], 40.00th=[ 2057], 50.00th=[ 2671], 60.00th=[ 3261], | 70.00th=[ 4146], 80.00th=[ 5407], 90.00th=[ 7308], 95.00th=[ 8848], | 99.00th=[ 11731], 99.50th=[ 12911], 99.90th=[ 25560], 99.95th=[149947], | 99.99th=[149947] bw ( KiB/s): min= 384, max= 752, per=24.68%, avg=535.13, stdev=94.05, samples=23 iops : min= 96, max= 188, avg=133.74, stdev=23.46, samples=23 lat (usec) : 100=1.95%, 250=0.26%, 500=1.27%, 750=13.64%, 1000=2.12% lat (msec) : 2=5.60%, 4=36.72%, 10=35.74%, 20=2.60%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=21.58%, ctx=5611, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2071KiB/s (2121kB/s), 515KiB/s-529KiB/s (528kB/s-542kB/s), io=23.4MiB (24.6MB), run=11390-11592msec WRITE: bw=2169KiB/s (2221kB/s), 538KiB/s-551KiB/s (551kB/s-564kB/s), io=24.6MiB (25.7MB), run=11390-11592msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23095: Mon Jan 13 13:57:40 2025 read: IOPS=98, BW=395KiB/s (404kB/s)(5992KiB/15184msec) clat (usec): min=1617, max=161905, avg=4121.36, stdev=5076.46 lat (usec): min=1618, max=161907, avg=4121.97, stdev=5076.48 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1827], 10.00th=[ 1942], 20.00th=[ 2114], | 30.00th=[ 2245], 40.00th=[ 2442], 50.00th=[ 2704], 60.00th=[ 3130], | 70.00th=[ 3949], 80.00th=[ 5538], 90.00th=[ 8455], 95.00th=[ 10552], | 99.00th=[ 15795], 99.50th=[ 17957], 99.90th=[ 26870], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 208, max= 752, per=24.83%, avg=392.50, stdev=159.70, samples=30 iops : min= 52, max= 188, avg=98.10, stdev=39.94, samples=30 write: IOPS=103, BW=415KiB/s (425kB/s)(6296KiB/15184msec) clat (usec): min=2088, max=29245, avg=5687.42, stdev=3177.52 lat (usec): min=2089, max=29245, avg=5688.15, stdev=3177.53 clat percentiles (usec): | 1.00th=[ 2507], 5.00th=[ 2868], 10.00th=[ 3064], 20.00th=[ 3359], | 30.00th=[ 3621], 40.00th=[ 3949], 50.00th=[ 4359], 60.00th=[ 5080], | 70.00th=[ 6456], 80.00th=[ 8029], 90.00th=[10159], 95.00th=[11863], | 99.00th=[16450], 99.50th=[18220], 99.90th=[24773], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 232, max= 736, per=24.84%, avg=411.17, stdev=159.13, samples=30 iops : min= 58, max= 184, avg=102.77, stdev=39.79, samples=30 lat (msec) : 2=6.54%, 4=49.32%, 10=35.58%, 20=8.27%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=10.88%, ctx=3569, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23097: Mon Jan 13 13:57:40 2025 read: IOPS=98, BW=392KiB/s (402kB/s)(5940KiB/15145msec) clat (usec): min=1595, max=161865, avg=4125.46, stdev=5049.91 lat (usec): min=1596, max=161867, avg=4126.06, stdev=5049.92 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1827], 10.00th=[ 1942], 20.00th=[ 2114], | 30.00th=[ 2245], 40.00th=[ 2442], 50.00th=[ 2704], 60.00th=[ 3130], | 70.00th=[ 3982], 80.00th=[ 5538], 90.00th=[ 8586], 95.00th=[ 10814], | 99.00th=[ 15008], 99.50th=[ 17171], 99.90th=[ 19792], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 160, max= 728, per=24.70%, avg=390.57, stdev=166.01, samples=30 iops : min= 40, max= 182, avg=97.57, stdev=41.54, samples=30 write: IOPS=104, BW=419KiB/s (429kB/s)(6348KiB/15145msec) clat (usec): min=2204, max=29252, avg=5646.24, stdev=3299.85 lat (usec): min=2205, max=29253, avg=5646.96, stdev=3299.86 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2769], 10.00th=[ 3064], 20.00th=[ 3359], | 30.00th=[ 3589], 40.00th=[ 3818], 50.00th=[ 4178], 60.00th=[ 4948], | 70.00th=[ 6390], 80.00th=[ 7963], 90.00th=[10159], 95.00th=[11731], | 99.00th=[17433], 99.50th=[20317], 99.90th=[28443], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 224, max= 736, per=25.13%, avg=415.90, stdev=159.53, samples=30 iops : min= 56, max= 184, avg=103.90, stdev=39.92, samples=30 lat (msec) : 2=6.09%, 4=51.43%, 10=33.98%, 20=8.17%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=10.72%, ctx=3522, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23099: Mon Jan 13 13:57:40 2025 read: IOPS=99, BW=398KiB/s (407kB/s)(6028KiB/15153msec) clat (usec): min=1515, max=26753, avg=4063.56, stdev=2993.13 lat (usec): min=1515, max=26754, avg=4064.21, stdev=2993.14 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1844], 10.00th=[ 1942], 20.00th=[ 2114], | 30.00th=[ 2278], 40.00th=[ 2442], 50.00th=[ 2704], 60.00th=[ 3195], | 70.00th=[ 4047], 80.00th=[ 5800], 90.00th=[ 8586], 95.00th=[10552], | 99.00th=[14091], 99.50th=[15795], 99.90th=[20055], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 176, max= 728, per=25.06%, avg=396.20, stdev=164.81, samples=30 iops : min= 44, max= 182, avg=99.00, stdev=41.24, samples=30 write: IOPS=103, BW=413KiB/s (423kB/s)(6260KiB/15153msec) clat (msec): min=2, max=164, avg= 5.73, stdev= 5.14 lat (msec): min=2, max=164, avg= 5.73, stdev= 5.14 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 18], 99.50th=[ 21], 99.90th=[ 28], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 216, max= 784, per=24.76%, avg=409.80, stdev=165.57, samples=30 iops : min= 54, max= 196, avg=102.40, stdev=41.42, samples=30 lat (msec) : 2=6.12%, 4=51.11%, 10=34.51%, 20=7.94%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=10.76%, ctx=3546, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23101: Mon Jan 13 13:57:40 2025 read: IOPS=99, BW=399KiB/s (409kB/s)(6052KiB/15150msec) clat (usec): min=1611, max=24323, avg=3993.58, stdev=2922.91 lat (usec): min=1611, max=24323, avg=3994.19, stdev=2922.91 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1827], 10.00th=[ 1958], 20.00th=[ 2114], | 30.00th=[ 2278], 40.00th=[ 2442], 50.00th=[ 2704], 60.00th=[ 3130], | 70.00th=[ 3916], 80.00th=[ 5669], 90.00th=[ 8160], 95.00th=[10421], | 99.00th=[14353], 99.50th=[17695], 99.90th=[20841], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 168, max= 744, per=25.06%, avg=396.13, stdev=159.59, samples=30 iops : min= 42, max= 186, avg=98.93, stdev=39.96, samples=30 write: IOPS=102, BW=412KiB/s (421kB/s)(6236KiB/15150msec) clat (msec): min=2, max=155, avg= 5.80, stdev= 4.98 lat (msec): min=2, max=155, avg= 5.80, stdev= 4.98 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 11], 95.00th=[ 12], | 99.00th=[ 18], 99.50th=[ 20], 99.90th=[ 29], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 192, max= 760, per=24.74%, avg=409.47, stdev=164.94, samples=30 iops : min= 48, max= 190, avg=102.27, stdev=41.29, samples=30 lat (msec) : 2=6.05%, 4=50.94%, 10=34.37%, 20=8.33%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=10.73%, ctx=3542, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1581KiB/s (1619kB/s), 392KiB/s-399KiB/s (402kB/s-409kB/s), io=23.4MiB (24.6MB), run=15145-15184msec WRITE: bw=1656KiB/s (1695kB/s), 412KiB/s-419KiB/s (421kB/s-429kB/s), io=24.6MiB (25.7MB), run=15145-15184msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23129: Mon Jan 13 13:57:44 2025 read: IOPS=122, BW=1958KiB/s (2005kB/s)(5920KiB/3023msec) clat (usec): min=61, max=18058, avg=4267.53, stdev=2880.80 lat (usec): min=61, max=18059, avg=4268.15, stdev=2880.77 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 73], 10.00th=[ 83], 20.00th=[ 2409], | 30.00th=[ 2835], 40.00th=[ 3326], 50.00th=[ 3982], 60.00th=[ 4555], | 70.00th=[ 5473], 80.00th=[ 6456], 90.00th=[ 7767], 95.00th=[ 9241], | 99.00th=[12518], 99.50th=[15139], 99.90th=[17957], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 1504, max= 2144, per=24.57%, avg=1940.67, stdev=255.50, samples=6 iops : min= 94, max= 134, avg=121.17, stdev=16.08, samples=6 write: IOPS=131, BW=2107KiB/s (2157kB/s)(6368KiB/3023msec) clat (usec): min=630, max=17345, avg=3599.03, stdev=3375.35 lat (usec): min=631, max=17346, avg=3599.86, stdev=3375.38 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 840], | 30.00th=[ 947], 40.00th=[ 1450], 50.00th=[ 2573], 60.00th=[ 3261], | 70.00th=[ 4293], 80.00th=[ 6128], 90.00th=[ 8848], 95.00th=[11076], | 99.00th=[14877], 99.50th=[17171], 99.90th=[17433], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 1760, max= 2656, per=25.28%, avg=2100.67, stdev=336.54, samples=6 iops : min= 110, max= 166, avg=131.17, stdev=21.13, samples=6 lat (usec) : 100=7.03%, 250=0.52%, 750=2.73%, 1000=13.80% lat (msec) : 2=6.64%, 4=28.39%, 10=35.03%, 20=5.86% cpu : usr=0.13%, sys=24.06%, ctx=1435, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23130: Mon Jan 13 13:57:44 2025 read: IOPS=119, BW=1905KiB/s (1951kB/s)(5744KiB/3015msec) clat (usec): min=60, max=14372, avg=4241.66, stdev=2603.94 lat (usec): min=60, max=14372, avg=4242.24, stdev=2603.95 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 72], 10.00th=[ 88], 20.00th=[ 2409], | 30.00th=[ 2868], 40.00th=[ 3359], 50.00th=[ 3949], 60.00th=[ 4686], | 70.00th=[ 5473], 80.00th=[ 6325], 90.00th=[ 7635], 95.00th=[ 8848], | 99.00th=[10814], 99.50th=[12125], 99.90th=[14353], 99.95th=[14353], | 99.99th=[14353] bw ( KiB/s): min= 1600, max= 2400, per=23.97%, avg=1893.33, stdev=313.15, samples=6 iops : min= 100, max= 150, avg=118.33, stdev=19.57, samples=6 write: IOPS=135, BW=2170KiB/s (2223kB/s)(6544KiB/3015msec) clat (usec): min=613, max=17272, avg=3583.55, stdev=3193.39 lat (usec): min=614, max=17272, avg=3584.41, stdev=3193.40 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 750], 10.00th=[ 824], 20.00th=[ 873], | 30.00th=[ 1020], 40.00th=[ 1729], 50.00th=[ 2802], 60.00th=[ 3392], | 70.00th=[ 4146], 80.00th=[ 5932], 90.00th=[ 8455], 95.00th=[ 9896], | 99.00th=[14222], 99.50th=[15664], 99.90th=[17171], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 1856, max= 2400, per=25.93%, avg=2154.67, stdev=224.15, samples=6 iops : min= 116, max= 150, avg=134.67, stdev=14.01, samples=6 lat (usec) : 100=5.08%, 250=0.65%, 500=0.39%, 750=3.12%, 1000=12.63% lat (msec) : 2=7.03%, 4=30.86%, 10=36.98%, 20=3.26% cpu : usr=0.10%, sys=24.98%, ctx=1456, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23131: Mon Jan 13 13:57:44 2025 read: IOPS=128, BW=2063KiB/s (2113kB/s)(6256KiB/3032msec) clat (usec): min=61, max=23090, avg=4261.81, stdev=2872.45 lat (usec): min=61, max=23091, avg=4262.42, stdev=2872.44 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 72], 10.00th=[ 90], 20.00th=[ 2540], | 30.00th=[ 2900], 40.00th=[ 3392], 50.00th=[ 3916], 60.00th=[ 4686], | 70.00th=[ 5342], 80.00th=[ 6063], 90.00th=[ 7111], 95.00th=[ 8586], | 99.00th=[14222], 99.50th=[18482], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 1600, max= 2464, per=26.12%, avg=2063.33, stdev=347.80, samples=6 iops : min= 100, max= 154, avg=128.83, stdev=21.80, samples=6 write: IOPS=124, BW=1989KiB/s (2037kB/s)(6032KiB/3032msec) clat (usec): min=626, max=21898, avg=3470.28, stdev=3270.17 lat (usec): min=627, max=21899, avg=3471.21, stdev=3270.18 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 766], 10.00th=[ 799], 20.00th=[ 848], | 30.00th=[ 1012], 40.00th=[ 1467], 50.00th=[ 2606], 60.00th=[ 3261], | 70.00th=[ 4047], 80.00th=[ 5538], 90.00th=[ 7898], 95.00th=[10028], | 99.00th=[14877], 99.50th=[19006], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 1600, max= 2560, per=23.54%, avg=1956.67, stdev=332.75, samples=6 iops : min= 100, max= 160, avg=122.17, stdev=20.90, samples=6 lat (usec) : 100=5.34%, 250=1.30%, 500=0.26%, 750=2.08%, 1000=12.63% lat (msec) : 2=7.29%, 4=31.25%, 10=35.55%, 20=4.04%, 50=0.26% cpu : usr=0.10%, sys=23.95%, ctx=1529, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23132: Mon Jan 13 13:57:44 2025 read: IOPS=126, BW=2018KiB/s (2066kB/s)(6032KiB/2989msec) clat (usec): min=59, max=28394, avg=4153.74, stdev=2868.62 lat (usec): min=59, max=28395, avg=4154.31, stdev=2868.63 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 72], 10.00th=[ 84], 20.00th=[ 2376], | 30.00th=[ 2900], 40.00th=[ 3359], 50.00th=[ 3916], 60.00th=[ 4621], | 70.00th=[ 5276], 80.00th=[ 5866], 90.00th=[ 7177], 95.00th=[ 8717], | 99.00th=[12387], 99.50th=[15139], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 1440, max= 2528, per=25.20%, avg=1990.40, stdev=411.92, samples=5 iops : min= 90, max= 158, avg=124.40, stdev=25.74, samples=5 write: IOPS=130, BW=2093KiB/s (2143kB/s)(6256KiB/2989msec) clat (usec): min=627, max=18185, avg=3427.45, stdev=3297.82 lat (usec): min=628, max=18186, avg=3428.34, stdev=3297.85 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 750], 10.00th=[ 783], 20.00th=[ 832], | 30.00th=[ 889], 40.00th=[ 1106], 50.00th=[ 2073], 60.00th=[ 3425], | 70.00th=[ 4228], 80.00th=[ 5932], 90.00th=[ 8094], 95.00th=[10028], | 99.00th=[16057], 99.50th=[16909], 99.90th=[18220], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 1664, max= 2720, per=25.26%, avg=2099.20, stdev=444.90, samples=5 iops : min= 104, max= 170, avg=131.20, stdev=27.81, samples=5 lat (usec) : 100=6.25%, 250=0.91%, 750=2.86%, 1000=16.28% lat (msec) : 2=6.77%, 4=27.21%, 10=36.07%, 20=3.52%, 50=0.13% cpu : usr=0.03%, sys=24.16%, ctx=1385, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7900KiB/s (8089kB/s), 1905KiB/s-2063KiB/s (1951kB/s-2113kB/s), io=23.4MiB (24.5MB), run=2989-3032msec WRITE: bw=8311KiB/s (8511kB/s), 1989KiB/s-2170KiB/s (2037kB/s-2223kB/s), io=24.6MiB (25.8MB), run=2989-3032msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23133: Mon Jan 13 13:57:45 2025 read: IOPS=87, BW=1403KiB/s (1437kB/s)(5920KiB/4219msec) clat (usec): min=1833, max=23940, avg=4412.17, stdev=3355.66 lat (usec): min=1833, max=23941, avg=4412.88, stdev=3355.68 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2040], 10.00th=[ 2147], 20.00th=[ 2376], | 30.00th=[ 2540], 40.00th=[ 2769], 50.00th=[ 2966], 60.00th=[ 3490], | 70.00th=[ 4228], 80.00th=[ 5604], 90.00th=[ 9241], 95.00th=[12387], | 99.00th=[17433], 99.50th=[22152], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 864, max= 2432, per=24.81%, avg=1352.00, stdev=584.26, samples=8 iops : min= 54, max= 152, avg=84.50, stdev=36.52, samples=8 write: IOPS=94, BW=1509KiB/s (1546kB/s)(6368KiB/4219msec) clat (usec): min=2381, max=32046, avg=6441.15, stdev=4016.79 lat (usec): min=2381, max=32047, avg=6442.20, stdev=4016.73 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 3032], 10.00th=[ 3294], 20.00th=[ 3621], | 30.00th=[ 3916], 40.00th=[ 4228], 50.00th=[ 4621], 60.00th=[ 5538], | 70.00th=[ 7242], 80.00th=[ 9241], 90.00th=[11994], 95.00th=[14484], | 99.00th=[20841], 99.50th=[24249], 99.90th=[32113], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 928, max= 2624, per=25.61%, avg=1468.00, stdev=669.15, samples=8 iops : min= 58, max= 164, avg=91.75, stdev=41.82, samples=8 lat (msec) : 2=1.56%, 4=47.53%, 10=37.76%, 20=12.37%, 50=0.78% cpu : usr=0.19%, sys=10.88%, ctx=883, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23134: Mon Jan 13 13:57:45 2025 read: IOPS=81, BW=1309KiB/s (1340kB/s)(5744KiB/4389msec) clat (usec): min=1825, max=24552, avg=4544.13, stdev=3719.12 lat (usec): min=1826, max=24552, avg=4544.95, stdev=3718.99 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2040], 10.00th=[ 2147], 20.00th=[ 2343], | 30.00th=[ 2474], 40.00th=[ 2704], 50.00th=[ 2999], 60.00th=[ 3458], | 70.00th=[ 4228], 80.00th=[ 5866], 90.00th=[ 9634], 95.00th=[12780], | 99.00th=[19530], 99.50th=[23200], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 702, max= 2336, per=23.32%, avg=1270.75, stdev=670.17, samples=8 iops : min= 43, max= 146, avg=79.12, stdev=41.87, samples=8 write: IOPS=93, BW=1491KiB/s (1527kB/s)(6544KiB/4389msec) clat (usec): min=2418, max=24107, avg=6534.90, stdev=3982.59 lat (usec): min=2419, max=24108, avg=6535.85, stdev=3982.60 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 3195], 10.00th=[ 3458], 20.00th=[ 3720], | 30.00th=[ 3982], 40.00th=[ 4359], 50.00th=[ 4883], 60.00th=[ 5669], | 70.00th=[ 7111], 80.00th=[ 9110], 90.00th=[12387], 95.00th=[15008], | 99.00th=[20317], 99.50th=[22938], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 894, max= 2720, per=24.96%, avg=1430.75, stdev=637.76, samples=8 iops : min= 55, max= 170, avg=89.12, stdev=39.91, samples=8 lat (msec) : 2=1.69%, 4=46.09%, 10=38.54%, 20=12.63%, 50=1.04% cpu : usr=0.05%, sys=10.71%, ctx=899, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23135: Mon Jan 13 13:57:45 2025 read: IOPS=93, BW=1503KiB/s (1539kB/s)(6256KiB/4162msec) clat (usec): min=1819, max=24567, avg=4229.44, stdev=3168.79 lat (usec): min=1820, max=24568, avg=4230.06, stdev=3168.78 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2057], 10.00th=[ 2147], 20.00th=[ 2311], | 30.00th=[ 2474], 40.00th=[ 2638], 50.00th=[ 2868], 60.00th=[ 3261], | 70.00th=[ 3916], 80.00th=[ 5735], 90.00th=[ 7963], 95.00th=[11469], | 99.00th=[15139], 99.50th=[21627], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 768, max= 2752, per=27.16%, avg=1480.00, stdev=781.92, samples=8 iops : min= 48, max= 172, avg=92.50, stdev=48.87, samples=8 write: IOPS=90, BW=1449KiB/s (1484kB/s)(6032KiB/4162msec) clat (usec): min=2655, max=24218, avg=6457.60, stdev=3844.09 lat (usec): min=2656, max=24220, avg=6460.05, stdev=3843.27 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3228], 10.00th=[ 3392], 20.00th=[ 3687], | 30.00th=[ 3916], 40.00th=[ 4146], 50.00th=[ 4621], 60.00th=[ 5669], | 70.00th=[ 7111], 80.00th=[ 9110], 90.00th=[12518], 95.00th=[14615], | 99.00th=[18744], 99.50th=[21627], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 864, max= 2432, per=24.21%, avg=1388.00, stdev=551.59, samples=8 iops : min= 54, max= 152, avg=86.75, stdev=34.47, samples=8 lat (msec) : 2=1.30%, 4=50.91%, 10=35.42%, 20=11.72%, 50=0.65% cpu : usr=0.17%, sys=10.98%, ctx=883, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23136: Mon Jan 13 13:57:45 2025 read: IOPS=85, BW=1372KiB/s (1405kB/s)(6032KiB/4395msec) clat (usec): min=1820, max=24631, avg=4716.45, stdev=3789.36 lat (usec): min=1821, max=24631, avg=4717.07, stdev=3789.36 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2073], 10.00th=[ 2180], 20.00th=[ 2376], | 30.00th=[ 2540], 40.00th=[ 2737], 50.00th=[ 3097], 60.00th=[ 3556], | 70.00th=[ 4490], 80.00th=[ 6521], 90.00th=[10028], 95.00th=[13173], | 99.00th=[20055], 99.50th=[23987], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 704, max= 2368, per=24.00%, avg=1308.00, stdev=586.93, samples=8 iops : min= 44, max= 148, avg=81.75, stdev=36.68, samples=8 write: IOPS=88, BW=1423KiB/s (1458kB/s)(6256KiB/4395msec) clat (usec): min=2539, max=32720, avg=6437.47, stdev=4151.52 lat (usec): min=2540, max=32721, avg=6438.45, stdev=4151.54 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 3032], 10.00th=[ 3359], 20.00th=[ 3621], | 30.00th=[ 3916], 40.00th=[ 4228], 50.00th=[ 4555], 60.00th=[ 5211], | 70.00th=[ 6718], 80.00th=[ 9110], 90.00th=[12518], 95.00th=[15008], | 99.00th=[20579], 99.50th=[23987], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 768, max= 2624, per=23.86%, avg=1368.00, stdev=734.86, samples=8 iops : min= 48, max= 164, avg=85.50, stdev=45.93, samples=8 lat (msec) : 2=1.43%, 4=47.14%, 10=37.24%, 20=13.15%, 50=1.04% cpu : usr=0.16%, sys=10.79%, ctx=886, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5450KiB/s (5581kB/s), 1309KiB/s-1503KiB/s (1340kB/s-1539kB/s), io=23.4MiB (24.5MB), run=4162-4395msec WRITE: bw=5734KiB/s (5871kB/s), 1423KiB/s-1509KiB/s (1458kB/s-1546kB/s), io=24.6MiB (25.8MB), run=4162-4395msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23146: Mon Jan 13 13:57:48 2025 read: IOPS=4, BW=4943KiB/s (5062kB/s)(7168KiB/1450msec) clat (usec): min=364, max=99075, avg=34353.77, stdev=33287.04 lat (usec): min=365, max=99076, avg=34354.45, stdev=33287.15 clat percentiles (usec): | 1.00th=[ 367], 5.00th=[ 367], 10.00th=[ 367], 20.00th=[ 424], | 30.00th=[30278], 40.00th=[30278], 50.00th=[30802], 60.00th=[34341], | 70.00th=[34341], 80.00th=[45351], 90.00th=[99091], 95.00th=[99091], | 99.00th=[99091], 99.50th=[99091], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min=10240, max=10240, per=76.15%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3531KiB/s (3616kB/s)(5120KiB/1450msec) clat (msec): min=26, max=886, avg=220.21, stdev=374.10 lat (msec): min=26, max=886, avg=220.24, stdev=374.12 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 40], 60.00th=[ 40], | 70.00th=[ 110], 80.00th=[ 110], 90.00th=[ 885], 95.00th=[ 885], | 99.00th=[ 885], 99.50th=[ 885], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 4096, max= 4096, per=21.76%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 50=58.33%, 100=8.33%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=13.52%, ctx=712, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23148: Mon Jan 13 13:57:48 2025 read: IOPS=1, BW=1345KiB/s (1377kB/s)(2048KiB/1523msec) clat (usec): min=32387, max=38739, avg=35563.76, stdev=4491.72 lat (usec): min=32388, max=38741, avg=35564.68, stdev=4492.03 clat percentiles (usec): | 1.00th=[32375], 5.00th=[32375], 10.00th=[32375], 20.00th=[32375], | 30.00th=[32375], 40.00th=[32375], 50.00th=[32375], 60.00th=[38536], | 70.00th=[38536], 80.00th=[38536], 90.00th=[38536], 95.00th=[38536], | 99.00th=[38536], 99.50th=[38536], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 2000, max= 2048, per=15.05%, avg=2024.00, stdev=33.94, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=6, BW=6724KiB/s (6885kB/s)(10.0MiB/1523msec) clat (msec): min=30, max=869, avg=134.36, stdev=258.76 lat (msec): min=30, max=869, avg=134.42, stdev=258.78 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 44], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 61], | 70.00th=[ 72], 80.00th=[ 72], 90.00th=[ 75], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 6144, max=12000, per=48.19%, avg=9072.00, stdev=4140.82, samples=2 iops : min= 6, max= 11, avg= 8.50, stdev= 3.54, samples=2 lat (msec) : 50=58.33%, 100=33.33%, 1000=8.33% cpu : usr=0.00%, sys=15.63%, ctx=332, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23150: Mon Jan 13 13:57:48 2025 read: IOPS=4, BW=5040KiB/s (5161kB/s)(6144KiB/1219msec) clat (usec): min=421, max=772582, avg=151082.45, stdev=304806.54 lat (usec): min=422, max=772582, avg=151083.20, stdev=304806.40 clat percentiles (usec): | 1.00th=[ 420], 5.00th=[ 420], 10.00th=[ 420], 20.00th=[ 23200], | 30.00th=[ 23200], 40.00th=[ 34341], 50.00th=[ 34341], 60.00th=[ 36963], | 70.00th=[ 39584], 80.00th=[ 39584], 90.00th=[775947], 95.00th=[775947], | 99.00th=[775947], 99.50th=[775947], 99.90th=[775947], 99.95th=[775947], | 99.99th=[775947] bw ( KiB/s): min= 6131, max= 6131, per=45.59%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 write: IOPS=4, BW=5040KiB/s (5161kB/s)(6144KiB/1219msec) clat (usec): min=33670, max=73062, avg=49336.15, stdev=15967.56 lat (usec): min=33736, max=73092, avg=49394.57, stdev=15948.31 clat percentiles (usec): | 1.00th=[33817], 5.00th=[33817], 10.00th=[33817], 20.00th=[33817], | 30.00th=[33817], 40.00th=[44827], 50.00th=[44827], 60.00th=[46924], | 70.00th=[63701], 80.00th=[63701], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 8175, max= 8175, per=43.42%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=66.67%, 100=16.67%, 1000=8.33% cpu : usr=0.00%, sys=23.40%, ctx=1138, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23152: Mon Jan 13 13:57:48 2025 read: IOPS=3, BW=3705KiB/s (3794kB/s)(5120KiB/1382msec) clat (msec): min=24, max=758, avg=185.39, stdev=320.80 lat (msec): min=24, max=758, avg=185.39, stdev=320.80 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 39], 60.00th=[ 39], | 70.00th=[ 77], 80.00th=[ 77], 90.00th=[ 760], 95.00th=[ 760], | 99.00th=[ 760], 99.50th=[ 760], 99.90th=[ 760], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 2048, max= 2048, per=15.23%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5187KiB/s (5311kB/s)(7168KiB/1382msec) clat (usec): min=25744, max=97616, avg=51144.06, stdev=27498.03 lat (usec): min=25776, max=97672, avg=51197.16, stdev=27505.63 clat percentiles (usec): | 1.00th=[25822], 5.00th=[25822], 10.00th=[25822], 20.00th=[26084], | 30.00th=[35914], 40.00th=[35914], 50.00th=[36439], 60.00th=[61604], | 70.00th=[61604], 80.00th=[74974], 90.00th=[98042], 95.00th=[98042], | 99.00th=[98042], 99.50th=[98042], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 8192, max= 8192, per=43.51%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=33.33%, 1000=8.33% cpu : usr=0.00%, sys=16.00%, ctx=901, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.1MiB/s (13.8MB/s), 1345KiB/s-5040KiB/s (1377kB/s-5161kB/s), io=20.0MiB (20.0MB), run=1219-1523msec WRITE: bw=18.4MiB/s (19.3MB/s), 3531KiB/s-6724KiB/s (3616kB/s-6885kB/s), io=28.0MiB (29.4MB), run=1219-1523msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23145: Mon Jan 13 13:57:48 2025 read: IOPS=5, BW=5434KiB/s (5565kB/s)(7168KiB/1319msec) clat (usec): min=14994, max=88970, avg=35860.51, stdev=26678.57 lat (usec): min=14995, max=88971, avg=35861.37, stdev=26678.51 clat percentiles (usec): | 1.00th=[15008], 5.00th=[15008], 10.00th=[15008], 20.00th=[17957], | 30.00th=[20579], 40.00th=[20579], 50.00th=[21890], 60.00th=[34866], | 70.00th=[34866], 80.00th=[52167], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min=10240, max=10240, per=77.61%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3882KiB/s (3975kB/s)(5120KiB/1319msec) clat (msec): min=17, max=837, avg=211.88, stdev=351.01 lat (msec): min=17, max=837, avg=211.93, stdev=351.01 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 54], 60.00th=[ 54], | 70.00th=[ 106], 80.00th=[ 106], 90.00th=[ 835], 95.00th=[ 835], | 99.00th=[ 835], 99.50th=[ 835], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 4096, max= 4096, per=22.17%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=33.33%, 100=25.00%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=7.74%, ctx=74, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23147: Mon Jan 13 13:57:48 2025 read: IOPS=1, BW=1460KiB/s (1495kB/s)(2048KiB/1403msec) clat (usec): min=32577, max=49028, avg=40802.89, stdev=11632.08 lat (usec): min=32578, max=49029, avg=40804.11, stdev=11632.32 clat percentiles (usec): | 1.00th=[32637], 5.00th=[32637], 10.00th=[32637], 20.00th=[32637], | 30.00th=[32637], 40.00th=[32637], 50.00th=[32637], 60.00th=[49021], | 70.00th=[49021], 80.00th=[49021], 90.00th=[49021], 95.00th=[49021], | 99.00th=[49021], 99.50th=[49021], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 2048, max= 2048, per=15.52%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7299KiB/s (7474kB/s)(10.0MiB/1403msec) clat (msec): min=18, max=794, avg=127.28, stdev=236.51 lat (msec): min=18, max=795, avg=127.43, stdev=236.80 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 22], | 30.00th=[ 26], 40.00th=[ 35], 50.00th=[ 36], 60.00th=[ 75], | 70.00th=[ 93], 80.00th=[ 93], 90.00th=[ 99], 95.00th=[ 793], | 99.00th=[ 793], 99.50th=[ 793], 99.90th=[ 793], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 6144, max= 6144, per=33.26%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=33.33%, 1000=8.33% cpu : usr=0.00%, sys=7.70%, ctx=45, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23149: Mon Jan 13 13:57:48 2025 read: IOPS=3, BW=4018KiB/s (4115kB/s)(6144KiB/1529msec) clat (msec): min=13, max=803, avg=191.05, stdev=301.59 lat (msec): min=13, max=803, avg=191.05, stdev=301.59 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 68], | 30.00th=[ 68], 40.00th=[ 71], 50.00th=[ 71], 60.00th=[ 95], | 70.00th=[ 96], 80.00th=[ 96], 90.00th=[ 802], 95.00th=[ 802], | 99.00th=[ 802], 99.50th=[ 802], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 4096, max= 6144, per=38.80%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=3, BW=4018KiB/s (4115kB/s)(6144KiB/1529msec) clat (usec): min=20256, max=90051, avg=55408.87, stdev=28169.27 lat (usec): min=20287, max=90118, avg=55459.08, stdev=28190.22 clat percentiles (usec): | 1.00th=[20317], 5.00th=[20317], 10.00th=[20317], 20.00th=[28705], | 30.00th=[28705], 40.00th=[46400], 50.00th=[46400], 60.00th=[67634], | 70.00th=[80217], 80.00th=[80217], 90.00th=[89654], 95.00th=[89654], | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 4096, max= 8192, per=33.26%, avg=6144.00, stdev=2896.31, samples=2 iops : min= 4, max= 8, avg= 6.00, stdev= 2.83, samples=2 lat (msec) : 20=8.33%, 50=25.00%, 100=58.33%, 1000=8.33% cpu : usr=0.00%, sys=7.85%, ctx=203, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23151: Mon Jan 13 13:57:48 2025 read: IOPS=3, BW=3299KiB/s (3378kB/s)(5120KiB/1552msec) clat (msec): min=53, max=844, avg=226.02, stdev=346.18 lat (msec): min=53, max=844, avg=226.02, stdev=346.18 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 54], 10.00th=[ 54], 20.00th=[ 54], | 30.00th=[ 62], 40.00th=[ 62], 50.00th=[ 84], 60.00th=[ 84], | 70.00th=[ 87], 80.00th=[ 87], 90.00th=[ 844], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 2048, max= 6144, per=31.04%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=4, BW=4619KiB/s (4729kB/s)(7168KiB/1552msec) clat (msec): min=25, max=117, avg=59.03, stdev=31.67 lat (msec): min=25, max=117, avg=59.09, stdev=31.65 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 36], | 30.00th=[ 38], 40.00th=[ 38], 50.00th=[ 55], 60.00th=[ 64], | 70.00th=[ 64], 80.00th=[ 81], 90.00th=[ 117], 95.00th=[ 117], | 99.00th=[ 117], 99.50th=[ 117], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 6144, max= 8192, per=38.80%, avg=7168.00, stdev=1448.15, samples=2 iops : min= 6, max= 8, avg= 7.00, stdev= 1.41, samples=2 lat (msec) : 50=25.00%, 100=58.33%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=5.61%, ctx=98, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.9MiB/s (13.5MB/s), 1460KiB/s-5434KiB/s (1495kB/s-5565kB/s), io=20.0MiB (20.0MB), run=1319-1552msec WRITE: bw=18.0MiB/s (18.9MB/s), 3882KiB/s-7299KiB/s (3975kB/s-7474kB/s), io=28.0MiB (29.4MB), run=1319-1552msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23160: Mon Jan 13 13:57:50 2025 read: IOPS=6, BW=27.8MiB/s (29.1MB/s)(12.0MiB/432msec) clat (msec): min=27, max=262, avg=139.92, stdev=117.72 lat (msec): min=27, max=262, avg=139.93, stdev=117.73 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 131], 50.00th=[ 131], 60.00th=[ 131], | 70.00th=[ 264], 80.00th=[ 264], 90.00th=[ 264], 95.00th=[ 264], | 99.00th=[ 264], 99.50th=[ 264], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] lat (msec) : 50=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=32.71%, ctx=1712, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23162: Mon Jan 13 13:57:50 2025 read: IOPS=1, BW=5634KiB/s (5769kB/s)(4096KiB/727msec) clat (nsec): min=215098k, max=215098k, avg=215097999.00, stdev= 0.00 lat (nsec): min=215099k, max=215099k, avg=215099036.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 215], 10.00th=[ 215], 20.00th=[ 215], | 30.00th=[ 215], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 215], | 70.00th=[ 215], 80.00th=[ 215], 90.00th=[ 215], 95.00th=[ 215], | 99.00th=[ 215], 99.50th=[ 215], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 8192, max= 8192, per=36.35%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=11.0MiB/s (11.5MB/s)(8192KiB/727msec) clat (msec): min=137, max=336, avg=236.69, stdev=140.45 lat (msec): min=137, max=336, avg=237.05, stdev=140.58 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 138], 10.00th=[ 138], 20.00th=[ 138], | 30.00th=[ 138], 40.00th=[ 138], 50.00th=[ 138], 60.00th=[ 338], | 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.14%, sys=25.34%, ctx=631, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23164: Mon Jan 13 13:57:50 2025 write: IOPS=3, BW=13.8MiB/s (14.5MB/s)(12.0MiB/869msec) clat (msec): min=188, max=433, avg=277.58, stdev=135.40 lat (msec): min=189, max=433, avg=277.88, stdev=135.42 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 190], 10.00th=[ 190], 20.00th=[ 190], | 30.00th=[ 190], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 211], | 70.00th=[ 435], 80.00th=[ 435], 90.00th=[ 435], 95.00th=[ 435], | 99.00th=[ 435], 99.50th=[ 435], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 8192, max= 8192, per=21.73%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=29.95%, ctx=30, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23166: Mon Jan 13 13:57:50 2025 write: IOPS=4, BW=17.5MiB/s (18.3MB/s)(12.0MiB/687msec) clat (msec): min=121, max=389, avg=223.15, stdev=145.26 lat (msec): min=121, max=389, avg=223.37, stdev=145.14 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 122], 10.00th=[ 122], 20.00th=[ 122], | 30.00th=[ 122], 40.00th=[ 159], 50.00th=[ 159], 60.00th=[ 159], | 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 8192, max= 8192, per=21.73%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=41.48%, ctx=39, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.0MiB/s (23.1MB/s), 5634KiB/s-27.8MiB/s (5769kB/s-29.1MB/s), io=16.0MiB (16.8MB), run=432-727msec WRITE: bw=36.8MiB/s (38.6MB/s), 11.0MiB/s-17.5MiB/s (11.5MB/s-18.3MB/s), io=32.0MiB (33.6MB), run=687-869msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23159: Mon Jan 13 13:57:50 2025 read: IOPS=4, BW=16.2MiB/s (16.9MB/s)(12.0MiB/743msec) clat (msec): min=128, max=315, avg=244.37, stdev=101.02 lat (msec): min=128, max=315, avg=244.37, stdev=101.02 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 129], 10.00th=[ 129], 20.00th=[ 129], | 30.00th=[ 129], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 288], | 70.00th=[ 317], 80.00th=[ 317], 90.00th=[ 317], 95.00th=[ 317], | 99.00th=[ 317], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min=16384, max=16384, per=90.10%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.13%, sys=9.57%, ctx=45, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23161: Mon Jan 13 13:57:50 2025 read: IOPS=1, BW=4546KiB/s (4655kB/s)(4096KiB/901msec) clat (nsec): min=303456k, max=303456k, avg=303455868.00, stdev= 0.00 lat (nsec): min=303458k, max=303458k, avg=303458061.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 305], 10.00th=[ 305], 20.00th=[ 305], | 30.00th=[ 305], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 305], | 70.00th=[ 305], 80.00th=[ 305], 90.00th=[ 305], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 8192, max= 8192, per=45.05%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9092KiB/s (9310kB/s)(8192KiB/901msec) clat (msec): min=224, max=355, avg=289.88, stdev=92.58 lat (msec): min=224, max=355, avg=290.08, stdev=92.58 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 226], 10.00th=[ 226], 20.00th=[ 226], | 30.00th=[ 226], 40.00th=[ 226], 50.00th=[ 226], 60.00th=[ 355], | 70.00th=[ 355], 80.00th=[ 355], 90.00th=[ 355], 95.00th=[ 355], | 99.00th=[ 355], 99.50th=[ 355], 99.90th=[ 355], 99.95th=[ 355], | 99.99th=[ 355] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.11%, sys=15.00%, ctx=50, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23163: Mon Jan 13 13:57:50 2025 write: IOPS=3, BW=14.7MiB/s (15.4MB/s)(12.0MiB/818msec) clat (msec): min=139, max=486, avg=267.83, stdev=190.09 lat (msec): min=140, max=486, avg=268.15, stdev=190.11 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 140], 10.00th=[ 140], 20.00th=[ 140], | 30.00th=[ 140], 40.00th=[ 178], 50.00th=[ 178], 60.00th=[ 178], | 70.00th=[ 485], 80.00th=[ 485], 90.00th=[ 485], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min=16384, max=16384, per=45.05%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.37%, sys=10.28%, ctx=164, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23165: Mon Jan 13 13:57:50 2025 write: IOPS=4, BW=19.5MiB/s (20.5MB/s)(12.0MiB/614msec) clat (msec): min=151, max=255, avg=194.34, stdev=54.45 lat (msec): min=152, max=255, avg=194.58, stdev=54.40 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 153], 10.00th=[ 153], 20.00th=[ 153], | 30.00th=[ 153], 40.00th=[ 176], 50.00th=[ 176], 60.00th=[ 176], | 70.00th=[ 255], 80.00th=[ 255], 90.00th=[ 255], 95.00th=[ 255], | 99.00th=[ 255], 99.50th=[ 255], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min=16384, max=16384, per=45.05%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=20.88%, ctx=173, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.8MiB/s (18.6MB/s), 4546KiB/s-16.2MiB/s (4655kB/s-16.9MB/s), io=16.0MiB (16.8MB), run=743-901msec WRITE: bw=35.5MiB/s (37.2MB/s), 9092KiB/s-19.5MiB/s (9310kB/s-20.5MB/s), io=32.0MiB (33.6MB), run=614-901msec