== sanity test 398b: DIO and buffer IO race ============== 03:51:31 (1746085891) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.31927 s, 38.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21616: Thu May 1 03:51:47 2025 read: IOPS=113, BW=454KiB/s (465kB/s)(5992KiB/13187msec) clat (usec): min=54, max=178487, avg=4837.02, stdev=5214.46 lat (usec): min=54, max=178487, avg=4838.49, stdev=5214.56 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 115], 10.00th=[ 2245], 20.00th=[ 2769], | 30.00th=[ 3195], 40.00th=[ 3687], 50.00th=[ 4293], 60.00th=[ 5014], | 70.00th=[ 5735], 80.00th=[ 6587], 90.00th=[ 8029], 95.00th=[ 9503], | 99.00th=[ 12780], 99.50th=[ 15270], 99.90th=[ 22414], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 280, max= 600, per=24.78%, avg=450.92, stdev=65.47, samples=26 iops : min= 70, max= 150, avg=112.62, stdev=16.40, samples=26 write: IOPS=119, BW=477KiB/s (489kB/s)(6296KiB/13187msec) clat (usec): min=410, max=24959, avg=3729.74, stdev=3439.81 lat (usec): min=411, max=24959, avg=3731.66, stdev=3439.90 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 594], 10.00th=[ 660], 20.00th=[ 766], | 30.00th=[ 906], 40.00th=[ 1745], 50.00th=[ 3064], 60.00th=[ 3752], | 70.00th=[ 4621], 80.00th=[ 6259], 90.00th=[ 8455], 95.00th=[10290], | 99.00th=[14746], 99.50th=[19006], 99.90th=[24511], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 352, max= 608, per=24.83%, avg=473.35, stdev=64.20, samples=26 iops : min= 88, max= 152, avg=118.23, stdev=16.07, samples=26 lat (usec) : 100=1.76%, 250=1.27%, 500=0.49%, 750=9.02%, 1000=7.55% lat (msec) : 2=4.26%, 4=30.66%, 10=40.46%, 20=4.26%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=23.05%, ctx=5725, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21617: Thu May 1 03:51:47 2025 read: IOPS=113, BW=455KiB/s (466kB/s)(5940KiB/13064msec) clat (usec): min=54, max=22236, avg=4745.25, stdev=2681.40 lat (usec): min=54, max=22236, avg=4746.17, stdev=2681.39 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 131], 10.00th=[ 2212], 20.00th=[ 2769], | 30.00th=[ 3163], 40.00th=[ 3654], 50.00th=[ 4228], 60.00th=[ 5080], | 70.00th=[ 5866], 80.00th=[ 6652], 90.00th=[ 8029], 95.00th=[ 9372], | 99.00th=[13829], 99.50th=[15401], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 288, max= 576, per=24.95%, avg=454.08, stdev=58.48, samples=26 iops : min= 72, max= 144, avg=113.46, stdev=14.61, samples=26 write: IOPS=121, BW=486KiB/s (498kB/s)(6348KiB/13064msec) clat (usec): min=435, max=178813, avg=3746.75, stdev=5497.05 lat (usec): min=436, max=178814, avg=3747.58, stdev=5497.06 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 570], 10.00th=[ 644], 20.00th=[ 750], | 30.00th=[ 865], 40.00th=[ 1401], 50.00th=[ 2966], 60.00th=[ 3785], | 70.00th=[ 4883], 80.00th=[ 6259], 90.00th=[ 8160], 95.00th=[ 9765], | 99.00th=[ 14222], 99.50th=[ 15139], 99.90th=[ 28181], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 352, max= 704, per=25.42%, avg=484.50, stdev=78.54, samples=26 iops : min= 88, max= 176, avg=121.08, stdev=19.64, samples=26 lat (usec) : 100=1.30%, 250=1.40%, 500=0.88%, 750=10.03%, 1000=7.49% lat (msec) : 2=4.72%, 4=28.16%, 10=41.93%, 20=3.94%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=23.07%, ctx=5788, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21618: Thu May 1 03:51:47 2025 read: IOPS=114, BW=459KiB/s (470kB/s)(6028KiB/13130msec) clat (usec): min=54, max=23663, avg=4615.40, stdev=2710.13 lat (usec): min=54, max=23664, avg=4616.42, stdev=2710.15 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 113], 10.00th=[ 2114], 20.00th=[ 2704], | 30.00th=[ 3097], 40.00th=[ 3556], 50.00th=[ 4178], 60.00th=[ 4817], | 70.00th=[ 5604], 80.00th=[ 6456], 90.00th=[ 7832], 95.00th=[ 9241], | 99.00th=[13042], 99.50th=[16450], 99.90th=[20579], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 216, max= 590, per=25.17%, avg=458.08, stdev=75.64, samples=26 iops : min= 54, max= 147, avg=114.50, stdev=18.87, samples=26 write: IOPS=119, BW=477KiB/s (488kB/s)(6260KiB/13130msec) clat (usec): min=432, max=184102, avg=3895.95, stdev=5645.60 lat (usec): min=433, max=184103, avg=3896.80, stdev=5645.56 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 594], 10.00th=[ 660], 20.00th=[ 775], | 30.00th=[ 922], 40.00th=[ 1942], 50.00th=[ 3097], 60.00th=[ 3916], | 70.00th=[ 4817], 80.00th=[ 6456], 90.00th=[ 8356], 95.00th=[ 10421], | 99.00th=[ 14222], 99.50th=[ 15795], 99.90th=[ 21890], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 312, max= 632, per=24.89%, avg=474.38, stdev=74.51, samples=26 iops : min= 78, max= 158, avg=118.58, stdev=18.59, samples=26 lat (usec) : 100=1.79%, 250=1.60%, 500=0.81%, 750=8.43%, 1000=7.52% lat (msec) : 2=4.52%, 4=29.85%, 10=40.76%, 20=4.59%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=22.89%, ctx=5729, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21619: Thu May 1 03:51:47 2025 read: IOPS=116, BW=465KiB/s (476kB/s)(6052KiB/13009msec) clat (usec): min=55, max=24586, avg=4793.85, stdev=2761.99 lat (usec): min=55, max=24587, avg=4795.53, stdev=2762.98 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 113], 10.00th=[ 2245], 20.00th=[ 2737], | 30.00th=[ 3228], 40.00th=[ 3720], 50.00th=[ 4359], 60.00th=[ 5080], | 70.00th=[ 5800], 80.00th=[ 6718], 90.00th=[ 7963], 95.00th=[ 9503], | 99.00th=[13698], 99.50th=[16319], 99.90th=[21103], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 232, max= 632, per=25.54%, avg=464.77, stdev=78.57, samples=26 iops : min= 58, max= 158, avg=116.15, stdev=19.61, samples=26 write: IOPS=119, BW=479KiB/s (491kB/s)(6236KiB/13009msec) clat (usec): min=435, max=174363, avg=3638.75, stdev=5405.53 lat (usec): min=436, max=174365, avg=3640.00, stdev=5406.31 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 594], 10.00th=[ 660], 20.00th=[ 758], | 30.00th=[ 873], 40.00th=[ 1401], 50.00th=[ 2802], 60.00th=[ 3589], | 70.00th=[ 4490], 80.00th=[ 5932], 90.00th=[ 7963], 95.00th=[ 10028], | 99.00th=[ 14091], 99.50th=[ 15664], 99.90th=[ 23987], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 272, max= 776, per=25.11%, avg=478.62, stdev=94.87, samples=26 iops : min= 68, max= 194, avg=119.62, stdev=23.69, samples=26 lat (usec) : 100=1.79%, 250=1.27%, 500=0.81%, 750=9.05%, 1000=7.88% lat (msec) : 2=4.92%, 4=29.26%, 10=40.33%, 20=4.49%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=23.16%, ctx=5692, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1821KiB/s (1865kB/s), 454KiB/s-465KiB/s (465kB/s-476kB/s), io=23.4MiB (24.6MB), run=13009-13187msec WRITE: bw=1906KiB/s (1952kB/s), 477KiB/s-486KiB/s (488kB/s-498kB/s), io=24.6MiB (25.7MB), run=13009-13187msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21612: Thu May 1 03:51:51 2025 read: IOPS=86, BW=346KiB/s (354kB/s)(5992KiB/17332msec) clat (usec): min=1557, max=179829, avg=4526.36, stdev=5663.75 lat (usec): min=1558, max=179830, avg=4527.04, stdev=5663.76 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1926], 10.00th=[ 2024], 20.00th=[ 2212], | 30.00th=[ 2409], 40.00th=[ 2671], 50.00th=[ 2999], 60.00th=[ 3490], | 70.00th=[ 4228], 80.00th=[ 5932], 90.00th=[ 9503], 95.00th=[ 11994], | 99.00th=[ 16909], 99.50th=[ 20317], 99.90th=[ 24249], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 152, max= 720, per=24.55%, avg=339.53, stdev=159.63, samples=34 iops : min= 38, max= 180, avg=84.88, stdev=39.91, samples=34 write: IOPS=90, BW=363KiB/s (372kB/s)(6296KiB/17332msec) clat (usec): min=2432, max=42545, avg=6657.23, stdev=4163.57 lat (usec): min=2433, max=42545, avg=6658.08, stdev=4163.60 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 3032], 10.00th=[ 3261], 20.00th=[ 3589], | 30.00th=[ 3949], 40.00th=[ 4359], 50.00th=[ 5014], 60.00th=[ 6128], | 70.00th=[ 7635], 80.00th=[ 9634], 90.00th=[11994], 95.00th=[14222], | 99.00th=[22938], 99.50th=[27657], 99.90th=[34341], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 176, max= 776, per=24.67%, avg=357.18, stdev=165.76, samples=34 iops : min= 44, max= 194, avg=89.29, stdev=41.44, samples=34 lat (msec) : 2=3.78%, 4=45.12%, 10=37.70%, 20=12.50%, 50=0.88% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=11.34%, ctx=3780, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21613: Thu May 1 03:51:51 2025 read: IOPS=85, BW=342KiB/s (350kB/s)(5940KiB/17361msec) clat (usec): min=1570, max=183388, avg=4615.61, stdev=5852.91 lat (usec): min=1570, max=183390, avg=4616.38, stdev=5852.93 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1926], 10.00th=[ 2040], 20.00th=[ 2245], | 30.00th=[ 2409], 40.00th=[ 2638], 50.00th=[ 2966], 60.00th=[ 3490], | 70.00th=[ 4293], 80.00th=[ 6063], 90.00th=[ 9765], 95.00th=[ 12518], | 99.00th=[ 17695], 99.50th=[ 21365], 99.90th=[ 25297], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 112, max= 760, per=24.43%, avg=337.88, stdev=174.57, samples=34 iops : min= 28, max= 190, avg=84.47, stdev=43.64, samples=34 write: IOPS=91, BW=366KiB/s (374kB/s)(6348KiB/17361msec) clat (usec): min=2363, max=34534, avg=6574.76, stdev=4055.64 lat (usec): min=2364, max=34535, avg=6575.72, stdev=4055.65 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2966], 10.00th=[ 3195], 20.00th=[ 3523], | 30.00th=[ 3916], 40.00th=[ 4359], 50.00th=[ 4948], 60.00th=[ 5932], | 70.00th=[ 7504], 80.00th=[ 9372], 90.00th=[12387], 95.00th=[14615], | 99.00th=[21890], 99.50th=[23987], 99.90th=[28181], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 176, max= 784, per=24.63%, avg=356.71, stdev=161.75, samples=34 iops : min= 44, max= 196, avg=89.18, stdev=40.44, samples=34 lat (msec) : 2=4.07%, 4=45.28%, 10=37.37%, 20=12.24%, 50=1.01% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.24%, ctx=3743, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21614: Thu May 1 03:51:51 2025 read: IOPS=86, BW=348KiB/s (356kB/s)(6028KiB/17337msec) clat (usec): min=1615, max=31582, avg=4614.96, stdev=3652.80 lat (usec): min=1615, max=31583, avg=4615.71, stdev=3652.78 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1926], 10.00th=[ 2024], 20.00th=[ 2212], | 30.00th=[ 2409], 40.00th=[ 2737], 50.00th=[ 3097], 60.00th=[ 3654], | 70.00th=[ 4555], 80.00th=[ 6456], 90.00th=[ 9896], 95.00th=[12911], | 99.00th=[17171], 99.50th=[19530], 99.90th=[28967], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 112, max= 768, per=24.77%, avg=342.59, stdev=163.29, samples=34 iops : min= 28, max= 192, avg=85.65, stdev=40.82, samples=34 write: IOPS=90, BW=361KiB/s (370kB/s)(6260KiB/17337msec) clat (msec): min=2, max=185, avg= 6.59, stdev= 5.97 lat (msec): min=2, max=185, avg= 6.59, stdev= 5.97 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 22], 99.50th=[ 24], 99.90th=[ 29], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 152, max= 736, per=24.46%, avg=354.12, stdev=169.16, samples=34 iops : min= 38, max= 184, avg=88.53, stdev=42.29, samples=34 lat (msec) : 2=4.30%, 4=44.17%, 10=38.25%, 20=12.40%, 50=0.85% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=11.23%, ctx=3658, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21615: Thu May 1 03:51:51 2025 read: IOPS=87, BW=349KiB/s (357kB/s)(6052KiB/17343msec) clat (usec): min=1537, max=28483, avg=4438.17, stdev=3403.78 lat (usec): min=1537, max=28484, avg=4438.95, stdev=3403.85 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1926], 10.00th=[ 2024], 20.00th=[ 2245], | 30.00th=[ 2409], 40.00th=[ 2704], 50.00th=[ 3064], 60.00th=[ 3556], | 70.00th=[ 4424], 80.00th=[ 5997], 90.00th=[ 9372], 95.00th=[12518], | 99.00th=[16319], 99.50th=[17957], 99.90th=[25560], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 136, max= 768, per=24.70%, avg=341.62, stdev=160.35, samples=34 iops : min= 34, max= 192, avg=85.38, stdev=40.10, samples=34 write: IOPS=89, BW=360KiB/s (368kB/s)(6236KiB/17343msec) clat (msec): min=2, max=187, avg= 6.77, stdev= 6.14 lat (msec): min=2, max=187, avg= 6.77, stdev= 6.14 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 22], 99.50th=[ 24], 99.90th=[ 43], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 160, max= 728, per=24.39%, avg=353.15, stdev=164.91, samples=34 iops : min= 40, max= 182, avg=88.26, stdev=41.23, samples=34 lat (msec) : 2=4.36%, 4=44.53%, 10=37.70%, 20=12.63%, 50=0.75% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=11.29%, ctx=3672, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1383KiB/s (1416kB/s), 342KiB/s-349KiB/s (350kB/s-357kB/s), io=23.4MiB (24.6MB), run=17332-17361msec WRITE: bw=1448KiB/s (1483kB/s), 360KiB/s-366KiB/s (368kB/s-374kB/s), io=24.6MiB (25.7MB), run=17332-17361msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21652: Thu May 1 03:51:56 2025 read: IOPS=99, BW=1593KiB/s (1631kB/s)(5920KiB/3716msec) clat (usec): min=58, max=21400, avg=5180.83, stdev=3069.08 lat (usec): min=59, max=21401, avg=5181.59, stdev=3069.17 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 101], 10.00th=[ 2212], 20.00th=[ 2999], | 30.00th=[ 3523], 40.00th=[ 4080], 50.00th=[ 4686], 60.00th=[ 5407], | 70.00th=[ 6718], 80.00th=[ 7504], 90.00th=[ 8848], 95.00th=[10290], | 99.00th=[13698], 99.50th=[15270], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 1312, max= 1756, per=24.36%, avg=1544.14, stdev=186.85, samples=7 iops : min= 82, max= 109, avg=96.29, stdev=11.64, samples=7 write: IOPS=107, BW=1714KiB/s (1755kB/s)(6368KiB/3716msec) clat (usec): min=673, max=19182, avg=4386.54, stdev=3536.74 lat (usec): min=674, max=19183, avg=4387.60, stdev=3536.71 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 881], 10.00th=[ 1020], 20.00th=[ 1172], | 30.00th=[ 1467], 40.00th=[ 2573], 50.00th=[ 3425], 60.00th=[ 4359], | 70.00th=[ 5407], 80.00th=[ 7373], 90.00th=[ 9372], 95.00th=[11600], | 99.00th=[15926], 99.50th=[18482], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 1376, max= 2016, per=25.14%, avg=1676.57, stdev=247.68, samples=7 iops : min= 86, max= 126, avg=104.57, stdev=15.47, samples=7 lat (usec) : 100=2.21%, 250=2.08%, 500=0.26%, 750=0.91%, 1000=4.17% lat (msec) : 2=13.41%, 4=24.61%, 10=45.57%, 20=6.64%, 50=0.13% cpu : usr=0.27%, sys=24.52%, ctx=1660, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21653: Thu May 1 03:51:56 2025 read: IOPS=96, BW=1551KiB/s (1588kB/s)(5744KiB/3704msec) clat (usec): min=64, max=18390, avg=5292.90, stdev=3047.97 lat (usec): min=64, max=18391, avg=5293.89, stdev=3047.91 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 110], 10.00th=[ 2343], 20.00th=[ 3130], | 30.00th=[ 3589], 40.00th=[ 4293], 50.00th=[ 4883], 60.00th=[ 5735], | 70.00th=[ 6456], 80.00th=[ 7373], 90.00th=[ 8848], 95.00th=[10814], | 99.00th=[15139], 99.50th=[17171], 99.90th=[18482], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 1376, max= 1660, per=24.14%, avg=1530.43, stdev=88.78, samples=7 iops : min= 86, max= 103, avg=95.43, stdev= 5.47, samples=7 write: IOPS=110, BW=1767KiB/s (1809kB/s)(6544KiB/3704msec) clat (usec): min=684, max=24271, avg=4297.53, stdev=3513.32 lat (usec): min=685, max=24272, avg=4304.51, stdev=3510.39 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 906], 10.00th=[ 988], 20.00th=[ 1139], | 30.00th=[ 1418], 40.00th=[ 2376], 50.00th=[ 3425], 60.00th=[ 4293], | 70.00th=[ 5407], 80.00th=[ 7242], 90.00th=[ 9503], 95.00th=[10552], | 99.00th=[14877], 99.50th=[15270], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 1280, max= 2075, per=25.68%, avg=1713.00, stdev=300.06, samples=7 iops : min= 80, max= 129, avg=106.86, stdev=18.69, samples=7 lat (usec) : 100=1.82%, 250=2.08%, 750=0.26%, 1000=5.47% lat (msec) : 2=13.93%, 4=23.05%, 10=46.09%, 20=7.03%, 50=0.26% cpu : usr=0.22%, sys=25.06%, ctx=1702, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21654: Thu May 1 03:51:56 2025 read: IOPS=103, BW=1659KiB/s (1699kB/s)(6256KiB/3771msec) clat (usec): min=51, max=18768, avg=5293.98, stdev=3087.42 lat (usec): min=51, max=18769, avg=5294.69, stdev=3087.42 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 112], 10.00th=[ 1975], 20.00th=[ 3326], | 30.00th=[ 3851], 40.00th=[ 4424], 50.00th=[ 4948], 60.00th=[ 5538], | 70.00th=[ 6259], 80.00th=[ 7242], 90.00th=[ 8586], 95.00th=[10945], | 99.00th=[16712], 99.50th=[17695], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 1344, max= 1948, per=25.44%, avg=1612.71, stdev=221.82, samples=7 iops : min= 84, max= 121, avg=100.57, stdev=13.81, samples=7 write: IOPS=99, BW=1600KiB/s (1638kB/s)(6032KiB/3771msec) clat (usec): min=683, max=20086, avg=4470.61, stdev=3640.58 lat (usec): min=684, max=20087, avg=4471.75, stdev=3640.70 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 889], 10.00th=[ 1004], 20.00th=[ 1172], | 30.00th=[ 1385], 40.00th=[ 2704], 50.00th=[ 3490], 60.00th=[ 4555], | 70.00th=[ 5866], 80.00th=[ 7439], 90.00th=[ 9634], 95.00th=[11469], | 99.00th=[15664], 99.50th=[17957], 99.90th=[20055], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 1312, max= 1760, per=23.08%, avg=1539.57, stdev=168.73, samples=7 iops : min= 82, max= 110, avg=96.00, stdev=10.57, samples=7 lat (usec) : 100=1.56%, 250=3.12%, 500=0.13%, 750=0.52%, 1000=4.56% lat (msec) : 2=13.15%, 4=20.70%, 10=48.83%, 20=7.29%, 50=0.13% cpu : usr=0.21%, sys=24.40%, ctx=1725, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21655: Thu May 1 03:51:56 2025 read: IOPS=99, BW=1597KiB/s (1635kB/s)(6032KiB/3778msec) clat (usec): min=54, max=16335, avg=5323.23, stdev=3008.10 lat (usec): min=54, max=16336, avg=5323.91, stdev=3008.14 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 102], 10.00th=[ 2343], 20.00th=[ 3163], | 30.00th=[ 3654], 40.00th=[ 4228], 50.00th=[ 4883], 60.00th=[ 5669], | 70.00th=[ 6718], 80.00th=[ 7439], 90.00th=[ 9634], 95.00th=[10814], | 99.00th=[13698], 99.50th=[16188], 99.90th=[16319], 99.95th=[16319], | 99.99th=[16319] bw ( KiB/s): min= 1245, max= 1792, per=24.22%, avg=1535.00, stdev=210.94, samples=7 iops : min= 77, max= 112, avg=95.71, stdev=13.30, samples=7 write: IOPS=103, BW=1656KiB/s (1696kB/s)(6256KiB/3778msec) clat (usec): min=631, max=17169, avg=4405.57, stdev=3588.60 lat (usec): min=631, max=17169, avg=4406.88, stdev=3588.62 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 914], 10.00th=[ 1037], 20.00th=[ 1156], | 30.00th=[ 1467], 40.00th=[ 2671], 50.00th=[ 3589], 60.00th=[ 4293], | 70.00th=[ 5211], 80.00th=[ 7111], 90.00th=[10159], 95.00th=[12125], | 99.00th=[15401], 99.50th=[16057], 99.90th=[17171], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 1440, max= 1888, per=24.31%, avg=1621.71, stdev=139.83, samples=7 iops : min= 90, max= 118, avg=101.14, stdev= 8.88, samples=7 lat (usec) : 100=2.08%, 250=1.95%, 750=0.91%, 1000=4.04% lat (msec) : 2=12.76%, 4=25.00%, 10=44.01%, 20=9.24% cpu : usr=0.26%, sys=23.88%, ctx=1767, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6340KiB/s (6492kB/s), 1551KiB/s-1659KiB/s (1588kB/s-1699kB/s), io=23.4MiB (24.5MB), run=3704-3778msec WRITE: bw=6670KiB/s (6830kB/s), 1600KiB/s-1767KiB/s (1638kB/s-1809kB/s), io=24.6MiB (25.8MB), run=3704-3778msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21656: Thu May 1 03:51:57 2025 read: IOPS=79, BW=1274KiB/s (1305kB/s)(5920KiB/4647msec) clat (usec): min=1756, max=19918, avg=4665.74, stdev=3420.70 lat (usec): min=1756, max=19919, avg=4666.44, stdev=3420.71 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2114], 10.00th=[ 2180], 20.00th=[ 2343], | 30.00th=[ 2573], 40.00th=[ 2933], 50.00th=[ 3294], 60.00th=[ 3851], | 70.00th=[ 4555], 80.00th=[ 6259], 90.00th=[10028], 95.00th=[12780], | 99.00th=[16712], 99.50th=[17433], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 672, max= 2368, per=24.63%, avg=1240.56, stdev=532.69, samples=9 iops : min= 42, max= 148, avg=77.44, stdev=33.34, samples=9 write: IOPS=85, BW=1370KiB/s (1403kB/s)(6368KiB/4647msec) clat (usec): min=2515, max=27069, avg=7172.93, stdev=4314.40 lat (usec): min=2517, max=27072, avg=7174.52, stdev=4314.42 clat percentiles (usec): | 1.00th=[ 2802], 5.00th=[ 3064], 10.00th=[ 3392], 20.00th=[ 3654], | 30.00th=[ 4047], 40.00th=[ 4686], 50.00th=[ 5604], 60.00th=[ 6587], | 70.00th=[ 8455], 80.00th=[10814], 90.00th=[13435], 95.00th=[15533], | 99.00th=[21890], 99.50th=[26870], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 832, max= 2656, per=25.16%, avg=1333.00, stdev=542.78, samples=9 iops : min= 52, max= 166, avg=83.22, stdev=33.98, samples=9 lat (msec) : 2=0.78%, 4=44.53%, 10=37.63%, 20=16.28%, 50=0.78% cpu : usr=0.28%, sys=11.56%, ctx=928, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21657: Thu May 1 03:51:57 2025 read: IOPS=76, BW=1219KiB/s (1248kB/s)(5744KiB/4712msec) clat (usec): min=1843, max=24506, avg=5208.12, stdev=4064.43 lat (usec): min=1844, max=24507, avg=5208.95, stdev=4064.44 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2024], 10.00th=[ 2180], 20.00th=[ 2343], | 30.00th=[ 2638], 40.00th=[ 2900], 50.00th=[ 3359], 60.00th=[ 4047], | 70.00th=[ 5473], 80.00th=[ 7570], 90.00th=[12125], 95.00th=[13829], | 99.00th=[19268], 99.50th=[20317], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 768, max= 2176, per=23.57%, avg=1187.22, stdev=503.36, samples=9 iops : min= 48, max= 136, avg=74.11, stdev=31.48, samples=9 write: IOPS=86, BW=1389KiB/s (1422kB/s)(6544KiB/4712msec) clat (usec): min=2282, max=27360, avg=6907.76, stdev=4108.05 lat (usec): min=2283, max=27362, avg=6909.31, stdev=4108.00 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 3130], 10.00th=[ 3458], 20.00th=[ 3720], | 30.00th=[ 3982], 40.00th=[ 4490], 50.00th=[ 5211], 60.00th=[ 6587], | 70.00th=[ 8029], 80.00th=[10159], 90.00th=[12518], 95.00th=[14746], | 99.00th=[20317], 99.50th=[23200], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 894, max= 2816, per=25.09%, avg=1329.56, stdev=596.09, samples=9 iops : min= 55, max= 176, avg=83.00, stdev=37.34, samples=9 lat (msec) : 2=1.95%, 4=42.19%, 10=38.80%, 20=16.02%, 50=1.04% cpu : usr=0.23%, sys=11.50%, ctx=908, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21658: Thu May 1 03:51:57 2025 read: IOPS=82, BW=1316KiB/s (1347kB/s)(6256KiB/4755msec) clat (usec): min=1737, max=22221, avg=5110.27, stdev=4027.25 lat (usec): min=1737, max=22222, avg=5110.96, stdev=4027.28 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2089], 10.00th=[ 2180], 20.00th=[ 2311], | 30.00th=[ 2540], 40.00th=[ 2868], 50.00th=[ 3326], 60.00th=[ 3982], | 70.00th=[ 5014], 80.00th=[ 7570], 90.00th=[11731], 95.00th=[13960], | 99.00th=[19530], 99.50th=[20579], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 768, max= 2848, per=25.13%, avg=1265.56, stdev=690.95, samples=9 iops : min= 48, max= 178, avg=79.00, stdev=43.25, samples=9 write: IOPS=79, BW=1269KiB/s (1299kB/s)(6032KiB/4755msec) clat (usec): min=2598, max=36528, avg=7169.39, stdev=4623.62 lat (usec): min=2599, max=36530, avg=7170.70, stdev=4623.74 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 3097], 10.00th=[ 3392], 20.00th=[ 3687], | 30.00th=[ 4146], 40.00th=[ 4621], 50.00th=[ 5276], 60.00th=[ 6652], | 70.00th=[ 7963], 80.00th=[10290], 90.00th=[13829], 95.00th=[16712], | 99.00th=[22938], 99.50th=[23725], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 704, max= 2368, per=22.74%, avg=1205.11, stdev=518.81, samples=9 iops : min= 44, max= 148, avg=75.22, stdev=32.47, samples=9 lat (msec) : 2=1.56%, 4=42.06%, 10=38.41%, 20=16.67%, 50=1.30% cpu : usr=0.42%, sys=11.15%, ctx=954, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21659: Thu May 1 03:51:57 2025 read: IOPS=79, BW=1278KiB/s (1309kB/s)(6032KiB/4720msec) clat (usec): min=1744, max=24241, avg=4952.89, stdev=4036.80 lat (usec): min=1744, max=24242, avg=4953.59, stdev=4036.84 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 2008], 10.00th=[ 2180], 20.00th=[ 2376], | 30.00th=[ 2606], 40.00th=[ 2802], 50.00th=[ 3294], 60.00th=[ 3884], | 70.00th=[ 4621], 80.00th=[ 6783], 90.00th=[11600], 95.00th=[13960], | 99.00th=[20841], 99.50th=[24249], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 832, max= 2592, per=24.13%, avg=1215.67, stdev=550.66, samples=9 iops : min= 52, max= 162, avg=75.89, stdev=34.45, samples=9 write: IOPS=82, BW=1325KiB/s (1357kB/s)(6256KiB/4720msec) clat (usec): min=2295, max=23770, avg=7139.25, stdev=4428.91 lat (usec): min=2296, max=23771, avg=7140.90, stdev=4428.73 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 3097], 10.00th=[ 3326], 20.00th=[ 3687], | 30.00th=[ 4047], 40.00th=[ 4490], 50.00th=[ 5342], 60.00th=[ 6456], | 70.00th=[ 8225], 80.00th=[10421], 90.00th=[13829], 95.00th=[16057], | 99.00th=[22152], 99.50th=[23200], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 864, max= 2528, per=24.15%, avg=1279.67, stdev=571.01, samples=9 iops : min= 54, max= 158, avg=79.89, stdev=35.73, samples=9 lat (msec) : 2=2.21%, 4=42.45%, 10=38.54%, 20=15.23%, 50=1.56% cpu : usr=0.11%, sys=11.49%, ctx=956, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5037KiB/s (5158kB/s), 1219KiB/s-1316KiB/s (1248kB/s-1347kB/s), io=23.4MiB (24.5MB), run=4647-4755msec WRITE: bw=5300KiB/s (5427kB/s), 1269KiB/s-1389KiB/s (1299kB/s-1422kB/s), io=24.6MiB (25.8MB), run=4647-4755msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21668: Thu May 1 03:51:59 2025 read: IOPS=10, BW=10.1MiB/s (10.6MB/s)(7168KiB/691msec) clat (msec): min=10, max=101, avg=45.60, stdev=37.17 lat (msec): min=10, max=101, avg=45.60, stdev=37.17 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 11], | 30.00th=[ 14], 40.00th=[ 14], 50.00th=[ 35], 60.00th=[ 73], | 70.00th=[ 73], 80.00th=[ 74], 90.00th=[ 102], 95.00th=[ 102], | 99.00th=[ 102], 99.50th=[ 102], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min=10240, max=10240, per=39.70%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7410KiB/s (7587kB/s)(5120KiB/691msec) clat (msec): min=18, max=162, avg=72.10, stdev=59.82 lat (msec): min=18, max=162, avg=72.19, stdev=59.82 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 66], 60.00th=[ 66], | 70.00th=[ 94], 80.00th=[ 94], 90.00th=[ 163], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 6144, max= 6144, per=17.01%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=16.67%, 100=33.33%, 250=16.67% cpu : usr=0.29%, sys=16.81%, ctx=80, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21670: Thu May 1 03:51:59 2025 read: IOPS=2, BW=2880KiB/s (2950kB/s)(2048KiB/711msec) clat (usec): min=40801, max=48512, avg=44656.85, stdev=5452.50 lat (usec): min=40802, max=48513, avg=44658.18, stdev=5452.46 clat percentiles (usec): | 1.00th=[40633], 5.00th=[40633], 10.00th=[40633], 20.00th=[40633], | 30.00th=[40633], 40.00th=[40633], 50.00th=[40633], 60.00th=[48497], | 70.00th=[48497], 80.00th=[48497], 90.00th=[48497], 95.00th=[48497], | 99.00th=[48497], 99.50th=[48497], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 2048, max= 2048, per=7.94%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=14, BW=14.1MiB/s (14.7MB/s)(10.0MiB/711msec) clat (msec): min=12, max=145, avg=58.31, stdev=45.30 lat (msec): min=13, max=145, avg=58.48, stdev=45.20 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 19], 50.00th=[ 42], 60.00th=[ 71], | 70.00th=[ 86], 80.00th=[ 86], 90.00th=[ 100], 95.00th=[ 146], | 99.00th=[ 146], 99.50th=[ 146], 99.90th=[ 146], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 8192, max= 8192, per=22.69%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=25.00%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=15.21%, ctx=76, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21672: Thu May 1 03:51:59 2025 read: IOPS=7, BW=7777KiB/s (7964kB/s)(6144KiB/790msec) clat (msec): min=22, max=111, avg=56.63, stdev=29.51 lat (msec): min=22, max=111, avg=56.63, stdev=29.51 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 49], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 61], 80.00th=[ 61], 90.00th=[ 112], 95.00th=[ 112], | 99.00th=[ 112], 99.50th=[ 112], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 4096, max= 4096, per=15.88%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7777KiB/s (7964kB/s)(6144KiB/790msec) clat (msec): min=17, max=150, avg=69.06, stdev=53.07 lat (msec): min=17, max=150, avg=69.15, stdev=53.08 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 21], | 30.00th=[ 21], 40.00th=[ 54], 50.00th=[ 54], 60.00th=[ 57], | 70.00th=[ 115], 80.00th=[ 115], 90.00th=[ 150], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 6144, max= 6144, per=17.01%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=33.33%, 250=25.00% cpu : usr=0.00%, sys=10.39%, ctx=68, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21674: Thu May 1 03:51:59 2025 read: IOPS=6, BW=6448KiB/s (6603kB/s)(5120KiB/794msec) clat (usec): min=12850, max=67815, avg=42013.44, stdev=24012.12 lat (usec): min=12852, max=67816, avg=42014.66, stdev=24012.05 clat percentiles (usec): | 1.00th=[12911], 5.00th=[12911], 10.00th=[12911], 20.00th=[12911], | 30.00th=[24511], 40.00th=[24511], 50.00th=[41157], 60.00th=[41157], | 70.00th=[63701], 80.00th=[63701], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 2048, max= 2048, per=7.94%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=8, BW=9028KiB/s (9244kB/s)(7168KiB/794msec) clat (msec): min=16, max=143, avg=76.24, stdev=51.07 lat (msec): min=16, max=143, avg=76.31, stdev=51.08 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 19], | 30.00th=[ 59], 40.00th=[ 59], 50.00th=[ 74], 60.00th=[ 86], | 70.00th=[ 86], 80.00th=[ 140], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 8192, max= 8192, per=22.69%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=16.67%, 100=41.67%, 250=16.67% cpu : usr=0.00%, sys=14.75%, ctx=70, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.2MiB/s (26.4MB/s), 2880KiB/s-10.1MiB/s (2950kB/s-10.6MB/s), io=20.0MiB (20.0MB), run=691-794msec WRITE: bw=35.3MiB/s (36.0MB/s), 7410KiB/s-14.1MiB/s (7587kB/s-14.7MB/s), io=28.0MiB (29.4MB), run=691-794msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21669: Thu May 1 03:51:59 2025 read: IOPS=8, BW=8463KiB/s (8666kB/s)(7168KiB/847msec) clat (msec): min=28, max=276, avg=73.96, stdev=89.86 lat (msec): min=28, max=276, avg=73.96, stdev=89.86 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 29], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 40], 60.00th=[ 54], | 70.00th=[ 54], 80.00th=[ 56], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 5389, max= 5389, per=22.29%, avg=5389.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 write: IOPS=5, BW=6045KiB/s (6190kB/s)(5120KiB/847msec) clat (usec): min=37429, max=54580, avg=44902.00, stdev=8787.22 lat (usec): min=37546, max=54610, avg=44968.70, stdev=8770.72 clat percentiles (usec): | 1.00th=[37487], 5.00th=[37487], 10.00th=[37487], 20.00th=[37487], | 30.00th=[38536], 40.00th=[38536], 50.00th=[39584], 60.00th=[39584], | 70.00th=[54264], 80.00th=[54264], 90.00th=[54789], 95.00th=[54789], | 99.00th=[54789], 99.50th=[54789], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 1796, max= 1796, per=5.31%, avg=1796.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=33.33%, 500=8.33% cpu : usr=0.12%, sys=29.43%, ctx=1166, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21671: Thu May 1 03:51:59 2025 read: IOPS=3, BW=3293KiB/s (3372kB/s)(2048KiB/622msec) clat (usec): min=508, max=29566, avg=15037.69, stdev=20546.71 lat (usec): min=509, max=29567, avg=15038.63, stdev=20546.88 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 510], 10.00th=[ 510], 20.00th=[ 510], | 30.00th=[ 510], 40.00th=[ 510], 50.00th=[ 510], 60.00th=[29492], | 70.00th=[29492], 80.00th=[29492], 90.00th=[29492], 95.00th=[29492], | 99.00th=[29492], 99.50th=[29492], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 3631, max= 3631, per=15.02%, avg=3631.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=16, BW=16.1MiB/s (16.9MB/s)(10.0MiB/622msec) clat (msec): min=25, max=156, avg=56.14, stdev=38.90 lat (msec): min=25, max=156, avg=56.21, stdev=38.90 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 27], | 30.00th=[ 33], 40.00th=[ 38], 50.00th=[ 39], 60.00th=[ 48], | 70.00th=[ 66], 80.00th=[ 66], 90.00th=[ 77], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min=14524, max=14524, per=42.91%, avg=14524.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=58.33%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=37.20%, ctx=365, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21673: Thu May 1 03:51:59 2025 read: IOPS=9, BW=9942KiB/s (10.2MB/s)(6144KiB/618msec) clat (usec): min=544, max=82316, avg=37920.68, stdev=27349.15 lat (usec): min=545, max=82317, avg=37921.91, stdev=27349.14 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 545], 10.00th=[ 545], 20.00th=[29492], | 30.00th=[29492], 40.00th=[29754], 50.00th=[29754], 60.00th=[32900], | 70.00th=[52691], 80.00th=[52691], 90.00th=[82314], 95.00th=[82314], | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 7301, max= 7301, per=30.20%, avg=7301.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9942KiB/s (10.2MB/s)(6144KiB/618msec) clat (msec): min=26, max=119, avg=53.49, stdev=33.39 lat (msec): min=26, max=119, avg=53.59, stdev=33.40 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 35], | 30.00th=[ 35], 40.00th=[ 42], 50.00th=[ 42], 60.00th=[ 47], | 70.00th=[ 54], 80.00th=[ 54], 90.00th=[ 120], 95.00th=[ 120], | 99.00th=[ 120], 99.50th=[ 120], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 9126, max= 9126, per=26.96%, avg=9126.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=58.33%, 100=25.00%, 250=8.33% cpu : usr=0.16%, sys=38.25%, ctx=1002, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21675: Thu May 1 03:51:59 2025 read: IOPS=10, BW=10.2MiB/s (10.7MB/s)(5120KiB/488msec) clat (usec): min=558, max=78630, avg=34745.90, stdev=34023.64 lat (usec): min=560, max=78631, avg=34747.40, stdev=34023.27 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 562], 10.00th=[ 562], 20.00th=[ 562], | 30.00th=[ 676], 40.00th=[ 676], 50.00th=[40633], 60.00th=[40633], | 70.00th=[53216], 80.00th=[53216], 90.00th=[78119], 95.00th=[78119], | 99.00th=[78119], 99.50th=[78119], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] write: IOPS=14, BW=14.3MiB/s (15.0MB/s)(7168KiB/488msec) clat (usec): min=28853, max=72190, avg=43786.00, stdev=15287.24 lat (usec): min=28979, max=72261, avg=43869.78, stdev=15278.24 clat percentiles (usec): | 1.00th=[28967], 5.00th=[28967], 10.00th=[28967], 20.00th=[30802], | 30.00th=[33817], 40.00th=[33817], 50.00th=[42730], 60.00th=[43779], | 70.00th=[43779], 80.00th=[53740], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] lat (usec) : 750=16.67% lat (msec) : 50=50.00%, 100=33.33% cpu : usr=0.21%, sys=48.25%, ctx=130, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=23.6MiB/s (24.8MB/s), 3293KiB/s-10.2MiB/s (3372kB/s-10.7MB/s), io=20.0MiB (20.0MB), run=488-847msec WRITE: bw=33.1MiB/s (34.7MB/s), 6045KiB/s-16.1MiB/s (6190kB/s-16.9MB/s), io=28.0MiB (29.4MB), run=488-847msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21682: Thu May 1 03:52:01 2025 read: IOPS=6, BW=25.2MiB/s (26.4MB/s)(12.0MiB/476msec) clat (msec): min=41, max=290, avg=154.68, stdev=126.12 lat (msec): min=41, max=290, avg=154.68, stdev=126.12 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 133], 50.00th=[ 133], 60.00th=[ 133], | 70.00th=[ 292], 80.00th=[ 292], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] lat (msec) : 50=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=26.32%, ctx=921, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21684: Thu May 1 03:52:01 2025 read: IOPS=1, BW=6759KiB/s (6921kB/s)(4096KiB/606msec) clat (nsec): min=212927k, max=212927k, avg=212927458.00, stdev= 0.00 lat (nsec): min=212930k, max=212930k, avg=212930188.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 213], 10.00th=[ 213], 20.00th=[ 213], | 30.00th=[ 213], 40.00th=[ 213], 50.00th=[ 213], 60.00th=[ 213], | 70.00th=[ 213], 80.00th=[ 213], 90.00th=[ 213], 95.00th=[ 213], | 99.00th=[ 213], 99.50th=[ 213], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 8192, max= 8192, per=30.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=13.2MiB/s (13.8MB/s)(8192KiB/606msec) clat (msec): min=133, max=211, avg=172.73, stdev=55.25 lat (msec): min=134, max=211, avg=173.02, stdev=55.12 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 134], 10.00th=[ 134], 20.00th=[ 134], | 30.00th=[ 134], 40.00th=[ 134], 50.00th=[ 134], 60.00th=[ 211], | 70.00th=[ 211], 80.00th=[ 211], 90.00th=[ 211], 95.00th=[ 211], | 99.00th=[ 211], 99.50th=[ 211], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 8192, max= 8192, per=21.58%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=51.57%, ctx=724, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21686: Thu May 1 03:52:01 2025 write: IOPS=3, BW=13.9MiB/s (14.6MB/s)(12.0MiB/863msec) clat (msec): min=103, max=506, avg=284.92, stdev=204.41 lat (msec): min=103, max=506, avg=285.18, stdev=204.34 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 104], | 30.00th=[ 104], 40.00th=[ 245], 50.00th=[ 245], 60.00th=[ 245], | 70.00th=[ 506], 80.00th=[ 506], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.23%, sys=36.43%, ctx=37, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21689: Thu May 1 03:52:01 2025 write: IOPS=4, BW=18.2MiB/s (19.1MB/s)(12.0MiB/659msec) clat (msec): min=128, max=340, avg=211.01, stdev=113.23 lat (msec): min=128, max=340, avg=211.26, stdev=113.28 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 129], 10.00th=[ 129], 20.00th=[ 129], | 30.00th=[ 129], 40.00th=[ 165], 50.00th=[ 165], 60.00th=[ 165], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 342], 95.00th=[ 342], | 99.00th=[ 342], 99.50th=[ 342], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min=16351, max=16351, per=43.06%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.15%, sys=50.91%, ctx=40, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=26.4MiB/s (27.7MB/s), 6759KiB/s-25.2MiB/s (6921kB/s-26.4MB/s), io=16.0MiB (16.8MB), run=476-606msec WRITE: bw=37.1MiB/s (38.9MB/s), 13.2MiB/s-18.2MiB/s (13.8MB/s-19.1MB/s), io=32.0MiB (33.6MB), run=606-863msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21683: Thu May 1 03:52:01 2025 read: IOPS=8, BW=33.7MiB/s (35.3MB/s)(12.0MiB/356msec) clat (msec): min=65, max=134, avg=108.82, stdev=37.47 lat (msec): min=65, max=134, avg=108.82, stdev=37.47 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 66], 10.00th=[ 66], 20.00th=[ 66], | 30.00th=[ 66], 40.00th=[ 127], 50.00th=[ 127], 60.00th=[ 127], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=22.82%, ctx=41, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21685: Thu May 1 03:52:01 2025 read: IOPS=1, BW=4602KiB/s (4713kB/s)(4096KiB/890msec) clat (nsec): min=288112k, max=288112k, avg=288111914.00, stdev= 0.00 lat (nsec): min=288114k, max=288114k, avg=288113715.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 288], 10.00th=[ 288], 20.00th=[ 288], | 30.00th=[ 288], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 288], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 8192, max= 8192, per=44.50%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9204KiB/s (9425kB/s)(8192KiB/890msec) clat (msec): min=204, max=388, avg=296.46, stdev=130.65 lat (msec): min=204, max=389, avg=296.80, stdev=130.58 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 205], 10.00th=[ 205], 20.00th=[ 205], | 30.00th=[ 205], 40.00th=[ 205], 50.00th=[ 205], 60.00th=[ 388], | 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.34%, sys=16.42%, ctx=186, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21687: Thu May 1 03:52:01 2025 write: IOPS=4, BW=18.2MiB/s (19.1MB/s)(12.0MiB/658msec) clat (msec): min=70, max=389, avg=216.40, stdev=161.56 lat (msec): min=70, max=390, avg=216.77, stdev=161.50 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 71], 10.00th=[ 71], 20.00th=[ 71], | 30.00th=[ 71], 40.00th=[ 190], 50.00th=[ 190], 60.00th=[ 190], | 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min=16384, max=16384, per=44.50%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.15%, sys=14.31%, ctx=43, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21688: Thu May 1 03:52:01 2025 write: IOPS=3, BW=15.0MiB/s (15.8MB/s)(12.0MiB/798msec) clat (msec): min=206, max=314, avg=245.52, stdev=59.91 lat (msec): min=207, max=314, avg=245.85, stdev=59.92 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 207], 10.00th=[ 207], 20.00th=[ 207], | 30.00th=[ 207], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 215], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 313], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min=16384, max=16384, per=44.50%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=22.71%, ctx=154, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.0MiB/s (18.8MB/s), 4602KiB/s-33.7MiB/s (4713kB/s-35.3MB/s), io=16.0MiB (16.8MB), run=356-890msec WRITE: bw=35.0MiB/s (37.7MB/s), 9204KiB/s-18.2MiB/s (9425kB/s-19.1MB/s), io=32.0MiB (33.6MB), run=658-890msec