== sanity test 398b: DIO and buffer IO race ============== 23:39:41 (1745465981) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.28404 s, 39.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1954: Wed Apr 23 23:39:55 2025 read: IOPS=130, BW=524KiB/s (536kB/s)(5992KiB/11439msec) clat (usec): min=54, max=148563, avg=4242.15, stdev=4326.50 lat (usec): min=55, max=148565, avg=4242.86, stdev=4326.49 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 1811], 10.00th=[ 2114], 20.00th=[ 2442], | 30.00th=[ 2835], 40.00th=[ 3294], 50.00th=[ 3785], 60.00th=[ 4293], | 70.00th=[ 4883], 80.00th=[ 5604], 90.00th=[ 6915], 95.00th=[ 7898], | 99.00th=[ 11600], 99.50th=[ 13960], 99.90th=[ 21890], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 232, max= 648, per=24.38%, avg=511.41, stdev=84.05, samples=22 iops : min= 58, max= 162, avg=127.77, stdev=21.05, samples=22 write: IOPS=137, BW=550KiB/s (564kB/s)(6296KiB/11439msec) clat (usec): min=431, max=22076, avg=3167.78, stdev=2753.05 lat (usec): min=432, max=22077, avg=3168.78, stdev=2753.15 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 611], | 30.00th=[ 758], 40.00th=[ 1860], 50.00th=[ 2573], 60.00th=[ 3228], | 70.00th=[ 4178], 80.00th=[ 5211], 90.00th=[ 7177], 95.00th=[ 8848], | 99.00th=[11076], 99.50th=[12125], 99.90th=[16319], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 272, max= 672, per=24.48%, avg=537.91, stdev=86.42, samples=22 iops : min= 68, max= 168, avg=134.41, stdev=21.60, samples=22 lat (usec) : 100=1.50%, 250=0.10%, 500=0.62%, 750=14.75%, 1000=2.38% lat (msec) : 2=5.14%, 4=37.01%, 10=36.39%, 20=2.02%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.33%, sys=21.98%, ctx=5859, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1956: Wed Apr 23 23:39:55 2025 read: IOPS=130, BW=522KiB/s (535kB/s)(5940KiB/11374msec) clat (usec): min=60, max=18673, avg=4150.24, stdev=2234.53 lat (usec): min=61, max=18674, avg=4150.89, stdev=2234.55 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 1827], 10.00th=[ 2089], 20.00th=[ 2409], | 30.00th=[ 2769], 40.00th=[ 3163], 50.00th=[ 3720], 60.00th=[ 4359], | 70.00th=[ 4948], 80.00th=[ 5669], 90.00th=[ 6915], 95.00th=[ 7832], | 99.00th=[11863], 99.50th=[13698], 99.90th=[15664], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 256, max= 632, per=24.52%, avg=514.45, stdev=77.03, samples=22 iops : min= 64, max= 158, avg=128.59, stdev=19.24, samples=22 write: IOPS=139, BW=558KiB/s (572kB/s)(6348KiB/11374msec) clat (usec): min=464, max=161101, avg=3231.27, stdev=4818.01 lat (usec): min=465, max=161102, avg=3233.44, stdev=4818.03 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 603], | 30.00th=[ 750], 40.00th=[ 1926], 50.00th=[ 2573], 60.00th=[ 3130], | 70.00th=[ 3982], 80.00th=[ 5211], 90.00th=[ 7046], 95.00th=[ 8455], | 99.00th=[ 12649], 99.50th=[ 13698], 99.90th=[ 15795], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 304, max= 664, per=24.72%, avg=543.18, stdev=76.14, samples=22 iops : min= 76, max= 166, avg=135.77, stdev=19.02, samples=22 lat (usec) : 100=1.63%, 250=0.13%, 500=0.62%, 750=14.97%, 1000=2.25% lat (msec) : 2=5.27%, 4=37.66%, 10=35.32%, 20=2.12%, 250=0.03% cpu : usr=0.18%, sys=22.49%, ctx=5821, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1959: Wed Apr 23 23:39:55 2025 read: IOPS=131, BW=527KiB/s (540kB/s)(6028KiB/11440msec) clat (usec): min=54, max=18720, avg=4071.68, stdev=2333.43 lat (usec): min=54, max=18721, avg=4072.67, stdev=2333.63 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 89], 10.00th=[ 1909], 20.00th=[ 2343], | 30.00th=[ 2704], 40.00th=[ 3163], 50.00th=[ 3654], 60.00th=[ 4293], | 70.00th=[ 4883], 80.00th=[ 5669], 90.00th=[ 6849], 95.00th=[ 8029], | 99.00th=[11863], 99.50th=[14353], 99.90th=[16712], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 192, max= 680, per=24.63%, avg=516.68, stdev=93.76, samples=22 iops : min= 48, max= 170, avg=129.14, stdev=23.45, samples=22 write: IOPS=136, BW=547KiB/s (560kB/s)(6260KiB/11440msec) clat (usec): min=440, max=160098, avg=3329.04, stdev=4883.42 lat (usec): min=441, max=160101, avg=3330.46, stdev=4883.61 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 603], | 30.00th=[ 766], 40.00th=[ 1909], 50.00th=[ 2606], 60.00th=[ 3228], | 70.00th=[ 4228], 80.00th=[ 5407], 90.00th=[ 7111], 95.00th=[ 8717], | 99.00th=[ 12256], 99.50th=[ 13829], 99.90th=[ 19530], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 256, max= 736, per=24.31%, avg=534.09, stdev=104.57, samples=22 iops : min= 64, max= 184, avg=133.50, stdev=26.13, samples=22 lat (usec) : 100=2.57%, 250=0.26%, 500=0.46%, 750=14.75%, 1000=1.95% lat (msec) : 2=6.87%, 4=35.19%, 10=35.55%, 20=2.38%, 250=0.03% cpu : usr=0.14%, sys=21.96%, ctx=5728, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1960: Wed Apr 23 23:39:55 2025 read: IOPS=134, BW=536KiB/s (549kB/s)(6052KiB/11290msec) clat (usec): min=52, max=18076, avg=4086.41, stdev=2306.97 lat (usec): min=52, max=18077, avg=4087.71, stdev=2306.31 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 1762], 10.00th=[ 2024], 20.00th=[ 2376], | 30.00th=[ 2704], 40.00th=[ 3130], 50.00th=[ 3687], 60.00th=[ 4228], | 70.00th=[ 4817], 80.00th=[ 5538], 90.00th=[ 6521], 95.00th=[ 8094], | 99.00th=[12780], 99.50th=[14615], 99.90th=[16188], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 248, max= 648, per=25.27%, avg=530.09, stdev=86.23, samples=22 iops : min= 62, max= 162, avg=132.50, stdev=21.54, samples=22 write: IOPS=138, BW=552KiB/s (566kB/s)(6236KiB/11291msec) clat (usec): min=463, max=169676, avg=3222.45, stdev=5038.96 lat (usec): min=464, max=169678, avg=3223.15, stdev=5038.96 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 586], | 30.00th=[ 725], 40.00th=[ 1795], 50.00th=[ 2540], 60.00th=[ 3195], | 70.00th=[ 4080], 80.00th=[ 5276], 90.00th=[ 6980], 95.00th=[ 8586], | 99.00th=[ 11600], 99.50th=[ 12780], 99.90th=[ 20055], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 304, max= 792, per=24.87%, avg=546.45, stdev=98.29, samples=22 iops : min= 76, max= 198, avg=136.59, stdev=24.55, samples=22 lat (usec) : 100=1.60%, 250=0.23%, 500=0.65%, 750=14.97%, 1000=2.34% lat (msec) : 2=6.22%, 4=37.24%, 10=34.15%, 20=2.54%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=21.98%, ctx=5751, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2099KiB/s (2149kB/s), 522KiB/s-536KiB/s (535kB/s-549kB/s), io=23.4MiB (24.6MB), run=11290-11440msec WRITE: bw=2198KiB/s (2250kB/s), 547KiB/s-558KiB/s (560kB/s-572kB/s), io=24.6MiB (25.7MB), run=11291-11440msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1953: Wed Apr 23 23:40:02 2025 read: IOPS=81, BW=326KiB/s (334kB/s)(5992KiB/18359msec) clat (usec): min=1677, max=164948, avg=4973.81, stdev=4959.86 lat (usec): min=1678, max=164950, avg=4974.84, stdev=4959.83 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2114], 10.00th=[ 2376], 20.00th=[ 2835], | 30.00th=[ 3490], 40.00th=[ 3949], 50.00th=[ 4178], 60.00th=[ 4424], | 70.00th=[ 4817], 80.00th=[ 6063], 90.00th=[ 8848], 95.00th=[ 10945], | 99.00th=[ 15533], 99.50th=[ 16909], 99.90th=[ 19792], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 136, max= 512, per=25.04%, avg=326.75, stdev=71.02, samples=36 iops : min= 34, max= 128, avg=81.58, stdev=17.76, samples=36 write: IOPS=85, BW=343KiB/s (351kB/s)(6296KiB/18359msec) clat (usec): min=2513, max=24622, avg=6875.98, stdev=2791.38 lat (usec): min=2513, max=24623, avg=6876.96, stdev=2791.37 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3359], 10.00th=[ 3752], 20.00th=[ 4555], | 30.00th=[ 5473], 40.00th=[ 6259], 50.00th=[ 6652], 60.00th=[ 6980], | 70.00th=[ 7439], 80.00th=[ 8356], 90.00th=[10159], 95.00th=[12256], | 99.00th=[16909], 99.50th=[19006], 99.90th=[22152], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 144, max= 544, per=25.04%, avg=342.31, stdev=61.59, samples=36 iops : min= 36, max= 136, avg=85.47, stdev=15.41, samples=36 lat (msec) : 2=1.27%, 4=26.33%, 10=63.83%, 20=8.33%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=12.64%, ctx=3613, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1955: Wed Apr 23 23:40:02 2025 read: IOPS=81, BW=325KiB/s (332kB/s)(5940KiB/18304msec) clat (usec): min=1598, max=164672, avg=4875.31, stdev=4900.30 lat (usec): min=1598, max=164675, avg=4876.17, stdev=4900.34 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2147], 10.00th=[ 2311], 20.00th=[ 2737], | 30.00th=[ 3458], 40.00th=[ 3884], 50.00th=[ 4146], 60.00th=[ 4359], | 70.00th=[ 4883], 80.00th=[ 5932], 90.00th=[ 8455], 95.00th=[ 10552], | 99.00th=[ 15139], 99.50th=[ 16450], 99.90th=[ 20055], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 120, max= 496, per=24.94%, avg=325.47, stdev=75.45, samples=36 iops : min= 30, max= 124, avg=81.31, stdev=18.84, samples=36 write: IOPS=86, BW=347KiB/s (355kB/s)(6348KiB/18304msec) clat (usec): min=2458, max=21746, avg=6917.91, stdev=2793.42 lat (usec): min=2459, max=21746, avg=6918.91, stdev=2793.42 clat percentiles (usec): | 1.00th=[ 2999], 5.00th=[ 3392], 10.00th=[ 3785], 20.00th=[ 4490], | 30.00th=[ 5473], 40.00th=[ 6194], 50.00th=[ 6652], 60.00th=[ 7046], | 70.00th=[ 7439], 80.00th=[ 8455], 90.00th=[10421], 95.00th=[12518], | 99.00th=[16909], 99.50th=[19268], 99.90th=[20317], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 200, max= 560, per=25.30%, avg=345.92, stdev=60.99, samples=36 iops : min= 50, max= 140, avg=86.42, stdev=15.18, samples=36 lat (msec) : 2=1.24%, 4=27.38%, 10=62.73%, 20=8.43%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=12.74%, ctx=3602, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1957: Wed Apr 23 23:40:02 2025 read: IOPS=81, BW=328KiB/s (336kB/s)(6028KiB/18386msec) clat (usec): min=1634, max=25252, avg=4784.42, stdev=2747.25 lat (usec): min=1634, max=25253, avg=4785.24, stdev=2747.22 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 2089], 10.00th=[ 2343], 20.00th=[ 2802], | 30.00th=[ 3425], 40.00th=[ 3851], 50.00th=[ 4113], 60.00th=[ 4359], | 70.00th=[ 4686], 80.00th=[ 5932], 90.00th=[ 8586], 95.00th=[10421], | 99.00th=[15401], 99.50th=[16319], 99.90th=[19530], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 136, max= 488, per=25.14%, avg=328.11, stdev=65.53, samples=36 iops : min= 34, max= 122, avg=81.94, stdev=16.29, samples=36 write: IOPS=85, BW=340KiB/s (349kB/s)(6260KiB/18386msec) clat (msec): min=2, max=167, avg= 7.09, stdev= 4.96 lat (msec): min=2, max=167, avg= 7.09, stdev= 4.96 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 18], 99.50th=[ 21], 99.90th=[ 25], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 176, max= 576, per=24.86%, avg=339.89, stdev=71.63, samples=36 iops : min= 44, max= 144, avg=84.89, stdev=17.97, samples=36 lat (msec) : 2=1.63%, 4=27.15%, 10=62.57%, 20=8.30%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=12.50%, ctx=3642, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1958: Wed Apr 23 23:40:02 2025 read: IOPS=82, BW=331KiB/s (339kB/s)(6052KiB/18279msec) clat (usec): min=1590, max=24837, avg=4916.67, stdev=2721.86 lat (usec): min=1591, max=24837, avg=4917.50, stdev=2721.86 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2114], 10.00th=[ 2409], 20.00th=[ 2868], | 30.00th=[ 3556], 40.00th=[ 3949], 50.00th=[ 4178], 60.00th=[ 4424], | 70.00th=[ 5014], 80.00th=[ 6128], 90.00th=[ 9110], 95.00th=[10814], | 99.00th=[13960], 99.50th=[15795], 99.90th=[18744], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 144, max= 552, per=25.32%, avg=330.36, stdev=72.77, samples=36 iops : min= 36, max= 138, avg=82.53, stdev=18.18, samples=36 write: IOPS=85, BW=341KiB/s (349kB/s)(6236KiB/18279msec) clat (msec): min=2, max=168, avg= 6.90, stdev= 4.94 lat (msec): min=2, max=168, avg= 6.90, stdev= 4.94 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 12], | 99.00th=[ 17], 99.50th=[ 20], 99.90th=[ 22], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 160, max= 520, per=24.96%, avg=341.25, stdev=59.83, samples=36 iops : min= 40, max= 130, avg=85.25, stdev=14.89, samples=36 lat (msec) : 2=1.63%, 4=26.63%, 10=62.83%, 20=8.66%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=12.81%, ctx=3621, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1306KiB/s (1337kB/s), 325KiB/s-331KiB/s (332kB/s-339kB/s), io=23.4MiB (24.6MB), run=18279-18386msec WRITE: bw=1367KiB/s (1400kB/s), 340KiB/s-347KiB/s (349kB/s-355kB/s), io=24.6MiB (25.7MB), run=18279-18386msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1992: Wed Apr 23 23:40:07 2025 read: IOPS=111, BW=1788KiB/s (1831kB/s)(5920KiB/3311msec) clat (usec): min=56, max=19495, avg=4572.76, stdev=2753.11 lat (usec): min=57, max=19496, avg=4573.40, stdev=2753.13 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 83], 10.00th=[ 1844], 20.00th=[ 2573], | 30.00th=[ 2999], 40.00th=[ 3556], 50.00th=[ 4228], 60.00th=[ 4883], | 70.00th=[ 5669], 80.00th=[ 6652], 90.00th=[ 7635], 95.00th=[ 9372], | 99.00th=[13042], 99.50th=[14091], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 1088, max= 2048, per=24.86%, avg=1738.00, stdev=338.11, samples=6 iops : min= 68, max= 128, avg=108.50, stdev=21.07, samples=6 write: IOPS=120, BW=1923KiB/s (1969kB/s)(6368KiB/3311msec) clat (usec): min=615, max=28847, avg=4010.40, stdev=3675.45 lat (usec): min=616, max=28848, avg=4013.25, stdev=3675.96 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 783], 10.00th=[ 816], 20.00th=[ 889], | 30.00th=[ 1045], 40.00th=[ 1893], 50.00th=[ 3097], 60.00th=[ 4113], | 70.00th=[ 5145], 80.00th=[ 6587], 90.00th=[ 8717], 95.00th=[10683], | 99.00th=[16581], 99.50th=[23200], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 1280, max= 2368, per=25.80%, avg=1897.83, stdev=427.09, samples=6 iops : min= 80, max= 148, avg=118.50, stdev=26.61, samples=6 lat (usec) : 100=3.78%, 250=0.78%, 750=1.56%, 1000=13.54% lat (msec) : 2=6.64%, 4=26.56%, 10=41.54%, 20=5.34%, 50=0.26% cpu : usr=0.03%, sys=24.62%, ctx=1507, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1993: Wed Apr 23 23:40:07 2025 read: IOPS=108, BW=1743KiB/s (1785kB/s)(5744KiB/3295msec) clat (usec): min=59, max=19553, avg=4553.42, stdev=2733.36 lat (usec): min=60, max=19553, avg=4554.00, stdev=2733.37 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 78], 10.00th=[ 110], 20.00th=[ 2704], | 30.00th=[ 3261], 40.00th=[ 3687], 50.00th=[ 4359], 60.00th=[ 4883], | 70.00th=[ 5604], 80.00th=[ 6456], 90.00th=[ 7832], 95.00th=[ 9634], | 99.00th=[11994], 99.50th=[12125], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 1024, max= 2144, per=24.26%, avg=1696.00, stdev=417.72, samples=6 iops : min= 64, max= 134, avg=106.00, stdev=26.11, samples=6 write: IOPS=124, BW=1986KiB/s (2034kB/s)(6544KiB/3295msec) clat (usec): min=645, max=26780, avg=3850.93, stdev=3671.96 lat (usec): min=646, max=26781, avg=3851.88, stdev=3671.90 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 783], 10.00th=[ 816], 20.00th=[ 914], | 30.00th=[ 1106], 40.00th=[ 1680], 50.00th=[ 2769], 60.00th=[ 3523], | 70.00th=[ 4555], 80.00th=[ 6325], 90.00th=[ 8717], 95.00th=[10683], | 99.00th=[16057], 99.50th=[20317], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 1344, max= 2432, per=25.96%, avg=1909.33, stdev=362.42, samples=6 iops : min= 84, max= 152, avg=119.33, stdev=22.65, samples=6 lat (usec) : 100=4.43%, 250=0.91%, 750=1.17%, 1000=12.24% lat (msec) : 2=9.51%, 4=26.56%, 10=39.45%, 20=5.34%, 50=0.39% cpu : usr=0.21%, sys=24.07%, ctx=1497, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1994: Wed Apr 23 23:40:07 2025 read: IOPS=114, BW=1826KiB/s (1870kB/s)(6256KiB/3426msec) clat (usec): min=63, max=20962, avg=4645.29, stdev=2727.62 lat (usec): min=64, max=20963, avg=4645.87, stdev=2727.64 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 86], 10.00th=[ 231], 20.00th=[ 2606], | 30.00th=[ 3130], 40.00th=[ 3916], 50.00th=[ 4555], 60.00th=[ 5276], | 70.00th=[ 5735], 80.00th=[ 6325], 90.00th=[ 8160], 95.00th=[ 9372], | 99.00th=[12518], 99.50th=[14615], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 768, max= 2080, per=24.64%, avg=1722.67, stdev=482.09, samples=6 iops : min= 48, max= 130, avg=107.67, stdev=30.13, samples=6 write: IOPS=110, BW=1761KiB/s (1803kB/s)(6032KiB/3426msec) clat (usec): min=633, max=30065, avg=4059.80, stdev=3723.48 lat (usec): min=633, max=30066, avg=4062.70, stdev=3725.43 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 799], 10.00th=[ 832], 20.00th=[ 906], | 30.00th=[ 1123], 40.00th=[ 2311], 50.00th=[ 3130], 60.00th=[ 4080], | 70.00th=[ 5014], 80.00th=[ 6325], 90.00th=[ 8356], 95.00th=[11338], | 99.00th=[15401], 99.50th=[21890], 99.90th=[30016], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 1088, max= 2272, per=23.35%, avg=1717.33, stdev=468.85, samples=6 iops : min= 68, max= 142, avg=107.33, stdev=29.30, samples=6 lat (usec) : 100=3.52%, 250=1.69%, 750=0.91%, 1000=11.72% lat (msec) : 2=6.90%, 4=25.52%, 10=45.18%, 20=4.17%, 50=0.39% cpu : usr=0.00%, sys=24.43%, ctx=1558, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1995: Wed Apr 23 23:40:07 2025 read: IOPS=116, BW=1865KiB/s (1909kB/s)(6032KiB/3235msec) clat (usec): min=60, max=19908, avg=4483.57, stdev=2813.43 lat (usec): min=60, max=19909, avg=4484.41, stdev=2813.36 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 80], 10.00th=[ 101], 20.00th=[ 2638], | 30.00th=[ 3163], 40.00th=[ 3720], 50.00th=[ 4178], 60.00th=[ 4883], | 70.00th=[ 5538], 80.00th=[ 6194], 90.00th=[ 7767], 95.00th=[ 9241], | 99.00th=[13435], 99.50th=[15401], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 1216, max= 2144, per=25.63%, avg=1792.00, stdev=320.64, samples=6 iops : min= 76, max= 134, avg=112.00, stdev=20.04, samples=6 write: IOPS=120, BW=1934KiB/s (1980kB/s)(6256KiB/3235msec) clat (usec): min=599, max=17494, avg=3702.27, stdev=3100.79 lat (usec): min=600, max=17495, avg=3705.33, stdev=3099.48 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 775], 10.00th=[ 832], 20.00th=[ 955], | 30.00th=[ 1221], 40.00th=[ 2180], 50.00th=[ 2900], 60.00th=[ 3621], | 70.00th=[ 4752], 80.00th=[ 5866], 90.00th=[ 8029], 95.00th=[ 9765], | 99.00th=[14615], 99.50th=[15139], 99.90th=[17433], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 1504, max= 2304, per=25.74%, avg=1893.33, stdev=294.27, samples=6 iops : min= 94, max= 144, avg=118.33, stdev=18.39, samples=6 lat (usec) : 100=4.82%, 250=1.04%, 500=0.13%, 750=1.56%, 1000=10.16% lat (msec) : 2=8.07%, 4=29.82%, 10=40.49%, 20=3.91% cpu : usr=0.22%, sys=24.30%, ctx=1537, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6991KiB/s (7159kB/s), 1743KiB/s-1865KiB/s (1785kB/s-1909kB/s), io=23.4MiB (24.5MB), run=3235-3426msec WRITE: bw=7356KiB/s (7532kB/s), 1761KiB/s-1986KiB/s (1803kB/s-2034kB/s), io=24.6MiB (25.8MB), run=3235-3426msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1988: Wed Apr 23 23:40:08 2025 read: IOPS=79, BW=1265KiB/s (1295kB/s)(5920KiB/4681msec) clat (usec): min=1909, max=21983, avg=4980.50, stdev=3400.44 lat (usec): min=1909, max=21984, avg=4981.20, stdev=3400.45 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2147], 10.00th=[ 2311], 20.00th=[ 2606], | 30.00th=[ 2900], 40.00th=[ 3392], 50.00th=[ 3851], 60.00th=[ 4359], | 70.00th=[ 4883], 80.00th=[ 6587], 90.00th=[ 9503], 95.00th=[12387], | 99.00th=[19006], 99.50th=[21890], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 800, max= 2176, per=24.38%, avg=1247.33, stdev=409.58, samples=9 iops : min= 50, max= 136, avg=77.78, stdev=25.63, samples=9 write: IOPS=85, BW=1360KiB/s (1393kB/s)(6368KiB/4681msec) clat (usec): min=2751, max=28565, avg=7086.62, stdev=3908.55 lat (usec): min=2752, max=28566, avg=7087.81, stdev=3908.55 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3261], 10.00th=[ 3490], 20.00th=[ 3884], | 30.00th=[ 4424], 40.00th=[ 5080], 50.00th=[ 6128], 60.00th=[ 6849], | 70.00th=[ 8160], 80.00th=[ 9634], 90.00th=[12387], 95.00th=[15401], | 99.00th=[20317], 99.50th=[27132], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 896, max= 2304, per=25.29%, avg=1361.11, stdev=425.54, samples=9 iops : min= 56, max= 144, avg=84.89, stdev=26.70, samples=9 lat (msec) : 2=0.39%, 4=37.37%, 10=47.92%, 20=13.41%, 50=0.91% cpu : usr=0.17%, sys=12.14%, ctx=907, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1989: Wed Apr 23 23:40:08 2025 read: IOPS=78, BW=1253KiB/s (1283kB/s)(5744KiB/4585msec) clat (usec): min=1853, max=24642, avg=4779.75, stdev=3366.96 lat (usec): min=1853, max=24643, avg=4780.41, stdev=3366.96 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2147], 10.00th=[ 2245], 20.00th=[ 2507], | 30.00th=[ 2835], 40.00th=[ 3163], 50.00th=[ 3589], 60.00th=[ 4146], | 70.00th=[ 4621], 80.00th=[ 6259], 90.00th=[ 9110], 95.00th=[12518], | 99.00th=[16188], 99.50th=[20055], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 736, max= 2112, per=24.45%, avg=1251.00, stdev=397.16, samples=9 iops : min= 46, max= 132, avg=78.00, stdev=24.93, samples=9 write: IOPS=89, BW=1427KiB/s (1462kB/s)(6544KiB/4585msec) clat (usec): min=2649, max=29559, avg=6939.79, stdev=3774.26 lat (usec): min=2650, max=29560, avg=6941.00, stdev=3774.29 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 3359], 10.00th=[ 3621], 20.00th=[ 3982], | 30.00th=[ 4490], 40.00th=[ 5276], 50.00th=[ 5997], 60.00th=[ 6915], | 70.00th=[ 7635], 80.00th=[ 8979], 90.00th=[11207], 95.00th=[14484], | 99.00th=[19006], 99.50th=[24511], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 960, max= 2528, per=26.54%, avg=1428.78, stdev=462.07, samples=9 iops : min= 60, max= 158, avg=89.11, stdev=29.00, samples=9 lat (msec) : 2=0.65%, 4=36.20%, 10=51.82%, 20=10.55%, 50=0.78% cpu : usr=0.24%, sys=12.19%, ctx=920, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1990: Wed Apr 23 23:40:08 2025 read: IOPS=85, BW=1365KiB/s (1398kB/s)(6256KiB/4584msec) clat (usec): min=1920, max=26321, avg=5191.20, stdev=3764.68 lat (usec): min=1921, max=26322, avg=5191.91, stdev=3764.76 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2147], 10.00th=[ 2311], 20.00th=[ 2606], | 30.00th=[ 2900], 40.00th=[ 3326], 50.00th=[ 3752], 60.00th=[ 4293], | 70.00th=[ 5211], 80.00th=[ 6980], 90.00th=[10814], 95.00th=[13173], | 99.00th=[18744], 99.50th=[21627], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 672, max= 2267, per=26.94%, avg=1378.22, stdev=498.45, samples=9 iops : min= 42, max= 141, avg=85.89, stdev=31.05, samples=9 write: IOPS=82, BW=1316KiB/s (1347kB/s)(6032KiB/4584msec) clat (usec): min=2373, max=22475, avg=6577.61, stdev=3212.35 lat (usec): min=2373, max=22476, avg=6578.80, stdev=3212.38 clat percentiles (usec): | 1.00th=[ 2802], 5.00th=[ 3326], 10.00th=[ 3523], 20.00th=[ 3916], | 30.00th=[ 4424], 40.00th=[ 4948], 50.00th=[ 5800], 60.00th=[ 6783], | 70.00th=[ 7504], 80.00th=[ 8356], 90.00th=[11207], 95.00th=[13304], | 99.00th=[19268], 99.50th=[21103], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 832, max= 2427, per=24.29%, avg=1307.33, stdev=454.21, samples=9 iops : min= 52, max= 151, avg=81.44, stdev=28.28, samples=9 lat (msec) : 2=0.65%, 4=38.41%, 10=48.31%, 20=12.11%, 50=0.52% cpu : usr=0.31%, sys=11.96%, ctx=913, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1991: Wed Apr 23 23:40:08 2025 read: IOPS=81, BW=1308KiB/s (1339kB/s)(6032KiB/4612msec) clat (usec): min=1833, max=20802, avg=5071.75, stdev=3530.64 lat (usec): min=1834, max=20803, avg=5072.46, stdev=3530.63 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2147], 10.00th=[ 2278], 20.00th=[ 2671], | 30.00th=[ 2933], 40.00th=[ 3326], 50.00th=[ 3818], 60.00th=[ 4359], | 70.00th=[ 4948], 80.00th=[ 6587], 90.00th=[10552], 95.00th=[13042], | 99.00th=[18220], 99.50th=[18744], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 768, max= 2176, per=25.63%, avg=1311.33, stdev=397.79, samples=9 iops : min= 48, max= 136, avg=81.78, stdev=24.87, samples=9 write: IOPS=84, BW=1356KiB/s (1389kB/s)(6256KiB/4612msec) clat (usec): min=2710, max=29247, avg=6689.36, stdev=3563.36 lat (usec): min=2711, max=29248, avg=6690.55, stdev=3563.44 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3326], 10.00th=[ 3523], 20.00th=[ 3982], | 30.00th=[ 4359], 40.00th=[ 4883], 50.00th=[ 5866], 60.00th=[ 6718], | 70.00th=[ 7373], 80.00th=[ 8586], 90.00th=[10683], 95.00th=[13435], | 99.00th=[22676], 99.50th=[27657], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 832, max= 2400, per=25.22%, avg=1357.44, stdev=451.19, samples=9 iops : min= 52, max= 150, avg=84.67, stdev=28.23, samples=9 lat (msec) : 2=0.91%, 4=35.03%, 10=51.69%, 20=11.72%, 50=0.65% cpu : usr=0.17%, sys=12.08%, ctx=939, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5117KiB/s (5240kB/s), 1253KiB/s-1365KiB/s (1283kB/s-1398kB/s), io=23.4MiB (24.5MB), run=4584-4681msec WRITE: bw=5383KiB/s (5513kB/s), 1316KiB/s-1427KiB/s (1347kB/s-1462kB/s), io=24.6MiB (25.8MB), run=4584-4681msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2008: Wed Apr 23 23:40:11 2025 read: IOPS=4, BW=4139KiB/s (4238kB/s)(7168KiB/1732msec) clat (usec): min=726, max=439791, avg=83929.09, stdev=157714.81 lat (usec): min=728, max=439793, avg=83930.35, stdev=157715.04 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 725], 10.00th=[ 725], 20.00th=[ 11076], | 30.00th=[ 25297], 40.00th=[ 25297], 50.00th=[ 28443], 60.00th=[ 31327], | 70.00th=[ 31327], 80.00th=[ 50594], 90.00th=[438305], 95.00th=[438305], | 99.00th=[438305], 99.50th=[438305], 99.90th=[438305], 99.95th=[438305], | 99.99th=[438305] bw ( KiB/s): min= 2048, max= 4087, per=26.88%, avg=3178.67, stdev=1037.52, samples=3 iops : min= 2, max= 3, avg= 2.67, stdev= 0.58, samples=3 write: IOPS=2, BW=2956KiB/s (3027kB/s)(5120KiB/1732msec) clat (msec): min=20, max=974, avg=225.64, stdev=419.02 lat (msec): min=20, max=974, avg=225.69, stdev=419.02 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 37], 60.00th=[ 37], | 70.00th=[ 64], 80.00th=[ 64], 90.00th=[ 978], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 1700, max= 1700, per=10.27%, avg=1700.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 20=8.33%, 50=50.00%, 100=16.67%, 500=8.33%, 1000=8.33% cpu : usr=0.00%, sys=14.79%, ctx=682, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2009: Wed Apr 23 23:40:11 2025 read: IOPS=1, BW=1188KiB/s (1216kB/s)(2048KiB/1724msec) clat (usec): min=38616, max=38644, avg=38630.84, stdev=19.85 lat (usec): min=38617, max=38646, avg=38632.00, stdev=20.57 clat percentiles (usec): | 1.00th=[38536], 5.00th=[38536], 10.00th=[38536], 20.00th=[38536], | 30.00th=[38536], 40.00th=[38536], 50.00th=[38536], 60.00th=[38536], | 70.00th=[38536], 80.00th=[38536], 90.00th=[38536], 95.00th=[38536], | 99.00th=[38536], 99.50th=[38536], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 1723, max= 2048, per=15.95%, avg=1885.50, stdev=229.81, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=5, BW=5940KiB/s (6082kB/s)(10.0MiB/1724msec) clat (msec): min=24, max=943, avg=156.83, stdev=296.25 lat (msec): min=24, max=943, avg=156.87, stdev=296.25 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 26], | 30.00th=[ 30], 40.00th=[ 30], 50.00th=[ 33], 60.00th=[ 35], | 70.00th=[ 40], 80.00th=[ 40], 90.00th=[ 372], 95.00th=[ 944], | 99.00th=[ 944], 99.50th=[ 944], 99.90th=[ 944], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 2048, max= 4096, per=19.31%, avg=3197.00, stdev=1046.64, samples=3 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=3 lat (msec) : 50=83.33%, 500=8.33%, 1000=8.33% cpu : usr=0.00%, sys=17.18%, ctx=422, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2010: Wed Apr 23 23:40:11 2025 read: IOPS=4, BW=4129KiB/s (4228kB/s)(6144KiB/1488msec) clat (usec): min=495, max=895123, avg=197714.17, stdev=356807.24 lat (usec): min=496, max=895123, avg=197715.37, stdev=356806.99 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 498], 10.00th=[ 498], 20.00th=[ 914], | 30.00th=[ 914], 40.00th=[ 979], 50.00th=[ 979], 60.00th=[ 25822], | 70.00th=[263193], 80.00th=[263193], 90.00th=[893387], 95.00th=[893387], | 99.00th=[893387], 99.50th=[893387], 99.90th=[893387], 99.95th=[893387], | 99.99th=[893387] bw ( KiB/s): min= 4096, max= 4096, per=34.64%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4129KiB/s (4228kB/s)(6144KiB/1488msec) clat (usec): min=30898, max=53225, avg=41928.42, stdev=8339.18 lat (usec): min=30933, max=53261, avg=41972.19, stdev=8340.22 clat percentiles (usec): | 1.00th=[30802], 5.00th=[30802], 10.00th=[30802], 20.00th=[35914], | 30.00th=[35914], 40.00th=[38011], 50.00th=[38011], 60.00th=[46400], | 70.00th=[46924], 80.00th=[46924], 90.00th=[53216], 95.00th=[53216], | 99.00th=[53216], 99.50th=[53216], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 8192, max= 8192, per=49.49%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 1000=16.67% lat (msec) : 50=50.00%, 100=8.33%, 500=8.33%, 1000=8.33% cpu : usr=0.00%, sys=25.02%, ctx=646, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2011: Wed Apr 23 23:40:11 2025 read: IOPS=4, BW=4745KiB/s (4859kB/s)(5120KiB/1079msec) clat (usec): min=957, max=39217, avg=29333.33, stdev=16073.63 lat (usec): min=958, max=39218, avg=29334.58, stdev=16073.47 clat percentiles (usec): | 1.00th=[ 955], 5.00th=[ 955], 10.00th=[ 955], 20.00th=[ 955], | 30.00th=[32113], 40.00th=[32113], 50.00th=[36439], 60.00th=[36439], | 70.00th=[37487], 80.00th=[37487], 90.00th=[39060], 95.00th=[39060], | 99.00th=[39060], 99.50th=[39060], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 8192, max= 8192, per=69.28%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6643KiB/s (6803kB/s)(7168KiB/1079msec) clat (msec): min=28, max=466, avg=123.23, stdev=154.35 lat (msec): min=28, max=466, avg=123.28, stdev=154.36 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 36], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 70], 60.00th=[ 107], | 70.00th=[ 107], 80.00th=[ 107], 90.00th=[ 468], 95.00th=[ 468], | 99.00th=[ 468], 99.50th=[ 468], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 6144, max= 6144, per=37.11%, avg=6144.00, stdev= 0.00, samples=2 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=2 lat (usec) : 1000=8.33% lat (msec) : 50=50.00%, 100=16.67%, 250=16.67%, 500=8.33% cpu : usr=0.00%, sys=31.45%, ctx=983, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.5MiB/s (12.1MB/s), 1188KiB/s-4745KiB/s (1216kB/s-4859kB/s), io=20.0MiB (20.0MB), run=1079-1732msec WRITE: bw=16.2MiB/s (16.0MB/s), 2956KiB/s-6643KiB/s (3027kB/s-6803kB/s), io=28.0MiB (29.4MB), run=1079-1732msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2003: Wed Apr 23 23:40:11 2025 read: IOPS=15, BW=15.1MiB/s (15.9MB/s)(7168KiB/463msec) clat (usec): min=13525, max=79339, avg=36889.95, stdev=22190.24 lat (usec): min=13525, max=79340, avg=36891.26, stdev=22190.46 clat percentiles (usec): | 1.00th=[13566], 5.00th=[13566], 10.00th=[13566], 20.00th=[23725], | 30.00th=[24249], 40.00th=[24249], 50.00th=[26084], 60.00th=[45351], | 70.00th=[45351], 80.00th=[46400], 90.00th=[79168], 95.00th=[79168], | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] write: IOPS=10, BW=10.8MiB/s (11.3MB/s)(5120KiB/463msec) clat (msec): min=18, max=101, avg=37.34, stdev=36.09 lat (msec): min=18, max=101, avg=37.46, stdev=36.11 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 21], 60.00th=[ 21], | 70.00th=[ 27], 80.00th=[ 27], 90.00th=[ 102], 95.00th=[ 102], | 99.00th=[ 102], 99.50th=[ 102], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] lat (msec) : 20=25.00%, 50=58.33%, 100=8.33%, 250=8.33% cpu : usr=0.00%, sys=21.86%, ctx=37, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2004: Wed Apr 23 23:40:11 2025 read: IOPS=1, BW=1176KiB/s (1204kB/s)(2048KiB/1742msec) clat (usec): min=15961, max=33525, avg=24743.41, stdev=12419.47 lat (usec): min=15962, max=33528, avg=24745.19, stdev=12420.82 clat percentiles (usec): | 1.00th=[15926], 5.00th=[15926], 10.00th=[15926], 20.00th=[15926], | 30.00th=[15926], 40.00th=[15926], 50.00th=[15926], 60.00th=[33424], | 70.00th=[33424], 80.00th=[33424], 90.00th=[33424], 95.00th=[33424], | 99.00th=[33424], 99.50th=[33424], 99.90th=[33424], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 1695, max= 2048, per=17.31%, avg=1871.50, stdev=249.61, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=5, BW=5878KiB/s (6019kB/s)(10.0MiB/1742msec) clat (msec): min=23, max=936, avg=168.16, stdev=298.28 lat (msec): min=23, max=936, avg=168.23, stdev=298.30 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 32], 50.00th=[ 33], 60.00th=[ 33], | 70.00th=[ 85], 80.00th=[ 85], 90.00th=[ 439], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 2048, max= 4096, per=20.99%, avg=3178.00, stdev=1040.33, samples=3 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=3 lat (msec) : 20=8.33%, 50=58.33%, 100=16.67%, 500=8.33%, 1000=8.33% cpu : usr=0.00%, sys=9.13%, ctx=60, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2005: Wed Apr 23 23:40:11 2025 read: IOPS=3, BW=3244KiB/s (3322kB/s)(6144KiB/1894msec) clat (msec): min=13, max=996, avg=270.73, stdev=394.13 lat (msec): min=13, max=996, avg=270.73, stdev=394.13 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 26], | 30.00th=[ 26], 40.00th=[ 48], 50.00th=[ 48], 60.00th=[ 79], | 70.00th=[ 464], 80.00th=[ 464], 90.00th=[ 995], 95.00th=[ 995], | 99.00th=[ 995], 99.50th=[ 995], 99.90th=[ 995], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 2048, max= 2048, per=18.94%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3244KiB/s (3322kB/s)(6144KiB/1894msec) clat (usec): min=22343, max=71682, avg=43509.11, stdev=22683.66 lat (usec): min=22395, max=71724, avg=43554.18, stdev=22681.70 clat percentiles (usec): | 1.00th=[22414], 5.00th=[22414], 10.00th=[22414], 20.00th=[24249], | 30.00th=[24249], 40.00th=[29754], 50.00th=[29754], 60.00th=[42206], | 70.00th=[70779], 80.00th=[70779], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 2048, max= 6144, per=27.06%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=8.33%, 50=50.00%, 100=25.00%, 500=8.33%, 1000=8.33% cpu : usr=0.00%, sys=7.03%, ctx=77, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2006: Wed Apr 23 23:40:11 2025 read: IOPS=4, BW=4621KiB/s (4732kB/s)(5120KiB/1108msec) clat (usec): min=18875, max=49769, avg=32446.63, stdev=12653.88 lat (usec): min=18877, max=49771, avg=32448.25, stdev=12653.81 clat percentiles (usec): | 1.00th=[19006], 5.00th=[19006], 10.00th=[19006], 20.00th=[19006], | 30.00th=[25035], 40.00th=[25035], 50.00th=[27657], 60.00th=[27657], | 70.00th=[41157], 80.00th=[41157], 90.00th=[49546], 95.00th=[49546], | 99.00th=[49546], 99.50th=[49546], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 8192, max= 8192, per=75.76%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6469KiB/s (6625kB/s)(7168KiB/1108msec) clat (msec): min=25, max=437, avg=126.98, stdev=140.66 lat (msec): min=25, max=437, avg=127.05, stdev=140.65 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 48], | 30.00th=[ 64], 40.00th=[ 64], 50.00th=[ 96], 60.00th=[ 107], | 70.00th=[ 107], 80.00th=[ 112], 90.00th=[ 439], 95.00th=[ 439], | 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 6131, max= 6144, per=40.54%, avg=6137.50, stdev= 9.19, samples=2 iops : min= 5, max= 6, avg= 5.50, stdev= 0.71, samples=2 lat (msec) : 20=8.33%, 50=50.00%, 100=16.67%, 250=16.67%, 500=8.33% cpu : usr=0.00%, sys=11.20%, ctx=75, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.6MiB/s (11.1MB/s), 1176KiB/s-15.1MiB/s (1204kB/s-15.9MB/s), io=20.0MiB (20.0MB), run=463-1894msec WRITE: bw=14.8MiB/s (15.5MB/s), 3244KiB/s-10.8MiB/s (3322kB/s-11.3MB/s), io=28.0MiB (29.4MB), run=463-1894msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2019: Wed Apr 23 23:40:13 2025 read: IOPS=4, BW=18.7MiB/s (19.6MB/s)(12.0MiB/641msec) clat (msec): min=114, max=368, avg=211.11, stdev=137.57 lat (msec): min=114, max=368, avg=211.11, stdev=137.57 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 115], 10.00th=[ 115], 20.00th=[ 115], | 30.00th=[ 115], 40.00th=[ 150], 50.00th=[ 150], 60.00th=[ 150], | 70.00th=[ 368], 80.00th=[ 368], 90.00th=[ 368], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=13.12%, ctx=64, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2021: Wed Apr 23 23:40:13 2025 read: IOPS=1, BW=5432KiB/s (5563kB/s)(4096KiB/754msec) clat (nsec): min=88558k, max=88558k, avg=88558003.00, stdev= 0.00 lat (nsec): min=88560k, max=88560k, avg=88560016.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[88605], 5.00th=[88605], 10.00th=[88605], 20.00th=[88605], | 30.00th=[88605], 40.00th=[88605], 50.00th=[88605], 60.00th=[88605], | 70.00th=[88605], 80.00th=[88605], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 8192, max= 8192, per=37.70%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.6MiB/s (11.1MB/s)(8192KiB/754msec) clat (msec): min=280, max=330, avg=305.34, stdev=35.17 lat (msec): min=280, max=330, avg=305.76, stdev=35.08 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 279], 10.00th=[ 279], 20.00th=[ 279], | 30.00th=[ 279], 40.00th=[ 279], 50.00th=[ 279], 60.00th=[ 330], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 8192, max= 8192, per=18.85%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.00%, sys=19.92%, ctx=127, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2023: Wed Apr 23 23:40:13 2025 write: IOPS=4, BW=17.4MiB/s (18.2MB/s)(12.0MiB/690msec) clat (msec): min=166, max=316, avg=225.58, stdev=79.79 lat (msec): min=166, max=316, avg=225.81, stdev=79.85 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 167], | 30.00th=[ 167], 40.00th=[ 194], 50.00th=[ 194], 60.00th=[ 194], | 70.00th=[ 317], 80.00th=[ 317], 90.00th=[ 317], 95.00th=[ 317], | 99.00th=[ 317], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=12.34%, ctx=148, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2025: Wed Apr 23 23:40:13 2025 write: IOPS=6, BW=24.7MiB/s (25.9MB/s)(12.0MiB/485msec) clat (msec): min=76, max=304, avg=157.22, stdev=127.91 lat (msec): min=76, max=304, avg=157.50, stdev=127.92 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 77], 10.00th=[ 77], 20.00th=[ 77], | 30.00th=[ 77], 40.00th=[ 91], 50.00th=[ 91], 60.00th=[ 91], | 70.00th=[ 305], 80.00th=[ 305], 90.00th=[ 305], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] lat (msec) : 100=66.67%, 500=33.33% cpu : usr=0.21%, sys=18.60%, ctx=142, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=21.2MiB/s (22.2MB/s), 5432KiB/s-18.7MiB/s (5563kB/s-19.6MB/s), io=16.0MiB (16.8MB), run=641-754msec WRITE: bw=42.4MiB/s (44.5MB/s), 10.6MiB/s-24.7MiB/s (11.1MB/s-25.9MB/s), io=32.0MiB (33.6MB), run=485-754msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2020: Wed Apr 23 23:40:13 2025 read: IOPS=10, BW=40.4MiB/s (42.4MB/s)(12.0MiB/297msec) clat (msec): min=55, max=139, avg=96.16, stdev=42.27 lat (msec): min=55, max=139, avg=96.16, stdev=42.27 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 56], 10.00th=[ 56], 20.00th=[ 56], | 30.00th=[ 56], 40.00th=[ 94], 50.00th=[ 94], 60.00th=[ 94], | 70.00th=[ 140], 80.00th=[ 140], 90.00th=[ 140], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] lat (msec) : 100=66.67%, 250=33.33% cpu : usr=0.00%, sys=43.92%, ctx=1524, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2022: Wed Apr 23 23:40:13 2025 read: IOPS=1, BW=5076KiB/s (5197kB/s)(4096KiB/807msec) clat (nsec): min=129381k, max=129381k, avg=129380635.00, stdev= 0.00 lat (nsec): min=129382k, max=129382k, avg=129381906.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 130], 10.00th=[ 130], 20.00th=[ 130], | 30.00th=[ 130], 40.00th=[ 130], 50.00th=[ 130], 60.00th=[ 130], | 70.00th=[ 130], 80.00th=[ 130], 90.00th=[ 130], 95.00th=[ 130], | 99.00th=[ 130], 99.50th=[ 130], 99.90th=[ 130], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 8192, max= 8192, per=40.35%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9.91MiB/s (10.4MB/s)(8192KiB/807msec) clat (msec): min=265, max=394, avg=330.24, stdev=91.33 lat (msec): min=265, max=395, avg=330.48, stdev=91.32 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 266], 10.00th=[ 266], 20.00th=[ 266], | 30.00th=[ 266], 40.00th=[ 266], 50.00th=[ 266], 60.00th=[ 397], | 70.00th=[ 397], 80.00th=[ 397], 90.00th=[ 397], 95.00th=[ 397], | 99.00th=[ 397], 99.50th=[ 397], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=30.77%, ctx=630, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2024: Wed Apr 23 23:40:13 2025 write: IOPS=3, BW=15.2MiB/s (15.0MB/s)(12.0MiB/787msec) clat (msec): min=229, max=300, avg=255.84, stdev=38.46 lat (msec): min=230, max=300, avg=256.28, stdev=38.83 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 230], 10.00th=[ 230], 20.00th=[ 230], | 30.00th=[ 230], 40.00th=[ 239], 50.00th=[ 239], 60.00th=[ 239], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 8192, max= 8192, per=20.18%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.13%, sys=36.13%, ctx=40, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2026: Wed Apr 23 23:40:13 2025 write: IOPS=5, BW=20.3MiB/s (21.3MB/s)(12.0MiB/592msec) clat (msec): min=93, max=253, avg=187.58, stdev=83.24 lat (msec): min=94, max=253, avg=187.83, stdev=83.26 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 94], 10.00th=[ 94], 20.00th=[ 94], | 30.00th=[ 94], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 215], | 70.00th=[ 253], 80.00th=[ 253], 90.00th=[ 253], 95.00th=[ 253], | 99.00th=[ 253], 99.50th=[ 253], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min=16384, max=16384, per=40.35%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.17%, sys=49.58%, ctx=39, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.8MiB/s (20.8MB/s), 5076KiB/s-40.4MiB/s (5197kB/s-42.4MB/s), io=16.0MiB (16.8MB), run=297-807msec WRITE: bw=39.7MiB/s (41.6MB/s), 9.91MiB/s-20.3MiB/s (10.4MB/s-21.3MB/s), io=32.0MiB (33.6MB), run=592-807msec