== sanity test 398b: DIO and buffer IO race ============== 15:29:22 (1744054162) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.13259 s, 44.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10241: Mon Apr 7 15:29:37 2025 read: IOPS=130, BW=521KiB/s (534kB/s)(5992KiB/11498msec) clat (usec): min=52, max=162714, avg=3509.19, stdev=4994.40 lat (usec): min=53, max=162715, avg=3510.00, stdev=4994.46 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 69], 10.00th=[ 80], 20.00th=[ 1745], | 30.00th=[ 2147], 40.00th=[ 2474], 50.00th=[ 2802], 60.00th=[ 3261], | 70.00th=[ 4015], 80.00th=[ 5407], 90.00th=[ 7111], 95.00th=[ 8717], | 99.00th=[ 11731], 99.50th=[ 14877], 99.90th=[ 34341], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 280, max= 856, per=24.86%, avg=515.91, stdev=109.16, samples=22 iops : min= 70, max= 214, avg=128.91, stdev=27.34, samples=22 write: IOPS=136, BW=548KiB/s (561kB/s)(6296KiB/11498msec) clat (usec): min=453, max=32831, avg=3894.16, stdev=3736.00 lat (usec): min=453, max=32832, avg=3897.11, stdev=3741.89 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 570], 20.00th=[ 644], | 30.00th=[ 832], 40.00th=[ 1844], 50.00th=[ 2966], 60.00th=[ 3949], | 70.00th=[ 4948], 80.00th=[ 6587], 90.00th=[ 8848], 95.00th=[11469], | 99.00th=[15664], 99.50th=[18220], 99.90th=[24511], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 352, max= 864, per=24.87%, avg=540.18, stdev=103.59, samples=22 iops : min= 88, max= 216, avg=135.00, stdev=25.90, samples=22 lat (usec) : 100=7.52%, 250=1.43%, 500=0.68%, 750=13.18%, 1000=3.29% lat (msec) : 2=7.26%, 4=31.58%, 10=30.37%, 20=4.39%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=23.57%, ctx=5876, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10242: Mon Apr 7 15:29:37 2025 read: IOPS=128, BW=513KiB/s (526kB/s)(5940KiB/11571msec) clat (usec): min=51, max=37963, avg=3398.19, stdev=2956.52 lat (usec): min=51, max=37963, avg=3398.88, stdev=2956.52 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 70], 10.00th=[ 81], 20.00th=[ 1762], | 30.00th=[ 2114], 40.00th=[ 2442], 50.00th=[ 2802], 60.00th=[ 3195], | 70.00th=[ 4047], 80.00th=[ 5211], 90.00th=[ 6980], 95.00th=[ 8291], | 99.00th=[12518], 99.50th=[16319], 99.90th=[27919], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 256, max= 688, per=24.72%, avg=512.91, stdev=99.57, samples=23 iops : min= 64, max= 172, avg=128.17, stdev=24.90, samples=23 write: IOPS=137, BW=549KiB/s (562kB/s)(6348KiB/11571msec) clat (usec): min=460, max=177209, avg=4040.75, stdev=5773.98 lat (usec): min=461, max=177210, avg=4041.54, stdev=5773.92 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 635], | 30.00th=[ 848], 40.00th=[ 1762], 50.00th=[ 3097], 60.00th=[ 4047], | 70.00th=[ 5145], 80.00th=[ 6521], 90.00th=[ 9110], 95.00th=[ 11338], | 99.00th=[ 16712], 99.50th=[ 19792], 99.90th=[ 34866], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 312, max= 752, per=25.15%, avg=546.30, stdev=110.09, samples=23 iops : min= 78, max= 188, avg=136.52, stdev=27.55, samples=23 lat (usec) : 100=7.06%, 250=1.50%, 500=0.91%, 750=13.12%, 1000=3.65% lat (msec) : 2=7.85%, 4=30.27%, 10=30.96%, 20=4.33%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=22.96%, ctx=5768, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10243: Mon Apr 7 15:29:37 2025 read: IOPS=130, BW=523KiB/s (536kB/s)(6028KiB/11515msec) clat (usec): min=54, max=180511, avg=3500.17, stdev=5281.89 lat (usec): min=54, max=180511, avg=3501.52, stdev=5282.26 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 69], 10.00th=[ 80], 20.00th=[ 1795], | 30.00th=[ 2147], 40.00th=[ 2442], 50.00th=[ 2802], 60.00th=[ 3261], | 70.00th=[ 4146], 80.00th=[ 5342], 90.00th=[ 6915], 95.00th=[ 8225], | 99.00th=[ 11600], 99.50th=[ 13829], 99.90th=[ 23725], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 280, max= 758, per=25.14%, avg=521.70, stdev=105.23, samples=23 iops : min= 70, max= 189, avg=130.30, stdev=26.23, samples=23 write: IOPS=135, BW=544KiB/s (557kB/s)(6260KiB/11515msec) clat (usec): min=438, max=38587, avg=3915.28, stdev=3942.13 lat (usec): min=439, max=38588, avg=3916.49, stdev=3942.09 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 627], | 30.00th=[ 807], 40.00th=[ 1614], 50.00th=[ 2999], 60.00th=[ 3916], | 70.00th=[ 4883], 80.00th=[ 6521], 90.00th=[ 9241], 95.00th=[11469], | 99.00th=[15926], 99.50th=[20317], 99.90th=[32113], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 336, max= 712, per=24.93%, avg=541.48, stdev=93.07, samples=23 iops : min= 84, max= 178, avg=135.26, stdev=23.20, samples=23 lat (usec) : 100=7.16%, 250=1.40%, 500=1.20%, 750=13.25%, 1000=2.93% lat (msec) : 2=8.17%, 4=30.92%, 10=29.82%, 20=4.79%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.12%, sys=23.20%, ctx=5717, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10244: Mon Apr 7 15:29:37 2025 read: IOPS=130, BW=523KiB/s (536kB/s)(6052KiB/11563msec) clat (usec): min=53, max=30994, avg=3302.24, stdev=2776.49 lat (usec): min=53, max=30994, avg=3303.12, stdev=2776.42 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 69], 10.00th=[ 78], 20.00th=[ 135], | 30.00th=[ 2114], 40.00th=[ 2409], 50.00th=[ 2769], 60.00th=[ 3294], | 70.00th=[ 4047], 80.00th=[ 5276], 90.00th=[ 6980], 95.00th=[ 8356], | 99.00th=[10945], 99.50th=[12518], 99.90th=[26870], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 192, max= 784, per=25.17%, avg=522.22, stdev=120.83, samples=23 iops : min= 48, max= 196, avg=130.48, stdev=30.25, samples=23 write: IOPS=134, BW=539KiB/s (552kB/s)(6236KiB/11563msec) clat (usec): min=445, max=212312, avg=4138.02, stdev=6890.39 lat (usec): min=446, max=212314, avg=4138.95, stdev=6890.45 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 635], | 30.00th=[ 848], 40.00th=[ 1565], 50.00th=[ 2999], 60.00th=[ 3949], | 70.00th=[ 5014], 80.00th=[ 6456], 90.00th=[ 9110], 95.00th=[ 11207], | 99.00th=[ 20579], 99.50th=[ 28705], 99.90th=[ 49546], 99.95th=[212861], | 99.99th=[212861] bw ( KiB/s): min= 200, max= 832, per=24.76%, avg=537.87, stdev=126.67, samples=23 iops : min= 50, max= 208, avg=134.39, stdev=31.73, samples=23 lat (usec) : 100=8.40%, 250=1.82%, 500=1.11%, 750=13.15%, 1000=2.99% lat (msec) : 2=7.81%, 4=29.72%, 10=30.34%, 20=4.04%, 50=0.59% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=22.89%, ctx=5705, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2075KiB/s (2125kB/s), 513KiB/s-523KiB/s (526kB/s-536kB/s), io=23.4MiB (24.6MB), run=11498-11571msec WRITE: bw=2173KiB/s (2225kB/s), 539KiB/s-549KiB/s (552kB/s-562kB/s), io=24.6MiB (25.7MB), run=11498-11571msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10246: Mon Apr 7 15:29:57 2025 read: IOPS=46, BW=186KiB/s (190kB/s)(5992KiB/32264msec) clat (usec): min=1615, max=165134, avg=3521.97, stdev=4993.29 lat (usec): min=1616, max=165135, avg=3522.77, stdev=4993.33 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1811], 10.00th=[ 1860], 20.00th=[ 1942], | 30.00th=[ 2008], 40.00th=[ 2114], 50.00th=[ 2311], 60.00th=[ 2671], | 70.00th=[ 3294], 80.00th=[ 3916], 90.00th=[ 6587], 95.00th=[ 9372], | 99.00th=[ 13698], 99.50th=[ 19006], 99.90th=[ 29230], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 72, max= 368, per=25.11%, avg=185.53, stdev=66.72, samples=64 iops : min= 18, max= 92, avg=46.31, stdev=16.71, samples=64 write: IOPS=48, BW=195KiB/s (200kB/s)(6296KiB/32264msec) clat (usec): min=8300, max=52813, avg=17097.21, stdev=5902.74 lat (usec): min=8301, max=52813, avg=17098.01, stdev=5902.75 clat percentiles (usec): | 1.00th=[ 8979], 5.00th=[10159], 10.00th=[10814], 20.00th=[11994], | 30.00th=[13566], 40.00th=[14877], 50.00th=[16319], 60.00th=[17433], | 70.00th=[18482], 80.00th=[20579], 90.00th=[25297], 95.00th=[28705], | 99.00th=[35914], 99.50th=[40109], 99.90th=[51643], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 80, max= 272, per=25.18%, avg=194.91, stdev=52.73, samples=64 iops : min= 20, max= 68, avg=48.66, stdev=13.19, samples=64 lat (msec) : 2=13.77%, 4=25.52%, 10=9.67%, 20=39.45%, 50=11.46% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.09%, sys=6.10%, ctx=3457, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10247: Mon Apr 7 15:29:57 2025 read: IOPS=45, BW=183KiB/s (187kB/s)(5940KiB/32479msec) clat (usec): min=1581, max=179236, avg=3664.75, stdev=5496.56 lat (usec): min=1582, max=179237, avg=3665.47, stdev=5496.56 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1795], 10.00th=[ 1860], 20.00th=[ 1926], | 30.00th=[ 2008], 40.00th=[ 2114], 50.00th=[ 2278], 60.00th=[ 2737], | 70.00th=[ 3326], 80.00th=[ 3949], 90.00th=[ 7177], 95.00th=[ 10159], | 99.00th=[ 16909], 99.50th=[ 21103], 99.90th=[ 30540], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 40, max= 344, per=24.77%, avg=183.03, stdev=75.19, samples=64 iops : min= 10, max= 86, avg=45.69, stdev=18.83, samples=64 write: IOPS=48, BW=195KiB/s (200kB/s)(6348KiB/32479msec) clat (usec): min=8056, max=57468, avg=16987.62, stdev=5953.99 lat (usec): min=8057, max=57469, avg=16988.44, stdev=5953.98 clat percentiles (usec): | 1.00th=[ 8717], 5.00th=[10159], 10.00th=[10945], 20.00th=[11994], | 30.00th=[13435], 40.00th=[14746], 50.00th=[16057], 60.00th=[17171], | 70.00th=[18220], 80.00th=[20317], 90.00th=[24773], 95.00th=[28967], | 99.00th=[37487], 99.50th=[40109], 99.90th=[56361], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 88, max= 280, per=25.17%, avg=194.78, stdev=54.43, samples=64 iops : min= 22, max= 70, avg=48.62, stdev=13.62, samples=64 lat (msec) : 2=14.42%, 4=24.25%, 10=9.41%, 20=40.76%, 50=11.07% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.11%, sys=6.07%, ctx=3458, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10248: Mon Apr 7 15:29:57 2025 read: IOPS=46, BW=186KiB/s (191kB/s)(6028KiB/32399msec) clat (usec): min=1590, max=39495, avg=3560.02, stdev=2923.78 lat (usec): min=1590, max=39496, avg=3560.73, stdev=2923.79 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1811], 10.00th=[ 1860], 20.00th=[ 1942], | 30.00th=[ 2024], 40.00th=[ 2147], 50.00th=[ 2376], 60.00th=[ 2868], | 70.00th=[ 3359], 80.00th=[ 3982], 90.00th=[ 7111], 95.00th=[ 9765], | 99.00th=[15270], 99.50th=[17433], 99.90th=[25297], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 64, max= 352, per=25.18%, avg=186.08, stdev=63.94, samples=64 iops : min= 16, max= 88, avg=46.48, stdev=15.99, samples=64 write: IOPS=48, BW=193KiB/s (198kB/s)(6260KiB/32399msec) clat (msec): min=8, max=193, avg=17.22, stdev= 7.48 lat (msec): min=8, max=193, avg=17.23, stdev= 7.48 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 16], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 26], 95.00th=[ 30], | 99.00th=[ 38], 99.50th=[ 43], 99.90th=[ 52], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 88, max= 272, per=24.88%, avg=192.58, stdev=55.73, samples=64 iops : min= 22, max= 68, avg=48.11, stdev=13.94, samples=64 lat (msec) : 2=12.92%, 4=26.33%, 10=9.64%, 20=39.49%, 50=11.52% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.08%, sys=6.06%, ctx=3442, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10249: Mon Apr 7 15:29:57 2025 read: IOPS=47, BW=188KiB/s (193kB/s)(6052KiB/32132msec) clat (usec): min=1526, max=39401, avg=3469.73, stdev=3097.63 lat (usec): min=1526, max=39402, avg=3470.48, stdev=3097.63 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1811], 10.00th=[ 1844], 20.00th=[ 1942], | 30.00th=[ 2024], 40.00th=[ 2114], 50.00th=[ 2278], 60.00th=[ 2606], | 70.00th=[ 3195], 80.00th=[ 3720], 90.00th=[ 7046], 95.00th=[ 9241], | 99.00th=[15795], 99.50th=[22414], 99.90th=[29492], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 56, max= 359, per=25.50%, avg=188.42, stdev=69.86, samples=64 iops : min= 14, max= 89, avg=47.05, stdev=17.45, samples=64 write: IOPS=48, BW=194KiB/s (199kB/s)(6236KiB/32132msec) clat (msec): min=7, max=208, avg=17.20, stdev= 7.59 lat (msec): min=7, max=208, avg=17.20, stdev= 7.59 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 22], 90.00th=[ 26], 95.00th=[ 29], | 99.00th=[ 36], 99.50th=[ 40], 99.90th=[ 47], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 72, max= 280, per=25.04%, avg=193.80, stdev=54.57, samples=64 iops : min= 18, max= 70, avg=48.39, stdev=13.67, samples=64 lat (msec) : 2=13.64%, 4=26.43%, 10=9.31%, 20=38.28%, 50=12.30% lat (msec) : 250=0.03% cpu : usr=0.12%, sys=6.10%, ctx=3425, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=739KiB/s (757kB/s), 183KiB/s-188KiB/s (187kB/s-193kB/s), io=23.4MiB (24.6MB), run=32132-32479msec WRITE: bw=774KiB/s (793kB/s), 193KiB/s-195KiB/s (198kB/s-200kB/s), io=24.6MiB (25.7MB), run=32132-32479msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10284: Mon Apr 7 15:30:01 2025 read: IOPS=155, BW=2481KiB/s (2541kB/s)(5920KiB/2386msec) clat (usec): min=59, max=16576, avg=3107.62, stdev=2313.26 lat (usec): min=60, max=16577, avg=3108.20, stdev=2313.29 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 73], 10.00th=[ 83], 20.00th=[ 103], | 30.00th=[ 2212], 40.00th=[ 2573], 50.00th=[ 2999], 60.00th=[ 3523], | 70.00th=[ 4047], 80.00th=[ 4752], 90.00th=[ 5735], 95.00th=[ 6718], | 99.00th=[ 9765], 99.50th=[13042], 99.90th=[16581], 99.95th=[16581], | 99.99th=[16581] bw ( KiB/s): min= 1888, max= 2810, per=24.56%, avg=2422.50, stdev=407.37, samples=4 iops : min= 118, max= 175, avg=151.25, stdev=25.26, samples=4 write: IOPS=166, BW=2669KiB/s (2733kB/s)(6368KiB/2386msec) clat (usec): min=657, max=31266, avg=3061.19, stdev=3224.62 lat (usec): min=658, max=31267, avg=3066.27, stdev=3226.65 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 734], 10.00th=[ 799], 20.00th=[ 889], | 30.00th=[ 979], 40.00th=[ 1156], 50.00th=[ 1549], 60.00th=[ 2638], | 70.00th=[ 3523], 80.00th=[ 5211], 90.00th=[ 7177], 95.00th=[ 8979], | 99.00th=[13960], 99.50th=[21365], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 2016, max= 3257, per=25.73%, avg=2670.25, stdev=590.32, samples=4 iops : min= 126, max= 203, avg=166.75, stdev=36.71, samples=4 lat (usec) : 100=9.37%, 250=1.04%, 500=0.52%, 750=3.12%, 1000=13.93% lat (msec) : 2=11.98%, 4=31.12%, 10=26.95%, 20=1.69%, 50=0.26% cpu : usr=0.08%, sys=28.68%, ctx=1810, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10285: Mon Apr 7 15:30:01 2025 read: IOPS=151, BW=2424KiB/s (2482kB/s)(5744KiB/2370msec) clat (usec): min=61, max=28597, avg=3236.75, stdev=3007.25 lat (usec): min=61, max=28597, avg=3237.82, stdev=3007.44 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 72], 10.00th=[ 80], 20.00th=[ 122], | 30.00th=[ 2278], 40.00th=[ 2540], 50.00th=[ 2868], 60.00th=[ 3294], | 70.00th=[ 3982], 80.00th=[ 4883], 90.00th=[ 6128], 95.00th=[ 7373], | 99.00th=[17171], 99.50th=[20841], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 1696, max= 2944, per=24.41%, avg=2408.00, stdev=529.94, samples=4 iops : min= 106, max= 184, avg=150.50, stdev=33.12, samples=4 write: IOPS=172, BW=2761KiB/s (2827kB/s)(6544KiB/2370msec) clat (usec): min=673, max=14283, avg=2806.95, stdev=2515.94 lat (usec): min=674, max=14284, avg=2809.30, stdev=2515.02 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 750], 10.00th=[ 791], 20.00th=[ 865], | 30.00th=[ 971], 40.00th=[ 1139], 50.00th=[ 1516], 60.00th=[ 2606], | 70.00th=[ 3425], 80.00th=[ 4948], 90.00th=[ 6325], 95.00th=[ 7832], | 99.00th=[11207], 99.50th=[13960], 99.90th=[14222], 99.95th=[14222], | 99.99th=[14222] bw ( KiB/s): min= 1920, max= 3200, per=26.21%, avg=2720.00, stdev=592.36, samples=4 iops : min= 120, max= 200, avg=170.00, stdev=37.02, samples=4 lat (usec) : 100=8.59%, 250=1.95%, 500=0.13%, 750=2.86%, 1000=14.58% lat (msec) : 2=12.24%, 4=31.90%, 10=26.17%, 20=1.30%, 50=0.26% cpu : usr=0.25%, sys=28.87%, ctx=1801, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10286: Mon Apr 7 15:30:01 2025 read: IOPS=161, BW=2577KiB/s (2638kB/s)(6256KiB/2428msec) clat (usec): min=56, max=28615, avg=3195.25, stdev=2931.87 lat (usec): min=56, max=28615, avg=3196.05, stdev=2931.98 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 72], 10.00th=[ 78], 20.00th=[ 120], | 30.00th=[ 2212], 40.00th=[ 2442], 50.00th=[ 2769], 60.00th=[ 3392], | 70.00th=[ 4178], 80.00th=[ 4883], 90.00th=[ 5932], 95.00th=[ 7242], | 99.00th=[12387], 99.50th=[27919], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 1728, max= 3001, per=25.12%, avg=2478.25, stdev=536.04, samples=4 iops : min= 108, max= 187, avg=154.75, stdev=33.32, samples=4 write: IOPS=155, BW=2484KiB/s (2544kB/s)(6032KiB/2428msec) clat (usec): min=669, max=21449, avg=2888.44, stdev=2681.93 lat (usec): min=670, max=21450, avg=2889.33, stdev=2681.96 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 750], 10.00th=[ 791], 20.00th=[ 881], | 30.00th=[ 1012], 40.00th=[ 1205], 50.00th=[ 1909], 60.00th=[ 2606], | 70.00th=[ 3294], 80.00th=[ 4883], 90.00th=[ 6521], 95.00th=[ 8094], | 99.00th=[11994], 99.50th=[14091], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 2048, max= 2874, per=23.81%, avg=2470.50, stdev=403.93, samples=4 iops : min= 128, max= 179, avg=154.25, stdev=25.04, samples=4 lat (usec) : 100=10.03%, 250=1.95%, 500=0.13%, 750=2.60%, 1000=11.72% lat (msec) : 2=11.07%, 4=33.59%, 10=27.21%, 20=1.30%, 50=0.39% cpu : usr=0.25%, sys=27.98%, ctx=1814, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10287: Mon Apr 7 15:30:01 2025 read: IOPS=159, BW=2557KiB/s (2618kB/s)(6032KiB/2359msec) clat (usec): min=62, max=29419, avg=3039.43, stdev=2770.85 lat (usec): min=62, max=29420, avg=3040.11, stdev=2770.75 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 72], 10.00th=[ 78], 20.00th=[ 116], | 30.00th=[ 2089], 40.00th=[ 2409], 50.00th=[ 2769], 60.00th=[ 3326], | 70.00th=[ 3916], 80.00th=[ 4555], 90.00th=[ 5538], 95.00th=[ 6915], | 99.00th=[13435], 99.50th=[15401], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 1760, max= 3040, per=25.22%, avg=2488.00, stdev=540.14, samples=4 iops : min= 110, max= 190, avg=155.50, stdev=33.76, samples=4 write: IOPS=165, BW=2652KiB/s (2716kB/s)(6256KiB/2359msec) clat (usec): min=659, max=33771, avg=2821.27, stdev=3242.01 lat (usec): min=660, max=33773, avg=2822.38, stdev=3242.07 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 775], 10.00th=[ 832], 20.00th=[ 914], | 30.00th=[ 988], 40.00th=[ 1106], 50.00th=[ 1385], 60.00th=[ 2245], | 70.00th=[ 3097], 80.00th=[ 4752], 90.00th=[ 6194], 95.00th=[ 7504], | 99.00th=[17171], 99.50th=[28705], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 1888, max= 2976, per=25.28%, avg=2624.00, stdev=499.17, samples=4 iops : min= 118, max= 186, avg=164.00, stdev=31.20, samples=4 lat (usec) : 100=9.24%, 250=2.34%, 500=0.26%, 750=1.95%, 1000=14.06% lat (msec) : 2=14.71%, 4=30.99%, 10=24.22%, 20=1.82%, 50=0.39% cpu : usr=0.13%, sys=28.20%, ctx=1598, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9865KiB/s (10.1MB/s), 2424KiB/s-2577KiB/s (2482kB/s-2638kB/s), io=23.4MiB (24.5MB), run=2359-2428msec WRITE: bw=10.1MiB/s (10.6MB/s), 2484KiB/s-2761KiB/s (2544kB/s-2827kB/s), io=24.6MiB (25.8MB), run=2359-2428msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10288: Mon Apr 7 15:30:08 2025 read: IOPS=40, BW=647KiB/s (663kB/s)(5920KiB/9145msec) clat (usec): min=2040, max=23944, avg=4484.43, stdev=2794.78 lat (usec): min=2040, max=23945, avg=4485.50, stdev=2794.72 clat percentiles (usec): | 1.00th=[ 2245], 5.00th=[ 2573], 10.00th=[ 2933], 20.00th=[ 3228], | 30.00th=[ 3392], 40.00th=[ 3523], 50.00th=[ 3654], 60.00th=[ 3785], | 70.00th=[ 3949], 80.00th=[ 4424], 90.00th=[ 6849], 95.00th=[11338], | 99.00th=[17171], 99.50th=[20317], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 384, max= 1088, per=24.70%, avg=641.67, stdev=176.80, samples=18 iops : min= 24, max= 68, avg=40.06, stdev=11.05, samples=18 write: IOPS=43, BW=696KiB/s (713kB/s)(6368KiB/9145msec) clat (usec): min=10245, max=38259, avg=18445.59, stdev=4549.87 lat (usec): min=10247, max=38261, avg=18447.29, stdev=4549.81 clat percentiles (usec): | 1.00th=[11076], 5.00th=[12911], 10.00th=[13435], 20.00th=[14222], | 30.00th=[15139], 40.00th=[16712], 50.00th=[17957], 60.00th=[19530], | 70.00th=[20579], 80.00th=[21890], 90.00th=[23725], 95.00th=[26084], | 99.00th=[34341], 99.50th=[37487], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 352, max= 832, per=25.42%, avg=695.00, stdev=115.50, samples=18 iops : min= 22, max= 52, avg=43.39, stdev= 7.20, samples=18 lat (msec) : 4=34.24%, 10=10.94%, 20=36.20%, 50=18.62% cpu : usr=0.14%, sys=9.18%, ctx=892, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10289: Mon Apr 7 15:30:08 2025 read: IOPS=38, BW=623KiB/s (638kB/s)(5744KiB/9216msec) clat (usec): min=2054, max=25801, avg=4377.42, stdev=2657.24 lat (usec): min=2055, max=25801, avg=4378.46, stdev=2657.18 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2573], 10.00th=[ 2933], 20.00th=[ 3163], | 30.00th=[ 3359], 40.00th=[ 3490], 50.00th=[ 3589], 60.00th=[ 3752], | 70.00th=[ 4015], 80.00th=[ 4424], 90.00th=[ 6390], 95.00th=[10814], | 99.00th=[16057], 99.50th=[17171], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 256, max= 928, per=24.08%, avg=625.67, stdev=188.60, samples=18 iops : min= 16, max= 58, avg=39.06, stdev=11.79, samples=18 write: IOPS=44, BW=710KiB/s (727kB/s)(6544KiB/9216msec) clat (usec): min=10352, max=36700, avg=18425.61, stdev=4679.05 lat (usec): min=10353, max=36702, avg=18427.26, stdev=4678.96 clat percentiles (usec): | 1.00th=[11469], 5.00th=[12780], 10.00th=[13304], 20.00th=[14222], | 30.00th=[15008], 40.00th=[16319], 50.00th=[17695], 60.00th=[19268], | 70.00th=[20317], 80.00th=[21890], 90.00th=[24511], 95.00th=[27657], | 99.00th=[32113], 99.50th=[35390], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 448, max= 832, per=25.81%, avg=705.67, stdev=104.94, samples=18 iops : min= 28, max= 52, avg=44.06, stdev= 6.55, samples=18 lat (msec) : 4=32.55%, 10=11.59%, 20=37.76%, 50=18.10% cpu : usr=0.16%, sys=9.19%, ctx=932, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10290: Mon Apr 7 15:30:08 2025 read: IOPS=43, BW=702KiB/s (719kB/s)(6256KiB/8914msec) clat (usec): min=1971, max=25147, avg=4426.90, stdev=3002.09 lat (usec): min=1972, max=25148, avg=4427.99, stdev=3002.05 clat percentiles (usec): | 1.00th=[ 2147], 5.00th=[ 2638], 10.00th=[ 2900], 20.00th=[ 3163], | 30.00th=[ 3294], 40.00th=[ 3458], 50.00th=[ 3589], 60.00th=[ 3720], | 70.00th=[ 3916], 80.00th=[ 4293], 90.00th=[ 6390], 95.00th=[10814], | 99.00th=[21627], 99.50th=[23462], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 384, max= 1024, per=27.46%, avg=713.41, stdev=202.63, samples=17 iops : min= 24, max= 64, avg=44.59, stdev=12.66, samples=17 write: IOPS=42, BW=677KiB/s (693kB/s)(6032KiB/8914msec) clat (usec): min=10066, max=41782, avg=18711.18, stdev=4697.63 lat (usec): min=10067, max=41783, avg=18713.55, stdev=4697.36 clat percentiles (usec): | 1.00th=[10945], 5.00th=[13173], 10.00th=[13566], 20.00th=[14484], | 30.00th=[15270], 40.00th=[16712], 50.00th=[18220], 60.00th=[19792], | 70.00th=[20841], 80.00th=[22414], 90.00th=[23987], 95.00th=[26870], | 99.00th=[34866], 99.50th=[41681], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 416, max= 832, per=24.51%, avg=670.12, stdev=97.63, samples=17 iops : min= 26, max= 52, avg=41.88, stdev= 6.10, samples=17 lat (msec) : 2=0.13%, 4=36.72%, 10=11.20%, 20=33.20%, 50=18.75% cpu : usr=0.11%, sys=9.64%, ctx=939, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10291: Mon Apr 7 15:30:08 2025 read: IOPS=41, BW=667KiB/s (683kB/s)(6032KiB/9038msec) clat (usec): min=1999, max=20444, avg=4481.22, stdev=2781.18 lat (usec): min=2000, max=20444, avg=4482.26, stdev=2781.13 clat percentiles (usec): | 1.00th=[ 2212], 5.00th=[ 2507], 10.00th=[ 2966], 20.00th=[ 3228], | 30.00th=[ 3359], 40.00th=[ 3490], 50.00th=[ 3621], 60.00th=[ 3752], | 70.00th=[ 3949], 80.00th=[ 4555], 90.00th=[ 6849], 95.00th=[10945], | 99.00th=[17171], 99.50th=[19268], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 320, max= 896, per=25.52%, avg=663.00, stdev=183.82, samples=18 iops : min= 20, max= 56, avg=41.39, stdev=11.44, samples=18 write: IOPS=43, BW=692KiB/s (709kB/s)(6256KiB/9038msec) clat (usec): min=9952, max=52967, avg=18729.18, stdev=5320.07 lat (usec): min=9953, max=52969, avg=18730.94, stdev=5320.08 clat percentiles (usec): | 1.00th=[11338], 5.00th=[12518], 10.00th=[13435], 20.00th=[14353], | 30.00th=[15139], 40.00th=[16319], 50.00th=[17695], 60.00th=[19530], | 70.00th=[20579], 80.00th=[22414], 90.00th=[24249], 95.00th=[27657], | 99.00th=[38011], 99.50th=[44827], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 416, max= 864, per=25.29%, avg=691.44, stdev=105.88, samples=18 iops : min= 26, max= 54, avg=43.17, stdev= 6.60, samples=18 lat (msec) : 2=0.13%, 4=35.42%, 10=10.68%, 20=35.81%, 50=17.84% lat (msec) : 100=0.13% cpu : usr=0.15%, sys=9.41%, ctx=924, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2599KiB/s (2661kB/s), 623KiB/s-702KiB/s (638kB/s-719kB/s), io=23.4MiB (24.5MB), run=8914-9216msec WRITE: bw=2734KiB/s (2800kB/s), 677KiB/s-710KiB/s (693kB/s-727kB/s), io=24.6MiB (25.8MB), run=8914-9216msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10303: Mon Apr 7 15:30:10 2025 read: IOPS=5, BW=5795KiB/s (5934kB/s)(7168KiB/1237msec) clat (usec): min=642, max=35645, avg=26731.73, stdev=11933.39 lat (usec): min=643, max=35646, avg=26732.77, stdev=11933.58 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 644], 10.00th=[ 644], 20.00th=[26346], | 30.00th=[27919], 40.00th=[27919], 50.00th=[30278], 60.00th=[32637], | 70.00th=[32637], 80.00th=[33424], 90.00th=[35390], 95.00th=[35390], | 99.00th=[35390], 99.50th=[35390], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 6144, max= 6144, per=46.62%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4139KiB/s (4238kB/s)(5120KiB/1237msec) clat (msec): min=22, max=819, avg=198.44, stdev=348.29 lat (msec): min=22, max=819, avg=198.48, stdev=348.28 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 23], | 30.00th=[ 25], 40.00th=[ 25], 50.00th=[ 43], 60.00th=[ 43], | 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 818], 95.00th=[ 818], | 99.00th=[ 818], 99.50th=[ 818], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 2043, max= 2043, per=11.07%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=75.00%, 100=8.33%, 1000=8.33% cpu : usr=0.00%, sys=16.18%, ctx=1071, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10304: Mon Apr 7 15:30:10 2025 read: IOPS=1, BW=1318KiB/s (1350kB/s)(2048KiB/1554msec) clat (msec): min=30, max=905, avg=467.84, stdev=618.87 lat (msec): min=30, max=905, avg=467.85, stdev=618.87 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 902], | 70.00th=[ 902], 80.00th=[ 902], 90.00th=[ 902], 95.00th=[ 902], | 99.00th=[ 902], 99.50th=[ 902], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 2043, max= 2043, per=15.50%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6589KiB/s (6748kB/s)(10.0MiB/1554msec) clat (msec): min=27, max=103, avg=60.98, stdev=30.19 lat (msec): min=27, max=103, avg=61.03, stdev=30.18 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 36], | 30.00th=[ 37], 40.00th=[ 41], 50.00th=[ 48], 60.00th=[ 49], | 70.00th=[ 101], 80.00th=[ 101], 90.00th=[ 103], 95.00th=[ 104], | 99.00th=[ 104], 99.50th=[ 104], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 2043, max= 2043, per=11.07%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=8.33%, 250=25.00%, 1000=8.33% cpu : usr=0.06%, sys=20.41%, ctx=362, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10305: Mon Apr 7 15:30:10 2025 read: IOPS=4, BW=4455KiB/s (4562kB/s)(6144KiB/1379msec) clat (usec): min=301, max=59635, avg=35871.97, stdev=22456.35 lat (usec): min=302, max=59636, avg=35872.77, stdev=22456.30 clat percentiles (usec): | 1.00th=[ 302], 5.00th=[ 302], 10.00th=[ 302], 20.00th=[24511], | 30.00th=[24511], 40.00th=[31065], 50.00th=[31065], 60.00th=[41157], | 70.00th=[58459], 80.00th=[58459], 90.00th=[59507], 95.00th=[59507], | 99.00th=[59507], 99.50th=[59507], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] write: IOPS=4, BW=4455KiB/s (4562kB/s)(6144KiB/1379msec) clat (msec): min=15, max=941, avg=188.64, stdev=369.59 lat (msec): min=15, max=941, avg=188.68, stdev=369.60 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 42], | 70.00th=[ 81], 80.00th=[ 81], 90.00th=[ 944], 95.00th=[ 944], | 99.00th=[ 944], 99.50th=[ 944], 99.90th=[ 944], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 4087, max= 4087, per=22.15%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 20=8.33%, 50=50.00%, 100=25.00%, 1000=8.33% cpu : usr=0.00%, sys=11.97%, ctx=549, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10306: Mon Apr 7 15:30:10 2025 read: IOPS=3, BW=3507KiB/s (3591kB/s)(5120KiB/1460msec) clat (usec): min=310, max=67198, avg=21093.49, stdev=30280.37 lat (usec): min=310, max=67198, avg=21094.18, stdev=30280.30 clat percentiles (usec): | 1.00th=[ 310], 5.00th=[ 310], 10.00th=[ 310], 20.00th=[ 310], | 30.00th=[ 424], 40.00th=[ 424], 50.00th=[ 437], 60.00th=[ 437], | 70.00th=[36963], 80.00th=[36963], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] write: IOPS=4, BW=4910KiB/s (5027kB/s)(7168KiB/1460msec) clat (msec): min=22, max=959, avg=190.72, stdev=340.86 lat (msec): min=22, max=959, avg=190.77, stdev=340.86 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 34], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 71], 60.00th=[ 104], | 70.00th=[ 104], 80.00th=[ 111], 90.00th=[ 961], 95.00th=[ 961], | 99.00th=[ 961], 99.50th=[ 961], 99.90th=[ 961], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 4087, max= 4087, per=22.15%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (usec) : 500=25.00% lat (msec) : 50=33.33%, 100=16.67%, 250=16.67%, 1000=8.33% cpu : usr=0.07%, sys=11.17%, ctx=343, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.9MiB/s (13.5MB/s), 1318KiB/s-5795KiB/s (1350kB/s-5934kB/s), io=20.0MiB (20.0MB), run=1237-1554msec WRITE: bw=18.0MiB/s (18.9MB/s), 4139KiB/s-6589KiB/s (4238kB/s-6748kB/s), io=28.0MiB (29.4MB), run=1237-1554msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10308: Mon Apr 7 15:30:10 2025 read: IOPS=4, BW=5044KiB/s (5165kB/s)(7168KiB/1421msec) clat (usec): min=13089, max=44061, avg=28721.23, stdev=11138.12 lat (usec): min=13090, max=44062, avg=28722.05, stdev=11138.21 clat percentiles (usec): | 1.00th=[13042], 5.00th=[13042], 10.00th=[13042], 20.00th=[18744], | 30.00th=[23725], 40.00th=[23725], 50.00th=[27657], 60.00th=[35914], | 70.00th=[35914], 80.00th=[38011], 90.00th=[44303], 95.00th=[44303], | 99.00th=[44303], 99.50th=[44303], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 6144, max= 6144, per=48.03%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3603KiB/s (3690kB/s)(5120KiB/1421msec) clat (msec): min=34, max=872, avg=241.95, stdev=355.71 lat (msec): min=34, max=872, avg=241.99, stdev=355.72 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 35], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 110], 60.00th=[ 110], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 2048, max= 2048, per=11.44%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=58.33%, 250=16.67%, 1000=8.33% cpu : usr=0.07%, sys=6.20%, ctx=65, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10309: Mon Apr 7 15:30:10 2025 read: IOPS=1, BW=1279KiB/s (1310kB/s)(2048KiB/1601msec) clat (msec): min=18, max=845, avg=431.97, stdev=584.41 lat (msec): min=18, max=845, avg=431.97, stdev=584.41 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 19], 60.00th=[ 844], | 70.00th=[ 844], 80.00th=[ 844], 90.00th=[ 844], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 2043, max= 2048, per=15.99%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=6, BW=6396KiB/s (6550kB/s)(10.0MiB/1601msec) clat (msec): min=35, max=159, avg=69.65, stdev=40.20 lat (msec): min=35, max=159, avg=69.71, stdev=40.21 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 40], | 30.00th=[ 40], 40.00th=[ 41], 50.00th=[ 58], 60.00th=[ 58], | 70.00th=[ 82], 80.00th=[ 82], 90.00th=[ 116], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min=14336, max=14336, per=80.05%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=33.33%, 250=16.67%, 1000=8.33% cpu : usr=0.06%, sys=6.31%, ctx=90, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10310: Mon Apr 7 15:30:10 2025 read: IOPS=3, BW=4085KiB/s (4183kB/s)(6144KiB/1504msec) clat (usec): min=12239, max=63067, avg=28550.54, stdev=20028.51 lat (usec): min=12239, max=63068, avg=28551.19, stdev=20028.51 clat percentiles (usec): | 1.00th=[12256], 5.00th=[12256], 10.00th=[12256], 20.00th=[14222], | 30.00th=[14222], 40.00th=[14484], 50.00th=[14484], 60.00th=[26608], | 70.00th=[40633], 80.00th=[40633], 90.00th=[63177], 95.00th=[63177], | 99.00th=[63177], 99.50th=[63177], 99.90th=[63177], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min=10240, max=10240, per=80.05%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=3, BW=4085KiB/s (4183kB/s)(6144KiB/1504msec) clat (msec): min=36, max=969, avg=216.85, stdev=369.27 lat (msec): min=36, max=969, avg=216.90, stdev=369.27 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 52], | 30.00th=[ 52], 40.00th=[ 80], 50.00th=[ 80], 60.00th=[ 82], | 70.00th=[ 85], 80.00th=[ 85], 90.00th=[ 969], 95.00th=[ 969], | 99.00th=[ 969], 99.50th=[ 969], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 2048, max=10240, per=34.31%, avg=6144.00, stdev=5792.62, samples=2 iops : min= 2, max= 10, avg= 6.00, stdev= 5.66, samples=2 lat (msec) : 20=25.00%, 50=25.00%, 100=41.67%, 1000=8.33% cpu : usr=0.00%, sys=4.99%, ctx=79, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10311: Mon Apr 7 15:30:10 2025 read: IOPS=3, BW=3360KiB/s (3440kB/s)(5120KiB/1524msec) clat (usec): min=12259, max=36557, avg=24411.56, stdev=11306.28 lat (usec): min=12260, max=36557, avg=24412.78, stdev=11306.50 clat percentiles (usec): | 1.00th=[12256], 5.00th=[12256], 10.00th=[12256], 20.00th=[12256], | 30.00th=[13566], 40.00th=[13566], 50.00th=[25560], 60.00th=[25560], | 70.00th=[34341], 80.00th=[34341], 90.00th=[36439], 95.00th=[36439], | 99.00th=[36439], 99.50th=[36439], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 8175, max= 8175, per=63.91%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4703KiB/s (4816kB/s)(7168KiB/1524msec) clat (msec): min=24, max=962, avg=194.53, stdev=339.96 lat (msec): min=24, max=962, avg=194.59, stdev=339.97 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 50], | 30.00th=[ 59], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 97], | 70.00th=[ 97], 80.00th=[ 109], 90.00th=[ 961], 95.00th=[ 961], | 99.00th=[ 961], 99.50th=[ 961], 99.90th=[ 961], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min=12263, max=12263, per=68.48%, avg=12263.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=41.67%, 100=25.00%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=5.06%, ctx=44, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.5MiB/s (13.1MB/s), 1279KiB/s-5044KiB/s (1310kB/s-5165kB/s), io=20.0MiB (20.0MB), run=1421-1601msec WRITE: bw=17.5MiB/s (18.3MB/s), 3603KiB/s-6396KiB/s (3690kB/s-6550kB/s), io=28.0MiB (29.4MB), run=1421-1601msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10319: Mon Apr 7 15:30:12 2025 read: IOPS=8, BW=33.1MiB/s (34.8MB/s)(12.0MiB/362msec) clat (msec): min=38, max=185, avg=117.44, stdev=73.94 lat (msec): min=38, max=185, avg=117.44, stdev=73.94 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 129], 50.00th=[ 129], 60.00th=[ 129], | 70.00th=[ 186], 80.00th=[ 186], 90.00th=[ 186], 95.00th=[ 186], | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] lat (msec) : 50=33.33%, 250=66.67% cpu : usr=0.00%, sys=30.47%, ctx=1146, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10321: Mon Apr 7 15:30:12 2025 read: IOPS=2, BW=8847KiB/s (9059kB/s)(4096KiB/463msec) clat (nsec): min=85816k, max=85816k, avg=85816323.00, stdev= 0.00 lat (nsec): min=85817k, max=85817k, avg=85817335.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[85459], 5.00th=[85459], 10.00th=[85459], 20.00th=[85459], | 30.00th=[85459], 40.00th=[85459], 50.00th=[85459], 60.00th=[85459], | 70.00th=[85459], 80.00th=[85459], 90.00th=[85459], 95.00th=[85459], | 99.00th=[85459], 99.50th=[85459], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] write: IOPS=4, BW=17.3MiB/s (18.1MB/s)(8192KiB/463msec) clat (msec): min=130, max=229, avg=179.63, stdev=69.90 lat (msec): min=130, max=229, avg=179.90, stdev=69.87 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 131], 10.00th=[ 131], 20.00th=[ 131], | 30.00th=[ 131], 40.00th=[ 131], 50.00th=[ 131], 60.00th=[ 230], | 70.00th=[ 230], 80.00th=[ 230], 90.00th=[ 230], 95.00th=[ 230], | 99.00th=[ 230], 99.50th=[ 230], 99.90th=[ 230], 99.95th=[ 230], | 99.99th=[ 230] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=29.22%, ctx=376, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10323: Mon Apr 7 15:30:12 2025 write: IOPS=5, BW=23.6MiB/s (24.8MB/s)(12.0MiB/508msec) clat (msec): min=107, max=275, avg=164.45, stdev=96.33 lat (msec): min=107, max=275, avg=164.71, stdev=96.32 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 108], 10.00th=[ 108], 20.00th=[ 108], | 30.00th=[ 108], 40.00th=[ 110], 50.00th=[ 110], 60.00th=[ 110], | 70.00th=[ 275], 80.00th=[ 275], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min=16384, max=16384, per=27.75%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=34.52%, ctx=108, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10325: Mon Apr 7 15:30:12 2025 write: IOPS=5, BW=21.6MiB/s (22.7MB/s)(12.0MiB/555msec) clat (msec): min=151, max=201, avg=177.11, stdev=25.40 lat (msec): min=151, max=202, avg=177.38, stdev=25.39 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 153], 10.00th=[ 153], 20.00th=[ 153], | 30.00th=[ 153], 40.00th=[ 178], 50.00th=[ 178], 60.00th=[ 178], | 70.00th=[ 203], 80.00th=[ 203], 90.00th=[ 203], 95.00th=[ 203], | 99.00th=[ 203], 99.50th=[ 203], 99.90th=[ 203], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min=16384, max=16384, per=27.75%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=35.56%, ctx=121, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=34.6MiB/s (36.2MB/s), 8847KiB/s-33.1MiB/s (9059kB/s-34.8MB/s), io=16.0MiB (16.8MB), run=362-463msec WRITE: bw=57.7MiB/s (60.5MB/s), 17.3MiB/s-23.6MiB/s (18.1MB/s-24.8MB/s), io=32.0MiB (33.6MB), run=463-555msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10318: Mon Apr 7 15:30:12 2025 read: IOPS=10, BW=40.8MiB/s (42.8MB/s)(12.0MiB/294msec) clat (msec): min=48, max=180, avg=95.10, stdev=74.14 lat (msec): min=48, max=180, avg=95.10, stdev=74.14 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 57], 50.00th=[ 57], 60.00th=[ 57], | 70.00th=[ 182], 80.00th=[ 182], 90.00th=[ 182], 95.00th=[ 182], | 99.00th=[ 182], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] lat (msec) : 50=33.33%, 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=18.09%, ctx=70, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10320: Mon Apr 7 15:30:12 2025 read: IOPS=1, BW=5761KiB/s (5899kB/s)(4096KiB/711msec) clat (nsec): min=160154k, max=160154k, avg=160154477.00, stdev= 0.00 lat (nsec): min=160157k, max=160157k, avg=160156928.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 161], | 70.00th=[ 161], 80.00th=[ 161], 90.00th=[ 161], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 7627, max= 7627, per=33.10%, avg=7627.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=11.3MiB/s (11.8MB/s)(8192KiB/711msec) clat (msec): min=97, max=444, avg=270.81, stdev=245.23 lat (msec): min=97, max=444, avg=271.01, stdev=245.12 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 97], 10.00th=[ 97], 20.00th=[ 97], | 30.00th=[ 97], 40.00th=[ 97], 50.00th=[ 97], 60.00th=[ 443], | 70.00th=[ 443], 80.00th=[ 443], 90.00th=[ 443], 95.00th=[ 443], | 99.00th=[ 443], 99.50th=[ 443], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 443] lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.28%, sys=12.39%, ctx=116, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10322: Mon Apr 7 15:30:12 2025 write: IOPS=4, BW=17.0MiB/s (18.9MB/s)(12.0MiB/667msec) clat (msec): min=62, max=347, avg=213.14, stdev=143.24 lat (msec): min=62, max=347, avg=213.41, stdev=143.25 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 63], | 30.00th=[ 63], 40.00th=[ 230], 50.00th=[ 230], 60.00th=[ 230], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.15%, sys=15.17%, ctx=64, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10324: Mon Apr 7 15:30:12 2025 write: IOPS=6, BW=24.2MiB/s (25.4MB/s)(12.0MiB/495msec) clat (msec): min=76, max=312, avg=158.48, stdev=133.40 lat (msec): min=76, max=312, avg=158.76, stdev=133.38 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 77], 10.00th=[ 77], 20.00th=[ 77], | 30.00th=[ 77], 40.00th=[ 87], 50.00th=[ 87], 60.00th=[ 87], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 313], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] lat (msec) : 100=66.67%, 500=33.33% cpu : usr=0.00%, sys=11.13%, ctx=47, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.5MiB/s (23.6MB/s), 5761KiB/s-40.8MiB/s (5899kB/s-42.8MB/s), io=16.0MiB (16.8MB), run=294-711msec WRITE: bw=45.0MiB/s (47.2MB/s), 11.3MiB/s-24.2MiB/s (11.8MB/s-25.4MB/s), io=32.0MiB (33.6MB), run=495-711msec