== sanity test 398b: DIO and buffer IO race ============== 08:51:09 (1748868669) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.25207 s, 40.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=3422: Mon Jun 2 08:51:26 2025 read: IOPS=104, BW=419KiB/s (429kB/s)(5992KiB/14299msec) clat (usec): min=49, max=204973, avg=4343.29, stdev=6272.68 lat (usec): min=50, max=204974, avg=4344.72, stdev=6272.80 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 90], 10.00th=[ 106], 20.00th=[ 2245], | 30.00th=[ 2671], 40.00th=[ 3097], 50.00th=[ 3523], 60.00th=[ 4146], | 70.00th=[ 5080], 80.00th=[ 6390], 90.00th=[ 8094], 95.00th=[ 9503], | 99.00th=[ 17171], 99.50th=[ 22676], 99.90th=[ 46400], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 208, max= 608, per=25.03%, avg=415.57, stdev=85.07, samples=28 iops : min= 52, max= 152, avg=103.79, stdev=21.27, samples=28 write: IOPS=110, BW=440KiB/s (451kB/s)(6296KiB/14299msec) clat (usec): min=426, max=62585, avg=4874.22, stdev=4538.95 lat (usec): min=427, max=62586, avg=4875.69, stdev=4538.52 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 611], 10.00th=[ 685], 20.00th=[ 799], | 30.00th=[ 1221], 40.00th=[ 3064], 50.00th=[ 4178], 60.00th=[ 5014], | 70.00th=[ 6259], 80.00th=[ 8029], 90.00th=[10421], 95.00th=[12780], | 99.00th=[20055], 99.50th=[23200], 99.90th=[38011], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 232, max= 622, per=25.11%, avg=436.36, stdev=90.80, samples=28 iops : min= 58, max= 155, avg=109.00, stdev=22.66, samples=28 lat (usec) : 50=0.03%, 100=4.13%, 250=3.61%, 500=0.68%, 750=7.81% lat (usec) : 1000=5.21% lat (msec) : 2=4.79%, 4=26.73%, 10=39.13%, 20=7.00%, 50=0.81% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.17%, sys=22.59%, ctx=5594, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3424: Mon Jun 2 08:51:26 2025 read: IOPS=103, BW=415KiB/s (425kB/s)(5940KiB/14301msec) clat (usec): min=51, max=36233, avg=4157.22, stdev=3033.61 lat (usec): min=51, max=36234, avg=4158.09, stdev=3033.62 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 91], 10.00th=[ 110], 20.00th=[ 2343], | 30.00th=[ 2769], 40.00th=[ 3163], 50.00th=[ 3621], 60.00th=[ 4146], | 70.00th=[ 5014], 80.00th=[ 6194], 90.00th=[ 7832], 95.00th=[ 9765], | 99.00th=[13173], 99.50th=[15795], 99.90th=[19006], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 216, max= 576, per=24.95%, avg=414.14, stdev=88.49, samples=28 iops : min= 54, max= 144, avg=103.46, stdev=22.14, samples=28 write: IOPS=110, BW=444KiB/s (455kB/s)(6348KiB/14301msec) clat (usec): min=425, max=199684, avg=5044.70, stdev=7005.04 lat (usec): min=426, max=199685, avg=5045.60, stdev=7005.03 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 586], 10.00th=[ 660], 20.00th=[ 766], | 30.00th=[ 1123], 40.00th=[ 2737], 50.00th=[ 3916], 60.00th=[ 4948], | 70.00th=[ 6259], 80.00th=[ 8094], 90.00th=[ 10683], 95.00th=[ 13304], | 99.00th=[ 22938], 99.50th=[ 28443], 99.90th=[ 55313], 99.95th=[200279], | 99.99th=[200279] bw ( KiB/s): min= 272, max= 600, per=25.13%, avg=436.71, stdev=82.04, samples=28 iops : min= 68, max= 150, avg=109.11, stdev=20.49, samples=28 lat (usec) : 100=3.84%, 250=3.39%, 500=0.65%, 750=8.85%, 1000=5.34% lat (msec) : 2=4.75%, 4=26.95%, 10=37.60%, 20=7.91%, 50=0.59% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.29%, sys=22.57%, ctx=5508, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3426: Mon Jun 2 08:51:26 2025 read: IOPS=104, BW=417KiB/s (427kB/s)(6028KiB/14460msec) clat (usec): min=51, max=38737, avg=4176.38, stdev=3303.16 lat (usec): min=51, max=38738, avg=4177.32, stdev=3303.23 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 83], 10.00th=[ 98], 20.00th=[ 2147], | 30.00th=[ 2638], 40.00th=[ 3097], 50.00th=[ 3556], 60.00th=[ 4178], | 70.00th=[ 5145], 80.00th=[ 6456], 90.00th=[ 8356], 95.00th=[ 9634], | 99.00th=[14615], 99.50th=[17171], 99.90th=[26084], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 120, max= 608, per=24.56%, avg=407.64, stdev=89.53, samples=28 iops : min= 30, max= 152, avg=101.86, stdev=22.37, samples=28 write: IOPS=108, BW=433KiB/s (443kB/s)(6260KiB/14460msec) clat (usec): min=434, max=231013, avg=5140.60, stdev=7520.85 lat (usec): min=435, max=231015, avg=5141.50, stdev=7520.88 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 627], 10.00th=[ 693], 20.00th=[ 791], | 30.00th=[ 1221], 40.00th=[ 2999], 50.00th=[ 4047], 60.00th=[ 5145], | 70.00th=[ 6390], 80.00th=[ 8094], 90.00th=[ 10683], 95.00th=[ 13566], | 99.00th=[ 20841], 99.50th=[ 25822], 99.90th=[ 52691], 99.95th=[231736], | 99.99th=[231736] bw ( KiB/s): min= 152, max= 576, per=24.28%, avg=421.93, stdev=87.60, samples=28 iops : min= 38, max= 144, avg=105.43, stdev=21.89, samples=28 lat (usec) : 100=5.08%, 250=3.61%, 500=0.55%, 750=7.91%, 1000=5.05% lat (msec) : 2=5.18%, 4=26.27%, 10=37.92%, 20=7.78%, 50=0.52% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.25%, sys=22.41%, ctx=5633, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3429: Mon Jun 2 08:51:26 2025 read: IOPS=105, BW=423KiB/s (433kB/s)(6052KiB/14301msec) clat (usec): min=48, max=29966, avg=4225.94, stdev=3236.66 lat (usec): min=49, max=29967, avg=4226.71, stdev=3236.64 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 89], 10.00th=[ 104], 20.00th=[ 2343], | 30.00th=[ 2802], 40.00th=[ 3163], 50.00th=[ 3621], 60.00th=[ 4113], | 70.00th=[ 5211], 80.00th=[ 6456], 90.00th=[ 8160], 95.00th=[ 9634], | 99.00th=[14615], 99.50th=[17695], 99.90th=[29492], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 231, max= 576, per=25.32%, avg=420.39, stdev=79.35, samples=28 iops : min= 57, max= 144, avg=104.96, stdev=19.92, samples=28 write: IOPS=109, BW=436KiB/s (447kB/s)(6236KiB/14301msec) clat (usec): min=438, max=210407, avg=4993.72, stdev=6994.55 lat (usec): min=439, max=210408, avg=4994.78, stdev=6994.66 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 611], 10.00th=[ 693], 20.00th=[ 799], | 30.00th=[ 1287], 40.00th=[ 3130], 50.00th=[ 4178], 60.00th=[ 5145], | 70.00th=[ 6194], 80.00th=[ 7701], 90.00th=[ 10290], 95.00th=[ 12911], | 99.00th=[ 18482], 99.50th=[ 31065], 99.90th=[ 67634], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 295, max= 640, per=24.93%, avg=433.25, stdev=82.47, samples=28 iops : min= 73, max= 160, avg=108.18, stdev=20.76, samples=28 lat (usec) : 50=0.03%, 100=4.43%, 250=3.45%, 500=0.42%, 750=8.01% lat (usec) : 1000=4.92% lat (msec) : 2=4.85%, 4=26.46%, 10=40.04%, 20=6.87%, 50=0.46% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.24%, sys=22.59%, ctx=5698, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1661KiB/s (1700kB/s), 415KiB/s-423KiB/s (425kB/s-433kB/s), io=23.4MiB (24.6MB), run=14299-14460msec WRITE: bw=1739KiB/s (1780kB/s), 433KiB/s-444KiB/s (443kB/s-455kB/s), io=24.6MiB (25.7MB), run=14299-14460msec rand-rw: (groupid=0, jobs=1): err= 0: pid=3423: Mon Jun 2 08:51:51 2025 read: IOPS=38, BW=154KiB/s (157kB/s)(5992KiB/38972msec) clat (usec): min=1587, max=203970, avg=4532.70, stdev=6194.48 lat (usec): min=1587, max=203972, avg=4533.69, stdev=6194.48 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1991], 10.00th=[ 2114], 20.00th=[ 2343], | 30.00th=[ 2507], 40.00th=[ 2704], 50.00th=[ 3064], 60.00th=[ 3687], | 70.00th=[ 4490], 80.00th=[ 5669], 90.00th=[ 8455], 95.00th=[ 11338], | 99.00th=[ 17433], 99.50th=[ 22152], 99.90th=[ 38011], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 55, max= 336, per=25.03%, avg=153.68, stdev=65.19, samples=77 iops : min= 13, max= 84, avg=38.35, stdev=16.31, samples=77 write: IOPS=40, BW=162KiB/s (165kB/s)(6296KiB/38972msec) clat (usec): min=9242, max=66538, avg=20365.86, stdev=7851.04 lat (usec): min=9243, max=66539, avg=20367.15, stdev=7850.91 clat percentiles (usec): | 1.00th=[10159], 5.00th=[11600], 10.00th=[12387], 20.00th=[13698], | 30.00th=[15401], 40.00th=[16909], 50.00th=[18482], 60.00th=[20317], | 70.00th=[22152], 80.00th=[25560], 90.00th=[31327], 95.00th=[36963], | 99.00th=[44827], 99.50th=[49021], 99.90th=[61080], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 40, max= 232, per=25.02%, avg=160.64, stdev=48.96, samples=77 iops : min= 10, max= 58, avg=40.09, stdev=12.26, samples=77 lat (msec) : 2=2.54%, 4=28.52%, 10=14.81%, 20=32.58%, 50=21.32% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.13%, sys=5.56%, ctx=3752, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3425: Mon Jun 2 08:51:51 2025 read: IOPS=37, BW=152KiB/s (156kB/s)(5940KiB/39093msec) clat (usec): min=1429, max=213834, avg=4401.72, stdev=6236.42 lat (usec): min=1430, max=213835, avg=4402.72, stdev=6236.41 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 2008], 10.00th=[ 2114], 20.00th=[ 2311], | 30.00th=[ 2540], 40.00th=[ 2769], 50.00th=[ 3064], 60.00th=[ 3654], | 70.00th=[ 4490], 80.00th=[ 5538], 90.00th=[ 8160], 95.00th=[ 10421], | 99.00th=[ 15795], 99.50th=[ 20579], 99.90th=[ 31589], 99.95th=[212861], | 99.99th=[212861] bw ( KiB/s): min= 16, max= 304, per=24.74%, avg=151.91, stdev=62.71, samples=78 iops : min= 4, max= 76, avg=37.91, stdev=15.69, samples=78 write: IOPS=40, BW=162KiB/s (166kB/s)(6348KiB/39093msec) clat (usec): min=8980, max=83549, avg=20435.35, stdev=8135.93 lat (usec): min=8981, max=83550, avg=20436.50, stdev=8135.88 clat percentiles (usec): | 1.00th=[10028], 5.00th=[11338], 10.00th=[12518], 20.00th=[14222], | 30.00th=[15401], 40.00th=[16909], 50.00th=[18482], 60.00th=[20317], | 70.00th=[22152], 80.00th=[25035], 90.00th=[31327], 95.00th=[36439], | 99.00th=[48497], 99.50th=[52691], 99.90th=[68682], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 56, max= 240, per=25.24%, avg=162.06, stdev=47.54, samples=78 iops : min= 14, max= 60, avg=40.45, stdev=11.90, samples=78 lat (msec) : 2=2.31%, 4=28.35%, 10=15.56%, 20=32.16%, 50=21.16% lat (msec) : 100=0.42%, 250=0.03% cpu : usr=0.13%, sys=5.52%, ctx=3748, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3427: Mon Jun 2 08:51:51 2025 read: IOPS=38, BW=154KiB/s (158kB/s)(6028KiB/39107msec) clat (usec): min=1662, max=62837, avg=4614.13, stdev=3965.25 lat (usec): min=1663, max=62838, avg=4615.26, stdev=3965.26 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1991], 10.00th=[ 2147], 20.00th=[ 2343], | 30.00th=[ 2540], 40.00th=[ 2769], 50.00th=[ 3228], 60.00th=[ 3982], | 70.00th=[ 4817], 80.00th=[ 6325], 90.00th=[ 8717], 95.00th=[11207], | 99.00th=[17433], 99.50th=[26346], 99.90th=[42730], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 40, max= 312, per=25.05%, avg=153.83, stdev=57.94, samples=78 iops : min= 10, max= 78, avg=38.37, stdev=14.53, samples=78 write: IOPS=40, BW=160KiB/s (164kB/s)(6260KiB/39107msec) clat (msec): min=8, max=229, avg=20.47, stdev= 9.57 lat (msec): min=8, max=229, avg=20.47, stdev= 9.57 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 21], | 70.00th=[ 23], 80.00th=[ 26], 90.00th=[ 32], 95.00th=[ 37], | 99.00th=[ 49], 99.50th=[ 53], 99.90th=[ 67], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 40, max= 232, per=24.88%, avg=159.74, stdev=50.48, samples=78 iops : min= 10, max= 58, avg=39.85, stdev=12.64, samples=78 lat (msec) : 2=2.60%, 4=26.95%, 10=16.47%, 20=32.45%, 50=21.09% lat (msec) : 100=0.39%, 250=0.03% cpu : usr=0.11%, sys=5.55%, ctx=3793, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3428: Mon Jun 2 08:51:51 2025 read: IOPS=39, BW=156KiB/s (160kB/s)(6052KiB/38731msec) clat (usec): min=1582, max=59051, avg=4352.01, stdev=3622.65 lat (usec): min=1583, max=59052, avg=4353.03, stdev=3622.65 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1975], 10.00th=[ 2114], 20.00th=[ 2311], | 30.00th=[ 2507], 40.00th=[ 2737], 50.00th=[ 3032], 60.00th=[ 3720], | 70.00th=[ 4490], 80.00th=[ 5538], 90.00th=[ 8160], 95.00th=[10683], | 99.00th=[17171], 99.50th=[21627], 99.90th=[42730], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 40, max= 304, per=25.35%, avg=155.65, stdev=63.80, samples=77 iops : min= 10, max= 76, avg=38.84, stdev=15.97, samples=77 write: IOPS=40, BW=161KiB/s (165kB/s)(6236KiB/38731msec) clat (msec): min=9, max=254, avg=20.54, stdev=10.09 lat (msec): min=9, max=254, avg=20.54, stdev=10.09 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 19], 60.00th=[ 21], | 70.00th=[ 22], 80.00th=[ 26], 90.00th=[ 32], 95.00th=[ 36], | 99.00th=[ 48], 99.50th=[ 57], 99.90th=[ 84], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 56, max= 240, per=25.01%, avg=160.53, stdev=49.22, samples=77 iops : min= 14, max= 60, avg=40.06, stdev=12.32, samples=77 lat (msec) : 2=2.67%, 4=29.10%, 10=14.75%, 20=32.39%, 50=20.70% lat (msec) : 100=0.36%, 500=0.03% cpu : usr=0.12%, sys=5.58%, ctx=3737, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=614KiB/s (629kB/s), 152KiB/s-156KiB/s (156kB/s-160kB/s), io=23.4MiB (24.6MB), run=38731-39107msec WRITE: bw=643KiB/s (658kB/s), 160KiB/s-162KiB/s (164kB/s-166kB/s), io=24.6MiB (25.7MB), run=38731-39107msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=3469: Mon Jun 2 08:51:55 2025 read: IOPS=121, BW=1937KiB/s (1984kB/s)(5920KiB/3056msec) clat (usec): min=54, max=24221, avg=3942.84, stdev=3211.98 lat (usec): min=54, max=24222, avg=3943.62, stdev=3211.95 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 79], 10.00th=[ 99], 20.00th=[ 121], | 30.00th=[ 2835], 40.00th=[ 3326], 50.00th=[ 3785], 60.00th=[ 4228], | 70.00th=[ 5080], 80.00th=[ 5932], 90.00th=[ 7439], 95.00th=[ 8586], | 99.00th=[17957], 99.50th=[20841], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 1728, max= 2272, per=25.29%, avg=1941.33, stdev=209.02, samples=6 iops : min= 108, max= 142, avg=121.33, stdev=13.06, samples=6 write: IOPS=130, BW=2084KiB/s (2134kB/s)(6368KiB/3056msec) clat (usec): min=692, max=48656, avg=3980.60, stdev=4392.47 lat (usec): min=693, max=48657, avg=3982.23, stdev=4393.11 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 881], 10.00th=[ 963], 20.00th=[ 1074], | 30.00th=[ 1172], 40.00th=[ 1352], 50.00th=[ 2343], 60.00th=[ 3589], | 70.00th=[ 4555], 80.00th=[ 6456], 90.00th=[ 9372], 95.00th=[11207], | 99.00th=[21365], 99.50th=[22676], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 1664, max= 2432, per=25.95%, avg=2096.00, stdev=295.89, samples=6 iops : min= 104, max= 152, avg=131.00, stdev=18.49, samples=6 lat (usec) : 100=5.34%, 250=5.86%, 500=0.13%, 750=0.39%, 1000=7.03% lat (msec) : 2=17.97%, 4=24.09%, 10=33.59%, 20=4.82%, 50=0.78% cpu : usr=0.23%, sys=28.09%, ctx=1472, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3471: Mon Jun 2 08:51:55 2025 read: IOPS=117, BW=1875KiB/s (1920kB/s)(5744KiB/3063msec) clat (usec): min=51, max=20413, avg=3930.05, stdev=3377.24 lat (usec): min=51, max=20414, avg=3930.80, stdev=3377.25 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 72], 10.00th=[ 93], 20.00th=[ 115], | 30.00th=[ 2606], 40.00th=[ 3359], 50.00th=[ 3720], 60.00th=[ 4293], | 70.00th=[ 5014], 80.00th=[ 5997], 90.00th=[ 7898], 95.00th=[ 9896], | 99.00th=[15139], 99.50th=[20317], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 1760, max= 1984, per=24.46%, avg=1877.33, stdev=89.75, samples=6 iops : min= 110, max= 124, avg=117.33, stdev= 5.61, samples=6 write: IOPS=133, BW=2136KiB/s (2188kB/s)(6544KiB/3063msec) clat (usec): min=653, max=41636, avg=3950.16, stdev=4645.74 lat (usec): min=653, max=41637, avg=3951.73, stdev=4646.35 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 881], 10.00th=[ 963], 20.00th=[ 1074], | 30.00th=[ 1205], 40.00th=[ 1401], 50.00th=[ 2008], 60.00th=[ 3326], | 70.00th=[ 4359], 80.00th=[ 6456], 90.00th=[ 8979], 95.00th=[11076], | 99.00th=[20841], 99.50th=[26870], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 1728, max= 2528, per=26.35%, avg=2128.00, stdev=278.79, samples=6 iops : min= 108, max= 158, avg=133.00, stdev=17.42, samples=6 lat (usec) : 100=6.51%, 250=6.12%, 500=0.39%, 750=0.78%, 1000=6.51% lat (msec) : 2=19.27%, 4=21.61%, 10=32.55%, 20=5.34%, 50=0.91% cpu : usr=0.29%, sys=26.91%, ctx=1450, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3474: Mon Jun 2 08:51:55 2025 read: IOPS=125, BW=2005KiB/s (2053kB/s)(6256KiB/3120msec) clat (usec): min=53, max=27737, avg=4150.66, stdev=3574.61 lat (usec): min=53, max=27738, avg=4151.45, stdev=3574.61 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 87], 10.00th=[ 99], 20.00th=[ 117], | 30.00th=[ 2638], 40.00th=[ 3326], 50.00th=[ 3982], 60.00th=[ 4686], | 70.00th=[ 5604], 80.00th=[ 6587], 90.00th=[ 8160], 95.00th=[ 9634], | 99.00th=[17957], 99.50th=[20841], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 1756, max= 2496, per=26.05%, avg=1999.33, stdev=293.77, samples=6 iops : min= 109, max= 156, avg=124.83, stdev=18.49, samples=6 write: IOPS=120, BW=1933KiB/s (1980kB/s)(6032KiB/3120msec) clat (usec): min=654, max=41994, avg=3898.69, stdev=4495.35 lat (usec): min=655, max=41995, avg=3899.84, stdev=4495.36 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 857], 10.00th=[ 922], 20.00th=[ 1057], | 30.00th=[ 1188], 40.00th=[ 1352], 50.00th=[ 1909], 60.00th=[ 3326], | 70.00th=[ 4293], 80.00th=[ 6652], 90.00th=[ 9241], 95.00th=[11207], | 99.00th=[19530], 99.50th=[38011], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 1564, max= 2112, per=23.17%, avg=1871.33, stdev=204.34, samples=6 iops : min= 97, max= 132, avg=116.83, stdev=13.00, samples=6 lat (usec) : 100=5.47%, 250=8.33%, 750=0.91%, 1000=6.90% lat (msec) : 2=17.84%, 4=19.27%, 10=35.55%, 20=5.08%, 50=0.65% cpu : usr=0.26%, sys=27.28%, ctx=1492, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3476: Mon Jun 2 08:51:55 2025 read: IOPS=123, BW=1982KiB/s (2029kB/s)(6032KiB/3044msec) clat (usec): min=53, max=24070, avg=3771.71, stdev=3232.95 lat (usec): min=53, max=24071, avg=3773.32, stdev=3232.84 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 88], 10.00th=[ 99], 20.00th=[ 117], | 30.00th=[ 2474], 40.00th=[ 3097], 50.00th=[ 3621], 60.00th=[ 4359], | 70.00th=[ 5014], 80.00th=[ 5997], 90.00th=[ 7832], 95.00th=[ 8848], | 99.00th=[13435], 99.50th=[21627], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 1664, max= 2432, per=25.43%, avg=1952.00, stdev=280.43, samples=6 iops : min= 104, max= 152, avg=122.00, stdev=17.53, samples=6 write: IOPS=128, BW=2055KiB/s (2105kB/s)(6256KiB/3044msec) clat (usec): min=601, max=34878, avg=4008.01, stdev=4386.95 lat (usec): min=602, max=34879, avg=4009.17, stdev=4386.99 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 914], 10.00th=[ 996], 20.00th=[ 1106], | 30.00th=[ 1237], 40.00th=[ 1450], 50.00th=[ 2024], 60.00th=[ 3621], | 70.00th=[ 5014], 80.00th=[ 6718], 90.00th=[ 8586], 95.00th=[10290], | 99.00th=[25560], 99.50th=[33424], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 1856, max= 2304, per=25.16%, avg=2032.00, stdev=159.04, samples=6 iops : min= 116, max= 144, avg=127.00, stdev= 9.94, samples=6 lat (usec) : 100=5.21%, 250=8.85%, 500=0.13%, 750=0.52%, 1000=4.95% lat (msec) : 2=20.18%, 4=19.53%, 10=36.72%, 20=2.86%, 50=1.04% cpu : usr=0.33%, sys=26.95%, ctx=1453, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7677KiB/s (7861kB/s), 1875KiB/s-2005KiB/s (1920kB/s-2053kB/s), io=23.4MiB (24.5MB), run=3044-3120msec WRITE: bw=8077KiB/s (8271kB/s), 1933KiB/s-2136KiB/s (1980kB/s-2188kB/s), io=24.6MiB (25.8MB), run=3044-3120msec rand-rw: (groupid=0, jobs=1): err= 0: pid=3470: Mon Jun 2 08:52:04 2025 read: IOPS=32, BW=520KiB/s (532kB/s)(5920KiB/11385msec) clat (usec): min=2024, max=53320, avg=4956.19, stdev=4194.87 lat (usec): min=2025, max=53320, avg=4957.31, stdev=4194.83 clat percentiles (usec): | 1.00th=[ 2311], 5.00th=[ 2573], 10.00th=[ 2704], 20.00th=[ 2966], | 30.00th=[ 3228], 40.00th=[ 3458], 50.00th=[ 3687], 60.00th=[ 4080], | 70.00th=[ 4621], 80.00th=[ 5604], 90.00th=[ 7898], 95.00th=[11731], | 99.00th=[22414], 99.50th=[23987], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 288, max= 1056, per=24.99%, avg=511.64, stdev=169.33, samples=22 iops : min= 18, max= 66, avg=31.77, stdev=10.63, samples=22 write: IOPS=34, BW=559KiB/s (573kB/s)(6368KiB/11385msec) clat (usec): min=10470, max=68338, avg=23944.36, stdev=8415.90 lat (usec): min=10471, max=68340, avg=23946.33, stdev=8415.96 clat percentiles (usec): | 1.00th=[12518], 5.00th=[13698], 10.00th=[15139], 20.00th=[16909], | 30.00th=[18744], 40.00th=[20841], 50.00th=[22676], 60.00th=[24773], | 70.00th=[26608], 80.00th=[29492], 90.00th=[33817], 95.00th=[40109], | 99.00th=[60031], 99.50th=[67634], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 384, max= 736, per=25.92%, avg=558.14, stdev=128.92, samples=22 iops : min= 24, max= 46, avg=34.68, stdev= 8.04, samples=22 lat (msec) : 4=27.73%, 10=17.45%, 20=21.22%, 50=32.81%, 100=0.78% cpu : usr=0.09%, sys=6.03%, ctx=945, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3472: Mon Jun 2 08:52:04 2025 read: IOPS=30, BW=491KiB/s (503kB/s)(5744KiB/11700msec) clat (usec): min=2255, max=51651, avg=4945.72, stdev=4349.35 lat (usec): min=2256, max=51652, avg=4947.37, stdev=4349.62 clat percentiles (usec): | 1.00th=[ 2343], 5.00th=[ 2638], 10.00th=[ 2769], 20.00th=[ 3032], | 30.00th=[ 3195], 40.00th=[ 3458], 50.00th=[ 3654], 60.00th=[ 3982], | 70.00th=[ 4555], 80.00th=[ 5342], 90.00th=[ 7767], 95.00th=[ 9896], | 99.00th=[23725], 99.50th=[27395], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 256, max= 736, per=23.91%, avg=489.35, stdev=156.04, samples=23 iops : min= 16, max= 46, avg=30.39, stdev= 9.66, samples=23 write: IOPS=34, BW=559KiB/s (573kB/s)(6544KiB/11700msec) clat (usec): min=11061, max=63171, avg=24130.08, stdev=8452.46 lat (usec): min=11063, max=63173, avg=24132.19, stdev=8452.72 clat percentiles (usec): | 1.00th=[12387], 5.00th=[14222], 10.00th=[15270], 20.00th=[17433], | 30.00th=[19006], 40.00th=[20579], 50.00th=[22414], 60.00th=[24249], | 70.00th=[26346], 80.00th=[29230], 90.00th=[35914], 95.00th=[40633], | 99.00th=[54264], 99.50th=[57934], 99.90th=[63177], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 287, max= 736, per=25.83%, avg=556.17, stdev=127.17, samples=23 iops : min= 17, max= 46, avg=34.57, stdev= 8.02, samples=23 lat (msec) : 4=28.26%, 10=16.28%, 20=20.83%, 50=33.85%, 100=0.78% cpu : usr=0.05%, sys=5.90%, ctx=957, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3473: Mon Jun 2 08:52:04 2025 read: IOPS=35, BW=570KiB/s (583kB/s)(6256KiB/10982msec) clat (usec): min=2127, max=34184, avg=4914.62, stdev=3729.15 lat (usec): min=2127, max=34185, avg=4915.87, stdev=3729.12 clat percentiles (usec): | 1.00th=[ 2180], 5.00th=[ 2474], 10.00th=[ 2704], 20.00th=[ 2868], | 30.00th=[ 3130], 40.00th=[ 3392], 50.00th=[ 3752], 60.00th=[ 4047], | 70.00th=[ 4490], 80.00th=[ 5735], 90.00th=[ 8291], 95.00th=[12125], | 99.00th=[19792], 99.50th=[26870], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 224, max= 1088, per=28.13%, avg=575.86, stdev=247.01, samples=21 iops : min= 14, max= 68, avg=35.90, stdev=15.45, samples=21 write: IOPS=34, BW=549KiB/s (562kB/s)(6032KiB/10982msec) clat (usec): min=10278, max=67490, avg=23861.65, stdev=8399.28 lat (usec): min=10281, max=67491, avg=23863.56, stdev=8399.21 clat percentiles (usec): | 1.00th=[11863], 5.00th=[13566], 10.00th=[15139], 20.00th=[16712], | 30.00th=[18482], 40.00th=[20317], 50.00th=[22152], 60.00th=[24511], | 70.00th=[26870], 80.00th=[29754], 90.00th=[34866], 95.00th=[40633], | 99.00th=[50070], 99.50th=[57410], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 384, max= 768, per=24.98%, avg=537.76, stdev=114.39, samples=21 iops : min= 24, max= 48, avg=33.52, stdev= 7.22, samples=21 lat (msec) : 4=30.08%, 10=16.80%, 20=22.40%, 50=30.34%, 100=0.39% cpu : usr=0.04%, sys=6.38%, ctx=959, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3475: Mon Jun 2 08:52:04 2025 read: IOPS=32, BW=526KiB/s (539kB/s)(6032KiB/11457msec) clat (usec): min=2023, max=27503, avg=4748.62, stdev=3561.42 lat (usec): min=2023, max=27504, avg=4749.72, stdev=3561.36 clat percentiles (usec): | 1.00th=[ 2311], 5.00th=[ 2540], 10.00th=[ 2704], 20.00th=[ 2900], | 30.00th=[ 3097], 40.00th=[ 3359], 50.00th=[ 3621], 60.00th=[ 3916], | 70.00th=[ 4359], 80.00th=[ 5145], 90.00th=[ 7635], 95.00th=[12518], | 99.00th=[21890], 99.50th=[24773], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 224, max= 928, per=25.36%, avg=519.18, stdev=207.22, samples=22 iops : min= 14, max= 58, avg=32.36, stdev=13.03, samples=22 write: IOPS=34, BW=546KiB/s (559kB/s)(6256KiB/11457msec) clat (usec): min=11494, max=65118, avg=24558.63, stdev=8287.91 lat (usec): min=11496, max=65119, avg=24560.57, stdev=8287.83 clat percentiles (usec): | 1.00th=[12125], 5.00th=[14353], 10.00th=[15926], 20.00th=[17695], | 30.00th=[19268], 40.00th=[21365], 50.00th=[22676], 60.00th=[25035], | 70.00th=[27657], 80.00th=[30278], 90.00th=[35390], 95.00th=[41157], | 99.00th=[52691], 99.50th=[56361], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 320, max= 672, per=25.19%, avg=542.41, stdev=110.21, samples=22 iops : min= 20, max= 42, avg=33.82, stdev= 6.93, samples=22 lat (msec) : 4=30.73%, 10=14.97%, 20=20.57%, 50=33.07%, 100=0.65% cpu : usr=0.10%, sys=5.91%, ctx=958, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2047KiB/s (2096kB/s), 491KiB/s-570KiB/s (503kB/s-583kB/s), io=23.4MiB (24.5MB), run=10982-11700msec WRITE: bw=2154KiB/s (2206kB/s), 546KiB/s-559KiB/s (559kB/s-573kB/s), io=24.6MiB (25.8MB), run=10982-11700msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=3497: Mon Jun 2 08:52:06 2025 read: IOPS=6, BW=6217KiB/s (6366kB/s)(7168KiB/1153msec) clat (msec): min=28, max=465, avg=103.85, stdev=160.00 lat (msec): min=28, max=465, avg=103.85, stdev=160.00 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 44], 60.00th=[ 56], | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 468], 95.00th=[ 468], | 99.00th=[ 468], 99.50th=[ 468], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 2048, max= 8159, per=28.73%, avg=5103.50, stdev=4321.13, samples=2 iops : min= 2, max= 7, avg= 4.50, stdev= 3.54, samples=2 write: IOPS=4, BW=4441KiB/s (4547kB/s)(5120KiB/1153msec) clat (msec): min=31, max=136, avg=76.04, stdev=45.00 lat (msec): min=31, max=136, avg=76.09, stdev=45.00 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 81], 60.00th=[ 81], | 70.00th=[ 101], 80.00th=[ 101], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 8159, max= 8159, per=32.81%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=33.33%, 250=8.33%, 500=8.33% cpu : usr=0.09%, sys=25.35%, ctx=874, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3499: Mon Jun 2 08:52:06 2025 read: IOPS=1, BW=2024KiB/s (2072kB/s)(2048KiB/1012msec) clat (usec): min=36332, max=54677, avg=45505.19, stdev=12971.81 lat (usec): min=36335, max=54678, avg=45506.81, stdev=12970.63 clat percentiles (usec): | 1.00th=[36439], 5.00th=[36439], 10.00th=[36439], 20.00th=[36439], | 30.00th=[36439], 40.00th=[36439], 50.00th=[36439], 60.00th=[54789], | 70.00th=[54789], 80.00th=[54789], 90.00th=[54789], 95.00th=[54789], | 99.00th=[54789], 99.50th=[54789], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 2039, max= 2048, per=11.50%, avg=2043.50, stdev= 6.36, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=9, BW=9.88MiB/s (10.4MB/s)(10.0MiB/1012msec) clat (msec): min=17, max=503, avg=86.53, stdev=147.91 lat (msec): min=17, max=503, avg=86.61, stdev=147.90 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 23], | 30.00th=[ 30], 40.00th=[ 32], 50.00th=[ 37], 60.00th=[ 38], | 70.00th=[ 57], 80.00th=[ 57], 90.00th=[ 87], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min=18358, max=18358, per=73.82%, avg=18358.00, stdev= 0.00, samples=1 iops : min= 17, max= 17, avg=17.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=58.33%, 100=25.00%, 750=8.33% cpu : usr=0.00%, sys=21.17%, ctx=119, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3501: Mon Jun 2 08:52:06 2025 read: IOPS=6, BW=6642KiB/s (6802kB/s)(6144KiB/925msec) clat (usec): min=464, max=49232, avg=22206.80, stdev=23964.55 lat (usec): min=465, max=49233, avg=22208.05, stdev=23964.45 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 465], 10.00th=[ 465], 20.00th=[ 502], | 30.00th=[ 502], 40.00th=[ 570], 50.00th=[ 570], 60.00th=[39584], | 70.00th=[42730], 80.00th=[42730], 90.00th=[49021], 95.00th=[49021], | 99.00th=[49021], 99.50th=[49021], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] write: IOPS=6, BW=6642KiB/s (6802kB/s)(6144KiB/925msec) clat (msec): min=29, max=551, avg=129.44, stdev=207.63 lat (msec): min=29, max=552, avg=129.49, stdev=207.65 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 55], | 70.00th=[ 73], 80.00th=[ 73], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 4096, max= 4096, per=16.47%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=16.67% lat (msec) : 50=50.00%, 100=16.67%, 750=8.33% cpu : usr=0.00%, sys=15.91%, ctx=482, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3503: Mon Jun 2 08:52:06 2025 read: IOPS=4, BW=5100KiB/s (5222kB/s)(5120KiB/1004msec) clat (usec): min=428, max=68961, avg=35807.96, stdev=25644.53 lat (usec): min=430, max=68963, avg=35809.23, stdev=25644.54 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 429], 10.00th=[ 429], 20.00th=[ 429], | 30.00th=[26608], 40.00th=[26608], 50.00th=[33424], 60.00th=[33424], | 70.00th=[49546], 80.00th=[49546], 90.00th=[68682], 95.00th=[68682], | 99.00th=[68682], 99.50th=[68682], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 8159, max= 8159, per=45.94%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=6, BW=7139KiB/s (7311kB/s)(7168KiB/1004msec) clat (msec): min=23, max=547, avg=115.02, stdev=191.62 lat (msec): min=23, max=548, avg=115.09, stdev=191.62 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 29], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 46], 60.00th=[ 50], | 70.00th=[ 50], 80.00th=[ 74], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 2048, max=10199, per=24.63%, avg=6123.50, stdev=5763.63, samples=2 iops : min= 2, max= 9, avg= 5.50, stdev= 4.95, samples=2 lat (usec) : 500=8.33% lat (msec) : 50=66.67%, 100=16.67%, 750=8.33% cpu : usr=0.00%, sys=17.25%, ctx=316, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.3MiB/s (18.2MB/s), 2024KiB/s-6642KiB/s (2072kB/s-6802kB/s), io=20.0MiB (20.0MB), run=925-1153msec WRITE: bw=24.3MiB/s (25.5MB/s), 4441KiB/s-9.88MiB/s (4547kB/s-10.4MB/s), io=28.0MiB (29.4MB), run=925-1153msec rand-rw: (groupid=0, jobs=1): err= 0: pid=3498: Mon Jun 2 08:52:06 2025 read: IOPS=6, BW=6360KiB/s (6513kB/s)(7168KiB/1127msec) clat (msec): min=16, max=409, avg=91.71, stdev=141.63 lat (msec): min=16, max=409, avg=91.71, stdev=141.63 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 23], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 39], 60.00th=[ 54], | 70.00th=[ 54], 80.00th=[ 75], 90.00th=[ 409], 95.00th=[ 409], | 99.00th=[ 409], 99.50th=[ 409], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 6144, max= 6144, per=37.92%, avg=6144.00, stdev= 0.00, samples=2 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4543KiB/s (4652kB/s)(5120KiB/1127msec) clat (msec): min=50, max=168, avg=94.93, stdev=48.04 lat (msec): min=50, max=168, avg=94.99, stdev=48.05 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 52], | 30.00th=[ 59], 40.00th=[ 59], 50.00th=[ 82], 60.00th=[ 82], | 70.00th=[ 115], 80.00th=[ 115], 90.00th=[ 169], 95.00th=[ 169], | 99.00th=[ 169], 99.50th=[ 169], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 2048, max= 6144, per=18.06%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=8.33%, 50=25.00%, 100=41.67%, 250=16.67%, 500=8.33% cpu : usr=0.00%, sys=7.28%, ctx=46, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3500: Mon Jun 2 08:52:06 2025 read: IOPS=1, BW=1620KiB/s (1659kB/s)(2048KiB/1264msec) clat (msec): min=17, max=103, avg=60.43, stdev=60.51 lat (msec): min=17, max=103, avg=60.44, stdev=60.51 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 18], 60.00th=[ 104], | 70.00th=[ 104], 80.00th=[ 104], 90.00th=[ 104], 95.00th=[ 104], | 99.00th=[ 104], 99.50th=[ 104], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 2048, max= 2048, per=12.64%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=7, BW=8101KiB/s (8296kB/s)(10.0MiB/1264msec) clat (msec): min=26, max=565, avg=112.54, stdev=164.80 lat (msec): min=26, max=565, avg=112.62, stdev=164.80 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 28], | 30.00th=[ 37], 40.00th=[ 42], 50.00th=[ 43], 60.00th=[ 48], | 70.00th=[ 138], 80.00th=[ 138], 90.00th=[ 146], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 6144, max= 6144, per=27.09%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=8.33%, 250=25.00%, 750=8.33% cpu : usr=0.08%, sys=8.00%, ctx=60, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3502: Mon Jun 2 08:52:06 2025 read: IOPS=4, BW=4907KiB/s (5025kB/s)(6144KiB/1252msec) clat (msec): min=16, max=119, avg=49.86, stdev=38.22 lat (msec): min=16, max=119, avg=49.86, stdev=38.22 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 18], | 30.00th=[ 18], 40.00th=[ 34], 50.00th=[ 34], 60.00th=[ 57], | 70.00th=[ 57], 80.00th=[ 57], 90.00th=[ 120], 95.00th=[ 120], | 99.00th=[ 120], 99.50th=[ 120], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 4096, max= 4096, per=25.28%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4907KiB/s (5025kB/s)(6144KiB/1252msec) clat (msec): min=24, max=645, avg=141.66, stdev=247.64 lat (msec): min=24, max=645, avg=141.72, stdev=247.65 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 25], 40.00th=[ 41], 50.00th=[ 41], 60.00th=[ 43], | 70.00th=[ 73], 80.00th=[ 73], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 2048, max= 4096, per=13.54%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (msec) : 20=16.67%, 50=41.67%, 100=25.00%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=8.71%, ctx=60, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3504: Mon Jun 2 08:52:06 2025 read: IOPS=4, BW=4235KiB/s (4337kB/s)(5120KiB/1209msec) clat (usec): min=17698, max=64986, avg=34776.39, stdev=18432.86 lat (usec): min=17699, max=64988, avg=34777.73, stdev=18432.99 clat percentiles (usec): | 1.00th=[17695], 5.00th=[17695], 10.00th=[17695], 20.00th=[17695], | 30.00th=[24511], 40.00th=[24511], 50.00th=[28705], 60.00th=[28705], | 70.00th=[38011], 80.00th=[38011], 90.00th=[64750], 95.00th=[64750], | 99.00th=[64750], 99.50th=[64750], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 4096, max= 4096, per=25.28%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5929KiB/s (6071kB/s)(7168KiB/1209msec) clat (msec): min=38, max=582, avg=143.19, stdev=195.58 lat (msec): min=39, max=582, avg=143.26, stdev=195.58 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 48], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 86], 60.00th=[ 97], | 70.00th=[ 97], 80.00th=[ 102], 90.00th=[ 584], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 2048, max= 6144, per=18.06%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=8.33%, 50=50.00%, 100=25.00%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=10.09%, ctx=54, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.8MiB/s (16.6MB/s), 1620KiB/s-6360KiB/s (1659kB/s-6513kB/s), io=20.0MiB (20.0MB), run=1127-1264msec WRITE: bw=22.2MiB/s (23.2MB/s), 4543KiB/s-8101KiB/s (4652kB/s-8296kB/s), io=28.0MiB (29.4MB), run=1127-1264msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=3512: Mon Jun 2 08:52:08 2025 read: IOPS=5, BW=22.3MiB/s (23.3MB/s)(12.0MiB/539msec) clat (msec): min=121, max=265, avg=170.29, stdev=82.77 lat (msec): min=121, max=265, avg=170.29, stdev=82.77 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 122], 10.00th=[ 122], 20.00th=[ 122], | 30.00th=[ 122], 40.00th=[ 124], 50.00th=[ 124], 60.00th=[ 124], | 70.00th=[ 266], 80.00th=[ 266], 90.00th=[ 266], 95.00th=[ 266], | 99.00th=[ 266], 99.50th=[ 266], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min=16253, max=16253, per=85.91%, avg=16253.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=29.00%, ctx=1545, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3514: Mon Jun 2 08:52:08 2025 read: IOPS=1, BW=4730KiB/s (4843kB/s)(4096KiB/866msec) clat (nsec): min=134894k, max=134894k, avg=134894308.00, stdev= 0.00 lat (nsec): min=134896k, max=134896k, avg=134895644.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 136], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 8192, max= 8192, per=43.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9460KiB/s (9687kB/s)(8192KiB/866msec) clat (msec): min=284, max=419, avg=352.14, stdev=95.07 lat (msec): min=285, max=419, avg=352.47, stdev=95.12 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 284], 10.00th=[ 284], 20.00th=[ 284], | 30.00th=[ 284], 40.00th=[ 284], 50.00th=[ 284], 60.00th=[ 418], | 70.00th=[ 418], 80.00th=[ 418], 90.00th=[ 418], 95.00th=[ 418], | 99.00th=[ 418], 99.50th=[ 418], 99.90th=[ 418], 99.95th=[ 418], | 99.99th=[ 418] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=26.47%, ctx=646, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3516: Mon Jun 2 08:52:08 2025 write: IOPS=4, BW=16.9MiB/s (17.7MB/s)(12.0MiB/711msec) clat (msec): min=118, max=328, avg=231.45, stdev=105.58 lat (msec): min=119, max=328, avg=231.73, stdev=105.62 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 247], 50.00th=[ 247], 60.00th=[ 247], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 8159, max= 8159, per=21.56%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=34.65%, ctx=66, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3518: Mon Jun 2 08:52:08 2025 write: IOPS=4, BW=19.7MiB/s (20.6MB/s)(12.0MiB/610msec) clat (msec): min=134, max=228, avg=192.68, stdev=51.02 lat (msec): min=134, max=228, avg=192.93, stdev=51.08 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 134], 10.00th=[ 134], 20.00th=[ 134], | 30.00th=[ 134], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 215], | 70.00th=[ 228], 80.00th=[ 228], 90.00th=[ 228], 95.00th=[ 228], | 99.00th=[ 228], 99.50th=[ 228], 99.90th=[ 228], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min=16384, max=16384, per=43.30%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=42.04%, ctx=170, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.5MiB/s (19.4MB/s), 4730KiB/s-22.3MiB/s (4843kB/s-23.3MB/s), io=16.0MiB (16.8MB), run=539-866msec WRITE: bw=36.0MiB/s (38.7MB/s), 9460KiB/s-19.7MiB/s (9687kB/s-20.6MB/s), io=32.0MiB (33.6MB), run=610-866msec rand-rw: (groupid=0, jobs=1): err= 0: pid=3511: Mon Jun 2 08:52:08 2025 read: IOPS=6, BW=25.4MiB/s (26.6MB/s)(12.0MiB/473msec) clat (msec): min=65, max=301, avg=155.58, stdev=127.28 lat (msec): min=65, max=301, avg=155.58, stdev=127.28 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 66], 10.00th=[ 66], 20.00th=[ 66], | 30.00th=[ 66], 40.00th=[ 101], 50.00th=[ 101], 60.00th=[ 101], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=15.89%, ctx=53, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3513: Mon Jun 2 08:52:08 2025 read: IOPS=1, BW=5596KiB/s (5730kB/s)(4096KiB/732msec) clat (nsec): min=57542k, max=57542k, avg=57541783.00, stdev= 0.00 lat (nsec): min=57544k, max=57544k, avg=57543701.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[57410], 5.00th=[57410], 10.00th=[57410], 20.00th=[57410], | 30.00th=[57410], 40.00th=[57410], 50.00th=[57410], 60.00th=[57410], | 70.00th=[57410], 80.00th=[57410], 90.00th=[57410], 95.00th=[57410], | 99.00th=[57410], 99.50th=[57410], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 8192, max= 8192, per=36.60%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.9MiB/s (11.5MB/s)(8192KiB/732msec) clat (msec): min=303, max=356, avg=330.27, stdev=37.66 lat (msec): min=303, max=357, avg=330.53, stdev=37.69 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 305], 10.00th=[ 305], 20.00th=[ 305], | 30.00th=[ 305], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 359], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 8192, max= 8192, per=22.18%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.00%, sys=14.09%, ctx=223, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3515: Mon Jun 2 08:52:08 2025 write: IOPS=3, BW=13.5MiB/s (14.2MB/s)(12.0MiB/887msec) clat (msec): min=124, max=397, avg=290.05, stdev=145.47 lat (msec): min=124, max=398, avg=290.34, stdev=145.50 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 125], 10.00th=[ 125], 20.00th=[ 125], | 30.00th=[ 125], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 347], | 70.00th=[ 397], 80.00th=[ 397], 90.00th=[ 397], 95.00th=[ 397], | 99.00th=[ 397], 99.50th=[ 397], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 5595, max= 5595, per=15.15%, avg=5595.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=11.85%, ctx=62, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3517: Mon Jun 2 08:52:08 2025 write: IOPS=6, BW=24.1MiB/s (25.3MB/s)(12.0MiB/498msec) clat (msec): min=65, max=231, avg=161.98, stdev=86.55 lat (msec): min=65, max=231, avg=162.28, stdev=86.59 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 65], 10.00th=[ 65], 20.00th=[ 65], | 30.00th=[ 65], 40.00th=[ 190], 50.00th=[ 190], 60.00th=[ 190], | 70.00th=[ 232], 80.00th=[ 232], 90.00th=[ 232], 95.00th=[ 232], | 99.00th=[ 232], 99.50th=[ 232], 99.90th=[ 232], 99.95th=[ 232], | 99.99th=[ 232] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=15.29%, ctx=39, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=21.9MiB/s (22.9MB/s), 5596KiB/s-25.4MiB/s (5730kB/s-26.6MB/s), io=16.0MiB (16.8MB), run=473-732msec WRITE: bw=36.1MiB/s (37.8MB/s), 10.9MiB/s-24.1MiB/s (11.5MB/s-25.3MB/s), io=32.0MiB (33.6MB), run=498-887msec