== sanity test 398b: DIO and buffer IO race ============== 14:32:58 (1744137178) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.1726 s, 42.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30384: Tue Apr 8 14:33:12 2025 read: IOPS=128, BW=514KiB/s (527kB/s)(5992KiB/11647msec) clat (usec): min=61, max=146246, avg=4315.58, stdev=4285.04 lat (usec): min=61, max=146247, avg=4317.30, stdev=4284.77 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 1696], 10.00th=[ 2180], 20.00th=[ 2606], | 30.00th=[ 2933], 40.00th=[ 3359], 50.00th=[ 3851], 60.00th=[ 4424], | 70.00th=[ 5014], 80.00th=[ 5669], 90.00th=[ 6849], 95.00th=[ 8094], | 99.00th=[ 12911], 99.50th=[ 13960], 99.90th=[ 15795], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 336, max= 688, per=24.97%, avg=511.13, stdev=73.24, samples=23 iops : min= 84, max= 172, avg=127.74, stdev=18.27, samples=23 write: IOPS=135, BW=541KiB/s (554kB/s)(6296KiB/11647msec) clat (usec): min=446, max=18316, avg=3248.31, stdev=2844.27 lat (usec): min=447, max=18316, avg=3250.12, stdev=2844.97 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 545], 10.00th=[ 570], 20.00th=[ 619], | 30.00th=[ 791], 40.00th=[ 1958], 50.00th=[ 2638], 60.00th=[ 3261], | 70.00th=[ 4228], 80.00th=[ 5538], 90.00th=[ 7177], 95.00th=[ 8717], | 99.00th=[11863], 99.50th=[13173], 99.90th=[18220], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 392, max= 688, per=24.97%, avg=535.17, stdev=74.60, samples=23 iops : min= 98, max= 172, avg=133.74, stdev=18.68, samples=23 lat (usec) : 100=2.08%, 250=0.20%, 500=0.29%, 750=13.96%, 1000=3.12% lat (msec) : 2=4.62%, 4=36.10%, 10=37.08%, 20=2.51%, 250=0.03% cpu : usr=0.21%, sys=22.65%, ctx=5600, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30385: Tue Apr 8 14:33:12 2025 read: IOPS=128, BW=516KiB/s (528kB/s)(5940KiB/11522msec) clat (usec): min=56, max=21750, avg=4221.01, stdev=2203.12 lat (usec): min=56, max=21751, avg=4221.93, stdev=2203.26 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 1876], 10.00th=[ 2180], 20.00th=[ 2540], | 30.00th=[ 2900], 40.00th=[ 3261], 50.00th=[ 3785], 60.00th=[ 4424], | 70.00th=[ 5080], 80.00th=[ 5800], 90.00th=[ 6849], 95.00th=[ 8094], | 99.00th=[11469], 99.50th=[13435], 99.90th=[16319], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 256, max= 702, per=25.17%, avg=515.13, stdev=86.75, samples=23 iops : min= 64, max= 175, avg=128.65, stdev=21.65, samples=23 write: IOPS=137, BW=551KiB/s (564kB/s)(6348KiB/11522msec) clat (usec): min=470, max=158611, avg=3276.88, stdev=4793.09 lat (usec): min=471, max=158612, avg=3279.48, stdev=4793.26 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 545], 10.00th=[ 570], 20.00th=[ 627], | 30.00th=[ 775], 40.00th=[ 1958], 50.00th=[ 2638], 60.00th=[ 3195], | 70.00th=[ 4047], 80.00th=[ 5276], 90.00th=[ 7046], 95.00th=[ 8586], | 99.00th=[ 11863], 99.50th=[ 13042], 99.90th=[ 20841], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 336, max= 766, per=25.61%, avg=548.87, stdev=87.96, samples=23 iops : min= 84, max= 191, avg=137.09, stdev=21.96, samples=23 lat (usec) : 100=1.60%, 250=0.29%, 500=0.23%, 750=14.88%, 1000=2.77% lat (msec) : 2=4.46%, 4=37.40%, 10=36.26%, 20=2.02%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=22.98%, ctx=5546, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30386: Tue Apr 8 14:33:12 2025 read: IOPS=128, BW=516KiB/s (528kB/s)(6028KiB/11691msec) clat (usec): min=53, max=16370, avg=4181.67, stdev=2266.35 lat (usec): min=53, max=16370, avg=4183.02, stdev=2268.27 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 1729], 10.00th=[ 2147], 20.00th=[ 2474], | 30.00th=[ 2835], 40.00th=[ 3294], 50.00th=[ 3720], 60.00th=[ 4359], | 70.00th=[ 5014], 80.00th=[ 5735], 90.00th=[ 6915], 95.00th=[ 8160], | 99.00th=[12125], 99.50th=[13304], 99.90th=[15926], 99.95th=[16319], | 99.99th=[16319] bw ( KiB/s): min= 208, max= 630, per=24.98%, avg=511.26, stdev=83.72, samples=23 iops : min= 52, max= 157, avg=127.65, stdev=20.92, samples=23 write: IOPS=133, BW=535KiB/s (548kB/s)(6260KiB/11691msec) clat (usec): min=418, max=144605, avg=3398.92, stdev=4614.05 lat (usec): min=419, max=144607, avg=3399.78, stdev=4614.02 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 545], 10.00th=[ 570], 20.00th=[ 627], | 30.00th=[ 807], 40.00th=[ 1926], 50.00th=[ 2769], 60.00th=[ 3326], | 70.00th=[ 4228], 80.00th=[ 5604], 90.00th=[ 7373], 95.00th=[ 8848], | 99.00th=[ 12256], 99.50th=[ 14746], 99.90th=[ 24249], 99.95th=[143655], | 99.99th=[143655] bw ( KiB/s): min= 296, max= 630, per=24.73%, avg=530.00, stdev=80.82, samples=23 iops : min= 74, max= 157, avg=132.35, stdev=20.21, samples=23 lat (usec) : 100=1.89%, 250=0.39%, 500=0.42%, 750=14.10%, 1000=2.96% lat (msec) : 2=4.59%, 4=36.98%, 10=35.97%, 20=2.60%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=22.47%, ctx=5651, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30387: Tue Apr 8 14:33:12 2025 read: IOPS=128, BW=516KiB/s (528kB/s)(6052KiB/11730msec) clat (usec): min=60, max=156000, avg=4395.72, stdev=4546.40 lat (usec): min=60, max=156001, avg=4396.40, stdev=4546.42 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 1745], 10.00th=[ 2114], 20.00th=[ 2507], | 30.00th=[ 2868], 40.00th=[ 3294], 50.00th=[ 3884], 60.00th=[ 4490], | 70.00th=[ 5145], 80.00th=[ 5800], 90.00th=[ 7177], 95.00th=[ 8455], | 99.00th=[ 12518], 99.50th=[ 13566], 99.90th=[ 16188], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 272, max= 648, per=24.97%, avg=511.13, stdev=74.29, samples=23 iops : min= 68, max= 162, avg=127.70, stdev=18.59, samples=23 write: IOPS=132, BW=532KiB/s (544kB/s)(6236KiB/11730msec) clat (usec): min=416, max=21290, avg=3230.53, stdev=2863.98 lat (usec): min=417, max=21291, avg=3231.57, stdev=2863.98 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 545], 10.00th=[ 570], 20.00th=[ 627], | 30.00th=[ 807], 40.00th=[ 1926], 50.00th=[ 2606], 60.00th=[ 3261], | 70.00th=[ 4047], 80.00th=[ 5473], 90.00th=[ 7177], 95.00th=[ 8979], | 99.00th=[11600], 99.50th=[13960], 99.90th=[18220], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 344, max= 720, per=24.55%, avg=526.13, stdev=89.07, samples=23 iops : min= 86, max= 180, avg=131.43, stdev=22.36, samples=23 lat (usec) : 100=1.69%, 250=0.46%, 500=0.26%, 750=14.16%, 1000=2.57% lat (msec) : 2=5.01%, 4=36.82%, 10=36.00%, 20=2.96%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=22.54%, ctx=5596, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2047KiB/s (2096kB/s), 514KiB/s-516KiB/s (527kB/s-528kB/s), io=23.4MiB (24.6MB), run=11522-11730msec WRITE: bw=2143KiB/s (2195kB/s), 532KiB/s-551KiB/s (544kB/s-564kB/s), io=24.6MiB (25.7MB), run=11522-11730msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30379: Tue Apr 8 14:33:16 2025 read: IOPS=95, BW=381KiB/s (390kB/s)(5992KiB/15729msec) clat (usec): min=1672, max=159741, avg=4353.34, stdev=5031.42 lat (usec): min=1672, max=159742, avg=4353.98, stdev=5031.42 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1909], 10.00th=[ 2024], 20.00th=[ 2212], | 30.00th=[ 2376], 40.00th=[ 2638], 50.00th=[ 2999], 60.00th=[ 3621], | 70.00th=[ 4424], 80.00th=[ 5669], 90.00th=[ 8586], 95.00th=[ 10945], | 99.00th=[ 16450], 99.50th=[ 17433], 99.90th=[ 23987], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 176, max= 808, per=24.93%, avg=379.61, stdev=158.03, samples=31 iops : min= 44, max= 202, avg=94.90, stdev=39.51, samples=31 write: IOPS=100, BW=400KiB/s (410kB/s)(6296KiB/15729msec) clat (usec): min=2259, max=26652, avg=5806.78, stdev=3296.99 lat (usec): min=2260, max=26653, avg=5807.55, stdev=3296.97 clat percentiles (usec): | 1.00th=[ 2507], 5.00th=[ 2868], 10.00th=[ 3032], 20.00th=[ 3326], | 30.00th=[ 3621], 40.00th=[ 3982], 50.00th=[ 4555], 60.00th=[ 5473], | 70.00th=[ 6456], 80.00th=[ 8094], 90.00th=[10159], 95.00th=[12256], | 99.00th=[17957], 99.50th=[21103], 99.90th=[24773], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 184, max= 736, per=24.93%, avg=397.68, stdev=153.69, samples=31 iops : min= 46, max= 184, avg=99.42, stdev=38.42, samples=31 lat (msec) : 2=4.20%, 4=47.88%, 10=39.55%, 20=8.01%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.92%, ctx=3642, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30381: Tue Apr 8 14:33:16 2025 read: IOPS=94, BW=377KiB/s (386kB/s)(5940KiB/15761msec) clat (usec): min=1615, max=161781, avg=4271.02, stdev=5045.03 lat (usec): min=1615, max=161782, avg=4271.68, stdev=5045.02 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1893], 10.00th=[ 2008], 20.00th=[ 2180], | 30.00th=[ 2409], 40.00th=[ 2671], 50.00th=[ 2966], 60.00th=[ 3458], | 70.00th=[ 4293], 80.00th=[ 5473], 90.00th=[ 8586], 95.00th=[ 10552], | 99.00th=[ 15664], 99.50th=[ 16909], 99.90th=[ 23462], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 160, max= 776, per=24.72%, avg=376.52, stdev=157.73, samples=31 iops : min= 40, max= 194, avg=94.13, stdev=39.43, samples=31 write: IOPS=100, BW=403KiB/s (412kB/s)(6348KiB/15761msec) clat (usec): min=2281, max=25570, avg=5892.49, stdev=3347.82 lat (usec): min=2281, max=25571, avg=5893.39, stdev=3347.76 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2868], 10.00th=[ 3064], 20.00th=[ 3326], | 30.00th=[ 3621], 40.00th=[ 3982], 50.00th=[ 4621], 60.00th=[ 5604], | 70.00th=[ 6718], 80.00th=[ 8291], 90.00th=[10683], 95.00th=[12518], | 99.00th=[17695], 99.50th=[20841], 99.90th=[24511], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 224, max= 768, per=25.03%, avg=399.23, stdev=141.71, samples=31 iops : min= 56, max= 192, avg=99.81, stdev=35.43, samples=31 lat (msec) : 2=4.65%, 4=48.76%, 10=37.30%, 20=8.89%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=11.78%, ctx=3702, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30382: Tue Apr 8 14:33:16 2025 read: IOPS=97, BW=388KiB/s (397kB/s)(6028KiB/15536msec) clat (usec): min=1604, max=23784, avg=4132.35, stdev=2843.10 lat (usec): min=1605, max=23784, avg=4133.03, stdev=2843.09 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1909], 10.00th=[ 2008], 20.00th=[ 2180], | 30.00th=[ 2442], 40.00th=[ 2671], 50.00th=[ 2999], 60.00th=[ 3523], | 70.00th=[ 4424], 80.00th=[ 5538], 90.00th=[ 8160], 95.00th=[10028], | 99.00th=[15008], 99.50th=[16909], 99.90th=[23462], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 160, max= 752, per=25.45%, avg=387.61, stdev=145.82, samples=31 iops : min= 40, max= 188, avg=96.90, stdev=36.46, samples=31 write: IOPS=100, BW=403KiB/s (413kB/s)(6260KiB/15536msec) clat (msec): min=2, max=163, avg= 5.90, stdev= 5.14 lat (msec): min=2, max=163, avg= 5.91, stdev= 5.14 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 17], 99.50th=[ 20], 99.90th=[ 26], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 184, max= 816, per=25.24%, avg=402.58, stdev=152.29, samples=31 iops : min= 46, max= 204, avg=100.65, stdev=38.07, samples=31 lat (msec) : 2=4.65%, 4=48.83%, 10=38.31%, 20=7.88%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=11.98%, ctx=3615, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30383: Tue Apr 8 14:33:16 2025 read: IOPS=96, BW=385KiB/s (394kB/s)(6052KiB/15712msec) clat (usec): min=1618, max=21449, avg=4310.70, stdev=3122.00 lat (usec): min=1618, max=21450, avg=4311.42, stdev=3122.09 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1893], 10.00th=[ 2008], 20.00th=[ 2180], | 30.00th=[ 2376], 40.00th=[ 2606], 50.00th=[ 2900], 60.00th=[ 3589], | 70.00th=[ 4555], 80.00th=[ 5800], 90.00th=[ 9110], 95.00th=[11207], | 99.00th=[15270], 99.50th=[17433], 99.90th=[20579], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 184, max= 856, per=25.16%, avg=383.13, stdev=160.46, samples=31 iops : min= 46, max= 214, avg=95.71, stdev=40.14, samples=31 write: IOPS=99, BW=397KiB/s (406kB/s)(6236KiB/15712msec) clat (msec): min=2, max=165, avg= 5.85, stdev= 5.20 lat (msec): min=2, max=165, avg= 5.85, stdev= 5.20 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 18], 99.50th=[ 21], 99.90th=[ 27], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 200, max= 752, per=24.75%, avg=394.74, stdev=153.48, samples=31 iops : min= 50, max= 188, avg=98.61, stdev=38.41, samples=31 lat (msec) : 2=4.88%, 4=48.57%, 10=36.88%, 20=9.28%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=11.99%, ctx=3627, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1524KiB/s (1560kB/s), 377KiB/s-388KiB/s (386kB/s-397kB/s), io=23.4MiB (24.6MB), run=15536-15761msec WRITE: bw=1595KiB/s (1633kB/s), 397KiB/s-403KiB/s (406kB/s-413kB/s), io=24.6MiB (25.7MB), run=15536-15761msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30415: Tue Apr 8 14:33:21 2025 read: IOPS=111, BW=1783KiB/s (1826kB/s)(5920KiB/3320msec) clat (usec): min=63, max=15338, avg=4596.67, stdev=2599.65 lat (usec): min=64, max=15339, avg=4599.46, stdev=2600.03 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 87], 10.00th=[ 2040], 20.00th=[ 2802], | 30.00th=[ 3392], 40.00th=[ 3818], 50.00th=[ 4424], 60.00th=[ 4817], | 70.00th=[ 5538], 80.00th=[ 6259], 90.00th=[ 7832], 95.00th=[ 9372], | 99.00th=[12387], 99.50th=[15270], 99.90th=[15401], 99.95th=[15401], | 99.99th=[15401] bw ( KiB/s): min= 1440, max= 1916, per=24.26%, avg=1743.33, stdev=174.18, samples=6 iops : min= 90, max= 119, avg=108.83, stdev=10.74, samples=6 write: IOPS=119, BW=1918KiB/s (1964kB/s)(6368KiB/3320msec) clat (usec): min=712, max=20774, avg=3943.66, stdev=3533.53 lat (usec): min=713, max=20775, avg=3944.56, stdev=3533.51 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 816], 10.00th=[ 848], 20.00th=[ 898], | 30.00th=[ 1074], 40.00th=[ 1778], 50.00th=[ 2769], 60.00th=[ 3884], | 70.00th=[ 5145], 80.00th=[ 6783], 90.00th=[ 8848], 95.00th=[10683], | 99.00th=[17171], 99.50th=[19268], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 1664, max= 2299, per=25.32%, avg=1913.83, stdev=250.77, samples=6 iops : min= 104, max= 143, avg=119.50, stdev=15.46, samples=6 lat (usec) : 100=3.52%, 250=1.17%, 750=0.52%, 1000=13.15% lat (msec) : 2=7.81%, 4=25.26%, 10=42.97%, 20=5.47%, 50=0.13% cpu : usr=0.18%, sys=24.10%, ctx=1509, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30416: Tue Apr 8 14:33:21 2025 read: IOPS=107, BW=1723KiB/s (1765kB/s)(5744KiB/3333msec) clat (usec): min=66, max=19449, avg=4680.46, stdev=2703.55 lat (usec): min=66, max=19450, avg=4681.09, stdev=2703.59 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 91], 10.00th=[ 2212], 20.00th=[ 2769], | 30.00th=[ 3163], 40.00th=[ 3752], 50.00th=[ 4293], 60.00th=[ 4948], | 70.00th=[ 5538], 80.00th=[ 6325], 90.00th=[ 7963], 95.00th=[ 8848], | 99.00th=[14091], 99.50th=[15664], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 1405, max= 1952, per=22.92%, avg=1646.83, stdev=196.11, samples=6 iops : min= 87, max= 122, avg=102.67, stdev=12.55, samples=6 write: IOPS=122, BW=1963KiB/s (2011kB/s)(6544KiB/3333msec) clat (usec): min=658, max=20947, avg=3951.57, stdev=3334.81 lat (usec): min=658, max=20948, avg=3952.52, stdev=3334.73 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 807], 10.00th=[ 857], 20.00th=[ 1057], | 30.00th=[ 1549], 40.00th=[ 2409], 50.00th=[ 2999], 60.00th=[ 3720], | 70.00th=[ 4621], 80.00th=[ 6521], 90.00th=[ 8717], 95.00th=[10421], | 99.00th=[14222], 99.50th=[18220], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 1536, max= 2139, per=24.53%, avg=1854.50, stdev=224.55, samples=6 iops : min= 96, max= 133, avg=115.67, stdev=13.95, samples=6 lat (usec) : 100=2.73%, 250=0.78%, 750=0.78%, 1000=8.46% lat (msec) : 2=9.64%, 4=32.55%, 10=40.23%, 20=4.69%, 50=0.13% cpu : usr=0.21%, sys=25.15%, ctx=1608, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30417: Tue Apr 8 14:33:21 2025 read: IOPS=119, BW=1910KiB/s (1956kB/s)(6256KiB/3275msec) clat (usec): min=71, max=20147, avg=4678.38, stdev=2941.48 lat (usec): min=71, max=20147, avg=4678.97, stdev=2941.50 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 85], 10.00th=[ 2089], 20.00th=[ 2835], | 30.00th=[ 3195], 40.00th=[ 3654], 50.00th=[ 4359], 60.00th=[ 4948], | 70.00th=[ 5604], 80.00th=[ 6325], 90.00th=[ 7767], 95.00th=[ 9634], | 99.00th=[15664], 99.50th=[19792], 99.90th=[20055], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 1469, max= 2336, per=26.11%, avg=1876.00, stdev=310.93, samples=6 iops : min= 91, max= 146, avg=117.00, stdev=19.58, samples=6 write: IOPS=115, BW=1842KiB/s (1886kB/s)(6032KiB/3275msec) clat (usec): min=686, max=18793, avg=3764.27, stdev=3259.28 lat (usec): min=687, max=18795, avg=3765.20, stdev=3259.33 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 816], 10.00th=[ 873], 20.00th=[ 988], | 30.00th=[ 1221], 40.00th=[ 1844], 50.00th=[ 2999], 60.00th=[ 3458], | 70.00th=[ 4228], 80.00th=[ 6390], 90.00th=[ 8717], 95.00th=[10814], | 99.00th=[16319], 99.50th=[16909], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 1568, max= 2144, per=23.76%, avg=1796.00, stdev=198.52, samples=6 iops : min= 98, max= 134, avg=112.00, stdev=12.38, samples=6 lat (usec) : 100=3.52%, 250=1.17%, 750=1.17%, 1000=9.11% lat (msec) : 2=10.16%, 4=30.86%, 10=38.93%, 20=4.95%, 50=0.13% cpu : usr=0.12%, sys=25.17%, ctx=1582, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30418: Tue Apr 8 14:33:21 2025 read: IOPS=113, BW=1820KiB/s (1863kB/s)(6032KiB/3315msec) clat (usec): min=73, max=29175, avg=4924.48, stdev=3386.56 lat (usec): min=74, max=29176, avg=4925.12, stdev=3386.55 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 88], 10.00th=[ 165], 20.00th=[ 2868], | 30.00th=[ 3359], 40.00th=[ 3949], 50.00th=[ 4555], 60.00th=[ 5080], | 70.00th=[ 5932], 80.00th=[ 6652], 90.00th=[ 8291], 95.00th=[ 9634], | 99.00th=[19530], 99.50th=[23462], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 1568, max= 1984, per=24.64%, avg=1770.67, stdev=192.71, samples=6 iops : min= 98, max= 124, avg=110.67, stdev=12.04, samples=6 write: IOPS=117, BW=1887KiB/s (1932kB/s)(6256KiB/3315msec) clat (usec): min=728, max=14914, avg=3688.39, stdev=3058.60 lat (usec): min=729, max=14916, avg=3689.27, stdev=3058.63 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 816], 10.00th=[ 848], 20.00th=[ 938], | 30.00th=[ 1254], 40.00th=[ 1778], 50.00th=[ 2769], 60.00th=[ 3589], | 70.00th=[ 4948], 80.00th=[ 6390], 90.00th=[ 8160], 95.00th=[10028], | 99.00th=[12518], 99.50th=[14615], 99.90th=[14877], 99.95th=[14877], | 99.99th=[14877] bw ( KiB/s): min= 1568, max= 2208, per=24.76%, avg=1872.00, stdev=224.23, samples=6 iops : min= 98, max= 138, avg=117.00, stdev=14.01, samples=6 lat (usec) : 100=3.39%, 250=1.69%, 500=0.13%, 750=0.26%, 1000=11.72% lat (msec) : 2=9.77%, 4=26.04%, 10=42.32%, 20=4.30%, 50=0.39% cpu : usr=0.21%, sys=24.74%, ctx=1529, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7186KiB/s (7359kB/s), 1723KiB/s-1910KiB/s (1765kB/s-1956kB/s), io=23.4MiB (24.5MB), run=3275-3333msec WRITE: bw=7561KiB/s (7742kB/s), 1842KiB/s-1963KiB/s (1886kB/s-2011kB/s), io=24.6MiB (25.8MB), run=3275-3333msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30419: Tue Apr 8 14:33:22 2025 read: IOPS=79, BW=1276KiB/s (1306kB/s)(5920KiB/4641msec) clat (usec): min=1887, max=39922, avg=5024.42, stdev=3740.46 lat (usec): min=1887, max=39924, avg=5025.15, stdev=3740.50 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2147], 10.00th=[ 2311], 20.00th=[ 2704], | 30.00th=[ 2966], 40.00th=[ 3425], 50.00th=[ 3851], 60.00th=[ 4293], | 70.00th=[ 4817], 80.00th=[ 6325], 90.00th=[ 9503], 95.00th=[12125], | 99.00th=[18482], 99.50th=[22414], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 928, max= 1888, per=24.79%, avg=1272.56, stdev=315.76, samples=9 iops : min= 58, max= 118, avg=79.44, stdev=19.70, samples=9 write: IOPS=85, BW=1372KiB/s (1405kB/s)(6368KiB/4641msec) clat (usec): min=2638, max=29273, avg=6816.00, stdev=3694.60 lat (usec): min=2639, max=29275, avg=6817.64, stdev=3694.49 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3294], 10.00th=[ 3490], 20.00th=[ 4047], | 30.00th=[ 4555], 40.00th=[ 5211], 50.00th=[ 5932], 60.00th=[ 6587], | 70.00th=[ 7373], 80.00th=[ 8717], 90.00th=[11469], 95.00th=[13566], | 99.00th=[24249], 99.50th=[27657], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 960, max= 1984, per=25.67%, avg=1386.22, stdev=333.82, samples=9 iops : min= 60, max= 124, avg=86.56, stdev=20.82, samples=9 lat (msec) : 2=1.04%, 4=34.90%, 10=51.95%, 20=11.07%, 50=1.04% cpu : usr=0.28%, sys=12.72%, ctx=958, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30420: Tue Apr 8 14:33:22 2025 read: IOPS=77, BW=1242KiB/s (1272kB/s)(5744KiB/4625msec) clat (usec): min=1815, max=25122, avg=4803.72, stdev=3320.63 lat (usec): min=1816, max=25122, avg=4804.49, stdev=3320.59 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2147], 10.00th=[ 2311], 20.00th=[ 2573], | 30.00th=[ 2900], 40.00th=[ 3326], 50.00th=[ 3687], 60.00th=[ 4146], | 70.00th=[ 4686], 80.00th=[ 5932], 90.00th=[ 9634], 95.00th=[12518], | 99.00th=[16581], 99.50th=[22414], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 736, max= 1792, per=24.37%, avg=1251.22, stdev=322.03, samples=9 iops : min= 46, max= 112, avg=78.11, stdev=20.09, samples=9 write: IOPS=88, BW=1415KiB/s (1449kB/s)(6544KiB/4625msec) clat (usec): min=2477, max=31518, avg=6913.27, stdev=3696.29 lat (usec): min=2478, max=31520, avg=6914.77, stdev=3696.31 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3261], 10.00th=[ 3490], 20.00th=[ 3982], | 30.00th=[ 4555], 40.00th=[ 5342], 50.00th=[ 5997], 60.00th=[ 6652], | 70.00th=[ 7308], 80.00th=[ 9241], 90.00th=[11863], 95.00th=[14746], | 99.00th=[19530], 99.50th=[19792], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 1120, max= 2144, per=26.19%, avg=1414.67, stdev=333.83, samples=9 iops : min= 70, max= 134, avg=88.33, stdev=20.83, samples=9 lat (msec) : 2=0.78%, 4=36.07%, 10=50.00%, 20=12.63%, 50=0.52% cpu : usr=0.15%, sys=12.67%, ctx=919, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30421: Tue Apr 8 14:33:22 2025 read: IOPS=84, BW=1347KiB/s (1379kB/s)(6256KiB/4646msec) clat (usec): min=1893, max=21550, avg=4945.60, stdev=3242.50 lat (usec): min=1894, max=21551, avg=4946.35, stdev=3242.47 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2180], 10.00th=[ 2343], 20.00th=[ 2606], | 30.00th=[ 2933], 40.00th=[ 3458], 50.00th=[ 3884], 60.00th=[ 4228], | 70.00th=[ 4883], 80.00th=[ 6456], 90.00th=[ 9634], 95.00th=[12518], | 99.00th=[16450], 99.50th=[17171], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 768, max= 1984, per=26.52%, avg=1361.33, stdev=439.22, samples=9 iops : min= 48, max= 124, avg=85.00, stdev=27.37, samples=9 write: IOPS=81, BW=1298KiB/s (1329kB/s)(6032KiB/4646msec) clat (usec): min=2592, max=35183, avg=7111.59, stdev=4118.79 lat (usec): min=2594, max=35187, avg=7113.16, stdev=4118.76 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3261], 10.00th=[ 3556], 20.00th=[ 4047], | 30.00th=[ 4621], 40.00th=[ 5276], 50.00th=[ 5997], 60.00th=[ 6915], | 70.00th=[ 7832], 80.00th=[ 9241], 90.00th=[11863], 95.00th=[14353], | 99.00th=[22676], 99.50th=[29230], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 896, max= 2048, per=23.96%, avg=1293.89, stdev=351.03, samples=9 iops : min= 56, max= 128, avg=80.78, stdev=21.96, samples=9 lat (msec) : 2=0.26%, 4=36.20%, 10=50.91%, 20=11.33%, 50=1.30% cpu : usr=0.22%, sys=12.87%, ctx=922, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30422: Tue Apr 8 14:33:22 2025 read: IOPS=80, BW=1293KiB/s (1324kB/s)(6032KiB/4665msec) clat (usec): min=1886, max=19200, avg=4861.91, stdev=3099.97 lat (usec): min=1887, max=19200, avg=4865.67, stdev=3098.04 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2180], 10.00th=[ 2343], 20.00th=[ 2704], | 30.00th=[ 3064], 40.00th=[ 3392], 50.00th=[ 3785], 60.00th=[ 4228], | 70.00th=[ 4752], 80.00th=[ 5932], 90.00th=[10290], 95.00th=[11863], | 99.00th=[15008], 99.50th=[16909], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 864, max= 1984, per=25.13%, avg=1290.33, stdev=346.85, samples=9 iops : min= 54, max= 124, avg=80.56, stdev=21.67, samples=9 write: IOPS=83, BW=1341KiB/s (1373kB/s)(6256KiB/4665msec) clat (usec): min=2523, max=27901, avg=7032.27, stdev=4003.35 lat (usec): min=2524, max=27903, avg=7033.69, stdev=4003.34 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 3097], 10.00th=[ 3556], 20.00th=[ 3949], | 30.00th=[ 4490], 40.00th=[ 5145], 50.00th=[ 6128], 60.00th=[ 6718], | 70.00th=[ 7635], 80.00th=[ 9241], 90.00th=[11994], 95.00th=[14746], | 99.00th=[22676], 99.50th=[25560], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 960, max= 2080, per=25.07%, avg=1354.22, stdev=364.65, samples=9 iops : min= 60, max= 130, avg=84.56, stdev=22.75, samples=9 lat (msec) : 2=1.04%, 4=37.37%, 10=48.44%, 20=11.98%, 50=1.17% cpu : usr=0.19%, sys=12.82%, ctx=938, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5134KiB/s (5258kB/s), 1242KiB/s-1347KiB/s (1272kB/s-1379kB/s), io=23.4MiB (24.5MB), run=4625-4665msec WRITE: bw=5402KiB/s (5532kB/s), 1298KiB/s-1415KiB/s (1329kB/s-1449kB/s), io=24.6MiB (25.8MB), run=4625-4665msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30436: Tue Apr 8 14:33:24 2025 read: IOPS=6, BW=6310KiB/s (6461kB/s)(7168KiB/1136msec) clat (usec): min=329, max=557102, avg=106574.41, stdev=200007.62 lat (usec): min=330, max=557103, avg=106575.56, stdev=200007.65 clat percentiles (usec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 19268], | 30.00th=[ 25297], 40.00th=[ 25297], 50.00th=[ 33424], 60.00th=[ 33817], | 70.00th=[ 33817], 80.00th=[ 77071], 90.00th=[557843], 95.00th=[557843], | 99.00th=[557843], 99.50th=[557843], 99.90th=[557843], 99.95th=[557843], | 99.99th=[557843] bw ( KiB/s): min= 2048, max= 8143, per=30.28%, avg=5095.50, stdev=4309.82, samples=2 iops : min= 2, max= 7, avg= 4.50, stdev= 3.54, samples=2 write: IOPS=4, BW=4507KiB/s (4615kB/s)(5120KiB/1136msec) clat (msec): min=31, max=103, avg=69.53, stdev=26.59 lat (msec): min=31, max=103, avg=69.57, stdev=26.58 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 59], 40.00th=[ 59], 50.00th=[ 78], 60.00th=[ 78], | 70.00th=[ 78], 80.00th=[ 78], 90.00th=[ 104], 95.00th=[ 104], | 99.00th=[ 104], 99.50th=[ 104], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 6107, max= 6107, per=25.92%, avg=6107.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 20=8.33%, 50=33.33%, 100=33.33%, 250=8.33%, 750=8.33% cpu : usr=0.09%, sys=25.46%, ctx=1123, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30437: Tue Apr 8 14:33:24 2025 read: IOPS=1, BW=1683KiB/s (1723kB/s)(2048KiB/1217msec) clat (usec): min=509, max=28431, avg=14470.82, stdev=19743.81 lat (usec): min=510, max=28432, avg=14471.73, stdev=19744.14 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 510], 10.00th=[ 510], 20.00th=[ 510], | 30.00th=[ 510], 40.00th=[ 510], 50.00th=[ 510], 60.00th=[28443], | 70.00th=[28443], 80.00th=[28443], 90.00th=[28443], 95.00th=[28443], | 99.00th=[28443], 99.50th=[28443], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 2035, max= 2048, per=12.13%, avg=2041.50, stdev= 9.19, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=8, BW=8414KiB/s (8616kB/s)(10.0MiB/1217msec) clat (msec): min=23, max=616, avg=117.00, stdev=179.82 lat (msec): min=23, max=616, avg=117.05, stdev=179.81 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 31], | 30.00th=[ 34], 40.00th=[ 37], 50.00th=[ 39], 60.00th=[ 46], | 70.00th=[ 120], 80.00th=[ 120], 90.00th=[ 124], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 4096, max= 8143, per=25.98%, avg=6119.50, stdev=2861.66, samples=2 iops : min= 4, max= 7, avg= 5.50, stdev= 2.12, samples=2 lat (usec) : 750=8.33% lat (msec) : 50=58.33%, 250=25.00%, 750=8.33% cpu : usr=0.00%, sys=18.91%, ctx=222, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30438: Tue Apr 8 14:33:24 2025 read: IOPS=5, BW=5265KiB/s (5391kB/s)(6144KiB/1167msec) clat (usec): min=328, max=44157, avg=32582.79, stdev=16246.89 lat (usec): min=328, max=44157, avg=32583.46, stdev=16246.91 clat percentiles (usec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[34866], | 30.00th=[34866], 40.00th=[36439], 50.00th=[36439], 60.00th=[36963], | 70.00th=[42730], 80.00th=[42730], 90.00th=[44303], 95.00th=[44303], | 99.00th=[44303], 99.50th=[44303], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 4096, max= 4096, per=24.34%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5265KiB/s (5391kB/s)(6144KiB/1167msec) clat (msec): min=24, max=644, avg=149.79, stdev=244.17 lat (msec): min=24, max=644, avg=149.84, stdev=244.18 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 33], 50.00th=[ 33], 60.00th=[ 82], | 70.00th=[ 90], 80.00th=[ 90], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 4087, max= 4096, per=17.37%, avg=4091.50, stdev= 6.36, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 lat (usec) : 500=8.33% lat (msec) : 50=66.67%, 100=16.67%, 750=8.33% cpu : usr=0.09%, sys=16.98%, ctx=249, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30439: Tue Apr 8 14:33:24 2025 read: IOPS=4, BW=4354KiB/s (4458kB/s)(5120KiB/1176msec) clat (usec): min=376, max=35377, avg=13045.59, stdev=17442.17 lat (usec): min=377, max=35378, avg=13046.30, stdev=17442.26 clat percentiles (usec): | 1.00th=[ 375], 5.00th=[ 375], 10.00th=[ 375], 20.00th=[ 375], | 30.00th=[ 437], 40.00th=[ 437], 50.00th=[ 478], 60.00th=[ 478], | 70.00th=[28443], 80.00th=[28443], 90.00th=[35390], 95.00th=[35390], | 99.00th=[35390], 99.50th=[35390], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] write: IOPS=5, BW=6095KiB/s (6242kB/s)(7168KiB/1176msec) clat (msec): min=33, max=646, avg=149.16, stdev=221.28 lat (msec): min=33, max=646, avg=149.23, stdev=221.28 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 40], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 64], 60.00th=[ 84], | 70.00th=[ 84], 80.00th=[ 123], 90.00th=[ 651], 95.00th=[ 651], | 99.00th=[ 651], 99.50th=[ 651], 99.90th=[ 651], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 2048, max= 6119, per=17.33%, avg=4083.50, stdev=2878.63, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (usec) : 500=25.00% lat (msec) : 50=33.33%, 100=25.00%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=16.94%, ctx=397, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.4MiB/s (17.2MB/s), 1683KiB/s-6310KiB/s (1723kB/s-6461kB/s), io=20.0MiB (20.0MB), run=1136-1217msec WRITE: bw=23.0MiB/s (24.1MB/s), 4507KiB/s-8414KiB/s (4615kB/s-8616kB/s), io=28.0MiB (29.4MB), run=1136-1217msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30432: Tue Apr 8 14:33:24 2025 read: IOPS=7, BW=7514KiB/s (7694kB/s)(7168KiB/954msec) clat (msec): min=16, max=618, avg=113.57, stdev=224.29 lat (msec): min=16, max=618, avg=113.58, stdev=224.29 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 17], | 30.00th=[ 17], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 22], | 70.00th=[ 22], 80.00th=[ 88], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 6144, max= 6144, per=38.85%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5367KiB/s (5496kB/s)(5120KiB/954msec) clat (usec): min=16071, max=59215, avg=30612.49, stdev=16756.61 lat (usec): min=16113, max=59314, avg=30700.32, stdev=16768.80 clat percentiles (usec): | 1.00th=[16057], 5.00th=[16057], 10.00th=[16057], 20.00th=[16057], | 30.00th=[21627], 40.00th=[21627], 50.00th=[27132], 60.00th=[27132], | 70.00th=[28967], 80.00th=[28967], 90.00th=[58983], 95.00th=[58983], | 99.00th=[58983], 99.50th=[58983], 99.90th=[58983], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 2048, max= 2048, per=9.25%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=41.67%, 50=33.33%, 100=16.67%, 750=8.33% cpu : usr=0.00%, sys=9.97%, ctx=52, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30433: Tue Apr 8 14:33:24 2025 read: IOPS=1, BW=1581KiB/s (1619kB/s)(2048KiB/1295msec) clat (usec): min=26325, max=41739, avg=34032.22, stdev=10899.06 lat (usec): min=26326, max=41741, avg=34034.08, stdev=10900.61 clat percentiles (usec): | 1.00th=[26346], 5.00th=[26346], 10.00th=[26346], 20.00th=[26346], | 30.00th=[26346], 40.00th=[26346], 50.00th=[26346], 60.00th=[41681], | 70.00th=[41681], 80.00th=[41681], 90.00th=[41681], 95.00th=[41681], | 99.00th=[41681], 99.50th=[41681], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 2048, max= 2048, per=12.95%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7907KiB/s (8097kB/s)(10.0MiB/1295msec) clat (msec): min=16, max=626, avg=117.91, stdev=183.41 lat (msec): min=16, max=626, avg=117.98, stdev=183.40 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 34], | 30.00th=[ 36], 40.00th=[ 43], 50.00th=[ 49], 60.00th=[ 52], | 70.00th=[ 117], 80.00th=[ 117], 90.00th=[ 150], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 6107, max= 6107, per=27.58%, avg=6107.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=16.67%, 250=16.67%, 750=8.33% cpu : usr=0.08%, sys=10.20%, ctx=93, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30434: Tue Apr 8 14:33:24 2025 read: IOPS=4, BW=4823KiB/s (4938kB/s)(6144KiB/1274msec) clat (msec): min=11, max=120, avg=47.45, stdev=39.42 lat (msec): min=11, max=120, avg=47.45, stdev=39.42 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 16], | 30.00th=[ 16], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 47], | 70.00th=[ 53], 80.00th=[ 53], 90.00th=[ 122], 95.00th=[ 122], | 99.00th=[ 122], 99.50th=[ 122], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 4087, max= 4087, per=25.84%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4823KiB/s (4938kB/s)(6144KiB/1274msec) clat (msec): min=23, max=607, avg=158.19, stdev=224.34 lat (msec): min=23, max=607, avg=158.43, stdev=224.22 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 55], 50.00th=[ 55], 60.00th=[ 68], | 70.00th=[ 150], 80.00th=[ 150], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 2043, max= 4096, per=13.86%, avg=3069.50, stdev=1451.69, samples=2 iops : min= 1, max= 4, avg= 2.50, stdev= 2.12, samples=2 lat (msec) : 20=16.67%, 50=33.33%, 100=25.00%, 250=16.67%, 750=8.33% cpu : usr=0.00%, sys=8.96%, ctx=113, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30435: Tue Apr 8 14:33:24 2025 read: IOPS=4, BW=4190KiB/s (4290kB/s)(5120KiB/1222msec) clat (usec): min=12334, max=60643, avg=31579.18, stdev=25126.56 lat (usec): min=12335, max=60644, avg=31579.97, stdev=25126.51 clat percentiles (usec): | 1.00th=[12387], 5.00th=[12387], 10.00th=[12387], 20.00th=[12387], | 30.00th=[13173], 40.00th=[13173], 50.00th=[14222], 60.00th=[14222], | 70.00th=[57410], 80.00th=[57410], 90.00th=[60556], 95.00th=[60556], | 99.00th=[60556], 99.50th=[60556], 99.90th=[60556], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min= 8192, max= 8192, per=51.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5866KiB/s (6007kB/s)(7168KiB/1222msec) clat (msec): min=56, max=629, avg=149.35, stdev=212.28 lat (msec): min=56, max=629, avg=149.41, stdev=212.29 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 59], | 30.00th=[ 61], 40.00th=[ 61], 50.00th=[ 65], 60.00th=[ 80], | 70.00th=[ 80], 80.00th=[ 97], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 2048, max= 6144, per=18.50%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=25.00%, 100=66.67%, 750=8.33% cpu : usr=0.00%, sys=7.86%, ctx=48, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.4MiB/s (16.2MB/s), 1581KiB/s-7514KiB/s (1619kB/s-7694kB/s), io=20.0MiB (20.0MB), run=954-1295msec WRITE: bw=21.6MiB/s (22.7MB/s), 4823KiB/s-7907KiB/s (4938kB/s-8097kB/s), io=28.0MiB (29.4MB), run=954-1295msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30444: Tue Apr 8 14:33:26 2025 read: IOPS=4, BW=19.2MiB/s (20.1MB/s)(12.0MiB/626msec) clat (msec): min=119, max=279, avg=194.74, stdev=80.52 lat (msec): min=119, max=279, avg=194.74, stdev=80.52 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 186], 50.00th=[ 186], 60.00th=[ 186], | 70.00th=[ 279], 80.00th=[ 279], 90.00th=[ 279], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min=14840, max=14840, per=56.70%, avg=14840.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=12.80%, ctx=49, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30445: Tue Apr 8 14:33:26 2025 read: IOPS=4, BW=16.7MiB/s (17.5MB/s)(4096KiB/240msec) clat (nsec): min=54596k, max=54596k, avg=54596138.00, stdev= 0.00 lat (nsec): min=54597k, max=54597k, avg=54597496.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[54789], 5.00th=[54789], 10.00th=[54789], 20.00th=[54789], | 30.00th=[54789], 40.00th=[54789], 50.00th=[54789], 60.00th=[54789], | 70.00th=[54789], 80.00th=[54789], 90.00th=[54789], 95.00th=[54789], | 99.00th=[54789], 99.50th=[54789], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] write: IOPS=8, BW=33.3MiB/s (34.0MB/s)(8192KiB/240msec) clat (msec): min=48, max=114, avg=81.58, stdev=46.50 lat (msec): min=49, max=114, avg=82.12, stdev=46.09 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 49], 10.00th=[ 49], 20.00th=[ 49], | 30.00th=[ 49], 40.00th=[ 49], 50.00th=[ 49], 60.00th=[ 115], | 70.00th=[ 115], 80.00th=[ 115], 90.00th=[ 115], 95.00th=[ 115], | 99.00th=[ 115], 99.50th=[ 115], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] lat (msec) : 50=33.33%, 100=33.33%, 250=33.33% cpu : usr=0.42%, sys=24.69%, ctx=61, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30446: Tue Apr 8 14:33:26 2025 write: IOPS=8, BW=32.8MiB/s (34.4MB/s)(12.0MiB/366msec) clat (msec): min=66, max=186, avg=110.31, stdev=66.40 lat (msec): min=66, max=187, avg=110.69, stdev=66.36 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 67], | 30.00th=[ 67], 40.00th=[ 79], 50.00th=[ 79], 60.00th=[ 79], | 70.00th=[ 188], 80.00th=[ 188], 90.00th=[ 188], 95.00th=[ 188], | 99.00th=[ 188], 99.50th=[ 188], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] lat (msec) : 100=66.67%, 250=33.33% cpu : usr=0.00%, sys=20.82%, ctx=67, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30447: Tue Apr 8 14:33:26 2025 write: IOPS=10, BW=42.6MiB/s (44.6MB/s)(12.0MiB/282msec) clat (msec): min=39, max=171, avg=87.34, stdev=72.92 lat (msec): min=39, max=171, avg=87.62, stdev=72.89 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 40], | 30.00th=[ 40], 40.00th=[ 52], 50.00th=[ 52], 60.00th=[ 52], | 70.00th=[ 171], 80.00th=[ 171], 90.00th=[ 171], 95.00th=[ 171], | 99.00th=[ 171], 99.50th=[ 171], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] lat (msec) : 50=33.33%, 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=26.69%, ctx=77, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.6MiB/s (26.8MB/s), 16.7MiB/s-19.2MiB/s (17.5MB/s-20.1MB/s), io=16.0MiB (16.8MB), run=240-626msec WRITE: bw=87.4MiB/s (91.7MB/s), 32.8MiB/s-42.6MiB/s (34.4MB/s-44.6MB/s), io=32.0MiB (33.6MB), run=240-366msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30449: Tue Apr 8 14:33:26 2025 read: IOPS=7, BW=30.5MiB/s (32.0MB/s)(12.0MiB/393msec) clat (usec): min=119548, max=120907, avg=120126.28, stdev=702.00 lat (usec): min=119550, max=120909, avg=120127.80, stdev=701.93 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 121], 10.00th=[ 121], 20.00th=[ 121], | 30.00th=[ 121], 40.00th=[ 121], 50.00th=[ 121], 60.00th=[ 121], | 70.00th=[ 122], 80.00th=[ 122], 90.00th=[ 122], 95.00th=[ 122], | 99.00th=[ 122], 99.50th=[ 122], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=35.97%, ctx=2382, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30450: Tue Apr 8 14:33:26 2025 read: IOPS=1, BW=7148KiB/s (7320kB/s)(4096KiB/573msec) clat (nsec): min=96845k, max=96845k, avg=96845359.00, stdev= 0.00 lat (nsec): min=96847k, max=96847k, avg=96846529.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[96994], 5.00th=[96994], 10.00th=[96994], 20.00th=[96994], | 30.00th=[96994], 40.00th=[96994], 50.00th=[96994], 60.00th=[96994], | 70.00th=[96994], 80.00th=[96994], 90.00th=[96994], 95.00th=[96994], | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 8192, max= 8192, per=28.65%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=13.0MiB/s (14.6MB/s)(8192KiB/573msec) clat (msec): min=143, max=303, avg=223.17, stdev=113.02 lat (msec): min=143, max=303, avg=223.48, stdev=113.04 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 144], 50.00th=[ 144], 60.00th=[ 305], | 70.00th=[ 305], 80.00th=[ 305], 90.00th=[ 305], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 8192, max= 8192, per=15.65%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.17%, sys=34.97%, ctx=911, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30451: Tue Apr 8 14:33:26 2025 write: IOPS=4, BW=19.2MiB/s (20.1MB/s)(12.0MiB/626msec) clat (msec): min=131, max=285, avg=202.62, stdev=77.52 lat (msec): min=131, max=285, avg=202.87, stdev=77.55 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 132], 10.00th=[ 132], 20.00th=[ 132], | 30.00th=[ 132], 40.00th=[ 192], 50.00th=[ 192], 60.00th=[ 192], | 70.00th=[ 284], 80.00th=[ 284], 90.00th=[ 284], 95.00th=[ 284], | 99.00th=[ 284], 99.50th=[ 284], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min=16384, max=16384, per=31.30%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.32%, sys=41.60%, ctx=29, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30452: Tue Apr 8 14:33:26 2025 write: IOPS=5, BW=21.6MiB/s (22.7MB/s)(12.0MiB/555msec) clat (msec): min=111, max=232, avg=169.59, stdev=60.64 lat (msec): min=111, max=232, avg=169.87, stdev=60.63 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 165], 50.00th=[ 165], 60.00th=[ 165], | 70.00th=[ 232], 80.00th=[ 232], 90.00th=[ 232], 95.00th=[ 232], | 99.00th=[ 232], 99.50th=[ 232], 99.90th=[ 232], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min=16384, max=16384, per=31.30%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.18%, sys=46.39%, ctx=62, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=27.9MiB/s (29.3MB/s), 7148KiB/s-30.5MiB/s (7320kB/s-32.0MB/s), io=16.0MiB (16.8MB), run=393-573msec WRITE: bw=51.1MiB/s (53.6MB/s), 13.0MiB/s-21.6MiB/s (14.6MB/s-22.7MB/s), io=32.0MiB (33.6MB), run=555-626msec