== sanity test 398b: DIO and buffer IO race ============== 07:45:56 (1749123956) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.18842 s, 42.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4306: Thu Jun 5 07:46:12 2025 read: IOPS=111, BW=447KiB/s (458kB/s)(5992KiB/13394msec) clat (usec): min=65, max=51461, avg=4893.57, stdev=3031.97 lat (usec): min=65, max=51463, avg=4894.62, stdev=3031.93 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 149], 10.00th=[ 2376], 20.00th=[ 2868], | 30.00th=[ 3326], 40.00th=[ 3752], 50.00th=[ 4359], 60.00th=[ 5080], | 70.00th=[ 5800], 80.00th=[ 6652], 90.00th=[ 8225], 95.00th=[ 9503], | 99.00th=[15401], 99.50th=[18220], 99.90th=[27919], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 216, max= 616, per=24.82%, avg=442.27, stdev=79.72, samples=26 iops : min= 54, max= 154, avg=110.46, stdev=19.97, samples=26 write: IOPS=117, BW=470KiB/s (481kB/s)(6296KiB/13394msec) clat (usec): min=454, max=157819, avg=3802.50, stdev=5138.67 lat (usec): min=454, max=157820, avg=3803.50, stdev=5138.68 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 586], 10.00th=[ 627], 20.00th=[ 783], | 30.00th=[ 1037], 40.00th=[ 2212], 50.00th=[ 2966], 60.00th=[ 3687], | 70.00th=[ 4621], 80.00th=[ 6063], 90.00th=[ 8160], 95.00th=[ 10290], | 99.00th=[ 14484], 99.50th=[ 19530], 99.90th=[ 30016], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 256, max= 616, per=24.77%, avg=462.27, stdev=83.50, samples=26 iops : min= 64, max= 154, avg=115.46, stdev=20.89, samples=26 lat (usec) : 100=1.60%, 250=0.91%, 500=0.13%, 750=9.64%, 1000=5.34% lat (msec) : 2=5.34%, 4=31.87%, 10=40.36%, 20=4.52%, 50=0.23% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.22%, sys=23.50%, ctx=5793, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4307: Thu Jun 5 07:46:12 2025 read: IOPS=110, BW=441KiB/s (452kB/s)(5940KiB/13471msec) clat (usec): min=55, max=45109, avg=4892.31, stdev=2866.95 lat (usec): min=55, max=45109, avg=4893.37, stdev=2866.62 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 1991], 10.00th=[ 2409], 20.00th=[ 2900], | 30.00th=[ 3294], 40.00th=[ 3818], 50.00th=[ 4424], 60.00th=[ 5014], | 70.00th=[ 5866], 80.00th=[ 6783], 90.00th=[ 8029], 95.00th=[ 9241], | 99.00th=[13829], 99.50th=[16909], 99.90th=[25297], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 232, max= 552, per=24.31%, avg=433.15, stdev=62.11, samples=26 iops : min= 58, max= 138, avg=108.27, stdev=15.50, samples=26 write: IOPS=117, BW=471KiB/s (483kB/s)(6348KiB/13471msec) clat (usec): min=504, max=160508, avg=3857.34, stdev=5214.37 lat (usec): min=505, max=160509, avg=3858.95, stdev=5214.66 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 586], 10.00th=[ 635], 20.00th=[ 791], | 30.00th=[ 1037], 40.00th=[ 2057], 50.00th=[ 2966], 60.00th=[ 3687], | 70.00th=[ 4686], 80.00th=[ 6259], 90.00th=[ 8586], 95.00th=[ 10552], | 99.00th=[ 15008], 99.50th=[ 18744], 99.90th=[ 22414], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 288, max= 624, per=24.55%, avg=458.12, stdev=85.77, samples=26 iops : min= 72, max= 156, avg=114.50, stdev=21.45, samples=26 lat (usec) : 100=1.14%, 250=0.98%, 500=0.10%, 750=9.31%, 1000=5.60% lat (msec) : 2=5.92%, 4=31.05%, 10=41.18%, 20=4.43%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=23.58%, ctx=5744, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4308: Thu Jun 5 07:46:12 2025 read: IOPS=112, BW=450KiB/s (461kB/s)(6028KiB/13398msec) clat (usec): min=65, max=160593, avg=4962.98, stdev=4772.60 lat (usec): min=66, max=160594, avg=4963.68, stdev=4772.62 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 1991], 10.00th=[ 2442], 20.00th=[ 2868], | 30.00th=[ 3326], 40.00th=[ 3818], 50.00th=[ 4490], 60.00th=[ 5080], | 70.00th=[ 5735], 80.00th=[ 6718], 90.00th=[ 8029], 95.00th=[ 9110], | 99.00th=[ 14353], 99.50th=[ 16188], 99.90th=[ 19792], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 280, max= 600, per=25.12%, avg=447.65, stdev=70.83, samples=26 iops : min= 70, max= 150, avg=111.88, stdev=17.69, samples=26 write: IOPS=116, BW=467KiB/s (478kB/s)(6260KiB/13398msec) clat (usec): min=470, max=31388, avg=3732.87, stdev=3353.19 lat (usec): min=471, max=31389, avg=3733.68, stdev=3353.21 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 578], 10.00th=[ 627], 20.00th=[ 758], | 30.00th=[ 988], 40.00th=[ 2245], 50.00th=[ 3032], 60.00th=[ 3720], | 70.00th=[ 4686], 80.00th=[ 6063], 90.00th=[ 8586], 95.00th=[10159], | 99.00th=[14484], 99.50th=[16057], 99.90th=[26346], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 336, max= 576, per=24.80%, avg=462.69, stdev=67.85, samples=26 iops : min= 84, max= 144, avg=115.65, stdev=16.93, samples=26 lat (usec) : 100=1.24%, 250=0.75%, 500=0.13%, 750=9.86%, 1000=5.53% lat (msec) : 2=4.56%, 4=31.32%, 10=42.02%, 20=4.46%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=23.11%, ctx=5895, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4309: Thu Jun 5 07:46:12 2025 read: IOPS=112, BW=451KiB/s (462kB/s)(6052KiB/13425msec) clat (usec): min=60, max=21776, avg=4811.95, stdev=2617.38 lat (usec): min=60, max=21776, avg=4813.30, stdev=2617.99 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 141], 10.00th=[ 2343], 20.00th=[ 2835], | 30.00th=[ 3294], 40.00th=[ 3884], 50.00th=[ 4490], 60.00th=[ 5080], | 70.00th=[ 5800], 80.00th=[ 6587], 90.00th=[ 7898], 95.00th=[ 9241], | 99.00th=[13173], 99.50th=[15664], 99.90th=[20579], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 232, max= 576, per=24.87%, avg=443.19, stdev=76.79, samples=26 iops : min= 58, max= 144, avg=110.69, stdev=19.18, samples=26 write: IOPS=116, BW=465KiB/s (476kB/s)(6236KiB/13425msec) clat (usec): min=463, max=153126, avg=3877.56, stdev=5128.19 lat (usec): min=464, max=153127, avg=3878.50, stdev=5128.24 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 570], 10.00th=[ 627], 20.00th=[ 758], | 30.00th=[ 979], 40.00th=[ 2057], 50.00th=[ 3097], 60.00th=[ 3818], | 70.00th=[ 4948], 80.00th=[ 6325], 90.00th=[ 8455], 95.00th=[ 10028], | 99.00th=[ 13960], 99.50th=[ 17433], 99.90th=[ 48497], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 272, max= 792, per=24.56%, avg=458.27, stdev=91.69, samples=26 iops : min= 68, max= 198, avg=114.46, stdev=22.95, samples=26 lat (usec) : 100=1.40%, 250=1.33%, 500=0.07%, 750=9.99%, 1000=5.66% lat (msec) : 2=4.75%, 4=29.56%, 10=42.90%, 20=4.13%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=23.17%, ctx=5828, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1782KiB/s (1825kB/s), 441KiB/s-451KiB/s (452kB/s-462kB/s), io=23.4MiB (24.6MB), run=13394-13471msec WRITE: bw=1866KiB/s (1911kB/s), 465KiB/s-471KiB/s (476kB/s-483kB/s), io=24.6MiB (25.7MB), run=13394-13471msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4302: Thu Jun 5 07:46:16 2025 read: IOPS=84, BW=338KiB/s (346kB/s)(5992KiB/17752msec) clat (usec): min=1593, max=203431, avg=4917.29, stdev=6222.23 lat (usec): min=1593, max=203432, avg=4918.00, stdev=6222.24 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1958], 10.00th=[ 2057], 20.00th=[ 2245], | 30.00th=[ 2507], 40.00th=[ 2933], 50.00th=[ 3490], 60.00th=[ 4146], | 70.00th=[ 4948], 80.00th=[ 6587], 90.00th=[ 10290], 95.00th=[ 12518], | 99.00th=[ 17433], 99.50th=[ 19792], 99.90th=[ 29230], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 152, max= 736, per=24.91%, avg=336.34, stdev=154.69, samples=35 iops : min= 38, max= 184, avg=84.00, stdev=38.70, samples=35 write: IOPS=88, BW=355KiB/s (363kB/s)(6296KiB/17752msec) clat (usec): min=2099, max=29832, avg=6542.70, stdev=4037.04 lat (usec): min=2100, max=29833, avg=6543.57, stdev=4037.15 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 2999], 10.00th=[ 3261], 20.00th=[ 3523], | 30.00th=[ 3785], 40.00th=[ 4293], 50.00th=[ 5080], 60.00th=[ 5997], | 70.00th=[ 7439], 80.00th=[ 9241], 90.00th=[11994], 95.00th=[14091], | 99.00th=[21627], 99.50th=[25035], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 160, max= 696, per=24.90%, avg=352.11, stdev=153.73, samples=35 iops : min= 40, max= 174, avg=87.94, stdev=38.48, samples=35 lat (msec) : 2=3.35%, 4=42.74%, 10=40.33%, 20=12.60%, 50=0.94% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=10.93%, ctx=3658, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4303: Thu Jun 5 07:46:16 2025 read: IOPS=84, BW=336KiB/s (344kB/s)(5940KiB/17675msec) clat (usec): min=1673, max=166542, avg=4757.27, stdev=5397.42 lat (usec): min=1673, max=166545, avg=4757.95, stdev=5397.45 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 1975], 10.00th=[ 2073], 20.00th=[ 2278], | 30.00th=[ 2573], 40.00th=[ 2933], 50.00th=[ 3425], 60.00th=[ 3949], | 70.00th=[ 4817], 80.00th=[ 6194], 90.00th=[ 9765], 95.00th=[ 12256], | 99.00th=[ 16909], 99.50th=[ 20055], 99.90th=[ 26870], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 104, max= 720, per=24.85%, avg=335.46, stdev=161.98, samples=35 iops : min= 26, max= 180, avg=83.80, stdev=40.49, samples=35 write: IOPS=89, BW=359KiB/s (368kB/s)(6348KiB/17675msec) clat (usec): min=2406, max=52487, avg=6637.58, stdev=4163.47 lat (usec): min=2407, max=52488, avg=6638.43, stdev=4163.57 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2966], 10.00th=[ 3195], 20.00th=[ 3556], | 30.00th=[ 3851], 40.00th=[ 4424], 50.00th=[ 5276], 60.00th=[ 6325], | 70.00th=[ 7570], 80.00th=[ 9372], 90.00th=[11863], 95.00th=[14091], | 99.00th=[20055], 99.50th=[25035], 99.90th=[44303], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 160, max= 696, per=25.21%, avg=356.46, stdev=150.89, samples=35 iops : min= 40, max= 174, avg=89.06, stdev=37.73, samples=35 lat (msec) : 2=2.86%, 4=43.75%, 10=40.23%, 20=12.40%, 50=0.68% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.21%, sys=11.40%, ctx=3695, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4304: Thu Jun 5 07:46:16 2025 read: IOPS=84, BW=340KiB/s (348kB/s)(6028KiB/17746msec) clat (usec): min=1641, max=29068, avg=4774.03, stdev=3604.80 lat (usec): min=1642, max=29069, avg=4774.84, stdev=3604.78 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1975], 10.00th=[ 2114], 20.00th=[ 2311], | 30.00th=[ 2573], 40.00th=[ 2900], 50.00th=[ 3425], 60.00th=[ 4047], | 70.00th=[ 4752], 80.00th=[ 6390], 90.00th=[10159], 95.00th=[12518], | 99.00th=[17695], 99.50th=[20055], 99.90th=[27132], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 120, max= 728, per=25.01%, avg=337.57, stdev=150.81, samples=35 iops : min= 30, max= 182, avg=84.37, stdev=37.72, samples=35 write: IOPS=88, BW=353KiB/s (361kB/s)(6260KiB/17746msec) clat (msec): min=2, max=169, avg= 6.69, stdev= 5.85 lat (msec): min=2, max=169, avg= 6.69, stdev= 5.85 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 15], | 99.00th=[ 22], 99.50th=[ 25], 99.90th=[ 44], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 168, max= 760, per=24.75%, avg=349.91, stdev=158.48, samples=35 iops : min= 42, max= 190, avg=87.46, stdev=39.63, samples=35 lat (msec) : 2=2.99%, 4=43.26%, 10=39.75%, 20=12.92%, 50=1.04% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=11.00%, ctx=3709, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4305: Thu Jun 5 07:46:16 2025 read: IOPS=85, BW=340KiB/s (349kB/s)(6052KiB/17779msec) clat (usec): min=1604, max=31495, avg=4757.42, stdev=3444.28 lat (usec): min=1605, max=31496, avg=4758.14, stdev=3444.31 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 1958], 10.00th=[ 2057], 20.00th=[ 2245], | 30.00th=[ 2507], 40.00th=[ 2999], 50.00th=[ 3556], 60.00th=[ 4113], | 70.00th=[ 5014], 80.00th=[ 6456], 90.00th=[ 9765], 95.00th=[12125], | 99.00th=[16909], 99.50th=[20055], 99.90th=[23462], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 119, max= 768, per=25.03%, avg=337.91, stdev=152.14, samples=35 iops : min= 29, max= 192, avg=84.37, stdev=38.11, samples=35 write: IOPS=87, BW=351KiB/s (359kB/s)(6236KiB/17779msec) clat (msec): min=2, max=156, avg= 6.73, stdev= 5.65 lat (msec): min=2, max=156, avg= 6.73, stdev= 5.65 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 23], 99.50th=[ 27], 99.90th=[ 50], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 159, max= 712, per=24.61%, avg=347.97, stdev=152.36, samples=35 iops : min= 39, max= 178, avg=86.89, stdev=38.16, samples=35 lat (msec) : 2=3.26%, 4=42.09%, 10=41.15%, 20=12.53%, 50=0.94% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=11.09%, ctx=3746, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1351KiB/s (1383kB/s), 336KiB/s-340KiB/s (344kB/s-349kB/s), io=23.4MiB (24.6MB), run=17675-17779msec WRITE: bw=1414KiB/s (1448kB/s), 351KiB/s-359KiB/s (359kB/s-368kB/s), io=24.6MiB (25.7MB), run=17675-17779msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4340: Thu Jun 5 07:46:21 2025 read: IOPS=102, BW=1638KiB/s (1677kB/s)(5920KiB/3615msec) clat (usec): min=61, max=18312, avg=5454.26, stdev=3264.96 lat (usec): min=62, max=18312, avg=5454.92, stdev=3264.96 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 100], 10.00th=[ 2540], 20.00th=[ 3064], | 30.00th=[ 3752], 40.00th=[ 4293], 50.00th=[ 4883], 60.00th=[ 5669], | 70.00th=[ 6521], 80.00th=[ 7701], 90.00th=[ 9503], 95.00th=[11207], | 99.00th=[15795], 99.50th=[17433], 99.90th=[18220], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 1440, max= 1760, per=25.30%, avg=1612.57, stdev=114.41, samples=7 iops : min= 90, max= 110, avg=100.57, stdev= 7.02, samples=7 write: IOPS=110, BW=1762KiB/s (1804kB/s)(6368KiB/3615msec) clat (usec): min=713, max=16158, avg=3898.33, stdev=3275.22 lat (usec): min=714, max=16159, avg=3899.57, stdev=3275.06 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 865], 10.00th=[ 898], 20.00th=[ 1037], | 30.00th=[ 1287], 40.00th=[ 1975], 50.00th=[ 2900], 60.00th=[ 3752], | 70.00th=[ 4752], 80.00th=[ 6783], 90.00th=[ 8979], 95.00th=[10159], | 99.00th=[14746], 99.50th=[15533], 99.90th=[16188], 99.95th=[16188], | 99.99th=[16188] bw ( KiB/s): min= 1440, max= 2171, per=26.30%, avg=1763.43, stdev=282.79, samples=7 iops : min= 90, max= 135, avg=110.00, stdev=17.65, samples=7 lat (usec) : 100=2.34%, 250=1.95%, 750=0.13%, 1000=8.33% lat (msec) : 2=12.63%, 4=24.22%, 10=43.88%, 20=6.51% cpu : usr=0.06%, sys=24.82%, ctx=1617, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4343: Thu Jun 5 07:46:21 2025 read: IOPS=98, BW=1582KiB/s (1620kB/s)(5744KiB/3630msec) clat (usec): min=56, max=20152, avg=5183.73, stdev=2860.78 lat (usec): min=57, max=20153, avg=5184.76, stdev=2861.29 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 93], 10.00th=[ 2245], 20.00th=[ 3097], | 30.00th=[ 3851], 40.00th=[ 4359], 50.00th=[ 4883], 60.00th=[ 5604], | 70.00th=[ 6456], 80.00th=[ 7373], 90.00th=[ 8455], 95.00th=[ 9896], | 99.00th=[13173], 99.50th=[15664], 99.90th=[20055], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 1280, max= 1792, per=24.59%, avg=1567.00, stdev=196.43, samples=7 iops : min= 80, max= 112, avg=97.71, stdev=12.34, samples=7 write: IOPS=112, BW=1803KiB/s (1846kB/s)(6544KiB/3630msec) clat (usec): min=648, max=19874, avg=4204.51, stdev=3523.66 lat (usec): min=649, max=19876, avg=4205.78, stdev=3523.68 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 857], 10.00th=[ 906], 20.00th=[ 1074], | 30.00th=[ 1336], 40.00th=[ 2409], 50.00th=[ 3195], 60.00th=[ 4015], | 70.00th=[ 5211], 80.00th=[ 7177], 90.00th=[ 9503], 95.00th=[10814], | 99.00th=[15401], 99.50th=[16319], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 1440, max= 2208, per=26.16%, avg=1754.14, stdev=315.50, samples=7 iops : min= 90, max= 138, avg=109.43, stdev=19.59, samples=7 lat (usec) : 100=2.60%, 250=1.30%, 500=0.26%, 750=0.39%, 1000=8.20% lat (msec) : 2=11.20%, 4=23.57%, 10=45.96%, 20=6.38%, 50=0.13% cpu : usr=0.17%, sys=25.32%, ctx=1612, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4345: Thu Jun 5 07:46:21 2025 read: IOPS=104, BW=1665KiB/s (1705kB/s)(6256KiB/3758msec) clat (usec): min=74, max=18950, avg=5232.68, stdev=3037.38 lat (usec): min=74, max=18951, avg=5233.47, stdev=3037.39 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 101], 10.00th=[ 2442], 20.00th=[ 3097], | 30.00th=[ 3589], 40.00th=[ 4113], 50.00th=[ 4621], 60.00th=[ 5407], | 70.00th=[ 6194], 80.00th=[ 7177], 90.00th=[ 9503], 95.00th=[11207], | 99.00th=[15270], 99.50th=[15926], 99.90th=[19006], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 1341, max= 1920, per=25.38%, avg=1617.29, stdev=230.38, samples=7 iops : min= 83, max= 120, avg=100.86, stdev=14.40, samples=7 write: IOPS=100, BW=1605KiB/s (1644kB/s)(6032KiB/3758msec) clat (usec): min=735, max=25607, avg=4498.52, stdev=3707.35 lat (usec): min=736, max=25609, avg=4501.41, stdev=3708.40 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 889], 10.00th=[ 963], 20.00th=[ 1270], | 30.00th=[ 1811], 40.00th=[ 2900], 50.00th=[ 3326], 60.00th=[ 4293], | 70.00th=[ 5538], 80.00th=[ 7242], 90.00th=[ 9503], 95.00th=[11600], | 99.00th=[16450], 99.50th=[18482], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 1277, max= 1760, per=23.03%, avg=1544.14, stdev=208.75, samples=7 iops : min= 79, max= 110, avg=96.29, stdev=13.10, samples=7 lat (usec) : 100=2.47%, 250=1.30%, 750=0.13%, 1000=5.47% lat (msec) : 2=10.94%, 4=26.43%, 10=45.05%, 20=8.07%, 50=0.13% cpu : usr=0.24%, sys=24.27%, ctx=1675, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4346: Thu Jun 5 07:46:21 2025 read: IOPS=102, BW=1633KiB/s (1673kB/s)(6032KiB/3693msec) clat (usec): min=63, max=16896, avg=5049.75, stdev=2914.05 lat (usec): min=64, max=16897, avg=5059.48, stdev=2917.55 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 87], 10.00th=[ 453], 20.00th=[ 3064], | 30.00th=[ 3621], 40.00th=[ 4178], 50.00th=[ 4752], 60.00th=[ 5538], | 70.00th=[ 6128], 80.00th=[ 7111], 90.00th=[ 8356], 95.00th=[10159], | 99.00th=[14746], 99.50th=[16909], 99.90th=[16909], 99.95th=[16909], | 99.99th=[16909] bw ( KiB/s): min= 1184, max= 2080, per=25.10%, avg=1599.43, stdev=274.27, samples=7 iops : min= 74, max= 130, avg=99.86, stdev=17.19, samples=7 write: IOPS=105, BW=1694KiB/s (1735kB/s)(6256KiB/3693msec) clat (usec): min=708, max=20096, avg=4484.51, stdev=3847.09 lat (usec): min=708, max=20097, avg=4489.56, stdev=3846.28 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 840], 10.00th=[ 889], 20.00th=[ 1090], | 30.00th=[ 1631], 40.00th=[ 2606], 50.00th=[ 3392], 60.00th=[ 4490], | 70.00th=[ 5473], 80.00th=[ 6980], 90.00th=[10028], 95.00th=[13435], | 99.00th=[16712], 99.50th=[18482], 99.90th=[20055], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 1376, max= 1952, per=25.15%, avg=1686.29, stdev=224.81, samples=7 iops : min= 86, max= 122, avg=105.29, stdev=13.96, samples=7 lat (usec) : 100=3.12%, 250=1.69%, 500=0.13%, 750=0.52%, 1000=7.94% lat (msec) : 2=9.51%, 4=24.35%, 10=45.18%, 20=7.42%, 50=0.13% cpu : usr=0.22%, sys=24.11%, ctx=1678, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6374KiB/s (6527kB/s), 1582KiB/s-1665KiB/s (1620kB/s-1705kB/s), io=23.4MiB (24.5MB), run=3615-3758msec WRITE: bw=6706KiB/s (6867kB/s), 1605KiB/s-1803KiB/s (1644kB/s-1846kB/s), io=24.6MiB (25.8MB), run=3615-3758msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4339: Thu Jun 5 07:46:22 2025 read: IOPS=79, BW=1267KiB/s (1297kB/s)(5920KiB/4673msec) clat (usec): min=1866, max=30535, avg=5003.01, stdev=4108.30 lat (usec): min=1867, max=30536, avg=5003.71, stdev=4108.33 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2147], 10.00th=[ 2278], 20.00th=[ 2442], | 30.00th=[ 2573], 40.00th=[ 2737], 50.00th=[ 3130], 60.00th=[ 3785], | 70.00th=[ 4817], 80.00th=[ 6980], 90.00th=[11469], 95.00th=[13829], | 99.00th=[19006], 99.50th=[21365], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 798, max= 2432, per=24.29%, avg=1222.56, stdev=541.79, samples=9 iops : min= 49, max= 152, avg=76.22, stdev=33.95, samples=9 write: IOPS=85, BW=1363KiB/s (1395kB/s)(6368KiB/4673msec) clat (usec): min=2864, max=27049, avg=6991.18, stdev=4161.66 lat (usec): min=2865, max=27051, avg=6992.65, stdev=4161.79 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3326], 10.00th=[ 3490], 20.00th=[ 3785], | 30.00th=[ 4015], 40.00th=[ 4424], 50.00th=[ 5145], 60.00th=[ 6456], | 70.00th=[ 7898], 80.00th=[10421], 90.00th=[13042], 95.00th=[15139], | 99.00th=[20841], 99.50th=[23200], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 830, max= 2592, per=24.90%, avg=1318.67, stdev=575.46, samples=9 iops : min= 51, max= 162, avg=82.22, stdev=36.14, samples=9 lat (msec) : 2=0.91%, 4=45.05%, 10=36.98%, 20=15.89%, 50=1.17% cpu : usr=0.09%, sys=11.82%, ctx=945, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4341: Thu Jun 5 07:46:22 2025 read: IOPS=75, BW=1207KiB/s (1236kB/s)(5744KiB/4758msec) clat (usec): min=1885, max=28828, avg=4924.09, stdev=4136.40 lat (usec): min=1886, max=28829, avg=4924.74, stdev=4136.42 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2180], 10.00th=[ 2278], 20.00th=[ 2376], | 30.00th=[ 2507], 40.00th=[ 2737], 50.00th=[ 3064], 60.00th=[ 3458], | 70.00th=[ 4359], 80.00th=[ 7111], 90.00th=[11600], 95.00th=[14222], | 99.00th=[19006], 99.50th=[21890], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 672, max= 2272, per=22.80%, avg=1147.78, stdev=531.34, samples=9 iops : min= 42, max= 142, avg=71.56, stdev=33.18, samples=9 write: IOPS=85, BW=1375KiB/s (1408kB/s)(6544KiB/4758msec) clat (usec): min=2741, max=29736, avg=7220.96, stdev=4815.44 lat (usec): min=2742, max=29738, avg=7222.40, stdev=4815.52 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3130], 10.00th=[ 3392], 20.00th=[ 3687], | 30.00th=[ 3916], 40.00th=[ 4293], 50.00th=[ 5211], 60.00th=[ 6259], | 70.00th=[ 8225], 80.00th=[10814], 90.00th=[13698], 95.00th=[17171], | 99.00th=[23725], 99.50th=[27919], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 832, max= 2720, per=24.42%, avg=1293.44, stdev=591.40, samples=9 iops : min= 52, max= 170, avg=80.67, stdev=36.94, samples=9 lat (msec) : 2=0.26%, 4=47.66%, 10=33.46%, 20=16.93%, 50=1.69% cpu : usr=0.17%, sys=11.58%, ctx=973, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4342: Thu Jun 5 07:46:22 2025 read: IOPS=83, BW=1336KiB/s (1368kB/s)(6256KiB/4682msec) clat (usec): min=1929, max=21993, avg=5061.13, stdev=4004.15 lat (usec): min=1930, max=21994, avg=5061.81, stdev=4004.16 clat percentiles (usec): | 1.00th=[ 2057], 5.00th=[ 2180], 10.00th=[ 2278], 20.00th=[ 2442], | 30.00th=[ 2638], 40.00th=[ 2835], 50.00th=[ 3163], 60.00th=[ 3752], | 70.00th=[ 5145], 80.00th=[ 7308], 90.00th=[10945], 95.00th=[14746], | 99.00th=[20317], 99.50th=[21365], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 798, max= 2848, per=26.13%, avg=1315.33, stdev=670.56, samples=9 iops : min= 49, max= 178, avg=82.11, stdev=42.00, samples=9 write: IOPS=80, BW=1288KiB/s (1319kB/s)(6032KiB/4682msec) clat (usec): min=2478, max=27680, avg=7103.89, stdev=4419.04 lat (usec): min=2479, max=27681, avg=7105.17, stdev=4419.16 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3163], 10.00th=[ 3490], 20.00th=[ 3687], | 30.00th=[ 3982], 40.00th=[ 4424], 50.00th=[ 5276], 60.00th=[ 6456], | 70.00th=[ 8291], 80.00th=[10290], 90.00th=[13304], 95.00th=[16188], | 99.00th=[22152], 99.50th=[25035], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 736, max= 2432, per=23.22%, avg=1229.89, stdev=503.29, samples=9 iops : min= 46, max= 152, avg=76.78, stdev=31.48, samples=9 lat (msec) : 2=0.39%, 4=45.96%, 10=36.72%, 20=15.36%, 50=1.56% cpu : usr=0.17%, sys=11.66%, ctx=960, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4344: Thu Jun 5 07:46:22 2025 read: IOPS=80, BW=1294KiB/s (1325kB/s)(6032KiB/4663msec) clat (usec): min=2005, max=20216, avg=5148.42, stdev=3988.21 lat (usec): min=2005, max=20217, avg=5149.38, stdev=3988.63 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2212], 10.00th=[ 2278], 20.00th=[ 2442], | 30.00th=[ 2638], 40.00th=[ 2900], 50.00th=[ 3326], 60.00th=[ 4178], | 70.00th=[ 5080], 80.00th=[ 6915], 90.00th=[11731], 95.00th=[14353], | 99.00th=[18744], 99.50th=[19792], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 896, max= 2592, per=24.71%, avg=1243.78, stdev=543.51, samples=9 iops : min= 56, max= 162, avg=77.56, stdev=33.99, samples=9 write: IOPS=83, BW=1342KiB/s (1374kB/s)(6256KiB/4663msec) clat (usec): min=2795, max=28074, avg=6929.02, stdev=4246.49 lat (usec): min=2796, max=28076, avg=6930.33, stdev=4246.58 clat percentiles (usec): | 1.00th=[ 2999], 5.00th=[ 3228], 10.00th=[ 3523], 20.00th=[ 3752], | 30.00th=[ 4047], 40.00th=[ 4293], 50.00th=[ 4883], 60.00th=[ 6194], | 70.00th=[ 8094], 80.00th=[10028], 90.00th=[13829], 95.00th=[15270], | 99.00th=[21103], 99.50th=[22676], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 928, max= 2464, per=24.62%, avg=1304.00, stdev=566.47, samples=9 iops : min= 58, max= 154, avg=81.33, stdev=35.32, samples=9 lat (msec) : 4=43.23%, 10=39.45%, 20=16.41%, 50=0.91% cpu : usr=0.19%, sys=11.73%, ctx=951, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5034KiB/s (5155kB/s), 1207KiB/s-1336KiB/s (1236kB/s-1368kB/s), io=23.4MiB (24.5MB), run=4663-4758msec WRITE: bw=5296KiB/s (5423kB/s), 1288KiB/s-1375KiB/s (1319kB/s-1408kB/s), io=24.6MiB (25.8MB), run=4663-4758msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4359: Thu Jun 5 07:46:24 2025 read: IOPS=15, BW=15.1MiB/s (15.8MB/s)(7168KiB/464msec) clat (usec): min=589, max=58214, avg=26407.58, stdev=20211.13 lat (usec): min=590, max=58215, avg=26408.99, stdev=20211.24 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 594], 10.00th=[ 594], 20.00th=[ 660], | 30.00th=[29492], 40.00th=[29492], 50.00th=[30016], 60.00th=[32637], | 70.00th=[32637], 80.00th=[33424], 90.00th=[58459], 95.00th=[58459], | 99.00th=[58459], 99.50th=[58459], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] write: IOPS=10, BW=10.8MiB/s (11.3MB/s)(5120KiB/464msec) clat (usec): min=31156, max=72202, avg=48480.89, stdev=16808.30 lat (usec): min=31228, max=72306, avg=48539.56, stdev=16825.55 clat percentiles (usec): | 1.00th=[31065], 5.00th=[31065], 10.00th=[31065], 20.00th=[31065], | 30.00th=[33817], 40.00th=[33817], 50.00th=[49546], 60.00th=[49546], | 70.00th=[55313], 80.00th=[55313], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] lat (usec) : 750=16.67% lat (msec) : 50=58.33%, 100=25.00% cpu : usr=0.00%, sys=38.23%, ctx=989, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4360: Thu Jun 5 07:46:24 2025 read: IOPS=1, BW=1752KiB/s (1794kB/s)(2048KiB/1169msec) clat (usec): min=36259, max=81820, avg=59040.10, stdev=32216.98 lat (usec): min=36260, max=81823, avg=59041.79, stdev=32217.60 clat percentiles (usec): | 1.00th=[36439], 5.00th=[36439], 10.00th=[36439], 20.00th=[36439], | 30.00th=[36439], 40.00th=[36439], 50.00th=[36439], 60.00th=[82314], | 70.00th=[82314], 80.00th=[82314], 90.00th=[82314], 95.00th=[82314], | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 4079, max= 4079, per=23.28%, avg=4079.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8760KiB/s (8970kB/s)(10.0MiB/1169msec) clat (msec): min=26, max=513, avg=101.10, stdev=148.07 lat (msec): min=26, max=513, avg=101.16, stdev=148.07 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 28], | 30.00th=[ 34], 40.00th=[ 40], 50.00th=[ 47], 60.00th=[ 48], | 70.00th=[ 94], 80.00th=[ 94], 90.00th=[ 125], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 3555, max=12239, per=32.20%, avg=7897.00, stdev=6140.52, samples=2 iops : min= 3, max= 11, avg= 7.00, stdev= 5.66, samples=2 lat (msec) : 50=58.33%, 100=25.00%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=24.14%, ctx=310, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4361: Thu Jun 5 07:46:24 2025 read: IOPS=9, BW=9974KiB/s (10.2MB/s)(6144KiB/616msec) clat (usec): min=473, max=66351, avg=23727.18, stdev=23713.33 lat (usec): min=474, max=66353, avg=23728.09, stdev=23713.69 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 474], 10.00th=[ 474], 20.00th=[ 6980], | 30.00th=[ 6980], 40.00th=[12256], 50.00th=[12256], 60.00th=[27919], | 70.00th=[28443], 80.00th=[28443], 90.00th=[66323], 95.00th=[66323], | 99.00th=[66323], 99.50th=[66323], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min=10240, max=10240, per=58.45%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9974KiB/s (10.2MB/s)(6144KiB/616msec) clat (msec): min=54, max=114, avg=76.90, stdev=24.18 lat (msec): min=54, max=114, avg=76.96, stdev=24.18 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 55], 10.00th=[ 55], 20.00th=[ 58], | 30.00th=[ 58], 40.00th=[ 64], 50.00th=[ 64], 60.00th=[ 72], | 70.00th=[ 99], 80.00th=[ 99], 90.00th=[ 115], 95.00th=[ 115], | 99.00th=[ 115], 99.50th=[ 115], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min=10240, max=10240, per=41.75%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 10=8.33%, 20=8.33%, 50=16.67%, 100=50.00%, 250=8.33% cpu : usr=0.00%, sys=29.43%, ctx=145, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4362: Thu Jun 5 07:46:24 2025 read: IOPS=4, BW=4719KiB/s (4832kB/s)(5120KiB/1085msec) clat (usec): min=350, max=460842, avg=97072.71, stdev=203564.12 lat (usec): min=350, max=460843, avg=97074.14, stdev=203564.30 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 351], 10.00th=[ 351], 20.00th=[ 351], | 30.00th=[ 371], 40.00th=[ 371], 50.00th=[ 1663], 60.00th=[ 1663], | 70.00th=[ 22152], 80.00th=[ 22152], 90.00th=[459277], 95.00th=[459277], | 99.00th=[459277], 99.50th=[459277], 99.90th=[459277], 99.95th=[459277], | 99.99th=[459277] bw ( KiB/s): min= 8192, max= 8192, per=46.76%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6606KiB/s (6765kB/s)(7168KiB/1085msec) clat (msec): min=29, max=132, avg=80.29, stdev=36.68 lat (msec): min=29, max=132, avg=80.35, stdev=36.69 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 44], | 30.00th=[ 70], 40.00th=[ 70], 50.00th=[ 82], 60.00th=[ 90], | 70.00th=[ 90], 80.00th=[ 116], 90.00th=[ 133], 95.00th=[ 133], | 99.00th=[ 133], 99.50th=[ 133], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 6144, max= 8192, per=29.23%, avg=7168.00, stdev=1448.15, samples=2 iops : min= 6, max= 8, avg= 7.00, stdev= 1.41, samples=2 lat (usec) : 500=16.67% lat (msec) : 2=8.33%, 50=25.00%, 100=25.00%, 250=16.67%, 500=8.33% cpu : usr=0.00%, sys=23.15%, ctx=335, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.1MiB/s (17.9MB/s), 1752KiB/s-15.1MiB/s (1794kB/s-15.8MB/s), io=20.0MiB (20.0MB), run=464-1169msec WRITE: bw=23.0MiB/s (25.1MB/s), 6606KiB/s-10.8MiB/s (6765kB/s-11.3MB/s), io=28.0MiB (29.4MB), run=464-1169msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4355: Thu Jun 5 07:46:24 2025 read: IOPS=11, BW=11.3MiB/s (11.9MB/s)(7168KiB/617msec) clat (msec): min=16, max=115, avg=46.27, stdev=36.95 lat (msec): min=16, max=115, avg=46.27, stdev=36.95 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 24], | 30.00th=[ 26], 40.00th=[ 26], 50.00th=[ 30], 60.00th=[ 35], | 70.00th=[ 35], 80.00th=[ 80], 90.00th=[ 116], 95.00th=[ 116], | 99.00th=[ 116], 99.50th=[ 116], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min=10219, max=10219, per=59.28%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8298KiB/s (8497kB/s)(5120KiB/617msec) clat (usec): min=25858, max=76447, avg=48943.32, stdev=21321.80 lat (usec): min=25934, max=76523, avg=49000.26, stdev=21329.55 clat percentiles (usec): | 1.00th=[25822], 5.00th=[25822], 10.00th=[25822], 20.00th=[25822], | 30.00th=[36439], 40.00th=[36439], 50.00th=[40109], 60.00th=[40109], | 70.00th=[65799], 80.00th=[65799], 90.00th=[76022], 95.00th=[76022], | 99.00th=[76022], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 8175, max= 8175, per=33.87%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=58.33%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=18.51%, ctx=84, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4356: Thu Jun 5 07:46:24 2025 read: IOPS=1, BW=1775KiB/s (1817kB/s)(2048KiB/1154msec) clat (usec): min=15022, max=27820, avg=21421.48, stdev=9049.65 lat (usec): min=15024, max=27821, avg=21422.95, stdev=9048.74 clat percentiles (usec): | 1.00th=[15008], 5.00th=[15008], 10.00th=[15008], 20.00th=[15008], | 30.00th=[15008], 40.00th=[15008], 50.00th=[15008], 60.00th=[27919], | 70.00th=[27919], 80.00th=[27919], 90.00th=[27919], 95.00th=[27919], | 99.00th=[27919], 99.50th=[27919], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 4096, max= 4096, per=23.76%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8873KiB/s (9086kB/s)(10.0MiB/1154msec) clat (msec): min=20, max=513, avg=109.89, stdev=146.83 lat (msec): min=20, max=513, avg=109.97, stdev=146.83 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 24], | 30.00th=[ 28], 40.00th=[ 39], 50.00th=[ 62], 60.00th=[ 79], | 70.00th=[ 107], 80.00th=[ 107], 90.00th=[ 124], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 4096, max=12288, per=33.94%, avg=8192.00, stdev=5792.62, samples=2 iops : min= 4, max= 12, avg= 8.00, stdev= 5.66, samples=2 lat (msec) : 20=8.33%, 50=41.67%, 100=16.67%, 250=25.00%, 750=8.33% cpu : usr=0.00%, sys=10.93%, ctx=133, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4357: Thu Jun 5 07:46:24 2025 read: IOPS=5, BW=5172KiB/s (5296kB/s)(6144KiB/1188msec) clat (msec): min=15, max=498, avg=131.82, stdev=183.28 lat (msec): min=15, max=498, avg=131.83, stdev=183.28 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 18], | 30.00th=[ 18], 40.00th=[ 67], 50.00th=[ 67], 60.00th=[ 94], | 70.00th=[ 100], 80.00th=[ 100], 90.00th=[ 498], 95.00th=[ 498], | 99.00th=[ 498], 99.50th=[ 498], 99.90th=[ 498], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 4087, max= 4096, per=23.73%, avg=4091.50, stdev= 6.36, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 write: IOPS=5, BW=5172KiB/s (5296kB/s)(6144KiB/1188msec) clat (usec): min=22121, max=88313, avg=63744.92, stdev=23065.03 lat (usec): min=22197, max=88390, avg=63902.95, stdev=23122.79 clat percentiles (usec): | 1.00th=[22152], 5.00th=[22152], 10.00th=[22152], 20.00th=[55313], | 30.00th=[55313], 40.00th=[67634], 50.00th=[67634], 60.00th=[72877], | 70.00th=[76022], 80.00th=[76022], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 2043, max= 8192, per=21.20%, avg=5117.50, stdev=4348.00, samples=2 iops : min= 1, max= 8, avg= 4.50, stdev= 4.95, samples=2 lat (msec) : 20=16.67%, 50=8.33%, 100=66.67%, 500=8.33% cpu : usr=0.08%, sys=13.40%, ctx=82, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4358: Thu Jun 5 07:46:24 2025 read: IOPS=4, BW=4310KiB/s (4413kB/s)(5120KiB/1188msec) clat (usec): min=13379, max=92851, avg=37451.98, stdev=32511.84 lat (usec): min=13380, max=92851, avg=37452.76, stdev=32511.89 clat percentiles (usec): | 1.00th=[13435], 5.00th=[13435], 10.00th=[13435], 20.00th=[13435], | 30.00th=[20055], 40.00th=[20055], 50.00th=[20841], 60.00th=[20841], | 70.00th=[40109], 80.00th=[40109], 90.00th=[92799], 95.00th=[92799], | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 2048, max= 6144, per=23.76%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=5, BW=6034KiB/s (6178kB/s)(7168KiB/1188msec) clat (msec): min=58, max=477, avg=140.90, stdev=149.41 lat (msec): min=58, max=477, avg=140.97, stdev=149.41 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 69], | 30.00th=[ 83], 40.00th=[ 83], 50.00th=[ 88], 60.00th=[ 102], | 70.00th=[ 102], 80.00th=[ 110], 90.00th=[ 477], 95.00th=[ 477], | 99.00th=[ 477], 99.50th=[ 477], 99.90th=[ 477], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 2048, max= 8192, per=21.21%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (msec) : 20=8.33%, 50=25.00%, 100=41.67%, 250=16.67%, 500=8.33% cpu : usr=0.00%, sys=6.40%, ctx=38, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.8MiB/s (17.7MB/s), 1775KiB/s-11.3MiB/s (1817kB/s-11.9MB/s), io=20.0MiB (20.0MB), run=617-1188msec WRITE: bw=23.6MiB/s (24.7MB/s), 5172KiB/s-8873KiB/s (5296kB/s-9086kB/s), io=28.0MiB (29.4MB), run=617-1188msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4369: Thu Jun 5 07:46:26 2025 read: IOPS=7, BW=30.6MiB/s (32.1MB/s)(12.0MiB/392msec) clat (msec): min=90, max=154, avg=117.30, stdev=33.18 lat (msec): min=90, max=154, avg=117.30, stdev=33.18 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 91], 10.00th=[ 91], 20.00th=[ 91], | 30.00th=[ 91], 40.00th=[ 107], 50.00th=[ 107], 60.00th=[ 107], | 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.26%, sys=23.53%, ctx=579, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4371: Thu Jun 5 07:46:26 2025 read: IOPS=1, BW=5885KiB/s (6026kB/s)(4096KiB/696msec) clat (nsec): min=155219k, max=155219k, avg=155218749.00, stdev= 0.00 lat (nsec): min=155220k, max=155220k, avg=155219934.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 157], 50.00th=[ 157], 60.00th=[ 157], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 8175, max= 8175, per=34.73%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=11.5MiB/s (12.1MB/s)(8192KiB/696msec) clat (msec): min=232, max=279, avg=255.83, stdev=33.15 lat (msec): min=232, max=279, avg=256.12, stdev=33.19 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 232], 10.00th=[ 232], 20.00th=[ 232], | 30.00th=[ 232], 40.00th=[ 232], 50.00th=[ 232], 60.00th=[ 279], | 70.00th=[ 279], 80.00th=[ 279], 90.00th=[ 279], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 8175, max= 8175, per=19.21%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.14%, sys=31.51%, ctx=112, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4373: Thu Jun 5 07:46:26 2025 write: IOPS=3, BW=15.6MiB/s (16.3MB/s)(12.0MiB/770msec) clat (msec): min=138, max=406, avg=247.61, stdev=140.42 lat (msec): min=138, max=406, avg=247.92, stdev=140.41 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 140], 10.00th=[ 140], 20.00th=[ 140], | 30.00th=[ 140], 40.00th=[ 199], 50.00th=[ 199], 60.00th=[ 199], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min=16384, max=16384, per=38.50%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.13%, sys=42.78%, ctx=52, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4375: Thu Jun 5 07:46:26 2025 write: IOPS=5, BW=20.7MiB/s (21.7MB/s)(12.0MiB/579msec) clat (msec): min=116, max=264, avg=189.36, stdev=73.93 lat (msec): min=116, max=264, avg=189.70, stdev=73.99 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 116], 10.00th=[ 116], 20.00th=[ 116], | 30.00th=[ 116], 40.00th=[ 188], 50.00th=[ 188], 60.00th=[ 188], | 70.00th=[ 264], 80.00th=[ 264], 90.00th=[ 264], 95.00th=[ 264], | 99.00th=[ 264], 99.50th=[ 264], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min=16384, max=16384, per=38.50%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=55.54%, ctx=39, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.0MiB/s (24.1MB/s), 5885KiB/s-30.6MiB/s (6026kB/s-32.1MB/s), io=16.0MiB (16.8MB), run=392-696msec WRITE: bw=41.6MiB/s (43.6MB/s), 11.5MiB/s-20.7MiB/s (12.1MB/s-21.7MB/s), io=32.0MiB (33.6MB), run=579-770msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4370: Thu Jun 5 07:46:26 2025 read: IOPS=3, BW=13.7MiB/s (14.3MB/s)(12.0MiB/879msec) clat (msec): min=230, max=337, avg=289.79, stdev=54.49 lat (msec): min=230, max=337, avg=289.79, stdev=54.49 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 230], 10.00th=[ 230], 20.00th=[ 230], | 30.00th=[ 230], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 8192, max= 8192, per=45.90%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=16.40%, ctx=69, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4372: Thu Jun 5 07:46:26 2025 read: IOPS=1, BW=4462KiB/s (4569kB/s)(4096KiB/918msec) clat (nsec): min=229589k, max=229589k, avg=229589244.00, stdev= 0.00 lat (nsec): min=229591k, max=229591k, avg=229590701.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 230], 10.00th=[ 230], 20.00th=[ 230], | 30.00th=[ 230], 40.00th=[ 230], 50.00th=[ 230], 60.00th=[ 230], | 70.00th=[ 230], 80.00th=[ 230], 90.00th=[ 230], 95.00th=[ 230], | 99.00th=[ 230], 99.50th=[ 230], 99.90th=[ 230], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 8000, max= 8000, per=44.83%, avg=8000.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=8924KiB/s (9138kB/s)(8192KiB/918msec) clat (msec): min=329, max=331, avg=330.51, stdev= 1.20 lat (msec): min=330, max=331, avg=330.86, stdev= 1.20 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 330], | 30.00th=[ 330], 40.00th=[ 330], 50.00th=[ 330], 60.00th=[ 334], | 70.00th=[ 334], 80.00th=[ 334], 90.00th=[ 334], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=8.93%, ctx=34, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4374: Thu Jun 5 07:46:26 2025 write: IOPS=3, BW=12.8MiB/s (13.4MB/s)(12.0MiB/941msec) clat (msec): min=247, max=328, avg=297.39, stdev=44.03 lat (msec): min=247, max=329, avg=297.79, stdev=44.00 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 247], 10.00th=[ 247], 20.00th=[ 247], | 30.00th=[ 247], 40.00th=[ 317], 50.00th=[ 317], 60.00th=[ 317], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 8192, max= 8192, per=23.85%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.11%, sys=7.66%, ctx=44, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4376: Thu Jun 5 07:46:26 2025 write: IOPS=3, BW=12.6MiB/s (13.2MB/s)(12.0MiB/954msec) clat (msec): min=73, max=516, avg=299.39, stdev=221.60 lat (msec): min=73, max=516, avg=299.74, stdev=221.61 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 73], 10.00th=[ 73], 20.00th=[ 73], | 30.00th=[ 73], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 518], 80.00th=[ 518], 90.00th=[ 518], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] lat (msec) : 100=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.31%, sys=16.05%, ctx=66, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.4MiB/s (18.3MB/s), 4462KiB/s-13.7MiB/s (4569kB/s-14.3MB/s), io=16.0MiB (16.8MB), run=879-918msec WRITE: bw=33.5MiB/s (35.2MB/s), 8924KiB/s-12.8MiB/s (9138kB/s-13.4MB/s), io=32.0MiB (33.6MB), run=918-954msec