== sanity test 398b: DIO and buffer IO race ============== 06:25:15 (1747218315) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.07177 s, 47.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10892: Wed May 14 06:25:32 2025 read: IOPS=102, BW=411KiB/s (421kB/s)(5992KiB/14565msec) clat (usec): min=55, max=42837, avg=4290.51, stdev=3484.11 lat (usec): min=56, max=42838, avg=4291.34, stdev=3484.10 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 81], 10.00th=[ 110], 20.00th=[ 2245], | 30.00th=[ 2606], 40.00th=[ 2966], 50.00th=[ 3490], 60.00th=[ 4228], | 70.00th=[ 5211], 80.00th=[ 6652], 90.00th=[ 8586], 95.00th=[10421], | 99.00th=[14615], 99.50th=[19268], 99.90th=[26084], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 264, max= 584, per=24.85%, avg=409.55, stdev=83.39, samples=29 iops : min= 66, max= 146, avg=102.31, stdev=20.95, samples=29 write: IOPS=108, BW=432KiB/s (443kB/s)(6296KiB/14565msec) clat (usec): min=471, max=164878, avg=5114.34, stdev=6167.27 lat (usec): min=472, max=164879, avg=5115.24, stdev=6167.26 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 578], 10.00th=[ 635], 20.00th=[ 816], | 30.00th=[ 1188], 40.00th=[ 2671], 50.00th=[ 4113], 60.00th=[ 5211], | 70.00th=[ 6456], 80.00th=[ 8356], 90.00th=[ 11207], 95.00th=[ 13960], | 99.00th=[ 21890], 99.50th=[ 25560], 99.90th=[ 33817], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 328, max= 632, per=24.93%, avg=430.24, stdev=66.67, samples=29 iops : min= 82, max= 158, avg=107.48, stdev=16.70, samples=29 lat (usec) : 100=3.91%, 250=3.81%, 500=0.20%, 750=8.43%, 1000=6.02% lat (msec) : 2=4.95%, 4=25.72%, 10=37.30%, 20=8.85%, 50=0.78% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=23.28%, ctx=5749, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10893: Wed May 14 06:25:32 2025 read: IOPS=102, BW=411KiB/s (420kB/s)(5940KiB/14467msec) clat (usec): min=53, max=50860, avg=4214.61, stdev=3573.96 lat (usec): min=53, max=50861, avg=4215.55, stdev=3573.93 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 82], 10.00th=[ 106], 20.00th=[ 2180], | 30.00th=[ 2638], 40.00th=[ 3032], 50.00th=[ 3490], 60.00th=[ 4047], | 70.00th=[ 5145], 80.00th=[ 6587], 90.00th=[ 8356], 95.00th=[ 9765], | 99.00th=[14746], 99.50th=[18482], 99.90th=[42730], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 280, max= 584, per=24.69%, avg=406.82, stdev=72.86, samples=28 iops : min= 70, max= 146, avg=101.68, stdev=18.22, samples=28 write: IOPS=109, BW=439KiB/s (449kB/s)(6348KiB/14467msec) clat (usec): min=480, max=170680, avg=5117.05, stdev=6160.58 lat (usec): min=481, max=170681, avg=5117.93, stdev=6160.66 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 570], 10.00th=[ 652], 20.00th=[ 824], | 30.00th=[ 1221], 40.00th=[ 2933], 50.00th=[ 4228], 60.00th=[ 5211], | 70.00th=[ 6390], 80.00th=[ 8356], 90.00th=[ 11338], 95.00th=[ 13960], | 99.00th=[ 18744], 99.50th=[ 22152], 99.90th=[ 38011], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 312, max= 568, per=24.81%, avg=428.25, stdev=57.57, samples=28 iops : min= 78, max= 142, avg=107.04, stdev=14.38, samples=28 lat (usec) : 100=4.30%, 250=3.81%, 500=0.36%, 750=8.20%, 1000=5.53% lat (msec) : 2=5.08%, 4=26.27%, 10=36.78%, 20=9.08%, 50=0.52% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.16%, sys=23.35%, ctx=5672, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10894: Wed May 14 06:25:32 2025 read: IOPS=104, BW=418KiB/s (428kB/s)(6028KiB/14412msec) clat (usec): min=50, max=173897, avg=4143.35, stdev=5439.24 lat (usec): min=50, max=173898, avg=4144.86, stdev=5439.00 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 79], 10.00th=[ 100], 20.00th=[ 1975], | 30.00th=[ 2540], 40.00th=[ 2900], 50.00th=[ 3359], 60.00th=[ 3982], | 70.00th=[ 5014], 80.00th=[ 6521], 90.00th=[ 8029], 95.00th=[ 9634], | 99.00th=[ 13698], 99.50th=[ 16909], 99.90th=[ 36439], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 288, max= 520, per=25.17%, avg=414.79, stdev=65.43, samples=28 iops : min= 72, max= 130, avg=103.68, stdev=16.33, samples=28 write: IOPS=108, BW=434KiB/s (445kB/s)(6260KiB/14412msec) clat (usec): min=480, max=60246, avg=5161.05, stdev=5110.32 lat (usec): min=481, max=60247, avg=5161.97, stdev=5110.26 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 562], 10.00th=[ 611], 20.00th=[ 816], | 30.00th=[ 1139], 40.00th=[ 2737], 50.00th=[ 4228], 60.00th=[ 5342], | 70.00th=[ 6718], 80.00th=[ 8586], 90.00th=[11469], 95.00th=[13566], | 99.00th=[19268], 99.50th=[26870], 99.90th=[58983], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 296, max= 520, per=24.81%, avg=428.21, stdev=56.03, samples=28 iops : min= 74, max= 130, avg=107.04, stdev=13.98, samples=28 lat (usec) : 100=4.95%, 250=4.10%, 500=0.26%, 750=8.98%, 1000=5.05% lat (msec) : 2=5.01%, 4=25.91%, 10=36.17%, 20=8.95%, 50=0.52% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.26%, sys=23.39%, ctx=5607, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10896: Wed May 14 06:25:32 2025 read: IOPS=105, BW=421KiB/s (431kB/s)(6052KiB/14366msec) clat (usec): min=56, max=32202, avg=4126.41, stdev=3202.75 lat (usec): min=56, max=32203, avg=4127.52, stdev=3202.61 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 82], 10.00th=[ 103], 20.00th=[ 2089], | 30.00th=[ 2573], 40.00th=[ 2933], 50.00th=[ 3425], 60.00th=[ 4015], | 70.00th=[ 5145], 80.00th=[ 6718], 90.00th=[ 8291], 95.00th=[ 9765], | 99.00th=[14353], 99.50th=[16188], 99.90th=[19268], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 224, max= 528, per=25.34%, avg=417.68, stdev=69.35, samples=28 iops : min= 56, max= 132, avg=104.39, stdev=17.31, samples=28 write: IOPS=108, BW=434KiB/s (444kB/s)(6236KiB/14366msec) clat (usec): min=471, max=198634, avg=5148.83, stdev=7079.80 lat (usec): min=472, max=198636, avg=5150.34, stdev=7079.64 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 562], 10.00th=[ 619], 20.00th=[ 791], | 30.00th=[ 1123], 40.00th=[ 2311], 50.00th=[ 3916], 60.00th=[ 4948], | 70.00th=[ 6390], 80.00th=[ 8455], 90.00th=[ 11338], 95.00th=[ 14353], | 99.00th=[ 22414], 99.50th=[ 29754], 99.90th=[ 50594], 99.95th=[198181], | 99.99th=[198181] bw ( KiB/s): min= 256, max= 568, per=24.96%, avg=430.82, stdev=75.21, samples=28 iops : min= 64, max= 142, avg=107.68, stdev=18.82, samples=28 lat (usec) : 100=4.69%, 250=3.71%, 500=0.36%, 750=8.95%, 1000=5.18% lat (msec) : 2=5.99%, 4=26.33%, 10=35.42%, 20=8.72%, 50=0.55% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.22%, sys=22.90%, ctx=5582, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1649KiB/s (1688kB/s), 411KiB/s-421KiB/s (420kB/s-431kB/s), io=23.4MiB (24.6MB), run=14366-14565msec WRITE: bw=1726KiB/s (1767kB/s), 432KiB/s-439KiB/s (443kB/s-449kB/s), io=24.6MiB (25.7MB), run=14366-14565msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10895: Wed May 14 06:25:53 2025 read: IOPS=42, BW=169KiB/s (173kB/s)(5992KiB/35403msec) clat (usec): min=1677, max=175229, avg=3988.88, stdev=5851.29 lat (usec): min=1678, max=175229, avg=3989.71, stdev=5851.28 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1926], 10.00th=[ 2008], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2638], | 70.00th=[ 3097], 80.00th=[ 4228], 90.00th=[ 8586], 95.00th=[ 10945], | 99.00th=[ 17433], 99.50th=[ 24511], 99.90th=[ 63701], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 56, max= 296, per=24.91%, avg=168.91, stdev=70.19, samples=70 iops : min= 14, max= 74, avg=42.14, stdev=17.55, samples=70 write: IOPS=44, BW=178KiB/s (182kB/s)(6296KiB/35403msec) clat (usec): min=8610, max=73864, avg=18641.73, stdev=7512.09 lat (usec): min=8611, max=73865, avg=18642.83, stdev=7512.08 clat percentiles (usec): | 1.00th=[ 9634], 5.00th=[10552], 10.00th=[11469], 20.00th=[12649], | 30.00th=[13829], 40.00th=[15401], 50.00th=[16909], 60.00th=[18220], | 70.00th=[19792], 80.00th=[22938], 90.00th=[30016], 95.00th=[34341], | 99.00th=[42730], 99.50th=[47973], 99.90th=[67634], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 64, max= 248, per=24.87%, avg=176.57, stdev=57.47, samples=70 iops : min= 16, max= 62, avg=44.06, stdev=14.39, samples=70 lat (msec) : 2=4.52%, 4=33.89%, 10=8.14%, 20=38.02%, 50=15.20% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.13%, sys=6.01%, ctx=3489, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10897: Wed May 14 06:25:53 2025 read: IOPS=42, BW=169KiB/s (173kB/s)(5940KiB/35228msec) clat (usec): min=1673, max=175342, avg=3905.12, stdev=5644.77 lat (usec): min=1674, max=175344, avg=3905.94, stdev=5644.79 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1926], 10.00th=[ 2008], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2638], | 70.00th=[ 3064], 80.00th=[ 4424], 90.00th=[ 7898], 95.00th=[ 10814], | 99.00th=[ 15926], 99.50th=[ 21103], 99.90th=[ 45876], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 39, max= 344, per=24.87%, avg=168.60, stdev=70.75, samples=70 iops : min= 9, max= 86, avg=42.09, stdev=17.72, samples=70 write: IOPS=45, BW=180KiB/s (185kB/s)(6348KiB/35228msec) clat (usec): min=8816, max=78281, avg=18491.27, stdev=7041.15 lat (usec): min=8817, max=78282, avg=18492.33, stdev=7041.14 clat percentiles (usec): | 1.00th=[ 9503], 5.00th=[10683], 10.00th=[11469], 20.00th=[12649], | 30.00th=[14091], 40.00th=[15533], 50.00th=[17171], 60.00th=[18744], | 70.00th=[20317], 80.00th=[22676], 90.00th=[28443], 95.00th=[32113], | 99.00th=[41681], 99.50th=[45876], 99.90th=[69731], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 80, max= 248, per=25.32%, avg=179.80, stdev=54.53, samples=70 iops : min= 20, max= 62, avg=44.89, stdev=13.65, samples=70 lat (msec) : 2=4.52%, 4=32.94%, 10=9.18%, 20=37.08%, 50=16.15% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.10%, sys=6.17%, ctx=3485, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10898: Wed May 14 06:25:53 2025 read: IOPS=42, BW=172KiB/s (176kB/s)(6028KiB/35055msec) clat (usec): min=1684, max=64621, avg=4112.94, stdev=4308.69 lat (usec): min=1685, max=64621, avg=4113.76, stdev=4308.67 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1926], 10.00th=[ 2024], 20.00th=[ 2147], | 30.00th=[ 2245], 40.00th=[ 2343], 50.00th=[ 2474], 60.00th=[ 2704], | 70.00th=[ 3261], 80.00th=[ 5080], 90.00th=[ 8979], 95.00th=[11469], | 99.00th=[19268], 99.50th=[27132], 99.90th=[52691], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 56, max= 344, per=25.35%, avg=171.90, stdev=66.96, samples=70 iops : min= 14, max= 86, avg=42.90, stdev=16.74, samples=70 write: IOPS=44, BW=179KiB/s (183kB/s)(6260KiB/35055msec) clat (msec): min=8, max=163, avg=18.37, stdev= 7.95 lat (msec): min=8, max=163, avg=18.37, stdev= 7.95 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 19], | 70.00th=[ 20], 80.00th=[ 23], 90.00th=[ 28], 95.00th=[ 33], | 99.00th=[ 45], 99.50th=[ 48], 99.90th=[ 58], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 64, max= 248, per=25.11%, avg=178.30, stdev=59.78, samples=70 iops : min= 16, max= 62, avg=44.50, stdev=14.95, samples=70 lat (msec) : 2=4.23%, 4=32.88%, 10=9.11%, 20=38.25%, 50=15.27% lat (msec) : 100=0.23%, 250=0.03% cpu : usr=0.10%, sys=6.12%, ctx=3444, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10899: Wed May 14 06:25:53 2025 read: IOPS=42, BW=172KiB/s (176kB/s)(6052KiB/35218msec) clat (usec): min=1664, max=39440, avg=3936.03, stdev=3722.55 lat (usec): min=1665, max=39441, avg=3936.89, stdev=3722.56 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1942], 10.00th=[ 2008], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2638], | 70.00th=[ 3195], 80.00th=[ 4555], 90.00th=[ 8455], 95.00th=[11469], | 99.00th=[18482], 99.50th=[25035], 99.90th=[36439], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 56, max= 336, per=25.25%, avg=171.19, stdev=71.79, samples=70 iops : min= 14, max= 84, avg=42.70, stdev=17.96, samples=70 write: IOPS=44, BW=177KiB/s (181kB/s)(6236KiB/35218msec) clat (msec): min=8, max=195, avg=18.72, stdev= 8.54 lat (msec): min=8, max=195, avg=18.72, stdev= 8.54 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 19], | 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 29], 95.00th=[ 34], | 99.00th=[ 41], 99.50th=[ 47], 99.90th=[ 79], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 64, max= 248, per=24.88%, avg=176.67, stdev=58.18, samples=70 iops : min= 16, max= 62, avg=44.07, stdev=14.53, samples=70 lat (msec) : 2=4.33%, 4=33.59%, 10=9.05%, 20=36.98%, 50=15.89% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.11%, sys=6.06%, ctx=3511, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=678KiB/s (695kB/s), 169KiB/s-172KiB/s (173kB/s-176kB/s), io=23.4MiB (24.6MB), run=35055-35403msec WRITE: bw=710KiB/s (727kB/s), 177KiB/s-180KiB/s (181kB/s-185kB/s), io=24.6MiB (25.7MB), run=35055-35403msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10936: Wed May 14 06:25:57 2025 read: IOPS=135, BW=2176KiB/s (2228kB/s)(5920KiB/2721msec) clat (usec): min=59, max=33416, avg=3346.71, stdev=3403.26 lat (usec): min=60, max=33417, avg=3347.67, stdev=3403.69 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 74], 10.00th=[ 85], 20.00th=[ 112], | 30.00th=[ 133], 40.00th=[ 2671], 50.00th=[ 3228], 60.00th=[ 3851], | 70.00th=[ 4621], 80.00th=[ 5735], 90.00th=[ 7046], 95.00th=[ 8291], | 99.00th=[14877], 99.50th=[21365], 99.90th=[33424], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 1792, max= 2592, per=25.27%, avg=2162.20, stdev=350.87, samples=5 iops : min= 112, max= 162, avg=135.00, stdev=21.82, samples=5 write: IOPS=146, BW=2340KiB/s (2396kB/s)(6368KiB/2721msec) clat (usec): min=621, max=16279, avg=3691.80, stdev=3448.28 lat (usec): min=622, max=16280, avg=3692.83, stdev=3448.33 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 824], 10.00th=[ 906], 20.00th=[ 1057], | 30.00th=[ 1188], 40.00th=[ 1369], 50.00th=[ 1860], 60.00th=[ 3195], | 70.00th=[ 4621], 80.00th=[ 6849], 90.00th=[ 9110], 95.00th=[11076], | 99.00th=[14615], 99.50th=[14877], 99.90th=[16319], 99.95th=[16319], | 99.99th=[16319] bw ( KiB/s): min= 1856, max= 3033, per=26.07%, avg=2347.40, stdev=525.76, samples=5 iops : min= 116, max= 189, avg=146.60, stdev=32.68, samples=5 lat (usec) : 100=7.42%, 250=9.11%, 500=0.52%, 750=1.30%, 1000=7.94% lat (msec) : 2=17.84%, 4=19.79%, 10=31.51%, 20=4.30%, 50=0.26% cpu : usr=0.07%, sys=30.70%, ctx=1447, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10937: Wed May 14 06:25:57 2025 read: IOPS=132, BW=2119KiB/s (2170kB/s)(5744KiB/2711msec) clat (usec): min=60, max=19510, avg=3496.94, stdev=2840.33 lat (usec): min=60, max=19511, avg=3497.61, stdev=2840.34 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 82], 10.00th=[ 94], 20.00th=[ 124], | 30.00th=[ 2376], 40.00th=[ 3032], 50.00th=[ 3359], 60.00th=[ 3851], | 70.00th=[ 4621], 80.00th=[ 5604], 90.00th=[ 7242], 95.00th=[ 8717], | 99.00th=[11207], 99.50th=[13566], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 1568, max= 2560, per=25.06%, avg=2144.00, stdev=411.67, samples=5 iops : min= 98, max= 160, avg=134.00, stdev=25.73, samples=5 write: IOPS=150, BW=2414KiB/s (2472kB/s)(6544KiB/2711msec) clat (usec): min=704, max=16136, avg=3473.98, stdev=3241.74 lat (usec): min=705, max=16138, avg=3474.96, stdev=3241.75 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 857], 10.00th=[ 955], 20.00th=[ 1106], | 30.00th=[ 1254], 40.00th=[ 1450], 50.00th=[ 1860], 60.00th=[ 2835], | 70.00th=[ 3982], 80.00th=[ 5866], 90.00th=[ 8717], 95.00th=[10290], | 99.00th=[14353], 99.50th=[15008], 99.90th=[16188], 99.95th=[16188], | 99.99th=[16188] bw ( KiB/s): min= 2240, max= 2624, per=26.94%, avg=2425.60, stdev=160.64, samples=5 iops : min= 140, max= 164, avg=151.60, stdev=10.04, samples=5 lat (usec) : 100=5.34%, 250=7.29%, 500=0.26%, 750=0.26%, 1000=6.38% lat (msec) : 2=21.87%, 4=25.39%, 10=29.04%, 20=4.17% cpu : usr=0.07%, sys=31.77%, ctx=1481, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10938: Wed May 14 06:25:57 2025 read: IOPS=139, BW=2235KiB/s (2289kB/s)(6256KiB/2799msec) clat (usec): min=59, max=34615, avg=3509.35, stdev=3663.14 lat (usec): min=59, max=34616, avg=3510.07, stdev=3663.15 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 72], 10.00th=[ 84], 20.00th=[ 109], | 30.00th=[ 143], 40.00th=[ 2802], 50.00th=[ 3326], 60.00th=[ 3982], | 70.00th=[ 4752], 80.00th=[ 6128], 90.00th=[ 7570], 95.00th=[ 8586], | 99.00th=[15926], 99.50th=[26346], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 1824, max= 2624, per=26.32%, avg=2251.80, stdev=306.97, samples=5 iops : min= 114, max= 164, avg=140.60, stdev=19.13, samples=5 write: IOPS=134, BW=2155KiB/s (2207kB/s)(6032KiB/2799msec) clat (usec): min=679, max=18550, avg=3661.23, stdev=3362.78 lat (usec): min=680, max=18552, avg=3663.10, stdev=3362.22 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 832], 10.00th=[ 930], 20.00th=[ 1123], | 30.00th=[ 1287], 40.00th=[ 1483], 50.00th=[ 2040], 60.00th=[ 3163], | 70.00th=[ 4490], 80.00th=[ 6390], 90.00th=[ 8586], 95.00th=[10421], | 99.00th=[16057], 99.50th=[17433], 99.90th=[18482], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 1824, max= 2459, per=23.95%, avg=2155.80, stdev=226.40, samples=5 iops : min= 114, max= 153, avg=134.60, stdev=13.92, samples=5 lat (usec) : 100=8.59%, 250=8.20%, 500=0.78%, 750=1.17%, 1000=5.99% lat (msec) : 2=17.97%, 4=20.57%, 10=32.94%, 20=3.39%, 50=0.39% cpu : usr=0.11%, sys=29.44%, ctx=1515, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10939: Wed May 14 06:25:57 2025 read: IOPS=135, BW=2171KiB/s (2223kB/s)(6032KiB/2778msec) clat (usec): min=57, max=19430, avg=3381.41, stdev=3258.30 lat (usec): min=57, max=19431, avg=3384.21, stdev=3256.68 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 73], 10.00th=[ 82], 20.00th=[ 109], | 30.00th=[ 128], 40.00th=[ 2343], 50.00th=[ 3359], 60.00th=[ 3982], | 70.00th=[ 4948], 80.00th=[ 5997], 90.00th=[ 7898], 95.00th=[ 8848], | 99.00th=[11600], 99.50th=[19268], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 1632, max= 2592, per=25.21%, avg=2156.80, stdev=361.05, samples=5 iops : min= 102, max= 162, avg=134.80, stdev=22.57, samples=5 write: IOPS=140, BW=2252KiB/s (2306kB/s)(6256KiB/2778msec) clat (usec): min=656, max=34398, avg=3677.38, stdev=3717.37 lat (usec): min=656, max=34399, avg=3678.87, stdev=3717.13 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 807], 10.00th=[ 881], 20.00th=[ 1057], | 30.00th=[ 1221], 40.00th=[ 1418], 50.00th=[ 1844], 60.00th=[ 3130], | 70.00th=[ 4686], 80.00th=[ 6259], 90.00th=[ 8225], 95.00th=[10814], | 99.00th=[14615], 99.50th=[22414], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 1824, max= 2912, per=25.24%, avg=2272.00, stdev=459.29, samples=5 iops : min= 114, max= 182, avg=142.00, stdev=28.71, samples=5 lat (usec) : 100=8.07%, 250=9.77%, 500=0.65%, 750=1.17%, 1000=8.07% lat (msec) : 2=17.58%, 4=17.97%, 10=31.90%, 20=4.56%, 50=0.26% cpu : usr=0.25%, sys=29.96%, ctx=1420, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8557KiB/s (8763kB/s), 2119KiB/s-2235KiB/s (2170kB/s-2289kB/s), io=23.4MiB (24.5MB), run=2711-2799msec WRITE: bw=9003KiB/s (9219kB/s), 2155KiB/s-2414KiB/s (2207kB/s-2472kB/s), io=24.6MiB (25.8MB), run=2711-2799msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10940: Wed May 14 06:26:03 2025 read: IOPS=43, BW=700KiB/s (717kB/s)(5920KiB/8455msec) clat (usec): min=1909, max=24360, avg=3685.10, stdev=3034.97 lat (usec): min=1910, max=24362, avg=3685.94, stdev=3034.97 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2073], 10.00th=[ 2114], 20.00th=[ 2278], | 30.00th=[ 2376], 40.00th=[ 2474], 50.00th=[ 2573], 60.00th=[ 2769], | 70.00th=[ 2966], 80.00th=[ 3785], 90.00th=[ 6980], 95.00th=[ 9503], | 99.00th=[19006], 99.50th=[20841], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 288, max= 1056, per=24.61%, avg=681.75, stdev=225.24, samples=16 iops : min= 18, max= 66, avg=42.50, stdev=14.06, samples=16 write: IOPS=47, BW=753KiB/s (771kB/s)(6368KiB/8455msec) clat (usec): min=8961, max=56031, avg=17782.87, stdev=6871.70 lat (usec): min=8962, max=56032, avg=17784.37, stdev=6871.73 clat percentiles (usec): | 1.00th=[ 9110], 5.00th=[10552], 10.00th=[11207], 20.00th=[12387], | 30.00th=[13566], 40.00th=[15008], 50.00th=[16188], 60.00th=[17433], | 70.00th=[19006], 80.00th=[21365], 90.00th=[26608], 95.00th=[32375], | 99.00th=[40633], 99.50th=[48497], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 416, max= 1024, per=25.38%, avg=739.81, stdev=221.02, samples=16 iops : min= 26, max= 64, avg=46.13, stdev=13.85, samples=16 lat (msec) : 2=0.78%, 4=38.80%, 10=7.94%, 20=39.58%, 50=12.76% lat (msec) : 100=0.13% cpu : usr=0.12%, sys=7.16%, ctx=880, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10941: Wed May 14 06:26:03 2025 read: IOPS=41, BW=665KiB/s (680kB/s)(5744KiB/8644msec) clat (usec): min=1880, max=35761, avg=3937.65, stdev=3669.43 lat (usec): min=1880, max=35762, avg=3938.46, stdev=3669.44 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2089], 10.00th=[ 2180], 20.00th=[ 2278], | 30.00th=[ 2409], 40.00th=[ 2507], 50.00th=[ 2606], 60.00th=[ 2802], | 70.00th=[ 3130], 80.00th=[ 4047], 90.00th=[ 7701], 95.00th=[11863], | 99.00th=[19268], 99.50th=[23200], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 288, max= 1184, per=23.91%, avg=662.24, stdev=269.75, samples=17 iops : min= 18, max= 74, avg=41.24, stdev=16.88, samples=17 write: IOPS=47, BW=757KiB/s (775kB/s)(6544KiB/8644msec) clat (usec): min=8774, max=45038, avg=17505.95, stdev=5976.66 lat (usec): min=8776, max=45040, avg=17507.66, stdev=5976.74 clat percentiles (usec): | 1.00th=[ 9896], 5.00th=[10814], 10.00th=[11338], 20.00th=[12387], | 30.00th=[13566], 40.00th=[15008], 50.00th=[16319], 60.00th=[17695], | 70.00th=[19006], 80.00th=[21103], 90.00th=[26608], 95.00th=[30278], | 99.00th=[36439], 99.50th=[37487], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 447, max= 992, per=25.69%, avg=748.76, stdev=213.56, samples=17 iops : min= 27, max= 62, avg=46.65, stdev=13.37, samples=17 lat (msec) : 2=0.65%, 4=36.59%, 10=7.16%, 20=42.32%, 50=13.28% cpu : usr=0.09%, sys=7.14%, ctx=899, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10942: Wed May 14 06:26:03 2025 read: IOPS=47, BW=756KiB/s (774kB/s)(6256KiB/8272msec) clat (usec): min=1785, max=28908, avg=3810.77, stdev=3208.63 lat (usec): min=1786, max=28909, avg=3811.58, stdev=3208.64 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2073], 10.00th=[ 2147], 20.00th=[ 2278], | 30.00th=[ 2376], 40.00th=[ 2442], 50.00th=[ 2573], 60.00th=[ 2769], | 70.00th=[ 3130], 80.00th=[ 4047], 90.00th=[ 7767], 95.00th=[10028], | 99.00th=[18744], 99.50th=[21365], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 224, max= 1312, per=27.50%, avg=761.81, stdev=334.99, samples=16 iops : min= 14, max= 82, avg=47.56, stdev=20.86, samples=16 write: IOPS=45, BW=729KiB/s (747kB/s)(6032KiB/8272msec) clat (usec): min=8978, max=46025, avg=17801.28, stdev=5910.01 lat (usec): min=8979, max=46026, avg=17802.82, stdev=5910.08 clat percentiles (usec): | 1.00th=[ 9765], 5.00th=[10814], 10.00th=[11469], 20.00th=[13042], | 30.00th=[14222], 40.00th=[15664], 50.00th=[16712], 60.00th=[17957], | 70.00th=[19268], 80.00th=[21103], 90.00th=[25560], 95.00th=[30016], | 99.00th=[36963], 99.50th=[42730], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 448, max= 960, per=24.76%, avg=721.88, stdev=186.11, samples=16 iops : min= 28, max= 60, avg=45.06, stdev=11.59, samples=16 lat (msec) : 2=1.30%, 4=39.06%, 10=8.59%, 20=38.15%, 50=12.89% cpu : usr=0.06%, sys=7.52%, ctx=882, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10943: Wed May 14 06:26:03 2025 read: IOPS=44, BW=711KiB/s (728kB/s)(6032KiB/8483msec) clat (usec): min=1892, max=35101, avg=3929.16, stdev=3644.39 lat (usec): min=1893, max=35101, avg=3930.00, stdev=3644.39 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2089], 10.00th=[ 2147], 20.00th=[ 2278], | 30.00th=[ 2376], 40.00th=[ 2474], 50.00th=[ 2638], 60.00th=[ 2802], | 70.00th=[ 3032], 80.00th=[ 4228], 90.00th=[ 8455], 95.00th=[11338], | 99.00th=[19268], 99.50th=[27657], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 319, max= 1248, per=24.98%, avg=691.81, stdev=248.79, samples=16 iops : min= 19, max= 78, avg=43.12, stdev=15.66, samples=16 write: IOPS=46, BW=737KiB/s (755kB/s)(6256KiB/8483msec) clat (usec): min=9073, max=41360, avg=17747.54, stdev=5749.34 lat (usec): min=9074, max=41362, avg=17749.11, stdev=5749.48 clat percentiles (usec): | 1.00th=[10028], 5.00th=[10945], 10.00th=[11600], 20.00th=[12911], | 30.00th=[13960], 40.00th=[15664], 50.00th=[16909], 60.00th=[17957], | 70.00th=[19006], 80.00th=[21103], 90.00th=[26346], 95.00th=[30540], | 99.00th=[36963], 99.50th=[39060], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 384, max= 928, per=24.90%, avg=725.75, stdev=199.48, samples=16 iops : min= 24, max= 58, avg=45.25, stdev=12.58, samples=16 lat (msec) : 2=0.91%, 4=38.15%, 10=7.29%, 20=40.89%, 50=12.76% cpu : usr=0.11%, sys=7.17%, ctx=894, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2771KiB/s (2837kB/s), 665KiB/s-756KiB/s (680kB/s-774kB/s), io=23.4MiB (24.5MB), run=8272-8644msec WRITE: bw=2915KiB/s (2985kB/s), 729KiB/s-757KiB/s (747kB/s-775kB/s), io=24.6MiB (25.8MB), run=8272-8644msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10958: Wed May 14 06:26:05 2025 read: IOPS=8, BW=8849KiB/s (9062kB/s)(7168KiB/810msec) clat (usec): min=465, max=430193, avg=85138.27, stdev=152657.17 lat (usec): min=465, max=430194, avg=85139.44, stdev=152657.22 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 465], 10.00th=[ 465], 20.00th=[ 29230], | 30.00th=[ 32113], 40.00th=[ 32113], 50.00th=[ 33162], 60.00th=[ 33424], | 70.00th=[ 33424], 80.00th=[ 37487], 90.00th=[429917], 95.00th=[429917], | 99.00th=[429917], 99.50th=[429917], 99.90th=[429917], 99.95th=[429917], | 99.99th=[429917] bw ( KiB/s): min=10240, max=10240, per=59.35%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6321KiB/s (6473kB/s)(5120KiB/810msec) clat (usec): min=26482, max=82134, avg=40600.40, stdev=23426.46 lat (usec): min=26511, max=82153, avg=40638.19, stdev=23417.19 clat percentiles (usec): | 1.00th=[26608], 5.00th=[26608], 10.00th=[26608], 20.00th=[26608], | 30.00th=[28443], 40.00th=[28443], 50.00th=[31327], 60.00th=[31327], | 70.00th=[34866], 80.00th=[34866], 90.00th=[82314], 95.00th=[82314], | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 8192, max= 8192, per=33.91%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=75.00%, 100=8.33%, 500=8.33% cpu : usr=0.00%, sys=25.96%, ctx=617, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10959: Wed May 14 06:26:05 2025 read: IOPS=2, BW=2554KiB/s (2615kB/s)(2048KiB/802msec) clat (usec): min=25765, max=37551, avg=31658.21, stdev=8334.10 lat (usec): min=25766, max=37552, avg=31659.60, stdev=8333.42 clat percentiles (usec): | 1.00th=[25822], 5.00th=[25822], 10.00th=[25822], 20.00th=[25822], | 30.00th=[25822], 40.00th=[25822], 50.00th=[25822], 60.00th=[37487], | 70.00th=[37487], 80.00th=[37487], 90.00th=[37487], 95.00th=[37487], | 99.00th=[37487], 99.50th=[37487], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 4096, max= 4096, per=23.74%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.5MiB/s (13.1MB/s)(10.0MiB/802msec) clat (msec): min=18, max=356, avg=70.16, stdev=103.41 lat (msec): min=19, max=356, avg=70.24, stdev=103.40 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 24], | 30.00th=[ 25], 40.00th=[ 27], 50.00th=[ 31], 60.00th=[ 31], | 70.00th=[ 57], 80.00th=[ 57], 90.00th=[ 100], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min=18432, max=18432, per=76.31%, avg=18432.00, stdev= 0.00, samples=1 iops : min= 18, max= 18, avg=18.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=66.67%, 100=16.67%, 500=8.33% cpu : usr=0.12%, sys=29.09%, ctx=146, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10961: Wed May 14 06:26:05 2025 read: IOPS=6, BW=7161KiB/s (7333kB/s)(6144KiB/858msec) clat (usec): min=512, max=460238, avg=92104.65, stdev=180990.52 lat (usec): min=513, max=460239, avg=92105.92, stdev=180990.27 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 515], 10.00th=[ 515], 20.00th=[ 537], | 30.00th=[ 537], 40.00th=[ 23200], 50.00th=[ 23200], 60.00th=[ 32375], | 70.00th=[ 35914], 80.00th=[ 35914], 90.00th=[459277], 95.00th=[459277], | 99.00th=[459277], 99.50th=[459277], 99.90th=[459277], 99.95th=[459277], | 99.99th=[459277] bw ( KiB/s): min= 4096, max= 4096, per=23.74%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=6, BW=7161KiB/s (7333kB/s)(6144KiB/858msec) clat (usec): min=18509, max=90323, avg=49547.69, stdev=27031.82 lat (usec): min=18527, max=90371, avg=49593.42, stdev=27033.86 clat percentiles (usec): | 1.00th=[18482], 5.00th=[18482], 10.00th=[18482], 20.00th=[30016], | 30.00th=[30016], 40.00th=[36439], 50.00th=[36439], 60.00th=[51119], | 70.00th=[70779], 80.00th=[70779], 90.00th=[90702], 95.00th=[90702], | 99.00th=[90702], 99.50th=[90702], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 8192, max= 8192, per=33.91%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 20=8.33%, 50=41.67%, 100=25.00%, 500=8.33% cpu : usr=0.00%, sys=23.57%, ctx=580, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10963: Wed May 14 06:26:05 2025 read: IOPS=4, BW=4313KiB/s (4417kB/s)(5120KiB/1187msec) clat (usec): min=25326, max=39465, avg=30718.44, stdev=5677.97 lat (usec): min=25328, max=39466, avg=30719.73, stdev=5677.53 clat percentiles (usec): | 1.00th=[25297], 5.00th=[25297], 10.00th=[25297], 20.00th=[25297], | 30.00th=[26084], 40.00th=[26084], 50.00th=[30802], 60.00th=[30802], | 70.00th=[32113], 80.00th=[32113], 90.00th=[39584], 95.00th=[39584], | 99.00th=[39584], 99.50th=[39584], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 8192, max= 8192, per=47.48%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=6039KiB/s (6184kB/s)(7168KiB/1187msec) clat (msec): min=34, max=450, avg=138.41, stdev=153.73 lat (msec): min=34, max=450, avg=138.77, stdev=153.57 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 51], | 30.00th=[ 62], 40.00th=[ 62], 50.00th=[ 67], 60.00th=[ 70], | 70.00th=[ 70], 80.00th=[ 239], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 4096, max= 4096, per=16.96%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 lat (msec) : 50=50.00%, 100=33.33%, 250=8.33%, 500=8.33% cpu : usr=0.17%, sys=22.85%, ctx=874, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.8MiB/s (17.7MB/s), 2554KiB/s-8849KiB/s (2615kB/s-9062kB/s), io=20.0MiB (20.0MB), run=802-1187msec WRITE: bw=23.6MiB/s (24.7MB/s), 6039KiB/s-12.5MiB/s (6184kB/s-13.1MB/s), io=28.0MiB (29.4MB), run=802-1187msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10962: Wed May 14 06:26:05 2025 read: IOPS=6, BW=6899KiB/s (7065kB/s)(7168KiB/1039msec) clat (msec): min=14, max=161, avg=48.03, stdev=52.00 lat (msec): min=14, max=161, avg=48.03, stdev=52.00 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 32], 60.00th=[ 43], | 70.00th=[ 43], 80.00th=[ 52], 90.00th=[ 163], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 6144, max= 6144, per=36.60%, avg=6144.00, stdev= 0.00, samples=2 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4928KiB/s (5046kB/s)(5120KiB/1039msec) clat (msec): min=30, max=508, avg=139.53, stdev=206.83 lat (msec): min=30, max=508, avg=139.60, stdev=206.84 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 48], 60.00th=[ 48], | 70.00th=[ 77], 80.00th=[ 77], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 8192, max= 8192, per=34.86%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=41.67%, 100=16.67%, 250=8.33%, 750=8.33% cpu : usr=0.10%, sys=10.21%, ctx=70, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10964: Wed May 14 06:26:05 2025 read: IOPS=1, BW=1843KiB/s (1888kB/s)(2048KiB/1111msec) clat (usec): min=37370, max=85053, avg=61211.89, stdev=33716.66 lat (usec): min=37371, max=85054, avg=61213.06, stdev=33716.76 clat percentiles (usec): | 1.00th=[37487], 5.00th=[37487], 10.00th=[37487], 20.00th=[37487], | 30.00th=[37487], 40.00th=[37487], 50.00th=[37487], 60.00th=[85459], | 70.00th=[85459], 80.00th=[85459], 90.00th=[85459], 95.00th=[85459], | 99.00th=[85459], 99.50th=[85459], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 2048, max= 2048, per=12.20%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=9, BW=9217KiB/s (9438kB/s)(10.0MiB/1111msec) clat (msec): min=22, max=519, avg=98.21, stdev=154.18 lat (msec): min=22, max=519, avg=98.26, stdev=154.18 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 23], | 30.00th=[ 25], 40.00th=[ 26], 50.00th=[ 27], 60.00th=[ 38], | 70.00th=[ 95], 80.00th=[ 95], 90.00th=[ 159], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 2048, max=10240, per=26.14%, avg=6144.00, stdev=5792.62, samples=2 iops : min= 2, max= 10, avg= 6.00, stdev= 5.66, samples=2 lat (msec) : 50=58.33%, 100=25.00%, 250=8.33%, 750=8.33% cpu : usr=0.09%, sys=7.84%, ctx=63, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10965: Wed May 14 06:26:05 2025 read: IOPS=5, BW=5747KiB/s (5885kB/s)(6144KiB/1069msec) clat (msec): min=15, max=443, avg=101.98, stdev=168.56 lat (msec): min=15, max=443, avg=101.98, stdev=168.56 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 21], | 30.00th=[ 21], 40.00th=[ 24], 50.00th=[ 24], 60.00th=[ 40], | 70.00th=[ 70], 80.00th=[ 70], 90.00th=[ 443], 95.00th=[ 443], | 99.00th=[ 443], 99.50th=[ 443], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 4096, max= 6144, per=30.50%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=5, BW=5747KiB/s (5885kB/s)(6144KiB/1069msec) clat (msec): min=21, max=175, avg=69.66, stdev=55.98 lat (msec): min=21, max=176, avg=69.72, stdev=55.98 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 55], 50.00th=[ 55], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 71], 90.00th=[ 176], 95.00th=[ 176], | 99.00th=[ 176], 99.50th=[ 176], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 2048, max= 8192, per=21.79%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (msec) : 20=8.33%, 50=41.67%, 100=33.33%, 250=8.33%, 500=8.33% cpu : usr=0.09%, sys=7.87%, ctx=48, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10966: Wed May 14 06:26:05 2025 read: IOPS=4, BW=4197KiB/s (4297kB/s)(5120KiB/1220msec) clat (usec): min=25841, max=82486, avg=43071.71, stdev=22841.85 lat (usec): min=25842, max=82487, avg=43072.90, stdev=22842.10 clat percentiles (usec): | 1.00th=[25822], 5.00th=[25822], 10.00th=[25822], 20.00th=[25822], | 30.00th=[29492], 40.00th=[29492], 50.00th=[35914], 60.00th=[35914], | 70.00th=[41681], 80.00th=[41681], 90.00th=[82314], 95.00th=[82314], | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 8192, max= 8192, per=48.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5875KiB/s (6016kB/s)(7168KiB/1220msec) clat (msec): min=28, max=532, avg=139.77, stdev=176.86 lat (msec): min=28, max=532, avg=139.84, stdev=176.86 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 45], | 30.00th=[ 67], 40.00th=[ 67], 50.00th=[ 79], 60.00th=[ 84], | 70.00th=[ 84], 80.00th=[ 146], 90.00th=[ 531], 95.00th=[ 531], | 99.00th=[ 531], 99.50th=[ 531], 99.90th=[ 531], 99.95th=[ 531], | 99.99th=[ 531] bw ( KiB/s): min= 4096, max= 6144, per=21.79%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 50=50.00%, 100=33.33%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=6.15%, ctx=46, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.4MiB/s (17.2MB/s), 1843KiB/s-6899KiB/s (1888kB/s-7065kB/s), io=20.0MiB (20.0MB), run=1039-1220msec WRITE: bw=22.0MiB/s (24.1MB/s), 4928KiB/s-9217KiB/s (5046kB/s-9438kB/s), io=28.0MiB (29.4MB), run=1039-1220msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10974: Wed May 14 06:26:07 2025 read: IOPS=2, BW=11.7MiB/s (12.3MB/s)(12.0MiB/1022msec) clat (msec): min=141, max=689, avg=338.07, stdev=304.97 lat (msec): min=141, max=689, avg=338.07, stdev=304.97 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 142], 10.00th=[ 142], 20.00th=[ 142], | 30.00th=[ 142], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 184], | 70.00th=[ 693], 80.00th=[ 693], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=7.05%, ctx=42, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10976: Wed May 14 06:26:07 2025 read: IOPS=0, BW=3131KiB/s (3207kB/s)(4096KiB/1308msec) clat (nsec): min=729316k, max=729316k, avg=729315504.00, stdev= 0.00 lat (nsec): min=729319k, max=729319k, avg=729318537.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 726], 5.00th=[ 726], 10.00th=[ 726], 20.00th=[ 726], | 30.00th=[ 726], 40.00th=[ 726], 50.00th=[ 726], 60.00th=[ 726], | 70.00th=[ 726], 80.00th=[ 726], 90.00th=[ 726], 95.00th=[ 726], | 99.00th=[ 726], 99.50th=[ 726], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 8192, max= 8192, per=65.41%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6263KiB/s (6413kB/s)(8192KiB/1308msec) clat (msec): min=114, max=447, avg=280.81, stdev=235.26 lat (msec): min=114, max=447, avg=281.09, stdev=235.27 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 115], 10.00th=[ 115], 20.00th=[ 115], | 30.00th=[ 115], 40.00th=[ 115], 50.00th=[ 115], 60.00th=[ 447], | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 8192, max= 8192, per=32.70%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=6.50%, ctx=59, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10978: Wed May 14 06:26:07 2025 write: IOPS=2, BW=10.6MiB/s (11.2MB/s)(12.0MiB/1127msec) clat (msec): min=67, max=663, avg=372.32, stdev=297.96 lat (msec): min=67, max=663, avg=372.57, stdev=297.92 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 68], | 30.00th=[ 68], 40.00th=[ 388], 50.00th=[ 388], 60.00th=[ 388], | 70.00th=[ 667], 80.00th=[ 667], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 8175, max= 8175, per=32.63%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.09%, sys=10.92%, ctx=53, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10979: Wed May 14 06:26:07 2025 write: IOPS=2, BW=10.9MiB/s (11.4MB/s)(12.0MiB/1099msec) clat (msec): min=102, max=821, avg=358.17, stdev=401.92 lat (msec): min=103, max=821, avg=358.44, stdev=401.91 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 104], | 30.00th=[ 104], 40.00th=[ 150], 50.00th=[ 150], 60.00th=[ 150], | 70.00th=[ 818], 80.00th=[ 818], 90.00th=[ 818], 95.00th=[ 818], | 99.00th=[ 818], 99.50th=[ 818], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min=16384, max=16384, per=65.40%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.09%, sys=10.93%, ctx=148, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.2MiB/s (12.8MB/s), 3131KiB/s-11.7MiB/s (3207kB/s-12.3MB/s), io=16.0MiB (16.8MB), run=1022-1308msec WRITE: bw=24.5MiB/s (25.7MB/s), 6263KiB/s-10.9MiB/s (6413kB/s-11.4MB/s), io=32.0MiB (33.6MB), run=1099-1308msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10972: Wed May 14 06:26:07 2025 read: IOPS=3, BW=12.3MiB/s (12.9MB/s)(12.0MiB/979msec) clat (msec): min=125, max=604, avg=322.64, stdev=250.27 lat (msec): min=125, max=604, avg=322.64, stdev=250.27 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 127], 10.00th=[ 127], 20.00th=[ 127], | 30.00th=[ 127], 40.00th=[ 239], 50.00th=[ 239], 60.00th=[ 239], | 70.00th=[ 609], 80.00th=[ 609], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=21.47%, ctx=2651, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10973: Wed May 14 06:26:07 2025 read: IOPS=0, BW=3451KiB/s (3534kB/s)(4096KiB/1187msec) clat (nsec): min=630622k, max=630622k, avg=630621896.00, stdev= 0.00 lat (nsec): min=630625k, max=630625k, avg=630624536.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 634], 5.00th=[ 634], 10.00th=[ 634], 20.00th=[ 634], | 30.00th=[ 634], 40.00th=[ 634], 50.00th=[ 634], 60.00th=[ 634], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 8192, max= 8192, per=59.35%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6901KiB/s (7067kB/s)(8192KiB/1187msec) clat (msec): min=141, max=383, avg=262.48, stdev=171.24 lat (msec): min=141, max=383, avg=262.75, stdev=171.27 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 142], 10.00th=[ 142], 20.00th=[ 142], | 30.00th=[ 142], 40.00th=[ 142], 50.00th=[ 142], 60.00th=[ 384], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 8192, max= 8192, per=33.83%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=21.42%, ctx=985, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10975: Wed May 14 06:26:07 2025 write: IOPS=2, BW=9082KiB/s (9300kB/s)(12.0MiB/1353msec) clat (msec): min=87, max=865, avg=441.34, stdev=393.72 lat (msec): min=88, max=865, avg=441.63, stdev=393.70 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 88], 10.00th=[ 88], 20.00th=[ 88], | 30.00th=[ 88], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 372], | 70.00th=[ 869], 80.00th=[ 869], 90.00th=[ 869], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 8192, max= 8192, per=33.83%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=19.08%, ctx=30, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10977: Wed May 14 06:26:07 2025 write: IOPS=2, BW=11.1MiB/s (11.6MB/s)(12.0MiB/1083msec) clat (msec): min=115, max=809, avg=348.85, stdev=399.17 lat (msec): min=116, max=809, avg=349.17, stdev=399.01 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 116], 10.00th=[ 116], 20.00th=[ 116], | 30.00th=[ 116], 40.00th=[ 122], 50.00th=[ 122], 60.00th=[ 122], | 70.00th=[ 810], 80.00th=[ 810], 90.00th=[ 810], 95.00th=[ 810], | 99.00th=[ 810], 99.50th=[ 810], 99.90th=[ 810], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min=16384, max=16384, per=67.65%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.00%, sys=25.97%, ctx=86, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.5MiB/s (14.1MB/s), 3451KiB/s-12.3MiB/s (3534kB/s-12.9MB/s), io=16.0MiB (16.8MB), run=979-1187msec WRITE: bw=23.7MiB/s (24.8MB/s), 6901KiB/s-11.1MiB/s (7067kB/s-11.6MB/s), io=32.0MiB (33.6MB), run=1083-1353msec