== sanity test 398b: DIO and buffer IO race ============== 08:05:22 (1744027522) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.24365 s, 40.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10063: Mon Apr 7 08:05:39 2025 read: IOPS=107, BW=430KiB/s (440kB/s)(5992KiB/13948msec) clat (usec): min=54, max=201674, avg=4261.33, stdev=6222.89 lat (usec): min=54, max=201675, avg=4262.18, stdev=6222.90 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 83], 10.00th=[ 110], 20.00th=[ 2073], | 30.00th=[ 2606], 40.00th=[ 2999], 50.00th=[ 3490], 60.00th=[ 4146], | 70.00th=[ 5145], 80.00th=[ 6390], 90.00th=[ 8225], 95.00th=[ 9765], | 99.00th=[ 13829], 99.50th=[ 19530], 99.90th=[ 52167], 99.95th=[202376], | 99.99th=[202376] bw ( KiB/s): min= 239, max= 544, per=24.84%, avg=425.81, stdev=76.32, samples=27 iops : min= 59, max= 136, avg=106.30, stdev=19.19, samples=27 write: IOPS=112, BW=451KiB/s (462kB/s)(6296KiB/13948msec) clat (usec): min=422, max=42889, avg=4743.81, stdev=4298.01 lat (usec): min=422, max=42890, avg=4745.42, stdev=4297.82 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 586], 10.00th=[ 668], 20.00th=[ 840], | 30.00th=[ 1188], 40.00th=[ 2769], 50.00th=[ 4047], 60.00th=[ 4948], | 70.00th=[ 6128], 80.00th=[ 7767], 90.00th=[10421], 95.00th=[12780], | 99.00th=[18482], 99.50th=[21365], 99.90th=[32375], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 295, max= 638, per=24.79%, avg=444.74, stdev=78.02, samples=27 iops : min= 73, max= 159, avg=111.04, stdev=19.51, samples=27 lat (usec) : 100=3.94%, 250=4.49%, 500=0.65%, 750=7.06%, 1000=6.48% lat (msec) : 2=5.24%, 4=25.88%, 10=38.54%, 20=7.19%, 50=0.46% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.29%, sys=23.47%, ctx=5853, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10064: Mon Apr 7 08:05:39 2025 read: IOPS=107, BW=428KiB/s (438kB/s)(5940KiB/13878msec) clat (usec): min=53, max=64179, avg=4100.47, stdev=3497.46 lat (usec): min=54, max=64180, avg=4101.38, stdev=3497.57 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 82], 10.00th=[ 112], 20.00th=[ 2212], | 30.00th=[ 2671], 40.00th=[ 3032], 50.00th=[ 3490], 60.00th=[ 4080], | 70.00th=[ 4948], 80.00th=[ 6259], 90.00th=[ 7898], 95.00th=[ 9241], | 99.00th=[13435], 99.50th=[18482], 99.90th=[38536], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 255, max= 550, per=25.00%, avg=428.48, stdev=69.53, samples=27 iops : min= 63, max= 137, avg=106.96, stdev=17.45, samples=27 write: IOPS=114, BW=457KiB/s (468kB/s)(6348KiB/13878msec) clat (usec): min=480, max=193293, avg=4845.92, stdev=6409.64 lat (usec): min=481, max=193293, avg=4847.02, stdev=6409.63 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 586], 10.00th=[ 652], 20.00th=[ 824], | 30.00th=[ 1123], 40.00th=[ 2573], 50.00th=[ 3916], 60.00th=[ 4883], | 70.00th=[ 6128], 80.00th=[ 7963], 90.00th=[ 10421], 95.00th=[ 12780], | 99.00th=[ 18482], 99.50th=[ 21103], 99.90th=[ 41157], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 295, max= 584, per=25.19%, avg=451.89, stdev=74.64, samples=27 iops : min= 73, max= 146, avg=112.81, stdev=18.68, samples=27 lat (usec) : 100=3.81%, 250=4.07%, 500=0.36%, 750=7.52%, 1000=6.80% lat (msec) : 2=4.85%, 4=27.25%, 10=37.92%, 20=6.87%, 50=0.49% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.27%, sys=23.71%, ctx=5723, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10066: Mon Apr 7 08:05:39 2025 read: IOPS=107, BW=430KiB/s (441kB/s)(6028KiB/14006msec) clat (usec): min=58, max=61783, avg=4120.30, stdev=3688.33 lat (usec): min=58, max=61784, avg=4121.25, stdev=3688.28 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 84], 10.00th=[ 105], 20.00th=[ 2114], | 30.00th=[ 2671], 40.00th=[ 3032], 50.00th=[ 3458], 60.00th=[ 4015], | 70.00th=[ 5145], 80.00th=[ 6325], 90.00th=[ 8225], 95.00th=[ 9634], | 99.00th=[14484], 99.50th=[18482], 99.90th=[46924], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 168, max= 592, per=25.11%, avg=430.43, stdev=96.01, samples=28 iops : min= 42, max= 148, avg=107.54, stdev=23.98, samples=28 write: IOPS=111, BW=447KiB/s (458kB/s)(6260KiB/14006msec) clat (usec): min=481, max=212820, avg=4921.91, stdev=6946.74 lat (usec): min=482, max=212822, avg=4923.02, stdev=6947.20 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 594], 10.00th=[ 676], 20.00th=[ 865], | 30.00th=[ 1336], 40.00th=[ 2835], 50.00th=[ 4015], 60.00th=[ 4948], | 70.00th=[ 5997], 80.00th=[ 7701], 90.00th=[ 10552], 95.00th=[ 12911], | 99.00th=[ 19530], 99.50th=[ 23987], 99.90th=[ 67634], 99.95th=[212861], | 99.99th=[212861] bw ( KiB/s): min= 216, max= 648, per=24.90%, avg=446.75, stdev=86.78, samples=28 iops : min= 54, max= 162, avg=111.61, stdev=21.68, samples=28 lat (usec) : 100=3.97%, 250=4.75%, 500=0.42%, 750=6.51%, 1000=5.99% lat (msec) : 2=5.76%, 4=27.51%, 10=37.08%, 20=7.39%, 50=0.52% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.25%, sys=23.41%, ctx=5792, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10068: Mon Apr 7 08:05:39 2025 read: IOPS=108, BW=434KiB/s (444kB/s)(6052KiB/13956msec) clat (usec): min=51, max=71892, avg=4008.04, stdev=3465.63 lat (usec): min=51, max=71893, avg=4008.96, stdev=3465.60 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 85], 10.00th=[ 109], 20.00th=[ 1975], | 30.00th=[ 2573], 40.00th=[ 2966], 50.00th=[ 3458], 60.00th=[ 4080], | 70.00th=[ 5080], 80.00th=[ 6456], 90.00th=[ 7898], 95.00th=[ 9110], | 99.00th=[13042], 99.50th=[15139], 99.90th=[26870], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 207, max= 592, per=25.09%, avg=429.96, stdev=78.51, samples=27 iops : min= 51, max= 148, avg=107.37, stdev=19.70, samples=27 write: IOPS=111, BW=447KiB/s (458kB/s)(6236KiB/13956msec) clat (usec): min=463, max=211888, avg=4999.68, stdev=6972.97 lat (usec): min=464, max=211890, avg=5001.17, stdev=6973.09 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 594], 10.00th=[ 709], 20.00th=[ 873], | 30.00th=[ 1287], 40.00th=[ 2769], 50.00th=[ 4015], 60.00th=[ 4948], | 70.00th=[ 6325], 80.00th=[ 8029], 90.00th=[ 10683], 95.00th=[ 13173], | 99.00th=[ 18482], 99.50th=[ 22676], 99.90th=[ 62653], 99.95th=[212861], | 99.99th=[212861] bw ( KiB/s): min= 231, max= 574, per=24.77%, avg=444.44, stdev=75.16, samples=27 iops : min= 57, max= 143, avg=111.00, stdev=18.84, samples=27 lat (usec) : 100=3.84%, 250=5.60%, 500=0.20%, 750=5.92%, 1000=7.10% lat (msec) : 2=5.31%, 4=26.33%, 10=38.15%, 20=7.10%, 50=0.36% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.27%, sys=23.73%, ctx=5821, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1714KiB/s (1756kB/s), 428KiB/s-434KiB/s (438kB/s-444kB/s), io=23.4MiB (24.6MB), run=13878-14006msec WRITE: bw=1795KiB/s (1838kB/s), 447KiB/s-457KiB/s (458kB/s-468kB/s), io=24.6MiB (25.7MB), run=13878-14006msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10065: Mon Apr 7 08:06:02 2025 read: IOPS=40, BW=163KiB/s (167kB/s)(5992KiB/36844msec) clat (usec): min=1619, max=203544, avg=4084.05, stdev=6092.52 lat (usec): min=1619, max=203545, avg=4084.96, stdev=6092.52 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1926], 10.00th=[ 2040], 20.00th=[ 2180], | 30.00th=[ 2311], 40.00th=[ 2442], 50.00th=[ 2704], 60.00th=[ 3228], | 70.00th=[ 3949], 80.00th=[ 4883], 90.00th=[ 7504], 95.00th=[ 10290], | 99.00th=[ 16057], 99.50th=[ 20579], 99.90th=[ 42730], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 64, max= 304, per=25.04%, avg=162.51, stdev=63.29, samples=73 iops : min= 16, max= 76, avg=40.53, stdev=15.83, samples=73 write: IOPS=42, BW=171KiB/s (175kB/s)(6296KiB/36844msec) clat (usec): min=9143, max=66246, avg=19460.50, stdev=7135.86 lat (usec): min=9144, max=66247, avg=19461.59, stdev=7135.83 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11469], 10.00th=[12256], 20.00th=[13566], | 30.00th=[15008], 40.00th=[16450], 50.00th=[17957], 60.00th=[19268], | 70.00th=[21103], 80.00th=[23725], 90.00th=[29492], 95.00th=[34341], | 99.00th=[44303], 99.50th=[46924], 99.90th=[51119], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 48, max= 240, per=25.09%, avg=170.62, stdev=50.12, samples=73 iops : min= 12, max= 60, avg=42.56, stdev=12.57, samples=73 lat (msec) : 2=3.84%, 4=30.53%, 10=11.98%, 20=34.99%, 50=18.55% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.12%, sys=5.95%, ctx=3677, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10067: Mon Apr 7 08:06:02 2025 read: IOPS=40, BW=161KiB/s (165kB/s)(5940KiB/36853msec) clat (usec): min=1551, max=197644, avg=4081.81, stdev=5844.25 lat (usec): min=1552, max=197645, avg=4082.76, stdev=5844.24 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1926], 10.00th=[ 2024], 20.00th=[ 2147], | 30.00th=[ 2278], 40.00th=[ 2442], 50.00th=[ 2671], 60.00th=[ 3195], | 70.00th=[ 4015], 80.00th=[ 4948], 90.00th=[ 7832], 95.00th=[ 10421], | 99.00th=[ 15401], 99.50th=[ 19792], 99.90th=[ 32637], 99.95th=[198181], | 99.99th=[198181] bw ( KiB/s): min= 16, max= 312, per=24.85%, avg=161.30, stdev=69.64, samples=73 iops : min= 4, max= 78, avg=40.23, stdev=17.47, samples=73 write: IOPS=43, BW=172KiB/s (176kB/s)(6348KiB/36853msec) clat (usec): min=9462, max=76425, avg=19343.90, stdev=7295.05 lat (usec): min=9463, max=76426, avg=19344.94, stdev=7295.04 clat percentiles (usec): | 1.00th=[10552], 5.00th=[11469], 10.00th=[12256], 20.00th=[13435], | 30.00th=[14746], 40.00th=[16319], 50.00th=[17957], 60.00th=[19006], | 70.00th=[20841], 80.00th=[23462], 90.00th=[29492], 95.00th=[34866], | 99.00th=[43779], 99.50th=[45351], 99.90th=[62129], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 72, max= 256, per=25.27%, avg=171.82, stdev=50.40, samples=73 iops : min= 18, max= 64, avg=42.86, stdev=12.64, samples=73 lat (msec) : 2=4.13%, 4=29.69%, 10=11.91%, 20=36.52%, 50=17.51% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.13%, sys=5.93%, ctx=3728, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10069: Mon Apr 7 08:06:02 2025 read: IOPS=40, BW=163KiB/s (167kB/s)(6028KiB/36968msec) clat (usec): min=1619, max=40439, avg=4291.23, stdev=3516.07 lat (usec): min=1620, max=40439, avg=4292.16, stdev=3516.05 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1926], 10.00th=[ 2040], 20.00th=[ 2180], | 30.00th=[ 2343], 40.00th=[ 2540], 50.00th=[ 2868], 60.00th=[ 3621], | 70.00th=[ 4228], 80.00th=[ 5473], 90.00th=[ 8848], 95.00th=[11076], | 99.00th=[17695], 99.50th=[22676], 99.90th=[34341], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 56, max= 336, per=25.09%, avg=162.84, stdev=59.87, samples=73 iops : min= 14, max= 84, avg=40.62, stdev=15.00, samples=73 write: IOPS=42, BW=169KiB/s (173kB/s)(6260KiB/36968msec) clat (msec): min=9, max=209, avg=19.43, stdev= 8.74 lat (msec): min=9, max=209, avg=19.43, stdev= 8.74 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 21], 80.00th=[ 24], 90.00th=[ 29], 95.00th=[ 34], | 99.00th=[ 45], 99.50th=[ 50], 99.90th=[ 77], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 64, max= 248, per=24.78%, avg=168.53, stdev=52.64, samples=73 iops : min= 16, max= 62, avg=42.04, stdev=13.21, samples=73 lat (msec) : 2=4.10%, 4=28.22%, 10=13.41%, 20=36.26%, 50=17.81% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.10%, sys=5.99%, ctx=3746, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10070: Mon Apr 7 08:06:02 2025 read: IOPS=41, BW=165KiB/s (169kB/s)(6052KiB/36686msec) clat (usec): min=1633, max=40691, avg=4037.95, stdev=3193.64 lat (usec): min=1634, max=40692, avg=4038.89, stdev=3193.61 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1926], 10.00th=[ 2024], 20.00th=[ 2147], | 30.00th=[ 2278], 40.00th=[ 2442], 50.00th=[ 2704], 60.00th=[ 3228], | 70.00th=[ 4047], 80.00th=[ 5211], 90.00th=[ 8225], 95.00th=[10028], | 99.00th=[14877], 99.50th=[20579], 99.90th=[30802], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 56, max= 296, per=25.45%, avg=165.19, stdev=63.88, samples=73 iops : min= 14, max= 74, avg=41.25, stdev=15.99, samples=73 write: IOPS=42, BW=170KiB/s (174kB/s)(6236KiB/36686msec) clat (msec): min=8, max=209, avg=19.55, stdev= 8.66 lat (msec): min=8, max=209, avg=19.55, stdev= 8.66 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 20], | 70.00th=[ 22], 80.00th=[ 24], 90.00th=[ 30], 95.00th=[ 34], | 99.00th=[ 43], 99.50th=[ 45], 99.90th=[ 72], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 64, max= 248, per=24.97%, avg=169.79, stdev=50.83, samples=73 iops : min= 16, max= 62, avg=42.40, stdev=12.74, samples=73 lat (msec) : 2=4.26%, 4=29.92%, 10=12.86%, 20=34.02%, 50=18.75% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.11%, sys=5.96%, ctx=3684, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=650KiB/s (665kB/s), 161KiB/s-165KiB/s (165kB/s-169kB/s), io=23.4MiB (24.6MB), run=36686-36968msec WRITE: bw=680KiB/s (696kB/s), 169KiB/s-172KiB/s (173kB/s-176kB/s), io=24.6MiB (25.7MB), run=36686-36968msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10110: Mon Apr 7 08:06:06 2025 read: IOPS=131, BW=2109KiB/s (2160kB/s)(5920KiB/2807msec) clat (usec): min=60, max=26111, avg=3547.37, stdev=3022.35 lat (usec): min=61, max=26111, avg=3548.11, stdev=3022.40 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 74], 10.00th=[ 88], 20.00th=[ 125], | 30.00th=[ 2442], 40.00th=[ 3032], 50.00th=[ 3425], 60.00th=[ 3916], | 70.00th=[ 4621], 80.00th=[ 5669], 90.00th=[ 7177], 95.00th=[ 8225], | 99.00th=[12256], 99.50th=[15270], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 1532, max= 2496, per=28.09%, avg=2091.00, stdev=431.87, samples=5 iops : min= 95, max= 156, avg=130.40, stdev=27.12, samples=5 write: IOPS=141, BW=2269KiB/s (2323kB/s)(6368KiB/2807msec) clat (usec): min=661, max=23947, avg=3599.55, stdev=3518.06 lat (usec): min=662, max=23947, avg=3601.45, stdev=3519.01 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 791], 10.00th=[ 898], 20.00th=[ 1090], | 30.00th=[ 1237], 40.00th=[ 1418], 50.00th=[ 1827], 60.00th=[ 3326], | 70.00th=[ 4359], 80.00th=[ 5866], 90.00th=[ 8455], 95.00th=[10290], | 99.00th=[17957], 99.50th=[21365], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 1788, max= 3040, per=29.14%, avg=2282.80, stdev=523.25, samples=5 iops : min= 111, max= 190, avg=142.40, stdev=32.79, samples=5 lat (usec) : 100=5.99%, 250=7.29%, 500=0.26%, 750=0.91%, 1000=7.55% lat (msec) : 2=19.14%, 4=22.14%, 10=32.94%, 20=3.39%, 50=0.39% cpu : usr=0.29%, sys=30.26%, ctx=1471, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10111: Mon Apr 7 08:06:06 2025 read: IOPS=124, BW=1994KiB/s (2042kB/s)(5744KiB/2881msec) clat (usec): min=61, max=16310, avg=3433.71, stdev=2833.56 lat (usec): min=61, max=16311, avg=3438.48, stdev=2829.82 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 81], 10.00th=[ 100], 20.00th=[ 129], | 30.00th=[ 2245], 40.00th=[ 2933], 50.00th=[ 3359], 60.00th=[ 3752], | 70.00th=[ 4424], 80.00th=[ 5604], 90.00th=[ 6980], 95.00th=[ 8356], | 99.00th=[12518], 99.50th=[14353], 99.90th=[16319], 99.95th=[16319], | 99.99th=[16319] bw ( KiB/s): min= 1728, max= 2240, per=27.07%, avg=2015.00, stdev=213.93, samples=5 iops : min= 108, max= 140, avg=125.80, stdev=13.27, samples=5 write: IOPS=141, BW=2271KiB/s (2326kB/s)(6544KiB/2881msec) clat (usec): min=671, max=49582, avg=3984.27, stdev=4238.92 lat (usec): min=672, max=49583, avg=3985.38, stdev=4238.92 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 824], 10.00th=[ 906], 20.00th=[ 1090], | 30.00th=[ 1270], 40.00th=[ 1532], 50.00th=[ 2343], 60.00th=[ 3589], | 70.00th=[ 4948], 80.00th=[ 6587], 90.00th=[ 9241], 95.00th=[11207], | 99.00th=[17171], 99.50th=[20579], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 2080, max= 2427, per=29.07%, avg=2277.40, stdev=147.79, samples=5 iops : min= 130, max= 151, avg=142.20, stdev= 9.07, samples=5 lat (usec) : 100=4.56%, 250=8.20%, 500=0.78%, 750=1.04%, 1000=6.90% lat (msec) : 2=16.80%, 4=25.65%, 10=31.12%, 20=4.56%, 50=0.39% cpu : usr=0.17%, sys=29.58%, ctx=1540, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10112: Mon Apr 7 08:06:06 2025 read: IOPS=121, BW=1945KiB/s (1991kB/s)(6256KiB/3217msec) clat (usec): min=62, max=26559, avg=3933.52, stdev=3945.35 lat (usec): min=62, max=26559, avg=3936.90, stdev=3945.03 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 84], 10.00th=[ 104], 20.00th=[ 137], | 30.00th=[ 2278], 40.00th=[ 2802], 50.00th=[ 3392], 60.00th=[ 3949], | 70.00th=[ 4948], 80.00th=[ 6259], 90.00th=[ 7963], 95.00th=[ 9765], | 99.00th=[23725], 99.50th=[25035], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 606, max= 2577, per=25.75%, avg=1917.17, stdev=707.72, samples=6 iops : min= 37, max= 161, avg=119.67, stdev=44.55, samples=6 write: IOPS=117, BW=1875KiB/s (1920kB/s)(6032KiB/3217msec) clat (usec): min=670, max=28315, avg=4360.19, stdev=5102.12 lat (usec): min=671, max=28317, avg=4361.34, stdev=5102.26 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 807], 10.00th=[ 889], 20.00th=[ 1057], | 30.00th=[ 1237], 40.00th=[ 1516], 50.00th=[ 2245], 60.00th=[ 3949], | 70.00th=[ 4948], 80.00th=[ 6325], 90.00th=[ 9896], 95.00th=[14877], | 99.00th=[26870], 99.50th=[26870], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 702, max= 2304, per=23.01%, avg=1802.00, stdev=587.30, samples=6 iops : min= 43, max= 144, avg=112.33, stdev=36.93, samples=6 lat (usec) : 100=4.82%, 250=8.20%, 500=0.52%, 750=1.69%, 1000=8.72% lat (msec) : 2=14.19%, 4=22.40%, 10=32.68%, 20=4.30%, 50=2.47% cpu : usr=0.16%, sys=26.71%, ctx=1557, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10113: Mon Apr 7 08:06:06 2025 read: IOPS=132, BW=2127KiB/s (2178kB/s)(6032KiB/2836msec) clat (usec): min=61, max=16986, avg=3577.75, stdev=3062.08 lat (usec): min=61, max=16987, avg=3578.45, stdev=3062.10 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 75], 10.00th=[ 86], 20.00th=[ 121], | 30.00th=[ 2245], 40.00th=[ 2900], 50.00th=[ 3392], 60.00th=[ 3851], | 70.00th=[ 4686], 80.00th=[ 5932], 90.00th=[ 7439], 95.00th=[ 8979], | 99.00th=[13566], 99.50th=[15533], 99.90th=[16909], 99.95th=[16909], | 99.99th=[16909] bw ( KiB/s): min= 1692, max= 2560, per=28.09%, avg=2091.20, stdev=316.45, samples=5 iops : min= 105, max= 160, avg=130.40, stdev=20.08, samples=5 write: IOPS=137, BW=2206KiB/s (2259kB/s)(6256KiB/2836msec) clat (usec): min=701, max=55566, avg=3745.66, stdev=4276.59 lat (usec): min=701, max=55567, avg=3751.23, stdev=4276.45 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 807], 10.00th=[ 881], 20.00th=[ 1090], | 30.00th=[ 1270], 40.00th=[ 1565], 50.00th=[ 2278], 60.00th=[ 3261], | 70.00th=[ 4228], 80.00th=[ 5800], 90.00th=[ 8979], 95.00th=[10683], | 99.00th=[20055], 99.50th=[20317], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 1792, max= 2560, per=27.84%, avg=2180.40, stdev=271.83, samples=5 iops : min= 112, max= 160, avg=136.00, stdev=16.99, samples=5 lat (usec) : 100=7.29%, 250=6.38%, 500=0.78%, 750=1.04%, 1000=7.16% lat (msec) : 2=16.02%, 4=25.91%, 10=30.73%, 20=4.17%, 50=0.39% lat (msec) : 100=0.13% cpu : usr=0.11%, sys=29.45%, ctx=1542, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7445KiB/s (7624kB/s), 1945KiB/s-2127KiB/s (1991kB/s-2178kB/s), io=23.4MiB (24.5MB), run=2807-3217msec WRITE: bw=7833KiB/s (8021kB/s), 1875KiB/s-2271KiB/s (1920kB/s-2326kB/s), io=24.6MiB (25.8MB), run=2807-3217msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10114: Mon Apr 7 08:06:12 2025 read: IOPS=40, BW=647KiB/s (662kB/s)(5920KiB/9153msec) clat (usec): min=1921, max=47172, avg=4647.61, stdev=3892.16 lat (usec): min=1922, max=47173, avg=4648.62, stdev=3892.12 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2089], 10.00th=[ 2180], 20.00th=[ 2343], | 30.00th=[ 2507], 40.00th=[ 2835], 50.00th=[ 3359], 60.00th=[ 4113], | 70.00th=[ 5276], 80.00th=[ 6325], 90.00th=[ 8094], 95.00th=[10683], | 99.00th=[18744], 99.50th=[30016], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 318, max= 1152, per=24.88%, avg=637.78, stdev=217.72, samples=18 iops : min= 19, max= 72, avg=39.67, stdev=13.61, samples=18 write: IOPS=43, BW=696KiB/s (712kB/s)(6368KiB/9153msec) clat (usec): min=9964, max=50523, avg=18549.23, stdev=6106.42 lat (usec): min=9965, max=50524, avg=18550.75, stdev=6106.48 clat percentiles (usec): | 1.00th=[10814], 5.00th=[11863], 10.00th=[12518], 20.00th=[13566], | 30.00th=[14484], 40.00th=[15926], 50.00th=[17171], 60.00th=[19006], | 70.00th=[20055], 80.00th=[21890], 90.00th=[26346], 95.00th=[31327], | 99.00th=[41681], 99.50th=[46400], 99.90th=[50594], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 415, max= 896, per=25.70%, avg=692.83, stdev=169.49, samples=18 iops : min= 25, max= 56, avg=43.11, stdev=10.67, samples=18 lat (msec) : 2=0.65%, 4=27.47%, 10=17.06%, 20=38.80%, 50=15.89% lat (msec) : 100=0.13% cpu : usr=0.09%, sys=7.31%, ctx=1033, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10115: Mon Apr 7 08:06:12 2025 read: IOPS=38, BW=615KiB/s (629kB/s)(5744KiB/9344msec) clat (usec): min=1902, max=47804, avg=4723.99, stdev=3930.88 lat (usec): min=1902, max=47804, avg=4724.83, stdev=3930.89 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2147], 10.00th=[ 2212], 20.00th=[ 2376], | 30.00th=[ 2507], 40.00th=[ 2704], 50.00th=[ 3097], 60.00th=[ 4293], | 70.00th=[ 5407], 80.00th=[ 6521], 90.00th=[ 8455], 95.00th=[11600], | 99.00th=[18744], 99.50th=[19530], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 288, max= 1120, per=23.91%, avg=612.78, stdev=227.75, samples=18 iops : min= 18, max= 70, avg=38.11, stdev=14.27, samples=18 write: IOPS=43, BW=700KiB/s (717kB/s)(6544KiB/9344msec) clat (usec): min=10061, max=59543, avg=18578.48, stdev=5888.32 lat (usec): min=10062, max=59545, avg=18580.05, stdev=5888.41 clat percentiles (usec): | 1.00th=[10945], 5.00th=[11994], 10.00th=[12649], 20.00th=[13698], | 30.00th=[14484], 40.00th=[15926], 50.00th=[17433], 60.00th=[19268], | 70.00th=[20841], 80.00th=[22152], 90.00th=[26608], 95.00th=[29230], | 99.00th=[35914], 99.50th=[37487], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 416, max= 864, per=25.50%, avg=687.56, stdev=165.89, samples=18 iops : min= 26, max= 54, avg=42.78, stdev=10.55, samples=18 lat (msec) : 2=1.04%, 4=25.52%, 10=16.93%, 20=37.76%, 50=18.62% lat (msec) : 100=0.13% cpu : usr=0.17%, sys=7.04%, ctx=1044, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10116: Mon Apr 7 08:06:12 2025 read: IOPS=44, BW=708KiB/s (725kB/s)(6256KiB/8830msec) clat (usec): min=1926, max=27270, avg=4482.50, stdev=3547.18 lat (usec): min=1927, max=27271, avg=4483.37, stdev=3547.20 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2114], 10.00th=[ 2212], 20.00th=[ 2343], | 30.00th=[ 2442], 40.00th=[ 2606], 50.00th=[ 2933], 60.00th=[ 3818], | 70.00th=[ 4686], 80.00th=[ 6325], 90.00th=[ 7963], 95.00th=[10814], | 99.00th=[21890], 99.50th=[24773], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 319, max= 1216, per=27.89%, avg=714.82, stdev=301.60, samples=17 iops : min= 19, max= 76, avg=44.53, stdev=18.99, samples=17 write: IOPS=42, BW=683KiB/s (700kB/s)(6032KiB/8830msec) clat (usec): min=9686, max=45668, avg=18729.61, stdev=6116.44 lat (usec): min=9687, max=45670, avg=18731.35, stdev=6116.31 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11994], 10.00th=[12649], 20.00th=[13698], | 30.00th=[14615], 40.00th=[16188], 50.00th=[17433], 60.00th=[19006], | 70.00th=[20579], 80.00th=[22414], 90.00th=[26870], 95.00th=[31327], | 99.00th=[41157], 99.50th=[45351], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 447, max= 928, per=25.05%, avg=675.29, stdev=157.22, samples=17 iops : min= 27, max= 58, avg=42.06, stdev=10.00, samples=17 lat (msec) : 2=0.65%, 4=31.25%, 10=16.02%, 20=35.42%, 50=16.67% cpu : usr=0.14%, sys=7.45%, ctx=1051, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10117: Mon Apr 7 08:06:12 2025 read: IOPS=41, BW=657KiB/s (673kB/s)(6032KiB/9180msec) clat (usec): min=1963, max=25290, avg=4573.02, stdev=3270.95 lat (usec): min=1964, max=25291, avg=4573.92, stdev=3271.02 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2114], 10.00th=[ 2212], 20.00th=[ 2409], | 30.00th=[ 2573], 40.00th=[ 2737], 50.00th=[ 3294], 60.00th=[ 4228], | 70.00th=[ 5211], 80.00th=[ 6325], 90.00th=[ 7767], 95.00th=[10552], | 99.00th=[18482], 99.50th=[24773], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 223, max= 1088, per=25.38%, avg=650.56, stdev=233.00, samples=18 iops : min= 13, max= 68, avg=40.56, stdev=14.72, samples=18 write: IOPS=42, BW=681KiB/s (698kB/s)(6256KiB/9180msec) clat (usec): min=10662, max=59279, avg=18891.50, stdev=6443.77 lat (usec): min=10664, max=59281, avg=18893.29, stdev=6444.26 clat percentiles (usec): | 1.00th=[10945], 5.00th=[11863], 10.00th=[12518], 20.00th=[13566], | 30.00th=[14877], 40.00th=[16057], 50.00th=[17695], 60.00th=[19006], | 70.00th=[20055], 80.00th=[22152], 90.00th=[28181], 95.00th=[32375], | 99.00th=[39584], 99.50th=[43779], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 351, max= 928, per=25.18%, avg=678.89, stdev=180.14, samples=18 iops : min= 21, max= 58, avg=42.33, stdev=11.41, samples=18 lat (msec) : 2=0.13%, 4=27.86%, 10=18.62%, 20=37.11%, 50=16.15% lat (msec) : 100=0.13% cpu : usr=0.15%, sys=7.34%, ctx=1068, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2563KiB/s (2625kB/s), 615KiB/s-708KiB/s (629kB/s-725kB/s), io=23.4MiB (24.5MB), run=8830-9344msec WRITE: bw=2697KiB/s (2762kB/s), 681KiB/s-700KiB/s (698kB/s-717kB/s), io=24.6MiB (25.8MB), run=8830-9344msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10137: Mon Apr 7 08:06:14 2025 read: IOPS=14, BW=14.8MiB/s (15.6MB/s)(7168KiB/472msec) clat (usec): min=554, max=55835, avg=34391.35, stdev=17789.97 lat (usec): min=555, max=55836, avg=34392.51, stdev=17789.96 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 553], 10.00th=[ 553], 20.00th=[27657], | 30.00th=[29492], 40.00th=[29492], 50.00th=[37487], 60.00th=[44303], | 70.00th=[44303], 80.00th=[45351], 90.00th=[55837], 95.00th=[55837], | 99.00th=[55837], 99.50th=[55837], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] write: IOPS=10, BW=10.6MiB/s (11.1MB/s)(5120KiB/472msec) clat (usec): min=25619, max=64100, avg=44805.99, stdev=18151.40 lat (usec): min=25683, max=64143, avg=44852.22, stdev=18140.18 clat percentiles (usec): | 1.00th=[25560], 5.00th=[25560], 10.00th=[25560], 20.00th=[25560], | 30.00th=[27132], 40.00th=[27132], 50.00th=[46400], 60.00th=[46400], | 70.00th=[61080], 80.00th=[61080], 90.00th=[64226], 95.00th=[64226], | 99.00th=[64226], 99.50th=[64226], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] lat (usec) : 750=8.33% lat (msec) : 50=66.67%, 100=25.00% cpu : usr=0.00%, sys=41.19%, ctx=562, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10138: Mon Apr 7 08:06:14 2025 read: IOPS=3, BW=3425KiB/s (3507kB/s)(2048KiB/598msec) clat (usec): min=37471, max=63288, avg=50380.14, stdev=18255.52 lat (usec): min=37473, max=63290, avg=50381.76, stdev=18255.42 clat percentiles (usec): | 1.00th=[37487], 5.00th=[37487], 10.00th=[37487], 20.00th=[37487], | 30.00th=[37487], 40.00th=[37487], 50.00th=[37487], 60.00th=[63177], | 70.00th=[63177], 80.00th=[63177], 90.00th=[63177], 95.00th=[63177], | 99.00th=[63177], 99.50th=[63177], 99.90th=[63177], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 4096, max= 4096, per=11.96%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=16, BW=16.7MiB/s (17.5MB/s)(10.0MiB/598msec) clat (usec): min=22187, max=66217, avg=45579.61, stdev=13455.98 lat (usec): min=22222, max=66248, avg=45636.37, stdev=13456.22 clat percentiles (usec): | 1.00th=[22152], 5.00th=[22152], 10.00th=[22152], 20.00th=[31589], | 30.00th=[41681], 40.00th=[42730], 50.00th=[43779], 60.00th=[45876], | 70.00th=[49546], 80.00th=[49546], 90.00th=[65799], 95.00th=[66323], | 99.00th=[66323], 99.50th=[66323], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min=14336, max=14336, per=29.90%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (msec) : 50=75.00%, 100=25.00% cpu : usr=0.00%, sys=38.19%, ctx=213, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10139: Mon Apr 7 08:06:14 2025 read: IOPS=12, BW=12.5MiB/s (13.1MB/s)(6144KiB/480msec) clat (usec): min=323, max=61876, avg=25842.19, stdev=28537.96 lat (usec): min=324, max=61877, avg=25843.14, stdev=28538.16 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 326], 10.00th=[ 326], 20.00th=[ 429], | 30.00th=[ 429], 40.00th=[ 519], 50.00th=[ 519], 60.00th=[42206], | 70.00th=[49546], 80.00th=[49546], 90.00th=[62129], 95.00th=[62129], | 99.00th=[62129], 99.50th=[62129], 99.90th=[62129], 99.95th=[62129], | 99.99th=[62129] write: IOPS=12, BW=12.5MiB/s (13.1MB/s)(6144KiB/480msec) clat (msec): min=25, max=100, avg=49.13, stdev=26.89 lat (msec): min=25, max=100, avg=49.18, stdev=26.91 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 40], 50.00th=[ 40], 60.00th=[ 49], | 70.00th=[ 50], 80.00th=[ 50], 90.00th=[ 102], 95.00th=[ 102], | 99.00th=[ 102], 99.50th=[ 102], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 50=58.33%, 100=8.33%, 250=8.33% cpu : usr=0.00%, sys=33.82%, ctx=520, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10140: Mon Apr 7 08:06:14 2025 read: IOPS=10, BW=10.0MiB/s (10.5MB/s)(5120KiB/499msec) clat (usec): min=425, max=39624, avg=11137.56, stdev=17066.20 lat (usec): min=425, max=39625, avg=11138.68, stdev=17066.05 clat percentiles (usec): | 1.00th=[ 424], 5.00th=[ 424], 10.00th=[ 424], 20.00th=[ 424], | 30.00th=[ 429], 40.00th=[ 429], 50.00th=[ 562], 60.00th=[ 562], | 70.00th=[14615], 80.00th=[14615], 90.00th=[39584], 95.00th=[39584], | 99.00th=[39584], 99.50th=[39584], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] write: IOPS=14, BW=14.0MiB/s (14.7MB/s)(7168KiB/499msec) clat (usec): min=29261, max=90116, avg=57362.81, stdev=23431.70 lat (usec): min=29331, max=90179, avg=57425.71, stdev=23428.31 clat percentiles (usec): | 1.00th=[29230], 5.00th=[29230], 10.00th=[29230], 20.00th=[29492], | 30.00th=[47449], 40.00th=[47449], 50.00th=[56886], 60.00th=[72877], | 70.00th=[72877], 80.00th=[76022], 90.00th=[89654], 95.00th=[89654], | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 20=8.33%, 50=33.33%, 100=33.33% cpu : usr=0.00%, sys=23.29%, ctx=110, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=33.4MiB/s (35.1MB/s), 3425KiB/s-14.8MiB/s (3507kB/s-15.6MB/s), io=20.0MiB (20.0MB), run=472-598msec WRITE: bw=46.8MiB/s (49.1MB/s), 10.6MiB/s-16.7MiB/s (11.1MB/s-17.5MB/s), io=28.0MiB (29.4MB), run=472-598msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10142: Mon Apr 7 08:06:14 2025 read: IOPS=9, BW=9583KiB/s (9813kB/s)(7168KiB/748msec) clat (msec): min=13, max=155, avg=60.12, stdev=52.52 lat (msec): min=13, max=155, avg=60.12, stdev=52.52 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 20], | 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 37], 60.00th=[ 73], | 70.00th=[ 73], 80.00th=[ 100], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 6144, max= 6144, per=22.44%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6845KiB/s (7009kB/s)(5120KiB/748msec) clat (usec): min=25169, max=81801, avg=49222.46, stdev=21572.55 lat (usec): min=25237, max=81866, avg=49290.01, stdev=21573.53 clat percentiles (usec): | 1.00th=[25297], 5.00th=[25297], 10.00th=[25297], 20.00th=[25297], | 30.00th=[36963], 40.00th=[36963], 50.00th=[45351], 60.00th=[45351], | 70.00th=[56886], 80.00th=[56886], 90.00th=[82314], 95.00th=[82314], | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 2048, max= 2048, per=5.34%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=41.67%, 100=33.33%, 250=8.33% cpu : usr=0.27%, sys=12.03%, ctx=49, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10143: Mon Apr 7 08:06:14 2025 read: IOPS=2, BW=2893KiB/s (2962kB/s)(2048KiB/708msec) clat (usec): min=30612, max=88633, avg=59623.11, stdev=41027.41 lat (usec): min=30613, max=88636, avg=59625.12, stdev=41028.17 clat percentiles (usec): | 1.00th=[30540], 5.00th=[30540], 10.00th=[30540], 20.00th=[30540], | 30.00th=[30540], 40.00th=[30540], 50.00th=[30540], 60.00th=[88605], | 70.00th=[88605], 80.00th=[88605], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 4096, max= 4096, per=14.96%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=14, BW=14.1MiB/s (14.8MB/s)(10.0MiB/708msec) clat (msec): min=36, max=102, avg=56.40, stdev=23.07 lat (msec): min=36, max=102, avg=56.47, stdev=23.08 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 37], | 30.00th=[ 38], 40.00th=[ 41], 50.00th=[ 42], 60.00th=[ 49], | 70.00th=[ 72], 80.00th=[ 72], 90.00th=[ 83], 95.00th=[ 103], | 99.00th=[ 103], 99.50th=[ 103], 99.90th=[ 103], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min=10240, max=10240, per=26.71%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=33.33%, 250=8.33% cpu : usr=0.14%, sys=19.80%, ctx=94, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10144: Mon Apr 7 08:06:14 2025 read: IOPS=8, BW=8533KiB/s (8738kB/s)(6144KiB/720msec) clat (usec): min=17081, max=33695, avg=23555.78, stdev=5881.10 lat (usec): min=17082, max=33696, avg=23557.12, stdev=5880.84 clat percentiles (usec): | 1.00th=[17171], 5.00th=[17171], 10.00th=[17171], 20.00th=[18482], | 30.00th=[18482], 40.00th=[22676], 50.00th=[22676], 60.00th=[24511], | 70.00th=[24773], 80.00th=[24773], 90.00th=[33817], 95.00th=[33817], | 99.00th=[33817], 99.50th=[33817], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 4096, max= 4096, per=14.96%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8533KiB/s (8738kB/s)(6144KiB/720msec) clat (msec): min=32, max=161, avg=93.36, stdev=49.55 lat (msec): min=32, max=161, avg=93.42, stdev=49.56 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 61], | 30.00th=[ 61], 40.00th=[ 68], 50.00th=[ 68], 60.00th=[ 99], | 70.00th=[ 140], 80.00th=[ 140], 90.00th=[ 161], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 6144, max= 6144, per=16.03%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=41.67%, 100=25.00%, 250=16.67% cpu : usr=0.00%, sys=13.63%, ctx=61, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10145: Mon Apr 7 08:06:14 2025 read: IOPS=7, BW=7496KiB/s (7676kB/s)(5120KiB/683msec) clat (usec): min=19403, max=58065, avg=38964.57, stdev=15923.50 lat (usec): min=19405, max=58065, avg=38965.76, stdev=15923.24 clat percentiles (usec): | 1.00th=[19530], 5.00th=[19530], 10.00th=[19530], 20.00th=[19530], | 30.00th=[25822], 40.00th=[25822], 50.00th=[44303], 60.00th=[44303], | 70.00th=[46924], 80.00th=[46924], 90.00th=[57934], 95.00th=[57934], | 99.00th=[57934], 99.50th=[57934], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 8192, max= 8192, per=29.92%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.2MiB/s (10.7MB/s)(7168KiB/683msec) clat (msec): min=33, max=117, avg=59.02, stdev=28.71 lat (msec): min=33, max=117, avg=59.09, stdev=28.71 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 39], | 30.00th=[ 45], 40.00th=[ 45], 50.00th=[ 52], 60.00th=[ 58], | 70.00th=[ 58], 80.00th=[ 71], 90.00th=[ 118], 95.00th=[ 118], | 99.00th=[ 118], 99.50th=[ 118], 99.90th=[ 118], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 8192, max= 8192, per=21.37%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=17.60%, ctx=67, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=26.7MiB/s (28.0MB/s), 2893KiB/s-9583KiB/s (2962kB/s-9813kB/s), io=20.0MiB (20.0MB), run=683-748msec WRITE: bw=37.4MiB/s (39.3MB/s), 6845KiB/s-14.1MiB/s (7009kB/s-14.8MB/s), io=28.0MiB (29.4MB), run=683-748msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10151: Mon Apr 7 08:06:16 2025 read: IOPS=2, BW=9958KiB/s (10.2MB/s)(12.0MiB/1234msec) clat (msec): min=78, max=717, avg=408.46, stdev=319.57 lat (msec): min=78, max=717, avg=408.46, stdev=319.57 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 80], 10.00th=[ 80], 20.00th=[ 80], | 30.00th=[ 80], 40.00th=[ 430], 50.00th=[ 430], 60.00th=[ 430], | 70.00th=[ 718], 80.00th=[ 718], 90.00th=[ 718], 95.00th=[ 718], | 99.00th=[ 718], 99.50th=[ 718], 99.90th=[ 718], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 8192, max= 8192, per=63.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 100=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=5.84%, ctx=38, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10152: Mon Apr 7 08:06:16 2025 read: IOPS=0, BW=3251KiB/s (3329kB/s)(4096KiB/1260msec) clat (nsec): min=621027k, max=621027k, avg=621027011.00, stdev= 0.00 lat (nsec): min=621029k, max=621029k, avg=621029377.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 625], 5.00th=[ 625], 10.00th=[ 625], 20.00th=[ 625], | 30.00th=[ 625], 40.00th=[ 625], 50.00th=[ 625], 60.00th=[ 625], | 70.00th=[ 625], 80.00th=[ 625], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 8192, max= 8192, per=63.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6502KiB/s (6658kB/s)(8192KiB/1260msec) clat (msec): min=191, max=431, avg=311.59, stdev=169.83 lat (msec): min=191, max=432, avg=312.11, stdev=170.18 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 192], 10.00th=[ 192], 20.00th=[ 192], | 30.00th=[ 192], 40.00th=[ 192], 50.00th=[ 192], 60.00th=[ 430], | 70.00th=[ 430], 80.00th=[ 430], 90.00th=[ 430], 95.00th=[ 430], | 99.00th=[ 430], 99.50th=[ 430], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=9.13%, ctx=45, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10153: Mon Apr 7 08:06:16 2025 write: IOPS=2, BW=9768KiB/s (10.0MB/s)(12.0MiB/1258msec) clat (msec): min=293, max=576, avg=415.04, stdev=145.93 lat (msec): min=293, max=577, avg=415.34, stdev=145.96 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 292], 10.00th=[ 292], 20.00th=[ 292], | 30.00th=[ 292], 40.00th=[ 376], 50.00th=[ 376], 60.00th=[ 376], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min=16384, max=16384, per=63.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.24%, sys=7.32%, ctx=37, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10154: Mon Apr 7 08:06:16 2025 write: IOPS=2, BW=11.4MiB/s (11.0MB/s)(12.0MiB/1049msec) clat (msec): min=61, max=543, avg=344.55, stdev=251.44 lat (msec): min=62, max=543, avg=344.86, stdev=251.44 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 62], 10.00th=[ 62], 20.00th=[ 62], | 30.00th=[ 62], 40.00th=[ 430], 50.00th=[ 430], 60.00th=[ 430], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 8175, max= 8192, per=31.47%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 100=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=9.16%, ctx=53, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.7MiB/s (13.3MB/s), 3251KiB/s-9958KiB/s (3329kB/s-10.2MB/s), io=16.0MiB (16.8MB), run=1234-1260msec WRITE: bw=25.4MiB/s (26.6MB/s), 6502KiB/s-11.4MiB/s (6658kB/s-11.0MB/s), io=32.0MiB (33.6MB), run=1049-1260msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10155: Mon Apr 7 08:06:16 2025 read: IOPS=3, BW=13.6MiB/s (14.3MB/s)(12.0MiB/882msec) clat (msec): min=124, max=562, avg=290.48, stdev=237.16 lat (msec): min=124, max=562, avg=290.48, stdev=237.16 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 125], 10.00th=[ 125], 20.00th=[ 125], | 30.00th=[ 125], 40.00th=[ 186], 50.00th=[ 186], 60.00th=[ 186], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=18.16%, ctx=1622, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10156: Mon Apr 7 08:06:16 2025 read: IOPS=0, BW=3534KiB/s (3619kB/s)(4096KiB/1159msec) clat (nsec): min=571604k, max=571604k, avg=571603555.00, stdev= 0.00 lat (nsec): min=571605k, max=571605k, avg=571604962.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 575], 5.00th=[ 575], 10.00th=[ 575], 20.00th=[ 575], | 30.00th=[ 575], 40.00th=[ 575], 50.00th=[ 575], 60.00th=[ 575], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 8192, max= 8192, per=57.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=7068KiB/s (7238kB/s)(8192KiB/1159msec) clat (msec): min=244, max=329, avg=286.89, stdev=59.95 lat (msec): min=244, max=329, avg=287.15, stdev=59.93 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 245], 10.00th=[ 245], 20.00th=[ 245], | 30.00th=[ 245], 40.00th=[ 245], 50.00th=[ 245], 60.00th=[ 330], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 8192, max= 8192, per=33.63%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=25.30%, ctx=845, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10157: Mon Apr 7 08:06:16 2025 write: IOPS=2, BW=9136KiB/s (9355kB/s)(12.0MiB/1345msec) clat (msec): min=231, max=831, avg=439.80, stdev=339.49 lat (msec): min=232, max=831, avg=440.07, stdev=339.51 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 232], 10.00th=[ 232], 20.00th=[ 232], | 30.00th=[ 232], 40.00th=[ 257], 50.00th=[ 257], 60.00th=[ 257], | 70.00th=[ 835], 80.00th=[ 835], 90.00th=[ 835], 95.00th=[ 835], | 99.00th=[ 835], 99.50th=[ 835], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 8192, max= 8192, per=33.63%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=20.31%, ctx=73, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10158: Mon Apr 7 08:06:16 2025 write: IOPS=2, BW=10.0MiB/s (11.5MB/s)(12.0MiB/1093msec) clat (msec): min=153, max=678, avg=361.27, stdev=279.07 lat (msec): min=153, max=678, avg=361.57, stdev=279.14 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 253], 50.00th=[ 253], 60.00th=[ 253], | 70.00th=[ 676], 80.00th=[ 676], 90.00th=[ 676], 95.00th=[ 676], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min=16351, max=16351, per=67.12%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=25.92%, ctx=110, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.8MiB/s (14.5MB/s), 3534KiB/s-13.6MiB/s (3619kB/s-14.3MB/s), io=16.0MiB (16.8MB), run=882-1159msec WRITE: bw=23.8MiB/s (24.9MB/s), 7068KiB/s-10.0MiB/s (7238kB/s-11.5MB/s), io=32.0MiB (33.6MB), run=1093-1345msec