== sanity test 398b: DIO and buffer IO race ============== 23:26:10 (1708835170) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 0.875818 s, 57.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25459: Sat Feb 24 23:26:25 2024 read: IOPS=120, BW=483KiB/s (495kB/s)(5992KiB/12398msec) clat (usec): min=37, max=143196, avg=3400.93, stdev=4760.16 lat (usec): min=38, max=143197, avg=3401.90, stdev=4760.08 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 90], | 30.00th=[ 1926], 40.00th=[ 2311], 50.00th=[ 2671], 60.00th=[ 3195], | 70.00th=[ 4047], 80.00th=[ 5407], 90.00th=[ 7373], 95.00th=[ 9503], | 99.00th=[ 12649], 99.50th=[ 14746], 99.90th=[ 28181], 99.95th=[143655], | 99.99th=[143655] bw ( KiB/s): min= 335, max= 670, per=25.74%, avg=480.13, stdev=89.76, samples=24 iops : min= 83, max= 167, avg=119.96, stdev=22.43, samples=24 write: IOPS=126, BW=508KiB/s (520kB/s)(6296KiB/12398msec) clat (usec): min=423, max=52246, avg=4592.55, stdev=4492.67 lat (usec): min=423, max=52247, avg=4593.40, stdev=4492.63 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 474], 10.00th=[ 498], 20.00th=[ 603], | 30.00th=[ 1123], 40.00th=[ 2638], 50.00th=[ 3654], 60.00th=[ 4686], | 70.00th=[ 5800], 80.00th=[ 7570], 90.00th=[10159], 95.00th=[13042], | 99.00th=[20841], 99.50th=[23200], 99.90th=[31851], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 296, max= 672, per=25.83%, avg=504.50, stdev=76.09, samples=24 iops : min= 74, max= 168, avg=126.04, stdev=19.05, samples=24 lat (usec) : 50=1.20%, 100=9.47%, 250=1.63%, 500=5.60%, 750=7.55% lat (usec) : 1000=1.89% lat (msec) : 2=6.64%, 4=27.41%, 10=31.22%, 20=6.71%, 50=0.62% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.18%, sys=21.71%, ctx=5311, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25460: Sat Feb 24 23:26:25 2024 read: IOPS=115, BW=464KiB/s (475kB/s)(5940KiB/12804msec) clat (usec): min=38, max=63705, avg=3270.79, stdev=3425.17 lat (usec): min=38, max=63706, avg=3271.68, stdev=3425.45 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 93], | 30.00th=[ 1860], 40.00th=[ 2212], 50.00th=[ 2638], 60.00th=[ 3032], | 70.00th=[ 3884], 80.00th=[ 5342], 90.00th=[ 7373], 95.00th=[ 9241], | 99.00th=[12911], 99.50th=[14222], 99.90th=[30540], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 304, max= 664, per=24.75%, avg=461.60, stdev=86.02, samples=25 iops : min= 76, max= 166, avg=115.32, stdev=21.52, samples=25 write: IOPS=123, BW=496KiB/s (508kB/s)(6348KiB/12804msec) clat (usec): min=379, max=140616, avg=4957.28, stdev=5801.96 lat (usec): min=380, max=140616, avg=4958.23, stdev=5802.06 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 474], 10.00th=[ 494], 20.00th=[ 635], | 30.00th=[ 1172], 40.00th=[ 2737], 50.00th=[ 3916], 60.00th=[ 5014], | 70.00th=[ 6194], 80.00th=[ 7963], 90.00th=[ 10945], 95.00th=[ 13960], | 99.00th=[ 21627], 99.50th=[ 25560], 99.90th=[ 40633], 99.95th=[141558], | 99.99th=[141558] bw ( KiB/s): min= 327, max= 622, per=24.98%, avg=487.84, stdev=76.83, samples=25 iops : min= 81, max= 155, avg=121.88, stdev=19.26, samples=25 lat (usec) : 50=0.94%, 100=9.70%, 250=1.86%, 500=5.83%, 750=7.62% lat (usec) : 1000=1.63% lat (msec) : 2=6.77%, 4=26.07%, 10=31.32%, 20=7.55%, 50=0.65% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.19%, sys=21.95%, ctx=5491, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25461: Sat Feb 24 23:26:25 2024 read: IOPS=117, BW=468KiB/s (480kB/s)(6028KiB/12869msec) clat (usec): min=38, max=32326, avg=3358.89, stdev=3346.09 lat (usec): min=39, max=32326, avg=3359.58, stdev=3346.09 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 63], 10.00th=[ 67], 20.00th=[ 85], | 30.00th=[ 1811], 40.00th=[ 2245], 50.00th=[ 2638], 60.00th=[ 3163], | 70.00th=[ 4015], 80.00th=[ 5800], 90.00th=[ 7963], 95.00th=[ 9503], | 99.00th=[14222], 99.50th=[16909], 99.90th=[25822], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 272, max= 622, per=24.78%, avg=462.20, stdev=81.85, samples=25 iops : min= 68, max= 155, avg=115.48, stdev=20.40, samples=25 write: IOPS=121, BW=486KiB/s (498kB/s)(6260KiB/12869msec) clat (usec): min=413, max=148997, avg=4938.39, stdev=5994.07 lat (usec): min=414, max=148999, avg=4939.58, stdev=5993.92 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 465], 10.00th=[ 494], 20.00th=[ 635], | 30.00th=[ 1029], 40.00th=[ 2671], 50.00th=[ 3818], 60.00th=[ 4817], | 70.00th=[ 6194], 80.00th=[ 8094], 90.00th=[ 10945], 95.00th=[ 14091], | 99.00th=[ 21103], 99.50th=[ 23462], 99.90th=[ 42730], 99.95th=[149947], | 99.99th=[149947] bw ( KiB/s): min= 328, max= 590, per=24.44%, avg=477.28, stdev=76.36, samples=25 iops : min= 82, max= 147, avg=119.24, stdev=19.09, samples=25 lat (usec) : 50=1.07%, 100=10.68%, 250=1.86%, 500=5.79%, 750=6.93% lat (usec) : 1000=2.51% lat (msec) : 2=6.18%, 4=25.78%, 10=30.57%, 20=7.85%, 50=0.75% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=21.53%, ctx=5348, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25462: Sat Feb 24 23:26:25 2024 read: IOPS=118, BW=473KiB/s (485kB/s)(6052KiB/12782msec) clat (usec): min=37, max=29494, avg=3288.10, stdev=3350.48 lat (usec): min=37, max=29495, avg=3288.87, stdev=3350.46 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 88], | 30.00th=[ 1795], 40.00th=[ 2180], 50.00th=[ 2606], 60.00th=[ 3064], | 70.00th=[ 3720], 80.00th=[ 5211], 90.00th=[ 7898], 95.00th=[ 9896], | 99.00th=[14746], 99.50th=[18482], 99.90th=[23987], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 264, max= 630, per=25.29%, avg=471.60, stdev=89.20, samples=25 iops : min= 66, max= 157, avg=117.76, stdev=22.28, samples=25 write: IOPS=121, BW=488KiB/s (500kB/s)(6236KiB/12782msec) clat (usec): min=408, max=142492, avg=4956.69, stdev=6277.04 lat (usec): min=409, max=142494, avg=4958.11, stdev=6276.97 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 469], 10.00th=[ 490], 20.00th=[ 635], | 30.00th=[ 971], 40.00th=[ 2474], 50.00th=[ 3752], 60.00th=[ 4752], | 70.00th=[ 6128], 80.00th=[ 7898], 90.00th=[ 11207], 95.00th=[ 14615], | 99.00th=[ 20579], 99.50th=[ 28443], 99.90th=[ 62653], 99.95th=[141558], | 99.99th=[141558] bw ( KiB/s): min= 304, max= 654, per=24.87%, avg=485.72, stdev=90.76, samples=25 iops : min= 76, max= 163, avg=121.28, stdev=22.65, samples=25 lat (usec) : 50=1.17%, 100=10.16%, 250=2.08%, 500=6.02%, 750=7.03% lat (usec) : 1000=2.54% lat (msec) : 2=7.78%, 4=25.85%, 10=28.48%, 20=8.07%, 50=0.72% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.16%, sys=21.57%, ctx=5339, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1866KiB/s (1911kB/s), 464KiB/s-483KiB/s (475kB/s-495kB/s), io=23.4MiB (24.6MB), run=12398-12869msec WRITE: bw=1954KiB/s (2000kB/s), 486KiB/s-508KiB/s (498kB/s-520kB/s), io=24.6MiB (25.7MB), run=12398-12869msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25463: Sat Feb 24 23:26:54 2024 read: IOPS=36, BW=145KiB/s (148kB/s)(5992KiB/41373msec) clat (usec): min=1427, max=143029, avg=3808.55, stdev=4994.71 lat (usec): min=1428, max=143031, avg=3809.40, stdev=4994.74 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1795], 10.00th=[ 1860], 20.00th=[ 2008], | 30.00th=[ 2114], 40.00th=[ 2245], 50.00th=[ 2474], 60.00th=[ 2835], | 70.00th=[ 3490], 80.00th=[ 4146], 90.00th=[ 7242], 95.00th=[ 10945], | 99.00th=[ 16581], 99.50th=[ 18482], 99.90th=[ 50594], 99.95th=[143655], | 99.99th=[143655] bw ( KiB/s): min= 31, max= 272, per=25.21%, avg=144.46, stdev=60.12, samples=82 iops : min= 7, max= 68, avg=36.02, stdev=15.03, samples=82 write: IOPS=38, BW=152KiB/s (156kB/s)(6296KiB/41373msec) clat (usec): min=11582, max=66171, avg=22614.35, stdev=8862.00 lat (usec): min=11583, max=66172, avg=22615.35, stdev=8862.03 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13173], 10.00th=[13960], 20.00th=[15401], | 30.00th=[16909], 40.00th=[19268], 50.00th=[21103], 60.00th=[22414], | 70.00th=[23987], 80.00th=[26608], 90.00th=[34866], 95.00th=[42730], | 99.00th=[53216], 99.50th=[57934], 99.90th=[65274], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 64, max= 208, per=25.28%, avg=151.68, stdev=45.69, samples=82 iops : min= 16, max= 52, avg=37.83, stdev=11.44, samples=82 lat (msec) : 2=9.60%, 4=28.78%, 10=7.10%, 20=25.78%, 50=27.80% lat (msec) : 100=0.91%, 250=0.03% cpu : usr=0.08%, sys=4.16%, ctx=3513, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25464: Sat Feb 24 23:26:54 2024 read: IOPS=35, BW=142KiB/s (145kB/s)(5940KiB/41876msec) clat (usec): min=1426, max=127175, avg=3747.49, stdev=4494.86 lat (usec): min=1427, max=127176, avg=3748.32, stdev=4494.86 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1762], 10.00th=[ 1844], 20.00th=[ 1975], | 30.00th=[ 2073], 40.00th=[ 2212], 50.00th=[ 2442], 60.00th=[ 2933], | 70.00th=[ 3490], 80.00th=[ 4113], 90.00th=[ 6915], 95.00th=[ 10683], | 99.00th=[ 17433], 99.50th=[ 20841], 99.90th=[ 30278], 99.95th=[127402], | 99.99th=[127402] bw ( KiB/s): min= 16, max= 296, per=24.73%, avg=141.70, stdev=63.69, samples=83 iops : min= 4, max= 74, avg=35.36, stdev=15.92, samples=83 write: IOPS=37, BW=152KiB/s (155kB/s)(6348KiB/41876msec) clat (usec): min=10735, max=69589, avg=22833.60, stdev=9159.35 lat (usec): min=10736, max=69589, avg=22834.62, stdev=9159.40 clat percentiles (usec): | 1.00th=[12256], 5.00th=[13304], 10.00th=[14222], 20.00th=[15533], | 30.00th=[17171], 40.00th=[19268], 50.00th=[21103], 60.00th=[22676], | 70.00th=[23987], 80.00th=[26870], 90.00th=[35390], 95.00th=[43254], | 99.00th=[54789], 99.50th=[61604], 99.90th=[68682], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 72, max= 208, per=25.11%, avg=150.66, stdev=44.69, samples=83 iops : min= 18, max= 52, avg=37.60, stdev=11.19, samples=83 lat (msec) : 2=10.68%, 4=27.47%, 10=7.13%, 20=25.59%, 50=28.06% lat (msec) : 100=1.04%, 250=0.03% cpu : usr=0.10%, sys=4.03%, ctx=3503, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25465: Sat Feb 24 23:26:54 2024 read: IOPS=36, BW=144KiB/s (148kB/s)(6028KiB/41808msec) clat (usec): min=1422, max=49770, avg=3891.81, stdev=3611.49 lat (usec): min=1423, max=49770, avg=3892.63, stdev=3611.49 clat percentiles (usec): | 1.00th=[ 1598], 5.00th=[ 1778], 10.00th=[ 1860], 20.00th=[ 2008], | 30.00th=[ 2114], 40.00th=[ 2311], 50.00th=[ 2606], 60.00th=[ 3163], | 70.00th=[ 3621], 80.00th=[ 4228], 90.00th=[ 8029], 95.00th=[11076], | 99.00th=[18220], 99.50th=[21627], 99.90th=[36963], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 32, max= 304, per=25.06%, avg=143.60, stdev=59.66, samples=83 iops : min= 8, max= 76, avg=35.82, stdev=14.94, samples=83 write: IOPS=37, BW=150KiB/s (153kB/s)(6260KiB/41808msec) clat (msec): min=10, max=141, avg=22.92, stdev= 9.44 lat (msec): min=10, max=141, avg=22.92, stdev= 9.44 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 23], | 70.00th=[ 25], 80.00th=[ 27], 90.00th=[ 35], 95.00th=[ 43], | 99.00th=[ 57], 99.50th=[ 62], 99.90th=[ 66], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 56, max= 208, per=24.83%, avg=148.96, stdev=45.13, samples=83 iops : min= 14, max= 52, avg=37.16, stdev=11.27, samples=83 lat (msec) : 2=9.60%, 4=28.03%, 10=8.07%, 20=25.07%, 50=28.19% lat (msec) : 100=1.01%, 250=0.03% cpu : usr=0.12%, sys=4.03%, ctx=3533, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25466: Sat Feb 24 23:26:54 2024 read: IOPS=36, BW=146KiB/s (150kB/s)(6052KiB/41439msec) clat (usec): min=1493, max=49995, avg=3781.45, stdev=3575.99 lat (usec): min=1494, max=49995, avg=3782.28, stdev=3576.01 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1778], 10.00th=[ 1860], 20.00th=[ 1991], | 30.00th=[ 2114], 40.00th=[ 2245], 50.00th=[ 2507], 60.00th=[ 2900], | 70.00th=[ 3490], 80.00th=[ 4146], 90.00th=[ 7308], 95.00th=[11076], | 99.00th=[18482], 99.50th=[22938], 99.90th=[36963], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 16, max= 320, per=25.39%, avg=145.46, stdev=61.56, samples=82 iops : min= 4, max= 80, avg=36.29, stdev=15.38, samples=82 write: IOPS=37, BW=150KiB/s (154kB/s)(6236KiB/41439msec) clat (msec): min=11, max=140, avg=22.86, stdev= 9.58 lat (msec): min=11, max=140, avg=22.86, stdev= 9.58 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 27], 90.00th=[ 36], 95.00th=[ 43], | 99.00th=[ 56], 99.50th=[ 59], 99.90th=[ 70], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 71, max= 208, per=25.02%, avg=150.15, stdev=45.97, samples=82 iops : min= 17, max= 52, avg=37.46, stdev=11.51, samples=82 lat (msec) : 2=10.22%, 4=28.19%, 10=7.58%, 20=24.61%, 50=28.32% lat (msec) : 100=1.04%, 250=0.03% cpu : usr=0.11%, sys=4.10%, ctx=3525, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=573KiB/s (587kB/s), 142KiB/s-146KiB/s (145kB/s-150kB/s), io=23.4MiB (24.6MB), run=41373-41876msec WRITE: bw=600KiB/s (615kB/s), 150KiB/s-152KiB/s (153kB/s-156kB/s), io=24.6MiB (25.7MB), run=41373-41876msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25517: Sat Feb 24 23:26:58 2024 read: IOPS=149, BW=2393KiB/s (2450kB/s)(5920KiB/2474msec) clat (usec): min=47, max=15995, avg=2906.88, stdev=2571.07 lat (usec): min=47, max=15995, avg=2907.52, stdev=2571.08 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 67], 10.00th=[ 70], 20.00th=[ 78], | 30.00th=[ 105], 40.00th=[ 2573], 50.00th=[ 2802], 60.00th=[ 3228], | 70.00th=[ 3916], 80.00th=[ 4948], 90.00th=[ 6325], 95.00th=[ 7242], | 99.00th=[10159], 99.50th=[13042], 99.90th=[16057], 99.95th=[16057], | 99.99th=[16057] bw ( KiB/s): min= 1952, max= 2752, per=24.98%, avg=2416.00, stdev=387.98, samples=4 iops : min= 122, max= 172, avg=151.00, stdev=24.25, samples=4 write: IOPS=160, BW=2574KiB/s (2636kB/s)(6368KiB/2474msec) clat (usec): min=582, max=31747, avg=3490.98, stdev=4783.66 lat (usec): min=583, max=31748, avg=3492.18, stdev=4783.53 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 668], 10.00th=[ 709], 20.00th=[ 766], | 30.00th=[ 840], 40.00th=[ 971], 50.00th=[ 1270], 60.00th=[ 2638], | 70.00th=[ 3621], 80.00th=[ 5211], 90.00th=[ 8586], 95.00th=[10683], | 99.00th=[28443], 99.50th=[29754], 99.90th=[31851], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 2112, max= 3808, per=26.02%, avg=2648.00, stdev=780.95, samples=4 iops : min= 132, max= 238, avg=165.50, stdev=48.81, samples=4 lat (usec) : 50=0.26%, 100=13.54%, 250=1.69%, 500=0.13%, 750=9.37% lat (usec) : 1000=12.37% lat (msec) : 2=7.55%, 4=26.95%, 10=24.35%, 20=2.47%, 50=1.30% cpu : usr=0.36%, sys=27.54%, ctx=1299, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25518: Sat Feb 24 23:26:58 2024 read: IOPS=145, BW=2331KiB/s (2387kB/s)(5744KiB/2464msec) clat (usec): min=48, max=23858, avg=2919.18, stdev=2763.61 lat (usec): min=49, max=23859, avg=2920.25, stdev=2763.45 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 74], | 30.00th=[ 102], 40.00th=[ 2409], 50.00th=[ 2868], 60.00th=[ 3163], | 70.00th=[ 3851], 80.00th=[ 5014], 90.00th=[ 6325], 95.00th=[ 7308], | 99.00th=[10814], 99.50th=[16450], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 2048, max= 2912, per=24.63%, avg=2382.75, stdev=381.79, samples=4 iops : min= 128, max= 182, avg=148.75, stdev=23.99, samples=4 write: IOPS=165, BW=2656KiB/s (2720kB/s)(6544KiB/2464msec) clat (usec): min=580, max=28536, avg=3444.54, stdev=3923.37 lat (usec): min=581, max=28537, avg=3446.64, stdev=3922.74 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 668], 10.00th=[ 701], 20.00th=[ 750], | 30.00th=[ 832], 40.00th=[ 996], 50.00th=[ 1434], 60.00th=[ 2999], | 70.00th=[ 4228], 80.00th=[ 5932], 90.00th=[ 8225], 95.00th=[10552], | 99.00th=[20317], 99.50th=[22414], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 2203, max= 3168, per=26.09%, avg=2654.75, stdev=460.17, samples=4 iops : min= 137, max= 198, avg=165.75, stdev=28.99, samples=4 lat (usec) : 50=0.39%, 100=13.28%, 250=1.30%, 500=0.26%, 750=10.55% lat (usec) : 1000=11.20% lat (msec) : 2=7.03%, 4=25.91%, 10=26.17%, 20=3.12%, 50=0.78% cpu : usr=0.24%, sys=27.81%, ctx=1357, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25519: Sat Feb 24 23:26:58 2024 read: IOPS=160, BW=2576KiB/s (2637kB/s)(6256KiB/2429msec) clat (usec): min=47, max=22272, avg=2945.12, stdev=2906.96 lat (usec): min=47, max=22273, avg=2949.87, stdev=2911.61 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 53], 10.00th=[ 67], 20.00th=[ 73], | 30.00th=[ 90], 40.00th=[ 2343], 50.00th=[ 2835], 60.00th=[ 3359], | 70.00th=[ 4228], 80.00th=[ 5145], 90.00th=[ 6194], 95.00th=[ 7504], | 99.00th=[14615], 99.50th=[17695], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 1888, max= 3072, per=26.13%, avg=2528.00, stdev=616.64, samples=4 iops : min= 118, max= 192, avg=158.00, stdev=38.54, samples=4 write: IOPS=155, BW=2483KiB/s (2543kB/s)(6032KiB/2429msec) clat (usec): min=569, max=34961, avg=3366.93, stdev=4450.98 lat (usec): min=570, max=34962, avg=3367.87, stdev=4450.99 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 644], 10.00th=[ 693], 20.00th=[ 750], | 30.00th=[ 840], 40.00th=[ 1020], 50.00th=[ 1287], 60.00th=[ 2376], | 70.00th=[ 3720], 80.00th=[ 5538], 90.00th=[ 8455], 95.00th=[10290], | 99.00th=[23200], 99.50th=[33162], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 2336, max= 2976, per=24.76%, avg=2520.00, stdev=305.12, samples=4 iops : min= 146, max= 186, avg=157.50, stdev=19.07, samples=4 lat (usec) : 50=0.91%, 100=15.76%, 250=1.04%, 500=0.52%, 750=10.03% lat (usec) : 1000=9.11% lat (msec) : 2=9.37%, 4=23.83%, 10=25.91%, 20=2.60%, 50=0.91% cpu : usr=0.29%, sys=27.88%, ctx=1307, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25520: Sat Feb 24 23:26:58 2024 read: IOPS=152, BW=2436KiB/s (2495kB/s)(6032KiB/2476msec) clat (usec): min=47, max=35757, avg=2828.21, stdev=3508.09 lat (usec): min=48, max=35758, avg=2829.73, stdev=3508.28 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 53], 10.00th=[ 63], 20.00th=[ 71], | 30.00th=[ 84], 40.00th=[ 184], 50.00th=[ 2540], 60.00th=[ 3097], | 70.00th=[ 3818], 80.00th=[ 4948], 90.00th=[ 6325], 95.00th=[ 7439], | 99.00th=[16909], 99.50th=[24249], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 1504, max= 3264, per=25.05%, avg=2422.75, stdev=732.03, samples=4 iops : min= 94, max= 204, avg=151.25, stdev=45.79, samples=4 write: IOPS=157, BW=2527KiB/s (2587kB/s)(6256KiB/2476msec) clat (usec): min=565, max=31271, avg=3583.52, stdev=4577.33 lat (usec): min=566, max=31272, avg=3584.48, stdev=4577.32 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 660], 10.00th=[ 693], 20.00th=[ 750], | 30.00th=[ 832], 40.00th=[ 979], 50.00th=[ 1336], 60.00th=[ 2966], | 70.00th=[ 4015], 80.00th=[ 5669], 90.00th=[ 8225], 95.00th=[11469], | 99.00th=[26608], 99.50th=[28181], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 1728, max= 3616, per=25.38%, avg=2582.50, stdev=779.26, samples=4 iops : min= 108, max= 226, avg=161.25, stdev=48.71, samples=4 lat (usec) : 50=1.04%, 100=17.06%, 250=1.69%, 500=0.26%, 750=10.55% lat (usec) : 1000=10.29% lat (msec) : 2=7.42%, 4=22.53%, 10=25.00%, 20=2.73%, 50=1.43% cpu : usr=0.20%, sys=26.38%, ctx=1278, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9674KiB/s (9906kB/s), 2331KiB/s-2576KiB/s (2387kB/s-2637kB/s), io=23.4MiB (24.5MB), run=2429-2476msec WRITE: bw=9.94MiB/s (10.4MB/s), 2483KiB/s-2656KiB/s (2543kB/s-2720kB/s), io=24.6MiB (25.8MB), run=2429-2476msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25513: Sat Feb 24 23:27:05 2024 read: IOPS=38, BW=612KiB/s (626kB/s)(5920KiB/9680msec) clat (usec): min=1810, max=36343, avg=3875.74, stdev=3135.45 lat (usec): min=1811, max=36344, avg=3876.58, stdev=3135.43 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2057], 10.00th=[ 2114], 20.00th=[ 2212], | 30.00th=[ 2343], 40.00th=[ 2474], 50.00th=[ 2638], 60.00th=[ 2835], | 70.00th=[ 3621], 80.00th=[ 5276], 90.00th=[ 6783], 95.00th=[ 9372], | 99.00th=[16581], 99.50th=[17171], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 352, max= 1024, per=24.72%, avg=602.84, stdev=170.15, samples=19 iops : min= 22, max= 64, avg=37.63, stdev=10.66, samples=19 write: IOPS=41, BW=658KiB/s (674kB/s)(6368KiB/9680msec) clat (usec): min=12106, max=50612, avg=20698.73, stdev=6113.40 lat (usec): min=12107, max=50614, avg=20700.39, stdev=6113.48 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13435], 10.00th=[13829], 20.00th=[15270], | 30.00th=[16712], 40.00th=[18744], 50.00th=[20055], 60.00th=[21365], | 70.00th=[22938], 80.00th=[24249], 90.00th=[27919], 95.00th=[33424], | 99.00th=[41157], 99.50th=[44303], 99.90th=[50594], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 416, max= 800, per=25.53%, avg=655.05, stdev=129.80, samples=19 iops : min= 26, max= 50, avg=40.89, stdev= 8.06, samples=19 lat (msec) : 2=1.30%, 4=33.85%, 10=11.07%, 20=27.60%, 50=26.04% lat (msec) : 100=0.13% cpu : usr=0.09%, sys=5.65%, ctx=944, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25514: Sat Feb 24 23:27:05 2024 read: IOPS=36, BW=585KiB/s (599kB/s)(5744KiB/9820msec) clat (usec): min=1752, max=24519, avg=3852.19, stdev=3229.32 lat (usec): min=1753, max=24520, avg=3853.03, stdev=3229.31 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2040], 10.00th=[ 2114], 20.00th=[ 2212], | 30.00th=[ 2343], 40.00th=[ 2474], 50.00th=[ 2606], 60.00th=[ 2835], | 70.00th=[ 3326], 80.00th=[ 5145], 90.00th=[ 6652], 95.00th=[ 8717], | 99.00th=[19792], 99.50th=[21627], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 288, max= 928, per=24.16%, avg=589.37, stdev=195.24, samples=19 iops : min= 18, max= 58, avg=36.79, stdev=12.17, samples=19 write: IOPS=41, BW=666KiB/s (682kB/s)(6544KiB/9820msec) clat (usec): min=11526, max=51994, avg=20608.96, stdev=6613.05 lat (usec): min=11526, max=51995, avg=20610.65, stdev=6613.01 clat percentiles (usec): | 1.00th=[12256], 5.00th=[12911], 10.00th=[13829], 20.00th=[15139], | 30.00th=[16319], 40.00th=[17957], 50.00th=[19530], 60.00th=[21365], | 70.00th=[22676], 80.00th=[23987], 90.00th=[28443], 95.00th=[35390], | 99.00th=[43779], 99.50th=[45351], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 320, max= 832, per=25.66%, avg=658.42, stdev=148.68, samples=19 iops : min= 20, max= 52, avg=41.11, stdev= 9.29, samples=19 lat (msec) : 2=1.43%, 4=33.85%, 10=9.77%, 20=29.56%, 50=25.26% lat (msec) : 100=0.13% cpu : usr=0.08%, sys=5.58%, ctx=928, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25515: Sat Feb 24 23:27:05 2024 read: IOPS=41, BW=662KiB/s (678kB/s)(6256KiB/9447msec) clat (usec): min=1739, max=23099, avg=3726.34, stdev=2899.53 lat (usec): min=1739, max=23100, avg=3727.18, stdev=2899.53 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2008], 10.00th=[ 2073], 20.00th=[ 2180], | 30.00th=[ 2278], 40.00th=[ 2376], 50.00th=[ 2540], 60.00th=[ 2769], | 70.00th=[ 3261], 80.00th=[ 4883], 90.00th=[ 6521], 95.00th=[10028], | 99.00th=[16712], 99.50th=[19268], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 288, max= 1152, per=27.70%, avg=675.56, stdev=283.44, samples=18 iops : min= 18, max= 72, avg=42.22, stdev=17.72, samples=18 write: IOPS=39, BW=639KiB/s (654kB/s)(6032KiB/9447msec) clat (usec): min=11705, max=47684, avg=21173.04, stdev=6733.30 lat (usec): min=11706, max=47685, avg=21174.66, stdev=6733.26 clat percentiles (usec): | 1.00th=[12649], 5.00th=[13566], 10.00th=[14091], 20.00th=[15533], | 30.00th=[16909], 40.00th=[18744], 50.00th=[20317], 60.00th=[21627], | 70.00th=[22676], 80.00th=[24511], 90.00th=[28705], 95.00th=[36439], | 99.00th=[44303], 99.50th=[47449], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 384, max= 768, per=24.73%, avg=634.67, stdev=132.74, samples=18 iops : min= 24, max= 48, avg=39.67, stdev= 8.30, samples=18 lat (msec) : 2=2.34%, 4=35.81%, 10=10.29%, 20=26.04%, 50=25.52% cpu : usr=0.10%, sys=5.68%, ctx=946, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25516: Sat Feb 24 23:27:05 2024 read: IOPS=39, BW=628KiB/s (643kB/s)(6032KiB/9607msec) clat (usec): min=1804, max=25150, avg=4022.63, stdev=3503.01 lat (usec): min=1804, max=25150, avg=4023.42, stdev=3503.03 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2024], 10.00th=[ 2089], 20.00th=[ 2212], | 30.00th=[ 2311], 40.00th=[ 2442], 50.00th=[ 2573], 60.00th=[ 2802], | 70.00th=[ 3294], 80.00th=[ 5211], 90.00th=[ 7767], 95.00th=[12387], | 99.00th=[18482], 99.50th=[22938], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 288, max= 960, per=25.47%, avg=621.32, stdev=187.96, samples=19 iops : min= 18, max= 60, avg=38.74, stdev=11.80, samples=19 write: IOPS=40, BW=651KiB/s (667kB/s)(6256KiB/9607msec) clat (usec): min=12107, max=51908, avg=20670.52, stdev=5874.94 lat (usec): min=12109, max=51909, avg=20672.27, stdev=5874.99 clat percentiles (usec): | 1.00th=[12518], 5.00th=[13435], 10.00th=[14353], 20.00th=[15270], | 30.00th=[16581], 40.00th=[18482], 50.00th=[20055], 60.00th=[21365], | 70.00th=[22938], 80.00th=[24249], 90.00th=[28705], 95.00th=[32375], | 99.00th=[38536], 99.50th=[43254], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 384, max= 832, per=25.26%, avg=648.21, stdev=135.27, samples=19 iops : min= 24, max= 52, avg=40.42, stdev= 8.45, samples=19 lat (msec) : 2=1.56%, 4=34.77%, 10=9.24%, 20=27.86%, 50=26.43% lat (msec) : 100=0.13% cpu : usr=0.07%, sys=5.52%, ctx=942, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2439KiB/s (2498kB/s), 585KiB/s-662KiB/s (599kB/s-678kB/s), io=23.4MiB (24.5MB), run=9447-9820msec WRITE: bw=2566KiB/s (2628kB/s), 639KiB/s-666KiB/s (654kB/s-682kB/s), io=24.6MiB (25.8MB), run=9447-9820msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25535: Sat Feb 24 23:27:07 2024 read: IOPS=10, BW=10.0MiB/s (11.5MB/s)(7168KiB/637msec) clat (usec): min=20565, max=55505, avg=30196.94, stdev=12836.78 lat (usec): min=20566, max=55506, avg=30198.13, stdev=12836.67 clat percentiles (usec): | 1.00th=[20579], 5.00th=[20579], 10.00th=[20579], 20.00th=[20841], | 30.00th=[24249], 40.00th=[24249], 50.00th=[24773], 60.00th=[25822], | 70.00th=[25822], 80.00th=[39584], 90.00th=[55313], 95.00th=[55313], | 99.00th=[55313], 99.50th=[55313], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min=12288, max=12288, per=52.26%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 write: IOPS=7, BW=8038KiB/s (8231kB/s)(5120KiB/637msec) clat (msec): min=51, max=106, avg=84.77, stdev=25.50 lat (msec): min=51, max=106, avg=84.84, stdev=25.52 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 52], | 30.00th=[ 64], 40.00th=[ 64], 50.00th=[ 99], 60.00th=[ 99], | 70.00th=[ 105], 80.00th=[ 105], 90.00th=[ 107], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 8192, max= 8192, per=24.89%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=33.33%, 250=16.67% cpu : usr=0.16%, sys=17.77%, ctx=108, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25536: Sat Feb 24 23:27:07 2024 read: IOPS=2, BW=2599KiB/s (2661kB/s)(2048KiB/788msec) clat (msec): min=30, max=101, avg=65.76, stdev=50.35 lat (msec): min=30, max=101, avg=65.76, stdev=50.35 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 102], | 70.00th=[ 102], 80.00th=[ 102], 90.00th=[ 102], 95.00th=[ 102], | 99.00th=[ 102], 99.50th=[ 102], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 2048, max= 2048, per=8.71%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.7MiB/s (13.3MB/s)(10.0MiB/788msec) clat (msec): min=35, max=142, avg=65.44, stdev=33.52 lat (msec): min=35, max=142, avg=65.51, stdev=33.50 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 39], | 30.00th=[ 47], 40.00th=[ 49], 50.00th=[ 55], 60.00th=[ 55], | 70.00th=[ 74], 80.00th=[ 74], 90.00th=[ 105], 95.00th=[ 142], | 99.00th=[ 142], 99.50th=[ 142], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min=10240, max=10240, per=31.11%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=33.33%, 250=25.00% cpu : usr=0.00%, sys=16.90%, ctx=52, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25537: Sat Feb 24 23:27:07 2024 read: IOPS=7, BW=8116KiB/s (8311kB/s)(6144KiB/757msec) clat (msec): min=20, max=187, avg=76.94, stdev=61.16 lat (msec): min=20, max=187, avg=76.94, stdev=61.16 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 55], 50.00th=[ 55], 60.00th=[ 58], | 70.00th=[ 106], 80.00th=[ 106], 90.00th=[ 188], 95.00th=[ 188], | 99.00th=[ 188], 99.50th=[ 188], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 6144, max= 6144, per=26.13%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=7, BW=8116KiB/s (8311kB/s)(6144KiB/757msec) clat (usec): min=39646, max=70321, avg=48993.17, stdev=10879.15 lat (usec): min=39666, max=70405, avg=49052.58, stdev=10894.05 clat percentiles (usec): | 1.00th=[39584], 5.00th=[39584], 10.00th=[39584], 20.00th=[43254], | 30.00th=[43254], 40.00th=[45876], 50.00th=[45876], 60.00th=[46400], | 70.00th=[48497], 80.00th=[48497], 90.00th=[70779], 95.00th=[70779], | 99.00th=[70779], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 8192, max= 8192, per=24.89%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=25.00%, 250=16.67% cpu : usr=0.13%, sys=14.15%, ctx=47, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25538: Sat Feb 24 23:27:07 2024 read: IOPS=5, BW=5878KiB/s (6019kB/s)(5120KiB/871msec) clat (msec): min=30, max=112, avg=71.09, stdev=33.18 lat (msec): min=30, max=112, avg=71.09, stdev=33.18 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 69], 60.00th=[ 69], | 70.00th=[ 95], 80.00th=[ 95], 90.00th=[ 113], 95.00th=[ 113], | 99.00th=[ 113], 99.50th=[ 113], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 4096, max= 4096, per=17.42%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8230KiB/s (8427kB/s)(7168KiB/871msec) clat (msec): min=38, max=136, avg=73.36, stdev=31.60 lat (msec): min=38, max=136, avg=73.43, stdev=31.61 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 54], | 30.00th=[ 55], 40.00th=[ 55], 50.00th=[ 72], 60.00th=[ 78], | 70.00th=[ 78], 80.00th=[ 82], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 8192, max= 8192, per=24.89%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=25.00%, 100=58.33%, 250=16.67% cpu : usr=0.11%, sys=13.22%, ctx=37, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.0MiB/s (24.1MB/s), 2599KiB/s-10.0MiB/s (2661kB/s-11.5MB/s), io=20.0MiB (20.0MB), run=637-871msec WRITE: bw=32.1MiB/s (33.7MB/s), 8038KiB/s-12.7MiB/s (8231kB/s-13.3MB/s), io=28.0MiB (29.4MB), run=637-871msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25540: Sat Feb 24 23:27:07 2024 read: IOPS=12, BW=12.6MiB/s (13.2MB/s)(7168KiB/557msec) clat (usec): min=515, max=61664, avg=35306.08, stdev=20117.60 lat (usec): min=517, max=61665, avg=35308.90, stdev=20116.59 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 515], 10.00th=[ 515], 20.00th=[27919], | 30.00th=[30278], 40.00th=[30278], 50.00th=[32375], 60.00th=[38536], | 70.00th=[38536], 80.00th=[55837], 90.00th=[61604], 95.00th=[61604], | 99.00th=[61604], 99.50th=[61604], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min=10240, max=10240, per=44.05%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=8, BW=9192KiB/s (9413kB/s)(5120KiB/557msec) clat (msec): min=31, max=102, avg=61.71, stdev=27.50 lat (msec): min=31, max=102, avg=61.77, stdev=27.53 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 65], 60.00th=[ 65], | 70.00th=[ 68], 80.00th=[ 68], 90.00th=[ 104], 95.00th=[ 104], | 99.00th=[ 104], 99.50th=[ 104], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 8192, max= 8192, per=25.17%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=50.00%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=36.15%, ctx=97, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25541: Sat Feb 24 23:27:07 2024 read: IOPS=2, BW=2325KiB/s (2380kB/s)(2048KiB/881msec) clat (usec): min=38515, max=52801, avg=45658.47, stdev=10101.24 lat (usec): min=38517, max=52803, avg=45660.48, stdev=10101.87 clat percentiles (usec): | 1.00th=[38536], 5.00th=[38536], 10.00th=[38536], 20.00th=[38536], | 30.00th=[38536], 40.00th=[38536], 50.00th=[38536], 60.00th=[52691], | 70.00th=[52691], 80.00th=[52691], 90.00th=[52691], 95.00th=[52691], | 99.00th=[52691], 99.50th=[52691], 99.90th=[52691], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 2027, max= 2027, per=8.72%, avg=2027.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.4MiB/s (11.9MB/s)(10.0MiB/881msec) clat (msec): min=22, max=186, avg=78.09, stdev=52.38 lat (msec): min=22, max=186, avg=78.17, stdev=52.39 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 23], | 30.00th=[ 30], 40.00th=[ 44], 50.00th=[ 71], 60.00th=[ 78], | 70.00th=[ 106], 80.00th=[ 106], 90.00th=[ 122], 95.00th=[ 186], | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 8110, max= 8110, per=24.92%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=25.00%, 250=33.33% cpu : usr=0.11%, sys=30.45%, ctx=70, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25542: Sat Feb 24 23:27:07 2024 read: IOPS=10, BW=10.5MiB/s (10.0MB/s)(6144KiB/574msec) clat (usec): min=4475, max=59644, avg=40063.74, stdev=21128.50 lat (usec): min=4477, max=59645, avg=40065.15, stdev=21128.40 clat percentiles (usec): | 1.00th=[ 4490], 5.00th=[ 4490], 10.00th=[ 4490], 20.00th=[27919], | 30.00th=[27919], 40.00th=[41681], 50.00th=[41681], 60.00th=[46924], | 70.00th=[59507], 80.00th=[59507], 90.00th=[59507], 95.00th=[59507], | 99.00th=[59507], 99.50th=[59507], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min=10240, max=10240, per=44.05%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.5MiB/s (10.0MB/s)(6144KiB/574msec) clat (msec): min=21, max=134, avg=55.30, stdev=41.19 lat (msec): min=21, max=134, avg=55.36, stdev=41.20 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 34], | 30.00th=[ 34], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 42], | 70.00th=[ 66], 80.00th=[ 66], 90.00th=[ 134], 95.00th=[ 134], | 99.00th=[ 134], 99.50th=[ 134], 99.90th=[ 134], 99.95th=[ 134], | 99.99th=[ 134] bw ( KiB/s): min=10240, max=10240, per=31.47%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 10=8.33%, 50=58.33%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=40.49%, ctx=118, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25543: Sat Feb 24 23:27:07 2024 read: IOPS=6, BW=6259KiB/s (6409kB/s)(5120KiB/818msec) clat (usec): min=37960, max=66819, avg=55781.20, stdev=11597.82 lat (usec): min=37961, max=66819, avg=55782.65, stdev=11597.87 clat percentiles (usec): | 1.00th=[38011], 5.00th=[38011], 10.00th=[38011], 20.00th=[38011], | 30.00th=[50594], 40.00th=[50594], 50.00th=[60556], 60.00th=[60556], | 70.00th=[63177], 80.00th=[63177], 90.00th=[66847], 95.00th=[66847], | 99.00th=[66847], 99.50th=[66847], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 6119, max= 6119, per=26.32%, avg=6119.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8763KiB/s (8973kB/s)(7168KiB/818msec) clat (msec): min=31, max=137, avg=76.65, stdev=45.59 lat (msec): min=31, max=137, avg=76.71, stdev=45.58 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 35], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 56], 60.00th=[ 110], | 70.00th=[ 110], 80.00th=[ 125], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 8159, max= 8159, per=25.07%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=41.67%, 250=25.00% cpu : usr=0.12%, sys=29.38%, ctx=96, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.7MiB/s (23.8MB/s), 2325KiB/s-12.6MiB/s (2380kB/s-13.2MB/s), io=20.0MiB (20.0MB), run=557-881msec WRITE: bw=31.8MiB/s (33.3MB/s), 8763KiB/s-11.4MiB/s (8973kB/s-11.9MB/s), io=28.0MiB (29.4MB), run=557-881msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25549: Sat Feb 24 23:27:09 2024 read: IOPS=7, BW=30.5MiB/s (31.9MB/s)(12.0MiB/394msec) clat (msec): min=72, max=148, avg=121.56, stdev=42.16 lat (msec): min=72, max=148, avg=121.56, stdev=42.16 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 73], 10.00th=[ 73], 20.00th=[ 73], | 30.00th=[ 73], 40.00th=[ 144], 50.00th=[ 144], 60.00th=[ 144], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.25%, sys=46.56%, ctx=118, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25550: Sat Feb 24 23:27:09 2024 read: IOPS=1, BW=6884KiB/s (7049kB/s)(4096KiB/595msec) clat (nsec): min=148770k, max=148770k, avg=148769987.00, stdev= 0.00 lat (nsec): min=148772k, max=148772k, avg=148771589.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 148], 10.00th=[ 148], 20.00th=[ 148], | 30.00th=[ 148], 40.00th=[ 148], 50.00th=[ 148], 60.00th=[ 148], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 8192, max= 8192, per=29.75%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=13.4MiB/s (14.1MB/s)(8192KiB/595msec) clat (msec): min=153, max=254, avg=203.75, stdev=71.45 lat (msec): min=153, max=254, avg=203.89, stdev=71.49 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 255], | 70.00th=[ 255], 80.00th=[ 255], 90.00th=[ 255], 95.00th=[ 255], | 99.00th=[ 255], 99.50th=[ 255], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 8192, max= 8192, per=17.65%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=32.32%, ctx=108, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25551: Sat Feb 24 23:27:09 2024 write: IOPS=4, BW=16.0MiB/s (17.8MB/s)(12.0MiB/706msec) clat (msec): min=175, max=300, avg=220.00, stdev=70.29 lat (msec): min=175, max=301, avg=220.32, stdev=70.27 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 176], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 184], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 8192, max= 8192, per=17.65%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.14%, sys=35.46%, ctx=91, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25552: Sat Feb 24 23:27:09 2024 write: IOPS=5, BW=21.4MiB/s (22.5MB/s)(12.0MiB/560msec) clat (msec): min=89, max=324, avg=181.78, stdev=125.21 lat (msec): min=89, max=324, avg=182.08, stdev=125.26 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 89], 10.00th=[ 89], 20.00th=[ 89], | 30.00th=[ 89], 40.00th=[ 133], 50.00th=[ 133], 60.00th=[ 133], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min=16384, max=16384, per=35.30%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=38.82%, ctx=72, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=26.9MiB/s (28.2MB/s), 6884KiB/s-30.5MiB/s (7049kB/s-31.9MB/s), io=16.0MiB (16.8MB), run=394-595msec WRITE: bw=45.3MiB/s (47.5MB/s), 13.4MiB/s-21.4MiB/s (14.1MB/s-22.5MB/s), io=32.0MiB (33.6MB), run=560-706msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25553: Sat Feb 24 23:27:09 2024 read: IOPS=4, BW=17.2MiB/s (18.1MB/s)(12.0MiB/696msec) clat (msec): min=108, max=411, avg=231.45, stdev=159.42 lat (msec): min=108, max=411, avg=231.45, stdev=159.42 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 109], 10.00th=[ 109], 20.00th=[ 109], | 30.00th=[ 109], 40.00th=[ 176], 50.00th=[ 176], 60.00th=[ 176], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 8062, max= 8062, per=34.25%, avg=8062.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=12.23%, ctx=38, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25554: Sat Feb 24 23:27:09 2024 read: IOPS=1, BW=6187KiB/s (6336kB/s)(4096KiB/662msec) clat (nsec): min=347042k, max=347042k, avg=347041929.00, stdev= 0.00 lat (nsec): min=347044k, max=347044k, avg=347043713.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 347], 10.00th=[ 347], 20.00th=[ 347], | 30.00th=[ 347], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 347], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 8192, max= 8192, per=34.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=12.1MiB/s (12.7MB/s)(8192KiB/662msec) clat (msec): min=92, max=218, avg=155.31, stdev=88.83 lat (msec): min=92, max=218, avg=155.55, stdev=88.92 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 93], 10.00th=[ 93], 20.00th=[ 93], | 30.00th=[ 93], 40.00th=[ 93], 50.00th=[ 93], 60.00th=[ 220], | 70.00th=[ 220], 80.00th=[ 220], 90.00th=[ 220], 95.00th=[ 220], | 99.00th=[ 220], 99.50th=[ 220], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 8192, max= 8192, per=16.55%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=14.37%, ctx=18, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25555: Sat Feb 24 23:27:09 2024 write: IOPS=5, BW=21.1MiB/s (22.2MB/s)(12.0MiB/568msec) clat (msec): min=174, max=207, avg=186.63, stdev=17.97 lat (msec): min=174, max=207, avg=186.88, stdev=17.90 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 176], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 180], 50.00th=[ 180], 60.00th=[ 180], | 70.00th=[ 207], 80.00th=[ 207], 90.00th=[ 207], 95.00th=[ 207], | 99.00th=[ 207], 99.50th=[ 207], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min=16384, max=16384, per=33.10%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=19.93%, ctx=38, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25556: Sat Feb 24 23:27:09 2024 write: IOPS=4, BW=18.0MiB/s (19.9MB/s)(12.0MiB/632msec) clat (msec): min=108, max=337, avg=209.76, stdev=116.99 lat (msec): min=108, max=338, avg=209.99, stdev=116.98 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 109], 10.00th=[ 109], 20.00th=[ 109], | 30.00th=[ 109], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 184], | 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min=16384, max=16384, per=33.10%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=19.49%, ctx=72, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.0MiB/s (24.1MB/s), 6187KiB/s-17.2MiB/s (6336kB/s-18.1MB/s), io=16.0MiB (16.8MB), run=662-696msec WRITE: bw=48.3MiB/s (50.7MB/s), 12.1MiB/s-21.1MiB/s (12.7MB/s-22.2MB/s), io=32.0MiB (33.6MB), run=568-662msec