== sanity test 398b: DIO and buffer IO race ============== 07:03:33 (1715339013) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.00789 s, 49.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1079: Fri May 10 07:03:49 2024 read: IOPS=115, BW=462KiB/s (473kB/s)(5992KiB/12963msec) clat (usec): min=46, max=185363, avg=3825.17, stdev=5546.70 lat (usec): min=47, max=185364, avg=3826.36, stdev=5546.71 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 62], 10.00th=[ 75], 20.00th=[ 930], | 30.00th=[ 2343], 40.00th=[ 2737], 50.00th=[ 3163], 60.00th=[ 3687], | 70.00th=[ 4686], 80.00th=[ 5866], 90.00th=[ 7570], 95.00th=[ 8979], | 99.00th=[ 13042], 99.50th=[ 17433], 99.90th=[ 22938], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 272, max= 582, per=24.50%, avg=452.04, stdev=65.38, samples=25 iops : min= 68, max= 145, avg=112.96, stdev=16.32, samples=25 write: IOPS=121, BW=486KiB/s (497kB/s)(6296KiB/12963msec) clat (usec): min=450, max=38022, avg=4547.99, stdev=4253.35 lat (usec): min=451, max=38023, avg=4549.02, stdev=4253.37 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 529], 10.00th=[ 562], 20.00th=[ 717], | 30.00th=[ 1172], 40.00th=[ 2704], 50.00th=[ 3785], 60.00th=[ 4752], | 70.00th=[ 5800], 80.00th=[ 7373], 90.00th=[ 9765], 95.00th=[12256], | 99.00th=[19792], 99.50th=[24773], 99.90th=[33424], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 352, max= 616, per=24.54%, avg=474.16, stdev=64.06, samples=25 iops : min= 88, max= 154, avg=118.48, stdev=16.05, samples=25 lat (usec) : 50=0.26%, 100=7.10%, 250=2.08%, 500=1.30%, 750=9.99% lat (usec) : 1000=3.55% lat (msec) : 2=4.92%, 4=28.39%, 10=36.17%, 20=5.70%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=23.78%, ctx=6213, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1081: Fri May 10 07:03:49 2024 read: IOPS=115, BW=461KiB/s (472kB/s)(5940KiB/12879msec) clat (usec): min=48, max=40085, avg=3609.73, stdev=3027.03 lat (usec): min=48, max=40086, avg=3610.64, stdev=3027.69 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 61], 10.00th=[ 73], 20.00th=[ 200], | 30.00th=[ 2343], 40.00th=[ 2704], 50.00th=[ 3064], 60.00th=[ 3589], | 70.00th=[ 4424], 80.00th=[ 5735], 90.00th=[ 7373], 95.00th=[ 8717], | 99.00th=[11994], 99.50th=[12911], 99.90th=[34866], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 272, max= 600, per=24.90%, avg=459.44, stdev=75.00, samples=25 iops : min= 68, max= 150, avg=114.80, stdev=18.74, samples=25 write: IOPS=123, BW=493KiB/s (505kB/s)(6348KiB/12879msec) clat (usec): min=461, max=152640, avg=4692.68, stdev=6001.40 lat (usec): min=462, max=152641, avg=4693.65, stdev=6001.55 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 537], 10.00th=[ 578], 20.00th=[ 750], | 30.00th=[ 1123], 40.00th=[ 2573], 50.00th=[ 3752], 60.00th=[ 4686], | 70.00th=[ 5669], 80.00th=[ 7177], 90.00th=[ 10159], 95.00th=[ 13042], | 99.00th=[ 19006], 99.50th=[ 25822], 99.90th=[ 51643], 99.95th=[152044], | 99.99th=[152044] bw ( KiB/s): min= 368, max= 624, per=25.12%, avg=485.32, stdev=69.32, samples=25 iops : min= 92, max= 156, avg=121.28, stdev=17.33, samples=25 lat (usec) : 50=0.26%, 100=7.62%, 250=1.89%, 500=0.88%, 750=9.73% lat (usec) : 1000=4.26% lat (msec) : 2=4.92%, 4=29.04%, 10=34.80%, 20=6.05%, 50=0.42% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.19%, sys=23.68%, ctx=6144, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1082: Fri May 10 07:03:49 2024 read: IOPS=115, BW=463KiB/s (474kB/s)(6028KiB/13012msec) clat (usec): min=45, max=51498, avg=3712.99, stdev=3436.20 lat (usec): min=46, max=51499, avg=3714.32, stdev=3436.27 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 60], 10.00th=[ 71], 20.00th=[ 129], | 30.00th=[ 2278], 40.00th=[ 2704], 50.00th=[ 3195], 60.00th=[ 3752], | 70.00th=[ 4555], 80.00th=[ 5866], 90.00th=[ 7570], 95.00th=[ 9110], | 99.00th=[12518], 99.50th=[20579], 99.90th=[36963], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 296, max= 592, per=24.69%, avg=455.60, stdev=57.09, samples=25 iops : min= 74, max= 148, avg=113.84, stdev=14.26, samples=25 write: IOPS=120, BW=481KiB/s (493kB/s)(6260KiB/13012msec) clat (usec): min=461, max=179231, avg=4691.28, stdev=6300.32 lat (usec): min=461, max=179232, avg=4692.18, stdev=6300.37 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 570], 20.00th=[ 725], | 30.00th=[ 1156], 40.00th=[ 2671], 50.00th=[ 3687], 60.00th=[ 4555], | 70.00th=[ 5604], 80.00th=[ 7242], 90.00th=[ 10683], 95.00th=[ 12780], | 99.00th=[ 19530], 99.50th=[ 25297], 99.90th=[ 50070], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 352, max= 576, per=24.41%, avg=471.52, stdev=70.68, samples=25 iops : min= 88, max= 144, avg=117.84, stdev=17.64, samples=25 lat (usec) : 50=0.49%, 100=7.75%, 250=2.44%, 500=0.88%, 750=9.83% lat (usec) : 1000=3.78% lat (msec) : 2=4.98%, 4=28.26%, 10=34.05%, 20=6.80%, 50=0.65% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.12%, sys=23.62%, ctx=6065, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1083: Fri May 10 07:03:49 2024 read: IOPS=118, BW=476KiB/s (487kB/s)(6052KiB/12719msec) clat (usec): min=48, max=155215, avg=3774.21, stdev=5041.26 lat (usec): min=48, max=155216, avg=3775.69, stdev=5041.81 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 60], 10.00th=[ 73], 20.00th=[ 1287], | 30.00th=[ 2376], 40.00th=[ 2769], 50.00th=[ 3163], 60.00th=[ 3621], | 70.00th=[ 4359], 80.00th=[ 5604], 90.00th=[ 7308], 95.00th=[ 8979], | 99.00th=[ 12649], 99.50th=[ 16057], 99.90th=[ 47973], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 288, max= 576, per=25.75%, avg=475.08, stdev=69.37, samples=25 iops : min= 72, max= 144, avg=118.72, stdev=17.35, samples=25 write: IOPS=122, BW=490KiB/s (502kB/s)(6236KiB/12719msec) clat (usec): min=472, max=48854, avg=4448.07, stdev=4408.21 lat (usec): min=473, max=48855, avg=4449.34, stdev=4408.33 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 562], 20.00th=[ 709], | 30.00th=[ 1012], 40.00th=[ 2507], 50.00th=[ 3687], 60.00th=[ 4555], | 70.00th=[ 5669], 80.00th=[ 7046], 90.00th=[10028], 95.00th=[12387], | 99.00th=[20317], 99.50th=[25297], 99.90th=[40109], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 368, max= 584, per=25.32%, avg=489.16, stdev=51.27, samples=25 iops : min= 92, max= 146, avg=122.24, stdev=12.83, samples=25 lat (usec) : 50=0.29%, 100=7.68%, 250=1.73%, 500=0.65%, 750=10.29% lat (usec) : 1000=4.23% lat (msec) : 2=5.27%, 4=29.10%, 10=33.92%, 20=6.15%, 50=0.65% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=23.79%, ctx=6019, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1845KiB/s (1890kB/s), 461KiB/s-476KiB/s (472kB/s-487kB/s), io=23.4MiB (24.6MB), run=12719-13012msec WRITE: bw=1932KiB/s (1978kB/s), 481KiB/s-493KiB/s (493kB/s-505kB/s), io=24.6MiB (25.7MB), run=12719-13012msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1076: Fri May 10 07:04:17 2024 read: IOPS=36, BW=146KiB/s (149kB/s)(5992KiB/41062msec) clat (usec): min=1480, max=205908, avg=4370.04, stdev=6040.81 lat (usec): min=1481, max=205909, avg=4370.85, stdev=6040.82 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1909], 10.00th=[ 2024], 20.00th=[ 2245], | 30.00th=[ 2474], 40.00th=[ 2868], 50.00th=[ 3589], 60.00th=[ 4047], | 70.00th=[ 4490], 80.00th=[ 5211], 90.00th=[ 7242], 95.00th=[ 9765], | 99.00th=[ 15795], 99.50th=[ 19792], 99.90th=[ 49546], 99.95th=[206570], | 99.99th=[206570] bw ( KiB/s): min= 48, max= 272, per=25.04%, avg=145.99, stdev=55.82, samples=82 iops : min= 12, max= 68, avg=36.45, stdev=14.00, samples=82 write: IOPS=38, BW=153KiB/s (157kB/s)(6296KiB/41062msec) clat (usec): min=11313, max=70689, avg=21876.26, stdev=7892.38 lat (usec): min=11313, max=70690, avg=21877.25, stdev=7892.36 clat percentiles (usec): | 1.00th=[12125], 5.00th=[13304], 10.00th=[14091], 20.00th=[15533], | 30.00th=[16909], 40.00th=[18482], 50.00th=[20317], 60.00th=[21890], | 70.00th=[23462], 80.00th=[26084], 90.00th=[32637], 95.00th=[38536], | 99.00th=[47449], 99.50th=[53216], 99.90th=[60556], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 48, max= 208, per=25.06%, avg=153.11, stdev=40.79, samples=82 iops : min= 12, max= 52, avg=38.23, stdev=10.24, samples=82 lat (msec) : 2=4.23%, 4=24.74%, 10=17.45%, 20=26.89%, 50=26.30% lat (msec) : 100=0.36%, 250=0.03% cpu : usr=0.09%, sys=5.38%, ctx=3870, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1077: Fri May 10 07:04:17 2024 read: IOPS=36, BW=145KiB/s (148kB/s)(5940KiB/41071msec) clat (usec): min=1615, max=155858, avg=4432.14, stdev=5442.44 lat (usec): min=1616, max=155860, avg=4433.02, stdev=5442.47 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1909], 10.00th=[ 2040], 20.00th=[ 2212], | 30.00th=[ 2474], 40.00th=[ 2900], 50.00th=[ 3589], 60.00th=[ 4047], | 70.00th=[ 4621], 80.00th=[ 5473], 90.00th=[ 7177], 95.00th=[ 9634], | 99.00th=[ 15533], 99.50th=[ 20579], 99.90th=[ 68682], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 16, max= 312, per=24.79%, avg=144.52, stdev=59.15, samples=82 iops : min= 4, max= 78, avg=36.09, stdev=14.81, samples=82 write: IOPS=38, BW=155KiB/s (158kB/s)(6348KiB/41071msec) clat (usec): min=10994, max=64252, avg=21681.02, stdev=7740.52 lat (usec): min=10995, max=64253, avg=21682.03, stdev=7740.50 clat percentiles (usec): | 1.00th=[12125], 5.00th=[13173], 10.00th=[14091], 20.00th=[15401], | 30.00th=[16712], 40.00th=[18482], 50.00th=[20055], 60.00th=[21627], | 70.00th=[23200], 80.00th=[25297], 90.00th=[32637], 95.00th=[38536], | 99.00th=[46924], 99.50th=[50070], 99.90th=[58983], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 72, max= 216, per=25.27%, avg=154.38, stdev=41.14, samples=82 iops : min= 18, max= 54, avg=38.55, stdev=10.34, samples=82 lat (msec) : 2=4.26%, 4=23.93%, 10=17.84%, 20=27.51%, 50=26.04% lat (msec) : 100=0.39%, 250=0.03% cpu : usr=0.11%, sys=5.45%, ctx=3887, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1078: Fri May 10 07:04:17 2024 read: IOPS=37, BW=148KiB/s (152kB/s)(6028KiB/40664msec) clat (usec): min=1568, max=69023, avg=4280.82, stdev=3399.60 lat (usec): min=1569, max=69024, avg=4281.67, stdev=3399.64 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1926], 10.00th=[ 2040], 20.00th=[ 2245], | 30.00th=[ 2507], 40.00th=[ 2933], 50.00th=[ 3621], 60.00th=[ 4113], | 70.00th=[ 4686], 80.00th=[ 5604], 90.00th=[ 7111], 95.00th=[ 9110], | 99.00th=[13566], 99.50th=[20055], 99.90th=[60031], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 48, max= 312, per=25.42%, avg=148.19, stdev=54.24, samples=81 iops : min= 12, max= 78, avg=37.00, stdev=13.58, samples=81 write: IOPS=38, BW=154KiB/s (158kB/s)(6260KiB/40664msec) clat (msec): min=11, max=177, avg=21.81, stdev= 8.67 lat (msec): min=11, max=177, avg=21.81, stdev= 8.67 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 22], | 70.00th=[ 24], 80.00th=[ 26], 90.00th=[ 33], 95.00th=[ 38], | 99.00th=[ 50], 99.50th=[ 54], 99.90th=[ 62], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 72, max= 208, per=25.16%, avg=153.72, stdev=40.86, samples=81 iops : min= 18, max= 52, avg=38.38, stdev=10.26, samples=81 lat (msec) : 2=4.04%, 4=24.74%, 10=18.36%, 20=26.66%, 50=25.72% lat (msec) : 100=0.46%, 250=0.03% cpu : usr=0.10%, sys=5.47%, ctx=3890, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1080: Fri May 10 07:04:17 2024 read: IOPS=36, BW=147KiB/s (151kB/s)(6052KiB/41124msec) clat (usec): min=1493, max=31231, avg=4130.38, stdev=2768.54 lat (usec): min=1494, max=31232, avg=4131.28, stdev=2768.58 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1876], 10.00th=[ 1991], 20.00th=[ 2180], | 30.00th=[ 2442], 40.00th=[ 2835], 50.00th=[ 3523], 60.00th=[ 3982], | 70.00th=[ 4555], 80.00th=[ 5342], 90.00th=[ 6718], 95.00th=[ 8979], | 99.00th=[13829], 99.50th=[19268], 99.90th=[28705], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 31, max= 288, per=25.22%, avg=147.06, stdev=59.20, samples=82 iops : min= 7, max= 72, avg=36.72, stdev=14.83, samples=82 write: IOPS=37, BW=152KiB/s (155kB/s)(6236KiB/41124msec) clat (msec): min=11, max=167, avg=22.32, stdev= 8.79 lat (msec): min=11, max=167, avg=22.32, stdev= 8.79 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 21], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 27], 90.00th=[ 34], 95.00th=[ 40], | 99.00th=[ 50], 99.50th=[ 55], 99.90th=[ 59], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 72, max= 208, per=24.74%, avg=151.16, stdev=39.60, samples=82 iops : min= 18, max= 52, avg=37.74, stdev= 9.94, samples=82 lat (msec) : 2=5.14%, 4=24.45%, 10=17.68%, 20=24.97%, 50=27.28% lat (msec) : 100=0.46%, 250=0.03% cpu : usr=0.10%, sys=5.37%, ctx=3906, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=584KiB/s (598kB/s), 145KiB/s-148KiB/s (148kB/s-152kB/s), io=23.4MiB (24.6MB), run=40664-41124msec WRITE: bw=611KiB/s (626kB/s), 152KiB/s-155KiB/s (155kB/s-158kB/s), io=24.6MiB (25.7MB), run=40664-41124msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1126: Fri May 10 07:04:23 2024 read: IOPS=76, BW=1229KiB/s (1258kB/s)(5920KiB/4817msec) clat (usec): min=50, max=46408, avg=9584.44, stdev=6137.42 lat (usec): min=51, max=46409, avg=9585.09, stdev=6137.45 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 65], 10.00th=[ 75], 20.00th=[ 135], | 30.00th=[ 8586], 40.00th=[ 9634], 50.00th=[10552], 60.00th=[11338], | 70.00th=[12256], 80.00th=[13304], 90.00th=[14877], 95.00th=[16909], | 99.00th=[31065], 99.50th=[36963], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 928, max= 1568, per=24.55%, avg=1194.11, stdev=187.38, samples=9 iops : min= 58, max= 98, avg=74.44, stdev=11.79, samples=9 write: IOPS=82, BW=1322KiB/s (1354kB/s)(6368KiB/4817msec) clat (usec): min=647, max=30878, avg=3171.19, stdev=3464.99 lat (usec): min=649, max=30879, avg=3174.29, stdev=3468.01 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 775], 10.00th=[ 807], 20.00th=[ 857], | 30.00th=[ 930], 40.00th=[ 1057], 50.00th=[ 1795], 60.00th=[ 2999], | 70.00th=[ 3589], 80.00th=[ 5080], 90.00th=[ 6718], 95.00th=[ 9110], | 99.00th=[17695], 99.50th=[24773], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 704, max= 1628, per=25.41%, avg=1300.44, stdev=299.68, samples=9 iops : min= 44, max= 101, avg=81.11, stdev=18.56, samples=9 lat (usec) : 100=8.20%, 250=1.82%, 750=1.95%, 1000=17.32% lat (msec) : 2=7.03%, 4=12.50%, 10=22.40%, 20=27.21%, 50=1.56% cpu : usr=0.17%, sys=22.80%, ctx=2206, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1128: Fri May 10 07:04:23 2024 read: IOPS=74, BW=1194KiB/s (1223kB/s)(5744KiB/4810msec) clat (usec): min=54, max=53158, avg=9611.24, stdev=6265.54 lat (usec): min=55, max=53159, avg=9611.84, stdev=6265.55 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 64], 10.00th=[ 70], 20.00th=[ 133], | 30.00th=[ 8586], 40.00th=[ 9634], 50.00th=[10814], 60.00th=[11731], | 70.00th=[12649], 80.00th=[13698], 90.00th=[15533], 95.00th=[17433], | 99.00th=[25297], 99.50th=[27395], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 800, max= 1472, per=24.26%, avg=1180.11, stdev=235.64, samples=9 iops : min= 50, max= 92, avg=73.67, stdev=14.80, samples=9 write: IOPS=85, BW=1360KiB/s (1393kB/s)(6544KiB/4810msec) clat (usec): min=673, max=72649, avg=3306.15, stdev=4663.79 lat (usec): min=674, max=72650, avg=3307.79, stdev=4663.52 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 742], 10.00th=[ 807], 20.00th=[ 865], | 30.00th=[ 963], 40.00th=[ 1270], 50.00th=[ 1975], 60.00th=[ 3032], | 70.00th=[ 3687], 80.00th=[ 5145], 90.00th=[ 6718], 95.00th=[ 8717], | 99.00th=[15926], 99.50th=[16712], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 1024, max= 1728, per=25.77%, avg=1318.67, stdev=244.72, samples=9 iops : min= 64, max= 108, avg=82.33, stdev=15.22, samples=9 lat (usec) : 100=8.59%, 250=1.30%, 500=0.39%, 750=2.99%, 1000=14.19% lat (msec) : 2=9.77%, 4=12.89%, 10=20.44%, 20=27.99%, 50=1.17% lat (msec) : 100=0.26% cpu : usr=0.15%, sys=22.23%, ctx=2096, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1130: Fri May 10 07:04:23 2024 read: IOPS=79, BW=1271KiB/s (1301kB/s)(6256KiB/4924msec) clat (usec): min=50, max=52390, avg=9279.61, stdev=6285.02 lat (usec): min=50, max=52391, avg=9280.25, stdev=6285.04 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 65], 10.00th=[ 71], 20.00th=[ 114], | 30.00th=[ 8586], 40.00th=[ 9634], 50.00th=[10421], 60.00th=[11207], | 70.00th=[12387], 80.00th=[13435], 90.00th=[15008], 95.00th=[17433], | 99.00th=[22676], 99.50th=[31065], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 768, max= 1920, per=25.36%, avg=1233.56, stdev=341.51, samples=9 iops : min= 48, max= 120, avg=77.00, stdev=21.42, samples=9 write: IOPS=76, BW=1225KiB/s (1254kB/s)(6032KiB/4924msec) clat (usec): min=654, max=57173, avg=3419.32, stdev=4636.89 lat (usec): min=656, max=57175, avg=3421.22, stdev=4636.82 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 750], 10.00th=[ 807], 20.00th=[ 873], | 30.00th=[ 955], 40.00th=[ 1172], 50.00th=[ 1614], 60.00th=[ 2835], | 70.00th=[ 3884], 80.00th=[ 5538], 90.00th=[ 6980], 95.00th=[ 8979], | 99.00th=[19792], 99.50th=[37487], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 670, max= 1664, per=23.20%, avg=1187.33, stdev=282.37, samples=9 iops : min= 41, max= 104, avg=74.11, stdev=17.85, samples=9 lat (usec) : 100=9.11%, 250=2.86%, 500=0.39%, 750=2.21%, 1000=14.45% lat (msec) : 2=9.37%, 4=8.59%, 10=22.92%, 20=28.12%, 50=1.69% lat (msec) : 100=0.26% cpu : usr=0.22%, sys=21.75%, ctx=2179, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1131: Fri May 10 07:04:23 2024 read: IOPS=77, BW=1232KiB/s (1262kB/s)(6032KiB/4896msec) clat (usec): min=52, max=41519, avg=9541.10, stdev=6266.54 lat (usec): min=52, max=41520, avg=9541.72, stdev=6266.56 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 65], 10.00th=[ 78], 20.00th=[ 137], | 30.00th=[ 8717], 40.00th=[ 9765], 50.00th=[10814], 60.00th=[11469], | 70.00th=[12518], 80.00th=[13304], 90.00th=[15008], 95.00th=[17957], | 99.00th=[26870], 99.50th=[39584], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 928, max= 1472, per=24.19%, avg=1176.56, stdev=189.51, samples=9 iops : min= 58, max= 92, avg=73.44, stdev=11.76, samples=9 write: IOPS=79, BW=1278KiB/s (1308kB/s)(6256KiB/4896msec) clat (usec): min=636, max=66758, avg=3303.79, stdev=4642.05 lat (usec): min=637, max=66760, avg=3304.93, stdev=4642.09 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 750], 10.00th=[ 791], 20.00th=[ 881], | 30.00th=[ 955], 40.00th=[ 1106], 50.00th=[ 1631], 60.00th=[ 2835], | 70.00th=[ 3621], 80.00th=[ 5276], 90.00th=[ 6783], 95.00th=[ 9634], | 99.00th=[19006], 99.50th=[32113], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 926, max= 1440, per=24.25%, avg=1240.67, stdev=174.83, samples=9 iops : min= 57, max= 90, avg=77.44, stdev=11.13, samples=9 lat (usec) : 100=8.20%, 250=2.34%, 500=0.26%, 750=2.73%, 1000=14.84% lat (msec) : 2=8.85%, 4=10.81%, 10=21.35%, 20=28.91%, 50=1.56% lat (msec) : 100=0.13% cpu : usr=0.20%, sys=21.98%, ctx=2136, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4864KiB/s (4981kB/s), 1194KiB/s-1271KiB/s (1223kB/s-1301kB/s), io=23.4MiB (24.5MB), run=4810-4924msec WRITE: bw=5118KiB/s (5241kB/s), 1225KiB/s-1360KiB/s (1254kB/s-1393kB/s), io=24.6MiB (25.8MB), run=4810-4924msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1127: Fri May 10 07:04:28 2024 read: IOPS=35, BW=568KiB/s (582kB/s)(5920KiB/10419msec) clat (usec): min=1949, max=44241, avg=4557.57, stdev=4343.51 lat (usec): min=1949, max=44243, avg=4558.30, stdev=4343.52 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2147], 10.00th=[ 2245], 20.00th=[ 2376], | 30.00th=[ 2507], 40.00th=[ 2638], 50.00th=[ 2868], 60.00th=[ 3163], | 70.00th=[ 3785], 80.00th=[ 5669], 90.00th=[ 8979], 95.00th=[14091], | 99.00th=[20841], 99.50th=[25822], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 320, max= 862, per=24.35%, avg=558.30, stdev=161.47, samples=20 iops : min= 20, max= 53, avg=34.85, stdev=10.01, samples=20 write: IOPS=38, BW=611KiB/s (626kB/s)(6368KiB/10419msec) clat (usec): min=12381, max=66364, avg=21919.28, stdev=7150.60 lat (usec): min=12382, max=66367, avg=21921.09, stdev=7150.68 clat percentiles (usec): | 1.00th=[12649], 5.00th=[13435], 10.00th=[14222], 20.00th=[15533], | 30.00th=[17433], 40.00th=[19268], 50.00th=[21103], 60.00th=[22414], | 70.00th=[23987], 80.00th=[25822], 90.00th=[31065], 95.00th=[34341], | 99.00th=[46400], 99.50th=[56361], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 415, max= 800, per=24.94%, avg=601.55, stdev=142.65, samples=20 iops : min= 25, max= 50, avg=37.55, stdev= 8.98, samples=20 lat (msec) : 2=0.13%, 4=34.64%, 10=8.98%, 20=26.17%, 50=29.82% lat (msec) : 100=0.26% cpu : usr=0.09%, sys=5.95%, ctx=885, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1129: Fri May 10 07:04:28 2024 read: IOPS=34, BW=550KiB/s (563kB/s)(5744KiB/10444msec) clat (usec): min=1978, max=26414, avg=4118.49, stdev=3620.95 lat (usec): min=1978, max=26414, avg=4119.19, stdev=3620.95 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2114], 10.00th=[ 2212], 20.00th=[ 2311], | 30.00th=[ 2442], 40.00th=[ 2573], 50.00th=[ 2835], 60.00th=[ 3064], | 70.00th=[ 3523], 80.00th=[ 4359], 90.00th=[ 7570], 95.00th=[11863], | 99.00th=[22152], 99.50th=[24511], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 192, max= 1120, per=24.07%, avg=551.95, stdev=236.31, samples=20 iops : min= 12, max= 70, avg=34.45, stdev=14.81, samples=20 write: IOPS=39, BW=627KiB/s (642kB/s)(6544KiB/10444msec) clat (msec): min=12, max=116, avg=21.90, stdev= 8.14 lat (msec): min=12, max=116, avg=21.90, stdev= 8.14 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 21], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 27], 90.00th=[ 31], 95.00th=[ 35], | 99.00th=[ 46], 99.50th=[ 50], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 416, max= 864, per=25.54%, avg=615.95, stdev=129.69, samples=20 iops : min= 26, max= 54, avg=38.45, stdev= 8.17, samples=20 lat (msec) : 2=0.13%, 4=35.42%, 10=7.68%, 20=27.86%, 50=28.65% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.08%, sys=5.91%, ctx=875, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1132: Fri May 10 07:04:28 2024 read: IOPS=39, BW=635KiB/s (651kB/s)(6256KiB/9847msec) clat (usec): min=1886, max=32206, avg=4368.55, stdev=3912.63 lat (usec): min=1887, max=32207, avg=4369.39, stdev=3912.56 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2114], 10.00th=[ 2245], 20.00th=[ 2376], | 30.00th=[ 2507], 40.00th=[ 2704], 50.00th=[ 2966], 60.00th=[ 3294], | 70.00th=[ 3752], 80.00th=[ 5211], 90.00th=[ 7832], 95.00th=[11600], | 99.00th=[24773], 99.50th=[29754], 99.90th=[32113], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 320, max= 1248, per=27.91%, avg=639.89, stdev=247.46, samples=19 iops : min= 20, max= 78, avg=39.95, stdev=15.49, samples=19 write: IOPS=38, BW=613KiB/s (627kB/s)(6032KiB/9847msec) clat (usec): min=12061, max=54213, avg=21565.12, stdev=7028.78 lat (usec): min=12063, max=54215, avg=21566.73, stdev=7028.91 clat percentiles (usec): | 1.00th=[12256], 5.00th=[13173], 10.00th=[13829], 20.00th=[15533], | 30.00th=[17171], 40.00th=[19006], 50.00th=[20579], 60.00th=[21890], | 70.00th=[23462], 80.00th=[26084], 90.00th=[30802], 95.00th=[35914], | 99.00th=[44827], 99.50th=[47973], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 352, max= 864, per=25.07%, avg=604.58, stdev=148.97, samples=19 iops : min= 22, max= 54, avg=37.74, stdev= 9.37, samples=19 lat (msec) : 2=0.52%, 4=36.85%, 10=9.51%, 20=25.78%, 50=27.21% lat (msec) : 100=0.13% cpu : usr=0.09%, sys=6.16%, ctx=882, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1133: Fri May 10 07:04:28 2024 read: IOPS=36, BW=588KiB/s (602kB/s)(6032KiB/10252msec) clat (usec): min=1939, max=33297, avg=4344.34, stdev=3890.20 lat (usec): min=1940, max=33298, avg=4345.05, stdev=3890.20 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2147], 10.00th=[ 2245], 20.00th=[ 2376], | 30.00th=[ 2540], 40.00th=[ 2671], 50.00th=[ 2868], 60.00th=[ 3228], | 70.00th=[ 3720], 80.00th=[ 5211], 90.00th=[ 8029], 95.00th=[12518], | 99.00th=[23987], 99.50th=[25035], 99.90th=[33424], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 320, max= 1120, per=25.17%, avg=577.25, stdev=198.71, samples=20 iops : min= 20, max= 70, avg=35.90, stdev=12.42, samples=20 write: IOPS=38, BW=610KiB/s (625kB/s)(6256KiB/10252msec) clat (usec): min=11521, max=68844, avg=22010.57, stdev=7496.74 lat (usec): min=11523, max=68848, avg=22012.25, stdev=7496.66 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13304], 10.00th=[13960], 20.00th=[15664], | 30.00th=[17695], 40.00th=[19530], 50.00th=[21103], 60.00th=[22414], | 70.00th=[23725], 80.00th=[26608], 90.00th=[31327], 95.00th=[35390], | 99.00th=[48497], 99.50th=[53740], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 383, max= 832, per=25.13%, avg=606.10, stdev=152.53, samples=20 iops : min= 23, max= 52, avg=37.70, stdev= 9.66, samples=20 lat (msec) : 2=0.39%, 4=35.94%, 10=8.98%, 20=25.91%, 50=28.39% lat (msec) : 100=0.39% cpu : usr=0.04%, sys=5.91%, ctx=870, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2293KiB/s (2348kB/s), 550KiB/s-635KiB/s (563kB/s-651kB/s), io=23.4MiB (24.5MB), run=9847-10444msec WRITE: bw=2413KiB/s (2471kB/s), 610KiB/s-627KiB/s (625kB/s-642kB/s), io=24.6MiB (25.8MB), run=9847-10444msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1148: Fri May 10 07:04:33 2024 read: IOPS=5, BW=5175KiB/s (5300kB/s)(7168KiB/1385msec) clat (msec): min=18, max=938, avg=162.76, stdev=342.37 lat (msec): min=18, max=938, avg=162.76, stdev=342.37 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 27], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 30], 60.00th=[ 37], | 70.00th=[ 37], 80.00th=[ 64], 90.00th=[ 936], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 6144, max= 6144, per=100.00%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3697KiB/s (3785kB/s)(5120KiB/1385msec) clat (usec): min=34075, max=77084, avg=48784.43, stdev=18178.11 lat (usec): min=34141, max=77205, avg=48849.75, stdev=18202.86 clat percentiles (usec): | 1.00th=[33817], 5.00th=[33817], 10.00th=[33817], 20.00th=[33817], | 30.00th=[37487], 40.00th=[37487], 50.00th=[38011], 60.00th=[38011], | 70.00th=[56886], 80.00th=[56886], 90.00th=[77071], 95.00th=[77071], | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 2048, max= 2048, per=25.77%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=58.33%, 100=25.00%, 1000=8.33% cpu : usr=0.00%, sys=6.21%, ctx=38, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1149: Fri May 10 07:04:33 2024 read: IOPS=0, BW=839KiB/s (859kB/s)(2048KiB/2442msec) clat (usec): min=19112, max=26791, avg=22952.33, stdev=5429.65 lat (usec): min=19113, max=26792, avg=22953.16, stdev=5429.70 clat percentiles (usec): | 1.00th=[19006], 5.00th=[19006], 10.00th=[19006], 20.00th=[19006], | 30.00th=[19006], 40.00th=[19006], 50.00th=[19006], 60.00th=[26870], | 70.00th=[26870], 80.00th=[26870], 90.00th=[26870], 95.00th=[26870], | 99.00th=[26870], 99.50th=[26870], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 2048, max= 2048, per=36.08%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4193KiB/s (4294kB/s)(10.0MiB/2442msec) clat (msec): min=28, max=986, avg=239.41, stdev=386.73 lat (msec): min=28, max=986, avg=239.47, stdev=386.72 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 34], | 30.00th=[ 35], 40.00th=[ 36], 50.00th=[ 37], 60.00th=[ 66], | 70.00th=[ 128], 80.00th=[ 128], 90.00th=[ 953], 95.00th=[ 986], | 99.00th=[ 986], 99.50th=[ 986], 99.90th=[ 986], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 4096, max=12288, per=100.00%, avg=8192.00, stdev=5792.62, samples=2 iops : min= 4, max= 12, avg= 8.00, stdev= 5.66, samples=2 lat (msec) : 20=8.33%, 50=50.00%, 100=16.67%, 250=8.33%, 1000=16.67% cpu : usr=0.08%, sys=5.53%, ctx=47, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1150: Fri May 10 07:04:33 2024 read: IOPS=1, BW=1703KiB/s (1744kB/s)(6144KiB/3608msec) clat (msec): min=27, max=787, avg=302.74, stdev=361.06 lat (msec): min=27, max=787, avg=302.74, stdev=361.06 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 45], | 30.00th=[ 45], 40.00th=[ 62], 50.00th=[ 62], 60.00th=[ 153], | 70.00th=[ 743], 80.00th=[ 743], 90.00th=[ 785], 95.00th=[ 785], | 99.00th=[ 785], 99.50th=[ 785], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 2048, max= 6144, per=60.14%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 write: IOPS=1, BW=1703KiB/s (1744kB/s)(6144KiB/3608msec) clat (msec): min=47, max=1062, avg=298.38, stdev=413.16 lat (msec): min=47, max=1062, avg=298.44, stdev=413.18 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 54], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 57], 60.00th=[ 75], | 70.00th=[ 498], 80.00th=[ 498], 90.00th=[ 1062], 95.00th=[ 1062], | 99.00th=[ 1062], 99.50th=[ 1062], 99.90th=[ 1062], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 4096, max= 4096, per=51.55%, avg=4096.00, stdev= 0.00, samples=3 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=3 lat (msec) : 50=25.00%, 100=33.33%, 250=8.33%, 500=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=3.58%, ctx=137, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1151: Fri May 10 07:04:33 2024 read: IOPS=1, BW=1430KiB/s (1464kB/s)(5120KiB/3580msec) clat (msec): min=16, max=851, avg=342.11, stdev=433.19 lat (msec): min=16, max=851, avg=342.11, stdev=433.19 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 17], | 30.00th=[ 23], 40.00th=[ 23], 50.00th=[ 40], 60.00th=[ 40], | 70.00th=[ 776], 80.00th=[ 776], 90.00th=[ 852], 95.00th=[ 852], | 99.00th=[ 852], 99.50th=[ 852], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 2048, max= 6144, per=72.16%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=1, BW=2002KiB/s (2050kB/s)(7168KiB/3580msec) clat (msec): min=30, max=1142, avg=266.83, stdev=421.08 lat (msec): min=30, max=1142, avg=266.89, stdev=421.09 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 34], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 45], 60.00th=[ 85], | 70.00th=[ 85], 80.00th=[ 498], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 2048, max= 6144, per=60.08%, avg=4774.33, stdev=2361.08, samples=3 iops : min= 2, max= 6, avg= 4.33, stdev= 2.08, samples=3 lat (msec) : 20=8.33%, 50=50.00%, 100=8.33%, 500=8.33%, 1000=16.67% cpu : usr=0.00%, sys=3.05%, ctx=35, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5676KiB/s (5813kB/s), 839KiB/s-5175KiB/s (859kB/s-5300kB/s), io=20.0MiB (20.0MB), run=1385-3608msec WRITE: bw=7947KiB/s (8138kB/s), 1703KiB/s-4193KiB/s (1744kB/s-4294kB/s), io=28.0MiB (29.4MB), run=1385-3608msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1152: Fri May 10 07:04:33 2024 read: IOPS=1, BW=1659KiB/s (1699kB/s)(7168KiB/4320msec) clat (usec): min=580, max=1072.2k, avg=586723.33, stdev=437354.84 lat (usec): min=580, max=1072.2k, avg=586724.42, stdev=437355.13 clat percentiles (usec): | 1.00th=[ 578], 5.00th=[ 578], 10.00th=[ 578], | 20.00th=[ 63177], 30.00th=[ 367002], 40.00th=[ 367002], | 50.00th=[ 801113], 60.00th=[ 859833], 70.00th=[ 859833], | 80.00th=[ 943719], 90.00th=[1069548], 95.00th=[1069548], | 99.00th=[1069548], 99.50th=[1069548], 99.90th=[1069548], | 99.95th=[1069548], 99.99th=[1069548] bw ( KiB/s): min= 1712, max= 4087, per=52.19%, avg=2473.75, stdev=1087.10, samples=4 iops : min= 1, max= 3, avg= 2.00, stdev= 0.82, samples=4 write: IOPS=1, BW=1185KiB/s (1214kB/s)(5120KiB/4320msec) clat (usec): min=25229, max=61623, avg=42318.15, stdev=13018.98 lat (usec): min=25325, max=61651, avg=42415.87, stdev=12994.93 clat percentiles (usec): | 1.00th=[25297], 5.00th=[25297], 10.00th=[25297], 20.00th=[25297], | 30.00th=[39060], 40.00th=[39060], 50.00th=[41681], 60.00th=[41681], | 70.00th=[43779], 80.00th=[43779], 90.00th=[61604], 95.00th=[61604], | 99.00th=[61604], 99.50th=[61604], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 2043, max= 5137, per=54.09%, avg=3590.00, stdev=2187.79, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 lat (usec) : 750=8.33% lat (msec) : 50=33.33%, 100=16.67%, 500=8.33%, 1000=25.00% cpu : usr=0.05%, sys=17.48%, ctx=1266, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1153: Fri May 10 07:04:33 2024 read: IOPS=0, BW=582KiB/s (596kB/s)(2048KiB/3519msec) clat (msec): min=47, max=1005, avg=526.45, stdev=677.11 lat (msec): min=47, max=1005, avg=526.45, stdev=677.11 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 48], 50.00th=[ 48], 60.00th=[ 1003], | 70.00th=[ 1003], 80.00th=[ 1003], 90.00th=[ 1003], 95.00th=[ 1003], | 99.00th=[ 1003], 99.50th=[ 1003], 99.90th=[ 1003], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 2048, max= 2048, per=43.21%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2910KiB/s (2980kB/s)(10.0MiB/3519msec) clat (msec): min=24, max=977, avg=246.23, stdev=344.54 lat (msec): min=24, max=978, avg=246.30, stdev=344.55 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 27], | 30.00th=[ 28], 40.00th=[ 37], 50.00th=[ 65], 60.00th=[ 120], | 70.00th=[ 275], 80.00th=[ 275], 90.00th=[ 776], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 2043, max= 6144, per=69.41%, avg=4606.75, stdev=1962.99, samples=4 iops : min= 1, max= 6, avg= 4.25, stdev= 2.36, samples=4 lat (msec) : 50=41.67%, 100=8.33%, 250=16.67%, 500=8.33%, 1000=16.67% cpu : usr=0.03%, sys=11.88%, ctx=376, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1154: Fri May 10 07:04:33 2024 read: IOPS=1, BW=1787KiB/s (1829kB/s)(6144KiB/3439msec) clat (usec): min=431, max=816276, avg=345971.65, stdev=359550.85 lat (usec): min=432, max=816277, avg=345972.50, stdev=359550.68 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 433], 10.00th=[ 433], 20.00th=[ 49546], | 30.00th=[ 49546], 40.00th=[ 84411], 50.00th=[ 84411], 60.00th=[396362], | 70.00th=[734004], 80.00th=[734004], 90.00th=[817890], 95.00th=[817890], | 99.00th=[817890], 99.50th=[817890], 99.90th=[817890], 99.95th=[817890], | 99.99th=[817890] bw ( KiB/s): min= 2048, max= 6144, per=72.01%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 write: IOPS=1, BW=1787KiB/s (1829kB/s)(6144KiB/3439msec) clat (msec): min=31, max=1068, avg=226.49, stdev=412.95 lat (msec): min=31, max=1068, avg=226.56, stdev=412.94 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 68], 50.00th=[ 68], 60.00th=[ 68], | 70.00th=[ 93], 80.00th=[ 93], 90.00th=[ 1070], 95.00th=[ 1070], | 99.00th=[ 1070], 99.50th=[ 1070], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 2048, max= 4096, per=46.29%, avg=3072.00, stdev=1182.41, samples=4 iops : min= 2, max= 4, avg= 3.00, stdev= 1.15, samples=4 lat (usec) : 500=8.33% lat (msec) : 50=25.00%, 100=33.33%, 500=8.33%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=11.40%, ctx=713, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1155: Fri May 10 07:04:33 2024 read: IOPS=1, BW=1493KiB/s (1529kB/s)(5120KiB/3430msec) clat (usec): min=655, max=847064, avg=349036.46, stdev=450988.64 lat (usec): min=656, max=847065, avg=349037.61, stdev=450988.32 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 660], 10.00th=[ 660], 20.00th=[ 660], | 30.00th=[ 1090], 40.00th=[ 1090], 50.00th=[ 58459], 60.00th=[ 58459], | 70.00th=[834667], 80.00th=[834667], 90.00th=[843056], 95.00th=[843056], | 99.00th=[843056], 99.50th=[843056], 99.90th=[843056], 99.95th=[843056], | 99.99th=[843056] bw ( KiB/s): min= 2048, max= 6144, per=86.41%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=2, BW=2090KiB/s (2140kB/s)(7168KiB/3430msec) clat (msec): min=45, max=1017, avg=240.43, stdev=360.30 lat (msec): min=45, max=1017, avg=240.48, stdev=360.30 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 50], | 30.00th=[ 55], 40.00th=[ 55], 50.00th=[ 75], 60.00th=[ 85], | 70.00th=[ 85], 80.00th=[ 359], 90.00th=[ 1020], 95.00th=[ 1020], | 99.00th=[ 1020], 99.50th=[ 1020], 99.90th=[ 1020], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 2048, max= 6144, per=54.00%, avg=3584.00, stdev=1960.81, samples=4 iops : min= 2, max= 6, avg= 3.50, stdev= 1.91, samples=4 lat (usec) : 750=8.33% lat (msec) : 2=8.33%, 50=16.67%, 100=33.33%, 500=8.33%, 1000=16.67% cpu : usr=0.00%, sys=12.92%, ctx=709, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4741KiB/s (4855kB/s), 582KiB/s-1787KiB/s (596kB/s-1829kB/s), io=20.0MiB (20.0MB), run=3430-4320msec WRITE: bw=6637KiB/s (6796kB/s), 1185KiB/s-2910KiB/s (1214kB/s-2980kB/s), io=28.0MiB (29.4MB), run=3430-4320msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1168: Fri May 10 07:04:38 2024 read: IOPS=0, BW=4050KiB/s (4147kB/s)(12.0MiB/3034msec) clat (msec): min=191, max=1732, avg=1010.21, stdev=775.02 lat (msec): min=191, max=1732, avg=1010.21, stdev=775.03 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 192], 10.00th=[ 192], 20.00th=[ 192], | 30.00th=[ 192], 40.00th=[ 1099], 50.00th=[ 1099], 60.00th=[ 1099], | 70.00th=[ 1737], 80.00th=[ 1737], 90.00th=[ 1737], 95.00th=[ 1737], | 99.00th=[ 1737], 99.50th=[ 1737], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=47.18%, ctx=710, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1170: Fri May 10 07:04:38 2024 read: IOPS=0, BW=1139KiB/s (1166kB/s)(4096KiB/3596msec) clat (nsec): min=224984k, max=224984k, avg=224983537.00, stdev= 0.00 lat (nsec): min=224984k, max=224984k, avg=224984304.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 226], 10.00th=[ 226], 20.00th=[ 226], | 30.00th=[ 226], 40.00th=[ 226], 50.00th=[ 226], 60.00th=[ 226], | 70.00th=[ 226], 80.00th=[ 226], 90.00th=[ 226], 95.00th=[ 226], | 99.00th=[ 226], 99.50th=[ 226], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2278KiB/s (2333kB/s)(8192KiB/3596msec) clat (msec): min=267, max=3101, avg=1684.60, stdev=2003.78 lat (msec): min=267, max=3101, avg=1684.79, stdev=2003.91 clat percentiles (msec): | 1.00th=[ 268], 5.00th=[ 268], 10.00th=[ 268], 20.00th=[ 268], | 30.00th=[ 268], 40.00th=[ 268], 50.00th=[ 268], 60.00th=[ 3104], | 70.00th=[ 3104], 80.00th=[ 3104], 90.00th=[ 3104], 95.00th=[ 3104], | 99.00th=[ 3104], 99.50th=[ 3104], 99.90th=[ 3104], 99.95th=[ 3104], | 99.99th=[ 3104] bw ( KiB/s): min= 8192, max= 8192, per=94.51%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=6.59%, ctx=59, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1171: Fri May 10 07:04:38 2024 write: IOPS=0, BW=3251KiB/s (3329kB/s)(12.0MiB/3780msec) clat (msec): min=104, max=3115, avg=1256.89, stdev=1625.04 lat (msec): min=104, max=3115, avg=1257.11, stdev=1625.00 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 105], 10.00th=[ 105], 20.00th=[ 105], | 30.00th=[ 105], 40.00th=[ 550], 50.00th=[ 550], 60.00th=[ 550], | 70.00th=[ 3104], 80.00th=[ 3104], 90.00th=[ 3104], 95.00th=[ 3104], | 99.00th=[ 3104], 99.50th=[ 3104], 99.90th=[ 3104], 99.95th=[ 3104], | 99.99th=[ 3104] bw ( KiB/s): min= 8192, max= 8192, per=94.51%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=6.54%, ctx=28, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1172: Fri May 10 07:04:38 2024 write: IOPS=0, BW=3387KiB/s (3468kB/s)(12.0MiB/3628msec) clat (msec): min=85, max=3119, avg=1208.18, stdev=1663.84 lat (msec): min=85, max=3120, avg=1208.52, stdev=1663.88 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 86], 10.00th=[ 86], 20.00th=[ 86], | 30.00th=[ 86], 40.00th=[ 418], 50.00th=[ 418], 60.00th=[ 418], | 70.00th=[ 3104], 80.00th=[ 3104], 90.00th=[ 3104], 95.00th=[ 3104], | 99.00th=[ 3104], 99.50th=[ 3104], 99.90th=[ 3104], 99.95th=[ 3104], | 99.99th=[ 3104] bw ( KiB/s): min= 8175, max= 8175, per=94.31%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=7.11%, ctx=32, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4556KiB/s (4666kB/s), 1139KiB/s-4050KiB/s (1166kB/s-4147kB/s), io=16.0MiB (16.8MB), run=3034-3596msec WRITE: bw=8669KiB/s (8877kB/s), 2278KiB/s-3387KiB/s (2333kB/s-3468kB/s), io=32.0MiB (33.6MB), run=3596-3780msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1165: Fri May 10 07:04:38 2024 read: IOPS=0, BW=3972KiB/s (4067kB/s)(12.0MiB/3094msec) clat (msec): min=47, max=2610, avg=1021.34, stdev=1387.82 lat (msec): min=47, max=2610, avg=1021.34, stdev=1387.82 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 405], | 70.00th=[ 2601], 80.00th=[ 2601], 90.00th=[ 2601], 95.00th=[ 2601], | 99.00th=[ 2601], 99.50th=[ 2601], 99.90th=[ 2601], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 500=33.33% cpu : usr=0.00%, sys=3.33%, ctx=34, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1166: Fri May 10 07:04:38 2024 read: IOPS=0, BW=1068KiB/s (1093kB/s)(4096KiB/3836msec) clat (nsec): min=362533k, max=362533k, avg=362532845.00, stdev= 0.00 lat (nsec): min=362534k, max=362534k, avg=362533623.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 363], | 30.00th=[ 363], 40.00th=[ 363], 50.00th=[ 363], 60.00th=[ 363], | 70.00th=[ 363], 80.00th=[ 363], 90.00th=[ 363], 95.00th=[ 363], | 99.00th=[ 363], 99.50th=[ 363], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2136KiB/s (2187kB/s)(8192KiB/3836msec) clat (msec): min=1014, max=2457, avg=1736.07, stdev=1019.87 lat (msec): min=1015, max=2457, avg=1736.46, stdev=1020.04 clat percentiles (msec): | 1.00th=[ 1011], 5.00th=[ 1011], 10.00th=[ 1011], 20.00th=[ 1011], | 30.00th=[ 1011], 40.00th=[ 1011], 50.00th=[ 1011], 60.00th=[ 2467], | 70.00th=[ 2467], 80.00th=[ 2467], 90.00th=[ 2467], 95.00th=[ 2467], | 99.00th=[ 2467], 99.50th=[ 2467], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 5595, max= 5595, per=66.15%, avg=5595.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=2.76%, ctx=30, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1167: Fri May 10 07:04:38 2024 write: IOPS=0, BW=3172KiB/s (3248kB/s)(12.0MiB/3874msec) clat (msec): min=74, max=3098, avg=1283.76, stdev=1600.38 lat (msec): min=75, max=3099, avg=1284.06, stdev=1600.52 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 75], 10.00th=[ 75], 20.00th=[ 75], | 30.00th=[ 75], 40.00th=[ 676], 50.00th=[ 676], 60.00th=[ 676], | 70.00th=[ 3104], 80.00th=[ 3104], 90.00th=[ 3104], 95.00th=[ 3104], | 99.00th=[ 3104], 99.50th=[ 3104], 99.90th=[ 3104], 99.95th=[ 3104], | 99.99th=[ 3104] bw ( KiB/s): min= 8126, max= 8126, per=96.07%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.05%, sys=3.54%, ctx=73, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1169: Fri May 10 07:04:38 2024 write: IOPS=3, BW=14.5MiB/s (15.3MB/s)(12.0MiB/825msec) clat (msec): min=85, max=567, avg=256.25, stdev=269.70 lat (msec): min=85, max=567, avg=256.43, stdev=269.66 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 86], 10.00th=[ 86], 20.00th=[ 86], | 30.00th=[ 86], 40.00th=[ 117], 50.00th=[ 117], 60.00th=[ 117], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=13.35%, ctx=37, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4271KiB/s (4374kB/s), 1068KiB/s-3972KiB/s (1093kB/s-4067kB/s), io=16.0MiB (16.8MB), run=3094-3836msec WRITE: bw=8458KiB/s (8661kB/s), 2136KiB/s-14.5MiB/s (2187kB/s-15.3MB/s), io=32.0MiB (33.6MB), run=825-3874msec