== sanity test 398b: DIO and buffer IO race ============== 07:01:01 (1715338861) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.43109 s, 35.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32709: Fri May 10 07:01:17 2024 read: IOPS=110, BW=443KiB/s (453kB/s)(5992KiB/13532msec) clat (usec): min=47, max=206809, avg=3981.08, stdev=6370.61 lat (usec): min=47, max=206811, avg=3982.45, stdev=6370.53 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 61], 10.00th=[ 71], 20.00th=[ 108], | 30.00th=[ 2311], 40.00th=[ 2704], 50.00th=[ 3228], 60.00th=[ 3752], | 70.00th=[ 4621], 80.00th=[ 6325], 90.00th=[ 8291], 95.00th=[ 9896], | 99.00th=[ 16319], 99.50th=[ 25297], 99.90th=[ 33817], 99.95th=[206570], | 99.99th=[206570] bw ( KiB/s): min= 184, max= 640, per=24.93%, avg=442.30, stdev=107.88, samples=27 iops : min= 46, max= 160, avg=110.56, stdev=26.94, samples=27 write: IOPS=116, BW=465KiB/s (476kB/s)(6296KiB/13532msec) clat (usec): min=473, max=30442, avg=4738.78, stdev=4573.12 lat (usec): min=474, max=30443, avg=4739.56, stdev=4573.13 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 545], 10.00th=[ 603], 20.00th=[ 709], | 30.00th=[ 1004], 40.00th=[ 2376], 50.00th=[ 3720], 60.00th=[ 4817], | 70.00th=[ 5932], 80.00th=[ 7832], 90.00th=[10814], 95.00th=[13829], | 99.00th=[19006], 99.50th=[21103], 99.90th=[30278], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 208, max= 664, per=24.98%, avg=463.93, stdev=108.17, samples=27 iops : min= 52, max= 166, avg=115.96, stdev=27.03, samples=27 lat (usec) : 50=0.49%, 100=8.69%, 250=2.64%, 500=0.59%, 750=11.13% lat (usec) : 1000=3.87% lat (msec) : 2=5.18%, 4=24.93%, 10=33.72%, 20=8.01%, 50=0.72% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=24.21%, ctx=5926, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32710: Fri May 10 07:01:17 2024 read: IOPS=110, BW=441KiB/s (451kB/s)(5940KiB/13472msec) clat (usec): min=46, max=38258, avg=3692.39, stdev=3319.67 lat (usec): min=46, max=38259, avg=3693.22, stdev=3319.48 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 60], 10.00th=[ 70], 20.00th=[ 102], | 30.00th=[ 2245], 40.00th=[ 2704], 50.00th=[ 3195], 60.00th=[ 3720], | 70.00th=[ 4555], 80.00th=[ 5932], 90.00th=[ 8094], 95.00th=[ 9634], | 99.00th=[13042], 99.50th=[15401], 99.90th=[32375], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 208, max= 632, per=24.59%, avg=436.15, stdev=104.48, samples=26 iops : min= 52, max= 158, avg=108.96, stdev=26.14, samples=26 write: IOPS=117, BW=471KiB/s (483kB/s)(6348KiB/13472msec) clat (usec): min=433, max=229209, avg=4962.70, stdev=7449.04 lat (usec): min=433, max=229210, avg=4964.75, stdev=7450.57 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 545], 10.00th=[ 586], 20.00th=[ 717], | 30.00th=[ 1037], 40.00th=[ 2409], 50.00th=[ 3720], 60.00th=[ 4752], | 70.00th=[ 5997], 80.00th=[ 8094], 90.00th=[ 11207], 95.00th=[ 14091], | 99.00th=[ 22152], 99.50th=[ 29230], 99.90th=[ 57410], 99.95th=[229639], | 99.99th=[229639] bw ( KiB/s): min= 264, max= 728, per=24.76%, avg=459.88, stdev=96.52, samples=26 iops : min= 66, max= 182, avg=114.88, stdev=24.12, samples=26 lat (usec) : 50=0.39%, 100=9.18%, 250=2.31%, 500=0.72%, 750=10.74% lat (usec) : 1000=4.00% lat (msec) : 2=4.88%, 4=26.40%, 10=33.01%, 20=7.58%, 50=0.72% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.14%, sys=24.24%, ctx=5958, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32711: Fri May 10 07:01:17 2024 read: IOPS=112, BW=451KiB/s (462kB/s)(6028KiB/13372msec) clat (usec): min=46, max=29037, avg=3763.82, stdev=3283.31 lat (usec): min=47, max=29038, avg=3764.45, stdev=3283.32 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 62], 10.00th=[ 74], 20.00th=[ 114], | 30.00th=[ 2343], 40.00th=[ 2737], 50.00th=[ 3195], 60.00th=[ 3720], | 70.00th=[ 4621], 80.00th=[ 6063], 90.00th=[ 8029], 95.00th=[ 9765], | 99.00th=[13566], 99.50th=[16712], 99.90th=[26084], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 192, max= 712, per=25.33%, avg=449.42, stdev=105.20, samples=26 iops : min= 48, max= 178, avg=112.27, stdev=26.34, samples=26 write: IOPS=117, BW=468KiB/s (479kB/s)(6260KiB/13372msec) clat (usec): min=465, max=203851, avg=4847.98, stdev=6945.98 lat (usec): min=465, max=203852, avg=4849.00, stdev=6945.86 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 545], 10.00th=[ 594], 20.00th=[ 709], | 30.00th=[ 947], 40.00th=[ 2147], 50.00th=[ 3687], 60.00th=[ 4752], | 70.00th=[ 6063], 80.00th=[ 7963], 90.00th=[ 10814], 95.00th=[ 13173], | 99.00th=[ 21103], 99.50th=[ 24511], 99.90th=[ 47973], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 256, max= 720, per=25.05%, avg=465.12, stdev=86.44, samples=26 iops : min= 64, max= 180, avg=116.19, stdev=21.67, samples=26 lat (usec) : 50=0.13%, 100=8.43%, 250=2.64%, 500=0.62%, 750=11.26% lat (usec) : 1000=4.39% lat (msec) : 2=4.62%, 4=26.79%, 10=32.98%, 20=7.29%, 50=0.81% lat (msec) : 250=0.03% cpu : usr=0.10%, sys=23.97%, ctx=5928, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32712: Fri May 10 07:01:17 2024 read: IOPS=112, BW=450KiB/s (460kB/s)(6052KiB/13463msec) clat (usec): min=47, max=184471, avg=3903.11, stdev=5686.47 lat (usec): min=47, max=184472, avg=3904.31, stdev=5686.76 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 61], 10.00th=[ 72], 20.00th=[ 105], | 30.00th=[ 2278], 40.00th=[ 2704], 50.00th=[ 3195], 60.00th=[ 3785], | 70.00th=[ 4686], 80.00th=[ 6194], 90.00th=[ 8291], 95.00th=[ 10028], | 99.00th=[ 13566], 99.50th=[ 15139], 99.90th=[ 26608], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 224, max= 752, per=24.94%, avg=442.50, stdev=103.97, samples=26 iops : min= 56, max= 188, avg=110.42, stdev=26.08, samples=26 write: IOPS=115, BW=463KiB/s (474kB/s)(6236KiB/13463msec) clat (usec): min=408, max=46134, avg=4772.67, stdev=4766.47 lat (usec): min=408, max=46135, avg=4774.27, stdev=4766.49 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 553], 10.00th=[ 594], 20.00th=[ 717], | 30.00th=[ 955], 40.00th=[ 2311], 50.00th=[ 3589], 60.00th=[ 4621], | 70.00th=[ 6128], 80.00th=[ 8029], 90.00th=[11207], 95.00th=[13435], | 99.00th=[20317], 99.50th=[23987], 99.90th=[34866], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 248, max= 704, per=24.67%, avg=458.12, stdev=98.82, samples=26 iops : min= 62, max= 176, avg=114.35, stdev=24.79, samples=26 lat (usec) : 50=0.36%, 100=9.24%, 250=2.12%, 500=0.59%, 750=10.74% lat (usec) : 1000=4.69% lat (msec) : 2=4.62%, 4=25.75%, 10=32.78%, 20=8.43%, 50=0.65% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=23.81%, ctx=5979, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1774KiB/s (1817kB/s), 441KiB/s-451KiB/s (451kB/s-462kB/s), io=23.4MiB (24.6MB), run=13372-13532msec WRITE: bw=1858KiB/s (1902kB/s), 463KiB/s-471KiB/s (474kB/s-483kB/s), io=24.6MiB (25.7MB), run=13372-13532msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32714: Fri May 10 07:01:47 2024 read: IOPS=35, BW=142KiB/s (145kB/s)(5992KiB/42295msec) clat (usec): min=1641, max=235917, avg=4596.88, stdev=6929.60 lat (usec): min=1642, max=235918, avg=4597.76, stdev=6929.62 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2040], 10.00th=[ 2147], 20.00th=[ 2409], | 30.00th=[ 2638], 40.00th=[ 2933], 50.00th=[ 3392], 60.00th=[ 4015], | 70.00th=[ 4621], 80.00th=[ 5276], 90.00th=[ 7767], 95.00th=[ 10683], | 99.00th=[ 17957], 99.50th=[ 27657], 99.90th=[ 43254], 99.95th=[235930], | 99.99th=[235930] bw ( KiB/s): min= 47, max= 272, per=25.10%, avg=141.56, stdev=58.38, samples=84 iops : min= 11, max= 68, avg=35.35, stdev=14.62, samples=84 write: IOPS=37, BW=149KiB/s (152kB/s)(6296KiB/42295msec) clat (usec): min=11137, max=74729, avg=22471.03, stdev=8449.81 lat (usec): min=11138, max=74730, avg=22472.04, stdev=8449.80 clat percentiles (usec): | 1.00th=[12518], 5.00th=[13829], 10.00th=[14484], 20.00th=[15795], | 30.00th=[17171], 40.00th=[19006], 50.00th=[20579], 60.00th=[22152], | 70.00th=[23725], 80.00th=[26608], 90.00th=[34866], 95.00th=[40633], | 99.00th=[51643], 99.50th=[55837], 99.90th=[65799], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 56, max= 208, per=25.24%, avg=148.89, stdev=42.20, samples=84 iops : min= 14, max= 52, avg=37.18, stdev=10.58, samples=84 lat (msec) : 2=1.92%, 4=27.12%, 10=16.60%, 20=26.86%, 50=26.82% lat (msec) : 100=0.65%, 250=0.03% cpu : usr=0.10%, sys=5.62%, ctx=3839, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32715: Fri May 10 07:01:47 2024 read: IOPS=34, BW=140KiB/s (143kB/s)(5940KiB/42551msec) clat (usec): min=1742, max=187904, avg=4484.63, stdev=5788.74 lat (usec): min=1743, max=187906, avg=4485.45, stdev=5788.75 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2024], 10.00th=[ 2147], 20.00th=[ 2376], | 30.00th=[ 2638], 40.00th=[ 2966], 50.00th=[ 3458], 60.00th=[ 4113], | 70.00th=[ 4621], 80.00th=[ 5276], 90.00th=[ 7177], 95.00th=[ 10421], | 99.00th=[ 15926], 99.50th=[ 19006], 99.90th=[ 42730], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 16, max= 280, per=24.77%, avg=139.70, stdev=60.35, samples=84 iops : min= 4, max= 70, avg=34.85, stdev=15.06, samples=84 write: IOPS=37, BW=149KiB/s (153kB/s)(6348KiB/42551msec) clat (usec): min=11115, max=68699, avg=22593.82, stdev=8516.31 lat (usec): min=11116, max=68701, avg=22594.91, stdev=8516.29 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13566], 10.00th=[14353], 20.00th=[15795], | 30.00th=[17171], 40.00th=[19006], 50.00th=[20841], 60.00th=[22414], | 70.00th=[23987], 80.00th=[26608], 90.00th=[34866], 95.00th=[41681], | 99.00th=[51119], 99.50th=[53740], 99.90th=[62129], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 72, max= 208, per=25.18%, avg=148.56, stdev=41.38, samples=84 iops : min= 18, max= 52, avg=37.06, stdev=10.33, samples=84 lat (msec) : 2=1.79%, 4=26.30%, 10=17.61%, 20=25.94%, 50=27.77% lat (msec) : 100=0.55%, 250=0.03% cpu : usr=0.09%, sys=5.61%, ctx=3799, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32716: Fri May 10 07:01:47 2024 read: IOPS=35, BW=142KiB/s (146kB/s)(6028KiB/42383msec) clat (usec): min=1684, max=74056, avg=4663.85, stdev=4011.46 lat (usec): min=1684, max=74057, avg=4664.74, stdev=4011.45 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2073], 10.00th=[ 2245], 20.00th=[ 2474], | 30.00th=[ 2704], 40.00th=[ 3097], 50.00th=[ 3785], 60.00th=[ 4228], | 70.00th=[ 4621], 80.00th=[ 5473], 90.00th=[ 8356], 95.00th=[11338], | 99.00th=[18220], 99.50th=[26346], 99.90th=[43254], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 40, max= 256, per=25.21%, avg=142.19, stdev=50.46, samples=84 iops : min= 10, max= 64, avg=35.48, stdev=12.66, samples=84 write: IOPS=36, BW=148KiB/s (151kB/s)(6260KiB/42383msec) clat (msec): min=11, max=233, avg=22.56, stdev= 9.57 lat (msec): min=11, max=233, avg=22.57, stdev= 9.57 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 21], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 27], 90.00th=[ 34], 95.00th=[ 40], | 99.00th=[ 50], 99.50th=[ 53], 99.90th=[ 63], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 64, max= 216, per=25.00%, avg=147.52, stdev=41.56, samples=84 iops : min= 16, max= 54, avg=36.81, stdev=10.46, samples=84 lat (msec) : 2=1.76%, 4=24.97%, 10=18.82%, 20=25.72%, 50=28.26% lat (msec) : 100=0.46%, 250=0.03% cpu : usr=0.10%, sys=5.66%, ctx=3824, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32717: Fri May 10 07:01:47 2024 read: IOPS=35, BW=144KiB/s (147kB/s)(6052KiB/42168msec) clat (usec): min=1649, max=42723, avg=4416.42, stdev=3371.93 lat (usec): min=1650, max=42724, avg=4417.26, stdev=3371.91 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 2073], 10.00th=[ 2212], 20.00th=[ 2409], | 30.00th=[ 2671], 40.00th=[ 2966], 50.00th=[ 3490], 60.00th=[ 4113], | 70.00th=[ 4621], 80.00th=[ 5145], 90.00th=[ 7635], 95.00th=[10683], | 99.00th=[16909], 99.50th=[19006], 99.90th=[41681], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 24, max= 272, per=25.43%, avg=143.44, stdev=55.81, samples=84 iops : min= 6, max= 68, avg=35.65, stdev=13.97, samples=84 write: IOPS=36, BW=148KiB/s (151kB/s)(6236KiB/42168msec) clat (msec): min=11, max=218, avg=22.74, stdev= 9.98 lat (msec): min=11, max=218, avg=22.74, stdev= 9.98 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 21], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 27], 90.00th=[ 35], 95.00th=[ 43], | 99.00th=[ 53], 99.50th=[ 59], 99.90th=[ 75], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 72, max= 207, per=25.02%, avg=147.63, stdev=41.23, samples=84 iops : min= 18, max= 51, avg=36.70, stdev=10.35, samples=84 lat (msec) : 2=1.33%, 4=27.08%, 10=18.07%, 20=25.36%, 50=27.41% lat (msec) : 100=0.72%, 250=0.03% cpu : usr=0.12%, sys=5.61%, ctx=3831, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=564KiB/s (578kB/s), 140KiB/s-144KiB/s (143kB/s-147kB/s), io=23.4MiB (24.6MB), run=42168-42551msec WRITE: bw=591KiB/s (605kB/s), 148KiB/s-149KiB/s (151kB/s-153kB/s), io=24.6MiB (25.7MB), run=42168-42551msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32765: Fri May 10 07:01:52 2024 read: IOPS=82, BW=1327KiB/s (1359kB/s)(5920KiB/4461msec) clat (usec): min=53, max=39959, avg=8663.21, stdev=5638.93 lat (usec): min=53, max=39959, avg=8664.50, stdev=5637.86 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 62], 10.00th=[ 72], 20.00th=[ 125], | 30.00th=[ 8029], 40.00th=[ 8848], 50.00th=[ 9765], 60.00th=[10421], | 70.00th=[11207], 80.00th=[12125], 90.00th=[13829], 95.00th=[16057], | 99.00th=[25297], 99.50th=[26870], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 862, max= 1536, per=24.16%, avg=1283.38, stdev=218.88, samples=8 iops : min= 53, max= 96, avg=80.00, stdev=14.01, samples=8 write: IOPS=89, BW=1427KiB/s (1462kB/s)(6368KiB/4461msec) clat (usec): min=655, max=25521, avg=3138.68, stdev=3569.58 lat (usec): min=656, max=25523, avg=3139.58, stdev=3569.62 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 725], 10.00th=[ 766], 20.00th=[ 824], | 30.00th=[ 873], 40.00th=[ 988], 50.00th=[ 1598], 60.00th=[ 2737], | 70.00th=[ 3556], 80.00th=[ 4948], 90.00th=[ 6783], 95.00th=[ 9503], | 99.00th=[20579], 99.50th=[23987], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 958, max= 1952, per=25.11%, avg=1403.25, stdev=341.79, samples=8 iops : min= 59, max= 122, avg=87.50, stdev=21.45, samples=8 lat (usec) : 100=9.24%, 250=1.56%, 500=0.13%, 750=3.65%, 1000=17.45% lat (msec) : 2=6.12%, 4=11.59%, 10=24.87%, 20=23.70%, 50=1.69% cpu : usr=0.18%, sys=22.06%, ctx=2009, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32766: Fri May 10 07:01:52 2024 read: IOPS=79, BW=1274KiB/s (1304kB/s)(5744KiB/4509msec) clat (usec): min=52, max=35177, avg=8584.75, stdev=5632.58 lat (usec): min=52, max=35178, avg=8585.44, stdev=5632.66 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 62], 10.00th=[ 65], 20.00th=[ 84], | 30.00th=[ 8094], 40.00th=[ 8848], 50.00th=[ 9896], 60.00th=[10421], | 70.00th=[11338], 80.00th=[12387], 90.00th=[14353], 95.00th=[16319], | 99.00th=[21103], 99.50th=[28181], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 512, max= 1657, per=24.01%, avg=1275.33, stdev=333.43, samples=9 iops : min= 32, max= 103, avg=79.56, stdev=20.81, samples=9 write: IOPS=90, BW=1451KiB/s (1486kB/s)(6544KiB/4509msec) clat (usec): min=629, max=33617, avg=3469.44, stdev=4612.26 lat (usec): min=630, max=33617, avg=3470.32, stdev=4612.34 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 725], 10.00th=[ 766], 20.00th=[ 816], | 30.00th=[ 865], 40.00th=[ 963], 50.00th=[ 2212], 60.00th=[ 2737], | 70.00th=[ 3687], 80.00th=[ 5276], 90.00th=[ 7177], 95.00th=[10028], | 99.00th=[27132], 99.50th=[29230], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 832, max= 1984, per=25.94%, avg=1449.33, stdev=390.05, samples=9 iops : min= 52, max= 124, avg=90.44, stdev=24.26, samples=9 lat (usec) : 100=10.42%, 250=0.78%, 500=0.13%, 750=3.91%, 1000=18.49% lat (msec) : 2=3.78%, 4=12.24%, 10=25.78%, 20=22.53%, 50=1.95% cpu : usr=0.24%, sys=21.12%, ctx=1976, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32767: Fri May 10 07:01:52 2024 read: IOPS=91, BW=1459KiB/s (1494kB/s)(6256KiB/4289msec) clat (usec): min=49, max=24569, avg=8094.49, stdev=5456.13 lat (usec): min=50, max=24570, avg=8095.10, stdev=5456.21 clat percentiles (usec): | 1.00th=[ 54], 5.00th=[ 61], 10.00th=[ 64], 20.00th=[ 84], | 30.00th=[ 7111], 40.00th=[ 8717], 50.00th=[ 9503], 60.00th=[10421], | 70.00th=[11338], 80.00th=[12256], 90.00th=[13698], 95.00th=[15270], | 99.00th=[20579], 99.50th=[23987], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 894, max= 1856, per=25.90%, avg=1375.75, stdev=286.19, samples=8 iops : min= 55, max= 116, avg=85.88, stdev=18.10, samples=8 write: IOPS=87, BW=1406KiB/s (1440kB/s)(6032KiB/4289msec) clat (usec): min=627, max=27080, avg=2963.04, stdev=3447.60 lat (usec): min=627, max=27081, avg=2963.96, stdev=3447.65 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 709], 10.00th=[ 750], 20.00th=[ 799], | 30.00th=[ 857], 40.00th=[ 922], 50.00th=[ 1254], 60.00th=[ 2737], | 70.00th=[ 3326], 80.00th=[ 4490], 90.00th=[ 6521], 95.00th=[ 9372], | 99.00th=[20317], 99.50th=[22676], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 960, max= 1824, per=23.90%, avg=1335.62, stdev=321.37, samples=8 iops : min= 60, max= 114, avg=83.37, stdev=20.12, samples=8 lat (usec) : 50=0.13%, 100=11.20%, 250=2.34%, 500=0.13%, 750=4.82% lat (usec) : 1000=16.28% lat (msec) : 2=4.56%, 4=12.24%, 10=23.31%, 20=23.96%, 50=1.04% cpu : usr=0.19%, sys=22.39%, ctx=1930, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=300: Fri May 10 07:01:52 2024 read: IOPS=83, BW=1340KiB/s (1372kB/s)(6032KiB/4503msec) clat (usec): min=50, max=35874, avg=8229.50, stdev=5437.95 lat (usec): min=50, max=35875, avg=8233.01, stdev=5442.62 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 62], 10.00th=[ 65], 20.00th=[ 88], | 30.00th=[ 7635], 40.00th=[ 8717], 50.00th=[ 9634], 60.00th=[10290], | 70.00th=[10945], 80.00th=[11863], 90.00th=[13829], 95.00th=[16057], | 99.00th=[19792], 99.50th=[20317], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 608, max= 1568, per=25.09%, avg=1333.00, stdev=288.47, samples=9 iops : min= 38, max= 98, avg=83.22, stdev=18.04, samples=9 write: IOPS=86, BW=1389KiB/s (1423kB/s)(6256KiB/4503msec) clat (usec): min=681, max=36091, avg=3559.32, stdev=4594.50 lat (usec): min=682, max=36093, avg=3560.24, stdev=4594.55 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 758], 10.00th=[ 799], 20.00th=[ 840], | 30.00th=[ 906], 40.00th=[ 996], 50.00th=[ 2057], 60.00th=[ 2802], | 70.00th=[ 3589], 80.00th=[ 5211], 90.00th=[ 8291], 95.00th=[12649], | 99.00th=[24773], 99.50th=[32113], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 736, max= 1824, per=24.87%, avg=1389.78, stdev=309.39, samples=9 iops : min= 46, max= 114, avg=86.78, stdev=19.25, samples=9 lat (usec) : 100=10.94%, 250=1.43%, 500=0.13%, 750=2.21%, 1000=18.23% lat (msec) : 2=4.95%, 4=11.72%, 10=25.65%, 20=23.57%, 50=1.17% cpu : usr=0.20%, sys=21.24%, ctx=1983, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5312KiB/s (5440kB/s), 1274KiB/s-1459KiB/s (1304kB/s-1494kB/s), io=23.4MiB (24.5MB), run=4289-4509msec WRITE: bw=5589KiB/s (5723kB/s), 1389KiB/s-1451KiB/s (1423kB/s-1486kB/s), io=24.6MiB (25.8MB), run=4289-4509msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32760: Fri May 10 07:01:58 2024 read: IOPS=36, BW=586KiB/s (600kB/s)(5920KiB/10109msec) clat (usec): min=2036, max=32473, avg=4494.36, stdev=3636.81 lat (usec): min=2045, max=32473, avg=4495.35, stdev=3636.64 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2376], 10.00th=[ 2474], 20.00th=[ 2638], | 30.00th=[ 2802], 40.00th=[ 2933], 50.00th=[ 3032], 60.00th=[ 3294], | 70.00th=[ 3720], 80.00th=[ 5342], 90.00th=[ 9110], 95.00th=[13173], | 99.00th=[17433], 99.50th=[23725], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 352, max= 926, per=24.87%, avg=582.20, stdev=151.19, samples=20 iops : min= 22, max= 57, avg=36.30, stdev= 9.37, samples=20 write: IOPS=39, BW=630KiB/s (645kB/s)(6368KiB/10109msec) clat (usec): min=12421, max=54825, avg=21200.47, stdev=5847.14 lat (usec): min=12422, max=54826, avg=21201.99, stdev=5847.05 clat percentiles (usec): | 1.00th=[12780], 5.00th=[13435], 10.00th=[14353], 20.00th=[15664], | 30.00th=[17433], 40.00th=[19006], 50.00th=[20841], 60.00th=[21890], | 70.00th=[23987], 80.00th=[25560], 90.00th=[28705], 95.00th=[32113], | 99.00th=[36963], 99.50th=[42730], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 416, max= 800, per=25.52%, avg=628.60, stdev=134.09, samples=20 iops : min= 26, max= 50, avg=39.20, stdev= 8.30, samples=20 lat (msec) : 4=35.55%, 10=8.72%, 20=27.21%, 50=28.39%, 100=0.13% cpu : usr=0.05%, sys=6.63%, ctx=876, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32761: Fri May 10 07:01:58 2024 read: IOPS=35, BW=561KiB/s (575kB/s)(5744KiB/10231msec) clat (usec): min=2024, max=25745, avg=4371.66, stdev=3293.04 lat (usec): min=2024, max=25746, avg=4372.56, stdev=3293.00 clat percentiles (usec): | 1.00th=[ 2147], 5.00th=[ 2376], 10.00th=[ 2474], 20.00th=[ 2606], | 30.00th=[ 2769], 40.00th=[ 2933], 50.00th=[ 3032], 60.00th=[ 3294], | 70.00th=[ 3752], 80.00th=[ 5211], 90.00th=[ 8848], 95.00th=[11863], | 99.00th=[19530], 99.50th=[24249], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 256, max= 992, per=24.05%, avg=563.10, stdev=194.34, samples=20 iops : min= 16, max= 62, avg=35.15, stdev=12.15, samples=20 write: IOPS=39, BW=640KiB/s (655kB/s)(6544KiB/10231msec) clat (usec): min=12496, max=42132, avg=21156.90, stdev=5480.61 lat (usec): min=12497, max=42133, avg=21158.23, stdev=5480.55 clat percentiles (usec): | 1.00th=[13042], 5.00th=[13829], 10.00th=[14484], 20.00th=[15795], | 30.00th=[17433], 40.00th=[19006], 50.00th=[20841], 60.00th=[22152], | 70.00th=[23987], 80.00th=[25560], 90.00th=[28443], 95.00th=[30802], | 99.00th=[36963], 99.50th=[38011], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 480, max= 800, per=25.79%, avg=635.10, stdev=101.77, samples=20 iops : min= 30, max= 50, avg=39.65, stdev= 6.32, samples=20 lat (msec) : 4=34.37%, 10=8.98%, 20=27.86%, 50=28.78% cpu : usr=0.08%, sys=6.40%, ctx=890, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32762: Fri May 10 07:01:58 2024 read: IOPS=39, BW=635KiB/s (650kB/s)(6256KiB/9858msec) clat (usec): min=2062, max=26547, avg=4447.20, stdev=3653.34 lat (usec): min=2063, max=26548, avg=4448.00, stdev=3653.29 clat percentiles (usec): | 1.00th=[ 2147], 5.00th=[ 2311], 10.00th=[ 2442], 20.00th=[ 2606], | 30.00th=[ 2769], 40.00th=[ 2933], 50.00th=[ 3097], 60.00th=[ 3359], | 70.00th=[ 3785], 80.00th=[ 5014], 90.00th=[ 7898], 95.00th=[12911], | 99.00th=[22414], 99.50th=[25822], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 352, max= 1120, per=27.41%, avg=641.68, stdev=220.02, samples=19 iops : min= 22, max= 70, avg=40.11, stdev=13.75, samples=19 write: IOPS=38, BW=612KiB/s (627kB/s)(6032KiB/9858msec) clat (usec): min=12684, max=47373, avg=21515.38, stdev=5760.12 lat (usec): min=12685, max=47374, avg=21516.77, stdev=5760.12 clat percentiles (usec): | 1.00th=[13042], 5.00th=[14091], 10.00th=[15008], 20.00th=[16188], | 30.00th=[17957], 40.00th=[19268], 50.00th=[20841], 60.00th=[22152], | 70.00th=[23725], 80.00th=[25560], 90.00th=[29492], 95.00th=[32113], | 99.00th=[40633], 99.50th=[43254], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 448, max= 800, per=24.69%, avg=608.00, stdev=117.82, samples=19 iops : min= 28, max= 50, avg=38.00, stdev= 7.36, samples=19 lat (msec) : 4=37.63%, 10=9.11%, 20=25.52%, 50=27.73% cpu : usr=0.08%, sys=6.83%, ctx=869, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32763: Fri May 10 07:01:58 2024 read: IOPS=37, BW=601KiB/s (615kB/s)(6032KiB/10037msec) clat (usec): min=1914, max=34535, avg=4443.22, stdev=3883.45 lat (usec): min=1914, max=34536, avg=4443.99, stdev=3883.43 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2343], 10.00th=[ 2442], 20.00th=[ 2573], | 30.00th=[ 2769], 40.00th=[ 2900], 50.00th=[ 3130], 60.00th=[ 3326], | 70.00th=[ 3785], 80.00th=[ 5014], 90.00th=[ 8225], 95.00th=[11994], | 99.00th=[24249], 99.50th=[30802], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 416, max= 832, per=25.49%, avg=596.75, stdev=122.59, samples=20 iops : min= 26, max= 52, avg=37.25, stdev= 7.72, samples=20 write: IOPS=38, BW=623KiB/s (638kB/s)(6256KiB/10037msec) clat (usec): min=12117, max=44473, avg=21364.72, stdev=5830.71 lat (usec): min=12118, max=44474, avg=21366.04, stdev=5830.61 clat percentiles (usec): | 1.00th=[12649], 5.00th=[13829], 10.00th=[14615], 20.00th=[16188], | 30.00th=[17695], 40.00th=[19006], 50.00th=[20841], 60.00th=[22152], | 70.00th=[23462], 80.00th=[25297], 90.00th=[29492], 95.00th=[33162], | 99.00th=[38536], 99.50th=[40109], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 448, max= 864, per=25.27%, avg=622.30, stdev=112.12, samples=20 iops : min= 28, max= 54, avg=38.85, stdev= 7.04, samples=20 lat (msec) : 2=0.52%, 4=35.03%, 10=10.16%, 20=25.52%, 50=28.78% cpu : usr=0.07%, sys=6.60%, ctx=880, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2341KiB/s (2397kB/s), 561KiB/s-635KiB/s (575kB/s-650kB/s), io=23.4MiB (24.5MB), run=9858-10231msec WRITE: bw=2463KiB/s (2522kB/s), 612KiB/s-640KiB/s (627kB/s-655kB/s), io=24.6MiB (25.8MB), run=9858-10231msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=313: Fri May 10 07:02:04 2024 read: IOPS=1, BW=1867KiB/s (1912kB/s)(7168KiB/3839msec) clat (usec): min=559, max=741128, avg=389167.39, stdev=356770.83 lat (usec): min=561, max=741130, avg=389168.78, stdev=356770.81 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 562], 10.00th=[ 562], 20.00th=[ 14484], | 30.00th=[ 36963], 40.00th=[ 36963], 50.00th=[501220], 60.00th=[700449], | 70.00th=[700449], 80.00th=[725615], 90.00th=[742392], 95.00th=[742392], | 99.00th=[742392], 99.50th=[742392], 99.90th=[742392], 99.95th=[742392], | 99.99th=[742392] bw ( KiB/s): min= 1721, max= 4096, per=61.81%, avg=2392.20, stdev=962.92, samples=5 iops : min= 1, max= 4, avg= 2.20, stdev= 1.10, samples=5 write: IOPS=1, BW=1334KiB/s (1366kB/s)(5120KiB/3839msec) clat (msec): min=34, max=832, avg=219.66, stdev=343.33 lat (msec): min=34, max=832, avg=219.72, stdev=343.33 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 35], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 79], 60.00th=[ 79], | 70.00th=[ 97], 80.00th=[ 97], 90.00th=[ 835], 95.00th=[ 835], | 99.00th=[ 835], 99.50th=[ 835], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 2048, max= 6144, per=75.59%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (usec) : 750=8.33% lat (msec) : 20=8.33%, 50=16.67%, 100=25.00%, 750=33.33%, 1000=8.33% cpu : usr=0.00%, sys=15.63%, ctx=1136, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=314: Fri May 10 07:02:04 2024 read: IOPS=0, BW=531KiB/s (544kB/s)(2048KiB/3855msec) clat (msec): min=73, max=659, avg=366.26, stdev=414.03 lat (msec): min=73, max=659, avg=366.26, stdev=414.03 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 74], 20.00th=[ 74], | 30.00th=[ 74], 40.00th=[ 74], 50.00th=[ 74], 60.00th=[ 659], | 70.00th=[ 659], 80.00th=[ 659], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 2048, max= 2048, per=52.92%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2656KiB/s (2720kB/s)(10.0MiB/3855msec) clat (msec): min=21, max=1568, avg=312.02, stdev=568.92 lat (msec): min=22, max=1568, avg=312.07, stdev=568.92 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 23], | 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 46], 60.00th=[ 58], | 70.00th=[ 68], 80.00th=[ 68], 90.00th=[ 1183], 95.00th=[ 1569], | 99.00th=[ 1569], 99.50th=[ 1569], 99.90th=[ 1569], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 4096, max=10240, per=100.00%, avg=7168.00, stdev=4344.46, samples=2 iops : min= 4, max= 10, avg= 7.00, stdev= 4.24, samples=2 lat (msec) : 50=41.67%, 100=33.33%, 750=8.33% cpu : usr=0.00%, sys=9.29%, ctx=332, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=315: Fri May 10 07:02:04 2024 read: IOPS=1, BW=1161KiB/s (1189kB/s)(6144KiB/5291msec) clat (usec): min=336, max=769891, avg=483504.12, stdev=359818.66 lat (usec): min=337, max=769892, avg=483505.14, stdev=359818.99 clat percentiles (usec): | 1.00th=[ 338], 5.00th=[ 338], 10.00th=[ 338], 20.00th=[ 64750], | 30.00th=[ 64750], 40.00th=[549454], 50.00th=[549454], 60.00th=[750781], | 70.00th=[767558], 80.00th=[767558], 90.00th=[767558], 95.00th=[767558], | 99.00th=[767558], 99.50th=[767558], 99.90th=[767558], 99.95th=[767558], | 99.99th=[767558] bw ( KiB/s): min= 1673, max= 4096, per=61.32%, avg=2373.00, stdev=1162.12, samples=4 iops : min= 1, max= 4, avg= 2.00, stdev= 1.41, samples=4 write: IOPS=1, BW=1161KiB/s (1189kB/s)(6144KiB/5291msec) clat (msec): min=24, max=1530, avg=398.05, stdev=601.12 lat (msec): min=24, max=1530, avg=398.10, stdev=601.13 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 105], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 1536], 95.00th=[ 1536], | 99.00th=[ 1536], 99.50th=[ 1536], 99.90th=[ 1536], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 1673, max= 4096, per=42.59%, avg=2308.00, stdev=1016.87, samples=5 iops : min= 1, max= 4, avg= 2.00, stdev= 1.22, samples=5 lat (usec) : 500=8.33% lat (msec) : 50=16.67%, 100=16.67%, 250=8.33%, 750=16.67%, 1000=25.00% cpu : usr=0.00%, sys=12.00%, ctx=1132, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=316: Fri May 10 07:02:04 2024 read: IOPS=1, BW=1704KiB/s (1745kB/s)(5120KiB/3004msec) clat (usec): min=425, max=532645, avg=214156.51, stdev=263664.48 lat (usec): min=425, max=532646, avg=214157.41, stdev=263664.89 clat percentiles (usec): | 1.00th=[ 424], 5.00th=[ 424], 10.00th=[ 424], 20.00th=[ 424], | 30.00th=[ 453], 40.00th=[ 453], 50.00th=[ 69731], 60.00th=[ 69731], | 70.00th=[467665], 80.00th=[467665], 90.00th=[530580], 95.00th=[530580], | 99.00th=[530580], 99.50th=[530580], 99.90th=[530580], 99.95th=[530580], | 99.99th=[530580] bw ( KiB/s): min= 2048, max= 6119, per=100.00%, avg=4083.50, stdev=2878.63, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=2, BW=2386KiB/s (2443kB/s)(7168KiB/3004msec) clat (msec): min=20, max=1521, avg=275.94, stdev=550.73 lat (msec): min=20, max=1521, avg=276.03, stdev=550.72 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 62], 40.00th=[ 62], 50.00th=[ 80], 60.00th=[ 112], | 70.00th=[ 112], 80.00th=[ 116], 90.00th=[ 1519], 95.00th=[ 1519], | 99.00th=[ 1519], 99.50th=[ 1519], 99.90th=[ 1519], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 2048, max= 6144, per=66.06%, avg=3579.75, stdev=1959.35, samples=4 iops : min= 2, max= 6, avg= 3.25, stdev= 1.89, samples=4 lat (usec) : 500=16.67% lat (msec) : 50=16.67%, 100=25.00%, 250=16.67%, 500=8.33%, 750=8.33% cpu : usr=0.03%, sys=10.56%, ctx=592, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3871KiB/s (3964kB/s), 531KiB/s-1867KiB/s (544kB/s-1912kB/s), io=20.0MiB (20.0MB), run=3004-5291msec WRITE: bw=5419KiB/s (5549kB/s), 1161KiB/s-2656KiB/s (1189kB/s-2720kB/s), io=28.0MiB (29.4MB), run=3004-5291msec rand-rw: (groupid=0, jobs=1): err= 0: pid=318: Fri May 10 07:02:04 2024 read: IOPS=1, BW=1839KiB/s (1883kB/s)(7168KiB/3898msec) clat (msec): min=32, max=753, avg=386.50, stdev=319.24 lat (msec): min=32, max=753, avg=386.50, stdev=319.24 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 51], | 30.00th=[ 121], 40.00th=[ 121], 50.00th=[ 418], 60.00th=[ 592], | 70.00th=[ 592], 80.00th=[ 743], 90.00th=[ 751], 95.00th=[ 751], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 1684, max= 4087, per=61.42%, avg=2382.00, stdev=965.95, samples=5 iops : min= 1, max= 3, avg= 1.80, stdev= 0.84, samples=5 write: IOPS=1, BW=1313KiB/s (1345kB/s)(5120KiB/3898msec) clat (msec): min=34, max=874, avg=238.09, stdev=357.74 lat (msec): min=34, max=874, avg=238.15, stdev=357.72 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 35], | 30.00th=[ 64], 40.00th=[ 64], 50.00th=[ 75], 60.00th=[ 75], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 1684, max= 4087, per=48.00%, avg=2606.33, stdev=1295.15, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 lat (msec) : 50=16.67%, 100=25.00%, 250=16.67%, 500=8.33%, 750=16.67% lat (msec) : 1000=16.67% cpu : usr=0.00%, sys=3.62%, ctx=56, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=319: Fri May 10 07:02:04 2024 read: IOPS=0, BW=530KiB/s (542kB/s)(2048KiB/3867msec) clat (usec): min=40484, max=59002, avg=49743.30, stdev=13093.98 lat (usec): min=40485, max=59003, avg=49744.53, stdev=13093.88 clat percentiles (usec): | 1.00th=[40633], 5.00th=[40633], 10.00th=[40633], 20.00th=[40633], | 30.00th=[40633], 40.00th=[40633], 50.00th=[40633], 60.00th=[58983], | 70.00th=[58983], 80.00th=[58983], 90.00th=[58983], 95.00th=[58983], | 99.00th=[58983], 99.50th=[58983], 99.90th=[58983], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 2043, max= 2048, per=52.75%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2648KiB/s (2712kB/s)(10.0MiB/3867msec) clat (msec): min=30, max=1643, avg=376.43, stdev=572.85 lat (msec): min=30, max=1643, avg=376.48, stdev=572.85 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 35], | 30.00th=[ 37], 40.00th=[ 66], 50.00th=[ 80], 60.00th=[ 82], | 70.00th=[ 510], 80.00th=[ 510], 90.00th=[ 1167], 95.00th=[ 1636], | 99.00th=[ 1636], 99.50th=[ 1636], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 2048, max= 8175, per=83.81%, avg=4551.00, stdev=3213.64, samples=3 iops : min= 2, max= 7, avg= 4.00, stdev= 2.65, samples=3 lat (msec) : 50=33.33%, 100=33.33%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=3.67%, ctx=45, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=320: Fri May 10 07:02:04 2024 read: IOPS=1, BW=1164KiB/s (1192kB/s)(6144KiB/5280msec) clat (msec): min=30, max=779, avg=333.31, stdev=343.17 lat (msec): min=30, max=779, avg=333.31, stdev=343.17 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 498], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 776], 95.00th=[ 776], | 99.00th=[ 776], 99.50th=[ 776], 99.90th=[ 776], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 1678, max= 4096, per=61.24%, avg=2375.00, stdev=1160.52, samples=4 iops : min= 1, max= 4, avg= 2.00, stdev= 1.41, samples=4 write: IOPS=1, BW=1164KiB/s (1192kB/s)(6144KiB/5280msec) clat (msec): min=27, max=1546, avg=546.01, stdev=595.72 lat (msec): min=27, max=1546, avg=546.05, stdev=595.71 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 67], | 30.00th=[ 67], 40.00th=[ 148], 50.00th=[ 148], 60.00th=[ 609], | 70.00th=[ 877], 80.00th=[ 877], 90.00th=[ 1552], 95.00th=[ 1552], | 99.00th=[ 1552], 99.50th=[ 1552], 99.90th=[ 1552], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 1678, max= 4096, per=42.52%, avg=2308.60, stdev=1015.94, samples=5 iops : min= 1, max= 4, avg= 1.80, stdev= 1.30, samples=5 lat (msec) : 50=33.33%, 100=8.33%, 250=8.33%, 500=8.33%, 750=16.67% lat (msec) : 1000=16.67% cpu : usr=0.00%, sys=2.88%, ctx=32, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=321: Fri May 10 07:02:04 2024 read: IOPS=1, BW=1718KiB/s (1759kB/s)(5120KiB/2981msec) clat (msec): min=16, max=481, avg=206.21, stdev=244.43 lat (msec): min=16, max=481, avg=206.21, stdev=244.43 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 17], | 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 51], 60.00th=[ 51], | 70.00th=[ 468], 80.00th=[ 468], 90.00th=[ 481], 95.00th=[ 481], | 99.00th=[ 481], 99.50th=[ 481], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 2048, max= 4096, per=79.22%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2405KiB/s (2462kB/s)(7168KiB/2981msec) clat (msec): min=28, max=1555, avg=277.66, stdev=565.17 lat (msec): min=28, max=1555, avg=277.71, stdev=565.18 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 30], | 30.00th=[ 30], 40.00th=[ 30], 50.00th=[ 51], 60.00th=[ 126], | 70.00th=[ 126], 80.00th=[ 126], 90.00th=[ 1552], 95.00th=[ 1552], | 99.00th=[ 1552], 99.50th=[ 1552], 99.90th=[ 1552], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 2048, max= 6144, per=75.43%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=16.67%, 50=25.00%, 100=16.67%, 250=16.67%, 500=16.67% cpu : usr=0.00%, sys=3.12%, ctx=36, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3879KiB/s (3972kB/s), 530KiB/s-1839KiB/s (542kB/s-1883kB/s), io=20.0MiB (20.0MB), run=2981-5280msec WRITE: bw=5430KiB/s (5561kB/s), 1164KiB/s-2648KiB/s (1192kB/s-2712kB/s), io=28.0MiB (29.4MB), run=2981-5280msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=331: Fri May 10 07:02:08 2024 read: IOPS=0, BW=3696KiB/s (3784kB/s)(12.0MiB/3325msec) clat (msec): min=146, max=1847, avg=1103.34, stdev=870.04 lat (msec): min=146, max=1847, avg=1103.34, stdev=870.04 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 146], 10.00th=[ 146], 20.00th=[ 146], | 30.00th=[ 146], 40.00th=[ 1318], 50.00th=[ 1318], 60.00th=[ 1318], | 70.00th=[ 1854], 80.00th=[ 1854], 90.00th=[ 1854], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=45.13%, ctx=1112, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=332: Fri May 10 07:02:08 2024 read: IOPS=0, BW=1513KiB/s (1549kB/s)(4096KiB/2708msec) clat (nsec): min=177971k, max=177971k, avg=177971213.00, stdev= 0.00 lat (nsec): min=177973k, max=177973k, avg=177972604.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 178], 10.00th=[ 178], 20.00th=[ 178], | 30.00th=[ 178], 40.00th=[ 178], 50.00th=[ 178], 60.00th=[ 178], | 70.00th=[ 178], 80.00th=[ 178], 90.00th=[ 178], 95.00th=[ 178], | 99.00th=[ 178], 99.50th=[ 178], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3025KiB/s (3098kB/s)(8192KiB/2708msec) clat (msec): min=375, max=2151, avg=1263.15, stdev=1255.93 lat (msec): min=375, max=2151, avg=1263.42, stdev=1255.92 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 376], 10.00th=[ 376], 20.00th=[ 376], | 30.00th=[ 376], 40.00th=[ 376], 50.00th=[ 376], 60.00th=[ 2165], | 70.00th=[ 2165], 80.00th=[ 2165], 90.00th=[ 2165], 95.00th=[ 2165], | 99.00th=[ 2165], 99.50th=[ 2165], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 8175, max= 8175, per=85.55%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=10.86%, ctx=77, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=333: Fri May 10 07:02:08 2024 write: IOPS=0, BW=3584KiB/s (3670kB/s)(12.0MiB/3429msec) clat (msec): min=235, max=2115, avg=1139.26, stdev=941.90 lat (msec): min=236, max=2115, avg=1139.61, stdev=941.76 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 236], 10.00th=[ 236], 20.00th=[ 236], | 30.00th=[ 236], 40.00th=[ 1070], 50.00th=[ 1070], 60.00th=[ 1070], | 70.00th=[ 2123], 80.00th=[ 2123], 90.00th=[ 2123], 95.00th=[ 2123], | 99.00th=[ 2123], 99.50th=[ 2123], 99.90th=[ 2123], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 7037, max= 8175, per=79.59%, avg=7606.00, stdev=804.69, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=7.61%, ctx=38, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=334: Fri May 10 07:02:08 2024 write: IOPS=1, BW=6547KiB/s (6704kB/s)(12.0MiB/1877msec) clat (msec): min=156, max=1486, avg=621.73, stdev=749.38 lat (msec): min=157, max=1486, avg=622.10, stdev=749.28 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 222], 50.00th=[ 222], 60.00th=[ 222], | 70.00th=[ 1485], 80.00th=[ 1485], 90.00th=[ 1485], 95.00th=[ 1485], | 99.00th=[ 1485], 99.50th=[ 1485], 99.90th=[ 1485], 99.95th=[ 1485], | 99.99th=[ 1485] bw ( KiB/s): min= 8192, max= 8192, per=85.73%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=14.02%, ctx=48, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4928KiB/s (5046kB/s), 1513KiB/s-3696KiB/s (1549kB/s-3784kB/s), io=16.0MiB (16.8MB), run=2708-3325msec WRITE: bw=9556KiB/s (9785kB/s), 3025KiB/s-6547KiB/s (3098kB/s-6704kB/s), io=32.0MiB (33.6MB), run=1877-3429msec rand-rw: (groupid=0, jobs=1): err= 0: pid=335: Fri May 10 07:02:08 2024 read: IOPS=0, BW=3515KiB/s (3599kB/s)(12.0MiB/3496msec) clat (msec): min=298, max=2272, avg=1163.12, stdev=1009.68 lat (msec): min=298, max=2272, avg=1163.12, stdev=1009.68 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 919], 50.00th=[ 919], 60.00th=[ 919], | 70.00th=[ 2265], 80.00th=[ 2265], 90.00th=[ 2265], 95.00th=[ 2265], | 99.00th=[ 2265], 99.50th=[ 2265], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7684, max= 8192, per=100.00%, avg=7938.00, stdev=359.21, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=2.29%, ctx=36, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=336: Fri May 10 07:02:08 2024 read: IOPS=0, BW=1445KiB/s (1480kB/s)(4096KiB/2834msec) clat (nsec): min=311736k, max=311736k, avg=311735847.00, stdev= 0.00 lat (nsec): min=311738k, max=311738k, avg=311738255.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 313], 10.00th=[ 313], 20.00th=[ 313], | 30.00th=[ 313], 40.00th=[ 313], 50.00th=[ 313], 60.00th=[ 313], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 313], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2891KiB/s (2960kB/s)(8192KiB/2834msec) clat (msec): min=231, max=2245, avg=1238.81, stdev=1424.23 lat (msec): min=231, max=2246, avg=1239.01, stdev=1424.33 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 232], 10.00th=[ 232], 20.00th=[ 232], | 30.00th=[ 232], 40.00th=[ 232], 50.00th=[ 232], 60.00th=[ 2232], | 70.00th=[ 2232], 80.00th=[ 2232], 90.00th=[ 2232], 95.00th=[ 2232], | 99.00th=[ 2232], 99.50th=[ 2232], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 8192, max= 8192, per=87.08%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=3.99%, ctx=175, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=337: Fri May 10 07:02:08 2024 write: IOPS=0, BW=3528KiB/s (3613kB/s)(12.0MiB/3483msec) clat (msec): min=267, max=2283, avg=1158.52, stdev=1028.01 lat (msec): min=268, max=2283, avg=1158.79, stdev=1027.99 clat percentiles (msec): | 1.00th=[ 268], 5.00th=[ 268], 10.00th=[ 268], 20.00th=[ 268], | 30.00th=[ 268], 40.00th=[ 927], 50.00th=[ 927], 60.00th=[ 927], | 70.00th=[ 2299], 80.00th=[ 2299], 90.00th=[ 2299], 95.00th=[ 2299], | 99.00th=[ 2299], 99.50th=[ 2299], 99.90th=[ 2299], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 8192, max= 8192, per=87.08%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.06%, sys=4.34%, ctx=38, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=338: Fri May 10 07:02:08 2024 write: IOPS=1, BW=4291KiB/s (4393kB/s)(12.0MiB/2864msec) clat (msec): min=73, max=2185, avg=944.95, stdev=1103.11 lat (msec): min=74, max=2185, avg=945.18, stdev=1103.15 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 74], 20.00th=[ 74], | 30.00th=[ 74], 40.00th=[ 575], 50.00th=[ 575], 60.00th=[ 575], | 70.00th=[ 2198], 80.00th=[ 2198], 90.00th=[ 2198], 95.00th=[ 2198], | 99.00th=[ 2198], 99.50th=[ 2198], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 8192, max= 8192, per=87.08%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.00%, sys=3.98%, ctx=134, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4686KiB/s (4799kB/s), 1445KiB/s-3515KiB/s (1480kB/s-3599kB/s), io=16.0MiB (16.8MB), run=2834-3496msec WRITE: bw=9408KiB/s (9634kB/s), 2891KiB/s-4291KiB/s (2960kB/s-4393kB/s), io=32.0MiB (33.6MB), run=2834-3483msec