== sanity test 398b: DIO and buffer IO race ============== 06:24:50 (1712485490) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.49836 s, 33.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30981: Sun Apr 7 06:25:05 2024 read: IOPS=129, BW=516KiB/s (529kB/s)(5992KiB/11606msec) clat (usec): min=40, max=175827, avg=3696.87, stdev=5463.93 lat (usec): min=41, max=175828, avg=3697.43, stdev=5463.94 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 64], 10.00th=[ 68], 20.00th=[ 1844], | 30.00th=[ 2212], 40.00th=[ 2573], 50.00th=[ 3064], 60.00th=[ 3621], | 70.00th=[ 4359], 80.00th=[ 5407], 90.00th=[ 6915], 95.00th=[ 8356], | 99.00th=[ 13960], 99.50th=[ 19006], 99.90th=[ 45876], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 232, max= 776, per=24.98%, avg=514.39, stdev=114.99, samples=23 iops : min= 58, max= 194, avg=128.57, stdev=28.75, samples=23 write: IOPS=135, BW=542KiB/s (555kB/s)(6296KiB/11606msec) clat (usec): min=403, max=23594, avg=3786.21, stdev=3502.77 lat (usec): min=404, max=23594, avg=3786.96, stdev=3502.68 clat percentiles (usec): | 1.00th=[ 453], 5.00th=[ 486], 10.00th=[ 510], 20.00th=[ 578], | 30.00th=[ 865], 40.00th=[ 2114], 50.00th=[ 3163], 60.00th=[ 3884], | 70.00th=[ 4752], 80.00th=[ 6259], 90.00th=[ 8717], 95.00th=[10683], | 99.00th=[14615], 99.50th=[17433], 99.90th=[23462], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 264, max= 760, per=24.94%, avg=537.65, stdev=107.01, samples=23 iops : min= 66, max= 190, avg=134.39, stdev=26.76, samples=23 lat (usec) : 50=0.62%, 100=7.49%, 250=0.59%, 500=4.10%, 750=10.12% lat (usec) : 1000=2.73% lat (msec) : 2=5.66%, 4=32.19%, 10=32.23%, 20=3.94%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=24.91%, ctx=5901, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30982: Sun Apr 7 06:25:05 2024 read: IOPS=129, BW=520KiB/s (532kB/s)(5940KiB/11426msec) clat (usec): min=43, max=47419, avg=3462.10, stdev=3029.45 lat (usec): min=44, max=47420, avg=3462.76, stdev=3029.44 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 64], 10.00th=[ 69], 20.00th=[ 1844], | 30.00th=[ 2180], 40.00th=[ 2507], 50.00th=[ 2835], 60.00th=[ 3458], | 70.00th=[ 4228], 80.00th=[ 5276], 90.00th=[ 6915], 95.00th=[ 8291], | 99.00th=[11863], 99.50th=[13698], 99.90th=[34341], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 264, max= 688, per=25.18%, avg=518.50, stdev=95.67, samples=22 iops : min= 66, max= 172, avg=129.59, stdev=23.95, samples=22 write: IOPS=138, BW=556KiB/s (569kB/s)(6348KiB/11426msec) clat (usec): min=417, max=150822, avg=3892.44, stdev=5379.49 lat (usec): min=418, max=150823, avg=3893.38, stdev=5379.62 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 490], 10.00th=[ 515], 20.00th=[ 603], | 30.00th=[ 857], 40.00th=[ 1958], 50.00th=[ 3097], 60.00th=[ 3916], | 70.00th=[ 4752], 80.00th=[ 6128], 90.00th=[ 8455], 95.00th=[ 10421], | 99.00th=[ 15795], 99.50th=[ 23200], 99.90th=[ 53740], 99.95th=[149947], | 99.99th=[149947] bw ( KiB/s): min= 344, max= 824, per=25.33%, avg=546.14, stdev=117.41, samples=22 iops : min= 86, max= 206, avg=136.50, stdev=29.40, samples=22 lat (usec) : 50=0.85%, 100=7.06%, 250=0.62%, 500=3.81%, 750=10.16% lat (usec) : 1000=2.67% lat (msec) : 2=7.19%, 4=31.93%, 10=31.58%, 20=3.74%, 50=0.33% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.20%, sys=24.78%, ctx=5819, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30983: Sun Apr 7 06:25:05 2024 read: IOPS=129, BW=517KiB/s (529kB/s)(6028KiB/11660msec) clat (usec): min=42, max=45812, avg=3478.78, stdev=3212.51 lat (usec): min=43, max=45812, avg=3480.11, stdev=3212.38 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 62], 10.00th=[ 68], 20.00th=[ 115], | 30.00th=[ 2147], 40.00th=[ 2507], 50.00th=[ 2933], 60.00th=[ 3556], | 70.00th=[ 4424], 80.00th=[ 5407], 90.00th=[ 6980], 95.00th=[ 8291], | 99.00th=[12125], 99.50th=[14353], 99.90th=[41681], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 176, max= 792, per=24.97%, avg=514.04, stdev=117.67, samples=23 iops : min= 44, max= 198, avg=128.48, stdev=29.43, samples=23 write: IOPS=134, BW=537KiB/s (550kB/s)(6260KiB/11660msec) clat (usec): min=404, max=193018, avg=4030.08, stdev=6164.87 lat (usec): min=405, max=193021, avg=4031.57, stdev=6164.62 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 490], 10.00th=[ 510], 20.00th=[ 594], | 30.00th=[ 906], 40.00th=[ 2343], 50.00th=[ 3294], 60.00th=[ 3949], | 70.00th=[ 4752], 80.00th=[ 6259], 90.00th=[ 8586], 95.00th=[ 10945], | 99.00th=[ 16057], 99.50th=[ 20579], 99.90th=[ 42206], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 232, max= 808, per=24.73%, avg=533.17, stdev=128.78, samples=23 iops : min= 58, max= 202, avg=133.26, stdev=32.22, samples=23 lat (usec) : 50=0.94%, 100=8.56%, 250=0.78%, 500=3.68%, 750=10.03% lat (usec) : 1000=2.38% lat (msec) : 2=6.05%, 4=30.89%, 10=32.29%, 20=3.94%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.27%, sys=24.32%, ctx=5831, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30984: Sun Apr 7 06:25:05 2024 read: IOPS=131, BW=525KiB/s (538kB/s)(6052KiB/11527msec) clat (usec): min=43, max=147354, avg=3567.19, stdev=4962.06 lat (usec): min=43, max=147355, avg=3568.10, stdev=4962.16 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 64], 10.00th=[ 69], 20.00th=[ 1795], | 30.00th=[ 2180], 40.00th=[ 2507], 50.00th=[ 2900], 60.00th=[ 3359], | 70.00th=[ 4293], 80.00th=[ 5342], 90.00th=[ 6783], 95.00th=[ 8094], | 99.00th=[ 12256], 99.50th=[ 20317], 99.90th=[ 62653], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 264, max= 800, per=25.49%, avg=524.87, stdev=121.30, samples=23 iops : min= 66, max= 200, avg=131.22, stdev=30.32, samples=23 write: IOPS=135, BW=541KiB/s (554kB/s)(6236KiB/11527msec) clat (usec): min=409, max=39056, avg=3862.10, stdev=3749.38 lat (usec): min=409, max=39057, avg=3862.98, stdev=3749.35 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 486], 10.00th=[ 510], 20.00th=[ 570], | 30.00th=[ 791], 40.00th=[ 1893], 50.00th=[ 3130], 60.00th=[ 3949], | 70.00th=[ 5080], 80.00th=[ 6521], 90.00th=[ 8717], 95.00th=[10814], | 99.00th=[16319], 99.50th=[20579], 99.90th=[27395], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 352, max= 704, per=25.05%, avg=540.17, stdev=97.20, samples=23 iops : min= 88, max= 176, avg=135.04, stdev=24.30, samples=23 lat (usec) : 50=0.72%, 100=7.75%, 250=0.52%, 500=4.33%, 750=10.32% lat (usec) : 1000=2.38% lat (msec) : 2=6.93%, 4=30.89%, 10=31.97%, 20=3.68%, 50=0.46% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.25%, sys=24.07%, ctx=5789, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2059KiB/s (2109kB/s), 516KiB/s-525KiB/s (529kB/s-538kB/s), io=23.4MiB (24.6MB), run=11426-11660msec WRITE: bw=2156KiB/s (2208kB/s), 537KiB/s-556KiB/s (550kB/s-569kB/s), io=24.6MiB (25.7MB), run=11426-11660msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30977: Sun Apr 7 06:25:31 2024 read: IOPS=39, BW=158KiB/s (162kB/s)(5992KiB/37846msec) clat (usec): min=1481, max=154338, avg=3163.77, stdev=4740.16 lat (usec): min=1482, max=154340, avg=3164.43, stdev=4740.25 clat percentiles (usec): | 1.00th=[ 1598], 5.00th=[ 1713], 10.00th=[ 1762], 20.00th=[ 1860], | 30.00th=[ 1909], 40.00th=[ 1991], 50.00th=[ 2073], 60.00th=[ 2212], | 70.00th=[ 2442], 80.00th=[ 3032], 90.00th=[ 6259], 95.00th=[ 8979], | 99.00th=[ 13304], 99.50th=[ 18482], 99.90th=[ 30016], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 47, max= 296, per=24.96%, avg=157.97, stdev=61.85, samples=75 iops : min= 11, max= 74, avg=39.41, stdev=15.51, samples=75 write: IOPS=41, BW=166KiB/s (170kB/s)(6296KiB/37846msec) clat (usec): min=11240, max=76810, avg=20961.84, stdev=7454.07 lat (usec): min=11241, max=76810, avg=20962.55, stdev=7454.09 clat percentiles (usec): | 1.00th=[11994], 5.00th=[12649], 10.00th=[13173], 20.00th=[14353], | 30.00th=[16319], 40.00th=[18482], 50.00th=[20317], 60.00th=[21627], | 70.00th=[22676], 80.00th=[24249], 90.00th=[30802], 95.00th=[35390], | 99.00th=[46400], 99.50th=[52167], 99.90th=[73925], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 48, max= 208, per=25.00%, avg=165.76, stdev=44.12, samples=75 iops : min= 12, max= 52, avg=41.36, stdev=11.08, samples=75 lat (msec) : 2=20.41%, 4=20.90%, 10=5.60%, 20=26.37%, 50=26.33% lat (msec) : 100=0.36%, 250=0.03% cpu : usr=0.07%, sys=4.19%, ctx=3283, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30978: Sun Apr 7 06:25:31 2024 read: IOPS=39, BW=157KiB/s (161kB/s)(5940KiB/37717msec) clat (usec): min=1482, max=154370, avg=3109.76, stdev=4783.32 lat (usec): min=1483, max=154371, avg=3110.37, stdev=4783.35 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1713], 10.00th=[ 1778], 20.00th=[ 1844], | 30.00th=[ 1909], 40.00th=[ 1975], 50.00th=[ 2073], 60.00th=[ 2180], | 70.00th=[ 2442], 80.00th=[ 2966], 90.00th=[ 5538], 95.00th=[ 8160], | 99.00th=[ 15008], 99.50th=[ 21103], 99.90th=[ 30540], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 15, max= 296, per=24.87%, avg=157.40, stdev=65.01, samples=75 iops : min= 3, max= 74, avg=39.24, stdev=16.24, samples=75 write: IOPS=42, BW=168KiB/s (172kB/s)(6348KiB/37717msec) clat (usec): min=11133, max=82599, avg=20785.51, stdev=7367.70 lat (usec): min=11133, max=82600, avg=20786.23, stdev=7367.75 clat percentiles (usec): | 1.00th=[11994], 5.00th=[12780], 10.00th=[13173], 20.00th=[14091], | 30.00th=[15926], 40.00th=[17957], 50.00th=[20317], 60.00th=[21365], | 70.00th=[22676], 80.00th=[24249], 90.00th=[30016], 95.00th=[35914], | 99.00th=[43779], 99.50th=[45351], 99.90th=[77071], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 47, max= 224, per=25.32%, avg=167.85, stdev=44.67, samples=75 iops : min= 11, max= 56, avg=41.85, stdev=11.21, samples=75 lat (msec) : 2=21.22%, 4=20.02%, 10=5.60%, 20=26.20%, 50=26.82% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.06%, sys=4.25%, ctx=3327, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30979: Sun Apr 7 06:25:31 2024 read: IOPS=40, BW=160KiB/s (164kB/s)(6028KiB/37615msec) clat (usec): min=1517, max=44718, avg=3060.73, stdev=2897.21 lat (usec): min=1518, max=44719, avg=3061.31, stdev=2897.22 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 1713], 10.00th=[ 1762], 20.00th=[ 1844], | 30.00th=[ 1926], 40.00th=[ 1991], 50.00th=[ 2089], 60.00th=[ 2212], | 70.00th=[ 2474], 80.00th=[ 3097], 90.00th=[ 5932], 95.00th=[ 8455], | 99.00th=[13173], 99.50th=[20317], 99.90th=[31065], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 40, max= 336, per=25.27%, avg=159.97, stdev=60.92, samples=75 iops : min= 10, max= 84, avg=39.89, stdev=15.25, samples=75 write: IOPS=41, BW=166KiB/s (170kB/s)(6260KiB/37615msec) clat (msec): min=11, max=191, avg=21.02, stdev= 8.23 lat (msec): min=11, max=191, avg=21.02, stdev= 8.23 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 22], | 70.00th=[ 23], 80.00th=[ 25], 90.00th=[ 31], 95.00th=[ 36], | 99.00th=[ 44], 99.50th=[ 51], 99.90th=[ 65], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 47, max= 223, per=25.05%, avg=166.05, stdev=43.87, samples=75 iops : min= 11, max= 55, avg=41.41, stdev=11.01, samples=75 lat (msec) : 2=19.92%, 4=21.81%, 10=5.92%, 20=24.61%, 50=27.44% lat (msec) : 100=0.26%, 250=0.03% cpu : usr=0.09%, sys=4.20%, ctx=3297, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30980: Sun Apr 7 06:25:31 2024 read: IOPS=39, BW=160KiB/s (164kB/s)(6052KiB/37888msec) clat (usec): min=1511, max=31073, avg=3047.11, stdev=2610.23 lat (usec): min=1512, max=31073, avg=3048.08, stdev=2610.08 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1713], 10.00th=[ 1778], 20.00th=[ 1844], | 30.00th=[ 1926], 40.00th=[ 2008], 50.00th=[ 2114], 60.00th=[ 2245], | 70.00th=[ 2442], 80.00th=[ 2966], 90.00th=[ 6194], 95.00th=[ 8717], | 99.00th=[12911], 99.50th=[15401], 99.90th=[25560], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 40, max= 296, per=25.12%, avg=159.00, stdev=62.29, samples=75 iops : min= 10, max= 74, avg=39.64, stdev=15.57, samples=75 write: IOPS=41, BW=165KiB/s (169kB/s)(6236KiB/37888msec) clat (msec): min=10, max=171, avg=21.27, stdev= 8.53 lat (msec): min=10, max=171, avg=21.27, stdev= 8.53 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 22], | 70.00th=[ 23], 80.00th=[ 25], 90.00th=[ 31], 95.00th=[ 38], | 99.00th=[ 48], 99.50th=[ 53], 99.90th=[ 74], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 55, max= 208, per=24.74%, avg=164.01, stdev=44.07, samples=75 iops : min= 13, max= 52, avg=40.89, stdev=11.04, samples=75 lat (msec) : 2=19.66%, 4=22.27%, 10=5.60%, 20=25.36%, 50=26.73% lat (msec) : 100=0.36%, 250=0.03% cpu : usr=0.08%, sys=4.14%, ctx=3274, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=634KiB/s (649kB/s), 157KiB/s-160KiB/s (161kB/s-164kB/s), io=23.4MiB (24.6MB), run=37615-37888msec WRITE: bw=664KiB/s (679kB/s), 165KiB/s-168KiB/s (169kB/s-172kB/s), io=24.6MiB (25.7MB), run=37615-37888msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31023: Sun Apr 7 06:25:36 2024 read: IOPS=96, BW=1537KiB/s (1574kB/s)(5920KiB/3852msec) clat (usec): min=47, max=39252, avg=7362.29, stdev=5121.43 lat (usec): min=47, max=39252, avg=7362.83, stdev=5121.45 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 64], 10.00th=[ 67], 20.00th=[ 71], | 30.00th=[ 6783], 40.00th=[ 7832], 50.00th=[ 8455], 60.00th=[ 9503], | 70.00th=[10159], 80.00th=[10945], 90.00th=[12125], 95.00th=[13042], | 99.00th=[20055], 99.50th=[26608], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 1152, max= 1760, per=24.56%, avg=1526.86, stdev=202.14, samples=7 iops : min= 72, max= 110, avg=95.43, stdev=12.63, samples=7 write: IOPS=103, BW=1653KiB/s (1693kB/s)(6368KiB/3852msec) clat (usec): min=585, max=36161, avg=2820.73, stdev=3599.83 lat (usec): min=586, max=36162, avg=2821.60, stdev=3599.80 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 635], 10.00th=[ 676], 20.00th=[ 725], | 30.00th=[ 758], 40.00th=[ 832], 50.00th=[ 1237], 60.00th=[ 2638], | 70.00th=[ 3261], 80.00th=[ 4424], 90.00th=[ 6194], 95.00th=[ 7308], | 99.00th=[17171], 99.50th=[32113], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 1344, max= 1984, per=25.37%, avg=1659.43, stdev=254.95, samples=7 iops : min= 84, max= 124, avg=103.71, stdev=15.93, samples=7 lat (usec) : 50=0.78%, 100=11.59%, 250=0.26%, 750=14.06%, 1000=10.42% lat (msec) : 2=3.39%, 4=11.85%, 10=30.47%, 20=16.28%, 50=0.91% cpu : usr=0.16%, sys=21.97%, ctx=1869, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31024: Sun Apr 7 06:25:36 2024 read: IOPS=94, BW=1515KiB/s (1552kB/s)(5744KiB/3791msec) clat (usec): min=45, max=26422, avg=7501.48, stdev=5119.28 lat (usec): min=45, max=26422, avg=7502.06, stdev=5119.28 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 59], 10.00th=[ 66], 20.00th=[ 72], | 30.00th=[ 6849], 40.00th=[ 7963], 50.00th=[ 8586], 60.00th=[ 9634], | 70.00th=[10421], 80.00th=[11207], 90.00th=[12911], 95.00th=[14091], | 99.00th=[21103], 99.50th=[23462], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 1088, max= 1792, per=24.33%, avg=1513.14, stdev=235.67, samples=7 iops : min= 68, max= 112, avg=94.57, stdev=14.73, samples=7 write: IOPS=107, BW=1726KiB/s (1768kB/s)(6544KiB/3791msec) clat (usec): min=568, max=62786, avg=2669.55, stdev=4131.18 lat (usec): min=568, max=62787, avg=2670.39, stdev=4131.17 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 635], 10.00th=[ 676], 20.00th=[ 717], | 30.00th=[ 758], 40.00th=[ 807], 50.00th=[ 938], 60.00th=[ 2147], | 70.00th=[ 2900], 80.00th=[ 4228], 90.00th=[ 5932], 95.00th=[ 7308], | 99.00th=[16057], 99.50th=[19530], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 1376, max= 2368, per=25.85%, avg=1691.43, stdev=422.28, samples=7 iops : min= 86, max= 148, avg=105.71, stdev=26.39, samples=7 lat (usec) : 50=0.78%, 100=10.81%, 250=0.65%, 500=0.13%, 750=15.10% lat (usec) : 1000=12.37% lat (msec) : 2=3.91%, 4=10.81%, 10=27.73%, 20=16.93%, 50=0.65% lat (msec) : 100=0.13% cpu : usr=0.00%, sys=21.69%, ctx=1821, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31025: Sun Apr 7 06:25:36 2024 read: IOPS=101, BW=1627KiB/s (1666kB/s)(6256KiB/3845msec) clat (usec): min=46, max=32358, avg=7248.40, stdev=5280.09 lat (usec): min=46, max=32358, avg=7248.96, stdev=5280.12 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 58], 10.00th=[ 67], 20.00th=[ 70], | 30.00th=[ 6063], 40.00th=[ 7767], 50.00th=[ 8455], 60.00th=[ 9372], | 70.00th=[10290], 80.00th=[11076], 90.00th=[12387], 95.00th=[13698], | 99.00th=[21890], 99.50th=[25035], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 1088, max= 1920, per=25.14%, avg=1563.43, stdev=291.78, samples=7 iops : min= 68, max= 120, avg=97.71, stdev=18.24, samples=7 write: IOPS=98, BW=1569KiB/s (1606kB/s)(6032KiB/3845msec) clat (usec): min=599, max=55829, avg=2664.08, stdev=3754.42 lat (usec): min=600, max=55830, avg=2664.97, stdev=3754.39 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 644], 10.00th=[ 685], 20.00th=[ 734], | 30.00th=[ 766], 40.00th=[ 865], 50.00th=[ 1221], 60.00th=[ 2507], | 70.00th=[ 3032], 80.00th=[ 4080], 90.00th=[ 6128], 95.00th=[ 6980], | 99.00th=[11863], 99.50th=[27132], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 1248, max= 2016, per=23.34%, avg=1526.86, stdev=259.12, samples=7 iops : min= 78, max= 126, avg=95.43, stdev=16.20, samples=7 lat (usec) : 50=1.17%, 100=12.89%, 250=0.65%, 500=0.13%, 750=12.50% lat (usec) : 1000=9.51% lat (msec) : 2=4.04%, 4=13.28%, 10=27.73%, 20=17.19%, 50=0.78% lat (msec) : 100=0.13% cpu : usr=0.18%, sys=21.20%, ctx=1868, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31026: Sun Apr 7 06:25:36 2024 read: IOPS=98, BW=1571KiB/s (1609kB/s)(6032KiB/3839msec) clat (usec): min=44, max=41746, avg=7408.28, stdev=5446.70 lat (usec): min=44, max=41747, avg=7410.76, stdev=5447.29 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 59], 10.00th=[ 65], 20.00th=[ 70], | 30.00th=[ 6587], 40.00th=[ 8029], 50.00th=[ 8848], 60.00th=[ 9634], | 70.00th=[10290], 80.00th=[10945], 90.00th=[12518], 95.00th=[13435], | 99.00th=[21627], 99.50th=[33817], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 1056, max= 2016, per=24.85%, avg=1545.14, stdev=313.38, samples=7 iops : min= 66, max= 126, avg=96.57, stdev=19.59, samples=7 write: IOPS=101, BW=1630KiB/s (1669kB/s)(6256KiB/3839msec) clat (usec): min=588, max=57536, avg=2656.00, stdev=3952.12 lat (usec): min=589, max=57538, avg=2659.27, stdev=3951.21 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 635], 10.00th=[ 668], 20.00th=[ 701], | 30.00th=[ 734], 40.00th=[ 783], 50.00th=[ 955], 60.00th=[ 2245], | 70.00th=[ 2999], 80.00th=[ 4178], 90.00th=[ 5997], 95.00th=[ 7635], | 99.00th=[13960], 99.50th=[25297], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 1312, max= 1888, per=24.81%, avg=1622.86, stdev=238.55, samples=7 iops : min= 82, max= 118, avg=101.43, stdev=14.91, samples=7 lat (usec) : 50=1.43%, 100=11.98%, 250=0.52%, 750=17.45%, 1000=8.46% lat (msec) : 2=3.39%, 4=11.20%, 10=27.99%, 20=16.67%, 50=0.78% lat (msec) : 100=0.13% cpu : usr=0.10%, sys=21.03%, ctx=1820, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6218KiB/s (6367kB/s), 1515KiB/s-1627KiB/s (1552kB/s-1666kB/s), io=23.4MiB (24.5MB), run=3791-3852msec WRITE: bw=6542KiB/s (6699kB/s), 1569KiB/s-1726KiB/s (1606kB/s-1768kB/s), io=24.6MiB (25.8MB), run=3791-3852msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31027: Sun Apr 7 06:25:42 2024 read: IOPS=38, BW=609KiB/s (624kB/s)(5920KiB/9721msec) clat (usec): min=1847, max=38147, avg=3659.13, stdev=3254.48 lat (usec): min=1848, max=38148, avg=3659.87, stdev=3254.42 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 1991], 10.00th=[ 2040], 20.00th=[ 2147], | 30.00th=[ 2245], 40.00th=[ 2343], 50.00th=[ 2442], 60.00th=[ 2638], | 70.00th=[ 3064], 80.00th=[ 4621], 90.00th=[ 6128], 95.00th=[10028], | 99.00th=[15533], 99.50th=[21365], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 288, max= 1024, per=25.16%, avg=597.89, stdev=187.82, samples=19 iops : min= 18, max= 64, avg=37.37, stdev=11.74, samples=19 write: IOPS=40, BW=655KiB/s (671kB/s)(6368KiB/9721msec) clat (usec): min=11736, max=48505, avg=21000.39, stdev=6616.64 lat (usec): min=11737, max=48506, avg=21001.36, stdev=6616.67 clat percentiles (usec): | 1.00th=[12125], 5.00th=[13042], 10.00th=[13698], 20.00th=[14353], | 30.00th=[16188], 40.00th=[18744], 50.00th=[20579], 60.00th=[21890], | 70.00th=[23200], 80.00th=[25297], 90.00th=[30278], 95.00th=[34341], | 99.00th=[42206], 99.50th=[45876], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 416, max= 832, per=26.00%, avg=650.11, stdev=156.06, samples=19 iops : min= 26, max= 52, avg=40.63, stdev= 9.75, samples=19 lat (msec) : 2=2.60%, 4=34.77%, 10=8.46%, 20=27.08%, 50=27.08% cpu : usr=0.08%, sys=4.88%, ctx=844, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31028: Sun Apr 7 06:25:42 2024 read: IOPS=35, BW=570KiB/s (584kB/s)(5744KiB/10078msec) clat (usec): min=1808, max=29003, avg=3800.22, stdev=3285.03 lat (usec): min=1809, max=29004, avg=3800.82, stdev=3285.03 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 1975], 10.00th=[ 2040], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2311], 50.00th=[ 2474], 60.00th=[ 2769], | 70.00th=[ 3032], 80.00th=[ 4490], 90.00th=[ 7504], 95.00th=[10814], | 99.00th=[17957], 99.50th=[20841], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 256, max= 960, per=23.74%, avg=564.16, stdev=207.79, samples=19 iops : min= 16, max= 60, avg=35.21, stdev=13.06, samples=19 write: IOPS=40, BW=649KiB/s (665kB/s)(6544KiB/10078msec) clat (usec): min=12052, max=85587, avg=21283.80, stdev=7289.37 lat (usec): min=12053, max=85588, avg=21284.81, stdev=7289.43 clat percentiles (usec): | 1.00th=[12518], 5.00th=[13173], 10.00th=[13566], 20.00th=[14877], | 30.00th=[16909], 40.00th=[18744], 50.00th=[20579], 60.00th=[22152], | 70.00th=[23462], 80.00th=[25297], 90.00th=[30278], 95.00th=[33424], | 99.00th=[40109], 99.50th=[48497], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 447, max= 832, per=25.60%, avg=639.95, stdev=154.28, samples=19 iops : min= 27, max= 52, avg=39.95, stdev= 9.71, samples=19 lat (msec) : 2=2.99%, 4=33.07%, 10=7.81%, 20=27.34%, 50=28.52% lat (msec) : 100=0.26% cpu : usr=0.05%, sys=5.01%, ctx=881, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31029: Sun Apr 7 06:25:42 2024 read: IOPS=41, BW=658KiB/s (674kB/s)(6256KiB/9506msec) clat (usec): min=1805, max=31674, avg=3397.31, stdev=2661.93 lat (usec): min=1806, max=31675, avg=3397.91, stdev=2661.92 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2008], 10.00th=[ 2040], 20.00th=[ 2114], | 30.00th=[ 2180], 40.00th=[ 2278], 50.00th=[ 2409], 60.00th=[ 2573], | 70.00th=[ 2933], 80.00th=[ 4047], 90.00th=[ 6259], 95.00th=[ 7898], | 99.00th=[13173], 99.50th=[18744], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 320, max= 1344, per=27.71%, avg=658.42, stdev=290.58, samples=19 iops : min= 20, max= 84, avg=41.11, stdev=18.14, samples=19 write: IOPS=39, BW=635KiB/s (650kB/s)(6032KiB/9506msec) clat (usec): min=12274, max=63387, avg=21669.42, stdev=7308.00 lat (usec): min=12275, max=63388, avg=21670.38, stdev=7308.05 clat percentiles (usec): | 1.00th=[12518], 5.00th=[13304], 10.00th=[13829], 20.00th=[15008], | 30.00th=[17171], 40.00th=[19268], 50.00th=[21103], 60.00th=[22152], | 70.00th=[23462], 80.00th=[25822], 90.00th=[30278], 95.00th=[35914], | 99.00th=[47449], 99.50th=[55837], 99.90th=[63177], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 416, max= 864, per=25.33%, avg=633.16, stdev=141.23, samples=19 iops : min= 26, max= 54, avg=39.53, stdev= 8.78, samples=19 lat (msec) : 2=2.47%, 4=38.15%, 10=8.46%, 20=23.96%, 50=26.69% lat (msec) : 100=0.26% cpu : usr=0.09%, sys=4.90%, ctx=839, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31030: Sun Apr 7 06:25:42 2024 read: IOPS=39, BW=625KiB/s (640kB/s)(6032KiB/9652msec) clat (usec): min=1825, max=28825, avg=3627.45, stdev=3264.38 lat (usec): min=1825, max=28826, avg=3628.03, stdev=3264.38 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 1958], 10.00th=[ 2024], 20.00th=[ 2114], | 30.00th=[ 2180], 40.00th=[ 2278], 50.00th=[ 2442], 60.00th=[ 2606], | 70.00th=[ 2900], 80.00th=[ 3851], 90.00th=[ 6783], 95.00th=[10945], | 99.00th=[18482], 99.50th=[19792], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 320, max= 1024, per=26.16%, avg=621.47, stdev=203.87, samples=19 iops : min= 20, max= 64, avg=38.84, stdev=12.74, samples=19 write: IOPS=40, BW=648KiB/s (664kB/s)(6256KiB/9652msec) clat (usec): min=11029, max=75636, avg=21166.41, stdev=6531.60 lat (usec): min=11030, max=75638, avg=21167.42, stdev=6531.65 clat percentiles (usec): | 1.00th=[12256], 5.00th=[13042], 10.00th=[13829], 20.00th=[15270], | 30.00th=[17433], 40.00th=[19268], 50.00th=[21103], 60.00th=[21890], | 70.00th=[23200], 80.00th=[24773], 90.00th=[28705], 95.00th=[32375], | 99.00th=[41681], 99.50th=[44303], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 448, max= 800, per=25.80%, avg=645.05, stdev=139.18, samples=19 iops : min= 28, max= 50, avg=40.32, stdev= 8.70, samples=19 lat (msec) : 2=3.65%, 4=35.94%, 10=6.38%, 20=25.26%, 50=28.65% lat (msec) : 100=0.13% cpu : usr=0.08%, sys=5.00%, ctx=859, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2377KiB/s (2434kB/s), 570KiB/s-658KiB/s (584kB/s-674kB/s), io=23.4MiB (24.5MB), run=9506-10078msec WRITE: bw=2500KiB/s (2561kB/s), 635KiB/s-655KiB/s (650kB/s-671kB/s), io=24.6MiB (25.8MB), run=9506-10078msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31044: Sun Apr 7 06:25:47 2024 read: IOPS=1, BW=1799KiB/s (1842kB/s)(7168KiB/3984msec) clat (msec): min=22, max=746, avg=288.41, stdev=319.60 lat (msec): min=22, max=746, avg=288.41, stdev=319.60 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 37], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 55], 60.00th=[ 510], | 70.00th=[ 510], 80.00th=[ 609], 90.00th=[ 751], 95.00th=[ 751], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 2043, max= 4096, per=47.79%, avg=2456.60, stdev=916.46, samples=5 iops : min= 1, max= 4, avg= 2.20, stdev= 1.10, samples=5 write: IOPS=1, BW=1285KiB/s (1316kB/s)(5120KiB/3984msec) clat (msec): min=71, max=1305, avg=390.85, stdev=529.24 lat (msec): min=71, max=1305, avg=390.91, stdev=529.23 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 71], 10.00th=[ 71], 20.00th=[ 71], | 30.00th=[ 87], 40.00th=[ 87], 50.00th=[ 94], 60.00th=[ 94], | 70.00th=[ 397], 80.00th=[ 397], 90.00th=[ 1301], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 2043, max= 4096, per=37.92%, avg=2729.00, stdev=1183.86, samples=3 iops : min= 1, max= 4, avg= 2.33, stdev= 1.53, samples=3 lat (msec) : 50=25.00%, 100=33.33%, 500=8.33%, 750=25.00% cpu : usr=0.05%, sys=4.57%, ctx=52, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31045: Sun Apr 7 06:25:47 2024 read: IOPS=0, BW=519KiB/s (531kB/s)(2048KiB/3947msec) clat (usec): min=34741, max=40258, avg=37499.98, stdev=3900.52 lat (usec): min=34742, max=40259, avg=37501.05, stdev=3901.04 clat percentiles (usec): | 1.00th=[34866], 5.00th=[34866], 10.00th=[34866], 20.00th=[34866], | 30.00th=[34866], 40.00th=[34866], 50.00th=[34866], 60.00th=[40109], | 70.00th=[40109], 80.00th=[40109], 90.00th=[40109], 95.00th=[40109], | 99.00th=[40109], 99.50th=[40109], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 2048, max= 2048, per=39.84%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2594KiB/s (2657kB/s)(10.0MiB/3947msec) clat (msec): min=44, max=932, avg=387.03, stdev=340.23 lat (msec): min=44, max=932, avg=387.09, stdev=340.24 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 60], | 30.00th=[ 88], 40.00th=[ 131], 50.00th=[ 157], 60.00th=[ 409], | 70.00th=[ 600], 80.00th=[ 600], 90.00th=[ 860], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 1706, max= 8192, per=50.24%, avg=3615.20, stdev=2726.84, samples=5 iops : min= 1, max= 8, avg= 3.00, stdev= 2.92, samples=5 lat (msec) : 50=25.00%, 100=16.67%, 250=16.67%, 500=8.33%, 750=16.67% lat (msec) : 1000=16.67% cpu : usr=0.05%, sys=4.31%, ctx=42, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31046: Sun Apr 7 06:25:47 2024 read: IOPS=1, BW=1879KiB/s (1924kB/s)(6144KiB/3270msec) clat (msec): min=30, max=709, avg=326.95, stdev=305.78 lat (msec): min=30, max=709, avg=326.95, stdev=305.78 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 174], 50.00th=[ 174], 60.00th=[ 338], | 70.00th=[ 676], 80.00th=[ 676], 90.00th=[ 709], 95.00th=[ 709], | 99.00th=[ 709], 99.50th=[ 709], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 2048, max= 6144, per=66.41%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 write: IOPS=1, BW=1879KiB/s (1924kB/s)(6144KiB/3270msec) clat (msec): min=38, max=897, avg=217.82, stdev=336.85 lat (msec): min=38, max=897, avg=217.87, stdev=336.85 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 61], 50.00th=[ 61], 60.00th=[ 75], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 894], 95.00th=[ 894], | 99.00th=[ 894], 99.50th=[ 894], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 2043, max= 4096, per=34.14%, avg=2456.60, stdev=916.46, samples=5 iops : min= 1, max= 4, avg= 2.20, stdev= 1.10, samples=5 lat (msec) : 50=25.00%, 100=25.00%, 250=16.67%, 500=8.33%, 750=16.67% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=4.83%, ctx=45, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31047: Sun Apr 7 06:25:47 2024 read: IOPS=1, BW=1583KiB/s (1621kB/s)(5120KiB/3234msec) clat (msec): min=21, max=823, avg=217.79, stdev=343.19 lat (msec): min=21, max=823, avg=217.80, stdev=343.19 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 27], 40.00th=[ 27], 50.00th=[ 60], 60.00th=[ 60], | 70.00th=[ 159], 80.00th=[ 159], 90.00th=[ 827], 95.00th=[ 827], | 99.00th=[ 827], 99.50th=[ 827], 99.90th=[ 827], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2216KiB/s (2270kB/s)(7168KiB/3234msec) clat (msec): min=44, max=951, avg=305.04, stdev=398.53 lat (msec): min=44, max=951, avg=305.10, stdev=398.52 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 57], | 30.00th=[ 60], 40.00th=[ 60], 50.00th=[ 64], 60.00th=[ 144], | 70.00th=[ 144], 80.00th=[ 818], 90.00th=[ 953], 95.00th=[ 953], | 99.00th=[ 953], 99.50th=[ 953], 99.90th=[ 953], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 2048, max= 6144, per=49.77%, avg=3581.75, stdev=1960.03, samples=4 iops : min= 2, max= 6, avg= 3.25, stdev= 1.89, samples=4 lat (msec) : 50=25.00%, 100=33.33%, 250=16.67%, 1000=25.00% cpu : usr=0.03%, sys=4.39%, ctx=48, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5141KiB/s (5264kB/s), 519KiB/s-1879KiB/s (531kB/s-1924kB/s), io=20.0MiB (20.0MB), run=3234-3984msec WRITE: bw=7197KiB/s (7370kB/s), 1285KiB/s-2594KiB/s (1316kB/s-2657kB/s), io=28.0MiB (29.4MB), run=3234-3984msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31048: Sun Apr 7 06:25:47 2024 read: IOPS=1, BW=1860KiB/s (1905kB/s)(7168KiB/3854msec) clat (usec): min=365, max=878878, avg=297640.24, stdev=368813.67 lat (usec): min=366, max=878880, avg=297641.22, stdev=368813.55 clat percentiles (usec): | 1.00th=[ 367], 5.00th=[ 367], 10.00th=[ 367], 20.00th=[ 3523], | 30.00th=[ 14091], 40.00th=[ 14091], 50.00th=[ 36963], 60.00th=[549454], | 70.00th=[549454], 80.00th=[599786], 90.00th=[876610], 95.00th=[876610], | 99.00th=[876610], 99.50th=[876610], 99.90th=[876610], 99.95th=[876610], | 99.99th=[876610] bw ( KiB/s): min= 2048, max= 4096, per=60.26%, avg=3069.75, stdev=1179.82, samples=4 iops : min= 2, max= 4, avg= 2.75, stdev= 0.96, samples=4 write: IOPS=1, BW=1328KiB/s (1360kB/s)(5120KiB/3854msec) clat (msec): min=52, max=713, avg=353.77, stdev=291.80 lat (msec): min=52, max=713, avg=353.79, stdev=291.81 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 54], 10.00th=[ 54], 20.00th=[ 54], | 30.00th=[ 65], 40.00th=[ 65], 50.00th=[ 397], 60.00th=[ 397], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 718], 95.00th=[ 718], | 99.00th=[ 718], 99.50th=[ 718], 99.90th=[ 718], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 2043, max= 4096, per=35.88%, avg=2558.75, stdev=1024.84, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 lat (usec) : 500=8.33% lat (msec) : 4=8.33%, 20=8.33%, 50=8.33%, 100=16.67%, 500=8.33% lat (msec) : 750=33.33%, 1000=8.33% cpu : usr=0.03%, sys=24.99%, ctx=595, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31049: Sun Apr 7 06:25:47 2024 read: IOPS=0, BW=509KiB/s (522kB/s)(2048KiB/4020msec) clat (msec): min=49, max=886, avg=468.20, stdev=591.71 lat (msec): min=49, max=886, avg=468.20, stdev=591.71 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 885], | 70.00th=[ 885], 80.00th=[ 885], 90.00th=[ 885], 95.00th=[ 885], | 99.00th=[ 885], 99.50th=[ 885], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 2048, max= 2048, per=40.20%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2547KiB/s (2608kB/s)(10.0MiB/4020msec) clat (msec): min=36, max=1342, avg=308.13, stdev=448.65 lat (msec): min=36, max=1342, avg=308.18, stdev=448.65 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 40], | 30.00th=[ 42], 40.00th=[ 51], 50.00th=[ 64], 60.00th=[ 68], | 70.00th=[ 430], 80.00th=[ 430], 90.00th=[ 860], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 2048, max= 6144, per=50.25%, avg=3584.00, stdev=1960.81, samples=4 iops : min= 2, max= 6, avg= 3.50, stdev= 1.91, samples=4 lat (msec) : 50=33.33%, 100=25.00%, 250=8.33%, 500=8.33%, 1000=16.67% cpu : usr=0.00%, sys=9.70%, ctx=334, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31050: Sun Apr 7 06:25:47 2024 read: IOPS=1, BW=1934KiB/s (1980kB/s)(6144KiB/3177msec) clat (usec): min=295, max=912691, avg=313661.65, stdev=458469.34 lat (usec): min=295, max=912691, avg=313662.38, stdev=458469.46 clat percentiles (usec): | 1.00th=[ 297], 5.00th=[ 297], 10.00th=[ 297], 20.00th=[ 379], | 30.00th=[ 379], 40.00th=[ 5342], 50.00th=[ 5342], 60.00th=[ 66847], | 70.00th=[893387], 80.00th=[893387], 90.00th=[910164], 95.00th=[910164], | 99.00th=[910164], 99.50th=[910164], 99.90th=[910164], 99.95th=[910164], | 99.99th=[910164] bw ( KiB/s): min= 2048, max= 6144, per=67.01%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 write: IOPS=1, BW=1934KiB/s (1980kB/s)(6144KiB/3177msec) clat (msec): min=27, max=899, avg=215.14, stdev=345.04 lat (msec): min=27, max=899, avg=215.16, stdev=345.04 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 56], | 70.00th=[ 241], 80.00th=[ 241], 90.00th=[ 902], 95.00th=[ 902], | 99.00th=[ 902], 99.50th=[ 902], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 2048, max= 4096, per=43.04%, avg=3069.75, stdev=1179.82, samples=4 iops : min= 2, max= 4, avg= 2.75, stdev= 0.96, samples=4 lat (usec) : 500=16.67% lat (msec) : 10=8.33%, 50=25.00%, 100=16.67%, 250=8.33%, 1000=25.00% cpu : usr=0.00%, sys=11.43%, ctx=586, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31051: Sun Apr 7 06:25:47 2024 read: IOPS=1, BW=1612KiB/s (1650kB/s)(5120KiB/3177msec) clat (msec): min=50, max=879, avg=399.52, stdev=387.45 lat (msec): min=50, max=879, avg=399.52, stdev=387.45 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 51], 10.00th=[ 51], 20.00th=[ 51], | 30.00th=[ 81], 40.00th=[ 81], 50.00th=[ 241], 60.00th=[ 241], | 70.00th=[ 751], 80.00th=[ 751], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 2048, max= 4096, per=53.61%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 write: IOPS=2, BW=2256KiB/s (2310kB/s)(7168KiB/3177msec) clat (msec): min=24, max=830, avg=167.88, stdev=294.59 lat (msec): min=24, max=830, avg=167.93, stdev=294.59 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 65], 80.00th=[ 133], 90.00th=[ 835], 95.00th=[ 835], | 99.00th=[ 835], 99.50th=[ 835], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 2048, max= 6144, per=50.25%, avg=3584.00, stdev=1960.81, samples=4 iops : min= 2, max= 6, avg= 3.50, stdev= 1.91, samples=4 lat (msec) : 50=25.00%, 100=33.33%, 250=16.67%, 750=8.33%, 1000=16.67% cpu : usr=0.00%, sys=13.76%, ctx=693, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5095KiB/s (5217kB/s), 509KiB/s-1934KiB/s (522kB/s-1980kB/s), io=20.0MiB (20.0MB), run=3177-4020msec WRITE: bw=7132KiB/s (7304kB/s), 1328KiB/s-2547KiB/s (1360kB/s-2608kB/s), io=28.0MiB (29.4MB), run=3177-4020msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31061: Sun Apr 7 06:25:52 2024 read: IOPS=0, BW=3269KiB/s (3347kB/s)(12.0MiB/3759msec) clat (msec): min=115, max=3477, avg=1247.61, stdev=1930.93 lat (msec): min=115, max=3477, avg=1247.61, stdev=1930.93 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 115], 10.00th=[ 115], 20.00th=[ 115], | 30.00th=[ 115], 40.00th=[ 150], 50.00th=[ 150], 60.00th=[ 150], | 70.00th=[ 3473], 80.00th=[ 3473], 90.00th=[ 3473], 95.00th=[ 3473], | 99.00th=[ 3473], 99.50th=[ 3473], 99.90th=[ 3473], 99.95th=[ 3473], | 99.99th=[ 3473] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.08%, sys=15.70%, ctx=1408, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31062: Sun Apr 7 06:25:52 2024 read: IOPS=0, BW=1029KiB/s (1054kB/s)(4096KiB/3981msec) clat (nsec): min=151597k, max=151597k, avg=151597455.00, stdev= 0.00 lat (nsec): min=151599k, max=151599k, avg=151598521.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 153], 10.00th=[ 153], 20.00th=[ 153], | 30.00th=[ 153], 40.00th=[ 153], 50.00th=[ 153], 60.00th=[ 153], | 70.00th=[ 153], 80.00th=[ 153], 90.00th=[ 153], 95.00th=[ 153], | 99.00th=[ 153], 99.50th=[ 153], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2058KiB/s (2107kB/s)(8192KiB/3981msec) clat (msec): min=235, max=3572, avg=1903.91, stdev=2359.84 lat (msec): min=235, max=3572, avg=1904.14, stdev=2359.87 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 236], 10.00th=[ 236], 20.00th=[ 236], | 30.00th=[ 236], 40.00th=[ 236], 50.00th=[ 236], 60.00th=[ 3574], | 70.00th=[ 3574], 80.00th=[ 3574], 90.00th=[ 3574], 95.00th=[ 3574], | 99.00th=[ 3574], 99.50th=[ 3574], 99.90th=[ 3574], 99.95th=[ 3574], | 99.99th=[ 3574] bw ( KiB/s): min= 8192, max= 8192, per=99.53%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=5.55%, ctx=115, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31064: Sun Apr 7 06:25:52 2024 write: IOPS=0, BW=3235KiB/s (3312kB/s)(12.0MiB/3799msec) clat (msec): min=160, max=3436, avg=1263.13, stdev=1882.17 lat (msec): min=160, max=3436, avg=1263.44, stdev=1882.12 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 192], 50.00th=[ 192], 60.00th=[ 192], | 70.00th=[ 3440], 80.00th=[ 3440], 90.00th=[ 3440], 95.00th=[ 3440], | 99.00th=[ 3440], 99.50th=[ 3440], 99.90th=[ 3440], 99.95th=[ 3440], | 99.99th=[ 3440] bw ( KiB/s): min= 8192, max= 8192, per=99.53%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=11.66%, ctx=866, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31067: Sun Apr 7 06:25:52 2024 write: IOPS=0, BW=3232KiB/s (3310kB/s)(12.0MiB/3802msec) clat (msec): min=85, max=3524, avg=1266.94, stdev=1955.64 lat (msec): min=85, max=3524, avg=1267.06, stdev=1955.73 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 86], 10.00th=[ 86], 20.00th=[ 86], | 30.00th=[ 86], 40.00th=[ 192], 50.00th=[ 192], 60.00th=[ 192], | 70.00th=[ 3540], 80.00th=[ 3540], 90.00th=[ 3540], 95.00th=[ 3540], | 99.00th=[ 3540], 99.50th=[ 3540], 99.90th=[ 3540], 99.95th=[ 3540], | 99.99th=[ 3540] bw ( KiB/s): min= 8192, max= 8192, per=99.53%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=5.89%, ctx=43, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4116KiB/s (4214kB/s), 1029KiB/s-3269KiB/s (1054kB/s-3347kB/s), io=16.0MiB (16.8MB), run=3759-3981msec WRITE: bw=8231KiB/s (8429kB/s), 2058KiB/s-3235KiB/s (2107kB/s-3312kB/s), io=32.0MiB (33.6MB), run=3799-3981msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31060: Sun Apr 7 06:25:52 2024 read: IOPS=0, BW=3110KiB/s (3185kB/s)(12.0MiB/3951msec) clat (msec): min=66, max=3558, avg=1312.16, stdev=1949.64 lat (msec): min=66, max=3558, avg=1312.16, stdev=1949.64 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 67], | 30.00th=[ 67], 40.00th=[ 313], 50.00th=[ 313], 60.00th=[ 313], | 70.00th=[ 3574], 80.00th=[ 3574], 90.00th=[ 3574], 95.00th=[ 3574], | 99.00th=[ 3574], 99.50th=[ 3574], 99.90th=[ 3574], 99.95th=[ 3574], | 99.99th=[ 3574] bw ( KiB/s): min= 5626, max= 5626, per=100.00%, avg=5626.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.05%, sys=2.61%, ctx=22, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31063: Sun Apr 7 06:25:52 2024 read: IOPS=0, BW=1072KiB/s (1098kB/s)(4096KiB/3821msec) clat (nsec): min=56599k, max=56599k, avg=56598556.00, stdev= 0.00 lat (nsec): min=56600k, max=56600k, avg=56600091.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[56361], 5.00th=[56361], 10.00th=[56361], 20.00th=[56361], | 30.00th=[56361], 40.00th=[56361], 50.00th=[56361], 60.00th=[56361], | 70.00th=[56361], 80.00th=[56361], 90.00th=[56361], 95.00th=[56361], | 99.00th=[56361], 99.50th=[56361], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 5665, max= 5665, per=100.00%, avg=5665.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2144KiB/s (2195kB/s)(8192KiB/3821msec) clat (msec): min=240, max=3516, avg=1878.75, stdev=2316.16 lat (msec): min=241, max=3516, avg=1878.96, stdev=2316.29 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 241], 10.00th=[ 241], 20.00th=[ 241], | 30.00th=[ 241], 40.00th=[ 241], 50.00th=[ 241], 60.00th=[ 3507], | 70.00th=[ 3507], 80.00th=[ 3507], 90.00th=[ 3507], 95.00th=[ 3507], | 99.00th=[ 3507], 99.50th=[ 3507], 99.90th=[ 3507], 99.95th=[ 3507], | 99.99th=[ 3507] bw ( KiB/s): min= 5665, max= 5665, per=71.93%, avg=5665.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=2.85%, ctx=59, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31065: Sun Apr 7 06:25:52 2024 write: IOPS=0, BW=2954KiB/s (3025kB/s)(12.0MiB/4160msec) clat (msec): min=60, max=3581, avg=1368.94, stdev=1926.34 lat (msec): min=61, max=3581, avg=1369.19, stdev=1926.42 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 62], 10.00th=[ 62], 20.00th=[ 62], | 30.00th=[ 62], 40.00th=[ 464], 50.00th=[ 464], 60.00th=[ 464], | 70.00th=[ 3574], 80.00th=[ 3574], 90.00th=[ 3574], 95.00th=[ 3574], | 99.00th=[ 3574], 99.50th=[ 3574], 99.90th=[ 3574], 99.95th=[ 3574], | 99.99th=[ 3574] bw ( KiB/s): min= 7922, max= 7922, per=100.00%, avg=7922.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=3.08%, ctx=157, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31066: Sun Apr 7 06:25:52 2024 write: IOPS=7, BW=28.6MiB/s (30.0MB/s)(12.0MiB/419msec) clat (msec): min=68, max=218, avg=139.21, stdev=75.38 lat (msec): min=68, max=218, avg=139.35, stdev=75.48 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 68], | 30.00th=[ 68], 40.00th=[ 132], 50.00th=[ 132], 60.00th=[ 132], | 70.00th=[ 220], 80.00th=[ 220], 90.00th=[ 220], 95.00th=[ 220], | 99.00th=[ 220], 99.50th=[ 220], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=20.81%, ctx=36, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4147KiB/s (4246kB/s), 1072KiB/s-3110KiB/s (1098kB/s-3185kB/s), io=16.0MiB (16.8MB), run=3821-3951msec WRITE: bw=7877KiB/s (8066kB/s), 2144KiB/s-28.6MiB/s (2195kB/s-30.0MB/s), io=32.0MiB (33.6MB), run=419-4160msec