== sanity test 398b: DIO and buffer IO race ============== 04:09:36 (1733648976) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 0.957774 s, 52.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1370: Sun Dec 8 04:09:50 2024 read: IOPS=127, BW=508KiB/s (520kB/s)(5992KiB/11794msec) clat (usec): min=46, max=161262, avg=3570.42, stdev=4969.99 lat (usec): min=47, max=161263, avg=3571.13, stdev=4969.97 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 59], 10.00th=[ 69], 20.00th=[ 174], | 30.00th=[ 2180], 40.00th=[ 2540], 50.00th=[ 2933], 60.00th=[ 3458], | 70.00th=[ 4359], 80.00th=[ 5604], 90.00th=[ 7177], 95.00th=[ 8717], | 99.00th=[ 12518], 99.50th=[ 14091], 99.90th=[ 25822], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 336, max= 712, per=25.27%, avg=503.22, stdev=95.56, samples=23 iops : min= 84, max= 178, avg=125.78, stdev=23.87, samples=23 write: IOPS=133, BW=534KiB/s (547kB/s)(6296KiB/11794msec) clat (usec): min=419, max=27421, avg=4052.77, stdev=4007.92 lat (usec): min=419, max=27422, avg=4054.94, stdev=4007.64 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 515], 10.00th=[ 545], 20.00th=[ 611], | 30.00th=[ 783], 40.00th=[ 1762], 50.00th=[ 3130], 60.00th=[ 4047], | 70.00th=[ 5145], 80.00th=[ 6783], 90.00th=[ 9503], 95.00th=[11994], | 99.00th=[17433], 99.50th=[21627], 99.90th=[25035], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 440, max= 768, per=25.40%, avg=529.65, stdev=76.52, samples=23 iops : min= 110, max= 192, avg=132.39, stdev=19.13, samples=23 lat (usec) : 50=0.59%, 100=8.53%, 250=0.72%, 500=1.30%, 750=13.57% lat (usec) : 1000=2.99% lat (msec) : 2=5.83%, 4=29.26%, 10=31.35%, 20=5.34%, 50=0.49% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=23.81%, ctx=6009, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1373: Sun Dec 8 04:09:50 2024 read: IOPS=123, BW=493KiB/s (505kB/s)(5940KiB/12056msec) clat (usec): min=47, max=39453, avg=3543.64, stdev=3221.50 lat (usec): min=47, max=39453, avg=3544.51, stdev=3221.30 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 59], 10.00th=[ 63], 20.00th=[ 139], | 30.00th=[ 2180], 40.00th=[ 2507], 50.00th=[ 2999], 60.00th=[ 3523], | 70.00th=[ 4293], 80.00th=[ 5538], 90.00th=[ 7308], 95.00th=[ 8455], | 99.00th=[12387], 99.50th=[19792], 99.90th=[33817], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 360, max= 648, per=24.21%, avg=482.00, stdev=71.76, samples=23 iops : min= 90, max= 162, avg=120.43, stdev=18.00, samples=23 write: IOPS=131, BW=527KiB/s (539kB/s)(6348KiB/12056msec) clat (usec): min=429, max=177518, avg=4236.89, stdev=5984.70 lat (usec): min=429, max=178461, avg=4238.14, stdev=6001.91 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 523], 10.00th=[ 537], 20.00th=[ 619], | 30.00th=[ 848], 40.00th=[ 2114], 50.00th=[ 3261], 60.00th=[ 4113], | 70.00th=[ 5342], 80.00th=[ 6849], 90.00th=[ 9372], 95.00th=[ 11863], | 99.00th=[ 19530], 99.50th=[ 21627], 99.90th=[ 33817], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 408, max= 744, per=24.58%, avg=512.57, stdev=76.15, samples=23 iops : min= 102, max= 186, avg=128.09, stdev=19.07, samples=23 lat (usec) : 50=0.62%, 100=8.56%, 250=0.88%, 500=1.11%, 750=13.25% lat (usec) : 1000=2.57% lat (msec) : 2=5.83%, 4=29.88%, 10=31.97%, 20=4.69%, 50=0.62% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=23.20%, ctx=6108, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1375: Sun Dec 8 04:09:50 2024 read: IOPS=126, BW=508KiB/s (520kB/s)(6028KiB/11867msec) clat (usec): min=46, max=34028, avg=3478.09, stdev=2949.94 lat (usec): min=46, max=34029, avg=3479.72, stdev=2949.43 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 58], 10.00th=[ 63], 20.00th=[ 231], | 30.00th=[ 2147], 40.00th=[ 2507], 50.00th=[ 2868], 60.00th=[ 3392], | 70.00th=[ 4293], 80.00th=[ 5473], 90.00th=[ 7177], 95.00th=[ 8717], | 99.00th=[12256], 99.50th=[14484], 99.90th=[22414], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 344, max= 680, per=25.29%, avg=503.52, stdev=84.39, samples=23 iops : min= 86, max= 170, avg=125.78, stdev=21.14, samples=23 write: IOPS=131, BW=528KiB/s (540kB/s)(6260KiB/11867msec) clat (usec): min=427, max=171704, avg=4187.61, stdev=5905.89 lat (usec): min=428, max=171705, avg=4188.76, stdev=5906.52 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 510], 10.00th=[ 529], 20.00th=[ 586], | 30.00th=[ 816], 40.00th=[ 2008], 50.00th=[ 3195], 60.00th=[ 4047], | 70.00th=[ 5080], 80.00th=[ 6652], 90.00th=[ 9241], 95.00th=[ 11863], | 99.00th=[ 18220], 99.50th=[ 20841], 99.90th=[ 41157], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 384, max= 694, per=24.90%, avg=519.09, stdev=78.58, samples=23 iops : min= 96, max= 173, avg=129.70, stdev=19.63, samples=23 lat (usec) : 50=0.65%, 100=8.72%, 250=0.52%, 500=1.24%, 750=13.25% lat (usec) : 1000=2.15% lat (msec) : 2=6.48%, 4=30.34%, 10=30.99%, 20=5.24%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=22.84%, ctx=6178, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1377: Sun Dec 8 04:09:50 2024 read: IOPS=128, BW=512KiB/s (525kB/s)(6052KiB/11810msec) clat (usec): min=46, max=35449, avg=3455.15, stdev=3003.90 lat (usec): min=46, max=35450, avg=3456.26, stdev=3004.40 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 58], 10.00th=[ 64], 20.00th=[ 125], | 30.00th=[ 2114], 40.00th=[ 2507], 50.00th=[ 2868], 60.00th=[ 3359], | 70.00th=[ 4146], 80.00th=[ 5473], 90.00th=[ 7308], 95.00th=[ 8717], | 99.00th=[12780], 99.50th=[15008], 99.90th=[28443], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 288, max= 784, per=25.29%, avg=503.52, stdev=100.72, samples=23 iops : min= 72, max= 196, avg=125.83, stdev=25.19, samples=23 write: IOPS=132, BW=528KiB/s (541kB/s)(6236KiB/11810msec) clat (usec): min=421, max=157384, avg=4173.24, stdev=5644.87 lat (usec): min=422, max=157385, avg=4174.61, stdev=5644.62 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 519], 10.00th=[ 537], 20.00th=[ 611], | 30.00th=[ 840], 40.00th=[ 1778], 50.00th=[ 3032], 60.00th=[ 3949], | 70.00th=[ 5080], 80.00th=[ 6915], 90.00th=[ 9765], 95.00th=[ 12256], | 99.00th=[ 17433], 99.50th=[ 22938], 99.90th=[ 33162], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 368, max= 688, per=24.95%, avg=520.22, stdev=93.20, samples=23 iops : min= 92, max= 172, avg=130.00, stdev=23.33, samples=23 lat (usec) : 50=0.55%, 100=8.89%, 250=0.78%, 500=1.60%, 750=12.53% lat (usec) : 1000=2.70% lat (msec) : 2=7.00%, 4=30.40%, 10=29.79%, 20=5.24%, 50=0.49% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=23.28%, ctx=6076, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1992KiB/s (2040kB/s), 493KiB/s-512KiB/s (505kB/s-525kB/s), io=23.4MiB (24.6MB), run=11794-12056msec WRITE: bw=2085KiB/s (2135kB/s), 527KiB/s-534KiB/s (539kB/s-547kB/s), io=24.6MiB (25.7MB), run=11794-12056msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1371: Sun Dec 8 04:10:12 2024 read: IOPS=45, BW=180KiB/s (184kB/s)(5992KiB/33287msec) clat (usec): min=1382, max=180607, avg=3675.49, stdev=5379.33 lat (usec): min=1382, max=180608, avg=3676.15, stdev=5379.35 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1778], 10.00th=[ 1844], 20.00th=[ 1942], | 30.00th=[ 2024], 40.00th=[ 2147], 50.00th=[ 2376], 60.00th=[ 2868], | 70.00th=[ 3490], 80.00th=[ 4359], 90.00th=[ 7111], 95.00th=[ 9765], | 99.00th=[ 14484], 99.50th=[ 17433], 99.90th=[ 32113], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 48, max= 312, per=25.09%, avg=179.89, stdev=66.08, samples=66 iops : min= 12, max= 78, avg=44.89, stdev=16.58, samples=66 write: IOPS=47, BW=189KiB/s (194kB/s)(6296KiB/33287msec) clat (usec): min=9259, max=49861, avg=17606.81, stdev=6287.94 lat (usec): min=9260, max=49862, avg=17607.57, stdev=6287.96 clat percentiles (usec): | 1.00th=[10159], 5.00th=[10814], 10.00th=[11338], 20.00th=[12387], | 30.00th=[13566], 40.00th=[14877], 50.00th=[16712], 60.00th=[17957], | 70.00th=[18744], 80.00th=[20579], 90.00th=[26346], 95.00th=[31065], | 99.00th=[39584], 99.50th=[42730], 99.90th=[45876], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 88, max= 248, per=25.17%, avg=188.74, stdev=52.77, samples=66 iops : min= 22, max= 62, avg=47.11, stdev=13.26, samples=66 lat (msec) : 2=13.22%, 4=24.61%, 10=9.08%, 20=41.34%, 50=11.72% lat (msec) : 250=0.03% cpu : usr=0.08%, sys=5.14%, ctx=3642, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1372: Sun Dec 8 04:10:12 2024 read: IOPS=44, BW=177KiB/s (182kB/s)(5940KiB/33489msec) clat (usec): min=1556, max=153834, avg=3687.61, stdev=4922.88 lat (usec): min=1557, max=153844, avg=3688.25, stdev=4923.08 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1778], 10.00th=[ 1860], 20.00th=[ 1942], | 30.00th=[ 2024], 40.00th=[ 2147], 50.00th=[ 2376], 60.00th=[ 2835], | 70.00th=[ 3490], 80.00th=[ 4178], 90.00th=[ 6849], 95.00th=[ 9634], | 99.00th=[ 15533], 99.50th=[ 19268], 99.90th=[ 34866], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 39, max= 336, per=24.73%, avg=177.35, stdev=71.93, samples=66 iops : min= 9, max= 84, avg=44.26, stdev=18.01, samples=66 write: IOPS=47, BW=190KiB/s (194kB/s)(6348KiB/33489msec) clat (usec): min=9153, max=57424, avg=17608.51, stdev=6335.58 lat (usec): min=9154, max=57425, avg=17609.30, stdev=6335.60 clat percentiles (usec): | 1.00th=[10028], 5.00th=[10814], 10.00th=[11338], 20.00th=[12256], | 30.00th=[13435], 40.00th=[15139], 50.00th=[16450], 60.00th=[17695], | 70.00th=[18744], 80.00th=[20579], 90.00th=[26346], 95.00th=[31327], | 99.00th=[38536], 99.50th=[41681], 99.90th=[48497], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 95, max= 248, per=25.08%, avg=188.14, stdev=52.00, samples=66 iops : min= 23, max= 62, avg=46.95, stdev=13.07, samples=66 lat (msec) : 2=12.57%, 4=25.16%, 10=8.82%, 20=41.60%, 50=11.78% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.11%, sys=5.18%, ctx=3674, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1374: Sun Dec 8 04:10:12 2024 read: IOPS=45, BW=182KiB/s (186kB/s)(6028KiB/33186msec) clat (usec): min=1519, max=39646, avg=3815.07, stdev=3246.28 lat (usec): min=1519, max=39647, avg=3815.80, stdev=3246.23 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1795], 10.00th=[ 1860], 20.00th=[ 1958], | 30.00th=[ 2057], 40.00th=[ 2180], 50.00th=[ 2474], 60.00th=[ 3228], | 70.00th=[ 3720], 80.00th=[ 5080], 90.00th=[ 7504], 95.00th=[ 9634], | 99.00th=[16581], 99.50th=[21627], 99.90th=[34866], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 56, max= 319, per=25.32%, avg=181.56, stdev=63.44, samples=66 iops : min= 14, max= 79, avg=45.29, stdev=15.88, samples=66 write: IOPS=47, BW=189KiB/s (193kB/s)(6260KiB/33186msec) clat (msec): min=9, max=171, avg=17.49, stdev= 7.37 lat (msec): min=9, max=171, avg=17.49, stdev= 7.37 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 26], 95.00th=[ 31], | 99.00th=[ 41], 99.50th=[ 45], 99.90th=[ 58], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 88, max= 256, per=25.08%, avg=188.11, stdev=54.70, samples=66 iops : min= 22, max= 64, avg=46.92, stdev=13.73, samples=66 lat (msec) : 2=12.11%, 4=23.93%, 10=11.07%, 20=42.09%, 50=10.71% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.11%, sys=5.22%, ctx=3692, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1376: Sun Dec 8 04:10:12 2024 read: IOPS=45, BW=182KiB/s (187kB/s)(6052KiB/33164msec) clat (usec): min=1478, max=31489, avg=3624.42, stdev=3026.77 lat (usec): min=1478, max=31490, avg=3625.02, stdev=3026.78 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1762], 10.00th=[ 1844], 20.00th=[ 1926], | 30.00th=[ 2008], 40.00th=[ 2147], 50.00th=[ 2343], 60.00th=[ 2835], | 70.00th=[ 3523], 80.00th=[ 4359], 90.00th=[ 7177], 95.00th=[ 9896], | 99.00th=[15926], 99.50th=[17957], 99.90th=[30278], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 56, max= 328, per=25.48%, avg=182.68, stdev=66.76, samples=66 iops : min= 14, max= 82, avg=45.59, stdev=16.74, samples=66 write: IOPS=47, BW=188KiB/s (193kB/s)(6236KiB/33164msec) clat (msec): min=8, max=163, avg=17.71, stdev= 7.12 lat (msec): min=8, max=163, avg=17.71, stdev= 7.12 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 27], 95.00th=[ 31], | 99.00th=[ 40], 99.50th=[ 45], 99.90th=[ 48], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 88, max= 248, per=25.02%, avg=187.65, stdev=51.96, samples=66 iops : min= 22, max= 62, avg=46.83, stdev=13.06, samples=66 lat (msec) : 2=14.58%, 4=23.54%, 10=9.21%, 20=41.47%, 50=11.17% lat (msec) : 250=0.03% cpu : usr=0.10%, sys=5.18%, ctx=3671, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=717KiB/s (734kB/s), 177KiB/s-182KiB/s (182kB/s-187kB/s), io=23.4MiB (24.6MB), run=33164-33489msec WRITE: bw=751KiB/s (769kB/s), 188KiB/s-190KiB/s (193kB/s-194kB/s), io=24.6MiB (25.7MB), run=33164-33489msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1416: Sun Dec 8 04:10:17 2024 read: IOPS=84, BW=1358KiB/s (1390kB/s)(5920KiB/4360msec) clat (usec): min=51, max=35563, avg=8353.00, stdev=5375.25 lat (usec): min=51, max=35563, avg=8353.55, stdev=5375.25 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 61], 10.00th=[ 64], 20.00th=[ 112], | 30.00th=[ 7570], 40.00th=[ 8586], 50.00th=[ 9241], 60.00th=[10028], | 70.00th=[10814], 80.00th=[11863], 90.00th=[13566], 95.00th=[15270], | 99.00th=[24511], 99.50th=[30540], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 736, max= 1536, per=24.37%, avg=1295.62, stdev=251.25, samples=8 iops : min= 46, max= 96, avg=80.88, stdev=15.67, samples=8 write: IOPS=91, BW=1461KiB/s (1496kB/s)(6368KiB/4360msec) clat (usec): min=619, max=108017, avg=3058.71, stdev=6015.89 lat (usec): min=620, max=108018, avg=3059.59, stdev=6015.93 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 693], 10.00th=[ 750], 20.00th=[ 791], | 30.00th=[ 840], 40.00th=[ 930], 50.00th=[ 1598], 60.00th=[ 2540], | 70.00th=[ 3326], 80.00th=[ 4883], 90.00th=[ 6325], 95.00th=[ 7635], | 99.00th=[ 15139], 99.50th=[ 22152], 99.90th=[108528], 99.95th=[108528], | 99.99th=[108528] bw ( KiB/s): min= 896, max= 2176, per=25.30%, avg=1415.50, stdev=456.18, samples=8 iops : min= 56, max= 136, avg=88.38, stdev=28.43, samples=8 lat (usec) : 100=9.37%, 250=0.91%, 750=5.08%, 1000=18.36% lat (msec) : 2=3.65%, 4=12.63%, 10=29.69%, 20=19.14%, 50=1.04% lat (msec) : 250=0.13% cpu : usr=0.11%, sys=21.68%, ctx=2067, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1418: Sun Dec 8 04:10:17 2024 read: IOPS=82, BW=1313KiB/s (1344kB/s)(5744KiB/4376msec) clat (usec): min=48, max=128691, avg=8796.26, stdev=8019.28 lat (usec): min=49, max=128691, avg=8796.86, stdev=8019.30 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 61], 10.00th=[ 68], 20.00th=[ 3785], | 30.00th=[ 7963], 40.00th=[ 8848], 50.00th=[ 9503], 60.00th=[ 10028], | 70.00th=[ 10814], 80.00th=[ 11469], 90.00th=[ 13042], 95.00th=[ 14877], | 99.00th=[ 22414], 99.50th=[ 26608], 99.90th=[128451], 99.95th=[128451], | 99.99th=[128451] bw ( KiB/s): min= 832, max= 1500, per=24.29%, avg=1291.50, stdev=213.72, samples=8 iops : min= 52, max= 93, avg=80.62, stdev=13.26, samples=8 write: IOPS=93, BW=1495KiB/s (1531kB/s)(6544KiB/4376msec) clat (usec): min=612, max=39977, avg=2943.51, stdev=3736.68 lat (usec): min=613, max=39979, avg=2944.72, stdev=3736.63 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 709], 10.00th=[ 750], 20.00th=[ 791], | 30.00th=[ 865], 40.00th=[ 955], 50.00th=[ 1319], 60.00th=[ 2507], | 70.00th=[ 3097], 80.00th=[ 4424], 90.00th=[ 6456], 95.00th=[ 8455], | 99.00th=[16319], 99.50th=[25822], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 1056, max= 2048, per=26.09%, avg=1459.62, stdev=357.02, samples=8 iops : min= 66, max= 128, avg=91.12, stdev=22.40, samples=8 lat (usec) : 50=0.13%, 100=7.81%, 250=1.17%, 750=5.60%, 1000=17.45% lat (msec) : 2=5.08%, 4=12.76%, 10=29.82%, 20=18.88%, 50=1.17% lat (msec) : 250=0.13% cpu : usr=0.09%, sys=24.30%, ctx=2071, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1419: Sun Dec 8 04:10:17 2024 read: IOPS=86, BW=1389KiB/s (1422kB/s)(6256KiB/4504msec) clat (usec): min=49, max=37984, avg=8288.86, stdev=5253.83 lat (usec): min=50, max=37985, avg=8289.40, stdev=5253.83 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 61], 10.00th=[ 63], 20.00th=[ 120], | 30.00th=[ 7701], 40.00th=[ 8717], 50.00th=[ 9241], 60.00th=[10028], | 70.00th=[10552], 80.00th=[11600], 90.00th=[13435], 95.00th=[15008], | 99.00th=[22938], 99.50th=[24511], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 479, max= 2368, per=26.14%, avg=1390.11, stdev=488.72, samples=9 iops : min= 29, max= 148, avg=86.78, stdev=30.76, samples=9 write: IOPS=83, BW=1339KiB/s (1371kB/s)(6032KiB/4504msec) clat (usec): min=606, max=175491, avg=3217.66, stdev=9373.92 lat (usec): min=606, max=175492, avg=3218.54, stdev=9373.92 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 701], 10.00th=[ 758], 20.00th=[ 799], | 30.00th=[ 857], 40.00th=[ 996], 50.00th=[ 1483], 60.00th=[ 2573], | 70.00th=[ 3163], 80.00th=[ 4424], 90.00th=[ 6259], 95.00th=[ 7832], | 99.00th=[ 21890], 99.50th=[ 25035], 99.90th=[175113], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 702, max= 1824, per=23.76%, avg=1329.56, stdev=358.61, samples=9 iops : min= 43, max= 114, avg=83.00, stdev=22.61, samples=9 lat (usec) : 50=0.13%, 100=9.77%, 250=1.04%, 750=4.69%, 1000=15.62% lat (msec) : 2=5.47%, 4=13.02%, 10=29.04%, 20=19.92%, 50=1.17% lat (msec) : 250=0.13% cpu : usr=0.13%, sys=20.83%, ctx=2249, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1420: Sun Dec 8 04:10:17 2024 read: IOPS=85, BW=1360KiB/s (1393kB/s)(6032KiB/4434msec) clat (usec): min=50, max=24434, avg=8324.99, stdev=4824.16 lat (usec): min=50, max=24434, avg=8325.53, stdev=4824.17 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 61], 10.00th=[ 65], 20.00th=[ 139], | 30.00th=[ 7832], 40.00th=[ 8586], 50.00th=[ 9241], 60.00th=[10159], | 70.00th=[10945], 80.00th=[11600], 90.00th=[12780], 95.00th=[14353], | 99.00th=[21103], 99.50th=[23462], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 576, max= 1632, per=24.53%, avg=1304.00, stdev=315.28, samples=8 iops : min= 36, max= 102, avg=81.50, stdev=19.70, samples=8 write: IOPS=88, BW=1411KiB/s (1445kB/s)(6256KiB/4434msec) clat (usec): min=651, max=145425, avg=3183.96, stdev=8066.50 lat (usec): min=652, max=145427, avg=3184.84, stdev=8066.54 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 717], 10.00th=[ 775], 20.00th=[ 807], | 30.00th=[ 873], 40.00th=[ 971], 50.00th=[ 1237], 60.00th=[ 2442], | 70.00th=[ 3097], 80.00th=[ 4146], 90.00th=[ 6063], 95.00th=[ 8455], | 99.00th=[ 20055], 99.50th=[ 39584], 99.90th=[145753], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 672, max= 1760, per=24.31%, avg=1360.00, stdev=346.34, samples=8 iops : min= 42, max= 110, avg=85.00, stdev=21.65, samples=8 lat (usec) : 100=8.98%, 250=0.91%, 750=3.91%, 1000=17.32% lat (msec) : 2=6.12%, 4=13.41%, 10=27.21%, 20=21.22%, 50=0.78% lat (msec) : 250=0.13% cpu : usr=0.07%, sys=21.41%, ctx=2100, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5318KiB/s (5446kB/s), 1313KiB/s-1389KiB/s (1344kB/s-1422kB/s), io=23.4MiB (24.5MB), run=4360-4504msec WRITE: bw=5595KiB/s (5729kB/s), 1339KiB/s-1495KiB/s (1371kB/s-1531kB/s), io=24.6MiB (25.8MB), run=4360-4504msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1413: Sun Dec 8 04:10:21 2024 read: IOPS=42, BW=675KiB/s (691kB/s)(5920KiB/8769msec) clat (usec): min=1826, max=174841, avg=3971.42, stdev=9325.99 lat (usec): min=1827, max=174842, avg=3972.04, stdev=9326.01 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2008], 10.00th=[ 2057], 20.00th=[ 2114], | 30.00th=[ 2180], 40.00th=[ 2278], 50.00th=[ 2442], 60.00th=[ 2638], | 70.00th=[ 2999], 80.00th=[ 3884], 90.00th=[ 6390], 95.00th=[ 10421], | 99.00th=[ 14091], 99.50th=[ 25560], 99.90th=[175113], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 416, max= 1248, per=24.23%, avg=656.94, stdev=223.75, samples=17 iops : min= 26, max= 78, avg=41.06, stdev=13.98, samples=17 write: IOPS=45, BW=726KiB/s (744kB/s)(6368KiB/8769msec) clat (usec): min=9900, max=43730, avg=18260.02, stdev=5980.46 lat (usec): min=9901, max=43731, avg=18261.27, stdev=5980.40 clat percentiles (usec): | 1.00th=[10290], 5.00th=[10945], 10.00th=[11600], 20.00th=[13042], | 30.00th=[14746], 40.00th=[16188], 50.00th=[17433], 60.00th=[18482], | 70.00th=[19792], 80.00th=[22152], 90.00th=[26608], 95.00th=[31065], | 99.00th=[37487], 99.50th=[40109], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 384, max= 960, per=25.01%, avg=713.41, stdev=190.92, samples=17 iops : min= 24, max= 60, avg=44.59, stdev=11.93, samples=17 lat (msec) : 2=2.47%, 4=36.33%, 10=6.77%, 20=39.84%, 50=14.45% lat (msec) : 250=0.13% cpu : usr=0.11%, sys=5.58%, ctx=886, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1414: Sun Dec 8 04:10:21 2024 read: IOPS=40, BW=650KiB/s (666kB/s)(5744KiB/8832msec) clat (usec): min=1773, max=32560, avg=3757.53, stdev=3575.04 lat (usec): min=1773, max=32561, avg=3758.19, stdev=3575.03 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 1975], 10.00th=[ 2040], 20.00th=[ 2114], | 30.00th=[ 2180], 40.00th=[ 2278], 50.00th=[ 2376], 60.00th=[ 2540], | 70.00th=[ 2900], 80.00th=[ 3916], 90.00th=[ 8291], 95.00th=[12125], | 99.00th=[17695], 99.50th=[25035], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 192, max= 1056, per=23.89%, avg=647.53, stdev=253.87, samples=17 iops : min= 12, max= 66, avg=40.47, stdev=15.87, samples=17 write: IOPS=46, BW=741KiB/s (759kB/s)(6544KiB/8832msec) clat (msec): min=9, max=173, avg=18.21, stdev= 9.57 lat (msec): min=9, max=173, avg=18.21, stdev= 9.57 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 19], | 70.00th=[ 21], 80.00th=[ 22], 90.00th=[ 27], 95.00th=[ 29], | 99.00th=[ 37], 99.50th=[ 38], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 416, max= 1056, per=25.34%, avg=722.82, stdev=191.69, samples=17 iops : min= 26, max= 66, avg=45.18, stdev=11.98, samples=17 lat (msec) : 2=3.52%, 4=33.98%, 10=6.25%, 20=40.10%, 50=16.02% lat (msec) : 250=0.13% cpu : usr=0.12%, sys=5.50%, ctx=883, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1415: Sun Dec 8 04:10:21 2024 read: IOPS=45, BW=735KiB/s (752kB/s)(6256KiB/8515msec) clat (usec): min=1790, max=24903, avg=3639.97, stdev=3211.19 lat (usec): min=1791, max=24904, avg=3640.66, stdev=3211.17 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 1991], 10.00th=[ 2040], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2311], 50.00th=[ 2409], 60.00th=[ 2573], | 70.00th=[ 2966], 80.00th=[ 3884], 90.00th=[ 6652], 95.00th=[11076], | 99.00th=[17433], 99.50th=[24249], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 384, max= 1504, per=27.14%, avg=735.82, stdev=302.64, samples=17 iops : min= 24, max= 94, avg=45.88, stdev=19.00, samples=17 write: IOPS=44, BW=708KiB/s (725kB/s)(6032KiB/8515msec) clat (msec): min=10, max=192, avg=18.78, stdev=10.53 lat (msec): min=10, max=192, avg=18.79, stdev=10.53 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 21], 80.00th=[ 22], 90.00th=[ 26], 95.00th=[ 31], | 99.00th=[ 39], 99.50th=[ 44], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 416, max= 960, per=24.80%, avg=707.53, stdev=176.76, samples=17 iops : min= 26, max= 60, avg=44.12, stdev=11.08, samples=17 lat (msec) : 2=3.39%, 4=37.37%, 10=7.42%, 20=36.33%, 50=15.36% lat (msec) : 250=0.13% cpu : usr=0.11%, sys=5.52%, ctx=872, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1417: Sun Dec 8 04:10:21 2024 read: IOPS=43, BW=698KiB/s (714kB/s)(6032KiB/8647msec) clat (usec): min=1760, max=22943, avg=3845.94, stdev=3052.86 lat (usec): min=1760, max=22943, avg=3846.56, stdev=3052.86 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 1975], 10.00th=[ 2040], 20.00th=[ 2114], | 30.00th=[ 2245], 40.00th=[ 2376], 50.00th=[ 2573], 60.00th=[ 2868], | 70.00th=[ 3490], 80.00th=[ 4752], 90.00th=[ 7635], 95.00th=[10814], | 99.00th=[16188], 99.50th=[17695], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 352, max= 1056, per=25.62%, avg=694.59, stdev=186.17, samples=17 iops : min= 22, max= 66, avg=43.41, stdev=11.64, samples=17 write: IOPS=45, BW=723KiB/s (741kB/s)(6256KiB/8647msec) clat (msec): min=9, max=174, avg=18.30, stdev= 9.58 lat (msec): min=9, max=174, avg=18.31, stdev= 9.58 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 20], 80.00th=[ 21], 90.00th=[ 25], 95.00th=[ 31], | 99.00th=[ 45], 99.50th=[ 46], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 384, max= 928, per=25.20%, avg=719.06, stdev=176.41, samples=17 iops : min= 24, max= 58, avg=44.94, stdev=11.03, samples=17 lat (msec) : 2=3.39%, 4=33.46%, 10=9.24%, 20=41.28%, 50=12.50% lat (msec) : 250=0.13% cpu : usr=0.08%, sys=5.60%, ctx=862, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2712KiB/s (2777kB/s), 650KiB/s-735KiB/s (666kB/s-752kB/s), io=23.4MiB (24.5MB), run=8515-8832msec WRITE: bw=2853KiB/s (2922kB/s), 708KiB/s-741KiB/s (725kB/s-759kB/s), io=24.6MiB (25.8MB), run=8515-8832msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1446: Sun Dec 8 04:10:25 2024 read: IOPS=2, BW=2636KiB/s (2700kB/s)(7168KiB/2719msec) clat (msec): min=12, max=613, avg=236.91, stdev=267.19 lat (msec): min=12, max=613, avg=236.91, stdev=267.19 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 24], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 44], 60.00th=[ 451], | 70.00th=[ 451], 80.00th=[ 489], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 2048, max= 4096, per=40.76%, avg=3069.75, stdev=1179.82, samples=4 iops : min= 2, max= 4, avg= 2.75, stdev= 0.96, samples=4 write: IOPS=1, BW=1883KiB/s (1928kB/s)(5120KiB/2719msec) clat (msec): min=21, max=533, avg=205.21, stdev=209.35 lat (msec): min=21, max=533, avg=205.24, stdev=209.35 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 157], 60.00th=[ 157], | 70.00th=[ 275], 80.00th=[ 275], 90.00th=[ 535], 95.00th=[ 535], | 99.00th=[ 535], 99.50th=[ 535], 99.90th=[ 535], 99.95th=[ 535], | 99.99th=[ 535] bw ( KiB/s): min= 1732, max= 4096, per=23.52%, avg=2479.75, stdev=1087.59, samples=4 iops : min= 1, max= 4, avg= 2.00, stdev= 1.41, samples=4 lat (msec) : 20=8.33%, 50=41.67%, 250=8.33%, 500=25.00%, 750=16.67% cpu : usr=0.04%, sys=2.83%, ctx=38, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1447: Sun Dec 8 04:10:25 2024 read: IOPS=0, BW=908KiB/s (930kB/s)(2048KiB/2256msec) clat (usec): min=16430, max=20872, avg=18651.72, stdev=3141.06 lat (usec): min=16432, max=20873, avg=18653.16, stdev=3139.93 clat percentiles (usec): | 1.00th=[16450], 5.00th=[16450], 10.00th=[16450], 20.00th=[16450], | 30.00th=[16450], 40.00th=[16450], 50.00th=[16450], 60.00th=[20841], | 70.00th=[20841], 80.00th=[20841], 90.00th=[20841], 95.00th=[20841], | 99.00th=[20841], 99.50th=[20841], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 2048, max= 2048, per=27.19%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4539KiB/s (4648kB/s)(10.0MiB/2256msec) clat (msec): min=23, max=952, avg=220.98, stdev=318.68 lat (msec): min=23, max=952, avg=221.03, stdev=318.68 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 24], | 30.00th=[ 26], 40.00th=[ 27], 50.00th=[ 77], 60.00th=[ 79], | 70.00th=[ 190], 80.00th=[ 190], 90.00th=[ 642], 95.00th=[ 953], | 99.00th=[ 953], 99.50th=[ 953], 99.90th=[ 953], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 4096, max=12288, per=77.69%, avg=8192.00, stdev=5792.62, samples=2 iops : min= 4, max= 12, avg= 8.00, stdev= 5.66, samples=2 lat (msec) : 20=8.33%, 50=41.67%, 100=16.67%, 250=16.67%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.04%, sys=3.41%, ctx=49, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1448: Sun Dec 8 04:10:25 2024 read: IOPS=2, BW=2891KiB/s (2961kB/s)(6144KiB/2125msec) clat (msec): min=10, max=140, avg=57.11, stdev=47.80 lat (msec): min=10, max=140, avg=57.11, stdev=47.80 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 26], | 30.00th=[ 26], 40.00th=[ 28], 50.00th=[ 28], 60.00th=[ 59], | 70.00th=[ 80], 80.00th=[ 80], 90.00th=[ 140], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 4096, max= 6144, per=67.98%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2891KiB/s (2961kB/s)(6144KiB/2125msec) clat (msec): min=27, max=966, avg=294.66, stdev=372.33 lat (msec): min=27, max=966, avg=294.71, stdev=372.31 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 74], 50.00th=[ 74], 60.00th=[ 169], | 70.00th=[ 493], 80.00th=[ 493], 90.00th=[ 969], 95.00th=[ 969], | 99.00th=[ 969], 99.50th=[ 969], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 2048, max= 4096, per=32.37%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 lat (msec) : 20=8.33%, 50=33.33%, 100=25.00%, 250=16.67%, 500=8.33% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=3.72%, ctx=48, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1450: Sun Dec 8 04:10:25 2024 read: IOPS=2, BW=2324KiB/s (2380kB/s)(5120KiB/2203msec) clat (msec): min=11, max=485, avg=128.25, stdev=202.72 lat (msec): min=11, max=485, avg=128.25, stdev=202.72 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 14], 50.00th=[ 32], 60.00th=[ 32], | 70.00th=[ 100], 80.00th=[ 100], 90.00th=[ 485], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 2048, max= 4096, per=40.79%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 write: IOPS=3, BW=3254KiB/s (3332kB/s)(7168KiB/2203msec) clat (msec): min=22, max=997, avg=215.48, stdev=351.39 lat (msec): min=22, max=997, avg=215.53, stdev=351.39 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 37], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 61], 60.00th=[ 132], | 70.00th=[ 132], 80.00th=[ 218], 90.00th=[ 995], 95.00th=[ 995], | 99.00th=[ 995], 99.50th=[ 995], 99.90th=[ 995], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 2048, max= 6144, per=38.84%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=16.67%, 50=33.33%, 100=16.67%, 250=16.67%, 500=8.33% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=4.36%, ctx=59, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7532KiB/s (7713kB/s), 908KiB/s-2891KiB/s (930kB/s-2961kB/s), io=20.0MiB (20.0MB), run=2125-2719msec WRITE: bw=10.3MiB/s (10.8MB/s), 1883KiB/s-4539KiB/s (1928kB/s-4648kB/s), io=28.0MiB (29.4MB), run=2125-2719msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1449: Sun Dec 8 04:10:25 2024 read: IOPS=2, BW=2665KiB/s (2729kB/s)(7168KiB/2690msec) clat (usec): min=313, max=552940, avg=315988.83, stdev=238299.76 lat (usec): min=314, max=552940, avg=315989.74, stdev=238299.59 clat percentiles (usec): | 1.00th=[ 314], 5.00th=[ 314], 10.00th=[ 314], 20.00th=[ 32113], | 30.00th=[175113], 40.00th=[175113], 50.00th=[476054], 60.00th=[484443], | 70.00th=[484443], 80.00th=[488637], 90.00th=[549454], 95.00th=[549454], | 99.00th=[549454], 99.50th=[549454], 99.90th=[549454], 99.95th=[549454], | 99.99th=[549454] bw ( KiB/s): min= 1783, max= 5361, per=37.57%, avg=2810.00, stdev=1705.25, samples=4 iops : min= 1, max= 5, avg= 2.50, stdev= 1.73, samples=4 write: IOPS=1, BW=1903KiB/s (1949kB/s)(5120KiB/2690msec) clat (msec): min=21, max=149, avg=90.21, stdev=56.79 lat (msec): min=21, max=149, avg=90.25, stdev=56.79 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 94], 60.00th=[ 94], | 70.00th=[ 142], 80.00th=[ 142], 90.00th=[ 150], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 1783, max= 7148, per=42.65%, avg=4465.50, stdev=3793.63, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (usec) : 500=8.33% lat (msec) : 50=25.00%, 100=8.33%, 250=25.00%, 500=25.00%, 750=8.33% cpu : usr=0.00%, sys=15.69%, ctx=1212, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1451: Sun Dec 8 04:10:25 2024 read: IOPS=0, BW=748KiB/s (766kB/s)(2048KiB/2738msec) clat (msec): min=64, max=660, avg=362.28, stdev=421.69 lat (msec): min=64, max=660, avg=362.28, stdev=421.69 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 65], 10.00th=[ 65], 20.00th=[ 65], | 30.00th=[ 65], 40.00th=[ 65], 50.00th=[ 65], 60.00th=[ 659], | 70.00th=[ 659], 80.00th=[ 659], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 1771, max= 2048, per=25.53%, avg=1909.50, stdev=195.87, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3740KiB/s (3830kB/s)(10.0MiB/2738msec) clat (msec): min=21, max=994, avg=200.13, stdev=331.91 lat (msec): min=21, max=994, avg=200.17, stdev=331.91 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 24], | 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 27], 60.00th=[ 29], | 70.00th=[ 197], 80.00th=[ 197], 90.00th=[ 600], 95.00th=[ 995], | 99.00th=[ 995], 99.50th=[ 995], 99.90th=[ 995], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 3543, max=10240, per=55.16%, avg=5775.33, stdev=3866.51, samples=3 iops : min= 3, max= 10, avg= 5.33, stdev= 4.04, samples=3 lat (msec) : 50=50.00%, 100=16.67%, 250=8.33%, 750=16.67%, 1000=8.33% cpu : usr=0.00%, sys=29.23%, ctx=192, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1452: Sun Dec 8 04:10:25 2024 read: IOPS=4, BW=4171KiB/s (4271kB/s)(6144KiB/1473msec) clat (usec): min=327, max=113797, avg=42778.06, stdev=52195.07 lat (usec): min=328, max=113797, avg=42778.75, stdev=52195.01 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 326], 10.00th=[ 326], 20.00th=[ 343], | 30.00th=[ 343], 40.00th=[ 375], 50.00th=[ 375], 60.00th=[ 42730], | 70.00th=[ 99091], 80.00th=[ 99091], 90.00th=[113771], 95.00th=[113771], | 99.00th=[113771], 99.50th=[113771], 99.90th=[113771], 99.95th=[113771], | 99.99th=[113771] write: IOPS=4, BW=4171KiB/s (4271kB/s)(6144KiB/1473msec) clat (msec): min=15, max=966, avg=194.78, stdev=378.76 lat (msec): min=16, max=966, avg=194.81, stdev=378.76 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 23], | 30.00th=[ 23], 40.00th=[ 28], 50.00th=[ 28], 60.00th=[ 53], | 70.00th=[ 85], 80.00th=[ 85], 90.00th=[ 969], 95.00th=[ 969], | 99.00th=[ 969], 99.50th=[ 969], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 4096, max= 4096, per=39.12%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 500=25.00% lat (msec) : 20=8.33%, 50=25.00%, 100=25.00%, 250=8.33%, 1000=8.33% cpu : usr=0.07%, sys=14.27%, ctx=133, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1453: Sun Dec 8 04:10:25 2024 read: IOPS=3, BW=3553KiB/s (3638kB/s)(5120KiB/1441msec) clat (usec): min=316, max=102468, avg=29833.05, stdev=44256.16 lat (usec): min=316, max=102469, avg=29833.80, stdev=44256.31 clat percentiles (usec): | 1.00th=[ 318], 5.00th=[ 318], 10.00th=[ 318], 20.00th=[ 318], | 30.00th=[ 359], 40.00th=[ 359], 50.00th=[ 3982], 60.00th=[ 3982], | 70.00th=[ 42206], 80.00th=[ 42206], 90.00th=[102237], 95.00th=[102237], | 99.00th=[102237], 99.50th=[102237], 99.90th=[102237], 99.95th=[102237], | 99.99th=[102237] write: IOPS=4, BW=4974KiB/s (5094kB/s)(7168KiB/1441msec) clat (msec): min=21, max=988, avg=179.76, stdev=357.48 lat (msec): min=21, max=988, avg=179.80, stdev=357.47 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 25], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 43], 60.00th=[ 63], | 70.00th=[ 63], 80.00th=[ 91], 90.00th=[ 986], 95.00th=[ 986], | 99.00th=[ 986], 99.50th=[ 986], 99.90th=[ 986], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 2043, max= 2043, per=19.51%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 4=8.33%, 50=41.67%, 100=16.67%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=12.99%, ctx=79, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7480KiB/s (7659kB/s), 748KiB/s-4171KiB/s (766kB/s-4271kB/s), io=20.0MiB (20.0MB), run=1441-2738msec WRITE: bw=10.2MiB/s (10.7MB/s), 1903KiB/s-4974KiB/s (1949kB/s-5094kB/s), io=28.0MiB (29.4MB), run=1441-2738msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1460: Sun Dec 8 04:10:28 2024 read: IOPS=1, BW=6707KiB/s (6868kB/s)(12.0MiB/1832msec) clat (msec): min=139, max=1130, avg=602.16, stdev=499.03 lat (msec): min=139, max=1130, avg=602.16, stdev=499.03 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 140], 10.00th=[ 140], 20.00th=[ 140], | 30.00th=[ 140], 40.00th=[ 535], 50.00th=[ 535], 60.00th=[ 535], | 70.00th=[ 1133], 80.00th=[ 1133], 90.00th=[ 1133], 95.00th=[ 1133], | 99.00th=[ 1133], 99.50th=[ 1133], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 8192, max= 8192, per=91.60%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=48.33%, ctx=744, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1462: Sun Dec 8 04:10:28 2024 read: IOPS=0, BW=2376KiB/s (2433kB/s)(4096KiB/1724msec) clat (nsec): min=114356k, max=114356k, avg=114356307.00, stdev= 0.00 lat (nsec): min=114359k, max=114359k, avg=114358822.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 115], 10.00th=[ 115], 20.00th=[ 115], | 30.00th=[ 115], 40.00th=[ 115], 50.00th=[ 115], 60.00th=[ 115], | 70.00th=[ 115], 80.00th=[ 115], 90.00th=[ 115], 95.00th=[ 115], | 99.00th=[ 115], 99.50th=[ 115], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 8192, max= 8192, per=91.60%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4752KiB/s (4866kB/s)(8192KiB/1724msec) clat (msec): min=177, max=1412, avg=795.15, stdev=872.84 lat (msec): min=178, max=1412, avg=795.38, stdev=872.86 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 178], 10.00th=[ 178], 20.00th=[ 178], | 30.00th=[ 178], 40.00th=[ 178], 50.00th=[ 178], 60.00th=[ 1418], | 70.00th=[ 1418], 80.00th=[ 1418], 90.00th=[ 1418], 95.00th=[ 1418], | 99.00th=[ 1418], 99.50th=[ 1418], 99.90th=[ 1418], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 8192, max= 8192, per=46.75%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=12.36%, ctx=183, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1465: Sun Dec 8 04:10:28 2024 write: IOPS=1, BW=6571KiB/s (6729kB/s)(12.0MiB/1870msec) clat (msec): min=191, max=1344, avg=612.30, stdev=636.60 lat (msec): min=192, max=1344, avg=612.55, stdev=636.61 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 192], 10.00th=[ 192], 20.00th=[ 192], | 30.00th=[ 192], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 1351], 80.00th=[ 1351], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 8192, max= 8192, per=46.75%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=11.56%, ctx=60, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1467: Sun Dec 8 04:10:28 2024 write: IOPS=1, BW=7091KiB/s (7261kB/s)(12.0MiB/1733msec) clat (msec): min=166, max=1355, avg=566.37, stdev=683.64 lat (msec): min=166, max=1355, avg=566.62, stdev=683.64 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 167], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 178], 60.00th=[ 178], | 70.00th=[ 1351], 80.00th=[ 1351], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 8192, max= 8192, per=46.75%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.06%, sys=11.26%, ctx=93, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8943KiB/s (9158kB/s), 2376KiB/s-6707KiB/s (2433kB/s-6868kB/s), io=16.0MiB (16.8MB), run=1724-1832msec WRITE: bw=17.1MiB/s (17.9MB/s), 4752KiB/s-7091KiB/s (4866kB/s-7261kB/s), io=32.0MiB (33.6MB), run=1724-1870msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1461: Sun Dec 8 04:10:28 2024 read: IOPS=1, BW=6603KiB/s (6761kB/s)(12.0MiB/1861msec) clat (msec): min=213, max=1370, avg=609.72, stdev=659.19 lat (msec): min=213, max=1370, avg=609.72, stdev=659.19 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 213], 10.00th=[ 213], 20.00th=[ 213], | 30.00th=[ 213], 40.00th=[ 245], 50.00th=[ 245], 60.00th=[ 245], | 70.00th=[ 1368], 80.00th=[ 1368], 90.00th=[ 1368], 95.00th=[ 1368], | 99.00th=[ 1368], 99.50th=[ 1368], 99.90th=[ 1368], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min= 5610, max= 8192, per=78.39%, avg=6901.00, stdev=1825.75, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=2.80%, ctx=47, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1463: Sun Dec 8 04:10:28 2024 read: IOPS=0, BW=2273KiB/s (2328kB/s)(4096KiB/1802msec) clat (nsec): min=202051k, max=202051k, avg=202051405.00, stdev= 0.00 lat (nsec): min=202053k, max=202053k, avg=202053198.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 203], 10.00th=[ 203], 20.00th=[ 203], | 30.00th=[ 203], 40.00th=[ 203], 50.00th=[ 203], 60.00th=[ 203], | 70.00th=[ 203], 80.00th=[ 203], 90.00th=[ 203], 95.00th=[ 203], | 99.00th=[ 203], 99.50th=[ 203], 99.90th=[ 203], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 8192, max= 8192, per=93.06%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4546KiB/s (4655kB/s)(8192KiB/1802msec) clat (msec): min=165, max=1415, avg=790.82, stdev=883.90 lat (msec): min=165, max=1416, avg=791.01, stdev=883.98 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 167], | 30.00th=[ 167], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 1418], | 70.00th=[ 1418], 80.00th=[ 1418], 90.00th=[ 1418], 95.00th=[ 1418], | 99.00th=[ 1418], 99.50th=[ 1418], 99.90th=[ 1418], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 8192, max= 8192, per=47.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.06%, sys=3.83%, ctx=44, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1466: Sun Dec 8 04:10:28 2024 write: IOPS=1, BW=6508KiB/s (6665kB/s)(12.0MiB/1888msec) clat (msec): min=38, max=1561, avg=619.99, stdev=822.99 lat (msec): min=38, max=1561, avg=620.21, stdev=822.91 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 259], 50.00th=[ 259], 60.00th=[ 259], | 70.00th=[ 1569], 80.00th=[ 1569], 90.00th=[ 1569], 95.00th=[ 1569], | 99.00th=[ 1569], 99.50th=[ 1569], 99.90th=[ 1569], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 8192, max= 8192, per=47.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 500=33.33% cpu : usr=0.00%, sys=3.92%, ctx=49, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1468: Sun Dec 8 04:10:28 2024 write: IOPS=2, BW=8286KiB/s (8485kB/s)(12.0MiB/1483msec) clat (msec): min=39, max=1305, avg=490.59, stdev=706.65 lat (msec): min=40, max=1305, avg=490.86, stdev=706.64 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 127], 50.00th=[ 127], 60.00th=[ 127], | 70.00th=[ 1301], 80.00th=[ 1301], 90.00th=[ 1301], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 8175, max= 8175, per=47.10%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33% cpu : usr=0.00%, sys=5.26%, ctx=38, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8804KiB/s (9015kB/s), 2273KiB/s-6603KiB/s (2328kB/s-6761kB/s), io=16.0MiB (16.8MB), run=1802-1861msec WRITE: bw=16.9MiB/s (17.8MB/s), 4546KiB/s-8286KiB/s (4655kB/s-8485kB/s), io=32.0MiB (33.6MB), run=1483-1888msec