== sanity test 398b: DIO and buffer IO race ============== 02:39:46 (1715582386) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.11092 s, 45.3 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1140: Mon May 13 02:40:01 2024 read: IOPS=123, BW=496KiB/s (508kB/s)(5992KiB/12084msec) clat (usec): min=46, max=169214, avg=3404.27, stdev=5307.72 lat (usec): min=47, max=169214, avg=3404.96, stdev=5307.70 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 59], 10.00th=[ 62], 20.00th=[ 76], | 30.00th=[ 1975], 40.00th=[ 2343], 50.00th=[ 2802], 60.00th=[ 3326], | 70.00th=[ 4113], 80.00th=[ 5276], 90.00th=[ 7373], 95.00th=[ 9241], | 99.00th=[ 12518], 99.50th=[ 15664], 99.90th=[ 35390], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 272, max= 856, per=24.95%, avg=495.50, stdev=111.70, samples=24 iops : min= 68, max= 214, avg=123.79, stdev=27.94, samples=24 write: IOPS=130, BW=521KiB/s (534kB/s)(6296KiB/12084msec) clat (usec): min=420, max=41547, avg=4370.27, stdev=4459.46 lat (usec): min=420, max=41548, avg=4371.26, stdev=4459.51 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 652], | 30.00th=[ 930], 40.00th=[ 2114], 50.00th=[ 3392], 60.00th=[ 4490], | 70.00th=[ 5538], 80.00th=[ 6980], 90.00th=[ 9765], 95.00th=[12649], | 99.00th=[19530], 99.50th=[26084], 99.90th=[36963], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 336, max= 872, per=25.03%, avg=520.46, stdev=115.41, samples=24 iops : min= 84, max= 218, avg=130.04, stdev=28.86, samples=24 lat (usec) : 50=0.62%, 100=11.46%, 250=1.17%, 500=0.94%, 750=12.27% lat (usec) : 1000=2.96% lat (msec) : 2=5.92%, 4=26.40%, 10=31.61%, 20=6.02%, 50=0.59% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=22.74%, ctx=5803, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1141: Mon May 13 02:40:01 2024 read: IOPS=122, BW=492KiB/s (504kB/s)(5940KiB/12079msec) clat (usec): min=47, max=44737, avg=3278.06, stdev=3050.46 lat (usec): min=47, max=44738, avg=3278.64, stdev=3050.46 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 59], 10.00th=[ 62], 20.00th=[ 77], | 30.00th=[ 1958], 40.00th=[ 2311], 50.00th=[ 2802], 60.00th=[ 3326], | 70.00th=[ 4113], 80.00th=[ 5473], 90.00th=[ 7177], 95.00th=[ 8848], | 99.00th=[11994], 99.50th=[13566], 99.90th=[20055], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 272, max= 728, per=24.70%, avg=490.50, stdev=95.15, samples=24 iops : min= 68, max= 182, avg=122.54, stdev=23.79, samples=24 write: IOPS=131, BW=526KiB/s (538kB/s)(6348KiB/12079msec) clat (usec): min=430, max=156010, avg=4475.88, stdev=5848.53 lat (usec): min=431, max=156012, avg=4476.72, stdev=5848.49 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 635], | 30.00th=[ 914], 40.00th=[ 2089], 50.00th=[ 3425], 60.00th=[ 4359], | 70.00th=[ 5473], 80.00th=[ 7177], 90.00th=[ 10290], 95.00th=[ 12911], | 99.00th=[ 19792], 99.50th=[ 21627], 99.90th=[ 51119], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 352, max= 864, per=25.11%, avg=522.12, stdev=110.22, samples=24 iops : min= 88, max= 216, avg=130.46, stdev=27.57, samples=24 lat (usec) : 50=0.42%, 100=11.10%, 250=1.20%, 500=0.88%, 750=12.96% lat (usec) : 1000=2.99% lat (msec) : 2=5.66%, 4=26.37%, 10=31.77%, 20=6.12%, 50=0.46% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.17%, sys=23.01%, ctx=5957, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1142: Mon May 13 02:40:01 2024 read: IOPS=124, BW=499KiB/s (511kB/s)(6028KiB/12083msec) clat (usec): min=46, max=56401, avg=3301.67, stdev=3350.45 lat (usec): min=46, max=56402, avg=3302.26, stdev=3350.44 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 58], 10.00th=[ 62], 20.00th=[ 77], | 30.00th=[ 1958], 40.00th=[ 2409], 50.00th=[ 2769], 60.00th=[ 3261], | 70.00th=[ 3982], 80.00th=[ 5342], 90.00th=[ 7308], 95.00th=[ 8717], | 99.00th=[11863], 99.50th=[14222], 99.90th=[40633], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 200, max= 696, per=25.07%, avg=497.96, stdev=104.70, samples=24 iops : min= 50, max= 174, avg=124.46, stdev=26.18, samples=24 write: IOPS=129, BW=518KiB/s (531kB/s)(6260KiB/12083msec) clat (usec): min=458, max=176879, avg=4473.96, stdev=6078.58 lat (usec): min=459, max=176881, avg=4474.77, stdev=6078.54 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 652], | 30.00th=[ 955], 40.00th=[ 2180], 50.00th=[ 3523], 60.00th=[ 4490], | 70.00th=[ 5473], 80.00th=[ 7242], 90.00th=[ 10159], 95.00th=[ 12649], | 99.00th=[ 17695], 99.50th=[ 20579], 99.90th=[ 39584], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 280, max= 680, per=24.82%, avg=515.96, stdev=84.81, samples=24 iops : min= 70, max= 170, avg=128.96, stdev=21.21, samples=24 lat (usec) : 50=0.72%, 100=10.94%, 250=1.14%, 500=0.98%, 750=11.91% lat (usec) : 1000=3.09% lat (msec) : 2=6.35%, 4=27.08%, 10=31.25%, 20=6.05%, 50=0.42% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.13%, sys=22.97%, ctx=5996, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1143: Mon May 13 02:40:01 2024 read: IOPS=125, BW=501KiB/s (513kB/s)(6052KiB/12090msec) clat (usec): min=46, max=27117, avg=3253.84, stdev=3069.46 lat (usec): min=46, max=27118, avg=3254.54, stdev=3069.42 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 58], 10.00th=[ 61], 20.00th=[ 76], | 30.00th=[ 1926], 40.00th=[ 2343], 50.00th=[ 2704], 60.00th=[ 3195], | 70.00th=[ 3982], 80.00th=[ 5211], 90.00th=[ 7504], 95.00th=[ 9372], | 99.00th=[12387], 99.50th=[16450], 99.90th=[19268], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 208, max= 814, per=25.04%, avg=497.21, stdev=119.71, samples=24 iops : min= 52, max= 203, avg=124.25, stdev=29.88, samples=24 write: IOPS=128, BW=516KiB/s (528kB/s)(6236KiB/12090msec) clat (usec): min=414, max=177391, avg=4528.01, stdev=6380.24 lat (usec): min=415, max=177392, avg=4529.08, stdev=6380.11 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 627], | 30.00th=[ 881], 40.00th=[ 1811], 50.00th=[ 3326], 60.00th=[ 4490], | 70.00th=[ 5473], 80.00th=[ 7242], 90.00th=[ 10290], 95.00th=[ 13566], | 99.00th=[ 19792], 99.50th=[ 25560], 99.90th=[ 47973], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 232, max= 806, per=24.68%, avg=513.17, stdev=119.37, samples=24 iops : min= 58, max= 201, avg=128.25, stdev=29.78, samples=24 lat (usec) : 50=0.88%, 100=11.39%, 250=1.27%, 500=1.04%, 750=12.60% lat (usec) : 1000=2.99% lat (msec) : 2=6.09%, 4=26.53%, 10=30.37%, 20=6.32%, 50=0.49% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=22.50%, ctx=5826, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1986KiB/s (2034kB/s), 492KiB/s-501KiB/s (504kB/s-513kB/s), io=23.4MiB (24.6MB), run=12079-12090msec WRITE: bw=2079KiB/s (2129kB/s), 516KiB/s-526KiB/s (528kB/s-538kB/s), io=24.6MiB (25.7MB), run=12079-12090msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1136: Mon May 13 02:40:30 2024 read: IOPS=37, BW=149KiB/s (153kB/s)(5992KiB/40089msec) clat (usec): min=1418, max=196741, avg=3800.79, stdev=5726.47 lat (usec): min=1419, max=196744, avg=3801.54, stdev=5726.50 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1795], 10.00th=[ 1876], 20.00th=[ 1975], | 30.00th=[ 2114], 40.00th=[ 2278], 50.00th=[ 2573], 60.00th=[ 3294], | 70.00th=[ 3720], 80.00th=[ 4359], 90.00th=[ 6915], 95.00th=[ 9634], | 99.00th=[ 15664], 99.50th=[ 16581], 99.90th=[ 32637], 99.95th=[196084], | 99.99th=[196084] bw ( KiB/s): min= 24, max= 304, per=25.40%, avg=149.61, stdev=57.66, samples=80 iops : min= 6, max= 76, avg=37.34, stdev=14.43, samples=80 write: IOPS=39, BW=157KiB/s (161kB/s)(6296KiB/40089msec) clat (usec): min=11636, max=64074, avg=21779.60, stdev=8053.97 lat (usec): min=11637, max=64075, avg=21780.38, stdev=8054.00 clat percentiles (usec): | 1.00th=[12649], 5.00th=[13566], 10.00th=[14091], 20.00th=[15008], | 30.00th=[16581], 40.00th=[18220], 50.00th=[20579], 60.00th=[21890], | 70.00th=[23200], 80.00th=[25297], 90.00th=[33162], 95.00th=[39060], | 99.00th=[50070], 99.50th=[53216], 99.90th=[60031], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 48, max= 208, per=25.43%, avg=156.91, stdev=43.94, samples=80 iops : min= 12, max= 52, avg=39.16, stdev=10.99, samples=80 lat (msec) : 2=10.71%, 4=26.20%, 10=9.64%, 20=26.37%, 50=26.53% lat (msec) : 100=0.52%, 250=0.03% cpu : usr=0.09%, sys=4.31%, ctx=3733, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1137: Mon May 13 02:40:30 2024 read: IOPS=36, BW=146KiB/s (149kB/s)(5940KiB/40734msec) clat (usec): min=1464, max=160752, avg=3723.98, stdev=4989.51 lat (usec): min=1465, max=160755, avg=3724.63, stdev=4989.57 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1778], 10.00th=[ 1860], 20.00th=[ 1975], | 30.00th=[ 2114], 40.00th=[ 2311], 50.00th=[ 2638], 60.00th=[ 3326], | 70.00th=[ 3654], 80.00th=[ 4146], 90.00th=[ 6390], 95.00th=[ 9372], | 99.00th=[ 15664], 99.50th=[ 17695], 99.90th=[ 40633], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 16, max= 280, per=24.73%, avg=145.67, stdev=60.93, samples=81 iops : min= 4, max= 70, avg=36.41, stdev=15.25, samples=81 write: IOPS=38, BW=156KiB/s (160kB/s)(6348KiB/40734msec) clat (usec): min=11067, max=81385, avg=22111.67, stdev=8385.54 lat (usec): min=11068, max=81386, avg=22112.48, stdev=8385.56 clat percentiles (usec): | 1.00th=[12649], 5.00th=[13566], 10.00th=[14222], 20.00th=[15401], | 30.00th=[16909], 40.00th=[18744], 50.00th=[20841], 60.00th=[22152], | 70.00th=[23725], 80.00th=[25297], 90.00th=[33817], 95.00th=[40109], | 99.00th=[50594], 99.50th=[54789], 99.90th=[73925], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 48, max= 208, per=25.16%, avg=155.25, stdev=43.32, samples=81 iops : min= 12, max= 52, avg=38.80, stdev=10.84, samples=81 lat (msec) : 2=10.06%, 4=27.38%, 10=8.79%, 20=25.55%, 50=27.64% lat (msec) : 100=0.55%, 250=0.03% cpu : usr=0.07%, sys=4.26%, ctx=3714, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1138: Mon May 13 02:40:30 2024 read: IOPS=37, BW=148KiB/s (152kB/s)(6028KiB/40641msec) clat (usec): min=1426, max=63111, avg=3852.54, stdev=3856.39 lat (usec): min=1427, max=63112, avg=3853.26, stdev=3856.47 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 1778], 10.00th=[ 1876], 20.00th=[ 2008], | 30.00th=[ 2147], 40.00th=[ 2343], 50.00th=[ 2638], 60.00th=[ 3326], | 70.00th=[ 3720], 80.00th=[ 4293], 90.00th=[ 6849], 95.00th=[10683], | 99.00th=[17433], 99.50th=[21627], 99.90th=[51643], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 40, max= 288, per=25.14%, avg=148.05, stdev=55.59, samples=81 iops : min= 10, max= 72, avg=36.94, stdev=13.90, samples=81 write: IOPS=38, BW=154KiB/s (158kB/s)(6260KiB/40641msec) clat (msec): min=11, max=175, avg=22.19, stdev= 8.75 lat (msec): min=11, max=175, avg=22.19, stdev= 8.75 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 26], 90.00th=[ 33], 95.00th=[ 40], | 99.00th=[ 50], 99.50th=[ 54], 99.90th=[ 68], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 48, max= 210, per=24.91%, avg=153.70, stdev=44.43, samples=81 iops : min= 12, max= 52, avg=38.35, stdev=11.10, samples=81 lat (msec) : 2=9.77%, 4=27.77%, 10=8.63%, 20=25.29%, 50=28.06% lat (msec) : 100=0.46%, 250=0.03% cpu : usr=0.09%, sys=4.29%, ctx=3727, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1139: Mon May 13 02:40:30 2024 read: IOPS=37, BW=149KiB/s (153kB/s)(6052KiB/40495msec) clat (usec): min=1408, max=62931, avg=3779.11, stdev=3635.18 lat (usec): min=1408, max=62931, avg=3779.80, stdev=3635.17 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1795], 10.00th=[ 1876], 20.00th=[ 2008], | 30.00th=[ 2147], 40.00th=[ 2343], 50.00th=[ 2671], 60.00th=[ 3359], | 70.00th=[ 3720], 80.00th=[ 4293], 90.00th=[ 6456], 95.00th=[ 9634], | 99.00th=[15139], 99.50th=[19268], 99.90th=[51643], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 32, max= 320, per=25.20%, avg=148.44, stdev=61.31, samples=80 iops : min= 8, max= 80, avg=37.06, stdev=15.37, samples=80 write: IOPS=38, BW=154KiB/s (158kB/s)(6236KiB/40495msec) clat (msec): min=11, max=176, avg=22.23, stdev= 9.41 lat (msec): min=11, max=176, avg=22.24, stdev= 9.41 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 26], 90.00th=[ 34], 95.00th=[ 42], | 99.00th=[ 54], 99.50th=[ 62], 99.90th=[ 72], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 48, max= 200, per=24.85%, avg=153.34, stdev=43.98, samples=80 iops : min= 12, max= 50, avg=38.29, stdev=11.02, samples=80 lat (msec) : 2=9.41%, 4=28.03%, 10=9.57%, 20=26.04%, 50=26.11% lat (msec) : 100=0.81%, 250=0.03% cpu : usr=0.10%, sys=4.25%, ctx=3775, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=589KiB/s (604kB/s), 146KiB/s-149KiB/s (149kB/s-153kB/s), io=23.4MiB (24.6MB), run=40089-40734msec WRITE: bw=617KiB/s (632kB/s), 154KiB/s-157KiB/s (158kB/s-161kB/s), io=24.6MiB (25.7MB), run=40089-40734msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1188: Mon May 13 02:40:35 2024 read: IOPS=76, BW=1217KiB/s (1246kB/s)(5920KiB/4866msec) clat (usec): min=53, max=48110, avg=9434.67, stdev=5802.38 lat (usec): min=54, max=48111, avg=9435.81, stdev=5802.93 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 66], 10.00th=[ 77], 20.00th=[ 6980], | 30.00th=[ 8356], 40.00th=[ 9372], 50.00th=[10028], 60.00th=[10683], | 70.00th=[11600], 80.00th=[12649], 90.00th=[13960], 95.00th=[16319], | 99.00th=[28443], 99.50th=[39060], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 640, max= 1376, per=23.25%, avg=1144.56, stdev=214.30, samples=9 iops : min= 40, max= 86, avg=71.44, stdev=13.33, samples=9 write: IOPS=81, BW=1309KiB/s (1340kB/s)(6368KiB/4866msec) clat (usec): min=643, max=35346, avg=3437.20, stdev=4731.23 lat (usec): min=643, max=35347, avg=3438.12, stdev=4731.23 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 750], 10.00th=[ 783], 20.00th=[ 824], | 30.00th=[ 889], 40.00th=[ 979], 50.00th=[ 1696], 60.00th=[ 2737], | 70.00th=[ 3621], 80.00th=[ 5211], 90.00th=[ 6849], 95.00th=[10290], | 99.00th=[28967], 99.50th=[31851], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 736, max= 1792, per=24.10%, avg=1247.78, stdev=351.59, samples=9 iops : min= 46, max= 112, avg=77.89, stdev=22.12, samples=9 lat (usec) : 100=7.68%, 250=0.65%, 750=3.12%, 1000=18.23% lat (msec) : 2=5.34%, 4=11.85%, 10=25.78%, 20=25.13%, 50=2.21% cpu : usr=0.14%, sys=21.19%, ctx=2095, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1190: Mon May 13 02:40:35 2024 read: IOPS=77, BW=1240KiB/s (1270kB/s)(5744KiB/4632msec) clat (usec): min=50, max=44954, avg=9078.13, stdev=5513.24 lat (usec): min=51, max=44955, avg=9080.48, stdev=5513.65 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 64], 10.00th=[ 74], 20.00th=[ 4146], | 30.00th=[ 8291], 40.00th=[ 9372], 50.00th=[10028], 60.00th=[10683], | 70.00th=[11469], 80.00th=[12649], 90.00th=[14353], 95.00th=[15926], | 99.00th=[22414], 99.50th=[30278], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 896, max= 1504, per=24.92%, avg=1226.67, stdev=195.96, samples=9 iops : min= 56, max= 94, avg=76.67, stdev=12.25, samples=9 write: IOPS=88, BW=1413KiB/s (1447kB/s)(6544KiB/4632msec) clat (usec): min=609, max=72385, avg=3335.37, stdev=4827.02 lat (usec): min=610, max=72386, avg=3336.38, stdev=4827.09 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 758], 10.00th=[ 783], 20.00th=[ 840], | 30.00th=[ 906], 40.00th=[ 1045], 50.00th=[ 2376], 60.00th=[ 2868], | 70.00th=[ 3490], 80.00th=[ 5211], 90.00th=[ 6980], 95.00th=[ 8848], | 99.00th=[17171], 99.50th=[20579], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 1152, max= 1792, per=26.51%, avg=1372.44, stdev=178.49, samples=9 iops : min= 72, max= 112, avg=85.78, stdev=11.16, samples=9 lat (usec) : 100=8.20%, 250=0.91%, 750=1.95%, 1000=17.97% lat (msec) : 2=5.73%, 4=14.06%, 10=26.30%, 20=23.83%, 50=0.91% lat (msec) : 100=0.13% cpu : usr=0.13%, sys=21.72%, ctx=2055, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1192: Mon May 13 02:40:35 2024 read: IOPS=83, BW=1332KiB/s (1364kB/s)(6256KiB/4697msec) clat (usec): min=53, max=47913, avg=9070.65, stdev=5738.03 lat (usec): min=54, max=47914, avg=9071.26, stdev=5738.03 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 66], 10.00th=[ 74], 20.00th=[ 510], | 30.00th=[ 8455], 40.00th=[ 9241], 50.00th=[ 9896], 60.00th=[10683], | 70.00th=[11600], 80.00th=[12649], 90.00th=[13829], 95.00th=[14615], | 99.00th=[28967], 99.50th=[36963], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 960, max= 1664, per=27.02%, avg=1329.78, stdev=212.33, samples=9 iops : min= 60, max= 104, avg=83.11, stdev=13.27, samples=9 write: IOPS=80, BW=1284KiB/s (1315kB/s)(6032KiB/4697msec) clat (usec): min=667, max=53567, avg=3031.89, stdev=4202.96 lat (usec): min=668, max=53569, avg=3032.97, stdev=4202.99 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 734], 10.00th=[ 775], 20.00th=[ 824], | 30.00th=[ 865], 40.00th=[ 938], 50.00th=[ 1237], 60.00th=[ 2671], | 70.00th=[ 3490], 80.00th=[ 4555], 90.00th=[ 6652], 95.00th=[ 8356], | 99.00th=[16581], 99.50th=[33424], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 864, max= 1696, per=24.03%, avg=1244.44, stdev=271.74, samples=9 iops : min= 54, max= 106, avg=77.78, stdev=16.98, samples=9 lat (usec) : 100=8.98%, 250=1.04%, 500=0.13%, 750=3.26%, 1000=17.97% lat (msec) : 2=5.34%, 4=10.68%, 10=25.52%, 20=25.91%, 50=1.04% lat (msec) : 100=0.13% cpu : usr=0.17%, sys=21.29%, ctx=2138, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1194: Mon May 13 02:40:35 2024 read: IOPS=82, BW=1323KiB/s (1355kB/s)(6032KiB/4560msec) clat (usec): min=53, max=24120, avg=9062.20, stdev=5317.02 lat (usec): min=53, max=24120, avg=9065.24, stdev=5317.97 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 64], 10.00th=[ 75], 20.00th=[ 1893], | 30.00th=[ 8291], 40.00th=[ 9110], 50.00th=[ 9896], 60.00th=[10683], | 70.00th=[11469], 80.00th=[12780], 90.00th=[14746], 95.00th=[16712], | 99.00th=[22152], 99.50th=[23725], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 992, max= 1504, per=26.44%, avg=1301.33, stdev=173.07, samples=9 iops : min= 62, max= 94, avg=81.33, stdev=10.82, samples=9 write: IOPS=85, BW=1372KiB/s (1405kB/s)(6256KiB/4560msec) clat (usec): min=682, max=64906, avg=2907.96, stdev=4280.88 lat (usec): min=682, max=64908, avg=2908.95, stdev=4280.95 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 750], 10.00th=[ 799], 20.00th=[ 848], | 30.00th=[ 898], 40.00th=[ 979], 50.00th=[ 1254], 60.00th=[ 2540], | 70.00th=[ 3195], 80.00th=[ 4359], 90.00th=[ 5932], 95.00th=[ 8160], | 99.00th=[17171], 99.50th=[19268], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 1056, max= 1664, per=26.23%, avg=1358.22, stdev=197.33, samples=9 iops : min= 66, max= 104, avg=84.89, stdev=12.33, samples=9 lat (usec) : 100=8.33%, 250=1.17%, 500=0.13%, 750=2.47%, 1000=19.01% lat (msec) : 2=5.99%, 4=12.50%, 10=25.26%, 20=24.22%, 50=0.78% lat (msec) : 100=0.13% cpu : usr=0.22%, sys=22.37%, ctx=2066, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4922KiB/s (5040kB/s), 1217KiB/s-1332KiB/s (1246kB/s-1364kB/s), io=23.4MiB (24.5MB), run=4560-4866msec WRITE: bw=5179KiB/s (5303kB/s), 1284KiB/s-1413KiB/s (1315kB/s-1447kB/s), io=24.6MiB (25.8MB), run=4560-4866msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1189: Mon May 13 02:40:41 2024 read: IOPS=35, BW=573KiB/s (586kB/s)(5920KiB/10339msec) clat (usec): min=1868, max=47050, avg=4522.27, stdev=5034.59 lat (usec): min=1869, max=47052, avg=4522.98, stdev=5034.58 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2073], 10.00th=[ 2114], 20.00th=[ 2212], | 30.00th=[ 2311], 40.00th=[ 2409], 50.00th=[ 2540], 60.00th=[ 2802], | 70.00th=[ 3359], 80.00th=[ 5080], 90.00th=[10028], 95.00th=[13960], | 99.00th=[23725], 99.50th=[40633], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 352, max= 960, per=24.16%, avg=558.40, stdev=166.27, samples=20 iops : min= 22, max= 60, avg=34.90, stdev=10.39, samples=20 write: IOPS=38, BW=616KiB/s (631kB/s)(6368KiB/10339msec) clat (usec): min=12317, max=47908, avg=21755.40, stdev=6585.55 lat (usec): min=12319, max=47909, avg=21756.73, stdev=6585.58 clat percentiles (usec): | 1.00th=[13042], 5.00th=[13566], 10.00th=[14091], 20.00th=[15401], | 30.00th=[17433], 40.00th=[19792], 50.00th=[21103], 60.00th=[22414], | 70.00th=[23987], 80.00th=[26084], 90.00th=[31065], 95.00th=[34866], | 99.00th=[42206], 99.50th=[42730], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 384, max= 800, per=25.08%, avg=609.60, stdev=156.59, samples=20 iops : min= 24, max= 50, avg=38.10, stdev= 9.79, samples=20 lat (msec) : 2=0.78%, 4=35.42%, 10=7.03%, 20=26.43%, 50=30.34% cpu : usr=0.06%, sys=5.02%, ctx=880, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1191: Mon May 13 02:40:41 2024 read: IOPS=34, BW=554KiB/s (568kB/s)(5744KiB/10363msec) clat (usec): min=1734, max=19950, avg=3601.75, stdev=2885.60 lat (usec): min=1734, max=19952, avg=3602.44, stdev=2885.63 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2057], 10.00th=[ 2114], 20.00th=[ 2180], | 30.00th=[ 2278], 40.00th=[ 2376], 50.00th=[ 2507], 60.00th=[ 2737], | 70.00th=[ 3032], 80.00th=[ 3752], 90.00th=[ 6390], 95.00th=[10421], | 99.00th=[15533], 99.50th=[19792], 99.90th=[20055], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 224, max= 926, per=23.86%, avg=551.50, stdev=210.57, samples=20 iops : min= 14, max= 57, avg=34.25, stdev=12.99, samples=20 write: IOPS=39, BW=631KiB/s (647kB/s)(6544KiB/10363msec) clat (usec): min=11641, max=99558, avg=22153.05, stdev=7359.44 lat (usec): min=11643, max=99560, avg=22154.46, stdev=7359.58 clat percentiles (usec): | 1.00th=[12256], 5.00th=[13435], 10.00th=[14091], 20.00th=[16057], | 30.00th=[18220], 40.00th=[20317], 50.00th=[21365], 60.00th=[22676], | 70.00th=[23987], 80.00th=[26608], 90.00th=[30802], 95.00th=[33817], | 99.00th=[41157], 99.50th=[43254], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 448, max= 800, per=25.58%, avg=621.95, stdev=120.85, samples=20 iops : min= 28, max= 50, avg=38.65, stdev= 7.60, samples=20 lat (msec) : 2=1.04%, 4=37.11%, 10=6.25%, 20=23.05%, 50=32.42% lat (msec) : 100=0.13% cpu : usr=0.08%, sys=4.90%, ctx=870, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1193: Mon May 13 02:40:41 2024 read: IOPS=38, BW=620KiB/s (635kB/s)(6256KiB/10086msec) clat (usec): min=1874, max=33130, avg=4085.75, stdev=3709.84 lat (usec): min=1875, max=33131, avg=4086.50, stdev=3709.83 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2024], 10.00th=[ 2114], 20.00th=[ 2180], | 30.00th=[ 2278], 40.00th=[ 2376], 50.00th=[ 2507], 60.00th=[ 2704], | 70.00th=[ 3195], 80.00th=[ 5342], 90.00th=[ 8455], 95.00th=[12256], | 99.00th=[20579], 99.50th=[24511], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 288, max= 1280, per=27.00%, avg=624.00, stdev=250.36, samples=20 iops : min= 18, max= 80, avg=39.00, stdev=15.65, samples=20 write: IOPS=37, BW=598KiB/s (612kB/s)(6032KiB/10086msec) clat (usec): min=11294, max=65843, avg=22492.15, stdev=7271.25 lat (usec): min=11295, max=65845, avg=22493.52, stdev=7271.45 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13566], 10.00th=[14353], 20.00th=[15795], | 30.00th=[18220], 40.00th=[20579], 50.00th=[21890], 60.00th=[23200], | 70.00th=[24511], 80.00th=[26608], 90.00th=[31851], 95.00th=[35914], | 99.00th=[46924], 99.50th=[47973], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 384, max= 800, per=24.55%, avg=596.80, stdev=137.45, samples=20 iops : min= 24, max= 50, avg=37.30, stdev= 8.59, samples=20 lat (msec) : 2=1.56%, 4=36.07%, 10=9.64%, 20=21.22%, 50=31.38% lat (msec) : 100=0.13% cpu : usr=0.12%, sys=5.06%, ctx=889, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1195: Mon May 13 02:40:41 2024 read: IOPS=36, BW=587KiB/s (601kB/s)(6032KiB/10274msec) clat (usec): min=1876, max=28776, avg=3831.73, stdev=3264.55 lat (usec): min=1876, max=28777, avg=3832.36, stdev=3264.55 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2040], 10.00th=[ 2114], 20.00th=[ 2212], | 30.00th=[ 2278], 40.00th=[ 2409], 50.00th=[ 2540], 60.00th=[ 2835], | 70.00th=[ 3261], 80.00th=[ 4686], 90.00th=[ 7046], 95.00th=[ 9765], | 99.00th=[18220], 99.50th=[20317], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 288, max= 1088, per=24.99%, avg=577.60, stdev=196.84, samples=20 iops : min= 18, max= 68, avg=36.10, stdev=12.30, samples=20 write: IOPS=38, BW=609KiB/s (624kB/s)(6256KiB/10274msec) clat (usec): min=12661, max=72781, avg=22564.79, stdev=7585.25 lat (usec): min=12662, max=72783, avg=22566.14, stdev=7585.24 clat percentiles (usec): | 1.00th=[12911], 5.00th=[13829], 10.00th=[14615], 20.00th=[16057], | 30.00th=[18220], 40.00th=[20317], 50.00th=[21890], 60.00th=[22938], | 70.00th=[24511], 80.00th=[26346], 90.00th=[30802], 95.00th=[35390], | 99.00th=[51119], 99.50th=[57934], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 384, max= 800, per=24.94%, avg=606.40, stdev=139.86, samples=20 iops : min= 24, max= 50, avg=37.90, stdev= 8.74, samples=20 lat (msec) : 2=0.91%, 4=37.11%, 10=8.85%, 20=21.61%, 50=30.99% lat (msec) : 100=0.52% cpu : usr=0.11%, sys=4.91%, ctx=888, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2311KiB/s (2367kB/s), 554KiB/s-620KiB/s (568kB/s-635kB/s), io=23.4MiB (24.5MB), run=10086-10363msec WRITE: bw=2432KiB/s (2490kB/s), 598KiB/s-631KiB/s (612kB/s-647kB/s), io=24.6MiB (25.8MB), run=10086-10363msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1218: Mon May 13 02:40:47 2024 read: IOPS=2, BW=3022KiB/s (3094kB/s)(7168KiB/2372msec) clat (usec): min=500, max=860284, avg=305529.75, stdev=352382.45 lat (usec): min=502, max=860285, avg=305531.04, stdev=352382.27 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 502], 10.00th=[ 502], 20.00th=[ 32637], | 30.00th=[127402], 40.00th=[127402], 50.00th=[177210], 60.00th=[181404], | 70.00th=[181404], 80.00th=[759170], 90.00th=[859833], 95.00th=[859833], | 99.00th=[859833], 99.50th=[859833], 99.90th=[859833], 99.95th=[859833], | 99.99th=[859833] bw ( KiB/s): min= 2048, max= 4096, per=77.13%, avg=3407.67, stdev=1177.54, samples=3 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=3 write: IOPS=2, BW=2159KiB/s (2210kB/s)(5120KiB/2372msec) clat (usec): min=23555, max=81542, avg=45878.20, stdev=27066.39 lat (usec): min=23616, max=81613, avg=45936.17, stdev=27073.62 clat percentiles (usec): | 1.00th=[23462], 5.00th=[23462], 10.00th=[23462], 20.00th=[23462], | 30.00th=[24511], 40.00th=[24511], 50.00th=[31589], 60.00th=[31589], | 70.00th=[68682], 80.00th=[68682], 90.00th=[81265], 95.00th=[81265], | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 2048, max= 2048, per=33.11%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=33.33%, 100=16.67%, 250=25.00%, 1000=16.67% cpu : usr=0.00%, sys=26.66%, ctx=786, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1220: Mon May 13 02:40:47 2024 read: IOPS=0, BW=442KiB/s (452kB/s)(2048KiB/4635msec) clat (msec): min=52, max=828, avg=440.60, stdev=549.17 lat (msec): min=52, max=828, avg=440.60, stdev=549.17 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 53], 50.00th=[ 53], 60.00th=[ 827], | 70.00th=[ 827], 80.00th=[ 827], 90.00th=[ 827], 95.00th=[ 827], | 99.00th=[ 827], 99.50th=[ 827], 99.90th=[ 827], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 2048, max= 2048, per=46.36%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2209KiB/s (2262kB/s)(10.0MiB/4635msec) clat (msec): min=18, max=1439, avg=375.14, stdev=540.99 lat (msec): min=18, max=1439, avg=375.19, stdev=540.99 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 26], | 30.00th=[ 42], 40.00th=[ 48], 50.00th=[ 61], 60.00th=[ 82], | 70.00th=[ 844], 80.00th=[ 844], 90.00th=[ 1116], 95.00th=[ 1435], | 99.00th=[ 1435], 99.50th=[ 1435], 99.90th=[ 1435], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 1762, max= 6144, per=69.85%, avg=4320.00, stdev=1900.73, samples=4 iops : min= 1, max= 6, avg= 4.00, stdev= 2.16, samples=4 lat (msec) : 20=8.33%, 50=25.00%, 100=33.33%, 1000=16.67% cpu : usr=0.00%, sys=8.87%, ctx=356, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1221: Mon May 13 02:40:47 2024 read: IOPS=1, BW=1882KiB/s (1927kB/s)(6144KiB/3265msec) clat (usec): min=429, max=973089, avg=182167.72, stdev=388536.61 lat (usec): min=430, max=973090, avg=182168.95, stdev=388536.47 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 429], 10.00th=[ 429], 20.00th=[ 947], | 30.00th=[ 947], 40.00th=[ 955], 50.00th=[ 955], 60.00th=[ 51643], | 70.00th=[ 65799], 80.00th=[ 65799], 90.00th=[977273], 95.00th=[977273], | 99.00th=[977273], 99.50th=[977273], 99.90th=[977273], 99.95th=[977273], | 99.99th=[977273] bw ( KiB/s): min= 4096, max= 6144, per=100.00%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=1, BW=1882KiB/s (1927kB/s)(6144KiB/3265msec) clat (msec): min=30, max=1030, avg=361.12, stdev=410.14 lat (msec): min=30, max=1031, avg=361.17, stdev=410.15 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 131], 50.00th=[ 131], 60.00th=[ 142], | 70.00th=[ 709], 80.00th=[ 709], 90.00th=[ 1028], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 2048, max= 4096, per=49.67%, avg=3072.00, stdev=1182.41, samples=4 iops : min= 2, max= 4, avg= 3.00, stdev= 1.15, samples=4 lat (usec) : 500=8.33%, 1000=16.67% lat (msec) : 50=8.33%, 100=16.67%, 250=25.00%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=10.32%, ctx=394, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1223: Mon May 13 02:40:47 2024 read: IOPS=1, BW=1187KiB/s (1216kB/s)(5120KiB/4312msec) clat (usec): min=371, max=931470, avg=346534.91, stdev=462797.88 lat (usec): min=371, max=931472, avg=346536.09, stdev=462798.25 clat percentiles (usec): | 1.00th=[ 371], 5.00th=[ 371], 10.00th=[ 371], 20.00th=[ 371], | 30.00th=[ 537], 40.00th=[ 537], 50.00th=[ 33162], 60.00th=[ 33162], | 70.00th=[767558], 80.00th=[767558], 90.00th=[935330], 95.00th=[935330], | 99.00th=[935330], 99.50th=[935330], 99.90th=[935330], 99.95th=[935330], | 99.99th=[935330] bw ( KiB/s): min= 2048, max= 6131, per=92.56%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=1, BW=1662KiB/s (1702kB/s)(7168KiB/4312msec) clat (msec): min=23, max=1177, avg=367.02, stdev=509.13 lat (msec): min=23, max=1177, avg=367.07, stdev=509.12 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 25], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 131], 60.00th=[ 144], | 70.00th=[ 144], 80.00th=[ 1036], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 2043, max= 6144, per=57.89%, avg=3580.50, stdev=1961.34, samples=4 iops : min= 1, max= 6, avg= 3.00, stdev= 2.16, samples=4 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=33.33%, 250=16.67%, 1000=16.67% cpu : usr=0.02%, sys=12.53%, ctx=604, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4419KiB/s (4525kB/s), 442KiB/s-3022KiB/s (452kB/s-3094kB/s), io=20.0MiB (20.0MB), run=2372-4635msec WRITE: bw=6186KiB/s (6334kB/s), 1662KiB/s-2209KiB/s (1702kB/s-2262kB/s), io=28.0MiB (29.4MB), run=2372-4635msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1217: Mon May 13 02:40:47 2024 read: IOPS=2, BW=2084KiB/s (2134kB/s)(7168KiB/3439msec) clat (msec): min=29, max=864, avg=283.14, stdev=358.76 lat (msec): min=29, max=864, avg=283.14, stdev=358.76 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 31], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 112], 60.00th=[ 169], | 70.00th=[ 169], 80.00th=[ 735], 90.00th=[ 869], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 2048, max= 4096, per=78.63%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 write: IOPS=1, BW=1489KiB/s (1525kB/s)(5120KiB/3439msec) clat (msec): min=51, max=1154, avg=290.56, stdev=483.53 lat (msec): min=51, max=1154, avg=290.62, stdev=483.54 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 52], | 30.00th=[ 64], 40.00th=[ 64], 50.00th=[ 67], 60.00th=[ 67], | 70.00th=[ 117], 80.00th=[ 117], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 2048, max= 4096, per=50.54%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (msec) : 50=25.00%, 100=25.00%, 250=25.00%, 750=8.33%, 1000=8.33% cpu : usr=0.06%, sys=4.60%, ctx=55, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1219: Mon May 13 02:40:47 2024 read: IOPS=0, BW=434KiB/s (445kB/s)(2048KiB/4717msec) clat (msec): min=59, max=928, avg=493.77, stdev=614.86 lat (msec): min=59, max=928, avg=493.77, stdev=614.86 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 59], | 30.00th=[ 59], 40.00th=[ 59], 50.00th=[ 59], 60.00th=[ 927], | 70.00th=[ 927], 80.00th=[ 927], 90.00th=[ 927], 95.00th=[ 927], | 99.00th=[ 927], 99.50th=[ 927], 99.90th=[ 927], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 2048, max= 2048, per=47.18%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2171KiB/s (2223kB/s)(10.0MiB/4717msec) clat (msec): min=34, max=1172, avg=372.79, stdev=514.90 lat (msec): min=34, max=1172, avg=372.86, stdev=514.89 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 36], | 30.00th=[ 58], 40.00th=[ 59], 50.00th=[ 63], 60.00th=[ 66], | 70.00th=[ 1011], 80.00th=[ 1011], 90.00th=[ 1167], 95.00th=[ 1167], | 99.00th=[ 1167], 99.50th=[ 1167], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 1793, max= 6144, per=57.92%, avg=3520.25, stdev=2030.31, samples=4 iops : min= 1, max= 6, avg= 3.25, stdev= 2.22, samples=4 lat (msec) : 50=16.67%, 100=50.00%, 1000=8.33% cpu : usr=0.04%, sys=3.63%, ctx=42, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1222: Mon May 13 02:40:47 2024 read: IOPS=1, BW=1356KiB/s (1389kB/s)(6144KiB/4530msec) clat (msec): min=16, max=1917, avg=405.69, stdev=748.59 lat (msec): min=16, max=1917, avg=405.69, stdev=748.59 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 124], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 1921], 95.00th=[ 1921], | 99.00th=[ 1921], 99.50th=[ 1921], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 2031, max= 4096, per=78.43%, avg=3404.67, stdev=1189.64, samples=3 iops : min= 1, max= 4, avg= 2.67, stdev= 1.53, samples=3 write: IOPS=1, BW=1356KiB/s (1389kB/s)(6144KiB/4530msec) clat (msec): min=44, max=980, avg=349.10, stdev=399.31 lat (msec): min=44, max=980, avg=349.15, stdev=399.32 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 49], | 30.00th=[ 49], 40.00th=[ 58], 50.00th=[ 58], 60.00th=[ 262], | 70.00th=[ 701], 80.00th=[ 701], 90.00th=[ 978], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 2031, max= 4096, per=50.44%, avg=3065.50, stdev=1184.75, samples=4 iops : min= 1, max= 4, avg= 2.50, stdev= 1.29, samples=4 lat (msec) : 20=8.33%, 50=33.33%, 100=8.33%, 250=8.33%, 500=16.67% lat (msec) : 750=8.33%, 1000=8.33% cpu : usr=0.02%, sys=2.63%, ctx=41, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1224: Mon May 13 02:40:47 2024 read: IOPS=1, BW=1139KiB/s (1166kB/s)(5120KiB/4497msec) clat (msec): min=18, max=196, avg=74.63, stdev=74.97 lat (msec): min=18, max=196, avg=74.63, stdev=74.97 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 22], 40.00th=[ 22], 50.00th=[ 42], 60.00th=[ 42], | 70.00th=[ 95], 80.00th=[ 95], 90.00th=[ 197], 95.00th=[ 197], | 99.00th=[ 197], 99.50th=[ 197], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1594KiB/s (1632kB/s)(7168KiB/4497msec) clat (msec): min=38, max=1901, avg=588.58, stdev=701.51 lat (msec): min=38, max=1901, avg=588.65, stdev=701.51 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 48], | 30.00th=[ 56], 40.00th=[ 56], 50.00th=[ 245], 60.00th=[ 827], | 70.00th=[ 827], 80.00th=[ 1011], 90.00th=[ 1905], 95.00th=[ 1905], | 99.00th=[ 1905], 99.50th=[ 1905], 99.90th=[ 1905], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 2048, max= 8192, per=67.39%, avg=4096.00, stdev=3547.24, samples=3 iops : min= 2, max= 8, avg= 4.00, stdev= 3.46, samples=3 lat (msec) : 20=8.33%, 50=33.33%, 100=16.67%, 250=16.67%, 1000=8.33% cpu : usr=0.02%, sys=2.42%, ctx=52, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4342KiB/s (4446kB/s), 434KiB/s-2084KiB/s (445kB/s-2134kB/s), io=20.0MiB (20.0MB), run=3439-4717msec WRITE: bw=6078KiB/s (6224kB/s), 1356KiB/s-2171KiB/s (1389kB/s-2223kB/s), io=28.0MiB (29.4MB), run=3439-4717msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1237: Mon May 13 02:40:51 2024 read: IOPS=0, BW=3613KiB/s (3700kB/s)(12.0MiB/3401msec) clat (msec): min=150, max=2423, avg=1133.24, stdev=1167.67 lat (msec): min=150, max=2423, avg=1133.24, stdev=1167.67 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 150], 10.00th=[ 150], 20.00th=[ 150], | 30.00th=[ 150], 40.00th=[ 827], 50.00th=[ 827], 60.00th=[ 827], | 70.00th=[ 2433], 80.00th=[ 2433], 90.00th=[ 2433], 95.00th=[ 2433], | 99.00th=[ 2433], 99.50th=[ 2433], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.03%, sys=39.00%, ctx=1367, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1238: Mon May 13 02:40:51 2024 read: IOPS=0, BW=1227KiB/s (1256kB/s)(4096KiB/3339msec) clat (nsec): min=133028k, max=133028k, avg=133027602.00, stdev= 0.00 lat (nsec): min=133029k, max=133029k, avg=133028917.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 133], | 30.00th=[ 133], 40.00th=[ 133], 50.00th=[ 133], 60.00th=[ 133], | 70.00th=[ 133], 80.00th=[ 133], 90.00th=[ 133], 95.00th=[ 133], | 99.00th=[ 133], 99.50th=[ 133], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2453KiB/s (2512kB/s)(8192KiB/3339msec) clat (msec): min=208, max=2994, avg=1601.65, stdev=1969.90 lat (msec): min=209, max=2994, avg=1601.98, stdev=1969.91 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 209], 10.00th=[ 209], 20.00th=[ 209], | 30.00th=[ 209], 40.00th=[ 209], 50.00th=[ 209], 60.00th=[ 3004], | 70.00th=[ 3004], 80.00th=[ 3004], 90.00th=[ 3004], 95.00th=[ 3004], | 99.00th=[ 3004], 99.50th=[ 3004], 99.90th=[ 3004], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 8192, max= 8192, per=88.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=6.35%, ctx=133, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1239: Mon May 13 02:40:51 2024 write: IOPS=0, BW=3485KiB/s (3569kB/s)(12.0MiB/3526msec) clat (msec): min=265, max=2990, avg=1174.82, stdev=1572.06 lat (msec): min=265, max=2990, avg=1174.96, stdev=1572.15 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 266], 10.00th=[ 266], 20.00th=[ 266], | 30.00th=[ 266], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 3004], 80.00th=[ 3004], 90.00th=[ 3004], 95.00th=[ 3004], | 99.00th=[ 3004], 99.50th=[ 3004], 99.90th=[ 3004], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 8192, max= 8192, per=88.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=6.41%, ctx=62, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1240: Mon May 13 02:40:51 2024 write: IOPS=0, BW=3808KiB/s (3899kB/s)(12.0MiB/3227msec) clat (msec): min=114, max=2908, avg=1075.18, stdev=1588.68 lat (msec): min=115, max=2909, avg=1075.41, stdev=1588.73 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 115], 10.00th=[ 115], 20.00th=[ 115], | 30.00th=[ 115], 40.00th=[ 203], 50.00th=[ 203], 60.00th=[ 203], | 70.00th=[ 2903], 80.00th=[ 2903], 90.00th=[ 2903], 95.00th=[ 2903], | 99.00th=[ 2903], 99.50th=[ 2903], 99.90th=[ 2903], 99.95th=[ 2903], | 99.99th=[ 2903] bw ( KiB/s): min= 8192, max= 8192, per=88.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.03%, sys=7.63%, ctx=58, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4817KiB/s (4933kB/s), 1227KiB/s-3613KiB/s (1256kB/s-3700kB/s), io=16.0MiB (16.8MB), run=3339-3401msec WRITE: bw=9293KiB/s (9516kB/s), 2453KiB/s-3808KiB/s (2512kB/s-3899kB/s), io=32.0MiB (33.6MB), run=3227-3526msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1233: Mon May 13 02:40:51 2024 read: IOPS=0, BW=3581KiB/s (3667kB/s)(12.0MiB/3431msec) clat (msec): min=133, max=2471, avg=1143.14, stdev=1200.92 lat (msec): min=133, max=2471, avg=1143.14, stdev=1200.92 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 134], 10.00th=[ 134], 20.00th=[ 134], | 30.00th=[ 134], 40.00th=[ 827], 50.00th=[ 827], 60.00th=[ 827], | 70.00th=[ 2467], 80.00th=[ 2467], 90.00th=[ 2467], 95.00th=[ 2467], | 99.00th=[ 2467], 99.50th=[ 2467], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 5785, max= 5785, per=100.00%, avg=5785.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=2.10%, ctx=34, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1234: Mon May 13 02:40:51 2024 read: IOPS=0, BW=1270KiB/s (1301kB/s)(4096KiB/3225msec) clat (nsec): min=53030k, max=53030k, avg=53029846.00, stdev= 0.00 lat (nsec): min=53031k, max=53031k, avg=53031468.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[53216], 5.00th=[53216], 10.00th=[53216], 20.00th=[53216], | 30.00th=[53216], 40.00th=[53216], 50.00th=[53216], 60.00th=[53216], | 70.00th=[53216], 80.00th=[53216], 90.00th=[53216], 95.00th=[53216], | 99.00th=[53216], 99.50th=[53216], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 8110, max= 8110, per=100.00%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2540KiB/s (2601kB/s)(8192KiB/3225msec) clat (msec): min=262, max=2908, avg=1585.39, stdev=1871.51 lat (msec): min=262, max=2908, avg=1585.59, stdev=1871.45 clat percentiles (msec): | 1.00th=[ 262], 5.00th=[ 262], 10.00th=[ 262], 20.00th=[ 262], | 30.00th=[ 262], 40.00th=[ 262], 50.00th=[ 262], 60.00th=[ 2903], | 70.00th=[ 2903], 80.00th=[ 2903], 90.00th=[ 2903], 95.00th=[ 2903], | 99.00th=[ 2903], 99.50th=[ 2903], 99.90th=[ 2903], 99.95th=[ 2903], | 99.99th=[ 2903] bw ( KiB/s): min= 8110, max= 8110, per=90.02%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=2.92%, ctx=66, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1235: Mon May 13 02:40:51 2024 write: IOPS=0, BW=3379KiB/s (3460kB/s)(12.0MiB/3637msec) clat (msec): min=134, max=3059, avg=1209.43, stdev=1609.34 lat (msec): min=134, max=3059, avg=1209.69, stdev=1609.33 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 435], 50.00th=[ 435], 60.00th=[ 435], | 70.00th=[ 3071], 80.00th=[ 3071], 90.00th=[ 3071], 95.00th=[ 3071], | 99.00th=[ 3071], 99.50th=[ 3071], 99.90th=[ 3071], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min= 8175, max= 8175, per=90.74%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.03%, sys=3.19%, ctx=50, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1236: Mon May 13 02:40:51 2024 write: IOPS=6, BW=24.1MiB/s (25.3MB/s)(12.0MiB/498msec) clat (msec): min=79, max=319, avg=165.19, stdev=134.26 lat (msec): min=80, max=320, avg=165.77, stdev=133.98 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 80], 10.00th=[ 80], 20.00th=[ 80], | 30.00th=[ 80], 40.00th=[ 96], 50.00th=[ 96], 60.00th=[ 96], | 70.00th=[ 321], 80.00th=[ 321], 90.00th=[ 321], 95.00th=[ 321], | 99.00th=[ 321], 99.50th=[ 321], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] lat (msec) : 100=66.67%, 500=33.33% cpu : usr=0.00%, sys=13.28%, ctx=36, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4775KiB/s (4890kB/s), 1270KiB/s-3581KiB/s (1301kB/s-3667kB/s), io=16.0MiB (16.8MB), run=3225-3431msec WRITE: bw=9010KiB/s (9226kB/s), 2540KiB/s-24.1MiB/s (2601kB/s-25.3MB/s), io=32.0MiB (33.6MB), run=498-3637msec