== sanity test 398b: DIO and buffer IO race ============== 07:32:42 (1734438762) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.17111 s, 43.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23730: Tue Dec 17 07:32:58 2024 read: IOPS=111, BW=447KiB/s (458kB/s)(5992KiB/13410msec) clat (usec): min=48, max=174437, avg=5041.57, stdev=5287.86 lat (usec): min=49, max=174438, avg=5042.29, stdev=5287.80 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 105], 10.00th=[ 2311], 20.00th=[ 2835], | 30.00th=[ 3228], 40.00th=[ 3720], 50.00th=[ 4359], 60.00th=[ 5145], | 70.00th=[ 5997], 80.00th=[ 6980], 90.00th=[ 8455], 95.00th=[ 10159], | 99.00th=[ 15664], 99.50th=[ 17433], 99.90th=[ 24511], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 264, max= 600, per=24.74%, avg=440.12, stdev=77.85, samples=26 iops : min= 66, max= 150, avg=109.88, stdev=19.52, samples=26 write: IOPS=117, BW=470KiB/s (481kB/s)(6296KiB/13410msec) clat (usec): min=414, max=22155, avg=3679.27, stdev=3262.39 lat (usec): min=415, max=22155, avg=3680.04, stdev=3262.41 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 545], 10.00th=[ 586], 20.00th=[ 717], | 30.00th=[ 947], 40.00th=[ 1991], 50.00th=[ 2999], 60.00th=[ 3785], | 70.00th=[ 4752], 80.00th=[ 6063], 90.00th=[ 8586], 95.00th=[10290], | 99.00th=[13304], 99.50th=[14484], 99.90th=[21365], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 320, max= 672, per=24.71%, avg=460.38, stdev=72.34, samples=26 iops : min= 80, max= 168, avg=114.96, stdev=18.10, samples=26 lat (usec) : 50=0.03%, 100=2.31%, 250=0.85%, 500=0.46%, 750=10.74% lat (usec) : 1000=5.24% lat (msec) : 2=4.56%, 4=29.95%, 10=40.43%, 20=5.31%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=22.88%, ctx=5774, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23732: Tue Dec 17 07:32:58 2024 read: IOPS=111, BW=446KiB/s (456kB/s)(5940KiB/13326msec) clat (usec): min=56, max=22387, avg=4866.37, stdev=2754.02 lat (usec): min=57, max=22388, avg=4867.94, stdev=2754.71 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 119], 10.00th=[ 2311], 20.00th=[ 2802], | 30.00th=[ 3294], 40.00th=[ 3785], 50.00th=[ 4359], 60.00th=[ 5145], | 70.00th=[ 5800], 80.00th=[ 6783], 90.00th=[ 8160], 95.00th=[ 9503], | 99.00th=[14091], 99.50th=[16188], 99.90th=[20841], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 216, max= 576, per=24.78%, avg=440.81, stdev=71.10, samples=26 iops : min= 54, max= 144, avg=110.12, stdev=17.81, samples=26 write: IOPS=119, BW=476KiB/s (488kB/s)(6348KiB/13326msec) clat (usec): min=406, max=169753, avg=3796.02, stdev=5477.99 lat (usec): min=407, max=169754, avg=3797.16, stdev=5478.00 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 553], 10.00th=[ 586], 20.00th=[ 701], | 30.00th=[ 922], 40.00th=[ 1729], 50.00th=[ 2933], 60.00th=[ 3654], | 70.00th=[ 4686], 80.00th=[ 6128], 90.00th=[ 8356], 95.00th=[ 10552], | 99.00th=[ 15139], 99.50th=[ 17695], 99.90th=[ 46400], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 264, max= 608, per=24.91%, avg=464.15, stdev=70.25, samples=26 iops : min= 66, max= 152, avg=115.96, stdev=17.56, samples=26 lat (usec) : 100=2.18%, 250=0.52%, 500=0.23%, 750=11.56%, 1000=5.11% lat (msec) : 2=4.98%, 4=29.52%, 10=40.59%, 20=5.08%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=23.19%, ctx=5872, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23734: Tue Dec 17 07:32:58 2024 read: IOPS=112, BW=449KiB/s (460kB/s)(6028KiB/13429msec) clat (usec): min=48, max=202070, avg=5013.76, stdev=5807.45 lat (usec): min=48, max=202071, avg=5014.78, stdev=5807.54 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 116], 10.00th=[ 2311], 20.00th=[ 2802], | 30.00th=[ 3261], 40.00th=[ 3752], 50.00th=[ 4359], 60.00th=[ 5080], | 70.00th=[ 5866], 80.00th=[ 6915], 90.00th=[ 8291], 95.00th=[ 9765], | 99.00th=[ 14353], 99.50th=[ 16188], 99.90th=[ 24249], 99.95th=[202376], | 99.99th=[202376] bw ( KiB/s): min= 256, max= 608, per=24.93%, avg=443.54, stdev=76.39, samples=26 iops : min= 64, max= 152, avg=110.81, stdev=19.10, samples=26 write: IOPS=116, BW=466KiB/s (477kB/s)(6260KiB/13429msec) clat (usec): min=470, max=24406, avg=3708.48, stdev=3289.46 lat (usec): min=470, max=24406, avg=3709.33, stdev=3289.40 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 545], 10.00th=[ 586], 20.00th=[ 701], | 30.00th=[ 922], 40.00th=[ 2089], 50.00th=[ 3032], 60.00th=[ 3687], | 70.00th=[ 4817], 80.00th=[ 6194], 90.00th=[ 8291], 95.00th=[ 9896], | 99.00th=[13960], 99.50th=[16188], 99.90th=[19792], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 320, max= 624, per=24.62%, avg=458.65, stdev=87.00, samples=26 iops : min= 80, max= 156, avg=114.58, stdev=21.82, samples=26 lat (usec) : 50=0.13%, 100=1.95%, 250=0.68%, 500=0.36%, 750=11.20% lat (usec) : 1000=4.72% lat (msec) : 2=4.23%, 4=30.89%, 10=41.11%, 20=4.62%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=23.14%, ctx=5885, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23736: Tue Dec 17 07:32:58 2024 read: IOPS=112, BW=449KiB/s (459kB/s)(6052KiB/13490msec) clat (usec): min=49, max=171189, avg=5074.98, stdev=5184.47 lat (usec): min=49, max=171190, avg=5076.49, stdev=5184.46 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 922], 10.00th=[ 2409], 20.00th=[ 2835], | 30.00th=[ 3261], 40.00th=[ 3752], 50.00th=[ 4359], 60.00th=[ 5080], | 70.00th=[ 5932], 80.00th=[ 6849], 90.00th=[ 8586], 95.00th=[ 10290], | 99.00th=[ 14877], 99.50th=[ 18220], 99.90th=[ 28443], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 280, max= 568, per=24.76%, avg=440.50, stdev=76.93, samples=26 iops : min= 70, max= 142, avg=110.04, stdev=19.28, samples=26 write: IOPS=115, BW=462KiB/s (473kB/s)(6236KiB/13490msec) clat (usec): min=472, max=20621, avg=3684.16, stdev=3320.55 lat (usec): min=473, max=20621, avg=3685.58, stdev=3320.35 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 545], 10.00th=[ 586], 20.00th=[ 709], | 30.00th=[ 889], 40.00th=[ 2024], 50.00th=[ 3032], 60.00th=[ 3752], | 70.00th=[ 4621], 80.00th=[ 5997], 90.00th=[ 8455], 95.00th=[10290], | 99.00th=[14484], 99.50th=[15926], 99.90th=[18220], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 344, max= 728, per=24.47%, avg=455.88, stdev=79.81, samples=26 iops : min= 86, max= 182, avg=113.88, stdev=19.98, samples=26 lat (usec) : 50=0.03%, 100=1.79%, 250=0.55%, 500=0.39%, 750=10.77% lat (usec) : 1000=5.86% lat (msec) : 2=3.55%, 4=31.05%, 10=40.49%, 20=5.34%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=22.84%, ctx=5937, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1780KiB/s (1823kB/s), 446KiB/s-449KiB/s (456kB/s-460kB/s), io=23.4MiB (24.6MB), run=13326-13490msec WRITE: bw=1864KiB/s (1908kB/s), 462KiB/s-476KiB/s (473kB/s-488kB/s), io=24.6MiB (25.7MB), run=13326-13490msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23731: Tue Dec 17 07:33:02 2024 read: IOPS=89, BW=356KiB/s (365kB/s)(5992KiB/16813msec) clat (usec): min=1640, max=174963, avg=4524.85, stdev=5612.54 lat (usec): min=1641, max=174964, avg=4525.58, stdev=5612.57 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1893], 10.00th=[ 1975], 20.00th=[ 2147], | 30.00th=[ 2311], 40.00th=[ 2573], 50.00th=[ 2966], 60.00th=[ 3523], | 70.00th=[ 4424], 80.00th=[ 5932], 90.00th=[ 9634], 95.00th=[ 12125], | 99.00th=[ 16909], 99.50th=[ 20579], 99.90th=[ 24773], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 152, max= 792, per=24.78%, avg=350.12, stdev=170.36, samples=33 iops : min= 38, max= 198, avg=87.42, stdev=42.61, samples=33 write: IOPS=93, BW=374KiB/s (383kB/s)(6296KiB/16813msec) clat (usec): min=2313, max=27832, avg=6335.01, stdev=4024.46 lat (usec): min=2314, max=27832, avg=6336.01, stdev=4024.39 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2868], 10.00th=[ 3064], 20.00th=[ 3326], | 30.00th=[ 3589], 40.00th=[ 3982], 50.00th=[ 4686], 60.00th=[ 5735], | 70.00th=[ 7308], 80.00th=[ 9110], 90.00th=[11731], 95.00th=[14353], | 99.00th=[21627], 99.50th=[23987], 99.90th=[26870], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 160, max= 758, per=24.92%, avg=368.55, stdev=174.39, samples=33 iops : min= 40, max= 189, avg=92.03, stdev=43.62, samples=33 lat (msec) : 2=5.89%, 4=46.87%, 10=34.64%, 20=11.65%, 50=0.91% lat (msec) : 250=0.03% cpu : usr=0.12%, sys=11.40%, ctx=3579, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23733: Tue Dec 17 07:33:02 2024 read: IOPS=88, BW=353KiB/s (361kB/s)(5940KiB/16838msec) clat (usec): min=1596, max=167017, avg=4521.80, stdev=5473.01 lat (usec): min=1597, max=167018, avg=4522.52, stdev=5473.04 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1893], 10.00th=[ 1975], 20.00th=[ 2147], | 30.00th=[ 2311], 40.00th=[ 2573], 50.00th=[ 2966], 60.00th=[ 3556], | 70.00th=[ 4293], 80.00th=[ 5932], 90.00th=[ 9241], 95.00th=[ 12387], | 99.00th=[ 17695], 99.50th=[ 19792], 99.90th=[ 24773], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 112, max= 760, per=24.67%, avg=348.55, stdev=178.91, samples=33 iops : min= 28, max= 190, avg=87.09, stdev=44.74, samples=33 write: IOPS=94, BW=377KiB/s (386kB/s)(6348KiB/16838msec) clat (usec): min=2313, max=55875, avg=6332.68, stdev=4099.02 lat (usec): min=2314, max=55877, avg=6333.55, stdev=4099.01 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2835], 10.00th=[ 3097], 20.00th=[ 3359], | 30.00th=[ 3621], 40.00th=[ 4015], 50.00th=[ 4686], 60.00th=[ 5669], | 70.00th=[ 7504], 80.00th=[ 9241], 90.00th=[11731], 95.00th=[14091], | 99.00th=[20841], 99.50th=[23987], 99.90th=[31327], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 168, max= 768, per=24.93%, avg=368.67, stdev=168.30, samples=33 iops : min= 42, max= 192, avg=92.12, stdev=42.11, samples=33 lat (msec) : 2=5.89%, 4=46.39%, 10=35.22%, 20=11.75%, 50=0.68% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.20%, sys=11.36%, ctx=3590, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23735: Tue Dec 17 07:33:02 2024 read: IOPS=88, BW=355KiB/s (363kB/s)(6028KiB/16993msec) clat (usec): min=1396, max=25959, avg=4499.67, stdev=3609.54 lat (usec): min=1396, max=25960, avg=4500.38, stdev=3609.53 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1876], 10.00th=[ 1975], 20.00th=[ 2114], | 30.00th=[ 2278], 40.00th=[ 2540], 50.00th=[ 2933], 60.00th=[ 3523], | 70.00th=[ 4359], 80.00th=[ 6390], 90.00th=[10028], 95.00th=[12387], | 99.00th=[17171], 99.50th=[19792], 99.90th=[23725], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 144, max= 784, per=24.42%, avg=345.09, stdev=173.13, samples=33 iops : min= 36, max= 196, avg=86.21, stdev=43.28, samples=33 write: IOPS=92, BW=368KiB/s (377kB/s)(6260KiB/16993msec) clat (msec): min=2, max=166, avg= 6.48, stdev= 5.78 lat (msec): min=2, max=166, avg= 6.48, stdev= 5.78 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 22], 99.50th=[ 26], 99.90th=[ 35], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 176, max= 792, per=24.10%, avg=356.48, stdev=180.22, samples=33 iops : min= 44, max= 198, avg=89.06, stdev=45.05, samples=33 lat (msec) : 2=5.63%, 4=47.82%, 10=32.58%, 20=13.18%, 50=0.75% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=11.28%, ctx=3593, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23737: Tue Dec 17 07:33:02 2024 read: IOPS=89, BW=359KiB/s (368kB/s)(6052KiB/16862msec) clat (usec): min=1606, max=25773, avg=4400.30, stdev=3419.40 lat (usec): min=1607, max=25773, avg=4400.96, stdev=3419.41 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1876], 10.00th=[ 1975], 20.00th=[ 2114], | 30.00th=[ 2311], 40.00th=[ 2606], 50.00th=[ 2999], 60.00th=[ 3523], | 70.00th=[ 4293], 80.00th=[ 6194], 90.00th=[ 9503], 95.00th=[11863], | 99.00th=[16712], 99.50th=[19268], 99.90th=[24249], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 152, max= 760, per=24.89%, avg=351.73, stdev=168.76, samples=33 iops : min= 38, max= 190, avg=87.91, stdev=42.20, samples=33 write: IOPS=92, BW=370KiB/s (379kB/s)(6236KiB/16862msec) clat (msec): min=2, max=179, avg= 6.51, stdev= 6.01 lat (msec): min=2, max=179, avg= 6.51, stdev= 6.01 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 15], | 99.00th=[ 23], 99.50th=[ 26], 99.90th=[ 32], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 184, max= 792, per=24.55%, avg=363.12, stdev=172.51, samples=33 iops : min= 46, max= 198, avg=90.76, stdev=43.14, samples=33 lat (msec) : 2=5.83%, 4=47.14%, 10=34.57%, 20=11.36%, 50=1.07% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=11.35%, ctx=3591, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1413KiB/s (1447kB/s), 353KiB/s-359KiB/s (361kB/s-368kB/s), io=23.4MiB (24.6MB), run=16813-16993msec WRITE: bw=1479KiB/s (1515kB/s), 368KiB/s-377KiB/s (377kB/s-386kB/s), io=24.6MiB (25.7MB), run=16813-16993msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23763: Tue Dec 17 07:33:09 2024 read: IOPS=57, BW=913KiB/s (935kB/s)(5920KiB/6481msec) clat (usec): min=61, max=36477, avg=11266.29, stdev=3837.52 lat (usec): min=62, max=36478, avg=11268.89, stdev=3838.36 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 7308], 10.00th=[ 8160], 20.00th=[ 8979], | 30.00th=[ 9896], 40.00th=[10552], 50.00th=[11076], 60.00th=[11731], | 70.00th=[12518], 80.00th=[13173], 90.00th=[14484], 95.00th=[15926], | 99.00th=[23725], 99.50th=[34341], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 736, max= 1120, per=24.74%, avg=903.42, stdev=143.51, samples=12 iops : min= 46, max= 70, avg=56.25, stdev= 8.89, samples=12 write: IOPS=61, BW=983KiB/s (1006kB/s)(6368KiB/6481msec) clat (usec): min=778, max=96186, avg=5751.11, stdev=7240.14 lat (usec): min=779, max=96188, avg=5752.28, stdev=7240.22 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 848], 10.00th=[ 988], 20.00th=[ 1795], | 30.00th=[ 2507], 40.00th=[ 3130], 50.00th=[ 3785], 60.00th=[ 4752], | 70.00th=[ 5997], 80.00th=[ 7963], 90.00th=[12518], 95.00th=[15533], | 99.00th=[42206], 99.50th=[42730], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 702, max= 1280, per=25.45%, avg=978.00, stdev=189.20, samples=12 iops : min= 43, max= 80, avg=60.92, stdev=11.85, samples=12 lat (usec) : 100=1.43%, 250=0.13%, 1000=5.34% lat (msec) : 2=5.34%, 4=16.41%, 10=29.82%, 20=39.58%, 50=1.82% lat (msec) : 100=0.13% cpu : usr=0.19%, sys=19.21%, ctx=2756, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23764: Tue Dec 17 07:33:09 2024 read: IOPS=54, BW=876KiB/s (897kB/s)(5744KiB/6557msec) clat (usec): min=70, max=40275, avg=11625.73, stdev=3919.63 lat (usec): min=71, max=40276, avg=11626.40, stdev=3919.64 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 7570], 10.00th=[ 8356], 20.00th=[ 9372], | 30.00th=[10028], 40.00th=[10421], 50.00th=[11207], 60.00th=[11994], | 70.00th=[12911], 80.00th=[13829], 90.00th=[15139], 95.00th=[16319], | 99.00th=[28443], 99.50th=[31065], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 672, max= 1120, per=23.85%, avg=871.08, stdev=134.70, samples=13 iops : min= 42, max= 70, avg=54.31, stdev= 8.47, samples=13 write: IOPS=62, BW=998KiB/s (1022kB/s)(6544KiB/6557msec) clat (usec): min=724, max=98975, avg=5747.00, stdev=7050.94 lat (usec): min=725, max=98976, avg=5748.31, stdev=7051.24 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 824], 10.00th=[ 906], 20.00th=[ 1549], | 30.00th=[ 2409], 40.00th=[ 2900], 50.00th=[ 3654], 60.00th=[ 4883], | 70.00th=[ 6128], 80.00th=[ 7963], 90.00th=[13829], 95.00th=[17433], | 99.00th=[22938], 99.50th=[37487], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 766, max= 1216, per=25.81%, avg=991.69, stdev=160.33, samples=13 iops : min= 47, max= 76, avg=61.85, stdev=10.17, samples=13 lat (usec) : 100=0.65%, 250=0.13%, 750=0.13%, 1000=6.38% lat (msec) : 2=5.47%, 4=17.32%, 10=28.39%, 20=39.58%, 50=1.82% lat (msec) : 100=0.13% cpu : usr=0.14%, sys=19.13%, ctx=2813, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23765: Tue Dec 17 07:33:09 2024 read: IOPS=60, BW=961KiB/s (985kB/s)(6256KiB/6507msec) clat (usec): min=59, max=82742, avg=11519.15, stdev=5126.40 lat (usec): min=59, max=82743, avg=11521.64, stdev=5126.36 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 7504], 10.00th=[ 8455], 20.00th=[ 9241], | 30.00th=[ 9896], 40.00th=[10421], 50.00th=[11076], 60.00th=[11731], | 70.00th=[12256], 80.00th=[13304], 90.00th=[15270], 95.00th=[16909], | 99.00th=[24773], 99.50th=[36963], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 734, max= 1152, per=26.34%, avg=962.08, stdev=140.54, samples=13 iops : min= 45, max= 72, avg=60.00, stdev= 8.88, samples=13 write: IOPS=57, BW=927KiB/s (949kB/s)(6032KiB/6507msec) clat (usec): min=761, max=50870, avg=5244.24, stdev=5654.50 lat (usec): min=763, max=50872, avg=5245.41, stdev=5654.50 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 873], 10.00th=[ 955], 20.00th=[ 1909], | 30.00th=[ 2409], 40.00th=[ 2769], 50.00th=[ 3326], 60.00th=[ 4359], | 70.00th=[ 5276], 80.00th=[ 7111], 90.00th=[12256], 95.00th=[15401], | 99.00th=[28181], 99.50th=[40633], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 608, max= 1152, per=23.95%, avg=920.31, stdev=170.46, samples=13 iops : min= 38, max= 72, avg=57.38, stdev=10.68, samples=13 lat (usec) : 100=0.91%, 250=0.39%, 1000=5.47% lat (msec) : 2=4.69%, 4=18.23%, 10=30.34%, 20=37.50%, 50=2.21% lat (msec) : 100=0.26% cpu : usr=0.05%, sys=20.07%, ctx=2862, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23766: Tue Dec 17 07:33:09 2024 read: IOPS=57, BW=922KiB/s (944kB/s)(6032KiB/6541msec) clat (usec): min=64, max=29694, avg=11723.84, stdev=3542.28 lat (usec): min=64, max=29695, avg=11724.65, stdev=3542.22 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 7373], 10.00th=[ 8291], 20.00th=[ 9372], | 30.00th=[10290], 40.00th=[10945], 50.00th=[11600], 60.00th=[12125], | 70.00th=[12911], 80.00th=[13698], 90.00th=[15401], 95.00th=[17433], | 99.00th=[25822], 99.50th=[27657], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 704, max= 1088, per=25.07%, avg=915.38, stdev=116.48, samples=13 iops : min= 44, max= 68, avg=57.08, stdev= 7.27, samples=13 write: IOPS=59, BW=956KiB/s (979kB/s)(6256KiB/6541msec) clat (usec): min=658, max=104203, avg=5391.18, stdev=7167.24 lat (usec): min=659, max=104205, avg=5393.32, stdev=7167.34 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 824], 10.00th=[ 922], 20.00th=[ 1434], | 30.00th=[ 2474], 40.00th=[ 2835], 50.00th=[ 3392], 60.00th=[ 4490], | 70.00th=[ 5800], 80.00th=[ 7570], 90.00th=[ 11994], 95.00th=[ 14484], | 99.00th=[ 21627], 99.50th=[ 56361], 99.90th=[104334], 99.95th=[104334], | 99.99th=[104334] bw ( KiB/s): min= 736, max= 1216, per=24.91%, avg=957.15, stdev=142.26, samples=13 iops : min= 46, max= 76, avg=59.69, stdev= 8.95, samples=13 lat (usec) : 100=0.52%, 250=0.13%, 750=0.26%, 1000=6.25% lat (msec) : 2=4.95%, 4=17.45%, 10=28.26%, 20=40.49%, 50=1.43% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.14%, sys=18.81%, ctx=2847, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3653KiB/s (3741kB/s), 876KiB/s-961KiB/s (897kB/s-985kB/s), io=23.4MiB (24.5MB), run=6481-6557msec WRITE: bw=3843KiB/s (3935kB/s), 927KiB/s-998KiB/s (949kB/s-1022kB/s), io=24.6MiB (25.8MB), run=6481-6557msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23767: Tue Dec 17 07:33:09 2024 read: IOPS=56, BW=911KiB/s (933kB/s)(5920KiB/6495msec) clat (usec): min=1850, max=30235, avg=6270.84, stdev=4940.74 lat (usec): min=1851, max=30235, avg=6271.56, stdev=4940.75 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2245], 10.00th=[ 2409], 20.00th=[ 2638], | 30.00th=[ 2933], 40.00th=[ 3359], 50.00th=[ 3949], 60.00th=[ 5211], | 70.00th=[ 7570], 80.00th=[ 9634], 90.00th=[12649], 95.00th=[15664], | 99.00th=[25822], 99.50th=[29230], 99.90th=[30278], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 672, max= 1216, per=24.57%, avg=898.17, stdev=178.09, samples=12 iops : min= 42, max= 76, avg=56.00, stdev=10.98, samples=12 write: IOPS=61, BW=980KiB/s (1004kB/s)(6368KiB/6495msec) clat (msec): min=2, max=102, avg=10.43, stdev= 7.98 lat (msec): min=2, max=102, avg=10.43, stdev= 7.98 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 15], 90.00th=[ 18], 95.00th=[ 23], | 99.00th=[ 42], 99.50th=[ 48], 99.90th=[ 103], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min= 702, max= 1248, per=25.37%, avg=975.67, stdev=191.58, samples=12 iops : min= 43, max= 78, avg=60.83, stdev=12.11, samples=12 lat (msec) : 2=0.39%, 4=28.91%, 10=41.15%, 20=24.74%, 50=4.69% lat (msec) : 250=0.13% cpu : usr=0.11%, sys=9.69%, ctx=1193, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23768: Tue Dec 17 07:33:09 2024 read: IOPS=54, BW=877KiB/s (898kB/s)(5744KiB/6553msec) clat (usec): min=1732, max=34397, avg=6676.58, stdev=5251.80 lat (usec): min=1733, max=34400, avg=6677.30, stdev=5251.99 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2089], 10.00th=[ 2343], 20.00th=[ 2704], | 30.00th=[ 2999], 40.00th=[ 3523], 50.00th=[ 4490], 60.00th=[ 6194], | 70.00th=[ 8094], 80.00th=[10421], 90.00th=[13042], 95.00th=[16909], | 99.00th=[26084], 99.50th=[31851], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 672, max= 1120, per=23.77%, avg=868.62, stdev=135.25, samples=13 iops : min= 42, max= 70, avg=54.15, stdev= 8.55, samples=13 write: IOPS=62, BW=999KiB/s (1023kB/s)(6544KiB/6553msec) clat (msec): min=2, max=104, avg=10.07, stdev= 7.75 lat (msec): min=2, max=104, avg=10.07, stdev= 7.75 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 14], 90.00th=[ 18], 95.00th=[ 21], | 99.00th=[ 31], 99.50th=[ 39], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 736, max= 1248, per=25.73%, avg=989.23, stdev=187.37, samples=13 iops : min= 46, max= 78, avg=61.69, stdev=11.76, samples=13 lat (msec) : 2=0.91%, 4=25.91%, 10=42.58%, 20=25.65%, 50=4.69% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.09%, sys=9.91%, ctx=1271, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23769: Tue Dec 17 07:33:09 2024 read: IOPS=59, BW=959KiB/s (982kB/s)(6256KiB/6522msec) clat (usec): min=1825, max=83413, avg=7037.23, stdev=6524.40 lat (usec): min=1826, max=83414, avg=7038.06, stdev=6524.41 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2278], 10.00th=[ 2409], 20.00th=[ 2737], | 30.00th=[ 3163], 40.00th=[ 3654], 50.00th=[ 5080], 60.00th=[ 6915], | 70.00th=[ 8586], 80.00th=[10552], 90.00th=[13304], 95.00th=[16909], | 99.00th=[30802], 99.50th=[43254], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 736, max= 1152, per=26.26%, avg=959.77, stdev=144.19, samples=13 iops : min= 46, max= 72, avg=59.92, stdev= 8.99, samples=13 write: IOPS=57, BW=925KiB/s (947kB/s)(6032KiB/6522msec) clat (usec): min=2760, max=59960, avg=9824.33, stdev=6442.34 lat (usec): min=2761, max=59960, avg=9825.62, stdev=6442.40 clat percentiles (usec): | 1.00th=[ 3097], 5.00th=[ 3589], 10.00th=[ 3884], 20.00th=[ 4948], | 30.00th=[ 5866], 40.00th=[ 6915], 50.00th=[ 8225], 60.00th=[ 9896], | 70.00th=[11338], 80.00th=[14091], 90.00th=[16909], 95.00th=[20579], | 99.00th=[34866], 99.50th=[46924], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 606, max= 1184, per=23.88%, avg=918.00, stdev=170.83, samples=13 iops : min= 37, max= 74, avg=57.31, stdev=10.81, samples=13 lat (msec) : 2=0.52%, 4=27.34%, 10=42.06%, 20=26.04%, 50=3.78% lat (msec) : 100=0.26% cpu : usr=0.17%, sys=9.62%, ctx=1272, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23770: Tue Dec 17 07:33:09 2024 read: IOPS=57, BW=921KiB/s (943kB/s)(6032KiB/6549msec) clat (usec): min=1658, max=75257, avg=6784.18, stdev=5924.49 lat (usec): min=1659, max=75257, avg=6784.95, stdev=5924.43 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2245], 10.00th=[ 2409], 20.00th=[ 2802], | 30.00th=[ 3195], 40.00th=[ 3621], 50.00th=[ 4686], 60.00th=[ 6259], | 70.00th=[ 8717], 80.00th=[10552], 90.00th=[12780], 95.00th=[15533], | 99.00th=[25560], 99.50th=[29492], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 544, max= 1184, per=24.98%, avg=913.08, stdev=173.63, samples=13 iops : min= 34, max= 74, avg=57.00, stdev=10.86, samples=13 write: IOPS=59, BW=955KiB/s (978kB/s)(6256KiB/6549msec) clat (usec): min=2401, max=57108, avg=10065.15, stdev=6386.11 lat (usec): min=2402, max=57109, avg=10066.66, stdev=6386.03 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3458], 10.00th=[ 3851], 20.00th=[ 5014], | 30.00th=[ 6390], 40.00th=[ 7308], 50.00th=[ 8717], 60.00th=[10290], | 70.00th=[11469], 80.00th=[13566], 90.00th=[17695], 95.00th=[21890], | 99.00th=[30540], 99.50th=[45351], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 736, max= 1248, per=24.77%, avg=952.46, stdev=153.61, samples=13 iops : min= 46, max= 78, avg=59.46, stdev= 9.63, samples=13 lat (msec) : 2=1.30%, 4=26.69%, 10=39.71%, 20=27.47%, 50=4.56% lat (msec) : 100=0.26% cpu : usr=0.14%, sys=9.48%, ctx=1223, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3655KiB/s (3743kB/s), 877KiB/s-959KiB/s (898kB/s-982kB/s), io=23.4MiB (24.5MB), run=6495-6553msec WRITE: bw=3846KiB/s (3938kB/s), 925KiB/s-999KiB/s (947kB/s-1023kB/s), io=24.6MiB (25.8MB), run=6495-6553msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23780: Tue Dec 17 07:33:12 2024 read: IOPS=3, BW=3340KiB/s (3420kB/s)(7168KiB/2146msec) clat (usec): min=423, max=1000.0k, avg=275220.52, stdev=368223.20 lat (usec): min=424, max=1000.0k, avg=275221.96, stdev=368223.06 clat percentiles (usec): | 1.00th=[ 424], 5.00th=[ 424], 10.00th=[ 424], | 20.00th=[ 12649], 30.00th=[ 28443], 40.00th=[ 28443], | 50.00th=[ 111674], 60.00th=[ 263193], 70.00th=[ 263193], | 80.00th=[ 509608], 90.00th=[1002439], 95.00th=[1002439], | 99.00th=[1002439], 99.50th=[1002439], 99.90th=[1002439], | 99.95th=[1002439], 99.99th=[1002439] bw ( KiB/s): min= 2048, max= 6144, per=35.93%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 write: IOPS=2, BW=2386KiB/s (2443kB/s)(5120KiB/2146msec) clat (usec): min=20187, max=82536, avg=42110.31, stdev=25171.27 lat (usec): min=20206, max=82604, avg=42163.95, stdev=25181.42 clat percentiles (usec): | 1.00th=[20317], 5.00th=[20317], 10.00th=[20317], 20.00th=[20317], | 30.00th=[23987], 40.00th=[23987], 50.00th=[34866], 60.00th=[34866], | 70.00th=[48497], 80.00th=[48497], 90.00th=[82314], 95.00th=[82314], | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 2048, max= 2048, per=15.40%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (usec) : 500=8.33% lat (msec) : 20=8.33%, 50=41.67%, 100=8.33%, 250=8.33%, 500=8.33% lat (msec) : 750=8.33% cpu : usr=0.00%, sys=30.49%, ctx=779, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23781: Tue Dec 17 07:33:12 2024 read: IOPS=0, BW=950KiB/s (973kB/s)(2048KiB/2156msec) clat (msec): min=40, max=525, avg=282.95, stdev=342.78 lat (msec): min=40, max=525, avg=282.95, stdev=342.78 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 41], 60.00th=[ 527], | 70.00th=[ 527], 80.00th=[ 527], 90.00th=[ 527], 95.00th=[ 527], | 99.00th=[ 527], 99.50th=[ 527], 99.90th=[ 527], 99.95th=[ 527], | 99.99th=[ 527] bw ( KiB/s): min= 2048, max= 2048, per=21.56%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4750KiB/s (4864kB/s)(10.0MiB/2156msec) clat (msec): min=19, max=1154, avg=153.23, stdev=353.23 lat (msec): min=20, max=1154, avg=153.30, stdev=353.22 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 23], | 30.00th=[ 24], 40.00th=[ 29], 50.00th=[ 31], 60.00th=[ 36], | 70.00th=[ 45], 80.00th=[ 45], 90.00th=[ 131], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 4096, max=10240, per=53.90%, avg=7168.00, stdev=4344.46, samples=2 iops : min= 4, max= 10, avg= 7.00, stdev= 4.24, samples=2 lat (msec) : 20=8.33%, 50=66.67%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=17.35%, ctx=356, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23782: Tue Dec 17 07:33:12 2024 read: IOPS=2, BW=2927KiB/s (2997kB/s)(6144KiB/2099msec) clat (usec): min=526, max=497048, avg=97317.62, stdev=196455.57 lat (usec): min=527, max=497050, avg=97318.68, stdev=196455.76 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 529], 10.00th=[ 529], 20.00th=[ 529], | 30.00th=[ 529], 40.00th=[ 19530], 50.00th=[ 19530], 60.00th=[ 24249], | 70.00th=[ 42206], 80.00th=[ 42206], 90.00th=[497026], 95.00th=[497026], | 99.00th=[497026], 99.50th=[497026], 99.90th=[497026], 99.95th=[497026], | 99.99th=[497026] bw ( KiB/s): min= 4096, max= 4096, per=43.12%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2927KiB/s (2997kB/s)(6144KiB/2099msec) clat (msec): min=27, max=1175, avg=243.23, stdev=457.62 lat (msec): min=27, max=1175, avg=243.28, stdev=457.63 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 34], | 30.00th=[ 34], 40.00th=[ 48], 50.00th=[ 48], 60.00th=[ 81], | 70.00th=[ 95], 80.00th=[ 95], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 4096, max= 4096, per=30.80%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 lat (usec) : 750=16.67% lat (msec) : 20=8.33%, 50=41.67%, 100=16.67%, 500=8.33% cpu : usr=0.00%, sys=11.68%, ctx=349, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23783: Tue Dec 17 07:33:12 2024 read: IOPS=2, BW=2437KiB/s (2495kB/s)(5120KiB/2101msec) clat (usec): min=334, max=511346, avg=117062.39, stdev=221590.86 lat (usec): min=335, max=511347, avg=117063.21, stdev=221591.00 clat percentiles (usec): | 1.00th=[ 334], 5.00th=[ 334], 10.00th=[ 334], 20.00th=[ 334], | 30.00th=[ 545], 40.00th=[ 545], 50.00th=[ 16712], 60.00th=[ 16712], | 70.00th=[ 56361], 80.00th=[ 56361], 90.00th=[509608], 95.00th=[509608], | 99.00th=[509608], 99.50th=[509608], 99.90th=[509608], 99.95th=[509608], | 99.99th=[509608] bw ( KiB/s): min= 2048, max= 6144, per=43.12%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=3, BW=3412KiB/s (3494kB/s)(7168KiB/2101msec) clat (msec): min=18, max=1132, avg=207.37, stdev=408.65 lat (msec): min=18, max=1132, avg=207.43, stdev=408.66 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 34], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 56], 60.00th=[ 75], | 70.00th=[ 75], 80.00th=[ 96], 90.00th=[ 1133], 95.00th=[ 1133], | 99.00th=[ 1133], 99.50th=[ 1133], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 2048, max= 6144, per=30.80%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 20=16.67%, 50=16.67%, 100=33.33%, 750=8.33% cpu : usr=0.05%, sys=12.62%, ctx=386, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9499KiB/s (9727kB/s), 950KiB/s-3340KiB/s (973kB/s-3420kB/s), io=20.0MiB (20.0MB), run=2099-2156msec WRITE: bw=12.0MiB/s (13.6MB/s), 2386KiB/s-4750KiB/s (2443kB/s-4864kB/s), io=28.0MiB (29.4MB), run=2099-2156msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23784: Tue Dec 17 07:33:12 2024 read: IOPS=3, BW=3148KiB/s (3224kB/s)(7168KiB/2277msec) clat (msec): min=23, max=1167, avg=274.46, stdev=424.00 lat (msec): min=23, max=1167, avg=274.46, stdev=424.00 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 54], | 30.00th=[ 65], 40.00th=[ 65], 50.00th=[ 69], 60.00th=[ 69], | 70.00th=[ 69], 80.00th=[ 477], 90.00th=[ 1167], 95.00th=[ 1167], | 99.00th=[ 1167], 99.50th=[ 1167], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 2048, max= 4096, per=38.20%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 write: IOPS=2, BW=2249KiB/s (2303kB/s)(5120KiB/2277msec) clat (msec): min=18, max=117, avg=59.51, stdev=44.82 lat (msec): min=18, max=117, avg=59.57, stdev=44.81 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 34], 60.00th=[ 34], | 70.00th=[ 99], 80.00th=[ 99], 90.00th=[ 117], 95.00th=[ 117], | 99.00th=[ 117], 99.50th=[ 117], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 2048, max= 2048, per=16.37%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=25.00%, 100=41.67%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=3.91%, ctx=55, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23785: Tue Dec 17 07:33:12 2024 read: IOPS=0, BW=894KiB/s (915kB/s)(2048KiB/2292msec) clat (usec): min=28753, max=49857, avg=39305.52, stdev=14922.83 lat (usec): min=28754, max=49858, avg=39306.86, stdev=14922.76 clat percentiles (usec): | 1.00th=[28705], 5.00th=[28705], 10.00th=[28705], 20.00th=[28705], | 30.00th=[28705], 40.00th=[28705], 50.00th=[28705], 60.00th=[50070], | 70.00th=[50070], 80.00th=[50070], 90.00th=[50070], 95.00th=[50070], | 99.00th=[50070], 99.50th=[50070], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 2048, max= 2048, per=22.92%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4468KiB/s (4575kB/s)(10.0MiB/2292msec) clat (msec): min=19, max=1183, avg=214.40, stdev=383.70 lat (msec): min=19, max=1183, avg=214.47, stdev=383.69 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 22], | 30.00th=[ 28], 40.00th=[ 33], 50.00th=[ 36], 60.00th=[ 38], | 70.00th=[ 124], 80.00th=[ 124], 90.00th=[ 600], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 6144, max= 6144, per=49.12%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=58.33%, 100=8.33%, 250=8.33%, 750=8.33% cpu : usr=0.09%, sys=5.06%, ctx=48, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23786: Tue Dec 17 07:33:12 2024 read: IOPS=2, BW=2750KiB/s (2816kB/s)(6144KiB/2234msec) clat (msec): min=11, max=464, avg=116.57, stdev=173.93 lat (msec): min=11, max=464, avg=116.57, stdev=173.93 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 19], | 30.00th=[ 19], 40.00th=[ 30], 50.00th=[ 30], 60.00th=[ 69], | 70.00th=[ 107], 80.00th=[ 107], 90.00th=[ 464], 95.00th=[ 464], | 99.00th=[ 464], 99.50th=[ 464], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 2048, max= 4096, per=34.38%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2750KiB/s (2816kB/s)(6144KiB/2234msec) clat (msec): min=24, max=1040, avg=244.80, stdev=394.77 lat (msec): min=24, max=1040, avg=244.87, stdev=394.77 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 55], | 30.00th=[ 55], 40.00th=[ 67], 50.00th=[ 67], 60.00th=[ 80], | 70.00th=[ 205], 80.00th=[ 205], 90.00th=[ 1045], 95.00th=[ 1045], | 99.00th=[ 1045], 99.50th=[ 1045], 99.90th=[ 1045], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 2048, max= 4096, per=21.83%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 lat (msec) : 20=16.67%, 50=16.67%, 100=33.33%, 250=16.67%, 500=8.33% cpu : usr=0.04%, sys=6.09%, ctx=68, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23787: Tue Dec 17 07:33:12 2024 read: IOPS=2, BW=2299KiB/s (2354kB/s)(5120KiB/2227msec) clat (msec): min=14, max=515, avg=133.28, stdev=215.56 lat (msec): min=14, max=515, avg=133.28, stdev=215.56 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15], | 30.00th=[ 22], 40.00th=[ 22], 50.00th=[ 31], 60.00th=[ 31], | 70.00th=[ 85], 80.00th=[ 85], 90.00th=[ 514], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 4096, max= 4096, per=45.84%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3219KiB/s (3296kB/s)(7168KiB/2227msec) clat (msec): min=23, max=1182, avg=220.99, stdev=424.78 lat (msec): min=23, max=1182, avg=221.07, stdev=424.77 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 40], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 61], 60.00th=[ 96], | 70.00th=[ 96], 80.00th=[ 100], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 2048, max= 6144, per=32.74%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=8.33%, 50=41.67%, 100=33.33%, 750=8.33% cpu : usr=0.13%, sys=5.84%, ctx=94, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8935KiB/s (9150kB/s), 894KiB/s-3148KiB/s (915kB/s-3224kB/s), io=20.0MiB (20.0MB), run=2227-2292msec WRITE: bw=12.2MiB/s (12.8MB/s), 2249KiB/s-4468KiB/s (2303kB/s-4575kB/s), io=28.0MiB (29.4MB), run=2227-2292msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23795: Tue Dec 17 07:33:16 2024 read: IOPS=0, BW=3881KiB/s (3974kB/s)(12.0MiB/3166msec) clat (msec): min=100, max=1792, avg=1049.85, stdev=864.47 lat (msec): min=100, max=1792, avg=1049.85, stdev=864.47 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 102], 10.00th=[ 102], 20.00th=[ 102], | 30.00th=[ 102], 40.00th=[ 1250], 50.00th=[ 1250], 60.00th=[ 1250], | 70.00th=[ 1787], 80.00th=[ 1787], 90.00th=[ 1787], 95.00th=[ 1787], | 99.00th=[ 1787], 99.50th=[ 1787], 99.90th=[ 1787], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.03%, sys=41.33%, ctx=1364, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23797: Tue Dec 17 07:33:16 2024 read: IOPS=0, BW=1554KiB/s (1591kB/s)(4096KiB/2636msec) clat (nsec): min=148893k, max=148893k, avg=148892817.00, stdev= 0.00 lat (nsec): min=148895k, max=148895k, avg=148895182.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 148], 10.00th=[ 148], 20.00th=[ 148], | 30.00th=[ 148], 40.00th=[ 148], 50.00th=[ 148], 60.00th=[ 148], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3108KiB/s (3182kB/s)(8192KiB/2636msec) clat (msec): min=986, max=1494, avg=1240.51, stdev=358.69 lat (msec): min=987, max=1494, avg=1240.70, stdev=358.78 clat percentiles (msec): | 1.00th=[ 986], 5.00th=[ 986], 10.00th=[ 986], 20.00th=[ 986], | 30.00th=[ 986], 40.00th=[ 986], 50.00th=[ 986], 60.00th=[ 1502], | 70.00th=[ 1502], 80.00th=[ 1502], 90.00th=[ 1502], 95.00th=[ 1502], | 99.00th=[ 1502], 99.50th=[ 1502], 99.90th=[ 1502], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 8192, max= 8192, per=79.08%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=12.49%, ctx=118, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23799: Tue Dec 17 07:33:16 2024 write: IOPS=0, BW=3885KiB/s (3978kB/s)(12.0MiB/3163msec) clat (msec): min=300, max=1636, avg=1049.56, stdev=682.60 lat (msec): min=300, max=1636, avg=1049.83, stdev=682.58 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 1217], 50.00th=[ 1217], 60.00th=[ 1217], | 70.00th=[ 1636], 80.00th=[ 1636], 90.00th=[ 1636], 95.00th=[ 1636], | 99.00th=[ 1636], 99.50th=[ 1636], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 8175, max= 8192, per=79.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=10.37%, ctx=268, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23801: Tue Dec 17 07:33:16 2024 write: IOPS=1, BW=5001KiB/s (5121kB/s)(12.0MiB/2457msec) clat (msec): min=116, max=1456, avg=812.24, stdev=671.22 lat (msec): min=116, max=1456, avg=812.53, stdev=671.21 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 117], 10.00th=[ 117], 20.00th=[ 117], | 30.00th=[ 117], 40.00th=[ 869], 50.00th=[ 869], 60.00th=[ 869], | 70.00th=[ 1452], 80.00th=[ 1452], 90.00th=[ 1452], 95.00th=[ 1452], | 99.00th=[ 1452], 99.50th=[ 1452], 99.90th=[ 1452], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 8192, max= 8192, per=79.08%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=13.88%, ctx=33, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5175KiB/s (5299kB/s), 1554KiB/s-3881KiB/s (1591kB/s-3974kB/s), io=16.0MiB (16.8MB), run=2636-3166msec WRITE: bw=10.1MiB/s (10.6MB/s), 3108KiB/s-5001KiB/s (3182kB/s-5121kB/s), io=32.0MiB (33.6MB), run=2457-3163msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23796: Tue Dec 17 07:33:17 2024 read: IOPS=0, BW=3811KiB/s (3903kB/s)(12.0MiB/3224msec) clat (msec): min=430, max=1744, avg=1067.79, stdev=657.97 lat (msec): min=430, max=1744, avg=1067.79, stdev=657.97 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 430], 10.00th=[ 430], 20.00th=[ 430], | 30.00th=[ 430], 40.00th=[ 1028], 50.00th=[ 1028], 60.00th=[ 1028], | 70.00th=[ 1737], 80.00th=[ 1737], 90.00th=[ 1737], 95.00th=[ 1737], | 99.00th=[ 1737], 99.50th=[ 1737], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33% cpu : usr=0.03%, sys=3.60%, ctx=29, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23798: Tue Dec 17 07:33:17 2024 read: IOPS=0, BW=1483KiB/s (1519kB/s)(4096KiB/2762msec) clat (nsec): min=450974k, max=450974k, avg=450973848.00, stdev= 0.00 lat (nsec): min=450975k, max=450975k, avg=450975411.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 451], 10.00th=[ 451], 20.00th=[ 451], | 30.00th=[ 451], 40.00th=[ 451], 50.00th=[ 451], 60.00th=[ 451], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2966KiB/s (3037kB/s)(8192KiB/2762msec) clat (msec): min=754, max=1524, avg=1139.39, stdev=544.80 lat (msec): min=754, max=1524, avg=1139.65, stdev=544.78 clat percentiles (msec): | 1.00th=[ 751], 5.00th=[ 751], 10.00th=[ 751], 20.00th=[ 751], | 30.00th=[ 751], 40.00th=[ 751], 50.00th=[ 751], 60.00th=[ 1519], | 70.00th=[ 1519], 80.00th=[ 1519], 90.00th=[ 1519], 95.00th=[ 1519], | 99.00th=[ 1519], 99.50th=[ 1519], 99.90th=[ 1519], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 8192, max= 8192, per=81.28%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=4.96%, ctx=190, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23800: Tue Dec 17 07:33:17 2024 write: IOPS=0, BW=3780KiB/s (3870kB/s)(12.0MiB/3251msec) clat (msec): min=463, max=1708, avg=1067.27, stdev=623.82 lat (msec): min=463, max=1709, avg=1067.55, stdev=623.83 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 464], 10.00th=[ 464], 20.00th=[ 464], | 30.00th=[ 464], 40.00th=[ 1028], 50.00th=[ 1028], 60.00th=[ 1028], | 70.00th=[ 1703], 80.00th=[ 1703], 90.00th=[ 1703], 95.00th=[ 1703], | 99.00th=[ 1703], 99.50th=[ 1703], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 8192, max= 8192, per=81.28%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=2.00%, ctx=35, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23802: Tue Dec 17 07:33:17 2024 write: IOPS=1, BW=4538KiB/s (4647kB/s)(12.0MiB/2708msec) clat (msec): min=280, max=1465, avg=896.87, stdev=594.20 lat (msec): min=280, max=1465, avg=897.13, stdev=594.19 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 279], 10.00th=[ 279], 20.00th=[ 279], | 30.00th=[ 279], 40.00th=[ 944], 50.00th=[ 944], 60.00th=[ 944], | 70.00th=[ 1469], 80.00th=[ 1469], 90.00th=[ 1469], 95.00th=[ 1469], | 99.00th=[ 1469], 99.50th=[ 1469], 99.90th=[ 1469], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 8078, max= 8175, per=80.63%, avg=8126.50, stdev=68.59, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.04%, sys=6.43%, ctx=182, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5082KiB/s (5204kB/s), 1483KiB/s-3811KiB/s (1519kB/s-3903kB/s), io=16.0MiB (16.8MB), run=2762-3224msec WRITE: bw=9.84MiB/s (10.3MB/s), 2966KiB/s-4538KiB/s (3037kB/s-4647kB/s), io=32.0MiB (33.6MB), run=2708-3251msec