== sanity test 398b: DIO and buffer IO race ============== 07:34:52 (1734438892) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.28353 s, 39.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28426: Tue Dec 17 07:35:08 2024 read: IOPS=116, BW=465KiB/s (476kB/s)(5992KiB/12892msec) clat (usec): min=56, max=189553, avg=4882.93, stdev=5508.76 lat (usec): min=57, max=189554, avg=4884.19, stdev=5508.80 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 1844], 10.00th=[ 2343], 20.00th=[ 2737], | 30.00th=[ 3130], 40.00th=[ 3654], 50.00th=[ 4359], 60.00th=[ 5014], | 70.00th=[ 5669], 80.00th=[ 6521], 90.00th=[ 7963], 95.00th=[ 9503], | 99.00th=[ 14484], 99.50th=[ 17433], 99.90th=[ 27132], 99.95th=[189793], | 99.99th=[189793] bw ( KiB/s): min= 231, max= 560, per=25.16%, avg=458.40, stdev=85.78, samples=25 iops : min= 57, max= 140, avg=114.52, stdev=21.51, samples=25 write: IOPS=122, BW=488KiB/s (500kB/s)(6296KiB/12892msec) clat (usec): min=423, max=27660, avg=3490.81, stdev=3152.23 lat (usec): min=423, max=27660, avg=3491.96, stdev=3152.38 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 545], 10.00th=[ 586], 20.00th=[ 685], | 30.00th=[ 889], 40.00th=[ 1942], 50.00th=[ 2900], 60.00th=[ 3490], | 70.00th=[ 4424], 80.00th=[ 5669], 90.00th=[ 7767], 95.00th=[ 9634], | 99.00th=[13566], 99.50th=[14877], 99.90th=[22676], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 295, max= 648, per=25.13%, avg=479.48, stdev=78.35, samples=25 iops : min= 73, max= 162, avg=119.80, stdev=19.62, samples=25 lat (usec) : 100=2.02%, 250=0.29%, 500=0.26%, 750=12.30%, 1000=4.56% lat (msec) : 2=4.07%, 4=32.58%, 10=39.55%, 20=4.13%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=23.05%, ctx=5813, majf=0, minf=37 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28427: Tue Dec 17 07:35:08 2024 read: IOPS=113, BW=454KiB/s (465kB/s)(5940KiB/13075msec) clat (usec): min=50, max=25177, avg=4834.28, stdev=2743.35 lat (usec): min=50, max=25178, avg=4835.01, stdev=2743.28 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 2073], 10.00th=[ 2343], 20.00th=[ 2802], | 30.00th=[ 3130], 40.00th=[ 3621], 50.00th=[ 4293], 60.00th=[ 4948], | 70.00th=[ 5800], 80.00th=[ 6718], 90.00th=[ 8029], 95.00th=[ 9634], | 99.00th=[14746], 99.50th=[16057], 99.90th=[19792], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 183, max= 608, per=24.85%, avg=452.73, stdev=83.59, samples=26 iops : min= 45, max= 152, avg=113.08, stdev=21.03, samples=26 write: IOPS=121, BW=486KiB/s (497kB/s)(6348KiB/13075msec) clat (usec): min=435, max=187104, avg=3664.65, stdev=5628.90 lat (usec): min=436, max=187105, avg=3665.40, stdev=5628.89 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 545], 10.00th=[ 578], 20.00th=[ 676], | 30.00th=[ 848], 40.00th=[ 1729], 50.00th=[ 2868], 60.00th=[ 3556], | 70.00th=[ 4490], 80.00th=[ 6063], 90.00th=[ 8291], 95.00th=[ 10159], | 99.00th=[ 13829], 99.50th=[ 16057], 99.90th=[ 21890], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 255, max= 656, per=25.26%, avg=481.96, stdev=92.49, samples=26 iops : min= 63, max= 164, avg=120.38, stdev=23.16, samples=26 lat (usec) : 100=1.66%, 250=0.26%, 500=0.33%, 750=12.60%, 1000=4.92% lat (msec) : 2=3.71%, 4=32.42%, 10=39.39%, 20=4.62%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=22.78%, ctx=5839, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28428: Tue Dec 17 07:35:08 2024 read: IOPS=114, BW=458KiB/s (469kB/s)(6028KiB/13175msec) clat (usec): min=48, max=186512, avg=4996.42, stdev=5473.45 lat (usec): min=48, max=186513, avg=4997.04, stdev=5473.46 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 1827], 10.00th=[ 2311], 20.00th=[ 2835], | 30.00th=[ 3294], 40.00th=[ 3785], 50.00th=[ 4359], 60.00th=[ 5014], | 70.00th=[ 5800], 80.00th=[ 6587], 90.00th=[ 8029], 95.00th=[ 9896], | 99.00th=[ 15795], 99.50th=[ 17433], 99.90th=[ 31065], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 199, max= 576, per=24.84%, avg=452.50, stdev=75.01, samples=26 iops : min= 49, max= 144, avg=113.04, stdev=18.82, samples=26 write: IOPS=118, BW=475KiB/s (487kB/s)(6260KiB/13175msec) clat (usec): min=464, max=19636, avg=3555.31, stdev=3327.11 lat (usec): min=464, max=19636, avg=3557.35, stdev=3327.46 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 545], 10.00th=[ 578], 20.00th=[ 660], | 30.00th=[ 832], 40.00th=[ 1647], 50.00th=[ 2802], 60.00th=[ 3556], | 70.00th=[ 4424], 80.00th=[ 5932], 90.00th=[ 8094], 95.00th=[10683], | 99.00th=[14484], 99.50th=[16319], 99.90th=[19268], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 279, max= 696, per=24.62%, avg=469.73, stdev=89.78, samples=26 iops : min= 69, max= 174, avg=117.35, stdev=22.51, samples=26 lat (usec) : 50=0.07%, 100=1.79%, 250=0.46%, 500=0.26%, 750=12.92% lat (usec) : 1000=4.39% lat (msec) : 2=3.97%, 4=31.22%, 10=39.45%, 20=5.34%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.10%, sys=23.17%, ctx=5750, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28429: Tue Dec 17 07:35:08 2024 read: IOPS=115, BW=462KiB/s (473kB/s)(6052KiB/13102msec) clat (usec): min=55, max=32126, avg=4792.59, stdev=2704.36 lat (usec): min=56, max=32126, avg=4794.03, stdev=2704.43 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 1926], 10.00th=[ 2343], 20.00th=[ 2835], | 30.00th=[ 3294], 40.00th=[ 3785], 50.00th=[ 4359], 60.00th=[ 5014], | 70.00th=[ 5669], 80.00th=[ 6521], 90.00th=[ 7898], 95.00th=[ 9110], | 99.00th=[13435], 99.50th=[16909], 99.90th=[26084], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 239, max= 608, per=25.22%, avg=459.50, stdev=71.06, samples=26 iops : min= 59, max= 152, avg=114.77, stdev=17.83, samples=26 write: IOPS=118, BW=476KiB/s (487kB/s)(6236KiB/13102msec) clat (usec): min=486, max=196720, avg=3702.17, stdev=5829.93 lat (usec): min=487, max=196721, avg=3702.96, stdev=5829.97 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 553], 10.00th=[ 586], 20.00th=[ 676], | 30.00th=[ 889], 40.00th=[ 2024], 50.00th=[ 2933], 60.00th=[ 3589], | 70.00th=[ 4490], 80.00th=[ 6194], 90.00th=[ 8225], 95.00th=[ 9765], | 99.00th=[ 14615], 99.50th=[ 15795], 99.90th=[ 18744], 99.95th=[196084], | 99.99th=[196084] bw ( KiB/s): min= 295, max= 696, per=24.88%, avg=474.62, stdev=72.70, samples=26 iops : min= 73, max= 174, avg=118.54, stdev=18.26, samples=26 lat (usec) : 100=1.69%, 250=0.46%, 500=0.10%, 750=12.08%, 1000=4.65% lat (msec) : 2=4.00%, 4=31.93%, 10=41.18%, 20=3.78%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=22.86%, ctx=5927, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1823KiB/s (1866kB/s), 454KiB/s-465KiB/s (465kB/s-476kB/s), io=23.4MiB (24.6MB), run=12892-13175msec WRITE: bw=1908KiB/s (1954kB/s), 475KiB/s-488KiB/s (487kB/s-500kB/s), io=24.6MiB (25.7MB), run=12892-13175msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28430: Tue Dec 17 07:35:12 2024 read: IOPS=86, BW=347KiB/s (355kB/s)(5992KiB/17261msec) clat (usec): min=1605, max=191192, avg=4712.65, stdev=5888.16 lat (usec): min=1606, max=191194, avg=4713.31, stdev=5888.20 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1860], 10.00th=[ 1975], 20.00th=[ 2180], | 30.00th=[ 2409], 40.00th=[ 2769], 50.00th=[ 3261], 60.00th=[ 3884], | 70.00th=[ 4686], 80.00th=[ 6521], 90.00th=[ 9896], 95.00th=[ 12125], | 99.00th=[ 16581], 99.50th=[ 18482], 99.90th=[ 23200], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 136, max= 800, per=24.79%, avg=344.85, stdev=160.59, samples=34 iops : min= 34, max= 200, avg=86.15, stdev=40.17, samples=34 write: IOPS=91, BW=365KiB/s (374kB/s)(6296KiB/17261msec) clat (usec): min=2099, max=35780, avg=6435.32, stdev=3898.65 lat (usec): min=2100, max=35781, avg=6436.15, stdev=3898.68 clat percentiles (usec): | 1.00th=[ 2507], 5.00th=[ 2900], 10.00th=[ 3097], 20.00th=[ 3425], | 30.00th=[ 3851], 40.00th=[ 4424], 50.00th=[ 5145], 60.00th=[ 6063], | 70.00th=[ 7308], 80.00th=[ 9110], 90.00th=[11469], 95.00th=[13829], | 99.00th=[21103], 99.50th=[25297], 99.90th=[32375], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 152, max= 768, per=24.85%, avg=361.79, stdev=156.27, samples=34 iops : min= 38, max= 192, avg=90.38, stdev=39.09, samples=34 lat (msec) : 2=5.60%, 4=41.31%, 10=40.49%, 20=11.82%, 50=0.75% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=10.85%, ctx=3623, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28431: Tue Dec 17 07:35:12 2024 read: IOPS=86, BW=345KiB/s (354kB/s)(5940KiB/17205msec) clat (usec): min=1564, max=186732, avg=4645.38, stdev=5758.43 lat (usec): min=1564, max=186733, avg=4646.09, stdev=5758.45 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1860], 10.00th=[ 1991], 20.00th=[ 2180], | 30.00th=[ 2474], 40.00th=[ 2868], 50.00th=[ 3326], 60.00th=[ 3884], | 70.00th=[ 4686], 80.00th=[ 6194], 90.00th=[ 9372], 95.00th=[ 11731], | 99.00th=[ 17695], 99.50th=[ 20317], 99.90th=[ 23987], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 104, max= 752, per=24.76%, avg=344.41, stdev=162.49, samples=34 iops : min= 26, max= 188, avg=86.06, stdev=40.65, samples=34 write: IOPS=92, BW=369KiB/s (378kB/s)(6348KiB/17205msec) clat (usec): min=2264, max=35340, avg=6433.10, stdev=4005.19 lat (usec): min=2265, max=35341, avg=6433.88, stdev=4005.19 clat percentiles (usec): | 1.00th=[ 2507], 5.00th=[ 2868], 10.00th=[ 3097], 20.00th=[ 3392], | 30.00th=[ 3752], 40.00th=[ 4293], 50.00th=[ 4948], 60.00th=[ 5932], | 70.00th=[ 7439], 80.00th=[ 9110], 90.00th=[11731], 95.00th=[14484], | 99.00th=[21365], 99.50th=[23200], 99.90th=[32375], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 160, max= 736, per=25.11%, avg=365.59, stdev=151.26, samples=34 iops : min= 40, max= 184, avg=91.35, stdev=37.83, samples=34 lat (msec) : 2=5.21%, 4=42.74%, 10=39.71%, 20=11.33%, 50=0.98% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=10.60%, ctx=3643, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28432: Tue Dec 17 07:35:12 2024 read: IOPS=88, BW=353KiB/s (361kB/s)(6028KiB/17079msec) clat (usec): min=1485, max=22512, avg=4560.84, stdev=3369.76 lat (usec): min=1486, max=22513, avg=4561.50, stdev=3369.78 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1844], 10.00th=[ 1975], 20.00th=[ 2180], | 30.00th=[ 2474], 40.00th=[ 2868], 50.00th=[ 3359], 60.00th=[ 3884], | 70.00th=[ 4621], 80.00th=[ 6128], 90.00th=[ 9241], 95.00th=[11994], | 99.00th=[16581], 99.50th=[19792], 99.90th=[21890], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 120, max= 736, per=25.33%, avg=352.38, stdev=151.17, samples=34 iops : min= 30, max= 184, avg=88.03, stdev=37.82, samples=34 write: IOPS=91, BW=367KiB/s (375kB/s)(6260KiB/17079msec) clat (msec): min=2, max=189, avg= 6.46, stdev= 6.04 lat (msec): min=2, max=189, avg= 6.46, stdev= 6.04 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 22], 99.50th=[ 24], 99.90th=[ 29], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 160, max= 824, per=25.07%, avg=365.09, stdev=159.46, samples=34 iops : min= 40, max= 206, avg=91.21, stdev=39.90, samples=34 lat (msec) : 2=5.34%, 4=42.77%, 10=39.68%, 20=11.30%, 50=0.88% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=10.63%, ctx=3607, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28433: Tue Dec 17 07:35:12 2024 read: IOPS=88, BW=353KiB/s (362kB/s)(6052KiB/17143msec) clat (usec): min=1476, max=23996, avg=4547.27, stdev=3360.03 lat (usec): min=1477, max=23996, avg=4547.91, stdev=3360.03 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1876], 10.00th=[ 1975], 20.00th=[ 2180], | 30.00th=[ 2409], 40.00th=[ 2802], 50.00th=[ 3326], 60.00th=[ 3916], | 70.00th=[ 4621], 80.00th=[ 6259], 90.00th=[ 9634], 95.00th=[11863], | 99.00th=[16450], 99.50th=[17957], 99.90th=[22414], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 120, max= 768, per=25.23%, avg=350.97, stdev=158.89, samples=34 iops : min= 30, max= 192, avg=87.68, stdev=39.75, samples=34 write: IOPS=90, BW=364KiB/s (372kB/s)(6236KiB/17143msec) clat (msec): min=2, max=207, avg= 6.53, stdev= 6.35 lat (msec): min=2, max=207, avg= 6.53, stdev= 6.35 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 22], 99.50th=[ 23], 99.90th=[ 29], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 160, max= 736, per=24.86%, avg=362.03, stdev=153.28, samples=34 iops : min= 40, max= 184, avg=90.44, stdev=38.36, samples=34 lat (msec) : 2=5.86%, 4=41.93%, 10=39.81%, 20=11.59%, 50=0.78% lat (msec) : 250=0.03% cpu : usr=0.11%, sys=10.66%, ctx=3656, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1391KiB/s (1424kB/s), 345KiB/s-353KiB/s (354kB/s-362kB/s), io=23.4MiB (24.6MB), run=17079-17261msec WRITE: bw=1456KiB/s (1491kB/s), 364KiB/s-369KiB/s (372kB/s-378kB/s), io=24.6MiB (25.7MB), run=17079-17261msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28469: Tue Dec 17 07:35:19 2024 read: IOPS=62, BW=1002KiB/s (1026kB/s)(5920KiB/5907msec) clat (usec): min=58, max=46689, avg=11482.85, stdev=4218.45 lat (usec): min=59, max=46690, avg=11483.53, stdev=4218.46 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 127], 10.00th=[ 8455], 20.00th=[ 9372], | 30.00th=[10159], 40.00th=[10945], 50.00th=[11600], 60.00th=[12256], | 70.00th=[13042], 80.00th=[13960], 90.00th=[15401], 95.00th=[16450], | 99.00th=[21890], 99.50th=[22676], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 800, max= 1120, per=24.87%, avg=968.55, stdev=101.30, samples=11 iops : min= 50, max= 70, avg=60.45, stdev= 6.35, samples=11 write: IOPS=67, BW=1078KiB/s (1104kB/s)(6368KiB/5907msec) clat (usec): min=661, max=23032, avg=3989.75, stdev=3742.69 lat (usec): min=661, max=23033, avg=3991.63, stdev=3743.45 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 832], 10.00th=[ 881], 20.00th=[ 1074], | 30.00th=[ 1860], 40.00th=[ 2540], 50.00th=[ 2835], 60.00th=[ 3392], | 70.00th=[ 3949], 80.00th=[ 5473], 90.00th=[ 8848], 95.00th=[13173], | 99.00th=[17171], 99.50th=[20317], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 670, max= 1280, per=25.71%, avg=1052.91, stdev=215.23, samples=11 iops : min= 41, max= 80, avg=65.73, stdev=13.61, samples=11 lat (usec) : 100=1.82%, 250=0.78%, 750=0.26%, 1000=8.72% lat (msec) : 2=6.77%, 4=20.83%, 10=21.35%, 20=38.28%, 50=1.17% cpu : usr=0.08%, sys=19.84%, ctx=2586, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28470: Tue Dec 17 07:35:19 2024 read: IOPS=59, BW=951KiB/s (973kB/s)(5744KiB/6042msec) clat (usec): min=52, max=31080, avg=11448.97, stdev=4314.52 lat (usec): min=52, max=31080, avg=11449.76, stdev=4315.01 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 93], 10.00th=[ 7963], 20.00th=[ 9372], | 30.00th=[10159], 40.00th=[10945], 50.00th=[11600], 60.00th=[12518], | 70.00th=[13173], 80.00th=[13960], 90.00th=[15401], 95.00th=[17171], | 99.00th=[22152], 99.50th=[27132], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 480, max= 1376, per=24.38%, avg=949.17, stdev=199.72, samples=12 iops : min= 30, max= 86, avg=59.25, stdev=12.47, samples=12 write: IOPS=67, BW=1083KiB/s (1109kB/s)(6544KiB/6042msec) clat (usec): min=724, max=54061, avg=4532.73, stdev=4910.49 lat (usec): min=725, max=54062, avg=4534.78, stdev=4909.80 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 832], 10.00th=[ 906], 20.00th=[ 1012], | 30.00th=[ 1450], 40.00th=[ 2671], 50.00th=[ 3064], 60.00th=[ 3654], | 70.00th=[ 4948], 80.00th=[ 6521], 90.00th=[10683], 95.00th=[14222], | 99.00th=[20579], 99.50th=[21103], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 768, max= 1696, per=26.49%, avg=1085.17, stdev=318.82, samples=12 iops : min= 48, max= 106, avg=67.75, stdev=19.99, samples=12 lat (usec) : 100=2.73%, 250=0.39%, 750=0.13%, 1000=10.03% lat (msec) : 2=6.25%, 4=18.10%, 10=23.44%, 20=36.85%, 50=1.95% lat (msec) : 100=0.13% cpu : usr=0.03%, sys=19.50%, ctx=2503, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28471: Tue Dec 17 07:35:19 2024 read: IOPS=63, BW=1017KiB/s (1041kB/s)(6256KiB/6151msec) clat (usec): min=57, max=29464, avg=11552.86, stdev=4144.12 lat (usec): min=58, max=29464, avg=11554.70, stdev=4144.08 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 112], 10.00th=[ 8029], 20.00th=[ 9372], | 30.00th=[10421], 40.00th=[11076], 50.00th=[11731], 60.00th=[12387], | 70.00th=[13173], 80.00th=[14091], 90.00th=[15664], 95.00th=[17433], | 99.00th=[24511], 99.50th=[25297], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 704, max= 1504, per=25.88%, avg=1007.83, stdev=202.63, samples=12 iops : min= 44, max= 94, avg=62.92, stdev=12.67, samples=12 write: IOPS=61, BW=981KiB/s (1004kB/s)(6032KiB/6151msec) clat (usec): min=648, max=22735, avg=4285.18, stdev=4082.96 lat (usec): min=649, max=22737, avg=4286.28, stdev=4082.96 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 816], 10.00th=[ 898], 20.00th=[ 1123], | 30.00th=[ 1631], 40.00th=[ 2540], 50.00th=[ 2900], 60.00th=[ 3654], | 70.00th=[ 4817], 80.00th=[ 6456], 90.00th=[ 8848], 95.00th=[12780], | 99.00th=[20317], 99.50th=[21627], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 576, max= 1248, per=23.17%, avg=949.17, stdev=210.61, samples=12 iops : min= 36, max= 78, avg=59.25, stdev=13.15, samples=12 lat (usec) : 100=2.47%, 250=0.26%, 500=0.13%, 750=0.78%, 1000=7.16% lat (msec) : 2=7.42%, 4=16.41%, 10=23.83%, 20=39.97%, 50=1.56% cpu : usr=0.07%, sys=19.38%, ctx=2639, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28472: Tue Dec 17 07:35:19 2024 read: IOPS=62, BW=999KiB/s (1023kB/s)(6032KiB/6037msec) clat (usec): min=54, max=23106, avg=11376.90, stdev=3922.27 lat (usec): min=55, max=23107, avg=11377.60, stdev=3922.29 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 118], 10.00th=[ 8160], 20.00th=[ 9372], | 30.00th=[10421], 40.00th=[10945], 50.00th=[11469], 60.00th=[12256], | 70.00th=[13173], 80.00th=[14091], 90.00th=[15270], 95.00th=[16909], | 99.00th=[20579], 99.50th=[22414], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 672, max= 1312, per=25.39%, avg=988.58, stdev=169.90, samples=12 iops : min= 42, max= 82, avg=61.58, stdev=10.66, samples=12 write: IOPS=64, BW=1036KiB/s (1061kB/s)(6256KiB/6037msec) clat (usec): min=663, max=49366, avg=4248.20, stdev=4565.40 lat (usec): min=664, max=49368, avg=4249.26, stdev=4565.41 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 824], 10.00th=[ 881], 20.00th=[ 1090], | 30.00th=[ 1532], 40.00th=[ 2606], 50.00th=[ 3032], 60.00th=[ 3425], | 70.00th=[ 4293], 80.00th=[ 5932], 90.00th=[10159], 95.00th=[13566], | 99.00th=[19006], 99.50th=[25560], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 768, max= 1376, per=25.31%, avg=1036.50, stdev=168.04, samples=12 iops : min= 48, max= 86, avg=64.58, stdev=10.51, samples=12 lat (usec) : 100=2.08%, 250=0.78%, 750=0.39%, 1000=8.07% lat (msec) : 2=8.33%, 4=18.36%, 10=19.92%, 20=41.15%, 50=0.91% cpu : usr=0.17%, sys=19.07%, ctx=2511, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3894KiB/s (3987kB/s), 951KiB/s-1017KiB/s (973kB/s-1041kB/s), io=23.4MiB (24.5MB), run=5907-6151msec WRITE: bw=4097KiB/s (4195kB/s), 981KiB/s-1083KiB/s (1004kB/s-1109kB/s), io=24.6MiB (25.8MB), run=5907-6151msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28465: Tue Dec 17 07:35:19 2024 read: IOPS=70, BW=1122KiB/s (1149kB/s)(5920KiB/5274msec) clat (usec): min=1816, max=21396, avg=5600.89, stdev=4158.49 lat (usec): min=1817, max=21397, avg=5601.53, stdev=4158.50 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2278], 10.00th=[ 2442], 20.00th=[ 2704], | 30.00th=[ 2933], 40.00th=[ 3261], 50.00th=[ 3621], 60.00th=[ 4359], | 70.00th=[ 5800], 80.00th=[ 9110], 90.00th=[11731], 95.00th=[15139], | 99.00th=[19006], 99.50th=[21365], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 864, max= 1408, per=29.05%, avg=1116.50, stdev=193.11, samples=10 iops : min= 54, max= 88, avg=69.70, stdev=11.98, samples=10 write: IOPS=75, BW=1207KiB/s (1236kB/s)(6368KiB/5274msec) clat (usec): min=2547, max=64156, avg=8010.50, stdev=5516.70 lat (usec): min=2548, max=64157, avg=8011.56, stdev=5516.74 clat percentiles (usec): | 1.00th=[ 3228], 5.00th=[ 3687], 10.00th=[ 3884], 20.00th=[ 4293], | 30.00th=[ 4817], 40.00th=[ 5407], 50.00th=[ 6456], 60.00th=[ 7439], | 70.00th=[ 8455], 80.00th=[10945], 90.00th=[14222], 95.00th=[17171], | 99.00th=[27657], 99.50th=[31851], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 992, max= 1472, per=29.75%, avg=1202.90, stdev=165.75, samples=10 iops : min= 62, max= 92, avg=75.10, stdev=10.46, samples=10 lat (msec) : 2=0.39%, 4=32.16%, 10=47.01%, 20=18.49%, 50=1.82% lat (msec) : 100=0.13% cpu : usr=0.09%, sys=10.01%, ctx=900, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28466: Tue Dec 17 07:35:19 2024 read: IOPS=57, BW=926KiB/s (948kB/s)(5744KiB/6203msec) clat (usec): min=1777, max=32710, avg=6212.27, stdev=4705.00 lat (usec): min=1778, max=32710, avg=6212.93, stdev=4705.02 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2311], 10.00th=[ 2409], 20.00th=[ 2671], | 30.00th=[ 2933], 40.00th=[ 3392], 50.00th=[ 3982], 60.00th=[ 5342], | 70.00th=[ 7242], 80.00th=[10028], 90.00th=[12649], 95.00th=[15926], | 99.00th=[22414], 99.50th=[24249], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 447, max= 1190, per=23.80%, avg=914.92, stdev=189.80, samples=12 iops : min= 27, max= 74, avg=57.00, stdev=12.02, samples=12 write: IOPS=65, BW=1055KiB/s (1080kB/s)(6544KiB/6203msec) clat (usec): min=2709, max=50519, avg=9591.26, stdev=5859.92 lat (usec): min=2710, max=50520, avg=9592.37, stdev=5859.95 clat percentiles (usec): | 1.00th=[ 3130], 5.00th=[ 3523], 10.00th=[ 4015], 20.00th=[ 4686], | 30.00th=[ 5538], 40.00th=[ 6915], 50.00th=[ 8029], 60.00th=[ 9110], | 70.00th=[11207], 80.00th=[14222], 90.00th=[18482], 95.00th=[20317], | 99.00th=[27132], 99.50th=[31327], 99.90th=[50594], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 734, max= 1475, per=25.72%, avg=1039.92, stdev=252.67, samples=12 iops : min= 45, max= 92, avg=64.83, stdev=15.95, samples=12 lat (msec) : 2=0.52%, 4=28.26%, 10=42.84%, 20=24.61%, 50=3.65% lat (msec) : 100=0.13% cpu : usr=0.05%, sys=10.01%, ctx=1107, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28467: Tue Dec 17 07:35:19 2024 read: IOPS=62, BW=1004KiB/s (1028kB/s)(6256KiB/6230msec) clat (usec): min=2023, max=34767, avg=6418.56, stdev=4716.74 lat (usec): min=2024, max=34767, avg=6419.28, stdev=4716.82 clat percentiles (usec): | 1.00th=[ 2089], 5.00th=[ 2245], 10.00th=[ 2474], 20.00th=[ 2704], | 30.00th=[ 3097], 40.00th=[ 3523], 50.00th=[ 4178], 60.00th=[ 5800], | 70.00th=[ 7898], 80.00th=[10421], 90.00th=[13042], 95.00th=[15795], | 99.00th=[21890], 99.50th=[22938], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 672, max= 1501, per=25.94%, avg=997.08, stdev=209.77, samples=12 iops : min= 42, max= 93, avg=62.25, stdev=12.93, samples=12 write: IOPS=60, BW=968KiB/s (991kB/s)(6032KiB/6230msec) clat (usec): min=2921, max=73687, avg=9677.51, stdev=6161.65 lat (usec): min=2922, max=73689, avg=9678.86, stdev=6161.62 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 3490], 10.00th=[ 4047], 20.00th=[ 4817], | 30.00th=[ 5997], 40.00th=[ 7111], 50.00th=[ 8356], 60.00th=[ 9765], | 70.00th=[10945], 80.00th=[13173], 90.00th=[17433], 95.00th=[20317], | 99.00th=[26608], 99.50th=[29492], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 608, max= 1280, per=23.56%, avg=952.92, stdev=190.52, samples=12 iops : min= 38, max= 80, avg=59.50, stdev=11.88, samples=12 lat (msec) : 4=28.91%, 10=40.62%, 20=27.08%, 50=3.26%, 100=0.13% cpu : usr=0.18%, sys=9.95%, ctx=1107, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28468: Tue Dec 17 07:35:19 2024 read: IOPS=69, BW=1116KiB/s (1143kB/s)(6032KiB/5404msec) clat (usec): min=1907, max=21575, avg=5576.60, stdev=3814.44 lat (usec): min=1908, max=21576, avg=5577.22, stdev=3814.45 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2311], 10.00th=[ 2474], 20.00th=[ 2737], | 30.00th=[ 3097], 40.00th=[ 3425], 50.00th=[ 3916], 60.00th=[ 4686], | 70.00th=[ 6128], 80.00th=[ 8717], 90.00th=[11076], 95.00th=[13042], | 99.00th=[19006], 99.50th=[20055], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 832, max= 1437, per=28.55%, avg=1097.30, stdev=176.60, samples=10 iops : min= 52, max= 89, avg=68.50, stdev=10.87, samples=10 write: IOPS=72, BW=1158KiB/s (1185kB/s)(6256KiB/5404msec) clat (usec): min=2892, max=26853, avg=8273.89, stdev=4595.85 lat (usec): min=2893, max=26854, avg=8275.12, stdev=4596.10 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 3687], 10.00th=[ 3916], 20.00th=[ 4555], | 30.00th=[ 5014], 40.00th=[ 5800], 50.00th=[ 6980], 60.00th=[ 7898], | 70.00th=[ 9634], 80.00th=[11469], 90.00th=[15401], 95.00th=[17433], | 99.00th=[24249], 99.50th=[26346], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 928, max= 1408, per=28.48%, avg=1151.70, stdev=159.07, samples=10 iops : min= 58, max= 88, avg=71.90, stdev= 9.98, samples=10 lat (msec) : 2=0.26%, 4=30.86%, 10=47.40%, 20=19.92%, 50=1.56% cpu : usr=0.11%, sys=9.98%, ctx=935, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3845KiB/s (3937kB/s), 926KiB/s-1122KiB/s (948kB/s-1149kB/s), io=23.4MiB (24.5MB), run=5274-6230msec WRITE: bw=4045KiB/s (4142kB/s), 968KiB/s-1207KiB/s (991kB/s-1236kB/s), io=24.6MiB (25.8MB), run=5274-6230msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28483: Tue Dec 17 07:35:22 2024 read: IOPS=3, BW=3435KiB/s (3517kB/s)(7168KiB/2087msec) clat (msec): min=13, max=834, avg=249.62, stdev=327.39 lat (msec): min=13, max=834, avg=249.62, stdev=327.39 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 25], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 64], 60.00th=[ 174], | 70.00th=[ 174], 80.00th=[ 592], 90.00th=[ 835], 95.00th=[ 835], | 99.00th=[ 835], 99.50th=[ 835], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 1809, max= 5417, per=27.63%, avg=2711.00, stdev=1804.00, samples=4 iops : min= 1, max= 5, avg= 2.00, stdev= 2.00, samples=4 write: IOPS=2, BW=2453KiB/s (2512kB/s)(5120KiB/2087msec) clat (usec): min=15259, max=79833, avg=55025.83, stdev=27199.00 lat (usec): min=15288, max=79861, avg=55070.45, stdev=27208.87 clat percentiles (usec): | 1.00th=[15270], 5.00th=[15270], 10.00th=[15270], 20.00th=[15270], | 30.00th=[39060], 40.00th=[39060], 50.00th=[66323], 60.00th=[66323], | 70.00th=[74974], 80.00th=[74974], 90.00th=[80217], 95.00th=[80217], | 99.00th=[80217], 99.50th=[80217], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 7223, max= 7223, per=52.58%, avg=7223.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=16.67%, 100=41.67%, 250=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=5.85%, ctx=270, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28484: Tue Dec 17 07:35:22 2024 read: IOPS=1, BW=1210KiB/s (1239kB/s)(2048KiB/1693msec) clat (msec): min=24, max=838, avg=431.72, stdev=575.87 lat (msec): min=24, max=838, avg=431.72, stdev=575.87 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 25], 40.00th=[ 25], 50.00th=[ 25], 60.00th=[ 844], | 70.00th=[ 844], 80.00th=[ 844], 90.00th=[ 844], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 1809, max= 1809, per=18.43%, avg=1809.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=5, BW=6048KiB/s (6194kB/s)(10.0MiB/1693msec) clat (msec): min=14, max=386, avg=81.49, stdev=110.73 lat (msec): min=14, max=386, avg=81.55, stdev=110.73 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 40], 50.00th=[ 42], 60.00th=[ 53], | 70.00th=[ 83], 80.00th=[ 83], 90.00th=[ 96], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 5427, max= 9045, per=52.67%, avg=7236.00, stdev=2558.31, samples=2 iops : min= 5, max= 8, avg= 6.50, stdev= 2.12, samples=2 lat (msec) : 20=25.00%, 50=25.00%, 100=33.33%, 500=8.33%, 1000=8.33% cpu : usr=0.18%, sys=9.99%, ctx=68, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28485: Tue Dec 17 07:35:22 2024 read: IOPS=17, BW=17.7MiB/s (18.6MB/s)(6144KiB/339msec) clat (usec): min=13223, max=24289, avg=17006.22, stdev=3959.04 lat (usec): min=13224, max=24300, avg=17008.79, stdev=3962.55 clat percentiles (usec): | 1.00th=[13173], 5.00th=[13173], 10.00th=[13173], 20.00th=[14222], | 30.00th=[14222], 40.00th=[15664], 50.00th=[15664], 60.00th=[16581], | 70.00th=[17957], 80.00th=[17957], 90.00th=[24249], 95.00th=[24249], | 99.00th=[24249], 99.50th=[24249], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] write: IOPS=17, BW=17.7MiB/s (18.6MB/s)(6144KiB/339msec) clat (usec): min=18806, max=46440, avg=33246.46, stdev=12507.85 lat (usec): min=18846, max=46653, avg=33338.80, stdev=12535.73 clat percentiles (usec): | 1.00th=[18744], 5.00th=[18744], 10.00th=[18744], 20.00th=[20055], | 30.00th=[20055], 40.00th=[28181], 50.00th=[28181], 60.00th=[40633], | 70.00th=[45351], 80.00th=[45351], 90.00th=[46400], 95.00th=[46400], | 99.00th=[46400], 99.50th=[46400], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] lat (msec) : 20=50.00%, 50=50.00% cpu : usr=0.00%, sys=24.85%, ctx=57, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28487: Tue Dec 17 07:35:22 2024 read: IOPS=13, BW=13.3MiB/s (13.0MB/s)(5120KiB/375msec) clat (usec): min=13180, max=49129, avg=26006.53, stdev=14733.02 lat (usec): min=13181, max=49130, avg=26007.68, stdev=14733.07 clat percentiles (usec): | 1.00th=[13173], 5.00th=[13173], 10.00th=[13173], 20.00th=[13173], | 30.00th=[13435], 40.00th=[13435], 50.00th=[25035], 60.00th=[25035], | 70.00th=[29230], 80.00th=[29230], 90.00th=[49021], 95.00th=[49021], | 99.00th=[49021], 99.50th=[49021], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] write: IOPS=18, BW=18.7MiB/s (19.6MB/s)(7168KiB/375msec) clat (usec): min=16180, max=58125, avg=26326.72, stdev=14736.28 lat (usec): min=16247, max=58192, avg=26394.64, stdev=14735.83 clat percentiles (usec): | 1.00th=[16188], 5.00th=[16188], 10.00th=[16188], 20.00th=[16909], | 30.00th=[18744], 40.00th=[18744], 50.00th=[20841], 60.00th=[24249], | 70.00th=[24249], 80.00th=[29230], 90.00th=[57934], 95.00th=[57934], | 99.00th=[57934], 99.50th=[57934], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] lat (msec) : 20=41.67%, 50=50.00%, 100=8.33% cpu : usr=0.00%, sys=17.65%, ctx=41, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9813KiB/s (10.0MB/s), 1210KiB/s-17.7MiB/s (1239kB/s-18.6MB/s), io=20.0MiB (20.0MB), run=339-2087msec WRITE: bw=13.4MiB/s (14.1MB/s), 2453KiB/s-18.7MiB/s (2512kB/s-19.6MB/s), io=28.0MiB (29.4MB), run=339-2087msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28486: Tue Dec 17 07:35:23 2024 read: IOPS=3, BW=3609KiB/s (3696kB/s)(7168KiB/1986msec) clat (usec): min=391, max=828445, avg=246653.34, stdev=333324.24 lat (usec): min=392, max=828446, avg=246654.77, stdev=333324.42 clat percentiles (usec): | 1.00th=[ 392], 5.00th=[ 392], 10.00th=[ 392], 20.00th=[ 408], | 30.00th=[ 627], 40.00th=[ 627], 50.00th=[ 90702], 60.00th=[210764], | 70.00th=[210764], 80.00th=[591397], 90.00th=[826278], 95.00th=[826278], | 99.00th=[826278], 99.50th=[826278], 99.90th=[826278], 99.95th=[826278], | 99.99th=[826278] bw ( KiB/s): min= 2048, max= 2048, per=26.18%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2578KiB/s (2640kB/s)(5120KiB/1986msec) clat (msec): min=19, max=131, avg=49.38, stdev=46.32 lat (msec): min=19, max=131, avg=49.44, stdev=46.34 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 20], | 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 36], 60.00th=[ 36], | 70.00th=[ 39], 80.00th=[ 39], 90.00th=[ 131], 95.00th=[ 131], | 99.00th=[ 131], 99.50th=[ 131], 99.90th=[ 131], 99.95th=[ 131], | 99.99th=[ 131] lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 20=8.33%, 50=25.00%, 100=8.33%, 250=16.67%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.10%, sys=22.92%, ctx=774, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28488: Tue Dec 17 07:35:23 2024 read: IOPS=1, BW=1245KiB/s (1275kB/s)(2048KiB/1645msec) clat (msec): min=50, max=589, avg=320.35, stdev=381.31 lat (msec): min=50, max=589, avg=320.35, stdev=381.31 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 51], 10.00th=[ 51], 20.00th=[ 51], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 592], | 70.00th=[ 592], 80.00th=[ 592], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 2048, max= 2048, per=26.18%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=6, BW=6225KiB/s (6374kB/s)(10.0MiB/1645msec) clat (msec): min=27, max=538, avg=93.31, stdev=157.15 lat (msec): min=27, max=538, avg=93.38, stdev=157.15 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 29], | 30.00th=[ 36], 40.00th=[ 41], 50.00th=[ 41], 60.00th=[ 42], | 70.00th=[ 58], 80.00th=[ 58], 90.00th=[ 83], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 2048, max=10240, per=49.87%, avg=5461.33, stdev=4263.25, samples=3 iops : min= 2, max= 10, avg= 5.33, stdev= 4.16, samples=3 lat (msec) : 50=58.33%, 100=25.00%, 750=16.67% cpu : usr=0.00%, sys=22.51%, ctx=295, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28489: Tue Dec 17 07:35:23 2024 read: IOPS=2, BW=2347KiB/s (2403kB/s)(6144KiB/2618msec) clat (usec): min=431, max=680797, avg=246145.89, stdev=301184.90 lat (usec): min=432, max=680798, avg=246146.94, stdev=301184.77 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 433], 10.00th=[ 433], 20.00th=[ 461], | 30.00th=[ 461], 40.00th=[ 55313], 50.00th=[ 55313], 60.00th=[173016], | 70.00th=[566232], 80.00th=[566232], 90.00th=[683672], 95.00th=[683672], | 99.00th=[683672], 99.50th=[683672], 99.90th=[683672], 99.95th=[683672], | 99.99th=[683672] bw ( KiB/s): min= 4096, max= 6131, per=65.37%, avg=5113.50, stdev=1438.96, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2347KiB/s (2403kB/s)(6144KiB/2618msec) clat (msec): min=27, max=895, avg=180.28, stdev=350.60 lat (msec): min=28, max=895, avg=180.34, stdev=350.61 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 30], | 30.00th=[ 30], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 41], | 70.00th=[ 58], 80.00th=[ 58], 90.00th=[ 894], 95.00th=[ 894], | 99.00th=[ 894], 99.50th=[ 894], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 4087, max= 4096, per=37.38%, avg=4093.00, stdev= 5.20, samples=3 iops : min= 3, max= 4, avg= 3.67, stdev= 0.58, samples=3 lat (usec) : 500=16.67% lat (msec) : 50=33.33%, 100=16.67%, 250=8.33%, 750=16.67%, 1000=8.33% cpu : usr=0.00%, sys=15.13%, ctx=683, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28490: Tue Dec 17 07:35:23 2024 read: IOPS=1, BW=1959KiB/s (2006kB/s)(5120KiB/2614msec) clat (usec): min=483, max=580534, avg=283938.74, stdev=284304.14 lat (usec): min=484, max=580534, avg=283939.59, stdev=284303.79 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 482], 10.00th=[ 482], 20.00th=[ 482], | 30.00th=[ 506], 40.00th=[ 506], 50.00th=[278922], 60.00th=[278922], | 70.00th=[557843], 80.00th=[557843], 90.00th=[583009], 95.00th=[583009], | 99.00th=[583009], 99.50th=[583009], 99.90th=[583009], 99.95th=[583009], | 99.99th=[583009] bw ( KiB/s): min= 2048, max= 6144, per=52.37%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=2, BW=2742KiB/s (2808kB/s)(7168KiB/2614msec) clat (msec): min=24, max=893, avg=159.67, stdev=324.40 lat (msec): min=24, max=893, avg=159.75, stdev=324.39 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 26], | 30.00th=[ 26], 40.00th=[ 26], 50.00th=[ 30], 60.00th=[ 37], | 70.00th=[ 37], 80.00th=[ 83], 90.00th=[ 894], 95.00th=[ 894], | 99.00th=[ 894], 99.50th=[ 894], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 2048, max= 6144, per=43.64%, avg=4778.67, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 4.67, stdev= 2.31, samples=3 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=41.67%, 100=8.33%, 500=8.33%, 750=16.67%, 1000=8.33% cpu : usr=0.08%, sys=16.65%, ctx=651, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7823KiB/s (8011kB/s), 1245KiB/s-3609KiB/s (1275kB/s-3696kB/s), io=20.0MiB (20.0MB), run=1645-2618msec WRITE: bw=10.7MiB/s (11.2MB/s), 2347KiB/s-6225KiB/s (2403kB/s-6374kB/s), io=28.0MiB (29.4MB), run=1645-2618msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28498: Tue Dec 17 07:35:26 2024 read: IOPS=1, BW=6437KiB/s (6591kB/s)(12.0MiB/1909msec) clat (msec): min=58, max=1034, avg=612.02, stdev=501.27 lat (msec): min=58, max=1034, avg=612.02, stdev=501.27 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 58], 10.00th=[ 58], 20.00th=[ 58], | 30.00th=[ 58], 40.00th=[ 743], 50.00th=[ 743], 60.00th=[ 743], | 70.00th=[ 1036], 80.00th=[ 1036], 90.00th=[ 1036], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 8192, max= 8192, per=95.46%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.05%, sys=51.10%, ctx=706, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28500: Tue Dec 17 07:35:26 2024 read: IOPS=0, BW=2186KiB/s (2238kB/s)(4096KiB/1874msec) clat (nsec): min=145275k, max=145275k, avg=145274507.00, stdev= 0.00 lat (nsec): min=145288k, max=145288k, avg=145287842.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 146], 10.00th=[ 146], 20.00th=[ 146], | 30.00th=[ 146], 40.00th=[ 146], 50.00th=[ 146], 60.00th=[ 146], | 70.00th=[ 146], 80.00th=[ 146], 90.00th=[ 146], 95.00th=[ 146], | 99.00th=[ 146], 99.50th=[ 146], 99.90th=[ 146], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 8175, max= 8175, per=95.26%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4371KiB/s (4476kB/s)(8192KiB/1874msec) clat (msec): min=375, max=1335, avg=855.63, stdev=678.31 lat (msec): min=376, max=1335, avg=855.82, stdev=678.24 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 376], 10.00th=[ 376], 20.00th=[ 376], | 30.00th=[ 376], 40.00th=[ 376], 50.00th=[ 376], 60.00th=[ 1334], | 70.00th=[ 1334], 80.00th=[ 1334], 90.00th=[ 1334], 95.00th=[ 1334], | 99.00th=[ 1334], 99.50th=[ 1334], 99.90th=[ 1334], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 8192, max= 8192, per=48.78%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=18.42%, ctx=54, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28502: Tue Dec 17 07:35:26 2024 write: IOPS=1, BW=6298KiB/s (6449kB/s)(12.0MiB/1951msec) clat (msec): min=248, max=1299, avg=640.76, stdev=574.22 lat (msec): min=249, max=1300, avg=641.06, stdev=574.22 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 249], 10.00th=[ 249], 20.00th=[ 249], | 30.00th=[ 249], 40.00th=[ 376], 50.00th=[ 376], 60.00th=[ 376], | 70.00th=[ 1301], 80.00th=[ 1301], 90.00th=[ 1301], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 8192, max= 8192, per=48.78%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=16.56%, ctx=106, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28504: Tue Dec 17 07:35:26 2024 write: IOPS=1, BW=8010KiB/s (8203kB/s)(12.0MiB/1534msec) clat (msec): min=145, max=1065, avg=492.31, stdev=499.86 lat (msec): min=145, max=1065, avg=492.78, stdev=499.69 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 146], 10.00th=[ 146], 20.00th=[ 146], | 30.00th=[ 146], 40.00th=[ 268], 50.00th=[ 268], 60.00th=[ 268], | 70.00th=[ 1062], 80.00th=[ 1062], 90.00th=[ 1062], 95.00th=[ 1062], | 99.00th=[ 1062], 99.50th=[ 1062], 99.90th=[ 1062], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 8192, max= 8192, per=48.78%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=21.92%, ctx=42, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8583KiB/s (8788kB/s), 2186KiB/s-6437KiB/s (2238kB/s-6591kB/s), io=16.0MiB (16.8MB), run=1874-1909msec WRITE: bw=16.4MiB/s (17.2MB/s), 4371KiB/s-8010KiB/s (4476kB/s-8203kB/s), io=32.0MiB (33.6MB), run=1534-1951msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28499: Tue Dec 17 07:35:26 2024 read: IOPS=1, BW=6200KiB/s (6349kB/s)(12.0MiB/1982msec) clat (msec): min=276, max=1292, avg=627.65, stdev=576.00 lat (msec): min=276, max=1292, avg=627.65, stdev=576.00 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 275], 10.00th=[ 275], 20.00th=[ 275], | 30.00th=[ 275], 40.00th=[ 313], 50.00th=[ 313], 60.00th=[ 313], | 70.00th=[ 1301], 80.00th=[ 1301], 90.00th=[ 1301], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 8192, max= 8192, per=99.10%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=3.23%, ctx=35, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28501: Tue Dec 17 07:35:26 2024 read: IOPS=0, BW=2071KiB/s (2120kB/s)(4096KiB/1978msec) clat (nsec): min=309636k, max=309636k, avg=309635577.00, stdev= 0.00 lat (nsec): min=309638k, max=309638k, avg=309637942.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 309], 10.00th=[ 309], 20.00th=[ 309], | 30.00th=[ 309], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 8192, max= 8192, per=99.10%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4142KiB/s (4241kB/s)(8192KiB/1978msec) clat (msec): min=240, max=1353, avg=797.26, stdev=787.29 lat (msec): min=240, max=1354, avg=797.61, stdev=787.20 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 241], 10.00th=[ 241], 20.00th=[ 241], | 30.00th=[ 241], 40.00th=[ 241], 50.00th=[ 241], 60.00th=[ 1351], | 70.00th=[ 1351], 80.00th=[ 1351], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 8175, max= 8175, per=49.82%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=6.17%, ctx=83, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28503: Tue Dec 17 07:35:26 2024 write: IOPS=1, BW=6153KiB/s (6301kB/s)(12.0MiB/1997msec) clat (msec): min=275, max=1290, avg=637.21, stdev=566.50 lat (msec): min=275, max=1290, avg=637.44, stdev=566.41 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 275], 10.00th=[ 275], 20.00th=[ 275], | 30.00th=[ 275], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 347], | 70.00th=[ 1284], 80.00th=[ 1284], 90.00th=[ 1284], 95.00th=[ 1284], | 99.00th=[ 1284], 99.50th=[ 1284], 99.90th=[ 1284], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 8192, max= 8192, per=49.93%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.05%, sys=2.81%, ctx=21, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28505: Tue Dec 17 07:35:26 2024 write: IOPS=1, BW=7018KiB/s (7186kB/s)(12.0MiB/1751msec) clat (msec): min=106, max=1272, avg=556.06, stdev=626.76 lat (msec): min=106, max=1272, avg=556.36, stdev=626.79 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 292], 50.00th=[ 292], 60.00th=[ 292], | 70.00th=[ 1267], 80.00th=[ 1267], 90.00th=[ 1267], 95.00th=[ 1267], | 99.00th=[ 1267], 99.50th=[ 1267], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 8192, max= 8192, per=49.93%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.11%, sys=10.46%, ctx=209, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8266KiB/s (8465kB/s), 2071KiB/s-6200KiB/s (2120kB/s-6349kB/s), io=16.0MiB (16.8MB), run=1978-1982msec WRITE: bw=16.0MiB/s (16.8MB/s), 4142KiB/s-7018KiB/s (4241kB/s-7186kB/s), io=32.0MiB (33.6MB), run=1751-1997msec