== sanity test 398b: DIO and buffer IO race ============== 12:27:45 (1729096065) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.21393 s, 41.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27885: Wed Oct 16 12:28:00 2024 read: IOPS=116, BW=464KiB/s (475kB/s)(5992KiB/12910msec) clat (usec): min=48, max=156643, avg=3978.72, stdev=4980.06 lat (usec): min=48, max=156644, avg=3979.42, stdev=4980.07 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 61], 10.00th=[ 73], 20.00th=[ 2057], | 30.00th=[ 2409], 40.00th=[ 2802], 50.00th=[ 3261], 60.00th=[ 3982], | 70.00th=[ 4883], 80.00th=[ 6063], 90.00th=[ 7767], 95.00th=[ 9241], | 99.00th=[ 12518], 99.50th=[ 16581], 99.90th=[ 32637], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 304, max= 744, per=24.25%, avg=450.84, stdev=89.76, samples=25 iops : min= 76, max= 186, avg=112.68, stdev=22.46, samples=25 write: IOPS=121, BW=488KiB/s (499kB/s)(6296KiB/12910msec) clat (usec): min=414, max=48733, avg=4350.38, stdev=4536.75 lat (usec): min=415, max=48733, avg=4351.49, stdev=4537.25 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 529], 10.00th=[ 562], 20.00th=[ 660], | 30.00th=[ 865], 40.00th=[ 2212], 50.00th=[ 3359], 60.00th=[ 4359], | 70.00th=[ 5276], 80.00th=[ 7046], 90.00th=[ 9896], 95.00th=[12911], | 99.00th=[19268], 99.50th=[23462], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 352, max= 752, per=24.32%, avg=473.56, stdev=103.36, samples=25 iops : min= 88, max= 188, avg=118.36, stdev=25.86, samples=25 lat (usec) : 50=0.16%, 100=6.90%, 250=1.37%, 500=1.01%, 750=12.30% lat (usec) : 1000=3.71% lat (msec) : 2=3.91%, 4=28.97%, 10=34.99%, 20=6.15%, 50=0.49% lat (msec) : 250=0.03% cpu : usr=0.27%, sys=23.18%, ctx=6201, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27886: Wed Oct 16 12:28:00 2024 read: IOPS=116, BW=465KiB/s (476kB/s)(5940KiB/12769msec) clat (usec): min=49, max=31531, avg=3910.78, stdev=3265.39 lat (usec): min=49, max=31532, avg=3913.69, stdev=3264.83 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 62], 10.00th=[ 75], 20.00th=[ 2008], | 30.00th=[ 2409], 40.00th=[ 2769], 50.00th=[ 3163], 60.00th=[ 3916], | 70.00th=[ 4883], 80.00th=[ 5997], 90.00th=[ 7701], 95.00th=[ 9110], | 99.00th=[15926], 99.50th=[19268], 99.90th=[30278], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 320, max= 624, per=24.82%, avg=461.44, stdev=84.38, samples=25 iops : min= 80, max= 156, avg=115.20, stdev=21.12, samples=25 write: IOPS=124, BW=497KiB/s (509kB/s)(6348KiB/12769msec) clat (usec): min=434, max=156914, avg=4320.23, stdev=5689.39 lat (usec): min=435, max=156915, avg=4321.10, stdev=5689.34 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 578], 20.00th=[ 660], | 30.00th=[ 889], 40.00th=[ 2024], 50.00th=[ 3294], 60.00th=[ 4293], | 70.00th=[ 5407], 80.00th=[ 6849], 90.00th=[ 9503], 95.00th=[ 12387], | 99.00th=[ 18482], 99.50th=[ 21627], 99.90th=[ 54789], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 352, max= 704, per=25.07%, avg=488.04, stdev=92.24, samples=25 iops : min= 88, max= 176, avg=121.84, stdev=23.10, samples=25 lat (usec) : 50=0.07%, 100=6.41%, 250=1.66%, 500=0.78%, 750=12.43% lat (usec) : 1000=3.91% lat (msec) : 2=5.01%, 4=28.55%, 10=34.93%, 20=5.70%, 50=0.49% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.20%, sys=23.62%, ctx=6208, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27887: Wed Oct 16 12:28:00 2024 read: IOPS=117, BW=471KiB/s (482kB/s)(6028KiB/12809msec) clat (usec): min=48, max=52763, avg=3787.06, stdev=3506.55 lat (usec): min=49, max=52763, avg=3790.03, stdev=3506.98 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 60], 10.00th=[ 72], 20.00th=[ 297], | 30.00th=[ 2311], 40.00th=[ 2638], 50.00th=[ 3097], 60.00th=[ 3720], | 70.00th=[ 4621], 80.00th=[ 5932], 90.00th=[ 7570], 95.00th=[ 9241], | 99.00th=[13304], 99.50th=[24249], 99.90th=[30540], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 200, max= 696, per=24.85%, avg=462.04, stdev=112.19, samples=25 iops : min= 50, max= 174, avg=115.48, stdev=28.08, samples=25 write: IOPS=122, BW=489KiB/s (500kB/s)(6260KiB/12809msec) clat (usec): min=418, max=186147, avg=4469.09, stdev=6336.25 lat (usec): min=419, max=186149, avg=4469.85, stdev=6336.29 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 529], 10.00th=[ 562], 20.00th=[ 652], | 30.00th=[ 906], 40.00th=[ 2180], 50.00th=[ 3523], 60.00th=[ 4359], | 70.00th=[ 5407], 80.00th=[ 7177], 90.00th=[ 10028], 95.00th=[ 12387], | 99.00th=[ 21365], 99.50th=[ 28443], 99.90th=[ 32900], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 296, max= 664, per=24.52%, avg=477.36, stdev=93.39, samples=25 iops : min= 74, max= 166, avg=119.32, stdev=23.34, samples=25 lat (usec) : 50=0.20%, 100=8.04%, 250=1.46%, 500=0.94%, 750=12.11% lat (usec) : 1000=3.42% lat (msec) : 2=5.34%, 4=28.09%, 10=33.33%, 20=6.25%, 50=0.75% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.23%, sys=22.92%, ctx=6213, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27888: Wed Oct 16 12:28:00 2024 read: IOPS=118, BW=473KiB/s (484kB/s)(6052KiB/12793msec) clat (usec): min=46, max=47157, avg=3844.93, stdev=3409.43 lat (usec): min=47, max=47158, avg=3845.51, stdev=3409.44 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 60], 10.00th=[ 71], 20.00th=[ 1729], | 30.00th=[ 2409], 40.00th=[ 2769], 50.00th=[ 3163], 60.00th=[ 3818], | 70.00th=[ 4686], 80.00th=[ 5997], 90.00th=[ 7701], 95.00th=[ 9503], | 99.00th=[12911], 99.50th=[21103], 99.90th=[30278], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 224, max= 664, per=25.12%, avg=466.92, stdev=94.84, samples=25 iops : min= 56, max= 166, avg=116.56, stdev=23.70, samples=25 write: IOPS=121, BW=487KiB/s (499kB/s)(6236KiB/12793msec) clat (usec): min=443, max=179551, avg=4407.10, stdev=6268.29 lat (usec): min=444, max=179553, avg=4409.02, stdev=6267.75 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 537], 10.00th=[ 570], 20.00th=[ 676], | 30.00th=[ 914], 40.00th=[ 2073], 50.00th=[ 3228], 60.00th=[ 4228], | 70.00th=[ 5342], 80.00th=[ 7308], 90.00th=[ 9896], 95.00th=[ 12125], | 99.00th=[ 20841], 99.50th=[ 25560], 99.90th=[ 55313], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 272, max= 712, per=24.70%, avg=480.96, stdev=109.20, samples=25 iops : min= 68, max= 178, avg=120.08, stdev=27.28, samples=25 lat (usec) : 50=0.46%, 100=8.07%, 250=1.11%, 500=0.85%, 750=11.72% lat (usec) : 1000=3.58% lat (msec) : 2=5.14%, 4=29.33%, 10=32.91%, 20=5.96%, 50=0.81% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.11%, sys=23.26%, ctx=6145, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1860KiB/s (1905kB/s), 464KiB/s-473KiB/s (475kB/s-484kB/s), io=23.4MiB (24.6MB), run=12769-12910msec WRITE: bw=1947KiB/s (1994kB/s), 487KiB/s-497KiB/s (499kB/s-509kB/s), io=24.6MiB (25.7MB), run=12769-12910msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27881: Wed Oct 16 12:28:22 2024 read: IOPS=43, BW=173KiB/s (177kB/s)(5992KiB/34589msec) clat (usec): min=1589, max=159220, avg=3577.78, stdev=5367.80 lat (usec): min=1589, max=159221, avg=3578.50, stdev=5367.80 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1811], 10.00th=[ 1876], 20.00th=[ 1942], | 30.00th=[ 2008], 40.00th=[ 2089], 50.00th=[ 2212], 60.00th=[ 2343], | 70.00th=[ 2606], 80.00th=[ 3621], 90.00th=[ 7308], 95.00th=[ 10159], | 99.00th=[ 17695], 99.50th=[ 27395], 99.90th=[ 40633], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 47, max= 336, per=24.98%, avg=173.10, stdev=71.35, samples=69 iops : min= 11, max= 84, avg=43.19, stdev=17.89, samples=69 write: IOPS=45, BW=182KiB/s (186kB/s)(6296KiB/34589msec) clat (usec): min=9719, max=64827, avg=18502.08, stdev=7007.00 lat (usec): min=9719, max=64828, avg=18502.93, stdev=7007.05 clat percentiles (usec): | 1.00th=[10552], 5.00th=[11207], 10.00th=[11600], 20.00th=[12649], | 30.00th=[13960], 40.00th=[15664], 50.00th=[17171], 60.00th=[18482], | 70.00th=[19530], 80.00th=[21890], 90.00th=[28443], 95.00th=[33424], | 99.00th=[41681], 99.50th=[45351], 99.90th=[59507], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 56, max= 240, per=25.06%, avg=181.68, stdev=56.08, samples=69 iops : min= 14, max= 60, avg=45.33, stdev=14.06, samples=69 lat (msec) : 2=13.87%, 4=25.98%, 10=6.48%, 20=39.68%, 50=13.87% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.09%, sys=5.23%, ctx=3444, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27882: Wed Oct 16 12:28:22 2024 read: IOPS=42, BW=171KiB/s (176kB/s)(5940KiB/34648msec) clat (usec): min=1534, max=157046, avg=3469.70, stdev=5262.42 lat (usec): min=1535, max=157047, avg=3470.41, stdev=5262.45 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1827], 10.00th=[ 1860], 20.00th=[ 1942], | 30.00th=[ 2008], 40.00th=[ 2089], 50.00th=[ 2180], 60.00th=[ 2343], | 70.00th=[ 2638], 80.00th=[ 3556], 90.00th=[ 6915], 95.00th=[ 9503], | 99.00th=[ 15270], 99.50th=[ 21627], 99.90th=[ 50070], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 16, max= 336, per=24.74%, avg=171.48, stdev=74.93, samples=69 iops : min= 4, max= 84, avg=42.78, stdev=18.79, samples=69 write: IOPS=45, BW=183KiB/s (188kB/s)(6348KiB/34648msec) clat (usec): min=9731, max=78426, avg=18519.05, stdev=7272.77 lat (usec): min=9732, max=78427, avg=18519.88, stdev=7272.80 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11076], 10.00th=[11600], 20.00th=[12518], | 30.00th=[13960], 40.00th=[15795], 50.00th=[17171], 60.00th=[18482], | 70.00th=[19530], 80.00th=[21627], 90.00th=[28705], 95.00th=[34341], | 99.00th=[42206], 99.50th=[45351], 99.90th=[68682], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 80, max= 248, per=25.20%, avg=182.72, stdev=55.92, samples=69 iops : min= 20, max= 62, avg=45.59, stdev=14.02, samples=69 lat (msec) : 2=13.83%, 4=25.65%, 10=6.84%, 20=39.42%, 50=14.10% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.12%, sys=5.16%, ctx=3440, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27883: Wed Oct 16 12:28:22 2024 read: IOPS=43, BW=175KiB/s (179kB/s)(6028KiB/34481msec) clat (usec): min=1607, max=50326, avg=3446.66, stdev=3274.74 lat (usec): min=1607, max=50327, avg=3447.33, stdev=3274.74 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1827], 10.00th=[ 1876], 20.00th=[ 1958], | 30.00th=[ 2040], 40.00th=[ 2114], 50.00th=[ 2212], 60.00th=[ 2409], | 70.00th=[ 2769], 80.00th=[ 4015], 90.00th=[ 7177], 95.00th=[ 9110], | 99.00th=[15008], 99.50th=[19792], 99.90th=[45876], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 40, max= 384, per=25.23%, avg=174.87, stdev=67.72, samples=68 iops : min= 10, max= 96, avg=43.66, stdev=16.97, samples=68 write: IOPS=45, BW=182KiB/s (186kB/s)(6260KiB/34481msec) clat (msec): min=9, max=177, avg=18.65, stdev= 8.20 lat (msec): min=9, max=177, avg=18.65, stdev= 8.20 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 20], 80.00th=[ 22], 90.00th=[ 30], 95.00th=[ 34], | 99.00th=[ 44], 99.50th=[ 47], 99.90th=[ 79], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 72, max= 264, per=24.96%, avg=180.99, stdev=58.23, samples=68 iops : min= 18, max= 66, avg=45.19, stdev=14.61, samples=68 lat (msec) : 2=12.73%, 4=26.50%, 10=7.78%, 20=39.29%, 50=13.57% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.10%, sys=5.21%, ctx=3442, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27884: Wed Oct 16 12:28:22 2024 read: IOPS=44, BW=176KiB/s (180kB/s)(6052KiB/34368msec) clat (usec): min=1601, max=47207, avg=3483.35, stdev=3419.78 lat (usec): min=1602, max=47208, avg=3484.05, stdev=3419.77 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1827], 10.00th=[ 1893], 20.00th=[ 1958], | 30.00th=[ 2040], 40.00th=[ 2114], 50.00th=[ 2212], 60.00th=[ 2343], | 70.00th=[ 2606], 80.00th=[ 3654], 90.00th=[ 7504], 95.00th=[10421], | 99.00th=[16188], 99.50th=[20055], 99.90th=[45351], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 40, max= 376, per=25.32%, avg=175.46, stdev=73.69, samples=68 iops : min= 10, max= 94, avg=43.81, stdev=18.46, samples=68 write: IOPS=45, BW=181KiB/s (186kB/s)(6236KiB/34368msec) clat (msec): min=9, max=177, avg=18.60, stdev= 8.14 lat (msec): min=9, max=177, avg=18.60, stdev= 8.14 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 20], 80.00th=[ 23], 90.00th=[ 29], 95.00th=[ 34], | 99.00th=[ 43], 99.50th=[ 50], 99.90th=[ 69], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 72, max= 248, per=24.98%, avg=181.10, stdev=55.37, samples=68 iops : min= 18, max= 62, avg=45.22, stdev=13.90, samples=68 lat (msec) : 2=12.40%, 4=27.83%, 10=6.45%, 20=39.62%, 50=13.51% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.11%, sys=5.28%, ctx=3433, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=693KiB/s (710kB/s), 171KiB/s-176KiB/s (176kB/s-180kB/s), io=23.4MiB (24.6MB), run=34368-34648msec WRITE: bw=726KiB/s (743kB/s), 181KiB/s-183KiB/s (186kB/s-188kB/s), io=24.6MiB (25.7MB), run=34368-34648msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27927: Wed Oct 16 12:28:28 2024 read: IOPS=81, BW=1308KiB/s (1339kB/s)(5920KiB/4527msec) clat (usec): min=51, max=22632, avg=8925.42, stdev=5215.34 lat (usec): min=51, max=22633, avg=8926.02, stdev=5215.35 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 62], 10.00th=[ 75], 20.00th=[ 155], | 30.00th=[ 8094], 40.00th=[ 9110], 50.00th=[10159], 60.00th=[10945], | 70.00th=[11731], 80.00th=[12780], 90.00th=[14353], 95.00th=[16188], | 99.00th=[18744], 99.50th=[22414], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 1024, max= 1664, per=25.12%, avg=1283.56, stdev=182.74, samples=9 iops : min= 64, max= 104, avg=80.22, stdev=11.42, samples=9 write: IOPS=87, BW=1407KiB/s (1440kB/s)(6368KiB/4527msec) clat (usec): min=613, max=27421, avg=2959.15, stdev=3007.51 lat (usec): min=614, max=27422, avg=2962.26, stdev=3006.55 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 725], 10.00th=[ 766], 20.00th=[ 848], | 30.00th=[ 930], 40.00th=[ 1172], 50.00th=[ 1909], 60.00th=[ 2802], | 70.00th=[ 3326], 80.00th=[ 4490], 90.00th=[ 6652], 95.00th=[ 8225], | 99.00th=[15401], 99.50th=[17695], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 832, max= 2112, per=25.93%, avg=1393.78, stdev=359.59, samples=9 iops : min= 52, max= 132, avg=87.11, stdev=22.47, samples=9 lat (usec) : 100=8.07%, 250=1.95%, 750=4.04%, 1000=13.54% lat (msec) : 2=8.85%, 4=13.67%, 10=23.44%, 20=26.04%, 50=0.39% cpu : usr=0.15%, sys=22.82%, ctx=1994, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27928: Wed Oct 16 12:28:28 2024 read: IOPS=79, BW=1278KiB/s (1309kB/s)(5744KiB/4493msec) clat (usec): min=54, max=28598, avg=8915.31, stdev=5360.06 lat (usec): min=54, max=28599, avg=8915.92, stdev=5360.06 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 63], 10.00th=[ 76], 20.00th=[ 130], | 30.00th=[ 8455], 40.00th=[ 9241], 50.00th=[10028], 60.00th=[10814], | 70.00th=[11600], 80.00th=[12518], 90.00th=[14222], 95.00th=[15795], | 99.00th=[21627], 99.50th=[25035], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 1024, max= 1472, per=24.74%, avg=1264.00, stdev=149.12, samples=8 iops : min= 64, max= 92, avg=79.00, stdev= 9.32, samples=8 write: IOPS=91, BW=1456KiB/s (1491kB/s)(6544KiB/4493msec) clat (usec): min=660, max=28881, avg=3129.21, stdev=3391.73 lat (usec): min=661, max=28883, avg=3134.29, stdev=3390.17 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 734], 10.00th=[ 766], 20.00th=[ 824], | 30.00th=[ 914], 40.00th=[ 1074], 50.00th=[ 2008], 60.00th=[ 2933], | 70.00th=[ 3589], 80.00th=[ 4752], 90.00th=[ 6915], 95.00th=[ 8979], | 99.00th=[16319], 99.50th=[18482], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 1088, max= 1792, per=26.49%, avg=1424.00, stdev=274.74, samples=8 iops : min= 68, max= 112, avg=89.00, stdev=17.17, samples=8 lat (usec) : 100=7.94%, 250=2.21%, 750=3.91%, 1000=15.76% lat (msec) : 2=6.90%, 4=13.28%, 10=24.35%, 20=24.74%, 50=0.91% cpu : usr=0.11%, sys=22.60%, ctx=2014, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27929: Wed Oct 16 12:28:28 2024 read: IOPS=83, BW=1335KiB/s (1367kB/s)(6256KiB/4687msec) clat (usec): min=50, max=35974, avg=8949.99, stdev=5663.92 lat (usec): min=51, max=35975, avg=8950.63, stdev=5663.98 clat percentiles (usec): | 1.00th=[ 54], 5.00th=[ 62], 10.00th=[ 70], 20.00th=[ 120], | 30.00th=[ 8160], 40.00th=[ 9241], 50.00th=[10290], 60.00th=[11076], | 70.00th=[12125], 80.00th=[13173], 90.00th=[14877], 95.00th=[16450], | 99.00th=[20841], 99.50th=[23200], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 896, max= 2176, per=25.74%, avg=1315.56, stdev=360.78, samples=9 iops : min= 56, max= 136, avg=82.22, stdev=22.55, samples=9 write: IOPS=80, BW=1287KiB/s (1318kB/s)(6032KiB/4687msec) clat (usec): min=645, max=33444, avg=3002.65, stdev=3587.31 lat (usec): min=646, max=33445, avg=3003.66, stdev=3587.31 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 742], 10.00th=[ 783], 20.00th=[ 848], | 30.00th=[ 979], 40.00th=[ 1188], 50.00th=[ 2008], 60.00th=[ 2737], | 70.00th=[ 3326], 80.00th=[ 4228], 90.00th=[ 5932], 95.00th=[ 8291], | 99.00th=[22152], 99.50th=[27132], 99.90th=[33424], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 672, max= 1664, per=23.02%, avg=1237.33, stdev=301.89, samples=9 iops : min= 42, max= 104, avg=77.33, stdev=18.87, samples=9 lat (usec) : 100=8.98%, 250=2.60%, 500=0.13%, 750=3.12%, 1000=12.63% lat (msec) : 2=8.98%, 4=14.45%, 10=20.83%, 20=27.21%, 50=1.04% cpu : usr=0.13%, sys=21.79%, ctx=2064, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27931: Wed Oct 16 12:28:28 2024 read: IOPS=83, BW=1342KiB/s (1374kB/s)(6032KiB/4494msec) clat (usec): min=49, max=39449, avg=8544.16, stdev=5899.73 lat (usec): min=49, max=39449, avg=8544.78, stdev=5899.73 clat percentiles (usec): | 1.00th=[ 54], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 93], | 30.00th=[ 7570], 40.00th=[ 9110], 50.00th=[ 9896], 60.00th=[10814], | 70.00th=[11863], 80.00th=[12911], 90.00th=[14353], 95.00th=[15533], | 99.00th=[22414], 99.50th=[31327], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 832, max= 1376, per=24.19%, avg=1236.00, stdev=179.34, samples=8 iops : min= 52, max= 86, avg=77.25, stdev=11.21, samples=8 write: IOPS=87, BW=1392KiB/s (1425kB/s)(6256KiB/4494msec) clat (usec): min=668, max=30412, avg=3113.80, stdev=3836.87 lat (usec): min=669, max=30413, avg=3114.76, stdev=3836.90 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 725], 10.00th=[ 766], 20.00th=[ 807], | 30.00th=[ 881], 40.00th=[ 996], 50.00th=[ 1254], 60.00th=[ 2638], | 70.00th=[ 3425], 80.00th=[ 4752], 90.00th=[ 6849], 95.00th=[ 8848], | 99.00th=[22152], 99.50th=[26346], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 896, max= 1920, per=24.40%, avg=1312.00, stdev=316.32, samples=8 iops : min= 56, max= 120, avg=82.00, stdev=19.77, samples=8 lat (usec) : 50=0.26%, 100=10.68%, 250=1.82%, 500=0.13%, 750=4.04% lat (usec) : 1000=17.06% lat (msec) : 2=6.77%, 4=10.94%, 10=22.79%, 20=23.83%, 50=1.69% cpu : usr=0.11%, sys=21.95%, ctx=1901, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5110KiB/s (5233kB/s), 1278KiB/s-1342KiB/s (1309kB/s-1374kB/s), io=23.4MiB (24.5MB), run=4493-4687msec WRITE: bw=5377KiB/s (5506kB/s), 1287KiB/s-1456KiB/s (1318kB/s-1491kB/s), io=24.6MiB (25.8MB), run=4493-4687msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27930: Wed Oct 16 12:28:32 2024 read: IOPS=40, BW=653KiB/s (669kB/s)(5920KiB/9066msec) clat (usec): min=1885, max=19594, avg=3735.08, stdev=2846.76 lat (usec): min=1886, max=19595, avg=3735.75, stdev=2846.76 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2024], 10.00th=[ 2114], 20.00th=[ 2212], | 30.00th=[ 2311], 40.00th=[ 2376], 50.00th=[ 2573], 60.00th=[ 2835], | 70.00th=[ 3163], 80.00th=[ 4359], 90.00th=[ 7046], 95.00th=[ 9896], | 99.00th=[15926], 99.50th=[19268], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 384, max= 896, per=24.78%, avg=647.06, stdev=169.24, samples=18 iops : min= 24, max= 56, avg=40.39, stdev=10.64, samples=18 write: IOPS=43, BW=702KiB/s (719kB/s)(6368KiB/9066msec) clat (usec): min=10729, max=39951, avg=19187.26, stdev=5819.95 lat (usec): min=10730, max=39953, avg=19188.52, stdev=5820.00 clat percentiles (usec): | 1.00th=[11338], 5.00th=[11731], 10.00th=[12387], 20.00th=[13698], | 30.00th=[15533], 40.00th=[17171], 50.00th=[18482], 60.00th=[19530], | 70.00th=[20841], 80.00th=[23462], 90.00th=[27395], 95.00th=[30278], | 99.00th=[38536], 99.50th=[39584], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 416, max= 928, per=25.49%, avg=700.33, stdev=166.19, samples=18 iops : min= 26, max= 58, avg=43.72, stdev=10.45, samples=18 lat (msec) : 2=1.82%, 4=35.42%, 10=8.59%, 20=34.77%, 50=19.40% cpu : usr=0.15%, sys=5.82%, ctx=893, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27932: Wed Oct 16 12:28:32 2024 read: IOPS=39, BW=626KiB/s (641kB/s)(5744KiB/9171msec) clat (usec): min=1870, max=20350, avg=3893.73, stdev=3087.92 lat (usec): min=1870, max=20350, avg=3894.40, stdev=3087.93 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2040], 10.00th=[ 2114], 20.00th=[ 2212], | 30.00th=[ 2311], 40.00th=[ 2409], 50.00th=[ 2573], 60.00th=[ 2802], | 70.00th=[ 3228], 80.00th=[ 4817], 90.00th=[ 7832], 95.00th=[10421], | 99.00th=[18220], 99.50th=[19268], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 288, max= 1184, per=23.96%, avg=625.56, stdev=270.77, samples=18 iops : min= 18, max= 74, avg=38.94, stdev=17.01, samples=18 write: IOPS=44, BW=714KiB/s (731kB/s)(6544KiB/9171msec) clat (usec): min=10347, max=42757, avg=18968.97, stdev=6055.12 lat (usec): min=10348, max=42758, avg=18970.25, stdev=6055.16 clat percentiles (usec): | 1.00th=[10945], 5.00th=[11600], 10.00th=[12256], 20.00th=[13698], | 30.00th=[15008], 40.00th=[16581], 50.00th=[17957], 60.00th=[19268], | 70.00th=[20579], 80.00th=[23462], 90.00th=[27395], 95.00th=[30278], | 99.00th=[36439], 99.50th=[39060], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 448, max= 928, per=25.75%, avg=707.22, stdev=177.25, samples=18 iops : min= 28, max= 58, avg=44.06, stdev=11.22, samples=18 lat (msec) : 2=1.69%, 4=33.46%, 10=8.85%, 20=38.15%, 50=17.84% cpu : usr=0.12%, sys=5.92%, ctx=864, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27933: Wed Oct 16 12:28:32 2024 read: IOPS=44, BW=709KiB/s (726kB/s)(6256KiB/8821msec) clat (usec): min=1850, max=30904, avg=3930.47, stdev=3227.62 lat (usec): min=1851, max=30905, avg=3931.28, stdev=3227.58 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2024], 10.00th=[ 2073], 20.00th=[ 2212], | 30.00th=[ 2311], 40.00th=[ 2442], 50.00th=[ 2573], 60.00th=[ 2900], | 70.00th=[ 3425], 80.00th=[ 5145], 90.00th=[ 7701], 95.00th=[11469], | 99.00th=[17433], 99.50th=[19268], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 384, max= 1280, per=27.40%, avg=715.29, stdev=278.96, samples=17 iops : min= 24, max= 80, avg=44.71, stdev=17.43, samples=17 write: IOPS=42, BW=684KiB/s (700kB/s)(6032KiB/8821msec) clat (usec): min=10648, max=42778, avg=19261.34, stdev=6178.73 lat (usec): min=10649, max=42780, avg=19262.62, stdev=6178.78 clat percentiles (usec): | 1.00th=[11076], 5.00th=[11863], 10.00th=[12256], 20.00th=[13435], | 30.00th=[15401], 40.00th=[16909], 50.00th=[18220], 60.00th=[19268], | 70.00th=[21103], 80.00th=[24511], 90.00th=[28181], 95.00th=[31065], | 99.00th=[39584], 99.50th=[42206], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 448, max= 928, per=24.53%, avg=673.88, stdev=160.99, samples=17 iops : min= 28, max= 58, avg=42.12, stdev=10.06, samples=17 lat (msec) : 2=2.08%, 4=35.81%, 10=9.64%, 20=35.42%, 50=17.06% cpu : usr=0.10%, sys=6.28%, ctx=867, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27934: Wed Oct 16 12:28:32 2024 read: IOPS=41, BW=671KiB/s (687kB/s)(6032KiB/8988msec) clat (usec): min=1757, max=31633, avg=4026.59, stdev=3669.20 lat (usec): min=1757, max=31634, avg=4027.32, stdev=3669.17 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2024], 10.00th=[ 2089], 20.00th=[ 2212], | 30.00th=[ 2311], 40.00th=[ 2474], 50.00th=[ 2638], 60.00th=[ 2933], | 70.00th=[ 3654], 80.00th=[ 4752], 90.00th=[ 6783], 95.00th=[12649], | 99.00th=[18744], 99.50th=[27132], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 416, max= 1120, per=25.00%, avg=652.82, stdev=193.66, samples=17 iops : min= 26, max= 70, avg=40.65, stdev=12.10, samples=17 write: IOPS=43, BW=696KiB/s (713kB/s)(6256KiB/8988msec) clat (usec): min=10694, max=46254, avg=18971.59, stdev=5580.65 lat (usec): min=10695, max=46255, avg=18972.86, stdev=5580.68 clat percentiles (usec): | 1.00th=[11338], 5.00th=[11600], 10.00th=[12256], 20.00th=[14222], | 30.00th=[15926], 40.00th=[17171], 50.00th=[18220], 60.00th=[19268], | 70.00th=[20579], 80.00th=[22676], 90.00th=[26084], 95.00th=[29492], | 99.00th=[36963], 99.50th=[45876], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 448, max= 926, per=25.07%, avg=688.59, stdev=159.59, samples=17 iops : min= 28, max= 57, avg=42.88, stdev= 9.78, samples=17 lat (msec) : 2=1.69%, 4=33.72%, 10=10.42%, 20=36.33%, 50=17.84% cpu : usr=0.16%, sys=6.11%, ctx=878, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2612KiB/s (2674kB/s), 626KiB/s-709KiB/s (641kB/s-726kB/s), io=23.4MiB (24.5MB), run=8821-9171msec WRITE: bw=2748KiB/s (2814kB/s), 684KiB/s-714KiB/s (700kB/s-731kB/s), io=24.6MiB (25.8MB), run=8821-9171msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27947: Wed Oct 16 12:28:36 2024 read: IOPS=3, BW=3545KiB/s (3630kB/s)(7168KiB/2022msec) clat (usec): min=398, max=906168, avg=244166.80, stdev=360860.59 lat (usec): min=399, max=906169, avg=244167.94, stdev=360860.63 clat percentiles (usec): | 1.00th=[ 400], 5.00th=[ 400], 10.00th=[ 400], 20.00th=[ 433], | 30.00th=[ 4178], 40.00th=[ 4178], 50.00th=[ 47973], 60.00th=[158335], | 70.00th=[158335], 80.00th=[591397], 90.00th=[910164], 95.00th=[910164], | 99.00th=[910164], 99.50th=[910164], 99.90th=[910164], 99.95th=[910164], | 99.99th=[910164] bw ( KiB/s): min= 4096, max= 6144, per=86.24%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2532KiB/s (2593kB/s)(5120KiB/2022msec) clat (msec): min=14, max=158, avg=59.94, stdev=56.78 lat (msec): min=15, max=158, avg=59.99, stdev=56.75 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 42], 60.00th=[ 42], | 70.00th=[ 50], 80.00th=[ 50], 90.00th=[ 159], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 2048, max= 6144, per=49.27%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (usec) : 500=16.67% lat (msec) : 10=8.33%, 20=8.33%, 50=33.33%, 250=16.67%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.05%, sys=48.74%, ctx=553, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27949: Wed Oct 16 12:28:36 2024 read: IOPS=1, BW=1081KiB/s (1107kB/s)(2048KiB/1894msec) clat (msec): min=64, max=267, avg=165.87, stdev=143.51 lat (msec): min=64, max=267, avg=165.88, stdev=143.51 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 65], 10.00th=[ 65], 20.00th=[ 65], | 30.00th=[ 65], 40.00th=[ 65], 50.00th=[ 65], 60.00th=[ 268], | 70.00th=[ 268], 80.00th=[ 268], 90.00th=[ 268], 95.00th=[ 268], | 99.00th=[ 268], 99.50th=[ 268], 99.90th=[ 268], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 2048, max= 2048, per=34.50%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5407KiB/s (5536kB/s)(10.0MiB/1894msec) clat (msec): min=15, max=646, avg=152.29, stdev=195.25 lat (msec): min=15, max=646, avg=152.35, stdev=195.25 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 19], | 30.00th=[ 31], 40.00th=[ 33], 50.00th=[ 53], 60.00th=[ 75], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 249], 95.00th=[ 651], | 99.00th=[ 651], 99.50th=[ 651], 99.90th=[ 651], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 6144, max=10240, per=98.54%, avg=8192.00, stdev=2896.31, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (msec) : 20=16.67%, 50=16.67%, 100=25.00%, 250=25.00%, 500=8.33% lat (msec) : 750=8.33% cpu : usr=0.00%, sys=18.91%, ctx=202, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27951: Wed Oct 16 12:28:36 2024 read: IOPS=1, BW=1781KiB/s (1824kB/s)(6144KiB/3449msec) clat (usec): min=376, max=1176.1k, avg=352666.62, stdev=503431.03 lat (usec): min=377, max=1176.1k, avg=352667.54, stdev=503430.94 clat percentiles (usec): | 1.00th=[ 375], 5.00th=[ 375], 10.00th=[ 375], | 20.00th=[ 4228], 30.00th=[ 4228], 40.00th=[ 46400], | 50.00th=[ 46400], 60.00th=[ 103285], 70.00th=[ 784335], | 80.00th=[ 784335], 90.00th=[1182794], 95.00th=[1182794], | 99.00th=[1182794], 99.50th=[1182794], 99.90th=[1182794], | 99.95th=[1182794], 99.99th=[1182794] bw ( KiB/s): min= 2048, max= 4096, per=57.44%, avg=3410.33, stdev=1179.82, samples=3 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=3 write: IOPS=1, BW=1781KiB/s (1824kB/s)(6144KiB/3449msec) clat (msec): min=28, max=778, avg=220.49, stdev=286.80 lat (msec): min=29, max=778, avg=220.53, stdev=286.79 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 46], 50.00th=[ 46], 60.00th=[ 205], | 70.00th=[ 224], 80.00th=[ 224], 90.00th=[ 776], 95.00th=[ 776], | 99.00th=[ 776], 99.50th=[ 776], 99.90th=[ 776], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 2048, max= 6131, per=49.22%, avg=4091.67, stdev=2041.50, samples=3 iops : min= 2, max= 5, avg= 3.67, stdev= 1.53, samples=3 lat (usec) : 500=8.33% lat (msec) : 10=8.33%, 50=33.33%, 250=25.00%, 1000=16.67% cpu : usr=0.00%, sys=9.25%, ctx=649, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27953: Wed Oct 16 12:28:36 2024 read: IOPS=1, BW=1778KiB/s (1820kB/s)(5120KiB/2880msec) clat (usec): min=510, max=602759, avg=247664.05, stdev=299796.73 lat (usec): min=511, max=602761, avg=247665.17, stdev=299796.97 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 510], 10.00th=[ 510], 20.00th=[ 510], | 30.00th=[ 40633], 40.00th=[ 40633], 50.00th=[ 47449], 60.00th=[ 47449], | 70.00th=[549454], 80.00th=[549454], 90.00th=[599786], 95.00th=[599786], | 99.00th=[599786], 99.50th=[599786], 99.90th=[599786], 99.95th=[599786], | 99.99th=[599786] bw ( KiB/s): min= 4096, max= 4096, per=68.99%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2489KiB/s (2549kB/s)(7168KiB/2880msec) clat (msec): min=33, max=823, avg=227.51, stdev=268.98 lat (msec): min=33, max=823, avg=227.56, stdev=268.97 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 74], | 30.00th=[ 138], 40.00th=[ 138], 50.00th=[ 161], 60.00th=[ 165], | 70.00th=[ 165], 80.00th=[ 199], 90.00th=[ 827], 95.00th=[ 827], | 99.00th=[ 827], 99.50th=[ 827], 99.90th=[ 827], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 2048, max= 8192, per=57.48%, avg=4778.67, stdev=3128.37, samples=3 iops : min= 2, max= 8, avg= 4.67, stdev= 3.06, samples=3 lat (usec) : 750=8.33% lat (msec) : 50=25.00%, 100=8.33%, 250=33.33%, 750=16.67%, 1000=8.33% cpu : usr=0.00%, sys=11.18%, ctx=656, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5938KiB/s (6080kB/s), 1081KiB/s-3545KiB/s (1107kB/s-3630kB/s), io=20.0MiB (20.0MB), run=1894-3449msec WRITE: bw=8313KiB/s (8513kB/s), 1781KiB/s-5407KiB/s (1824kB/s-5536kB/s), io=28.0MiB (29.4MB), run=1894-3449msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27948: Wed Oct 16 12:28:36 2024 read: IOPS=2, BW=2378KiB/s (2435kB/s)(7168KiB/3014msec) clat (msec): min=12, max=766, avg=165.68, stdev=272.41 lat (msec): min=12, max=766, avg=165.68, stdev=272.41 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 15], | 30.00th=[ 29], 40.00th=[ 29], 50.00th=[ 29], 60.00th=[ 150], | 70.00th=[ 150], 80.00th=[ 161], 90.00th=[ 768], 95.00th=[ 768], | 99.00th=[ 768], 99.50th=[ 768], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 4087, max= 6144, per=88.50%, avg=5115.50, stdev=1454.52, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 write: IOPS=1, BW=1699KiB/s (1740kB/s)(5120KiB/3014msec) clat (msec): min=33, max=1471, avg=357.42, stdev=626.67 lat (msec): min=33, max=1471, avg=357.48, stdev=626.67 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 45], 60.00th=[ 45], | 70.00th=[ 203], 80.00th=[ 203], 90.00th=[ 1469], 95.00th=[ 1469], | 99.00th=[ 1469], 99.50th=[ 1469], 99.90th=[ 1469], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 2048, max= 4087, per=37.91%, avg=3067.50, stdev=1441.79, samples=2 iops : min= 2, max= 3, avg= 2.50, stdev= 0.71, samples=2 lat (msec) : 20=16.67%, 50=41.67%, 250=25.00%, 1000=8.33% cpu : usr=0.00%, sys=3.02%, ctx=61, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27950: Wed Oct 16 12:28:36 2024 read: IOPS=0, BW=578KiB/s (592kB/s)(2048KiB/3543msec) clat (usec): min=36341, max=72404, avg=54373.36, stdev=25500.33 lat (usec): min=36343, max=72405, avg=54374.55, stdev=25500.15 clat percentiles (usec): | 1.00th=[36439], 5.00th=[36439], 10.00th=[36439], 20.00th=[36439], | 30.00th=[36439], 40.00th=[36439], 50.00th=[36439], 60.00th=[72877], | 70.00th=[72877], 80.00th=[72877], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 2048, max= 2048, per=35.43%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2890KiB/s (2960kB/s)(10.0MiB/3543msec) clat (msec): min=30, max=1289, avg=339.05, stdev=475.33 lat (msec): min=30, max=1289, avg=339.11, stdev=475.32 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 32], | 30.00th=[ 39], 40.00th=[ 46], 50.00th=[ 68], 60.00th=[ 69], | 70.00th=[ 493], 80.00th=[ 493], 90.00th=[ 1099], 95.00th=[ 1284], | 99.00th=[ 1284], 99.50th=[ 1284], 99.90th=[ 1284], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 1780, max=10240, per=64.18%, avg=5193.67, stdev=4460.06, samples=3 iops : min= 1, max= 10, avg= 4.67, stdev= 4.73, samples=3 lat (msec) : 50=41.67%, 100=25.00%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=2.68%, ctx=58, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27952: Wed Oct 16 12:28:36 2024 read: IOPS=1, BW=1777KiB/s (1820kB/s)(6144KiB/3457msec) clat (msec): min=11, max=1294, avg=325.19, stdev=503.90 lat (msec): min=11, max=1294, avg=325.19, stdev=503.90 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 18], | 30.00th=[ 18], 40.00th=[ 29], 50.00th=[ 29], 60.00th=[ 144], | 70.00th=[ 456], 80.00th=[ 456], 90.00th=[ 1301], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 1793, max= 4096, per=43.19%, avg=2496.25, stdev=1073.25, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 write: IOPS=1, BW=1777KiB/s (1820kB/s)(6144KiB/3457msec) clat (msec): min=32, max=752, avg=243.97, stdev=275.32 lat (msec): min=32, max=752, avg=244.03, stdev=275.30 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 38], | 30.00th=[ 38], 40.00th=[ 129], 50.00th=[ 129], 60.00th=[ 159], | 70.00th=[ 355], 80.00th=[ 355], 90.00th=[ 751], 95.00th=[ 751], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 1793, max= 6144, per=36.39%, avg=2944.50, stdev=2136.38, samples=4 iops : min= 1, max= 6, avg= 2.50, stdev= 2.38, samples=4 lat (msec) : 20=16.67%, 50=25.00%, 250=25.00%, 500=16.67%, 1000=8.33% cpu : usr=0.00%, sys=2.84%, ctx=233, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27954: Wed Oct 16 12:28:36 2024 read: IOPS=2, BW=3028KiB/s (3100kB/s)(5120KiB/1691msec) clat (usec): min=13412, max=44383, avg=23898.58, stdev=12154.58 lat (usec): min=13413, max=44384, avg=23899.52, stdev=12154.72 clat percentiles (usec): | 1.00th=[13435], 5.00th=[13435], 10.00th=[13435], 20.00th=[13435], | 30.00th=[16581], 40.00th=[16581], 50.00th=[20841], 60.00th=[20841], | 70.00th=[23987], 80.00th=[23987], 90.00th=[44303], 95.00th=[44303], | 99.00th=[44303], 99.50th=[44303], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 4096, max= 4096, per=70.87%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4239KiB/s (4341kB/s)(7168KiB/1691msec) clat (msec): min=41, max=716, avg=222.74, stdev=228.21 lat (msec): min=41, max=716, avg=222.80, stdev=228.19 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 66], | 30.00th=[ 153], 40.00th=[ 153], 50.00th=[ 159], 60.00th=[ 186], | 70.00th=[ 186], 80.00th=[ 243], 90.00th=[ 718], 95.00th=[ 718], | 99.00th=[ 718], 99.50th=[ 718], 99.90th=[ 718], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 4087, max= 8192, per=75.87%, avg=6139.50, stdev=2902.67, samples=2 iops : min= 3, max= 8, avg= 5.50, stdev= 3.54, samples=2 lat (msec) : 20=16.67%, 50=33.33%, 100=8.33%, 250=33.33%, 750=8.33% cpu : usr=0.00%, sys=6.09%, ctx=76, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5780KiB/s (5919kB/s), 578KiB/s-3028KiB/s (592kB/s-3100kB/s), io=20.0MiB (20.0MB), run=1691-3543msec WRITE: bw=8093KiB/s (8287kB/s), 1699KiB/s-4239KiB/s (1740kB/s-4341kB/s), io=28.0MiB (29.4MB), run=1691-3543msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27964: Wed Oct 16 12:28:41 2024 read: IOPS=0, BW=3442KiB/s (3525kB/s)(12.0MiB/3570msec) clat (msec): min=242, max=2100, avg=1181.43, stdev=928.98 lat (msec): min=242, max=2100, avg=1181.43, stdev=928.98 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 243], 10.00th=[ 243], 20.00th=[ 243], | 30.00th=[ 243], 40.00th=[ 1200], 50.00th=[ 1200], 60.00th=[ 1200], | 70.00th=[ 2106], 80.00th=[ 2106], 90.00th=[ 2106], 95.00th=[ 2106], | 99.00th=[ 2106], 99.50th=[ 2106], 99.90th=[ 2106], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 5483, max= 8192, per=100.00%, avg=6837.50, stdev=1915.55, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.03%, sys=1.54%, ctx=33, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27966: Wed Oct 16 12:28:41 2024 read: IOPS=0, BW=2174KiB/s (2226kB/s)(4096KiB/1884msec) clat (nsec): min=356170k, max=356170k, avg=356169738.00, stdev= 0.00 lat (nsec): min=356171k, max=356171k, avg=356171332.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 355], 10.00th=[ 355], 20.00th=[ 355], | 30.00th=[ 355], 40.00th=[ 355], 50.00th=[ 355], 60.00th=[ 355], | 70.00th=[ 355], 80.00th=[ 355], 90.00th=[ 355], 95.00th=[ 355], | 99.00th=[ 355], 99.50th=[ 355], 99.90th=[ 355], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4348KiB/s (4453kB/s)(8192KiB/1884msec) clat (msec): min=120, max=1336, avg=728.24, stdev=859.76 lat (msec): min=120, max=1336, avg=728.46, stdev=859.72 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 121], 10.00th=[ 121], 20.00th=[ 121], | 30.00th=[ 121], 40.00th=[ 121], 50.00th=[ 121], 60.00th=[ 1334], | 70.00th=[ 1334], 80.00th=[ 1334], 90.00th=[ 1334], 95.00th=[ 1334], | 99.00th=[ 1334], 99.50th=[ 1334], 99.90th=[ 1334], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 5595, max= 5595, per=32.17%, avg=5595.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=4.20%, ctx=33, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27968: Wed Oct 16 12:28:41 2024 write: IOPS=4, BW=16.9MiB/s (17.8MB/s)(12.0MiB/708msec) clat (msec): min=123, max=357, avg=222.11, stdev=121.13 lat (msec): min=123, max=357, avg=222.41, stdev=121.12 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 124], 10.00th=[ 124], 20.00th=[ 124], | 30.00th=[ 124], 40.00th=[ 186], 50.00th=[ 186], 60.00th=[ 186], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min=16351, max=16351, per=94.01%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.14%, sys=15.13%, ctx=123, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27969: Wed Oct 16 12:28:41 2024 write: IOPS=4, BW=18.3MiB/s (19.2MB/s)(12.0MiB/654msec) clat (msec): min=106, max=346, avg=206.60, stdev=124.97 lat (msec): min=106, max=346, avg=206.89, stdev=124.96 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 167], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min=16384, max=16384, per=94.20%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.15%, sys=12.86%, ctx=72, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4589KiB/s (4700kB/s), 2174KiB/s-3442KiB/s (2226kB/s-3525kB/s), io=16.0MiB (16.8MB), run=1884-3570msec WRITE: bw=16.0MiB/s (17.8MB/s), 4348KiB/s-18.3MiB/s (4453kB/s-19.2MB/s), io=32.0MiB (33.6MB), run=654-1884msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27962: Wed Oct 16 12:28:41 2024 read: IOPS=0, BW=3465KiB/s (3548kB/s)(12.0MiB/3546msec) clat (msec): min=185, max=2269, avg=1164.83, stdev=1047.40 lat (msec): min=185, max=2269, avg=1164.83, stdev=1047.40 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 186], 10.00th=[ 186], 20.00th=[ 186], | 30.00th=[ 186], 40.00th=[ 1036], 50.00th=[ 1036], 60.00th=[ 1036], | 70.00th=[ 2265], 80.00th=[ 2265], 90.00th=[ 2265], 95.00th=[ 2265], | 99.00th=[ 2265], 99.50th=[ 2265], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 5580, max= 8192, per=100.00%, avg=6886.00, stdev=1846.96, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.03%, sys=31.42%, ctx=1669, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27963: Wed Oct 16 12:28:41 2024 read: IOPS=0, BW=2271KiB/s (2325kB/s)(4096KiB/1804msec) clat (nsec): min=198286k, max=198286k, avg=198286172.00, stdev= 0.00 lat (nsec): min=198288k, max=198288k, avg=198288009.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 199], 10.00th=[ 199], 20.00th=[ 199], | 30.00th=[ 199], 40.00th=[ 199], 50.00th=[ 199], 60.00th=[ 199], | 70.00th=[ 199], 80.00th=[ 199], 90.00th=[ 199], 95.00th=[ 199], | 99.00th=[ 199], 99.50th=[ 199], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4541KiB/s (4650kB/s)(8192KiB/1804msec) clat (msec): min=143, max=1421, avg=782.23, stdev=903.45 lat (msec): min=143, max=1421, avg=782.51, stdev=903.46 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 144], 50.00th=[ 144], 60.00th=[ 1418], | 70.00th=[ 1418], 80.00th=[ 1418], 90.00th=[ 1418], 95.00th=[ 1418], | 99.00th=[ 1418], 99.50th=[ 1418], 99.90th=[ 1418], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 8192, max= 8192, per=91.98%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.06%, sys=12.31%, ctx=157, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27965: Wed Oct 16 12:28:41 2024 write: IOPS=0, BW=3340KiB/s (3420kB/s)(12.0MiB/3679msec) clat (msec): min=318, max=1937, avg=1207.91, stdev=821.63 lat (msec): min=318, max=1938, avg=1208.23, stdev=821.55 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 317], 10.00th=[ 317], 20.00th=[ 317], | 30.00th=[ 317], 40.00th=[ 1368], 50.00th=[ 1368], 60.00th=[ 1368], | 70.00th=[ 1938], 80.00th=[ 1938], 90.00th=[ 1938], 95.00th=[ 1938], | 99.00th=[ 1938], 99.50th=[ 1938], 99.90th=[ 1938], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 5542, max= 8192, per=77.11%, avg=6867.00, stdev=1873.83, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=7.01%, ctx=50, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27967: Wed Oct 16 12:28:41 2024 write: IOPS=4, BW=19.9MiB/s (20.8MB/s)(12.0MiB/604msec) clat (msec): min=99, max=378, avg=198.33, stdev=156.62 lat (msec): min=100, max=379, avg=198.68, stdev=156.58 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 101], 10.00th=[ 101], 20.00th=[ 101], | 30.00th=[ 101], 40.00th=[ 116], 50.00th=[ 116], 60.00th=[ 116], | 70.00th=[ 380], 80.00th=[ 380], 90.00th=[ 380], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=35.66%, ctx=87, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4620KiB/s (4731kB/s), 2271KiB/s-3465KiB/s (2325kB/s-3548kB/s), io=16.0MiB (16.8MB), run=1804-3546msec WRITE: bw=8907KiB/s (9121kB/s), 3340KiB/s-19.9MiB/s (3420kB/s-20.8MB/s), io=32.0MiB (33.6MB), run=604-3679msec