== sanity test 398b: DIO and buffer IO race ============== 03:01:59 (1735372919) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.13288 s, 44.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25534: Sat Dec 28 03:02:14 2024 read: IOPS=124, BW=499KiB/s (511kB/s)(5992KiB/12014msec) clat (usec): min=52, max=168293, avg=4629.98, stdev=4873.12 lat (usec): min=52, max=168294, avg=4630.70, stdev=4873.06 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 1942], 10.00th=[ 2311], 20.00th=[ 2737], | 30.00th=[ 3130], 40.00th=[ 3589], 50.00th=[ 4113], 60.00th=[ 4686], | 70.00th=[ 5342], 80.00th=[ 6128], 90.00th=[ 7308], 95.00th=[ 8455], | 99.00th=[ 13173], 99.50th=[ 15533], 99.90th=[ 29230], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 272, max= 752, per=24.90%, avg=497.42, stdev=85.82, samples=24 iops : min= 68, max= 188, avg=124.29, stdev=21.43, samples=24 write: IOPS=131, BW=524KiB/s (537kB/s)(6296KiB/12014msec) clat (usec): min=473, max=16855, avg=3175.72, stdev=2702.98 lat (usec): min=474, max=16855, avg=3176.73, stdev=2703.15 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 537], 10.00th=[ 553], 20.00th=[ 627], | 30.00th=[ 848], 40.00th=[ 1909], 50.00th=[ 2638], 60.00th=[ 3294], | 70.00th=[ 4047], 80.00th=[ 5342], 90.00th=[ 7111], 95.00th=[ 8455], | 99.00th=[11469], 99.50th=[12125], 99.90th=[14353], 99.95th=[16909], | 99.99th=[16909] bw ( KiB/s): min= 328, max= 752, per=24.96%, avg=522.17, stdev=78.16, samples=24 iops : min= 82, max= 188, avg=130.46, stdev=19.58, samples=24 lat (usec) : 100=1.56%, 250=0.16%, 500=0.29%, 750=13.38%, 1000=3.39% lat (msec) : 2=4.79%, 4=35.61%, 10=38.28%, 20=2.47%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=23.99%, ctx=6010, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25536: Sat Dec 28 03:02:14 2024 read: IOPS=124, BW=499KiB/s (511kB/s)(5940KiB/11908msec) clat (usec): min=49, max=17477, avg=4405.61, stdev=2380.57 lat (usec): min=49, max=17478, avg=4406.33, stdev=2380.58 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 1942], 10.00th=[ 2278], 20.00th=[ 2638], | 30.00th=[ 3032], 40.00th=[ 3425], 50.00th=[ 3949], 60.00th=[ 4621], | 70.00th=[ 5145], 80.00th=[ 5932], 90.00th=[ 7177], 95.00th=[ 8586], | 99.00th=[13566], 99.50th=[14353], 99.90th=[16581], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 304, max= 584, per=24.60%, avg=491.48, stdev=68.58, samples=23 iops : min= 76, max= 146, avg=122.70, stdev=17.22, samples=23 write: IOPS=133, BW=533KiB/s (546kB/s)(6348KiB/11908msec) clat (usec): min=481, max=167439, avg=3332.33, stdev=5034.12 lat (usec): min=481, max=167440, avg=3333.33, stdev=5034.13 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 619], | 30.00th=[ 766], 40.00th=[ 1745], 50.00th=[ 2573], 60.00th=[ 3261], | 70.00th=[ 4146], 80.00th=[ 5538], 90.00th=[ 7308], 95.00th=[ 8848], | 99.00th=[ 12256], 99.50th=[ 14091], 99.90th=[ 26346], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 360, max= 696, per=24.78%, avg=518.48, stdev=80.82, samples=23 iops : min= 90, max= 174, avg=129.48, stdev=20.19, samples=23 lat (usec) : 50=0.03%, 100=1.69%, 250=0.16%, 500=0.29%, 750=15.04% lat (usec) : 1000=2.86% lat (msec) : 2=4.07%, 4=35.84%, 10=37.11%, 20=2.83%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=23.77%, ctx=6047, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25538: Sat Dec 28 03:02:14 2024 read: IOPS=125, BW=502KiB/s (514kB/s)(6028KiB/12003msec) clat (usec): min=50, max=168936, avg=4556.05, stdev=4917.58 lat (usec): min=50, max=168937, avg=4556.63, stdev=4917.59 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 1778], 10.00th=[ 2245], 20.00th=[ 2606], | 30.00th=[ 3032], 40.00th=[ 3458], 50.00th=[ 4015], 60.00th=[ 4621], | 70.00th=[ 5276], 80.00th=[ 5997], 90.00th=[ 7308], 95.00th=[ 8848], | 99.00th=[ 13304], 99.50th=[ 15926], 99.90th=[ 27132], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 288, max= 688, per=25.12%, avg=501.92, stdev=82.50, samples=24 iops : min= 72, max= 172, avg=125.33, stdev=20.66, samples=24 write: IOPS=130, BW=522KiB/s (534kB/s)(6260KiB/12003msec) clat (usec): min=395, max=20287, avg=3231.93, stdev=2853.47 lat (usec): min=395, max=20288, avg=3232.62, stdev=2853.46 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 611], | 30.00th=[ 807], 40.00th=[ 1860], 50.00th=[ 2606], 60.00th=[ 3261], | 70.00th=[ 4228], 80.00th=[ 5407], 90.00th=[ 7373], 95.00th=[ 8979], | 99.00th=[11863], 99.50th=[13304], 99.90th=[18744], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 352, max= 776, per=24.90%, avg=520.96, stdev=95.30, samples=24 iops : min= 88, max= 194, avg=130.08, stdev=23.84, samples=24 lat (usec) : 100=2.18%, 250=0.13%, 500=0.36%, 750=14.26%, 1000=2.93% lat (msec) : 2=4.13%, 4=35.22%, 10=38.12%, 20=2.57%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=23.95%, ctx=5963, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25540: Sat Dec 28 03:02:14 2024 read: IOPS=126, BW=507KiB/s (519kB/s)(6052KiB/11934msec) clat (usec): min=48, max=20457, avg=4472.17, stdev=2366.10 lat (usec): min=48, max=20457, avg=4472.81, stdev=2366.08 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 1926], 10.00th=[ 2278], 20.00th=[ 2671], | 30.00th=[ 3064], 40.00th=[ 3490], 50.00th=[ 3982], 60.00th=[ 4621], | 70.00th=[ 5342], 80.00th=[ 6128], 90.00th=[ 7242], 95.00th=[ 8586], | 99.00th=[12649], 99.50th=[14222], 99.90th=[19530], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 304, max= 648, per=24.86%, avg=496.65, stdev=67.86, samples=23 iops : min= 76, max= 162, avg=124.00, stdev=16.97, samples=23 write: IOPS=130, BW=523KiB/s (535kB/s)(6236KiB/11934msec) clat (usec): min=408, max=168926, avg=3264.32, stdev=5019.34 lat (usec): min=409, max=168926, avg=3265.23, stdev=5019.39 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 529], 10.00th=[ 562], 20.00th=[ 619], | 30.00th=[ 799], 40.00th=[ 1762], 50.00th=[ 2638], 60.00th=[ 3261], | 70.00th=[ 4047], 80.00th=[ 5407], 90.00th=[ 7046], 95.00th=[ 8356], | 99.00th=[ 11076], 99.50th=[ 14091], 99.90th=[ 19530], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 407, max= 680, per=24.46%, avg=511.61, stdev=72.02, samples=23 iops : min= 101, max= 170, avg=127.74, stdev=18.06, samples=23 lat (usec) : 50=0.03%, 100=1.60%, 250=0.23%, 500=0.36%, 750=14.16% lat (usec) : 1000=2.93% lat (msec) : 2=4.79%, 4=35.90%, 10=37.63%, 20=2.31%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=24.34%, ctx=6081, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1999KiB/s (2047kB/s), 499KiB/s-507KiB/s (511kB/s-519kB/s), io=23.4MiB (24.6MB), run=11908-12014msec WRITE: bw=2093KiB/s (2143kB/s), 522KiB/s-533KiB/s (534kB/s-546kB/s), io=24.6MiB (25.7MB), run=11908-12014msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25535: Sat Dec 28 03:02:17 2024 read: IOPS=99, BW=398KiB/s (407kB/s)(5992KiB/15070msec) clat (usec): min=1499, max=200078, avg=4053.11, stdev=5832.81 lat (usec): min=1499, max=200080, avg=4053.76, stdev=5832.81 clat percentiles (usec): | 1.00th=[ 1598], 5.00th=[ 1713], 10.00th=[ 1811], 20.00th=[ 1958], | 30.00th=[ 2147], 40.00th=[ 2376], 50.00th=[ 2704], 60.00th=[ 3163], | 70.00th=[ 3884], 80.00th=[ 5538], 90.00th=[ 8291], 95.00th=[ 10421], | 99.00th=[ 13829], 99.50th=[ 15926], 99.90th=[ 25297], 99.95th=[200279], | 99.99th=[200279] bw ( KiB/s): min= 152, max= 840, per=24.87%, avg=395.97, stdev=178.95, samples=30 iops : min= 38, max= 210, avg=98.97, stdev=44.75, samples=30 write: IOPS=104, BW=418KiB/s (428kB/s)(6296KiB/15070msec) clat (usec): min=2011, max=27322, avg=5670.34, stdev=3395.42 lat (usec): min=2012, max=27322, avg=5671.13, stdev=3395.37 clat percentiles (usec): | 1.00th=[ 2343], 5.00th=[ 2671], 10.00th=[ 2835], 20.00th=[ 3097], | 30.00th=[ 3359], 40.00th=[ 3720], 50.00th=[ 4293], 60.00th=[ 5211], | 70.00th=[ 6652], 80.00th=[ 8160], 90.00th=[10159], 95.00th=[12125], | 99.00th=[18220], 99.50th=[20579], 99.90th=[25822], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 160, max= 880, per=24.97%, avg=416.23, stdev=181.36, samples=30 iops : min= 40, max= 220, avg=104.03, stdev=45.35, samples=30 lat (msec) : 2=11.07%, 4=47.10%, 10=33.46%, 20=8.01%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=11.12%, ctx=3578, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25537: Sat Dec 28 03:02:17 2024 read: IOPS=98, BW=394KiB/s (403kB/s)(5940KiB/15077msec) clat (usec): min=1539, max=172383, avg=4075.62, stdev=5296.78 lat (usec): min=1540, max=172385, avg=4076.22, stdev=5296.82 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1745], 10.00th=[ 1827], 20.00th=[ 1975], | 30.00th=[ 2147], 40.00th=[ 2343], 50.00th=[ 2671], 60.00th=[ 3195], | 70.00th=[ 3949], 80.00th=[ 5473], 90.00th=[ 8291], 95.00th=[ 10552], | 99.00th=[ 15664], 99.50th=[ 17171], 99.90th=[ 20055], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 136, max= 784, per=24.69%, avg=393.07, stdev=179.89, samples=30 iops : min= 34, max= 196, avg=98.13, stdev=44.93, samples=30 write: IOPS=105, BW=421KiB/s (431kB/s)(6348KiB/15077msec) clat (usec): min=2139, max=26995, avg=5640.92, stdev=3324.80 lat (usec): min=2140, max=26996, avg=5641.66, stdev=3324.76 clat percentiles (usec): | 1.00th=[ 2278], 5.00th=[ 2704], 10.00th=[ 2900], 20.00th=[ 3130], | 30.00th=[ 3359], 40.00th=[ 3687], 50.00th=[ 4178], 60.00th=[ 5145], | 70.00th=[ 6587], 80.00th=[ 8029], 90.00th=[10290], 95.00th=[12125], | 99.00th=[16909], 99.50th=[18482], 99.90th=[24249], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 208, max= 808, per=25.10%, avg=418.40, stdev=174.54, samples=30 iops : min= 52, max= 202, avg=104.47, stdev=43.63, samples=30 lat (msec) : 2=10.22%, 4=47.95%, 10=32.85%, 20=8.76%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=11.33%, ctx=3581, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25539: Sat Dec 28 03:02:17 2024 read: IOPS=100, BW=404KiB/s (413kB/s)(6028KiB/14939msec) clat (usec): min=1496, max=25518, avg=3987.55, stdev=3042.14 lat (usec): min=1496, max=25518, avg=3988.23, stdev=3042.10 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 1729], 10.00th=[ 1827], 20.00th=[ 1991], | 30.00th=[ 2180], 40.00th=[ 2376], 50.00th=[ 2704], 60.00th=[ 3163], | 70.00th=[ 3916], 80.00th=[ 5407], 90.00th=[ 8717], 95.00th=[10552], | 99.00th=[15008], 99.50th=[17171], 99.90th=[22676], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 160, max= 824, per=24.92%, avg=396.76, stdev=174.78, samples=29 iops : min= 40, max= 206, avg=99.10, stdev=43.64, samples=29 write: IOPS=104, BW=419KiB/s (429kB/s)(6260KiB/14939msec) clat (msec): min=2, max=165, avg= 5.65, stdev= 5.21 lat (msec): min=2, max=165, avg= 5.65, stdev= 5.21 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 10], 95.00th=[ 12], | 99.00th=[ 18], 99.50th=[ 21], 99.90th=[ 25], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 192, max= 776, per=24.55%, avg=409.17, stdev=169.37, samples=29 iops : min= 48, max= 194, avg=102.21, stdev=42.29, samples=29 lat (msec) : 2=10.03%, 4=48.70%, 10=33.17%, 20=7.71%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.18%, ctx=3570, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25541: Sat Dec 28 03:02:17 2024 read: IOPS=100, BW=402KiB/s (411kB/s)(6052KiB/15069msec) clat (usec): min=1459, max=19337, avg=4038.80, stdev=3066.29 lat (usec): min=1460, max=19338, avg=4039.41, stdev=3066.28 clat percentiles (usec): | 1.00th=[ 1598], 5.00th=[ 1713], 10.00th=[ 1827], 20.00th=[ 1975], | 30.00th=[ 2147], 40.00th=[ 2409], 50.00th=[ 2769], 60.00th=[ 3228], | 70.00th=[ 4047], 80.00th=[ 5473], 90.00th=[ 8979], 95.00th=[10945], | 99.00th=[15664], 99.50th=[16712], 99.90th=[18744], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 160, max= 800, per=25.10%, avg=399.67, stdev=172.82, samples=30 iops : min= 40, max= 200, avg=99.87, stdev=43.22, samples=30 write: IOPS=103, BW=414KiB/s (424kB/s)(6236KiB/15069msec) clat (msec): min=2, max=151, avg= 5.68, stdev= 5.02 lat (msec): min=2, max=151, avg= 5.69, stdev= 5.02 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 5], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 11], 95.00th=[ 12], | 99.00th=[ 19], 99.50th=[ 24], 99.90th=[ 28], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 192, max= 800, per=24.73%, avg=412.20, stdev=177.79, samples=30 iops : min= 48, max= 200, avg=103.00, stdev=44.48, samples=30 lat (msec) : 2=10.61%, 4=47.10%, 10=33.27%, 20=8.59%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=10.99%, ctx=3589, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1593KiB/s (1631kB/s), 394KiB/s-404KiB/s (403kB/s-413kB/s), io=23.4MiB (24.6MB), run=14939-15077msec WRITE: bw=1667KiB/s (1707kB/s), 414KiB/s-421KiB/s (424kB/s-431kB/s), io=24.6MiB (25.7MB), run=14939-15077msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25565: Sat Dec 28 03:02:24 2024 read: IOPS=62, BW=1001KiB/s (1025kB/s)(5920KiB/5914msec) clat (usec): min=60, max=56735, avg=11240.58, stdev=4399.75 lat (usec): min=61, max=56736, avg=11241.23, stdev=4399.78 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 7046], 10.00th=[ 7832], 20.00th=[ 8979], | 30.00th=[ 9634], 40.00th=[10290], 50.00th=[10814], 60.00th=[11338], | 70.00th=[12125], 80.00th=[13304], 90.00th=[15270], 95.00th=[17433], | 99.00th=[24511], 99.50th=[28705], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 672, max= 1248, per=26.09%, avg=986.00, stdev=168.72, samples=11 iops : min= 42, max= 78, avg=61.55, stdev=10.60, samples=11 write: IOPS=67, BW=1077KiB/s (1103kB/s)(6368KiB/5914msec) clat (usec): min=627, max=30540, avg=4312.96, stdev=4328.81 lat (usec): min=627, max=30541, avg=4313.88, stdev=4328.82 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 791], 10.00th=[ 816], 20.00th=[ 1074], | 30.00th=[ 2180], 40.00th=[ 2474], 50.00th=[ 2835], 60.00th=[ 3556], | 70.00th=[ 4424], 80.00th=[ 6063], 90.00th=[ 9503], 95.00th=[13698], | 99.00th=[22938], 99.50th=[25822], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 606, max= 1440, per=26.92%, avg=1070.36, stdev=285.60, samples=11 iops : min= 37, max= 90, avg=66.82, stdev=17.99, samples=11 lat (usec) : 100=0.78%, 250=0.65%, 750=0.39%, 1000=9.11% lat (msec) : 2=3.26%, 4=21.35%, 10=28.78%, 20=33.85%, 50=1.69% lat (msec) : 100=0.13% cpu : usr=0.15%, sys=21.00%, ctx=2798, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25567: Sat Dec 28 03:02:24 2024 read: IOPS=58, BW=944KiB/s (966kB/s)(5744KiB/6086msec) clat (usec): min=59, max=27889, avg=11213.70, stdev=3503.94 lat (usec): min=60, max=27889, avg=11214.34, stdev=3503.94 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 7242], 10.00th=[ 8094], 20.00th=[ 8979], | 30.00th=[ 9634], 40.00th=[10290], 50.00th=[10945], 60.00th=[11600], | 70.00th=[12125], 80.00th=[13435], 90.00th=[15270], 95.00th=[16319], | 99.00th=[23725], 99.50th=[27657], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 704, max= 1184, per=24.84%, avg=938.67, stdev=143.33, samples=12 iops : min= 44, max= 74, avg=58.67, stdev= 8.96, samples=12 write: IOPS=67, BW=1075KiB/s (1101kB/s)(6544KiB/6086msec) clat (usec): min=706, max=79306, avg=5006.78, stdev=5969.33 lat (usec): min=706, max=79308, avg=5007.77, stdev=5969.32 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 791], 10.00th=[ 832], 20.00th=[ 1172], | 30.00th=[ 2073], 40.00th=[ 2507], 50.00th=[ 2900], 60.00th=[ 4047], | 70.00th=[ 5800], 80.00th=[ 7635], 90.00th=[11863], 95.00th=[15270], | 99.00th=[22152], 99.50th=[25822], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 704, max= 1472, per=26.96%, avg=1072.00, stdev=206.26, samples=12 iops : min= 44, max= 92, avg=67.00, stdev=12.89, samples=12 lat (usec) : 100=0.52%, 250=0.26%, 500=0.13%, 750=0.52%, 1000=8.98% lat (msec) : 2=5.34%, 4=17.32%, 10=30.73%, 20=34.24%, 50=1.82% lat (msec) : 100=0.13% cpu : usr=0.08%, sys=20.48%, ctx=2744, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25569: Sat Dec 28 03:02:24 2024 read: IOPS=61, BW=989KiB/s (1013kB/s)(6256KiB/6324msec) clat (usec): min=58, max=34054, avg=10913.63, stdev=3581.72 lat (usec): min=58, max=34054, avg=10914.23, stdev=3581.75 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 7177], 10.00th=[ 8160], 20.00th=[ 8979], | 30.00th=[ 9503], 40.00th=[10028], 50.00th=[10945], 60.00th=[11469], | 70.00th=[12125], 80.00th=[12649], 90.00th=[14353], 95.00th=[15795], | 99.00th=[21365], 99.50th=[32113], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 576, max= 1184, per=25.90%, avg=978.67, stdev=203.06, samples=12 iops : min= 36, max= 74, avg=61.17, stdev=12.69, samples=12 write: IOPS=59, BW=954KiB/s (977kB/s)(6032KiB/6324msec) clat (usec): min=662, max=70956, avg=5242.52, stdev=6711.45 lat (usec): min=663, max=70957, avg=5243.49, stdev=6711.44 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 807], 10.00th=[ 857], 20.00th=[ 1319], | 30.00th=[ 2147], 40.00th=[ 2474], 50.00th=[ 2900], 60.00th=[ 3949], | 70.00th=[ 5276], 80.00th=[ 6718], 90.00th=[11994], 95.00th=[17433], | 99.00th=[32637], 99.50th=[40633], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 544, max= 1280, per=23.21%, avg=922.67, stdev=209.99, samples=12 iops : min= 34, max= 80, avg=57.67, stdev=13.12, samples=12 lat (usec) : 100=1.17%, 250=0.39%, 750=0.26%, 1000=7.68% lat (msec) : 2=5.34%, 4=16.67%, 10=31.25%, 20=35.16%, 50=1.95% lat (msec) : 100=0.13% cpu : usr=0.13%, sys=18.85%, ctx=2889, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25571: Sat Dec 28 03:02:24 2024 read: IOPS=59, BW=952KiB/s (975kB/s)(6032KiB/6337msec) clat (usec): min=62, max=26080, avg=11172.76, stdev=3426.22 lat (usec): min=62, max=26081, avg=11173.50, stdev=3426.24 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 7046], 10.00th=[ 7767], 20.00th=[ 8979], | 30.00th=[ 9503], 40.00th=[10290], 50.00th=[10945], 60.00th=[11600], | 70.00th=[12256], 80.00th=[13435], 90.00th=[14746], 95.00th=[16909], | 99.00th=[22938], 99.50th=[24773], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 606, max= 1152, per=24.83%, avg=938.33, stdev=159.09, samples=12 iops : min= 37, max= 72, avg=58.50, stdev=10.10, samples=12 write: IOPS=61, BW=987KiB/s (1011kB/s)(6256KiB/6337msec) clat (usec): min=632, max=91119, avg=5299.97, stdev=6854.10 lat (usec): min=632, max=91121, avg=5300.93, stdev=6854.12 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 791], 10.00th=[ 824], 20.00th=[ 1037], | 30.00th=[ 2147], 40.00th=[ 2671], 50.00th=[ 3163], 60.00th=[ 4178], | 70.00th=[ 5473], 80.00th=[ 6980], 90.00th=[12649], 95.00th=[16450], | 99.00th=[27657], 99.50th=[34866], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 766, max= 1376, per=24.74%, avg=983.67, stdev=194.22, samples=12 iops : min= 47, max= 86, avg=61.33, stdev=12.24, samples=12 lat (usec) : 100=0.65%, 500=0.39%, 750=0.52%, 1000=9.37% lat (msec) : 2=4.17%, 4=15.36%, 10=31.38%, 20=35.94%, 50=2.08% lat (msec) : 100=0.13% cpu : usr=0.09%, sys=19.00%, ctx=2904, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3780KiB/s (3870kB/s), 944KiB/s-1001KiB/s (966kB/s-1025kB/s), io=23.4MiB (24.5MB), run=5914-6337msec WRITE: bw=3977KiB/s (4072kB/s), 954KiB/s-1077KiB/s (977kB/s-1103kB/s), io=24.6MiB (25.8MB), run=5914-6337msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25564: Sat Dec 28 03:02:24 2024 read: IOPS=61, BW=978KiB/s (1002kB/s)(5920KiB/6052msec) clat (usec): min=1857, max=40181, avg=6739.26, stdev=5547.59 lat (usec): min=1858, max=40182, avg=6740.19, stdev=5547.59 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2089], 10.00th=[ 2212], 20.00th=[ 2474], | 30.00th=[ 2966], 40.00th=[ 3392], 50.00th=[ 4359], 60.00th=[ 6783], | 70.00th=[ 8455], 80.00th=[ 9896], 90.00th=[13042], 95.00th=[16909], | 99.00th=[28443], 99.50th=[39060], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 608, max= 1344, per=25.88%, avg=970.33, stdev=207.55, samples=12 iops : min= 38, max= 84, avg=60.50, stdev=12.99, samples=12 write: IOPS=65, BW=1052KiB/s (1077kB/s)(6368KiB/6052msec) clat (usec): min=2590, max=42753, avg=8893.04, stdev=5843.82 lat (usec): min=2591, max=42753, avg=8894.10, stdev=5843.76 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3228], 10.00th=[ 3589], 20.00th=[ 4146], | 30.00th=[ 4948], 40.00th=[ 5997], 50.00th=[ 7242], 60.00th=[ 8586], | 70.00th=[10028], 80.00th=[12649], 90.00th=[16712], 95.00th=[21627], | 99.00th=[29492], 99.50th=[36439], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 638, max= 1344, per=26.55%, avg=1047.58, stdev=262.68, samples=12 iops : min= 39, max= 84, avg=65.33, stdev=16.52, samples=12 lat (msec) : 2=1.30%, 4=30.34%, 10=43.23%, 20=20.96%, 50=4.17% cpu : usr=0.08%, sys=8.86%, ctx=1149, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25566: Sat Dec 28 03:02:24 2024 read: IOPS=58, BW=933KiB/s (956kB/s)(5744KiB/6154msec) clat (usec): min=1677, max=41085, avg=6615.10, stdev=5769.95 lat (usec): min=1677, max=41086, avg=6615.69, stdev=5769.96 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2147], 10.00th=[ 2245], 20.00th=[ 2540], | 30.00th=[ 2868], 40.00th=[ 3359], 50.00th=[ 4359], 60.00th=[ 6063], | 70.00th=[ 7898], 80.00th=[10028], 90.00th=[13173], 95.00th=[17695], | 99.00th=[31065], 99.50th=[34866], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 448, max= 1152, per=24.89%, avg=933.33, stdev=210.87, samples=12 iops : min= 28, max= 72, avg=58.33, stdev=13.18, samples=12 write: IOPS=66, BW=1063KiB/s (1089kB/s)(6544KiB/6154msec) clat (usec): min=2634, max=61860, avg=9046.00, stdev=6266.51 lat (usec): min=2636, max=61862, avg=9047.02, stdev=6266.51 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3326], 10.00th=[ 3687], 20.00th=[ 4359], | 30.00th=[ 5342], 40.00th=[ 6390], 50.00th=[ 7439], 60.00th=[ 8455], | 70.00th=[10159], 80.00th=[12780], 90.00th=[16909], 95.00th=[19792], | 99.00th=[31065], 99.50th=[38011], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 768, max= 1376, per=26.42%, avg=1042.67, stdev=193.19, samples=12 iops : min= 48, max= 86, avg=65.17, stdev=12.07, samples=12 lat (msec) : 2=1.43%, 4=29.04%, 10=43.88%, 20=21.87%, 50=3.65% lat (msec) : 100=0.13% cpu : usr=0.11%, sys=8.84%, ctx=1171, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25568: Sat Dec 28 03:02:24 2024 read: IOPS=61, BW=980KiB/s (1003kB/s)(6256KiB/6386msec) clat (usec): min=1758, max=33511, avg=6652.19, stdev=5242.06 lat (usec): min=1758, max=33512, avg=6652.79, stdev=5242.07 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2089], 10.00th=[ 2311], 20.00th=[ 2540], | 30.00th=[ 2933], 40.00th=[ 3359], 50.00th=[ 4490], 60.00th=[ 6783], | 70.00th=[ 8586], 80.00th=[ 9896], 90.00th=[13304], 95.00th=[16909], | 99.00th=[26870], 99.50th=[29230], 99.90th=[33424], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 512, max= 1280, per=25.96%, avg=973.33, stdev=222.73, samples=12 iops : min= 32, max= 80, avg=60.83, stdev=13.92, samples=12 write: IOPS=59, BW=945KiB/s (967kB/s)(6032KiB/6386msec) clat (usec): min=2460, max=79251, avg=9917.62, stdev=7361.27 lat (usec): min=2461, max=79253, avg=9918.78, stdev=7361.36 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3326], 10.00th=[ 3851], 20.00th=[ 4686], | 30.00th=[ 5800], 40.00th=[ 6718], 50.00th=[ 8291], 60.00th=[ 9503], | 70.00th=[11076], 80.00th=[13829], 90.00th=[17957], 95.00th=[21103], | 99.00th=[36963], 99.50th=[53216], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 608, max= 1280, per=23.31%, avg=920.00, stdev=180.05, samples=12 iops : min= 38, max= 80, avg=57.50, stdev=11.25, samples=12 lat (msec) : 2=1.17%, 4=29.43%, 10=41.41%, 20=23.44%, 50=4.30% lat (msec) : 100=0.26% cpu : usr=0.08%, sys=9.04%, ctx=1259, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25570: Sat Dec 28 03:02:24 2024 read: IOPS=59, BW=950KiB/s (973kB/s)(6032KiB/6351msec) clat (usec): min=1849, max=32086, avg=6885.05, stdev=5384.40 lat (usec): min=1849, max=32086, avg=6885.64, stdev=5384.40 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2114], 10.00th=[ 2343], 20.00th=[ 2671], | 30.00th=[ 3032], 40.00th=[ 3621], 50.00th=[ 4752], 60.00th=[ 6325], | 70.00th=[ 8717], 80.00th=[10552], 90.00th=[13829], 95.00th=[18220], | 99.00th=[26346], 99.50th=[27132], 99.90th=[32113], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 480, max= 1152, per=24.60%, avg=922.42, stdev=183.15, samples=12 iops : min= 30, max= 72, avg=57.58, stdev=11.36, samples=12 write: IOPS=61, BW=985KiB/s (1009kB/s)(6256KiB/6351msec) clat (msec): min=2, max=101, avg= 9.44, stdev= 7.39 lat (msec): min=2, max=101, avg= 9.44, stdev= 7.39 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 9], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 13], 90.00th=[ 17], 95.00th=[ 20], | 99.00th=[ 34], 99.50th=[ 41], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 608, max= 1216, per=24.66%, avg=973.17, stdev=185.90, samples=12 iops : min= 38, max= 76, avg=60.75, stdev=11.66, samples=12 lat (msec) : 2=1.04%, 4=28.12%, 10=42.32%, 20=24.48%, 50=3.91% lat (msec) : 250=0.13% cpu : usr=0.08%, sys=8.94%, ctx=1244, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3751KiB/s (3841kB/s), 933KiB/s-980KiB/s (956kB/s-1003kB/s), io=23.4MiB (24.5MB), run=6052-6386msec WRITE: bw=3946KiB/s (4041kB/s), 945KiB/s-1063KiB/s (967kB/s-1089kB/s), io=24.6MiB (25.8MB), run=6052-6386msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25582: Sat Dec 28 03:02:30 2024 read: IOPS=3, BW=3201KiB/s (3278kB/s)(7168KiB/2239msec) clat (msec): min=13, max=912, avg=171.30, stdev=327.56 lat (msec): min=13, max=912, avg=171.30, stdev=327.56 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 62], 40.00th=[ 62], 50.00th=[ 63], 60.00th=[ 67], | 70.00th=[ 67], 80.00th=[ 68], 90.00th=[ 911], 95.00th=[ 911], | 99.00th=[ 911], 99.50th=[ 911], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 4096, max= 6144, per=100.00%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2287KiB/s (2342kB/s)(5120KiB/2239msec) clat (msec): min=16, max=853, avg=196.96, stdev=367.06 lat (msec): min=16, max=853, avg=197.02, stdev=367.07 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 17], | 30.00th=[ 29], 40.00th=[ 29], 50.00th=[ 39], 60.00th=[ 39], | 70.00th=[ 49], 80.00th=[ 49], 90.00th=[ 852], 95.00th=[ 852], | 99.00th=[ 852], 99.50th=[ 852], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 4096, max= 4096, per=63.86%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=25.00%, 100=33.33%, 1000=16.67% cpu : usr=0.04%, sys=3.93%, ctx=49, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25583: Sat Dec 28 03:02:30 2024 read: IOPS=0, BW=528KiB/s (541kB/s)(2048KiB/3878msec) clat (usec): min=22653, max=57277, avg=39965.44, stdev=24483.19 lat (usec): min=22655, max=57278, avg=39966.75, stdev=24482.30 clat percentiles (usec): | 1.00th=[22676], 5.00th=[22676], 10.00th=[22676], 20.00th=[22676], | 30.00th=[22676], 40.00th=[22676], 50.00th=[22676], 60.00th=[57410], | 70.00th=[57410], 80.00th=[57410], 90.00th=[57410], 95.00th=[57410], | 99.00th=[57410], 99.50th=[57410], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 2048, max= 2048, per=44.71%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2641KiB/s (2704kB/s)(10.0MiB/3878msec) clat (msec): min=15, max=2350, avg=378.01, stdev=748.38 lat (msec): min=16, max=2350, avg=378.08, stdev=748.38 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 21], | 30.00th=[ 28], 40.00th=[ 33], 50.00th=[ 33], 60.00th=[ 49], | 70.00th=[ 220], 80.00th=[ 220], 90.00th=[ 944], 95.00th=[ 2366], | 99.00th=[ 2366], 99.50th=[ 2366], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 4096, max=12288, per=100.00%, avg=8192.00, stdev=5792.62, samples=2 iops : min= 4, max= 12, avg= 8.00, stdev= 5.66, samples=2 lat (msec) : 20=8.33%, 50=50.00%, 100=16.67%, 250=8.33%, 1000=8.33% cpu : usr=0.05%, sys=3.30%, ctx=74, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25584: Sat Dec 28 03:02:30 2024 read: IOPS=1, BW=1374KiB/s (1407kB/s)(6144KiB/4470msec) clat (msec): min=46, max=1383, avg=501.33, stdev=542.72 lat (msec): min=46, max=1383, avg=501.34, stdev=542.72 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 51], | 30.00th=[ 51], 40.00th=[ 78], 50.00th=[ 78], 60.00th=[ 684], | 70.00th=[ 768], 80.00th=[ 768], 90.00th=[ 1385], 95.00th=[ 1385], | 99.00th=[ 1385], 99.50th=[ 1385], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 1896, max= 6144, per=73.40%, avg=3362.67, stdev=2409.90, samples=3 iops : min= 1, max= 6, avg= 3.00, stdev= 2.65, samples=3 write: IOPS=1, BW=1374KiB/s (1407kB/s)(6144KiB/4470msec) clat (msec): min=14, max=1060, avg=231.69, stdev=411.09 lat (msec): min=14, max=1061, avg=231.74, stdev=411.10 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 34], 50.00th=[ 34], 60.00th=[ 61], | 70.00th=[ 188], 80.00th=[ 188], 90.00th=[ 1062], 95.00th=[ 1062], | 99.00th=[ 1062], 99.50th=[ 1062], 99.90th=[ 1062], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 1896, max= 4096, per=47.30%, avg=3034.00, stdev=1227.86, samples=4 iops : min= 1, max= 4, avg= 2.75, stdev= 1.50, samples=4 lat (msec) : 20=8.33%, 50=25.00%, 100=25.00%, 250=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.02%, sys=2.48%, ctx=146, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25586: Sat Dec 28 03:02:30 2024 read: IOPS=1, BW=1963KiB/s (2010kB/s)(5120KiB/2608msec) clat (msec): min=17, max=1280, avg=286.33, stdev=556.70 lat (msec): min=17, max=1280, avg=286.33, stdev=556.70 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 22], | 70.00th=[ 93], 80.00th=[ 93], 90.00th=[ 1284], 95.00th=[ 1284], | 99.00th=[ 1284], 99.50th=[ 1284], 99.90th=[ 1284], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2748KiB/s (2814kB/s)(7168KiB/2608msec) clat (msec): min=14, max=972, avg=166.34, stdev=355.85 lat (msec): min=14, max=973, avg=166.41, stdev=355.85 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 23], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 41], 60.00th=[ 42], | 70.00th=[ 42], 80.00th=[ 44], 90.00th=[ 969], 95.00th=[ 969], | 99.00th=[ 969], 99.50th=[ 969], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 6144, max= 8192, per=100.00%, avg=7168.00, stdev=1448.15, samples=2 iops : min= 6, max= 8, avg= 7.00, stdev= 1.41, samples=2 lat (msec) : 20=25.00%, 50=50.00%, 100=8.33%, 1000=8.33% cpu : usr=0.00%, sys=3.15%, ctx=47, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4582KiB/s (4692kB/s), 528KiB/s-3201KiB/s (541kB/s-3278kB/s), io=20.0MiB (20.0MB), run=2239-4470msec WRITE: bw=6414KiB/s (6568kB/s), 1374KiB/s-2748KiB/s (1407kB/s-2814kB/s), io=28.0MiB (29.4MB), run=2239-4470msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25585: Sat Dec 28 03:02:30 2024 read: IOPS=1, BW=2013KiB/s (2062kB/s)(7168KiB/3560msec) clat (usec): min=404, max=917275, avg=408575.39, stdev=453287.53 lat (usec): min=405, max=917275, avg=408576.93, stdev=453287.16 clat percentiles (usec): | 1.00th=[ 404], 5.00th=[ 404], 10.00th=[ 404], 20.00th=[ 498], | 30.00th=[ 60556], 40.00th=[ 60556], 50.00th=[127402], 60.00th=[851444], | 70.00th=[851444], 80.00th=[910164], 90.00th=[918553], 95.00th=[918553], | 99.00th=[918553], 99.50th=[918553], 99.90th=[918553], 99.95th=[918553], | 99.99th=[918553] bw ( KiB/s): min= 2003, max= 4096, per=54.45%, avg=2448.60, stdev=921.13, samples=5 iops : min= 1, max= 4, avg= 2.20, stdev= 1.10, samples=5 write: IOPS=1, BW=1438KiB/s (1473kB/s)(5120KiB/3560msec) clat (msec): min=31, max=368, avg=128.75, stdev=138.77 lat (msec): min=31, max=368, avg=128.83, stdev=138.75 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 74], 60.00th=[ 74], | 70.00th=[ 126], 80.00th=[ 126], 90.00th=[ 368], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 2048, max= 4096, per=43.37%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 lat (usec) : 500=16.67% lat (msec) : 50=16.67%, 100=16.67%, 250=16.67%, 500=8.33%, 1000=25.00% cpu : usr=0.03%, sys=18.49%, ctx=970, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25587: Sat Dec 28 03:02:30 2024 read: IOPS=0, BW=450KiB/s (461kB/s)(2048KiB/4554msec) clat (msec): min=41, max=1273, avg=657.39, stdev=870.64 lat (msec): min=41, max=1273, avg=657.39, stdev=870.64 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 42], 50.00th=[ 42], 60.00th=[ 1267], | 70.00th=[ 1267], 80.00th=[ 1267], 90.00th=[ 1267], 95.00th=[ 1267], | 99.00th=[ 1267], 99.50th=[ 1267], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 2048, max= 2048, per=45.54%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2249KiB/s (2303kB/s)(10.0MiB/4554msec) clat (msec): min=24, max=1098, avg=312.66, stdev=443.42 lat (msec): min=24, max=1098, avg=312.73, stdev=443.40 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 33], | 30.00th=[ 37], 40.00th=[ 39], 50.00th=[ 41], 60.00th=[ 42], | 70.00th=[ 802], 80.00th=[ 802], 90.00th=[ 944], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 2048, max=10240, per=86.15%, avg=5424.00, stdev=4281.64, samples=3 iops : min= 2, max= 10, avg= 5.00, stdev= 4.36, samples=3 lat (msec) : 50=58.33%, 100=8.33%, 1000=16.67% cpu : usr=0.04%, sys=21.94%, ctx=448, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25588: Sat Dec 28 03:02:30 2024 read: IOPS=1, BW=1386KiB/s (1420kB/s)(6144KiB/4432msec) clat (usec): min=381, max=1825.2k, avg=510553.53, stdev=702296.38 lat (usec): min=382, max=1825.2k, avg=510554.58, stdev=702296.26 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 383], 10.00th=[ 383], | 20.00th=[ 31851], 30.00th=[ 31851], 40.00th=[ 68682], | 50.00th=[ 68682], 60.00th=[ 417334], 70.00th=[ 717226], | 80.00th=[ 717226], 90.00th=[1820328], 95.00th=[1820328], | 99.00th=[1820328], 99.50th=[1820328], 99.90th=[1820328], | 99.95th=[1820328], 99.99th=[1820328] bw ( KiB/s): min= 2048, max= 6144, per=75.90%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 write: IOPS=1, BW=1386KiB/s (1420kB/s)(6144KiB/4432msec) clat (msec): min=28, max=1027, avg=226.01, stdev=397.29 lat (msec): min=28, max=1027, avg=226.08, stdev=397.30 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 41], 50.00th=[ 41], 60.00th=[ 43], | 70.00th=[ 188], 80.00th=[ 188], 90.00th=[ 1028], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 2048, max= 4096, per=48.79%, avg=3072.00, stdev=1182.41, samples=4 iops : min= 2, max= 4, avg= 3.00, stdev= 1.15, samples=4 lat (usec) : 500=8.33% lat (msec) : 50=41.67%, 100=8.33%, 250=8.33%, 500=8.33%, 750=8.33% cpu : usr=0.05%, sys=9.37%, ctx=804, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25589: Sat Dec 28 03:02:30 2024 read: IOPS=1, BW=1488KiB/s (1523kB/s)(5120KiB/3442msec) clat (usec): min=369, max=852679, avg=341794.89, stdev=439105.17 lat (usec): min=370, max=852680, avg=341795.66, stdev=439105.28 clat percentiles (usec): | 1.00th=[ 371], 5.00th=[ 371], 10.00th=[ 371], 20.00th=[ 371], | 30.00th=[ 375], 40.00th=[ 375], 50.00th=[ 65799], 60.00th=[ 65799], | 70.00th=[792724], 80.00th=[792724], 90.00th=[851444], 95.00th=[851444], | 99.00th=[851444], 99.50th=[851444], 99.90th=[851444], 99.95th=[851444], | 99.99th=[851444] bw ( KiB/s): min= 2048, max= 6144, per=91.08%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=2, BW=2083KiB/s (2132kB/s)(7168KiB/3442msec) clat (msec): min=19, max=1031, avg=239.72, stdev=371.50 lat (msec): min=19, max=1031, avg=239.77, stdev=371.51 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 27], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 62], 60.00th=[ 133], | 70.00th=[ 133], 80.00th=[ 380], 90.00th=[ 1028], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 2048, max= 6144, per=56.93%, avg=3584.00, stdev=1960.81, samples=4 iops : min= 2, max= 6, avg= 3.50, stdev= 1.91, samples=4 lat (usec) : 500=16.67% lat (msec) : 20=8.33%, 50=16.67%, 100=16.67%, 250=8.33%, 500=8.33% lat (msec) : 1000=16.67% cpu : usr=0.03%, sys=13.40%, ctx=591, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4497KiB/s (4605kB/s), 450KiB/s-2013KiB/s (461kB/s-2062kB/s), io=20.0MiB (20.0MB), run=3442-4554msec WRITE: bw=6296KiB/s (6447kB/s), 1386KiB/s-2249KiB/s (1420kB/s-2303kB/s), io=28.0MiB (29.4MB), run=3442-4554msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25598: Sat Dec 28 03:02:34 2024 read: IOPS=1, BW=4123KiB/s (4222kB/s)(12.0MiB/2980msec) clat (usec): min=1890, max=2759.6k, avg=985898.31, stdev=1539170.32 lat (usec): min=1891, max=2759.6k, avg=985899.52, stdev=1539169.72 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 1893], 10.00th=[ 1893], | 20.00th=[ 1893], 30.00th=[ 1893], 40.00th=[ 196084], | 50.00th=[ 196084], 60.00th=[ 196084], 70.00th=[2768241], | 80.00th=[2768241], 90.00th=[2768241], 95.00th=[2768241], | 99.00th=[2768241], 99.50th=[2768241], 99.90th=[2768241], | 99.95th=[2768241], 99.99th=[2768241] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 2=33.33%, 250=33.33% cpu : usr=0.03%, sys=41.39%, ctx=660, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25599: Sat Dec 28 03:02:34 2024 read: IOPS=0, BW=1335KiB/s (1367kB/s)(4096KiB/3068msec) clat (nsec): min=197756k, max=197756k, avg=197755507.00, stdev= 0.00 lat (nsec): min=197758k, max=197758k, avg=197757798.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 199], 10.00th=[ 199], 20.00th=[ 199], | 30.00th=[ 199], 40.00th=[ 199], 50.00th=[ 199], 60.00th=[ 199], | 70.00th=[ 199], 80.00th=[ 199], 90.00th=[ 199], 95.00th=[ 199], | 99.00th=[ 199], 99.50th=[ 199], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2670KiB/s (2734kB/s)(8192KiB/3068msec) clat (msec): min=99, max=2762, avg=1430.99, stdev=1882.56 lat (msec): min=100, max=2762, avg=1431.42, stdev=1882.40 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 101], 10.00th=[ 101], 20.00th=[ 101], | 30.00th=[ 101], 40.00th=[ 101], 50.00th=[ 101], 60.00th=[ 2769], | 70.00th=[ 2769], 80.00th=[ 2769], 90.00th=[ 2769], 95.00th=[ 2769], | 99.00th=[ 2769], 99.50th=[ 2769], 99.90th=[ 2769], 99.95th=[ 2769], | 99.99th=[ 2769] bw ( KiB/s): min= 8192, max= 8192, per=76.70%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.03%, sys=12.85%, ctx=378, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25600: Sat Dec 28 03:02:34 2024 write: IOPS=0, BW=4039KiB/s (4136kB/s)(12.0MiB/3042msec) clat (msec): min=121, max=2742, avg=1006.71, stdev=1503.50 lat (msec): min=121, max=2742, avg=1006.95, stdev=1503.55 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 122], 10.00th=[ 122], 20.00th=[ 122], | 30.00th=[ 122], 40.00th=[ 157], 50.00th=[ 157], 60.00th=[ 157], | 70.00th=[ 2735], 80.00th=[ 2735], 90.00th=[ 2735], 95.00th=[ 2735], | 99.00th=[ 2735], 99.50th=[ 2735], 99.90th=[ 2735], 99.95th=[ 2735], | 99.99th=[ 2735] bw ( KiB/s): min=16351, max=16351, per=100.00%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=8.39%, ctx=40, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25601: Sat Dec 28 03:02:34 2024 write: IOPS=5, BW=22.6MiB/s (23.7MB/s)(12.0MiB/530msec) clat (msec): min=83, max=303, avg=167.37, stdev=118.99 lat (msec): min=83, max=303, avg=167.69, stdev=119.00 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 84], 10.00th=[ 84], 20.00th=[ 84], | 30.00th=[ 84], 40.00th=[ 115], 50.00th=[ 115], 60.00th=[ 115], | 70.00th=[ 305], 80.00th=[ 305], 90.00th=[ 305], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.19%, sys=48.02%, ctx=79, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5340KiB/s (5468kB/s), 1335KiB/s-4123KiB/s (1367kB/s-4222kB/s), io=16.0MiB (16.8MB), run=2980-3068msec WRITE: bw=10.4MiB/s (10.9MB/s), 2670KiB/s-22.6MiB/s (2734kB/s-23.7MB/s), io=32.0MiB (33.6MB), run=530-3068msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25602: Sat Dec 28 03:02:34 2024 read: IOPS=0, BW=3558KiB/s (3643kB/s)(12.0MiB/3454msec) clat (msec): min=195, max=2933, avg=1131.21, stdev=1561.35 lat (msec): min=195, max=2933, avg=1131.21, stdev=1561.35 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 197], 10.00th=[ 197], 20.00th=[ 197], | 30.00th=[ 197], 40.00th=[ 266], 50.00th=[ 266], 60.00th=[ 266], | 70.00th=[ 2937], 80.00th=[ 2937], 90.00th=[ 2937], 95.00th=[ 2937], | 99.00th=[ 2937], 99.50th=[ 2937], 99.90th=[ 2937], 99.95th=[ 2937], | 99.99th=[ 2937] bw ( KiB/s): min= 8143, max= 8192, per=100.00%, avg=8167.50, stdev=34.65, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=2.29%, ctx=43, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25603: Sat Dec 28 03:02:34 2024 read: IOPS=0, BW=1200KiB/s (1229kB/s)(4096KiB/3414msec) clat (nsec): min=297541k, max=297541k, avg=297540547.00, stdev= 0.00 lat (nsec): min=297542k, max=297542k, avg=297541615.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 296], 10.00th=[ 296], 20.00th=[ 296], | 30.00th=[ 296], 40.00th=[ 296], 50.00th=[ 296], 60.00th=[ 296], | 70.00th=[ 296], 80.00th=[ 296], 90.00th=[ 296], 95.00th=[ 296], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2400KiB/s (2457kB/s)(8192KiB/3414msec) clat (msec): min=46, max=3009, avg=1528.26, stdev=2095.35 lat (msec): min=46, max=3010, avg=1528.60, stdev=2095.31 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 3004], | 70.00th=[ 3004], 80.00th=[ 3004], 90.00th=[ 3004], 95.00th=[ 3004], | 99.00th=[ 3004], 99.50th=[ 3004], 99.90th=[ 3004], 99.95th=[ 3004], | 99.99th=[ 3004] lat (msec) : 50=33.33%, 500=33.33% cpu : usr=0.03%, sys=5.01%, ctx=42, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25604: Sat Dec 28 03:02:34 2024 write: IOPS=0, BW=3577KiB/s (3663kB/s)(12.0MiB/3435msec) clat (msec): min=183, max=2934, avg=1132.96, stdev=1560.90 lat (msec): min=184, max=2934, avg=1133.26, stdev=1560.91 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 279], 50.00th=[ 279], 60.00th=[ 279], | 70.00th=[ 2937], 80.00th=[ 2937], 90.00th=[ 2937], 95.00th=[ 2937], | 99.00th=[ 2937], 99.50th=[ 2937], 99.90th=[ 2937], 99.95th=[ 2937], | 99.99th=[ 2937] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.12%, sys=3.38%, ctx=107, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25605: Sat Dec 28 03:02:34 2024 write: IOPS=5, BW=20.3MiB/s (21.3MB/s)(12.0MiB/592msec) clat (msec): min=125, max=223, avg=178.10, stdev=49.38 lat (msec): min=125, max=223, avg=178.39, stdev=49.38 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 126], 10.00th=[ 126], 20.00th=[ 126], | 30.00th=[ 126], 40.00th=[ 186], 50.00th=[ 186], 60.00th=[ 186], | 70.00th=[ 224], 80.00th=[ 224], 90.00th=[ 224], 95.00th=[ 224], | 99.00th=[ 224], 99.50th=[ 224], 99.90th=[ 224], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.17%, sys=26.73%, ctx=75, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4743KiB/s (4857kB/s), 1200KiB/s-3558KiB/s (1229kB/s-3643kB/s), io=16.0MiB (16.8MB), run=3414-3454msec WRITE: bw=9539KiB/s (9768kB/s), 2400KiB/s-20.3MiB/s (2457kB/s-21.3MB/s), io=32.0MiB (33.6MB), run=592-3435msec