== sanity test 398b: DIO and buffer IO race ============== 21:04:43 (1745629483) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.14735 s, 43.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10656: Fri Apr 25 21:04:58 2025 read: IOPS=129, BW=517KiB/s (530kB/s)(5992KiB/11582msec) clat (usec): min=50, max=159830, avg=3469.50, stdev=5140.82 lat (usec): min=50, max=159831, avg=3470.13, stdev=5140.81 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 69], 10.00th=[ 78], 20.00th=[ 151], | 30.00th=[ 2057], 40.00th=[ 2376], 50.00th=[ 2737], 60.00th=[ 3228], | 70.00th=[ 4015], 80.00th=[ 5211], 90.00th=[ 6783], 95.00th=[ 8356], | 99.00th=[ 14222], 99.50th=[ 21365], 99.90th=[ 38011], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 312, max= 920, per=24.81%, avg=514.26, stdev=139.16, samples=23 iops : min= 78, max= 230, avg=128.52, stdev=34.77, samples=23 write: IOPS=135, BW=544KiB/s (557kB/s)(6296KiB/11582msec) clat (usec): min=464, max=44373, avg=3991.16, stdev=4132.56 lat (usec): min=465, max=44374, avg=3992.02, stdev=4132.45 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 603], | 30.00th=[ 824], 40.00th=[ 1975], 50.00th=[ 3130], 60.00th=[ 3949], | 70.00th=[ 5080], 80.00th=[ 6521], 90.00th=[ 8979], 95.00th=[11600], | 99.00th=[16712], 99.50th=[22152], 99.90th=[40633], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 368, max= 856, per=24.77%, avg=537.57, stdev=125.19, samples=23 iops : min= 92, max= 214, avg=134.35, stdev=31.28, samples=23 lat (usec) : 100=8.95%, 250=1.01%, 500=1.04%, 750=13.35%, 1000=2.54% lat (msec) : 2=7.26%, 4=31.02%, 10=29.49%, 20=4.75%, 50=0.55% lat (msec) : 250=0.03% cpu : usr=0.27%, sys=22.99%, ctx=5416, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10658: Fri Apr 25 21:04:58 2025 read: IOPS=128, BW=514KiB/s (527kB/s)(5940KiB/11549msec) clat (usec): min=53, max=34040, avg=3288.08, stdev=2945.41 lat (usec): min=54, max=34041, avg=3288.68, stdev=2945.43 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 69], 10.00th=[ 79], 20.00th=[ 245], | 30.00th=[ 2114], 40.00th=[ 2376], 50.00th=[ 2769], 60.00th=[ 3163], | 70.00th=[ 3884], 80.00th=[ 5145], 90.00th=[ 6783], 95.00th=[ 8029], | 99.00th=[11731], 99.50th=[17695], 99.90th=[30802], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 296, max= 750, per=24.18%, avg=501.18, stdev=120.57, samples=22 iops : min= 74, max= 187, avg=125.23, stdev=30.07, samples=22 write: IOPS=137, BW=550KiB/s (563kB/s)(6348KiB/11549msec) clat (usec): min=433, max=159503, avg=4134.57, stdev=5890.34 lat (usec): min=434, max=159504, avg=4135.42, stdev=5890.48 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 619], | 30.00th=[ 857], 40.00th=[ 1745], 50.00th=[ 3163], 60.00th=[ 3982], | 70.00th=[ 5014], 80.00th=[ 6521], 90.00th=[ 9110], 95.00th=[ 11600], | 99.00th=[ 17957], 99.50th=[ 22938], 99.90th=[ 73925], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 352, max= 792, per=24.61%, avg=533.95, stdev=112.71, samples=22 iops : min= 88, max= 198, avg=133.41, stdev=28.18, samples=22 lat (usec) : 100=8.33%, 250=1.37%, 500=0.75%, 750=13.05%, 1000=3.42% lat (msec) : 2=7.52%, 4=31.02%, 10=29.56%, 20=4.46%, 50=0.46% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.16%, sys=23.10%, ctx=5396, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10661: Fri Apr 25 21:04:58 2025 read: IOPS=130, BW=522KiB/s (535kB/s)(6028KiB/11538msec) clat (usec): min=53, max=54081, avg=3418.48, stdev=3515.02 lat (usec): min=54, max=54081, avg=3419.06, stdev=3515.02 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 69], 10.00th=[ 78], 20.00th=[ 1647], | 30.00th=[ 2089], 40.00th=[ 2376], 50.00th=[ 2704], 60.00th=[ 3163], | 70.00th=[ 4015], 80.00th=[ 5276], 90.00th=[ 6849], 95.00th=[ 8717], | 99.00th=[12125], 99.50th=[22414], 99.90th=[42730], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 256, max= 696, per=24.47%, avg=507.18, stdev=125.89, samples=22 iops : min= 64, max= 174, avg=126.77, stdev=31.46, samples=22 write: IOPS=135, BW=543KiB/s (556kB/s)(6260KiB/11538msec) clat (usec): min=433, max=155877, avg=4013.65, stdev=5591.03 lat (usec): min=434, max=155878, avg=4014.36, stdev=5591.04 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 603], | 30.00th=[ 783], 40.00th=[ 1369], 50.00th=[ 2868], 60.00th=[ 3851], | 70.00th=[ 5014], 80.00th=[ 6718], 90.00th=[ 9110], 95.00th=[ 11207], | 99.00th=[ 19530], 99.50th=[ 22676], 99.90th=[ 38536], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 376, max= 704, per=24.26%, avg=526.45, stdev=92.14, samples=22 iops : min= 94, max= 176, avg=131.59, stdev=23.00, samples=22 lat (usec) : 100=7.85%, 250=1.56%, 500=1.04%, 750=14.00%, 1000=3.39% lat (msec) : 2=7.42%, 4=30.21%, 10=29.69%, 20=4.13%, 50=0.65% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.21%, sys=23.21%, ctx=5428, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10663: Fri Apr 25 21:04:58 2025 read: IOPS=131, BW=526KiB/s (539kB/s)(6052KiB/11497msec) clat (usec): min=54, max=158451, avg=3411.38, stdev=5143.64 lat (usec): min=54, max=158452, avg=3412.23, stdev=5143.78 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 69], 10.00th=[ 79], 20.00th=[ 668], | 30.00th=[ 2057], 40.00th=[ 2376], 50.00th=[ 2704], 60.00th=[ 3097], | 70.00th=[ 3785], 80.00th=[ 5014], 90.00th=[ 6980], 95.00th=[ 8356], | 99.00th=[ 11731], 99.50th=[ 16188], 99.90th=[ 58983], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 304, max= 720, per=24.75%, avg=513.05, stdev=104.60, samples=22 iops : min= 76, max= 180, avg=128.23, stdev=26.19, samples=22 write: IOPS=135, BW=542KiB/s (555kB/s)(6236KiB/11497msec) clat (usec): min=463, max=46242, avg=3998.62, stdev=4162.21 lat (usec): min=463, max=46243, avg=3999.53, stdev=4162.03 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 611], | 30.00th=[ 840], 40.00th=[ 1631], 50.00th=[ 3064], 60.00th=[ 3982], | 70.00th=[ 5145], 80.00th=[ 6521], 90.00th=[ 8979], 95.00th=[11469], | 99.00th=[17695], 99.50th=[24511], 99.90th=[39060], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 392, max= 696, per=24.28%, avg=526.86, stdev=92.85, samples=22 iops : min= 98, max= 174, avg=131.68, stdev=23.24, samples=22 lat (usec) : 100=7.94%, 250=1.60%, 500=1.07%, 750=12.89%, 1000=3.29% lat (msec) : 2=7.81%, 4=31.41%, 10=28.97%, 20=4.46%, 50=0.49% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.10%, sys=22.94%, ctx=5460, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2073KiB/s (2123kB/s), 514KiB/s-526KiB/s (527kB/s-539kB/s), io=23.4MiB (24.6MB), run=11497-11582msec WRITE: bw=2171KiB/s (2223kB/s), 542KiB/s-550KiB/s (555kB/s-563kB/s), io=24.6MiB (25.7MB), run=11497-11582msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10657: Fri Apr 25 21:05:22 2025 read: IOPS=42, BW=169KiB/s (173kB/s)(5992KiB/35365msec) clat (usec): min=1640, max=162457, avg=3582.99, stdev=5234.07 lat (usec): min=1640, max=162458, avg=3583.94, stdev=5234.05 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1811], 10.00th=[ 1876], 20.00th=[ 1958], | 30.00th=[ 2040], 40.00th=[ 2180], 50.00th=[ 2409], 60.00th=[ 2868], | 70.00th=[ 3163], 80.00th=[ 3621], 90.00th=[ 6587], 95.00th=[ 9241], | 99.00th=[ 15926], 99.50th=[ 21103], 99.90th=[ 49021], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 56, max= 312, per=25.03%, avg=169.17, stdev=63.48, samples=70 iops : min= 14, max= 78, avg=42.14, stdev=15.88, samples=70 write: IOPS=44, BW=178KiB/s (182kB/s)(6296KiB/35365msec) clat (usec): min=8792, max=81681, avg=18988.58, stdev=7213.29 lat (usec): min=8792, max=81681, avg=18989.53, stdev=7213.22 clat percentiles (usec): | 1.00th=[10159], 5.00th=[11207], 10.00th=[11994], 20.00th=[13042], | 30.00th=[14353], 40.00th=[16319], 50.00th=[17957], 60.00th=[19268], | 70.00th=[20579], 80.00th=[22414], 90.00th=[28181], 95.00th=[33817], | 99.00th=[43779], 99.50th=[46924], 99.90th=[76022], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 56, max= 248, per=25.07%, avg=177.51, stdev=50.86, samples=70 iops : min= 14, max= 62, avg=44.23, stdev=12.73, samples=70 lat (msec) : 2=12.50%, 4=27.83%, 10=6.74%, 20=35.48%, 50=17.29% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.10%, sys=5.92%, ctx=3491, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10659: Fri Apr 25 21:05:22 2025 read: IOPS=41, BW=167KiB/s (171kB/s)(5940KiB/35491msec) clat (usec): min=1646, max=162467, avg=3537.23, stdev=5303.41 lat (usec): min=1646, max=162469, avg=3538.00, stdev=5303.45 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1827], 10.00th=[ 1876], 20.00th=[ 1958], | 30.00th=[ 2057], 40.00th=[ 2212], 50.00th=[ 2442], 60.00th=[ 2900], | 70.00th=[ 3163], 80.00th=[ 3687], 90.00th=[ 6390], 95.00th=[ 9110], | 99.00th=[ 14484], 99.50th=[ 16319], 99.90th=[ 77071], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 16, max= 312, per=24.75%, avg=167.34, stdev=67.31, samples=70 iops : min= 4, max= 78, avg=41.77, stdev=16.85, samples=70 write: IOPS=44, BW=179KiB/s (183kB/s)(6348KiB/35491msec) clat (usec): min=9030, max=75090, avg=18984.86, stdev=7104.93 lat (usec): min=9031, max=75091, avg=18985.74, stdev=7104.90 clat percentiles (usec): | 1.00th=[10159], 5.00th=[11338], 10.00th=[11863], 20.00th=[13042], | 30.00th=[14484], 40.00th=[16319], 50.00th=[17957], 60.00th=[19006], | 70.00th=[20579], 80.00th=[22414], 90.00th=[29230], 95.00th=[33817], | 99.00th=[42206], 99.50th=[45876], 99.90th=[60031], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 64, max= 240, per=25.14%, avg=177.97, stdev=49.42, samples=70 iops : min= 16, max= 60, avg=44.43, stdev=12.37, samples=70 lat (msec) : 2=12.24%, 4=27.51%, 10=7.36%, 20=35.58%, 50=17.19% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.10%, sys=5.87%, ctx=3520, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10660: Fri Apr 25 21:05:22 2025 read: IOPS=42, BW=170KiB/s (174kB/s)(6028KiB/35416msec) clat (usec): min=1614, max=43276, avg=3525.24, stdev=3072.47 lat (usec): min=1614, max=43277, avg=3526.02, stdev=3072.45 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1827], 10.00th=[ 1893], 20.00th=[ 1975], | 30.00th=[ 2073], 40.00th=[ 2212], 50.00th=[ 2442], 60.00th=[ 2868], | 70.00th=[ 3195], 80.00th=[ 3752], 90.00th=[ 6849], 95.00th=[ 9765], | 99.00th=[15270], 99.50th=[19268], 99.90th=[32375], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 48, max= 352, per=25.16%, avg=170.09, stdev=61.97, samples=70 iops : min= 12, max= 88, avg=42.46, stdev=15.50, samples=70 write: IOPS=44, BW=177KiB/s (181kB/s)(6260KiB/35416msec) clat (msec): min=9, max=200, avg=19.16, stdev= 8.52 lat (msec): min=9, max=200, avg=19.17, stdev= 8.52 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 29], 95.00th=[ 34], | 99.00th=[ 44], 99.50th=[ 48], 99.90th=[ 82], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 56, max= 239, per=24.88%, avg=176.14, stdev=49.74, samples=70 iops : min= 14, max= 59, avg=43.97, stdev=12.45, samples=70 lat (msec) : 2=11.52%, 4=28.39%, 10=7.06%, 20=36.13%, 50=16.70% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.11%, sys=5.84%, ctx=3492, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10662: Fri Apr 25 21:05:22 2025 read: IOPS=43, BW=173KiB/s (177kB/s)(6052KiB/35081msec) clat (usec): min=1665, max=49110, avg=3499.88, stdev=3023.67 lat (usec): min=1666, max=49110, avg=3500.61, stdev=3023.67 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1827], 10.00th=[ 1876], 20.00th=[ 1958], | 30.00th=[ 2040], 40.00th=[ 2180], 50.00th=[ 2442], 60.00th=[ 2835], | 70.00th=[ 3163], 80.00th=[ 3687], 90.00th=[ 6849], 95.00th=[ 9634], | 99.00th=[14877], 99.50th=[18482], 99.90th=[27657], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 48, max= 360, per=25.50%, avg=172.37, stdev=65.77, samples=70 iops : min= 12, max= 90, avg=43.03, stdev=16.46, samples=70 write: IOPS=44, BW=178KiB/s (182kB/s)(6236KiB/35081msec) clat (msec): min=8, max=184, avg=19.04, stdev= 8.40 lat (msec): min=8, max=184, avg=19.04, stdev= 8.40 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 28], 95.00th=[ 34], | 99.00th=[ 45], 99.50th=[ 50], 99.90th=[ 82], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 56, max= 240, per=25.06%, avg=177.40, stdev=49.18, samples=70 iops : min= 14, max= 60, avg=44.29, stdev=12.31, samples=70 lat (msec) : 2=12.66%, 4=27.70%, 10=7.13%, 20=35.38%, 50=16.93% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.11%, sys=5.90%, ctx=3459, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=677KiB/s (693kB/s), 167KiB/s-173KiB/s (171kB/s-177kB/s), io=23.4MiB (24.6MB), run=35081-35491msec WRITE: bw=708KiB/s (725kB/s), 177KiB/s-179KiB/s (181kB/s-183kB/s), io=24.6MiB (25.7MB), run=35081-35491msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10708: Fri Apr 25 21:05:25 2025 read: IOPS=148, BW=2372KiB/s (2429kB/s)(5920KiB/2496msec) clat (usec): min=54, max=22775, avg=2638.93, stdev=2605.77 lat (usec): min=55, max=22775, avg=2640.38, stdev=2606.04 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 67], 10.00th=[ 71], 20.00th=[ 82], | 30.00th=[ 96], 40.00th=[ 2212], 50.00th=[ 2573], 60.00th=[ 2933], | 70.00th=[ 3392], 80.00th=[ 4555], 90.00th=[ 5800], 95.00th=[ 6718], | 99.00th=[ 9372], 99.50th=[15926], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 1184, max= 2906, per=24.78%, avg=2342.50, stdev=787.77, samples=4 iops : min= 74, max= 181, avg=146.25, stdev=49.09, samples=4 write: IOPS=159, BW=2550KiB/s (2611kB/s)(6368KiB/2497msec) clat (usec): min=666, max=33650, avg=3591.10, stdev=4592.28 lat (usec): min=667, max=33651, avg=3591.88, stdev=4592.30 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 725], 10.00th=[ 766], 20.00th=[ 848], | 30.00th=[ 930], 40.00th=[ 1123], 50.00th=[ 1565], 60.00th=[ 2999], | 70.00th=[ 4080], 80.00th=[ 5538], 90.00th=[ 8094], 95.00th=[10159], | 99.00th=[30016], 99.50th=[32637], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 1440, max= 3328, per=25.49%, avg=2534.75, stdev=825.02, samples=4 iops : min= 90, max= 208, avg=158.25, stdev=51.60, samples=4 lat (usec) : 100=14.58%, 250=2.21%, 500=0.13%, 750=4.04%, 1000=13.93% lat (msec) : 2=9.51%, 4=27.86%, 10=24.61%, 20=1.95%, 50=1.17% cpu : usr=0.20%, sys=26.48%, ctx=1343, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10709: Fri Apr 25 21:05:25 2025 read: IOPS=141, BW=2267KiB/s (2321kB/s)(5744KiB/2534msec) clat (usec): min=58, max=42374, avg=2889.95, stdev=3431.08 lat (usec): min=58, max=42374, avg=2890.46, stdev=3431.08 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 69], 10.00th=[ 71], 20.00th=[ 83], | 30.00th=[ 105], 40.00th=[ 2245], 50.00th=[ 2638], 60.00th=[ 3097], | 70.00th=[ 3884], 80.00th=[ 4752], 90.00th=[ 5866], 95.00th=[ 7570], | 99.00th=[10159], 99.50th=[27657], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 672, max= 3072, per=23.81%, avg=2250.60, stdev=932.65, samples=5 iops : min= 42, max= 192, avg=140.40, stdev=58.24, samples=5 write: IOPS=161, BW=2582KiB/s (2644kB/s)(6544KiB/2534msec) clat (usec): min=668, max=30493, avg=3405.83, stdev=4396.00 lat (usec): min=669, max=30494, avg=3407.77, stdev=4396.01 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 709], 10.00th=[ 742], 20.00th=[ 799], | 30.00th=[ 873], 40.00th=[ 996], 50.00th=[ 1549], 60.00th=[ 2933], | 70.00th=[ 3818], 80.00th=[ 5080], 90.00th=[ 7504], 95.00th=[ 9241], | 99.00th=[24511], 99.50th=[28181], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 896, max= 3168, per=25.72%, avg=2557.40, stdev=939.90, samples=5 iops : min= 56, max= 198, avg=159.60, stdev=58.64, samples=5 lat (usec) : 100=13.28%, 250=1.56%, 500=0.91%, 750=5.86%, 1000=15.49% lat (msec) : 2=7.55%, 4=26.69%, 10=25.52%, 20=1.69%, 50=1.43% cpu : usr=0.16%, sys=26.21%, ctx=1289, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10710: Fri Apr 25 21:05:25 2025 read: IOPS=168, BW=2701KiB/s (2766kB/s)(6256KiB/2316msec) clat (usec): min=56, max=29646, avg=3006.08, stdev=2999.97 lat (usec): min=57, max=29647, avg=3006.85, stdev=2999.77 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 70], 10.00th=[ 72], 20.00th=[ 85], | 30.00th=[ 2040], 40.00th=[ 2507], 50.00th=[ 2868], 60.00th=[ 3261], | 70.00th=[ 3654], 80.00th=[ 4817], 90.00th=[ 6128], 95.00th=[ 7242], | 99.00th=[10159], 99.50th=[29492], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 2080, max= 2976, per=28.10%, avg=2656.00, stdev=397.97, samples=4 iops : min= 130, max= 186, avg=166.00, stdev=24.87, samples=4 write: IOPS=162, BW=2604KiB/s (2667kB/s)(6032KiB/2316msec) clat (usec): min=631, max=25767, avg=2960.41, stdev=3100.35 lat (usec): min=632, max=25768, avg=2961.23, stdev=3100.37 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 865], | 30.00th=[ 947], 40.00th=[ 1156], 50.00th=[ 1729], 60.00th=[ 2671], | 70.00th=[ 3425], 80.00th=[ 4621], 90.00th=[ 6390], 95.00th=[ 8848], | 99.00th=[16909], 99.50th=[20841], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 2368, max= 2880, per=26.47%, avg=2632.00, stdev=219.19, samples=4 iops : min= 148, max= 180, avg=164.50, stdev=13.70, samples=4 lat (usec) : 100=12.89%, 250=1.82%, 500=0.13%, 750=2.99%, 1000=13.41% lat (msec) : 2=9.77%, 4=33.98%, 10=23.18%, 20=1.30%, 50=0.52% cpu : usr=0.30%, sys=30.11%, ctx=1397, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10711: Fri Apr 25 21:05:25 2025 read: IOPS=154, BW=2467KiB/s (2526kB/s)(6032KiB/2445msec) clat (usec): min=55, max=15124, avg=2676.25, stdev=2424.56 lat (usec): min=55, max=15124, avg=2676.80, stdev=2424.60 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 67], 10.00th=[ 71], 20.00th=[ 80], | 30.00th=[ 97], 40.00th=[ 2278], 50.00th=[ 2704], 60.00th=[ 3032], | 70.00th=[ 3752], 80.00th=[ 4490], 90.00th=[ 5866], 95.00th=[ 6980], | 99.00th=[ 9110], 99.50th=[12518], 99.90th=[15139], 99.95th=[15139], | 99.99th=[15139] bw ( KiB/s): min= 1344, max= 2816, per=24.68%, avg=2333.00, stdev=668.19, samples=4 iops : min= 84, max= 176, avg=145.50, stdev=41.61, samples=4 write: IOPS=159, BW=2559KiB/s (2620kB/s)(6256KiB/2445msec) clat (usec): min=633, max=50966, avg=3378.69, stdev=4770.73 lat (usec): min=634, max=50967, avg=3379.52, stdev=4770.73 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 701], 10.00th=[ 758], 20.00th=[ 824], | 30.00th=[ 898], 40.00th=[ 1037], 50.00th=[ 1631], 60.00th=[ 2704], | 70.00th=[ 3654], 80.00th=[ 4948], 90.00th=[ 7635], 95.00th=[10028], | 99.00th=[31327], 99.50th=[35914], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 1632, max= 3161, per=25.47%, avg=2533.00, stdev=681.70, samples=4 iops : min= 102, max= 197, avg=158.00, stdev=42.48, samples=4 lat (usec) : 100=14.97%, 250=1.69%, 500=0.39%, 750=4.95%, 1000=14.19% lat (msec) : 2=8.33%, 4=29.43%, 10=23.31%, 20=2.08%, 50=0.52% lat (msec) : 100=0.13% cpu : usr=0.29%, sys=26.72%, ctx=1322, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9452KiB/s (9679kB/s), 2267KiB/s-2701KiB/s (2321kB/s-2766kB/s), io=23.4MiB (24.5MB), run=2316-2534msec WRITE: bw=9945KiB/s (10.2MB/s), 2550KiB/s-2604KiB/s (2611kB/s-2667kB/s), io=24.6MiB (25.8MB), run=2316-2534msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10704: Fri Apr 25 21:05:31 2025 read: IOPS=41, BW=660KiB/s (676kB/s)(5920KiB/8973msec) clat (usec): min=1815, max=20397, avg=4631.29, stdev=2881.08 lat (usec): min=1815, max=20398, avg=4632.32, stdev=2881.02 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2212], 10.00th=[ 2769], 20.00th=[ 3294], | 30.00th=[ 3392], 40.00th=[ 3523], 50.00th=[ 3654], 60.00th=[ 3818], | 70.00th=[ 4293], 80.00th=[ 5276], 90.00th=[ 6980], 95.00th=[11076], | 99.00th=[17433], 99.50th=[20317], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 416, max= 1088, per=24.56%, avg=647.41, stdev=175.06, samples=17 iops : min= 26, max= 68, avg=40.41, stdev=10.93, samples=17 write: IOPS=44, BW=710KiB/s (727kB/s)(6368KiB/8973msec) clat (usec): min=9182, max=39710, avg=18195.21, stdev=5542.87 lat (usec): min=9183, max=39711, avg=18196.98, stdev=5542.71 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11469], 10.00th=[12256], 20.00th=[13304], | 30.00th=[14484], 40.00th=[15926], 50.00th=[17433], 60.00th=[18744], | 70.00th=[20055], 80.00th=[21627], 90.00th=[26084], 95.00th=[28705], | 99.00th=[36439], 99.50th=[37487], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 416, max= 928, per=25.17%, avg=698.24, stdev=143.52, samples=17 iops : min= 26, max= 58, avg=43.59, stdev= 8.91, samples=17 lat (msec) : 2=1.30%, 4=29.95%, 10=14.19%, 20=38.93%, 50=15.62% cpu : usr=0.13%, sys=8.94%, ctx=973, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10705: Fri Apr 25 21:05:32 2025 read: IOPS=39, BW=632KiB/s (647kB/s)(5744KiB/9084msec) clat (usec): min=1788, max=25762, avg=4335.52, stdev=2494.43 lat (usec): min=1788, max=25762, avg=4336.56, stdev=2494.36 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2180], 10.00th=[ 2507], 20.00th=[ 3261], | 30.00th=[ 3359], 40.00th=[ 3458], 50.00th=[ 3589], 60.00th=[ 3720], | 70.00th=[ 4146], 80.00th=[ 5014], 90.00th=[ 6783], 95.00th=[ 9110], | 99.00th=[15795], 99.50th=[20579], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 288, max= 1024, per=23.94%, avg=631.11, stdev=206.16, samples=18 iops : min= 18, max= 64, avg=39.44, stdev=12.89, samples=18 write: IOPS=45, BW=720KiB/s (738kB/s)(6544KiB/9084msec) clat (usec): min=9909, max=57774, avg=18331.93, stdev=6122.39 lat (usec): min=9910, max=57775, avg=18333.55, stdev=6122.31 clat percentiles (usec): | 1.00th=[10552], 5.00th=[11469], 10.00th=[12387], 20.00th=[13566], | 30.00th=[14746], 40.00th=[16057], 50.00th=[16909], 60.00th=[18482], | 70.00th=[19792], 80.00th=[21365], 90.00th=[25035], 95.00th=[31065], | 99.00th=[39060], 99.50th=[42730], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 480, max= 928, per=25.96%, avg=720.00, stdev=144.89, samples=18 iops : min= 30, max= 58, avg=45.00, stdev= 9.06, samples=18 lat (msec) : 2=0.91%, 4=30.21%, 10=14.06%, 20=39.32%, 50=15.36% lat (msec) : 100=0.13% cpu : usr=0.06%, sys=8.94%, ctx=935, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10706: Fri Apr 25 21:05:32 2025 read: IOPS=45, BW=729KiB/s (747kB/s)(6256KiB/8579msec) clat (usec): min=1923, max=16644, avg=4322.12, stdev=2069.00 lat (usec): min=1924, max=16644, avg=4323.19, stdev=2068.91 clat percentiles (usec): | 1.00th=[ 2089], 5.00th=[ 2442], 10.00th=[ 2966], 20.00th=[ 3228], | 30.00th=[ 3359], 40.00th=[ 3458], 50.00th=[ 3556], 60.00th=[ 3720], | 70.00th=[ 4146], 80.00th=[ 5276], 90.00th=[ 6652], 95.00th=[ 8094], | 99.00th=[14222], 99.50th=[16057], 99.90th=[16581], 99.95th=[16581], | 99.99th=[16581] bw ( KiB/s): min= 352, max= 1184, per=27.85%, avg=734.12, stdev=274.22, samples=17 iops : min= 22, max= 74, avg=45.88, stdev=17.14, samples=17 write: IOPS=43, BW=703KiB/s (720kB/s)(6032KiB/8579msec) clat (usec): min=9272, max=43785, avg=18080.81, stdev=5771.54 lat (usec): min=9273, max=43788, avg=18082.55, stdev=5771.36 clat percentiles (usec): | 1.00th=[10814], 5.00th=[11469], 10.00th=[11994], 20.00th=[13173], | 30.00th=[14222], 40.00th=[15664], 50.00th=[16909], 60.00th=[18220], | 70.00th=[19530], 80.00th=[21890], 90.00th=[26346], 95.00th=[29230], | 99.00th=[36439], 99.50th=[42730], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 448, max= 896, per=25.24%, avg=700.24, stdev=149.19, samples=17 iops : min= 28, max= 56, avg=43.76, stdev= 9.32, samples=17 lat (msec) : 2=0.13%, 4=34.51%, 10=15.23%, 20=37.11%, 50=13.02% cpu : usr=0.12%, sys=9.83%, ctx=955, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10707: Fri Apr 25 21:05:32 2025 read: IOPS=42, BW=676KiB/s (693kB/s)(6032KiB/8918msec) clat (usec): min=1772, max=20394, avg=4414.47, stdev=2486.65 lat (usec): min=1773, max=20395, avg=4415.61, stdev=2486.59 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2114], 10.00th=[ 2507], 20.00th=[ 3195], | 30.00th=[ 3359], 40.00th=[ 3458], 50.00th=[ 3589], 60.00th=[ 3851], | 70.00th=[ 4228], 80.00th=[ 5473], 90.00th=[ 6587], 95.00th=[ 9372], | 99.00th=[15008], 99.50th=[20317], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 224, max= 928, per=25.35%, avg=668.24, stdev=188.60, samples=17 iops : min= 14, max= 58, avg=41.76, stdev=11.79, samples=17 write: IOPS=43, BW=702KiB/s (718kB/s)(6256KiB/8918msec) clat (usec): min=9564, max=57887, avg=18252.27, stdev=5859.33 lat (usec): min=9565, max=57888, avg=18253.96, stdev=5859.19 clat percentiles (usec): | 1.00th=[10814], 5.00th=[12125], 10.00th=[12649], 20.00th=[13435], | 30.00th=[14746], 40.00th=[15664], 50.00th=[16909], 60.00th=[18220], | 70.00th=[20055], 80.00th=[21627], 90.00th=[25822], 95.00th=[29754], | 99.00th=[38536], 99.50th=[42206], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 352, max= 864, per=25.11%, avg=696.47, stdev=155.76, samples=17 iops : min= 22, max= 54, avg=43.53, stdev= 9.73, samples=17 lat (msec) : 2=0.91%, 4=31.51%, 10=14.58%, 20=37.63%, 50=15.23% lat (msec) : 100=0.13% cpu : usr=0.17%, sys=9.15%, ctx=958, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2637KiB/s (2700kB/s), 632KiB/s-729KiB/s (647kB/s-747kB/s), io=23.4MiB (24.5MB), run=8579-9084msec WRITE: bw=2774KiB/s (2841kB/s), 702KiB/s-720KiB/s (718kB/s-738kB/s), io=24.6MiB (25.8MB), run=8579-9084msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10724: Fri Apr 25 21:05:33 2025 read: IOPS=18, BW=18.5MiB/s (19.4MB/s)(7168KiB/378msec) clat (usec): min=367, max=54878, avg=26393.99, stdev=19686.21 lat (usec): min=368, max=54879, avg=26394.72, stdev=19686.21 clat percentiles (usec): | 1.00th=[ 367], 5.00th=[ 367], 10.00th=[ 367], 20.00th=[ 611], | 30.00th=[27919], 40.00th=[27919], 50.00th=[31065], 60.00th=[33817], | 70.00th=[33817], 80.00th=[35914], 90.00th=[54789], 95.00th=[54789], | 99.00th=[54789], 99.50th=[54789], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] write: IOPS=13, BW=13.2MiB/s (13.9MB/s)(5120KiB/378msec) clat (usec): min=15065, max=56535, avg=29788.99, stdev=16399.62 lat (usec): min=15078, max=56570, avg=29823.84, stdev=16402.31 clat percentiles (usec): | 1.00th=[15008], 5.00th=[15008], 10.00th=[15008], 20.00th=[15008], | 30.00th=[19268], 40.00th=[19268], 50.00th=[25035], 60.00th=[25035], | 70.00th=[33162], 80.00th=[33162], 90.00th=[56361], 95.00th=[56361], | 99.00th=[56361], 99.50th=[56361], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 20=16.67%, 50=50.00%, 100=16.67% cpu : usr=0.00%, sys=38.20%, ctx=714, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10725: Fri Apr 25 21:05:33 2025 read: IOPS=3, BW=3363KiB/s (3444kB/s)(2048KiB/609msec) clat (usec): min=34097, max=40099, avg=37098.70, stdev=4243.97 lat (usec): min=34098, max=40100, avg=37099.63, stdev=4243.60 clat percentiles (usec): | 1.00th=[34341], 5.00th=[34341], 10.00th=[34341], 20.00th=[34341], | 30.00th=[34341], 40.00th=[34341], 50.00th=[34341], 60.00th=[40109], | 70.00th=[40109], 80.00th=[40109], 90.00th=[40109], 95.00th=[40109], | 99.00th=[40109], 99.50th=[40109], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] write: IOPS=16, BW=16.4MiB/s (17.2MB/s)(10.0MiB/609msec) clat (usec): min=20745, max=72536, avg=45099.60, stdev=17965.15 lat (usec): min=20786, max=72552, avg=45169.84, stdev=17954.87 clat percentiles (usec): | 1.00th=[20841], 5.00th=[20841], 10.00th=[20841], 20.00th=[25822], | 30.00th=[33162], 40.00th=[39584], 50.00th=[39584], 60.00th=[41681], | 70.00th=[62129], 80.00th=[62129], 90.00th=[70779], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] lat (msec) : 50=75.00%, 100=25.00% cpu : usr=0.00%, sys=36.68%, ctx=522, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10726: Fri Apr 25 21:05:33 2025 read: IOPS=7, BW=7670KiB/s (7855kB/s)(6144KiB/801msec) clat (usec): min=531, max=57634, avg=29845.42, stdev=18714.50 lat (usec): min=532, max=57635, avg=29846.16, stdev=18714.49 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 529], 10.00th=[ 529], 20.00th=[26346], | 30.00th=[26346], 40.00th=[26346], 50.00th=[26346], 60.00th=[28705], | 70.00th=[39584], 80.00th=[39584], 90.00th=[57410], 95.00th=[57410], | 99.00th=[57410], 99.50th=[57410], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 2048, max= 2048, per=8.01%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7670KiB/s (7855kB/s)(6144KiB/801msec) clat (msec): min=24, max=273, avg=96.69, stdev=90.18 lat (msec): min=24, max=273, avg=96.73, stdev=90.19 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 69], 50.00th=[ 69], 60.00th=[ 83], | 70.00th=[ 91], 80.00th=[ 91], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 6144, max= 6144, per=17.16%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=50.00%, 100=33.33%, 500=8.33% cpu : usr=0.12%, sys=23.00%, ctx=1105, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10727: Fri Apr 25 21:05:33 2025 read: IOPS=11, BW=11.8MiB/s (12.4MB/s)(5120KiB/422msec) clat (usec): min=338, max=77880, avg=30210.22, stdev=29333.20 lat (usec): min=338, max=77881, avg=30210.80, stdev=29333.23 clat percentiles (usec): | 1.00th=[ 338], 5.00th=[ 338], 10.00th=[ 338], 20.00th=[ 338], | 30.00th=[15139], 40.00th=[15139], 50.00th=[23987], 60.00th=[23987], | 70.00th=[33817], 80.00th=[33817], 90.00th=[78119], 95.00th=[78119], | 99.00th=[78119], 99.50th=[78119], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] write: IOPS=16, BW=16.6MiB/s (17.4MB/s)(7168KiB/422msec) clat (usec): min=15163, max=73945, avg=36104.76, stdev=18330.82 lat (usec): min=15194, max=73980, avg=36148.34, stdev=18326.95 clat percentiles (usec): | 1.00th=[15139], 5.00th=[15139], 10.00th=[15139], 20.00th=[27657], | 30.00th=[28443], 40.00th=[28443], 50.00th=[32900], 60.00th=[36963], | 70.00th=[36963], 80.00th=[38011], 90.00th=[73925], 95.00th=[73925], | 99.00th=[73925], 99.50th=[73925], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] lat (usec) : 500=8.33% lat (msec) : 20=16.67%, 50=58.33%, 100=16.67% cpu : usr=0.00%, sys=39.90%, ctx=497, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=24.0MiB/s (26.2MB/s), 3363KiB/s-18.5MiB/s (3444kB/s-19.4MB/s), io=20.0MiB (20.0MB), run=378-801msec WRITE: bw=34.0MiB/s (36.7MB/s), 7670KiB/s-16.6MiB/s (7855kB/s-17.4MB/s), io=28.0MiB (29.4MB), run=378-801msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10728: Fri Apr 25 21:05:33 2025 read: IOPS=10, BW=10.0MiB/s (10.5MB/s)(7168KiB/698msec) clat (msec): min=19, max=117, avg=53.89, stdev=32.49 lat (msec): min=19, max=117, avg=53.89, stdev=32.49 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 33], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 47], 60.00th=[ 50], | 70.00th=[ 50], 80.00th=[ 73], 90.00th=[ 118], 95.00th=[ 118], | 99.00th=[ 118], 99.50th=[ 118], 99.90th=[ 118], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min=10219, max=10219, per=39.82%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7335KiB/s (7511kB/s)(5120KiB/698msec) clat (usec): min=42343, max=64143, avg=52032.46, stdev=9193.13 lat (usec): min=42373, max=64208, avg=52079.71, stdev=9201.66 clat percentiles (usec): | 1.00th=[42206], 5.00th=[42206], 10.00th=[42206], 20.00th=[42206], | 30.00th=[44303], 40.00th=[44303], 50.00th=[51643], 60.00th=[51643], | 70.00th=[57934], 80.00th=[57934], 90.00th=[64226], 95.00th=[64226], | 99.00th=[64226], 99.50th=[64226], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 4087, max= 4087, per=11.38%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=11.62%, ctx=49, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10729: Fri Apr 25 21:05:33 2025 read: IOPS=2, BW=2566KiB/s (2628kB/s)(2048KiB/798msec) clat (msec): min=14, max=141, avg=78.09, stdev=89.56 lat (msec): min=14, max=141, avg=78.09, stdev=89.56 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15], | 30.00th=[ 15], 40.00th=[ 15], 50.00th=[ 15], 60.00th=[ 142], | 70.00th=[ 142], 80.00th=[ 142], 90.00th=[ 142], 95.00th=[ 142], | 99.00th=[ 142], 99.50th=[ 142], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 2048, max= 2048, per=7.98%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.5MiB/s (13.1MB/s)(10.0MiB/798msec) clat (usec): min=43254, max=84779, avg=58020.31, stdev=13801.94 lat (usec): min=43328, max=84846, avg=58076.56, stdev=13798.65 clat percentiles (usec): | 1.00th=[43254], 5.00th=[43254], 10.00th=[43254], 20.00th=[44303], | 30.00th=[47449], 40.00th=[49021], 50.00th=[52167], 60.00th=[57410], | 70.00th=[68682], 80.00th=[68682], 90.00th=[73925], 95.00th=[84411], | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min=10240, max=10240, per=28.50%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=50.00%, 250=8.33% cpu : usr=0.00%, sys=10.54%, ctx=51, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10730: Fri Apr 25 21:05:33 2025 read: IOPS=8, BW=8463KiB/s (8666kB/s)(6144KiB/726msec) clat (usec): min=25142, max=62206, avg=44010.99, stdev=14838.74 lat (usec): min=25142, max=62207, avg=44011.66, stdev=14838.77 clat percentiles (usec): | 1.00th=[25035], 5.00th=[25035], 10.00th=[25035], 20.00th=[27395], | 30.00th=[27395], 40.00th=[45876], 50.00th=[45876], 60.00th=[48497], | 70.00th=[54789], 80.00th=[54789], 90.00th=[62129], 95.00th=[62129], | 99.00th=[62129], 99.50th=[62129], 99.90th=[62129], 99.95th=[62129], | 99.99th=[62129] bw ( KiB/s): min= 8192, max= 8192, per=31.92%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8463KiB/s (8666kB/s)(6144KiB/726msec) clat (msec): min=40, max=123, avg=74.24, stdev=28.45 lat (msec): min=40, max=123, avg=74.29, stdev=28.46 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 60], | 30.00th=[ 60], 40.00th=[ 63], 50.00th=[ 63], 60.00th=[ 72], | 70.00th=[ 88], 80.00th=[ 88], 90.00th=[ 124], 95.00th=[ 124], | 99.00th=[ 124], 99.50th=[ 124], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 8192, max= 8192, per=22.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=50.00%, 250=8.33% cpu : usr=0.00%, sys=9.10%, ctx=39, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10731: Fri Apr 25 21:05:33 2025 read: IOPS=6, BW=6506KiB/s (6662kB/s)(5120KiB/787msec) clat (usec): min=14424, max=53256, avg=30953.83, stdev=18496.59 lat (usec): min=14425, max=53257, avg=30954.70, stdev=18496.33 clat percentiles (usec): | 1.00th=[14484], 5.00th=[14484], 10.00th=[14484], 20.00th=[14484], | 30.00th=[16319], 40.00th=[16319], 50.00th=[22152], 60.00th=[22152], | 70.00th=[48497], 80.00th=[48497], 90.00th=[53216], 95.00th=[53216], | 99.00th=[53216], 99.50th=[53216], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] write: IOPS=8, BW=9108KiB/s (9327kB/s)(7168KiB/787msec) clat (msec): min=28, max=234, avg=78.97, stdev=71.42 lat (msec): min=28, max=234, avg=79.02, stdev=71.41 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 41], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 48], 60.00th=[ 70], | 70.00th=[ 70], 80.00th=[ 90], 90.00th=[ 234], 95.00th=[ 234], | 99.00th=[ 234], 99.50th=[ 234], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 8175, max= 8175, per=22.75%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=50.00%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=11.83%, ctx=55, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.1MiB/s (26.3MB/s), 2566KiB/s-10.0MiB/s (2628kB/s-10.5MB/s), io=20.0MiB (20.0MB), run=698-798msec WRITE: bw=35.1MiB/s (36.8MB/s), 7335KiB/s-12.5MiB/s (7511kB/s-13.1MB/s), io=28.0MiB (29.4MB), run=698-798msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10737: Fri Apr 25 21:05:36 2025 read: IOPS=2, BW=9533KiB/s (9762kB/s)(12.0MiB/1289msec) clat (msec): min=130, max=965, avg=425.55, stdev=468.63 lat (msec): min=130, max=965, avg=425.55, stdev=468.63 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 131], 10.00th=[ 131], 20.00th=[ 131], | 30.00th=[ 131], 40.00th=[ 182], 50.00th=[ 182], 60.00th=[ 182], | 70.00th=[ 969], 80.00th=[ 969], 90.00th=[ 969], 95.00th=[ 969], | 99.00th=[ 969], 99.50th=[ 969], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 8192, max= 8192, per=64.45%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.00%, sys=5.20%, ctx=37, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10739: Fri Apr 25 21:05:36 2025 read: IOPS=0, BW=3279KiB/s (3358kB/s)(4096KiB/1249msec) clat (nsec): min=33223k, max=33223k, avg=33222856.00, stdev= 0.00 lat (nsec): min=33225k, max=33225k, avg=33224761.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[33162], 5.00th=[33162], 10.00th=[33162], 20.00th=[33162], | 30.00th=[33162], 40.00th=[33162], 50.00th=[33162], 60.00th=[33162], | 70.00th=[33162], 80.00th=[33162], 90.00th=[33162], 95.00th=[33162], | 99.00th=[33162], 99.50th=[33162], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 8192, max= 8192, per=64.45%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6559KiB/s (6716kB/s)(8192KiB/1249msec) clat (msec): min=198, max=993, avg=595.87, stdev=562.62 lat (msec): min=198, max=993, avg=596.13, stdev=562.55 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 199], 10.00th=[ 199], 20.00th=[ 199], | 30.00th=[ 199], 40.00th=[ 199], 50.00th=[ 199], 60.00th=[ 995], | 70.00th=[ 995], 80.00th=[ 995], 90.00th=[ 995], 95.00th=[ 995], | 99.00th=[ 995], 99.50th=[ 995], 99.90th=[ 995], 99.95th=[ 995], | 99.99th=[ 995] lat (msec) : 50=33.33%, 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=7.85%, ctx=50, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10741: Fri Apr 25 21:05:36 2025 write: IOPS=2, BW=9009KiB/s (9225kB/s)(12.0MiB/1364msec) clat (msec): min=76, max=1041, avg=442.13, stdev=522.92 lat (msec): min=77, max=1041, avg=442.42, stdev=522.92 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 78], 10.00th=[ 78], 20.00th=[ 78], | 30.00th=[ 78], 40.00th=[ 209], 50.00th=[ 209], 60.00th=[ 209], | 70.00th=[ 1045], 80.00th=[ 1045], 90.00th=[ 1045], 95.00th=[ 1045], | 99.00th=[ 1045], 99.50th=[ 1045], 99.90th=[ 1045], 99.95th=[ 1045], | 99.99th=[ 1045] lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.07%, sys=6.24%, ctx=20, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10743: Fri Apr 25 21:05:36 2025 write: IOPS=2, BW=9445KiB/s (9672kB/s)(12.0MiB/1301msec) clat (msec): min=111, max=910, avg=428.09, stdev=424.30 lat (msec): min=112, max=910, avg=428.48, stdev=424.20 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 264], 50.00th=[ 264], 60.00th=[ 264], | 70.00th=[ 911], 80.00th=[ 911], 90.00th=[ 911], 95.00th=[ 911], | 99.00th=[ 911], 99.50th=[ 911], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 8192, max= 8192, per=34.10%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.15%, sys=7.85%, ctx=56, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.4MiB/s (13.0MB/s), 3279KiB/s-9533KiB/s (3358kB/s-9762kB/s), io=16.0MiB (16.8MB), run=1249-1289msec WRITE: bw=23.5MiB/s (24.6MB/s), 6559KiB/s-9445KiB/s (6716kB/s-9672kB/s), io=32.0MiB (33.6MB), run=1249-1364msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10738: Fri Apr 25 21:05:36 2025 read: IOPS=2, BW=10.1MiB/s (10.6MB/s)(12.0MiB/1189msec) clat (msec): min=45, max=944, avg=392.19, stdev=483.81 lat (msec): min=45, max=944, avg=392.19, stdev=483.81 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 46], | 30.00th=[ 46], 40.00th=[ 186], 50.00th=[ 186], 60.00th=[ 186], | 70.00th=[ 944], 80.00th=[ 944], 90.00th=[ 944], 95.00th=[ 944], | 99.00th=[ 944], 99.50th=[ 944], 99.90th=[ 944], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 8192, max= 8192, per=75.75%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=23.23%, ctx=1558, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10740: Fri Apr 25 21:05:36 2025 read: IOPS=0, BW=2704KiB/s (2769kB/s)(4096KiB/1515msec) clat (nsec): min=912844k, max=912844k, avg=912844035.00, stdev= 0.00 lat (nsec): min=912845k, max=912845k, avg=912845045.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 911], 5.00th=[ 911], 10.00th=[ 911], 20.00th=[ 911], | 30.00th=[ 911], 40.00th=[ 911], 50.00th=[ 911], 60.00th=[ 911], | 70.00th=[ 911], 80.00th=[ 911], 90.00th=[ 911], 95.00th=[ 911], | 99.00th=[ 911], 99.50th=[ 911], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 8192, max= 8192, per=75.75%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5407KiB/s (5537kB/s)(8192KiB/1515msec) clat (msec): min=266, max=297, avg=281.94, stdev=22.16 lat (msec): min=266, max=297, avg=282.25, stdev=22.21 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 266], 10.00th=[ 266], 20.00th=[ 266], | 30.00th=[ 266], 40.00th=[ 266], 50.00th=[ 266], 60.00th=[ 296], | 70.00th=[ 296], 80.00th=[ 296], 90.00th=[ 296], 95.00th=[ 296], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] lat (msec) : 500=66.67%, 1000=33.33% cpu : usr=0.00%, sys=11.10%, ctx=816, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10742: Fri Apr 25 21:05:36 2025 write: IOPS=1, BW=7979KiB/s (8171kB/s)(12.0MiB/1540msec) clat (msec): min=96, max=1125, avg=498.21, stdev=550.01 lat (msec): min=97, max=1125, avg=498.51, stdev=550.00 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 97], 10.00th=[ 97], 20.00th=[ 97], | 30.00th=[ 97], 40.00th=[ 275], 50.00th=[ 275], 60.00th=[ 275], | 70.00th=[ 1133], 80.00th=[ 1133], 90.00th=[ 1133], 95.00th=[ 1133], | 99.00th=[ 1133], 99.50th=[ 1133], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=12.67%, ctx=66, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10744: Fri Apr 25 21:05:36 2025 write: IOPS=2, BW=8411KiB/s (8613kB/s)(12.0MiB/1461msec) clat (msec): min=86, max=1085, avg=467.65, stdev=539.82 lat (msec): min=86, max=1085, avg=467.93, stdev=539.82 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 87], 10.00th=[ 87], 20.00th=[ 87], | 30.00th=[ 87], 40.00th=[ 232], 50.00th=[ 232], 60.00th=[ 232], | 70.00th=[ 1083], 80.00th=[ 1083], 90.00th=[ 1083], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=15.14%, ctx=95, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.6MiB/s (11.1MB/s), 2704KiB/s-10.1MiB/s (2769kB/s-10.6MB/s), io=16.0MiB (16.8MB), run=1189-1515msec WRITE: bw=20.8MiB/s (21.8MB/s), 5407KiB/s-8411KiB/s (5537kB/s-8613kB/s), io=32.0MiB (33.6MB), run=1461-1540msec