== sanity test 398b: DIO and buffer IO race ============== 18:32:06 (1746138726) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.06393 s, 24.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5865: Thu May 1 18:32:47 2025 read: IOPS=42, BW=171KiB/s (175kB/s)(5992KiB/35073msec) clat (usec): min=51, max=314891, avg=10560.30, stdev=12498.90 lat (usec): min=52, max=314892, avg=10561.75, stdev=12499.09 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 99], 10.00th=[ 113], 20.00th=[ 289], | 30.00th=[ 5735], 40.00th=[ 7308], 50.00th=[ 8848], 60.00th=[ 10290], | 70.00th=[ 12780], 80.00th=[ 16909], 90.00th=[ 22414], 95.00th=[ 26346], | 99.00th=[ 39584], 99.50th=[ 50594], 99.90th=[145753], 99.95th=[316670], | 99.99th=[316670] bw ( KiB/s): min= 79, max= 312, per=25.66%, avg=170.37, stdev=54.28, samples=70 iops : min= 19, max= 78, avg=42.34, stdev=13.58, samples=70 write: IOPS=44, BW=180KiB/s (184kB/s)(6296KiB/35073msec) clat (usec): min=527, max=187539, avg=12183.84, stdev=13086.60 lat (usec): min=527, max=187539, avg=12185.45, stdev=13086.73 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 734], 10.00th=[ 791], 20.00th=[ 1188], | 30.00th=[ 2311], 40.00th=[ 5145], 50.00th=[ 8979], 60.00th=[ 12387], | 70.00th=[ 16188], 80.00th=[ 21365], 90.00th=[ 28705], 95.00th=[ 35914], | 99.00th=[ 48497], 99.50th=[ 60556], 99.90th=[ 90702], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 48, max= 328, per=25.71%, avg=178.71, stdev=60.26, samples=70 iops : min= 12, max= 82, avg=44.43, stdev=15.11, samples=70 lat (usec) : 100=2.51%, 250=7.10%, 500=0.55%, 750=3.39%, 1000=5.47% lat (msec) : 2=5.92%, 4=5.37%, 10=25.39%, 20=26.66%, 50=16.93% lat (msec) : 100=0.62%, 250=0.07%, 500=0.03% cpu : usr=0.12%, sys=13.14%, ctx=5125, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5866: Thu May 1 18:32:47 2025 read: IOPS=41, BW=164KiB/s (168kB/s)(5940KiB/36146msec) clat (usec): min=60, max=203240, avg=10362.25, stdev=10891.56 lat (usec): min=61, max=203242, avg=10363.64, stdev=10891.47 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 95], 10.00th=[ 108], 20.00th=[ 135], | 30.00th=[ 5604], 40.00th=[ 7242], 50.00th=[ 8717], 60.00th=[ 10552], | 70.00th=[ 13042], 80.00th=[ 16450], 90.00th=[ 21627], 95.00th=[ 27395], | 99.00th=[ 41157], 99.50th=[ 64226], 99.90th=[ 92799], 99.95th=[202376], | 99.99th=[202376] bw ( KiB/s): min= 7, max= 312, per=24.57%, avg=163.12, stdev=60.42, samples=72 iops : min= 1, max= 78, avg=40.49, stdev=15.16, samples=72 write: IOPS=43, BW=176KiB/s (180kB/s)(6348KiB/36146msec) clat (usec): min=524, max=159233, avg=12969.10, stdev=14358.67 lat (usec): min=525, max=159234, avg=12972.95, stdev=14358.74 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 734], 10.00th=[ 791], 20.00th=[ 1090], | 30.00th=[ 2212], 40.00th=[ 5538], 50.00th=[ 9634], 60.00th=[ 13042], | 70.00th=[ 16909], 80.00th=[ 21627], 90.00th=[ 30278], 95.00th=[ 38011], | 99.00th=[ 64750], 99.50th=[ 82314], 99.90th=[125305], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 16, max= 393, per=24.99%, avg=173.67, stdev=61.75, samples=72 iops : min= 4, max= 98, avg=43.12, stdev=15.52, samples=72 lat (usec) : 100=3.39%, 250=7.42%, 500=0.42%, 750=3.58%, 1000=6.64% lat (msec) : 2=5.21%, 4=4.85%, 10=22.79%, 20=27.64%, 50=16.60% lat (msec) : 100=1.33%, 250=0.13% cpu : usr=0.19%, sys=12.47%, ctx=4955, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5867: Thu May 1 18:32:47 2025 read: IOPS=42, BW=171KiB/s (175kB/s)(6028KiB/35332msec) clat (usec): min=56, max=95311, avg=9957.65, stdev=9010.77 lat (usec): min=57, max=95312, avg=9958.89, stdev=9010.72 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 95], 10.00th=[ 109], 20.00th=[ 196], | 30.00th=[ 5800], 40.00th=[ 7308], 50.00th=[ 8717], 60.00th=[10028], | 70.00th=[12256], 80.00th=[15139], 90.00th=[21103], 95.00th=[25560], | 99.00th=[38536], 99.50th=[44303], 99.90th=[80217], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 64, max= 312, per=25.90%, avg=172.00, stdev=57.60, samples=69 iops : min= 16, max= 78, avg=42.81, stdev=14.43, samples=69 write: IOPS=44, BW=177KiB/s (181kB/s)(6260KiB/35332msec) clat (usec): min=496, max=319497, avg=12880.65, stdev=16371.76 lat (usec): min=497, max=319500, avg=12883.14, stdev=16371.80 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 734], 10.00th=[ 791], 20.00th=[ 1029], | 30.00th=[ 2245], 40.00th=[ 5276], 50.00th=[ 9110], 60.00th=[ 12518], | 70.00th=[ 16319], 80.00th=[ 21365], 90.00th=[ 29754], 95.00th=[ 38011], | 99.00th=[ 61080], 99.50th=[ 82314], 99.90th=[162530], 99.95th=[320865], | 99.99th=[320865] bw ( KiB/s): min= 8, max= 336, per=25.24%, avg=175.40, stdev=53.91, samples=70 iops : min= 2, max= 84, avg=43.66, stdev=13.56, samples=70 lat (usec) : 100=3.61%, 250=6.67%, 500=0.29%, 750=3.81%, 1000=6.45% lat (msec) : 2=5.11%, 4=4.43%, 10=25.72%, 20=26.56%, 50=16.02% lat (msec) : 100=1.17%, 250=0.13%, 500=0.03% cpu : usr=0.14%, sys=12.72%, ctx=5108, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5868: Thu May 1 18:32:47 2025 read: IOPS=43, BW=173KiB/s (177kB/s)(6052KiB/35070msec) clat (usec): min=69, max=74693, avg=9783.94, stdev=8731.55 lat (usec): min=70, max=74695, avg=9786.00, stdev=8732.11 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 94], 10.00th=[ 106], 20.00th=[ 157], | 30.00th=[ 5604], 40.00th=[ 7177], 50.00th=[ 8455], 60.00th=[10159], | 70.00th=[12256], 80.00th=[15139], 90.00th=[21103], 95.00th=[25822], | 99.00th=[39060], 99.50th=[45351], 99.90th=[71828], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 80, max= 312, per=26.33%, avg=174.81, stdev=54.72, samples=69 iops : min= 20, max= 78, avg=43.42, stdev=13.76, samples=69 write: IOPS=44, BW=178KiB/s (182kB/s)(6236KiB/35070msec) clat (usec): min=452, max=304350, avg=12916.00, stdev=16280.45 lat (usec): min=452, max=304353, avg=12917.78, stdev=16280.15 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 734], 10.00th=[ 791], 20.00th=[ 1106], | 30.00th=[ 2057], 40.00th=[ 4424], 50.00th=[ 8848], 60.00th=[ 12911], | 70.00th=[ 16581], 80.00th=[ 21627], 90.00th=[ 31065], 95.00th=[ 39060], | 99.00th=[ 59507], 99.50th=[ 89654], 99.90th=[168821], 99.95th=[304088], | 99.99th=[304088] bw ( KiB/s): min= 16, max= 336, per=25.50%, avg=177.23, stdev=57.67, samples=70 iops : min= 4, max= 84, avg=44.03, stdev=14.50, samples=70 lat (usec) : 100=3.78%, 250=6.67%, 500=0.62%, 750=3.58%, 1000=5.99% lat (msec) : 2=6.02%, 4=4.88%, 10=24.15%, 20=26.95%, 50=16.34% lat (msec) : 100=0.81%, 250=0.16%, 500=0.03% cpu : usr=0.18%, sys=12.59%, ctx=4918, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=664KiB/s (680kB/s), 164KiB/s-173KiB/s (168kB/s-177kB/s), io=23.4MiB (24.6MB), run=35070-36146msec WRITE: bw=696KiB/s (712kB/s), 176KiB/s-180KiB/s (180kB/s-184kB/s), io=24.6MiB (25.7MB), run=35070-36146msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5860: Thu May 1 18:34:14 2025 read: IOPS=12, BW=48.8KiB/s (49.0kB/s)(5992KiB/122746msec) clat (msec): min=2, max=311, avg=10.60, stdev=12.36 lat (msec): min=2, max=311, avg=10.60, stdev=12.36 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 13], 90.00th=[ 21], 95.00th=[ 28], | 99.00th=[ 46], 99.50th=[ 53], 99.90th=[ 180], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 7, max= 144, per=25.28%, avg=49.05, stdev=28.38, samples=242 iops : min= 1, max= 36, avg=11.97, stdev= 7.13, samples=242 write: IOPS=12, BW=51.3KiB/s (52.5kB/s)(6296KiB/122746msec) clat (msec): min=24, max=230, avg=67.78, stdev=26.05 lat (msec): min=24, max=230, avg=67.78, stdev=26.05 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 36], 10.00th=[ 41], 20.00th=[ 47], | 30.00th=[ 53], 40.00th=[ 58], 50.00th=[ 64], 60.00th=[ 69], | 70.00th=[ 77], 80.00th=[ 87], 90.00th=[ 102], 95.00th=[ 116], | 99.00th=[ 159], 99.50th=[ 169], 99.90th=[ 215], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 8, max= 88, per=25.17%, avg=51.09, stdev=15.03, samples=244 iops : min= 2, max= 22, avg=12.48, stdev= 3.79, samples=244 lat (msec) : 4=1.63%, 10=33.20%, 20=8.69%, 50=18.46%, 100=32.55% lat (msec) : 250=5.44%, 500=0.03% cpu : usr=0.04%, sys=2.87%, ctx=3535, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5861: Thu May 1 18:34:14 2025 read: IOPS=12, BW=48.1KiB/s (49.3kB/s)(5940KiB/123425msec) clat (msec): min=2, max=302, avg=10.91, stdev=13.13 lat (msec): min=2, max=302, avg=10.91, stdev=13.13 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 13], 90.00th=[ 22], 95.00th=[ 29], | 99.00th=[ 51], 99.50th=[ 79], 99.90th=[ 155], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 7, max= 144, per=25.28%, avg=49.04, stdev=27.41, samples=240 iops : min= 1, max= 36, avg=12.04, stdev= 6.85, samples=240 write: IOPS=12, BW=51.4KiB/s (52.7kB/s)(6348KiB/123425msec) clat (msec): min=21, max=224, avg=67.45, stdev=25.16 lat (msec): min=21, max=224, avg=67.45, stdev=25.16 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 37], 10.00th=[ 41], 20.00th=[ 47], | 30.00th=[ 53], 40.00th=[ 58], 50.00th=[ 64], 60.00th=[ 70], | 70.00th=[ 75], 80.00th=[ 85], 90.00th=[ 102], 95.00th=[ 115], | 99.00th=[ 153], 99.50th=[ 169], 99.90th=[ 190], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 8, max= 88, per=25.10%, avg=50.94, stdev=14.88, samples=246 iops : min= 2, max= 22, avg=12.52, stdev= 3.77, samples=246 lat (msec) : 4=1.63%, 10=32.36%, 20=8.66%, 50=19.01%, 100=32.98% lat (msec) : 250=5.34%, 500=0.03% cpu : usr=0.04%, sys=2.84%, ctx=3470, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5862: Thu May 1 18:34:14 2025 read: IOPS=12, BW=49.1KiB/s (50.3kB/s)(6028KiB/122646msec) clat (msec): min=2, max=148, avg=10.85, stdev=10.26 lat (msec): min=2, max=148, avg=10.85, stdev=10.26 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 22], 95.00th=[ 28], | 99.00th=[ 45], 99.50th=[ 65], 99.90th=[ 140], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 7, max= 152, per=25.63%, avg=49.72, stdev=27.83, samples=241 iops : min= 1, max= 38, avg=12.23, stdev= 6.98, samples=241 write: IOPS=12, BW=51.0KiB/s (52.3kB/s)(6260KiB/122646msec) clat (msec): min=20, max=405, avg=67.80, stdev=27.05 lat (msec): min=20, max=405, avg=67.81, stdev=27.05 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 36], 10.00th=[ 41], 20.00th=[ 47], | 30.00th=[ 53], 40.00th=[ 58], 50.00th=[ 64], 60.00th=[ 70], | 70.00th=[ 77], 80.00th=[ 85], 90.00th=[ 101], 95.00th=[ 117], | 99.00th=[ 153], 99.50th=[ 178], 99.90th=[ 199], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 16, max= 88, per=25.09%, avg=50.94, stdev=15.46, samples=244 iops : min= 4, max= 22, avg=12.54, stdev= 3.89, samples=244 lat (msec) : 4=1.53%, 10=32.55%, 20=9.24%, 50=18.52%, 100=32.94% lat (msec) : 250=5.18%, 500=0.03% cpu : usr=0.05%, sys=2.81%, ctx=3460, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5863: Thu May 1 18:34:14 2025 read: IOPS=12, BW=49.6KiB/s (50.8kB/s)(6052KiB/121975msec) clat (msec): min=2, max=104, avg=10.27, stdev= 8.50 lat (msec): min=2, max=104, avg=10.27, stdev= 8.50 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 13], 90.00th=[ 20], 95.00th=[ 27], | 99.00th=[ 42], 99.50th=[ 48], 99.90th=[ 100], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 7, max= 136, per=26.15%, avg=50.74, stdev=25.85, samples=236 iops : min= 1, max= 34, avg=12.40, stdev= 6.49, samples=236 write: IOPS=12, BW=51.1KiB/s (52.4kB/s)(6236KiB/121975msec) clat (msec): min=17, max=360, avg=68.15, stdev=27.58 lat (msec): min=17, max=360, avg=68.15, stdev=27.58 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 36], 10.00th=[ 40], 20.00th=[ 46], | 30.00th=[ 53], 40.00th=[ 58], 50.00th=[ 64], 60.00th=[ 70], | 70.00th=[ 77], 80.00th=[ 86], 90.00th=[ 104], 95.00th=[ 120], | 99.00th=[ 157], 99.50th=[ 169], 99.90th=[ 234], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 7, max= 88, per=24.96%, avg=50.67, stdev=15.93, samples=243 iops : min= 1, max= 22, avg=12.37, stdev= 4.04, samples=243 lat (msec) : 4=2.05%, 10=32.75%, 20=9.70%, 50=18.16%, 100=31.67% lat (msec) : 250=5.63%, 500=0.03% cpu : usr=0.09%, sys=2.79%, ctx=3499, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=195KiB/s (199kB/s), 48.1KiB/s-49.6KiB/s (49.3kB/s-50.8kB/s), io=23.4MiB (24.6MB), run=121975-123425msec WRITE: bw=204KiB/s (209kB/s), 51.0KiB/s-51.4KiB/s (52.3kB/s-52.7kB/s), io=24.6MiB (25.7MB), run=121975-123425msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5976: Thu May 1 18:34:21 2025 read: IOPS=60, BW=971KiB/s (994kB/s)(5920KiB/6096msec) clat (usec): min=53, max=42076, avg=7472.91, stdev=6859.48 lat (usec): min=54, max=42077, avg=7473.94, stdev=6859.47 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 96], 10.00th=[ 103], 20.00th=[ 121], | 30.00th=[ 693], 40.00th=[ 5538], 50.00th=[ 6915], 60.00th=[ 8586], | 70.00th=[ 9896], 80.00th=[12125], 90.00th=[15795], 95.00th=[19006], | 99.00th=[29230], 99.50th=[42206], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 702, max= 1440, per=24.94%, avg=967.82, stdev=240.27, samples=11 iops : min= 43, max= 90, avg=60.18, stdev=15.27, samples=11 write: IOPS=65, BW=1045KiB/s (1070kB/s)(6368KiB/6096msec) clat (usec): min=749, max=91536, avg=8265.95, stdev=9849.80 lat (usec): min=750, max=91622, avg=8267.72, stdev=9851.64 clat percentiles (usec): | 1.00th=[ 832], 5.00th=[ 979], 10.00th=[ 1074], 20.00th=[ 1205], | 30.00th=[ 1500], 40.00th=[ 2147], 50.00th=[ 4424], 60.00th=[ 7439], | 70.00th=[10552], 80.00th=[14746], 90.00th=[20579], 95.00th=[25297], | 99.00th=[46924], 99.50th=[56361], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 669, max= 1568, per=25.78%, avg=1052.18, stdev=316.33, samples=11 iops : min= 41, max= 98, avg=65.45, stdev=19.92, samples=11 lat (usec) : 100=3.91%, 250=9.51%, 500=0.52%, 750=0.65%, 1000=2.86% lat (msec) : 2=17.32%, 4=5.86%, 10=29.30%, 20=22.53%, 50=7.16% lat (msec) : 100=0.39% cpu : usr=0.39%, sys=16.65%, ctx=1157, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5977: Thu May 1 18:34:21 2025 read: IOPS=61, BW=986KiB/s (1009kB/s)(5744KiB/5827msec) clat (usec): min=54, max=54623, avg=7862.69, stdev=7106.09 lat (usec): min=54, max=54623, avg=7863.69, stdev=7106.13 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 92], 10.00th=[ 105], 20.00th=[ 133], | 30.00th=[ 4424], 40.00th=[ 6390], 50.00th=[ 7439], 60.00th=[ 8717], | 70.00th=[10290], 80.00th=[12387], 90.00th=[15270], 95.00th=[20055], | 99.00th=[33424], 99.50th=[42730], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 666, max= 1437, per=25.82%, avg=1001.82, stdev=216.29, samples=11 iops : min= 41, max= 89, avg=62.18, stdev=13.39, samples=11 write: IOPS=70, BW=1123KiB/s (1150kB/s)(6544KiB/5827msec) clat (usec): min=643, max=41926, avg=7211.80, stdev=7998.68 lat (usec): min=643, max=41928, avg=7213.69, stdev=7999.89 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 996], 10.00th=[ 1057], 20.00th=[ 1188], | 30.00th=[ 1483], 40.00th=[ 2024], 50.00th=[ 3163], 60.00th=[ 5932], | 70.00th=[ 9241], 80.00th=[13698], 90.00th=[18482], 95.00th=[23725], | 99.00th=[33817], 99.50th=[35390], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 862, max= 1628, per=27.46%, avg=1120.82, stdev=249.69, samples=11 iops : min= 53, max= 101, avg=69.64, stdev=15.56, samples=11 lat (usec) : 100=3.39%, 250=7.94%, 500=0.91%, 750=1.04%, 1000=2.47% lat (msec) : 2=18.10%, 4=8.20%, 10=28.12%, 20=23.05%, 50=6.64% lat (msec) : 100=0.13% cpu : usr=0.24%, sys=17.14%, ctx=1165, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5978: Thu May 1 18:34:21 2025 read: IOPS=66, BW=1059KiB/s (1085kB/s)(6256KiB/5905msec) clat (usec): min=53, max=46919, avg=7716.32, stdev=7223.52 lat (usec): min=54, max=46920, avg=7717.60, stdev=7223.60 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 91], 10.00th=[ 103], 20.00th=[ 121], | 30.00th=[ 269], 40.00th=[ 5866], 50.00th=[ 7373], 60.00th=[ 9110], | 70.00th=[10683], 80.00th=[12649], 90.00th=[15795], 95.00th=[19792], | 99.00th=[31851], 99.50th=[39584], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 702, max= 1437, per=27.85%, avg=1080.73, stdev=208.15, samples=11 iops : min= 43, max= 89, avg=67.18, stdev=13.08, samples=11 write: IOPS=63, BW=1022KiB/s (1046kB/s)(6032KiB/5905msec) clat (usec): min=623, max=61526, avg=7586.06, stdev=8315.91 lat (usec): min=624, max=61528, avg=7588.60, stdev=8315.08 clat percentiles (usec): | 1.00th=[ 857], 5.00th=[ 1020], 10.00th=[ 1090], 20.00th=[ 1237], | 30.00th=[ 1549], 40.00th=[ 2212], 50.00th=[ 3261], 60.00th=[ 6521], | 70.00th=[10421], 80.00th=[14484], 90.00th=[19006], 95.00th=[22938], | 99.00th=[33817], 99.50th=[48497], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 798, max= 1248, per=24.83%, avg=1013.73, stdev=130.18, samples=11 iops : min= 49, max= 78, avg=63.00, stdev= 8.21, samples=11 lat (usec) : 100=4.43%, 250=10.81%, 500=0.65%, 750=0.39%, 1000=1.69% lat (msec) : 2=16.80%, 4=7.94%, 10=25.52%, 20=25.00%, 50=6.64% lat (msec) : 100=0.13% cpu : usr=0.15%, sys=16.65%, ctx=1164, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5979: Thu May 1 18:34:21 2025 read: IOPS=61, BW=977KiB/s (1001kB/s)(6032KiB/6173msec) clat (usec): min=66, max=68578, avg=8348.44, stdev=8176.94 lat (usec): min=67, max=68578, avg=8349.35, stdev=8176.93 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 95], 10.00th=[ 104], 20.00th=[ 130], | 30.00th=[ 2311], 40.00th=[ 5997], 50.00th=[ 7701], 60.00th=[ 9241], | 70.00th=[11338], 80.00th=[13960], 90.00th=[17171], 95.00th=[20317], | 99.00th=[35914], 99.50th=[45876], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 672, max= 1341, per=24.76%, avg=960.50, stdev=190.64, samples=12 iops : min= 42, max= 83, avg=59.50, stdev=11.77, samples=12 write: IOPS=63, BW=1013KiB/s (1038kB/s)(6256KiB/6173msec) clat (usec): min=602, max=53762, avg=7550.89, stdev=8621.98 lat (usec): min=603, max=53763, avg=7553.12, stdev=8621.89 clat percentiles (usec): | 1.00th=[ 922], 5.00th=[ 1029], 10.00th=[ 1123], 20.00th=[ 1303], | 30.00th=[ 1647], 40.00th=[ 2442], 50.00th=[ 3294], 60.00th=[ 5669], | 70.00th=[ 9503], 80.00th=[13304], 90.00th=[19268], 95.00th=[24249], | 99.00th=[42206], 99.50th=[46924], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 736, max= 1277, per=24.70%, avg=1008.17, stdev=164.66, samples=12 iops : min= 46, max= 79, avg=62.50, stdev=10.04, samples=12 lat (usec) : 100=4.17%, 250=9.64%, 500=0.52%, 750=0.26%, 1000=2.08% lat (msec) : 2=15.49%, 4=11.72%, 10=23.18%, 20=25.52%, 50=7.16% lat (msec) : 100=0.26% cpu : usr=0.31%, sys=16.54%, ctx=1212, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3880KiB/s (3973kB/s), 971KiB/s-1059KiB/s (994kB/s-1085kB/s), io=23.4MiB (24.5MB), run=5827-6173msec WRITE: bw=4082KiB/s (4180kB/s), 1013KiB/s-1123KiB/s (1038kB/s-1150kB/s), io=24.6MiB (25.8MB), run=5827-6173msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5981: Thu May 1 18:34:46 2025 read: IOPS=12, BW=197KiB/s (202kB/s)(5920KiB/29991msec) clat (usec): min=3445, max=69102, avg=9674.81, stdev=7668.12 lat (usec): min=3447, max=69103, avg=9676.46, stdev=7668.02 clat percentiles (usec): | 1.00th=[ 3916], 5.00th=[ 4424], 10.00th=[ 4883], 20.00th=[ 5669], | 30.00th=[ 6259], 40.00th=[ 6718], 50.00th=[ 7373], 60.00th=[ 8094], | 70.00th=[ 9372], 80.00th=[11469], 90.00th=[15664], 95.00th=[22414], | 99.00th=[56886], 99.50th=[62129], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 31, max= 448, per=24.73%, avg=193.12, stdev=88.63, samples=59 iops : min= 1, max= 28, avg=11.64, stdev= 5.62, samples=59 write: IOPS=13, BW=212KiB/s (217kB/s)(6368KiB/29991msec) clat (msec): min=25, max=151, avg=66.26, stdev=21.53 lat (msec): min=25, max=151, avg=66.26, stdev=21.53 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 38], 10.00th=[ 42], 20.00th=[ 47], | 30.00th=[ 53], 40.00th=[ 59], 50.00th=[ 63], 60.00th=[ 69], | 70.00th=[ 75], 80.00th=[ 84], 90.00th=[ 95], 95.00th=[ 107], | 99.00th=[ 129], 99.50th=[ 142], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 64, max= 288, per=25.47%, avg=209.39, stdev=45.08, samples=59 iops : min= 4, max= 18, avg=12.66, stdev= 2.97, samples=59 lat (msec) : 4=0.65%, 10=34.77%, 20=9.64%, 50=15.62%, 100=35.55% lat (msec) : 250=3.78% cpu : usr=0.03%, sys=3.06%, ctx=882, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5982: Thu May 1 18:34:46 2025 read: IOPS=11, BW=188KiB/s (192kB/s)(5744KiB/30632msec) clat (msec): min=3, max=105, avg=10.21, stdev= 8.47 lat (msec): min=3, max=105, avg=10.21, stdev= 8.47 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 13], 90.00th=[ 18], 95.00th=[ 23], | 99.00th=[ 49], 99.50th=[ 67], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 31, max= 448, per=24.36%, avg=190.23, stdev=93.18, samples=60 iops : min= 1, max= 28, avg=11.58, stdev= 5.77, samples=60 write: IOPS=13, BW=214KiB/s (219kB/s)(6544KiB/30632msec) clat (msec): min=24, max=156, avg=65.42, stdev=20.66 lat (msec): min=24, max=156, avg=65.43, stdev=20.66 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 38], 10.00th=[ 42], 20.00th=[ 48], | 30.00th=[ 54], 40.00th=[ 59], 50.00th=[ 64], 60.00th=[ 68], | 70.00th=[ 73], 80.00th=[ 80], 90.00th=[ 92], 95.00th=[ 104], | 99.00th=[ 129], 99.50th=[ 138], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 64, max= 367, per=25.83%, avg=212.34, stdev=50.84, samples=61 iops : min= 4, max= 22, avg=12.97, stdev= 3.08, samples=61 lat (msec) : 4=0.65%, 10=31.25%, 20=11.85%, 50=15.62%, 100=37.11% lat (msec) : 250=3.52% cpu : usr=0.06%, sys=2.89%, ctx=887, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5983: Thu May 1 18:34:46 2025 read: IOPS=13, BW=214KiB/s (220kB/s)(6256KiB/29174msec) clat (usec): min=3483, max=84222, avg=9982.93, stdev=7361.68 lat (usec): min=3486, max=84223, avg=9986.53, stdev=7360.93 clat percentiles (usec): | 1.00th=[ 3851], 5.00th=[ 4686], 10.00th=[ 5145], 20.00th=[ 5866], | 30.00th=[ 6521], 40.00th=[ 7111], 50.00th=[ 7832], 60.00th=[ 8586], | 70.00th=[ 9896], 80.00th=[11469], 90.00th=[17695], 95.00th=[25035], | 99.00th=[37487], 99.50th=[40633], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 32, max= 512, per=28.07%, avg=219.23, stdev=109.61, samples=57 iops : min= 2, max= 32, avg=13.46, stdev= 6.88, samples=57 write: IOPS=12, BW=207KiB/s (212kB/s)(6032KiB/29174msec) clat (msec): min=23, max=172, avg=66.47, stdev=20.80 lat (msec): min=23, max=172, avg=66.47, stdev=20.80 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 39], 10.00th=[ 43], 20.00th=[ 49], | 30.00th=[ 55], 40.00th=[ 59], 50.00th=[ 66], 60.00th=[ 71], | 70.00th=[ 75], 80.00th=[ 81], 90.00th=[ 91], 95.00th=[ 101], | 99.00th=[ 144], 99.50th=[ 157], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 63, max= 320, per=25.00%, avg=205.52, stdev=44.56, samples=58 iops : min= 3, max= 20, avg=12.59, stdev= 2.88, samples=58 lat (msec) : 4=0.78%, 10=35.29%, 20=10.55%, 50=15.10%, 100=35.68% lat (msec) : 250=2.60% cpu : usr=0.05%, sys=3.07%, ctx=888, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5984: Thu May 1 18:34:46 2025 read: IOPS=12, BW=199KiB/s (204kB/s)(6032KiB/30252msec) clat (msec): min=3, max=105, avg=10.84, stdev=10.63 lat (msec): min=3, max=105, avg=10.85, stdev=10.63 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 12], 90.00th=[ 20], 95.00th=[ 30], | 99.00th=[ 58], 99.50th=[ 84], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 32, max= 448, per=25.65%, avg=200.32, stdev=100.25, samples=59 iops : min= 2, max= 28, avg=12.19, stdev= 6.41, samples=59 write: IOPS=12, BW=207KiB/s (212kB/s)(6256KiB/30252msec) clat (msec): min=26, max=147, avg=66.50, stdev=20.94 lat (msec): min=26, max=147, avg=66.51, stdev=20.94 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 37], 10.00th=[ 44], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 59], 50.00th=[ 64], 60.00th=[ 70], | 70.00th=[ 75], 80.00th=[ 83], 90.00th=[ 94], 95.00th=[ 107], | 99.00th=[ 134], 99.50th=[ 136], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 95, max= 320, per=25.00%, avg=205.47, stdev=48.80, samples=60 iops : min= 5, max= 20, avg=12.50, stdev= 3.18, samples=60 lat (msec) : 4=0.13%, 10=35.68%, 20=8.46%, 50=14.97%, 100=36.59% lat (msec) : 250=4.17% cpu : usr=0.03%, sys=2.99%, ctx=876, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=782KiB/s (801kB/s), 188KiB/s-214KiB/s (192kB/s-220kB/s), io=23.4MiB (24.5MB), run=29174-30632msec WRITE: bw=823KiB/s (842kB/s), 207KiB/s-214KiB/s (212kB/s-219kB/s), io=24.6MiB (25.8MB), run=29174-30632msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6023: Thu May 1 18:34:48 2025 read: IOPS=9, BW=9713KiB/s (9946kB/s)(7168KiB/738msec) clat (usec): min=546, max=143810, avg=49560.68, stdev=48770.28 lat (usec): min=547, max=143812, avg=49562.17, stdev=48770.54 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 545], 10.00th=[ 545], 20.00th=[ 627], | 30.00th=[ 34866], 40.00th=[ 34866], 50.00th=[ 44303], 60.00th=[ 55837], | 70.00th=[ 55837], 80.00th=[ 66847], 90.00th=[143655], 95.00th=[143655], | 99.00th=[143655], 99.50th=[143655], 99.90th=[143655], 99.95th=[143655], | 99.99th=[143655] bw ( KiB/s): min=10240, max=10240, per=50.85%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6938KiB/s (7104kB/s)(5120KiB/738msec) clat (msec): min=29, max=105, avg=53.37, stdev=31.24 lat (msec): min=29, max=106, avg=53.43, stdev=31.25 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 30], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 46], 60.00th=[ 46], | 70.00th=[ 56], 80.00th=[ 56], 90.00th=[ 107], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 2048, max= 2048, per=7.26%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 50=41.67%, 100=25.00%, 250=16.67% cpu : usr=0.14%, sys=26.46%, ctx=627, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6024: Thu May 1 18:34:48 2025 read: IOPS=1, BW=2014KiB/s (2062kB/s)(2048KiB/1017msec) clat (usec): min=48169, max=72639, avg=60404.48, stdev=17303.11 lat (usec): min=48170, max=72641, avg=60405.91, stdev=17303.37 clat percentiles (usec): | 1.00th=[47973], 5.00th=[47973], 10.00th=[47973], 20.00th=[47973], | 30.00th=[47973], 40.00th=[47973], 50.00th=[47973], 60.00th=[72877], | 70.00th=[72877], 80.00th=[72877], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 2048, max= 2048, per=10.17%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9.83MiB/s (10.3MB/s)(10.0MiB/1017msec) clat (msec): min=29, max=122, avg=71.24, stdev=31.45 lat (msec): min=29, max=122, avg=71.33, stdev=31.46 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 33], | 30.00th=[ 36], 40.00th=[ 64], 50.00th=[ 71], 60.00th=[ 79], | 70.00th=[ 90], 80.00th=[ 90], 90.00th=[ 106], 95.00th=[ 123], | 99.00th=[ 123], 99.50th=[ 123], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 6144, max= 6144, per=21.79%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=50.00%, 250=16.67% cpu : usr=0.00%, sys=26.87%, ctx=277, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6026: Thu May 1 18:34:48 2025 read: IOPS=8, BW=8969KiB/s (9185kB/s)(6144KiB/685msec) clat (usec): min=529, max=49054, avg=19275.80, stdev=22332.08 lat (usec): min=530, max=49055, avg=19276.98, stdev=22332.10 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 529], 10.00th=[ 529], 20.00th=[ 545], | 30.00th=[ 545], 40.00th=[ 1287], 50.00th=[ 1287], 60.00th=[20841], | 70.00th=[43254], 80.00th=[43254], 90.00th=[49021], 95.00th=[49021], | 99.00th=[49021], 99.50th=[49021], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 8192, max= 8192, per=40.68%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8969KiB/s (9185kB/s)(6144KiB/685msec) clat (msec): min=39, max=151, avg=83.85, stdev=39.82 lat (msec): min=39, max=151, avg=83.91, stdev=39.85 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 58], | 30.00th=[ 58], 40.00th=[ 67], 50.00th=[ 67], 60.00th=[ 87], | 70.00th=[ 103], 80.00th=[ 103], 90.00th=[ 153], 95.00th=[ 153], | 99.00th=[ 153], 99.50th=[ 153], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 8192, max= 8192, per=29.06%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 2=8.33%, 50=33.33%, 100=25.00%, 250=16.67% cpu : usr=0.00%, sys=26.32%, ctx=302, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6028: Thu May 1 18:34:48 2025 read: IOPS=7, BW=7367KiB/s (7544kB/s)(5120KiB/695msec) clat (usec): min=502, max=107028, avg=51658.04, stdev=38925.40 lat (usec): min=504, max=107030, avg=51659.65, stdev=38925.51 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 502], 10.00th=[ 502], 20.00th=[ 502], | 30.00th=[ 34866], 40.00th=[ 34866], 50.00th=[ 55313], 60.00th=[ 55313], | 70.00th=[ 61080], 80.00th=[ 61080], 90.00th=[107480], 95.00th=[107480], | 99.00th=[107480], 99.50th=[107480], 99.90th=[107480], 99.95th=[107480], | 99.99th=[107480] bw ( KiB/s): min= 8175, max= 8175, per=40.60%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.1MiB/s (10.6MB/s)(7168KiB/695msec) clat (msec): min=25, max=165, avg=60.72, stdev=48.71 lat (msec): min=25, max=165, avg=60.79, stdev=48.68 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 29], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 43], 60.00th=[ 50], | 70.00th=[ 50], 80.00th=[ 72], 90.00th=[ 167], 95.00th=[ 167], | 99.00th=[ 167], 99.50th=[ 167], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 8175, max= 8175, per=29.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=50.00%, 100=25.00%, 250=16.67% cpu : usr=0.00%, sys=30.98%, ctx=299, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.7MiB/s (20.6MB/s), 2014KiB/s-9713KiB/s (2062kB/s-9946kB/s), io=20.0MiB (20.0MB), run=685-1017msec WRITE: bw=27.5MiB/s (28.9MB/s), 6938KiB/s-10.1MiB/s (7104kB/s-10.6MB/s), io=28.0MiB (29.4MB), run=685-1017msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6025: Thu May 1 18:34:48 2025 read: IOPS=5, BW=6003KiB/s (6147kB/s)(7168KiB/1194msec) clat (msec): min=18, max=201, avg=88.09, stdev=69.06 lat (msec): min=18, max=201, avg=88.09, stdev=69.06 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 34], | 30.00th=[ 38], 40.00th=[ 38], 50.00th=[ 65], 60.00th=[ 105], | 70.00th=[ 105], 80.00th=[ 157], 90.00th=[ 203], 95.00th=[ 203], | 99.00th=[ 203], 99.50th=[ 203], 99.90th=[ 203], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 2048, max= 8159, per=32.87%, avg=5103.50, stdev=4321.13, samples=2 iops : min= 2, max= 7, avg= 4.50, stdev= 3.54, samples=2 write: IOPS=4, BW=4288KiB/s (4391kB/s)(5120KiB/1194msec) clat (msec): min=46, max=116, avg=73.88, stdev=28.45 lat (msec): min=47, max=116, avg=73.96, stdev=28.44 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 49], 40.00th=[ 49], 50.00th=[ 79], 60.00th=[ 79], | 70.00th=[ 81], 80.00th=[ 81], 90.00th=[ 117], 95.00th=[ 117], | 99.00th=[ 117], 99.50th=[ 117], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 6119, max= 6119, per=28.15%, avg=6119.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=25.00%, 250=33.33% cpu : usr=0.00%, sys=3.94%, ctx=29, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6027: Thu May 1 18:34:48 2025 read: IOPS=1, BW=1553KiB/s (1590kB/s)(2048KiB/1319msec) clat (usec): min=34225, max=77204, avg=55715.04, stdev=30390.30 lat (usec): min=34227, max=77206, avg=55716.99, stdev=30390.77 clat percentiles (usec): | 1.00th=[34341], 5.00th=[34341], 10.00th=[34341], 20.00th=[34341], | 30.00th=[34341], 40.00th=[34341], 50.00th=[34341], 60.00th=[77071], | 70.00th=[77071], 80.00th=[77071], 90.00th=[77071], 95.00th=[77071], | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 2048, max= 2048, per=13.19%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=7, BW=7763KiB/s (7950kB/s)(10.0MiB/1319msec) clat (msec): min=39, max=227, avg=103.97, stdev=70.26 lat (msec): min=39, max=227, avg=104.05, stdev=70.27 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 51], 50.00th=[ 77], 60.00th=[ 88], | 70.00th=[ 150], 80.00th=[ 150], 90.00th=[ 215], 95.00th=[ 228], | 99.00th=[ 228], 99.50th=[ 228], 99.90th=[ 228], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 2048, max= 8192, per=23.55%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (msec) : 50=33.33%, 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=6.22%, ctx=37, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6029: Thu May 1 18:34:48 2025 read: IOPS=4, BW=5020KiB/s (5140kB/s)(6144KiB/1224msec) clat (msec): min=14, max=180, avg=59.28, stdev=64.40 lat (msec): min=14, max=180, avg=59.28, stdev=64.40 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 23], 50.00th=[ 23], 60.00th=[ 34], | 70.00th=[ 83], 80.00th=[ 83], 90.00th=[ 182], 95.00th=[ 182], | 99.00th=[ 182], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 4087, max= 4087, per=26.32%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=4, BW=5020KiB/s (5140kB/s)(6144KiB/1224msec) clat (msec): min=44, max=379, avg=142.36, stdev=140.05 lat (msec): min=44, max=379, avg=142.45, stdev=140.11 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 54], 50.00th=[ 54], 60.00th=[ 74], | 70.00th=[ 251], 80.00th=[ 251], 90.00th=[ 380], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 4087, max= 4087, per=18.80%, avg=4087.00, stdev= 0.00, samples=2 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=2 lat (msec) : 20=8.33%, 50=33.33%, 100=33.33%, 250=16.67%, 500=8.33% cpu : usr=0.25%, sys=3.27%, ctx=34, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6030: Thu May 1 18:34:48 2025 read: IOPS=4, BW=4429KiB/s (4535kB/s)(5120KiB/1156msec) clat (usec): min=17581, max=72495, avg=43889.61, stdev=23297.73 lat (usec): min=17583, max=72497, avg=43891.32, stdev=23297.76 clat percentiles (usec): | 1.00th=[17695], 5.00th=[17695], 10.00th=[17695], 20.00th=[17695], | 30.00th=[29230], 40.00th=[29230], 50.00th=[36439], 60.00th=[36439], | 70.00th=[63701], 80.00th=[63701], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 8192, max= 8192, per=52.76%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6201KiB/s (6350kB/s)(7168KiB/1156msec) clat (msec): min=79, max=237, avg=118.46, stdev=58.34 lat (msec): min=79, max=237, avg=118.55, stdev=58.34 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 81], 10.00th=[ 81], 20.00th=[ 85], | 30.00th=[ 86], 40.00th=[ 86], 50.00th=[ 88], 60.00th=[ 102], | 70.00th=[ 102], 80.00th=[ 155], 90.00th=[ 239], 95.00th=[ 239], | 99.00th=[ 239], 99.50th=[ 239], 99.90th=[ 239], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 4096, max= 6144, per=23.55%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 20=8.33%, 50=16.67%, 100=50.00%, 250=25.00% cpu : usr=0.26%, sys=4.76%, ctx=29, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.2MiB/s (15.9MB/s), 1553KiB/s-6003KiB/s (1590kB/s-6147kB/s), io=20.0MiB (20.0MB), run=1156-1319msec WRITE: bw=21.2MiB/s (22.3MB/s), 4288KiB/s-7763KiB/s (4391kB/s-7950kB/s), io=28.0MiB (29.4MB), run=1156-1319msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6038: Thu May 1 18:34:51 2025 read: IOPS=3, BW=15.5MiB/s (16.3MB/s)(12.0MiB/773msec) clat (msec): min=150, max=400, avg=244.78, stdev=135.54 lat (msec): min=150, max=400, avg=244.78, stdev=135.54 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 150], 10.00th=[ 150], 20.00th=[ 150], | 30.00th=[ 150], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 184], | 70.00th=[ 401], 80.00th=[ 401], 90.00th=[ 401], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 8175, max= 8175, per=76.94%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=22.02%, ctx=803, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6039: Thu May 1 18:34:51 2025 read: IOPS=0, BW=2656KiB/s (2720kB/s)(4096KiB/1542msec) clat (nsec): min=234527k, max=234527k, avg=234527174.00, stdev= 0.00 lat (nsec): min=234531k, max=234531k, avg=234530798.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 234], 10.00th=[ 234], 20.00th=[ 234], | 30.00th=[ 234], 40.00th=[ 234], 50.00th=[ 234], 60.00th=[ 234], | 70.00th=[ 234], 80.00th=[ 234], 90.00th=[ 234], 95.00th=[ 234], | 99.00th=[ 234], 99.50th=[ 234], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 8078, max= 8078, per=76.03%, avg=8078.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5313KiB/s (5440kB/s)(8192KiB/1542msec) clat (msec): min=569, max=703, avg=636.70, stdev=95.00 lat (msec): min=569, max=704, avg=637.07, stdev=94.95 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 567], 10.00th=[ 567], 20.00th=[ 567], | 30.00th=[ 567], 40.00th=[ 567], 50.00th=[ 567], 60.00th=[ 701], | 70.00th=[ 701], 80.00th=[ 701], 90.00th=[ 701], 95.00th=[ 701], | 99.00th=[ 701], 99.50th=[ 701], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 8192, max= 8192, per=38.55%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=66.67% cpu : usr=0.00%, sys=12.13%, ctx=298, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6040: Thu May 1 18:34:51 2025 write: IOPS=2, BW=8192KiB/s (8389kB/s)(12.0MiB/1500msec) clat (msec): min=374, max=613, avg=481.34, stdev=121.51 lat (msec): min=374, max=613, avg=481.66, stdev=121.62 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 376], 10.00th=[ 376], 20.00th=[ 376], | 30.00th=[ 376], 40.00th=[ 456], 50.00th=[ 456], 60.00th=[ 456], | 70.00th=[ 617], 80.00th=[ 617], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 8143, max= 8143, per=38.32%, avg=8143.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.00%, sys=17.81%, ctx=81, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6042: Thu May 1 18:34:51 2025 write: IOPS=2, BW=10.6MiB/s (11.2MB/s)(12.0MiB/1128msec) clat (msec): min=215, max=604, avg=356.86, stdev=215.05 lat (msec): min=216, max=604, avg=357.52, stdev=214.66 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 215], 10.00th=[ 215], 20.00th=[ 215], | 30.00th=[ 215], 40.00th=[ 251], 50.00th=[ 251], 60.00th=[ 251], | 70.00th=[ 609], 80.00th=[ 609], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 8159, max= 8192, per=38.47%, avg=8175.50, stdev=23.33, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=24.05%, ctx=84, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.4MiB/s (10.9MB/s), 2656KiB/s-15.5MiB/s (2720kB/s-16.3MB/s), io=16.0MiB (16.8MB), run=773-1542msec WRITE: bw=20.8MiB/s (21.8MB/s), 5313KiB/s-10.6MiB/s (5440kB/s-11.2MB/s), io=32.0MiB (33.6MB), run=1128-1542msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6041: Thu May 1 18:34:52 2025 read: IOPS=2, BW=9191KiB/s (9411kB/s)(12.0MiB/1337msec) clat (msec): min=268, max=695, avg=440.06, stdev=225.61 lat (msec): min=268, max=695, avg=440.06, stdev=225.61 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 271], 10.00th=[ 271], 20.00th=[ 271], | 30.00th=[ 271], 40.00th=[ 355], 50.00th=[ 355], 60.00th=[ 355], | 70.00th=[ 693], 80.00th=[ 693], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 8192, max= 8192, per=96.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.30%, sys=1.05%, ctx=19, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6043: Thu May 1 18:34:52 2025 read: IOPS=0, BW=2133KiB/s (2185kB/s)(4096KiB/1920msec) clat (nsec): min=286923k, max=286923k, avg=286923194.00, stdev= 0.00 lat (nsec): min=286925k, max=286925k, avg=286925204.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 288], 10.00th=[ 288], 20.00th=[ 288], | 30.00th=[ 288], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 288], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 8192, max= 8192, per=96.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4267KiB/s (4369kB/s)(8192KiB/1920msec) clat (msec): min=677, max=933, avg=805.50, stdev=180.72 lat (msec): min=678, max=933, avg=805.76, stdev=180.61 clat percentiles (msec): | 1.00th=[ 676], 5.00th=[ 676], 10.00th=[ 676], 20.00th=[ 676], | 30.00th=[ 676], 40.00th=[ 676], 50.00th=[ 676], 60.00th=[ 936], | 70.00th=[ 936], 80.00th=[ 936], 90.00th=[ 936], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 8192, max= 8192, per=49.93%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.16%, sys=3.18%, ctx=70, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6044: Thu May 1 18:34:52 2025 write: IOPS=1, BW=6153KiB/s (6301kB/s)(12.0MiB/1997msec) clat (msec): min=185, max=1067, avg=635.81, stdev=441.36 lat (msec): min=186, max=1068, avg=636.20, stdev=441.36 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 186], 10.00th=[ 186], 20.00th=[ 186], | 30.00th=[ 186], 40.00th=[ 651], 50.00th=[ 651], 60.00th=[ 651], | 70.00th=[ 1070], 80.00th=[ 1070], 90.00th=[ 1070], 95.00th=[ 1070], | 99.00th=[ 1070], 99.50th=[ 1070], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 8143, max= 8143, per=49.63%, avg=8143.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=4.81%, ctx=41, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6045: Thu May 1 18:34:52 2025 write: IOPS=1, BW=6689KiB/s (6850kB/s)(12.0MiB/1837msec) clat (msec): min=155, max=1135, avg=587.65, stdev=500.06 lat (msec): min=155, max=1135, avg=588.11, stdev=500.20 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 472], 50.00th=[ 472], 60.00th=[ 472], | 70.00th=[ 1133], 80.00th=[ 1133], 90.00th=[ 1133], 95.00th=[ 1133], | 99.00th=[ 1133], 99.50th=[ 1133], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 8175, max= 8175, per=49.82%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.11%, sys=2.12%, ctx=24, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8533KiB/s (8738kB/s), 2133KiB/s-9191KiB/s (2185kB/s-9411kB/s), io=16.0MiB (16.8MB), run=1337-1920msec WRITE: bw=16.0MiB/s (16.8MB/s), 4267KiB/s-6689KiB/s (4369kB/s-6850kB/s), io=32.0MiB (33.6MB), run=1837-1997msec