== sanity test 398b: DIO and buffer IO race ============== 00:28:18 (1746160098) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.30019 s, 21.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4280: Fri May 2 00:29:05 2025 read: IOPS=35, BW=142KiB/s (145kB/s)(5992KiB/42306msec) clat (usec): min=52, max=389972, avg=12527.28, stdev=15335.30 lat (usec): min=52, max=389973, avg=12530.39, stdev=15334.68 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 102], 10.00th=[ 117], 20.00th=[ 167], | 30.00th=[ 6587], 40.00th=[ 8586], 50.00th=[ 10552], 60.00th=[ 12780], | 70.00th=[ 15795], 80.00th=[ 20055], 90.00th=[ 26084], 95.00th=[ 31065], | 99.00th=[ 46924], 99.50th=[ 53740], 99.90th=[219153], 99.95th=[387974], | 99.99th=[387974] bw ( KiB/s): min= 8, max= 256, per=24.88%, avg=140.85, stdev=43.04, samples=84 iops : min= 2, max= 64, avg=34.89, stdev=10.76, samples=84 write: IOPS=37, BW=149KiB/s (152kB/s)(6296KiB/42306msec) clat (usec): min=634, max=141397, avg=14786.67, stdev=15356.08 lat (usec): min=634, max=141399, avg=14789.71, stdev=15357.42 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 775], 10.00th=[ 832], 20.00th=[ 1106], | 30.00th=[ 2737], 40.00th=[ 6915], 50.00th=[ 11600], 60.00th=[ 15270], | 70.00th=[ 19792], 80.00th=[ 25560], 90.00th=[ 35390], 95.00th=[ 42730], | 99.00th=[ 65799], 99.50th=[ 81265], 99.90th=[125305], 99.95th=[141558], | 99.99th=[141558] bw ( KiB/s): min= 64, max= 256, per=25.23%, avg=149.59, stdev=43.25, samples=83 iops : min= 16, max= 64, avg=37.08, stdev=10.88, samples=83 lat (usec) : 100=2.21%, 250=8.17%, 500=0.46%, 750=1.89%, 1000=7.75% lat (msec) : 2=4.79%, 4=4.65%, 10=16.44%, 20=28.61%, 50=23.50% lat (msec) : 100=1.37%, 250=0.13%, 500=0.03% cpu : usr=0.16%, sys=11.63%, ctx=4799, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4281: Fri May 2 00:29:05 2025 read: IOPS=35, BW=140KiB/s (144kB/s)(5940KiB/42291msec) clat (usec): min=71, max=203996, avg=12167.74, stdev=12020.12 lat (usec): min=71, max=203997, avg=12169.12, stdev=12020.07 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 98], 10.00th=[ 114], 20.00th=[ 159], | 30.00th=[ 6325], 40.00th=[ 8586], 50.00th=[ 10421], 60.00th=[ 12649], | 70.00th=[ 15795], 80.00th=[ 19792], 90.00th=[ 25560], 95.00th=[ 32113], | 99.00th=[ 41681], 99.50th=[ 61080], 99.90th=[ 96994], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 8, max= 272, per=24.66%, avg=139.58, stdev=48.43, samples=84 iops : min= 2, max= 68, avg=34.56, stdev=12.08, samples=84 write: IOPS=37, BW=150KiB/s (154kB/s)(6348KiB/42291msec) clat (usec): min=521, max=386932, avg=15090.98, stdev=18377.64 lat (usec): min=523, max=386934, avg=15092.85, stdev=18377.99 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 775], 10.00th=[ 840], 20.00th=[ 1106], | 30.00th=[ 2212], 40.00th=[ 5604], 50.00th=[ 11338], 60.00th=[ 15401], | 70.00th=[ 19792], 80.00th=[ 25822], 90.00th=[ 36439], 95.00th=[ 43779], | 99.00th=[ 64226], 99.50th=[ 82314], 99.90th=[162530], 99.95th=[387974], | 99.99th=[387974] bw ( KiB/s): min= 79, max= 296, per=25.35%, avg=150.31, stdev=46.69, samples=83 iops : min= 19, max= 74, avg=37.24, stdev=11.68, samples=83 lat (usec) : 100=2.64%, 250=8.07%, 500=0.42%, 750=1.76%, 1000=8.07% lat (msec) : 2=5.79%, 4=4.46%, 10=16.21%, 20=27.80%, 50=22.59% lat (msec) : 100=2.02%, 250=0.13%, 500=0.03% cpu : usr=0.19%, sys=11.44%, ctx=4783, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4282: Fri May 2 00:29:05 2025 read: IOPS=35, BW=142KiB/s (146kB/s)(6028KiB/42382msec) clat (usec): min=53, max=390118, avg=12345.20, stdev=15144.61 lat (usec): min=54, max=390120, avg=12347.55, stdev=15143.97 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 96], 10.00th=[ 109], 20.00th=[ 141], | 30.00th=[ 5932], 40.00th=[ 8455], 50.00th=[ 10552], 60.00th=[ 12780], | 70.00th=[ 15795], 80.00th=[ 20055], 90.00th=[ 26346], 95.00th=[ 31065], | 99.00th=[ 42730], 99.50th=[ 49021], 99.90th=[139461], 99.95th=[392168], | 99.99th=[392168] bw ( KiB/s): min= 8, max= 255, per=24.74%, avg=140.01, stdev=45.79, samples=84 iops : min= 2, max= 63, avg=34.70, stdev=11.52, samples=84 write: IOPS=36, BW=148KiB/s (151kB/s)(6260KiB/42382msec) clat (usec): min=547, max=161658, avg=15003.54, stdev=16076.59 lat (usec): min=549, max=161660, avg=15005.24, stdev=16076.64 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 766], 10.00th=[ 848], 20.00th=[ 1221], | 30.00th=[ 2442], 40.00th=[ 6194], 50.00th=[ 11207], 60.00th=[ 15008], | 70.00th=[ 20055], 80.00th=[ 25560], 90.00th=[ 35390], 95.00th=[ 45351], | 99.00th=[ 65799], 99.50th=[ 77071], 99.90th=[128451], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 24, max= 272, per=24.44%, avg=144.95, stdev=43.58, samples=84 iops : min= 6, max= 68, avg=35.94, stdev=10.90, samples=84 lat (usec) : 100=3.32%, 250=8.11%, 500=0.62%, 750=2.18%, 1000=7.00% lat (msec) : 2=5.37%, 4=4.98%, 10=15.40%, 20=27.90%, 50=23.14% lat (msec) : 100=1.73%, 250=0.23%, 500=0.03% cpu : usr=0.17%, sys=11.61%, ctx=4976, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4284: Fri May 2 00:29:05 2025 read: IOPS=35, BW=143KiB/s (147kB/s)(6052KiB/42175msec) clat (usec): min=66, max=99051, avg=12161.11, stdev=11304.09 lat (usec): min=67, max=99052, avg=12167.12, stdev=11318.38 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 98], 10.00th=[ 111], 20.00th=[ 149], | 30.00th=[ 6325], 40.00th=[ 8586], 50.00th=[10552], 60.00th=[12649], | 70.00th=[15664], 80.00th=[19792], 90.00th=[26346], 95.00th=[32637], | 99.00th=[46400], 99.50th=[56886], 99.90th=[91751], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 64, max= 231, per=25.45%, avg=144.04, stdev=40.53, samples=83 iops : min= 16, max= 57, avg=35.66, stdev=10.07, samples=83 write: IOPS=36, BW=148KiB/s (151kB/s)(6236KiB/42175msec) clat (usec): min=609, max=383504, avg=15061.54, stdev=18169.57 lat (usec): min=609, max=383505, avg=15072.56, stdev=18164.97 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 783], 10.00th=[ 857], 20.00th=[ 1270], | 30.00th=[ 2671], 40.00th=[ 6325], 50.00th=[ 11076], 60.00th=[ 15401], | 70.00th=[ 19530], 80.00th=[ 25297], 90.00th=[ 35390], 95.00th=[ 43779], | 99.00th=[ 65274], 99.50th=[ 86508], 99.90th=[147850], 99.95th=[383779], | 99.99th=[383779] bw ( KiB/s): min= 8, max= 248, per=24.76%, avg=146.82, stdev=41.22, samples=84 iops : min= 2, max= 62, avg=36.36, stdev=10.33, samples=84 lat (usec) : 100=2.96%, 250=8.59%, 500=0.62%, 750=1.82%, 1000=7.00% lat (msec) : 2=5.11%, 4=4.69%, 10=16.31%, 20=28.22%, 50=23.08% lat (msec) : 100=1.40%, 250=0.16%, 500=0.03% cpu : usr=0.20%, sys=11.51%, ctx=4840, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=567KiB/s (580kB/s), 140KiB/s-143KiB/s (144kB/s-147kB/s), io=23.4MiB (24.6MB), run=42175-42382msec WRITE: bw=593KiB/s (607kB/s), 148KiB/s-150KiB/s (151kB/s-154kB/s), io=24.6MiB (25.7MB), run=42175-42382msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4285: Fri May 2 00:31:22 2025 read: IOPS=8, BW=33.4KiB/s (34.2kB/s)(5992KiB/179266msec) clat (msec): min=2, max=395, avg=15.29, stdev=15.02 lat (msec): min=2, max=395, avg=15.29, stdev=15.02 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 16], 80.00th=[ 20], 90.00th=[ 28], 95.00th=[ 35], | 99.00th=[ 69], 99.50th=[ 78], 99.90th=[ 157], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 7, max= 120, per=25.99%, avg=34.57, stdev=20.74, samples=343 iops : min= 1, max= 30, avg= 8.40, stdev= 5.21, samples=343 write: IOPS=8, BW=35.1KiB/s (35.0kB/s)(6296KiB/179266msec) clat (msec): min=39, max=282, avg=99.19, stdev=34.65 lat (msec): min=39, max=282, avg=99.19, stdev=34.65 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 54], 10.00th=[ 61], 20.00th=[ 70], | 30.00th=[ 79], 40.00th=[ 86], 50.00th=[ 93], 60.00th=[ 103], | 70.00th=[ 113], 80.00th=[ 127], 90.00th=[ 144], 95.00th=[ 161], | 99.00th=[ 205], 99.50th=[ 224], 99.90th=[ 268], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 7, max= 64, per=24.90%, avg=34.85, stdev= 9.16, samples=357 iops : min= 1, max= 16, avg= 8.47, stdev= 2.36, samples=357 lat (msec) : 4=0.16%, 10=17.90%, 20=21.58%, 50=9.57%, 100=28.94% lat (msec) : 250=21.65%, 500=0.20% cpu : usr=0.05%, sys=2.18%, ctx=3749, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4286: Fri May 2 00:31:23 2025 read: IOPS=8, BW=33.1KiB/s (33.9kB/s)(5940KiB/179426msec) clat (msec): min=3, max=436, avg=15.70, stdev=16.61 lat (msec): min=3, max=436, avg=15.70, stdev=16.61 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 17], 80.00th=[ 21], 90.00th=[ 28], 95.00th=[ 35], | 99.00th=[ 62], 99.50th=[ 73], 99.90th=[ 251], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 7, max= 120, per=25.94%, avg=34.50, stdev=21.24, samples=339 iops : min= 1, max= 30, avg= 8.29, stdev= 5.33, samples=339 write: IOPS=8, BW=35.4KiB/s (36.2kB/s)(6348KiB/179426msec) clat (msec): min=31, max=266, avg=98.22, stdev=32.26 lat (msec): min=31, max=266, avg=98.23, stdev=32.26 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 55], 10.00th=[ 61], 20.00th=[ 70], | 30.00th=[ 79], 40.00th=[ 87], 50.00th=[ 94], 60.00th=[ 103], | 70.00th=[ 111], 80.00th=[ 124], 90.00th=[ 142], 95.00th=[ 157], | 99.00th=[ 201], 99.50th=[ 211], 99.90th=[ 222], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 7, max= 64, per=24.99%, avg=34.99, stdev= 9.12, samples=357 iops : min= 1, max= 16, avg= 8.41, stdev= 2.34, samples=357 lat (msec) : 4=0.03%, 10=17.58%, 20=20.67%, 50=10.25%, 100=29.30% lat (msec) : 250=22.07%, 500=0.10% cpu : usr=0.03%, sys=2.22%, ctx=3736, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4287: Fri May 2 00:31:23 2025 read: IOPS=8, BW=33.6KiB/s (34.4kB/s)(6028KiB/179381msec) clat (msec): min=3, max=253, avg=15.46, stdev=12.69 lat (msec): min=3, max=253, avg=15.47, stdev=12.69 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 17], 80.00th=[ 21], 90.00th=[ 29], 95.00th=[ 37], | 99.00th=[ 56], 99.50th=[ 70], 99.90th=[ 159], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 7, max= 120, per=25.74%, avg=34.23, stdev=20.38, samples=347 iops : min= 1, max= 30, avg= 8.24, stdev= 5.14, samples=347 write: IOPS=8, BW=34.9KiB/s (35.7kB/s)(6260KiB/179381msec) clat (msec): min=38, max=453, avg=99.53, stdev=33.65 lat (msec): min=38, max=453, avg=99.53, stdev=33.65 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 56], 10.00th=[ 63], 20.00th=[ 71], | 30.00th=[ 81], 40.00th=[ 88], 50.00th=[ 95], 60.00th=[ 104], | 70.00th=[ 112], 80.00th=[ 125], 90.00th=[ 142], 95.00th=[ 157], | 99.00th=[ 203], 99.50th=[ 215], 99.90th=[ 279], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 7, max= 56, per=24.66%, avg=34.52, stdev= 8.92, samples=357 iops : min= 1, max= 14, avg= 8.31, stdev= 2.30, samples=357 lat (msec) : 4=0.03%, 10=18.03%, 20=20.83%, 50=10.71%, 100=28.12% lat (msec) : 250=22.17%, 500=0.10% cpu : usr=0.06%, sys=2.17%, ctx=3693, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4288: Fri May 2 00:31:23 2025 read: IOPS=8, BW=34.0KiB/s (34.8kB/s)(6052KiB/177966msec) clat (msec): min=3, max=253, avg=15.17, stdev=12.43 lat (msec): min=3, max=253, avg=15.17, stdev=12.43 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 28], 95.00th=[ 36], | 99.00th=[ 62], 99.50th=[ 71], 99.90th=[ 115], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 7, max= 111, per=26.65%, avg=35.45, stdev=21.11, samples=336 iops : min= 1, max= 27, avg= 8.52, stdev= 5.31, samples=336 write: IOPS=8, BW=35.0KiB/s (35.9kB/s)(6236KiB/177966msec) clat (msec): min=35, max=625, avg=99.25, stdev=35.60 lat (msec): min=35, max=625, avg=99.25, stdev=35.60 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 55], 10.00th=[ 62], 20.00th=[ 71], | 30.00th=[ 80], 40.00th=[ 87], 50.00th=[ 95], 60.00th=[ 103], | 70.00th=[ 112], 80.00th=[ 125], 90.00th=[ 144], 95.00th=[ 161], | 99.00th=[ 201], 99.50th=[ 218], 99.90th=[ 284], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 7, max= 64, per=24.74%, avg=34.64, stdev= 9.15, samples=354 iops : min= 1, max= 16, avg= 8.32, stdev= 2.36, samples=354 lat (msec) : 4=0.20%, 10=19.21%, 20=20.02%, 50=10.29%, 100=28.55% lat (msec) : 250=21.61%, 500=0.10%, 750=0.03% cpu : usr=0.03%, sys=2.22%, ctx=3720, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=134KiB/s (137kB/s), 33.1KiB/s-34.0KiB/s (33.9kB/s-34.8kB/s), io=23.4MiB (24.6MB), run=177966-179426msec WRITE: bw=140KiB/s (143kB/s), 34.9KiB/s-35.4KiB/s (35.7kB/s-36.2kB/s), io=24.6MiB (25.7MB), run=177966-179426msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4439: Fri May 2 00:31:34 2025 read: IOPS=38, BW=619KiB/s (634kB/s)(5920KiB/9559msec) clat (usec): min=75, max=68091, avg=10898.84, stdev=12110.72 lat (usec): min=76, max=68093, avg=10899.97, stdev=12110.81 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 93], 10.00th=[ 100], 20.00th=[ 115], | 30.00th=[ 133], 40.00th=[ 881], 50.00th=[ 8586], 60.00th=[12518], | 70.00th=[16057], 80.00th=[20317], 90.00th=[27395], 95.00th=[32113], | 99.00th=[54264], 99.50th=[61080], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 251, max= 858, per=24.55%, avg=614.67, stdev=165.14, samples=18 iops : min= 15, max= 53, avg=37.78, stdev=10.36, samples=18 write: IOPS=41, BW=666KiB/s (682kB/s)(6368KiB/9559msec) clat (usec): min=903, max=218628, avg=13634.83, stdev=18957.10 lat (usec): min=905, max=218629, avg=13636.78, stdev=18957.06 clat percentiles (usec): | 1.00th=[ 955], 5.00th=[ 1057], 10.00th=[ 1156], 20.00th=[ 1352], | 30.00th=[ 2008], 40.00th=[ 3982], 50.00th=[ 6587], 60.00th=[ 10814], | 70.00th=[ 16712], 80.00th=[ 24249], 90.00th=[ 34341], 95.00th=[ 41157], | 99.00th=[ 72877], 99.50th=[139461], 99.90th=[219153], 99.95th=[219153], | 99.99th=[219153] bw ( KiB/s): min= 283, max= 1312, per=25.03%, avg=659.17, stdev=258.68, samples=18 iops : min= 17, max= 82, avg=40.56, stdev=16.32, samples=18 lat (usec) : 100=4.82%, 250=13.54%, 500=0.39%, 750=0.26%, 1000=2.08% lat (msec) : 2=14.71%, 4=5.86%, 10=13.80%, 20=21.09%, 50=21.22% lat (msec) : 100=1.82%, 250=0.39% cpu : usr=0.21%, sys=11.39%, ctx=1039, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4440: Fri May 2 00:31:34 2025 read: IOPS=38, BW=609KiB/s (623kB/s)(5744KiB/9436msec) clat (usec): min=86, max=102984, avg=12411.21, stdev=13859.25 lat (usec): min=86, max=102985, avg=12417.99, stdev=13857.69 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 100], 10.00th=[ 110], 20.00th=[ 127], | 30.00th=[ 141], 40.00th=[ 6718], 50.00th=[ 10159], 60.00th=[ 13566], | 70.00th=[ 17957], 80.00th=[ 22938], 90.00th=[ 28967], 95.00th=[ 35390], | 99.00th=[ 63701], 99.50th=[ 89654], 99.90th=[103285], 99.95th=[103285], | 99.99th=[103285] bw ( KiB/s): min= 319, max= 990, per=24.41%, avg=611.28, stdev=170.75, samples=18 iops : min= 19, max= 61, avg=37.78, stdev=10.55, samples=18 write: IOPS=43, BW=694KiB/s (710kB/s)(6544KiB/9436msec) clat (usec): min=886, max=114945, avg=12052.56, stdev=14432.05 lat (usec): min=888, max=114946, avg=12057.02, stdev=14432.07 clat percentiles (usec): | 1.00th=[ 971], 5.00th=[ 1074], 10.00th=[ 1139], 20.00th=[ 1319], | 30.00th=[ 1975], 40.00th=[ 3195], 50.00th=[ 5735], 60.00th=[ 9634], | 70.00th=[ 14746], 80.00th=[ 20841], 90.00th=[ 34866], 95.00th=[ 41157], | 99.00th=[ 51119], 99.50th=[ 58983], 99.90th=[114820], 99.95th=[114820], | 99.99th=[114820] bw ( KiB/s): min= 444, max= 864, per=25.98%, avg=684.39, stdev=142.21, samples=18 iops : min= 27, max= 54, avg=42.39, stdev= 8.89, samples=18 lat (usec) : 100=2.34%, 250=14.06%, 500=0.39%, 750=0.52%, 1000=1.43% lat (msec) : 2=15.36%, 4=7.16%, 10=14.45%, 20=21.22%, 50=21.48% lat (msec) : 100=1.30%, 250=0.26% cpu : usr=0.16%, sys=11.41%, ctx=954, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4441: Fri May 2 00:31:34 2025 read: IOPS=40, BW=654KiB/s (670kB/s)(6256KiB/9564msec) clat (usec): min=80, max=64647, avg=11812.03, stdev=12501.66 lat (usec): min=80, max=64649, avg=11818.18, stdev=12504.29 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 91], 10.00th=[ 100], 20.00th=[ 117], | 30.00th=[ 137], 40.00th=[ 578], 50.00th=[10159], 60.00th=[13566], | 70.00th=[17433], 80.00th=[22152], 90.00th=[29492], 95.00th=[35914], | 99.00th=[44827], 99.50th=[50594], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 446, max= 1024, per=26.34%, avg=659.67, stdev=155.54, samples=18 iops : min= 27, max= 64, avg=40.83, stdev= 9.84, samples=18 write: IOPS=39, BW=631KiB/s (646kB/s)(6032KiB/9564msec) clat (usec): min=798, max=218885, avg=12769.52, stdev=18510.63 lat (usec): min=799, max=218886, avg=12771.47, stdev=18510.63 clat percentiles (usec): | 1.00th=[ 938], 5.00th=[ 1020], 10.00th=[ 1106], 20.00th=[ 1254], | 30.00th=[ 2008], 40.00th=[ 2933], 50.00th=[ 4686], 60.00th=[ 9896], | 70.00th=[ 15401], 80.00th=[ 20317], 90.00th=[ 33424], 95.00th=[ 43779], | 99.00th=[ 74974], 99.50th=[ 93848], 99.90th=[219153], 99.95th=[219153], | 99.99th=[219153] bw ( KiB/s): min= 382, max= 992, per=23.49%, avg=618.78, stdev=184.29, samples=18 iops : min= 23, max= 62, avg=38.28, stdev=11.80, samples=18 lat (usec) : 100=5.47%, 250=14.06%, 500=0.65%, 750=0.26%, 1000=1.69% lat (msec) : 2=13.54%, 4=8.33%, 10=10.42%, 20=22.79%, 50=21.09% lat (msec) : 100=1.56%, 250=0.13% cpu : usr=0.17%, sys=11.58%, ctx=1031, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4442: Fri May 2 00:31:34 2025 read: IOPS=40, BW=652KiB/s (668kB/s)(6032KiB/9253msec) clat (usec): min=79, max=69313, avg=12176.34, stdev=12821.40 lat (usec): min=79, max=69315, avg=12177.51, stdev=12821.46 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 96], 10.00th=[ 105], 20.00th=[ 119], | 30.00th=[ 135], 40.00th=[ 3163], 50.00th=[10945], 60.00th=[13829], | 70.00th=[17695], 80.00th=[21890], 90.00th=[27657], 95.00th=[36439], | 99.00th=[51119], 99.50th=[60556], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 377, max= 924, per=25.40%, avg=636.00, stdev=151.20, samples=18 iops : min= 23, max= 57, avg=39.11, stdev= 9.52, samples=18 write: IOPS=42, BW=676KiB/s (692kB/s)(6256KiB/9253msec) clat (usec): min=905, max=118673, avg=11647.66, stdev=14541.01 lat (usec): min=906, max=118675, avg=11650.35, stdev=14542.16 clat percentiles (usec): | 1.00th=[ 979], 5.00th=[ 1045], 10.00th=[ 1123], 20.00th=[ 1287], | 30.00th=[ 1811], 40.00th=[ 2802], 50.00th=[ 4424], 60.00th=[ 8586], | 70.00th=[ 15008], 80.00th=[ 21627], 90.00th=[ 31589], 95.00th=[ 39584], | 99.00th=[ 58983], 99.50th=[ 76022], 99.90th=[119014], 99.95th=[119014], | 99.99th=[119014] bw ( KiB/s): min= 448, max= 1248, per=25.22%, avg=664.17, stdev=186.71, samples=18 iops : min= 28, max= 78, avg=40.89, stdev=11.83, samples=18 lat (usec) : 100=3.65%, 250=14.97%, 750=0.13%, 1000=1.04% lat (msec) : 2=15.89%, 4=8.59%, 10=9.90%, 20=22.40%, 50=21.61% lat (msec) : 100=1.69%, 250=0.13% cpu : usr=0.09%, sys=11.94%, ctx=990, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2504KiB/s (2564kB/s), 609KiB/s-654KiB/s (623kB/s-670kB/s), io=23.4MiB (24.5MB), run=9253-9564msec WRITE: bw=2635KiB/s (2698kB/s), 631KiB/s-694KiB/s (646kB/s-710kB/s), io=24.6MiB (25.8MB), run=9253-9564msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4444: Fri May 2 00:32:07 2025 read: IOPS=8, BW=139KiB/s (142kB/s)(5920KiB/42557msec) clat (msec): min=4, max=113, avg=13.41, stdev=12.89 lat (msec): min=4, max=113, avg=13.41, stdev=12.89 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 15], 90.00th=[ 23], 95.00th=[ 35], | 99.00th=[ 85], 99.50th=[ 108], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 31, max= 384, per=25.65%, avg=142.33, stdev=79.45, samples=81 iops : min= 1, max= 24, avg= 8.51, stdev= 5.03, samples=81 write: IOPS=9, BW=150KiB/s (153kB/s)(6368KiB/42557msec) clat (msec): min=39, max=313, avg=93.80, stdev=38.28 lat (msec): min=39, max=313, avg=93.80, stdev=38.28 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 50], 10.00th=[ 53], 20.00th=[ 62], | 30.00th=[ 71], 40.00th=[ 80], 50.00th=[ 88], 60.00th=[ 96], | 70.00th=[ 106], 80.00th=[ 116], 90.00th=[ 146], 95.00th=[ 165], | 99.00th=[ 230], 99.50th=[ 271], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 32, max= 256, per=25.72%, avg=150.18, stdev=46.75, samples=83 iops : min= 2, max= 16, avg= 9.00, stdev= 3.02, samples=83 lat (msec) : 10=24.61%, 20=17.45%, 50=8.72%, 100=30.21%, 250=18.75% lat (msec) : 500=0.26% cpu : usr=0.02%, sys=2.28%, ctx=881, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4445: Fri May 2 00:32:07 2025 read: IOPS=8, BW=133KiB/s (136kB/s)(5744KiB/43138msec) clat (msec): min=3, max=107, avg=13.56, stdev=13.20 lat (msec): min=3, max=107, avg=13.56, stdev=13.20 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 16], 90.00th=[ 24], 95.00th=[ 38], | 99.00th=[ 81], 99.50th=[ 95], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 31, max= 352, per=26.28%, avg=145.87, stdev=86.07, samples=78 iops : min= 1, max= 22, avg= 8.76, stdev= 5.53, samples=78 write: IOPS=9, BW=152KiB/s (155kB/s)(6544KiB/43138msec) clat (msec): min=32, max=226, avg=92.90, stdev=36.36 lat (msec): min=32, max=226, avg=92.90, stdev=36.36 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 50], 10.00th=[ 55], 20.00th=[ 63], | 30.00th=[ 71], 40.00th=[ 79], 50.00th=[ 87], 60.00th=[ 94], | 70.00th=[ 106], 80.00th=[ 118], 90.00th=[ 138], 95.00th=[ 167], | 99.00th=[ 211], 99.50th=[ 220], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 32, max= 227, per=25.68%, avg=149.97, stdev=45.57, samples=86 iops : min= 2, max= 14, avg= 9.01, stdev= 2.94, samples=86 lat (msec) : 4=0.13%, 10=25.65%, 20=14.84%, 50=7.55%, 100=33.59% lat (msec) : 250=18.23% cpu : usr=0.06%, sys=2.27%, ctx=879, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4446: Fri May 2 00:32:07 2025 read: IOPS=9, BW=151KiB/s (155kB/s)(6256KiB/41350msec) clat (msec): min=4, max=123, avg=14.25, stdev=14.72 lat (msec): min=4, max=123, avg=14.26, stdev=14.72 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 16], 90.00th=[ 26], 95.00th=[ 39], | 99.00th=[ 86], 99.50th=[ 117], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 31, max= 448, per=28.37%, avg=157.47, stdev=102.55, samples=79 iops : min= 1, max= 28, avg= 9.42, stdev= 6.48, samples=79 write: IOPS=9, BW=146KiB/s (149kB/s)(6032KiB/41350msec) clat (msec): min=42, max=248, avg=94.38, stdev=37.21 lat (msec): min=42, max=248, avg=94.38, stdev=37.21 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 51], 10.00th=[ 56], 20.00th=[ 64], | 30.00th=[ 73], 40.00th=[ 81], 50.00th=[ 86], 60.00th=[ 95], | 70.00th=[ 106], 80.00th=[ 122], 90.00th=[ 142], 95.00th=[ 169], | 99.00th=[ 239], 99.50th=[ 249], 99.90th=[ 249], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min= 63, max= 224, per=24.77%, avg=144.68, stdev=42.77, samples=82 iops : min= 3, max= 14, avg= 8.61, stdev= 2.75, samples=82 lat (msec) : 10=26.69%, 20=18.23%, 50=6.25%, 100=32.03%, 250=16.80% cpu : usr=0.01%, sys=2.38%, ctx=901, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4447: Fri May 2 00:32:07 2025 read: IOPS=8, BW=142KiB/s (146kB/s)(6032KiB/42415msec) clat (usec): min=4345, max=87399, avg=14308.17, stdev=12963.26 lat (usec): min=4347, max=87401, avg=14311.52, stdev=12962.40 clat percentiles (usec): | 1.00th=[ 4883], 5.00th=[ 5932], 10.00th=[ 6325], 20.00th=[ 7635], | 30.00th=[ 8291], 40.00th=[ 9110], 50.00th=[10028], 60.00th=[11076], | 70.00th=[13173], 80.00th=[16319], 90.00th=[26084], 95.00th=[42730], | 99.00th=[82314], 99.50th=[85459], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 31, max= 415, per=26.97%, avg=149.66, stdev=78.51, samples=79 iops : min= 1, max= 25, avg= 8.86, stdev= 4.89, samples=79 write: IOPS=9, BW=147KiB/s (151kB/s)(6256KiB/42415msec) clat (msec): min=44, max=309, avg=94.53, stdev=39.67 lat (msec): min=44, max=309, avg=94.53, stdev=39.67 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 53], 10.00th=[ 57], 20.00th=[ 63], | 30.00th=[ 71], 40.00th=[ 79], 50.00th=[ 86], 60.00th=[ 94], | 70.00th=[ 104], 80.00th=[ 116], 90.00th=[ 144], 95.00th=[ 182], | 99.00th=[ 232], 99.50th=[ 247], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 31, max= 256, per=25.15%, avg=146.86, stdev=47.41, samples=84 iops : min= 1, max= 16, avg= 8.69, stdev= 3.03, samples=84 lat (msec) : 10=24.22%, 20=17.71%, 50=7.03%, 100=33.85%, 250=17.06% lat (msec) : 500=0.13% cpu : usr=0.03%, sys=2.32%, ctx=890, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=555KiB/s (569kB/s), 133KiB/s-151KiB/s (136kB/s-155kB/s), io=23.4MiB (24.5MB), run=41350-43138msec WRITE: bw=584KiB/s (598kB/s), 146KiB/s-152KiB/s (149kB/s-155kB/s), io=24.6MiB (25.8MB), run=41350-43138msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4489: Fri May 2 00:32:11 2025 read: IOPS=4, BW=4165KiB/s (4265kB/s)(7168KiB/1721msec) clat (usec): min=503, max=106973, avg=77454.19, stdev=35805.86 lat (usec): min=504, max=106976, avg=77455.89, stdev=35805.97 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 502], 10.00th=[ 502], 20.00th=[ 73925], | 30.00th=[ 82314], 40.00th=[ 82314], 50.00th=[ 83362], 60.00th=[ 96994], | 70.00th=[ 96994], 80.00th=[ 98042], 90.00th=[107480], 95.00th=[107480], | 99.00th=[107480], 99.50th=[107480], 99.90th=[107480], 99.95th=[107480], | 99.99th=[107480] bw ( KiB/s): min= 2048, max= 6095, per=46.32%, avg=4071.33, stdev=2023.50, samples=3 iops : min= 2, max= 5, avg= 3.33, stdev= 1.53, samples=3 write: IOPS=2, BW=2975KiB/s (3046kB/s)(5120KiB/1721msec) clat (msec): min=123, max=250, avg=172.95, stdev=53.06 lat (msec): min=123, max=250, avg=173.07, stdev=53.05 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 125], 10.00th=[ 125], 20.00th=[ 125], | 30.00th=[ 140], 40.00th=[ 140], 50.00th=[ 146], 60.00th=[ 146], | 70.00th=[ 205], 80.00th=[ 205], 90.00th=[ 251], 95.00th=[ 251], | 99.00th=[ 251], 99.50th=[ 251], 99.90th=[ 251], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 4063, max= 4096, per=33.15%, avg=4079.50, stdev=23.33, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 lat (usec) : 750=8.33% lat (msec) : 100=41.67%, 250=41.67%, 500=8.33% cpu : usr=0.00%, sys=13.66%, ctx=512, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4490: Fri May 2 00:32:11 2025 read: IOPS=1, BW=1144KiB/s (1172kB/s)(2048KiB/1790msec) clat (msec): min=89, max=166, avg=127.78, stdev=54.77 lat (msec): min=89, max=166, avg=127.78, stdev=54.77 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 89], 10.00th=[ 89], 20.00th=[ 89], | 30.00th=[ 89], 40.00th=[ 89], 50.00th=[ 89], 60.00th=[ 167], | 70.00th=[ 167], 80.00th=[ 167], 90.00th=[ 167], 95.00th=[ 167], | 99.00th=[ 167], 99.50th=[ 167], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 2043, max= 2043, per=23.24%, avg=2043.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5721KiB/s (5858kB/s)(10.0MiB/1790msec) clat (msec): min=51, max=618, avg=147.18, stdev=171.53 lat (msec): min=51, max=619, avg=147.28, stdev=171.52 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 61], | 30.00th=[ 71], 40.00th=[ 71], 50.00th=[ 72], 60.00th=[ 87], | 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 192], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 4087, max= 8175, per=49.83%, avg=6131.00, stdev=2044.00, samples=3 iops : min= 3, max= 7, avg= 5.00, stdev= 2.00, samples=3 lat (msec) : 100=66.67%, 250=25.00%, 750=8.33% cpu : usr=0.00%, sys=14.31%, ctx=224, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4491: Fri May 2 00:32:11 2025 read: IOPS=2, BW=2637KiB/s (2700kB/s)(6144KiB/2330msec) clat (usec): min=584, max=1527.3k, avg=312400.98, stdev=597379.43 lat (usec): min=585, max=1527.3k, avg=312402.38, stdev=597379.54 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 586], 10.00th=[ 586], | 20.00th=[ 51643], 30.00th=[ 51643], 40.00th=[ 56361], | 50.00th=[ 56361], 60.00th=[ 80217], 70.00th=[ 158335], | 80.00th=[ 158335], 90.00th=[1535116], 95.00th=[1535116], | 99.00th=[1535116], 99.50th=[1535116], 99.90th=[1535116], | 99.95th=[1535116], 99.99th=[1535116] bw ( KiB/s): min= 8192, max= 8192, per=93.21%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2637KiB/s (2700kB/s)(6144KiB/2330msec) clat (msec): min=39, max=125, avg=68.71, stdev=30.49 lat (msec): min=39, max=125, avg=68.79, stdev=30.48 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 55], 50.00th=[ 55], 60.00th=[ 71], | 70.00th=[ 73], 80.00th=[ 73], 90.00th=[ 126], 95.00th=[ 126], | 99.00th=[ 126], 99.50th=[ 126], 99.90th=[ 126], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 4096, max= 6131, per=41.56%, avg=5113.50, stdev=1438.96, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 lat (usec) : 750=8.33% lat (msec) : 50=16.67%, 100=50.00%, 250=16.67% cpu : usr=0.09%, sys=11.08%, ctx=654, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4492: Fri May 2 00:32:11 2025 read: IOPS=2, BW=2505KiB/s (2565kB/s)(5120KiB/2044msec) clat (msec): min=5, max=258, avg=103.05, stdev=101.01 lat (msec): min=5, max=258, avg=103.06, stdev=101.01 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 6], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 59], 60.00th=[ 59], | 70.00th=[ 146], 80.00th=[ 146], 90.00th=[ 259], 95.00th=[ 259], | 99.00th=[ 259], 99.50th=[ 259], 99.90th=[ 259], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 4087, max= 4087, per=46.50%, avg=4087.00, stdev= 0.00, samples=2 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3507KiB/s (3591kB/s)(7168KiB/2044msec) clat (msec): min=46, max=326, avg=172.02, stdev=92.57 lat (msec): min=46, max=327, avg=172.23, stdev=92.71 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 112], | 30.00th=[ 150], 40.00th=[ 150], 50.00th=[ 153], 60.00th=[ 161], | 70.00th=[ 161], 80.00th=[ 257], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 4039, max= 4087, per=33.08%, avg=4071.00, stdev=27.71, samples=3 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=3 lat (msec) : 10=8.33%, 50=16.67%, 100=8.33%, 250=41.67%, 500=25.00% cpu : usr=0.05%, sys=11.75%, ctx=381, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8790KiB/s (9001kB/s), 1144KiB/s-4165KiB/s (1172kB/s-4265kB/s), io=20.0MiB (20.0MB), run=1721-2330msec WRITE: bw=12.0MiB/s (12.6MB/s), 2637KiB/s-5721KiB/s (2700kB/s-5858kB/s), io=28.0MiB (29.4MB), run=1721-2330msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4493: Fri May 2 00:32:11 2025 read: IOPS=3, BW=3598KiB/s (3685kB/s)(7168KiB/1992msec) clat (msec): min=44, max=453, avg=125.46, stdev=148.96 lat (msec): min=44, max=453, avg=125.46, stdev=148.96 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 52], 40.00th=[ 52], 50.00th=[ 52], 60.00th=[ 84], | 70.00th=[ 84], 80.00th=[ 144], 90.00th=[ 456], 95.00th=[ 456], | 99.00th=[ 456], 99.50th=[ 456], 99.90th=[ 456], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 2048, max= 6059, per=49.40%, avg=4064.67, stdev=2005.59, samples=3 iops : min= 2, max= 5, avg= 3.33, stdev= 1.53, samples=3 write: IOPS=2, BW=2570KiB/s (2632kB/s)(5120KiB/1992msec) clat (msec): min=128, max=250, avg=204.62, stdev=47.87 lat (msec): min=128, max=250, avg=204.72, stdev=47.85 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 129], 10.00th=[ 129], 20.00th=[ 129], | 30.00th=[ 192], 40.00th=[ 192], 50.00th=[ 218], 60.00th=[ 218], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 251], 95.00th=[ 251], | 99.00th=[ 251], 99.50th=[ 251], 99.90th=[ 251], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 2019, max= 4087, per=23.60%, avg=2718.00, stdev=1185.68, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 lat (msec) : 50=16.67%, 100=25.00%, 250=41.67%, 500=16.67% cpu : usr=0.00%, sys=2.61%, ctx=30, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4494: Fri May 2 00:32:11 2025 read: IOPS=0, BW=887KiB/s (908kB/s)(2048KiB/2310msec) clat (usec): min=42538, max=56253, avg=49396.29, stdev=9698.08 lat (usec): min=42540, max=56255, avg=49398.07, stdev=9698.38 clat percentiles (usec): | 1.00th=[42730], 5.00th=[42730], 10.00th=[42730], 20.00th=[42730], | 30.00th=[42730], 40.00th=[42730], 50.00th=[42730], 60.00th=[56361], | 70.00th=[56361], 80.00th=[56361], 90.00th=[56361], 95.00th=[56361], | 99.00th=[56361], 99.50th=[56361], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 2043, max= 2048, per=24.86%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4433KiB/s (4539kB/s)(10.0MiB/2310msec) clat (msec): min=77, max=401, avg=192.62, stdev=106.52 lat (msec): min=77, max=401, avg=192.72, stdev=106.51 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 79], 10.00th=[ 79], 20.00th=[ 91], | 30.00th=[ 110], 40.00th=[ 133], 50.00th=[ 155], 60.00th=[ 165], | 70.00th=[ 241], 80.00th=[ 241], 90.00th=[ 330], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 2043, max= 6144, per=31.05%, avg=3576.50, stdev=1959.98, samples=4 iops : min= 1, max= 6, avg= 3.00, stdev= 2.16, samples=4 lat (msec) : 50=8.33%, 100=25.00%, 250=50.00%, 500=16.67% cpu : usr=0.13%, sys=3.72%, ctx=50, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4495: Fri May 2 00:32:11 2025 read: IOPS=2, BW=2468KiB/s (2528kB/s)(6144KiB/2489msec) clat (msec): min=20, max=748, avg=157.57, stdev=289.99 lat (msec): min=20, max=748, avg=157.57, stdev=289.99 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 42], | 70.00th=[ 78], 80.00th=[ 78], 90.00th=[ 751], 95.00th=[ 751], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 4096, max= 4096, per=49.78%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2468KiB/s (2528kB/s)(6144KiB/2489msec) clat (msec): min=123, max=452, avg=210.37, stdev=127.07 lat (msec): min=124, max=452, avg=210.46, stdev=127.06 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 125], 10.00th=[ 125], 20.00th=[ 128], | 30.00th=[ 128], 40.00th=[ 130], 50.00th=[ 130], 60.00th=[ 190], | 70.00th=[ 241], 80.00th=[ 241], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 2043, max= 2048, per=17.76%, avg=2046.33, stdev= 2.89, samples=3 iops : min= 1, max= 2, avg= 1.67, stdev= 0.58, samples=3 lat (msec) : 50=33.33%, 100=8.33%, 250=41.67%, 500=8.33%, 750=8.33% cpu : usr=0.04%, sys=2.37%, ctx=41, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4496: Fri May 2 00:32:11 2025 read: IOPS=2, BW=2079KiB/s (2129kB/s)(5120KiB/2463msec) clat (msec): min=24, max=600, avg=168.96, stdev=243.37 lat (msec): min=24, max=600, avg=168.96, stdev=243.37 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 26], | 30.00th=[ 42], 40.00th=[ 42], 50.00th=[ 69], 60.00th=[ 69], | 70.00th=[ 110], 80.00th=[ 110], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 2048, max= 6144, per=49.78%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=2, BW=2910KiB/s (2980kB/s)(7168KiB/2463msec) clat (msec): min=132, max=306, avg=214.01, stdev=66.32 lat (msec): min=132, max=306, avg=214.13, stdev=66.34 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 148], | 30.00th=[ 174], 40.00th=[ 174], 50.00th=[ 226], 60.00th=[ 230], | 70.00th=[ 230], 80.00th=[ 284], 90.00th=[ 305], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 2023, max= 4096, per=23.63%, avg=2722.33, stdev=1189.70, samples=3 iops : min= 1, max= 4, avg= 2.33, stdev= 1.53, samples=3 lat (msec) : 50=16.67%, 100=8.33%, 250=50.00%, 500=16.67%, 750=8.33% cpu : usr=0.08%, sys=2.68%, ctx=38, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8228KiB/s (8426kB/s), 887KiB/s-3598KiB/s (908kB/s-3685kB/s), io=20.0MiB (20.0MB), run=1992-2489msec WRITE: bw=11.2MiB/s (11.8MB/s), 2468KiB/s-4433KiB/s (2528kB/s-4539kB/s), io=28.0MiB (29.4MB), run=1992-2489msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4504: Fri May 2 00:32:16 2025 read: IOPS=1, BW=5394KiB/s (5524kB/s)(12.0MiB/2278msec) clat (msec): min=254, max=1660, avg=751.91, stdev=787.81 lat (msec): min=254, max=1660, avg=751.91, stdev=787.81 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 255], 10.00th=[ 255], 20.00th=[ 255], | 30.00th=[ 255], 40.00th=[ 342], 50.00th=[ 342], 60.00th=[ 342], | 70.00th=[ 1653], 80.00th=[ 1653], 90.00th=[ 1653], 95.00th=[ 1653], | 99.00th=[ 1653], 99.50th=[ 1653], 99.90th=[ 1653], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 8126, max= 8126, per=100.00%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.48%, sys=15.63%, ctx=857, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4505: Fri May 2 00:32:16 2025 read: IOPS=0, BW=1485KiB/s (1520kB/s)(4096KiB/2759msec) clat (nsec): min=1579.4M, max=1579.4M, avg=1579362609.00, stdev= 0.00 lat (nsec): min=1579.4M, max=1579.4M, avg=1579365154.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1586], 5.00th=[ 1586], 10.00th=[ 1586], 20.00th=[ 1586], | 30.00th=[ 1586], 40.00th=[ 1586], 50.00th=[ 1586], 60.00th=[ 1586], | 70.00th=[ 1586], 80.00th=[ 1586], 90.00th=[ 1586], 95.00th=[ 1586], | 99.00th=[ 1586], 99.50th=[ 1586], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2969KiB/s (3040kB/s)(8192KiB/2759msec) clat (msec): min=528, max=614, avg=571.60, stdev=60.67 lat (msec): min=529, max=614, avg=571.96, stdev=60.71 clat percentiles (msec): | 1.00th=[ 531], 5.00th=[ 531], 10.00th=[ 531], 20.00th=[ 531], | 30.00th=[ 531], 40.00th=[ 531], 50.00th=[ 531], 60.00th=[ 617], | 70.00th=[ 617], 80.00th=[ 617], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 5997, max= 5997, per=50.50%, avg=5997.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 750=66.67% cpu : usr=0.18%, sys=11.17%, ctx=436, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4506: Fri May 2 00:32:16 2025 write: IOPS=1, BW=4498KiB/s (4606kB/s)(12.0MiB/2732msec) clat (msec): min=431, max=1603, avg=895.91, stdev=622.83 lat (msec): min=431, max=1603, avg=896.31, stdev=622.81 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 430], 10.00th=[ 430], 20.00th=[ 430], | 30.00th=[ 430], 40.00th=[ 651], 50.00th=[ 651], 60.00th=[ 651], | 70.00th=[ 1603], 80.00th=[ 1603], 90.00th=[ 1603], 95.00th=[ 1603], | 99.00th=[ 1603], 99.50th=[ 1603], 99.90th=[ 1603], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min= 6041, max= 8192, per=59.92%, avg=7116.50, stdev=1520.99, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=10.58%, ctx=73, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4507: Fri May 2 00:32:16 2025 write: IOPS=1, BW=4921KiB/s (5039kB/s)(12.0MiB/2497msec) clat (msec): min=213, max=1683, avg=817.98, stdev=768.76 lat (msec): min=214, max=1683, avg=819.37, stdev=768.42 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 213], 10.00th=[ 213], 20.00th=[ 213], | 30.00th=[ 213], 40.00th=[ 558], 50.00th=[ 558], 60.00th=[ 558], | 70.00th=[ 1687], 80.00th=[ 1687], 90.00th=[ 1687], 95.00th=[ 1687], | 99.00th=[ 1687], 99.50th=[ 1687], 99.90th=[ 1687], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 8192, max= 8192, per=68.98%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=10.54%, ctx=94, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5938KiB/s (6081kB/s), 1485KiB/s-5394KiB/s (1520kB/s-5524kB/s), io=16.0MiB (16.8MB), run=2278-2759msec WRITE: bw=11.6MiB/s (12.2MB/s), 2969KiB/s-4921KiB/s (3040kB/s-5039kB/s), io=32.0MiB (33.6MB), run=2497-2759msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4508: Fri May 2 00:32:17 2025 read: IOPS=1, BW=4411KiB/s (4516kB/s)(12.0MiB/2786msec) clat (msec): min=423, max=1769, avg=903.94, stdev=751.11 lat (msec): min=423, max=1769, avg=903.94, stdev=751.11 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 426], 10.00th=[ 426], 20.00th=[ 426], | 30.00th=[ 426], 40.00th=[ 518], 50.00th=[ 518], 60.00th=[ 518], | 70.00th=[ 1770], 80.00th=[ 1770], 90.00th=[ 1770], 95.00th=[ 1770], | 99.00th=[ 1770], 99.50th=[ 1770], 99.90th=[ 1770], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 7831, max= 8192, per=100.00%, avg=8011.50, stdev=255.27, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.07%, sys=1.11%, ctx=23, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4509: Fri May 2 00:32:17 2025 read: IOPS=0, BW=1276KiB/s (1306kB/s)(4096KiB/3211msec) clat (nsec): min=1790.1M, max=1790.1M, avg=1790143369.00, stdev= 0.00 lat (nsec): min=1790.1M, max=1790.1M, avg=1790145940.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1787], 5.00th=[ 1787], 10.00th=[ 1787], 20.00th=[ 1787], | 30.00th=[ 1787], 40.00th=[ 1787], 50.00th=[ 1787], 60.00th=[ 1787], | 70.00th=[ 1787], 80.00th=[ 1787], 90.00th=[ 1787], 95.00th=[ 1787], | 99.00th=[ 1787], 99.50th=[ 1787], 99.90th=[ 1787], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2551KiB/s (2612kB/s)(8192KiB/3211msec) clat (msec): min=169, max=1195, avg=682.15, stdev=725.62 lat (msec): min=169, max=1195, avg=682.63, stdev=725.65 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 169], 10.00th=[ 169], 20.00th=[ 169], | 30.00th=[ 169], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 1200], | 70.00th=[ 1200], 80.00th=[ 1200], 90.00th=[ 1200], 95.00th=[ 1200], | 99.00th=[ 1200], 99.50th=[ 1200], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] lat (msec) : 250=33.33% cpu : usr=0.19%, sys=1.40%, ctx=31, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4510: Fri May 2 00:32:17 2025 write: IOPS=0, BW=3907KiB/s (4001kB/s)(12.0MiB/3145msec) clat (msec): min=471, max=1865, avg=1007.15, stdev=750.88 lat (msec): min=471, max=1865, avg=1007.40, stdev=750.85 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 472], 10.00th=[ 472], 20.00th=[ 472], | 30.00th=[ 472], 40.00th=[ 684], 50.00th=[ 684], 60.00th=[ 684], | 70.00th=[ 1871], 80.00th=[ 1871], 90.00th=[ 1871], 95.00th=[ 1871], | 99.00th=[ 1871], 99.50th=[ 1871], 99.90th=[ 1871], 99.95th=[ 1871], | 99.99th=[ 1871] bw ( KiB/s): min= 8175, max= 8192, per=82.57%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.06%, sys=0.83%, ctx=18, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4511: Fri May 2 00:32:17 2025 write: IOPS=0, BW=3717KiB/s (3806kB/s)(12.0MiB/3306msec) clat (msec): min=236, max=2298, avg=1044.06, stdev=1101.24 lat (msec): min=236, max=2298, avg=1044.37, stdev=1101.21 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 236], 10.00th=[ 236], 20.00th=[ 236], | 30.00th=[ 236], 40.00th=[ 600], 50.00th=[ 600], 60.00th=[ 600], | 70.00th=[ 2299], 80.00th=[ 2299], 90.00th=[ 2299], 95.00th=[ 2299], | 99.00th=[ 2299], 99.50th=[ 2299], 99.90th=[ 2299], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 8192, max= 8192, per=82.66%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.12%, sys=2.24%, ctx=67, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5102KiB/s (5225kB/s), 1276KiB/s-4411KiB/s (1306kB/s-4516kB/s), io=16.0MiB (16.8MB), run=2786-3211msec WRITE: bw=9912KiB/s (10.1MB/s), 2551KiB/s-3907KiB/s (2612kB/s-4001kB/s), io=32.0MiB (33.6MB), run=3145-3306msec