== sanity test 398b: DIO and buffer IO race ============== 20:12:07 (1746144727) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.24368 s, 22.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30500: Thu May 1 20:12:47 2025 read: IOPS=42, BW=170KiB/s (174kB/s)(5992KiB/35349msec) clat (usec): min=51, max=345517, avg=9491.45, stdev=12659.51 lat (usec): min=51, max=345519, avg=9492.69, stdev=12659.75 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 90], 10.00th=[ 98], 20.00th=[ 120], | 30.00th=[ 799], 40.00th=[ 5932], 50.00th=[ 7701], 60.00th=[ 9634], | 70.00th=[ 11863], 80.00th=[ 15795], 90.00th=[ 22414], 95.00th=[ 27132], | 99.00th=[ 39584], 99.50th=[ 44303], 99.90th=[ 64226], 99.95th=[346031], | 99.99th=[346031] bw ( KiB/s): min= 72, max= 344, per=25.47%, avg=171.91, stdev=60.12, samples=69 iops : min= 18, max= 86, avg=42.62, stdev=15.12, samples=69 write: IOPS=44, BW=178KiB/s (182kB/s)(6296KiB/35349msec) clat (usec): min=467, max=111750, avg=13243.83, stdev=14561.03 lat (usec): min=468, max=111751, avg=13246.39, stdev=14560.43 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 717], 10.00th=[ 758], 20.00th=[ 922], | 30.00th=[ 2089], 40.00th=[ 4948], 50.00th=[ 8979], 60.00th=[ 12780], | 70.00th=[ 17171], 80.00th=[ 23200], 90.00th=[ 33817], 95.00th=[ 42206], | 99.00th=[ 62653], 99.50th=[ 72877], 99.90th=[102237], 99.95th=[111674], | 99.99th=[111674] bw ( KiB/s): min= 87, max= 416, per=25.58%, avg=180.86, stdev=56.40, samples=69 iops : min= 21, max= 104, avg=44.86, stdev=14.16, samples=69 lat (usec) : 100=5.70%, 250=7.58%, 500=0.81%, 750=5.11%, 1000=6.61% lat (msec) : 2=4.52%, 4=4.95%, 10=21.52%, 20=24.25%, 50=17.48% lat (msec) : 100=1.37%, 250=0.07%, 500=0.03% cpu : usr=0.18%, sys=12.52%, ctx=4603, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30502: Thu May 1 20:12:47 2025 read: IOPS=41, BW=168KiB/s (172kB/s)(5940KiB/35441msec) clat (usec): min=68, max=123105, avg=9232.09, stdev=9695.38 lat (usec): min=69, max=123107, avg=9233.28, stdev=9695.36 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 91], 10.00th=[ 98], 20.00th=[ 116], | 30.00th=[ 685], 40.00th=[ 5735], 50.00th=[ 7504], 60.00th=[ 9503], | 70.00th=[ 11994], 80.00th=[ 15795], 90.00th=[ 21627], 95.00th=[ 27132], | 99.00th=[ 40109], 99.50th=[ 44303], 99.90th=[ 66847], 99.95th=[123208], | 99.99th=[123208] bw ( KiB/s): min= 8, max= 328, per=24.73%, avg=166.94, stdev=69.31, samples=70 iops : min= 2, max= 82, avg=41.39, stdev=17.39, samples=70 write: IOPS=44, BW=179KiB/s (183kB/s)(6348KiB/35441msec) clat (usec): min=498, max=338009, avg=13518.86, stdev=16190.12 lat (usec): min=499, max=338010, avg=13520.44, stdev=16189.88 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 734], 10.00th=[ 775], 20.00th=[ 1172], | 30.00th=[ 2442], 40.00th=[ 5473], 50.00th=[ 9634], 60.00th=[ 13173], | 70.00th=[ 17695], 80.00th=[ 22938], 90.00th=[ 32375], 95.00th=[ 41681], | 99.00th=[ 60031], 99.50th=[ 63701], 99.90th=[111674], 99.95th=[337642], | 99.99th=[337642] bw ( KiB/s): min= 87, max= 367, per=25.46%, avg=180.00, stdev=57.49, samples=69 iops : min= 21, max= 91, avg=44.65, stdev=14.42, samples=69 lat (usec) : 100=5.60%, 250=8.33%, 500=0.49%, 750=4.10%, 1000=5.34% lat (msec) : 2=5.40%, 4=5.73%, 10=21.52%, 20=24.93%, 50=17.42% lat (msec) : 100=0.98%, 250=0.13%, 500=0.03% cpu : usr=0.15%, sys=12.56%, ctx=4669, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30504: Thu May 1 20:12:47 2025 read: IOPS=42, BW=171KiB/s (175kB/s)(6028KiB/35313msec) clat (usec): min=57, max=97076, avg=9207.86, stdev=9545.26 lat (usec): min=57, max=97076, avg=9208.92, stdev=9545.22 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 88], 10.00th=[ 96], 20.00th=[ 116], | 30.00th=[ 231], 40.00th=[ 5604], 50.00th=[ 7701], 60.00th=[ 9634], | 70.00th=[12125], 80.00th=[16057], 90.00th=[22414], 95.00th=[27657], | 99.00th=[36439], 99.50th=[43254], 99.90th=[64750], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 55, max= 336, per=26.06%, avg=175.90, stdev=59.83, samples=68 iops : min= 13, max= 84, avg=43.79, stdev=14.98, samples=68 write: IOPS=44, BW=177KiB/s (182kB/s)(6260KiB/35313msec) clat (usec): min=482, max=511180, avg=13531.94, stdev=19498.88 lat (usec): min=482, max=511182, avg=13533.13, stdev=19498.90 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 717], 10.00th=[ 766], 20.00th=[ 955], | 30.00th=[ 2180], 40.00th=[ 5211], 50.00th=[ 9372], 60.00th=[ 13304], | 70.00th=[ 17171], 80.00th=[ 22414], 90.00th=[ 32375], 95.00th=[ 42206], | 99.00th=[ 63177], 99.50th=[ 69731], 99.90th=[185598], 99.95th=[509608], | 99.99th=[509608] bw ( KiB/s): min= 15, max= 380, per=25.40%, avg=179.61, stdev=65.23, samples=69 iops : min= 3, max= 95, avg=44.71, stdev=16.39, samples=69 lat (usec) : 100=6.12%, 250=8.79%, 500=0.68%, 750=4.65%, 1000=6.58% lat (msec) : 2=4.39%, 4=4.95%, 10=20.31%, 20=24.84%, 50=17.42% lat (msec) : 100=1.14%, 250=0.10%, 750=0.03% cpu : usr=0.13%, sys=12.37%, ctx=4528, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30505: Thu May 1 20:12:47 2025 read: IOPS=42, BW=170KiB/s (174kB/s)(6052KiB/35546msec) clat (usec): min=48, max=361588, avg=10081.51, stdev=13280.59 lat (usec): min=48, max=361589, avg=10083.29, stdev=13281.88 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 91], 10.00th=[ 100], 20.00th=[ 126], | 30.00th=[ 4146], 40.00th=[ 6194], 50.00th=[ 8029], 60.00th=[ 9765], | 70.00th=[ 12387], 80.00th=[ 16450], 90.00th=[ 23725], 95.00th=[ 28705], | 99.00th=[ 41681], 99.50th=[ 50070], 99.90th=[ 67634], 99.95th=[362808], | 99.99th=[362808] bw ( KiB/s): min= 71, max= 392, per=25.46%, avg=171.87, stdev=64.20, samples=70 iops : min= 17, max= 98, avg=42.67, stdev=16.09, samples=70 write: IOPS=43, BW=175KiB/s (180kB/s)(6236KiB/35546msec) clat (usec): min=420, max=129776, avg=12844.80, stdev=14026.08 lat (usec): min=421, max=129778, avg=12846.06, stdev=14026.04 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 725], 10.00th=[ 775], 20.00th=[ 1037], | 30.00th=[ 2114], 40.00th=[ 4621], 50.00th=[ 9110], 60.00th=[ 12780], | 70.00th=[ 16319], 80.00th=[ 22676], 90.00th=[ 31327], 95.00th=[ 41157], | 99.00th=[ 60031], 99.50th=[ 65274], 99.90th=[ 98042], 99.95th=[129500], | 99.99th=[129500] bw ( KiB/s): min= 32, max= 396, per=24.63%, avg=174.17, stdev=64.64, samples=71 iops : min= 8, max= 99, avg=43.25, stdev=16.20, samples=71 lat (usec) : 50=0.03%, 100=4.88%, 250=7.91%, 500=0.52%, 750=3.48% lat (usec) : 1000=6.61% lat (msec) : 2=5.24%, 4=5.18%, 10=22.56%, 20=24.22%, 50=17.84% lat (msec) : 100=1.46%, 250=0.03%, 500=0.03% cpu : usr=0.09%, sys=12.71%, ctx=4735, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=676KiB/s (692kB/s), 168KiB/s-171KiB/s (172kB/s-175kB/s), io=23.4MiB (24.6MB), run=35313-35546msec WRITE: bw=707KiB/s (724kB/s), 175KiB/s-179KiB/s (180kB/s-183kB/s), io=24.6MiB (25.7MB), run=35313-35546msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30498: Thu May 1 20:14:18 2025 read: IOPS=11, BW=47.2KiB/s (48.3kB/s)(5992KiB/126908msec) clat (msec): min=2, max=349, avg=10.75, stdev=12.66 lat (msec): min=2, max=349, avg=10.75, stdev=12.66 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 21], 95.00th=[ 30], | 99.00th=[ 46], 99.50th=[ 57], 99.90th=[ 99], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 7, max= 143, per=25.40%, avg=47.75, stdev=27.61, samples=248 iops : min= 1, max= 35, avg=11.63, stdev= 6.89, samples=248 write: IOPS=12, BW=49.6KiB/s (50.8kB/s)(6296KiB/126908msec) clat (msec): min=19, max=231, avg=70.21, stdev=28.16 lat (msec): min=19, max=231, avg=70.22, stdev=28.16 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 36], 10.00th=[ 41], 20.00th=[ 47], | 30.00th=[ 54], 40.00th=[ 59], 50.00th=[ 66], 60.00th=[ 71], | 70.00th=[ 79], 80.00th=[ 90], 90.00th=[ 107], 95.00th=[ 128], | 99.00th=[ 163], 99.50th=[ 176], 99.90th=[ 213], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 15, max= 103, per=25.04%, avg=49.34, stdev=15.63, samples=252 iops : min= 3, max= 25, avg=12.03, stdev= 3.95, samples=252 lat (msec) : 4=2.80%, 10=31.18%, 20=9.70%, 50=17.68%, 100=31.74% lat (msec) : 250=6.87%, 500=0.03% cpu : usr=0.07%, sys=2.64%, ctx=3469, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30499: Thu May 1 20:14:18 2025 read: IOPS=11, BW=46.9KiB/s (48.0kB/s)(5940KiB/126626msec) clat (msec): min=2, max=352, avg=10.82, stdev=13.11 lat (msec): min=2, max=352, avg=10.83, stdev=13.11 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 22], 95.00th=[ 29], | 99.00th=[ 46], 99.50th=[ 59], 99.90th=[ 138], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 7, max= 160, per=25.41%, avg=47.77, stdev=28.64, samples=247 iops : min= 1, max= 40, avg=11.72, stdev= 7.20, samples=247 write: IOPS=12, BW=50.1KiB/s (51.3kB/s)(6348KiB/126626msec) clat (msec): min=21, max=186, avg=69.46, stdev=27.24 lat (msec): min=21, max=186, avg=69.46, stdev=27.24 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 37], 10.00th=[ 41], 20.00th=[ 47], | 30.00th=[ 54], 40.00th=[ 59], 50.00th=[ 64], 60.00th=[ 71], | 70.00th=[ 79], 80.00th=[ 89], 90.00th=[ 106], 95.00th=[ 124], | 99.00th=[ 161], 99.50th=[ 178], 99.90th=[ 186], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 15, max= 104, per=25.39%, avg=50.02, stdev=15.50, samples=252 iops : min= 3, max= 26, avg=12.28, stdev= 3.94, samples=252 lat (msec) : 4=2.86%, 10=30.27%, 20=9.90%, 50=18.03%, 100=32.52% lat (msec) : 250=6.38%, 500=0.03% cpu : usr=0.04%, sys=2.65%, ctx=3436, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30501: Thu May 1 20:14:18 2025 read: IOPS=11, BW=47.4KiB/s (48.6kB/s)(6028KiB/127090msec) clat (msec): min=2, max=160, avg=11.14, stdev=11.29 lat (msec): min=2, max=160, avg=11.14, stdev=11.29 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 23], 95.00th=[ 32], | 99.00th=[ 51], 99.50th=[ 80], 99.90th=[ 142], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 7, max= 152, per=25.43%, avg=47.80, stdev=28.65, samples=250 iops : min= 1, max= 38, avg=11.68, stdev= 7.20, samples=250 write: IOPS=12, BW=49.3KiB/s (50.4kB/s)(6260KiB/127090msec) clat (msec): min=22, max=499, avg=70.29, stdev=29.42 lat (msec): min=22, max=499, avg=70.30, stdev=29.41 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 36], 10.00th=[ 41], 20.00th=[ 48], | 30.00th=[ 55], 40.00th=[ 60], 50.00th=[ 66], 60.00th=[ 71], | 70.00th=[ 79], 80.00th=[ 89], 90.00th=[ 104], 95.00th=[ 124], | 99.00th=[ 167], 99.50th=[ 180], 99.90th=[ 226], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 8, max= 96, per=24.89%, avg=49.03, stdev=15.83, samples=253 iops : min= 2, max= 24, avg=11.99, stdev= 4.02, samples=253 lat (msec) : 4=3.22%, 10=29.79%, 20=10.48%, 50=16.70%, 100=33.76% lat (msec) : 250=6.02%, 500=0.03% cpu : usr=0.07%, sys=2.65%, ctx=3436, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30503: Thu May 1 20:14:18 2025 read: IOPS=11, BW=47.0KiB/s (49.1kB/s)(6052KiB/126126msec) clat (msec): min=2, max=161, avg=10.42, stdev= 9.95 lat (msec): min=2, max=161, avg=10.42, stdev= 9.95 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 14], 90.00th=[ 21], 95.00th=[ 29], | 99.00th=[ 44], 99.50th=[ 52], 99.90th=[ 142], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 7, max= 176, per=25.96%, avg=48.81, stdev=28.65, samples=246 iops : min= 1, max= 44, avg=11.93, stdev= 7.19, samples=246 write: IOPS=12, BW=49.4KiB/s (50.6kB/s)(6236KiB/126126msec) clat (msec): min=17, max=537, avg=70.59, stdev=30.53 lat (msec): min=17, max=537, avg=70.59, stdev=30.53 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 36], 10.00th=[ 41], 20.00th=[ 48], | 30.00th=[ 53], 40.00th=[ 59], 50.00th=[ 66], 60.00th=[ 71], | 70.00th=[ 80], 80.00th=[ 90], 90.00th=[ 110], 95.00th=[ 126], | 99.00th=[ 165], 99.50th=[ 178], 99.90th=[ 211], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 16, max= 112, per=25.01%, avg=49.27, stdev=16.49, samples=251 iops : min= 4, max= 28, avg=12.05, stdev= 4.17, samples=251 lat (msec) : 4=2.99%, 10=31.80%, 20=9.24%, 50=17.61%, 100=31.64% lat (msec) : 250=6.67%, 750=0.03% cpu : usr=0.07%, sys=2.68%, ctx=3451, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=189KiB/s (193kB/s), 46.9KiB/s-47.0KiB/s (48.0kB/s-49.1kB/s), io=23.4MiB (24.6MB), run=126126-127090msec WRITE: bw=198KiB/s (203kB/s), 49.3KiB/s-50.1KiB/s (50.4kB/s-51.3kB/s), io=24.6MiB (25.7MB), run=126126-127090msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30618: Thu May 1 20:14:26 2025 read: IOPS=58, BW=930KiB/s (952kB/s)(5920KiB/6367msec) clat (usec): min=61, max=88469, avg=7845.40, stdev=7717.47 lat (usec): min=62, max=88471, avg=7846.32, stdev=7717.51 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 94], 10.00th=[ 102], 20.00th=[ 122], | 30.00th=[ 3195], 40.00th=[ 5669], 50.00th=[ 7177], 60.00th=[ 8717], | 70.00th=[10814], 80.00th=[13042], 90.00th=[16319], 95.00th=[20055], | 99.00th=[23462], 99.50th=[35390], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 447, max= 1408, per=26.08%, avg=927.67, stdev=264.00, samples=12 iops : min= 27, max= 88, avg=57.83, stdev=16.55, samples=12 write: IOPS=62, BW=1000KiB/s (1024kB/s)(6368KiB/6367msec) clat (usec): min=742, max=81497, avg=8355.38, stdev=9824.24 lat (usec): min=743, max=81499, avg=8356.86, stdev=9824.30 clat percentiles (usec): | 1.00th=[ 955], 5.00th=[ 1057], 10.00th=[ 1123], 20.00th=[ 1287], | 30.00th=[ 1844], 40.00th=[ 2507], 50.00th=[ 4293], 60.00th=[ 7177], | 70.00th=[10945], 80.00th=[14746], 90.00th=[20055], 95.00th=[25297], | 99.00th=[42206], 99.50th=[70779], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 542, max= 1472, per=26.56%, avg=994.25, stdev=254.43, samples=12 iops : min= 33, max= 92, avg=62.00, stdev=15.98, samples=12 lat (usec) : 100=3.78%, 250=9.11%, 500=0.65%, 750=0.39%, 1000=1.30% lat (msec) : 2=16.41%, 4=8.85%, 10=26.04%, 20=25.91%, 50=7.03% lat (msec) : 100=0.52% cpu : usr=0.25%, sys=16.92%, ctx=1166, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30619: Thu May 1 20:14:26 2025 read: IOPS=53, BW=853KiB/s (874kB/s)(5744KiB/6732msec) clat (usec): min=52, max=61226, avg=6949.88, stdev=7685.99 lat (usec): min=53, max=61227, avg=6950.81, stdev=7685.94 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 91], 10.00th=[ 99], 20.00th=[ 113], | 30.00th=[ 131], 40.00th=[ 2835], 50.00th=[ 6390], 60.00th=[ 7701], | 70.00th=[ 9896], 80.00th=[12649], 90.00th=[16057], 95.00th=[19006], | 99.00th=[30540], 99.50th=[50594], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 32, max= 1341, per=23.48%, avg=835.08, stdev=411.83, samples=13 iops : min= 2, max= 83, avg=52.00, stdev=25.56, samples=13 write: IOPS=60, BW=972KiB/s (995kB/s)(6544KiB/6732msec) clat (usec): min=656, max=168814, avg=9998.76, stdev=17696.76 lat (usec): min=657, max=168816, avg=10000.27, stdev=17696.74 clat percentiles (usec): | 1.00th=[ 857], 5.00th=[ 988], 10.00th=[ 1045], 20.00th=[ 1188], | 30.00th=[ 1516], 40.00th=[ 2089], 50.00th=[ 3228], 60.00th=[ 6980], | 70.00th=[ 10421], 80.00th=[ 14091], 90.00th=[ 22676], 95.00th=[ 31589], | 99.00th=[105382], 99.50th=[114820], 99.90th=[168821], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 64, max= 1596, per=25.00%, avg=935.62, stdev=462.27, samples=13 iops : min= 4, max= 99, avg=58.31, stdev=28.73, samples=13 lat (usec) : 100=5.08%, 250=11.59%, 500=0.91%, 750=1.04%, 1000=2.86% lat (msec) : 2=17.97%, 4=7.68%, 10=22.79%, 20=22.01%, 50=6.12% lat (msec) : 100=1.30%, 250=0.65% cpu : usr=0.21%, sys=15.58%, ctx=1109, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30620: Thu May 1 20:14:26 2025 read: IOPS=68, BW=1089KiB/s (1115kB/s)(6256KiB/5746msec) clat (usec): min=69, max=41501, avg=7619.58, stdev=6992.38 lat (usec): min=69, max=41501, avg=7620.61, stdev=6992.49 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 96], 10.00th=[ 103], 20.00th=[ 124], | 30.00th=[ 1254], 40.00th=[ 5669], 50.00th=[ 7242], 60.00th=[ 8455], | 70.00th=[10290], 80.00th=[13173], 90.00th=[16450], 95.00th=[20055], | 99.00th=[33817], 99.50th=[41157], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 640, max= 1696, per=31.22%, avg=1110.36, stdev=318.62, samples=11 iops : min= 40, max= 106, avg=69.09, stdev=19.92, samples=11 write: IOPS=65, BW=1050KiB/s (1075kB/s)(6032KiB/5746msec) clat (usec): min=726, max=81421, avg=7255.49, stdev=9476.48 lat (usec): min=727, max=81422, avg=7256.86, stdev=9476.52 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 1012], 10.00th=[ 1074], 20.00th=[ 1205], | 30.00th=[ 1549], 40.00th=[ 2114], 50.00th=[ 2933], 60.00th=[ 5014], | 70.00th=[ 7832], 80.00th=[12125], 90.00th=[19530], 95.00th=[23987], | 99.00th=[41157], 99.50th=[62129], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 830, max= 1472, per=28.11%, avg=1052.27, stdev=222.41, samples=11 iops : min= 51, max= 92, avg=65.45, stdev=13.97, samples=11 lat (usec) : 100=4.17%, 250=10.03%, 500=0.52%, 750=0.39%, 1000=1.95% lat (msec) : 2=17.45%, 4=9.51%, 10=27.73%, 20=21.35%, 50=6.51% lat (msec) : 100=0.39% cpu : usr=0.17%, sys=17.75%, ctx=1172, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30622: Thu May 1 20:14:26 2025 read: IOPS=60, BW=975KiB/s (999kB/s)(6032KiB/6184msec) clat (usec): min=86, max=81439, avg=7253.03, stdev=7570.62 lat (usec): min=87, max=81441, avg=7253.93, stdev=7570.67 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 96], 10.00th=[ 103], 20.00th=[ 122], | 30.00th=[ 184], 40.00th=[ 5473], 50.00th=[ 6718], 60.00th=[ 7898], | 70.00th=[ 9503], 80.00th=[12518], 90.00th=[16188], 95.00th=[19006], | 99.00th=[28181], 99.50th=[33817], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 798, max= 1536, per=29.36%, avg=1044.18, stdev=227.97, samples=11 iops : min= 49, max= 96, avg=65.18, stdev=14.34, samples=11 write: IOPS=63, BW=1012KiB/s (1036kB/s)(6256KiB/6184msec) clat (usec): min=741, max=125481, avg=8281.04, stdev=12416.57 lat (usec): min=742, max=125482, avg=8282.42, stdev=12416.59 clat percentiles (usec): | 1.00th=[ 906], 5.00th=[ 996], 10.00th=[ 1074], 20.00th=[ 1205], | 30.00th=[ 1549], 40.00th=[ 2212], 50.00th=[ 3458], 60.00th=[ 6194], | 70.00th=[ 9110], 80.00th=[ 13435], 90.00th=[ 20317], 95.00th=[ 26608], | 99.00th=[ 77071], 99.50th=[ 96994], 99.90th=[125305], 99.95th=[125305], | 99.99th=[125305] bw ( KiB/s): min= 95, max= 1568, per=26.71%, avg=999.67, stdev=387.82, samples=12 iops : min= 5, max= 98, avg=62.33, stdev=24.43, samples=12 lat (usec) : 100=3.12%, 250=12.24%, 500=0.65%, 750=0.39%, 1000=2.60% lat (msec) : 2=17.19%, 4=7.55%, 10=27.86%, 20=21.22%, 50=6.51% lat (msec) : 100=0.52%, 250=0.13% cpu : usr=0.05%, sys=16.29%, ctx=1080, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3558KiB/s (3643kB/s), 853KiB/s-1089KiB/s (874kB/s-1115kB/s), io=23.4MiB (24.5MB), run=5746-6732msec WRITE: bw=3743KiB/s (3833kB/s), 972KiB/s-1050KiB/s (995kB/s-1075kB/s), io=24.6MiB (25.8MB), run=5746-6732msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30621: Thu May 1 20:14:45 2025 read: IOPS=14, BW=238KiB/s (243kB/s)(5920KiB/24924msec) clat (msec): min=2, max=109, avg= 8.98, stdev= 9.36 lat (msec): min=2, max=109, avg= 8.99, stdev= 9.36 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 12], 90.00th=[ 18], 95.00th=[ 23], | 99.00th=[ 47], 99.50th=[ 63], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 64, max= 832, per=24.96%, avg=235.63, stdev=130.74, samples=49 iops : min= 4, max= 52, avg=14.49, stdev= 8.16, samples=49 write: IOPS=15, BW=255KiB/s (262kB/s)(6368KiB/24924msec) clat (msec): min=17, max=150, avg=53.84, stdev=19.25 lat (msec): min=17, max=150, avg=53.85, stdev=19.25 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 30], 10.00th=[ 33], 20.00th=[ 38], | 30.00th=[ 42], 40.00th=[ 47], 50.00th=[ 52], 60.00th=[ 57], | 70.00th=[ 62], 80.00th=[ 67], 90.00th=[ 81], 95.00th=[ 92], | 99.00th=[ 115], 99.50th=[ 131], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 95, max= 448, per=25.65%, avg=254.73, stdev=76.72, samples=49 iops : min= 5, max= 28, avg=15.67, stdev= 4.78, samples=49 lat (msec) : 4=7.68%, 10=28.91%, 20=8.59%, 50=27.60%, 100=25.78% lat (msec) : 250=1.43% cpu : usr=0.04%, sys=3.58%, ctx=948, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30623: Thu May 1 20:14:45 2025 read: IOPS=14, BW=227KiB/s (232kB/s)(5744KiB/25358msec) clat (usec): min=2275, max=52609, avg=8536.05, stdev=6555.18 lat (usec): min=2276, max=52610, avg=8537.52, stdev=6555.12 clat percentiles (usec): | 1.00th=[ 3195], 5.00th=[ 3687], 10.00th=[ 3851], 20.00th=[ 4490], | 30.00th=[ 5145], 40.00th=[ 5604], 50.00th=[ 6194], 60.00th=[ 7111], | 70.00th=[ 8455], 80.00th=[11338], 90.00th=[16581], 95.00th=[20579], | 99.00th=[36439], 99.50th=[41681], 99.90th=[52691], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 32, max= 541, per=23.70%, avg=223.70, stdev=128.79, samples=50 iops : min= 2, max= 33, avg=13.66, stdev= 8.02, samples=50 write: IOPS=16, BW=258KiB/s (264kB/s)(6544KiB/25358msec) clat (msec): min=21, max=155, avg=54.20, stdev=21.16 lat (msec): min=21, max=155, avg=54.20, stdev=21.16 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 30], 10.00th=[ 33], 20.00th=[ 37], | 30.00th=[ 41], 40.00th=[ 46], 50.00th=[ 51], 60.00th=[ 56], | 70.00th=[ 60], 80.00th=[ 69], 90.00th=[ 81], 95.00th=[ 96], | 99.00th=[ 130], 99.50th=[ 140], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 64, max= 415, per=25.59%, avg=254.06, stdev=77.39, samples=50 iops : min= 4, max= 25, avg=15.56, stdev= 4.88, samples=50 lat (msec) : 4=5.60%, 10=30.47%, 20=8.20%, 50=28.65%, 100=24.61% lat (msec) : 250=2.47% cpu : usr=0.11%, sys=3.43%, ctx=932, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30624: Thu May 1 20:14:45 2025 read: IOPS=16, BW=256KiB/s (262kB/s)(6256KiB/24417msec) clat (usec): min=2927, max=53572, avg=8523.14, stdev=6834.61 lat (usec): min=2928, max=53573, avg=8525.95, stdev=6834.15 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 3523], 10.00th=[ 3916], 20.00th=[ 4621], | 30.00th=[ 5014], 40.00th=[ 5538], 50.00th=[ 6259], 60.00th=[ 6915], | 70.00th=[ 7898], 80.00th=[11600], 90.00th=[16319], 95.00th=[19268], | 99.00th=[42730], 99.50th=[48497], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 64, max= 768, per=28.04%, avg=264.66, stdev=149.38, samples=47 iops : min= 4, max= 48, avg=16.38, stdev= 9.41, samples=47 write: IOPS=15, BW=247KiB/s (253kB/s)(6032KiB/24417msec) clat (msec): min=18, max=131, avg=55.25, stdev=20.29 lat (msec): min=18, max=131, avg=55.25, stdev=20.29 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 30], 10.00th=[ 33], 20.00th=[ 39], | 30.00th=[ 43], 40.00th=[ 47], 50.00th=[ 53], 60.00th=[ 58], | 70.00th=[ 63], 80.00th=[ 69], 90.00th=[ 83], 95.00th=[ 96], | 99.00th=[ 122], 99.50th=[ 127], 99.90th=[ 132], 99.95th=[ 132], | 99.99th=[ 132] bw ( KiB/s): min= 64, max= 384, per=24.69%, avg=245.13, stdev=70.61, samples=48 iops : min= 4, max= 24, avg=15.17, stdev= 4.47, samples=48 lat (msec) : 4=5.60%, 10=33.98%, 20=9.24%, 50=24.48%, 100=24.74% lat (msec) : 250=1.95% cpu : usr=0.05%, sys=3.58%, ctx=941, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30625: Thu May 1 20:14:45 2025 read: IOPS=15, BW=242KiB/s (248kB/s)(6032KiB/24945msec) clat (msec): min=2, max=108, avg= 9.36, stdev= 9.41 lat (msec): min=2, max=108, avg= 9.36, stdev= 9.41 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 9], 80.00th=[ 12], 90.00th=[ 18], 95.00th=[ 25], | 99.00th=[ 51], 99.50th=[ 56], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 32, max= 576, per=26.02%, avg=245.63, stdev=135.68, samples=48 iops : min= 2, max= 36, avg=15.02, stdev= 8.55, samples=48 write: IOPS=15, BW=251KiB/s (257kB/s)(6256KiB/24945msec) clat (msec): min=20, max=180, avg=54.65, stdev=20.64 lat (msec): min=20, max=180, avg=54.65, stdev=20.64 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 30], 10.00th=[ 32], 20.00th=[ 37], | 30.00th=[ 43], 40.00th=[ 50], 50.00th=[ 54], 60.00th=[ 58], | 70.00th=[ 62], 80.00th=[ 68], 90.00th=[ 80], 95.00th=[ 92], | 99.00th=[ 129], 99.50th=[ 153], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 96, max= 448, per=25.15%, avg=249.78, stdev=73.52, samples=49 iops : min= 6, max= 28, avg=15.29, stdev= 4.62, samples=49 lat (msec) : 4=5.47%, 10=32.03%, 20=7.42%, 50=25.52%, 100=27.86% lat (msec) : 250=1.69% cpu : usr=0.00%, sys=3.63%, ctx=952, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=945KiB/s (967kB/s), 227KiB/s-256KiB/s (232kB/s-262kB/s), io=23.4MiB (24.5MB), run=24417-25358msec WRITE: bw=994KiB/s (1018kB/s), 247KiB/s-258KiB/s (253kB/s-264kB/s), io=24.6MiB (25.8MB), run=24417-25358msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30666: Thu May 1 20:14:47 2025 read: IOPS=7, BW=7601KiB/s (7784kB/s)(7168KiB/943msec) clat (usec): min=485, max=109964, avg=60013.49, stdev=44874.47 lat (usec): min=487, max=109966, avg=60014.97, stdev=44874.66 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 486], 10.00th=[ 486], 20.00th=[ 594], | 30.00th=[ 56886], 40.00th=[ 56886], 50.00th=[ 65274], 60.00th=[ 82314], | 70.00th=[ 82314], 80.00th=[104334], 90.00th=[109577], 95.00th=[109577], | 99.00th=[109577], 99.50th=[109577], 99.90th=[109577], 99.95th=[109577], | 99.99th=[109577] bw ( KiB/s): min=10240, max=10240, per=57.70%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5429KiB/s (5560kB/s)(5120KiB/943msec) clat (msec): min=46, max=152, avg=91.03, stdev=49.01 lat (msec): min=46, max=152, avg=91.10, stdev=49.01 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 73], 60.00th=[ 73], | 70.00th=[ 134], 80.00th=[ 134], 90.00th=[ 153], 95.00th=[ 153], | 99.00th=[ 153], 99.50th=[ 153], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 2048, max= 2048, per=8.24%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=16.67%, 100=33.33%, 250=33.33% cpu : usr=0.11%, sys=19.53%, ctx=607, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30667: Thu May 1 20:14:47 2025 read: IOPS=1, BW=1775KiB/s (1817kB/s)(2048KiB/1154msec) clat (msec): min=83, max=111, avg=97.39, stdev=20.15 lat (msec): min=83, max=111, avg=97.39, stdev=20.15 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 84], 10.00th=[ 84], 20.00th=[ 84], | 30.00th=[ 84], 40.00th=[ 84], 50.00th=[ 84], 60.00th=[ 112], | 70.00th=[ 112], 80.00th=[ 112], 90.00th=[ 112], 95.00th=[ 112], | 99.00th=[ 112], 99.50th=[ 112], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 1750, max= 2039, per=10.68%, avg=1894.50, stdev=204.35, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=8, BW=8873KiB/s (9086kB/s)(10.0MiB/1154msec) clat (msec): min=44, max=169, avg=90.30, stdev=38.70 lat (msec): min=44, max=169, avg=90.38, stdev=38.70 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 51], | 30.00th=[ 70], 40.00th=[ 73], 50.00th=[ 78], 60.00th=[ 85], | 70.00th=[ 120], 80.00th=[ 120], 90.00th=[ 133], 95.00th=[ 169], | 99.00th=[ 169], 99.50th=[ 169], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 6119, max=10502, per=33.45%, avg=8310.50, stdev=3099.25, samples=2 iops : min= 5, max= 10, avg= 7.50, stdev= 3.54, samples=2 lat (msec) : 50=8.33%, 100=58.33%, 250=33.33% cpu : usr=0.00%, sys=22.98%, ctx=194, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30668: Thu May 1 20:14:47 2025 read: IOPS=8, BW=8486KiB/s (8690kB/s)(6144KiB/724msec) clat (usec): min=646, max=96211, avg=51333.21, stdev=32347.32 lat (usec): min=648, max=96213, avg=51334.53, stdev=32347.25 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 644], 10.00th=[ 644], 20.00th=[42730], | 30.00th=[42730], 40.00th=[42730], 50.00th=[42730], 60.00th=[51643], | 70.00th=[73925], 80.00th=[73925], 90.00th=[95945], 95.00th=[95945], | 99.00th=[95945], 99.50th=[95945], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 8175, max= 8175, per=46.07%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8486KiB/s (8690kB/s)(6144KiB/724msec) clat (msec): min=34, max=111, avg=63.05, stdev=28.55 lat (msec): min=34, max=111, avg=63.12, stdev=28.53 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 44], | 30.00th=[ 44], 40.00th=[ 52], 50.00th=[ 52], 60.00th=[ 57], | 70.00th=[ 82], 80.00th=[ 82], 90.00th=[ 112], 95.00th=[ 112], | 99.00th=[ 112], 99.50th=[ 112], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 8175, max= 8175, per=32.90%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=33.33%, 100=50.00%, 250=8.33% cpu : usr=0.14%, sys=27.80%, ctx=714, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30669: Thu May 1 20:14:47 2025 read: IOPS=5, BW=5912KiB/s (6054kB/s)(5120KiB/866msec) clat (usec): min=601, max=88199, avg=52156.64, stdev=32553.01 lat (usec): min=603, max=88200, avg=52158.14, stdev=32552.74 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 603], 10.00th=[ 603], 20.00th=[ 603], | 30.00th=[47449], 40.00th=[47449], 50.00th=[56361], 60.00th=[56361], | 70.00th=[67634], 80.00th=[67634], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 6144, max= 6144, per=34.62%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8277KiB/s (8476kB/s)(7168KiB/866msec) clat (msec): min=29, max=182, avg=84.23, stdev=54.31 lat (msec): min=29, max=182, avg=84.31, stdev=54.30 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 43], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 65], 60.00th=[ 96], | 70.00th=[ 96], 80.00th=[ 125], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 8192, max= 8192, per=32.97%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=25.00%, 100=50.00%, 250=16.67% cpu : usr=0.00%, sys=22.89%, ctx=854, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.3MiB/s (18.2MB/s), 1775KiB/s-8486KiB/s (1817kB/s-8690kB/s), io=20.0MiB (20.0MB), run=724-1154msec WRITE: bw=24.3MiB/s (25.4MB/s), 5429KiB/s-8873KiB/s (5560kB/s-9086kB/s), io=28.0MiB (29.4MB), run=724-1154msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30671: Thu May 1 20:14:48 2025 read: IOPS=5, BW=5493KiB/s (5625kB/s)(7168KiB/1305msec) clat (msec): min=12, max=205, avg=79.60, stdev=74.06 lat (msec): min=12, max=205, avg=79.60, stdev=74.06 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 22], | 30.00th=[ 27], 40.00th=[ 27], 50.00th=[ 59], 60.00th=[ 75], | 70.00th=[ 75], 80.00th=[ 157], 90.00th=[ 205], 95.00th=[ 205], | 99.00th=[ 205], 99.50th=[ 205], 99.90th=[ 205], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 4096, max= 6107, per=32.51%, avg=5101.50, stdev=1421.99, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3923KiB/s (4018kB/s)(5120KiB/1305msec) clat (msec): min=43, max=308, avg=127.37, stdev=111.61 lat (msec): min=43, max=308, avg=127.46, stdev=111.62 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 44], | 30.00th=[ 52], 40.00th=[ 52], 50.00th=[ 71], 60.00th=[ 71], | 70.00th=[ 163], 80.00th=[ 163], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 2048, max= 2048, per=9.32%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=25.00%, 100=33.33%, 250=25.00%, 500=8.33% cpu : usr=0.31%, sys=3.68%, ctx=38, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30672: Thu May 1 20:14:48 2025 read: IOPS=1, BW=1619KiB/s (1658kB/s)(2048KiB/1265msec) clat (usec): min=23384, max=45477, avg=34430.58, stdev=15622.02 lat (usec): min=23385, max=45478, avg=34431.73, stdev=15622.26 clat percentiles (usec): | 1.00th=[23462], 5.00th=[23462], 10.00th=[23462], 20.00th=[23462], | 30.00th=[23462], 40.00th=[23462], 50.00th=[23462], 60.00th=[45351], | 70.00th=[45351], 80.00th=[45351], 90.00th=[45351], 95.00th=[45351], | 99.00th=[45351], 99.50th=[45351], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 2048, max= 2048, per=13.05%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=7, BW=8095KiB/s (8289kB/s)(10.0MiB/1265msec) clat (msec): min=47, max=149, avg=95.71, stdev=37.25 lat (msec): min=48, max=149, avg=95.79, stdev=37.26 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 63], | 30.00th=[ 63], 40.00th=[ 65], 50.00th=[ 79], 60.00th=[ 108], | 70.00th=[ 125], 80.00th=[ 125], 90.00th=[ 148], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 4096, max= 8192, per=27.97%, avg=6144.00, stdev=2896.31, samples=2 iops : min= 4, max= 8, avg= 6.00, stdev= 2.83, samples=2 lat (msec) : 50=25.00%, 100=33.33%, 250=41.67% cpu : usr=0.16%, sys=6.80%, ctx=57, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30673: Thu May 1 20:14:48 2025 read: IOPS=4, BW=5086KiB/s (5208kB/s)(6144KiB/1208msec) clat (msec): min=27, max=112, avg=63.11, stdev=30.81 lat (msec): min=27, max=112, avg=63.12, stdev=30.81 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 40], | 30.00th=[ 40], 40.00th=[ 58], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 83], 80.00th=[ 83], 90.00th=[ 113], 95.00th=[ 113], | 99.00th=[ 113], 99.50th=[ 113], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 2048, max= 6131, per=26.06%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=4, BW=5086KiB/s (5208kB/s)(6144KiB/1208msec) clat (msec): min=59, max=150, avg=97.04, stdev=33.99 lat (msec): min=60, max=150, avg=97.13, stdev=33.99 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 61], 10.00th=[ 61], 20.00th=[ 77], | 30.00th=[ 77], 40.00th=[ 82], 50.00th=[ 82], 60.00th=[ 89], | 70.00th=[ 126], 80.00th=[ 126], 90.00th=[ 150], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 4087, max= 6144, per=23.28%, avg=5115.50, stdev=1454.52, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 lat (msec) : 50=16.67%, 100=58.33%, 250=25.00% cpu : usr=0.00%, sys=4.72%, ctx=39, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30674: Thu May 1 20:14:48 2025 read: IOPS=4, BW=4491KiB/s (4599kB/s)(5120KiB/1140msec) clat (msec): min=20, max=119, avg=59.91, stdev=40.13 lat (msec): min=20, max=119, avg=59.92, stdev=40.13 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 46], 60.00th=[ 46], | 70.00th=[ 82], 80.00th=[ 82], 90.00th=[ 120], 95.00th=[ 120], | 99.00th=[ 120], 99.50th=[ 120], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 8175, max= 8175, per=52.09%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6288KiB/s (6439kB/s)(7168KiB/1140msec) clat (msec): min=67, max=139, avg=108.18, stdev=30.33 lat (msec): min=67, max=139, avg=108.26, stdev=30.33 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 69], | 30.00th=[ 99], 40.00th=[ 99], 50.00th=[ 121], 60.00th=[ 128], | 70.00th=[ 128], 80.00th=[ 136], 90.00th=[ 140], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 6131, max= 6144, per=27.94%, avg=6137.50, stdev= 9.19, samples=2 iops : min= 5, max= 6, avg= 5.50, stdev= 0.71, samples=2 lat (msec) : 50=25.00%, 100=33.33%, 250=41.67% cpu : usr=0.00%, sys=4.21%, ctx=41, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.3MiB/s (16.1MB/s), 1619KiB/s-5493KiB/s (1658kB/s-5625kB/s), io=20.0MiB (20.0MB), run=1140-1305msec WRITE: bw=21.5MiB/s (22.5MB/s), 3923KiB/s-8095KiB/s (4018kB/s-8289kB/s), io=28.0MiB (29.4MB), run=1140-1305msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30681: Thu May 1 20:14:51 2025 read: IOPS=1, BW=7362KiB/s (7539kB/s)(12.0MiB/1669msec) clat (msec): min=26, max=1210, avg=537.41, stdev=608.36 lat (msec): min=26, max=1210, avg=537.41, stdev=608.36 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 376], 50.00th=[ 376], 60.00th=[ 376], | 70.00th=[ 1217], 80.00th=[ 1217], 90.00th=[ 1217], 95.00th=[ 1217], | 99.00th=[ 1217], 99.50th=[ 1217], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 500=33.33% cpu : usr=0.96%, sys=11.81%, ctx=497, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30682: Thu May 1 20:14:51 2025 read: IOPS=0, BW=1732KiB/s (1773kB/s)(4096KiB/2365msec) clat (nsec): min=1244.2M, max=1244.2M, avg=1244189372.00, stdev= 0.00 lat (nsec): min=1244.2M, max=1244.2M, avg=1244191854.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1250], 5.00th=[ 1250], 10.00th=[ 1250], 20.00th=[ 1250], | 30.00th=[ 1250], 40.00th=[ 1250], 50.00th=[ 1250], 60.00th=[ 1250], | 70.00th=[ 1250], 80.00th=[ 1250], 90.00th=[ 1250], 95.00th=[ 1250], | 99.00th=[ 1250], 99.50th=[ 1250], 99.90th=[ 1250], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3464KiB/s (3547kB/s)(8192KiB/2365msec) clat (msec): min=400, max=698, avg=549.39, stdev=211.25 lat (msec): min=400, max=698, avg=549.62, stdev=211.20 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 401], 10.00th=[ 401], 20.00th=[ 401], | 30.00th=[ 401], 40.00th=[ 401], 50.00th=[ 401], 60.00th=[ 701], | 70.00th=[ 701], 80.00th=[ 701], 90.00th=[ 701], 95.00th=[ 701], | 99.00th=[ 701], 99.50th=[ 701], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 8175, max= 8175, per=59.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.38%, sys=12.35%, ctx=444, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30683: Thu May 1 20:14:51 2025 write: IOPS=1, BW=6095KiB/s (6242kB/s)(12.0MiB/2016msec) clat (msec): min=142, max=1244, avg=665.33, stdev=553.17 lat (msec): min=143, max=1245, avg=665.70, stdev=553.23 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 609], 50.00th=[ 609], 60.00th=[ 609], | 70.00th=[ 1250], 80.00th=[ 1250], 90.00th=[ 1250], 95.00th=[ 1250], | 99.00th=[ 1250], 99.50th=[ 1250], 99.90th=[ 1250], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 7937, max= 8126, per=57.97%, avg=8031.50, stdev=133.64, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.05%, sys=13.35%, ctx=82, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30684: Thu May 1 20:14:51 2025 write: IOPS=1, BW=5818KiB/s (5958kB/s)(12.0MiB/2112msec) clat (msec): min=159, max=1363, avg=691.91, stdev=613.71 lat (msec): min=159, max=1363, avg=692.27, stdev=613.66 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 550], 50.00th=[ 550], 60.00th=[ 550], | 70.00th=[ 1368], 80.00th=[ 1368], 90.00th=[ 1368], 95.00th=[ 1368], | 99.00th=[ 1368], 99.50th=[ 1368], 99.90th=[ 1368], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min= 8192, max= 8192, per=59.13%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=11.65%, ctx=56, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6928KiB/s (7094kB/s), 1732KiB/s-7362KiB/s (1773kB/s-7539kB/s), io=16.0MiB (16.8MB), run=1669-2365msec WRITE: bw=13.5MiB/s (14.2MB/s), 3464KiB/s-6095KiB/s (3547kB/s-6242kB/s), io=32.0MiB (33.6MB), run=2016-2365msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30685: Thu May 1 20:14:51 2025 read: IOPS=1, BW=5756KiB/s (5894kB/s)(12.0MiB/2135msec) clat (msec): min=90, max=1388, avg=701.77, stdev=651.88 lat (msec): min=90, max=1388, avg=701.78, stdev=651.88 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 91], 10.00th=[ 91], 20.00th=[ 91], | 30.00th=[ 91], 40.00th=[ 625], 50.00th=[ 625], 60.00th=[ 625], | 70.00th=[ 1385], 80.00th=[ 1385], 90.00th=[ 1385], 95.00th=[ 1385], | 99.00th=[ 1385], 99.50th=[ 1385], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.09%, sys=0.94%, ctx=14, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30686: Thu May 1 20:14:51 2025 read: IOPS=0, BW=1665KiB/s (1705kB/s)(4096KiB/2460msec) clat (nsec): min=1322.1M, max=1322.1M, avg=1322069390.00, stdev= 0.00 lat (nsec): min=1322.1M, max=1322.1M, avg=1322071755.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1318], 5.00th=[ 1318], 10.00th=[ 1318], 20.00th=[ 1318], | 30.00th=[ 1318], 40.00th=[ 1318], 50.00th=[ 1318], 60.00th=[ 1318], | 70.00th=[ 1318], 80.00th=[ 1318], 90.00th=[ 1318], 95.00th=[ 1318], | 99.00th=[ 1318], 99.50th=[ 1318], 99.90th=[ 1318], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 8159, max= 8159, per=100.00%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3330KiB/s (3410kB/s)(8192KiB/2460msec) clat (msec): min=489, max=619, avg=554.28, stdev=91.68 lat (msec): min=489, max=619, avg=554.62, stdev=91.64 clat percentiles (msec): | 1.00th=[ 489], 5.00th=[ 489], 10.00th=[ 489], 20.00th=[ 489], | 30.00th=[ 489], 40.00th=[ 489], 50.00th=[ 489], 60.00th=[ 617], | 70.00th=[ 617], 80.00th=[ 617], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 5572, max= 5572, per=41.83%, avg=5572.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.20%, sys=2.72%, ctx=32, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30687: Thu May 1 20:14:51 2025 write: IOPS=1, BW=5447KiB/s (5578kB/s)(12.0MiB/2256msec) clat (msec): min=95, max=1547, avg=736.21, stdev=740.95 lat (msec): min=95, max=1547, avg=736.53, stdev=740.95 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 95], 10.00th=[ 95], 20.00th=[ 95], | 30.00th=[ 95], 40.00th=[ 567], 50.00th=[ 567], 60.00th=[ 567], | 70.00th=[ 1552], 80.00th=[ 1552], 90.00th=[ 1552], 95.00th=[ 1552], | 99.00th=[ 1552], 99.50th=[ 1552], 99.90th=[ 1552], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min=11314, max=11314, per=84.94%, avg=11314.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.27%, sys=1.06%, ctx=12, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30688: Thu May 1 20:14:51 2025 write: IOPS=1, BW=6763KiB/s (6925kB/s)(12.0MiB/1817msec) clat (msec): min=113, max=1234, avg=600.03, stdev=575.32 lat (msec): min=113, max=1235, avg=600.49, stdev=575.45 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 113], 10.00th=[ 113], 20.00th=[ 113], | 30.00th=[ 113], 40.00th=[ 451], 50.00th=[ 451], 60.00th=[ 451], | 70.00th=[ 1234], 80.00th=[ 1234], 90.00th=[ 1234], 95.00th=[ 1234], | 99.00th=[ 1234], 99.50th=[ 1234], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 8126, max= 8126, per=61.01%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.28%, sys=2.70%, ctx=24, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6660KiB/s (6820kB/s), 1665KiB/s-5756KiB/s (1705kB/s-5894kB/s), io=16.0MiB (16.8MB), run=2135-2460msec WRITE: bw=13.0MiB/s (13.6MB/s), 3330KiB/s-6763KiB/s (3410kB/s-6925kB/s), io=32.0MiB (33.6MB), run=1817-2460msec