== sanity test 398b: DIO and buffer IO race ============== 19:22:29 (1746141749) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.94415 s, 25.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30612: Thu May 1 19:23:10 2025 read: IOPS=41, BW=165KiB/s (169kB/s)(5992KiB/36413msec) clat (usec): min=62, max=350434, avg=11255.52, stdev=12989.12 lat (usec): min=62, max=350435, avg=11258.01, stdev=12989.29 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 99], 10.00th=[ 113], 20.00th=[ 725], | 30.00th=[ 6128], 40.00th=[ 7767], 50.00th=[ 9503], 60.00th=[ 11338], | 70.00th=[ 13960], 80.00th=[ 17433], 90.00th=[ 22938], 95.00th=[ 30016], | 99.00th=[ 38011], 99.50th=[ 48497], 99.90th=[122160], 99.95th=[350225], | 99.99th=[350225] bw ( KiB/s): min= 8, max= 256, per=25.50%, avg=164.71, stdev=49.89, samples=72 iops : min= 2, max= 64, avg=40.88, stdev=12.53, samples=72 write: IOPS=43, BW=173KiB/s (177kB/s)(6296KiB/36413msec) clat (usec): min=449, max=127524, avg=12276.73, stdev=13276.01 lat (usec): min=450, max=127525, avg=12280.61, stdev=13276.25 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 742], 10.00th=[ 799], 20.00th=[ 1139], | 30.00th=[ 2376], 40.00th=[ 5080], 50.00th=[ 8979], 60.00th=[ 12256], | 70.00th=[ 16188], 80.00th=[ 20579], 90.00th=[ 28705], 95.00th=[ 36963], | 99.00th=[ 62653], 99.50th=[ 70779], 99.90th=[102237], 99.95th=[127402], | 99.99th=[127402] bw ( KiB/s): min= 79, max= 336, per=25.96%, avg=175.48, stdev=52.65, samples=71 iops : min= 19, max= 84, avg=43.56, stdev=13.25, samples=71 lat (usec) : 100=2.67%, 250=6.35%, 500=0.65%, 750=3.32%, 1000=5.89% lat (msec) : 2=5.73%, 4=5.08%, 10=23.11%, 20=29.49%, 50=16.57% lat (msec) : 100=1.01%, 250=0.10%, 500=0.03% cpu : usr=0.14%, sys=12.42%, ctx=5083, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30614: Thu May 1 19:23:10 2025 read: IOPS=40, BW=162KiB/s (166kB/s)(5940KiB/36564msec) clat (usec): min=56, max=99170, avg=10429.43, stdev=9249.42 lat (usec): min=57, max=99170, avg=10430.71, stdev=9249.55 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 97], 10.00th=[ 112], 20.00th=[ 161], | 30.00th=[ 5538], 40.00th=[ 7242], 50.00th=[ 8848], 60.00th=[10814], | 70.00th=[13304], 80.00th=[17171], 90.00th=[22152], 95.00th=[27657], | 99.00th=[39060], 99.50th=[42206], 99.90th=[55837], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 8, max= 344, per=25.05%, avg=161.81, stdev=56.21, samples=73 iops : min= 2, max= 86, avg=40.12, stdev=14.09, samples=73 write: IOPS=43, BW=174KiB/s (178kB/s)(6348KiB/36564msec) clat (usec): min=490, max=336156, avg=13129.95, stdev=15906.15 lat (usec): min=491, max=336157, avg=13140.66, stdev=15912.28 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 758], 10.00th=[ 807], 20.00th=[ 1123], | 30.00th=[ 2442], 40.00th=[ 5407], 50.00th=[ 9896], 60.00th=[ 13304], | 70.00th=[ 17433], 80.00th=[ 21890], 90.00th=[ 29492], 95.00th=[ 38011], | 99.00th=[ 63701], 99.50th=[ 70779], 99.90th=[135267], 99.95th=[337642], | 99.99th=[337642] bw ( KiB/s): min= 72, max= 320, per=25.92%, avg=175.24, stdev=52.83, samples=72 iops : min= 18, max= 80, avg=43.47, stdev=13.27, samples=72 lat (usec) : 100=2.99%, 250=7.10%, 500=0.33%, 750=2.60%, 1000=7.13% lat (msec) : 2=5.24%, 4=5.53%, 10=22.23%, 20=27.83%, 50=17.81% lat (msec) : 100=1.11%, 250=0.07%, 500=0.03% cpu : usr=0.11%, sys=12.37%, ctx=5103, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30616: Thu May 1 19:23:10 2025 read: IOPS=40, BW=164KiB/s (168kB/s)(6028KiB/36766msec) clat (usec): min=52, max=344321, avg=11208.96, stdev=13390.73 lat (usec): min=52, max=344323, avg=11210.72, stdev=13390.55 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 95], 10.00th=[ 108], 20.00th=[ 231], | 30.00th=[ 5669], 40.00th=[ 7570], 50.00th=[ 9372], 60.00th=[ 11469], | 70.00th=[ 13960], 80.00th=[ 17171], 90.00th=[ 23725], 95.00th=[ 28967], | 99.00th=[ 41157], 99.50th=[ 56886], 99.90th=[143655], 99.95th=[346031], | 99.99th=[346031] bw ( KiB/s): min= 7, max= 295, per=25.27%, avg=163.26, stdev=51.77, samples=73 iops : min= 1, max= 73, avg=40.49, stdev=12.97, samples=73 write: IOPS=42, BW=170KiB/s (174kB/s)(6260KiB/36766msec) clat (usec): min=523, max=130273, avg=12558.29, stdev=13812.72 lat (usec): min=524, max=130273, avg=12561.94, stdev=13811.88 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 742], 10.00th=[ 799], 20.00th=[ 1106], | 30.00th=[ 2180], 40.00th=[ 4883], 50.00th=[ 8848], 60.00th=[ 12387], | 70.00th=[ 16581], 80.00th=[ 21365], 90.00th=[ 29230], 95.00th=[ 39060], | 99.00th=[ 60031], 99.50th=[ 73925], 99.90th=[119014], 99.95th=[130548], | 99.99th=[130548] bw ( KiB/s): min= 23, max= 335, per=25.05%, avg=169.32, stdev=60.22, samples=73 iops : min= 5, max= 83, avg=42.00, stdev=15.05, samples=73 lat (usec) : 100=3.48%, 250=6.38%, 500=0.59%, 750=2.90%, 1000=6.61% lat (msec) : 2=5.50%, 4=5.11%, 10=23.11%, 20=26.76%, 50=18.42% lat (msec) : 100=0.98%, 250=0.13%, 500=0.03% cpu : usr=0.20%, sys=12.32%, ctx=5066, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30618: Thu May 1 19:23:10 2025 read: IOPS=40, BW=163KiB/s (167kB/s)(6052KiB/37160msec) clat (usec): min=49, max=120588, avg=10587.04, stdev=10156.62 lat (usec): min=49, max=120589, avg=10591.53, stdev=10161.65 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 95], 10.00th=[ 109], 20.00th=[ 143], | 30.00th=[ 5407], 40.00th=[ 7242], 50.00th=[ 8979], 60.00th=[ 11076], | 70.00th=[ 13304], 80.00th=[ 17171], 90.00th=[ 22152], 95.00th=[ 28443], | 99.00th=[ 40109], 99.50th=[ 51643], 99.90th=[ 87557], 99.95th=[121111], | 99.99th=[121111] bw ( KiB/s): min= 55, max= 280, per=25.27%, avg=163.23, stdev=56.74, samples=73 iops : min= 13, max= 70, avg=40.44, stdev=14.20, samples=73 write: IOPS=41, BW=168KiB/s (172kB/s)(6236KiB/37160msec) clat (usec): min=458, max=438253, avg=13393.82, stdev=17616.89 lat (usec): min=459, max=438256, avg=13395.99, stdev=17616.45 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 750], 10.00th=[ 816], 20.00th=[ 1123], | 30.00th=[ 2638], 40.00th=[ 6652], 50.00th=[ 10028], 60.00th=[ 13566], | 70.00th=[ 17171], 80.00th=[ 22152], 90.00th=[ 30016], 95.00th=[ 39060], | 99.00th=[ 62129], 99.50th=[ 73925], 99.90th=[164627], 99.95th=[438305], | 99.99th=[438305] bw ( KiB/s): min= 63, max= 296, per=24.92%, avg=168.44, stdev=52.61, samples=73 iops : min= 15, max= 74, avg=41.75, stdev=13.14, samples=73 lat (usec) : 50=0.03%, 100=3.35%, 250=7.39%, 500=0.68%, 750=2.47% lat (usec) : 1000=6.84% lat (msec) : 2=4.98%, 4=4.39%, 10=22.27%, 20=28.58%, 50=17.81% lat (msec) : 100=1.07%, 250=0.10%, 500=0.03% cpu : usr=0.10%, sys=12.29%, ctx=5136, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=646KiB/s (662kB/s), 162KiB/s-165KiB/s (166kB/s-169kB/s), io=23.4MiB (24.6MB), run=36413-37160msec WRITE: bw=677KiB/s (693kB/s), 168KiB/s-174KiB/s (172kB/s-178kB/s), io=24.6MiB (25.7MB), run=36413-37160msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30611: Thu May 1 19:24:45 2025 read: IOPS=11, BW=45.7KiB/s (46.8kB/s)(5992KiB/131222msec) clat (msec): min=3, max=357, avg=11.42, stdev=13.10 lat (msec): min=3, max=357, avg=11.42, stdev=13.10 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 16], 90.00th=[ 21], 95.00th=[ 30], | 99.00th=[ 49], 99.50th=[ 70], 99.90th=[ 115], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 7, max= 144, per=25.52%, avg=46.44, stdev=26.85, samples=256 iops : min= 1, max= 36, avg=11.36, stdev= 6.77, samples=256 write: IOPS=11, BW=47.0KiB/s (49.1kB/s)(6296KiB/131222msec) clat (msec): min=24, max=258, avg=72.35, stdev=29.62 lat (msec): min=24, max=258, avg=72.36, stdev=29.62 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 39], 10.00th=[ 43], 20.00th=[ 50], | 30.00th=[ 56], 40.00th=[ 62], 50.00th=[ 67], 60.00th=[ 73], | 70.00th=[ 81], 80.00th=[ 90], 90.00th=[ 108], 95.00th=[ 128], | 99.00th=[ 184], 99.50th=[ 203], 99.90th=[ 241], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 15, max= 80, per=25.15%, avg=47.78, stdev=14.72, samples=261 iops : min= 3, max= 20, avg=11.69, stdev= 3.77, samples=261 lat (msec) : 4=1.69%, 10=29.69%, 20=11.91%, 50=16.08%, 100=33.79% lat (msec) : 250=6.77%, 500=0.07% cpu : usr=0.06%, sys=2.77%, ctx=3599, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30613: Thu May 1 19:24:45 2025 read: IOPS=11, BW=45.1KiB/s (46.2kB/s)(5940KiB/131676msec) clat (msec): min=2, max=305, avg=11.38, stdev=11.81 lat (msec): min=2, max=305, avg=11.38, stdev=11.81 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 16], 90.00th=[ 22], 95.00th=[ 29], | 99.00th=[ 46], 99.50th=[ 57], 99.90th=[ 109], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 7, max= 135, per=25.00%, avg=45.50, stdev=27.57, samples=258 iops : min= 1, max= 33, avg=11.07, stdev= 6.95, samples=258 write: IOPS=12, BW=48.2KiB/s (49.4kB/s)(6348KiB/131676msec) clat (msec): min=23, max=254, avg=72.17, stdev=30.46 lat (msec): min=23, max=254, avg=72.18, stdev=30.46 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 38], 10.00th=[ 42], 20.00th=[ 48], | 30.00th=[ 54], 40.00th=[ 60], 50.00th=[ 66], 60.00th=[ 73], | 70.00th=[ 81], 80.00th=[ 92], 90.00th=[ 110], 95.00th=[ 132], | 99.00th=[ 180], 99.50th=[ 207], 99.90th=[ 247], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 15, max= 88, per=25.20%, avg=47.89, stdev=15.40, samples=262 iops : min= 3, max= 22, avg=11.66, stdev= 3.93, samples=262 lat (msec) : 4=1.37%, 10=29.30%, 20=12.04%, 50=17.38%, 100=32.49% lat (msec) : 250=7.36%, 500=0.07% cpu : usr=0.04%, sys=2.76%, ctx=3622, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30615: Thu May 1 19:24:45 2025 read: IOPS=11, BW=46.1KiB/s (47.2kB/s)(6028KiB/130723msec) clat (msec): min=2, max=200, avg=11.81, stdev=11.56 lat (msec): min=2, max=200, avg=11.81, stdev=11.56 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 13], 80.00th=[ 16], 90.00th=[ 23], 95.00th=[ 29], | 99.00th=[ 48], 99.50th=[ 73], 99.90th=[ 188], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 7, max= 152, per=25.60%, avg=46.60, stdev=26.39, samples=257 iops : min= 1, max= 38, avg=11.45, stdev= 6.64, samples=257 write: IOPS=11, BW=47.9KiB/s (49.0kB/s)(6260KiB/130723msec) clat (msec): min=25, max=304, avg=71.96, stdev=30.59 lat (msec): min=25, max=304, avg=71.96, stdev=30.59 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 48], | 30.00th=[ 55], 40.00th=[ 61], 50.00th=[ 66], 60.00th=[ 72], | 70.00th=[ 80], 80.00th=[ 91], 90.00th=[ 110], 95.00th=[ 132], | 99.00th=[ 182], 99.50th=[ 203], 99.90th=[ 245], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 8, max= 88, per=25.16%, avg=47.80, stdev=15.76, samples=260 iops : min= 2, max= 22, avg=11.75, stdev= 3.98, samples=260 lat (msec) : 4=1.20%, 10=28.48%, 20=13.05%, 50=17.90%, 100=32.36% lat (msec) : 250=6.97%, 500=0.03% cpu : usr=0.05%, sys=2.73%, ctx=3625, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30617: Thu May 1 19:24:45 2025 read: IOPS=11, BW=46.5KiB/s (47.7kB/s)(6052KiB/130031msec) clat (msec): min=2, max=197, avg=11.54, stdev=11.15 lat (msec): min=2, max=197, avg=11.54, stdev=11.15 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 16], 90.00th=[ 22], 95.00th=[ 29], | 99.00th=[ 46], 99.50th=[ 53], 99.90th=[ 188], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 7, max= 159, per=25.96%, avg=47.24, stdev=26.17, samples=254 iops : min= 1, max= 39, avg=11.50, stdev= 6.59, samples=254 write: IOPS=11, BW=47.0KiB/s (49.1kB/s)(6236KiB/130031msec) clat (msec): min=26, max=473, avg=72.07, stdev=31.31 lat (msec): min=26, max=473, avg=72.07, stdev=31.31 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 38], 10.00th=[ 43], 20.00th=[ 50], | 30.00th=[ 55], 40.00th=[ 61], 50.00th=[ 66], 60.00th=[ 72], | 70.00th=[ 80], 80.00th=[ 90], 90.00th=[ 108], 95.00th=[ 129], | 99.00th=[ 174], 99.50th=[ 192], 99.90th=[ 334], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 7, max= 80, per=25.11%, avg=47.71, stdev=15.05, samples=259 iops : min= 1, max= 20, avg=11.62, stdev= 3.85, samples=259 lat (msec) : 4=1.04%, 10=29.98%, 20=12.17%, 50=16.83%, 100=32.65% lat (msec) : 250=7.26%, 500=0.07% cpu : usr=0.03%, sys=2.83%, ctx=3669, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=182KiB/s (187kB/s), 45.1KiB/s-46.5KiB/s (46.2kB/s-47.7kB/s), io=23.4MiB (24.6MB), run=130031-131676msec WRITE: bw=191KiB/s (196kB/s), 47.9KiB/s-48.2KiB/s (49.0kB/s-49.4kB/s), io=24.6MiB (25.7MB), run=130031-131676msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30740: Thu May 1 19:24:53 2025 read: IOPS=51, BW=829KiB/s (849kB/s)(5920KiB/7140msec) clat (usec): min=52, max=56387, avg=8692.80, stdev=9164.60 lat (usec): min=52, max=56389, avg=8696.12, stdev=9162.77 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 92], 10.00th=[ 101], 20.00th=[ 119], | 30.00th=[ 147], 40.00th=[ 5866], 50.00th=[ 7701], 60.00th=[ 9503], | 70.00th=[11600], 80.00th=[14615], 90.00th=[20055], 95.00th=[26084], | 99.00th=[38536], 99.50th=[42730], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 416, max= 1440, per=25.22%, avg=831.36, stdev=279.79, samples=14 iops : min= 26, max= 90, avg=51.79, stdev=17.58, samples=14 write: IOPS=55, BW=892KiB/s (913kB/s)(6368KiB/7140msec) clat (usec): min=811, max=103140, avg=9706.21, stdev=12088.57 lat (usec): min=812, max=103142, avg=9708.32, stdev=12088.66 clat percentiles (usec): | 1.00th=[ 881], 5.00th=[ 1037], 10.00th=[ 1139], 20.00th=[ 1352], | 30.00th=[ 1811], 40.00th=[ 2737], 50.00th=[ 4047], 60.00th=[ 6652], | 70.00th=[ 11994], 80.00th=[ 17433], 90.00th=[ 26608], 95.00th=[ 34866], | 99.00th=[ 47449], 99.50th=[ 69731], 99.90th=[103285], 99.95th=[103285], | 99.99th=[103285] bw ( KiB/s): min= 416, max= 1370, per=25.88%, avg=897.50, stdev=283.39, samples=14 iops : min= 26, max= 85, avg=55.93, stdev=17.73, samples=14 lat (usec) : 100=4.30%, 250=11.59%, 500=0.65%, 1000=2.21% lat (msec) : 2=16.54%, 4=8.59%, 10=20.70%, 20=21.22%, 50=13.67% lat (msec) : 100=0.39%, 250=0.13% cpu : usr=0.24%, sys=14.95%, ctx=1127, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30741: Thu May 1 19:24:53 2025 read: IOPS=50, BW=801KiB/s (820kB/s)(5744KiB/7175msec) clat (usec): min=76, max=94286, avg=8651.51, stdev=10433.12 lat (usec): min=76, max=94287, avg=8652.67, stdev=10433.10 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 96], 10.00th=[ 104], 20.00th=[ 119], | 30.00th=[ 149], 40.00th=[ 2900], 50.00th=[ 7308], 60.00th=[ 9110], | 70.00th=[11469], 80.00th=[15664], 90.00th=[20055], 95.00th=[23462], | 99.00th=[52167], 99.50th=[64226], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 192, max= 1370, per=24.24%, avg=798.86, stdev=268.27, samples=14 iops : min= 12, max= 85, avg=49.71, stdev=16.70, samples=14 write: IOPS=57, BW=912KiB/s (934kB/s)(6544KiB/7175msec) clat (usec): min=880, max=109199, avg=9726.04, stdev=12227.77 lat (usec): min=881, max=109202, avg=9728.57, stdev=12227.40 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 1037], 10.00th=[ 1106], 20.00th=[ 1352], | 30.00th=[ 1729], 40.00th=[ 2442], 50.00th=[ 4080], 60.00th=[ 8094], | 70.00th=[ 12518], 80.00th=[ 17695], 90.00th=[ 25297], 95.00th=[ 30278], | 99.00th=[ 46924], 99.50th=[ 73925], 99.90th=[109577], 99.95th=[109577], | 99.99th=[109577] bw ( KiB/s): min= 448, max= 1408, per=25.94%, avg=899.43, stdev=265.67, samples=14 iops : min= 28, max= 88, avg=56.00, stdev=16.59, samples=14 lat (usec) : 100=2.99%, 250=12.50%, 500=1.04%, 750=0.39%, 1000=1.82% lat (msec) : 2=17.58%, 4=8.85%, 10=19.40%, 20=22.27%, 50=12.11% lat (msec) : 100=0.91%, 250=0.13% cpu : usr=0.13%, sys=14.08%, ctx=1123, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30742: Thu May 1 19:24:53 2025 read: IOPS=53, BW=861KiB/s (882kB/s)(6256KiB/7266msec) clat (usec): min=74, max=56759, avg=7946.68, stdev=8448.24 lat (usec): min=75, max=56761, avg=7949.40, stdev=8449.73 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 93], 10.00th=[ 101], 20.00th=[ 115], | 30.00th=[ 130], 40.00th=[ 322], 50.00th=[ 7504], 60.00th=[ 9503], | 70.00th=[11207], 80.00th=[14877], 90.00th=[19792], 95.00th=[22152], | 99.00th=[27919], 99.50th=[54789], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 447, max= 1240, per=26.43%, avg=871.29, stdev=251.79, samples=14 iops : min= 27, max= 77, avg=54.00, stdev=15.76, samples=14 write: IOPS=51, BW=830KiB/s (850kB/s)(6032KiB/7266msec) clat (usec): min=738, max=107726, avg=10957.30, stdev=12724.92 lat (usec): min=740, max=107728, avg=10961.08, stdev=12725.05 clat percentiles (usec): | 1.00th=[ 938], 5.00th=[ 1074], 10.00th=[ 1139], 20.00th=[ 1369], | 30.00th=[ 1909], 40.00th=[ 2933], 50.00th=[ 5342], 60.00th=[ 10290], | 70.00th=[ 14877], 80.00th=[ 19530], 90.00th=[ 27919], 95.00th=[ 35390], | 99.00th=[ 58459], 99.50th=[ 66847], 99.90th=[107480], 99.95th=[107480], | 99.99th=[107480] bw ( KiB/s): min= 541, max= 1216, per=23.95%, avg=830.43, stdev=234.56, samples=14 iops : min= 33, max= 76, avg=51.43, stdev=14.78, samples=14 lat (usec) : 100=4.69%, 250=15.23%, 500=0.78%, 750=0.13%, 1000=1.17% lat (msec) : 2=14.32%, 4=7.68%, 10=16.80%, 20=25.13%, 50=13.28% lat (msec) : 100=0.65%, 250=0.13% cpu : usr=0.17%, sys=14.22%, ctx=1117, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30743: Thu May 1 19:24:53 2025 read: IOPS=52, BW=834KiB/s (854kB/s)(6032KiB/7232msec) clat (usec): min=54, max=78124, avg=8526.13, stdev=10362.51 lat (usec): min=54, max=78126, avg=8527.38, stdev=10362.39 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 93], 10.00th=[ 104], 20.00th=[ 117], | 30.00th=[ 137], 40.00th=[ 383], 50.00th=[ 7635], 60.00th=[10028], | 70.00th=[11600], 80.00th=[14484], 90.00th=[19530], 95.00th=[25035], | 99.00th=[53740], 99.50th=[77071], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 128, max= 1312, per=24.99%, avg=823.71, stdev=298.89, samples=14 iops : min= 8, max= 82, avg=51.14, stdev=18.74, samples=14 write: IOPS=54, BW=865KiB/s (886kB/s)(6256KiB/7232msec) clat (usec): min=779, max=104414, avg=9984.76, stdev=13423.65 lat (usec): min=780, max=104417, avg=9986.94, stdev=13423.51 clat percentiles (usec): | 1.00th=[ 881], 5.00th=[ 1012], 10.00th=[ 1139], 20.00th=[ 1352], | 30.00th=[ 1893], 40.00th=[ 2638], 50.00th=[ 4080], 60.00th=[ 7373], | 70.00th=[ 11469], 80.00th=[ 16712], 90.00th=[ 25297], 95.00th=[ 32375], | 99.00th=[ 64226], 99.50th=[ 84411], 99.90th=[104334], 99.95th=[104334], | 99.99th=[104334] bw ( KiB/s): min= 192, max= 1370, per=24.94%, avg=864.86, stdev=342.80, samples=14 iops : min= 12, max= 85, avg=53.71, stdev=21.43, samples=14 lat (usec) : 100=3.52%, 250=15.49%, 500=0.78%, 750=0.52%, 1000=2.73% lat (msec) : 2=14.19%, 4=8.72%, 10=16.93%, 20=24.74%, 50=10.29% lat (msec) : 100=1.95%, 250=0.13% cpu : usr=0.17%, sys=14.35%, ctx=1050, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3296KiB/s (3376kB/s), 801KiB/s-861KiB/s (820kB/s-882kB/s), io=23.4MiB (24.5MB), run=7140-7266msec WRITE: bw=3468KiB/s (3551kB/s), 830KiB/s-912KiB/s (850kB/s-934kB/s), io=24.6MiB (25.8MB), run=7140-7266msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30735: Thu May 1 19:25:17 2025 read: IOPS=11, BW=190KiB/s (195kB/s)(5920KiB/31142msec) clat (usec): min=3579, max=72918, avg=10697.99, stdev=9603.00 lat (usec): min=3581, max=72919, avg=10699.51, stdev=9602.91 clat percentiles (usec): | 1.00th=[ 3851], 5.00th=[ 4490], 10.00th=[ 5014], 20.00th=[ 5669], | 30.00th=[ 6128], 40.00th=[ 6849], 50.00th=[ 7504], 60.00th=[ 8291], | 70.00th=[ 9634], 80.00th=[12780], 90.00th=[19792], 95.00th=[28443], | 99.00th=[61080], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 31, max= 702, per=24.90%, avg=189.47, stdev=105.12, samples=62 iops : min= 1, max= 43, avg=11.45, stdev= 6.58, samples=62 write: IOPS=12, BW=204KiB/s (209kB/s)(6368KiB/31142msec) clat (msec): min=30, max=161, avg=68.14, stdev=24.88 lat (msec): min=30, max=161, avg=68.14, stdev=24.88 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 38], 10.00th=[ 42], 20.00th=[ 49], | 30.00th=[ 54], 40.00th=[ 58], 50.00th=[ 63], 60.00th=[ 67], | 70.00th=[ 74], 80.00th=[ 86], 90.00th=[ 104], 95.00th=[ 121], | 99.00th=[ 146], 99.50th=[ 159], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 64, max= 319, per=25.40%, avg=203.44, stdev=59.40, samples=62 iops : min= 4, max= 19, avg=12.32, stdev= 3.71, samples=62 lat (msec) : 4=0.52%, 10=33.33%, 20=9.64%, 50=16.15%, 100=34.24% lat (msec) : 250=6.12% cpu : usr=0.03%, sys=3.04%, ctx=872, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30736: Thu May 1 19:25:17 2025 read: IOPS=11, BW=183KiB/s (187kB/s)(5744KiB/31435msec) clat (usec): min=3199, max=79400, avg=10426.08, stdev=9453.87 lat (usec): min=3200, max=79400, avg=10427.75, stdev=9453.81 clat percentiles (usec): | 1.00th=[ 3884], 5.00th=[ 4555], 10.00th=[ 4948], 20.00th=[ 5604], | 30.00th=[ 6194], 40.00th=[ 6587], 50.00th=[ 7111], 60.00th=[ 8291], | 70.00th=[ 9634], 80.00th=[11469], 90.00th=[18744], 95.00th=[31589], | 99.00th=[56886], 99.50th=[58983], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 32, max= 479, per=24.63%, avg=187.43, stdev=105.41, samples=60 iops : min= 2, max= 29, avg=11.45, stdev= 6.56, samples=60 write: IOPS=13, BW=208KiB/s (213kB/s)(6544KiB/31435msec) clat (msec): min=22, max=191, avg=67.52, stdev=26.09 lat (msec): min=22, max=191, avg=67.52, stdev=26.09 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 37], 10.00th=[ 40], 20.00th=[ 46], | 30.00th=[ 52], 40.00th=[ 57], 50.00th=[ 63], 60.00th=[ 68], | 70.00th=[ 75], 80.00th=[ 87], 90.00th=[ 103], 95.00th=[ 122], | 99.00th=[ 150], 99.50th=[ 157], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 63, max= 320, per=25.80%, avg=206.68, stdev=60.48, samples=62 iops : min= 3, max= 20, avg=12.66, stdev= 3.87, samples=62 lat (msec) : 4=0.65%, 10=33.85%, 20=7.94%, 50=17.71%, 100=33.98% lat (msec) : 250=5.86% cpu : usr=0.10%, sys=2.96%, ctx=863, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30737: Thu May 1 19:25:17 2025 read: IOPS=12, BW=207KiB/s (212kB/s)(6256KiB/30236msec) clat (msec): min=3, max=120, avg=10.99, stdev=11.41 lat (msec): min=3, max=120, avg=11.00, stdev=11.41 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 13], 90.00th=[ 19], 95.00th=[ 30], | 99.00th=[ 72], 99.50th=[ 93], 99.90th=[ 121], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 63, max= 544, per=27.37%, avg=208.30, stdev=116.90, samples=60 iops : min= 3, max= 34, avg=12.80, stdev= 7.48, samples=60 write: IOPS=12, BW=199KiB/s (204kB/s)(6032KiB/30236msec) clat (msec): min=29, max=186, avg=68.66, stdev=24.69 lat (msec): min=29, max=186, avg=68.67, stdev=24.69 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 37], 10.00th=[ 42], 20.00th=[ 48], | 30.00th=[ 54], 40.00th=[ 60], 50.00th=[ 65], 60.00th=[ 70], | 70.00th=[ 77], 80.00th=[ 86], 90.00th=[ 101], 95.00th=[ 121], | 99.00th=[ 148], 99.50th=[ 150], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 64, max= 352, per=24.81%, avg=198.70, stdev=57.19, samples=60 iops : min= 4, max= 22, avg=12.20, stdev= 3.64, samples=60 lat (msec) : 4=0.52%, 10=35.81%, 20=9.77%, 50=15.49%, 100=33.20% lat (msec) : 250=5.21% cpu : usr=0.00%, sys=3.21%, ctx=891, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30738: Thu May 1 19:25:17 2025 read: IOPS=12, BW=193KiB/s (197kB/s)(6032KiB/31296msec) clat (msec): min=3, max=123, avg=10.84, stdev=10.60 lat (msec): min=3, max=123, avg=10.84, stdev=10.61 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 12], 90.00th=[ 22], 95.00th=[ 31], | 99.00th=[ 57], 99.50th=[ 69], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 31, max= 448, per=25.46%, avg=193.75, stdev=99.42, samples=61 iops : min= 1, max= 28, avg=11.79, stdev= 6.18, samples=61 write: IOPS=12, BW=200KiB/s (205kB/s)(6256KiB/31296msec) clat (msec): min=27, max=213, avg=69.50, stdev=26.02 lat (msec): min=27, max=213, avg=69.51, stdev=26.02 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 37], 10.00th=[ 42], 20.00th=[ 48], | 30.00th=[ 54], 40.00th=[ 59], 50.00th=[ 66], 60.00th=[ 72], | 70.00th=[ 79], 80.00th=[ 87], 90.00th=[ 103], 95.00th=[ 122], | 99.00th=[ 148], 99.50th=[ 184], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 64, max= 319, per=24.89%, avg=199.37, stdev=57.80, samples=62 iops : min= 4, max= 19, avg=12.15, stdev= 3.61, samples=62 lat (msec) : 4=0.52%, 10=36.07%, 20=7.16%, 50=16.54%, 100=33.98% lat (msec) : 250=5.73% cpu : usr=0.00%, sys=3.04%, ctx=871, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=762KiB/s (780kB/s), 183KiB/s-207KiB/s (187kB/s-212kB/s), io=23.4MiB (24.5MB), run=30236-31435msec WRITE: bw=802KiB/s (821kB/s), 199KiB/s-208KiB/s (204kB/s-213kB/s), io=24.6MiB (25.8MB), run=30236-31435msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30774: Thu May 1 19:25:22 2025 read: IOPS=2, BW=2509KiB/s (2569kB/s)(7168KiB/2857msec) clat (usec): min=895, max=1138.7k, avg=349365.70, stdev=534349.82 lat (usec): min=896, max=1138.7k, avg=349367.38, stdev=534349.86 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 898], 10.00th=[ 898], | 20.00th=[ 2343], 30.00th=[ 54264], 40.00th=[ 54264], | 50.00th=[ 60556], 60.00th=[ 66323], 70.00th=[ 66323], | 80.00th=[1115685], 90.00th=[1132463], 95.00th=[1132463], | 99.00th=[1132463], 99.50th=[1132463], 99.90th=[1132463], | 99.95th=[1132463], 99.99th=[1132463] bw ( KiB/s): min= 2048, max= 8192, per=87.06%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=1, BW=1792KiB/s (1835kB/s)(5120KiB/2857msec) clat (msec): min=32, max=198, avg=75.48, stdev=70.24 lat (msec): min=32, max=198, avg=75.54, stdev=70.22 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 45], 60.00th=[ 45], | 70.00th=[ 66], 80.00th=[ 66], 90.00th=[ 199], 95.00th=[ 199], | 99.00th=[ 199], 99.50th=[ 199], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 2048, max= 2048, per=24.87%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 1000=8.33% lat (msec) : 4=8.33%, 50=25.00%, 100=33.33%, 250=8.33% cpu : usr=0.07%, sys=15.20%, ctx=622, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30775: Thu May 1 19:25:22 2025 read: IOPS=0, BW=649KiB/s (664kB/s)(2048KiB/3158msec) clat (usec): min=79884, max=86144, avg=83014.41, stdev=4426.40 lat (usec): min=79886, max=86145, avg=83016.10, stdev=4426.20 clat percentiles (usec): | 1.00th=[80217], 5.00th=[80217], 10.00th=[80217], 20.00th=[80217], | 30.00th=[80217], 40.00th=[80217], 50.00th=[80217], 60.00th=[86508], | 70.00th=[86508], 80.00th=[86508], 90.00th=[86508], 95.00th=[86508], | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 2048, max= 2048, per=34.82%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3243KiB/s (3320kB/s)(10.0MiB/3158msec) clat (msec): min=21, max=2333, avg=293.35, stdev=718.75 lat (msec): min=21, max=2333, avg=293.42, stdev=718.75 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 24], | 30.00th=[ 27], 40.00th=[ 56], 50.00th=[ 56], 60.00th=[ 61], | 70.00th=[ 79], 80.00th=[ 79], 90.00th=[ 209], 95.00th=[ 2333], | 99.00th=[ 2333], 99.50th=[ 2333], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 2048, max=12288, per=87.05%, avg=7168.00, stdev=7240.77, samples=2 iops : min= 2, max= 12, avg= 7.00, stdev= 7.07, samples=2 lat (msec) : 50=25.00%, 100=58.33%, 250=8.33% cpu : usr=0.00%, sys=7.54%, ctx=425, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30776: Thu May 1 19:25:22 2025 read: IOPS=1, BW=2024KiB/s (2073kB/s)(6144KiB/3035msec) clat (usec): min=474, max=79263, avg=34411.65, stdev=37550.30 lat (usec): min=475, max=79263, avg=34412.81, stdev=37550.04 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 474], 10.00th=[ 474], 20.00th=[ 482], | 30.00th=[ 482], 40.00th=[ 881], 50.00th=[ 881], 60.00th=[60031], | 70.00th=[65274], 80.00th=[65274], 90.00th=[79168], 95.00th=[79168], | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min=10219, max=10219, per=100.00%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=1, BW=2024KiB/s (2073kB/s)(6144KiB/3035msec) clat (msec): min=36, max=2287, avg=454.21, stdev=898.92 lat (msec): min=36, max=2287, avg=454.26, stdev=898.92 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 62], | 30.00th=[ 62], 40.00th=[ 69], 50.00th=[ 69], 60.00th=[ 125], | 70.00th=[ 146], 80.00th=[ 146], 90.00th=[ 2299], 95.00th=[ 2299], | 99.00th=[ 2299], 99.50th=[ 2299], 99.90th=[ 2299], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 4096, max= 6131, per=62.10%, avg=5113.50, stdev=1438.96, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 lat (usec) : 500=16.67%, 1000=8.33% lat (msec) : 50=8.33%, 100=41.67%, 250=16.67% cpu : usr=0.00%, sys=5.87%, ctx=223, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30777: Thu May 1 19:25:22 2025 read: IOPS=1, BW=1470KiB/s (1506kB/s)(5120KiB/3482msec) clat (usec): min=437, max=73843, avg=27150.55, stdev=32404.65 lat (usec): min=438, max=73843, avg=27151.79, stdev=32404.50 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 437], 10.00th=[ 437], 20.00th=[ 437], | 30.00th=[ 510], 40.00th=[ 510], 50.00th=[13566], 60.00th=[13566], | 70.00th=[47449], 80.00th=[47449], 90.00th=[73925], 95.00th=[73925], | 99.00th=[73925], 99.50th=[73925], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] write: IOPS=2, BW=2059KiB/s (2108kB/s)(7168KiB/3482msec) clat (msec): min=66, max=2266, avg=464.00, stdev=799.61 lat (msec): min=66, max=2266, avg=464.09, stdev=799.61 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 111], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 150], 60.00th=[ 176], | 70.00th=[ 176], 80.00th=[ 342], 90.00th=[ 2265], 95.00th=[ 2265], | 99.00th=[ 2265], 99.50th=[ 2265], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 2039, max= 2048, per=24.82%, avg=2043.33, stdev= 4.51, samples=3 iops : min= 1, max= 2, avg= 1.33, stdev= 0.58, samples=3 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 20=8.33%, 50=8.33%, 100=16.67%, 250=33.33%, 500=8.33% cpu : usr=0.00%, sys=6.09%, ctx=247, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5882KiB/s (6023kB/s), 649KiB/s-2509KiB/s (664kB/s-2569kB/s), io=20.0MiB (20.0MB), run=2857-3482msec WRITE: bw=8234KiB/s (8432kB/s), 1792KiB/s-3243KiB/s (1835kB/s-3320kB/s), io=28.0MiB (29.4MB), run=2857-3482msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30778: Thu May 1 19:25:23 2025 read: IOPS=1, BW=1891KiB/s (1936kB/s)(7168KiB/3791msec) clat (msec): min=22, max=1614, avg=446.05, stdev=647.65 lat (msec): min=22, max=1614, avg=446.05, stdev=647.65 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 32], | 30.00th=[ 55], 40.00th=[ 55], 50.00th=[ 124], 60.00th=[ 155], | 70.00th=[ 155], 80.00th=[ 1116], 90.00th=[ 1620], 95.00th=[ 1620], | 99.00th=[ 1620], 99.50th=[ 1620], 99.90th=[ 1620], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 2048, max= 4096, per=64.67%, avg=3309.67, stdev=1103.65, samples=3 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=3 write: IOPS=1, BW=1351KiB/s (1383kB/s)(5120KiB/3791msec) clat (msec): min=59, max=227, avg=122.19, stdev=67.23 lat (msec): min=59, max=227, avg=122.28, stdev=67.21 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 60], 10.00th=[ 60], 20.00th=[ 60], | 30.00th=[ 73], 40.00th=[ 73], 50.00th=[ 108], 60.00th=[ 108], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 228], 95.00th=[ 228], | 99.00th=[ 228], 99.50th=[ 228], 99.90th=[ 228], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 2048, max= 3785, per=40.70%, avg=2916.50, stdev=1228.24, samples=2 iops : min= 2, max= 3, avg= 2.50, stdev= 0.71, samples=2 lat (msec) : 50=16.67%, 100=25.00%, 250=41.67% cpu : usr=0.03%, sys=2.93%, ctx=54, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30779: Thu May 1 19:25:23 2025 read: IOPS=0, BW=512KiB/s (524kB/s)(2048KiB/4001msec) clat (usec): min=29234, max=40933, avg=35084.24, stdev=8272.69 lat (usec): min=29236, max=40936, avg=35086.32, stdev=8273.33 clat percentiles (usec): | 1.00th=[29230], 5.00th=[29230], 10.00th=[29230], 20.00th=[29230], | 30.00th=[29230], 40.00th=[29230], 50.00th=[29230], 60.00th=[41157], | 70.00th=[41157], 80.00th=[41157], 90.00th=[41157], 95.00th=[41157], | 99.00th=[41157], 99.50th=[41157], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 1928, max= 2048, per=38.84%, avg=1988.00, stdev=84.85, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2559KiB/s (2621kB/s)(10.0MiB/4001msec) clat (msec): min=74, max=2403, avg=380.53, stdev=715.01 lat (msec): min=74, max=2403, avg=380.62, stdev=715.00 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 75], 10.00th=[ 75], 20.00th=[ 77], | 30.00th=[ 99], 40.00th=[ 101], 50.00th=[ 142], 60.00th=[ 153], | 70.00th=[ 279], 80.00th=[ 279], 90.00th=[ 292], 95.00th=[ 2400], | 99.00th=[ 2400], 99.50th=[ 2400], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 5785, max= 6107, per=82.98%, avg=5946.00, stdev=227.69, samples=2 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=2 lat (msec) : 50=16.67%, 100=33.33%, 250=25.00%, 500=16.67% cpu : usr=0.03%, sys=3.20%, ctx=56, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30780: Thu May 1 19:25:23 2025 read: IOPS=1, BW=1675KiB/s (1715kB/s)(6144KiB/3669msec) clat (msec): min=25, max=412, avg=121.26, stdev=150.77 lat (msec): min=25, max=412, avg=121.27, stdev=150.77 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 48], 50.00th=[ 48], 60.00th=[ 57], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 2043, max= 6144, per=79.98%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 write: IOPS=1, BW=1675KiB/s (1715kB/s)(6144KiB/3669msec) clat (msec): min=66, max=2287, avg=484.14, stdev=885.15 lat (msec): min=66, max=2287, avg=484.24, stdev=885.21 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 82], | 30.00th=[ 82], 40.00th=[ 117], 50.00th=[ 117], 60.00th=[ 125], | 70.00th=[ 228], 80.00th=[ 228], 90.00th=[ 2299], 95.00th=[ 2299], | 99.00th=[ 2299], 99.50th=[ 2299], 99.90th=[ 2299], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 2043, max= 4096, per=47.57%, avg=3408.67, stdev=1182.71, samples=3 iops : min= 1, max= 4, avg= 2.67, stdev= 1.53, samples=3 lat (msec) : 50=25.00%, 100=25.00%, 250=33.33%, 500=8.33% cpu : usr=0.14%, sys=2.48%, ctx=48, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30781: Thu May 1 19:25:23 2025 read: IOPS=1, BW=1493KiB/s (1529kB/s)(5120KiB/3429msec) clat (usec): min=48006, max=81281, avg=57466.24, stdev=13809.43 lat (usec): min=48017, max=81283, avg=57469.77, stdev=13807.91 clat percentiles (usec): | 1.00th=[47973], 5.00th=[47973], 10.00th=[47973], 20.00th=[47973], | 30.00th=[48497], 40.00th=[48497], 50.00th=[52167], 60.00th=[52167], | 70.00th=[57410], 80.00th=[57410], 90.00th=[81265], 95.00th=[81265], | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 8126, max= 8126, per=100.00%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2090KiB/s (2141kB/s)(7168KiB/3429msec) clat (msec): min=97, max=2254, avg=439.15, stdev=800.59 lat (msec): min=97, max=2254, avg=439.32, stdev=800.81 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 97], 10.00th=[ 97], 20.00th=[ 124], | 30.00th=[ 138], 40.00th=[ 138], 50.00th=[ 142], 60.00th=[ 159], | 70.00th=[ 159], 80.00th=[ 163], 90.00th=[ 2265], 95.00th=[ 2265], | 99.00th=[ 2265], 99.50th=[ 2265], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 2039, max= 4063, per=37.91%, avg=2716.67, stdev=1165.97, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 lat (msec) : 50=16.67%, 100=33.33%, 250=41.67% cpu : usr=0.06%, sys=3.18%, ctx=52, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5119KiB/s (5242kB/s), 512KiB/s-1891KiB/s (524kB/s-1936kB/s), io=20.0MiB (20.0MB), run=3429-4001msec WRITE: bw=7166KiB/s (7338kB/s), 1351KiB/s-2559KiB/s (1383kB/s-2621kB/s), io=28.0MiB (29.4MB), run=3429-4001msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30790: Thu May 1 19:25:26 2025 read: IOPS=4, BW=19.2MiB/s (20.1MB/s)(12.0MiB/625msec) clat (msec): min=57, max=387, avg=200.79, stdev=168.82 lat (msec): min=57, max=387, avg=200.79, stdev=168.82 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 58], 10.00th=[ 58], 20.00th=[ 58], | 30.00th=[ 58], 40.00th=[ 159], 50.00th=[ 159], 60.00th=[ 159], | 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min=16384, max=16384, per=93.90%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.32%, sys=11.86%, ctx=45, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30791: Thu May 1 19:25:26 2025 read: IOPS=1, BW=4362KiB/s (4467kB/s)(4096KiB/939msec) clat (nsec): min=184537k, max=184537k, avg=184536514.00, stdev= 0.00 lat (nsec): min=184540k, max=184540k, avg=184539622.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 184], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 8192, max= 8192, per=46.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=8724KiB/s (8934kB/s)(8192KiB/939msec) clat (msec): min=303, max=427, avg=365.90, stdev=87.62 lat (msec): min=304, max=428, avg=366.57, stdev=87.24 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 305], 10.00th=[ 305], 20.00th=[ 305], | 30.00th=[ 305], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 430], | 70.00th=[ 430], 80.00th=[ 430], 90.00th=[ 430], 95.00th=[ 430], | 99.00th=[ 430], 99.50th=[ 430], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=12.05%, ctx=62, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30792: Thu May 1 19:25:26 2025 write: IOPS=2, BW=9029KiB/s (9245kB/s)(12.0MiB/1361msec) clat (msec): min=235, max=567, avg=434.76, stdev=175.33 lat (msec): min=236, max=567, avg=435.23, stdev=175.35 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 236], 10.00th=[ 236], 20.00th=[ 236], | 30.00th=[ 236], 40.00th=[ 502], 50.00th=[ 502], 60.00th=[ 502], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 8192, max= 8192, per=34.03%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=66.67% cpu : usr=0.00%, sys=8.38%, ctx=54, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30793: Thu May 1 19:25:26 2025 write: IOPS=2, BW=10.0MiB/s (11.5MB/s)(12.0MiB/1095msec) clat (msec): min=224, max=588, avg=350.07, stdev=206.76 lat (msec): min=224, max=589, avg=350.51, stdev=206.78 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 224], 10.00th=[ 224], 20.00th=[ 224], | 30.00th=[ 224], 40.00th=[ 239], 50.00th=[ 239], 60.00th=[ 239], | 70.00th=[ 592], 80.00th=[ 592], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 8143, max= 8192, per=33.92%, avg=8167.50, stdev=34.65, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.37%, sys=8.41%, ctx=47, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.0MiB/s (17.9MB/s), 4362KiB/s-19.2MiB/s (4467kB/s-20.1MB/s), io=16.0MiB (16.8MB), run=625-939msec WRITE: bw=23.5MiB/s (24.7MB/s), 8724KiB/s-10.0MiB/s (8934kB/s-11.5MB/s), io=32.0MiB (33.6MB), run=939-1361msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30795: Thu May 1 19:25:26 2025 read: IOPS=4, BW=17.1MiB/s (17.0MB/s)(12.0MiB/700msec) clat (msec): min=91, max=315, avg=224.83, stdev=117.67 lat (msec): min=91, max=315, avg=224.83, stdev=117.67 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 92], 10.00th=[ 92], 20.00th=[ 92], | 30.00th=[ 92], 40.00th=[ 268], 50.00th=[ 268], 60.00th=[ 268], | 70.00th=[ 317], 80.00th=[ 317], 90.00th=[ 317], 95.00th=[ 317], | 99.00th=[ 317], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 8159, max= 8159, per=69.87%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=1.14%, sys=17.88%, ctx=492, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30796: Thu May 1 19:25:26 2025 read: IOPS=0, BW=2919KiB/s (2990kB/s)(4096KiB/1403msec) clat (nsec): min=214698k, max=214698k, avg=214698111.00, stdev= 0.00 lat (nsec): min=214700k, max=214700k, avg=214700447.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 215], 10.00th=[ 215], 20.00th=[ 215], | 30.00th=[ 215], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 215], | 70.00th=[ 215], 80.00th=[ 215], 90.00th=[ 215], 95.00th=[ 215], | 99.00th=[ 215], 99.50th=[ 215], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 8192, max= 8192, per=70.16%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5839KiB/s (5979kB/s)(8192KiB/1403msec) clat (msec): min=524, max=589, avg=556.93, stdev=45.53 lat (msec): min=525, max=589, avg=557.65, stdev=45.11 clat percentiles (msec): | 1.00th=[ 527], 5.00th=[ 527], 10.00th=[ 527], 20.00th=[ 527], | 30.00th=[ 527], 40.00th=[ 527], 50.00th=[ 527], 60.00th=[ 592], | 70.00th=[ 592], 80.00th=[ 592], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 8192, max= 8192, per=36.88%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=66.67% cpu : usr=0.07%, sys=13.20%, ctx=238, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30797: Thu May 1 19:25:26 2025 write: IOPS=2, BW=9423KiB/s (9649kB/s)(12.0MiB/1304msec) clat (msec): min=204, max=599, avg=424.08, stdev=201.41 lat (msec): min=206, max=599, avg=425.29, stdev=200.40 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 205], 10.00th=[ 205], 20.00th=[ 205], | 30.00th=[ 205], 40.00th=[ 468], 50.00th=[ 468], 60.00th=[ 468], | 70.00th=[ 600], 80.00th=[ 600], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 8192, max= 8192, per=36.88%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=23.18%, ctx=102, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30798: Thu May 1 19:25:26 2025 write: IOPS=2, BW=8331KiB/s (8531kB/s)(12.0MiB/1475msec) clat (msec): min=206, max=639, avg=481.06, stdev=238.72 lat (msec): min=206, max=639, avg=481.50, stdev=238.76 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 207], 10.00th=[ 207], 20.00th=[ 207], | 30.00th=[ 207], 40.00th=[ 600], 50.00th=[ 600], 60.00th=[ 600], | 70.00th=[ 642], 80.00th=[ 642], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 8192, max= 8192, per=36.88%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=66.67% cpu : usr=0.07%, sys=17.16%, ctx=71, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.4MiB/s (11.0MB/s), 2919KiB/s-17.1MiB/s (2990kB/s-17.0MB/s), io=16.0MiB (16.8MB), run=700-1403msec WRITE: bw=21.7MiB/s (22.7MB/s), 5839KiB/s-9423KiB/s (5979kB/s-9649kB/s), io=32.0MiB (33.6MB), run=1304-1475msec