== sanity test 398b: DIO and buffer IO race ============== 23:03:31 (1746155011) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.9241 s, 26.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22762: Thu May 1 23:04:09 2025 read: IOPS=43, BW=175KiB/s (180kB/s)(5992KiB/34160msec) clat (usec): min=61, max=302776, avg=13030.28, stdev=11323.38 lat (usec): min=62, max=302777, avg=13033.97, stdev=11321.25 clat percentiles (usec): | 1.00th=[ 98], 5.00th=[ 130], 10.00th=[ 5014], 20.00th=[ 6652], | 30.00th=[ 8225], 40.00th=[ 9634], 50.00th=[ 11076], 60.00th=[ 12780], | 70.00th=[ 15008], 80.00th=[ 18220], 90.00th=[ 23200], 95.00th=[ 28443], | 99.00th=[ 43254], 99.50th=[ 48497], 99.90th=[ 95945], 99.95th=[304088], | 99.99th=[304088] bw ( KiB/s): min= 48, max= 288, per=24.84%, avg=174.38, stdev=43.07, samples=68 iops : min= 12, max= 72, avg=43.28, stdev=10.84, samples=68 write: IOPS=46, BW=184KiB/s (189kB/s)(6296KiB/34160msec) clat (usec): min=484, max=73948, avg=9189.09, stdev=9773.23 lat (usec): min=485, max=73949, avg=9190.90, stdev=9772.97 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 922], | 30.00th=[ 1598], 40.00th=[ 3326], 50.00th=[ 6259], 60.00th=[ 8717], | 70.00th=[11994], 80.00th=[15926], 90.00th=[22938], 95.00th=[28705], | 99.00th=[41157], 99.50th=[51643], 99.90th=[55837], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 32, max= 304, per=24.94%, avg=183.28, stdev=53.73, samples=68 iops : min= 8, max= 76, avg=45.51, stdev=13.37, samples=68 lat (usec) : 100=0.65%, 250=2.05%, 500=0.13%, 750=3.42%, 1000=8.07% lat (msec) : 2=5.50%, 4=5.44%, 10=28.26%, 20=32.13%, 50=13.93% lat (msec) : 100=0.39%, 500=0.03% cpu : usr=0.15%, sys=12.84%, ctx=5162, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22763: Thu May 1 23:04:09 2025 read: IOPS=44, BW=177KiB/s (181kB/s)(5940KiB/33579msec) clat (usec): min=65, max=304963, avg=12399.49, stdev=11383.80 lat (usec): min=66, max=304964, avg=12401.50, stdev=11383.43 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[ 126], 10.00th=[ 4555], 20.00th=[ 6325], | 30.00th=[ 7767], 40.00th=[ 8979], 50.00th=[ 10552], 60.00th=[ 12256], | 70.00th=[ 14484], 80.00th=[ 17433], 90.00th=[ 21627], 95.00th=[ 27132], | 99.00th=[ 40109], 99.50th=[ 47449], 99.90th=[120062], 99.95th=[304088], | 99.99th=[304088] bw ( KiB/s): min= 39, max= 344, per=25.08%, avg=176.03, stdev=49.38, samples=67 iops : min= 9, max= 86, avg=43.70, stdev=12.43, samples=67 write: IOPS=47, BW=189KiB/s (194kB/s)(6348KiB/33579msec) clat (usec): min=500, max=89699, avg=9447.57, stdev=10144.92 lat (usec): min=501, max=89700, avg=9450.51, stdev=10145.48 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 971], | 30.00th=[ 1663], 40.00th=[ 3523], 50.00th=[ 6456], 60.00th=[ 9241], | 70.00th=[12256], 80.00th=[16188], 90.00th=[23200], 95.00th=[30016], | 99.00th=[43779], 99.50th=[47449], 99.90th=[61080], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 63, max= 296, per=25.54%, avg=187.73, stdev=54.39, samples=67 iops : min= 15, max= 74, avg=46.63, stdev=13.65, samples=67 lat (usec) : 100=0.94%, 250=1.89%, 500=0.13%, 750=3.09%, 1000=7.71% lat (msec) : 2=6.22%, 4=5.70%, 10=29.33%, 20=31.35%, 50=13.18% lat (msec) : 100=0.39%, 250=0.03%, 500=0.03% cpu : usr=0.24%, sys=12.76%, ctx=5064, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22764: Thu May 1 23:04:09 2025 read: IOPS=44, BW=180KiB/s (184kB/s)(6028KiB/33538msec) clat (usec): min=73, max=107075, avg=12239.91, stdev=8532.01 lat (usec): min=73, max=107076, avg=12241.46, stdev=8532.48 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 135], 10.00th=[ 4490], 20.00th=[ 6259], | 30.00th=[ 7701], 40.00th=[ 9110], 50.00th=[ 10552], 60.00th=[ 12387], | 70.00th=[ 14484], 80.00th=[ 17433], 90.00th=[ 21890], 95.00th=[ 26870], | 99.00th=[ 40109], 99.50th=[ 49546], 99.90th=[ 80217], 99.95th=[107480], | 99.99th=[107480] bw ( KiB/s): min= 63, max= 296, per=25.53%, avg=179.25, stdev=49.53, samples=67 iops : min= 15, max= 74, avg=44.51, stdev=12.40, samples=67 write: IOPS=46, BW=187KiB/s (191kB/s)(6260KiB/33538msec) clat (usec): min=494, max=271978, avg=9542.99, stdev=12217.16 lat (usec): min=494, max=271980, avg=9545.22, stdev=12217.03 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 742], 10.00th=[ 799], 20.00th=[ 947], | 30.00th=[ 1532], 40.00th=[ 3064], 50.00th=[ 6652], 60.00th=[ 8979], | 70.00th=[ 11863], 80.00th=[ 16057], 90.00th=[ 23462], 95.00th=[ 30278], | 99.00th=[ 46400], 99.50th=[ 52691], 99.90th=[ 69731], 99.95th=[270533], | 99.99th=[270533] bw ( KiB/s): min= 88, max= 327, per=25.33%, avg=186.16, stdev=47.66, samples=67 iops : min= 22, max= 81, avg=46.24, stdev=11.93, samples=67 lat (usec) : 100=0.88%, 250=2.15%, 500=0.13%, 750=2.80%, 1000=8.37% lat (msec) : 2=6.18%, 4=5.44%, 10=29.39%, 20=30.83%, 50=13.31% lat (msec) : 100=0.46%, 250=0.03%, 500=0.03% cpu : usr=0.19%, sys=12.99%, ctx=5100, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22765: Thu May 1 23:04:09 2025 read: IOPS=44, BW=178KiB/s (183kB/s)(6052KiB/33952msec) clat (usec): min=62, max=91686, avg=12536.15, stdev=8764.83 lat (usec): min=62, max=91687, avg=12537.51, stdev=8764.64 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 126], 10.00th=[ 4621], 20.00th=[ 6456], | 30.00th=[ 7767], 40.00th=[ 9241], 50.00th=[10945], 60.00th=[12780], | 70.00th=[14746], 80.00th=[17695], 90.00th=[22152], 95.00th=[27395], | 99.00th=[44303], 99.50th=[54264], 99.90th=[74974], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 110, max= 280, per=25.61%, avg=179.79, stdev=45.16, samples=66 iops : min= 27, max= 70, avg=44.65, stdev=11.30, samples=66 write: IOPS=45, BW=184KiB/s (188kB/s)(6236KiB/33952msec) clat (usec): min=475, max=284806, avg=9469.68, stdev=12136.46 lat (usec): min=476, max=284809, avg=9471.04, stdev=12136.49 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 938], | 30.00th=[ 1663], 40.00th=[ 3130], 50.00th=[ 6325], 60.00th=[ 9110], | 70.00th=[ 11600], 80.00th=[ 15795], 90.00th=[ 23200], 95.00th=[ 30802], | 99.00th=[ 42730], 99.50th=[ 46400], 99.90th=[ 71828], 99.95th=[283116], | 99.99th=[283116] bw ( KiB/s): min= 15, max= 304, per=24.89%, avg=182.96, stdev=53.20, samples=67 iops : min= 3, max= 76, avg=45.43, stdev=13.36, samples=67 lat (usec) : 100=0.85%, 250=2.21%, 500=0.23%, 750=2.86%, 1000=8.27% lat (msec) : 2=5.89%, 4=5.18%, 10=28.81%, 20=31.67%, 50=13.57% lat (msec) : 100=0.42%, 500=0.03% cpu : usr=0.22%, sys=12.73%, ctx=5037, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=703KiB/s (720kB/s), 175KiB/s-180KiB/s (180kB/s-184kB/s), io=23.4MiB (24.6MB), run=33538-34160msec WRITE: bw=736KiB/s (754kB/s), 184KiB/s-189KiB/s (188kB/s-194kB/s), io=24.6MiB (25.7MB), run=33538-34160msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22758: Thu May 1 23:04:29 2025 read: IOPS=27, BW=112KiB/s (115kB/s)(5992KiB/53553msec) clat (msec): min=2, max=317, avg=13.45, stdev=13.91 lat (msec): min=2, max=317, avg=13.45, stdev=13.90 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 14], 80.00th=[ 18], 90.00th=[ 27], 95.00th=[ 36], | 99.00th=[ 61], 99.50th=[ 70], 99.90th=[ 148], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 24, max= 216, per=25.02%, avg=111.58, stdev=47.36, samples=107 iops : min= 6, max= 54, avg=27.59, stdev=11.85, samples=107 write: IOPS=29, BW=118KiB/s (120kB/s)(6296KiB/53553msec) clat (usec): min=5380, max=95270, avg=21109.03, stdev=11587.31 lat (usec): min=5381, max=95271, avg=21110.46, stdev=11587.25 clat percentiles (usec): | 1.00th=[ 7504], 5.00th=[ 9503], 10.00th=[10552], 20.00th=[12649], | 30.00th=[14222], 40.00th=[15926], 50.00th=[17957], 60.00th=[20317], | 70.00th=[23462], 80.00th=[27919], 90.00th=[34866], 95.00th=[43779], | 99.00th=[68682], 99.50th=[74974], 99.90th=[91751], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 31, max= 208, per=25.30%, avg=118.15, stdev=46.36, samples=106 iops : min= 7, max= 52, avg=29.23, stdev=11.61, samples=106 lat (msec) : 4=1.04%, 10=29.04%, 20=40.85%, 50=26.73%, 100=2.28% lat (msec) : 250=0.03%, 500=0.03% cpu : usr=0.06%, sys=5.76%, ctx=3373, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22759: Thu May 1 23:04:29 2025 read: IOPS=27, BW=111KiB/s (113kB/s)(5940KiB/53724msec) clat (msec): min=2, max=296, avg=13.13, stdev=13.12 lat (msec): min=2, max=296, avg=13.13, stdev=13.12 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 17], 90.00th=[ 26], 95.00th=[ 36], | 99.00th=[ 59], 99.50th=[ 71], 99.90th=[ 94], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 8, max= 256, per=24.65%, avg=109.93, stdev=48.70, samples=107 iops : min= 2, max= 64, avg=27.22, stdev=12.19, samples=107 write: IOPS=29, BW=118KiB/s (121kB/s)(6348KiB/53724msec) clat (usec): min=5295, max=99531, avg=21434.72, stdev=11359.18 lat (usec): min=5297, max=99532, avg=21437.09, stdev=11359.59 clat percentiles (usec): | 1.00th=[ 7177], 5.00th=[ 9896], 10.00th=[11207], 20.00th=[13042], | 30.00th=[14484], 40.00th=[16057], 50.00th=[17957], 60.00th=[20055], | 70.00th=[23200], 80.00th=[28967], 90.00th=[36963], 95.00th=[43779], | 99.00th=[64750], 99.50th=[70779], 99.90th=[73925], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 8, max= 216, per=25.08%, avg=117.10, stdev=44.38, samples=107 iops : min= 2, max= 54, avg=29.02, stdev=11.14, samples=107 lat (msec) : 4=1.01%, 10=27.86%, 20=43.07%, 50=25.65%, 100=2.38% lat (msec) : 500=0.03% cpu : usr=0.15%, sys=5.62%, ctx=3431, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22760: Thu May 1 23:04:29 2025 read: IOPS=28, BW=113KiB/s (116kB/s)(6028KiB/53188msec) clat (msec): min=2, max=113, avg=12.89, stdev=10.86 lat (msec): min=2, max=113, avg=12.89, stdev=10.86 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 17], 90.00th=[ 27], 95.00th=[ 35], | 99.00th=[ 54], 99.50th=[ 65], 99.90th=[ 102], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 31, max= 240, per=25.55%, avg=113.95, stdev=47.81, samples=105 iops : min= 7, max= 60, avg=28.18, stdev=11.99, samples=105 write: IOPS=29, BW=118KiB/s (121kB/s)(6260KiB/53188msec) clat (msec): min=5, max=321, avg=21.45, stdev=13.99 lat (msec): min=5, max=321, avg=21.45, stdev=13.99 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 21], | 70.00th=[ 24], 80.00th=[ 29], 90.00th=[ 38], 95.00th=[ 46], | 99.00th=[ 63], 99.50th=[ 73], 99.90th=[ 93], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 15, max= 216, per=25.06%, avg=117.01, stdev=47.32, samples=106 iops : min= 3, max= 54, avg=28.94, stdev=11.86, samples=106 lat (msec) : 4=0.88%, 10=30.50%, 20=40.30%, 50=25.88%, 100=2.34% lat (msec) : 250=0.07%, 500=0.03% cpu : usr=0.10%, sys=5.73%, ctx=3388, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22761: Thu May 1 23:04:29 2025 read: IOPS=28, BW=113KiB/s (116kB/s)(6052KiB/53419msec) clat (msec): min=2, max=113, avg=13.04, stdev=11.13 lat (msec): min=2, max=113, avg=13.04, stdev=11.13 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 14], 80.00th=[ 17], 90.00th=[ 26], 95.00th=[ 35], | 99.00th=[ 61], 99.50th=[ 74], 99.90th=[ 102], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 23, max= 239, per=25.41%, avg=113.33, stdev=47.62, samples=105 iops : min= 5, max= 59, avg=28.09, stdev=11.92, samples=105 write: IOPS=29, BW=117KiB/s (120kB/s)(6236KiB/53419msec) clat (msec): min=4, max=282, avg=21.46, stdev=13.17 lat (msec): min=4, max=282, avg=21.46, stdev=13.17 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 21], | 70.00th=[ 24], 80.00th=[ 29], 90.00th=[ 36], 95.00th=[ 45], | 99.00th=[ 61], 99.50th=[ 77], 99.90th=[ 100], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 16, max= 216, per=24.85%, avg=116.04, stdev=43.85, samples=106 iops : min= 4, max= 54, avg=28.76, stdev=10.99, samples=106 lat (msec) : 4=0.88%, 10=28.39%, 20=42.15%, 50=26.27%, 100=2.21% lat (msec) : 250=0.07%, 500=0.03% cpu : usr=0.09%, sys=5.68%, ctx=3397, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=447KiB/s (458kB/s), 111KiB/s-113KiB/s (113kB/s-116kB/s), io=23.4MiB (24.6MB), run=53188-53724msec WRITE: bw=468KiB/s (479kB/s), 117KiB/s-118KiB/s (120kB/s-121kB/s), io=24.6MiB (25.7MB), run=53188-53724msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22828: Thu May 1 23:04:40 2025 read: IOPS=38, BW=612KiB/s (626kB/s)(5920KiB/9681msec) clat (usec): min=79, max=71384, avg=14179.06, stdev=10653.90 lat (usec): min=80, max=71386, avg=14180.26, stdev=10654.00 clat percentiles (usec): | 1.00th=[ 97], 5.00th=[ 116], 10.00th=[ 139], 20.00th=[ 6325], | 30.00th=[ 8848], 40.00th=[10683], 50.00th=[12649], 60.00th=[14877], | 70.00th=[17171], 80.00th=[19792], 90.00th=[27657], 95.00th=[34341], | 99.00th=[49546], 99.50th=[63177], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 415, max= 1056, per=25.41%, avg=606.47, stdev=168.12, samples=19 iops : min= 25, max= 66, avg=37.37, stdev=10.59, samples=19 write: IOPS=41, BW=658KiB/s (674kB/s)(6368KiB/9681msec) clat (usec): min=681, max=70503, avg=10991.78, stdev=11604.48 lat (usec): min=682, max=70504, avg=10993.63, stdev=11604.60 clat percentiles (usec): | 1.00th=[ 955], 5.00th=[ 1074], 10.00th=[ 1188], 20.00th=[ 1450], | 30.00th=[ 2409], 40.00th=[ 4293], 50.00th=[ 7242], 60.00th=[ 9634], | 70.00th=[13304], 80.00th=[20579], 90.00th=[27919], 95.00th=[33162], | 99.00th=[50594], 99.50th=[56886], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 415, max= 992, per=26.16%, avg=656.89, stdev=170.24, samples=19 iops : min= 25, max= 62, avg=40.53, stdev=10.72, samples=19 lat (usec) : 100=0.65%, 250=4.95%, 500=0.39%, 750=0.13%, 1000=0.91% lat (msec) : 2=13.80%, 4=5.60%, 10=22.79%, 20=30.60%, 50=19.27% lat (msec) : 100=0.91% cpu : usr=0.13%, sys=12.92%, ctx=1406, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22829: Thu May 1 23:04:40 2025 read: IOPS=35, BW=572KiB/s (586kB/s)(5744KiB/10034msec) clat (usec): min=75, max=63530, avg=14070.75, stdev=9502.47 lat (usec): min=75, max=63531, avg=14071.96, stdev=9502.53 clat percentiles (usec): | 1.00th=[ 98], 5.00th=[ 117], 10.00th=[ 145], 20.00th=[ 6980], | 30.00th=[ 9241], 40.00th=[11338], 50.00th=[13173], 60.00th=[15664], | 70.00th=[17695], 80.00th=[20841], 90.00th=[25035], 95.00th=[31589], | 99.00th=[45351], 99.50th=[49546], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 191, max= 800, per=23.53%, avg=561.63, stdev=175.96, samples=19 iops : min= 11, max= 50, avg=34.74, stdev=11.01, samples=19 write: IOPS=40, BW=652KiB/s (668kB/s)(6544KiB/10034msec) clat (usec): min=880, max=70365, avg=11663.02, stdev=11631.54 lat (usec): min=882, max=70368, avg=11664.82, stdev=11631.68 clat percentiles (usec): | 1.00th=[ 1012], 5.00th=[ 1123], 10.00th=[ 1205], 20.00th=[ 1483], | 30.00th=[ 2474], 40.00th=[ 5145], 50.00th=[ 8455], 60.00th=[10945], | 70.00th=[14746], 80.00th=[20841], 90.00th=[29230], 95.00th=[34341], | 99.00th=[47449], 99.50th=[55837], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 384, max= 1306, per=25.11%, avg=630.47, stdev=195.78, samples=19 iops : min= 24, max= 81, avg=39.05, stdev=12.25, samples=19 lat (usec) : 100=0.65%, 250=4.95%, 500=0.39%, 750=0.13%, 1000=0.39% lat (msec) : 2=13.67%, 4=5.99%, 10=19.53%, 20=32.68%, 50=21.09% lat (msec) : 100=0.52% cpu : usr=0.11%, sys=12.97%, ctx=1354, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22831: Thu May 1 23:04:40 2025 read: IOPS=40, BW=646KiB/s (662kB/s)(6256KiB/9681msec) clat (usec): min=55, max=76745, avg=14147.52, stdev=10502.80 lat (usec): min=56, max=76747, avg=14149.79, stdev=10502.81 clat percentiles (usec): | 1.00th=[ 97], 5.00th=[ 121], 10.00th=[ 178], 20.00th=[ 7177], | 30.00th=[ 8848], 40.00th=[10552], 50.00th=[12256], 60.00th=[14484], | 70.00th=[17695], 80.00th=[19792], 90.00th=[26084], 95.00th=[32900], | 99.00th=[56361], 99.50th=[63701], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 320, max= 1213, per=26.91%, avg=642.37, stdev=212.01, samples=19 iops : min= 20, max= 75, avg=39.79, stdev=13.20, samples=19 write: IOPS=38, BW=623KiB/s (638kB/s)(6032KiB/9681msec) clat (usec): min=780, max=77378, avg=10584.62, stdev=11233.31 lat (usec): min=782, max=77380, avg=10587.64, stdev=11234.01 clat percentiles (usec): | 1.00th=[ 857], 5.00th=[ 1057], 10.00th=[ 1172], 20.00th=[ 1467], | 30.00th=[ 2245], 40.00th=[ 3982], 50.00th=[ 6915], 60.00th=[ 9765], | 70.00th=[14222], 80.00th=[18482], 90.00th=[25822], 95.00th=[30802], | 99.00th=[53216], 99.50th=[66847], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 320, max= 1024, per=24.24%, avg=608.63, stdev=204.86, samples=19 iops : min= 20, max= 64, avg=37.68, stdev=12.77, samples=19 lat (usec) : 100=0.78%, 250=4.43%, 500=0.39%, 750=0.13%, 1000=1.56% lat (msec) : 2=11.98%, 4=6.77%, 10=22.27%, 20=33.59%, 50=16.93% lat (msec) : 100=1.17% cpu : usr=0.11%, sys=13.06%, ctx=1420, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22833: Thu May 1 23:04:40 2025 read: IOPS=39, BW=627KiB/s (642kB/s)(6032KiB/9623msec) clat (usec): min=67, max=54689, avg=13535.58, stdev=9977.09 lat (usec): min=67, max=54691, avg=13536.75, stdev=9977.09 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[ 118], 10.00th=[ 133], 20.00th=[ 5866], | 30.00th=[ 8291], 40.00th=[10159], 50.00th=[12256], 60.00th=[14353], | 70.00th=[16319], 80.00th=[20317], 90.00th=[26870], 95.00th=[32900], | 99.00th=[45351], 99.50th=[51643], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 224, max= 862, per=25.92%, avg=618.74, stdev=152.82, samples=19 iops : min= 14, max= 53, avg=38.26, stdev= 9.36, samples=19 write: IOPS=40, BW=650KiB/s (666kB/s)(6256KiB/9623msec) clat (usec): min=803, max=55692, avg=10892.23, stdev=11417.29 lat (usec): min=805, max=55694, avg=10894.14, stdev=11417.25 clat percentiles (usec): | 1.00th=[ 840], 5.00th=[ 1090], 10.00th=[ 1172], 20.00th=[ 1467], | 30.00th=[ 2278], 40.00th=[ 4015], 50.00th=[ 7046], 60.00th=[ 9896], | 70.00th=[14222], 80.00th=[19006], 90.00th=[26608], 95.00th=[34866], | 99.00th=[47973], 99.50th=[55837], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 224, max= 1056, per=25.58%, avg=642.32, stdev=205.14, samples=19 iops : min= 14, max= 66, avg=39.74, stdev=12.84, samples=19 lat (usec) : 100=1.17%, 250=5.21%, 500=0.39%, 750=0.13%, 1000=1.43% lat (msec) : 2=12.89%, 4=6.64%, 10=21.87%, 20=30.34%, 50=19.27% lat (msec) : 100=0.65% cpu : usr=0.19%, sys=12.43%, ctx=1332, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2387KiB/s (2444kB/s), 572KiB/s-646KiB/s (586kB/s-662kB/s), io=23.4MiB (24.5MB), run=9623-10034msec WRITE: bw=2511KiB/s (2572kB/s), 623KiB/s-658KiB/s (638kB/s-674kB/s), io=24.6MiB (25.8MB), run=9623-10034msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22830: Thu May 1 23:04:43 2025 read: IOPS=29, BW=471KiB/s (483kB/s)(5920KiB/12557msec) clat (usec): min=2633, max=94265, avg=12112.66, stdev=12835.74 lat (usec): min=2635, max=94266, avg=12113.92, stdev=12835.72 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3228], 10.00th=[ 3654], 20.00th=[ 4359], | 30.00th=[ 5014], 40.00th=[ 5932], 50.00th=[ 7111], 60.00th=[ 9241], | 70.00th=[11994], 80.00th=[16319], 90.00th=[28705], 95.00th=[36963], | 99.00th=[61604], 99.50th=[89654], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 192, max= 1344, per=24.84%, avg=470.68, stdev=337.42, samples=25 iops : min= 12, max= 84, avg=29.24, stdev=21.04, samples=25 write: IOPS=31, BW=507KiB/s (519kB/s)(6368KiB/12557msec) clat (msec): min=3, max=106, avg=20.03, stdev=16.08 lat (msec): min=3, max=106, avg=20.03, stdev=16.08 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 15], 60.00th=[ 19], | 70.00th=[ 24], 80.00th=[ 30], 90.00th=[ 42], 95.00th=[ 53], | 99.00th=[ 83], 99.50th=[ 99], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 192, max= 1248, per=25.34%, avg=505.24, stdev=360.12, samples=25 iops : min= 12, max= 78, avg=31.40, stdev=22.46, samples=25 lat (msec) : 4=7.16%, 10=42.32%, 20=24.87%, 50=20.83%, 100=4.69% lat (msec) : 250=0.13% cpu : usr=0.03%, sys=5.93%, ctx=872, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22832: Thu May 1 23:04:43 2025 read: IOPS=28, BW=458KiB/s (469kB/s)(5744KiB/12534msec) clat (usec): min=2530, max=99220, avg=12525.02, stdev=13278.96 lat (usec): min=2530, max=99221, avg=12526.26, stdev=13278.99 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3163], 10.00th=[ 3523], 20.00th=[ 4113], | 30.00th=[ 4883], 40.00th=[ 5669], 50.00th=[ 7111], 60.00th=[ 9110], | 70.00th=[12780], 80.00th=[18220], 90.00th=[29492], 95.00th=[43254], | 99.00th=[66847], 99.50th=[77071], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 160, max= 1440, per=24.08%, avg=456.40, stdev=335.21, samples=25 iops : min= 10, max= 90, avg=28.12, stdev=21.12, samples=25 write: IOPS=32, BW=522KiB/s (535kB/s)(6544KiB/12534msec) clat (usec): min=4207, max=85060, avg=19319.45, stdev=14084.18 lat (usec): min=4209, max=85062, avg=19321.89, stdev=14084.05 clat percentiles (usec): | 1.00th=[ 4752], 5.00th=[ 5932], 10.00th=[ 6783], 20.00th=[ 7832], | 30.00th=[ 9372], 40.00th=[10814], 50.00th=[14222], 60.00th=[18744], | 70.00th=[24249], 80.00th=[30016], 90.00th=[40633], 95.00th=[45876], | 99.00th=[63701], 99.50th=[69731], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 159, max= 1408, per=26.09%, avg=520.32, stdev=367.10, samples=25 iops : min= 9, max= 88, avg=32.12, stdev=23.07, samples=25 lat (msec) : 4=7.94%, 10=40.89%, 20=23.18%, 50=24.61%, 100=3.39% cpu : usr=0.07%, sys=6.07%, ctx=878, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22834: Thu May 1 23:04:43 2025 read: IOPS=30, BW=495KiB/s (507kB/s)(6256KiB/12635msec) clat (msec): min=2, max=145, avg=12.63, stdev=14.76 lat (msec): min=2, max=145, avg=12.63, stdev=14.76 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 9], | 70.00th=[ 13], 80.00th=[ 19], 90.00th=[ 31], 95.00th=[ 43], | 99.00th=[ 66], 99.50th=[ 94], 99.90th=[ 146], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 191, max= 1568, per=25.96%, avg=491.96, stdev=395.34, samples=25 iops : min= 11, max= 98, avg=30.24, stdev=24.93, samples=25 write: IOPS=29, BW=477KiB/s (489kB/s)(6032KiB/12635msec) clat (msec): min=3, max=101, avg=20.30, stdev=15.07 lat (msec): min=3, max=101, avg=20.31, stdev=15.07 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 15], 60.00th=[ 20], | 70.00th=[ 26], 80.00th=[ 32], 90.00th=[ 42], 95.00th=[ 51], | 99.00th=[ 72], 99.50th=[ 79], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 191, max= 1152, per=23.39%, avg=466.36, stdev=325.07, samples=25 iops : min= 11, max= 72, avg=28.64, stdev=20.49, samples=25 lat (msec) : 4=9.64%, 10=41.28%, 20=20.96%, 50=24.09%, 100=3.78% lat (msec) : 250=0.26% cpu : usr=0.17%, sys=5.64%, ctx=867, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22835: Thu May 1 23:04:43 2025 read: IOPS=29, BW=478KiB/s (490kB/s)(6032KiB/12608msec) clat (usec): min=2156, max=99220, avg=12930.17, stdev=14100.61 lat (usec): min=2157, max=99221, avg=12931.40, stdev=14100.63 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 3294], 10.00th=[ 3621], 20.00th=[ 4424], | 30.00th=[ 5080], 40.00th=[ 5866], 50.00th=[ 7308], 60.00th=[ 9372], | 70.00th=[12125], 80.00th=[17695], 90.00th=[31589], 95.00th=[41681], | 99.00th=[74974], 99.50th=[90702], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 192, max= 1376, per=24.91%, avg=471.96, stdev=342.10, samples=25 iops : min= 12, max= 86, avg=29.24, stdev=21.50, samples=25 write: IOPS=31, BW=496KiB/s (508kB/s)(6256KiB/12608msec) clat (msec): min=4, max=106, avg=19.35, stdev=15.75 lat (msec): min=4, max=106, avg=19.35, stdev=15.75 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 17], | 70.00th=[ 24], 80.00th=[ 30], 90.00th=[ 41], 95.00th=[ 52], | 99.00th=[ 74], 99.50th=[ 101], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 159, max= 1344, per=24.70%, avg=492.44, stdev=370.74, samples=25 iops : min= 9, max= 84, avg=30.52, stdev=23.34, samples=25 lat (msec) : 4=7.55%, 10=42.32%, 20=23.83%, 50=22.14%, 100=3.91% lat (msec) : 250=0.26% cpu : usr=0.24%, sys=5.75%, ctx=875, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1896KiB/s (1941kB/s), 458KiB/s-495KiB/s (469kB/s-507kB/s), io=23.4MiB (24.5MB), run=12534-12635msec WRITE: bw=1994KiB/s (2042kB/s), 477KiB/s-522KiB/s (489kB/s-535kB/s), io=24.6MiB (25.8MB), run=12534-12635msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22851: Thu May 1 23:04:45 2025 read: IOPS=9, BW=9.79MiB/s (10.3MB/s)(7168KiB/715msec) clat (usec): min=501, max=86380, avg=34873.62, stdev=39099.97 lat (usec): min=503, max=86382, avg=34875.41, stdev=39099.76 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 502], 10.00th=[ 502], 20.00th=[ 594], | 30.00th=[ 857], 40.00th=[ 857], 50.00th=[14484], 60.00th=[66847], | 70.00th=[66847], 80.00th=[74974], 90.00th=[86508], 95.00th=[86508], | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min=10240, max=10240, per=51.05%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=6, BW=7161KiB/s (7333kB/s)(5120KiB/715msec) clat (usec): min=44535, max=74673, avg=64804.96, stdev=12562.82 lat (usec): min=44603, max=74741, avg=64860.68, stdev=12563.63 clat percentiles (usec): | 1.00th=[44303], 5.00th=[44303], 10.00th=[44303], 20.00th=[44303], | 30.00th=[61604], 40.00th=[61604], 50.00th=[68682], 60.00th=[68682], | 70.00th=[74974], 80.00th=[74974], 90.00th=[74974], 95.00th=[74974], | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 2048, max= 2048, per=7.29%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67%, 1000=8.33% lat (msec) : 20=8.33%, 50=8.33%, 100=58.33% cpu : usr=0.14%, sys=24.93%, ctx=438, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22854: Thu May 1 23:04:45 2025 read: IOPS=1, BW=2006KiB/s (2054kB/s)(2048KiB/1021msec) clat (msec): min=52, max=104, avg=78.55, stdev=36.58 lat (msec): min=52, max=104, avg=78.55, stdev=36.58 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 53], 50.00th=[ 53], 60.00th=[ 105], | 70.00th=[ 105], 80.00th=[ 105], 90.00th=[ 105], 95.00th=[ 105], | 99.00th=[ 105], 99.50th=[ 105], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 2048, max= 2048, per=10.21%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9.79MiB/s (10.3MB/s)(10.0MiB/1021msec) clat (msec): min=39, max=129, avg=70.48, stdev=32.42 lat (msec): min=39, max=129, avg=70.54, stdev=32.41 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 42], | 30.00th=[ 43], 40.00th=[ 53], 50.00th=[ 54], 60.00th=[ 64], | 70.00th=[ 91], 80.00th=[ 91], 90.00th=[ 120], 95.00th=[ 130], | 99.00th=[ 130], 99.50th=[ 130], 99.90th=[ 130], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 4096, max= 4096, per=14.59%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=25.00%, 100=50.00%, 250=25.00% cpu : usr=0.10%, sys=26.76%, ctx=163, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22855: Thu May 1 23:04:45 2025 read: IOPS=9, BW=9862KiB/s (10.1MB/s)(6144KiB/623msec) clat (usec): min=484, max=95572, avg=42154.76, stdev=37063.32 lat (usec): min=484, max=95575, avg=42156.02, stdev=37063.63 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 486], 10.00th=[ 486], 20.00th=[ 627], | 30.00th=[ 627], 40.00th=[39060], 50.00th=[39060], 60.00th=[56886], | 70.00th=[60556], 80.00th=[60556], 90.00th=[95945], 95.00th=[95945], | 99.00th=[95945], 99.50th=[95945], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 8175, max= 8175, per=40.76%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9862KiB/s (10.1MB/s)(6144KiB/623msec) clat (usec): min=24147, max=97273, avg=58913.00, stdev=28842.67 lat (usec): min=24211, max=97291, avg=58977.89, stdev=28831.55 clat percentiles (usec): | 1.00th=[24249], 5.00th=[24249], 10.00th=[24249], 20.00th=[30016], | 30.00th=[30016], 40.00th=[54789], 50.00th=[54789], 60.00th=[64226], | 70.00th=[83362], 80.00th=[83362], 90.00th=[96994], 95.00th=[96994], | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 8175, max= 8175, per=29.11%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=25.00%, 100=58.33% cpu : usr=0.00%, sys=29.74%, ctx=328, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22857: Thu May 1 23:04:45 2025 read: IOPS=7, BW=7420KiB/s (7598kB/s)(5120KiB/690msec) clat (usec): min=455, max=70631, avg=16113.37, stdev=30666.44 lat (usec): min=456, max=70632, avg=16114.49, stdev=30666.59 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 457], 10.00th=[ 457], 20.00th=[ 457], | 30.00th=[ 537], 40.00th=[ 537], 50.00th=[ 562], 60.00th=[ 562], | 70.00th=[ 8356], 80.00th=[ 8356], 90.00th=[70779], 95.00th=[70779], | 99.00th=[70779], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 8175, max= 8175, per=40.76%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.1MiB/s (10.6MB/s)(7168KiB/690msec) clat (msec): min=22, max=117, avg=76.34, stdev=33.81 lat (msec): min=22, max=117, avg=76.40, stdev=33.80 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 51], | 30.00th=[ 58], 40.00th=[ 58], 50.00th=[ 86], 60.00th=[ 101], | 70.00th=[ 101], 80.00th=[ 102], 90.00th=[ 117], 95.00th=[ 117], | 99.00th=[ 117], 99.50th=[ 117], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 8175, max= 8175, per=29.11%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=16.67% lat (msec) : 10=8.33%, 50=16.67%, 100=33.33%, 250=16.67% cpu : usr=0.00%, sys=23.80%, ctx=416, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.6MiB/s (20.5MB/s), 2006KiB/s-9.79MiB/s (2054kB/s-10.3MB/s), io=20.0MiB (20.0MB), run=623-1021msec WRITE: bw=27.4MiB/s (28.8MB/s), 7161KiB/s-10.1MiB/s (7333kB/s-10.6MB/s), io=28.0MiB (29.4MB), run=623-1021msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22852: Thu May 1 23:04:45 2025 read: IOPS=7, BW=7877KiB/s (8066kB/s)(7168KiB/910msec) clat (msec): min=15, max=173, avg=65.36, stdev=57.23 lat (msec): min=15, max=173, avg=65.36, stdev=57.23 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 21], | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 41], 60.00th=[ 58], | 70.00th=[ 58], 80.00th=[ 111], 90.00th=[ 174], 95.00th=[ 174], | 99.00th=[ 174], 99.50th=[ 174], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 6131, max= 6131, per=30.84%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5626KiB/s (5761kB/s)(5120KiB/910msec) clat (msec): min=23, max=115, avg=59.42, stdev=37.52 lat (msec): min=23, max=115, avg=59.49, stdev=37.51 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 24], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 52], 60.00th=[ 52], | 70.00th=[ 77], 80.00th=[ 77], 90.00th=[ 116], 95.00th=[ 116], | 99.00th=[ 116], 99.50th=[ 116], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 2043, max= 2043, per=7.34%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=25.00%, 250=25.00% cpu : usr=0.00%, sys=6.27%, ctx=40, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22853: Thu May 1 23:04:45 2025 read: IOPS=2, BW=2090KiB/s (2140kB/s)(2048KiB/980msec) clat (usec): min=25402, max=93180, avg=59291.59, stdev=47925.84 lat (usec): min=25404, max=93182, avg=59293.56, stdev=47926.66 clat percentiles (usec): | 1.00th=[25297], 5.00th=[25297], 10.00th=[25297], 20.00th=[25297], | 30.00th=[25297], 40.00th=[25297], 50.00th=[25297], 60.00th=[92799], | 70.00th=[92799], 80.00th=[92799], 90.00th=[92799], 95.00th=[92799], | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 2048, max= 2048, per=10.30%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.2MiB/s (10.7MB/s)(10.0MiB/980msec) clat (msec): min=18, max=167, avg=72.56, stdev=46.97 lat (msec): min=18, max=167, avg=72.65, stdev=46.96 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 30], | 30.00th=[ 36], 40.00th=[ 40], 50.00th=[ 59], 60.00th=[ 61], | 70.00th=[ 108], 80.00th=[ 108], 90.00th=[ 113], 95.00th=[ 167], | 99.00th=[ 167], 99.50th=[ 167], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 4096, max= 4096, per=14.71%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=33.33%, 250=25.00% cpu : usr=0.10%, sys=6.74%, ctx=48, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22856: Thu May 1 23:04:45 2025 read: IOPS=6, BW=6911KiB/s (7077kB/s)(6144KiB/889msec) clat (msec): min=19, max=106, avg=52.55, stdev=39.61 lat (msec): min=19, max=106, avg=52.55, stdev=39.61 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 23], | 30.00th=[ 23], 40.00th=[ 24], 50.00th=[ 24], 60.00th=[ 46], | 70.00th=[ 99], 80.00th=[ 99], 90.00th=[ 107], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 2048, max= 2048, per=10.30%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6911KiB/s (7077kB/s)(6144KiB/889msec) clat (msec): min=30, max=216, avg=92.45, stdev=68.40 lat (msec): min=30, max=217, avg=92.73, stdev=68.39 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 43], | 30.00th=[ 43], 40.00th=[ 66], 50.00th=[ 66], 60.00th=[ 81], | 70.00th=[ 120], 80.00th=[ 120], 90.00th=[ 218], 95.00th=[ 218], | 99.00th=[ 218], 99.50th=[ 218], 99.90th=[ 218], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 6144, max= 6144, per=22.07%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=25.00%, 250=25.00% cpu : usr=0.00%, sys=5.97%, ctx=42, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22858: Thu May 1 23:04:45 2025 read: IOPS=4, BW=4971KiB/s (5090kB/s)(5120KiB/1030msec) clat (msec): min=15, max=179, avg=76.48, stdev=67.78 lat (msec): min=15, max=179, avg=76.48, stdev=67.78 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 17], 50.00th=[ 77], 60.00th=[ 77], | 70.00th=[ 96], 80.00th=[ 96], 90.00th=[ 180], 95.00th=[ 180], | 99.00th=[ 180], 99.50th=[ 180], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 8192, max= 8192, per=41.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6959KiB/s (7126kB/s)(7168KiB/1030msec) clat (msec): min=24, max=180, avg=85.45, stdev=49.37 lat (msec): min=24, max=180, avg=85.51, stdev=49.37 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 55], | 30.00th=[ 65], 40.00th=[ 65], 50.00th=[ 83], 60.00th=[ 89], | 70.00th=[ 89], 80.00th=[ 105], 90.00th=[ 182], 95.00th=[ 182], | 99.00th=[ 182], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 6144, max= 8192, per=25.75%, avg=7168.00, stdev=1448.15, samples=2 iops : min= 6, max= 8, avg= 7.00, stdev= 1.41, samples=2 lat (msec) : 20=16.67%, 50=8.33%, 100=50.00%, 250=25.00% cpu : usr=0.00%, sys=8.16%, ctx=55, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.4MiB/s (20.4MB/s), 2090KiB/s-7877KiB/s (2140kB/s-8066kB/s), io=20.0MiB (20.0MB), run=889-1030msec WRITE: bw=27.2MiB/s (28.5MB/s), 5626KiB/s-10.2MiB/s (5761kB/s-10.7MB/s), io=28.0MiB (29.4MB), run=889-1030msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22865: Thu May 1 23:04:47 2025 read: IOPS=10, BW=42.9MiB/s (44.9MB/s)(12.0MiB/280msec) clat (msec): min=4, max=138, avg=76.81, stdev=67.62 lat (msec): min=4, max=138, avg=76.81, stdev=67.62 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 87], 50.00th=[ 87], 60.00th=[ 87], | 70.00th=[ 140], 80.00th=[ 140], 90.00th=[ 140], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] lat (msec) : 10=33.33%, 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=26.88%, ctx=809, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22867: Thu May 1 23:04:47 2025 read: IOPS=1, BW=5032KiB/s (5153kB/s)(4096KiB/814msec) clat (nsec): min=151028k, max=151028k, avg=151027923.00, stdev= 0.00 lat (nsec): min=151030k, max=151030k, avg=151029973.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 153], 10.00th=[ 153], 20.00th=[ 153], | 30.00th=[ 153], 40.00th=[ 153], 50.00th=[ 153], 60.00th=[ 153], | 70.00th=[ 153], 80.00th=[ 153], 90.00th=[ 153], 95.00th=[ 153], | 99.00th=[ 153], 99.50th=[ 153], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 8192, max= 8192, per=40.70%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9.83MiB/s (10.3MB/s)(8192KiB/814msec) clat (msec): min=286, max=345, avg=316.15, stdev=41.27 lat (msec): min=287, max=345, avg=316.34, stdev=41.37 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 288], 10.00th=[ 288], 20.00th=[ 288], | 30.00th=[ 288], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 347], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 8192, max= 8192, per=21.75%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=29.64%, ctx=762, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22869: Thu May 1 23:04:47 2025 write: IOPS=3, BW=13.8MiB/s (14.5MB/s)(12.0MiB/870msec) clat (msec): min=234, max=335, avg=273.36, stdev=54.26 lat (msec): min=234, max=335, avg=273.78, stdev=54.12 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 234], 10.00th=[ 234], 20.00th=[ 234], | 30.00th=[ 234], 40.00th=[ 251], 50.00th=[ 251], 60.00th=[ 251], | 70.00th=[ 334], 80.00th=[ 334], 90.00th=[ 334], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 8192, max= 8192, per=21.75%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=36.02%, ctx=88, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22871: Thu May 1 23:04:47 2025 write: IOPS=3, BW=14.3MiB/s (14.9MB/s)(12.0MiB/842msec) clat (msec): min=166, max=360, avg=275.09, stdev=99.00 lat (msec): min=167, max=361, avg=275.39, stdev=99.03 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 167], | 30.00th=[ 167], 40.00th=[ 296], 50.00th=[ 296], 60.00th=[ 296], | 70.00th=[ 363], 80.00th=[ 363], 90.00th=[ 363], 95.00th=[ 363], | 99.00th=[ 363], 99.50th=[ 363], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min=16351, max=16351, per=43.41%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=40.31%, ctx=48, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.7MiB/s (20.6MB/s), 5032KiB/s-42.9MiB/s (5153kB/s-44.9MB/s), io=16.0MiB (16.8MB), run=280-814msec WRITE: bw=36.8MiB/s (38.6MB/s), 9.83MiB/s-14.3MiB/s (10.3MB/s-14.9MB/s), io=32.0MiB (33.6MB), run=814-870msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22866: Thu May 1 23:04:48 2025 read: IOPS=4, BW=17.1MiB/s (17.0MB/s)(12.0MiB/700msec) clat (msec): min=103, max=389, avg=217.08, stdev=151.55 lat (msec): min=103, max=389, avg=217.08, stdev=151.55 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 105], 10.00th=[ 105], 20.00th=[ 105], | 30.00th=[ 105], 40.00th=[ 159], 50.00th=[ 159], 60.00th=[ 159], | 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=3.72%, ctx=21, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22868: Thu May 1 23:04:48 2025 read: IOPS=0, BW=3807KiB/s (3898kB/s)(4096KiB/1076msec) clat (nsec): min=535140k, max=535140k, avg=535140195.00, stdev= 0.00 lat (nsec): min=535141k, max=535141k, avg=535141157.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 535], 5.00th=[ 535], 10.00th=[ 535], 20.00th=[ 535], | 30.00th=[ 535], 40.00th=[ 535], 50.00th=[ 535], 60.00th=[ 535], | 70.00th=[ 535], 80.00th=[ 535], 90.00th=[ 535], 95.00th=[ 535], | 99.00th=[ 535], 99.50th=[ 535], 99.90th=[ 535], 99.95th=[ 535], | 99.99th=[ 535] bw ( KiB/s): min= 8159, max= 8159, per=53.59%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=7613KiB/s (7796kB/s)(8192KiB/1076msec) clat (msec): min=150, max=323, avg=237.33, stdev=122.49 lat (msec): min=150, max=324, avg=237.58, stdev=122.49 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 150], 10.00th=[ 150], 20.00th=[ 150], | 30.00th=[ 150], 40.00th=[ 150], 50.00th=[ 150], 60.00th=[ 326], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 8159, max= 8159, per=29.26%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.09%, sys=4.74%, ctx=26, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22870: Thu May 1 23:04:48 2025 write: IOPS=2, BW=10.2MiB/s (10.7MB/s)(12.0MiB/1175msec) clat (msec): min=140, max=694, avg=381.79, stdev=283.67 lat (msec): min=140, max=694, avg=382.09, stdev=283.74 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 140], 10.00th=[ 140], 20.00th=[ 140], | 30.00th=[ 140], 40.00th=[ 313], 50.00th=[ 313], 60.00th=[ 313], | 70.00th=[ 693], 80.00th=[ 693], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 8159, max= 8159, per=29.26%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.34%, sys=3.92%, ctx=30, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22872: Thu May 1 23:04:48 2025 write: IOPS=2, BW=10.0MiB/s (11.5MB/s)(12.0MiB/1091msec) clat (msec): min=161, max=594, avg=348.83, stdev=222.01 lat (msec): min=162, max=594, avg=349.22, stdev=221.86 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 163], 10.00th=[ 163], 20.00th=[ 163], | 30.00th=[ 163], 40.00th=[ 292], 50.00th=[ 292], 60.00th=[ 292], | 70.00th=[ 592], 80.00th=[ 592], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min=16384, max=16384, per=58.75%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=13.58%, ctx=103, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=14.9MiB/s (15.6MB/s), 3807KiB/s-17.1MiB/s (3898kB/s-17.0MB/s), io=16.0MiB (16.8MB), run=700-1076msec WRITE: bw=27.2MiB/s (28.6MB/s), 7613KiB/s-10.0MiB/s (7796kB/s-11.5MB/s), io=32.0MiB (33.6MB), run=1076-1175msec