== sanity test 398b: DIO and buffer IO race ============== 01:41:09 (1746078069) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.03078 s, 24.8 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1205: Thu May 1 01:41:51 2025 read: IOPS=40, BW=163KiB/s (166kB/s)(5992KiB/36872msec) clat (usec): min=52, max=284164, avg=11094.48, stdev=11473.81 lat (usec): min=52, max=284165, avg=11095.81, stdev=11473.66 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 98], 10.00th=[ 115], 20.00th=[ 4178], | 30.00th=[ 6390], 40.00th=[ 7767], 50.00th=[ 9372], 60.00th=[ 11600], | 70.00th=[ 14091], 80.00th=[ 17171], 90.00th=[ 22152], 95.00th=[ 26870], | 99.00th=[ 38011], 99.50th=[ 44827], 99.90th=[119014], 99.95th=[283116], | 99.99th=[283116] bw ( KiB/s): min= 47, max= 318, per=25.15%, avg=161.23, stdev=51.23, samples=73 iops : min= 11, max= 79, avg=39.99, stdev=12.78, samples=73 write: IOPS=42, BW=171KiB/s (175kB/s)(6296KiB/36872msec) clat (usec): min=494, max=166044, avg=12705.25, stdev=14759.64 lat (usec): min=494, max=166046, avg=12706.79, stdev=14759.68 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 742], 10.00th=[ 799], 20.00th=[ 979], | 30.00th=[ 1860], 40.00th=[ 4555], 50.00th=[ 9110], 60.00th=[ 12125], | 70.00th=[ 16188], 80.00th=[ 21365], 90.00th=[ 30540], 95.00th=[ 40109], | 99.00th=[ 67634], 99.50th=[ 79168], 99.90th=[143655], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 39, max= 312, per=25.25%, avg=169.40, stdev=51.19, samples=73 iops : min= 9, max= 78, avg=42.04, stdev=12.86, samples=73 lat (usec) : 100=2.67%, 250=5.47%, 500=0.46%, 750=2.93%, 1000=7.91% lat (msec) : 2=5.50%, 4=4.52%, 10=23.40%, 20=29.52%, 50=16.34% lat (msec) : 100=1.11%, 250=0.13%, 500=0.03% cpu : usr=0.23%, sys=12.35%, ctx=5066, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1206: Thu May 1 01:41:51 2025 read: IOPS=40, BW=162KiB/s (166kB/s)(5940KiB/36718msec) clat (usec): min=51, max=182553, avg=10987.71, stdev=12044.82 lat (usec): min=52, max=182555, avg=10991.38, stdev=12045.89 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 98], 10.00th=[ 109], 20.00th=[ 416], | 30.00th=[ 6063], 40.00th=[ 7701], 50.00th=[ 9503], 60.00th=[ 11207], | 70.00th=[ 13435], 80.00th=[ 16450], 90.00th=[ 22414], 95.00th=[ 26608], | 99.00th=[ 40109], 99.50th=[ 46924], 99.90th=[179307], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 15, max= 271, per=25.14%, avg=161.12, stdev=52.51, samples=73 iops : min= 3, max= 67, avg=39.92, stdev=13.17, samples=73 write: IOPS=43, BW=173KiB/s (177kB/s)(6348KiB/36718msec) clat (usec): min=515, max=302718, avg=12690.63, stdev=15369.67 lat (usec): min=515, max=302719, avg=12692.20, stdev=15369.81 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 734], 10.00th=[ 791], 20.00th=[ 1057], | 30.00th=[ 2278], 40.00th=[ 5669], 50.00th=[ 9503], 60.00th=[ 12518], | 70.00th=[ 15926], 80.00th=[ 21365], 90.00th=[ 29230], 95.00th=[ 38011], | 99.00th=[ 64226], 99.50th=[ 81265], 99.90th=[108528], 99.95th=[304088], | 99.99th=[304088] bw ( KiB/s): min= 31, max= 374, per=25.57%, avg=171.56, stdev=60.12, samples=73 iops : min= 7, max= 93, avg=42.53, stdev=15.06, samples=73 lat (usec) : 100=2.67%, 250=6.67%, 500=0.46%, 750=3.39%, 1000=6.67% lat (msec) : 2=5.18%, 4=4.75%, 10=22.79%, 20=29.52%, 50=16.70% lat (msec) : 100=0.98%, 250=0.20%, 500=0.03% cpu : usr=0.16%, sys=12.24%, ctx=5008, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1207: Thu May 1 01:41:51 2025 read: IOPS=40, BW=161KiB/s (165kB/s)(6028KiB/37460msec) clat (usec): min=70, max=197802, avg=11058.23, stdev=11106.89 lat (usec): min=71, max=197803, avg=11061.66, stdev=11106.45 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 99], 10.00th=[ 115], 20.00th=[ 2769], | 30.00th=[ 6194], 40.00th=[ 7635], 50.00th=[ 9503], 60.00th=[ 11338], | 70.00th=[ 13435], 80.00th=[ 16909], 90.00th=[ 23200], 95.00th=[ 28181], | 99.00th=[ 39060], 99.50th=[ 47973], 99.90th=[135267], 99.95th=[198181], | 99.99th=[198181] bw ( KiB/s): min= 39, max= 303, per=24.73%, avg=158.54, stdev=53.14, samples=74 iops : min= 9, max= 75, avg=39.30, stdev=13.32, samples=74 write: IOPS=41, BW=167KiB/s (171kB/s)(6260KiB/37460msec) clat (usec): min=492, max=304266, avg=13120.20, stdev=15794.53 lat (usec): min=493, max=304268, avg=13122.49, stdev=15794.21 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 758], 10.00th=[ 807], 20.00th=[ 1139], | 30.00th=[ 2474], 40.00th=[ 5997], 50.00th=[ 9503], 60.00th=[ 12780], | 70.00th=[ 16909], 80.00th=[ 21890], 90.00th=[ 29492], 95.00th=[ 36963], | 99.00th=[ 66323], 99.50th=[ 81265], 99.90th=[152044], 99.95th=[304088], | 99.99th=[304088] bw ( KiB/s): min= 39, max= 295, per=24.50%, avg=164.36, stdev=48.70, samples=74 iops : min= 9, max= 73, avg=40.77, stdev=12.22, samples=74 lat (usec) : 100=2.73%, 250=6.15%, 500=0.49%, 750=2.57%, 1000=6.84% lat (msec) : 2=4.82%, 4=4.36%, 10=24.12%, 20=29.30%, 50=17.19% lat (msec) : 100=1.20%, 250=0.20%, 500=0.03% cpu : usr=0.20%, sys=12.17%, ctx=5122, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1208: Thu May 1 01:41:51 2025 read: IOPS=41, BW=167KiB/s (171kB/s)(6052KiB/36166msec) clat (usec): min=48, max=317295, avg=10923.04, stdev=12455.79 lat (usec): min=48, max=317297, avg=10925.06, stdev=12456.11 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 100], 10.00th=[ 116], 20.00th=[ 1254], | 30.00th=[ 5997], 40.00th=[ 7570], 50.00th=[ 9372], 60.00th=[ 10945], | 70.00th=[ 13698], 80.00th=[ 16909], 90.00th=[ 21103], 95.00th=[ 26346], | 99.00th=[ 45351], 99.50th=[ 54789], 99.90th=[121111], 99.95th=[316670], | 99.99th=[316670] bw ( KiB/s): min= 23, max= 312, per=26.01%, avg=166.74, stdev=51.28, samples=72 iops : min= 5, max= 78, avg=41.39, stdev=12.85, samples=72 write: IOPS=43, BW=172KiB/s (177kB/s)(6236KiB/36166msec) clat (usec): min=565, max=144550, avg=12435.79, stdev=13520.18 lat (usec): min=567, max=144552, avg=12437.69, stdev=13520.44 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 996], | 30.00th=[ 2114], 40.00th=[ 5211], 50.00th=[ 9241], 60.00th=[ 12649], | 70.00th=[ 16319], 80.00th=[ 21365], 90.00th=[ 28967], 95.00th=[ 35914], | 99.00th=[ 57410], 99.50th=[ 70779], 99.90th=[126354], 99.95th=[143655], | 99.99th=[143655] bw ( KiB/s): min= 31, max= 312, per=25.57%, avg=171.58, stdev=50.20, samples=72 iops : min= 7, max= 78, avg=42.60, stdev=12.62, samples=72 lat (usec) : 50=0.03%, 100=2.51%, 250=6.38%, 500=0.55%, 750=3.71% lat (usec) : 1000=6.77% lat (msec) : 2=5.01%, 4=4.49%, 10=24.15%, 20=28.71%, 50=16.44% lat (msec) : 100=1.07%, 250=0.13%, 500=0.03% cpu : usr=0.22%, sys=12.21%, ctx=5103, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=641KiB/s (656kB/s), 161KiB/s-167KiB/s (165kB/s-171kB/s), io=23.4MiB (24.6MB), run=36166-37460msec WRITE: bw=671KiB/s (687kB/s), 167KiB/s-173KiB/s (171kB/s-177kB/s), io=24.6MiB (25.7MB), run=36166-37460msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1210: Thu May 1 01:43:20 2025 read: IOPS=11, BW=47.6KiB/s (48.8kB/s)(5992KiB/125843msec) clat (msec): min=3, max=330, avg=12.55, stdev=12.96 lat (msec): min=3, max=330, avg=12.56, stdev=12.96 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 16], 90.00th=[ 23], 95.00th=[ 31], | 99.00th=[ 57], 99.50th=[ 68], 99.90th=[ 118], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 7, max= 200, per=25.55%, avg=48.29, stdev=27.92, samples=245 iops : min= 1, max= 50, avg=11.76, stdev= 7.02, samples=245 write: IOPS=12, BW=50.0KiB/s (51.2kB/s)(6296KiB/125843msec) clat (msec): min=18, max=262, avg=67.84, stdev=27.17 lat (msec): min=18, max=262, avg=67.84, stdev=27.17 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 36], 10.00th=[ 40], 20.00th=[ 47], | 30.00th=[ 53], 40.00th=[ 57], 50.00th=[ 63], 60.00th=[ 69], | 70.00th=[ 77], 80.00th=[ 86], 90.00th=[ 101], 95.00th=[ 121], | 99.00th=[ 159], 99.50th=[ 174], 99.90th=[ 241], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 15, max= 96, per=25.10%, avg=49.70, stdev=15.09, samples=250 iops : min= 3, max= 24, avg=12.12, stdev= 3.83, samples=250 lat (msec) : 4=0.85%, 10=25.55%, 20=16.24%, 50=18.88%, 100=33.01% lat (msec) : 250=5.40%, 500=0.07% cpu : usr=0.04%, sys=2.87%, ctx=3762, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1211: Thu May 1 01:43:20 2025 read: IOPS=11, BW=46.8KiB/s (47.9kB/s)(5940KiB/126937msec) clat (msec): min=3, max=196, avg=12.44, stdev=11.62 lat (msec): min=3, max=196, avg=12.44, stdev=11.62 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 16], 90.00th=[ 22], 95.00th=[ 32], | 99.00th=[ 61], 99.50th=[ 73], 99.90th=[ 131], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 7, max= 143, per=25.10%, avg=47.44, stdev=26.60, samples=247 iops : min= 1, max= 35, avg=11.59, stdev= 6.69, samples=247 write: IOPS=12, BW=50.0KiB/s (51.2kB/s)(6348KiB/126937msec) clat (msec): min=22, max=236, avg=68.14, stdev=28.19 lat (msec): min=22, max=236, avg=68.15, stdev=28.19 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 35], 10.00th=[ 40], 20.00th=[ 46], | 30.00th=[ 51], 40.00th=[ 57], 50.00th=[ 63], 60.00th=[ 69], | 70.00th=[ 77], 80.00th=[ 87], 90.00th=[ 105], 95.00th=[ 120], | 99.00th=[ 169], 99.50th=[ 186], 99.90th=[ 230], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 7, max= 88, per=25.00%, avg=49.50, stdev=15.19, samples=252 iops : min= 1, max= 22, avg=12.10, stdev= 3.83, samples=252 lat (msec) : 4=1.24%, 10=25.49%, 20=15.95%, 50=19.56%, 100=31.74% lat (msec) : 250=6.02% cpu : usr=0.03%, sys=2.89%, ctx=3791, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1212: Thu May 1 01:43:20 2025 read: IOPS=12, BW=48.0KiB/s (49.2kB/s)(6028KiB/125552msec) clat (msec): min=3, max=229, avg=12.48, stdev=11.43 lat (msec): min=3, max=229, avg=12.48, stdev=11.43 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 14], 80.00th=[ 16], 90.00th=[ 23], 95.00th=[ 30], | 99.00th=[ 56], 99.50th=[ 67], 99.90th=[ 132], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 7, max= 144, per=25.74%, avg=48.65, stdev=26.53, samples=245 iops : min= 1, max= 36, avg=11.94, stdev= 6.69, samples=245 write: IOPS=12, BW=49.9KiB/s (51.1kB/s)(6260KiB/125552msec) clat (msec): min=22, max=292, avg=68.04, stdev=28.93 lat (msec): min=22, max=292, avg=68.04, stdev=28.93 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 35], 10.00th=[ 40], 20.00th=[ 45], | 30.00th=[ 52], 40.00th=[ 57], 50.00th=[ 63], 60.00th=[ 68], | 70.00th=[ 78], 80.00th=[ 86], 90.00th=[ 104], 95.00th=[ 125], | 99.00th=[ 174], 99.50th=[ 203], 99.90th=[ 241], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 8, max= 96, per=24.98%, avg=49.46, stdev=16.04, samples=250 iops : min= 2, max= 24, avg=12.14, stdev= 4.06, samples=250 lat (msec) : 4=0.75%, 10=25.59%, 20=16.73%, 50=19.86%, 100=31.28% lat (msec) : 250=5.76%, 500=0.03% cpu : usr=0.06%, sys=2.85%, ctx=3792, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1213: Thu May 1 01:43:20 2025 read: IOPS=12, BW=48.3KiB/s (49.5kB/s)(6052KiB/125212msec) clat (msec): min=2, max=118, avg=11.83, stdev= 9.22 lat (msec): min=2, max=118, avg=11.83, stdev= 9.22 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 15], 90.00th=[ 21], 95.00th=[ 30], | 99.00th=[ 46], 99.50th=[ 55], 99.90th=[ 99], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 7, max= 136, per=26.10%, avg=49.33, stdev=27.55, samples=243 iops : min= 1, max= 34, avg=12.02, stdev= 6.91, samples=243 write: IOPS=12, BW=49.8KiB/s (50.0kB/s)(6236KiB/125212msec) clat (msec): min=20, max=433, avg=68.72, stdev=29.81 lat (msec): min=20, max=433, avg=68.72, stdev=29.81 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 35], 10.00th=[ 40], 20.00th=[ 47], | 30.00th=[ 52], 40.00th=[ 58], 50.00th=[ 64], 60.00th=[ 69], | 70.00th=[ 77], 80.00th=[ 86], 90.00th=[ 106], 95.00th=[ 124], | 99.00th=[ 163], 99.50th=[ 190], 99.90th=[ 264], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 15, max= 88, per=25.03%, avg=49.55, stdev=14.99, samples=249 iops : min= 3, max= 22, avg=12.08, stdev= 3.79, samples=249 lat (msec) : 4=0.88%, 10=26.82%, 20=16.28%, 50=18.78%, 100=30.96% lat (msec) : 250=6.18%, 500=0.10% cpu : usr=0.03%, sys=2.88%, ctx=3773, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=189KiB/s (194kB/s), 46.8KiB/s-48.3KiB/s (47.9kB/s-49.5kB/s), io=23.4MiB (24.6MB), run=125212-126937msec WRITE: bw=198KiB/s (203kB/s), 49.8KiB/s-50.0KiB/s (50.0kB/s-51.2kB/s), io=24.6MiB (25.7MB), run=125212-126937msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1332: Thu May 1 01:43:28 2025 read: IOPS=61, BW=983KiB/s (1006kB/s)(5920KiB/6024msec) clat (usec): min=69, max=58578, avg=7908.82, stdev=7679.63 lat (usec): min=70, max=58579, avg=7909.90, stdev=7679.60 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 101], 10.00th=[ 114], 20.00th=[ 135], | 30.00th=[ 3949], 40.00th=[ 5997], 50.00th=[ 7242], 60.00th=[ 8586], | 70.00th=[10290], 80.00th=[12256], 90.00th=[16057], 95.00th=[20579], | 99.00th=[40109], 99.50th=[53216], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 416, max= 1696, per=25.89%, avg=979.33, stdev=330.33, samples=12 iops : min= 26, max= 106, avg=60.92, stdev=20.65, samples=12 write: IOPS=66, BW=1057KiB/s (1082kB/s)(6368KiB/6024msec) clat (usec): min=681, max=95163, avg=7605.49, stdev=10300.68 lat (usec): min=683, max=95165, avg=7607.12, stdev=10300.88 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 1057], 10.00th=[ 1139], 20.00th=[ 1319], | 30.00th=[ 1745], 40.00th=[ 2278], 50.00th=[ 3523], 60.00th=[ 5538], | 70.00th=[ 8586], 80.00th=[12256], 90.00th=[18744], 95.00th=[23462], | 99.00th=[61080], 99.50th=[77071], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 384, max= 1568, per=26.42%, avg=1051.33, stdev=351.83, samples=12 iops : min= 24, max= 98, avg=65.42, stdev=22.04, samples=12 lat (usec) : 100=2.34%, 250=9.77%, 500=0.78%, 750=0.65%, 1000=1.56% lat (msec) : 2=17.45%, 4=9.90%, 10=29.56%, 20=21.22%, 50=5.86% lat (msec) : 100=0.91% cpu : usr=0.15%, sys=17.18%, ctx=1124, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1333: Thu May 1 01:43:28 2025 read: IOPS=56, BW=907KiB/s (929kB/s)(5744KiB/6333msec) clat (usec): min=81, max=54906, avg=6933.25, stdev=7037.36 lat (usec): min=81, max=54907, avg=6934.17, stdev=7037.37 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 97], 10.00th=[ 106], 20.00th=[ 123], | 30.00th=[ 145], 40.00th=[ 4948], 50.00th=[ 6456], 60.00th=[ 7767], | 70.00th=[ 9110], 80.00th=[11600], 90.00th=[16057], 95.00th=[18220], | 99.00th=[27395], 99.50th=[41157], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 63, max= 1341, per=23.73%, avg=897.33, stdev=341.83, samples=12 iops : min= 3, max= 83, avg=55.75, stdev=21.51, samples=12 write: IOPS=64, BW=1033KiB/s (1058kB/s)(6544KiB/6333msec) clat (usec): min=807, max=99733, avg=9155.29, stdev=13299.01 lat (usec): min=807, max=99734, avg=9156.85, stdev=13299.05 clat percentiles (usec): | 1.00th=[ 857], 5.00th=[ 979], 10.00th=[ 1090], 20.00th=[ 1254], | 30.00th=[ 1598], 40.00th=[ 2114], 50.00th=[ 3359], 60.00th=[ 6456], | 70.00th=[ 10159], 80.00th=[ 14615], 90.00th=[ 22676], 95.00th=[ 31851], | 99.00th=[ 73925], 99.50th=[ 78119], 99.90th=[100140], 99.95th=[100140], | 99.99th=[100140] bw ( KiB/s): min= 223, max= 1504, per=25.50%, avg=1014.58, stdev=369.75, samples=12 iops : min= 13, max= 94, avg=63.08, stdev=23.20, samples=12 lat (usec) : 100=2.86%, 250=12.37%, 500=0.65%, 1000=3.26% lat (msec) : 2=17.45%, 4=8.46%, 10=26.30%, 20=20.18%, 50=7.03% lat (msec) : 100=1.43% cpu : usr=0.11%, sys=15.93%, ctx=1127, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1334: Thu May 1 01:43:28 2025 read: IOPS=66, BW=1069KiB/s (1095kB/s)(6256KiB/5852msec) clat (usec): min=54, max=75916, avg=7329.69, stdev=8172.68 lat (usec): min=54, max=75917, avg=7345.29, stdev=8165.31 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 97], 10.00th=[ 106], 20.00th=[ 121], | 30.00th=[ 180], 40.00th=[ 5145], 50.00th=[ 6128], 60.00th=[ 7635], | 70.00th=[ 9503], 80.00th=[12125], 90.00th=[16057], 95.00th=[20055], | 99.00th=[43779], 99.50th=[55313], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 416, max= 1405, per=28.52%, avg=1078.45, stdev=303.02, samples=11 iops : min= 26, max= 87, avg=67.18, stdev=18.78, samples=11 write: IOPS=64, BW=1031KiB/s (1055kB/s)(6032KiB/5852msec) clat (usec): min=642, max=74908, avg=7533.85, stdev=8879.84 lat (usec): min=643, max=74909, avg=7535.44, stdev=8879.84 clat percentiles (usec): | 1.00th=[ 857], 5.00th=[ 1037], 10.00th=[ 1123], 20.00th=[ 1303], | 30.00th=[ 1680], 40.00th=[ 2343], 50.00th=[ 3589], 60.00th=[ 6063], | 70.00th=[ 9241], 80.00th=[13435], 90.00th=[18744], 95.00th=[24249], | 99.00th=[35390], 99.50th=[60031], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 448, max= 1373, per=25.50%, avg=1014.45, stdev=301.87, samples=11 iops : min= 28, max= 85, avg=63.18, stdev=18.68, samples=11 lat (usec) : 100=3.78%, 250=11.85%, 500=0.52%, 750=0.91%, 1000=1.43% lat (msec) : 2=16.93%, 4=8.33%, 10=27.60%, 20=21.74%, 50=6.12% lat (msec) : 100=0.78% cpu : usr=0.26%, sys=17.67%, ctx=1150, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1335: Thu May 1 01:43:28 2025 read: IOPS=61, BW=989KiB/s (1013kB/s)(6032KiB/6097msec) clat (usec): min=69, max=83176, avg=7062.81, stdev=7924.20 lat (usec): min=70, max=83176, avg=7064.72, stdev=7923.44 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 92], 10.00th=[ 102], 20.00th=[ 116], | 30.00th=[ 137], 40.00th=[ 4178], 50.00th=[ 6128], 60.00th=[ 7898], | 70.00th=[10159], 80.00th=[12649], 90.00th=[16450], 95.00th=[20055], | 99.00th=[30802], 99.50th=[40633], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 223, max= 1376, per=25.92%, avg=980.33, stdev=359.37, samples=12 iops : min= 13, max= 86, avg=60.92, stdev=22.73, samples=12 write: IOPS=64, BW=1026KiB/s (1051kB/s)(6256KiB/6097msec) clat (usec): min=661, max=98621, avg=8409.57, stdev=11429.98 lat (usec): min=661, max=99124, avg=8412.41, stdev=11440.18 clat percentiles (usec): | 1.00th=[ 873], 5.00th=[ 996], 10.00th=[ 1090], 20.00th=[ 1254], | 30.00th=[ 1582], 40.00th=[ 2442], 50.00th=[ 4047], 60.00th=[ 6325], | 70.00th=[ 9765], 80.00th=[13042], 90.00th=[20317], 95.00th=[28181], | 99.00th=[60556], 99.50th=[78119], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 223, max= 1440, per=25.57%, avg=1017.42, stdev=322.39, samples=12 iops : min= 13, max= 90, avg=63.25, stdev=20.39, samples=12 lat (usec) : 100=4.30%, 250=13.15%, 500=0.52%, 750=0.65%, 1000=2.73% lat (msec) : 2=15.36%, 4=8.20%, 10=25.26%, 20=21.87%, 50=7.16% lat (msec) : 100=0.78% cpu : usr=0.21%, sys=16.13%, ctx=1130, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3782KiB/s (3873kB/s), 907KiB/s-1069KiB/s (929kB/s-1095kB/s), io=23.4MiB (24.5MB), run=5852-6333msec WRITE: bw=3979KiB/s (4075kB/s), 1026KiB/s-1057KiB/s (1051kB/s-1082kB/s), io=24.6MiB (25.8MB), run=5852-6333msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1327: Thu May 1 01:43:51 2025 read: IOPS=12, BW=202KiB/s (207kB/s)(5920KiB/29250msec) clat (usec): min=3417, max=53933, avg=11216.20, stdev=7657.68 lat (usec): min=3418, max=53934, avg=11218.26, stdev=7657.65 clat percentiles (usec): | 1.00th=[ 3687], 5.00th=[ 4424], 10.00th=[ 5145], 20.00th=[ 6128], | 30.00th=[ 6783], 40.00th=[ 7635], 50.00th=[ 8717], 60.00th=[10028], | 70.00th=[11863], 80.00th=[14746], 90.00th=[21103], 95.00th=[26084], | 99.00th=[44827], 99.50th=[45876], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 31, max= 510, per=25.67%, avg=204.56, stdev=95.16, samples=57 iops : min= 1, max= 31, avg=12.47, stdev= 5.97, samples=57 write: IOPS=13, BW=218KiB/s (223kB/s)(6368KiB/29250msec) clat (msec): min=25, max=143, avg=62.86, stdev=21.62 lat (msec): min=25, max=143, avg=62.86, stdev=21.62 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 33], 10.00th=[ 39], 20.00th=[ 45], | 30.00th=[ 50], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 64], | 70.00th=[ 72], 80.00th=[ 81], 90.00th=[ 93], 95.00th=[ 104], | 99.00th=[ 133], 99.50th=[ 140], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 127, max= 384, per=25.87%, avg=217.03, stdev=53.64, samples=58 iops : min= 7, max= 24, avg=13.26, stdev= 3.48, samples=58 lat (msec) : 4=1.04%, 10=27.60%, 20=14.06%, 50=21.09%, 100=33.20% lat (msec) : 250=2.99% cpu : usr=0.06%, sys=3.29%, ctx=951, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1328: Thu May 1 01:43:51 2025 read: IOPS=11, BW=191KiB/s (196kB/s)(5744KiB/30035msec) clat (usec): min=3162, max=62996, avg=10803.70, stdev=8076.94 lat (usec): min=3164, max=62997, avg=10805.18, stdev=8076.85 clat percentiles (usec): | 1.00th=[ 3556], 5.00th=[ 4359], 10.00th=[ 4752], 20.00th=[ 5538], | 30.00th=[ 6325], 40.00th=[ 7111], 50.00th=[ 8455], 60.00th=[ 9765], | 70.00th=[11469], 80.00th=[14353], 90.00th=[19006], 95.00th=[23200], | 99.00th=[50070], 99.50th=[60556], 99.90th=[63177], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 32, max= 512, per=23.92%, avg=190.68, stdev=105.63, samples=59 iops : min= 2, max= 32, avg=11.69, stdev= 6.62, samples=59 write: IOPS=13, BW=218KiB/s (223kB/s)(6544KiB/30035msec) clat (msec): min=24, max=136, avg=63.79, stdev=21.94 lat (msec): min=24, max=136, avg=63.80, stdev=21.94 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 35], 10.00th=[ 39], 20.00th=[ 45], | 30.00th=[ 50], 40.00th=[ 56], 50.00th=[ 61], 60.00th=[ 67], | 70.00th=[ 73], 80.00th=[ 81], 90.00th=[ 95], 95.00th=[ 108], | 99.00th=[ 124], 99.50th=[ 126], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 127, max= 384, per=25.76%, avg=216.15, stdev=51.67, samples=59 iops : min= 7, max= 24, avg=13.29, stdev= 3.33, samples=59 lat (msec) : 4=1.43%, 10=27.60%, 20=13.41%, 50=20.18%, 100=33.33% lat (msec) : 250=4.04% cpu : usr=0.07%, sys=3.07%, ctx=928, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1329: Thu May 1 01:43:51 2025 read: IOPS=13, BW=219KiB/s (224kB/s)(6256KiB/28597msec) clat (usec): min=3273, max=69847, avg=10599.91, stdev=7649.33 lat (usec): min=3273, max=69848, avg=10601.47, stdev=7649.26 clat percentiles (usec): | 1.00th=[ 3392], 5.00th=[ 4424], 10.00th=[ 4883], 20.00th=[ 5735], | 30.00th=[ 6456], 40.00th=[ 7439], 50.00th=[ 8356], 60.00th=[10028], | 70.00th=[11469], 80.00th=[13829], 90.00th=[17433], 95.00th=[22152], | 99.00th=[45876], 99.50th=[67634], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 32, max= 640, per=27.48%, avg=219.05, stdev=122.43, samples=57 iops : min= 2, max= 40, avg=13.32, stdev= 7.72, samples=57 write: IOPS=13, BW=211KiB/s (216kB/s)(6032KiB/28597msec) clat (msec): min=25, max=149, avg=64.59, stdev=21.22 lat (msec): min=25, max=149, avg=64.59, stdev=21.22 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 36], 10.00th=[ 41], 20.00th=[ 46], | 30.00th=[ 53], 40.00th=[ 58], 50.00th=[ 62], 60.00th=[ 67], | 70.00th=[ 72], 80.00th=[ 83], 90.00th=[ 94], 95.00th=[ 105], | 99.00th=[ 125], 99.50th=[ 136], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 127, max= 352, per=25.04%, avg=210.11, stdev=44.50, samples=57 iops : min= 7, max= 22, avg=12.75, stdev= 2.89, samples=57 lat (msec) : 4=1.30%, 10=29.43%, 20=16.54%, 50=16.67%, 100=32.94% lat (msec) : 250=3.12% cpu : usr=0.12%, sys=3.33%, ctx=939, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1330: Thu May 1 01:43:51 2025 read: IOPS=12, BW=205KiB/s (210kB/s)(6032KiB/29454msec) clat (usec): min=3000, max=62619, avg=10614.96, stdev=6416.55 lat (usec): min=3001, max=62619, avg=10616.47, stdev=6416.62 clat percentiles (usec): | 1.00th=[ 4178], 5.00th=[ 4752], 10.00th=[ 5342], 20.00th=[ 6063], | 30.00th=[ 6915], 40.00th=[ 7635], 50.00th=[ 8586], 60.00th=[ 9765], | 70.00th=[11863], 80.00th=[13829], 90.00th=[18482], 95.00th=[22414], | 99.00th=[32637], 99.50th=[39060], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 32, max= 416, per=25.52%, avg=203.43, stdev=97.16, samples=58 iops : min= 2, max= 26, avg=12.57, stdev= 6.09, samples=58 write: IOPS=13, BW=212KiB/s (217kB/s)(6256KiB/29454msec) clat (msec): min=20, max=154, avg=65.00, stdev=21.86 lat (msec): min=20, max=154, avg=65.01, stdev=21.86 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 36], 10.00th=[ 42], 20.00th=[ 48], | 30.00th=[ 53], 40.00th=[ 58], 50.00th=[ 63], 60.00th=[ 67], | 70.00th=[ 72], 80.00th=[ 80], 90.00th=[ 93], 95.00th=[ 106], | 99.00th=[ 144], 99.50th=[ 150], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 96, max= 352, per=25.30%, avg=212.26, stdev=47.89, samples=58 iops : min= 6, max= 22, avg=13.12, stdev= 3.06, samples=58 lat (msec) : 4=0.39%, 10=29.43%, 20=15.23%, 50=16.15%, 100=35.16% lat (msec) : 250=3.65% cpu : usr=0.12%, sys=3.13%, ctx=941, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=797KiB/s (817kB/s), 191KiB/s-219KiB/s (196kB/s-224kB/s), io=23.4MiB (24.5MB), run=28597-30035msec WRITE: bw=839KiB/s (859kB/s), 211KiB/s-218KiB/s (216kB/s-223kB/s), io=24.6MiB (25.8MB), run=28597-30035msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1368: Thu May 1 01:43:56 2025 read: IOPS=2, BW=3061KiB/s (3134kB/s)(7168KiB/2342msec) clat (usec): min=668, max=1360.7k, avg=250510.40, stdev=491714.19 lat (usec): min=670, max=1360.7k, avg=250512.04, stdev=491714.31 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 668], 10.00th=[ 668], | 20.00th=[ 51643], 30.00th=[ 56361], 40.00th=[ 56361], | 50.00th=[ 56886], 60.00th=[ 71828], 70.00th=[ 71828], | 80.00th=[ 156238], 90.00th=[1367344], 95.00th=[1367344], | 99.00th=[1367344], 99.50th=[1367344], 99.90th=[1367344], | 99.95th=[1367344], 99.99th=[1367344] bw ( KiB/s): min= 2043, max= 6144, per=47.25%, avg=3411.67, stdev=2366.27, samples=3 iops : min= 1, max= 6, avg= 3.00, stdev= 2.65, samples=3 write: IOPS=2, BW=2186KiB/s (2239kB/s)(5120KiB/2342msec) clat (msec): min=28, max=206, avg=109.97, stdev=86.05 lat (msec): min=29, max=207, avg=110.03, stdev=86.03 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 29], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 69], 60.00th=[ 69], | 70.00th=[ 199], 80.00th=[ 199], 90.00th=[ 207], 95.00th=[ 207], | 99.00th=[ 207], 99.50th=[ 207], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 8192, max= 8192, per=81.03%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=16.67%, 100=41.67%, 250=25.00% cpu : usr=0.00%, sys=14.78%, ctx=891, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1369: Thu May 1 01:43:56 2025 read: IOPS=0, BW=790KiB/s (809kB/s)(2048KiB/2592msec) clat (msec): min=69, max=115, avg=92.70, stdev=32.67 lat (msec): min=69, max=115, avg=92.70, stdev=32.67 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 70], 10.00th=[ 70], 20.00th=[ 70], | 30.00th=[ 70], 40.00th=[ 70], 50.00th=[ 70], 60.00th=[ 116], | 70.00th=[ 116], 80.00th=[ 116], 90.00th=[ 116], 95.00th=[ 116], | 99.00th=[ 116], 99.50th=[ 116], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 2048, max= 2048, per=28.36%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3951KiB/s (4045kB/s)(10.0MiB/2592msec) clat (msec): min=22, max=1384, avg=225.30, stdev=413.48 lat (msec): min=22, max=1384, avg=225.40, stdev=413.49 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 65], 50.00th=[ 73], 60.00th=[ 86], | 70.00th=[ 171], 80.00th=[ 171], 90.00th=[ 259], 95.00th=[ 1385], | 99.00th=[ 1385], 99.50th=[ 1385], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 6144, max=10240, per=81.03%, avg=8192.00, stdev=2896.31, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (msec) : 50=25.00%, 100=33.33%, 250=25.00%, 500=8.33% cpu : usr=0.12%, sys=9.03%, ctx=175, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1370: Thu May 1 01:43:56 2025 read: IOPS=2, BW=2566KiB/s (2628kB/s)(6144KiB/2394msec) clat (usec): min=587, max=182577, avg=82139.18, stdev=58249.97 lat (usec): min=589, max=182579, avg=82140.74, stdev=58249.94 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 586], 10.00th=[ 586], 20.00th=[ 70779], | 30.00th=[ 70779], 40.00th=[ 73925], 50.00th=[ 73925], 60.00th=[ 78119], | 70.00th=[ 86508], 80.00th=[ 86508], 90.00th=[183501], 95.00th=[183501], | 99.00th=[183501], 99.50th=[183501], 99.90th=[183501], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 4055, max= 4055, per=56.16%, avg=4055.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2566KiB/s (2628kB/s)(6144KiB/2394msec) clat (msec): min=37, max=1534, avg=300.71, stdev=604.69 lat (msec): min=37, max=1534, avg=300.77, stdev=604.70 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 45], 50.00th=[ 45], 60.00th=[ 61], | 70.00th=[ 87], 80.00th=[ 87], 90.00th=[ 1536], 95.00th=[ 1536], | 99.00th=[ 1536], 99.50th=[ 1536], 99.90th=[ 1536], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 2027, max= 4096, per=30.28%, avg=3061.50, stdev=1463.00, samples=2 iops : min= 1, max= 4, avg= 2.50, stdev= 2.12, samples=2 lat (usec) : 750=8.33% lat (msec) : 50=25.00%, 100=50.00%, 250=8.33% cpu : usr=0.00%, sys=9.95%, ctx=350, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1371: Thu May 1 01:43:56 2025 read: IOPS=1, BW=1805KiB/s (1849kB/s)(5120KiB/2836msec) clat (usec): min=520, max=96449, avg=58613.79, stdev=40361.84 lat (usec): min=521, max=96451, avg=58615.35, stdev=40362.04 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 523], 10.00th=[ 523], 20.00th=[ 523], | 30.00th=[32900], 40.00th=[32900], 50.00th=[81265], 60.00th=[81265], | 70.00th=[81265], 80.00th=[81265], 90.00th=[95945], 95.00th=[95945], | 99.00th=[95945], 99.50th=[95945], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2528KiB/s (2588kB/s)(7168KiB/2836msec) clat (msec): min=49, max=1503, avg=345.70, stdev=515.48 lat (msec): min=49, max=1503, avg=345.77, stdev=515.49 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 95], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 180], 60.00th=[ 186], | 70.00th=[ 186], 80.00th=[ 271], 90.00th=[ 1502], 95.00th=[ 1502], | 99.00th=[ 1502], 99.50th=[ 1502], 99.90th=[ 1502], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 2048, max= 4096, per=33.47%, avg=3383.67, stdev=1157.58, samples=3 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=3 lat (usec) : 750=8.33% lat (msec) : 50=16.67%, 100=33.33%, 250=25.00%, 500=8.33% cpu : usr=0.07%, sys=8.43%, ctx=268, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7221KiB/s (7395kB/s), 790KiB/s-3061KiB/s (809kB/s-3134kB/s), io=20.0MiB (20.0MB), run=2342-2836msec WRITE: bw=9.87MiB/s (10.4MB/s), 2186KiB/s-3951KiB/s (2239kB/s-4045kB/s), io=28.0MiB (29.4MB), run=2342-2836msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1372: Thu May 1 01:43:56 2025 read: IOPS=2, BW=2333KiB/s (2389kB/s)(7168KiB/3073msec) clat (msec): min=24, max=1356, avg=312.30, stdev=486.56 lat (msec): min=24, max=1356, avg=312.30, stdev=486.55 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 36], | 30.00th=[ 85], 40.00th=[ 85], 50.00th=[ 85], 60.00th=[ 117], | 70.00th=[ 117], 80.00th=[ 485], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 1796, max= 6131, per=45.37%, avg=2990.25, stdev=2096.45, samples=4 iops : min= 1, max= 5, avg= 2.00, stdev= 2.00, samples=4 write: IOPS=1, BW=1666KiB/s (1706kB/s)(5120KiB/3073msec) clat (msec): min=63, max=262, avg=156.06, stdev=80.94 lat (msec): min=63, max=262, avg=156.12, stdev=80.94 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 64], | 30.00th=[ 93], 40.00th=[ 93], 50.00th=[ 159], 60.00th=[ 159], | 70.00th=[ 205], 80.00th=[ 205], 90.00th=[ 262], 95.00th=[ 262], | 99.00th=[ 262], 99.50th=[ 262], 99.90th=[ 262], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 2043, max= 7185, per=50.00%, avg=4614.00, stdev=3635.94, samples=2 iops : min= 1, max= 7, avg= 4.00, stdev= 4.24, samples=2 lat (msec) : 50=16.67%, 100=33.33%, 250=25.00%, 500=16.67% cpu : usr=0.00%, sys=2.60%, ctx=41, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1373: Thu May 1 01:43:56 2025 read: IOPS=0, BW=659KiB/s (675kB/s)(2048KiB/3107msec) clat (msec): min=36, max=155, avg=95.71, stdev=84.43 lat (msec): min=36, max=155, avg=95.71, stdev=84.43 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 157], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 2048, max= 2048, per=31.07%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3296KiB/s (3375kB/s)(10.0MiB/3107msec) clat (msec): min=63, max=1564, avg=284.63, stdev=454.24 lat (msec): min=64, max=1564, avg=284.81, stdev=454.19 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 72], | 30.00th=[ 85], 40.00th=[ 100], 50.00th=[ 140], 60.00th=[ 163], | 70.00th=[ 215], 80.00th=[ 215], 90.00th=[ 245], 95.00th=[ 1569], | 99.00th=[ 1569], 99.50th=[ 1569], 99.90th=[ 1569], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 2031, max= 8192, per=63.30%, avg=5841.33, stdev=3329.78, samples=3 iops : min= 1, max= 8, avg= 5.33, stdev= 3.79, samples=3 lat (msec) : 50=8.33%, 100=33.33%, 250=50.00% cpu : usr=0.10%, sys=3.06%, ctx=65, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1374: Thu May 1 01:43:56 2025 read: IOPS=2, BW=2104KiB/s (2155kB/s)(6144KiB/2920msec) clat (msec): min=31, max=274, avg=99.71, stdev=103.99 lat (msec): min=31, max=274, avg=99.71, stdev=104.00 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 41], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 2043, max= 6144, per=62.11%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 write: IOPS=2, BW=2104KiB/s (2155kB/s)(6144KiB/2920msec) clat (msec): min=80, max=1442, avg=367.66, stdev=529.71 lat (msec): min=80, max=1442, avg=367.75, stdev=529.71 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 81], 10.00th=[ 81], 20.00th=[ 96], | 30.00th=[ 96], 40.00th=[ 174], 50.00th=[ 174], 60.00th=[ 182], | 70.00th=[ 232], 80.00th=[ 232], 90.00th=[ 1435], 95.00th=[ 1435], | 99.00th=[ 1435], 99.50th=[ 1435], 99.90th=[ 1435], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 2043, max= 4096, per=29.57%, avg=2729.00, stdev=1183.86, samples=3 iops : min= 1, max= 4, avg= 2.33, stdev= 1.53, samples=3 lat (msec) : 50=33.33%, 100=16.67%, 250=33.33%, 500=8.33% cpu : usr=0.03%, sys=4.04%, ctx=57, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1375: Thu May 1 01:43:56 2025 read: IOPS=1, BW=1862KiB/s (1907kB/s)(5120KiB/2750msec) clat (msec): min=56, max=128, avg=91.16, stdev=30.78 lat (msec): min=56, max=128, avg=91.16, stdev=30.78 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 57], | 30.00th=[ 70], 40.00th=[ 70], 50.00th=[ 84], 60.00th=[ 84], | 70.00th=[ 117], 80.00th=[ 117], 90.00th=[ 129], 95.00th=[ 129], | 99.00th=[ 129], 99.50th=[ 129], 99.90th=[ 129], 99.95th=[ 129], | 99.99th=[ 129] bw ( KiB/s): min= 2023, max= 6144, per=61.96%, avg=4083.50, stdev=2913.99, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 write: IOPS=2, BW=2607KiB/s (2669kB/s)(7168KiB/2750msec) clat (msec): min=67, max=1498, avg=322.93, stdev=520.11 lat (msec): min=67, max=1498, avg=323.02, stdev=520.11 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 82], | 30.00th=[ 120], 40.00th=[ 120], 50.00th=[ 140], 60.00th=[ 178], | 70.00th=[ 178], 80.00th=[ 180], 90.00th=[ 1502], 95.00th=[ 1502], | 99.00th=[ 1502], 99.50th=[ 1502], 99.90th=[ 1502], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 2048, max= 6071, per=36.73%, avg=3389.00, stdev=2322.68, samples=3 iops : min= 2, max= 5, avg= 3.00, stdev= 1.73, samples=3 lat (msec) : 100=41.67%, 250=50.00% cpu : usr=0.04%, sys=3.31%, ctx=41, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6592KiB/s (6750kB/s), 659KiB/s-2333KiB/s (675kB/s-2389kB/s), io=20.0MiB (20.0MB), run=2750-3107msec WRITE: bw=9228KiB/s (9450kB/s), 1666KiB/s-3296KiB/s (1706kB/s-3375kB/s), io=28.0MiB (29.4MB), run=2750-3107msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1384: Thu May 1 01:43:59 2025 read: IOPS=5, BW=21.9MiB/s (23.0MB/s)(12.0MiB/547msec) clat (msec): min=44, max=267, avg=159.36, stdev=111.72 lat (msec): min=44, max=267, avg=159.36, stdev=111.72 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 45], | 30.00th=[ 45], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 167], | 70.00th=[ 268], 80.00th=[ 268], 90.00th=[ 268], 95.00th=[ 268], | 99.00th=[ 268], 99.50th=[ 268], 99.90th=[ 268], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 8159, max= 8159, per=63.25%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=21.25%, ctx=654, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1386: Thu May 1 01:43:59 2025 read: IOPS=0, BW=3225KiB/s (3303kB/s)(4096KiB/1270msec) clat (nsec): min=244461k, max=244461k, avg=244461150.00, stdev= 0.00 lat (nsec): min=244464k, max=244464k, avg=244463822.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 245], 10.00th=[ 245], 20.00th=[ 245], | 30.00th=[ 245], 40.00th=[ 245], 50.00th=[ 245], 60.00th=[ 245], | 70.00th=[ 245], 80.00th=[ 245], 90.00th=[ 245], 95.00th=[ 245], | 99.00th=[ 245], 99.50th=[ 245], 99.90th=[ 245], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 8192, max= 8192, per=63.50%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6450KiB/s (6605kB/s)(8192KiB/1270msec) clat (msec): min=465, max=541, avg=503.04, stdev=53.78 lat (msec): min=465, max=541, avg=503.35, stdev=53.77 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 464], 10.00th=[ 464], 20.00th=[ 464], | 30.00th=[ 464], 40.00th=[ 464], 50.00th=[ 464], 60.00th=[ 542], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 8175, max= 8175, per=31.68%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=19.70%, ctx=260, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1388: Thu May 1 01:43:59 2025 write: IOPS=2, BW=9950KiB/s (10.2MB/s)(12.0MiB/1235msec) clat (msec): min=292, max=515, avg=376.54, stdev=120.98 lat (msec): min=293, max=515, avg=376.90, stdev=121.01 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 292], 10.00th=[ 292], 20.00th=[ 292], | 30.00th=[ 292], 40.00th=[ 321], 50.00th=[ 321], 60.00th=[ 321], | 70.00th=[ 514], 80.00th=[ 514], 90.00th=[ 514], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min=16351, max=16351, per=63.37%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.00%, sys=22.12%, ctx=83, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1389: Thu May 1 01:43:59 2025 write: IOPS=2, BW=11.0MiB/s (12.6MB/s)(12.0MiB/1001msec) clat (msec): min=230, max=466, avg=309.68, stdev=136.21 lat (msec): min=231, max=467, avg=310.17, stdev=136.11 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 230], 10.00th=[ 230], 20.00th=[ 230], | 30.00th=[ 230], 40.00th=[ 232], 50.00th=[ 232], 60.00th=[ 232], | 70.00th=[ 468], 80.00th=[ 468], 90.00th=[ 468], 95.00th=[ 468], | 99.00th=[ 468], 99.50th=[ 468], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min=16384, max=16384, per=63.50%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=28.37%, ctx=90, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.6MiB/s (13.2MB/s), 3225KiB/s-21.9MiB/s (3303kB/s-23.0MB/s), io=16.0MiB (16.8MB), run=547-1270msec WRITE: bw=25.2MiB/s (26.4MB/s), 6450KiB/s-11.0MiB/s (6605kB/s-12.6MB/s), io=32.0MiB (33.6MB), run=1001-1270msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1383: Thu May 1 01:43:59 2025 read: IOPS=4, BW=17.4MiB/s (18.3MB/s)(12.0MiB/689msec) clat (msec): min=65, max=496, avg=224.79, stdev=236.70 lat (msec): min=65, max=496, avg=224.80, stdev=236.70 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 66], 10.00th=[ 66], 20.00th=[ 66], | 30.00th=[ 66], 40.00th=[ 112], 50.00th=[ 112], 60.00th=[ 112], | 70.00th=[ 498], 80.00th=[ 498], 90.00th=[ 498], 95.00th=[ 498], | 99.00th=[ 498], 99.50th=[ 498], 99.90th=[ 498], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=1.02%, sys=7.85%, ctx=51, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1385: Thu May 1 01:43:59 2025 read: IOPS=0, BW=2775KiB/s (2842kB/s)(4096KiB/1476msec) clat (nsec): min=220106k, max=220106k, avg=220106051.00, stdev= 0.00 lat (nsec): min=220109k, max=220109k, avg=220108891.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 220], 10.00th=[ 220], 20.00th=[ 220], | 30.00th=[ 220], 40.00th=[ 220], 50.00th=[ 220], 60.00th=[ 220], | 70.00th=[ 220], 80.00th=[ 220], 90.00th=[ 220], 95.00th=[ 220], | 99.00th=[ 220], 99.50th=[ 220], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 8192, max= 8192, per=73.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5550KiB/s (5683kB/s)(8192KiB/1476msec) clat (msec): min=398, max=811, avg=605.40, stdev=292.01 lat (msec): min=399, max=812, avg=605.84, stdev=292.15 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 401], 10.00th=[ 401], 20.00th=[ 401], | 30.00th=[ 401], 40.00th=[ 401], 50.00th=[ 401], 60.00th=[ 810], | 70.00th=[ 810], 80.00th=[ 810], 90.00th=[ 810], 95.00th=[ 810], | 99.00th=[ 810], 99.50th=[ 810], 99.90th=[ 810], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 5785, max= 5785, per=26.06%, avg=5785.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.41%, sys=7.39%, ctx=58, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1387: Thu May 1 01:43:59 2025 write: IOPS=2, BW=9069KiB/s (9286kB/s)(12.0MiB/1355msec) clat (msec): min=301, max=644, avg=423.82, stdev=191.62 lat (msec): min=301, max=644, avg=424.16, stdev=191.62 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 326], 50.00th=[ 326], 60.00th=[ 326], | 70.00th=[ 642], 80.00th=[ 642], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min=12011, max=12011, per=54.10%, avg=12011.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.07%, sys=10.27%, ctx=55, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1390: Thu May 1 01:43:59 2025 write: IOPS=3, BW=14.3MiB/s (14.0MB/s)(12.0MiB/841msec) clat (msec): min=221, max=345, avg=273.78, stdev=64.33 lat (msec): min=222, max=346, avg=274.26, stdev=64.18 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 222], 10.00th=[ 222], 20.00th=[ 222], | 30.00th=[ 222], 40.00th=[ 253], 50.00th=[ 253], 60.00th=[ 253], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 8192, max= 8192, per=36.90%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.36%, sys=11.43%, ctx=112, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.8MiB/s (11.4MB/s), 2775KiB/s-17.4MiB/s (2842kB/s-18.3MB/s), io=16.0MiB (16.8MB), run=689-1476msec WRITE: bw=21.7MiB/s (22.7MB/s), 5550KiB/s-14.3MiB/s (5683kB/s-14.0MB/s), io=32.0MiB (33.6MB), run=841-1476msec