== sanity test 398b: DIO and buffer IO race ============== 03:17:48 (1734682668) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.81784 s, 27.7 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27250: Fri Dec 20 03:18:35 2024 read: IOPS=35, BW=141KiB/s (145kB/s)(5992KiB/42409msec) clat (usec): min=59, max=317009, avg=12797.90, stdev=15120.55 lat (usec): min=60, max=317012, avg=12799.36, stdev=15120.31 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 85], 10.00th=[ 98], 20.00th=[ 635], | 30.00th=[ 7177], 40.00th=[ 8979], 50.00th=[ 10683], 60.00th=[ 12649], | 70.00th=[ 15270], 80.00th=[ 19530], 90.00th=[ 25297], 95.00th=[ 30802], | 99.00th=[ 49546], 99.50th=[ 60031], 99.90th=[233833], 99.95th=[316670], | 99.99th=[316670] bw ( KiB/s): min= 23, max= 262, per=24.72%, avg=139.94, stdev=44.67, samples=84 iops : min= 5, max= 65, avg=34.69, stdev=11.23, samples=84 write: IOPS=37, BW=148KiB/s (152kB/s)(6296KiB/42409msec) clat (usec): min=499, max=133437, avg=14628.52, stdev=15094.92 lat (usec): min=501, max=133439, avg=14632.57, stdev=15096.49 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 734], 10.00th=[ 791], 20.00th=[ 1156], | 30.00th=[ 2769], 40.00th=[ 7111], 50.00th=[ 11207], 60.00th=[ 14877], | 70.00th=[ 19268], 80.00th=[ 24511], 90.00th=[ 34341], 95.00th=[ 43254], | 99.00th=[ 66847], 99.50th=[ 73925], 99.90th=[126354], 99.95th=[133694], | 99.99th=[133694] bw ( KiB/s): min= 55, max= 280, per=25.05%, avg=148.31, stdev=46.28, samples=83 iops : min= 13, max= 70, avg=36.80, stdev=11.60, samples=83 lat (usec) : 100=5.31%, 250=3.97%, 500=0.42%, 750=3.39%, 1000=6.38% lat (msec) : 2=3.78%, 4=4.65%, 10=18.62%, 20=29.56%, 50=21.87% lat (msec) : 100=1.86%, 250=0.16%, 500=0.03% cpu : usr=0.19%, sys=10.96%, ctx=5179, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27251: Fri Dec 20 03:18:35 2024 read: IOPS=35, BW=141KiB/s (145kB/s)(5940KiB/42000msec) clat (usec): min=61, max=173890, avg=12471.97, stdev=11519.50 lat (usec): min=61, max=173891, avg=12474.12, stdev=11519.21 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 85], 10.00th=[ 97], 20.00th=[ 824], | 30.00th=[ 7373], 40.00th=[ 8979], 50.00th=[ 10814], 60.00th=[ 12911], | 70.00th=[ 15533], 80.00th=[ 19268], 90.00th=[ 25822], 95.00th=[ 30278], | 99.00th=[ 46924], 99.50th=[ 58983], 99.90th=[116917], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 8, max= 240, per=24.98%, avg=141.37, stdev=48.62, samples=83 iops : min= 2, max= 60, avg=35.06, stdev=12.18, samples=83 write: IOPS=37, BW=151KiB/s (155kB/s)(6348KiB/42000msec) clat (usec): min=507, max=343130, avg=14683.12, stdev=17488.17 lat (usec): min=507, max=343131, avg=14686.57, stdev=17488.11 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 1057], | 30.00th=[ 2769], 40.00th=[ 6849], 50.00th=[ 11338], 60.00th=[ 14877], | 70.00th=[ 18744], 80.00th=[ 24249], 90.00th=[ 34341], 95.00th=[ 41157], | 99.00th=[ 69731], 99.50th=[ 83362], 99.90th=[149947], 99.95th=[341836], | 99.99th=[341836] bw ( KiB/s): min= 32, max= 278, per=25.39%, avg=150.33, stdev=45.22, samples=83 iops : min= 8, max= 69, avg=37.30, stdev=11.31, samples=83 lat (usec) : 100=5.47%, 250=3.87%, 500=0.23%, 750=3.42%, 1000=6.67% lat (msec) : 2=4.04%, 4=4.39%, 10=18.29%, 20=30.96%, 50=20.80% lat (msec) : 100=1.63%, 250=0.20%, 500=0.03% cpu : usr=0.16%, sys=11.33%, ctx=5167, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27252: Fri Dec 20 03:18:35 2024 read: IOPS=35, BW=144KiB/s (147kB/s)(6028KiB/41905msec) clat (usec): min=46, max=340668, avg=12498.31, stdev=13370.14 lat (usec): min=46, max=340669, avg=12499.61, stdev=13370.22 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 84], 10.00th=[ 97], 20.00th=[ 217], | 30.00th=[ 7242], 40.00th=[ 9110], 50.00th=[ 10945], 60.00th=[ 13173], | 70.00th=[ 15533], 80.00th=[ 19268], 90.00th=[ 25035], 95.00th=[ 30802], | 99.00th=[ 46400], 99.50th=[ 51119], 99.90th=[ 89654], 99.95th=[341836], | 99.99th=[341836] bw ( KiB/s): min= 15, max= 231, per=25.34%, avg=143.43, stdev=43.49, samples=83 iops : min= 3, max= 57, avg=35.46, stdev=10.94, samples=83 write: IOPS=37, BW=149KiB/s (153kB/s)(6260KiB/41905msec) clat (usec): min=554, max=158180, avg=14625.19, stdev=15573.36 lat (usec): min=555, max=158182, avg=14627.16, stdev=15573.62 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 1106], | 30.00th=[ 2409], 40.00th=[ 6915], 50.00th=[ 11207], 60.00th=[ 14484], | 70.00th=[ 19268], 80.00th=[ 25297], 90.00th=[ 35390], 95.00th=[ 43779], | 99.00th=[ 69731], 99.50th=[ 79168], 99.90th=[112722], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 31, max= 248, per=25.11%, avg=148.63, stdev=42.56, samples=83 iops : min= 7, max= 62, avg=36.76, stdev=10.76, samples=83 lat (usec) : 50=0.03%, 100=5.47%, 250=4.36%, 500=0.10%, 750=4.07% lat (usec) : 1000=5.70% lat (msec) : 2=4.52%, 4=4.33%, 10=17.19%, 20=30.79%, 50=21.74% lat (msec) : 100=1.56%, 250=0.10%, 500=0.03% cpu : usr=0.20%, sys=11.27%, ctx=5156, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27253: Fri Dec 20 03:18:35 2024 read: IOPS=36, BW=146KiB/s (149kB/s)(6052KiB/41582msec) clat (usec): min=59, max=125904, avg=12251.86, stdev=11121.31 lat (usec): min=60, max=125905, avg=12253.48, stdev=11121.35 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 80], 10.00th=[ 93], 20.00th=[ 204], | 30.00th=[ 6783], 40.00th=[ 8979], 50.00th=[ 10683], 60.00th=[ 12780], | 70.00th=[ 15533], 80.00th=[ 19006], 90.00th=[ 24773], 95.00th=[ 31065], | 99.00th=[ 46400], 99.50th=[ 61080], 99.90th=[101188], 99.95th=[126354], | 99.99th=[126354] bw ( KiB/s): min= 71, max= 256, per=25.95%, avg=146.87, stdev=40.94, samples=82 iops : min= 17, max= 64, avg=36.45, stdev=10.36, samples=82 write: IOPS=37, BW=150KiB/s (154kB/s)(6236KiB/41582msec) clat (usec): min=514, max=272280, avg=14624.09, stdev=16534.34 lat (usec): min=514, max=272282, avg=14626.51, stdev=16536.04 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 725], 10.00th=[ 783], 20.00th=[ 1188], | 30.00th=[ 2769], 40.00th=[ 6259], 50.00th=[ 10945], 60.00th=[ 14484], | 70.00th=[ 19268], 80.00th=[ 25297], 90.00th=[ 34866], 95.00th=[ 43254], | 99.00th=[ 67634], 99.50th=[ 73925], 99.90th=[137364], 99.95th=[270533], | 99.99th=[270533] bw ( KiB/s): min= 23, max= 248, per=25.22%, avg=149.29, stdev=43.54, samples=83 iops : min= 5, max= 62, avg=37.06, stdev=10.96, samples=83 lat (usec) : 100=6.05%, 250=3.84%, 500=0.42%, 750=3.87%, 1000=5.34% lat (msec) : 2=4.82%, 4=4.59%, 10=17.87%, 20=29.82%, 50=21.55% lat (msec) : 100=1.63%, 250=0.16%, 500=0.03% cpu : usr=0.24%, sys=11.15%, ctx=5064, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=566KiB/s (580kB/s), 141KiB/s-146KiB/s (145kB/s-149kB/s), io=23.4MiB (24.6MB), run=41582-42409msec WRITE: bw=593KiB/s (607kB/s), 148KiB/s-151KiB/s (152kB/s-155kB/s), io=24.6MiB (25.7MB), run=41582-42409msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27246: Fri Dec 20 03:19:59 2024 read: IOPS=11, BW=47.8KiB/s (48.9kB/s)(5992KiB/125448msec) clat (msec): min=2, max=347, avg=11.30, stdev=14.92 lat (msec): min=2, max=347, avg=11.30, stdev=14.92 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 14], 90.00th=[ 24], 95.00th=[ 33], | 99.00th=[ 63], 99.50th=[ 87], 99.90th=[ 157], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 7, max= 168, per=25.99%, avg=49.13, stdev=30.25, samples=241 iops : min= 1, max= 42, avg=11.97, stdev= 7.62, samples=241 write: IOPS=12, BW=50.2KiB/s (51.4kB/s)(6296KiB/125448msec) clat (msec): min=20, max=303, avg=68.81, stdev=34.88 lat (msec): min=20, max=303, avg=68.81, stdev=34.88 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 32], 10.00th=[ 36], 20.00th=[ 43], | 30.00th=[ 48], 40.00th=[ 54], 50.00th=[ 59], 60.00th=[ 67], | 70.00th=[ 75], 80.00th=[ 91], 90.00th=[ 118], 95.00th=[ 140], | 99.00th=[ 186], 99.50th=[ 222], 99.90th=[ 257], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 7, max= 112, per=25.19%, avg=49.87, stdev=21.07, samples=249 iops : min= 1, max= 28, avg=12.15, stdev= 5.35, samples=249 lat (msec) : 4=2.96%, 10=31.48%, 20=8.56%, 50=22.49%, 100=26.17% lat (msec) : 250=8.24%, 500=0.10% cpu : usr=0.03%, sys=2.82%, ctx=3490, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27247: Fri Dec 20 03:19:59 2024 read: IOPS=11, BW=46.8KiB/s (47.9kB/s)(5940KiB/126860msec) clat (msec): min=2, max=350, avg=11.55, stdev=15.07 lat (msec): min=2, max=350, avg=11.55, stdev=15.07 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 14], 90.00th=[ 26], 95.00th=[ 36], | 99.00th=[ 61], 99.50th=[ 80], 99.90th=[ 140], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 7, max= 136, per=25.04%, avg=47.32, stdev=29.54, samples=248 iops : min= 1, max= 34, avg=11.57, stdev= 7.42, samples=248 write: IOPS=12, BW=50.0KiB/s (51.2kB/s)(6348KiB/126860msec) clat (msec): min=21, max=404, avg=69.00, stdev=35.87 lat (msec): min=21, max=404, avg=69.00, stdev=35.87 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 34], 10.00th=[ 36], 20.00th=[ 42], | 30.00th=[ 47], 40.00th=[ 54], 50.00th=[ 60], 60.00th=[ 68], | 70.00th=[ 77], 80.00th=[ 89], 90.00th=[ 117], 95.00th=[ 144], | 99.00th=[ 188], 99.50th=[ 209], 99.90th=[ 296], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 8, max= 96, per=25.03%, avg=49.56, stdev=20.89, samples=252 iops : min= 2, max= 24, avg=12.12, stdev= 5.29, samples=252 lat (msec) : 4=2.64%, 10=32.06%, 20=7.29%, 50=23.54%, 100=26.30% lat (msec) : 250=8.04%, 500=0.13% cpu : usr=0.09%, sys=2.79%, ctx=3522, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27248: Fri Dec 20 03:19:59 2024 read: IOPS=11, BW=47.0KiB/s (49.1kB/s)(6028KiB/125600msec) clat (msec): min=2, max=130, avg=11.08, stdev=10.98 lat (msec): min=2, max=130, avg=11.08, stdev=10.98 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 15], 90.00th=[ 23], 95.00th=[ 32], | 99.00th=[ 57], 99.50th=[ 71], 99.90th=[ 95], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 7, max= 160, per=25.63%, avg=48.44, stdev=28.72, samples=247 iops : min= 1, max= 40, avg=11.88, stdev= 7.25, samples=247 write: IOPS=12, BW=49.8KiB/s (51.0kB/s)(6260KiB/125600msec) clat (msec): min=20, max=380, avg=69.42, stdev=36.17 lat (msec): min=20, max=380, avg=69.42, stdev=36.17 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 33], 10.00th=[ 37], 20.00th=[ 43], | 30.00th=[ 49], 40.00th=[ 54], 50.00th=[ 59], 60.00th=[ 66], | 70.00th=[ 74], 80.00th=[ 92], 90.00th=[ 118], 95.00th=[ 144], | 99.00th=[ 201], 99.50th=[ 209], 99.90th=[ 300], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 7, max= 96, per=25.07%, avg=49.65, stdev=21.01, samples=250 iops : min= 1, max= 24, avg=12.18, stdev= 5.32, samples=250 lat (msec) : 4=2.77%, 10=31.05%, 20=9.37%, 50=22.36%, 100=26.24% lat (msec) : 250=8.14%, 500=0.07% cpu : usr=0.09%, sys=2.78%, ctx=3475, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27249: Fri Dec 20 03:19:59 2024 read: IOPS=12, BW=48.6KiB/s (49.7kB/s)(6052KiB/124576msec) clat (msec): min=2, max=154, avg=10.81, stdev=10.86 lat (msec): min=2, max=154, avg=10.81, stdev=10.86 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 23], 95.00th=[ 33], | 99.00th=[ 51], 99.50th=[ 62], 99.90th=[ 144], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 7, max= 152, per=26.82%, avg=50.69, stdev=29.94, samples=237 iops : min= 1, max= 38, avg=12.41, stdev= 7.49, samples=237 write: IOPS=12, BW=50.1KiB/s (51.3kB/s)(6236KiB/124576msec) clat (msec): min=18, max=394, avg=69.22, stdev=36.08 lat (msec): min=18, max=394, avg=69.22, stdev=36.08 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 33], 10.00th=[ 36], 20.00th=[ 43], | 30.00th=[ 48], 40.00th=[ 54], 50.00th=[ 59], 60.00th=[ 67], | 70.00th=[ 77], 80.00th=[ 91], 90.00th=[ 120], 95.00th=[ 146], | 99.00th=[ 194], 99.50th=[ 207], 99.90th=[ 228], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 15, max= 104, per=25.19%, avg=49.87, stdev=20.93, samples=248 iops : min= 3, max= 26, avg=12.20, stdev= 5.28, samples=248 lat (msec) : 4=2.96%, 10=31.48%, 20=8.85%, 50=22.85%, 100=25.94% lat (msec) : 250=7.88%, 500=0.03% cpu : usr=0.09%, sys=2.81%, ctx=3499, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=189KiB/s (194kB/s), 46.8KiB/s-48.6KiB/s (47.9kB/s-49.7kB/s), io=23.4MiB (24.6MB), run=124576-126860msec WRITE: bw=198KiB/s (203kB/s), 49.8KiB/s-50.2KiB/s (51.0kB/s-51.4kB/s), io=24.6MiB (25.7MB), run=124576-126860msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27369: Fri Dec 20 03:20:14 2024 read: IOPS=27, BW=443KiB/s (453kB/s)(5920KiB/13376msec) clat (usec): min=62, max=91894, avg=27133.08, stdev=16061.73 lat (usec): min=63, max=91896, avg=27134.37, stdev=16061.79 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 96], 10.00th=[ 114], 20.00th=[17433], | 30.00th=[23725], 40.00th=[26084], 50.00th=[28181], 60.00th=[31065], | 70.00th=[34341], 80.00th=[38536], 90.00th=[44303], 95.00th=[51643], | 99.00th=[71828], 99.50th=[73925], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 320, max= 576, per=24.60%, avg=436.46, stdev=76.02, samples=26 iops : min= 20, max= 36, avg=27.00, stdev= 4.67, samples=26 write: IOPS=29, BW=476KiB/s (488kB/s)(6368KiB/13376msec) clat (usec): min=614, max=82351, avg=8290.91, stdev=9784.58 lat (usec): min=615, max=82354, avg=8292.98, stdev=9784.60 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 1037], 10.00th=[ 1139], 20.00th=[ 1303], | 30.00th=[ 1647], 40.00th=[ 2671], 50.00th=[ 4228], 60.00th=[ 7439], | 70.00th=[ 9372], 80.00th=[14746], 90.00th=[20055], 95.00th=[26608], | 99.00th=[44303], 99.50th=[55837], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 223, max= 702, per=25.49%, avg=475.81, stdev=132.06, samples=26 iops : min= 13, max= 43, avg=29.46, stdev= 8.27, samples=26 lat (usec) : 100=2.99%, 250=4.17%, 500=0.65%, 750=0.39%, 1000=1.95% lat (msec) : 2=16.15%, 4=7.68%, 10=11.59%, 20=12.24%, 50=39.06% lat (msec) : 100=3.12% cpu : usr=0.16%, sys=12.63%, ctx=2150, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27370: Fri Dec 20 03:20:14 2024 read: IOPS=26, BW=426KiB/s (436kB/s)(5744KiB/13497msec) clat (usec): min=55, max=102925, avg=26652.67, stdev=17426.67 lat (usec): min=56, max=102927, avg=26654.74, stdev=17427.12 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 89], 10.00th=[ 100], 20.00th=[ 322], | 30.00th=[ 22676], 40.00th=[ 25560], 50.00th=[ 28443], 60.00th=[ 32113], | 70.00th=[ 34866], 80.00th=[ 39060], 90.00th=[ 44827], 95.00th=[ 49021], | 99.00th=[ 74974], 99.50th=[ 83362], 99.90th=[103285], 99.95th=[103285], | 99.99th=[103285] bw ( KiB/s): min= 32, max= 736, per=23.23%, avg=412.08, stdev=132.58, samples=26 iops : min= 2, max= 46, avg=25.54, stdev= 8.36, samples=26 write: IOPS=30, BW=485KiB/s (496kB/s)(6544KiB/13497msec) clat (usec): min=834, max=135606, avg=9247.00, stdev=13830.82 lat (usec): min=835, max=135608, avg=9249.31, stdev=13830.71 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 1045], 10.00th=[ 1123], 20.00th=[ 1336], | 30.00th=[ 1762], 40.00th=[ 2737], 50.00th=[ 4817], 60.00th=[ 7177], | 70.00th=[ 9503], 80.00th=[ 13304], 90.00th=[ 21103], 95.00th=[ 31851], | 99.00th=[ 66847], 99.50th=[ 98042], 99.90th=[135267], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 96, max= 992, per=25.03%, avg=467.38, stdev=197.08, samples=26 iops : min= 6, max= 62, avg=29.00, stdev=12.36, samples=26 lat (usec) : 100=4.69%, 250=4.43%, 500=0.65%, 1000=1.82% lat (msec) : 2=15.89%, 4=7.94%, 10=12.76%, 20=11.33%, 50=37.24% lat (msec) : 100=2.99%, 250=0.26% cpu : usr=0.22%, sys=12.16%, ctx=2024, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27371: Fri Dec 20 03:20:14 2024 read: IOPS=29, BW=474KiB/s (485kB/s)(6256KiB/13204msec) clat (usec): min=57, max=115538, avg=25655.50, stdev=17216.65 lat (usec): min=59, max=115540, avg=25656.72, stdev=17216.66 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 88], 10.00th=[ 99], 20.00th=[ 293], | 30.00th=[ 21627], 40.00th=[ 25035], 50.00th=[ 27919], 60.00th=[ 30802], | 70.00th=[ 33424], 80.00th=[ 36963], 90.00th=[ 43254], 95.00th=[ 50070], | 99.00th=[ 76022], 99.50th=[102237], 99.90th=[115868], 99.95th=[115868], | 99.99th=[115868] bw ( KiB/s): min= 160, max= 960, per=26.42%, avg=468.73, stdev=162.48, samples=26 iops : min= 10, max= 60, avg=29.12, stdev=10.24, samples=26 write: IOPS=28, BW=457KiB/s (468kB/s)(6032KiB/13204msec) clat (usec): min=757, max=112984, avg=8073.49, stdev=12381.64 lat (usec): min=758, max=112987, avg=8076.94, stdev=12382.38 clat percentiles (usec): | 1.00th=[ 824], 5.00th=[ 1037], 10.00th=[ 1123], 20.00th=[ 1270], | 30.00th=[ 1663], 40.00th=[ 2442], 50.00th=[ 3884], 60.00th=[ 5669], | 70.00th=[ 8225], 80.00th=[ 11863], 90.00th=[ 17695], 95.00th=[ 30016], | 99.00th=[ 55837], 99.50th=[104334], 99.90th=[112722], 99.95th=[112722], | 99.99th=[112722] bw ( KiB/s): min= 160, max= 800, per=23.85%, avg=445.27, stdev=154.60, samples=26 iops : min= 10, max= 50, avg=27.65, stdev= 9.66, samples=26 lat (usec) : 100=5.21%, 250=4.95%, 500=0.52%, 1000=2.08% lat (msec) : 2=15.10%, 4=8.33%, 10=11.72%, 20=10.16%, 50=38.80% lat (msec) : 100=2.60%, 250=0.52% cpu : usr=0.23%, sys=12.25%, ctx=2096, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27372: Fri Dec 20 03:20:14 2024 read: IOPS=28, BW=460KiB/s (471kB/s)(6032KiB/13118msec) clat (usec): min=48, max=116606, avg=25235.03, stdev=17919.75 lat (usec): min=48, max=116608, avg=25236.26, stdev=17919.72 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 90], 10.00th=[ 100], 20.00th=[ 131], | 30.00th=[ 20317], 40.00th=[ 24511], 50.00th=[ 27132], 60.00th=[ 30278], | 70.00th=[ 33817], 80.00th=[ 39060], 90.00th=[ 45876], 95.00th=[ 52691], | 99.00th=[ 68682], 99.50th=[ 81265], 99.90th=[116917], 99.95th=[116917], | 99.99th=[116917] bw ( KiB/s): min= 128, max= 896, per=25.24%, avg=447.73, stdev=149.80, samples=26 iops : min= 8, max= 56, avg=27.81, stdev= 9.39, samples=26 write: IOPS=29, BW=477KiB/s (488kB/s)(6256KiB/13118msec) clat (usec): min=771, max=85569, avg=8918.78, stdev=13286.37 lat (usec): min=773, max=85571, avg=8920.85, stdev=13286.35 clat percentiles (usec): | 1.00th=[ 938], 5.00th=[ 1074], 10.00th=[ 1139], 20.00th=[ 1287], | 30.00th=[ 1631], 40.00th=[ 2180], 50.00th=[ 3490], 60.00th=[ 6259], | 70.00th=[ 8848], 80.00th=[14353], 90.00th=[19530], 95.00th=[36439], | 99.00th=[74974], 99.50th=[82314], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 128, max= 992, per=25.10%, avg=468.62, stdev=200.53, samples=26 iops : min= 8, max= 62, avg=29.12, stdev=12.57, samples=26 lat (usec) : 50=0.13%, 100=4.69%, 250=6.38%, 500=0.13%, 750=0.39% lat (usec) : 1000=1.69% lat (msec) : 2=17.06%, 4=8.07%, 10=11.20%, 20=10.81%, 50=34.90% lat (msec) : 100=4.43%, 250=0.13% cpu : usr=0.14%, sys=12.23%, ctx=1989, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1775KiB/s (1817kB/s), 426KiB/s-474KiB/s (436kB/s-485kB/s), io=23.4MiB (24.5MB), run=13118-13497msec WRITE: bw=1867KiB/s (1912kB/s), 457KiB/s-485KiB/s (468kB/s-496kB/s), io=24.6MiB (25.8MB), run=13118-13497msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27365: Fri Dec 20 03:20:31 2024 read: IOPS=12, BW=198KiB/s (203kB/s)(5920KiB/29893msec) clat (msec): min=3, max=109, avg=11.30, stdev=11.54 lat (msec): min=3, max=109, avg=11.30, stdev=11.54 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 13], 90.00th=[ 24], 95.00th=[ 35], | 99.00th=[ 59], 99.50th=[ 75], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 32, max= 510, per=25.46%, avg=198.86, stdev=105.88, samples=58 iops : min= 2, max= 31, avg=12.17, stdev= 6.67, samples=58 write: IOPS=13, BW=213KiB/s (218kB/s)(6368KiB/29893msec) clat (msec): min=26, max=158, avg=64.34, stdev=21.15 lat (msec): min=26, max=158, avg=64.34, stdev=21.15 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 37], 10.00th=[ 40], 20.00th=[ 47], | 30.00th=[ 53], 40.00th=[ 57], 50.00th=[ 61], 60.00th=[ 65], | 70.00th=[ 73], 80.00th=[ 81], 90.00th=[ 94], 95.00th=[ 107], | 99.00th=[ 122], 99.50th=[ 144], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 63, max= 384, per=25.76%, avg=211.75, stdev=61.74, samples=59 iops : min= 3, max= 24, avg=12.98, stdev= 3.91, samples=59 lat (msec) : 4=1.56%, 10=32.94%, 20=7.81%, 50=18.75%, 100=35.29% lat (msec) : 250=3.65% cpu : usr=0.02%, sys=3.17%, ctx=883, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27366: Fri Dec 20 03:20:31 2024 read: IOPS=11, BW=188KiB/s (192kB/s)(5744KiB/30634msec) clat (usec): min=3631, max=55890, avg=10869.46, stdev=9277.03 lat (usec): min=3633, max=55891, avg=10871.62, stdev=9277.18 clat percentiles (usec): | 1.00th=[ 3752], 5.00th=[ 4359], 10.00th=[ 4883], 20.00th=[ 5538], | 30.00th=[ 6194], 40.00th=[ 6587], 50.00th=[ 7308], 60.00th=[ 8029], | 70.00th=[ 9896], 80.00th=[13435], 90.00th=[22152], 95.00th=[33817], | 99.00th=[49546], 99.50th=[55313], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 31, max= 512, per=24.21%, avg=189.07, stdev=105.53, samples=60 iops : min= 1, max= 32, avg=11.57, stdev= 6.69, samples=60 write: IOPS=13, BW=214KiB/s (219kB/s)(6544KiB/30634msec) clat (msec): min=27, max=190, avg=65.12, stdev=22.40 lat (msec): min=27, max=190, avg=65.13, stdev=22.40 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 37], 10.00th=[ 40], 20.00th=[ 46], | 30.00th=[ 53], 40.00th=[ 58], 50.00th=[ 63], 60.00th=[ 67], | 70.00th=[ 73], 80.00th=[ 80], 90.00th=[ 93], 95.00th=[ 107], | 99.00th=[ 131], 99.50th=[ 167], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 95, max= 352, per=25.88%, avg=212.72, stdev=59.88, samples=61 iops : min= 5, max= 22, avg=13.05, stdev= 3.81, samples=61 lat (msec) : 4=0.78%, 10=32.29%, 20=8.20%, 50=18.36%, 100=36.98% lat (msec) : 250=3.39% cpu : usr=0.10%, sys=3.03%, ctx=898, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27367: Fri Dec 20 03:20:31 2024 read: IOPS=13, BW=213KiB/s (219kB/s)(6256KiB/29303msec) clat (msec): min=3, max=105, avg=11.41, stdev=10.43 lat (msec): min=3, max=105, avg=11.41, stdev=10.43 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 15], 90.00th=[ 24], 95.00th=[ 33], | 99.00th=[ 56], 99.50th=[ 58], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 31, max= 640, per=27.50%, avg=214.81, stdev=115.74, samples=58 iops : min= 1, max= 40, avg=13.09, stdev= 7.40, samples=58 write: IOPS=12, BW=206KiB/s (211kB/s)(6032KiB/29303msec) clat (msec): min=30, max=162, avg=65.81, stdev=23.05 lat (msec): min=30, max=162, avg=65.82, stdev=23.05 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 36], 10.00th=[ 39], 20.00th=[ 46], | 30.00th=[ 53], 40.00th=[ 57], 50.00th=[ 62], 60.00th=[ 69], | 70.00th=[ 75], 80.00th=[ 85], 90.00th=[ 95], 95.00th=[ 109], | 99.00th=[ 140], 99.50th=[ 150], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 96, max= 320, per=24.79%, avg=203.78, stdev=53.61, samples=58 iops : min= 6, max= 20, avg=12.40, stdev= 3.43, samples=58 lat (msec) : 4=0.39%, 10=33.46%, 20=10.68%, 50=19.01%, 100=32.55% lat (msec) : 250=3.91% cpu : usr=0.00%, sys=3.23%, ctx=896, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27368: Fri Dec 20 03:20:31 2024 read: IOPS=12, BW=199KiB/s (203kB/s)(6032KiB/30362msec) clat (usec): min=3686, max=79285, avg=10714.33, stdev=9583.66 lat (usec): min=3687, max=79286, avg=10717.61, stdev=9583.55 clat percentiles (usec): | 1.00th=[ 3818], 5.00th=[ 4359], 10.00th=[ 4883], 20.00th=[ 5604], | 30.00th=[ 6063], 40.00th=[ 6587], 50.00th=[ 7373], 60.00th=[ 8455], | 70.00th=[10290], 80.00th=[12780], 90.00th=[20317], 95.00th=[31327], | 99.00th=[54264], 99.50th=[71828], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 32, max= 448, per=25.16%, avg=196.52, stdev=103.36, samples=60 iops : min= 2, max= 28, avg=12.03, stdev= 6.48, samples=60 write: IOPS=12, BW=206KiB/s (211kB/s)(6256KiB/30362msec) clat (msec): min=28, max=155, avg=67.00, stdev=22.51 lat (msec): min=28, max=155, avg=67.00, stdev=22.51 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 37], 10.00th=[ 41], 20.00th=[ 48], | 30.00th=[ 54], 40.00th=[ 59], 50.00th=[ 64], 60.00th=[ 69], | 70.00th=[ 77], 80.00th=[ 84], 90.00th=[ 99], 95.00th=[ 109], | 99.00th=[ 140], 99.50th=[ 148], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 96, max= 319, per=24.88%, avg=204.50, stdev=51.71, samples=60 iops : min= 6, max= 19, avg=12.53, stdev= 3.22, samples=60 lat (msec) : 4=1.04%, 10=32.81%, 20=9.90%, 50=16.67%, 100=34.90% lat (msec) : 250=4.69% cpu : usr=0.02%, sys=3.14%, ctx=920, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=782KiB/s (801kB/s), 188KiB/s-213KiB/s (192kB/s-219kB/s), io=23.4MiB (24.5MB), run=29303-30634msec WRITE: bw=823KiB/s (842kB/s), 206KiB/s-214KiB/s (211kB/s-219kB/s), io=24.6MiB (25.8MB), run=29303-30634msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27409: Fri Dec 20 03:20:39 2024 read: IOPS=1, BW=1916KiB/s (1962kB/s)(7168KiB/3742msec) clat (msec): min=40, max=1181, avg=220.66, stdev=424.98 lat (msec): min=40, max=1181, avg=220.66, stdev=424.98 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 45], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 50], 60.00th=[ 55], | 70.00th=[ 55], 80.00th=[ 129], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 2048, max= 6144, per=100.00%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 write: IOPS=1, BW=1368KiB/s (1401kB/s)(5120KiB/3742msec) clat (msec): min=109, max=1264, avg=433.48, stdev=481.98 lat (msec): min=110, max=1264, avg=433.54, stdev=481.98 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 110], 10.00th=[ 110], 20.00th=[ 110], | 30.00th=[ 171], 40.00th=[ 171], 50.00th=[ 178], 60.00th=[ 178], | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 1267], 95.00th=[ 1267], | 99.00th=[ 1267], 99.50th=[ 1267], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 2048, max= 4087, per=68.09%, avg=2727.67, stdev=1177.22, samples=3 iops : min= 2, max= 3, avg= 2.33, stdev= 0.58, samples=3 lat (msec) : 50=33.33%, 100=8.33%, 250=33.33%, 500=8.33% cpu : usr=0.00%, sys=2.97%, ctx=57, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27410: Fri Dec 20 03:20:39 2024 read: IOPS=0, BW=506KiB/s (518kB/s)(2048KiB/4046msec) clat (usec): min=28040, max=61848, avg=44944.36, stdev=23906.05 lat (usec): min=28042, max=61850, avg=44946.38, stdev=23905.44 clat percentiles (usec): | 1.00th=[27919], 5.00th=[27919], 10.00th=[27919], 20.00th=[27919], | 30.00th=[27919], 40.00th=[27919], 50.00th=[27919], 60.00th=[61604], | 70.00th=[61604], 80.00th=[61604], 90.00th=[61604], 95.00th=[61604], | 99.00th=[61604], 99.50th=[61604], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 2035, max= 2048, per=71.36%, avg=2041.50, stdev= 9.19, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2531KiB/s (2592kB/s)(10.0MiB/4046msec) clat (msec): min=61, max=2336, avg=393.90, stdev=692.72 lat (msec): min=61, max=2336, avg=393.97, stdev=692.73 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 62], 10.00th=[ 62], 20.00th=[ 72], | 30.00th=[ 86], 40.00th=[ 111], 50.00th=[ 130], 60.00th=[ 192], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 451], 95.00th=[ 2333], | 99.00th=[ 2333], 99.50th=[ 2333], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 4087, max= 6107, per=100.00%, avg=4596.50, stdev=1007.01, samples=4 iops : min= 3, max= 5, avg= 4.00, stdev= 0.82, samples=4 lat (msec) : 50=8.33%, 100=33.33%, 250=33.33%, 500=16.67% cpu : usr=0.00%, sys=3.19%, ctx=77, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27411: Fri Dec 20 03:20:39 2024 read: IOPS=0, BW=859KiB/s (879kB/s)(6144KiB/7156msec) clat (msec): min=35, max=1123, avg=255.38, stdev=427.82 lat (msec): min=35, max=1123, avg=255.38, stdev=427.82 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 61], 50.00th=[ 61], 60.00th=[ 106], | 70.00th=[ 159], 80.00th=[ 159], 90.00th=[ 1116], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 4096, max= 6144, per=100.00%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=0, BW=859KiB/s (879kB/s)(6144KiB/7156msec) clat (msec): min=121, max=2447, avg=926.02, stdev=1057.36 lat (msec): min=121, max=2447, avg=926.25, stdev=1057.58 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 122], 10.00th=[ 122], 20.00th=[ 165], | 30.00th=[ 165], 40.00th=[ 190], 50.00th=[ 190], 60.00th=[ 542], | 70.00th=[ 2089], 80.00th=[ 2089], 90.00th=[ 2433], 95.00th=[ 2433], | 99.00th=[ 2433], 99.50th=[ 2433], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 1723, max= 4096, per=74.66%, avg=2990.75, stdev=1283.11, samples=4 iops : min= 1, max= 4, avg= 2.75, stdev= 1.50, samples=4 lat (msec) : 50=16.67%, 100=8.33%, 250=41.67%, 750=8.33% cpu : usr=0.00%, sys=1.51%, ctx=62, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27412: Fri Dec 20 03:20:39 2024 read: IOPS=0, BW=1003KiB/s (1027kB/s)(5120KiB/5103msec) clat (msec): min=26, max=384, avg=115.94, stdev=151.22 lat (msec): min=26, max=384, avg=115.95, stdev=151.22 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 42], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 46], | 70.00th=[ 82], 80.00th=[ 82], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 2039, max= 6144, per=100.00%, avg=4091.50, stdev=2902.67, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 write: IOPS=1, BW=1405KiB/s (1438kB/s)(7168KiB/5103msec) clat (msec): min=74, max=2377, avg=637.30, stdev=842.04 lat (msec): min=74, max=2377, avg=637.38, stdev=842.04 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 75], 10.00th=[ 75], 20.00th=[ 88], | 30.00th=[ 178], 40.00th=[ 178], 50.00th=[ 207], 60.00th=[ 472], | 70.00th=[ 472], 80.00th=[ 1062], 90.00th=[ 2366], 95.00th=[ 2366], | 99.00th=[ 2366], 99.50th=[ 2366], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 2039, max= 6131, per=100.00%, avg=4088.67, stdev=2046.01, samples=3 iops : min= 1, max= 5, avg= 3.33, stdev= 2.08, samples=3 lat (msec) : 50=25.00%, 100=25.00%, 250=16.67%, 500=16.67% cpu : usr=0.02%, sys=2.02%, ctx=45, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2862KiB/s (2931kB/s), 506KiB/s-1916KiB/s (518kB/s-1962kB/s), io=20.0MiB (20.0MB), run=3742-7156msec WRITE: bw=4007KiB/s (4103kB/s), 859KiB/s-2531KiB/s (879kB/s-2592kB/s), io=28.0MiB (29.4MB), run=3742-7156msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27414: Fri Dec 20 03:20:40 2024 read: IOPS=1, BW=1208KiB/s (1237kB/s)(7168KiB/5936msec) clat (usec): min=537, max=1281.4k, avg=654691.11, stdev=570601.96 lat (usec): min=539, max=1281.5k, avg=654692.85, stdev=570601.91 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 537], 10.00th=[ 537], | 20.00th=[ 90702], 30.00th=[ 189793], 40.00th=[ 189793], | 50.00th=[ 624952], 60.00th=[1115685], 70.00th=[1115685], | 80.00th=[1283458], 90.00th=[1283458], 95.00th=[1283458], | 99.00th=[1283458], 99.50th=[1283458], 99.90th=[1283458], | 99.95th=[1283458], 99.99th=[1283458] bw ( KiB/s): min= 1729, max= 4096, per=97.19%, avg=2393.80, stdev=961.53, samples=5 iops : min= 1, max= 4, avg= 2.20, stdev= 1.10, samples=5 write: IOPS=0, BW=863KiB/s (883kB/s)(5120KiB/5936msec) clat (msec): min=30, max=1022, avg=259.02, stdev=428.28 lat (msec): min=30, max=1022, avg=259.10, stdev=428.27 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 100], 60.00th=[ 100], | 70.00th=[ 111], 80.00th=[ 111], 90.00th=[ 1020], 95.00th=[ 1020], | 99.00th=[ 1020], 99.50th=[ 1020], 99.90th=[ 1020], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 2048, max= 4096, per=79.20%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 lat (usec) : 750=8.33% lat (msec) : 50=16.67%, 100=16.67%, 250=16.67%, 750=8.33% cpu : usr=0.02%, sys=17.52%, ctx=1063, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27415: Fri Dec 20 03:20:40 2024 read: IOPS=0, BW=345KiB/s (353kB/s)(2048KiB/5940msec) clat (msec): min=91, max=200, avg=145.96, stdev=76.46 lat (msec): min=91, max=200, avg=145.96, stdev=76.46 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 92], 10.00th=[ 92], 20.00th=[ 92], | 30.00th=[ 92], 40.00th=[ 92], 50.00th=[ 92], 60.00th=[ 201], | 70.00th=[ 201], 80.00th=[ 201], 90.00th=[ 201], 95.00th=[ 201], | 99.00th=[ 201], 99.50th=[ 201], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 2043, max= 2048, per=83.05%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=1, BW=1724KiB/s (1765kB/s)(10.0MiB/5940msec) clat (msec): min=36, max=2437, avg=555.81, stdev=787.46 lat (msec): min=36, max=2438, avg=555.89, stdev=787.47 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 52], | 30.00th=[ 54], 40.00th=[ 60], 50.00th=[ 81], 60.00th=[ 144], | 70.00th=[ 961], 80.00th=[ 961], 90.00th=[ 1234], 95.00th=[ 2433], | 99.00th=[ 2433], 99.50th=[ 2433], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 1721, max=10219, per=100.00%, avg=6023.67, stdev=4250.02, samples=3 iops : min= 1, max= 9, avg= 5.00, stdev= 4.00, samples=3 lat (msec) : 50=8.33%, 100=41.67%, 250=16.67%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=6.87%, ctx=116, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27416: Fri Dec 20 03:20:40 2024 read: IOPS=0, BW=739KiB/s (757kB/s)(6144KiB/8315msec) clat (msec): min=67, max=1278, avg=819.08, stdev=548.32 lat (msec): min=67, max=1278, avg=819.08, stdev=548.32 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 186], | 30.00th=[ 186], 40.00th=[ 986], 50.00th=[ 986], 60.00th=[ 1116], | 70.00th=[ 1267], 80.00th=[ 1267], 90.00th=[ 1284], 95.00th=[ 1284], | 99.00th=[ 1284], 99.50th=[ 1284], 99.90th=[ 1284], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 1762, max= 2048, per=76.11%, avg=1874.60, stdev=154.22, samples=5 iops : min= 1, max= 2, avg= 1.20, stdev= 0.45, samples=5 write: IOPS=0, BW=739KiB/s (757kB/s)(6144KiB/8315msec) clat (msec): min=33, max=2467, avg=552.01, stdev=955.60 lat (msec): min=33, max=2467, avg=552.07, stdev=955.60 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 44], | 30.00th=[ 44], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 176], | 70.00th=[ 518], 80.00th=[ 518], 90.00th=[ 2467], 95.00th=[ 2467], | 99.00th=[ 2467], 99.50th=[ 2467], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 1762, max= 4096, per=67.96%, avg=2343.20, stdev=990.22, samples=5 iops : min= 1, max= 4, avg= 2.00, stdev= 1.22, samples=5 lat (msec) : 50=16.67%, 100=16.67%, 250=16.67%, 750=8.33%, 1000=8.33% cpu : usr=0.01%, sys=10.81%, ctx=1224, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27417: Fri Dec 20 03:20:40 2024 read: IOPS=1, BW=1047KiB/s (1072kB/s)(5120KiB/4892msec) clat (usec): min=351, max=1151.7k, avg=309454.96, stdev=488843.41 lat (usec): min=352, max=1151.7k, avg=309456.37, stdev=488843.65 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 351], 10.00th=[ 351], | 20.00th=[ 351], 30.00th=[ 502], 40.00th=[ 502], | 50.00th=[ 73925], 60.00th=[ 73925], 70.00th=[ 320865], | 80.00th=[ 320865], 90.00th=[1149240], 95.00th=[1149240], | 99.00th=[1149240], 99.50th=[1149240], 99.90th=[1149240], | 99.95th=[1149240], 99.99th=[1149240] bw ( KiB/s): min= 2048, max= 6107, per=100.00%, avg=4077.50, stdev=2870.15, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=1, BW=1465KiB/s (1500kB/s)(7168KiB/4892msec) clat (msec): min=32, max=2483, avg=474.79, stdev=894.05 lat (msec): min=32, max=2483, avg=474.86, stdev=894.06 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33], | 30.00th=[ 99], 40.00th=[ 99], 50.00th=[ 120], 60.00th=[ 169], | 70.00th=[ 169], 80.00th=[ 388], 90.00th=[ 2500], 95.00th=[ 2500], | 99.00th=[ 2500], 99.50th=[ 2500], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 2035, max= 6144, per=100.00%, avg=3578.50, stdev=1966.56, samples=4 iops : min= 1, max= 6, avg= 3.00, stdev= 2.45, samples=4 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=16.67%, 100=16.67%, 250=16.67%, 500=16.67% cpu : usr=0.00%, sys=9.94%, ctx=432, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2463KiB/s (2522kB/s), 345KiB/s-1208KiB/s (353kB/s-1237kB/s), io=20.0MiB (20.0MB), run=4892-8315msec WRITE: bw=3448KiB/s (3531kB/s), 739KiB/s-1724KiB/s (757kB/s-1765kB/s), io=28.0MiB (29.4MB), run=4892-8315msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27431: Fri Dec 20 03:20:45 2024 read: IOPS=0, BW=4076KiB/s (4173kB/s)(12.0MiB/3015msec) clat (msec): min=110, max=2127, avg=967.45, stdev=1042.19 lat (msec): min=110, max=2127, avg=967.45, stdev=1042.19 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 667], 50.00th=[ 667], 60.00th=[ 667], | 70.00th=[ 2123], 80.00th=[ 2123], 90.00th=[ 2123], 95.00th=[ 2123], | 99.00th=[ 2123], 99.50th=[ 2123], 99.90th=[ 2123], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=2.69%, ctx=39, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27433: Fri Dec 20 03:20:45 2024 read: IOPS=0, BW=1348KiB/s (1380kB/s)(4096KiB/3039msec) clat (nsec): min=212036k, max=212036k, avg=212036053.00, stdev= 0.00 lat (nsec): min=212039k, max=212039k, avg=212038751.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 213], 10.00th=[ 213], 20.00th=[ 213], | 30.00th=[ 213], 40.00th=[ 213], 50.00th=[ 213], 60.00th=[ 213], | 70.00th=[ 213], 80.00th=[ 213], 90.00th=[ 213], 95.00th=[ 213], | 99.00th=[ 213], 99.50th=[ 213], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2696KiB/s (2760kB/s)(8192KiB/3039msec) clat (msec): min=597, max=2180, avg=1389.02, stdev=1119.99 lat (msec): min=597, max=2181, avg=1389.30, stdev=1119.85 clat percentiles (msec): | 1.00th=[ 600], 5.00th=[ 600], 10.00th=[ 600], 20.00th=[ 600], | 30.00th=[ 600], 40.00th=[ 600], 50.00th=[ 600], 60.00th=[ 2165], | 70.00th=[ 2165], 80.00th=[ 2165], 90.00th=[ 2165], 95.00th=[ 2165], | 99.00th=[ 2165], 99.50th=[ 2165], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 8192, max= 8192, per=79.05%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.03%, sys=2.76%, ctx=49, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27435: Fri Dec 20 03:20:45 2024 write: IOPS=0, BW=3886KiB/s (3979kB/s)(12.0MiB/3162msec) clat (msec): min=385, max=2055, avg=1019.56, stdev=904.26 lat (msec): min=386, max=2055, avg=1019.98, stdev=904.13 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 388], 10.00th=[ 388], 20.00th=[ 388], | 30.00th=[ 388], 40.00th=[ 617], 50.00th=[ 617], 60.00th=[ 617], | 70.00th=[ 2056], 80.00th=[ 2056], 90.00th=[ 2056], 95.00th=[ 2056], | 99.00th=[ 2056], 99.50th=[ 2056], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 6243, max= 8192, per=69.65%, avg=7217.50, stdev=1378.15, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.06%, sys=3.51%, ctx=52, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27437: Fri Dec 20 03:20:45 2024 write: IOPS=1, BW=4262KiB/s (4365kB/s)(12.0MiB/2883msec) clat (msec): min=228, max=1927, avg=940.08, stdev=882.62 lat (msec): min=228, max=1928, avg=940.48, stdev=882.57 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 228], 10.00th=[ 228], 20.00th=[ 228], | 30.00th=[ 228], 40.00th=[ 667], 50.00th=[ 667], 60.00th=[ 667], | 70.00th=[ 1921], 80.00th=[ 1921], 90.00th=[ 1921], 95.00th=[ 1921], | 99.00th=[ 1921], 99.50th=[ 1921], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 8192, max= 8192, per=79.05%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.21%, sys=5.20%, ctx=110, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5391KiB/s (5521kB/s), 1348KiB/s-4076KiB/s (1380kB/s-4173kB/s), io=16.0MiB (16.8MB), run=3015-3039msec WRITE: bw=10.1MiB/s (10.6MB/s), 2696KiB/s-4262KiB/s (2760kB/s-4365kB/s), io=32.0MiB (33.6MB), run=2883-3162msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27429: Fri Dec 20 03:20:45 2024 read: IOPS=1, BW=6505KiB/s (6661kB/s)(12.0MiB/1889msec) clat (usec): min=1895, max=1515.4k, avg=606520.22, stdev=801296.88 lat (usec): min=1897, max=1515.4k, avg=606521.91, stdev=801296.46 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 1893], 10.00th=[ 1893], | 20.00th=[ 1893], 30.00th=[ 1893], 40.00th=[ 304088], | 50.00th=[ 304088], 60.00th=[ 304088], 70.00th=[1518339], | 80.00th=[1518339], 90.00th=[1518339], 95.00th=[1518339], | 99.00th=[1518339], 99.50th=[1518339], 99.90th=[1518339], | 99.95th=[1518339], 99.99th=[1518339] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 2=33.33%, 500=33.33% cpu : usr=0.05%, sys=45.13%, ctx=411, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27430: Fri Dec 20 03:20:45 2024 read: IOPS=0, BW=1277KiB/s (1308kB/s)(4096KiB/3207msec) clat (nsec): min=330929k, max=330929k, avg=330928670.00, stdev= 0.00 lat (nsec): min=330932k, max=330932k, avg=330931532.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 330], | 30.00th=[ 330], 40.00th=[ 330], 50.00th=[ 330], 60.00th=[ 330], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2554KiB/s (2616kB/s)(8192KiB/3207msec) clat (msec): min=629, max=2185, avg=1407.24, stdev=1100.58 lat (msec): min=629, max=2191, avg=1410.27, stdev=1104.52 clat percentiles (msec): | 1.00th=[ 625], 5.00th=[ 625], 10.00th=[ 625], 20.00th=[ 625], | 30.00th=[ 625], 40.00th=[ 625], 50.00th=[ 625], 60.00th=[ 2198], | 70.00th=[ 2198], 80.00th=[ 2198], 90.00th=[ 2198], 95.00th=[ 2198], | 99.00th=[ 2198], 99.50th=[ 2198], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 6068, max= 6068, per=59.39%, avg=6068.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.09%, sys=10.39%, ctx=282, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27432: Fri Dec 20 03:20:45 2024 write: IOPS=1, BW=5163KiB/s (5287kB/s)(12.0MiB/2380msec) clat (msec): min=108, max=1648, avg=759.40, stdev=796.91 lat (msec): min=108, max=1648, avg=759.71, stdev=796.87 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 109], 10.00th=[ 109], 20.00th=[ 109], | 30.00th=[ 109], 40.00th=[ 523], 50.00th=[ 523], 60.00th=[ 523], | 70.00th=[ 1653], 80.00th=[ 1653], 90.00th=[ 1653], 95.00th=[ 1653], | 99.00th=[ 1653], 99.50th=[ 1653], 99.90th=[ 1653], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 8175, max= 8175, per=80.01%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=9.96%, ctx=86, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27436: Fri Dec 20 03:20:45 2024 write: IOPS=2, BW=11.7MiB/s (12.3MB/s)(12.0MiB/1025msec) clat (msec): min=231, max=389, avg=331.77, stdev=87.40 lat (msec): min=231, max=390, avg=332.29, stdev=87.52 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 232], 10.00th=[ 232], 20.00th=[ 232], | 30.00th=[ 232], 40.00th=[ 376], 50.00th=[ 376], 60.00th=[ 376], | 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 8192, max= 8192, per=80.18%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=28.61%, ctx=92, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5109KiB/s (5231kB/s), 1277KiB/s-6505KiB/s (1308kB/s-6661kB/s), io=16.0MiB (16.8MB), run=1889-3207msec WRITE: bw=9.98MiB/s (10.5MB/s), 2554KiB/s-11.7MiB/s (2616kB/s-12.3MB/s), io=32.0MiB (33.6MB), run=1025-3207msec