== sanity test 398b: DIO and buffer IO race ============== 08:39:39 (1726835979) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.33547 s, 21.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=13164: Fri Sep 20 08:40:31 2024 read: IOPS=31, BW=126KiB/s (129kB/s)(5992KiB/47546msec) clat (usec): min=59, max=210096, avg=13651.33, stdev=13896.15 lat (usec): min=60, max=210097, avg=13652.50, stdev=13896.18 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 81], 10.00th=[ 89], 20.00th=[ 115], | 30.00th=[ 6194], 40.00th=[ 9241], 50.00th=[ 11731], 60.00th=[ 14222], | 70.00th=[ 17433], 80.00th=[ 22414], 90.00th=[ 31065], 95.00th=[ 37487], | 99.00th=[ 54264], 99.50th=[ 63177], 99.90th=[111674], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 23, max= 255, per=24.50%, avg=123.70, stdev=42.27, samples=94 iops : min= 5, max= 63, avg=30.57, stdev=10.65, samples=94 write: IOPS=33, BW=132KiB/s (136kB/s)(6296KiB/47546msec) clat (usec): min=596, max=409818, avg=17021.92, stdev=20932.38 lat (usec): min=597, max=409819, avg=17023.53, stdev=20932.33 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 725], 10.00th=[ 775], 20.00th=[ 1123], | 30.00th=[ 2933], 40.00th=[ 6521], 50.00th=[ 12387], 60.00th=[ 17171], | 70.00th=[ 21890], 80.00th=[ 28967], 90.00th=[ 40109], 95.00th=[ 50070], | 99.00th=[ 81265], 99.50th=[111674], 99.90th=[183501], 99.95th=[408945], | 99.99th=[408945] bw ( KiB/s): min= 64, max= 312, per=24.89%, avg=131.43, stdev=44.48, samples=93 iops : min= 16, max= 78, avg=32.52, stdev=11.14, samples=93 lat (usec) : 100=7.36%, 250=5.24%, 500=0.23%, 750=4.10%, 1000=5.99% lat (msec) : 2=3.48%, 4=5.21%, 10=12.47%, 20=27.21%, 50=25.39% lat (msec) : 100=2.96%, 250=0.33%, 500=0.03% cpu : usr=0.12%, sys=10.34%, ctx=5135, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13165: Fri Sep 20 08:40:31 2024 read: IOPS=31, BW=126KiB/s (129kB/s)(5940KiB/46994msec) clat (usec): min=44, max=207259, avg=12897.53, stdev=13143.48 lat (usec): min=44, max=207261, avg=12901.85, stdev=13143.31 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 80], 10.00th=[ 90], 20.00th=[ 115], | 30.00th=[ 5800], 40.00th=[ 8979], 50.00th=[ 10945], 60.00th=[ 13173], | 70.00th=[ 16450], 80.00th=[ 20841], 90.00th=[ 28967], 95.00th=[ 35914], | 99.00th=[ 53216], 99.50th=[ 61080], 99.90th=[ 92799], 99.95th=[206570], | 99.99th=[206570] bw ( KiB/s): min= 16, max= 255, per=24.92%, avg=125.83, stdev=47.73, samples=93 iops : min= 4, max= 63, avg=31.16, stdev=11.96, samples=93 write: IOPS=33, BW=135KiB/s (138kB/s)(6348KiB/46994msec) clat (usec): min=468, max=378086, avg=17350.95, stdev=20598.09 lat (usec): min=469, max=378088, avg=17355.39, stdev=20596.55 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 725], 10.00th=[ 783], 20.00th=[ 1139], | 30.00th=[ 2868], 40.00th=[ 7439], 50.00th=[ 13042], 60.00th=[ 17695], | 70.00th=[ 22676], 80.00th=[ 29230], 90.00th=[ 41157], 95.00th=[ 50594], | 99.00th=[ 80217], 99.50th=[ 93848], 99.90th=[187696], 99.95th=[379585], | 99.99th=[379585] bw ( KiB/s): min= 40, max= 264, per=25.35%, avg=133.87, stdev=41.73, samples=93 iops : min= 10, max= 66, avg=33.18, stdev=10.47, samples=93 lat (usec) : 50=0.03%, 100=7.42%, 250=5.11%, 500=0.29%, 750=4.07% lat (usec) : 1000=5.76% lat (msec) : 2=4.46%, 4=4.04%, 10=13.41%, 20=27.08%, 50=24.93% lat (msec) : 100=3.16%, 250=0.20%, 500=0.03% cpu : usr=0.11%, sys=10.30%, ctx=4922, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13166: Fri Sep 20 08:40:31 2024 read: IOPS=31, BW=128KiB/s (131kB/s)(6028KiB/47154msec) clat (usec): min=51, max=195381, avg=13309.19, stdev=13695.88 lat (usec): min=51, max=195381, avg=13311.54, stdev=13695.38 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 78], 10.00th=[ 88], 20.00th=[ 110], | 30.00th=[ 4883], 40.00th=[ 8586], 50.00th=[ 10814], 60.00th=[ 13829], | 70.00th=[ 17433], 80.00th=[ 22414], 90.00th=[ 30802], 95.00th=[ 38011], | 99.00th=[ 50594], 99.50th=[ 60556], 99.90th=[109577], 99.95th=[196084], | 99.99th=[196084] bw ( KiB/s): min= 16, max= 264, per=25.51%, avg=128.84, stdev=47.23, samples=93 iops : min= 4, max= 66, avg=31.84, stdev=11.82, samples=93 write: IOPS=33, BW=133KiB/s (136kB/s)(6260KiB/47154msec) clat (usec): min=508, max=484878, avg=17116.42, stdev=21916.01 lat (usec): min=509, max=484880, avg=17118.27, stdev=21916.09 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 725], 10.00th=[ 775], 20.00th=[ 1106], | 30.00th=[ 2540], 40.00th=[ 5997], 50.00th=[ 11207], 60.00th=[ 16581], | 70.00th=[ 22414], 80.00th=[ 30278], 90.00th=[ 41681], 95.00th=[ 52167], | 99.00th=[ 76022], 99.50th=[ 94897], 99.90th=[152044], 99.95th=[484443], | 99.99th=[484443] bw ( KiB/s): min= 47, max= 247, per=25.30%, avg=133.59, stdev=40.10, samples=93 iops : min= 11, max= 61, avg=33.03, stdev=10.02, samples=93 lat (usec) : 100=8.33%, 250=4.79%, 500=0.39%, 750=4.10%, 1000=6.15% lat (msec) : 2=4.13%, 4=4.59%, 10=14.23%, 20=24.25%, 50=25.39% lat (msec) : 100=3.35%, 250=0.26%, 500=0.03% cpu : usr=0.13%, sys=10.30%, ctx=4893, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13167: Fri Sep 20 08:40:31 2024 read: IOPS=32, BW=130KiB/s (134kB/s)(6052KiB/46391msec) clat (usec): min=59, max=83408, avg=12857.15, stdev=12256.33 lat (usec): min=59, max=83409, avg=12859.14, stdev=12255.66 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 81], 10.00th=[ 90], 20.00th=[ 114], | 30.00th=[ 5276], 40.00th=[ 8717], 50.00th=[10814], 60.00th=[13698], | 70.00th=[16712], 80.00th=[21627], 90.00th=[29230], 95.00th=[36963], | 99.00th=[49021], 99.50th=[56361], 99.90th=[77071], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 56, max= 272, per=26.31%, avg=132.88, stdev=45.00, samples=90 iops : min= 14, max= 68, avg=32.91, stdev=11.32, samples=90 write: IOPS=33, BW=134KiB/s (138kB/s)(6236KiB/46391msec) clat (usec): min=576, max=343305, avg=17007.25, stdev=19774.27 lat (usec): min=577, max=343308, avg=17008.68, stdev=19774.26 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 1139], | 30.00th=[ 2900], 40.00th=[ 6783], 50.00th=[ 11994], 60.00th=[ 16909], | 70.00th=[ 22152], 80.00th=[ 29492], 90.00th=[ 40633], 95.00th=[ 51119], | 99.00th=[ 77071], 99.50th=[ 95945], 99.90th=[149947], 99.95th=[341836], | 99.99th=[341836] bw ( KiB/s): min= 24, max= 248, per=25.67%, avg=135.53, stdev=44.69, samples=91 iops : min= 6, max= 62, avg=33.58, stdev=11.16, samples=91 lat (usec) : 100=7.42%, 250=5.60%, 500=0.33%, 750=3.58%, 1000=6.41% lat (msec) : 2=3.74%, 4=4.69%, 10=14.23%, 20=25.81%, 50=25.03% lat (msec) : 100=2.99%, 250=0.13%, 500=0.03% cpu : usr=0.14%, sys=10.28%, ctx=4845, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=505KiB/s (517kB/s), 126KiB/s-130KiB/s (129kB/s-134kB/s), io=23.4MiB (24.6MB), run=46391-47546msec WRITE: bw=529KiB/s (541kB/s), 132KiB/s-135KiB/s (136kB/s-138kB/s), io=24.6MiB (25.7MB), run=46391-47546msec rand-rw: (groupid=0, jobs=1): err= 0: pid=13168: Fri Sep 20 08:42:32 2024 read: IOPS=8, BW=35.7KiB/s (36.6kB/s)(5992KiB/167846msec) clat (msec): min=3, max=426, avg=13.92, stdev=15.65 lat (msec): min=3, max=426, avg=13.93, stdev=15.65 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 17], 90.00th=[ 28], 95.00th=[ 40], | 99.00th=[ 61], 99.50th=[ 71], 99.90th=[ 113], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 7, max= 167, per=26.16%, avg=37.15, stdev=23.92, samples=319 iops : min= 1, max= 41, avg= 9.01, stdev= 6.04, samples=319 write: IOPS=9, BW=37.5KiB/s (38.4kB/s)(6296KiB/167846msec) clat (msec): min=33, max=296, avg=93.20, stdev=36.94 lat (msec): min=33, max=296, avg=93.20, stdev=36.94 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 49], 10.00th=[ 55], 20.00th=[ 64], | 30.00th=[ 71], 40.00th=[ 79], 50.00th=[ 86], 60.00th=[ 94], | 70.00th=[ 105], 80.00th=[ 117], 90.00th=[ 140], 95.00th=[ 165], | 99.00th=[ 220], 99.50th=[ 247], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 7, max= 64, per=24.98%, avg=37.22, stdev=11.99, samples=334 iops : min= 1, max= 16, avg= 9.02, stdev= 3.07, samples=334 lat (msec) : 4=0.10%, 10=25.33%, 20=15.82%, 50=9.83%, 100=31.41% lat (msec) : 250=17.25%, 500=0.26% cpu : usr=0.05%, sys=2.29%, ctx=3549, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13169: Fri Sep 20 08:42:32 2024 read: IOPS=8, BW=35.3KiB/s (36.1kB/s)(5940KiB/168350msec) clat (msec): min=3, max=378, avg=14.02, stdev=15.25 lat (msec): min=3, max=378, avg=14.02, stdev=15.25 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 14], 80.00th=[ 17], 90.00th=[ 29], 95.00th=[ 38], | 99.00th=[ 67], 99.50th=[ 81], 99.90th=[ 117], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 7, max= 112, per=25.71%, avg=36.50, stdev=22.87, samples=321 iops : min= 1, max= 28, avg= 8.83, stdev= 5.75, samples=321 write: IOPS=9, BW=37.7KiB/s (38.6kB/s)(6348KiB/168350msec) clat (msec): min=34, max=322, avg=92.79, stdev=38.01 lat (msec): min=34, max=322, avg=92.79, stdev=38.01 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 49], 10.00th=[ 55], 20.00th=[ 64], | 30.00th=[ 71], 40.00th=[ 78], 50.00th=[ 85], 60.00th=[ 93], | 70.00th=[ 104], 80.00th=[ 117], 90.00th=[ 140], 95.00th=[ 167], | 99.00th=[ 230], 99.50th=[ 266], 99.90th=[ 296], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 15, max= 72, per=25.07%, avg=37.36, stdev=11.97, samples=335 iops : min= 3, max= 18, avg= 9.05, stdev= 3.06, samples=335 lat (msec) : 4=0.26%, 10=25.29%, 20=14.75%, 50=10.38%, 100=31.97% lat (msec) : 250=16.93%, 500=0.42% cpu : usr=0.03%, sys=2.25%, ctx=3495, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13170: Fri Sep 20 08:42:32 2024 read: IOPS=9, BW=36.1KiB/s (36.0kB/s)(6028KiB/167014msec) clat (msec): min=3, max=130, avg=14.22, stdev=12.05 lat (msec): min=3, max=130, avg=14.23, stdev=12.05 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 14], 80.00th=[ 19], 90.00th=[ 32], 95.00th=[ 40], | 99.00th=[ 57], 99.50th=[ 68], 99.90th=[ 97], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 7, max= 144, per=26.17%, avg=37.16, stdev=22.51, samples=321 iops : min= 1, max= 36, avg= 8.99, stdev= 5.65, samples=321 write: IOPS=9, BW=37.5KiB/s (38.4kB/s)(6260KiB/167014msec) clat (msec): min=34, max=456, avg=92.84, stdev=38.39 lat (msec): min=34, max=456, avg=92.84, stdev=38.39 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 48], 10.00th=[ 55], 20.00th=[ 64], | 30.00th=[ 72], 40.00th=[ 79], 50.00th=[ 85], 60.00th=[ 93], | 70.00th=[ 103], 80.00th=[ 116], 90.00th=[ 140], 95.00th=[ 165], | 99.00th=[ 228], 99.50th=[ 255], 99.90th=[ 317], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 7, max= 72, per=24.95%, avg=37.17, stdev=12.41, samples=333 iops : min= 1, max= 18, avg= 8.99, stdev= 3.20, samples=333 lat (msec) : 4=0.29%, 10=25.26%, 20=14.49%, 50=11.00%, 100=32.29% lat (msec) : 250=16.41%, 500=0.26% cpu : usr=0.05%, sys=2.27%, ctx=3554, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13171: Fri Sep 20 08:42:32 2024 read: IOPS=9, BW=36.2KiB/s (37.1kB/s)(6052KiB/167049msec) clat (msec): min=3, max=129, avg=13.71, stdev=11.24 lat (msec): min=3, max=129, avg=13.71, stdev=11.24 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 14], 80.00th=[ 18], 90.00th=[ 29], 95.00th=[ 38], | 99.00th=[ 59], 99.50th=[ 68], 99.90th=[ 99], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 7, max= 112, per=26.63%, avg=37.82, stdev=21.12, samples=316 iops : min= 1, max= 28, avg= 9.11, stdev= 5.29, samples=316 write: IOPS=9, BW=37.3KiB/s (38.2kB/s)(6236KiB/167049msec) clat (msec): min=28, max=456, avg=93.67, stdev=40.87 lat (msec): min=28, max=456, avg=93.67, stdev=40.87 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 48], 10.00th=[ 54], 20.00th=[ 63], | 30.00th=[ 70], 40.00th=[ 77], 50.00th=[ 86], 60.00th=[ 93], | 70.00th=[ 103], 80.00th=[ 118], 90.00th=[ 144], 95.00th=[ 176], | 99.00th=[ 236], 99.50th=[ 257], 99.90th=[ 305], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 7, max= 72, per=24.81%, avg=36.97, stdev=12.41, samples=333 iops : min= 1, max= 18, avg= 8.90, stdev= 3.18, samples=333 lat (msec) : 4=0.10%, 10=25.33%, 20=15.82%, 50=10.94%, 100=31.09% lat (msec) : 250=16.47%, 500=0.26% cpu : usr=0.05%, sys=2.28%, ctx=3563, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=143KiB/s (146kB/s), 35.3KiB/s-36.2KiB/s (36.1kB/s-37.1kB/s), io=23.4MiB (24.6MB), run=167014-168350msec WRITE: bw=149KiB/s (153kB/s), 37.3KiB/s-37.7KiB/s (38.2kB/s-38.6kB/s), io=24.6MiB (25.7MB), run=167014-168350msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=13318: Fri Sep 20 08:42:50 2024 read: IOPS=21, BW=347KiB/s (356kB/s)(5920KiB/17043msec) clat (usec): min=67, max=108462, avg=33761.73, stdev=26980.15 lat (usec): min=68, max=108464, avg=33763.11, stdev=26980.20 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 86], 10.00th=[ 94], 20.00th=[ 114], | 30.00th=[ 578], 40.00th=[ 31589], 50.00th=[ 38536], 60.00th=[ 43779], | 70.00th=[ 50594], 80.00th=[ 56886], 90.00th=[ 67634], 95.00th=[ 74974], | 99.00th=[ 98042], 99.50th=[104334], 99.90th=[108528], 99.95th=[108528], | 99.99th=[108528] bw ( KiB/s): min= 159, max= 608, per=24.66%, avg=345.44, stdev=97.50, samples=34 iops : min= 9, max= 38, avg=21.18, stdev= 6.09, samples=34 write: IOPS=23, BW=374KiB/s (383kB/s)(6368KiB/17043msec) clat (usec): min=886, max=100279, avg=11087.26, stdev=15148.27 lat (usec): min=887, max=100282, avg=11089.37, stdev=15148.23 clat percentiles (usec): | 1.00th=[ 963], 5.00th=[ 1057], 10.00th=[ 1123], 20.00th=[ 1287], | 30.00th=[ 1614], 40.00th=[ 2704], 50.00th=[ 4621], 60.00th=[ 9110], | 70.00th=[ 12518], 80.00th=[ 18482], 90.00th=[ 27132], 95.00th=[ 40109], | 99.00th=[ 82314], 99.50th=[ 98042], 99.90th=[100140], 99.95th=[100140], | 99.99th=[100140] bw ( KiB/s): min= 95, max= 736, per=25.22%, avg=371.74, stdev=155.05, samples=34 iops : min= 5, max= 46, avg=22.82, stdev= 9.77, samples=34 lat (usec) : 100=6.12%, 250=8.07%, 500=0.13%, 750=0.39%, 1000=1.43% lat (msec) : 2=17.45%, 4=6.77%, 10=8.20%, 20=10.55%, 50=24.61% lat (msec) : 100=15.89%, 250=0.39% cpu : usr=0.01%, sys=10.10%, ctx=1864, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13319: Fri Sep 20 08:42:51 2024 read: IOPS=21, BW=349KiB/s (357kB/s)(5744KiB/16473msec) clat (usec): min=59, max=159909, avg=33401.95, stdev=27510.16 lat (usec): min=60, max=159910, avg=33405.96, stdev=27510.43 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 85], 10.00th=[ 94], 20.00th=[ 111], | 30.00th=[ 388], 40.00th=[ 32375], 50.00th=[ 39584], 60.00th=[ 44303], | 70.00th=[ 49021], 80.00th=[ 55313], 90.00th=[ 64226], 95.00th=[ 72877], | 99.00th=[106431], 99.50th=[145753], 99.90th=[160433], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 63, max= 576, per=25.63%, avg=359.10, stdev=98.27, samples=31 iops : min= 3, max= 36, avg=21.97, stdev= 6.21, samples=31 write: IOPS=24, BW=397KiB/s (407kB/s)(6544KiB/16473msec) clat (usec): min=884, max=191131, avg=10742.94, stdev=18073.06 lat (usec): min=884, max=191133, avg=10745.09, stdev=18073.02 clat percentiles (usec): | 1.00th=[ 922], 5.00th=[ 1029], 10.00th=[ 1090], 20.00th=[ 1205], | 30.00th=[ 1434], 40.00th=[ 2311], 50.00th=[ 3818], 60.00th=[ 7046], | 70.00th=[ 11076], 80.00th=[ 15270], 90.00th=[ 27657], 95.00th=[ 38536], | 99.00th=[ 91751], 99.50th=[111674], 99.90th=[191890], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 128, max= 922, per=26.44%, avg=389.69, stdev=179.82, samples=32 iops : min= 8, max= 57, avg=23.91, stdev=11.21, samples=32 lat (usec) : 100=5.73%, 250=8.07%, 500=0.52%, 750=0.13%, 1000=1.95% lat (msec) : 2=18.75%, 4=7.55%, 10=9.37%, 20=9.24%, 50=24.09% lat (msec) : 100=13.67%, 250=0.91% cpu : usr=0.13%, sys=9.57%, ctx=1776, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13320: Fri Sep 20 08:42:51 2024 read: IOPS=23, BW=369KiB/s (377kB/s)(6256KiB/16972msec) clat (usec): min=64, max=128526, avg=32189.62, stdev=27275.48 lat (usec): min=65, max=128526, avg=32190.90, stdev=27275.48 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 80], 10.00th=[ 86], 20.00th=[ 102], | 30.00th=[ 128], 40.00th=[ 31065], 50.00th=[ 37487], 60.00th=[ 42206], | 70.00th=[ 49546], 80.00th=[ 55313], 90.00th=[ 63177], 95.00th=[ 73925], | 99.00th=[ 99091], 99.50th=[109577], 99.90th=[128451], 99.95th=[128451], | 99.99th=[128451] bw ( KiB/s): min= 160, max= 733, per=26.30%, avg=368.52, stdev=130.13, samples=33 iops : min= 10, max= 45, avg=22.58, stdev= 8.00, samples=33 write: IOPS=22, BW=355KiB/s (364kB/s)(6032KiB/16972msec) clat (usec): min=890, max=155493, avg=11527.85, stdev=16286.84 lat (usec): min=890, max=155493, avg=11530.80, stdev=16286.55 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 1045], 10.00th=[ 1123], 20.00th=[ 1287], | 30.00th=[ 1909], 40.00th=[ 2999], 50.00th=[ 4883], 60.00th=[ 9896], | 70.00th=[ 13435], 80.00th=[ 19530], 90.00th=[ 27657], 95.00th=[ 34866], | 99.00th=[ 84411], 99.50th=[126354], 99.90th=[156238], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 127, max= 702, per=23.69%, avg=349.21, stdev=141.28, samples=33 iops : min= 7, max= 43, avg=21.36, stdev= 8.79, samples=33 lat (usec) : 100=9.51%, 250=7.16%, 500=0.52%, 1000=2.08% lat (msec) : 2=13.93%, 4=7.03%, 10=7.94%, 20=9.77%, 50=26.17% lat (msec) : 100=15.23%, 250=0.65% cpu : usr=0.04%, sys=9.87%, ctx=1858, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13321: Fri Sep 20 08:42:51 2024 read: IOPS=22, BW=353KiB/s (361kB/s)(6032KiB/17094msec) clat (usec): min=63, max=94356, avg=32955.56, stdev=25450.26 lat (usec): min=63, max=94358, avg=32956.83, stdev=25450.29 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 84], 10.00th=[ 95], 20.00th=[ 115], | 30.00th=[ 545], 40.00th=[32637], 50.00th=[38011], 60.00th=[43779], | 70.00th=[48497], 80.00th=[56361], 90.00th=[63701], 95.00th=[69731], | 99.00th=[87557], 99.50th=[92799], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 95, max= 540, per=24.71%, avg=346.18, stdev=91.56, samples=34 iops : min= 5, max= 33, avg=21.15, stdev= 5.73, samples=34 write: IOPS=22, BW=366KiB/s (375kB/s)(6256KiB/17094msec) clat (usec): min=772, max=106875, avg=11662.46, stdev=14201.22 lat (usec): min=774, max=106878, avg=11664.65, stdev=14201.25 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 1020], 10.00th=[ 1123], 20.00th=[ 1303], | 30.00th=[ 1958], 40.00th=[ 3294], 50.00th=[ 5800], 60.00th=[ 9765], | 70.00th=[ 13698], 80.00th=[ 20841], 90.00th=[ 31327], 95.00th=[ 39060], | 99.00th=[ 63701], 99.50th=[ 87557], 99.90th=[106431], 99.95th=[106431], | 99.99th=[106431] bw ( KiB/s): min= 64, max= 798, per=24.45%, avg=360.41, stdev=152.22, samples=34 iops : min= 4, max= 49, avg=22.03, stdev= 9.60, samples=34 lat (usec) : 100=6.64%, 250=7.55%, 500=0.39%, 750=0.26%, 1000=2.08% lat (msec) : 2=14.19%, 4=6.51%, 10=8.85%, 20=9.90%, 50=28.78% lat (msec) : 100=14.71%, 250=0.13% cpu : usr=0.16%, sys=9.79%, ctx=1909, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1401KiB/s (1435kB/s), 347KiB/s-369KiB/s (356kB/s-377kB/s), io=23.4MiB (24.5MB), run=16473-17094msec WRITE: bw=1474KiB/s (1510kB/s), 355KiB/s-397KiB/s (364kB/s-407kB/s), io=24.6MiB (25.8MB), run=16473-17094msec rand-rw: (groupid=0, jobs=1): err= 0: pid=13313: Fri Sep 20 08:43:16 2024 read: IOPS=8, BW=140KiB/s (143kB/s)(5920KiB/42298msec) clat (usec): min=3636, max=98129, avg=14681.53, stdev=13152.42 lat (usec): min=3638, max=98130, avg=14683.15, stdev=13152.47 clat percentiles (usec): | 1.00th=[ 4817], 5.00th=[ 5997], 10.00th=[ 6456], 20.00th=[ 7504], | 30.00th=[ 8455], 40.00th=[ 9241], 50.00th=[10159], 60.00th=[11731], | 70.00th=[12911], 80.00th=[17171], 90.00th=[28181], 95.00th=[43779], | 99.00th=[70779], 99.50th=[96994], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 31, max= 544, per=25.61%, avg=143.19, stdev=86.98, samples=81 iops : min= 1, max= 34, avg= 8.41, stdev= 5.55, samples=81 write: IOPS=9, BW=151KiB/s (154kB/s)(6368KiB/42298msec) clat (msec): min=38, max=240, avg=92.44, stdev=37.15 lat (msec): min=38, max=240, avg=92.45, stdev=37.15 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 49], 10.00th=[ 54], 20.00th=[ 62], | 30.00th=[ 69], 40.00th=[ 75], 50.00th=[ 83], 60.00th=[ 93], | 70.00th=[ 104], 80.00th=[ 123], 90.00th=[ 146], 95.00th=[ 167], | 99.00th=[ 205], 99.50th=[ 230], 99.90th=[ 241], 99.95th=[ 241], | 99.99th=[ 241] bw ( KiB/s): min= 31, max= 255, per=25.31%, avg=149.07, stdev=51.47, samples=84 iops : min= 1, max= 15, avg= 8.79, stdev= 3.31, samples=84 lat (msec) : 4=0.13%, 10=23.31%, 20=16.80%, 50=10.03%, 100=32.29% lat (msec) : 250=17.45% cpu : usr=0.04%, sys=2.39%, ctx=896, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13314: Fri Sep 20 08:43:16 2024 read: IOPS=8, BW=134KiB/s (138kB/s)(5744KiB/42772msec) clat (usec): min=3529, max=94276, avg=15160.90, stdev=13695.17 lat (usec): min=3530, max=94278, avg=15164.83, stdev=13693.46 clat percentiles (usec): | 1.00th=[ 4686], 5.00th=[ 5473], 10.00th=[ 6259], 20.00th=[ 7308], | 30.00th=[ 8455], 40.00th=[ 9503], 50.00th=[10421], 60.00th=[11731], | 70.00th=[13829], 80.00th=[19530], 90.00th=[29230], 95.00th=[41681], | 99.00th=[81265], 99.50th=[87557], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 31, max= 448, per=25.90%, avg=144.78, stdev=84.73, samples=78 iops : min= 1, max= 28, avg= 8.59, stdev= 5.36, samples=78 write: IOPS=9, BW=153KiB/s (157kB/s)(6544KiB/42772msec) clat (msec): min=36, max=303, avg=91.18, stdev=39.47 lat (msec): min=36, max=303, avg=91.18, stdev=39.47 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 45], 10.00th=[ 53], 20.00th=[ 59], | 30.00th=[ 69], 40.00th=[ 74], 50.00th=[ 82], 60.00th=[ 91], | 70.00th=[ 102], 80.00th=[ 118], 90.00th=[ 148], 95.00th=[ 165], | 99.00th=[ 218], 99.50th=[ 243], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 63, max= 317, per=25.73%, avg=151.58, stdev=49.84, samples=85 iops : min= 3, max= 19, avg= 9.02, stdev= 3.21, samples=85 lat (msec) : 4=0.13%, 10=21.48%, 20=16.28%, 50=11.33%, 100=34.11% lat (msec) : 250=16.41%, 500=0.26% cpu : usr=0.05%, sys=2.35%, ctx=916, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13315: Fri Sep 20 08:43:16 2024 read: IOPS=9, BW=152KiB/s (155kB/s)(6256KiB/41235msec) clat (msec): min=4, max=126, avg=14.42, stdev=14.02 lat (msec): min=4, max=126, avg=14.42, stdev=14.02 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 12], | 70.00th=[ 14], 80.00th=[ 17], 90.00th=[ 26], 95.00th=[ 36], | 99.00th=[ 99], 99.50th=[ 123], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 31, max= 447, per=27.91%, avg=156.02, stdev=96.81, samples=80 iops : min= 1, max= 27, avg= 9.40, stdev= 6.14, samples=80 write: IOPS=9, BW=146KiB/s (150kB/s)(6032KiB/41235msec) clat (msec): min=29, max=298, avg=94.18, stdev=39.71 lat (msec): min=29, max=298, avg=94.19, stdev=39.70 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 48], 10.00th=[ 54], 20.00th=[ 63], | 30.00th=[ 70], 40.00th=[ 78], 50.00th=[ 87], 60.00th=[ 95], | 70.00th=[ 107], 80.00th=[ 122], 90.00th=[ 146], 95.00th=[ 165], | 99.00th=[ 234], 99.50th=[ 257], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 63, max= 256, per=24.71%, avg=145.55, stdev=50.38, samples=82 iops : min= 3, max= 16, avg= 8.73, stdev= 3.25, samples=82 lat (msec) : 10=22.40%, 20=20.18%, 50=10.42%, 100=29.69%, 250=16.93% lat (msec) : 500=0.39% cpu : usr=0.04%, sys=2.47%, ctx=907, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13316: Fri Sep 20 08:43:16 2024 read: IOPS=8, BW=143KiB/s (147kB/s)(6032KiB/42053msec) clat (msec): min=3, max=120, avg=15.08, stdev=13.75 lat (msec): min=3, max=120, avg=15.08, stdev=13.75 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 12], | 70.00th=[ 15], 80.00th=[ 18], 90.00th=[ 29], 95.00th=[ 42], | 99.00th=[ 79], 99.50th=[ 99], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 31, max= 352, per=26.44%, avg=147.81, stdev=81.70, samples=81 iops : min= 1, max= 22, avg= 8.93, stdev= 5.17, samples=81 write: IOPS=9, BW=149KiB/s (152kB/s)(6256KiB/42053msec) clat (msec): min=29, max=255, avg=92.72, stdev=39.31 lat (msec): min=29, max=255, avg=92.72, stdev=39.31 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 47], 10.00th=[ 52], 20.00th=[ 60], | 30.00th=[ 68], 40.00th=[ 77], 50.00th=[ 85], 60.00th=[ 95], | 70.00th=[ 106], 80.00th=[ 122], 90.00th=[ 140], 95.00th=[ 165], | 99.00th=[ 241], 99.50th=[ 251], 99.90th=[ 257], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 31, max= 224, per=25.23%, avg=148.60, stdev=48.68, samples=84 iops : min= 1, max= 14, avg= 8.98, stdev= 3.19, samples=84 lat (msec) : 4=0.26%, 10=20.70%, 20=19.40%, 50=11.59%, 100=30.08% lat (msec) : 250=17.71%, 500=0.26% cpu : usr=0.03%, sys=2.46%, ctx=908, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=560KiB/s (573kB/s), 134KiB/s-152KiB/s (138kB/s-155kB/s), io=23.4MiB (24.5MB), run=41235-42772msec WRITE: bw=589KiB/s (603kB/s), 146KiB/s-153KiB/s (150kB/s-157kB/s), io=24.6MiB (25.8MB), run=41235-42772msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=13360: Fri Sep 20 08:43:26 2024 read: IOPS=0, BW=881KiB/s (902kB/s)(7168KiB/8134msec) clat (usec): min=432, max=1418.9k, avg=573621.69, stdev=646096.05 lat (usec): min=434, max=1418.9k, avg=573623.08, stdev=646095.99 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 433], 10.00th=[ 433], | 20.00th=[ 979], 30.00th=[ 65799], 40.00th=[ 65799], | 50.00th=[ 258999], 60.00th=[ 859833], 70.00th=[ 859833], | 80.00th=[1400898], 90.00th=[1417675], 95.00th=[1417675], | 99.00th=[1417675], 99.50th=[1417675], 99.90th=[1417675], | 99.95th=[1417675], 99.99th=[1417675] bw ( KiB/s): min= 2023, max= 4087, per=100.00%, avg=3054.00, stdev=1183.60, samples=4 iops : min= 1, max= 3, avg= 2.00, stdev= 1.15, samples=4 write: IOPS=0, BW=629KiB/s (645kB/s)(5120KiB/8134msec) clat (msec): min=101, max=2154, avg=796.14, stdev=953.82 lat (msec): min=101, max=2154, avg=796.34, stdev=953.70 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 102], 10.00th=[ 102], 20.00th=[ 102], | 30.00th=[ 106], 40.00th=[ 106], 50.00th=[ 165], 60.00th=[ 165], | 70.00th=[ 1452], 80.00th=[ 1452], 90.00th=[ 2165], 95.00th=[ 2165], | 99.00th=[ 2165], 99.50th=[ 2165], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 2023, max= 4071, per=72.11%, avg=2541.00, stdev=1020.02, samples=4 iops : min= 1, max= 3, avg= 1.50, stdev= 1.00, samples=4 lat (usec) : 500=8.33%, 1000=8.33% lat (msec) : 100=8.33%, 250=25.00%, 500=8.33%, 1000=8.33% cpu : usr=0.05%, sys=5.98%, ctx=589, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13361: Fri Sep 20 08:43:26 2024 read: IOPS=1, BW=1025KiB/s (1049kB/s)(2048KiB/1999msec) clat (usec): min=549, max=246774, avg=123661.94, stdev=174107.52 lat (usec): min=551, max=246777, avg=123664.19, stdev=174107.95 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 553], 10.00th=[ 553], 20.00th=[ 553], | 30.00th=[ 553], 40.00th=[ 553], 50.00th=[ 553], 60.00th=[246416], | 70.00th=[246416], 80.00th=[246416], 90.00th=[246416], 95.00th=[246416], | 99.00th=[246416], 99.50th=[246416], 99.90th=[246416], 99.95th=[246416], | 99.99th=[246416] bw ( KiB/s): min= 2048, max= 2048, per=81.37%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5123KiB/s (5246kB/s)(10.0MiB/1999msec) clat (msec): min=45, max=523, avg=166.71, stdev=137.81 lat (msec): min=46, max=523, avg=166.79, stdev=137.83 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 87], | 30.00th=[ 90], 40.00th=[ 109], 50.00th=[ 124], 60.00th=[ 131], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 264], 95.00th=[ 523], | 99.00th=[ 523], 99.50th=[ 523], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 2048, max= 8192, per=100.00%, avg=5461.33, stdev=3128.37, samples=3 iops : min= 2, max= 8, avg= 5.33, stdev= 3.06, samples=3 lat (usec) : 750=8.33% lat (msec) : 50=8.33%, 100=16.67%, 250=50.00%, 500=8.33%, 750=8.33% cpu : usr=0.05%, sys=14.41%, ctx=142, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13362: Fri Sep 20 08:43:26 2024 read: IOPS=0, BW=946KiB/s (969kB/s)(6144KiB/6494msec) clat (usec): min=466, max=1887.1k, avg=686809.27, stdev=820536.56 lat (usec): min=467, max=1887.1k, avg=686810.70, stdev=820536.31 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 465], 10.00th=[ 465], | 20.00th=[ 486], 30.00th=[ 486], 40.00th=[ 221250], | 50.00th=[ 221250], 60.00th=[ 476054], 70.00th=[1535116], | 80.00th=[1535116], 90.00th=[1887437], 95.00th=[1887437], | 99.00th=[1887437], 99.50th=[1887437], 99.90th=[1887437], | 99.95th=[1887437], 99.99th=[1887437] bw ( KiB/s): min= 2035, max= 6144, per=100.00%, avg=3409.00, stdev=2368.59, samples=3 iops : min= 1, max= 6, avg= 3.00, stdev= 2.65, samples=3 write: IOPS=0, BW=946KiB/s (969kB/s)(6144KiB/6494msec) clat (msec): min=49, max=1533, avg=368.82, stdev=578.71 lat (msec): min=49, max=1533, avg=369.21, stdev=578.55 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 68], | 30.00th=[ 68], 40.00th=[ 102], 50.00th=[ 102], 60.00th=[ 144], | 70.00th=[ 317], 80.00th=[ 317], 90.00th=[ 1536], 95.00th=[ 1536], | 99.00th=[ 1536], 99.50th=[ 1536], 99.90th=[ 1536], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 2035, max= 4096, per=87.02%, avg=3066.50, stdev=1183.59, samples=4 iops : min= 1, max= 4, avg= 2.50, stdev= 1.29, samples=4 lat (usec) : 500=16.67% lat (msec) : 50=8.33%, 100=8.33%, 250=25.00%, 500=16.67% cpu : usr=0.03%, sys=8.87%, ctx=563, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13363: Fri Sep 20 08:43:26 2024 read: IOPS=0, BW=638KiB/s (653kB/s)(5120KiB/8024msec) clat (msec): min=40, max=2968, avg=1404.13, stdev=1244.72 lat (msec): min=40, max=2968, avg=1404.13, stdev=1244.72 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 41], | 30.00th=[ 220], 40.00th=[ 220], 50.00th=[ 1854], 60.00th=[ 1854], | 70.00th=[ 1938], 80.00th=[ 1938], 90.00th=[ 2970], 95.00th=[ 2970], | 99.00th=[ 2970], 99.50th=[ 2970], 99.90th=[ 2970], 99.95th=[ 2970], | 99.99th=[ 2970] bw ( KiB/s): min= 2015, max= 4063, per=100.00%, avg=2708.67, stdev=1173.00, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 write: IOPS=0, BW=893KiB/s (915kB/s)(7168KiB/8024msec) clat (msec): min=56, max=326, avg=141.42, stdev=85.44 lat (msec): min=57, max=326, avg=141.50, stdev=85.44 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 103], | 30.00th=[ 122], 40.00th=[ 122], 50.00th=[ 126], 60.00th=[ 127], | 70.00th=[ 127], 80.00th=[ 133], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 2043, max= 4096, per=100.00%, avg=3564.25, stdev=1014.57, samples=4 iops : min= 1, max= 4, avg= 2.75, stdev= 1.26, samples=4 lat (msec) : 50=8.33%, 100=8.33%, 250=50.00%, 500=8.33% cpu : usr=0.04%, sys=9.97%, ctx=855, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2518KiB/s (2578kB/s), 638KiB/s-1025KiB/s (653kB/s-1049kB/s), io=20.0MiB (20.0MB), run=1999-8134msec WRITE: bw=3525KiB/s (3610kB/s), 629KiB/s-5123KiB/s (645kB/s-5246kB/s), io=28.0MiB (29.4MB), run=1999-8134msec rand-rw: (groupid=0, jobs=1): err= 0: pid=13365: Fri Sep 20 08:43:26 2024 read: IOPS=0, BW=889KiB/s (910kB/s)(7168KiB/8066msec) clat (msec): min=66, max=1493, avg=502.06, stdev=598.70 lat (msec): min=66, max=1493, avg=502.07, stdev=598.70 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 72], | 30.00th=[ 79], 40.00th=[ 79], 50.00th=[ 121], 60.00th=[ 502], | 70.00th=[ 502], 80.00th=[ 1183], 90.00th=[ 1502], 95.00th=[ 1502], | 99.00th=[ 1502], 99.50th=[ 1502], 99.90th=[ 1502], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 1825, max= 4087, per=94.05%, avg=2388.00, stdev=954.29, samples=5 iops : min= 1, max= 3, avg= 1.80, stdev= 0.84, samples=5 write: IOPS=0, BW=635KiB/s (650kB/s)(5120KiB/8066msec) clat (msec): min=82, max=1689, avg=833.70, stdev=743.13 lat (msec): min=82, max=1689, avg=833.84, stdev=743.21 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 83], 10.00th=[ 83], 20.00th=[ 83], | 30.00th=[ 148], 40.00th=[ 148], 50.00th=[ 760], 60.00th=[ 760], | 70.00th=[ 1502], 80.00th=[ 1502], 90.00th=[ 1687], 95.00th=[ 1687], | 99.00th=[ 1687], 99.50th=[ 1687], 99.90th=[ 1687], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 1822, max= 3650, per=67.21%, avg=2388.50, stdev=847.31, samples=4 iops : min= 1, max= 3, avg= 1.50, stdev= 1.00, samples=4 lat (msec) : 100=33.33%, 250=16.67%, 500=8.33%, 1000=8.33% cpu : usr=0.00%, sys=1.74%, ctx=75, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13366: Fri Sep 20 08:43:26 2024 read: IOPS=0, BW=411KiB/s (421kB/s)(2048KiB/4986msec) clat (msec): min=218, max=684, avg=451.58, stdev=329.19 lat (msec): min=218, max=684, avg=451.58, stdev=329.19 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 220], 10.00th=[ 220], 20.00th=[ 220], | 30.00th=[ 220], 40.00th=[ 220], 50.00th=[ 220], 60.00th=[ 684], | 70.00th=[ 684], 80.00th=[ 684], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 1924, max= 2043, per=78.12%, avg=1983.50, stdev=84.15, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2054KiB/s (2103kB/s)(10.0MiB/4986msec) clat (msec): min=135, max=2077, avg=394.97, stdev=596.99 lat (msec): min=135, max=2077, avg=395.06, stdev=596.99 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 138], 40.00th=[ 142], 50.00th=[ 148], 60.00th=[ 218], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 338], 95.00th=[ 2072], | 99.00th=[ 2072], 99.50th=[ 2072], 99.90th=[ 2072], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 2035, max= 6144, per=100.00%, avg=3681.00, stdev=1717.25, samples=5 iops : min= 1, max= 6, avg= 3.00, stdev= 2.12, samples=5 lat (msec) : 250=58.33%, 500=25.00%, 750=8.33% cpu : usr=0.10%, sys=2.99%, ctx=52, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13367: Fri Sep 20 08:43:26 2024 read: IOPS=0, BW=768KiB/s (787kB/s)(6144KiB/7998msec) clat (msec): min=46, max=2201, avg=610.39, stdev=874.05 lat (msec): min=46, max=2201, avg=610.39, stdev=874.05 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 83], 50.00th=[ 83], 60.00th=[ 211], | 70.00th=[ 1070], 80.00th=[ 1070], 90.00th=[ 2198], 95.00th=[ 2198], | 99.00th=[ 2198], 99.50th=[ 2198], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 2035, max= 4087, per=100.00%, avg=2554.50, stdev=1021.69, samples=4 iops : min= 1, max= 3, avg= 2.00, stdev= 0.82, samples=4 write: IOPS=0, BW=768KiB/s (787kB/s)(6144KiB/7998msec) clat (msec): min=141, max=1781, avg=707.39, stdev=746.55 lat (msec): min=141, max=1781, avg=707.47, stdev=746.56 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 142], 10.00th=[ 142], 20.00th=[ 194], | 30.00th=[ 194], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 376], | 70.00th=[ 1536], 80.00th=[ 1536], 90.00th=[ 1787], 95.00th=[ 1787], | 99.00th=[ 1787], 99.50th=[ 1787], 99.90th=[ 1787], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 1910, max= 4096, per=68.20%, avg=2424.00, stdev=936.37, samples=5 iops : min= 1, max= 4, avg= 1.80, stdev= 1.30, samples=5 lat (msec) : 50=16.67%, 100=8.33%, 250=33.33%, 500=8.33% cpu : usr=0.00%, sys=1.33%, ctx=51, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13368: Fri Sep 20 08:43:26 2024 read: IOPS=0, BW=643KiB/s (658kB/s)(5120KiB/7965msec) clat (msec): min=110, max=1941, avg=737.66, stdev=774.45 lat (msec): min=110, max=1941, avg=737.66, stdev=774.45 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 228], 40.00th=[ 228], 50.00th=[ 321], 60.00th=[ 321], | 70.00th=[ 1083], 80.00th=[ 1083], 90.00th=[ 1938], 95.00th=[ 1938], | 99.00th=[ 1938], 99.50th=[ 1938], 99.90th=[ 1938], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 2048, max= 4096, per=100.00%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 write: IOPS=0, BW=900KiB/s (922kB/s)(7168KiB/7965msec) clat (msec): min=78, max=1633, avg=558.08, stdev=688.41 lat (msec): min=78, max=1633, avg=558.22, stdev=688.45 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 80], 10.00th=[ 80], 20.00th=[ 97], | 30.00th=[ 155], 40.00th=[ 155], 50.00th=[ 213], 60.00th=[ 245], | 70.00th=[ 245], 80.00th=[ 1485], 90.00th=[ 1636], 95.00th=[ 1636], | 99.00th=[ 1636], 99.50th=[ 1636], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 2048, max= 6144, per=100.00%, avg=4778.67, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 4.67, stdev= 2.31, samples=3 lat (msec) : 100=16.67%, 250=41.67%, 500=8.33% cpu : usr=0.03%, sys=1.43%, ctx=56, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2539KiB/s (2600kB/s), 411KiB/s-889KiB/s (421kB/s-910kB/s), io=20.0MiB (20.0MB), run=4986-8066msec WRITE: bw=3555KiB/s (3640kB/s), 635KiB/s-2054KiB/s (650kB/s-2103kB/s), io=28.0MiB (29.4MB), run=4986-8066msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=13382: Fri Sep 20 08:43:32 2024 read: IOPS=1, BW=5745KiB/s (5883kB/s)(12.0MiB/2139msec) clat (msec): min=23, max=1923, avg=704.66, stdev=1057.68 lat (msec): min=23, max=1923, avg=704.66, stdev=1057.68 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 24], | 30.00th=[ 24], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 167], | 70.00th=[ 1921], 80.00th=[ 1921], 90.00th=[ 1921], 95.00th=[ 1921], | 99.00th=[ 1921], 99.50th=[ 1921], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33% cpu : usr=0.00%, sys=40.93%, ctx=274, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13383: Fri Sep 20 08:43:32 2024 read: IOPS=0, BW=1061KiB/s (1087kB/s)(4096KiB/3859msec) clat (nsec): min=214680k, max=214680k, avg=214680043.00, stdev= 0.00 lat (nsec): min=214683k, max=214683k, avg=214682746.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 215], 10.00th=[ 215], 20.00th=[ 215], | 30.00th=[ 215], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 215], | 70.00th=[ 215], 80.00th=[ 215], 90.00th=[ 215], 95.00th=[ 215], | 99.00th=[ 215], 99.50th=[ 215], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2123KiB/s (2174kB/s)(8192KiB/3859msec) clat (msec): min=1012, max=2610, avg=1811.55, stdev=1129.76 lat (msec): min=1013, max=2610, avg=1811.85, stdev=1129.72 clat percentiles (msec): | 1.00th=[ 1011], 5.00th=[ 1011], 10.00th=[ 1011], 20.00th=[ 1011], | 30.00th=[ 1011], 40.00th=[ 1011], 50.00th=[ 1011], 60.00th=[ 2601], | 70.00th=[ 2601], 80.00th=[ 2601], 90.00th=[ 2601], 95.00th=[ 2601], | 99.00th=[ 2601], 99.50th=[ 2601], 99.90th=[ 2601], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 8159, max= 8159, per=99.57%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=6.19%, ctx=68, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13384: Fri Sep 20 08:43:32 2024 write: IOPS=0, BW=3625KiB/s (3712kB/s)(12.0MiB/3390msec) clat (msec): min=386, max=2362, avg=1121.19, stdev=1080.88 lat (msec): min=386, max=2362, avg=1121.51, stdev=1080.84 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 388], 10.00th=[ 388], 20.00th=[ 388], | 30.00th=[ 388], 40.00th=[ 617], 50.00th=[ 617], 60.00th=[ 617], | 70.00th=[ 2366], 80.00th=[ 2366], 90.00th=[ 2366], 95.00th=[ 2366], | 99.00th=[ 2366], 99.50th=[ 2366], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 8175, max= 8175, per=99.77%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=9.21%, ctx=88, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13385: Fri Sep 20 08:43:32 2024 write: IOPS=0, BW=3073KiB/s (3147kB/s)(12.0MiB/3999msec) clat (msec): min=291, max=2185, avg=1313.19, stdev=955.46 lat (msec): min=296, max=2185, avg=1314.73, stdev=953.35 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 292], 10.00th=[ 292], 20.00th=[ 292], | 30.00th=[ 292], 40.00th=[ 1469], 50.00th=[ 1469], 60.00th=[ 1469], | 70.00th=[ 2198], 80.00th=[ 2198], 90.00th=[ 2198], 95.00th=[ 2198], | 99.00th=[ 2198], 99.50th=[ 2198], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 8192, max= 8192, per=99.98%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=7.13%, ctx=88, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4246KiB/s (4348kB/s), 1061KiB/s-5745KiB/s (1087kB/s-5883kB/s), io=16.0MiB (16.8MB), run=2139-3859msec WRITE: bw=8194KiB/s (8391kB/s), 2123KiB/s-3625KiB/s (2174kB/s-3712kB/s), io=32.0MiB (33.6MB), run=3390-3999msec rand-rw: (groupid=0, jobs=1): err= 0: pid=13381: Fri Sep 20 08:43:32 2024 read: IOPS=0, BW=3435KiB/s (3518kB/s)(12.0MiB/3577msec) clat (msec): min=495, max=2254, avg=1184.31, stdev=939.69 lat (msec): min=495, max=2254, avg=1184.31, stdev=939.69 clat percentiles (msec): | 1.00th=[ 498], 5.00th=[ 498], 10.00th=[ 498], 20.00th=[ 498], | 30.00th=[ 498], 40.00th=[ 802], 50.00th=[ 802], 60.00th=[ 802], | 70.00th=[ 2265], 80.00th=[ 2265], 90.00th=[ 2265], 95.00th=[ 2265], | 99.00th=[ 2265], 99.50th=[ 2265], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 8110, max= 8192, per=100.00%, avg=8151.00, stdev=57.98, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=1.93%, ctx=34, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13386: Fri Sep 20 08:43:32 2024 read: IOPS=0, BW=915KiB/s (937kB/s)(4096KiB/4476msec) clat (nsec): min=259538k, max=259538k, avg=259537647.00, stdev= 0.00 lat (nsec): min=259541k, max=259541k, avg=259541263.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 259], 5.00th=[ 259], 10.00th=[ 259], 20.00th=[ 259], | 30.00th=[ 259], 40.00th=[ 259], 50.00th=[ 259], 60.00th=[ 259], | 70.00th=[ 259], 80.00th=[ 259], 90.00th=[ 259], 95.00th=[ 259], | 99.00th=[ 259], 99.50th=[ 259], 99.90th=[ 259], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1830KiB/s (1874kB/s)(8192KiB/4476msec) clat (msec): min=1391, max=2781, avg=2086.48, stdev=982.81 lat (msec): min=1391, max=2781, avg=2086.77, stdev=982.83 clat percentiles (msec): | 1.00th=[ 1385], 5.00th=[ 1385], 10.00th=[ 1385], 20.00th=[ 1385], | 30.00th=[ 1385], 40.00th=[ 1385], 50.00th=[ 1385], 60.00th=[ 2769], | 70.00th=[ 2769], 80.00th=[ 2769], 90.00th=[ 2769], 95.00th=[ 2769], | 99.00th=[ 2769], 99.50th=[ 2769], 99.90th=[ 2769], 99.95th=[ 2769], | 99.99th=[ 2769] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=2.88%, ctx=212, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13387: Fri Sep 20 08:43:32 2024 write: IOPS=0, BW=3200KiB/s (3277kB/s)(12.0MiB/3840msec) clat (msec): min=505, max=2274, avg=1260.57, stdev=912.24 lat (msec): min=505, max=2274, avg=1260.92, stdev=912.28 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 506], 10.00th=[ 506], 20.00th=[ 506], | 30.00th=[ 506], 40.00th=[ 1003], 50.00th=[ 1003], 60.00th=[ 1003], | 70.00th=[ 2265], 80.00th=[ 2265], 90.00th=[ 2265], 95.00th=[ 2265], | 99.00th=[ 2265], 99.50th=[ 2265], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 8143, max= 8175, per=100.00%, avg=8159.00, stdev=22.63, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 750=33.33% cpu : usr=0.05%, sys=1.90%, ctx=29, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=13388: Fri Sep 20 08:43:32 2024 write: IOPS=0, BW=2645KiB/s (2708kB/s)(12.0MiB/4646msec) clat (msec): min=362, max=3080, avg=1538.24, stdev=1395.72 lat (msec): min=362, max=3080, avg=1538.57, stdev=1395.74 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 363], | 30.00th=[ 363], 40.00th=[ 1167], 50.00th=[ 1167], 60.00th=[ 1167], | 70.00th=[ 3071], 80.00th=[ 3071], 90.00th=[ 3071], 95.00th=[ 3071], | 99.00th=[ 3071], 99.50th=[ 3071], 99.90th=[ 3071], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33% cpu : usr=0.04%, sys=2.56%, ctx=59, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3660KiB/s (3748kB/s), 915KiB/s-3435KiB/s (937kB/s-3518kB/s), io=16.0MiB (16.8MB), run=3577-4476msec WRITE: bw=7053KiB/s (7222kB/s), 1830KiB/s-3200KiB/s (1874kB/s-3277kB/s), io=32.0MiB (33.6MB), run=3840-4646msec