-----============= acceptance-small: lustre-rsync-test ============----- Fri May 23 17:42:47 EDT 2025 oleg303-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/lustre-rsync-test.*ex: No such file or directory excepting tests: 2b oleg303-client.virtnet: executing check_config_client /mnt/lustre oleg303-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg303-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800aaa00800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800aaa00800.idle_timeout=debug disable quota as required oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 nobody:x:99: == lustre-rsync-test test 2a: Replicate files created by dbench. ========================================================== 17:42:59 (1748036579) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl1' lustre-MDT0000: Registered changelog user cl1 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2a.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:43:01 EDT 2025 waiting for dbench pid 8776 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs failed to create barrier semaphore 1 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 664 42.31 MB/sec warmup 1 sec latency 24.540 ms 2 1225 23.07 MB/sec warmup 2 sec latency 63.983 ms 2 1984 17.64 MB/sec warmup 3 sec latency 34.795 ms 2 2922 16.74 MB/sec warmup 4 sec latency 29.044 ms 2 3852 15.90 MB/sec warmup 5 sec latency 27.657 ms 2 4369 13.53 MB/sec warmup 6 sec latency 48.812 ms 2 5070 12.90 MB/sec warmup 7 sec latency 32.980 ms 2 5855 11.72 MB/sec warmup 8 sec latency 25.565 ms 2 6851 12.10 MB/sec warmup 9 sec latency 26.704 ms 2 7557 11.84 MB/sec warmup 10 sec latency 12.691 ms 2 8082 10.92 MB/sec warmup 11 sec latency 43.070 ms 2 9551 3.73 MB/sec execute 1 sec latency 29.002 ms 2 10651 12.17 MB/sec execute 2 sec latency 29.423 ms 2 11243 9.23 MB/sec execute 3 sec latency 8.884 ms 2 11734 7.90 MB/sec execute 4 sec latency 45.929 ms 2 12446 7.67 MB/sec execute 5 sec latency 22.029 ms 2 13121 7.16 MB/sec execute 6 sec latency 37.776 ms 2 14172 8.93 MB/sec execute 7 sec latency 32.327 ms 2 14691 8.24 MB/sec execute 8 sec latency 9.757 ms 2 15139 7.50 MB/sec execute 9 sec latency 58.063 ms 2 15817 7.63 MB/sec execute 10 sec latency 23.488 ms 2 16482 7.17 MB/sec execute 11 sec latency 29.791 ms 2 17204 7.59 MB/sec execute 12 sec latency 36.042 ms 2 18031 7.96 MB/sec execute 13 sec latency 24.104 ms 2 18489 7.49 MB/sec execute 14 sec latency 61.141 ms 2 18996 7.23 MB/sec execute 15 sec latency 43.407 ms 2 19637 7.19 MB/sec execute 16 sec latency 29.995 ms 2 20337 7.07 MB/sec execute 17 sec latency 31.612 ms 2 21318 7.73 MB/sec execute 18 sec latency 30.655 ms 2 21806 7.50 MB/sec execute 19 sec latency 10.547 ms 2 22251 7.21 MB/sec execute 20 sec latency 46.304 ms 2 22906 7.28 MB/sec execute 21 sec latency 31.786 ms 2 23520 7.00 MB/sec execute 22 sec latency 36.376 ms 2 24300 7.29 MB/sec execute 23 sec latency 33.664 ms 2 25116 7.50 MB/sec execute 24 sec latency 23.337 ms 2 25575 7.25 MB/sec execute 25 sec latency 65.367 ms 2 26031 7.11 MB/sec execute 26 sec latency 25.637 ms 2 26660 7.09 MB/sec execute 27 sec latency 39.435 ms 2 27292 6.96 MB/sec execute 28 sec latency 26.888 ms 2 28213 7.23 MB/sec execute 29 sec latency 28.777 ms 2 28822 7.30 MB/sec execute 30 sec latency 9.982 ms 2 29252 7.10 MB/sec execute 31 sec latency 56.922 ms 2 29889 7.16 MB/sec execute 32 sec latency 32.688 ms 2 30448 6.97 MB/sec execute 33 sec latency 29.719 ms 2 31183 7.09 MB/sec execute 34 sec latency 32.565 ms 2 32073 7.27 MB/sec execute 35 sec latency 31.401 ms 2 32566 7.15 MB/sec execute 36 sec latency 9.726 ms 2 32954 7.00 MB/sec execute 37 sec latency 45.943 ms 2 33602 7.04 MB/sec execute 38 sec latency 29.858 ms 2 34202 6.89 MB/sec execute 39 sec latency 30.941 ms 2 34962 7.06 MB/sec execute 40 sec latency 33.504 ms 2 35787 7.19 MB/sec execute 41 sec latency 24.788 ms 2 36240 7.05 MB/sec execute 42 sec latency 65.454 ms 2 36715 6.97 MB/sec execute 43 sec latency 26.835 ms 2 37328 6.96 MB/sec execute 44 sec latency 31.447 ms 2 37957 6.88 MB/sec execute 45 sec latency 36.593 ms 2 38869 7.05 MB/sec execute 46 sec latency 34.288 ms 2 39470 7.10 MB/sec execute 47 sec latency 10.625 ms 2 39923 6.98 MB/sec execute 48 sec latency 56.238 ms 2 40537 7.02 MB/sec execute 49 sec latency 38.218 ms 2 41085 6.90 MB/sec execute 50 sec latency 32.281 ms 2 41825 6.97 MB/sec execute 51 sec latency 25.948 ms 2 42721 7.10 MB/sec execute 52 sec latency 32.069 ms 2 43200 7.02 MB/sec execute 53 sec latency 10.497 ms 2 43592 6.92 MB/sec execute 54 sec latency 44.475 ms 2 44259 6.95 MB/sec execute 55 sec latency 26.058 ms 2 44946 6.89 MB/sec execute 56 sec latency 37.987 ms 2 45785 6.99 MB/sec execute 57 sec latency 32.924 ms 2 46487 7.07 MB/sec execute 58 sec latency 24.706 ms 2 46938 6.97 MB/sec execute 59 sec latency 50.143 ms 2 cleanup 60 sec 0 cleanup 60 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 13366 4.467 14.276 Close 9813 1.024 9.205 Rename 570 6.144 9.671 Unlink 2718 1.787 7.673 Qpathinfo 12128 1.116 7.620 Qfileinfo 2108 0.116 1.294 Qfsinfo 2244 0.076 1.288 Sfileinfo 1100 2.909 5.670 Find 4700 0.342 6.984 WriteX 6610 0.662 8.348 ReadX 21114 0.012 1.576 LockX 44 0.845 1.025 UnlockX 44 0.877 1.065 Flush 940 19.110 65.445 Throughput 6.96517 MB/sec 2 clients 2 procs max_latency=65.454 ms stopping dbench in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:44:13 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2a.lustre-rsync-test dbench successfully finished Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 7 seconds Changelog records consumed: 19384 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0000: changelog user 'cl1' not found PASS 2a (84s) == lustre-rsync-test test 2b: Replicate files changed by dbench. ========================================================== 17:44:23 (1748036663) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl2' lustre-MDT0000: Registered changelog user cl2 waiting child_pid=$(pgrep dbench), 360 secs left ... looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2b.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:44:25 EDT 2025 waiting for dbench pid 9658 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 604 41.94 MB/sec warmup 1 sec latency 12.871 ms 2 1091 22.93 MB/sec warmup 2 sec latency 60.121 ms 2 1795 17.55 MB/sec warmup 3 sec latency 33.659 ms 2 2477 14.36 MB/sec warmup 4 sec latency 27.462 ms 2 3579 15.37 MB/sec warmup 5 sec latency 32.931 ms 2 4138 13.36 MB/sec warmup 6 sec latency 9.047 ms 2 4629 12.00 MB/sec warmup 7 sec latency 47.896 ms 2 5342 11.35 MB/sec warmup 8 sec latency 28.242 ms 2 6049 10.63 MB/sec warmup 9 sec latency 27.190 ms 2 7190 11.55 MB/sec warmup 10 sec latency 26.813 ms 2 7778 10.80 MB/sec warmup 11 sec latency 9.039 ms 2 8993 6.72 MB/sec execute 1 sec latency 51.625 ms 2 9861 8.94 MB/sec execute 2 sec latency 31.433 ms PIDs: 9658 9659 9660 Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 4074 Resuming dbench 2 10439 4.37 MB/sec execute 6 sec latency 3148.150 ms 2 11106 5.12 MB/sec execute 7 sec latency 3143.251 ms 2 11576 4.67 MB/sec execute 8 sec latency 49.392 ms 2 12276 5.14 MB/sec execute 9 sec latency 36.506 ms 2 12996 4.96 MB/sec execute 10 sec latency 26.667 ms 2 13847 5.70 MB/sec execute 11 sec latency 28.658 ms 2 14541 6.20 MB/sec execute 12 sec latency 30.066 ms 2 14994 5.80 MB/sec execute 13 sec latency 52.931 ms 2 15520 5.76 MB/sec execute 14 sec latency 46.835 ms 2 16125 5.69 MB/sec execute 15 sec latency 25.840 ms 2 16817 5.66 MB/sec execute 16 sec latency 30.757 ms Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 23458 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 2745 Wait for dbench to finish 2 16828 5.15 MB/sec execute 17 sec latency 1671.540 ms 2 17835 5.88 MB/sec execute 18 sec latency 1671.741 ms 2 18367 5.72 MB/sec execute 19 sec latency 16.544 ms 2 18767 5.58 MB/sec execute 20 sec latency 54.306 ms 2 19448 5.68 MB/sec execute 21 sec latency 33.318 ms 2 20161 5.59 MB/sec execute 22 sec latency 26.841 ms 2 21076 5.98 MB/sec execute 23 sec latency 29.117 ms 2 21708 6.12 MB/sec execute 24 sec latency 13.689 ms 2 22139 5.92 MB/sec execute 25 sec latency 70.146 ms 2 22796 6.04 MB/sec execute 26 sec latency 24.899 ms 2 23393 5.86 MB/sec execute 27 sec latency 37.775 ms 2 24227 6.10 MB/sec execute 28 sec latency 33.531 ms 2 25076 6.36 MB/sec execute 29 sec latency 23.666 ms 2 25556 6.19 MB/sec execute 30 sec latency 50.602 ms 2 26006 6.10 MB/sec execute 31 sec latency 35.410 ms 2 26638 6.12 MB/sec execute 32 sec latency 32.874 ms 2 27292 6.04 MB/sec execute 33 sec latency 26.204 ms 2 28210 6.29 MB/sec execute 34 sec latency 32.954 ms 2 28818 6.38 MB/sec execute 35 sec latency 12.054 ms 2 29244 6.23 MB/sec execute 36 sec latency 49.223 ms 2 29905 6.31 MB/sec execute 37 sec latency 24.400 ms 2 30529 6.18 MB/sec execute 38 sec latency 36.940 ms 2 31318 6.34 MB/sec execute 39 sec latency 36.720 ms 2 32174 6.52 MB/sec execute 40 sec latency 31.371 ms 2 32643 6.39 MB/sec execute 41 sec latency 67.681 ms 2 33091 6.32 MB/sec execute 42 sec latency 27.129 ms 2 33739 6.33 MB/sec execute 43 sec latency 30.550 ms 2 34382 6.27 MB/sec execute 44 sec latency 27.846 ms 2 35258 6.45 MB/sec execute 45 sec latency 32.430 ms 2 35879 6.52 MB/sec execute 46 sec latency 15.491 ms 2 36315 6.40 MB/sec execute 47 sec latency 56.152 ms 2 36937 6.46 MB/sec execute 48 sec latency 41.452 ms 2 37517 6.34 MB/sec execute 49 sec latency 35.520 ms 2 38269 6.44 MB/sec execute 50 sec latency 27.307 ms 2 39162 6.57 MB/sec execute 51 sec latency 30.438 ms 2 39657 6.50 MB/sec execute 52 sec latency 10.021 ms 2 40048 6.41 MB/sec execute 53 sec latency 47.303 ms 2 40705 6.45 MB/sec execute 54 sec latency 37.528 ms 2 41404 6.40 MB/sec execute 55 sec latency 23.034 ms 2 42223 6.51 MB/sec execute 56 sec latency 36.877 ms 2 42895 6.59 MB/sec execute 57 sec latency 33.976 ms 2 43342 6.50 MB/sec execute 58 sec latency 61.156 ms 2 43855 6.45 MB/sec execute 59 sec latency 29.519 ms 2 cleanup 60 sec 0 cleanup 61 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 12602 4.669 1671.732 Close 9250 1.008 2.178 Rename 528 6.118 9.555 Unlink 2504 1.798 4.042 Qpathinfo 11386 1.370 3143.237 Qfileinfo 1996 0.118 1.308 Qfsinfo 2054 0.078 1.277 Sfileinfo 1018 2.889 5.459 Find 4378 0.318 6.075 WriteX 6164 0.654 7.195 ReadX 19618 0.011 0.602 LockX 40 0.840 1.130 UnlockX 40 0.848 1.068 Flush 868 22.719 3148.140 Throughput 6.44918 MB/sec 2 clients 2 procs max_latency=3148.150 ms stopping dbench in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:45:38 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2b.lustre-rsync-test dbench successfully finished Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 26202 Clear changelog after use: no Errors: 0 lustre_rsync took 4 seconds Changelog records consumed: 11379 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 lustre-MDT0000: changelog user 'cl2' not found PASS 2b (82s) == lustre-rsync-test test complete, duration 178 sec ===== 17:45:45 (1748036745)