-----============= acceptance-small: lustre-rsync-test ============----- Fri May 23 17:42:57 EDT 2025 oleg242-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/lustre-rsync-test.*ex: No such file or directory excepting tests: 2b oleg242-client.virtnet: executing check_config_client /mnt/lustre oleg242-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg242-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b62ff800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b62ff800.idle_timeout=debug disable quota as required oleg242-server: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 nobody:x:99: == lustre-rsync-test test 2a: Replicate files created by dbench. ========================================================== 17:43:07 (1748036587) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' lustre-MDT0000: Registered changelog user cl1 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2a.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:43:10 EDT 2025 waiting for dbench pid 9652 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs failed to create barrier semaphore 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 493 34.41 MB/sec warmup 1 sec latency 14.742 ms 2 969 21.81 MB/sec warmup 2 sec latency 70.815 ms 2 1649 17.47 MB/sec warmup 3 sec latency 33.797 ms 2 2345 13.95 MB/sec warmup 4 sec latency 23.711 ms 2 3177 13.65 MB/sec warmup 5 sec latency 36.051 ms 2 3885 13.25 MB/sec warmup 6 sec latency 23.362 ms 2 4360 11.59 MB/sec warmup 7 sec latency 49.599 ms 2 4969 11.28 MB/sec warmup 8 sec latency 22.788 ms 2 5519 10.13 MB/sec warmup 9 sec latency 30.018 ms 2 6133 9.61 MB/sec warmup 10 sec latency 28.509 ms 2 7110 10.46 MB/sec warmup 11 sec latency 31.875 ms 2 8004 1.48 MB/sec execute 1 sec latency 57.688 ms 2 8608 5.10 MB/sec execute 2 sec latency 29.208 ms 2 9195 3.80 MB/sec execute 3 sec latency 28.091 ms 2 9869 5.51 MB/sec execute 4 sec latency 29.087 ms 2 10755 7.11 MB/sec execute 5 sec latency 30.085 ms 2 11183 6.38 MB/sec execute 6 sec latency 12.807 ms 2 11609 5.71 MB/sec execute 7 sec latency 71.436 ms 2 12217 6.07 MB/sec execute 8 sec latency 33.474 ms 2 12776 5.52 MB/sec execute 9 sec latency 36.491 ms 2 13419 6.03 MB/sec execute 10 sec latency 32.454 ms 2 14275 6.69 MB/sec execute 11 sec latency 31.557 ms 2 14702 6.38 MB/sec execute 12 sec latency 11.294 ms 2 15106 6.00 MB/sec execute 13 sec latency 47.306 ms 2 15688 6.19 MB/sec execute 14 sec latency 28.795 ms 2 16227 5.85 MB/sec execute 15 sec latency 26.651 ms 2 16792 5.79 MB/sec execute 16 sec latency 37.476 ms 2 17754 6.55 MB/sec execute 17 sec latency 32.717 ms 2 18190 6.37 MB/sec execute 18 sec latency 10.969 ms 2 18611 6.10 MB/sec execute 19 sec latency 62.479 ms 2 19154 6.24 MB/sec execute 20 sec latency 37.430 ms 2 19637 5.98 MB/sec execute 21 sec latency 38.733 ms 2 20219 5.87 MB/sec execute 22 sec latency 35.702 ms 2 21095 6.25 MB/sec execute 23 sec latency 29.483 ms 2 21683 6.38 MB/sec execute 24 sec latency 13.378 ms 2 22101 6.16 MB/sec execute 25 sec latency 48.177 ms 2 22539 6.05 MB/sec execute 26 sec latency 35.078 ms 2 23134 6.07 MB/sec execute 27 sec latency 37.705 ms 2 23733 5.97 MB/sec execute 28 sec latency 32.133 ms 2 24533 6.20 MB/sec execute 29 sec latency 32.841 ms 2 25183 6.36 MB/sec execute 30 sec latency 13.659 ms 2 25612 6.19 MB/sec execute 31 sec latency 65.107 ms 2 26058 6.11 MB/sec execute 32 sec latency 26.511 ms 2 26660 6.12 MB/sec execute 33 sec latency 35.225 ms 2 27262 6.04 MB/sec execute 34 sec latency 28.199 ms 2 28061 6.23 MB/sec execute 35 sec latency 31.763 ms 2 28702 6.36 MB/sec execute 36 sec latency 28.001 ms 2 29146 6.22 MB/sec execute 37 sec latency 48.315 ms 2 29571 6.15 MB/sec execute 38 sec latency 38.832 ms 2 30148 6.16 MB/sec execute 39 sec latency 35.058 ms 2 30788 6.09 MB/sec execute 40 sec latency 30.655 ms 2 31439 6.21 MB/sec execute 41 sec latency 32.599 ms 2 32227 6.36 MB/sec execute 42 sec latency 23.762 ms 2 32668 6.24 MB/sec execute 43 sec latency 63.649 ms 2 33100 6.18 MB/sec execute 44 sec latency 38.322 ms 2 33681 6.19 MB/sec execute 45 sec latency 33.466 ms 2 34322 6.12 MB/sec execute 46 sec latency 29.116 ms 2 34981 6.23 MB/sec execute 47 sec latency 33.678 ms 2 35771 6.36 MB/sec execute 48 sec latency 23.222 ms 2 36198 6.25 MB/sec execute 49 sec latency 61.891 ms 2 36616 6.20 MB/sec execute 50 sec latency 29.877 ms 2 37199 6.21 MB/sec execute 51 sec latency 33.007 ms 2 37844 6.15 MB/sec execute 52 sec latency 34.162 ms 2 38516 6.25 MB/sec execute 53 sec latency 31.431 ms 2 39314 6.36 MB/sec execute 54 sec latency 27.504 ms 2 39760 6.27 MB/sec execute 55 sec latency 58.252 ms 2 40190 6.22 MB/sec execute 56 sec latency 36.331 ms 2 40783 6.23 MB/sec execute 57 sec latency 31.100 ms 2 41460 6.18 MB/sec execute 58 sec latency 31.580 ms 2 42406 6.32 MB/sec execute 59 sec latency 28.481 ms 2 cleanup 60 sec 0 cleanup 60 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 12277 4.932 17.632 Close 9024 1.136 3.431 Rename 520 6.740 12.489 Unlink 2476 1.940 5.520 Qpathinfo 11124 1.211 14.680 Qfileinfo 1940 0.125 1.488 Qfsinfo 2040 0.084 1.546 Sfileinfo 1000 3.314 7.774 Find 4300 0.362 7.248 WriteX 6062 0.715 7.352 ReadX 19268 0.013 0.417 LockX 40 0.983 1.343 UnlockX 40 0.971 1.525 Flush 860 19.736 71.426 Throughput 6.32387 MB/sec 2 clients 2 procs max_latency=71.436 ms stopping dbench in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:44:22 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2a.lustre-rsync-test dbench successfully finished Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 6 seconds Changelog records consumed: 17575 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0001: changelog user 'cl1' not found lustre-MDT0000: changelog user 'cl1' not found PASS 2a (85s) == lustre-rsync-test test 2b: Replicate files changed by dbench. ========================================================== 17:44:32 (1748036672) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl2 cl2' lustre-MDT0000: Registered changelog user cl2 waiting child_pid=$(pgrep dbench), 360 secs left ... looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2b.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:44:34 EDT 2025 waiting for dbench pid 10621 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 500 34.93 MB/sec warmup 1 sec latency 13.827 ms 2 936 21.77 MB/sec warmup 2 sec latency 76.381 ms 2 1580 17.43 MB/sec warmup 3 sec latency 38.321 ms 2 2167 13.36 MB/sec warmup 4 sec latency 26.399 ms 2 2978 13.40 MB/sec warmup 5 sec latency 31.028 ms 2 3813 13.25 MB/sec warmup 6 sec latency 22.926 ms 2 4292 11.55 MB/sec warmup 7 sec latency 52.893 ms 2 4787 10.55 MB/sec warmup 8 sec latency 28.170 ms 2 5448 10.11 MB/sec warmup 9 sec latency 30.112 ms 2 6134 9.61 MB/sec warmup 10 sec latency 26.009 ms 2 7180 10.50 MB/sec warmup 11 sec latency 27.591 ms 2 8151 3.84 MB/sec execute 1 sec latency 52.803 ms 2 8817 5.26 MB/sec execute 2 sec latency 31.015 ms PIDs: 10621 10622 10623 Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 3706 Resuming dbench 2 9128 2.00 MB/sec execute 6 sec latency 3115.142 ms 2 9863 3.34 MB/sec execute 7 sec latency 27.772 ms 2 10742 4.66 MB/sec execute 8 sec latency 35.596 ms 2 11186 4.46 MB/sec execute 9 sec latency 11.767 ms 2 11617 4.17 MB/sec execute 10 sec latency 46.287 ms 2 12274 4.60 MB/sec execute 11 sec latency 24.924 ms 2 12904 4.31 MB/sec execute 12 sec latency 30.300 ms 2 13646 5.04 MB/sec execute 13 sec latency 29.064 ms 2 14485 5.58 MB/sec execute 14 sec latency 21.498 ms 2 14965 5.30 MB/sec execute 15 sec latency 56.019 ms 2 15455 5.18 MB/sec execute 16 sec latency 23.053 ms Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 21281 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 2563 Only in /mnt/lustre/d2b.lustre-rsync-test/clients/client0/~dmtmp/WORD: ~WRL1146.TMP Only in /mnt/lustre/d2b.lustre-rsync-test/clients/client1/~dmtmp/WORD: ~WRL1146.TMP lustre-rsync-test test_2b: @@@@@@ FAIL: Failure in replication; differences found. Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6459:error() = /home/green/git/lustre-release/lustre/tests/lustre-rsync-test.sh:113:check_diff() = /home/green/git/lustre-release/lustre/tests/lustre-rsync-test.sh:323:test_2b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6817:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6864:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6690:run_test() = /home/green/git/lustre-release/lustre/tests/lustre-rsync-test.sh:339:main() Dumping lctl log to /tmp/testlogs//lustre-rsync-test.test_2b.*.1748036704.log rsync: chown "/tmp/testlogs/.lustre-rsync-test.test_2b.debug_log.oleg242-server.1748036704.log.5R0E0u" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.lustre-rsync-test.test_2b.dmesg.oleg242-server.1748036704.log.uzPcfB" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001: Deregistered changelog user #2 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2