-----============= acceptance-small: lustre-rsync-test ============----- Fri May 23 17:42:47 EDT 2025 oleg119-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/lustre-rsync-test.*ex: No such file or directory excepting tests: 2b oleg119-client.virtnet: executing check_config_client /mnt/lustre oleg119-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg119-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012a592000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a592000.idle_timeout=debug disable quota as required oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 nobody:x:99: == lustre-rsync-test test 2a: Replicate files created by dbench. ========================================================== 17:42:59 (1748036579) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' lustre-MDT0000: Registered changelog user cl1 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2a.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:43:01 EDT 2025 waiting for dbench pid 9628 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs failed to create barrier semaphore 1 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 664 42.31 MB/sec warmup 1 sec latency 17.285 ms 2 1221 23.07 MB/sec warmup 2 sec latency 81.381 ms 2 1977 17.63 MB/sec warmup 3 sec latency 31.633 ms 2 2922 16.74 MB/sec warmup 4 sec latency 29.521 ms 2 3830 15.90 MB/sec warmup 5 sec latency 24.156 ms 2 4362 13.53 MB/sec warmup 6 sec latency 41.433 ms 2 5052 12.90 MB/sec warmup 7 sec latency 34.878 ms 2 5720 11.46 MB/sec warmup 8 sec latency 31.053 ms 2 6545 11.70 MB/sec warmup 9 sec latency 29.611 ms 2 7404 11.78 MB/sec warmup 10 sec latency 20.848 ms 2 7902 10.85 MB/sec warmup 11 sec latency 55.837 ms 2 9213 1.31 MB/sec execute 1 sec latency 35.758 ms 2 10039 7.47 MB/sec execute 2 sec latency 25.770 ms 2 10919 9.16 MB/sec execute 3 sec latency 26.965 ms 2 11420 7.23 MB/sec execute 4 sec latency 56.311 ms 2 12043 7.64 MB/sec execute 5 sec latency 33.230 ms 2 12660 6.54 MB/sec execute 6 sec latency 34.110 ms 2 13424 7.16 MB/sec execute 7 sec latency 28.016 ms 2 14369 8.01 MB/sec execute 8 sec latency 31.282 ms 2 14877 7.42 MB/sec execute 9 sec latency 52.467 ms 2 15369 7.05 MB/sec execute 10 sec latency 33.757 ms 2 16029 7.01 MB/sec execute 11 sec latency 28.389 ms 2 16674 6.82 MB/sec execute 12 sec latency 37.384 ms 2 17737 7.79 MB/sec execute 13 sec latency 26.002 ms 2 18227 7.47 MB/sec execute 14 sec latency 10.344 ms 2 18722 7.09 MB/sec execute 15 sec latency 62.033 ms 2 19404 7.19 MB/sec execute 16 sec latency 24.386 ms 2 20130 6.97 MB/sec execute 17 sec latency 23.626 ms 2 20989 7.30 MB/sec execute 18 sec latency 30.727 ms 2 21697 7.52 MB/sec execute 19 sec latency 13.834 ms 2 22160 7.20 MB/sec execute 20 sec latency 52.319 ms 2 22823 7.29 MB/sec execute 21 sec latency 32.424 ms 2 23472 7.02 MB/sec execute 22 sec latency 34.549 ms 2 24305 7.31 MB/sec execute 23 sec latency 33.129 ms 2 25151 7.52 MB/sec execute 24 sec latency 25.250 ms 2 25626 7.28 MB/sec execute 25 sec latency 51.630 ms 2 26259 7.36 MB/sec execute 26 sec latency 30.868 ms 2 26846 7.12 MB/sec execute 27 sec latency 29.494 ms 2 27569 7.21 MB/sec execute 28 sec latency 24.981 ms 2 28540 7.47 MB/sec execute 29 sec latency 28.906 ms 2 29093 7.33 MB/sec execute 30 sec latency 14.769 ms 2 29549 7.21 MB/sec execute 31 sec latency 58.507 ms 2 30216 7.19 MB/sec execute 32 sec latency 29.375 ms 2 30892 7.11 MB/sec execute 33 sec latency 25.033 ms 2 31945 7.48 MB/sec execute 34 sec latency 28.633 ms 2 32464 7.36 MB/sec execute 35 sec latency 9.723 ms 2 32954 7.21 MB/sec execute 36 sec latency 54.719 ms 2 33630 7.25 MB/sec execute 37 sec latency 30.588 ms 2 34349 7.15 MB/sec execute 38 sec latency 25.159 ms 2 35258 7.34 MB/sec execute 39 sec latency 28.142 ms 2 35903 7.40 MB/sec execute 40 sec latency 14.179 ms 2 36366 7.25 MB/sec execute 41 sec latency 55.139 ms 2 36996 7.29 MB/sec execute 42 sec latency 29.007 ms 2 37575 7.14 MB/sec execute 43 sec latency 31.026 ms 2 38406 7.26 MB/sec execute 44 sec latency 25.294 ms 2 39211 7.37 MB/sec execute 45 sec latency 28.339 ms 2 39732 7.27 MB/sec execute 46 sec latency 10.936 ms 2 40141 7.19 MB/sec execute 47 sec latency 55.370 ms 2 40779 7.18 MB/sec execute 48 sec latency 26.680 ms 2 41466 7.11 MB/sec execute 49 sec latency 28.241 ms 2 42379 7.26 MB/sec execute 50 sec latency 34.240 ms 2 42981 7.30 MB/sec execute 51 sec latency 13.030 ms 2 43406 7.18 MB/sec execute 52 sec latency 74.190 ms 2 44051 7.22 MB/sec execute 53 sec latency 26.416 ms 2 44625 7.10 MB/sec execute 54 sec latency 36.335 ms 2 45475 7.18 MB/sec execute 55 sec latency 32.318 ms 2 46309 7.29 MB/sec execute 56 sec latency 26.048 ms 2 46824 7.21 MB/sec execute 57 sec latency 52.395 ms 2 47295 7.15 MB/sec execute 58 sec latency 55.663 ms 2 47966 7.14 MB/sec execute 59 sec latency 26.262 ms 2 cleanup 60 sec 0 cleanup 60 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 13948 4.319 10.908 Close 10280 0.987 3.697 Rename 580 6.003 9.506 Unlink 2764 1.724 3.493 Qpathinfo 12676 1.075 6.764 Qfileinfo 2220 0.115 1.243 Qfsinfo 2260 0.075 1.150 Sfileinfo 1126 2.822 5.757 Find 4852 0.327 5.948 WriteX 6886 0.641 7.161 ReadX 21508 0.011 0.138 LockX 44 0.803 1.099 UnlockX 44 0.847 0.987 Flush 966 17.967 74.147 Throughput 7.14157 MB/sec 2 clients 2 procs max_latency=74.190 ms stopping dbench in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:44:13 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2a.lustre-rsync-test dbench successfully finished Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 8 seconds Changelog records consumed: 19852 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0001: changelog user 'cl1' not found lustre-MDT0000: changelog user 'cl1' not found PASS 2a (86s) == lustre-rsync-test test 2b: Replicate files changed by dbench. ========================================================== 17:44:24 (1748036664) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl2 cl2' lustre-MDT0000: Registered changelog user cl2 waiting child_pid=$(pgrep dbench), 360 secs left ... looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2b.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:44:27 EDT 2025 waiting for dbench pid 10600 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 642 42.12 MB/sec warmup 1 sec latency 13.921 ms 2 1147 22.99 MB/sec warmup 2 sec latency 69.955 ms 2 1884 17.58 MB/sec warmup 3 sec latency 26.178 ms 2 2763 15.98 MB/sec warmup 4 sec latency 29.561 ms 2 3797 15.90 MB/sec warmup 5 sec latency 28.429 ms 2 4311 13.48 MB/sec warmup 6 sec latency 46.570 ms 2 4994 12.90 MB/sec warmup 7 sec latency 37.775 ms 2 5622 11.43 MB/sec warmup 8 sec latency 26.218 ms 2 6458 11.54 MB/sec warmup 9 sec latency 26.467 ms 2 7341 11.78 MB/sec warmup 10 sec latency 34.837 ms 2 7836 10.82 MB/sec warmup 11 sec latency 48.298 ms 2 9097 6.65 MB/sec execute 1 sec latency 45.667 ms 2 9868 8.82 MB/sec execute 2 sec latency 35.885 ms PIDs: 10600 10601 10602 Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 3 seconds Changelog records consumed: 4074 Resuming dbench 2 10404 4.26 MB/sec execute 6 sec latency 3247.802 ms 2 11060 5.02 MB/sec execute 7 sec latency 3247.856 ms 2 11516 4.54 MB/sec execute 8 sec latency 53.349 ms 2 12142 5.04 MB/sec execute 9 sec latency 31.870 ms 2 12819 4.66 MB/sec execute 10 sec latency 24.871 ms 2 13674 5.50 MB/sec execute 11 sec latency 26.078 ms 2 14562 6.14 MB/sec execute 12 sec latency 22.978 ms 2 15043 5.75 MB/sec execute 13 sec latency 50.021 ms 2 15714 5.99 MB/sec execute 14 sec latency 38.608 ms 2 16365 5.68 MB/sec execute 15 sec latency 26.169 ms 2 17170 6.18 MB/sec execute 16 sec latency 36.240 ms Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 23926 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 2809 Wait for dbench to finish 2 17171 5.52 MB/sec execute 18 sec latency 1878.756 ms 2 18003 5.89 MB/sec execute 19 sec latency 1878.284 ms 2 18476 5.66 MB/sec execute 20 sec latency 56.000 ms 2 18964 5.56 MB/sec execute 21 sec latency 36.786 ms 2 19591 5.60 MB/sec execute 22 sec latency 42.757 ms 2 20262 5.57 MB/sec execute 23 sec latency 36.887 ms 2 21331 6.16 MB/sec execute 24 sec latency 31.116 ms 2 21878 6.04 MB/sec execute 25 sec latency 9.064 ms 2 22322 5.88 MB/sec execute 26 sec latency 45.297 ms 2 22989 5.98 MB/sec execute 27 sec latency 41.018 ms 2 23686 5.89 MB/sec execute 28 sec latency 29.023 ms 2 24533 6.13 MB/sec execute 29 sec latency 34.739 ms 2 25242 6.31 MB/sec execute 30 sec latency 14.013 ms 2 25687 6.14 MB/sec execute 31 sec latency 45.787 ms 2 26346 6.23 MB/sec execute 32 sec latency 23.354 ms 2 26949 6.08 MB/sec execute 33 sec latency 32.571 ms 2 27773 6.27 MB/sec execute 34 sec latency 29.761 ms 2 28638 6.49 MB/sec execute 35 sec latency 29.269 ms 2 29127 6.34 MB/sec execute 36 sec latency 44.074 ms 2 29638 6.26 MB/sec execute 37 sec latency 46.679 ms 2 30284 6.27 MB/sec execute 38 sec latency 29.950 ms 2 30991 6.24 MB/sec execute 39 sec latency 28.636 ms 2 31995 6.57 MB/sec execute 40 sec latency 30.651 ms 2 32518 6.49 MB/sec execute 41 sec latency 9.599 ms 2 32957 6.37 MB/sec execute 42 sec latency 54.126 ms 2 33630 6.43 MB/sec execute 43 sec latency 32.010 ms 2 34339 6.36 MB/sec execute 44 sec latency 28.066 ms 2 35194 6.50 MB/sec execute 45 sec latency 31.728 ms 2 35874 6.61 MB/sec execute 46 sec latency 14.454 ms 2 36317 6.49 MB/sec execute 47 sec latency 63.920 ms 2 36966 6.55 MB/sec execute 48 sec latency 28.651 ms 2 37535 6.43 MB/sec execute 49 sec latency 31.344 ms 2 38280 6.52 MB/sec execute 50 sec latency 35.463 ms 2 39197 6.66 MB/sec execute 51 sec latency 22.560 ms 2 39732 6.59 MB/sec execute 52 sec latency 28.538 ms 2 40149 6.53 MB/sec execute 53 sec latency 58.925 ms 2 40793 6.54 MB/sec execute 54 sec latency 27.939 ms 2 41487 6.48 MB/sec execute 55 sec latency 28.450 ms 2 42423 6.63 MB/sec execute 56 sec latency 27.254 ms 2 43041 6.68 MB/sec execute 57 sec latency 9.870 ms 2 43521 6.59 MB/sec execute 58 sec latency 45.727 ms 2 44212 6.63 MB/sec execute 59 sec latency 28.419 ms 2 44945 6.57 MB/sec execute 60 sec latency 27.027 ms 2 cleanup 61 sec 0 cleanup 61 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 12900 4.303 11.556 Close 9516 0.984 4.698 Rename 544 6.005 10.107 Unlink 2548 1.730 3.487 Qpathinfo 11756 1.070 6.506 Qfileinfo 2084 0.114 1.281 Qfsinfo 2108 0.075 1.468 Sfileinfo 1032 2.822 5.128 Find 4498 0.312 6.100 WriteX 6412 1.226 1878.746 ReadX 20614 0.011 1.157 LockX 44 0.835 1.070 UnlockX 44 0.844 1.096 Flush 890 25.638 3247.846 Throughput 6.57481 MB/sec 2 clients 2 procs max_latency=3247.856 ms stopping dbench in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:45:40 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2b.lustre-rsync-test dbench successfully finished Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 26734 Clear changelog after use: no Errors: 0 lustre_rsync took 4 seconds Changelog records consumed: 11729 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001: Deregistered changelog user #2 lustre-MDT0001: changelog user 'cl2' not found lustre-MDT0000: changelog user 'cl2' not found PASS 2b (84s) == lustre-rsync-test test complete, duration 180 sec ===== 17:45:48 (1748036748)