-----============= acceptance-small: lustre-rsync-test ============----- Fri May 23 17:43:06 EDT 2025 oleg353-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/lustre-rsync-test.*ex: No such file or directory excepting tests: 2b oleg353-client.virtnet: executing check_config_client /mnt/lustre oleg353-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg353-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff880137305000.idle_timeout=debug osc.lustre-OST0001-osc-ffff880137305000.idle_timeout=debug disable quota as required oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 osd-ldiskfs.track_declares_assert=1 nobody:x:99: == lustre-rsync-test test 2a: Replicate files created by dbench. ========================================================== 17:43:16 (1748036596) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' lustre-MDT0000: Registered changelog user cl1 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2a.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:43:18 EDT 2025 waiting for dbench pid 9613 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs failed to create barrier semaphore 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 524 36.10 MB/sec warmup 1 sec latency 13.041 ms 2 1192 23.04 MB/sec warmup 2 sec latency 10.013 ms 2 1953 17.63 MB/sec warmup 3 sec latency 10.920 ms 2 2773 15.98 MB/sec warmup 4 sec latency 12.720 ms 2 3785 15.90 MB/sec warmup 5 sec latency 10.269 ms 2 4294 13.47 MB/sec warmup 6 sec latency 14.227 ms 2 4955 12.89 MB/sec warmup 7 sec latency 11.624 ms 2 5578 11.41 MB/sec warmup 8 sec latency 10.192 ms 2 6458 11.54 MB/sec warmup 9 sec latency 10.374 ms 2 7349 11.78 MB/sec warmup 10 sec latency 10.005 ms 2 7875 10.83 MB/sec warmup 11 sec latency 15.936 ms 2 9186 1.24 MB/sec execute 1 sec latency 13.440 ms 2 10031 7.46 MB/sec execute 2 sec latency 10.236 ms 2 10906 9.16 MB/sec execute 3 sec latency 10.070 ms 2 11439 7.23 MB/sec execute 4 sec latency 19.162 ms 2 12113 7.65 MB/sec execute 5 sec latency 10.019 ms 2 12738 6.58 MB/sec execute 6 sec latency 11.517 ms 2 13627 7.60 MB/sec execute 7 sec latency 9.781 ms 2 14475 8.22 MB/sec execute 8 sec latency 11.895 ms 2 14996 7.48 MB/sec execute 9 sec latency 26.031 ms 2 15675 7.65 MB/sec execute 10 sec latency 14.472 ms 2 16319 7.07 MB/sec execute 11 sec latency 9.598 ms 2 17176 7.62 MB/sec execute 12 sec latency 10.253 ms 2 18035 8.00 MB/sec execute 13 sec latency 10.375 ms 2 18558 7.55 MB/sec execute 14 sec latency 17.435 ms 2 19248 7.65 MB/sec execute 15 sec latency 16.717 ms 2 19898 7.25 MB/sec execute 16 sec latency 9.495 ms 2 20714 7.63 MB/sec execute 17 sec latency 10.747 ms 2 21587 7.91 MB/sec execute 18 sec latency 9.727 ms 2 22105 7.57 MB/sec execute 19 sec latency 15.513 ms 2 22758 7.65 MB/sec execute 20 sec latency 9.791 ms 2 23348 7.34 MB/sec execute 21 sec latency 14.676 ms 2 24100 7.50 MB/sec execute 22 sec latency 13.681 ms 2 25088 7.85 MB/sec execute 23 sec latency 10.230 ms 2 25594 7.57 MB/sec execute 24 sec latency 18.278 ms 2 26155 7.42 MB/sec execute 25 sec latency 11.296 ms 2 26854 7.39 MB/sec execute 26 sec latency 18.546 ms 2 27627 7.53 MB/sec execute 27 sec latency 9.841 ms 2 28588 7.75 MB/sec execute 28 sec latency 9.537 ms 2 29120 7.58 MB/sec execute 29 sec latency 18.374 ms 2 29802 7.65 MB/sec execute 30 sec latency 9.839 ms 2 30419 7.44 MB/sec execute 31 sec latency 17.321 ms 2 31290 7.56 MB/sec execute 32 sec latency 12.646 ms 2 32185 7.79 MB/sec execute 33 sec latency 11.154 ms 2 32704 7.60 MB/sec execute 34 sec latency 14.691 ms 2 33389 7.65 MB/sec execute 35 sec latency 11.803 ms 2 34031 7.47 MB/sec execute 36 sec latency 10.163 ms 2 34936 7.64 MB/sec execute 37 sec latency 10.999 ms 2 35771 7.77 MB/sec execute 38 sec latency 10.039 ms 2 36298 7.61 MB/sec execute 39 sec latency 16.706 ms 2 36968 7.65 MB/sec execute 40 sec latency 15.038 ms 2 37619 7.50 MB/sec execute 41 sec latency 10.786 ms 2 38477 7.64 MB/sec execute 42 sec latency 9.400 ms 2 39346 7.76 MB/sec execute 43 sec latency 10.158 ms 2 39939 7.63 MB/sec execute 44 sec latency 15.298 ms 2 40665 7.66 MB/sec execute 45 sec latency 15.959 ms 2 41447 7.57 MB/sec execute 46 sec latency 8.817 ms 2 42589 7.85 MB/sec execute 47 sec latency 10.322 ms 2 43156 7.76 MB/sec execute 48 sec latency 8.373 ms 2 43815 7.68 MB/sec execute 49 sec latency 14.493 ms 2 44581 7.67 MB/sec execute 50 sec latency 8.856 ms 2 45526 7.79 MB/sec execute 51 sec latency 8.257 ms 2 46456 7.88 MB/sec execute 52 sec latency 9.156 ms 2 47075 7.78 MB/sec execute 53 sec latency 12.283 ms 2 47826 7.80 MB/sec execute 54 sec latency 9.820 ms 2 48576 7.73 MB/sec execute 55 sec latency 11.306 ms 2 49688 7.96 MB/sec execute 56 sec latency 8.987 ms 2 50205 7.87 MB/sec execute 57 sec latency 9.970 ms 2 50844 7.81 MB/sec execute 58 sec latency 13.895 ms 2 51521 7.79 MB/sec execute 59 sec latency 11.826 ms 2 cleanup 60 sec 0 cleanup 60 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 15192 4.392 16.706 Close 11194 0.906 5.741 Rename 636 6.490 10.741 Unlink 3014 1.623 6.473 Qpathinfo 13816 1.097 11.285 Qfileinfo 2416 0.119 1.408 Qfsinfo 2468 0.076 1.508 Sfileinfo 1230 3.004 15.024 Find 5300 0.333 6.135 WriteX 7500 0.860 18.540 ReadX 23524 0.011 0.835 LockX 48 0.912 1.714 UnlockX 48 0.924 1.285 Flush 1052 5.400 26.023 Throughput 7.78962 MB/sec 2 clients 2 procs max_latency=26.031 ms stopping dbench in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:44:31 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2a.lustre-rsync-test dbench successfully finished Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 8 seconds Changelog records consumed: 21316 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0001: changelog user 'cl1' not found lustre-MDT0000: changelog user 'cl1' not found PASS 2a (86s) == lustre-rsync-test test 2b: Replicate files changed by dbench. ========================================================== 17:44:42 (1748036682) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl2 cl2' lustre-MDT0000: Registered changelog user cl2 waiting child_pid=$(pgrep dbench), 360 secs left ... looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2b.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:44:44 EDT 2025 waiting for dbench pid 10581 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 535 37.42 MB/sec warmup 1 sec latency 14.729 ms 2 1246 23.08 MB/sec warmup 2 sec latency 8.359 ms 2 2033 17.72 MB/sec warmup 3 sec latency 9.894 ms 2 2982 16.76 MB/sec warmup 4 sec latency 9.945 ms 2 3900 16.02 MB/sec warmup 5 sec latency 8.766 ms 2 4462 13.60 MB/sec warmup 6 sec latency 13.255 ms 2 5178 12.94 MB/sec warmup 7 sec latency 9.025 ms 2 5948 11.77 MB/sec warmup 8 sec latency 9.249 ms 2 7109 12.79 MB/sec warmup 9 sec latency 8.534 ms 2 7704 11.84 MB/sec warmup 10 sec latency 8.799 ms 2 8369 11.15 MB/sec warmup 11 sec latency 19.849 ms 2 10060 13.84 MB/sec execute 1 sec latency 10.119 ms 2 10966 13.19 MB/sec execute 2 sec latency 9.450 ms PIDs: 10581 10582 10583 Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 3 seconds Changelog records consumed: 4520 Resuming dbench 2 11202 4.75 MB/sec execute 6 sec latency 3225.185 ms 2 11863 4.66 MB/sec execute 7 sec latency 3225.522 ms 2 12598 4.94 MB/sec execute 8 sec latency 11.224 ms 2 13546 5.84 MB/sec execute 9 sec latency 9.700 ms 2 14453 6.67 MB/sec execute 10 sec latency 9.657 ms 2 15008 6.20 MB/sec execute 11 sec latency 23.348 ms 2 15714 6.46 MB/sec execute 12 sec latency 12.158 ms 2 16431 6.06 MB/sec execute 13 sec latency 9.734 ms 2 17463 6.87 MB/sec execute 14 sec latency 10.073 ms 2 18154 7.04 MB/sec execute 15 sec latency 9.319 ms 2 18743 6.72 MB/sec execute 16 sec latency 11.843 ms Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 25836 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 3079 Wait for dbench to finish 2 18748 6.05 MB/sec execute 17 sec latency 1713.284 ms 2 19477 6.20 MB/sec execute 18 sec latency 1720.865 ms 2 20234 6.08 MB/sec execute 19 sec latency 8.552 ms 2 21354 6.79 MB/sec execute 20 sec latency 9.679 ms 2 21886 6.63 MB/sec execute 21 sec latency 9.782 ms 2 22516 6.51 MB/sec execute 22 sec latency 18.903 ms 2 23160 6.52 MB/sec execute 23 sec latency 24.223 ms 2 23921 6.47 MB/sec execute 24 sec latency 8.366 ms 2 25011 6.98 MB/sec execute 25 sec latency 8.615 ms 2 25557 6.84 MB/sec execute 26 sec latency 15.798 ms 2 26158 6.73 MB/sec execute 27 sec latency 9.466 ms 2 26889 6.73 MB/sec execute 28 sec latency 12.502 ms 2 27793 6.98 MB/sec execute 29 sec latency 8.261 ms 2 28691 7.16 MB/sec execute 30 sec latency 9.969 ms 2 29227 6.98 MB/sec execute 31 sec latency 12.871 ms 2 29924 7.05 MB/sec execute 32 sec latency 9.597 ms 2 30622 6.88 MB/sec execute 33 sec latency 8.714 ms 2 31621 7.12 MB/sec execute 34 sec latency 9.621 ms 2 32345 7.25 MB/sec execute 35 sec latency 8.926 ms 2 32914 7.09 MB/sec execute 36 sec latency 15.361 ms 2 33625 7.14 MB/sec execute 37 sec latency 15.151 ms 2 34355 7.04 MB/sec execute 38 sec latency 8.810 ms 2 35454 7.39 MB/sec execute 39 sec latency 8.942 ms 2 35985 7.30 MB/sec execute 40 sec latency 8.910 ms 2 36661 7.22 MB/sec execute 41 sec latency 12.038 ms 2 37397 7.21 MB/sec execute 42 sec latency 8.746 ms 2 38384 7.31 MB/sec execute 43 sec latency 8.018 ms 2 39335 7.49 MB/sec execute 44 sec latency 8.635 ms 2 39968 7.37 MB/sec execute 45 sec latency 12.700 ms 2 40751 7.40 MB/sec execute 46 sec latency 8.730 ms 2 41555 7.35 MB/sec execute 47 sec latency 7.356 ms 2 42759 7.62 MB/sec execute 48 sec latency 8.373 ms 2 43400 7.53 MB/sec execute 49 sec latency 11.474 ms 2 44219 7.57 MB/sec execute 50 sec latency 9.614 ms 2 45041 7.50 MB/sec execute 51 sec latency 12.454 ms 2 46262 7.76 MB/sec execute 52 sec latency 8.277 ms 2 46892 7.68 MB/sec execute 53 sec latency 12.245 ms 2 47679 7.72 MB/sec execute 54 sec latency 9.274 ms 2 48515 7.64 MB/sec execute 55 sec latency 8.519 ms 2 49697 7.88 MB/sec execute 56 sec latency 8.407 ms 2 50356 7.80 MB/sec execute 57 sec latency 8.133 ms 2 51159 7.84 MB/sec execute 58 sec latency 14.747 ms 2 51923 7.73 MB/sec execute 59 sec latency 9.766 ms 2 cleanup 60 sec 0 cleanup 61 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 15048 4.315 1720.858 Close 11096 0.841 7.636 Rename 646 6.011 9.668 Unlink 3028 1.512 4.115 Qpathinfo 13702 1.029 13.478 Qfileinfo 2426 0.112 1.153 Qfsinfo 2536 0.071 1.125 Sfileinfo 1232 2.758 12.147 Find 5376 1.505 3225.512 WriteX 7550 0.800 11.665 ReadX 24092 0.010 0.845 LockX 52 0.807 1.256 UnlockX 52 0.835 1.219 Flush 1058 4.915 23.334 Throughput 7.73292 MB/sec 2 clients 2 procs max_latency=3225.522 ms stopping dbench in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:45:57 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2b.lustre-rsync-test dbench successfully finished Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 28914 Clear changelog after use: no Errors: 0 lustre_rsync took 10 seconds Changelog records consumed: 13923 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001: Deregistered changelog user #2 lustre-MDT0001: changelog user 'cl2' not found lustre-MDT0000: changelog user 'cl2' not found PASS 2b (90s) == lustre-rsync-test test complete, duration 186 sec ===== 17:46:12 (1748036772)