-----============= acceptance-small: lustre-rsync-test ============----- Fri May 23 17:42:55 EDT 2025 oleg216-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/lustre-rsync-test.*ex: No such file or directory excepting tests: 2b oleg216-client.virtnet: executing check_config_client /mnt/lustre oleg216-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg216-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b64cd000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b64cd000.idle_timeout=debug disable quota as required oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 nobody:x:99: == lustre-rsync-test test 2a: Replicate files created by dbench. ========================================================== 17:43:06 (1748036586) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' lustre-MDT0000: Registered changelog user cl1 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2a.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:43:08 EDT 2025 waiting for dbench pid 9647 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs failed to create barrier semaphore 1 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 498 34.67 MB/sec warmup 1 sec latency 15.075 ms 2 983 21.90 MB/sec warmup 2 sec latency 72.755 ms 2 1713 17.49 MB/sec warmup 3 sec latency 34.732 ms 2 2426 14.05 MB/sec warmup 4 sec latency 28.925 ms 2 3366 14.18 MB/sec warmup 5 sec latency 30.176 ms 2 4010 13.35 MB/sec warmup 6 sec latency 9.798 ms 2 4486 11.68 MB/sec warmup 7 sec latency 49.877 ms 2 5184 11.32 MB/sec warmup 8 sec latency 30.910 ms 2 5897 10.43 MB/sec warmup 9 sec latency 28.263 ms 2 6732 10.65 MB/sec warmup 10 sec latency 33.619 ms 2 7412 10.71 MB/sec warmup 11 sec latency 25.711 ms 2 8377 3.61 MB/sec execute 1 sec latency 69.547 ms 2 8995 5.04 MB/sec execute 2 sec latency 36.936 ms 2 9679 5.08 MB/sec execute 3 sec latency 27.318 ms 2 10675 8.55 MB/sec execute 4 sec latency 30.730 ms 2 11138 7.49 MB/sec execute 5 sec latency 9.729 ms 2 11597 6.52 MB/sec execute 6 sec latency 53.532 ms 2 12274 6.84 MB/sec execute 7 sec latency 23.563 ms 2 12909 6.14 MB/sec execute 8 sec latency 30.842 ms 2 13664 6.96 MB/sec execute 9 sec latency 28.094 ms 2 14481 7.51 MB/sec execute 10 sec latency 33.576 ms 2 14930 6.95 MB/sec execute 11 sec latency 44.513 ms 2 15426 6.66 MB/sec execute 12 sec latency 35.352 ms 2 16038 6.66 MB/sec execute 13 sec latency 30.178 ms 2 16669 6.50 MB/sec execute 14 sec latency 40.531 ms 2 17685 7.37 MB/sec execute 15 sec latency 29.418 ms 2 18178 7.12 MB/sec execute 16 sec latency 10.904 ms 2 18622 6.78 MB/sec execute 17 sec latency 48.809 ms 2 19240 6.90 MB/sec execute 18 sec latency 33.665 ms 2 19817 6.59 MB/sec execute 19 sec latency 31.777 ms 2 20533 6.80 MB/sec execute 20 sec latency 31.008 ms 2 21407 7.11 MB/sec execute 21 sec latency 26.184 ms 2 21878 6.92 MB/sec execute 22 sec latency 9.958 ms 2 22303 6.70 MB/sec execute 23 sec latency 63.549 ms 2 22939 6.78 MB/sec execute 24 sec latency 37.722 ms 2 23552 6.56 MB/sec execute 25 sec latency 27.517 ms 2 24305 6.82 MB/sec execute 26 sec latency 33.847 ms 2 25127 7.03 MB/sec execute 27 sec latency 23.619 ms 2 25604 6.83 MB/sec execute 28 sec latency 59.467 ms 2 26061 6.71 MB/sec execute 29 sec latency 33.180 ms 2 26682 6.71 MB/sec execute 30 sec latency 32.145 ms 2 27312 6.61 MB/sec execute 31 sec latency 26.623 ms 2 28226 6.86 MB/sec execute 32 sec latency 33.674 ms 2 28828 6.93 MB/sec execute 33 sec latency 10.382 ms 2 29291 6.77 MB/sec execute 34 sec latency 51.920 ms 2 29881 6.83 MB/sec execute 35 sec latency 28.689 ms 2 30434 6.66 MB/sec execute 36 sec latency 33.162 ms 2 31183 6.78 MB/sec execute 37 sec latency 27.463 ms 2 32079 6.96 MB/sec execute 38 sec latency 37.517 ms 2 32589 6.85 MB/sec execute 39 sec latency 10.395 ms 2 33010 6.77 MB/sec execute 40 sec latency 58.272 ms 2 33623 6.77 MB/sec execute 41 sec latency 30.348 ms 2 34308 6.68 MB/sec execute 42 sec latency 28.928 ms 2 35132 6.82 MB/sec execute 43 sec latency 31.086 ms 2 35814 6.92 MB/sec execute 44 sec latency 22.637 ms 2 36277 6.80 MB/sec execute 45 sec latency 57.040 ms 2 36783 6.73 MB/sec execute 46 sec latency 31.416 ms 2 37398 6.73 MB/sec execute 47 sec latency 31.389 ms 2 38008 6.68 MB/sec execute 48 sec latency 32.909 ms 2 39042 6.94 MB/sec execute 49 sec latency 29.174 ms 2 39522 6.87 MB/sec execute 50 sec latency 9.955 ms 2 39979 6.77 MB/sec execute 51 sec latency 49.637 ms 2 40618 6.80 MB/sec execute 52 sec latency 31.774 ms 2 41231 6.70 MB/sec execute 53 sec latency 29.595 ms 2 42036 6.83 MB/sec execute 54 sec latency 26.049 ms 2 42871 6.93 MB/sec execute 55 sec latency 30.394 ms 2 43339 6.83 MB/sec execute 56 sec latency 47.932 ms 2 43863 6.77 MB/sec execute 57 sec latency 26.348 ms 2 44568 6.77 MB/sec execute 58 sec latency 24.981 ms 2 45469 6.85 MB/sec execute 59 sec latency 31.403 ms 2 cleanup 60 sec 0 cleanup 60 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 13245 4.570 11.908 Close 9808 1.048 5.799 Rename 570 6.238 10.364 Unlink 2615 1.775 2.961 Qpathinfo 12040 1.130 7.815 Qfileinfo 2124 0.117 1.659 Qfsinfo 2206 0.075 1.275 Sfileinfo 1088 3.019 5.862 Find 4668 0.323 6.843 WriteX 6616 0.660 6.804 ReadX 21107 0.011 1.109 LockX 44 0.882 1.234 UnlockX 44 0.927 1.239 Flush 922 18.470 63.539 Throughput 6.85081 MB/sec 2 clients 2 procs max_latency=69.547 ms stopping dbench in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:44:21 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2a.lustre-rsync-test dbench successfully finished Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 6 seconds Changelog records consumed: 18861 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0001: changelog user 'cl1' not found lustre-MDT0000: changelog user 'cl1' not found PASS 2a (84s) == lustre-rsync-test test 2b: Replicate files changed by dbench. ========================================================== 17:44:30 (1748036670) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl2 cl2' lustre-MDT0000: Registered changelog user cl2 waiting child_pid=$(pgrep dbench), 360 secs left ... looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2b.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:44:33 EDT 2025 waiting for dbench pid 10618 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 603 41.94 MB/sec warmup 1 sec latency 12.954 ms 2 1041 22.89 MB/sec warmup 2 sec latency 70.946 ms 2 1749 17.52 MB/sec warmup 3 sec latency 27.033 ms 2 2416 14.05 MB/sec warmup 4 sec latency 27.885 ms 2 3372 14.18 MB/sec warmup 5 sec latency 33.095 ms 2 4022 13.35 MB/sec warmup 6 sec latency 9.702 ms 2 4487 11.68 MB/sec warmup 7 sec latency 42.175 ms 2 5199 11.32 MB/sec warmup 8 sec latency 31.634 ms 2 5916 10.45 MB/sec warmup 9 sec latency 25.618 ms 2 6860 10.90 MB/sec warmup 10 sec latency 32.427 ms 2 7537 10.76 MB/sec warmup 11 sec latency 13.741 ms 2 8742 8.81 MB/sec execute 1 sec latency 44.901 ms 2 9493 6.20 MB/sec execute 2 sec latency 24.888 ms PIDs: 10618 10619 10620 Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 3 seconds Changelog records consumed: 3978 Resuming dbench 2 9975 3.70 MB/sec execute 6 sec latency 3138.823 ms 2 10817 5.13 MB/sec execute 7 sec latency 3138.892 ms 2 11331 4.85 MB/sec execute 8 sec latency 50.170 ms 2 11802 4.71 MB/sec execute 9 sec latency 34.029 ms 2 12436 4.91 MB/sec execute 10 sec latency 35.232 ms 2 13095 4.79 MB/sec execute 11 sec latency 28.405 ms 2 14142 6.14 MB/sec execute 12 sec latency 34.657 ms 2 14661 5.92 MB/sec execute 13 sec latency 10.711 ms 2 15154 5.61 MB/sec execute 14 sec latency 45.837 ms 2 15872 5.83 MB/sec execute 15 sec latency 28.058 ms 2 16584 5.68 MB/sec execute 16 sec latency 27.956 ms Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 22839 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 2743 Wait for dbench to finish 2 16584 5.15 MB/sec execute 17 sec latency 1626.002 ms 2 17497 5.68 MB/sec execute 18 sec latency 1632.654 ms 2 18161 5.87 MB/sec execute 19 sec latency 14.758 ms 2 18658 5.65 MB/sec execute 20 sec latency 53.095 ms 2 19369 5.81 MB/sec execute 21 sec latency 37.017 ms 2 20090 5.70 MB/sec execute 22 sec latency 33.263 ms 2 20977 6.01 MB/sec execute 23 sec latency 36.281 ms 2 21720 6.24 MB/sec execute 24 sec latency 23.488 ms 2 22210 6.05 MB/sec execute 25 sec latency 52.192 ms 2 22915 6.16 MB/sec execute 26 sec latency 31.987 ms 2 23673 6.06 MB/sec execute 27 sec latency 27.270 ms 2 24623 6.38 MB/sec execute 28 sec latency 26.844 ms 2 25310 6.48 MB/sec execute 29 sec latency 16.534 ms 2 25796 6.32 MB/sec execute 30 sec latency 40.643 ms 2 26513 6.40 MB/sec execute 31 sec latency 29.907 ms 2 27250 6.31 MB/sec execute 32 sec latency 32.324 ms 2 28209 6.57 MB/sec execute 33 sec latency 27.503 ms 2 28854 6.65 MB/sec execute 34 sec latency 10.967 ms 2 29340 6.51 MB/sec execute 35 sec latency 47.664 ms 2 30057 6.57 MB/sec execute 36 sec latency 23.908 ms 2 30796 6.49 MB/sec execute 37 sec latency 26.057 ms 2 31749 6.71 MB/sec execute 38 sec latency 28.426 ms 2 32415 6.78 MB/sec execute 39 sec latency 17.424 ms 2 32910 6.65 MB/sec execute 40 sec latency 50.214 ms 2 33616 6.70 MB/sec execute 41 sec latency 31.840 ms 2 34352 6.63 MB/sec execute 42 sec latency 24.510 ms 2 35309 6.82 MB/sec execute 43 sec latency 33.021 ms 2 35952 6.88 MB/sec execute 44 sec latency 9.302 ms 2 36442 6.76 MB/sec execute 45 sec latency 56.833 ms 2 37165 6.81 MB/sec execute 46 sec latency 43.110 ms 2 37876 6.73 MB/sec execute 47 sec latency 26.231 ms 2 38782 6.90 MB/sec execute 48 sec latency 31.524 ms 2 39474 6.95 MB/sec execute 49 sec latency 12.293 ms 2 39958 6.85 MB/sec execute 50 sec latency 46.767 ms 2 40648 6.89 MB/sec execute 51 sec latency 36.624 ms 2 41404 6.82 MB/sec execute 52 sec latency 23.395 ms 2 42263 6.93 MB/sec execute 53 sec latency 36.729 ms 2 43003 7.02 MB/sec execute 54 sec latency 14.034 ms 2 43521 6.92 MB/sec execute 55 sec latency 41.768 ms 2 44250 6.96 MB/sec execute 56 sec latency 24.062 ms 2 45011 6.90 MB/sec execute 57 sec latency 23.465 ms 2 46175 7.14 MB/sec execute 58 sec latency 25.460 ms 2 46858 7.08 MB/sec execute 59 sec latency 46.679 ms 2 cleanup 60 sec 0 cleanup 60 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 13666 4.520 3138.886 Close 10072 0.928 3.319 Rename 586 11.147 1632.637 Unlink 2752 1.617 3.383 Qpathinfo 12414 1.018 13.486 Qfileinfo 2158 0.108 1.122 Qfsinfo 2280 0.067 1.545 Sfileinfo 1142 2.632 6.779 Find 4792 0.295 6.218 WriteX 6760 0.599 7.740 ReadX 21506 0.010 0.710 LockX 44 0.791 1.377 UnlockX 44 0.805 0.995 Flush 954 17.019 56.802 Throughput 7.08043 MB/sec 2 clients 2 procs max_latency=3138.892 ms stopping dbench in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:45:45 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2b.lustre-rsync-test dbench successfully finished Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 25581 Clear changelog after use: no Errors: 0 lustre_rsync took 4 seconds Changelog records consumed: 12679 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001: Deregistered changelog user #2 lustre-MDT0001: changelog user 'cl2' not found lustre-MDT0000: changelog user 'cl2' not found PASS 2b (83s) == lustre-rsync-test test complete, duration 177 sec ===== 17:45:53 (1748036753)