-----============= acceptance-small: lustre-rsync-test ============----- Fri May 23 17:43:03 EDT 2025 oleg313-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/lustre-rsync-test.*ex: No such file or directory excepting tests: 2b oleg313-client.virtnet: executing check_config_client /mnt/lustre oleg313-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg313-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012a687800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a687800.idle_timeout=debug disable quota as required oleg313-server: oleg313-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 osd-ldiskfs.track_declares_assert=1 nobody:x:99: == lustre-rsync-test test 2a: Replicate files created by dbench. ========================================================== 17:43:14 (1748036594) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' lustre-MDT0000: Registered changelog user cl1 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2a.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:43:16 EDT 2025 waiting for dbench pid 9618 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs failed to create barrier semaphore 1 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 542 38.20 MB/sec warmup 1 sec latency 12.937 ms 2 1226 23.07 MB/sec warmup 2 sec latency 9.195 ms 2 1986 17.64 MB/sec warmup 3 sec latency 11.518 ms 2 2896 16.41 MB/sec warmup 4 sec latency 9.710 ms 2 3824 15.90 MB/sec warmup 5 sec latency 11.282 ms 2 4360 13.52 MB/sec warmup 6 sec latency 14.721 ms 2 5031 12.90 MB/sec warmup 7 sec latency 11.439 ms 2 5660 11.44 MB/sec warmup 8 sec latency 10.307 ms 2 6489 11.69 MB/sec warmup 9 sec latency 9.554 ms 2 7376 11.78 MB/sec warmup 10 sec latency 9.882 ms 2 7902 10.85 MB/sec warmup 11 sec latency 27.540 ms 2 9244 1.28 MB/sec execute 1 sec latency 12.953 ms 2 10086 7.48 MB/sec execute 2 sec latency 9.605 ms 2 10934 9.15 MB/sec execute 3 sec latency 9.732 ms 2 11463 7.28 MB/sec execute 4 sec latency 20.746 ms 2 12130 7.65 MB/sec execute 5 sec latency 9.947 ms 2 12785 6.59 MB/sec execute 6 sec latency 10.974 ms 2 13627 7.60 MB/sec execute 7 sec latency 11.136 ms 2 14468 8.21 MB/sec execute 8 sec latency 9.936 ms 2 14996 7.48 MB/sec execute 9 sec latency 16.782 ms 2 15691 7.65 MB/sec execute 10 sec latency 10.347 ms 2 16352 7.07 MB/sec execute 11 sec latency 10.614 ms 2 17205 7.62 MB/sec execute 12 sec latency 10.000 ms 2 18045 7.99 MB/sec execute 13 sec latency 10.206 ms 2 18610 7.56 MB/sec execute 14 sec latency 24.791 ms 2 19283 7.66 MB/sec execute 15 sec latency 14.155 ms 2 19949 7.26 MB/sec execute 16 sec latency 10.170 ms 2 20795 7.64 MB/sec execute 17 sec latency 11.038 ms 2 21648 7.90 MB/sec execute 18 sec latency 9.403 ms 2 22185 7.59 MB/sec execute 19 sec latency 14.880 ms 2 22844 7.66 MB/sec execute 20 sec latency 13.774 ms 2 23500 7.36 MB/sec execute 21 sec latency 11.130 ms 2 24316 7.64 MB/sec execute 22 sec latency 11.086 ms 2 25179 7.85 MB/sec execute 23 sec latency 9.435 ms 2 25687 7.59 MB/sec execute 24 sec latency 18.635 ms 2 26371 7.65 MB/sec execute 25 sec latency 15.371 ms 2 27034 7.41 MB/sec execute 26 sec latency 9.345 ms 2 27883 7.64 MB/sec execute 27 sec latency 9.961 ms 2 28734 7.81 MB/sec execute 28 sec latency 10.187 ms 2 29297 7.61 MB/sec execute 29 sec latency 15.529 ms 2 29989 7.66 MB/sec execute 30 sec latency 10.070 ms 2 30720 7.52 MB/sec execute 31 sec latency 10.022 ms 2 31690 7.76 MB/sec execute 32 sec latency 11.382 ms 2 32354 7.81 MB/sec execute 33 sec latency 9.775 ms 2 32928 7.63 MB/sec execute 34 sec latency 17.503 ms 2 33602 7.66 MB/sec execute 35 sec latency 9.239 ms 2 34334 7.54 MB/sec execute 36 sec latency 10.767 ms 2 35333 7.75 MB/sec execute 37 sec latency 10.461 ms 2 35942 7.79 MB/sec execute 38 sec latency 10.140 ms 2 36527 7.66 MB/sec execute 39 sec latency 13.580 ms 2 37209 7.66 MB/sec execute 40 sec latency 12.913 ms 2 37922 7.56 MB/sec execute 41 sec latency 9.348 ms 2 39012 7.87 MB/sec execute 42 sec latency 9.869 ms 2 39521 7.77 MB/sec execute 43 sec latency 10.019 ms 2 40116 7.68 MB/sec execute 44 sec latency 14.068 ms 2 40793 7.66 MB/sec execute 45 sec latency 9.897 ms 2 41543 7.60 MB/sec execute 46 sec latency 7.793 ms 2 42695 7.86 MB/sec execute 47 sec latency 10.034 ms 2 43274 7.77 MB/sec execute 48 sec latency 9.082 ms 2 43995 7.80 MB/sec execute 49 sec latency 22.724 ms 2 44698 7.67 MB/sec execute 50 sec latency 16.434 ms 2 45610 7.79 MB/sec execute 51 sec latency 10.644 ms 2 46516 7.90 MB/sec execute 52 sec latency 9.357 ms 2 47147 7.78 MB/sec execute 53 sec latency 12.573 ms 2 47942 7.80 MB/sec execute 54 sec latency 9.450 ms 2 48754 7.76 MB/sec execute 55 sec latency 9.892 ms 2 49923 8.00 MB/sec execute 56 sec latency 8.658 ms 2 50491 7.89 MB/sec execute 57 sec latency 11.629 ms 2 51206 7.91 MB/sec execute 58 sec latency 10.429 ms 2 51939 7.80 MB/sec execute 59 sec latency 8.600 ms 2 cleanup 60 sec 0 cleanup 60 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 15294 4.356 16.424 Close 11270 0.900 6.473 Rename 648 6.383 10.602 Unlink 3036 1.598 2.837 Qpathinfo 13938 1.089 8.018 Qfileinfo 2464 0.119 1.339 Qfsinfo 2510 0.075 1.941 Sfileinfo 1232 2.960 17.493 Find 5334 0.330 6.206 WriteX 7612 0.859 6.190 ReadX 24264 0.011 0.863 LockX 52 0.846 1.120 UnlockX 52 0.886 1.196 Flush 1062 5.391 24.765 Throughput 7.80437 MB/sec 2 clients 2 procs max_latency=24.791 ms stopping dbench in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:44:28 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2a.lustre-rsync-test dbench successfully finished Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 9 seconds Changelog records consumed: 21474 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0001: changelog user 'cl1' not found lustre-MDT0000: changelog user 'cl1' not found PASS 2a (86s) == lustre-rsync-test test 2b: Replicate files changed by dbench. ========================================================== 17:44:40 (1748036680) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl2 cl2' lustre-MDT0000: Registered changelog user cl2 waiting child_pid=$(pgrep dbench), 360 secs left ... looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2b.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:44:42 EDT 2025 waiting for dbench pid 10588 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs 1 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 648 42.12 MB/sec warmup 1 sec latency 10.420 ms 2 1403 26.01 MB/sec warmup 2 sec latency 8.599 ms 2 2133 17.78 MB/sec warmup 3 sec latency 11.859 ms 2 3183 17.06 MB/sec warmup 4 sec latency 9.287 ms 2 3967 16.02 MB/sec warmup 5 sec latency 9.610 ms 2 4569 13.72 MB/sec warmup 6 sec latency 14.308 ms 2 5296 12.97 MB/sec warmup 7 sec latency 12.427 ms 2 6000 11.80 MB/sec warmup 8 sec latency 9.858 ms 2 7134 12.79 MB/sec warmup 9 sec latency 9.971 ms 2 7703 11.84 MB/sec warmup 10 sec latency 9.463 ms 2 8371 11.15 MB/sec warmup 11 sec latency 14.845 ms 2 10083 13.77 MB/sec execute 1 sec latency 10.944 ms 2 10971 13.13 MB/sec execute 2 sec latency 8.867 ms PIDs: 10588 10589 10590 Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 4553 Resuming dbench 2 11249 4.73 MB/sec execute 6 sec latency 3185.861 ms 2 11880 4.65 MB/sec execute 7 sec latency 3185.894 ms 2 12640 4.93 MB/sec execute 8 sec latency 11.357 ms 2 13552 5.82 MB/sec execute 9 sec latency 10.657 ms 2 14479 6.65 MB/sec execute 10 sec latency 9.249 ms 2 15069 6.21 MB/sec execute 11 sec latency 13.253 ms 2 15763 6.45 MB/sec execute 12 sec latency 15.532 ms 2 16530 6.21 MB/sec execute 13 sec latency 9.520 ms 2 17544 6.86 MB/sec execute 14 sec latency 9.612 ms 2 18175 7.03 MB/sec execute 15 sec latency 9.668 ms 2 18793 6.83 MB/sec execute 16 sec latency 18.411 ms Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 26027 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 3048 Wait for dbench to finish 2 18794 6.18 MB/sec execute 17 sec latency 1652.221 ms 2 19480 6.21 MB/sec execute 18 sec latency 1652.249 ms 2 20213 6.09 MB/sec execute 19 sec latency 9.088 ms 2 21344 6.80 MB/sec execute 20 sec latency 9.227 ms 2 21894 6.64 MB/sec execute 21 sec latency 9.858 ms 2 22530 6.53 MB/sec execute 22 sec latency 12.819 ms 2 23190 6.53 MB/sec execute 23 sec latency 33.360 ms 2 23934 6.48 MB/sec execute 24 sec latency 9.126 ms 2 25035 7.00 MB/sec execute 25 sec latency 9.196 ms 2 25602 6.85 MB/sec execute 26 sec latency 12.759 ms 2 26281 6.95 MB/sec execute 27 sec latency 9.186 ms 2 26908 6.74 MB/sec execute 28 sec latency 9.572 ms 2 27802 6.99 MB/sec execute 29 sec latency 8.441 ms 2 28691 7.17 MB/sec execute 30 sec latency 9.201 ms 2 29247 6.99 MB/sec execute 31 sec latency 23.364 ms 2 29943 7.06 MB/sec execute 32 sec latency 12.071 ms 2 30646 6.89 MB/sec execute 33 sec latency 9.283 ms 2 31626 7.13 MB/sec execute 34 sec latency 8.827 ms 2 32350 7.25 MB/sec execute 35 sec latency 9.925 ms 2 32914 7.10 MB/sec execute 36 sec latency 14.134 ms 2 33623 7.15 MB/sec execute 37 sec latency 10.050 ms 2 34367 7.05 MB/sec execute 38 sec latency 10.754 ms 2 35481 7.40 MB/sec execute 39 sec latency 8.577 ms 2 35994 7.30 MB/sec execute 40 sec latency 9.587 ms 2 36648 7.22 MB/sec execute 41 sec latency 12.534 ms 2 37328 7.21 MB/sec execute 42 sec latency 10.585 ms 2 38091 7.17 MB/sec execute 43 sec latency 9.783 ms 2 39254 7.49 MB/sec execute 44 sec latency 8.302 ms 2 39836 7.36 MB/sec execute 45 sec latency 12.289 ms 2 40565 7.40 MB/sec execute 46 sec latency 8.786 ms 2 41354 7.32 MB/sec execute 47 sec latency 8.633 ms 2 42443 7.49 MB/sec execute 48 sec latency 9.749 ms 2 43134 7.52 MB/sec execute 49 sec latency 8.633 ms 2 43849 7.46 MB/sec execute 50 sec latency 11.430 ms 2 44689 7.45 MB/sec execute 51 sec latency 10.879 ms 2 45867 7.64 MB/sec execute 52 sec latency 9.623 ms 2 46629 7.67 MB/sec execute 53 sec latency 7.998 ms 2 47373 7.61 MB/sec execute 54 sec latency 23.586 ms 2 48209 7.60 MB/sec execute 55 sec latency 10.036 ms 2 49350 7.74 MB/sec execute 56 sec latency 8.600 ms 2 50159 7.80 MB/sec execute 57 sec latency 7.843 ms 2 50922 7.74 MB/sec execute 58 sec latency 18.482 ms 2 51810 7.73 MB/sec execute 59 sec latency 9.393 ms 2 cleanup 60 sec 0 cleanup 61 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 15006 4.098 19.293 Close 11066 0.838 4.647 Rename 646 6.064 10.035 Unlink 3020 2.556 3185.882 Qpathinfo 13660 1.262 3185.766 Qfileinfo 2416 0.113 1.333 Qfsinfo 2524 0.072 1.040 Sfileinfo 1232 2.788 8.855 Find 5340 0.309 6.248 WriteX 7512 1.246 1652.241 ReadX 24080 0.010 0.735 LockX 52 0.806 1.138 UnlockX 52 0.813 1.232 Flush 1056 5.040 33.338 Throughput 7.73453 MB/sec 2 clients 2 procs max_latency=3185.894 ms stopping dbench in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:45:55 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2b.lustre-rsync-test dbench successfully finished Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 29074 Clear changelog after use: no Errors: 0 lustre_rsync took 7 seconds Changelog records consumed: 13883 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001: Deregistered changelog user #2 lustre-MDT0001: changelog user 'cl2' not found lustre-MDT0000: changelog user 'cl2' not found PASS 2b (86s) == lustre-rsync-test test complete, duration 183 sec ===== 17:46:07 (1748036767)