-----============= acceptance-small: lustre-rsync-test ============----- Fri May 23 17:42:56 EDT 2025 oleg418-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/lustre-rsync-test.*ex: No such file or directory excepting tests: 2b oleg418-client.virtnet: executing check_config_client /mnt/lustre oleg418-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg418-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800abb2c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800abb2c800.idle_timeout=debug disable quota as required oleg418-server: oleg418-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 nobody:x:99: == lustre-rsync-test test 2a: Replicate files created by dbench. ========================================================== 17:43:10 (1748036590) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' lustre-MDT0000: Registered changelog user cl1 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2a.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:43:12 EDT 2025 waiting for dbench pid 9633 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs failed to create barrier semaphore 1 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 664 42.31 MB/sec warmup 1 sec latency 51.778 ms 2 1262 23.11 MB/sec warmup 2 sec latency 58.371 ms 2 2118 17.78 MB/sec warmup 3 sec latency 31.085 ms 2 3019 16.77 MB/sec warmup 4 sec latency 26.256 ms 2 3978 16.02 MB/sec warmup 5 sec latency 24.202 ms 2 4543 13.65 MB/sec warmup 6 sec latency 43.602 ms 2 5336 12.97 MB/sec warmup 7 sec latency 36.140 ms 2 6135 12.02 MB/sec warmup 8 sec latency 25.626 ms 2 7304 13.08 MB/sec warmup 9 sec latency 21.516 ms 2 7854 11.91 MB/sec warmup 10 sec latency 42.811 ms 2 8577 11.69 MB/sec warmup 11 sec latency 28.982 ms 2 10336 15.41 MB/sec execute 1 sec latency 31.827 ms 2 11105 13.28 MB/sec execute 2 sec latency 11.271 ms 2 11639 9.49 MB/sec execute 3 sec latency 47.128 ms 2 12438 9.31 MB/sec execute 4 sec latency 21.387 ms 2 13218 8.50 MB/sec execute 5 sec latency 26.088 ms 2 14336 10.36 MB/sec execute 6 sec latency 25.115 ms 2 14919 9.33 MB/sec execute 7 sec latency 40.343 ms 2 15601 9.37 MB/sec execute 8 sec latency 26.109 ms 2 16310 8.47 MB/sec execute 9 sec latency 32.947 ms 2 17192 9.00 MB/sec execute 10 sec latency 31.165 ms 2 18116 9.37 MB/sec execute 11 sec latency 17.881 ms 2 18651 8.71 MB/sec execute 12 sec latency 47.557 ms 2 19396 8.73 MB/sec execute 13 sec latency 22.926 ms 2 20187 8.37 MB/sec execute 14 sec latency 20.910 ms 2 21301 9.20 MB/sec execute 15 sec latency 27.145 ms 2 21939 8.84 MB/sec execute 16 sec latency 8.441 ms 2 22494 8.56 MB/sec execute 17 sec latency 43.474 ms 2 23228 8.46 MB/sec execute 18 sec latency 34.344 ms 2 24205 8.63 MB/sec execute 19 sec latency 26.359 ms 2 25147 8.95 MB/sec execute 20 sec latency 19.787 ms 2 25675 8.60 MB/sec execute 21 sec latency 54.488 ms 2 26382 8.63 MB/sec execute 22 sec latency 41.619 ms 2 27164 8.39 MB/sec execute 23 sec latency 29.512 ms 2 28145 8.68 MB/sec execute 24 sec latency 27.022 ms 2 28865 8.71 MB/sec execute 25 sec latency 11.669 ms 2 29406 8.45 MB/sec execute 26 sec latency 44.259 ms 2 30195 8.46 MB/sec execute 27 sec latency 28.192 ms 2 30978 8.35 MB/sec execute 28 sec latency 25.524 ms 2 32095 8.74 MB/sec execute 29 sec latency 26.366 ms 2 32685 8.55 MB/sec execute 30 sec latency 42.435 ms 2 33349 8.59 MB/sec execute 31 sec latency 38.724 ms 2 34036 8.36 MB/sec execute 32 sec latency 28.888 ms 2 34957 8.52 MB/sec execute 33 sec latency 26.941 ms 2 35871 8.66 MB/sec execute 34 sec latency 18.006 ms 2 36404 8.45 MB/sec execute 35 sec latency 55.856 ms 2 37148 8.47 MB/sec execute 36 sec latency 35.541 ms 2 37925 8.34 MB/sec execute 37 sec latency 25.733 ms 2 39059 8.67 MB/sec execute 38 sec latency 29.954 ms 2 39707 8.53 MB/sec execute 39 sec latency 9.152 ms 2 40235 8.42 MB/sec execute 40 sec latency 44.301 ms 2 40976 8.38 MB/sec execute 41 sec latency 29.682 ms 2 41945 8.48 MB/sec execute 42 sec latency 24.470 ms 2 42878 8.61 MB/sec execute 43 sec latency 25.130 ms 2 43399 8.45 MB/sec execute 44 sec latency 48.806 ms 2 44116 8.47 MB/sec execute 45 sec latency 26.895 ms 2 44926 8.36 MB/sec execute 46 sec latency 21.440 ms 2 45925 8.50 MB/sec execute 47 sec latency 32.473 ms 2 46626 8.52 MB/sec execute 48 sec latency 11.581 ms 2 47182 8.42 MB/sec execute 49 sec latency 59.102 ms 2 47952 8.40 MB/sec execute 50 sec latency 33.700 ms 2 48737 8.34 MB/sec execute 51 sec latency 26.630 ms 2 49857 8.55 MB/sec execute 52 sec latency 24.727 ms 2 50428 8.45 MB/sec execute 53 sec latency 46.318 ms 2 51135 8.47 MB/sec execute 54 sec latency 36.169 ms 2 51875 8.34 MB/sec execute 55 sec latency 28.753 ms 2 52892 8.46 MB/sec execute 56 sec latency 26.808 ms 2 53680 8.52 MB/sec execute 57 sec latency 13.107 ms 2 54231 8.40 MB/sec execute 58 sec latency 37.062 ms 2 55035 8.41 MB/sec execute 59 sec latency 26.894 ms 2 cleanup 60 sec 0 cleanup 60 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 16058 3.707 13.601 Close 11798 0.831 3.825 Rename 688 5.189 7.603 Unlink 3246 1.488 5.705 Qpathinfo 14576 0.957 6.762 Qfileinfo 2550 0.105 1.145 Qfsinfo 2680 0.064 1.059 Sfileinfo 1332 2.406 5.658 Find 5652 0.294 6.292 WriteX 7988 0.562 7.737 ReadX 25268 0.010 0.900 LockX 52 0.718 0.852 UnlockX 52 0.736 0.880 Flush 1126 15.728 59.091 Throughput 8.41112 MB/sec 2 clients 2 procs max_latency=59.102 ms stopping dbench in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:44:24 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2a.lustre-rsync-test dbench successfully finished Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 7 seconds Changelog records consumed: 22772 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0001: changelog user 'cl1' not found lustre-MDT0000: changelog user 'cl1' not found PASS 2a (86s) == lustre-rsync-test test 2b: Replicate files changed by dbench. ========================================================== 17:44:36 (1748036676) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl2 cl2' lustre-MDT0000: Registered changelog user cl2 waiting child_pid=$(pgrep dbench), 360 secs left ... looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2b.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:44:38 EDT 2025 waiting for dbench pid 10606 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs 1 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 667 42.32 MB/sec warmup 1 sec latency 58.336 ms 2 1271 23.14 MB/sec warmup 2 sec latency 20.300 ms 2 2123 17.78 MB/sec warmup 3 sec latency 25.516 ms 2 3177 17.06 MB/sec warmup 4 sec latency 23.636 ms 2 3995 16.02 MB/sec warmup 5 sec latency 24.565 ms 2 4549 13.67 MB/sec warmup 6 sec latency 52.120 ms 2 5361 12.98 MB/sec warmup 7 sec latency 33.068 ms 2 6176 12.05 MB/sec warmup 8 sec latency 30.217 ms 2 7345 13.08 MB/sec warmup 9 sec latency 23.284 ms 2 7904 11.94 MB/sec warmup 10 sec latency 42.999 ms 2 8667 11.70 MB/sec warmup 11 sec latency 31.385 ms 2 10530 17.03 MB/sec execute 1 sec latency 26.853 ms 2 11206 12.13 MB/sec execute 2 sec latency 8.142 ms PIDs: 10606 10607 10608 Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 4636 Resuming dbench 2 11459 4.76 MB/sec execute 5 sec latency 2799.907 ms 2 12205 5.47 MB/sec execute 6 sec latency 29.932 ms 2 12992 5.20 MB/sec execute 7 sec latency 25.351 ms 2 13990 6.39 MB/sec execute 8 sec latency 29.973 ms 2 14667 6.71 MB/sec execute 9 sec latency 8.794 ms 2 15188 6.25 MB/sec execute 10 sec latency 42.656 ms 2 15937 6.47 MB/sec execute 11 sec latency 34.378 ms 2 16686 6.34 MB/sec execute 12 sec latency 25.250 ms 2 17852 7.36 MB/sec execute 13 sec latency 28.421 ms 2 18433 7.07 MB/sec execute 14 sec latency 45.197 ms 2 19027 6.86 MB/sec execute 15 sec latency 25.243 ms Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 27408 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 3081 Wait for dbench to finish 2 19036 6.27 MB/sec execute 17 sec latency 1452.499 ms 2 19827 6.31 MB/sec execute 18 sec latency 1452.596 ms 2 20714 6.71 MB/sec execute 19 sec latency 28.297 ms 2 21633 7.00 MB/sec execute 20 sec latency 20.070 ms 2 22139 6.75 MB/sec execute 21 sec latency 61.105 ms 2 22895 6.86 MB/sec execute 22 sec latency 32.582 ms 2 23676 6.72 MB/sec execute 23 sec latency 31.561 ms 2 24665 7.07 MB/sec execute 24 sec latency 34.240 ms 2 25352 7.16 MB/sec execute 25 sec latency 8.376 ms 2 25907 7.02 MB/sec execute 26 sec latency 47.751 ms 2 26674 7.03 MB/sec execute 27 sec latency 20.617 ms 2 27440 6.96 MB/sec execute 28 sec latency 32.288 ms 2 28538 7.40 MB/sec execute 29 sec latency 26.865 ms 2 29126 7.26 MB/sec execute 30 sec latency 45.055 ms 2 29707 7.16 MB/sec execute 31 sec latency 37.453 ms 2 30502 7.14 MB/sec execute 32 sec latency 33.130 ms 2 31398 7.35 MB/sec execute 33 sec latency 25.430 ms 2 32286 7.50 MB/sec execute 34 sec latency 27.424 ms 2 32791 7.33 MB/sec execute 35 sec latency 56.491 ms 2 33540 7.39 MB/sec execute 36 sec latency 37.573 ms 2 34311 7.28 MB/sec execute 37 sec latency 26.318 ms 2 35319 7.49 MB/sec execute 38 sec latency 34.111 ms 2 36000 7.53 MB/sec execute 39 sec latency 8.331 ms 2 36503 7.39 MB/sec execute 40 sec latency 49.474 ms 2 37284 7.42 MB/sec execute 41 sec latency 25.560 ms 2 38061 7.37 MB/sec execute 42 sec latency 29.056 ms 2 39190 7.66 MB/sec execute 43 sec latency 21.356 ms 2 39754 7.56 MB/sec execute 44 sec latency 54.647 ms 2 40359 7.49 MB/sec execute 45 sec latency 20.921 ms 2 41103 7.46 MB/sec execute 46 sec latency 46.225 ms 2 41997 7.60 MB/sec execute 47 sec latency 29.525 ms 2 42921 7.70 MB/sec execute 48 sec latency 23.947 ms 2 43481 7.59 MB/sec execute 49 sec latency 43.674 ms 2 44212 7.61 MB/sec execute 50 sec latency 38.233 ms 2 44993 7.54 MB/sec execute 51 sec latency 22.841 ms 2 46109 7.79 MB/sec execute 52 sec latency 30.568 ms 2 46730 7.71 MB/sec execute 53 sec latency 8.269 ms 2 47330 7.64 MB/sec execute 54 sec latency 42.685 ms 2 48098 7.63 MB/sec execute 55 sec latency 33.376 ms 2 49060 7.74 MB/sec execute 56 sec latency 23.120 ms 2 50010 7.83 MB/sec execute 57 sec latency 24.191 ms 2 50587 7.73 MB/sec execute 58 sec latency 53.628 ms 2 51347 7.75 MB/sec execute 59 sec latency 32.014 ms 2 52164 7.70 MB/sec execute 60 sec latency 25.056 ms 2 cleanup 61 sec 0 cleanup 61 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 14941 3.948 1452.588 Close 10972 0.840 2.771 Rename 644 5.275 12.600 Unlink 3017 1.501 3.343 Qpathinfo 13553 0.965 9.272 Qfileinfo 2398 0.106 1.018 Qfsinfo 2520 0.063 0.971 Sfileinfo 1222 2.431 6.417 Find 5312 0.289 5.338 WriteX 7460 0.571 5.752 ReadX 23970 0.010 0.099 LockX 52 0.766 1.351 UnlockX 52 0.781 1.277 Flush 1056 21.190 2799.898 Throughput 7.70339 MB/sec 2 clients 2 procs max_latency=2799.907 ms stopping dbench in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:45:51 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2b.lustre-rsync-test dbench successfully finished Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 30488 Clear changelog after use: no Errors: 0 lustre_rsync took 9 seconds Changelog records consumed: 13797 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001: Deregistered changelog user #2 lustre-MDT0001: changelog user 'cl2' not found lustre-MDT0000: changelog user 'cl2' not found PASS 2b (89s) == lustre-rsync-test test complete, duration 188 sec ===== 17:46:05 (1748036765)