-----============= acceptance-small: lustre-rsync-test ============----- Fri May 23 17:42:55 EDT 2025 oleg156-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/lustre-rsync-test.*ex: No such file or directory excepting tests: 2b oleg156-client.virtnet: executing check_config_client /mnt/lustre oleg156-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg156-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6cf5800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6cf5800.idle_timeout=debug disable quota as required oleg156-server: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 nobody:x:99: == lustre-rsync-test test 2a: Replicate files created by dbench. ========================================================== 17:43:05 (1748036585) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' lustre-MDT0000: Registered changelog user cl1 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2a.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:43:07 EDT 2025 waiting for dbench pid 9653 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs failed to create barrier semaphore 1 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 598 41.58 MB/sec warmup 1 sec latency 13.655 ms 2 1060 22.89 MB/sec warmup 2 sec latency 65.069 ms 2 1730 17.52 MB/sec warmup 3 sec latency 37.188 ms 2 2436 14.05 MB/sec warmup 4 sec latency 28.902 ms 2 3421 14.46 MB/sec warmup 5 sec latency 32.242 ms 2 4048 13.35 MB/sec warmup 6 sec latency 9.840 ms 2 4529 11.69 MB/sec warmup 7 sec latency 50.243 ms 2 5221 11.33 MB/sec warmup 8 sec latency 30.873 ms 2 5960 10.47 MB/sec warmup 9 sec latency 24.562 ms 2 6911 10.91 MB/sec warmup 10 sec latency 25.898 ms 2 7556 10.76 MB/sec warmup 11 sec latency 10.147 ms 2 8693 8.77 MB/sec execute 1 sec latency 51.911 ms 2 9338 5.02 MB/sec execute 2 sec latency 31.383 ms 2 10099 7.84 MB/sec execute 3 sec latency 34.902 ms 2 10936 9.00 MB/sec execute 4 sec latency 28.434 ms 2 11427 7.48 MB/sec execute 5 sec latency 42.710 ms 2 12043 7.78 MB/sec execute 6 sec latency 28.217 ms 2 12608 6.80 MB/sec execute 7 sec latency 31.060 ms 2 13293 6.75 MB/sec execute 8 sec latency 25.787 ms 2 14290 8.00 MB/sec execute 9 sec latency 31.895 ms 2 14798 7.49 MB/sec execute 10 sec latency 10.163 ms 2 15205 7.00 MB/sec execute 11 sec latency 50.589 ms 2 15891 7.12 MB/sec execute 12 sec latency 29.178 ms 2 16584 6.84 MB/sec execute 13 sec latency 32.384 ms 2 17493 7.40 MB/sec execute 14 sec latency 27.975 ms 2 18132 7.54 MB/sec execute 15 sec latency 18.768 ms 2 18614 7.15 MB/sec execute 16 sec latency 52.626 ms 2 19253 7.25 MB/sec execute 17 sec latency 28.316 ms 2 19898 6.92 MB/sec execute 18 sec latency 27.271 ms 2 20682 7.21 MB/sec execute 19 sec latency 29.609 ms 2 21529 7.54 MB/sec execute 20 sec latency 30.489 ms 2 22001 7.23 MB/sec execute 21 sec latency 54.077 ms 2 22450 7.07 MB/sec execute 22 sec latency 70.502 ms 2 23061 7.05 MB/sec execute 23 sec latency 38.691 ms 2 23738 6.90 MB/sec execute 24 sec latency 30.077 ms 2 24674 7.21 MB/sec execute 25 sec latency 32.038 ms 2 25289 7.29 MB/sec execute 26 sec latency 10.242 ms 2 25758 7.08 MB/sec execute 27 sec latency 57.822 ms 2 26386 7.14 MB/sec execute 28 sec latency 33.442 ms 2 27018 6.94 MB/sec execute 29 sec latency 24.919 ms 2 27793 7.16 MB/sec execute 30 sec latency 32.763 ms 2 28660 7.33 MB/sec execute 31 sec latency 23.676 ms 2 29127 7.14 MB/sec execute 32 sec latency 51.870 ms 2 29618 7.03 MB/sec execute 33 sec latency 32.714 ms 2 30243 7.02 MB/sec execute 34 sec latency 33.060 ms 2 30873 6.92 MB/sec execute 35 sec latency 25.993 ms 2 31794 7.14 MB/sec execute 36 sec latency 35.057 ms 2 32389 7.19 MB/sec execute 37 sec latency 10.905 ms 2 32823 7.04 MB/sec execute 38 sec latency 60.323 ms 2 33430 7.08 MB/sec execute 39 sec latency 31.909 ms 2 33950 6.93 MB/sec execute 40 sec latency 37.956 ms 2 34548 6.88 MB/sec execute 41 sec latency 31.701 ms 2 35516 7.17 MB/sec execute 42 sec latency 40.385 ms 2 35980 7.08 MB/sec execute 43 sec latency 10.773 ms 2 36421 6.95 MB/sec execute 44 sec latency 51.730 ms 2 37057 6.99 MB/sec execute 45 sec latency 26.564 ms 2 37649 6.87 MB/sec execute 46 sec latency 29.658 ms 2 38417 7.00 MB/sec execute 47 sec latency 28.749 ms 2 39268 7.13 MB/sec execute 48 sec latency 32.634 ms 2 39734 7.00 MB/sec execute 49 sec latency 56.898 ms 2 40181 6.93 MB/sec execute 50 sec latency 40.432 ms 2 40790 6.93 MB/sec execute 51 sec latency 32.861 ms 2 41466 6.86 MB/sec execute 52 sec latency 34.747 ms 2 42396 7.01 MB/sec execute 53 sec latency 30.355 ms 2 43029 7.05 MB/sec execute 54 sec latency 9.771 ms 2 43509 6.95 MB/sec execute 55 sec latency 52.084 ms 2 44177 6.99 MB/sec execute 56 sec latency 30.295 ms 2 44898 6.92 MB/sec execute 57 sec latency 26.493 ms 2 45781 7.02 MB/sec execute 58 sec latency 33.086 ms 2 46510 7.10 MB/sec execute 59 sec latency 22.535 ms 2 cleanup 60 sec 0 cleanup 60 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 13486 4.457 11.850 Close 9902 1.014 3.351 Rename 570 6.182 10.143 Unlink 2724 1.776 9.320 Qpathinfo 12238 1.121 8.140 Qfileinfo 2130 0.120 1.211 Qfsinfo 2242 0.074 1.137 Sfileinfo 1095 2.902 5.857 Find 4722 0.345 7.104 WriteX 6647 0.661 5.982 ReadX 21178 0.011 1.235 LockX 44 0.881 1.147 UnlockX 44 0.910 1.182 Flush 942 18.436 70.494 Throughput 7.10299 MB/sec 2 clients 2 procs max_latency=70.502 ms stopping dbench in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:44:19 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2a.lustre-rsync-test dbench successfully finished Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 7 seconds Changelog records consumed: 19146 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0001: changelog user 'cl1' not found lustre-MDT0000: changelog user 'cl1' not found PASS 2a (85s) == lustre-rsync-test test 2b: Replicate files changed by dbench. ========================================================== 17:44:30 (1748036670) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl2 cl2' lustre-MDT0000: Registered changelog user cl2 waiting child_pid=$(pgrep dbench), 360 secs left ... looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2b.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:44:33 EDT 2025 waiting for dbench pid 10623 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 541 38.20 MB/sec warmup 1 sec latency 14.097 ms 2 985 21.90 MB/sec warmup 2 sec latency 76.067 ms 2 1660 17.47 MB/sec warmup 3 sec latency 32.062 ms 2 2374 13.95 MB/sec warmup 4 sec latency 28.977 ms 2 3277 14.14 MB/sec warmup 5 sec latency 30.041 ms 2 3963 13.35 MB/sec warmup 6 sec latency 16.901 ms 2 4443 11.65 MB/sec warmup 7 sec latency 53.146 ms 2 5069 11.29 MB/sec warmup 8 sec latency 44.703 ms 2 5718 10.18 MB/sec warmup 9 sec latency 28.406 ms 2 6535 10.53 MB/sec warmup 10 sec latency 30.760 ms 2 7381 10.71 MB/sec warmup 11 sec latency 22.060 ms 2 8347 3.54 MB/sec execute 1 sec latency 61.412 ms 2 8982 5.05 MB/sec execute 2 sec latency 32.672 ms PIDs: 10623 10624 10625 Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 3 seconds Changelog records consumed: 3786 Resuming dbench 2 9338 2.05 MB/sec execute 5 sec latency 2844.536 ms 2 10123 3.86 MB/sec execute 6 sec latency 2844.548 ms 2 10975 5.03 MB/sec execute 7 sec latency 28.574 ms 2 11449 4.62 MB/sec execute 8 sec latency 50.620 ms 2 12045 5.10 MB/sec execute 9 sec latency 42.102 ms 2 12600 4.69 MB/sec execute 10 sec latency 37.587 ms 2 13254 4.73 MB/sec execute 11 sec latency 28.316 ms 2 14269 5.90 MB/sec execute 12 sec latency 31.161 ms 2 14715 5.68 MB/sec execute 13 sec latency 11.398 ms 2 15170 5.41 MB/sec execute 14 sec latency 50.216 ms 2 15806 5.62 MB/sec execute 15 sec latency 33.364 ms Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 22932 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 2597 Wait for dbench to finish 2 15809 5.07 MB/sec execute 17 sec latency 1654.417 ms 2 16541 4.98 MB/sec execute 18 sec latency 1654.459 ms 2 17382 5.39 MB/sec execute 19 sec latency 32.465 ms 2 18065 5.67 MB/sec execute 20 sec latency 22.135 ms 2 18533 5.47 MB/sec execute 21 sec latency 57.290 ms 2 19027 5.38 MB/sec execute 22 sec latency 36.389 ms 2 19639 5.43 MB/sec execute 23 sec latency 41.824 ms 2 20339 5.42 MB/sec execute 24 sec latency 27.447 ms 2 21336 5.96 MB/sec execute 25 sec latency 34.348 ms 2 21825 5.86 MB/sec execute 26 sec latency 11.296 ms 2 22292 5.71 MB/sec execute 27 sec latency 39.240 ms 2 22949 5.81 MB/sec execute 28 sec latency 31.418 ms 2 23654 5.73 MB/sec execute 29 sec latency 32.807 ms 2 24512 5.97 MB/sec execute 30 sec latency 26.213 ms 2 25194 6.13 MB/sec execute 31 sec latency 28.582 ms 2 25675 5.99 MB/sec execute 32 sec latency 40.569 ms 2 26282 6.08 MB/sec execute 33 sec latency 38.889 ms 2 26877 5.93 MB/sec execute 34 sec latency 31.686 ms 2 27608 6.07 MB/sec execute 35 sec latency 40.826 ms 2 28524 6.28 MB/sec execute 36 sec latency 26.124 ms 2 29006 6.19 MB/sec execute 37 sec latency 11.121 ms 2 29413 6.07 MB/sec execute 38 sec latency 58.627 ms 2 30070 6.14 MB/sec execute 39 sec latency 34.463 ms 2 30764 6.07 MB/sec execute 40 sec latency 26.453 ms 2 31587 6.22 MB/sec execute 41 sec latency 34.612 ms 2 32266 6.34 MB/sec execute 42 sec latency 23.312 ms 2 32712 6.23 MB/sec execute 43 sec latency 59.211 ms 2 33207 6.17 MB/sec execute 44 sec latency 39.383 ms 2 33847 6.18 MB/sec execute 45 sec latency 32.803 ms 2 34462 6.14 MB/sec execute 46 sec latency 33.800 ms 2 35509 6.43 MB/sec execute 47 sec latency 26.017 ms 2 36000 6.36 MB/sec execute 48 sec latency 10.945 ms 2 36471 6.27 MB/sec execute 49 sec latency 53.466 ms 2 37135 6.32 MB/sec execute 50 sec latency 36.913 ms 2 37854 6.26 MB/sec execute 51 sec latency 26.249 ms 2 38704 6.38 MB/sec execute 52 sec latency 30.048 ms 2 39427 6.48 MB/sec execute 53 sec latency 22.591 ms 2 39914 6.39 MB/sec execute 54 sec latency 51.955 ms 2 40562 6.43 MB/sec execute 55 sec latency 36.514 ms 2 41251 6.34 MB/sec execute 56 sec latency 23.269 ms 2 42074 6.47 MB/sec execute 57 sec latency 29.030 ms 2 42947 6.58 MB/sec execute 58 sec latency 26.291 ms 2 43427 6.49 MB/sec execute 59 sec latency 52.544 ms 2 44142 6.54 MB/sec execute 60 sec latency 39.871 ms 2 cleanup 61 sec 0 cleanup 61 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 12887 4.388 16.427 Close 9543 1.944 2844.542 Rename 540 6.100 9.237 Unlink 2538 1.766 9.750 Qpathinfo 11704 1.097 12.019 Qfileinfo 2036 0.118 1.336 Qfsinfo 2096 0.075 1.364 Sfileinfo 1059 2.884 5.939 Find 4482 0.326 6.473 WriteX 6352 0.653 18.451 ReadX 19964 0.011 1.205 LockX 40 0.826 1.007 UnlockX 40 0.912 1.094 Flush 897 18.359 59.198 Throughput 6.53508 MB/sec 2 clients 2 procs max_latency=2844.548 ms stopping dbench in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:45:46 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2b.lustre-rsync-test dbench successfully finished Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 25528 Clear changelog after use: no Errors: 0 lustre_rsync took 4 seconds Changelog records consumed: 11963 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001: Deregistered changelog user #2 lustre-MDT0001: changelog user 'cl2' not found lustre-MDT0000: changelog user 'cl2' not found PASS 2b (84s) == lustre-rsync-test test complete, duration 179 sec ===== 17:45:54 (1748036754)