-----============= acceptance-small: lustre-rsync-test ============----- Fri May 23 17:42:46 EDT 2025 oleg302-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/lustre-rsync-test.*ex: No such file or directory excepting tests: 2b oleg302-client.virtnet: executing check_config_client /mnt/lustre oleg302-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg302-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012a562800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a562800.idle_timeout=debug disable quota as required oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 nobody:x:99: == lustre-rsync-test test 2a: Replicate files created by dbench. ========================================================== 17:42:58 (1748036578) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl1' lustre-MDT0000: Registered changelog user cl1 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2a.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:43:00 EDT 2025 waiting for dbench pid 8750 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs failed to create barrier semaphore 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 522 36.11 MB/sec warmup 1 sec latency 16.607 ms 2 1180 23.02 MB/sec warmup 2 sec latency 56.107 ms 2 2004 17.67 MB/sec warmup 3 sec latency 28.644 ms 2 2934 16.74 MB/sec warmup 4 sec latency 26.729 ms 2 3885 15.90 MB/sec warmup 5 sec latency 25.846 ms 2 4443 13.59 MB/sec warmup 6 sec latency 43.156 ms 2 5184 12.94 MB/sec warmup 7 sec latency 24.952 ms 2 5948 11.77 MB/sec warmup 8 sec latency 26.402 ms 2 7051 12.77 MB/sec warmup 9 sec latency 26.126 ms 2 7665 11.84 MB/sec warmup 10 sec latency 8.976 ms 2 8253 11.13 MB/sec warmup 11 sec latency 39.504 ms 2 9868 11.18 MB/sec execute 1 sec latency 26.577 ms 2 10865 13.39 MB/sec execute 2 sec latency 23.041 ms 2 11387 9.37 MB/sec execute 3 sec latency 51.458 ms 2 12085 9.38 MB/sec execute 4 sec latency 19.106 ms 2 12763 7.77 MB/sec execute 5 sec latency 29.063 ms 2 13639 8.76 MB/sec execute 6 sec latency 26.922 ms 2 14499 9.29 MB/sec execute 7 sec latency 27.661 ms 2 14997 8.33 MB/sec execute 8 sec latency 40.712 ms 2 15665 8.42 MB/sec execute 9 sec latency 41.341 ms 2 16306 7.71 MB/sec execute 10 sec latency 25.471 ms 2 17123 8.25 MB/sec execute 11 sec latency 36.313 ms 2 17978 8.60 MB/sec execute 12 sec latency 27.636 ms 2 18467 8.03 MB/sec execute 13 sec latency 47.451 ms 2 18964 7.72 MB/sec execute 14 sec latency 33.798 ms 2 19635 7.65 MB/sec execute 15 sec latency 26.906 ms 2 20275 7.47 MB/sec execute 16 sec latency 31.054 ms 2 21338 8.17 MB/sec execute 17 sec latency 28.893 ms 2 21839 7.90 MB/sec execute 18 sec latency 9.507 ms 2 22310 7.58 MB/sec execute 19 sec latency 41.662 ms 2 22995 7.64 MB/sec execute 20 sec latency 35.186 ms 2 23702 7.44 MB/sec execute 21 sec latency 27.119 ms 2 24598 7.77 MB/sec execute 22 sec latency 44.781 ms 2 25259 7.85 MB/sec execute 23 sec latency 14.018 ms 2 25708 7.56 MB/sec execute 24 sec latency 57.294 ms 2 26371 7.63 MB/sec execute 25 sec latency 35.556 ms 2 27030 7.38 MB/sec execute 26 sec latency 24.320 ms 2 27805 7.61 MB/sec execute 27 sec latency 33.303 ms 2 28663 7.79 MB/sec execute 28 sec latency 21.735 ms 2 29149 7.57 MB/sec execute 29 sec latency 40.784 ms 2 29652 7.43 MB/sec execute 30 sec latency 34.053 ms 2 30307 7.40 MB/sec execute 31 sec latency 34.794 ms 2 31011 7.34 MB/sec execute 32 sec latency 29.313 ms 2 32041 7.70 MB/sec execute 33 sec latency 35.892 ms 2 32558 7.56 MB/sec execute 34 sec latency 9.711 ms 2 33002 7.45 MB/sec execute 35 sec latency 39.156 ms 2 33681 7.43 MB/sec execute 36 sec latency 29.030 ms 2 34378 7.33 MB/sec execute 37 sec latency 26.416 ms 2 35305 7.52 MB/sec execute 38 sec latency 32.281 ms 2 35927 7.57 MB/sec execute 39 sec latency 13.438 ms 2 36382 7.41 MB/sec execute 40 sec latency 57.457 ms 2 37038 7.45 MB/sec execute 41 sec latency 25.154 ms 2 37679 7.30 MB/sec execute 42 sec latency 30.236 ms 2 38486 7.45 MB/sec execute 43 sec latency 28.305 ms 2 39311 7.56 MB/sec execute 44 sec latency 30.110 ms 2 39789 7.42 MB/sec execute 45 sec latency 46.107 ms 2 40305 7.34 MB/sec execute 46 sec latency 46.360 ms 2 40951 7.32 MB/sec execute 47 sec latency 31.212 ms 2 41659 7.28 MB/sec execute 48 sec latency 24.127 ms 2 42678 7.53 MB/sec execute 49 sec latency 33.327 ms 2 43196 7.43 MB/sec execute 50 sec latency 9.643 ms 2 43610 7.33 MB/sec execute 51 sec latency 45.459 ms 2 44294 7.35 MB/sec execute 52 sec latency 28.069 ms 2 44988 7.28 MB/sec execute 53 sec latency 31.441 ms 2 45925 7.42 MB/sec execute 54 sec latency 28.004 ms 2 46540 7.45 MB/sec execute 55 sec latency 13.549 ms 2 46979 7.34 MB/sec execute 56 sec latency 42.047 ms 2 47619 7.37 MB/sec execute 57 sec latency 28.978 ms 2 48234 7.26 MB/sec execute 58 sec latency 28.138 ms 2 49046 7.35 MB/sec execute 59 sec latency 28.248 ms 2 cleanup 60 sec 0 cleanup 60 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 13986 4.279 10.983 Close 10348 0.977 2.942 Rename 602 5.938 9.468 Unlink 2804 1.707 3.765 Qpathinfo 12766 1.091 7.532 Qfileinfo 2246 0.114 1.190 Qfsinfo 2366 0.073 1.008 Sfileinfo 1132 2.788 5.439 Find 4974 0.323 6.473 WriteX 7019 0.642 6.369 ReadX 22512 0.010 0.179 LockX 48 0.847 1.143 UnlockX 48 0.861 1.181 Flush 977 17.707 57.444 Throughput 7.34942 MB/sec 2 clients 2 procs max_latency=57.457 ms stopping dbench in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:44:13 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2a.lustre-rsync-test dbench successfully finished Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 8 seconds Changelog records consumed: 20248 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0000: changelog user 'cl1' not found PASS 2a (85s) == lustre-rsync-test test 2b: Replicate files changed by dbench. ========================================================== 17:44:23 (1748036663) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl2' lustre-MDT0000: Registered changelog user cl2 waiting child_pid=$(pgrep dbench), 360 secs left ... looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2b.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:44:24 EDT 2025 waiting for dbench pid 9636 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 654 42.13 MB/sec warmup 1 sec latency 13.467 ms 2 1147 22.99 MB/sec warmup 2 sec latency 57.215 ms 2 1840 17.57 MB/sec warmup 3 sec latency 36.766 ms 2 2562 14.45 MB/sec warmup 4 sec latency 25.340 ms 2 3636 15.45 MB/sec warmup 5 sec latency 31.950 ms 2 4226 13.39 MB/sec warmup 6 sec latency 9.310 ms 2 4706 12.03 MB/sec warmup 7 sec latency 43.843 ms 2 5393 11.36 MB/sec warmup 8 sec latency 27.728 ms 2 6145 10.68 MB/sec warmup 9 sec latency 22.128 ms 2 7241 11.57 MB/sec warmup 10 sec latency 25.600 ms 2 7790 10.80 MB/sec warmup 11 sec latency 47.540 ms 2 9121 6.87 MB/sec execute 1 sec latency 34.939 ms 2 10047 10.40 MB/sec execute 2 sec latency 26.672 ms PIDs: 9636 9637 9638 Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 3 seconds Changelog records consumed: 4154 Resuming dbench 2 10645 5.58 MB/sec execute 5 sec latency 2989.388 ms 2 11235 5.23 MB/sec execute 6 sec latency 2989.520 ms 2 11797 5.07 MB/sec execute 7 sec latency 58.483 ms 2 12455 5.26 MB/sec execute 8 sec latency 26.565 ms 2 13153 5.21 MB/sec execute 9 sec latency 26.980 ms 2 14266 6.60 MB/sec execute 10 sec latency 28.051 ms 2 14798 6.28 MB/sec execute 11 sec latency 9.456 ms 2 15255 6.09 MB/sec execute 12 sec latency 46.635 ms 2 15910 6.13 MB/sec execute 13 sec latency 29.281 ms 2 16596 5.95 MB/sec execute 14 sec latency 35.394 ms 2 17535 6.52 MB/sec execute 15 sec latency 27.757 ms Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 24402 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 2789 Wait for dbench to finish 2 17535 5.83 MB/sec execute 17 sec latency 1829.764 ms 2 18184 6.03 MB/sec execute 18 sec latency 1829.826 ms 2 18660 5.79 MB/sec execute 19 sec latency 48.480 ms 2 19328 5.94 MB/sec execute 20 sec latency 30.929 ms 2 19997 5.73 MB/sec execute 21 sec latency 22.446 ms 2 20799 6.08 MB/sec execute 22 sec latency 31.686 ms 2 21673 6.38 MB/sec execute 23 sec latency 21.462 ms 2 22139 6.16 MB/sec execute 24 sec latency 54.522 ms 2 22812 6.28 MB/sec execute 25 sec latency 35.596 ms 2 23462 6.09 MB/sec execute 26 sec latency 29.991 ms 2 24291 6.36 MB/sec execute 27 sec latency 31.999 ms 2 25137 6.58 MB/sec execute 28 sec latency 23.518 ms 2 25618 6.40 MB/sec execute 29 sec latency 50.291 ms 2 26155 6.31 MB/sec execute 30 sec latency 31.912 ms 2 26846 6.32 MB/sec execute 31 sec latency 30.363 ms 2 27627 6.47 MB/sec execute 32 sec latency 26.861 ms 2 28549 6.68 MB/sec execute 33 sec latency 27.792 ms 2 29093 6.57 MB/sec execute 34 sec latency 31.281 ms 2 29570 6.49 MB/sec execute 35 sec latency 50.078 ms 2 30235 6.50 MB/sec execute 36 sec latency 30.575 ms 2 30904 6.45 MB/sec execute 37 sec latency 25.125 ms 2 31963 6.79 MB/sec execute 38 sec latency 27.667 ms 2 32479 6.70 MB/sec execute 39 sec latency 9.089 ms 2 32954 6.58 MB/sec execute 40 sec latency 54.778 ms 2 33625 6.63 MB/sec execute 41 sec latency 30.515 ms 2 34324 6.55 MB/sec execute 42 sec latency 26.101 ms 2 35194 6.70 MB/sec execute 43 sec latency 26.240 ms 2 35883 6.81 MB/sec execute 44 sec latency 13.408 ms 2 36320 6.68 MB/sec execute 45 sec latency 53.928 ms 2 36962 6.73 MB/sec execute 46 sec latency 32.535 ms 2 37540 6.61 MB/sec execute 47 sec latency 29.961 ms 2 38380 6.71 MB/sec execute 48 sec latency 26.566 ms 2 39250 6.88 MB/sec execute 49 sec latency 27.163 ms 2 39739 6.76 MB/sec execute 50 sec latency 47.491 ms 2 40199 6.70 MB/sec execute 51 sec latency 37.327 ms 2 40847 6.70 MB/sec execute 52 sec latency 28.069 ms 2 41506 6.64 MB/sec execute 53 sec latency 29.372 ms 2 42433 6.79 MB/sec execute 54 sec latency 34.981 ms 2 43048 6.84 MB/sec execute 55 sec latency 10.920 ms 2 43510 6.74 MB/sec execute 56 sec latency 43.748 ms 2 44181 6.78 MB/sec execute 57 sec latency 23.578 ms 2 44835 6.68 MB/sec execute 58 sec latency 25.728 ms 2 45595 6.80 MB/sec execute 59 sec latency 34.435 ms 2 cleanup 60 sec 0 cleanup 61 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 13107 4.534 1829.820 Close 9666 0.972 2.321 Rename 552 5.881 8.966 Unlink 2598 1.706 3.005 Qpathinfo 11908 1.575 2989.510 Qfileinfo 2102 0.116 1.207 Qfsinfo 2168 0.075 1.543 Sfileinfo 1040 2.784 6.337 Find 4610 0.307 6.322 WriteX 6500 0.642 6.625 ReadX 20916 0.011 0.811 LockX 44 0.818 1.158 UnlockX 44 0.848 0.969 Flush 898 17.561 58.469 Throughput 6.79927 MB/sec 2 clients 2 procs max_latency=2989.520 ms stopping dbench in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:45:37 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2b.lustre-rsync-test dbench successfully finished Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 27190 Clear changelog after use: no Errors: 0 lustre_rsync took 5 seconds Changelog records consumed: 11938 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 lustre-MDT0000: changelog user 'cl2' not found PASS 2b (82s) == lustre-rsync-test test complete, duration 179 sec ===== 17:45:45 (1748036745)