-----============= acceptance-small: lustre-rsync-test ============----- Fri May 23 17:43:00 EDT 2025 oleg301-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/lustre-rsync-test.*ex: No such file or directory excepting tests: 2b oleg301-client.virtnet: executing check_config_client /mnt/lustre oleg301-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg301-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6cca000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6cca000.idle_timeout=debug disable quota as required oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 nobody:x:99: == lustre-rsync-test test 2a: Replicate files created by dbench. ========================================================== 17:43:09 (1748036589) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' lustre-MDT0000: Registered changelog user cl1 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2a.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:43:12 EDT 2025 waiting for dbench pid 9660 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs failed to create barrier semaphore 1 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 664 42.31 MB/sec warmup 1 sec latency 17.395 ms 2 1190 23.04 MB/sec warmup 2 sec latency 64.096 ms 2 1913 17.63 MB/sec warmup 3 sec latency 35.938 ms 2 2768 15.98 MB/sec warmup 4 sec latency 28.221 ms 2 3780 15.90 MB/sec warmup 5 sec latency 32.095 ms 2 4285 13.45 MB/sec warmup 6 sec latency 50.887 ms 2 4808 12.06 MB/sec warmup 7 sec latency 45.346 ms 2 5453 11.37 MB/sec warmup 8 sec latency 30.667 ms 2 6179 10.74 MB/sec warmup 9 sec latency 31.751 ms 2 7212 11.57 MB/sec warmup 10 sec latency 34.087 ms 2 7709 10.76 MB/sec warmup 11 sec latency 10.894 ms 2 8751 8.58 MB/sec execute 1 sec latency 50.383 ms 2 9449 5.98 MB/sec execute 2 sec latency 32.052 ms 2 10277 8.14 MB/sec execute 3 sec latency 32.345 ms 2 10968 8.93 MB/sec execute 4 sec latency 23.194 ms 2 11395 7.41 MB/sec execute 5 sec latency 47.510 ms 2 11900 6.76 MB/sec execute 6 sec latency 28.804 ms 2 12517 6.73 MB/sec execute 7 sec latency 31.662 ms 2 13144 6.48 MB/sec execute 8 sec latency 33.345 ms 2 14180 7.91 MB/sec execute 9 sec latency 31.098 ms 2 14675 7.46 MB/sec execute 10 sec latency 10.003 ms 2 15119 6.93 MB/sec execute 11 sec latency 63.791 ms 2 15773 7.08 MB/sec execute 12 sec latency 34.501 ms 2 16407 6.63 MB/sec execute 13 sec latency 27.347 ms 2 17185 7.12 MB/sec execute 14 sec latency 36.005 ms 2 18003 7.48 MB/sec execute 15 sec latency 26.602 ms 2 18481 7.09 MB/sec execute 16 sec latency 52.103 ms 2 18964 6.88 MB/sec execute 17 sec latency 40.536 ms 2 19612 6.87 MB/sec execute 18 sec latency 28.020 ms 2 20262 6.76 MB/sec execute 19 sec latency 29.570 ms 2 21271 7.38 MB/sec execute 20 sec latency 28.845 ms 2 21791 7.20 MB/sec execute 21 sec latency 12.250 ms 2 22248 6.94 MB/sec execute 22 sec latency 64.474 ms 2 22900 7.02 MB/sec execute 23 sec latency 33.847 ms 2 23528 6.78 MB/sec execute 24 sec latency 28.462 ms 2 24305 7.05 MB/sec execute 25 sec latency 36.755 ms 2 25119 7.26 MB/sec execute 26 sec latency 23.027 ms 2 25595 7.03 MB/sec execute 27 sec latency 57.061 ms 2 26036 6.91 MB/sec execute 28 sec latency 40.870 ms 2 26670 6.90 MB/sec execute 29 sec latency 31.134 ms 2 27312 6.79 MB/sec execute 30 sec latency 28.971 ms 2 28227 7.04 MB/sec execute 31 sec latency 35.992 ms 2 28834 7.11 MB/sec execute 32 sec latency 10.021 ms 2 29275 6.94 MB/sec execute 33 sec latency 49.938 ms 2 29905 6.99 MB/sec execute 34 sec latency 22.131 ms 2 30484 6.83 MB/sec execute 35 sec latency 35.651 ms 2 31288 6.95 MB/sec execute 36 sec latency 35.818 ms 2 32103 7.11 MB/sec execute 37 sec latency 25.913 ms 2 32631 7.00 MB/sec execute 38 sec latency 10.367 ms 2 33037 6.92 MB/sec execute 39 sec latency 81.163 ms 2 33683 6.91 MB/sec execute 40 sec latency 32.190 ms 2 34355 6.83 MB/sec execute 41 sec latency 26.859 ms 2 35238 7.02 MB/sec execute 42 sec latency 29.507 ms 2 35883 7.07 MB/sec execute 43 sec latency 15.978 ms 2 36310 6.93 MB/sec execute 44 sec latency 58.655 ms 2 36916 6.98 MB/sec execute 45 sec latency 41.570 ms 2 37479 6.85 MB/sec execute 46 sec latency 32.598 ms 2 38104 6.81 MB/sec execute 47 sec latency 27.619 ms 2 39118 7.07 MB/sec execute 48 sec latency 29.524 ms 2 39634 6.99 MB/sec execute 49 sec latency 9.738 ms 2 40074 6.90 MB/sec execute 50 sec latency 46.745 ms 2 40775 6.92 MB/sec execute 51 sec latency 33.761 ms 2 41475 6.86 MB/sec execute 52 sec latency 36.611 ms 2 42425 7.01 MB/sec execute 53 sec latency 36.317 ms 2 43066 7.05 MB/sec execute 54 sec latency 9.067 ms 2 43562 6.95 MB/sec execute 55 sec latency 55.659 ms 2 44262 6.98 MB/sec execute 56 sec latency 27.619 ms 2 45004 6.92 MB/sec execute 57 sec latency 33.714 ms 2 46105 7.16 MB/sec execute 58 sec latency 22.080 ms 2 46658 7.10 MB/sec execute 59 sec latency 9.113 ms 2 cleanup 60 sec 0 cleanup 60 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 13512 4.431 16.567 Close 9927 1.004 7.774 Rename 572 6.142 9.323 Unlink 2729 1.760 6.900 Qpathinfo 12249 1.112 9.408 Qfileinfo 2134 0.121 1.219 Qfsinfo 2244 0.078 1.202 Sfileinfo 1099 2.912 7.981 Find 4730 0.345 6.647 WriteX 6668 0.669 5.880 ReadX 21191 0.012 0.245 LockX 44 0.852 1.044 UnlockX 44 0.890 1.317 Flush 946 18.717 81.145 Throughput 7.09849 MB/sec 2 clients 2 procs max_latency=81.163 ms stopping dbench in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:44:24 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2a.lustre-rsync-test dbench successfully finished Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 8 seconds Changelog records consumed: 19225 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0001: changelog user 'cl1' not found lustre-MDT0000: changelog user 'cl1' not found PASS 2a (86s) == lustre-rsync-test test 2b: Replicate files changed by dbench. ========================================================== 17:44:35 (1748036675) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl2 cl2' lustre-MDT0000: Registered changelog user cl2 waiting child_pid=$(pgrep dbench), 360 secs left ... looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2b.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:44:38 EDT 2025 waiting for dbench pid 10629 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 664 42.31 MB/sec warmup 1 sec latency 12.339 ms 2 1181 23.01 MB/sec warmup 2 sec latency 62.362 ms 2 1965 17.63 MB/sec warmup 3 sec latency 25.591 ms 2 2889 16.26 MB/sec warmup 4 sec latency 30.550 ms 2 3805 15.90 MB/sec warmup 5 sec latency 29.616 ms 2 4304 13.48 MB/sec warmup 6 sec latency 45.283 ms 2 4834 12.07 MB/sec warmup 7 sec latency 39.404 ms 2 5547 11.39 MB/sec warmup 8 sec latency 25.963 ms 2 6324 11.35 MB/sec warmup 9 sec latency 31.143 ms 2 7253 11.58 MB/sec warmup 10 sec latency 27.410 ms 2 7778 10.80 MB/sec warmup 11 sec latency 40.482 ms 2 8835 6.67 MB/sec execute 1 sec latency 57.379 ms 2 9520 5.16 MB/sec execute 2 sec latency 28.094 ms PIDs: 10629 10630 10631 Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 3 seconds Changelog records consumed: 3991 Resuming dbench 2 10000 3.34 MB/sec execute 6 sec latency 3428.093 ms 2 10864 4.86 MB/sec execute 7 sec latency 3428.055 ms 2 11370 4.39 MB/sec execute 8 sec latency 44.982 ms 2 11854 4.31 MB/sec execute 9 sec latency 37.968 ms 2 12517 4.54 MB/sec execute 10 sec latency 35.655 ms 2 13150 4.56 MB/sec execute 11 sec latency 27.173 ms 2 14207 5.81 MB/sec execute 12 sec latency 27.613 ms 2 14702 5.61 MB/sec execute 13 sec latency 11.149 ms 2 15182 5.34 MB/sec execute 14 sec latency 47.842 ms 2 15852 5.56 MB/sec execute 15 sec latency 38.947 ms 2 16542 5.43 MB/sec execute 16 sec latency 26.117 ms Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 23216 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 2712 Wait for dbench to finish 2 16543 4.91 MB/sec execute 18 sec latency 1678.353 ms 2 17389 5.33 MB/sec execute 19 sec latency 1678.609 ms 2 18081 5.65 MB/sec execute 20 sec latency 21.653 ms 2 18549 5.43 MB/sec execute 21 sec latency 60.305 ms 2 19194 5.60 MB/sec execute 22 sec latency 37.618 ms 2 19757 5.39 MB/sec execute 23 sec latency 38.022 ms 2 20391 5.38 MB/sec execute 24 sec latency 29.114 ms 2 21420 5.94 MB/sec execute 25 sec latency 30.767 ms 2 21919 5.83 MB/sec execute 26 sec latency 12.453 ms 2 22329 5.68 MB/sec execute 27 sec latency 56.139 ms 2 22990 5.78 MB/sec execute 28 sec latency 34.820 ms 2 23673 5.70 MB/sec execute 29 sec latency 28.048 ms 2 24514 5.94 MB/sec execute 30 sec latency 31.123 ms 2 25189 6.10 MB/sec execute 31 sec latency 23.216 ms 2 25619 5.95 MB/sec execute 32 sec latency 52.151 ms 2 26115 5.88 MB/sec execute 33 sec latency 26.109 ms 2 26747 5.90 MB/sec execute 34 sec latency 27.996 ms 2 27364 5.86 MB/sec execute 35 sec latency 30.511 ms 2 28396 6.24 MB/sec execute 36 sec latency 29.112 ms 2 28881 6.16 MB/sec execute 37 sec latency 9.973 ms 2 29324 6.04 MB/sec execute 38 sec latency 69.508 ms 2 29947 6.11 MB/sec execute 39 sec latency 36.743 ms 2 30576 5.99 MB/sec execute 40 sec latency 31.346 ms 2 31345 6.17 MB/sec execute 41 sec latency 32.200 ms 2 32198 6.33 MB/sec execute 42 sec latency 29.707 ms 2 32650 6.20 MB/sec execute 43 sec latency 54.268 ms 2 33100 6.14 MB/sec execute 44 sec latency 40.061 ms 2 33741 6.15 MB/sec execute 45 sec latency 31.670 ms 2 34382 6.10 MB/sec execute 46 sec latency 36.156 ms 2 35305 6.28 MB/sec execute 47 sec latency 32.428 ms 2 35923 6.35 MB/sec execute 48 sec latency 18.652 ms 2 36391 6.24 MB/sec execute 49 sec latency 45.976 ms 2 37063 6.30 MB/sec execute 50 sec latency 33.014 ms 2 37714 6.20 MB/sec execute 51 sec latency 26.189 ms 2 38517 6.34 MB/sec execute 52 sec latency 32.337 ms 2 39403 6.47 MB/sec execute 53 sec latency 26.703 ms 2 39929 6.37 MB/sec execute 54 sec latency 53.133 ms 2 40595 6.42 MB/sec execute 55 sec latency 33.947 ms 2 41364 6.37 MB/sec execute 56 sec latency 24.430 ms 2 42387 6.52 MB/sec execute 57 sec latency 29.310 ms 2 43064 6.57 MB/sec execute 58 sec latency 12.388 ms 2 43574 6.49 MB/sec execute 59 sec latency 47.202 ms 2 44320 6.53 MB/sec execute 60 sec latency 33.458 ms 2 cleanup 61 sec 0 cleanup 61 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 12816 5.126 3428.065 Close 9470 0.979 2.913 Rename 540 5.943 9.583 Unlink 2530 1.749 3.766 Qpathinfo 11676 1.080 8.385 Qfileinfo 2044 0.119 1.244 Qfsinfo 2076 0.075 1.113 Sfileinfo 1050 2.833 13.620 Find 4462 0.324 6.597 WriteX 6318 0.653 6.789 ReadX 19828 0.012 0.894 LockX 40 0.840 1.155 UnlockX 40 0.882 1.159 Flush 880 18.518 69.491 Throughput 6.52858 MB/sec 2 clients 2 procs max_latency=3428.093 ms stopping dbench in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:45:50 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2b.lustre-rsync-test dbench successfully finished Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 25927 Clear changelog after use: no Errors: 0 lustre_rsync took 4 seconds Changelog records consumed: 11717 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001: Deregistered changelog user #2 lustre-MDT0001: changelog user 'cl2' not found lustre-MDT0000: changelog user 'cl2' not found PASS 2b (84s) == lustre-rsync-test test complete, duration 179 sec ===== 17:45:59 (1748036759)