-----============= acceptance-small: lustre-rsync-test ============----- Fri May 23 17:42:47 EDT 2025 oleg336-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/lustre-rsync-test.*ex: No such file or directory excepting tests: 2b oleg336-client.virtnet: executing check_config_client /mnt/lustre oleg336-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg336-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012ab05800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012ab05800.idle_timeout=debug disable quota as required oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 nobody:x:99: == lustre-rsync-test test 2a: Replicate files created by dbench. ========================================================== 17:42:59 (1748036579) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl1' lustre-MDT0000: Registered changelog user cl1 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2a.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:43:01 EDT 2025 waiting for dbench pid 8765 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs failed to create barrier semaphore 1 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 664 42.31 MB/sec warmup 1 sec latency 12.473 ms 2 1200 23.05 MB/sec warmup 2 sec latency 61.559 ms 2 1986 17.64 MB/sec warmup 3 sec latency 34.147 ms 2 2919 16.73 MB/sec warmup 4 sec latency 27.573 ms 2 3846 15.90 MB/sec warmup 5 sec latency 29.146 ms 2 4367 13.53 MB/sec warmup 6 sec latency 41.919 ms 2 5070 12.90 MB/sec warmup 7 sec latency 37.132 ms 2 5832 11.55 MB/sec warmup 8 sec latency 24.075 ms 2 6829 12.10 MB/sec warmup 9 sec latency 23.409 ms 2 7546 11.84 MB/sec warmup 10 sec latency 11.635 ms 2 8070 10.92 MB/sec warmup 11 sec latency 39.015 ms 2 9549 3.77 MB/sec execute 1 sec latency 26.630 ms 2 10667 12.21 MB/sec execute 2 sec latency 27.444 ms 2 11260 9.24 MB/sec execute 3 sec latency 10.425 ms 2 11748 7.91 MB/sec execute 4 sec latency 41.240 ms 2 12446 7.68 MB/sec execute 5 sec latency 24.048 ms 2 13131 7.20 MB/sec execute 6 sec latency 32.575 ms 2 14176 8.93 MB/sec execute 7 sec latency 28.476 ms 2 14691 8.24 MB/sec execute 8 sec latency 9.746 ms 2 15174 7.52 MB/sec execute 9 sec latency 53.037 ms 2 15833 7.64 MB/sec execute 10 sec latency 35.326 ms 2 16491 7.23 MB/sec execute 11 sec latency 25.495 ms 2 17204 7.59 MB/sec execute 12 sec latency 30.738 ms 2 18043 7.97 MB/sec execute 13 sec latency 20.808 ms 2 18489 7.49 MB/sec execute 14 sec latency 56.733 ms 2 19001 7.23 MB/sec execute 15 sec latency 41.991 ms 2 19635 7.19 MB/sec execute 16 sec latency 31.494 ms 2 20262 7.05 MB/sec execute 17 sec latency 33.069 ms 2 21284 7.73 MB/sec execute 18 sec latency 27.322 ms 2 21786 7.50 MB/sec execute 19 sec latency 10.292 ms 2 22238 7.20 MB/sec execute 20 sec latency 46.831 ms 2 22884 7.28 MB/sec execute 21 sec latency 38.071 ms 2 23487 7.00 MB/sec execute 22 sec latency 32.917 ms 2 24301 7.29 MB/sec execute 23 sec latency 29.165 ms 2 25131 7.51 MB/sec execute 24 sec latency 22.772 ms 2 25608 7.26 MB/sec execute 25 sec latency 39.612 ms 2 26110 7.12 MB/sec execute 26 sec latency 25.818 ms 2 26767 7.10 MB/sec execute 27 sec latency 34.312 ms 2 27456 7.03 MB/sec execute 28 sec latency 27.163 ms 2 28480 7.45 MB/sec execute 29 sec latency 36.382 ms 2 28988 7.30 MB/sec execute 30 sec latency 9.575 ms 2 29416 7.13 MB/sec execute 31 sec latency 42.101 ms 2 30112 7.17 MB/sec execute 32 sec latency 38.630 ms 2 30801 7.06 MB/sec execute 33 sec latency 28.647 ms 2 31712 7.29 MB/sec execute 34 sec latency 33.574 ms 2 32354 7.35 MB/sec execute 35 sec latency 14.103 ms 2 32793 7.17 MB/sec execute 36 sec latency 43.093 ms 2 33457 7.23 MB/sec execute 37 sec latency 28.288 ms 2 34063 7.07 MB/sec execute 38 sec latency 28.488 ms 2 34864 7.20 MB/sec execute 39 sec latency 28.327 ms 2 35720 7.37 MB/sec execute 40 sec latency 29.046 ms 2 36194 7.22 MB/sec execute 41 sec latency 60.746 ms 2 36679 7.13 MB/sec execute 42 sec latency 23.364 ms 2 37344 7.12 MB/sec execute 43 sec latency 25.916 ms 2 37982 7.06 MB/sec execute 44 sec latency 30.648 ms 2 39027 7.34 MB/sec execute 45 sec latency 27.259 ms 2 39524 7.25 MB/sec execute 46 sec latency 10.388 ms 2 39982 7.13 MB/sec execute 47 sec latency 52.567 ms 2 40665 7.17 MB/sec execute 48 sec latency 27.519 ms 2 41327 7.06 MB/sec execute 49 sec latency 30.087 ms 2 42073 7.18 MB/sec execute 50 sec latency 31.840 ms 2 42902 7.28 MB/sec execute 51 sec latency 33.358 ms 2 43371 7.17 MB/sec execute 52 sec latency 42.885 ms 2 43922 7.14 MB/sec execute 53 sec latency 25.618 ms 2 44576 7.09 MB/sec execute 54 sec latency 26.843 ms 2 45206 7.06 MB/sec execute 55 sec latency 34.744 ms 2 46238 7.27 MB/sec execute 56 sec latency 29.194 ms 2 46742 7.20 MB/sec execute 57 sec latency 9.797 ms 2 47175 7.13 MB/sec execute 58 sec latency 62.877 ms 2 47858 7.13 MB/sec execute 59 sec latency 27.689 ms 2 cleanup 60 sec 0 cleanup 60 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 13793 4.362 11.880 Close 10170 0.996 3.741 Rename 576 6.096 9.567 Unlink 2750 1.728 2.876 Qpathinfo 12566 1.098 7.666 Qfileinfo 2187 0.117 1.358 Qfsinfo 2256 0.075 1.195 Sfileinfo 1114 2.860 6.067 Find 4822 0.335 6.375 WriteX 6792 0.652 6.295 ReadX 21439 0.011 1.618 LockX 44 0.877 1.182 UnlockX 44 0.861 1.128 Flush 956 17.981 62.864 Throughput 7.13088 MB/sec 2 clients 2 procs max_latency=62.877 ms stopping dbench in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:44:14 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2a.lustre-rsync-test dbench successfully finished Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 7 seconds Changelog records consumed: 19814 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0000: changelog user 'cl1' not found PASS 2a (85s) == lustre-rsync-test test 2b: Replicate files changed by dbench. ========================================================== 17:44:24 (1748036664) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl2' lustre-MDT0000: Registered changelog user cl2 waiting child_pid=$(pgrep dbench), 360 secs left ... looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2b.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:44:25 EDT 2025 waiting for dbench pid 9647 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 608 41.94 MB/sec warmup 1 sec latency 14.055 ms 2 1082 22.93 MB/sec warmup 2 sec latency 51.780 ms 2 1787 17.54 MB/sec warmup 3 sec latency 23.835 ms 2 2476 14.35 MB/sec warmup 4 sec latency 34.529 ms 2 3562 15.37 MB/sec warmup 5 sec latency 29.276 ms 2 4153 13.36 MB/sec warmup 6 sec latency 9.271 ms 2 4635 12.00 MB/sec warmup 7 sec latency 41.147 ms 2 5360 11.36 MB/sec warmup 8 sec latency 23.347 ms 2 6112 10.67 MB/sec warmup 9 sec latency 32.368 ms 2 7199 11.55 MB/sec warmup 10 sec latency 23.057 ms 2 7778 10.80 MB/sec warmup 11 sec latency 44.566 ms 2 9013 6.76 MB/sec execute 1 sec latency 50.155 ms 2 9873 8.92 MB/sec execute 2 sec latency 28.044 ms PIDs: 9647 9648 9649 Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 4096 Resuming dbench 2 10460 4.55 MB/sec execute 5 sec latency 2962.791 ms 2 11124 5.28 MB/sec execute 6 sec latency 2962.929 ms 2 11629 4.80 MB/sec execute 7 sec latency 56.701 ms 2 12341 5.28 MB/sec execute 8 sec latency 39.308 ms 2 13050 5.09 MB/sec execute 9 sec latency 26.300 ms 2 14002 6.02 MB/sec execute 10 sec latency 29.398 ms 2 14625 6.31 MB/sec execute 11 sec latency 9.325 ms 2 15082 5.92 MB/sec execute 12 sec latency 57.560 ms 2 15714 6.14 MB/sec execute 13 sec latency 32.520 ms 2 16376 5.81 MB/sec execute 14 sec latency 25.365 ms 2 17195 6.31 MB/sec execute 15 sec latency 28.719 ms Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 23910 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 2801 Wait for dbench to finish 2 17200 5.64 MB/sec execute 17 sec latency 1816.050 ms 2 18043 6.02 MB/sec execute 18 sec latency 1816.080 ms 2 18534 5.78 MB/sec execute 19 sec latency 68.172 ms 2 19170 5.95 MB/sec execute 20 sec latency 35.777 ms 2 19775 5.72 MB/sec execute 21 sec latency 27.886 ms 2 20558 5.96 MB/sec execute 22 sec latency 27.966 ms 2 21485 6.29 MB/sec execute 23 sec latency 25.914 ms 2 22010 6.15 MB/sec execute 24 sec latency 45.317 ms 2 22507 6.05 MB/sec execute 25 sec latency 38.975 ms 2 23170 6.08 MB/sec execute 26 sec latency 32.960 ms 2 23886 6.04 MB/sec execute 27 sec latency 24.970 ms 2 24902 6.50 MB/sec execute 28 sec latency 26.824 ms 2 25429 6.39 MB/sec execute 29 sec latency 11.656 ms 2 25893 6.27 MB/sec execute 30 sec latency 42.052 ms 2 26587 6.32 MB/sec execute 31 sec latency 23.357 ms 2 27277 6.23 MB/sec execute 32 sec latency 28.261 ms 2 28223 6.49 MB/sec execute 33 sec latency 27.533 ms 2 28849 6.58 MB/sec execute 34 sec latency 9.435 ms 2 29305 6.43 MB/sec execute 35 sec latency 44.136 ms 2 29989 6.50 MB/sec execute 36 sec latency 24.063 ms 2 30680 6.36 MB/sec execute 37 sec latency 23.583 ms 2 31584 6.58 MB/sec execute 38 sec latency 34.536 ms 2 32274 6.70 MB/sec execute 39 sec latency 33.343 ms 2 32765 6.57 MB/sec execute 40 sec latency 40.928 ms 2 33374 6.63 MB/sec execute 41 sec latency 46.319 ms 2 33976 6.50 MB/sec execute 42 sec latency 25.994 ms 2 34733 6.60 MB/sec execute 43 sec latency 32.668 ms 2 35649 6.76 MB/sec execute 44 sec latency 31.225 ms 2 36184 6.68 MB/sec execute 45 sec latency 16.547 ms 2 36633 6.61 MB/sec execute 46 sec latency 43.289 ms 2 37286 6.61 MB/sec execute 47 sec latency 27.406 ms 2 37943 6.55 MB/sec execute 48 sec latency 30.884 ms 2 38878 6.71 MB/sec execute 49 sec latency 26.835 ms 2 39486 6.76 MB/sec execute 50 sec latency 9.690 ms 2 39951 6.66 MB/sec execute 51 sec latency 44.826 ms 2 40589 6.70 MB/sec execute 52 sec latency 35.117 ms 2 41234 6.60 MB/sec execute 53 sec latency 26.703 ms 2 42031 6.73 MB/sec execute 54 sec latency 30.343 ms 2 42850 6.83 MB/sec execute 55 sec latency 25.236 ms 2 43330 6.74 MB/sec execute 56 sec latency 43.175 ms 2 43818 6.68 MB/sec execute 57 sec latency 34.336 ms 2 44458 6.68 MB/sec execute 58 sec latency 36.608 ms 2 45151 6.65 MB/sec execute 59 sec latency 28.247 ms 2 cleanup 60 sec 0 cleanup 60 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 13004 4.734 2962.921 Close 9614 0.978 3.112 Rename 550 5.963 9.454 Unlink 2568 1.728 3.098 Qpathinfo 11850 1.383 1816.068 Qfileinfo 2098 0.114 1.585 Qfsinfo 2160 0.076 1.319 Sfileinfo 1046 2.813 5.997 Find 4606 0.308 5.995 WriteX 6454 0.643 7.513 ReadX 20878 0.010 0.345 LockX 44 0.811 0.989 UnlockX 44 0.860 1.013 Flush 898 17.628 68.164 Throughput 6.65129 MB/sec 2 clients 2 procs max_latency=2962.929 ms stopping dbench in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:45:38 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2b.lustre-rsync-test dbench successfully finished Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 26710 Clear changelog after use: no Errors: 0 lustre_rsync took 5 seconds Changelog records consumed: 11831 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 lustre-MDT0000: changelog user 'cl2' not found PASS 2b (81s) == lustre-rsync-test test complete, duration 178 sec ===== 17:45:45 (1748036745)