-----============= acceptance-small: lustre-rsync-test ============----- Wed Oct 16 10:24:40 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 2b === lustre-rsync-test: start setup 10:24:47 (1729088687) === oleg101-client.virtnet: executing check_config_client /mnt/lustre oleg101-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg101-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012a525800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a525800.idle_timeout=debug disable quota as required oleg101-server: oleg101-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === lustre-rsync-test: finish setup 10:25:01 (1729088701) === nobody:x:99: == lustre-rsync-test test 1A: Simple Replication ========= 10:25:02 (1729088702) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' lustre-MDT0000: Registered changelog user cl1 Replication #1 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 0 seconds Changelog records consumed: 26 Replication #2 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 26 Clear changelog after use: no Errors: 0 lustre_rsync took 0 seconds Changelog records consumed: 3 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0001: changelog user 'cl1' not found lustre-MDT0000: changelog user 'cl1' not found PASS 1A (10s) == lustre-rsync-test test 1a: Replicate create/delete operations in ROOT directory ========================================================== 10:25:12 (1729088712) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl2 cl2' lustre-MDT0000: Registered changelog user cl2 Replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 11 File: '/tmp/target/root-dir' Size: 40 Blocks: 0 IO Block: 4096 directory Device: 27h/39d Inode: 35145 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-10-16 10:25:15.000000000 -0400 Modify: 2024-10-16 10:25:15.000000000 -0400 Change: 2024-10-16 10:25:16.035000000 -0400 Birth: - File: '/tmp/target/root-file' Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: 27h/39d Inode: 35146 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-10-16 10:25:16.000000000 -0400 Modify: 2024-10-16 10:25:15.000000000 -0400 Change: 2024-10-16 10:25:16.075000000 -0400 Birth: - stat: cannot stat '/tmp/target/root-file2': No such file or directory File: '/tmp/target/root-file3' Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: 27h/39d Inode: 37981 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-10-16 10:25:16.000000000 -0400 Modify: 2024-10-16 10:25:15.000000000 -0400 Change: 2024-10-16 10:25:16.117000000 -0400 Birth: - stat: cannot stat '/tmp/target/root-dir1': No such file or directory stat: cannot stat '/tmp/target/root-file4': No such file or directory lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001: Deregistered changelog user #2 lustre-MDT0001: changelog user 'cl2' not found lustre-MDT0000: changelog user 'cl2' not found PASS 1a (9s) == lustre-rsync-test test 2a: Replicate files created by dbench. ========================================================== 10:25:21 (1729088721) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl3 cl3' lustre-MDT0000: Registered changelog user cl3 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2a.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2a.lustre-rsync-test at Wed Oct 16 10:25:26 EDT 2024 waiting for dbench pid 11897 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs failed to create barrier semaphore 1 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 215 16.70 MB/sec warmup 1 sec latency 54.530 ms 2 370 13.16 MB/sec warmup 2 sec latency 45.626 ms 2 614 13.97 MB/sec warmup 3 sec latency 34.261 ms 2 777 10.67 MB/sec warmup 4 sec latency 34.199 ms 2 919 8.70 MB/sec warmup 5 sec latency 32.546 ms 2 1144 7.65 MB/sec warmup 6 sec latency 31.701 ms 2 1410 7.42 MB/sec warmup 7 sec latency 35.812 ms 2 1617 6.53 MB/sec warmup 8 sec latency 31.596 ms 2 1848 5.85 MB/sec warmup 9 sec latency 30.839 ms 2 2068 5.32 MB/sec warmup 10 sec latency 30.144 ms 2 2335 5.06 MB/sec warmup 11 sec latency 30.000 ms 2 2990 9.22 MB/sec execute 1 sec latency 42.748 ms 2 3391 6.58 MB/sec execute 2 sec latency 42.365 ms 2 3707 6.58 MB/sec execute 3 sec latency 26.760 ms 2 3866 5.43 MB/sec execute 4 sec latency 36.041 ms 2 4019 4.46 MB/sec execute 5 sec latency 39.972 ms 2 4248 3.80 MB/sec execute 6 sec latency 33.782 ms 2 4444 3.39 MB/sec execute 7 sec latency 28.451 ms 2 4664 3.29 MB/sec execute 8 sec latency 25.971 ms 2 4951 3.60 MB/sec execute 9 sec latency 28.918 ms 2 5114 3.26 MB/sec execute 10 sec latency 39.894 ms 2 5254 2.99 MB/sec execute 11 sec latency 41.115 ms 2 5417 2.76 MB/sec execute 12 sec latency 38.316 ms 2 5657 2.59 MB/sec execute 13 sec latency 29.152 ms 2 5905 2.58 MB/sec execute 14 sec latency 40.320 ms 2 6060 2.52 MB/sec execute 15 sec latency 39.107 ms 2 6484 2.96 MB/sec execute 16 sec latency 32.357 ms 2 6916 3.02 MB/sec execute 17 sec latency 32.341 ms 2 7239 3.22 MB/sec execute 18 sec latency 20.114 ms 2 7441 3.18 MB/sec execute 19 sec latency 28.124 ms 2 7612 3.03 MB/sec execute 20 sec latency 30.600 ms 2 7873 2.92 MB/sec execute 21 sec latency 50.299 ms 2 8080 2.83 MB/sec execute 22 sec latency 27.022 ms 2 8350 2.82 MB/sec execute 23 sec latency 35.059 ms 2 8673 2.95 MB/sec execute 24 sec latency 24.504 ms 2 8833 2.85 MB/sec execute 25 sec latency 50.000 ms 2 9074 2.75 MB/sec execute 26 sec latency 43.988 ms 2 9293 2.67 MB/sec execute 27 sec latency 27.342 ms 2 9539 2.67 MB/sec execute 28 sec latency 26.138 ms 2 9991 2.90 MB/sec execute 29 sec latency 28.061 ms 2 10442 2.98 MB/sec execute 30 sec latency 28.422 ms 2 10798 3.10 MB/sec execute 31 sec latency 22.515 ms 2 11051 3.09 MB/sec execute 32 sec latency 24.349 ms 2 11302 3.00 MB/sec execute 33 sec latency 22.251 ms 2 11466 2.94 MB/sec execute 34 sec latency 31.144 ms 2 11714 2.93 MB/sec execute 35 sec latency 23.619 ms 2 12050 3.02 MB/sec execute 36 sec latency 23.390 ms 2 12264 2.95 MB/sec execute 37 sec latency 25.287 ms 2 12479 2.88 MB/sec execute 38 sec latency 25.436 ms 2 12729 2.82 MB/sec execute 39 sec latency 22.637 ms 2 13024 2.82 MB/sec execute 40 sec latency 22.777 ms 2 13259 2.80 MB/sec execute 41 sec latency 25.553 ms 2 13852 2.99 MB/sec execute 42 sec latency 24.627 ms 2 14227 3.11 MB/sec execute 43 sec latency 26.316 ms 2 14463 3.10 MB/sec execute 44 sec latency 25.217 ms 2 14648 3.04 MB/sec execute 45 sec latency 28.188 ms 2 14891 2.99 MB/sec execute 46 sec latency 45.640 ms 2 15094 2.95 MB/sec execute 47 sec latency 24.959 ms 2 15348 2.94 MB/sec execute 48 sec latency 23.563 ms 2 15656 3.00 MB/sec execute 49 sec latency 21.324 ms 2 15863 2.95 MB/sec execute 50 sec latency 28.968 ms 2 16039 2.90 MB/sec execute 51 sec latency 27.591 ms 2 16317 2.85 MB/sec execute 52 sec latency 28.148 ms 2 16599 2.85 MB/sec execute 53 sec latency 23.816 ms 2 17023 2.95 MB/sec execute 54 sec latency 29.003 ms 2 17400 2.98 MB/sec execute 55 sec latency 31.488 ms 2 17789 3.07 MB/sec execute 56 sec latency 30.133 ms 2 18008 3.06 MB/sec execute 57 sec latency 24.037 ms 2 18176 3.02 MB/sec execute 58 sec latency 45.855 ms 2 18426 2.98 MB/sec execute 59 sec latency 32.181 ms 2 cleanup 60 sec 0 cleanup 60 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 5540 12.263 49.979 Close 3999 2.662 13.601 Rename 236 18.054 42.317 Unlink 1178 4.086 19.305 Qpathinfo 4972 3.007 24.509 Qfileinfo 857 0.237 6.718 Qfsinfo 964 0.193 5.601 Sfileinfo 430 11.110 26.334 Find 1954 0.728 20.752 WriteX 2711 1.741 22.810 ReadX 8906 0.028 2.025 LockX 20 2.824 5.810 UnlockX 20 2.580 4.546 Flush 402 13.275 50.290 Throughput 2.97679 MB/sec 2 clients 2 procs max_latency=50.299 ms stopping dbench in /mnt/lustre/d2a.lustre-rsync-test at Wed Oct 16 10:26:38 EDT 2024 with return code 0 clean dbench files in /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2a.lustre-rsync-test dbench successfully finished Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl3 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 10 seconds Changelog records consumed: 7702 lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0000: Deregistered changelog user #3 lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0001: Deregistered changelog user #3 lustre-MDT0001: changelog user 'cl3' not found lustre-MDT0000: changelog user 'cl3' not found PASS 2a (92s) SKIP: lustre-rsync-test test_2b skipping ALWAYS excluded test 2b == lustre-rsync-test test 2c: Replicate files while dbench is running. ========================================================== 10:26:54 (1729088814) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl4 cl4' lustre-MDT0000: Registered changelog user cl4 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2c.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2c.lustre-rsync-test at Wed Oct 16 10:26:57 EDT 2024 waiting for dbench pid 13035 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 314 22.97 MB/sec warmup 1 sec latency 26.868 ms 2 702 21.20 MB/sec warmup 2 sec latency 20.257 ms 2 968 14.54 MB/sec warmup 3 sec latency 19.334 ms 2 1248 11.54 MB/sec warmup 4 sec latency 20.282 ms 2 1614 10.46 MB/sec warmup 5 sec latency 22.704 ms 2 1915 8.81 MB/sec warmup 6 sec latency 22.944 ms 2 2190 7.64 MB/sec warmup 7 sec latency 27.490 ms 2 2534 7.20 MB/sec warmup 8 sec latency 21.325 ms 2 2995 7.45 MB/sec warmup 9 sec latency 31.435 ms Running lustre_rsync 2 3521 7.67 MB/sec warmup 10 sec latency 34.550 ms 2 3814 7.23 MB/sec warmup 11 sec latency 26.353 ms 2 4291 0.69 MB/sec execute 1 sec latency 36.560 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 5 seconds Changelog records consumed: 1179 2 4538 0.87 MB/sec execute 2 sec latency 35.820 ms 2 4829 1.44 MB/sec execute 3 sec latency 26.529 ms 2 5115 2.58 MB/sec execute 4 sec latency 27.486 ms 2 5293 2.13 MB/sec execute 5 sec latency 38.414 ms 2 5547 1.84 MB/sec execute 6 sec latency 38.414 ms 13035 13036 13037 Running lustre_rsync 2 5760 1.66 MB/sec execute 7 sec latency 30.685 ms 2 6000 1.79 MB/sec execute 8 sec latency 48.821 ms 2 6437 2.50 MB/sec execute 9 sec latency 26.665 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 8921 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 1162 2 6829 2.88 MB/sec execute 10 sec latency 26.591 ms 2 7179 3.21 MB/sec execute 11 sec latency 22.003 ms 2 7395 3.14 MB/sec execute 12 sec latency 23.384 ms 2 7587 2.94 MB/sec execute 13 sec latency 28.529 ms 2 7807 2.77 MB/sec execute 14 sec latency 45.327 ms 13035 13036 13037 Running lustre_rsync 2 7995 2.64 MB/sec execute 15 sec latency 28.150 ms 2 8219 2.64 MB/sec execute 16 sec latency 31.554 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 10082 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 835 2 8505 2.84 MB/sec execute 17 sec latency 27.393 ms 2 8693 2.70 MB/sec execute 18 sec latency 40.924 ms 2 8886 2.57 MB/sec execute 19 sec latency 29.548 ms 2 9107 2.47 MB/sec execute 20 sec latency 38.275 ms 2 9388 2.44 MB/sec execute 21 sec latency 58.048 ms 13035 13036 13037 Running lustre_rsync 2 9663 2.46 MB/sec execute 22 sec latency 30.811 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 10916 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 664 2 10219 2.79 MB/sec execute 23 sec latency 26.034 ms 2 10746 3.07 MB/sec execute 24 sec latency 21.853 ms 2 11017 3.06 MB/sec execute 25 sec latency 21.169 ms 2 11283 2.95 MB/sec execute 26 sec latency 19.726 ms 2 11534 2.88 MB/sec execute 27 sec latency 28.014 ms 13035 13036 13037 Running lustre_rsync 2 11784 2.88 MB/sec execute 28 sec latency 24.836 ms 2 12061 2.99 MB/sec execute 29 sec latency 25.416 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 11579 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 876 2 12259 2.90 MB/sec execute 30 sec latency 29.926 ms 2 12479 2.81 MB/sec execute 31 sec latency 46.816 ms 2 12740 2.75 MB/sec execute 32 sec latency 27.379 ms 2 13012 2.74 MB/sec execute 33 sec latency 29.232 ms 2 13231 2.72 MB/sec execute 34 sec latency 39.694 ms 13035 13036 13037 Running lustre_rsync 2 13662 2.90 MB/sec execute 35 sec latency 28.600 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 12454 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 702 2 14219 3.09 MB/sec execute 36 sec latency 30.742 ms 2 14476 3.08 MB/sec execute 37 sec latency 21.416 ms 2 14683 3.02 MB/sec execute 38 sec latency 25.151 ms 2 14947 2.96 MB/sec execute 39 sec latency 35.683 ms 2 15183 2.91 MB/sec execute 40 sec latency 24.193 ms 13035 13036 13037 Running lustre_rsync 2 15438 2.90 MB/sec execute 41 sec latency 23.821 ms 2 15756 2.98 MB/sec execute 42 sec latency 27.500 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 13155 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 844 2 15960 2.92 MB/sec execute 43 sec latency 21.968 ms 2 16203 2.86 MB/sec execute 44 sec latency 29.300 ms 2 16491 2.85 MB/sec execute 45 sec latency 24.738 ms 2 16753 2.84 MB/sec execute 46 sec latency 30.934 ms 2 17169 2.97 MB/sec execute 47 sec latency 26.137 ms 13035 13036 13037 Running lustre_rsync 2 17704 3.11 MB/sec execute 48 sec latency 23.607 ms 2 17950 3.11 MB/sec execute 49 sec latency 37.252 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 13998 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 743 2 18141 3.06 MB/sec execute 50 sec latency 23.625 ms 2 18323 3.00 MB/sec execute 51 sec latency 33.126 ms 2 18522 2.96 MB/sec execute 52 sec latency 50.170 ms 2 18724 2.92 MB/sec execute 53 sec latency 31.010 ms 2 18992 2.91 MB/sec execute 54 sec latency 24.480 ms 13035 13036 13037 Running lustre_rsync 2 19287 2.97 MB/sec execute 55 sec latency 27.464 ms 2 19470 2.92 MB/sec execute 56 sec latency 30.406 ms 2 19687 2.88 MB/sec execute 57 sec latency 32.177 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 14740 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 802 2 19915 2.84 MB/sec execute 58 sec latency 25.552 ms 2 20179 2.83 MB/sec execute 59 sec latency 31.645 ms 2 cleanup 60 sec 0 cleanup 61 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 5729 11.824 58.026 Close 4273 2.445 9.818 Rename 242 17.474 46.791 Unlink 1137 3.886 13.362 Qpathinfo 5247 2.797 25.078 Qfileinfo 914 0.227 3.028 Qfsinfo 926 0.174 5.204 Sfileinfo 500 11.037 37.673 Find 2006 0.674 26.217 WriteX 2832 1.715 22.866 ReadX 8816 0.030 1.849 LockX 16 2.237 3.170 UnlockX 16 2.176 3.604 Flush 412 13.633 50.158 Throughput 2.83145 MB/sec 2 clients 2 procs max_latency=58.048 ms stopping dbench in /mnt/lustre/d2c.lustre-rsync-test at Wed Oct 16 10:28:10 EDT 2024 with return code 0 clean dbench files in /mnt/lustre/d2c.lustre-rsync-test /mnt/lustre/d2c.lustre-rsync-test /mnt/lustre/d2c.lustre-rsync-test removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2c.lustre-rsync-test dbench successfully finished Running lustre_rsync Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 15541 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 662 lustre-MDT0000: clear the changelog for cl4 of all records lustre-MDT0001: clear the changelog for cl4 of all records lustre-MDT0000: clear the changelog for cl4 of all records lustre-MDT0000: Deregistered changelog user #4 lustre-MDT0001: clear the changelog for cl4 of all records lustre-MDT0001: Deregistered changelog user #4 lustre-MDT0001: changelog user 'cl4' not found lustre-MDT0000: changelog user 'cl4' not found PASS 2c (89s) == lustre-rsync-test test 3a: Replicate files created by createmany ========================================================== 10:28:23 (1729088903) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl5 cl5' lustre-MDT0000: Registered changelog user cl5 total: 1000 open/close in 6.13 seconds: 163.26 ops/second Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl5 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 23 seconds Changelog records consumed: 2000 lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0001: clear the changelog for cl5 of all records lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0000: Deregistered changelog user #5 lustre-MDT0001: clear the changelog for cl5 of all records lustre-MDT0001: Deregistered changelog user #5 lustre-MDT0001: changelog user 'cl5' not found lustre-MDT0000: changelog user 'cl5' not found PASS 3a (42s) == lustre-rsync-test test 3b: Replicate files created by writemany ========================================================== 10:29:05 (1729088945) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl6 cl6' lustre-MDT0000: Registered changelog user cl6 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl6 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 64 seconds Changelog records consumed: 31650 lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0000: Deregistered changelog user #6 lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0001: Deregistered changelog user #6 lustre-MDT0001: changelog user 'cl6' not found lustre-MDT0000: changelog user 'cl6' not found PASS 3b (130s) == lustre-rsync-test test 3c: Replicate files created by createmany/unlinkmany ========================================================== 10:31:15 (1729089075) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl7 cl7' lustre-MDT0000: Registered changelog user cl7 total: 1000 open/close in 3.43 seconds: 291.52 ops/second - unlinked 0 (time 1729089083 ; total 0 ; last 0) total: 1000 unlinks in 2 seconds: 500.000000 unlinks/second Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl7 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 4 seconds Changelog records consumed: 3000 lustre-MDT0000: clear the changelog for cl7 of all records lustre-MDT0001: clear the changelog for cl7 of all records lustre-MDT0000: clear the changelog for cl7 of all records lustre-MDT0000: Deregistered changelog user #7 lustre-MDT0001: clear the changelog for cl7 of all records lustre-MDT0001: Deregistered changelog user #7 lustre-MDT0001: changelog user 'cl7' not found lustre-MDT0000: changelog user 'cl7' not found PASS 3c (18s) == lustre-rsync-test test 4: Replicate files created by iozone ========================================================== 10:31:33 (1729089093) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl8 cl8' lustre-MDT0000: Registered changelog user cl8 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl8 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 676 /home/green/git/lustre-release/lustre/tests/lustre-rsync-test.sh: line 450: 17272 Killed END_RUN_FILE=${DIR}/$tdir/run LOAD_PID_FILE=${DIR}/$tdir/pid MOUNT=${DIR}/$tdir run_iozone.sh Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl8 Starting changelog record: 53528 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 1021 lustre-MDT0000: clear the changelog for cl8 of all records lustre-MDT0001: clear the changelog for cl8 of all records lustre-MDT0000: clear the changelog for cl8 of all records lustre-MDT0000: Deregistered changelog user #8 lustre-MDT0001: clear the changelog for cl8 of all records lustre-MDT0001: Deregistered changelog user #8 lustre-MDT0001: changelog user 'cl8' not found lustre-MDT0000: changelog user 'cl8' not found PASS 4 (99s) == lustre-rsync-test test 5a: Stop / start lustre_rsync == 10:33:12 (1729089192) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl9 cl9' lustre-MDT0000: Registered changelog user cl9 total: 2000 open/close in 7.69 seconds: 259.96 ops/second Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl9 Starting changelog record: 0 Clear changelog after use: no lustre_rsync halting. Errors: 0 lustre_rsync took 30 seconds Changelog records consumed: 2949 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl9 Starting changelog record: 57497 Clear changelog after use: no Errors: 0 lustre_rsync took 12 seconds Changelog records consumed: 1052 lustre-MDT0000: clear the changelog for cl9 of all records lustre-MDT0001: clear the changelog for cl9 of all records lustre-MDT0000: clear the changelog for cl9 of all records lustre-MDT0000: Deregistered changelog user #9 lustre-MDT0001: clear the changelog for cl9 of all records lustre-MDT0001: Deregistered changelog user #9 lustre-MDT0001: changelog user 'cl9' not found lustre-MDT0000: changelog user 'cl9' not found PASS 5a (69s) == lustre-rsync-test test 5b: Kill / restart lustre_rsync ========================================================== 10:34:21 (1729089261) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl10 cl10' lustre-MDT0000: Registered changelog user cl10 total: 2000 open/close in 7.51 seconds: 266.30 ops/second /home/green/git/lustre-release/lustre/tests/lustre-rsync-test.sh: line 539: 19526 Killed $LRSYNC -s $DIR -t $TGT -t $TGT2 -m $MDT0 -u $CL_USER -l $LREPL_LOG -D $LRSYNC_LOG Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl10 Starting changelog record: 61074 Clear changelog after use: no Errors: 0 lustre_rsync took 16 seconds Changelog records consumed: 1475 lustre-MDT0000: clear the changelog for cl10 of all records lustre-MDT0001: clear the changelog for cl10 of all records lustre-MDT0000: clear the changelog for cl10 of all records lustre-MDT0000: Deregistered changelog user #10 lustre-MDT0001: clear the changelog for cl10 of all records lustre-MDT0001: Deregistered changelog user #10 lustre-MDT0001: changelog user 'cl10' not found lustre-MDT0000: changelog user 'cl10' not found PASS 5b (70s) == lustre-rsync-test test 6: lustre_rsync large no of hard links ========================================================== 10:35:31 (1729089331) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl11 cl11' lustre-MDT0000: Registered changelog user cl11 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl11 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 0 seconds Changelog records consumed: 129 lustre-MDT0000: clear the changelog for cl11 of all records lustre-MDT0001: clear the changelog for cl11 of all records lustre-MDT0000: clear the changelog for cl11 of all records lustre-MDT0000: Deregistered changelog user #11 lustre-MDT0001: clear the changelog for cl11 of all records lustre-MDT0001: Deregistered changelog user #11 lustre-MDT0001: changelog user 'cl11' not found lustre-MDT0000: changelog user 'cl11' not found PASS 6 (9s) == lustre-rsync-test test 7: lustre_rsync stripesize ===== 10:35:40 (1729089340) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl12 cl12' lustre-MDT0000: Registered changelog user cl12 total: 100 open/close in 0.38 seconds: 260.22 ops/second Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /mnt/lustre/tgt Statuslog: /tmp/lustre_rsync.log Changelog registration: cl12 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 5 seconds Changelog records consumed: 202 lustre-MDT0000: clear the changelog for cl12 of all records lustre-MDT0001: clear the changelog for cl12 of all records lustre-MDT0000: clear the changelog for cl12 of all records lustre-MDT0000: Deregistered changelog user #12 lustre-MDT0001: clear the changelog for cl12 of all records lustre-MDT0001: Deregistered changelog user #12 lustre-MDT0001: changelog user 'cl12' not found lustre-MDT0000: changelog user 'cl12' not found PASS 7 (13s) == lustre-rsync-test test 8: Replicate multiple file/directory moves ========================================================== 10:35:53 (1729089353) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl13 cl13' lustre-MDT0000: Registered changelog user cl13 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl13 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 30 seconds Changelog records consumed: 3501 lustre-MDT0000: clear the changelog for cl13 of all records lustre-MDT0001: clear the changelog for cl13 of all records lustre-MDT0000: clear the changelog for cl13 of all records lustre-MDT0000: Deregistered changelog user #13 lustre-MDT0001: clear the changelog for cl13 of all records lustre-MDT0001: Deregistered changelog user #13 lustre-MDT0001: changelog user 'cl13' not found lustre-MDT0000: changelog user 'cl13' not found PASS 8 (66s) == lustre-rsync-test test 9: Replicate recursive directory removal ========================================================== 10:36:59 (1729089419) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl14 cl14' lustre-MDT0000: Registered changelog user cl14 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl14 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 0 seconds Changelog records consumed: 3 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl14 Starting changelog record: 66383 Clear changelog after use: no Errors: 0 lustre_rsync took 0 seconds Changelog records consumed: 3 lustre-MDT0000: clear the changelog for cl14 of all records lustre-MDT0001: clear the changelog for cl14 of all records lustre-MDT0000: clear the changelog for cl14 of all records lustre-MDT0000: Deregistered changelog user #14 lustre-MDT0001: clear the changelog for cl14 of all records lustre-MDT0001: Deregistered changelog user #14 lustre-MDT0001: changelog user 'cl14' not found lustre-MDT0000: changelog user 'cl14' not found PASS 9 (6s) == lustre-rsync-test test complete, duration 744 sec ===== 10:37:05 (1729089425) === lustre-rsync-test: start cleanup 10:37:06 (1729089426) === === lustre-rsync-test: finish cleanup 10:37:06 (1729089426) ===