-----============= acceptance-small: lustre-rsync-test ============----- Tue Nov 5 05:16:53 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 2b === lustre-rsync-test: start setup 05:17:01 (1730801821) === oleg104-client.virtnet: executing check_config_client /mnt/lustre oleg104-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg104-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800aa534800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800aa534800.idle_timeout=debug disable quota as required oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === lustre-rsync-test: finish setup 05:17:17 (1730801837) === nobody:x:99: == lustre-rsync-test test 1A: Simple Replication ========= 05:17:18 (1730801838) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' lustre-MDT0000: Registered changelog user cl1 Replication #1 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 0 seconds Changelog records consumed: 26 Replication #2 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 26 Clear changelog after use: no Errors: 0 lustre_rsync took 0 seconds Changelog records consumed: 3 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0001: changelog user 'cl1' not found lustre-MDT0000: changelog user 'cl1' not found PASS 1A (10s) == lustre-rsync-test test 1a: Replicate create/delete operations in ROOT directory ========================================================== 05:17:29 (1730801849) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl2 cl2' lustre-MDT0000: Registered changelog user cl2 Replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 11 File: '/tmp/target/root-dir' Size: 40 Blocks: 0 IO Block: 4096 directory Device: 27h/39d Inode: 36096 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-11-05 05:17:32.000000000 -0500 Modify: 2024-11-05 05:17:32.000000000 -0500 Change: 2024-11-05 05:17:32.969000000 -0500 Birth: - File: '/tmp/target/root-file' Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: 27h/39d Inode: 36097 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-11-05 05:17:32.000000000 -0500 Modify: 2024-11-05 05:17:32.000000000 -0500 Change: 2024-11-05 05:17:32.994000000 -0500 Birth: - stat: cannot stat '/tmp/target/root-file2': No such file or directory File: '/tmp/target/root-file3' Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: 27h/39d Inode: 36954 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-11-05 05:17:33.000000000 -0500 Modify: 2024-11-05 05:17:32.000000000 -0500 Change: 2024-11-05 05:17:33.024000000 -0500 Birth: - stat: cannot stat '/tmp/target/root-dir1': No such file or directory stat: cannot stat '/tmp/target/root-file4': No such file or directory lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001: Deregistered changelog user #2 lustre-MDT0001: changelog user 'cl2' not found lustre-MDT0000: changelog user 'cl2' not found PASS 1a (11s) == lustre-rsync-test test 2a: Replicate files created by dbench. ========================================================== 05:17:39 (1730801859) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl3 cl3' lustre-MDT0000: Registered changelog user cl3 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2a.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2a.lustre-rsync-test at Tue Nov 5 05:17:44 EST 2024 waiting for dbench pid 11921 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs failed to create barrier semaphore 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 188 15.14 MB/sec warmup 1 sec latency 59.377 ms 2 353 12.64 MB/sec warmup 2 sec latency 50.842 ms 2 564 13.05 MB/sec warmup 3 sec latency 45.245 ms 2 763 10.67 MB/sec warmup 4 sec latency 31.695 ms 2 915 8.70 MB/sec warmup 5 sec latency 33.983 ms 2 1127 7.66 MB/sec warmup 6 sec latency 27.524 ms 2 1377 7.42 MB/sec warmup 7 sec latency 45.858 ms 2 1557 6.52 MB/sec warmup 8 sec latency 34.653 ms 2 1776 5.84 MB/sec warmup 9 sec latency 35.872 ms 2 2011 5.30 MB/sec warmup 10 sec latency 25.056 ms 2 2217 4.87 MB/sec warmup 11 sec latency 27.467 ms 2 2877 8.17 MB/sec execute 1 sec latency 37.940 ms 2 3208 6.17 MB/sec execute 2 sec latency 40.104 ms 2 3569 6.88 MB/sec execute 3 sec latency 27.606 ms 2 3780 5.82 MB/sec execute 4 sec latency 30.344 ms 2 3906 4.77 MB/sec execute 5 sec latency 59.735 ms 2 4052 3.98 MB/sec execute 6 sec latency 32.808 ms 2 4261 3.48 MB/sec execute 7 sec latency 43.426 ms 2 4427 3.14 MB/sec execute 8 sec latency 43.767 ms 2 4552 2.87 MB/sec execute 9 sec latency 38.924 ms 2 4790 2.82 MB/sec execute 10 sec latency 32.794 ms 2 5060 3.10 MB/sec execute 11 sec latency 35.475 ms 2 5227 2.87 MB/sec execute 12 sec latency 33.370 ms 2 5398 2.67 MB/sec execute 13 sec latency 33.698 ms 2 5622 2.52 MB/sec execute 14 sec latency 30.526 ms 2 5890 2.51 MB/sec execute 15 sec latency 26.543 ms 2 6046 2.46 MB/sec execute 16 sec latency 41.148 ms 2 6478 2.88 MB/sec execute 17 sec latency 29.881 ms 2 6899 2.93 MB/sec execute 18 sec latency 30.459 ms 2 7186 3.12 MB/sec execute 19 sec latency 31.119 ms 2 7363 3.08 MB/sec execute 20 sec latency 32.364 ms 2 7525 2.96 MB/sec execute 21 sec latency 30.814 ms 2 7698 2.83 MB/sec execute 22 sec latency 31.822 ms 2 7871 2.74 MB/sec execute 23 sec latency 63.983 ms 2 8024 2.66 MB/sec execute 24 sec latency 41.397 ms 2 8268 2.65 MB/sec execute 25 sec latency 27.538 ms 2 8560 2.78 MB/sec execute 26 sec latency 28.103 ms 2 8761 2.69 MB/sec execute 27 sec latency 27.952 ms 2 8966 2.60 MB/sec execute 28 sec latency 25.454 ms 2 9260 2.54 MB/sec execute 29 sec latency 24.604 ms 2 9526 2.55 MB/sec execute 30 sec latency 23.063 ms 2 9868 2.71 MB/sec execute 31 sec latency 26.224 ms 2 10338 2.79 MB/sec execute 32 sec latency 27.335 ms 2 10724 2.95 MB/sec execute 33 sec latency 26.121 ms 2 10921 2.93 MB/sec execute 34 sec latency 28.059 ms 2 11107 2.87 MB/sec execute 35 sec latency 24.212 ms 2 11326 2.80 MB/sec execute 36 sec latency 50.817 ms 2 11527 2.75 MB/sec execute 37 sec latency 37.750 ms 2 11766 2.75 MB/sec execute 38 sec latency 24.426 ms 2 12064 2.83 MB/sec execute 39 sec latency 24.874 ms 2 12264 2.77 MB/sec execute 40 sec latency 22.921 ms 2 12459 2.71 MB/sec execute 41 sec latency 23.843 ms 2 12682 2.66 MB/sec execute 42 sec latency 24.201 ms 2 12993 2.66 MB/sec execute 43 sec latency 21.941 ms 2 13216 2.64 MB/sec execute 44 sec latency 35.656 ms 2 13639 2.79 MB/sec execute 45 sec latency 27.406 ms 2 14059 2.83 MB/sec execute 46 sec latency 32.194 ms 2 14328 2.89 MB/sec execute 47 sec latency 28.125 ms 2 14483 2.87 MB/sec execute 48 sec latency 47.761 ms 2 14616 2.83 MB/sec execute 49 sec latency 38.519 ms 2 14750 2.77 MB/sec execute 50 sec latency 35.114 ms 2 14949 2.73 MB/sec execute 51 sec latency 48.070 ms 2 15116 2.70 MB/sec execute 52 sec latency 32.124 ms 2 15329 2.69 MB/sec execute 53 sec latency 33.145 ms 2 15593 2.75 MB/sec execute 54 sec latency 28.151 ms 2 15778 2.71 MB/sec execute 55 sec latency 32.101 ms 2 15956 2.67 MB/sec execute 56 sec latency 29.854 ms 2 16174 2.62 MB/sec execute 57 sec latency 22.190 ms 2 16380 2.59 MB/sec execute 58 sec latency 24.609 ms 2 16627 2.59 MB/sec execute 59 sec latency 30.403 ms 2 cleanup 60 sec 0 cleanup 61 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 4979 13.544 63.968 Close 3656 2.914 12.115 Rename 216 19.615 39.709 Unlink 1004 4.689 14.970 Qpathinfo 4546 3.268 32.211 Qfileinfo 807 0.243 4.028 Qfsinfo 842 0.188 12.049 Sfileinfo 412 12.526 31.290 Find 1758 0.816 26.439 WriteX 2476 1.973 25.376 ReadX 8162 0.035 3.318 LockX 20 3.143 8.255 UnlockX 20 2.657 4.937 Flush 348 15.752 50.800 Throughput 2.59132 MB/sec 2 clients 2 procs max_latency=63.983 ms stopping dbench in /mnt/lustre/d2a.lustre-rsync-test at Tue Nov 5 05:18:57 EST 2024 with return code 0 clean dbench files in /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2a.lustre-rsync-test dbench successfully finished Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl3 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 11 seconds Changelog records consumed: 7028 lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0000: Deregistered changelog user #3 lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0001: Deregistered changelog user #3 lustre-MDT0001: changelog user 'cl3' not found lustre-MDT0000: changelog user 'cl3' not found PASS 2a (94s) SKIP: lustre-rsync-test test_2b skipping ALWAYS excluded test 2b == lustre-rsync-test test 2c: Replicate files while dbench is running. ========================================================== 05:19:14 (1730801954) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl4 cl4' lustre-MDT0000: Registered changelog user cl4 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2c.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2c.lustre-rsync-test at Tue Nov 5 05:19:18 EST 2024 waiting for dbench pid 13063 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 230 17.74 MB/sec warmup 1 sec latency 38.687 ms 2 433 15.23 MB/sec warmup 2 sec latency 41.567 ms 2 688 14.11 MB/sec warmup 3 sec latency 35.418 ms 2 845 10.77 MB/sec warmup 4 sec latency 44.358 ms 2 979 8.75 MB/sec warmup 5 sec latency 35.460 ms 2 1176 7.67 MB/sec warmup 6 sec latency 43.206 ms 2 1423 7.43 MB/sec warmup 7 sec latency 33.154 ms 2 1615 6.54 MB/sec warmup 8 sec latency 34.114 ms 2 1808 5.85 MB/sec warmup 9 sec latency 52.358 ms Running lustre_rsync 2 2012 5.30 MB/sec warmup 10 sec latency 30.267 ms 2 2196 4.86 MB/sec warmup 11 sec latency 30.581 ms 2 2613 2.08 MB/sec execute 1 sec latency 35.968 ms 2 2986 5.52 MB/sec execute 2 sec latency 39.040 ms 2 3356 4.94 MB/sec execute 3 sec latency 38.408 ms 2 3629 5.30 MB/sec execute 4 sec latency 39.672 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 7 seconds Changelog records consumed: 764 2 3814 4.70 MB/sec execute 5 sec latency 49.025 ms 2 3929 4.02 MB/sec execute 6 sec latency 41.729 ms 2 4058 3.45 MB/sec execute 7 sec latency 39.872 ms 2 4255 3.07 MB/sec execute 8 sec latency 77.712 ms 2 4383 2.80 MB/sec execute 9 sec latency 47.103 ms 13063 13064 13065 Running lustre_rsync 2 4543 2.59 MB/sec execute 10 sec latency 40.259 ms 2 4757 2.57 MB/sec execute 11 sec latency 39.908 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 7832 Clear changelog after use: no Errors: 0 lustre_rsync took 3 seconds Changelog records consumed: 1033 2 4995 2.85 MB/sec execute 12 sec latency 39.135 ms 2 5169 2.66 MB/sec execute 13 sec latency 33.750 ms 2 5342 2.49 MB/sec execute 14 sec latency 28.665 ms 2 5545 2.34 MB/sec execute 15 sec latency 33.521 ms 2 5756 2.23 MB/sec execute 16 sec latency 26.370 ms 13063 13064 13065 Running lustre_rsync 2 6001 2.26 MB/sec execute 17 sec latency 24.600 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 8864 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 619 2 6435 2.59 MB/sec execute 18 sec latency 24.128 ms 2 6827 2.78 MB/sec execute 19 sec latency 30.066 ms 2 7129 2.95 MB/sec execute 20 sec latency 46.176 ms 2 7341 2.94 MB/sec execute 21 sec latency 31.444 ms 2 7504 2.83 MB/sec execute 22 sec latency 37.740 ms 13063 13064 13065 Running lustre_rsync 2 7723 2.71 MB/sec execute 23 sec latency 23.007 ms 2 7892 2.63 MB/sec execute 24 sec latency 47.598 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 9482 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 646 2 8051 2.56 MB/sec execute 25 sec latency 68.623 ms 2 8245 2.55 MB/sec execute 26 sec latency 29.651 ms 2 8513 2.68 MB/sec execute 27 sec latency 24.736 ms 2 8670 2.59 MB/sec execute 28 sec latency 44.729 ms 2 8831 2.52 MB/sec execute 29 sec latency 38.597 ms 13063 13064 13065 Running lustre_rsync 2 8995 2.44 MB/sec execute 30 sec latency 28.025 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 10127 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 561 2 9194 2.38 MB/sec execute 31 sec latency 33.513 ms 2 9455 2.38 MB/sec execute 32 sec latency 31.032 ms 2 9673 2.37 MB/sec execute 33 sec latency 26.468 ms 2 10087 2.57 MB/sec execute 34 sec latency 32.685 ms 2 10607 2.78 MB/sec execute 35 sec latency 26.882 ms 13063 13064 13065 Running lustre_rsync 2 10802 2.72 MB/sec execute 36 sec latency 25.613 ms 2 10987 2.72 MB/sec execute 37 sec latency 39.835 ms 2 11150 2.65 MB/sec execute 38 sec latency 37.171 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 10687 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 618 2 11370 2.59 MB/sec execute 39 sec latency 31.218 ms 2 11535 2.55 MB/sec execute 40 sec latency 34.865 ms 2 11720 2.55 MB/sec execute 41 sec latency 30.922 ms 2 12004 2.59 MB/sec execute 42 sec latency 25.237 ms 2 12218 2.58 MB/sec execute 43 sec latency 26.125 ms 13063 13064 13065 Running lustre_rsync 2 12375 2.53 MB/sec execute 44 sec latency 28.362 ms 2 12535 2.48 MB/sec execute 45 sec latency 26.986 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 11304 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 718 2 12753 2.43 MB/sec execute 46 sec latency 34.661 ms 2 13007 2.44 MB/sec execute 47 sec latency 27.787 ms 2 13183 2.42 MB/sec execute 48 sec latency 35.196 ms 2 13573 2.56 MB/sec execute 49 sec latency 48.518 ms 2 13919 2.59 MB/sec execute 50 sec latency 37.580 ms 13063 13064 13065 Running lustre_rsync 2 14227 2.66 MB/sec execute 51 sec latency 35.252 ms 2 14393 2.65 MB/sec execute 52 sec latency 39.916 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 12021 Clear changelog after use: no Errors: 0 lustre_rsync took 3 seconds Changelog records consumed: 632 2 14520 2.61 MB/sec execute 53 sec latency 36.873 ms 2 14666 2.57 MB/sec execute 54 sec latency 30.976 ms 2 14817 2.52 MB/sec execute 55 sec latency 37.419 ms 2 14971 2.49 MB/sec execute 56 sec latency 40.964 ms 2 15113 2.46 MB/sec execute 57 sec latency 50.857 ms 2 15282 2.46 MB/sec execute 58 sec latency 37.727 ms 13063 13064 13065 Running lustre_rsync 2 15435 2.42 MB/sec execute 59 sec latency 34.348 ms 2 cleanup 60 sec Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 12652 Clear changelog after use: no Errors: 0 lustre_rsync took 3 seconds Changelog records consumed: 635 0 cleanup 61 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 4509 14.884 77.676 Close 3271 3.269 22.832 Rename 202 21.562 48.488 Unlink 964 5.336 29.776 Qpathinfo 4068 3.601 34.973 Qfileinfo 700 0.261 4.367 Qfsinfo 804 0.222 8.636 Sfileinfo 386 14.310 68.610 Find 1605 0.855 29.101 WriteX 2248 2.102 25.956 ReadX 7393 0.031 3.522 LockX 16 3.313 5.608 UnlockX 16 3.271 5.367 Flush 328 16.531 47.589 Throughput 2.42487 MB/sec 2 clients 2 procs max_latency=77.712 ms stopping dbench in /mnt/lustre/d2c.lustre-rsync-test at Tue Nov 5 05:20:31 EST 2024 with return code 0 clean dbench files in /mnt/lustre/d2c.lustre-rsync-test /mnt/lustre/d2c.lustre-rsync-test /mnt/lustre/d2c.lustre-rsync-test removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2c.lustre-rsync-test dbench successfully finished Running lustre_rsync Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 13286 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 272 lustre-MDT0000: clear the changelog for cl4 of all records lustre-MDT0001: clear the changelog for cl4 of all records lustre-MDT0000: clear the changelog for cl4 of all records lustre-MDT0000: Deregistered changelog user #4 lustre-MDT0001: clear the changelog for cl4 of all records lustre-MDT0001: Deregistered changelog user #4 lustre-MDT0001: changelog user 'cl4' not found lustre-MDT0000: changelog user 'cl4' not found PASS 2c (94s) == lustre-rsync-test test 3a: Replicate files created by createmany ========================================================== 05:20:48 (1730802048) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl5 cl5' lustre-MDT0000: Registered changelog user cl5 total: 1000 open/close in 7.51 seconds: 133.18 ops/second Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl5 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 41 seconds Changelog records consumed: 2000 lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0001: clear the changelog for cl5 of all records lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0000: Deregistered changelog user #5 lustre-MDT0001: clear the changelog for cl5 of all records lustre-MDT0001: Deregistered changelog user #5 lustre-MDT0001: changelog user 'cl5' not found lustre-MDT0000: changelog user 'cl5' not found PASS 3a (68s) == lustre-rsync-test test 3b: Replicate files created by writemany ========================================================== 05:21:56 (1730802116) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl6 cl6' lustre-MDT0000: Registered changelog user cl6 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl6 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 56 seconds Changelog records consumed: 24111 lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0000: Deregistered changelog user #6 lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0001: Deregistered changelog user #6 lustre-MDT0001: changelog user 'cl6' not found lustre-MDT0000: changelog user 'cl6' not found PASS 3b (123s) == lustre-rsync-test test 3c: Replicate files created by createmany/unlinkmany ========================================================== 05:24:00 (1730802240) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl7 cl7' lustre-MDT0000: Registered changelog user cl7 total: 1000 open/close in 4.29 seconds: 233.12 ops/second - unlinked 0 (time 1730802248 ; total 0 ; last 0) total: 1000 unlinks in 3 seconds: 333.333344 unlinks/second Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl7 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 6 seconds Changelog records consumed: 3000 lustre-MDT0000: clear the changelog for cl7 of all records lustre-MDT0001: clear the changelog for cl7 of all records lustre-MDT0000: clear the changelog for cl7 of all records lustre-MDT0000: Deregistered changelog user #7 lustre-MDT0001: clear the changelog for cl7 of all records lustre-MDT0001: Deregistered changelog user #7 lustre-MDT0001: changelog user 'cl7' not found lustre-MDT0000: changelog user 'cl7' not found PASS 3c (21s) == lustre-rsync-test test 4: Replicate files created by iozone ========================================================== 05:24:21 (1730802261) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl8 cl8' lustre-MDT0000: Registered changelog user cl8 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl8 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 675 /home/green/git/lustre-release/lustre/tests/lustre-rsync-test.sh: line 450: 17320 Killed END_RUN_FILE=${DIR}/$tdir/run LOAD_PID_FILE=${DIR}/$tdir/pid MOUNT=${DIR}/$tdir run_iozone.sh Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl8 Starting changelog record: 43343 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 760 lustre-MDT0000: clear the changelog for cl8 of all records lustre-MDT0001: clear the changelog for cl8 of all records lustre-MDT0000: clear the changelog for cl8 of all records lustre-MDT0000: Deregistered changelog user #8 lustre-MDT0001: clear the changelog for cl8 of all records lustre-MDT0001: Deregistered changelog user #8 lustre-MDT0001: changelog user 'cl8' not found lustre-MDT0000: changelog user 'cl8' not found PASS 4 (100s) == lustre-rsync-test test 5a: Stop / start lustre_rsync == 05:26:01 (1730802361) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl9 cl9' lustre-MDT0000: Registered changelog user cl9 total: 2000 open/close in 8.67 seconds: 230.79 ops/second Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl9 Starting changelog record: 0 Clear changelog after use: no lustre_rsync halting. Errors: 0 lustre_rsync took 30 seconds Changelog records consumed: 2799 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl9 Starting changelog record: 46902 Clear changelog after use: no Errors: 0 lustre_rsync took 12 seconds Changelog records consumed: 1202 lustre-MDT0000: clear the changelog for cl9 of all records lustre-MDT0001: clear the changelog for cl9 of all records lustre-MDT0000: clear the changelog for cl9 of all records lustre-MDT0000: Deregistered changelog user #9 lustre-MDT0001: clear the changelog for cl9 of all records lustre-MDT0001: Deregistered changelog user #9 lustre-MDT0001: changelog user 'cl9' not found lustre-MDT0000: changelog user 'cl9' not found PASS 5a (68s) == lustre-rsync-test test 5b: Kill / restart lustre_rsync ========================================================== 05:27:09 (1730802429) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl10 cl10' lustre-MDT0000: Registered changelog user cl10 total: 2000 open/close in 7.85 seconds: 254.92 ops/second /home/green/git/lustre-release/lustre/tests/lustre-rsync-test.sh: line 539: 19570 Killed $LRSYNC -s $DIR -t $TGT -t $TGT2 -m $MDT0 -u $CL_USER -l $LREPL_LOG -D $LRSYNC_LOG Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl10 Starting changelog record: 50932 Clear changelog after use: no Errors: 0 lustre_rsync took 11 seconds Changelog records consumed: 1172 lustre-MDT0000: clear the changelog for cl10 of all records lustre-MDT0001: clear the changelog for cl10 of all records lustre-MDT0000: clear the changelog for cl10 of all records lustre-MDT0000: Deregistered changelog user #10 lustre-MDT0001: clear the changelog for cl10 of all records lustre-MDT0001: Deregistered changelog user #10 lustre-MDT0001: changelog user 'cl10' not found lustre-MDT0000: changelog user 'cl10' not found PASS 5b (68s) == lustre-rsync-test test 6: lustre_rsync large no of hard links ========================================================== 05:28:17 (1730802497) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl11 cl11' lustre-MDT0000: Registered changelog user cl11 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl11 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 0 seconds Changelog records consumed: 129 lustre-MDT0000: clear the changelog for cl11 of all records lustre-MDT0001: clear the changelog for cl11 of all records lustre-MDT0000: clear the changelog for cl11 of all records lustre-MDT0000: Deregistered changelog user #11 lustre-MDT0001: clear the changelog for cl11 of all records lustre-MDT0001: Deregistered changelog user #11 lustre-MDT0001: changelog user 'cl11' not found lustre-MDT0000: changelog user 'cl11' not found PASS 6 (8s) == lustre-rsync-test test 7: lustre_rsync stripesize ===== 05:28:25 (1730802505) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl12 cl12' lustre-MDT0000: Registered changelog user cl12 total: 100 open/close in 0.31 seconds: 319.10 ops/second Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /mnt/lustre/tgt Statuslog: /tmp/lustre_rsync.log Changelog registration: cl12 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 6 seconds Changelog records consumed: 202 lustre-MDT0000: clear the changelog for cl12 of all records lustre-MDT0001: clear the changelog for cl12 of all records lustre-MDT0000: clear the changelog for cl12 of all records lustre-MDT0000: Deregistered changelog user #12 lustre-MDT0001: clear the changelog for cl12 of all records lustre-MDT0001: Deregistered changelog user #12 lustre-MDT0001: changelog user 'cl12' not found lustre-MDT0000: changelog user 'cl12' not found PASS 7 (15s) == lustre-rsync-test test 8: Replicate multiple file/directory moves ========================================================== 05:28:40 (1730802520) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl13 cl13' lustre-MDT0000: Registered changelog user cl13 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl13 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 26 seconds Changelog records consumed: 3501 lustre-MDT0000: clear the changelog for cl13 of all records lustre-MDT0001: clear the changelog for cl13 of all records lustre-MDT0000: clear the changelog for cl13 of all records lustre-MDT0000: Deregistered changelog user #13 lustre-MDT0001: clear the changelog for cl13 of all records lustre-MDT0001: Deregistered changelog user #13 lustre-MDT0001: changelog user 'cl13' not found lustre-MDT0000: changelog user 'cl13' not found PASS 8 (56s) == lustre-rsync-test test 9: Replicate recursive directory removal ========================================================== 05:29:36 (1730802576) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl14 cl14' lustre-MDT0000: Registered changelog user cl14 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl14 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 0 seconds Changelog records consumed: 3 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl14 Starting changelog record: 55938 Clear changelog after use: no Errors: 0 lustre_rsync took 0 seconds Changelog records consumed: 3 lustre-MDT0000: clear the changelog for cl14 of all records lustre-MDT0001: clear the changelog for cl14 of all records lustre-MDT0000: clear the changelog for cl14 of all records lustre-MDT0000: Deregistered changelog user #14 lustre-MDT0001: clear the changelog for cl14 of all records lustre-MDT0001: Deregistered changelog user #14 lustre-MDT0001: changelog user 'cl14' not found lustre-MDT0000: changelog user 'cl14' not found PASS 9 (6s) == lustre-rsync-test test complete, duration 768 sec ===== 05:29:42 (1730802582) === lustre-rsync-test: start cleanup 05:29:42 (1730802582) === === lustre-rsync-test: finish cleanup 05:29:43 (1730802583) ===