-----============= acceptance-small: lustre-rsync-test ============----- Mon May 13 03:59:47 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 2b === lustre-rsync-test: start setup 03:59:51 (1715587191) === oleg232-client.virtnet: executing check_config_client /mnt/lustre oleg232-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg232-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b424f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b424f000.idle_timeout=debug disable quota as required oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === lustre-rsync-test: finish setup 03:59:57 (1715587197) === nobody:x:99: debug_raw_pointers=Y debug_raw_pointers=Y == lustre-rsync-test test 1A: Simple Replication ========= 03:59:58 (1715587198) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl1' lustre-MDT0000: Registered changelog user cl1 Replication #1 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 0 seconds Changelog records consumed: 26 Replication #2 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 26 Clear changelog after use: no Errors: 0 lustre_rsync took 0 seconds Changelog records consumed: 3 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0000: changelog user 'cl1' not found PASS 1A (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == lustre-rsync-test test 1a: Replicate create/delete operations in ROOT directory ========================================================== 04:00:04 (1715587204) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl2' lustre-MDT0000: Registered changelog user cl2 Replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 0 seconds Changelog records consumed: 11 File: '/tmp/target/root-dir' Size: 40 Blocks: 0 IO Block: 4096 directory Device: 27h/39d Inode: 33256 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-05-13 04:00:05.000000000 -0400 Modify: 2024-05-13 04:00:05.000000000 -0400 Change: 2024-05-13 04:00:06.108000000 -0400 Birth: - File: '/tmp/target/root-file' Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: 27h/39d Inode: 33998 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-05-13 04:00:06.000000000 -0400 Modify: 2024-05-13 04:00:05.000000000 -0400 Change: 2024-05-13 04:00:06.133000000 -0400 Birth: - stat: cannot stat '/tmp/target/root-file2': No such file or directory File: '/tmp/target/root-file3' Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: 27h/39d Inode: 33257 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-05-13 04:00:06.000000000 -0400 Modify: 2024-05-13 04:00:05.000000000 -0400 Change: 2024-05-13 04:00:06.157000000 -0400 Birth: - stat: cannot stat '/tmp/target/root-dir1': No such file or directory stat: cannot stat '/tmp/target/root-file4': No such file or directory lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 lustre-MDT0000: changelog user 'cl2' not found PASS 1a (4s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == lustre-rsync-test test 2a: Replicate files created by dbench. ========================================================== 04:00:10 (1715587210) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl3' lustre-MDT0000: Registered changelog user cl3 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2a.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2a.lustre-rsync-test at Mon May 13 04:00:12 EDT 2024 waiting for dbench pid 10549 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs failed to create barrier semaphore 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 322 23.48 MB/sec warmup 1 sec latency 26.346 ms 2 673 21.17 MB/sec warmup 2 sec latency 57.961 ms 2 916 14.51 MB/sec warmup 3 sec latency 46.615 ms 2 1493 13.02 MB/sec warmup 4 sec latency 40.581 ms 2 2010 10.61 MB/sec warmup 5 sec latency 35.064 ms 2 2477 9.57 MB/sec warmup 6 sec latency 44.608 ms 2 3355 10.11 MB/sec warmup 7 sec latency 47.150 ms 2 3963 10.01 MB/sec warmup 8 sec latency 14.723 ms 2 4365 9.02 MB/sec warmup 9 sec latency 73.604 ms 2 4818 8.44 MB/sec warmup 10 sec latency 34.591 ms 2 5386 8.26 MB/sec warmup 11 sec latency 40.802 ms 2 6764 12.62 MB/sec execute 1 sec latency 42.457 ms 2 7467 12.09 MB/sec execute 2 sec latency 17.632 ms 2 7891 8.34 MB/sec execute 3 sec latency 50.241 ms 2 8507 8.58 MB/sec execute 4 sec latency 27.011 ms 2 9119 7.07 MB/sec execute 5 sec latency 36.797 ms 2 9868 7.70 MB/sec execute 6 sec latency 33.029 ms 2 10806 8.55 MB/sec execute 7 sec latency 36.739 ms 2 11371 7.87 MB/sec execute 8 sec latency 43.197 ms 2 11900 7.41 MB/sec execute 9 sec latency 49.949 ms 2 12622 7.34 MB/sec execute 10 sec latency 36.719 ms 2 13421 7.68 MB/sec execute 11 sec latency 32.794 ms 2 14431 8.34 MB/sec execute 12 sec latency 29.198 ms 2 14928 7.80 MB/sec execute 13 sec latency 53.707 ms 2 15385 7.49 MB/sec execute 14 sec latency 45.207 ms 2 16078 7.43 MB/sec execute 15 sec latency 29.042 ms 2 16828 7.35 MB/sec execute 16 sec latency 34.745 ms 2 17956 8.13 MB/sec execute 17 sec latency 30.856 ms 2 18477 7.76 MB/sec execute 18 sec latency 57.145 ms 2 19001 7.54 MB/sec execute 19 sec latency 30.983 ms 2 19581 7.48 MB/sec execute 20 sec latency 46.296 ms 2 20208 7.30 MB/sec execute 21 sec latency 52.386 ms 2 21143 7.64 MB/sec execute 22 sec latency 32.784 ms 2 21797 7.71 MB/sec execute 23 sec latency 9.073 ms 2 22248 7.45 MB/sec execute 24 sec latency 58.353 ms 2 22949 7.51 MB/sec execute 25 sec latency 35.317 ms 2 23702 7.36 MB/sec execute 26 sec latency 27.758 ms 2 24832 7.85 MB/sec execute 27 sec latency 30.426 ms 2 25547 7.71 MB/sec execute 28 sec latency 7.883 ms 2 26062 7.57 MB/sec execute 29 sec latency 58.251 ms 2 26883 7.55 MB/sec execute 30 sec latency 36.590 ms 2 27866 7.76 MB/sec execute 31 sec latency 29.166 ms 2 28818 7.93 MB/sec execute 32 sec latency 23.758 ms 2 29385 7.74 MB/sec execute 33 sec latency 54.588 ms 2 30193 7.78 MB/sec execute 34 sec latency 44.958 ms 2 31061 7.75 MB/sec execute 35 sec latency 29.059 ms 2 32185 8.09 MB/sec execute 36 sec latency 21.048 ms 2 32680 7.90 MB/sec execute 37 sec latency 67.912 ms 2 33085 7.79 MB/sec execute 38 sec latency 37.344 ms 2 33663 7.76 MB/sec execute 39 sec latency 69.286 ms 2 34342 7.65 MB/sec execute 40 sec latency 30.341 ms 2 35220 7.81 MB/sec execute 41 sec latency 32.708 ms 2 35903 7.86 MB/sec execute 42 sec latency 19.089 ms 2 36316 7.70 MB/sec execute 43 sec latency 53.531 ms 2 36777 7.60 MB/sec execute 44 sec latency 37.613 ms 2 37398 7.58 MB/sec execute 45 sec latency 42.516 ms 2 37986 7.52 MB/sec execute 46 sec latency 33.843 ms 2 38894 7.64 MB/sec execute 47 sec latency 36.600 ms 2 39541 7.68 MB/sec execute 48 sec latency 8.840 ms 2 39983 7.55 MB/sec execute 49 sec latency 54.985 ms 2 40694 7.58 MB/sec execute 50 sec latency 30.575 ms 2 41406 7.50 MB/sec execute 51 sec latency 34.595 ms 2 42082 7.57 MB/sec execute 52 sec latency 39.902 ms 2 42926 7.66 MB/sec execute 53 sec latency 28.797 ms 2 43341 7.54 MB/sec execute 54 sec latency 75.571 ms 2 43843 7.47 MB/sec execute 55 sec latency 30.028 ms 2 44490 7.46 MB/sec execute 56 sec latency 37.056 ms 2 45312 7.52 MB/sec execute 57 sec latency 28.591 ms 2 46240 7.62 MB/sec execute 58 sec latency 35.414 ms 2 46768 7.54 MB/sec execute 59 sec latency 9.605 ms 2 cleanup 60 sec 0 cleanup 60 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 14202 3.956 11.280 Close 10362 0.953 3.264 Rename 604 5.722 10.734 Unlink 2928 1.765 3.997 Qpathinfo 12790 1.035 8.139 Qfileinfo 2232 0.127 1.314 Qfsinfo 2402 0.083 1.701 Sfileinfo 1148 2.963 6.329 Find 4993 0.343 7.082 WriteX 7012 0.620 7.753 ReadX 22502 0.017 0.839 LockX 48 0.774 1.083 UnlockX 48 0.815 1.935 Flush 1012 20.899 75.563 Throughput 7.54148 MB/sec 2 clients 2 procs max_latency=75.571 ms stopping dbench in /mnt/lustre/d2a.lustre-rsync-test at Mon May 13 04:01:25 EDT 2024 with return code 0 clean dbench files in /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2a.lustre-rsync-test dbench successfully finished Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl3 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 10 seconds Changelog records consumed: 19198 lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0000: Deregistered changelog user #3 lustre-MDT0000: changelog user 'cl3' not found PASS 2a (88s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: lustre-rsync-test test_2b skipping ALWAYS excluded test 2b debug_raw_pointers=Y debug_raw_pointers=Y == lustre-rsync-test test 2c: Replicate files while dbench is running. ========================================================== 04:01:40 (1715587300) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl4' lustre-MDT0000: Registered changelog user cl4 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2c.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2c.lustre-rsync-test at Mon May 13 04:01:42 EDT 2024 waiting for dbench pid 11699 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 636 42.11 MB/sec warmup 1 sec latency 12.003 ms 2 989 21.91 MB/sec warmup 2 sec latency 66.804 ms 2 1689 17.48 MB/sec warmup 3 sec latency 33.711 ms 2 2376 13.98 MB/sec warmup 4 sec latency 37.758 ms 2 3297 14.14 MB/sec warmup 5 sec latency 32.419 ms 2 3944 13.35 MB/sec warmup 6 sec latency 20.536 ms 2 4363 11.59 MB/sec warmup 7 sec latency 67.437 ms 2 4984 11.28 MB/sec warmup 8 sec latency 37.360 ms 2 5573 10.14 MB/sec warmup 9 sec latency 29.875 ms Running lustre_rsync 2 6325 10.22 MB/sec warmup 10 sec latency 48.619 ms 2 7183 10.50 MB/sec warmup 11 sec latency 44.291 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 4 seconds Changelog records consumed: 2412 2 8029 1.63 MB/sec execute 1 sec latency 55.928 ms 2 8742 5.22 MB/sec execute 2 sec latency 30.323 ms 2 9326 3.88 MB/sec execute 3 sec latency 28.620 ms 2 10033 6.26 MB/sec execute 4 sec latency 38.259 ms 2 10896 7.52 MB/sec execute 5 sec latency 45.268 ms 11699 11700 11701 Running lustre_rsync 2 11381 6.48 MB/sec execute 6 sec latency 53.085 ms 2 11863 6.06 MB/sec execute 7 sec latency 35.649 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 21650 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 2261 2 12450 6.12 MB/sec execute 8 sec latency 42.032 ms 2 13061 5.82 MB/sec execute 9 sec latency 31.643 ms 2 13927 6.70 MB/sec execute 10 sec latency 35.560 ms 2 14583 6.95 MB/sec execute 11 sec latency 25.850 ms 2 15055 6.48 MB/sec execute 12 sec latency 46.236 ms 11699 11700 11701 Running lustre_rsync Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 23910 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 1487 2 15776 6.68 MB/sec execute 13 sec latency 33.280 ms 2 16479 6.39 MB/sec execute 14 sec latency 23.723 ms 2 17204 6.78 MB/sec execute 15 sec latency 36.839 ms 2 18003 7.13 MB/sec execute 16 sec latency 29.375 ms 2 18449 6.78 MB/sec execute 17 sec latency 52.922 ms 11699 11700 11701 Running lustre_rsync 2 18781 6.59 MB/sec execute 18 sec latency 46.709 ms 2 19393 6.59 MB/sec execute 19 sec latency 46.616 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 25396 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 1471 2 19989 6.32 MB/sec execute 20 sec latency 32.685 ms 2 20709 6.66 MB/sec execute 21 sec latency 47.692 ms 2 21549 6.93 MB/sec execute 22 sec latency 28.310 ms 2 22010 6.67 MB/sec execute 23 sec latency 66.014 ms 2 22334 6.47 MB/sec execute 24 sec latency 43.808 ms 11699 11700 11701 Running lustre_rsync 2 22905 6.54 MB/sec execute 25 sec latency 62.083 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 26866 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 1465 2 23432 6.33 MB/sec execute 26 sec latency 36.654 ms 2 23927 6.27 MB/sec execute 27 sec latency 38.166 ms 2 24727 6.52 MB/sec execute 28 sec latency 39.957 ms 2 25234 6.60 MB/sec execute 29 sec latency 12.916 ms 2 25616 6.40 MB/sec execute 30 sec latency 78.681 ms 11699 11700 11701 Running lustre_rsync 2 25946 6.30 MB/sec execute 31 sec latency 46.242 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 28330 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 1325 2 26481 6.30 MB/sec execute 32 sec latency 40.505 ms 2 26943 6.14 MB/sec execute 33 sec latency 39.572 ms 2 27626 6.27 MB/sec execute 34 sec latency 38.190 ms 2 28502 6.48 MB/sec execute 35 sec latency 37.746 ms 2 28977 6.38 MB/sec execute 36 sec latency 11.287 ms 11699 11700 11701 Running lustre_rsync 2 29307 6.24 MB/sec execute 37 sec latency 54.860 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 29654 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 1409 2 29853 6.31 MB/sec execute 38 sec latency 42.284 ms 2 30313 6.16 MB/sec execute 39 sec latency 38.546 ms 2 30867 6.10 MB/sec execute 40 sec latency 43.189 ms 2 31778 6.31 MB/sec execute 41 sec latency 33.371 ms 2 32418 6.37 MB/sec execute 42 sec latency 8.841 ms 11699 11700 11701 Running lustre_rsync 2 32848 6.26 MB/sec execute 43 sec latency 62.122 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 31062 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 1441 2 33468 6.32 MB/sec execute 44 sec latency 32.006 ms 2 33994 6.20 MB/sec execute 45 sec latency 32.808 ms 2 34731 6.30 MB/sec execute 46 sec latency 42.440 ms 2 35544 6.44 MB/sec execute 47 sec latency 43.253 ms 2 36078 6.38 MB/sec execute 48 sec latency 9.008 ms 11699 11700 11701 Running lustre_rsync 2 36467 6.28 MB/sec execute 49 sec latency 57.400 ms 2 37001 6.32 MB/sec execute 50 sec latency 46.462 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 32502 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 1487 2 37424 6.22 MB/sec execute 51 sec latency 43.232 ms 2 37963 6.16 MB/sec execute 52 sec latency 44.172 ms 2 38866 6.32 MB/sec execute 53 sec latency 32.651 ms 2 39459 6.37 MB/sec execute 54 sec latency 10.599 ms 2 39856 6.28 MB/sec execute 55 sec latency 52.319 ms 11699 11700 11701 Running lustre_rsync 2 40307 6.22 MB/sec execute 56 sec latency 38.984 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 33988 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 1500 2 40929 6.23 MB/sec execute 57 sec latency 36.339 ms 2 41623 6.21 MB/sec execute 58 sec latency 33.871 ms 2 42650 6.43 MB/sec execute 59 sec latency 29.732 ms 2 cleanup 60 sec 0 cleanup 60 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 12298 4.661 19.926 Close 9040 1.106 6.504 Rename 520 6.485 13.667 Unlink 2526 1.983 6.112 Qpathinfo 11208 1.178 16.697 Qfileinfo 1940 0.143 1.394 Qfsinfo 2042 0.092 1.512 Sfileinfo 1000 3.411 8.604 Find 4319 0.383 8.563 WriteX 6062 0.716 10.200 ReadX 19274 0.020 0.813 LockX 40 0.975 1.491 UnlockX 40 0.974 1.322 Flush 860 23.536 78.667 Throughput 6.42979 MB/sec 2 clients 2 procs max_latency=78.681 ms stopping dbench in /mnt/lustre/d2c.lustre-rsync-test at Mon May 13 04:02:54 EDT 2024 with return code 0 clean dbench files in /mnt/lustre/d2c.lustre-rsync-test /mnt/lustre/d2c.lustre-rsync-test /mnt/lustre/d2c.lustre-rsync-test removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2c.lustre-rsync-test dbench successfully finished Running lustre_rsync Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 35487 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 1368 lustre-MDT0000: clear the changelog for cl4 of all records lustre-MDT0000: clear the changelog for cl4 of all records lustre-MDT0000: Deregistered changelog user #4 lustre-MDT0000: changelog user 'cl4' not found PASS 2c (84s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == lustre-rsync-test test 3a: Replicate files created by createmany ========================================================== 04:03:06 (1715587386) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl5' lustre-MDT0000: Registered changelog user cl5 total: 1000 open/close in 2.95 seconds: 339.32 ops/second Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl5 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 15 seconds Changelog records consumed: 2000 lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0000: Deregistered changelog user #5 lustre-MDT0000: changelog user 'cl5' not found PASS 3a (26s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == lustre-rsync-test test 3b: Replicate files created by writemany ========================================================== 04:03:33 (1715587413) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl6' lustre-MDT0000: Registered changelog user cl6 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl6 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 34 seconds Changelog records consumed: 42033 lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0000: Deregistered changelog user #6 lustre-MDT0000: changelog user 'cl6' not found PASS 3b (100s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == lustre-rsync-test test 3c: Replicate files created by createmany/unlinkmany ========================================================== 04:05:14 (1715587514) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl7' lustre-MDT0000: Registered changelog user cl7 total: 1000 open/close in 2.32 seconds: 430.38 ops/second - unlinked 0 (time 1715587519 ; total 0 ; last 0) total: 1000 unlinks in 1 seconds: 1000.000000 unlinks/second Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl7 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 3000 lustre-MDT0000: clear the changelog for cl7 of all records lustre-MDT0000: clear the changelog for cl7 of all records lustre-MDT0000: Deregistered changelog user #7 lustre-MDT0000: changelog user 'cl7' not found PASS 3c (10s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == lustre-rsync-test test 4: Replicate files created by iozone ========================================================== 04:05:26 (1715587526) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl8' lustre-MDT0000: Registered changelog user cl8 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl8 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 625 /home/green/git/lustre-release/lustre/tests/lustre-rsync-test.sh: line 450: 15971 Killed END_RUN_FILE=${DIR}/$tdir/run LOAD_PID_FILE=${DIR}/$tdir/pid MOUNT=${DIR}/$tdir run_iozone.sh Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl8 Starting changelog record: 84512 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 1248 lustre-MDT0000: clear the changelog for cl8 of all records lustre-MDT0000: clear the changelog for cl8 of all records lustre-MDT0000: Deregistered changelog user #8 lustre-MDT0000: changelog user 'cl8' not found PASS 4 (96s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == lustre-rsync-test test 5a: Stop / start lustre_rsync == 04:07:03 (1715587623) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl9' lustre-MDT0000: Registered changelog user cl9 total: 2000 open/close in 4.38 seconds: 457.13 ops/second Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl9 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 24 seconds Changelog records consumed: 4000 kill: sending signal to 17162 failed: No such process Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl9 Starting changelog record: 89759 Clear changelog after use: no Errors: 0 lustre_rsync took 0 seconds Changelog records consumed: 1 lustre-MDT0000: clear the changelog for cl9 of all records lustre-MDT0000: clear the changelog for cl9 of all records lustre-MDT0000: Deregistered changelog user #9 lustre-MDT0000: changelog user 'cl9' not found PASS 5a (46s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == lustre-rsync-test test 5b: Kill / restart lustre_rsync ========================================================== 04:07:50 (1715587670) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl10' lustre-MDT0000: Registered changelog user cl10 total: 2000 open/close in 4.46 seconds: 448.17 ops/second Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl10 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 24 seconds Changelog records consumed: 4000 kill: sending signal to 18242 failed: No such process Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl10 Starting changelog record: 93759 Clear changelog after use: no Errors: 0 lustre_rsync took 0 seconds Changelog records consumed: 1 lustre-MDT0000: clear the changelog for cl10 of all records lustre-MDT0000: clear the changelog for cl10 of all records lustre-MDT0000: Deregistered changelog user #10 lustre-MDT0000: changelog user 'cl10' not found PASS 5b (45s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == lustre-rsync-test test 6: lustre_rsync large no of hard links ========================================================== 04:08:37 (1715587717) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl11' lustre-MDT0000: Registered changelog user cl11 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl11 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 0 seconds Changelog records consumed: 129 lustre-MDT0000: clear the changelog for cl11 of all records lustre-MDT0000: clear the changelog for cl11 of all records lustre-MDT0000: Deregistered changelog user #11 lustre-MDT0000: changelog user 'cl11' not found PASS 6 (5s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == lustre-rsync-test test 7: lustre_rsync stripesize ===== 04:08:44 (1715587724) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl12' lustre-MDT0000: Registered changelog user cl12 total: 100 open/close in 0.28 seconds: 361.37 ops/second Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /mnt/lustre/tgt Statuslog: /tmp/lustre_rsync.log Changelog registration: cl12 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 7 seconds Changelog records consumed: 202 lustre-MDT0000: clear the changelog for cl12 of all records lustre-MDT0000: clear the changelog for cl12 of all records lustre-MDT0000: Deregistered changelog user #12 lustre-MDT0000: changelog user 'cl12' not found PASS 7 (12s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == lustre-rsync-test test 8: Replicate multiple file/directory moves ========================================================== 04:08:58 (1715587738) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl13' lustre-MDT0000: Registered changelog user cl13 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl13 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 16 seconds Changelog records consumed: 3501 lustre-MDT0000: clear the changelog for cl13 of all records lustre-MDT0000: clear the changelog for cl13 of all records lustre-MDT0000: Deregistered changelog user #13 lustre-MDT0000: changelog user 'cl13' not found PASS 8 (34s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == lustre-rsync-test test 9: Replicate recursive directory removal ========================================================== 04:09:33 (1715587773) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl14' lustre-MDT0000: Registered changelog user cl14 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl14 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 0 seconds Changelog records consumed: 3 Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl14 Starting changelog record: 98384 Clear changelog after use: no Errors: 0 lustre_rsync took 0 seconds Changelog records consumed: 3 lustre-MDT0000: clear the changelog for cl14 of all records lustre-MDT0000: clear the changelog for cl14 of all records lustre-MDT0000: Deregistered changelog user #14 lustre-MDT0000: changelog user 'cl14' not found PASS 9 (4s) debug_raw_pointers=0 debug_raw_pointers=0 == lustre-rsync-test test complete, duration 590 sec ===== 04:09:37 (1715587777) === lustre-rsync-test: start cleanup 04:09:38 (1715587778) === === lustre-rsync-test: finish cleanup 04:09:38 (1715587778) ===