== lustre-rsync-test test 2c: Replicate files while dbench is running. ========================================================== 16:33:47 (1746131627) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl4' lustre-MDT0000: Registered changelog user cl4 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2c.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2c.lustre-rsync-test at Thu May 1 16:33:51 EDT 2025 waiting for dbench pid 12188 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs 2 of 2 processes prepared for launch 0 sec releasing clients 2 147 12.56 MB/sec warmup 1 sec latency 72.760 ms 2 251 9.53 MB/sec warmup 2 sec latency 58.477 ms 2 345 8.26 MB/sec warmup 3 sec latency 86.291 ms 2 481 8.41 MB/sec warmup 4 sec latency 69.388 ms 2 665 8.46 MB/sec warmup 5 sec latency 133.952 ms 2 721 7.10 MB/sec warmup 6 sec latency 145.705 ms 2 792 6.13 MB/sec warmup 7 sec latency 169.706 ms 2 879 5.41 MB/sec warmup 8 sec latency 154.306 ms Running lustre_rsync 2 947 4.84 MB/sec warmup 9 sec latency 162.134 ms 2 1074 4.58 MB/sec warmup 10 sec latency 163.435 ms 2 1194 4.19 MB/sec warmup 11 sec latency 46.362 ms 2 1467 4.47 MB/sec execute 1 sec latency 72.623 ms 2 1569 2.32 MB/sec execute 2 sec latency 197.817 ms 2 1686 1.61 MB/sec execute 3 sec latency 158.725 ms 2 1820 1.27 MB/sec execute 4 sec latency 58.679 ms 2 1924 1.06 MB/sec execute 5 sec latency 122.695 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 9 seconds Changelog records consumed: 394 2 2082 0.94 MB/sec execute 6 sec latency 36.490 ms 2 2235 0.85 MB/sec execute 7 sec latency 48.762 ms 2 2389 1.04 MB/sec execute 8 sec latency 151.601 ms 2 2487 1.09 MB/sec execute 9 sec latency 111.373 ms 2 2766 1.63 MB/sec execute 10 sec latency 49.478 ms 12188 12191 12192 Running lustre_rsync 2 2980 1.76 MB/sec execute 11 sec latency 138.658 ms 2 3209 1.75 MB/sec execute 12 sec latency 138.603 ms 2 3495 2.24 MB/sec execute 13 sec latency 91.452 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 4319 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 759 2 3659 2.12 MB/sec execute 14 sec latency 43.535 ms 2 3802 2.13 MB/sec execute 15 sec latency 54.754 ms 2 3904 2.03 MB/sec execute 16 sec latency 40.453 ms 2 3998 1.91 MB/sec execute 17 sec latency 60.824 ms 2 4114 1.81 MB/sec execute 18 sec latency 54.656 ms 12188 12191 12192 Running lustre_rsync 2 4241 1.73 MB/sec execute 19 sec latency 233.735 ms 2 4295 1.66 MB/sec execute 20 sec latency 151.246 ms 2 4362 1.60 MB/sec execute 21 sec latency 159.170 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 5077 Clear changelog after use: no Errors: 0 lustre_rsync took 3 seconds Changelog records consumed: 563 2 4452 1.54 MB/sec execute 22 sec latency 130.178 ms 2 4550 1.50 MB/sec execute 23 sec latency 108.185 ms 2 4699 1.52 MB/sec execute 24 sec latency 81.027 ms 2 4811 1.47 MB/sec execute 25 sec latency 67.485 ms 2 4994 1.64 MB/sec execute 26 sec latency 50.919 ms 12188 12191 12192 Running lustre_rsync 2 5080 1.58 MB/sec execute 27 sec latency 228.615 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 5639 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 323 2 5193 1.53 MB/sec execute 28 sec latency 47.900 ms 2 5311 1.49 MB/sec execute 29 sec latency 159.622 ms 2 5421 1.44 MB/sec execute 30 sec latency 76.208 ms 2 5550 1.40 MB/sec execute 31 sec latency 220.072 ms 2 5711 1.37 MB/sec execute 32 sec latency 54.480 ms 12188 12191 12192 Running lustre_rsync 2 5863 1.40 MB/sec execute 33 sec latency 48.979 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 5961 Clear changelog after use: no Errors: 0 lustre_rsync took 0 seconds Changelog records consumed: 361 2 5969 1.37 MB/sec execute 34 sec latency 177.803 ms 2 6053 1.37 MB/sec execute 35 sec latency 144.470 ms 2 6327 1.51 MB/sec execute 36 sec latency 123.717 ms 2 6537 1.56 MB/sec execute 37 sec latency 139.821 ms 2 6742 1.56 MB/sec execute 38 sec latency 171.362 ms 12188 12191 12192 Running lustre_rsync 2 6907 1.57 MB/sec execute 39 sec latency 152.610 ms 2 7108 1.69 MB/sec execute 40 sec latency 46.834 ms 2 7204 1.66 MB/sec execute 41 sec latency 65.586 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 6321 Clear changelog after use: no Errors: 0 lustre_rsync took 3 seconds Changelog records consumed: 251 2 7318 1.67 MB/sec execute 42 sec latency 72.274 ms 2 7393 1.63 MB/sec execute 43 sec latency 53.444 ms 2 7473 1.61 MB/sec execute 44 sec latency 54.165 ms 2 7568 1.57 MB/sec execute 45 sec latency 53.272 ms 2 7650 1.54 MB/sec execute 46 sec latency 77.565 ms 12188 12191 12192 Running lustre_rsync 2 7756 1.51 MB/sec execute 47 sec latency 65.762 ms 2 7786 1.48 MB/sec execute 48 sec latency 369.393 ms 2 7834 1.46 MB/sec execute 49 sec latency 185.029 ms 2 7889 1.43 MB/sec execute 50 sec latency 226.948 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 6571 Clear changelog after use: no Errors: 0 lustre_rsync took 4 seconds Changelog records consumed: 483 2 7937 1.41 MB/sec execute 51 sec latency 163.565 ms 2 8016 1.39 MB/sec execute 52 sec latency 127.007 ms 2 8080 1.37 MB/sec execute 53 sec latency 184.640 ms 2 8184 1.38 MB/sec execute 54 sec latency 174.432 ms 2 8299 1.36 MB/sec execute 55 sec latency 50.509 ms 12188 12191 12192 Running lustre_rsync 2 8379 1.34 MB/sec execute 56 sec latency 53.780 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 7053 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 262 2 8571 1.42 MB/sec execute 57 sec latency 45.737 ms 2 8663 1.40 MB/sec execute 58 sec latency 239.181 ms 2 8736 1.38 MB/sec execute 59 sec latency 91.393 ms 2 cleanup 60 sec 2 cleanup 61 sec 0 cleanup 62 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 2625 22.515 77.554 Close 1924 5.506 42.056 Rename 106 34.491 66.226 Unlink 506 9.737 57.432 Qpathinfo 2400 5.481 76.179 Qfileinfo 420 0.389 8.432 Qfsinfo 408 0.467 14.900 Sfileinfo 200 22.009 68.899 Find 902 1.145 49.222 WriteX 1278 2.797 47.396 ReadX 4067 0.047 4.485 LockX 8 5.279 10.654 UnlockX 8 4.994 8.873 Flush 178 105.277 369.360 Throughput 1.37531 MB/sec 2 clients 2 procs max_latency=369.393 ms stopping dbench in /mnt/lustre/d2c.lustre-rsync-test at Thu May 1 16:35:05 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2c.lustre-rsync-test /mnt/lustre/d2c.lustre-rsync-test /mnt/lustre/d2c.lustre-rsync-test Running lustre_rsync removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2c.lustre-rsync-test dbench successfully finished Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 7314 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 338 Only in /tmp/target/d2c.lustre-rsync-test: clients Only in /tmp/target/d2c.lustre-rsync-test: client.txt Only in /tmp/target/d2c.lustre-rsync-test: dbench Only in /tmp/target/d2c.lustre-rsync-test/lib64: ld-linux-x86-64.so.2 Only in /tmp/target/d2c.lustre-rsync-test/lib64: libc.so.6 Only in /tmp/target/d2c.lustre-rsync-test/lib64: libpopt.so.0 lustre-rsync-test test_2c: @@@@@@ FAIL: Failure in replication; differences found. Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/lustre-rsync-test.sh:115:check_diff() = /home/green/git/lustre-release/lustre/tests/lustre-rsync-test.sh:373:test_2c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/lustre-rsync-test.sh:380:main() Dumping lctl log to /tmp/testlogs//lustre-rsync-test.test_2c.*.1746131715.log lustre-MDT0000: clear the changelog for cl4 of all records rsync: chown "/tmp/testlogs/.lustre-rsync-test.test_2c.debug_log.oleg117-server.1746131715.log.VeKGS6" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.lustre-rsync-test.test_2c.dmesg.oleg117-server.1746131715.log.FAcMWT" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] lustre-MDT0000: Deregistered changelog user #4