== lustre-rsync-test test 2c: Replicate files while dbench is running. ========================================================== 03:51:02 (1743493862) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl4 cl4' lustre-MDT0000: Registered changelog user cl4 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2c.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2c.lustre-rsync-test at Tue Apr 1 03:51:21 EDT 2025 waiting for dbench pid 13288 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs 2 of 2 processes prepared for launch 0 sec releasing clients 2 52 4.18 MB/sec warmup 1 sec latency 145.372 ms 2 120 5.43 MB/sec warmup 2 sec latency 160.196 ms 2 157 4.34 MB/sec warmup 3 sec latency 200.656 ms 2 197 3.90 MB/sec warmup 4 sec latency 156.575 ms Running lustre_rsync 2 234 3.56 MB/sec warmup 5 sec latency 149.625 ms 2 267 3.33 MB/sec warmup 6 sec latency 189.005 ms 2 300 3.15 MB/sec warmup 7 sec latency 214.260 ms 2 333 2.99 MB/sec warmup 8 sec latency 168.532 ms 2 374 2.94 MB/sec warmup 9 sec latency 139.210 ms 2 437 3.06 MB/sec warmup 10 sec latency 111.977 ms 2 485 3.07 MB/sec warmup 11 sec latency 117.206 ms 2 628 5.93 MB/sec execute 1 sec latency 148.512 ms 2 676 3.11 MB/sec execute 2 sec latency 171.120 ms 2 711 2.09 MB/sec execute 3 sec latency 121.106 ms 2 738 1.63 MB/sec execute 4 sec latency 130.143 ms 2 778 1.32 MB/sec execute 5 sec latency 148.708 ms 2 808 1.14 MB/sec execute 6 sec latency 128.787 ms 2 863 1.00 MB/sec execute 7 sec latency 120.955 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 14 seconds Changelog records consumed: 111 2 896 0.91 MB/sec execute 8 sec latency 114.731 ms 2 931 0.83 MB/sec execute 9 sec latency 114.078 ms 2 974 0.76 MB/sec execute 10 sec latency 130.912 ms 2 1041 0.88 MB/sec execute 11 sec latency 103.388 ms 2 1120 0.82 MB/sec execute 12 sec latency 78.081 ms 13288 13289 13290 Running lustre_rsync 2 1163 0.76 MB/sec execute 13 sec latency 142.525 ms 2 1217 0.71 MB/sec execute 14 sec latency 93.754 ms 2 1255 0.67 MB/sec execute 15 sec latency 98.426 ms 2 1387 0.99 MB/sec execute 16 sec latency 131.109 ms 2 1426 0.93 MB/sec execute 17 sec latency 120.929 ms 2 1481 0.89 MB/sec execute 18 sec latency 139.150 ms 2 1517 0.84 MB/sec execute 19 sec latency 114.409 ms 2 1575 0.81 MB/sec execute 20 sec latency 171.462 ms 2 1626 0.77 MB/sec execute 21 sec latency 101.055 ms 2 1668 0.74 MB/sec execute 22 sec latency 129.147 ms 2 1705 0.71 MB/sec execute 23 sec latency 135.734 ms 2 1741 0.68 MB/sec execute 24 sec latency 223.391 ms 2 1790 0.66 MB/sec execute 25 sec latency 92.080 ms 2 1831 0.64 MB/sec execute 26 sec latency 96.014 ms 2 1878 0.61 MB/sec execute 27 sec latency 126.916 ms 2 1972 0.60 MB/sec execute 28 sec latency 71.715 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 2102 Clear changelog after use: no Errors: 0 lustre_rsync took 15 seconds Changelog records consumed: 380 2 2008 0.58 MB/sec execute 29 sec latency 187.864 ms 2 2068 0.57 MB/sec execute 30 sec latency 118.891 ms 2 2139 0.55 MB/sec execute 31 sec latency 98.118 ms 2 2184 0.54 MB/sec execute 32 sec latency 92.533 ms 2 2259 0.53 MB/sec execute 33 sec latency 117.906 ms 13288 13289 13290 Running lustre_rsync 2 2345 0.58 MB/sec execute 34 sec latency 138.216 ms 2 2379 0.56 MB/sec execute 35 sec latency 106.166 ms 2 2417 0.56 MB/sec execute 36 sec latency 139.204 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 2481 Clear changelog after use: no Errors: 0 lustre_rsync took 3 seconds Changelog records consumed: 481 2 2462 0.56 MB/sec execute 37 sec latency 98.227 ms 2 2564 0.57 MB/sec execute 38 sec latency 96.211 ms 2 2618 0.56 MB/sec execute 39 sec latency 95.886 ms 2 2838 0.70 MB/sec execute 40 sec latency 104.958 ms 2 2953 0.75 MB/sec execute 41 sec latency 85.033 ms 2 3013 0.73 MB/sec execute 42 sec latency 100.727 ms 13288 13289 13290 Running lustre_rsync 2 3200 0.75 MB/sec execute 43 sec latency 104.695 ms 2 3330 0.78 MB/sec execute 44 sec latency 81.962 ms 2 3440 0.84 MB/sec execute 45 sec latency 100.566 ms 2 3552 0.88 MB/sec execute 46 sec latency 95.713 ms 2 3623 0.87 MB/sec execute 47 sec latency 91.450 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 2961 Clear changelog after use: no Errors: 0 lustre_rsync took 6 seconds Changelog records consumed: 213 2 3683 0.86 MB/sec execute 48 sec latency 91.915 ms 2 3735 0.86 MB/sec execute 49 sec latency 120.690 ms 2 3806 0.87 MB/sec execute 50 sec latency 91.032 ms 2 3861 0.85 MB/sec execute 51 sec latency 103.763 ms 2 3911 0.84 MB/sec execute 52 sec latency 95.702 ms 13288 13289 13290 Running lustre_rsync 2 3954 0.83 MB/sec execute 53 sec latency 150.657 ms 2 3987 0.81 MB/sec execute 54 sec latency 141.208 ms 2 4034 0.80 MB/sec execute 55 sec latency 100.797 ms 2 4087 0.78 MB/sec execute 56 sec latency 92.016 ms 2 4150 0.77 MB/sec execute 57 sec latency 68.178 ms 2 4197 0.76 MB/sec execute 58 sec latency 137.513 ms 2 4238 0.75 MB/sec execute 59 sec latency 289.336 ms Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 3173 Clear changelog after use: no Errors: 0 lustre_rsync took 6 seconds Changelog records consumed: 370 2 cleanup 60 sec 2 cleanup 61 sec 2 cleanup 62 sec 2 cleanup 63 sec 0 cleanup 63 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1262 48.937 289.319 Close 933 13.489 79.348 Rename 53 77.592 142.492 Unlink 252 22.209 70.857 Qpathinfo 1180 11.840 135.989 Qfileinfo 228 1.268 41.953 Qfsinfo 222 1.381 25.007 Sfileinfo 108 62.709 138.197 Find 446 4.865 108.472 WriteX 748 8.737 89.096 ReadX 1980 0.102 11.179 LockX 4 12.808 15.532 UnlockX 4 10.337 13.781 Flush 98 60.838 171.093 Throughput 0.750933 MB/sec 2 clients 2 procs max_latency=289.336 ms stopping dbench in /mnt/lustre/d2c.lustre-rsync-test at Tue Apr 1 03:52:37 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2c.lustre-rsync-test /mnt/lustre/d2c.lustre-rsync-test /mnt/lustre/d2c.lustre-rsync-test Running lustre_rsync removed directory 'clients/client1/~dmtmp/WORD' removed directory 'clients/client1/~dmtmp/PARADOX' removed directory 'clients/client1/~dmtmp/ACCESS' removed directory 'clients/client1/~dmtmp/PM' removed directory 'clients/client1/~dmtmp/PWRPNT' removed directory 'clients/client1/~dmtmp/EXCEL' removed directory 'clients/client1/~dmtmp/WORDPRO' removed directory 'clients/client1/~dmtmp/COREL' removed directory 'clients/client1/~dmtmp/SEED' removed directory 'clients/client1/~dmtmp' removed directory 'clients/client1' removed directory 'clients/client0/~dmtmp/WORD' removed directory 'clients/client0/~dmtmp/PARADOX' removed directory 'clients/client0/~dmtmp/ACCESS' removed directory 'clients/client0/~dmtmp/PM' removed directory 'clients/client0/~dmtmp/PWRPNT' removed directory 'clients/client0/~dmtmp/EXCEL' removed directory 'clients/client0/~dmtmp/WORDPRO' removed directory 'clients/client0/~dmtmp/COREL' removed directory 'clients/client0/~dmtmp/SEED' removed directory 'clients/client0/~dmtmp' removed directory 'clients/client0' removed directory 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2c.lustre-rsync-test dbench successfully finished Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 3542 Clear changelog after use: no Errors: 0 lustre_rsync took 4 seconds Changelog records consumed: 308 Only in /tmp/target/d2c.lustre-rsync-test: clients Only in /tmp/target/d2c.lustre-rsync-test: client.txt Only in /tmp/target/d2c.lustre-rsync-test: dbench Only in /tmp/target/d2c.lustre-rsync-test/lib64: ld-linux-x86-64.so.2 Only in /tmp/target/d2c.lustre-rsync-test/lib64: libc.so.6 Only in /tmp/target/d2c.lustre-rsync-test/lib64: libpopt.so.0 lustre-rsync-test test_2c: @@@@@@ FAIL: Failure in replication; differences found. Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7235:error() = /home/green/git/lustre-release/lustre/tests/lustre-rsync-test.sh:115:check_diff() = /home/green/git/lustre-release/lustre/tests/lustre-rsync-test.sh:373:test_2c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7608:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7671:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7489:run_test() = /home/green/git/lustre-release/lustre/tests/lustre-rsync-test.sh:380:main() Dumping lctl log to /tmp/testlogs//lustre-rsync-test.test_2c.*.1743493977.log rsync: chown "/tmp/testlogs/.lustre-rsync-test.test_2c.debug_log.oleg641-server.1743493977.log.zYTWAK" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.lustre-rsync-test.test_2c.dmesg.oleg641-server.1743493977.log.9cZgn0" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1670) [generator=3.1.3] lustre-MDT0001: clear the changelog for cl4 of all records lustre-MDT0001: Deregistered changelog user #4 lustre-MDT0000: clear the changelog for cl4 of all records lustre-MDT0000: Deregistered changelog user #4