== lustre-rsync-test test 2b: Replicate files changed by dbench. ========================================================== 08:06:53 (1711627613) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' lustre-MDT0000: Registered changelog user cl1 waiting child_pid=$(pgrep dbench), 360 secs left ... looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2b.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2b.lustre-rsync-test at Thu Mar 28 08:06:56 EDT 2024 waiting for dbench pid 9440 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs failed to create barrier semaphore 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 289 21.41 MB/sec warmup 1 sec latency 38.865 ms 2 640 21.06 MB/sec warmup 2 sec latency 23.432 ms 2 900 14.45 MB/sec warmup 3 sec latency 22.106 ms 2 1214 11.53 MB/sec warmup 4 sec latency 19.783 ms 2 1632 10.47 MB/sec warmup 5 sec latency 19.073 ms 2 2008 8.84 MB/sec warmup 6 sec latency 19.056 ms 2 2405 8.01 MB/sec warmup 7 sec latency 16.451 ms 2 2980 8.38 MB/sec warmup 8 sec latency 17.985 ms 2 3648 8.58 MB/sec warmup 9 sec latency 21.960 ms 2 3978 8.01 MB/sec warmup 10 sec latency 16.392 ms 2 4314 7.35 MB/sec warmup 11 sec latency 30.854 ms 2 5028 6.28 MB/sec execute 1 sec latency 22.084 ms 2 5251 3.33 MB/sec execute 2 sec latency 44.400 ms PIDs: 9440 9441 9442 Stopping dbench 9440 9441 9442 stopped Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 19 seconds Changelog records consumed: 2144 changelog record number: before lrsync 2144, after 2144 Resuming dbench 2 5261 0.29 MB/sec execute 23 sec latency 21267.134 ms 2 5375 0.28 MB/sec execute 24 sec latency 43.069 ms 2 5515 0.28 MB/sec execute 25 sec latency 54.507 ms 2 5695 0.29 MB/sec execute 26 sec latency 29.082 ms 2 5865 0.36 MB/sec execute 27 sec latency 59.239 ms 2 5995 0.37 MB/sec execute 28 sec latency 37.487 ms 2 6154 0.42 MB/sec execute 29 sec latency 43.902 ms 2 6495 0.70 MB/sec execute 30 sec latency 35.523 ms 2 6828 0.79 MB/sec execute 31 sec latency 51.435 ms 2 7136 0.96 MB/sec execute 32 sec latency 30.258 ms Stopping dbench 9440 9441 9442 stopped Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 2144 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 729 changelog record number: before lrsync 2872, after 2872 Files /mnt/lustre/d2b.lustre-rsync-test/clients/client0/~dmtmp/PARADOX/ANSWER.DB and /tmp/target/d2b.lustre-rsync-test/clients/client0/~dmtmp/PARADOX/ANSWER.DB differ Files /mnt/lustre/d2b.lustre-rsync-test/clients/client1/~dmtmp/PARADOX/ANSWER.DB and /tmp/target/d2b.lustre-rsync-test/clients/client1/~dmtmp/PARADOX/ANSWER.DB differ lustre-rsync-test test_2b: @@@@@@ FAIL: Failure in replication; differences found. Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6975:error() = /home/green/git/lustre-release/lustre/tests/lustre-rsync-test.sh:115:check_diff() = /home/green/git/lustre-release/lustre/tests/lustre-rsync-test.sh:358:test_2b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7315:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7186:run_test() = /home/green/git/lustre-release/lustre/tests/lustre-rsync-test.sh:387:main() Dumping lctl log to /tmp/testlogs//lustre-rsync-test.test_2b.*.1711627666.log 9440 9441 9442 /home/green/git/lustre-release/lustre/tests/rundbench: line 157: 9440 Killed $RUN dbench -c client.txt "$@" dbench killed by signal 9 stopping dbench in /mnt/lustre/d2b.lustre-rsync-test at Thu Mar 28 08:07:48 EDT 2024 with return code 0 clean dbench files in /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test lustre-MDT0001: clear the changelog for cl1 of all records rsync: chown "/tmp/testlogs/.lustre-rsync-test.test_2b.debug_log.oleg122-server.1711627666.log.xFYhqe" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.lustre-rsync-test.test_2b.dmesg.oleg122-server.1711627666.log.JpLVzP" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] lustre-MDT0001: Deregistered changelog user #1 removed 'clients/client0/filler.004' removed 'clients/client0/filler.003' removed 'clients/client0/filler.002' removed 'clients/client0/filler.001' removed 'clients/client0/~dmtmp/EXCEL/SALES.XLS' removed 'clients/client0/~dmtmp/EXCEL/SALES1.XLS' removed 'clients/client0/~dmtmp/EXCEL/RESULTS3.PRN' removed 'clients/client0/~dmtmp/EXCEL/RESULTS2.PRN' removed 'clients/client0/~dmtmp/EXCEL/PCMAGCD.XLS' removed 'clients/client0/~dmtmp/EXCEL/SALES1.PRN' removed 'clients/client0/~dmtmp/EXCEL/RESULTS.XLS' removed 'clients/client0/~dmtmp/EXCEL/RESULTS1.PRN' removed directory: 'clients/client0/~dmtmp/EXCEL' removed 'clients/client0/~dmtmp/ACCESS/INV.PRN' removed 'clients/client0/~dmtmp/ACCESS/FASTENER.MDB' removed 'clients/client0/~dmtmp/ACCESS/SALES.PRN' removed 'clients/client0/~dmtmp/ACCESS/LABELS.PRN' removed directory: 'clients/client0/~dmtmp/ACCESS' removed 'clients/client0/~dmtmp/COREL/TEAPOT2.CPT' removed 'clients/client0/~dmtmp/COREL/CDRBARS.CFG' removed 'clients/client0/~dmtmp/COREL/GRAPHIC1.CDR' removed 'clients/client0/~dmtmp/COREL/MAILBOX2.CPT' removed 'clients/client0/~dmtmp/COREL/GRAPH2.CDR' removed 'clients/client0/~dmtmp/COREL/CDRROLS.CFG' removed 'clients/client0/~dmtmp/COREL/ROSE.CPT' removed 'clients/client0/~dmtmp/COREL/GRAPH2.BAK' removed 'clients/client0/~dmtmp/COREL/TRAIN.CDR' removed 'clients/client0/~dmtmp/COREL/GRAPH1.CDR' removed 'clients/client0/~dmtmp/COREL/GRAPH1.BAK' removed 'clients/client0/~dmtmp/COREL/CORELDRW.CDT' removed 'clients/client0/~dmtmp/COREL/SIGN2.CPT' removed 'clients/client0/~dmtmp/COREL/ARTISTIC.ACL' removed 'clients/client0/~dmtmp/COREL/GRAPHIC2.CDR' removed directory: 'clients/client0/~dmtmp/COREL' removed 'clients/client0/~dmtmp/WORD/TIPS.DOC' removed 'clients/client0/~dmtmp/WORD/HANDOUT.DOC' removed 'clients/client0/~dmtmp/WORD/CHAP10.DOC' removed 'clients/client0/~dmtmp/WORD/FACTS.DOC' removed 'clients/client0/~dmtmp/WORD/BASEMACH.DOC' removed directory: 'clients/client0/~dmtmp/WORD' removed 'clients/client0/~dmtmp/PM/T1.XLS' removed 'clients/client0/~dmtmp/PM/EVENTS.DOC' removed 'clients/client0/~dmtmp/PM/MOVED.DOC' removed directory: 'clients/client0/~dmtmp/PM' removed 'clients/client0/~dmtmp/SEED/MEDIUM.FIL' removed 'clients/client0/~dmtmp/SEED/LARGE.FIL' removed 'clients/client0/~dmtmp/SEED/SMALL.FIL' removed directory: 'clients/client0/~dmtmp/SEED' removed 'clients/client0/~dmtmp/PWRPNT/TRIDOTS.POT' removed 'clients/client0/~dmtmp/PWRPNT/PPTOOLS1.PPA' removed 'clients/client0/~dmtmp/PWRPNT/NEWPCB.PPT' removed 'clients/client0/~dmtmp/PWRPNT/ZD16.BMP' removed 'clients/client0/~dmtmp/PWRPNT/NEWTIPS.PPT' removed 'clients/client0/~dmtmp/PWRPNT/PCBENCHM.PPT' removed 'clients/client0/~dmtmp/PWRPNT/TIPS.PPT' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed 'clients/client0/~dmtmp/PARADOX/COURSES.PX' removed 'clients/client0/~dmtmp/PARADOX/CHANGED.DB' removed 'clients/client0/~dmtmp/PARADOX/STUDENTS.XG0' removed 'clients/client0/~dmtmp/PARADOX/STUDENTS.YG1' removed 'clients/client0/~dmtmp/PARADOX/ENROLL.PX' removed 'clients/client0/~dmtmp/PARADOX/STUDENTS.MB' removed 'clients/client0/~dmtmp/PARADOX/STUDENTS.TV' removed 'clients/client0/~dmtmp/PARADOX/COURSES.Y04' removed 'clients/client0/~dmtmp/PARADOX/STUDENTS.YG0' removed 'clients/client0/~dmtmp/PARADOX/ANSWER.DB' removed 'clients/client0/~dmtmp/PARADOX/COURSES.VAL' removed 'clients/client0/~dmtmp/PARADOX/COURSES.X04' removed 'clients/client0/~dmtmp/PARADOX/ENROLL.VAL' removed 'clients/client0/~dmtmp/PARADOX/COURSES.DB' removed 'clients/client0/~dmtmp/PARADOX/COURSES.FSL' removed 'clients/client0/~dmtmp/PARADOX/ENROLL.Y03' removed 'clients/client0/~dmtmp/PARADOX/STUDENTS.PX' removed 'clients/client0/~dmtmp/PARADOX/ANSWER.MB' removed 'clients/client0/~dmtmp/PARADOX/STUDENTS.VAL' removed 'clients/client0/~dmtmp/PARADOX/ENROLL.X03' removed 'clients/client0/~dmtmp/PARADOX/STUDENTS.XG1' removed 'clients/client0/~dmtmp/PARADOX/STUDENTS.DB' removed 'clients/client0/~dmtmp/PARADOX/ENROLL.DB' removed 'clients/client0/~dmtmp/PARADOX/STUDENTS.FSL' removed directory: 'clients/client0/~dmtmp/PARADOX' removed 'clients/client0/~dmtmp/lustre-MDT0000: clear the changelog for cl1 of all records WORDPRO/BENCHS1.LWP' removed 'clients/client0/~dmtmp/WORDPRO/BENCHS1A.PRN' removed 'clients/client0/~dmtmp/WORDPRO/BENCHS.LWP' removed 'clients/client0/~dmtmp/WORDPRO/BENCHS1.PRN' removed 'clients/client0/~dmtmp/WORDPRO/NEWS1_1.LWP' removed 'clients/client0/~dmtmp/WORDPRO/NEWS1_1A.PRN' removed 'clients/client0/~dmtmp/WORDPRO/NEWS1_1A.LWP' removed 'clients/client0/~dmtmp/WORDPRO/RESULTS.XLS' removed 'clients/client0/~dmtmp/WORDPRO/NEWS1_1B.PRN' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp' removed 'clients/client0/filler.000' removed directory: 'clients/client0' removed 'clients/client1/filler.004' removed 'clients/client1/filler.003' removed 'clients/client1/filler.002' removed 'clients/client1/filler.001' removed 'clients/client1/~dmtmp/EXCEL/SALES.XLS' removed 'clients/client1/~dmtmp/EXCEL/SALES1.XLS' removed 'clients/client1/~dmtmp/EXCEL/RESULTS3.PRN' removed 'clients/client1/~dmtmp/EXCEL/RESULTS2.PRN' removed 'clients/client1/~dmtmp/EXCEL/PCMAGCD.XLS' removed 'clients/client1/~dmtmp/EXCEL/SALES1.PRN' removed 'clients/client1/~dmtmp/EXCEL/RESULTS.XLS' removed 'clients/client1/~dmtmp/EXCEL/RESULTS1.PRN' removed directory: 'clients/client1/~dmtmp/EXCEL' removed 'clients/client1/~dmtmp/ACCESS/INV.PRN' removed 'clients/client1/~dmtmp/ACCESS/FASTENER.MDB' removed 'clients/client1/~dmtmp/ACCESS/SALES.PRN' removed 'clients/client1/~dmtmp/ACCESS/LABELS.PRN' removed directory: 'clients/client1/~dmtmp/ACCESS' removed 'clients/client1/~dmtmp/COREL/TEAPOT2.CPT' removed 'clients/client1/~dmtmp/COREL/CDRBARS.CFG' removed 'clients/client1/~dmtmp/COREL/GRAPHIC1.CDR' removed 'clients/client1/~dmtmp/COREL/MAILBOX2.CPT' removed 'clients/client1/~dmtmp/COREL/GRAPH2.CDR' removed 'clients/client1/~dmtmp/COREL/CDRROLS.CFG' removed 'clients/client1/~dmtmp/COREL/ROSE.CPT' removed 'clients/client1/~dmtmp/COREL/GRAPH2.BAK' removed 'clients/client1/~dmtmp/COREL/TRAIN.CDR' removed 'clients/client1/~dmtmp/COREL/GRAPH1.CDR' removed 'clients/client1/~dmtmp/COREL/GRAPH1.BAK' removed 'clients/client1/~dmtmp/COREL/CORELDRW.CDT' removed 'clients/client1/~dmtmp/COREL/SIGN2.CPT' removed 'clients/client1/~dmtmp/COREL/ARTISTIC.ACL' removed 'clients/client1/~dmtmp/COREL/GRAPHIC2.CDR' removed directory: 'clients/client1/~dmtmp/COREL' removed 'clients/client1/~dmtmp/WORD/TIPS.DOC' removed 'clients/client1/~dmtmp/WORD/HANDOUT.DOC' removed 'clients/client1/~dmtmp/WORD/CHAP10.DOC' removed 'clients/client1/~dmtmp/WORD/FACTS.DOC' removed 'clients/client1/~dmtmp/WORD/BASEMACH.DOC' removed directory: 'clients/client1/~dmtmp/WORD' removed 'clients/client1/~dmtmp/PM/T1.XLS' removed 'clients/client1/~dmtmp/PM/EVENTS.DOC' removed 'clients/client1/~dmtmp/PM/MOVED.DOC' removed directory: 'clients/client1/~dmtmp/PM' removed 'clients/client1/~dmtmp/SEED/MEDIUM.FIL' removed 'clients/client1/~dmtmp/SEED/LARGE.FIL' removed 'clients/client1/~dmtmp/SEED/SMALL.FIL' removed directory: 'clients/client1/~dmtmp/SEED' removed 'clients/client1/~dmtmp/PWRPNT/TRIDOTS.POT' removed 'clients/client1/~dmtmp/PWRPNT/PPTOOLS1.PPA' removed 'clients/client1/~dmtmp/PWRPNT/NEWPCB.PPT' removed 'clients/client1/~dmtmp/PWRPNT/ZD16.BMP' removed 'clients/client1/~dmtmp/PWRPNT/NEWTIPS.PPT' removed 'clients/client1/~dmtmp/PWRPNT/PCBENCHM.PPT' removed 'clients/client1/~dmtmp/PWRPNT/TIPS.PPT' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed 'clients/client1/~dmtmp/PARADOX/COURSES.PX' removed 'clients/client1/~dmtmp/PARADOX/CHANGED.DB' removed 'clients/client1/~dmtmp/PARADOX/STUDENTS.XG0' removed 'clients/client1/~dmtmp/PARADOX/STUDENTS.YG1' removed 'clients/client1/~dmtmp/PARADOX/ENROLL.PX' removed 'clients/client1/~dmtmp/PARADOX/STUDENTS.MB' removed 'clients/client1/~dmtmp/PARADOX/STUDENTS.TV' removed 'clients/client1/~dmtmp/PARADOX/COURSES.Y04' removed 'clients/client1/~dmtmp/PARADOX/STUDENTS.YG0' removed 'clients/client1/~dmtmp/PARADOX/ANSWER.DB' removed 'clients/client1/~dmtmp/PARADOX/COURSES.VAL' removed 'clients/client1/~dmtmp/PARADOX/COURSES.X04' removed 'clients/client1/~dmtmp/PARADOX/ENROLL.VAL' removed 'clients/client1/~dmtmp/PARADOX/COURSlustre-MDT0000: Deregistered changelog user #1 ES.DB' removed 'clients/client1/~dmtmp/PARADOX/COURSES.FSL' removed 'clients/client1/~dmtmp/PARADOX/ENROLL.Y03' removed 'clients/client1/~dmtmp/PARADOX/STUDENTS.PX' removed 'clients/client1/~dmtmp/PARADOX/ANSWER.MB' removed 'clients/client1/~dmtmp/PARADOX/STUDENTS.VAL' removed 'clients/client1/~dmtmp/PARADOX/ENROLL.X03' removed 'clients/client1/~dmtmp/PARADOX/STUDENTS.XG1' removed 'clients/client1/~dmtmp/PARADOX/STUDENTS.DB' removed 'clients/client1/~dmtmp/PARADOX/ENROLL.DB' removed 'clients/client1/~dmtmp/PARADOX/STUDENTS.FSL' removed directory: 'clients/client1/~dmtmp/PARADOX' removed 'clients/client1/~dmtmp/WORDPRO/BENCHS1.LWP' removed 'clients/client1/~dmtmp/WORDPRO/BENCHS1A.PRN' removed 'clients/client1/~dmtmp/WORDPRO/BENCHS.LWP' removed 'clients/client1/~dmtmp/WORDPRO/BENCHS1.PRN' removed 'clients/client1/~dmtmp/WORDPRO/NEWS1_1.LWP' removed 'clients/client1/~dmtmp/WORDPRO/NEWS1_1A.PRN' removed 'clients/client1/~dmtmp/WORDPRO/NEWS1_1A.LWP' removed 'clients/client1/~dmtmp/WORDPRO/RESULTS.XLS' removed 'clients/client1/~dmtmp/WORDPRO/NEWS1_1B.PRN' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp' removed 'clients/client1/filler.000' removed directory: 'clients/client1' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2b.lustre-rsync-test dbench successfully finished