== recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 14:17:39 (1740683859) timeout=300 Failing ost1 on oleg159-server Stopping /mnt/lustre-ost1 (opts:) on oleg159-server 14:17:48 (1740683868) shut down facet: ost1 facet_host: oleg159-server facet_failover_host: oleg159-server Failover ost1 to oleg159-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-OST0000 14:18:01 (1740683881) targets are mounted 14:18:01 (1740683881) facet_failover done oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec client: ssh: Could not resolve hostname client: Name or service not known pdsh@oleg159-client: client: ssh exited with exit code 255 PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9691) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9692) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9693) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9694) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9695) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9696) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9697) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9698) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9699) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9700) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9701) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9702) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9703) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9704) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9705) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9706) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9707) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9708) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9709) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9710) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9711) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9712) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9713) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9714) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9715) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9716) - No such process PID TTY TIME CMD 9717 pts/0 00:00:00 touch PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9718) - No such process PID TTY TIME CMD 9719 pts/0 00:00:00 touch PID TTY TIME CMD 9720 pts/0 00:00:00 touch PID TTY TIME CMD 9721 pts/0 00:00:00 touch PID TTY TIME CMD 9722 pts/0 00:00:00 touch PID TTY TIME CMD 9723 pts/0 00:00:00 touch PID TTY TIME CMD 9724 pts/0 00:00:00 touch PID TTY TIME CMD 9725 pts/0 00:00:00 touch PID TTY TIME CMD 9726 pts/0 00:00:00 touch PID TTY TIME CMD 9727 pts/0 00:00:00 touch PID TTY TIME CMD 9728 pts/0 00:00:00 touch PID TTY TIME CMD 9729 pts/0 00:00:00 touch PID TTY TIME CMD 9730 pts/0 00:00:00 touch PID TTY TIME CMD 9731 pts/0 00:00:00 touch PID TTY TIME CMD 9732 pts/0 00:00:00 touch PID TTY TIME CMD 9733 pts/0 00:00:00 touch PID TTY TIME CMD 9734 pts/0 00:00:00 touch PID TTY TIME CMD 9735 pts/0 00:00:00 touch PID TTY TIME CMD 9736 pts/0 00:00:00 touch PID TTY TIME CMD 9737 pts/0 00:00:00 touch PID TTY TIME CMD 9738 pts/0 00:00:00 touch PID TTY TIME CMD 9739 pts/0 00:00:00 touch PID TTY TIME CMD 9740 pts/0 00:00:00 touch rc 23 Failing mds1 on oleg159-server Stopping /mnt/lustre-mds1 (opts:) on oleg159-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4898: 9717 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4898: 9720 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4898: 9721 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 9719 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 9722 Killed touch $DIR/$tdir/$tfile_$i 14:18:07 (1740683887) shut down facet: mds1 facet_host: oleg159-server facet_failover_host: oleg159-server Failover mds1 to oleg159-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-MDT0000 14:18:20 (1740683900) targets are mounted 14:18:20 (1740683900) facet_failover done oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 9723 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 9724 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 9725 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 9726 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 9727 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 9728 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 9729 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 9730 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 9731 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 9732 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 9733 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 9734 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 9736 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 9737 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 9738 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4045: 9735 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4045: 9739 Killed touch $DIR/$tdir/$tfile_$i recovery-small test_144b: @@@@@@ FAIL: Create thread still running Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7235:error() = /home/green/git/lustre-release/lustre/tests/recovery-small.sh:3215:test_144b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7608:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7671:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7474:run_test() = /home/green/git/lustre-release/lustre/tests/recovery-small.sh:3217:main() Dumping lctl log to /tmp/testlogs//recovery-small.test_144b.*.1740683908.log timeout=20 rsync: chown "/tmp/testlogs/.recovery-small.test_144b.debug_log.oleg159-server.1740683908.log.z6mTMq" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.recovery-small.test_144b.dmesg.oleg159-server.1740683908.log.X2l45P" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]