== recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 14:17:35 (1740683855) timeout=300 Failing ost1 on oleg205-server Stopping /mnt/lustre-ost1 (opts:) on oleg205-server 14:17:37 (1740683857) shut down facet: ost1 facet_host: oleg205-server facet_failover_host: oleg205-server Failover ost1 to oleg205-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 14:17:51 (1740683871) targets are mounted 14:17:51 (1740683871) facet_failover done oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec client: ssh: Could not resolve hostname client: Name or service not known pdsh@oleg205-client: client: ssh exited with exit code 255 PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9660) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9661) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9662) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9663) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9664) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9665) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9666) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9667) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9668) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9669) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9670) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9671) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9672) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9673) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9674) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9675) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9676) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9677) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9678) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9679) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9680) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9681) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9682) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9683) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9684) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (9685) - No such process PID TTY TIME CMD 9686 pts/0 00:00:00 touch PID TTY TIME CMD 9687 pts/0 00:00:00 touch PID TTY TIME CMD 9688 pts/0 00:00:00 touch PID TTY TIME CMD 9689 pts/0 00:00:00 touch PID TTY TIME CMD 9690 pts/0 00:00:00 touch /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3209: 9686 Killed touch $DIR/$tdir/$tfile_$i PID TTY TIME CMD 9691 pts/0 00:00:00 touch PID TTY TIME CMD 9692 pts/0 00:00:00 touch PID TTY TIME CMD 9693 pts/0 00:00:00 touch /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3209: 9687 Killed touch $DIR/$tdir/$tfile_$i PID TTY TIME CMD 9694 pts/0 00:00:00 touch PID TTY TIME CMD 9695 pts/0 00:00:00 touch PID TTY TIME CMD 9696 pts/0 00:00:00 touch PID TTY TIME CMD 9697 pts/0 00:00:00 touch PID TTY TIME CMD 9698 pts/0 00:00:00 touch PID TTY TIME CMD 9699 pts/0 00:00:00 touch PID TTY TIME CMD 9700 pts/0 00:00:00 touch PID TTY TIME CMD 9701 pts/0 00:00:00 touch PID TTY TIME CMD 9702 pts/0 00:00:00 touch PID TTY TIME CMD 9703 pts/0 00:00:00 touch PID TTY TIME CMD 9704 pts/0 00:00:00 touch PID TTY TIME CMD 9705 pts/0 00:00:00 touch /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3209: 9688 Killed touch $DIR/$tdir/$tfile_$i PID TTY TIME CMD 9706 pts/0 00:00:00 touch PID TTY TIME CMD 9707 pts/0 00:00:00 touch PID TTY TIME CMD 9708 pts/0 00:00:00 touch PID TTY TIME CMD 9709 pts/0 00:00:00 touch rc 24 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4178: 9689 Killed touch $DIR/$tdir/$tfile_$i Failing mds1 on oleg205-server Stopping /mnt/lustre-mds1 (opts:) on oleg205-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4898: 9690 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4898: 9691 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 9692 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 9693 Killed touch $DIR/$tdir/$tfile_$i 14:17:57 (1740683877) shut down facet: mds1 facet_host: oleg205-server facet_failover_host: oleg205-server Failover mds1 to oleg205-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-MDT0000 14:18:10 (1740683890) targets are mounted 14:18:10 (1740683890) facet_failover done /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4045: 9694 Killed touch $DIR/$tdir/$tfile_$i oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 9695 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 9696 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 9697 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 9698 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 9699 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 9700 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 9702 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 9703 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 9704 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 9701 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 9705 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 9706 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 9707 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 9708 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 9709 Killed touch $DIR/$tdir/$tfile_$i recovery-small test_144b: @@@@@@ FAIL: Create thread still running Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7235:error() = /home/green/git/lustre-release/lustre/tests/recovery-small.sh:3215:test_144b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7608:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7671:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7474:run_test() = /home/green/git/lustre-release/lustre/tests/recovery-small.sh:3217:main() Dumping lctl log to /tmp/testlogs//recovery-small.test_144b.*.1740683899.log timeout=20 rsync: chown "/tmp/testlogs/.recovery-small.test_144b.debug_log.oleg205-server.1740683899.log.zw7JLh" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.recovery-small.test_144b.dmesg.oleg205-server.1740683899.log.YDtbEe" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]