== recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 14:17:33 (1740683853) striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d144b.recovery-small timeout=300 Failing ost1 on oleg312-server Stopping /mnt/lustre-ost1 (opts:) on oleg312-server 14:17:37 (1740683857) shut down facet: ost1 facet_host: oleg312-server facet_failover_host: oleg312-server Failover ost1 to oleg312-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0000 14:17:50 (1740683870) targets are mounted 14:17:50 (1740683870) facet_failover done oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3204: 360: command not found client: ssh: Could not resolve hostname client: Name or service not known pdsh@oleg312-client: client: ssh exited with exit code 255 PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10538) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10539) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10540) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10541) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10542) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10543) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10544) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10545) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10546) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10547) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10548) - No such process PID TTY TIME CMD 10549 pts/0 00:00:00 touch PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10550) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10551) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10552) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10553) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10554) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10555) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10556) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10557) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10558) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10559) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10560) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10561) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10562) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10563) - No such process PID TTY TIME CMD 10564 pts/0 00:00:00 touch PID TTY TIME CMD 10565 pts/0 00:00:00 touch PID TTY TIME CMD 10566 pts/0 00:00:00 touch PID TTY TIME CMD 10567 pts/0 00:00:00 touch PID TTY TIME CMD 10568 pts/0 00:00:00 touch PID TTY TIME CMD 10569 pts/0 00:00:00 touch PID TTY TIME CMD 10570 pts/0 00:00:00 touch PID TTY TIME CMD 10571 pts/0 00:00:00 touch /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3209: 10570 Killed touch $DIR/$tdir/$tfile_$i PID TTY TIME CMD 10572 pts/0 00:00:00 touch PID TTY TIME CMD 10573 pts/0 00:00:00 touch PID TTY TIME CMD 10574 pts/0 00:00:00 touch PID TTY TIME CMD 10575 pts/0 00:00:00 touch /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3209: 10567 Killed touch $DIR/$tdir/$tfile_$i PID TTY TIME CMD 10576 pts/0 00:00:00 touch PID TTY TIME CMD 10577 pts/0 00:00:00 touch PID TTY TIME CMD 10578 pts/0 00:00:00 touch PID TTY TIME CMD 10579 pts/0 00:00:00 touch PID TTY TIME CMD 10580 pts/0 00:00:00 touch PID TTY TIME CMD 10581 pts/0 00:00:00 touch PID TTY TIME CMD 10582 pts/0 00:00:00 touch PID TTY TIME CMD 10583 pts/0 00:00:00 touch PID TTY TIME CMD 10584 pts/0 00:00:00 touch PID TTY TIME CMD 10585 pts/0 00:00:00 touch PID TTY TIME CMD 10586 pts/0 00:00:00 touch PID TTY TIME CMD 10587 pts/0 00:00:00 touch rc 25 Failing mds1 on oleg312-server Stopping /mnt/lustre-mds1 (opts:) on oleg312-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10565 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10569 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10574 Killed touch $DIR/$tdir/$tfile_$i 14:17:56 (1740683876) shut down facet: mds1 facet_host: oleg312-server facet_failover_host: oleg312-server Failover mds1 to oleg312-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 14:18:09 (1740683889) targets are mounted 14:18:09 (1740683889) facet_failover done /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4045: 10564 Killed touch $DIR/$tdir/$tfile_$i oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10549 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10566 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10568 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10571 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10572 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10573 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10576 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 10577 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 10578 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 10579 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 10580 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 10581 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 10582 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 10586 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4045: 10585 Killed touch $DIR/$tdir/$tfile_$i recovery-small test_144b: @@@@@@ FAIL: Create thread still running Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7235:error() = /home/green/git/lustre-release/lustre/tests/recovery-small.sh:3215:test_144b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7608:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7671:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7474:run_test() = /home/green/git/lustre-release/lustre/tests/recovery-small.sh:3217:main() Dumping lctl log to /tmp/testlogs//recovery-small.test_144b.*.1740683895.log /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4842: 10584 Killed touch $DIR/$tdir/$tfile_$i timeout=20 rsync: chown "/tmp/testlogs/.recovery-small.test_144b.debug_log.oleg312-server.1740683895.log.sRoY5v" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.recovery-small.test_144b.dmesg.oleg312-server.1740683895.log.vZ5k8x" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]