== recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 14:17:42 (1740683862) striped dir -i0 -c1 -H crush /mnt/lustre/d144b.recovery-small timeout=300 Failing ost1 on oleg308-server Stopping /mnt/lustre-ost1 (opts:) on oleg308-server 14:17:47 (1740683867) shut down facet: ost1 facet_host: oleg308-server facet_failover_host: oleg308-server Failover ost1 to oleg308-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 14:18:00 (1740683880) targets are mounted 14:18:00 (1740683880) facet_failover done oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3204: 360: command not found client: ssh: Could not resolve hostname client: Name or service not known pdsh@oleg308-client: client: ssh exited with exit code 255 PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10565) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10566) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10567) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10568) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10569) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10570) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10571) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10572) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10573) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10574) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10575) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10576) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10577) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10578) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10579) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10580) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10581) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10582) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10583) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10584) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10585) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10586) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10587) - No such process PID TTY TIME CMD 10588 pts/0 00:00:00 touch PID TTY TIME CMD 10589 pts/0 00:00:00 touch PID TTY TIME CMD 10590 pts/0 00:00:00 touch PID TTY TIME CMD 10591 pts/0 00:00:00 touch PID TTY TIME CMD 10592 pts/0 00:00:00 touch PID TTY TIME CMD 10593 pts/0 00:00:00 touch /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3209: 10588 Killed touch $DIR/$tdir/$tfile_$i PID TTY TIME CMD 10594 pts/0 00:00:00 touch PID TTY TIME CMD 10595 pts/0 00:00:00 touch PID TTY TIME CMD 10596 pts/0 00:00:00 touch PID TTY TIME CMD 10597 pts/0 00:00:00 touch /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3209: 10589 Killed touch $DIR/$tdir/$tfile_$i PID TTY TIME CMD 10598 pts/0 00:00:00 touch PID TTY TIME CMD 10599 pts/0 00:00:00 touch PID TTY TIME CMD 10600 pts/0 00:00:00 touch PID TTY TIME CMD 10601 pts/0 00:00:00 touch PID TTY TIME CMD 10602 pts/0 00:00:00 touch PID TTY TIME CMD 10603 pts/0 00:00:00 touch /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3209: 10591 Killed touch $DIR/$tdir/$tfile_$i PID TTY TIME CMD 10604 pts/0 00:00:00 touch PID TTY TIME CMD 10605 pts/0 00:00:00 touch PID TTY TIME CMD 10606 pts/0 00:00:00 touch PID TTY TIME CMD 10607 pts/0 00:00:00 touch PID TTY TIME CMD 10608 pts/0 00:00:00 touch PID TTY TIME CMD 10609 pts/0 00:00:00 touch PID TTY TIME CMD 10610 pts/0 00:00:00 touch PID TTY TIME CMD 10611 pts/0 00:00:00 touch PID TTY TIME CMD 10612 pts/0 00:00:00 touch PID TTY TIME CMD 10613 pts/0 00:00:00 touch /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3209: 10593 Killed touch $DIR/$tdir/$tfile_$i PID TTY TIME CMD 10614 pts/0 00:00:00 touch rc 27 Failing mds1 on oleg308-server Stopping /mnt/lustre-mds1 (opts:) on oleg308-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4898: 10590 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10594 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10597 Killed touch $DIR/$tdir/$tfile_$i 14:18:06 (1740683886) shut down facet: mds1 facet_host: oleg308-server facet_failover_host: oleg308-server Failover mds1 to oleg308-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0000 14:18:19 (1740683899) targets are mounted 14:18:19 (1740683899) facet_failover done oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10592 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10595 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10596 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10598 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10599 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10602 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10603 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 10600 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 10601 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 10604 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 10605 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 10606 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 10607 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 10608 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4045: 10612 Killed touch $DIR/$tdir/$tfile_$i recovery-small test_144b: @@@@@@ FAIL: Create thread still running Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7235:error() = /home/green/git/lustre-release/lustre/tests/recovery-small.sh:3215:test_144b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7608:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7671:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7474:run_test() = /home/green/git/lustre-release/lustre/tests/recovery-small.sh:3217:main() Dumping lctl log to /tmp/testlogs//recovery-small.test_144b.*.1740683905.log /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4842: 10609 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4842: 10613 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4842: 10614 Killed touch $DIR/$tdir/$tfile_$i timeout=20 rsync: chown "/tmp/testlogs/.recovery-small.test_144b.debug_log.oleg308-server.1740683905.log.5uqzqv" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.recovery-small.test_144b.dmesg.oleg308-server.1740683905.log.5RwckN" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]