== recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 15:41:23 (1740688883) striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d144b.recovery-small timeout=300 Failing ost1 on oleg104-server Stopping /mnt/lustre-ost1 (opts:) on oleg104-server 15:41:27 (1740688887) shut down facet: ost1 facet_host: oleg104-server facet_failover_host: oleg104-server Failover ost1 to oleg104-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-OST0000 15:41:41 (1740688901) targets are mounted 15:41:41 (1740688901) facet_failover done oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec client: ssh: Could not resolve hostname client: Name or service not known pdsh@oleg104-client: client: ssh exited with exit code 255 PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (18756) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (18757) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (18758) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (18759) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (18760) - No such process PID TTY TIME CMD 18761 pts/0 00:00:00 touch PID TTY TIME CMD 18762 pts/0 00:00:00 touch PID TTY TIME CMD 18763 pts/0 00:00:00 touch PID TTY TIME CMD 18764 pts/0 00:00:00 touch PID TTY TIME CMD 18765 pts/0 00:00:00 touch PID TTY TIME CMD 18766 pts/0 00:00:00 touch PID TTY TIME CMD 18767 pts/0 00:00:00 touch PID TTY TIME CMD 18768 pts/0 00:00:00 touch PID TTY TIME CMD 18769 pts/0 00:00:00 touch PID TTY TIME CMD 18770 pts/0 00:00:00 touch PID TTY TIME CMD 18771 pts/0 00:00:00 touch PID TTY TIME CMD 18772 pts/0 00:00:00 touch PID TTY TIME CMD 18773 pts/0 00:00:00 touch PID TTY TIME CMD 18774 pts/0 00:00:00 touch PID TTY TIME CMD 18775 pts/0 00:00:00 touch PID TTY TIME CMD 18776 pts/0 00:00:00 touch PID TTY TIME CMD 18777 pts/0 00:00:00 touch PID TTY TIME CMD 18778 pts/0 00:00:00 touch PID TTY TIME CMD 18779 pts/0 00:00:00 touch PID TTY TIME CMD 18780 pts/0 00:00:00 touch PID TTY TIME CMD 18781 pts/0 00:00:00 touch PID TTY TIME CMD 18782 pts/0 00:00:00 touch PID TTY TIME CMD 18783 pts/0 00:00:00 touch PID TTY TIME CMD 18784 pts/0 00:00:00 touch PID TTY TIME CMD 18785 pts/0 00:00:00 touch PID TTY TIME CMD 18786 pts/0 00:00:00 touch PID TTY TIME CMD 18787 pts/0 00:00:00 touch PID TTY TIME CMD 18788 pts/0 00:00:00 touch PID TTY TIME CMD 18789 pts/0 00:00:00 touch PID TTY TIME CMD 18790 pts/0 00:00:00 touch PID TTY TIME CMD 18791 pts/0 00:00:00 touch PID TTY TIME CMD 18792 pts/0 00:00:00 touch PID TTY TIME CMD 18793 pts/0 00:00:00 touch PID TTY TIME CMD 18794 pts/0 00:00:00 touch PID TTY TIME CMD 18795 pts/0 00:00:00 touch PID TTY TIME CMD 18796 pts/0 00:00:00 touch PID TTY TIME CMD 18797 pts/0 00:00:00 touch PID TTY TIME CMD 18798 pts/0 00:00:00 touch PID TTY TIME CMD 18799 pts/0 00:00:00 touch PID TTY TIME CMD 18800 pts/0 00:00:00 touch PID TTY TIME CMD 18801 pts/0 00:00:00 touch PID TTY TIME CMD 18802 pts/0 00:00:00 touch PID TTY TIME CMD 18803 pts/0 00:00:00 touch PID TTY TIME CMD 18804 pts/0 00:00:00 touch PID TTY TIME CMD 18805 pts/0 00:00:00 touch rc 45 Failing mds1 on oleg104-server Stopping /mnt/lustre-mds1 (opts:) on oleg104-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4898: 18761 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4898: 18762 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 18764 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 18766 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 18767 Killed touch $DIR/$tdir/$tfile_$i 15:41:48 (1740688908) shut down facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server Failover mds1 to oleg104-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0000 15:42:01 (1740688921) targets are mounted 15:42:01 (1740688921) facet_failover done oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 18765 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 18768 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 18769 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 18770 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 18771 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 18772 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 18773 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 18775 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 18763 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 18774 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 18776 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 18777 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 18778 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 18779 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 18780 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 18782 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4045: 18783 Killed touch $DIR/$tdir/$tfile_$i recovery-small test_144b: @@@@@@ FAIL: Create thread still running Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7235:error() = /home/green/git/lustre-release/lustre/tests/recovery-small.sh:3215:test_144b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7608:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7671:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7489:run_test() = /home/green/git/lustre-release/lustre/tests/recovery-small.sh:3217:main() Dumping lctl log to /tmp/testlogs//recovery-small.test_144b.*.1740688927.log /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4842: 18781 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4842: 18787 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4842: 18788 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4842: 18789 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4842: 18790 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4842: 18793 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4842: 18794 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4842: 18795 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4842: 18796 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4866: 18797 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4842: 18791 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4842: 18798 Killed touch $DIR/$tdir/$tfile_$i timeout=20 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4842: 18792 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4842: 18800 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4842: 18801 Killed touch $DIR/$tdir/$tfile_$i rsync: chown "/tmp/testlogs/.recovery-small.test_144b.debug_log.oleg104-server.1740688927.log.CZ4BvN" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.recovery-small.test_144b.dmesg.oleg104-server.1740688927.log.9RK3mr" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1: 18799 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1: 18802 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1: 18803 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1: 18804 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1: 18805 Killed touch $DIR/$tdir/$tfile_$i