-----============= acceptance-small: recovery-small ============----- Thu Feb 27 14:17:31 EST 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg342-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/recovery-small.*ex: No such file or directory excepting tests: === recovery-small: start setup 14:17:36 (1740683856) === oleg342-client.virtnet: executing check_config_client /mnt/lustre oleg342-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg342-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6e2e000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6e2e000.idle_timeout=debug disable quota as required oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all osd-ldiskfs.track_declares_assert=1 === recovery-small: finish setup 14:17:46 (1740683866) === == recovery-small test 144b: orphan cleanup shouldn't be blocked for no objects+failover situation ========================================================== 14:17:47 (1740683867) striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d144b.recovery-small timeout=300 Failing ost1 on oleg342-server Stopping /mnt/lustre-ost1 (opts:) on oleg342-server 14:17:55 (1740683875) shut down facet: ost1 facet_host: oleg342-server facet_failover_host: oleg342-server Failover ost1 to oleg342-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0000 14:18:08 (1740683888) targets are mounted 14:18:08 (1740683888) facet_failover done oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec client: ssh: Could not resolve hostname client: Name or service not known pdsh@oleg342-client: client: ssh exited with exit code 255 PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10516) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10517) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10518) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10519) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10520) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10521) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10522) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10523) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10524) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10525) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10526) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10527) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10528) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10529) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10530) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10531) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10532) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10533) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10534) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10535) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10536) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10537) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10538) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10539) - No such process PID TTY TIME CMD 10540 pts/0 00:00:00 touch PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10541) - No such process PID TTY TIME CMD 10542 pts/0 00:00:00 touch PID TTY TIME CMD 10543 pts/0 00:00:00 touch PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10544) - No such process PID TTY TIME CMD /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3212: kill: (10545) - No such process PID TTY TIME CMD 10546 pts/0 00:00:00 touch PID TTY TIME CMD 10547 pts/0 00:00:00 touch PID TTY TIME CMD 10548 pts/0 00:00:00 touch /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3209: 10543 Killed touch $DIR/$tdir/$tfile_$i PID TTY TIME CMD 10549 pts/0 00:00:00 touch PID TTY TIME CMD 10550 pts/0 00:00:00 touch PID TTY TIME CMD 10551 pts/0 00:00:00 touch PID TTY TIME CMD 10552 pts/0 00:00:00 touch PID TTY TIME CMD 10553 pts/0 00:00:00 touch /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3209: 10542 Killed touch $DIR/$tdir/$tfile_$i PID TTY TIME CMD 10554 pts/0 00:00:00 touch PID TTY TIME CMD 10555 pts/0 00:00:00 touch PID TTY TIME CMD 10556 pts/0 00:00:00 touch /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3209: 10549 Killed touch $DIR/$tdir/$tfile_$i PID TTY TIME CMD 10557 pts/0 00:00:00 touch PID TTY TIME CMD 10558 pts/0 00:00:00 touch PID TTY TIME CMD 10559 pts/0 00:00:00 touch PID TTY TIME CMD 10560 pts/0 00:00:00 touch PID TTY TIME CMD 10561 pts/0 00:00:00 touch PID TTY TIME CMD 10562 pts/0 00:00:00 touch PID TTY TIME CMD 10563 pts/0 00:00:00 touch PID TTY TIME CMD 10564 pts/0 00:00:00 touch PID TTY TIME CMD 10565 pts/0 00:00:00 touch /home/green/git/lustre-release/lustre/tests/recovery-small.sh: line 3209: 10548 Killed touch $DIR/$tdir/$tfile_$i rc 23 Failing mds1 on oleg342-server Stopping /mnt/lustre-mds1 (opts:) on oleg342-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4898: 10540 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10546 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10550 Killed touch $DIR/$tdir/$tfile_$i 14:18:15 (1740683895) shut down facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server Failover mds1 to oleg342-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-MDT0000 14:18:28 (1740683908) targets are mounted 14:18:28 (1740683908) facet_failover done oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10547 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10551 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10552 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10553 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10554 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10555 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10556 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4707: 10557 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 10558 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 10559 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 10560 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 10561 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 10562 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 10563 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 10564 Killed touch $DIR/$tdir/$tfile_$i /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4067: 10565 Killed touch $DIR/$tdir/$tfile_$i recovery-small test_144b: @@@@@@ FAIL: Create thread still running Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7235:error() = /home/green/git/lustre-release/lustre/tests/recovery-small.sh:3215:test_144b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7608:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7671:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7474:run_test() = /home/green/git/lustre-release/lustre/tests/recovery-small.sh:3217:main() Dumping lctl log to /tmp/testlogs//recovery-small.test_144b.*.1740683914.log timeout=20 rsync: chown "/tmp/testlogs/.recovery-small.test_144b.debug_log.oleg342-server.1740683914.log.5LFUpV" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.recovery-small.test_144b.dmesg.oleg342-server.1740683914.log.kQPvzH" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 144b (60s) == recovery-small test complete, duration 76 sec ========= 14:18:47 (1740683927) recovery-small: FAIL: test_144b Create thread still running === recovery-small: start cleanup 14:18:47 (1740683927) === === recovery-small: finish cleanup 14:18:48 (1740683928) ===