== recovery-small test 144a: MDT failover should stop precreation threads ========================================================== 15:38:30 (1776195510)
timeout=300
Failing ost1 on oleg433-server
Stopping /mnt/lustre-ost1 (opts:) on oleg433-server
reboot facets: ost1
Failover ost1 to oleg433-server
mount facets: ost1
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1
Started lustre-OST0000
oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Failing mds1 on oleg433-server
Stopping /mnt/lustre-mds1 (opts:) on oleg433-server
reboot facets: mds1
Failover mds1 to oleg433-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 94997 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 94998 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 94999 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95000 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95001 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95002 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95003 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95004 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95005 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95007 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95008 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95009 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95010 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95011 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3637: 95012 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3637: 95015 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3615: 95006 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3615: 95014 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3615: 95016 Killed                  touch $DIR/$tdir/$tfile_$i
Failing mds1 on oleg433-server
Stopping /mnt/lustre-mds1 (opts:) on oleg433-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4341: 95013 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95017 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95019 Killed                  touch $DIR/$tdir/$tfile_$i
reboot facets: mds1
Failover mds1 to oleg433-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all 8
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4285: 95018 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4285: 95020 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4285: 95021 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4285: 95023 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4285: 95026 Killed                  touch $DIR/$tdir/$tfile_$i
pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3117: 95024 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 2185: 95022 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 2185: 95027 Killed                  touch $DIR/$tdir/$tfile_$i
Started lustre-MDT0000
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3637: 95028 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3637: 95029 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3615: 95025 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 3615: 95030 Killed                  touch $DIR/$tdir/$tfile_$i
oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95031 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95032 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95033 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95034 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95035 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95036 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95037 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95038 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95039 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95040 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95041 Killed                  touch $DIR/$tdir/$tfile_$i
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4180: 95042 Killed                  touch $DIR/$tdir/$tfile_$i
timeout=20