== recovery-small test 51: failover MDS during recovery == 10:48:43 (1787842123)
fail_loc=0x00001310
Failing mds1 on oleg636-server
Stopping /mnt/lustre-mds1 (opts:) on oleg636-server
10:48:51 (1787842131) shut down
facet: mds1 facet_host: oleg636-server facet_failover_host: oleg636-server
Failover mds1 to oleg636-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all
pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1
Started lustre-MDT0000
10:49:14 (1787842154) targets are mounted
10:49:14 (1787842154) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg636-server
Stopping /mnt/lustre-mds1 (opts:) on oleg636-server
10:49:22 (1787842162) shut down
facet: mds1 facet_host: oleg636-server facet_failover_host: oleg636-server
Failover mds1 to oleg636-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all
pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1
Started lustre-MDT0000
10:49:45 (1787842185) targets are mounted
10:49:45 (1787842185) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg636-server
Stopping /mnt/lustre-mds1 (opts:) on oleg636-server
10:49:56 (1787842196) shut down
facet: mds1 facet_host: oleg636-server facet_failover_host: oleg636-server
Failover mds1 to oleg636-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all
pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1
Started lustre-MDT0000
10:50:21 (1787842221) targets are mounted
10:50:21 (1787842221) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg636-server
Stopping /mnt/lustre-mds1 (opts:) on oleg636-server
10:50:39 (1787842239) shut down
facet: mds1 facet_host: oleg636-server facet_failover_host: oleg636-server
Failover mds1 to oleg636-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all
pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1
Started lustre-MDT0000
10:51:00 (1787842260) targets are mounted
10:51:00 (1787842260) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg636-server
Stopping /mnt/lustre-mds1 (opts:) on oleg636-server
10:51:28 (1787842288) shut down
facet: mds1 facet_host: oleg636-server facet_failover_host: oleg636-server
Failover mds1 to oleg636-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all
pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1
Started lustre-MDT0000
10:51:51 (1787842311) targets are mounted
10:51:51 (1787842311) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg636-server
Stopping /mnt/lustre-mds1 (opts:) on oleg636-server
10:52:24 (1787842344) shut down
facet: mds1 facet_host: oleg636-server facet_failover_host: oleg636-server
Failover mds1 to oleg636-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all
pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1
Started lustre-MDT0000
10:52:47 (1787842367) targets are mounted
10:52:47 (1787842367) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg636-server
Stopping /mnt/lustre-mds1 (opts:) on oleg636-server
10:53:24 (1787842404) shut down
facet: mds1 facet_host: oleg636-server facet_failover_host: oleg636-server
Failover mds1 to oleg636-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all
pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1
Started lustre-MDT0000
10:53:47 (1787842427) targets are mounted
10:53:47 (1787842427) facet_failover done
writemany returned 0