== recovery-small test 51: failover MDS during recovery == 15:35:13 (1782243313)
fail_loc=0x00001310
Failing mds1 on oleg425-server
Stopping /mnt/lustre-mds1 (opts:) on oleg425-server
15:35:21 (1782243321) shut down
facet: mds1 facet_host: oleg425-server facet_failover_host: oleg425-server
Failover mds1 to oleg425-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all
pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1
Started lustre-MDT0000
15:35:42 (1782243342) targets are mounted
15:35:42 (1782243342) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg425-server
Stopping /mnt/lustre-mds1 (opts:) on oleg425-server
15:35:49 (1782243349) shut down
facet: mds1 facet_host: oleg425-server facet_failover_host: oleg425-server
Failover mds1 to oleg425-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all
pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1
Started lustre-MDT0000
15:36:12 (1782243372) targets are mounted
15:36:12 (1782243372) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg425-server
Stopping /mnt/lustre-mds1 (opts:) on oleg425-server
15:36:24 (1782243384) shut down
facet: mds1 facet_host: oleg425-server facet_failover_host: oleg425-server
Failover mds1 to oleg425-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all
pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1
Started lustre-MDT0000
15:36:47 (1782243407) targets are mounted
15:36:47 (1782243407) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg425-server
Stopping /mnt/lustre-mds1 (opts:) on oleg425-server
15:37:05 (1782243425) shut down
facet: mds1 facet_host: oleg425-server facet_failover_host: oleg425-server
Failover mds1 to oleg425-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all
pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1
Started lustre-MDT0000
15:37:27 (1782243447) targets are mounted
15:37:27 (1782243447) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg425-server
Stopping /mnt/lustre-mds1 (opts:) on oleg425-server
15:37:57 (1782243477) shut down
facet: mds1 facet_host: oleg425-server facet_failover_host: oleg425-server
Failover mds1 to oleg425-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all
pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1
Started lustre-MDT0000
15:38:29 (1782243509) targets are mounted
15:38:29 (1782243509) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg425-server
Stopping /mnt/lustre-mds1 (opts:) on oleg425-server
15:39:02 (1782243542) shut down
facet: mds1 facet_host: oleg425-server facet_failover_host: oleg425-server
Failover mds1 to oleg425-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all
pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1
Started lustre-MDT0000
15:39:24 (1782243564) targets are mounted
15:39:24 (1782243564) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg425-server
Stopping /mnt/lustre-mds1 (opts:) on oleg425-server
15:40:01 (1782243601) shut down
facet: mds1 facet_host: oleg425-server facet_failover_host: oleg425-server
Failover mds1 to oleg425-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all
pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1
Started lustre-MDT0000
15:40:24 (1782243624) targets are mounted
15:40:24 (1782243624) facet_failover done
writemany returned 0