== recovery-small test 51: failover MDS during recovery == 07:07:44 (1787396864)
fail_loc=0x00001310
Failing mds1 on oleg226-server
Stopping /mnt/lustre-mds1 (opts:) on oleg226-server
07:07:54 (1787396874) shut down
facet: mds1 facet_host: oleg226-server facet_failover_host: oleg226-server
Failover mds1 to oleg226-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all
pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1
Started lustre-MDT0000
07:08:29 (1787396909) targets are mounted
07:08:29 (1787396909) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg226-server
Stopping /mnt/lustre-mds1 (opts:) on oleg226-server
07:08:40 (1787396920) shut down
facet: mds1 facet_host: oleg226-server facet_failover_host: oleg226-server
Failover mds1 to oleg226-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all
pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1
Started lustre-MDT0000
07:09:16 (1787396956) targets are mounted
07:09:16 (1787396956) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg226-server
Stopping /mnt/lustre-mds1 (opts:) on oleg226-server
07:09:30 (1787396970) shut down
facet: mds1 facet_host: oleg226-server facet_failover_host: oleg226-server
Failover mds1 to oleg226-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all
pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1
Started lustre-MDT0000
07:09:56 (1787396996) targets are mounted
07:09:57 (1787396997) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg226-server
Stopping /mnt/lustre-mds1 (opts:) on oleg226-server
07:10:16 (1787397016) shut down
facet: mds1 facet_host: oleg226-server facet_failover_host: oleg226-server
Failover mds1 to oleg226-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all
pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1
Started lustre-MDT0000
07:10:40 (1787397040) targets are mounted
07:10:40 (1787397040) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg226-server
Stopping /mnt/lustre-mds1 (opts:) on oleg226-server
07:11:10 (1787397070) shut down
facet: mds1 facet_host: oleg226-server facet_failover_host: oleg226-server
Failover mds1 to oleg226-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all
pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1
Started lustre-MDT0000
07:11:42 (1787397102) targets are mounted
07:11:42 (1787397102) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg226-server
Stopping /mnt/lustre-mds1 (opts:) on oleg226-server
07:12:17 (1787397137) shut down
facet: mds1 facet_host: oleg226-server facet_failover_host: oleg226-server
Failover mds1 to oleg226-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all
pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1
Started lustre-MDT0000
07:12:54 (1787397174) targets are mounted
07:12:54 (1787397174) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg226-server
Stopping /mnt/lustre-mds1 (opts:) on oleg226-server
07:13:35 (1787397215) shut down
facet: mds1 facet_host: oleg226-server facet_failover_host: oleg226-server
Failover mds1 to oleg226-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all
pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1
Started lustre-MDT0000
07:14:09 (1787397249) targets are mounted
07:14:09 (1787397249) facet_failover done
writemany returned 0