== recovery-small test 51: failover MDS during recovery == 08:41:48 (1786192908)
fail_loc=0x00001310
Failing mds1 on oleg139-server
Stopping /mnt/lustre-mds1 (opts:) on oleg139-server
08:41:53 (1786192913) shut down
facet: mds1 facet_host: oleg139-server facet_failover_host: oleg139-server
Failover mds1 to oleg139-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1
Started lustre-MDT0000
08:42:08 (1786192928) targets are mounted
08:42:08 (1786192928) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg139-server
Stopping /mnt/lustre-mds1 (opts:) on oleg139-server
08:42:12 (1786192932) shut down
facet: mds1 facet_host: oleg139-server facet_failover_host: oleg139-server
Failover mds1 to oleg139-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1
Started lustre-MDT0000
08:42:28 (1786192948) targets are mounted
08:42:28 (1786192948) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg139-server
Stopping /mnt/lustre-mds1 (opts:) on oleg139-server
08:42:35 (1786192955) shut down
facet: mds1 facet_host: oleg139-server facet_failover_host: oleg139-server
Failover mds1 to oleg139-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1
Started lustre-MDT0000
08:42:51 (1786192971) targets are mounted
08:42:51 (1786192971) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg139-server
Stopping /mnt/lustre-mds1 (opts:) on oleg139-server
08:43:03 (1786192983) shut down
facet: mds1 facet_host: oleg139-server facet_failover_host: oleg139-server
Failover mds1 to oleg139-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1
Started lustre-MDT0000
08:43:19 (1786192999) targets are mounted
08:43:19 (1786192999) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg139-server
Stopping /mnt/lustre-mds1 (opts:) on oleg139-server
08:43:42 (1786193022) shut down
facet: mds1 facet_host: oleg139-server facet_failover_host: oleg139-server
Failover mds1 to oleg139-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1
Started lustre-MDT0000
08:43:58 (1786193038) targets are mounted
08:43:58 (1786193038) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg139-server
Stopping /mnt/lustre-mds1 (opts:) on oleg139-server
08:44:26 (1786193066) shut down
facet: mds1 facet_host: oleg139-server facet_failover_host: oleg139-server
Failover mds1 to oleg139-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1
Started lustre-MDT0000
08:44:42 (1786193082) targets are mounted
08:44:42 (1786193082) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg139-server
Stopping /mnt/lustre-mds1 (opts:) on oleg139-server
08:45:15 (1786193115) shut down
facet: mds1 facet_host: oleg139-server facet_failover_host: oleg139-server
Failover mds1 to oleg139-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1
Started lustre-MDT0000
08:45:31 (1786193131) targets are mounted
08:45:31 (1786193131) facet_failover done
writemany returned 0