== recovery-small test 51: failover MDS during recovery == 21:01:25 (1782867685)
fail_loc=0x00001310
Failing mds1 on oleg137-server
Stopping /mnt/lustre-mds1 (opts:) on oleg137-server
21:01:29 (1782867689) shut down
facet: mds1 facet_host: oleg137-server facet_failover_host: oleg137-server
Failover mds1 to oleg137-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all
pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1
Started lustre-MDT0000
21:01:43 (1782867703) targets are mounted
21:01:43 (1782867703) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg137-server
Stopping /mnt/lustre-mds1 (opts:) on oleg137-server
21:01:46 (1782867706) shut down
facet: mds1 facet_host: oleg137-server facet_failover_host: oleg137-server
Failover mds1 to oleg137-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all
pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1
Started lustre-MDT0000
21:02:00 (1782867720) targets are mounted
21:02:00 (1782867720) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg137-server
Stopping /mnt/lustre-mds1 (opts:) on oleg137-server
21:02:07 (1782867727) shut down
facet: mds1 facet_host: oleg137-server facet_failover_host: oleg137-server
Failover mds1 to oleg137-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all
pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1
Started lustre-MDT0000
21:02:22 (1782867742) targets are mounted
21:02:22 (1782867742) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg137-server
Stopping /mnt/lustre-mds1 (opts:) on oleg137-server
21:02:35 (1782867755) shut down
facet: mds1 facet_host: oleg137-server facet_failover_host: oleg137-server
Failover mds1 to oleg137-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all
pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1
Started lustre-MDT0000
21:02:49 (1782867769) targets are mounted
21:02:49 (1782867769) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg137-server
Stopping /mnt/lustre-mds1 (opts:) on oleg137-server
21:03:12 (1782867792) shut down
facet: mds1 facet_host: oleg137-server facet_failover_host: oleg137-server
Failover mds1 to oleg137-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all
pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1
Started lustre-MDT0000
21:03:27 (1782867807) targets are mounted
21:03:27 (1782867807) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg137-server
Stopping /mnt/lustre-mds1 (opts:) on oleg137-server
21:03:54 (1782867834) shut down
facet: mds1 facet_host: oleg137-server facet_failover_host: oleg137-server
Failover mds1 to oleg137-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all
pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1
Started lustre-MDT0000
21:04:09 (1782867849) targets are mounted
21:04:09 (1782867849) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg137-server
Stopping /mnt/lustre-mds1 (opts:) on oleg137-server
21:04:41 (1782867881) shut down
facet: mds1 facet_host: oleg137-server facet_failover_host: oleg137-server
Failover mds1 to oleg137-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all
pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1
Started lustre-MDT0000
21:04:56 (1782867896) targets are mounted
21:04:56 (1782867896) facet_failover done
writemany returned 0