== recovery-small test 51: failover MDS during recovery == 04:29:53 (1789547393)
fail_loc=0x00001310
Failing mds1 on oleg342-server
Stopping /mnt/lustre-mds1 (opts:) on oleg342-server
04:29:56 (1789547396) shut down
facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server
Failover mds1 to oleg342-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all
pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1
Started lustre-MDT0000
04:30:11 (1789547411) targets are mounted
04:30:11 (1789547411) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg342-server
Stopping /mnt/lustre-mds1 (opts:) on oleg342-server
04:30:14 (1789547414) shut down
facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server
Failover mds1 to oleg342-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all
pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1
Started lustre-MDT0000
04:30:28 (1789547428) targets are mounted
04:30:28 (1789547428) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg342-server
Stopping /mnt/lustre-mds1 (opts:) on oleg342-server
04:30:35 (1789547435) shut down
facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server
Failover mds1 to oleg342-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all
pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1
Started lustre-MDT0000
04:30:50 (1789547450) targets are mounted
04:30:50 (1789547450) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg342-server
Stopping /mnt/lustre-mds1 (opts:) on oleg342-server
04:31:02 (1789547462) shut down
facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server
Failover mds1 to oleg342-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all
pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1
Started lustre-MDT0000
04:31:16 (1789547476) targets are mounted
04:31:16 (1789547476) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg342-server
Stopping /mnt/lustre-mds1 (opts:) on oleg342-server
04:31:39 (1789547499) shut down
facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server
Failover mds1 to oleg342-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all
pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1
Started lustre-MDT0000
04:31:53 (1789547513) targets are mounted
04:31:53 (1789547513) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg342-server
Stopping /mnt/lustre-mds1 (opts:) on oleg342-server
04:32:21 (1789547541) shut down
facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server
Failover mds1 to oleg342-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all
pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1
Started lustre-MDT0000
04:32:35 (1789547555) targets are mounted
04:32:35 (1789547555) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg342-server
Stopping /mnt/lustre-mds1 (opts:) on oleg342-server
04:33:07 (1789547587) shut down
facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server
Failover mds1 to oleg342-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all
pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1
Started lustre-MDT0000
04:33:21 (1789547601) targets are mounted
04:33:21 (1789547601) facet_failover done
writemany returned 0