== recovery-small test 51: failover MDS during recovery == 10:26:06 (1781274366)
fail_loc=0x00001310
Failing mds1 on oleg310-server
Stopping /mnt/lustre-mds1 (opts:) on oleg310-server
10:26:15 (1781274375) shut down
facet: mds1 facet_host: oleg310-server facet_failover_host: oleg310-server
Failover mds1 to oleg310-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all
pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1
Started lustre-MDT0000
10:26:46 (1781274406) targets are mounted
10:26:46 (1781274406) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg310-server
Stopping /mnt/lustre-mds1 (opts:) on oleg310-server
10:26:53 (1781274413) shut down
facet: mds1 facet_host: oleg310-server facet_failover_host: oleg310-server
Failover mds1 to oleg310-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all
pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1
Started lustre-MDT0000
10:27:15 (1781274435) targets are mounted
10:27:15 (1781274435) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg310-server
Stopping /mnt/lustre-mds1 (opts:) on oleg310-server
10:27:27 (1781274447) shut down
facet: mds1 facet_host: oleg310-server facet_failover_host: oleg310-server
Failover mds1 to oleg310-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all
pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1
Started lustre-MDT0000
10:27:51 (1781274471) targets are mounted
10:27:51 (1781274471) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg310-server
Stopping /mnt/lustre-mds1 (opts:) on oleg310-server
10:28:10 (1781274490) shut down
facet: mds1 facet_host: oleg310-server facet_failover_host: oleg310-server
Failover mds1 to oleg310-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all
pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1
Started lustre-MDT0000
10:28:36 (1781274516) targets are mounted
10:28:36 (1781274516) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg310-server
Stopping /mnt/lustre-mds1 (opts:) on oleg310-server
10:29:05 (1781274545) shut down
facet: mds1 facet_host: oleg310-server facet_failover_host: oleg310-server
Failover mds1 to oleg310-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all
pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1
Started lustre-MDT0000
10:29:36 (1781274576) targets are mounted
10:29:36 (1781274576) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg310-server
Stopping /mnt/lustre-mds1 (opts:) on oleg310-server
10:30:10 (1781274610) shut down
facet: mds1 facet_host: oleg310-server facet_failover_host: oleg310-server
Failover mds1 to oleg310-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all
pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1
Started lustre-MDT0000
10:30:35 (1781274635) targets are mounted
10:30:35 (1781274635) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg310-server
Stopping /mnt/lustre-mds1 (opts:) on oleg310-server
10:31:13 (1781274673) shut down
facet: mds1 facet_host: oleg310-server facet_failover_host: oleg310-server
Failover mds1 to oleg310-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all
pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1
Started lustre-MDT0000
10:31:37 (1781274697) targets are mounted
10:31:37 (1781274697) facet_failover done
writemany returned 0