== recovery-small test 51: failover MDS during recovery == 21:31:10 (1789608670)
fail_loc=0x00001310
Failing mds1 on oleg260-server
Stopping /mnt/lustre-mds1 (opts:) on oleg260-server
21:31:19 (1789608679) shut down
facet: mds1 facet_host: oleg260-server facet_failover_host: oleg260-server
Failover mds1 to oleg260-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all
pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1
Started lustre-MDT0000
21:31:52 (1789608712) targets are mounted
21:31:52 (1789608712) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg260-server
Stopping /mnt/lustre-mds1 (opts:) on oleg260-server
21:32:01 (1789608721) shut down
facet: mds1 facet_host: oleg260-server facet_failover_host: oleg260-server
Failover mds1 to oleg260-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all
pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1
Started lustre-MDT0000
21:32:35 (1789608755) targets are mounted
21:32:35 (1789608755) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg260-server
Stopping /mnt/lustre-mds1 (opts:) on oleg260-server
21:32:46 (1789608766) shut down
facet: mds1 facet_host: oleg260-server facet_failover_host: oleg260-server
Failover mds1 to oleg260-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all
pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1
Started lustre-MDT0000
21:33:19 (1789608799) targets are mounted
21:33:19 (1789608799) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg260-server
Stopping /mnt/lustre-mds1 (opts:) on oleg260-server
21:33:37 (1789608817) shut down
facet: mds1 facet_host: oleg260-server facet_failover_host: oleg260-server
Failover mds1 to oleg260-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all
pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1
Started lustre-MDT0000
21:34:09 (1789608849) targets are mounted
21:34:09 (1789608849) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg260-server
Stopping /mnt/lustre-mds1 (opts:) on oleg260-server
21:34:37 (1789608877) shut down
facet: mds1 facet_host: oleg260-server facet_failover_host: oleg260-server
Failover mds1 to oleg260-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all
pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1
Started lustre-MDT0000
21:35:10 (1789608910) targets are mounted
21:35:10 (1789608910) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg260-server
Stopping /mnt/lustre-mds1 (opts:) on oleg260-server
21:35:43 (1789608943) shut down
facet: mds1 facet_host: oleg260-server facet_failover_host: oleg260-server
Failover mds1 to oleg260-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all
pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1
Started lustre-MDT0000
21:36:18 (1789608978) targets are mounted
21:36:18 (1789608978) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg260-server
Stopping /mnt/lustre-mds1 (opts:) on oleg260-server
21:36:59 (1789609019) shut down
facet: mds1 facet_host: oleg260-server facet_failover_host: oleg260-server
Failover mds1 to oleg260-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all
pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1
Started lustre-MDT0000
21:37:31 (1789609051) targets are mounted
21:37:31 (1789609051) facet_failover done
writemany returned 0