== recovery-small test 51: failover MDS during recovery == 01:40:37 (1788932437)
fail_loc=0x00001310
Failing mds1 on oleg609-server
Stopping /mnt/lustre-mds1 (opts:) on oleg609-server
01:40:45 (1788932445) shut down
facet: mds1 facet_host: oleg609-server facet_failover_host: oleg609-server
Failover mds1 to oleg609-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg609-server: oleg609-server.virtnet: executing set_default_debug -1 all
pdsh@oleg609-client: oleg609-server: ssh exited with exit code 1
Started lustre-MDT0000
01:41:10 (1788932470) targets are mounted
01:41:10 (1788932470) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg609-server
Stopping /mnt/lustre-mds1 (opts:) on oleg609-server
01:41:17 (1788932477) shut down
facet: mds1 facet_host: oleg609-server facet_failover_host: oleg609-server
Failover mds1 to oleg609-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg609-server: oleg609-server.virtnet: executing set_default_debug -1 all
pdsh@oleg609-client: oleg609-server: ssh exited with exit code 1
Started lustre-MDT0000
01:41:42 (1788932502) targets are mounted
01:41:42 (1788932502) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg609-server
Stopping /mnt/lustre-mds1 (opts:) on oleg609-server
01:41:54 (1788932514) shut down
facet: mds1 facet_host: oleg609-server facet_failover_host: oleg609-server
Failover mds1 to oleg609-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg609-server: oleg609-server.virtnet: executing set_default_debug -1 all
pdsh@oleg609-client: oleg609-server: ssh exited with exit code 1
Started lustre-MDT0000
01:42:15 (1788932535) targets are mounted
01:42:15 (1788932535) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg609-server
Stopping /mnt/lustre-mds1 (opts:) on oleg609-server
01:42:32 (1788932552) shut down
facet: mds1 facet_host: oleg609-server facet_failover_host: oleg609-server
Failover mds1 to oleg609-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg609-server: oleg609-server.virtnet: executing set_default_debug -1 all
pdsh@oleg609-client: oleg609-server: ssh exited with exit code 1
Started lustre-MDT0000
01:43:04 (1788932584) targets are mounted
01:43:04 (1788932584) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg609-server
Stopping /mnt/lustre-mds1 (opts:) on oleg609-server
01:43:31 (1788932611) shut down
facet: mds1 facet_host: oleg609-server facet_failover_host: oleg609-server
Failover mds1 to oleg609-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg609-server: oleg609-server.virtnet: executing set_default_debug -1 all
pdsh@oleg609-client: oleg609-server: ssh exited with exit code 1
Started lustre-MDT0000
01:43:55 (1788932635) targets are mounted
01:43:55 (1788932635) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg609-server
Stopping /mnt/lustre-mds1 (opts:) on oleg609-server
01:44:29 (1788932669) shut down
facet: mds1 facet_host: oleg609-server facet_failover_host: oleg609-server
Failover mds1 to oleg609-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg609-server: oleg609-server.virtnet: executing set_default_debug -1 all
pdsh@oleg609-client: oleg609-server: ssh exited with exit code 1
Started lustre-MDT0000
01:45:01 (1788932701) targets are mounted
01:45:01 (1788932701) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg609-server
Stopping /mnt/lustre-mds1 (opts:) on oleg609-server
01:45:40 (1788932740) shut down
facet: mds1 facet_host: oleg609-server facet_failover_host: oleg609-server
Failover mds1 to oleg609-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg609-server: oleg609-server.virtnet: executing set_default_debug -1 all
pdsh@oleg609-client: oleg609-server: ssh exited with exit code 1
Started lustre-MDT0000
01:46:10 (1788932770) targets are mounted
01:46:10 (1788932770) facet_failover done
writemany returned 0