== recovery-small test 51: failover MDS during recovery == 07:31:42 (1788867102)
fail_loc=0x00001310
Failing mds1 on oleg335-server
Stopping /mnt/lustre-mds1 (opts:) on oleg335-server
07:31:50 (1788867110) shut down
facet: mds1 facet_host: oleg335-server facet_failover_host: oleg335-server
Failover mds1 to oleg335-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all
pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1
Started lustre-MDT0000
07:32:14 (1788867134) targets are mounted
07:32:14 (1788867134) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg335-server
Stopping /mnt/lustre-mds1 (opts:) on oleg335-server
07:32:21 (1788867141) shut down
facet: mds1 facet_host: oleg335-server facet_failover_host: oleg335-server
Failover mds1 to oleg335-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all
pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1
Started lustre-MDT0000
07:32:44 (1788867164) targets are mounted
07:32:44 (1788867164) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg335-server
Stopping /mnt/lustre-mds1 (opts:) on oleg335-server
07:32:55 (1788867175) shut down
facet: mds1 facet_host: oleg335-server facet_failover_host: oleg335-server
Failover mds1 to oleg335-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all
pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1
Started lustre-MDT0000
07:33:28 (1788867208) targets are mounted
07:33:28 (1788867208) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg335-server
Stopping /mnt/lustre-mds1 (opts:) on oleg335-server
07:33:45 (1788867225) shut down
facet: mds1 facet_host: oleg335-server facet_failover_host: oleg335-server
Failover mds1 to oleg335-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all
pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1
Started lustre-MDT0000
07:34:18 (1788867258) targets are mounted
07:34:18 (1788867258) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg335-server
Stopping /mnt/lustre-mds1 (opts:) on oleg335-server
07:34:44 (1788867284) shut down
facet: mds1 facet_host: oleg335-server facet_failover_host: oleg335-server
Failover mds1 to oleg335-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all
pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1
Started lustre-MDT0000
07:35:09 (1788867309) targets are mounted
07:35:09 (1788867309) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg335-server
Stopping /mnt/lustre-mds1 (opts:) on oleg335-server
07:35:41 (1788867341) shut down
facet: mds1 facet_host: oleg335-server facet_failover_host: oleg335-server
Failover mds1 to oleg335-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all
pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1
Started lustre-MDT0000
07:36:04 (1788867364) targets are mounted
07:36:04 (1788867364) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg335-server
Stopping /mnt/lustre-mds1 (opts:) on oleg335-server
07:36:40 (1788867400) shut down
facet: mds1 facet_host: oleg335-server facet_failover_host: oleg335-server
Failover mds1 to oleg335-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all
pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1
Started lustre-MDT0000
07:37:03 (1788867423) targets are mounted
07:37:03 (1788867423) facet_failover done
writemany returned 0