== recovery-small test 51: failover MDS during recovery == 18:46:59 (1789080419)
fail_loc=0x00001310
Failing mds1 on oleg244-server
Stopping /mnt/lustre-mds1 (opts:) on oleg244-server
18:47:09 (1789080429) shut down
facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server
Failover mds1 to oleg244-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all
pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1
Started lustre-MDT0000
18:47:41 (1789080461) targets are mounted
18:47:41 (1789080461) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg244-server
Stopping /mnt/lustre-mds1 (opts:) on oleg244-server
18:47:50 (1789080470) shut down
facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server
Failover mds1 to oleg244-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all
pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1
Started lustre-MDT0000
18:48:26 (1789080506) targets are mounted
18:48:26 (1789080506) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg244-server
Stopping /mnt/lustre-mds1 (opts:) on oleg244-server
18:48:39 (1789080519) shut down
facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server
Failover mds1 to oleg244-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all
pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1
Started lustre-MDT0000
18:49:12 (1789080552) targets are mounted
18:49:12 (1789080552) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg244-server
Stopping /mnt/lustre-mds1 (opts:) on oleg244-server
18:49:30 (1789080570) shut down
facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server
Failover mds1 to oleg244-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all
pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1
Started lustre-MDT0000
18:50:04 (1789080604) targets are mounted
18:50:04 (1789080604) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg244-server
Stopping /mnt/lustre-mds1 (opts:) on oleg244-server
18:50:33 (1789080633) shut down
facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server
Failover mds1 to oleg244-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all
pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1
Started lustre-MDT0000
18:50:58 (1789080658) targets are mounted
18:50:58 (1789080658) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg244-server
Stopping /mnt/lustre-mds1 (opts:) on oleg244-server
18:51:33 (1789080693) shut down
facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server
Failover mds1 to oleg244-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all
pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1
Started lustre-MDT0000
18:51:57 (1789080717) targets are mounted
18:51:57 (1789080717) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg244-server
Stopping /mnt/lustre-mds1 (opts:) on oleg244-server
18:52:35 (1789080755) shut down
facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server
Failover mds1 to oleg244-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all
pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1
Started lustre-MDT0000
18:53:09 (1789080789) targets are mounted
18:53:09 (1789080789) facet_failover done
writemany returned 0