== recovery-small test 51: failover MDS during recovery == 07:18:41 (1787829521)
fail_loc=0x00001310
Failing mds1 on oleg444-server
Stopping /mnt/lustre-mds1 (opts:) on oleg444-server
07:18:49 (1787829529) shut down
facet: mds1 facet_host: oleg444-server facet_failover_host: oleg444-server
Failover mds1 to oleg444-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all
pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1
Started lustre-MDT0000
07:19:14 (1787829554) targets are mounted
07:19:14 (1787829554) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg444-server
Stopping /mnt/lustre-mds1 (opts:) on oleg444-server
07:19:22 (1787829562) shut down
facet: mds1 facet_host: oleg444-server facet_failover_host: oleg444-server
Failover mds1 to oleg444-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all
pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1
Started lustre-MDT0000
07:19:44 (1787829584) targets are mounted
07:19:44 (1787829584) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg444-server
Stopping /mnt/lustre-mds1 (opts:) on oleg444-server
07:19:56 (1787829596) shut down
facet: mds1 facet_host: oleg444-server facet_failover_host: oleg444-server
Failover mds1 to oleg444-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all
pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1
Started lustre-MDT0000
07:20:21 (1787829621) targets are mounted
07:20:21 (1787829621) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg444-server
Stopping /mnt/lustre-mds1 (opts:) on oleg444-server
07:20:39 (1787829639) shut down
facet: mds1 facet_host: oleg444-server facet_failover_host: oleg444-server
Failover mds1 to oleg444-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all
pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1
Started lustre-MDT0000
07:21:03 (1787829663) targets are mounted
07:21:03 (1787829663) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg444-server
Stopping /mnt/lustre-mds1 (opts:) on oleg444-server
07:21:32 (1787829692) shut down
facet: mds1 facet_host: oleg444-server facet_failover_host: oleg444-server
Failover mds1 to oleg444-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all
pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1
Started lustre-MDT0000
07:22:05 (1787829725) targets are mounted
07:22:05 (1787829725) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg444-server
Stopping /mnt/lustre-mds1 (opts:) on oleg444-server
07:22:40 (1787829760) shut down
facet: mds1 facet_host: oleg444-server facet_failover_host: oleg444-server
Failover mds1 to oleg444-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all
pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1
Started lustre-MDT0000
07:23:15 (1787829795) targets are mounted
07:23:15 (1787829795) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg444-server
Stopping /mnt/lustre-mds1 (opts:) on oleg444-server
07:23:54 (1787829834) shut down
facet: mds1 facet_host: oleg444-server facet_failover_host: oleg444-server
Failover mds1 to oleg444-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all
pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1
Started lustre-MDT0000
07:24:28 (1787829868) targets are mounted
07:24:28 (1787829868) facet_failover done
writemany returned 0