== recovery-small test 51: failover MDS during recovery == 20:50:05 (1773103805)
fail_loc=0x00001310
Failing mds1 on oleg424-server
Stopping /mnt/lustre-mds1 (opts:) on oleg424-server
20:50:09 (1773103809) shut down
facet: mds1 facet_host: oleg424-server facet_failover_host: oleg424-server
Failover mds1 to oleg424-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all
pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1
Started lustre-MDT0000
20:50:23 (1773103823) targets are mounted
20:50:23 (1773103823) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg424-server
Stopping /mnt/lustre-mds1 (opts:) on oleg424-server
20:50:26 (1773103826) shut down
facet: mds1 facet_host: oleg424-server facet_failover_host: oleg424-server
Failover mds1 to oleg424-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all
pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1
Started lustre-MDT0000
20:50:40 (1773103840) targets are mounted
20:50:40 (1773103840) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg424-server
Stopping /mnt/lustre-mds1 (opts:) on oleg424-server
20:50:48 (1773103848) shut down
facet: mds1 facet_host: oleg424-server facet_failover_host: oleg424-server
Failover mds1 to oleg424-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all
pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1
Started lustre-MDT0000
20:51:02 (1773103862) targets are mounted
20:51:02 (1773103862) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg424-server
Stopping /mnt/lustre-mds1 (opts:) on oleg424-server
20:51:15 (1773103875) shut down
facet: mds1 facet_host: oleg424-server facet_failover_host: oleg424-server
Failover mds1 to oleg424-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all
pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1
Started lustre-MDT0000
20:51:29 (1773103889) targets are mounted
20:51:29 (1773103889) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg424-server
Stopping /mnt/lustre-mds1 (opts:) on oleg424-server
20:51:51 (1773103911) shut down
facet: mds1 facet_host: oleg424-server facet_failover_host: oleg424-server
Failover mds1 to oleg424-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all
pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1
Started lustre-MDT0000
20:52:05 (1773103925) targets are mounted
20:52:05 (1773103925) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg424-server
Stopping /mnt/lustre-mds1 (opts:) on oleg424-server
20:52:32 (1773103952) shut down
facet: mds1 facet_host: oleg424-server facet_failover_host: oleg424-server
Failover mds1 to oleg424-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all
pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1
Started lustre-MDT0000
20:52:46 (1773103966) targets are mounted
20:52:46 (1773103966) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg424-server
Stopping /mnt/lustre-mds1 (opts:) on oleg424-server
20:53:18 (1773103998) shut down
facet: mds1 facet_host: oleg424-server facet_failover_host: oleg424-server
Failover mds1 to oleg424-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all
pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1
Started lustre-MDT0000
20:53:32 (1773104012) targets are mounted
20:53:32 (1773104012) facet_failover done
writemany returned 0