== recovery-small test 51: failover MDS during recovery == 09:21:25 (1788355285)
fail_loc=0x00001310
Failing mds1 on oleg225-server
Stopping /mnt/lustre-mds1 (opts:) on oleg225-server
09:21:29 (1788355289) shut down
facet: mds1 facet_host: oleg225-server facet_failover_host: oleg225-server
Failover mds1 to oleg225-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
09:21:43 (1788355303) targets are mounted
09:21:43 (1788355303) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg225-server
Stopping /mnt/lustre-mds1 (opts:) on oleg225-server
09:21:46 (1788355306) shut down
facet: mds1 facet_host: oleg225-server facet_failover_host: oleg225-server
Failover mds1 to oleg225-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
09:22:00 (1788355320) targets are mounted
09:22:00 (1788355320) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg225-server
Stopping /mnt/lustre-mds1 (opts:) on oleg225-server
09:22:07 (1788355327) shut down
facet: mds1 facet_host: oleg225-server facet_failover_host: oleg225-server
Failover mds1 to oleg225-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
09:22:21 (1788355341) targets are mounted
09:22:21 (1788355341) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg225-server
Stopping /mnt/lustre-mds1 (opts:) on oleg225-server
09:22:34 (1788355354) shut down
facet: mds1 facet_host: oleg225-server facet_failover_host: oleg225-server
Failover mds1 to oleg225-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
09:22:48 (1788355368) targets are mounted
09:22:48 (1788355368) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg225-server
Stopping /mnt/lustre-mds1 (opts:) on oleg225-server
09:23:10 (1788355390) shut down
facet: mds1 facet_host: oleg225-server facet_failover_host: oleg225-server
Failover mds1 to oleg225-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
09:23:25 (1788355405) targets are mounted
09:23:25 (1788355405) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg225-server
Stopping /mnt/lustre-mds1 (opts:) on oleg225-server
09:23:53 (1788355433) shut down
facet: mds1 facet_host: oleg225-server facet_failover_host: oleg225-server
Failover mds1 to oleg225-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
09:24:07 (1788355447) targets are mounted
09:24:07 (1788355447) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg225-server
Stopping /mnt/lustre-mds1 (opts:) on oleg225-server
09:24:40 (1788355480) shut down
facet: mds1 facet_host: oleg225-server facet_failover_host: oleg225-server
Failover mds1 to oleg225-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all
pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1
Started lustre-MDT0000
09:24:54 (1788355494) targets are mounted
09:24:54 (1788355494) facet_failover done
writemany returned 0