== recovery-small test 51: failover MDS during recovery == 08:00:24 (1788523224)
fail_loc=0x00001310
Failing mds1 on oleg311-server
Stopping /mnt/lustre-mds1 (opts:) on oleg311-server
08:00:32 (1788523232) shut down
facet: mds1 facet_host: oleg311-server facet_failover_host: oleg311-server
Failover mds1 to oleg311-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all
pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1
Started lustre-MDT0000
08:00:54 (1788523254) targets are mounted
08:00:54 (1788523254) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg311-server
Stopping /mnt/lustre-mds1 (opts:) on oleg311-server
08:01:01 (1788523261) shut down
facet: mds1 facet_host: oleg311-server facet_failover_host: oleg311-server
Failover mds1 to oleg311-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all
pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1
Started lustre-MDT0000
08:01:24 (1788523284) targets are mounted
08:01:24 (1788523284) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg311-server
Stopping /mnt/lustre-mds1 (opts:) on oleg311-server
08:01:34 (1788523294) shut down
facet: mds1 facet_host: oleg311-server facet_failover_host: oleg311-server
Failover mds1 to oleg311-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all
pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1
Started lustre-MDT0000
08:01:56 (1788523316) targets are mounted
08:01:56 (1788523316) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg311-server
Stopping /mnt/lustre-mds1 (opts:) on oleg311-server
08:02:12 (1788523332) shut down
facet: mds1 facet_host: oleg311-server facet_failover_host: oleg311-server
Failover mds1 to oleg311-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all
pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1
Started lustre-MDT0000
08:02:42 (1788523362) targets are mounted
08:02:42 (1788523362) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg311-server
Stopping /mnt/lustre-mds1 (opts:) on oleg311-server
08:03:10 (1788523390) shut down
facet: mds1 facet_host: oleg311-server facet_failover_host: oleg311-server
Failover mds1 to oleg311-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all
pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1
Started lustre-MDT0000
08:03:44 (1788523424) targets are mounted
08:03:44 (1788523424) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg311-server
Stopping /mnt/lustre-mds1 (opts:) on oleg311-server
08:04:15 (1788523455) shut down
facet: mds1 facet_host: oleg311-server facet_failover_host: oleg311-server
Failover mds1 to oleg311-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all
pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1
Started lustre-MDT0000
08:04:47 (1788523487) targets are mounted
08:04:47 (1788523487) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg311-server
Stopping /mnt/lustre-mds1 (opts:) on oleg311-server
08:05:25 (1788523525) shut down
facet: mds1 facet_host: oleg311-server facet_failover_host: oleg311-server
Failover mds1 to oleg311-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all
pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1
Started lustre-MDT0000
08:05:47 (1788523547) targets are mounted
08:05:47 (1788523547) facet_failover done
writemany returned 0