== recovery-small test 51: failover MDS during recovery == 19:59:42 (1788652782)
fail_loc=0x00001310
Failing mds1 on oleg105-server
Stopping /mnt/lustre-mds1 (opts:) on oleg105-server
19:59:51 (1788652791) shut down
facet: mds1 facet_host: oleg105-server facet_failover_host: oleg105-server
Failover mds1 to oleg105-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all
pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1
Started lustre-MDT0000
20:00:18 (1788652818) targets are mounted
20:00:18 (1788652818) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg105-server
Stopping /mnt/lustre-mds1 (opts:) on oleg105-server
20:00:25 (1788652825) shut down
facet: mds1 facet_host: oleg105-server facet_failover_host: oleg105-server
Failover mds1 to oleg105-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all
pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1
Started lustre-MDT0000
20:00:59 (1788652859) targets are mounted
20:00:59 (1788652859) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg105-server
Stopping /mnt/lustre-mds1 (opts:) on oleg105-server
20:01:10 (1788652870) shut down
facet: mds1 facet_host: oleg105-server facet_failover_host: oleg105-server
Failover mds1 to oleg105-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all
pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1
Started lustre-MDT0000
20:01:33 (1788652893) targets are mounted
20:01:33 (1788652893) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg105-server
Stopping /mnt/lustre-mds1 (opts:) on oleg105-server
20:01:50 (1788652910) shut down
facet: mds1 facet_host: oleg105-server facet_failover_host: oleg105-server
Failover mds1 to oleg105-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all
pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1
Started lustre-MDT0000
20:02:15 (1788652935) targets are mounted
20:02:15 (1788652935) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg105-server
Stopping /mnt/lustre-mds1 (opts:) on oleg105-server
20:02:43 (1788652963) shut down
facet: mds1 facet_host: oleg105-server facet_failover_host: oleg105-server
Failover mds1 to oleg105-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all
pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1
Started lustre-MDT0000
20:03:05 (1788652985) targets are mounted
20:03:05 (1788652985) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg105-server
Stopping /mnt/lustre-mds1 (opts:) on oleg105-server
20:03:37 (1788653017) shut down
facet: mds1 facet_host: oleg105-server facet_failover_host: oleg105-server
Failover mds1 to oleg105-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all
pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1
Started lustre-MDT0000
20:04:08 (1788653048) targets are mounted
20:04:08 (1788653048) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg105-server
Stopping /mnt/lustre-mds1 (opts:) on oleg105-server
20:04:45 (1788653085) shut down
facet: mds1 facet_host: oleg105-server facet_failover_host: oleg105-server
Failover mds1 to oleg105-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all
pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1
Started lustre-MDT0000
20:05:07 (1788653107) targets are mounted
20:05:07 (1788653107) facet_failover done
writemany returned 0