== recovery-small test 51: failover MDS during recovery == 12:53:05 (1774889585)
fail_loc=0x00001310
Failing mds1 on oleg611-server
Stopping /mnt/lustre-mds1 (opts:) on oleg611-server
12:53:11 (1774889591) shut down
facet: mds1 facet_host: oleg611-server facet_failover_host: oleg611-server
Failover mds1 to oleg611-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg611-server: oleg611-server.virtnet: executing set_default_debug -1 all
pdsh@oleg611-client: oleg611-server: ssh exited with exit code 1
Started lustre-MDT0000
12:53:30 (1774889610) targets are mounted
12:53:30 (1774889610) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg611-server
Stopping /mnt/lustre-mds1 (opts:) on oleg611-server
12:53:35 (1774889615) shut down
facet: mds1 facet_host: oleg611-server facet_failover_host: oleg611-server
Failover mds1 to oleg611-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg611-server: oleg611-server.virtnet: executing set_default_debug -1 all
pdsh@oleg611-client: oleg611-server: ssh exited with exit code 1
Started lustre-MDT0000
12:53:55 (1774889635) targets are mounted
12:53:55 (1774889635) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg611-server
Stopping /mnt/lustre-mds1 (opts:) on oleg611-server
12:54:06 (1774889646) shut down
facet: mds1 facet_host: oleg611-server facet_failover_host: oleg611-server
Failover mds1 to oleg611-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg611-server: oleg611-server.virtnet: executing set_default_debug -1 all
pdsh@oleg611-client: oleg611-server: ssh exited with exit code 1
Started lustre-MDT0000
12:54:24 (1774889664) targets are mounted
12:54:24 (1774889664) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg611-server
Stopping /mnt/lustre-mds1 (opts:) on oleg611-server
12:54:40 (1774889680) shut down
facet: mds1 facet_host: oleg611-server facet_failover_host: oleg611-server
Failover mds1 to oleg611-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg611-server: oleg611-server.virtnet: executing set_default_debug -1 all
pdsh@oleg611-client: oleg611-server: ssh exited with exit code 1
Started lustre-MDT0000
12:54:57 (1774889697) targets are mounted
12:54:57 (1774889697) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg611-server
Stopping /mnt/lustre-mds1 (opts:) on oleg611-server
12:55:22 (1774889722) shut down
facet: mds1 facet_host: oleg611-server facet_failover_host: oleg611-server
Failover mds1 to oleg611-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg611-server: oleg611-server.virtnet: executing set_default_debug -1 all
pdsh@oleg611-client: oleg611-server: ssh exited with exit code 1
Started lustre-MDT0000
12:55:41 (1774889741) targets are mounted
12:55:41 (1774889741) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg611-server
Stopping /mnt/lustre-mds1 (opts:) on oleg611-server
12:56:11 (1774889771) shut down
facet: mds1 facet_host: oleg611-server facet_failover_host: oleg611-server
Failover mds1 to oleg611-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg611-server: oleg611-server.virtnet: executing set_default_debug -1 all
pdsh@oleg611-client: oleg611-server: ssh exited with exit code 1
Started lustre-MDT0000
12:56:31 (1774889791) targets are mounted
12:56:31 (1774889791) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg611-server
Stopping /mnt/lustre-mds1 (opts:) on oleg611-server
12:57:07 (1774889827) shut down
facet: mds1 facet_host: oleg611-server facet_failover_host: oleg611-server
Failover mds1 to oleg611-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg611-server: oleg611-server.virtnet: executing set_default_debug -1 all
pdsh@oleg611-client: oleg611-server: ssh exited with exit code 1
Started lustre-MDT0000
12:57:25 (1774889845) targets are mounted
12:57:25 (1774889845) facet_failover done
writemany returned 0