== recovery-small test 51: failover MDS during recovery == 21:43:35 (1781228615)
fail_loc=0x00001310
Failing mds1 on oleg203-server
Stopping /mnt/lustre-mds1 (opts:) on oleg203-server
21:43:44 (1781228624) shut down
facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server
Failover mds1 to oleg203-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all
pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1
Started lustre-MDT0000
21:44:18 (1781228658) targets are mounted
21:44:18 (1781228658) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg203-server
Stopping /mnt/lustre-mds1 (opts:) on oleg203-server
21:44:26 (1781228666) shut down
facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server
Failover mds1 to oleg203-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all
pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1
Started lustre-MDT0000
21:44:59 (1781228699) targets are mounted
21:44:59 (1781228699) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg203-server
Stopping /mnt/lustre-mds1 (opts:) on oleg203-server
21:45:12 (1781228712) shut down
facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server
Failover mds1 to oleg203-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all
pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1
Started lustre-MDT0000
21:45:45 (1781228745) targets are mounted
21:45:45 (1781228745) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg203-server
Stopping /mnt/lustre-mds1 (opts:) on oleg203-server
21:46:03 (1781228763) shut down
facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server
Failover mds1 to oleg203-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all
pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1
Started lustre-MDT0000
21:46:31 (1781228791) targets are mounted
21:46:31 (1781228791) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg203-server
Stopping /mnt/lustre-mds1 (opts:) on oleg203-server
21:47:03 (1781228823) shut down
facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server
Failover mds1 to oleg203-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all
pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1
Started lustre-MDT0000
21:47:41 (1781228861) targets are mounted
21:47:41 (1781228861) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg203-server
Stopping /mnt/lustre-mds1 (opts:) on oleg203-server
21:48:15 (1781228895) shut down
facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server
Failover mds1 to oleg203-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all
pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1
Started lustre-MDT0000
21:48:51 (1781228931) targets are mounted
21:48:51 (1781228931) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg203-server
Stopping /mnt/lustre-mds1 (opts:) on oleg203-server
21:49:33 (1781228973) shut down
facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server
Failover mds1 to oleg203-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all
pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1
Started lustre-MDT0000
21:49:57 (1781228997) targets are mounted
21:49:57 (1781228997) facet_failover done
writemany returned 0