== recovery-small test 51: failover MDS during recovery == 21:12:02 (1788570722)
fail_loc=0x00001310
Failing mds1 on oleg326-server
Stopping /mnt/lustre-mds1 (opts:) on oleg326-server
21:12:16 (1788570736) shut down
facet: mds1 facet_host: oleg326-server facet_failover_host: oleg326-server
Failover mds1 to oleg326-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all
pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1
Started lustre-MDT0000
21:12:41 (1788570761) targets are mounted
21:12:41 (1788570761) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg326-server
Stopping /mnt/lustre-mds1 (opts:) on oleg326-server
21:12:52 (1788570772) shut down
facet: mds1 facet_host: oleg326-server facet_failover_host: oleg326-server
Failover mds1 to oleg326-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all
pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1
Started lustre-MDT0000
21:13:15 (1788570795) targets are mounted
21:13:15 (1788570795) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg326-server
Stopping /mnt/lustre-mds1 (opts:) on oleg326-server
21:13:26 (1788570806) shut down
facet: mds1 facet_host: oleg326-server facet_failover_host: oleg326-server
Failover mds1 to oleg326-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all
pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1
Started lustre-MDT0000
21:13:49 (1788570829) targets are mounted
21:13:49 (1788570829) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg326-server
Stopping /mnt/lustre-mds1 (opts:) on oleg326-server
21:14:08 (1788570848) shut down
facet: mds1 facet_host: oleg326-server facet_failover_host: oleg326-server
Failover mds1 to oleg326-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all
pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1
Started lustre-MDT0000
21:14:31 (1788570871) targets are mounted
21:14:31 (1788570871) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg326-server
Stopping /mnt/lustre-mds1 (opts:) on oleg326-server
21:14:59 (1788570899) shut down
facet: mds1 facet_host: oleg326-server facet_failover_host: oleg326-server
Failover mds1 to oleg326-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all
pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1
Started lustre-MDT0000
21:15:21 (1788570921) targets are mounted
21:15:21 (1788570921) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg326-server
Stopping /mnt/lustre-mds1 (opts:) on oleg326-server
21:15:54 (1788570954) shut down
facet: mds1 facet_host: oleg326-server facet_failover_host: oleg326-server
Failover mds1 to oleg326-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all
pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1
Started lustre-MDT0000
21:16:20 (1788570980) targets are mounted
21:16:20 (1788570980) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg326-server
Stopping /mnt/lustre-mds1 (opts:) on oleg326-server
21:16:59 (1788571019) shut down
facet: mds1 facet_host: oleg326-server facet_failover_host: oleg326-server
Failover mds1 to oleg326-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all
pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1
Started lustre-MDT0000
21:17:32 (1788571052) targets are mounted
21:17:32 (1788571052) facet_failover done
writemany returned 0