== recovery-small test 51: failover MDS during recovery == 09:17:40 (1788355060)
fail_loc=0x00001310
Failing mds1 on oleg145-server
Stopping /mnt/lustre-mds1 (opts:) on oleg145-server
09:17:44 (1788355064) shut down
facet: mds1 facet_host: oleg145-server facet_failover_host: oleg145-server
Failover mds1 to oleg145-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all
pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1
Started lustre-MDT0000
09:17:59 (1788355079) targets are mounted
09:17:59 (1788355079) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg145-server
Stopping /mnt/lustre-mds1 (opts:) on oleg145-server
09:18:01 (1788355081) shut down
facet: mds1 facet_host: oleg145-server facet_failover_host: oleg145-server
Failover mds1 to oleg145-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all
pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1
Started lustre-MDT0000
09:18:16 (1788355096) targets are mounted
09:18:16 (1788355096) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg145-server
Stopping /mnt/lustre-mds1 (opts:) on oleg145-server
09:18:23 (1788355103) shut down
facet: mds1 facet_host: oleg145-server facet_failover_host: oleg145-server
Failover mds1 to oleg145-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all
pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1
Started lustre-MDT0000
09:18:38 (1788355118) targets are mounted
09:18:38 (1788355118) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg145-server
Stopping /mnt/lustre-mds1 (opts:) on oleg145-server
09:18:51 (1788355131) shut down
facet: mds1 facet_host: oleg145-server facet_failover_host: oleg145-server
Failover mds1 to oleg145-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all
pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1
Started lustre-MDT0000
09:19:05 (1788355145) targets are mounted
09:19:05 (1788355145) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg145-server
Stopping /mnt/lustre-mds1 (opts:) on oleg145-server
09:19:28 (1788355168) shut down
facet: mds1 facet_host: oleg145-server facet_failover_host: oleg145-server
Failover mds1 to oleg145-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all
pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1
Started lustre-MDT0000
09:19:43 (1788355183) targets are mounted
09:19:43 (1788355183) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg145-server
Stopping /mnt/lustre-mds1 (opts:) on oleg145-server
09:20:10 (1788355210) shut down
facet: mds1 facet_host: oleg145-server facet_failover_host: oleg145-server
Failover mds1 to oleg145-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all
pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1
Started lustre-MDT0000
09:20:25 (1788355225) targets are mounted
09:20:25 (1788355225) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg145-server
Stopping /mnt/lustre-mds1 (opts:) on oleg145-server
09:20:58 (1788355258) shut down
facet: mds1 facet_host: oleg145-server facet_failover_host: oleg145-server
Failover mds1 to oleg145-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all
pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1
Started lustre-MDT0000
09:21:13 (1788355273) targets are mounted
09:21:13 (1788355273) facet_failover done
writemany returned 0