== recovery-small test 51: failover MDS during recovery == 07:14:38 (1781694878)
fail_loc=0x00001310
Failing mds1 on oleg241-server
Stopping /mnt/lustre-mds1 (opts:) on oleg241-server
07:14:42 (1781694882) shut down
facet: mds1 facet_host: oleg241-server facet_failover_host: oleg241-server
Failover mds1 to oleg241-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
07:14:57 (1781694897) targets are mounted
07:14:57 (1781694897) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg241-server
Stopping /mnt/lustre-mds1 (opts:) on oleg241-server
07:15:00 (1781694900) shut down
facet: mds1 facet_host: oleg241-server facet_failover_host: oleg241-server
Failover mds1 to oleg241-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
07:15:16 (1781694916) targets are mounted
07:15:16 (1781694916) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg241-server
Stopping /mnt/lustre-mds1 (opts:) on oleg241-server
07:15:23 (1781694923) shut down
facet: mds1 facet_host: oleg241-server facet_failover_host: oleg241-server
Failover mds1 to oleg241-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
07:15:38 (1781694938) targets are mounted
07:15:38 (1781694938) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg241-server
Stopping /mnt/lustre-mds1 (opts:) on oleg241-server
07:15:51 (1781694951) shut down
facet: mds1 facet_host: oleg241-server facet_failover_host: oleg241-server
Failover mds1 to oleg241-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
07:16:06 (1781694966) targets are mounted
07:16:06 (1781694966) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg241-server
Stopping /mnt/lustre-mds1 (opts:) on oleg241-server
07:16:29 (1781694989) shut down
facet: mds1 facet_host: oleg241-server facet_failover_host: oleg241-server
Failover mds1 to oleg241-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
07:16:44 (1781695004) targets are mounted
07:16:44 (1781695004) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg241-server
Stopping /mnt/lustre-mds1 (opts:) on oleg241-server
07:17:12 (1781695032) shut down
facet: mds1 facet_host: oleg241-server facet_failover_host: oleg241-server
Failover mds1 to oleg241-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
07:17:27 (1781695047) targets are mounted
07:17:27 (1781695047) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg241-server
Stopping /mnt/lustre-mds1 (opts:) on oleg241-server
07:17:59 (1781695079) shut down
facet: mds1 facet_host: oleg241-server facet_failover_host: oleg241-server
Failover mds1 to oleg241-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all
pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1
Started lustre-MDT0000
07:18:15 (1781695095) targets are mounted
07:18:15 (1781695095) facet_failover done
writemany returned 0