== recovery-small test 51: failover MDS during recovery == 22:37:39 (1788835059)
fail_loc=0x00001310
Failing mds1 on oleg618-server
Stopping /mnt/lustre-mds1 (opts:) on oleg618-server
22:37:49 (1788835069) shut down
facet: mds1 facet_host: oleg618-server facet_failover_host: oleg618-server
Failover mds1 to oleg618-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all
pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1
Started lustre-MDT0000
22:38:20 (1788835100) targets are mounted
22:38:20 (1788835100) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg618-server
Stopping /mnt/lustre-mds1 (opts:) on oleg618-server
22:38:27 (1788835107) shut down
facet: mds1 facet_host: oleg618-server facet_failover_host: oleg618-server
Failover mds1 to oleg618-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all
pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1
Started lustre-MDT0000
22:38:51 (1788835131) targets are mounted
22:38:51 (1788835131) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg618-server
Stopping /mnt/lustre-mds1 (opts:) on oleg618-server
22:39:03 (1788835143) shut down
facet: mds1 facet_host: oleg618-server facet_failover_host: oleg618-server
Failover mds1 to oleg618-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all
pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1
Started lustre-MDT0000
22:39:27 (1788835167) targets are mounted
22:39:27 (1788835167) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg618-server
Stopping /mnt/lustre-mds1 (opts:) on oleg618-server
22:39:45 (1788835185) shut down
facet: mds1 facet_host: oleg618-server facet_failover_host: oleg618-server
Failover mds1 to oleg618-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all
pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1
Started lustre-MDT0000
22:40:19 (1788835219) targets are mounted
22:40:19 (1788835219) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg618-server
Stopping /mnt/lustre-mds1 (opts:) on oleg618-server
22:40:46 (1788835246) shut down
facet: mds1 facet_host: oleg618-server facet_failover_host: oleg618-server
Failover mds1 to oleg618-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all
pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1
Started lustre-MDT0000
22:41:20 (1788835280) targets are mounted
22:41:20 (1788835280) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg618-server
Stopping /mnt/lustre-mds1 (opts:) on oleg618-server
22:41:52 (1788835312) shut down
facet: mds1 facet_host: oleg618-server facet_failover_host: oleg618-server
Failover mds1 to oleg618-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all
pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1
Started lustre-MDT0000
22:42:18 (1788835338) targets are mounted
22:42:18 (1788835338) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg618-server
Stopping /mnt/lustre-mds1 (opts:) on oleg618-server
22:42:57 (1788835377) shut down
facet: mds1 facet_host: oleg618-server facet_failover_host: oleg618-server
Failover mds1 to oleg618-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg618-server: oleg618-server.virtnet: executing set_default_debug -1 all
pdsh@oleg618-client: oleg618-server: ssh exited with exit code 1
Started lustre-MDT0000
22:43:31 (1788835411) targets are mounted
22:43:31 (1788835411) facet_failover done
writemany returned 0