== recovery-small test 51: failover MDS during recovery == 14:27:21 (1779128841)
fail_loc=0x00001310
Failing mds1 on oleg215-server
Stopping /mnt/lustre-mds1 (opts:) on oleg215-server
14:27:31 (1779128851) shut down
facet: mds1 facet_host: oleg215-server facet_failover_host: oleg215-server
Failover mds1 to oleg215-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all
pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1
Started lustre-MDT0000
14:28:07 (1779128887) targets are mounted
14:28:07 (1779128887) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg215-server
Stopping /mnt/lustre-mds1 (opts:) on oleg215-server
14:28:16 (1779128896) shut down
facet: mds1 facet_host: oleg215-server facet_failover_host: oleg215-server
Failover mds1 to oleg215-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all
pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1
Started lustre-MDT0000
14:28:39 (1779128919) targets are mounted
14:28:39 (1779128919) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg215-server
Stopping /mnt/lustre-mds1 (opts:) on oleg215-server
14:28:52 (1779128932) shut down
facet: mds1 facet_host: oleg215-server facet_failover_host: oleg215-server
Failover mds1 to oleg215-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all
pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1
Started lustre-MDT0000
14:29:15 (1779128955) targets are mounted
14:29:15 (1779128955) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg215-server
Stopping /mnt/lustre-mds1 (opts:) on oleg215-server
14:29:33 (1779128973) shut down
facet: mds1 facet_host: oleg215-server facet_failover_host: oleg215-server
Failover mds1 to oleg215-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all
pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1
Started lustre-MDT0000
14:29:57 (1779128997) targets are mounted
14:29:57 (1779128997) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg215-server
Stopping /mnt/lustre-mds1 (opts:) on oleg215-server
14:30:25 (1779129025) shut down
facet: mds1 facet_host: oleg215-server facet_failover_host: oleg215-server
Failover mds1 to oleg215-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all
pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1
Started lustre-MDT0000
14:30:53 (1779129053) targets are mounted
14:30:53 (1779129053) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg215-server
Stopping /mnt/lustre-mds1 (opts:) on oleg215-server
14:31:29 (1779129089) shut down
facet: mds1 facet_host: oleg215-server facet_failover_host: oleg215-server
Failover mds1 to oleg215-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all
pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1
Started lustre-MDT0000
14:31:51 (1779129111) targets are mounted
14:31:51 (1779129111) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg215-server
Stopping /mnt/lustre-mds1 (opts:) on oleg215-server
14:32:29 (1779129149) shut down
facet: mds1 facet_host: oleg215-server facet_failover_host: oleg215-server
Failover mds1 to oleg215-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all
pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1
Started lustre-MDT0000
14:32:53 (1779129173) targets are mounted
14:32:53 (1779129173) facet_failover done
writemany returned 0