== recovery-small test 51: failover MDS during recovery == 07:16:27 (1789125387)
fail_loc=0x00001310
Failing mds1 on oleg421-server
Stopping /mnt/lustre-mds1 (opts:) on oleg421-server
07:16:38 (1789125398) shut down
facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server
Failover mds1 to oleg421-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all
pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1
Started lustre-MDT0000
07:17:10 (1789125430) targets are mounted
07:17:10 (1789125430) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg421-server
Stopping /mnt/lustre-mds1 (opts:) on oleg421-server
07:17:17 (1789125437) shut down
facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server
Failover mds1 to oleg421-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all
pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1
Started lustre-MDT0000
07:17:40 (1789125460) targets are mounted
07:17:40 (1789125460) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg421-server
Stopping /mnt/lustre-mds1 (opts:) on oleg421-server
07:17:54 (1789125474) shut down
facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server
Failover mds1 to oleg421-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all
pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1
Started lustre-MDT0000
07:18:27 (1789125507) targets are mounted
07:18:27 (1789125507) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg421-server
Stopping /mnt/lustre-mds1 (opts:) on oleg421-server
07:18:45 (1789125525) shut down
facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server
Failover mds1 to oleg421-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all
pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1
Started lustre-MDT0000
07:19:10 (1789125550) targets are mounted
07:19:10 (1789125550) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg421-server
Stopping /mnt/lustre-mds1 (opts:) on oleg421-server
07:19:37 (1789125577) shut down
facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server
Failover mds1 to oleg421-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all
pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1
Started lustre-MDT0000
07:20:05 (1789125605) targets are mounted
07:20:05 (1789125605) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg421-server
Stopping /mnt/lustre-mds1 (opts:) on oleg421-server
07:20:38 (1789125638) shut down
facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server
Failover mds1 to oleg421-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all
pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1
Started lustre-MDT0000
07:21:08 (1789125668) targets are mounted
07:21:08 (1789125668) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg421-server
Stopping /mnt/lustre-mds1 (opts:) on oleg421-server
07:21:46 (1789125706) shut down
facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server
Failover mds1 to oleg421-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all
pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1
Started lustre-MDT0000
07:22:21 (1789125741) targets are mounted
07:22:21 (1789125741) facet_failover done
writemany returned 0