== recovery-small test 51: failover MDS during recovery == 01:09:47 (1787375387)
fail_loc=0x00001310
Failing mds1 on oleg104-server
Stopping /mnt/lustre-mds1 (opts:) on oleg104-server
01:09:57 (1787375397) shut down
facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server
Failover mds1 to oleg104-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all
pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1
Started lustre-MDT0000
01:10:23 (1787375423) targets are mounted
01:10:23 (1787375423) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg104-server
Stopping /mnt/lustre-mds1 (opts:) on oleg104-server
01:10:32 (1787375432) shut down
facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server
Failover mds1 to oleg104-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all
pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1
Started lustre-MDT0000
01:10:55 (1787375455) targets are mounted
01:10:55 (1787375455) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg104-server
Stopping /mnt/lustre-mds1 (opts:) on oleg104-server
01:11:06 (1787375466) shut down
facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server
Failover mds1 to oleg104-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all
pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1
Started lustre-MDT0000
01:11:31 (1787375491) targets are mounted
01:11:31 (1787375491) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg104-server
Stopping /mnt/lustre-mds1 (opts:) on oleg104-server
01:11:49 (1787375509) shut down
facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server
Failover mds1 to oleg104-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all
pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1
Started lustre-MDT0000
01:12:21 (1787375541) targets are mounted
01:12:21 (1787375541) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg104-server
Stopping /mnt/lustre-mds1 (opts:) on oleg104-server
01:12:49 (1787375569) shut down
facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server
Failover mds1 to oleg104-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all
pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1
Started lustre-MDT0000
01:13:22 (1787375602) targets are mounted
01:13:22 (1787375602) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg104-server
Stopping /mnt/lustre-mds1 (opts:) on oleg104-server
01:13:55 (1787375635) shut down
facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server
Failover mds1 to oleg104-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all
pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1
Started lustre-MDT0000
01:14:17 (1787375657) targets are mounted
01:14:17 (1787375657) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg104-server
Stopping /mnt/lustre-mds1 (opts:) on oleg104-server
01:14:54 (1787375694) shut down
facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server
Failover mds1 to oleg104-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all
pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1
Started lustre-MDT0000
01:15:24 (1787375724) targets are mounted
01:15:24 (1787375724) facet_failover done
writemany returned 0