== recovery-small test 51: failover MDS during recovery == 13:05:46 (1790787946)
fail_loc=0x00001310
Failing mds1 on oleg238-server
Stopping /mnt/lustre-mds1 (opts:) on oleg238-server
13:05:56 (1790787956) shut down
facet: mds1 facet_host: oleg238-server facet_failover_host: oleg238-server
Failover mds1 to oleg238-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
13:06:31 (1790787991) targets are mounted
13:06:31 (1790787991) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg238-server
Stopping /mnt/lustre-mds1 (opts:) on oleg238-server
13:06:42 (1790788002) shut down
facet: mds1 facet_host: oleg238-server facet_failover_host: oleg238-server
Failover mds1 to oleg238-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
13:07:20 (1790788040) targets are mounted
13:07:20 (1790788040) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg238-server
Stopping /mnt/lustre-mds1 (opts:) on oleg238-server
13:07:39 (1790788059) shut down
facet: mds1 facet_host: oleg238-server facet_failover_host: oleg238-server
Failover mds1 to oleg238-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
13:08:16 (1790788096) targets are mounted
13:08:16 (1790788096) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg238-server
Stopping /mnt/lustre-mds1 (opts:) on oleg238-server
13:08:41 (1790788121) shut down
facet: mds1 facet_host: oleg238-server facet_failover_host: oleg238-server
Failover mds1 to oleg238-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
13:09:19 (1790788159) targets are mounted
13:09:19 (1790788159) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg238-server
Stopping /mnt/lustre-mds1 (opts:) on oleg238-server
13:09:48 (1790788188) shut down
facet: mds1 facet_host: oleg238-server facet_failover_host: oleg238-server
Failover mds1 to oleg238-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
13:10:22 (1790788222) targets are mounted
13:10:22 (1790788222) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg238-server
Stopping /mnt/lustre-mds1 (opts:) on oleg238-server
13:10:57 (1790788257) shut down
facet: mds1 facet_host: oleg238-server facet_failover_host: oleg238-server
Failover mds1 to oleg238-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
13:11:24 (1790788284) targets are mounted
13:11:24 (1790788284) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg238-server
Stopping /mnt/lustre-mds1 (opts:) on oleg238-server
13:12:05 (1790788325) shut down
facet: mds1 facet_host: oleg238-server facet_failover_host: oleg238-server
Failover mds1 to oleg238-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all
pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1
Started lustre-MDT0000
13:12:48 (1790788368) targets are mounted
13:12:48 (1790788368) facet_failover done
writemany returned 0