== recovery-small test 51: failover MDS during recovery == 10:49:14 (1788360554)
fail_loc=0x00001310
Failing mds1 on oleg221-server
Stopping /mnt/lustre-mds1 (opts:) on oleg221-server
10:49:17 (1788360557) shut down
facet: mds1 facet_host: oleg221-server facet_failover_host: oleg221-server
Failover mds1 to oleg221-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all
pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1
Started lustre-MDT0000
10:49:32 (1788360572) targets are mounted
10:49:32 (1788360572) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg221-server
Stopping /mnt/lustre-mds1 (opts:) on oleg221-server
10:49:36 (1788360576) shut down
facet: mds1 facet_host: oleg221-server facet_failover_host: oleg221-server
Failover mds1 to oleg221-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all
pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1
Started lustre-MDT0000
10:49:56 (1788360596) targets are mounted
10:49:56 (1788360596) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg221-server
Stopping /mnt/lustre-mds1 (opts:) on oleg221-server
10:50:07 (1788360607) shut down
facet: mds1 facet_host: oleg221-server facet_failover_host: oleg221-server
Failover mds1 to oleg221-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all
pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1
Started lustre-MDT0000
10:50:30 (1788360630) targets are mounted
10:50:30 (1788360630) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg221-server
Stopping /mnt/lustre-mds1 (opts:) on oleg221-server
10:50:47 (1788360647) shut down
facet: mds1 facet_host: oleg221-server facet_failover_host: oleg221-server
Failover mds1 to oleg221-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all
pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1
Started lustre-MDT0000
10:51:08 (1788360668) targets are mounted
10:51:08 (1788360668) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg221-server
Stopping /mnt/lustre-mds1 (opts:) on oleg221-server
10:51:34 (1788360694) shut down
facet: mds1 facet_host: oleg221-server facet_failover_host: oleg221-server
Failover mds1 to oleg221-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all
pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1
Started lustre-MDT0000
10:51:51 (1788360711) targets are mounted
10:51:51 (1788360711) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg221-server
Stopping /mnt/lustre-mds1 (opts:) on oleg221-server
10:52:21 (1788360741) shut down
facet: mds1 facet_host: oleg221-server facet_failover_host: oleg221-server
Failover mds1 to oleg221-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all
pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1
Started lustre-MDT0000
10:52:39 (1788360759) targets are mounted
10:52:39 (1788360759) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg221-server
Stopping /mnt/lustre-mds1 (opts:) on oleg221-server
10:53:13 (1788360793) shut down
facet: mds1 facet_host: oleg221-server facet_failover_host: oleg221-server
Failover mds1 to oleg221-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all
pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1
Started lustre-MDT0000
10:53:31 (1788360811) targets are mounted
10:53:31 (1788360811) facet_failover done
writemany returned 0