== recovery-small test 51: failover MDS during recovery == 09:00:07 (1789304407)
fail_loc=0x00001310
Failing mds1 on oleg228-server
Stopping /mnt/lustre-mds1 (opts:) on oleg228-server
09:00:15 (1789304415) shut down
facet: mds1 facet_host: oleg228-server facet_failover_host: oleg228-server
Failover mds1 to oleg228-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all
pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1
Started lustre-MDT0000
09:00:47 (1789304447) targets are mounted
09:00:47 (1789304447) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg228-server
Stopping /mnt/lustre-mds1 (opts:) on oleg228-server
09:00:55 (1789304455) shut down
facet: mds1 facet_host: oleg228-server facet_failover_host: oleg228-server
Failover mds1 to oleg228-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all
pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1
Started lustre-MDT0000
09:01:27 (1789304487) targets are mounted
09:01:27 (1789304487) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg228-server
Stopping /mnt/lustre-mds1 (opts:) on oleg228-server
09:01:39 (1789304499) shut down
facet: mds1 facet_host: oleg228-server facet_failover_host: oleg228-server
Failover mds1 to oleg228-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all
pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1
Started lustre-MDT0000
09:02:13 (1789304533) targets are mounted
09:02:13 (1789304533) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg228-server
Stopping /mnt/lustre-mds1 (opts:) on oleg228-server
09:02:31 (1789304551) shut down
facet: mds1 facet_host: oleg228-server facet_failover_host: oleg228-server
Failover mds1 to oleg228-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all
pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1
Started lustre-MDT0000
09:03:04 (1789304584) targets are mounted
09:03:04 (1789304584) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg228-server
Stopping /mnt/lustre-mds1 (opts:) on oleg228-server
09:03:32 (1789304612) shut down
facet: mds1 facet_host: oleg228-server facet_failover_host: oleg228-server
Failover mds1 to oleg228-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all
pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1
Started lustre-MDT0000
09:04:05 (1789304645) targets are mounted
09:04:05 (1789304645) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg228-server
Stopping /mnt/lustre-mds1 (opts:) on oleg228-server
09:04:38 (1789304678) shut down
facet: mds1 facet_host: oleg228-server facet_failover_host: oleg228-server
Failover mds1 to oleg228-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all
pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1
Started lustre-MDT0000
09:05:11 (1789304711) targets are mounted
09:05:11 (1789304711) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg228-server
Stopping /mnt/lustre-mds1 (opts:) on oleg228-server
09:05:48 (1789304748) shut down
facet: mds1 facet_host: oleg228-server facet_failover_host: oleg228-server
Failover mds1 to oleg228-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all
pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1
Started lustre-MDT0000
09:06:21 (1789304781) targets are mounted
09:06:21 (1789304781) facet_failover done
writemany returned 0