== recovery-small test 51: failover MDS during recovery == 06:44:11 (1786185851)
fail_loc=0x00001310
Failing mds1 on oleg351-server
Stopping /mnt/lustre-mds1 (opts:) on oleg351-server
06:44:20 (1786185860) shut down
facet: mds1 facet_host: oleg351-server facet_failover_host: oleg351-server
Failover mds1 to oleg351-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Started lustre-MDT0000
06:44:52 (1786185892) targets are mounted
06:44:52 (1786185892) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg351-server
Stopping /mnt/lustre-mds1 (opts:) on oleg351-server
06:44:59 (1786185899) shut down
facet: mds1 facet_host: oleg351-server facet_failover_host: oleg351-server
Failover mds1 to oleg351-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Started lustre-MDT0000
06:45:24 (1786185924) targets are mounted
06:45:24 (1786185924) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg351-server
Stopping /mnt/lustre-mds1 (opts:) on oleg351-server
06:45:39 (1786185939) shut down
facet: mds1 facet_host: oleg351-server facet_failover_host: oleg351-server
Failover mds1 to oleg351-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Started lustre-MDT0000
06:46:05 (1786185965) targets are mounted
06:46:05 (1786185965) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg351-server
Stopping /mnt/lustre-mds1 (opts:) on oleg351-server
06:46:26 (1786185986) shut down
facet: mds1 facet_host: oleg351-server facet_failover_host: oleg351-server
Failover mds1 to oleg351-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Started lustre-MDT0000
06:46:57 (1786186017) targets are mounted
06:46:57 (1786186017) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg351-server
Stopping /mnt/lustre-mds1 (opts:) on oleg351-server
06:47:25 (1786186045) shut down
facet: mds1 facet_host: oleg351-server facet_failover_host: oleg351-server
Failover mds1 to oleg351-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Started lustre-MDT0000
06:47:50 (1786186070) targets are mounted
06:47:50 (1786186070) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg351-server
Stopping /mnt/lustre-mds1 (opts:) on oleg351-server
06:48:22 (1786186102) shut down
facet: mds1 facet_host: oleg351-server facet_failover_host: oleg351-server
Failover mds1 to oleg351-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Started lustre-MDT0000
06:48:50 (1786186130) targets are mounted
06:48:50 (1786186130) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg351-server
Stopping /mnt/lustre-mds1 (opts:) on oleg351-server
06:49:29 (1786186169) shut down
facet: mds1 facet_host: oleg351-server facet_failover_host: oleg351-server
Failover mds1 to oleg351-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all
pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1
Started lustre-MDT0000
06:49:55 (1786186195) targets are mounted
06:49:55 (1786186195) facet_failover done
writemany returned 0