== recovery-small test 51: failover MDS during recovery == 00:02:04 (1783137724)
fail_loc=0x00001310
Failing mds1 on oleg135-server
Stopping /mnt/lustre-mds1 (opts:) on oleg135-server
00:02:14 (1783137734) shut down
facet: mds1 facet_host: oleg135-server facet_failover_host: oleg135-server
Failover mds1 to oleg135-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all
pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1
Started lustre-MDT0000
00:02:40 (1783137760) targets are mounted
00:02:40 (1783137760) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg135-server
Stopping /mnt/lustre-mds1 (opts:) on oleg135-server
00:02:48 (1783137768) shut down
facet: mds1 facet_host: oleg135-server facet_failover_host: oleg135-server
Failover mds1 to oleg135-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all
pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1
Started lustre-MDT0000
00:03:12 (1783137792) targets are mounted
00:03:12 (1783137792) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg135-server
Stopping /mnt/lustre-mds1 (opts:) on oleg135-server
00:03:25 (1783137805) shut down
facet: mds1 facet_host: oleg135-server facet_failover_host: oleg135-server
Failover mds1 to oleg135-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all
pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1
Started lustre-MDT0000
00:03:49 (1783137829) targets are mounted
00:03:50 (1783137830) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg135-server
Stopping /mnt/lustre-mds1 (opts:) on oleg135-server
00:04:10 (1783137850) shut down
facet: mds1 facet_host: oleg135-server facet_failover_host: oleg135-server
Failover mds1 to oleg135-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all
pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1
Started lustre-MDT0000
00:04:37 (1783137877) targets are mounted
00:04:37 (1783137877) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg135-server
Stopping /mnt/lustre-mds1 (opts:) on oleg135-server
00:05:08 (1783137908) shut down
facet: mds1 facet_host: oleg135-server facet_failover_host: oleg135-server
Failover mds1 to oleg135-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all
pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1
Started lustre-MDT0000
00:05:42 (1783137942) targets are mounted
00:05:42 (1783137942) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg135-server
Stopping /mnt/lustre-mds1 (opts:) on oleg135-server
00:06:18 (1783137978) shut down
facet: mds1 facet_host: oleg135-server facet_failover_host: oleg135-server
Failover mds1 to oleg135-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all
pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1
Started lustre-MDT0000
00:06:43 (1783138003) targets are mounted
00:06:43 (1783138003) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg135-server
Stopping /mnt/lustre-mds1 (opts:) on oleg135-server
00:07:21 (1783138041) shut down
facet: mds1 facet_host: oleg135-server facet_failover_host: oleg135-server
Failover mds1 to oleg135-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all
pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1
Started lustre-MDT0000
00:07:44 (1783138064) targets are mounted
00:07:45 (1783138065) facet_failover done
writemany returned 0