== recovery-small test 51: failover MDS during recovery == 19:56:35 (1789430195)
fail_loc=0x00001310
Failing mds1 on oleg253-server
Stopping /mnt/lustre-mds1 (opts:) on oleg253-server
19:56:43 (1789430203) shut down
facet: mds1 facet_host: oleg253-server facet_failover_host: oleg253-server
Failover mds1 to oleg253-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all
pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1
Started lustre-MDT0000
19:57:05 (1789430225) targets are mounted
19:57:05 (1789430225) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg253-server
Stopping /mnt/lustre-mds1 (opts:) on oleg253-server
19:57:14 (1789430234) shut down
facet: mds1 facet_host: oleg253-server facet_failover_host: oleg253-server
Failover mds1 to oleg253-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all
pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1
Started lustre-MDT0000
19:57:52 (1789430272) targets are mounted
19:57:52 (1789430272) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg253-server
Stopping /mnt/lustre-mds1 (opts:) on oleg253-server
19:58:05 (1789430285) shut down
facet: mds1 facet_host: oleg253-server facet_failover_host: oleg253-server
Failover mds1 to oleg253-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all
pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1
Started lustre-MDT0000
19:58:29 (1789430309) targets are mounted
19:58:29 (1789430309) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg253-server
Stopping /mnt/lustre-mds1 (opts:) on oleg253-server
19:58:47 (1789430327) shut down
facet: mds1 facet_host: oleg253-server facet_failover_host: oleg253-server
Failover mds1 to oleg253-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all
pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1
Started lustre-MDT0000
19:59:13 (1789430353) targets are mounted
19:59:13 (1789430353) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg253-server
Stopping /mnt/lustre-mds1 (opts:) on oleg253-server
19:59:41 (1789430381) shut down
facet: mds1 facet_host: oleg253-server facet_failover_host: oleg253-server
Failover mds1 to oleg253-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all
pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1
Started lustre-MDT0000
20:00:12 (1789430412) targets are mounted
20:00:12 (1789430412) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg253-server
Stopping /mnt/lustre-mds1 (opts:) on oleg253-server
20:00:45 (1789430445) shut down
facet: mds1 facet_host: oleg253-server facet_failover_host: oleg253-server
Failover mds1 to oleg253-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all
pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1
Started lustre-MDT0000
20:01:16 (1789430476) targets are mounted
20:01:16 (1789430476) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg253-server
Stopping /mnt/lustre-mds1 (opts:) on oleg253-server
20:01:55 (1789430515) shut down
facet: mds1 facet_host: oleg253-server facet_failover_host: oleg253-server
Failover mds1 to oleg253-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all
pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1
Started lustre-MDT0000
20:02:26 (1789430546) targets are mounted
20:02:26 (1789430546) facet_failover done
writemany returned 0