== recovery-small test 51: failover MDS during recovery == 12:26:23 (1777911983)
fail_loc=0x00001310
Failing mds1 on oleg359-server
Stopping /mnt/lustre-mds1 (opts:) on oleg359-server
12:26:27 (1777911987) shut down
facet: mds1 facet_host: oleg359-server facet_failover_host: oleg359-server
Failover mds1 to oleg359-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all
pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1
Started lustre-MDT0000
12:26:41 (1777912001) targets are mounted
12:26:41 (1777912001) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg359-server
Stopping /mnt/lustre-mds1 (opts:) on oleg359-server
12:26:45 (1777912005) shut down
facet: mds1 facet_host: oleg359-server facet_failover_host: oleg359-server
Failover mds1 to oleg359-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all
pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1
Started lustre-MDT0000
12:26:58 (1777912018) targets are mounted
12:26:58 (1777912018) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg359-server
Stopping /mnt/lustre-mds1 (opts:) on oleg359-server
12:27:06 (1777912026) shut down
facet: mds1 facet_host: oleg359-server facet_failover_host: oleg359-server
Failover mds1 to oleg359-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all
pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1
Started lustre-MDT0000
12:27:20 (1777912040) targets are mounted
12:27:20 (1777912040) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg359-server
Stopping /mnt/lustre-mds1 (opts:) on oleg359-server
12:27:32 (1777912052) shut down
facet: mds1 facet_host: oleg359-server facet_failover_host: oleg359-server
Failover mds1 to oleg359-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all
pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1
Started lustre-MDT0000
12:27:46 (1777912066) targets are mounted
12:27:46 (1777912066) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg359-server
Stopping /mnt/lustre-mds1 (opts:) on oleg359-server
12:28:08 (1777912088) shut down
facet: mds1 facet_host: oleg359-server facet_failover_host: oleg359-server
Failover mds1 to oleg359-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all
pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1
Started lustre-MDT0000
12:28:22 (1777912102) targets are mounted
12:28:22 (1777912102) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg359-server
Stopping /mnt/lustre-mds1 (opts:) on oleg359-server
12:28:49 (1777912129) shut down
facet: mds1 facet_host: oleg359-server facet_failover_host: oleg359-server
Failover mds1 to oleg359-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all
pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1
Started lustre-MDT0000
12:29:03 (1777912143) targets are mounted
12:29:03 (1777912143) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg359-server
Stopping /mnt/lustre-mds1 (opts:) on oleg359-server
12:29:35 (1777912175) shut down
facet: mds1 facet_host: oleg359-server facet_failover_host: oleg359-server
Failover mds1 to oleg359-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all
pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1
Started lustre-MDT0000
12:29:49 (1777912189) targets are mounted
12:29:49 (1777912189) facet_failover done
writemany returned 0