== recovery-small test 51: failover MDS during recovery == 03:00:44 (1785222044)
fail_loc=0x00001310
Failing mds1 on oleg406-server
Stopping /mnt/lustre-mds1 (opts:) on oleg406-server
03:00:51 (1785222051) shut down
facet: mds1 facet_host: oleg406-server facet_failover_host: oleg406-server
Failover mds1 to oleg406-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all
pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1
Started lustre-MDT0000
03:01:14 (1785222074) targets are mounted
03:01:14 (1785222074) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg406-server
Stopping /mnt/lustre-mds1 (opts:) on oleg406-server
03:01:22 (1785222082) shut down
facet: mds1 facet_host: oleg406-server facet_failover_host: oleg406-server
Failover mds1 to oleg406-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all
pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1
Started lustre-MDT0000
03:01:44 (1785222104) targets are mounted
03:01:44 (1785222104) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg406-server
Stopping /mnt/lustre-mds1 (opts:) on oleg406-server
03:01:54 (1785222114) shut down
facet: mds1 facet_host: oleg406-server facet_failover_host: oleg406-server
Failover mds1 to oleg406-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all
pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1
Started lustre-MDT0000
03:02:26 (1785222146) targets are mounted
03:02:26 (1785222146) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg406-server
Stopping /mnt/lustre-mds1 (opts:) on oleg406-server
03:02:42 (1785222162) shut down
facet: mds1 facet_host: oleg406-server facet_failover_host: oleg406-server
Failover mds1 to oleg406-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all
pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1
Started lustre-MDT0000
03:03:13 (1785222193) targets are mounted
03:03:13 (1785222193) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg406-server
Stopping /mnt/lustre-mds1 (opts:) on oleg406-server
03:03:39 (1785222219) shut down
facet: mds1 facet_host: oleg406-server facet_failover_host: oleg406-server
Failover mds1 to oleg406-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all
pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1
Started lustre-MDT0000
03:04:10 (1785222250) targets are mounted
03:04:10 (1785222250) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg406-server
Stopping /mnt/lustre-mds1 (opts:) on oleg406-server
03:04:44 (1785222284) shut down
facet: mds1 facet_host: oleg406-server facet_failover_host: oleg406-server
Failover mds1 to oleg406-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all
pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1
Started lustre-MDT0000
03:05:21 (1785222321) targets are mounted
03:05:21 (1785222321) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg406-server
Stopping /mnt/lustre-mds1 (opts:) on oleg406-server
03:05:59 (1785222359) shut down
facet: mds1 facet_host: oleg406-server facet_failover_host: oleg406-server
Failover mds1 to oleg406-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all
pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1
Started lustre-MDT0000
03:06:22 (1785222382) targets are mounted
03:06:22 (1785222382) facet_failover done
writemany returned 0