== recovery-small test 51: failover MDS during recovery == 20:50:54 (1787100654)
fail_loc=0x00001310
Failing mds1 on oleg334-server
Stopping /mnt/lustre-mds1 (opts:) on oleg334-server
20:51:06 (1787100666) shut down
facet: mds1 facet_host: oleg334-server facet_failover_host: oleg334-server
Failover mds1 to oleg334-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all
pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1
Started lustre-MDT0000
20:51:42 (1787100702) targets are mounted
20:51:42 (1787100702) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg334-server
Stopping /mnt/lustre-mds1 (opts:) on oleg334-server
20:51:51 (1787100711) shut down
facet: mds1 facet_host: oleg334-server facet_failover_host: oleg334-server
Failover mds1 to oleg334-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all
pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1
Started lustre-MDT0000
20:52:25 (1787100745) targets are mounted
20:52:26 (1787100746) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg334-server
Stopping /mnt/lustre-mds1 (opts:) on oleg334-server
20:52:40 (1787100760) shut down
facet: mds1 facet_host: oleg334-server facet_failover_host: oleg334-server
Failover mds1 to oleg334-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all
pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1
Started lustre-MDT0000
20:53:18 (1787100798) targets are mounted
20:53:18 (1787100798) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg334-server
Stopping /mnt/lustre-mds1 (opts:) on oleg334-server
20:53:40 (1787100820) shut down
facet: mds1 facet_host: oleg334-server facet_failover_host: oleg334-server
Failover mds1 to oleg334-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all
pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1
Started lustre-MDT0000
20:54:13 (1787100853) targets are mounted
20:54:13 (1787100853) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg334-server
Stopping /mnt/lustre-mds1 (opts:) on oleg334-server
20:54:45 (1787100885) shut down
facet: mds1 facet_host: oleg334-server facet_failover_host: oleg334-server
Failover mds1 to oleg334-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all
pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1
Started lustre-MDT0000
20:55:14 (1787100914) targets are mounted
20:55:14 (1787100914) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg334-server
Stopping /mnt/lustre-mds1 (opts:) on oleg334-server
20:55:46 (1787100946) shut down
facet: mds1 facet_host: oleg334-server facet_failover_host: oleg334-server
Failover mds1 to oleg334-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all
pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1
Started lustre-MDT0000
20:56:09 (1787100969) targets are mounted
20:56:09 (1787100969) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg334-server
Stopping /mnt/lustre-mds1 (opts:) on oleg334-server
20:56:45 (1787101005) shut down
facet: mds1 facet_host: oleg334-server facet_failover_host: oleg334-server
Failover mds1 to oleg334-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all
pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1
Started lustre-MDT0000
20:57:06 (1787101026) targets are mounted
20:57:06 (1787101026) facet_failover done
writemany returned 0