== recovery-small test 51: failover MDS during recovery == 06:57:31 (1787137051)
fail_loc=0x00001310
Failing mds1 on oleg137-server
Stopping /mnt/lustre-mds1 (opts:) on oleg137-server
06:57:41 (1787137061) shut down
facet: mds1 facet_host: oleg137-server facet_failover_host: oleg137-server
Failover mds1 to oleg137-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all
pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1
Started lustre-MDT0000
06:58:14 (1787137094) targets are mounted
06:58:14 (1787137094) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg137-server
Stopping /mnt/lustre-mds1 (opts:) on oleg137-server
06:58:22 (1787137102) shut down
facet: mds1 facet_host: oleg137-server facet_failover_host: oleg137-server
Failover mds1 to oleg137-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all
pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1
Started lustre-MDT0000
06:58:57 (1787137137) targets are mounted
06:58:57 (1787137137) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg137-server
Stopping /mnt/lustre-mds1 (opts:) on oleg137-server
06:59:10 (1787137150) shut down
facet: mds1 facet_host: oleg137-server facet_failover_host: oleg137-server
Failover mds1 to oleg137-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all
pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1
Started lustre-MDT0000
06:59:43 (1787137183) targets are mounted
06:59:43 (1787137183) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg137-server
Stopping /mnt/lustre-mds1 (opts:) on oleg137-server
07:00:05 (1787137205) shut down
facet: mds1 facet_host: oleg137-server facet_failover_host: oleg137-server
Failover mds1 to oleg137-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all
pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1
Started lustre-MDT0000
07:00:36 (1787137236) targets are mounted
07:00:36 (1787137236) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg137-server
Stopping /mnt/lustre-mds1 (opts:) on oleg137-server
07:01:03 (1787137263) shut down
facet: mds1 facet_host: oleg137-server facet_failover_host: oleg137-server
Failover mds1 to oleg137-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all
pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1
Started lustre-MDT0000
07:01:30 (1787137290) targets are mounted
07:01:30 (1787137290) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg137-server
Stopping /mnt/lustre-mds1 (opts:) on oleg137-server
07:02:04 (1787137324) shut down
facet: mds1 facet_host: oleg137-server facet_failover_host: oleg137-server
Failover mds1 to oleg137-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all
pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1
Started lustre-MDT0000
07:02:39 (1787137359) targets are mounted
07:02:39 (1787137359) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg137-server
Stopping /mnt/lustre-mds1 (opts:) on oleg137-server
07:03:17 (1787137397) shut down
facet: mds1 facet_host: oleg137-server facet_failover_host: oleg137-server
Failover mds1 to oleg137-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all
pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1
Started lustre-MDT0000
07:03:51 (1787137431) targets are mounted
07:03:51 (1787137431) facet_failover done
writemany returned 0