== recovery-small test 51: failover MDS during recovery == 18:47:17 (1787870837)
fail_loc=0x00001310
Failing mds1 on oleg639-server
Stopping /mnt/lustre-mds1 (opts:) on oleg639-server
18:47:25 (1787870845) shut down
facet: mds1 facet_host: oleg639-server facet_failover_host: oleg639-server
Failover mds1 to oleg639-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg639-server: oleg639-server.virtnet: executing set_default_debug -1 all
pdsh@oleg639-client: oleg639-server: ssh exited with exit code 1
Started lustre-MDT0000
18:47:47 (1787870867) targets are mounted
18:47:47 (1787870867) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg639-server
Stopping /mnt/lustre-mds1 (opts:) on oleg639-server
18:47:54 (1787870874) shut down
facet: mds1 facet_host: oleg639-server facet_failover_host: oleg639-server
Failover mds1 to oleg639-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg639-server: oleg639-server.virtnet: executing set_default_debug -1 all
pdsh@oleg639-client: oleg639-server: ssh exited with exit code 1
Started lustre-MDT0000
18:48:17 (1787870897) targets are mounted
18:48:18 (1787870898) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg639-server
Stopping /mnt/lustre-mds1 (opts:) on oleg639-server
18:48:28 (1787870908) shut down
facet: mds1 facet_host: oleg639-server facet_failover_host: oleg639-server
Failover mds1 to oleg639-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg639-server: oleg639-server.virtnet: executing set_default_debug -1 all
pdsh@oleg639-client: oleg639-server: ssh exited with exit code 1
Started lustre-MDT0000
18:48:49 (1787870929) targets are mounted
18:48:49 (1787870929) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg639-server
Stopping /mnt/lustre-mds1 (opts:) on oleg639-server
18:49:06 (1787870946) shut down
facet: mds1 facet_host: oleg639-server facet_failover_host: oleg639-server
Failover mds1 to oleg639-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg639-server: oleg639-server.virtnet: executing set_default_debug -1 all
pdsh@oleg639-client: oleg639-server: ssh exited with exit code 1
Started lustre-MDT0000
18:49:26 (1787870966) targets are mounted
18:49:26 (1787870966) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg639-server
Stopping /mnt/lustre-mds1 (opts:) on oleg639-server
18:49:51 (1787870991) shut down
facet: mds1 facet_host: oleg639-server facet_failover_host: oleg639-server
Failover mds1 to oleg639-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg639-server: oleg639-server.virtnet: executing set_default_debug -1 all
pdsh@oleg639-client: oleg639-server: ssh exited with exit code 1
Started lustre-MDT0000
18:50:12 (1787871012) targets are mounted
18:50:12 (1787871012) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg639-server
Stopping /mnt/lustre-mds1 (opts:) on oleg639-server
18:50:42 (1787871042) shut down
facet: mds1 facet_host: oleg639-server facet_failover_host: oleg639-server
Failover mds1 to oleg639-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg639-server: oleg639-server.virtnet: executing set_default_debug -1 all
pdsh@oleg639-client: oleg639-server: ssh exited with exit code 1
Started lustre-MDT0000
18:51:03 (1787871063) targets are mounted
18:51:03 (1787871063) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg639-server
Stopping /mnt/lustre-mds1 (opts:) on oleg639-server
18:51:40 (1787871100) shut down
facet: mds1 facet_host: oleg639-server facet_failover_host: oleg639-server
Failover mds1 to oleg639-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg639-server: oleg639-server.virtnet: executing set_default_debug -1 all
pdsh@oleg639-client: oleg639-server: ssh exited with exit code 1
Started lustre-MDT0000
18:52:11 (1787871131) targets are mounted
18:52:11 (1787871131) facet_failover done
writemany returned 0