== recovery-small test 51: failover MDS during recovery == 16:30:34 (1778617834)
fail_loc=0x00001310
Failing mds1 on oleg139-server
Stopping /mnt/lustre-mds1 (opts:) on oleg139-server
16:30:45 (1778617845) shut down
facet: mds1 facet_host: oleg139-server facet_failover_host: oleg139-server
Failover mds1 to oleg139-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1
Started lustre-MDT0000
16:31:18 (1778617878) targets are mounted
16:31:18 (1778617878) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg139-server
Stopping /mnt/lustre-mds1 (opts:) on oleg139-server
16:31:27 (1778617887) shut down
facet: mds1 facet_host: oleg139-server facet_failover_host: oleg139-server
Failover mds1 to oleg139-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1
Started lustre-MDT0000
16:32:01 (1778617921) targets are mounted
16:32:01 (1778617921) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg139-server
Stopping /mnt/lustre-mds1 (opts:) on oleg139-server
16:32:16 (1778617936) shut down
facet: mds1 facet_host: oleg139-server facet_failover_host: oleg139-server
Failover mds1 to oleg139-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1
Started lustre-MDT0000
16:32:49 (1778617969) targets are mounted
16:32:49 (1778617969) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg139-server
Stopping /mnt/lustre-mds1 (opts:) on oleg139-server
16:33:09 (1778617989) shut down
facet: mds1 facet_host: oleg139-server facet_failover_host: oleg139-server
Failover mds1 to oleg139-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1
Started lustre-MDT0000
16:33:46 (1778618026) targets are mounted
16:33:46 (1778618026) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg139-server
Stopping /mnt/lustre-mds1 (opts:) on oleg139-server
16:34:16 (1778618056) shut down
facet: mds1 facet_host: oleg139-server facet_failover_host: oleg139-server
Failover mds1 to oleg139-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1
Started lustre-MDT0000
16:34:49 (1778618089) targets are mounted
16:34:49 (1778618089) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg139-server
Stopping /mnt/lustre-mds1 (opts:) on oleg139-server
16:35:24 (1778618124) shut down
facet: mds1 facet_host: oleg139-server facet_failover_host: oleg139-server
Failover mds1 to oleg139-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1
Started lustre-MDT0000
16:35:57 (1778618157) targets are mounted
16:35:57 (1778618157) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg139-server
Stopping /mnt/lustre-mds1 (opts:) on oleg139-server
16:36:37 (1778618197) shut down
facet: mds1 facet_host: oleg139-server facet_failover_host: oleg139-server
Failover mds1 to oleg139-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all
pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1
Started lustre-MDT0000
16:37:03 (1778618223) targets are mounted
16:37:03 (1778618223) facet_failover done
writemany returned 0