== recovery-small test 51: failover MDS during recovery == 01:37:54 (1789450674)
fail_loc=0x00001310
Failing mds1 on oleg446-server
Stopping /mnt/lustre-mds1 (opts:) on oleg446-server
01:38:04 (1789450684) shut down
facet: mds1 facet_host: oleg446-server facet_failover_host: oleg446-server
Failover mds1 to oleg446-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
Started lustre-MDT0000
01:38:36 (1789450716) targets are mounted
01:38:37 (1789450717) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg446-server
Stopping /mnt/lustre-mds1 (opts:) on oleg446-server
01:38:44 (1789450724) shut down
facet: mds1 facet_host: oleg446-server facet_failover_host: oleg446-server
Failover mds1 to oleg446-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
Started lustre-MDT0000
01:39:09 (1789450749) targets are mounted
01:39:09 (1789450749) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg446-server
Stopping /mnt/lustre-mds1 (opts:) on oleg446-server
01:39:21 (1789450761) shut down
facet: mds1 facet_host: oleg446-server facet_failover_host: oleg446-server
Failover mds1 to oleg446-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
Started lustre-MDT0000
01:39:53 (1789450793) targets are mounted
01:39:53 (1789450793) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg446-server
Stopping /mnt/lustre-mds1 (opts:) on oleg446-server
01:40:11 (1789450811) shut down
facet: mds1 facet_host: oleg446-server facet_failover_host: oleg446-server
Failover mds1 to oleg446-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
Started lustre-MDT0000
01:40:44 (1789450844) targets are mounted
01:40:44 (1789450844) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg446-server
Stopping /mnt/lustre-mds1 (opts:) on oleg446-server
01:41:15 (1789450875) shut down
facet: mds1 facet_host: oleg446-server facet_failover_host: oleg446-server
Failover mds1 to oleg446-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
Started lustre-MDT0000
01:41:46 (1789450906) targets are mounted
01:41:46 (1789450906) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg446-server
Stopping /mnt/lustre-mds1 (opts:) on oleg446-server
01:42:19 (1789450939) shut down
facet: mds1 facet_host: oleg446-server facet_failover_host: oleg446-server
Failover mds1 to oleg446-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
Started lustre-MDT0000
01:42:43 (1789450963) targets are mounted
01:42:43 (1789450963) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg446-server
Stopping /mnt/lustre-mds1 (opts:) on oleg446-server
01:43:21 (1789451001) shut down
facet: mds1 facet_host: oleg446-server facet_failover_host: oleg446-server
Failover mds1 to oleg446-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
Started lustre-MDT0000
01:43:45 (1789451025) targets are mounted
01:43:45 (1789451025) facet_failover done
writemany returned 0