== recovery-small test 51: failover MDS during recovery == 11:14:23 (1787584463)
fail_loc=0x00001310
Failing mds1 on oleg154-server
Stopping /mnt/lustre-mds1 (opts:) on oleg154-server
11:14:32 (1787584472) shut down
facet: mds1 facet_host: oleg154-server facet_failover_host: oleg154-server
Failover mds1 to oleg154-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all
pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1
Started lustre-MDT0000
11:15:05 (1787584505) targets are mounted
11:15:05 (1787584505) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg154-server
Stopping /mnt/lustre-mds1 (opts:) on oleg154-server
11:15:13 (1787584513) shut down
facet: mds1 facet_host: oleg154-server facet_failover_host: oleg154-server
Failover mds1 to oleg154-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all
pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1
Started lustre-MDT0000
11:15:47 (1787584547) targets are mounted
11:15:47 (1787584547) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg154-server
Stopping /mnt/lustre-mds1 (opts:) on oleg154-server
11:15:59 (1787584559) shut down
facet: mds1 facet_host: oleg154-server facet_failover_host: oleg154-server
Failover mds1 to oleg154-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all
pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1
Started lustre-MDT0000
11:16:33 (1787584593) targets are mounted
11:16:33 (1787584593) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg154-server
Stopping /mnt/lustre-mds1 (opts:) on oleg154-server
11:16:52 (1787584612) shut down
facet: mds1 facet_host: oleg154-server facet_failover_host: oleg154-server
Failover mds1 to oleg154-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all
pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1
Started lustre-MDT0000
11:17:27 (1787584647) targets are mounted
11:17:27 (1787584647) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg154-server
Stopping /mnt/lustre-mds1 (opts:) on oleg154-server
11:17:55 (1787584675) shut down
facet: mds1 facet_host: oleg154-server facet_failover_host: oleg154-server
Failover mds1 to oleg154-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all
pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1
Started lustre-MDT0000
11:18:29 (1787584709) targets are mounted
11:18:29 (1787584709) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg154-server
Stopping /mnt/lustre-mds1 (opts:) on oleg154-server
11:19:02 (1787584742) shut down
facet: mds1 facet_host: oleg154-server facet_failover_host: oleg154-server
Failover mds1 to oleg154-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all
pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1
Started lustre-MDT0000
11:19:40 (1787584780) targets are mounted
11:19:40 (1787584780) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg154-server
Stopping /mnt/lustre-mds1 (opts:) on oleg154-server
11:20:20 (1787584820) shut down
facet: mds1 facet_host: oleg154-server facet_failover_host: oleg154-server
Failover mds1 to oleg154-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all
pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1
Started lustre-MDT0000
11:20:56 (1787584856) targets are mounted
11:20:56 (1787584856) facet_failover done
writemany returned 0