== recovery-small test 51: failover MDS during recovery == 13:39:17 (1758735557)
fail_loc=0x00001310
Failing mds1 on oleg231-server
Stopping /mnt/lustre-mds1 (opts:) on oleg231-server
13:39:20 (1758735560) shut down
facet: mds1 facet_host: oleg231-server facet_failover_host: oleg231-server
Failover mds1 to oleg231-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all
pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1
Started lustre-MDT0000
13:39:35 (1758735575) targets are mounted
13:39:35 (1758735575) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg231-server
Stopping /mnt/lustre-mds1 (opts:) on oleg231-server
13:39:38 (1758735578) shut down
facet: mds1 facet_host: oleg231-server facet_failover_host: oleg231-server
Failover mds1 to oleg231-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all
pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1
Started lustre-MDT0000
13:39:52 (1758735592) targets are mounted
13:39:52 (1758735592) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg231-server
Stopping /mnt/lustre-mds1 (opts:) on oleg231-server
13:39:59 (1758735599) shut down
facet: mds1 facet_host: oleg231-server facet_failover_host: oleg231-server
Failover mds1 to oleg231-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all
pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1
Started lustre-MDT0000
13:40:13 (1758735613) targets are mounted
13:40:13 (1758735613) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg231-server
Stopping /mnt/lustre-mds1 (opts:) on oleg231-server
13:40:26 (1758735626) shut down
facet: mds1 facet_host: oleg231-server facet_failover_host: oleg231-server
Failover mds1 to oleg231-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all
pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1
Started lustre-MDT0000
13:40:40 (1758735640) targets are mounted
13:40:40 (1758735640) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg231-server
Stopping /mnt/lustre-mds1 (opts:) on oleg231-server
13:41:02 (1758735662) shut down
facet: mds1 facet_host: oleg231-server facet_failover_host: oleg231-server
Failover mds1 to oleg231-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all
pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1
Started lustre-MDT0000
13:41:16 (1758735676) targets are mounted
13:41:16 (1758735676) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg231-server
Stopping /mnt/lustre-mds1 (opts:) on oleg231-server
13:41:44 (1758735704) shut down
facet: mds1 facet_host: oleg231-server facet_failover_host: oleg231-server
Failover mds1 to oleg231-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all
pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1
Started lustre-MDT0000
13:41:59 (1758735719) targets are mounted
13:41:59 (1758735719) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg231-server
Stopping /mnt/lustre-mds1 (opts:) on oleg231-server
13:42:31 (1758735751) shut down
facet: mds1 facet_host: oleg231-server facet_failover_host: oleg231-server
Failover mds1 to oleg231-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all
pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1
Started lustre-MDT0000
13:42:45 (1758735765) targets are mounted
13:42:45 (1758735765) facet_failover done
writemany returned 0