== recovery-small test 51: failover MDS during recovery == 02:58:51 (1785221931)
fail_loc=0x00001310
Failing mds1 on oleg421-server
Stopping /mnt/lustre-mds1 (opts:) on oleg421-server
02:59:02 (1785221942) shut down
facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server
Failover mds1 to oleg421-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all
pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1
Started lustre-MDT0000
02:59:41 (1785221981) targets are mounted
02:59:41 (1785221981) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg421-server
Stopping /mnt/lustre-mds1 (opts:) on oleg421-server
02:59:56 (1785221996) shut down
facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server
Failover mds1 to oleg421-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all
pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1
Started lustre-MDT0000
03:00:28 (1785222028) targets are mounted
03:00:28 (1785222028) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg421-server
Stopping /mnt/lustre-mds1 (opts:) on oleg421-server
03:00:39 (1785222039) shut down
facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server
Failover mds1 to oleg421-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all
pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1
Started lustre-MDT0000
03:01:11 (1785222071) targets are mounted
03:01:11 (1785222071) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg421-server
Stopping /mnt/lustre-mds1 (opts:) on oleg421-server
03:01:28 (1785222088) shut down
facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server
Failover mds1 to oleg421-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all
pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1
Started lustre-MDT0000
03:01:59 (1785222119) targets are mounted
03:01:59 (1785222119) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg421-server
Stopping /mnt/lustre-mds1 (opts:) on oleg421-server
03:02:26 (1785222146) shut down
facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server
Failover mds1 to oleg421-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all
pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1
Started lustre-MDT0000
03:02:48 (1785222168) targets are mounted
03:02:48 (1785222168) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg421-server
Stopping /mnt/lustre-mds1 (opts:) on oleg421-server
03:03:20 (1785222200) shut down
facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server
Failover mds1 to oleg421-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all
pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1
Started lustre-MDT0000
03:03:43 (1785222223) targets are mounted
03:03:43 (1785222223) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg421-server
Stopping /mnt/lustre-mds1 (opts:) on oleg421-server
03:04:19 (1785222259) shut down
facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server
Failover mds1 to oleg421-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all
pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1
Started lustre-MDT0000
03:04:48 (1785222288) targets are mounted
03:04:49 (1785222289) facet_failover done
writemany returned 0