== recovery-small test 51: failover MDS during recovery == 18:40:19 (1787784019)
fail_loc=0x00001310
Failing mds1 on oleg150-server
Stopping /mnt/lustre-mds1 (opts:) on oleg150-server
18:40:27 (1787784027) shut down
facet: mds1 facet_host: oleg150-server facet_failover_host: oleg150-server
Failover mds1 to oleg150-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all
pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1
Started lustre-MDT0000
18:41:01 (1787784061) targets are mounted
18:41:01 (1787784061) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg150-server
Stopping /mnt/lustre-mds1 (opts:) on oleg150-server
18:41:09 (1787784069) shut down
facet: mds1 facet_host: oleg150-server facet_failover_host: oleg150-server
Failover mds1 to oleg150-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all
pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1
Started lustre-MDT0000
18:41:34 (1787784094) targets are mounted
18:41:34 (1787784094) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg150-server
Stopping /mnt/lustre-mds1 (opts:) on oleg150-server
18:41:47 (1787784107) shut down
facet: mds1 facet_host: oleg150-server facet_failover_host: oleg150-server
Failover mds1 to oleg150-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all
pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1
Started lustre-MDT0000
18:42:20 (1787784140) targets are mounted
18:42:20 (1787784140) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg150-server
Stopping /mnt/lustre-mds1 (opts:) on oleg150-server
18:42:38 (1787784158) shut down
facet: mds1 facet_host: oleg150-server facet_failover_host: oleg150-server
Failover mds1 to oleg150-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all
pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1
Started lustre-MDT0000
18:43:12 (1787784192) targets are mounted
18:43:12 (1787784192) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg150-server
Stopping /mnt/lustre-mds1 (opts:) on oleg150-server
18:43:39 (1787784219) shut down
facet: mds1 facet_host: oleg150-server facet_failover_host: oleg150-server
Failover mds1 to oleg150-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all
pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1
Started lustre-MDT0000
18:44:03 (1787784243) targets are mounted
18:44:03 (1787784243) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg150-server
Stopping /mnt/lustre-mds1 (opts:) on oleg150-server
18:44:36 (1787784276) shut down
facet: mds1 facet_host: oleg150-server facet_failover_host: oleg150-server
Failover mds1 to oleg150-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all
pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1
Started lustre-MDT0000
18:44:59 (1787784299) targets are mounted
18:45:00 (1787784300) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg150-server
Stopping /mnt/lustre-mds1 (opts:) on oleg150-server
18:45:37 (1787784337) shut down
facet: mds1 facet_host: oleg150-server facet_failover_host: oleg150-server
Failover mds1 to oleg150-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all
pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1
Started lustre-MDT0000
18:46:00 (1787784360) targets are mounted
18:46:00 (1787784360) facet_failover done
writemany returned 0