== recovery-small test 51: failover MDS during recovery == 14:30:54 (1781029854)
fail_loc=0x00001310
Failing mds1 on oleg254-server
Stopping /mnt/lustre-mds1 (opts:) on oleg254-server
14:30:58 (1781029858) shut down
facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server
Failover mds1 to oleg254-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all
kernel.panic_on_warn = 0
pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1
Started lustre-MDT0000
14:31:14 (1781029874) targets are mounted
14:31:14 (1781029874) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg254-server
Stopping /mnt/lustre-mds1 (opts:) on oleg254-server
14:31:18 (1781029878) shut down
facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server
Failover mds1 to oleg254-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all
kernel.panic_on_warn = 0
pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1
Started lustre-MDT0000
14:31:35 (1781029895) targets are mounted
14:31:35 (1781029895) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg254-server
Stopping /mnt/lustre-mds1 (opts:) on oleg254-server
14:31:43 (1781029903) shut down
facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server
Failover mds1 to oleg254-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all
kernel.panic_on_warn = 0
pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1
Started lustre-MDT0000
14:31:58 (1781029918) targets are mounted
14:31:58 (1781029918) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg254-server
Stopping /mnt/lustre-mds1 (opts:) on oleg254-server
14:32:11 (1781029931) shut down
facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server
Failover mds1 to oleg254-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all
kernel.panic_on_warn = 0
pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1
Started lustre-MDT0000
14:32:26 (1781029946) targets are mounted
14:32:26 (1781029946) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg254-server
Stopping /mnt/lustre-mds1 (opts:) on oleg254-server
14:32:49 (1781029969) shut down
facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server
Failover mds1 to oleg254-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all
kernel.panic_on_warn = 0
pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1
Started lustre-MDT0000
14:33:05 (1781029985) targets are mounted
14:33:05 (1781029985) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg254-server
Stopping /mnt/lustre-mds1 (opts:) on oleg254-server
14:33:33 (1781030013) shut down
facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server
Failover mds1 to oleg254-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all
kernel.panic_on_warn = 0
pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1
Started lustre-MDT0000
14:33:49 (1781030029) targets are mounted
14:33:49 (1781030029) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg254-server
Stopping /mnt/lustre-mds1 (opts:) on oleg254-server
14:34:22 (1781030062) shut down
facet: mds1 facet_host: oleg254-server facet_failover_host: oleg254-server
Failover mds1 to oleg254-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all
kernel.panic_on_warn = 0
pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1
Started lustre-MDT0000
14:34:38 (1781030078) targets are mounted
14:34:38 (1781030078) facet_failover done
writemany returned 0