== recovery-small test 51: failover MDS during recovery == 14:45:45 (1790102745)
fail_loc=0x00001310
Failing mds1 on oleg235-server
Stopping /mnt/lustre-mds1 (opts:) on oleg235-server
14:45:54 (1790102754) shut down
facet: mds1 facet_host: oleg235-server facet_failover_host: oleg235-server
Failover mds1 to oleg235-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all
pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1
Started lustre-MDT0000
14:46:20 (1790102780) targets are mounted
14:46:20 (1790102780) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg235-server
Stopping /mnt/lustre-mds1 (opts:) on oleg235-server
14:46:29 (1790102789) shut down
facet: mds1 facet_host: oleg235-server facet_failover_host: oleg235-server
Failover mds1 to oleg235-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all
pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1
Started lustre-MDT0000
14:47:02 (1790102822) targets are mounted
14:47:02 (1790102822) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg235-server
Stopping /mnt/lustre-mds1 (opts:) on oleg235-server
14:47:14 (1790102834) shut down
facet: mds1 facet_host: oleg235-server facet_failover_host: oleg235-server
Failover mds1 to oleg235-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all
pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1
Started lustre-MDT0000
14:47:47 (1790102867) targets are mounted
14:47:47 (1790102867) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg235-server
Stopping /mnt/lustre-mds1 (opts:) on oleg235-server
14:48:05 (1790102885) shut down
facet: mds1 facet_host: oleg235-server facet_failover_host: oleg235-server
Failover mds1 to oleg235-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all
pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1
Started lustre-MDT0000
14:48:32 (1790102912) targets are mounted
14:48:32 (1790102912) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg235-server
Stopping /mnt/lustre-mds1 (opts:) on oleg235-server
14:49:01 (1790102941) shut down
facet: mds1 facet_host: oleg235-server facet_failover_host: oleg235-server
Failover mds1 to oleg235-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all
pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1
Started lustre-MDT0000
14:49:34 (1790102974) targets are mounted
14:49:34 (1790102974) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg235-server
Stopping /mnt/lustre-mds1 (opts:) on oleg235-server
14:50:08 (1790103008) shut down
facet: mds1 facet_host: oleg235-server facet_failover_host: oleg235-server
Failover mds1 to oleg235-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all
pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1
Started lustre-MDT0000
14:50:44 (1790103044) targets are mounted
14:50:44 (1790103044) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg235-server
Stopping /mnt/lustre-mds1 (opts:) on oleg235-server
14:51:28 (1790103088) shut down
facet: mds1 facet_host: oleg235-server facet_failover_host: oleg235-server
Failover mds1 to oleg235-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all
pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1
Started lustre-MDT0000
14:52:02 (1790103122) targets are mounted
14:52:02 (1790103122) facet_failover done
writemany returned 0