== recovery-small test 51: failover MDS during recovery == 00:26:42 (1789187202)
fail_loc=0x00001310
Failing mds1 on oleg231-server
Stopping /mnt/lustre-mds1 (opts:) on oleg231-server
00:26:48 (1789187208) shut down
facet: mds1 facet_host: oleg231-server facet_failover_host: oleg231-server
Failover mds1 to oleg231-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all
pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1
Started lustre-MDT0000
00:27:07 (1789187227) targets are mounted
00:27:07 (1789187227) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg231-server
Stopping /mnt/lustre-mds1 (opts:) on oleg231-server
00:27:12 (1789187232) shut down
facet: mds1 facet_host: oleg231-server facet_failover_host: oleg231-server
Failover mds1 to oleg231-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all
pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1
Started lustre-MDT0000
00:27:32 (1789187252) targets are mounted
00:27:32 (1789187252) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg231-server
Stopping /mnt/lustre-mds1 (opts:) on oleg231-server
00:27:42 (1789187262) shut down
facet: mds1 facet_host: oleg231-server facet_failover_host: oleg231-server
Failover mds1 to oleg231-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all
pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1
Started lustre-MDT0000
00:28:00 (1789187280) targets are mounted
00:28:00 (1789187280) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg231-server
Stopping /mnt/lustre-mds1 (opts:) on oleg231-server
00:28:15 (1789187295) shut down
facet: mds1 facet_host: oleg231-server facet_failover_host: oleg231-server
Failover mds1 to oleg231-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all
pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1
Started lustre-MDT0000
00:28:35 (1789187315) targets are mounted
00:28:35 (1789187315) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg231-server
Stopping /mnt/lustre-mds1 (opts:) on oleg231-server
00:29:00 (1789187340) shut down
facet: mds1 facet_host: oleg231-server facet_failover_host: oleg231-server
Failover mds1 to oleg231-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all
pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1
Started lustre-MDT0000
00:29:18 (1789187358) targets are mounted
00:29:18 (1789187358) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg231-server
Stopping /mnt/lustre-mds1 (opts:) on oleg231-server
00:29:48 (1789187388) shut down
facet: mds1 facet_host: oleg231-server facet_failover_host: oleg231-server
Failover mds1 to oleg231-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all
pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1
Started lustre-MDT0000
00:30:07 (1789187407) targets are mounted
00:30:07 (1789187407) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg231-server
Stopping /mnt/lustre-mds1 (opts:) on oleg231-server
00:30:41 (1789187441) shut down
facet: mds1 facet_host: oleg231-server facet_failover_host: oleg231-server
Failover mds1 to oleg231-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all
pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1
Started lustre-MDT0000
00:30:57 (1789187457) targets are mounted
00:30:57 (1789187457) facet_failover done
writemany returned 0