== recovery-small test 51: failover MDS during recovery == 03:43:55 (1785397435)
fail_loc=0x00001310
Failing mds1 on oleg151-server
Stopping /mnt/lustre-mds1 (opts:) on oleg151-server
03:44:05 (1785397445) shut down
facet: mds1 facet_host: oleg151-server facet_failover_host: oleg151-server
Failover mds1 to oleg151-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all
pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1
Started lustre-MDT0000
03:44:41 (1785397481) targets are mounted
03:44:41 (1785397481) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg151-server
Stopping /mnt/lustre-mds1 (opts:) on oleg151-server
03:44:50 (1785397490) shut down
facet: mds1 facet_host: oleg151-server facet_failover_host: oleg151-server
Failover mds1 to oleg151-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all
pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1
Started lustre-MDT0000
03:45:16 (1785397516) targets are mounted
03:45:16 (1785397516) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg151-server
Stopping /mnt/lustre-mds1 (opts:) on oleg151-server
03:45:29 (1785397529) shut down
facet: mds1 facet_host: oleg151-server facet_failover_host: oleg151-server
Failover mds1 to oleg151-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all
pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1
Started lustre-MDT0000
03:46:02 (1785397562) targets are mounted
03:46:02 (1785397562) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg151-server
Stopping /mnt/lustre-mds1 (opts:) on oleg151-server
03:46:23 (1785397583) shut down
facet: mds1 facet_host: oleg151-server facet_failover_host: oleg151-server
Failover mds1 to oleg151-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all
pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1
Started lustre-MDT0000
03:46:57 (1785397617) targets are mounted
03:46:57 (1785397617) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg151-server
Stopping /mnt/lustre-mds1 (opts:) on oleg151-server
03:47:30 (1785397650) shut down
facet: mds1 facet_host: oleg151-server facet_failover_host: oleg151-server
Failover mds1 to oleg151-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all
pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1
Started lustre-MDT0000
03:48:08 (1785397688) targets are mounted
03:48:08 (1785397688) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg151-server
Stopping /mnt/lustre-mds1 (opts:) on oleg151-server
03:48:43 (1785397723) shut down
facet: mds1 facet_host: oleg151-server facet_failover_host: oleg151-server
Failover mds1 to oleg151-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all
pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1
Started lustre-MDT0000
03:49:18 (1785397758) targets are mounted
03:49:18 (1785397758) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg151-server
Stopping /mnt/lustre-mds1 (opts:) on oleg151-server
03:50:02 (1785397802) shut down
facet: mds1 facet_host: oleg151-server facet_failover_host: oleg151-server
Failover mds1 to oleg151-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all
pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1
Started lustre-MDT0000
03:50:39 (1785397839) targets are mounted
03:50:39 (1785397839) facet_failover done
writemany returned 0