== recovery-small test 51: failover MDS during recovery == 20:42:42 (1763343762)
fail_loc=0x00001310
Failing mds1 on oleg106-server
Stopping /mnt/lustre-mds1 (opts:) on oleg106-server
20:42:49 (1763343769) shut down
facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server
Failover mds1 to oleg106-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all
pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1
Started lustre-MDT0000
20:43:08 (1763343788) targets are mounted
20:43:08 (1763343788) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg106-server
Stopping /mnt/lustre-mds1 (opts:) on oleg106-server
20:43:15 (1763343795) shut down
facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server
Failover mds1 to oleg106-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all
pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1
Started lustre-MDT0000
20:43:45 (1763343825) targets are mounted
20:43:45 (1763343825) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg106-server
Stopping /mnt/lustre-mds1 (opts:) on oleg106-server
20:43:55 (1763343835) shut down
facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server
Failover mds1 to oleg106-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all
pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1
Started lustre-MDT0000
20:44:17 (1763343857) targets are mounted
20:44:17 (1763343857) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg106-server
Stopping /mnt/lustre-mds1 (opts:) on oleg106-server
20:44:34 (1763343874) shut down
facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server
Failover mds1 to oleg106-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all
pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1
Started lustre-MDT0000
20:44:54 (1763343894) targets are mounted
20:44:54 (1763343894) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg106-server
Stopping /mnt/lustre-mds1 (opts:) on oleg106-server
20:45:20 (1763343920) shut down
facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server
Failover mds1 to oleg106-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all
pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1
Started lustre-MDT0000
20:45:52 (1763343952) targets are mounted
20:45:52 (1763343952) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg106-server
Stopping /mnt/lustre-mds1 (opts:) on oleg106-server
20:46:23 (1763343983) shut down
facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server
Failover mds1 to oleg106-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all
pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1
Started lustre-MDT0000
20:46:43 (1763344003) targets are mounted
20:46:43 (1763344003) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg106-server
Stopping /mnt/lustre-mds1 (opts:) on oleg106-server
20:47:21 (1763344041) shut down
facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server
Failover mds1 to oleg106-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all
pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1
Started lustre-MDT0000
20:47:52 (1763344072) targets are mounted
20:47:52 (1763344072) facet_failover done
writemany returned 0