== recovery-small test 51: failover MDS during recovery == 20:35:51 (1712277351)
fail_loc=0x00001310
Failing mds1 on oleg215-server
Stopping /mnt/lustre-mds1 (opts:) on oleg215-server
20:35:54 (1712277354) shut down
Failover mds1 to oleg215-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all
pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1
Started lustre-MDT0000
20:36:07 (1712277367) targets are mounted
20:36:07 (1712277367) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg215-server
Stopping /mnt/lustre-mds1 (opts:) on oleg215-server
20:36:10 (1712277370) shut down
Failover mds1 to oleg215-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all
pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1
Started lustre-MDT0000
20:36:22 (1712277382) targets are mounted
20:36:22 (1712277382) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg215-server
Stopping /mnt/lustre-mds1 (opts:) on oleg215-server
20:36:29 (1712277389) shut down
Failover mds1 to oleg215-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all
pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1
Started lustre-MDT0000
20:36:42 (1712277402) targets are mounted
20:36:42 (1712277402) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg215-server
Stopping /mnt/lustre-mds1 (opts:) on oleg215-server
20:36:53 (1712277413) shut down
Failover mds1 to oleg215-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all
pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1
Started lustre-MDT0000
20:37:05 (1712277425) targets are mounted
20:37:06 (1712277426) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg215-server
Stopping /mnt/lustre-mds1 (opts:) on oleg215-server
20:37:27 (1712277447) shut down
Failover mds1 to oleg215-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all
pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1
Started lustre-MDT0000
20:37:39 (1712277459) targets are mounted
20:37:39 (1712277459) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg215-server
Stopping /mnt/lustre-mds1 (opts:) on oleg215-server
20:38:06 (1712277486) shut down
Failover mds1 to oleg215-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all
pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1
Started lustre-MDT0000
20:38:18 (1712277498) targets are mounted
20:38:18 (1712277498) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg215-server
Stopping /mnt/lustre-mds1 (opts:) on oleg215-server
20:38:50 (1712277530) shut down
Failover mds1 to oleg215-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all
pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1
Started lustre-MDT0000
20:39:03 (1712277543) targets are mounted
20:39:03 (1712277543) facet_failover done
writemany returned 0