== recovery-small test 51: failover MDS during recovery == 21:46:55 (1723254415)
fail_loc=0x00001310
Failing mds1 on oleg353-server
Stopping /mnt/lustre-mds1 (opts:) on oleg353-server
21:46:59 (1723254419) shut down
Failover mds1 to oleg353-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all
pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1
Started lustre-MDT0000
21:47:14 (1723254434) targets are mounted
21:47:14 (1723254434) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg353-server
Stopping /mnt/lustre-mds1 (opts:) on oleg353-server
21:47:17 (1723254437) shut down
Failover mds1 to oleg353-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all
pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1
Started lustre-MDT0000
21:47:30 (1723254450) targets are mounted
21:47:30 (1723254450) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg353-server
Stopping /mnt/lustre-mds1 (opts:) on oleg353-server
21:47:37 (1723254457) shut down
Failover mds1 to oleg353-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all
pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1
Started lustre-MDT0000
21:47:52 (1723254472) targets are mounted
21:47:52 (1723254472) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg353-server
Stopping /mnt/lustre-mds1 (opts:) on oleg353-server
21:48:04 (1723254484) shut down
Failover mds1 to oleg353-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all
pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1
Started lustre-MDT0000
21:48:17 (1723254497) targets are mounted
21:48:17 (1723254497) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg353-server
Stopping /mnt/lustre-mds1 (opts:) on oleg353-server
21:48:38 (1723254518) shut down
Failover mds1 to oleg353-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all
pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1
Started lustre-MDT0000
21:48:52 (1723254532) targets are mounted
21:48:52 (1723254532) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg353-server
Stopping /mnt/lustre-mds1 (opts:) on oleg353-server
21:49:18 (1723254558) shut down
Failover mds1 to oleg353-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all
pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1
Started lustre-MDT0000
21:49:32 (1723254572) targets are mounted
21:49:32 (1723254572) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg353-server
Stopping /mnt/lustre-mds1 (opts:) on oleg353-server
21:50:04 (1723254604) shut down
Failover mds1 to oleg353-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all
pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1
Started lustre-MDT0000
21:50:18 (1723254618) targets are mounted
21:50:18 (1723254618) facet_failover done
writemany returned 0