== recovery-small test 51: failover MDS during recovery == 00:57:30 (1708840650)
fail_loc=0x00001310
Failing mds1 on oleg356-server
Stopping /mnt/lustre-mds1 (opts:) on oleg356-server
00:57:33 (1708840653) shut down
Failover mds1 to oleg356-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all
pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1
Started lustre-MDT0000
00:57:47 (1708840667) targets are mounted
00:57:47 (1708840667) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg356-server
Stopping /mnt/lustre-mds1 (opts:) on oleg356-server
00:57:50 (1708840670) shut down
Failover mds1 to oleg356-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all
pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1
Started lustre-MDT0000
00:58:03 (1708840683) targets are mounted
00:58:03 (1708840683) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg356-server
Stopping /mnt/lustre-mds1 (opts:) on oleg356-server
00:58:10 (1708840690) shut down
Failover mds1 to oleg356-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all
pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1
Started lustre-MDT0000
00:58:24 (1708840704) targets are mounted
00:58:24 (1708840704) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg356-server
Stopping /mnt/lustre-mds1 (opts:) on oleg356-server
00:58:36 (1708840716) shut down
Failover mds1 to oleg356-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all
pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1
Started lustre-MDT0000
00:58:49 (1708840729) targets are mounted
00:58:49 (1708840729) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg356-server
Stopping /mnt/lustre-mds1 (opts:) on oleg356-server
00:59:11 (1708840751) shut down
Failover mds1 to oleg356-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all
pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1
Started lustre-MDT0000
00:59:25 (1708840765) targets are mounted
00:59:25 (1708840765) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg356-server
Stopping /mnt/lustre-mds1 (opts:) on oleg356-server
00:59:52 (1708840792) shut down
Failover mds1 to oleg356-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all
pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1
Started lustre-MDT0000
01:00:06 (1708840806) targets are mounted
01:00:06 (1708840806) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg356-server
Stopping /mnt/lustre-mds1 (opts:) on oleg356-server
01:00:38 (1708840838) shut down
Failover mds1 to oleg356-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all
pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1
Started lustre-MDT0000
01:00:51 (1708840851) targets are mounted
01:00:51 (1708840851) facet_failover done
writemany returned 0