== recovery-small test 51: failover MDS during recovery == 03:52:18 (1779695538)
fail_loc=0x00001310
Failing mds1 on oleg425-server
Stopping /mnt/lustre-mds1 (opts:) on oleg425-server
03:52:26 (1779695546) shut down
facet: mds1 facet_host: oleg425-server facet_failover_host: oleg425-server
Failover mds1 to oleg425-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all
pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1
Started lustre-MDT0000
03:52:52 (1779695572) targets are mounted
03:52:52 (1779695572) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg425-server
Stopping /mnt/lustre-mds1 (opts:) on oleg425-server
03:52:59 (1779695579) shut down
facet: mds1 facet_host: oleg425-server facet_failover_host: oleg425-server
Failover mds1 to oleg425-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all
pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1
Started lustre-MDT0000
03:53:22 (1779695602) targets are mounted
03:53:22 (1779695602) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg425-server
Stopping /mnt/lustre-mds1 (opts:) on oleg425-server
03:53:34 (1779695614) shut down
facet: mds1 facet_host: oleg425-server facet_failover_host: oleg425-server
Failover mds1 to oleg425-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all
pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1
Started lustre-MDT0000
03:54:08 (1779695648) targets are mounted
03:54:08 (1779695648) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg425-server
Stopping /mnt/lustre-mds1 (opts:) on oleg425-server
03:54:25 (1779695665) shut down
facet: mds1 facet_host: oleg425-server facet_failover_host: oleg425-server
Failover mds1 to oleg425-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all
pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1
Started lustre-MDT0000
03:54:50 (1779695690) targets are mounted
03:54:50 (1779695690) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg425-server
Stopping /mnt/lustre-mds1 (opts:) on oleg425-server
03:55:18 (1779695718) shut down
facet: mds1 facet_host: oleg425-server facet_failover_host: oleg425-server
Failover mds1 to oleg425-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all
pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1
Started lustre-MDT0000
03:55:53 (1779695753) targets are mounted
03:55:53 (1779695753) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg425-server
Stopping /mnt/lustre-mds1 (opts:) on oleg425-server
03:56:25 (1779695785) shut down
facet: mds1 facet_host: oleg425-server facet_failover_host: oleg425-server
Failover mds1 to oleg425-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all
pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1
Started lustre-MDT0000
03:56:59 (1779695819) targets are mounted
03:56:59 (1779695819) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg425-server
Stopping /mnt/lustre-mds1 (opts:) on oleg425-server
03:57:36 (1779695856) shut down
facet: mds1 facet_host: oleg425-server facet_failover_host: oleg425-server
Failover mds1 to oleg425-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all
pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1
Started lustre-MDT0000
03:57:59 (1779695879) targets are mounted
03:57:59 (1779695879) facet_failover done
writemany returned 0