== recovery-small test 51: failover MDS during recovery == 10:08:35 (1779977315)
fail_loc=0x00001310
Failing mds1 on oleg452-server
Stopping /mnt/lustre-mds1 (opts:) on oleg452-server
10:08:44 (1779977324) shut down
facet: mds1 facet_host: oleg452-server facet_failover_host: oleg452-server
Failover mds1 to oleg452-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all
pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1
Started lustre-MDT0000
10:09:09 (1779977349) targets are mounted
10:09:09 (1779977349) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg452-server
Stopping /mnt/lustre-mds1 (opts:) on oleg452-server
10:09:19 (1779977359) shut down
facet: mds1 facet_host: oleg452-server facet_failover_host: oleg452-server
Failover mds1 to oleg452-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all
pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1
Started lustre-MDT0000
10:09:51 (1779977391) targets are mounted
10:09:51 (1779977391) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg452-server
Stopping /mnt/lustre-mds1 (opts:) on oleg452-server
10:10:02 (1779977402) shut down
facet: mds1 facet_host: oleg452-server facet_failover_host: oleg452-server
Failover mds1 to oleg452-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all
pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1
Started lustre-MDT0000
10:10:24 (1779977424) targets are mounted
10:10:24 (1779977424) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg452-server
Stopping /mnt/lustre-mds1 (opts:) on oleg452-server
10:10:42 (1779977442) shut down
facet: mds1 facet_host: oleg452-server facet_failover_host: oleg452-server
Failover mds1 to oleg452-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all
pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1
Started lustre-MDT0000
10:11:06 (1779977466) targets are mounted
10:11:06 (1779977466) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg452-server
Stopping /mnt/lustre-mds1 (opts:) on oleg452-server
10:11:35 (1779977495) shut down
facet: mds1 facet_host: oleg452-server facet_failover_host: oleg452-server
Failover mds1 to oleg452-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all
pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1
Started lustre-MDT0000
10:12:07 (1779977527) targets are mounted
10:12:07 (1779977527) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg452-server
Stopping /mnt/lustre-mds1 (opts:) on oleg452-server
10:12:41 (1779977561) shut down
facet: mds1 facet_host: oleg452-server facet_failover_host: oleg452-server
Failover mds1 to oleg452-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all
pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1
Started lustre-MDT0000
10:13:03 (1779977583) targets are mounted
10:13:03 (1779977583) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg452-server
Stopping /mnt/lustre-mds1 (opts:) on oleg452-server
10:13:40 (1779977620) shut down
facet: mds1 facet_host: oleg452-server facet_failover_host: oleg452-server
Failover mds1 to oleg452-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all
pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1
Started lustre-MDT0000
10:14:04 (1779977644) targets are mounted
10:14:04 (1779977644) facet_failover done
writemany returned 0