== recovery-small test 51: failover MDS during recovery == 14:28:07 (1787855287)
fail_loc=0x00001310
Failing mds1 on oleg602-server
Stopping /mnt/lustre-mds1 (opts:) on oleg602-server
14:28:14 (1787855294) shut down
facet: mds1 facet_host: oleg602-server facet_failover_host: oleg602-server
Failover mds1 to oleg602-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg602-server: oleg602-server.virtnet: executing set_default_debug -1 all
pdsh@oleg602-client: oleg602-server: ssh exited with exit code 1
Started lustre-MDT0000
14:28:37 (1787855317) targets are mounted
14:28:37 (1787855317) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg602-server
Stopping /mnt/lustre-mds1 (opts:) on oleg602-server
14:28:44 (1787855324) shut down
facet: mds1 facet_host: oleg602-server facet_failover_host: oleg602-server
Failover mds1 to oleg602-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg602-server: oleg602-server.virtnet: executing set_default_debug -1 all
pdsh@oleg602-client: oleg602-server: ssh exited with exit code 1
Started lustre-MDT0000
14:29:05 (1787855345) targets are mounted
14:29:05 (1787855345) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg602-server
Stopping /mnt/lustre-mds1 (opts:) on oleg602-server
14:29:17 (1787855357) shut down
facet: mds1 facet_host: oleg602-server facet_failover_host: oleg602-server
Failover mds1 to oleg602-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg602-server: oleg602-server.virtnet: executing set_default_debug -1 all
pdsh@oleg602-client: oleg602-server: ssh exited with exit code 1
Started lustre-MDT0000
14:29:47 (1787855387) targets are mounted
14:29:47 (1787855387) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg602-server
Stopping /mnt/lustre-mds1 (opts:) on oleg602-server
14:30:04 (1787855404) shut down
facet: mds1 facet_host: oleg602-server facet_failover_host: oleg602-server
Failover mds1 to oleg602-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg602-server: oleg602-server.virtnet: executing set_default_debug -1 all
pdsh@oleg602-client: oleg602-server: ssh exited with exit code 1
Started lustre-MDT0000
14:30:27 (1787855427) targets are mounted
14:30:27 (1787855427) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg602-server
Stopping /mnt/lustre-mds1 (opts:) on oleg602-server
14:30:55 (1787855455) shut down
facet: mds1 facet_host: oleg602-server facet_failover_host: oleg602-server
Failover mds1 to oleg602-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg602-server: oleg602-server.virtnet: executing set_default_debug -1 all
pdsh@oleg602-client: oleg602-server: ssh exited with exit code 1
Started lustre-MDT0000
14:31:17 (1787855477) targets are mounted
14:31:17 (1787855477) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg602-server
Stopping /mnt/lustre-mds1 (opts:) on oleg602-server
14:31:50 (1787855510) shut down
facet: mds1 facet_host: oleg602-server facet_failover_host: oleg602-server
Failover mds1 to oleg602-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg602-server: oleg602-server.virtnet: executing set_default_debug -1 all
pdsh@oleg602-client: oleg602-server: ssh exited with exit code 1
Started lustre-MDT0000
14:32:14 (1787855534) targets are mounted
14:32:14 (1787855534) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg602-server
Stopping /mnt/lustre-mds1 (opts:) on oleg602-server
14:32:53 (1787855573) shut down
facet: mds1 facet_host: oleg602-server facet_failover_host: oleg602-server
Failover mds1 to oleg602-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg602-server: oleg602-server.virtnet: executing set_default_debug -1 all
pdsh@oleg602-client: oleg602-server: ssh exited with exit code 1
Started lustre-MDT0000
14:33:27 (1787855607) targets are mounted
14:33:27 (1787855607) facet_failover done
writemany returned 0