== recovery-small test 51: failover MDS during recovery == 17:40:43 (1786830043)
fail_loc=0x00001310
Failing mds1 on oleg235-server
Stopping /mnt/lustre-mds1 (opts:) on oleg235-server
17:40:51 (1786830051) shut down
facet: mds1 facet_host: oleg235-server facet_failover_host: oleg235-server
Failover mds1 to oleg235-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all
pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1
Started lustre-MDT0000
17:41:16 (1786830076) targets are mounted
17:41:16 (1786830076) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg235-server
Stopping /mnt/lustre-mds1 (opts:) on oleg235-server
17:41:23 (1786830083) shut down
facet: mds1 facet_host: oleg235-server facet_failover_host: oleg235-server
Failover mds1 to oleg235-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all
pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1
Started lustre-MDT0000
17:41:57 (1786830117) targets are mounted
17:41:57 (1786830117) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg235-server
Stopping /mnt/lustre-mds1 (opts:) on oleg235-server
17:42:09 (1786830129) shut down
facet: mds1 facet_host: oleg235-server facet_failover_host: oleg235-server
Failover mds1 to oleg235-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all
pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1
Started lustre-MDT0000
17:42:45 (1786830165) targets are mounted
17:42:45 (1786830165) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg235-server
Stopping /mnt/lustre-mds1 (opts:) on oleg235-server
17:43:05 (1786830185) shut down
facet: mds1 facet_host: oleg235-server facet_failover_host: oleg235-server
Failover mds1 to oleg235-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all
pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1
Started lustre-MDT0000
17:43:40 (1786830220) targets are mounted
17:43:40 (1786830220) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg235-server
Stopping /mnt/lustre-mds1 (opts:) on oleg235-server
17:44:10 (1786830250) shut down
facet: mds1 facet_host: oleg235-server facet_failover_host: oleg235-server
Failover mds1 to oleg235-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all
pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1
Started lustre-MDT0000
17:44:42 (1786830282) targets are mounted
17:44:42 (1786830282) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg235-server
Stopping /mnt/lustre-mds1 (opts:) on oleg235-server
17:45:17 (1786830317) shut down
facet: mds1 facet_host: oleg235-server facet_failover_host: oleg235-server
Failover mds1 to oleg235-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all
pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1
Started lustre-MDT0000
17:45:50 (1786830350) targets are mounted
17:45:50 (1786830350) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg235-server
Stopping /mnt/lustre-mds1 (opts:) on oleg235-server
17:46:28 (1786830388) shut down
facet: mds1 facet_host: oleg235-server facet_failover_host: oleg235-server
Failover mds1 to oleg235-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all
pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1
Started lustre-MDT0000
17:47:05 (1786830425) targets are mounted
17:47:05 (1786830425) facet_failover done
writemany returned 0