== recovery-small test 51: failover MDS during recovery == 01:11:51 (1787375511)
fail_loc=0x00001310
Failing mds1 on oleg237-server
Stopping /mnt/lustre-mds1 (opts:) on oleg237-server
01:11:58 (1787375518) shut down
facet: mds1 facet_host: oleg237-server facet_failover_host: oleg237-server
Failover mds1 to oleg237-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Started lustre-MDT0000
01:12:21 (1787375541) targets are mounted
01:12:21 (1787375541) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg237-server
Stopping /mnt/lustre-mds1 (opts:) on oleg237-server
01:12:28 (1787375548) shut down
facet: mds1 facet_host: oleg237-server facet_failover_host: oleg237-server
Failover mds1 to oleg237-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Started lustre-MDT0000
01:12:51 (1787375571) targets are mounted
01:12:51 (1787375571) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg237-server
Stopping /mnt/lustre-mds1 (opts:) on oleg237-server
01:13:02 (1787375582) shut down
facet: mds1 facet_host: oleg237-server facet_failover_host: oleg237-server
Failover mds1 to oleg237-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Started lustre-MDT0000
01:13:26 (1787375606) targets are mounted
01:13:26 (1787375606) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg237-server
Stopping /mnt/lustre-mds1 (opts:) on oleg237-server
01:13:43 (1787375623) shut down
facet: mds1 facet_host: oleg237-server facet_failover_host: oleg237-server
Failover mds1 to oleg237-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Started lustre-MDT0000
01:14:07 (1787375647) targets are mounted
01:14:07 (1787375647) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg237-server
Stopping /mnt/lustre-mds1 (opts:) on oleg237-server
01:14:35 (1787375675) shut down
facet: mds1 facet_host: oleg237-server facet_failover_host: oleg237-server
Failover mds1 to oleg237-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Started lustre-MDT0000
01:15:10 (1787375710) targets are mounted
01:15:10 (1787375710) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg237-server
Stopping /mnt/lustre-mds1 (opts:) on oleg237-server
01:15:43 (1787375743) shut down
facet: mds1 facet_host: oleg237-server facet_failover_host: oleg237-server
Failover mds1 to oleg237-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Started lustre-MDT0000
01:16:05 (1787375765) targets are mounted
01:16:05 (1787375765) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg237-server
Stopping /mnt/lustre-mds1 (opts:) on oleg237-server
01:16:42 (1787375802) shut down
facet: mds1 facet_host: oleg237-server facet_failover_host: oleg237-server
Failover mds1 to oleg237-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all
pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1
Started lustre-MDT0000
01:17:15 (1787375835) targets are mounted
01:17:15 (1787375835) facet_failover done
writemany returned 0