== recovery-small test 51: failover MDS during recovery == 00:30:15 (1734672615)
fail_loc=0x00001310
Failing mds1 on oleg303-server
Stopping /mnt/lustre-mds1 (opts:) on oleg303-server
00:30:20 (1734672620) shut down
facet: mds1 facet_host: oleg303-server facet_failover_host: oleg303-server
Failover mds1 to oleg303-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1
Started lustre-MDT0000
00:30:35 (1734672635) targets are mounted
00:30:35 (1734672635) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg303-server
Stopping /mnt/lustre-mds1 (opts:) on oleg303-server
00:30:39 (1734672639) shut down
facet: mds1 facet_host: oleg303-server facet_failover_host: oleg303-server
Failover mds1 to oleg303-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1
Started lustre-MDT0000
00:30:54 (1734672654) targets are mounted
00:30:54 (1734672654) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg303-server
Stopping /mnt/lustre-mds1 (opts:) on oleg303-server
00:31:01 (1734672661) shut down
facet: mds1 facet_host: oleg303-server facet_failover_host: oleg303-server
Failover mds1 to oleg303-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1
Started lustre-MDT0000
00:31:16 (1734672676) targets are mounted
00:31:16 (1734672676) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg303-server
Stopping /mnt/lustre-mds1 (opts:) on oleg303-server
00:31:29 (1734672689) shut down
facet: mds1 facet_host: oleg303-server facet_failover_host: oleg303-server
Failover mds1 to oleg303-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1
Started lustre-MDT0000
00:31:44 (1734672704) targets are mounted
00:31:44 (1734672704) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg303-server
Stopping /mnt/lustre-mds1 (opts:) on oleg303-server
00:32:13 (1734672733) shut down
facet: mds1 facet_host: oleg303-server facet_failover_host: oleg303-server
Failover mds1 to oleg303-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1
Started lustre-MDT0000
00:32:28 (1734672748) targets are mounted
00:32:28 (1734672748) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg303-server
Stopping /mnt/lustre-mds1 (opts:) on oleg303-server
00:32:56 (1734672776) shut down
facet: mds1 facet_host: oleg303-server facet_failover_host: oleg303-server
Failover mds1 to oleg303-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1
Started lustre-MDT0000
00:33:10 (1734672790) targets are mounted
00:33:10 (1734672790) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg303-server
Stopping /mnt/lustre-mds1 (opts:) on oleg303-server
00:33:43 (1734672823) shut down
facet: mds1 facet_host: oleg303-server facet_failover_host: oleg303-server
Failover mds1 to oleg303-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1
Started lustre-MDT0000
00:33:58 (1734672838) targets are mounted
00:33:58 (1734672838) facet_failover done
writemany returned 0