== recovery-small test 51: failover MDS during recovery == 10:42:08 (1749480128)
fail_loc=0x00001310
Failing mds1 on oleg334-server
Stopping /mnt/lustre-mds1 (opts:) on oleg334-server
10:42:11 (1749480131) shut down
facet: mds1 facet_host: oleg334-server facet_failover_host: oleg334-server
Failover mds1 to oleg334-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all
pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1
Started lustre-MDT0000
10:42:25 (1749480145) targets are mounted
10:42:25 (1749480145) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg334-server
Stopping /mnt/lustre-mds1 (opts:) on oleg334-server
10:42:28 (1749480148) shut down
facet: mds1 facet_host: oleg334-server facet_failover_host: oleg334-server
Failover mds1 to oleg334-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all
pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1
Started lustre-MDT0000
10:42:41 (1749480161) targets are mounted
10:42:41 (1749480161) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg334-server
Stopping /mnt/lustre-mds1 (opts:) on oleg334-server
10:42:48 (1749480168) shut down
facet: mds1 facet_host: oleg334-server facet_failover_host: oleg334-server
Failover mds1 to oleg334-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all
pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1
Started lustre-MDT0000
10:43:02 (1749480182) targets are mounted
10:43:02 (1749480182) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg334-server
Stopping /mnt/lustre-mds1 (opts:) on oleg334-server
10:43:13 (1749480193) shut down
facet: mds1 facet_host: oleg334-server facet_failover_host: oleg334-server
Failover mds1 to oleg334-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all
pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1
Started lustre-MDT0000
10:43:26 (1749480206) targets are mounted
10:43:26 (1749480206) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg334-server
Stopping /mnt/lustre-mds1 (opts:) on oleg334-server
10:43:48 (1749480228) shut down
facet: mds1 facet_host: oleg334-server facet_failover_host: oleg334-server
Failover mds1 to oleg334-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all
pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1
Started lustre-MDT0000
10:44:01 (1749480241) targets are mounted
10:44:01 (1749480241) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg334-server
Stopping /mnt/lustre-mds1 (opts:) on oleg334-server
10:44:28 (1749480268) shut down
facet: mds1 facet_host: oleg334-server facet_failover_host: oleg334-server
Failover mds1 to oleg334-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all
pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1
Started lustre-MDT0000
10:44:42 (1749480282) targets are mounted
10:44:42 (1749480282) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg334-server
Stopping /mnt/lustre-mds1 (opts:) on oleg334-server
10:45:13 (1749480313) shut down
facet: mds1 facet_host: oleg334-server facet_failover_host: oleg334-server
Failover mds1 to oleg334-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all
pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1
Started lustre-MDT0000
10:45:26 (1749480326) targets are mounted
10:45:26 (1749480326) facet_failover done
writemany returned 0