== recovery-small test 51: failover MDS during recovery == 02:52:21 (1780383141)
fail_loc=0x00001310
Failing mds1 on oleg126-server
Stopping /mnt/lustre-mds1 (opts:) on oleg126-server
02:52:30 (1780383150) shut down
facet: mds1 facet_host: oleg126-server facet_failover_host: oleg126-server
Failover mds1 to oleg126-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all
pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1
Started lustre-MDT0000
02:53:06 (1780383186) targets are mounted
02:53:06 (1780383186) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg126-server
Stopping /mnt/lustre-mds1 (opts:) on oleg126-server
02:53:15 (1780383195) shut down
facet: mds1 facet_host: oleg126-server facet_failover_host: oleg126-server
Failover mds1 to oleg126-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all
pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1
Started lustre-MDT0000
02:53:44 (1780383224) targets are mounted
02:53:44 (1780383224) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg126-server
Stopping /mnt/lustre-mds1 (opts:) on oleg126-server
02:54:01 (1780383241) shut down
facet: mds1 facet_host: oleg126-server facet_failover_host: oleg126-server
Failover mds1 to oleg126-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all
pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1
Started lustre-MDT0000
02:54:31 (1780383271) targets are mounted
02:54:31 (1780383271) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg126-server
Stopping /mnt/lustre-mds1 (opts:) on oleg126-server
02:54:52 (1780383292) shut down
facet: mds1 facet_host: oleg126-server facet_failover_host: oleg126-server
Failover mds1 to oleg126-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all
pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1
Started lustre-MDT0000
02:55:25 (1780383325) targets are mounted
02:55:25 (1780383325) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg126-server
Stopping /mnt/lustre-mds1 (opts:) on oleg126-server
02:55:55 (1780383355) shut down
facet: mds1 facet_host: oleg126-server facet_failover_host: oleg126-server
Failover mds1 to oleg126-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all
pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1
Started lustre-MDT0000
02:56:27 (1780383387) targets are mounted
02:56:27 (1780383387) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg126-server
Stopping /mnt/lustre-mds1 (opts:) on oleg126-server
02:57:01 (1780383421) shut down
facet: mds1 facet_host: oleg126-server facet_failover_host: oleg126-server
Failover mds1 to oleg126-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all
pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1
Started lustre-MDT0000
02:57:35 (1780383455) targets are mounted
02:57:35 (1780383455) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg126-server
Stopping /mnt/lustre-mds1 (opts:) on oleg126-server
02:58:14 (1780383494) shut down
facet: mds1 facet_host: oleg126-server facet_failover_host: oleg126-server
Failover mds1 to oleg126-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all
pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1
Started lustre-MDT0000
02:58:48 (1780383528) targets are mounted
02:58:48 (1780383528) facet_failover done
writemany returned 0