== recovery-small test 51: failover MDS during recovery == 06:24:46 (1769081086)
fail_loc=0x00001310
Failing mds1 on oleg459-server
Stopping /mnt/lustre-mds1 (opts:) on oleg459-server
06:24:54 (1769081094) shut down
facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server
Failover mds1 to oleg459-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1
Started lustre-MDT0000
06:25:14 (1769081114) targets are mounted
06:25:14 (1769081114) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg459-server
Stopping /mnt/lustre-mds1 (opts:) on oleg459-server
06:25:22 (1769081122) shut down
facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server
Failover mds1 to oleg459-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1
Started lustre-MDT0000
06:25:40 (1769081140) targets are mounted
06:25:40 (1769081140) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg459-server
Stopping /mnt/lustre-mds1 (opts:) on oleg459-server
06:25:50 (1769081150) shut down
facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server
Failover mds1 to oleg459-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1
Started lustre-MDT0000
06:26:11 (1769081171) targets are mounted
06:26:11 (1769081171) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg459-server
Stopping /mnt/lustre-mds1 (opts:) on oleg459-server
06:26:28 (1769081188) shut down
facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server
Failover mds1 to oleg459-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1
Started lustre-MDT0000
06:26:48 (1769081208) targets are mounted
06:26:48 (1769081208) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg459-server
Stopping /mnt/lustre-mds1 (opts:) on oleg459-server
06:27:14 (1769081234) shut down
facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server
Failover mds1 to oleg459-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1
Started lustre-MDT0000
06:27:33 (1769081253) targets are mounted
06:27:33 (1769081253) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg459-server
Stopping /mnt/lustre-mds1 (opts:) on oleg459-server
06:28:04 (1769081284) shut down
facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server
Failover mds1 to oleg459-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1
Started lustre-MDT0000
06:28:25 (1769081305) targets are mounted
06:28:25 (1769081305) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg459-server
Stopping /mnt/lustre-mds1 (opts:) on oleg459-server
06:29:00 (1769081340) shut down
facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server
Failover mds1 to oleg459-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all
pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1
Started lustre-MDT0000
06:29:20 (1769081360) targets are mounted
06:29:20 (1769081360) facet_failover done
writemany returned 0