== recovery-small test 51: failover MDS during recovery == 02:16:42 (1779689802)
fail_loc=0x00001310
Failing mds1 on oleg357-server
Stopping /mnt/lustre-mds1 (opts:) on oleg357-server
02:16:47 (1779689807) shut down
facet: mds1 facet_host: oleg357-server facet_failover_host: oleg357-server
Failover mds1 to oleg357-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all
pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1
Started lustre-MDT0000
02:17:07 (1779689827) targets are mounted
02:17:07 (1779689827) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg357-server
Stopping /mnt/lustre-mds1 (opts:) on oleg357-server
02:17:12 (1779689832) shut down
facet: mds1 facet_host: oleg357-server facet_failover_host: oleg357-server
Failover mds1 to oleg357-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all
pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1
Started lustre-MDT0000
02:17:31 (1779689851) targets are mounted
02:17:31 (1779689851) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg357-server
Stopping /mnt/lustre-mds1 (opts:) on oleg357-server
02:17:41 (1779689861) shut down
facet: mds1 facet_host: oleg357-server facet_failover_host: oleg357-server
Failover mds1 to oleg357-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all
pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1
Started lustre-MDT0000
02:18:02 (1779689882) targets are mounted
02:18:02 (1779689882) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg357-server
Stopping /mnt/lustre-mds1 (opts:) on oleg357-server
02:18:17 (1779689897) shut down
facet: mds1 facet_host: oleg357-server facet_failover_host: oleg357-server
Failover mds1 to oleg357-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all
pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1
Started lustre-MDT0000
02:18:38 (1779689918) targets are mounted
02:18:38 (1779689918) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg357-server
Stopping /mnt/lustre-mds1 (opts:) on oleg357-server
02:19:03 (1779689943) shut down
facet: mds1 facet_host: oleg357-server facet_failover_host: oleg357-server
Failover mds1 to oleg357-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all
pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1
Started lustre-MDT0000
02:19:24 (1779689964) targets are mounted
02:19:24 (1779689964) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg357-server
Stopping /mnt/lustre-mds1 (opts:) on oleg357-server
02:19:55 (1779689995) shut down
facet: mds1 facet_host: oleg357-server facet_failover_host: oleg357-server
Failover mds1 to oleg357-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all
pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1
Started lustre-MDT0000
02:20:14 (1779690014) targets are mounted
02:20:14 (1779690014) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg357-server
Stopping /mnt/lustre-mds1 (opts:) on oleg357-server
02:20:49 (1779690049) shut down
facet: mds1 facet_host: oleg357-server facet_failover_host: oleg357-server
Failover mds1 to oleg357-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all
pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1
Started lustre-MDT0000
02:21:09 (1779690069) targets are mounted
02:21:09 (1779690069) facet_failover done
writemany returned 0