== recovery-small test 51: failover MDS during recovery == 20:50:18 (1775868618)
fail_loc=0x00001310
Failing mds1 on oleg418-server
Stopping /mnt/lustre-mds1 (opts:) on oleg418-server
20:50:27 (1775868627) shut down
facet: mds1 facet_host: oleg418-server facet_failover_host: oleg418-server
Failover mds1 to oleg418-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all
pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1
Started lustre-MDT0000
20:50:59 (1775868659) targets are mounted
20:50:59 (1775868659) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg418-server
Stopping /mnt/lustre-mds1 (opts:) on oleg418-server
20:51:07 (1775868667) shut down
facet: mds1 facet_host: oleg418-server facet_failover_host: oleg418-server
Failover mds1 to oleg418-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all
pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1
Started lustre-MDT0000
20:51:41 (1775868701) targets are mounted
20:51:41 (1775868701) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg418-server
Stopping /mnt/lustre-mds1 (opts:) on oleg418-server
20:51:54 (1775868714) shut down
facet: mds1 facet_host: oleg418-server facet_failover_host: oleg418-server
Failover mds1 to oleg418-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all
pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1
Started lustre-MDT0000
20:52:27 (1775868747) targets are mounted
20:52:27 (1775868747) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg418-server
Stopping /mnt/lustre-mds1 (opts:) on oleg418-server
20:52:45 (1775868765) shut down
facet: mds1 facet_host: oleg418-server facet_failover_host: oleg418-server
Failover mds1 to oleg418-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all
pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1
Started lustre-MDT0000
20:53:20 (1775868800) targets are mounted
20:53:20 (1775868800) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg418-server
Stopping /mnt/lustre-mds1 (opts:) on oleg418-server
20:53:49 (1775868829) shut down
facet: mds1 facet_host: oleg418-server facet_failover_host: oleg418-server
Failover mds1 to oleg418-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all
pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1
Started lustre-MDT0000
20:54:23 (1775868863) targets are mounted
20:54:23 (1775868863) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg418-server
Stopping /mnt/lustre-mds1 (opts:) on oleg418-server
20:54:57 (1775868897) shut down
facet: mds1 facet_host: oleg418-server facet_failover_host: oleg418-server
Failover mds1 to oleg418-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all
pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1
Started lustre-MDT0000
20:55:33 (1775868933) targets are mounted
20:55:33 (1775868933) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg418-server
Stopping /mnt/lustre-mds1 (opts:) on oleg418-server
20:56:10 (1775868970) shut down
facet: mds1 facet_host: oleg418-server facet_failover_host: oleg418-server
Failover mds1 to oleg418-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all
pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1
Started lustre-MDT0000
20:56:45 (1775869005) targets are mounted
20:56:45 (1775869005) facet_failover done
writemany returned 0