== recovery-small test 51: failover MDS during recovery == 00:14:10 (1790741650)
fail_loc=0x00001310
Failing mds1 on oleg157-server
Stopping /mnt/lustre-mds1 (opts:) on oleg157-server
00:14:19 (1790741659) shut down
facet: mds1 facet_host: oleg157-server facet_failover_host: oleg157-server
Failover mds1 to oleg157-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all
pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1
Started lustre-MDT0000
00:14:53 (1790741693) targets are mounted
00:14:53 (1790741693) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg157-server
Stopping /mnt/lustre-mds1 (opts:) on oleg157-server
00:15:01 (1790741701) shut down
facet: mds1 facet_host: oleg157-server facet_failover_host: oleg157-server
Failover mds1 to oleg157-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all
pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1
Started lustre-MDT0000
00:15:33 (1790741733) targets are mounted
00:15:33 (1790741733) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg157-server
Stopping /mnt/lustre-mds1 (opts:) on oleg157-server
00:15:46 (1790741746) shut down
facet: mds1 facet_host: oleg157-server facet_failover_host: oleg157-server
Failover mds1 to oleg157-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all
pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1
Started lustre-MDT0000
00:16:19 (1790741779) targets are mounted
00:16:19 (1790741779) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg157-server
Stopping /mnt/lustre-mds1 (opts:) on oleg157-server
00:16:37 (1790741797) shut down
facet: mds1 facet_host: oleg157-server facet_failover_host: oleg157-server
Failover mds1 to oleg157-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all
pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1
Started lustre-MDT0000
00:17:10 (1790741830) targets are mounted
00:17:10 (1790741830) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg157-server
Stopping /mnt/lustre-mds1 (opts:) on oleg157-server
00:17:37 (1790741857) shut down
facet: mds1 facet_host: oleg157-server facet_failover_host: oleg157-server
Failover mds1 to oleg157-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all
pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1
Started lustre-MDT0000
00:18:10 (1790741890) targets are mounted
00:18:10 (1790741890) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg157-server
Stopping /mnt/lustre-mds1 (opts:) on oleg157-server
00:18:44 (1790741924) shut down
facet: mds1 facet_host: oleg157-server facet_failover_host: oleg157-server
Failover mds1 to oleg157-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all
pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1
Started lustre-MDT0000
00:19:16 (1790741956) targets are mounted
00:19:16 (1790741956) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg157-server
Stopping /mnt/lustre-mds1 (opts:) on oleg157-server
00:19:53 (1790741993) shut down
facet: mds1 facet_host: oleg157-server facet_failover_host: oleg157-server
Failover mds1 to oleg157-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all
pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1
Started lustre-MDT0000
00:20:18 (1790742018) targets are mounted
00:20:18 (1790742018) facet_failover done
writemany returned 0