== recovery-small test 51: failover MDS during recovery == 15:35:32 (1746128132)
fail_loc=0x00001310
Failing mds1 on oleg448-server
Stopping /mnt/lustre-mds1 (opts:) on oleg448-server
15:35:38 (1746128138) shut down
facet: mds1 facet_host: oleg448-server facet_failover_host: oleg448-server
Failover mds1 to oleg448-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all
pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1
Started lustre-MDT0000
15:35:54 (1746128154) targets are mounted
15:35:55 (1746128155) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg448-server
Stopping /mnt/lustre-mds1 (opts:) on oleg448-server
15:36:00 (1746128160) shut down
facet: mds1 facet_host: oleg448-server facet_failover_host: oleg448-server
Failover mds1 to oleg448-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all
pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1
Started lustre-MDT0000
15:36:17 (1746128177) targets are mounted
15:36:17 (1746128177) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg448-server
Stopping /mnt/lustre-mds1 (opts:) on oleg448-server
15:36:27 (1746128187) shut down
facet: mds1 facet_host: oleg448-server facet_failover_host: oleg448-server
Failover mds1 to oleg448-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all
pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1
Started lustre-MDT0000
15:36:45 (1746128205) targets are mounted
15:36:45 (1746128205) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg448-server
Stopping /mnt/lustre-mds1 (opts:) on oleg448-server
15:37:00 (1746128220) shut down
facet: mds1 facet_host: oleg448-server facet_failover_host: oleg448-server
Failover mds1 to oleg448-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all
pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1
Started lustre-MDT0000
15:37:18 (1746128238) targets are mounted
15:37:18 (1746128238) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg448-server
Stopping /mnt/lustre-mds1 (opts:) on oleg448-server
15:37:43 (1746128263) shut down
facet: mds1 facet_host: oleg448-server facet_failover_host: oleg448-server
Failover mds1 to oleg448-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all
pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1
Started lustre-MDT0000
15:38:00 (1746128280) targets are mounted
15:38:00 (1746128280) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg448-server
Stopping /mnt/lustre-mds1 (opts:) on oleg448-server
15:38:30 (1746128310) shut down
facet: mds1 facet_host: oleg448-server facet_failover_host: oleg448-server
Failover mds1 to oleg448-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all
pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1
Started lustre-MDT0000
15:38:47 (1746128327) targets are mounted
15:38:47 (1746128327) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg448-server
Stopping /mnt/lustre-mds1 (opts:) on oleg448-server
15:39:20 (1746128360) shut down
facet: mds1 facet_host: oleg448-server facet_failover_host: oleg448-server
Failover mds1 to oleg448-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all
pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1
Started lustre-MDT0000
15:39:37 (1746128377) targets are mounted
15:39:37 (1746128377) facet_failover done
writemany returned 0