== recovery-small test 51: failover MDS during recovery == 10:15:04 (1746627304)
fail_loc=0x00001310
Failing mds1 on oleg342-server
Stopping /mnt/lustre-mds1 (opts:) on oleg342-server
10:15:06 (1746627306) shut down
facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server
Failover mds1 to oleg342-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all
pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1
Started lustre-MDT0000
10:15:21 (1746627321) targets are mounted
10:15:21 (1746627321) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg342-server
Stopping /mnt/lustre-mds1 (opts:) on oleg342-server
10:15:25 (1746627325) shut down
facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server
Failover mds1 to oleg342-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all
pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1
Started lustre-MDT0000
10:15:41 (1746627341) targets are mounted
10:15:41 (1746627341) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg342-server
Stopping /mnt/lustre-mds1 (opts:) on oleg342-server
10:15:48 (1746627348) shut down
facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server
Failover mds1 to oleg342-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all
pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1
Started lustre-MDT0000
10:16:01 (1746627361) targets are mounted
10:16:01 (1746627361) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg342-server
Stopping /mnt/lustre-mds1 (opts:) on oleg342-server
10:16:13 (1746627373) shut down
facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server
Failover mds1 to oleg342-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all
pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1
Started lustre-MDT0000
10:16:27 (1746627387) targets are mounted
10:16:27 (1746627387) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg342-server
Stopping /mnt/lustre-mds1 (opts:) on oleg342-server
10:16:48 (1746627408) shut down
facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server
Failover mds1 to oleg342-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all
pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1
Started lustre-MDT0000
10:17:01 (1746627421) targets are mounted
10:17:01 (1746627421) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg342-server
Stopping /mnt/lustre-mds1 (opts:) on oleg342-server
10:17:28 (1746627448) shut down
facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server
Failover mds1 to oleg342-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all
pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1
Started lustre-MDT0000
10:17:40 (1746627460) targets are mounted
10:17:40 (1746627460) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg342-server
Stopping /mnt/lustre-mds1 (opts:) on oleg342-server
10:18:12 (1746627492) shut down
facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server
Failover mds1 to oleg342-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all
pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1
Started lustre-MDT0000
10:18:24 (1746627504) targets are mounted
10:18:24 (1746627504) facet_failover done
writemany returned 0