== recovery-small test 51: failover MDS during recovery == 17:51:23 (1746136283)
fail_loc=0x00001310
Failing mds1 on oleg308-server
Stopping /mnt/lustre-mds1 (opts:) on oleg308-server
17:51:27 (1746136287) shut down
facet: mds1 facet_host: oleg308-server facet_failover_host: oleg308-server
Failover mds1 to oleg308-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all
pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1
Started lustre-MDT0000
17:51:41 (1746136301) targets are mounted
17:51:41 (1746136301) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg308-server
Stopping /mnt/lustre-mds1 (opts:) on oleg308-server
17:51:45 (1746136305) shut down
facet: mds1 facet_host: oleg308-server facet_failover_host: oleg308-server
Failover mds1 to oleg308-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all
pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1
Started lustre-MDT0000
17:51:59 (1746136319) targets are mounted
17:51:59 (1746136319) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg308-server
Stopping /mnt/lustre-mds1 (opts:) on oleg308-server
17:52:06 (1746136326) shut down
facet: mds1 facet_host: oleg308-server facet_failover_host: oleg308-server
Failover mds1 to oleg308-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all
pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1
Started lustre-MDT0000
17:52:20 (1746136340) targets are mounted
17:52:20 (1746136340) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg308-server
Stopping /mnt/lustre-mds1 (opts:) on oleg308-server
17:52:33 (1746136353) shut down
facet: mds1 facet_host: oleg308-server facet_failover_host: oleg308-server
Failover mds1 to oleg308-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all
pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1
Started lustre-MDT0000
17:52:48 (1746136368) targets are mounted
17:52:48 (1746136368) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg308-server
Stopping /mnt/lustre-mds1 (opts:) on oleg308-server
17:53:11 (1746136391) shut down
facet: mds1 facet_host: oleg308-server facet_failover_host: oleg308-server
Failover mds1 to oleg308-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all
pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1
Started lustre-MDT0000
17:53:25 (1746136405) targets are mounted
17:53:25 (1746136405) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg308-server
Stopping /mnt/lustre-mds1 (opts:) on oleg308-server
17:53:53 (1746136433) shut down
facet: mds1 facet_host: oleg308-server facet_failover_host: oleg308-server
Failover mds1 to oleg308-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all
pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1
Started lustre-MDT0000
17:54:07 (1746136447) targets are mounted
17:54:07 (1746136447) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg308-server
Stopping /mnt/lustre-mds1 (opts:) on oleg308-server
17:54:40 (1746136480) shut down
facet: mds1 facet_host: oleg308-server facet_failover_host: oleg308-server
Failover mds1 to oleg308-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all
pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1
Started lustre-MDT0000
17:54:54 (1746136494) targets are mounted
17:54:54 (1746136494) facet_failover done
writemany returned 0