== recovery-small test 51: failover MDS during recovery == 13:37:45 (1741628265)
fail_loc=0x00001310
Failing mds1 on oleg410-server
Stopping /mnt/lustre-mds1 (opts:) on oleg410-server
13:37:48 (1741628268) shut down
facet: mds1 facet_host: oleg410-server facet_failover_host: oleg410-server
Failover mds1 to oleg410-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all
pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1
Started lustre-MDT0000
13:38:01 (1741628281) targets are mounted
13:38:01 (1741628281) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg410-server
Stopping /mnt/lustre-mds1 (opts:) on oleg410-server
13:38:04 (1741628284) shut down
facet: mds1 facet_host: oleg410-server facet_failover_host: oleg410-server
Failover mds1 to oleg410-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all
pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1
Started lustre-MDT0000
13:38:16 (1741628296) targets are mounted
13:38:16 (1741628296) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg410-server
Stopping /mnt/lustre-mds1 (opts:) on oleg410-server
13:38:23 (1741628303) shut down
facet: mds1 facet_host: oleg410-server facet_failover_host: oleg410-server
Failover mds1 to oleg410-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all
pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1
Started lustre-MDT0000
13:38:36 (1741628316) targets are mounted
13:38:36 (1741628316) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg410-server
Stopping /mnt/lustre-mds1 (opts:) on oleg410-server
13:38:47 (1741628327) shut down
facet: mds1 facet_host: oleg410-server facet_failover_host: oleg410-server
Failover mds1 to oleg410-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all
pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1
Started lustre-MDT0000
13:39:00 (1741628340) targets are mounted
13:39:00 (1741628340) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg410-server
Stopping /mnt/lustre-mds1 (opts:) on oleg410-server
13:39:22 (1741628362) shut down
facet: mds1 facet_host: oleg410-server facet_failover_host: oleg410-server
Failover mds1 to oleg410-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all
pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1
Started lustre-MDT0000
13:39:34 (1741628374) targets are mounted
13:39:34 (1741628374) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg410-server
Stopping /mnt/lustre-mds1 (opts:) on oleg410-server
13:40:01 (1741628401) shut down
facet: mds1 facet_host: oleg410-server facet_failover_host: oleg410-server
Failover mds1 to oleg410-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all
pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1
Started lustre-MDT0000
13:40:14 (1741628414) targets are mounted
13:40:14 (1741628414) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg410-server
Stopping /mnt/lustre-mds1 (opts:) on oleg410-server
13:40:45 (1741628445) shut down
facet: mds1 facet_host: oleg410-server facet_failover_host: oleg410-server
Failover mds1 to oleg410-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all
pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1
Started lustre-MDT0000
13:40:58 (1741628458) targets are mounted
13:40:58 (1741628458) facet_failover done
writemany returned 0