== recovery-small test 51: failover MDS during recovery == 18:53:26 (1745621606)
fail_loc=0x00001310
Failing mds1 on oleg244-server
Stopping /mnt/lustre-mds1 (opts:) on oleg244-server
18:53:31 (1745621611) shut down
facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server
Failover mds1 to oleg244-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all
pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1
Started lustre-MDT0000
18:53:48 (1745621628) targets are mounted
18:53:48 (1745621628) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg244-server
Stopping /mnt/lustre-mds1 (opts:) on oleg244-server
18:53:52 (1745621632) shut down
facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server
Failover mds1 to oleg244-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all
pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1
Started lustre-MDT0000
18:54:07 (1745621647) targets are mounted
18:54:07 (1745621647) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg244-server
Stopping /mnt/lustre-mds1 (opts:) on oleg244-server
18:54:16 (1745621656) shut down
facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server
Failover mds1 to oleg244-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all
pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1
Started lustre-MDT0000
18:54:32 (1745621672) targets are mounted
18:54:32 (1745621672) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg244-server
Stopping /mnt/lustre-mds1 (opts:) on oleg244-server
18:54:45 (1745621685) shut down
facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server
Failover mds1 to oleg244-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all
pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1
Started lustre-MDT0000
18:55:01 (1745621701) targets are mounted
18:55:01 (1745621701) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg244-server
Stopping /mnt/lustre-mds1 (opts:) on oleg244-server
18:55:24 (1745621724) shut down
facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server
Failover mds1 to oleg244-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all
pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1
Started lustre-MDT0000
18:55:41 (1745621741) targets are mounted
18:55:41 (1745621741) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg244-server
Stopping /mnt/lustre-mds1 (opts:) on oleg244-server
18:56:10 (1745621770) shut down
facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server
Failover mds1 to oleg244-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all
pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1
Started lustre-MDT0000
18:56:27 (1745621787) targets are mounted
18:56:27 (1745621787) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg244-server
Stopping /mnt/lustre-mds1 (opts:) on oleg244-server
18:57:00 (1745621820) shut down
facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server
Failover mds1 to oleg244-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all
pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1
Started lustre-MDT0000
18:57:16 (1745621836) targets are mounted
18:57:16 (1745621836) facet_failover done
writemany returned 0