== recovery-small test 51: failover MDS during recovery == 02:53:39 (1727765619)
fail_loc=0x00001310
Failing mds1 on oleg346-server
Stopping /mnt/lustre-mds1 (opts:) on oleg346-server
02:53:45 (1727765625) shut down
Failover mds1 to oleg346-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all
pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1
Started lustre-MDT0000
02:54:00 (1727765640) targets are mounted
02:54:00 (1727765640) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg346-server
Stopping /mnt/lustre-mds1 (opts:) on oleg346-server
02:54:03 (1727765643) shut down
Failover mds1 to oleg346-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all
pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1
Started lustre-MDT0000
02:54:18 (1727765658) targets are mounted
02:54:18 (1727765658) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg346-server
Stopping /mnt/lustre-mds1 (opts:) on oleg346-server
02:54:26 (1727765666) shut down
Failover mds1 to oleg346-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all
pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1
Started lustre-MDT0000
02:54:42 (1727765682) targets are mounted
02:54:42 (1727765682) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg346-server
Stopping /mnt/lustre-mds1 (opts:) on oleg346-server
02:54:55 (1727765695) shut down
Failover mds1 to oleg346-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all
pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1
Started lustre-MDT0000
02:55:11 (1727765711) targets are mounted
02:55:11 (1727765711) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg346-server
Stopping /mnt/lustre-mds1 (opts:) on oleg346-server
02:55:34 (1727765734) shut down
Failover mds1 to oleg346-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all
pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1
Started lustre-MDT0000
02:55:51 (1727765751) targets are mounted
02:55:51 (1727765751) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg346-server
Stopping /mnt/lustre-mds1 (opts:) on oleg346-server
02:56:26 (1727765786) shut down
Failover mds1 to oleg346-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all
pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1
Started lustre-MDT0000
02:56:41 (1727765801) targets are mounted
02:56:41 (1727765801) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg346-server
Stopping /mnt/lustre-mds1 (opts:) on oleg346-server
02:57:14 (1727765834) shut down
Failover mds1 to oleg346-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all
pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1
Started lustre-MDT0000
02:57:30 (1727765850) targets are mounted
02:57:30 (1727765850) facet_failover done
writemany returned 0