== recovery-small test 51: failover MDS during recovery == 12:11:44 (1724775104)
fail_loc=0x00001310
Failing mds1 on oleg304-server
Stopping /mnt/lustre-mds1 (opts:) on oleg304-server
12:11:50 (1724775110) shut down
Failover mds1 to oleg304-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all
pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1
Started lustre-MDT0000
12:12:06 (1724775126) targets are mounted
12:12:06 (1724775126) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg304-server
Stopping /mnt/lustre-mds1 (opts:) on oleg304-server
12:12:09 (1724775129) shut down
Failover mds1 to oleg304-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all
pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1
Started lustre-MDT0000
12:12:22 (1724775142) targets are mounted
12:12:22 (1724775142) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg304-server
Stopping /mnt/lustre-mds1 (opts:) on oleg304-server
12:12:29 (1724775149) shut down
Failover mds1 to oleg304-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all
pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1
Started lustre-MDT0000
12:12:44 (1724775164) targets are mounted
12:12:44 (1724775164) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg304-server
Stopping /mnt/lustre-mds1 (opts:) on oleg304-server
12:12:56 (1724775176) shut down
Failover mds1 to oleg304-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all
pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1
Started lustre-MDT0000
12:13:10 (1724775190) targets are mounted
12:13:10 (1724775190) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg304-server
Stopping /mnt/lustre-mds1 (opts:) on oleg304-server
12:13:31 (1724775211) shut down
Failover mds1 to oleg304-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all
pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1
Started lustre-MDT0000
12:13:44 (1724775224) targets are mounted
12:13:44 (1724775224) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg304-server
Stopping /mnt/lustre-mds1 (opts:) on oleg304-server
12:14:11 (1724775251) shut down
Failover mds1 to oleg304-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all
pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1
Started lustre-MDT0000
12:14:24 (1724775264) targets are mounted
12:14:24 (1724775264) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg304-server
Stopping /mnt/lustre-mds1 (opts:) on oleg304-server
12:14:55 (1724775295) shut down
Failover mds1 to oleg304-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all
pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1
Started lustre-MDT0000
12:15:08 (1724775308) targets are mounted
12:15:08 (1724775308) facet_failover done
writemany returned 0