== recovery-small test 51: failover MDS during recovery == 05:05:19 (1731751519)
fail_loc=0x00001310
Failing mds1 on oleg303-server
Stopping /mnt/lustre-mds1 (opts:) on oleg303-server
05:05:22 (1731751522) shut down
Failover mds1 to oleg303-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1
Started lustre-MDT0000
05:05:35 (1731751535) targets are mounted
05:05:35 (1731751535) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg303-server
Stopping /mnt/lustre-mds1 (opts:) on oleg303-server
05:05:37 (1731751537) shut down
Failover mds1 to oleg303-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1
Started lustre-MDT0000
05:05:51 (1731751551) targets are mounted
05:05:51 (1731751551) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg303-server
Stopping /mnt/lustre-mds1 (opts:) on oleg303-server
05:05:57 (1731751557) shut down
Failover mds1 to oleg303-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1
Started lustre-MDT0000
05:06:11 (1731751571) targets are mounted
05:06:11 (1731751571) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg303-server
Stopping /mnt/lustre-mds1 (opts:) on oleg303-server
05:06:22 (1731751582) shut down
Failover mds1 to oleg303-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1
Started lustre-MDT0000
05:06:35 (1731751595) targets are mounted
05:06:35 (1731751595) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg303-server
Stopping /mnt/lustre-mds1 (opts:) on oleg303-server
05:06:56 (1731751616) shut down
Failover mds1 to oleg303-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1
Started lustre-MDT0000
05:07:10 (1731751630) targets are mounted
05:07:10 (1731751630) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg303-server
Stopping /mnt/lustre-mds1 (opts:) on oleg303-server
05:07:37 (1731751657) shut down
Failover mds1 to oleg303-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1
Started lustre-MDT0000
05:07:50 (1731751670) targets are mounted
05:07:50 (1731751670) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg303-server
Stopping /mnt/lustre-mds1 (opts:) on oleg303-server
05:08:21 (1731751701) shut down
Failover mds1 to oleg303-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all
pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1
Started lustre-MDT0000
05:08:34 (1731751714) targets are mounted
05:08:34 (1731751714) facet_failover done
writemany returned 0