== recovery-small test 51: failover MDS during recovery == 05:43:15 (1734432195)
fail_loc=0x00001310
Failing mds1 on oleg328-server
Stopping /mnt/lustre-mds1 (opts:) on oleg328-server
05:43:18 (1734432198) shut down
facet: mds1 facet_host: oleg328-server facet_failover_host: oleg328-server
Failover mds1 to oleg328-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1
Started lustre-MDT0000
05:43:32 (1734432212) targets are mounted
05:43:32 (1734432212) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg328-server
Stopping /mnt/lustre-mds1 (opts:) on oleg328-server
05:43:34 (1734432214) shut down
facet: mds1 facet_host: oleg328-server facet_failover_host: oleg328-server
Failover mds1 to oleg328-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1
Started lustre-MDT0000
05:43:48 (1734432228) targets are mounted
05:43:48 (1734432228) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg328-server
Stopping /mnt/lustre-mds1 (opts:) on oleg328-server
05:43:54 (1734432234) shut down
facet: mds1 facet_host: oleg328-server facet_failover_host: oleg328-server
Failover mds1 to oleg328-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1
Started lustre-MDT0000
05:44:08 (1734432248) targets are mounted
05:44:08 (1734432248) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg328-server
Stopping /mnt/lustre-mds1 (opts:) on oleg328-server
05:44:19 (1734432259) shut down
facet: mds1 facet_host: oleg328-server facet_failover_host: oleg328-server
Failover mds1 to oleg328-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1
Started lustre-MDT0000
05:44:33 (1734432273) targets are mounted
05:44:33 (1734432273) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg328-server
Stopping /mnt/lustre-mds1 (opts:) on oleg328-server
05:44:55 (1734432295) shut down
facet: mds1 facet_host: oleg328-server facet_failover_host: oleg328-server
Failover mds1 to oleg328-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1
Started lustre-MDT0000
05:45:08 (1734432308) targets are mounted
05:45:08 (1734432308) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg328-server
Stopping /mnt/lustre-mds1 (opts:) on oleg328-server
05:45:35 (1734432335) shut down
facet: mds1 facet_host: oleg328-server facet_failover_host: oleg328-server
Failover mds1 to oleg328-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1
Started lustre-MDT0000
05:45:48 (1734432348) targets are mounted
05:45:48 (1734432348) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg328-server
Stopping /mnt/lustre-mds1 (opts:) on oleg328-server
05:46:20 (1734432380) shut down
facet: mds1 facet_host: oleg328-server facet_failover_host: oleg328-server
Failover mds1 to oleg328-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1
Started lustre-MDT0000
05:46:33 (1734432393) targets are mounted
05:46:33 (1734432393) facet_failover done
writemany returned 0