== recovery-small test 51: failover MDS during recovery == 22:38:52 (1746067132)
fail_loc=0x00001310
Failing mds1 on oleg204-server
Stopping /mnt/lustre-mds1 (opts:) on oleg204-server
22:38:56 (1746067136) shut down
facet: mds1 facet_host: oleg204-server facet_failover_host: oleg204-server
Failover mds1 to oleg204-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all
pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1
Started lustre-MDT0000
22:39:11 (1746067151) targets are mounted
22:39:11 (1746067151) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg204-server
Stopping /mnt/lustre-mds1 (opts:) on oleg204-server
22:39:15 (1746067155) shut down
facet: mds1 facet_host: oleg204-server facet_failover_host: oleg204-server
Failover mds1 to oleg204-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all
pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1
Started lustre-MDT0000
22:39:32 (1746067172) targets are mounted
22:39:32 (1746067172) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg204-server
Stopping /mnt/lustre-mds1 (opts:) on oleg204-server
22:39:40 (1746067180) shut down
facet: mds1 facet_host: oleg204-server facet_failover_host: oleg204-server
Failover mds1 to oleg204-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all
pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1
Started lustre-MDT0000
22:39:55 (1746067195) targets are mounted
22:39:55 (1746067195) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg204-server
Stopping /mnt/lustre-mds1 (opts:) on oleg204-server
22:40:08 (1746067208) shut down
facet: mds1 facet_host: oleg204-server facet_failover_host: oleg204-server
Failover mds1 to oleg204-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all
pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1
Started lustre-MDT0000
22:40:24 (1746067224) targets are mounted
22:40:24 (1746067224) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg204-server
Stopping /mnt/lustre-mds1 (opts:) on oleg204-server
22:40:47 (1746067247) shut down
facet: mds1 facet_host: oleg204-server facet_failover_host: oleg204-server
Failover mds1 to oleg204-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all
pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1
Started lustre-MDT0000
22:41:01 (1746067261) targets are mounted
22:41:01 (1746067261) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg204-server
Stopping /mnt/lustre-mds1 (opts:) on oleg204-server
22:41:29 (1746067289) shut down
facet: mds1 facet_host: oleg204-server facet_failover_host: oleg204-server
Failover mds1 to oleg204-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all
pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1
Started lustre-MDT0000
22:41:44 (1746067304) targets are mounted
22:41:44 (1746067304) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg204-server
Stopping /mnt/lustre-mds1 (opts:) on oleg204-server
22:42:17 (1746067337) shut down
facet: mds1 facet_host: oleg204-server facet_failover_host: oleg204-server
Failover mds1 to oleg204-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all
pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1
Started lustre-MDT0000
22:42:31 (1746067351) targets are mounted
22:42:31 (1746067351) facet_failover done
writemany returned 0