== recovery-small test 51: failover MDS during recovery == 04:35:39 (1743496539)
fail_loc=0x00001310
Failing mds1 on oleg623-server
Stopping /mnt/lustre-mds1 (opts:) on oleg623-server
04:35:57 (1743496557) shut down
facet: mds1 facet_host: oleg623-server facet_failover_host: oleg623-server
Failover mds1 to oleg623-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg623-server: oleg623-server.virtnet: executing set_default_debug -1 all
pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1
Started lustre-MDT0000
04:36:39 (1743496599) targets are mounted
04:36:39 (1743496599) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg623-server
Stopping /mnt/lustre-mds1 (opts:) on oleg623-server
04:36:55 (1743496615) shut down
facet: mds1 facet_host: oleg623-server facet_failover_host: oleg623-server
Failover mds1 to oleg623-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg623-server: oleg623-server.virtnet: executing set_default_debug -1 all
pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1
Started lustre-MDT0000
04:37:34 (1743496654) targets are mounted
04:37:34 (1743496654) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg623-server
Stopping /mnt/lustre-mds1 (opts:) on oleg623-server
04:37:54 (1743496674) shut down
facet: mds1 facet_host: oleg623-server facet_failover_host: oleg623-server
Failover mds1 to oleg623-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg623-server: oleg623-server.virtnet: executing set_default_debug -1 all
pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1
Started lustre-MDT0000
04:38:41 (1743496721) targets are mounted
04:38:41 (1743496721) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg623-server
Stopping /mnt/lustre-mds1 (opts:) on oleg623-server
04:39:10 (1743496750) shut down
facet: mds1 facet_host: oleg623-server facet_failover_host: oleg623-server
Failover mds1 to oleg623-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg623-server: oleg623-server.virtnet: executing set_default_debug -1 all
pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1
Started lustre-MDT0000
04:40:02 (1743496802) targets are mounted
04:40:02 (1743496802) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg623-server
Stopping /mnt/lustre-mds1 (opts:) on oleg623-server
04:40:40 (1743496840) shut down
facet: mds1 facet_host: oleg623-server facet_failover_host: oleg623-server
Failover mds1 to oleg623-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg623-server: oleg623-server.virtnet: executing set_default_debug -1 all
pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1
Started lustre-MDT0000
04:41:24 (1743496884) targets are mounted
04:41:24 (1743496884) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg623-server
Stopping /mnt/lustre-mds1 (opts:) on oleg623-server
04:42:09 (1743496929) shut down
facet: mds1 facet_host: oleg623-server facet_failover_host: oleg623-server
Failover mds1 to oleg623-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg623-server: oleg623-server.virtnet: executing set_default_debug -1 all
pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1
Started lustre-MDT0000
04:43:00 (1743496980) targets are mounted
04:43:00 (1743496980) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg623-server
Stopping /mnt/lustre-mds1 (opts:) on oleg623-server
04:43:47 (1743497027) shut down
facet: mds1 facet_host: oleg623-server facet_failover_host: oleg623-server
Failover mds1 to oleg623-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg623-server: oleg623-server.virtnet: executing set_default_debug -1 all
pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1
Started lustre-MDT0000
04:44:37 (1743497077) targets are mounted
04:44:37 (1743497077) facet_failover done
writemany returned 0