== recovery-small test 51: failover MDS during recovery == 12:39:30 (1761237570)
fail_loc=0x00001310
Failing mds1 on oleg339-server
Stopping /mnt/lustre-mds1 (opts:) on oleg339-server
12:39:38 (1761237578) shut down
facet: mds1 facet_host: oleg339-server facet_failover_host: oleg339-server
Failover mds1 to oleg339-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all
pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1
Started lustre-MDT0000
12:40:00 (1761237600) targets are mounted
12:40:00 (1761237600) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg339-server
Stopping /mnt/lustre-mds1 (opts:) on oleg339-server
12:40:08 (1761237608) shut down
facet: mds1 facet_host: oleg339-server facet_failover_host: oleg339-server
Failover mds1 to oleg339-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all
pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1
Started lustre-MDT0000
12:40:40 (1761237640) targets are mounted
12:40:40 (1761237640) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg339-server
Stopping /mnt/lustre-mds1 (opts:) on oleg339-server
12:40:51 (1761237651) shut down
facet: mds1 facet_host: oleg339-server facet_failover_host: oleg339-server
Failover mds1 to oleg339-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all
pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1
Started lustre-MDT0000
12:41:21 (1761237681) targets are mounted
12:41:21 (1761237681) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg339-server
Stopping /mnt/lustre-mds1 (opts:) on oleg339-server
12:41:38 (1761237698) shut down
facet: mds1 facet_host: oleg339-server facet_failover_host: oleg339-server
Failover mds1 to oleg339-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all
pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1
Started lustre-MDT0000
12:42:03 (1761237723) targets are mounted
12:42:03 (1761237723) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg339-server
Stopping /mnt/lustre-mds1 (opts:) on oleg339-server
12:42:31 (1761237751) shut down
facet: mds1 facet_host: oleg339-server facet_failover_host: oleg339-server
Failover mds1 to oleg339-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all
pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1
Started lustre-MDT0000
12:42:52 (1761237772) targets are mounted
12:42:53 (1761237773) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg339-server
Stopping /mnt/lustre-mds1 (opts:) on oleg339-server
12:43:26 (1761237806) shut down
facet: mds1 facet_host: oleg339-server facet_failover_host: oleg339-server
Failover mds1 to oleg339-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all
pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1
Started lustre-MDT0000
12:43:46 (1761237826) targets are mounted
12:43:46 (1761237826) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg339-server
Stopping /mnt/lustre-mds1 (opts:) on oleg339-server
12:44:24 (1761237864) shut down
facet: mds1 facet_host: oleg339-server facet_failover_host: oleg339-server
Failover mds1 to oleg339-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all
pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1
Started lustre-MDT0000
12:44:53 (1761237893) targets are mounted
12:44:53 (1761237893) facet_failover done
writemany returned 0