== recovery-small test 51: failover MDS during recovery == 20:30:05 (1767317405)
fail_loc=0x00001310
Failing mds1 on oleg359-server
Stopping /mnt/lustre-mds1 (opts:) on oleg359-server
20:30:16 (1767317416) shut down
facet: mds1 facet_host: oleg359-server facet_failover_host: oleg359-server
Failover mds1 to oleg359-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all
pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1
Started lustre-MDT0000
20:31:03 (1767317463) targets are mounted
20:31:03 (1767317463) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg359-server
Stopping /mnt/lustre-mds1 (opts:) on oleg359-server
20:31:12 (1767317472) shut down
facet: mds1 facet_host: oleg359-server facet_failover_host: oleg359-server
Failover mds1 to oleg359-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all
pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1
Started lustre-MDT0000
20:31:47 (1767317507) targets are mounted
20:31:47 (1767317507) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg359-server
Stopping /mnt/lustre-mds1 (opts:) on oleg359-server
20:32:02 (1767317522) shut down
facet: mds1 facet_host: oleg359-server facet_failover_host: oleg359-server
Failover mds1 to oleg359-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all
pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1
Started lustre-MDT0000
20:32:48 (1767317568) targets are mounted
20:32:48 (1767317568) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg359-server
Stopping /mnt/lustre-mds1 (opts:) on oleg359-server
20:33:08 (1767317588) shut down
facet: mds1 facet_host: oleg359-server facet_failover_host: oleg359-server
Failover mds1 to oleg359-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all
pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1
Started lustre-MDT0000
20:33:43 (1767317623) targets are mounted
20:33:43 (1767317623) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg359-server
Stopping /mnt/lustre-mds1 (opts:) on oleg359-server
20:34:12 (1767317652) shut down
facet: mds1 facet_host: oleg359-server facet_failover_host: oleg359-server
Failover mds1 to oleg359-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all
pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1
Started lustre-MDT0000
20:34:44 (1767317684) targets are mounted
20:34:44 (1767317684) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg359-server
Stopping /mnt/lustre-mds1 (opts:) on oleg359-server
20:35:17 (1767317717) shut down
facet: mds1 facet_host: oleg359-server facet_failover_host: oleg359-server
Failover mds1 to oleg359-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all
pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1
Started lustre-MDT0000
20:35:50 (1767317750) targets are mounted
20:35:50 (1767317750) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg359-server
Stopping /mnt/lustre-mds1 (opts:) on oleg359-server
20:36:29 (1767317789) shut down
facet: mds1 facet_host: oleg359-server facet_failover_host: oleg359-server
Failover mds1 to oleg359-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all
pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1
Started lustre-MDT0000
20:37:02 (1767317822) targets are mounted
20:37:02 (1767317822) facet_failover done
writemany returned 0