== recovery-small test 51: failover MDS during recovery == 13:15:11 (1750180511)
fail_loc=0x00001310
Failing mds1 on oleg143-server
Stopping /mnt/lustre-mds1 (opts:) on oleg143-server
13:15:16 (1750180516) shut down
facet: mds1 facet_host: oleg143-server facet_failover_host: oleg143-server
Failover mds1 to oleg143-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all
pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1
Started lustre-MDT0000
13:15:30 (1750180530) targets are mounted
13:15:30 (1750180530) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg143-server
Stopping /mnt/lustre-mds1 (opts:) on oleg143-server
13:15:33 (1750180533) shut down
facet: mds1 facet_host: oleg143-server facet_failover_host: oleg143-server
Failover mds1 to oleg143-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all
pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1
Started lustre-MDT0000
13:15:48 (1750180548) targets are mounted
13:15:48 (1750180548) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg143-server
Stopping /mnt/lustre-mds1 (opts:) on oleg143-server
13:15:56 (1750180556) shut down
facet: mds1 facet_host: oleg143-server facet_failover_host: oleg143-server
Failover mds1 to oleg143-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all
pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1
Started lustre-MDT0000
13:16:10 (1750180570) targets are mounted
13:16:10 (1750180570) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg143-server
Stopping /mnt/lustre-mds1 (opts:) on oleg143-server
13:16:24 (1750180584) shut down
facet: mds1 facet_host: oleg143-server facet_failover_host: oleg143-server
Failover mds1 to oleg143-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all
pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1
Started lustre-MDT0000
13:16:39 (1750180599) targets are mounted
13:16:39 (1750180599) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg143-server
Stopping /mnt/lustre-mds1 (opts:) on oleg143-server
13:17:02 (1750180622) shut down
facet: mds1 facet_host: oleg143-server facet_failover_host: oleg143-server
Failover mds1 to oleg143-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all
pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1
Started lustre-MDT0000
13:17:17 (1750180637) targets are mounted
13:17:17 (1750180637) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg143-server
Stopping /mnt/lustre-mds1 (opts:) on oleg143-server
13:17:45 (1750180665) shut down
facet: mds1 facet_host: oleg143-server facet_failover_host: oleg143-server
Failover mds1 to oleg143-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all
pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1
Started lustre-MDT0000
13:18:00 (1750180680) targets are mounted
13:18:00 (1750180680) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg143-server
Stopping /mnt/lustre-mds1 (opts:) on oleg143-server
13:18:34 (1750180714) shut down
facet: mds1 facet_host: oleg143-server facet_failover_host: oleg143-server
Failover mds1 to oleg143-server
mount facets: mds1
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all
pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1
Started lustre-MDT0000
13:18:49 (1750180729) targets are mounted
13:18:49 (1750180729) facet_failover done
writemany returned 0