== recovery-small test 51: failover MDS during recovery == 06:45:18 (1733399118)
fail_loc=0x00001310
Failing mds1 on oleg129-server
Stopping /mnt/lustre-mds1 (opts:) on oleg129-server
06:45:20 (1733399120) shut down
Failover mds1 to oleg129-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all
pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1
Started lustre-MDT0000
06:45:33 (1733399133) targets are mounted
06:45:33 (1733399133) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg129-server
Stopping /mnt/lustre-mds1 (opts:) on oleg129-server
06:45:36 (1733399136) shut down
Failover mds1 to oleg129-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all
pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1
Started lustre-MDT0000
06:45:49 (1733399149) targets are mounted
06:45:49 (1733399149) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg129-server
Stopping /mnt/lustre-mds1 (opts:) on oleg129-server
06:45:56 (1733399156) shut down
Failover mds1 to oleg129-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all
pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1
Started lustre-MDT0000
06:46:09 (1733399169) targets are mounted
06:46:09 (1733399169) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg129-server
Stopping /mnt/lustre-mds1 (opts:) on oleg129-server
06:46:20 (1733399180) shut down
Failover mds1 to oleg129-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all
pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1
Started lustre-MDT0000
06:46:33 (1733399193) targets are mounted
06:46:33 (1733399193) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg129-server
Stopping /mnt/lustre-mds1 (opts:) on oleg129-server
06:46:55 (1733399215) shut down
Failover mds1 to oleg129-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all
pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1
Started lustre-MDT0000
06:47:08 (1733399228) targets are mounted
06:47:08 (1733399228) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg129-server
Stopping /mnt/lustre-mds1 (opts:) on oleg129-server
06:47:34 (1733399254) shut down
Failover mds1 to oleg129-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all
pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1
Started lustre-MDT0000
06:47:47 (1733399267) targets are mounted
06:47:47 (1733399267) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg129-server
Stopping /mnt/lustre-mds1 (opts:) on oleg129-server
06:48:19 (1733399299) shut down
Failover mds1 to oleg129-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all
pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1
Started lustre-MDT0000
06:48:31 (1733399311) targets are mounted
06:48:31 (1733399311) facet_failover done
writemany returned 0