== recovery-small test 51: failover MDS during recovery == 00:58:16 (1708840696)
fail_loc=0x00001310
Failing mds1 on oleg113-server
Stopping /mnt/lustre-mds1 (opts:) on oleg113-server
00:58:19 (1708840699) shut down
Failover mds1 to oleg113-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all
pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1
Started lustre-MDT0000
00:58:33 (1708840713) targets are mounted
00:58:33 (1708840713) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg113-server
Stopping /mnt/lustre-mds1 (opts:) on oleg113-server
00:58:36 (1708840716) shut down
Failover mds1 to oleg113-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all
pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1
Started lustre-MDT0000
00:58:51 (1708840731) targets are mounted
00:58:51 (1708840731) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg113-server
Stopping /mnt/lustre-mds1 (opts:) on oleg113-server
00:58:58 (1708840738) shut down
Failover mds1 to oleg113-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all
pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1
Started lustre-MDT0000
00:59:12 (1708840752) targets are mounted
00:59:12 (1708840752) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg113-server
Stopping /mnt/lustre-mds1 (opts:) on oleg113-server
00:59:24 (1708840764) shut down
Failover mds1 to oleg113-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all
pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1
Started lustre-MDT0000
00:59:38 (1708840778) targets are mounted
00:59:38 (1708840778) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg113-server
Stopping /mnt/lustre-mds1 (opts:) on oleg113-server
00:59:59 (1708840799) shut down
Failover mds1 to oleg113-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all
pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1
Started lustre-MDT0000
01:00:14 (1708840814) targets are mounted
01:00:14 (1708840814) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg113-server
Stopping /mnt/lustre-mds1 (opts:) on oleg113-server
01:00:40 (1708840840) shut down
Failover mds1 to oleg113-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all
pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1
Started lustre-MDT0000
01:00:55 (1708840855) targets are mounted
01:00:55 (1708840855) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg113-server
Stopping /mnt/lustre-mds1 (opts:) on oleg113-server
01:01:26 (1708840886) shut down
Failover mds1 to oleg113-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all
pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1
Started lustre-MDT0000
01:01:39 (1708840899) targets are mounted
01:01:39 (1708840899) facet_failover done
writemany returned 0