== recovery-small test 51: failover MDS during recovery == 08:18:10 (1746793090)
fail_loc=0x00001310
Failing mds1 on oleg301-server
Stopping /mnt/lustre-mds1 (opts:) on oleg301-server
08:18:13 (1746793093) shut down
facet: mds1 facet_host: oleg301-server facet_failover_host: oleg301-server
Failover mds1 to oleg301-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all
pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1
Started lustre-MDT0000
08:18:25 (1746793105) targets are mounted
08:18:25 (1746793105) facet_failover done
will failover at 1 5 10 20 25 30
test_51: failover in 1 sec
Failing mds1 on oleg301-server
Stopping /mnt/lustre-mds1 (opts:) on oleg301-server
08:18:28 (1746793108) shut down
facet: mds1 facet_host: oleg301-server facet_failover_host: oleg301-server
Failover mds1 to oleg301-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all
pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1
Started lustre-MDT0000
08:18:40 (1746793120) targets are mounted
08:18:40 (1746793120) facet_failover done
test_51: failover in 5 sec
Failing mds1 on oleg301-server
Stopping /mnt/lustre-mds1 (opts:) on oleg301-server
08:18:47 (1746793127) shut down
facet: mds1 facet_host: oleg301-server facet_failover_host: oleg301-server
Failover mds1 to oleg301-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all
pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1
Started lustre-MDT0000
08:19:01 (1746793141) targets are mounted
08:19:01 (1746793141) facet_failover done
test_51: failover in 10 sec
Failing mds1 on oleg301-server
Stopping /mnt/lustre-mds1 (opts:) on oleg301-server
08:19:12 (1746793152) shut down
facet: mds1 facet_host: oleg301-server facet_failover_host: oleg301-server
Failover mds1 to oleg301-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all
pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1
Started lustre-MDT0000
08:19:25 (1746793165) targets are mounted
08:19:25 (1746793165) facet_failover done
test_51: failover in 20 sec
Failing mds1 on oleg301-server
Stopping /mnt/lustre-mds1 (opts:) on oleg301-server
08:19:47 (1746793187) shut down
facet: mds1 facet_host: oleg301-server facet_failover_host: oleg301-server
Failover mds1 to oleg301-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all
pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1
Started lustre-MDT0000
08:20:00 (1746793200) targets are mounted
08:20:00 (1746793200) facet_failover done
test_51: failover in 25 sec
Failing mds1 on oleg301-server
Stopping /mnt/lustre-mds1 (opts:) on oleg301-server
08:20:27 (1746793227) shut down
facet: mds1 facet_host: oleg301-server facet_failover_host: oleg301-server
Failover mds1 to oleg301-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all
pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1
Started lustre-MDT0000
08:20:41 (1746793241) targets are mounted
08:20:41 (1746793241) facet_failover done
test_51: failover in 30 sec
Failing mds1 on oleg301-server
Stopping /mnt/lustre-mds1 (opts:) on oleg301-server
08:21:13 (1746793273) shut down
facet: mds1 facet_host: oleg301-server facet_failover_host: oleg301-server
Failover mds1 to oleg301-server
mount facets: mds1
Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all
pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1
Started lustre-MDT0000
08:21:27 (1746793287) targets are mounted
08:21:27 (1746793287) facet_failover done
writemany returned 0