== recovery-small test 50: failover MDS under load ======= 04:46:39 (1787820399) writemany pid 42489 Failing mds1 on oleg108-server Stopping /mnt/lustre-mds1 (opts:) on oleg108-server 04:46:58 (1787820418) shut down facet: mds1 facet_host: oleg108-server facet_failover_host: oleg108-server Failover mds1 to oleg108-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 04:47:32 (1787820452) targets are mounted 04:47:32 (1787820452) facet_failover done oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds1 on oleg108-server Stopping /mnt/lustre-mds1 (opts:) on oleg108-server 04:48:52 (1787820532) shut down facet: mds1 facet_host: oleg108-server facet_failover_host: oleg108-server Failover mds1 to oleg108-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 04:49:26 (1787820566) targets are mounted 04:49:26 (1787820566) facet_failover done oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds1 on oleg108-server Stopping /mnt/lustre-mds1 (opts:) on oleg108-server 04:50:48 (1787820648) shut down facet: mds1 facet_host: oleg108-server facet_failover_host: oleg108-server Failover mds1 to oleg108-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 04:51:14 (1787820674) targets are mounted 04:51:14 (1787820674) facet_failover done oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec writemany returned 0