== recovery-small test 50: failover MDS under load ======= 02:07:23 (1790921243) writemany pid 42609 Failing mds1 on oleg631-server Stopping /mnt/lustre-mds1 (opts:) on oleg631-server 02:07:40 (1790921260) shut down facet: mds1 facet_host: oleg631-server facet_failover_host: oleg631-server Failover mds1 to oleg631-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg631-server: oleg631-server.virtnet: executing set_default_debug -1 all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-MDT0000 02:08:04 (1790921284) targets are mounted 02:08:04 (1790921284) facet_failover done oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds1 on oleg631-server Stopping /mnt/lustre-mds1 (opts:) on oleg631-server 02:09:22 (1790921362) shut down facet: mds1 facet_host: oleg631-server facet_failover_host: oleg631-server Failover mds1 to oleg631-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg631-server: oleg631-server.virtnet: executing set_default_debug -1 all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-MDT0000 02:09:54 (1790921394) targets are mounted 02:09:54 (1790921394) facet_failover done oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds1 on oleg631-server Stopping /mnt/lustre-mds1 (opts:) on oleg631-server 02:11:13 (1790921473) shut down facet: mds1 facet_host: oleg631-server facet_failover_host: oleg631-server Failover mds1 to oleg631-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg631-server: oleg631-server.virtnet: executing set_default_debug -1 all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-MDT0000 02:11:40 (1790921500) targets are mounted 02:11:40 (1790921500) facet_failover done oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec writemany returned 0