== recovery-small test 50: failover MDS under load ======= 20:10:02 (1788307802) writemany pid 42456 Failing mds1 on oleg658-server Stopping /mnt/lustre-mds1 (opts:) on oleg658-server 20:10:20 (1788307820) shut down facet: mds1 facet_host: oleg658-server facet_failover_host: oleg658-server Failover mds1 to oleg658-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg658-server: oleg658-server.virtnet: executing set_default_debug -1 all pdsh@oleg658-client: oleg658-server: ssh exited with exit code 1 Started lustre-MDT0000 20:10:45 (1788307845) targets are mounted 20:10:45 (1788307845) facet_failover done oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds1 on oleg658-server Stopping /mnt/lustre-mds1 (opts:) on oleg658-server 20:12:07 (1788307927) shut down facet: mds1 facet_host: oleg658-server facet_failover_host: oleg658-server Failover mds1 to oleg658-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg658-server: oleg658-server.virtnet: executing set_default_debug -1 all pdsh@oleg658-client: oleg658-server: ssh exited with exit code 1 Started lustre-MDT0000 20:12:31 (1788307951) targets are mounted 20:12:31 (1788307951) facet_failover done oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Failing mds1 on oleg658-server Stopping /mnt/lustre-mds1 (opts:) on oleg658-server 20:13:51 (1788308031) shut down facet: mds1 facet_host: oleg658-server facet_failover_host: oleg658-server Failover mds1 to oleg658-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg658-server: oleg658-server.virtnet: executing set_default_debug -1 all pdsh@oleg658-client: oleg658-server: ssh exited with exit code 1 Started lustre-MDT0000 20:14:14 (1788308054) targets are mounted 20:14:14 (1788308054) facet_failover done oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec writemany returned 0