		******************************************
		* Overview of Dynamic LNet Configuration *
		******************************************

Original Authors:
=================
Amir Shehata <amir.shehata@intel.com>

Contents
1. Introduction
	1.1 Objectives
	1.3 Motivation
	1.2 Overview
2. YAML Interface
3. DLC API
4. LNet Backend Design
	4.1 router.c
		4.1.1 Enabling/Disabling Routing and Router Pools
		4.1.2 Configuring Router Pools
	4.2 api-ni.c
		4.2.1 Dynamically Adding an NI
		4.2.2 Dynamically Deleting an NI
		4.2.3 Internal Functions

===================
= 1. Introduction =
===================

1.1 Objectives
==============

Dynamic LNet Configuration (DLC) strives to accomplish four objectives:

. Allow changes to some LNet parameters to be dynamic so they can be altered
  without having to stop/start LNet.
. Remove the limits on how many NI's or route's which can be configured.
. Work towards replacing the LNet module parameters with user space scripts
  (i.e. /etc/rc.d and /etc/sysconfig/network-scripts) which operate much more
  like conventional network config systems on Linux.
. Allow sysadmins to use a command line utility for making dynamic changes to
  the LNet configuration.

1.2 Motivation
==============

DLC introduces a new shared library. The DLC C API Library is provided as part
of the standard lustre installation. This library must be released under LGPL
license to facilitate third-party development. This API will serve the
following purposes:

. Provide a funnel for all LNet configurations. This way there isn't multiple
  ways to configure the same thing
. Provide a C API which configures specific parameters. This C API will
  translate into IOCTL calls to the kernel module.
. Provide a YAML interface, which ensures that YAML parsing is done in one
  location.
. Provide more detailed return values from the kernel presented in YAML
  format.
. Provide a set of API calls to parse out the YAML error returns if need be.

1.3 Overview
============

DLC introduces a userspace library which exposes a set of APIs to:

. configure/unconfigure LNet
. add/delete/show routes
. add/delete/show networks
. set router buffers sizes
. enable/disable routing
. import YAML configuration
. export configuration in YAML format
. show peer credit information

DLC API uses IOCTLs to communicate with the kernel. This has the following
advantage:

. No need to do any string parsing in the kernel
. Bypass the 4K limit, since each command will be sent to the kernel
  separately. For example, if there are 100 routes to configure then this
  translates to 100 ioctls to the kernel.
. Provide a more intuitive user facing interface. The user need not know how
  to talk to the kernel via ioctl. This will be achieved in two ways:
	a. The configuration can be presented in YAML syntax, as shown below
	b. A more traditional C API will be exported, Each translates to exactly
	   one ioctl to the kernel module.

=====================
= 2. YAML Interface =
=====================

The DLC API provides functions to configure specific parameters together with
higher level functions which accept a YAML formatted configuration and
configure all parameters described therein.

To aid in parsing and building YAML stings a third party library, libyaml, is
used. lnet/utils/cyaml/cyaml.c is written to interface with the libyaml
library. When parsing a yaml string or file, it builds an internal tree
representation of the YAML string. lnet/utils/cyaml/cyaml.c then provides
functions to extract data from the tree. It also provides functions to build
a YAML tree, which can then be printed textually.

The YAML API is used by the DLC API to parse and extract YAML configuration,
and to build a YAML representation of the success or error status per DLC API.

==============
= 3. DLC API =
==============

Refer to: lnet/utils/lnetconfig/liblnetconfig.h

==========================
= 4. LNet Backend Design =
==========================

4.1 lnet/lnet/router.c
=======================

In router.c the following three functions are used to add, delete and get a
route.

. lnet_add_route() : adds a new route entry.
. lnet_del_route() : removes an existing routing entry.
. lnet_get_route() : retrieves information on an existing route.

These functions will be called from the ioctl handler.

The Router buffer pools will be controlled via the following APIs:

. lnet_rtrpools_enable()  : enable routing and allocate router buffers
. lnet_rtrpools_disable() : disable routing and free router buffers
. lnet_rtrpools_adjust()  : configure the number of router buffer pools

4.1.1 Enabling/Disabling Routing and Router Pools
=================================================

lnet_rtrpools_enable()
lnet_rtrpools_disable()

These functions will explicitly enable and disable rtrpools respectively. If the
rtrpools are being enabled then the default values of the tiny, small and
large buffers are used.

    #define LNET_NRB_TINY        (LNET_NRB_TINY_MIN * 4)
    #define LNET_NRB_SMALL        (LNET_NRB_SMALL_MIN * 4)
    #define LNET_NRB_LARGE        (LNET_NRB_LARGE_MIN * 4)

4.1.2 Configuring Router Pools
==============================

lnet_rtrpools_adjust()

This API will set the tiny, small and large buffers. rtrpools must be enabled
prior to calling this API. If not enabled, this API is a no-op.

Below is the behavior when adjusting the router pools:

. A value of 0 indicates that the default values (listed above) are used.
. Each pool size will be run through the current sizing algorithm to ensure the
  given values are not below the minimum standards. As such, a user could get
  the defaults for all three pools by just passing in "0 0 0" for pool sizes.
. If a pool size is being made bigger, add the new buffers to the pool checking
  if there are any queued messages waiting for a buffer on that pool. If so,
  schedule those messages to be sent with the new buffer.
. If a pool size is being made smaller, change the maximum value for the number
  of buffers on the pool only. When the system returns buffers to the pool, it
  will see there are "excess" buffers on the pool and will discard the buffer
  rather than return it to the pool.

4.2 lnet/lnet/api-ni.c
=======================

The file api-ni.c contains the interface routines for working with network
interfaces. Four new routines will be added to this file to support this
feature.

lnet_dyn_add_ni()     : adds an NI dynamically
lnet_dyn_del_ni()     : deletes an NI dynamically
lnet_startup_lndni()  : starts a single LND.
lnet_shutdown_lndni() : shuts down a single LND.

4.2.1 Dynamically Adding an NI
==============================

lnet_dyn_add_ni()

This function is called to startup a network passed in through userspace as a
string. This function verifies that only one network is passed in.  If more
than one network is passed in, it fails with -EINVAL error number.

Peer timeout, peer credits, and credits are passed in as well, since DLC makes
these tunable parameters configurable per network interface.

This function does the following:

. Parses the network string passed in.
. Initializes a new ping info structure in preparation of adding the Network
  Interface.
. Starts up the NI
. Updates the global ping info.
. If there is any failure in any of these steps, appropriate cleanup action is
  taken.

4.2.2 Dynamically Deleting an NI
================================

lnet_dyn_del_ni()

This function does the following:

. Allocates a new ping info for the number of networks without the one being
  removed.
. Shuts down the NI
. Updates the global ping info appropriately.

4.2.3 Internal Functions
========================

lnet_startup_lndni()

The code was restructured to breakup the function lnet_startup_lndnis() into
two functions: lnet_startup_lndni() and lnet_startup_lndnis(). This way the
new lnet_dyn_add_ni() can call lnet_startup_lndni() to startup one NI, and on
standard startup lnet_startup_lndnis() can start up all NIs configured in the
module parameters, by calling lnet_startup_lndni() repeatedly.

lnet_shutdown_lndni()

This function was introduced to shutdown only one NI. It coexists with the
original lnet_shutdown_lndnis() which shuts down all NIs, which is needed when
LNet is being unconfigured.

lnet_shutdown_lndni() is called from lnet_dyn_del_ni() and from
lnet_startup_lndni() in case there is a failure starting the NI.

