Building Lustre with Mellanox OFED Support

In order to have a successful build of Lustre with Mellanox OFED support, it’s important to keep in mind some of the basics of a Lustre installation:

• Lustre servers must use the Lustre patched kernel
• Lustre clients can use the standard kernel (Lustre patched kernel is optional)
• Use the latest manufacturer-provided IB drivers on servers and clients for the best performance
• The build environment must match the runtime environment in terms of: kernel version, Linux distribution version, MOFED version,
• Successfully built RPMs does not equate to fully functional RPMS.  Make sure that MOFED support is working after installing the packages
• The build procedure must be repeated when upgrading the kernel, MOFED, or Lustre version.

Assuming the Lustre clients use a different version of the kernel than the Lustre servers, a different build environment is required for the build of packages targeting clients, and packages targeting servers.

The simpler case is the build of packages targeting Lustre clients which can utilize a Linux distribution officially supported by Mellanox.  For example, to build Lustre packages with MOFED support for a supported Linux distribution, simply:

1. Download the MOFED iso, mount, and proceed with mlnxofedinstall.sh (located in the mount directory)
2. Download the Lustre source repo (git clone git://git.hpdd.intel.com/fs/lustre-release.git) and checkout a release tag.  See here for a list of release tags.
3. Change directories to “lustre-release/”
4. Execute “sh ./autogen.sh” to generate a “configure” script
5. Proceed with “./configure –with-o2ib=/usr/src/ofa_kernel/default” to include MOFED support
6. The last step is to build the rpms with “make rpms”

Building packages targeting Lustre servers is more complex due to the facts that:

• MOFED packages depend on the custom target kernel
• Lustre server packages depend on the custom MOFED packages
• Intel provides pre-built e2fsprogs rpms which will be supplemental to the custom-built Lustre rpms
• Intel provides pre-built Lustre rpms which will not be compatible with the custom-built Lustre rpms.  NOTE that the kernel packages which must be used in the build and runtime environment are also contained in this same file archive.

A fail-safe procedure for making sure that the correct symbols get used in the build of Lustre server packages is:

1. Download and install the custom Linux kernel and headers. Reboot the build environment and verify that “lustre” is in the name of the kernel by checking the output of “uname -r”
2. Download the MOFED iso and mount
3. Add MOFED support for the Lustre kernel.  Note that this can be done once and only once by saving the created tarball (see the output of mlnxofedinstall) or by specifying the flag and path for a custom ISO.  Change directories to the mount directory and run “./mlnxofedinstall –add-kernel-support”
4. Download the Lustre source repo (git clone git://git.hpdd.intel.com/fs/lustre-release.git) and checkout a release tag.  See here for a list of release tags.   Change directories to “lustre-release/”.
5. Execute “sh ./autogen.sh” to generate a “configure” script
6. Proceed with “./configure –with-o2ib=/usr/src/ofa_kernel/default” to include custom MOFED support
7. The last step is to build the rpms with “make rpms”

Building Lustre rpms will result in several custom RPMs being created in the current working directory.  When using any of the Lustre-related RPMs from Intel, avoid using the Intel-provided version if you have a custom version as a result of your build.

Verifying results:  after the RPMs are built, it’s important to ensure that the RPMs are fully functional when installed on a target Lustre server.

On a Lustre server, follow the same order of installation of components as used in the build for fail-safe resolution of dependencies:

1. Install the same Lustre-patched kernel as used during build. Reboot and verify “lustre” is included in the output of “uname -r”
2. Install MOFED by utilizing the tarball or iso created during build.  Untar or mount the MOFED bundle.  Run “mlnxofedinstall” (located in the untarred/mount directory).
3. Install the custom Linux server rpms
4. Specify the ib0 device as the Lustre traffic carrier.  Add “options lnet networks=”o2ib1(ib0)” to /etc/modprobe.d/lustre.conf
5. Load the Lustre module with “modprobe lustre”
6. Verify successful module load with “echo $?” or “lsmod | grep lustre”
7. Verify “o2ib” is in the output of “lctl list_nids”

For additional details, or advanced procedures- e.g. building a Linux kernel with inclusion of custom patches- see the references below.

References:

Pre-built CentOS 7 Lustre packages and kernel

Lustre filesystem configuration

Installing MOFED for Lustre

Lustre build guide