Skip to main content
NetApp artificial intelligence solutions

Deploy the Lustre HA Cluster and Clients

Contributors netapp-jsnyder
PDFs

Prepare the server operating system, install NVIDIA DOCA-OFED, deploy the Lustre HA cluster, and configure Lustre clients by using NetApp Ansible playbooks.

Before you begin, complete and secure the inventory described in Customize the Lustre Ansible inventory.

Deploy the Lustre HA cluster

Steps

  1. From the building_blocks/playbooks directory in your working copy, prepare the operating system on the Lustre servers. Replace <inventory_path> with the path to your customized inventory directory:

    ansible-playbook -i <inventory_path>/lustre_inventory.yml deploy_lustre_os/deploy_lustre_os_playbook.yml

  2. Install NVIDIA DOCA-OFED on each Lustre server. Build the kernel modules against the kernel that the previous step installed. Follow the procedure for your kernel in NVIDIA DOCA-Host installation and upgrade. The following example shows a DOCA-OFED installation on RHEL or Rocky Linux.

    1. Install the DOCA host repository package and refresh the package cache. Replace <doca_host_package> with the DOCA host package for your operating system and architecture:

      dnf install -y wget tar
wget https://www.mellanox.com/downloads/DOCA/<doca_host_package>.rpm
rpm -i <doca_host_package>.rpm
dnf clean all
dnf makecache

    2. Build the DOCA kernel modules for the running kernel. The script prints the path of the package it creates:

      dnf install -y doca-extra
/opt/mellanox/doca/tools/doca-kernel-support

    3. Install the kernel module repository package that the script created, then refresh the package cache. Replace <doca_kernel_repo> with the path from the previous step:

      rpm -Uvh <doca_kernel_repo>.rpm
dnf makecache

    4. Install the DOCA-OFED user space, kernel, and NVMe packages. Replace <kernel_version> with the running kernel version:

      dnf install -y doca-ofed-userspace
dnf install -y --disablerepo=doca doca-kernel-<kernel_version>
dnf install -y kmod-mlnx-nvme

  3. Run the main deployment playbook:

    ansible-playbook -i <inventory_path>/lustre_inventory.yml lustre_playbook.yml
    Note Adjust --forks for the size of the deployment to control how many hosts Ansible configures in parallel. For example, add --forks 20 for a larger cluster.

The playbook provisions E-Series volume groups or DDP pools and volumes, configures NVMe-oF host connectivity, formats and mounts the Lustre targets, configures LNet, applies performance tuning, and configures the Pacemaker HA resources.

Note A deployment with one building block forms a two-node Pacemaker cluster. Node A is given an extra vote in the case of failover. To achieve quorum in a two-node cluster, consider configuring a qdevice. When a deployment contains multiple building blocks, each building block contributes a two-node server pair to the larger Pacemaker cluster, up to the documented cluster limit. Review the fencing and quorum configuration in the HA administration user guide so the cluster can safely recover targets during a node failure.

Deploy Lustre clients

Deploy Lustre clients on any system that requires file system access by customizing the client deployment template and running the client playbook.

Steps

  1. Select a compatible client operating system and kernel from the Lustre Support Matrix. Build and install the Lustre client packages for that kernel from netapp-lustre if they are not already installed.

  2. Create a working copy of the clients template directory, including its shared passwords.yml file, and select the template that matches your fabric:

  3. Customize lustre_client_inventory.yml, host_vars, and group_vars for your clients. Line references in the table point to the RoCE client template files in the ansible-lustre release/1.0.0 tag; the InfiniBand client template uses the same variables except where noted.

    Field Ansible variable Template file (release/1.0.0) Scope Notes

    Client hosts

    lustre_clients

    lustre_client_inventory.yml#L4

    Per client

    List each client host name.

    Client SSH user and become password

    ssh_client_user / ssh_client_become_pass

    passwords.yml#L2

    Cluster-wide

    Set the become password only if the SSH user is not root.

    LNet interfaces

    eseries_lustre_lnet

    group_vars/all.yml#L7

    Cluster-wide

    Client-side LNet interface names (values at L13).

    Mount MGS NIDs

    lustre_client_mounts.mgsnode

    group_vars/all.yml#L17

    Cluster-wide

    MGS NIDs used to mount the file system (values at L20).

    File system name

    lustre_client_mounts.fsname

    group_vars/all.yml#L21

    Cluster-wide

    Must match the cluster fsname.

    Mount point

    lustre_client_mounts.mount_point

    group_vars/all.yml#L22

    Cluster-wide

    Client mount directory.

    Client management IP

    ansible_host

    host_vars/client_01.yml#L4

    Repeat per client

    One host_vars/client_NN.yml file per client.

    Storage fabric interfaces

    eseries_roce_interfaces

    host_vars/client_01.yml#L10

    Repeat per client

    Frontend interface addresses (values at L15). The InfiniBand template uses eseries_ipoib_interfaces here.

    Repeat the settings for each client. Create one host_vars/client_NN.yml file for each client and add the corresponding host to lustre_client_inventory.yml. Populate the shared clients/passwords.yml file and encrypt it with Ansible Vault before running the client playbook.

  4. Run the client playbook from the client template directory:

    ansible-playbook -i lustre_client_inventory.yml lustre_client_playbook.yml

The client playbook configures the client network interfaces and LNet, and can create a persistent systemd mount unit for the Lustre file system.

After you finish

Verify the deployment.