Configure BeeGFS clients

08/13/2024 Contributors

You must install and configure the BeeGFS client on any hosts that need access to the BeeGFS file system, such as compute or GPU nodes. For this task, you can use Ansible and the BeeGFS collection.

Steps

If needed, set up passwordless SSH from the Ansible control node to each of the hosts you want to configure as BeeGFS clients:

ssh-copy-id <user>@<HOSTNAME_OR_IP>

Under host_vars/, create a file for each BeeGFS client named <HOSTNAME>.yml with the following content, filling in the placeholder text with the correct information for your environment:

# BeeGFS Client
ansible_host: <MANAGEMENT_IP>
# OPTIONAL: If you want to use the NetApp E-Series Host Collection’s IPoIB role to configure InfiniBand interfaces for clients to connect to BeeGFS file systems:
eseries_ipoib_interfaces:
  - name: <INTERFACE>
    address: <IP>/<SUBNET_MASK> # Example: 100.127.1.1/16
  - name: <INTERFACE>
    address: <IP>/<SUBNET_MASK>

If deploying with a two subnet addressing scheme, two InfiniBand interfaces must be configured on each client, one in each of the two storage IPoIB subnets. If using the example subnets and recommended ranges for each BeeGFS service listed here, clients should have one interface configured in the range of 100.127.1.0 to 100.127.99.255 and the other in 100.128.1.0 to 100.128.99.255.

Create a new file client_inventory.yml, and then populate the following parameters at the top:

# BeeGFS client inventory.
all:
  vars:
    ansible_ssh_user: <USER> # This is the user Ansible should use to connect to each client.
    ansible_become_password: <PASSWORD> # This is the password Ansible will use for privilege escalation, and requires the ansible_ssh_user be root, or have sudo privileges.
The defaults set by the BeeGFS HA role are based on the testing performed as part of this NetApp Verified Architecture and differ from the typical BeeGFS client defaults.

Do not store passwords in plain text. Instead, use the Ansible Vault (see the Ansible documentation for Encrypting content with Ansible Vault) or use the --ask-become-pass option when running the playbook.

In the client_inventory.yml file, list all hosts that should be configured as BeeGFS clients under the beegfs_clients group, and then specify any additional configuration required to build the BeeGFS client kernel module.

  children:
    # Ansible group representing all BeeGFS clients:
    beegfs_clients:
      hosts:
        beegfs_01:
        beegfs_02:
        beegfs_03:
        beegfs_04:
        beegfs_05:
        beegfs_06:
        beegfs_07:
        beegfs_08:
        beegfs_09:
        beegfs_10:
      vars:
        # OPTION 1: If you’re using the NVIDIA OFED drivers and they are already installed:
        eseries_ib_skip: True # Skip installing inbox drivers when using the IPoIB role.
        beegfs_client_ofed_enable: True
        beegfs_client_ofed_include_path: "/usr/src/ofa_kernel/default/include"
        # OPTION 2: If you’re using inbox IB/RDMA drivers and they are already installed:
        eseries_ib_skip: True # Skip installing inbox drivers when using the IPoIB role.
        # OPTION 3: If you want to use inbox IB/RDMA drivers and need them installed/configured.
        eseries_ib_skip: False # Default value.
        beegfs_client_ofed_enable: False # Default value.

When using the NVIDIA OFED drivers, make sure that beegfs_client_ofed_include_path points to the correct "header include path" for your Linux installation. For more information, see the BeeGFS documentation for RDMA support.

In the client_inventory.yml file, list the BeeGFS file systems you want mounted at the bottom of any previously defined vars.

        beegfs_client_mounts:
          - sysMgmtdHost: 100.127.101.0 # Primary IP of the BeeGFS management service.
            mount_point: /mnt/beegfs    # Path to mount BeeGFS on the client.
            connInterfaces:
              - <INTERFACE> # Example: ibs4f1
              - <INTERFACE>
            beegfs_client_config:
              # Maximum number of simultaneous connections to the same node.
              connMaxInternodeNum: 128 # BeeGFS Client Default: 12
              # Allocates the number of buffers for transferring IO.
              connRDMABufNum: 36 # BeeGFS Client Default: 70
              # Size of each allocated RDMA buffer
              connRDMABufSize: 65536 # BeeGFS Client Default: 8192
              # Required when using the BeeGFS client with the shared-disk HA solution.
              # This does require BeeGFS targets be mounted in the default “sync” mode.
              # See the documentation included with the BeeGFS client role for full details.
              sysSessionChecksEnabled: false

The beegfs_client_config represents the settings that were tested. See the documentation included with the netapp_eseries.beegfs collection’s beegfs_client role for a comprehensive overview of all options. This includes details around mounting multiple BeeGFS file systems or mounting the same BeeGFS file system multiple times.

Create a new client_playbook.yml file, and then populate the following parameters:

# BeeGFS client playbook.
- hosts: beegfs_clients
  any_errors_fatal: true
  gather_facts: true
  collections:
    - netapp_eseries.beegfs
    - netapp_eseries.host
  tasks:
    - name: Ensure IPoIB is configured
      import_role:
        name: ipoib
    - name: Verify the BeeGFS clients are configured.
      import_role:
        name: beegfs_client

Omit importing the netapp_eseries.host collection and ipoib role if you have already installed the required IB/RDMA drivers and configured IPs on the appropriate IPoIB interfaces.

To install and build the client and mount BeeGFS, run the following command:
```
ansible-playbook -i client_inventory.yml client_playbook.yml
```
Before you place the BeeGFS file system in production, we strongly recommend that you log in to any clients and run beegfs-fsck --checkfs to ensure that all nodes are reachable and there are no issues reported.

Configure BeeGFS clients

Creating your file...