Skip to main content
BeeGFS on NetApp with E-Series Storage

Configure Individual File Nodes

Contributors iamjoemccormick
PDFs

Specify configuration for individual file nodes using host variables (host_vars).

Overview

This section walks through populating a host_vars/<FILE_NODE_HOSTNAME>.yml file for each file node in the cluster. These files should only contain configuration unique to a particular file node. This commonly includes:

  • Defining the IP or hostname Ansible should use to connect to the node.

  • Configuring additional interfaces and cluster IPs used for HA cluster services (Pacemaker and Corosync) to communicate to other file nodes. By default these services use the same network as the management interface, but additional interfaces should be available for redundancy. Common practice is to define additional IPs on the storage network, avoiding the need for an additional cluster or management network.

    • The performance of any networks used for cluster communication is not critical for file system performance. With the default cluster configuration generally at least a 1Gbps network will provide sufficient performance for cluster operations such as synchronizing node states and coordinating cluster resource state changes. Slow/busy networks may cause resource state changes to take longer than usual, and in extreme cases could result in nodes being evicted from the cluster if they cannot send heartbeats in a reasonable time frame.

  • Configuring interfaces used for connecting to block nodes over the desired protocol (for example: iSCSI/iSER, NVMe/IB, NVMe/RoCE, FCP, etc.)

Steps

Referencing the IP addressing scheme defined in the Plan the File System section, for each file node in the cluster create a file host_vars/<FILE_NODE_HOSTNAME>/yml and populate it as follows:

  1. At the top specify the IP or hostname Ansible should use to SSH to the node and manage it:

    ansible_host: "<MANAGEMENT_IP>"

  2. Configure additional IPs that can be used for cluster traffic:

    1. If the network type is InfiniBand (using IPoIB):

      eseries_ipoib_interfaces:
- name: <INTERFACE>  # Example: ib0 or i1b
  address: <IP/SUBNET> # Example: 100.127.100.1/16
- name: <INTERFACE>  # Additional interfaces as needed.
  address: <IP/SUBNET>

    2. If the network type is RDMA over Converged Ethernet (RoCE):

      eseries_roce_interfaces:
- name: <INTERFACE>  # Example: eth0.
  address: <IP/SUBNET> # Example: 100.127.100.1/16
- name: <INTERFACE>  # Additional interfaces as needed.
  address: <IP/SUBNET>

    3. If the network type is Ethernet (TCP only, no RDMA):

      eseries_ip_interfaces:
- name: <INTERFACE>  # Example: eth0.
  address: <IP/SUBNET> # Example: 100.127.100.1/16
- name: <INTERFACE>  # Additional interfaces as needed.
  address: <IP/SUBNET>

  3. Indicate what IPs should be used for cluster traffic, with preferred IPs listed higher:

    beegfs_ha_cluster_node_ips:
- <MANAGEMENT_IP> # Including the management IP is typically but not required.
- <IP_ADDRESS>    # Ex: 100.127.100.1
- <IP_ADDRESS>    # Additional IPs as needed.
    Note IPs configured in step two will not be used as cluster IPs unless they are included in the beegfs_ha_cluster_node_ips list. This allows you to configure additional IPs/interfaces using Ansible that can be used for other purposes if desired.

  4. If the file node needs to communicate to block nodes over an IP-based protocol, IPs will need to be configured on the appropriate interface, and any packages required for that protocol installed/configured.

    1. If using iSCSI:

      eseries_iscsi_interfaces:
- name: <INTERFACE>  # Example: eth0.
  address: <IP/SUBNET> # Example: 100.127.100.1/16

    2. If using iSER:

      eseries_ib_iser_interfaces:
- name: <INTERFACE>  # Example: ib0.
  address: <IP/SUBNET> # Example: 100.127.100.1/16
  configure: true # If the file node is directly connected to the block node set to true to setup OpenSM.

    3. If using NVMe/IB:

      eseries_nvme_ib_interfaces:
- name: <INTERFACE>  # Example: ib0.
  address: <IP/SUBNET> # Example: 100.127.100.1/16
  configure: true # If the file node is directly connected to the block node set to true to setup OpenSM.

    4. If using NVMe/RoCE:

      eseries_nvme_roce_interfaces:
- name: <INTERFACE>  # Example: eth0.
  address: <IP/SUBNET> # Example: 100.127.100.1/16

    5. Other Protocols:

      1. If using NVMe/FC, configuring individual interfaces is not required. The BeeGFS cluster deployment will automatically detect the protocol and install/configure requirements as needed. If you are using a fabric to connect file and block nodes, ensure switches are properly zoned following NetApp and your switch vendor's best practices.

      2. Use of FCP or SAS do not require installing or configuring additional software. If using FCP, ensure switches are properly zoned following NetApp and your switch vendor's best practices.

      3. Use of IB SRP is not recommended at this time. Use NVMe/IB or iSER depending on what your E-Series block nodes support.

Click here for an example of a complete inventory file representing a single file node.

Advanced: Toggling NVIDIA ConnectX VPI Adapters between Ethernet and InfiniBand Mode

NVIDIA ConnectX-Virtual Protocol Interconnect® (VPI) adapters support both InfiniBand and Ethernet as the transport layer. Switching between modes is not automatically negotiated, and must be configured using the mstconfig tool included in mstflint, an open source package that is part of the Mellanox Firmare Tools (MFT). Changing the mode of the adapters only need to be done once. This can be done manually, or included in the Ansible inventory as part of any interfaces configured using the eseries-[ib|ib_iser|ipoib|nvme_ib|nvme_roce|roce]_interfaces: section of the inventory, to have it checked/applied automatically.

For example to change an interface current in InfiniBand mode to Ethernet so it can be used for RoCE:

  1. For each interface you want to configure specify mstconfig as a mapping (or dictionary) that specifies LINK_TYPE_P<N> where <N> is determined by the HCA's port number for the interface. The <N> value can be determined by running grep PCI_SLOT_NAME /sys/class/net/<INTERFACE_NAME>/device/uevent and adding 1 to the last number from the PCI slot name and converting to decimal.

    1. For example given PCI_SLOT_NAME=0000:2f:00.2 (2 + 1 → HCA port 3) → LINK_TYPE_P3: eth:

      eseries_roce_interfaces:
- name: <INTERFACE>
  address: <IP/SUBNET>
  mstconfig:
    LINK_TYPE_P3: eth

For additional details refer to the NetApp E-Series Host collection's documentation for the interface type/protocol you are using.