Skip to main content

Prepare the worker node

Contributors netapp-aruldeepa
PDFs

All worker nodes in the Kubernetes cluster must be able to mount the volumes you have provisioned for your pods. To prepare the worker nodes, you must install NFS, iSCSI, NVMe/TCP, or FC tools based on your driver selection.

Selecting the right tools

If you are using a combination of drivers, you should install all required tools for your drivers. Recent versions of RedHat CoreOS have the tools installed by default.

NFS tools

Install the NFS tools if you are using: ontap-nas, ontap-nas-economy, ontap-nas-flexgroup, azure-netapp-files, gcp-cvs.

iSCSI tools

Install the iSCSI tools if you are using: ontap-san, ontap-san-economy, solidfire-san.

NVMe tools

Install the NVMe tools if you are using ontap-san for nonvolatile memory express (NVMe) over TCP (NVMe/TCP) protocol.

Note We recommend ONTAP 9.12 or later for NVMe/TCP.
SCSI over FC tools

SCSI over Fibre Channel (FC) is a tech preview feature in the Trident 24.10 release.

Install the FC tools if you are using ontap-san with sanType fcp (SCSI over FC).

Refer to Ways to configure FC & FC-NVMe SAN hosts for more information.

Node service discovery

Trident attempts to automatically detect if the node can run iSCSI or NFS services.

Note Node service discovery identifies discovered services but does not guarantee services are properly configured. Conversely, the absence of a discovered service does not guarantee the volume mount will fail.
Review events

Trident creates events for the node to identify the discovered services. To review these events, run:

kubectl get event -A --field-selector involvedObject.name=<Kubernetes node name>
Review discovered services

Trident identifies services enabled for each node on the Trident node CR. To view the discovered services, run:

tridentctl get node -o wide -n <Trident namespace>

NFS volumes

Install the NFS tools using the commands for your operating system. Ensure the NFS service is started up during boot time.

RHEL 8+
sudo yum install -y nfs-utils
Ubuntu
sudo apt-get install -y nfs-common
Warning Reboot your worker nodes after installing the NFS tools to prevent failure when attaching volumes to containers.

iSCSI volumes

Trident can automatically establish an iSCSI session, scan LUNs, and discover multipath devices, format them, and mount them to a pod.

iSCSI self-healing capabilities

For ONTAP systems, Trident runs iSCSI self-healing every five minutes to:

  1. Identify the desired iSCSI session state and the current iSCSI session state.

  2. Compare the desired state to the current state to identify needed repairs. Trident determines repair priorities and when to preempt repairs.

  3. Perform repairs required to return the current iSCSI session state to the desired iSCSI session state.

Note Logs of self-healing activity are located in the trident-main container on the respective Daemonset pod. To view logs, you must have set debug to "true" during Trident installation.

Trident iSCSI self-healing capabilities can help prevent:

  • Stale or unhealthy iSCSI sessions that could occur after a network connectivity issue. In the case of a stale session, Trident waits seven minutes before logging out to reestablish the connection with a portal.

    Note For example, if CHAP secrets were rotated on the storage controller and the network loses connectivity, the old (stale) CHAP secrets could persist. Self-healing can recognize this and automatically reestablish the session to apply the updated CHAP secrets.

  • Missing iSCSI sessions

  • Missing LUNs

Points to consider before upgrading Trident

  • If only per-node igroups (introduced in 23.04+) are in use, iSCSI self-healing will initiate SCSI rescans for all devices in the SCSI bus.

  • If only backend-scoped igroups (deprecated as of 23.04) are in use, iSCSI self-healing will initiate SCSI rescans for exact LUN IDs in the SCSI bus.

  • If a mix of per-node igroups and backend-scoped igroups are in use, iSCSI self-healing will initiate SCSI rescans for exact LUN IDs in the SCSI bus.

Install the iSCSI tools

Install the iSCSI tools using the commands for your operating system.

Before you begin

  • Each node in the Kubernetes cluster must have a unique IQN. This is a necessary prerequisite.

  • If using RHCOS version 4.5 or later, or other RHEL-compatible Linux distribution, with the solidfire-san driver and Element OS 12.5 or earlier, ensure that the CHAP authentication algorithm is set to MD5 in /etc/iscsi/iscsid.conf. Secure FIPS-compliant CHAP algorithms SHA1, SHA-256, and SHA3-256 are available with Element 12.7.

    sudo sed -i 's/^\(node.session.auth.chap_algs\).*/\1 = MD5/' /etc/iscsi/iscsid.conf

  • When using worker nodes that run RHEL/RedHat CoreOS with iSCSI PVs, specify the discard mountOption in the StorageClass to perform inline space reclamation. Refer to RedHat documentation.

RHEL 8+

  1. Install the following system packages:

    sudo yum install -y lsscsi iscsi-initiator-utils device-mapper-multipath

  2. Check that iscsi-initiator-utils version is 6.2.0.874-2.el7 or later:

    rpm -q iscsi-initiator-utils

  3. Enable multipathing:

    sudo mpathconf --enable --with_multipathd y --find_multipaths n
    Note Ensure etc/multipath.conf contains find_multipaths no under defaults.

  4. Ensure that iscsid and multipathd are running:

    sudo systemctl enable --now iscsid multipathd

  5. Enable and start iscsi:

    sudo systemctl enable --now iscsi
Ubuntu

  1. Install the following system packages:

    sudo apt-get install -y open-iscsi lsscsi sg3-utils multipath-tools scsitools

  2. Check that open-iscsi version is 2.0.874-5ubuntu2.10 or later (for bionic) or 2.0.874-7.1ubuntu6.1 or later (for focal):

    dpkg -l open-iscsi

  3. Set scanning to manual:

    sudo sed -i 's/^\(node.session.scan\).*/\1 = manual/' /etc/iscsi/iscsid.conf

  4. Enable multipathing:

    sudo tee /etc/multipath.conf <<-EOF
defaults {
    user_friendly_names yes
    find_multipaths no
}
EOF
sudo systemctl enable --now multipath-tools.service
sudo service multipath-tools restart
    Note Ensure etc/multipath.conf contains find_multipaths no under defaults.

  5. Ensure that open-iscsi and multipath-tools are enabled and running:

    sudo systemctl status multipath-tools
sudo systemctl enable --now open-iscsi.service
sudo systemctl status open-iscsi
    Note For Ubuntu 18.04, you must discover target ports with iscsiadm before starting open-iscsi for the iSCSI daemon to start. You can alternatively modify the iscsi service to start iscsid automatically.

Configure or disable iSCSI self healing

You can configure the following Trident iSCSI self-healing settings to fix stale sessions:

  • iSCSI self-healing interval: Determines the frequency at which iSCSI self-healing is invoked (default: 5 minutes). You can configure it to run more frequently by setting a smaller number or less frequently by setting a larger number.

Note

Setting the iSCSI self-healing interval to 0 stops iSCSI self-healing completely. We do not recommend disabling iSCSI Self-healing; it should only be disabled in certain scenarios when iSCSI self-healing is not working as intended or for debugging purposes.

  • iSCSI Self-Healing Wait Time: Determines the duration iSCSI self-healing waits before logging out of an unhealthy session and trying to log in again (default: 7 minutes). You can configure it to a larger number so that sessions that are identified as unhealthy have to wait longer before being logged out and then an attempt is made to log back in, or a smaller number to log out and log in earlier.

Helm

To configure or change iSCSI self-healing settings, pass the iscsiSelfHealingInterval and iscsiSelfHealingWaitTime parameters during the helm installation or helm update.

The following example sets the iSCSI self-healing interval to 3 minutes and self-healing wait time to 6 minutes:

helm install trident trident-operator-100.2410.0.tgz --set iscsiSelfHealingInterval=3m0s --set iscsiSelfHealingWaitTime=6m0s -n trident
tridentctl

To configure or change iSCSI self-healing settings, pass the iscsi-self-healing-interval and iscsi-self-healing-wait-time parameters during the tridentctl installation or update.

The following example sets the iSCSI self-healing interval to 3 minutes and self-healing wait time to 6 minutes:

tridentctl install --iscsi-self-healing-interval=3m0s --iscsi-self-healing-wait-time=6m0s -n trident

NVMe/TCP volumes

Install the NVMe tools using the commands for your operating system.

Note

  • NVMe requires RHEL 9 or later.

  • If the kernel version of your Kubernetes node is too old or if the NVMe package is not available for your kernel version, you might have to update the kernel version of your node to one with the NVMe package.
RHEL 9
sudo yum install nvme-cli
sudo yum install linux-modules-extra-$(uname -r)
sudo modprobe nvme-tcp
Ubuntu
sudo apt install nvme-cli
sudo apt -y install linux-modules-extra-$(uname -r)
sudo modprobe nvme-tcp

Verify installation

After installation, verify that each node in the Kubernetes cluster has a unique NQN using the command:

cat /etc/nvme/hostnqn
Warning Trident modifies the ctrl_device_tmo value to ensure NVMe doesn't give up on the path if it goes down. Do not change this setting.

Install the FC tools

Install the FC tools using the commands for your operating system.

  • When using worker nodes that run RHEL/RedHat CoreOS with FC PVs, specify the discard mountOption in the StorageClass to perform inline space reclamation. Refer to RedHat documentation.

RHEL 8+

  1. Install the following system packages:

    sudo yum install -y lsscsi device-mapper-multipath

  2. Enable multipathing:

    sudo mpathconf --enable --with_multipathd y --find_multipaths n
    Note Ensure etc/multipath.conf contains find_multipaths no under defaults.

  3. Ensure that multipathd is running:

    sudo systemctl enable --now multipathd
Ubuntu

  1. Install the following system packages:

    sudo apt-get install -y lsscsi sg3-utils multipath-tools scsitools

  2. Enable multipathing:

    sudo tee /etc/multipath.conf <<-EOF
defaults {
    user_friendly_names yes
    find_multipaths no
}
EOF
sudo systemctl enable --now multipath-tools.service
sudo service multipath-tools restart
    Note Ensure etc/multipath.conf contains find_multipaths no under defaults.

  3. Ensure that multipath-tools is enabled and running:

    sudo systemctl status multipath-tools