Prepare the worker node

Selecting the right tools

If you are using a combination of drivers, you should install all required tools for your drivers. Recent versions of RedHat CoreOS have the tools installed by default.

NFS tools

Install the NFS tools if you are using: ontap-nas, ontap-nas-economy, ontap-nas-flexgroup, azure-netapp-files, gcp-cvs.

iSCSI tools

Install the iSCSI tools if you are using: ontap-san, ontap-san-economy, solidfire-san.

NVMe tools

Install the NVMe tools if you are using ontap-san for nonvolatile memory express (NVMe) over TCP (NVMe/TCP) protocol.

NetApp recommends ONTAP 9.12 or later for NVMe/TCP.

SCSI over FC tools

Install the FC tools if you are using ontap-san with sanType fcp (SCSI over FC).

Refer to Ways to configure FC & FC-NVMe SAN hosts for more information.

Node service discovery

Trident attempts to automatically detect if the node can run iSCSI or NFS services.

Node service discovery identifies discovered services but does not guarantee services are properly configured. Conversely, the absence of a discovered service does not guarantee the volume mount will fail.

Review events

Trident creates events for the node to identify the discovered services. To review these events, run:

kubectl get event -A --field-selector involvedObject.name=<Kubernetes node name>

Review discovered services

Trident identifies services enabled for each node on the Trident node CR. To view the discovered services, run:

tridentctl get node -o wide -n <Trident namespace>

NFS volumes

Install the NFS tools using the commands for your operating system. Ensure the NFS service is started up during boot time.

sudo yum install -y nfs-utils

Reboot your worker nodes after installing the NFS tools to prevent failure when attaching volumes to containers.

iSCSI volumes

Trident can automatically establish an iSCSI session, scan LUNs, and discover multipath devices, format them, and mount them to a pod.

iSCSI self-healing capabilities

For ONTAP systems, Trident runs iSCSI self-healing every five minutes to:

Identify the desired iSCSI session state and the current iSCSI session state.
Compare the desired state to the current state to identify needed repairs. Trident determines repair priorities and when to preempt repairs.
Perform repairs required to return the current iSCSI session state to the desired iSCSI session state.

Logs of self-healing activity are located in the trident-main container on the respective Daemonset pod. To view logs, you must have set debug to "true" during Trident installation.

Trident iSCSI self-healing capabilities can help prevent:

Stale or unhealthy iSCSI sessions that could occur after a network connectivity issue. In the case of a stale session, Trident waits seven minutes before logging out to reestablish the connection with a portal.

For example, if CHAP secrets were rotated on the storage controller and the network loses connectivity, the old (stale) CHAP secrets could persist. Self-healing can recognize this and automatically reestablish the session to apply the updated CHAP secrets.

Missing iSCSI sessions
Missing LUNs

Points to consider before upgrading Trident

If only per-node igroups (introduced in 23.04+) are in use, iSCSI self-healing will initiate SCSI rescans for all devices in the SCSI bus.
If only backend-scoped igroups (deprecated as of 23.04) are in use, iSCSI self-healing will initiate SCSI rescans for exact LUN IDs in the SCSI bus.
If a mix of per-node igroups and backend-scoped igroups are in use, iSCSI self-healing will initiate SCSI rescans for exact LUN IDs in the SCSI bus.

Install the iSCSI tools

Install the iSCSI tools using the commands for your operating system.

Before you begin

Each node in the Kubernetes cluster must have a unique IQN. This is a necessary prerequisite.
If using RHCOS version 4.5 or later, or other RHEL-compatible Linux distribution, with the solidfire-san driver and Element OS 12.5 or earlier, ensure that the CHAP authentication algorithm is set to MD5 in /etc/iscsi/iscsid.conf. Secure FIPS-compliant CHAP algorithms SHA1, SHA-256, and SHA3-256 are available with Element 12.7.
```
sudo sed -i 's/^\(node.session.auth.chap_algs\).*/\1 = MD5/' /etc/iscsi/iscsid.conf
```
When using worker nodes that run RHEL/RedHat CoreOS with iSCSI PVs, specify the discard mountOption in the StorageClass to perform inline space reclamation. Refer to RedHat documentation.

Install the following system packages:

sudo yum install -y lsscsi iscsi-initiator-utils device-mapper-multipath

Check that iscsi-initiator-utils version is 6.2.0.874-2.el7 or later:
```
rpm -q iscsi-initiator-utils
```
Enable multipathing:
```
sudo mpathconf --enable --with_multipathd y --find_multipaths n
```
Ensure etc/multipath.conf contains find_multipaths no under defaults.

Ensure that iscsid and multipathd are running:

sudo systemctl enable --now iscsid multipathd

Enable and start iscsi:
```
sudo systemctl enable --now iscsi
```

Configure or disable iSCSI self healing

You can configure the following Trident iSCSI self-healing settings to fix stale sessions:

iSCSI self-healing interval: Determines the frequency at which iSCSI self-healing is invoked (default: 5 minutes). You can configure it to run more frequently by setting a smaller number or less frequently by setting a larger number.

Setting the iSCSI self-healing interval to 0 stops iSCSI self-healing completely. We do not recommend disabling iSCSI Self-healing; it should only be disabled in certain scenarios when iSCSI self-healing is not working as intended or for debugging purposes.

iSCSI Self-Healing Wait Time: Determines the duration iSCSI self-healing waits before logging out of an unhealthy session and trying to log in again (default: 7 minutes). You can configure it to a larger number so that sessions that are identified as unhealthy have to wait longer before being logged out and then an attempt is made to log back in, or a smaller number to log out and log in earlier.

To configure or change iSCSI self-healing settings, pass the iscsiSelfHealingInterval and iscsiSelfHealingWaitTime parameters during the helm installation or helm update.

The following example sets the iSCSI self-healing interval to 3 minutes and self-healing wait time to 6 minutes:

helm install trident trident-operator-100.2502.0.tgz --set iscsiSelfHealingInterval=3m0s --set iscsiSelfHealingWaitTime=6m0s -n trident

NVMe/TCP volumes

Install the NVMe tools using the commands for your operating system.

NVMe requires RHEL 9 or later.
If the kernel version of your Kubernetes node is too old or if the NVMe package is not available for your kernel version, you might have to update the kernel version of your node to one with the NVMe package.

sudo yum install nvme-cli
sudo yum install linux-modules-extra-$(uname -r)
sudo modprobe nvme-tcp

Verify installation

After installation, verify that each node in the Kubernetes cluster has a unique NQN using the command:

cat /etc/nvme/hostnqn

Trident modifies the ctrl_device_tmo value to ensure NVMe doesn't give up on the path if it goes down. Do not change this setting.

SCSI over FC volumes

You can now use the Fibre Channel (FC) protocol with Trident to provision and manage storage resources on ONTAP system.

Prerequisites

Configure the required network and node settings for FC.

Network settings

Get the WWPN of the target interfaces. Refer to network interface show for more information.
Get the WWPN for the interfaces on initiator (Host).

Refer to the corresponding host operating system utilities.
Configure zoning on the FC switch using WWPNs of the Host and target.

Refer to the respecive switch vendor documentation for information.

Refer to the following ONTAP documentation for details:
- Fibre Channel and FCoE zoning overview
- Ways to configure FC & FC-NVMe SAN hosts

Install the FC tools

Install the FC tools using the commands for your operating system.

When using worker nodes that run RHEL/RedHat CoreOS with FC PVs, specify the discard mountOption in the StorageClass to perform inline space reclamation. Refer to RedHat documentation.