Kubernetes and Trident objects
You can interact with Kubernetes and Trident using REST APIs by reading and writing resource objects. There are several resource objects that dictate the relationship between Kubernetes and Trident, Trident and storage, and Kubernetes and storage. Some of these objects are managed through Kubernetes and the others are managed through Trident.
How do the objects interact with one another?
Perhaps the easiest way to understand the objects, what they are for, and how they interact, is to follow a single request for storage from a Kubernetes user:
-
A user creates a
PersistentVolumeClaim
requesting a newPersistentVolume
of a particular size from a KubernetesStorageClass
that was previously configured by the administrator. -
The Kubernetes
StorageClass
identifies Trident as its provisioner and includes parameters that tell Trident how to provision a volume for the requested class. -
Trident looks at its own
StorageClass
with the same name that identifies the matchingBackends
andStoragePools
that it can use to provision volumes for the class. -
Trident provisions storage on a matching backend and creates two objects: a
PersistentVolume
in Kubernetes that tells Kubernetes how to find, mount, and treat the volume, and a volume in Trident that retains the relationship between thePersistentVolume
and the actual storage. -
Kubernetes binds the
PersistentVolumeClaim
to the newPersistentVolume
. Pods that include thePersistentVolumeClaim
mount that PersistentVolume on any host that it runs on. -
A user creates a
VolumeSnapshot
of an existing PVC, using aVolumeSnapshotClass
that points to Trident. -
Trident identifies the volume that is associated with the PVC and creates a snapshot of the volume on its backend. It also creates a
VolumeSnapshotContent
that instructs Kubernetes on how to identify the snapshot. -
A user can create a
PersistentVolumeClaim
usingVolumeSnapshot
as the source. -
Trident identifies the required snapshot and performs the same set of steps involved in creating a
PersistentVolume
and aVolume
.
For further reading about Kubernetes objects, we highly recommend that you read the Persistent Volumes section of the Kubernetes documentation. |
Kubernetes PersistentVolumeClaim
objects
A Kubernetes PersistentVolumeClaim
object is a request for storage made by a Kubernetes cluster user.
In addition to the standard specification, Trident allows users to specify the following volume-specific annotations if they want to override the defaults that you set in the backend configuration:
Annotation | Volume Option | Supported Drivers |
---|---|---|
trident.netapp.io/fileSystem |
fileSystem |
ontap-san, solidfire-san,ontap-san-economy |
trident.netapp.io/cloneFromPVC |
cloneSourceVolume |
ontap-nas, |
trident.netapp.io/splitOnClone |
splitOnClone |
ontap-nas, ontap-san |
trident.netapp.io/protocol |
protocol |
any |
trident.netapp.io/exportPolicy |
exportPolicy |
ontap-nas, |
trident.netapp.io/snapshotPolicy |
snapshotPolicy |
ontap-nas, |
trident.netapp.io/snapshotReserve |
snapshotReserve |
ontap-nas, |
trident.netapp.io/snapshotDirectory |
snapshotDirectory |
ontap-nas, |
trident.netapp.io/unixPermissions |
unixPermissions |
ontap-nas, |
trident.netapp.io/blockSize |
blockSize |
solidfire-san |
If the created PV has the Delete
reclaim policy, Trident deletes both the PV and the backing volume when the PV becomes released (that is, when the user deletes the PVC). Should the delete action fail, Trident marks the PV as such and periodically retries the operation until it succeeds or the PV is manually deleted. If the PV uses the Retain
policy, Trident ignores it and assumes the administrator will clean it up from Kubernetes and the backend, allowing the volume to be backed up or inspected before its removal. Note that deleting the PV does not cause Trident to delete the backing volume. You should remove it using the REST API (tridentctl
).
Trident supports the creation of Volume Snapshots using the CSI specification: you can create a Volume Snapshot and use it as a Data Source to clone existing PVCs. This way, point-in-time copies of PVs can be exposed to Kubernetes in the form of snapshots. The snapshots can then be used to create new PVs. Take a look at On-Demand Volume Snapshots
to see how this would work.
Trident also provides the cloneFromPVC
and splitOnClone
annotations for creating clones. You can use these annotations to clone a PVC without having to use the CSI implementation.
Here is an example: If a user already has a PVC called mysql
, the user can create a new PVC called mysqlclone
by using the annotation, such as trident.netapp.io/cloneFromPVC: mysql
. With this annotation set, Trident clones the volume corresponding to the mysql PVC, instead of provisioning a volume from scratch.
Consider the following points:
-
We recommend cloning an idle volume.
-
A PVC and its clone should be in the same Kubernetes namespace and have the same storage class.
-
With the
ontap-nas
andontap-san
drivers, it might be desirable to set the PVC annotationtrident.netapp.io/splitOnClone
in conjunction withtrident.netapp.io/cloneFromPVC
. Withtrident.netapp.io/splitOnClone
set totrue
, Trident splits the cloned volume from the parent volume and thus, completely decoupling the life cycle of the cloned volume from its parent at the expense of losing some storage efficiency. Not settingtrident.netapp.io/splitOnClone
or setting it tofalse
results in reduced space consumption on the backend at the expense of creating dependencies between the parent and clone volumes such that the parent volume cannot be deleted unless the clone is deleted first. A scenario where splitting the clone makes sense is cloning an empty database volume where it's expected for the volume and its clone to greatly diverge and not benefit from storage efficiencies offered by ONTAP.
The sample-input
directory contains examples of PVC definitions for use with Trident. Refer to [Trident Volume
objects] for a full description of the parameters and settings associated with Trident volumes.
Kubernetes PersistentVolume
objects
A Kubernetes PersistentVolume
object represents a piece of storage that is made available to the Kubernetes cluster. It has a lifecycle that is independent of the pod that uses it.
Trident creates PersistentVolume objects and registers them with the Kubernetes cluster automatically based on the volumes that it provisions. You are not expected to manage them yourself.
|
When you create a PVC that refers to a Trident-based StorageClass
, Trident provisions a new volume using the corresponding storage class and registers a new PV for that volume. In configuring the provisioned volume and corresponding PV, Trident follows the following rules:
-
Trident generates a PV name for Kubernetes and an internal name that it uses to provision the storage. In both cases, it is assuring that the names are unique in their scope.
-
The size of the volume matches the requested size in the PVC as closely as possible, though it might be rounded up to the nearest allocatable quantity, depending on the platform.
Kubernetes StorageClass
objects
Kubernetes StorageClass
objects are specified by name in PersistentVolumeClaims
to provision storage with a set of properties. The storage class itself identifies the provisioner to be used and defines that set of properties in terms the provisioner understands.
It is one of two basic objects that need to be created and managed by the administrator. The other is the Trident backend object.
A Kubernetes StorageClass
object that uses Trident looks like this:
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: <Name> provisioner: csi.trident.netapp.io mountOptions: <Mount Options> parameters: <Trident Parameters> allowVolumeExpansion: true volumeBindingMode: Immediate
These parameters are Trident-specific and tell Trident how to provision volumes for the class.
The storage class parameters are:
Attribute | Type | Required | Description |
---|---|---|---|
attributes |
map[string]string |
no |
See the attributes section below |
storagePools |
map[string]StringList |
no |
Map of backend names to lists |
additionalStoragePools |
map[string]StringList |
no |
Map of backend names |
excludeStoragePools |
map[string]StringList |
no |
Map of backend names to |
Storage attributes and their possible values can be classified into storage pool selection attributes and Kubernetes attributes.
Storage pool selection attributes
These parameters determine which Trident-managed storage pools should be utilized to provision volumes of a given type.
Attribute | Type | Values | Offer | Request | Supported by |
---|---|---|---|---|---|
media1 |
string |
hdd, hybrid, ssd |
Pool contains media of this type; hybrid means both |
Media type specified |
ontap-nas, ontap-nas-economy, ontap-nas-flexgroup, ontap-san, solidfire-san |
provisioningType |
string |
thin, thick |
Pool supports this provisioning method |
Provisioning method specified |
thick: all ontap; thin: all ontap & solidfire-san |
backendType |
string |
ontap-nas, ontap-nas-economy, ontap-nas-flexgroup, ontap-san, solidfire-san, gcp-cvs, azure-netapp-files, ontap-san-economy |
Pool belongs to this type of backend |
Backend specified |
All drivers |
snapshots |
bool |
true, false |
Pool supports volumes with snapshots |
Volume with snapshots enabled |
ontap-nas, ontap-san, solidfire-san, gcp-cvs |
clones |
bool |
true, false |
Pool supports cloning volumes |
Volume with clones enabled |
ontap-nas, ontap-san, solidfire-san, gcp-cvs |
encryption |
bool |
true, false |
Pool supports encrypted volumes |
Volume with encryption enabled |
ontap-nas, ontap-nas-economy, ontap-nas-flexgroups, ontap-san |
IOPS |
int |
positive integer |
Pool is capable of guaranteeing IOPS in this range |
Volume guaranteed these IOPS |
solidfire-san |
1: Not supported by ONTAP Select systems
In most cases, the values requested directly influence provisioning; for instance, requesting thick provisioning results in a thickly provisioned volume. However, an Element storage pool uses its offered IOPS minimum and maximum to set QoS values, rather than the requested value. In this case, the requested value is used only to select the storage pool.
Ideally, you can use attributes
alone to model the qualities of the storage you need to satisfy the needs of a particular class. Trident automatically discovers and selects storage pools that match all of the attributes
that you specify.
If you find yourself unable to use attributes
to automatically select the right pools for a class, you can use the storagePools
and additionalStoragePools
parameters to further refine the pools or even to select a specific set of pools.
You can use the storagePools
parameter to further restrict the set of pools that match any specified attributes
. In other words, Trident uses the intersection of pools identified by the attributes
and storagePools
parameters for provisioning. You can use either parameter alone or both together.
You can use the additionalStoragePools
parameter to extend the set of pools that Trident uses for provisioning, regardless of any pools selected by the attributes
and storagePools
parameters.
You can use the excludeStoragePools
parameter to filter the set of pools that Trident uses for provisioning. Using this parameter removes any pools that match.
In the storagePools
and additionalStoragePools
parameters, each entry takes the form <backend>:<storagePoolList>
, where <storagePoolList>
is a comma-separated list of storage pools for the specified backend. For example, a value for additionalStoragePools
might look like ontapnas_192.168.1.100:aggr1,aggr2;solidfire_192.168.1.101:bronze
.
These lists accept regex values for both the backend and list values. You can use tridentctl get backend
to get the list of backends and their pools.
Kubernetes attributes
These attributes have no impact on the selection of storage pools/backends by Trident during dynamic provisioning. Instead, these attributes simply supply parameters supported by Kubernetes Persistent Volumes. Worker nodes are responsible for filesystem create operations and might require filesystem utilities, such as xfsprogs.
Attribute | Type | Values | Description | Relevant Drivers | Kubernetes Version |
---|---|---|---|---|---|
fsType |
string |
ext4, ext3, xfs |
The file system type for block |
solidfire-san, ontap-nas, ontap-nas-economy, ontap-nas-flexgroup, ontap-san, ontap-san-economy |
All |
allowVolumeExpansion |
boolean |
true, false |
Enable or disable support for growing the PVC size |
ontap-nas, ontap-nas-economy, ontap-nas-flexgroup, ontap-san, ontap-san-economy, solidfire-san, gcp-cvs, azure-netapp-files |
1.11+ |
volumeBindingMode |
string |
Immediate, WaitForFirstConsumer |
Choose when volume binding and dynamic provisioning occurs |
All |
1.19 - 1.26 |
|
Kubernetes VolumeSnapshotClass
objects
Kubernetes VolumeSnapshotClass
objects are analogous to StorageClasses
. They help define multiple classes of storage and are referenced by volume snapshots to associate the snapshot with the required snapshot class. Each volume snapshot is associated with a single volume snapshot class.
A VolumeSnapshotClass
should be defined by an administrator in order to create snapshots. A volume snapshot class is created with the following definition:
apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: csi-snapclass driver: csi.trident.netapp.io deletionPolicy: Delete
The driver
specifies to Kubernetes that requests for volume snapshots of the csi-snapclass
class are handled by Trident. The deletionPolicy
specifies the action to be taken when a snapshot must be deleted. When deletionPolicy
is set to Delete
, the volume snapshot objects as well as the underlying snapshot on the storage cluster are removed when a snapshot is deleted. Alternatively, setting it to Retain
means that VolumeSnapshotContent
and the physical snapshot are retained.
Kubernetes VolumeSnapshot
objects
A Kubernetes VolumeSnapshot
object is a request to create a snapshot of a volume. Just as a PVC represents a request made by a user for a volume, a volume snapshot is a request made by a user to create a snapshot of an existing PVC.
When a volume snapshot request comes in, Trident automatically manages the creation of the snapshot for the volume on the backend and exposes the snapshot by creating a unique
VolumeSnapshotContent
object. You can create snapshots from existing PVCs and use the snapshots as a DataSource when creating new PVCs.
The lifecyle of a VolumeSnapshot is independent of the source PVC: a snapshot persists even after the source PVC is deleted. When deleting a PVC which has associated snapshots, Trident marks the backing volume for this PVC in a Deleting state, but does not remove it completely. The volume is removed when all the associated snapshots are deleted. |
Kubernetes VolumeSnapshotContent
objects
A Kubernetes VolumeSnapshotContent
object represents a snapshot taken from an already provisioned volume. It is analogous to a PersistentVolume
and signifies a provisioned snapshot on the storage cluster. Similar to PersistentVolumeClaim
and PersistentVolume
objects, when a snapshot is created, the VolumeSnapshotContent
object maintains a one-to-one mapping to the VolumeSnapshot
object, which had requested the snapshot creation.
The VolumeSnapshotContent
object contains details that uniquely identify the snapshot, such as the snapshotHandle
. This snapshotHandle
is a unique combination of the name of the PV and the name of the VolumeSnapshotContent
object.
When a snapshot request comes in, Trident creates the snapshot on the backend. After the snapshot is created, Trident configures a VolumeSnapshotContent
object and thus exposes the snapshot to the Kubernetes API.
Typically, you do not need to manage the VolumeSnapshotContent object. An exception to this is when you want to import a volume snapshot created outside of Astra Trident.
|
Kubernetes CustomResourceDefinition
objects
Kubernetes Custom Resources are endpoints in the Kubernetes API that are defined by the administrator and are used to group similar objects. Kubernetes supports the creation of custom resources for storing a collection of objects. You can obtain these resource definitions by running kubectl get crds
.
Custom Resource Definitions (CRDs) and their associated object metadata are stored by Kubernetes in its metadata store. This eliminates the need for a separate store for Trident.
Astra Trident uses CustomResourceDefinition
objects to preserve the identity of Trident objects, such as Trident backends, Trident storage classes, and Trident volumes. These objects are managed by Trident. In addition, the CSI volume snapshot framework introduces some CRDs that are required to define volume snapshots.
CRDs are a Kubernetes construct. Objects of the resources defined above are created by Trident. As a simple example, when a backend is created using tridentctl
, a corresponding tridentbackends
CRD object is created for consumption by Kubernetes.
Here are a few points to keep in mind about Trident's CRDs:
-
When Trident is installed, a set of CRDs are created and can be used like any other resource type.
-
When uninstalling Trident by using the
tridentctl uninstall
command, Trident pods are deleted but the created CRDs are not cleaned up. Refer to Uninstall Trident to understand how Trident can be completely removed and reconfigured from scratch.
Astra Trident StorageClass
objects
Trident creates matching storage classes for Kubernetes StorageClass
objects that specify csi.trident.netapp.io
in their provisioner field. The storage class name matches that of the Kubernetes StorageClass
object it represents.
With Kubernetes, these objects are created automatically when a Kubernetes StorageClass that uses Trident as a provisioner is registered.
|
Storage classes comprise a set of requirements for volumes. Trident matches these requirements with the attributes present in each storage pool; if they match, that storage pool is a valid target for provisioning volumes using that storage class.
You can create storage class configurations to directly define storage classes by using the REST API. However, for Kubernetes deployments, we expect them to be created when registering new Kubernetes StorageClass
objects.
Astra Trident backend objects
Backends represent the storage providers on top of which Trident provisions volumes; a single Trident instance can manage any number of backends.
This is one of the two object types that you create and manage yourself. The other is the Kubernetes StorageClass object.
|
For more information about how to construct these objects, refer to configuring backends.
Astra Trident StoragePool
objects
Storage pools represent the distinct locations available for provisioning on each backend. For ONTAP, these correspond to aggregates in SVMs. For NetApp HCI/SolidFire, these correspond to administrator-specified QoS bands. For Cloud Volumes Service, these correspond to cloud provider regions. Each storage pool has a set of distinct storage attributes, which define its performance characteristics and data protection characteristics.
Unlike the other objects here, storage pool candidates are always discovered and managed automatically.
Astra Trident Volume
objects
Volumes are the basic unit of provisioning, comprising backend endpoints, such as NFS shares and iSCSI LUNs. In Kubernetes, these correspond directly to PersistentVolumes
. When you create a volume, ensure that it has a storage class, which determines where that volume can be provisioned, along with a size.
|
A volume configuration defines the properties that a provisioned volume should have.
Attribute | Type | Required | Description |
---|---|---|---|
version |
string |
no |
Version of the Trident API ("1") |
name |
string |
yes |
Name of volume to create |
storageClass |
string |
yes |
Storage class to use when provisioning the volume |
size |
string |
yes |
Size of the volume to provision in bytes |
protocol |
string |
no |
Protocol type to use; "file" or "block" |
internalName |
string |
no |
Name of the object on the storage system; generated by Trident |
cloneSourceVolume |
string |
no |
ontap (nas, san) & solidfire-*: Name of the volume to clone from |
splitOnClone |
string |
no |
ontap (nas, san): Split the clone from its parent |
snapshotPolicy |
string |
no |
ontap-*: Snapshot policy to use |
snapshotReserve |
string |
no |
ontap-*: Percentage of volume reserved for snapshots |
exportPolicy |
string |
no |
ontap-nas*: Export policy to use |
snapshotDirectory |
bool |
no |
ontap-nas*: Whether the snapshot directory is visible |
unixPermissions |
string |
no |
ontap-nas*: Initial UNIX permissions |
blockSize |
string |
no |
solidfire-*: Block/sector size |
fileSystem |
string |
no |
File system type |
Trident generates internalName
when creating the volume. This consists of two steps. First, it prepends the storage prefix (either the default trident
or the prefix in the backend configuration) to the volume name, resulting in a name of the form <prefix>-<volume-name>
. It then proceeds to sanitize the name, replacing characters not permitted in the backend. For ONTAP backends, it replaces hyphens with underscores (thus, the internal name becomes <prefix>_<volume-name>
). For Element backends, it replaces underscores with hyphens.
You can use volume configurations to directly provision volumes using the REST API, but in Kubernetes deployments we expect most users to use the standard Kubernetes PersistentVolumeClaim
method. Trident creates this volume object automatically as part of the provisioning
process.
Astra Trident Snapshot
objects
Snapshots are a point-in-time copy of volumes, which can be used to provision new volumes or restore state. In Kubernetes, these correspond directly to VolumeSnapshotContent
objects. Each snapshot is associated with a volume, which is the source of the data for the snapshot.
Each Snapshot
object includes the properties listed below:
Attribute | Type | Required | Description |
---|---|---|---|
version |
String |
Yes |
Version of the Trident API ("1") |
name |
String |
Yes |
Name of the Trident snapshot object |
internalName |
String |
Yes |
Name of the Trident snapshot object on the storage system |
volumeName |
String |
Yes |
Name of the Persistent Volume for which the snapshot is created |
volumeInternalName |
String |
Yes |
Name of the associated Trident volume object on the storage system |
In Kubernetes, these objects are managed automatically. You can view them to see what Trident provisioned. |
When a Kubernetes VolumeSnapshot
object request is created, Trident works by creating a snapshot object on the backing storage system. The internalName
of this snapshot object is generated by combining the prefix snapshot-
with the UID
of the VolumeSnapshot
object (for example, snapshot-e8d8a0ca-9826-11e9-9807-525400f3f660
). volumeName
and volumeInternalName
are populated by getting the details of the backing
volume.
Astra Trident ResourceQuota
object
The Trident deamonset consumes a system-node-critical
Priority Class—the highest Priority Class available in Kubernetes—to ensure Astra Trident can identify and clean up volumes during graceful node shutdown and allow Trident daemonset pods to preempt workloads with a lower priority in clusters where there is high resource pressure.
To accomplish this, Astra Trident employs a ResourceQuota
object to ensure a "system-node-critical" Priority Class on the Trident daemonset is satisfied. Prior to deployment and daemonset creation, Astra Trident looks for the ResourceQuota
object and, if not discovered, applies it.
If you need more control over the default Resource Quota and Priority Class, you can generate a custom.yaml
or configure the ResourceQuota
object using Helm chart.
The following is an example of a `ResourceQuota`object prioritizing the Trident daemonset.
apiVersion: <version> kind: ResourceQuota metadata: name: trident-csi labels: app: node.csi.trident.netapp.io spec: scopeSelector: matchExpressions: - operator : In scopeName: PriorityClass values: ["system-node-critical"]
For more information on Resource Quotas, refer to Kubernetes: Resource Quotas.
Clean up ResourceQuota
if installation fails
In the rare case where installation fails after the ResourceQuota
object is created, first try uninstalling and then reinstall.
If that doesn't work, manually remove the ResourceQuota
object.
Remove ResourceQuota
If you prefer to control your own resource allocation, you can remove the Astra Trident ResourceQuota
object using the command:
kubectl delete quota trident-csi -n trident