Skip to main content

Fail over to an older snapshot using Trident Protect

Contributors netapp-shwetav

Use Trident Protect to fail over a replicated application to an earlier snapshot, allowing recovery from a point in time before data was damaged or compromised. By default, a failover uses the most recently transferred snapshot.

Caution Failing over to an older snapshot causes ONTAP SnapMirror to delete the snapshots on the destination cluster that are newer than the selected snapshot, because they are no longer valid. Add the annotation in this procedure only if this data loss is acceptable.
Before you begin
  • An AppMirrorRelationship exists and is in the Established state. To create one, refer to Set up a replication relationship. To verify the state, run kubectl get amr -n <namespace> <relationship_name> -o=jsonpath='{.status}'.

  • The source and destination clusters use ONTAP storage backends with SnapMirror replication. Azure NetApp Files and other non-ONTAP backends are not supported. Refer to Requirements for SnapMirror replication.

  • A snapshot schedule (with snapshotRetention greater than 1) exists on the source application.

Identify the snapshot

Each snapshot has a completion time. Get its archive path with kubectl on the source cluster, or with the CLI.

Note Use the CLI when the source cluster is down or the snapshot custom resource is not available. Run the command on the destination cluster, or on any cluster that has the source AppVault.
Use kubectl

Use kubectl on the source cluster to get the archive path.

  1. On the source cluster, list the snapshots and note the completionTimestamp of the snapshot you want:

    kubectl get snapshots -n my-app-namespace
  2. Get its archive path:

    kubectl get snapshots <snapshot_name> -n my-app-namespace -o jsonpath='{.status.appArchivePath}'
Use the CLI

Use the Trident Protect CLI to get the archive path. On the destination cluster, or on any cluster that has the source AppVault, list the snapshots. TIMESTAMP is the completion time and PATH is the archive path.

tridentctl-protect get appvaultcontent <source_appvault_name> \
--app <app_name> \
--show-resources snapshot \
--show-paths
Note The AppVault CR of the source application must exist on the cluster where you run the command, and your environment must be able to connect to the object storage bucket defined in the AppVault CR. If network restrictions prevent access, run the CLI from within a pod on the cluster.

The basename is everything after the last slash in the path. For example:

20231114173500_schedule-20231114173500_f8df8bd3-a727-4a14-b954-9666aa618b55
Note Use the basename, not the name of the snapshot custom resource.
Result

The basename of the selected snapshot is recorded.

Fail over to the selected snapshot

Specify the snapshot to promote, and acknowledge the data loss by adding an annotation.

Steps
  1. On the destination cluster, add the annotation to the AppMirrorRelationship. Without the annotation, a failover can use only the most recently transferred snapshot.

    kubectl annotate appmirrorrelationship <relationship_name> -n my-app-namespace protect.trident.netapp.io/allow-promotion-with-older-snapshots="true"
  2. Edit the AppMirrorRelationship CR file (for example, trident-protect-relationship.yaml):

    • Set spec.desiredState to Promoted.

    • Set spec.promotedSnapshot to the basename you recorded in Identify the snapshot.

      For example:

      apiVersion: protect.trident.netapp.io/v1
      kind: AppMirrorRelationship
      metadata:
        name: <relationship_name>
        namespace: <namespace>
        annotations:
          protect.trident.netapp.io/allow-promotion-with-older-snapshots: "true"
      spec:
        desiredState: Promoted
        promotedSnapshot: <basename>
  3. Apply the CR:

    kubectl apply -f trident-protect-relationship.yaml -n my-app-namespace
  4. Check the status of the relationship:

    kubectl get amr -n my-app-namespace <relationship_name> -o=jsonpath='{.status}' | jq
Result

The relationship moves to the Promoted state, and the application runs on the destination cluster with the data from the selected snapshot.

If the relationship stays in the Promoting state and status.error shows The snapshot provided is invalid or older than the most recently transferred snapshot, the annotation is missing or is not set to true. Add the annotation and try again.

What's next?

To return to the original cluster, perform a reverse resync, and then reverse the replication direction.

Caution To return to the original cluster, do not perform a normal resync. A normal resync discards the failed-over application and returns the relationship to the newest source snapshot.