Skip to main content
BlueXP copy and sync

BlueXP copy and sync technical FAQ

Contributors netapp-bcammett netapp-ivanad

This FAQ can help if you’re just looking for a quick answer to a question.

Getting started

The following questions relate to getting started with BlueXP copy and sync.

How does BlueXP copy and sync work?

BlueXP copy and sync uses the NetApp data broker software to sync data from a source to a target (this is called a sync relationship).

A data broker group controls the sync relationships between your sources and targets. After you set up a sync relationship, BlueXP copy and sync analyzes your source system and breaks it up into multiple replication streams to push to your selected target data.

After the initial copy, the service syncs any changed data based on the schedule that you set.

How does the 14-day free trial work?

The 14-day free trial starts when you sign up for the BlueXP copy and sync service. You're not subject to NetApp charges for BlueXP copy and sync relationships you create for 14 days. However, all resource charges for any data brokers that you deploy still applies.

How much does BlueXP copy and sync cost?

There are two types of costs associated with using BlueXP copy and sync: service charges and resource charges.

Service charges

For pay-as-you-go pricing, BlueXP copy and sync service charges are hourly, based on the number of sync relationships that you create.

BlueXP copy and sync licenses are also available through your NetApp representative. Each license enables 20 sync relationships for 12 months.

Note BlueXP copy and sync relationships are free for Cloud Volumes Service and Azure NetApp Files.

Resource charges

The resource charges are related to the compute and storage costs for running the data broker in the cloud.

How is BlueXP copy and sync billed and how do I manage my subscription?

There are two ways to pay for sync relationships after your 14-day free trial ends. The first option is to subscribe from AWS or Azure, which enables you to pay-as-you-go or to pay annually. The second option is to purchase licenses directly from NetApp. In each case, your subscription will be managed through your provider marketplace and not via the BlueXP copy and sync user interface.

Can I use BlueXP copy and sync outside the cloud?

Yes, you can use BlueXP copy and sync in a non-cloud architecture. The source and target can reside on-premises and so can the data broker software.

Note the following key points about using BlueXP copy and sync outside of the cloud:

  • A data broker group needs an internet connection to communicate with the BlueXP copy and sync service.

  • If you don't purchase a license directly from NetApp, you will need an AWS or Azure account for the PAYGO BlueXP copy and sync service billing.

How do I access BlueXP copy and sync?

BlueXP copy and sync is available from the BlueXP website in the Sync tab.

What's a data broker group?

Each data broker belongs to a data broker group. Grouping data brokers together helps improve the performance of sync relationships.

Supported sources and targets

The following questions related to the source and targets that are supported in a sync relationship.

Which sources and targets does BlueXP copy and sync support?

BlueXP copy and sync supports many different types of sync relationships. View the entire list.

What versions of NFS and SMB does BlueXP copy and sync support?

BlueXP copy and sync supports NFS version 3 and later, and SMB version 1 and later.

When Amazon S3 is the target, can the data be tiered to a specific S3 storage class?

Yes, you can choose a specific S3 storage class when AWS S3 is the target:

  • Standard (this is the default class)

  • Intelligent-Tiering

  • Standard-Infrequent Access

  • One Zone-Infrequent Access

  • Glacier Deep Archive

  • Glacier Flexible Retrieval

  • Glacier Instant Retrieval

What about storage tiers for Azure Blob storage?

You can choose a specific Azure Blob storage tier when a Blob container is the target:

  • Hot storage

  • Cool storage

Do you support Google Cloud storage tiers?

Yes, you can choose a specific storage class when a Google Cloud Storage bucket is the target:

  • Standard

  • Nearline

  • Coldline

  • Archive

Networking

The following questions relate to networking requirements for BlueXP copy and sync.

What are the networking requirements for BlueXP copy and sync?

The BlueXP copy and sync environment requires that a data broker group is connected with the source and the target through the selected protocol or object storage API (Amazon S3, Azure Blob, IBM Cloud Object Storage).

In addition, a data broker group needs an outbound internet connection over port 443 so it can communicate with the BlueXP copy and sync service and contact a few other services and repositories.

For more details, review networking requirements.

Can I use a proxy server with the data broker?

Yes.

BlueXP copy and sync supports proxy servers with or without basic authentication. If you specify a proxy server when you deploy a data broker, all HTTP and HTTPS traffic from the data broker is routed through the proxy. Note that non-HTTP traffic such as NFS or SMB can’t be routed through a proxy server.

The only proxy server limitation is when using data-in-flight encryption with an NFS or Azure NetApp Files sync relationship. The encrypted data is sent over HTTPS and isn’t routable through a proxy server.

Data synchronization

The following questions relate to how data synchronization works.

How often does synchronization occur?

The default schedule is set for daily synchronization. After the initial synchronization, you can:

  • Modify the sync schedule to your desired number of days, hours, or minutes

  • Disable the sync schedule

  • Delete the sync schedule (no data will be lost; only the sync relationship will be removed)

What is the minimum sync schedule?

You can schedule a relationship to sync data as often as every 1 minute.

Does the data broker group retry when a file fails to sync? Or does it timeout?

A data broker group doesn't timeout when a single file fails to transfer. Instead, the data broker group retries 3 times before skipping the file. The retry value is configurable in the settings for a sync relationship.

What if I have a very large dataset?

If a single directory contains 600,000 files or more, contact us so that we can help you configure the data broker group to handle the payload. We might need to add additional memory to the data broker group.

Note that there's no limit to the total number of files in the mount point. The extra memory is required for large directories with 600,000 files or more, regardless of their level in the hierarchy (top directory or subdirectory).

Security

The following questions related to security.

Is BlueXP copy and sync secure?

Yes. All BlueXP copy and sync service networking connectivity is done using Amazon Simple Queue Service (SQS).

All communication between the data broker group and Amazon S3, Azure Blob, Google Cloud Storage, and IBM Cloud Object Storage is done through the HTTPS protocol.

If you're using BlueXP copy and sync with on-premises (source or destination) systems, here's a few recommended connectivity options:

  • An AWS Direct Connect, Azure ExpressRoute, or Google Cloud Interconnect connection, which is non-internet routed (and can only communicate with the cloud networks that you specify)

  • A VPN connection between your on-premises gateway device and your cloud networks

  • For extra secure data transfer with S3 buckets, Azure Blob storage, or Google Cloud Storage, an Amazon Private S3 Endpoint, Azure Virtual Network service endpoints, or Private Google Access may be established.

Any of these methods establishes a secure connection between your on-premises NAS servers and a BlueXP copy and sync data broker group.

Is data encrypted by BlueXP copy and sync?

  • BlueXP copy and sync supports data-in-flight encryption between source and target NFS servers. Learn more.

  • For SMB, BlueXP copy and sync supports SMB 3.0 and 3.11 data that you've encrypted on the server side. BlueXP copy and sync copies the encrypted data from the source to the target where the data remains encrypted.

    BlueXP copy and sync cannot encrypt SMB data itself.

  • When an Amazon S3 bucket is the target in a sync relationship, you can choose whether to enable data encryption using AWS KMS encryption or AES-256 encryption.

  • When a Google Storage bucket is the target in a sync relationship, you can choose whether to use the default, Google-managed encryption key or your own KMS key.

Permissions

The following questions relate to data permissions.

Are SMB data permissions synced to the target location?

You can set up BlueXP copy and sync to preserve access control lists (ACLs) between a source SMB share and a target SMB share, and from a source SMB share to object storage (except for ONTAP S3).

Note BlueXP copy and sync doesn't support copying ACLs from object storage to SMB shares.

Are NFS data permissions synced to the target location?

BlueXP copy and sync automatically copies NFS permissions between NFS servers as follows:

  • NFS version 3: BlueXP copy and sync copies the permissions and the user group owner.

  • NFS version 4: BlueXP copy and sync copies the ACLs.

Object storage metadata

What kinds of sync relationships preserve object storage metadata?

BlueXP copy and sync copies object storage metadata from the source to the target for the following types of sync relationships:

  • Amazon S3 → Amazon S3 1

  • Amazon S3 → StorageGRID

  • StorageGRID → Amazon S3

  • StorageGRID → StorageGRID

  • StorageGRID → Google Cloud Storage

  • Google Cloud Storage → StorageGRID 1

  • Google Cloud Storage → IBM Cloud Object Storage 1

  • Google Cloud Storage → Amazon S3 1

  • Amazon S3 → Google Cloud Storage

  • IBM Cloud Object Storage → Google Cloud Storage

  • StorageGRID → IBM Cloud Object Storage

  • IBM Cloud Object Storage → StorageGRID

  • IBM Cloud Object Storage → IBM Cloud Object Storage

What kinds of metadata are replicated during syncs where NFS or SMB are the source?

Metadata such as user ID, modification time, access time, and GID are replicated by default. Users may opt into replicating ACL from CIFs by marking it as required when creating a sync relationship.

Performance

The following questions relate to BlueXP copy and sync performance.

What does the progress indicator for a sync relationship represent?

The sync relationship shows the throughput of the data broker group's network adapter. If you accelerated sync performance by using multiple data brokers, then the throughput is the sum of all traffic. This throughput refreshes every 20 seconds.

I'm experiencing performance issues. Can we limit the number of concurrent transfers?

If you have very large files (multiple TiBs each), it can take a long time to complete the transfer process and performance might be impacted.

Limiting the number of concurrent transfers can help. Contact us for help.

Why am I experiencing low performance with Azure NetApp Files?

When you sync data to or from Azure NetApp Files, you might experience failures and performance issues if the disk service level is Standard.

Change the service level to Premium or Ultra to enhance the sync performance.

Why am I experiencing low performance with Cloud Volumes Service for AWS?

When you sync data to or from a cloud volume, you might experience failures and performance issues if the level of performance for the cloud volume is Standard.

Change the Service level to Premium or Extreme to enhance the sync performance.

How many data brokers are required in a group?

When you create a new relationship, you start with a single data broker in a group (unless you selected an existing data broker that belongs to an accelerated sync relationship). In many cases, a single data broker can meet the performance requirements for a sync relationship. If it doesn't, you can accelerate sync performance by adding additional data brokers to the group. But you should first check other factors that can impact sync performance.

Multiple factors can impact data transfer performance. The overall sync performance might be impacted due to network bandwidth, latency, and network topology, as well as the data broker VM specs and storage system performance. For example, a single data broker in a group can reach 100 MB/s, while disk throughput on the target might only allow 64 MB/s. As a result, the data broker group keeps trying to copy the data, but the target can't meet the performance of the data broker group.

So be sure to check the performance of your networking and the disk throughput on the target.

Then you can consider accelerating sync performance by adding an additional data brokers to a group to share the load of that relationship. Learn how to accelerate sync performance.

Deleting things

The following questions relate to deleting sync relationships and data from sources and targets.

What happens if I delete my BlueXP copy and sync relationship?

Deleting a relationship stops all future data syncs and terminates payment. Any data that was synced to the target remains as-is.

What happens if I delete something from my source server? Is it removed from the target too?

By default, if you have an active sync relationship, the item deleted on the source server is not deleted from the target during the next synchronization. But there is an option in the sync settings for each relationship, where you can define that BlueXP copy and sync will delete files in the target location if they were deleted from the source.

What happens if I delete something from my target? Is it removed from my source too?

If an item is deleted from the target, it will not be removed from the source. The relationship is one-way—from source to target. On the next sync cycle, BlueXP copy and sync compares the source to the target, identifies that the item is missing, and BlueXP copy and sync copies it again from the source to the target.

Data broker deep dive

The following question relates to the data broker.

Can you explain the architecture of the data broker?

Sure. Here are the most important points:

  • The data broker is a node.js application running on a Linux host.

  • BlueXP copy and sync deploys the data broker as follows:

    • AWS: From an AWS CloudFormation template

    • Azure: From Azure Resource Manager

    • Google: From Google Cloud Deployment Manager

    • If you use your own Linux host, you need to manually install the software

  • The data broker software automatically upgrades itself to the latest version.

  • The data broker uses AWS SQS as a reliable and secure communication channel and for control and monitoring. SQS also provides a persistency layer.

  • You can add additional data brokers to a group to increase transfer speed and add high availability. There is service resiliency if one data broker fails.