Installing the data broker on a Linux host Edit on GitHub Request doc changes

Contributors netapp-bcammett

When you create a sync relationship, choose the On-Prem Data Broker option to install the data broker software on an on-premises Linux host, or on an existing Linux host in the cloud. Cloud Sync guides you through the installation process, but the requirements and steps are repeated on this page to help you prepare for installation.

Linux host requirements

  • Operating system: Red Hat Enterprise Linux 7, CentOS 7, or Fedora

    The command yum update all must be run on the host before you install the data broker.

    A Red Hat Enterprise Linux system must be registered with Red Hat Subscription Management. If it is not registered, the system cannot access repositories to update required 3rd party software during installation.

  • RAM: 16 GB

  • CPU: 4 cores

  • Free disk space: 10 GB

Networking requirements

  • The Linux host must have a connection to the source and target.

  • The Linux host must have an outbound internet connection so the data broker can poll the Cloud Sync service for tasks over port 443.

    If you need to limit outbound connectivity, see the list of endpoints that the data broker contacts.

  • NetApp recommends configuring the source, target, and data broker to use a Network Time Protocol (NTP) service. The time difference between the three components should not exceed 5 minutes.

AWS permissions for S3 buckets

If an S3 bucket is the source or target in the relationship, Cloud Sync prompts you for an AWS user that has programmatic access and the permissions included in this NetApp-provided policy. Refer to the steps below for more details.

GCP permissions for Google Cloud Storage buckets

If a GCP Storage bucket is the source or target in the relationship, Cloud Sync prompts you to provide GCP access. Refer to the steps below for more details.

Installing the data broker

You can install a data broker on a Linux host when you create a sync relationship.

Steps
  1. Click Create New Sync Relationship.

  2. On the Define Sync Relationship page, choose a source and target and click Continue.

    Complete the steps until you reach the Data Broker page.

  3. On the Data Broker page, click On-Prem Data Broker.

    If you already have a data broker, you’ll need to click the A screenshot if the plus icon icon first.

    A screenshot of the Data Broker page that enables you to choose between an AWS

    Even though the option is labeled On-Prem Data Broker, it applies to a Linux host on your premises or in the cloud.
  4. Enter a name for the data broker and click Continue.

    The instructions load in a separate browser tab.

  5. Complete the installation by following the steps in one of the sections below:

Installation steps when the source or target are not S3 or GCP

If S3 or GCP Storage are not the source or target in the relationship, then you should follow these steps.

Steps
  1. Download the installer, switch to superuser and make the installer executable:

    curl <URI> -o data_broker_installer.sh
    sudo -s
    chmod +x data_broker_installer.sh

    URI

    Cloud Sync displays the URI of the installation file when you follow the prompts to deploy the On-Prem Data Broker. That URI isn’t repeated here because the link is generated dynamically and can be used only once. Follow these steps to obtain the URI from Cloud Sync.

  2. Install the data broker software using one of the following options:

    • No proxy configuration:

      ./data_broker_installer.sh

    • Proxy configuration:

      ./data_broker_installer.sh -h <proxy_host> -p <proxy_port>

    • Proxy configuration with authentication:

      ./data_broker_installer.sh -h <proxy_host> -p <proxy_port> -u <proxy_username> -w <proxy_password>

  3. Once the data broker is available, click Continue in Cloud Sync.

  4. Complete the pages in the wizard to create the new sync relationship.

Installation steps when AWS S3 is the source or target

If an S3 bucket is the source or target in the relationship, Cloud Sync prompts you for an AWS user that has programmatic access and specific permissions.

Watch the following video to learn how to install the data broker when S3 is the source or target. Or scroll down to read the steps.

Steps
  1. Prepare the Linux host for AWS access:

    1. Create an IAM policy using this NetApp-provided policy. View AWS instructions.

    2. Create an IAM user that has programmatic access. View AWS instructions.

      Be sure to copy the AWS keys because you need to specify them when you install the data broker software.

  2. Download the installer, switch to superuser, and make the installer executable:

    curl <URI> -o data_broker_installer.sh
    sudo -s
    chmod +x data_broker_installer.sh

    URI

    Cloud Sync displays the URI of the installation file when you follow the prompts to deploy the On-Prem Data Broker. That URI isn’t repeated here because the link is generated dynamically and can be used only once. Follow these steps to obtain the URI from Cloud Sync.

  3. Install the data broker software using one of the following options:

    • No proxy configuration:

      ./data_broker_installer.sh -a <access_key> -s <secret_key>

    • Proxy configuration:

      ./data_broker_installer.sh -a <access_key> -s <secret_key> -h <proxy_host> -p <proxy_port>

    • Proxy configuration with authentication:

      ./data_broker_installer.sh -a <access_key> -s <secret_key> -h <proxy_host> -p <proxy_port> -u <proxy_username> -w <proxy_password>

      AWS keys are stored on the data broker, which runs in your VPC. NetApp does not use the keys outside of the data broker.
  4. Once the data broker is available, click Continue in Cloud Sync.

  5. Complete the pages in the wizard to create the new sync relationship.

Installation steps when GCP Storage is the source or target

If a GCP Storage bucket is the source or target in the relationship, Cloud Sync prompts you to set up the data broker with GCP access. Refer to the steps below for more details.

Steps
  1. Prepare the Linux host for GCP access:

    1. Create a GCP service account that has Storage Admin permissions, if you don’t already have one.

    2. Create a service account key saved in JSON format. View GCP instructions.

      The file should contain at least the following properties: "project_id", "private_key", "client_email"

      When you create a key, the file gets generated and downloaded to your machine.
    3. Save the JSON file to the Linux host.

  2. Download the installer, switch to superuser and make the installer executable:

    curl <URI> -o data_broker_installer.sh
    sudo -s
    chmod +x data_broker_installer.sh

    URI

    Cloud Sync displays the URI of the installation file when you follow the prompts to deploy the On-Prem Data Broker. That URI isn’t repeated here because the link is generated dynamically and can be used only once. Follow these steps to obtain the URI from Cloud Sync.

  3. Install the data broker software using one of the following options:

    • No proxy configuration:

      ./data_broker_installer.sh -g <absolute_path_to_the_json_file>

      Example
      ./data_broker_installer.sh -g /var/google/permissions-GC-bbc20118554a.json

    • Proxy configuration:

      ./data_broker_installer.sh -g <absolute_path_to_the_json_file> -h <proxy_host> -p <proxy_port>

    • Proxy configuration with authentication:

      ./data_broker_installer.sh -g <absolute_path_to_the_json_file> -h <proxy_host> -p <proxy_port> -u <proxy_username> -w <proxy_password>

  4. Once the data broker is available, click Continue in Cloud Sync.

  5. Complete the pages in the wizard to create the new sync relationship.