Skip to main content
How to enable StorageGRID in your environment

Configure StorageGRID search integration service

Contributors netapp-aronk

This guide provides detailed instructions for configuring NetApp StorageGRID
11.6 search integration service with either Amazon OpenSearch Service or on-premises
Elasticsearch.

Introduction

StorageGRID supports three types of platform services.

  • StorageGRID CloudMirror replication. Mirror specific objects from a StorageGRID bucket to a specified external destination.

  • Notifications. Per-bucket event notifications to send notifications about specific actions performed on objects to a specified external Amazon Simple Notification Service (Amazon SNS).

  • Search integration service. Send Simple Storage Service (S3) object metadata to a specified Elasticsearch index where you can search or analyze the metadata by using the external service.

Platform services are configured by the S3 tenant through the Tenant Manager UI. For more information, see Considerations for using platform services.

This document serves as a supplement to the StorageGRID 11.6 Tenant Guide and provides step by step instructions and examples for the endpoint and bucket configuration for search integration services. The Amazon Web Services (AWS) or on-premises Elasticsearch setup instructions included here are for basic testing or demo purposes only.

Audiences should be familiar with Grid Manager, Tenant Manager, and have access to the S3 browser to perform basic upload (PUT) and download (GET) operations for StorageGRID search integration testing.

Create tenant and enable platform services

  1. Create an S3 tenant by using Grid Manager, enter a display name, and select the S3 protocol.

  2. On the Permission page, select the Allow Platform Services option. Optionally, select other permissions, if necessary.

    Select permissions screenshot

  3. Set up the tenant root user initial password or, if identify federation is enabled on the grid, select which federated group has root access permission to configure the tenant account.

  4. Click Sign In As Root and select Bucket: Create and Manage Buckets.

    This takes you to the Tenant Manager page.

  5. From Tenant Manager, select My Access Keys to create and download the S3 access key for later testing.

Search integration services with Amazon OpenSearch

Amazon OpenSearch (formerly Elasticsearch) service setup

Use this procedure for a quick and simple setup of the OpenSearch service for testing/demo purposes only. If you are using on-premises Elasticsearch for search integration services, see the section Search integration services with on premises Elasticsearch.

Note You must have a valid AWS console login, access key, secret access key, and permission to subscribe to the OpenSearch service.

  1. Create a new domain using the instructions from AWS OpenSearch Service Getting Started, except for the following:

    • Step 4. Domain name: sgdemo

    • Step 10. Fine-grained access control: deselect the Enable Fine-Grained Access Control option.

    • Step 12. Access policy: select Configure Level Access Policy, select the JSON tab to modify the access policy by using the following example:

      • Replace the highlighted text with your own AWS Identity and Access Management (IAM) ID and user name.

      • Replace the highlighted text (the IP address) with the public IP address of your local computer that you used to access the AWS console.

      • Open a browser tab to https://checkip.amazonaws.com to find your public IP.

        {

    "Version": "2012-10-17",
    "Statement": [
        {
        "Effect": "Allow",
        "Principal":
        {"AWS": "arn:aws:iam:: nnnnnn:user/xyzabc"},
        "Action": "es:*",
        "Resource": "arn:aws:es:us-east-1:nnnnnn:domain/sgdemo/*"
        },
        {
        "Effect": "Allow",
        "Principal": {"AWS": "*"},
        "Action": [
        "es:ESHttp*"
                ],
        "Condition": {
            "IpAddress": {
                "aws:SourceIp": [ "nnn.nnn.nn.n/nn"
                    ]
                }
        },
        "Resource": "arn:aws:es:us-east-1:nnnnnn:domain/sgdemo/*"
        }
    ]
}

        Search integration screenshot

  2. Wait 15 to 20 minutes for the domain to become active.

    Activating domain screenshot

  3. Click OpenSearch Dashboards URL to open the domain in a new tab to access the dashboard. If you get an access denied error, verify that the access policy source IP address is correctly set to your computer public IP to allow access to the domain dashboard.

  4. On the dashboard welcome page, select Explore On Your Own. From the menu, go to Management → Dev Tools

  5. Under Dev Tools → Console , enter PUT <index> where you use the index for storing StorageGRID object metadata. We use the index name 'sgmetadata' in the following example. Click the small triangle symbol to execute the PUT command. The expected result displays on the right panel as shown in the following example screenshot.

    Using put command for index screenshot

  6. Verify that the index is visible from Amazon OpenSearch UI under sgdomain > Indices.

    Verifying-the-index screenshot

Platform services endpoint configuration

To configure the platform services endpoints, follow these steps:

  1. In Tenant Manager, go to STORAGE(S3) > Platform services endpoints.

  2. Click Create Endpoint, enter the following, and then click Continue:

    • Display name example aws-opensearch

    • The domain endpoint in the example screenshot under Step 2 of the preceding procedure in the URI field.

    • The domain ARN used in Step 2 of the preceding procedure in the URN field and add /<index>/_doc to the end of ARN.

      In this example, URN becomes arn:aws:es:us-east-1:211234567890:domain/sgdemo /sgmedata/_doc.

      end-points-details screenshot

  3. To access the Amazon OpenSearch sgdomain, choose Access Key as the authentication type and then enter the Amazon S3 access key and secret key. To go the next page, click Continue.

    authenticate connections to endpoints screenshot

  4. To verify the endpoint, select Use Operating System CA Certificate and Test and Create Endpoint. If verification is successful, an endpoint screen similar to the following figure displays. If verification fails, verify that the URN includes /<index>/_doc at the end of the path and the AWS access key and secret key are correct.

    platform service endpoints screenshot

Search integration services with on premises Elasticsearch

On premises Elasticsearch setup

This procedure is for a quick setup of on premises Elasticsearch and Kibana using docker for testing purposes only. If the Elasticsearch and Kibana server already exists, go to Step 5.

  1. Follow this Docker installation procedure to install docker. We use the CentOS Docker install procedure in this setup.

    sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo yum install docker-ce docker-ce-cli containerd.io
sudo systemctl start docker

    • To start docker after reboot, enter the following:

      sudo systemctl enable docker

    • Set the vm.max_map_count value to 262144:

      sysctl -w vm.max_map_count=262144

    • To keep the setting after reboot, enter the following:

      echo 'vm.max_map_count=262144' >> /etc/sysctl.conf

  2. Follow the Elasticsearch Quick start guide self-managed section to install and run the Elasticsearch and Kibana docker. In this example, we installed version 8.1.

    Tip Note down the user name/password and token created by Elasticsearch, you need these to start the Kibana UI and StorageGRID platform endpoint authentication.

    search integration elasticsearch screenshot

  3. After the Kibana docker container has started, the URL link https://0.0.0.0:5601 displays in the console. Replace 0.0.0.0 with the server IP address in the URL.

  4. Log in to the Kibana UI by using user name elastic and the password generated by Elastic in the preceding step.

  5. For first time login, on the dashboard welcome page, select Explore On Your Own. From the menu, select Management > Dev Tools.

  6. On the Dev Tools Console screen, enter PUT <index> where you use this index for storing StorageGRID object metadata. We use the index name sgmetadata in this example. Click the small triangle symbol to execute the PUT command. The expected result displays on the right panel as shown in the following example screenshot.

    Execute put command screenshot

Platform services endpoint configuration

To configure endpoints for platform services, follow these steps:

  1. On Tenant Manager, go to STORAGE(S3) > Platform services endpoints

  2. Click Create Endpoint, enter the following, and then click Continue:

    • Display name example: elasticsearch

    • URI: https://<elasticsearch-server-ip or hostname>:9200

    • URN: urn:<something>:es:::<some-unique-text>/<index-name>/_doc where the index-name is the name you used on the Kibana console.
      Example: urn:local:es:::sgmd/sgmetadata/_doc

      Platform service endpoint details screenshot

  3. Select Basic HTTP as the authentication type, enter the user name elastic and the password generated by the Elasticsearch installation process. To go to the next page, click Continue.

    Platform service endpoint authentication screenshot

  4. Select Do Not Verify Certificate and Test and Create Endpoint to verify the endpoint. If verification is successful, an endpoint screen similar to the following screenshot displays. If the verification fails, verify the URN, URI, and username/password entries are correct.

    Successfully verified endpoint

Bucket search integration service configuration

After the platform service endpoint is created, the next step is to configure this service at bucket level to send object metadata to the defined endpoint whenever an object is created, deleted, or its metadata or tags are updated.

You can configure search integration by using Tenant Manager to apply a custom
StorageGRID configuration XML to a bucket as follows:

  1. In Tenant Manager, go to STORAGE(S3) > Buckets

  2. Click Create Bucket, enter the bucket name (for example, sgmetadata-test) and accept the default us-east-1 region.

  3. Click Continue > Create Bucket.

  4. To bring up the bucket Overview page, click the bucket name, then select Platform Services.

  5. Select the Enable Search Integration dialog box. In the provided XML box, enter the configuration XML using this syntax.

    The highlighted URN must match the platform services endpoint that you defined. You can open another browser tab to access the Tenant Manager and copy the URN from the defined platform services endpoint.

    In this example, we used no prefix, meaning that the metadata for every object in this bucket is sent to the Elasticsearch endpoint defined previously.

    <MetadataNotificationConfiguration>
    <Rule>
        <ID>Rule-1</ID>
        <Status>Enabled</Status>
        <Prefix></Prefix>
        <Destination>
            <Urn> urn:local:es:::sgmd/sgmetadata/_doc</Urn>
        </Destination>
    </Rule>
</MetadataNotificationConfiguration>

  6. Use S3 Browser to connect to StorageGRID with the tenant access/secret key, upload test objects to sgmetadata-test bucket and add tags or custom metadata to objects.

    Upload test objects screenshot

  7. Use the Kibana UI to verify that the object metadata was loaded to sgmetadata’s index.

    1. From the menu, select Management > Dev Tools.

    2. Paste the sample query to the console panel on the left and click the triangle symbol to execute it.

      The query 1 sample result in the following example screenshot shows four records. This matches number of objects in the bucket.

      GET sgmetadata/_search
{
    "query": {
        "match_all": { }
}
}

      Query 1 sample result screenshot

      The query 2 sample result in the following screenshot shows two records with tag type jpg.

      GET sgmetadata/_search
{
    "query": {
        "match": {
            "tags.type": {
                "query" : "jpg" }
                }
            }
}

      Query 2 sample

Where to find additional information

To learn more about the information that is described in this document, review the following documents and/or websites:

By Angela Cheng