Understanding the search integration service
Suggest changes
PDFs
You can enable search integration for an S3 bucket if you want to use an external search and data analysis service for your object metadata.
The search integration service is a custom StorageGRID service that automatically and asynchronously sends S3 object metadata to a destination endpoint whenever an object or its metadata is updated. You can then use sophisticated search, data analysis, visualization, or machine learning tools provided by the destination service to search, analyze, and gain insights from your object data.
You can enable the search integration service for any versioned or unversioned bucket. Search integration is configured by associating metadata notification configuration XML with the bucket that specifies which objects to act on and the destination for the object metadata.
Notifications are generated in the form of a JSON document named with the bucket name, object name, and version ID, if any. Each metadata notification contains a standard set of system metadata for the object in addition to all of the object's tags and user metadata.
|
For tags and user metadata, StorageGRID passes dates and numbers to Elasticsearch as strings or as S3 event notifications. To configure Elasticsearch to interpret these strings as dates or numbers, follow the Elasticsearch instructions for dynamic field mapping and for mapping date formats. You must enable the dynamic field mappings on the index before you configure the search integration service. After a document is indexed, you cannot edit the document's field types in the index. |
Notifications are generated and queued for delivery whenever:
-
An object is created.
-
An object is deleted, including when objects are deleted as a result of the operation of the grid's ILM policy.
-
Object metadata or tags are added, updated, or deleted. The complete set of metadata and tags is always sent on update — not just the changed values.
After you add metadata notification configuration XML to a bucket, notifications are sent for any new objects that you create and for any objects that you modify by updating its data, user metadata, or tags. However, notifications are not sent for any objects that were already in the bucket. To ensure that object metadata for all objects in the bucket is sent to the destination, you should do either of the following:
-
Configure the search integration service immediately after creating the bucket and before adding any objects.
-
Perform an action on all objects already in the bucket that will trigger a metadata notification message to be sent to the destination.
The StorageGRID search integration service supports an Elasticsearch cluster as a destination. As with the other platform services, the destination is specified in the endpoint whose URN is used in the configuration XML for the service. Use the Interoperability Matrix Tool to determine the supported versions of Elasticsearch.