Ingest

Ingest refers to the process of a client application saving an object to the StorageGRID Webscale system.

Data flow

When a client saves an object to the StorageGRID Webscale system, the Storage Node’s LDR service assigns it a unique identifier, which is returned to the client to use for later retrieval or delete operations. The object is evaluated against the active ILM policy, and then copies are made and distributed according to the ILM policy’s ILM rules. The LDR service sends object data to disk for storage and object metadata to the DDS service for management.
Ingest

  1. The client application creates the object and sends it to the StorageGRID Webscale system through an HTTP request. The object is evaluated against the system’s ILM policy. Copies of object data are made and LDR services chosen as a result of an ADC topology query to other Storage Node’s that make up the storage pool.
  2. The LDR service saves the object data as a replicated copy to disk.
  3. The LDR service sends the object metadata to a DDS service.
  4. The DDS service saves the object metadata to disk.
  5. The DDS service propagates copies of object metadata to other DDS services via the underlying distributed key-value store. These copies are also saved to disk.
  6. The LDR service returns a unique identifier for the object to the client. The identifier type depends on the client's interface (S3 or Swift).

Ingest related attributes

The table shows the primary attributes used to track the ingest of an object to the StorageGRID Webscale system:

Component Attribute changes
LDR > Storage Total Usable Space (STAS): The total amount of object storage space that is currently available to be used to store objects decreases by an amount roughly equivalent to the newly ingested object’s size.

Total Usable Space (Percent) (SAVP): The total amount of object storage space (displayed as a percentage) that is currently available decreases by an amount roughly equivalent to the newly ingested object’s size.

Total Persistent Data (SPSD): The estimate of the size of the persistently stored data increases by an amount roughly equivalent to the object’s size.

Objects Committed (OCOM): The number of object store operations for replicated copies that have been processed by the LDR service increases by one.

DDS > Data Store Total Objects (DATC): The total number of objects that are known by this DDS service increases by one.

S3 and Swift Objects (SDOC): When a data object is ingested by a client application that interfaces to the StorageGRID Webscale system through the S3 or Swift REST API, the total number of data objects known by this DDS service increases by one.