Ingest data flow

An ingest, or save, operation consists of a defined data flow between the client and the StorageGRID Webscale system. The system uses attributes to track the ingestion of the object and the available storage space.

Data flow

When a client saves an object to the StorageGRID Webscale system, the Storage Node processes the request and stores the metadata and data to disk.


Ingest

  1. The client application creates the object and sends it to the StorageGRID Webscale system through an HTTP PUT request.
  2. The LDR service saves the object data as a replicated copy to disk.
  3. The LDR service sends the object metadata to a DDS service.
  4. The DDS service saves the object metadata to disk.
  5. The DDS service propagates copies of object metadata to other DDS services via the metadata store. These copies are also saved to disk.
  6. The LDR service returns an HTTP 200 OK response to the client to acknowledge that the object has been ingested.
  7. The object is then evaluated against the system's ILM policy.

Ingest related attributes

The table shows the primary attributes used to track the ingest of an object to the StorageGRID Webscale system:

Component Attribute changes
Nodes > Storage Object Data: The amount of replicated object data stored on the object store increases by an amount roughly equivalent to the newly ingested object's size. Object data may be initially stored as two replicated copies from dual commit, resulting in twice the space used.
Nodes > Storage Node > Objects Total Objects: The total number of S3 and Swift data objects known by this grid node increases by one.