Adding storage capacity for erasure-coded objects

If your ILM policy includes a rule that makes erasure-coded copies of objects, you must plan where to add new storage and when to add new storage. The amount of storage you add and the timing of the addition can affect the grid's usable storage capacity.

The first step in planning a storage expansion is to examine the rules in your ILM policy that create erasure-coded objects. Because StorageGRID creates k+m fragments for every erasure-coded object and stores each fragment on a different Storage Node, you must ensure that at least k+m Storage Nodes are writable after the expansion. If the erasure-coding profile provides site-loss protection, you must also make sure to add storage to each site.

It is not necessary to add k+m Storage Nodes in every storage expansion. The number of nodes you need to add depends on how full existing nodes are. If all Storage Nodes are 100% full, you must add at least k+m new Storage Nodes to permit additional objects to be stored. However, if you add Storage Nodes while existing nodes still have space available, StorageGRID can distribute fragments for new objects across the free space on both the old and new Storage Nodes. The amount of usable space that is created by the expansion depends both on the amount of new capacity added and the space remaining on the existing Storage Nodes.

Optimizing the total amount of usable space created by an expansion requires making some calculations.

Rule of thumb for adding storage capacity for erasure-coded objects

If you want to avoid detailed calculations, you can simply add two Storage Nodes per site when existing Storage Nodes reach 70% capacity.

This rule of thumb provides reasonable results across a wide range of erasure-coding schemes for both single-site grids and for grids where erasure coding provides site-loss protection.

For a better understanding of the factors that lead to this rule of thumb or to develop a more precise plan for your site, review the next section. For a custom recommendation optimized for your situation, contact your NetApp account representative.

Calculating the number of expansion Storage Nodes to add for erasure-coded objects

To optimize how you expand a deployment that stores erasure-coded objects, you must consider many factors:
  • Erasure-coding scheme in use
  • Characteristics of the storage pool used for erasure coding, including the number of nodes at each site and the amount of free space on each node
  • Whether the grid was previously expanded
  • Exact nature of the ILM policy, such as whether ILM rules make both replicated and erasure-coded objects

The following examples can help you understand the impact of the erasure-coding scheme, the number of nodes in the storage pool, and the amount of free space on each node.

Similar considerations affect the calculations required for an ILM policy that stores both replicated and erasure-coded data, or for a grid that has been previously expanded (where the amount of free space per Storage Node might not be approximately the same on all nodes).

Example 1: Expanding a one-site grid that uses 2+1 erasure coding

This example shows how to develop an expansion plan for a simple grid that includes only three Storage Nodes.
Note: Using only three Storage Nodes is not a recommended configuration: if this were an actual production grid, it should use a minimum of k+m + 1 Storage Nodes for redundancy, which equals four Storage Nodes (2+1+1) for this example.
Suppose the deployment has the following characteristics:
  • All data is stored using the 2+1 erasure-coding scheme.
  • There is one site with three Storage Nodes. Each Storage Node has a total capacity of 100 TB.

Assume that you want to expand by adding new 100 TB Storage Nodes, and that you want to eventually use 100% of the storage capacity of all nodes, old and new.

You have a number of options:

Example 2: Expanding a three-site grid that uses 6+3 erasure coding

This example shows how to develop an expansion plan for a multi-site grid that has an erasure-coding scheme with a larger number of fragments. Despite the differences between these examples, the recommended expansion plan is very similar.

Suppose the deployment has the following characteristics:
  • All data is stored using the 6+3 erasure coding scheme
  • There are three sites, and each site has four Storage Nodes (12 nodes in total). Each node has a total capacity of 100 TB.

Assume that you want to expand by adding new 100 TB Storage Nodes, and that you want to eventually use 100% of the storage capacity of all nodes, old and new.

You have a number of options: