If your ILM policy includes a rule that makes erasure-coded copies of objects, you must plan where to add new storage and when to add new storage. The amount of storage you add and the timing of the addition can affect the grid's usable storage capacity.
The first step in planning a storage expansion is to examine the rules in your ILM policy that create erasure-coded objects. Because StorageGRID creates k+m fragments for every erasure-coded object and stores each fragment on a different Storage Node, you must ensure that at least k+m Storage Nodes are writable after the expansion. If the erasure-coding profile provides site-loss protection, you must also make sure to add storage to each site.
It is not necessary to add k+m Storage Nodes in every storage expansion. The number of nodes you need to add depends on how full existing nodes are. If all Storage Nodes are 100% full, you must add at least k+m new Storage Nodes to permit additional objects to be stored. However, if you add Storage Nodes while existing nodes still have space available, StorageGRID can distribute fragments for new objects across the free space on both the old and new Storage Nodes. The amount of usable space that is created by the expansion depends both on the amount of new capacity added and the space remaining on the existing Storage Nodes.
Optimizing the total amount of usable space created by an expansion requires making some calculations.
If you want to avoid detailed calculations, you can simply add two Storage Nodes per site when existing Storage Nodes reach 70% capacity.
This rule of thumb provides reasonable results across a wide range of erasure-coding schemes for both single-site grids and for grids where erasure coding provides site-loss protection.
For a better understanding of the factors that lead to this rule of thumb or to develop a more precise plan for your site, review the next section. For a custom recommendation optimized for your situation, contact your NetApp account representative.
The following examples can help you understand the impact of the erasure-coding scheme, the number of nodes in the storage pool, and the amount of free space on each node.
Similar considerations affect the calculations required for an ILM policy that stores both replicated and erasure-coded data, or for a grid that has been previously expanded (where the amount of free space per Storage Node might not be approximately the same on all nodes).
Assume that you want to expand by adding new 100 TB Storage Nodes, and that you want to eventually use 100% of the storage capacity of all nodes, old and new.
You have a number of options:
You are adding 3×100 TB (300 TB) of additional capacity. Using the 2+1 erasure coding scheme means that every object is stored as three fragments, and each fragment is saved on a different Storage Node. After the expansion is complete, when new objects are stored to the grid, all fragments are placed on the new nodes.
You can add the new nodes when existing nodes are 100% full.
The expansion Storage Nodes add 2×100 TB (200 TB) of additional storage capacity to the deployment. Each object is stored as three fragments. To use 100% of the new storage capacity there must be 100 TB of capacity free on the existing Storage Nodes, or 100/3 = 33 TB) free on each existing node.
In other words, you can expand when the existing Storage Nodes have 67 TB of content and are 67% full.
The expansion Storage Node adds 100 TB of storage. To use all 100 TB on the expansion Storage Node, there must be 200 TB of free capacity available on the three existing Storage Nodes, or 200/3 = 67 TB) free per node.
In other words, you must expand when the existing Storage Nodes have 33 TB of content and are only 33% full.
This example shows how to develop an expansion plan for a multi-site grid that has an erasure-coding scheme with a larger number of fragments. Despite the differences between these examples, the recommended expansion plan is very similar.
Assume that you want to expand by adding new 100 TB Storage Nodes, and that you want to eventually use 100% of the storage capacity of all nodes, old and new.
You have a number of options:
The expansion Storage Nodes add 900 TB (9×100 TB) of additional capacity. Using the 6+3 erasure coding scheme means that every object is stored as 9 fragments and each fragment is saved to a different Storage Node. After the expansion is complete, when new object are stored to the grid, all fragments are placed on the new nodes.
This expansion adds k+m nodes. Adding twelve new nodes (four per site) would be preferred to maintain redundancy. If you add only k+m expansion Storage Nodes when existing nodes are 100% full, all new objects must be stored on the expansion nodes. If any writeable node becomes unavailable, even temporarily, StorageGRID cannot meet ILM requirements.
If you expand by adding two 100 TB Storage Nodes per site, or six nodes in total, the new nodes add 600 TB (6×100 TB) of additional storage capacity. Every object is stored as 9 fragments. To use all 600 TB of the new capacity there must be 300 TB available on the existing 12 Storage Nodes, or 300/12 = 25 TB of free space per existing node.
In other words, you can expand when the existing Storage Nodes are 75% full.
The expansion Storage Nodes add 300 TB (3×100 TB) of capacity in total. To use all 300 TB of expansion capacity there must be 600 TB of free space available on the 12 existing Storage Nodes, or 600/12 = 50 TB of free space per node.
In other words, you must expand when the existing Storage Nodes are only 50% full.