Skip to main content

Rebalance FlexGroup volumes by moving files

Contributors netapp-lenida netapp-aherbin netapp-dbagwell netapp-mwallis
PDFs

Beginning with ONTAP 9.12.1, you can rebalance FlexGroup volumes by non-disruptively moving files from one constituent in a FlexGroup to another constituent.

FlexGroup rebalancing helps redistribute capacity when imbalances develop over time due to the addition of new files and file growth. After you manually start the rebalance operation, ONTAP selects the files and moves them automatically and non-disruptively.

Note

You should be aware that FlexGroup rebalancing degrades system performance when large numbers of files are moved as part of a single rebalancing event or over multiple rebalancing events because of the creation of multi-part inodes. Every file moved as part of a rebalancing event has 2 multi-part inodes associated with that file. The larger the number of files with multi-part inodes as a percentage of the total number of files in a FlexGroup, the greater the performance impact. Certain use cases, such as a FlexVol to FlexGroup conversion, can result in a significant amount of multi-part inode creation.

Rebalancing is available only when all nodes in the cluster are running ONTAP 9.12.1 or later releases. You must enable granular data functionality on any FlexGroup volume that runs the rebalancing operation. Once that functionality is enabled, you cannot revert to ONTAP 9.11.1 and earlier versions unless you delete this volume or restore from a Snapshot copy that was created before the setting was enabled.

Beginning with ONTAP 9.14.1, ONTAP introduces an algorithm to non-disruptively and proactively move files in volumes that have granular data enabled without user interaction. The algorithm operates in very specific, targeted scenarios to alleviate performance bottlenecks. Scenarios where this algorithm might act include very heavy write load on a particular set of files on one node in the cluster or a continually growing file in a very hot parent directory.

Beginning with ONTAP 9.16.1, you can also enable advanced capacity balancing to redistribute a large file's data between FlexGroup member volumes.

FlexGroup rebalancing considerations

You should be aware of how FlexGroup rebalancing works and how it interacts with other ONTAP features.

  • FlexVol to FlexGroup conversion

    It is recommended that you not use automatic FlexGroup rebalancing after a FlexVol to FlexGroup conversion. Instead, you can use the disruptive retroactive file move feature available in ONTAP 9.10.1 and later, by entering the volume rebalance file-move command. For command syntax, see the volume rebalance file-move start man page.

    Rebalancing with the automatic FlexGroup rebalancing feature can degrade performance when moving large numbers of files, like when you perform a FlexVol to FlexGroup conversion, and as much as 50 to 85% of the data on the FlexVol volume is moved to a new constituent.

  • Minimum and maximum file size

    File selection for automatic rebalancing is based on blocks saved. The minimum file size considered for rebalancing is 100 MB by default (can be configured as low as 20 MB using the min-file-size parameter shown below) and the maximum file size is 100 GB.

  • Files in Snapshot copies

    You can configure FlexGroup rebalancing to only consider files to be moved which are not currently present in any Snapshot copies. When rebalancing is started, a notification displays if a Snapshot copy operation is scheduled anytime during a rebalancing operation.

    Snapshot copies are restricted if a file is being moved and is undergoing framing at the destination. A Snapshot copy restore operation is not allowed while file rebalancing is in progress.

    Any Snapshot copy created after the granular-data option is enabled cannot be replicated to a system running ONTAP 9.11.1 and earlier versions because ONTAP 9.11.1 and earlier versions do not support multi-part inodes.

  • SnapMirror operations

    FlexGroup rebalancing should take place between scheduled SnapMirror operations. A SnapMirror operation might fail if a file is being relocated before a SnapMirror operation begins if that file move does not complete within the 24-minute SnapMirror retry period. Any new file relocation that begins after a SnapMirror transfer has started will not fail.

  • File-based compression storage efficiency

    With file-based compression storage efficiency, the file is decompressed before it’s moved to the destination, so the compression savings is lost. The compression savings is regained after a manually initiated background scanner runs on the FlexGroup volume after rebalancing. However, if any file is associated with a Snapshot copy on any volume, the file will be ignored for compression.

  • Deduplication

    Moving deduplicated files can cause increased overall usage for the FlexGroup volume. During file rebalancing, only unique blocks are moved to the destination, freeing that capacity on the source. Shared blocks remain on the source and are copied to the destination. While this achieves the goal of reducing the used capacity on a nearly full source constituent, it can also lead to increased overall usage on the FlexGroup volume due to copies of shared blocks on the new destinations. This is also possible when files that are part of a Snapshot copy are moved. The space savings is not fully recognized until the Snapshot copy schedule recycles and there are no longer copies of the files in Snapshot copies.

  • FlexClone volumes

    If file rebalancing is in progress when a FlexClone volume is created, the rebalancing will not be performed on the FlexClone volume. Rebalancing on the FlexClone volume should be performed after it is created.

  • File move

    When a file is moved during a FlexGroup rebalancing operation, the file size is reported as part of quota accounting on both the source and destination constituents. Once the move is completed, quota accounting returns to normal, and the file size is only reported on the new destination.

  • Autonomous Ransomware Protection

    Beginning with ONTAP 9.13.1, Autonomous Ransomware Protection is supported during disruptive and non-disruptive rebalance operations.

  • Object store volumes

    Volume capacity rebalancing is not supported on object store volumes, such as S3 buckets.

Enable FlexGroup rebalancing

Beginning with ONTAP 9.12.1, you can enable automatic non-disruptive FlexGroup volume rebalancing to redistribute files between FlexGroup constituents.

Beginning with ONTAP 9.13.1, you can schedule a single FlexGroup rebalancing operation to begin at a date and time in the future.

Before you begin

You must have enabled the granular-data option on the FlexGroup volume before enabling FlexGroup rebalancing. You can enable it by using one of these methods:

  • When you create FlexGroup volume using the volume create command

  • By modifying an existing FlexGroup volume to enable the setting using the volume modify command

  • Setting it automatically when FlexGroup rebalancing is initiated using the volume rebalance command

    Note If you are using ONTAP 9.16.1 or later and FlexGroup advanced capacity balancing is enabled using either the granular-data advanced option in the ONTAP CLI or using System Manager, FlexGroup rebalancing is also enabled.
Steps

You can manage FlexGroup rebalancing by using ONTAP System Manager or the ONTAP CLI.

System Manager

  1. Navigate to Storage > Volumes and locate the FlexGroup volume to rebalance.

  2. Select Dropdown icon to view the volume details.

  3. Under FlexGroup Balance Status select Rebalance.

    Note The Rebalance option is only available when the FlexGroup status is out of balance.

  4. In the Rebalance Volume window, change the default settings as needed.

  5. To schedule the rebalancing operation, select Rebalance Later and enter the date and time.

CLI

  1. Start automatic rebalancing:

    volume rebalance start -vserver <SVM name> -volume <volume name>

    Optionally, you can specify the following options:

    [[-max-runtime] <time interval>] Maximum Runtime

    [-max-threshold <percent>] Maximum Imbalance Threshold per Constituent

    [-min-threshold <percent>] Minimum Imbalance Threshold per Constituent

    [-max-file-moves <integer>] Maximum Concurrent File Moves per Constituent

    [-min-file-size {<integer>[KB|MB|GB|TB|PB]}] Minimum file size

    [-start-time <mm/dd/yyyy-00:00:00>] Schedule rebalance start date and time

    [-exclude-snapshots {true|false}] Exclude files stuck in Snapshot copies

    Example:

    volume rebalance start -vserver vs0 -volume fg1

Modify FlexGroup rebalance configurations

You can change a FlexGroup rebalancing configuration to update the imbalance threshold, number of concurrent files moves minimum file size, maximum runtime, and to include or exclude Snapshot copies. Options to modify your FlexGroup rebalancing schedule are available beginning with ONTAP 9.13.1.

System Manager

  1. Navigate to Storage > Volumes and locate the FlexGroup volume to rebalance.

  2. Select Dropdown icon to view the volume details.

  3. Under FlexGroup Balance Status select Rebalance.

    Note The Rebalance option is only available when the FlexGroup status is out of balance.

  4. In the Rebalance Volume window, change the default settings as needed.

CLI

  1. Modify automatic rebalancing:

    volume rebalance modify -vserver <SVM name> -volume <volume name>

    You can specify one or more of the following options:

    [[-max-runtime] <time interval>] Maximum Runtime

    [-max-threshold <percent>] Maximum Imbalance Threshold per Constituent

    [-min-threshold <percent>] Minimum Imbalance Threshold per Constituent

    [-max-file-moves <integer>] Maximum Concurrent File Moves per Constituent

    [-min-file-size {<integer>[KB|MB|GB|TB|PB]}] Minimum file size

    [-start-time <mm/dd/yyyy-00:00:00>] Schedule rebalance start date and time

    [-exclude-snapshots {true|false}] Exclude files stuck in Snapshot copies

Stop FlexGroup rebalance

After FlexGroup rebalancing is enabled or scheduled, you can stop it at any time.

System Manager

  1. Navigate to Storage > Volumes and locate the FlexGroup volume.

  2. Select Dropdown icon to view the volume details.

  3. Select Stop Rebalance.

CLI

  1. Stop FlexGroup rebalancing:

    volume rebalance stop -vserver <SVM name> -volume <volume name>

View FlexGroup rebalance status

You can display the status about a FlexGroup rebalance operation, the FlexGroup rebalance configuration, the rebalance operation time, and the rebalance instance details.

System Manager

  1. Navigate to Storage > Volumes and locate the FlexGroup volume.

  2. Select Dropdown icon to view the FlexGroup details.

  3. FlexGroup Balance Status is displayed near the bottom of the details pane.

  4. To view information about the last rebalance operation, select Last Volume Rebalance Status.

CLI

  1. View the status of a FlexGroup rebalance operation:

    volume rebalance show

    Example of rebalance state:

    > volume rebalance show
Vserver: vs0
                                                        Target     Imbalance
Volume       State                  Total      Used     Used       Size     %
------------ ------------------ --------- --------- --------- --------- -----
fg1          idle                     4GB   115.3MB         -       8KB    0%

    Example of rebalance configuration details:

    > volume rebalance show -config
Vserver: vs0
                    Max            Threshold         Max          Min          Exclude
Volume              Runtime        Min     Max       File Moves   File Size    Snapshot
---------------     ------------   -----   -----     ----------   ---------    ---------
fg1                 6h0m0s         5%      20%          25          4KB          true

    Example of rebalance time details:

    > volume rebalance show -time
Vserver: vs0
Volume               Start Time                    Runtime        Max Runtime
----------------     -------------------------     -----------    -----------
fg1                  Wed Jul 20 16:06:11 2022      0h1m16s        6h0m0s

    Example of rebalance instance details:

        > volume rebalance show -instance
    Vserver Name: vs0
    Volume Name: fg1
    Is Constituent: false
    Rebalance State: idle
    Rebalance Notice Messages: -
    Total Size: 4GB
    AFS Used Size: 115.3MB
    Constituent Target Used Size: -
    Imbalance Size: 8KB
    Imbalance Percentage: 0%
    Moved Data Size: -
    Maximum Constituent Imbalance Percentage: 1%
    Rebalance Start Time: Wed Jul 20 16:06:11 2022
    Rebalance Stop Time: -
    Rebalance Runtime: 0h1m32s
    Rebalance Maximum Runtime: 6h0m0s
    Maximum Imbalance Threshold per Constituent: 20%
    Minimum Imbalance Threshold per Constituent: 5%
    Maximum Concurrent File Moves per Constituent: 25
    Minimum File Size: 4KB
    Exclude Files Stuck in Snapshot Copies: true