Upgrade to BeeGFS 8

03/10/2026 Contributors

Follow these steps to upgrade your BeeGFS HA cluster from BeeGFS 7.4 to BeeGFS 8.

Overview

BeeGFS 8 introduces several significant changes which require additional setup before upgrading from BeeGFS 7. This document guides you through preparing your cluster for BeeGFS 8's new requirements, and then upgrading to BeeGFS 8.

Before upgrading to BeeGFS 8, ensure your system is running at least BeeGFS 7.4. Any cluster running a release earlier than BeeGFS 7.4 must first upgrade to BeeGFS 7.4 before upgrading to BeeGFS 8.

Key changes in BeeGFS 8

BeeGFS 8 introduces the following major changes:

License enforcement: BeeGFS 8 requires a license to use premium features such as storage pools, remote storage targets, BeeOND, and more. Acquire a valid license for your BeeGFS cluster before upgrading. If needed, you can obtain a temporary BeeGFS 8 evaluation license from the BeeGFS License Portal.
Management Service Database Migration: To enable configuration with the new TOML-based format in BeeGFS 8, you must manually migrate your BeeGFS 7 management service database to the updated BeeGFS 8 format.
TLS encryption: BeeGFS 8 introduces TLS for secure communication between services. You will need to generate and distribute TLS certificates for the BeeGFS management service and the beegfs command-line utility as part of the upgrade.

For more details and additional changes in BeeGFS 8, see the BeeGFS 8.0.0 Upgrade Guide.

Upgrading to BeeGFS 8 requires cluster downtime and BeeGFS 7 clients cannot connect to BeeGFS 8 clusters. Coordinate the upgrade timing carefully to minimize operational impact.

Prepare your BeeGFS cluster for the upgrade

Before starting the upgrade, carefully prepare your environment to ensure a smooth transition and minimize downtime.

Verify that all servers and clients are using supported Linux distributions and kernel versions for BeeGFS 8.
Ensure your cluster is in a healthy state, with all BeeGFS services running on their preferred nodes. From a file node running BeeGFS services, verify all Pacemaker resources are running on their preferred nodes:
```
pcs status
```
Record and back up your cluster configuration. The BeeGFS management service data will be backed up later as part of the management database upgrade process.
1. Run the following commands on a BeeGFS client and save the output for reference:
  beegfs-ctl --getentryinfo --verbose /path/to/beegfs/mountpoint
2. If using mirroring, gather detailed state information so you can identify the original primary target if both mirror members enter "needs-resync" state:
  beegfs-ctl --listtargets --longnodes --state --spaceinfo --mirrorgroups --nodetype=meta beegfs-ctl --listtargets --longnodes --state --spaceinfo --mirrorgroups --nodetype=storage
3. If using mirroring, verify all buddy groups are in GOOD state before proceeding with the upgrade.
Stop the BeeGFS system in the following order:
1. On each client, unmount the BeeGFS filesystem and stop the beegfs-client service. For each client, run:
  systemctl stop beegfs-client
2. For each Pacemaker cluster, disable STONITH so you can validate cluster integrity after the upgrade without triggering unnecessary node reboots:
  pcs property set stonith-enabled=false
3. On all Pacemaker clusters, stop the cluster:
  pcs cluster stop --all

Upgrade the BeeGFS management package

On all file nodes in the cluster, add the BeeGFS 8 package repository for your Linux distribution. Instructions for adding the official BeeGFS repository for your Linux distribution can be found at the BeeGFS download page. Otherwise, configure your local BeeGFS mirror repository accordingly.

On the first two file nodes of your cluster, update only the BeeGFS management daemon package to BeeGFS 8:

For RHEL, after configuring your BeeGFS repository, run the following commands:
```
dnf clean all
dnf update beegfs-mgmtd
```

Upgrade the management database

On one of the file nodes running the BeeGFS management service, follow these steps to migrate the management database from BeeGFS 7 to BeeGFS 8.

List all NVMe devices and identify the management target:
```
nvme netapp smdevices | grep mgmt_tgt
```
Note the device path from the output (for example, /dev/nvmeXnY).
1. Mount the management target device to the existing management target mount point (replace /dev/nvmeXnY with your device path):
  mount /dev/nvmeXnY /mnt/mgmt_tgt_mgmt01/

Perform a backup of the management data directory:

tar --force-local -cpzf /mnt/mgmt_tgt_mgmt01/data_beegfs_7_backup_$(date +'%F_%T').tar.gz /mnt/mgmt_tgt_mgmt01/data

Import your BeeGFS 7 management data into the new database format by running:

/opt/beegfs/sbin/beegfs-mgmtd --import-from-v7=/mnt/mgmt_tgt_mgmt01/data/

Expected output:

Created new database version 3 at "/var/lib/beegfs/mgmtd.sqlite". Successfully imported v7 management data from "/mnt/mgmt_tgt_mgmt01/data/".

The automatic import may not succeed in all cases due to stricter validation requirements in BeeGFS 8. For example, if targets are assigned to non-existent storage pools, the import fails. If the migration fails, do not continue with the upgrade. Contact NetApp support for assistance with resolving the database migration issues. You can downgrade the BeeGFS 8 management packages and continue running BeeGFS 7 while the issue is addressed.

Move the generated SQLite file to the management service mount:
```
mv /var/lib/beegfs/mgmtd.sqlite /mnt/mgmt_tgt_mgmt01/data/
```
Move the generated beegfs-mgmtd.toml to the management service mount:
```
mv /etc/beegfs/beegfs-mgmtd.toml /mnt/mgmt_tgt_mgmt01/mgmt_config/
```
You will prepare the beegfs-mgmtd.toml configuration file after completing the licensing and TLS configuration steps in the following sections.

Upgrade remaining BeeGFS packages

After completing the management database migration, upgrade the remaining BeeGFS packages on all file nodes in the cluster. For RHEL, after configuring your BeeGFS repository, run the following commands:

dnf clean all
dnf update beegfs-storage beegfs-meta libbeegfs-ib

Configure licensing

Install the BeeGFS license packages on all nodes that run the BeeGFS management service. This is typically the first two file nodes of your cluster:
```
dnf install libbeegfs-license
```
Download your BeeGFS 8 license file to the management nodes and place it at:
```
/etc/beegfs/license.pem
```

Configure TLS encryption

BeeGFS 8 requires TLS encryption for secure communication between management services and the BeeGFS CLI tool (beegfs). Choose one of three configuration methods: use certificates signed by a trusted Certificate Authority, create your own local CA to sign certificates for your BeeGFS cluster, or disable TLS entirely for troubleshooting (not recommended for production).

Configure TLS encryption for the management services on your file nodes using the Configure TLS Encryption for BeeGFS 8 guide.

Client TLS configuration will be completed later after upgrading the BeeGFS clients. At this stage, only configure TLS for the management services and beegfs-tool on the file nodes.

Update management service configuration

Prepare the BeeGFS 8 management service configuration file by manually transferring settings from your BeeGFS 7 configuration file into the /mnt/mgmt_tgt_mgmt01/mgmt_config/beegfs-mgmtd.toml file.

On the management node with the management target mounted, reference the /mnt/mgmt_tgt_mgmt01/mgmt_config/beegfs-mgmtd.conf management service file for BeeGFS 7 and transfer all settings to the /mnt/mgmt_tgt_mgmt01/mgmt_config/beegfs-mgmtd.toml file. For a basic setup, your beegfs-mgmtd.toml may look like the following:

beemsg-port = 8008
grpc-port = 8010
log-level = "info"
node-offline-timeout = "900s"
quota-enable = false
auth-disable = false
auth-file = "/etc/beegfs/<mgmt_service_ip>_connAuthFile"
db-file = "/mnt/mgmt_tgt_mgmt01/data/mgmtd.sqlite"
license-disable = false
license-cert-file = "/etc/beegfs/license.pem"
tls-disable = false
tls-cert-file = "/etc/beegfs/mgmtd_tls_cert.pem"
tls-key-file = "/etc/beegfs/mgmtd_tls_key.pem"
interfaces = ['i1b:mgmt_1', 'i2b:mgmt_2']

Adjust all paths as necessary to match your environment and TLS configuration.

On each file node running management services, modify your systemd service file to point to the new configuration file location.

sudo sed -i 's|ExecStart=.*|ExecStart=nice -n -3 /opt/beegfs/sbin/beegfs-mgmtd --config-file /mnt/mgmt_tgt_mgmt01/mgmt_config/beegfs-mgmtd.toml|' /etc/systemd/system/beegfs-mgmtd.service

Reload systemd:
```
systemctl daemon-reload
```

For each file node running management services, open port 8010 for the management service's gRPC communication.

Add TCP port 8010 to the beegfs zone:

sudo firewall-cmd --zone=beegfs --permanent --add-port=8010/tcp

Reload the firewall to apply the change:
```
sudo firewall-cmd --reload
```

Update the BeeGFS monitor script

The Pacemaker beegfs-monitor Open Cluster Framework (OCF) script requires updates to support the new TOML configuration format and systemd service management. Update the script on one node in the cluster, then copy the updated script to all other nodes.

Create a backup of the current script:

cp /usr/lib/ocf/resource.d/eseries/beegfs-monitor /usr/lib/ocf/resource.d/eseries/beegfs-monitor.bak.$(date +%F)

Update the management configuration file path from .conf to .toml:

sed -i 's|mgmt_config/beegfs-mgmtd\.conf|mgmt_config/beegfs-mgmtd.toml|' /usr/lib/ocf/resource.d/eseries/beegfs-monitor

Alternatively, manually locate the following block in the script:

case $type in
  management)
    conf_path="${configuration_mount}/mgmt_config/beegfs-mgmtd.conf"
    ;;

And replace it with:

case $type in
  management)
    conf_path="${configuration_mount}/mgmt_config/beegfs-mgmtd.toml"
    ;;

Update the get_interfaces() and get_subnet_ips() functions to support TOML configuration:

Open the script in a text editor:

vi /usr/lib/ocf/resource.d/eseries/beegfs-monitor

Locate the two functions: get_interfaces() and get_subnet_ips().
Delete both entire functions, starting at get_interfaces() to the end of get_subnet_ips().

Copy and paste the following updated functions in their place:

# Return network communication interface name(s) from the BeeGFS resource's connInterfaceFile
get_interfaces() {
  # Determine BeeGFS service network IP interfaces.
  if [ "$type" = "management" ]; then
    interfaces_line=$(grep "^interfaces =" "$conf_path")
    interfaces_list=$(echo "$interfaces_line" | sed "s/.*= \[\(.*\)\]/\1/")
    interfaces=$(echo "$interfaces_list" | tr -d "'" | tr -d " " | tr ',' '\n')

    for entry in $interfaces; do
      echo "$entry" | cut -d ':' -f 1
    done
  else
    connInterfacesFile_path=$(grep "^connInterfacesFile" "$conf_path" | tr -d "[:space:]" | cut -f 2 -d "=")

    if [ -f "$connInterfacesFile_path" ]; then
      while read -r entry; do
        echo "$entry" | cut -f 1 -d ':'
      done < "$connInterfacesFile_path"
    fi
  fi
}

# Return list containing all the BeeGFS resource's usable IP addresses. *Note that these are filtered by the connNetFilterFile entries.
get_subnet_ips() {
  # Determine all possible BeeGFS service network IP addresses.
  if [ "$type" != "management" ]; then
    connNetFilterFile_path=$(grep "^connNetFilterFile" "$conf_path" | tr -d "[:space:]" | cut -f 2 -d "=")

    filter_ips=""
    if [ -n "$connNetFilterFile_path" ] && [ -e $connNetFilterFile_path ]; then
      while read -r filter; do
        filter_ips="$filter_ips $(get_ipv4_subnet_addresses $filter)"
      done < $connNetFilterFile_path
    fi

    echo "$filter_ips"
  fi
}

Save and exit the text editor.
Run the following command to check the script for syntax errors before proceeding. If no output appears, the script is syntactically correct.
```
bash -n /usr/lib/ocf/resource.d/eseries/beegfs-monitor
```

Copy the updated beegfs-monitor OCF script to all other nodes in the cluster:

scp /usr/lib/ocf/resource.d/eseries/beegfs-monitor user@node:/usr/lib/ocf/resource.d/eseries/beegfs-monitor

Bring the cluster back online

After you complete all the previous upgrade steps, bring the cluster back online by starting the BeeGFS services on all nodes.
```
pcs cluster start --all
```

Verify the beegfs-mgmtd service started successfully:

journalctl -xeu beegfs-mgmtd

Expected output includes lines such as:

Started Cluster Controlled beegfs-mgmtd.
Loaded config file from "/mnt/mgmt_tgt_mgmt01/mgmt_config/beegfs-mgmtd.toml"
Successfully initialized certificate verification library.
Successfully loaded license certificate: TMP-113489268
Opened database at "/mnt/mgmt_tgt_mgmt01/data/mgmtd.sqlite"
Listening for BeeGFS connections on [::]:8008
Serving gRPC requests on [::]:8010

If errors appear in the journal logs, review the management configuration file paths and verify that you transferred all values correctly from the BeeGFS 7 configuration file.

Run pcs status and verify the cluster is healthy and services are started on their preferred nodes.
Once the cluster is verified to be healthy, re-enable STONITH:
```
pcs property set stonith-enabled=true
```
Proceed to the next section to upgrade the BeeGFS clients and verify the cluster's health.

Upgrade BeeGFS clients

After upgrading your cluster to BeeGFS 8, you must also upgrade all BeeGFS clients.

If you have not already stopped the BeeGFS client service, stop it now:
```
systemctl stop beegfs-client
```
Add the BeeGFS 8 package repository for your Linux distribution. Instructions for adding the official BeeGFS repositories can be found at the BeeGFS download page. Otherwise, configure your local BeeGFS mirror repository accordingly.

Upgrade the BeeGFS client packages:

For Ubuntu/Debian:

apt-get update
apt-get install --only-upgrade beegfs-client

Configure TLS for the client. TLS is required to use the BeeGFS CLI from client nodes. Reference the Configure TLS for BeeGFS 8 clients section in the TLS deployment guide.

TLS is not required for the beegfs-client service to mount the BeeGFS file system. TLS is only required for using the BeeGFS CLI tools and other administrative services.
Start the BeeGFS client service:
```
systemctl start beegfs-client
```
If mounting BeeGFS fails, check the system journal for errors. If you see messages like "Unable to proceed without a working root metadata node" or "Remote I/O error", the BeeGFS 7 import may not have completed successfully. Contact NetApp support for assistance.
(Optional) Install and configure the new beegfs CLI tool on client nodes or other systems where you want to run BeeGFS administrative commands:
1. Install the beegfs-tools package:
  
  For Ubuntu/Debian:
  apt-get install beegfs-tools
2. Configure TLS for the beegfs tool. TLS is required to use the BeeGFS CLI from a client node. Reference the Configure TLS for BeeGFS 8 clients section in the TLS deployment guide.

Verify the upgrade

After finishing the upgrade to BeeGFS 8, run the following commands to verify the upgrade was successful.

Verify the root inode is owned by the same metadata node as before. This should happen automatically if you used the import-from-v7 functionality in the management service:
```
beegfs entry info /mnt/beegfs
```

Verify that all nodes and targets are online and in a healthy state:

beegfs health check

If the "Available Capacity" check warns that targets are low on free space, you can adjust the "capacity pool" thresholds defined in the beegfs-mgmtd.toml file so they are better suited to your environment.