Skip to main content

Creating node configuration files

Contributors netapp-lhalbert

Node configuration files are small text files that provide the information the StorageGRID host service needs to start a node and connect it to the appropriate network and block storage resources. Node configuration files are used for virtual nodes and are not used for appliance nodes.

Where do I put the node configuration files?

You must place the configuration file for each StorageGRID node in the /etc/storagegrid/nodes directory on the host where the node will run. For example, if you plan to run one Admin Node, one Gateway Node, and one Storage Node on HostA, you must place three node configuration files in /etc/storagegrid/nodes on HostA. You can create the configuration files directly on each host using a text editor, such as vim or nano, or you can create them elsewhere and move them to each host.

What do I name the node configuration files?

The names of the configuration files are significant. The format is <node-name>.conf, where <node-name> is a name you assign to the node. This name appears in the StorageGRID Installer and is used for node maintenance operations, such as node migration.

Node names must follow these rules:

  • Must be unique

  • Must start with a letter

  • Can contain the characters A through Z and a through z

  • Can contain the numbers 0 through 9

  • Can contain one or more hyphens (-)

  • Must be no more than 32 characters, not including the .conf extension

Any files in /etc/storagegrid/nodes that do not follow these naming conventions will not be parsed by the host service.

If you have a multi-site topology planned for your grid, a typical node naming scheme might be:

<site>-<node type>-<node number>.conf

For example, you might use dc1-adm1.conf for the first Admin Node in Data Center 1, and dc2-sn3.conf for the third Storage Node in Data Center 2. However, you can use any scheme you like, as long as all node names follow the naming rules.

What is in a node configuration file?

The configuration files contain key/value pairs, with one key and one value per line. For each key/value pair, you must follow these rules:

  • The key and the value must be separated by an equal sign (=) and optional whitespace.

  • The keys can contain no spaces.

  • The values can contain embedded spaces.

  • Any leading or trailing whitespace is ignored.

Some keys are required for every node, while others are optional or only required for certain node types.

The table defines the acceptable values for all supported keys. In the middle column:

R: required
BP: best practice
O: optional

Key R, BP, or O? Value

ADMIN_IP

BP

Grid Network IPv4 address of the primary Admin Node for the grid to which this node belongs. Use the same value you specified for GRID_NETWORK_IP for the grid node with NODE_TYPE = VM_Admin_Node and ADMIN_ROLE = Primary. If you omit this parameter, the node attempts to discover a primary Admin Node using mDNS.

See “How grid nodes discover the primary Admin Node.”

Note: This value is ignored, and might be prohibited, on the primary Admin Node.

ADMIN_NETWORK_CONFIG

O

DHCP, STATIC, or DISABLED

ADMIN_NETWORK_ESL

O

Comma-separated list of subnets in CIDR notation to which this node should communicate via the Admin Network gateway.

Example: 172.16.0.0/21,172.17.0.0/21

ADMIN_NETWORK_GATEWAY

O (R)

IPv4 address of the local Admin Network gateway for this node. Must be on the subnet defined by ADMIN_NETWORK_IP and ADMIN_NETWORK_MASK. This value is ignored for DHCP-configured networks.

Note: This parameter is required if ADMIN_NETWORK_ESL is specified.

Examples:

  • 1.1.1.1

  • 10.224.4.81

ADMIN_NETWORK_IP

O

IPv4 address of this node on the Admin Network. This key is only required when ADMIN_NETWORK_CONFIG = STATIC; do not specify it for other values.

Examples:

  • 1.1.1.1

  • 10.224.4.81

ADMIN_NETWORK_MAC

O

The MAC address for the Admin Network interface in the container.

This field is optional. If omitted, a MAC address will be generated automatically.

Must be 6 pairs of hexadecimal digits separated by colons.

Example: b2:9c:02:c2:27:10

ADMIN_NETWORK_MASK

O

IPv4 netmask for this node, on the Admin Network. This key is only required when ADMIN_NETWORK_CONFIG = STATIC; do not specify it for other values.

Examples:

  • 255.255.255.0

  • 255.255.248.0

ADMIN_NETWORK_MTU

O

The maximum transmission unit (MTU) for this node on the Admin Network. Do not specify if ADMIN_NETWORK_CONFIG = DHCP. If specified, the value must be between 1280 and 9216. If omitted, 1500 is used.

If you want to use jumbo frames, set the MTU to a value suitable for jumbo frames, such as 9000. Otherwise, keep the default value.

IMPORTANT: The MTU value of the network must match the value configured on the switch port the node is connected to. Otherwise, network performance issues or packet loss might occur.

Examples:

  • 1500

  • 8192

ADMIN_NETWORK_TARGET

BP

Name of the host device that you will use for Admin Network access by the StorageGRID node. Only network interface names are supported. Typically, you use a different interface name than what was specified for GRID_NETWORK_TARGET or CLIENT_NETWORK_TARGET.

Note: Do not use bond or bridge devices as the network target. Either configure a VLAN (or other virtual interface) on top of the bond device, or use a bridge and virtual Ethernet (veth) pair.

Best practice: Specify a value even if this node will not initially have an Admin Network IP address. Then you can add an Admin Network IP address later, without having to reconfigure the node on the host.

Examples:

  • bond0.1002

  • ens256

ADMIN_NETWORK_TARGET_TYPE

O

Interface

(This is the only supported value.)

ADMIN_NETWORK_TARGET_TYPE_INTERFACE_CLONE_MAC

BP

True or False

Set the key to "true" to cause the StorageGRID container use the MAC address of the host host target interface on the Admin Network.

Best practice: In networks where promiscuous mode would be required, use the ADMIN_NETWORK_TARGET_TYPE_INTERFACE_CLONE_MAC key instead.

For more details on MAC cloning, see the considerations and recommendations for MAC address cloning.

ADMIN_ROLE

R

Primary or Non-Primary

This key is only required when NODE_TYPE = VM_Admin_Node; do not specify it for other node types.

BLOCK_DEVICE_AUDIT_LOGS

R

Path and name of the block device special file this node will use for persistent storage of audit logs. This key is only required for nodes with NODE_TYPE = VM_Admin_Node; do not specify it for other node types.

Examples:

  • /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0

  • /dev/disk/by-id/wwn-0x600a09800059d6df000060d757b475fd

  • /dev/mapper/sgws-adm1-audit-logs

BLOCK_DEVICE_RANGEDB_00

BLOCK_DEVICE_RANGEDB_01

BLOCK_DEVICE_RANGEDB_02

BLOCK_DEVICE_RANGEDB_03

BLOCK_DEVICE_RANGEDB_04

BLOCK_DEVICE_RANGEDB_05

BLOCK_DEVICE_RANGEDB_06

BLOCK_DEVICE_RANGEDB_07

BLOCK_DEVICE_RANGEDB_08

BLOCK_DEVICE_RANGEDB_09

BLOCK_DEVICE_RANGEDB_10

BLOCK_DEVICE_RANGEDB_11

BLOCK_DEVICE_RANGEDB_12

BLOCK_DEVICE_RANGEDB_13

BLOCK_DEVICE_RANGEDB_14

BLOCK_DEVICE_RANGEDB_15

R

Path and name of the block device special file this node will use for persistent object storage. This key is only required for nodes with NODE_TYPE = VM_Storage_Node; do not specify it for other node types.

Only BLOCK_DEVICE_RANGEDB_00 is required; the rest are optional. The block device specified for BLOCK_DEVICE_RANGEDB_00 must be at least 4 TB; the others can be smaller.

Note: Do not leave gaps. If you specify BLOCK_DEVICE_RANGEDB_05, you must also specify BLOCK_DEVICE_RANGEDB_04.

Examples:

  • /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0

  • /dev/disk/by-id/wwn-0x600a09800059d6df000060d757b475fd

  • /dev/mapper/sgws-sn1-rangedb-0

BLOCK_DEVICE_TABLES

R

Path and name of the block device special file this node will use for persistent storage of database tables. This key is only required for nodes with NODE_TYPE = VM_Admin_Node; do not specify it for other node types.

Examples:

  • /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0

  • /dev/disk/by-id/wwn-0x600a09800059d6df000060d757b475fd

  • /dev/mapper/sgws-adm1-tables

BLOCK_DEVICE_VAR_LOCAL

R

Path and name of the block device special file this node will use for its /var/local persistent storage.

Examples:

  • /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0

  • /dev/disk/by-id/wwn-0x600a09800059d6df000060d757b475fd

  • /dev/mapper/sgws-sn1-var-local

CLIENT_NETWORK_CONFIG

O

DHCP, STATIC, or DISABLED

CLIENT_NETWORK_GATEWAY

O

IPv4 address of the local Client Network gateway for this node, which must be on the subnet defined by CLIENT_NETWORK_IP and CLIENT_NETWORK_MASK. This value is ignored for DHCP-configured networks.

Examples:

  • 1.1.1.1

  • 10.224.4.81

CLIENT_NETWORK_IP

O

IPv4 address of this node on the Client Network. This key is only required when CLIENT_NETWORK_CONFIG = STATIC; do not specify it for other values.

Examples:

  • 1.1.1.1

  • 10.224.4.81

CLIENT_NETWORK_MAC

O

The MAC address for the Client Network interface in the container.

This field is optional. If omitted, a MAC address will be generated automatically.

Must be 6 pairs of hexadecimal digits separated by colons.

Example: b2:9c:02:c2:27:20

CLIENT_NETWORK_MASK

O

IPv4 netmask for this node on the Client Network. This key is only required when CLIENT_NETWORK_CONFIG = STATIC; do not specify it for other values.

Examples:

  • 255.255.255.0

  • 255.255.248.0

CLIENT_NETWORK_MTU

O

The maximum transmission unit (MTU) for this node on the Client Network. Do not specify if CLIENT_NETWORK_CONFIG = DHCP. If specified, the value must be between 1280 and 9216. If omitted, 1500 is used.

If you want to use jumbo frames, set the MTU to a value suitable for jumbo frames, such as 9000. Otherwise, keep the default value.

IMPORTANT: The MTU value of the network must match the value configured on the switch port the node is connected to. Otherwise, network performance issues or packet loss might occur.

Examples:

  • 1500

  • 8192

CLIENT_NETWORK_TARGET

BP

Name of the host device that you will use for Client Network access by the StorageGRID node. Only network interface names are supported. Typically, you use a different interface name than what was specified for GRID_NETWORK_TARGET or ADMIN_NETWORK_TARGET.

Note: Do not use bond or bridge devices as the network target. Either configure a VLAN (or other virtual interface) on top of the bond device, or use a bridge and virtual Ethernet (veth) pair.

Best practice: Specify a value even if this node will not initially have a Client Network IP address. Then you can add a Client Network IP address later, without having to reconfigure the node on the host.

Examples:

  • bond0.1003

  • ens423

CLIENT_NETWORK_TARGET_TYPE

O

Interface

(This is only supported value.)

CLIENT_NETWORK_TARGET_TYPE_INTERFACE_CLONE_MAC

BP

True or False

Set the key to "true" to cause the StorageGRID container to use the MAC address of the host target interface on the Client Network.

Best practice: In networks where promiscuous mode would be required, use the CLIENT_NETWORK_TARGET_TYPE_INTERFACE_CLONE_MAC key instead.

For more details on MAC cloning, see the considerations and recommendations for MAC address cloning.

GRID_NETWORK_CONFIG

BP

STATIC or DHCP

(Defaults to STATIC if not specified.)

GRID_NETWORK_GATEWAY

R

IPv4 address of the local Grid Network gateway for this node, which must be on the subnet defined by GRID_NETWORK_IP and GRID_NETWORK_MASK. This value is ignored for DHCP-configured networks.

If the Grid Network is a single subnet with no gateway, use either the standard gateway address for the subnet (X.Y.Z.1) or this node's GRID_NETWORK_IP value; either value will simplify potential future Grid Network expansions.

GRID_NETWORK_IP

R

IPv4 address of this node on the Grid Network. This key is only required when GRID_NETWORK_CONFIG = STATIC; do not specify it for other values.

Examples:

  • 1.1.1.1

  • 10.224.4.81

GRID_NETWORK_MAC

O

The MAC address for the Grid Network interface in the container.

This field is optional. If omitted, a MAC address will be generated automatically.

Must be 6 pairs of hexadecimal digits separated by colons.

Example: b2:9c:02:c2:27:30

GRID_NETWORK_MASK

O

IPv4 netmask for this node on the Grid Network. This key is only required when GRID_NETWORK_CONFIG = STATIC; do not specify it for other values.

Examples:

  • 255.255.255.0

  • 255.255.248.0

GRID_NETWORK_MTU

O

The maximum transmission unit (MTU) for this node on the Grid Network. Do not specify if GRID_NETWORK_CONFIG = DHCP. If specified, the value must be between 1280 and 9216. If omitted, 1500 is used.

If you want to use jumbo frames, set the MTU to a value suitable for jumbo frames, such as 9000. Otherwise, keep the default value.

IMPORTANT: The MTU value of the network must match the value configured on the switch port the node is connected to. Otherwise, network performance issues or packet loss might occur.

IMPORTANT: For the best network performance, all nodes should be configured with similar MTU values on their Grid Network interfaces. The Grid Network MTU mismatch alert is triggered if there is a significant difference in MTU settings for the Grid Network on individual nodes. The MTU values do not have to be the same for all network types.

Examples:

  • 1500

  • 8192

GRID_NETWORK_TARGET

R

Name of the host device that you will use for Grid Network access by the StorageGRID node. Only network interface names are supported. Typically, you use a different interface name than what was specified for ADMIN_NETWORK_TARGET or CLIENT_NETWORK_TARGET.

Note: Do not use bond or bridge devices as the network target. Either configure a VLAN (or other virtual interface) on top of the bond device, or use a bridge and virtual Ethernet (veth) pair.

Examples:

  • bond0.1001

  • ens192

GRID_NETWORK_TARGET_TYPE

O

Interface

(This is the only supported value.)

GRID_NETWORK_TARGET_TYPE_INTERFACE_CLONE_MAC

BP

True or False

Set the value of the key to "true" to cause the StorageGRID container to use the MAC address of the host target interface on the Grid Network.

Best practice: In networks where promiscuous mode would be required, use the GRID_NETWORK_TARGET_TYPE_INTERFACE_CLONE_MAC key instead.

For more details on MAC cloning, see the considerations and recommendations for MAC address cloning.

MAXIMUM_RAM

O

The maximum amount of RAM that this node is allowed to consume. If this key is omitted, the node has no memory restrictions. When setting this field for a production-level node, specify a value that is at least 24 GB and 16 to 32 GB less than the total system RAM.

Note: The RAM value affects a node's actual metadata reserved space. See the instructions for administering StorageGRID for a description of what Metadata Reserved Space is.

The format for this field is <number><unit>, where <unit> can be b, k, m, or g.

Examples:

24g

38654705664b

Note: If you want to use this option, you must enable kernel support for memory cgroups.

NODE_TYPE

R

Type of node:

  • VM_Admin_Node

  • VM_Storage_Node

  • VM_Archive_Node

  • VM_API_Gateway

PORT_REMAP

O

Remaps any port used by a node for internal grid node communications or external communications. Remapping ports is necessary if enterprise networking policies restrict one or more ports used by StorageGRID, as described in “Internal grid node communications” or “External communications.”

IMPORTANT: Do not remap the ports you are planning to use to configure load balancer endpoints.

Note: If only PORT_REMAP is set, the mapping that you specify is used for both inbound and outbound communications. If PORT_REMAP_INBOUND is also specified, PORT_REMAP applies only to outbound communications.

The format used is: <network type>/<protocol>/<default port used by grid node>/<new port>, where network type is grid, admin, or client, and protocol is tcp or udp.

For example:

PORT_REMAP = client/tcp/18082/443

PORT_REMAP_INBOUND

O

Remaps inbound communications to the specified port. If you specify PORT_REMAP_INBOUND but do not specify a value for PORT_REMAP, outbound communications for the port are unchanged.

IMPORTANT: Do not remap the ports you are planning to use to configure load balancer endpoints.

The format used is: <network type>/<protocol:>/<remapped port >/<default port used by grid node>, where network type is grid, admin, or client, and protocol is tcp or udp.

For example:

PORT_REMAP_INBOUND = grid/tcp/3022/22