Restore network connectivity for isolated nodes
PDF of this doc site
- Get started
Install and maintain appliance hardware
SG100 and SG1000 services appliances
- Prepare for installation (SG100 and SG1000)
SG6000 storage appliances
- Prepare for installation (SG6000)
- Configure hardware (SG6000)
SG5700 storage appliances
- Prepare for installation (SG5700)
- Configure hardware (SG5700)
SG5600 storage appliances
- Prepare for installation (SG5600)
- Configure hardware (SG5600)
- SG100 and SG1000 services appliances
Install and upgrade software
- Upgrade StorageGRID software
- Install Red Hat Enterprise Linux or CentOS
- Install Ubuntu or Debian
Perform system administration
- Manage security settings
- Manage Admin Nodes
- Manage Archive Nodes
Manage objects with ILM
- ILM and object lifecycle
- Create storage grades, storage pools, EC profiles, and regions
- Administer StorageGRID
- Use a tenant account
- S3 REST API supported operations and limitations
Monitor and maintain StorageGRID
Monitor and troubleshoot
- Troubleshoot a StorageGRID system
- Expand your grid
Recover and maintain
Grid node recovery procedures
- Recover from Storage Node failures
- Recover from Admin Node failures
- All grid node types: Replace Linux node
- Grid node decommission
Network maintenance procedures
- Grid node procedures
- Grid node recovery procedures
Review audit logs
- Audit messages and the object lifecycle
- Monitor and troubleshoot
Under certain circumstances, such as site- or grid-wide IP address changes, one or more groups of nodes might not be able to contact the rest of the grid.
In the Grid Manager (SUPPORT > Tools > Grid topology), if a node is gray, or if a node is blue with many of its services showing a status other than Running, you should check for node isolation.
Some of the consequences of having isolated nodes include the following:
If multiple nodes are isolated, you might not be able to sign in to or access the Grid Manager.
If multiple nodes are isolated, the storage usage and quota values shown on the Dashboard for the Tenant Manager might be out of date. The totals will be updated when network connectivity is restored.
To resolve the isolation issue, you run a command line utility on each isolated node or on one node in a group (all nodes in a subnet that does not contain the primary Admin Node) that is isolated from the grid. The utility provides the nodes with the IP address of a non-isolated node in the grid, which allows the isolated node or group of nodes to contact the entire grid again.
|If the multicast Domain Name System (mDNS) is disabled in the networks, the command line utility might have to be run on each isolated node.|
Access the node and check
/var/local/log/dynip.logfor isolation messages.
[2018-01-09T19:11:00.545] UpdateQueue - WARNING -- Possible isolation, no contact with other nodes. If this warning persists, manual action may be required.
If you are using the VMware console, it will contain a message that the node might be isolated.
On Linux deployments, isolation messages would appear in
If the isolation messages are recurring and persistent, run the following command:
<address>is the IP address of a remote node that is connected to the grid.
# /usr/sbin/add_node_ip.py 10.224.4.210 Retrieving local host information Validating remote node at address 10.224.4.210 Sending node IP hint for 10.224.4.210 to local node Local node found on remote node. Update complete.
Verify the following for each node that was previously isolated:
The node’s services have started.
The status of the Dynamic IP Service is “Running” after you run the
In the Grid Topology tree, the node no longer appears disconnected from the rest of the grid.
If running the