Request doc changes
Edit this page
Learn how to contributeRecover from Gateway Node failures
Contributors
You must complete a sequence of tasks in exact order to recover from a Gateway Node failure.
Related information
-
PDF of this doc site
-
Install and maintain appliance hardware
-
Install and upgrade software
-
Perform system administration
-
Monitor and maintain StorageGRID
-
Recover and maintain
-
Grid node recovery procedures
-
-
-
Collection of separate PDF docs
Creating your file...
This may take a few minutes. Thanks for your patience.
Your file is ready
- StorageGRID 11.6 Docs
- Release notes
-
Get started
-
Grid primer
- Overview
- Hybrid clouds with StorageGRID
- Architecture and network topology
-
Object management
-
How to use StorageGRID
- Explore Grid Manager
- Explore Tenant Manager
- Control StorageGRID access
- Manage tenants and client connections
- Configure network settings
- Configure security settings
- Configure system settings
- Use information lifecycle management
-
Monitor operations
- Perform maintenance
- Download Recovery Package
- Use support options
-
Networking guidelines
-
-
Install and maintain appliance hardware
-
SG100 and SG1000 services appliances
- Overview
- SG100 and SG1000 applications
- Installation and deployment overview
-
Prepare for installation (SG100 and SG1000)
-
Install hardware (SG100 and SG1000)
-
Configure StorageGRID connections (SG100 and SG1000)
-
Configure BMC interface (SG100 and SG1000)
- Optional: Enable node encryption
-
Deploy services appliance node
-
Troubleshoot hardware installation (SG100 and SG1000)
-
Maintain appliance
- Place appliance into maintenance mode
- Turn controller identify LED on and off
- Locate controller in data center
- Shut down the appliance
- Replace the services appliance
- Replace one or both power supplies in the services appliance
- Replace fan in services appliance
- Replace drive in services appliance
- Change link configuration of services appliance
- Change MTU setting
- Check DNS server configuration
- Monitor node encryption in maintenance mode (SG100 and SG1000)
- Clear key management server configuration
-
SG6000 storage appliances
-
Overview
- Installation and deployment overview
-
Prepare for installation (SG6000)
-
Install hardware (SG6000)
- Register hardware
- SG6060 and SG6060X: Install 60-drive shelves into cabinet or rack
- SG6060 and SG6060X: Install drives
- SGF6024: Install 24-drive shelves into cabinet or rack
- SG6000-CN: Install into cabinet or rack
- Cable appliance (SG6000)
- SG6060 and SG6060X: Cable optional expansion shelves
- Connect power cords and apply power (SG6000)
- View status indicators and buttons on SG6000-CN controller
- View boot-up status codes for SG6000 and SG6060X storage controllers
-
Configure hardware (SG6000)
- Deploy appliance Storage Node
- Monitor storage appliance installation
-
Automate appliance installation and configuration (SG6000)
- Overview of installation REST APIs
-
Troubleshoot hardware installation (SG6000)
-
Maintain SG6000 appliance
- Place appliance into maintenance mode
-
Upgrade SANtricity OS on storage controllers
- Upgrade drive firmware using SANtricity System Manager
- Add expansion shelf to deployed SG6060
- Turn controller identify LED on and off
- Locate controller in data center
- Replace storage controller
- Replace hardware components in storage controller shelf
- Replace hardware components in optional 60-drive expansion shelf
- Shut down SG6000-CN controller
- Power on SG6000-CN controller and verify operation
- Replace SG6000-CN controller
- Replace one or both power supplies in the SG6000-CN controller
- Remove SG6000-CN controller from cabinet or rack
- Reinstall SG6000-CN controller into cabinet or rack
- Remove SG6000-CN controller cover
- Reinstall SG6000-CN controller cover
-
Replace Fibre Channel HBA in SG6000-CN controller
- Change link configuration of SG6000-CN controller
- Change MTU setting
- Check DNS server configuration
- Monitor node encryption in maintenance mode (SG6000)
- Clear key management server configuration
-
-
SG5700 storage appliances
-
Overview
- Installation and deployment overview
-
Prepare for installation (SG5700)
-
Install hardware (SG5700)
-
Configure hardware (SG5700)
- Deploy appliance Storage Node
- Monitor storage appliance installation
-
Automate appliance installation and configuration (SG5700)
- Overview of installation REST APIs
-
Troubleshoot hardware installation (SG5700)
-
Maintain SG5700 appliance
- Place appliance into maintenance mode
-
Upgrade SANtricity OS on storage controller
- Upgrade drive firmware using SANtricity System Manager
- Replace E2800 controller
- Replace E5700SG controller
- Replace other hardware components
- Change link configuration of E5700SG controller
- Change MTU setting
- Check DNS server configuration
-
Monitor node encryption in maintenance mode (SG5700)
-
-
SG5600 storage appliances
-
Overview
- Installation and deployment overview
-
Prepare for installation (SG5600)
-
Install hardware (SG5600)
-
Configure hardware (SG5600)
- Deploy appliance Storage Node
- Monitor storage appliance installation
-
Automate appliance installation and configuration (SG5600)
- Overview of installation REST APIs
-
Troubleshoot hardware installation (SG5600)
-
Maintain SG5600 appliance
- Place appliance into maintenance mode
- Upgrade SANtricity OS on storage controllers using Grid Manager
- Upgrade SANtricity OS on E2700 controller using maintenance mode
- Upgrade drive firmware using SANtricity Storage Manager
- Replace E2700 controller
- Replace E5600SG controller
- Replace other hardware components
- Change link configuration of E5600SG controller
- Change MTU setting
- Check DNS server configuration
-
Monitor node encryption in maintenance mode (SG5600)
-
-
-
Install and upgrade software
-
Upgrade StorageGRID software
-
Install Red Hat Enterprise Linux or CentOS
- Overview
-
Plan and prepare for Red Hat or CentOS installation
-
Deploy virtual grid nodes (Red Hat or CentOS)
-
Configure the grid and complete installation (Red Hat or CentOS)
- Navigate to the Grid Manager
- Specify the StorageGRID license information
- Add sites
- Specify Grid Network subnets
- Approve pending grid nodes
- Specify Network Time Protocol server information
- Specify Domain Name System server information
- Specify the StorageGRID system passwords
- Review your configuration and complete installation
- Post-installation guidelines
- Automate the installation (Red Hat or CentOS)
- Overview of the installation REST API
- Where to go next
- Troubleshoot installation issues
- Example /etc/sysconfig/network-scripts
-
Install Ubuntu or Debian
- Overview
-
Plan and prepare for Ubuntu or Debian installation
-
Deploy virtual grid nodes (Ubuntu or Debian)
-
Configure grid and complete installation (Ubuntu or Debian)
- Navigate to the Grid Manager
- Specify the StorageGRID license information
- Add sites
- Specify Grid Network subnets
- Approve pending grid nodes
- Specify Network Time Protocol server information
- Specify Domain Name System server information
- Specify the StorageGRID system passwords
- Review your configuration and complete installation
- Post-installation guidelines
- Automate the installation (Ubuntu or Debian)
- Overview of the installation REST API
- Where to go next
- Troubleshoot installation issues
- Example /etc/network/interfaces
-
Install VMware
- Overview
-
Plan and prepare for VMware installation
-
Deploy virtual machine grid nodes (VMware)
-
Configure the grid and complete installation (VMware)
- Navigate to the Grid Manager
- Specify the StorageGRID license information
- Add sites
- Specify Grid Network subnets
- Approve pending grid nodes
- Specify Network Time Protocol server information
- Specify Domain Name System server information
- Specify the StorageGRID system passwords
- Review your configuration and complete installation
- Post-installation guidelines
- Automate the installation (VMware)
- Overview of the installation REST API
- Where to go next
- Troubleshoot installation issues
-
-
Perform system administration
-
Administer StorageGRID
- Overview
-
Get started with StorageGRID
-
Control access to StorageGRID
- Change the provisioning passphrase
- Change node console passwords
- Control access through firewalls
- Use identity federation
- Manage admin groups
- Deactivate features with the API
- Manage users
-
Use single sign-on (SSO)
- Configure single sign-on
- Requirements for using single sign-on
- Confirm federated users can sign in
- Use sandbox mode
- Create relying party trusts in AD FS
- Create enterprise applications in Azure
- Create SP connections in PingFederate
- Disable single sign-on
- Temporarily disable and reenable single sign-on for one Admin Node
-
Manage security settings
-
Manage certificates
-
Configure key management servers
- Overview
- Review StorageGRID encryption methods
- Overview of KMS and appliance configuration
- Considerations and requirements for using a key management server
- Considerations for changing the KMS for a site
- Configure StorageGRID as a client in the KMS
- Add a key management server (KMS)
- View KMS details
- View encrypted nodes
- Edit a key management server (KMS)
- Remove a key management server (KMS)
-
Manage proxy settings
-
Manage untrusted Client Networks
-
-
Manage tenants
-
Configure S3 and Swift client connections
-
Manage networks and connections
-
Use AutoSupport
-
Manage Storage Nodes
-
Manage Admin Nodes
-
Manage Archive Nodes
-
Migrate data into StorageGRID
-
Manage objects with ILM
- Overview
-
ILM and object lifecycle
- What an ILM policy is
- What an ILM rule is
-
Create storage grades, storage pools, EC profiles, and regions
- Create and assign storage grades
-
Configure storage pools
-
Use Cloud Storage Pools
- What a Cloud Storage Pool is
- Lifecycle of Cloud Storage Pool object
- When to use Cloud Storage Pools
- Considerations for Cloud Storage Pools
- Compare Cloud Storage Pools and CloudMirror replication
-
Create Cloud Storage Pool
- Edit Cloud Storage Pool
- Remove Cloud Storage Pool
- Troubleshoot Cloud Storage Pools
-
Configure Erasure Coding profiles
- Configure regions (optional and S3 only)
-
Create ILM rule
-
Create ILM policy
- Work with ILM rules and ILM policies
-
Use S3 Object Lock with ILM
-
Example ILM rules and policies
- Example 1: ILM rules and policy for object storage
- Example 2: ILM rules and policy for EC object size filtering
- Example 3: ILM rules and policy for better protection for image files
- Example 4: ILM rules and policy for S3 versioned objects
- Example 5: ILM rules and policy for Strict ingest behavior
- Example 6: Change ILM policy
- Example 7: Compliant ILM policy for S3 Object Lock
-
System hardening
-
Configure FabricPool
-
-
Use StorageGRID
-
Use a tenant account
- Overview
-
How to sign in and sign out
- Tenant Manager Dashboard
-
Tenant Management API
-
Manage system access
-
Manage S3 tenant accounts
-
Manage S3 platform services
-
Use S3
-
Overview
- Configure tenant accounts and connections
-
How StorageGRID implements S3 REST API
-
S3 REST API supported operations and limitations
-
StorageGRID S3 REST API operations
- GET Bucket consistency request
- PUT Bucket consistency request
- GET Bucket last access time request
- PUT Bucket last access time request
- DELETE Bucket metadata notification configuration request
- GET Bucket metadata notification configuration request
- PUT Bucket metadata notification configuration request
- GET Storage Usage request
-
Deprecated bucket requests for legacy Compliance
- Bucket and group access policies
- Configure security for REST API
-
Monitor and audit operations
-
Benefits of active, idle, and concurrent HTTP connections
-
-
Use Swift
-
-
Monitor and maintain StorageGRID
-
Monitor and troubleshoot
- Overview
- View the Dashboard
-
View the Nodes page
-
Information you should monitor regularly
-
Manage alerts and alarms
- Configure audit messages and log destinations
-
Use an external syslog server
-
Use SNMP monitoring
-
Collect additional StorageGRID data
- Use charts and graphs
- Use text reports
- Monitor PUT and GET performance
- Monitor object verification operations
- Monitor events
- Review audit messages
- Collect log files and system data
- Manually trigger an AutoSupport message
- View the Grid Topology tree
- Review support metrics
- Run diagnostics
- Create custom monitoring applications
-
Troubleshoot a StorageGRID system
- Troubleshooting overview
-
Troubleshoot object and storage issues
- Confirm object data locations
- Object store failures
- Verify object integrity
-
Troubleshoot lost and missing object data
- Troubleshoot the Low object data storage alert
- Troubleshoot Low read-only watermark override alerts
- Troubleshoot the Storage Status (SSTS) alarm
- Troubleshoot delivery of platform services messages (SMTT alarm)
- Troubleshoot metadata issues
- Troubleshoot certificate errors
- Troubleshoot Admin Node and user interface issues
- Troubleshoot network, hardware, and platform issues
- Troubleshoot external syslog server
- Alerts reference
- Commonly used Prometheus metrics
-
Alarms reference (legacy system)
-
Log files reference
-
Expand your grid
-
Recover and maintain
-
Overview
-
StorageGRID hotfix procedure
-
Grid node recovery procedures
- Warnings and considerations for grid node recovery
- Gather required materials for grid node recovery
- Select node recovery procedure
-
Recover from Storage Node failures
- Recover Storage Node down more than 15 days
-
Recover appliance Storage Node
- Prepare appliance Storage Node for reinstallation
- Start StorageGRID appliance installation
- Monitor StorageGRID appliance installation
- Select Start Recovery to configure appliance Storage Node
- Remount and reformat appliance storage volumes (“Manual Steps”)
- Restore object data to storage volume for appliance
- Check storage state after recovering appliance Storage Node
-
Recover from storage volume failure where system drive is intact
-
Recover from system drive failure
-
Recover from Admin Node failures
-
Recover from primary Admin Node failures
- Copy audit logs from failed primary Admin Node
- Replace primary Admin Node
- Configure replacement primary Admin Node
- Restore audit log on recovered primary Admin Node
- Reset preferred sender on recovered primary Admin Node
- Restore Admin Node database when recovering primary Admin Node
- Restore Prometheus metrics when recovering primary Admin Node
-
Recover from non-primary Admin Node failures
- Copy audit logs from failed non-primary Admin Node
- Replace non-primary Admin Node
- Select Start Recovery to configure non-primary Admin Node
- Restore audit log on recovered non-primary Admin Node
- Reset preferred sender on recovered non-primary Admin Node
- Restore Admin Node database when recovering non-primary Admin Node
- Restore Prometheus metrics when recovering non-primary Admin Node
-
-
Recover from Gateway Node failures
-
Recover from Archive Node failures
- All grid node types: Replace VMware node
-
All grid node types: Replace Linux node
-
Replace failed node with services appliance
- How site recovery is performed by technical support
-
Decommission procedure
-
Grid node decommission
-
Prepare to decommission grid nodes
- Gather required materials
- Access Decommission Nodes page
- Decommission disconnected grid nodes
- Decommission connected grid nodes
- Pause and resume decommission process for Storage Nodes
- Troubleshoot node decommissioning
-
-
Site decommission
-
-
Network maintenance procedures
-
Host-level and middleware procedures
-
Grid node procedures
- View Server Manager status and version
- View current status of all services
- Start Server Manager and all services
- Restart Server Manager and all services
- Stop Server Manager and all services
- View current status of service
- Stop service
- Place appliance into maintenance mode
- Force service to terminate
- Start or restart service
- Remove port remaps
- Remove port remaps on bare metal hosts
-
Reboot grid node
- Shut down grid node
- Power down host
-
Power off and on all nodes in grid
-
Use a DoNotStart file
-
Troubleshoot Server Manager
-
Appliance node cloning
-
-
Review audit logs
-
Overview
-
Audit log file and message formats
-
Audit messages and the object lifecycle
-
Audit messages
-
Audit message categories
-
Audit message reference
- APCT: Archive Purge from Cloud-Tier
- ARCB: Archive Object Retrieve Begin
- ARCE: Archive Object Retrieve End
- ARCT: Archive Retrieve from Cloud-Tier
- AREM: Archive Object Remove
- ASCE: Archive Object Store End
- ASCT: Archive Store Cloud-Tier
- ATCE: Archive Object Store Begin
- AVCC: Archive Validate Cloud-Tier Configuration
- CBRB: Object Receive Begin
- CBRE: Object Receive End
- CBSB: Object Send Begin
- CBSE: Object Send End
- ECMC: Missing Erasure Coded Data Fragment
- ECOC: Corrupt Erasure Coded Data Fragment
- ETAF: Security Authentication Failed
- GNRG: GNDS Registration
- GNUR: GNDS Unregistration
- GTED: Grid Task Ended
- GTST: Grid Task Started
- GTSU: Grid Task Submitted
- IDEL: ILM Initiated Delete
- LKCU: Overwritten Object Cleanup
- LLST: Location Lost
- MGAU: Management audit message
- OLST: System Detected Lost Object
- ORLM: Object Rules Met
- OVWR: Object Overwrite
- SADD: Security Audit Disable
- SADE: Security Audit Enable
- SCMT: Object Store Commit
- SDEL: S3 DELETE
- SGET: S3 GET
- SHEA: S3 HEAD
- SPOS: S3 POST
- SPUT: S3 PUT
- SREM: Object Store Remove
- SUPD: S3 Metadata Updated
- SVRF: Object Store Verify Fail
- SVRU: Object Store Verify Unknown
- SYSD: Node Stop
- SYST: Node Stopping
- SYSU: Node Start
- VLST: User Initiated Volume Lost
- WDEL: Swift DELETE
- WGET: Swift GET
- WHEA: Swift HEAD
- WPUT: Swift PUT
-
-
-
- Enable StorageGRID
- Other versions
- Legal notices