StorageGRID software logs

You can use StorageGRID logs to troubleshoot issues.

General StorageGRID logs

File name Notes Found on
/var/local/log/bycast.log The file bycast.log is the primary StorageGRID troubleshooting file.

The file bycast-err.log contains a subset of bycast.log (messages with severity ERROR and CRITICAL). CRITICAL messages are also displayed in the system. Select Support > Tools > Grid Topology. Then select Site > Node > SSM > Events.

All nodes
/var/local/log/bycast-err.log
/var/local/core/ Contains any core dump files created if the program terminates abnormally. Possible causes include assertion failures, violations, or thread timeouts.
Note: The file /var/local/core/kexec_cmd usually exists on appliance nodes and does not indicate an error.

Server Manager logs

File name Notes Found on
/var/local/log/servermanager.log Log file for the Server Manager application running on the server. All nodes
/var/local/log/GridstatBackend.errlog Log file for the Server Manager GUI backend application.
/var/local/log/gridstat.errlog Log file for the Server Manager GUI.

Logs for StorageGRID services

File name Notes Found on
/var/local/log/acct.errlog   Storage Nodes running the ADC service
/var/local/log/adc.errlog Contains the Standard Error (stderr) stream of the corresponding services. There is one log file per service. These files are generally empty unless there are problems with the service. Storage Nodes running the ADC service
/var/local/log/ams.errlog   Admin Nodes
/var/local/log/arc.errlog   Archive Nodes
/var/local/log/cassandra/system.log Information for the metadata store (Cassandra database) that can be used if problems occur when adding new Storage Nodes, or if the nodetool repair task stalls. Storage Nodes
/var/local/log/cassandra-reaper.log Information for the Cassandra Reaper service, which performs repairs of the data in the Cassandra database. Storage Nodes
/var/local/log/cassandra-reaper.errlog Error information for the Cassandra Reaper service. Storage Nodes
/var/local/log/chunk.errlog   Storage Nodes
/var/local/log/clb.errlog Error information for the CLB service.
Note: The CLB service is deprecated.
Gateway Nodes
/var/local/log/cmn.errlog   Admin Nodes
/var/local/log/cms.errlog This log file might be present on systems that have been upgraded from an older version of StorageGRID. It contains legacy information. Storage Nodes
/var/local/log/cts.errlog This log file is only created if the Target Type is Cloud Tiering - Simple Storage Service (S3). Archive Nodes
/var/local/log/dds.errlog   Storage Nodes
/var/local/log/dmv.errlog   Storage Nodes
/var/local/log/dynip* Contains logs related to the dynip service, which monitors the grid for dynamic IP changes and updates local configuration. All nodes
/var/local/log/grafana.log The log associated with the Grafana service, which is used for metrics visualization in the Grid Manager. Admin Nodes
/var/local/log/hagroups.log The log associated with high availability groups. Admin Nodes and Gateway Nodes
/var/local/log/hagroups_events.log Tracks state changes, such as transition from BACKUP to MASTER or FAULT. Admin Nodes and Gateway Nodes
/var/local/log/idnt.errlog   Storage Nodes running the ADC service
/var/local/log/jaeger.log The log associated with the jaeger service, which is used for trace collection. All nodes
/var/local/log/kstn.errlog   Storage Nodes running the ADC service
/var/local/log/ldr.errlog   Storage Nodes
/var/local/log/miscd/*.log Contains logs for the MISCd service (Information Service Control Daemon), which provides an interface for querying and managing services on other nodes and for managing environmental configurations on the node such as querying the state of services running on other nodes. All nodes
/var/local/log/nginx/*.log Contains logs for the nginx service, which acts as an authentication and secure communication mechanism for various grid services (such as Prometheus and Dynip) to be able to talk to services on other nodes over HTTPS APIs. All nodes
/var/local/log/nginx-gw/*.log Contains logs for the restricted admin ports on Admin Nodes and for the Load Balancer service, which provides load balancing of S3 and Swift traffic from clients to Storage Nodes. Admin Nodes and Gateway Nodes
/var/local/log/persistence* Contains logs for the Persistence service, which manages files on the root disk that need to persist across a reboot. All nodes
/var/local/log/prometheus.log

For all nodes, contains the node exporter service log and the ade-exporter metrics service log.

For Admin Nodes, also contains logs for the Prometheus and Alert Manager services.

All nodes
/var/local/log/raft.log Contains the output of the library used by the RSM service for the Raft protocol. Storage Nodes with RSM service
/var/local/log/rms.errlog Contains logs for the Replicated State Machine Service (RSM) service, which is used for S3 platform services. Storage Nodes with RSM service
/var/local/log/ssm.errlog   All nodes
/var/local/log/update-s3vs-domains.log Contains logs related to processing updates for the S3 virtual hosted domain names configuration.

See the instructions for implementing S3 client applications.

Admin and Gateway Nodes
/var/local/log/update-snmp-firewall.* Contain logs related to the firewall ports being managed for SNMP. All nodes
/var/local/log/update-sysl.log Contains logs related to changes made to the system syslog configuration. All nodes
/var/local/log/update-traffic-classes.log Contains logs related to changes to the traffic classifiers configuration. Admin and Gateway Nodes
/var/local/log/update-utcn.log Contains logs related to Untrusted Client Network mode on this node. All nodes

NMS logs

File name Notes Found on
/var/local/log/nms.log
  • Captures notifications from the Grid Manager and the Tenant Manager.
  • Captures events related to the operation of the NMS service, for example, alarm processing, email notifications, and configuration changes.
  • Contains XML bundle updates resulting from configuration changes made in the system.
  • Contains error messages related to the attribute downsampling done once a day.
  • Contains Java web server error messages, for example, page generation errors and HTTP Status 500 errors.
Admin Nodes
/var/local/log/nms.errlog

Contains error messages related to MySQL database upgrades.

Contains the Standard Error (stderr) stream of the corresponding services. There is one log file per service. These files are generally empty unless there are problems with the service.

Admin Nodes
/var/local/log/nms.requestlog Contains information about outgoing connections from the Management API to internal StorageGRID services. Admin Nodes