INFORMATIONAL

This message occurs when a node initiates an automatic giveback to its partner following a takeover that was due to a panic on the partner.

(None).

Failover monitor: Automatic giveback was initiated following a takeover that was caused by a panic on the partner.

(None).

INFORMATIONAL

This message occurs when a node initiates an automatic giveback to its partner following a takeover that was due to a reboot of the partner.

(None).

Failover monitor: Automatic giveback was initiated following a takeover that was caused by the partner reboot.

(None).

EMERGENCY

This event is issued when auto giveback is disabled due to a ping-pong situation (autoGB followed by t/o followed by autoGB…​)

Examine the logs and/or console output from the partner node. Resolve the issue which prevents the node from staying up.

Failover monitor: Automatic giveback is being disabled due to exceeding %d attempts in %d minutes.

attempts (INT): Number of automatic givebacks attempted
minutes (INT): Time period where the automatic givebacks were attempted

INFORMATIONAL

This message occurs when an automatic giveback is delayed because 'Delay Before Auto Giveback' is set to a non-zero number. If you want to eliminate the delay before automatic giveback occurs, you can use the command "storage failover modify -delay-seconds" to set it to zero.

(None).

Failover monitor: Automatic giveback was delayed by %d seconds due to a non-zero value of 'Delay Before Auto Giveback'.

seconds (INT): Number of seconds by which automatic giveback was delayed.

NOTICE

This event is generated when we release the disk reservations in preparation for an automatic giveback.

(None).

Failover monitor: Releasing disk reservations in preparation for an automatic giveback

(None).

INFORMATIONAL

This event is issued when an automatic giveback is initiated.

(None).

Failover monitor: Automatic giveback started

(None).

ERROR

This event is issued when one or more subsystems have vetoed the automatic giveback.

Terminate the long-running jobs and auto giveback will be successful next time it is attempted.

Failover monitor: Automatic giveback has been deferred due to long running operations

(None).

NOTICE

This event is issued when the failover monitor cancels a pending takeover.

(None).

Failover monitor: pending takeover cancelled

(None).

ERROR

This message occurs when the failover monitor determines that an error was observed in the partner's mailbox.

Correct the issues preventing the node from accessing the partner's mailbox disks. Check for cabling, host bus adapter (HBA), storage controller or drive/LUN issues. You can also use Multipathing to provide a redundant connection to the mailbox disk.

Failover monitor: partner mailbox error detected.

(None).

NOTICE

This event is issued when the failover monitor has determined that an error in the backup's mailbox has been fixed.

(None).

Failover monitor: backup mailbox OK

(None).

INFORMATIONAL

This event is generated when we want to initiate an automatic giveback and we're checking for long running operations which might veto our plans.

(None).

Failover monitor: Checking for long running operations in preparation for an automatic giveback.

(None).

NOTICE

This event is issued when the failover monitor cancels a pending takeover issued through a CLI.

(None).

Failover monitor: takeover cannot be performed because of reason (%s)

reason (STRING): Reason why takeover cannot occur

INFORMATIONAL

This event is issued when we detect an altered firmware status update from the partner.

(None).

Failover monitor: partner %s

reason (STRING): Partner status

ALERT

This event occurs when the failover monitor detects that the giveback process is hung.

Collect the resulting core file and provide it to Customer Support.

Failover monitor: giveback process is hung ('%s')

moduleName (STRING): The name of the module that the hang occurred in.

ALERT

This event is called when the giveback retry count has been exceeded. This situation exists when the system is unable either to takeover or giveback. It may be due to either a hardware bug (e.g., the disk subsystem is hung) or a software bug.

Examine the logs and determine why the giveback is failing. Correct that problem and retry the giveback.

Failover monitor: giveback has exceeded max retry count

retries (INT): Number of retries attempted.

NOTICE

This message occurs when the system starts a negotiated takeover of its partner, and requests a graceful shutdown of the partner.

(None).

Negotiated failover: starting takeover and shutdown of partner (%s), will take over in at most %d secs. Reason: %s.

partnerName (STRING): Name of partner node.
maxTakeoverTime (INT): Maximum amount of time to wait for the partner to shut down before starting takeover, in seconds.
partnerReason (STRING): Reason for the initiation of the takeover.

NOTICE

This event is called when the system clears a request for takeover by its partner.

(None).

Negotiated failover: clearing partner takeover request

(None).

ERROR

This event is called when the system has been asked to shutdown by its partner as the result of the negotiated failover mechanism, but the system can not shut down due to a specific reason.

Using the information provided in the message, determine why shutdown cannot be invoked. Resolve that problem and retry the takeover request.

Negotiated failover: delaying shutdown due to %s

why (STRING): Indicates the cause of the delay.

ERROR

This event is called when negotiated failover is disabled for a particular module.

Examine previous messages for failures related to the type of NFO.

Negotiated failover: disabling negotiated failover for module %s

mod (STRING): Negotiated failover module or type.

ERROR

This event is called when negotiated failover is disabled due to Shelf Count message version mismatch.

Upgrade both nodes to the same release.

Negotiated failover: disabling negotiated failover due to version mis-match.

(None).

NOTICE

This event is called when negotiated failover is enabled for a particular module.

(None).

Negotiated failover: enabling negotiated failover for module %s

mod (STRING): NFO module (or type)

NOTICE

This event is called when the maximum time the system will wait for the partner to shutdown gracefully has passed. At this point the system takes over by force.

(None).

Negotiated failover: partner graceful shutdown appears hung, taking over

(None).

ERROR

This event is called when a module which is participating in negotiated failover changes from "unimpaired" to "impaired" or vice versa.

Check the state of the module listed in the message.

Negotiated failover: module %s is now %s

mod (STRING): Type of negotiated failover
impairment (STRING): Either unimpared or impaired

NOTICE

This event is called when the system sees that the partner has finished shutting down gracefully during negotiated failover.

(None).

Negotiated failover: partner has shutdown

(None).

ERROR

This message occurs when the system rejects a request by its partner to take it over because the system is itself impaired.

Use the "storage failover show" command to determine why takeover is not possible. Resolve that problem and retry the takeover request.

Negotiated failover: rejecting takeover request by partner due to own impairment.

(None).

ERROR

This event is called when the system rejects a request by its partner to take it over because the system has itself recently requested takeover by its partner. Rejecting this request prevents each system trying to takeover its partner simultaneously. If the partner persists and the system doesn't become impaired the request will soon be granted.

Resolve any impairment issues reported in previous messages that would cause takeover. If takeover is requested by the operator, it should only be requested on one node.

Negotiated failover: rejecting takeover request by partner due to own recent takeover request.

(None).

NOTICE

This event is called when the system has been asked to shutdown by its partner as the result of the negotiated failover mechanism. The system responds by shutting down gracefully, shutting down services in an orderly manner.

(None).

Negotiated failover: starting graceful shutdown.

(None).

ERROR

This message occurs when the system is waiting for the partner to shutdown gracefully and failover is disabled, canceling the pending takeover.

Use the "storage failover modify -enabled true" command to reenable failover.

Negotiated failover: pending takeover canceled.

(None).

NOTICE

This event is issued when we detect that the partner node is not responsive.

(None).

Failover monitor: partner not responding

(None).

NOTICE

This event is issued when we detect that the partner node, which was previously not responsive, is now OK.

(None).

Failover monitor: partner ok

(None).

INFORMATIONAL

This event is generated when we release the disk reservations in preparation for a manual giveback.

(None).

Failover monitor: Releasing disk reservations in preparation for giveback

(None).

ALERT

This event is emitted when we detect that the partner sees more of our disk shelves than we do. In other words, it sees more shelves on its FCAL B loop than we see on our A loop. This is probably due to a cabling problem or a broken FCAL host adaptor. If "disk_shelf" negotiated failover is enabled, this condition should lead to a takeover by the partner if the partner is otherwise able to take us over.

Resolve cabling issues which are preventing both nodes from seeing the same disks.

Disk shelf count mismatch: partner sees more of our A shelves on its B loop (%d) than we do (%d).

bShelves (INT): Number of our shelves which the partner can see
aShelves (INT): Number of shelves which we can see

INFORMATIONAL

This event is issued when a state transition is detected. Typically, this indication means that the failover monitor is about to either takeover its partner or giveback to its partner. This can happen as the result of timers going of, operator command, or an indication from the partner that a fault has been detected.

(None).

Failover monitor: %s -→ %s

oldState (STRING): The old failover monitor state.
newState (STRING): The new failover monitor state.
elem (STRING): The name of the FSM element that has caused the state transition to occur. This value is dependent upon the FSM implementation.

ALERT

This event is issued when an automatic takeover is initiated after detecting that the partner boot process is hung trying to load the kernel.

Please capture console log of partner filer and contact Customer Support

Failover monitor: automatic takeover attempted after detecting that partner is hung loading kernel while booting

(None).

INFORMATIONAL

This message occurs when an operator-requested disaster recovery (DR) takeover is initiated.

(None).

Failover monitor: takeover attempted after "cf forcetakeover -d" command.

(None).

INFORMATIONAL

This event is issued when a filer takes over its partner while booting up in takeover mode.

(None).

Failover monitor: takeover resumption attempted after reboot

(None).

INFORMATIONAL

This message occurs when an operator-requested forced takeover is initiated.

(None).

Failover monitor: takeover attempted after "cf forcetakeover" or "storage failover takeover -option force" in advanced privilege.

(None).

ALERT

This event is issued when an automatic takeover is initiated after detecting that the partner has panicked due to a multi-disk failure

Please check the connectivity of the partner to it's disks and shelves and contact customer support.

Failover monitor: takeover attempted after multi-disk failure on partner

(None).

INFORMATIONAL

This message occurs when an operator-requested takeover is initiated with the "cf takeover -n" or "storage failover takeover -option allow-version-mismatch" command.

(None).

Failover monitor: takeover attempted after "cf takeover -n" or "storage failover takeover -option allow-version-mismatch" command.

(None).

INFORMATIONAL

This message occurs when an operator-requested takeover is initiated with the "storage failover takeover" command.

(None).

Failover monitor: takeover attempted after "storage failover takeover" command.

(None).

ALERT

This message occurs when a node detects no heartbeat from the partner, indicating that the partner is not functioning. The node will attempt an automatic takeover.

Contact NetApp technical support.

Failover monitor: Takeover initiated after no heartbeat was detected from the partner node.

(None).

INFORMATIONAL

This message occurs when an operator-requested takeover is initiated with the "storage failover takeover -option immediate" command, or when Kernel Cluster Services Connectivity, Liveness and Availability Monitor (CLAM) triggers a takeover after determining that the partner node is out of "CLAM quorom".

(None).

Failover monitor: %s attempted

reason (STRING): Reason a takeover of the partner was triggered. Possible values are "Operator initiated immediate takeover" and "CLAM initiated takeover".

INFORMATIONAL

This message occurs when one node in a high-availability (HA) pair initiates an automatic takeover after detecting that its partner node has halted.

(None).

Failover monitor: Node initiated automatic takeover after detecting that its partner node has halted.

(None).

INFORMATIONAL

This message occurs when one node in a High Availability (HA) pair initiates an automatic takeover after detecting that its partner node is rebooting.

(None).

Failover monitor: One node initiated automatic takeover after detecting that its partner node is rebooting.

(None).

NOTICE

This event is issued when an automatic takeover is initiated after detecting that the operator timer has expired. This may happen when the operator has failed to respond to a question or not entered a required command during boot.

(None).

Failover monitor: takeover attempted after operator timeout expired on partner

(None).

ALERT

This message occurs when an automatic takeover is initiated after detecting that the partner has panicked.

Attempt to find the panic string in the event logs by using the "event log show" command from the CLI, and then look up the string by using the Panic Message Analyzer tool on the NetApp support site: http://mysupport.netapp.com/NOW/cgi-bin/pmsg/. Contact NetApp technical support to confirm the analysis.

Failover monitor: takeover attempted after partner panic.

(None).

ALERT

This event is issued when an automatic takeover is initiated after detecting that the partner's power-on self-test has failed.

Please run hardware diagnostics on partner filer and contact Customer Support

Failover monitor: takeover attempted after partner POST failed

(None).

ALERT

This message occurs when an automatic takeover is initiated after detecting that the partner boot process is hung.

Capture console log of partner node and contact NetApp technical support.

Failover monitor: automatic takeover attempted after detecting that partner is hung in boot.

(None).

ALERT

This message occurs when an automatic takeover is initiated after detecting that the partner died very shortly after booting up.

Contact NetApp technical support.

Failover monitor: takeover attempted after partner went down shortly after booting up

(None).

ALERT

This message occurs when an automatic takeover is initiated by the local node after it detects that the partner node has panicked, and that the partner has not initiated a main memory core dump to disk in a reasonable amount of time.

Capture the console log of the partner node, and then contact NetApp technical support.

A takeover was attempted by the local node after sparecore timeout expired on the partner node.

(None).

ERROR

This message occurs when the failover monitor determines that takeover by the partner is disabled.

Find the reason for the error message (it is surrounded by parentheses). Based on that reason, the corrective action is one of the following: -allowed: Takeover is allowed; you do not need to take any action. -Controller failover (CFO) is not initialized: Make sure that the high-availability (HA) pair is set up correctly. Contact NetApp technical support if you need assistance. -Controller is in non-HA mode: Set the HA mode to "ha" by using the "storage failover modify -mode ha" command to activate HA functionality. -Takeover disabled: Use the "storage failover modify -enabled true" command to reenable HA functionality. -partner mailbox disks not accessible or invalid: Check connectivity to all disks by running the "run local storage show" command on each node, and then comparing the results. Resolve the differences in disks visible to both systems. Verify that the interconnect cables are properly cabled. Failover monitor version mismatch: Make sure that both the local and partner node are running the same version of Data ONTAP®. -Takeover disabled by partner: Use the "storage failover modify -enabled true" command to reenable HA functionality. -Takeover disabled by operator: Use the "storage failover modify -enabled true" command to reenable HA functionality. -NVRAM size mismatch: Make sure that the local NVRAM (nonvolatile random-access memory) size matches the partner node. -version mismatch: Make sure that both the local and partner nodes are running the same version of Data ONTAP. -interconnect error: Make sure that the interconnect link is connected and functioning. -partner booting: Wait for the partner node to complete its booting process, and then try takeover. -shelf too hot: Make sure that the disk shelf temperature is properly regulated. -partner is performing revert: Wait for the partner node to complete the revert process, and then try takeover. -revert is in progress: Wait for the local node to complete the revert process, and then try takeover. -partner is attempting takeover: Cannot perform a takeover operation while the partner node is attempting a takeover. -takeover is in progress: The local node is already taken over or is trying to take over the partner node. -partner halted in notakeover mode: The partner node was most likely halted using the "halt -f" command; reboot the partner node, and then try again. -unsynchronized log: Make sure that the interconnect link is connected and functioning. -unknown notakeover reasons: Contact NetApp technical support. -waiting for partner to recover: The partner has not booted completely after giveback; wait for the partner to come back up completely. -low memory: Contact NetApp technical support to upgrade. -local halt in progress: The local node is about to halt; try again after the reboot. -status of backup mailbox is uncertain: Check connectivity to all disks by running the "run local storage show" command on each node, and then comparing the results. Resolve the differences in disks visible to both systems. Verify that the interconnect cables are properly cabled. -automatic takeover disabled: Use the "storage failover takeover" command manually. -metrocluster disaster recovery operation is in progress: The local node is performing a MetroCluster(tm) disaster recovery operation; wait for it to finish, and then try again. -This node or partner node is in switchover state and the MetroCluster configuration option "node-object-limit" is off in the disaster recovery(DR) group of this node: Retry takeover after doing a switchback.

Failover monitor: takeover of %s by %s disabled (%s).

local (STRING): Name of the local node.
partner (STRING): Name of the partner node.
reason (STRING): Reason takeover is disabled.

NOTICE

This event is issued when the failover monitor determines that takeover by the partner has been enabled.

(None).

Failover monitor: takeover of %s by %s enabled

local (STRING): Name of local node
partner (STRING): Name of partner node

INFORMATIONAL

This event is issued as part of the takeover countdown processing in the FSM.

(None).

Failover monitor: takeover scheduled in %d seconds

secsTillTakeover (INT): Number of seconds until takeover occurs

NOTICE

This event is issued when we are delaying takeover due to status indications received from the partner.

(None).

Failover monitor: takeover delayed, partner %s

reason (STRING): Description of why takeover is being delayed
secsTillTakeover (INT): Number of seconds until takeover will be started

EMERGENCY

This message occurs when the failover monitor detects that the takeover process is hung. Subsequent to this event, the takeover node will panic.

Attempt to find the panic string in the event logs by using the "event log show" command from the CLI, and then look up the string by using the Panic Message Analyzer tool on the NetApp support site: http://mysupport.netapp.com/NOW/cgi-bin/pmsg/. Contact NetApp technical support to confirm the analysis.

Failover monitor: takeover process is hung ('%s').

moduleName (STRING): Name of the module that the hang occurred in.

ERROR

This message occurs when the failover monitor determines that takeover of the partner is disabled.

Find the reason for the error message (it is surrounded by parentheses). Based on that reason, the corrective action is one of the following: -allowed: Takeover is allowed; you do not need to take any action. -Controller failover (CFO) is not initialized: Make sure that the high-availability (HA) pair is set up correctly. Contact NetApp technical support for assistance. -Controller is in non-HA mode: Set HA mode to "ha" by using the "storage failover modify -mode ha" command to activate HA functionality. -HA takeover disabled: Use the "storage failover modify -enabled true" command to reenable HA functionality. -partner mailbox disks not accessible or invalid: Check connectivity to all disks by running the "storage show" nodeshell command on each node and comparing the results. Resolve the differences in disks visible to both systems. Verify that the interconnect cables are properly cabled. -failover monitor version mismatch: Make sure that both the local and partner node are running the same version of Data ONTAP®. -Takeover disabled by partner: Use the "storage failover modify -enabled true" command to reenable HA functionality. -Takeover disabled by operator: Use the "storage failover modify -enabled true" command to reenable HA functionality. -NVRAM size mismatch: Make sure that the local NVRAM (nonvolatile random-access memory) size matches the partner node. -version mismatch: Make sure that both the local and partner nodes are running the same version of Data ONTAP. -interconnect error: Make sure that the interconnect link is connected and functioning. -partner booting: Wait for the partner node to complete its booting process, and then try takeover. -shelf too hot: Make sure that the disk shelf temperature is properly regulated. -partner is performing revert: Wait for the partner node to complete the revert process, and then try takeover. -revert is in progress: Wait for the local node to complete the revert process, and then try takeover. -partner is attempting takeover: Cannot perform a takeover operation while the partner node is attempting a takeover. -takeover is in progress: The local node is already taken over or is trying to take over the partner node. -partner halted in notakeover mode: The partner node was most likely halted using the "halt -f" or "system node halt -inhibit-takeover" command; reboot the partner node, and then try again. -unsynchronized log: Make sure that the interconnect link is connected and functioning. -unknown notakeover reasons: Contact NetApp technical support. -waiting for partner to recover: The partner has not booted completely after giveback; wait for the partner to come back up completely. -low memory: Contact NetApp technical support to upgrade. -local halt in progress: The local node is about to halt; try again after the reboot. -status of backup mailbox is uncertain: Check connectivity to all disks by running the "run local storage show" command on each node, and then comparing the results. Resolve the differences in disks visible to both systems. Verify that the interconnect cables are properly cabled. -automatic takeover disabled: Use the "storage failover takeover" command manually. -metrocluster disaster recovery operation is in progress: The local node is performing a MetroCluster(tm) disaster recovery operation; wait for it to finish, and then try again. -This node or the partner node is in switchover state and the MetroCluster configuration option "node-object-limit" is off in the disaster recovery(DR) group of this node: Retry takeover after doing a switchback.

Failover monitor: takeover of %s disabled (%s).

partner (STRING): Name of the partner node.
reason (STRING): Description of why takeover cannot occur.

NOTICE

This event is issued when the failover monitor determines that takeover of the partner has been enabled.

(None).

Failover monitor: takeover of %s enabled

partner (STRING): Name of partner node