Deprecated as of version 9.0.

NOTICE

This message occurs at the start of a takeover, end of a successful takeover, start of a CFO giveback, and completion of an SFO giveback. It records the maximum, minimum, and average CPU and disk utilization on the node executing the takeover or giveback.

(None).

CPU and disk utilization during the %d seconds %s: cpu_util_high: %lld; cpu_util_low: %lld; cpu_util_avg: %lld; disk_util_high: %lld; disk_util_low: %lld; disk_util_avg: %lld

window_sz (INT): Duration, in seconds, over which CPU and disk utilization are tracked.
when (STRING): Event during which CPU and disk utilization are tracked.
cpu_util_high (LONGINT): Maximum CPU utilization.
cpu_util_low (LONGINT): Minimum CPU utilization.
cpu_util_avg (LONGINT): Average CPU utilization.
disk_util_high (LONGINT): Maximum disk utilization.
disk_util_low (LONGINT): Minimum disk utilization.
disk_util_avg (LONGINT): Average disk utilization.

NOTICE

This event is issued when we discover that the partner has previously taken us over, forcing us to invalidate our own nvram contents. This is a normal condition, subsequent to a takeover/giveback.

(None).

Failover monitor: node was previously taken over, nvram may be discarded

(None).

ERROR

This message occurs when the system discovers that disk inventory gathering has been disabled. During normal operation, the high-availability (HA) nodes transmit their disk inventory data at regular intervals. This is intended to prevent a situation in which loop connectivity problems are unnoticed until a takeover event occurs. If this event occurs, contact NetApp technical support.

Use the "sysconfig" and "storage" nodeshell commands to determine whether there are problems with the loop, adapter, or shelf. Resolve those problems.

Failover monitor: HA disk inventory disabled.

(None).

INFORMATIONAL

This event is issued when we're using a debug build and failover monitor reservations are released.

n/a

Failover monitor: released disk reservations.

(None).

NOTICE

This message occurs when the release of a reservation on a disk fails in preparation for a giveback event. The error indicates that a disk is not ready, that it failed, or that it does not exist. If the reservation is detected by the partner node, it will reboot.

Look for the cf.disk.releaseFailed event in the EMS log to find the name of the disk where the reservation could not be released. Follow the corrective action described in the cf.disk.releaseFailed event to address any problems with the disk.

Could not release disk reservations of at least one disk.

(None).

ALERT

This message occurs when the local node system identifier is the same as the partner's. This could happen if the HA-Interconnect is configured for loopback in maintenance mode systems or if the system was not properly configured. The local node will halt in this case and the partner node will do a takeover of the local node resources, provided takeover is enabled.

If this message occurs only while the system is configured in maintenance mode, it can be ignored as HA-interconnect loopback tests send a message with a node's own system identifier to itself. If this message occurs while a system is not in maintenance mode, check if the HA-interconnect cables are properly connected. If cabling is correct, contact NetApp technical support for assistance.

Partner ID %u is the same as that of this node. This node will halt and the partner will perform a takeover, if takeover is enabled.

id (INT): System ID.

NOTICE

This event occurs when we are aborting a takeover that was initiated during a previous boot sequence. This event should only occur under unusual circumstances, indicating successful recovery from a software failure.

(None).

Failover monitor: giveback of previous takeover complete

(None).

ALERT

This message occurs when an error during early takeover prevents the node from booting into takeover mode. The node instead boots up without taking over its partner and also releases its partner resources, allowing the partner node to boot up. Note: Early takeover occurs when a node boots up after rebooting while in takeover mode.

Check the EMS log for the cf.rsrc.takeoverFail error or other errors indicating why the node could not boot into takeover mode.

Early takeover failed; node will boot without taking over partner node. Partner resources released, allowing partner node to boot.

(None).

ERROR

This event is issued if the monitor fast timeout thread has been blocked for an unacceptable amount of time. The event indicates a heavy load on the system and may result in an unexpected (false) takeover.

Check CPU load and make sure system is not over subscribed.

WARNING failover monitor fast timeout was blocked for %lld secs

secs (LONGINT): Number of seconds that the High Availability (HA) node has been blocked

ERROR

This event is issued when a giveback has been cancelled due to an ongoing metrocluster disaster recovery operation.

Check the status of metrocluster disaster recovery operation by executing command 'metrocluster operation show'. If the command reports metrocluster disaster recovery operation is in progress wait for it to complete and then issue a manual giveback.

Failover monitor: giveback cancelled

(None).

NOTICE

This message occurs when a giveback is canceled due to a preexisting state, such as an active CIFS session, a reconstruction, and so on.

To override, use the "storage failover giveback -override-vetoes true" command.

Failover monitor: giveback canceled.

partner_node_uuid (STRING): UUID of the partner node.

NOTICE

This message occurs when giveback succeeds.

(None).

Failover monitor: giveback completed

token (STRING): Unique token that identifies a failover instance.
partner_node_uuid (STRING): UUID of the partner node.

NOTICE

This message occurs when a giveback is completed successfully.

(None).

Failover monitor: giveback duration time is %llu seconds.

giveback_duration (LONGINT): Giveback duration time.

ALERT

This message occurs when the failover monitor determines that a giveback has failed. The reason code is a string that describes the reason for the failure.

Resolve the issue based on the reason logged in the message.

Failover monitor: giveback failed '%s'

reason (STRING): Internal reason code for the failure.
token (STRING): Unique token that identifies a failover instance.
partner_node_uuid (STRING): UUID of the partner node.

ALERT

This message occurs when the takeover node detects that the takeover process has not been completed within the expected time, and/or normal attempts to give back partner resources also fail. Subsequent to this event, the takeover node will panic and reboot.

Attempt to find the panic string in the event logs by using the "event log show" command from the CLI, and then look up the string by using the Panic Message Analyzer tool on the NetApp support site: http://mysupport.netapp.com/NOW/cgi-bin/pmsg/. Contact NetApp technical support to confirm the analysis.

Failover monitor: forcing reboot to clear state.

partner_node_uuid (STRING): UUID of the partner node.

NOTICE

This message occurs when the failover monitor initiates a giveback.

(None).

Failover monitor: giveback started with token %s. "override-vetoes" set to %s, and "require-partner-waiting" set to %s.

token (STRING): Unique token that identifies a failover instance.
override_vetoes (STRING): Flag that indicates whether the system overrides veto checks during a giveback operation. This flag corresponds to the "-override-vetoes" parameter of the "storage failover giveback" command. When the parameter is set to true, some veto checks made by subsystems on the source node might be overridden.
require_partner_waiting (STRING): Flag that indicates whether, during a giveback, the storage is given back regardless of whether the partner node is available to take back the storage. This flag corresponds to the "-require-partner-waiting" parameter of the "storage failover giveback" command. When set to true, the parameter might cause the giveback to proceed, even if the destination node is not ready to receive the aggregate being migrated.
partner_node_uuid (STRING): UUID of the partner node.

ALERT

This message occurs when GIVEBACK_DONE is not written to the backup mailbox after all other giveback processing is done. The issuing node is no longer in takeover mode, but the partner node cannot boot (without operator intervention) because the partner mailbox claims it has been taken over.

Boot the previously taken over node. During the boot operation, the node requests confirmation to proceed.

Failover Monitor: Unexpected error %d while trying to update backup mailbox during giveback

errcode (INT): Error code.
partner_node_uuid (STRING): UUID of the partner node.

INFORMATIONAL

This event is issued if we are unable to update the partner state as part of halt processing. This occurrence of this event should not affect the operation of the High Availability (HA) pair.

(None).

halt: Unable to update failover monitor with NoTakeover state

(None).

ERROR

This message occurs when the fast timeout thread is blocked for a very long time and the system can identify threads that might have been responsible for the fast timeout thread not being scheduled.

Determine why the process is consuming the CPU, and either correct the problem, or end the offending process.

Failover monitor: Process %s ran continuously for %llu ms.

procName (STRING): Name of the process that is consuming the CPU.
schedTime (LONGINT): Time for which the process ran without releasing the CPU.

ALERT

This message occurs when failover monitor initialization fails. If this event occurs, the failover monitor cannot be started. The node will reboot after this event.

Check the logs for other messages from the failing component listed in the message by using the "event log show" command from the CLI. Also check for errors from other components or errors indicating hardware failures. If the problem occurs again after the node reboots, contact NetApp technical support.

Failover monitor: initialize(%s) fails.

component (STRING): Software component that has failed to initialize.

ERROR

This event is issued when we detect a possible mismatch of kernel versions in the High Availability (HA) pair. This situation is allowed, although takeover may be disabled if the mismatch imposes version differences in the metadata formats (nvram, filesystem, etc.) of the system.

Upgrade both nodes to the same release.

Failover monitor: possible kernel mismatch detected local '%s', partner '%s'

myVersion (STRING): My version
partnerVersion (STRING): The partner's version

INFORMATIONAL

This event is issued when we detect a possible mismatch of kernel versions in the High Availability (HA) pair has been resolved.

(None).

Failover monitor: possible kernel mismatch resolved

(None).

INFORMATIONAL

This event is issued when the failover monitor is launched. It occurs very early in the system startup sequence.

(None).

Launching failover monitor

(None).

Deprecated as of version 9.7.

NOTICE

This message occurs during an SFO aggregate giveback, when system settings indicate that giveback should be vetoed but the veto was overridden by the automated nondisruptive update procedure. The automated nondisruptive update procedure verifies the expected state of aggregate.

(None).

"%s" subsystem veto was overridden during giveback operation of "%s" aggregate.

subsystem (STRING): Name of the vetoed subsystem.
aggregate (STRING): Name of the aggregate.

INFORMATIONAL

This message reports the status of a local mailbox disk read.

(None).

(None).

returncode (INT): Status returned by the read of the local mailbox disk.

ALERT

This message occurs when the local node does not have sufficient memory to run failover monitor services.

Verify that the recommended amount of memory is installed on the system. If there is sufficient memory, the error might be related to hardware issues. In this case, capture the console logs, and then call NetApp technical support.

Takeover is disabled due to insufficient memory.

(None).

INFORMATIONAL

This message occurs on system boot when the failover monitor detects that no takeover is in progress.

(None).

(None).

status (INT): Failover monitor status as reported by the mailbox disk.

ERROR

This message occurs when the system discovers that the NVRAM mirror consistency option has been disabled. This option should ONLY be disabled under operator control. If mirror consistency is disabled, a takeover can result in a loss of recently logged data.

Run the "cf enable mirrorconsistency" advanced privilege nodeshell command to reenable mirror consistency.

Failover monitor: NVRAM mirror consistency is disabled.

(None).

ERROR

This message occurs when the HA mode is set to "ha" but no interconnect adapter is found. This is an error indicating a misconfiguration of the system.

Install the high-availability (HA) interconnect adapter or set the HA mode to "non_ha" by using the "storage failover modify -mode non_ha" command.

Warning: HA mode is set to "ha" but the interconnect adapter was not found.

(None).

ERROR

This event is issued if the failover monitor has been blocked for an unacceptable amount of time. The event indicates a heavy load on the system and may result in an unexpected (false) takeover.

Check CPU load and make sure system is not over subscribed.

WARNING failover monitor was blocked for %lld secs

secs (LONGINT): Number of seconds that the failover monitor has been blocked

INFORMATIONAL

This message occurs when an early release of reservations is not done.

(None).

(None).

state (INT): Partner firmware state.
version (INT): Partner firmware version.

INFORMATIONAL

This message occurs when there is no progress in the firmware status received from the partner.

(None).

(None).

(None).

INFORMATIONAL

This message occurs when no firmware state is obtained over the High Availability (HA) interconnect but the mailbox disks are found.

(None).

(None).

status (INT): Status of the active/active configuration based on the mailbox disks.

INFORMATIONAL

This messages indicates the status of the local mailbox disks.

(None).

(None).

returncode (INT): Return value from the call to read the local mailbox disks.
mbstatus (INT): Current status of the active/active configuration.

ERROR

This message occurs when no local mailbox disks are detected, even though the partner performed a giveback.

Check connectivity to all disks by running the "run local storage show" command on each partner, and then comparing the results.

Could not find the local mailbox disks after a giveback. Check connectivity to all disks.

(None).

ERROR

This message occurs when Data ONTAP® cannot access the local mailbox disks and cannot determine partner status through the high-availability (HA) interconnect.

Check connectivity to all disks by running the "run local storage show" command on each partner, and then comparing the results. Verify that the interconnect cables are properly cabled.

Could not find the local mailbox disks. Could not determine the firmware state of the partner through the HA interconnect.

(None).

ERROR

This message occurs when the system cannot identify the serial number of the partner because the firmware variable is not set.

1) Use the "storage failover show" command to verify that that high-availability (HA) is enabled. 2) If HA is enabled, there might be too many environment variables defined. Halt the system, and then enter the "printenv" command at the LOADER prompt. Use the "unsetenv" command to remove unneeded environment variables.

Unknown partner serial number: firmware %s variable is not set.

variable (STRING): Name of the firmware variable.

ERROR

This message indicates that we cannot do takeover during a no-rc boot.

Reboot the node normally

Failover monitor: reboot normally to enable takeover

(None).

NOTICE

This event is issued when we discover that a mailbox is uninitialized.

(None).

Failover monitor: uninitialized %s mailbox data detected

whose (STRING): Indicates which mailbox is uninitialized

ERROR

This event is issued when we discover that failover between the High Availability (HA) pair has been disabled. Failover may be disabled under operator control or when a condition has been discovered (e.g., kernel mismatch) that necessitates disabling of the HA pair.

Resolve the reason provided in the message.

Failover monitor: takeover disabled (%s)

reason (STRING): The reason code for disabling the HA pair

ERROR

This event is issued when we discover that the operator has disabled takeover-by-partner.

If takeover by the partner is desired, re-enable takeover.

Failover monitor: takeover by partner disabled

(None).

ERROR

This event is issued when we discover that the operator has disabled the nvram mirror.

Re-enable NVRAM mirroring

Failover monitor: nvram mirror disabled

(None).

NOTICE

This event is issued when the operator has manually intervened and has forced an overwrite of failover monitor state.

(None).

System continuing after overwriting failover monitor state!

(None).

ALERT

This message occurs when a node panics too soon after the completion of a takeover. The node reboots in normal mode to avoid recursive panics.

Contact NetApp technical support.

Failover monitor: Panic occurred too soon after takeover was completed (currentTime %llu ms, Takeover completed %llu ms).

currentTime (LONGINT): Time when the panic occurred.
ToDoneTime (LONGINT): Time when the takeover was completed.

EMERGENCY

This message occurs when the node panics after taking over the partner node. When the node comes back up, it will do so in takeover mode.

Attempt to find the panic string in the event logs by using the "event log show" command from the CLI, and then look up the string by using the Panic Message Analyzer tool on the NetApp support site: http://mysupport.netapp.com/NOW/cgi-bin/pmsg/. Contact NetApp technical support to confirm the analysis.

Failover monitor: Panic in takeover mode; takeover will occur on reboot.

(None).

ALERT

This message occurs when a node panics while a forced giveback is in progress. The node performs giveback and releases partner resources on reboot.

Capture the console log and contact NetApp technical support.

Failover monitor: Panic during forced giveback; node will release partner resources on reboot.

(None).

ALERT

This message occurs when a node panics while the takeover is in progress. The node reboots in normal mode with takeover disabled.

Capture the console log and contact NetApp technical support.

Failover monitor: Panic during takeover; takeover will be disabled on reboot.

(None).

INFORMATIONAL

This event is issued to announce the name of the partner.

(None).

Failover monitor: partner '%s'

partner (STRING): The name of the High Availability (HA) partner

INFORMATIONAL

This event is issued to announce a change in the name of the partner.

(None).

Failover monitor: partner hostname has changed: '%s'

partner (STRING): The name of the High Availability (HA) partner

INFORMATIONAL

This message reports the firmware status of the partner.

(None).

(None).

state (INT): Partner firmware status.

INFORMATIONAL

This message occurs when there is a change in the partner firmware state.

(None).

(None).

prevstate (STRING): Previously reported partner firmware state.
newstate (STRING): New firmware state, as reported by the partner.
progresscounter (LONGINT): New progress counter, as reported by the partner.

INFORMATIONAL

This message occurs when the partner is using a different version of the interconnect firmware.

(None).

(None).

version (INT): Partner firmware version.

INFORMATIONAL

This event is issued to announce the system id of the partner.

(None).

Failover monitor: partner system id: %u

sysid (LONGINT): The sysid of the High Availability (HA) partner

INFORMATIONAL

This event is issued to announce a change in the system id of the partner.

(None).

Failover monitor: partner system id has changed: %u

sysid (LONGINT): The sysid of the High Availability (HA) partner

NOTICE

This event is issued to indicate that the partner's volumes have been brought on-line as part of early takeover processing.

(None).

Failover monitor: partner volumes on-line

(None).

INFORMATIONAL

This event is issued when the failover monitor initiates a replay-only takeover, which essentially means performing takeover till the partner logs have been replayed, and then initiating a giveback.

(None).

Failover monitor: Starting replay-only takeover. A giveback will be initiated after the partner logs have been replayed.

(None).

INFORMATIONAL

This message occurs if a node panics in takeover mode and replay of the partner logs will be attempted on reboot.

(None).

Failover monitor: replay of partner logs will be attempted on reboot.

(None).

EMERGENCY

This event is issued if we discover that disk reservations have been disabled. If this event occurs, contact NetApp technical support.

(Call support).

Failover monitor: disk reservations disabled

(None).

ERROR

This message occurs when ONTAP® cannot reserve a high-availability (HA) partner mailbox disk during a takeover.

Check connectivity to all disks by using the "storage disk show -fields diskpathnames" command to verify each node in the HA pair has access to all disks. If some disks are not fully accessible, confirm the disks are correctly cabled. To check whether one HA node cannot access disks that are visible to the HA partner node, use the "storage failover show -fields local-missing-disks, partner-missing-disks" command.

Takeover has been aborted because the partner mailbox disk: %s could not be reserved. Error: %u.

diskname (STRING): Partner mailbox disk that ONTAP could not reserve.
disk_error (INT): Disk reservation error that was encountered.

NOTICE

This message occurs when the High Availability slow timeout thread has been blocked for an unacceptable amount of time. The event indicates a heavy load on the system and may result in an unexpected takeover.

Check CPU load and make sure system is not over subscribed. Contact NetApp technical support for further assistance.

High Availability slow timeout was blocked for %lld secs.

secs (LONGINT): Number of seconds that the High Availability (HA) slow timeout thread has been blocked.

Deprecated as of version 9.7.

NOTICE

This message occurs during an SFO aggregate giveback, when the SnapMirror® subsystem indicates that giveback should be vetoed but the veto was overridden by the automated nondisruptive update procedure. The automated nondisruptive update procedure verifies the expected state of the aggregate.

(None).

"%s" subsystem veto was overridden during giveback operation of "%s" aggregate.

subsystem (STRING): Name of the vetoed subsystem.
aggregate (STRING): Name of the aggregate.

ERROR

This event is issued when a "soft error" has occurred in the failover monitor.

Resolve the failure listed in the message.

Failover monitor: %s

reason (STRING): Description of the failure.

NOTICE

This message occurs when a takeover succeeds.

(None).

Failover monitor: takeover completed

token (STRING): Unique token that identifies a failover instance.
partner_node_uuid (STRING): UUID of the partner node.

ERROR

This message occurs when the takeover detection time is set to a value less than the DEFAULT_FIRMWARE_TIMEOUTS setting. This can result in false takeovers and takeovers without diagnostic core dumps.

Modify the takeover detection time to the recommended value by using the "storage failover modify -detection-time" command.

Takeover detection time is set to %d seconds, which is below the recommended value of %d seconds. False takeovers and takeovers without diagnostic core dumps might occur.

SECONDS (INT): Value that the takeover detection time is set to.
FIRMWARE_TIMEOUT_DEF (INT): Recommended value.

ERROR

This message occurs when the takeover detection time is set to a value less than the KERNEL_TIMEOUT setting (as specified by the "sk.process.timeout.override" option). This can result in takeovers without accompanying diagnostic core dumps of the taken over node.

Set the takeover detection time to the recommended value by using the "storage failover modify -detection-time" command.

Takeover detection time is set to %d seconds, which is below %d (= sk.process.timeout.override + 5) seconds. Takeovers without diagnostic core dumps might occur.

SECONDS (INT): Value that the takeover detection time is set to.
KERNEL_TIMEOUT (INT): Minimum value that should be used.

INFORMATIONAL

This message occurs when a takeover is completed successfully.

(None).

Failover monitor: takeover duration time is %llu seconds.

takeover_duration (LONGINT): Takeover duration time.

ALERT

This message occurs when the failover monitor determines that a takeover has failed. The reason code is a string that describes the reason for the failure. Any data LIFs that were migrated as part of the takeover operation are not automatically reverted.

Resolve the issue based on the reason logged in the message.

Failover monitor: takeover failed '%s'

reason (STRING): Internal reason code for the failure.
token (STRING): Unique token that identifies a failover instance.
partner_node_uuid (STRING): UUID of the partner node.

NOTICE

This message occurs when the failover monitor initiates a takeover.

(None).

Failover monitor: takeover started

token (STRING): Unique token that identifies a failover instance.
partner_node_uuid (STRING): UUID of the partner node.

INFORMATIONAL

This event is when we determine our status as time master or slave.

(None).

Acting as time %s

masterOrSlave (STRING): Master or Slave

INFORMATIONAL

This message occurs when the current setting of takeover detection time is shorter than the minimum takeover detection time allowed by this version of Data ONTAP®. This can result in false takeovers or takeovers without diagnostic core dumps. Data ONTAP resets the takeover detection time to the new minimum.

(None).

Takeover detection time was set to %d seconds, shorter than the minimum allowed. Reset the detection time to a new minimum of %d seconds.

SECONDS (INT): Current takeover detection seconds.
FIRMWARE_TIMEOUT_DEF (INT): New default takeover detection seconds.

INFORMATIONAL

This message occurs when you set the takeover or giveback transit timeout to a value other than the default value. During takeover or giveback, if the timeout is exceeded by a subsystem during the takeover/giveback processing, a panic occurs. If the timeout is set too high, longer client outages might occur instead of aborting the takeover/giveback.

(None).

(None).

SECONDS (INT): Transit timeout value (in seconds).
DEFAULT_VAL (INT): Default transit timeout value (in seconds).

NOTICE

This event is issued when we initiate an undo of a failed takeover.

(None).

Failover monitor: initiate giveback due to failed takeover

(None).

ERROR

This message occurs when the HA mode is set to "non_ha" but the HA mode was set to "ha" previously. This is not an error, but indicates a possible misconfiguration of the system.

Determine whether the HA mode should be set to "ha", and if so, set it.

Warning: HA mode is set to "non_ha" but the node once had a storage failover partner.

(None).

ALERT

This event occurs when a version mismatch is detected during internode initialization. Each node transmits its version information to its partner. If a mismatch is detected, the High Availability (HA) takeover capability is disabled.

Boot both nodes on the same release.

Failover monitor: %s version mismatch detected: %d/%d

subsystem (STRING): The name of the versioned subsystem
myVersion (INT): My version
partnerVersion (INT): The partner's version

INFORMATIONAL

This message occurs when a system waits, during boot, for a module to come up before declaring itself ready for giveback. Examples include waiting for the NVRAM battery to be charged.

(None).

Failover monitor: waited %llu seconds for module %s.

secs (LONGINT): Amount of time spent waiting, in seconds.
module_name (STRING): Name of the module the system is waiting for.