Skip to main content
Novus

Novus storage system environment and processing

Contributors dmp-netapp
PDFs

There are several important aspects of the Novus environment and its processing that you should understand.

Novus Data Director and ONTAP data server internal communication

The Novus Data Directors communicate with the ONTAP data servers to support file access and management. Most of this is initiated by the NDD rather than the ONTAP servers.

ONTAP data server discovery

The Novus Data Directors need to discover what data servers are available. ONTAP never initiates communication with the Novus Data Director. An operator or a tool registers each data server through the mds.conf configuration file, which is used at startup time; data servers can also be manually added using the command mds-admin ds add. Each data server is represented as a row in the distributed metadata store including its host, export, ports, state and capacity.

ONTAP data server availability

There are several processes used to determine the availability of each ONTAP data server.

Heartbeat

The MDS sends an NFSv3 NULL RPC call over TCP to each online data server. By default it does this every 5 seconds using a heartbeat message.

Going offline

After a set number of failures in a row, the data server is marked as offline. The MDS stops placing new files on it and informs clients to give back layouts that point to it.

Coming back

After enough successful probes in succession, it returns to online. A cool down period stops it from flipping back and forth.

Client errors don't count

When a client reports an error it's only logged. ONTAP data server health is decided by the probes alone.

ONTAP failover is invisible

If an ONTAP HA pair does a takeover, the MDS only notices through probe timeouts.

ONTAP data server data content

The NDD doesn't ask ONTAP what it stores. It already knows, because it creates every backing file:

  • When a file is created, the MDS picks the data servers, creates the backing files on them, and saves the result in the stripe map in the distributed metadata store. The stripe map records which data server and NFS file handle hold each stripe and mirror.

  • Backing files on ONTAP have a specific name format in the NDD.

  • Capacity is the only operational value the NDD reads from ONTAP using statvfs() on each mounted export.

  • File size isn't verified against ONTAP. It's determined from the client's layout commit.

High-availability and resiliency for the NDD cluster

The metadata plane anchored by the Novus Data Directors ensures high availability and resilience in the system.

Cluster design

Novus uses a multi-host cluster to ensure high availability and fault tolerance for the NDD services.

Three hosts with three failure domains

The minimum NDD cluster is three physical hosts or virtual machines (VMs) deployed on separate servers. Each host runs one MDS process and one metadata store node. The cluster is designed to survive the loss of one host.

All replicas serve

Every MDS replica handles clients at the same time with none designated in standby mode. All healthy replicas announce the same VIP through BGP. BFD (fast link-failure detection) is required because BGP's own timers alone are not a completely acceptable failover mechanism. A replica announces the VIP only when its NFS service, gateway and metadata store connection are all ready for client handling.

Metadata replication

The distributed metadata store keeps three synchronous copies. An arbitrator prevents a split-brain scenario by choosing to go unavailable rather than letting two sides both write.

Restarts are safe

MDS processes keep no state of their own; the distributed metadata store holds the authoritative metadata.

Limitations and cautions

You should also be aware of the following limitations and cautions:

  • Subtree ownership doesn't fail over automatically. The namespace can be split into subtrees, each owned by one MDS. If an owner fails, its subtree hangs until the owner comes back, even though the VIP has already moved.

  • One host failure affects every tenant on that host because each tenant has one replica per host.

  • Clients must mount the VIP and not a host IP. Otherwise the client can't fail over.

  • Hard mounts may block during recovery. A blocked mount doesn't mean data was lost.

  • Always perform maintenance one replica at a time. Take the replica out of the VIP and drain it first.