SAS cabling rules and concepts - shelves with IOM12/IOM12B modules
Suggest changes
PDFs
Disk shelves with IOM12/IOM12B modules can be cabled in HA pair and single-controller configurations (for supported platforms) by applying the SAS cabling rules: configuration rules, controller slot numbering rules, shelf-to-shelf connection rules, controller-to-stack connection rules, and if applicable, mini-SAS HD SAS optical cable rules.
|
The SAS cabling rules regarding controller slot numbering rules, shelf-to-shelf connection rules, and controller-to-stack connection rules described in this guide are the same rules that apply to all SAS disk shelves, whether they have IOM12/IOM12B, IOM6, or IOM3 modules. However, the information in this guide is specific to the unique characteristics of disk shelves with IOM12/IOM12B modules and their use in supported configurations. |
The SAS cabling rules regarding configuration rules and mini-SAS HD SAS optical cable rules described in this guide are specific to disk shelves with IOM12/IOM12B modules.
The SAS cabling rules described in this guide balance SAS cabling between the on-board SAS ports and host bus adapter SAS ports to provide highly available storage controller configurations and meet the following goals:
-
Provide a single, easily understood universal algorithm for all SAS products and configurations
-
Yield the same physical cabling when generating the Bill of Materials (BOM), followed in the factory, and in the field
-
Are verifiable by configuration-checking software and tools
-
Provide maximum possible resilience to maintain availability and minimize the reliance on controller takeovers
You should avoid deviating from the rules; deviations might reduce reliability, universality, and commonality.
Configuration rules
Disk shelves with IOM12/IOM12B modules are supported on specific types of HA pair and single-controller configurations.
|
|
The FAS25XX platforms are not addressed in this content. |
|
|
For current information about supported cabling configurations for your platform model, see the Hardware Universe. |
-
HA pair configurations must be cabled as multipath HA or quad-path HA configurations with the following exceptions:
-
Platforms with internal storage do not support quad-path HA connectivity.
-
A FAS2820 HA pair can be cabled as tri-path HA.
Information about the FAS2820 connectivity can be found in the Tri-path HA connectivity section.
-
Platforms with internal storage can be cabled as single-path HA configurations (from port 0b/0b1 to external shelves) to support connectivity to an external SAS tape backup device (from port 0a).
-
|
|
For FAS2820 HA pairs, although the cabling to external shelves is single-path HA, because of each controller's internal connection of port 0b to its local expander (IOM12G) and port 0c to its partner’s expander, the HA pair configuration is multipath HA. |
-
Single-controller configurations must be cabled as multipath or quad-path configurations, with the following exceptions:
-
FAS2600 series single-controller configurations can be cabled as single-path configurations.
Because the internal storage uses single-path connectivity, ONTAP issues occasional warnings that mixed paths are detected. To avoid these warnings, you can use single-path connectivity to the external disk shelves. Additionally, you can use single-path connectivity when an external SAS tape backup device is used.
-
FAS2600 series single-controller configurations do not support quad-path connectivity.
-
Controller slot numbering rules
For the purpose of applying cabling rules across all supported HA pairs and single-controller configurations, a controller slot numbering convention is used.
-
For all HA pairs and single-controller configurations, the following applies:
-
A SAS HBA in a physical PCI slot is defined as occupying PCI slot 1, 2, 3, and so on regardless of the slot's physical label on a controller.
For example, if SAS HBAs occupied physical PCI slots 3, 5, and 7, they would be designated as slots 1, 2, and 3 for the purpose of applying the SAS cabling rules.
-
An onboard SAS HBA is defined as occupying PCI slot 0 just as it is labeled on a controller.
-
Each port in each slot is defined just as it is labeled on a controller. For example, slot 0 with two ports is referred to as 0a and 0b. Slot 1 with four ports is referred to as 1a, 1b, 1c, and 1d.
In this document, slots and the slot ports are depicted as follows:
-
Shelf-to-shelf connection rules
When you have more than one disk shelf in a stack of disk shelves, they connect to each other through each SAS domain (IOM A and IOM B) using the applicable “standard” or “double-wide” shelf-to-shelf cabling. Your use of “standard” or “double-wide” shelf-to-shelf cabling depends on the configuration you have.
Standard shelf-to-shelf connectivity
-
Standard shelf-to-shelf connectivity is used in any stack of disk shelves with more than one disk shelf.
One cable connection is needed between disk shelves in each domain—domain A (IOM A) and domain B (IOM B).
-
Best practice is to use IOM ports 3 and 1 for standard shelf-to-shelf connectivity.
From the logical first shelf to the logical last shelf in a stack, you connect IOM port 3 to the next shelf's IOM port 1 in domain A and then domain B.
Double-wide shelf-to-shelf connectivity
-
Double-wide shelf-to-shelf connectivity is used in quad-pathed (quad-path HA and quad-path) configurations.
-
Double-wide shelf-to-shelf connectivity requires two cable connections between disk shelves in each domain—domain A (IOM A) and domain B (IOM B).
The first cable connection is cabled as standard shelf-to-shelf connectivity (using IOM ports 3 and 1); the second cable connection is cabled as double-wide shelf-to-shelf connectivity (using IOM ports 4 and 2).
From the logical first shelf to the logical last shelf in a stack, you connect IOM port 3 to the next shelf's IOM port 1 in domain A and then domain B. From the logical first shelf to the logical last shelf in a stack, you connect IOM port 4 to the next shelf's IOM port 2 in domain A and then domain B. (IOM ports cabled as double-wide connectivity are shown with blue.)
Controller-to-stack connection rules
You can correctly cable the SAS connections from each controller to each stack in an HA pair or in a single-controller configuration by understanding that SAS disk shelves use software-based disk ownership, how controller ports A/C and B/D are connected to stacks, how controller ports A/C and B/D are organized into port pairs, and how platforms with internal storage have their controller ports connected to stacks.
SAS disk shelf software-based disk ownership rule
SAS disk shelves use software-based disk ownership (not hardware-based disk ownership). This means that disk drive ownership is stored on the disk drive rather than it being determined by the topology of the storage system's physical connections (as it is for hardware-based disk ownership). Specifically, disk drive ownership is assigned by ONTAP (automatically or by CLI commands), not by how you cable the controller-to-stack connections.
SAS disk shelves should never be cabled using the hardware-based disk ownership scheme.
Controller A and C port connection rules (for platforms without internal storage)
-
A and C ports are always the primary paths to a stack.
-
A and C ports always connect to the logical first disk shelf in a stack.
-
A and C ports always connect to disk shelf IOM ports 1 and 2.
IOM port 2 is only used for quad-path HA and quad-path configurations.
-
Controller 1 A and C ports always connect to IOM A (domain A).
-
Controller 2 A and C ports always connect to IOM B (domain B).
The following illustration highlights how controller ports A and C connect in a multipath HA configuration with one quad-port HBA and two stacks of disk shelves. Connections to stack 1 are shown in blue. Connections to stack 2 are shown in orange.
Controller B and D port connection rules (for platforms without internal storage)
-
B and D ports are always the secondary paths to a stack.
-
B and D ports always connect to the logical last disk shelf in a stack.
-
B and D ports always connect to disk shelf IOM ports 3 and 4.
IOM port 4 is only used for quad-path HA and quad-path configurations.
-
Controller 1 B and D ports always connect to IOM B (domain B).
-
Controller 2 B and D ports always connect to IOM A (domain A).
-
B and D ports are connected to the stacks by offsetting the order of the PCI slots by one so that the first port on the first slot is cabled last.
The following illustration highlights how controller ports B and D connect in a multipath HA configuration with one quad-port HBA and two stacks of disk shelves. Connections to stack 1 are shown in blue. Connections to stack 2 are shown in orange.
Port pair connection rules (for platforms without internal storage)
Controller SAS ports A, B, C, and D are organized into port pairs using a method that leverages all of the SAS ports for system resiliency and consistency when cabling controller-to-stack connections in HA pair and single-controller configurations.
-
Port pairs consist of a controller A or C SAS port and a controller B or D SAS port.
A and C SAS ports connect to the logical first shelf in a stack. B and D SAS ports connect to the logical last shelf in a stack.
-
Port pairs use all SAS ports on each controller in your system.
You increase system resiliency by incorporating all SAS ports (on an HBA in a physical PCI slot [slot 1-N] and on board the controller [slot 0]) into port pairs. Do not exclude any SAS ports.
-
Port pairs are identified and organized as follows:
-
List A ports and then C ports in sequence of slots (0,1, 2, 3, and so on).
For example: 1a, 2a, 3a, 1c, 2c, 3c
-
List B ports and then D ports in sequence of slots (0,1, 2, 3, and so on).
For example: 1b, 2b, 3b, 1d, 2d, 3d
-
Rewrite the D and B port list so that the first port in the list is moved to the end of the list.
For example:
Offsetting the order of the slots by one balances port pairs across multiple slots (physical PCI slots and on board slots) when more than one slot of SAS ports is available; therefore, preventing a stack from being cabled to a single SAS HBA.
-
Pair the A and C ports (listed in step 1) to the D and B ports (listed in step 2) in the order that they are listed.
For example: 1a/2b, 2a/3b, 3a/1d,1c/2d, 2c/3d, 3c/1b.
For an HA pair, the list of port pairs you identify for the first controller is also applicable to the second controller.
-
-
When cabling your system, you can use port pairs in the order in which you identified them or you can skip port pairs:
-
Use port pairs in the order in which you identified (listed) them when all port pairs are needed to cable the stacks in your system.
For example, if you identified six port pairs for your system and you have six stacks to cable as multipath, you cable the port pairs in the order in which you listed them:
1a/2b, 2a/3b, 3a/1d,1c/2d, 2c/3d, 3c/1b
-
Skip port pairs (use every other port pair) when not all port pairs are needed to cable the stacks in your system.
For example, if you identified six port pairs for your system and you have three stacks to cable as multipath, you cable every other port pair in your list:
When you have more port pairs than you need to cable the stacks in your system, the best practice is to skip port pairs to optimize the SAS ports on your system. By optimizing SAS ports, you optimize your system's performance.
-
Controller-to-stack cabling worksheets are convenient tools for identifying and organizing port pairs so that you can cable the controller-to-stack connections for your HA pair or single-controller configuration.
Controller 0b/0b1 and 0a port connection rules for platforms with internal storage
Platforms with internal storage have a unique set of connection rules because each controller must maintain same domain connectivity between the internal storage (port 0b/0b1) and the stack. This means that when a controller is located in slot A of the chassis (controller 1) it is in domain A (IOM A) and therefore port 0b/0b1 must connect to IOM A in the stack. When a controller is located in slot B of the chassis (controller 2) it is in domain B (IOM B) and therefore port 0b/0b1 must connect to IOM B in the stack.
|
|
FAS25XX platforms are not addressed in this content. |
|
|
If you do not connect the 0b/0b1 port to the correct domain (cross-connect domains), you expose your system to resiliency issues that prevent you from performing nondisruptive procedures safely. |
-
Controller 0b/0b1 port (internal storage port):
-
Controller 1 0b/0b1 port always connects to IOM A (domain A).
-
Controller 2 0b/0b1 port always connects to IOM B (domain B).
-
Port 0b/0b1 is always the primary path.
-
Port 0b/0b1 always connects to the logical last disk shelf in a stack.
-
Port 0b/0b1 always connect to disk shelf IOM port 3.
-
-
Controller 0a port (internal HBA port):
-
Controller 1 0a port always connects to IOM B (domain B).
-
Controller 2 0a port always connects to IOM A (domain A).
-
Port 0a is always the secondary path.
-
Port 0a always connects to the logical first disk shelf in a stack.
-
Port 0a always connect to disk shelf IOM port 1.
-
The following illustration highlights internal storage port (0b/0b1) domain connectivity to an external stack of shelves:
Tri-path HA connectivity
Tri-path HA connectivity is available on FAS2820 HA pairs. Tri-path HA connectivity has three paths from each controller to internal (IOM12G) and external shelves:
-
Each controller’s internal connection of port 0b to its local IOM12G and port 0c to its partner’s IOM12G gives an HA pair multipath HA connectivity.
-
The cabling of each controller's external storage ports, 0a and 0b1, gives an HA pair tri-path HA connectivity.
Ports 0a and 0b1 are cabled across the two controllers when there are no external shelves, or they are cabled to external shelves to achieve tri-path HA connectivity.
The following shows the controller's internal connections and external cabling that achieves tri-path HA connectivity:
The FAS2820 external SAS ports:
-
The 0a port is from the internal HBA (like other platforms with an internal shelf).
-
The 0b1 port is from the internal shelf (like the 0b ports on other platforms with an internal shelf).
-
The 0b2 port is not used. It is disabled. If a cable is connected to it, an error message is generated.
FAS2820 HA pair cabling examples can be found in the Controller-to-stack cabling worksheets and cabling examples for platforms with internal storage section.
Mini-SAS HD SAS optical cable rules
You can use mini-SAS HD SAS optical cables—multimode active optical cable (AOC) cables with mini-SAS HD-to-mini-SAS HD connectors and multimode (OM4) breakout cables with mini-SAS HD-to-LC connectors—to achieve long distance SAS connectivity for certain configurations that have disk shelves with IOM12 modules.
-
Your platform and version of ONTAP must support the use of mini-SAS HD SAS optical cables: multimode active optical cable (AOC) cables with mini-SAS HD-to-mini-SAS HD connectors and multimode (OM4) breakout cables with mini-SAS HD-to-LC connectors.
-
SAS optical multimode AOC cables with mini-SAS HD-to-mini-SAS HD connectors can be used for controller-to-stack and shelf-to-shelf connections, and are available in lengths up to 50 meters.
-
If you are using SAS optical multimode (OM4) breakout cables with mini-SAS HD-to-LC connectors (for patch panels), the following rules apply:
-
You can use these cables for controller-to-stack and shelf-to-shelf connections.
If you use multimode breakout cables for shelf-to-shelf connections, you can only use them once within a stack of disk shelves. You must use multimode AOC cables to connect the remaining shelf-to-shelf connections.
For quad-path HA and quad-path configurations, if you use multimode breakout cables for the shelf-to-shelf double-wide connections between two disk shelves, the best practice is to use identically paired breakout cables.
-
You must connect all eight (four pairs) of the LC breakout connectors to the patch panel.
-
You need to supply the patch panels and inter-panel cables.
The inter-panel cables must be the same mode as the breakout cable: OM4 multimode.
-
Up to one pair of patch panels can be used in a path.
-
The point-to-point (mini-SAS HD-to-mini-SAS HD) path of any multimode cable cannot exceed 100 meters.
The path includes the set of breakout cables, patch panels, and inter-panel cables.
-
The total end-to-end path (sum of point-to-point paths from the controller to the last shelf) cannot exceed 300 meters.
The total path includes the set of breakout cables, patch panels, and inter-panel cables.
-
-
The SAS cables can be SAS copper, SAS optical, or a mix.
If you are using a mix of SAS copper cables and SAS optical cables, the following rules apply:
-
Shelf-to-shelf connections in a stack must be all SAS copper cables or all SAS optical cables.
-
If the shelf-to-shelf connections are SAS optical cables, the controller-to-stack connections to that stack must also be SAS optical cables.
-
If the shelf-to-shelf connections are SAS copper cables, the controller-to-stack connections to that stack can be SAS optical cables or SAS copper cables.
-