OCP Extended PCIe(NVMe & CXL) Connectivity Workstream

• OCP Server Project
• PCIe Extended Connectivity Workstream
• July 20^th, 2023
• ​
• Mohamad El-Batal – Seagate
• Richard Ward – Astera-Labs
• LK Bupathi – Ayar-Labs
• ​

​

Connect. Collaborate. Accelerate.

Extended PCIe Cable Market Needs Timeline

PCIe-5.0

PCIe-6.0 🡪

Aug-22

Fabric Optimized

CXL3.0 Specification Release

1H-25

Early CXL1.1/2.0 PCIe-5.0 Pooled/Shared Appliances

1H-27

Mature CXL3.0/3.1 PCIe-6.0 Pooled/Shared Emerging Memory Appliance Deployment

2H-25

First PCIe-5.0 Host attached NVMe HDD Mass Storage JBOD

1H-24

Early Pooled DRAM PCIe-5.0 CXL1.1/2.0 Deployments

1H-23

NVMe HDD EDU2 Availability from Seagate

2H-22

NVMe HDDs Engineering Demo Units(EDU) Demonstrated

Years ago

PCIe-4.0 NVMe SSDs are deployed in JBOFs

Nov-2020

Fabric Capable CXL2.0 Specification Release

1H-26

PCIe-3.0 NVMe Possible HDD CTUs Availability

2H-24

NVMe HDD Possible CDU Availability

1H-23

Expected CXL3.1 Specification Release

• PCIe-5.0 NVMe/CXL x4, x8 & x16
• PCIe-5.0 NVMe/CXL x4, x8 & x16

2H-23

PCIe-5.0 Host attached NVMe SSD Primary Storage JBOF

2H-26

PCIe-6.0 Host attached CXL/NVMe SSD Primary JBOF

2H-27

NVMe attached SSD & HDD to replace SATA JBODs

CXL3.0/3.1

PCIe-6.0 Hosts

CXL2.0+

PCIe-5.0

Hosts

CXL1.1

PCIe-5.0

Hosts

🡪

OCP Requirement Workstream Charter 

• Research and document the future Compute, Storage, Accelerator and Memory connectivity scenarios for the NVMe & CXL enabled disaggregated datacenter
• Identify and document Commonalities and Differences between Switches, Compute, Accelerator, Networking, Storage and Memory appliance connectivity requirements
• Explore the Cost, Bandwidth, Latency, Density, Distance expectations with Electrical and Optical solutions to meet the PCIe NVMe & CXL requirements
• Produce detailed extended intra-rack & inter-rack PCIe (NVMe & CXL) connectivity High-Level scenarios and requirements

Switch

Accelerator

Compute

Networking

Storage

Memory

PCIe-SIG or SFF Specification Charter 

• Explore, study and specify optimal cable and connector solutions for Compute, Storage, Accelerator and Memory intra-rack connectivity scenarios and requirements described by the OCP workstream
• Identify Commonalities and Differences between NVMe & CXL appliance connectivity requirements 
• Investigate existing industry cost/performance optimized optical and copper cables & connectors to leverage volume production cost optimized components 
• Produce detailed extended PCIe (NVMe & CXL) detailed intra-rack & Inter-rack connectivity specification 

Switch

Accelerator

Compute

Networking

Storage

Memory

CXL & NVMe Connectivity Scenarios & Requirements

NVMe & CXL Connectivity for DAC, AEC, DOC and AOC:

• Switches, Compute, Accelerator, Memory, Networking and Storage PCIe-5.0 & PCIe-6.0 connectivity using x4 & x8 & x16 Direct-Connect, as well as x8 and x16 fanout to multiple x4 or x8
• Multiple scenarios were considered and analyzed for Intra-Rack and Inter-Rack extended connectivity with regards to the data-path and management cabling options
• We compared these scenarios to similar scenarios requirements for Intra-Appliance connectivity and concluded that common assumptions regarding data-path and management are common

U-Heights & Connection Position of Various CXL Chassis

• 1U Servers and/or Switches
• 2U Servers, GPUs blades, EDSFF Memory or Storage
• 3U PCIe CEM cards, NICs, RNICs, DPUs, IPUs …etc.
• 4U Enterprise Servers or High-Density Storage
• 4U High-Density Storage Enclosure
• 5U Potential 19” Half-width chassis vertically mounted solutions
• 6U Potential 21” Half-width chassis vertically mounted solutions 

​

An enclosure with Cable Management Arm (CMA) would require a minimum of 1M longer cables than a standard chassis. All above cable length scenarios in section-4 are analyzed without CMAs due to CXL latency budget concerns

Intra-Rack CXL Connectivity Position of Various Chassis

3.0M from Bott/Top-Of-Rack (B/TOR) to Top/Bott-Of-Rack (T/BOR)

​

2.0M from Mid-Of-Rack (MOR) to Top/Bott-Of-Rack (T/BOR)

Inter-Rack CXL Connectivity Position of Various Chassis

• 3.0M from Top-Of-Rack (TOR) to adjacent Top-Of-Rack (TOR) w/ Ceiling Tracks
• 3.0M from Bott-Of-Rack (BOR) to adjacent Bott-Of-Rack (BOR) w/ Subfloor Tracks

• 5.0M from Mid-Of-Rack (MOR) to adjacent Mid-Of-Rack (MOR) w/ Ceiling of Subfloor Tracks
• 5.0M from Top-Of-Rack (TOR) to adjacent Bott-Of-Rack (BOR) w/ Ceiling of Subfloor Tracks
• 5.0M from Bott-Of-Rack (BOR) to adjacent Top-Of-Rack (TOR) w/ Ceiling of Subfloor Tracks

​

• 7.0M from Top-Of-Rack (TOR) to adjacent Top-Of-Rack (TOR) w/ Subfloor Tracks
• 7.0M from Bott-Of-Rack (BOR) to adjacent Bot-Of-Rack (TOR) w/ Ceiling Tracks

​

Extended PCIe 5.0 and 6.0 CXL Connectivity Solutions

In conclusion we identified 23 different CXL extended connectivity scenarios that varied in cable length requirements from 0.5 Meters up to 7 Meters in lengths and concluded the following:

• DAC cables would satisfy 0.5-Meter and 1-Meter connections with:
• 1x4 to 1x4
• 1x8 to 1x8, 2x4 to 2x4, 1x8 to 2x4
• 1x16 to 1x16, 2x8 to 2x8, 4x4 to 4x4, 1x16 to 2x8 and 1x16 to 4x4
• AEC cables would satisfy 2-Meter and 3-Meter connections with:
• 1x4 to 1x4
• 1x8 to 1x8, 2x4 to 2x4, 1x8 to 2x4
• 1x16 to 1x16, 2x8 to 2x8, 4x4 to 4x4, 1x16 to 2x8 and 1x16 to 4x4
• D0C and AOC cables would satisfy 5-Meter to 7-Meter connections with:
• 1x4 to 1x4
• 1x8 to 1x8, 2x4 to 2x4, 1x8 to 2x4
• 1x16 to 1x16, 2x8 to 2x8, 4x4 to 4x4, 1x16 to 2x8 and 1x16 to 4x4

​

U-Heights & Connection Position of Various NVMe Chassis

• 0.5M from Near-End-Shelf (NES) (w/o CMA) to adjacent Near-End-Shelf (NES) (w/o CMA)
• 1.0M from Mid-End-Shelf (MES) (w/o CMA) to adjacent Mid-End-Shelf (MES) (w/o CMA)
• 2.0M from Near-End-Rack (TOR) (w/o CMA) to adjacent Near-End-Shelf (NES) (w/ CMA)
• 2.0M from Mid-Of-Rack (MOR) (w/o CMA) to Top/Bott-Of-Rack (T/BOR) (w/o CMA)
• 3.0M from Bott-Of-Rack (BOR) (w/o CMA) to Top-Of-Rack (TOR) (w/o CMA)
• 3.0M from Top-Of-Rack (TOR) (w/o CMA) to adjacent Top-Of-Rack (TOR) (w/o CMA)
• 5.0M from Top-Of-Rack (TOR) (w/o CMA) to adjacent Top-Of-Rack (TOR) (w CMA)
• 5.0M from Mid-Of-Rack (MOR) (w/o CMA) to adjacent Mid-Of-Rack (MOR) (w/o CMA)
• 7.0M from Bott-Of-Rack (BOR) (w/o CMA) to adjacent Bott-Of-Rack (BOR) (w/o CMA)
• 7.0M from Mid-Of-Rack (MOR) (w/o CMA) to adjacent Mid-Of-Rack (MOR) (w CMA)
• 10.0M from Bott-Of-Rack (BOR) (w/o CMA) to adjacent Bott-Of-Rack (BOR) (w/ CMA)
• 10.0M from Bott-Of-Rack (BOR) (w/ CMA) to adjacent Bott-Of-Rack (BOR) (w/ CMA)

​

An enclosure with CMA would require a minimum of 1M longer cables than a standard chassis. Some of the above NVMe cable length scenarios in section-5 are analyzed with CMAs since the extra cable latency is less of a concern.

Extended PCIe 5.0 and 6.0 NVMe Connectivity Solutions

In conclusion we identified 30 different NVMe extended connectivity scenarios that varied in cable length requirements from 0.5 Meters up to 10 Meters in lengths and concluded the following:

• DAC cables would satisfy 0.5-Meter and 1-Meter connections with:
• 1x4 to 1x4
• 1x8 to 1x8, 2x4 to 2x4, 1x8 to 2x4
• 1x16 to 1x16, 2x8 to 2x8, 4x4 to 4x4, 1x16 to 2x8 and 1x16 to 4x4
• AEC cables would satisfy 2-Meter and 3-Meter connections with:
• 1x4 to 1x4
• 1x8 to 1x8, 2x4 to 2x4, 1x8 to 2x4
• 1x16 to 1x16, 2x8 to 2x8, 4x4 to 4x4, 1x16 to 2x8 and 1x16 to 4x4
• D0C and AOC cables would satisfy 5-Meter to 10-Meter connections with:
• 1x4 to 1x4
• 1x8 to 1x8, 2x4 to 2x4, 1x8 to 2x4
• 1x16 to 1x16, 2x8 to 2x8, 4x4 to 4x4, 1x16 to 2x8 and 1x16 to 4x4

​

PCIe 5.0 and 6.0 DAC Specific Requirements

• DAC cables must be completely passive on data-path signals but must contain an SMBus target controller and signals to provide Vital Product Data (VPD) information for the cable on both ends, but not used to pass communication across the cable to identify the endpoint or root-port.
• Out-of-Band management between devices on extended connectivity solutions shall not require the use of sideband signals like I2C or I3C, which adds more wires across the cable thus more overall cost.
• DAC cables shall require host and endpoint support for SRIS with Spread Spectrum Clocking (SSC) and SRNS with standard single frequency clocking both as mandatory functions in order to save on the added cabling cost and EMI reduction.
• DAC cables shall support PCIe port bifurcation in SRIS/SRNS mod. All additional optional SMbus PCIe differential REFCLK(+/-) and PERST# signals are not required.
• A shared synchronization clock can be connected between Root and Endpoint devices over sideband control-plane management interface on a separate clock synchronization cable if desired by the architecture.

​

PCIe 5.0 and 6.0 DAC Validation Analysis

• Bulk cable performance is ultimately a function of cable construction and process technology. Impedances can be adjusted to optimize loss performance based on the constraints of the form factor requirements.

​

• Focus on 30 & 32 AWG to keep the system level bend radius and cable weight acceptable for most rack-level system solutions.

​

• There is more to signal integrity than the loss budget, since many factors play a big role in how long of a cable can be used including the trace routing and impedance matching as well as any reflections or crosstalk. The system designer must qualify the complete end to end solution.

​

• The Passive Cable Channel dB loss budget also depends greatly on the cable gauge and type used. The DAC cable provider must specify their loss per distance as well as their connector's loss.

Different DAC Cable Connectivity Requirements

• DAC cables can optionally support different connector width on each end to allow for Octopus cable for 1 x8 to 2 x4, and 1 x16 to 2 x8 or 4 x4 configurations with PCIe bifurcated connections.

PCIe 5.0 and 6.0 AEC Specific Requirements

• AEC cables support Re-Timer devices on both end modules, forming an Active-Active connection type.
• AEC must also contain an SMBus target controller and signals to provide Vital Product Data (VPD) information for the cable on both ends, but not used to pass communication across the cable to identify the endpoint or root-port.
• Out-of-Band management between devices on extended connectivity solutions shall not require the use of sideband signals like I2C or I3C, which adds more wires across the cable thus more overall cost.
• AEC cables shall require host and endpoint support for SRIS with Spread Spectrum Clocking (SSC) and SRNS with standard single frequency clocking both as mandatory functions in order to save on the added cabling cost and EMI reduction.
• AEC cables shall support PCIe port bifurcation in SRIS/SRNS mod. All additional optional SMbus PCIe differential REFCLK(+/-) and PERST# signals are not required.
• A shared synchronization clock can be connected between Root and Endpoint devices over sideband control-plane management interface on a separate clock synchronization cable if desired by the architecture.

​

PCIe 5.0 and 6.0 AEC Validation Analysis

• With the additional link budget from including retimers, the maximum length of the cable will depend on the line-side Package and PCB loss and the gauge of the cable (AWG).

​

• The Active Cable Channel dB-loss budget also depends on the cable gauge and type used. The AEC cable vendor shall provide the HCL/DCL and CPL losses to calculate Root and Device link margins to the Retimer IC.
• AEC cables can optionally support different connector widths on each end to allow for Octopus cable for 1 x8 to 2 x4, and 1 x16 to 2 x8 or 4 x4 configurations with PCIe bifurcated connections.
• The Analysis showed that PCIe-6.0 AEC cable lengths of 3-Meters leaves zero db-loss margin, so additional analysis is needed for full 6.0 validation.

Different AEC Cable Connectivity Requirements

• AEC cables can optionally support different connector widths on each end to allow for Octopus cable for 1 x8 to 2 x4, and 1 x16 to 2 x8 or 4 x4 configurations with PCIe bifurcated connections.

PCI-SIG Standard Retimer Package Definition

• The PCI-SIG defines the PCIe-4.0 Retimer Specification, here: https://members.pcisig.com/wg/PCI-SIG/document/17251

• The PCIe-5.0 & PCIe-6.0 Retimer Specification are under development.

PCIe 5.0 and 6.0 DOC Specific Requirements

• With DOC cables, the primary physical medium for data transfer is optical fibers. In some cases, especially as data-rates increase, optical fibers offer longer reach, better bend radius, lower weight, higher power efficiency, and lower latency relative to electrical cables.
• As compute, memory, storage, and switch devices work in the electrical domain, there is a need to convert electrical signals into the optical domain.
• Out-of-Band management between devices on extended connectivity solutions shall not require the use of sideband signals like I2C or I3C, which adds more electrical wires across the cable thus more overall cost.

With Direct Optical Connection (DOC) the transition from electrical to optical is performed in one of two ways

Different DOC Cable Connectivity Requirements

• In typical DOC implementations, the light source that contains lasers that power the optical engine are expected to be external to the data-path devices. They could be incorporated into the system either as a module mounted on the PCB or as a pluggable like the ELSF (External Laser Small Form Factor Pluggable) module specified by OIF. Solutions maybe realized with integrated lasers if power and cost are within requirements.

PCIe 5.0 and 6.0 AOC Specific Requirements

• AOC cables must support built-in Copper to Optical transceiver devices on both end connectors, in order to provide support for the complete standard PCIe-5.0 at 1x10-12 BER requirements, as well as PCIe-6.0 at 1x10-6 BER requirements as far as the Physical transport layer is concerned, while keeping in mind the need for the overall connectivity solution to enhance the BER at the Link layer for the specific protocols that we are addressing being NVMe and CXL.
• In the case of AOC, the Cable Connector needs to convert electrical signals into the optical domain since the main CXL or NVMe host or device will have only the connector socket and will support either DAC, AEC or AOC cables of the same connector type.
• Out-of-Band management between devices on extended connectivity solutions shall not require the use of sideband signals like I2C or I3C, which adds more electrical wires across the cable thus more overall cost.

Different AOC Cable Connectivity Requirements

• AOC cable source and destination PCB route and db-loss budget must also be specified to meet the standard PCIe driver complianc. AOC cables must contain an SMBus target controller and signals to provide Vendor Product Data (VPD) information on both ends.
• AOC cables must use Multi-Mode (MM) or Single-Mode (SM) Micro-Fiber cabling that comply with the optical transceiver’s requirements used on both connector ends.

Changes Required to Current CDFP Solution

Direct Attached Storage(DAS) NVMe JBOD/JBOF as well as Compute Express Link(CXL) pooled memory expansion require external host node PCIe physical layer connectivity:

• The older SFF8674 PCIe external connector not ideal beyond PCIe-4.0 x4
• Changes are required to the CDFP connector with x4, x8 and x16 to extend at PCIe 5.0 & 6.0 speeds beyond 1-Meter in the form of:
• Direct Attached Cable(DAC)
• Active Electrical Cable(AEC)
• Active Optical Cable(AOC)

Additional proposed connector form-factor by Molex for DAC and AEC is the CDFP-stye2: https://members.snia.org/document/dl/27465

QSFP/QSFP-DD and/or OSFP/OSFP-XD

PCIe-5.0 Cabling:

• x4 Full-Duplex Tx/Rx of 32Gbps using 56Gbps NRZ SR4 🡪 200Gbps QSFP
• AEC & AOC Power budget of <2.5W per QSFP
• Bit Error Rate(BER) of 10^(-12) PCIe-5.0 Requirement
• x8 Full-Duplex Tx/Rx of 32Gbps using 56Gbps NRZ SR8 🡪 400Gbps QSFP-DD/OSFP
• AEC & AOC Power budget of <5W per QSFP
• Bit Error Rate(BER) of 10^(-12) PCIe-5.0 Requirement
• x16 Full-Duplex Tx/Rx of 32Gbps using 56Gbps NRZ SR16 🡪 800Gbps OSFP-XD
• AEC & AOC Power budget of <10W per QSFP
• Bit Error Rate(BER) of 10^(-12) PCIe-5.0 Requirement

​

PCIe-6.0 Cabling: 

• x4 Full-Duplex Tx/Rx of 64Gbps using 112Gbps PAM4 SR4 🡪 400Gbps QSFP
• AEC & AOC Power budget of <5W per QSFP-DD
• Bit Error Rate(BER) of 10^(-6) PCIe-6.0 Requirement
• x8 Full-Duplex Tx/Rx of 64Gbps using 112Gbps PAM4 SR8 🡪 800Gbps QSFP-DD/OSFP
• AEC & AOC Power budget of <10W per QSFP-DD
• Bit Error Rate(BER) of 10^(-6) PCIe-6.0 Requirement
• x16 Full-Duplex Tx/Rx of 64Gbps using 112Gbps PAM4 SR16 🡪 1.6Tbps OSFP-XD
• AEC & AOC Power budget of <20W per QSFP-DD
• Bit Error Rate(BER) of 10^(-6) PCIe-6.0 Requirement

NVMe & CXL PCIe Cabling Market Analysis

Identified two Intra-Rack markets for PCIe Cables w/ ~$1B/year potential revenue:

1. NVMe DAS SSD & HDD Storage JBOD connectivity to Compute Nodes 
• HDD Storage Market by 2025 ~3 Zettabytes
• SSD Storage Market by 2025 ~1 Zettabyte
• ~200Mu HDDs plus ~50Mu SSDs are sold Yearly
• >200Mu will go in JBODs/JBOFs with ~50 Device per box >4Mu JBODs
• Datacenters update their storage infrastructure once each ~4-5 years
• Assuming ~4 cables per JBOD/JBOF we expect ~4Mu cables per year
• AOC represent 1/4^th of the cables over >3M length at ~$200 per cable at ~$200M
• AEC represent 1/4^th of the cables 1M-3M length at ~$100 per cable at ~$100M 
• DAC represent 1/2 of the cables ~1M  length at ~$50 per cable at ~$100M
2. CXL Pooled Memory connectivity to Compute Nodes
• GPU clusters can be a major enabler with >100 cables per cluster
• Market size is still unknown, but expected to require >20 cables per memory pool
• Assuming >100K Memory Pooling Appliances and > 100K GPU clusters per year >6Mu Cables/year
• AOC represent 1/4^th of the cables over >3M length at ~$200 per cable at ~$300M
• AEC represent 1/4^th of the cables 1M-3M length at ~$100 per cable at ~$150M 
• DAC represent 1/2 of the cables ~1M  length at ~$50 per cable at ~$150M

Thank you

mohamad.elbatal@seagate.com

Connect. Collaborate. Accelerate.