OCP External PCIe(NVMe & CXL) Connectivity Workstream

• OCP Server Project
• PCIe External Connectivity Workstream
• Dec 12^th, 2022

​

Connect. Collaborate. Accelerate.

Seagate Internal

Extended PCIe Cable Market Needs Timeline

PCIe Gen5

PCIe Gen6 🡪

Aug-22

Fabric Optimized

CXL3.0 Specification Release

2H-25

Early CXL1.1/2.0 PCIe-Gen5 Pooled/Shared Appliances

2H-27

Mature CXL3.0/3.1 PCIe-Gen6 Pooled/Shared Emerging Memory Appliance Deployment

2H-26

First PCIe-Gen5 Host attached NVMe HDD Mass Storage JBOD

1H-24

Early Pooled DRAM PCIe-Gen5 CXL1.1/2.0 Deployments

1H-23

NVMe HDD EDU2 Availability from Seagate

2H-22

NVMe HDDs Engineering Demo Units(EDU) Demonstrated

Years ago

PCIe-Gen4 NVMe SSDs are deployed in JBOFs

Nov-2020

Fabric Capable CXL2.0 Specification Release

1H-26

PCIe-Gen3 NVMe Possible HDD CTUs Availability

2H-24

NVMe HDD Possible CDU Availability

1H-23

Expected CXL3.1 Specification Release

• PCIe-Gen5 NVMe/CXL x4, x8 & x16
• PCIe-Gen5 NVMe/CXL x4, x8 & x16

2H-23

PCIe-Gen5 Host attached NVMe SSD Primary Storage JBOF

1H-27

PCIe-Gen6 Host attached CXL/NVMe SSD Primary JBOF

2H-27

NVMe attached SSD & HDD to replace SATA JBODs

CXL3.0/3.1

Gen6 Hosts

CXL2.0+

Gen5 Hosts

CXL1.1

Gen5 Hosts

🡪

Seagate Internal

NVMe Connectivity Scenarios & Requirements

NVMe Connectivity for DAC, AEC and AOC:

• Compute to Storage x4 & x8 & x16 PCIe-Gen5
• Accelerator to Storage x4 & x8 & x16 PCIe-Gen5
• Storage Expansion x4 & x8 & x16 PCIe-Gen5
• Distance 0.5 to 5 Meter distance requirements (Wen Mei Hwu nVidia)
• Cable Management Arm(CMA) Restrictions (Barrett Edwards - WDC)
• Bend Radius Restrictions for CMA 320-deg ?
• Extra length needed for CMA on each end 1.2 to 1.5M ?
• Consider Interoperability between Connectors of different width
• Support for different width and connectors types on ether sides of the cable
• Support multiple side-band signals for bifurcation
• Support either bifurcated and non-bifurcate cables
• Support for SRIS/SRNS specific cables
• DAC, Gen5 32Gb/s, BER 10^-12, 28 to 30 gauge Cable, Bend-Radius Budget?
• With Passive CEM card RP/EP 8”> x >4” from connector on main PCB on both ends 🡪 0.5M
• With Active CEM card RP/EP 4”> x >2” from connector on main PCB on both ends 🡪 0.5M to 1M
• AEC, Gen5 32Gb/s, BER 10^-12, 32 gauge Cable, 36-DBL each end
• Bend-Radius Budget 5x outside diameter of the cable
• Sideband signals of SmBus/I2C/I3C are localized to the connector and will not connect between the two ends of the cable
• REFCLK & PERST are not needed due to the proposed use of SRIS/SRNS
• Re-Timer Based Solutions 🡪 1M to 3M @ 32-gauge
• Power budget for x4, x8 & 16 with 1.15W per lane including DC/DC deficiency
• For a x4 cable 5.2mm Diameter (Don from Credo) add cable bends
• For a x8 cable 6.8mm Diameter (Don from Credo) add cable bends – x8 Retimer package size
• For a x16 cable we use two of the x8 cables add cable bends – x16 Retimer package size 8.90 mm x 22.80 mm
• Re-Driver Based Solutions 🡪 1M to 2M? @ 32-gauge – Alvin Cox Re-driver add significant Jitter and could prevent us from any improvements over DAC cable.
• Active device on both-ends are in most cases
• Active device is added on one end of the other on the Root-port or Endpoint card if active device in not needed on both ends of the wire
• AOC or other Optical NPO/CPO, Gen5 32Gb/s , BER 10^-12 🡪 1M to 10M
• Electrical specification – Chip to module distance
• Long-Reach Full Standard PCIe Gen5 36 DBL budget – Required for most Active cable solutions in order to interoperate with standard PCIe Gen5 transceivers
• Medium to Short Reach DBL budget ?
• Very-Short-Reach DBL budget ?
• Multi-Mode or Single-Mode fiber cables can be used as long as they are cost effective
• Bend-Radius Budget 10x optical fiber diameter ~30mm
• Sideband signals of SmBus/I2C/I3C are localized to the connector and will not connect between the two ends of the cable
• REFCLK & PERST are not needed due to the proposed use of SRIS/SRNS

Seagate Internal

CXL Connectivity Scenarios & Requirements

CXL Connectivity for DAC, AEC and AOC:

• Compute Cluster x8 & x16 PCIe-Gen5
• Compute to Accelerator x8 & x16 PCIe-Gen5
• Compute to Memory x8 & x16 PCIe-Gen5
• Accelerator Cluster x8 & x16 PCIe-Gen5
• Accelerator to Memory x8 & x16 PCIe-Gen5
• Distance 0.5 to 5 Meter distance requirements
• Bend radius
• Consider Interoperability between Connectors of different width
• Support for different width and connectors types on ether sides of the cable
• Support multiple side-band signals for bifurcation
• Support either bifurcated and non-bifurcate cables
• Support for SRIS/SRNS specific cables
• DAC, Gen5 32Gb/s, BER 10^-12, 8-DBL? each end, 28 to 30-gauge Cable, Bend-Radius Budget 🡪 0.5M to 1M
• With Active RP/EP 6”> x >2” from connector on main PCB on both ends
• AEC, Gen5 32Gb/s, BER 10^-12, 16-DBL each end, 32-gauge Cable, Bend-Radius Budget 🡪 2M to 4M
• Re-Timer Based?
• Re-Driver Based?
• Power Budget?
• Additional Pins Needed?
• Active device on both-ends ?
• Active device on one-end?
• AOC, Gen5 32Gb/s , BER 10^-12, 16-DBL each end, Multi-Mode Cable, Bend-Radius Budget 🡪 5M to 10M

Seagate Internal

OCP Workstream Charter 

• Research and document the future Compute, Storage, Accelerator and Memory connectivity scenarios for the NVMe & CXL enabled disaggregated datacenter
• Identify and document Commonalities and Differences between Compute, Accelerator, Storage and Memory appliance connectivity requirements
• Explore the Cost, Bandwidth, Latency, Density, Distance expectations with Electrical and Optical solutions to meet the PCIe NVMe & CXL requirements
• Produce detailed external intra-rack PCIe (NVMe & CXL) connectivity High-Level scenarios and requirements

Seagate Internal

SFF Specification Charter 

• Explore, study and specify optimal cable and connector solutions for Compute, Storage, Accelerator and Memory intra-rack connectivity scenarios and requirements described by the OCP workstream
• Identify Commonalities and Differences between NVMe & CXL appliance connectivity requirements 
• Investigate existing industry cost/performance optimized optical and copper cables & connectors to leverage volume production cost optimized components 
• Produce detailed external PCIe (NVMe & CXL) detailed intra-rack connectivity specification 

Seagate Internal

NVMe & CXL PCIe Cabling Market Analysis

Identified two Intra-Rack markets for PCIe Cables w/ ~$600M/year potential revenue:

1. NVMe DAS SSD & HDD Storage JBOD connectivity to Compute Nodes 
• HDD Storage Market by 2025 ~3 Zettabytes
• SSD Storage Market by 2025 ~1 Zettabyte
• ~200Mu HDDs plus ~50Mu SSDs are sold Yearly
• >200Mu will go in JBODs/JBOFs with ~50 Device per box 🡪 ~4Mu JBODs
• Datacenters update their storage infrastructure once each ~4-5 years
• Assuming ~4 cables per JBOD/JBOF we expect ~4Mu cables per year
• AOC represent 1/4^th of the cables over >3M length at ~$200 per cable 🡪 ~$200M
• AEC represent 1/4^th of the cables 1M-3M length at ~$100 per cable 🡪 ~$100M 
• DAC represent 1/2 of the cables ~1M  length at ~$50 per cable 🡪 ~$100M
2. CXL Pooled Memory connectivity to Compute Nodes
• Market size is still unknow, but expected to require ~20 cables per memory pool
• Assuming ~100K Memory Pooling Appliances per year 🡪 ~2Mu Cables/year
• AOC represent 1/4^th of the cables over >3M length at ~$200 per cable 🡪 ~$100M
• AEC represent 1/4^th of the cables 1M-3M length at ~$100 per cable 🡪 ~$50M 
• DAC represent 1/2 of the cables ~1M  length at ~$50 per cable 🡪 ~$50M

Seagate Internal

Option-1: Changes Required to Current CDFP Solution

Direct Attached Storage(DAS) NVMe JBOD/JBOF as well as Compute Express Link(CXL) pooled memory expansion require external host node PCIe physical layer connectivity:

• The older SFF8674 PCIe external connector not ideal beyond PCIe Gen4 x4
• Changes are required to the CDFP connector with x8 and x16 to extend at PCIe Gen5 & Gen6 speeds beyond 1-Meter at 10^(-12) BER in the form of:
• Direct Attached Cable(DAC)
• Active Electrical Cable(AEC)
• Active Optical Cable(AOC)

Seagate Internal

Option-2: QSFP/QSFP-DD and/or OSFP/OSFP-XD

PCIe Gen5 Cabling:

• x4 Full-Duplex Tx/Rx of 32Gbps using 56Gbps NRZ SR4 🡪 200Gbps QSFP
• AEC & AOC Power budget of <2.5W per QSFP
• Bit Error Rate(BER) of 10^(-12) PCIe Gen5 Requirement
• x8 Full-Duplex Tx/Rx of 32Gbps using 56Gbps NRZ SR8 🡪 400Gbps QSFP-DD/OSFP
• AEC & AOC Power budget of <5W per QSFP
• Bit Error Rate(BER) of 10^(-12) PCIe Gen5 Requirement
• x16 Full-Duplex Tx/Rx of 32Gbps using 56Gbps NRZ SR16 🡪 800Gbps OSFP-XD
• AEC & AOC Power budget of <10W per QSFP
• Bit Error Rate(BER) of 10^(-12) PCIe Gen5 Requirement

PCIe Gen6 Cabling: 

• x4 Full-Duplex Tx/Rx of 64Gbps using 112Gbps PAM4 SR4 🡪 400Gbps QSFP
• AEC & AOC Power budget of <5W per QSFP-DD
• Bit Error Rate(BER) of 10^(-6) PCIe Gen6 Requirement
• x8 Full-Duplex Tx/Rx of 64Gbps using 112Gbps PAM4 SR8 🡪 800Gbps QSFP-DD/OSFP
• AEC & AOC Power budget of <10W per QSFP-DD
• Bit Error Rate(BER) of 10^(-6) PCIe Gen6 Requirement
• x16 Full-Duplex Tx/Rx of 64Gbps using 112Gbps PAM4 SR16 🡪 1.6Tbps OSFP-XD
• AEC & AOC Power budget of <20W per QSFP-DD
• Bit Error Rate(BER) of 10^(-6) PCIe Gen6 Requirement
• Can’t use FEC for CXL, due to ns latency sensitive, but could use FEC for NVMe BER needs of 10^(-12)

Seagate Internal

QSFP-DD Pinout Option

• x8 Full-Duplex Tx/Rx
• x4 Full-Duplex Tx/Rx (half-populated)
• 4-pos Vendor Specific Definition (VSx)
• Single-PCB solution
• Support for DAC, AEC, and AOC
• Support for 26-32AWG Twin-ax Cable
• Available with Riding Heat Sink (RHS)

​

8.5mm

13.8mm

60mm

Form Factor Outline

Port� Profile

Seagate Internal

OSFP-XD Pinout Option

• x16 Full-Duplex Tx/Rx
• x8 Full-Duplex Tx/Rx (half-populated)
• 4-pos Reserved For Future Use (RFFU)
• Single-PCB solution
• Support for DAC, AEC, and AOC
• Support for 26-32AWG Twin-ax Cable
• Available with Integrated Heat Sink (IHS) or Riding Heat Sink (RHS)

​

Source: https://osfpmsa.org/assets/pdf/OSFP1600_and_OSFP-XD.pdf

https://osfpmsa.org/assets/pdf/OSFP_Module_Specification_Rev5_0.pdf

10.5mm*

22.8mm

95mm

*w/ RHS

Form Factor Outline

Port� Profile

Seagate Internal

Option-3: CW-WDM MSA Primarily for CXL

CXL 1.1/2.0:

• Fully integrated electro-optical Chiplet supporting up to 8 WDM optical ports
• Each port is a x8 link
• Full-duplex Tx/Rx at 32 Gbps
• Each port has 8λ per Tx/Rx fiber, based on the CW-WDM MSA
• Port bandwidth is 256 Gbps, up to 2 Tbps per Chiplet
• Bit Error Rate (BER) of 10^(-12) meeting PCIe Gen5 requirement. No FEC required.
• <1W per dual-port (x16) Chiplet
• Re-driver-based Chiplet taking advantage of PCIe-Gen5 MR/LR SerDes on host ASIC
• Latency around 1ns (electrical interface to optical interface)

CXL 3.0: 

• Scalable architecture
• Data rate at 64 Gbps
• Support PAM4 modulation scheme
• CW-WDM MSA supports 16λ and 32λ per fiber

Seagate Internal

Thank you

mohamad.elbatal@seagate.com

Connect. Collaborate. Accelerate.

Seagate Internal