1 of 23

2 of 23

SERVER

OCP NIC Community Update�November 2023

�����Sub-group Project wiki: https://www.opencompute.org/wiki/Server/NIC�

Mailing List: https://ocp-all.groups.io/g/OCP-NIC

NIC3.0

3 of 23

Agenda

  • OCP Global Summit – Break-out session
  • OCP Global Summit – Experience Lab
  • OCP NIC rev1.3 spec released
  • PCIe Gen6 update
  • Double-width SFF

​

​

​

4 of 23

OCP Global Summit 2023

Break-out session/Presentation:

2 presentations delivered

  • OCP NIC in 2023: 800G, PCIe Gen6 & Test Integration by Broadcom, Dell & Meta
  • Ensuring Predictable Thermal Performance for Higher Speed OCP 3.0 NICs by Dell

​

5 of 23

OCP Global Summit 2023

Experience Center demo:

Servers, NICs, Test fixtures and Cable/Connectors ecosystem demo by

Amphenol, Broadcom, Dell, Intel, Meta, Portwell, TE connectivity

6 of 23

OCP NIC 3.0 rev 1.3 released

A list of updates and maintenance which Thomas would help to walk through

What’s NOT in rev1.3

  • PCIe Gen6
  • D-SFF

​

​

Thomas

7 of 23

SIWG PCIe Gen6 Channel Simulation

Feedback from SNIA regarding proposal to slightly increase crosstalk

Insertion loss & Crosstalk goes hand-in-hand for overall channel ability to recover signal. Might have implications to NIC insertion loss budget

​

Tanmoy

Line Rate

Insertion Loss

Return Loss

Power Sum Near End Crosstalk

Power Sum Far End Crosstalk

64 GT/s PAM4

(PCIe 6.0)

≥(-0.1–0.040625*f)dB

(0.01≤f≤16 GHz)

 

≥(1.75-0.15625*f)dB

(16<f≤24 GHz)

≤(-25+0.625*f)dB

(0.01≤f≤24 GHz)

 

iRL1,4 ≤ -28 dB

≤(-65+0.625*f)dB

(0.01≤f≤24 GHz)

 

ccICNNEXT2 ≤ 149uV

≤(-60+0.625*f)dB

(0.01≤f≤24 GHz)

 

ccICNFEXT3 ≤ 110uV

8 of 23

Double-wide SFF for Smart NICs

Spans over two 4C+ connectors

Regular and Tall options

160 Watt

x32 PCIe lanes

13% more board space than LFF

​

Opens:

    • Management scheme
    • Latching mechanism
    • More

​

We need initial 3D models for other groups evaluations

​

​

Illustration only

Arnon/Harvey

Presented on: 8/9/2023

9 of 23

D-SFF option preference

Option 1: Follow M-FLW

Option 2: Increase gap support thumbscrews

Customer A

Required cost for surprise removal is high.

Platform incline to internal latch version

Width of faceplate of thumbscrew became a limitation trade-off for other system feature

Customer B

Embrace internal lock since the start of OCP

​

Customer C

Majority thumbscrews. Some internal latch is niche

​

​

Vendor B

Ejector for Customer D, Internal lock Customer A & B

Thumbscrews for channel ODM CSP partners

On website wide distributions is thumbscrews

​

​

Vendor A

Internal lock for OEM

Thumbscrews for the rest of market

Able to support all flavors of OCP NIC including thumbscrews. Give the flexibility to customer

​

​

Customer D Wearing OCP HAT

Which form factor can drive most adoptions?

​

​

FLW adjusted the gaps to squeeze in tubing manifold or boot E1.S

While unable to support dual-thumb screws, it support dual internal lock or dual ejector latch

Aligned

10 of 23

D-SFF features

System Must Have: (Meta/Damien, Intel/Thomas/Paul, HPE/Mujeeb, Nvidia/Yuval)

  • Something to manage 1 DSFF vs 2 SFF
    • Primary, secondary
  • Manage diverse lane mappings (more than LFF definition)
    • Rootport, endpoint mixed usage
  • Pin-out discussion
    • Direction of PERST and CLK
    • Flex pins

​

​

Preferred:

  • Pin-out stayed identical
  • Accurate Timing Sync feature
    • PPS signal

Mechanical Must Have: (Amphenol/Zhineng, Dell/Jason, Intel/Johnathanh, Molex/Dubi, TE/Kyle, Meta/Ivy?, Broadcom/Jason, Nvidia/Arnon(acting) )

  • Mechanical tolerance
  • Ejection mechanism
    • Latch mechanism on both side?
    • Ejector rail reference?
  • Card height
    • SFF, TSFF, 1RU-DSFF?

​

​

​

​

Electrical Must Have: (Meta/Tanmoy?, Intel/Thomas/Robert, Dell/Jason, Broadcom/Jason, Nvidia/Yuval, HPE/Mujeeb

  • Insertion loss for PCIe Gen6 D-SFF
    • -6dB trend is only for SFF

​

​

​

11 of 23

12 of 23

Gather feedback throughout the year

Provide your feedback at:�https://tinyurl.com/feedback2ocpnic

13 of 23

SIWG for PAM4 Gen6 spec

Xtalk spec to meet BER requirement

    • SFF-TA-1002 proposal(rev 1.4.1) in progress, will leverage
    • SFF-TA-1009 proposal(rev 3.1.1) for device spec
    • WIP to check for channel simulations
    • Working with device vendor to meet Rx spec

Anthony/Tanmoy

Connector Spec

14 of 23

Channel Simulation

Based on Gen5 models as same Nyquist for Gen6

    • Root complex/Switch ~ 20dB + Package model
    • AIC/OCP ~7dB + package model
    • Tighter PSNEXT proposal (-60dB @16Ghz), ccICN comparison
    • Gen5 models are close to meet the spec

​

Anthony/Tanmoy

Straddle Mount

AIC/OCP

15 of 23

Channel Gen5 vs Gen6

~32dB

~28dB

CPU

NIC

CPU

NIC

7dB

6dB?

Gen 5

Gen 6

16 of 23

SI Test Fixtures

CLB 5.0 32GT/s status:

    • Dell to present at IC meeting to release design to Wiki

​

CBB 5.0 32GT/s status:

    • Multi-vendor testing results within predicted performance, suitable for CEM equivalent testing
    • Dell to present at IC meeting to release design to Wiki

OCP NIC 3.0 SFF CLB 5.0 prototype

Calibration

CLB board 1

Impedance Coupon

CLB board 2

Jason

OCP NIC 3.0 SFF CBB 5.0 block diagram

17 of 23

OCP NIC 3.0 ejection mechanism

Customer A

Customer B

Customer D

Customer C

Customer X,Y,Z

18 of 23

D-SFF Option 1: FLW rev1.0 SFF recommendation

thumbscrew possible

No thumbscrew

  • Dimensions follow M-FLW rev1.0
  • Middle NIC card unable to support Pull-tab/Thumbscrew version
  • Opens: Does this dimension support “ejector latch” faceplate?

​

19 of 23

D-SFF Option 2: Define separation to fit Pull-tab/Thumbscrew

Thumbscrew support

?

  • Dimensions follow M-HPM Type4 (M-DNO4) and M-HPM Type7 (M-DNO7, aka wide 21” FLW)

​

Thumbscrew support

20 of 23

D-SFF option preference

Option 1: Follow M-FLW

Option 2: Increase gap support thumbscrews

Customer A

Required cost for surprise removal is high.

Platform incline to internal latch version

Width of faceplate of thumbscrew became a limitation trade-off for other system feature

Customer B

Embrace internal lock since the start of OCP

​

Customer C

Majority thumbscrews. Some internal latch is niche

​

​

Vendor B

Ejector for Customer D, Internal lock Customer A & B

Thumbscrews for channel ODM CSP partners

On website wide distributions is thumbscrews

​

​

Vendor A

Internal lock for OEM

Thumbscrews for the rest of market

Able to support all flavors of OCP NIC including thumbscrews. Give the flexibility to customer

​

​

Customer D Wearing OCP HAT

Which form factor can drive most adoptions?

​

​

21 of 23

OCP NIC 3.0 spec 1.2.1 PCIe mapping

Paul

22 of 23

Mixed mode LFF use case examples

 

PCIe Interface A

PCIe Interface B

#

0,1

2,3

4,5

6,7

8,9

10,11

12,13

14,15

16,17

18,19

20,21

22,23

24,25

26,27

28,29

30,31

1

Endpoint

Root Port

Root Port

Root Port

Root Port

 

REFCLK0

REFCLK2

 

PERST_N0

EXT_PERST_0

EXT_PERST_1

EXT_PERST_2

EXT_PERST_3

​

​

​

​

​

​

​

​

​

2

Endpoint

Endpoint

Endpoint

Endpoint

Root Port

Root Port

Root Port

Root Port

 

REFCLK0

REFCLK1

REFCLK2

REFCLK3

REFCLK4

 

PERST_N0

PERST_N1

PERST_N2

PERST_N3

EXT_PERST_0

EXT_PERST_1

EXT_PERST_2

EXT_PERST_3

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

3

Root Port

Root Port

Root Port

Root Port

Root Port

Root Port

Root Port

Root Port

Root Port

Root Port

Root Port

Root Port

Root Port

Root Port

Root Port

Root Port

 

REFCLK0

 

EXT_PERST_0

EXT_PERST_1

EXT_PERST_2

EXT_PERST_3

EXT_PERST_4

EXT_PERST_5

EXT_PERST_6

EXT_PERST_7

EXT_PERST_8

EXT_PERST_9

EXT_PERST_10

EXT_PERST_11

EXT_PERST_12

EXT_PERST_13

EXT_PERST_14

EXT_PERST_15

​

​

​

4

Endpoint

Root Port

 

REFCLK0

REFCLK2

 

PERST_N0

PERST_N1

Paul

23 of 23

SDNO Class A to D