1 of 15

2 of 15

SERVER

OCP NIC Community Update�April 2023

�����Sub-group Project wiki: https://www.opencompute.org/wiki/Server/NIC�

Mailing List: https://ocp-all.groups.io/g/OCP-NIC

NIC3.0

3 of 15

Agenda

  • TSFF TTF challenge
  • SFF-TA-1002 PCIe Gen6 Crosstalk
  • SmartNIC summit
  • PCIe Gen5 Thermal Test Fixture
  • SI test fixture

​

​

​

4 of 15

Thermal Test Fixture for TSFF feedbacks

Nvidia feedback TSFF has air bypass issue. It was similar on SFF testing fixture:

  • It makes the NIC thermal performance is worse comparing to the case if assuming no air bypass from the front panel
  • It is also not suitable to use candlestick to capture NIC local air velocity as the candlestick measurement shows large variation and low accuracy at especially at low air velocity and cold aisle

Therefore, Nvidia propose to use air duct as shown in Figure at the right bottom to regulate the air only going through the NIC portion (avoid air bypass on the sides).

​

Anna to provide additional study results (will share in May monthly call):

    • Report the local air velocity and compare with candlestick sensor reading
    • Limit the lowest air velocity for the candlestick measurement
    • List the options that can solve the problem

​

1 slot test fixture

5 of 15

Thermal Test Fixture for TSFF feedbacks

OCP 3.0 Spec:

​

6 of 15

Feedback from part of the community:

  • The issue is well-received and makes sense
  • NIC thermal test fixture is used for the comparison purpose as it mimics the NIC performance in the real chassis (will have bypass)
  • Using candlestick method has been recommended in NIC 3.0 spec, it requires a community decision if we would like to propose a change

​

​

Call for Action:

- Looking for more feedback and more involvement on this decision making, and to have a follow-up call

​

Thermal Test Fixture for TSFF feedbacks

7 of 15

SIWG for PAM4 Gen6 spec

Issues with Xtalk spec to meet BER requirement

    • SFF-TA-1002 proposal – in progress, can leverage
    • Base spec for package is relaxed – difficult to meet BER target
    • ICR@nyquist can be another metric for channel compliance
    • Proposing to have ad-hoc sessions to close on the proposal

​

Anthony/Tanmoy

8 of 15

SmartNIC Summit

  • SmartNIC Summit June 13-15, 2023 held in San Jose, CA
  • SmartNIC Summit Organizer & OCP Foundation seeking our group support in sharing OCP NIC in SmartNIC Summit
    • Demos
    • Presentations
  • So far Meta, Dell, Nvidia interested

​

9 of 15

Test fixture using existing TTF with Gen5 rated cable assemblies

    • TTF was designed with Megtron 6 to support speeds up to 32GT/s
    • Procured two cable assemblies for 32GT/s signaling transmission
      • Samtec 12”, Amphenol 6”
    • Testing with NVIDIA samples with a Dell R760 server

Lab experimentation

    • Long term traffic testing - complete
    • Margin testing – in process

Additional Notes

    • TTF MCU firmware will be contributed to the OCP Github
    • Dell is supportive of sub-group approved changes to the TTF

​

PCIe® 5.0 Thermal Test Fixture (TTF)

OCP NIC 3.0 TTF in use

Example cable assembly from Samtec

Jason

10 of 15

SI Test Fixtures

CLB 5.0 32GT/s status:

    • 2nd set protos
      • Keysight, Broadcom received CLB samples. Keysight provided impedance & SDD21 results (next slide)
      • Impedance ~89 ohms, within manufacturing tolerance
    • First protos impedance mismatch due to manufacturing defect yielding ~95 ohms, out of tolerance
    • Dell will proceed forward to release the design

CBB 5.0 32GT/s status:

    • Gerbered March 3rd , New arrival date: Mid-may
    • Based on 4.0 design; updates to align with PCI-SIG®

OCP NIC 3.0 SFF CLB 5.0 prototype

Calibration

CLB board 1

Impedance Coupon

CLB board 2

Jason

OCP NIC 3.0 SFF CBB 5.0 block diagram

11 of 15

CLB measurements

Impedance (diff pair avg’d)

2 inch

5 inch

10 inch

First board bottom route

96.5 ohms

99.3 ohms

97.1 ohms

First board top route

93.7 ohms

93.9 ohms

97.4 ohms

Second board bottom route

87.6 ohms

88.0 ohms

89.1 ohms

Second board top route

89.6 ohms

91.6 ohms

91.7 ohms

SDD21 @ 16 GHz

2 inch

5 inch

10 inch

First board bottom route

-3.07 dB

-7.18 dB

-13.93 dB

First board top route

-3.20 dB

-7.15 dB

-13.76 dB

Second board bottom route

-2.88 dB

-6.83 dB

-12.92 dB

Second board top route

-2.67 dB

-6.41 dB

-12.80 dB

Bottom

Top

1st board

2 inch

2nd board

2 inch

Bottom

Top

Contributed by Keysight

Requested Top/Bottom differential impedence target = 85.00 ohm +/- 10%

​

1st board manufacturer simulated top = 84.99 ohm

1st board manufacturer simulated bottom = 85.08 ohm

2nd board manufacturer simulated top = 85.10 ohm

2nd board manufacturer simulated bottom = 85.35 ohm

​

12 of 15

Software contributions, Support model

Dell contracted a firm developing MCU firmware for:

  • Thermal test fixture
  • LED test fixture
  • Systems mgmt test fixture

Code base to be contributed to the OCP Github

LED & Systems mgmt HW collateral to be contributed to Wiki

​

Support discussion (cont):

  • UNH IOL offers fully assembled CLBs, CBBs
    • Cannot provide technical support
  • Test fixture user guides posted on our Wiki
    • Dell will author CLB, CBB guides. We welcome volunteers!

Jason

Thermal test fixture console (draft)

13 of 15

14 of 15

Thermal Test Fixture for TSFF feedbacks

Nvidia feedback TSFF has air bypass issue. It was similar on SFF testing fixture:

  • It makes

​

​

Make the thermal performance worse comparing to the single NIC thermal testing

The test fixture has candlestick to measure air velocity (LFM) since NIC3.0 spec cooling based on air velocity

​

The issue is Nvidia found candlestick is inaccurate for low airflow especially for cold-aisle case

15 of 15

Gather feedback throughout the year

Provide your feedback at:�https://tinyurl.com/feedback2ocpnic