1 of 14

Corundum status updates

Alex Forencich

9/11/2023

2 of 14

Agenda

  • Announcements
  • Project ideas
  • Status updates

3 of 14

Announcements

  • Next Corundum developer meeting: September 25 at 9:00 AM PDT
  • Next switch development meeting: September 18 at 9:00 AM PDT

4 of 14

Project ideas

  • Some relatively self-contained project ideas:
  • Oversampled 1000BASE-X on a 10GBASE-R transceiver
  • Open-source 40G MAC+PCS
  • MAC+PHY+GT wrappers for switchable 1/10/25/40/100 G
  • RISC-V management core
  • Open-source FT4232 JTAG adapter with Alveo connectors
  • Improved transmit scheduler (WFQ, rate limiting, etc.)
  • SR-IOV support
  • Multi-head support (multiple PCIe and/or AXI host interfaces)
  • Improved application section SoC support

5 of 14

Status update summary

  • MAC control and PFC
  • Corundum updates
  • Potential timestamp handling improvements
  • Architectural changes
    • Single HW interface, split in SW
    • Switch integration
    • Additional app section passthrough

​

6 of 14

MAC control and PFC/pause frames

  • MAC control layer (IEEE 802.3 clause 31)
    • Exchange MAC control messages
  • PFC and pause frame support
    • Hardware flow control for Ethernet
    • Generate/receive PFC and pause frames using MAC control layer
  • Status: Integrated into Corundum
    • Currently, only pause frames are supported
    • Test setup: TX NIC at gen 3 x16, RX NIC at gen 3 x4, RX pauses TX
    • Handling pause frames on RX seems lossless
    • Generating pause frames on TX still results in some retransmits

7 of 14

Corundum updates

  • Internal MAC control layer added to Corundum
    • For use when the MAC does not support pause frames/PFC
  • New port control registers to control LFC/PFC
  • All targets updated to add LFC/PFC connections to MACs
  • Fixed PCIe subsystem vendor IDs on UltraScale
  • Possibly fixed PCIe class code on newer versions of Vivado

8 of 14

Potential timestamp handling improvements

  • High-resolution, time-of-day timestamps are large
    • PTP ToD timestamp is 96 bits (48 bit seconds, 32.16 fixed point ns)
    • PTP relative timestamp is 64 bits (48.16 fixed point ns)
  • MSBs are effectively constant when operating at high data rates
  • Can we save logic resources by truncating and reconstructing the MSBs?
  • Can both timestamp formats be supported efficiently?
  • Have to handle timestamp rollover, CDC, and long wires

​

9 of 14

Supporting both 64 and 96 bit formats

  • 96 bit timestamp is difficult to truncate due to rollover behavior (32 bit ns rolls over at 1 billion)
  • Idea: define 96 bit timestamp as offset of (truncated) 64 bit timestamp
    • Natural binary rollover means offsets are easy to apply
    • Timestamp logic only has to handle one format
    • 64 bit timestamp can be monotonic (no stepping)
    • Offset changes with seconds rollover and with time changes
    • Distribute offset along with full reference timestamps

10 of 14

MSB reconstruction

  • Timestamps are split into two portions with overlap
    • E.g. 48 bit fine, 48+2 bit coarse (2 bits overlap)
    • Overlap defines window for reconstruction (N future, 2^k-N past)
  • Timestamp capture logic only handles fine portion
    • Lower resource consumption, simplifies timestamp math
  • Coarse portion is transferred via slow serial link (similar to SPI)
    • ~1 MHz, easy to send long distances and synchronize
  • PHC shifts out 2 coarse timestamps (current + subsequent)
  • Reconstruction logic stores several timestamps in LUTRAM, indexed via overlap bits
    • Reconstruction is a simple table lookup

11 of 14

Time transfer

  • PTP reference clock distributed to whole device
  • PHC runs “ahead” by a number of clock cycles (e.g. 256)
  • Serial data synchronous to PTP reference clock transfers next PTP time in both formats, step value, etc.
  • Endpoints extract data and latch it into local clock on reference edge
    • Can compensate for pipeline delays with “early” reference edge
    • Can generate whatever timestamp format is required
    • Also have access to full 32-bit fractional ns value

12 of 14

Single HW interface

  • Currently, Corundum supports multiple HW interfaces, with each interface potentially connecting to multiple ports
  • Is this really necessary?
    • New queue management logic trades area for performance
    • Dynamic queue allocation in driver + RX indirection tables mean software can smoothly expose each port as on OS-level interface
    • With embedded switch, it’s likely going to make even more sense to manage this sort of thing in software

13 of 14

Switch integration

  • Sharing code between NIC and switch opens up some interesting possibilities
  • Switchable/splittable 10G/25G/100G ports
    • How to handle splitting ports with application section interfaces?
  • E-switch for internal routing for SRIOV, etc.
    • Additional internal switch ports for application section?
  • Switch likely will use additional clocks
    • Would the application section interface clocking change?
    • Should the PCIe clock be decoupled from the core clock?

​

14 of 14

App section passthrough

  • Some applications require more “stuff” to get passed through to application section
  • Can this be done with macros?
    • Probably need to `include two files
      • Port definitions (included in module port list)
      • Port connections (included in instance port list)
      • What about parameters?

​