Corundum status updates

Alex Forencich

6/4/2025

Agenda

• Announcements
• Status

Announcements

• Next dev meeting: July 2 at 9 AM PDT
• Next switch meeting: June 11 at 9 AM PDT

​

• Corundum developer meetings:
• 1^st Wednesday of each month, 9 AM PDT
• Dev meetings for 2025: July 2, Aug 6, Sept 3, Oct 1, Nov 5, Dec 3
• Switch meetings:
• 2^nd and 4^th Wednesday of each month, 9 AM PDT
• Google calendar linked on wiki page

Status

• Deprecations
• SV rewrite progress
• New MAC+PHY+GT wrapper
• 10G/25G Eth low latency mode (sync gearbox)
• Corundum-ng
• New network stack
• Versal architecture notes
• VCK5000 + Vivado

Deprecations

• Added deprecation notices to verilog-ethernet, verilog-axi, verilog-axis, verilog-uart, and verilog-i2c
• All of these repos will be superseded by taxi
• No further development or support will be provided for the old permissively-licensed code
• Future development will be sustained by licensing the new system verilog code
• Permission from MLE to use their license agreement
• Terms and rates are negotiable
• Porting assistance also available

SV rewrite

• Rewrite corundum and all libraries in SV, change to CERN OHL strongly reciprocal license
• Library porting: new Taxi transport library
• Available on GitHub now: https://github.com/fpganinja/taxi
• Wrote SV interfaces for AXI/AXI stream/AXI lite
• Porting core library modules is progressing
• New MAC+PCS+GT wrappers for 10G/25G
• Example designs added for several different boards

SV tooling status

• Icarus Verilog
• Interfaces not usable (no modports????)
• Verilator
• Finally works with cocotb! SV interfaces also work!
• Vivado
• Works in Vivado 2021.2 (intentionally not using the latest Vivado)
• Quartus
• Builds on Quartus Prime Pro 23.1 (have not tested in HW yet)
• Does NOT build on Quartus Prime Lite/Standard (very old SV)
• Sv2v
• Possibly an option for Quartus Lite/Standard, maybe even ISE
• Currently there are a few bugs in sv2v

SV interface improvement

Porting progress

• Done: Most AXI-stream, AXI RAM, tri-mode MACs, 10G/25G MACs, PTP, LFSR, UART, MDIO, I2C, XFCP
• To do (short term): AXI/AXI lite crossbars, adapters, DMA, PCIe, sync primitives
• To do (long term): New UDP stack

Example designs

• “Kitchen sink” designs that bring up many peripherals
• Current designs light up Ethernet ports + UARTs
• Simple loopback, works well with network tester
• XFCP on UART for monitoring/statistics and control
• Eventually will bring up external DRAM and add AXI infrastructure components
• TODO:
• Intel parts (need wrapper)
• Boards that need I2C bring-up (need to port I2C init code)
• FMC (6x QSFP28/10x SFP+)

New MAC+PHY+GT wrapper

• New wrapper uses combined MAC+PHY module
• Lower footprint, lower latency
• Utilize modular design + CDC primitives
• Simulation mode (remove GT instance) for CI, etc.
• Done
• Integrated statistics
• Custom reset logic
• Low latency GT wrapper, sync gearbox, buffer bypass
• TODO
• DRP interface logic
• Similar wrappers for Intel parts
• Support 1G/10G/25G/100G, AN, SGMII (1G), USXGMII (10G)
• Integrated qJTAG
• Integrated high-resolution timestamping

10G/25G Eth Low Latency Mode

• Added CFG_LOW_LATENCY parameter to MAC+PHY+GT
• Added new low-latency GT wrapper
• TCL scripts generate both GT variants
• Only need to change the parameter
• MAC+PHY and common PHY logic updated to support gearboxes
• Timing and area trade-off
• 161/403 vs 156/390 MHz, sync and stall signals, clock enables, etc.

Gearbox

• Problem: 64b/66b line code operates on 66-bit blocks
• Serializers are generally powers of 2 (+ maybe 5), but 66 = 2*3*11
• Solution: use 64 bit serdes, re-pack 64 bits into 66 bits
• Problem: need stall cycles
• Sync gearbox: datapath has to stall appropriately
• Effective latency varies cycle-by-cycle
• Async gearbox: add a FIFO and use a different clock division to absorb the stalls in the FIFO
• Latency effectively constant cycle-to-cycle, but is difficult to characterize and varies between resets, link disruptions, etc.

Synchronous gearbox interface

• RX: easy, just add “valid” signal and pipeline appropriately
• TX: need the correct sequence at the end of the transmit pipeline
• Create a “loop” from the GT (wrapper) and back
• gbx_req_stall – stall request, datapath stalls when high
• gbx_req_start – sequence start, datapath forwards this signal
• gbx_start – start signal from datapath
• gbx_req_stall/gbx_req_start generated with counters
• gbx_start used to generate txsequence for GTH/GTY

Synchronous gearbox interface

Synchronous gearbox interface

• Status:
• Main datapath modifications are complete
• Tested on VCU118, VCU108, KCU105, and ZCU106
• Have a strategy to track the RX gearbox state
• TODO: TX and RX timestamp correction, optimizations
• Longer-term TODO: ultra low latency with fabric gearbox?
• 7-series support: due to differences in FPGA clocking primitives (need 16 clocks per quad), it’s unclear what the best approach will be, so tabling this for now

“Next generation” Corundum

• Three variants:
• Corundum-micro
• Intended for SoCs, size-optimized, supports ~1-4 ports of ~1G Eth
• Corundum-lite
• Balanced performance, 1-4 ports of 10/25G Eth, maybe 100G
• Corundum-ng
• Max performance, 100-400G Ethernet
• All will use the same driver and support timestamping, checksum and segmentation offloading

Corundum-micro

• Size-optimized design, ~1-4 ports at ~1 Gbps
• Lots of cycles per packet at 1 Gbps -> micro coded design
• Intended for SoC -> AXI interface, but could also support PCIe
• Supports basic NIC offloads (segmentation and checksum), scatter/gather DMA, and timestamping
• Primarily a self-contained core
• May be a good method for prototyping the new descriptor format and developing the driver

Corundum-lite

• Balanced design, ~1-4 ports at 10G/25G and possibly 100G
• Supports multiple TX/RX queues, SR-IOV
• Both PCIe and AXI for the host interface
• May support PCIe/AXI dual head (e.g. for MPSoC boards)
• May support application section
• May be available as a self-contained core

Corundum-ng

• Maximum performance, 100G-400G
• Target Intel Agilex I R-tile (500 MHz, 1024 bit)
• Will support other devices as well
• Multi-headed PCIe? (for dual gen 4/5 x8 mode)
• May support PCIe/AXI dual head (Zynq/Versal)
• Local AXI DMA? (DRAM, HBM)
• Application section

​

New network stack

• Old UDP/IP stack has some significant limitations
• Build a more flexible replacement
• Support IP and UDP natively, have a mechanism for supporting additional protocols like TCP and RDMA
• More of a socket-style interface
• Configuration via register interface
• ARP, etc. handled either by a soft core or a more compact state machine
• Embedded RDMA likely on the roadmap

Versal Architecture notes

• Versal devices, like Zynq, are not FPGAs, they are ARM SoCs with an eFPGA
• They boot like an SoC, the SoC has dedicated peripherals and IO pins, the SoC portion can do quite a bit with the FPGA “blank”
• Internal network on chip (NoC)
• Two different PCIe subsystems
• Hardened XDMA/QDMA
• How would Corundum run on Versal?

Versal PCIe

• Versal has two different PCIe blocks: PCIE4/5 and CPM4/5
• Either PCIE4 only, PCIE4+CPM4, or PCIE5+CPM5
• PCIE4: Basically PCIE4C from US+, located in PL, TLP interfaces, up to gen 3 x16 or gen 4 x8 (~128 Gbps total)
• PCIE5: Same as PCIE4 but adds gen 5 x4
• CPM4: Two PCIe controllers, connects to NoC or PL (TLP)
• Supports gen 3 x16 or (dual) gen 4 x8
• Hardened XDMA and QDMA
• CPM5: Same as CPM4, but adds gen 4 x16 and gen 5 x8
• Dedicated transceivers

Versal CPM notes

• CPM4: 512 bit TLP interface to PL (per controller)
• On highest grade devices, can do gen 4 x16 with XDMA/QDMA
• CPM5: 1024 bit TLP interface to PL (per controller)
• Both CPM4 and CPM5 support hardened QDMA and AXI bridge modes
• Doesn’t seem like QDMA can be used as a “datamover”
• AXI bridge mode effectively useless due to no direct access to PCIe address space (“AXI BAR”) and no real way to map ops to PFs/VFs
• Conclusion: only realistic option is bifurcated CPM4/CPM5 in TLP mode

VCK5000 + Vivado

• PCIe form factor card, Versal VC1902 part
• Initially introduced at a low price for AI applications ($2745)
• Two SKUs were available, one “AI-engine only” and a more expensive one which can be used with Vivado
• These units support secure boot features like AES and RSA - was one of these features used to “lock” the cards to only manufacturer-supplied bitstreams?

VCK5000 + Vivado

• I recently acquired several VCK5000 ES1 boards, marked “encryption disabled”
• Most/all security features are disabled
• Unsure if it’s the same situation with production boards
• Theoretically, if they aren’t locked they should simply be usable with Vivado
• However, there are no public board files or documentation
• Need to determine pinout, settings, etc. to use it
• Is this just security-through-obscurity?

VCK5000 + Vivado

• Versal is an ARM SoC – many things are fixed
• Boot is via OSPI on fixed pins, hard DDRMC use fixed pins, etc.
• Not a whole lot of stuff on the board
• It does look kinda like an Alveo – MSP430, 30 pin header

VCK5000 + Vivado

• First attempt – Successfully generated bitstream, but got a PLM stall when loading it
• Apparently there is debug info from the PMC during boot
• After excluding OSPI pins, only ~4 settings, one worked
• Debug info indicated PL was not starting up
• Enabled sysmon channels, found VCC_RAM was reading 0
• Maybe the MSP430 is waiting for something?

VCK5000 + Vivado

• The card is running a good image, maybe the settings can be extracted from that
• Versal is an ARM SoC – Vivado cannot read the OSPI flash
• Versal is an ARM SoC – can connect to the ARM cores from Vitis and read anything we want, including GPIO mux settings
• Found some other peripherals, and one MIO pin driving high
• After adding that MIO pin, the design booted successfully
• Next steps: bring up PCIe and Ethernet interfaces