Corundum status updates

Alex Forencich

12/5/2022

Agenda

• Status updates

Status update summary

• Bugs
• Apparent bug in Xilinx PCIe IP core (in progress)
• FIFO memory inference issue (in progress)
• Xilinx PCIe core MSI-X settings issue (fixed)
• Incorrect in-flight write request limit (fixed)
• APIs for clocks and statistics
• AXI Virtual FIFO (in progress)
• DRAM integration (in progress)
• Batched completion write support (in progress)
• Variable-length descriptor support (todo)
• Management soft core (todo)
• 10G/25G switching (HW done, SW todo)

Bugs: Apparent PCIe HIP bug

• US+ PCIe HIP drops operations under heavy load
• Appears to be related to completion buffer management
• Disabling client tags may be a workaround
• PCIe HIP provides tags, instead of DMA engine
• Status: implemented tag management in HIP model, need to update DMA engine

Bugs: FIFO memory inference issue

• Seeing TX packets with incorrect IP layer checksums, only on Intel devices
• MLE traced the issue to FIFO between TX engine and TX checksum compute block incorrectly setting “enable” bit
• Appears to be a Quartus tool bug related to merging pipeline registers into MLABs
• Connecting RAM output register to logic analyzer or adding “preserve” attribute results in the bug disappearing

Bugs: Xilinx PCIe core MSI-X settings issue

• Issue opened on github reporting that NIC functionality didn’t work when DMA benchmark application was enabled
• Reproduced in HW, mqnic-dump indicated some sort of interrupt issue
• Lspci reported incorrect MSI-X table offsets
• Apparently, changing the BAR config in the Xilinx PCIe IP core blows away the MSI-X settings
• Fixed by re-applying all MSI-X settings after BAR updates in config.tcl

Bugs: Incorrect in-flight write request limit

• While debugging the previous bug, discovered that the DMA write BW was too low (~76 Gbps instead of ~110 Gbps)
• Thought the issue was linked, but it wasn’t…
• Strangely, NIC RX can hit 100 Gbps line rate, unclear why only DMA benchmark module is affected
• Traced problem to low TX limit setting for write DMA module
• Originally used to prevent head-of-line blocking issues, but with credit-based flow control implemented, this is no longer necessary
• Fixed by removing all PCIe DMA parameter overrides
• Default settings should provide highest performance

APIs for clocks and statistics

• APIs added to core kernel module and userspace library for clocking and statistics
• Clocks:
• Read nominal and measured clock frequencies
• Conversion between cycles and ns
• Statistics:
• Read statistics counter at specified index
• TODO: counter allocation and naming

AXI virtual FIFO

• Large packet buffer capability in DRAM
• Store both packet data as well as sideband data (tid, tdest, tuser)
• Intent is to support operation at 100G with all packet sizes
• 2x DDR4 2400 channels or 2x HBM ports
• Main bottleneck is memory BW, so need efficient encoding scheme for framing and sideband data
• Status: core FIFO working in sim, preparing HW test, encode/decode is TODO

DRAM integration

• Provide access to DRAM and HBM from application section
• AXI passthrough for all memory channels
• Dedicated PCIe BAR for host software access and P2P DMA
• Connection to DMA subsystem (unified DMA address space)
• Potentially provide multiple clocking modes
• Fully async (all ports driven directly from interface clocks, min. latency)
• Synchronous (include async FIFOs to sync to first interface clock, core clock, custom clock, etc.)
• TODO
• Intel DDR4 (AXI?), Intel HBM (switching, bursts), 7-series
• PCIe BAR, DMA subsystem connection

RAM BAR/AXI port

• Currently provide AXI lite ports for NIC and app control
• Exposed as PCIe BAR0 and BAR2 on PCIe designs
• Add AXI port + BAR4 to access on-card RAM
• Full AXI supporting bursting and interleaving
• Supports P2P DMA, write combining, etc.
• Can also pass through to application section for low latency operations
• Internally shared with DMA subsystem
• Transparently handle DMA operations based on address

​

DMA benchmark application

• DMA benchmark application is a useful test/sanity check
• Extend DMA benchmark to test more internal components
• Use AXI virtual FIFO to test DDR/HBM channels
• Support DMA benchmark application on all targets
• Status:
• Refactored to put DMA benchmark components in dedicated module
• Updated to use register blocks
• Added for all Alveo boards (100G only)
• TODO: DRAM test, add to other boards

Batched completion write support

• Writing completions separately is inefficient
• PCIe TLP header overhead
• Batch completions in per-queue buffers, write in blocks
• Less overhead
• Can issue block write + event/IRQ + queue pointer write concurrently
• Effectively doubles as interrupt coalescing
• Status: initial version working in HW
• Performance is similar

Baseline (MTU 9000)

• ADM-PCIE-9V3
• 8192 TXQ
• 256 RXQ
• 64 EQ + 64 MSI-X
• 256 PCIe tags w/FC
• 2x Xeon 6130
• MTU 9000
• Writeback off
• MMIO pointers
• No IRQ rate limiting

Writeback on, but not used (MTU 9000)

• ADM-PCIE-9V3
• 8192 TXQ
• 256 RXQ
• 64 EQ + 64 MSI-X
• 256 PCIe tags w/FC
• 2x Xeon 6130
• MTU 9000
• Writeback on
• MMIO pointers
• No IRQ rate limiting

Full queue pointer writeback (MTU 9000)

• ADM-PCIE-9V3
• 8192 TXQ
• 256 RXQ
• 64 EQ + 64 MSI-X
• 256 PCIe tags w/FC
• 2x Xeon 6130
• MTU 9000
• Writeback on
• Writeback pointers
• No IRQ rate limiting

Writeback with IRQ rate limiting (MTU 9000)

• ADM-PCIE-9V3
• 8192 TXQ
• 256 RXQ
• 64 EQ + 64 MSI-X
• 256 PCIe tags w/FC
• 2x Xeon 6130
• MTU 9000
• Writeback on
• Writeback pointers
• IRQ rate limiting (10 us)

New completion write (MTU 9000)

• ADM-PCIE-9V3
• 8192 TXQ
• 256 RXQ
• 64 EQ + 64 MSI-X
• 256 PCIe tags w/FC
• 2x Xeon 6130
• MTU 9000
• New completion write implementation
• IRQ rate limiting

Baseline (MTU 1500)

• ADM-PCIE-9V3
• 8192 TXQ
• 256 RXQ
• 64 EQ + 64 MSI-X
• 256 PCIe tags w/FC
• 2x Xeon 6130
• MTU 1500
• Writeback off
• MMIO pointers
• IRQ rate limiting

New completion write (MTU 1500)

• ADM-PCIE-9V3
• 8192 TXQ
• 256 RXQ
• 64 EQ + 64 MSI-X
• 256 PCIe tags w/FC
• 2x Xeon 6130
• MTU 1500
• New completion write implementation
• IRQ rate limiting

Batched completion write support

• ~64K buffer space, ~64 rings of ~1024 bytes (32 entries)
• Truncated queue pointers so wrap point aligns
• Block write and pointer writeback
• Triggered on half ring boundary in scratch buffer
• Also triggered by event generation
• Event generation
• Triggered by packet count and/or timeout (configurable)

Variable-length descriptor support

• Current queue management logic does not handle variable-length descriptors
• Implement simplified queue managers (state storage only)
• Implement descriptor fetch and header parsing logic
• Read descriptors in blocks up to some block size, parse length/type fields and hand off to transmit/receive engines
• Potential extensions to support LSO (descriptor duplication)
• Status: working on supporting components

Current descriptor format

• Small - 16 bytes
• 32 bit length, 64 bit pointer
• Fixed-size blocks for scatter-gather
• Inflexible
• Extra overhead for small packets

struct mqnic_desc {

__u8 rsvd0[2];

__u16 tx_csum_cmd;

__u32 len;

__u64 addr;

};

Descriptor framing format

• Split framing from contents
• Descriptor fetch only has to parse framing information
• Descriptor format can be modified without changing fetch logic
• 16 byte blocks, 2 byte header
• 256 blocks / 4096 bytes max size
• 1 byte type field to support multiple descriptor formats

struct desc_hdr {

__u8 len;

__u8 type;

}

​

struct desc_block {

__u8 rsvd[16];

};

​

struct desc {

struct desc_hdr desc_hdr;

__u8 rsvd[14];

struct desc_block[];

};

Proposed descriptor format

struct desc_with_inline_data {

struct desc_hdr desc_hdr;

__u16 flags;

__u32 opcode;

__u8 data_seg_count;

__u8 rsvd0;

__u16 inl_data_len;

__u8 rsvd1[4];

union {

struct desc_data_seg data;

char inl_data[];

} segs[] __attribute__ ((aligned(16)));

};

struct desc_hdr {

__u8 len;

__u8 type;

}

​

struct desc_data_seg {

union {

struct desc_hdr desc_hdr;

__u8 rsvd0[2];

};

__u16 flags;

__u16 tx_csum_cmd;

__u16 len;

__u64 addr;

};

Queue state storage

• Some state must be stored in HW for every queue
• Supporting 10K+ queues requires efficient storage
• Utilize URAM blocks on US+ (4K x 64)
• Need to add writeback pointer, VF index, and LSO offset
• Can reclaim a couple of other fields
• Status: Implemented queue state storage module

Queue state storage (current)

URAM is 4096 x 64, so

2 URAM = 4096 queues

Queue state storage (current)

URAM is 4096 x 64, so

3 URAM = 4096 queues

​

Can fit into 2 URAM with writeback disabled and 6 bit VF index

Management soft core

• Soft core for board-level management
• Handle board and device specific functions
• Present unified interface to driver
• Location: board-level or inside core?
• API
• Low-level – direct register access, etc. Board-dependent diagnostics
• Medium-level – Somewhat abstracted operations
• High-level – High-level operations, board and FPGA independent
• Core: probably vexriscv
• JTAG debug interface, can simulate with icarus verilog
• Status: core bring-up

10G/25G switching

• Run-time switching between 10G and 25G should be doable
• Change speeds and mix and match speeds without reloading FPGA
• Run two QPLLs, dynamically switch channel clock sources
• Possibly can also switch EQ settings (DFE vs LPM)
• Requires resetting both RX and TX
• Previously blocked on FIFO reset rework and PHY integration updates
• Status: Working from userspace, need to move to softcore FW
• TODO: update transmit engine to handle dropped TX packets when PTP is enabled

To-do list for stable release

• Application section [done]
• Register space reorganization [done]
• Shared interface datapath [initial version done]
• Self-contained checksum offloading (no checksum in desc)
• Variable-length descriptor support [in progress]
• On-FPGA management soft core (no board-specific kernel code)
• Overall goal: stabilize driver/hardware interface