1 of 10

EJ-FAT

Stacey Sheldon

Yatish Kumar, Michael Goodrich, Graham Heyes

2 of 10

End to End science problem and motivation

*

SC22 | Dallas, TX | hpc accelerates.

2

Compute Node

ESNET

DOE needs to stream N x 400G A/D samples from very large instruments to its HPC / Supercomputers for processing

Extreme Scale Streaming ( 400G / 800G / 1.2T … )

Very long latencies across North America ( 30 ms / 60 ms … )

No time for retransmission. Data loss is treated as “noise” and compute keeps going

Compute Node

DAQ

3 of 10

End to End science problem and motivation

*

SC22 | Dallas, TX | hpc accelerates.

3

DOE has dozens of large instruments spread out across the country
Each instrument will produce Nx400G streams of DAQ data based on emerging sensors
This requires real time processing in order to:

Preview and tune the experiment to ensure well focused beams, x-rays and samples
Use compute to identify events, particles, image frames etc.

Select what to store
Discard the raw DAQs for up to 1000000:1 data reduction

Compute Node

ESNET

Compute Node

DAQ

4 of 10

Design Goals

4

DAQ

EJ-FAT

FPGA + Switch

Compute Node

ESNET

Unidirectional UDP Stream

1) Separation of IP Addresses

2) In network sorting of Event Data

3) Stateless load balancing

4) Compute node feedback for dynamic LB

5) Hit-less reconfiguration of LB table

6) Unidirectional UDP streaming

AS: 345 203.58.0.0/16

AS: 999 102.12.2.0/8

Event Q Feedback

5 of 10

Implementation - Data Plane Packet Rewrite

5

Eth

LB IP

UDP

LB

Data

Eth	LB Dst MAC	Next HOP Router / CN MAC
IP	LB Dst IP	CN Dst IP
UDP	LB well known port	CN UDP Dst port
LB	Event ID	— Header Popped

Eth

CN IP

UDP

Data

** Fine Print : Src IP rewritten to LB IP, ARP/ND used to resolve next-hop Dst MAC, UDP Dst port used to select a CPU within the CN, LB responds to upstream ARP/ND/ping, packets dropped if wrong Eth, IP addr/port or LB header.

6 of 10

Implementation - Data Plane Selecting a CN

6

Event ID

0

8

63

> 53,000	last epoch
> 209,000	this epoch
TBD	next epoch
unused	—-

0	CN #A
1	CN #B
…	CN #B
511	CN #Z

CN A	ETH	IP	UDP
CN B	ETH	IP	UDP
CN Y	ETH	IP	UDP
etc..

9

Select Epoch

Select

Calendar

Slot

0	CN #Y
1	CN #Y
…	…
511	CN #Z

0	CN #Y
1	CN #O
…	…
511	CN #W

Compute Node Details

Multiple active epoch tables ensures make before break forwarding
Atomic transition between epochs on event # boundaries
Multiple entries for the same CN controls “weight” of that CN
512 entry epoch table balances table size vs. # CN

it is adjustable by recompiling the FPGA

Indirection for CN Details, helps compress memory and speeds up table programming

7 of 10

Implementation - Control Plane Updating the LB

7

Time

0	CN 0
1	CN 1
2	CN 2
N	CN 1

Start with a default LB

CN 0	oooo______
CN 1	oooooooo__
CN 2	o_________

Monitor CN Q Fill

Build a new epoch table

Activate New Epoch

Release Old Epoch

Wait

0	CN 0
1	CN 1
2	CN 2
N	CN 1

0	CN 0
1	CN 1
2	CN 2
N	CN 1

CN 0	oooo______
CN 1	ooooo_____
CN 2	oooo______

Monitor CN Q Fill

0	CN 0
1	CN 1
2	CN 2
N	CN 2

0	CN 0
1	CN 1
2	CN 2
N	CN 2

0	CN 0
1	CN 1
2	CN 2
N	CN 2

8 of 10

Test Setup

8

DAQ Source Emulator

Real Load Balancer HW

Compute Node

DAQ Packet Sequence Generator SW

DAQ.pcapng

DPDK pcap Replay

- multi DAQ

- network jitter

- random order DAQ

- event overlap

FPGA Server

Compute Node Server

U280 FPGA

PCIe @ 100Gpbs�Packet Inject�+ P4 Table Config

tcpdump

- HW TS (ns)

- 100Gbps Rx

CN.pcapng

Ethernet @ 100Gpbs

This is the physical setup for the test results in this paper:� - It is a packet level test centered on the LB

Streaming has been done in a larger system, with full science streaming and reconstruction

9 of 10

Test Results

9

10 of 10

Thank You

Contact : smartnic@es.net

Implemented using : https://github.com/esnet/esnet-smartnic-hw

10