1 of 21

David Lawrence

Jun. 22, 2024

1

Experimental Physics Software and Computing Infrastructure

SoLID Collaboration Meeting - Streaming Readout - David Lawrence - Jun. 22, 2024

2 of 21

Streaming Grand Challenge

2

slide from Jan. 5, 2024 presentation by Rolf Ent

The Streaming Grand Challenge began in 2018. �

Significant progress has been made since then on several fronts that include deployment of SRO-capable fast electronics, firmware development, and software (ERSAP, JANA2, InstaRec,...).

​

Phase II of the SRO GC is now beginning.

Experimental Physics Software and Computing Infrastructure

SoLID Collaboration Meeting - Streaming Readout - David Lawrence - Jun. 22, 2024

3 of 21

Streaming Grand Challenge Phase II

​

3

Diagram from early 2024 by Rolf Ent, Markus Diefenthaler, Brad Sawatsky, and David Lawrence

Experimental Physics Software and Computing Infrastructure

SoLID Collaboration Meeting - Streaming Readout - David Lawrence - Jun. 22, 2024

4 of 21

BDX

CLAS12

Hall-C

Q1

Q2

Q3

Q4

Q1

Q2

Q3

Q4

Q1

Q2

Q3

Q4

Q1

Q2

Q3

Q4

Q1

Q2

Q3

Q4

CY2024

CY2025

CY2026

CY2027

CY2028

HMS + SHMS

full scale

DCRB firmware

(capture 2 sectors)

Stream to CC

(partial detector)

PRAD Electronics on loan (1 crate)

Data Stream - partial

(Hall-B to NERSC)

Auto. Calib.

(HMS)

Auto. Calib.

(SHMS)

Legacy Data Formatter

SRO Capable Hardware (all)

Event Filters

Event Identifiers

Auto. Calib.

(FTOF)

Auto. Calib.

(PCAL)

Auto. Calib.

(TOF/CTOF/DC)

Full SRO hardware capability

Auto. Calib.

(ECAL)

Event Identifiers

Event Filters

CLAS12

full scale

RTDP Tool Functional

Auto. Calib.

(RICH/SVT)

SRO GCII

5 of 21

5

SHMS

HMS

CODA

streaming

EJFAT

Filter Module

Filter Module

Filter Module

ET

Data

Lake

monitoring

Analyzer

(recon)

ROOT files

(recon)

Calib.

calib

DB

monitoring

Filter Module

EJFAT reassembly

JANA2

EVIO SRO Parser

Event Identifiers

Legacy Data Formatter

Event Filters

Experimental Physics Software and Computing Infrastructure

SoLID Collaboration Meeting - Streaming Readout - David Lawrence - Jun. 22, 2024

6 of 21

ESnet/JLab FPGA Accelerated Transport

6

Experimental Physics Software and Computing Infrastructure

SoLID Collaboration Meeting - Streaming Readout - David Lawrence - Jun. 22, 2024

7 of 21

GCII: JIRIAF + EJFAT + NERSC Testing

7

80Gbps returning to JLab

100Gbps from JLab to NERSC

Stream exercise sending CLAS12 data from JLab to NERSC using the JIRIAF system​

Virginia

California

CLAS12 Data

(archival)

PI: Vardan Gyurjyan�LDRD Project�(funding FY23, FY24)

PI: Graham Heyes

ASCR Project

Project Lead: Michael Goodrich

Experimental Physics Software and Computing Infrastructure

SoLID Collaboration Meeting - Streaming Readout - David Lawrence - Jun. 22, 2024

8 of 21

ERSAP - Environment for Real-time Streaming, Acquisition and Processing

8

EIC prototype calorimeter SRO pipeline at DESY. CODA & ERSAP

recent beam test at DESY included ERSAP environment for online processing

ERSAP design is event reactive actors, networked by data pipelines.�

• Compositional actors with conditional data routing at runtime.�

• Flow-based programming paradigm

​

Funded through JLab SciOPS

Experimental Physics Software and Computing Infrastructure

SoLID Collaboration Meeting - Streaming Readout - David Lawrence - Jun. 22, 2024

9 of 21

GCII:

9

DPPGE + DPOL (SRO Replacement for JCEdit + COOL)

Graphical Configuration of multi-process, distributed system

Vardan Gyurjyan

Experimental Physics Software and Computing Infrastructure

SoLID Collaboration Meeting - Streaming Readout - David Lawrence - Jun. 22, 2024

10 of 21

GCII: Standard Test Setup

10

Hanjie L. - Hall-C

Dmitry R. - EIC

Brad S. - SciComp+ENP

David L. - EPSCI

Experimental Physics Software and Computing Infrastructure

SoLID Collaboration Meeting - Streaming Readout - David Lawrence - Jun. 22, 2024

11 of 21

Simple Example of a Streaming Readout (SRO) System

11

Highly configurable multi-stream source allows realistic streaming simulations

Onsite components will implement first stages of data filtering/reduction

Offsite processing must incorporate built-in calibration latencies and storage. This will also help inform HPDF design

GCII: Real-Time Development Platform

Experimental Physics Software and Computing Infrastructure

SoLID Collaboration Meeting - Streaming Readout - David Lawrence - Jun. 22, 2024

12 of 21

HOSS!

HOSS = Hall-D Online Skim System

This is responsible for distributing raw data to multiple RAID disks��DAQ system writes to files in RAM disk.

Files transferred using RDMA only after file is closed

Fine time structure naturally introduced wherever buffering is implemented

12

EPJ Web of Conferences 251, 04005 (2021)�https://doi.org/10.1051/epjconf/202125104005

Experimental Physics Software and Computing Infrastructure

SoLID Collaboration Meeting - Streaming Readout - David Lawrence - Jun. 22, 2024

13 of 21

Streaming Data

13

Perception

Reality

Experimental Physics Software and Computing Infrastructure

SoLID Collaboration Meeting - Streaming Readout - David Lawrence - Jun. 22, 2024

14 of 21

GCII: CLAS12 Data Capture

14

May 16th beam test where data streamed from CLAS12 to Data Center and packets captured with hardware timestamps.

beam trip

355 MB/s

(sectors 2,5)

May 2024

(RG-E 5nA)

Dec. 2023

(RG-K 100nA)

Drift Chambers + all FADC250 for sectors 2 and 5

ECAL and PCAL for sector 2

Data stream from VTPs in Hall-B to Data Center

Packets captured with hardware timestamps est. x10s of us resolution

Experimental Physics Software and Computing Infrastructure

SoLID Collaboration Meeting - Streaming Readout - David Lawrence - Jun. 22, 2024

15 of 21

PODIO ROOT Streaming

15

Disclaimer: This is not an efficient way to stream data and will almost certainly not be part of the final ePIC standard streaming configuration. It does provide a continuous streaming source that can be consumed by the current ePIC reconstruction software allowing other components of the streaming system to be sketched out.

Multiple pieces

  1. Modified PODIO (allows data to come in form other than ROOT file on disk)�Added openTDirectory() to ROOTReader as alternative to openFiles()�https://github.com/AIDASoft/podio/issues/565 �https://github.com/AIDASoft/podio/pull/579�
  2. podio2tcp utility (read from ROOT file and send data over network) �Use zmq PUSH-PULL to do automatic load balancing�https://github.com/JeffersonLab/SRO-RTDP/tree/main/src/utilities/cpp/podio2tcp�
  3. podiostream JANA2 event source plugin�Reads events from network and provides them to EICrecon

Experimental Physics Software and Computing Infrastructure

SoLID Collaboration Meeting - Streaming Readout - David Lawrence - Jun. 22, 2024

16 of 21

FABRIC Testbed

16

https://portal.fabric-testbed.net/

Create configuration of network+compute

(includes GPU and FPGA elements)

​

No large HPC resources, but good for testing distributed applications

Experimental Physics Software and Computing Infrastructure

SoLID Collaboration Meeting - Streaming Readout - David Lawrence - Jun. 22, 2024

17 of 21

FABRIC test

17

Tested sending data from CERN to 8 locations in US.

​

Four EICrecon processes running at each location

​

32 consumers total

(event processing rate is few seconds per event)

​

Input file: SIDIS 10x100

/work/eic2/EPIC/EVGEN/SIDIS/pythia6-eic/1.0.0/10x100/q2_0to1/pythia_ep_noradcor_10x100_q2_0.000000001_1.0_run48.ab.hepmc3.tree.root

Experimental Physics Software and Computing Infrastructure

SoLID Collaboration Meeting - Streaming Readout - David Lawrence - Jun. 22, 2024

18 of 21

18

  • Hall-B deployment now rivals Hall-D in online images analyzed.�
  • Mini-Hydra uses labeler only for offline calibration checkout�
  • Interest from NASA at Open House. Meeting scheduled for Thursday

Experimental Physics Software and Computing Infrastructure

SoLID Collaboration Meeting - Streaming Readout - David Lawrence - Jun. 22, 2024

19 of 21

19

Summary

  • Streaming Readout will be the standard for the next generation of experiments at JLab
  • To get to there from here requires a number of pieces be developed and coupled into a coherent system
  • A key requirement is automated calibration of the detector.
    • This needs to be considered during detector development!
  • Monitoring at all stages is needed

Experimental Physics Software and Computing Infrastructure

SoLID Collaboration Meeting - Streaming Readout - David Lawrence - Jun. 22, 2024

20 of 21

20

Experimental Physics Software and Computing Infrastructure

SoLID Collaboration Meeting - Streaming Readout - David Lawrence - Jun. 22, 2024

21 of 21

ESnet/JLab FPGA Accelerated Transport

(part of IRIAD effort)

​

21

Funded through DOE ASCR IRIAD project

PI: Graham Heyes

Project Lead: Michael Goodrich

Partnership between ESnet and JLab to provide high bandwidth, low latency transport of experimental data

​

  • Data packet headers re-written in FPGA to redirect to processing nodes�
  • Smart load balancing by Control Plane application based on telemetry from processing nodes

Experimental Physics Software and Computing Infrastructure

SoLID Collaboration Meeting - Streaming Readout - David Lawrence - Jun. 22, 2024