1 of 26

Analysis Systems introduction

Luke Kreczko for SWIFT-HEP WP5

​

1

2023.11.22 SWIFT-HEP #6

2 of 26

Good morning!

2

3 of 26

Key points in HEP Data Analysis

Physics

Last mile of long chain of data recording and processing.

Goals: gain insight and create new knowledge

Computing

Analysis workflow (data + software) depends on experiment, analysis group, subset of data (signal + relevant backgrounds), analysis iteration.

Flexibility is paramount.

3

4 of 26

WP5 key points

Seamless Access to Computing Resources

Ease of access to distributed computing resources through user-friendly interfaces and “industry standards”

Efficiency

Caching for fast iterations, portability for mapping algorithms to hardware accelerators (GPU, FPGA)

4

5 of 26

Anatomy of an analysis workflow

​

The cycle (oversimplified)

5

data

Task

Task

Task

data

Task

Task

data

Task

data

New idea/extension of existing work

Publication

create/modify code

Run analysis on laptop/cluster/grid

Understand results

New ideas for improvement,

mistakes identified, or updates

One cycle as short as a day or as long as a month

# The eternal loop of research

while True:

idea = find_new_idea_or_use_existing_work()

code = create_or_modify_code(idea)

result = run_analysis_on_any_computer_available(code)

understanding = try_to_understand_results(result)

if understanding.has_aha_moment():

publish_results(understanding)

celebrate() # Optional: do a little dance

else:

groan() # Sigh deeply and prepare for the next iteration

find_coffee() # Important: Refuel the researcher

​

new_ideas = generate_new_ideas(understanding)

if new_ideas.contains('Eureka'):

break # Break out of the loop to accept your Nobel Prize

else:

feedback = collect_feedback(new_ideas)

apply_feedback_to_idea(feedback)

​

# Sleep is for the weak; let's do another cycle!

time.sleep(for_how_long='Just a blink, I promise')

​

# Note: This loop does not account for weekends or holidays (which are often mythical in research)

​

​

6 of 26

SWIFT-HEP WP 5 in a nutshell

​

WP1

Analysis workflow

WP5

6

Analysis step

output

Data lake

DIRAC

caching

data

Task

Task

Task

data

Task

Task

data

Task

data

Caching Opportunities: The intermediate data at various stages of the workflow are opportune points for caching. By storing these results, we can avoid redundant computations in iterative analysis, thus saving time and resources.

7 of 26

SWIFT-HEP WP 5 in a nutshell

​

WP1

Analysis workflow

WP5

7

Virtual Analysis Facility

​

​

​

​

​

Analysis step

output

Data lake

Heterogeneous hardware

Specialised hardware (GPU, FPGA, etc)

DIRAC

portability

caching

GPU

capable

CPU-only

data

Task

Task

Task

data

Task

Task

data

Task

data

Adaptive Algorithms: automatically detect and utilize GPUs, FPGAs, and other accelerators when available.

​

Fallback Mechanism: In the absence of specialized hardware, algorithms revert to utilizing conventional CPUs without compromising functionality.

​

8 of 26

Scheduling

with

coffea-casa

8

From coffea-casa docs

Uses Dask and dask-jobqueue

The international view: IRIS-HEP

9 of 26

INFN: a CMS project

9

From Analysis facilities forum (28.07.2022)

10 of 26

SWIFT-HEP + GridPP == experiment agnostic?

10

10

*no relation to IRIS-HEP; **no relation to DIRAC

​

​

​

​

​

​

​

As simple as adding DIRAC jobqueue to dask-jobqueue?

Shared Jupyter-hub via iris.ac.uk*

Can run on DiRAC**

Data lake

(WP1)

Virtual analysis facility

(WP1)

11 of 26

Roadmap overview

11

Dask to DIRAC interface (dask-dirac)

Connect to data lake (caching)

Specify resource requirements per analysis component (portability)

Data lake to DIRAC (via Rucio)

Virtual analysis facility

DIRAC workflow manager

Via tags (slide 26)

REST API

FileCatalog?

Closes example of what we want to achieve: Dask-based Distributed Analysis Facility (kubernetes slides)

Caching at analysis step level

WP1

WP5

1

2

3

4

12 of 26

Roadmap overview

12

Dask to DIRAC interface (dask-dirac)

Connect to data lake (caching)

Specify resource requirements per analysis component (portability)

Data lake to DIRAC (via Rucio)

Virtual analysis facility

DIRAC workflow manager

Via tags (slide 26)

REST API

FileCatalog?

Closes example of what we want to achieve: Dask-based Distributed Analysis Facility (kubernetes slides)

Caching at analysis step level

WP1

WP5

1

2

3

4

More on this in Sam’s talk

13 of 26

Analysis Grand Challenges

-

The medium for testing

13

14 of 26

Analysis Grand Challenges (IRIS-HEP)

IRIS-HEP are planning to verify work through several analysis grand challenges

Aiming for a realistic workflow, e.g.

  • Existing analysis, their example: Higgs → tau tau
  • Approx 200 TB of input data, their example: CMS NanoAOD
  • Testing performance (speed, resource usage)
  • Outputs: statistical inference, tables, control plots, HEP Data
  • Other metrics: reproducibility of results (e.g. with REANA)

→ ACG repo: https://github.com/iris-hep/analysis-grand-challenge

​

​

14

15 of 26

SWIFT-HEP Phase 2

15

16 of 26

Phase 2?

Phase 1 is a technology test/prototype

Phase 2 would be the production system:

Jupyter-hub with IRIS IAM on STFC cloud***

Sample of analyses beyond ACGs

Test phase 1 at scale

16

*no relation to IRIS-HEP; **no relation to DIRAC

Shared Jupyter-hub via iris.ac.uk*

Can run on DiRAC**

Data lake

(WP1)

Virtual analysis facility

(WP1)

✅

✅

*** These could also be institutional resources with access to both local and GridPP resources

17 of 26

Phase 2 Roadmap overview

17

Jupyter-hub with IRIS IAM

Workflow converter (to Dask)

Data lake to DIRAC (via Rucio)

Virtual analysis facility

DIRACX workflow manager

REST API

Would also include optimising caching and portability as analysis sample size increases.

In other words: phase 2 is a superset of phase 1.

Dask-dirac interface

WP1

WP5

1

2

4

18 of 26

Summary

and

Outlook

In Phase 1 WP5 aims for a prototype demonstrating the key goals on WP1 deliverables

WP5 is in progress, using AGCs as benchmark - details in Sam’s talk

Phase 2 will look towards a production system:

  • Extend capabilities beyond ACG benchmarks to encompass wider analyses
  • Integrate JupyterHub or an equivalent platform to provide a user-friendly interface for researchers.

18

19 of 26

Backup slides

19

20 of 26

INFN: a CMS project

20

From Analysis facilities forum (28.07.2022)

21 of 26

But:

21

Is there a way to standardize this functionality and make it available to everyone?

22 of 26

Dask network layer simplified

22

23 of 26

Dask network layer simplified

23

mermaidjs source​

24 of 26

Dask network layer simplified

24

So where is the problem?​

​

Imagine network boundary between scheduler and workers​

​

Scheduler port is accessible from workers​

​

Worker port is ONLY accessible to scheduler if connection is recycled (part of ESTABLISHED --> firewall OK)​

​

Default Dask operation: this can happen at RANDOM (most likely for small # of workers)​

25 of 26

Dask network layer: A general fix​

25

We know connections can be recycled and bypass firewall if they are part of an ESTABLISHED connection​

​

We also know of a working solution in our field: The HTCondor Connection Broker​

​

Workers, schedulers, etc connect to a SHARED_PORT​

​

As long as SHARED_PORT is open in firewall on a node accessible to both scheduler and workers --> connection can be established​

​

Most simple solution: Can the Dask Connection proxy be rewritten to hold worker connections?​

​

What are the downsides for 100-1000 worker nodes?​

​

​

26 of 26

CPU vs accelerators

No clear optimal way yet, first draft will require multiple versions of a “stage”:

26

register:

my_namespace::my_stage:

cpu: my_module.my_stage

gpu: my_module.my_stage_gpu

GPU version is used if a GPU is detected, CPU version otherwise