1 of 34

IRIS-HEP�US ATLAS

L2/L3 Managers

MEETING

OAC-1836650

�

Peter Elmer

(Original slides from

Gordon Watts)

2 of 34

One thing…

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

2

Microsoft’s PowerPoint AI suggested this background after typing “US CMS IRIS-HEP”

3 of 34

IRIS-HEP

EVERYONE PROBABLY KNOWS THIS

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

3

4 of 34

IRIS-HEP

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

4

PI: Peter Elmer (Princeton), co-PIs: Brian Bockelman (Morgridge Institute), Gordon Watts (U.Washington) with

UC-Berkeley, University of Chicago, University of Cincinnati, Cornell University, Indiana University, MIT, U.Michigan-Ann Arbor, U.Nebraska-Lincoln, New York University, Stanford University, UC-Santa Cruz, UC-San Diego, U.Illinois at Urbana-Champaign, U.Puerto Rico-Mayaguez and U.Wisconsin-Madison

18 Universities across the USA

~28 FTE’s spread across ~60 people

Computational and data science research to enable discoveries in fundamental physics

Designed to address key elements of the Roadmap for HEP Software and Computing R&D for the 2020’s.

5 of 34

IRIS-HEP

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

5

Institute Aims, in brief:

  • Enable new approaches to computing and software that maximize, and potentially radically extend, the physics reach of the detectors.
  • Make improvements in software efficiency, scalability and performance and make use of the advances in CPU, storage and network technologies, that allow the experiments to maximize their physics reach within their computing budgets.
  • Significantly improve the long-term sustainability of the software through the lifetime of the HL-LHC.
  • Play the role of intellectual hub for the larger software R&D effort required to ensure the success of the HL-LHC scientific program
  • Lead research into deployment of the resulting systems on distributed high throughput computing.
  • Work to improve scientific software more broadly and create opportunities for a more diverse participation in scientific software and computing.

Part of the measuring stick we use to decide what projects and challenges to take on

6 of 34

What are we up to?

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

6

http://iris-hep.org

Details and contacts of all many of the projects we are involved in.

7 of 34

IRIS-HEP R&D Overview

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

7

As an organization we operate at all levels

8 of 34

Multi Experiment

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

8

We strive for solutions useful to the LHC community*

*as best we can

9 of 34

Multi Experiment

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

9

With an eye towards an even broader community

10 of 34

Structure and Executive Board

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

10

And

Mike Hildreth

And

Kaushik De

11 of 34

Steering Board

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

11

Represents the major stakeholders and partners for the IRIS-HEP project. Will meet quarterly with the IRIS-HEP Executive Board to learn the status of the project and provide feedback on the large-scale priorities and current strategy of the Institute.

The steering board meets quarterly with the executive board:

​

12 of 34

Advisory Board

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

12

Provides annual non-stakeholder feedback on the goals and evolving project plans, and evaluates how well the Institute, as a project, is achieving its overall mission as defined with NSF. The Advisory Panel consists of 7 fixed members with an option of inviting ad-hoc additional members as needed for particular topics.

​

The first in-person meeting with the Advisory Panel took place on 9 September, 2019:

​

​

Extremely useful feedback for us as a project as we began to get critical mass in terms of staffing.

The next Advisory Panel meeting will take place in summer, 2020, after the IRIS-HEP Retreat and Strategic Planning exercise.

Maybe (COVID-19)

13 of 34

Project Round Up

PROJECTS WE ARE INVOLVED IN, LEADING, ETC.

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

13

14 of 34

Innovative Algorithms

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

14

Algorithms for real-time processing of detector data in the software trigger and offline reconstruction are critical components of HEP’s computing challenge.

IA Project Page

ACTS – Experiment independent, parallel, track reconstruction

exploratory-ml – Using ML to transform analysis workflows

FastPID – Using generative models to simulate PID detectors in LHCb

GPU Trigger – Allen GPU trigger framework for LHCb

ML For Jet Physics – Using ML for jet taggers, boosted objects, etc.

mkFit – Fully Vectorized and Parallel Kalman Filter for use in collider experiments.

ML on FPGA – Fast inference for use in low latency environments like a L1 trigger.

15 of 34

DOMA

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

15

XCache – Regional caches to store, on-demand, datasets (CMS/ATLAS T2 Facilities)

iDDS – Moving from byte/file level delivery to event delivery

Modeling Data Workflows – Using Run 2 to predict Run 3, 4, and beyond

ServiceX – Delivering columnar data on demand.

SkyhookDM – Ceph for selection, projection, aggregation, indexing of data

TPC – Third Party Copy – using modern protocols to move data between datacenters

The DOMA focus area performs fundamental R&D related to the central challenges of organizing, managing, and providing access to exabytes of data from processing systems of various kinds.

16 of 34

Analysis Systems

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

16

Develop sustainable analysis tools to extend the physics reach of the HL-LHC experiments: create greater functionality to enable new techniques; reducing time-to-insight and physics; lowering the barriers for smaller teams; and streamlining analysis preservation, reproducibility, and reuse.

ADL Benchmarks – Comparison data query tasks in different Analysis Description languages

AMPGEN – Fitting, multibody decays using isobar model (LHCb)

FunctionalADL – A Functional Analysis Description Language with SQL roots

Awesome-hep – Awesome list of high energy and particle physics software.

Awkward Array – Hierarchical numpy

DecayLanguage – Convert particle decay descriptions between digital representations

Histogram Projects – boost-histogram, Aghast, etc.

MadMiner – Likelihood free inference from Monte Carlo and simulation

17 of 34

Analysis Systems

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

17

Develop sustainable analysis tools to extend the physics reach of the HL-LHC experiments: create greater functionality to enable new techniques; reducing time-to-insight and physics; lowering the barriers for smaller teams; and streamlining analysis preservation, reproducibility, and reuse.

Particle – Accessing PDG info in python

PPX – Cross-platform Probabilistic Programming eXecution protocol for MC to Inference Engine connections

Scikit-HEP – Python package which brings in common HEP tools

pyhf – Python implementation of HistFactory (including toys!) based on TensorFlow and PyTorch as backends

RECAST – Framework for live-archiving of existing analyses

ROOT-conda – installing root from conda forge

uproot – Loads TTree’s from ROOT files into awkward arrays (writes too!)

18 of 34

SSL

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

18

Provide the Institute and the HL-LHC experiments with scalable platforms needed for development in context, perform facilities and systems R&D.

It has grown to more than that:

  • Leading the charge to re-imagine the compute facility
  • Developing a common substrate for HEP
  • Flexible computing, optimal use of resources, minimal sysadmin effort

River (UChicago): REANA, ServiceX, ATLAS analytics, COVID calculations, training platform, the SLATE project. Backfilled by OSG.

19 of 34

OSG

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

19

  • The OSG is a consortium dedicated to the advancement of all of open science via the practice of Distributed High Throughput Computing, and the advancement of its state of the art.
  • IRIS-HEP is the funding mechanism to support LHC needs from the consortium.
    • Effort form IRIS-HEP is roughly ⅓ of the total effort in OSG today.
  • At a high level policy, only shared interests between US ATLAS and US CMS ops programs are within scope of the OSG effort in IRIS-HEP
    • There are activities in the OSG consortium more broadly that serve multiple domains, DOE-NP, and cosmic and intensity frontier experiments in DOE-HEP, plus one of the experiments but not the other. Such activities are not within scope of IRIS-HEP.
  • A lot of “operations” R&D goes on inside OSG
    • Globus transition
    • Packages are constantly being retired
    • New workflows – e.g. containers, ML, GPU’s
    • Works tightly with DOMA and SSL as a result

20 of 34

Blueprint Workshops

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

20

  • 21 Jun - 22 Jun, 2019 - Blueprint: Analysis Systems R&D on Scalable Platforms (NYU)
  • 10 Sep - 11 Sep, 2019 - Blueprint: Accelerated Machine Learning and Inference (Fermilab)
  • 23 Oct - 25 Oct, 2019 - Blueprint: A Coordinated Ecosystem for HL-LHC Computing R&D (Catholic University of America, Washington DC)
  • 20 Feb, 2020 - Training Blueprint (Vidyo)

Small workshops 3-4 times a year with experts from outside and inside the field to facilitate effective collaborations by building and maintaining a common vision.

See indico for workshops material

21 of 34

Training

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

21

How to build a sustainable and scalable training framework that grows skills through multiple stages of people’s careers?

Collaborate with FIRST-HEP and Carpentries

CoDaS summer school, FIRST-HEP/ATLAS software training, ML hackathon at University of Puerto Rico, Software Carpentry Workshop, etc.

A number of our members took these courses and have become teachers. SSL often provides infrastructure for the training.

22 of 34

Towards Year 3

AND BEYOND…

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

22

23 of 34

High Level Overview

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

23

  1. The NSF 18 Month Review Results
  2. Focus Area Discussions
  3. PI Discussions
  4. US Ops Programs Discussion
  5. Steering Board Meeting
  6. Full-Team Retreat
  7. PEP Draft

This is a conversation leading to Year 3 and out plans

24 of 34

High Level Overview

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

24

  1. The NSF 18 Month Review Results
  2. Focus Area Discussions
  3. PI Discussions
  4. US Ops Programs Discussion
  5. Steering Board Meeting
  6. Full-Team Retreat
  7. PEP Draft
  • Successful (thank you for your help!)
  • Better Metrics and Milestones
  • Grand Challenges
  • Evidence of impact outside of HEP

25 of 34

High Level Overview

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

25

  1. The NSF 18 Month Review Results
  2. Focus Area Discussions
  3. PI Discussions
  4. US Ops Programs Discussion
  5. Steering Board Meeting
  6. Full-Team Retreat
  7. PEP Draft
  • Focus Area Leads view of how things are progressing
  • In progress
  • Questions like – how are people performing, future grand challenge ideas, etc.

26 of 34

High Level Overview

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

26

  1. The NSF 18 Month Review Results
  2. Focus Area Discussions
  3. PI Discussions
  4. US Ops Programs Discussion
  5. Steering Board Meeting
  6. Full-Team Retreat
  7. PEP Draft
  • PI’s discuss their team’s progress in the various focus areas, their vision for the future, how it interfaces with the focus areas
  • Will start in a week or so
  • Questions look very similar to the Focus area questions

27 of 34

High Level Overview

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

27

  1. The NSF 18 Month Review Results
  2. Focus Area Discussions
  3. PI Discussions
  4. US Ops Programs Discussion
  5. Steering Board Meeting
  6. Full-Team Retreat
  7. PEP Draft
  • This meeting and the same one with US ATLAS
  • Discussions & interaction at the full-team retreat
  • Any other interactions you think would be good

28 of 34

High Level Overview

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

28

  1. The NSF 18 Month Review Results
  2. Focus Area Discussions
  3. PI Discussions
  4. US Ops Programs Discussion
  5. Steering Board Meeting
  6. Full-Team Retreat
  7. PEP Draft
  • Occurs on May 26 - indico
  • Y3 plans will be the main theme

29 of 34

High Level Overview

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

29

  1. The NSF 18 Month Review Results
  2. Focus Area Discussions
  3. PI Discussions
  4. US Ops Programs Discussion
  5. Steering Board Meeting
  6. Full-Team Retreat
  7. PEP Draft
  • Occurs on May 26-28 - indico
  • Virtual Meeting – in planning stages
  • US OPS welcome
  • Focus area talks with PI input to present a first draft of Y3 vision
    • With direction for out years
  • Presentations by full team
  • Result: Content of draft PEP

30 of 34

High Level Overview

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

30

  1. The NSF 18 Month Review Results
  2. Focus Area Discussions
  3. PI Discussions
  4. US Ops Programs Discussion
  5. Steering Board Meeting
  6. Full-Team Retreat
  7. PEP Draft
  • Iterated over by PI’s
  • Given to the NSF
  • Iterate until approved
  • Year 3 starts Sept 1, 2020

31 of 34

Brainstorming…

I’VE PROVIDED SOME STRUCTURE, BUT IT ISN’T MEANT TO BE LIMITING…

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

31

32 of 34

Challenges

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

32

  • 10 PB Data reconstruction in one day
    • Connection to a “data lake”? Rucio as a source? iDDS?
    • Does data lake need a blueprint to get the various experiments talking? Is that something that naturally happens in the WLCG?
    • OSG and ESNET connections?

“Grand” just means it involves many areas of IRIS-HEP

Some ideas to get discussion started…

  • Vertical slice of analysis chain – from ServiceX to fitted limit plots
    • 500TB?
    • Could we make this auto-diff’able?
  • Distributed SSL facility, 3 centers, common k8 substrate

33 of 34

Workshops, Training, Blueprint

P.Elmer (Princeton) from slides by G. Watts (UW/SEATTLE)

33

Blueprints to focus joint work with CMS – especially if it might involve outside experts

Common, topical workshops

Joint training efforts

Connections to Snowmass?

Evolution of connections to CCE?

Connections to any funded NSF AI Institute?

​

34 of 34

COVID-19

Not directly related to IRIS-HEP/US-ATLAS collaboration, but there are also efforts to (1) find ways to contribute “Big Science” experience and capabilities as part of responding to the many COVID-19 challenges and (2) understanding how research activities can be effective “in remote”.

​

“Science Responds” (https://science-responds.org/) set up to enable discussions, promote things that are being done….