1 of 22

Software & HSF Status Report

LHCC, February 2024

P. Buncic (EP-SFT), E. Rodrigues (University of Liverpool)

2 of 22

SFT Projects

3 of 22

Summary of evolutionary changes in EP-SFT group

  • New GL (G Ganis), DGL (P Buncic) since 01/07/2023
  • Changes in group organization
    • Simulation: flattened structure (new PL: A Ribon)
    • Stacks (previously known as SPI), including Key4Hep (new PL: A Sailer)
    • Starting of SFTLabs: cradle for ideas, <20% member time
      • Currently: investigate possible role of Julia
    • Change of ROOT PL
      • A Naumann took WP1 coordination role in the NGT (Next Generation Triggers) project
      • New ROOT PL: D Piparo
    • New Machine Learning Project
      • Expectations
        • Attract interest from developing programs, e.g. NGT
        • Act as synergising point for solutions developed in the community
        • Build-up critical expertise around available technologies
      • Will start from re-arranging current ML related activities from Simulation and ROOT
      • Project Leader: Lorenzo Moneta

3

4 of 22

SFT new projects view

4

Simulation Tools

​

​

Geant4

Analysis And Processing Tools

​

ROOT

Software Distribution Tools

​

CernVM-FS

Software Stacks / SPI

Coordination Activities (HSF, EP R&D, …)

ALICE

ATLAS

CMS

LHCb

EP R&D

IT projects

HSF

FCC

…

Stakeholders

ML4EP

5 of 22

SFT PoW

5

  • One day Program of Work meeting held on 15/01/2024
  • SFT Project presented they planned Program of Work for 2024
  • The experiments representatives invited to the meeting and asked to provide a consolidated feedback
  • The feedback was received at Architects Forum meeting on 01/02/2024

6 of 22

SFT PoW Feedback Summary

6

7 of 22

SFT PoW Feedback Summary

7

8 of 22

SFT PoW Feedback Summary

8

9 of 22

SFT PoW Feedback Summary

9

10 of 22

SFT PoW Feedback Summary

10

11 of 22

Analysis: Organization & Personpower

  • Change of ROOT Project Leader
    • A Naumann participation in NGT (Next Generation Triggers) project to be compensated by an LD position
  • New hires
    • 2 staff LD positions (started 01/02/2024!)
    • R&D Software Engineer on ROOT vacancy notice (QUEST) just published
    • Two QUEST vacancies opened soon (“Parallel-GPU Software Developer on ROOT” and “Data analysis tools developer on ROOT”
  • ATLAS physics PhD student with Uni Oslo
    • Should start after March Doctoral student selection round
  • Still under discussion:
    • Openlab (distributed analysis in the cloud)
    • CERN Knowledge Transfer (anomaly detection, math tools)

11

12 of 22

Analysis: General news

​

​

​

  • Two releases: 6.28/12 and 6.30.04
    • Bug fixes and major performance improvements, targeting the C++ interpreter and Python bindings.
    • Targeting ATLAS data taking 2024 and 6.30.04 the one targeting LHCb and CMS 2024 data taking.
  • 1st ROOT Hackathon: the Fixathon
    • Took place at CERN on February 13-14.
    • Main objectives - all achieved:
      • Close as many ROOT issues as possible
        • 59 issues closed (for comparison, 441 were closed in total during 2023), 17 solutions proposed, work started on another 23
      • Connect further with ROOT community, inclusively
        • ~30 people, from early career physicists to senior scientists,
        • Onboard early career colleagues teaching how to contribute to a large open-source scientific software

12

13 of 22

Analysis: RNTuple validation status

13

​

  • Technical work on the RNTuple large-scale testing effort started with teams from EP-SFT and IT-ST (12/2023)
    • CERN-IT provides a benchmark setup (80 nodes, 20PB storage, 100GbE)
    • Experiment liaison persons for ATLAS and CMS identified
  • First successful tests of the Analysis Grand Challenge performed
    • Tests based on community benchmark for analysis with single-client with many cores (>100) and input data from EOS/XRootD
    • First issues found and being addressed (ROOT, XRootD)
    • Once single-client tests are understood, we will move on to multi-client tests (i.e.: emulate what happens on a shared infrastructure)
  • After the AGC tests, we will continue with workloads from experiments
    • Still to be defined, experiment contact persons are in the loop
  • This activity is now on track
    • Many thanks for CERN-IT, WLCG management and the experiments for prompt reaction and follow up

�

14 of 22

Simulation: General news

  • New Geant4 version 11.2 released as scheduled on December 8th:
    • https://cern.ch/geant4-data/ReleaseNotes/ReleaseNotes.11.2.html
    • The first patch, 11.2.p01, is expected in a few weeks.
    • New monthly development release of Geant4 (11.2.ref01), including the developments since the release 11.2 (i.e. in December and January) was deployed and is installed on CVMFS at CERN.

​

  • Technical Forums and Courses
    • Presentation of the new Geant4 release 11.2 to the users Geant4 Technical Forum (February 15th)
    • The following Geant4 Technical Forum, in Spring, will be dedicated to the presentation of the plan of work of Geant4 to the users.
    • The next Beginners Course on Geant4 will take place in hybrid form (in person and remote) on April 15th to 19th:

​

​

14

15 of 22

Simulation: Geant4 assessment of AdePT and Celeritas

  • Geant4 assessment of the R&D GPU projects AdePT and Celeritas held on December 13-14.

  • The final report is now available
    • Common Report from the GPU projects Celeritas and AdePT Assessment
    • Specific Recommendations to Celeritas
    • Specific Recommendations to AdePT

​

16 of 22

Simulation: Selected quotes from the draft report

“The panel does recommend continuing discussions and exchanges between the two projects about findings, difficulties and solutions. The panel does not recommend however merging – at least for the time being – the activities of the two projects into a single one.”

“The two projects have demonstrated the need to evolve in depth the geometry representation, shown as the main limiting factor today. The two projects anticipates they will be ready by September. December leaves then some time to perform batteries of tests. For this delta-assessment, the panel considers proposing a series of application benchmarks together with a series of distributions of various key observables and performance measurements.

“The panel proposes a delta-assessment by December 2024.”

17 of 22

CernVM: File System

  • New TECH student joined 1 Feb, bringing the team finally back to nominal size (3 FTEs)
    • Will focus on container tools.
  • The CVMFS team in collaboration with the EESSI and EuroHPC gave an online tutorial on CVMFS best practices on HPC
    • > 100 connected participants during full 3 hours from HPC centers across Europe and beyond
      • (registrations from, e.g., SURF, HPCNow!, Institut Pasteur, Barcelona Supercomputing Centre, ITER, CNRS, HP, SDSC, AWS, Azure, NERSC)
    • Very positive feedback

​

  • Re-submitted the tutorial to ISC'24

​

17

18 of 22

Community (via HSF)

19 of 22

HSF: Activities highlights

​

  • New conveners appointed in the various WGs. 2024 plans being presented
  • Preparation of LHCC Focus Session on MC Generators
    • Related-ish to the Event generators' and N(n)LO codes' acceleration workshop in Nov. ’23
    • Several preparation meetings held since Dec. 2023 with representatives from the Gen. community and WLCG MB software liaisons
    • Report finalised and circulated among community members, for a presentation today

​

  • Co-organising the joint workshop with WLCG @ DESY in May 13-17
    • Registration is open !
    • Programme planning is advancing and will include
      • Joint session with WLCG on Analysis Facilities
      • Community Software Projects
      • Training Community Lessons and Plans
      • HSF’s 10th birthday and plans for evolution
  • Organised 6th IML Workshop @ CERN (29 Jan - 2 Feb)
    • Large in-person participation (> 100 people) from experiments ML community
    • Tutorials (Quantum ML, Transformers) and presentations from invited speakers
    • Very successful poster sessions with a new format
      • all contributed talks encouraged to present a poster for further discussions

19

20 of 22

HSF: PyHEP.dev workshops

  • Series of HSF workshops part of the PyHEP WG activities
  • .dev: new workshops since 2023 complementing the “good old” PyHEP “users” workshops, held since 2018 and online since 2020.
  • Planning/coordination meeting for developers of HEP software in Python.
  • Jul 25‒28, 2023 at Princeton (hosted by IRIS-HEP, 38 participants).
  • Aug 26‒30, 2024 in Aachen (IRIS-HEP & ErUM-Data-Hub, targeting 40‒50). Advert went out.
  • Preparation: participants set goals for themselves in GitHub Issues, starting a month before the workshop, and followed up on these plans during and after the workshop.
  • Format: introductory (lightning) presentations followed by discussions in topical groups.
  • Result: minutes taken here. The 2024 edition will also have a formal conclusions document.

Indico for the 2023 workshop

21 of 22

Backup

22 of 22

Request from the LHCC referees for topics to be covered

  • ROOT:
    • The issue of more personpower expected
  • Geant4:
    • G4 assessment in December on Adept vs Celeritas
  • CVMFS:
    • Collaboration with EESSI and EuroHPC to "train" HPCs to use CVMFS and lowering the threshold for adoption
  • Python in HEP developer workshops.
    • It will be good to hear on the outcome regarding the planned roadmap and priorities for the upcoming year.

​