Mini-Challenges: Building from DC24
Derek Weitzel / University of Nebraska Lincoln, Shawn McKee / University of Michigan & Diego Davila / University of California San Diego
IRIS-HEP 2024 Retreat, Seattle Washington (https://indico.cern.ch/event/1374962/timetable/)
September 4, 2024
WLCG Data Challenges
The WLCG Data Challenges are a ~biennial series of four increasingly-complex exercises which started in 2021 and are aimed at demonstrating readiness at the HL-LHC scale.
Next data challenge (DC26?) targets 50% of HL-LHC scale and includes T1/T2 and any improvements we can integrate into our infrastructure.
These data challenges provide many benefits, allowing sites, networks and experiments to evaluate their progress, motivate and validate their developments in hardware and software and show readiness of technologies at suitable scale.
For USLHC, we believe it is critical to fully participate in future challenges, both by preparing and testing before each and analyzing the results after each.
2
IRIS-HEP Retreat Sep 4th, 2024
Plan and Overview
The plan for today is to discuss future mini-challenges (note document should be open for everyone to contribute to: https://docs.google.com/document/d/1Xjq8CvotPAYSsp28Ntn_hKZAQiyS4pwpX5t_p_a_6oM/edit?usp=sharing )
Topical Questions to think about for this presentation:
What is a mini-challenge?
What do we want to test?
Who wants to participate?
What role does IRIS-HEP play?
What technologies should we test?
When do we want to run our first tests?
How (what tools) will we run mini-challenges?
3
IRIS-HEP Retreat Sep 4th, 2024
Mini-Challenge Introduction, History and Context
During DC21, 2 objectives, apart from achieving data rates, were established:
So DC became a platform for showcasing new “technologies”
Then on a presentation: “The path to the next WLCG Data Challenge” from Mario Lassing at the LCHOPN-LHCONE meeting in October 2022[*], the following is stated about the different technologies to be tested during DC24:
Start with a series of distributed, constrained, and isolated ramp-up challenges .. …Independently organized, and report via WLCG/DOMA
4
IRIS-HEP Retreat Sep 4th, 2024
What is a “mini-challenge”
Given the cadence and scope of the WLCG Network Data Challenges, it is important to understand what we mean by “mini-challenge”. Here is a possible definition:
The goal is to make it easy to test and track both our capabilities and capacities, finding and fixing bottlenecks, identifying bad architectures and hardware and improving our visibility into how our sites perform as part of a globally distributed infrastructure.
5
IRIS-HEP Retreat Sep 4th, 2024
Goals for DC26
For DC26 (or DC27 if it moves later) we are targeting:
Mini-challenges should help get us there…
6
IRIS-HEP Retreat Sep 4th, 2024
Running an upgrade Campaign
7
IRIS-HEP Retreat Sep 4th, 2024
Needed Visibility for Data Challenges
For DC24, a site network monitoring campaign was undertaken to provide better visibility into each site’s capabilities (see CERN Gitlab)
We need to continue to improve (and verify) the monitoring we have, since this underlies all our attempts to identify friction points in our infrastructure.
For DC26, we would like to have at least 80% of our WAN traffic identified via the SciTags Initiative (there is also an IRIS-HEP metric for number of USLHC sites marking traffic; currently the number is ‘1’ [Nebraska])
We need to identify what monitoring is missing and fix any incorrect monitoring and clarify any misleading monitoring.
8
IRIS-HEP Retreat Sep 4th, 2024
Carry out a Monitoring validation
Q. Who can do this?
9
IRIS-HEP Retreat Sep 4th, 2024
Running periodic 1-to-1 load tests (NANO-challenge?)
This requires each participant site to have a validated site network monitoring
Q. Who can do this?
10
IRIS-HEP Retreat Sep 4th, 2024
Management and Logistics for Mini-Challenges
Commonly mini-challenges that test new technologies are run independently, (specially if they do not interfere with day-by-day Ops) by the people around the related project
On the other hand, those that run throughput tests are run in coordination with the site admins of the sites involved and the relevant R&E network responsibles.
In any case every mini-challenge should be reported in advance to the WLCG DOMA conveners.
11
IRIS-HEP Retreat Sep 4th, 2024
USATLAS and USCMS Activities: What is in Common?
If we are going to undertake various mini-challenges that are experiment-based, there may be benefit in running them at the same time.
Certain resources or network transit locations may be shared and identifying any contention can be an important outcome from running tests at the same time so we can address the issues as they are observed.
On the other hand, we don’t want to make mini-challenges heavy weight or over constrained as to when they be run. One of the advantages of mini-challenges are supposed to be the ability to quickly and easily test capability or capacity for a site or set of sites.
12
IRIS-HEP Retreat Sep 4th, 2024
Preparing Technologies and Capabilities
HL-LHC will require more resources than we can currently afford.
The WLCG data challenges are designed to regularly test where we are relative to where we need to be for HL-LHC.
Possible technologies to test and, if beneficial, integrate
13
IRIS-HEP Retreat Sep 4th, 2024
Site Testing, Motivation and Benefits
We hope to have sites as enthusiastic participants in planning and executing various mini-challenges.
This is a great opportunity to identify how each site performs and identify where there are issues.
It should be a significant help in defining how the site should evolve their hardware, software and architecture.
14
IRIS-HEP Retreat Sep 4th, 2024
Transforming our Sites
The data challenges provide us with an opportunity to evaluate our existing hardware, software and architecture to identify bottlenecks, limitations and misconfigurations.
Given that HL-LHC is ~6 years away, now is the perfect time to re-evaluate our site’s hardware configuration and architecture so that we can have a suitable baseline ready for HL-LHC requirements.
It is critical that sites understand how they fit into our globally distributed infrastructure so they can meet the HL-LHC requirements and use-cases.
15
IRIS-HEP Retreat Sep 4th, 2024
Transforming our Sites (cont’d)
An example of the above is the case of Caltech that went from performing at 80Gbps during DC24 to ~250Gbps by analyzing the results from DC24 and upgrading and tuning their system
16
Caltech (Prod) => UCSD (test) HTTP-TPC transfer throughput
IRIS-HEP Retreat Sep 4th, 2024
Mini-Challenges and Ongoing Testing
The mini-challenges prior to DC24 turned out to be very beneficial for finding problems in our infrastructure and we should plan to have regular mini-challenges going forward
Ongoing mini-challenges a few times a year provide important guidance and validation for site changes in hardware, software and tunings.
17
IRIS-HEP Retreat Sep 4th, 2024
Getting Effort to Analyze Results
One important issue is ensuring that we have enough effort targeting analyzing results we obtain.
For DC24, we didn’t really have enough effort to fully benefit from all the work that went into running DC24 and gathering the data.
We need to make sure we have effort not just to organize and run mini-challenges but to follow through and thoroughly explore the results.
18
IRIS-HEP Retreat Sep 4th, 2024
Questions To Discuss (and Try to Answer)
How do we best organize mini-challenges in common between USATLAS/USCMS?
Who will organize and operate the tests?
Who will analyse results and identify bottlenecks?
What technology-focus mini-challenges are needed?
How do sites share and implement beneficial changes identified by mini-challenges?
19
IRIS-HEP Retreat Sep 4th, 2024
Summary
We (IRIS-HEP/OSG-LHC/US-LHC) need to clarify and document existing plans, mini-challenges and goals for the next year and for DC26/DC27
We have an opportunity to leverage DC24 results to improve our infrastructure, to drive technology deployment, to show value and to demonstrate capabilities at scale.
Let’s Discuss
20
IRIS-HEP Retreat Sep 4th, 2024
Acknowledgements
Thanks to Justas Balcas and Asif Shaw for their contributions to the slides
We would like to thank the WLCG, HEPiX, perfSONAR and OSG organizations for their work on the topics presented.
In addition we want to explicitly acknowledge the support of the National Science Foundation which supported this work via:
21
IRIS-HEP Retreat Sep 4th, 2024
Background Material
Here are some resources we know about:
Presentations
Some Google Docs
22
IRIS-HEP Retreat Sep 4th, 2024
DC24 Links
Official DC24 report
https://zenodo.org/records/11402618
DC24 Network Activities and Results: https://docs.google.com/presentation/d/1s0VvbXEpj1PN9umFT8wgsHsHmG9EYucymbalKNrvuKQ/edit#slide=id.p1
Katy Ellis LHCONE/LHCOPN DC24 presentation: https://docs.google.com/presentation/d/1Tm3pCMkfHj5KHTW3PXbgS7mdHf72Ir27qr1JgMbrnRg/edit#slide=id.g1ea89411ecb_0_4
Next Steps Towards DC26: https://docs.google.com/presentation/d/1mMx6QaihWJWpbVEQgxNjZXRT5_s4SkBTXu0SpELtuvI/edit#slide=id.gd170caf633_1_0
DC24 ATLAS Retrospective: https://docs.google.com/presentation/d/1Lh_D57BvWn13AFClhhuczm-j-tKV-yMez_oD4yYUtBo/edit#slide=id.gd170caf633_1_0
23
IRIS-HEP Retreat Sep 4th, 2024
Backup Slides
24
DC24 Throughput
25
Flexible target (green line) was to be met for 48 hours
IRIS-HEP Retreat Sep 4th, 2024
DC24 Throughput Annotations
DC24 met the baseline target, but not the flexible one
Wide-area Networking was NOT the bottleneck
FTS was one of the primary limitations
26
FTS tokens refresh load
Volatile datasets included as a source
FTS weekly DB defrag
Degradation due to rucio daemons db contention
submission paused to give the cleaner time to clean
not enough pressure on FTS switched token off increased FTS memory
stopped submissions installed second high memory FTS instance for T2s. Cleanup 3M cancelled transfers
1.4 Tb/s peak for 4h
Start of flexible model injections
Increased concurrent transfers in FTS
IRIS-HEP Retreat Sep 4th, 2024
UCSMS DC24 summary
The planned target for FNAL was 32GB/s
The maximum injected/achieved rates for FNAL were 26/31GB/s for reads and 37/39 GB/s for writes making this a success for the T1.
Moreover, sustained rates of 24 GB/s for writes and 36 GB/s for reads were observed for 21 and 12 hrs. respectively
From the T2s perspective the initial targets set by CMS centrally, ranging from 1.2 to 4.8 GB/s, were dimmed lower than expected by USCMS and all sites performed above them.
On the last day, these rates were significantly increased as per USCMS request. The new rates ranged from 4 to 14 GB/s. This time only Nebraska performed above the new rates with other sites hitting marks around 50% of the updated targets.
27
IRIS-HEP Retreat Sep 4th, 2024
USCMS planned upgrades
28
IRIS-HEP Retreat Sep 4th, 2024