Host Tuning: Status and Plans

Shawn McKee / University of Michigan

LHCOPN/LHCONE Meeting #57, Naples, IT

(https://indico.cern.ch/event/1677691/ )

October 1, 2026

Host-Tuning Evaluation

Quick summary: we wanted to enable and test ESnet Fasterdata recommendations as applied to our production storage hosts.

Goal: Improve the throughput and/or reduce the load per transfer, decreasing time for transfers & improving net use.

As part of the ongoing perfSONAR upgrade campaign, we created a fasterdata-tuning.sh (bash) script to implement the recommendations from the Fasterdata web pages.

The WLCG host-tuning Capability Mini-Challenge was done Feb 2-3 and involved USATLAS and USCMS sites.

2

LHCOPN/LHCONE #57 – Naples

Sites and Paths Tested

3

LHCOPN/LHCONE #57 – Naples

fasterdata-tuning.sh Workflow

Two profiles: measurement (perfSONAR) and dtn (storage/transfer nodes)

Workflow: audit (read-only, current vs. recommended) → apply → save / diff / restore

Tunes TCP buffers, backlog and txqueuelen, NIC rings and offloads, congestion control (BBR/BBRv3), fq pacing, CPU governor, tuned-adm profile

Baseline auto-saved before apply, so every test can be compared and reverted

Docs: osg-htc.org/networking/perfsonar/tools_scripts/fasterdata-tuning/

4

LHCOPN/LHCONE #57 – Naples

Host-Tuning Challenge Outcome

​

USATLAS:

• We set up to gather a significant amount of monitoring: CheckMK plots, Ganglia plots, ESnet Stardust and various logs
• Problems with conflicting production (merge) jobs at MWT2 made tests on 2nd day fail
• Fixed a dCache proxying misconfiguration; tuned re-run showed best transfers we have seen (MWT2->AGLT2)
• Other direction exposed MSU outbound issue identified and addressed post-challenge.

USCMS: tuning DTNs lifted single-node iperf3 from 21 to 95–97 Gbps on 100G NICs

• End-to-end FTS gains only where storage kept up (UNL 60 → 90 Gbps)
• UCSD storage-bound (FTS unchanged); Vanderbilt 25G NICs regressed

​

Mini-challenge details available from LHCOPN/LHCONE #56 presentation

5

LHCOPN/LHCONE #57 – Naples

Key Result: Bottleneck Decides

Same dtn profile, three USCMS outcomes: UNL improves; UCSD unchanged (storage-bound); Vanderbilt worse (25G NIC already full)

USATLAS: found and fixed a dCache proxying misconfiguration at AGLT2, then saw the best MWT2 → AGLT2 transfers yet

Takeaway: audit first, then tune the layer that is actually providing the bottleneck

6

LHCOPN/LHCONE #57 – Naples

Where Each Site Was Limited

7

LHCOPN/LHCONE #57 – Naples

What Changed Since April (#56)

Results written up: CHEP 2026 proceedings paper and arXiv preprint submitted Sep 25 (11 authors, 7 sites + ESnet)

Script now has a separate dtn profile; audit mode documented on the OSG pages

LHCONE path finding (400G WAN test, not host tuning): NET2 ↔ Prague capped at 100 Gb/s by ESnet path selection pinning traffic to one path; reported and fixed by ESnet

UK collaboration: comparing tuning configs with Tim Chown (Jisc) and Raul Lopes (Jisc), who is also running 800G+ tests for EuroHPC (see backup); Jisc keen to contribute to DC27 network testing

DC27 dates fixed(?): Feb 22 – Mar 5, 2027 (TBD @ WLCG/HSF)

8

LHCOPN/LHCONE #57 – Naples

LHCONE Path Finding: NET2 ↔ Prague

9

Capped at 100G?!

LHCOPN/LHCONE #57 – Naples

DC27 Site Checklist

• ALL systems are dual-stacked
• SciTags is enabled and configured on storage
• Jumbo frames are in use on storage hosts
• perfSONAR is upgraded (hardware and software)
• Systems are tuned for optimal distributed computing performance (host-tuning + switch and service configs)
• Site network monitoring is capturing all WAN data flows
• Need to understand how to differentiate traffic by type: LHCOPN, LHCONE, R&E and commodity

​

10

LHCOPN/LHCONE #57 – Naples

Proposed Phase 2 Methodology

Tune one parameter group at a time (buffers, pacing, rings/offloads, congestion control, storage I/O) instead of the full profile

• Measure network (iperf3) and storage (fio) separately, then end to end (FTS)
• Tune both ends; save → apply → test → restore on every host
• Record CPU, I/O wait and retransmits systematically (Fireflies/SciTags + FTS records in UC Elasticsearch)
• Gate settings by NIC headroom so no host regresses (Vanderbilt case)
• Expand beyond the US: at least one European pair and one transatlantic path

11

LHCOPN/LHCONE #57 – Naples

Tuning Decision Tree for Phase 2

12

LHCOPN/LHCONE #57 – Naples

Road to DC27

13

LHCOPN/LHCONE #57 – Naples

Near To Mid-Term Plans

The host-tuning will likely need multiple iterations before DC27

• If the things we tuned are NOT the bottleneck, no improvement
• Phase 2 capability challenge: Nov 2026 – Jan 2027, more sites, per-parameter testing
• Tuning clearly helps network throughput (iperf3); storage needs its own study
• Priority: fix settings that reduced throughput — Vanderbilt (25G NICs) and a reported ~50% drop on a RAL storage host; comparing audit output with RAL’s own tuned settings (Raul, Jisc)

More broadly, continued mini-challenges for both capacity and capability.

Not detailed in this talk but very important: getting more advocates to drive capability challenges for WLCG:

• File sizes (increase x10 to understand cost-benefit)
• Broader deployment of Network optimizations (Jumbo frames, pacing, new protocols)
• SDN (SENSE/Rucio), assuming testing shows benefits.

14

LHCOPN/LHCONE #57 – Naples

How the Community Can Help

Sites: run fasterdata-tuning.sh in audit mode (dtn target) on a storage node and send us the output

Already-tuned sites (JISC has some cases that beat our profile): share your settings so we can diff them against the profile

NRENs: volunteer a path and a contact for Phase 2; report ECMP or path-pinning cases like NET2–Prague

Everyone: follow the WLCG DOMA mini-challenge calendar and the WLCG Site Networking Support meeting (3rd Thursday of the month; 10 AM Eastern)

15

LHCOPN/LHCONE #57 – Naples

Summary / Conclusions

Tuning only helps when it removes the binding constraint (audit first); written up for CHEP 2026.

These results inform our Phase 2 plans. Much work remains in implementing capabilities and testing our setups.

Ideally, the testing we do (and have done) should inform our purchasing by identifying the most beneficial configurations/hardware.

We need a multi-site, multi-region team (including NREN partners) for Phase 2 host-tuning.

Questions, Comments, Discussion?

​

16

LHCOPN/LHCONE #57 – Naples

Acknowledgements

We would like to thank the WLCG, HEPiX, perfSONAR, ESnet, JISC (Tim Chown, Raul Lopes), USLHC and OSG organizations and the participating sites for their work on the topics presented.

In addition we want to explicitly acknowledge the support of the National Science Foundation which supported this work via:

• IRIS-HEP: NSF PHY-2323298

​

17

LHCOPN/LHCONE #57 – Naples

Backup Slides

18

USCMS Results

19

Table 1. Summary of Results (Units are Gbps)

Better / Better

Worse / Worse?

Better / Same

Quick Conclusion: Overall the tuning seems to improve at best and do little harm at worst

Summary of the summary

USCMS: Throughput per file (input data: FTS records)

20

Better

Definitely Worse

Same

… also maybe something to look at

Update: Vanderbilt’s 25G NIC was already the ceiling; profile needs per-site review

Backup: Jisc 800G+ EuroHPC Tests

• Raul Lopes (Jisc), with Mark Feit (Internet2) on hardware, is testing 800G+ paths for EuroHPC; a write-up is in progress and a recorded R&E network call talk exists
• Tools: the perftest suite (ib_write_bw) drives close to 800 Gb/s without hitting CPU limits, while iperf2/iperf3 struggle; qperf also used
• perftest is being integrated into perfSONAR (“coming soon”; github.com/linux-rdma/perftest)
• Raul’s tuning beats our fasterdata-tuning.sh profile in some scenarios (not all); we will compare the settings and fold improvements into the script
• At these speeds latency and loss say most of what we need; no need to fill links (~20 Gb/s single-stream iperf3 UK–US is already respectable)
• Open: which workflow tools and perimeter (Science DMZ) designs will EuroHPC centres use? New UK AI HPC centres (Isambard/Bristol, Dawn/Cambridge) start at 100G to Janet

21

LHCOPN/LHCONE #57 – Naples