Release Updates

Tim Theisen

OSG/PATh Staff Meeting - 2022-03-23

Summary (since 2021-10-20, 5 months)

• OSG 3.5 - 12 releases total (upcoming repository updates ease 3.6 transition)
• 1 High Priority release (HTCondor Security)
• 3 Data Only releases
• 1 IGTF release (3 day turnaround, expired CA certs in Let's Encrypt CA bundle)
• Weekly VO Package releases in February due to DN changes
• OSG 3.6 - 11 releases total
• 1 High Priority release (HTCondor Security)
• 2 Data Only releases
• 1 IGTF release (3 day turnaround, expired CA certs in Let's Encrypt CA bundle)
• Weekly VO Package releases in February due to DN changes
• Devops - Open Science Pool
• 7 HTCondor Update releases
• 3 HTCondor Beta releases (1-2 weeks before general release)
• HTCondor
• 1 Security Release (8.8.16, 9.0.10. 9.6.0)
• 7 HTCondor Update releases
• 4 HTCondor LTS (9.0.x) and Feature (9.x.0) releases
• Docker Images are updated on release day
• Community testing going well

Notable Events

• HTCondor Security Release Day (Interleaved security and regular release)
• Released six HTCondor versions + four OSG builds of HTCondor in a single day
• Changed plan midstream to avoid manual steps
• Several release preparation steps cannot be performed until source in github.com
• Too much to do, turned into a twelve hour day and only accomplished the necessities
• Recommendation: Do the security release on Tuesday and the regular release on Thursday
• HTCondor Security Release Numbering
• 9.6.0 version should have been 9.5.5 version
• Let’s Encrypt CA Certificates Expiration
• Took three days to release CA Certificate Packages
• CA Certification Package builds failed on EL 8
• Found the expired CA certificates and removed them and pushed the change upstream
• Defects found in testing (not really notable)
• gratia-probe 2.2, osg-flock (refactoring out GSI, minor issues found, expected)
• htvault-config, htgettoken (new packages, minor issues found, expected)
• Defects discovered within a month after release: none

Challenges

• Transition away from GSI
• Job submission to ARC CEs from the Glidein factory
• Until now, only the old nordugrid GAHP was available.
• We need the new ARC REST interface available along with the nordugrid GAHP
• Special build of HTCondor in OSG devops repository
• Currently, it's HTCondor 9.7.0 with Globus included (nordugrid requires Globus)
• Provides both nordugrid GAHP and ARC CE REST interface
• How long do we need to support this special build?
• Special build for Open Science Pool Access Point (still has GSI)
• Open Science Pool no longer needs GSI
• Get rid of special build
• Move OSPool APs from devops repository to OSG 3.6-upcoming-testing repository
• Based on feedback from OSG All Hands meeting, the HTCondor Team agreed to extend support for HTCondor 9.0(which has GSI) in the CHTC repos until 2023-02-01

Challenges (continued)

• OSG 3.5 is kind of brittle at this point
• Newer gfal2 in EPEL precludes installing XRootD, StashCache, and Worker Node
• Can install Worker Node by enabling osg-upcoming
• No work-around for XRootD 4 (should be on XRootD 5 anyway)
• Support ends in five and a half weeks
• Devops paradigm increases release team workload
• Packaging and distributing Go programs (all dependencies included)
• stashcp (includes stash_plugin for HTCondor)
• xrootd-monitoring-shoveler
• What about security issues in a dependency? Who monitors?