LIGO containers in diverse computing environments��Thomas P Downes�Center for Gravitation, Cosmology & Astrophysics�University of Wisconsin-Milwaukee�LIGO Scientific Collaboration
LIGO-Virgo Advanced Detector Network
O1: September 2015 -- January 2016�O2: December 2016 -- August 2017�O3: ~1 year of observing TBA
Upper-right: LIGO Hanford, Washington State, USA�Lower-right: Virgo ca. Pisa, Italy�Unshown: LIGO Livingston, Louisiana, USA
UW-Milwaukee�and the CGCA
Kenwood Interdisciplinary Research Complex (2016)
“Modeled” LIGO searches compare data to many simulations
Images courtesy LIGO Laboratory & Fisher Price
Small amount of data: ~1MiB/sec!
Increasing demand for LIGO Computing
As our detectors become more sensitive we are seeing increased demand
Approximately a factor of 2-3 in growth each observing run!�We need to make greater use of resources not directly managed by LIGO
Researcher / administrator attention is our scarcest resource!
LIGO Computing Environment and Practices
Approach cannot be sustained from either user or administrator perspective!
Technical debt: it seemed like a good deal at the time...
The low-cost approach to development suddenly has costs when you have more and better data!
Must make it easier for development practices to more closely mimic what “we want the users to do” at similar up-front cost in time and technical understanding.
Contemporary tools are really good for moving fast
Reject thesis that scientific use cases are special: use standard tools!
Even really smart people have work that can and should be performed by a robot
Continuous integration w/fork + merge to reduce impact of broken changes to code
Continuous deployment w/agnostic outputs
(Tarballs, Docker image, .deb/.rpm, pypi)
Users can self-deploy to their workstation, but can we continuously deploy to the grid?
Webhook Automation
GitLab Container Registry produces nightly build/public release of LIGO Algorithm Library��docker pull containers.ligo.org/lscsoft/lalsuite:nightly��Below: API-triggered DockerHub rebuilds of our cluster login and job environment
GitLab allows me to automate webhooks on behalf of all LIGO researchers who “docker push” to our container registry
Publishing of Docker images to CVMFS for use with Singularity
DockerHub or GitLab Container Registry builds container and generates webhook�[DockerHub: +1 hour @ 5GB worker node image]�[GitLab Container Registry: Θ(minutes)]
LIGO Webhook Relay validates and forwards event to CVMFS Publisher
CVMFS Publisher receives event and places it in job queue
Job queue pulls container images and publishes them 1-by-1�[+13 minutes @ 5GB]
Available to clients at
/cvmfs/ligo-containers.opensciencegrid.org
Within hour, a developer can test changes via Docker or on Open Science Grid using Singularity and CVMFS!
Thanks, CERN + Open Science Grid!
Success so far...
Problems so far...
The infrastructure is freely available
These applications are distributed as fairly simple Docker Compose applications