Preparing users for the next generation of data intensive astronomy
Alex Clarke - Operations Data Scientist
alex.clarke@skao.int
Ugur Yilmaz - DevOps Engineer
ugur.yilmaz@skao.int
Collaborations Workshop - 3 May 2023
Slide /
Outline
Q&A: Talking preferred, alternative: https://app.sli.do/event/74yBfmatPgA59tfQKiu3AS
at slido.com with the code #2270043
2
Slide /
Slido
3
Slide /
The traditional way to do astronomy
4
Edwin Hubble (Image credit: NASA)
Slide /
Observation and Data Lifecycle (example based on Atacama Large Millimeter/submillimeter Array)
5
Observation Planning
The first step is to plan the observation. This involves choosing the target, the observing mode, and the observing time.
Observation execution
Once the observation is planned, it is executed by the ALMA staff. The data is received in real time and stored on disk.
Data processing
The data is then processed by the ALMA staff. This involves calibration and imaging
Data access
The data is then made available to the public through the ALMA data archive (using CASA)
Data archiving
The processed data is then archived in the ALMA archive. A distributed data store around the world
05
01
02
03
04
Several weeks/ months
3-10 EB, 1EB = 1TB
Data rate is around 30Gb/s or Streaming 6 HD movies every second
Slide /
Two telescopes, one observatory
6
SKA Low
131072 antennas in the Western Australian desert
SKA mid
197 dishes in South Africa
Slide /
Data Processing in SKA
7
https://www.bing.com/images/create/a-diagram-of-the-machine-learning-approach-for-est/6451ad7d4e204e69806c6670514360ca?id=uwRjEMFgPsFIxI3Nv8BqLQ%3d%3d&view=detailv2&idpp=genimg&idpclose=1&FORM=SYDBIC
Slide /
How does SKA data get to the users?
8
SKA MID
SKA LOW
8 Tb/s
9 Tb/s
100 Gb/s
SKA Regional Centres
100 Gb/s
20 Tb/s
2 Pb/s
Central Signal Processor
Central Signal Processor
Science data processor
Science data processor
Slide /
SKA Regional Centres (SRCs)
9
Interoperability
Heterogeneous SKA data from different SRCs and other observatories
Support to Science Community
Support community on SKA data use, SRC services use, Training, Project Impact Dissemination
Visualization
Advanced visualizers for SKA data and data from other observatories
Science Enabling Applications
Analysis Tools, Notebooks,
Workflows execution
Machine Learning, etc
Distributed Data Processing
Computing capabilities provided by the SRCNet to allow data processing
Data Discovery
Discovery of SKA data from the SRCNet, local or remote, transparently to the user
Data Management
Dissemination of Data to SRCs and Distributed Data Storage
Slide /
SKA data model: Discovery and re-use
All SKA Data Products will (in time) become public
Following from the example given by Hubble and SDSS, this will be the biggest generator of science
10
Slide /
SKA Software Vision and Strategic Objectives
It’s hard to do this all from ground up!
We are a global collaboration committed to building and delivering a world-class software and computing ecosystem, enabling the SKAO community to achieve transformational science.
Our culture of innovation, shared governance, and respect for diversity creates mutual value for all.
11
Slide /
ARTs
12
Slide /
Slide /
Software Quality & Delivery: SKAO Context
“How do we make sense of this so that we can deliver to a consistent level of quality, and efficiently use resources available, on time, and on budget”
13
28 Agile teams
> 200 people
Slide /
SAFe way to develop and collaborate
14
Slide /
Look at what Qualities needed? -
they influence approach
15
Slide /
How can the SKAO promote open and reproducible science
16
Slide /
Data challenge reproducibility awards
17
Reproducibility Awards Gold/Silver/Bronze scheme: https://sdc2.astronomers.skatelescope.org/sdc2-challenge/reproducibility-awards
Slide /
Maximising science output at regional centres
Users will bring their code to the data at Science Regional Centres
We want reproducibility and sustainability build into these tools at the Regional Centres
These things make it easier for users to take care of their workflows, and ultimately be more productive
As an observatory we have a responsibility to our users to make their lives as easy as possible, but understanding how our approach will influence users is not trivial.
18
Slide /
Thank you!
Q & A
We recognise and acknowledge the Indigenous peoples and cultures that have traditionally lived on the lands on which our facilities are located.
Slide /
Slido
20
Slide /
User notes 2
Mini-workshop or demo session abstract
The Square Kilometre Array Observatory (SKAO) is a global radio telescope that aims to transform our understanding of the Universe, and deliver benefits to society through global collaboration and innovation. Thousands of radio antennas and dishes will be spread across Australia and South Africa, allowing astronomers to observe the universe in unprecedented detail from 2028.
The SKAO is an Intergovernmental Organisation currently involving 16 countries with the headquarters based at Jodrell Bank in Cheshire. The distributed nature of this large project presents a challenge, with over 25 teams deploying hardware and software spread amongst 100 organisations around the world. We operate under a Scaled Agile Framework (SAFe) approach, which promotes workflow patterns that enable a large number of teams to collaborate and deliver work towards a large common goal.
The SKAO will deliver data at an unprecedented volume and rate, where it will mostly not be feasible for users to download data to their personal or even institutional computing facilities. In anticipation of this we are working on tools that will enable scientists to explore and process data remotely at dedicated facilities no matter where they live around the world. A key driver for science output is ensuring that data products and software pipelines are easily discoverable, explorable, and understandable. Achieving this depends on us deploying user friendly, flexible and efficient services that allow scientists to maximise their output from the data, and so a key challenge for us is ensuring that our software and operations teams are working together effectively towards these goals.
In this talk we will discuss how our different teams work together, and how we are implementing software pipelines and data discovery tools that will maximise scientific output. The SKAO is committed to ensuring the long-term sustainability of our software, data and systems, and we will discuss how we are encouraging users to align with our values, making science inclusive and reproducible. We want these themes built directly into the data management systems, software pipelines, and compute resources so it is easier for users to take care of their workflows, and ultimately lead to being more confident and productive. As an observatory we have a responsibility to our users to make their lives as easy as possible, but understanding how our approach will influence users is not trivial. At the end of our talk we are keen to gather some ideas from you on how organisations can make users' lives easier and encourage healthy working behaviours.
Mini-workshop or demo session audience
People don't need any prior knowledge about astronomy, science or software. This talk will present how we are building the observatory, how users will be affected by a shift in scientific workflows, and how we can make it easier for them to work in a sustainable and reproducible way.
Mini-workshop or demo session theme(s)
Technical development (e.g. relating to software sustainability, software products and digital tools, infrastructure and documentation, software development skills, and training)
Mini-workshop or demo session description
Alex and Ugur will give a joint presentation as discussed in the abstract, and we will be leaving time for a brief discussion as we would love to hear the thoughts of others on these themes and bring that feedback back to SKAO. We hope that showing people how the SKAO will operate will inspire people on how to design systems for big data workflows. We want to create a research ecosystem system which takes care of its users, so that it is easier for them to explore and analyse data in a reproducible and sustainable way. One of the themes of our talk will be how data and software tools influence peoples workflows - if it's hard to install and clunky to use, users are less likely to use it and even less likely to bother making a piece of work reproducible, ultimately making their work harder in the long run. We want to highlight how an organisation like SKAO can make a big impact on encouraging healthy working behaviour, creating more efficient and happier users. We don't see any barriers to remote participants engaging with our session. We might use a shared goolge doc, or Sli.do to collect questions, thoughts, and feedback throughout the session.
We saw potential for this to be a 25 minute talk followed by a 20+ minute discussion around ways organisations influence their users workflows and how they can make life easier for their users (emphasis on organisations that serve data/software/compute resources), as we are keen on gathering ideas around this. But we have currently decided on cutting this down to a 20+10 session for efficiency. However, if you think there is scope/desire for this to be 45-60 minute session we would be happy to plan for that, but we thought we would initially be conservative and not use up time that could be used for other sessions. Happy to discuss more if there is interest in a longer session from us!
Any special requirements or other information you would like to include about your proposed session?
We are interested in using Sli.do. We will check it out and see if we can use it to get feedback and thoughts throughout our talk and discussion.
Scheduled
CW23 Day 2: Wednesday, 3 May 2023 from 10:00 - 10:30 BST (09:00 - 09:30 UTC; 30 minutes)
21
Slide /
Requirements
The purpose of
the requirements specifications
is to
capture stakeholders needs.
Requirement Specifications
are managed throughout the life of the project
to create a consensus
of what should be delivered as a system.
22
Slide /
Data Flow
23
Slide /
Proposal Observation Flow
OSO is the suite of software tools that support the entire science process from proposal submission to data-product delivery.
24
Slide /
Observatory Science Operations Tools
Provide a science-oriented user interface for the following functions:
OSO tools provide easy to use science (and operations) oriented interfaces for using these functions, with aim to hide, as much as possible, the complexity of the telescopes.
25
Slide /
Observatory Science and Operations Tools (2 of 2)
26
Slide /
Observation Operations
27
Slide /
Functional Observation
28
Slide /
Standardisation … for everything
Throughout the Software Development Life Cycle:
29
Slide /