Rubin Computing in 180”

Rubin-LSST France - June 2023

Dominique Boutigny - LAPP

Or… "How to get bananas from such a mess ? "

DC2

Abott et al. http://arxiv.org/abs/2305.17173

An insane amount of data

​

2

DESC DC2 == Rubin DP0.2

• Input: 51 TB - 2.9 M files
• Output: 3 PB - 54 M directories - 201 M files

​

• 58 M individual tasks (quantas)
• 2.3 Millions CPU hours to process

DC2 is this tiny sky patch

DP0 / DC2 is only 8% of 1 full LSST year

The initial DC2 processing took us 2 years to complete

We are still unable to process the small area corresponding to the Deep Drilling Field

The first Rubin data release will occur 6 months after the first 6 months of data …

​

… And it will be extremely complicated

Tools

3

Software:

The (in)famous Stack

• 90 packages - 39 681 files
• Goes from Instrument Signature Removal up to PSF modeling, deblending, flux and shape measurements

Workflow:

• Quantum task generation
• Individual tasks + their interconnexions

I/O - Metadata - Bookeeping

The (in)famous Butler

• Keep track of the relationship between metadata and files physical location on disk
• Allow to retrieve datasets from metadata

Handled through a posgreSQL database

• The butler hides database complexity behind standard interfaces

Computing for Science Collaborations / DESC

4

Rubin will produce basic catalogs, common to all Science Collaborations

All science specific catalogs:

• Photo-z
• Fancy shape measurements for weak lensing
• Cluster identification
• Strong lensing
• …

Will be under the Science Collabs' responsibility

DESC:

• Main computing center at NERSC
• Super Computers
• Not really well suited for DESC usage, … but no real choice

At CC-IN2P3 we maintain the whole DESC software and catalogs

Access through:

• GCRCatalogs
• Dask
• Qserv

CC-IN2P3 will provide flexibility in Rubin data analysis

It is super important for us to identify which services we should focus on to produce the best possible bananas