HATS/LSDB working group

February 21, 2025

1

LINCC Frameworks Agenda

• Plans
• Format changes we are considering
• Rubin updates
• LSDB server at data.lsdb.io
• Nested-pandas representation
• Firefly-over-LSDB at University of Washington server

2

Plans

Early March - Operational Rehearsal 5 for Rubin

Testing scaling of import; 950k detectors compared to around 10k takes so far

Mid March - Internal Rubin meeting

Mid April - Last chance for large change to format

Mid to late April - Rubin first photon

Early June - IVOA meeting in Maryland

​

Additionally, focus on usability of LSDB

Enable scientific analysis, scaling and ease of use

​

3

Format changes we are considering

• Leafs can be folders, to enable additive catalogs (such as alerts)
• https://github.com/astronomy-commons/lsdb/issues/435
• Index catalogs, margins catalogs generated by default and incorporated into the main catalog
• https://github.com/astronomy-commons/hats/issues/456

​

​

• [Not format related, but potential HATS change] Rowgroups in parquet files?

4

Rubin data

• Updates on weekly processing
• Tickets that we opened
• Added value catalogs

​

5

Rubin data

• Updates on weekly processing
• Tickets that we opened
• Added value catalogs

​

6

Rubin data

• Updates on weekly processing
• Tickets that we opened
• Added value catalogs
• Crossmatched to ZTF, PanSTARRs
• Add magnitudes to the tables (data management provides only flux)
• Detections sorted in time

​

7

News (that I am aware)

8

• IPAC
• ZTF Data Release 23 in HATS on Fornax, public soon
• Helping set up Firefly-over-LSDB at UW
• Space Telescope
• PanSTARRS in HATS at S3 open bucket: s3://stpubdata/panstarrs/ps1/public/hats/
• SPlus
• Helping set up HATS-server at UW
• Linea
• Results document

LSDB server

• lsdb-server understands column- and row-filtering, making some server-side optimizations possible
• Without this, all filtering happens client-side, obliging higher data transfer rates
• Must be run as a proxy server since it only handles Parquet files and not the other files that are part of a HATS catalog
• Must be fronted by nginx since it does not handle require Range headers
• nginx will handle those for proxies

​

Status:

• Running on epyc but not installed, and not visible through the firewall

9

lsdb-server testing

10

~100e3 observations

~1e6 observations

lsdb-server surprises

• Tried to use back-end filtering in combination with cone search, thinking that ra, dec +/- radius would describe a square that included the whole cone
• But the observations returned were too low until I doubled the bounds
• Filtering columns (4 out of ~150) doesn’t yield the kind of benefit I’d expect

11

nested-pandas representation changes

12

v 0.3.5

Firefly-over-LSDB

Still in progress! Maintenance on epyc 2025-02-21

13

Future Activities (next month)

• Joint Technical meeting
• Week of March 10

​

• Operational Rehearsal 5
• Early March

​

14

Next demo meeting

March 21.

​

Normal office hours in the Friday time slot in the meantime.

15

Any other business?

16