1 of 25

Francisco Förster; G. Cabrera-Vives; E. Castillo-Navarrete; P. A. Estévez; P. Sánchez-Sáez; J. Arredondo; F. E. Bauer; R. Carrasco-Davis; M. Catelan; F. Elorrieta; S. Eyheramendy; P. Huijse; G. Pignata; E. Reyes; I. Reyes; D. Rodríguez-Mancini; D. Ruz-Mieres; C. Valenzuela; I. Álvarez-Maldonado; N. Astorga; J. Borissova; A. Clocchiatti; D. De Cicco; C. Donoso-Oliva; M. J. Graham; L. Hernández-García; R. Kurtev; A. Mahabal; J.C. Maureira; R. Molina-Ferreiro; A. Moya; A. Muñoz Arancibia; W. Palma; M. Pérez-Carrasco; A. Papageorgiou; P. Protopapas; M. Romero; L. Sabatini-Gacitua; A. Sánchez; J. San Martı́n; C. Sepúlveda-Cobo; E. Vera; J. R. Vergara

The Universe in a stream: the ALeRCE broker

2 of 25

ALeRCE is a Chilean-led initiative to build a community broker for ZTF, LSST, and other large etendue survey telescopes

Carrasco-Davis et al. 2020 + Förster et al. 2020 + Sánchez-Sáez et al. 2020 (Submitted)

3 of 25

ALeRCE: from HiTS to LSST

~10-100x

~1-10x

4 of 25

Goals

To facilitate the study of variable and transients objects:

  • Fast classification of transients, variable stars and active galactic nuclei

  • Flexibility to adapt to different science cases (taxonomy, data products)

  • Connect survey and follow up resources in Chile and abroad

5 of 25

The ALeRCE collaboration

Valdivia, Chile, November 2018

Santiago, Chile, June 2019

La Serena, Chile, March 2019

Concepción, Chile, Jan 2020

ALeRCE users

6 of 25

Agile Methodology

  • Two week sprints
  • Product owners
  • Detailed sprint planning (Notion)
  • Very short daily meetings
  • Weekly area meetings

Well aligned teams:

    • infrastructure
    • machine learning
    • astronomy

7 of 25

Scientific Questions

Transients

Progenitors of stellar explosions (outermost layers) & explosion physics (ejecta structure)

Variable stars

Low mass microlensing events, changing mode stellar pulsators, rapid reaction to eclipsing events, eruptive events

Supermassive black holes

Changing state AGNs, reverberation mapping studies, detection of intermediate mass black holes, tidal disruption events

8 of 25

ALeRCE infrastructure

Distributed Storage

Container orchestrator

Distributed messaging

Distributed database

Services

Feature computation

Model training

Training

Deployment

9 of 25

ALeRCE pipeline

Code your step

Install APF

Configure your step

Run locally

See examples

Build the image

Deploy

10 of 25

Classification

Taxonomy

Complex & growing taxonomy

Stamp classifier

Light curve classifier

+ Forecasting service

+ Outlier detection

Classification models

11 of 25

Light Curve Classifier

Stamp Classifier

Convolutional Neural Network

Carrasco-Davis et al. 2020 (Submitted)

Hierarchical Random Forest Classifier

(using light curves with at least 6 observations)

Sánchez-Sáez et al. 2020 (Submitted)

AGN, SN, VS, asteroid, bogus

SN Ia, SN Ibc, SN II, SLSNe,

QSO, AGN, Blazar, YSO, CV/Nova,

LPV, E, DSCT, RRL, CEP, Periodic Other

12 of 25

Stamp Classifier vs Light Curve Classifier

13 of 25

SNe detected by ALeRCE (stamp classifier)

ALeRCE reports more SNe within the first day of detection, using only the public alert stream. We do not report previously reported SNe.

14 of 25

Why single stamp classification?

First detection magnitude vs initial magnitude change rate

1st ZTF detection

+8 hr ALeRCE TNS report

+21 hr spectroscopic confirmation (Ib)

+63 hr 2nd ZTF detection

1st ZTF detection

+4.5 hr ALeRCE TNS report

+42 hr spectroscopic confirmation (Ic)

+96 hr 2nd ZTF detection

One detection reports can be critical to catching very young SNe.

15 of 25

Light curve classifier candidates

Number of candidates per class obtained for 868,371 sources with enough alerts until 2020/06/09

Normalized magnitude distributions in the r band for sources in the labeled set (LS; red) and candidates from the unlabeled ZTF set (blue)

Extragalactic sources

Galactic sources

16 of 25

Web Interfaces

17 of 25

Web Interfaces

Jupyter Notebooks

TOMs

Output stream

(real-time follow-up)

http://alerce.science

API

18 of 25

API

  • ZTF Database:

http://ztf.alerce.online

  • Avro/Stamps:

http://avro.alerce.online

  • catsHTM Cone Search & Xmatch:

http://catshtm.alerce.online *

  • Finding Chart API.

https://findingchart.alerce.online

  • TNS API.

https://tns.alerce.online/search

* Soumagnac & Ofek (2018), (Ofek 2014; ascl.soft 07005)

19 of 25

New: database, API & client

Database

Detailed description of new schema here

API

Documentation in http://dev.api.alerce.online

Client

Clone new client and pip install -e .

Notebooks

Supernova, AGN and variable stars

Preview available for LSST PCW.

Any feedback is welcome!

New ALeRCE database

Time series, filter dependent

Static

Static, filter dependent

detection: light curves & other relevant time dependent information.

non-detection: limiting magnitudes

data_quality: data quality related time dependent information

object: basic object statistics

xmatch: points to the detailed xmatch tables (allwise, ps1_ztf, gaia_ztf, ss_stf)

magstat: statistics per band per object

feature: object computed features

reference: object statistics for every reference image used

20 of 25

The new ALeRCE explorer

Modular design

Open source

AWS based

New DB connection

New taxonomy

New classifiers

More statistics

Black hole mode!

Try the beta version!

https://dev.alerce.online

21 of 25

Jupyter Notebooks

  • API

  • Transients

  • Variable Stars

  • Active Galactic Nuclei

  • Asteroids

  • PanSTARRS

  • Avro inspection

New: preview the new DB, API and client (newDB notebooks)

LC and stamp classifiers...

22 of 25

Mobile Phones

23 of 25

Other tools

ALeRCE TOM

(https://tom.alerce.online/)

  • Create Target Manually to connect to Target & Observation Managers
  • Observe scheduler to LCO.
  • API endpoint to create targets.
  • ALeRCE plugin to query ZTF DB.
  • Upcoming: Integration with SN Hunter and Reporter.

Reporter/Challenger (http://reporter.alerce.online/)

  • List SN candidates from SN Hunter
  • List Bogus reports from SN Hunter
  • Create Challenge associated to a training/test sets.
  • Subscribe to a Challenge.
  • Send model results and get Metrics.

Xmatch service (http://xmatch.alerce.online/)

  • Find xmatch between ZTF and your own catalog
  • Connect with ZTF explorer tool

24 of 25

Summary

  • ALeRCE: broker for ZTF, LSST and other large etendue telescopes
  • Interdisciplinary research team born from HiTS survey + young developer team building distributed and scalable system
  • Connecting with transient and variable communities to enable effective follow up and larger impact (SOXS, SCORPIO, 4MOST, MOONS key instruments). We need your feedback!
  • Products: living catalog of objects, annotated & classified streams, API, client, Explorer, SN hunter, TOM toolkit connection, reporter. Spark batch processing.
  • Learning from ZTF to prepare for LSST: infrastructure, databases, classification, visualization, transfer learning, forecasting, outlier detection
  • A large effort is needed by the community to compile training sets in preparation for the new paradigm of machine learning aided astronomy!

25 of 25

Follow us...