Plans and Policies for LSST Alert Distribution Eric Bellm Alert - - PowerPoint PPT Presentation

1

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

Plans and Policies for LSST Alert Distribution Eric Bellm Alert Production Science Lead, LSST DM University of Washington August 16, 2018

2

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

After an LSST image is taken, image subtraction will identify sources that move or vary within 60 seconds. All of these Difference Image Analysis Sources (DIASources) will be packaged with contextual information into world-public alert packets for distribution. The full stream of alerts will be sent to a small number of “community brokers”, who will add value to the stream and provide user access. LSST’s alert packets are world-public. Additionally, LSST data rights holders will have access to a limited-capability simple filtering service (the “mini-broker”). There are many open questions about this ecosystem that we are now resolving.

LSST’s Prompt Processing system will enable real-time science.

3

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

Review of the Alert Production System Overview of the LSST mini-broker Towards a selection process for community brokers Outline

4

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

The Alert Production System

5

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

Alert Production identifies time-varying objects.

6

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

Rich alert packets enable powerful filtering.

Data Products Definition Document ls.st/dpdd

7

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

Testing a system using open source technologies: Apache Avro (binary serialization format for alerts) Apache Kafka (realtime streaming platform; used for alert distribution) LSST-scale experiments detailed in Data Management Technical Notes #28 (dmtn-028.lsst.io) & #81 (dmtn-081.lsst.io) acceptable performance at LSST scale Met NSF Level 2 milestone for Alert Distribution (LDM-503-5) in July A fork of this technology stack has recently been put into production by the Zwicky Transient Facility (ZTF) collaboration, with realtime distribution of alerts to ANTARES, Lasair (UK), ALeRCE (Chile), LCO

We are testing the performance of new alert distribution technologies.

8

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

The LSST Mini-Broker

9

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

Users can create filters that return (in near real-time) a subset of LSST alerts based only on data in the alert packet

e.g., can use lightcurve, variability parameters, colors, etc., but no crossmatches to external catalogs still provides a powerful initial selection

Runs in the LSST Data Access Center

(-> users must have data rights)

Will have TBD capacity limits on runtime, number of passed alerts, number of filters run The LSST mini-broker provides simple filtering capabilities.

10

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

Simple filters can enable powerful searches.

11

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

A range of output channels are being explored.

12

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

Towards selection of community brokers

13

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

May:

• Provide public access to alerts
• Crossmatch to other catalogs or data streams
• Classify events
• Provide filtering, visualization, and search
• Coordinate scientific activity and/or followup observations
• Aggregate alert annotations (community classifications, etc.)
• … probably more!

Community brokers are vital to enabling scientific utilization of the LSST alert stream.

14

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

Outbound bandwidth from the datacenter is the expected bottleneck Early planning led to an expectation of ~4 full streams this is not firmly established; we are revisiting the issue If true capacity is not greater than the number of credible brokers, a broker selection process is needed. The number of community brokers will be finite.

15

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

Require demonstration of technical capability (see next slide) but no functional requirements, e.g.,

• No requirement to receive the full stream
• No requirement to redistribute the full stream
• No requirement to make products world public

But evaluation criteria (subsequent slides) are expected to favor proposals that offer these! Any institution worldwide can propose Hope to enable a rich broker ecosystem offering diverse capabilities

We are considering an “open” call for community brokers.

16

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

Large inbound and outbound network bandwidth (the full alert stream is a few TB/night) Petabytes of disk capacity Databases handling of billions of sources Compute resources to handle sophisticated classification and filtering tasks in real time at scale Appropriate personnel to develop and maintain the service Institutional & funding support to ensure the longevity and stability of the service. Brokers must demonstrate adequate resources to accomplish their proposed goals.

17

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

Serve a large community Enable high-profile science Provide unique capabilities Contribute to LSST’s four science pillars Take advantage of the unique aspects of the LSST alert stream (real-time, world-public) We propose brokers be evaluated on their contribution to the scientific utilization of LSST.

18

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

Scientific value --- see previous slides Availability of data products and services to the world community Scientific Validity --- demonstrated on precursor surveys or data Integration with the Time-Domain Ecosystem --- e.g., followup telescopes, other services Community adoption Complementarity to other selected brokers Applicable existing agreements, if any

Suggested broker evaluation criteria have been drafted.

19

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

2018: Plans and Policies for LSST Alert Distribution document issued 2019: Issue call for Letters of Intent Document bulk transport format and interface Set up canned alert stream with sample precursor data Invite full proposals for community brokers from selected LOIs 2020: Broker proposals due Produce commissioning alerts with ComCam with substantial latency 2021: Finalize number of full streams Selection of community brokers for early operations Produce commissioning alerts with LSSTCam with substantial latency 2022: Integrate selected brokers with the LSST Alert System Begin full LSST operations 2023+: Period reviews of broker performance, potential revisions

Proposed Timeline

20

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

The LSST alert stream provides a powerful real-time view of the dynamic sky, but downstream tools are required to identify

• bjects of interest.

We are drafting policies to guide selection of community brokers and will release them later this year. We are prototyping technologies for the LSST mini-broker. Community brokers using the ZTF alert stream provide a near- term opportunity to test-drive LSST time-domain science. Summary