http://ls.st/lse-163 Zeljko Ivezic, Robert Lupton & Mario Juric - - PowerPoint PPT Presentation

Data Products Definition Document (LSE-163) 
 


Zeljko Ivezic, Robert Lupton & Mario Juric LSST JTM Meeting, Mar 6, 2017 http://ls.st/lse-163

Why DPDD?

http://ls.st/lse-163

LSST Products in LSST Science Requirements

Large Synoptic Survey Telescope Data Products Definition Document

LSST Document LSE-163

Mario Juri´ c∗ , R.H. Lupton, T. Axelrod, J.F. Bosch, G.P. Dubois-Felsmann, ˇ

• Z. Ivezi´

c, A.C. Becker, J. Becla, A.J. Connolly, M. Freemon, J. Kantor, K-T Lim, D. Shaw,

• M. Strauss, and J.A. Tyson

for the LSST Project May 5, 2016

Abstract This document describes the data products and processing services to be delivered by the Large Synoptic Survey Telescope (LSST). The LSST will deliver three levels of data products and services. Level 1 (nightly) data products will include images, difference im- ages, catalogs of sources and objects detected in difference images, and catalogs of Solar System objects. Their primary purpose is to enable rapid follow-up of time-domain events. Level 2 (annual) data products will include well calibrated single-epoch images, deep coadds, and catalogs of objects, sources, and forced sources, enabling static sky and precision time-domain science. Level 3 (user-created) data product services will enable science cases that greatly benefit from co-location of user processing and/or data within the LSST Archive

• Center. LSST will also devote 10% of observing time to programs

with special cadence. Their data products will be created using the same software and hardware as Levels 1 and 2. All data products will be made available using user-friendly databases and web services.

∗Please direct comments to <mjuric@lsst.org>.

DPDD

Contents

1 Preface 4 2 Introduction 5 2.1 The Large Synoptic Survey Telescope . . . . . . . . . . . . . . 5 2.2 General Image Processing Concepts for LSST . . . . . . . . . 6 2.3 Classes of LSST Data Products . . . . . . . . . . . . . . . . . 7 2.4 Conceptual Design of Science Pipelines . . . . . . . . . . . . . 9 3 General Considerations 16 3.1 Estimator and Naming Conventions . . . . . . . . . . . . . . . 16 3.2 Image Characterization Data . . . . . . . . . . . . . . . . . . . 17 3.3 Fluxes and Magnitudes . . . . . . . . . . . . . . . . . . . . . . 18 3.4 Uniqueness of IDs across database versions . . . . . . . . . . . 19 3.5 Repeatability of Queries . . . . . . . . . . . . . . . . . . . . . 19 4 Level 1 Data Products 20 4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 4.2 Level 1 Data Processing . . . . . . . . . . . . . . . . . . . . . 21 4.2.1 Difference Image Analysis . . . . . . . . . . . . . . . . 21 4.2.2 Solar System Object Processing . . . . . . . . . . . . . 23 4.3 Level 1 Catalogs . . . . . . . . . . . . . . . . . . . . . . . . . 24 4.3.1 DIASource Table . . . . . . . . . . . . . . . . . . . . . 26 4.3.2 DIAObject Table . . . . . . . . . . . . . . . . . . . . . 32 4.3.3 SSObject Table . . . . . . . . . . . . . . . . . . . . . . 34 4.3.4 Precovery Measurements . . . . . . . . . . . . . . . . . 36 4.3.5 Reprocessing the Level 1 Data Set . . . . . . . . . . . . 36 4.4 Level 1 Image Products . . . . . . . . . . . . . . . . . . . . . . 38 4.4.1 Visit Images . . . . . . . . . . . . . . . . . . . . . . . . 38 4.4.2 Difference Images . . . . . . . . . . . . . . . . . . . . . 38 4.4.3 Image Differencing Templates . . . . . . . . . . . . . . 38 4.5 Alerts to DIASources . . . . . . . . . . . . . . . . . . . . . . . 39 4.5.1 Information Contained in Each Alert . . . . . . . . . . 39 4.5.2 Receiving and Filtering the Alerts . . . . . . . . . . . . 40

CONTENTS 3 5 Level 2 Data Products 42 5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 5.2 Level 2 Data Processing . . . . . . . . . . . . . . . . . . . . . 43 5.2.1 Object Characterization Measures . . . . . . . . . . . . 45 5.2.2 Supporting Science Cases Requiring Full Posteriors . . 48 5.2.3 Source Characterization . . . . . . . . . . . . . . . . . 49 5.2.4 Forced Photometry . . . . . . . . . . . . . . . . . . . . 50 5.2.5 Crowded Field Photometry . . . . . . . . . . . . . . . 50 5.3 The Level 2 Catalogs . . . . . . . . . . . . . . . . . . . . . . . 50 5.3.1 The Object Table . . . . . . . . . . . . . . . . . . . . . 51 5.3.2 Source Table . . . . . . . . . . . . . . . . . . . . . . . 57 5.3.3 ForcedSource Table . . . . . . . . . . . . . . . . . . . 60 5.4 Level 2 Image Products . . . . . . . . . . . . . . . . . . . . . . 61 5.4.1 Visit Images . . . . . . . . . . . . . . . . . . . . . . . . 61 5.4.2 Calibration Data . . . . . . . . . . . . . . . . . . . . . 61 5.4.3 Coadded Images . . . . . . . . . . . . . . . . . . . . . 61 5.5 Data Release Availability and Retention Policies . . . . . . . . 63 6 Level 3 Data Products and Capabilities 65 6.1 Level 3 Data Products and Associated Storage Resources . . . 65 6.2 Level 3 Processing Resources . . . . . . . . . . . . . . . . . . . 66 6.3 Level 3 Programming Environment and Framework . . . . . . 67 6.4 Migration of Level 3 data products to Level 2 . . . . . . . . . 69 7 Data Products for Special Programs 70

The main classes of LSST data products: 1) Images: single visit, coadded images, difference images 2) Catalogs: Level 1: DIA Sources, DIA Objects, SS Objects, Alerts Level 2: Sources, Forced Sources, Objects 3) Alerts

Motivation for these revisions: 
 


• it’s been ~3 years since the last revisions of DPDD

(LSE-163, what) and “Science Pipelines” Document (LDM-151, how)

• a lot of DM development happened over last 2 years:

useful feedback and potential for design improvements

• last chance for major edits (~3 years to first light!)

The main changes in DPDD:

• a pedagogical introduction to overall pipeline design, with

high-level block diagrams: provide a reader (both stakeholders and developers) with an understanding of the complete DM system to be constructed or operated

• changes in nightly Level 1 processing and a simplification of

its interaction with DRP (data release processing): tentative

• changes in forced photometry: measured on both direct and

difference images: tentative

• revisions of measured parameters (e.g. dipole fit for DIA

Sources, adaptive moments, options for galaxy models)

• general cleanup

DPDD: Fun for the whole Subsystem!

http://ls.st/LSE-163

Single Visit Processing

• “classical” astronomical image processing (e.g. similar to SDSS)
• prototype implementation operational on HSC data
• rudimentary QA analysis tools available
• PSF pipeline is a non-trivial task
• calibration products pipeline is implied above

Image Coaddition

• algorithmic research not

completed

• rudimentary

implementation available Detect and deblend sources

• association issues not fully

resolved

• rudimentary

implementation available

• “StackFit” measurements

Multifit

• modeling issues not fully

resolved

• requires major resources
• direct impact on science

Image Differencing

• algorithmic research not

completed (DCR, seeing)

• rudimentary

implementation available Difference image analysis

• rudimentary

implementation available

• real time (60 sec)

processing, which must be exceedingly robust: scary! Alert Generation

• if DIA works fine, it

shouldn’t be too hard to construct Alert stream

• lots of issues with external

brokers

MOPS

• rudimentary

implementation available

• recent work in the

context of NEOs showed that MOPS is in a much more sorry state than we thought

• in addition to

components that we don’t have, we might need to rewrite those that we thought we did have

Baseline: we plan to “catch up” with the new data acquiring during the DR processing period, and then replace the “live” Level 1 db with the new DRP one. This switch would likely be disruptive for Level 1 science and downstream brokers. Alternative proposal:

• use a running 6-12 month wide window to compute summary

quantities in DIA Objects: a “living” Level 1 db (DIA Objects retain the memory of only that window, and not of all the old data)

• freeze the dataset at the beginning of DR processing
• produce Level 1 db as part of DRP (using all data), but do not

replace the “living” Level 1 db Level 1 DR Processing:

• produce difference images for all visits and find all DIA Sources
• associate DIA Sources to DIA Objects and SS Objects
• go back (2nd pass) to single-visit images and difference images and

perform all required processing for all DIA Sources from DIAObjects and SS Objects (image models for SNR>5 DIA Sources and Forced PSF Photometry for all DIA Sources)