http://ls.st/lse-163 Zeljko Ivezic, Robert Lupton & Mario Juric - PowerPoint PPT Presentation

Data Products Definition Document (LSE-163) 
 
 http://ls.st/lse-163 Zeljko Ivezic, Robert Lupton & Mario Juric LSST JTM Meeting, Mar 6, 2017

Why DPDD? http://ls.st/lse-163

LSST Products in LSST Science Requirements

Large Synoptic Survey Telescope Data Products Definition Document DPDD LSST Document LSE-163 Mario Juri´ c ∗ , R.H. Lupton, T. Axelrod, J.F. Bosch, G.P. Dubois-Felsmann, ˇ Z. Ivezi´ c, A.C. Becker, J. Becla, A.J. Connolly, M. Freemon, J. Kantor, K-T Lim, D. Shaw, M. Strauss, and J.A. Tyson for the LSST Project May 5, 2016 Abstract This document describes the data products and processing services to be delivered by the Large Synoptic Survey Telescope (LSST). The LSST will deliver three levels of data products and services. Level 1 (nightly) data products will include images, di ff erence im- ages, catalogs of sources and objects detected in di ff erence images, and catalogs of Solar System objects. Their primary purpose is to enable rapid follow-up of time-domain events. Level 2 (annual) data products will include well calibrated single-epoch images, deep coadds, and catalogs of objects, sources, and forced sources, enabling static sky and precision time-domain science. Level 3 (user-created) data product services will enable science cases that greatly benefit from co-location of user processing and/or data within the LSST Archive Center. LSST will also devote 10% of observing time to programs with special cadence. Their data products will be created using the same software and hardware as Levels 1 and 2. All data products will be made available using user-friendly databases and web services. ∗ Please direct comments to < mjuric@lsst.org > .

Contents 1 Preface 4 2 Introduction 5 2.1 The Large Synoptic Survey Telescope . . . . . . . . . . . . . . 5 2.2 General Image Processing Concepts for LSST . . . . . . . . . 6 2.3 Classes of LSST Data Products . . . . . . . . . . . . . . . . . 7 2.4 Conceptual Design of Science Pipelines . . . . . . . . . . . . . 9 3 General Considerations 16 3.1 Estimator and Naming Conventions . . . . . . . . . . . . . . . 16 3.2 Image Characterization Data . . . . . . . . . . . . . . . . . . . 17 3.3 Fluxes and Magnitudes . . . . . . . . . . . . . . . . . . . . . . 18 3.4 Uniqueness of IDs across database versions . . . . . . . . . . . 19 3.5 Repeatability of Queries . . . . . . . . . . . . . . . . . . . . . 19 4 Level 1 Data Products 20 4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 4.2 Level 1 Data Processing . . . . . . . . . . . . . . . . . . . . . 21 4.2.1 Di ff erence Image Analysis . . . . . . . . . . . . . . . . 21 4.2.2 Solar System Object Processing . . . . . . . . . . . . . 23 4.3 Level 1 Catalogs . . . . . . . . . . . . . . . . . . . . . . . . . 24 4.3.1 DIASource Table . . . . . . . . . . . . . . . . . . . . . 26 4.3.2 DIAObject Table . . . . . . . . . . . . . . . . . . . . . 32 4.3.3 SSObject Table . . . . . . . . . . . . . . . . . . . . . . 34 4.3.4 Precovery Measurements . . . . . . . . . . . . . . . . . 36 4.3.5 Reprocessing the Level 1 Data Set . . . . . . . . . . . . 36 4.4 Level 1 Image Products . . . . . . . . . . . . . . . . . . . . . . 38 4.4.1 Visit Images . . . . . . . . . . . . . . . . . . . . . . . . 38 4.4.2 Di ff erence Images . . . . . . . . . . . . . . . . . . . . . 38 4.4.3 Image Di ff erencing Templates . . . . . . . . . . . . . . 38 4.5 Alerts to DIASources . . . . . . . . . . . . . . . . . . . . . . . 39 4.5.1 Information Contained in Each Alert . . . . . . . . . . 39 4.5.2 Receiving and Filtering the Alerts . . . . . . . . . . . . 40

CONTENTS 3 5 Level 2 Data Products 42 5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 5.2 Level 2 Data Processing . . . . . . . . . . . . . . . . . . . . . 43 5.2.1 Object Characterization Measures . . . . . . . . . . . . 45 5.2.2 Supporting Science Cases Requiring Full Posteriors . . 48 5.2.3 Source Characterization . . . . . . . . . . . . . . . . . 49 5.2.4 Forced Photometry . . . . . . . . . . . . . . . . . . . . 50 5.2.5 Crowded Field Photometry . . . . . . . . . . . . . . . 50 5.3 The Level 2 Catalogs . . . . . . . . . . . . . . . . . . . . . . . 50 5.3.1 The Object Table . . . . . . . . . . . . . . . . . . . . . 51 5.3.2 Source Table . . . . . . . . . . . . . . . . . . . . . . . 57 5.3.3 ForcedSource Table . . . . . . . . . . . . . . . . . . . 60 5.4 Level 2 Image Products . . . . . . . . . . . . . . . . . . . . . . 61 5.4.1 Visit Images . . . . . . . . . . . . . . . . . . . . . . . . 61 5.4.2 Calibration Data . . . . . . . . . . . . . . . . . . . . . 61 5.4.3 Coadded Images . . . . . . . . . . . . . . . . . . . . . 61 5.5 Data Release Availability and Retention Policies . . . . . . . . 63 6 Level 3 Data Products and Capabilities 65 6.1 Level 3 Data Products and Associated Storage Resources . . . 65 6.2 Level 3 Processing Resources . . . . . . . . . . . . . . . . . . . 66 6.3 Level 3 Programming Environment and Framework . . . . . . 67 6.4 Migration of Level 3 data products to Level 2 . . . . . . . . . 69 7 Data Products for Special Programs 70

The main classes of LSST data products: 1) Images : single visit, coadded images, difference images 2) Catalogs : Level 1: DIA Sources, DIA Objects, SS Objects, Alerts Level 2: Sources, Forced Sources, Objects 3) Alerts


 
 Motivation for these revisions: - it’s been ~3 years since the last revisions of DPDD (LSE-163, what) and “Science Pipelines” Document (LDM-151, how) - a lot of DM development happened over last 2 years: useful feedback and potential for design improvements - last chance for major edits (~3 years to first light!)

The main changes in DPDD: - a pedagogical introduction to overall pipeline design, with high-level block diagrams: provide a reader (both stakeholders and developers) with an understanding of the complete DM system to be constructed or operated - changes in nightly Level 1 processing and a simplification of its interaction with DRP (data release processing): tentative - changes in forced photometry: measured on both direct and difference images: tentative - revisions of measured parameters (e.g. dipole fit for DIA Sources, adaptive moments, options for galaxy models) - general cleanup

DPDD: Fun for the whole Subsystem! http://ls.st/LSE-163

Single Visit Processing - “classical” astronomical image processing (e.g. similar to SDSS) - prototype implementation operational on HSC data - rudimentary QA analysis tools available - PSF pipeline is a non-trivial task - calibration products pipeline is implied above

Image Coaddition - algorithmic research not completed - rudimentary implementation available Detect and deblend sources - association issues not fully resolved - rudimentary implementation available - “StackFit” measurements Multifit - modeling issues not fully resolved - requires major resources - direct impact on science

Image Differencing - algorithmic research not completed (DCR, seeing) - rudimentary implementation available Difference image analysis - rudimentary implementation available - real time (60 sec) processing, which must be exceedingly robust: scary! Alert Generation - if DIA works fine, it shouldn’t be too hard to construct Alert stream - lots of issues with external brokers

MOPS - rudimentary implementation available - recent work in the context of NEOs showed that MOPS is in a much more sorry state than we thought - in addition to components that we don’t have, we might need to rewrite those that we thought we did have

Baseline: we plan to “catch up” with the new data acquiring during the DR processing period, and then replace the “live” Level 1 db with the new DRP one. This switch would likely be disruptive for Level 1 science and downstream brokers. Alternative proposal: - use a running 6-12 month wide window to compute summary quantities in DIA Objects: a “living” Level 1 db (DIA Objects retain the memory of only that window, and not of all the old data) - freeze the dataset at the beginning of DR processing - produce Level 1 db as part of DRP (using all data), but do not replace the “living” Level 1 db Level 1 DR Processing: - produce difference images for all visits and find all DIA Sources - associate DIA Sources to DIA Objects and SS Objects - go back (2nd pass) to single-visit images and difference images and perform all required processing for all DIA Sources from DIAObjects and SS Objects (image models for SNR>5 DIA Sources and Forced PSF Photometry for all DIA Sources)