DM System Design and LDM-148 KIAN-TAT LIM, DM SOFTWARE ARCHITECT - - PowerPoint PPT Presentation

DM System Design and LDM-148

KIAN-TAT LIM, DM SOFTWARE ARCHITECT

What Are We Building?

Title Text

2 Joint Technical Meeting - 2017-03-06 - Glendale

A system composed of services that need to be operated.

• Note that not all services built by DM or using DM software are operated by the Data

Products and Processing Department; some are operated by the Science Operations Department and some by the Observatory Operations Department. Running on software that needs to be released as a product as well as data center infrastructure that needs to be maintained. Everything that follows is "to the best of my current understanding"; it may be wrong and should be challenged if wrong now or in the future.

3

Service and Payload Components

Data Management System

Data Backbone Level 1 Domain Users Community Alert Brokers Level 2 Domain Prompt Processing Offline Processing OCS Driven Batch Image and EFD Archiving Level 1 Quality Control Alert Broker Feed Telemetry Gateway Alert Mini- Broker Calibration Products Production Data Release Production Level 2 Quality Control Data Access Centers (x2) Bulk Data Distribution Data Access Web Svcs Science Platform (DAC) Other Data Partners Observatory Alert Users EPO Analysis and Developer Support Science Platform (Commiss.) Developer Services Integration & Test Science Platform (Sci. Verif.) Calibration QA Payload Alert Production Payload Monthly Calibration Payload Annual Calibration Payload Template Generation Payload DRP Payload DayMOPS Payload Daily Cal. Update Payload

Domains are not exactly enclaves or centers

4

Software "Stack"

Middleware and Utilities Frameworks Plugins

afw meas_base meas_modelfit meas_extensions_ shapeHSM cameraGeom coord detection display geom image math table pex_exceptions, base, utils daf_base pex_config log

• bs_base

ip_isr ip_diffim

• bs_cfht
• bs_decam
• bs_subaru
• bs_lsstSim

jointcal pipe_tasks pipe_base coadd_utils meas_algorithms meas_deblender meas_astrom meas_extensions_ convolved

Middleware/utilities, frameworks, and "plugins" (or independent computation packages) "obs_" packages are more middleware configuration than computation

5

Middleware and Infrastructure

Task Framework

Config API Logging API MultiNodeTask API MultiCoreTask API Control and Orchestration Authentication & Authorization Provisioning & Resource Management Monitoring Data Butler Science Algorithms Activator (base) SuperTask Task Command Line Activator SUIT Activator Orchestration Activator

Data Backbone

RDBMS Qserv Files

How do Stack apps become services? Task framework, control and orchestration, other key infrastructure

Future of LDM-148

6 Joint Technical Meeting - 2017-03-06 - Glendale

Brief summaries of the functions of each component Interfaces between components and outside the subsystem Links to requirements in both the DM System Requirements (LSE-61) and Interface Control Documents (ICDs) Partial draft available now at https://ldm-148.lsst.io/v/draft Components complete by end of March Interfaces and linkages complete by DM Review dry run