Generation 3 Middleware overview : (some of) the same slides as last - - PowerPoint PPT Presentation

Generation 3 Middleware

• verview: (some of) the same slides as last time, almost

status: not the same slides as last time!

LSST2018

What's in the Gen3 Middleware?

The SuperTask Framework: a replacement for CmdLineTask

• Pipelines are lists of SuperTasks
• Preflight turns a Pipeline and a Data ID Expression into a

directed acyclic QuantumGraph whose nodes are Datasets or Quanta.

• Execution runs the processing described by a QuantumGraph

(possibly in parallel, possibly at scale).

2

PipelineTask PipelineTasks

LSST2018

What's in the Gen3 Middleware?

The Generation 3 Butler: a replacement for the current Butler

• Almost the same get and put interface for reading and writing

Datasets.

• Predefined data model for Data ID keys, called DataUnits, that

lets us capture the relationships between different kinds of data.

• New interfaces for subsets and other expressions.
• Very different data repository concept.
• Totally different implementation.

3

LSST2018

What's a PipelineTask?

Just a CmdLineTask, with a few modifications:

• configurable, introspectable input/output DatasetTypes
• a runQuantum method that does the work

Unlike CmdLineTask, SuperTasks don't run themselves or parse arguments; they provide enough information for something else to run them.

4

LSST2018

What's a Quantum?

An indivisible, independent unit of processing. A Quantum has:

• A PipelineTask that executes it.
• A collection of input Datasets (subdivided into predicted and

actual).

• A collection of output Datasets.

These naturally form a directed acyclic graph (DAG), which we call the QuantumGraph.

5

QuantumGraph

quantum 1 Task=ProcessCcd raw visit=320 sensor=10 raw visit=321 sensor=10 quantum 2 Task=ProcessCcd src visit=320 sensor=10 PVI visit=320 sensor=10 quantum 3 Task=JointCal src visit=321 sensor=10 PVI visit=321 sensor=10 WCS visit=321 sensor=10 WCS visit=320 sensor=10 quantum 4 Task=MakeWarps quantum 5 Task=MakeWarps quantum 6 Task=AssembleCoadd warp visit=320 tract=12 patch=50 warp visit=321 tract=12 patch=50 coadd tract=12 patch=50 filter=r flat begin=2019-04-10 sensor=10 filter=r

6

LSST2018

What's a Pipeline?

A list of PipelineTask classes and their Config instances. It's conceptually a graph as well (though it will behave like flat a collection in Python).

ProcessCcd MakeWarps AssembleCoadd raw PVI warp JointCal coadd src WCS flat

7

LSST2018

How it all works

Preflight Solver Pipeline

PipelineTask 1 Config 1 PipelineTask 2 Config 2 PipelineTask N Config N

Data ID Expression Preflight Registry QuantumGraph

Quantum 1 Quantum 2 Quantum M

Execution Configuration Execution Butler(s) Pipeline Executor

Executor Utilities Common Configuration SuperTask Core Framework PipelineTask Extensions Operator Inputs Gen3 Butler Components

8

LSST2018

Gen3 Butler Components

Master Repository

stores and manages Datasets

Registry

SQL database that manages metadata groups Datasets into Collections labels Datasets with DataUnits defines DatasetTypes dynamically

Datastore

stores Datasets understands file formats

Butler

high-level interface to the Datasets in a single Collection

9

From Butler From user

Butler.get

RC_w_2018_42 Collection camera=HSC DataUnit sensor=30 DataUnit

Registry

PVI DatasetType

dataset_id Datastore Python Object

10

From Butler From user

Butler.put

RC_w_2018_42 Collection camera=HSC DataUnit sensor=30 DataUnit

Registry

PVI DatasetType

dataset_id Datastore Python Object

11

LSST2018

Gen3 Butler Design Principles

Make our data model explicit, as a SQL schema.

• Different cameras are mapped to our data model; we don't map
• ur processing to theirs.
• Predefine the set of allowed keys for Data ID dicts ("DataUnits").
• The relationships between DataUnits are part of the data model:

○ Many-to-One: e.g. a Visit has a PhysicalFilter. ○ Many-to-Many: e.g. a Visit-Sensor combination spatially

• verlaps several Tract-Patch combinations.

12

LSST2018

Gen3 Butler Design Principles

Separate Camera-specialization from filename mapping.

• Mappers will no longer exist.
• Cameras are themselves DataUnits, and obs_* packages are

responsible for defining other DataUnits associated with Cameras.

• When we have files, filename templates will be customizable in

the Butler client config at the DatasetType level and the Camera level.

13

LSST2018

Gen3 Butler Design Principles

Define Collections in the database, not the filesystem.

• Multiple cameras, runs, etc. in a single data repository.
• Datasets may be associated with multiple Collections, which are

just database tags.

• A Butler instance retrieves Datasets from a single Collection.
• The Collection for a processing run typically includes its inputs

and its outputs (simulates Gen2 repository chaining).

14

LSST2018

Schema: DataUnits

also, without tables:

• ExposureRange
• SkyPix
• Label

15

LSST2018

Schema: Dataset

DataUnit fields Not shown: provenance tables (details are just distracting at this level)

16

LSST2018

Gen3 Butler Registry Status

SQLite Registry is basically working for low-level operations:

• registering DatasetTypes
• registering Cameras and SkyMaps
• get / put / single-file ingest
• single-Camera, single-SkyMap expressions for Preflight
• transactions

What's missing:

• Subsets/Transfers (copy stuff from one repo to another)
• Provenance is nominally there, but hasn't really been tested.
• Details of the schema have not been finalized.

17

LSST2018

Gen3 Butler Datastore Status

PosixDatastore is basically working for low-level operations:

• get / put
• single-file ingest
• composites

Chained- and InMemoryDatastore have been prototyped; need Registry support to really be evaluatored What's missing:

• testing, polishing, edge-case support
• more concrete Datastores implementations

18

LSST2018

Gen3 Butler Status: Repo Conversion

We already have a script to (non-destructively) create a snapshot Gen3 view into a set of Gen2 repos. It's actually running in ci_hsc - after the (Gen2-based) processing completes, we make a Gen3 repo and re-run all of the checks. It doesn't do everything yet:

• raw data isn't transformed on read properly
• the script doesn't work on calibration repos yet.

19

LSST2018

Why we can't all use Gen3 Butler yet

20

It's integration time!

• obs_ package integration is totally different (no Mappers!), and

still being prototyped.

• There are a lot of hacks in obs_ packages we don't want to

propagate to Gen3; that means we need to actually fix those issues properly.

• It's not quite a drop-in replacement, so our CmdLineTasks

can't use it directly.

LSST2018

PipelineTask Status

Base class is ready to be used, but many of our concrete CmdLineTasks aren't ready to use it - so we're getting them in shape first (e.g. run -> runDataRef, butlers in LoadReferencesTask). Preflight and "laptop" execution framework are working - in simple tests, with toy-example PipelineTasks. But no one has "run them in anger."

21

LSST2018

What We're Doing Now

• About to start writing PipelineTask versions of existing

CmdLineTasks.

○ PipelineTask versions will use Gen3 Butler,

CmdLineTask versions will continue to use Gen2.

• Putting together raw ingest hooks and customizations in obs_

packages.

• Standing up first versions of production Registries and

Datastores.

22

LSST2018

What We're Doing Now

Overall: The basic pieces are there, but we need to use them to learn how they need to be improved. To be able to use them, we need:

• concrete PipelineTasks that aren't just toy examples
• real obs_ packages with Gen3 integration

23