Towards Structure-Aware Earth System Data Management Jakob Lttgau, - - PowerPoint PPT Presentation

▶

towards structure aware earth system data management

Towards Structure-Aware Earth System Data Management Jakob Lttgau, - - PowerPoint PPT Presentation

Sep 13, 2023 268 likes •346 views

Towards Structure-Aware Earth System Data Management Towards Structure-Aware Earth System Data Management Jakob Lttgau, Julian Kunkel, Bryan N. Lawrence, Sandro Fiore, Huang Hua Jakob Luettgau (DKRZ) Middleware for Earth System Data PDSW SC17

slide-1

SLIDE 1

Towards Structure-Aware Earth System Data Management

Towards Structure-Aware Earth System Data Management

Jakob Lüttgau, Julian Kunkel, Bryan N. Lawrence, Sandro Fiore, Huang Hua

Jakob Luettgau (DKRZ) Middleware for Earth System Data PDSW SC17 1 / 5

slide-2

SLIDE 2

Towards Structure-Aware Earth System Data Management

How to manage Earth System Data?

What to optimize? Write throughput on generation? Avoid transformation?

Jakob Luettgau (DKRZ) Middleware for Earth System Data PDSW SC17 2 / 5

slide-3

SLIDE 3

Towards Structure-Aware Earth System Data Management

Data Representations

Different views to the same data. Suboptimal serialization on storage.

Jakob Luettgau (DKRZ) Middleware for Earth System Data PDSW SC17 3 / 5

slide-4

SLIDE 4

Towards Structure-Aware Earth System Data Management

Middleware for Earth System Data

Adaptively choose backends. Discriminate by data, metadata and access type.

Jakob Luettgau (DKRZ) Middleware for Earth System Data PDSW SC17 4 / 5

slide-5

SLIDE 5

Towards Structure-Aware Earth System Data Management

Summary, Status and Outlook

Architectures likely to become more heterogeneous Systems prohibitively complex for manual optimization Status Reports and design documents publicly available:

http://esiwace.eu (WP4, Deliverables 4.1 and 4.2)

Prototype to demonstrate viability of adaptive tier selection Outlook Open development of middleware, licensed under LGPL:

http://github.com/ESiWACE

Backends being developed for Object Storage and MongoDB NVM backends as hardware becomes more widely available

Jakob Luettgau (DKRZ) Middleware for Earth System Data PDSW SC17 5 / 5

slide-6

SLIDE 6

The ESiWACE project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 675191

Disclaimer: This material reflects only the author’s view and the EU-Commission is not responsible for any use that may be made of the information it contains

Jakob Luettgau (DKRZ) Middleware for Earth System Data PDSW SC17 6 / 5