Large-scale EXecution for Industry & Society - - PowerPoint PPT Presentation

▶

Aug 27, 2023 514 likes •599 views

Co-funded by the Horizon 2020 Framework Programme of the European Union Grant Agreement Number 825532 Large-scale EXecution for Industry & Society www.lexis-project.eu HPC, BIG DATA, IOT AND AI FUTURE INDUSTRY-DRIVEN COLLABORATIVE

SLIDE 1

1 | LEXIS Data System

Co-funded by the Horizon 2020 Framework Programme of the European Union Grant Agreement Number 825532

www.lexis-project.eu

Large-scale EXecution for Industry & Society

STEPHAN HACHINGER Leibniz Supercomputing Centre

HPC, BIG DATA, IOT AND AI FUTURE INDUSTRY-DRIVEN COLLABORATIVE STRATEGIC TOPICS (PART 2)

MARC LEVRIER Atos

SLIDE 2

2 | LEXIS Data System

LEXIS PROJECT CHALLENGES

Dynamic data-aware and complex workflows orchestration
On both Cloud and HPC resources
Federation of participating supercomputing systems
Real-time deadline-aware workflows over both Cloud and HPC
Data sharing between Cloud and HPC resources
Accelerated by dedicated Burst Buffer nodes, high bandwidth network and FPGA cards for on-line processing
Cross-site data and metadata management
Distributed data management & Big Data and Data Discovery
Harmonised orchestration across data centres with their local AAI
Providing access to the HPC/BD/Cloud resources also for SMEs/Industry to:
execute complex workflows
real time resource consumption data monitoring
accounting and billing information spanning multiple HPC centres
Web portal and interfaces for workflow specification and execution
Seamless integration of remote visualization services

SLIDE 3

3 | LEXIS Data System

LEXIS PLATFORM AND DATA HANDLING

LEXIS Workflows need fast & flexible data exchange DDI endpoints: Distributed Data Infrastructure (iRODS/EUDAT-B2SAFE)

SLIDE 4

4 | LEXIS Data System

LEXIS PLATFORM – DISTRIBUTED DATA INFRASTRUCTURE

Key Points

Unified view on data in LEXIS – federated iRODS zones (IT4I, LRZ, …)
“filesystem-like”, top-level directories e.g. /IT4ILexisZone
transparent access to all files via all iRODS servers
Physical file storage policies implemented as iRODS rules, e.g.
Cross-site mirroring
Low-level storage tiering (in each computing/data centre)
“Non-invasive” data-curation approach
DataCite metadata subset stored in iRODS
EUDAT-B2HANDLE PIDs
Directory/access-rights structure fixed on project (top-)level
Uses LEXIS cross-provider AAI

SLIDE 5

5 | LEXIS Data System

LEXIS DISTRIBUTED DATA INFRASTRUCTURE

iRODS/iCAT Servers IT4I (redundant)

IT4I: CEPH Storage

iRODS/iCAT Servers LRZ (redundant)

LRZ: „DSS“ IBM Spectrum Scale/GPFS

HPC Cloud

FEDERATION – MIRRORING – PREFETCH

LRZLexisZone IT4ILexisZone

EUDAT/B2SAFE

rchestrated

via staging REST API Leveraging iRODS & EUDAT B2SAFE (and B2HANDLE, B2STAGE)

rchestrated

via staging REST API HPC Cloud Burst Buffer Burst Buffer

SLIDE 6

6 | LEXIS Data System

LEXIS DATA SYSTEM

Moving and accessing data via APIs

DDI

(Distributed Data Infrastructure with Metadata Handling / FAIR)

Portal / Data / Workflows / Visualisation AAI

(Authentication

&Authorization

Infrastructure)

Monitoring System Data Discovery REST API Data staging/ up-/download REST APIs Monitoring/ Billing API

Weather & Climate Data API + Storage Local Storage Libraries/ Systems

SLIDE 7

7 | LEXIS Data System

CONSORTIUM

LEXIS CONTACTS

Large-scale EXecution for Industry & Society

Stephan Hachinger (LRZ, WP3 lead) stephan.hachinger@lrz.de Marc Levrier (Atos, WP2 lead) marc.levrier@atos.net Jan Martinovič (IT4I, Project coordinator) jan.martinovic@vsb.cz