Large-scale EXecution for Industry & Society - - PowerPoint PPT Presentation

large scale execution for industry society
SMART_READER_LITE
LIVE PREVIEW

Large-scale EXecution for Industry & Society - - PowerPoint PPT Presentation

Co-funded by the Horizon 2020 Framework Programme of the European Union Grant Agreement Number 825532 Large-scale EXecution for Industry & Society www.lexis-project.eu HPC, BIG DATA, IOT AND AI FUTURE INDUSTRY-DRIVEN COLLABORATIVE


slide-1
SLIDE 1

1 | LEXIS Data System

Co-funded by the Horizon 2020 Framework Programme of the European Union Grant Agreement Number 825532

www.lexis-project.eu

Large-scale EXecution for Industry & Society

STEPHAN HACHINGER Leibniz Supercomputing Centre

HPC, BIG DATA, IOT AND AI FUTURE INDUSTRY-DRIVEN COLLABORATIVE STRATEGIC TOPICS (PART 2)

MARC LEVRIER Atos

slide-2
SLIDE 2

2 | LEXIS Data System

LEXIS PROJECT CHALLENGES

  • Dynamic data-aware and complex workflows orchestration
  • On both Cloud and HPC resources
  • Federation of participating supercomputing systems
  • Real-time deadline-aware workflows over both Cloud and HPC
  • Data sharing between Cloud and HPC resources
  • Accelerated by dedicated Burst Buffer nodes, high bandwidth network and FPGA cards for on-line processing
  • Cross-site data and metadata management
  • Distributed data management & Big Data and Data Discovery
  • Harmonised orchestration across data centres with their local AAI
  • Providing access to the HPC/BD/Cloud resources also for SMEs/Industry to:
  • execute complex workflows
  • real time resource consumption data monitoring
  • accounting and billing information spanning multiple HPC centres
  • Web portal and interfaces for workflow specification and execution
  • Seamless integration of remote visualization services
slide-3
SLIDE 3

3 | LEXIS Data System

LEXIS PLATFORM AND DATA HANDLING

LEXIS Workflows need fast & flexible data exchange DDI endpoints: Distributed Data Infrastructure (iRODS/EUDAT-B2SAFE)

slide-4
SLIDE 4

4 | LEXIS Data System

LEXIS PLATFORM – DISTRIBUTED DATA INFRASTRUCTURE

Key Points

  • Unified view on data in LEXIS – federated iRODS zones (IT4I, LRZ, …)
  • “filesystem-like”, top-level directories e.g. /IT4ILexisZone
  • transparent access to all files via all iRODS servers
  • Physical file storage policies implemented as iRODS rules, e.g.
  • Cross-site mirroring
  • Low-level storage tiering (in each computing/data centre)
  • “Non-invasive” data-curation approach
  • DataCite metadata subset stored in iRODS
  • EUDAT-B2HANDLE PIDs
  • Directory/access-rights structure fixed on project (top-)level
  • Uses LEXIS cross-provider AAI
slide-5
SLIDE 5

5 | LEXIS Data System

LEXIS DISTRIBUTED DATA INFRASTRUCTURE

iRODS/iCAT Servers IT4I (redundant)

IT4I: CEPH Storage

iRODS/iCAT Servers LRZ (redundant)

LRZ: „DSS“ IBM Spectrum Scale/GPFS

HPC Cloud

FEDERATION – MIRRORING – PREFETCH

LRZLexisZone IT4ILexisZone

EUDAT/B2SAFE

  • rchestrated

via staging REST API Leveraging iRODS & EUDAT B2SAFE (and B2HANDLE, B2STAGE)

  • rchestrated

via staging REST API HPC Cloud Burst Buffer Burst Buffer

slide-6
SLIDE 6

6 | LEXIS Data System

LEXIS DATA SYSTEM

Moving and accessing data via APIs

DDI

(Distributed Data Infrastructure with Metadata Handling / FAIR)

Portal / Data / Workflows / Visualisation AAI

(Authentication

&Authorization

Infrastructure)

Monitoring System Data Discovery REST API Data staging/ up-/download REST APIs Monitoring/ Billing API

Weather & Climate Data API + Storage Local Storage Libraries/ Systems

slide-7
SLIDE 7

7 | LEXIS Data System

CONSORTIUM

LEXIS CONTACTS

Large-scale EXecution for Industry & Society

Stephan Hachinger (LRZ, WP3 lead) stephan.hachinger@lrz.de Marc Levrier (Atos, WP2 lead) marc.levrier@atos.net Jan Martinovič (IT4I, Project coordinator) jan.martinovic@vsb.cz