Large-scale EXecution for Industry & Society - PowerPoint PPT Presentation

Co-funded by the Horizon 2020 Framework Programme of the European Union Grant Agreement Number 825532 Large-scale EXecution for Industry & Society www.lexis-project.eu HPC, BIG DATA, IOT AND AI FUTURE INDUSTRY-DRIVEN COLLABORATIVE STRATEGIC TOPICS (PART 2) STEPHAN HACHINGER Leibniz Supercomputing Centre MARC LEVRIER Atos 1 | LEXIS Data System

LEXIS PROJECT CHALLENGES Dynamic data-aware and complex workflows orchestration • On both Cloud and HPC resources ◦ ◦ Federation of participating supercomputing systems ◦ Real-time deadline-aware workflows over both Cloud and HPC Data sharing between Cloud and HPC resources • ◦ Accelerated by dedicated Burst Buffer nodes, high bandwidth network and FPGA cards for on-line processing Cross-site data and metadata management • Distributed data management & Big Data and Data Discovery ◦ • Harmonised orchestration across data centres with their local AAI Providing access to the HPC/BD/Cloud resources also for SMEs/Industry to: • execute complex workflows ◦ real time resource consumption data monitoring ◦ accounting and billing information spanning multiple HPC centres ◦ Web portal and interfaces for workflow specification and execution • Seamless integration of remote visualization services • 2 | LEXIS Data System

LEXIS PLATFORM AND DATA HANDLING LEXIS Workflows need fast & flexible data exchange DDI endpoints: Distributed Data Infrastructure (iRODS/EUDAT-B2SAFE) 3 | LEXIS Data System

LEXIS PLATFORM – DISTRIBUTED DATA INFRASTRUCTURE Key Points Unified view on data in LEXIS – federated iRODS zones (IT4I, LRZ, …) • “filesystem-like”, top-level directories e.g. /IT4ILexisZone • transparent access to all files via all iRODS servers • Physical file storage policies implemented as iRODS rules, e.g. • Cross-site mirroring • Low-level storage tiering (in each computing/data centre) • “Non-invasive” data-curation approach • DataCite metadata subset stored in iRODS • EUDAT-B2HANDLE PIDs • Directory/access-rights structure fixed on project (top-)level • Uses LEXIS cross-provider AAI • 4 | LEXIS Data System

LEXIS DISTRIBUTED DATA INFRASTRUCTURE Leveraging iRODS & EUDAT B2SAFE (and B2HANDLE, B2STAGE) HPC Cloud HPC Cloud Burst Burst Buffer Buffer orchestrated orchestrated via staging via staging REST API REST API iRODS/iCAT iRODS/iCAT Servers LRZ Servers IT4I (redundant) (redundant) FEDERATION – MIRRORING – PREFETCH LRZLexisZone IT4ILexisZone LRZ: „DSS“ IT4I: IBM Spectrum CEPH EUDAT/B2SAFE Scale/GPFS Storage 5 | LEXIS Data System

LEXIS DATA SYSTEM Moving and accessing data via APIs Portal / Monitoring Data / Workflows / Visualisation System Data Data staging/ Monitoring/ AAI Discovery up-/download Billing REST API REST APIs API (Authentication & Authorization Infrastructure) DDI (Distributed Data Infrastructure with Metadata Handling / FAIR) Local Storage Weather & Climate Libraries/ Systems Data API + Storage 6 | LEXIS Data System

LEXIS CONTACTS Large-scale EXecution Stephan Hachinger (LRZ, WP3 lead) for Industry & Society stephan.hachinger@lrz.de Marc Levrier (Atos, WP2 lead) marc.levrier@atos.net Jan Martinovi č (IT4I, Project coordinator) jan.martinovic@vsb.cz CONSORTIUM 7 | LEXIS Data System