A Web-Based Platform for Publication and Distributed Execution of - - PowerPoint PPT Presentation

▶
a web based platform for publication and distributed
SMART_READER_LITE
LIVE PREVIEW

A Web-Based Platform for Publication and Distributed Execution of - - PowerPoint PPT Presentation

Nov 27, 2022 249 likes •563 views

14th International Symposium on Parallel and Distributed Computing A Web-Based Platform for Publication and Distributed Execution of Computing Applications Oleg Sukhoroslov, Sergey Volkov, Alexander Afanasiev Institute for Information

slide-1
SLIDE 1

14th International Symposium on Parallel and Distributed Computing

A Web-Based Platform for Publication and Distributed Execution

  • f Computing Applications

Oleg Sukhoroslov, Sergey Volkov, Alexander Afanasiev

Institute for Information Transmission Problems (Moscow, Russia)

slide-2
SLIDE 2

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 2 / 30

Motivation

Researchers Applications Computing Resources

slide-3
SLIDE 3

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 3 / 30

Challenges

  • Convenient access to computational applications
  • Execution of applications on heterogeneous distributed

computing resources

  • Automation of workfmows involving multiple applications
  • Sharing applications/workfmows with colleagues
slide-4
SLIDE 4

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 4 / 30

Current Solutions

Hosted vs Standalone Arbitrary Resources Application Sharing Application Composition

Remote API Grid Middleware User-level Toolkits / Workflow Systems Scientific Gateways Web Service Toolkits

slide-5
SLIDE 5

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 5 / 30

Sharing a Scientifjc Application

  • Publish a source code
  • Send an executable
  • Create a web-based interface
  • Run a web service
slide-6
SLIDE 6

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 6 / 30

Scientifjc Application as a Service

  • Software as a Service (SaaS)
  • No need to install software and deal with computing resources
  • Centralized maintenance and accelerated feature delivery
  • Application composition and integration with third-party tools
  • Collaboration
  • Publication and reproducibility
slide-7
SLIDE 7

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 7 / 30

MathCloud (2009-2013)

  • Software toolkit for building, deployment, discovery and

composition of computational web services

  • Based on the unifjed web service interface

– Follows REST (Representational state transfer) architectural style

  • Main components

– Service Runtime Environment (Container) – Service Catalogue – Workfmow Management System (WfMS) – Security Mechanism – Client Interfaces

slide-8
SLIDE 8

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 8 / 30

MathCloud Architecture

Service Container Workfmow Management System Service Catalogue HTTPS + REST API External Application Web UI

slide-9
SLIDE 9

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 9 / 30

Problems

  • Lack of convenient infrastructure to host services
  • Sharing an application implies sharing a resource
  • Service user cannot override the resource
slide-10
SLIDE 10

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 10 / 30

Everest (2014-...)

  • Leverage cloud computing models to implement a web-based platform

supporting

– Describing and hosting computational applications as services – Binding applications to external computing resources – Running applications on arbitrary sets of resources – Sharing applications and resources with other users

  • Platform as a Service (PaaS)

– Accessible via web browser and REST API – No installation is required

  • Combination of existing approaches + PaaS

– Uniform REST interface for accessing applications – Web UI for application description – Automatic generation of web UI for application invocation

slide-11
SLIDE 11

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 11 / 30

Everest Architecture

slide-12
SLIDE 12

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 12 / 30

Application: Interface

slide-13
SLIDE 13

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 13 / 30

POV-Ray: Parameters

slide-14
SLIDE 14

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 14 / 30

POV-Ray: Submit Form

slide-15
SLIDE 15

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 15 / 30

Application: Implementation

slide-16
SLIDE 16

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 16 / 30

Command Application Skeleton

slide-17
SLIDE 17

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 17 / 30

POV-Ray: Confjguration

./povray_run.sh +Iscene.pov +F${format} +W${width} +H${height} +Q${quality} -D +A -Oimage

slide-18
SLIDE 18

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 18 / 30

Parameter Sweep Application

slide-19
SLIDE 19

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 19 / 30

Example: Virtual Screening

parameter n from 1 to 100 step 1 input_files @run.sh vina write_score.py protein.pdbqt input_files ligand${n}.pdbqt config.txt command ./run.sh

  • utput_files ligand${n}_out.pdbqt log.txt @score

criterion min $affinity

slide-20
SLIDE 20

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 20 / 30

Integration with Computing Resources

slide-21
SLIDE 21

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 21 / 30

Everest Agent

  • A mediator between the resource and the platform
  • Supporting servers, clusters and resources behind a fjrewall
  • Security mechanisms: white list, execution of tasks in Docker containers
  • Open Source: https://gitlab.com/everest/agent/
slide-22
SLIDE 22

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 22 / 30

Integration with EGI

slide-23
SLIDE 23

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 23 / 30

Binding Applications to Resources

slide-24
SLIDE 24

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 24 / 30

Resource Binding: Challenges

  • Dynamic binding

– Protecting users/resources from malicious/broken code

  • Common practices (trust, code signing, verifjcation, publication)
  • Using virtualization and sandboxing solutions (Docker, Firejail)

– Making applications portable across resources

  • Run an application in a preconfjgured Docker container
  • Build a portable application package (CDE, CARE)
  • Binding with multiple resources

– Sheduling of application tasks across heterogeneous

distributed computing resources

slide-25
SLIDE 25

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 25 / 30

Programming Access to Applications

  • Why?

– Automation

  • Repetitve application runs
  • Use of multiple applications (pipelines, workfmows)

– Integration with external systems and third-party tools

  • How?

– Accessing application via web service interface (REST API)

  • HTTP + JSON, any modern programming language

– Using client library (Python API)

  • Implemented on top of REST API
slide-26
SLIDE 26

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 26 / 30

Python API

import everest session = everest.Session( 'https://everest.distcomp.org', token = '...' ) appA = everest.App('52b1d2d13b...', session) appB = everest.App('...', session) appC = everest.App('...', session) appD = everest.App('...', session) jobA = appA.run({'a': '...'}) jobB = appB.run({'b': jobA.output('out1')}) jobC = appC.run({'c': jobA.output('out2')}) jobD = appD.run({'d1': jobB.output('out'), 'd2': jobC.output('out')}) print(jobD.result()) session.close()

slide-27
SLIDE 27

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 27 / 30

Everest Architecture

slide-28
SLIDE 28

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 28 / 30

Experimental Evaluation

  • Setup

– Single server: 2 quad-core Xeon E5620 (2.4 GHz), 24GB RAM, Ubuntu 12.04 – Applications: Sleep, Autodock Vina, Parameter Sweep

  • Raw job submission tests

– Capable of serving 1000 concurrent clients with acceptable latencies – Input fjle uploads negatively impact throughput and latency

  • End-to-end tests (complete job life cycle)

– Job processing overhead introduced by Everest+agent is 10s of seconds – Could be improved to better accommodate short jobs

  • Scalability tests

– 100 agents with 10 slots running in difgerent locations – Maximum observed overhead for 1000 jobs is 23 seconds

  • Real application runs

– Ad-hoc grid: 3 servers + 3 clusters (316 cores) – Autodock Vina, Parameter Sweep application from geophysics domain

slide-29
SLIDE 29

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 29 / 30

Future Work

  • Supporting more complex many-task applications
  • Implementing interaction with a running application
  • Integration with other types of computing resources
  • Optimization of data transfer
  • Effjcient scheduling of applications across multiple resources
  • Improving performance and scalability
slide-30
SLIDE 30

01.07.2015 A Web-Based Platform for Publication and Distributed Execution of Computing Applications 30 / 30

Conclusion

  • A web-based platform supporting scientifjc computing

– Enables users with minimal skills to publish and share scientifjc

applications as services

– Executes applications on external resources attached by users – Implements decoupling of published applications from resources – Supports programmatic access to the platform's functionality

  • More information: http://everest.distcomp.org/