Combining Data Sharing with the Master-Worker Paradigm in the - - PowerPoint PPT Presentation

▶
combining data sharing with the master worker paradigm in
SMART_READER_LITE
LIVE PREVIEW

Combining Data Sharing with the Master-Worker Paradigm in the - - PowerPoint PPT Presentation

May 06, 2023 262 likes •482 views

Managed by Combining Data Sharing with the Master-Worker Paradigm in the Common Component Architecture Gabriel Antoniu, Hinde Lilia Bouziane, Mathieu Jan, Christian Prez, Thierry Priol PARIS research-team IRISA / INRIA Rennes - France

slide-1
SLIDE 1

Managed by

Combining Data Sharing with the Master-Worker Paradigm in the Common Component Architecture

Gabriel Antoniu, Hinde Lilia Bouziane, Mathieu Jan, Christian Pérez, Thierry Priol

PARIS research-team IRISA / INRIA Rennes - France

slide-2
SLIDE 2

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

2

Outline of the talk

  • Introduction

– Context – Overview of the Common Component Architecture

  • Data sharing paradigm

– Objectives – Our proposition : data port model – Data sharing on operation invocations

  • Support of the master-worker paradigm

– Limits with existing component models – A proposed model

  • Data sharing in a master-worker paradigm
  • Conclusions and perspectives
slide-3
SLIDE 3

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

3

Introduction

  • Context

– Complex applications

  • e-Science

– Different resource kinds

  • multi-core processors
  • SMP machines
  • clusters
  • grids
  • Challenges

– Simplifying application programming – Independence from resource kinds – HPC

u

Component model approach

slide-4
SLIDE 4

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

4

Software component models

  • Black box
  • Ports

– Method invocations, events/messages/streams

  • Several component models

– Common Component Architecture/CCA Forum (CCA) – CORBA Component Model/OMG (CCM) – Fractal/ObjectWeb – Grid.it-ASSIST/UNIPI – Etc.

Software component

PROVIDED PORTS REQUIRED PORTS

(client interfaces)

Software component

(server interfaces)

slide-5
SLIDE 5

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

5

Overview of the CCA (1/2)

  • CCA forum

– US laboratories and academic institutions

  • Specification based on the Scientific Interface

Definition Language

– Standard interfaces

  • Several implementations

– Local frameworks

  • Ccaffeine, SCIRun2, etc.

– Distributed frameworks

  • Xcat, SCIRun2, Legion-CCA, etc.
slide-6
SLIDE 6

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

6

Overview of the CCA (2/2)

  • All is done at runtime
  • Examples of specified SIDL

interfaces

– BuilderService

  • Component creation

createInstance(…)

  • Composition

connect(…)

– Services

  • Port declarations

addProvidesPort(…) registerUsesPort(…)

  • Getting a port reference

getPort(…)

fw

A

createInstance(A) setServices registerUsesPort(aP)

aP

connect(aP, bP)

B

createInstance(B) setServices addProvidesPort(bP)

bP

slide-7
SLIDE 7

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

7

Outline of the talk

  • Introduction

– Context – Overview of the Common Component Architecture

  • Data sharing paradigm

– Objectives – Our proposition : data port model – Data sharing on operation invocations

  • Support of the master-worker paradigm

– Limits with existing component models – A proposed model

  • Data sharing in a master-worker paradigm
  • Conclusions and perspectives
slide-8
SLIDE 8

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

8

Problem overview

  • Multiple concurrent accesses to a piece of data
  • Data localization and management

– Intra-process : local shared memory – Intra-cluster : distributed shared memory (DSM) – Intra-grid : grid data sharing service (JuxMem, PARIS/INRIA )

Data

B C A D

Read / write Read / write write Read

slide-9
SLIDE 9

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

9

Our proposal: data port model

component D { shares array<float> u; }; component A { accesses array<float> v; }; …

Transparent data localization and management – Local shared memory, DSM, JuxMem

Data

data sharing management

B D A

data_ref

accesses port shares port Principle: transparent data sharing

slide-10
SLIDE 10

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

10

Data ports in CCA

User View

interface ExtendedServices : Services { void createAccessPort (in string portName, in string typeDataName, in TypeMap properties); void createSharesPort (...); } interface AccessPort : Port {

  • paque get_pointer();

long get_size(); void acquire(); void acquireR(); void release(); } interface SharesPort : AccessPort { void associate (in opaque ptr, in long size); void disassociate(); }

Component implementation example

class CompImpl { Services srv; AccessPort myPort; void setServices (Services services ){ srv = services; srv.createAccessPort ("myPort", "arrayFloat",..); } void computeSum(){ myPort = srv.getPort(“myPort"); myPort.acquireR(); ptr = myPort.getPointer(); size = myPort.get_size(); for ( i = 0; i < size; i++) sum += ptr[i]; myPort.release (); } }

slide-11
SLIDE 11

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

11

Data sharing on operation invocation

  • Data sharing in SIDL

– Add “&” notation

  • Orthogonal to parameter modes

– In/out/inout determines who is the data “publisher” – Example

  • Mapping of arguments to data ports

– Example

data publisher data access (R/W) in& caller all/always

  • ut&

callee callee: always caller: after return caller callee in& shares port accesses port

  • ut&

accesses port shares port

interface compute { void foo (in matrix & m1, out matrix & m2); } callee caller

slide-12
SLIDE 12

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

12

Example of use

void foo (AccessPort& dpm1, SharesPort& dpm2) { ptr = dpm1.get_pointer(); size = dpm1.get_size(); res = f77_foo(ptr, size); dpm2->associate(res, size); } callee caller interface compute { void foo (in matrix & m1, out matrix & m2); } Matrix* ptr = allocate_matrix(…) SharesPort* dp1 = createSharesPort(“p1”, “matrix”) dp1->associate(ptr, size); AccessPort* dp2 = createAccessPort(“p2”, “matrix”); to_server-> foo(dp1, dp2);

slide-13
SLIDE 13

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

13

Outline of the talk

  • Introduction

– Context – Overview of the Common Component Architecture

  • Data sharing paradigm

– Objectives – Our proposition : data port model – Data sharing on operation invocations

  • Support of the master-worker paradigm

– Limits with existing component models – A proposed model

  • Data sharing in a master-worker paradigm
  • Conclusions and perspectives
slide-14
SLIDE 14

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

14

Problem overview

worker worker worker

Master

worker

Workers collection Requests transport Scheduling Fault tolerance

  • Simultaneous independent computations (~ForAll loop)
  • Dedicated API/environments

– BOINC, XTremWEB, DIET, NetSolve, Nimrod/G, ...

slide-15
SLIDE 15

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

15

Limits with component models

  • Different infrastructures

– Multi-core processors, SMP, clusters, grids, etc.

  • Resources dependant properties

– Number of workers – Request transport and scheduling policy W W

worker master master

request delivery policy

W W

worker

  • At the burden of the programmer

– Complex – No transparence

  • Objectives

– Transparency – Re-use existing MW environments

slide-16
SLIDE 16

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

16

Overview of the proposal

Programmer/designer view

Collection

binding master worker Exposed provided port

Resources infrastructure

Round-Robin

Framework implementation view #workers + Pattern selection

Set of request transport mechanism patterns

  • 1. Random
  • 2. Round-Robin
  • 3. NetSolve
  • 4. Diet
slide-17
SLIDE 17

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

17

Master-worker paradigm in CCA

interface CollBuilderService : BuilderService { CollectionID createInstanceCollection (in string instanceCollName, in string className, in TypeMap properties); BindingID bind (in CollectionID collID, in string collPortName, in ComponentID elem, in string elemPortName); } interface CollectionManagement: Port { bool addElement(in CollectionID collID); bool subElement(in CollectionID collID); } interface PatternInstantiation: Port {…}

master

createInstance(master)

fw

setServices registerUsesPort(mP)

createInstanceCollection(coll) setServices addProvidesPort(cP) addProvidesPort(m)

mP

connect(mP, cP)

1 2 3

w1

coll cP wP

w2 Wx m addElement(coll) createInstance(Wx) bind(cP, wP)

slide-18
SLIDE 18

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

18

Outline of the talk

  • Introduction

– Context – Overview of the Common Component Architecture

  • Data sharing paradigm

– Objectives – Our proposition : data port model – Data sharing on operation invocations

  • Support of the master-worker paradigm

– Limits with existing component models – A proposed model

  • Data sharing in a master-worker paradigm
  • Conclusions and perspectives
slide-19
SLIDE 19

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

19

Data sharing on operation invocation

interface compute { void foo (in matrix & m1,

  • ut matrix & m2);

}

  • Data sharing on operation invocations

– Operation declarations level

  • Master-worker paradigm

– Composition level

  • Data-sharing & master-worker paradigm

– Dynamic data ports creation at each operation invocation (worker side)

master m1

W W

worker1 M2_x M2_2 M2_1

slide-20
SLIDE 20

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies

20

Conclusions and perspectives

  • Data sharing and transparent data access model in CCA

– Data ports model: «shares» and «accesses»

  • Improving the support of master-worker paradigm in CCA

– Collections, request delivery policy patterns – Resources infrastructure independent programming – Adaptation of workers and requests delivery at the burden of the framework

  • Support of data sharing on operation invocation

– Data sharing in MW paradigm

  • Perspectives

– Implementation within a CCA framework

  • Data sharing and master-worker paradigm

– Benchmarks – Shared components in a distributed environment

slide-21
SLIDE 21

Managed by

Questions ?