Fault tolerance based on the Publish- subscribe Paradigm for the - - PowerPoint PPT Presentation

Fault tolerance based on the Publish- subscribe Paradigm for the BonjourGrid Middleware

Heithem ABBES, Christophe CERIN, Mohamed JEMNI and Walid SAAD

Grid 2010 - 27 October 2010

University of Paris XIII INSTITUT GALILEE Laboratoire d’Informatique de Paris Nord (LIPN) Université of Tunis École Supérieure des Sciences et Tehniques de Tunis Unité de Recherche UTIC

Outline

• Introduction
• Objectives
• Design of BonjourGrid
• Integration of Boinc and Condor
• Fault tolerance approach
• Experimentation and validation
• Conclusion and future works

2

Introduction (1/3)

• P2P systems have allowed large improvements in

the field of file sharing over Internet.

3

Introduction (1/3)

• P2P systems have allowed large improvements in

the field of file sharing over Internet.

3

• Gnutella, Kazaa and Freenet

Introduction (1/3)

➡ Decentralized architecture ➡ No coordination between machines

• P2P systems have allowed large improvements in

the field of file sharing over Internet.

3

• Gnutella, Kazaa and Freenet

Introduction (2/3)

• Grid computing : obtaining an infrastructure
• ffering computing power for users

applications.

• Coordination between machines during

application execution.

• Centralized or hierarchical architectures

(Globus, Glite, Condor).

4

Introduction (2/3)

• Grid computing : obtaining an infrastructure
• ffering computing power for users

applications.

• Coordination between machines during

application execution.

• Centralized or hierarchical architectures

(Globus, Glite, Condor). ➡ No scalability ➡ Complicated procedure of installation ➡ Complicated configuration phase for an ordinary user

4

Introduction (2/3)

• Grid computing : obtaining an infrastructure
• ffering computing power for users

applications.

• Coordination between machines during

application execution.

• Centralized or hierarchical architectures

(Globus, Glite, Condor). ➡ No scalability ➡ Complicated procedure of installation ➡ Complicated configuration phase for an ordinary user

4

Introduction (3/3)

• Desktop Grid led the community to build computing

systems based on voluntary machines.

• Current systems use Master/Worker model

5

Introduction (3/3)

• Desktop Grid led the community to build computing

systems based on voluntary machines.

• Current systems use Master/Worker model

5

• United Devices, BOINC, PLANETLAB, XtremWeb

Introduction (3/3)

• Desktop Grid led the community to build computing

systems based on voluntary machines.

• Current systems use Master/Worker model
• Application domains
• Global climate prediction (BOINC)
• Search for extraterrestrial intelligence (SETI@Home)
• Cosmic rays study (XtremWeb).

5

• United Devices, BOINC, PLANETLAB, XtremWeb

Introduction (3/3)

• Desktop Grid led the community to build computing

systems based on voluntary machines.

• Current systems use Master/Worker model
• Application domains
• Global climate prediction (BOINC)
• Search for extraterrestrial intelligence (SETI@Home)
• Cosmic rays study (XtremWeb).

5

• United Devices, BOINC, PLANETLAB, XtremWeb

✓ Demonstrate the potential of Desktop Grid

Introduction (3/3)

• Desktop Grid led the community to build computing

systems based on voluntary machines.

• Current systems use Master/Worker model
• Application domains
• Global climate prediction (BOINC)
• Search for extraterrestrial intelligence (SETI@Home)
• Cosmic rays study (XtremWeb).

✴ Suffer from being hardly scalable due to centralized control ✴ Rely on permanent administrative staff who guarantees the master operation

5

• United Devices, BOINC, PLANETLAB, XtremWeb

✓ Demonstrate the potential of Desktop Grid

Objectives of BonjourGrid

• Design a multi-platform coordinators and fault tolerant

system using existing desktop grid middleware

• Reduce the centralization factor: no static coordinator
• Benefit from the existing decentralized service discovery

tools (Publish / Subscribe)

• Create coordinators on demand, automatically and

without administrator intervention.

• Each coordinator selects machines to participate in the

execution of a given application.

6

Coordinateur

1 Computing Element (CE) = 1 coordinator + N workers

Design of BonjourGrid

7

Coordinateur

1 Computing Element (CE) = 1 coordinator + N workers

Design of BonjourGrid

7

Coordinateur

1 Computing Element (CE) = 1 coordinator + N workers

Design of BonjourGrid

7

Coordinateur

1 Computing Element (CE) = 1 coordinator + N workers 1 instance: 1 CE managed by a middleware

Design of BonjourGrid

7

Coordinateur

1 Computing Element (CE) = 1 coordinator + N workers Controls and orchestrate multiple instances 1 instance: 1 CE managed by a middleware

Design of BonjourGrid

7

Coordinateur

1 Computing Element (CE) = 1 coordinator + N workers Controls and orchestrate multiple instances 1 instance: 1 CE managed by a middleware

Design of BonjourGrid

Introduction of the concept of meta-grids

7

A A

8

Design of BonjourGrid

A A A

8

Design of BonjourGrid

A B A A

8

Design of BonjourGrid

A B A A C

8

Design of BonjourGrid

A B A A D C

8

Design of BonjourGrid

A B A A D C

8

Design of BonjourGrid

A B A A D C

8

Design of BonjourGrid

A B A A D C

8

Design of BonjourGrid

A B A A D C

8

Design of BonjourGrid

A B A A D C

8

Design of BonjourGrid

A B A A D C

8

Design of BonjourGrid

A B A A D C

8

Design of BonjourGrid

A B A A D C

8

Design of BonjourGrid

A B A A D C

8

Design of BonjourGrid

A B A A D C

8

Design of BonjourGrid

A B A A D C A computing element for each user

8

Design of BonjourGrid

A B A A D C A computing element for each user No static coordinator

8

Design of BonjourGrid

A B A A D C A computing element for each user No static coordinator Each user can specify a middleware for his computing element

8

Design of BonjourGrid

• BonjourGrid is based on :
• A resource discovery protocol
• Fully decentralized
• A computing element
• Executes and handles the various tasks of an application

(Condor, Boinc, XtremWeb)

• A global coordination protocol
• Manages and controls all resources, services and computing

elements

• Does not depend on any specific machine or centralized element

Components of BonjourGrid

9

Discovery protocol

• Based on Bonjour protocol
• Multicast IP network
• An implementation by Apple of ZeroConf protocol.
• Structured around three functionalities :
• Dynamic allocation of IP addresses without DHCP
• Resolution of names and IP addresses without DNS
• Services discovery without directory server
• Motivations
• Industrial protocol approved by Apple
• Different versions for the 3 OS (Windows, Linux, MaxOS)
• Linux and MacOS distributions integrate Bonjour
• Evolution of networks (10 Gb/s 10 * x Gb/s) => low risk of

network congestion for multicast protocols

10

Computing element (CE)

• Each coordinator creates dynamically its CE
• CE = Coordinator + set of workers
• CE functionalities
• Allocates workers
• Submits and run tasks on workers
• Schedules and get results
• Computing systems
• XtremWeb, Condor or Boinc

11

Computing element (CE)

• Each coordinator creates dynamically its CE
• CE = Coordinator + set of workers
• CE functionalities
• Allocates workers
• Submits and run tasks on workers
• Schedules and get results
• Computing systems
• XtremWeb, Condor or Boinc

11

1 specific CE for each user

Coordination protocol

• Each machine can have one of the three states (Idle, Worker or

Coordinator).

• A machine announces its state by publishing the specific service to

this state :

• IdleService for idle state
• WorkerService for worker state
• CoordinatorService for coordinator state
• When machine state changes:
• it publishes the appropriate service to advertise this new state,
• after having deactivated the old one.
• Every machine can discover machines that are in a given state:
• A machine launches a discovery on a particular service instead of

permanently receiving all new events.

• Restrict communication between machines.

12

Layered architecture

13

Layered architecture

Publish/Subscribe

13

Layered architecture

Publish/Subscribe Connection to BonjourGrid

13

Layered architecture

Publish/Subscribe Connection to BonjourGrid Resources discovery

13

Layered architecture

Publish/Subscribe Connection to BonjourGrid Resources discovery Resources characteristics

13

Layered architecture

Publish/Subscribe Connection to BonjourGrid Resources discovery Resources characteristics Establishment of CE network

13

Layered architecture

Publish/Subscribe Connection to BonjourGrid Resources discovery Resources characteristics Establishment of CE network XtremWeb

13

Layered architecture

Publish/Subscribe Connection to BonjourGrid Resources discovery Resources characteristics Establishment of CE network XtremWeb Condor

13

Layered architecture

Publish/Subscribe Connection to BonjourGrid Resources discovery Resources characteristics Establishment of CE network XtremWeb Condor Boinc

13

Layered architecture

Publish/Subscribe Connection to BonjourGrid Resources discovery Resources characteristics Establishment of CE network XtremWeb Condor Boinc Deployment

• f a computing system

13

Integration of Boinc in BonjourGrid

!"#$%&'()"%*'#"+ !"#$%&'()#*+,- !"#$"#%&'()*++, !"#$%&'()''*&+, !"#$%&'(%)&*""+ !"#$"%&#'"($) !"#$%&'(") !"#$%#&'%()%*+, !"#$"%&'$()$* !"#$%&'(")

Account,mail,Certificate… URL,ProjectName,NbreWorker… IP,Hostname,CPU,Memory… ServiceType,HostName.. URL,ProjectName… IP,Hostname,CPU,Memory…

!"#$%&

!"#$"%&'&()*

Integration of Condor in BonjourGrid

!"#"$%&'(#)*"++, !"#"$%&'()#*$+ !"#$%#&'()*+,,- !""#$%&&'()* !"#$"%&#'"($) !"#$%&'(") !"#$"%&'$()$* !"#$%&'(") !"#$%#&'%()%*+,

Host/IP Access levels security, Mail and Networks parameters ServiceType,HostName.. IP,Hostname,CPU,Memory… IP,PoolName,ManagerName, CollectorName,DomainName,NbreWorker… IP,PoolName,ManagerName, CollectorName,DomainName… IP,Hostname,CPU,Memory…

!"#$"%&

!"#$"%&'&()*

Fault tolerance approach

• Each computing system is responsible for :
• controlling and monitoring application tasks

execution

• the fault-tolerance of its workers within a

computing element

➡ The failure of workers is not the responsibility of BonjourGrid

• The failure of coordinators is in the charge of

BonjourGrid

16

Fault tolerance approach

• Solution
• Create dynamically backup coordinators for each

application,

• Provide k backup (k is a configuration setting that

must be fixed before construction of the CE) for each application, using a passive replication

Fault tolerance approach

Idle Coordinator Worker

18

Fault tolerance approach

Idle Coordinator Worker

18

• Construction of a computing element and 2 backups of the coordinator

Fault tolerance approach

Idle Coordinator Worker

18

• Construction of a computing element and 2 backups of the coordinator

Fault tolerance approach

Idle Coordinator Worker

18

• Construction of a computing element and 2 backups of the coordinator

Fault tolerance approach

Idle Coordinator Worker

18

• Construction of a computing element and 2 backups of the coordinator

Fault tolerance approach

• The workers go back to idle state when the coordinator is disabled

Idle Coordinator Worker

18

• Construction of a computing element and 2 backups of the coordinator

Fault tolerance approach

• The workers go back to idle state when the coordinator is disabled

Idle Coordinator Worker

18

Problem: The coordinator has not yet completed the application

• Construction of a computing element and 2 backups of the coordinator

Fault tolerance approach

• Solution : Status field to distinguish between :
• Stop due to failure

➡ Status = 0 (application is in execution)

• Stop following the end of application

➡ Status = 1 (application finished)

19

Fault tolerance approach

• Solution : Status field to distinguish between :
• Stop due to failure

➡ Status = 0 (application is in execution)

• Stop following the end of application

➡ Status = 1 (application finished)

Idle Coordinator Worker Status

19

Fault tolerance approach

• Solution : Status field to distinguish between :
• Stop due to failure

➡ Status = 0 (application is in execution)

• Stop following the end of application

➡ Status = 1 (application finished)

Idle Coordinator Worker Status

19

Fault tolerance approach

• Solution : Status field to distinguish between :
• Stop due to failure

➡ Status = 0 (application is in execution)

• Stop following the end of application

➡ Status = 1 (application finished)

Idle Coordinator Worker Status

19

Fault tolerance approach

• Solution : Status field to distinguish between :
• Stop due to failure

➡ Status = 0 (application is in execution)

• Stop following the end of application

➡ Status = 1 (application finished)

Idle Coordinator Worker Status

19

Experimentations

• System evaluation based on
• a set of specific applications?
• a specific arrival pattern (Poisson’s Law) ?
• Workload model very close to the reality
• Feitelson and Lublin
• Inputs of the workload model
• Number of nodes (system size)
• Arrival time of applications
• Maximum number of parallel tasks
• Tasks execution times

20

Application ID Arrived Time (s) Execution Time (s) Nbre of parallel tasks 1 19 4 32 2 39 11 13 3 69 13 16 4 98 87 1 5 299 100 4

Experimentations

• Emulation of a set of users and a set of applications
• 1 CE is dynamically created for each application
• Emulator
• Parameters :
• list of machines
• list of applications
• workload model
• Submit an application following the arrival pattern of

applications in the workload

• Look for free machine on which a coordinator will start to initiate

the application tasks execution

• The CE is released when application tasks finish

21

• Calculate: (end time of an application - submission time)
• Analyze the delay caused by the decentralization
• Analyze the behavior of BonjourGrid with :
• Boinc
• Condor
• Setup
• BonjourGrid : N machines (dynamic infrastructure)
• Boinc or Condor : 1 coordinator + N-1 workers (static

infrastructure)

22

Experimentations

Experimentations - Boinc

1 4 16 64 256 1024 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 105 110 115 120 125 130 10 20 30 40 50 60 70 80 90 100 110 120 130 Time (s) in logscale(2) Nbre Of // Tasks #Applications Turnaround time of BOINC Turnaround time of BonjourGrid Nbre of Tasks per App.

23

Setup :

• 130 applications (2 to 128 // tasks)
• 200 machines on Grid5000 (Orsay’s node)
• Execution times vary from 1 to 500 seconds

Results :

• With BonjourGrid, 60% of applications give a delay varying from 24 to 1277 s
• BonjourGrid gives execution times < Boinc when the tasks number is important

Experimentations - Condor

1 4 16 64 256 1024 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 105 110 115 120 125 130 10 20 30 40 50 60 70 80 90 100 110 120 130 Time (s) in logscale(2) Nbre Of // Tasks #Applications Turnaround time of Condor Turnaround time of BonjourGrid Nbre of Tasks per App.

24

Setup :

• 130 applications (2 to 128 // tasks)
• 200 machines on Grid5000 (Orsay’s node)
• Execution times vary from 1 to 500 seconds

Results :

• With BonjourGrid, 35% of applications generate a delay around 30 s
• BonjourGrid generates more important delays for applications which are preceded by

applications with a large number (>100) of tasks

Experimentations - Fault tolerance

• Using virtual machines to save the state of coordinators
• XEN virtualization system
• 10 applications, with parallel tasks ranging from 2 to 128 tasks
• 50 machines on the node of Nancy (Grid5000)
• Faults scenarios by injecting faults during the execution of

applications

• Recovery Time
• Time of recovery of the coordinator
• Time to re-establish the connection workers

25

!"#$%&'()*# +)%,-.(/0 12&%/(2# !""#$%&'("#) *"%&+,!"&$"#) !"#$"%&'()*+&' !"#$%&'()*+,'

• .)&&%&/'012'

!"#$%&&$ !"#$%&'(")*' !""#$%&'("#) *"%&+,!"&$"#) !"#$"%&'()*+&' !"#$%&&$ !"#$%&&$ !"#$%&'(")*' !"#$%&'()*# !%+,-.$&-&/-(0# !"#$%&'()*# +)%,-.(/0 1&-/(2&# !"#$"%&'()*+&' !"#$%&'()*+,'

• .,$/,*0'123'

!"#$%&'( )"&*"+,-,%$( .,#/01",2( !"#$%&'()*+,'-.)&&%&/'012'

Main coordinator Backup coordinator Worker

Save Snapshot (1) Migrate Snapshot (2) Restore Snapshot (3) Establishment of connection(4)

Fault tolerance framework

BOINC

• Average delay of 197 sec
• Almost stable delay which does not depend on number of tasks
• Boinc allows the continuation of work after coordinator failure

27

1 4 16 64 256 1024 1 2 3 4 5 6 7 8 9 10 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 200 Time (s) in logscale(2) Nbre Of // Tasks #Applications Turnaround time of BOINC Turnaround time of BOINC-FT Nbre of Tasks per App. Level of Fault Injection .

Experimentations - Fault tolerance

28

1 4 16 64 256 1024 1 2 3 4 5 6 7 8 9 10 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 200 Time (s) in logscale(2) Nbre Of // Tasks #Applications Turnaround time of CONDOR Turnaround time of CONDOR-FT Nbre of Tasks per App. Level of Fault Injection .

Condor

• Average delay of 238 sec
• Condor recovers tasks that have not completed their executions

Experimentations - Fault tolerance

Conclusion

• BonjourGrid: A novel approach for making a

collaborative and decentralized Desktop Grid systems.

• Publish/Subscribe protocol
• Orchestrate the participants
• A computing system (Boinc, Condor or XtremWeb) for the

execution level of an application.

• BonjourGrid makes a distributed control over resources

and does not depend on a central element.

• BonjourGrid implements a Fault-tolerant mechanism

for coordinators

• BonjourGrid favors collaborative execution and Meta-

Grids orchestration

29

Future works

• Minimize the amount of information transferred

between coordinators

• Include reservation rules based on history traces
• f the previous executions
• Integrate economic models
• Add a new layer for security issue

30

Our background is coordination, our future will be coordination of clouds!

Thanks. Any Questions ?