!"#$% Pascal Felber, Raluca Halalai, Lorenzo Leonini, Etienne - - PowerPoint PPT Presentation

▶
pascal felber raluca halalai lorenzo leonini etienne rivi
SMART_READER_LITE
LIVE PREVIEW

!"#$% Pascal Felber, Raluca Halalai, Lorenzo Leonini, Etienne - - PowerPoint PPT Presentation

Oct 28, 2022 493 likes •829 views

On The Integration Of !"#$% Pascal Felber, Raluca Halalai, Lorenzo Leonini, Etienne Rivire, Valerio Schiavoni, Jos Valerio Universit de Neuchtel, Switzerland SimGrid User Days 2012 - Ecully, France !"#$% 2 About Me

slide-1
SLIDE 1

!"#$%

On The Integration Of

Pascal Felber, Raluca Halalai, Lorenzo Leonini, Etienne Rivière, Valerio Schiavoni, José Valerio Université de Neuchâtel, Switzerland SimGrid User Days 2012 - Ecully, France

!"#$%

slide-2
SLIDE 2

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

About Me

  • 2010-now : PhD student at the University of

Neuchâtel, Switzerland.

  • Dependable and Distributed Computing Group.
  • Topic: large-scale distributed systems, cloud

computing.

  • 2007-2009: Research Engineer at INRIA Grenoble,

SARDES team.

2

slide-3
SLIDE 3

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

Outline

  • SPLAY overview
  • Integrating SPLAY and SimGrid
  • Based on rough ideas
  • Suggestions are welcome, collaborations

even more!

3

slide-4
SLIDE 4

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

Motivations

  • Developing, testing and tuning distributed

applications is hard

  • In Computer Science research, fixing the

gap of simplicity between pseudocode description and implementation is hard

  • Using worldwide testbeds is hard

4

slide-5
SLIDE 5

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

What is

5

slide-6
SLIDE 6

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

What is

5

slide-7
SLIDE 7

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

What is

5

slide-8
SLIDE 8

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

What is

  • Machines contributed by universities,

companies,etc.

  • ~1000 nodes at 531 sites (May 2012)
  • Shared resources, no privileged access
  • University-quality Internet links
  • High resource contention
  • Faults, churn, packet-loss is the norm
  • Challenging conditions

5

slide-9
SLIDE 9

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

  • A testbed is a set of

machines for testing your distributed applications/ protocols

  • Several different testbeds!

6

networks of idle workstations your local machine

  • ur new cluster@UniNE
slide-10
SLIDE 10

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

Daily Job With Distributed Systems

7

slide-11
SLIDE 11

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

1

Daily Job With Distributed Systems

code 1 • Write (testbed specific) code

  • Local tests, in-house cluster, PlanetLab...

7

slide-12
SLIDE 12

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

2 1

Daily Job With Distributed Systems

code debug 2 • Debug (in this context, a nightmare)

7

slide-13
SLIDE 13

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

2 1

Daily Job With Distributed Systems

code debug 2 • Debug (in this context, a nightmare)

7

slide-14
SLIDE 14

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

3 2 1

Daily Job With Distributed Systems

code debug deploy 3 • Deploy, with testbed specific scripts

7

slide-15
SLIDE 15

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

3 2 1

Daily Job With Distributed Systems

code debug deploy 3 • Deploy, with testbed specific scripts

7

slide-16
SLIDE 16

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

4 3 2 1

Daily Job With Distributed Systems

code debug deploy get logs 4 • Get logs, with testbed specific scripts

7

slide-17
SLIDE 17

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

4 3 2 1

Daily Job With Distributed Systems

code debug deploy get logs 4 • Get logs, with testbed specific scripts

7

slide-18
SLIDE 18

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

5 4 3 2 1

Daily Job With Distributed Systems

code debug deploy get logs plots 5 • Produce plots, hopefully

7

slide-19
SLIDE 19

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

5 4 3 2 1

Daily Job With Distributed Systems

code debug deploy get logs plots 5 • Produce plots, hopefully

7

slide-20
SLIDE 20

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

at a Glance

  • Supports the development, evaluation, testing, and

tuning of distributed applications on any testbed:

  • In-house cluster, shared testbeds, emulated

environments

  • Provides an easy-to-use pseudocode-like language

based on Lua

  • Write Once, Deploy&Run Everywhere
  • Open-source: http://www.splay-project.org

!"#$%

code debug deploy get logs plots

8

slide-21
SLIDE 21

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

at a Glance

  • Supports the development, evaluation, testing, and

tuning of distributed applications on any testbed:

  • In-house cluster, shared testbeds, emulated

environments

  • Provides an easy-to-use pseudocode-like language

based on Lua

  • Write Once, Deploy&Run Everywhere
  • Open-source: http://www.splay-project.org

!"#$%

code debug deploy get logs plots

8

slide-22
SLIDE 22

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

at a Glance

  • Supports the development, evaluation, testing, and

tuning of distributed applications on any testbed:

  • In-house cluster, shared testbeds, emulated

environments

  • Provides an easy-to-use pseudocode-like language

based on Lua

  • Write Once, Deploy&Run Everywhere
  • Open-source: http://www.splay-project.org

!"#$%

code debug deploy get logs plots

g n u p l

  • t

i s y

  • u

r f r i e n d

8

slide-23
SLIDE 23

http://www.splay-project.org

Language

  • Based on
  • High-level scripting language
  • Made for interaction with C
  • Bytecode-based, garbage collection
  • Must be close to pseudo code (focus on the algorithm)
  • & favor “natural ways” of expressing algorithms (e.g. RPCs)
  • SPLAY applications can be run locally without the

deployment infrastructure (for debugging & testing)

  • Comprehensive set of

libraries (can be extended)

9

event-based model (discussed below).

events/threads crypto* io (fs)* sb_fs misc sb_stdlib stdlib* log rpc json* llenc socketevents sb_socket luasocket*

splay::app

* : main dependencies : third!party and lua libraries

Figure 6:

Lua

!"#$%

slide-24
SLIDE 24

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

Why ?

  • Light & Fast
  • (Very) Close to equivalent code in C
  • Concise
  • Allow developers to focus on ideas more

than implementation details

  • Key for researchers and students
  • Sandbox thanks to the possibility of easily

redefine (even built-in) functions

10

slide-25
SLIDE 25

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

Concise

11

slide-26
SLIDE 26

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

Concise

Pseudo code as published

  • n original

paper Executable code using SPLAY libraries

11

slide-27
SLIDE 27

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

Network Emulation

  • Similar in the spirit to existing network

emulators (ModelNet, Emulab)

  • Novel features:
  • multi-user
  • multi-topology
  • user-space

12

!"#$%

1 3.3 5 10 10 20 30 40 50 60 upload rate (Mbps) data transfer time Stream from 1 to 3 Stream from 2 to 4

1 2 3 4 10 Mbps 10ms

slide-28
SLIDE 28

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

Network Emulation

  • XML/TCS formats to define topologies
  • Support for delay, lossy channels
  • User-land bandwidth shaping
  • Emulation of congestion at inner nodes in

a completely decentralized fashion

  • Dynamic adjustments of token-bucket

bandwidth shapers

13

!"#$%

slide-29
SLIDE 29

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

  • Integrating the two systems to get the best
  • f both world
  • Two options:
  • 1. Use Splay apps within SimGrid
  • 2. Use SimGrid apps within Splay

14

!"#$%

slide-30
SLIDE 30

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

Splay within SimGrid

  • Evaluate Splay applications with SimGrid
  • Different execution models, network

emulation

  • Splay adapter for SimGrid Lua Bindings
  • Customized sandbox
  • Customized Splay RPC libraries

15

slide-31
SLIDE 31

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

SimGrid within Splay

  • Evaluate SimGrid applications within Splay

testbed

  • Easier if applications are written in Lua
  • Alternatively, exploit Splay binary shipping
  • On-the-fly per-job relaxation of Splay

sandbox to allow binary code

  • What about sandboxing C applications?

16

slide-32
SLIDE 32

!"#$%

On The Integration of Splay & SimGrid - Valerio Schiavoni - University of Neuchâtel

  • Distributed systems raise a number of

issues for their evaluation

  • Their implementation, debug, deployment

and tuning is hard

  • leverages Lua and centralized

controller to produce an easy to use yet powerful working environment T a k e

  • a

w a y S l i d e

!"#$%

17