Experiences using Grid Computing Technologies to Solve Optimization - - PDF document

1

Departamento de Lenguajes y Ciencias de la Computación

Antonio J. Nebro

PhD in Computer Science Associate Professor Universidad de Málaga SPAIN

Experiences using Grid Computing Technologies to Solve Optimization Problems

The 2007 High Performance Computing & Simulation (HPCS'07) Conference

Prague, Czech Republic, June 4-6, 2007

PANEL Parallel Processing Systems: Present and Future Trends

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

2 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

• In our research group we are interested in applying

parallelism to solve optimization problems using metaheuristics.

• In this line of work, grid computing appears as a

discipline providing a potential huge computing power that can be used to solve very hard optimization problems.

• In this talk, I will comment our experiences using these

systems and I will discuss about what we expect concerning grid computing and optimization in the future.

Summary

2

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

3 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

• Introduction
• Case study 1: Condor and enumerative search
• Case study 2: Condor, MW, and DNA sequencing
• Case study 3: ProActive and optimizing the VRP
• Conclusions and future plans

Table of Contents

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

4 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

• MSc in Computer Science (1991)

– Masther thesis:

• “Parallelization of a Branch and Bound Algorithm for Solving the

TSP”

• Parallel computer: Transputers (16 nodes machine)
• Parallel programming: Occam
• PhD in Computer Science (1999)

– PhD thesis:

• “A Distributed Runtime System for Implementing Concurrent

Objects”

• Parallel computers: LAN of workstations, cluster of four Digital

AlphaServer (4 CPU multiprocessors)

• Parallel programming: sockets, PVM, POSIX threads (Pthreads)

Experiences in parallel computing

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

3

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

5 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

• There many computers in the labs of the Computer

Science Department of the University of Málaga

– Currently about 300 processors

• Question:

– ¿How can we use them together to solve a problem?

• Using known passing message libraries (sockets, PVM,

MPI) is not a solution

– Easy to use tool: heterogeneity, fault tolerance, dynamicity – Machines are idle in the nights and in the weekends – Variable availability during the day

• SOLUTION: using grid technologies

Motivation

Introduction

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

6 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

• Time table

Motivation

Introduction

4

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

7 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

• Key paper
• Distributed branch and bound algorithm
• 2500 processors
• A hard instance of the QAP (Quadratic Assignment Problem) was
• solved. ESTIMATED TIME: seven years
• Grid computing software: Condor

Motivation

Introduction

• K. Anstreicher, N. Brixius, J.-P. Goux, and J. Linderoth

Solving Large Quadratic Assignment Problems on Computational Grids. Mathematical Programming, 2002, Vol. 91, pp. 563–588.

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

8 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

• Goals

– To apply grid technologies to an enumerative search algorithm for multiobjective optimization – Heterogeneous distributed system – Grid system: Condor

Case study 1: Condor and enumerative search

A.J. Nebro, E. Alba, F. Luna. Multi-objective Optimization Using Grid Computing. Soft Computing, Vol. 11, No. 6. (April 2007), pp. 531-540.

Case study 1: Condor

5

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

9 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

• Condor (http://www.cs.wisc.edu/condor/)

– Project of the University of Wisconsin – It is a distributed scheduler

• Some features

– Checkpointing – Remote system calls – Several universes (working modes): standard, vanilla, PVM, MPI, Java – Interoperable with Globus – Suitable to master/slave applications through the MW library – Opportunistic computing

Case study 1: Condor and enumerative search

Case study 1: Condor

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

10 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

• Our requirements

– Our application is composed of hundred/thousands of independent tasks – All the results (output files) are gathered offline – Heterogeneous machines

Case study 1: Condor and enumerative search

110 Processors

Case study 1: Condor

6

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

11 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

• Obtained results

– Two hard problems were solved – Problem 1

• Time needed: 2 days
• Total time reported by Condor: 90 days

– Problem 2

• Timed needed: 27 days
• Total time reported by Condor: 775 days
• Analysis:

– There is not dynamic load balancing – Condor is unable to migrate works between heterogeneous machines

Case study 1: Condor and enumerative search

Case study 1: Condor

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

12 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

• Our experience

– Condor is easy to install and administrate – Very easy to use if the tasks are independent

• The programs do not need to be modified nor recompiled
• Only linking with the Condor libraries is needed

– Very stable

• Few problems in this sense

– Well documented – Opportunistic computing works – The lack of dynamic load balancing can be a problem

Case study 1: Condor and enumerative search

Case study 1: Condor

7

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

13 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

• Goals

– To design a distributed genetic algorithm to solve an

• ptimization problem from the biotechnology field

– Grid system: Condor and the MW (Master/Worker) library

• MW: C++ library allowing to develop master/slave applications on

top of Condor

• Not difficult to use if your are familiar with C++

Case study 2: Condor, MW and Genetic Algorithms

A.J. Nebro, G. Luque, F. Luna, E. Alba. DNA Fragment Assembly Using a Grid Based Genetic Algorithm. Computers and Operations Research. In Press. 2007

Case study 2: Condor and MW

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

14 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

• Our requirements

– Designing a distributed genetic algorithm (GA) to optimize a biotechnology problem (DNA sequencing) – Ability to use up to 150 processors – Analysis of the master-worker scheme when applied to implement a grid-enabled distributed GA

Case study 2: Condor, MW and Genetic Algorithms

Case study 2: Condor and MW

8

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

15 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

• Obtained results

Case study 2: Condor, MW and Genetic Algorithms

Case study 2: Condor and MW

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

16 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

• Our experience

– A local search method had to be applied to the slave computation to obtain a favorable computation/communication ratio – MW: easy to use, difficulties to send complex data structures

Case study 2: Condor, MW and Genetic Algorithms

Without Local Search WITH Local Search

Case study 2: Condor and MW

9

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

17 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

• Goals

– To solve very large instances of the VRP (Vehicle Routing Problem)

• Benchmark: VLSVRP

– Grid System: ProActive

Case study 3: ProActive and Genetic Algorithms

• B. Dorronsoro, D. Arias, F. Luna, A.J. Nebro, E. Alba

A Grid-Based Hybrid Cellular Genetic Algorithm for Very Large Scale Instances of the CVRP. HPCS 2007. Special Session on Parallel and Grid Computing for Optimization

Case study 3: ProActive

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

18 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

• ProActive (http://www-sop.inria.fr/oasis/ProActive/)

– Java Grid middleware library – It is aimed at parallel, distributed and multi-threaded computing – Main principles:

• active objects
• asynchronous communication
• future synchronization

– High level programming model – Communication through Java RMI

Case study 3: ProActive and Genetic Algorithms

Case study 3: ProActive

10

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

19 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

• Our requirements

– Designing a grid- enabled genetic algorithm to solve large instances of the VRP – Non-trivial model:

• Ring of processes
• Master/slave in the

processes

– Local search in the slaves – Up to 125 machines

Case study 3: ProActive and Genetic Algorithms

Case study 3: ProActive

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

20 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

Case study 3: ProActive and Genetic Algorithms

• Obtained results

– Most of the solutions of the VLSVRP instances are estimated

BNS: Best knowN Solution

–We obtain the best solution in 7 out of the 12 problems –Found a new best solution for VLS26

Case study 3: ProActive

11

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

21 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

Case study 3: ProActive and Genetic Algorithms

• Our experience

– ProActive is not difficult to use if you are familiar with Java

• No problem with heterogeneous architectures

– The same advantages of using JAVA RMI

• Sending complex data structures is very simple

– The same drawbacks

• Slow communication

– No opportunistic computing

Case study 3: ProActive

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

22 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

Conclusions

• As conclusions

– Grid computing technologies are here – Useful and easy-to-use in local administrative domains

• In our experience (in a public university)

– Using computers in the labs is not a major problem – But using the desktop computers is not so easy

• Our department has more than 100 hundred people
• They are reluctant to give their computers, even when you tell

them about opportunistic computing

Conclusions and future plans

12

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

23 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

Future plans

• Our current work

– We have now more than 300 processors in our labs – Use ProActive to explore more complex parallel models – We are considering now BOINC (http://boinc.berkeley.edu)

• SETI@HOME
• Master/worker again
• But it works by using the screensavers of the PCs
• More potential number of users giving the idle computing time of

their desktop computers

• It simplifies distributed computing across different administrative

domains

Conclusions and future plans

Experiences using Grid Computing Technologies to Solve Optimization Problems Antonio J. Nebro

24 - 24

Panel:

Parallel Processing Systems: Present and Future Trends

Conclusions and future plans Case study 3: ProActive Case study 2: Condor and MW Case study 1: Condor Introduction

Thanks for your attention!!

Conclusions and future plans

• QUESTIONS
• COMMENTS