Experiment Driven Research Emmanuel Jeannot INRIA LaBRI 2011 - - PowerPoint PPT Presentation

Experiment Driven Research

Emmanuel Jeannot INRIA – LaBRI 2011 ComplexHPC Spring School Amsterdam, May 2011

The ComplexHPC Action

What is ComplexHPC?

Experiment Driven Research Emmanuel Jeannot 1/33

High Performance Computing on Complex Environments

• E. Jeannot

2/20

Modern infrastructures are more and more complex

Millions of cores:

• More and more
• General purpose or specialized

Core Processor Node Cluster/Heter. env. Large-scale env.

Thousands of processors:

• CPU/GPU
• Intrinsically parallel (ILP, TLP)

Thousands of nodes:

• SMP
• RAM (NUMA)

Tens of clusters:

• Homogeneity
• Power consumption

Heterogeneous env.:

• Use of available resources
• Processor and network heter.

Large scale env.:

• WAN (latency)
• High computing/storage capacity

High Performance Computing on Complex Environments

• E. Jeannot

3/20

Complexity

Characteristics of modern infrastructures

Modern infrastructures are already:

• Hierarchical
• Heterogeneous (data transfer & computation)
• Of different scales

Near future:

• Large scale infrastructures
• Dozens of sites
• Several heterogeneous computers/clusters per sites
• Thousands of processors per parallel computer
• Tens of cores on each processor

Lot of power: do we need it?

1 Large-scale env. 20 Sites 100 Clusters 105 Processors 108 Cores

High Performance Computing on Complex Environments

• E. Jeannot

4/20

Applications

High-performance applications:

• Larger and larger data sets
• Higher and higher computational

requirements

• Relevant

applications:

! Environmental simulations ! Molecular dynamics ! Satellite imaging ! Medicine (modeling and simulation)

High Performance Computing on Complex Environments

• E. Jeannot

5/20

Goal of the ComplexHPC Action

Rationale:

• Enormous computational demand
• Architectural advances: potential to meet requirements
• No integrated solution to master the complexity
• Research is fragmented

Goals:

• Overcome research fragmentation: foster HPC efforts to increase Europe

competitiveness

• Tackle the problem at every level (from cores to large-scale env.)
• Vertical integration: provide new integrated solutions for large-scale computing

for future platforms

• Train new generations of scientists in high-performance and heterogeneous

computing

High Performance Computing on Complex Environments

• E. Jeannot

6/20

Activities within the Action

Summer/spring schools

• Forum for students and

young researchers

• Second and four years

Working groups

• 4 working groups
• Meet twice a year
• Identify synergies
• Exchange ideas

Action meetings

• Transversal to working

groups

• Once a year

MC meetings

• Organization of the network
• Action steering
• Inclusion of new members

Visits

• Implement synergies
• 60 short-term visits (priority

to young researchers)

• 12 long-term visits

(fellowship)

International workshops

• HCW (in conjunction with

IPDPS)

• Heteropar (in conjunction

with Europar)

High Performance Computing on Complex Environments

• E. Jeannot

7/20

Scientific programme

Four working groups:

• Numerical analysis
• Efficient use of complex systems (comp. or comm. library)
• Algorithms and tools for mapping applications
• Applications

Target vertical aspects of the architectural structure Each WG managed by a its own leader (specialist, nominated by MC) Each participant of the Action will join at least one WG

Numerical analysis Libraries

• Alg. & tools for mapping

Applications

STSM

! Short Term Scientific Mission

• An exchange program is open for visits and exchanges before end of june.
• Do not hesitate to apply

! Emmanuel.jeannot@inria.fr, ! Krzysztof Kurowski: krzysztof.kurowski@man.poznan.pl

Experiment Driven Research Emmanuel Jeannot 8/33

My solution is better than yours

My Solution/Your Solution:

• Workload A: 0.25
• Workload B: 1
• Workload C: 4

Average: 5.25/3= 1.75 On average, My Solution is 1.75 more efficient than Your Solution Efficiency Workload A Workload B Workload C My Solution 10 20 40 Your Solution 40 20 10

Emmanuel Jeannot 9/33 Experiment Driven Research

Environment Stack

Problem of experiments

• Testing and validating solutions and models as a scientific

problematic

• Questions:

! what is a good experiments? ! which methodologies and tools to perform experiments? ! advantages and drawbacks of these methodologies/tools?

Infrastructure Services-protocols Middleware Applications

Experimental validation

Research issues at each layer of the stack

• algorithms
• software
• data
• models
• …
• Example:

Emmanuel Jeannot 10/33 Experiment Driven Research

Outline

! Importance, role and properties of experiments in computer science ! Different experimental methodologies ! Statistical analysis of results

Emmanuel Jeannot 11/33 Experiment Driven Research

The discipline of computing: a science

“The discipline of computing is the systematic study of algorithmic processes that describe and transform information: their theory, analysis design, efficiency, implementation and application” Peter J. Denning et al. COMPUTING AS A DISCIPLINE

! Confusion: computer science is not only science it is also engineering, technology, etc (!biology or physic) ! Problem of vulgarization ! Two ways for classifying knowledge:

• Analytic: using mathematics to tract models
• Experimental: gathering facts through observation

Emmanuel Jeannot 12/33 Experiment Driven Research

The discipline of computing: an experimental science

The reality of computer science:

• information
• computers, network, algorithms, programs, etc.

Studied objects (hardware, programs, data, protocols, algorithms, network): more and more complex. Modern infrastructures:

• Processors have very nice features

! Cache ! Hyperthreading ! Multi-core

• Operating system impacts the performance

(process scheduling, socket implementation, etc.)

• The runtime environment plays a role

(MPICH!OPENMPI)

• Middleware have an impact (Globus!GridSolve)
• Various parallel architectures that can be:

! Heterogeneous ! Hierarchical ! Distributed ! Dynamic Emmanuel Jeannot 13/33 Experiment Driven Research

Analytic modeling

Purely analytical (math) models:

• Demonstration of properties (theorem)
• Models need to be tractable: over-

simplification?

• Good to understand the basic of the problem
• Most of the time ones still perform a

experiments (at least for comparison)

For a practical impact (especially in distributed computing): analytic study not always possible or not sufficient

Emmanuel Jeannot 14/33 Experiment Driven Research

Experimental culture: great successes

Experimental computer science at its best [Denning1980]:

• Queue models (Jackson, Gordon, Newel,

‘50s and 60’s). Stochastic models validated experimentally

• Paging algorithms (Belady, end of the

60’s). Experiments to show that LRU is better than FIFO

Emmanuel Jeannot 15/33 Experiment Driven Research

Experimental culture not comparable with other science

Different studies:

• In the 90ʼs: between 40% and 50% of CS ACM papers requiring experimental

validation had none (15% in optical engineering) [Lukovicz et al.]

• “Too many articles have no experimental

validation” [Zelkowitz and Wallace 98]: 612 articles published by IEEE.

• Quantitatively more experiments

with times

Computer science not at the same level than some other sciences:

• Nobody redo experiments (no funding).
• Lack of tool and methodologies.

M.V. Zelkowitz and D.R. Wallace. Experimental models for validating technology. Computer, 31(5):23-31, May 1998.

Emmanuel Jeannot 16/33 Experiment Driven Research

Error bars in scientific papers

Who has ever published a paper with error bars? In computer science, very few papers contain error bars:

Euro-Par Nb Papers With Error Bar Percentage 2007 89 5 5.6% 2008 89 3 3.4% 2009 86 2 2.3% 3 last conf 264 10 3.8%

Emmanuel Jeannot 17/33 Experiment Driven Research

Three paradigms of computer science

Theory Three feedback loops of the three paradigm of CS [Denning 89], [Feitelson 07]

Definition Theorem Proof Result interpretation

Modeling

Observation Model Prediction Experimental test

Design

Idea/need Design Implementation Experimental validation

Emmanuel Jeannot 18/33 Experiment Driven Research

Two types of experiments

• Test and compare:
• 1. Model validation (comparing models

with reality): is my hypothesis valid?

• 2. Quantitative validation (measuring

performance): is my solution better?

• Can occur at the same time. Ex.

validation of the implementation

• f an algorithm:
• grounding modeling is precise
• design is correct

Idea/need Design Implementation Experimental validation Observation Model Prediction Experimental test

Emmanuel Jeannot 19/33 Experiment Driven Research

Main advantages

[Tichy 98]: ! Experiments : testing hypothesis, algorithms or programs help construct a database of knowledge on theories, methods and tools used for such study. ! Observations: unexpected or negative results "eliminate some less fruitful field of study, erroneous approaches or false hypothesis.

Emmanuel Jeannot 20/33 Experiment Driven Research

“Good experiments”

A good experiment should fulfill the following properties [Algorille project 05]

• Reproducibility: must give the same result with the same input
• Extensibility: must target possible comparisons with other works and

extensions (more/other processors, larger data sets, different architectures)

• Applicability: must define realistic parameters and must allow for an

easy calibration

• “Revisability”: when an implementation does not perform as expected,

must help to identify the reasons

Emmanuel Jeannot 21/33 Experiment Driven Research

Outline

! Importance, role and properties of experiments in computer science (and in grid computing) ! Different experimental methodologies

• Simulation
• Emulation
• Benchmarking
• Real-scale

! Tools for performing experiments

Emmanuel Jeannot 22/33 Experiment Driven Research

Experimental Validation

A good alternative to analytical validation:

• Provides a comparison between algorithms and programs
• Provides a validation of the model or helps to define the validity domain of the

model

Several methodologies:

• Simulation (SimGrid, NS, …)
• Emulation (MicroGrid, Wrekavoc, …)
• Benchmarking (NAS, SPEC, Linpack, ….)
• Real-scale (Grid’5000, PlanetLab, …)

Emmanuel Jeannot 23/33 Experiment Driven Research

Properties of methodologies

Enabling good experiments: Control:

• essential to know which part of the model or the implementation are evaluated
• allows testing and evaluating each part independently

Reproducibility:

• base of the experimental protocol
• Ensured experimental environment

Realism:

• Experimental condition: always (somehow) synthetic conditions
• Level of abstraction depends on the chosen environment
• Three levels of realism:
• 1. Qualitative: experiment says A1#A2 then in reality A1#A2
• 2. Quantitative: experiment says A1=k*A2 then in reality A1=k*A2
• 3. Predictive.

! Problem of validation

Emmanuel Jeannot 24/33 Experiment Driven Research

Simulation

Simulation [Quinson 08]: predict parts of the behavior of a system using an approximate model

• Model = Collection of attributes + set of rules governing how elements interact
• Simulator: computing the interactions according to the rules

Models wanted features:

• Accuracy/realism: correspondence between simulation and real-world
• Scalability: actually usable by computers (fast enough)
• Tractability: actually usable by human beings (understandable)
• “Instanciability”: can actually describe real settings (no magic parameters)

"Scientific challenges

Emmanuel Jeannot 25/33 Experiment Driven Research

Emulation

Emulation: executing a real application on a model of the environment Two approaches:

• Sandbox/virtual machine: confined execution on (a) real machine(s).

syscall catch. Ex: MicroGrid

• Degradation of the environment (to make it heterogeneous): direct
• execution. Ex: Wrekavoc

Emmanuel Jeannot 26/33 Experiment Driven Research

Benchmark

Synthetic application:

• Test workload
• Model of a real application workload
• Shared by other scientists
• Do not care for the output (e.g. random matrix multiplication).

Classical benchmark:

• NAS parallel benchmarks (diff. kernels, size and class).
• Linpack (Top 500
• SPEC
• Montage workflow
• Archive:

! Grid Workload archive (GWA) ! Failure trace archive (FTA)

Emmanuel Jeannot 27/33 Experiment Driven Research

In-situ/Real scale

Real application executed on real (dedicated) hardware/ environment Challenges:

• Configuration
• “Genericity”
• Experiment cycle time
• Ease of use, cost

Emmanuel Jeannot 28/33 Experiment Driven Research

A unifed Taxonomy [GJQ09]

Simulation Emulation In-Situ (real scale) Benchmarking Real application Real environnement Model of the environnement Model of the application Grid’5000 Das Planet Lab Linpack Montage Workflow NAS SimGRID GridSim P2PSim MicroGRID Wrekavoc ModelNet

Emmanuel Jeannot 29/33 Experiment Driven Research

Warning: running a benchmark on an emulator is different than doing a simulation See: J. Gustedt, E. Jeannot and M. Quinson Experimental Methodologies for Large-Scale Systems: a Survey. PPL, 19(3):399–418, September 2009

Performing an evaluation study

The Art of Computer Systems Performance Analysis [Jain 91, Chap. 2]:

1. State the goal and study and define system boundaries 2. List system services and possible outcome 3. Select performance metrics 4. List System and workload parameter 5. Select factors and their values 6. Select evaluation techniques 7. Select workload 8. Design experiments 9. Analyze and interpret the data

• 10. Present the results. Start over, if necessary.

Emmanuel Jeannot 30/33 Experiment Driven Research

State Goals and define the system

• State the goal of your study and define system boundaries:

! XP on collective communication diff. XP on scheduling workload

Emmanuel Jeannot 31/33 Experiment Driven Research

List services and outcome

A system deliver a set of services with different possible outcomes:

• Database answer queries: correct, incorrect or no answer
• Network transfer packet: in order, out of order, loss
• Scheduler allocate tasks: respect deadline

Useful to select the right metric and workload.

Emmanuel Jeannot 32/33 Experiment Driven Research

Select metrics

In general metrics are related to speed, accuracy or availability of services. A good metric should be easy to evaluate and reflect high level criteria. Online scheduler, different criteria: stretch, throughput, response time, etc. Some metrics are correlated some are not (multicriteria study).

Emmanuel Jeannot 33/33 Experiment Driven Research

List parameters

List all parameters that affect performance:

• System/environment parameters (e.g. number of machines, speed of the

network)

• Workload/application parameters (e.g. number of tasks, amount of operation,

size of packets).

After first pass of analysis, you may discover other parameters. Keep the list as comprehensive as possible.

Emmanuel Jeannot 34/33 Experiment Driven Research

Select Factor to Study

List of parameters:

• Some vary during evaluation (factors)
• Some do not

Variation of factors are (sometimes) called levels. You can start with a short list of factors and a small number of levels Parameters suspected to have a high impact on on the performance should be preferably selected as factor. Attention: take care of choosing factors based on their impact and not because thy are easy to vary or measure.

Emmanuel Jeannot 35/33 Experiment Driven Research

Select evaluation technique

! Simulation, ! Emulation, ! Benchmarking, ! In Situ, ! Analytical modeling.

Emmanuel Jeannot 36/33 Experiment Driven Research

Select Workload

What is the input of your evaluation? Different form:

• Real application
• Traces
• Synthetic workload
• Analytical model

In any case, a workload should be representative of the system usage. It might be necessary to produce representative workload (measure and modeling of a system).

Emmanuel Jeannot 37/33 Experiment Driven Research

Design the experiments

List of factors and levels:

• Decide the sequence of experiments that give the maximum of information with

minimal effort.

Two phases:

1. Large number of factor but small number of levels: determine relative effect of factors (fractional, factorial experiment design). 2. Reduced number of factors, levels of factors that have significant impact is increased

Emmanuel Jeannot 38/33 Experiment Driven Research

Analyze and Interpret the Data

Take into account variability of the results Comparing means can lead to inaccurate conclusion (use median, quantiles, etc.) Use statistical technique to compare alterative Interpret results: experimental analysis only produces results and not conclusion!

Emmanuel Jeannot 39/33 Experiment Driven Research

Present results

Final step. You can do better than curve plotting:

• Error bar
• Box plot
• Violin plot
• Histogram
• ECDF
• Scatter plot

Emmanuel Jeannot 40/33 Experiment Driven Research