OCELOTL: CLOSE ENCOUNTERS OF THE THIRD KIND 8th International - - PowerPoint PPT Presentation

▶
ocelotl close encounters of the third kind
SMART_READER_LITE
LIVE PREVIEW

OCELOTL: CLOSE ENCOUNTERS OF THE THIRD KIND 8th International - - PowerPoint PPT Presentation

Aug 27, 2022 140 likes •615 views

OCELOTL: LARGE TRACE OVERVIEWS BASED ON MULTIDIMENSIONAL DATA AGGREGATION OCELOTL: CLOSE ENCOUNTERS OF THE THIRD KIND 8th International Parallel Tools Workshop 1st October 2014 Damien Dosimont 1 2 , Youenn Corre 1 2 , Lucas M. Schnorr 3 ,

slide-1
SLIDE 1

OCELOTL: LARGE TRACE OVERVIEWS BASED ON MULTIDIMENSIONAL DATA AGGREGATION

OCELOTL: CLOSE ENCOUNTERS OF THE THIRD KIND

8th International Parallel Tools Workshop 1st October 2014

Damien Dosimont 1 2, Youenn Corre 1 2, Lucas M. Schnorr 3, Guillaume Huard 2 1, Jean-Marc Vincent 2 1

1 Inria,

first.last@inria.fr,

2 Univ. Grenoble Alpes, LIG, CNRS, F-38000 Grenoble, France

first.last@imag.fr

3 Informatics Institute, UFRGS, Porto Alegre

schnorr@inf.ufrgs.br

slide-2
SLIDE 2

INTRODUCTION

slide-3
SLIDE 3

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

TRACE VISUALIZATION PROBLEMATIC

▶ Trace contents:

  • SPACE = application structure:

▶ hardware components: clusters, machines, cores, etc. ▶ software components: processes, threads, etc.

  • TIME = timestamped events:

▶ function calls, communications, CPU load, malloc, etc.

▶ Traces can be HUGE

→ scalability issues of space-time representations

3

slide-4
SLIDE 4

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

PROBLEMATIC VISUALIZATION

4

slide-5
SLIDE 5

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

OUR PROPOSAL MULTIDIMENSIONAL OVERVIEWS

▶ Several overviews generated thanks to data aggregation

  • Temporal
  • Spatiotemporal

▶ Showing meaningful information (phases, perturbations) ▶ Possibility to adjust dynamically the level of details

5

slide-6
SLIDE 6

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

EXAMPLE : TEMPORAL OVERVIEW

6

slide-7
SLIDE 7

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

EXAMPLE : SPATIOTEMPORAL OVERVIEW

0s 20s 40s 60s SA SB SC MPI_Init MPI_Recv MPI_Allreduce Temporal Perturbation Heterogeneous Spatiotemporal Behavior Grid'5000 Nancy, 700 Cores, NASBP LU, class C 7

slide-8
SLIDE 8

THEORETICAL BACKGROUND : LAMARCHE-PERRIN METHODOLOGY

slide-9
SLIDE 9

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

ADAPTING AN AGGREGATION METHODOLOGY

9

slide-10
SLIDE 10

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

INFORMATION LOSS: KL DIVERGENCE

lossE = ∑

e∈E

ρe log2 ( ρe ρE )

10

slide-11
SLIDE 11

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

COMPLEXITY REDUCTION: SHANNON ENTROPY

gainE = ρE log2 ρE − ∑

e∈E

ρe log2 ρe

11

slide-12
SLIDE 12

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

TRADE-OFF: PIC

pICE = p gainE −(1−p) lossE pICP = ∑

E∈P

pICE

▶ For a given p: choose P with the highest pIC ▶ Aggregate in priority most homogeneous values

12

slide-13
SLIDE 13

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

VIVA: SPATIAL AGGREGATION (SCHNORR & LP)

13

slide-14
SLIDE 14

TEMPORAL OVERVIEW

slide-15
SLIDE 15

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

TEMPORAL AGGREGATION AND VISUALIZATION

IRQ Process 1 Process 2 Time

1 2 3 12 3

Spatial Aggregation Temporal Aggregation

15

slide-16
SLIDE 16

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

MINIMUM INFORMATION LOSS: P=0

16

slide-17
SLIDE 17

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

MAXIMUM COMPLEXITY REDUCTION: P=1

17

slide-18
SLIDE 18

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

INTERESTING TRADE-OFF

18

slide-19
SLIDE 19

SPATIOTEMPORAL OVERVIEW

slide-20
SLIDE 20

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

GENERATE A TRACE MICROSCOPIC MODEL

s1 s2 s3 s4 s5 s6 s7 s8 s9 s10 s11 s12

S

SA SB SC t1 t6 t9 t14 t19

T |X| = 2, ρx(s, t) = dx(s, t)/d(t) ∈ [0, 1], ρ1(s, t) = 1 − ρ2(s, t)

20

slide-21
SLIDE 21

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

AGGREGATE THE MICROSCOPIC MODEL

s1 s2 s3 s4 s5 s6 s7 s8 s9 s10 s11 s12

S

SA SB SC

p = 0 p = a, 0 < a < 1

s1 s2 s3 s4 s5 s6 s7 s8 s9 s10 s11 s12

S

SA SB SC

p = b, 0 < a < b < 1 p = 1

21

slide-22
SLIDE 22

IMPLEMENTATION AND FEATURES

slide-23
SLIDE 23

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

OCELOTL TOOL

▶ Implementation of the overview techniques ▶ Generic architecture. Add:

  • Your own aggregation operator (dimensions, metric)
  • Your own visualization

▶ Persistent caches to avoid long recomputations ▶ Integrated in Framesoc:

  • Trace and tools management
  • Fast trace reading (DB queries)
  • Interaction with other analysis tools
  • Also enable to add you own tools

23

slide-24
SLIDE 24

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

FRAMESOC

▶ Trace format compatibility : Pajé (Akypuera: tool to convert

from OTF2, Tau), LTTng, KPTrace

24

slide-25
SLIDE 25

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

PERFORMANCE: TEMPORAL ANALYSIS

Total analysis time as a function of trace size (100 time slices)

2 4 6 8 10 12 14 0,5 1 1,5 2 2,5 3 3,5 4 No cache With cache

Size (GB) Time (min) 25

slide-26
SLIDE 26

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

PERFORMANCE: TEMPORAL ANALYSIS

Total analysis time as a function of trace size (1000 time slices)

2 4 6 8 10 12 14 1 2 3 4 5 6 7 8 9 No Cache With Cache

Size (GB) Time (min) 26

slide-27
SLIDE 27

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

PERFORMANCE: SPATIOTEMPORAL ANALYSIS

Total analysis time as a function of trace size (30 time slices)

2 4 6 8 10 12 14 1 2 3 4 5 6 No cache With cache

Size (GB) Time (min) 27

slide-28
SLIDE 28

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

DEMONSTRATION

28

slide-29
SLIDE 29

CONCLUSION

slide-30
SLIDE 30

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

CONCLUSION

▶ Visualizations based on spatiotemporal data

aggregation

  • Solves screen, computing and analyst capability

limitations

  • Gives meaningful information about homogeneity

(phases, perturbations)

▶ Implementation:

  • Interaction (zoom, switch to other tools)
  • Helps to drastically reduce computation times (caches)
  • Generic architecture: add your own aggregation and

visualization

▶ Future work:

  • Extend methodology and design new algorithms

(H(S) × H(S) × I(T), surface, etc.)

  • Improve visualization and interaction to get more details
  • Framesoc: native compatibility with OTF2 (soon)

30

slide-31
SLIDE 31

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

LINKS Ocelotl:

http://github.com/dosimont/ocelotl

Framesoc:

http://github.com/generoso/framesoc

Viva:

http://github.com/schnorr/viva

31

slide-32
SLIDE 32

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

THANK YOU FOR YOUR ATTENTION

32

slide-33
SLIDE 33

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

OCELOTL: TEMPORAL AGGREGATION (1)

33

slide-34
SLIDE 34

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

OCELOTL: TEMPORAL AGGREGATION (2)

34

slide-35
SLIDE 35

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

GENERATE A TRACE MICROSCOPIC MODEL

IRQ Process 1 Process 2 Time

1 2.1 1 3 4.1 2 4.1 4

IRQ Process 1 Process 2

2 4.9 3 2.4

IRQ Process 1 Process 2

And so on...

1 2 3

35

slide-36
SLIDE 36

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

DATA AGGREGATION METHODOLOGY

▶ A1. Choose a model and a metric ▶ A2. Choose on which dimension(s) aggregate ▶ A3. Define the operands ▶ A4. Constrain the aggregation : → partitions P allowed ▶ A5. Define the operator ▶ A6. Define the trigger - the aggregation condition ▶ A7. Build the algorithm satisfying A1-A6

36

slide-37
SLIDE 37

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

A2-A5

▶ A2. We aggregate simultaneously on T and S ▶ A3. Operands: (s, t) ∈ S × T ▶ A4. Constraint: A(S × T) = H(S) × I(T)

Aggregation result is a partition P(S × T) ∈ A(S × T)

▶ A5. Operator: + ▶ A6. Trigger: maximize pIC of the partition P(S × T)

s1 s2 s3 s4 s5 s6 s7 s8 s9 s10 s11 s12

S

SA SB SC 37

slide-38
SLIDE 38

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

BEST CUT ALGORITHM

▶ Compute the partition with the highest pIC :

  • Cut an area : time, space (or no cut)
  • Best cut: the partition P where ∑

E∈P pICE is max

  • Recursively cut and evaluate the partitions of E1, E2 ∈ P
  • Useless recomputation is avoided

W V Q P O J C D B A 1 2 3 4 5 6 1 2 3 4 5 6 3 6 3 6 2 5 1 4 2 5 1 4 2 5 1 4 2 5 1 4 3 2 3 2 6 5 6 5 ? ? ? ? 1 4 1 4 ? 2 5 2 5 ? 1 2 3 4 5 6 2 1 2 1 5 4 5 4 ? 1 2 3 4 5 6 3 2 2 1 6 5 5 4 ? 1 4 3 6 ? 2 1 5 4 3 6 1 2 3 4 5 6 3 6 2 5 3 6 2 5 ? E ? ? ? ?

38

slide-39
SLIDE 39

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

  • A6. TRIGGER THE AGGREGATION

▶ Quantification of data reduction and information loss

  • aggregate the homogeneous areas
  • preserve the microscopic information of the heterogeneous

areas

▶ Each (Sk, T(i,j)) ∈ A(S × T) has an associated gain and loss ▶ gain and loss of a partition P(S × T) is the sum of gain and

loss of its content (Sk, T(i,j)) ∈ P(S × T)

s1 s2 s3 s4 s5 s6 s7 s8 s9 s10 s11 s12

S

SA SB SC 39

slide-40
SLIDE 40

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

ELMQVIST-FEKETE CRITERIA

▶ Shneiderman : overview, zoom and filter, then get details

  • n demand

▶ Elmqvist & Fekete: guidelines to design an overview

visualization based on hierarchical aggregation

  • G1. Entity Budget
  • G2. Visual Summary
  • G3. Visual Simplicity
  • G4. Discriminability
  • G5. Fidelity
  • G6. Interpretability

40

slide-41
SLIDE 41

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

VISUALIZATIONS NOT FULFILLING THESE CRITERIA (1)

Example of Gantt chart - space-time diagram

41

slide-42
SLIDE 42

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

VISUALIZATIONS NOT FULFILLING THESE CRITERIA (2)

KPTrace: G1 (time), G2, G4, G5

42

slide-43
SLIDE 43

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

VISUALIZATIONS NOT FULFILLING THESE CRITERIA (2)

Pajé: G1 (space), G2

43

slide-44
SLIDE 44

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

CG CLASS C, 64 PROCESSES ON G5K RENNES

44

slide-45
SLIDE 45

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

LU CLASS C, 700 PROCESSES ON G5K NANCY

0s 20s 40s 60s SA SB SC MPI_Init MPI_Recv MPI_Allreduce Temporal Perturbation Heterogeneous Spatiotemporal Behavior

45

slide-46
SLIDE 46

. . . . . Introduction . . . . . Theoretical Background . . . . Temporal Overview . . Spatiotemporal Overview . . . . . . Implementation and Features Conclusion

PERFORMANCES (SPATIOTEMPORAL)

Case A Case B Case C Case D Application CG, class C CG, class C LU, class C LU, class B Processes 64 512 700 900 Site Rennes Grenoble Nancy Rennes Clusters (nodes) parapide(8) adonis(9), edel(24), genepi(31) graphene(26), graphite(4), griffon(67) paradent(38), parapide(21), parapluie(18) Event number 3,838,144 49,149,440 218,457,456 177,376,729 Trace size 136.9 MB 1.8 GB 8.3 GB 6.7 GB Ocelotl computation times (30 time slices) Trace reading + Microscopic description 5 s 31 s 222 s 174 s Aggregation <1s <1s 2s 2s

46