Semantic GP Frameworks: Alignment in the Error Space and - - PowerPoint PPT Presentation

SLIDE 1

Semantic GP Frameworks: Alignment in the Error Space and Equivalence classes

Stefano Ruberto

GSSI

July 5, 2016

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 1 / 48

SLIDE 2

Overview

1

Optimization problem

2

Introduction to Genetic Programming

3

Software Engineering point of view

4

A case study

5

Semantic Genetic Programming

6

Overview

7

Semantic space and error space

8

Alignment in the Error Space

9

Reconstruction of the Optimum

10 A New Goal For GP 11 How to search for Optimally Aligned Individuals? 12 ESAGP 13 Experimental Study 14 Computational Complexity 15 ESAGP : observations 16 Future Works: semantic based equivalence classes

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 2 / 48

SLIDE 3

Optimization Problem

Solution space and fitness

Solve an optimization problem to find the best solutions in a (often huge) candidate set. Considering a pair (S, f ) where S is the set of all possible solutions (search space) and f the function: f = S → R f measures the quality of the solution in S and is called fitness function

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 3 / 48

SLIDE 4

Optimization Problems

Definitions

Global Optimum and Local Optimum

Maximization Problem

We look for the solution s ∈ S such that f (s) ≥ f (i), ∀i ∈ S

Local maxima

the solution s ∈ S such that f (s) ≥ f (i), ∀i ∈ Ns where Ns is in the neighbourhood

• f s, given some criteria

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 4 / 48

SLIDE 5

Supervised Learning

Symbolic Regression

Symbolic Regression      x11 x12 · · · x1n y1 x21 x22 · · · x2n y2 . . . . . . ... . . . . . . xm1 xm2 · · · xmn ym      In a regression problem we look for the function g such that ∀i = 1, 2, · · · , m holds that g(xi1, xi2, · · · , xin) = yi

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 5 / 48

SLIDE 6

Introduction to Genetic Programming.

The Genetic Programming is framed within the broader family of evolutionary algorithms (EA) [Bac96].The EA are inspired by Darwin’s theory of evolution in its various aspects and, specifically for GP, on the iterative process based on reproduction, mutation, competition and selection.

Note

One of the main features of the GP is to output, as a result of the evolution, a real algorithm. White box approach as opposed to black box one.

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 6 / 48

SLIDE 7

Introduction to Genetic Programming.

Example applications

Genetic programming has produced results that can be called “human competitive” from a wide variety of fields, here is some example of successful application [PK14],[ES15]: Regression or Classification of non-linear problem. In telecommunications: speech quality estimation In finance: evolving effective bidding strategies. Networks: network coding. Clinical applications: Cancer detectors, seizure detectors, mental health diagnosis, etc

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 7 / 48

SLIDE 8

Introduction to Genetic Programming.

Application example in Software Egineering

Search Based Software Engineering

Software engineering is ideal for the application of metaheuristic search techniques, such as genetic algorithms, simulated annealing and tabu search. Such search-based techniques could provide solutions to the difficult problems of balancing competing constraints and may suggest ways of finding acceptable solutions in situations where perfect solutions are either theoretically impossible or practically infeasible. Mark Harman 2001 [HJ01] There is usually a need to balance competing constraints. Occasionally there is a need to cope with inconsistency. There are often many potential solutions. There is typically no perfect answer. . . but good ones can be recognised. There are sometimes no precise rules for computing the best solution.

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 8 / 48

SLIDE 9

Introduction to Genetic Programming.

Application example in Software Egineering

Fundamental steps

Representation of the problem. Definition of the fitness function Existing Applications of Optimization Techniques to Software engineering [Har07] Accurate cost estimates Staff allocations in project planning Requirements to form the next release Optimizing design decisions Optimizing source code Optimizing test data generation Optimizing test data selection Optimizing maintenance and reverse engineering ... many others...

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 9 / 48

SLIDE 10

Introduction to Genetic Programming.

Application example Adaptive Software Engineering

Genetic Improvement for Adaptive Software Engineering [HJL+14]

• Stefano Ruberto (GSSI)

Semantic GP July 5, 2016 10 / 48

SLIDE 11

A case study

Performance measure

Using Machine Learning Techniques for Predicting Performance Robustness of Software Under Uncertainty

Measurement-based performance evaluation process in order to support stakeholders in the evaluation of systems that have to meet performance requirements. Uncertainty is critical in the performance domain when it relates to workload, operational profile, and resource demand. It is necessary to sample uncertain parameters The application-level monitoring of software system samples is very expensive in terms of time consumption and resource usage. Sampled Input Param.: Number Of Users, Think Time, Number Of Items To Cart, Cpu Catalog. Measured Param.: Response time, Throughput,CPU Utilization.

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 11 / 48

SLIDE 12

A case study

Procedure

Uncertainty

Workload Exp. distr. Operational profile Gamma distr. sample

Specification

Resource Uniform distr. demand

Use both model predictions and software measurements to evaluate software performances

Machine Learning

Performance Performance

System Monitoring

Software system

Performance Robustness

measurement prediction

Results Interpretation

Sample count Sample count Sample count Tolerance level ML models Performance measurements and predictions

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 12 / 48

SLIDE 13

A case study

Results

Results on the real-world enterprise application, i.e., the JPetStore

Performance Requirements TH RT U (TH, U) (TH, RT) (RT, U) (TH, RT, U) Measurements and Predictions 0.68 0.95 0.75 0.66 0.66 0.73 0.64 Measurements 0.69 0.89 0.74 0.66 — — —

TH ¡ RT ¡ U ¡ TH, ¡U ¡ Meas ¡and ¡Pred ¡ 30.02 ¡ 30.02 ¡ 30.02 ¡ 30.02 ¡ Measurements ¡ 125.25 ¡ 78.25 ¡ 121.00 ¡ 124.75 ¡

0 ¡ 20 ¡ 40 ¡ 60 ¡ 80 ¡ 100 ¡ 120 ¡ 140 ¡

Computa?onal ¡?me ¡(hours) ¡

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 13 / 48

SLIDE 14

Introduction to Genetic Programming

Solution structure

Unlike other evolutionary algorithms Genetic Programming (GP) has a variable solution size : often tree structures are used. Functional symbols: F = {f1, f2, ..., fn} Terminal symbols: T = {t1, t2, ..., tn} Example: y = 2 ∗ x2 + x

Figure: Syntactic tree.

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 14 / 48

SLIDE 15

Introduction to Genetic Programming

Evolution cycle

Initialization: full, grow, RH&H, etc. Selection: Tournament Selection, Fitness Proportional Selection, Ranking Selection, etc. Genetic operators: Mutation, Crossover, etc. Fitness function: RMSE, etc. .

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 15 / 48

SLIDE 16

Open Problems

Overview

Bloat: During the evolution, the number of nodes in the trees start growing in a way non proportional to fitness improvement. Premature convergence: The loss of genetic diversity in the population trapped in a local optima. Overfitting: Given a hypothesis space H , a hypothesis h ∈ H is said to

• verfit the training data if there exists some alternative

hypothesis h′ ∈ H, such that h has smaller error than h′ over the training examples, and h′ has a smaller error than h over the entire distribution of instances.

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 16 / 48

SLIDE 17

Semantic Genetic Programming

Overview

Goal: maximize phenotype diversity in population to counteract premature convergence exploring larger areas of the solution space.

Structural diversity (Genotype) it’s not enough!

y = 2 ∗ x2 + x

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 17 / 48

SLIDE 18

Semantic Genetic Programming

[Nguyen at al., 2011][Ngu11], [Moraglio et al., 2012] [MKJ12], . . .

Taking the test cases input vector x the software gives as output the semantic vector s.      x11 x12 · · · x1n y1 x21 x22 · · · x2n y2 . . . . . . ... . . . . . . xm1 xm2 · · · xmn ym     

• s ≡ [p(x11, x12, · · · , x1n), p(x21, x22, · · · , x2n), · · · , p(xm1, xm2, · · · , xmn)]

The target is expressed as the vector :

• t ≡ [y1, y2, · · · , ym]

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 18 / 48

SLIDE 19

Overview

summary

Presenting a new way of exploiting semantic awareness and geometry in GP (looking for alignments in the error space ESAGP) Presenting two different methods that implement this idea Discussing the obtained results both on training and on test data for real life applications and benchmark problems. Future works and discussion:

A general GP framework based on Equivalence Classes EC EC incorporate: ESAGP, Linear Scaling and other simple yet effective algorithm. Preliminary results.

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 19 / 48

SLIDE 20

Genotypic and Semantic Space

Semantic space Genotypic space

?

The target is also represented by a point in the semantic space and usually it does not correspond to the origin

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 20 / 48

SLIDE 21

From Semantic to Error Space

error vector of an individual P the vector:

• eP =

sP − t This is a point in a(nother) n-dimensional space, that we call error space.

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 21 / 48

SLIDE 22

From Semantic to Error Space

Example Genotypic space Error space

target

?

Each point in the semantic space is translated by subtracting the target What we get is the error space In the error space, the target is represented by the

• rigin

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 22 / 48

SLIDE 23

Alignment in the Error Space

Error Space Definition: Optimally Aligned Individuals

Two GP individuals A and B are optimally aligned if a scalar constant k exists such that: eA = k · eB In other words, two individuals are

• ptimally aligned if the straight line

that joins their error vectors also intersects the origin. If we find two optimally aligned individuals, we are able to reconstruct a globally optimal solution analytically

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 23 / 48

SLIDE 24

Reconstruction of the Optimum

Let A and B be optimally aligned individuals then:

• eA = k ·

eB − − − − − − − − − − − − − − − − − − →

applying the def. of error vector

sA − t = k · ( sB − t) Obtaining: t =

1 1−k ·

sA −

k 1−k ·

sB Now, we construct an individual with the following genotype:

: :

Important!!

The semantics of this individual is equal to t and thus it is a global

• ptimum !

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 24 / 48

SLIDE 25

A New Goal For GP

If we find two optimally aligned individuals, then we are able to reconstruct a globally optimal solution analytically

Note

This holds regardless of the quality of each one of these two individuals! Thus now the objective of GP can be finding two optimally aligned individuals (instead of searching directly for a globally

• ptimal solution).

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 25 / 48

SLIDE 26

How to search for Optimally Aligned Individuals?

There is a very large number of ways of doing it! We just implemented one of them. The idea:

Error space

Assume that these are the error vectors of the individuals in a GP population (for instance, the initial population of a GP run)

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 26 / 48

SLIDE 27

How to search for Optimally Aligned Individuals?

Error space

We consider a particular direction (point) that (informally speaking) “stands in the middle” of the error vectors

• f the individuals in the

population. We call this point attractor.

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 27 / 48

SLIDE 28

How to search for Optimally Aligned Individuals?

Error space

Than, we make GP “push” the individuals in the population towards alignment with the attractor

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 28 / 48

SLIDE 29

The idea of Optimal Alignment can be extended

• Definition. Optimally Coplanar Individuals

Three GP individuals A, B and C are optimally coplanar if the bi-dimensional plane on which eA, eB and eC lie also intersects the origin of the error space. The proof is not given here (see the paper!). It is based on the idea that the concept of

• ptimal coplanarity can be

seen as an “iteration” of the concept of optimal

• alignment. In fact, in this

figure and are aligned with , and and are aligned with the origin.

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 29 / 48

SLIDE 30

Proposed GP Systems

ESAGP-1 Objective: finding two individuals whose error vectors are aligned on (i.e. belong to) a straight line that intersects the origin. ESAGP-2 Objective: finding three individuals whose error vectors belong to a bi-dimensional plane intersecting the origin. ESAGP-µ Objective: finding µ + 1 individuals whose error vectors belong to a µ-dimensional plane intersecting the origin. [RVCS14]

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 30 / 48

SLIDE 31

ESAGP-1

Algorithm

ESAGP-1 maintains an archive of all the ”semantically new” individuals that have been found during the GP run. Every time a new individual P is generated, the algorithm checks whether it is optimally aligned with any of the individuals already in the archive. If so, the algorithm terminates Otherwise, P is added to the archive, unless the archive already contains an individual with the same semantics, and the algorithm continues.

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 31 / 48

SLIDE 32

ESAGP-1

Fitness

The fitness of an individual is the angle between its error vector and the attractor, and it has to be minimized. Error is never used directly as fitness!

Error Space θ

In this work, as attractor, we used:

• a =
• P∈Pop
• eP
• eP

Remark: the angle between two vectors eA and a is: θ = arccos

• eA ×

a

• eA ·

eA

• Stefano Ruberto (GSSI)

Semantic GP July 5, 2016 32 / 48

SLIDE 33

Experimental Study

Dataset

Two complex real-life applications (prediction of pharmacokinetic parameters in drug discovery)

Dataset Name(ID) Number of features Number of instances Short goal explanatjon

%F

241 359

Predictjng the value of human oral bioavailability of a candidate new drug as a functjon of its molecular descriptors

LD50

626 234

Predictjng the value of the toxicity of a candidate new drug as a functjon of its molecular descriptors

Both these datasets are freely available

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 33 / 48

SLIDE 34

Experimental Results (Errors)

ESAGP−1 ESAGP−2 GS−GP ST−GP

100 200 300 30 35 40 45 50 55 60

Number of Generations Training Error (%F)

100 200 300 1800 2000 2200 2400 2600

Number of Generations Training Error (LD50)

100 200 300 30 35 40 45 50 55 60

Number of Generations Test Error (%F)

100 200 300 1800 2000 2200 2400 2600

Number of Generations Test Error (LD50)

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 34 / 48

SLIDE 35

Experimental Results (Size)

10 20 30 40 50 50 100 150 200 250 300

Number of Generations Number of Nodes (%F)

10 20 30 40 50 50 100 150 200

Number of Generations Number of Nodes (LD50)

ESAGP−1 ESAGP−2 GS−GP ST−GP

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 35 / 48

SLIDE 36

Computational Complexity

Example

ESAGP-1 system add no further fitness evaluation to the complexity of standard GP; it just adds comparisons among (error) vectors that have already been calculated (and stored), in order to check for alignment. In the worst case, ESAGP-1 does n vector comparisons (where n is the archive size). In the worst case, n is equal to (population size × number of generations) (it does not take into account semantic repetitions that are very frequent). In practice: the overhead of time given by the vector comparisons is compensated by the fact that the ESAGP system evolves smaller trees, so the ESAGP system has approximately the same running time as ST-GP.

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 36 / 48

SLIDE 37

Observations

ESAGP

ESAGP-1 and -2 find comparable solutions to the ones of GS-GP after 350 (even 2000!) generations and better than the ones of ST-GP on the test set in 350 generations. Individuals evolved by ESAGP-1 are smaller than the ones of ESAGP-2, which are smaller than the ones of ST-GP, which are much smaller than the ones of GS-GP Adding dimensions seems beneficial (ESAGP-2 consistently

• utperforms ESAGP-1)

Looking for alignment can be easier than directly looking for a global optimum. In our experimental study, we have obtained solutions of comparable quality as other systems, and smaller, earlier.

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 37 / 48

SLIDE 38

Future Works: Semantic based equivalence classes

Motivations

Semantic based equivalence classes (EQC) Different definitions of equivalence are used Exploring the solution space by classes may be more efficient

One single individual represent the class (no ”duplicate” solutions) It is easy to analytically obtain the best individual of the class Probably not all the interesting solutions are in a solution space reachable directly by evolution structured like GP Interesting equivalence classes can capture property of the domain (e.g. position invariance, scale invariance, rotation invariance etc.)

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 38 / 48

SLIDE 39

Test GP Syetem using EQC

GPMUL, GPPLUS

EQC concept is actualized by means of two different novel genetic programming systems, in which two different definitions of equivalence are used.

Equivalent by Translation: GPPLUS

2 individuals are equivalent if P1 − P2 = k and ki = kj ∀i, j

Equivalent by Scale: GPMUL

2 individuals are equivalent if

• P1
• P2 =

k and ki = kj ∀i, j

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 39 / 48

SLIDE 40

Approximate EQC and fitness function

GOAL

The goal of these GP system is to evolve P equivalent to t To measure the distance from the EQC of the target we use the dispersion of values in k e.g. the variance

Filter redundant solutions

Establishing a threshold on variance of k we can reject ”duplicate” semantics.

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 40 / 48

SLIDE 41

Preliminary experimental validation

test problems

Dataset # Features # Instances Airfoil Self-Noise [BPM89] 5 1502 Concrete Compressive Strength [CVS13] 8 1029 Parkinson Voice Recording (TOTAL) [CVS14] 19 5875 Parkinson Voice Recording (MOTOR) [CVS14] 19 5875 Concrete Slump Test [Yeh09] 9 102 Yacht Hydrodynamics [OLG07] 6 307

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 41 / 48

SLIDE 42

Preliminary experimental validation

filter vs NON-filter

Test set of dataset ”Concrete Compressive Strength” filter vs NON-filter

0e+00 1e+07 2e+07 3e+07 4e+07 9 10 11 12 13 14 15 0e+00 1e+07 2e+07 3e+07 4e+07 9 10 11 12 13 14 15 Computational Effort Median Best Fitness FGPPLUS GPPLUS 0e+00 2e+07 4e+07 6e+07 8 9 10 11 12 13 14 0e+00 2e+07 4e+07 6e+07 8 9 10 11 12 13 14 Computational Effort Median Best Fitness FGPMUL GPMUL

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 42 / 48

SLIDE 43

Preliminary experimental validation

General peformance example

Performance of GPPLUS and GPMUL are comparable with Linear scaling and Geometric Semantic GP (example on dataset ”slump”)

FGPPLUS FGPMUL FLS GSGP 2 3 4 5 6 7 Best Fitness

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 43 / 48

SLIDE 44

Preliminary experimental validation

Effect on solutions sizes

Technique Dataset P-value average size % Diff pop. Average size P-value Best size % Diff on best ind. Size GPMUL airfoil 0,00

• 47,66

0,01

• 43,03

concrete 0,59 10,38 0,53 8,63 motor 0,34 33,98 0,48 45,32 motor total 0,00

• 63,76

0,00

• 63,00

slump 0,00

• 93,31

0,00

• 88,70

yacht 0,00

• 44,85

0,00

• 40,39

GPPLUS airfoil 0,00

• 24,39

0,00

• 28,24

concrete 0,27

• 5,77

0,94 0,90 motor 0,06

• 18,88

0,05

• 35,05

motor total 0,74

• 17,98

0,99

• 15,07

slump 0,00

• 58,43

0,00

• 50,16

yacht 0,00

• 61,48

0,00

• 57,62

LS airfoil 0,00

• 37,70

0,05

• 19,02

concrete 0,19

• 6,43

0,33

• 10,75

motor 0,33

• 13,22

0,23 6,14 motor total 0,23

• 26,27

0,27

• 27,34

slump 0,00

• 81,94

0,00

• 81,09

yacht 0,00

• 63,96

0,00

• 63,85

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 44 / 48

SLIDE 45

Conclusion

ESAGP and EQC methods do not optimise directly the error results have comparable or better error solution size is smaller ESAGP is very fast Simple EQC can match partial input (advantages over linear scaling)

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 45 / 48

SLIDE 46

[Bac96]

• T. Back.

Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms. Oxford University Press, 1996. [BPM89]

• T. Brooks, D. Pope, and A. Marcolini.

Airfoil self-noise and prediction. Technical report, NASA RP-1218, 1989. [CVS13] Mauro Castelli, Leonardo Vanneschi, and Sara Silva. Prediction of high performance concrete strength using genetic programming with geometric semantic genetic

• perators.

Expert Systems with Applications, 40(17):6856–6862, 2013. [CVS14] Mauro Castelli, Leonardo Vanneschi, and Sara Silva. Prediction of the unified Parkinson’s disease rating scale assessment using a genetic programming system with geometric semantic genetic operators. Expert Systems with Applications, 41(10):4608 – 4616, 2014. [ES15] Agoston E Eiben and Jim Smith. From evolutionary computation to the evolution of things. Nature, 521(7553):476–482, 2015. [Har07] Mark Harman. The current state and future of search based software engineering. In 2007 Future of Software Engineering, pages 342–357. IEEE Computer Society, 2007. [HJ01] Mark Harman and Bryan F Jones. Search-based software engineering. Information and Software Technology, 43(14):833 – 839, 2001. [HJL+14] Mark Harman, Yue Jia, William B Langdon, Justyna Petke, Iman Hemati Moghadam, Shin Yoo, and Fan Wu. Genetic improvement for adaptive software engineering (keynote). In Proceedings of the 9th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, pages 1–4. ACM, 2014. Stefano Ruberto (GSSI) Semantic GP July 5, 2016 46 / 48

SLIDE 47

[MKJ12] Alberto Moraglio, Krzysztof Krawiec, and Colin G. Johnson. Geometric semantic genetic programming. In Carlos A. Coello Coello, Vincenzo Cutello, Kalyanmoy Deb, Stephanie Forrest, Giuseppe Nicosia, and Mario Pavone, editors, Parallel Problem Solving from Nature, PPSN XII (part 1), volume 7491 of LNCS, pages 21–31. Springer, 2012. [Ngu11] Quang Uy Nguyen. Examining Semantic Diversity and Semantic Locality of Operators in Genetic Programming. PhD thesis, University College Dublin, Ireland, 18 July 2011. [OLG07]

• I. Ortigosa, R. L´
• pez, and J. Garc´

ıa. A neural networks approach to residuary resistance of sailing yachts prediction. Proceedings of the International Conference on Marine Engineering MARINE, 2007:250, 2007. [PK14] Riccardo Poli and John Koza. Genetic Programming. Springer, 2014. [RVCS14] Stefano Ruberto, Leonardo Vanneschi, Mauro Castelli, and Sara Silva. ESAGP - a semantic gp framework based on alignment in the error space. In Genetic Programming, pages 150–161. Springer, 2014. [Yeh09] I-C Yeh. Simulation of concrete slump using neural networks. Proceedings of the Institution of Civil Engineers-Construction Materials, 162(1):11–18, 2009. Stefano Ruberto (GSSI) Semantic GP July 5, 2016 47 / 48

SLIDE 48

The End

Stefano Ruberto (GSSI) Semantic GP July 5, 2016 48 / 48