Metaheuristics applied to the feature selection problem Tom Fredrik - - PowerPoint PPT Presentation

Introduction Problem Algorithms Case study Fontainebleu Conclusion

Metaheuristics applied to the feature selection problem

Tom Fredrik B. Klaussen

Department of Mathematics University of Oslo

Master presentation 28th June - 2006

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion What we will see Motivation Feature selection

I will present 2 new methods for the FSP I will present a number of adapted methods, from other scientific fields. I will present a visualization algorithm for the FSP

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion What we will see Motivation Feature selection

I will present 2 new methods for the FSP I will present a number of adapted methods, from other scientific fields. I will present a visualization algorithm for the FSP

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion What we will see Motivation Feature selection

I will present 2 new methods for the FSP I will present a number of adapted methods, from other scientific fields. I will present a visualization algorithm for the FSP

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion What we will see Motivation Feature selection

Segmentation

We want to separate a set of data into distinct subregions: Figure: Ground truth for Fontainebleu dataset

Curse of dimensionality Natural to add more data to get better separation, but results

• ften deteriorate

Solution: Try to use the “best” set of data

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion What we will see Motivation Feature selection

Segmentation

We want to separate a set of data into distinct subregions: Figure: Ground truth for Fontainebleu dataset

Curse of dimensionality Natural to add more data to get better separation, but results

• ften deteriorate

Solution: Try to use the “best” set of data

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion What we will see Motivation Feature selection

Segmentation

We want to separate a set of data into distinct subregions: Figure: Ground truth for Fontainebleu dataset

Curse of dimensionality Natural to add more data to get better separation, but results

• ften deteriorate

Solution: Try to use the “best” set of data

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion What we will see Motivation Feature selection

Feature selection

The Feature Selection Problem is: An Image Analysis Problem A Parameter Estimation Problem A Discrete Optimization Problem

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion What we will see Motivation Feature selection

Feature selection

The Feature Selection Problem is: An Image Analysis Problem A Parameter Estimation Problem A Discrete Optimization Problem

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion What we will see Motivation Feature selection

Feature selection

The Feature Selection Problem is: An Image Analysis Problem A Parameter Estimation Problem A Discrete Optimization Problem

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion What we will see Motivation Feature selection

Feature selection

The Feature Selection Problem is: An Image Analysis Problem A Parameter Estimation Problem A Discrete Optimization Problem

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion What we will see Motivation Feature selection

Feature selection

The Feature Selection Problem is: An Image Analysis Problem A Parameter Estimation Problem A Discrete Optimization Problem

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Problem formulation Regularization Motivation for metaheuristics

Standard formulation max

c

p(x|ωc) = 1 (2π)d/2|Σc|1/2 exp(−1 2(x − ˆ µc)TΣ−1

c (x − ˆ

µc)) Cholesky factorization LcLT

c = Σc

Equivalent, more easily calculated formulation max

c

p∗(x|ωc) = −1 2||L−1

c (x − ˆ

µc)||2 − log(

N

• i=1

(L−1

c )i,i)

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Problem formulation Regularization Motivation for metaheuristics

Standard formulation max

c

p(x|ωc) = 1 (2π)d/2|Σc|1/2 exp(−1 2(x − ˆ µc)TΣ−1

c (x − ˆ

µc)) Cholesky factorization LcLT

c = Σc

Equivalent, more easily calculated formulation max

c

p∗(x|ωc) = −1 2||L−1

c (x − ˆ

µc)||2 − log(

N

• i=1

(L−1

c )i,i)

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Problem formulation Regularization Motivation for metaheuristics

Standard formulation max

c

p(x|ωc) = 1 (2π)d/2|Σc|1/2 exp(−1 2(x − ˆ µc)TΣ−1

c (x − ˆ

µc)) Cholesky factorization LcLT

c = Σc

Equivalent, more easily calculated formulation max

c

p∗(x|ωc) = −1 2||L−1

c (x − ˆ

µc)||2 − log(

N

• i=1

(L−1

c )i,i)

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Problem formulation Regularization Motivation for metaheuristics

Motivation Inverting nearly singular matrices is numerically very unstable. How can we get more stability? Regularization Side effect: stabilizes the parameter estimation on its own merit. Choice of regularizer ˆ Σc(α) = αˆ Σc + (1 − α)ˆ Σ (1) ˆ Σ(λ) = λˆ Σ + (1 − λ)ˆ σ2I (2) which combined yields: ˆ Σc(α, λ) = αˆ Σc + (1 − α)ˆ Σ(λ) (3)

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Problem formulation Regularization Motivation for metaheuristics

Motivation Inverting nearly singular matrices is numerically very unstable. How can we get more stability? Regularization Side effect: stabilizes the parameter estimation on its own merit. Choice of regularizer ˆ Σc(α) = αˆ Σc + (1 − α)ˆ Σ (1) ˆ Σ(λ) = λˆ Σ + (1 − λ)ˆ σ2I (2) which combined yields: ˆ Σc(α, λ) = αˆ Σc + (1 − α)ˆ Σ(λ) (3)

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Problem formulation Regularization Motivation for metaheuristics

Motivation Inverting nearly singular matrices is numerically very unstable. How can we get more stability? Regularization Side effect: stabilizes the parameter estimation on its own merit. Choice of regularizer ˆ Σc(α) = αˆ Σc + (1 − α)ˆ Σ (1) ˆ Σ(λ) = λˆ Σ + (1 − λ)ˆ σ2I (2) which combined yields: ˆ Σc(α, λ) = αˆ Σc + (1 − α)ˆ Σ(λ) (3)

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Problem formulation Regularization Motivation for metaheuristics

Motivation Inverting nearly singular matrices is numerically very unstable. How can we get more stability? Regularization Side effect: stabilizes the parameter estimation on its own merit. Choice of regularizer ˆ Σc(α) = αˆ Σc + (1 − α)ˆ Σ (1) ˆ Σ(λ) = λˆ Σ + (1 − λ)ˆ σ2I (2) which combined yields: ˆ Σc(α, λ) = αˆ Σc + (1 − α)ˆ Σ(λ) (3)

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Problem formulation Regularization Motivation for metaheuristics

Running time of exact algorithms Exact algorithms are available for the Feature Selection Problem, but running time is exponential. Local search Local search is an easy way, to obtain relatively good solutions, but cannot exit a local optimum. Metaheuristics

Metaheuristics are a class of algorithms that uses knowledge of the problem topology in order to move from one place in the search-space to another place in hopefully an intelligent manner. Common to them all are the fact that they use more or less local information to decide where to go next. This use of local information, while maybe carefully devised, and often effective, can in general not guarantee that we find the best solution to our problem. In fact we can only hope that we are in the vicinity of a good solution.

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Problem formulation Regularization Motivation for metaheuristics

Running time of exact algorithms Exact algorithms are available for the Feature Selection Problem, but running time is exponential. Local search Local search is an easy way, to obtain relatively good solutions, but cannot exit a local optimum. Metaheuristics

Metaheuristics are a class of algorithms that uses knowledge of the problem topology in order to move from one place in the search-space to another place in hopefully an intelligent manner. Common to them all are the fact that they use more or less local information to decide where to go next. This use of local information, while maybe carefully devised, and often effective, can in general not guarantee that we find the best solution to our problem. In fact we can only hope that we are in the vicinity of a good solution.

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Problem formulation Regularization Motivation for metaheuristics

Running time of exact algorithms Exact algorithms are available for the Feature Selection Problem, but running time is exponential. Local search Local search is an easy way, to obtain relatively good solutions, but cannot exit a local optimum. Metaheuristics

Metaheuristics are a class of algorithms that uses knowledge of the problem topology in order to move from one place in the search-space to another place in hopefully an intelligent manner. Common to them all are the fact that they use more or less local information to decide where to go next. This use of local information, while maybe carefully devised, and often effective, can in general not guarantee that we find the best solution to our problem. In fact we can only hope that we are in the vicinity of a good solution.

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search

Floating searches Floating searches are a collection of essentially greedy construction heuristics. Genetic Algorithms (GA) Genetic Algorithm a population based metaheuristic. Based on evolution theory. Simulated Annealing (SA) Simulated Annealing is perhaps the best known metaheuristic. Based on the notion of cooling an alloy to minimize tension.

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search

Floating searches Floating searches are a collection of essentially greedy construction heuristics. Genetic Algorithms (GA) Genetic Algorithm a population based metaheuristic. Based on evolution theory. Simulated Annealing (SA) Simulated Annealing is perhaps the best known metaheuristic. Based on the notion of cooling an alloy to minimize tension.

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search

Floating searches Floating searches are a collection of essentially greedy construction heuristics. Genetic Algorithms (GA) Genetic Algorithm a population based metaheuristic. Based on evolution theory. Simulated Annealing (SA) Simulated Annealing is perhaps the best known metaheuristic. Based on the notion of cooling an alloy to minimize tension.

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search

Huff-Puff Based on path relinking, is designed to increase the diversity compared to GA. Variable Neigborhood Search (VNS) Variable Neigbhorhood Search is based on the fact that local

• ptimum are not necessarily local optimum in all neighborhoods. A

global optimum is however a global optimum in every neighborhood. Roaming Search Roaming search is based on VNS, it is designed to give added diversity compared to VNS.

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search

Huff-Puff Based on path relinking, is designed to increase the diversity compared to GA. Variable Neigborhood Search (VNS) Variable Neigbhorhood Search is based on the fact that local

• ptimum are not necessarily local optimum in all neighborhoods. A

global optimum is however a global optimum in every neighborhood. Roaming Search Roaming search is based on VNS, it is designed to give added diversity compared to VNS.

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search

Huff-Puff Based on path relinking, is designed to increase the diversity compared to GA. Variable Neigborhood Search (VNS) Variable Neigbhorhood Search is based on the fact that local

• ptimum are not necessarily local optimum in all neighborhoods. A

global optimum is however a global optimum in every neighborhood. Roaming Search Roaming search is based on VNS, it is designed to give added diversity compared to VNS.

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search

Define the projection and signum function:

P(k) =    k ≤ 0 k 0 ≤ k ≤ N N k ≥ N sgn(x) =    −1 x < 0 x = 0 1 x > 0

we may specify a move probability with respect to the guidance function g(k) p(θ|t, k, r) = 1 + sgn(∇g(k))

• g(P(k+1))+g(P(k−1))

2

r/t 2

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search

Define the projection and signum function:

P(k) =    k ≤ 0 k 0 ≤ k ≤ N N k ≥ N sgn(x) =    −1 x < 0 x = 0 1 x > 0

we may specify a move probability with respect to the guidance function g(k) p(θ|t, k, r) = 1 + sgn(∇g(k))

• g(P(k+1))+g(P(k−1))

2

r/t 2

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search

RoamingSA RoamingSA uses a guidance function to guide the search a particular number of features. Defined as the low probability parts

• f the guidance function.

Guided RoamingSA(GRSA) Guided Roaming SA is a hyper heuristic that repeatedly call RoamingSA, but as time passes by, alters the guidance function as more data is collected.

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search

RoamingSA RoamingSA uses a guidance function to guide the search a particular number of features. Defined as the low probability parts

• f the guidance function.

Guided RoamingSA(GRSA) Guided Roaming SA is a hyper heuristic that repeatedly call RoamingSA, but as time passes by, alters the guidance function as more data is collected.

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search

Iterated Locals Search (ILS)

Motivation When local searches get stuck in a local optimum, an iterated local search uses small perturbations in the search space to avoid a local “baisin of attraction”. If perturbation is too small, risks returning to the same baisin. Tabulist Keep a list of the most recently visisted local optimums, if we get a certain number of revisits, increase perturbation.

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search

Iterated Locals Search (ILS)

Motivation When local searches get stuck in a local optimum, an iterated local search uses small perturbations in the search space to avoid a local “baisin of attraction”. If perturbation is too small, risks returning to the same baisin. Tabulist Keep a list of the most recently visisted local optimums, if we get a certain number of revisits, increase perturbation.

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Traditional aproaches Some new aproaches Guidance function RoamingSA Iterated Local Search

Iterated Locals Search (ILS)

Motivation When local searches get stuck in a local optimum, an iterated local search uses small perturbations in the search space to avoid a local “baisin of attraction”. If perturbation is too small, risks returning to the same baisin. Tabulist Keep a list of the most recently visisted local optimums, if we get a certain number of revisits, increase perturbation.

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Bar plots Convergence Discussion

No regularization, CV No regularization, test Regularized, CV Regularized, test

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Bar plots Convergence Discussion

Figure: Selected runs of 3 algorithms, to demonstrate different convergence functions. The runs are for the Rosis set, without regularization.

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Bar plots Convergence Discussion

Inconsistency between crossvalidation and test It is difficult to conclude with a single best methdod. Since we see a large discrepancy between test and CV. However deviance is low, which suggests underlaying effects. Best methods GRSA, and the add-l-rem-r method do well in all tests. The ILS is very good on CV, but seems to overadapt.

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Bar plots Convergence Discussion

Inconsistency between crossvalidation and test It is difficult to conclude with a single best methdod. Since we see a large discrepancy between test and CV. However deviance is low, which suggests underlaying effects. Best methods GRSA, and the add-l-rem-r method do well in all tests. The ILS is very good on CV, but seems to overadapt.

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Summary

I have presented 2 new methods for the FSP I have presented a number of adapted methods, from other scientific fields. The visualization algorithm will be shown to you now.

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Summary

I have presented 2 new methods for the FSP I have presented a number of adapted methods, from other scientific fields. The visualization algorithm will be shown to you now.

Klaussen Metaheuristics applied to the feature selection problem

Introduction Problem Algorithms Case study Fontainebleu Conclusion Summary

I have presented 2 new methods for the FSP I have presented a number of adapted methods, from other scientific fields. The visualization algorithm will be shown to you now.

Klaussen Metaheuristics applied to the feature selection problem