Outline Heuristics for Continuos Optimization DM812 METAHEURISTICS - - PowerPoint PPT Presentation

DM812 METAHEURISTICS

Lecture 13

Heuristic Methods for Continuous Optimization

Marco Chiarandini

Department of Mathematics and Computer Science University of Southern Denmark, Odense, Denmark <marco@imada.sdu.dk>

An Example Heuristics for Continuos Optimization

Outline

• 1. An Application Example in Econometrics
• 2. Heuristics for Continuos Optimization

An Example Heuristics for Continuos Optimization

Outline

• 1. An Application Example in Econometrics
• 2. Heuristics for Continuos Optimization

An Example Heuristics for Continuos Optimization

Capital Asset Pricing Model

Tool for pricing an individual asset i Individual security’s reward-to-risk ratio = βi · Market’s securities reward-to-risk ratio

• E(Ri) − Rf
• = βi ·
• E(Rm) − Rf
• βi sensitivity of the asset returns to market returns

Under normality assumption and least squares method: βi = Cov(Ri, Rm) Var(Rm) Alternatively: Rit − Rft = β0 + β1 · (Rmt − Rft) Use more robust techniques than least squares to determine β0 and β1

[Winker, Lyra, Sharpe, 2008]

An Example Heuristics for Continuos Optimization

Least Median of Squares

Yt = β0 + β1Xt + ǫt ǫ2

t =

• Yt − β0 − β1Xt

2 least squares method: min

n

• t=1

ǫ2

t

least median of squares method: min

• median
• ǫ2

t

• An Example

Heuristics for Continuos Optimization

−0.05 0.00 0.05 0.10 −0.5 0.0 0.5 1.0 1.5 2.0 0.002 0.004 0.006 0.008 0.010

beta

−0.005 0.000 0.005 0.010 1.2 1.4 1.6 1.8 2.0 0.00010 0.00015 0.00020

beta

An Example Heuristics for Continuos Optimization

Four solutions corresponding to four different local optima (red line: least squares; blue line: least median of squares)

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ++ + + + + + + + ++ + + + + + + + + + + + + + + + + + + ++ + + + + + + + + + + + + + + + + ++ + + + + + + + + + + + + + + + ++ −0.02 0.00 0.02 0.04 −0.04 0.02 0.06 X Yt

median(εt

2) = 5.2e−05

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ++ + + + + + + + ++ + + + + + + + + + + + + + + + + + + ++ + + + + + + + + + + + + + + + + ++ + + + + + + + + + + + + + + + ++ −0.02 0.00 0.02 0.04 −0.04 0.02 0.06 X Yt

median(εt

2) = 0.00014

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ++ + + + + + + + ++ + + + + + + + + + + + + + + + + + + ++ + + + + + + + + + + + + + + + + ++ + + + + + + + + + + + + + + + ++ −0.02 0.00 0.02 0.04 −0.04 0.02 0.06 Yt

median(εt

2) = 8.6e−05

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ++ + + + + + + + ++ + + + + + + + + + + + + + + + + + + ++ + + + + + + + + + + + + + + + + ++ + + + + + + + + + + + + + + + ++ −0.02 0.00 0.02 0.04 −0.04 0.02 0.06 Yt

median(εt

2) = 6.9e−05 An Example Heuristics for Continuos Optimization

Outline

• 1. An Application Example in Econometrics
• 2. Heuristics for Continuos Optimization

An Example Heuristics for Continuos Optimization

Optimization Heuristics

Nelder-Mead Simulated Annealing Differential Evolution Particle Swarm Optimization Genetic Algorithm Ant Colony Optimization

An Example Heuristics for Continuos Optimization

Nelder-Mead

Simplex based method [Spendley et al. (1962)]

An Example Heuristics for Continuos Optimization

Nelder-Mead (cont.)

An Example Heuristics for Continuos Optimization

Nelder-Mead (cont.)

Nelder-Mead simplex method [Nelder and Mead, 1965]:

An Example Heuristics for Continuos Optimization

Nelder-Mead (cont.)

Example:

An Example Heuristics for Continuos Optimization

Simulated Annealing

Simulated Annealing (SA): determine initial candidate solution s set initial temperature T = T0 while termination condition is not satisfied do while keep T constant, that is, Tmax iterations not elapsed do probabilistically choose a neighbor s′ of s using proposal mechanism accept s′ as new search position with probability: p(T, s, s′) := ( 1 if f(s′) ≤ f(s) exp f(s)−f(s′)

T

• therwise

update T according to annealing schedule

Proposal mechanism The next candidate point is generated from a Gaussian Markov kernel with scale proportional to the actual temperature.

An Example Heuristics for Continuos Optimization

Simulated Annealing

Annealing schedule logarithmic cooling schedule [Belisle (1992)] T = T0 ln(⌊ i−1

Imax ⌋Imax + e)

−40 −20 20 40 0.0 0.2 0.4 0.6 0.8 1.0 x Temperature 200 400 600 800 1000 2 4 6 8 10 x Cooling

threshold accepting [Dueck and Scheuer (1990)] accept if ∆ < τ

An Example Heuristics for Continuos Optimization

Differential Evolution

Differential Evolution (DE) determine initial population P while termination criterion is not satisfied do for each solution x of P do generate solution u from three solutions of P by mutation generate solution v from u by recombination with solution x select between x and v solutions

An Example Heuristics for Continuos Optimization

Differential Evolution (cont.)

Solution representation: x = (x1, x2, . . . , xp) Mutation: u = r1 + F · (r2 − r3) F ∈ [0, 2] and (r1, r2, r3) ∈ P Recombination: vj =

• uj

if p < CR or j = r xj

• therwise

j = 1, 2, . . . , p Selection: replace x with v if f(v) is better

An Example Heuristics for Continuos Optimization

Differential Evolution (cont.)

[http://www.icsi.berkeley.edu/~storn/code.html

• K. Price and R. Storn, 1995]

An Example Heuristics for Continuos Optimization

Particle Swarm Optimization

Particle Swarm Optimization

An Example Heuristics for Continuos Optimization

Particle Swarm Optimization

An Example Heuristics for Continuos Optimization

Generation of Initial Solutions

Point generators: Left: Uniform random distribution (pseudo random number generator) Right: Quasi-Monte Carlo method: low discrepancy sequence generator

[Bratley, Fox and Niederreiter, 1994]

• 0.0

0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 β β1

• 0.0

0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 β β1

• (for other methods see spatial point process from spatial statistics)