MATH529 Fundamentals of Optimization Unconstrained Optimization II - - PowerPoint PPT Presentation

MATH529 – Fundamentals of Optimization Unconstrained Optimization II

Marco A. Montes de Oca

Mathematical Sciences, University of Delaware, USA

1 / 31

Recap

2 / 31

Optimization via Calculus

Example Find the local and global minimizers and maximizers on R of f (x) = 3x4 − 4x3 + 1.

3 / 31

Optimization via Calculus

Graph of f (x) = 3x4 − 4x3 + 1.

4 / 31

Optimization via Calculus

Two theorems summarize the basic facts about global

• ptimization of one variable functions.

Theorem (1st order condition (Necessary, but not sufficient)) Suppose that f (x) is a differentiable function on R (or the function’s domain I). If x⋆ is a global minimizer of f (x), then f ′(x⋆) = 0.

5 / 31

Optimization via Calculus

Two theorems summarize the basic facts about global

• ptimization of one variable functions.

Theorem (1st order condition (Necessary, but not sufficient)) Suppose that f (x) is a differentiable function on R (or the function’s domain I). If x⋆ is a global minimizer of f (x), then f ′(x⋆) = 0. Theorem (2nd order condition (Sufficient, but not necessary)) Suppose that f (x), f ′(x), and f ′′(x) are all continuous on R (or I) and that x⋆ is a critical point of f (x). a) If f ′′(x) ≥ 0 for all x ∈ R (or I), then x⋆ is a global minimizer

• f f (x) on R (or I).

b) If f ′′(x) > 0 for all x ∈ I such that x = x⋆, then x⋆ is a strict global minimizer of f (x) on R (or I).

6 / 31

Optimization via Calculus

Local optimization is easier to verify. Theorem (1st order condition (Necessary, but not sufficient)) Suppose that f (x) is a differentiable function on R (or I). If x⋆ is a local minimizer of f (x), then f ′(x⋆) = 0.

7 / 31

Optimization via Calculus

Local optimization is easier to verify. Theorem (1st order condition (Necessary, but not sufficient)) Suppose that f (x) is a differentiable function on R (or I). If x⋆ is a local minimizer of f (x), then f ′(x⋆) = 0. Theorem (2nd order condition (Sufficient, but not necessary)) Suppose that f (x), f ′(x), and f ′′(x) are all continuous on R (or I) and that x⋆ is a critical point of f (x). If f ′′(x⋆) > 0, then x⋆ is a strict local minimizer of f (x).

8 / 31

Optimization via Calculus

Exercise Find the local and global minimizers and maximizers on I = (−1, 1) of f (x) = ln(1 − x2).

9 / 31

Optimization via Calculus What about functions of many variables?

10 / 31

Optimization via Calculus What about functions of many variables? Extend theorems that allow us to identify and classifly local minimizers of one variable functions to multivariable cases.

11 / 31

Optimization via Calculus

Notation: A vector in Rn is an ordered n-tuple x =        x1 x2 x3 . . . xn       

• f real numbers

called components of x. If x and y are vectors in Rn, then their dot product or inner product is defined by x · y = xTy = (x1, x2, x3, . . . , xn)        y1 y2 y3 . . . yn        =

n

• i=1

xiyi where xT is the transpose of x.

12 / 31

Optimization via Calculus

Notation: If f (x) is a function of n variables with continuous first and second partial derivatives on Rn, then the gradient of f (x) is the vector ∇f =          

∂f ∂x1 ∂f ∂x2 ∂f ∂x3

. . .

∂f ∂xn

         

13 / 31

Optimization via Calculus

Notation: The Hessian of f (x), denoted by ∇2f or Hf , is the symmetric n × n matrix ∇2f = Hf =              

∂2f ∂x2

1

∂2f ∂x1∂x2 ∂2f ∂x1∂x3

. . .

∂2f ∂x1∂xn ∂2f ∂x2∂x1 ∂2f ∂x2

2

∂2f ∂x2∂x3

. . .

∂2f ∂x2∂xn ∂2f ∂x3∂x1 ∂2f ∂x3∂x2 ∂2f ∂x2

3

. . .

∂2f ∂x3∂xn

. . . . . . . . . ... . . .

∂2f ∂xn∂x1 ∂2f ∂xn∂x2 ∂2f ∂xn∂x3

. . .

∂2f ∂x2

n

             

14 / 31

Optimization via Calculus

Definition Suppose f (x) is a real-valued function defined on a subset D of

• Rn. A point x⋆ in D is:

A global minimizer for f (x) on D if f (x⋆) ≤ f (x) for all x ∈ D;

15 / 31

Optimization via Calculus

Definition Suppose f (x) is a real-valued function defined on a subset D of

• Rn. A point x⋆ in D is:

A global minimizer for f (x) on D if f (x⋆) ≤ f (x) for all x ∈ D; A strict global minimizer for f (x) on D if f (x⋆) < f (x) for all x ∈ D such that x = x⋆;

16 / 31

Optimization via Calculus

Definition Suppose f (x) is a real-valued function defined on a subset D of

• Rn. A point x⋆ in D is:

A global minimizer for f (x) on D if f (x⋆) ≤ f (x) for all x ∈ D; A strict global minimizer for f (x) on D if f (x⋆) < f (x) for all x ∈ D such that x = x⋆; A local minimizer for f (x) if there is a positive number δ such that f (x⋆) ≤ f (x) for all x ∈ D for which x in B(x⋆, δ);

17 / 31

Optimization via Calculus

Definition Suppose f (x) is a real-valued function defined on a subset D of

• Rn. A point x⋆ in D is:

A global minimizer for f (x) on D if f (x⋆) ≤ f (x) for all x ∈ D; A strict global minimizer for f (x) on D if f (x⋆) < f (x) for all x ∈ D such that x = x⋆; A local minimizer for f (x) if there is a positive number δ such that f (x⋆) ≤ f (x) for all x ∈ D for which x in B(x⋆, δ); A strict local minimizer for f (x) if there is a positive number δ such that f (x⋆) < f (x) for all x ∈ D for which x in B(x⋆, δ) and x = x⋆;

18 / 31

Optimization via Calculus

Definition Suppose f (x) is a real-valued function defined on a subset D of

• Rn. A point x⋆ in D is:

A global minimizer for f (x) on D if f (x⋆) ≤ f (x) for all x ∈ D; A strict global minimizer for f (x) on D if f (x⋆) < f (x) for all x ∈ D such that x = x⋆; A local minimizer for f (x) if there is a positive number δ such that f (x⋆) ≤ f (x) for all x ∈ D for which x in B(x⋆, δ); A strict local minimizer for f (x) if there is a positive number δ such that f (x⋆) < f (x) for all x ∈ D for which x in B(x⋆, δ) and x = x⋆; A critical point (also called a stationary point) of f (x) if the first partial derivatives of f (x) exist at x⋆ and ∂f

∂xi = 0, for

i = 1, 2, 3 . . . , n.

19 / 31

Optimization via Calculus

Theorem (Multivariable Taylor’s formula) Suppose that x, x⋆ are points in Rn and that f (x) is a real-valued function of n variables with continuous first and second partial derivatives on some open set containing the line segment [x⋆, x] = {w ∈ Rn : w = x⋆ + t(x − x⋆) , 0 ≤ t ≤ 1} joining x⋆ and

• x. Then, there exists a z ∈ [x⋆, x] such that

f (x) = f (x⋆) + ∇f (x⋆)T(x − x⋆) + 1 2(x − x⋆)THf (z)(x − x⋆)

20 / 31

Optimization via Calculus

Theorem (Local minimizer identification) Suppose that f (x) is a real-valued function for which all first partial derivatives of f (x) exist on a subset D ∈ Rn. If x⋆ is an interior point of D that is a local minimizer of f (x), then ∇f (x⋆) = 0.

21 / 31

Optimization via Calculus

Theorem (Classification of minimizers (maximizers)) Suppose that x⋆ is a critical point of a function f (x) with continuous first and second partial derivatives on Rn. Then: x⋆ is a global minimizer of f (x) if (x − x⋆)THf (z)(x − x⋆) ≥ 0 for all x ∈ Rn and all z ∈ [x⋆, x]; x⋆ is a strict global minimizer of f (x) if (x − x⋆)THf (z)(x − x⋆) > 0 for all x ∈ Rn such that x = x⋆ and for all z ∈ [x⋆, x]; x⋆ is a global maximizer of f (x) if (x − x⋆)THf (z)(x − x⋆) ≤ 0 for all x ∈ Rn and all z ∈ [x⋆, x]; x⋆ is a strict global maximizer of f (x) if (x − x⋆)THf (z)(x − x⋆) < 0 for all x ∈ Rn such that x = x⋆ and for all z ∈ [x⋆, x];

22 / 31

Optimization via Calculus

Practical ways to use the previous theorem: Conditions that involve the form (x − x⋆)THf (z)(x − x⋆), or in general vTAv, where A is a symmetric square matrix, call for methods to identify whether A (in our case the Hessian of the

• bjective function) is positive or negative (semi)definite.

23 / 31

Optimization via Calculus

Quadratic forms: Let A =      a11 a12 . . . a1n a21 a22 . . . a2n . . . . . . ... . . . an1 an2 . . . ann      . The quadratic form QA(x) = xTAx = a11x2

1 + a12x1x2 + a13x1x3 + . . . + aijxixj + . . . + aiix2 i + . . . + annx2 n.

24 / 31

Optimization via Calculus

Example Write the quadratic form associated with the following matrix: A =     −1 2 −3 2 1/2 −1 2 1/2 4 −3 −1 4 5     .

25 / 31

Optimization via Calculus

Determining whether a quadratic form QA(x) > 0 for all x ∈ Rn. Example in class . . . .

26 / 31