Nonlinear Optimization I Dr. Thomas M. Surowiec Humboldt University - - PowerPoint PPT Presentation

Introduction Optimality Conditions

Nonlinear Optimization I

• Dr. Thomas M. Surowiec

Humboldt University of Berlin Department of Mathematics

Summer 2013

• Dr. Thomas M. Surowiec

BMS Couse NLO, Summer 2013

Introduction Optimality Conditions

Course Information

1

Lectures:

Tuesdays 15-17 RUD 26 1.304 Thursdays 11-13 RUD 1.013

2

Recitations: Thursdays 15-17 RUD 25 3.006

3

Office Hours: By appointment.

Office: RUD 2.425 Phone: x5497 Email: surowiec math.hu-berlin.de Website: http://www.mathematik.hu-berlin.de/ surowiec/

• Dr. Thomas M. Surowiec

BMS Couse NLO, Summer 2013

Introduction Optimality Conditions

Course Information & Grading

1

Lecture Notes, Slides.

Complete lecture notes (should be!) available at some point during the semester. Slides available following each lecture on my website. Slides = Lecture Notes − Proofs − Prose.

2

Homeworks:

1 Homework series per week with 4 problems. Problems to be solved on the board during the following recitation.

3

Final Exam: oral.

• Dr. Thomas M. Surowiec

BMS Couse NLO, Summer 2013

Introduction Optimality Conditions

Definition of an Optimization Problem

Definition (Minimization Problem) Let X ⊂ Rn an arbitrary set and f : X → R a continuous function. The problem is to find an x∗ ∈ X such that f(x∗) ≤ f(x), ∀x ∈ X. Alternate formulations: min f(x) subject to (s.t.) x ∈ X, or min

x∈X f(x)

• Dr. Thomas M. Surowiec

BMS Couse NLO, Summer 2013

Introduction Optimality Conditions

Examples of Optimization Problems

Basic Idea: Given observations or measurements of a system of interest, how can we determine certain intrinsic properties? Example Let M be a point of mass with mass m affixed to the end of a vertical spring. At equilibrium, M is located at the origin. K is the restoring force applied by spring to M upon displacement. For small (vertical) displacements y, the spring applies K = −ˆ ky (Hooke’s law).

• Dr. Thomas M. Surowiec

BMS Couse NLO, Summer 2013

Introduction Optimality Conditions

Examples of Optimization Problems

Example (cont.) ˆ k is an as yet unknown positive spring constant. y(t) := Position of M at time t. Ignoring damping and friction, Newton’s law states: m¨ y = −ˆ ky, (1) i.e. mass m times acceleration ¨ y equals the opposing force of the spring −ˆ ky. (1) is called “the undamped harmonic oscillator equation.” Usually, friction and damping forces behave proportionally to the velocity ( ˙ y) of M. We model with by −r ˙ y with fixed r > 0.

• Dr. Thomas M. Surowiec

BMS Couse NLO, Summer 2013

Introduction Optimality Conditions

Examples of Optimization Problems

Example (cont.) Together with (1), we obtain m¨ y + r ˙ y + ˆ ky = 0. (2) Setting c := r/m, k := ˆ k/m, (2) becomes: ¨ y + c ˙ y + ky = 0. (3) Assume at time t = 0: y(0) = y0 ∈ R, ˙ y(0) = 0. (4) Given endtime T > 0, we consider this initial boundary value problem (IVP) on the interval [0, T].

• Dr. Thomas M. Surowiec

BMS Couse NLO, Summer 2013

Introduction Optimality Conditions

Examples of Optimization Problems

Example (cont.) Suppose we do not know x := (c, k)T. For j := 1, . . . , N, we are given measurements

• yjN

j=1 of the spring

deviation at time instances tj = (j − 1)T/(N − 1). Let y(x; , t) be the solution of our IVP for a given x. By solving the unconstrained optimization problem min

x∈R2 f(x) := 1

2

N

• j=1

|y(x; tj) − y j|2, (5) we seek to determine the spring constant k and damping factor c. Note y(·, t) is differentiable wrt to x provided c2 − 4k = 0. (5) : “Nonlinear Least Squares Problem.”

• Dr. Thomas M. Surowiec

BMS Couse NLO, Summer 2013

Introduction Optimality Conditions

Examples of Optimization Problems

Basic Idea Deciding product capacity based on fixed and variable costs. Example x output quantity. Kv(x) variable costs, Kf(x) = c > 0 fixed costs. K(x) := Kv(x) + Kf(x), x ∈ R total costs. Normally one looks for an x∗, which minimizes total costs of K(x), i.e. x∗ = argmin {Kf(x) + Kv(x) s.t. x ∈ R} = argmin {Kv(x) s.t. x ∈ R} (6) In general, x∗ is not unique. We therefore write: x∗ ∈ argmin {Kv(x) s.t. x ∈ R} (7)

• Dr. Thomas M. Surowiec

BMS Couse NLO, Summer 2013

Introduction Optimality Conditions

Constrained Optimization

Particularly in the previous example, one often has constraints on x. When X Rn, we often have X = X1 ∩ X2 ∩ X3, where X1 =

• x ∈ Rn |ci(x) = 0, i ∈ I1
• X2 =
• x ∈ Rn |ci(x) ≤ 0, i ∈ I2
• X3 =
• x ∈ Rn |xi ∈ Z, i ∈ I3
• Ii, i = 1, 2, 3 are called index sets.

X1, X2, X3 are called equality, inequality, and integer constraints, respectively.

• Dr. Thomas M. Surowiec

BMS Couse NLO, Summer 2013

Introduction Optimality Conditions

Catagories of Optimization Problems

X set of discrete points → discrete or combinatorial optimization. Otherwise, continuous optimization. f, ci for any i is nondifferentiable → nonsmooth optimization. Example Nonsmooth Problems f(x) := || · ||2 (f nondifferentiable at 0). c(x) = |x| − x2 = 0. Optimizing optimization problems (game theory, optimal control, etc.)

• Dr. Thomas M. Surowiec

BMS Couse NLO, Summer 2013

Introduction Optimality Conditions

Notions of Solutions

Definition (Solution Types) Let f : X → R with X ⊂ Rn. The point x∗ ∈ X is called a

1

(strict) global minimizer of f (on X) if and only if f(x∗) ≤ f(x), (f(x∗) < f(x)) ∀x ∈ X \ {x∗} .

2

(strict) local minimizer of f (on X) if and only if there exists a neighborhood U of x∗ such that f(x∗) ≤ f(x), (f(x∗) < f(x)) ∀x ∈ (X ∩ U) \ {x∗} . The optimal objective value of f(x∗) is called a (strict) local minimum.

• Dr. Thomas M. Surowiec

BMS Couse NLO, Summer 2013

Introduction Optimality Conditions

Stationary Points

For f ∈ C1, we denote its gradient by ∇f(x) = ∂f ∂x1 (x), . . . , ∂f ∂xn (x) T . If f : X → R is directionally differentiable, then its directional derivative at x in direction d is denoted by f ′(x; d) := lim

α↓0

f(x + αd) − f(x) α . Definition (Stationarity) Let X ⊂ Rn be an open set and f : X → R directionally differentiable. The point x∗ ∈ X is called a stationary point of f, if ∇f(x∗) = 0. holds true.

• Dr. Thomas M. Surowiec

BMS Couse NLO, Summer 2013

Introduction Optimality Conditions

Fermat’s Theorem

Theorem Let X ⊂ Rn be an open set and f : X → R a continuously differentiable

• function. If x∗ ∈ X is a local minimizer of f (on X), then

∇f(x∗) = 0. i.e. x∗ is a stationary point. Proof. On the board.

• Dr. Thomas M. Surowiec

BMS Couse NLO, Summer 2013

Introduction Optimality Conditions

A Fair Warning

Fermat’s Theorem, regardless of what variant, is a first order necessary condition. Example Let f(x) = −x2 with x∗ = 0. Here, ∇f(x∗) = −2x∗ = 0, but x∗ is in fact a global maximizer of f. In the previous example, ∇2f(x∗) = −2 indicates that f is concave (down). Hence, x∗ cannot be even a local minimum. This leads to a general result.

• Dr. Thomas M. Surowiec

BMS Couse NLO, Summer 2013

Introduction Optimality Conditions

Continuity of Eigenvalues

Lemma Let Sn be the vector space of symmetrical n × n-matrices. For A ∈ Sn let λ(A) ∈ R be the smallest eigenvalue of A. Then the following estimate holds true: |λ(A) − λ(B)| ≤ ||A − B|| for all A, B ∈ Sn In other words, the mapping that takes a real symmetric matrix to its smallest eigenvalue in Lipschitz continuous with modulus 1.

• Dr. Thomas M. Surowiec

BMS Couse NLO, Summer 2013

Introduction Optimality Conditions

An Important Implication

Lemma 12 allows us to make the following observation: f ∈ C2 ⇒ ∇2f : Rn → Rn×n continuous wrt some matrix norm || · ||. ∇2f is symmetric. ∇2f(x∗) positive definite implies ∇2f(x) pos. def. for ||x − x∗|| suff. small. In general: Small perturbations (in Sn!) of symmetric pos. def. matrices remain pos. def.

• Dr. Thomas M. Surowiec

BMS Couse NLO, Summer 2013

Introduction Optimality Conditions

Second-Order Necessary Optimality Conditions

Theorem Let X ⊂ Rn be open and f : X → R be twice continuously differentiable. If x∗ ∈ X is a local minimizer of f on (X), then ∇f(x∗) = 0 and the Hessian ∇2f(x∗) is positive semi-definite. Proof. On the board. Question: What happens if we try to apply this as a sufficient condition? Example f(x) = x2

1 − x4 2, x∗ = (0, 0)T.

• Dr. Thomas M. Surowiec

BMS Couse NLO, Summer 2013

Introduction Optimality Conditions

Second-Order Sufficient Optimality Conditions

Theorem Let X ⊂ Rn be open and f : X → R twice continuously differentiable. If

1

∇f(x∗) = 0 and

2

∇2f(x∗) is positive definite, then x∗ is a strict local minimizer of f (on X). Proof. On the board.

• Dr. Thomas M. Surowiec

BMS Couse NLO, Summer 2013