Optimization with side constraints Philipp Warode October 2, 2019 - PowerPoint PPT Presentation

Optimization with side constraints Philipp Warode October 2, 2019 Mathematics Preparatory Course 2019 – Philipp Warode

Optimization with side constraints We want to maximize max − x 2 − y 2 with the side constraint x + y = 1 Mathematics Preparatory Course 2019 – Philipp Warode

Simple solution Reduce numbers of variables by inserting side constraints in the objective function Example: Side constraint: x + y = 1 ⇔ y = 1 − x Objective function: max − x 2 − y 2 = max − x 2 − ( 1 − x ) 2 Not always possible, e.g. if the side constraint can’t be solved uniquely: x 2 + y 2 = 1 Mathematics Preparatory Course 2019 – Philipp Warode

Lagrange Multipliers Mathematics Preparatory Course 2019 – Philipp Warode

Lagrange Multipliers y 9 / 4 8 / 4 7 / 4 6 / 4 5 / 4 4 / 4 3 / 4 2 / 4 1 / 4 x f ( x , y ) = − x 2 − y 2 g ( x , y ) = x + y − 1 = 0 Mathematics Preparatory Course 2019 – Philipp Warode

Lagrange Multipliers y ∇ f 9 / 4 8 / 4 7 / 4 ∇ g 6 / 4 5 / 4 4 / 4 3 / 4 2 / 4 1 / 4 x f ( x , y ) = − x 2 − y 2 g ( x , y ) = x + y − 1 = 0 In the maximum, ∇ f (gradient of the objective) and ∇ g (gradient of the constraint) are linear dependent Mathematics Preparatory Course 2019 – Philipp Warode

Lagrange Multipliers y ∇ f 9 / 4 8 / 4 7 / 4 ∇ g 6 / 4 5 / 4 4 / 4 3 / 4 2 / 4 1 / 4 x f ( x , y ) = − x 2 − y 2 g ( x , y ) = x + y − 1 = 0 In the maximum, ∇ f (gradient of the objective) and ∇ g (gradient of the constraint) are linear dependent Necessary condition for local optimum: ∇ f = λ ∇ g Mathematics Preparatory Course 2019 – Philipp Warode

Lagrange Multipliers Theorem Let x ( 0 ) = ( x ( 0 ) n , . . . , x ( 0 ) n ) be an optimal solution of max f ( x ) s.t. g j ( x ) = 0 ∀ j ∈ { 1 , . . . , m } and f , g partially differentiable with ∇ g j ( x ( 0 ) ) � = 0 for j = 1 , . . . , m then there are λ 1 , . . . , λ m such that m � ∇ f ( x ( 0 ) ) + λ j ∇ g j ( x ( 0 ) ) = 0 . j = 1 Mathematics Preparatory Course 2019 – Philipp Warode

Lagrange Multipliers 1 Create Lagrange function m � L ( x , λ ) := f ( x ) + λ j g j ( x ) j = 1 Mathematics Preparatory Course 2019 – Philipp Warode

Lagrange Multipliers 1 Create Lagrange function m � L ( x , λ ) := f ( x ) + λ j g j ( x ) j = 1 2 Compute the partial derivatives m � ∂ L ( x , λ ) = ∂ ∂ f ( x ) + g j ( x ) λ j ∂ x i ∂ x i ∂ x i j = 1 ∂ L ( x , λ ) = g j ( x ) ∂λ j Mathematics Preparatory Course 2019 – Philipp Warode

Lagrange Multipliers 1 Create Lagrange function m � L ( x , λ ) := f ( x ) + λ j g j ( x ) j = 1 2 Compute the partial derivatives m � ∂ L ( x , λ ) = ∂ ∂ f ( x ) + g j ( x ) λ j ∂ x i ∂ x i ∂ x i j = 1 ∂ L ( x , λ ) = g j ( x ) ∂λ j 3 Solve the system of equations ∂ L ( x , λ ) = 0 for all i and ∂ L ( x , λ ) = 0 for all j ∂ x i ∂λ j Mathematics Preparatory Course 2019 – Philipp Warode