MATH529 Fundamentals of Optimization Fundamentals of Constrained - PowerPoint PPT Presentation

MATH529 – Fundamentals of Optimization Fundamentals of Constrained Optimization II Marco A. Montes de Oca Mathematical Sciences, University of Delaware, USA 1 / 33

Multiple equality constraints: Example A motor company makes cars, trucks and vans. Its revenue is R ( c , t , v ), where c , t , v are the number of cars, trucks, vans (respectively) it produces per year. Suppose production is constrained by the amount of steel available, and by the amount of aluminum available. Assume each car, truck, van needs s c , s t , s v units of steel, respectively, and a c , a t , a v units of aluminum, respectively. Suppose S units of steel and A of aluminum are available. Write the constraints as equations. Use Lagrange multipliers to determine optimality conditions. What are the interpretations of each Lagrange multiplier? 2 / 33

Multiple equality constraints Illustrations by Steuard Jensen. ( http://www.slimy.com/~steuard/teaching/tutorials/Lagrange.html ) 3 / 33

Multiple equality constraints The Lagrangian can be extended to simultaneously consider multiple constraints. For example, let f ( x ) be subject to: g ( x ) = c and h ( x ) = d . The Lagrangian function may be defined as follows: L ( x , λ, µ ) = f ( x ) + λ ( c − g ( x )) + µ ( d − h ( x )). The new first-order condition is now: c − g ( x ) = 0, d − h ( x ) = 0, ∇ f ( x ) − λ ∇ g ( x ) − µ ∇ h ( x ) = 0. 4 / 33

Second-order conditions Let z be the objective function, that is, let z = f ( x , y ). Let also g ( x , y ) = c be a constraint that a solution to the optimization problem must satisfy. Then, dg = g x dx + g y dy = 0, which means that d x and d y are not independent. So, we can find dy by fixing dx , that is, dy = − g x g y dx . Note that since g x and g y both depend on x and y , so does dy . Now, d 2 z = d ( dz ) = ∂ ( dz ) ∂ x dx + ∂ ( dz ) ∂ y dy = 5 / 33

Second-order conditions Let z be the objective function, that is, let z = f ( x , y ). Let also g ( x , y ) = c be a constraint that a solution to the optimization problem must satisfy. Then, dg = g x dx + g y dy = 0, which means that d x and d y are not independent. So, we can find dy by fixing dx , that is, dy = − g x g y dx . Note that since g x and g y both depend on x and y , so does dy . Now, d 2 z = d ( dz ) = ∂ ( dz ) ∂ x dx + ∂ ( dz ) ∂ y dy = ∂ ( f x dx + f y dy ) dx + ∂ ( f x dx + f y dy ) dy = ∂ x ∂ y 6 / 33

Second-order conditions Let z be the objective function, that is, let z = f ( x , y ). Let also g ( x , y ) = c be a constraint that a solution to the optimization problem must satisfy. Then, dg = g x dx + g y dy = 0, which means that d x and d y are not independent. So, we can find dy by fixing dx , that is, dy = − g x g y dx . Note that since g x and g y both depend on x and y , so does dy . Now, d 2 z = d ( dz ) = ∂ ( dz ) ∂ x dx + ∂ ( dz ) ∂ y dy = ∂ ( f x dx + f y dy ) dx + ∂ ( f x dx + f y dy ) dy = ∂ x ∂ y ∂ ( f x dx ) dx + ∂ ( f y dy ) dx + ∂ ( f x dx ) dy + ∂ ( f y dy ) dy ∂ x ∂ x ∂ y ∂ y 7 / 33

Second-order conditions ∂ ( f x dx ) dx + ∂ ( f y dy ) dx + ∂ ( f x dx ) dy + ∂ ( f y dy ) dy = ∂ x ∂ x ∂ y ∂ y 8 / 33

Second-order conditions ∂ ( f x dx ) dx + ∂ ( f y dy ) dx + ∂ ( f x dx ) dy + ∂ ( f y dy ) dy = ∂ x ∂ x ∂ y ∂ y f xx ( dx ) 2 + 9 / 33

Second-order conditions ∂ ( f x dx ) dx + ∂ ( f y dy ) dx + ∂ ( f x dx ) dy + ∂ ( f y dy ) dy = ∂ x ∂ x ∂ y ∂ y ∂ ( dy ) f xx ( dx ) 2 +( f yx dy + f y ∂ x ) dx + 10 / 33

Second-order conditions ∂ ( f x dx ) dx + ∂ ( f y dy ) dx + ∂ ( f x dx ) dy + ∂ ( f y dy ) dy = ∂ x ∂ x ∂ y ∂ y ∂ ( dy ) f xx ( dx ) 2 +( f yx dy + f y ∂ x ) dx + f xy dxdy + 11 / 33

Second-order conditions ∂ ( f x dx ) dx + ∂ ( f y dy ) dx + ∂ ( f x dx ) dy + ∂ ( f y dy ) dy = ∂ x ∂ x ∂ y ∂ y ∂ ( dy ) ∂ ( dy ) f xx ( dx ) 2 +( f yx dy + f y ∂ x ) dx + f xy dxdy +( f yy dy + f y ∂ y ) dy = 12 / 33

Second-order conditions ∂ ( f x dx ) dx + ∂ ( f y dy ) dx + ∂ ( f x dx ) dy + ∂ ( f y dy ) dy = ∂ x ∂ x ∂ y ∂ y ∂ ( dy ) ∂ ( dy ) f xx ( dx ) 2 +( f yx dy + f y ∂ x ) dx + f xy dxdy +( f yy dy + f y ∂ y ) dy = f xx ( dx ) 2 + 2 f xy dxdy + f yy ( dy ) 2 + f y ( ∂ ( dy ) ∂ x dx + ∂ ( dy ) ∂ y dy ) = 13 / 33

Second-order conditions ∂ ( f x dx ) dx + ∂ ( f y dy ) dx + ∂ ( f x dx ) dy + ∂ ( f y dy ) dy = ∂ x ∂ x ∂ y ∂ y ∂ ( dy ) ∂ ( dy ) f xx ( dx ) 2 +( f yx dy + f y ∂ x ) dx + f xy dxdy +( f yy dy + f y ∂ y ) dy = f xx ( dx ) 2 + 2 f xy dxdy + f yy ( dy ) 2 + f y ( ∂ ( dy ) ∂ x dx + ∂ ( dy ) ∂ y dy ) = f xx ( dx ) 2 + 2 f xy dxdy + f yy ( dy ) 2 + f y ( d ( dy )) = � f xx � � dx � f xy + f y ( d 2 y ) ( dx dy ) f yx f yy dy 14 / 33

Second-order conditions ∂ ( f x dx ) dx + ∂ ( f y dy ) dx + ∂ ( f x dx ) dy + ∂ ( f y dy ) dy = ∂ x ∂ x ∂ y ∂ y f xx ( dx ) 2 + ( f yx dy + f y ∂ ( dy ) ∂ ( dy ) ∂ x ) dx + f xy dxdy + ( f yy dy + f y ∂ y ) dy = f xx ( dx ) 2 + 2 f xy dxdy + f yy ( dy ) 2 + f y ( ∂ ( dy ) ∂ x dx + ∂ ( dy ) ∂ y dy ) = f xx ( dx ) 2 + 2 f xy dxdy + f yy ( dy ) 2 + f y ( d ( dy )) = � f xx � � dx � f xy + f y ( d 2 y ) ( dx dy ) f yx f yy dy � �� � Quadratic form associated with unconstrained problem 15 / 33

Second-order conditions ∂ ( f x dx ) dx + ∂ ( f y dy ) dx + ∂ ( f x dx ) dy + ∂ ( f y dy ) dy = ∂ x ∂ x ∂ y ∂ y f xx ( dx ) 2 + ( f yx dy + f y ∂ ( dy ) ∂ ( dy ) ∂ x ) dx + f xy dxdy + ( f yy dy + f y ∂ y ) dy = f xx ( dx ) 2 + 2 f xy dxdy + f yy ( dy ) 2 + f y ( ∂ ( dy ) ∂ x dx + ∂ ( dy ) ∂ y dy ) = f xx ( dx ) 2 + 2 f xy dxdy + f yy ( dy ) 2 + f y ( d ( dy )) = � f xx � � dx � f xy f y ( d 2 y ) ( dx dy ) + f yx f yy dy � �� � New 1st degree term in constrained case 16 / 33

Second-order conditions To eliminate this 1st degree term, recall that dy = − g x g y dx , so ∂ x ( − g x ∂ y ( − g x d 2 y = ∂ g y dx ) dx + ∂ g y dx ) dy = 17 / 33

Second-order conditions To eliminate this 1st degree term, recall that dy = − g x g y dx , so ∂ x ( − g x ∂ y ( − g x d 2 y = ∂ g y dx ) dx + ∂ g y dx ) dy = − g y g xx ( g y ) 2 dxdx + g yx g x ( g y ) 2 dxdx − g y g xy ( g y ) 2 dxdy + g yy g x ( g y ) 2 dxdy = 18 / 33

Second-order conditions To eliminate this 1st degree term, recall that dy = − g x g y dx , so ∂ x ( − g x ∂ y ( − g x d 2 y = ∂ g y dx ) dx + ∂ g y dx ) dy = − g y g xx ( g y ) 2 dxdx + g yx g x ( g y ) 2 dxdx − g y g xy ( g y ) 2 dxdy + g yy g x ( g y ) 2 dxdy = − g xx g y dxdx + g yx g x ( g y ) 2 dxdx − g xy g y dxdy + g yy g x ( g y ) 2 dxdy = 19 / 33

Second-order conditions To eliminate this 1st degree term, recall that dy = − g x g y dx , so ∂ x ( − g x ∂ y ( − g x d 2 y = ∂ g y dx ) dx + ∂ g y dx ) dy = − g y g xx ( g y ) 2 dxdx + g yx g x ( g y ) 2 dxdx − g y g xy ( g y ) 2 dxdy + g yy g x ( g y ) 2 dxdy = − g xx g y dxdx + g yx g x ( g y ) 2 dxdx − g xy g y dxdy + g yy g x ( g y ) 2 dxdy = g y dxdx − 2 g xy − g xx g y dxdy − gyy g y dydy . 20 / 33

Second-order conditions Substituting d 2 y back in the expression for d 2 z : d 2 z = � � � � � � f xx − f y f xy − f y f yy − f y g y g xx dxdx + 2 g y g xy dxdy + g y g yy dydy 21 / 33

Second-order conditions Substituting d 2 y back in the expression for d 2 z : d 2 z = � � � � � � f xx − f y f xy − f y f yy − f y g y g xx dxdx + 2 g y g xy dxdy + g y g yy dydy Finally, since f y g y = λ (recall the derivation of the Lagrange multiplier equation), we have: L xx = f xx − λ g xx L xy = L yx = f xy − λ g xy L yy = f yy − λ g yy d 2 z = L xx dxdx + 2 L xy dxdy + L yy dydy Since this is a quadratic form, we can now establish a determinantal test. 22 / 33

Second-order conditions What we are looking for: Necessary conditions For a maximum of z : d 2 z should be negative semidefinite, subject to dg = 0 . For a minimum of z : d 2 z should be positive semidefinite, subject to dg = 0 . Sufficient conditions For a maximum of z : d 2 z should be negative definite, subject to dg = 0 . For a minimum of z : d 2 z should be positive definite, subject to dg = 0 . 23 / 33

Second-order conditions Problem: What are the conditions for the sign of definiteness of q = au 2 + 2 huv + bv 2 subject to α u + β v = 0 ? 24 / 33

Second-order conditions Since α u + β v = 0, v = − α u β , thus q = au 2 + 2 hu ( − α u β ) 2 = ( a β 2 − 2 h αβ + b α 2 ) u 2 β ) + b ( − α u β 2 . Therefore, the sign of q depends on the sign of a β 2 − 2 h αβ + b α 2 , which is equal to the negative of: � � 0 α β � � � � α a h � � � � β h b � � 25 / 33

Second-order conditions Thus, q is positive definite subject to α u + β v = 0, iff � � 0 α β � � � � α < 0 and a h � � � � β h b � � � � 0 α β � � � � q is negative definite subject to α u + β v = 0, iff α a h > 0 � � � � β h b � � 26 / 33