Optimization and Simulation Constrained optimization Michel Bierlaire Transport and Mobility Laboratory School of Architecture, Civil and Environmental Engineering Ecole Polytechnique F´ ed´ erale de Lausanne M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 1 / 72
The problem Outline The problem 1 Duality 2 Feasible directions 3 Optimality conditions 4 Convex constraints Lagrange multipliers: necessary conditions Lagrange multipliers: sufficient conditions Algorithms 5 Constrained Newton Interior point methods Augmented lagrangian Sequential quadratic programming M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 2 / 72
The problem Optimization: the problem x ∈ R n f ( x ) min subject to h ( x ) = 0 g ( x ) ≤ 0 X ⊆ R n x ∈ Modeling elements 1 Decision variables: x 2 Objective function: f : R n → R ( n > 0) 3 Constraints: equality: h : R → R m ( m ≥ 0) inequality: g : R n → R p ( p ≥ 0) X is a convex set M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 3 / 72
The problem The problem Assumptions x i , i = 1 , . . . , n , are continuous variables f , g and h are sufficiently differentiable Y = { x ∈ R n | h ( x ) = 0 , g ( x ) ≤ 0 and x ∈ X } is non empty Local minimum x ∗ ∈ Y is a local minimum of the above problem if there exists ε > 0 such that f ( x ∗ ) ≤ f ( x ) ∀ x ∈ Y such that � x − x ∗ � < ε. Global minimum x ∗ ∈ Y is a global minimum of the above problem if f ( x ∗ ) ≤ f ( x ) ∀ x ∈ Y . M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 4 / 72
Duality Outline The problem 1 Duality 2 Feasible directions 3 Optimality conditions 4 Convex constraints Lagrange multipliers: necessary conditions Lagrange multipliers: sufficient conditions Algorithms 5 Constrained Newton Interior point methods Augmented lagrangian Sequential quadratic programming M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 5 / 72
Duality Lagrangian Assume X = R n in the above problem Consider λ ∈ R m Consider µ ∈ R p Definition The function L : R n + m + p → R defined as f ( x ) + λ T h ( x ) + µ T g ( x ) L ( x , λ, µ ) = f ( x ) + � m i =1 λ i h i ( x ) + � p = j =1 µ j g j ( x ) is called the lagrangian function. M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 6 / 72
Duality Dual function Dual function The function q : R m + p → R defined as q ( λ, µ ) = min x ∈ R n L ( x , λ, µ ) is called the dual function of the optimization problem. Dual variables Parameters λ and µ are called dual variables. x are called primal variables. Bound on the optimal solution If x ∗ is a global minimum of the optimization problem, then, for any λ ∈ R m and any µ ∈ R , µ ≥ 0, we have q ( λ, µ ) ≤ f ( x ∗ ) . M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 7 / 72
Duality Dual problem Constrain the dual function to be bounded Let X q ⊆ R m + p be the domain of q , that is X q = { λ, µ | q ( λ, µ ) > −∞} Dual problem max λ,µ q ( λ, µ ) subject to µ ≥ 0 and ( λ, µ ) ∈ X q is called the dual problem of the original problem, which is called the primal problem in this context. M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 8 / 72
Duality Duality results Weak duality theorem Let x ∗ be a global minimum of the primal problem, and ( λ ∗ , µ ∗ ) a global maximum of the dual problem. Then, q ( λ ∗ , µ ∗ ) ≤ f ( x ∗ ) . Convexity-concavity of the dual problem The objective function of the dual problem is concave. The feasible set of the dual problem is convex. M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 9 / 72
Feasible directions Outline The problem 1 Duality 2 Feasible directions 3 Optimality conditions 4 Convex constraints Lagrange multipliers: necessary conditions Lagrange multipliers: sufficient conditions Algorithms 5 Constrained Newton Interior point methods Augmented lagrangian Sequential quadratic programming M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 10 / 72
Feasible directions Feasible directions Definitions x ∈ R n is a feasible point if it verifies the constraints Given x feasible, d is a feasible direction in x if there is η > 0 such that x + α d is feasible for any 0 ≤ α ≤ η . Convex constraints Let X ⊆ R n be a convex set, and x , y ∈ X , x � = y . The direction d = y − x is feasible in x . Moreover, for each 0 ≤ α ≤ 1, α x + (1 − α ) y is feasible. M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 11 / 72
Feasible directions Feasible directions Interior point Let X ⊆ R n Let x be an interior point, that is there exists ε > 0 such that � x − z � ≤ ε = ⇒ z ∈ X . Then, any direction d is feasible in x . M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 12 / 72
Feasible directions Feasible sequence Definition Consider the generic optimization problem Let x + ∈ R n be a feasible point The sequence ( x k ) k is said to be feasible in x + if lim k →∞ x k = x + , ∃ k 0 such that x k is feasible if k ≥ k 0 , x k � = x + for all k . M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 13 / 72
Feasible directions Feasible sequence Example One equality constraint h ( x ) = x 2 1 − x 2 = 0 , Feasible point: x + = (0 , 0) T Feasible sequence: 1 � � x k = k 1 k 2 M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 14 / 72
Feasible directions Feasible sequence • h ( x ) = x 2 1 − x 2 = 0 x 2 • • • • • • • • • • • • • • • • • • • x + = 0 -1 -0.5 0 0.5 1 x 1 M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 15 / 72
Feasible directions Feasible direction at the limit Main idea Consider the sequence of directions x k − x + d k = � x k − x + � , and take the limit. Directions d k are not necessarily feasible The sequence may not always converge Subsequences must then be considered M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 16 / 72
Feasible directions Feasible direction at the limit • d 1 h ( x ) = x 2 1 − x 2 = 0 x 2 d 2 d 3 • • • d • • • • • • • • • • • • • • • • x + = 0 -1 -0.5 0 0.5 1 x 1 M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 17 / 72
Feasible directions Feasible direction at the limit Example Constraint: h ( x ) = x 2 1 − x 2 = 0 Feasible point: x + = (0 , 0) T Feasible sequence: � � ( − 1) k x k = k 1 k 2 Sequence of directions: ( − 1) k k � � √ k 2 +1 d k = 1 k 2 +1 , √ Two feasible directions at the limit M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 18 / 72
Feasible directions Feasible direction at the limit h ( x ) = x 2 • 1 − x 2 = 0 d 1 x 2 d 2 d 3 d 4 • • d ′′ • d ′ • • • • • • • • • • • • • • • x + = 0 -1 -0.5 0 0.5 1 x 1 M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 19 / 72
Feasible directions Feasible direction at the limit Definition Consider the generic optimization problem Let x + ∈ R n be feasible Let ( x k ) k be a feasible sequence in x + Then, d � = 0 is a feasible direction at the limit in x + for the sequence ( x k ) k if there exists a subsequence ( x k i ) i such that x k i − x + d � d � = lim � x k i − x + � . i →∞ M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 20 / 72
Feasible directions Feasible direction at the limit Notes It is sometimes called a tangent direction. THe set of all tangent directions is called the tangent cone . Any feasible direction d is also a feasible direction at the limit, for the sequence x k = x + + 1 k d . M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 21 / 72
Feasible directions Linearized cone Definition Consider the generic optimization problem Let x + ∈ R n be feasible The set of directions d such that d T ∇ g i ( x + ) ≤ 0 , ∀ i = 1 , . . . , p such that g i ( x + ) = 0 , and d T ∇ h i ( x + ) = 0 , i = 1 , . . . , m , as well as their multiples α d , α > 0, is the linearized cone at x + . M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 22 / 72
Feasible directions Linearized cone h ( x ) = x 2 1 − x 2 = 0 x 2 d ′′ d ′ x + = 0 ∇ h ( x + ) -1 -0.5 0 0.5 1 x 1 M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 23 / 72
Feasible directions Linearized cone Theorem: Consider the generic optimization problem Let x + ∈ R n be feasible If d is a feasible direction at the limit at x + Then d belongs to the linearized cone at x + M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 24 / 72
Feasible directions Constraint qualification Definition: Consider the generic optimization problem Let x + ∈ R n be feasible The constraint qualification condition is verified if every direction in the linearized cone at x + is also in the tangent cone, that is, it is a feasible direction at the limit at x + . This is verified in particular if the constraints are linear, or if the gradients of the constraints active at x + are linearly independent. M. Bierlaire (TRANSP-OR ENAC EPFL) Optimization and Simulation 25 / 72
Recommend
More recommend
