Computational Optimization Constrained Optimization Algorithms - PowerPoint PPT Presentation

Computational Optimization Constrained Optimization Algorithms – Feasible Descent Methods Continued

Next Problem Consider Next Hardest Problem min ( ) f x ≥ . . s t Ax b How could we adapt gradient projection or other linear equality constrained techniques to this problem?

Active Set Methods MW 16.5 Change one item of working set at a time

Interior point algorithms MW 16.6 Traverse interior of set (a little more later)

Gradient Projection MW 16.7 Change many elements of working set at once

Gradient Projection Method NW 16.7 For equality constrained approach we projected the gradient back to the feasible region. We could do this for − ∇ ( xk ) f Inequalities too if xk projection is cheap 1 + min ' ' x Qx x c 2 ≤ ≤ . . s t l x u

Projection for bounds constraints Projection is closest point in the set to x 1 − ≤ ≤ 2 � min || || . . x s s t s u s 2 < ⎧ � � if x i i i ⎪ = ≤ ≤ � ⎨ s x if x u x u i i i i i i � ⎪ > ⎩ u if x u i i i

Cauchy Point x − Take a step tg i i The projected step is a function of t = − ( ) ( ) x t P x tg i i Cauchy point finds t that minimizes ( ( )) q x t e.g. An exact linesearch along the projected direction

Plus: Cauchy point can change many constraints in working set g x(t) xk

Gradient Projection Method for QP NW 16.5 Start with feasible x0 For x = 0,1,2,… if xk satisfies KKT then optimal Set x =xk and find cauchy point xc; x k+1 is an approximate solution of QP using active set of xc fixed and rest feasible. approximation just needs to find feasible decrease. End;

KKT have nice form min ( ) f x KKT of ≤ ≤ . . s t l x u Lagrangian is λ α = − λ − − α − ( , , ) ( ) '( ) ( ) L x f x x l u x Primal Feasibility ≤ ≤ l x u Dual feasibility ∇ − λ + α = λ ≥ α ≥ ( ) 0 0, 0 f x Complimentarity λ − = α − = = ( ) 0 ( ) 0 1,.., x l u x i n i i i i i i

KKT have nice form ∂ ( ) f x Need = α − λ α λ ≥ , 0 ∂ i i i i x ∂ ∂ i ( ) ( ) f x f x ≥ α = λ = 0, , 0 So if ∂ ∂ i i x x i i ∂ ∂ ( ) ( ) f x f x < α = λ = − 0, 0, if ∂ ∂ i i x x i i So if complimentarity holds λ − = α − = = ( ) 0 ( ) 0 1,.., x l u x i n i i i i i i Point is optimal

Active Set Summary Active set QP methods widely used Simplex method is an active set method for LP Can do hot start (start from good solution) Projection method effective when constraints have easy to compute projections. Gradient methods can still be slow Interior methods usually better for LP and QP but no hot start.