Augmented Lagrangians and Decomposition in Convex and Nonconvex - - PowerPoint PPT Presentation

Augmented Lagrangians and Decomposition in Convex and Nonconvex Programming

Terry Rockafellar University of Washington, Seattle

DIMACS Workshop on ADMM and Proximal Splitting Methods in Optimization Rutgers University, New Brunswick NJ June 11-13, 2018

Extended Optimization Model for Decomposition Work

Ingredients: mappings Fj : I Rnj → I Rm, just C1 or C2 for j = 1, . . . , q, functions fj : I Rnj → (−∞, ∞], just lsc for j = 1, . . . , q, function g : I Rm → (−∞, ∞], lsc, convex, pos. homogeneous subspace S ⊂ I Rn = I Rn1 × · · · × I Rnq with complement S⊥ Problem minimize q

j=1 fj(xj) + g

q

j=1 Fj(xj)

• ver (x1, . . . , xq) ∈ S

Specializations of the coupling term: g(u) = δK(u) for a closed convex cone K, or g(u) = ||u||p Specializations of the coupling space: S =

• (x1, . . . , xq)
• x1 = · · · = xq
• , for the splitting version

S taken to be all of I Rn (thereby “dropping out”), S⊥ = {0} Convex case: fj convex and Fj = Aj affine is just one possiblity

Reformulation With Enhanced Separability

Expansion Lemma g q

j=1 Fj(xj)

• ≤ α

⇐ ⇒ ∃ uj ∈ I Rm for j = 1, . . . , q such that q

j=1 uj = 0 and q j=1 g

• Fj(xj) + uj
• ≤ α

Extended coupling space: now in I Rn × [I Rm|q S =

• (x1, . . . , xq, u1, . . . , uq)
• (x1, . . . , xq) ∈ S, q

j=1 uj = 0

• ,

S

⊥ =

• (v1, . . . , vq, y1, . . . , yq)
• (v1, . . . , vq) ∈ S⊥, y1 = · · · = yq
• Expanded problem (equivalent)

min q

j=1

• fj(xj) + g(Fj(xj) + uj)
• ver (x1, . . . , xq, u1, . . . , uq) ∈ S

− → separability achieved in the objective: ϕ(x1, . . . , xq, u1, . . . , uq) = ϕ1(x1, u1) + · · · + ϕq(xq, uq)

Linkage Problems and Subgradients

Optimization goal minimize some lsc function ϕ on a subspace S First-order condition for local optimality: ¯ w ∈ S and ∃¯ z ∈ ∂ϕ( ¯ w) such that ¯ z ∈ S⊥ smooth case: ¯ z = ∇ϕ( ¯ w) ⊥ S Linkage problem — in terms of the subgradient mapping ∂ϕ given ϕ and S, find such a pair ( ¯ w, ¯ z) ∈ [gph ∂ϕ] ∩ [S × S⊥] Regular subgradients: notation ¯ z ∈ ∂ϕ( ¯ w) ϕ(w) ≥ ϕ( ¯ w) + ¯ z·(w − ¯ w) + o(||w − ¯ w||) General subgradients: notation ¯ z ∈ ∂ϕ( ¯ w) ∃ zν → ¯ z with zν ∈ ∂ϕ(wν), wν → ¯ w, ϕ(wν) → ϕ( ¯ w)

Local Elicitation of Convexity/Monotonicity

Key observation minimizing ϕ on over S ← → minimizing ϕe = ϕ + e

2d2 S over S

dS(w) = distance of w from S, e = elicitation parameter Second-order variational sufficient condition: for ¯ z ∈ ∂ϕ( ¯ w) ∃ e ≥ 0, ε > 0, open convex nbhd W × Z of ( ¯ w, ¯ z), and lsc convex ψ on W such that gph ∂ψ coincides in W × Z with gph Te,ε =

• (w, z) ∈ gph ∂ϕe
• ϕe(w) ≤ ϕe( ¯

w) + ε

• and, on that common set, furthermore ψ(w) = ϕe(w)

Example: ϕ ∈ C2, ¯ z = ∇ϕ( ¯ w), ∇2ϕ( ¯ w) pos.definite relative to S Criterion for local max monotonicity This condition = ⇒ Te,ε is max monotone around ( ¯ w, ¯ z), and it is ⇐ ⇒ when ¯ z is a regular subgradient of ϕ at ¯ w

Progressive Decoupling of Linkages

for finding a local minimizer ¯ w of ϕ on the subspace S ⊂ I RN Algorithm with parameters r > e ≥ 0 In iteration ν, having wν ∈ S and zν ∈ S⊥, find

• wν = (local) argminw∈I

RN

• ϕ(w) − zν · w + r

2||w − wν||2

and update by wν+1 = PS( wν), zν+1 = zν− (r − e)[ wν− wν+1] − → builds on Spingarn’s method and Pennanen’s PPA localization Convergence Theorem (for e high enough) Let ( ˜ w, ˜ z) satisfy the first-order condition and the second-order variational condition. Then ∃ nbhd W × Z of ( ˜ w, ˜ z) such that if (w0, z0) ∈ W × Z then (wν, zν) ∈ W × Z with wν = unique local minimizer on W , and (wν, zν) → some ( ¯ w, ¯ z) satisfying the first-order condition, such that ¯ w = local argmin of ϕ on W ∩ S

Application to the Expanded Decomposition Model

min q

j=1[ fj(xj) + g(Fj(xj) + uj) ] over (x1, . . . , xq, u1, . . . , uq) ∈ S

S =

• (x1, . . . , xq, u1, . . . , uq)
• (x1, . . . , xq) ∈ S, q

j=1 uj = 0

• ,

S

⊥=

• (v1, . . . , vq, y1, . . . , yq)
• (v1, . . . , vq) ∈ S⊥, y1 = · · · = yq
• Algorithm (with parameters r > e ≥ 0)

(xν

1 , . . . , xν q , uν 1, . . . , uν q) ∈ S,

(vν

1 , . . . , vν q , yν, . . . , yν) ∈ S ⊥ −

→ determine ( xν

j ,

uν

j ) as the local minimizer of ϕν j (xj, uj) =

fj(xj) + g(Fj(xj) + uj) − vν

j ·xj − yν·uj + r 2||xj − xν j ||2 + r 2||uj − uν j ||2

Then let uν = 1

q

q

j=1

uν

j

and update by (xν+1

1

, . . . , xν+1

q

) = PS( xν

j , . . . ,

xν

j ),

uν+1

j

= uν

j −

uν vν+1

j

= vν

j − (r − e)[

xν

j − xν+1 j

], yν+1 = yν − (r − e) uν Convergence: local under local optimality conditions as above

Implementation with Augmented Lagrangians

Auxiliary problems: min fj(xj)+g(F ν

j (xj)), F ν j (xj) = Fj(xj)+uν j

Note: g is lsc convex pos.homog., so g∗ = δY for some Y ⊂ I Rm Examples: g = δK for cone K yields Y = polar cone K ∗ g = || · ||p yields Y = unit ball for dual norm || · ||q Associated Lagrangians: Lν

j (xj, yj) = fj(xj) + yj·F ν j (xj) − δY (yj)

Augmented Lagrangians (with parameter r > 0): Lν

j,r(xj, yj) = fj(xj) + yj·F ν j (xj) + r 2||F ν j (xj)||2 − 1 2r d2 Y

• yj + rF ν

j (xj)

• Algorithm in condensed form

From (xν

1 , . . . , xν q ) ∈ S, (vν 1 , . . . , vν q ) ∈ S⊥, uν 1 + · · · + uν q = 0, yν,

get

• xν

j = (local) argminxj

• Lν

j,r(xj, yν) − vν j ·xj + r 2||xj − xν j ||2

then take uν

j = ∇yLν j,r(xν+1 j

, yν) and update just as before augmented Lagrangians can furthermore help with elicitation

References

[1] J.E. Spingarn (1985) “Applications of the metiond of partial inverses to convex programming: decomposition,” Mathematical Programming 32, 199–221. [2]

• T. Pennanen (2002) “Local convergence of the proximal

point algorithm and multiplier methods without monotonicity,” Mathematics of Operations Research 27, 170–191. [3] R.T. Rockafellar (2017) “Progressive decoupling of linkages in monotone variational inequalities and convex optimization” [4] R.T. Rockafellar (2018) “Variational convexity and local monotonicity of subgradient mappings” [5] R.T. Rockafellar (2018) “Progressive decoupling of linkages in optimization and variational inequalities with elicitable convexity

• r monotonicity”

nearly finished website: www.math.washington.edu/∼rtr/mypage.html