Duality (I) Lijun Zhang zlj@nju.edu.cn http://cs.nju.edu.cn/zlj - - PowerPoint PPT Presentation

Duality (I)

Lijun Zhang zlj@nju.edu.cn http://cs.nju.edu.cn/zlj

Outline

 The Lagrange Dual Function

 The Lagrange Dual Function  Lower Bound on Optimal Value  The Lagrange Dual Function and Conjugate Functions

 The Lagrange Dual Problem

 Making Dual Constraints Explicit  Weak Duality  Strong Duality and Slater’s Constraint Qualification

Optimization Problems

 Standard Form

 Domain is nonempty  Denote the optimal value by

∗

 We do not assume the problem is convex

𝒠 dom 𝑔

∩

• dom ℎ
• min

𝑔

𝑦

• s. t.

𝑔

𝑦 0,

𝑗 1, . . . , 𝑛 ℎ 𝑦 0 𝑗 1, . . . , 𝑞 1

 The Lagrangian

• 
• 

: the Lagrange multiplier associated with

the -th inequality constraint  𝜉: the Lagrange multiplier associated with the -th equality constraint

•  Vectors

and : dual variables or Lagrange multiplier vectors

The Lagrangian

𝑀 𝑦, 𝜇, 𝜉 𝑔

𝑦 𝜇𝑔 𝑦 𝜉ℎ 𝑦

The Lagrange Dual Function



•  When

is unbounded below in ,  is concave

 𝑕 is the pointwise infimum of a family of affine functions of 𝜇, 𝑤

 It is unconstrained

𝑕 𝜇, 𝜉 inf

∈𝒠 𝑀 𝑦, 𝜇, 𝜉

inf

∈𝒠 𝑔 𝑦 𝜇𝑔 𝑦 𝜉ℎ 𝑦

Outline

 The Lagrange Dual Function

 The Lagrange Dual Function  Lower Bound on Optimal Value  The Lagrange Dual Function and Conjugate Functions

 The Lagrange Dual Problem

 Making Dual Constraints Explicit  Weak Duality  Strong Duality and Slater’s Constraint Qualification

Lower Bounds on

 For any and any  Proof

 is a feasible point for original problem  Since  Therefore

𝑕 𝜇, 𝜉 𝑞∗ 𝑔

𝑦

0, ℎ 𝑦 𝜇𝑔

𝑦

𝜉ℎ 𝑦

• 𝑀 𝑦

, 𝜇, 𝜉 𝑔

𝑦

𝜇𝑔

𝑦

𝜉ℎ 𝑦

• 𝑔

𝑦

Lower Bounds on

 For any and any  Proof

 Hence  Note that

• for any feasible

 Discussions

 The lower bound is vacuous, when 𝑕 𝜇, 𝜉 ∞  It is nontrivial only when 𝜇 ≽ 0, 𝜇, 𝜉 ∈ dom 𝑕  Dual feasible: 𝜇, 𝜉 with 𝜇 ≽ 0, 𝜇, 𝜉 ∈ dom 𝑕 𝑕 𝜇, 𝜉 𝑞∗ 𝑕 𝜇, 𝜉 inf

∈𝒠 𝑀 𝑦, 𝜇, 𝜉 𝑀 𝑦

, 𝜇, 𝜉 𝑔

𝑦

Example

 A Simple Problem with

 Lower bound from a dual feasible point

 Solid curve: objective function 𝑔

•  Dashed curve: constraint function 𝑔
•  Feasible set: 0.46, 0.46 (indicated by the two

dotted vertical lines)

Example

 A Simple Problem with

 Lower bound from a dual feasible point

 Optimal point and value: 𝑦∗ 0.46, 𝑞∗ 1.54  Dotted curves: 𝑀 𝑦, 𝜇 for 𝜇 0.1, 0.2, … , 1.0.

• Each has a minimum value smaller than 𝑞∗ as on

the feasible set (and for 𝜇 0), 𝑀 𝑦, 𝜇 𝑔

𝑦

𝑀 𝑦, 𝜇 𝑔

𝑦

𝑕 𝜇 inf

∈𝒠 𝑀 𝑦, 𝜇

Example

 The dual function

 Neither

nor is convex, but the dual

function is concave  Horizontal dashed line:

∗ (the optimal

value of the problem)

 Rewrite (1) as unconstrained problem



• is the indicator function for the

nonpositive reals 

is the indicator function of

Linear Approximation Interpretation

min 𝑔

𝑦 𝐽 𝑔 𝑦

𝐽 ℎ 𝑦

• 2

𝐽 𝑣 0 𝑣 0, ∞ 𝑣 0.

 In the formulation (2)



• expresses our irritation or displeasure

associated with a constraint function value

• : zero if

, infinite if 

• gives our displeasure for an equality

constraint value

•  Our displeasure rises from zero to infinite as
• transitions from nonpositive to positive

Linear Approximation Interpretation

 In the formulation (2)

 Suppose we replace with linear function

, where

• , and

with 𝜉  Objective becomes the Lagrangian  Dual function value is optimal value of

Linear Approximation Interpretation

min 𝑔

𝑦 𝜇𝑔 𝑦 𝜉ℎ 𝑦

• 3

𝑀 𝑦, 𝜇, 𝜉 𝑔

𝑦 𝜇𝑔 𝑦 𝜉ℎ 𝑦

 In the formulation (3)

 We replace and with linear or “soft” displeasure functions  For an inequality constraint, our displeasure is zero when , and is positive when

• (assuming
• )

 In (2), any nonpositive value of is acceptable  In (3), we actually derive pleasure from constraints that have margin, i.e., from

• Linear Approximation

Interpretation

 Interpretation of Lower Bound

 The linear function is an underestimator of the indicator function  Lower Bound Property

Linear Approximation Interpretation

𝜇𝑣 𝐽 𝑣 𝜉𝑣 𝐽𝑣 𝑀 𝑦, 𝜇, 𝜉 𝑔

𝑦 𝜇𝑔 𝑦 𝜉ℎ 𝑦

• 𝑔

𝑦 𝐽 𝑔 𝑦

𝐽 ℎ 𝑦

Example

 Least-squares Solution of Linear Equations



•  No inequality constraints

 (linear) equality constraints

 Lagrangian

 Domain:

• min

𝑦𝑦

• s. t.

𝐵𝑦 𝑐 𝑀 𝑦, 𝜉 𝑦𝑦 𝜉𝐵𝑦 𝑐

Example

 Least-squares Solution of Linear Equations  Dual Function

 Optimality condition

𝑕 𝜉 inf

𝑀 𝑦, 𝜉 inf 𝑦𝑦 𝜉𝐵𝑦 𝑐

𝛼

𝑀 𝑦, 𝜉 2𝑦 𝐵𝜉 0 ⇒ 𝑦 1/2 𝐵𝜉

min 𝑦𝑦

• s. t.

𝐵𝑦 𝑐

Example

 Least-squares Solution of Linear Equations  Dual Function

 Concave Function

 Lower Bound Property

⇒ 𝑕 𝜉 𝑀 1/2 𝐵𝜉, 𝜉 1/4 𝜉𝐵𝐵𝜉 𝑐𝜉 1/4 𝜉𝐵𝐵𝜉 𝑐𝜉 inf 𝑦𝑦 𝐵𝑦 𝑐 min 𝑦𝑦

• s. t.

𝐵𝑦 𝑐

Example

 Standard Form LP

 Inequality constraints:

•  Lagrangian

 Dual Function

min 𝑑𝑦

• s. t.

𝐵𝑦 𝑐 𝑦 ≽ 0 𝑀 𝑦, 𝜇, 𝜉 𝑑𝑦 ∑ 𝜇𝑦 𝜉 𝐵𝑦 𝑐

• 𝑐𝜉 𝑑 𝐵𝜉 𝜇 𝑦

𝑕 𝜇, 𝜉 inf

𝑀 𝑦, 𝜇, 𝜉

𝑐𝜉 inf

𝑑 𝐵𝜉 𝜇 𝑦

Example

 Standard Form LP

 Inequality constraints:

•  Dual Function

 The lower bound is nontrivial only when and satisfy and

• min

𝑑𝑦

• s. t.

𝐵𝑦 𝑐 𝑦 ≽ 0 𝑕 𝜇, 𝜉 𝑐𝜉 𝐵𝜉 𝜇 𝑑 0, ∞ otherwise.

Outline

 The Lagrange Dual Function

 The Lagrange Dual Function  Lower Bound on Optimal Value  The Lagrange Dual Function and Conjugate Functions

 The Lagrange Dual Problem

 Making Dual Constraints Explicit  Weak Duality  Strong Duality and Slater’s Constraint Qualification

Conjugate Function



• Its conjugate function is

 dom 𝑔∗ 𝑧|𝑔∗ 𝑧 ∞  𝑔∗ is always convex 𝑔∗ 𝑧 sup

∈

𝑧𝑦 𝑔 𝑦

The Lagrange Dual Function and Conjugate Functions

 A Simple Example  Lagrangian  Dual Function

min 𝑔 𝑦

• s. t.

𝑦 0 𝑀 𝑦, 𝜉 𝑔 𝑦 𝜉𝑦 𝑕 𝜉 inf

𝑔 𝑦 𝜉𝑦

sup

• 𝜉 𝑦 𝑔 𝑦

𝑔∗ 𝜉

The Lagrange Dual Function and Conjugate Functions

 A More General Example  Lagrangian  Dual Function

min 𝑔

𝑦

• s. t.

𝐵𝑦 ≼ 𝑐 𝐷𝑦 𝑒 𝑀 𝑦, 𝜉 𝑔

𝑦 𝜇 𝐵𝑦 𝑐 𝜉 𝐷𝑦 𝑒

𝑕 𝜉 inf

𝑔 𝑦 𝜇 𝐵𝑦 𝑐 𝜉 𝐷𝑦 𝑒

𝑐𝜇 𝑒𝜉 inf

𝑔 𝑦 𝐵𝜇 𝐷𝜉 𝑦

𝑐𝜇 𝑒𝜉 𝑔

• ∗𝐵𝜇 𝐷𝜉

The Lagrange Dual Function and Conjugate Functions

 A More General Example  Lagrangian  Dual Function



• ∗

min 𝑔

𝑦

• s. t.

𝐵𝑦 ≼ 𝑐 𝐷𝑦 𝑒 𝑀 𝑦, 𝜉 𝑔

𝑦 𝜇 𝐵𝑦 𝑐 𝜉 𝐷𝑦 𝑒

𝑕 𝜉 𝑐𝜇 𝑒𝜉 𝑔

• ∗𝐵𝜇 𝐷𝜉

Example

 Equality Constrained Norm Minimization  Conjugate of  The Dual Function

min 𝑦

• s. t.

𝐵𝑦 𝑐 𝑔

• ∗ 𝑧 0 𝑧 ∗ 1,

∞ otherwise. 𝑕 𝜉 𝑐𝜉 𝑔

• ∗ 𝐵𝜉 𝑐𝜉 𝐵𝜉 ∗ 1,

∞ otherwise.

Example

 Entropy Maximization  Conjugate of  The Dual Function

min 𝑔

𝑦

𝑦 log 𝑦

• s. t.

𝐵𝑦 ≼ 𝑐 𝟐𝑦 1 𝑔

• ∗ 𝑧

𝑓

• 𝑕 𝜇, 𝜉 𝑐𝜇 𝜉 𝑔
• ∗𝐵𝜇 𝑤𝟐

= 𝑐𝜇 𝜉 ∑ 𝑓

• 𝑐𝜇 𝜉 𝑓 ∑

𝑓

Outline

 The Lagrange Dual Function

 The Lagrange Dual Function  Lower Bound on Optimal Value  The Lagrange Dual Function and Conjugate Functions

 The Lagrange Dual Problem

 Making Dual Constraints Explicit  Weak Duality  Strong Duality and Slater’s Constraint Qualification

The Lagrange Dual Problem

 For any and any

 What is the best lower bound?

 Lagrange Dual Problem  Primal Problem

𝑕 𝜇, 𝜉 𝑞∗ max 𝑕 𝜇, 𝜉

• s. t.

𝜇 ≽ 0 min 𝑔

𝑦

• s. t.

𝑔

𝑦 0,

𝑗 1, . . . , 𝑛 ℎ 𝑦 0 𝑗 1, . . . , 𝑞 1

The Lagrange Dual Problem

 For any and any

 What is the best lower bound?

 Lagrange Dual Problem

 Dual feasible: with  Dual optimal or optimal Lagrange multipliers:

∗ ∗

 A convex optimization problem

𝑕 𝜇, 𝜉 𝑞∗ max 𝑕 𝜇, 𝜉

• s. t.

𝜇 ≽ 0

Outline

 The Lagrange Dual Function

 The Lagrange Dual Function  Lower Bound on Optimal Value  The Lagrange Dual Function and Conjugate Functions

 The Lagrange Dual Problem

 Making Dual Constraints Explicit  Weak Duality  Strong Duality and Slater’s Constraint Qualification

Making Dual Constraints Explicit

 Motivation

 The may have dimension  Identify the equality constraints that are ‘hidden’ or ‘implicit’ in

 Standard Form LP  Dual Function

dom 𝑕 𝜇, 𝜉 𝑕 𝜇, 𝜉 ∞ 𝑕 𝜇, 𝜉 𝑐𝜉 𝐵𝜉 𝜇 𝑑 0, ∞ otherwise. min 𝑑𝑦

• s. t.

𝐵𝑦 𝑐 𝑦 ≽ 0

Example

 Lagrange Dual of Standard Form LP

 Lagrange Dual Problem  An Equivalent Problem

 Make equality constraints explicit

max 𝑕 𝜇, 𝜉 𝑐𝜉 𝐵𝜉 𝜇 𝑑 0, ∞ otherwise.

• s. t.

𝜇 ≽ 0 max 𝑐𝜉

• s. t.

𝐵𝜉 𝜇 𝑑 0 𝜇 ≽ 0

Example

 Lagrange Dual of Standard Form LP

 Lagrange Dual Problem  Another Equivalent Problem

 An LP in inequality form

max 𝑕 𝜇, 𝜉 𝑐𝜉 𝐵𝜉 𝜇 𝑑 0, ∞ otherwise.

• s. t.

𝜇 ≽ 0 max 𝑐𝜉

• s. t.

𝐵𝜉 𝑑 ≽ 0

Standard Form LP Inequality Form LP

Lagrange Dual

Example

 Lagrange Dual of Inequality Form LP

 Inequality form LP (Primal Problem)  Lagrangian  Lagrange dual function

min 𝑑𝑦

• s. t.

𝐵𝑦 ≼ 𝑐 𝑀 𝑦, 𝜇 𝑑𝑦 𝜇 𝐵𝑦 𝑐 𝑐𝜇 𝐵𝜇 𝑑 𝑦 𝑕 𝜇 inf

𝑀 𝑦, 𝜇 𝑐𝜇 inf 𝐵𝜇 𝑑 𝑦

𝑐𝜇 𝐵𝜇 𝑑 0, ∞ otherwise.

Example

 Lagrange Dual of Inequality Form LP

 Inequality form LP (Primal Problem)  Lagrange Dual Problem  An Equivalent Problem

min 𝑑𝑦

• s. t.

𝐵𝑦 ≼ 𝑐 max 𝑕 𝜇, 𝜉 𝑐𝜇 𝐵𝜇 𝑑 0, ∞ otherwise.

• s. t.

𝜇 ≽ 0 max 𝑐𝜇

• s. t.

𝐵𝜇 𝑑 0 𝜇 ≽ 0

Example

 Lagrange Dual of Inequality Form LP

 Inequality form LP (Primal Problem)  An Equivalent Problem

 An LP in standard form

min 𝑑𝑦

• s. t.

𝐵𝑦 ≼ 𝑐 max 𝑐𝜇

• s. t.

𝐵𝜇 𝑑 0 𝜇 ≽ 0

Standard Form LP Inequality Form LP

Lagrange Dual

Outline

 The Lagrange Dual Function

 The Lagrange Dual Function  Lower Bound on Optimal Value  The Lagrange Dual Function and Conjugate Functions

 The Lagrange Dual Problem

 Making Dual Constraints Explicit  Weak Duality  Strong Duality and Slater’s Constraint Qualification

Weak Duality

 For any and any

 What is the best lower bound?

 Lagrange Dual Problem

 Optimal value

∗

 Weak Duality

𝑕 𝜇, 𝜉 𝑞∗ max 𝑕 𝜇, 𝜉

• s. t.

𝜇 ≽ 0 𝑒∗ 𝑞∗

Does not rely

• n convexity!

Weak Duality

 Weak Duality

 If the primal problem is unbounded below, i.e.,

∗

, we must have

∗

, i.e., the Lagrange dual problem is infeasible  If

∗

, we must have

∗

 Optimal duality gap

 Nonegative

𝑞∗ 𝑒∗ 𝑒∗ 𝑞∗

Outline

 The Lagrange Dual Function

 The Lagrange Dual Function  Lower Bound on Optimal Value  The Lagrange Dual Function and Conjugate Functions

 The Lagrange Dual Problem

 Making Dual Constraints Explicit  Weak Duality  Strong Duality and Slater’s Constraint Qualification

Strong Duality

 Strong Duality

 The optimal duality gap is zero  The best bound that can be obtained from the Lagrange dual function is tight  In general, does not hold

 Usually hold for convex optimization



• are convex

∗ ∗

min 𝑔

𝑦

• s. t.

𝑔

𝑦 0,

𝑗 1, . . . , 𝑛 𝐵𝑦 𝑐

Slater’s Constraint Qualification

 Constraint Qualifications

 Sufficient conditions for strong duality

 Slater’s condition

 such that  Such a point is called strictly feasible

 If Slater’s condition holds and the problem is convex

 Strong duality holds  Dual optimal value is attained when

∗

𝑔

𝑦 0,

𝑗 1, … , 𝑛, 𝐵𝑦 𝑐

Slater’s Constraint Qualification

 Constraint Qualifications

 Sufficient conditions for strong duality

 Slater’s condition (weaker form)

 If the first constraint functions are affine  such that  When constraints are all linear equalities and inequalities, and

is open

 Reduce to feasibility

𝑔

𝑦 0,

𝑗 1, … , 𝑙 𝑔

𝑦 0,

𝑗 𝑙 1, … , 𝑛 𝐵𝑦 𝑐

Example

 Least-squares Solution of Linear Equations  Dual Problem  Slater’s condition

 The primal problem is feasible, i.e.,

 Strong duality always holds

 Even when

min 𝑦𝑦

• s. t.

𝐵𝑦 𝑐 max 1/4 ν𝐵𝐵𝜉 𝑐𝜉

Example

 Lagrange dual of LP  Strong duality holds for any LP

 If the primal problem is feasible or the dual problem is feasible

 Strong duality may fail

 If both the primal and dual problems are infeasible

Standard Form LP Inequality Form LP

Lagrange Dual

Example

 QCQP (Primal Problem)

 𝑄 ∈ 𝐓

and 𝑄 ∈ 𝐓 , 𝑗 1, … , 𝑛

 Dual Problem

 𝑄 𝜇 𝑄 ∑ 𝜇𝑄

• , 𝑟 𝜇 𝑟 ∑

𝜇𝑟

•  𝑠 𝜇 𝑠

∑

𝜇𝑠

•  Slater’s condition

 ∃𝑦 , 1/2 𝑦𝑄𝑦 𝑟

𝑦 𝑠 0, 𝑗 1, … , 𝑛

min 1/2 𝑦𝑄𝑦 𝑟

𝑦 𝑠

• s. t.

1/2 𝑦𝑄𝑦 𝑟

𝑦 𝑠 0,

𝑗 1, … , 𝑛 max 1/2 𝑟 𝜇 𝑄 𝜇 𝑟 𝜇 𝑠𝜇

• s. t.

𝜇 ≽ 0

Example

 A Nonconvex Quadratic Problem (Primal Problem)



•  Lagrangian

 Dual Function

min 𝑦𝐵𝑦 2𝑐𝑦

• s. t.

𝑦𝑦 1 𝑀 𝑦, 𝜇 𝑦𝐵𝑦 2𝑐𝑦 𝜇 𝑦𝑦 1 𝑦 𝐵 𝜇𝐽 𝑦 2𝑐𝑦 𝜇 𝑕 𝜇 𝑐 𝐵 𝜇𝐽 𝑐 𝜇 𝐵 𝜇𝐽 ≽ 0, 𝑐 ∈ ℛ 𝐵 𝜇𝐽 ∞ otherwise

Example

 A Nonconvex Quadratic Problem (Primal Problem)



•  Dual Problem

 A convex optimization problem

min 𝑦𝐵𝑦 2𝑐𝑦

• s. t.

𝑦𝑦 1 max 𝑐 𝐵 𝜇𝐽 𝑐 𝜇

• s. t.

𝐵 𝜇𝐽 ≽ 0, 𝑐 ∈ ℛ 𝐵 𝜇𝐽

Example

 A Nonconvex Quadratic Problem (Primal Problem)



•  Dual Problem

 A convex optimization problem 

and : eigenvalues and corresponding

(orthonormal) eigenvectors of

min 𝑦𝐵𝑦 2𝑐𝑦

• s. t.

𝑦𝑦 1 max ∑ 𝑟

𝑐 / 𝜇 𝜇

• 𝜇
• s. t.

𝜇 𝜇 𝐵

Strong duality holds

Example

 A Nonconvex Quadratic Problem (Primal Problem)



•  Dual Problem

 A convex optimization problem 

and : eigenvalues and corresponding

(orthonormal) eigenvectors of

min 𝑦𝐵𝑦 2𝑐𝑦

• s. t.

𝑦𝑦 1 max ∑ 𝑟

𝑐 / 𝜇 𝜇

• 𝜇
• s. t.

𝜇 𝜇 𝐵

Strong duality holds for any

• ptimization

problem with quadratic

• bjective

and

• ne

quadratic inequality constraint, provided Slater’s condition holds

Summary

 The Lagrange Dual Function

 The Lagrange Dual Function  Lower Bound on Optimal Value  The Lagrange Dual Function and Conjugate Functions

 The Lagrange Dual Problem

 Making Dual Constraints Explicit  Weak Duality  Strong Duality and Slater’s Constraint Qualification