Duality (II) Lijun Zhang zlj@nju.edu.cn http://cs.nju.edu.cn/zlj - - PowerPoint PPT Presentation

Duality (II)

Lijun Zhang zlj@nju.edu.cn http://cs.nju.edu.cn/zlj

Outline

 Saddle-point Interpretation

 Max-min Characterization of Weak and Strong Duality  Saddle-point Interpretation  Game Interpretation

 Optimality Conditions

 Certificate of Suboptimality and Stopping Criteria  Complementary Slackness  KKT Optimality Conditions  Solving the Primal Problem via the Dual

 Examples  Generalized Inequalities

Outline

 Saddle-point Interpretation

 Max-min Characterization of Weak and Strong Duality  Saddle-point Interpretation  Game Interpretation

 Optimality Conditions

 Certificate of Suboptimality and Stopping Criteria  Complementary Slackness  KKT Optimality Conditions  Solving the Primal Problem via the Dual

 Examples  Generalized Inequalities

More Symmetric Form

 Assume no equality constraint

 Suppose for some . Then,

≽

by and

•  If

, then the optimal choice of is and

≽

• sup

≽

𝑀 𝑦, 𝜇 sup

≽

𝑔

𝑦 𝜇𝑔 𝑦

• 𝑔

𝑦 𝑔 𝑦 0, 𝑗 1, … , 𝑛

∞ otherwise

More Symmetric Form

 Optimal Value of Primal Problem  Optimal Value of Dual Problem  Weak Duality  Strong Duality

 Min and Max can be switched

𝑞⋆ inf

sup ≽

𝑀 𝑦, 𝜇 𝑒⋆ sup

≽

inf

𝑀 𝑦, 𝜇

sup

≽

inf

𝑀 𝑦, 𝜇 inf sup ≽

𝑀 𝑦, 𝜇 sup

≽

inf

𝑀 𝑦, 𝜇 inf sup ≽

𝑀 𝑦, 𝜇

A More General Form

 Max-min Inequality

 For any

• and any
•  Strong Max-min Property

 Hold only in special cases

sup

∈

inf

∈ 𝑔 𝑥, 𝑨 inf ∈ sup ∈

𝑔 𝑥, 𝑨 sup

∈

inf

∈ 𝑔 𝑥, 𝑨 inf ∈ sup ∈

𝑔 𝑥, 𝑨

Outline

 Saddle-point Interpretation

 Max-min Characterization of Weak and Strong Duality  Saddle-point Interpretation  Game Interpretation

 Optimality Conditions

 Certificate of Suboptimality and Stopping Criteria  Complementary Slackness  KKT Optimality Conditions  Solving the Primal Problem via the Dual

 Examples  Generalized Inequalities

Saddle-point Interpretation

 is a saddle point for

 minimizes , maximizes

𝑔 𝑥 , 𝑨 𝑔 𝑥 , 𝑨̃ 𝑔 𝑥, 𝑨̃ , ∀𝑥 ∈ 𝑋, 𝑨 ∈ 𝑎 𝑔 𝑥 , 𝑨̃ inf

∈ 𝑔𝑥, 𝑨̃ ,

𝑔 𝑥 , 𝑨̃ sup

∈

𝑔 𝑥 , 𝑨

https: / / en.wikipedia.org/ wiki/ Saddle_point

Saddle-point Interpretation

 is a saddle point for

 minimizes , maximizes

 Imply the strong max-min property

𝑔 𝑥 , 𝑨 𝑔 𝑥 , 𝑨̃ 𝑔 𝑥, 𝑨̃ , ∀𝑥 ∈ 𝑋, 𝑨̃ ∈ 𝑎 𝑔 𝑥 , 𝑨̃ inf

∈ 𝑔𝑥, 𝑨̃ ,

𝑔 𝑥 , 𝑨̃ sup

∈

𝑔 𝑥 , 𝑨 sup

∈

inf

∈ 𝑔 𝑥, 𝑨 inf ∈ 𝑔𝑥, 𝑨̃ 𝑔 𝑥

, 𝑨̃ 𝑔 𝑥 , 𝑨̃ sup

∈

𝑔 𝑥 , 𝑨 inf

∈ sup ∈

𝑔 𝑥, 𝑨 ⇒ sup

∈

inf

∈ 𝑔 𝑥, 𝑨 inf ∈ sup ∈

𝑔 𝑥, 𝑨 ⇒ sup

∈

inf

∈ 𝑔 𝑥, 𝑨 inf ∈ sup ∈

𝑔 𝑥, 𝑨

Saddle-point Interpretation

 is a saddle point for

 minimizes , maximizes

 If

⋆ ⋆ are primal and dual optimal

points and strong duality holds,

⋆ ⋆

form a saddle-point.  If is saddle-point, then is primal

• ptimal,

is dual optimal, and the duality gap is zero.

𝑔 𝑥 , 𝑨 𝑔 𝑥 , 𝑨̃ 𝑔 𝑥, 𝑨̃ , ∀𝑥 ∈ 𝑋, 𝑨̃ ∈ 𝑎 𝑔 𝑥 , 𝑨̃ inf

∈ 𝑔𝑥, 𝑨̃ ,

𝑔 𝑥 , 𝑨̃ sup

∈

𝑔 𝑥 , 𝑨

Outline

 Saddle-point Interpretation

 Max-min Characterization of Weak and Strong Duality  Saddle-point Interpretation  Game Interpretation

 Optimality Conditions

 Certificate of Suboptimality and Stopping Criteria  Complementary Slackness  KKT Optimality Conditions  Solving the Primal Problem via the Dual

 Examples  Generalized Inequalities

Continuous Zero-sum Game

 Two players

 The 1st player chooses , and the 2nd player selects  Player 1 pays an amount to player 2

 Goals

 Player 1 wants to minimize  Player 2 wants to maximize

 Continuous game

 The choices are vectors, and not discrete

Continuous Zero-sum Game

 Player 1 makes his choice first

 Player 2 wants to maximize payoff and the resulting payoff is

∈

 Player 1 knows that player 2 will follow this strategy, and so will choose to make

∈

as small as possible  Thus, player 1 chooses  The payoff

argmin

∈

sup

∈

𝑔 𝑥, 𝑨 inf

∈ sup ∈

𝑔 𝑥, 𝑨

Continuous Zero-sum Game

 Player 2 makes his choice first

 Player 1 wants to minimize payoff and the resulting payoff is ∈  Player 2 knows that player 1 will follow this strategy, and so will choose to make

∈

as large as possible  Thus, player 2 chooses  The payoff

argmax

∈

inf

∈ 𝑔 𝑥, 𝑨

sup

∈

inf

∈ 𝑔 𝑥, 𝑨

Continuous Zero-sum Game

 Max-min Inequality

 Player 1 wants to minimize  Player 2 wants to maximize

sup

∈

inf

∈ 𝑔 𝑥, 𝑨 inf ∈ sup ∈

𝑔 𝑥, 𝑨

Player 1 plays first Player 2 plays first

It is better for a player to go second

Continuous Zero-sum Game

 Strong Max-min Property

 Player 1 wants to minimize  Player 2 wants to maximize

sup

∈

inf

∈ 𝑔 𝑥, 𝑨 inf ∈ sup ∈

𝑔 𝑥, 𝑨

Player 1 plays first Player 2 plays first

There is no advantage to playing second

Continuous Zero-sum Game

 Strong Max-min Property  Saddle-point Property

 If is a saddle-point for (and ), then it is called a solution of the game

 𝑥 : the optimal strategy for player 1  𝑨̃: the optimal strategy for player 2  No advantage to playing second

sup

∈

inf

∈ 𝑔 𝑥, 𝑨 inf ∈ sup ∈

𝑔 𝑥, 𝑨

Player 1 plays first Player 2 plays first

A Special Case

 Payoff is the Lagrangian;

•  Player 1 chooses the primal variable

while player 2 chooses the dual variable  The optimal choice for player 2, if she must choose first, is any dual optimal ⋆

 The resulting payoff: 𝑒⋆

 Conversely, if player 1 chooses first, his

• ptimal choice is any primal optimal

⋆

 The resulting payoff: 𝑞⋆

 Duality gap: advantage of going second

Outline

 Saddle-point Interpretation

 Max-min Characterization of Weak and Strong Duality  Saddle-point Interpretation  Game Interpretation

 Optimality Conditions

 Certificate of Suboptimality and Stopping Criteria  Complementary Slackness  KKT Optimality Conditions  Solving the Primal Problem via the Dual

 Examples  Generalized Inequalities

Certificate of Suboptimality

 Dual Feasible

 A lower bound on the optimal value

• f the primal problem

⋆

 Provides a proof or certificate  Bound how suboptimal a given feasible point is, without knowing the value of

⋆

• ⋆
•  𝑦 is 𝜗-suboptimal for primal problem

 (𝜇, 𝜉 is 𝜗-suboptimal for dual

Certificate of Suboptimality

 Gap between Primal & Dual Objectives

•  Referred to as duality gap associated with

primal feasible and dual feasible  localizes the optimal value of the primal (and dual) problems to an interval

⋆

• ⋆
•  The width of the interval is the duality gap

 If duality gap of is , then is primal optimal and is dual optimal

Stopping Criteria

 Optimization algorithms produce a sequence of primal feasible

and dual

feasible

• for

 Required absolute accuracy:  A Nonheuristic Stopping Criterion

•  Guarantees when algorithm terminates,
• is -suboptimal

Stopping Criteria

 A Relative Accuracy  Nonheuristic Stopping Criteria

 If

𝑕 𝜇 , 𝜉 0, 𝑔

𝑦

𝑕 𝜇 , 𝜉 𝑕 𝜇 , 𝜉 𝜗

• r

𝑔

𝑦

0, 𝑔

𝑦

𝑕 𝜇 , 𝜉 𝑔

𝑦

𝜗

 Then

⋆

, and the relative error satisfies

𝑔

𝑦

𝑞⋆ |𝑞⋆| 𝜗

Outline

 Saddle-point Interpretation

 Max-min Characterization of Weak and Strong Duality  Saddle-point Interpretation  Game Interpretation

 Optimality Conditions

 Certificate of Suboptimality and Stopping Criteria  Complementary Slackness  KKT Optimality Conditions  Solving the Primal Problem via the Dual

 Examples  Generalized Inequalities

Complementary Slackness

 Suppose Strong Duality Holds

 For primal optimal

⋆ & dual optimal ⋆ ⋆

 First line: the optimal duality gap is zero  Second line: definition of the dual function  Third line: infimum of Lagrangian over 𝑦 is less than or equal to its value at 𝑦 𝑦⋆

𝑔

𝑦⋆ 𝑕 𝜇⋆, 𝜉⋆

inf

𝑔 𝑦 ∑

𝜇

⋆𝑔 𝑦

• ∑

𝜉

⋆ℎ 𝑦

• 𝑔

𝑦⋆ ∑

𝜇

⋆𝑔 𝑦⋆ ∑

𝜉

⋆ℎ 𝑦⋆

• 𝑔

𝑦⋆

Complementary Slackness

 Suppose Strong Duality Holds

 For primal optimal

⋆ & dual optimal ⋆ ⋆

 Last line: 𝜇

⋆ 0, 𝑔 𝑦⋆ 0, 𝑗 1, … , 𝑛 and

ℎ 𝑦⋆ 0, 𝑗 1, … , 𝑞  We conclude that the two inequalities in this chain hold with equality

𝑔

𝑦⋆ 𝑕 𝜇⋆, 𝜉⋆

inf

𝑔 𝑦 ∑

𝜇

⋆𝑔 𝑦

• ∑

𝜉

⋆ℎ 𝑦

• 𝑔

𝑦⋆ ∑

𝜇

⋆𝑔 𝑦⋆ ∑

𝜉

⋆ℎ 𝑦⋆

• 𝑔

𝑦⋆

Complementary Slackness

 Suppose Strong Duality Holds

 For primal optimal

⋆ & dual optimal ⋆ ⋆

 Equality in the third line implies 𝑦⋆ minimizes 𝑀 𝑦, 𝜇⋆, 𝜉⋆  Equality in the last line implies ∑

𝜇

⋆𝑔 𝑦⋆

• 𝑔

𝑦⋆ 𝑕 𝜇⋆, 𝜉⋆

inf

𝑔 𝑦 ∑

𝜇

⋆𝑔 𝑦

• ∑

𝜉

⋆ℎ 𝑦

• 𝑔

𝑦⋆ ∑

𝜇

⋆𝑔 𝑦⋆ ∑

𝜉

⋆ℎ 𝑦⋆

• 𝑔

𝑦⋆

Complementary Slackness

 Complementary Slackness

 Derived from

• ⋆

⋆

•  Holds for any primal optimal

⋆ and dual

• ptimal ⋆

⋆ (when strong duality holds)

 Other expressions

• ⋆
• ⋆
• ⋆
• ⋆

 𝑗-th optimal Lagrange multiplier is zero unless 𝑗-th constraint is active at the optimum

𝜇

⋆𝑔 𝑦⋆ 0,

𝑗 1, … , 𝑛

Outline

 Saddle-point Interpretation

 Max-min Characterization of Weak and Strong Duality  Saddle-point Interpretation  Game Interpretation

 Optimality Conditions

 Certificate of Suboptimality and Stopping Criteria  Complementary Slackness  KKT Optimality Conditions  Solving the Primal Problem via the Dual

 Examples  Generalized Inequalities

KKT Conditions for Nonconvex Problems



⋆ and ⋆ ⋆ : any primal and dual

• ptimal points with zero duality gap



⋆ minimizes ⋆ ⋆ ⋆ ⋆ ⋆

• ⋆
• ⋆
• ⋆
• ⋆
• ⋆

KKT Conditions for Nonconvex Problems



⋆ and ⋆ ⋆ : any primal and dual

• ptimal points with zero duality gap

 Karush-Kuhn-Tucker (KKT) conditions

𝑔

𝑦⋆ 0, 𝑗 1, … , 𝑛

ℎ 𝑦⋆ 0, 𝑗 1, … , 𝑞 𝜇

⋆ 0, 𝑗 1, … , 𝑛

𝜇

⋆𝑔 𝑦⋆ 0, 𝑗 1, … , 𝑛

𝛼𝑔

𝑦⋆ ∑

𝜇

⋆𝛼𝑔 𝑦⋆ ∑

𝜉

⋆𝛼ℎ 𝑦⋆ 0

• For
• ptimization

problem with differentiable

• bjective

and constraint functions for which strong duality obtains, any pair of primal and dual optimal must satisfy KKT conditions. Necessary Condition

KKT Conditions for Convex Problems

 If are convex,

are affine,

satisfy  Then, and are primal and dual

• ptimal, with zero duality gap.

𝑔

𝑦

0, 𝑗 1, … , 𝑛 ℎ 𝑦 0, 𝑗 1, … , 𝑞 𝜇 0, 𝑗 1, … , 𝑛 𝜇 𝑔

𝑦

0, 𝑗 1, … , 𝑛 𝛼𝑔

𝑦

∑ 𝜇 𝛼𝑔

𝑦

∑ 𝜉 𝛼ℎ 𝑦

• For

any convex

• ptimization

problem with differentiable objective and constraint functions, any points that satisfy the KKT conditions are primal and dual optimal, and have zero duality gap. Sufficient Condition

KKT Conditions for Convex Problems

 For convex problem satisfying Slater’s condition, KKT conditions provide necessary and sufficient conditions for optimality.

 Slater’s condition implies that optimal duality gap is zero and dual optimum is attained  is optimal if and only if there are that, together with , satisfy the KKT conditions

KKT Conditions for Convex Problems

 The KKT conditions play an important role in optimization.

 In a few special cases it is possible to solve the KKT conditions.  More generally, many algorithms for convex optimization can be nterpreted as methods for solving the KKT conditions

Example

 Equality Constrained Convex Quadratic Minimization

 Primal Problem (with

• )

 KKT conditions

𝐵𝑦⋆ 𝑐, 𝑄𝑦⋆ 𝑟 𝐵𝜉⋆ 0 ⇔ 𝑄 𝐵 𝐵 𝑦⋆ 𝑤⋆ = 𝑟 𝑐

 Solving this set of 𝑛 𝑜 equations in 𝑛 𝑜 variables 𝑦⋆, 𝜉⋆ gives optimal primal and dual variables

min 1/2 𝑦𝑄𝑦 𝑟𝑦 𝑠

• s. t.

𝐵𝑦 𝑐

Outline

 Saddle-point Interpretation

 Max-min Characterization of Weak and Strong Duality  Saddle-point Interpretation  Game Interpretation

 Optimality Conditions

 Certificate of Suboptimality and Stopping Criteria  Complementary Slackness  KKT Optimality Conditions  Solving the Primal Problem via the Dual

 Examples  Generalized Inequalities

Solving the Primal Problem via the Dual

 If strong duality holds and a dual

• ptimal solution

⋆ ⋆ exists, any

primal optimal point is also a minimizer

• f

⋆ ⋆

 Suppose the minimizer of

⋆ ⋆

below is unique

 If solution is primal feasible, it’s primal optimal  If not primal feasible, no optimal point exists

min 𝑔

𝑦 𝜇 ⋆𝑔 𝑦

• 𝜉

⋆ℎ 𝑦

Example

 Entropy Maximization

 Primal Problem (with domain

• )

 Dual Problem (

: the -th column of )

 Assume weak Slater’s condition holds

 There exists an 𝑦 ≻ 0 with 𝐵𝑦 ≼ 𝑐, 𝟐𝑦 1  So strong duality holds and an optimal solution 𝜇⋆, 𝜉⋆ exists

min 𝑔

𝑦 ∑

𝑦 log 𝑦

• s. t.

𝐵𝑦 ≼ 𝑐 𝟐𝑦 1 max 𝑐𝜇 𝜉 𝑓 ∑ 𝑓

• s. t.

𝜇 ≽ 0

Example

 Entropy Maximization

 Suppose we have solved the dual problem  The Lagrangian at

⋆ ⋆

is

 Strictly convex on 𝒠 and bounded below  So it has a unique solution  If 𝑦⋆ is primal feasible, it must be the optimal solution of the primal problem  If 𝑦⋆ is not primal feasible, we can conclude that the primal optimum is not attained

𝑀 𝑦, 𝜇⋆, 𝜉⋆ ∑ 𝑦 log 𝑦

• 𝜇⋆ 𝐵𝑦 𝑐 𝜉⋆𝟐𝑦 1

𝑦

⋆ 1/ exp 𝑏 𝜇⋆ 𝜉⋆ 1 ,

𝑗 1, … , 𝑜

Outline

 Saddle-point Interpretation

 Max-min Characterization of Weak and Strong Duality  Saddle-point Interpretation  Game Interpretation

 Optimality Conditions

 Certificate of Suboptimality and Stopping Criteria  Complementary Slackness  KKT Optimality Conditions  Solving the Primal Problem via the Dual

 Examples  Generalized Inequalities

Examples

 Introduce New Variables and Equality Constraints  Transform the Objective  Implicit Constraints

Introduce New Variables and Equality Constraints

 Unconstrained Problem

 Lagrange dual function: constant

⋆

 strong duality holds (𝑞⋆ 𝑒⋆, but it is not useful

 Reformulation

 Lagrangian of the reformulated problem

𝑀 𝑦, 𝑧, 𝜉 𝑔

𝑧 𝜉 𝐵𝑦 𝑐 𝑧

min 𝑔

𝑧

• s. t.

𝐵𝑦 𝑐 𝑧 min 𝑔

𝐵𝑦 𝑐

Introduce New Variables and Equality Constraints

 Unconstrained Problem

 Find dual function by minimizing

 Minimizing over 𝑦, 𝑕 𝜉 ∞ unless 𝐵𝑤 0

 When

• minimizing

gives

𝑕 𝜉 𝑐𝜉 inf

𝑔 𝑧 𝜉𝑧 𝑐𝜉 𝑔

• ∗ 𝜉

 𝑔

• ∗: conjugate of 𝑔
•  Dual problem

 More useful

max 𝑐𝜉 𝑔

• ∗ 𝜉
• s. t.

𝐵𝜉 0

Example

 Unconstrained Geometric Program

 Problem

min log ∑ exp 𝑏

𝑦 𝑐

•  Add new variables & equality constraints

 𝑏

: 𝑗-th row of 𝐵

 Conjugate of the log-sum-exp function

min 𝑔

𝑧 log ∑

exp 𝑧

• s. t.

𝐵𝑦 𝑐 𝑧 𝑔

• ∗ 𝜉 ∑

𝜉 log 𝜉

• 𝜉 ≽ 0, 𝟐𝜉 1

∞ otherwise

Introduce New Variables and Equality Constraints

 Unconstrained Geometric Program

 Primal Problem  Dual of the reformulated problem

 An entropy maximization problem

max 𝑐𝜉 ∑ 𝜉 log 𝜉

• s. t.

𝟐𝜉 1 𝐵𝜉 0 𝜉 ≽ 0 min 𝑔

𝑧 log ∑

exp 𝑧

• s. t.

𝐵𝑦 𝑐 𝑧

Example

 Norm Approximation Problem

 Problem (with any norm )

min 𝐵𝑦 𝑐

 Constant Lagrange dual function (not useful)

 Reformulate the problem  Lagrange dual problem

 The conjugate of a norm is the indicator function of the dual norm unit ball

min 𝑧

• s. t.

𝐵𝑦 𝑐 𝑧 max 𝑐𝜉

• s. t.

𝜉 ∗ 1, 𝐵𝜉 0

Introduce New Variables and Equality Constraints

 Constraint Functions



•  Introduce
•  The Lagrangian for the above problem

min 𝑔

𝐵𝑦 𝑐

• s. t.

𝑔

𝐵𝑦 𝑐 0,

𝑗 1, … , 𝑛 min 𝑔

𝑧

• s. t.

𝑔

𝑧 0,

𝑗 1, … , 𝑛 𝐵𝑦 𝑐 𝑧, 𝑗 0, … , 𝑛 𝑀 𝑦, 𝑧, … , 𝑧, 𝜇, 𝜉, … , 𝜉 𝑔

𝑧 ∑

𝜇𝑔

𝑧 ∑

𝜉

𝐵𝑦 𝑐 𝑧

Introduce New Variables and Equality Constraints

 Constraint Functions

 Dual function (by minimizing over

)

 Minimum over 𝑦 is ∞ unless ∑ 𝐵

• 𝜉 0

In this case, for 𝜇 ≻ 0, 𝑕 𝜇, 𝜉, … , 𝜉 𝜉

𝑐

inf

,…,

𝑔

𝑧 𝜇𝑔 𝑧 𝜉 𝑧

• 𝜉

𝑐 inf 𝑔 𝑧 𝜉 𝑧 𝜇 inf 𝑔 𝑧 𝜉/𝜇 𝑧

• 𝜉

𝑐 𝑔

• ∗ 𝜉 𝜇𝑔
• ∗ 𝜉/𝜇

Introduce New Variables and Equality Constraints

 Constraint Functions

 What happens when (but some

• )

 If 𝜇 0 & 𝜉 0, the dual function is ∞  If 𝜇 0 & 𝜉 0, terms involving 𝑧, 𝜉, 𝜇 are 0

 The expression for is valid for all if

 Take 𝜇𝑔

• ∗ 𝜉/𝜇 0, when 𝜇 0 & 𝜉 0

 Take 𝜇𝑔

• ∗ 𝜉/𝜇 ∞, when 𝜇 0 & 𝜉 0

 Dual Problem

max ∑ 𝜉

𝑐 𝑔

• ∗ 𝜉 ∑

𝜇𝑔

• ∗ 𝜉/𝜇
• s. t.

𝜇 ≽ 0, ∑ 𝐵

𝜉 0

Example

 Inequality Constrained Geometric Program

 Problem

 Let 𝑔

𝑧 log ∑

𝑓

•  Conjugate of 𝑔
• min

log ∑ 𝑓

• s. t.

log ∑ 𝑓

• 0, 𝑗 1, … , 𝑛

𝑔

• ∗ 𝜉 ∑

𝜉 log 𝜉

• 𝜉 ≽ 0, 𝟐𝜉 1

∞ otherwise

Example

 Inequality Constrained Geometric Program

 Dual problem is

max 𝑐

𝜉 ∑

𝜉 log 𝜉

• ∑

𝑐

𝜉 ∑

𝜉 log 𝜉/𝜇

• s. t.

𝜉 ≽ 0, 𝟐𝜉 1 𝜉 ≽ 0, 𝟐𝜉 𝜇, 𝑗 1, … , 𝑛 𝜇 0, 𝑗 1, … , 𝑛 ∑ 𝐵

𝜉 0

Transform the Objective

 Replace the Objective by an Increasing Function of

 The resulting problem is equivalent  The dual of this equivalent problem can be very different from dual of original problem

Example

 Minimum Norm Problem

min 𝐵𝑦 𝑐

 Reformulate this problem as

 Introduce new variables and replace the

• bjective by half its square

 Equivalent to the original problem

 Dual of the reformulated problem

min 1/2 𝑧

• s. t.

𝐵𝑦 𝑐 𝑧 max 1/2 𝜉 ∗

𝑐𝜉

• s. t.

𝐵𝜉 0

Implicit Constraints

 Include Some of the Constraints in the Objective Function

 Modifying the objective function to be infinite when the constraint is violated

Example

 Linear Program with Box Constraints

 Problem

 𝐵 ∈ 𝐒 and 𝑚 ≺ 𝑣  𝑚 ≼ 𝑦 ≼ 𝑣 are called box constraints

 Derive the dual of this linear program

min 𝑑𝑦

• s. t.

𝐵𝑦 𝑐 𝑚 ≼ 𝑦 ≼ 𝑣 min 𝑐𝜉 𝜇

𝑣 𝜇 𝑚

• s. t.

𝐵𝜉 𝜇 𝜇 𝑑 0 𝜇 ≽ 0, 𝜇 ≽ 0

Example

 Linear Program with Box Constraints

 Problem

 𝐵 ∈ 𝐒 and 𝑚 ≺ 𝑣  𝑚 ≼ 𝑦 ≼ 𝑣 are called box constraints

 Reformulate the problem as

 Here, we define

min 𝑑𝑦

• s. t.

𝐵𝑦 𝑐 𝑚 ≼ 𝑦 ≼ 𝑣 min 𝑔

𝑦

• s. t.

𝐵𝑦 𝑐 𝑔

𝑦 𝑑𝑦 𝑚 ≼ 𝑦 ≼ 𝑣

∞ otherwise

Implicit Constraints

 Linear Program with Box Constraints

 Dual function

 𝑧

max 𝑧, 0 , 𝑧 max 𝑧, 0

 We can derive an analytical formula for 𝑕, which is a concave piecewise-linear function

 Dual problem

max 𝑐𝜉 𝑣 𝐵𝜉 𝑑 𝑚 𝐵𝜉 𝑑

 Unconstrained problem  Different form from the dual of original problem

𝑕 𝜉 inf

≼≼ 𝑑𝑦 𝜉 𝐵𝑦 𝑐

𝑐𝜉 𝑣 𝐵𝜉 𝑑 𝑚 𝐵𝜉 𝑑

Outline

 Saddle-point Interpretation

 Max-min Characterization of Weak and Strong Duality  Saddle-point Interpretation  Game Interpretation

 Optimality Conditions

 Certificate of Suboptimality and Stopping Criteria  Complementary Slackness  KKT Optimality Conditions  Solving the Primal Problem via the Dual

 Examples  Generalized Inequalities

Generalized Inequalities

 Problems with Generalized Inequality Constraints

 Primal Problem

 𝐿 ⊆ 𝐒 are proper cones  Do not assume convexity of the problem  Assume the domain is nonempty

min 𝑔

𝑦

• s. t.

𝑔

𝑦 ≼ 0, 𝑗 1, … , 𝑛

ℎ 𝑦 0, 𝑗 1, … , 𝑞

The Lagrange Dual

 Lagrangian

𝑀 𝑦, 𝜇, 𝜉 𝑔

𝑦 𝜇 𝑔 𝑦 ⋯ 𝜇 𝑔 𝑦

𝜉ℎ 𝑦 ⋯ 𝜉ℎ 𝑦

 𝜇 𝜇, … , 𝜇 , 𝜇 ∈ 𝐒, 𝜉 𝜉, … , 𝜉

 Dual Function

𝑕 𝜇, 𝜉 inf

∈𝒠 𝑀 𝑦, 𝜇, 𝜉

inf

∈𝒠 𝑔 𝑦 ∑

𝜇

𝑔 𝑦

• ∑

𝜉ℎ 𝑦

•  Lagrangian is affine in dual variables; Dual

function is pointwise infimum of Lagrangian. So, dual function is concave

The Lagrange Dual

 Nonnegativity on dual variables

• ∗



• ∗ the dual cone of
•  Lagrange multipliers must be dual

nonnegative

 Weak Duality

 If

• ∗

and

• then
•  So, for any primal feasible

and

• ∗

𝑔

𝑦

∑ 𝜇

𝑔 𝑦

∑ 𝜉ℎ 𝑦

• 𝑔

𝑦

•  Taking the infimum over

yields

⋆

The Lagrange Dual

 Lagrange dual optimization problem

 Always have weak duality (

⋆ ⋆)

whether or not the primal problem is convex

 Primal Problem

max 𝑕 𝜇, 𝜉

• s. t.

𝜇 ≽

∗ 0, 𝑗 1, … , 𝑛

min 𝑔

𝑦

• s. t.

𝑔

𝑦 ≼ 0, 𝑗 1, … , 𝑛

ℎ 𝑦 0, 𝑗 1, … , 𝑞

The Lagrange Dual

 Slater’s Condition and Strong Duality

 Strong duality:

⋆ ⋆

 Holds when primal problem is convex and satisfies appropriate constraint qualifications

 For problem (convex and

• convex )

 Generalized version of Slater’s condition

 ∃𝑦 ∈ relint 𝒠, 𝐵𝑦 𝑐, 𝑔

𝑦 ≺ 0, 𝑗 1, … , 𝑛

 Implies strong duality and the dual optimum is attained

min 𝑔

𝑦

• s. t.

𝑔

𝑦 ≼ 0, 𝑗 1, … , 𝑛

𝐵𝑦 𝑐

Example

 Lagrange Dual of Cone Program in Standard Form

 Primal Problem

 𝐵 ∈ 𝐒, 𝑐 ∈ 𝐒 and 𝐿 ⊆ 𝐒 is a proper cone

 Lagrangian: 𝑀 𝑦, 𝜇, 𝜉 𝑑𝑦 𝜇𝑦 𝜉 𝐵𝑦 𝑐  Dual function

min 𝑑𝑦

• s. t.

𝐵𝑦 𝑐 𝑦 ≽ 0 𝑕 𝜇, 𝜉 inf

𝑀 𝑦, 𝜇, 𝜉 𝑐𝜉 𝐵𝜉 𝜇 𝑑 0,

∞ otherwise.

Example

 Lagrange Dual of Cone Program in Standard Form

 Dual problem  Eliminating and defining gives

 A cone program in inequality form  Involving the dual generalized inequality  Strong duality (Slater condition): 𝑦 ≻ 0, 𝐵𝑦 𝑐

max 𝑐𝜉

• s. t.

𝐵𝜉 𝑑 𝜇 𝜇 ≽∗ 0 max 𝑐𝑧

• s. t.

𝐵𝑧 ≼∗ 𝑑

Optimality Conditions

 Complementary Slackness

 Assume primal and dual optimal values are equal, and attained at

⋆ ⋆ ⋆

 Complementary slackness

 𝑦⋆ minimizes 𝑀 𝑦, 𝜇⋆, 𝜉⋆  The two sums in the second line are zero  The second sum is zero ⇒ ∑ 𝜇

⋆𝑔 𝑦⋆ 0 ⇒

• 𝜇

⋆𝑔 𝑦⋆ 0, 𝑗 1, … , 𝑛

𝑔

𝑦⋆ 𝑕 𝜇⋆, 𝜉⋆

𝑔

𝑦⋆ ∑

𝜇

⋆𝑔 𝑦⋆ ∑

𝜉

⋆ℎ 𝑦⋆

• 𝑔

𝑦⋆

Optimality Conditions

 Complementary Slackness

 Assume primal and dual optimal values are equal, and attained at

⋆ ⋆ ⋆

 Complementary slackness

 From𝜇

⋆𝑔 𝑦⋆ 0, we can conclude

𝜇

⋆ ≻

∗ 0 ⇒ 𝑔

𝑦⋆ 0, 𝑔 𝑦⋆ ≺ 0 ⇒ 𝜇 ⋆ 0

 Possible to satisfy𝜇

⋆𝑔 𝑦⋆ 0 with 𝜇 ⋆

0 & 𝑔

𝑦⋆ 0

𝑔

𝑦⋆ 𝑕 𝜇⋆, 𝜉⋆

𝑔

𝑦⋆ ∑

𝜇

⋆𝑔 𝑦⋆ ∑

𝜉

⋆ℎ 𝑦⋆

• 𝑔

𝑦⋆

Optimality Conditions

 KKT Conditions

 Additionally assume

are differentiable

 Generalize the KKT conditions to problems with generalized inequalities 

⋆ minimizes ⋆ ⋆

 𝐸𝑔

𝑦⋆ ∈ ℝ: derivative of 𝑔 evaluated at 𝑦⋆

𝛼𝑔

𝑦⋆ 𝐸𝑔 𝑦⋆ 𝜇 ⋆

• 𝜉

⋆𝛼ℎ 𝑦⋆ 0

Optimality Conditions

 KKT Conditions

 If strong duality holds, any primal optimal

⋆ and dual optimal ⋆ ⋆ must satisfy the

• ptimality conditions (or KKT conditions)

 If the primal problem is convex, the converse also holds

𝑔

𝑦⋆ ≼ 0,

𝑗 1, … , 𝑛 ℎ 𝑦⋆ 0, 𝑗 1, … , 𝑞 𝜇

⋆ ≽

∗ 0,

𝑗 1, … , 𝑛 𝜇

⋆𝑔 𝑦⋆ 0,

𝑗 1, … , 𝑛 𝛼𝑔

𝑦⋆

𝐸𝑔

𝑦⋆ 𝜇 ⋆

𝜉

⋆𝛼ℎ 𝑦⋆ 0

Summary

 Saddle-point Interpretation

 Max-min Characterization of Weak and Strong Duality  Saddle-point Interpretation  Game Interpretation

 Optimality Conditions

 Certificate of Suboptimality and Stopping Criteria  Complementary Slackness  KKT Optimality Conditions  Solving the Primal Problem via the Dual

 Examples  Generalized Inequalities