Convex Optimization ( EE227A: UC Berkeley ) Lecture 8 Weak duality - PowerPoint PPT Presentation

Convex Optimization ( EE227A: UC Berkeley ) Lecture 8 Weak duality 14 Feb, 2013 ◦ Suvrit Sra

Primal problem Let f i : R n → R ( 0 ≤ i ≤ m ). Generic nonlinear program min f 0 ( x ) s.t. f i ( x ) ≤ 0 , 1 ≤ i ≤ m, (P) x ∈ { dom f 0 ∩ dom f 1 · · · ∩ dom f m } . 2 / 9

Primal problem Let f i : R n → R ( 0 ≤ i ≤ m ). Generic nonlinear program min f 0 ( x ) s.t. f i ( x ) ≤ 0 , 1 ≤ i ≤ m, (P) x ∈ { dom f 0 ∩ dom f 1 · · · ∩ dom f m } . Def. Domain: The set D := { dom f 0 ∩ dom f 1 · · · ∩ dom f m } 2 / 9

Primal problem Let f i : R n → R ( 0 ≤ i ≤ m ). Generic nonlinear program min f 0 ( x ) s.t. f i ( x ) ≤ 0 , 1 ≤ i ≤ m, (P) x ∈ { dom f 0 ∩ dom f 1 · · · ∩ dom f m } . Def. Domain: The set D := { dom f 0 ∩ dom f 1 · · · ∩ dom f m } ◮ We call ( P ) the primal problem ◮ The variable x is the primal variable 2 / 9

Primal problem Let f i : R n → R ( 0 ≤ i ≤ m ). Generic nonlinear program min f 0 ( x ) s.t. f i ( x ) ≤ 0 , 1 ≤ i ≤ m, (P) x ∈ { dom f 0 ∩ dom f 1 · · · ∩ dom f m } . Def. Domain: The set D := { dom f 0 ∩ dom f 1 · · · ∩ dom f m } ◮ We call ( P ) the primal problem ◮ The variable x is the primal variable ◮ We will attach to ( P ) a dual problem 2 / 9

Primal problem Let f i : R n → R ( 0 ≤ i ≤ m ). Generic nonlinear program min f 0 ( x ) s.t. f i ( x ) ≤ 0 , 1 ≤ i ≤ m, (P) x ∈ { dom f 0 ∩ dom f 1 · · · ∩ dom f m } . Def. Domain: The set D := { dom f 0 ∩ dom f 1 · · · ∩ dom f m } ◮ We call ( P ) the primal problem ◮ The variable x is the primal variable ◮ We will attach to ( P ) a dual problem ◮ In our initial derivation: no restriction to convexity. 2 / 9

Lagrangian To the primal problem, associate Lagrangian L : R n × R m → R , � m L ( x, λ ) := f 0 ( x ) + i =1 λ i f i ( x ) . 3 / 9

Lagrangian To the primal problem, associate Lagrangian L : R n × R m → R , � m L ( x, λ ) := f 0 ( x ) + i =1 λ i f i ( x ) . ♠ Variables λ ∈ R m called Lagrange multipliers 3 / 9

Lagrangian To the primal problem, associate Lagrangian L : R n × R m → R , � m L ( x, λ ) := f 0 ( x ) + i =1 λ i f i ( x ) . ♠ Variables λ ∈ R m called Lagrange multipliers ♠ Suppose x is feasible, and λ ≥ 0 . Then, we get the lower-bound: λ ∈ R m f 0 ( x ) ≥ L ( x, λ ) ∀ x ∈ X , + . 3 / 9

Lagrangian To the primal problem, associate Lagrangian L : R n × R m → R , � m L ( x, λ ) := f 0 ( x ) + i =1 λ i f i ( x ) . ♠ Variables λ ∈ R m called Lagrange multipliers ♠ Suppose x is feasible, and λ ≥ 0 . Then, we get the lower-bound: λ ∈ R m f 0 ( x ) ≥ L ( x, λ ) ∀ x ∈ X , + . ♠ Lagrangian helps write problem in unconstrained form 3 / 9

Lagrangian λ ∈ R m Claim: Since, f 0 ( x ) ≥ L ( x, λ ) ∀ x ∈ X , + , primal optimal p ∗ = inf x ∈X sup L ( x, λ ) . λ ≥ 0 4 / 9

Lagrangian λ ∈ R m Claim: Since, f 0 ( x ) ≥ L ( x, λ ) ∀ x ∈ X , + , primal optimal p ∗ = inf x ∈X sup L ( x, λ ) . λ ≥ 0 Proof: ♠ If x is not feasible, then some f i ( x ) > 0 4 / 9

Lagrangian λ ∈ R m Claim: Since, f 0 ( x ) ≥ L ( x, λ ) ∀ x ∈ X , + , primal optimal p ∗ = inf x ∈X sup L ( x, λ ) . λ ≥ 0 Proof: ♠ If x is not feasible, then some f i ( x ) > 0 ♠ In this case, inner sup is + ∞ , so claim true by definition 4 / 9

Lagrangian λ ∈ R m Claim: Since, f 0 ( x ) ≥ L ( x, λ ) ∀ x ∈ X , + , primal optimal p ∗ = inf x ∈X sup L ( x, λ ) . λ ≥ 0 Proof: ♠ If x is not feasible, then some f i ( x ) > 0 ♠ In this case, inner sup is + ∞ , so claim true by definition ♠ If x is feasible, each f i ( x ) ≤ 0 , so sup λ � i λ i f i ( x ) = 0 4 / 9

Lagrange dual function Def. We define the Lagrangian dual as g ( λ ) := inf x L ( x, λ ) . 5 / 9

Lagrange dual function Def. We define the Lagrangian dual as g ( λ ) := inf x L ( x, λ ) . Observations: ◮ g is pointwise inf of affine functions of λ ◮ Thus, g is concave; it may take value −∞ 5 / 9

Lagrange dual function Def. We define the Lagrangian dual as g ( λ ) := inf x L ( x, λ ) . Observations: ◮ g is pointwise inf of affine functions of λ ◮ Thus, g is concave; it may take value −∞ ◮ Recall: f 0 ( x ) ≥ L ( x, λ ) ∀ x ∈ X ; thus 5 / 9

Lagrange dual function Def. We define the Lagrangian dual as g ( λ ) := inf x L ( x, λ ) . Observations: ◮ g is pointwise inf of affine functions of λ ◮ Thus, g is concave; it may take value −∞ ◮ Recall: f 0 ( x ) ≥ L ( x, λ ) ∀ x ∈ X ; thus f 0 ( x ) ≥ inf x ′ L ( x ′ , λ ) = g ( λ ) ◮ ∀ x ∈ X , 5 / 9

Lagrange dual function Def. We define the Lagrangian dual as g ( λ ) := inf x L ( x, λ ) . Observations: ◮ g is pointwise inf of affine functions of λ ◮ Thus, g is concave; it may take value −∞ ◮ Recall: f 0 ( x ) ≥ L ( x, λ ) ∀ x ∈ X ; thus f 0 ( x ) ≥ inf x ′ L ( x ′ , λ ) = g ( λ ) ◮ ∀ x ∈ X , ◮ Now minimize over x on lhs, to obtain p ∗ ≥ g ( λ ) . ∀ λ ∈ R m + 5 / 9

Lagrange dual problem sup g ( λ ) s.t. λ ≥ 0 . λ 6 / 9

Lagrange dual problem sup g ( λ ) s.t. λ ≥ 0 . λ ◮ dual feasible: if λ ≥ 0 and g ( λ ) > −∞ ◮ dual optimal: λ ∗ if sup is achieved 6 / 9

Lagrange dual problem sup g ( λ ) s.t. λ ≥ 0 . λ ◮ dual feasible: if λ ≥ 0 and g ( λ ) > −∞ ◮ dual optimal: λ ∗ if sup is achieved ◮ Lagrange dual is always concave, regardless of original 6 / 9

Weak duality Def. Denote dual optimal value by d ∗ , i.e., d ∗ := sup g ( λ ) . λ ≥ 0 7 / 9

Weak duality Def. Denote dual optimal value by d ∗ , i.e., d ∗ := sup g ( λ ) . λ ≥ 0 Theorem (Weak-duality): For problem (P), we have p ∗ ≥ d ∗ . 7 / 9

Weak duality Def. Denote dual optimal value by d ∗ , i.e., d ∗ := sup g ( λ ) . λ ≥ 0 Theorem (Weak-duality): For problem (P), we have p ∗ ≥ d ∗ . + , p ∗ ≥ g ( λ ) . Proof: We showed that for all λ ∈ R m Thus, it follows that p ∗ ≥ sup g ( λ ) = d ∗ . 7 / 9

Equality constraints min f 0 ( x ) s.t. f i ( x ) ≤ 0 , i = 1 , . . . , m, h i ( x ) = 0 , i = 1 , . . . , p. Exercise: Show that we get the Lagrangian dual + × R p : ( λ, ν ) �→ inf g : R m L ( x, λ, ν ) , x where the Lagrange variable ν corresponding to the equality constraints is unconstrained. Hint: Represent h i ( x ) = 0 as h i ( x ) ≤ 0 and − h i ( x ) ≤ 0 . 8 / 9

Equality constraints min f 0 ( x ) s.t. f i ( x ) ≤ 0 , i = 1 , . . . , m, h i ( x ) = 0 , i = 1 , . . . , p. Exercise: Show that we get the Lagrangian dual + × R p : ( λ, ν ) �→ inf g : R m L ( x, λ, ν ) , x where the Lagrange variable ν corresponding to the equality constraints is unconstrained. Hint: Represent h i ( x ) = 0 as h i ( x ) ≤ 0 and − h i ( x ) ≤ 0 . Again, we see that p ∗ ≥ sup λ ≥ 0 ,ν g ( λ, ν ) = d ∗ 8 / 9

Some duals ◮ Least-norm solution of linear equations: min x T x s.t. Ax = b ◮ Linear programming standard form ◮ Study example (5.7) in BV (binary QP) 9 / 9