CS599: Convex and Combinatorial Optimization Fall 2013 Lecture 3: - PowerPoint PPT Presentation

CS599: Convex and Combinatorial Optimization Fall 2013 Lecture 3: Linear Programming Duality II Instructor: Shaddin Dughmi

Announcements Today: wrap up linear programming Readings on website

Outline Recall 1 Formal Proof of Strong Duality of LP 2 Consequences of Duality 3 More Examples of Duality 4

Weak and Strong Duality Primal LP Dual LP c ⊺ x b ⊺ y maximize minimize subject to Ax ≤ b subject to A ⊺ y ≥ c x ≥ 0 y ≥ 0 Theorem (Weak Duality) OPT ( primal ) ≤ OPT ( dual ) . Theorem (Strong Duality) OPT ( primal ) = OPT ( dual ) . Recall 1/17

Informal Proof of Strong Duality Recall the physical interpretation of duality Recall 2/17

Informal Proof of Strong Duality Recall the physical interpretation of duality When ball is stationary at x , we expect force c to be neutralized only by constraints that are tight. i.e. force multipliers y ≥ 0 s.t. y ⊺ A = c y i ( b i − a i x ) = 0 Recall 2/17

Informal Proof of Strong Duality Recall the physical interpretation of duality When ball is stationary at x , we expect force c to be neutralized only by constraints that are tight. i.e. force multipliers y ≥ 0 s.t. y ⊺ A = c y i ( b i − a i x ) = 0 y ⊺ b − c ⊺ x = y ⊺ b − y T Ax = � y i ( b i − a i x ) = 0 i We found a primal and dual solution that are equal in value! Recall 2/17

Outline Recall 1 Formal Proof of Strong Duality of LP 2 Consequences of Duality 3 More Examples of Duality 4

Separating Hyperplane Theorem If A, B ⊆ R n are disjoint convex sets, then there is a hyperplane separating them. That is, there is a ∈ R n and b ∈ R such that a ⊺ x ≤ b for every x ∈ A and a ⊺ y ≥ b for every y ∈ B . Formal Proof of Strong Duality of LP 3/17

Definition A convex cone is a convex subset of R n which is closed under nonnegative scaling and convex combinations. Definition The convex cone generated by vectors u 1 , . . . , u m ∈ R n is the set of all nonnegative-weighted sums of these vectors (also known as conic combinations). � m � � Cone ( u 1 , . . . , u m ) = α i u i : α i ≥ 0 ∀ i i =1 Formal Proof of Strong Duality of LP 4/17

The following follows from the separating hyperplane Theorem. Farkas’ Lemma Let C be the convex cone generated by vectors u 1 , . . . , u m ∈ R n , and let w ∈ R n . Exactly one of the following is true: w ∈ C There is z ∈ R n such that z · u i ≤ 0 for all i , and z · w ≥ 0 . Formal Proof of Strong Duality of LP 4/17

Equivalently: Theorem of the Alternative One of the following is true, where U = [ u 1 , . . . , u m ] The system Uy = w , y ≥ 0 has a solution The system U ⊺ z ≤ 0 , z ⊺ w ≥ 0 has a solution. Formal Proof of Strong Duality of LP 4/17

Formal Proof of Strong Duality Primal LP Dual LP maximize c ⊺ x minimize b ⊺ y subject to Ax ≤ b subject to A ⊺ y = c y ≥ 0 Given v , by Farkas’ Lemma one of the following is true � A ⊺ � � c � The system y = , y ≥ 0 has a solution. 1 b ⊺ v OPT ( dual ) ≤ v � c � � � The system A ; b z ≤ 0 , z ⊺ ≥ 0 has a solution. 2 v � z 1 � , where z 1 ∈ R n and z 2 ∈ R Let z = z 2 Setting x = − z 1 /z 2 gives Ax ≤ b , c T x ≥ v . OPT ( primal ) ≥ v Formal Proof of Strong Duality of LP 5/17

Outline Recall 1 Formal Proof of Strong Duality of LP 2 Consequences of Duality 3 More Examples of Duality 4

Complementary Slackness Primal LP Dual LP maximize c ⊺ x minimize y ⊺ b subject to Ax ≤ b subject to A ⊺ y ≥ c x ≥ 0 y ≥ 0 Consequences of Duality 6/17

Complementary Slackness Primal LP Dual LP maximize c ⊺ x minimize y ⊺ b subject to Ax ≤ b subject to A ⊺ y ≥ c x ≥ 0 y ≥ 0 Let s i = ( b − Ax ) i be the i ’th primal slack variable Let t j = ( A ⊺ y − c ) j be the j ’th dual slack variable Consequences of Duality 6/17

Complementary Slackness Primal LP Dual LP maximize c ⊺ x minimize y ⊺ b subject to Ax ≤ b subject to A ⊺ y ≥ c x ≥ 0 y ≥ 0 Let s i = ( b − Ax ) i be the i ’th primal slack variable Let t j = ( A ⊺ y − c ) j be the j ’th dual slack variable Complementary Slackness x 1 x 2 x 3 x 4 y 1 a 11 a 12 a 13 a 14 b 1 x and y are optimal if and only if y 2 a 21 a 22 a 23 a 24 b 2 x j t j = 0 for all j = 1 , . . . , n y 3 a 31 a 32 a 33 a 34 b 3 y i s i = 0 for all i = 1 , . . . , m c 1 c 2 c 3 c 4 Consequences of Duality 6/17

Interpretation of Complementary Slackness Economic Interpretation Given an optimal primal production vector x and optimal dual offer prices y , Facility produces only products for which it is indifferent between sale and production. Only raw materials that are binding constraints on production are priced greater than 0 Consequences of Duality 7/17

Interpretation of Complementary Slackness Physical Interpretation Only walls adjacent to the balls equilibrium position push back on it. Consequences of Duality 7/17

Proof of Complementary Slackness Primal LP Dual LP maximize c ⊺ x minimize y ⊺ b subject to Ax ≤ b subject to A ⊺ y ≥ c x ≥ 0 y ≥ 0 Consequences of Duality 8/17

Proof of Complementary Slackness Primal LP Dual LP maximize c ⊺ x minimize y ⊺ b subject to Ax + s = b subject to A ⊺ y − t = c x ≥ 0 y ≥ 0 s ≥ 0 t ≥ 0 Can equivalently rewrite LP using slack variables Consequences of Duality 8/17

Proof of Complementary Slackness Primal LP Dual LP maximize c ⊺ x minimize y ⊺ b subject to Ax + s = b subject to A ⊺ y − t = c x ≥ 0 y ≥ 0 s ≥ 0 t ≥ 0 Can equivalently rewrite LP using slack variables y ⊺ b − c ⊺ x = y ⊺ ( Ax + s ) − ( y ⊺ A − t ⊺ ) x = y ⊺ s + t ⊺ x Consequences of Duality 8/17

Proof of Complementary Slackness Primal LP Dual LP maximize c ⊺ x minimize y ⊺ b subject to Ax + s = b subject to A ⊺ y − t = c x ≥ 0 y ≥ 0 s ≥ 0 t ≥ 0 Can equivalently rewrite LP using slack variables y ⊺ b − c ⊺ x = y ⊺ ( Ax + s ) − ( y ⊺ A − t ⊺ ) x = y ⊺ s + t ⊺ x Gap between primal and dual objectives is 0 if and only if complementary slackness holds. Consequences of Duality 8/17

Recovering Primal from Dual Will encounter LPs where the dual is easier to solve than primal Complementary slackness allows us to recover the primal optimal from the dual optimal, and vice versa. Consequences of Duality 9/17

Recovering Primal from Dual Will encounter LPs where the dual is easier to solve than primal Complementary slackness allows us to recover the primal optimal from the dual optimal, and vice versa. Assuming non-degeneracy: Every vertex of primal [dual] is the solution of exactly n [ m ] tight constraints. Consequences of Duality 9/17

Recovering Primal from Dual Will encounter LPs where the dual is easier to solve than primal Complementary slackness allows us to recover the primal optimal from the dual optimal, and vice versa. Assuming non-degeneracy: Every vertex of primal [dual] is the solution of exactly n [ m ] tight constraints. Primal LP Dual LP ( n variables, m + n constraints) ( m variables, m + n constraints) maximize c ⊺ x minimize y ⊺ b subject to Ax ≤ b subject to A ⊺ y ≥ c x ≥ 0 y ≥ 0 Consequences of Duality 9/17

Recovering Primal from Dual Will encounter LPs where the dual is easier to solve than primal Complementary slackness allows us to recover the primal optimal from the dual optimal, and vice versa. Assuming non-degeneracy: Every vertex of primal [dual] is the solution of exactly n [ m ] tight constraints. Primal LP Dual LP ( n variables, m + n constraints) ( m variables, m + n constraints) maximize c ⊺ x minimize y ⊺ b subject to Ax ≤ b subject to A ⊺ y ≥ c x ≥ 0 y ≥ 0 Let y be dual optimal. By non-degeneracy: Exactly m of the m + n dual constraints are tight at y Exactly n dual constraints are loose Consequences of Duality 9/17

Recovering Primal from Dual Will encounter LPs where the dual is easier to solve than primal Complementary slackness allows us to recover the primal optimal from the dual optimal, and vice versa. Assuming non-degeneracy: Every vertex of primal [dual] is the solution of exactly n [ m ] tight constraints. Primal LP Dual LP ( n variables, m + n constraints) ( m variables, m + n constraints) maximize c ⊺ x minimize y ⊺ b subject to Ax ≤ b subject to A ⊺ y ≥ c x ≥ 0 y ≥ 0 Let y be dual optimal. By non-degeneracy: Exactly m of the m + n dual constraints are tight at y Exactly n dual constraints are loose n loose dual constraints impose n tight primal constraints Consequences of Duality 9/17

Recovering Primal from Dual Will encounter LPs where the dual is easier to solve than primal Complementary slackness allows us to recover the primal optimal from the dual optimal, and vice versa. Assuming non-degeneracy: Every vertex of primal [dual] is the solution of exactly n [ m ] tight constraints. Primal LP Dual LP ( n variables, m + n constraints) ( m variables, m + n constraints) maximize c ⊺ x minimize y ⊺ b subject to Ax ≤ b subject to A ⊺ y ≥ c x ≥ 0 y ≥ 0 Let y be dual optimal. By non-degeneracy: Exactly m of the m + n dual constraints are tight at y Exactly n dual constraints are loose n loose dual constraints impose n tight primal constraints Assuming non-degeneracy, solving the linear equation yields a unique primal optimum solution x . Consequences of Duality 9/17