COMP331/557 Chapter 4: Duality Theory (Bertsimas & Tsitsiklis, - PowerPoint PPT Presentation

COMP331/557 Chapter 4: Duality Theory (Bertsimas & Tsitsiklis, Chapter 4) 130

Example minimize + 2 x 2 x 1 s.t. x 1 ≥ 2 ≥ 2 x 2 − x 1 + x 2 ≥ 1 + ≥ 5 x 1 x 2 x 1 , x 2 ≥ 0 Goal: Find an upper bound on the optimal solution value z ∗ . Easy: Any feasible solution provides one. Examples: ⇒ z ∗ ≤ 14 ◮ ( x 1 , x 2 ) = ( 4 , 5 ) ⇒ z ∗ ≤ 11 ◮ ( x 1 , x 2 ) = ( 3 , 4 ) ⇒ z ∗ ≤ 10 ◮ ( x 1 , x 2 ) = ( 2 , 4 ) ⇒ z ∗ ≤ 8 ◮ ( x 1 , x 2 ) = ( 2 , 3 ) 131

Example minimize + 2 x 2 ← z x 1 s.t. x 1 ≥ 2 ← C 1 ≥ 2 ← C 2 x 2 − x 1 + x 2 ≥ 1 ← C 3 + ≥ 5 ← C 4 x 1 x 2 x 1 , x 2 ≥ 0 New goal: Find a lower bound on the optimal solution value. Examples: ◮ C 4 ⇒ z ≥ 5 ◮ C 1 + 2 C 2 ⇒ z ≥ 6 ◮ 3 C 1 + 2 C 3 ⇒ ◮ 3 C 2 − C 3 ⇒ 132

Example minimize + 2 x 2 ← z x 1 s.t. x 1 ≥ 2 ← C 1 ≥ 2 ← C 2 x 2 − x 1 + x 2 ≥ 1 ← C 3 + ≥ 5 ← C 4 x 1 x 2 x 1 , x 2 ≥ 0 Idea: Add non-negative combination p 1 · C 1 + p 2 · C 2 + p 3 · C 3 + p 4 · C 4 of the constraints, s.t.: z = x 1 + 2 x 2 ≥ ( p 1 − p 3 + p 4 ) · x 1 + ( p 2 + p 3 + p 4 ) · x 2 ≥ 2 p 1 + 2 p 2 + p 3 + 5 p 4 Dual Problem: Find the best such lower bound. 133

More general minimize + · · · + ← z c 1 x 1 c n x n s.t. a 11 x 1 + · · · + a 1 n x n ≥ b 1 ← C 1 + · · · + ≥ ← C 2 a 21 x 1 a 2 n x n b 2 . . ... . . . . a m 1 x 1 + · · · + a mn x n ≥ b m ← C m x 1 , . . . , x n ≥ 0 Consider: p 1 C 1 + p 2 C 2 + · · · + p m C m Q: What are the conditions on p 1 , . . . , p m so that this combination lower bounds z ? a 11 p 1 + a 21 p 2 + · · · + a m 1 p m ≤ c 1 . . . . . . . . . . . ≤ . a 1 n p 1 + a 1 n p 2 + · · · + a mn p m ≤ c n p 1 , p 2 , . . . , p m ≥ 0 Q: What lower bound do we get? 134

Primal and Dual LP Primal: Decision variables x 1 , . . . , x n . minimize c 1 x 1 + · · · + c n x n s.t. a 11 x 1 + · · · + a 1 n x n ≥ b 1 c T x min + · · · + ≥ a 21 x 1 a 2 n x n b 2 s.t. A x ≥ b . . ... . . . . ≥ 0 x a m 1 x 1 + · · · + a mn x n ≥ b m x 1 , . . . , x n ≥ 0 Dual: Decision variables p 1 , . . . , p m . maximize + · · · + b 1 p 1 b m p m s.t. a 11 p 1 + · · · + a m 1 p m ≤ c 1 b T p max + · · · + ≤ a 12 p 1 a m 2 p m c 2 A T p s.t. ≤ c . . ... . . . . p ≥ 0 a 1 n p 1 + · · · + a mn p m ≤ c n p 1 , . . . , p m ≥ 0 135

Primal and Dual Example (1) Primal: min + 2 x 2 x 1 s.t. 2 x 1 + x 2 ≥ 7 − x 1 + 3 x 2 ≥ 1 x 1 + 4 x 2 ≥ 5 x 1 , x 2 ≥ 0 Dual: 136

Primal and Dual Example (2) Primal: min − x 1 + 4 x 2 s.t. 3 x 1 + 2 x 2 ≥ 9 − 3 x 2 ≤ 3 x 1 x 1 , x 2 ≥ 0 Dual: 137

Primal and Dual Example (3) Primal: min − x 1 + 4 x 2 s.t. 3 x 1 + 2 x 2 ≥ 9 − 3 x 2 = 3 x 1 x 1 , x 2 ≥ 0 Dual: 138

Primal and Dual Linear Program Consider the general linear program: Obtain a lower bound: c T · x p T · b min max a i T · x ≥ b i s.t. p i ≥ 0 for i ∈ M 1 s.t. for i ∈ M 1 a i T · x ≤ b i p i ≤ 0 for i ∈ M 2 for i ∈ M 2 a i T · x = b i p i free for i ∈ M 3 for i ∈ M 3 A T j · p ≤ c j for j ∈ N 1 x j ≥ 0 for j ∈ N 1 A T j · p ≥ c j for j ∈ N 2 x j ≤ 0 for j ∈ N 2 A T x j free for j ∈ N 3 j · p = c j for j ∈ N 3 The linear program on the right hand side is the dual linear program of the primal linear program on the left hand side. 139

Primal and Dual Variables and Constraints primal LP (minimize) dual LP (maximize) ≥ b i ≥ 0 constraints ≤ b i ≤ 0 variables = b i free ≥ 0 ≤ c i variables ≤ 0 ≥ c i constraints free = c i 140

Examples primal LP dual LP p T · b max c T · x min A T · p s.t. = c s.t. A · x ≥ b p ≥ 0 c T · x min p T · b max s.t. A · x = b A T · p s.t. ≤ c x ≥ 0 141

Basic Properties of the Dual Linear Program Theorem 4.1. The dual of the dual LP is the primal LP. Proof: Primal in general form: Dual: c T · x p T · b min max a i T · x ≥ b i s.t. p i ≥ 0 for i ∈ M 1 s.t. for i ∈ M 1 a i T · x ≤ b i p i ≤ 0 for i ∈ M 2 for i ∈ M 2 p i free for i ∈ M 3 a i T · x = b i for i ∈ M 3 A T j · p ≤ c j for j ∈ N 1 x j ≥ 0 for j ∈ N 1 A T j · p ≥ c j for j ∈ N 2 x j ≤ 0 for j ∈ N 2 A T x j free for j ∈ N 3 j · p = c j for j ∈ N 3 142

Basic Properties of the Dual Linear Program Proof (cont.): Dual: Dual (in primal form): p T · b − p T · b max min s.t. p i ≥ 0 for i ∈ M 1 − A T s.t. j · p ≥ − c j for j ∈ N 1 p i ≤ 0 for i ∈ M 2 − A T j · p ≤ − c j for j ∈ N 2 p i free for i ∈ M 3 − A T j · p = − c j for j ∈ N 3 A T j · p ≤ c j for j ∈ N 1 p i ≥ 0 for i ∈ M 1 A T j · p ≥ c j for j ∈ N 2 p i ≤ 0 for i ∈ M 2 A T j · p = c j for j ∈ N 3 p i free for i ∈ M 3 143

Basic Properties of the Dual Linear Program Proof (cont.): Dual (in primal form): Dual of Dual: − p T · b min − A T s.t. j · p ≥ − c j for j ∈ N 1 − A T j · p ≤ − c j for j ∈ N 2 − A T j · p = − c j for j ∈ N 3 p i ≥ 0 for i ∈ M 1 p i ≤ 0 for i ∈ M 2 p i free for i ∈ M 3 144

Equavalence of the Dual LP Theorem 4.2. Let Π 1 and Π 2 be two LPs where Π 2 has been obtained from Π 1 by (several) transformations of the following type: i replace a free variable by the difference of two non-negative variables; ii introduce a slack variable in order to replace an inequality constraint by an equation; iii if some row of a feasible equality system is a linear combination of the other rows, eliminate this row. Then the dual of Π 1 is equivalent to the dual of Π 2 . 145

Weak Duality Theorem Theorem 4.3. If x is a feasible solution to the primal LP (minimization problem) and p a feasible solution to the dual LP (maximization problem), then c T · x ≥ p T · b . Corollary 4.4. Consider a primal-dual pair of linear programs as above. a If the primal LP is unbounded (i. e., optimal cost = −∞ ), then the dual LP is infeasible. b If the dual LP is unbounded (i. e., optimal cost = ∞ ), then the primal LP is infeasible. c If x and p are feasible solutions to the primal and dual LP, resp., and if c T · x = p T · b , then x and p are optimal solutions. 146

Strong Duality Theorem Theorem 4.5. If an LP has an optimal solution, so does its dual and the optimal costs are equal. 147

Different Possibilities for Primal and Dual LP primal \ dual finite optimum unbounded infeasible finite optimum possible impossible impossible unbounded impossible impossible possible infeasible impossible possible possible Example of infeasible primal and dual LP: min x 1 + 2 x 2 max p 1 + 3 p 2 s.t. x 1 + x 2 = 1 s.t. p 1 + 2 p 2 = 1 2 x 1 + 2 x 2 = 3 p 1 + 2 p 2 = 2 148

Complementary Slackness Consider the following pair of primal and dual LPs: c T · x p T · b min max p T · A = c T s.t. A · x ≥ b s.t. p ≥ 0 If x and p are feasible solutions, then c T · x = p T · A · x ≥ p T · b .Thus, c T · x = p T · b for all i : p i = 0 if a i T · x > b i . ⇐ ⇒ Theorem 4.6. Consider an arbitrary pair of primal and dual LPs. Let x and p be feasible solutions to the primal and dual LP, respectively. Then x and p are both optimal if and only if u i := p i ( a i T · x − b i ) = 0 for all i , (1) v j := ( c j − p T · A j ) x j = 0 for all j . (2) 149

Complementary Slackness Example Consider the following LP in standard form and its dual: min 13 x 1 + 10 x 2 + 6 x 3 max 8 p 1 + 3 p 2 s.t. 5 x 1 + x 2 + 3 x 3 = 8 s.t. 5 p 1 + 3 p 2 ≤ 13 3 x 1 + = 3 p 1 + p 2 ≤ 10 x 2 x 1 , x 2 , x 3 ≥ 0 3 p 1 ≤ 6 Claim: x ∗ = ( 1 , 0 , 1 ) is a non-degenerate optimal solution to the primal. Verify this using complementary slackness! 150

Geometric View Consider pair of primal and dual LPs with A ∈ R m × n and rank ( A ) = n : c T · x p T · b min max m � a i T · x ≥ b i , s.t. i = 1 , . . . , m s.t. p i · a i = c i = 1 p ≥ 0 Let I ⊆ { 1 , . . . , m } with | I | = n and a i , i ∈ I , linearly independent. a i T · x = b i , i ∈ I , has unique solution x I (basic solution) = ⇒ Let p ∈ R m (dual vector). Then x , p are optimal solutions if i a i T · x ≥ b i for all i (primal feasibility) ii p i = 0 for all i �∈ I (complementary slackness) iii � m i = 1 p i · a i = c (dual feasibility) iv p ≥ 0 (dual feasibility) (ii) and (iii) imply � i ∈ I p i · a i = c which has a unique solution p I . The a i , i ∈ I , form basis for dual LP and p I is corresponding basic solution. 151

Geometric View (cont.) a 3 a 1 A a 5 a 2 a 4 a 1 B a 3 a 4 a 2 a 1 C c a 1 a 5 D a 1 152

Dual Variables as Marginal Costs Consider the primal dual pair: c T · x p T · b min max p T · A ≤ c T s.t. A · x = b s.t. x ≥ 0 Let x ∗ be optimal basic feasible solution to primal LP with basis B , i. e., x ∗ B = B − 1 · b B > 0 (i. e., x ∗ non-degenerate). and assume that x ∗ Replace b by b + d . For small d , the basis B remains feasible and optimal: B − 1 · ( b + d ) = B − 1 · b + B − 1 · d ≥ 0 (feasibility) c T = c T − c BT · B − 1 · A ≥ 0 ¯ (optimality) Optimal cost of perturbed problem is c BT · B − 1 · ( b + d ) = c BT · x ∗ B + ( c BT · B − 1 ) · d � �� � = p T Thus, p i is the marginal cost per unit increase of b i . 153