CS 473: Algorithms Chandra Chekuri Ruta Mehta University of - - PowerPoint PPT Presentation
CS 473: Algorithms Chandra Chekuri Ruta Mehta University of Illinois, Urbana-Champaign Fall 2016 Chandra & Ruta (UIUC) CS473 1 Fall 2016 1 / 39 CS 473: Algorithms, Fall 2016 LP Duality Lecture 20 November 2, 2016 Chandra & Ruta
Chandra Chekuri Ruta Mehta
University of Illinois, Urbana-Champaign
Fall 2016
Chandra & Ruta (UIUC) CS473 1 Fall 2016 1 / 39
November 2, 2016
Chandra & Ruta (UIUC) CS473 2 Fall 2016 2 / 39
max cx subject to Ax = b which is compact form for max c1x1 + c2x2 + . . . + cnxn ai1x1 + ai2x2 + . . . + ainxn = bi 1 ≤ i ≤ m Question: Is this a geneal LP problem or is it some how easy?
Chandra & Ruta (UIUC) CS473 3 Fall 2016 3 / 39
max cx subject to Ax = b Basically reduces to linear system solving. Three cases for Ax = b. The system Ax = b is infeasible, that is, no solution The system Ax = b has a unique solution x∗ when rank([A b]) = n (full rank). Optimum solution value is cx∗ The system Ax = b has infinite solutions when rank([A b]) < n. There all vectors of the form x∗ + y are feasible where y is null-space(A) = {y | Ay = 0}. Let d be dimension of null-space(A) and let e1, e2, . . . , ed be an
cx = cx∗ + cy = cx∗ + c(λ1e1 + λ2e2 + . . . + λded. If cei = 0 for any i then optimum solution value is unbounded. Otherwise cx∗.
Chandra & Ruta (UIUC) CS473 4 Fall 2016 4 / 39
Two basic canonical forms: max cx, Ax = b, x ≥ 0 max cx, Ax ≤ b, x ≥ 0 What makes LP non-trivial and different from linear system solving is the additional non-negativity constraint on variables.
Chandra & Ruta (UIUC) CS473 5 Fall 2016 5 / 39
Chandra & Ruta (UIUC) CS473 6 Fall 2016 6 / 39
Consider the program maximize 4x1+ 2x2 subject to x1+ 3x2 ≤ 5 2x1− 4x2 ≤ 10 x1+ x2 ≤ 7 x1 ≤ 5
Chandra & Ruta (UIUC) CS473 7 Fall 2016 7 / 39
Consider the program maximize 4x1+ 2x2 subject to x1+ 3x2 ≤ 5 2x1− 4x2 ≤ 10 x1+ x2 ≤ 7 x1 ≤ 5
1
(0, 1) satisfies all the constraints and gives value 2 for the
Chandra & Ruta (UIUC) CS473 7 Fall 2016 7 / 39
Consider the program maximize 4x1+ 2x2 subject to x1+ 3x2 ≤ 5 2x1− 4x2 ≤ 10 x1+ x2 ≤ 7 x1 ≤ 5
1
(0, 1) satisfies all the constraints and gives value 2 for the
2
Thus, optimal value σ∗ is at least 4.
Chandra & Ruta (UIUC) CS473 7 Fall 2016 7 / 39
Consider the program maximize 4x1+ 2x2 subject to x1+ 3x2 ≤ 5 2x1− 4x2 ≤ 10 x1+ x2 ≤ 7 x1 ≤ 5
1
(0, 1) satisfies all the constraints and gives value 2 for the
2
Thus, optimal value σ∗ is at least 4.
3
(2, 0) also feasible, and gives a better bound of 8.
Chandra & Ruta (UIUC) CS473 7 Fall 2016 7 / 39
Consider the program maximize 4x1+ 2x2 subject to x1+ 3x2 ≤ 5 2x1− 4x2 ≤ 10 x1+ x2 ≤ 7 x1 ≤ 5
1
(0, 1) satisfies all the constraints and gives value 2 for the
2
Thus, optimal value σ∗ is at least 4.
3
(2, 0) also feasible, and gives a better bound of 8.
4
How good is 8 when compared with σ∗?
Chandra & Ruta (UIUC) CS473 7 Fall 2016 7 / 39
1
Let us multiply the first constraint by 2 and the and add it to second constraint 2( x1+ 3x2 ) ≤ 2(5) +1( 2x1− 4x2 ) ≤ 1(10) 4x1+ 2x2 ≤ 20
2
Thus, 20 is an upper bound on the optimum value!
Chandra & Ruta (UIUC) CS473 8 Fall 2016 8 / 39
1
Multiply first equation by y1, second by y2, third by y3 and fourth by y4 (all of y1, y2, y3, y4 being positive) and add y1( x1+ 3x2 ) ≤ y1(5) +y2( 2x1− 4x2 ) ≤ y2(10) +y3( x1+ x2 ) ≤ y3(7) +y4( x1 ) ≤ y4(5) (y1 + 2y2 + y3 + y4)x1 + (3y1 − 4y2 + y3)x2 ≤ . . .
2
5y1 + 10y2 + 7y3 + 5y4 is an upper bound, provided coefficients of xi are same as in the objective function, i.e., y1 + 2y2 + y3 + y4 = 4 3y1 − 4y2 + y3 = 2
3
The best upper bound is when 5y1 + 10y2 + 7y3 + 5y4 is minimized!
Chandra & Ruta (UIUC) CS473 9 Fall 2016 9 / 39
Thus, the optimum value of program maximize 4x1 + 2x2 subject to x1 + 3x2 ≤ 5 2x1 − 4x2 ≤ 10 x1 + x2 ≤ 7 x1 ≤ 5 is upper bounded by the optimal value of the program minimize 5y1 + 10y2 + 7y3 + 5y4 subject to y1 + 2y2 + y3 + y4 = 4 3y1 − 4y2 + y3 = 2 y1, y2 ≥ 0
Chandra & Ruta (UIUC) CS473 10 Fall 2016 10 / 39
Given a linear program Π in canonical form maximize d
j=1 cjxj
subject to d
j=1 aijxj ≤ bi
i = 1, 2, . . . n the dual Dual(Π) is given by minimize n
i=1 biyi
subject to n
i=1 yiaij = cj
j = 1, 2, . . . d yi ≥ 0 i = 1, 2, . . . n
Chandra & Ruta (UIUC) CS473 11 Fall 2016 11 / 39
Given a linear program Π in canonical form maximize d
j=1 cjxj
subject to d
j=1 aijxj ≤ bi
i = 1, 2, . . . n the dual Dual(Π) is given by minimize n
i=1 biyi
subject to n
i=1 yiaij = cj
j = 1, 2, . . . d yi ≥ 0 i = 1, 2, . . . n
Dual(Dual(Π)) is equivalent to Π
Chandra & Ruta (UIUC) CS473 11 Fall 2016 11 / 39
If x′ is a feasible solution to Π and y ′ is a feasible solution to Dual(Π) then c · x′ ≤ y ′ · b.
Chandra & Ruta (UIUC) CS473 12 Fall 2016 12 / 39
If x′ is a feasible solution to Π and y ′ is a feasible solution to Dual(Π) then c · x′ ≤ y ′ · b.
If x∗ is an optimal solution to Π and y ∗ is an optimal solution to Dual(Π) then c · x∗ = y ∗ · b. Many applications! Maxflow-Mincut theorem can be deduced from duality.
Chandra & Ruta (UIUC) CS473 12 Fall 2016 12 / 39
We already saw the proof by the way we derived it but we will do it again formally. Since y ′ is feasible to Dual(Π): y ′A = c Therefore c · x′ = y ′Ax′ Since x′ is feasible Ax′ ≤ b and hence, c · x′ = y ′Ax′ ≤ y ′ · b
Chandra & Ruta (UIUC) CS473 13 Fall 2016 13 / 39
maximize 4x1+ x2+ 3x3 subject to x1+ 4x2 ≤ 2 2x1− x2+ x3 ≤ 4 x1, x2, x3 ≥ 0
Chandra & Ruta (UIUC) CS473 14 Fall 2016 14 / 39
maximize 4x1+ x2+ 3x3 subject to x1+ 4x2 ≤ 2 2x1− x2+ x3 ≤ 4 x1, x2, x3 ≥ 0 Choose non-negative y1, y2 and multiply inequalities maximize 4x1+ x2+ 3x3 subject to y1(x1+ 4x2 ) ≤ 2y1 y2(2x1− x2+ x3) ≤ 4y2 x1, x2, x3 ≥ 0
Chandra & Ruta (UIUC) CS473 14 Fall 2016 14 / 39
Choose non-negative y1, y2 and multiply inequalities maximize 4x1+ x2+ 3x3 subject to y1(x1+ 4x2 ) ≤ 2y1 y2(2x1− x2+ x3) ≤ 4y2 x1, x2, x3 ≥ 0 Adding the inequalities we get an inequality below that is valid for any feasible x and any non-negative y: (y1 + 2y2)x1 + (4y1 − y2)x2 + y2 ≤ 2y1 + 4y2 Suppose we choose y1, y2 such that y1 + 2y2 ≥ 4 and 4y2 − y2 ≥ 1 and 2y1 ≥ 3 Then, since x1, x2, x3 ≥ 0, we have 4x1 + x2 + 3x3 ≤ 2y1 + 4y2
Chandra & Ruta (UIUC) CS473 15 Fall 2016 15 / 39
maximize 4x1+ x2+ 3x3 subject to x1+ 4x2 ≤ 2 2x1− x2+ x3 ≤ 4 x1, x2, x3 ≥ 0 is upper bounded by minimize 2y1+ 4y2 subject to y1+ 2y2 ≥ 4 4y1− y2 ≥ 1 2y1 ≥ 3 y1, y2 ≥ 0
Chandra & Ruta (UIUC) CS473 16 Fall 2016 16 / 39
Compactly, For the primal LP max cx subject to Ax ≤ b, x ≥ 0 the dual LP is min yb subject to yA ≥ c, y ≥ 0
Chandra & Ruta (UIUC) CS473 17 Fall 2016 17 / 39
Assume primal LP is a maximization LP. For a given LP, Dual is another LP. The variables in the dual correspond to “non-trival” primal constraints and vice-versa. Dual of the dual LP give us back the primal LP. Weak and strong duality theorems. If primal is unbounded (objective achieves infinity) then dual LP is infeasible. Why? If dual LP had a feasible solution it would upper bound the primal LP which is not possible. If primal is infeasible then dual LP is unbounded. Primal and dual optimum solutions satisfy complementary slackness conditions (discussed soon).
Chandra & Ruta (UIUC) CS473 18 Fall 2016 18 / 39
Chandra & Ruta (UIUC) CS473 19 Fall 2016 19 / 39
Input:G = (V = L ∪ R, E) max
xuv s.t.
xuv ≤ 1 ∀v ∈ V . xuv ≥ 0 ∀uv ∈ E
Chandra & Ruta (UIUC) CS473 20 Fall 2016 20 / 39
s-t flow in directed graph G = (V , E) with capacities c. Assume for simplicity that no incoming edges into s. max
x(s, v)
x(u, v) −
x(v, w) = 0 ∀v ∈ V \ {s, t} x(u, v) ≤ c(u, v) ∀(u, v) ∈ E x(u, v) ≥ 0 ∀(u, v) ∈ E.
Chandra & Ruta (UIUC) CS473 21 Fall 2016 21 / 39
Chandra & Ruta (UIUC) CS473 22 Fall 2016 22 / 39
Chandra & Ruta (UIUC) CS473 23 Fall 2016 23 / 39
Suppose we want to solve LP of the form: max cx subject to Ax ≤ b It is an optimization problem. Can we reduce it to a decision problem?
Chandra & Ruta (UIUC) CS473 24 Fall 2016 24 / 39
Suppose we want to solve LP of the form: max cx subject to Ax ≤ b It is an optimization problem. Can we reduce it to a decision problem? Yes, via binary search. Find the largest values of σ such that the system of inequalities Ax ≤ b, cx ≥ σ is feasible. Feasible implies that there is at least one solution. Caveat: to do binary search need to know the range of numbers. Skip for now since we need to worry about precision issues etc.
Chandra & Ruta (UIUC) CS473 24 Fall 2016 24 / 39
Suppose we have a system of m inequalities in n variables defined by Ax ≤ b How can we convince some one that there is a feasible solution? How can we convince some one that there is no feasible solution?
Chandra & Ruta (UIUC) CS473 25 Fall 2016 25 / 39
Let A ∈ Rm×n and b ∈ Rm. The system Ax ≤ b is either feasible
yA = 0 and yb < 0. In other words, if Ax ≤ b is infeasible we can demonstrate it via the following compact contradiction. Find a non-negative combination of the rows of A (given by certificate y) to derive 0 < 1. The preceding theorem can be used to prove strong duality. A fair amount of formal detail though geometric intuition is reasonable.
Chandra & Ruta (UIUC) CS473 26 Fall 2016 26 / 39
From the theorem of alternatives we can derive a useful version of Farkas lemma.
A system Ax = b, x ≥ 0 is either feasible or there is a y such that yA ≥ 0 and yb < 0. Then the following hold: Nice geometric interpretation.
Chandra & Ruta (UIUC) CS473 27 Fall 2016 27 / 39
Let x∗ be any optimum solution to primal LP Π in the canonical form max cx, Ax ≤ b, x ≥ 0 and y ∗ be an optimum solution to the dual LP Dual(Π) which is min yb, yA ≥ c, y ≥ 0. Then the following hold. If y ∗
i > 0 then n j=1 aijxj = bj (the primal constraint for row i
is tight). If x∗
j > 0 then m i=1 yiaij = cj (the dual constraint for row j is
tight). The converse also hold: if x∗ and y ∗ are primal and dual feasible and satisfy complementary slackness conditions then both must be
Very useful in various applications. Nice geometric interpretation.
Chandra & Ruta (UIUC) CS473 28 Fall 2016 28 / 39
Chandra & Ruta (UIUC) CS473 29 Fall 2016 29 / 39
Find a vector x ∈ Z d (integer values) that maximize d
j=1 cjxj
subject to d
j=1 aijxj ≤ bi
for i = 1 . . . n Input is matrix A = (aij) ∈ Rn×d, column vector b = (bi) ∈ Rn, and row vector c = (cj) ∈ Rd
Chandra & Ruta (UIUC) CS473 30 Fall 2016 30 / 39
maximize x1 + 6x2 subject to x1 ≤ 200 x2 ≤ 300 x1 + x2 ≤ 400 x1, x2 ≥ 0 Suppose we want x1, x2 to be integer valued.
Chandra & Ruta (UIUC) CS473 31 Fall 2016 31 / 39
x2 x1 300 200
1
Feasible values of x1 and x2 are integer points in shaded region
2
Optimization function is a line; moving the line until it just leaves the final integer point in feasible region, gives optimal values
Chandra & Ruta (UIUC) CS473 32 Fall 2016 32 / 39
x2 x1 300 200
1
Feasible values of x1 and x2 are integer points in shaded region
2
Optimization function is a line; moving the line until it just leaves the final integer point in feasible region, gives optimal values
Chandra & Ruta (UIUC) CS473 32 Fall 2016 32 / 39
x2 x1 300 200
1
Feasible values of x1 and x2 are integer points in shaded region
2
Optimization function is a line; moving the line until it just leaves the final integer point in feasible region, gives optimal values
Chandra & Ruta (UIUC) CS473 32 Fall 2016 32 / 39
Can model many difficult discrete optimization problems as integer programs! Therefore integer programming is a hard problem. NP-hard. Can relax integer program to linear program and approximate. Practice: integer programs are solved by a variety of methods
1
branch and bound
2
branch and cut
3
adding cutting planes
4
linear programming plays a fundamental role
Chandra & Ruta (UIUC) CS473 33 Fall 2016 33 / 39
Given undirected graph G = (V , E) a subset of nodes S ⊆ V is an independent set (also called a stable set) if for there are no edges between nodes in S. That is, if u, v ∈ S then (u, v) ∈ E. Input Graph G = (V , E) Goal Find maximum sized independent set in G
Chandra & Ruta (UIUC) CS473 34 Fall 2016 34 / 39
Given undirected graph G = (V , E) a subset of nodes S ⊆ V is a dominating set if for all v ∈ V , either v ∈ S or a neighbor of v is in S. Input Graph G = (V , E), weights w(v) ≥ 0 for v ∈ V Goal Find minimum weight dominating set in G
Chandra & Ruta (UIUC) CS473 35 Fall 2016 35 / 39
Input Graph G = (V , E), edge capacities c(e), e ∈ E. s, t ∈ V Goal Find minimum capacity s-t cut in G.
Chandra & Ruta (UIUC) CS473 36 Fall 2016 36 / 39
Suppose we know that for a linear program all vertices have integer coordinates.
Chandra & Ruta (UIUC) CS473 37 Fall 2016 37 / 39
Suppose we know that for a linear program all vertices have integer coordinates. Then solving linear program is same as solving integer program. We know how to solve linear programs efficiently (polynomial time) and hence we get an integer solution for free!
Chandra & Ruta (UIUC) CS473 37 Fall 2016 37 / 39
Suppose we know that for a linear program all vertices have integer coordinates. Then solving linear program is same as solving integer program. We know how to solve linear programs efficiently (polynomial time) and hence we get an integer solution for free! Luck or Structure:
1
Linear program for flows with integer capacities have integer vertices
2
Linear program for matchings in bipartite graphs have integer vertices
3
A complicated linear program for matchings in general graphs have integer vertices. All of above problems can hence be solved efficiently.
Chandra & Ruta (UIUC) CS473 37 Fall 2016 37 / 39
Meta Theorem: A combinatorial optimization problem can be solved efficiently if and only if there is a linear program for problem with integer vertices. Consequence of the Ellipsoid method for solving linear programming. In a sense linear programming and other geometric generalizations such as convex programming are the most general problems that we can solve efficiently.
Chandra & Ruta (UIUC) CS473 38 Fall 2016 38 / 39
1
Linear Programming is a useful and powerful (modeling) problem.
2
Can be solved in polynomial time. Practical solvers available commercially as well as in open source. Whether there is a strongly polynomial time algorithm is a major open problem.
3
Geometry and linear algebra are important to understand the structure of LP and in algorithm design. Vertex solutions imply that LPs have poly-sized optimum solutions. This implies that LP is in NP.
4
Duality is a critical tool in the theory of linear programming. Duality implies the Linear Programming is in co-NP. Do you see why?
5
Integer Programming in NP-Complete. LP-based techniques critical in heuristically solving integer programs.
Chandra & Ruta (UIUC) CS473 39 Fall 2016 39 / 39