CS672: Approximation ALgorithms Spring 2017 Crash Course in Linear - PowerPoint PPT Presentation

CS672: Approximation ALgorithms Spring 2017 Crash Course in Linear Programming Instructor: Shaddin Dughmi

Outline Linear Programming Basics 1 Duality and Its Interpretations 2 3 Properties of Duals Weak and Strong Duality 4 5 Consequences of Duality Uses and Examples of Duality 6 Solvability of LP 7

Outline Linear Programming Basics 1 Duality and Its Interpretations 2 3 Properties of Duals Weak and Strong Duality 4 5 Consequences of Duality Uses and Examples of Duality 6 Solvability of LP 7

A Brief History The forefather of convex optimization problems, and the most ubiquitous. Developed by Kantorovich during World War II (1939) for planning the Soviet army’s expenditures and returns. Kept secret. Discovered a few years later by George Dantzig, who in 1947 developed the simplex method for solving linear programs John von Neumann developed LP duality in 1947, and applied it to game theory Polynomial-time algorithms: Ellipsoid method (Khachiyan 1979), interior point methods (Karmarkar 1984). Linear Programming Basics 1/37

LP General Form minimize (or maximize) c ⊺ x for i ∈ C 1 . a ⊺ i x ≤ b i , subject to for i ∈ C 2 . a ⊺ i x ≥ b i , for i ∈ C 3 . a ⊺ i x = b i , Decision variables: x ∈ R n Parameters: c ∈ R n defines the linear objective function a i ∈ R n and b i ∈ R define the i ’th constraint. Linear Programming Basics 2/37

Standard Form maximize c ⊺ x subject to a ⊺ i x ≤ b i , for i = 1 , . . . , m. x j ≥ 0 , for j = 1 , . . . , n. Every LP can be transformed to this form minimizing c ⊺ x is equivalent to maximizing − c ⊺ x ≥ constraints can be flipped by multiplying by − 1 Each equality constraint can be replaced by two inequalities Uconstrained variable x j can be replaced by x + j − x − j , where both x + j and x − j are constrained to be nonnegative. Linear Programming Basics 3/37

Geometric View Linear Programming Basics 4/37

Geometric View Linear Programming Basics 4/37

A 2-D example maximize x 1 + x 2 subject to x 1 + 2 x 2 ≤ 2 2 x 1 + x 2 ≤ 2 x 1 , x 2 ≥ 0 Linear Programming Basics 5/37

Application: Optimal Production n products, m raw materials Every unit of product j uses a ij units of raw material i There are b i units of material i available Product j yields profit c j per unit Facility wants to maximize profit subject to available raw materials maximize c ⊺ x subject to a ⊺ i x ≤ b i , for i = 1 , . . . , m. x j ≥ 0 , for j = 1 , . . . , n. Linear Programming Basics 6/37

Terminology Hyperplane: The region defined by a linear equality Halfspace: The region defined by a linear inequality a ⊺ i x ≤ b i . Polyhedron: The intersection of a set of linear inequalities Feasible region of an LP is a polyhedron Polytope: Bounded polyhedron Equivalently: convex hull of a finite set of points Vertex: A point x is a vertex of polyhedron P if � ∃ y � = 0 with x + y ∈ P and x − y ∈ P Face of P : The intersection with P of a hyperplane H disjoint from the interior of P Linear Programming Basics 7/37

Basic Facts about LPs and Polyhedrons Fact Feasible regions of LPs (i.e. polyhedrons) are convex Linear Programming Basics 8/37

Basic Facts about LPs and Polyhedrons Fact Feasible regions of LPs (i.e. polyhedrons) are convex Fact Set of optimal solutions of an LP is convex In fact, a face of the polyhedron intersection of P with hyperplane c ⊺ x = OPT Linear Programming Basics 8/37

Basic Facts about LPs and Polyhedrons Fact Feasible regions of LPs (i.e. polyhedrons) are convex Fact Set of optimal solutions of an LP is convex In fact, a face of the polyhedron intersection of P with hyperplane c ⊺ x = OPT Fact At a vertex, n linearly independent constraints are satisfied with equality (a.k.a. tight) Linear Programming Basics 8/37

Basic Facts about LPs and Polyhedrons Fact An LP either has an optimal solution, or is unbounded or infeasible Linear Programming Basics 9/37

Fundamental Theorem of LP If an LP in standard form has an optimal solution, then it has a vertex optimal solution. Linear Programming Basics 10/37

Fundamental Theorem of LP If an LP in standard form has an optimal solution, then it has a vertex optimal solution. Proof Assume not, and take a non-vertex optimal solution x with the maximum number of tight constraints Linear Programming Basics 10/37

Fundamental Theorem of LP If an LP in standard form has an optimal solution, then it has a vertex optimal solution. Proof Assume not, and take a non-vertex optimal solution x with the maximum number of tight constraints There is y � = 0 s.t. x ± y are feasible Linear Programming Basics 10/37

Fundamental Theorem of LP If an LP in standard form has an optimal solution, then it has a vertex optimal solution. Proof Assume not, and take a non-vertex optimal solution x with the maximum number of tight constraints There is y � = 0 s.t. x ± y are feasible y is perpendicular to the objective function and the tight constraints at x . i.e. c ⊺ y = 0 , and a ⊺ i y = 0 whenever the i ’th constraint is tight for x . Linear Programming Basics 10/37

Fundamental Theorem of LP If an LP in standard form has an optimal solution, then it has a vertex optimal solution. Proof Assume not, and take a non-vertex optimal solution x with the maximum number of tight constraints There is y � = 0 s.t. x ± y are feasible y is perpendicular to the objective function and the tight constraints at x . i.e. c ⊺ y = 0 , and a ⊺ i y = 0 whenever the i ’th constraint is tight for x . Can choose y s.t. y j < 0 for some j Linear Programming Basics 10/37

Fundamental Theorem of LP If an LP in standard form has an optimal solution, then it has a vertex optimal solution. Proof Assume not, and take a non-vertex optimal solution x with the maximum number of tight constraints There is y � = 0 s.t. x ± y are feasible y is perpendicular to the objective function and the tight constraints at x . i.e. c ⊺ y = 0 , and a ⊺ i y = 0 whenever the i ’th constraint is tight for x . Can choose y s.t. y j < 0 for some j Let α be the largest constant such that x + αy is feasible Such an α exists Linear Programming Basics 10/37

Fundamental Theorem of LP If an LP in standard form has an optimal solution, then it has a vertex optimal solution. Proof Assume not, and take a non-vertex optimal solution x with the maximum number of tight constraints There is y � = 0 s.t. x ± y are feasible y is perpendicular to the objective function and the tight constraints at x . i.e. c ⊺ y = 0 , and a ⊺ i y = 0 whenever the i ’th constraint is tight for x . Can choose y s.t. y j < 0 for some j Let α be the largest constant such that x + αy is feasible Such an α exists An additional constraint becomes tight at x + αy , a contradiction. Linear Programming Basics 10/37

Counting non-zero Variables Corollary If an LP in standard form has an optimal solution, then there is an optimal solution with at most m non-zero variables. maximize c ⊺ x a ⊺ i x ≤ b i , subject to for i = 1 , . . . , m. x j ≥ 0 , for j = 1 , . . . , n. e.g. for optimal production with n products and m raw materials, there is an optimal plan with at most m products. Linear Programming Basics 11/37

Outline Linear Programming Basics 1 Duality and Its Interpretations 2 3 Properties of Duals Weak and Strong Duality 4 5 Consequences of Duality Uses and Examples of Duality 6 Solvability of LP 7

Linear Programming Duality Primal LP Dual LP maximize c ⊺ x minimize b ⊺ y subject to Ax ≤ b subject to A ⊺ y = c y ≥ 0 A ∈ R m × n , c ∈ R n , b ∈ R m y i is the dual variable corresponding to primal constraint A i x ≤ b i A T j y ≥ c j is the dual constraint corresponding to primal variable x j Duality and Its Interpretations 12/37

Linear Programming Duality: Standard Form, and Visualization Primal LP Dual LP maximize c ⊺ x minimize y ⊺ b subject to Ax ≤ b subject to A ⊺ y ≥ c x ≥ 0 y ≥ 0 Duality and Its Interpretations 13/37

Linear Programming Duality: Standard Form, and Visualization Primal LP Dual LP maximize c ⊺ x minimize y ⊺ b subject to Ax ≤ b subject to A ⊺ y ≥ c x ≥ 0 y ≥ 0 x 1 x 2 x 3 x 4 y 1 a 11 a 12 a 13 a 14 b 1 y 2 a 21 a 22 a 23 a 24 b 2 y 3 a 31 a 32 a 33 a 34 b 3 c 1 c 2 c 3 c 4 Duality and Its Interpretations 13/37

Linear Programming Duality: Standard Form, and Visualization Primal LP Dual LP maximize c ⊺ x minimize y ⊺ b subject to Ax ≤ b subject to A ⊺ y ≥ c x ≥ 0 y ≥ 0 x 1 x 2 x 3 x 4 y 1 a 11 a 12 a 13 a 14 b 1 y 2 a 21 a 22 a 23 a 24 b 2 y 3 a 31 a 32 a 33 a 34 b 3 c 1 c 2 c 3 c 4 y i is the dual variable corresponding to primal constraint A i x ≤ b i A T j y ≥ c j is the dual constraint corresponding to primal variable x j Duality and Its Interpretations 13/37

Interpretation 1: Economic Interpretation Recall the Optimal Production problem from last lecture n products, m raw materials Every unit of product j uses a ij units of raw material i There are b i units of material i available Product j yields profit c j per unit Facility wants to maximize profit subject to available raw materials Duality and Its Interpretations 14/37

Interpretation 1: Economic Interpretation Primal LP � n max j =1 c j x j � n j =1 a ij x j ≤ b i , for i ∈ [ m ] . s.t. x j ≥ 0 , for j ∈ [ n ] . Duality and Its Interpretations 14/37