Linear Programming Illustration Courtesy: Kevin Wayne & Denis - PowerPoint PPT Presentation

CSC373 Week 6: Linear Programming Illustration Courtesy: Kevin Wayne & Denis Pankratov 373F19 - Nisarg Shah & Karan Singh 1

Recap • Network flow ➢ Ford-Fulkerson algorithm ➢ Ways to make the running time polynomial ➢ Correctness using max-flow, min-cut ➢ Applications: o Edge-disjoint paths o Multiple sources/sinks o Circulation o Circulation with lower bounds o Survey design o Image segmentation 373F19 - Nisarg Shah & Karan Singh 2

Brewery Example • A brewery can invest its inventory of corn, hops and malt into producing some amount of ale and some amount of beer ➢ Per unit resource requirement and profit of the two items are as given below Example Courtesy: Kevin Wayne 373F19 - Nisarg Shah & Karan Singh 3

Brewery Example • Suppose it produces 𝐵 units of ale and 𝐶 units of beer • Then we want to solve this program: 373F19 - Nisarg Shah & Karan Singh 4

Linear Function • 𝑔: ℝ 𝑜 → ℝ is a linear function if 𝑔 𝑦 = 𝑏 𝑈 𝑦 for some 𝑏 ∈ ℝ 𝑜 𝑈 3 𝑦 1 ➢ Example: 𝑔 𝑦 1 , 𝑦 2 = 3𝑦 1 − 5𝑦 2 = −5 𝑦 2 • Linear constraints: ➢ For a linear function 𝑕: ℝ 𝑜 → ℝ and 𝑑 ∈ ℝ , 𝑕 𝑦 = 𝑑 ➢ Line in the plane (or a hyperplane in ℝ 𝑜 ) ➢ Example: 5𝑦 1 + 7𝑦 2 = 10 373F19 - Nisarg Shah & Karan Singh 5

Linear Function • Geometrically, 𝑏 is the normal vector of the line(or hyperplane) represented by 𝑏 𝑈 𝑦 = 𝑑 373F19 - Nisarg Shah & Karan Singh 6

Linear Inequality • 𝑏 𝑈 𝑦 ≤ 𝑑 represents a “half - space” 373F19 - Nisarg Shah & Karan Singh 7

Linear Programming • Maximize/minimize a linear function subject to linear equality/inequality constraints 373F19 - Nisarg Shah & Karan Singh 8

Geometrically… 373F19 - Nisarg Shah & Karan Singh 9

Back to Brewery Example 373F19 - Nisarg Shah & Karan Singh 10

Back to Brewery Example 373F19 - Nisarg Shah & Karan Singh 11

Optimal Solution At A Vertex • Claim: Regardless of the objective function, the optimal solution must be at a vertex 373F19 - Nisarg Shah & Karan Singh 12

Convexity • Convex set 𝑇 : If 𝑦, 𝑧 ∈ 𝑇 and 𝜇 ∈ [0,1] , then 𝜇𝑦 + 1 − 𝜇 𝑧 ∈ 𝑇 too. • Vertex: A point which cannot be written as a strict convex combination of any two points in the set • Observation: Feasible region of an LP is a convex set 373F19 - Nisarg Shah & Karan Singh 13

Optimal Solution At A Vertex • Proof intuition: ➢ If 𝑦 is not a vertex, we can move towards the boundary in a direction where the objective value does not decrease o Take some direction 𝑒 such that you can move by at least 𝜗 in both 𝑒 and −𝑒 directions while remaining within the region o Objective must not decrease in at least one of {𝑒, −𝑒} directions ➢ Reach a point that is “tight” for at least one more constraint ➢ Repeat until we are at a vertex 373F19 - Nisarg Shah & Karan Singh 14

LP, Standard Formulation • Input: 𝑑, 𝑏 1 , 𝑏 2 , … , 𝑏 𝑛 ∈ ℝ 𝑜 , 𝑐 ∈ ℝ 𝑛 ➢ There are 𝑜 variables and 𝑛 constraints • Goal: 373F19 - Nisarg Shah & Karan Singh 15

LP, Standard Matrix Form • Input: 𝑑, 𝑏 1 , 𝑏 2 , … , 𝑏 𝑛 ∈ ℝ 𝑜 , 𝑐 ∈ ℝ 𝑛 ➢ There are 𝑜 variables and 𝑛 constraints • Goal: 373F19 - Nisarg Shah & Karan Singh 16

Convert to Standard Form • What if the LP is not in standard form? ➢ Constraints that use ≥ o 𝑏 𝑈 𝑦 ≥ 𝑐 ⇔ −𝑏 𝑈 𝑦 ≤ −𝑐 ➢ Constraints that use equality o 𝑏 𝑈 𝑦 = 𝑐 ⇔ 𝑏 𝑈 𝑦 ≤ 𝑐, 𝑏 𝑈 𝑦 ≥ 𝑐 ➢ Objective function is a minimization o Minimize 𝑑 𝑈 𝑦 ⇔ Maximize −𝑑 𝑈 𝑦 ➢ Variable is unconstrained o 𝑦 with no constraint ⇔ Replace 𝑦 by two variables 𝑦 ′ and 𝑦 ′′ , replace every occurrence of 𝑦 with 𝑦 ′ − 𝑦 ′′ , and add constraints 𝑦 ′ ≥ 0, 𝑦 ′′ ≥ 0 373F19 - Nisarg Shah & Karan Singh 17

LP Transformation Example 373F19 - Nisarg Shah & Karan Singh 18

Optimal Solution • Does this LP always have an optimal solution? • No! The LP can fail for two reasons 1. It is infeasible , i.e. 𝑦 𝐵𝑦 ≤ 𝑐} = ∅ o Example: 𝑦 1 ≤ 1 and 𝑦 1 ≥ 2 (or −𝑦 1 ≤ −2 ) constraints 2. It is unbounded , i.e. you can get arbitrarily large or small objective values o Example: maximize 𝑦 1 subject to 𝑦 1 ≥ 0 • We know that if the LP has an optimal solution, it must be at a vertex. 373F19 - Nisarg Shah & Karan Singh 19

Simplex Algorithm • Simple algorithm, easy to specify geometrically • Worst-case running time is exponential • Excellent performance in practice 373F19 - Nisarg Shah & Karan Singh 20

Simplex: Geometric View 373F19 - Nisarg Shah & Karan Singh 21

Algorithmic Implementation Move to a neighbor vertex with better objective value Start at a Is there a vertex of neighbor vertex feasible with better polytope objective value? Terminate, declare the current solution and value as optimal 373F19 - Nisarg Shah & Karan Singh 22

How Do We Implement This? • We’ll work with the slack form of LP ➢ Convenient for implementing simplex operations ➢ We want to maximize 𝑨 in the slack form, but for now, forget about the maximization objective 373F19 - Nisarg Shah & Karan Singh 23

Slack Form 373F19 - Nisarg Shah & Karan Singh 24

Slack Form 373F19 - Nisarg Shah & Karan Singh 25

Simplex: Step 1 • Start at a feasible vertex ➢ How do we find a feasible vertex? ➢ For now, assume 𝑐 ≥ 0 (each 𝑐 𝑗 ≥ 0 ) o In this case, 𝑦 = 0 is a feasible vertex. o In the slack form, this means setting the nonbasic variables to 0 ➢ We’ll later see what to do in the general case 373F19 - Nisarg Shah & Karan Singh 26

Simple: Step 2 • What next? Let’s look at an example • To increase the value of 𝑨 : ➢ Find a nonbasic variable with a positive coefficient o This is called an entering variable ➢ See how much you can increase its value without violating any constraints 373F19 - Nisarg Shah & Karan Singh 27

Simple: Step 2 This is because the current values of 𝑦 2 and 𝑦 3 are 0 , and we need 𝑦 4 , 𝑦 5 , 𝑦 6 ≥ 0 373F19 - Nisarg Shah & Karan Singh 28

Simple: Step 2 Tightest obstacle ➢ Solve the tightest obstacle for the nonbasic variable 𝑦 1 = 9 − 𝑦 2 4 − 𝑦 3 2 − 𝑦 6 4 o Substitute the entering variable (called pivot) in other equations o Now 𝑦 1 becomes basic and 𝑦 6 becomes non-basic o 𝑦 6 is called the leaving variable 373F19 - Nisarg Shah & Karan Singh 29

Simplex: Step 2 • After one iteration of this step: ➢ The basic feasible solution (i.e. substituting 0 for all nonbasic variables) improves from 𝑨 = 0 to 𝑨 = 27 • Repeat! 373F19 - Nisarg Shah & Karan Singh 30

Simplex: Step 2 373F19 - Nisarg Shah & Karan Singh 31

Simplex: Step 2 373F19 - Nisarg Shah & Karan Singh 32

Simplex: Step 2 • There is no leaving variable (nonbasic variable with positive coefficient). • What now? Nothing! We are done. • Take the basic feasible solution ( 𝑦 3 = 𝑦 5 = 𝑦 6 = 0 ). • Gives the optimal value 𝑨 = 28 • In the optimal solution, 𝑦 1 = 8 , 𝑦 2 = 4, 𝑦 3 = 0 373F19 - Nisarg Shah & Karan Singh 33

Simplex Overview Move to a neighbor vertex with better objective value Start at a Is there a vertex of neighbor vertex feasible with better polytope objective value? Terminate, declare the current solution and value as optimal 373F19 - Nisarg Shah & Karan Singh 34

Simplex Overview Move to a neighbor vertex with better objective value Is there a Assuming 𝑐 ≥ neighbor vertex 0 , start with a basic feasible with better solution objective value? Terminate, declare the current solution and value as optimal 373F19 - Nisarg Shah & Karan Singh 35

Simplex Overview Move to a neighbor vertex with better objective value Assuming 𝑐 ≥ Is there a leaving 0 , start with a variable? (coefficient > 0 in basic feasible 𝑨 ) solution Terminate, declare the current solution and value as optimal 373F19 - Nisarg Shah & Karan Singh 36

Simplex Overview Pivot on a leaving variable Assuming 𝑐 ≥ Is there a leaving 0 , start with a variable? (coefficient > 0 in basic feasible 𝑨 ) solution Terminate, declare the current solution and value as optimal 373F19 - Nisarg Shah & Karan Singh 37

Simplex Overview Pivot on a leaving variable Assuming 𝑐 ≥ Is there a leaving 0 , start with a variable? (coefficient > 0 in basic feasible 𝑨 ) solution Terminate, declare optimal value 373F19 - Nisarg Shah & Karan Singh 38