[PPT] - IEOR E4008: Computational Discrete Optimization Yuri Faenza IEOR PowerPoint Presentation

SLIDE 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . IEOR E4008: Computational Discrete Optimization Yuri Faenza – IEOR Department Jan 23th, 2018 Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

SLIDE 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Logistics Instructor: Yuri Faenza Assistant Professor @ IEOR from 2016 Research area: Discrete Optimization Schedule: MW, 10:10-11:25 Room: 303 Mudd Office Hours: M, 5:30-7pm or by appointment – Mudd 334 Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

SLIDE 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Website Course website on Courseworks: ◮ Slides, lecture notes, links to further material,... ◮ I’ll (try to) upload the slides of each lecture before class starts, so you can print them if you want to. Piazza: Online discussions on topics of the class. Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

SLIDE 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Grading 30%: Assignments, roughly one every two weeks. 10%: Class participation. 60%: Project. Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

I

will list some projects ~

you

can come up with idea for c project, and talk to me

SLIDE 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Examples of final projects ◮ How much would we save if all taxis were replaced by car sharing? ◮ How to solve discrete optimization problems with algorithms inspired by statistical physics and genetics. ◮ How to summarize the content of documents using machine learning and submodular functions. ◮ How can we visit all streets in a neighborhood as quickly as possible? ◮ How can we build index funds using algorithms on graphs? Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

SLIDE 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Examples of assignments In an image, each pixel is represented by a 3-dimensional vector in the RGB space, i.e. each component represents the amount of one of the three primary colors (red, green blue) that appears in the pixel. Each component has value between 0 and 256. Adapt one of the algorithms seen in class to the following

problem. You are given an image as a set of pixels in the RGB space, and an

integer k. Reduce the size of the picture by representing it using at most k colors. Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

SLIDE 7 Image compression via clustering: an example Original image, 302KB 2 colors, 13KB 16 colors, 44KB 32 colors, 73KB Yuri Faenza, Columbia University IEORE4004: Optimization Models and Methods

SLIDE 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . What is this course about? Discrete optimization: choose the solution of maximum profit (or minimum cost) from a discrete family. Some discrete optimization problems you have probably seen in basic classes: ◮ Many problems on graphs (e.g. shortest path). Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

÷ :

" in .

SLIDE 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . What is this course about? Discrete optimization: choose the solution of maximum profit (or minimum cost) from a discrete family. Some discrete optimization problems you have probably seen in basic classes: ◮ Many problems on graphs (e.g. shortest path). ◮ Integer Programming. min cx Ax ≤ b x ∈ Zn Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

f

↳

integer

I Linear programming program

SLIDE 10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Why a class focused on Discrete Optimization? Many applications: production planning network design biomedicine DNA assembling machine learning ... In general, very hard to solve ⇒ not a unique approach. Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

SLIDE 11 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example: online ads ◮ Where does the 97% of Google revenue come from1? Advertising! Google Adwords: ◮ Company i chooses: ◮ a monthly budget Bi; ◮ for each word w a cost c(i, w) that the company is willing to pay to have its ad shown when word w is searched on Google. ◮ When w is searched, Google chooses a set of ads to display. If the user clicks on the ad of company i, Google gets c(i, w) from the company (unless the company has finished its budget Bi). 1Source: http://www.wordstream.com/blog/ws/2011/07/18/most-expensive-keywords-google- adwords?dt=1 Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization Bis 25$ we car di ,w )

_

70$

tire

cci , w ) = s $ = race cci ,w ) = 3$

SLIDE 12 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Which word produces the biggest revenue2? Which is the most expensive3? 2Source: https://searchenginewatch.com/2016/05/31/the-most-expensive-100-google- adwords-keywords-in-the-us/ 3Source: http://www.wordstream.com/blog/ws/2011/07/18/most-expensive-keywords-google- adwords?dt=1 Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

SLIDE 13 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Adwords problem

8 8

3 4 3 1

Who should we assign each word to? Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization WORDS TO Y OTA CAR BUDGET RACE G- H

SLIDE 14 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Adwords problem

8 8

3 4 3 1

Who should we assign each word to? Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization TOYOTA WORDS f CAR [ I

④

O 2X CAR 2 x RACE ① STRATEGY I

:/GRCED

)

Bugg , , . agg , q , µ , won , f) TO THE HIGHEST BIDDER WHO STILL HAS RACE

③

7)

G SOME BUDGET LEFT [ I I GM can → TOYOTA → G CAR → TOYOTA → 9 RACE → a- ti → I RACE → c- n

a

I

10

SLIDE 15 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Adwords problem

8 8

3 4 3 1

Who should we assign each word to? Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization TOYOTA WORDS I L CAR ⑦

yy3

2x CAR 2x RACE £

)

BUDGET ① CAR → TOYOTA

79

RACE

TSU

can → Gn → 3 [ I [ gu RACE → TOYOTA → 3 RACE → g- n → y

II

SLIDE 16 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Adwords problem

8 8

3 4 3 1

Who should we assign each word to? What if arrivals are random?

Dr. Balasubramanian Sivan, Google Research.

Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

SLIDE 17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Stable Marriage problem Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization A marriage is stable if

§

m , w : m prefers w to his partner w a m " hav x , m ,W as above is called a " blocking pain,

SLIDE 18 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Stable Marriage problem Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization my W1 is a blocking

pain

I not a stable matching

SLIDE 19 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Stable Marriage problem Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization NO BLOCKING PAIR ¥ STABLE MATCHING

SLIDE 20 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Stable Marriage problem Applications include: Allocations of students to high schools in NYC, of doctors to hospitals, of kidney donors to patients, ... Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

SLIDE 21 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A mathematical challenge From a Procter & Gamble ad from 1962: Imagine that Toody and Muldoon want to drive around the country and visit each of the 33 locations represented by dots on the contest map, and that in doing so, they want to travel the shortest possible route. You should plan a route for them from location to location which will result in the shortest total mileage from Chicago, Illi- nois back to Chicago, Illinois. 10.000 $ in 1962 correspond to ≈ 80.000 $ in today’s value. Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization TO

¥-1

SLIDE 22 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . TSP and applications Travelling Salesman Problem (TSP): find a tour of minimum length. Today’s algorithms can solve problems with tens of thousands of cities, and there are still open challenges. Applications: Pick-up & delivery, Guiding industrial machines, Mapping genomes, ... Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization FREE App SOLVING TSP INSTANCES : CONCORDE

SLIDE 23 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Discrete Optimization for Machine Learning Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

i

Ip ,

,p4 ?

' " '

"sHHGReTurj

Let f ( S) = AREA COVERED BY ACTIVATING seasons IN POSITION S

f (

sit ) +

fcsntsfcs

) + FCT)

SLIDE 24 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Discrete Optimization for Machine Learning Those functions have diminishing returns and are called Submodular Functions. Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

SLIDE 25 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . What is this course about? 1: Problems and Applications ◮ Minimum spanning tree, steiner minimum tree, bin packing. Network design and beyond. ◮ TSP: approximation algorithms, heuristics, Integer Programming and branch and bound. Extensions: vehicle routing problems. ◮ Matching problems, stable marriage problem. Applications to the New York school system and to the hospital-intern problem. Beyond stability. ◮ Online matching and online advertisement: adwords and secretary problems. ◮ Submodular functions and their applications in machine learning: MAP inference, document summarization, influence in social networks. Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

SLIDE 26 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . What is this course about? 2: Algorithms ◮ For most of those problems, we will see one or more algorithms for their

solutions. Why?

◮ Often, those algorithms contain ideas that are problem-independent and can be reused for your favourite application. ◮ How “well” we want to solve the problem depends on the application. Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

SLIDE 27 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . What is this course about? 3: Implementations We will test our algorithm in practice using: ◮ Python, ◮ Gurobi, ◮ and their interaction. Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

SLIDE 28 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Do we really need to learn all this? ◮ Objection 1: There are only a finite number of solutions to these problems. I don’t need a smart algorithm, I have a supercomputer! ◮ Objection 2: I’ll buy an expensive solver and let it do the job! Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

SLIDE 29 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Do we really need to learn all this? ◮ Objection 1: There are only a finite number of solutions to these problems. I don’t need a smart algorithm, I have a supercomputer! ◮ Objection 2: I’ll buy an expensive solver and let it do the job! Let’s prove those objections wrong! Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

SLIDE 30 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Number of solutions to TSP vs. very large sets cities 10 33 100 tours 106 1037 10158 Number of cells in a human body: ≈ 1014. Number of particles in the visible universe: ≈ 1089. Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

(

, n cities ⇒ # feasible solutions roughly n !

SLIDE 31 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Graphs, trees ◮ Graph G(V , E) Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

is

In

, . . in } n=s ↳

edges .lk#hk.vsl.k.inD

h

floss

4

f

J 3

SLIDE 32 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Graphs, trees ◮ Graph G(V , E) ◮ Degree Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

OY

degrees

SLIDE 33 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Graphs, trees ◮ Graph G(V , E) ◮ Degree ◮ Path Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

O
.

.

repetition

allowed

( of either

nodes

r

edges

)

SLIDE 34 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Graphs, trees ◮ Graph G(V , E) ◮ Degree ◮ Path ◮ Walk (open and closed) Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization as a path , but repetitions allowed , first node # last node / " " " first nodes last node '

1
pen walk

closed walk

SLIDE 35 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Graphs, trees ◮ Graph G(V , E) ◮ Degree ◮ Path ◮ Walk (open and closed) ◮ Cycle Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

/

%

every path is an

pen walk

every cycle is a closed

walk

SLIDE 36 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Graphs, trees ◮ Graph G(V , E) ◮ Degree ◮ Path ◮ Walk (open and closed) ◮ Cycle ◮ Tree Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization TREE NOT A TREE ( 7 cycle )

O

I CONNECTED GRAPH O WITH NO CYCLE A TREE ° O

(

NOT CONNECTED ) O

SLIDE 37 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Graphs, trees ◮ Graph G(V , E) ◮ Degree ◮ Path ◮ Walk (open and closed) ◮ Cycle ◮ Tree ◮ Weighted graph Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization 2 3 I I 3 4 25

!

weight can be

n the

edges

r
n

the nodes C or both)

SLIDE 38 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Graphs, trees ◮ Graph G(V , E) ◮ Degree ◮ Path ◮ Walk (open and closed) ◮ Cycle ◮ Tree ◮ Weighted graph ◮ Subgraph Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization O O O O s a subgraph

f

O O O O

[

We Cdn choose

°

any selection

f nodes

and

edges ]

SLIDE 39 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Graphs, trees ◮ Graph G(V , E) ◮ Degree ◮ Path ◮ Walk (open and closed) ◮ Cycle ◮ Tree ◮ Weighted graph ◮ Subgraph ◮ Induced Subgraph Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization O Vl O ✓ 3 ✓

20 04

s the subgraph

f

VI O

u

's induced by

VI. Vz

, Vs Va

Go

k
f

we choose a subset of " vertices , and take ¥ edges between them ]

SLIDE 40 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Graphs, trees ◮ Graph G(V , E) ◮ Degree ◮ Path ◮ Walk (open and closed) ◮ Cycle ◮ Tree ◮ Weighted graph ◮ Subgraph ◮ Induced Subgraph ◮ Multigraph Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization the same

c)

edge can

Xan

:O :

than

nce

¥

.

SLIDE 41 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . TSP: a formulation The Travelling Salesman Problem (TSP): ◮ Given: A graph G(V , E) with weights w ≥ 0 on the edges. ◮ Find: a tour – i.e. a cycle passing through all vertices of the graph – of minimum total weight. Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization cities 1

y

roads

between cities

00

O ① 5 t I tht 6 = 16 ② 2 t h t 3 t s = 14 the 16 ⇒ we prefer ②

SLIDE 42 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Greedy algorithm for TSP Input: A graph G(V , E); w : E → R≥0. L = [0]. For i = 0, . . . , n − 2: ◮ Pick j∈V \ L minimizing w(L[i], j). ◮ Set L[i + 1] = j. Output L. Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization

±

① IT MAY NOT GIVE You THG OPTIMUM ② IT HAV Bf FAL AWAY From Ttb OPTIMUM But IT IS FAST AND EASY To IMPLEMENT

SLIDE 43 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pros and cons of Greedy Pros: ◮ Fast ◮ Intuitively, seems to do “the right thing”. Cons: ◮ We have no idea how far it is from the optimum. Yuri Faenza, Columbia University IEOR E4008: Computational Discrete Optimization