A Distributed Algorithm for Large-Scale Generalized Matching Faraz - - PowerPoint PPT Presentation

Faraz Makari, Baruch Awerbuch, Rainer Gemulla, Rohit Khandekar, Julián Mestre, Mauro Sozio

A Distributed Algorithm for Large-Scale Generalized Matching

Recommender systems

1

? ? 2 ? ? ? 4 ? ? ? 5 ?

Given:

• A user-item feedback matrix

Goal:

• Recommend additional

items users may like

Recommender systems

4 1 2 5 1 3 4 4 2 3 5 2

1

Given:

• A user-item feedback matrix

Goal:

• Recommend additional

items users may like

Approach:

I. Predict missing ratings

Recommender systems

4 1 2 5 1 3 4 4 2 3 5 2

1

Given:

• A user-item feedback matrix

Goal:

• Recommend additional

items users may like

Approach:

I. Predict missing ratings

• II. Recommend items with

highest predicted ratings

Recommender systems

4 1 2 5 1 3 4 4 2 3 5 2

1

Given:

• A user-item feedback matrix

Goal:

• Recommend additional

items users may like

Approach:

I. Predict missing ratings

• II. Recommend items with

highest predicted ratings

Recommender systems

4 1 2 5 1 3 4 4 2 3 5 2

1

Given:

• A user-item feedback matrix

Goal:

• Recommend additional

items users may like

Approach:

I. Predict missing ratings

• II. Recommend items with

highest predicted ratings How to recommend items under constraints?

Generalized Bipartite Matching (GBM)

2

# Recommendation = 1 Available DVD per movie = 2

4 Constraints:

• Neither too few nor too many recommendations
• Number of DVDs limited
• …

1 5 1 3 4 2 3 2

Users Predicted ratings DVDs

Generalized Bipartite Matching (GBM)

2

Constraints:

• Neither too few nor too many recommendations
• Number of DVDs limited
• …

# Recommendation = 1

4 1 5 1 3 4 2 3 2

Users Predicted ratings DVDs Available DVD per movie = 2

Generalized Bipartite Matching (GBM)

1 ≤ # Recommendations ≤ 2

Goal

• Recommend items to users s.t.

I.

Interesting for users

II.

Neither too few nor too many

• III. Availability of items satisfied

3

Maximum weight matching LB and UB constraints

4 1 5 1 3 4 2 3 2

Users Predicted ratings DVDs Available DVD per movie = 2 s.t.

Generalized Bipartite Matching (GBM)

3

1 ≤ # Recommendations ≤ 2

4 1 5 1 3 4 2 3 2

Users Predicted ratings DVDs Available DVD per movie = 2

Goal

• Recommend items to users s.t.

I.

Interesting for users

II.

Neither too few nor too many

• III. Availability of items satisfied

Maximum weight matching LB and UB constraints s.t.

Challenge

GBM optimally solvable in polynomial time

• E.g., linear programming
• Available solvers handle small instances very well

4

Challenge

GBM optimally solvable in polynomial time

• E.g., linear programming
• Available solvers handle small instances very well

Real applications can be large

• E.g., Netflix has >20M users, >20k movies
• Available solvers do not scale to large problems

4

Challenge

GBM optimally solvable in polynomial time

• E.g., linear programming
• Available solvers handle small instances very well

Real applications can be large

• E.g., Netflix has >20M users, >20k movies
• Available solvers do not scale to large problems

Goal:

• Efficient and scalable algorithm for large-scale GBM instances

4

Framework

Phase 1: Approximate LP

• Compute “edge probabilities’’

using linear programming

5

Phase 1

2

0.7 0.4 0.9 0.3 0.5 0.7 0.3 0.6 0.4

Framework

Phase 1: Approximate LP

• Compute “edge probabilities’’

using linear programming

Phase 2: Round

• Select edges based on

probabilities from phase 1

1 1 1 1 1 1 Phase 2

5

2

Phase 1 0.7 0.4 0.9 0.3 0.5 0.7 0.3 0.6 0.4

Phase 1: Computing edge probabilities

Algorithm for Mixed Packing Covering (MPC) LPs (like GBM LP)

• Gradient-based multiplicative weights update algorithm
• Approximately solves MPC LPs (ε: approx. parameter)
• LB and UB constraints satisfied up to (1±ε) Almost feasible
• Objective value (1-ε) of the optimum Near-optimal
• Poly-log rounds
• Easy to implement: Each round involves matrix-vector multiplications

6

• Contrib. 1

Phase 1: Computing edge probabilities

Algorithm for Mixed Packing Covering (MPC) LPs (like GBM LP)

7

• Contrib. 1

0.9 0.5 0.6 0.2 0.1 0.2 0.2 0.8 0.5 1 ≤ deg ≤ 2 0 ≤ deg ≤ 2

Phase 1: Computing edge probabilities

Algorithm for Mixed Packing Covering (MPC) LPs (like GBM LP)

8

• Contrib. 1

0.9 0.5 0.6 0.2 0.1 0.2 0.2 0.8 0.5 1 ≤ deg ≤ 2 1.3 0.9 0.4 1.3 0 ≤ deg ≤ 2 2.3 0.9 0.8

Phase 1: Computing edge probabilities

Algorithm for Mixed Packing Covering (MPC) LPs (like GBM LP)

9

• Contrib. 1

0.6 0.6 0.8 0.3 0.4 0.5 0.2 0.7 0.7 1 ≤ deg ≤ 2 1.2 1.5 0.7 1.4 0 ≤ deg ≤ 2 2.1 1.4 1.3

Phase 1: Computing edge probabilities

Algorithm for Mixed Packing Covering (MPC) LPs (like GBM LP)

10

• Contrib. 1

0.7 0.4 0.9 0.3 0.5 0.7 0.3 0.4 0.6 1 ≤ deg ≤ 2 1.1 1.7 1.0 1.0 0 ≤ deg ≤ 2 2.0 1.4 1.4

Phase 1: Computing edge probabilities

Algorithm for Mixed Packing Covering (MPC) LPs (like GBM LP) Distributed Processing for GBM (details in paper)

• Communication depends on # nodes not # edges
• All computation in parallel

10

• Contrib. 1

0.7 0.4 0.9 0.3 0.5 0.7 0.3 0.4 0.6 1 ≤ deg ≤ 2 1.1 1.7 1.0 1.0 0 ≤ deg ≤ 2 2.0 1.4 1.4

• Contrib. 2

Phase 2: Selecting edges

• Given: Edge probabilities from phase 1
• Goal: Select edges to be in final solution s. t.

I.

LB and UB constraints satisfied (up to rounding)

II.

• Approx. guarantee preserved (in expectation)

0.7 0.4 0.9 0.3 0.5 0.7 0.3 0.4 0.6

11

1.1 1.7 1.0 1.0 2.0 1.4 1.4

Phase 2: Selecting edges

• Given: Edge probabilities from phase 1
• Goal: Select edges to be in final solution s. t.

I.

LB and UB constraints satisfied (up to rounding)

II.

• Approx. guarantee preserved (in expectation)

Naïve approach: Round independently using prob. from phase 1

0.7 0.4 0.9 0.3 0.5 0.7 0.3 0.4 0.6

11

1.1 1.7 1.0 1.0 2.0 1.4 1.4

Phase 2: Selecting edges

• Given: Edge probabilities from phase 1
• Goal: Select edges to be in final solution s. t.

I.

LB and UB constraints satisfied (up to rounding) ✖

II.

• Approx. guarantee preserved (in expectation) ✔

Naïve approach: Round independently using prob. from phase 1

0.7 0.4 0.9 0.3 0.5 0.7 0.3 0.4 0.6

11

1.1 1.7 1.0 1.0 2.0 1.4 1.4

Phase 2: Selecting edges

0.7 0.4 0.9 0.3 0.5 0.7 0.3 0.4 0.6

12

1.1 1.7 1.0 1.0 2.0 1.4 1.4

• Given: Edge probabilities from phase 1
• Goal: Select edges to be in final solution s. t.

I.

LB and UB constraints satisfied (up to rounding)

II.

• Approx. guarantee preserved (in expectation)
• Seq. algorithm [Gandhi et al. 06]:

Phase 2: Selecting edges

0.7 0.4 0.9 0.3 0.5 0.7 0.3 0.4 0.6

13

1.1 1.7 1.0 1.0 2.0 1.4 1.4

• Given: Edge probabilities from phase 1
• Goal: Select edges to be in final solution s. t.

I.

LB and UB constraints satisfied (up to rounding)

II.

• Approx. guarantee preserved (in expectation)
• Seq. algorithm [Gandhi et al. 06]:
• 1. Find a cycle/maximal path

Phase 2: Selecting edges

0.7 0.4 0.9 0.3 0.5 0.7 0.3 0.4 0.6

14

1.1 1.7 1.0 1.0 2.0 1.4 1.4

• Given: Edge probabilities from phase 1
• Goal: Select edges to be in final solution s. t.

I.

LB and UB constraints satisfied (up to rounding)

II.

• Approx. guarantee preserved (in expectation)
• Seq. algorithm [Gandhi et al. 06]:
• 1. Find a cycle/maximal path
• 2. Modify edge prob. on the

cycle/maximal path (details omitted)

• If edge prob. = 1, include in solution
• If edge prob. = 0, remove from graph

Cycle

0.7 0.4 1 0.3 0.4 0.7 0.3 0.3 0.7

15

1.1 1.7 1.0 1.0 2.0 1.4 1.4

Phase 2: Selecting edges

• Given: Edge probabilities from phase 1
• Goal: Select edges to be in final solution s. t.

I.

LB and UB constraints satisfied (up to rounding)

II.

• Approx. guarantee preserved (in expectation)
• Seq. algorithm [Gandhi et al. 06]:
• 1. Find a cycle/maximal path
• 2. Modify edge prob. on the

cycle/maximal path (details omitted)

• If edge prob. = 1, include in solution
• If edge prob. = 0, remove from graph

Cycle

0.7 0.4 1 0.3 0.4 0.7 0.3 0.3 0.7

16

1.1 1.7 1.0 1.0 2.0 1.4 1.4

Phase 2: Selecting edges

• Given: Edge probabilities from phase 1
• Goal: Select edges to be in final solution s. t.

I.

LB and UB constraints satisfied (up to rounding)

II.

• Approx. guarantee preserved (in expectation)
• Seq. algorithm [Gandhi et al. 06]:
• 1. Find a cycle/maximal path
• 2. Modify edge prob. on the

cycle/maximal path (details omitted)

• If edge prob. = 1, include in solution
• If edge prob. = 0, remove from graph
• 3. Repeat until graph is empty

Cycle

1 0.1 1

17

2.0 1.4 1.4 1.1 1.7 1.0 1.0 1

Phase 2: Selecting edges

0.3 0.4 1

• Given: Edge probabilities from phase 1
• Goal: Select edges to be in final solution s. t.

I.

LB and UB constraints satisfied (up to rounding)

II.

• Approx. guarantee preserved (in expectation)
• Seq. algorithm [Gandhi et al. 06]:
• 1. Find a cycle/maximal path
• 2. Modify edge prob. on the

cycle/maximal path (details omitted)

• If edge prob. = 1, include in solution
• If edge prob. = 0, remove from graph
• 3. Repeat until graph is empty

Maximal path

1 0.4 1

18

2.0 1.4 1.7 1.4 1.7 1.0 1.0 1

Phase 2: Selecting edges

0.7 1

• Given: Edge probabilities from phase 1
• Goal: Select edges to be in final solution s. t.

I.

LB and UB constraints satisfied (up to rounding)

II.

• Approx. guarantee preserved (in expectation)
• Seq. algorithm [Gandhi et al. 06]:
• 1. Find a cycle/maximal path
• 2. Modify edge prob. on the

cycle/maximal path (details omitted)

• If edge prob. = 1, include in solution
• If edge prob. = 0, remove from graph
• 3. Repeat until graph is empty

Maximal path

1 1 1

19

2.0 2.0 2.0 2.0 2.0 1.0 1.0 1

Phase 2: Selecting edges

1 1

• Given: Edge probabilities from phase 1
• Goal: Select edges to be in final solution s. t.

I.

LB and UB constraints satisfied (up to rounding)

II.

• Approx. guarantee preserved (in expectation)
• Seq. algorithm [Gandhi et al. 06]:
• 1. Find a cycle/maximal path
• 2. Modify edge prob. on the

cycle/maximal path (details omitted)

• If edge prob. = 1, include in solution
• If edge prob. = 0, remove from graph
• 3. Repeat until graph is empty

1 1 1

20

2.0 2.0 2.0 2.0 2.0 1.0 1.0 1

Phase 2: Selecting edges

1 1

• Given: Edge probabilities from phase 1
• Goal: Select edges to be in final solution s. t.

I.

LB and UB constraints satisfied (up to rounding) ✔

II.

• Approx. guarantee preserved (in expectation) ✔
• Seq. algorithm [Gandhi et al. 06]:
• 1. Find a cycle/maximal path
• 2. Modify edge prob. on the

cycle/maximal path (details omitted)

• If edge prob. = 1, include in solution
• If edge prob. = 0, remove from graph
• 3. Repeat until graph is empty

Phase 2: Selecting edges

21

How to distribute?

• A local cycle is a global cycle
• A local maximal path is not

a global maximal path

• Order of processing cycles has no

affect on approx. guarantess

• Contrib. 3

Partition Partition

Phase 2: Selecting edges

22

• Contrib. 3

Distributed algorithm:

• 1. Partition edges uniformly across

compute nodes

• 2. Process all cycles in each partition
• 3. Merge all partitions
• 4. Repeat until graph is “small enough”
• 5. On last partition: Process all cycles

and maximal paths using seq. alg.

Partition Partition

Phase 2: Selecting edges

23

• Contrib. 3

Partition Partition

Distributed algorithm:

• 1. Partition edges uniformly across

compute nodes

• 2. Process all cycles in each partition
• 3. Merge all partitions
• 4. Repeat until graph is “small enough”
• 5. On last partition: Process all cycles

and maximal paths using seq. alg.

Large-Scale (Semi-) Synthetic Experiments

Dataset # users # items # edges Netflix-predicted 490k 18k 3.2B Synthetic 10M 1M 1B

Gurobi ran out of memory on a large-memory server with 512GB RAM

24

Cluster of commodity nodes

Nodes x cores Total time (h) 1x8 2x8 4x8 8x8 16x8 10 20 30 40 50 60

Rounding Netflix−predicted Fractional solution Netflix−predicted Rounding Synthetic Fractional solution Synthetic

Insufficient memory Insufficient memory Insufficient memory Insufficient memory Insufficient memory

Powerful server

Large-Scale (Semi-) Synthetic Experiments

Dataset # users # items # edges Netflix-predicted 490k 18k 3.2B Synthetic 10M 1M 1B

Gurobi ran out of memory on a large-memory server with 512GB RAM

Scales to very large problem instances

24

Cluster of commodity nodes

Nodes x cores Total time (h) 1x8 2x8 4x8 8x8 16x8 10 20 30 40 50 60

Rounding Netflix−predicted Fractional solution Netflix−predicted Rounding Synthetic Fractional solution Synthetic

Insufficient memory Insufficient memory Insufficient memory Insufficient memory Insufficient memory

Powerful server

Summary

25

• Recommending items to users under constraints
• Contributions:
• A scalable distributed algorithm for large-scale GBM
• A simple and efficient algorithm for general MPC LPs
• Effective distributed processing for GBM
• A distributed randomized rounding for GBM
• Experiments indicate scalability to instances with millions

and nodes and billions of edges

Summary

25

• Recommending items to users under constraints
• Contributions:
• A scalable distributed algorithm for large-scale GBM
• A simple and efficient algorithm for general MPC LPs
• Effective distributed processing for GBM
• A distributed randomized rounding for GBM
• Experiments indicate scalability to instances with millions

and nodes and billions of edges

Thank you Questions?