VirtuCast: Multicast and Aggregation with In-Network Processing - - PowerPoint PPT Presentation

▶
virtucast multicast and aggregation with in network
SMART_READER_LITE
LIVE PREVIEW

VirtuCast: Multicast and Aggregation with In-Network Processing - - PowerPoint PPT Presentation

Mar 20, 2023 8 likes •1.09k views

VirtuCast: Multicast and Aggregation with In-Network Processing OPODIS 2013 Matthias Rost & Stefan Schmid TU Berlin & Telekom Innovation Laboratories (T-Labs) December 19th, 2013 EURECOM Matthias Rost (TU Berlin & T-Labs)

slide-1
SLIDE 1

VirtuCast: Multicast and Aggregation with In-Network Processing

OPODIS 2013 Matthias Rost & Stefan Schmid

TU Berlin & Telekom Innovation Laboratories (T-Labs)

December 19th, 2013 EURECOM

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 1

slide-2
SLIDE 2

Introducing CVSAP

Our Work in a Nutshell

Virtualization on the rise: SDN + NFV How to compute virtual aggregation / multicasting trees? Where to place in-network processing functionality? Our Answer New Model: Constrained Virtual Steiner Arborescence Problem New Algorithm: VirtuCast Objective: Jointly minimize . . . bandwidth number of processing nodes

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 2

slide-3
SLIDE 3

Introducing CVSAP

Communication Schemes: Multicast

processing = duplication + reroute

sender receiver receiver receiver processing node

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 3

slide-4
SLIDE 4

Introducing CVSAP

Communication Schemes: Multicast

processing = duplication + reroute

Figure: Hierarchy of processing nodes

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 3

slide-5
SLIDE 5

Introducing CVSAP

Communication Schemes: Aggregation

processing = merge + reroute

sender receiver processing node sender sender

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 4

slide-6
SLIDE 6

Introducing CVSAP

Communication Schemes: Aggregation

processing = merge + reroute

Figure: Hierarchy of processing nodes

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 4

slide-7
SLIDE 7

Introducing CVSAP Introductory Example

Introductory Example

Aggregation scenario grid graph with 14 senders and one receiver Virtualized links Flow can be routed along arbitrary paths receiver sender

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 5

slide-8
SLIDE 8

Introducing CVSAP Introductory Example

Without in-network processing: Unicast

Solution Method minimal cost flow Solution uses 43 edges 0 processing nodes receiver sender

Figure: Unicast solution

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 6

slide-9
SLIDE 9

Introducing CVSAP Introductory Example

With in-network processing at all nodes

Solution Method Steiner arborescence Solution uses 16 edges 9 processing nodes

receiver processing sender sender with processing node

Figure: Aggregation tree

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 7

slide-10
SLIDE 10

Introducing CVSAP Introductory Example

How to Trade-off?

vs.

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 8

slide-11
SLIDE 11

Introducing CVSAP Introductory Example

Our Solution: CVSAP & VirtuCast

Solution uses 26 edges 2 processing nodes

receiver processing sender node

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 9

slide-12
SLIDE 12

Introducing CVSAP Introductory Example

Our Solution: CVSAP & VirtuCast

Solution uses 26 edges 2 processing nodes New Model Constrained Virtual Steiner Arboresence Problem (CVSAP) New Solution Method VirtuCast algorithm

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 9

slide-13
SLIDE 13

Definition of the Constrained Virtual Steiner Arborescence Problem

slide-14
SLIDE 14

Introducing CVSAP Definition of CVSAP

Multicast Aggregation

Multicasting scenario can be reduced onto the aggregation scenario We only consider the aggregation scenario.

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 11

slide-15
SLIDE 15

Introducing CVSAP Definition of CVSAP

Input to the Constained Virtual Steiner Arborescence Problem

Graph Directed Graph G = (VG, EG) Root r ∈ VG, i.e. single receiver Terminals T ⊂ VG, i.e. sender Steiner sites S ⊂ VG, i.e. potential processing locations

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 12

slide-16
SLIDE 16

Introducing CVSAP Definition of CVSAP

Input to the Constained Virtual Steiner Arborescence Problem

Graph Directed Graph G = (VG, EG) Root r ∈ VG, i.e. single receiver Terminals T ⊂ VG, i.e. sender Steiner sites S ⊂ VG, i.e. potential processing locations Important No processing functionality can be placed on non-Steiner nodes.

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 12

slide-17
SLIDE 17

Introducing CVSAP Definition of CVSAP

Input to the Constained Virtual Steiner Arborescence Problem

Graph Directed Graph G = (VG, EG) Root r ∈ VG, i.e. single receiver Terminals T ⊂ VG, i.e. sender Steiner sites S ⊂ VG, i.e. potential processing locations Important No processing functionality can be placed on non-Steiner nodes. Costs for edges for opening Steiner sites Capacities for edges for Steiner sites & the root

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 12

slide-18
SLIDE 18

Introducing CVSAP Definition of CVSAP

CVSAP Solution

Virtual Links sender & processing nodes are connected via paths

receiver processing sender node

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 13

slide-19
SLIDE 19

Introducing CVSAP Definition of CVSAP

Solution Structure

Virtual Arborescence directed tree towards root r terminals are leaves non Steiner sites are forbidden if a Steiner site is included, processing functionality is placed edges represent paths in underlying network

Figure: Virtual Arborescence

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 14

slide-20
SLIDE 20

Introducing CVSAP Definition of CVSAP

Constrained Virtual Steiner Arborescence Problem

Definition Find a Virtual Arborescence such that Degree constraints degrees of root r and Steiner sites are bounded by ur and uS Reasoning aggregation nodes are not able to handle arbitrary many incoming flows multicasting nodes are not able to duplicate an incoming stream arbitrarily many times

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 15

slide-21
SLIDE 21

Introducing CVSAP Definition of CVSAP

Constrained Virtual Steiner Arborescence Problem

Definition Find a Virtual Arborescence such that Degree constraints Edge capacities edge capacities in the underlying network are not violated

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 15

slide-22
SLIDE 22

Introducing CVSAP Definition of CVSAP

Constrained Virtual Steiner Arborescence Problem

Definition Find a Virtual Arborescence such that Degree constraints Edge capacities minimizing sum of edge costs + sum of installation costs

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 15

slide-23
SLIDE 23

Applications

slide-24
SLIDE 24

Introducing CVSAP Applications

Applications

Network Application Technology, e.g. multicast ISP service replication / cache placement [8, 9] middleboxes / NFV + SDN backbone

  • ptical multicast [5]

ROADM1 + SDH all application-level multicast [12] different aggregation sensor network value & message aggrega- tion [4, 6] source routing ISP network analytics: Gigascope [3] middleboxes / NFV + SDN data center big data / map-reduce: Cam- doop [2] SDN

1reconfigurable optical add/drop multiplexer Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 17

slide-25
SLIDE 25

Introducing CVSAP Applications

Gigascope [3]

Problem Network monitoring and analysis of large communication networks 500 GB data / day in 2003 Idea Develop structured query language (similar to SQL) for analysis Move analysis close to source to reduce traffic Implemented and used at AT&T (2003) Modeled via aggregation CVSAP single receiver instantiates query processing nodes can perform e.g. join operations, reducing output

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 18

slide-26
SLIDE 26

Introducing CVSAP Applications

Camdoop [2]

Problem Optimize data-center for MapReduce (running on the whole system) At Facebook: in 82% of cases output has only 5.4% size of input Idea Use direct-interconnect 3D-torus network Devise disjoint set of paths for shuffle operations Aggregate values (if possible) at each node to reduce traffic Modeled via aggregation CVSAP processing nodes can perform aggregation of data receiver(s) run final reduce operation and store result

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 19

slide-27
SLIDE 27

Solution Approach

slide-28
SLIDE 28

Algorithms Solution Approach

Overview of Solution Approach

CVSAP novel problem inapproximable (if P = NP) Goal: exact algorithm solves CVSAP to optimality non-polynomial runtime

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 21

slide-29
SLIDE 29

Algorithms Solution Approach

Overview of Solution Approach

CVSAP novel problem inapproximable (if P = NP) Goal: exact algorithm solves CVSAP to optimality non-polynomial runtime Motivation for exact algorithms application dependent: allows trading-off runtime with solution quality, e.g. when designing new networks baseline for heuristics

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 21

slide-30
SLIDE 30

Algorithms Solution Approach

Overview of Solution Approach

CVSAP novel problem inapproximable (if P = NP) Goal: exact algorithm solves CVSAP to optimality non-polynomial runtime Motivation for exact algorithms application dependent: allows trading-off runtime with solution quality, e.g. when designing new networks baseline for heuristics Solution Approach: Integer Programming (IP) lower bounds are computed on-the-fly

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 21

slide-31
SLIDE 31

Algorithms Solution Approach

Our Algorithms for CVSAP

Developed two different IP formulations Multi-Commodity Flow based bad lower bounds cannot be used on large instances Single-Commodity Flow based good lower bounds can be used to solve large instances VirtuCast

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 22

slide-32
SLIDE 32

Algorithms Solution Approach

Single- vs. Multi-Commodity Flows

Single-Commodity Flow Formulation computes aggregated flow on edges independently of the origin does not represent virtual arborescence

Figure: Single-commodity

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 23

slide-33
SLIDE 33

Algorithms Solution Approach

Single- vs. Multi-Commodity Flows

Example: 6000 edges and 200 Steiner sites Single-commodity: 6000 integer variables Multi-commodity: 1,200,000 binary variables

Figure: Single-commodity Figure: Multi-commodity

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 23

slide-34
SLIDE 34

VirtuCast

slide-35
SLIDE 35

Algorithms VirtuCast

VirtuCast Algorithm

Outline of VirtuCast

1 Solve single-commodity flow IP formulation. 2 Decompose IP solution into Virtual Arborescence.

How to decompose?

(a) IP solution

→

(b) Virtual Arborescence

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 25

slide-36
SLIDE 36

IP Formulation

slide-37
SLIDE 37

Algorithms VirtuCast: IP Formulation

Extended Graph

Additional nodes source o+ sinks o−

r and o− S

Additional edges

  • −

r

  • −

S

  • +

receiver Steiner sender site

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 27

slide-38
SLIDE 38

Algorithms VirtuCast: IP Formulation

Outline of IP Formulation

Variables ∀ s ∈ S. xs ∈ {0, 1} ∀ e ∈ Eext. fe ∈ Z≥0 Constraints

1 single-commodity flow on extended graph 2 capacity constraints 3 connectivity inequalities Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 28

slide-39
SLIDE 39

Algorithms VirtuCast: IP Formulation

Outline of IP Formulation

Variables ∀ s ∈ S. xs ∈ {0, 1} ∀ e ∈ Eext. fe ∈ Z≥0 Constraints

1 single-commodity flow on extended graph

terminals receive one unit of flow activated Steiner sites receive one unit of flow flow preservation on all original nodes

2 capacity constraints 3 connectivity inequalities Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 28

slide-40
SLIDE 40

Algorithms VirtuCast: IP Formulation

Outline of IP Formulation

Variables ∀ s ∈ S. xs ∈ {0, 1} ∀ e ∈ Eext. fe ∈ Z≥0 Constraints

1 single-commodity flow on extended graph 2 capacity constraints

enforce degree constraints enforce that edge capacities hold

3 connectivity inequalities Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 28

slide-41
SLIDE 41

Algorithms VirtuCast: IP Formulation

Outline of IP Formulation

Variables ∀ s ∈ S. xs ∈ {0, 1} ∀ e ∈ Eext. fe ∈ Z≥0 Constraints

1 single-commodity flow on extended graph 2 capacity constraints 3 connectivity inequalities Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 28

slide-42
SLIDE 42

Algorithms VirtuCast: IP Formulation

Connectivity Inequalities

∀ W ⊆ VG, s ∈ W ∩ S = ∅. f (δ+

E R

ext(W )) ≥ xs

From each activated Steiner site, there exists a path towards o−

r .

Exponentially many constraints, but . . . can be separated in polynomial time.

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 29

slide-43
SLIDE 43

Algorithms VirtuCast: IP Formulation

Example

Scenario

receiver Steiner sender site

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 30

slide-44
SLIDE 44

Algorithms VirtuCast: IP Formulation

Example

Extended Graph

  • −

r

  • −

S

  • +

receiver Steiner sender site

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 30

slide-45
SLIDE 45

Algorithms VirtuCast: IP Formulation

Example

Solution

  • −

r

  • −

S

  • +

1 1 1 1 1 1 1 1 1 1 1 3

receiver Steiner sender site activated

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 30

slide-46
SLIDE 46

Decomposition Algorithm

slide-47
SLIDE 47

Algorithms VirtuCast: Decomposition Algorithm

Decomposing flow is non-trivial.

Flow solution is . . . not a tree and not a DAG [7]. Flow solution . . . contains cycles and represents arbitrary hierarchies.

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 32

slide-48
SLIDE 48

Algorithms VirtuCast: Decomposition Algorithm

Outline of Decomposition Algorithm

Iterate

1 select a terminal t 2 construct path P from t towards o−

r or o− S

3 remove one unit of flow along P 4 connect t to the second last node of P and remove t

After each iteration Problem size reduced by one.

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 33

slide-49
SLIDE 49

Algorithms VirtuCast: Decomposition Algorithm

Outline of Decomposition Algorithm

Reduced problem must be feasible Removing flow must not invalidate any connectivity inequalities. Principle: Repair & Redirect decrease flow on path edge by edge if connectivity inequalities are violated repair increment flow on edge to remain feasible redirect choose another path from the current node

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 34

slide-50
SLIDE 50

Algorithms VirtuCast: Decomposition Algorithm

Decomposition Example I

  • −

r

  • −

S

  • +

t1 v r s

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 35

slide-51
SLIDE 51

Algorithms VirtuCast: Decomposition Algorithm

Decomposition Example I P = o+, t1, v, r, o−

r

  • −

r

  • −

S

  • +

t1 v r s

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 35

slide-52
SLIDE 52

Algorithms VirtuCast: Decomposition Algorithm

Decomposition Example I P = o+, t1, v, r, o−

r

  • −

r

  • −

S

  • +

t1 v r s

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 35

slide-53
SLIDE 53

Algorithms VirtuCast: Decomposition Algorithm

Decomposition Example I P = o+, t1, v, r, o−

r

  • −

r

  • −

S

  • +

t1 v r s

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 35

slide-54
SLIDE 54

Algorithms VirtuCast: Decomposition Algorithm

Decomposition Example I P = o+, t1, v, r, o−

r

  • −

r

  • −

S

  • +

t1 v r s

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 35

slide-55
SLIDE 55

Algorithms VirtuCast: Decomposition Algorithm

Redirecting Flow

  • −

r

  • −

S

  • +

t1 v r s W

Violation of Connectivity Inequality f (δ+

E R

ext(W )) ≥ xs

∀ W ⊆ VG, s ∈ W ∩ S = ∅

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 36

slide-56
SLIDE 56

Algorithms VirtuCast: Decomposition Algorithm

Redirecting Flow

Redirection towards o−

S is possible!

There exists a path from v towards o−

S in W .

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 37

slide-57
SLIDE 57

Algorithms VirtuCast: Decomposition Algorithm

Redirecting Flow

Redirection towards o−

S is possible!

There exists a path from v towards o−

S in W .

Reasoning

1 Flow preservation holds within W . 2 s could reach o−

r via v before the reduction of flow.

3 v receives at least one unit of flow. 4 Flow leaving v must eventually terminate at o−

S .

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 37

slide-58
SLIDE 58

Algorithms VirtuCast: Decomposition Algorithm

Decomposition Example II P = o+, t1, v, s, o−

S

  • −

r

  • −

S

  • +

t1 v r s

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 38

slide-59
SLIDE 59

Algorithms VirtuCast: Decomposition Algorithm

Decomposition Example II P = o+, t1, v, s, o−

S

  • −

r

  • −

S

  • +

t1 v r s

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 38

slide-60
SLIDE 60

Algorithms VirtuCast: Decomposition Algorithm

Decomposition Example II

  • −

r

  • −

S

  • +

v r s

Solution

s

t1, v, s

t1

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 38

slide-61
SLIDE 61

Algorithms VirtuCast: Decomposition Algorithm

Decomposition Example II

  • −

r

  • −

S

  • +

v r s

Solution

s

t1, v, s

t1 t2 t3

t2, s

  • t

3

, s

  • Matthias Rost (TU Berlin & T-Labs)

VirtuCast (OPODIS ’13) EURECOM, December 2013 38

slide-62
SLIDE 62

Algorithms VirtuCast: Decomposition Algorithm

Decomposition Example II

  • −

r

  • −

S

  • +

v r

Solution

s

t1, v, s

t1 t2 t3

t2, s

  • t

3

, s

  • Matthias Rost (TU Berlin & T-Labs)

VirtuCast (OPODIS ’13) EURECOM, December 2013 38

slide-63
SLIDE 63

Algorithms VirtuCast: Decomposition Algorithm

Decomposition Example II

  • −

r

  • −

S

  • +

v r

Solution

s

t1, v, s

t1 t2 t3

t2, s

  • t

3

, s

  • Matthias Rost (TU Berlin & T-Labs)

VirtuCast (OPODIS ’13) EURECOM, December 2013 38

slide-64
SLIDE 64

Algorithms VirtuCast: Decomposition Algorithm

Decomposition Example II

Final Solution

s

  • t1

, v , s

  • t1

t2 t3

t2, s t

3

, s

r

s, v, r

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 38

slide-65
SLIDE 65

Algorithms VirtuCast: Decomposition Algorithm

Runtime of Decomposition Algorithm

Theorem Given an optimal solution, the Decompososition Algorithm computes a Virtual Arborescence in time O

  • |VG|2 · |EG| · (|VG| + |EG|)
  • .

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 39

slide-66
SLIDE 66

Proof of Correctness

slide-67
SLIDE 67

Algorithms VirtuCast: Proof of Correctness

Outline of Proof

Cost-preserving mapping

ˆ TG ∈ FCVSAP (ˆ x, ˆ f) ∈ FIP

easy via Decomposition algorithm Theorem Algorithm VirtuCast solves CVSAP to optimality.

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 41

slide-68
SLIDE 68

Implementation

slide-69
SLIDE 69

Implementation

Overview over Implementation

VirtuCast is implemented in C++ using SCIP [1]. Separation of connectivity inequalities is implemented using the Edmonds-Karp algorithm. FlowDecoRound heuristic to generate primal solutions during the branch-and-bound process [11].

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 43

slide-70
SLIDE 70

FlowDecoRound Heuristic

slide-71
SLIDE 71

Implementation FlowDecoRound Heuristic

Outline

Goal Develop a fast primal heuristic for generating solutions during the branch-and-bound process. Important Note The algorithm takes as input a fractional solution (ˆ x, ˆ f ) ∈ FLP.

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 45

slide-72
SLIDE 72

Implementation FlowDecoRound Heuristic

Outline

1 Select terminal randomly and connect it according to local flow

decomposition

If node t is connected to an inactive Steiner site s ∈ S, place s into the set of terminals.

2 Connect all unconnected terminals using shortest paths. 3 Prune active Steiner nodes. Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 46

slide-73
SLIDE 73

Implementation FlowDecoRound Heuristic

Phase 1

1 Randomly choose a terminal t

(a) Compute flow decomposition from t to o−

S , o− r with flow f (o+, t)

(b) Prune infeasible paths (c) Choose a path uniformly at random according to the flow value and connect t to the respective node. (d) If t was connected to inactive Steiner site s ∈ S, place s into the set of terminals. (e) Remove t from the set of terminals.

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 47

slide-74
SLIDE 74

Implementation FlowDecoRound Heuristic

Phase 2

2 Choose an unconnected terminal t randomly

Compute shortest path towards any of the activated Steiner nodes, while not introducing a cycle into the Virtual Arborescence. Connect node according to found shortest path or abort if no path was found.

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 48

slide-75
SLIDE 75

Implementation FlowDecoRound Heuristic

Phase 3

3 Iterate over active Steiner nodes s ∈ S in decreasing order of its cost

divided by the number of incoming connections.

Temporarily, disconnect all nodes connected to s and remove the

  • utgoing connection of s.

Try to reconnect all unconnected nodes using shortest paths. If a cheaper solution was found, accept it. Otherwise restore previous solution.

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 49

slide-76
SLIDE 76

Computational Evaluation

slide-77
SLIDE 77

Computational Evaluation Test Set

Test Set I: n × n Grid Graphs

uniform costs uniform edge costs uniform installation costs Sizes n nodes edges Steiner sites terminals 16 256 960 51 64 20 400 1520 80 100

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 51

slide-78
SLIDE 78

Computational Evaluation Test Set

Test Set II: Synthetic ISP Topologies [10]

Figure: IGen topology with 1600 nodes

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 52

slide-79
SLIDE 79

Computational Evaluation Test Set

Test Set II: Synthetic ISP Topologies [10]

non-uniform costs metric edge costs uniformly distributed installation costs Size Name nodes edges Steiner sites terminals IGen.1600 1600 6816 200 300 IGen.3200 3200 19410 400 600

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 53

slide-80
SLIDE 80

Computational Evaluation Computational Setup

Computational Setup

General 25 instances for each test set and each graph size. Terminate experiments after 2 hours of runtime. Multi-commodity flow formulation is solved with CPLEX

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 54

slide-81
SLIDE 81

VirtuCast Performance

slide-82
SLIDE 82

Computational Evaluation VirtuCast Performance

VirtuCast - Objective Gap: Grids

16 × 16 After 30 minutes: median gap around 2 % After 120 minutes: median gap around 1 %

  • 300

860 1420 2120 2820 3520 4220 4920 5620 6320 7020 1 2 3 4 5 6 time [s]

  • bjective gap [%]

16x16 Grid 1 2 3 4 5 6

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 56

slide-83
SLIDE 83

Computational Evaluation VirtuCast Performance

VirtuCast - Objective Gap: Grids

20 × 20 After 30 minutes: median gap around 4 % After 120 minutes: median gap around 3 %

  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • 300

860 1420 2120 2820 3520 4220 4920 5620 6320 7020 1 5 20 100 time [s]

  • bjective gap [%]

20x20 Grid 1 5 20 100 Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 56

slide-84
SLIDE 84

Computational Evaluation VirtuCast Performance

VirtuCast - Objective Gap: IGen

IGen.1600 After 30 minutes: gap below 0.3 % After 120 minutes: median gap below 0.1 %

  • ● ● ● ● ●
  • time [s]
  • bjective gap [%]

IGen.1600 300 860 1420 2120 2820 3520 4220 4920 5620 6320 7020 0.3 0.6 gap ≥ 1 0.2

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 57

slide-85
SLIDE 85

Computational Evaluation VirtuCast Performance

VirtuCast - Objective Gap: IGen

IGen.3200 After 30 minutes: median gap around 4 % After 120 minutes: median gap around 3 %

  • ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

time [s]

  • bjective gap [%]

IGen.3200 300 860 1420 2120 2820 3520 4220 4920 5620 6320 7020 4 8 12 ∞ 4

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 57

slide-86
SLIDE 86

Performance of FlowDecoRound

slide-87
SLIDE 87

Computational Evaluation Performance of FlowDecoRound

Performance of FlowDecoRound: Grids

  • ●
  • ● ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • 0.5

1.0 2.0 4.0 8.0 16.0 32.0 64.0 50 100 200 400 800 1600 3200 6400

time [s]

  • bjective gap [%]

Origin of Solution

  • FlowDecoRound

Integrality (LP) LP Diving (SCIP)

16x16 Grid

  • ●
  • ● ●
  • ●
  • ●
  • ●
  • ● ●
  • ●
  • 2

4 8 16 32 64 50 100 200 400 800 1600 3200 6400

time [s]

  • bjective gap [%]

Origin of Solution

  • FlowDecoRound

LP Diving (SCIP)

20x20 Grid

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 59

slide-88
SLIDE 88

Computational Evaluation Performance of FlowDecoRound

Performance of FlowDecoRound: IGen

  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • 0.025

0.050 0.100 0.200 0.400 0.800 1.600 3.200 6.400 13.200 25 50 100 200 400 800 1600 3200 6400

time [s]

  • bjective gap [%]

Origin of Solution

  • FlowDecoRound

Integrality (LP) LP Diving (SCIP)

IGen.1600

  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • ●
  • 0.2

0.4 0.8 1.6 3.2 6.4 200 400 800 1600 3200 6400

time [s]

  • bjective gap [%]

Origin of Solution

  • FlowDecoRound

LP Diving (SCIP)

IGen.3200

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 60

slide-89
SLIDE 89

Comparison with MCF

slide-90
SLIDE 90

Computational Evaluation Comparison with MCF

Computational Results of MCF

IGen.3200 Cannot be solved (efficiently) using MCF formulation: more than 6,000,000 variables IGen.1600: Strength of MCF formulation VirtuCast’s lower bound improves upon MCF’s lower bound by around 90% w.r.t to the best known solution.

1200 1700 2200 2700 3200 3700 4200 4700 5200 5700 6200 6700 7200 80 90 100

IGen.1600

time [s] relative improvement [%] 80 90 100 Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 62

slide-91
SLIDE 91

Conclusion

slide-92
SLIDE 92

Conclusion Related Work

Related Work

Molnar: Constrained Spanning Tree Problems [7] Shows that optimal solution is a ‘spanning hierarchy’ and not a DAG. Oliveira et. al: Flow Streaming Cache Placement Problem [9] Consider a weaker variant of multicasting CVSAP without bandwidth Give weak approximation algorithm Shi: Scalability in Overlay Multicasting [12] Provided heuristic and showed improvement in scalability.

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 64

slide-93
SLIDE 93

Conclusion Future Work

Future Work

Model Extensions Generalize CVSAP for multiple concurrent multicast / aggregation sessions. Try to incorporate service-chaining (EU project UNIFY). Heuristics for CVSAP Currently testing different approaches. Algorithmically challenging problem.

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 65

slide-94
SLIDE 94

Conclusion Summary

Conclusion

Motivation Network virtualization enables virtual multicasting / aggregation trees. NFV enables placement of processing functionality. Goals: Improve scalability or reduce costs. Summary Concise graph theoretic definition of CVSAP. Algorithm to solve CVSAP: VirtuCast. Computational Evaluation:

Feasible to solve realistically sized instances using VirtuCast. Significant Improvement over naive multi-commodity flow IP.

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 66

slide-95
SLIDE 95

Conclusion Summary

Discussion

Restriction of single-commodity flow model: no path semantics iterative aggregation of flows no control over path length / latency Advantages yields good solutions quickly models multicast scenarios accurately aggregation compression is limited (at each node) Applications to BigFoot? Can CVSAP be used to model workloads in private clouds? If not, which model extensions are necessary?

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 67

slide-96
SLIDE 96

Conclusion Summary

Thanks for your attention.

Project homepage www.net.t-labs.tu-berlin.de/~stefan/cvsap.html OPODIS ’13 link.springer.com/chapter/10.1007/978-3-319-03850-6_16 Technical Report arxiv.org/abs/1310.0346

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 68

slide-97
SLIDE 97

Other Current Work

slide-98
SLIDE 98

Other Current Work IPDPS 2014

IPDPS Paper 2014

It’s About Time: On Optimal Virtual Network Embeddings under Temporal Flexibilities Joint work with Stefan Schmid und Anja Feldmann Algorithms for embedding & scheduling virtual networks under temporal flexibilities

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 70

slide-99
SLIDE 99

Other Current Work IPDPS 2014

Outline

Temporal Virtual Network Embedding Problem Requests consisting of node allocations and link allocations need to be embedded over time Temporal specification allows for flexibility in scheduling requests Task Find embedding of requests and a schedule to . . . maximize number of embedded requests minimize makespan . . .

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 71

slide-100
SLIDE 100

Other Current Work IPDPS 2014

Contribution

Continuous-Time Requests may be scheduled at (real valued) times Avoids discretization (errors) Uses fewer variables IP formulations ∆: represents state changes only (bad idea) Σ: represent state changes explicitly (better idea) cΣ: Σ-model using symmetry & state-space reductions (best idea) Greedy Heuristic based on cΣ-model

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 72

slide-101
SLIDE 101

Other Current Work IPDPS 2014

Computational Evaluation: One-day workload

Scenario consider scenarios with 20 requests over time poisson inter-arrival time weibull duration (heavy tailed) node-mappings are fixed link-mappings are not fixed 30, 60, 90, 120, . . . , 300 minutes of flexibility Task Decide which requests to embed & when to embed & how to route flow.

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 73

slide-102
SLIDE 102

Other Current Work IPDPS 2014

Objective Gap

  • bjective gap [%]

∆ Σ cΣ 30 60 90 120 150 180 210 240 270 300 ∆ ΣcΣ∆ ΣcΣ∆ ΣcΣ∆ ΣcΣ∆ ΣcΣ∆ ΣcΣ∆ ΣcΣ∆ ΣcΣ∆ ΣcΣ∆ ΣcΣ∆ ΣcΣ 1 10 100 ∞ 1 10 100 ∞

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 74

slide-103
SLIDE 103

Other Current Work IPDPS 2014

Runtime

  • runtime [s]

∆ Σ cΣ 30 60 90 120 150 180 210 240 270 300 ∆ ΣcΣ∆ ΣcΣ∆ ΣcΣ∆ ΣcΣ∆ ΣcΣ∆ ΣcΣ∆ ΣcΣ∆ ΣcΣ∆ ΣcΣ∆ ΣcΣ∆ ΣcΣ 1 10 100 3600 1 10 100 3600

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 75

slide-104
SLIDE 104

Other Current Work IPDPS 2014

Benefit of Flexibility & Performance of Heuristic

  • 30

90 150 240 10 30 50 improvement of objective [%] 60 120 210 300 −20 −10 relative performance of cΣH [%]

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 76

slide-105
SLIDE 105

Other Current Work IPDPS 2014

Discussion & Future Work

Future Work Incorporate flexible duration of requests Develop heuristics for other objectives as well Evaluate our approach in conjunction with embedding heuristics Applications to BigFoot? Scheduling and routing of (time insensitive) background data transfers? Plan bandwidth intense jobs like VM migration ahead? . . .

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 77

slide-106
SLIDE 106

Other Current Work IPDPS 2014

References I

[1] T. Achterberg. SCIP: Solving Constraint Integer Programs. Mathematical Programming Computation, 1(1):1–41, 2009. [2] P. Costa, A. Donnelly, A. Rowstron, and G. O. Shea. Camdoop: Exploiting In-network Aggregation for Big Data Applications. In Proc. USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2012. [3] C. Cranor, T. Johnson, O. Spataschek, and V. Shkapenyuk. Gigascope: A Stream Database for Network Applications. In Proc. ACM SIGMOD International Conference on Management of Data, pages 647–651, 2003. [4] M. Ding, X. Cheng, and G. Xue. Aggregation tree construction in sensor networks. In Vehicular Technology Conference, 2003. VTC 2003-Fall. 2003 IEEE 58th, volume 4, pages 2168–2172. IEEE, 2003. 01285913.pdf. [5] C. Hermsmeyer, E. Hernandez-Valencia, D. Stoll, and O. Tamm. Ethernet aggregation and core network models for effcient and reliable IPTV services. Bell Labs Technical Journal, 12(1):57–76, 2007.

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 78

slide-107
SLIDE 107

Other Current Work IPDPS 2014

References II

[6] B. Krishnamachari, D. Estrin, and S. Wicker. Modelling data-centric routing in wireless sensor networks. In IEEE infocom, volume 2, pages 39–44, 2002. [7] M. Molnár. Hierarchies to Solve Constrained Connected Spanning Problems. Technical Report lrimm-00619806, University Montpellier 2, LIRMM, 2011. [8] S. Narayana, W. Jiang, J. Rexford, and M. Chiang. Joint Server Selection and Routing for Geo-Replicated Services. In Proc. Workshop on Distributed Cloud Computing (DCC), 2013. [9] C. Oliveira and P. Pardalos. Streaming Cache Placement. In Mathematical Aspects of Network Routing Optimization, Springer Optimization and Its Applications, pages 117–133. Springer New York, 2011. [10] B. Quoitin, V. Van den Schrieck, P. François, and O. Bonaventure. IGen: Generation of router-level Internet topologies through network design heuristics. In Proc. 21st International Teletraffic Congress (ITC), pages 1–8, 2009.

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 79

slide-108
SLIDE 108

Other Current Work IPDPS 2014

References III

[11] M. Rost and S. Schmid. CVSAP-Project Website. http://www.net.t-labs.tu-berlin.de/~stefan/cvsap.html, 2013. [12] S. Shi. A Proposal for A Scalable Internet Multicast Architecture. Technical Report WUCS-01-03, Washington University, 2001.

Matthias Rost (TU Berlin & T-Labs) VirtuCast (OPODIS ’13) EURECOM, December 2013 80