Achieving High Utilization with Software-Driven WAN Chi-Yao Hong - - PowerPoint PPT Presentation

achieving high utilization with software driven wan
SMART_READER_LITE
LIVE PREVIEW

Achieving High Utilization with Software-Driven WAN Chi-Yao Hong - - PowerPoint PPT Presentation

Achieving High Utilization with Software-Driven WAN Chi-Yao Hong (UIUC) Srikanth Kandula Ratul Mahajan Ming Zhang Vijay Gill Mohan Nanduri Roger Wattenhofer Microsoft Tuesday, August 13, 13 Background: Inter-DC WANs Dublin% Sea,le%


slide-1
SLIDE 1

Achieving High Utilization with Software-Driven WAN

Chi-Yao Hong (UIUC) Srikanth Kandula Ratul Mahajan

Microsoft

Ming Zhang Vijay Gill Mohan Nanduri Roger Wattenhofer

Tuesday, August 13, 13

slide-2
SLIDE 2

Background: Inter-DC WANs

Hong%Kong% Seoul% Sea,le% Los%Angeles% New%York% Miami% Dublin% Barcelona%

Tuesday, August 13, 13

slide-3
SLIDE 3

Background: Inter-DC WANs

Hong%Kong% Seoul% Sea,le% Los%Angeles% New%York% Miami% Dublin% Barcelona%

Inter-DC WANs are critical

Tuesday, August 13, 13

slide-4
SLIDE 4

Background: Inter-DC WANs

Hong%Kong% Seoul% Sea,le% Los%Angeles% New%York% Miami% Dublin% Barcelona%

Inter-DC WANs are critical Inter-DC WANs are highly expensive

Tuesday, August 13, 13

slide-5
SLIDE 5

Two key problems

Poor efficiency

average utilization over time

  • f busy links is only 30-50%

Poor sharing

little support for flexible resource sharing

Tuesday, August 13, 13

slide-6
SLIDE 6

Two key problems

Poor efficiency

average utilization over time

  • f busy links is only 30-50%

Poor sharing

little support for flexible resource sharing

Why?

Tuesday, August 13, 13

slide-7
SLIDE 7

One cause of inefficiency: lack of coordination

Norm. traffic rate Time (~ one day) mean

Tuesday, August 13, 13

slide-8
SLIDE 8

One cause of inefficiency: lack of coordination

Background traffic Non-background traffic Norm. traffic rate Time (~ one day)

Tuesday, August 13, 13

slide-9
SLIDE 9

One cause of inefficiency: lack of coordination

Background traffic Non-background traffic Norm. traffic rate Time (~ one day) peak before rate adaptation peak after rate adaptation > 50% peak reduction

Tuesday, August 13, 13

slide-10
SLIDE 10

Another cause of inefficiency: local, greedy resource allocation

MPLS TE (Multiprotocol Label Switching Traffic Engineering) greedily selects shortest path fulfilling capacity constraint

Tuesday, August 13, 13

slide-11
SLIDE 11

Local, greedy resource allocation hurts efficiency

Flow Src → Dst A 1→6 B 3→6 C 4→6

1 2 3 4 5 6 7

flow arrival order: A, B, C each link can carry at most one flow

MPLS-TE

Tuesday, August 13, 13

slide-12
SLIDE 12

Local, greedy resource allocation hurts efficiency

Flow Src → Dst A 1→6 B 3→6 C 4→6

1 2 3 4 5 6 7

flow arrival order: A, B, C each link can carry at most one flow

MPLS-TE

Tuesday, August 13, 13

slide-13
SLIDE 13

Local, greedy resource allocation hurts efficiency

Flow Src → Dst A 1→6 B 3→6 C 4→6

1 2 3 4 5 6 7

flow arrival order: A, B, C each link can carry at most one flow

MPLS-TE

Tuesday, August 13, 13

slide-14
SLIDE 14

Local, greedy resource allocation hurts efficiency

Flow Src → Dst A 1→6 B 3→6 C 4→6

1 2 3 4 5 6 7

flow arrival order: A, B, C each link can carry at most one flow

MPLS-TE

Tuesday, August 13, 13

slide-15
SLIDE 15

1 2 3 5 6 7 1 2 3 5 6 7

Optimal

Local, greedy resource allocation hurts efficiency

flow arrival order: A, B, C each link can carry at most one flow

MPLS-TE

Tuesday, August 13, 13

slide-16
SLIDE 16

Poor sharing

Tuesday, August 13, 13

slide-17
SLIDE 17

Poor sharing

  • When services compete today, they can get

higher throughput by sending faster

Tuesday, August 13, 13

slide-18
SLIDE 18

Poor sharing

  • Mapping services onto different queues at

switches helps, but # services ≫ # queues

(4 - 8 typically)

  • When services compete today, they can get

higher throughput by sending faster

(hundreds)

Tuesday, August 13, 13

slide-19
SLIDE 19

Poor sharing

  • Mapping services onto different queues at

switches helps, but # services ≫ # queues

(4 - 8 typically)

  • When services compete today, they can get

higher throughput by sending faster

Borrowing the idea of edge rate limiting, we can have better sharing without many queues

(hundreds)

Tuesday, August 13, 13

slide-20
SLIDE 20

Our solution

Tuesday, August 13, 13

slide-21
SLIDE 21

SWAN

Our solution

: Software-driven WAN

Tuesday, August 13, 13

slide-22
SLIDE 22

SWAN

Our solution

: Software-driven WAN

  • high utilization
  • flexible sharing

Tuesday, August 13, 13

slide-23
SLIDE 23

System flow

WAN switches Hosts

Tuesday, August 13, 13

slide-24
SLIDE 24

System flow

WAN switches SWAN controller

traffic demand topology, traffic

Hosts

Tuesday, August 13, 13

slide-25
SLIDE 25

System flow

WAN switches SWAN controller

traffic demand topology, traffic [global optimization for high utilization]

Hosts

Tuesday, August 13, 13

slide-26
SLIDE 26

System flow

WAN switches

rate allocation network configuration

SWAN controller

traffic demand topology, traffic [global optimization for high utilization]

Hosts

Tuesday, August 13, 13

slide-27
SLIDE 27

System flow

WAN switches

rate allocation network configuration [rate limiting] [forwarding plane update]

SWAN controller

traffic demand topology, traffic [global optimization for high utilization]

Hosts

Tuesday, August 13, 13

slide-28
SLIDE 28

Challenges

  • scalable allocation computation
  • congestion-free data plane update
  • working with limited switch memory

Tuesday, August 13, 13

slide-29
SLIDE 29

Challenge #1: How to compute allocation in a time-efficient manner?

Tuesday, August 13, 13

slide-30
SLIDE 30

Computing resource allocation

Path-constrained, multi-commodity flow problem

  • allocate higher-priority traffic first
  • ensure weighted max-min fairness within a class

Solving at the granularity of {DC pairs, priority class}-tuple

  • split the allocation fairly among service flows

Tuesday, August 13, 13

slide-31
SLIDE 31

But computing max-min fairness is hard

[Danna, Mandal, Singh; INFOCOM’12]

State-of-the-art takes minutes at our target scale

As it needs to solve a long sequence of LPs: # LPs = O(# saturated edges)

Tuesday, August 13, 13

slide-32
SLIDE 32

Approximated max-min fairness

Max demand α

MCF solver Maximize throughput Prefer shorter paths

upper & lower bounds

Tuesday, August 13, 13

slide-33
SLIDE 33

Approximated max-min fairness

Max demand α

MCF solver Maximize throughput Prefer shorter paths

upper & lower bounds freeze saturated flow rates

Tuesday, August 13, 13

slide-34
SLIDE 34

Approximated max-min fairness

Max demand α α2

...

MCF solver Maximize throughput Prefer shorter paths

freeze saturated flow rates upper & lower bounds

Tuesday, August 13, 13

slide-35
SLIDE 35

Performance

Theoretical bound: Empirical efficiency (with α ¡= 2):

  • only 4% of flows deviate over 5% from

their fair share rate

  • sub-second computational time

Tuesday, August 13, 13

slide-36
SLIDE 36

Fairness: SWAN vs. MPLS TE

1 2 3 4 100 200 300 400 500 600 1 2 3 4 100 200 300 400 500 600

Flow goodput [versus max-min fair rate] Flow index [increasing order of demand] SWAN; α=2 MPLS TE

Tuesday, August 13, 13

slide-37
SLIDE 37

Challenge #2: Congestion-free update

How to update forwarding plane without causing transient congestion?

Tuesday, August 13, 13

slide-38
SLIDE 38

Congestion-free update is hard

initial state target state A B B A

Tuesday, August 13, 13

slide-39
SLIDE 39

Congestion-free update is hard

initial state target state A B B A A B

Tuesday, August 13, 13

slide-40
SLIDE 40

Congestion-free update is hard

initial state target state A B B A A B

✘ ✘

B A

Tuesday, August 13, 13

slide-41
SLIDE 41

In fact, congestion-free update sequence might not exist!

Tuesday, August 13, 13

slide-42
SLIDE 42

Idea

Leave a small amount of scratch capacity on each link

Tuesday, August 13, 13

slide-43
SLIDE 43

A=2/3 B=2/3 B=2/3 A=2/3

Slack = 1/3 of link capacity ...

  • Init. state

target state

Tuesday, August 13, 13

slide-44
SLIDE 44

A=2/3 B=2/3 B=2/3 A=2/3

Slack = 1/3 of link capacity ...

B=1/3 A=2/3 B=1/3

  • Init. state

target state

Tuesday, August 13, 13

slide-45
SLIDE 45

A=2/3 B=2/3 B=2/3 A=2/3

Slack = 1/3 of link capacity ...

B=1/3 B=1/3 A=2/3 B=1/3 A=2/3 B=1/3

  • Init. state

target state

Tuesday, August 13, 13

slide-46
SLIDE 46

A=2/3 B=2/3 B=2/3 A=2/3

Slack = 1/3 of link capacity ...

B=1/3 B=1/3 A=2/3 B=1/3 A=2/3 B=1/3 Does slack guarantee that congestion-free update always exists?

  • Init. state

target state

Tuesday, August 13, 13

slide-47
SLIDE 47

Yes!

With slack :

  • we prove there exists a congestion-free

update in steps

  • ne step = multiple updates

whose order can be arbitrary

Tuesday, August 13, 13

slide-48
SLIDE 48

Yes!

With slack :

  • we prove there exists a congestion-free

update in steps

  • ne step = multiple updates

whose order can be arbitrary It exsits, but how to find it?

Tuesday, August 13, 13

slide-49
SLIDE 49

Congestion-free update: LP-based solution

  • rate variable:

step flow path

Tuesday, August 13, 13

slide-50
SLIDE 50

Congestion-free update: LP-based solution

  • rate variable:

step flow path

  • input: and
  • output: ...

Tuesday, August 13, 13

slide-51
SLIDE 51

Congestion-free update: LP-based solution

  • rate variable:

step flow path

  • input: and
  • output: ...
  • congestion-free constraint:

∀i,j on a link link capacity

Tuesday, August 13, 13

slide-52
SLIDE 52

Utilizing all the capacity

non-background is congestion-free background has bounded congestion

using 90% capacity (s ¡= ¡10%) using 100% capacity (s ¡= ¡0%)

Tuesday, August 13, 13

slide-53
SLIDE 53

0.01% 0.1% 1% 10% 100 200 300

SWAN versus one-shot update

0.01% 0.1% 1% 10% 100 200 300

CCDF over links & updates Overloaded traffic [MB]

[data-driven evaluation; s = 10% for non-background]

Tuesday, August 13, 13

slide-54
SLIDE 54

0.01% 0.1% 1% 10% 100 200 300

SWAN versus one-shot update

0.01% 0.1% 1% 10% 100 200 300

CCDF over links & updates Overloaded traffic [MB]

0.01% 0.1% 1% 10% 100 200 300

One-shot update brings heavy packet drops

  • ne-shot

non-background

  • ne-shot

background [data-driven evaluation; s = 10% for non-background]

Tuesday, August 13, 13

slide-55
SLIDE 55

0.01% 0.1% 1% 10% 100 200 300

SWAN versus one-shot update

0.01% 0.1% 1% 10% 100 200 300

CCDF over links & updates Overloaded traffic [MB]

0.01% 0.1% 1% 10% 100 200 300

  • ne-shot

background 0.01% 0.1% 1% 10% 100 200 300 SWAN background

SWAN non-background: congestion-free background: much better than one-shot

[data-driven evaluation; s = 10% for non-background]

Tuesday, August 13, 13

slide-56
SLIDE 56

Working with limited switch memory

Challenge #3

Tuesday, August 13, 13

slide-57
SLIDE 57

Why does switch memory matter?

Tuesday, August 13, 13

slide-58
SLIDE 58

Why does switch memory matter?

How many we need?

  • 50 sites = 2,500 pairs
  • 3 priority classes
  • static k-shortest path routing

[by data-driven analysis]

Tuesday, August 13, 13

slide-59
SLIDE 59

Why does switch memory matter?

How many we need?

  • 50 sites = 2,500 pairs
  • 3 priority classes
  • static k-shortest path routing

[by data-driven analysis] it requires 20K rules to fully use network capacity

Tuesday, August 13, 13

slide-60
SLIDE 60

Why does switch memory matter?

Commodity switches has limited memory:

  • today’s OpenFlow switch: 1-4K rules
  • next generation: 16K rules

How many we need?

  • 50 sites = 2,500 pairs
  • 3 priority classes
  • static k-shortest path routing

[by data-driven analysis] it requires 20K rules to fully use network capacity

[Broadcom Trident II]

Tuesday, August 13, 13

slide-61
SLIDE 61

Hardness

Finding the set of paths with a given size that carries the most traffic is NP-complete

[Hartman et al., INFOCOM’12]

Tuesday, August 13, 13

slide-62
SLIDE 62

Heuristic: Dynamic path set adaptation

Tuesday, August 13, 13

slide-63
SLIDE 63

Heuristic: Dynamic path set adaptation

Observation:

  • working path set ≪ total needed paths

Tuesday, August 13, 13

slide-64
SLIDE 64

Heuristic: Dynamic path set adaptation

  • important ones that carry more traffic and

provide basic connectivity

  • 10x fewer rules than static k-shortest path routing

Path selection: Observation:

  • working path set ≪ total needed paths

Tuesday, August 13, 13

slide-65
SLIDE 65

Heuristic: Dynamic path set adaptation

  • important ones that carry more traffic and

provide basic connectivity

  • 10x fewer rules than static k-shortest path routing

Path selection:

Rule update:

  • multi-stage rule update
  • with 10% memory slack, typically 2 stages needed

Observation:

  • working path set ≪ total needed paths

Tuesday, August 13, 13

slide-66
SLIDE 66

Overall workflow

Compute resource allocation

Tuesday, August 13, 13

slide-67
SLIDE 67

Overall workflow

Compute resource allocation

if enough gain

Compute rule update plan

if not

Tuesday, August 13, 13

slide-68
SLIDE 68

Overall workflow

Compute resource allocation

if enough gain

Compute rule update plan

if not

Compute congestion- free update plan

Tuesday, August 13, 13

slide-69
SLIDE 69

Overall workflow

Compute resource allocation Notify services with decrease allocation

if enough gain

Compute rule update plan

if not

Compute congestion- free update plan

Tuesday, August 13, 13

slide-70
SLIDE 70

Overall workflow

Compute resource allocation Notify services with decrease allocation

if enough gain

Compute rule update plan

if not

Compute congestion- free update plan Update network

Tuesday, August 13, 13

slide-71
SLIDE 71

Overall workflow

Compute resource allocation Notify services with decrease allocation

if enough gain

Compute rule update plan

if not

Compute congestion- free update plan Update network Notify services with increase allocation

Tuesday, August 13, 13

slide-72
SLIDE 72

Evaluation platforms

Arista

Cisco N3K Blade Server

Prototype

  • 5 DCs across 3 continents;

10 switches

Tuesday, August 13, 13

slide-73
SLIDE 73

Evaluation platforms

Arista

Cisco N3K Blade Server

Prototype

  • 5 DCs across 3 continents;

10 switches Data-driven evaluation

  • 40+ DCs across 3

continents, 80+ switches

  • G-scale: 12 DCs, 24 switches

Tuesday, August 13, 13

slide-74
SLIDE 74

Prototype evaluation

0.2 0.4 0.6 0.8 1 1 2 3 4 5 6 Interactive Elastic Elastic Background

Time [minute] Goodput (normalized & stacked) Traffic: (∀DC-pair) 125 TCP flows per class

Tuesday, August 13, 13

slide-75
SLIDE 75

Prototype evaluation

0.2 0.4 0.6 0.8 1 1 2 3 4 5 6 Interactive Elastic Elastic Background

Time [minute] Goodput (normalized & stacked)

0.2 0.4 0.6 0.8 1 1 2 3 4 5 6 Interactive Elastic Elastic Background

(impractical)

  • ptimal line

High utilization SWAN’s goodput: 98% of an optimal method

dips due to rate adaptation Traffic: (∀DC-pair) 125 TCP flows per class

Tuesday, August 13, 13

slide-76
SLIDE 76

Prototype evaluation

0.2 0.4 0.6 0.8 1 1 2 3 4 5 6 Interactive Elastic Elastic Background

Time [minute] Goodput (normalized & stacked)

0.2 0.4 0.6 0.8 1 1 2 3 4 5 6 Interactive Elastic Elastic Background

(impractical)

  • ptimal line

High utilization SWAN’s goodput: 98% of an optimal method

Flexible sharing Interactive protected; background rate-adapted

dips due to rate adaptation Traffic: (∀DC-pair) 125 TCP flows per class

Tuesday, August 13, 13

slide-77
SLIDE 77

Data-driven evaluation

  • f 40+ DCs

0.2 0.4 0.6 0.8 1

SWAN SWAN w/o Rate Control MPLS TE

Goodput [versus optimal]

Tuesday, August 13, 13

slide-78
SLIDE 78

Data-driven evaluation

  • f 40+ DCs

0.2 0.4 0.6 0.8 1

SWAN SWAN w/o Rate Control MPLS TE

Goodput [versus optimal] 0.2 0.4 0.6 0.8 1

SWAN SWAN w/o Rate Control MPLS TE

99.0%

Near-optimal total goodput under a practical setting

Tuesday, August 13, 13

slide-79
SLIDE 79

Data-driven evaluation

  • f 40+ DCs

0.2 0.4 0.6 0.8 1

SWAN SWAN w/o Rate Control MPLS TE

Goodput [versus optimal] 0.2 0.4 0.6 0.8 1

SWAN SWAN w/o Rate Control MPLS TE

99.0% 58.3% 0.2 0.4 0.6 0.8 1

SWAN SWAN w/o Rate Control MPLS TE

SWAN carries ~60% more traffic than MPLS-TE

Tuesday, August 13, 13

slide-80
SLIDE 80

Data-driven evaluation

  • f 40+ DCs

0.2 0.4 0.6 0.8 1

SWAN SWAN w/o Rate Control MPLS TE

Goodput [versus optimal] 0.2 0.4 0.6 0.8 1

SWAN SWAN w/o Rate Control MPLS TE

99.0% 58.3% 0.2 0.4 0.6 0.8 1

SWAN SWAN w/o Rate Control MPLS TE

0.2 0.4 0.6 0.8 1

SWAN SWAN w/o Rate Control MPLS TE

70.3%

SWAN w/o rate control still carries 20% more traffic than MPLS TE

Tuesday, August 13, 13

slide-81
SLIDE 81

SWAN : Software-driven WAN

Tuesday, August 13, 13

slide-82
SLIDE 82

SWAN : Software-driven WAN

✔ High utilization and flexible sharing via global

rate and route coordination

Tuesday, August 13, 13

slide-83
SLIDE 83

SWAN : Software-driven WAN

✔ High utilization and flexible sharing via global

rate and route coordination

✔ Scalable allocation via approximated max-min

fairness

Tuesday, August 13, 13

slide-84
SLIDE 84

SWAN : Software-driven WAN

✔ High utilization and flexible sharing via global

rate and route coordination

✔ Congestion-free update in bounded stages ✔ Scalable allocation via approximated max-min

fairness

Tuesday, August 13, 13

slide-85
SLIDE 85

SWAN : Software-driven WAN

✔ High utilization and flexible sharing via global

rate and route coordination

✔ Congestion-free update in bounded stages ✔ Scalable allocation via approximated max-min

fairness

✔ Using commodity switches with limited

memory

Tuesday, August 13, 13

slide-86
SLIDE 86

Conclusion

Achieving high utilization itself is easy, but coupling it with flexible sharing and change management is hard

Tuesday, August 13, 13

slide-87
SLIDE 87

Conclusion

Achieving high utilization itself is easy, but coupling it with flexible sharing and change management is hard

Approximating max-min fairness with low computational time

Tuesday, August 13, 13

slide-88
SLIDE 88

Conclusion

Achieving high utilization itself is easy, but coupling it with flexible sharing and change management is hard

Approximating max-min fairness with low computational time Keeping scratch capacity of links and switch memory to enable quick transitions

Tuesday, August 13, 13

slide-89
SLIDE 89

Chi-Yao Hong (UIUC) Srikanth Kandula Ratul Mahajan

Microsoft

Ming Zhang Vijay Gill Mohan Nanduri Roger Wattenhofer

  • n the job market!

Thanks!

Tuesday, August 13, 13

slide-90
SLIDE 90

SWAN versus B4

SWAN B4 high utilization yes yes scalable rate and route computation bounded error heuristic congestion-free update in bounded steps no using commodity switches with limited # forwarding rules yes no

Tuesday, August 13, 13

slide-91
SLIDE 91

Demo Video: SWAN achieves high utilization

Tuesday, August 13, 13

slide-92
SLIDE 92

Demo Video: SWAN provides flexible sharing

Tuesday, August 13, 13

slide-93
SLIDE 93

Controller crashes

  • Run stateless backup instances

Failure handling

Link and switch failures

  • SWAN controller performs one-time global

recomputation

  • Network agent notifies SWAN controller
  • During the recovery, data plane will still operate

Controller bugs

  • Work in progress

Tuesday, August 13, 13

slide-94
SLIDE 94

Demo Video: SWAN handles link failures gracefully

Tuesday, August 13, 13

slide-95
SLIDE 95

SWAN controller

Global allocation at {SrcDC, DstDC, ServicePriority}-level

  • map flows to priority queues at switches (via DSCP bits)
  • support a few priorities (e.g.,

background < elastic < interactive) Dst

30% 50% 20%

Src

Label-based forwarding (“tunnels”)

  • by tagging

VLAN IDs

  • SWAN controller

globally computes how to split traffic at ingress switches

Tuesday, August 13, 13

slide-96
SLIDE 96

Link-level fairness != network-wide fairness

S1 S2 S3 D1 D2 D3

1/2 1/2 1/2 1/2

Link-level

S1 S2 S3 D1 D2 D3

2/3 1/3 1/3 2/3

Network-wide

Tuesday, August 13, 13

slide-97
SLIDE 97

Time for network update

Tuesday, August 13, 13

slide-98
SLIDE 98

How much stretch capacity is needed?

s

max steps 50% 1 30% 3 10% 9 0% ∞ goodput 79% 91% 100% 100%

[data-driven evaluation]

99th pctl. 1 2 3 6

> >> >>

Tuesday, August 13, 13

slide-99
SLIDE 99

if not

Our heuristic: dynamic path selection

Solve rate allocation

if can fit in memory

Greedily select important paths Solve again with selected paths fin. Using 10x fewer paths than static k-shortest path routing

Tuesday, August 13, 13

slide-100
SLIDE 100
  • ld

rules new rules

Rule update with memory constraints

Option #1: Fully utilize all the switch memory Rule update may disrupt traffic switch memory

?

Tuesday, August 13, 13

slide-101
SLIDE 101
  • ld

rules new rules

Rule update with memory constraints

Option #2: Leave 50% slack [Reitblatt et al.; SIGCOMM’12] Waste a half switch memory switch memory

Tuesday, August 13, 13

slide-102
SLIDE 102

Multi-stage rule update

switch memory active rules

Tuesday, August 13, 13

slide-103
SLIDE 103

Multi-stage rule update

switch memory active rules

Tuesday, August 13, 13

slide-104
SLIDE 104

Multi-stage rule update

switch memory active rules

Tuesday, August 13, 13

slide-105
SLIDE 105

Multi-stage rule update

switch memory active rules

Tuesday, August 13, 13

slide-106
SLIDE 106

Multi-stage rule update

switch memory active rules

Tuesday, August 13, 13

slide-107
SLIDE 107

Multi-stage rule update

switch memory active rules

Tuesday, August 13, 13

slide-108
SLIDE 108

Multi-stage rule update

switch memory active rules

Tuesday, August 13, 13

slide-109
SLIDE 109

Multi-stage rule update

switch memory active rules # stages bound: f(memory slack)

Tuesday, August 13, 13

slide-110
SLIDE 110

Multi-stage rule update

switch memory active rules # stages bound: f(memory slack) When slack=10%, 2 stages for 95% of time

Tuesday, August 13, 13