6/10/2002 Motivation Motivation Tradable properties (ilities - - PDF document

SLIDE 1

6/10/2002 1

A A Utility Utility-

• Centered Approach

Centered Approach to Designing Dependable Internet Services to Designing Dependable Internet Services

George Candea, Armando Fox George Candea, Armando Fox and other ROC and other ROC-

• ers

ers Stanford University Stanford University

ROC Retreat / Lake Tahoe / June 10, 2002 2 George Candea

Motivation Motivation

• Tradable properties (“

Tradable properties (“ilities ilities”) in system design: functionality, ”) in system design: functionality, usability, maintainability, performance, portability, security, usability, maintainability, performance, portability, security, availability, development cost, … availability, development cost, …

• Examples of

Examples of multiway multiway tradeoffs: tradeoffs:

• Inktomi

Inktomi: data quality : data quality ↔ ↔ performance+ availability+ cost performance+ availability+ cost

• Akamai

Akamai: : security+ manageability security+ manageability ↔ ↔ performance+ availability+ cost performance+ availability+ cost

• Yahoo:

Yahoo: cost+ portability cost+ portability ↔ ↔ performance+ functionality performance+ functionality

• Key observation: tradeoffs improve service by providing a

Key observation: tradeoffs improve service by providing a better match between service properties and app requirements better match between service properties and app requirements

• Small systems: right mix is a matter of optimization

Small systems: right mix is a matter of optimization Giant scale: indispensable to the very possibility of building s Giant scale: indispensable to the very possibility of building sys ys

ROC Retreat / Lake Tahoe / June 10, 2002 3 George Candea

Issues Issues

• Making the right tradeoffs is mostly

Making the right tradeoffs is mostly art art

• 75% of system deployments fail or don’t meet requirements

75% of system deployments fail or don’t meet requirements (Yankee Group, 1998) (Yankee Group, 1998)

• Deployment costs exceed expectations (Forrester Research:

Deployment costs exceed expectations (Forrester Research: 25% of Fortune 1000 reported 10 25% of Fortune 1000 reported 10-

• 49% higher costs)

49% higher costs)

• To make it

To make it engineering engineering, we need three things: , we need three things:

1.

• 1. A straightforward model for the design space

A straightforward model for the design space

2.

• 2. Simple, but comprehensive vocabulary for describing properties

Simple, but comprehensive vocabulary for describing properties and the outcome of making tradeoffs and the outcome of making tradeoffs

3.

• 3. Step

Step-

• by

by-

• step process for trading properties among each other to

step process for trading properties among each other to maximize usefulness of system maximize usefulness of system

ROC Retreat / Lake Tahoe / June 10, 2002 4 George Candea

Proposed Process Proposed Process

1. 1.

Identify set of relevant axes that span design space in Identify set of relevant axes that span design space in req req spec spec

(“spanning set” (“spanning set” any interesting tradeoff can be expressed in terms of the axes) any interesting tradeoff can be expressed in terms of the axes)

2. 2.

State system utility functions State system utility functions w.r.t w.r.t. each axis . each axis

3. 3.

Identify major design areas; Identify major design areas; choose representative design for each; then choose representative design for each; then

• find their coordinates in design space

find their coordinates in design space

• compute overall utility by combining individual utilities

compute overall utility by combining individual utilities

4. 4.

Choose design area that maximizes utility; repeat w/in scope of Choose design area that maximizes utility; repeat w/in scope of chosen area chosen area

• iterative process, with successive refining

iterative process, with successive refining

ROC Retreat / Lake Tahoe / June 10, 2002 5 George Candea

Bank of America Bank of America (

(http://

http://www.bofa.com www.bofa.com)

)

• System model: service takes inputs and must return

System model: service takes inputs and must return

• utputs within specified amount of time
• utputs within specified amount of time
• Spanning set for design space:

Spanning set for design space:

• Q

Quality of data: consistency with real account uality of data: consistency with real account

• A

Availability: % of requests that are completed as required vailability: % of requests that are completed as required

• P

Performance: Throughput and latency for reads/writes erformance: Throughput and latency for reads/writes

• S

Security: ITSEC levels ecurity: ITSEC levels

• C

Cost of ownership: $ amount/year (including initial cost,

• st of ownership: $ amount/year (including initial cost,

amortized over expected lifetime of system) amortized over expected lifetime of system)

ROC Retreat / Lake Tahoe / June 10, 2002 6 George Candea

bofa.com bofa.com : Quality of Data (Fidelity)

: Quality of Data (Fidelity)

Quality [%] Utility [normalized]

• Utility = how useful is a given level of quality

Utility = how useful is a given level of quality

100

1

SLIDE 2

6/10/2002 2

ROC Retreat / Lake Tahoe / June 10, 2002 7 George Candea

bofa.com bofa.com : Availability

: Availability

Availability [%] Utility [normalized] 100

1

98

• Can choose salient points, then interpolate

Can choose salient points, then interpolate

ROC Retreat / Lake Tahoe / June 10, 2002 8 George Candea

bofa.com bofa.com : Performance/Latency

: Performance/Latency

• Max. Latency [sec]

Utility [normalized] 10

1

10-1 5

ROC Retreat / Lake Tahoe / June 10, 2002 9 George Candea

bofa.com bofa.com : Performance/Throughput

: Performance/Throughput

• Min. Throughput

[# responses / sec] Utility [normalized] 2000

1

1000

ROC Retreat / Lake Tahoe / June 10, 2002 10 George Candea

bofa.com bofa.com : Security

: Security

Security [ITSEC EAL] Utility [normalized] 7

1 3 4 5 6 2 1

ROC Retreat / Lake Tahoe / June 10, 2002 11 George Candea

bofa.com bofa.com : Cost

: Cost

TCO [M$/year] Utility [normalized] 4

1

3 1

ROC Retreat / Lake Tahoe / June 10, 2002 12 George Candea

Proposed Process Overview Proposed Process Overview

1. 1.

Identify set of relevant axes that span design space Identify set of relevant axes that span design space

(“spanning set” (“spanning set” any interesting tradeoff can be expressed in terms of the axes) any interesting tradeoff can be expressed in terms of the axes)

2. 2.

State system utility functions with respect to each axis State system utility functions with respect to each axis

3. 3.

Identify major design areas; Identify major design areas; choose representative design for each; then choose representative design for each; then

• find their coordinates in design space

find their coordinates in design space

• compute overall utility by combining individual utilities

compute overall utility by combining individual utilities

4. 4.

Choose design area that maximizes utility; repeat w/in scope of Choose design area that maximizes utility; repeat w/in scope of chosen area chosen area

• iterative process, with successive refining

iterative process, with successive refining

SLIDE 3

6/10/2002 3

ROC Retreat / Lake Tahoe / June 10, 2002 13 George Candea

Design Space Navigation: Phase 1 Design Space Navigation: Phase 1

• Region # 1: distributed DB, geographically distributed app

Region # 1: distributed DB, geographically distributed app servers, distributed web servers, caches everywhere servers, distributed web servers, caches everywhere

• Region # 2: centralized DB, app server, web servers; no web

Region # 2: centralized DB, app server, web servers; no web caches caches

0 -

• 0.26

0.26 0.7 0.7-

• 0.9

0.9 0 -

• 1.0

1.0 0.6 0.6 -

• 0.8

0.8 0.8 0.8 -

• 0.9

0.9 0.2 0.2 -

• 0.4

0.4 1.0 1.0

# 2 # 2

0.5 0.5-

• 0.7

0.7 0.9 0.9 – – 1.0 1.0 0.9 0.9 -

• 1.0

1.0 0.9 0.9 -

• 1.0

1.0 1.0 1.0

# 1 # 1

Overall Overall

multiply multiply

Total Cost of Total Cost of Ownership Ownership Security Security Performance Performance Throughput Throughput Performance Performance Latency Latency Availability Availability Quality Quality Type Type

• choose Area

choose Area # 2 # 2

ROC Retreat / Lake Tahoe / June 10, 2002 14 George Candea

Design Space Navigation: Phase 2 Design Space Navigation: Phase 2

• Design # 1 (w/in Region # 1): Sun Solaris 8, Oracle 8i, BEA

Design # 1 (w/in Region # 1): Sun Solaris 8, Oracle 8i, BEA WebLogic WebLogic 7.0, Netscape 7.0, Netscape-

• Enterprise 3.6

Enterprise 3.6

• Design # 2 (w/in Region # 2):

Design # 2 (w/in Region # 2): RedHat RedHat Linux 7.2, proprietary Linux 7.2, proprietary DBMS, proprietary app server, Apache 2.0 DBMS, proprietary app server, Apache 2.0

0 – – 0.13 0.13 0.8 0.8 -

• 0.9

0.9 0 – – 0.5 0.5 0.8 0.8 0.9 0.9 0.3 0.3 -

• 0.4

0.4 1.0 1.0

# 2 # 2

0.05 0.05 – – 0.21 0.21 0.7 0.7 -

• 0.8

0.8 0.5 0.5 – – 1.0 1.0 0.8 0.8 0.8 0.8 0.2 0.2 – – 0.4 0.4 1.0 1.0

# 1 # 1

Overall Overall Total Cost of Total Cost of Ownership Ownership Security Security Performance Performance Throughput Throughput Performance Performance Latency Latency Availability Availability Quality Quality Type Type

• choose # 1 (much further refinement possible,

choose # 1 (much further refinement possible, config config, etc.) , etc.)

ROC Retreat / Lake Tahoe / June 10, 2002 15 George Candea

Proposed Process Overview Proposed Process Overview

1. 1.

Identify set of relevant axes that span design space Identify set of relevant axes that span design space

(“spanning set” (“spanning set” any interesting tradeoff can be expressed in terms of the axes) any interesting tradeoff can be expressed in terms of the axes)

2. 2.

State system utility functions with respect to each axis State system utility functions with respect to each axis

3. 3.

Identify major design areas; Identify major design areas; choose representative design for each; then choose representative design for each; then

• find their coordinates in design space

find their coordinates in design space

• compute overall utility by combining individual utilities

compute overall utility by combining individual utilities

4. 4.

Choose design area that maximizes utility; repeat w/in scope of Choose design area that maximizes utility; repeat w/in scope of chosen area chosen area

• iterate until confidence band gets sufficiently narrow

iterate until confidence band gets sufficiently narrow

ROC Retreat / Lake Tahoe / June 10, 2002 16 George Candea

Alternate View Alternate View

• Design space = multidimensional hyperspace spanned by the

Design space = multidimensional hyperspace spanned by the axes described earlier and utility as an extra axis axes described earlier and utility as an extra axis

• Candidate designs = “discrete manifold” in this space

Candidate designs = “discrete manifold” in this space

• process of making tradeoffs is analogous to navigating this

process of making tradeoffs is analogous to navigating this manifold manifold

• Search for a global max with no cliffs around it (i.e., a smooth

Search for a global max with no cliffs around it (i.e., a smooth plateau) to ensure robustness plateau) to ensure robustness

• Can break design up into orthogonal subsystems that only

Can break design up into orthogonal subsystems that only concern themselves with subspaces (thus, only some of the concern themselves with subspaces (thus, only some of the axes) axes) makes it easier to design and develop makes it easier to design and develop

ROC Retreat / Lake Tahoe / June 10, 2002 17 George Candea

Benefits: Art vs. Engineering Benefits: Art vs. Engineering

• Make requirements and tradeoffs more explicit (thus,

Make requirements and tradeoffs more explicit (thus, easier to evaluate easier to evaluate and and to change later) to change later)

• Closer match between requirements and delivered

Closer match between requirements and delivered system system

• Use for dynamic adaptation

Use for dynamic adaptation (blur design points into (blur design points into regions; at design time you choose region, at runtime regions; at design time you choose region, at runtime you navigate w/in region to choose point) you navigate w/in region to choose point)

ROC Retreat / Lake Tahoe / June 10, 2002 18 George Candea

Difficulties Difficulties

• Stating utility functions can be a major effort

Stating utility functions can be a major effort

• Some properties are hard to quantify (

Some properties are hard to quantify (note note: we only need to : we only need to to to compare them, not measure on some absolute scale) compare them, not measure on some absolute scale)

• Utility

Utility-

• centered design process may require hierarchical

centered design process may require hierarchical decomposition of axes (typically application decomposition of axes (typically application-

• specific)

specific)

• hierarchical utility composition

hierarchical utility composition

• Utility units must be uniform across all axes, to enable

Utility units must be uniform across all axes, to enable comparison comparison

• The comparison must include the ability to say “how much

The comparison must include the ability to say “how much better” one point is than another better” one point is than another

• Unlike engineering, where you have struts, bolts, panels, etc.

Unlike engineering, where you have struts, bolts, panels, etc. we are far from having standardized components in software we are far from having standardized components in software engineering engineering

SLIDE 4

6/10/2002 4

ROC Retreat / Lake Tahoe / June 10, 2002 19 George Candea

More… More…

http:// http://RR.stanford.edu RR.stanford.edu