SLIDE 1
Prophecy : Using History for High Throughput Fault Tolerance - - PowerPoint PPT Presentation
Prophecy : Using History for High Throughput Fault Tolerance - - PowerPoint PPT Presentation
Prophecy : Using History for High Throughput Fault Tolerance Siddhartha Sen Joint work with Wyatt Lloyd and Mike Freedman Princeton University Non crash failures happen Non crash failures happen Non crash failures happen Non crash
SLIDE 2
SLIDE 3
Non‐crash failures happen Non crash failures happen
Model as Byzantine (malicious) (malicious)
SLIDE 4
Mask Byzantine faults Mask Byzantine faults
Clients Service
SLIDE 5
Mask Byzantine faults Mask Byzantine faults
Throughput Throughput
R li t d i Clients Replicated service
SLIDE 6
Mask Byzantine faults Mask Byzantine faults
Throughput Throughput
R li t d i Clients Replicated service
SLIDE 7
Mask Byzantine faults Mask Byzantine faults
Throughput Throughput
R li t d i Clients Replicated service
SLIDE 8
Mask Byzantine faults Mask Byzantine faults
Throughput Throughput
R li t d i Clients Replicated service
SLIDE 9
Mask Byzantine faults Mask Byzantine faults
Throughput Throughput
Linearizability ( t i t ) (strong consistency)
R li t d i Clients Replicated service
SLIDE 10
Byzantine fault tolerance (BFT) Byzantine fault tolerance (BFT)
- Low throughput
Low throughput difi li
- Modifies clients
- Long‐lived sessions
SLIDE 11
Prophecy Prophecy
- High throughput + good consistency
High throughput + good consistency f l h
- No free lunch:
– Read‐mostly workloads – Slightly weakened consistency
SLIDE 12
Byzantine fault tolerance (BFT) Byzantine fault tolerance (BFT)
- Low throughput
D Prophecy
Low throughput difi li
D‐Prophecy
- Modifies clients
Prophecy
- Long‐lived sessions
SLIDE 13
Traditional BFT reads Traditional BFT reads
application application
Clients Replica Group
SLIDE 14
Traditional BFT reads Traditional BFT reads
application application Agree?
Clients Replica Group
SLIDE 15
A cache solution A cache solution
application cache application
Clients Replica Group
SLIDE 16
A cache solution A cache solution
application cache application Agree?
Clients Replica Group
SLIDE 17
A cache solution A cache solution
application cache application
Problems:
Agree?
- Huge cache
- Invalidation
Clients
Invalidation
Replica Group
SLIDE 18
A compact cache A compact cache
application cache application
Requests Responses req1 resp1 req2 resp2
Clients
req3 resp3
Replica Group
SLIDE 19
A compact cache A compact cache
application cache application
Requests Responses sketch(req1) sketch(resp1) sketch(req2) sketch(resp2)
Clients
sketch(req3) sketch(resp3)
Replica Group
SLIDE 20
A sketcher A sketcher
application sketcher application
Clients Replica Group
SLIDE 21
Executing a read Executing a read
sketch webpage
…… …… …… ……
Clients
…… ……
Replica Group
SLIDE 22
Executing a read Executing a read
sketch webpage
…… …… …… ……
Clients
…… ……
Replica Group
SLIDE 23
Executing a read Executing a read
sketch webpage
…… …… …… …… ……
Clients
…… …… ……
Replica Group
SLIDE 24
Executing a read Executing a read
sketch webpage
…… ……
Agree?
…… …… ……
Clients
…… …… ……
Replica Group
SLIDE 25
Executing a read Executing a read
sketch webpage
…… ……
Agree?
…… …… ……
Fast, load‐balanced reads
Clients
…… …… ……
Replica Group
SLIDE 26
Executing a read Executing a read
sketch webpage
…… ……
Agree?
…… …… ……
Clients
…… …… ……
Replica Group
SLIDE 27
Executing a read Executing a read
sketch webpage
…… …… …… ……
Clients
…… ……
Replica Group
SLIDE 28
Executing a read Executing a read
sketch webpage
…… ……
key‐value store
…… ……
replicated state machine Clients
…… ……
Replica Group
SLIDE 29
Executing a read Executing a read
sketch webpage
…… …… …… …… …… ……
Clients
…… ……
Replica Group
SLIDE 30
Executing a read Executing a read
sketch webpage
…… …… …… …… …… ……
Clients
…… ……
Replica Group
SLIDE 31
Executing a read Executing a read
sketch webpage
…… …… …… …… …… ……
Clients
…… ……
Replica Group
SLIDE 32
Executing a read Executing a read
sketch webpage
…… ……
Agree?
…… …… ……
Clients
…… …… ……
Replica Group
SLIDE 33
Executing a read Executing a read
sketch webpage
…… ……
Agree?
…… …… ……
Maintain a fresh cache
Clients
……
fresh cache
…… ……
Replica Group
SLIDE 34
Did hi li i bilit ? Did we achieve linearizability? NO!
SLIDE 35
Executing a read Executing a read
sketch webpage
…… …… …… …… …… ……
Clients
…… ……
Replica Group
SLIDE 36
Executing a read Executing a read
sketch webpage
…… …… …… …… …… ……
Clients
…… ……
Replica Group
SLIDE 37
Executing a read Executing a read
sketch webpage
…… ……
Agree?
…… …… ……
Clients
…… …… ……
Replica Group
SLIDE 38
Executing a read Executing a read
sketch webpage
…… …… …… …… …… ……
Clients
…… ……
Replica Group
SLIDE 39
Executing a read Executing a read
sketch webpage
…… ……
Agree?
…… …… ……
Clients
…… …… ……
Replica Group
SLIDE 40
Executing a read Executing a read
sketch webpage
…… ……
Agree?
…… …… ……
Fast reads may be stale
Clients
……
may be stale
…… ……
Replica Group
SLIDE 41
Load balancing Load balancing
sketch webpage
…… …… …… ……
Clients
…… ……
Replica Group
SLIDE 42
Load balancing Load balancing
sketch webpage
…… ……
Agree?
…… …… …… ……
Clients
…… ……
Replica Group
SLIDE 43
Load balancing Load balancing
sketch webpage
…… ……
Agree?
…… …… …… ……
Pr(k stale) = gk
Clients
…… ……
Replica Group
SLIDE 44
D‐Prophecy vs. BFT D Prophecy vs. BFT
Traditional BFT:
- Each replica executes read
p
- Linearizability
D‐Prophecy:
- One replica executes read
Clients
- “Delay‐once” linearizability
Replica Group
SLIDE 45
Byzantine fault tolerance (BFT) Byzantine fault tolerance (BFT)
- Low throughput
D Prophecy
Low throughput difi li
D‐Prophecy
- Modifies clients
Prophecy
- Long‐lived sessions
SLIDE 46
Key‐exchange overhead Key exchange overhead
SLIDE 47
Key‐exchange overhead Key exchange overhead
11%
SLIDE 48
Key‐exchange overhead Key exchange overhead
11% 3%
SLIDE 49
Internet services Internet services
Cli Clients Replica Group
SLIDE 50
A proxy solution A proxy solution
Sketcher
Cli
Proxy
Clients Replica Group
SLIDE 51
A proxy solution A proxy solution
C lid t Consolidate sketchers
Sketcher
Cli
Proxy
Clients Replica Group
SLIDE 52
A proxy solution A proxy solution
C lid t Consolidate sketchers
Sketcher
Cli Clients Replica Group
SLIDE 53
A proxy solution A proxy solution
Sk t h t Sketcher must be fail‐stop
Sketcher
Cli
d
Clients
Trusted
Replica Group
SLIDE 54
A proxy solution
Sketcher must be fail‐stop
A proxy solution
iddl b l d
Sketcher must be fail stop
- Trust middlebox already
- Small and simple
Sketcher
Cli
d
Clients
Trusted
Replica Group
SLIDE 55
Executing a read Executing a read
…… …… ……
q
……
Sketcher
Cli
d
q
Clients
Trusted
…… ……
Replica Group
SLIDE 56
Executing a read Executing a read
…… …… …… ……
Sketcher
Cli
d
Clients
Trusted
…… ……
Replica Group
SLIDE 57
Executing a read Executing a read
…… …… …… ……
Sketcher
Cli
d
…… ……
Clients
Trusted
…… ……
Replica Group
SLIDE 58
Executing a read Executing a read
…… …… ……
Sketcher
……
Cli
d
…… ……
Req Resp
( )
Clients
Trusted
…… ……
s(q) ⋅⋅⋅ ⋅⋅⋅
Replica Group
SLIDE 59
Executing a read Executing a read
…… …… …… ……
Sketcher
Cli
d
Clients
Trusted
…… ……
Replica Group
SLIDE 60
Executing a read Executing a read
…… …… …… ……
Sketcher
Cli
d
Clients
Trusted
…… ……
Replica Group
SLIDE 61
Executing a read Executing a read
…… …… …… ……
Sketcher
Cli
d
…… ……
Clients
Trusted
…… ……
Replica Group
SLIDE 62
Executing a read Executing a read
…… …… …… ……
Sketcher
Cli
d
…… ……
Clients
Trusted
…… ……
Req Resp
( )
Replica Group
s(q) ⋅⋅⋅ ⋅⋅⋅
SLIDE 63
Executing a read Executing a read
…… …… …… ……
Sketcher
Cli
d
…… ……
Clients
Trusted
…… ……
Req Resp
( )
Replica Group
s(q) ⋅⋅⋅ ⋅⋅⋅
SLIDE 64
Prophecy Prophecy
…… …… …… ……
Sketcher
Cli
d
…… ……
Clients
Trusted
…… ……
Replica Group
SLIDE 65
Prophecy Prophecy
…… ……
Fast, load‐balanced reads
…… ……
Sketcher
Cli
d
…… ……
Clients
Trusted
…… ……
Replica Group
SLIDE 66
Prophecy Prophecy
…… ……
Fast reads may be stale
…… ……
Sketcher
Cli
d
…… ……
Clients
Trusted
…… ……
Req Resp
( )
Replica Group
s(q) ⋅⋅⋅ ⋅⋅⋅
SLIDE 67
Delay‐once linearizability Delay once linearizability
SLIDE 68
Delay‐once linearizability Delay once linearizability
SLIDE 69
Delay‐once linearizability Delay once linearizability
〈 W, R, W, W, R, R, W, R 〉
SLIDE 70
Delay‐once linearizability Delay once linearizability
Read‐after‐write property
〈 W, R, W, W, R, R, W, R 〉
SLIDE 71
Delay‐once linearizability Delay once linearizability
Read‐after‐write property
〈 W, R, W, W, R, R, W, R 〉
SLIDE 72
Example application Example application
- Upload embarrassing photos
Upload embarrassing photos
- 1. Remove colleagues from ACL
2 Upload photos
- 2. Upload photos
- 3. (Refresh)
- Weak may reorder
- Delay‐once preserves order
SLIDE 73
Byzantine fault tolerance (BFT) Byzantine fault tolerance (BFT)
- Low throughput
D Prophecy
Low throughput difi li
D‐Prophecy
- Modifies clients
Prophecy
- Long‐lived sessions
SLIDE 74
Implementation Implementation
- Modified PBFT
Modified PBFT
– PBFT is stable, complete Competitive with Zyzzyva et al – Competitive with Zyzzyva et. al.
- C++, Tamer async I/O
– Sketcher: ∼2000 LOC – PBFT library: ∼1140 LOC – PBFT client: ∼1000 LOC
SLIDE 75
Evaluation Evaluation
- Prophecy vs proxied‐PBFT
Prophecy vs. proxied PBFT
– Proxied systems
- D‐Prophecy vs. PBFT
– Non‐proxied systems
SLIDE 76
Evaluation Evaluation
- Prophecy vs proxied‐PBFT
Prophecy vs. proxied PBFT
– Proxied systems
- We will study:
– Performance on “null” workloads – Performance on null workloads – Performance with real replicated service Where system bottlenecks how to scale – Where system bottlenecks, how to scale
SLIDE 77
Basic setup Basic setup
Sk t h Sketcher
Clients
(concurrent)
Clients (100) Replica Group (PBFT)
SLIDE 78
SLIDE 79
Fraction of failed Fraction of failed fast reads
SLIDE 80
Fraction of failed Alexa top sites: Fraction of failed fast reads Alexa top sites: < 15%
SLIDE 81
Small benefit on null reads Small benefit on null reads
SLIDE 82
Small benefit on null reads Small benefit on null reads
SLIDE 83
Apache webserver setup Apache webserver setup
Sk t h Sketcher
Clients Clients Replica Group
SLIDE 84
Large benefit on real workload Large benefit on real workload
SLIDE 85
Large benefit on real workload Large benefit on real workload
3.7x
SLIDE 86
Large benefit on real workload Large benefit on real workload
3.7x 2.0x
SLIDE 87
Large benefit on real workload Large benefit on real workload
3.7x 2.0x
SLIDE 88
Benefit grows with work Benefit grows with work
SLIDE 89
Benefit grows with work Benefit grows with work
SLIDE 90
Benefit grows with work Benefit grows with work
SLIDE 91
Benefit grows with work Benefit grows with work
94μs (Apache)
SLIDE 92
Benefit grows with work Benefit grows with work
94μs (Apache) Null workloads are misleading! are misleading!
SLIDE 93
Benefit grows with work Benefit grows with work
SLIDE 94
Single sketcher bottlenecks Single sketcher bottlenecks
SLIDE 95
Single sketcher bottlenecks Single sketcher bottlenecks
SLIDE 96
Scaling out Scaling out
SLIDE 97
Scales linearly with replicas Scales linearly with replicas
SLIDE 98
Summary Summary
- Prophecy good for Internet services
p y g
– Fast, load‐balanced reads
- D Prophecy good for traditional services
- D‐Prophecy good for traditional services
- Prophecy scales linearly while PBFT stays flat
Prophecy scales linearly while PBFT stays flat
- Limitations:
– Read‐mostly workloads (meas. study corroborates) – Delay‐once linearizability (useful for many apps)
SLIDE 99
Thank You
SLIDE 100
Additional slides
SLIDE 101
Transitions Transitions
- Prophecy good for read‐mostly workloads
Prophecy good for read mostly workloads i i i i ?
- Are transitions rare in practice?
SLIDE 102
Measurement study Measurement study
- Alexa top sites
Alexa top sites
- Access main page every 20 sec for 24 hrs
SLIDE 103
Mostly static content Mostly static content
SLIDE 104
Mostly static content Mostly static content
SLIDE 105
Mostly static content Mostly static content
15%
SLIDE 106
Dynamic content Dynamic content
- Rabin fingerprinting on transitions
Rabin fingerprinting on transitions
- 43% differ by single contiguous change
- 43% differ by single contiguous change
- Sampled 4000 of them, over half due to: