Demystifying Distributed Transactions with the - - PowerPoint PPT Presentation
Demystifying Distributed Transactions with the - - PowerPoint PPT Presentation
Demystifying Distributed Transactions with the Fairness-Isolation-Throughput Tradeoff Jose Faleiro Yale University Distributed txns: Too expensive? Several popular databases eschew distributed txns Distributed txns: Too expensive?
Distributed txns: Too expensive?
- Several popular databases eschew distributed txns
Distributed txns: Too expensive?
- Several popular databases eschew distributed txns
Distributed txns: Too expensive?
- Several popular databases eschew distributed txns
Distributed txns: Too expensive?
- Several popular databases eschew distributed txns
Distributed txns: Too expensive?
- Several popular databases eschew distributed txns
Distributed txns: Too expensive?
- Several popular databases eschew distributed txns
… and many more
This talk – Tradeoffs in distributed txns
- Do distributed txns always imply terrible performance?
- No
- For databases which support distributed txns
- What does the space of tradeoffs look like?
Fairness-Isolation-Throughput tradeoff
- Any system implementing distributed transactions can get
at most two of these three properties
But first, some context
Early databases
“A transaction ... has the properties
- f atomicity (all or nothing),
durability (effects survive failures) and consistency (a correct transformation). … The [transaction] concept may have applicability to programming systems in general.”
- Jim Gray, DB pioneer
ACID abstraction
- “Holy grail” of database system correctness
- Atomicity, Consistency, Isolation, Durability
ACID abstraction
- “Holy grail” of database system correctness
- Atomicity, Consistency, Isolation, Durability
Atomicity
- “All-or-nothing” guarantee
- Either all of a txn’s updates succeed or all fail
- Txns don’t need to think about partial writes
- No dangling references
- E.g., If secondary index points to a record, then the record must exist
Isolation
- Guarantees that if a pair of txns conflicts, then one “sees”
the other’s writes
- Each txn executes as if it has the system to itself
- Relieves developers from reasoning about txn interleavings
How to build the abstraction?
- Mutexes
- Protect shared data-structures from modification
- Concurrency control
- Logical locking, multi-versioning, etc.
- Logging
On a single node
How to build the abstraction?
On multiple nodes Things are more complicated …
Atomicity by example
T: Read R Read B . . . Write R Write B
Atomicity by example
T: Read R Read B . . . Write R Write B
Atomicity by example
R B T: Read R Read B . . . Write R Write B
Atomicity by example
T: Read R Read B . . . Write R Write B
Atomicity by example
R T: Read R Read B . . . Write R Write B
Atomicity by example
T: Write Wr Write Wb
Red must learn of blue’s willingness to commit/abort Blue must learn of red’s willingness to commit/abort
Atomicity by example
T: Write Wr Write Wb
In general… A partition must learn of every
- ther partition’s willingess to
commit/abort
Atomicity by example
T: Write Wr Write Wb
In general… A partition must learn of every
- ther partition’s willingess to
commit/abort
Atomicity by example
T: Write Wr Write Wb
In general… A partition must learn of every
- ther partition’s willingess to
commit/abort
Atomicity by example
T: Write Wr Write Wb
In general… A partition must learn of every
- ther partition’s willingess to
commit/abort
Distributed Transactions entail unavoidable coordination
Isolation
T1: Read B . . . Write B
Suppose T1 observes T0’s writes
Write R T0: Read R Read B . . . Write R Write B
Isolation
Suppose T1 observes T0’s writes
Write R T0: Read R Read B . . . Write R Write B T1: Read B . . . Write B
WAIT! Distributed Transactions entail unavoidable coordination
Isolation
T1: Read B . . . Write B
Suppose T1 observes T0’s writes
Write R T0: Read R Read B . . . Write R Write B
T1 must wait for commit protocol to finish
In order to support distributed transactions…
- Atomicity necessitates distributed coordination
- Isolation requires waiting
- Conflicts induce waiting
- The result?
- Confliciting transactions must wait for distributed coordination to finish
- Penalizes even non-distributed transactions!
Get rid of distributed transactions!
Txnr Txnb
- Only single partition txns
Get rid of distributed transactions!
Txnr Txnb
- Only single partition txns
Get rid of distributed transactions!
Txnr Txnb
- Only single partition txns
- No cross-talk between
partitions
Get rid of distributed transactions!
Txnr Txnb
- Only single partition txns
- No cross-talk between
partitions
- Easy scale-out
Txny Txng Txnr
Get rid of distributed transactions!
Txnr Txnb
- Only single partition txns
- No cross-talk between
partitions
- Easy scale-out
Txny Txng Txnr
Get rid of distributed transactions!
Txnr Txnb
- Only single partition txns
- No cross-talk between
partitions
- Easy scale-out
Txny Txng Txnr
Get rid of distributed transactions!
Txnr Txnb
- Only single partition txns
- No cross-talk between
partitions
- Easy scale-out
Txny Txng Txnr
Get rid of distributed transactions!
Txnr Txnb
- Only single partition txns
- No cross-talk between
partitions
- Easy scale-out
Txny Txng Txnr
Get rid of distributed transactions!
Txnr Txnb
- Only single partition txns
- No cross-talk between
partitions
- Easy scale-out
Txny Txng Txnr
Get rid of distributed transactions!
Txnr Txnb
- Only single partition txns
- No cross-talk between
partitions
- Easy scale-out
Txny Txng Txnr
Get rid of distributed transactions!
Txnr Txnb
- Only single partition txns
- No cross-talk between
partitions
- Easy scale-out
Txny Txng Txnr
Get rid of distributed transactions!
Txnr Txnb
- Only single partition txns
- No cross-talk between
partitions
- Easy scale-out
Txny Txng Txnr
Get rid of distributed transactions!
Txnr Txnb
- Only single partition txns
- No cross-talk between
partitions
- Easy scale-out
Txny Txng Txnr
… and many more
Get rid of distributed transactions?
No atomicty and isolation More complexity L L
Get rid of distributed transactions?
- Square pegs and round
holes Txnr Txnb
Get rid of distributed transactions?
- Square pegs and round
holes
- Explicitly deal with
inconsistencies Txnr Txnb Index on documents with the word “Cats” Document(s) not yet inserted
Get rid of distributed transactions?
- Square pegs and round
holes
- Explicitly deal with
inconsistencies
- Sometimes no correctness
guarantees possible Txnr Txnb Savings + Checking >= 0 Deduct from Savings Deduct from Checking
Get rid of distributed transactions?
“In retrospect I think that [not supporting distributed transactions] was a mistake. … a lot of people did want distributed transactions, and so they hand-rolled their
- wn protocols, sometimes incorrectly, and it
would have been better to build it into the infrastructure.“
- Jeff Dean, Google
Get rid of distributed transactions?
“In retrospect I think that [not supporting distributed transactions] was a mistake. … a lot of people did want distributed transactions, and so they hand-rolled their
- wn protocols, sometimes incorrectly, and it
would have been better to build it into the infrastructure.“
- Jeff Dean
Get rid of distributed transactions?
“In retrospect I think that [not supporting distributed transactions] was a mistake. … a lot of people did want distributed transactions, and so they hand-rolled their
- wn protocols, sometimes incorrectly, and it
would have been better to build it into the infrastructure.“
- Jeff Dean
Get rid of distributed transactions?
“In retrospect I think that [not supporting distributed transactions] was a mistake. … a lot of people did want distributed transactions, and so they hand-rolled their
- wn protocols, sometimes incorrectly, and it
would have been better to build it into the infrastructure.“
- Jeff Dean
Get rid of distributed transactions?
“In retrospect I think that [not supporting distributed transactions] was a mistake. … a lot of people did want distributed transactions, and so they hand-rolled their
- wn protocols, sometimes incorrectly, and it
would have been better to build it into the infrastructure.“
- Jeff Dean
Get rid of distributed transactions?
“In retrospect I think that [not supporting distributed transactions] was a mistake. … a lot of people did want distributed transactions, and so they hand-rolled their
- wn protocols, sometimes incorrectly, and it
would have been better to build it into the infrastructure.“
- Jeff Dean
Get rid of distributed transactions?
“In retrospect I think that [not supporting distributed transactions] was a mistake. … a lot of people did want distributed transactions, and so they hand-rolled their
- wn protocols, sometimes incorrectly, and it
would have been better to build it into the infrastructure.“
- Jeff Dean
Get rid of distributed transactions?
“In retrospect I think that [not supporting distributed transactions] was a mistake. … a lot of people did want distributed transactions, and so they hand-rolled their
- wn protocols, sometimes incorrectly, and it
would have been better to build it into the infrastructure.“
- Jeff Dean
Get rid of distributed transactions?
“In retrospect I think that [not supporting distributed transactions] was a mistake. … a lot of people did want distributed transactions, and so they hand-rolled their
- wn protocols, sometimes incorrectly, and it
would have been better to build it into the infrastructure.“
- Jeff Dean
Get rid of distributed transactions?
“In retrospect I think that [not supporting distributed transactions] was a mistake. … a lot of people did want distributed transactions, and so they hand-rolled their
- wn protocols, sometimes incorrectly, and it
would have been better to build it into the infrastructure.“
- Jeff Dean
… and many more
Why are distributed txns expensive?
Why are distributed txns expensive?
T1: Read B . . . Write B
Suppose T1 observes T0’s writes
Write R T0: Read R Read B . . . Write R Write B
Why are distributed txns expensive?
Suppose T1 observes T0’s writes
Write R T0: Read R Read B . . . Write R Write B T1: Read B . . . Write B
WAIT! Distributed Transactions entail unavoidable coordination
Why are distributed txns expensive?
T1: Read B . . . Write B
Suppose T1 observes T0’s writes
Write R T0: Read R Read B . . . Write R Write B
T1 must wait for commit protocol to finish
Why are distributed txns expensive?
- Mechanisms for atomicity and isolation overlap in time
- Atomicity: Distributed coordination
- Isolation: Wait during distributed coordination
Fairness-Isolation-Throughput tradeoff
- Any system implementing distributed transactions can get
at most two of these three properties
- Three classes of systems
- Fairness-Isolation
- Isolation-Throughput
- Throughput-Fairness
FIT Tradeoff
- Poor performance of distributed transactions
attributable to two fundamental issues
- Expensive commit protocol (required due to atomicity)
- Waiting (required for isolation)
- Commit protocol and waiting overlap in time
- Space characterized by how to separate commit
protocol from waiting
Intuition
- Badness results from overlapping commit with isolation
- To avoid impact of coordination, separate the two
Option 1: Weaken isolation
- Allow conflicting txns to execute without observing each
- ther’s writes
- Implementable without making txns wait for each other
- Susceptible to concurrency bugs
- Transactions execute against potentially stale state
- E.g., RAMP transactions
Option 2: Re-order coordination
- Move coordination outside of transaction boundaries
- Amortize coordination across several transactions
- Compromises fairness because we penalize certain txns to
benefit overall throughput
- E.g., Calvin, G-Store
FIT Tradeoff
- Fairness-Isolation
- Give up throughput
- Fairness-Throughput
- Give up isolation
- Isolation-Throughput
- Give up fairness
FIT Tradeoff
- Fairness-Isolation
- Give up throughput
- Fairness-Throughput
- Give up isolation
- Isolation-Throughput
- Give up fairness
Atomicity and isolation mechanisms overlap
FIT Tradeoff
- Fairness-Isolation
- Give up throughput
- Fairness-Throughput
- Give up isolation
- Isolation-Throughput
- Give up fairness
Atomicity and isolation mechanisms are decoupled
Weak Isolation Example
- Read Atomic Multi-Partition (RAMP) transactions
- Decouples concurrent transactions
- Research system
- Appeared in SIGMOD 2014
- Peter Bailis et al. from UC Berkeley
RAMP transactions
R B G
RAMP transactions
R B G T1: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1
RAMP transactions
R B G T1: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1
RAMP transactions
R B G T1: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1
RAMP transactions
R B G T1: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1
RAMP transactions
R B G T1: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1
Run a commit protocol
RAMP transactions
R B G T1: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1
Run a commit protocol Commit protocol represents coordination required for atomicity
RAMP transactions
R, RT1 B, BT1 G, GT1 T1: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1
RAMP transactions
R B G T1: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1 T2: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1
RAMP transactions
R B G T1: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1 T2: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1
RAMP transactions
R B G T1: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1 T2: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1
RAMP transactions
T1: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1 T2: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1 R B G
T1 and T2 read the same snapshot
RAMP transactions
R B G T1: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1 T2: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1
RAMP transactions
T1: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1 T2: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1
Commit T1
R, RT1 B, BT1 G, GT1
RAMP transactions
T2: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1 R, RT1 B, BT1 G, GT1 T1: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1
RAMP transactions
R, RT1, RT2 B, BT1, BT2 G, GT1, GT2 T1: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1 T2: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1
Commit T2
RAMP transactions
R, RT1, RT2 B, BT1, BT2 G, GT1, GT2 T1: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1 T2: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1
T1 and T2 don’t see each
- ther’s writes
RAMP transactions
R, RT1, RT2 B, BT1, BT2 G, GT1, GT2 T1: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1 T2: R0, G0, B0 = Read R G B R1, G1, B1 = f(R0, G0, B0) Write R1 G1 B1
State must be merged in some app dependent manner
RAMP transactions
- Decouples execution of concurrent txns
- “Synchronization independence”
- Great for scalability
- More resources means more throughput
- Weak isolation
- Diverged state must be reconciled and merged
- Cannot enforce important class of constraints
RAMP transactions
- Decouples execution of concurrent txns
- “Synchronization independence”
- Great for scalability
- More resources means more throughput
- Weak isolation
- Diverged state must be reconciled and merged
- Cannot enforce important class of constraints
Txns never blocks others
RAMP transactions
- Decouples execution of concurrent txns
- “Synchronization independence”
- Great for scalability
- More resources means more throughput
- Weak isolation
- Diverged state must be reconciled and merged
- Cannot enforce important class of constraints
Extra development effort
RAMP transactions
- Decouples execution of concurrent txns
- “Synchronization independence”
- Great for scalability
- More resources means more throughput
- Weak isolation
- Diverged state must be reconciled and merged
- Cannot enforce important class of constraints
Irrelevant for many applications
Fairness-Isolation-Throughput tradeoff
- Any system implementing distributed transactions can get
at most two of these three properties
- Three classes of systems
- Fairness-Isolation
- Isolation-Throughput
- Throughput-Fairness
Why are distributed txns expensive?
T1: Read B . . . Write B
Suppose T1 observes T0’s writes
Write R T0: Read R Read B . . . Write R Write B
T1 must wait for commit protocol to finish
Re-ordered coordination example
- Move distributed coordination outside txn boundaries
- Amortize its cost across several txns
- Guarantee isolation
- Conflicts still induce waiting
- But txns don’t wait for distributed coordination
- By re-ordering coordination, some txns are penalized
- Unfairly delay txns to benefit overall throughput
G-Store
- Built for workloads with temporal locality
- E.g., multi-player games
- Research system
- Appeared in SoCC 2010
- Sudipto Das et al. from UC Santa Barbara
- Built as a transaction layer on top of Hbase
G-Store
R B G Y
Supports txns on “KeyGroups”
G-Store
R B G Y
G-Store
R B G R’ B’ G’ Y
KeyGroup
G-Store
R B G R’ B’ G’ Y Txns on R, B, G, and Y, execute on Yellow node
G-Store
R B G R’ B’ G’ Y Update B
G-Store
R’ B’ G’ Y Txns on R, B, G, and Y, execute on Yellow node
No coordination on txns against a KeyGroup
G-Store
- Txns on a KeyGroup do not require coordination
- Keys are on a single shard
- Supports arbitrary distributed txns
- KeyGroups can be created on demand
G-Store
- Txns on a KeyGroup do not require coordination
- Keys are on a single shard
- Supports arbitrary distributed txns
- KeyGroups can be created on demand
Don’t distributed transactions necessitate coordination?
KeyGrouping protocol
R B G Y
Coordination happens here
KeyGrouping protocol
R B G Y
And here
G-Store’s position in FIT?
- Guarantees isolation
- Txns are serializable
- Distributed coordination does not occur on txn boundaries
- Coordination is amortized across txns on a KeyGroup
- Sacrifices fairness
- Txns on as yet unformed KeyGroups pay high latency
Fairness-Isolation example
T1: Read B . . . Write B
Suppose T1 observes T0’s writes
Write R T0: Read R Read B . . . Write R Write B
T1 must wait for commit protocol to finish
Fairness-Isolation example
Suppose T1 observes T0’s writes
Write R T0: Read R Read B . . . Write R Write B T1: Read B . . . Write B
Atomicity and isolation mechanisms overlap in time
How to use the FIT tradeoff?
- If designing a distributed DB
- Use FIT to make design desicisions that meet your requirements
How to use the FIT tradeoff?
- If designing a distributed DB
- Use FIT to make design desicisions that meet your requirements
- If choosing a distributed DB
- Try to assign it to a point in the FIT tradeoff space
How to use the FIT tradeoff?
- If designing a distributed DB
- Use FIT to make design desicisions that meet your requirements
- If choosing a distributed DB
- Try to assign it to a point in the FIT tradeoff space
- Where do your requirements lie within FIT?
- Sanity check that they’re plausible
How to use the FIT tradeoff?
- If designing a distributed DB
- Use FIT to make design desicisions that meet your requirements
- If choosing a distributed DB
- Try to assign it to a point in the FIT tradeoff space
- Where do your requirements lie within FIT?
- Sanity check that they’re plausible
- Good rule of thumb: What happens under contention?
Conclusions
- Distributed txns do not always entail terrible performance
- Systems implementing distributed txns subject to FIT
- Fairness-Isolation-Throughput tradeoff
- Can get at most two of these three properties
- Uses of FIT
- When building new systems
- Yardstick to compare distributed databases
- Sanity check requirements
Conclusions
- Distributed txns do not always entail terrible performance
- Systems implementing distributed txns subject to FIT
- Fairness-Isolation-Throughput tradeoff
- Can get at most two of these three properties
- Uses of FIT
- When building new systems
- Yardstick to compare distributed databases
- Sanity check requirements