Increasing Performance in Byzantine Fault-Tolerant Systems with - - PowerPoint PPT Presentation

▶
increasing performance in byzantine fault tolerant
SMART_READER_LITE
LIVE PREVIEW

Increasing Performance in Byzantine Fault-Tolerant Systems with - - PowerPoint PPT Presentation

Dec 11, 2022 177 likes •622 views

Increasing Performance in Byzantine Fault-Tolerant Systems with On-Demand Replica Consistency Tobias Distler and R udiger Kapitza System Software Group Friedrich-Alexander University Erlangen-Nuremberg EuroSys April 11, 2011 supported by

slide-1
SLIDE 1

Increasing Performance in Byzantine Fault-Tolerant Systems with On-Demand Replica Consistency

Tobias Distler and R¨ udiger Kapitza

System Software Group Friedrich-Alexander University Erlangen-Nuremberg

EuroSys April 11, 2011

supported by

slide-2
SLIDE 2

Byzantine Fault Tolerance (BFT)

Agreement-based Byzantine fault tolerance

3f + 1 replicas to tolerate f faults BFT agreement protocol Client-side voting

Drawbacks

High resource usage Performance overhead for agreement

Replica0 Replica1 Replica2 Replica3 Client

[Castro et al., Practical Byzantine fault tolerance, OSDI ’99]

REFIT Project Research Goal

Resource-efficient BFT systems

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 2

slide-3
SLIDE 3

Making BFT Systems More Resource-Efficient

REFIT Project Research Goal

Resource-efficient BFT systems

Optimizing Resource Usage

Reduced number of replicas Same performance Recent examples

SPARE [Distler et al., NDSS ’11] ZZ [Wood et al., EuroSys ’11]

Optimizing Performance

Default number of replicas Increased performance

⇒ ODRC

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 3

slide-4
SLIDE 4

Where to Optimize?

Agreement stage

BFT agreement protocol Sequence of agreed requests

Execution stage

Service application Request processing

Agreement0 Execution0 Agreement1 Execution1 Agreement2 Execution2 Agreement3 Execution3 Client

[Yin et al., Separating agreement from execution for Byzantine fault tolerant services, SOSP ’03]

Observations

Past optimizations have significantly reduced agreement overhead Non-trivial services: response times are dominated by execution stage

ODRC Approach

Reducing the load on the execution stage

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 4

slide-5
SLIDE 5

Basic Approach

Traditional BFT systems

All 3f + 1 replicas process all requests Client waits for f + 1 identical replies

Insight

In the absence of faults, a client only needs f + 1 replies to make progress ODRC

Each request is processed by only f + 1 replicas Load distribution across replicas Additional replicas process the request in case of faults

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 5

slide-6
SLIDE 6

Talk Outline ODRC

Selective Request Execution On-Demand Replica Consistency Evaluation Conclusion

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 6

slide-7
SLIDE 7

Talk Outline ODRC

Selective Request Execution On-Demand Replica Consistency Evaluation Conclusion

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 7

slide-8
SLIDE 8

Architecture

Selector

Selects requests for execution Stores requests that have not been selected

Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 8

slide-9
SLIDE 9

Application State

Set of objects

Examples: files, directories, ... Assumption: requests carry information about object access

Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client State Objects

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 9

slide-10
SLIDE 10

Application State Distribution

Object distribution scheme

Each object is maintained on f + 1 replicas, unmaintained on others State of unmaintained objects may be outdated

Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client Maintained Objects

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 10

slide-11
SLIDE 11

Selective Request Execution in Action

Normal-case operation

Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

Garbage collection of stored requests

Periodic object checkpoints of maintained objects Stable checkpoint: f + 1 identical checkpoints

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 11

slide-12
SLIDE 12

Selective Request Execution in Action

Normal-case operation

Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

inc() inc() inc() inc()

Garbage collection of stored requests

Periodic object checkpoints of maintained objects Stable checkpoint: f + 1 identical checkpoints

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 11

slide-13
SLIDE 13

Selective Request Execution in Action

Normal-case operation

Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

inc() inc() inc() inc()

Garbage collection of stored requests

Periodic object checkpoints of maintained objects Stable checkpoint: f + 1 identical checkpoints

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 11

slide-14
SLIDE 14

Selective Request Execution in Action

Normal-case operation

Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

inc() inc()

1 1

Garbage collection of stored requests

Periodic object checkpoints of maintained objects Stable checkpoint: f + 1 identical checkpoints

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 11

slide-15
SLIDE 15

Selective Request Execution in Action

Normal-case operation

Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

inc() inc() inc() inc() inc() inc()

1 1

Garbage collection of stored requests

Periodic object checkpoints of maintained objects Stable checkpoint: f + 1 identical checkpoints

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 11

slide-16
SLIDE 16

Selective Request Execution in Action

Normal-case operation

Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

inc() inc() inc() inc() inc() inc()

1 1

Garbage collection of stored requests

Periodic object checkpoints of maintained objects Stable checkpoint: f + 1 identical checkpoints

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 11

slide-17
SLIDE 17

Selective Request Execution in Action

Normal-case operation

Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

inc() inc() inc() inc()

1 1 1 1

Garbage collection of stored requests

Periodic object checkpoints of maintained objects Stable checkpoint: f + 1 identical checkpoints

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 11

slide-18
SLIDE 18

Talk Outline ODRC

Selective Request Execution On-Demand Replica Consistency Evaluation Conclusion

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 12

slide-19
SLIDE 19

Multi-Object Operations

Access of multiple objects

Only unmaintained objects ⇒ store request At least one maintained object

Update unmaintained objects Process request Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

inc() inc() inc() inc()

1 1 1 1

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 13

slide-20
SLIDE 20

Multi-Object Operations

Access of multiple objects

Only unmaintained objects ⇒ store request At least one maintained object

Update unmaintained objects Process request Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

inc() inc() inc() inc() inc() inc() inc() inc()

1 1 1 1

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 13

slide-21
SLIDE 21

Multi-Object Operations

Access of multiple objects

Only unmaintained objects ⇒ store request At least one maintained object

Update unmaintained objects Process request Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

inc() inc() inc() inc() inc() inc() inc() inc()

1 1 1 1

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 13

slide-22
SLIDE 22

Multi-Object Operations

Access of multiple objects

Only unmaintained objects ⇒ store request At least one maintained object

Update unmaintained objects Process request Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

inc() inc() inc() inc() inc() inc() inc() inc()

1 1 1 1

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 13

slide-23
SLIDE 23

Multi-Object Operations

Access of multiple objects

Only unmaintained objects ⇒ store request At least one maintained object

Update unmaintained objects Process request Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

inc() inc() inc() inc() inc() inc() inc()

1 1 1 2 1

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 13

slide-24
SLIDE 24

Multi-Object Operations

Access of multiple objects

Only unmaintained objects ⇒ store request At least one maintained object

Update unmaintained objects Process request Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

inc() inc() inc() inc() inc() inc() inc()

Cross-border request

1 1 1 2 1

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 13

slide-25
SLIDE 25

Multi-Object Operations

Access of multiple objects

Only unmaintained objects ⇒ store request At least one maintained object

Update unmaintained objects Process request Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

inc() inc() inc() inc() inc() inc() inc()

1 1 1 2 1

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 13

slide-26
SLIDE 26

Multi-Object Operations

Access of multiple objects

Only unmaintained objects ⇒ store request At least one maintained object

Update unmaintained objects Process request Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

inc() inc() inc() inc() inc() inc()

1 1 1 1 2 1

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 13

slide-27
SLIDE 27

Multi-Object Operations

Access of multiple objects

Only unmaintained objects ⇒ store request At least one maintained object

Update unmaintained objects Process request Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

inc() inc() inc() inc() inc() inc()

1 1 1 1 2 1

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 13

slide-28
SLIDE 28

Multi-Object Operations

Access of multiple objects

Only unmaintained objects ⇒ store request At least one maintained object

Update unmaintained objects Process request Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

inc() inc() inc() inc() inc()

1 1 2 1 1 2 1

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 13

slide-29
SLIDE 29

Multi-Object Operations

Access of multiple objects

Only unmaintained objects ⇒ store request At least one maintained object

Update unmaintained objects Process request Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

inc() inc() inc() inc() inc()

1 1 2 1 1 2 1

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 13

slide-30
SLIDE 30

Multi-Object Operations

Access of multiple objects

Only unmaintained objects ⇒ store request At least one maintained object

Update unmaintained objects Process request Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

inc() inc() inc() inc()

1 1 2 1 2 1 2 1

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 13

slide-31
SLIDE 31

Multi-Object Operations

Access of multiple objects

Only unmaintained objects ⇒ store request At least one maintained object

Update unmaintained objects Process request Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

inc() inc() inc() inc()

1 1 2 1 2 1 2 1

On-Demand Replica Consistency

  • nly if a request demands it
  • nly to the extent demanded

by a request

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 13

slide-32
SLIDE 32

Cross-Border Requests

Additional consistency overhead

Processed by more than f+1 replicas Goal: minimize number of cross-border requests

Optimized object distribution

Application-centric strategies Consider object dependencies

Selector0 Selector1 Selector2 Selector3

Example: Network File System (NFS)

Assign files and their parent directories to the same replicas Subdirectories may be assigned to different replicas

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 14

slide-33
SLIDE 33

Handling Faults

Providing additional replies on demand

Standard BFT clients Request retransmission after timeout Additional replicas process the request

Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

1 8

Replies

inc() inc() inc() inc()

8 1 2 1 2 1 2 1

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 15

slide-34
SLIDE 34

Handling Faults

Providing additional replies on demand

Standard BFT clients Request retransmission after timeout Additional replicas process the request

Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

1 8

Replies

Retransmit!

inc() inc() inc() inc()

8 1 2 1 2 1 2 1

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 15

slide-35
SLIDE 35

Handling Faults

Providing additional replies on demand

Standard BFT clients Request retransmission after timeout Additional replicas process the request

Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

1 8

Replies

inc() inc() inc() inc() force force force force

8 1 2 1 2 1 2 1

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 15

slide-36
SLIDE 36

Handling Faults

Providing additional replies on demand

Standard BFT clients Request retransmission after timeout Additional replicas process the request

Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

1 8

Replies

inc() inc() inc() inc()

8 1 2 1 2 1 2 1

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 15

slide-37
SLIDE 37

Handling Faults

Providing additional replies on demand

Standard BFT clients Request retransmission after timeout Additional replicas process the request

Agreement0 Selector0 Execution0 Agreement1 Selector1 Execution1 Agreement2 Selector2 Execution2 Agreement3 Selector3 Execution3 Client

1 8

Replies

inc() inc()

1 1

8 1 2 1 1 2 1 1 2 1

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 15

slide-38
SLIDE 38

Talk Outline ODRC

Selective Request Execution On-Demand Replica Consistency Evaluation Conclusion

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 16

slide-39
SLIDE 39

NFS Evaluation: Postmark Benchmark

5 10 15 20 25 30 35 40 45 50

Number of clients

100 200 300 400 500 600 700 800 900 1000

Duration [s]

BFT Unreplicated NFS ODRC 4 replicas (f=1), write-mostly workload

61% throughput increase over BFT

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 17

slide-40
SLIDE 40

NFS Evaluation: Postmark Benchmark

5 10 15 20 25 30 35 40 45 50

Number of clients

100 200 300 400 500 600 700 800 900 1000

Duration [s]

BFT Unreplicated NFS ODRC ODRC (optimized distribution) 4 replicas (f=1), write-mostly workload

90% throughput increase over BFT

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 17

slide-41
SLIDE 41

NFS Evaluation: Append-Only Micro-Benchmark

  • 15
  • 10
  • 5

5 10 15

Time [s]

10 20 30 40 50 60 70 80 90

  • Avg. response time [ms]

Replica fault

4 ODRC replicas 20 clients

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 18

slide-42
SLIDE 42

Talk Outline ODRC

Selective Request Execution On-Demand Replica Consistency Evaluation Conclusion

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 19

slide-43
SLIDE 43

Conclusion

Execution matters! Traditional BFT systems

All replicas process all requests Consistency overhead

ODRC

Selective request execution based on object access On-demand replica consistency Additional replies in case of faults

Thank you very much. Questions?

Tobias Distler (distler@cs.fau.de) Increasing Performance in BFT Systems with ODRC 20