Ti e E ffi cient Server Audit Problem, Deduplicated Re-execution, - - PowerPoint PPT Presentation

Tie Efficient Server Audit Problem, Deduplicated Re-execution, and the Web

Cheng Tan, Lingfan Yu, Joshua B. Leners*, and Michael Walfjsh

NYU Department of Computer Science, Courant Institute *Two Sigma Investments

Amazon Web Services company

Alice employee employee wiki PHP

Amazon Web Services company

web server OS PHP runtime hypervisor hardware database wiki PHP Alice employee employee

request response

Amazon Web Services company

web server OS PHP runtime hypervisor hardware database wiki PHP Alice employee employee

request response

• Alice has confjdence in the wiki's PHP code

Amazon Web Services company

web server OS PHP runtime hypervisor hardware database wiki PHP Alice employee employee

request response

• Alice has confjdence in the wiki's PHP code
• Still, lots of things can go wrong ...

Amazon Web Services company

web server OS PHP runtime hypervisor hardware database wiki PHP Alice employee employee

request response

• Alice has confjdence in the wiki's PHP code
• Still, lots of things can go wrong ...

kiwi PHP

Amazon Web Services company

web server OS PHP runtime hypervisor hardware database wiki PHP Alice employee employee

request response

• Alice has confjdence in the wiki's PHP code
• Still, lots of things can go wrong ...

PCP runtime

Amazon Web Services company

web server OS PHP runtime hypervisor hardware database wiki PHP Alice employee employee

request response

• Alice has confjdence in the wiki's PHP code
• Still, lots of things can go wrong ...

OS hypervisor OS

Amazon Web Services company

web server OS PHP runtime hypervisor hardware database wiki PHP Alice employee employee

request response

• Alice has confjdence in the wiki's PHP code
• Still, lots of things can go wrong ...

web server

Amazon Web Services company

web server OS PHP runtime hypervisor hardware database wiki PHP Alice employee employee

request response

• Alice has confjdence in the wiki's PHP code
• Still, lots of things can go wrong ...

kiwi PHP PCP runtime OS hypervisor OS web server

Amazon Web Services company

web server OS PHP runtime hypervisor hardware database wiki PHP Alice employee employee

request response

• Alice has confjdence in the wiki's PHP code
• Still, lots of things can go wrong ...

kiwi PHP PCP runtime OS hypervisor OS web server

Amazon Web Services company

web server OS PHP runtime hypervisor hardware database wiki PHP Alice employee employee

request response

• Alice has confjdence in the wiki's PHP code
• Still, lots of things can go wrong ...

kiwi PHP PCP runtime OS hypervisor OS web server

• Tius, Alice wants to audit the delivered responses

– Are they derived from executing the actual application?

Tie Efficient Server Audit Problem

program server

Tie Efficient Server Audit Problem

program requests responses server

• nline phase

clients

Tie Efficient Server Audit Problem

program requests responses server

• nline phase

clients

• 2. server is concurrent
• 1. server is untrusted; can respond arbitrarily

Tie Efficient Server Audit Problem

program requests responses server

• nline phase

clients

• 2. server is concurrent
• 1. server is untrusted; can respond arbitrarily

trace collector trace

Amazon Web Services company

request response

web server OS PHP runtime hypervisor hardware OS hypervisor database OS web server wiki PHP kiwi PHP PCP runtime Alice employee employee wiki PHP

trace collector

Tie Efficient Server Audit Problem

program verifier audit phase requests responses server

• nline phase

clients trace collector trace

• 2. server is concurrent
• 1. server is untrusted; can respond arbitrarily

Tie Efficient Server Audit Problem

program verifier audit phase requests responses server

• nline phase

clients trace collector trace = ? program + requests responses

• 2. server is concurrent
• 1. server is untrusted; can respond arbitrarily

Tie Efficient Server Audit Problem

program verifier audit phase requests responses server

• nline phase

clients trace collector trace = ? program + requests responses

• 2. server is concurrent
• 1. server is untrusted; can respond arbitrarily
• 4. server overhead is low; legacy applications supported
• 3. verifjer is weaker than server

Tie Efficient Server Audit Problem

program verifier audit phase requests responses server

• nline phase

clients = ? program + requests responses trace collector trace

• 1. server is untrusted…
• 2. server is concurrent
• 3. verifjer is weaker than server
• 4. server overhead is low...

Tie Efficient Server Audit Problem

program verifier audit phase requests responses server

• nline phase

clients = ? program + requests responses

• Combination of these four is a new problem.
• Execution integrity is complementary to program verifjcation.

trace collector trace

• 1. server is untrusted…
• 2. server is concurrent
• 3. verifjer is weaker than server
• 4. server overhead is low...
• 1. server is untrusted…
• 2. server is concurrent
• 3. verifjer is weaker than server
• 4. server overhead is low...

program verifier audit phase requests responses server

• nline phase

clients trace trace collector

• 1. server is untrusted…
• 2. server is concurrent
• 3. verifjer is weaker than server
• 4. server overhead is low...

What about naive re-execution?

program verifier audit phase requests responses server

• nline phase

clients trace trace collector

• 1. server is untrusted…
• 2. server is concurrent
• 3. verifjer is weaker than server
• 4. server overhead is low...

What about naive re-execution?

= ?

delivered responses produced responses

program verifier audit phase requests responses server

• nline phase

clients trace trace collector

What about naive re-execution?

= ?

delivered responses

• Tiis does not save the verifjer work.
• 1. server is untrusted…
• 2. server is concurrent
• 3. v
• 3. veri

erifj fjer is weaker than server er is weaker than server

• 4. server overhead is low...

❌ ✔ ✔ ✔

produced responses

program verifier audit phase requests responses server

• nline phase

clients trace trace collector

= ?

produced responses delivered responses

• Tiis does not save the verifjer work.
• Instead, we will accelerate re-execution.
• 1. server is untrusted…
• 2. server is concurrent
• 3. verifjer is weaker than server
• 4. server overhead is low...
• 1. server is untrusted…
• 2. server is concurrent
• 3. v
• 3. veri

erifj fjer is weaker than server er is weaker than server

• 4. server overhead is low...

❌ ✔ ✔ ✔

• 1. server is untrusted…
• 2. server is concurrent
• 3. verifjer is weaker than server
• 4. server overhead is low...

❓ ✔ ❓ ❓

What about naive re-execution?

Rest of the talk

• 1. How does the verifjer accelerate re-execution?
• 2. Why are shared objects (such as DBs) challenging?
• 3. Does our implementation for PHP perform well?

(these two are in tension)

Rest of the talk

• 1. How does the verifjer accelerate re-execution?
• 2. Why are shared objects (such as DBs) challenging?
• 3. Does our implementation for PHP perform well?
• 1. How does the verifjer accelerate re-execution?

Accelerating re-execution: a 30,000-foot view

server (online) verifier (offline) advice

• Deduplicate computation across requests

Poirot’s observation: repeated computation

• T. Kim, R. Chandra, and N. Zeldovich.

Efficient patch-based audi>ng for web applica>ons. OSDI, 2012

Poirot’s observation: repeated computation

• T. Kim, R. Chandra, and N. Zeldovich.

Efficient patch-based audi>ng for web applica>ons. OSDI, 2012

Poirot’s observation: repeated computation

reqi reqj “My paper” “Another paper”

• T. Kim, R. Chandra, and N. Zeldovich.

Efficient patch-based audi>ng for web applica>ons. OSDI, 2012

Poirot’s observation: repeated computation

reqi reqj “My paper” “Another paper”

• T. Kim, R. Chandra, and N. Zeldovich.

Efficient patch-based audi>ng for web applica>ons. OSDI, 2012

requires trusting the requires trusting the advice advice

We accelerate re-execution without trusting the server

server (online) verifier (offline) C: tag→{set of reqs} for each tag: – execute C(tag) with SIMD-on-demand – conduct unanimity checks

We accelerate re-execution without trusting the server

SIMD-on-demand re-executes identical instructions once.

server reqi reqj verifier reqi+reqj server (online) verifier (offline) C: tag→{set of reqs} for each tag: – execute C(tag) with SIMD-on-demand – conduct unanimity checks

SIMD-on-demand eliminates redundant computation

main(a,b): c ← a * b c ← c + 1 reqi: reqj: a=1;b=2 a=2;b=1

SIMD-on-demand eliminates redundant computation

main(a,b): c ← a * b c ← c + 1 reqi: reqj: a=1;b=2 a=2;b=1 reqi+reqj c=[2,2]

+1

* *

a=[1,2] b=[2,1]

• Multi-value represents different values of the same variable.

• Verifjer collapses multi-value to scalar if possible.

c=3 reqi+reqj c=[2,2]

+1

* *

a=[1,2] b=[2,1] c=2

• Multi-value represents different values of the same variable.

SIMD-on-demand eliminates redundant computation

main(a,b): c ← a * b c ← c + 1 reqi: reqj: a=1;b=2 a=2;b=1

Recap

• Verifjer re-executes in an accelerated way …
• ... by exploiting advice from the server ...
• ... without trusting that advice.

• 1. How does the verifjer accelerate re-execution?
• 2. Why are shared objects (such as DBs) challenging?
• 3. Does our implementation for PHP perform well?

• Will try to give some intuition for the difficulties
• Solutions in the paper, rigorous proofs in tech report

server (online)

UPDATE database put get SELECT key-value store

• For now, assume simple storage model

– Read-write registers, named with letters

• Will try to give some intuition for the difficulties
• Solutions in the paper, rigorous proofs in tech report

register B register A write(B,12) 12←read(B) 56←read(A) write(A,56)

server (online)

Central challenge: re-execution is out of order

write(B,12) 12←read(B) register B 56←read(A) write(A,56) register A

verifier (offline)

register B

…

server (online)

Central challenge: re-execution is out of order

write(B,12) 12←read(B) register B 56←read(A) write(A,56) register A

verifier (offline)

register B

…

server (online)

…

Central challenge: re-execution is out of order

write(B,12) 12←read(B) register B 56←read(A) write(A,56) register A

verifier (offline)

register B write(B,12) ←read(B)

?

…

server (online)

…

How can the verifjer re-execute reads? Attempt 1:

• Server logs read values; verifjer supplies from log

advice

reqi

register B READ 12

… … write(B,12) 12←read(B) register B 56←read(A) write(A,56) register A register B write(B,12) 12←read(B)

…

verifier (offline) server (online)

…

How can the verifjer re-execute reads? Attempt 1:

• Server logs read values; verifjer supplies from log
• Tiis can fool the verifjer

advice

reqi

register B READ 12

… …

reqi

register B READ 999

write(B,12) 12←read(B) register B 56←read(A) write(A,56) register A register B write(B,12) 12←read(B) 999←read(B)

…

999←read(B)

verifier (offline) server (online)

…

How can the verifjer re-execute reads? Attempt 2:

advice

reqj

register B WRITE 12

… …

reqk

register B READ

write(B,12) 12←read(B) register B 56←read(A) write(A,56) register A register B write(B,12) 12←read(B)

• Server: logs operands
• Verifjer: simulates reads using log and checks writes

…

verifier (offline) server (online)

…

How can the verifjer re-execute reads? Attempt 2:

advice

reqj

register B WRITE 12

… …

reqk

register B READ

write(B,12) 12←read(B) register B 56←read(A) write(A,56) register A register B write(B,12) 12←read(B)

• Server: logs operands
• Verifjer: simulates reads using log and checks writes

…

verifier (offline) server (online)

…

How can the verifjer re-execute reads? Attempt 2:

advice

reqj

register B WRITE 12

… …

reqk

register B READ

check if “WRITE 12” write(B,12) 12←read(B) register B 56←read(A) write(A,56) register A register B write(B,12) 12←read(B)

• Server: logs operands
• Verifjer: simulates reads using log and checks writes

…

verifier (offline) server (online)

…

Another challenge is validating the log order

• Order in log could be nonsensical
• Verifjer must check consistency of log:

– Is log order consistent with observed request order?

req1 r e q

2

resp1 resp2 r e q

3

resp4

server

r e q

4

resp3

… …

req2

• p

req3

• p

req2

• p

req1

• p

?

consistent with

• Tiis check must be efficient

• 1. How does the verifjer accelerate re-execution?
• 2. Why are shared objects (such as DBs) challenging?
• 3. Does our implementation for PHP perform well?

A built system: Orochi

• Orochi targets apps based on PHP and SQL (“LAMP”)
• Server and verifjer: modifjed PHP runtimes
• Includes techniques for deduplicating database queries
• Details

– Built atop HipHop VM (HHVM) – 20K lines of C++, PHP, Bash, Python

• Is auditing efficient for the verifjer?
• What is the price of verifjability?
• How compatible is Orochi with legacy applications?

Evaluation questions

• Applications:

– MediaWiki, phpBB and HotCRP

• Workloads:

– MediaWiki: Wikipedia 2007 trace – phpBB: 7-day’s posts from CentOS forum – HotCRP: Simulation of SIGCOMM’09

Our workloads see a lot of redundant computation

MediaWiki’s workload (20K requests)

propor>on of iden>cal instruc>ons the number of requests

200 400 600 800 CPU >me take (s) PHP DB Others

Orochi's verifjer is efficient

beaer

10.9x

MediaWiki’s workload phpBB’s workload HotCRP’s workload

5.6x 6.2x

* Pessimistically estimated from the original online execution Naive re-execu>on* Orochi Naive re-execu>on* Orochi Naive re-execu>on* Orochi

Orochi's verifjer achieves speedups compared to naive replay

Tie price of verifjability is tolerable

CPU

MediaWiki’s workload phpBB’s workload HotCRP’s workload

4.7% 8.6% 5.9%

trace (per req) advice (per req) Orochi’s

• verhead

7.1KB 1.7KB 11.4% 5.7KB 0.3KB 2.7%

Network

MediaWiki’s workload phpBB’s workload HotCRP’s workload

3.2KB 0.4KB 10.9%

Storage

MediaWiki’s workload phpBB’s workload HotCRP’s workload

1.0x 1.7x 1.5x Verifjer needs to store the trace and advice for one audit epoch.

Orochi requires modest application adjustments

• Lines of code modifjed:

– 346 lines of code change for MediaWiki – 270 lines of code change for phpBB – 67 lines of code change for HotCRP

• Most of the changes are due to

– PHP features that our implementation does not support – Modifying the application to respect object semantics

Recap of evaluation

• Verifjer: 5.6--10.9x speedup over naive re-execution
• Costs: storage at verifjer, <10% overhead on server
• Compatibility: Modest application changes

Related work, future work, and wrap-up

Related work

• Efficient execution integrity

– Replication: BFT – Attestation: TPMs, SGX – Probabilistic proofs: Pepper, CMT, Pinocchio, Pantry, SNARKs

• Computation deduplication (Delta execution, iTireads)
• Record-replay

– Untrusted recorder: Accountable Virtual Machines – Accelerated replayer: Poirot – Multiprocessor: RecPlay, LEAP, DoublePlay, PRES, ODR, …

• 1. server is untrusted
• 2. server is concurrent
• 3. verifjer is weaker than server
• 4. server overhead is low; compatibility
• 1. server is untrusted
• 2. server is concurrent
• 3. verifjer is weaker than server
• 4. server overhead is low; compatibility

❌

• 1. server is untrusted
• 2. server is concurrent
• 3. verifjer is weaker than server
• 4. server overhead is low; compatibility

❌ ❌ – Attestation: TPMs, SGX

Related work

• Efficient execution integrity

– Replication: BFT – Attestation: TPMs, SGX – Probabilistic proofs: Pepper, CMT, Pinocchio, Pantry, SNARKs

• Computation deduplication (Delta execution, iTireads)
• Record-replay

– Untrusted recorder: Accountable Virtual Machines – Accelerated replayer: Poirot – Multiprocessor: RecPlay, LEAP, DoublePlay, PRES, ODR, …

• 1. server is untrusted
• 2. server is concurrent
• 3. verifjer is weaker than server
• 4. server overhead is low; compatibility
• 1. server is untrusted
• 2. server is concurrent
• 3. verifjer is weaker than server
• 4. server overhead is low; compatibility

❌

• 1. server is untrusted
• 2. server is concurrent
• 3. verifjer is weaker than server
• 4. server overhead is low; compatibility

❌ ❌

• 1. server is untrusted
• 2. server is concurrent
• 3. verifjer is weaker than server
• 4. server overhead is low; compatibility

❌ ❌ – Probabilistic proofs: Pepper, CMT, Pinocchio, Pantry, SNARKs

Related work

• Efficient execution integrity

– Replication: BFT – Attestation: TPMs, SGX – Probabilistic proofs: Pepper, CMT, Pinocchio, Pantry, SNARKs

• Computation deduplication (Delta execution, iTireads)
• Record-replay

– Untrusted recorder: Accountable Virtual Machines – Accelerated replayer: Poirot – Multiprocessor: RecPlay, LEAP, DoublePlay, PRES, ODR, …

• 1. server is untrusted
• 2. server is concurrent
• 3. verifjer is weaker than server
• 4. server overhead is low; compatibility
• 1. server is untrusted
• 2. server is concurrent
• 3. verifjer is weaker than server
• 4. server overhead is low; compatibility

❌

• 1. server is untrusted
• 2. server is concurrent
• 3. verifjer is weaker than server
• 4. server overhead is low; compatibility

❌ ❌

• 1. server is untrusted
• 2. server is concurrent
• 3. verifjer is weaker than server
• 4. server overhead is low; compatibility

❌ ❌

• Computation deduplication (Delta execution, iTireads)
• Record-replay

Wrap-up and future work

• Our solution to the Efficient Server Audit Problem:

– Includes a new accelerated re-execution technique – Includes new algorithms for verifying concurrent executions – Comes with a rigorous proof of correctness

• Our instantiation for PHP, SQL web apps:

– 5-10x speedups over a naive replay; <10% CPU overhead on server

• Future work includes:

– SGX integration – Extend to multiple interacting servers – Accelerate other record-replay systems