2 3 4 Service Server Client Server ConnecOon Recovered! - PowerPoint PPT Presentation

Presented
at
DSN‐DCCS
2011
in
Hong
Kong
on
6/28/11
 TRODS 
 Transparent
Recovery
for
 Object
Delivery
Services
 Wya$
Lloyd ,
Michael
J.
Freedman
 Princeton
University


2


3


4


Service
 Server
 Client
 Server
 ConnecOon
 Recovered!
 Server
 Server
 5


Object
Delivery
Services
 • Read‐Only
 • StaOc
Content
 • Webpages,
Images,
Videos
 6


Work
Now
 • Can’t
Modify
Clients
 7


Key
Idea
 • Coerce
client
to
help
 – To
idenOfy
connecOons
that
need
recovery
 – To
reliably
store
informaOon
 • Yet
client
is
unmodified
and
unaware
 – Exploit
TCP
spec
to
control
client’s
stack
 8


Object
Delivery
Cluster
 Service
 Liveness
 Server
 Monitor
 Server
 Load
 Balancer
 Server
 Server
 9


Failure
 Service
 Liveness
 Server
 Monitor
 Server
 Load
 Balancer
 Server
 Server
 10


TRODS
 Service
 Liveness
 Monitor
 ?
 Server
 Load
 Client
 Balancer
 Server
 Server
 11


TRODS
 Service
 Liveness
 Monitor
 ?
 Server
 Load
 Client
 Balancer
 Server
 Store
 Server
 12


Road
to
Recovery
 Step 
 
 
 
 
 
 
 
 Technique
 Redirect
to
live
server
……………….
Liveness
monitor
updates



 
 
 
 
 
 
 
 
 
load
balancer
 Induce
client
to
send
packet………
Coerce
client’s
TCP
stack
 ConOnue
ConnecOon
 
Determine
Phase…………………
Use
packet
+
stored
info
 
IdenOfy
Object…………………….
Stored
Info
 
Find
Offset
…………………………..
Use
packet
+
stored
info
 13


Coercing
Clients
 • Always
Leave
A
Packet
Unacknowledged
 Exploit
TCP
Spec
for
Recovery
IniOaOon!
 Client
 Server
 FIN/ACK
 Request
 ACK
 SYN
 ACK
 ACK
 ACK
 Response 1
 Response 2
 Response 3
 SYN/ACK
 ACK
 FIN
 Retransmit
Queue
 Retransmit
Queue
 Request
 FIN/ACK
 SYN
 Response 3
 Response 2
 Response 1
 SYN/ACK
 FIN
 Always
Something
Here
 14


ConOnuing
the
ConnecOon
 • Determine
Phase:
 1) TCP
Setup
 2) HTTP
Setup
 TRODS
Saves
Info
 3) HTTP
Download
 4) TCP
Teardown
 15


ConOnuing
the
Download
 • HTTP

ObjectID
 • Offset
 =
TCP
Ack
–
HTTP

ObjectISN
 HTTP
ObjectISN
 TCP
Ack
 HTTP
 S HTTP
 Y Resp
 Object
 N 
 Header
 TCP
ISN
 16


ConOnuing
the
Download
 • HTTP

ObjectID
 • Offset
 =
TCP
Ack
–
HTTP

ObjectISN
 HTTP
ObjectISN
 TCP
Ack
 HTTP
 S HTTP
 Y Resp
 Object
 N 
 Header
 TCP
ISN
 17


Persistent
Store
 • Key‐Value
Store
 +
Corner
Cases
Handled
 +
Unlimited
Objects
 KV
 – SOll
Efficient
(1
save
only)
 T IP
 TCP
 Payload
 • TCP
Timestamp
 S
 +
Very
Efficient
(1
machine
only)
 – 1
Million
Object
Limit
 Exploit
TCP
Spec
for
Persistence!
 – Corner
Cases
 18
 18


Recover
the
ConnecOon
 • IniOate
New
ConnecOon
 – GET
ObjectID
…
 – Range:
bytes=Offset‐
 • Splice
ConnecOons
Together
 • Works
with
Unmodified
Servers!
 19


TRODS
 1) Packet
ManipulaOon
 Server
 IP
 TCP
 …
 TCP
 IP
 TCP’
 …
 TRODS
 IP
 20


TRODS
 1) Packet
ManipulaOon
 Server
 2) Protocol
InspecOon
 Response 1
 TCP
 Request
 TRODS
 ObjISN
 ObjID
 IP
 Request
 21


TRODS
 1) Packet
ManipulaOon
 Server
 2) Protocol
InspecOon
 3) Blocks
ConnecOon
 TCP
 Response 1
 TRODS
 ObjID
 ObjISN
 IP
 22


TRODS
 1) Packet
ManipulaOon
 Server
 2) Protocol
InspecOon
 IP
 TCP
 …
 3) Blocks
ConnecOon
 4) State
InjecOon
 TCP
 IP
 TCP
 …
 TRODS
 TS
 IP
 23


TRODS
 1) Packet
ManipulaOon
 Server
 2) Protocol
InspecOon
 3) Blocks
ConnecOon
 ?
 4) State
InjecOon
 TCP
 5) Recovery
IniOaOon
 TRODS
 IP
 Ack
 24


Failure
Walkthrough
 Service
 Server
 Liveness
 Response 1
 SYN/ACK
 ID
 ISN
 TCP
 Monitor
 TRODS
 IP
 Server
 Load
 Client
 SYN
 Balancer
 ACK
 Request
 TCP
 TRODS
 IP
 Server
 KV
Store
 TCP
 TRODS
 IP
 25
 25


Failure
Walkthrough
 Service
 Liveness
 !
 Monitor
 Server
 Response 2
 Load
 Response 3
 Client
 Response 4
 FIN
 ?
 Balancer
 ACK
 ACK
 TCP
 ACK
 TRODS
 FIN
 ACK
 IP
 ID
 ISN
 Server
 KV
Store
 TCP
 TRODS
 IP
 26
 26


Related
Work
 • New
Transport
 – Trickles,
SCTP,
TCP
Migrate,
…
 • TCP
 – FT‐TCP ,
ST‐TCP,
Backdoors,
…
 • HTTP
 – CoRAL ,
… 
 27


ImplementaOon
 • Linux
Kernel
Module
 • 3,000
lines
of
C
 • ~CoRAL
 – OpOmisOc
subset
of
CoRAL
 28


Experiments
 • AddiOonal
Latency
 – Normal
 – Failure
 • Throughput
 – Lighppd
@
Princeton
 – Apache

@
Emulab
 – Hybrid
TS
&
KV
Throughput
 – Failure
 29


Normal
Case
Latency
 • TRODS‐TimeStamp
(TS)
 – Median:
+
0.009
ms
 – 99 th :
+
0.012
ms
 • TRODS‐Key‐Value
(KV)
 – Median:
+
0.137
ms
 – 99 th :
+
0.148
ms
 30


Recovery
Latency
 1 ~15%
 0.8 ~35%
 0.6 CDF 0.4 ~50%
 0.2 0 ~0 .2ms 20ms 200ms 3s Additional Latency Blink
of
an
eye
 31


ThroughPut
Per
Server
 Raw
 120
ops/s
 120
ops/s
 30
ops/s
 Frontend
 30
ops/s
 30
ops/s/server
 TPPS
 20
ops/s/server
 32


Lighppd
 9%
 22500 Unmodified Requests / Sec / Server TRODS-TS 20000 38%
 TRODS-KV 17500 ~CoRAL KV/Server:
 1/8
 15000 KV/Server:
 1/4
 12500 10000 7%
 7500 KV/Server:
 1/34
 66%
 5000 KV/Server:
 1/2
 2500 1KB 2KB 4KB 8KB 16KB 32KB 64KB 128KB Web Object Size 33


Apache
 1 Normalized TPPS 0.8 0.6 0.4 Unmodified FT-TCP(cold) 0.2 TRODS-TS ~CoRAL TRODS-KV FT-TCP(hot) 0 1KB 2KB 4KB 8KB 16KB 32KB 64KB Web Object Size 34


Summary
 • Recover
Object
Delivery
ConnecOons
 Unmodified
 Unmodified
 ^
 ^
 • Exploit
TCP
SpecificaOon
to
Coerce

Clients
 – To
send
recovery‐starOng
packets
 – To
provide
persistent
storage
 • EvaluaOon
 – Low
Latency
 – High
Throughput
Per
Server
 35


Summary
 • Recover
Object
Delivery
ConnecOons
 Unmodified
 ^
 • Exploit
TCP
SpecificaOon
to
Coerce

Clients
 – To
send
recovery‐starOng
packets
 – To
provide
persistent
storage
 • EvaluaOon
 – Low
Latency
 – High
Throughput
Per
Server
 • QuesOons?
 36