2 3 4 Service Server Client Server ConnecOon Recovered! - - PowerPoint PPT Presentation

Wya$
Lloyd,
Michael
J.
Freedman


Princeton
University


TRODS


Transparent
Recovery
for
 Object
Delivery
Services


Presented
at
DSN‐DCCS
2011
in
Hong
Kong
on
6/28/11


2


3


4


Client
 Server
 Server


Service


Server
 Server


ConnecOon
 Recovered!


5


Object
Delivery
Services


• Read‐Only

• StaOc
Content

• Webpages,
Images,
Videos


6


Work
Now


• Can’t
Modify
Clients


7


Key
Idea


• Coerce
client
to
help


– To
idenOfy
connecOons
that
need
recovery
 – To
reliably
store
informaOon


• Yet
client
is
unmodified
and
unaware


– Exploit
TCP
spec
to
control
client’s
stack


8


Object
Delivery
Cluster


9


Server
 Server


Service


Server
 Server
 Liveness
 Monitor
 Load
 Balancer


Failure


10


Server
 Server


Service


Server
 Server
 Liveness
 Monitor
 Load
 Balancer


TRODS


11


Client
 Server


Service


Server
 Server
 Liveness
 Monitor
 Load
 Balancer


?


TRODS


12


Client
 Server


Service


Server
 Server
 Liveness
 Monitor
 Load
 Balancer


?


Store


Road
to
Recovery


Step 
 
 
 
 
 
 
 
Technique


Redirect
to
live
server
……………….
Liveness
monitor
updates



 
 
 
 
 
 
 
 
 
load
balancer
 Induce
client
to
send
packet………
Coerce
client’s
TCP
stack
 ConOnue
ConnecOon
 
Determine
Phase…………………
Use
packet
+
stored
info
 
IdenOfy
Object…………………….
Stored
Info
 
Find
Offset
…………………………..
Use
packet
+
stored
info


13


Coercing
Clients


• Always
Leave
A
Packet
Unacknowledged


SYN
 SYN/ACK
 ACK
 Request
 Response1
 ACK
 Response2
 ACK
 Response3
 FIN/ACK
 FIN


Client
 Server


14


SYN
 Request
 Retransmit
Queue
 Retransmit
Queue
 SYN/ACK
 Response1
 Response2
 Response3
 FIN/ACK
 FIN
 ACK
 ACK


Always
Something
Here


Exploit
TCP
Spec
for
Recovery
IniOaOon!


ConOnuing
the
ConnecOon


• Determine
Phase:


1) TCP
Setup
 2) HTTP
Setup
 3) HTTP
Download
 4) TCP
Teardown


15


TRODS
Saves
Info


ConOnuing
the
Download


• HTTP

ObjectID

• Offset


16


=
TCP
Ack
–
HTTP

ObjectISN


HTTP
ObjectISN
 TCP
Ack


TCP
ISN


HTTP
 Resp
 Header


HTTP
 Object


S Y N 


ConOnuing
the
Download


• HTTP

ObjectID

• Offset


17


=
TCP
Ack
–
HTTP

ObjectISN


HTTP
ObjectISN
 TCP
Ack


TCP
ISN


HTTP
 Resp
 Header


HTTP
 Object


S Y N 


Persistent
Store


• Key‐Value
Store


+
Corner
Cases
Handled
 +
Unlimited
Objects
 – SOll
Efficient
(1
save
only)


• TCP
Timestamp


+
Very
Efficient
(1
machine
only)
 – 1
Million
Object
Limit
 – Corner
Cases


18
 18


IP
 TCP
 Payload


T S
 KV


Exploit
TCP
Spec
for
Persistence!


Recover
the
ConnecOon


• IniOate
New
ConnecOon


– GET
ObjectID
…
 – Range:
bytes=Offset‐


• Splice
ConnecOons
Together

• Works
with
Unmodified
Servers!


19


TRODS


20


1) Packet
ManipulaOon
 Server


TCP
 IP
 TRODS


IP
 TCP
 …
 IP
 TCP’
 …


Server


TCP
 IP
 TRODS


TRODS


21


1) Packet
ManipulaOon
 2) Protocol
InspecOon


Request
 Request


ObjISN
 ObjID


Response1


Server


TCP
 IP
 TRODS


TRODS


22


1) Packet
ManipulaOon
 2) Protocol
InspecOon
 3) Blocks
ConnecOon


Response1


ObjISN
 ObjID


Server


TCP
 IP
 TRODS


TRODS


23


1) Packet
ManipulaOon
 2) Protocol
InspecOon
 3) Blocks
ConnecOon
 4) State
InjecOon


IP
 TCP
 …
 IP
 TCP
 …


TS


TRODS


24


1) Packet
ManipulaOon
 2) Protocol
InspecOon
 3) Blocks
ConnecOon
 4) State
InjecOon
 5) Recovery
IniOaOon
 Server


TCP
 IP
 TRODS


Ack


?


Server


TCP
 IP
 TRODS


Server


TCP
 IP
 TRODS


Server


TCP
 IP
 TRODS


Failure
Walkthrough


25
 25


Client


Service


Liveness
 Monitor
 Load
 Balancer
 KV
Store


ISN
 ID
 SYN
 ACK
 Request
 SYN/ACK
 Response1


Server


TCP
 IP
 TRODS


Server


TCP
 IP
 TRODS


Failure
Walkthrough


26
 26


Client
 Liveness
 Monitor
 Load
 Balancer
 KV
Store


ACK
 Response2


?
 !


ACK
 Response3
 ACK
 Response4
 FIN
 FIN
 ACK
 ISN
 ID


Service


Related
Work


• New
Transport


– Trickles,
SCTP,
TCP
Migrate,
…


• TCP


– FT‐TCP,
ST‐TCP,
Backdoors,
…


• HTTP


– CoRAL,
…


27


ImplementaOon


• Linux
Kernel
Module

• 3,000
lines
of
C

• ~CoRAL


– OpOmisOc
subset
of
CoRAL


28


Experiments


• AddiOonal
Latency


– Normal
 – Failure


• Throughput


– Lighppd
@
Princeton
 – Apache

@
Emulab
 – Hybrid
TS
&
KV
Throughput
 – Failure


29


Normal
Case
Latency


• TRODS‐TimeStamp
(TS)


– Median:
+
0.009
ms
 – 99th:
+
0.012
ms


• TRODS‐Key‐Value
(KV)


– Median:
+
0.137
ms
 – 99th:
+
0.148
ms


30


0.2 0.4 0.6 0.8 1 ~0 .2ms 20ms 200ms 3s CDF Additional Latency

Recovery
Latency


31


Blink
of
an
eye


~35%
 ~50%
 ~15%


ThroughPut
Per
Server


32


120
ops/s
 120
ops/s
 Raw
 30
ops/s
 30
ops/s
 Frontend
 30
ops/s/server
 20
ops/s/server
 TPPS


33


2500 5000 7500 10000 12500 15000 17500 20000 22500 1KB 2KB 4KB 8KB 16KB 32KB 64KB 128KB Requests / Sec / Server Web Object Size ~CoRAL TRODS-KV TRODS-TS Unmodified

KV/Server:
1/8
 KV/Server:
1/34
 KV/Server:
1/4
 KV/Server:
1/2


Lighppd


9%
 38%
 7%
 66%


34


Apache


FT-TCP(cold) ~CoRAL FT-TCP(hot) TRODS-KV TRODS-TS Unmodified 0.2 0.4 0.6 0.8 1 1KB 2KB 4KB 8KB 16KB 32KB 64KB Normalized TPPS Web Object Size

Summary


• Recover
Object
Delivery
ConnecOons

• Exploit
TCP
SpecificaOon
to
Coerce

Clients


– To
send
recovery‐starOng
packets
 – To
provide
persistent
storage


• EvaluaOon


– Low
Latency
 – High
Throughput
Per
Server


35


Unmodified
 ^
 Unmodified
 ^


Summary


• Recover
Object
Delivery
ConnecOons

• Exploit
TCP
SpecificaOon
to
Coerce

Clients


– To
send
recovery‐starOng
packets
 – To
provide
persistent
storage


• EvaluaOon


– Low
Latency
 – High
Throughput
Per
Server


36


• QuesOons?


Unmodified
 ^