Data Provenance at Internet Scale: Architecture, Experiences, and - - PowerPoint PPT Presentation
Data Provenance at Internet Scale: Architecture, Experiences, and the Road Ahead Ang Chen, Yang Wu, Andreas Haeberlen, Boon Thau Loo, Wenchao Zhou Motivation D E A foo.com Alice C B An example scenario: network routing System
Ang Chen, Yang Wu, Andreas Haeberlen, Boon Thau Loo, Wenchao Zhou
– System administrator observes strange behavior – Example: the route to foo.com has suddenly changed – Anomalies in distributed systems
2
Alice foo.com A D E B C
– System administrator observes strange behavior – Example: the route to foo.com has suddenly changed – Anomalies in distributed systems
2
Alice foo.com Route r1 A D E B C
– System administrator observes strange behavior – Example: the route to foo.com has suddenly changed – Anomalies in distributed systems
2
Alice foo.com Route r2 A D E B C
– System administrator observes strange behavior – Example: the route to foo.com has suddenly changed – Anomalies in distributed systems
2
Why did my route to foo.com change?!
Alice foo.com Route r2 A D E B C
– System administrator observes strange behavior – Example: the route to foo.com has suddenly changed – Anomalies in distributed systems
2
Why did my route to foo.com change?!
Alice foo.com Route r2
Innocent Reason?
A D E B C
– System administrator observes strange behavior – Example: the route to foo.com has suddenly changed – Anomalies in distributed systems
2
Why did my route to foo.com change?!
Alice foo.com Route r2
Innocent Reason?
A D E B C
Software Bugs?
– System administrator observes strange behavior – Example: the route to foo.com has suddenly changed – Anomalies in distributed systems
2
Why did my route to foo.com change?!
Alice foo.com Route r2
Innocent Reason? Malicious Attack?
A D E B C
Software Bugs?
– Network consists of nodes (routers, middleboxes, ...) – The state of a node is a set of tuples (routes, config, ...)
3
Alice foo.com A B C D E
– Network consists of nodes (routers, middleboxes, ...) – The state of a node is a set of tuples (routes, config, ...)
3
Alice foo.com
route(A, B)
A B C D E
…… route(A, foo.com) route(A, D)
– Network consists of nodes (routers, middleboxes, ...) – The state of a node is a set of tuples (routes, config, ...)
3
Alice foo.com
route(A, B)
A B C D E
…… route(A, foo.com) route(A, D) link(A, B) link(A, D)
– Network consists of nodes (routers, middleboxes, ...) – The state of a node is a set of tuples (routes, config, ...) – Idea: Explanation as reasoning about distributed state dependencies
3
Alice foo.com A B C D E
route(A, foo.com)
– Network consists of nodes (routers, middleboxes, ...) – The state of a node is a set of tuples (routes, config, ...) – Idea: Explanation as reasoning about distributed state dependencies
3
Alice foo.com A B C D E
route(B, foo.com) route(A, foo.com) link(A, B)
– Network consists of nodes (routers, middleboxes, ...) – The state of a node is a set of tuples (routes, config, ...) – Idea: Explanation as reasoning about distributed state dependencies
3
Alice foo.com
route(C, foo.com) link(C, foo.com)
A B C D E
route(B, foo.com) link(B, C) route(A, foo.com) link(A, B)
4
Alice foo.com
route(C, foo.com) link(C, foo.com)
A B C D E
route(B, foo.com) link(B, C) route(A, foo.com) link(A, B) route(D, foo.com) link(D, E) route(E, foo.com) link(E, B)
[SIGMOD 2010]
– Explains the derivation of tuples – Captures the dependencies between tuples as a graph
4
route(C, foo.com) link(C, foo.com) route(B, foo.com) link(B, C) route(A, foo.com) link(A, B) route(D, foo.com) link(D, E) route(E, foo.com) link(E, B)
[SIGMOD 2010]
– Explains the derivation of tuples – Captures the dependencies between tuples as a graph – Explanation of a tuple is an acyclic graph rooted at the tuple
4
route(C, foo.com) link(C, foo.com) route(B, foo.com) link(B, C) route(A, foo.com) link(A, B)
[SIGMOD 2010]
5
Ph.D. dissertation work of Ang Chen (2017), Chen Chen (2017), Yang Wu (2017), and Wenchao Zhou (2012).
Deeper diagnostics and repair Explanations
6
Alice foo.com Route r2 A D E B C
Why did my route to foo.com change?!
7
The Network Alice foo.com Route r2 A D E B C
Q: Explain why the route to foo.com changed to r2.
7
The Network
A: Because someone accessed Router D and changed the configuration from X to Y.
Alice foo.com Route r2 A D E B C
Q: Explain why the route to foo.com changed to r2.
7
The Network
A: Because someone accessed Router D and changed the configuration from X to Y.
Alice foo.com Route r2 A D E B C
Not realistic: adversary can tell lies
Q: Explain why the route to foo.com changed to r2.
7
8
The Network
Q: Explain why the route to foo.com changed to r2.
Alice foo.com Route r2 A D E B C
Problem: adversary can …
... fabricate plausible (yet incorrect) response … point accusation towards innocent nodes
I should cover up the intrusion.
8
The Network
Q: Explain why the route to foo.com changed to r2.
Alice foo.com Route r2 A D E B C
Problem: adversary can …
... fabricate plausible (yet incorrect) response … point accusation towards innocent nodes
Everything is fine. Router E advertised a new route.
– Split cross-node communications
9
route(C, foo.com) link(C, foo.com) route(B, foo.com) link(B, C) route(A, foo.com) link(A, B) route(D, foo.com) link(D, E) route(E, foo.com) link(E, B)
SOSP 2011
– Split cross-node communications
9
SOSP 2011
– Split cross-node communications
9
SOSP 2011
– Split cross-node communications
9
SOSP 2011
– Split cross-node communications
9
RECV SEND
SOSP 2011
– Split cross-node communications
9
RECV SEND
SOSP 2011
10
The Network
Q: Why did my route to foo.com change to r2? A: Because someone accessed Router D and changed its router configuration from X to Y.
Alice foo.com Route r2 A D E B C
10
The Network
Q: Why did my route to foo.com change to r2? A: Because someone accessed Router D and changed its router configuration from X to Y.
Alice foo.com Route r2 A D E B C
10
The Network
Q: Why did my route to foo.com change to r2? A: Because someone accessed Router D and changed its router configuration from X to Y.
Alice foo.com Route r2 A D E B C
10
The Network
Q: Why did my route to foo.com change to r2? A: Because someone accessed Router D and changed its router configuration from X to Y.
Alice foo.com Route r2 A D E B C
Aha, at least I know which node is compromised.
11
not happening?
handled by existing tools
11
Internet HTTP Server Data Center Network Controller
not happening?
handled by existing tools
11
Internet HTTP Server Data Center Network Controller
???
Why is the HTTP server
NOT getting requests?
not happening?
handled by existing tools
11
17% 83%
Outages
48% 52%
NANOG-user
26% 74%
floodlight-dev
Missing events Positive events NANOG-user Floodlight-dev Outages
12
[SIGCOMM 2014]
13
Internet HTTP Server Data Center Network Controller
[SIGCOMM 2014]
13
Internet HTTP Server Data Center Network Controller
No HTTP Packet arrived at HTTP Server
???
Why is the HTTP server
NOT getting requests?
[SIGCOMM 2014]
13
Internet HTTP Server Data Center Network Controller
No HTTP Packet arrived at HTTP Server No Forwarding-FlowEntry installed at Switch
???
???
Why is the HTTP server
NOT getting requests?
[SIGCOMM 2014]
13
Internet HTTP Server Data Center Network Controller
No HTTP Packet arrived at HTTP Server No Forwarding-FlowEntry installed at Switch HTTP Packet received at Switch
???
??? HTTP Packet
Why is the HTTP server
NOT getting requests?
[SIGCOMM 2014]
13
Internet HTTP Server Data Center Network Controller
No HTTP Packet arrived at HTTP Server No Forwarding-FlowEntry installed at Switch HTTP Packet received at Switch Dropping-FlowEntry existed at Switch
???
??? HTTP Packet Dropping- FlowEntry
Why is the HTTP server
NOT getting requests?
[SIGCOMM 2014]
13
Internet HTTP Server Data Center Network Controller
No HTTP Packet arrived at HTTP Server No Forwarding-FlowEntry installed at Switch HTTP Packet received at Switch Dropping-FlowEntry existed at Switch … Program
???
??? HTTP Packet Dropping- FlowEntry
Why is the HTTP server
NOT getting requests?
[SIGCOMM 2014]
13
Internet HTTP Server Data Center Network Controller
No HTTP Packet arrived at HTTP Server No Forwarding-FlowEntry installed at Switch HTTP Packet received at Switch Dropping-FlowEntry existed at Switch … … Program
…
???
??? HTTP Packet Dropping- FlowEntry
Why is the HTTP server
NOT getting requests?
[SIGCOMM 2014]
13
Find all the ways a missing event could have occurred, and show why each of them did not happen.
14
Find all the ways a missing event could have occurred, Philly and show why each of them did not happen.
14
Find all the ways a missing event could have occurred, Philly Santa Cruz and show why each of them did not happen.
14
Find all the ways a missing event could have occurred,
Why did Bob NOT arrive at CIDR?
Philly Santa Cruz and show why each of them did not happen.
14
Find all the ways a missing event could have occurred,
Why did Bob NOT arrive at CIDR?
Philly Santa Cruz and show why each of them did not happen.
14
Find all the ways a missing event could have occurred,
Why did Bob NOT arrive at CIDR?
Philly Santa Cruz and show why each of them did not happen.
14
Find all the ways a missing event could have occurred,
Why did Bob NOT arrive at CIDR?
Philly Santa Cruz and show why each of them did not happen.
14
Find all the ways a missing event could have occurred,
Why did Bob NOT arrive at CIDR?
Philly Santa Cruz and show why each of them did not happen.
14
15
r
Root cause Symptom
15
r
Root cause Symptom
C received packet B sent packet Rule match on B
15
r
Root cause Symptom
15
r
Root cause Symptom Packet arrives at the wrong server
15
r
Root cause Symptom Packet arrives at the wrong server Rule 7: Next-hop=port2
15
16
From: alice@xyz.com To: Admin (bob@xyz.com) Title: Help! My server is receiving suspicious traffic from 4.3.2.0/24--it should have been sent to the low-security server. Packets from 4.3.3.0/24 are still being routed correctly. Can you help?
16
From: alice@xyz.com To: Admin (bob@xyz.com) Title: Help! My server is receiving suspicious traffic from 4.3.2.0/24--it should have been sent to the low-security server. Packets from 4.3.3.0/24 are still being routed correctly. Can you help?
Working reference!
16
From: alice@xyz.com To: Admin (bob@xyz.com) Title: Help! My server is receiving suspicious traffic from 4.3.2.0/24--it should have been sent to the low-security server. Packets from 4.3.3.0/24 are still being routed correctly. Can you help?
Outages mailing list Sept.—Dec. 2014: 66% have references! Working reference!
16
Web server 1 DPI Web server 2
S1 S2 S3 S4 S5 S6
Bob
16
and the reference
Web server 1 DPI Web server 2
S1 S2 S3 S4 S5 S6
Bob
16
4.3.3.1 fails 4.3.2.1 works
[SIGCOMM 2016]
17
4.3.3.1 fails 4.3.2.1 works
Differential Provenance
[SIGCOMM 2016]
17
4.3.3.1 fails 4.3.2.1 works
Differential Provenance
[SIGCOMM 2016]
17
4.3.3.1 fails 4.3.2.1 works
Differential Provenance Rule 7’s next hop is wrong!
[SIGCOMM 2016]
17
two trees
faulty rule root rootProvenance (Symptom) Provenance (Reference)
18
two trees
Provenance (Symptom) Provenance (Reference)
18
Roll back the execution to a divergence point Change the faulty node to be like the correct node Roll forward the execution to align the trees
19
20
S0
5
S2
3 4
S1 Backup Web Server Main Web Server DNS Server
20
SDN Controller S0
5
S2
3 4
S1 Backup Web Server Main Web Server DNS Server
20
SDN Controller S0
5
S2
3 4
S1 Backup Web Server Main Web Server DNS Server HTTP traffic
20
SDN Controller S0
5
S2
3 4
S1 Backup Web Server Main Web Server DNS Server Off-loading HTTP HTTP traffic
20
else if (switch == S1 && protocol == HTTP) then action = output:3.
SDN Controller S0
5
S2
3 4
S1 Backup Web Server Main Web Server DNS Server Off-loading HTTP HTTP traffic
20
else if (switch == S1 && protocol == HTTP) then action = output:3.
SDN Controller S0
5
S2
3 4
S1 Backup Web Server Main Web Server DNS Server Off-loading HTTP HTTP traffic
20
else if (switch == S1 && protocol == HTTP) then action = output:3. else if (switch == S1 && protocol == HTTP) then action = output:5.
SDN Controller S0
5
S2
3 4
S1 Backup Web Server Main Web Server DNS Server Off-loading HTTP HTTP traffic
20
else if (switch == S1 && protocol == HTTP) then action = output:3. else if (switch == S1 && protocol == HTTP) then action = output:5.
SDN Controller S0
5
S2
3 4
S1 Backup Web Server Main Web Server DNS Server Off-loading HTTP HTTP traffic
20
else if (switch == S1 && protocol == HTTP) then action = output:3. else if (switch == S1 && protocol == HTTP) then action = output:5.
SDN Controller S0
5
S2
3 4
S1 Backup Web Server Main Web Server DNS Server
Copy-and-paste bug!!!
Off-loading HTTP HTTP traffic
20
else if (switch == S1 && protocol == HTTP) then action = output:3. else if (switch == S1 && protocol == HTTP) then action = output:5.
SDN Controller S0
5
S2
3 4
S1 Backup Web Server Main Web Server DNS Server
Copy-and-paste bug!!!
Off-loading HTTP HTTP traffic
20
else if (switch == S1 && protocol == HTTP) then action = output:3. else if (switch == S1 && protocol == HTTP) then action = output:5.
SDN Controller S0
5
S2
3 4
S1 Backup Web Server Main Web Server DNS Server Why is the backup web server not getting requests?
Copy-and-paste bug!!!
Off-loading HTTP HTTP traffic
20
else if (switch == S1 && protocol == HTTP) then action = output:3. else if (switch == S1 && protocol == HTTP) then action = output:5.
SDN Controller S0
5
S2
3 4
S1 Backup Web Server Main Web Server DNS Server
Why is the backup web server not getting requests?
Copy-and-paste bug!!!
Off-loading HTTP HTTP traffic
20
HTTP Packet received at Main Web Server Matching Flow Entry installed at S1
Executed If Clause in Controller Program
PacketIn received at Controller
meta provenance
[NSDI 2017]
21
Full support of Network Datalog (declarative) Uses Z3 SMT solver to enumerate repairs More details are in [HotNets’15, NSDI’17] Support a subset of Pyretic (Python + DSL) Support a subset of Trema (imperative Ruby)
22
Ph.D. dissertation work of Ang Chen (2017), Chen Chen (2017), Yang Wu (2017), and Wenchao Zhou (2012).
Deeper diagnostics and repair Explanations
23
rich area of exploration!
– Network forensics on the data plane – Privacy-preserving provenance on sensitive networks – Probabilistic provenance – Automated repairs of complex events – Timing-based provenance – and more….
24
– Ang Chen, Chen Chen, Ling Ding, Qiong Fei, Andreas Haeberlen, Zachary Ives, Yang Li, Boon Thau Loo, Suyog Mapara, Arjun Narayan, Yiqing Ren, Micah Sherr, Shengzhi Sun, Tao Tao, Yang Wu, Mingchen Zhao, Wenchao Zhou
25