Declarative, Secure, Convergent Edge Computation Christopher Meiklejohn Université catholique de Louvain, Belgium 1
Internet of Things 2
Internet of Things but, more generally… 2
Edge Computation • Logical extremes Pushing both computation and data to the logical extremes of the network 3
Edge Computation • Logical extremes Pushing both computation and data to the logical extremes of the network • Arbitrary computation Support arbitrary computations regardless of location of data in the network 3
Edge Computation • Logical extremes Pushing both computation and data to the logical extremes of the network • Arbitrary computation Support arbitrary computations regardless of location of data in the network • Self-organizing, resilient Directed diffusion, Cornell circa-1990; self-organizing systems that coordinate to complete computations 3
Example Application Hospital Refrigerators 4
Hospital Refrigerators Typical Topology 5
6
Internet 7
Internet HDFS 8
Client Hadoop Internet Client Internet HDFS 9
Internet HDFS Client Spark Internet Client 10
Hospital Refrigerators Ideal Execution 11
Spark Internet HDFS 12
Spark Internet HDFS 13
Spark Internet HDFS 14
Spark Internet HDFS 15
Spark Internet HDFS 16
Spark Client Internet HDFS 17
Problem Connectivity 18
Spark Internet HDFS 19
Spark Internet HDFS 20
Spark Internet HDFS 21
Spark Internet HDFS 22
Spark Internet HDFS 23
Spark Internet HDFS 24
Spark Internet HDFS 25
Solution Local Decisions 26
Spark Internet HDFS 27
Spark Internet HDFS 28
Spark Internet HDFS 29
Spark Internet HDFS 30
Spark Internet HDFS 31
Local Decisions • Not new for backup (80s-90s) Backup communication mechanisms for critical systems; POTS backup for ISDN, etc. 32
Local Decisions • Not new for backup (80s-90s) Backup communication mechanisms for critical systems; POTS backup for ISDN, etc. • Not new for storage (90s-00s) EMC’s “phone home” via POTS when disks failed in NAS devices to signal for replacement unit 32
Solution Transitive Dissemination 33
Spark Internet HDFS 34
Spark Internet HDFS 35
Spark Internet HDFS 36
Spark Internet HDFS 37
Spark Internet HDFS 38
Spark Client Internet HDFS 39
Problem State Transmission 40
Internet 41
Internet 42
Internet 43
Internet 44
Solution Aggregate Dissemination 45
Internet 46
Internet 47
Internet 48
Internet HDFS 49
Internet HDFS = = 50
Internet ? = = 51
Internet ? = = 52
Local Computation • Reduce state transmission Perform some local computation to reduce transmitted state on the wire 53
Local Computation • Reduce state transmission Perform some local computation to reduce transmitted state on the wire • Make local decisions Make decisions based on results of local computation 53
Databases Consistency Models 54
Databases Strong Consistency 55
R1 C1 C2 56
R1 C1 C2 57
R1 Read C1 C2 58
R1 Read C1 C2 59
R1 C1 C2 60
R1 CAS C1 C2 61
R1 C1 C2 62
R1 CAS C1 C2 63
I won’t diagram the Paxos protocol 64
Value 2 R1 R2 R3 Paxos C1 C2 Value 2 Value 1 65
Databases Eventual Consistency 66
R1 R2 R3 C1 C2 67
R1 R2 R3 C1 C2 68
R1 R2 R3 Read C1 C2 69
R1 R2 R3 Read C1 C2 70
R1 R2 R3 Write C1 C2 71
R1 R2 R3 Write C1 C2 72
R1 R2 R3 Write C1 C1 C2 73
R1 R2 R3 Read C1 C2 74
R1 R2 R3 Write C1 C2 75
Eventual Consistency As The Model 76
Clients Own Their Data 77
78
79
80
Computations Mergeability & Provenance 81
A C A B 82
A F D’ C A F D’’ B 83
A F D’ C ≤ ≤ D’ D’’ D A F D’’ B 84
A F D’ D C ≤ ≤ D’ D’’ D Merge A F D’’ D B 85
Example Application Preliminary Results 86
Preliminary Results • Conflict-Free Replicated Data Types Distributed data structures designed for convergence [Shapiro et al., 2011] 87
Preliminary Results • Conflict-Free Replicated Data Types Distributed data structures designed for convergence [Shapiro et al., 2011] • Lattice Processing Make decisions based on results of local computation [Meiklejohn & Van Roy, 2015] 87
Preliminary Results • Conflict-Free Replicated Data Types Distributed data structures designed for convergence [Shapiro et al., 2011] • Lattice Processing Make decisions based on results of local computation [Meiklejohn & Van Roy, 2015] • Selective Hearing Scalable, epidemic broadcast based runtime system [Meiklejohn & Van Roy, 2015/2016] 87
Conflict-Free Replicated Data Types • Collection of types Sets, counters, registers, flags, maps 88
Conflict-Free Replicated Data Types • Collection of types Sets, counters, registers, flags, maps • Strong Eventual Consistency (SEC) Objects that receive the same updates, regardless of order, will reach equivalent state 88
RA RB RC
add(1) RA {1} (1, {a}, {}) RB RC
add(1) RA {1} (1, {a}, {}) RB add(1) RC {1} (1, {c}, {})
add(1) RA {1} (1, {a}, {}) RB add(1) remove(1) RC {1} {} (1, {c}, {c}) (1, {c}, {})
Recommend
More recommend
