SLIDE 1 1/19
Distributed Programming using Role-Parametric Session Types in Go
David Castro1, Raymond Hu1, Sung-Shik Jongmans1,2, Nicholas Ng1, Nobuko Yoshida1
1 Imperial College London 2 Open University of the Netherlands
SLIDE 2 1/19
Distributed Programming using Role-Parametric Session Types in Go
David Castro1, Raymond Hu1, Sung-Shik Jongmans1,2, Nicholas Ng1, Nobuko Yoshida1
1 Imperial College London 2 Open University of the Netherlands
SLIDE 3 1/19
Distributed Programming using Role-Parametric Session Types in Go
David Castro1, Raymond Hu1, Sung-Shik Jongmans1,2, Nicholas Ng1, Nobuko Yoshida1
1 Imperial College London 2 Open University of the Netherlands
SLIDE 4
2/19
Introduction (distributed programming in Go) Long-term research agenda: Development of theory and tools to help programmers write safe concurrent programs
SLIDE 5
2/19
Introduction (distributed programming in Go) Long-term research agenda: Development of theory and tools to help Go programmers write safe concurrent Go programs [CC’16, POPL’17, ICSE’18]
SLIDE 6
3/19
Introduction (distributed programming in Go) (a) Modern, popular systems language
SLIDE 7
3/19
Introduction (distributed programming in Go) (a) Modern, popular systems language (b) Primacy of CSP-based concurrency features Lightweight threads, called goroutines Higher-order, typed native channels (across shared memory) First-order, untyped API channels (across a network)
SLIDE 8
3/19
Introduction (distributed programming in Go) (a) Modern, popular systems language (b) Primacy of CSP-based concurrency features Lightweight threads, called goroutines Higher-order, typed native channels (across shared memory) First-order, untyped API channels (across a network) (c) Survey: “Users least agreed that they are able to effectively debug uses of Go’s concurrency features”
SLIDE 9 3/19
Introduction (distributed programming in Go) (a) Modern, popular systems language (b) Primacy of CSP-based concurrency features Lightweight threads, called goroutines Higher-order, typed native channels (across shared memory) First-order, untyped API channels (across a network) (c) Survey: “Users least agreed that they are able to effectively debug uses of Go’s concurrency features” m u l t i p a r t y s e s s i
t y p e s ? [ P O P L ’ 8 ]
SLIDE 10
4/19
Introduction (distributed programming in Go) Motivating example: htcat (https://github.com/htcat/htcat) Parallel downloader of webpages Post-factum verification very difficult Our safe-by-construction version: PGet ( )
SLIDE 11
5/19
Introduction (distributed programming in Go) M F1 F2 Fn S Master Fetchers Server
SLIDE 12 5/19
Introduction (distributed programming in Go) M F1 F2 Fn S Master Fetchers Server f e a t u r e 1 : p a r a m e t e r i s a t i
( i n # F e t c h e r s )
SLIDE 13 5/19
Introduction (distributed programming in Go) M F1 F2 Fn S Master Fetchers Server
Local Remote
f e a t u r e 1 : p a r a m e t e r i s a t i
( i n # F e t c h e r s ) shared memory channel TCP channel
SLIDE 14 5/19
Introduction (distributed programming in Go) M F1 F2 Fn S Master Fetchers Server
Local Remote
f e a t u r e 1 : p a r a m e t e r i s a t i
( i n # F e t c h e r s ) f e a t u r e 2 : m i x e d t r a n s p
t s & d i s p a r a t e a b s t r a c t i
s shared memory channel TCP channel
SLIDE 15 5/19
Introduction (distributed programming in Go) M F1 F2 Fn S Master Fetchers Server
Local Remote
f e a t u r e 1 : p a r a m e t e r i s a t i
( i n # F e t c h e r s ) f e a t u r e 2 : m i x e d t r a n s p
t s & d i s p a r a t e a b s t r a c t i
s
M → F1 : : :GetSize(string)
SLIDE 16 5/19
Introduction (distributed programming in Go) M F1 F2 Fn S Master Fetchers Server
Local Remote
f e a t u r e 1 : p a r a m e t e r i s a t i
( i n # F e t c h e r s ) f e a t u r e 2 : m i x e d t r a n s p
t s & d i s p a r a t e a b s t r a c t i
s
F1 → S: : :HttpReq(byte[])
SLIDE 17 5/19
Introduction (distributed programming in Go) M F1 F2 Fn S Master Fetchers Server
Local Remote
f e a t u r e 1 : p a r a m e t e r i s a t i
( i n # F e t c h e r s ) f e a t u r e 2 : m i x e d t r a n s p
t s & d i s p a r a t e a b s t r a c t i
s
F1 → S: : :HttpReq(byte[]). . . S → F1 : : :HttpRes(byte[])
SLIDE 18 5/19
Introduction (distributed programming in Go) M F1 F2 Fn S Master Fetchers Server
Local Remote
f e a t u r e 1 : p a r a m e t e r i s a t i
( i n # F e t c h e r s ) f e a t u r e 2 : m i x e d t r a n s p
t s & d i s p a r a t e a b s t r a c t i
s
F1 → M: : :Size(int)
SLIDE 19 5/19
Introduction (distributed programming in Go) M F1 F2 Fn S Master Fetchers Server
Local Remote
f e a t u r e 1 : p a r a m e t e r i s a t i
( i n # F e t c h e r s ) f e a t u r e 2 : m i x e d t r a n s p
t s & d i s p a r a t e a b s t r a c t i
s
M → F[1. .n]: : :GetData(string,int,int)
SLIDE 20 5/19
Introduction (distributed programming in Go) M F1 F2 Fn S Master Fetchers Server
Local Remote
f e a t u r e 1 : p a r a m e t e r i s a t i
( i n # F e t c h e r s ) f e a t u r e 2 : m i x e d t r a n s p
t s & d i s p a r a t e a b s t r a c t i
s
F[1. .n] → S: : :HttpReq(byte[])
SLIDE 21 5/19
Introduction (distributed programming in Go) M F1 F2 Fn S Master Fetchers Server
Local Remote
f e a t u r e 1 : p a r a m e t e r i s a t i
( i n # F e t c h e r s ) f e a t u r e 2 : m i x e d t r a n s p
t s & d i s p a r a t e a b s t r a c t i
s
F[1. .n] → S: : :HttpReq(byte[]). . . S → F[1. .n]: : :HttpRes(byte[])
SLIDE 22 5/19
Introduction (distributed programming in Go) M F1 F2 Fn S Master Fetchers Server
Local Remote
f e a t u r e 1 : p a r a m e t e r i s a t i
( i n # F e t c h e r s ) f e a t u r e 2 : m i x e d t r a n s p
t s & d i s p a r a t e a b s t r a c t i
s
F[1. .n] → M: : :Data(string,chan chan chan)
SLIDE 23 5/19
Introduction (distributed programming in Go) M F1 F2 Fn S Master Fetchers Server
Local Remote
f e a t u r e 1 : p a r a m e t e r i s a t i
( i n # F e t c h e r s ) f e a t u r e 2 : m i x e d t r a n s p
t s & d i s p a r a t e a b s t r a c t i
s
F[1. .n] → M: : :Data(string,chan chan chan)
SLIDE 24 5/19
Introduction (distributed programming in Go) M F1 F2 Fn S Master Fetchers Server
Local Remote
f e a t u r e 1 : p a r a m e t e r i s a t i
( i n # F e t c h e r s ) f e a t u r e 2 : m i x e d t r a n s p
t s & d i s p a r a t e a b s t r a c t i
s feature 3: channel passing
F[1. .n] → M: : :Data(string,chan chan chan)
SLIDE 25 5/19
Introduction (distributed programming in Go) M F1 F2 Fn S Master Fetchers Server
Local Remote
f e a t u r e 1 : p a r a m e t e r i s a t i
( i n # F e t c h e r s ) f e a t u r e 2 : m i x e d t r a n s p
t s & d i s p a r a t e a b s t r a c t i
s feature 3: channel passing
SLIDE 26 5/19
Introduction (distributed programming in Go) M F1 F2 Fn S F1 Master Fetchers Server
Local Remote
f e a t u r e 1 : p a r a m e t e r i s a t i
( i n # F e t c h e r s ) f e a t u r e 2 : m i x e d t r a n s p
t s & d i s p a r a t e a b s t r a c t i
s feature 3: channel passing f e a t u r e 4 : h e t e r
e n e
s r
e s
SLIDE 27
6/19
Introduction (distributed programming in Go) Features: Parameterisation (in #Fetchers) Mixed transports & disparate abstractions Channel passing Heterogeneous roles
SLIDE 28
6/19
Introduction (distributed programming in Go) Features: Parameterisation (in #Fetchers) Mixed transports & disparate abstractions Channel passing Heterogeneous roles Challenges (safety ): Protocol compliance Deadlock-freedom
SLIDE 29
6/19
Introduction (distributed programming in Go) Features: Parameterisation (in #Fetchers) Mixed transports & disparate abstractions Channel passing Heterogeneous roles Challenges (safety ): Protocol compliance Deadlock-freedom multiparty session types:
SLIDE 30 6/19
Introduction (distributed programming in Go) Features: Parameterisation (in #Fetchers) Mixed transports & disparate abstractions Channel passing Heterogeneous roles Challenges (safety ): Protocol compliance Deadlock-freedom multiparty session types:
- real programs need more expressive theory and impl.
SLIDE 31
7/19
Introduction (multiparty session types; MPST) W1 W2 W3 processes
SLIDE 32
7/19
Introduction (multiparty session types; MPST) W1 W2 W3 processes G global type G =
W1 → W2 : : :Int. . . W2 → W3 : : :Bool
SLIDE 33 7/19
Introduction (multiparty session types; MPST) W1 W2 W3 processes G global type L1 L2 L3 local types
p r
e c t
G =
W1 → W2 : : :Int. . . W2 → W3 : : :Bool
L1 = W2!
! !Int
L2 = W1?
? ?Int. . . W3! ! !Bool
L3 = W2?
? ?Bool
SLIDE 34 7/19
Introduction (multiparty session types; MPST) W1 W2 W3 processes G global type L1 L2 L3 local types
p r
e c t type- check
G =
W1 → W2 : : :Int. . . W2 → W3 : : :Bool
L1 = W2!
! !Int
L2 = W1?
? ?Int. . . W3! ! !Bool
L3 = W2?
? ?Bool
SLIDE 35 7/19
Introduction (multiparty session types; MPST) well-typed ⇒ protocol compliance ∧ deadlock-freedom W1 W2 W3 processes G global type L1 L2 L3 local types
p r
e c t type- check
G =
W1 → W2 : : :Int. . . W2 → W3 : : :Bool
L1 = W2!
! !Int
L2 = W1?
? ?Int. . . W3! ! !Bool
L3 = W2?
? ?Bool
SLIDE 36 7/19
Introduction (multiparty session types; MPST) well-typed ⇒ protocol compliance ∧ deadlock-freedom W1 W2 W3 processes G global type L1 L2 L3 local types
p r
e c t type- check
G =
W1 → W2 : : :Int. . . W2 → W3 : : :Bool
L1 = W2!
! !Int
L2 = W1?
? ?Int. . . W3! ! !Bool
L3 = W2?
? ?Bool
SLIDE 37
8/19
Contributions Theory: MPST + parameterisation + role heterogeneity Proofs of decidability and correctness
SLIDE 38
8/19
Contributions Theory: MPST + parameterisation + role heterogeneity Proofs of decidability and correctness Implementation: Extension to Scribble [FASE’16, FASE’17] Artifact (reusable and available )
SLIDE 39
8/19
Contributions Theory: MPST + parameterisation + role heterogeneity Proofs of decidability and correctness Implementation: Extension to Scribble [FASE’16, FASE’17] Artifact (reusable and available ) Evaluation: Competitive performance Wide applicability
SLIDE 40
9/19
Theory Easy part: Parameterisation G = foreach
foreach foreach W[i:1. .n-1, j:2. .n] do do do W[i] → W[j]: : :Msg
SLIDE 41
9/19
Theory Easy part: Parameterisation G = foreach
foreach foreach W[i:1. .n-1, j:2. .n] do do do W[i] → W[j]: : :Msg
Hard part: Role heterogeneity How to infer from G there exist three role variants? (first Worker; middle Workers; last Worker)
SLIDE 42
10/19
Theory G = foreach
foreach foreach W[i:1. .n-1, j:2. .n] do do do W[i] → W[j]: : :Msg
Key insight: Behaviour of Worker x is determined by the intervals in which x occurs (i.e., if x and y are contained in the same intervals, Workers x and y behave the same)
SLIDE 43
10/19
Theory G = foreach
foreach foreach W[i:1. .n-1, j:2. .n] do do do W[i] → W[j]: : :Msg
Key insight: Behaviour of Worker x is determined by the intervals in which x occurs (i.e., if x and y are contained in the same intervals, Workers x and y behave the same) x ∈ 1.
.n-1 ∧ x ∈ 2. .n ⇒ x ∈ 2. .n-1
(middle Worker) x ∈ 1.
.n-1 ∧ x /
∈ 2.
.n ⇒ x = 1
(first Worker) x / ∈ 1.
.n-1 ∧ x ∈ 2. .n ⇒ x = n
(last Worker) x / ∈ 1.
.n-1 ∧ x /
∈ 2.
.n ⇒ ⊥
SLIDE 44 11/19
Theory
- 1. Infer role variants as triples r[D, ¯
D], where: r is a role name D is a set of intervals ¯ D is a set of “co-intervals”
SLIDE 45 11/19
Theory
- 1. Infer role variants as triples r[D, ¯
D], where: r is a role name D is a set of intervals ¯ D is a set of “co-intervals”
- 2. Project G onto inferred role variants, e.g.:
G ↾ W[{1.
.n-1, 2. .n}, ∅] = W[self self self-1]? ? ?Msg. . . W[self self self+1]! ! !Msg
G ↾ W[{1.
.n-1}, {2. .n}] = W[self self self+1]! ! !Msg
G ↾ W[{2.
.n}, {1. .n-1}] = W[self self self-1]? ? ?Msg
SLIDE 46
12/19
Theory Theorem: Inferring role variants is decidable Theorem: Checking well-formedness is decidable Theorem: Projecting well-formed global types is semantics-preserving, i.e., correct
SLIDE 47 13/19
Implementation Extension of protocol description language Scribble (http://www.scribble.org) protocol spec state machines APIs
role variants, well-formedness, and projection (using Z3) Go code generation
global type local types (role variant-specific)
SLIDE 48 13/19
Implementation Extension of protocol description language Scribble (http://www.scribble.org) protocol spec state machines APIs
role variants, well-formedness, and projection (using Z3) Go code generation
global type local types (role variant-specific) A P I s g u i d e p r
r a m m e r t
a r d s s a f e t y
SLIDE 49
14/19
Implementation (demo video)
SLIDE 50
15/19
Implementation Guarantees: Protocol compliance Deadlock-freedom (up to “protocol-unrelated” program behaviour, premature termination, and delegation) Achieved through: Native Go typing Lightweight run-time checks for linearity
SLIDE 51
16/19
Evaluation (benchmarks) shm tcp Microbenchmarks Speed-up (t1/t2) of Scribble (t2) vs. native Go (t1) Per communication: ∼20ns
SLIDE 52
16/19
Evaluation (benchmarks) shm tcp Microbenchmarks Speed-up (t1/t2) of Scribble (t2) vs. native Go (t1) Per communication: ∼20ns Computer Language Benchmark Games (CLBG)
SLIDE 53
17/19
Evaluation (expressiveness) 21 patterns, topologies, and applications (each uses various features of our framework)
SLIDE 54
18/19
Conclusion Also in the paper: Branching, selection, recursion, merge Implementation Transport independence Linearity checks (Go does not have linear types) Technical report with all details: https://www.doc.ic.ac.uk/research/ technicalreports/2018/#4
SLIDE 55
19/19
Conclusion Theory: MPST + parameterisation + role heterogeneity Proofs of decidability and correctness Implementation: Extension to Scribble Artifact (reusable and available ) Evaluation: Competitive performance Wide applicability
