Verifying concurrent software using movers in CSPEC Tej Chajed , - PowerPoint PPT Presentation

Verifying concurrent software using movers in CSPEC Tej Chajed , Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich MIT CSAIL and *Microsoft

Concurrent software is difficult to get right Programmer cannot reason about code in sequence… � 2

Concurrent software is difficult to get right Programmer cannot reason about code in sequence… instead, must consider many executions: � 3

Concurrent software is difficult to get right Programmer cannot reason about code in sequence… instead, must consider many executions: … � 3

Goal: verify concurrent software � 4

Challenge for formal verification • Proofs must also cover every execution • Many approaches to managing this complexity • movers [Lipton, 1975] • rely-guarantee [1983] • RGSep [CONCUR 2007] • FCSL [PLDI 2015] • Iris [POPL 2017, LICS 2018, others] • many others � 5

Challenge for formal verification • Proofs must also cover every execution • Many approaches to managing this complexity • movers [Lipton, 1975] • rely-guarantee [1983] • RGSep [CONCUR 2007] • FCSL [PLDI 2015] • Iris [POPL 2017, LICS 2018, others] • many others • This work: our experience using movers � 5

Movers: reduce concurrent executions to sequential ones time 1 2 3 blue thread 1 A 2 3 B green thread A B � 6

Movers: reduce concurrent executions to sequential ones 1 2 3 blue thread 1 A 2 3 B green thread A B movers has the same e ff ect as 1 2 3 A B � 6

Movers: reduce concurrent executions to sequential ones 1 2 3 blue thread 1 A 2 3 B green thread A B movers has the same e ff ect as 1 2 3 A B sequential reasoning 2 1 3 A B � 6

Prior systems with mover reasoning CIVL [CAV ’15, CAV ’18] framework relies pen & paper proofs IronFleet [SOSP ’15] only move network send/receive � 7

Contribution: CSPEC • Framework for verifying concurrency in systems software • general-purpose movers • patterns to support mover reasoning • machine checked in Coq to support extensibility � 8

Contribution: CSPEC • Framework for verifying concurrency in systems software • general-purpose movers • patterns to support mover reasoning • machine checked in Coq to support extensibility • Case studies using CSPEC • Lock-free file-system concurrency • Spinlock on top of x86-TSO (see paper) � 8

Case study: mail server using file-system concurrency file system spool mbox � 9

Mail servers exploit file-system concurrency # accept def deliver(msg): file system # spool create (“/spool/$TID”) spool mbox write (“/spool/$TID”, msg) # store while True: 1 2 3 t = time.time() if link (“/spool/$TID”, “/mbox/$t”): break # cleanup unlink (“/spool/$TID”) � 10

Mail servers exploit file-system concurrency msg # accept def deliver(msg): file system # spool create (“/spool/$TID”) spool mbox write (“/spool/$TID”, msg) # store while True: 1 2 3 t = time.time() if link (“/spool/$TID”, “/mbox/$t”): break # cleanup unlink (“/spool/$TID”) � 11

Spooling avoids reading partially-written messages $TID =10 # accept def deliver(msg): file system # spool create (“/spool/$TID”) spool mbox write (“/spool/$TID”, msg) # store while True: 1 2 3 t = time.time() if link (“/spool/$TID”, “/mbox/$t”): break # cleanup unlink (“/spool/$TID”) � 12

Spooling avoids reading partially-written messages $TID =10 # accept def deliver(msg): file system # spool create (“/spool/$TID”) spool mbox write (“/spool/$TID”, msg) # store 10 10 while True: 1 2 3 t = time.time() if link (“/spool/$TID”, “/mbox/$t”): break # cleanup unlink (“/spool/$TID”) � 12

Threads use unique IDs to avoid conflicts msg $TID =10 $TID =11 # accept def deliver(msg): file system # spool create (“/spool/$TID”) spool mbox write (“/spool/$TID”, msg) # store 10 while True: 1 2 3 t = time.time() if link (“/spool/$TID”, “/mbox/$t”): break # cleanup unlink (“/spool/$TID”) � 13

Threads use unique IDs to avoid conflicts $TID =10 $TID =11 # accept def deliver(msg): file system # spool create (“/spool/$TID”) spool mbox write (“/spool/$TID”, msg) # store 10 while True: 1 2 3 t = time.time() if link (“/spool/$TID”, “/mbox/$t”): break # cleanup unlink (“/spool/$TID”) � 14

Threads use unique IDs to avoid conflicts $TID =10 $TID =11 # accept def deliver(msg): file system # spool create (“/spool/$TID”) spool mbox write (“/spool/$TID”, msg) # store 10 11 while True: 1 2 3 t = time.time() if link (“/spool/$TID”, “/mbox/$t”): break # cleanup unlink (“/spool/$TID”) � 14

Timestamps help generate unique message names # accept def deliver(msg): file system # spool create (“/spool/$TID”) spool mbox write (“/spool/$TID”, msg) # store 10 11 while True: 1 2 3 4 t = time.time() if link (“/spool/$TID”, “/mbox/$t”): link(/spool/11, /mbox/4) break # cleanup unlink (“/spool/$TID”) � 15

Timestamps help generate unique message names # accept def deliver(msg): file system # spool create (“/spool/$TID”) spool mbox write (“/spool/$TID”, msg) # store 10 11 while True: 1 2 3 4 t = time.time() if link (“/spool/$TID”, “/mbox/$t”): link(/spool/10, /mbox/4) break # cleanup EEXISTS ✗ unlink (“/spool/$TID”) � 16

Timestamps help generate unique message names # accept def deliver(msg): file system # spool create (“/spool/$TID”) spool mbox write (“/spool/$TID”, msg) # store 10 11 while True: 1 2 3 4 5 t = time.time() if link (“/spool/$TID”, “/mbox/$t”): link(/spool/10, /mbox/5) break # cleanup unlink (“/spool/$TID”) � 17

Delivery concurrency does not use locks # accept def deliver(msg): file system # spool create (“/spool/$TID”) spool mbox write (“/spool/$TID”, msg) # store 10 while True: 1 2 3 4 5 t = time.time() if link (“/spool/$TID”, “/mbox/$t”): break # cleanup unlink (“/spool/$TID”) � 18

Delivery concurrency does not use locks # accept def deliver(msg): file system # spool create (“/spool/$TID”) spool mbox write (“/spool/$TID”, msg) # store while True: 1 2 3 4 5 t = time.time() if link (“/spool/$TID”, “/mbox/$t”): break # cleanup unlink (“/spool/$TID”) � 19

Proving delivery correct in CSPEC delivery specification implementation and proof file-system spec CSPEC provides supporting definitions CSPEC and theorems � 20

Proof engineer reasons about file-system operations def deliver(msg): create (“/spool/$TID”, msg) while True: t = time.time() if link (“/spool/$TID”, “/mbox/$t”): break unlink (“/spool/$TID”) create( link( link( unlink( /sp/$TID, /sp/$TID, /sp/$TID, /sp/$TID) msg) /mbox/$t) /mbox/$t) ✓ ✓ ✓ EEXISTS ✗ � 21

Proof engineer reasons about file-system operations collapsed to def deliver(msg): one operation create (“/spool/$TID”) create (“/spool/$TID”, msg) write (“/spool/$TID”, msg) while True: t = time.time() if link (“/spool/$TID”, “/mbox/$t”): break unlink (“/spool/$TID”) create( link( link( unlink( /sp/$TID, /sp/$TID, /sp/$TID, /sp/$TID) msg) /mbox/$t) /mbox/$t) ✓ ✓ ✓ EEXISTS ✗ � 21

Proof engineer reasons about interleaving of file- system operations def deliver(msg): create (“/spool/$TID”, msg) while True: t = time.time() if link (“/spool/$TID”, “/mbox/$t”): break unlink (“/spool/$TID”) create( link( link( unlink( /sp/$TID, /sp/$TID, /sp/$TID, /sp/$TID) create link unlink msg) /mbox/$t) /mbox/$t) ✓ ✓ ✓ ✓ EEXISTS ✗ We assume file-system operations are atomic � 22

Proving atomicity of delivery atomicity : concurrent deliveries appear create create link link link unlink unlink to execute all at once (in some order) ✓ ✓ ✗ deliver deliver create link unlink create link link unlink ✓ ✓ ✗ � 23

Proving atomicity of delivery atomicity : concurrent deliveries appear create create link link link unlink unlink to execute all at once (in some order) ✓ ✓ ✗ Step 1: developer identifies commit point deliver deliver create link unlink create link link unlink ✓ ✓ ✗ � 23

Proving atomicity of delivery atomicity : concurrent deliveries appear create create link link link unlink unlink to execute all at once (in some order) ✓ ✓ ✗ Step 1: developer identifies commit point Step 2: prove operation occurs logically at commit point deliver deliver create link unlink create link link unlink ✓ ✓ ✗ � 23

Example of movers for this execution create create link link link unlink unlink ✓ ✓ ✗ � 24

Example of movers for this execution create create link link link unlink unlink ✓ ✓ ✗ create link create link link unlink unlink ✓ ✓ ✗ � 24

Example of movers for this execution create create link link link unlink unlink ✓ ✓ ✗ create link create link link unlink unlink ✓ ✓ ✗ create link unlink create link link unlink ✓ ✓ ✗ � 24

Right mover can be reordered after any green thread operation A A r r � 25