Deterministic Fast User Space Synchronization Alexander Zpke - PowerPoint PPT Presentation

Deterministic Fast User Space Synchronization Alexander Züpke alexander.zuepke@hs-rm.de RheinMain University of Applied Sciences Wiesbaden, Germany

OSPERT A. Züpke Overview 2013-07-09 2 / 42  Futex Basics  Challenge: Futexes for Partitioning Systems  New Approach  Mutexes  Condition Variables  Locking of Wait Queues  Robustness  Future Work  Summary

OSPERT A. Züpke Mutex State Transitions 2013-07-09 3 / 42  Unlocked ↔ Locked unlocked  Fast path: use atomic ops  No system call involved!  Contention: first waiter locked  Atomically indicate pending waiters  System call: suspend caller locked w/ contention 1 waiter  Kernel allocates a wait queue object  Contention: multiple waiters locked w/ contention  Append to existing wait queue 2+ waiters  Wait queue order depends, sorting if necessary

OSPERT A. Züpke Futexes in Linux 2013-07-09 4 / 42  Futex := 32-bit integer variable in user space  atomic CAS or LL/SC operations in the fast path  Glibc provides:  Mutexes and Condition Variables  Semaphores, Reader-Writer Locks, Barriers, …  Linux kernel provides system calls to:  suspend the caller  wake a given number of waiters  First prototype in Linux kernel version 2.5.7

OSPERT A. Züpke Futexes in Linux 2013-07-09 5 / 42  Futex API #include <linux/futex.h> int futex(int *uaddr, int op, int val, const struct timespec *timeout, int *uaddr2, int val3);  Operations Suspend calling thread on futex uaddr FUTEX_WAIT  Wake val threads waiting on futex uaddr FUTEX_WAKE  Move threads waiting on uaddr to uaddr2 FUTEX_REQUEUE   … more operations available → see FUTEX(2) man page

OSPERT A. Züpke Motivation 2013-07-09 6 / 42  Linux Implementation  Requires system calls only on contention  Supports an arbitrary number of futexes  No kernel resources required until suspension  Also supports PI mutexes & condition variables  Futexes are really nice … for Un*x Kernels

OSPERT A. Züpke Motivation 2013-07-09 7 / 42  Linux Implementation  Requires system calls only on contention  Supports an arbitrary number of futexes  No kernel resources required until suspension  Also supports PI mutexes & condition variables  But:  Can we use futexes in partitioned environments?  For highly safety critical systems?  Kernels without SLAB allocator?

OSPERT A. Züpke Motivation 2013-07-09 8 / 42  Define ”Partitioning”  space and time partitioning  Isolated (groups of) processes  kernel resources are partitioned Partition A Partition B Futex a thread SHM

OSPERT A. Züpke Motivation 2013-07-09 9 / 42  Define ”Partitioning”  space and time partitioning  Isolated (groups of) processes  kernel resources are partitioned Partition A Partition B lock lock Futex SHM

OSPERT A. Züpke Motivation 2013-07-09 10 / 42  Define ”Partitioning”  space and time partitioning  Isolated (groups of) processes  kernel resources are partitioned  Problem Partition A Partition B  Q: Wait queue belongs to lock lock Partition A or Partition B? Futex  Pre-allocated w. queues? SHM ? ?  Too pessimistic! Kernel Obj

OSPERT A. Züpke Motivation 2013-07-09 11 / 42  Define ”Partitioning”  space and time partitioning  Isolated (groups of) processes  kernel resources are partitioned  Problem Partition A Partition B  Q: Wait queue belongs to lock lock Partition A or Partition B? Futex  Pre-allocated w. queues? SHM ? ?  Too pessimistic!  Idea: get rid of kernel object! Kernel Obj

OSPERT A. Züpke Motivation 2013-07-09 12 / 42  Get rid of the kernel object!  The Linux Futex implementation uses:  array of futex hash entries  lock  list head in-kernel objects  in-kernel object  list node in futex hash  key (futex address)  wait queue  lock pointer

OSPERT A. Züpke Motivation 2013-07-09 13 / 42  Get rid of the kernel object!  The Linux Futex implementation uses:  array of futex hash entries  lock  list head in-kernel objects  in-kernel object  list node in futex hash  key (futex address) put into  wait queue TCB  lock pointer

OSPERT A. Züpke Requirements 2013-07-09 14 / 42  Identify correct wait queue futex Thread ID  use thread ID of the first waiter of 1st waiter  put thread ID into user space, next to futex  Wait queue implementation in linear space  a priority sorted wait queue would be nice  Locking of the wait queue  assume a single kernel lock for now → more on that later

OSPERT A. Züpke Requirements 2013-07-09 15 / 42  Algorithms need bounded WCET  depends on # of waiters  # of waiters probably not known in advance → tricky across partition boundaries  Wait Queues  doubly-linked lists are O(1) ... except searching  sorted wait queues with O(log n) are acceptable if the upper bound of O(log n) is known  O(n) is only acceptable if n is bounded  Pick FIFO-ordered doubly-linked list for now

OSPERT A. Züpke Mutex Protocol 2013-07-09 16 / 42  Example Futex Encoding: Lock Holder ID < T | W > Waiters Bit  2 processes Wait Queue Q  3 threads  futex in shared memory  mutex protocol Process A Process B a thread a b  Symbols Futex SHM c  T: lock holder's thread ID  W:bit indicating non-empty wait queue  Q: thread ID of first waiting thread

OSPERT A. Züpke Mutex Protocol 2013-07-09 17 / 42  Sequence  0. initial state: mutex unlocked  1. yellow tries to lock & suceeds  2. blue tries & sets W & suspends  3. green tries & suspends Process A Process B  4. yellow unlocks & wakes a b  5. blue becomes owner 0 | 0 0  6. blue unlocks & wakes SHM c  7. green becomes owner  8. green unlocks → mutex unlocked

OSPERT A. Züpke Mutex Protocol 2013-07-09 18 / 42  Sequence  0. initial state: mutex unlocked  1. yellow tries to lock & suceeds  2. blue tries & sets W & suspends  3. green tries & suspends Process A Process B  4. yellow unlocks & wakes a lock b  5. blue becomes owner 0 | 0 0  6. blue unlocks & wakes SHM c  7. green becomes owner  8. green unlocks → mutex unlocked

OSPERT A. Züpke Mutex Protocol 2013-07-09 19 / 42  Sequence  0. initial state: mutex unlocked  1. yellow tries to lock & suceeds  2. blue tries & sets W & suspends  3. green tries & suspends Process A Process B  4. yellow unlocks & wakes a lock holder b  5. blue becomes owner a | 0 0  6. blue unlocks & wakes SHM c  7. green becomes owner  8. green unlocks → mutex unlocked

OSPERT A. Züpke Mutex Protocol 2013-07-09 20 / 42  Sequence  0. initial state: mutex unlocked  1. yellow tries to lock & suceeds  2. blue tries & sets W & suspends  3. green tries & suspends Process A Process B  4. yellow unlocks & wakes a lock holder lock b  5. blue becomes owner a | 0 0  6. blue unlocks & wakes SHM c  7. green becomes owner  8. green unlocks → mutex unlocked

OSPERT A. Züpke Mutex Protocol 2013-07-09 21 / 42  Sequence  0. initial state: mutex unlocked  1. yellow tries to lock & suceeds  2. blue tries & sets W & suspends  3. green tries & suspends Process A Process B  4. yellow unlocks & wakes a lock holder lock b  5. blue becomes owner a | W 0  6. blue unlocks & wakes SHM c  7. green becomes owner  8. green unlocks → mutex unlocked

OSPERT A. Züpke Mutex Protocol 2013-07-09 22 / 42  Sequence  0. initial state: mutex unlocked  1. yellow tries to lock & suceeds  2. blue tries & sets W & suspends  3. green tries & suspends Process A Process B  4. yellow unlocks & wakes a lock holder  5. blue becomes owner a | W b  6. blue unlocks & wakes SHM c  7. green becomes owner Wait  8. green unlocks → mutex unlocked b Queue

OSPERT A. Züpke Mutex Protocol 2013-07-09 23 / 42  Sequence  0. initial state: mutex unlocked  1. yellow tries to lock & suceeds  2. blue tries & sets W & suspends  3. green tries & suspends Process A Process B  4. yellow unlocks & wakes a lock holder  5. blue becomes owner a | W lock b  6. blue unlocks & wakes SHM c  7. green becomes owner Wait  8. green unlocks → mutex unlocked b Queue

OSPERT A. Züpke Mutex Protocol 2013-07-09 24 / 42  Sequence  0. initial state: mutex unlocked  1. yellow tries to lock & suceeds  2. blue tries & sets W & suspends  3. green tries & suspends Process A Process B  4. yellow unlocks & wakes a lock holder  5. blue becomes owner a | W b  6. blue unlocks & wakes SHM  7. green becomes owner Wait  8. green unlocks → mutex unlocked b c Queue

OSPERT A. Züpke Mutex Protocol 2013-07-09 25 / 42  Sequence  0. initial state: mutex unlocked  1. yellow tries to lock & suceeds  2. blue tries & sets W & suspends  3. green tries & suspends Process A Process B  4. yellow unlocks & wakes a unlock  5. blue becomes owner a | W b  6. blue unlocks & wakes SHM  7. green becomes owner Wait  8. green unlocks → mutex unlocked b c Queue