TimerShield Protecting High-Priority Tasks from Low-Priority Timer - - PowerPoint PPT Presentation
TimerShield Protecting High-Priority Tasks from Low-Priority Timer Interference Pratyush Patel 1,2 , Manohar Vanga 1 , Bjrn Brandenburg 1 1 MPI-SWS, 2 Carnegie Mellon University RTAS 2017 April 18, 2017 Pittsburgh, USA Kaiserslautern,
Pratyush Patel1,2, Manohar Vanga1, Björn Brandenburg1
1MPI-SWS, 2Carnegie Mellon University
RTAS 2017 April 18, 2017 Pittsburgh, USA
Kaiserslautern, Germany
2
PREEMPT_RT
hrtimers
3
PREEMPT_RT
hrtimers
Default high-resolution timer subsystem
4
PREEMPT_RT
hrtimers
5
Unnecessary low- priority timer-interrupt interference Default high-resolution timer subsystem
PREEMPT_RT
hrtimers
6
Unnecessary low- priority timer-interrupt interference Default high-resolution timer subsystem
TimerShield
PREEMPT_RT
hrtimers
7
Unnecessary low- priority timer-interrupt interference Default high-resolution timer subsystem A drop-in replacement for hrtimers
TimerShield
Unnecessary low- priority timer-interrupt interference
hrtimers
Default high-resolution timer subsystem
TimerShield
A drop-in replacement for hrtimers
8
PREEMPT_RT
Unnecessary low- priority timer-interrupt interference PREEMPT_RT
hrtimers
Default high-resolution timer subsystem Eliminates low-priority timer-interrupt interference
TimerShield
9
A drop-in replacement for hrtimers
Timers and the Interference Problem TimerShield Design Evaluation
10
Timers and the Interference Problem TimerShield Design Evaluation
11
12
Core 1 Timer Core 2 Timer Core 3 Timer Core 4 Timer
13
Core-local timers with cycle precision Core 1 Timer Core 2 Timer Core 3 Timer Core 4 Timer
14
Can be programmed to raise an interrupt at a desired time Core-local timers with cycle precision Core 1 Timer Core 2 Timer Core 3 Timer Core 4 Timer
15
Job Releases Tasks can be woken up periodically using timers
16
Job Releases Tasks can be woken up periodically using timers Budget Enforcement Schedulers use timers to prevent budget overruns
17
Job Releases Tasks can be woken up periodically using timers Budget Enforcement Schedulers use timers to prevent budget overruns Self-Suspensions Tasks can use POSIX clock_nanosleep() to suspend themselves
18
19
Uniprocessor Partitioned Multiprocessor Fixed-priority scheduling
LP
2 4 6 8 10 12
Low-Priority Task
20
LP
2 4 6 8 10 12
Low-Priority Task Calls clock_nanosleep(6)
21
LP
2 4 6 8 10 12
Low-Priority Task Calls clock_nanosleep(6) Timer hardware is programmed to fire at the specified time
22
HP LP
2 4 6 8 10 12
Low-Priority Task High-Priority Task High-priority task is released
23
HP LP
2 4 6 8 10 12
Low-Priority Task High-Priority Task At t = 6, timer hardware fires an interrupt
24
HP LP
2 4 6 8 10 12
Low-Priority Task High-Priority Task Timer Handler At t = 6, timer hardware fires an interrupt HP is preempted to service the interrupt (LP task is woken up)
25
HP is preempted to service the interrupt (LP task is woken up) HP task resumes LP HP HP LP
2 4 6 8 10 12
Low-Priority Task High-Priority Task Timer Handler
26
Unnecessary interference LP HP HP LP
2 4 6 8 10 12
Low-Priority Task High-Priority Task Timer Handler
27
13 8 17 1 11 20 15
28
Linux’s hrtimer subsystem
13 8 17 1 11 20 15
29
Multiplexes many software timers on a single hardware timer using a time-ordered red-black tree Linux’s hrtimer subsystem
13 8 17 1 11 20 15
30
Earliest expiring timer is programmed into hardware Linux’s hrtimer subsystem
13 8 17 1 11 20 15
31
Earliest expiring timer is programmed into hardware Linux’s hrtimer subsystem But, earliest timer could belong to the lowest-priority task!
13 8 17 1 11 20 15
32
Earliest expiring timer is programmed into hardware Linux’s hrtimer subsystem But, earliest timer could belong to the lowest-priority task! May interrupt a higher-priority task!
13 8 17 1 11 20 15
33
Earliest expiring timer is programmed into hardware Linux’s hrtimer subsystem But, earliest timer could belong to the lowest-priority task! May interrupt a higher-priority task!
Key Problem hrtimers does not take into account the priority of the process that created the timer
Timers and the Interference Problem TimerShield Design Evaluation
34
LP
2 4 6 8 10 12
Low-Priority Task
35
LP
2 4 6 8 10 12
Low-Priority Task High-Priority Task
36
LP
2 4 6 8 10 12
Low-Priority Task High-Priority Task Mask all the low-priority timers
37
HP LP
2 4 6 8 10 12
Low-Priority Task High-Priority Task Mask all the low-priority timers
38
Process the expired low-priority timers
HP LP
2 4 6 8 10 12
Low-Priority Task High-Priority Task Mask all the low-priority timers
39
Mask all the low-priority timers
HP LP
2 4 6 8 10 12
Low-Priority Task High-Priority Task Timer Handler Process the expired low-priority timers Timer processing shifted
40
LP HP LP
2 4 6 8 10 12
Low-Priority Task High-Priority Task Timer Handler Timer processing (interrupt top-half) is safely deferred
41
LP
2 4 6 8 10 12
Low-Priority Task
42
Timer inherits task priority
the earliest timer with priority ≥ HP
LP
2 4 6 8 10 12
Low-Priority Task High-Priority Task
43
the earliest timer with priority ≥ HP
HP LP
2 4 6 8 10 12
Low-Priority Task High-Priority Task
44
(lower priority ones can still be deferred)
HP LP
2 4 6 8 10 12
Low-Priority Task High-Priority Task
45
the earliest timer with priority ≥ HP
HP LP
2 4 6 8 10 12
Low-Priority Task High-Priority Task Timer Handler
46
the earliest timer with priority ≥ HP
(lower priority ones can still be deferred)
HP LP
2 4 6 8 10 12
Low-Priority Task High-Priority Task Timer Handler
47
These operations need to be inexpensive to work well in practice
the earliest timer with priority ≥ HP
(lower priority ones can still be deferred)
48
1: Find the earliest timer at each priority level
49
1: Find the earliest timer at each priority level 2: Among these, find the earliest timer in the priority range [curr_task_prio, max_system_prio]
50
1: Find the earliest timer at each priority level
A Range Minimum Query! (RMQ)
2: Among these, find the earliest timer in the priority range [curr_task_prio, max_system_prio]
1 2 3
……
140 NULL Priority Level
51
1 2 3
……
140 NULL Priority Level
52
Earliest timer for each priority level
10 20 10 30 20 40 10
[0] [1] [2] [3] min [0,1] min [2, 3] min [0, 3]
53
10 20 10 30 20 40 10
[0] [1] [2] [3] min [0,1] min [2, 3] min [0, 3]
54
Leaf nodes are the earliest timers for each priority level
10 20 10 30 20 40 10
[0] [1] [2] [3] min [0,1] min [2, 3] min [0, 3]
55
Provides an efficient, O(log N) mechanism to find the earliest timer in the priority range [curr_task_prio, max_sys_prio]
10 20 10 30 20 40 10
[0] [1] [2] [3] min [0,1] min [2, 3] min [0, 3]
56
N = number of (fixed) priority levels Constant time
Provides an efficient, O(log N) mechanism to find the earliest timer in the priority range [curr_task_prio, max_sys_prio]
57
Further details in the paper! Open-source implementation at
https://people.mpi-sws.org/~bbb/papers/details/rtas17p/
Timers and the Interference Problem TimerShield Design Evaluation
58
Prototyped in PREEMPT_RT
Intel Core-i5 4 x 3.2Ghz ARM Cortex-A53 4 x 1.2Ghz
59
Prototyped in PREEMPT_RT
Intel Core-i5 4 x 3.2Ghz ARM Cortex-A53 4 x 1.2Ghz
Details in paper
60
How costly are the new queueing data structures? How effective is TimerShield at isolating high-priority tasks from low-priority timer interrupts?
61
How is the context-switch duration affected?
How costly are the new queueing data structures? How effective is TimerShield at isolating high-priority tasks from low-priority timer interrupts?
62
How is the context-switch duration affected?
63
We measured the response time of a high-priority task with varying number of low-priority, timer-using tasks
64
Taskset parameters based on S. Kramer, D. Ziegenbein, and A. Hamann, “Real world automotive benchmark for free,” in WATERS, 2015.
We measured the response time of a high-priority task with varying number of low-priority, timer-using tasks 1 KHz control loop with
computation time
We measured the response time of a high-priority task with varying number of low-priority, timer-using tasks
65
1 KHz control loop with
computation time
Taskset parameters based on S. Kramer, D. Ziegenbein, and A. Hamann, “Real world automotive benchmark for free,” in WATERS, 2015.
cyclictest tasks which periodically call clock_nanosleep()
We measured the response time of a high-priority task with varying number of low-priority, timer-using tasks
66
From 1 to 100 LP cyclictest tasks 1 KHz control loop with
computation time cyclictest tasks which periodically call clock_nanosleep()
Taskset parameters based on S. Kramer, D. Ziegenbein, and A. Hamann, “Real world automotive benchmark for free,” in WATERS, 2015.
67
68
Means 60% of the measured samples have a response time ≤ 214.8us
69
1 LP cyclictest
70
1 LP cyclictest 50 LP cyclictests 100 LP cyclictests
71
1 LP cyclictest 50 LP cyclictests 100 LP cyclictests Long tail, high unpredictability
72
73
Response time with 1, 50 or 100 LP timers remains consistent!
74
Response time with 1, 50 or 100 LP timers remains consistent! Slight shift due to cache effects of increasing number
75
Linux (and POSIX) provide no protection, and specifies no upper limit on timer creation
76
We measured the response time of a high-priority task with a single, unprivileged, user-space task that spawned timers Linux (and POSIX) provide no protection, and specifies no upper limit on timer creation
We measured the response time of a high-priority task with a single, unprivileged, user-space task that spawned timers
77
Using Linux’s timerfd API Linux (and POSIX) provide no protection, and specifies no upper limit on timer creation
78
Idle system 100 LP timers 1000 LP timers
79
Nearly 45us (22%) response- time increase with 1000 low-priority timers
80
1000 LP timers 100 LP timers Idle System
81
1000 LP timers 100 LP timers Idle System
TimerShield protects high-priority task response times from low-priority timer interrupts!
How costly are the new queueing data structures? How effective is TimerShield at isolating high-priority tasks from low-priority timer interrupts?
82
How is the context-switch duration affected?
83
Note: Results for a scenario without a timer-heavy load can be found in the paper.
During context-switches, TimerShield processes expired timers, performs a RMQ, and optionally reprograms hardware
84
We measured the total additional time incurred by TimerShield during context-switches in a timer-heavy scenario
Note: Results for a scenario without a timer-heavy load can be found in the paper.
During context-switches, TimerShield processes expired timers, performs a RMQ, and optionally reprograms hardware
1 high-priority and 50 low-priority timer-using tasks of the same priority
85
We measured the total additional time incurred by TimerShield during context-switches in a timer-heavy scenario
Note: Results for a scenario without a timer-heavy load can be found in the paper.
During context-switches, TimerShield processes expired timers, performs a RMQ, and optionally reprograms hardware
86
1 2 3 4 5 6 7 8 9 Mean Median 99.9th percentile Max microseconds (us)
87
1 2 3 4 5 6 7 8 9 Mean Median 99.9th percentile Max microseconds (us)
Mean and median delay (typical case) is much less than a microsecond
88
1 2 3 4 5 6 7 8 9 Mean Median 99.9th percentile Max microseconds (us)
These reflect batch processing of multiple timers that were deferred
89
2 4 6 8 10 12 14 16 Timer Processing Delay microseconds (us) Hrtimers TimerShield
We measured the worst- case increase in HP task response time under hrtimers with the same experimental setup
90
2 4 6 8 10 12 14 16 Timer Processing Delay microseconds (us) Hrtimers TimerShield
hrtimers takes longer due to the repetitive switches to interrupt context!
91
2 4 6 8 10 12 14 16 Timer Processing Delay microseconds (us) Hrtimers TimerShield
Context-switch delay due to TimerShield is small, and its batch processing of timers is faster than hrtimers
How costly are the new queueing data structures? How effective is TimerShield at isolating high-priority tasks from low-priority timer interrupts?
92
How is the context-switch duration affected?
93
Note: Results for a scenario with a timer-heavy load can be found in the paper.
The worst case for TimerShield’s data-structures is with a single timer, because each operation modifies the segment tree
94
Note: Results for a scenario with a timer-heavy load can be found in the paper.
We measured the timer enqueue and dequeue cost on both subsystems for this setup The worst case for TimerShield’s data-structures is with a single timer, because each operation modifies the segment tree
95
Lower is Better
0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 Mean Median 99.9th percentile Max microseconds (us) Hrtimers TimerShield
96
0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 Mean Median 99.9th percentile Max microseconds (us) Hrtimers TimerShield
Performs negligibly worse on average Favourable towards the max, but the difference is miniscule Lower is Better
97
0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 Mean Median 99.9th percentile Max microseconds (us) Hrtimers TimerShield
Both subsystems have very similar dequeue costs Lower is Better
Impossible for high-priority tasks to be interrupted by low-priority timers under TimerShield
98
Note: Further experiments, including results for ARM, can be found in the paper.
99
Note: Further experiments, including results for ARM, can be found in the paper.
Additional context-switch delay is small, and batch timer processing is faster with TimerShield Impossible for high-priority tasks to be interrupted by low-priority timers under TimerShield
TimerShield’s data structure costs are comparable to hrtimers
100
Note: Further experiments, including results for ARM, can be found in the paper.
Additional context-switch delay is small, and batch timer processing is faster with TimerShield Impossible for high-priority tasks to be interrupted by low-priority timers under TimerShield
Implementation currently assumes unchanging timer priorities
101
Implementation currently assumes unchanging timer priorities
102
Real-time locking protocols, or users, may change task priorities
Implementation currently assumes unchanging timer priorities
103
Real-time locking protocols, or users, may change task priorities
Works implicitly for the immediate priority ceiling protocol Implicitly works if priority is changed with no pending timers
Implementation currently assumes unchanging timer priorities
104
Real-time locking protocols, or users, may change task priorities
Works implicitly for the immediate priority ceiling protocol Implicitly works if priority is changed with no pending timers Can be easily extended to deal with dynamic priorities
Support for Earliest Deadline First (EDF) schedulers Applying similar techniques to other, multiplexed interrupt sources such as network packet interrupts
105
106
Low-priority timer interrupts have a significant negative impact on high-priority task execution
107
FP scheduling on uniprocessor/partitioned multiprocessors
Low-priority timer interrupts have a significant negative impact on high-priority task execution
108
Existing high-resolution timer subsystems, such as Linux hrtimers, are not priority aware
FP scheduling on uniprocessor/partitioned multiprocessors
Low-priority timer interrupts have a significant negative impact on high-priority task execution Existing high-resolution timer subsystems, such as Linux hrtimers, are not priority aware TimerShield completely avoids low-priority timer interrupt interference with small overheads
109
FP scheduling on uniprocessor/partitioned multiprocessors
110
Source Code https://people.mpi-sws.org/~bbb/papers/details/rtas17p/
111
LP
112
Not deferring the wakeup of a low-priority task might allow it to execute on a different, possibly idle CPU HP HP LP
2 4 6 8 10 12 2 4 6 8 10 12
CPU 1 CPU 2
10 20 10 30 20 40 10
[0] [1] [2] [3] min [0,1] min [2, 3] min [0, 3] Leaf nodes correspond to the earliest timer
red-black tree Priority Level
113
10 20 10 30 20 40 10
[0] [1] [2] [3] min [0,1] min [2, 3] Parent nodes store the minimum of their child nodes, and depict the earliest timer for the resulting priority range min [0, 3] Priority Range
114
Increase in text segment 2 KiB Increase in data segment 35 KiB per core
115
How big is TimerShield code, and what are it’s memory requirements?
0.2 0.4 0.6 0.8 1 1.2 Mean Median 99.9th percentile Max microseconds (us) Hrtimers TimerShield
116
Lower is Better
117
0.1 0.2 0.3 0.4 0.5 0.6 Mean Median 99.9th percentile Max microseconds (us) Hrtimers TimerShield
Lower is Better
118
20 40 60 80 100 120 140 Hrtimers TimerShield Requests/ms
With 1000 background LP timers
Idle Throughput: 7044.4 requests/ms
Lower is Better