Final Demos on Thursday! 1. Robotic Arm Controller for Diagnostics - - PowerPoint PPT Presentation

Final Demos on Thursday!

• 1. Robotic Arm Controller for Diagnostics
• 2. PiCar
• 3. Facial Recognition Based Attendance
• 4. Interactive Sign Language Training System
• 5. WashU Course Buddy IoT
• 6. Smart “Soft Real-Time” Security Camera
• 7. Automated Garage Door System

Guidelines: https://www.cse.wustl.edu/~lu/cse520s/slides/project-guidelines.pdf

1

Class in the Fall

Ø CSE 521S Wireless Sensor Networks

q More properly: Internet of Things

Ø https://www.cse.wustl.edu/~lu/cse521s/

2

Adaptive QoS Control for Real-Time Systems

Chenyang Lu

CSE 520S

Challenges

Ø Classical real-time scheduling theory relies on accurate knowledge about workload and platform. Ø New challenges under uncertainties

q Maintain robust real-time properties in face of

• unknown and varying workload
• system failure
• platform upgrade

q Tuning, testing and certification of adaptive real-time systems

4

Challenge 1: Workload Uncertainties

Ø Task execution times

q Heavily influenced by sensor data or user input q Unknown and time-varying

Ø Disturbances

q Aperiodic events q Resource contention from subsystems q Denial of Service attacks

Ø Examples: power grid management, autonomous vehicles.

5

Challenge 2: System Failure

Ø Only maintaining functional reliability is not sufficient. Must also maintain robust real-time properties!

6

1. Norbert fails. 2. Move its tasks to other processors. hermione & harry are

• verloaded!

Challenge 3: System Upgrade

Ø Goal: Portable application across HW/OS platforms

q Same application “works” on multiple platforms

Ø Existing real-time middleware

ü Support functional portability û Lack QoS portability: must manually reconfigure applications on

different platforms to achieve desired QoS

• Profile execution times
• Determine/implement allocation and task rate
• Test/analyze schedulability

Time-consuming and expensive!

7

Example: nORB Middleware

8

Timer thread Worker thread

Server

… … Conn. thread … … … … … … Priority queues

nORB* Application

CORBA Objects Client

T1: 2 Hz T2: 12 Hz

Ma Manually set

• f
• ffline

Conn. thread Operation Request Lanes

Challenge 4: Certification

Ø Uncertainties call for adaptive solutions. But… Ø Adaptation can make things worse. Ø Adaptive systems are difficult to test and certify.

9

0.2 0.4 0.6 0.8 1 100 200 300 Time (sampling period) CPU utilization P1 P2 Set Point

An unstable adaptive system

Adaptive QoS Control

Ø Develop software feedback control in middleware

q Achieve robust real-time properties for many applications

Ø Apply control theory to design and analyze control algorithms

q Facilitate certification of embedded software

10

Available resources? HW failure? Drivers/OS/HW? Applications Sensor/human input? Disturbance? Adaptive QoS Control Middleware

Maintain QoS guarantees

• w/

w/o accurate knowledge about workload/platform

• w/

w/o hand tuning

Adaptive QoS Control Middleware

Ø FCS/nORB: Single server control Ø FC-ORB: Distributed systems with end-to-end tasks

11

Feedback Control Real-Time Scheduling

Ø Developers specify

q Performance specs

• CPU utilization = 70%; Deadline miss ratio = 1%.

q Tunable parameters

• Range of task rate: digital control loop, video/data display
• Quality levels: image quality, filters
• Admission control

Ø Guarantee specs by tuning parameters based on feedbacks

q Automatic: No need for hand tuning q Transparent from developers q Performance Portability!

12

A Feedback Control Loop

13

FC-U

Monitor

HW? Drivers/OS? Application?

Sensors, Inputs Specs

Us = 70%

Parameters

R1: [1, 5] Hz R2: [10, 20] Hz Middleware

Actuator Controller

U(k) {Ri(k+1)}

The FC-U Algorithm

Us: utilization reference Ku: control parameter Ri(0): initial rate 1. Get utilization U(k) from Utilization Monitor. 2. Utilization Controller: B(k+1) = B(k)+ Ku*(Us–U(k)) /* Integral Controller */ 3. Rate Actuator adjusts task rates Ri(k+1) = (B(k+1)/B(0))Ri(0) 4. Inform clients of new task rates.

14

The Family of FCS Algorithms

Ø FC-U controls utilization

q Performance spec: U(k) = Us ü Meet all deadlines if Us £ schedulable utilization bound û Relatively low utilization if utilization bound is pessimistic

Ø FC-M controls miss ratio

q Performance spec: M(k) = Ms ü High utilization ü Does not require utilization bound to be known a priori û Small but non-zero deadline miss ratio: M(k) > 0

Ø FC-UM combines FC-U and FC-M

q Performance specs: Us, Ms ü Allow higher utilization than FC-U ü No deadline misses in “nominal” case q Performance bounded by FC-M

15

16

Feedback Control Loop

Actuator Monitor Reference Controlled variable

+ -

error sample Manipulated variable change

Computing System Software Feedback Control Loop

Controller control input

Dynamic Response

17

Settling time

Controlled variable Time Reference

Steady State Transient State Steady state error Stability

Control Analysis

Ø Rigorously designed based on feedback control theory Ø Analytic guarantees on

q Stability q Steady state performance q Transient state: settling time and overshoot q Robustness against variation in execution time

Ø Do not assume accurate knowledge of execution time

18

FCS/nORB Architecture

19

Timer thread worker thread conn. thread

Server

… …

util monitor miss monitor controller rate assigner conn. thread

… …

rate modulator

feedback lane … … … Operation Request Lanes …

Priority Queues

FCS/nORB Application

CORBA Objects Client

Implementation

Ø Running on top of COTS Linux Ø Deadline Miss Monitor

q Instrument operation request lanes q Time-stamp operation request and response on each lane

Ø CPU Utilization Monitor

q Interface with Linux /proc/stat file q Count idle time: “Coarse” granularity at jiffy (10 ms)

Ø Only controls server delay

20

Offline or Online?

Ø Offline

q FCS executed in testing phase on a new platform q Turned off after entering steady state ü No run-time overhead û Cannot deal with varying workload

Ø Online

û Run-time overhead (actually small…) ü Robustness in face of changing execution times

21

Set-up

Ø OS: Redhat Linux Ø Hardware platform

q Server A: 1.8GHz Celeron, 512 MB RAM q Server B: 1.99GHz Pentium 4, 256 MB RAM q Same client q Connected via 100 Mbps LAN

Ø Experiment

q Overhead q Steady execution time (offline case) q Varying execution time (on-line case)

22

Server Overhead

• Overhead: FC-UM > FC-M > FC-U
• FC-UM increases CPU utilization by <1% for a 4s sampling period.

23

Sampling Period = 4 sec

Server Overhead per Sampling Period

5 10 15 20 25 30 35 40 FC-U FC-M FC-UM Overhead (ms)

Performance Portability

Steady Execution Time

24

0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 25 50 75 100 125 150 175 200 Time (4 sec) U(k) B(k) M(k)

FC-U on Server A 1.8GHz Celeron, 512 MB RAM

0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 25 50 75 100 125 150 175 200 Time (4 sec) U(k) B(k) M(k)

• Same CPU utilization (and no deadline miss) on different

platforms w/o hand-tuning!

FC-U on Server B 1.99GHz Pentium 4, 256 MB RAM

Us = = 70% 70%

Steady-state Deadline Miss Ratio

1.49 0.00 0.50 1.00 1.50 2.00

FC-U FC-M FC-UM %

Average Deadline Miss Ratio in Steady State

25

Ms = 1. 1.5%

• FC-M enforces miss ratio spec
• FC-U, FC-UM causes no deadline misses

Steady-State CPU Utilization

Average CPU Utilization in Steady State

70.01 98.93 74.97 20 40 60 80 100

FC-U FC-M FC-UM %

26

Us = = 70% 70% Us = = 75% 75%

• FC-U, FC-UM enforces utilization spec
• FC-M achieves higher utilization

Robust Guarantees

Varying Execution Time

27

0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 50 100 150 200 250 300 350 400 Time (4 sec) U(k) B(k) M(k)

Same CPU utilization and no deadline miss in steady state despite changes in execution times!

Tolerance to Load Increase

Ø Surprise

q Server crashes under FC-M when execution time increases q FCS/nORB threads run at real-time priority q Kernel starvation when CPU utilization reaches 100%

Ø Tolerance margin of load increase

q FC-U, FC-UM: margin = 1/Us-1

• Us=70% à can tolerate (1/0.7-1)=43% increase in execution time

q FC-M: small and “unknown” margin

• Unsuitable when execution time can increase unexpectedly

28

Summary of Experimental Results

Ø FCS algorithms enforces specified CPU utilization or miss ratio in steady state

q Experimental validation of control design and analysis of FCS

Ø Performance portability: FCS/nORB achieves the same performance guarantee when

q platform changes q execution time changes (within tolerance margin)

Ø Overhead acceptable à FCS can be used online

29

Summary: FCS/nORB

Ø Enable robust, performance-portable real-time software Ø Program application once à runs on multiple platforms with robust performance guarantees!

30

References

Ø C. Lu, J.A. Stankovic, G. Tao, and S.H. Son, Feedback Control Real-Time Scheduling: Framework, Modeling, and Algorithms, Real-Time Systems, Special Issue on Control-theoretical Approaches to Real-Time Computing, 23(1/2): 85-126, July/September 2002. Ø C. Lu, X. Wang and C.D. Gill, Feedback Control Real-Time Scheduling in ORB Middleware, IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS), May 2003. Critique

31

Adaptive QoS Control Middleware

Ø FCS/nORB: Single server control Ø FC-ORB: Distributed systems with end-to-end tasks

q Handle end-to-end tasks q Fault tolerance

32

End-to-End Task Model

Ø Periodic task Ti = chain of subtasks {Tij} on different processors

q All subtasks run at a same rate q End-to-end deadline

Ø Task rate can be adjusted within a range

q Trade-off between video quality and rate q Higher rate à better video quality & higher CPU utilization

33

T1 T2 T3 T11 T12 T13 P1 P2 P3

Precedence Constraints Subtask

End-to-End Utilization Control

Ø CPU utilization

q Too high à system overload à crash q Too low à poor performance (e.g. poor video quality) q Utilization < schedulable bound à meet deadlines

Ø Uncertainties: varying task execution times

q Adjust task rates to compensate for variations

34

T1 T2 T3 T11 T12 T13 P1 P2 P3

Precedence Constraints Subtask

Challenges

Ø Multi-Input-Multi-Output (MIMO) control Ø Utilizations are coupled due to end-to-end tasks

q Rate change affects all processors in the task chain

Ø Constraints on task rates Ø Stability assurance

35 50% 80% 50% 80% 30% 60% 60%

Controller CPU utilization

T1 T2 T3 T11 T12 T13 P1 P2 P3

EUCON – End-to-end Utilization CONtrol

36

Model Predictive Controller

Utilization Monitor Rate Modulator RM UM UM RM

Manipulated variables: Task rate changes Controlled Variables: CPU utilizations

• Centralized control
• Designed based on Model Predictive Control (MPC) theory
• Invoked periodically to control the utilizations of all processors

ú ú ú û ù ê ê ê ë é ) ( ) (

1

k u k u

n

! ú ú ú û ù ê ê ê ë é D D ) ( ) (

1

k r k r

m

!

ú ú ú û ù ê ê ê ë é

m m R

R R R

max, 1 max, min, 1 min,

! !

Allowed rate range for tasks Desired utilization bounds (constraints)

ú ú ú û ù ê ê ê ë é

n

B B !

1

Control Theoretic Methodology

1. Model the controlled system 2. Design a controller 3. Analyze stability

37

ui(k) = ui(k −1) + gi c jlΔrj(k −1)

T jl ∈Si

∑

Dynamic Model: One Processor

Ø Si: set of subtasks on Pi Ø cjl: estimated execution time of Til running on Pi

q may not be correct

Ø gi: utilization gain of Pi

q ratio between actual and estimated change in utilization q unknown: models uncertainty in execution times

38

• G: diagonal matrix of utilization gains
• F: subtask allocation matrix

– models the coupling among processors – fij = cjl if task Tj has a subtask Tjl on processor Pi – fij = 0 if Tj has no subtask on Pi

Dynamic Model: Multiple Processors

ú û ù ê ë é =

31 22 21 11

c c c c F

39

u(k) = u(k-1) + GFDr(k-1)

T1 T2 T11

P1 P2

T21 T22 T3 T31

Model Predictive Control

At a sampling instant Ø Compute inputs for several future sampling periods Dr(k), Dr(k+1), ... Dr(k+M-1) to minimize a cost function in the future Ø Cost in the future is predicted using i) feedback u(k-1) ii) approximate dynamic model Ø Apply Dr(k) to the system At the next sampling instant: Ø Shift time and re-compute Dr(k+1), Dr(k+2), ... Dr(k+M) based on feedback u(k)

40

Model Predictive Controller in EUCON

41

Least Squares Solver

ú ú ú û ù ê ê ê ë é ) ( ) (

1

k u k u

n

! ú ú ú û ù ê ê ê ë é D D ) ( ) (

1

k r k r

m

!

Model Predictive Controller System Model Cost Function Reference Trajectory

ú ú ú û ù ê ê ê ë é

n

B B !

1

Rate Constraints Least Squares Solver

ú ú ú û ù ê ê ê ë é ) ( ) (

1

k u k u

n

! ú ú ú û ù ê ê ê ë é D D ) ( ) (

1

k r k r

m

!

Model Predictive Controller System Model Cost Function Reference Trajectory

ú ú ú û ù ê ê ê ë é

n

B B !

1

Rate Constraints

ú ú ú û ù ê ê ê ë é + D + D ) 1 ( ) 1 (

1

k r k r

m

!

Difference with reference trajectory Desired trajectory for u(k) to converge to B Constrained

• ptimization

solver

Stability Analysis

Ø Stability: system converges to equilibrium point from any initial condition

q Equilibrium point = utilization set points B q Utilization of all processors à their set points whenever feasible

Ø Derive stability condition in term of G

q Tolerable range of variation in execution times

42

Guarantees on utilization despite uncertainty!

Simulation: Stable System

0.2 0.4 0.6 0.8 1

50 100 150 200 250 300

Time (sampling period) CPU utilization P1 P2 Set Point

43

execution time factor = 0.5 (actual execution times = ½ estimates)

Simulation: Unstable System

44

execution time factor = 7 (actual execution times = 7x estimates)

0.2 0.4 0.6 0.8 1 100 200 300 Time (sampling period) CPU utilization P1 P2 Set Point

Stability

Ø Stability condition à tolerable range of execution times Analytical assurance on utilizations despite uncertainty

45

Overestimation

• f execution

times prevents

• scillation

actual execution time / estimation Predicted bound for stability

FC-ORB

Feedback Controlled Object Request Broker

Ø End-to-end utilization control

q Maintains desired utilizations on all processors

Ø End-to-end ORB architecture

q Specialized for rate adaptation

Ø Task migration

q Reliability in terms of both functionality and real-time performance

46

End-to-End Utilization Control Service

47

q Implements EUCON (End-to-end Utilization CONtrol) q Provides functional and performance portability

Remote request lanes

Priority Manager Rate Modulator Model Predictive Controller

Remote request lanes

Utilization Monitor

Controlled variables: Utilizations Manipulated variables: Rate changes

Priority Manager Rate Modulator Utilization Monitor Priority Manager Rate Modulator Utilization Monitor

End-to-End Object Request Broker

Ø Release guard for end-to-end tasks Ø Priority management

q Rate adaptation à continuous priority changes û Thread-per-priority à high overhead ü Thread-per-subtask: change priority only when the order of task

rates changes

48

Remote request lanes

Priority Manager Rate Modulator

Remote request lanes

Utilization Monitor Priority Manager Rate Modulator Utilization Monitor Priority Manager Rate Modulator Utilization Monitor

Task Migration

Ø Fault model: permanent processor failure Ø Subtasks have backups on different processors Ø Utilization control + fault tolerance

q Automatic controller reconfiguration q Handle overload caused by task migration

49

Remote request lanes

Priority Manager Rate Modulator Model Predictive Controller

Remote request lanes

Utilization Monitor

ú ú ú û ù ê ê ê ë é ) ( ) ( ) (

3 2 1

k u k u k u

Utilizations

ú û ù ê ë é D D ) ( ) (

2 1

k r k r

Rate changes

Priority Manager Rate Modulator Utilization Monitor Priority Manager Rate Modulator Utilization Monitor

ú û ù ê ë é ) ( ) (

2 1

k u k u

FC-ORB Implementation

Ø Implemented based on FCS/nORB, nORB and ACE Ø Specialized for memory constrained distributed real-time systems Ø 7017 lines of C++ code Ø Controller is implemented as a Dynamic Link Library (DLL) generated by MATLAB

50

Experimental Setup

Ø 12 tasks (25 subtasks) and 4 Pentium IV processors Ø KURT Linux 2.4.22 Ø Rate Monotonic Scheduling Ø Subtasks on Norbert have backups on other processors

51

Goal 1: Robust Utilization Control

52

Execution times change at runtime Disturbance from external resource contention

Desired utilization: 73% (0.73)

Goal 2: Performance Portability

0.2 0.4 0.6 0.8 1 200 400 600 800 1000 1200 1400 1600 Time (sec) CPU utilization ron harry norbert hermione

53

Same utilization: portable performance on systems with different capacities

0.2 0.4 0.6 0.8 1 200 400 600 800 1000 1200 1400 1600 Time (sec) CPU utilization ron harry norbert hermione

exec time = 2x expected (running on slow machines) exec times = expected/4 (running on fast machines) Desired utilization: 73% (0.73)

Goal 3: Fault Tolerance

54

1. Norbert fails. 2. move its tasks to other processors. 3. reconfigure controller 4. control utilization by adjusting task rates

T1 T2 T11 T12 P1 P2 Norbert

73% 73%

T13 T3

10 100% % !!

73%

Summary: FC-ORB

Ø Robust utilization control, despite

q

unknown or varying execution times

q

external disturbances

Ø Performance portability Ø Fault tolerance, in terms of

q

functionality

q

real-time performance

55

Conclusion: Adaptive QoS Control

Ø Feedback control à robust real-time performance under uncertainty Ø Middleware: provides reusable control services to real-time applications Ø Control analysis: tuning and certification of adaptive software Ø More

q Advanced control: event-driven, discrete configurations. q Coordination of multiple control policies q Sophisticated fault tolerance techniques q Certification/testing methodologies

56

Reading

Ø Control of a single server

q FCS/nORB: Feedback Control Real-Time Scheduling in ORB Middleware,

RTAS’03.

q FCS: Feedback Control Real-Time Scheduling: Framework, Modeling, and

Algorithms, Real-Time Systems, 2002.

Ø Centralized control of distributed systems

q FC-ORB: Enhancing the Robustness of Distributed Real-Time Middleware via

End-to-End Utilization Control, RTSS’05.

q EUCON: Feedback Utilization Control in Distributed Real-Time Systems with

End-to-End Tasks, RTSS’05, IEEE TPDS.

57