On Self-adaptive Resource Allocation through Reinforcement Learning - - PowerPoint PPT Presentation
On Self-adaptive Resource Allocation through Reinforcement Learning Jacopo Panerati , Filippo Sironi , Matteo Carminati , Martina Maggio , Giovanni Beltrame , Piotr J. Gmytrasiewicz , Donatella Sciuto and Marco D.
Jacopo Panerati†, Filippo Sironi‡, Matteo Carminati‡, Martina Maggio§, Giovanni Beltrame†, Piotr J. Gmytrasiewicz¶, Donatella Sciuto‡ and Marco D. Santambrogio‡
†Polytechnique Montr´
eal, ‡Politecnico Milano, §Lund University, ¶University of Illinois Chicago
Politecnico di Torino - Turin, 25 June 2013
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Rationale
Methodology
(1) Reinforcement Learning (RL).
Objective
(2) Self-adaptive Computing.
Research Question
Is RL a suitable approach for self-adaptive computing?
2/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Rationale
Methodology
(1) Reinforcement Learning (RL).
Objective
(2) Self-adaptive Computing.
Research Question
Is RL a suitable approach for self-adaptive computing?
2/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Rationale
Methodology
(1) Reinforcement Learning (RL).
Objective
(2) Self-adaptive Computing.
Research Question
Is RL a suitable approach for self-adaptive computing?
2/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Rationale
Methodology
(1) Reinforcement Learning (RL).
Objective
(2) Self-adaptive Computing.
Research Question
Is RL a suitable approach for self-adaptive computing?
2/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Rationale
Methodology
(1) Reinforcement Learning (RL).
Objective
(2) Self-adaptive Computing.
Research Question
Is RL a suitable approach for self-adaptive computing?
2/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Rationale
Methodology
(1) Reinforcement Learning (RL).
Objective
(2) Self-adaptive Computing.
Research Question
Is RL a suitable approach for self-adaptive computing?
2/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
A Typical Machine Learning Problem
Generic (Informal) Steps
3/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
A Typical Machine Learning Problem
Generic (Informal) Steps
3/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
A Typical Machine Learning Problem
Generic (Informal) Steps
3/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
A Typical Machine Learning Problem
Generic (Informal) Steps
3/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
A Typical Machine Learning Problem
Generic (Informal) Steps
3/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Machine Learning Methodologies
Supervised Learning
Classification Algorithms
when labels are known to belong to a finite set C
Regression Algorithms
when labels are known to belong to R Unsupervised Learning
Clustering Algorithms
when labels are unknown but their cardinality K is assumed be fixed
4/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Machine Learning Methodologies
Supervised Learning
Classification Algorithms
when labels are known to belong to a finite set C
Regression Algorithms
when labels are known to belong to R Unsupervised Learning
Clustering Algorithms
when labels are unknown but their cardinality K is assumed be fixed
4/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Machine Learning Methodologies
Supervised Learning
Classification Algorithms
when labels are known to belong to a finite set C
Regression Algorithms
when labels are known to belong to R Unsupervised Learning
Clustering Algorithms
when labels are unknown but their cardinality K is assumed be fixed
4/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Example of Classification Problem
Hand-Writing
Recognition of hand-written digits is a typical classifcation problem. Data-points are matrices of pixels (∈ Rd) and the label set C is {0,1,2,..,9}.
5/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Example of Classification Problem
Hand-Writing
Recognition of hand-written digits is a typical classifcation problem. Data-points are matrices of pixels (∈ Rd) and the label set C is {0,1,2,..,9}.
5/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Machine Learning Methodologies
Supervised Learning
Classification Algorithms
when labels are known to belong to a finite set C
Regression Algorithms
when labels are known to belong to R Unsupervised Learning
Clustering Algorithms
when labels are unknown but their cardinality K is assumed be fixed
6/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Machine Learning Methodologies
Supervised Learning
Classification Algorithms
when labels are known to belong to a finite set C
Regression Algorithms
when labels are known to belong to R Unsupervised Learning
Clustering Algorithms
when labels are unknown but their cardinality K is assumed be fixed
6/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Machine Learning Methodologies
Supervised Learning
Classification Algorithms
when labels are known to belong to a finite set C
Regression Algorithms
when labels are known to belong to R Unsupervised Learning
Clustering Algorithms
when labels are unknown but their cardinality K is assumed be fixed
6/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Machine Learning Methodologies
Supervised Learning
Classification Algorithms
when labels are known to belong to a finite set C
Regression Algorithms
when labels are known to belong to R Unsupervised Learning
Clustering Algorithms
when labels are unknown but their cardinality K is assumed be fixed
6/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Example of Clustering Problem
Space Exploration
Clustering algorithms can be used to identify patterns in remotely (e.g. in space) sensed data and improve the scientific return by sending to the ground station only statistically significant data [1].
1
1http://nssdc.gsfc.nasa.gov/
7/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Example of Clustering Problem
Space Exploration
Clustering algorithms can be used to identify patterns in remotely (e.g. in space) sensed data and improve the scientific return by sending to the ground station only statistically significant data [1].
1
1http://nssdc.gsfc.nasa.gov/
7/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Reinforcements in Behavioural Psychology
Definition
In behavioural psychology, reinforcement consists of the strengthening of a behaviour associated to a stimulus through its repetition.
Pioneers
B.F. Skinner (1904-1990), together with E. Thorndike (1874-1949), is considered to be
8/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Reinforcements in Behavioural Psychology
Definition
In behavioural psychology, reinforcement consists of the strengthening of a behaviour associated to a stimulus through its repetition.
Pioneers
B.F. Skinner (1904-1990), together with E. Thorndike (1874-1949), is considered to be
8/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Reinforcements in Behavioural Psychology
Definition
In behavioural psychology, reinforcement consists of the strengthening of a behaviour associated to a stimulus through its repetition.
Pioneers
B.F. Skinner (1904-1990), together with E. Thorndike (1874-1949), is considered to be
8/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Reinforcements in Behavioural Psychology
Definition
In behavioural psychology, reinforcement consists of the strengthening of a behaviour associated to a stimulus through its repetition.
Pioneers
B.F. Skinner (1904-1990), together with E. Thorndike (1874-1949), is considered to be
8/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Reinforcements in Behavioural Psychology
Definition
In behavioural psychology, reinforcement consists of the strengthening of a behaviour associated to a stimulus through its repetition.
Pioneers
B.F. Skinner (1904-1990), together with E. Thorndike (1874-1949), is considered to be
8/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Reinforcements in Behavioural Psychology
Definition
In behavioural psychology, reinforcement consists of the strengthening of a behaviour associated to a stimulus through its repetition.
Pioneers
B.F. Skinner (1904-1990), together with E. Thorndike (1874-1949), is considered to be
8/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Reinforcements in Behavioural Psychology
Definition
In behavioural psychology, reinforcement consists of the strengthening of a behaviour associated to a stimulus through its repetition.
Pioneers
B.F. Skinner (1904-1990), together with E. Thorndike (1874-1949), is considered to be
8/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Reinforcements in Behavioural Psychology
Definition
In behavioural psychology, reinforcement consists of the strengthening of a behaviour associated to a stimulus through its repetition.
Pioneers
B.F. Skinner (1904-1990), together with E. Thorndike (1874-1949), is considered to be
8/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Pavlov’s Dog
A precursor of Skinner theories
Ivan Pavlov (1849-1936) made conditioning famous with his experiment of drooling dogs.
9/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Pavlov’s Dog
A precursor of Skinner theories
Ivan Pavlov (1849-1936) made conditioning famous with his experiment of drooling dogs.
9/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
reinforcement learning in computer science is something a bit different both from supervised/unsupervised learning and reinforcements in behavioural psychology..
10/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (I)
Supervised/Unsupervised Machine Learning
data-point → label (or a cluster)
Reinforcements in Behavioural Psychology
stimulus → behaviour
Reinforcement Learning
state of the world → action
11/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (I)
Supervised/Unsupervised Machine Learning
data-point → label (or a cluster)
Reinforcements in Behavioural Psychology
stimulus → behaviour
Reinforcement Learning
state of the world → action
11/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (I)
Supervised/Unsupervised Machine Learning
data-point → label (or a cluster)
Reinforcements in Behavioural Psychology
stimulus → behaviour
Reinforcement Learning
state of the world → action
11/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (I)
Supervised/Unsupervised Machine Learning
data-point → label (or a cluster)
Reinforcements in Behavioural Psychology
stimulus → behaviour
Reinforcement Learning
state of the world → action
11/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (I)
Supervised/Unsupervised Machine Learning
data-point → label (or a cluster)
Reinforcements in Behavioural Psychology
stimulus → behaviour
Reinforcement Learning
state of the world → action
11/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (I)
Supervised/Unsupervised Machine Learning
data-point → label (or a cluster)
Reinforcements in Behavioural Psychology
stimulus → behaviour
Reinforcement Learning
state of the world → action
11/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (I)
Supervised/Unsupervised Machine Learning
data-point → label (or a cluster)
Reinforcements in Behavioural Psychology
stimulus → behaviour
Reinforcement Learning
state of the world → action
11/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (I)
Supervised/Unsupervised Machine Learning
data-point → label (or a cluster)
Reinforcements in Behavioural Psychology
stimulus → behaviour
Reinforcement Learning
state of the world → action
11/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (I)
Supervised/Unsupervised Machine Learning
data-point → label (or a cluster)
Reinforcements in Behavioural Psychology
stimulus → behaviour
Reinforcement Learning
state of the world → action
11/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (I)
Supervised/Unsupervised Machine Learning
data-point → label (or a cluster)
Reinforcements in Behavioural Psychology
stimulus → behaviour
Reinforcement Learning
state of the world → action
11/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (II)
Reinforcement Learning
state of the world → action → new state of the world → action → ..
Because the performance metric of RL (i.e., the collected rewardS)
is computed over time, solving a RL problem allows to make
12/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (II)
Reinforcement Learning
state of the world → action → new state of the world → action → ..
Because the performance metric of RL (i.e., the collected rewardS)
is computed over time, solving a RL problem allows to make
12/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (II)
Reinforcement Learning
state of the world → action → new state of the world → action → ..
Because the performance metric of RL (i.e., the collected rewardS)
is computed over time, solving a RL problem allows to make
12/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (II)
Reinforcement Learning
state of the world → action → new state of the world → action → ..
Because the performance metric of RL (i.e., the collected rewardS)
is computed over time, solving a RL problem allows to make
12/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (II)
Reinforcement Learning
state of the world → action → new state of the world → action → ..
Because the performance metric of RL (i.e., the collected rewardS)
is computed over time, solving a RL problem allows to make
12/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (II)
Reinforcement Learning
state of the world → action → new state of the world → action → ..
Because the performance metric of RL (i.e., the collected rewardS)
is computed over time, solving a RL problem allows to make
12/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (II)
Reinforcement Learning
state of the world → action → new state of the world → action → ..
Because the performance metric of RL (i.e., the collected rewardS)
is computed over time, solving a RL problem allows to make
12/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (III)
If today was a sunny day
fail your exams later in the summer” RL is not an epicurean carpe diem methodology, but a more farsighted and judicious approach. The point is, not how long you live, but how nobly you live.
13/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (III)
If today was a sunny day
fail your exams later in the summer” RL is not an epicurean carpe diem methodology, but a more farsighted and judicious approach. The point is, not how long you live, but how nobly you live.
13/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (III)
If today was a sunny day
fail your exams later in the summer” RL is not an epicurean carpe diem methodology, but a more farsighted and judicious approach. The point is, not how long you live, but how nobly you live.
13/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (III)
If today was a sunny day
fail your exams later in the summer” RL is not an epicurean carpe diem methodology, but a more farsighted and judicious approach. The point is, not how long you live, but how nobly you live.
13/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (III)
If today was a sunny day
fail your exams later in the summer” RL is not an epicurean carpe diem methodology, but a more farsighted and judicious approach. The point is, not how long you live, but how nobly you live.
13/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Why Reinforcement Learning is Different (III)
If today was a sunny day
fail your exams later in the summer” RL is not an epicurean carpe diem methodology, but a more farsighted and judicious approach. The point is, not how long you live, but how nobly you live.
13/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
moving on to self-adaptive computing..
14/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Typical Properties of Self-adaptive Computing
Self-configuration
The system requires limited or no human intervention in order to set-up.
Self-optimization
The system is able to achieve user-defined goals autonomously, without human interaction.
Self-healing
The system can detect and recover from faults without human intervention. Together with self-protection, these are the properties identified in [3] for autonomic system.
15/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Typical Properties of Self-adaptive Computing
Self-configuration
The system requires limited or no human intervention in order to set-up.
Self-optimization
The system is able to achieve user-defined goals autonomously, without human interaction.
Self-healing
The system can detect and recover from faults without human intervention. Together with self-protection, these are the properties identified in [3] for autonomic system.
15/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-configuration Example
Multi-platform software
Software that is able to run on different hardware configurations seamlessly is a good example of self-configuration. Hardware Inst.Tools Software
Detect Config. Install Run
16/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-configuration Example
Multi-platform software
Software that is able to run on different hardware configurations seamlessly is a good example of self-configuration. Hardware Inst.Tools Software
Detect Config. Install Run
16/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Typical Properties of Self-adaptive Computing
Self-configuration
The system requires limited or no human intervention in order to set-up.
Self-optimization
The system is able to achieve user-defined goals autonomously, without human interaction.
Self-healing
The system can detect and recover from faults without human intervention. Together with self-protection, these are the properties identified in [3] for autonomic system.
17/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Typical Properties of Self-adaptive Computing
Self-configuration
The system requires limited or no human intervention in order to set-up.
Self-optimization
The system is able to achieve user-defined goals autonomously, without human interaction.
Self-healing
The system can detect and recover from faults without human intervention. Together with self-protection, these are the properties identified in [3] for autonomic system.
17/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-optimization Example
Smart Video Players
Players that can adjust media encoding in order to maintain a certain Quality of Service (QoS) can be considered self-optimizing applications. Video Manager Encoder
Detect Quality Control Play
18/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-optimization Example
Smart Video Players
Players that can adjust media encoding in order to maintain a certain Quality of Service (QoS) can be considered self-optimizing applications. Video Manager Encoder
Detect Quality Control Play
18/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Typical Properties of Self-adaptive Computing
Self-configuration
The system requires limited or no human intervention in order to set-up.
Self-optimization
The system is able to achieve user-defined goals autonomously, without human interaction.
Self-healing
The system can detect and recover from faults without human intervention. Together with self-protection, these are the properties identified in [3] for autonomic system.
19/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Typical Properties of Self-adaptive Computing
Self-configuration
The system requires limited or no human intervention in order to set-up.
Self-optimization
The system is able to achieve user-defined goals autonomously, without human interaction.
Self-healing
The system can detect and recover from faults without human intervention. Together with self-protection, these are the properties identified in [3] for autonomic system.
19/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-healing Example
Reconfigurable Logic
FPGAs are a good playground for self-healing implementation. Part of the hardware resources can be used to verify the correct functioning of the rest of the logic and force reconfiguration when a fault is detected. Prog.Logic Listener µContr.
Detect Fault Inform Reconfigure
20/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-healing Example
Reconfigurable Logic
FPGAs are a good playground for self-healing implementation. Part of the hardware resources can be used to verify the correct functioning of the rest of the logic and force reconfiguration when a fault is detected. Prog.Logic Listener µContr.
Detect Fault Inform Reconfigure
20/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Typical Properties of Self-adaptive Computing
Self-configuration
The system requires limited or no human intervention in order to set-up.
Self-optimization
The system is able to achieve user-defined goals autonomously, without human interaction.
Self-healing
The system can detect and recover from faults without human intervention. Together with self-protection, these are the properties identified in [3] for autonomic system.
21/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Typical Properties of Self-adaptive Computing
Self-configuration
The system requires limited or no human intervention in order to set-up.
Self-optimization
The system is able to achieve user-defined goals autonomously, without human interaction.
Self-healing
The system can detect and recover from faults without human intervention. Together with self-protection, these are the properties identified in [3] for autonomic system.
21/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Research Question
Is RL a suitable approach for self-adaptive computing?
22/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Case Study
Testing Environment
Objective of our Experiments
Enabling self-adaptive properties in applications of the PARSEC[4] benchmark suite through reinforcement learning algorithms.
23/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Case Study
Testing Environment
Objective of our Experiments
Enabling self-adaptive properties in applications of the PARSEC[4] benchmark suite through reinforcement learning algorithms.
23/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Case Study
Testing Environment
Objective of our Experiments
Enabling self-adaptive properties in applications of the PARSEC[4] benchmark suite through reinforcement learning algorithms.
23/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Tests Set-Up
Reinforcement Learning Framework
→ heart rate of the PARSEC benchmark application measured through Heart Rate Monitor (HRM) APIs [5]
→ (1) number of cores on which the PARSEC benchmark application is scheduled 2 and (2) CPU frequency 3
→whether a user-defined target (in heartbeats/s) is met or not
2sched setaffinity system call 3cpufrequtils package
24/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Tests Set-Up
Reinforcement Learning Framework
→ heart rate of the PARSEC benchmark application measured through Heart Rate Monitor (HRM) APIs [5]
→ (1) number of cores on which the PARSEC benchmark application is scheduled 2 and (2) CPU frequency 3
→whether a user-defined target (in heartbeats/s) is met or not
2sched setaffinity system call 3cpufrequtils package
24/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Tests Set-Up
Reinforcement Learning Framework
→ heart rate of the PARSEC benchmark application measured through Heart Rate Monitor (HRM) APIs [5]
→ (1) number of cores on which the PARSEC benchmark application is scheduled 2 and (2) CPU frequency 3
→whether a user-defined target (in heartbeats/s) is met or not
2sched setaffinity system call 3cpufrequtils package
24/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Tests Set-Up
Reinforcement Learning Framework
→ heart rate of the PARSEC benchmark application measured through Heart Rate Monitor (HRM) APIs [5]
→ (1) number of cores on which the PARSEC benchmark application is scheduled 2 and (2) CPU frequency 3
→whether a user-defined target (in heartbeats/s) is met or not
2sched setaffinity system call 3cpufrequtils package
24/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Tests Set-Up
Reinforcement Learning Framework
→ heart rate of the PARSEC benchmark application measured through Heart Rate Monitor (HRM) APIs [5]
→ (1) number of cores on which the PARSEC benchmark application is scheduled 2 and (2) CPU frequency 3
→whether a user-defined target (in heartbeats/s) is met or not
2sched setaffinity system call 3cpufrequtils package
24/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Tests Set-Up
Reinforcement Learning Framework
→ heart rate of the PARSEC benchmark application measured through Heart Rate Monitor (HRM) APIs [5]
→ (1) number of cores on which the PARSEC benchmark application is scheduled 2 and (2) CPU frequency 3
→whether a user-defined target (in heartbeats/s) is met or not
2sched setaffinity system call 3cpufrequtils package
24/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Tests Set-Up
Reinforcement Learning Framework
→ heart rate of the PARSEC benchmark application measured through Heart Rate Monitor (HRM) APIs [5]
→ (1) number of cores on which the PARSEC benchmark application is scheduled 2 and (2) CPU frequency 3
→whether a user-defined target (in heartbeats/s) is met or not
2sched setaffinity system call 3cpufrequtils package
24/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Tests Set-Up
Reinforcement Learning Framework
→ heart rate of the PARSEC benchmark application measured through Heart Rate Monitor (HRM) APIs [5]
→ (1) number of cores on which the PARSEC benchmark application is scheduled 2 and (2) CPU frequency 3
→whether a user-defined target (in heartbeats/s) is met or not
2sched setaffinity system call 3cpufrequtils package
24/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-configuration
1 2 3 4 5 6 7 8 9 10
5 10 15 20 25
50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 9501000 1 2 3 4
time(s) cores blackscholes managed exploiting ADP and core allocation.
25/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-configuration
1 2 3 4 5 6 7 8 9 10
5 10 15 20 25
50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 9501000 1 2 3 4
time(s) cores blackscholes managed exploiting ADP and core allocation.
25/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-configuration
1 2 3 4 5 6 7 8 9 10
5 10 15 20 25
50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 9501000 1 2 3 4
time(s) cores blackscholes managed exploiting ADP and core allocation.
25/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-configuration
1 2 3 4 5 6 7 8 9 10
5 10 15 20 25
50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 9501000 1 2 3 4
time(s) cores blackscholes managed exploiting ADP and core allocation .
25/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-configuration
1 2 3 4 5 6 7 8 9 10
5 10 15 20 25
50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 9501000 1 2 3 4
time(s) cores blackscholes managed exploiting ADP and core allocation.
25/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-configuration
1 2 3 4 5 6 7 8 9 10
5 10 15 20 25
50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 9501000 1 2 3 4
time(s) cores blackscholes managed exploiting ADP and core allocation.
25/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-configuration
1 2 3 4 5 6 7 8 9 10
5 10 15 20 25
50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 9501000 1 2 3 4
time(s) cores blackscholes managed exploiting ADP and core allocation.
25/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-configuration
1 2 3 4 5 6 7 8 9 10
5 10 15 20 25
50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 9501000 1 2 3 4
time(s) cores blackscholes managed exploiting ADP and core allocation.
25/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-configuration
1 2 3 4 5 6 7 8 9 10
5 10 15 20 25
50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 9501000 1 2 3 4
time(s) cores blackscholes managed exploiting ADP and core allocation.
25/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-configuration
1 2 3 4 5 6 7 8 9 10
5 10 15 20 25
50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 9501000 1 2 3 4
time(s) cores blackscholes managed exploiting ADP and core allocation.
25/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-configuration
1 2 3 4 5 6 7 8 9 10
5 10 15 20 25
50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 9501000 1 2 3 4
time(s) cores blackscholes managed exploiting ADP and core allocation.
25/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-optimization
1 2 3 4 5 6 7 8 9 10
0.5 1 1.5 2 2.5
50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 1 2 3 4
time (s) cores canneal managed exploiting ADP and core allocation.
26/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-optimization
1 2 3 4 5 6 7 8 9 10
0.5 1 1.5 2 2.5
50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 1 2 3 4
time (s) cores canneal managed exploiting ADP and core allocation.
26/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-healing
1 2 3 4 5 6 7 8 9 10
0.5 1 1.5 2 2.5
1 2 3 4
cores
50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 1 14
time (s) frequency canneal managed exploiting ADP, core allocation, and frequency scaling.
27/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-healing
1 2 3 4 5 6 7 8 9 10
0.5 1 1.5 2 2.5
1 2 3 4
cores
50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 1 14
time (s) frequency canneal managed exploiting ADP, core allocation, and frequency scaling .
27/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-healing
1 2 3 4 5 6 7 8 9 10
0.5 1 1.5 2 2.5
1 2 3 4
cores
50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 1 14
time (s) frequency canneal managed exploiting ADP, core allocation, and frequency scaling.
27/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Self-healing
1 2 3 4 5 6 7 8 9 10
0.5 1 1.5 2 2.5
1 2 3 4
cores
50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 1 14
time (s) frequency canneal managed exploiting ADP, core allocation, and frequency scaling.
27/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Conclusions
and behavioural psychology
computing properties
28/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Conclusions
and behavioural psychology
computing properties
28/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Conclusions
and behavioural psychology
computing properties
28/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Conclusions
and behavioural psychology
computing properties
28/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Conclusions
and behavioural psychology
computing properties
28/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
Q&A
4
4http://www.dilbert.com/
29/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
References I
no, “Using clustering and metric learning to improve science return of remote sensed imagery,” ACM Trans.
http://doi.acm.org/10.1145/2168752.2168765
Free Press, 1965.
NJ, USA, 2011, aAI3445564.
30/31 – mistlab.ca
POLYTECHNIQUE MONTR´ EAL Rationale Reinforcement Learning Self-adaptive Computing Case Study Conclusions References
References II
management via self-adaptive computing,” in Proceedings of the 49th Annual Design Automation Conference, ser. DAC ’12. New York, NY, USA: ACM, 2012,
31/31 – mistlab.ca