SLIDE 1
Transfer Learning for Improving Model Predictions in Highly - - PowerPoint PPT Presentation
Transfer Learning for Improving Model Predictions in Highly - - PowerPoint PPT Presentation
Transfer Learning for Improving Model Predictions in Highly Configurable Software Pooyan Jamshidi, Miguel Velez , Christian Kstner, Norbert Siegmund, Prasad Kawthekar Highly Configurable Systems In Dynamic Environments [Credit to CMU
SLIDE 2
SLIDE 3
SLIDE 4
Highly Configurable Systems In Dynamic Environments
[Credit to CMU CoBot]
SLIDE 5
Options Influence Performance
SLIDE 6
Adapt to Different Environments
TurtleBot
SLIDE 7
Adapt to Different Environments
50 + 3*C1 + 15*C2 - 7*C2 *C3 TurtleBot
SLIDE 8
Adapt to Different Environments
50 + 3*C1 + 15*C2 - 7*C2 *C3 TurtleBot
SLIDE 9
Classic Sensitivity Analysis
50 + 3*C1 + 15*C2 - 7*C2 *C3 TurtleBot
SLIDE 10
Classic Sensitivity Analysis
50 + 3*C1 + 15*C2 - 7*C2 *C3 TurtleBot
Measure
SLIDE 11
Classic Sensitivity Analysis
50 + 3*C1 + 15*C2 - 7*C2 *C3 TurtleBot
Measure Learn
SLIDE 12
Classic Sensitivity Analysis
50 + 3*C1 + 15*C2 - 7*C2 *C3 TurtleBot
Measure Learn
SLIDE 13
Applications
50 + 3*C1 + 15*C2 - 7*C2 *C3 TurtleBot
Measure Learn
Optimization Adaptation + Reasoning + Debugging +
SLIDE 14
Measuring Performance is Expensive
25 options × 10 values = 1025 configurations Measure
[Credit to Peng Hou]
SLIDE 15
Transfer Learning
SLIDE 16
Reuse Data From Similar System
TurtleBot
Measure
SLIDE 17
Reuse Data From Similar System
TurtleBot
Measure
Simulator (Gazebo)
SLIDE 18
Reuse Data From Similar System
TurtleBot
Measure Measure
Simulator (Gazebo)
SLIDE 19
Reuse Data From Similar System
TurtleBot
Measure Reuse Measure
Simulator (Gazebo)
SLIDE 20
Reuse Data From Similar System
50 + 3*C1 + 15*C2 - 7*C2 *C3 TurtleBot
Measure Reuse Measure
Learn with TL
Simulator (Gazebo)
SLIDE 21
Reuse Data From Similar System
50 + 3*C1 + 15*C2 - 7*C2 *C3 TurtleBot
Measure Reuse Measure
Learn with TL
Simulator (Gazebo)
SLIDE 22
Transfer Between Different Systems
SLIDE 23
Transfer Between Different Systems
SLIDE 24
Transfer Between Different Systems
SLIDE 25
Transfer Between Different Systems
(0, 0) → (7, 12) (0, 0) → (0, 10)
SLIDE 26
Exploiting Similarity
SLIDE 27
Function Prediction
1 3 5 9 10 11 12 13 14 50 100 150 200 Target function Target samples
SLIDE 28
Prediction without Transfer Learning
SLIDE 29
Prediction with More Data but without Transfer Learning
SLIDE 30
Prediction with Transfer Learning
SLIDE 31
Technical Details
SLIDE 32
Prediction With Transfer Learning
true function GP mean GP variance
- bservation
selected point true minimum
Motivations:
- 1. mean estimates + variance
- 2. all computation are linear algebra
- 3. good estimations with few data
SLIDE 33
Prediction With Transfer Learning
true function GP mean GP variance
- bservation
selected point true minimum
Motivations:
- 1. mean estimates + variance
- 2. all computation are linear algebra
- 3. good estimations when few data
https://github.com/pooyanjamshidi/transferlearning
SLIDE 34
Prediction With Transfer Learning
true function GP mean GP variance
- bservation
selected point true minimum
Motivations:
- 1. mean estimates + variance
- 2. all computation are linear algebra
- 3. good estimations when few data
https://github.com/pooyanjamshidi/transferlearning pjsamshid@cs.cmu.edu
SLIDE 35
Evaluation
SLIDE 36
RQ1: Improve prediction accuracy? RQ2: Tradeoffs among number of source and target samples? RQ3: Fast enough for self-adaptive systems?
Case Study and Controlled Experiments
SLIDE 37
Autonomous service robot Environmental change 3 stream processing apps Workload change NoSQL DBMS Workload & hardware changes
Analyzed Systems
SLIDE 38
Prediction Accuracy
SLIDE 39
5 10 15 20 25 5 10 15 20 25 5 10 15 20 25
Performance Prediction for CoBot
Source Model
SLIDE 40
5 10 15 20 25 5 10 15 20 25 5 10 15 20 25 5 10 15 20 25 5 10 15 20 25 5 10 15 20 25
Performance Prediction for CoBot
Source Model Target Model
SLIDE 41
5 10 15 20 25 5 10 15 20 25 5 10 15 20 25 5 10 15 20 25 5 10 15 20 25 5 10 15 20 25
Performance Prediction for CoBot
Source Model Target Model
5 10 15 20 25 5 10 15 20 25 5 10 15 20 25
Prediction with 4 samples
CPU usage[%]
SLIDE 42
5 10 15 20 25 5 10 15 20 25 5 10 15 20 25 5 10 15 20 25 5 10 15 20 25 5 10 15 20 25 5 10 15 20 25 5 10 15 20 25 5 10 15 20 25
Performance Prediction for CoBot
Source Model Target Model
5 10 15 20 25 5 10 15 20 25 5 10 15 20 25
Prediction with 4 samples Prediction with TL
CPU usage[%] CPU usage[%]
SLIDE 43
Number of Source and Target Samples
SLIDE 44
Prediction Error with Different Source and Target Samples
% Source %Target
SLIDE 45
Prediction Error with Different Source and Target Samples
% Source %Target
SLIDE 46
Prediction Error with Different Source and Target Samples
% Source %Target
SLIDE 47
Prediction Error with Different Source and Target Samples
% Source %Target
SLIDE 48
Prediction Error with Different Source and Target Samples
% Source %Target
SLIDE 49
Prediction Error with Different Source and Target Samples
% Source %Target
SLIDE 50
Accuracy and Costs
% Source %Target B u d g e t
SLIDE 51
Prediction error of other systems
SLIDE 52
Applicable in Self-Adaptive Systems
SLIDE 53
Low model training overhead Produces accurate models Models can improve at each adaptation cycle
Transfer Learning in Self-Adaptive Systems
SLIDE 54
Future Work, Insights, and Ideas
SLIDE 55
Selecting from Multiple Sources
Source Simulator Target Simulator Source Robot Target Robot (1) (2) (3) (4) (5)
C1 C2 C3
SLIDE 56
Active Learning with Transfer Learning
50 + 3*C1 + 15*C2 - 7*C2 *C3 TurtleBot
Measure Reuse Measure
Learn with TL
Simulator (Gazebo)
Iteratively find best sample points that maximize knowledge
SLIDE 57
Integrating Transfer Learning in MAPE-K
Contribute to Knowledge Assist in self-optimization Support online learning
SLIDE 58
Summary
SLIDE 59
Transfer Learning for Improving Model Prediction in Highly Configurable Software
Reuse data from similar system Improves model accuracy Applicable in self-adaptive systems
SLIDE 60