Distributed Meta Optimization of Reinforcement Learning Agents Greg - - PowerPoint PPT Presentation
Distributed Meta Optimization of Reinforcement Learning Agents Greg Heinrich, Iuri Frosio - GTC San Jose, March 2019 Contents Introduction to Reinforcement Learning Introduction to Metaoptimization (on distributed systems) / Maglev
Greg Heinrich, Iuri Frosio - GTC San Jose, March 2019
2
Contents
Introduction to Reinforcement Learning Introduction to Metaoptimization (on distributed systems) / Maglev Metaoptimization and Reinforcement Learning (on distributed systems) HyperTrick Results Conclusion
3
(available at https://openreview.net/forum?id=r1VGvBcxl¬eId=r1VGvBcxl). Open source implementation: https://github.com/NVlabs/GA3C.
4
Image from www.33rdsquare.com
5
✓ Environment ✓ Agent ✓ Observable status St ✓ Reward Rt ✓ Action at ✓ Policy at = π(St)
6
7
R0 R1 R2 R3 R4
8
9
(available at https://openreview.net/forum?id=r1VGvBcxl¬eId=r1VGvBcxl). Open source implementation: https://github.com/NVlabs/GA3C.
10
Master model
St, Rt R0 R1 R2 R3 R4 at = π(St) St, Rt R0 R1 R2 R3 R4 at = π(St) St, Rt R0 R1 R2 R3 R4 at = π(St)
Agent 1 Agent 2 Agent 16
11
(available at https://openreview.net/forum?id=r1VGvBcxl¬eId=r1VGvBcxl). Open source implementation: https://github.com/NVlabs/GA3C.
12
Pear, pear, pear, pear, … Empty, empty, … Fig, fig, fig, fig, fig, fig, Strawberry, Strawberry, Strawberry, … …
data labels 100% utilization / occupancy status, reward action
13
Master model
St, Rt R0 R1 R2 R3 R4 at = π(St) St, Rt R0 R1 R2 R3 R4 at = π(St) St, Rt R0 R1 R2 R3 R4 at = π(St)
Agent 1 Agent 2 Agent 16
t_max
14
Master model
St, Rt R0 R1 R2 R3 R4 at = π(St) St, Rt R0 R1 R2 R3 R4 at = π(St) St, Rt R0 R1 R2 R3 R4 at = π(St)
Agent 1 Agent 2 Agent 16
15
Master model
St, Rt R0 R1 R2 R3 R4 at = π(St) St, Rt R0 R1 R2 R3 R4 at = π(St) St, Rt R0 R1 R2 R3 R4 at = π(St)
Agent 1 Agent 2 Agent 16
t_max
16
Master model
St, Rt R0 R1 R2 R3 R4 at = π(St) St, Rt R0 R1 R2 R3 R4 at = π(St) St, Rt R0 R1 R2 R3 R4 at = π(St)
Agent 1 Agent 2 Agent 16
t_max
17
Master model
St predictors
{St}
prediction queue
Agent 1 Agent 2
Agent N {at} at
18
R0 R1 R2 R4
trainers
{St,Rt}
training queue Δπ(∙)
Agent 1 Agent 2
Agent N St,Rt
Master model
19
Master model
St predictors
{St}
prediction queue
Agent 1 Agent 2
Agent N {at} at
R0 R1 R2 R4
trainers
{St,Rt}
training queue Δπ(∙)
20
CPU for environment simulation GPU for inference / training
21
t_max = 4 [Play to the end - Monte Carlo] No variance (collected rewards are real) High bias (we played only once) One update every t_max frames 1 4
22
t_max = 2 Value network High variance (noisy value network) Low bias (unbiased net, many agents) More updates per second 1 4 Value network: from here, approximately 2.5 t_max affects bias, variance, computational cost (number of updates per second, batch size)
23
Hyperparameter search in 2015: The search for the optimal learning rate:
https://arxiv.org/pdf/1602.01783.pdf
24
hyperparameters search
convergence
but … We can run parallel instances of GA3C on a distributed system
sighted agent)
stability* of GA3C
* See G. Heinrich, I. Frosio, Metaoptimization on a Distributed System forDeep Reinforcement Learning, ttps://arxiv.org/abs/1902.02725.
25
Contents
Introduction to Reinforcement Learning Introduction to Metaoptimization (on distributed systems) / Maglev Metaoptimization and Reinforcement Learning (on distributed systems) HyperTrick Results Conclusion
26 26
GA3C Agent Concorde
▪ Number of layers and their width ▪ Choice of activations
▪ Learning rate ▪ Reward decay rate (𝛅)
▪ back-propagation window size (tmax)
▪ Choice of optimizer ▪ Number of training episodes
▪ Environment model.
Exhaustive search is intractable
Source: Christian Kath
27 27
Example: Tree of Parzen Estimators
minimize.
▪ Exploitation v.s. exploration. ▪ Wall time v.s. resource efficiency.
Random Search.
before historical records can be leveraged. Warm Starts are needed to cut down complexity over time.
→
28 28
Metric Variance
experiments inconclusive.
improvement if it works under a variety of conditions. Meta Optimization should be part of data scientists’ daily routine.
→
29 29
Complex Pipelines
reduced to a single Python function, or Docker container. Meta Optimization must be independent of task scheduling.
30 30
Architecture
Traceable Machine Learning Workflows
Experiments
isolation, or combined for maximum traceability.
31 31
Knowledge Base
connected.
through their relationships with
No Information silo.
Model
id 6a-e7-59 job_id ae-45-4a param_set_id 45-4a-26 creator bob created 2018-09-28 dataset_id ac-73-fc
Workflow
id fb-d5-7a Desc my_v0.1.1 spec ... project DormRoom creator bob created 2018-09-28
Job
id ae-45-4a scm SHA:3487c creator bob workflow_id fb-d5-7a created 2018-09-28
ExperimentSet
id 03-ad-24 config yml creator bob project DormRoom created 2018-09-28
ParamSet
id 45-4a-26 exp_id 78-3f-58
Metric
id 69-bd-4c param_set_id 45-4a-26 job_id ae-45-4a model_id 6a-e7-59 name xentropy value 0.01 creator bob created 2018-09-28 dataset_id ac-73-fc
Dataset
id ac-73-fc vdisk zzz://... project DormRoom creator bob created 2018-09-28
32 32
Main SDK Features
types.
methods.
parameter picking methods.
33
Contents
Introduction to Reinforcement Learning Introduction to Metaoptimization (on distributed systems) / Maglev Metaoptimization and Reinforcement Learning (on distributed systems) HyperTrick Results Conclusion
34 34
35
Contents
Introduction to Reinforcement Learning Introduction to Metaoptimization (on distributed systems) / Maglev Metaoptimization and Reinforcement Learning (on distributed systems) HyperTrick Results Conclusion
36 36
Successive Halving (SH) [1]
Terminate ½ of workers every N*2P units of work (P is a phase index). Requires synchronization between workers. Assumes relative perf over time is constant.
[1] https://arxiv.org/abs/1502.07943
37 37
Successive Halving (SH) [1]
Terminate ½ of workers every N*2P units of work (P is a phase index). Requires synchronization between workers. Assumes relative perf over time is constant. Run several instances of SH in parallel with different values of N. Does not assume constant relative perf but still requires synchronization.
HyperBand [2]
[1] https://arxiv.org/abs/1502.07943 [2] http://arxiv.org/abs/1603.06560
38 38
Successive Halving (SH) [1] HyperTrick [3]
Terminate ½ of workers every N*2P units of work (P is a phase index). Requires synchronization between workers. Assumes relative perf over time is constant. Run several instances of SH in parallel with different values of N. Does not assume constant relative perf but still requires synchronization.
Parameters:
Worker:
while maglev.should_continue(): work() maglev.report_metrics()
MagLev:
bottom √r quantile. Expected number of workers at end of phase P is N(1-r)P
HyperBand [2]
[1] https://arxiv.org/abs/1502.07943 [2] http://arxiv.org/abs/1603.06560 [3] https://arxiv.org/abs/1902.02725
39 39
Successive Halving HyperTrick
No context switches, shorter wall time Must support preemption
40
Contents
Introduction to Reinforcement Learning Introduction to Metaoptimization (on distributed systems) / Maglev Metaoptimization and Reinforcement Learning (on distributed systems) HyperTrick Results Conclusion
41 41
42 42
Boxing Centipede Pacman Pong
unpainted area = saved resources
43 43
→
Cluster occupancy (Pong) Timelines (Pong) Best Score v.s. Wall Time (Pong)
44 44
45 45
Take Away Related Talks
RL is unstable => meta optimization is useful. Hypertrick, a new algorithm for metaoptimization. GA3C + HyperTrick + Maglev is effective. Our paper: https://arxiv.org/abs/1902.02725 GA3C: https://github.com/NVlabs/GA3C MagLev info: Yehia Khoja (ykhoja@nvidia.com)
S9649
Wed 9:00am
NVIDIA's AI Infrastructure for Self-Driving Cars
Clement Farabet
S9613
Wed 10:00am
Deep Active Learning
Adam Lesnikowski
S9911
Wed 2:00pm
Determinism In Deep Learning
Duncan Riach
S9630
Thu 2:00pm
Scaling Up DL for Autonomous Driving
Jose Alvarez
S9987
Thu 9:00am
MagLev: production-grade AI platform...
Divya Vavili