Reinforcement Learning in a Physics Inspired Semi-Markov Environment - - PowerPoint PPT Presentation

Reinforcement Learning in a Physics Inspired Semi-Markov Environment

National Research Council of Canada, Digital Technologies Ottawa, Canada colin.bellinver@nrc-cnrc.gc.ca https://web.cs.dal.ca/~bellinger May, 2020

CLEAN CLEAN

Colin Bellinger, Rory Coles, Mark Crowley, and Isaac Tamblyn

Background

RL and Materials Science

• Material are critical to advancements in

science and society

• Carbon fibre
• Bicycles, sailboats, hockey sticks, etc
• Materials design process
• Iterative process
• Time consuming, expensive and human

dependent,

• AI aided human design could expedite design

RL and Materials Science

Overview

• Investigate suitability of deep RL for applications in materials

science

• Semi-Markov decision processes
• Present a new physics-inspired semi-Markov learn

environment

• Evaluation value-based deep RL algorithms in the

environment

Background

AI For Sequential Decision Making

• Markov decision process
• Formalization of sequential decision making

problems

• Choice of action

influence reward and next state

• States emitted from environment satisfy Markov

property (fully observable)

• State

is sufficient to select action

at ∈ 풜 rt st+1 s0, a0, r1, s1, a1, r2, s2, a2, r3, s3, . . . st+1 at

Background

AI For Sequential Decision Making

• Partially observable (PO) MDP
• Complete state information is not available
• Observation of next state

is NOT sufficient to select action

• Semi-Markov Decision process: for states
• States

are partially observable

• State

are fully observable

st+1 at 풮 = {s0, s1, . . . , sn} 풮′ ⊂ 풮 풮∖풮′

Background

Reinforcement Learning

• RL agent interacts with Environment
• Given state

, the agent picks an action

• Receives:
• Reward

,

• Change in the state of the environment
• Agents aims to learn optimal policy:
• Maximize the discounted sum of rewards
• Online and simulated experience

t = 0,1,2,...

풮t 풜t ℛt+1 풮t+1

π : S × A → [0,1]

Physics Inspired RL Environment

Phase Change Dynamic

• Phases describe the state of a material
• Solid, liquid, gas
• Material phases reacts to temperature , pressure , heat

, and work

• Many to few mapping between , ,

, and material phases

• Adding heat

increases , and removing it decreases

• Doing positive work

increase , and negative work decreases

• Relationship between

, , , and is different at some phase boundaries

T P Q W T P Q W Q+ T Q− T W+ P W− P Q W T P

Physics Inspired RL Environment

Phase Change RL Environment

* http://clean.energyscience.ca/gyms

• Approximates phase

change dynamics

• 3 phases

3 phases

Physics Inspired RL Environment

Phase Change RL Environment

* http://clean.energyscience.ca/gyms

• Approximates phase

change dynamics

• 3 phases
• 2 phase change

boundaries Phase change boundaries

Physics Inspired RL Environment

Phase Change RL Environment

* http://clean.energyscience.ca/gyms

Fixed goal with ,

r(S) = 1 S = Ms = (Ts, Ps)

• Approximates phase

change dynamics

• 3 phases
• 2 phase change

boundaries

• 1 goal

Ms

Physics Inspired RL Environment

Phase Change RL Environment

* http://clean.energyscience.ca/gyms

Start states with Increasing difficulty

• Approximates phase

change dynamics

• 3 phases
• 2 phase change

boundaries

• 1 goal
• 3 start states

Physics Inspired RL Environment

Phase Change RL Environment

* http://clean.energyscience.ca/gyms

4 actions, 


풜 = {a0 = Q−, a1 = Q+, a2 = W−, a3 = W+}

Physics Inspired RL Environment

Phase Change RL Environment

* http://clean.energyscience.ca/gyms

Agent at time in state , taking action

t St = (x, y) = at = 1

Moves to state at time

St+1 = (x − 1,y) = t + 1

Physics Inspired RL Environment

Phase Change RL Environment

* http://clean.energyscience.ca/gyms

A sequence of two actions is required to move into the next phase

• 1. Action

state 
 at time

at = 3 → St+1 = (x, y) = t + 1

• 2. Action

state 
 at time

at = 1 → St+2 = (x, y) = t + 2

Partially observable

Experimental Setup

RL Algorithms

• Deep Q-Learning (DQN) (Mnih, 2013)
• Off-policy TD control algorithm Q-learning (Watkins, 1989)
• Network input
• Network output
• greedy action selection
• Trained with experience replay
• Batches of 128 tuples

st = (Tt, Pt)

• t = Q(st, a|θ)

ϵ at = maxaQ(st, a|θ) (st, at, rt+1, st+1)

Current state (2-dimensional) Hidden layer (1x78) Q-values (4-dimensional)

Experimental Setup

RL Algorithms

• Recurrent Q-Learning (DRQN) (Hausknecht, 2015)
• 128 unit Gated Recurrent Unit
• Trained with experience replay
• Batches of 128 tuples
• Improves:
• Partial observability via internal memory

(st, at, rt+1, st+1)

Current state (2-dimensional) GRU (1x128) Q-values (4-dimensional)

Experimental Setup

RL Algorithms

• Hindsight Experience Replay (HER)
• Training framework: Agents to learn from their mistakes
• 5% of failed action recast as successes
• Improves:
• Sample efficiency
• Learning from sparse and binary reward signals

Experimental Setup

Questions of Interest

• What is the impact of PO phase change boundaries?
• Number of steps to goal
• Number of episodes of training
• Is RNN memory sufficient to solve the semi-Markov problem?
• Does HER improve the sample efficiency of DRQN?

Results

DQN - Different Starting Points

• DQN Agent on EX_hard significantly worse
• Higher variance
• More steps
• Agent does not learn a good policy
• What makes EX_hard significantly more

challenging?

Semi-Markov

Easy Moderate Hard

Large number

• f steps

Near optimal number of steps

Results

DQN - Semi-Markov versus Markov Phase Change Env.

• DQN agents is significantly inhibited by phase change boundary
• Agent must remember last 2 steps
• 2 versus 1 PO boundary has major impact

Semi-Markov

Faster learning and lower variance without POMDP boundaries

Markov

Results

DQN Versus DRQN - Different Starting Points

• DRQN consistently outperforms DQN
• DQN: mean steps = 2,400, after 10,000 episode
• DRQN: mean steps = 530, after 10,000 episode

DQN Semi-Markov DRQN Semi- Markov

DQN: High variance and mean steps DRQN: Low variance and mean steps

Results

DRQN - Semi-Markov versus Markov Phase Change Env.

• Does DRQN solve the PO phase change boundaries?
• EX_hard
• Semi-Markov: mean steps = 530, after 10,000

episode

• Markov: mean steps = 40, after 3,750 episodes

Semi-Markov Markov

No, notable difference in means

Results

DRQN + HER Versus DQN + HER

• HER improves learning rate and lowers variance
• DQN:
• Standard: mean steps = 2,400, after 10,000

episode

• HER: mean steps = 195, after 6,000 episode
• DRQN:
• Standard: mean steps = 530, after 10,000 episode
• HER: mean steps = 145, after 4,000 episode

DRQN Semi-Markov DQN Semi-Markov

Results

State Visit Frequency

Bouncing around the “walls” Low rates of success Struggling with the phase boundary

High frequency Low frequency

DRQN+HER - Early performance (0-10 episodes)

Results

State Visit Frequency

Early life (0-10 episodes)

Still struggling with the phase boundary

High frequency Low frequency

DRQN+HER - Midlife (190-200 episodes) More direct path to goal

Results

State Visit Frequency

Early life (0-10 episodes) Midlife (190-200 episodes)

High frequency Low frequency

DRQN+HER - Later life (390-400 episodes) Improved, but has not master the boundaries

Conclusion

• Introduced physics-inspired Phase Change Environment
• Semi-Markov sequential decision making problem based on material phases
• Findings include
• DRQN benefits from training with HER
• DRQN improved performance on PO phase change boundaries
• RNN memory not sufficient to completely solve the problem
• Approximately 3x optimal number of step on EX_hard

Summary of Results

• RL with recurrent memory has potential for simple PO material problem
• Significant refinement required for real-world problems
• Materials science is a great testbed for general RL research
• Domain properties include:
• Partial observability
• Observation costs
• Spatial-temporal variability of sensors
• Availability of weak models

Conclusion

Discussion and Future work

Thank you

Colin Bellinger, Rory Coles, Mark Crowley, and Isaac Tamblyn

colin.bellinver@nrc-cnrc.gc.ca https://web.cs.dal.ca/~bellinger

CLEAN CLEAN

References

(Watkins, 1989) Watkins, C.J. and Dayan, P ., 1992. Q-learning. Machine learning, 8(3-4), pp.279-292. (Mnih, 2013) Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D. and Riedmiller, M., 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. (Hausknecht, 2015) Hausknecht, M. and Stone, P ., 2015, September. Deep recurrent q- learning for partially observable mdps. In 2015 AAAI Fall Symposium Series. (Andrychowicz, 2017) Andrychowicz, M., Wolski, F ., Ray, A., Schneider, J., Fong, R., Welinder, P ., McGrew, B., Tobin, J., Abbeel, O.P . and Zaremba, W., 2017. Hindsight experience replay. In Advances in neural information processing systems (pp. 5048-5058).