Variational Option Discovery Algorithms Achiam, Edwards, Amodei, - - PowerPoint PPT Presentation

▶
variational option discovery algorithms
SMART_READER_LITE
LIVE PREVIEW

Variational Option Discovery Algorithms Achiam, Edwards, Amodei, - - PowerPoint PPT Presentation

Mar 13, 2024 145 likes •568 views

Variational Option Discovery Algorithms Achiam, Edwards, Amodei, Abbeel Topic: Hierarchical Reinforcement Learning Presenter: Harris Chan Overview Motivation : Reward-free option discovery Contributions Background : Universal

slide-1
SLIDE 1

Variational Option Discovery Algorithms

Achiam, Edwards, Amodei, Abbeel Topic: Hierarchical Reinforcement Learning Presenter: Harris Chan

slide-2
SLIDE 2

Overview

  • Motivation: Reward-free option discovery
  • Contributions
  • Background: Universal Policies, Variational Autoencoder
  • Method: Variational Option Discovery Algorithms, VALOR, Curriculum
  • Results
  • Discussions & Limitations
slide-3
SLIDE 3

Overview

  • Motivation: Reward-free option discovery
  • Contributions
  • Background: Universal Policies, Variational Autoencoder
  • Method: Variational Option Discovery Algorithms, VALOR, Curriculum
  • Results
  • Discussions & Limitations
slide-4
SLIDE 4

Humans find new ways to interact with environment

slide-5
SLIDE 5

Motivation: Reward-Free Option Discovery

Reward-free Option Discovery: RL agent learn skills (options) without environment reward Research Questions:

  • How can we learn diverse set of skills?
  • Do these skills match with human priors on what are useful skills?
  • Can we use these learned skills for downstream tasks?
slide-6
SLIDE 6

Limitations of Prior Related Works

  • Information Theoretic approaches: mutual info between options

and states, not full trajectories:

  • Multi-goal Reinforcement learning (goal or instruction conditioned

policies) requires:

  • Extrinsic reward signal (e.g. did the agent achieve the goal/instruction?)
  • Hand-crafted instruction space (e.g. XY coordinate of agent)
  • Intrinsic Motivations: suffers from catastrophic forgetting
  • Intrinsic reward decays over time, may forget how to revisit
slide-7
SLIDE 7

Overview

  • Motivation: Reward-free option discovery
  • Contributions
  • Background: Universal Policies, Variational Autoencoder
  • Method: Variational Option Discovery Algorithms, VALOR, Curriculum
  • Results
  • Discussions & Limitations
slide-8
SLIDE 8

Contributions

1.

Problem: Reward-free options discovery, which aims to learn interesting behaviours without environment rewards (unsupervised)

2.

Introduced a general framework Variational Option Discovery objective & algorithm

1.

Connected Variational Option Discovery and Variational Autoencoder (VAE)

3.

Specific instantiation: VALOR and Curriculum learning:

1.

VALOR: a decoder architecture using Bi-LSTM over only (some) states in trajectory

2.

Curriculum learning for increasing number of skills when agent mastered current skills

  • 4. Empirically tested on simulated robotics environments

1.

VALOR can learn diverse behaviours in variety of environments

2.

Learned policies are universal, can be interpolated and used in hierarchies

slide-9
SLIDE 9

Overview

  • Motivation: Reward-free option discovery
  • Contributions
  • Background: Universal Policies, Variational Autoencoder
  • Method: Variational Option Discovery Algorithms, VALOR, Curriculum
  • Results
  • Discussions & Limitations
slide-10
SLIDE 10

Background: Universal Policies

  • …

…

slide-11
SLIDE 11

Background: Variational Autoencoders (VAE)

  • Objective Function: Evidence Lowerbound (ELBO)
slide-12
SLIDE 12

Overview

  • Motivation: Reward-free option discovery
  • Contributions
  • Background: Universal Policies, Variational Autoencoder
  • Method: Variational Option Discovery Algorithms, VALOR, Curriculum
  • Results
  • Discussions & Limitations
slide-13
SLIDE 13

Intuition: Why VAE + Universal Policies?

Skill 1 Skill 100 . . . ? ?

Trajectory

Latent Data

?

slide-14
SLIDE 14

Variational Option Discovery Algorithms (VODA)

  • Entropy Regularization

Decoder Reconstruction

slide-15
SLIDE 15

… …

Algorithm:

  • 3. Update policy via RL to maximize:
  • 4. Update decoder with supervised learning

Variational Option Discovery Algorithms (VODA)

slide-16
SLIDE 16

Variational Option Discovery Algorithms (VODA)

slide-17
SLIDE 17

VAE vs VODA

VA E VODA

slide-18
SLIDE 18

VAE vs VODA

  • How?

“Reconstruction” “KL on prior”

slide-19
SLIDE 19

VAE vs VODA: Equivalence Proof

slide-20
SLIDE 20

Connection to existing works: VIC

Variational Intrinsic Controls (VIC):

(VODA)

  • 3. Decoder
  • nly sees first

and last state

slide-21
SLIDE 21

Connection to existing works: DIAYN

Diversity Is All You Need (DIAYN):

(VODA)

  • 1. Factorizes probability:
slide-22
SLIDE 22

VALOR: Variational Autoencoding Learning of Options by Reinforcement

slide-23
SLIDE 23

Curriculum on Contexts

  • Curriculum

Uniform Training Iteration

slide-24
SLIDE 24

Overview

  • Motivation: Reward-free option discovery
  • Contributions
  • Background: Universal Policies, Variational Autoencoder
  • Method: Variational Option Discovery Algorithms, VALOR, Curriculum
  • Results
  • Discussions & Limitations
slide-25
SLIDE 25

Experiments

1. What are the best practices when training VODAs?

1. Does the curriculum learning approach help? 2. Does embedding the discrete context help vs. one-hot vector?

2. What are the qualitative results from running VODA?

1. Are the learned behaviors recognizably distinct to a human? 2. Are there substantial differences between algorithms?

3. Are the learned behaviors useful for downstream control tasks?

slide-26
SLIDE 26

Environments: Locomotion environments

HalfCheetah Swimmer Ant Note: State is given as vectors, not raw pixels

slide-27
SLIDE 27

Implementation Details (Brief)

  • ,
slide-28
SLIDE 28

Curriculum learning on contexts does help

slide-29
SLIDE 29

…But struggle in high dimensional environment

slide-30
SLIDE 30

Embedding context is better than one-hot

  • Embedding

One-Hot

slide-31
SLIDE 31

Qualitatively learns some interesting behaviors

  • VALOR/VIC able to

find locomotion gaits that travel in variety

  • f speeds/directions
  • DIAYN learns

behaviours that ‘attain target state’ (fixed/unmoving target state)

  • Note: Original DIAYN

use SAC

VALO R DIAY N

Source: https://varoptdisc.github.io/

slide-32
SLIDE 32

Qualitative results (Quantified)

  • Behaviours
slide-33
SLIDE 33

Can somewhat interpolate behaviours

  • Interpolating between context embeddings yields reasonably smooth

behaviours

  • X-Y Traces for behaviours learned by VALOR

Point Env Ant Env Embedding 1 Embedding 2 Interpolated embedding

slide-34
SLIDE 34

Experiment: Downstream tasks on Ant-Maze

slide-35
SLIDE 35

Overview

  • Motivation: Reward-free option discovery
  • Contributions
  • Background: Universal Policies, Variational Autoencoder
  • Method: Variational Option Discovery Algorithms, VALOR, Curriculum
  • Results
  • Discussions & Limitations
slide-36
SLIDE 36

Discussion and Limitations

  • Learned behaviours are unnatural
  • Due to using purely information theoretic approach?
  • Struggle in high dimensional environments (e.g. Toddler)
  • Need better performance metrics for evaluating discovered

behaviours

  • Hierarchies built on top of learned contexts do not outperform

task-specific policies learned from scratch

  • But at least universal enough to be able to adapt to more complex tasks
  • Specific curriculum on context equation seems unprincipled/hacky
slide-37
SLIDE 37

Follow Up Works

slide-38
SLIDE 38

Future Research Directions

  • Fix “unnaturalness” of learned behaviours: incorporate human

priors?

  • Distinguish trajectories in ways which corresponds to human intuition
  • Leverage demonstration? Human-in-the-loop feedback?
  • Architectures: Use Transformers instead of Bi-LSTM for decoder
  • As done in NLP: ELMO (Bi-LSTM) vs BERT (Transformer)
slide-39
SLIDE 39

Contributions

1.

Problem: Reward-free options discovery, which aims to learn interesting behaviours without environment rewards (unsupervised)

2.

Introduced a general framework Variational Option Discovery objective & algorithm

1.

Connected Variational Option Discovery and Variational Autoencoder (VAE)

3.

Specific instantiation: VALOR and Curriculum learning:

1.

VALOR: a decoder architecture using Bi-LSTM over only (some) states in trajectory

2.

Curriculum learning for increasing number of skills when agent mastered current skills

  • 4. Empirically tested on simulated robotics environments

1.

VALOR can learn diverse behaviours in variety of environments

2.

Learned policies are universal, can be interpolated and used in hierarchies

slide-40
SLIDE 40

References

  • 1. Achiam, et al. Variational Option Discovery Algorithms
  • 2. (VIC) Variational Intrinsic Control
  • 3. (DIAYN) Diversity Is All You Need
  • 4. Rich Sutton’s page on Options Discovery