Path following with reinforcement learning for autonomous cars - - - PowerPoint PPT Presentation

▶
path following with reinforcement learning for autonomous
SMART_READER_LITE
LIVE PREVIEW

Path following with reinforcement learning for autonomous cars - - - PowerPoint PPT Presentation

Sep 11, 2023 463 likes •715 views

Path following with reinforcement learning for autonomous cars - Mozzam Motiwala (IAS) Index Basics of Reinforcement Learning Model Based vs Model Free Reinforcement Learning Autonomous Car collision avoidance What is Reinforcement

slide-1
SLIDE 1

Path following with reinforcement learning for autonomous cars

  • Mozzam Motiwala (IAS)
slide-2
SLIDE 2

Index

  • Basics of Reinforcement Learning
  • Model Based vs Model Free Reinforcement

Learning

  • Autonomous Car collision avoidance
slide-3
SLIDE 3

What is Reinforcement Learning?

  • Learning by trial and error only based on a

reward signal[1]

Exploration vs Exploitation?

https://towardsdatascience.com/solving-the-multi-armed- bandit-problem-b72de40db97c

slide-4
SLIDE 4

Markov-Desicion Process

Reward Function? Transition Function? Policy? Optimal Policy?

[1]

slide-5
SLIDE 5

Some terminalogy

  • Value Function:
  • Action Value Function:

Why Discounting Factor?

slide-6
SLIDE 6

Gridworld

[1]

slide-7
SLIDE 7

Finding Optimal Policy

[1]

slide-8
SLIDE 8

Cart Pole Balancing Problem

https://www.youtube.com/watch?v=Lt-KLtkDlh8

https://towardsdatascience.com/cartpole-introduction-to- reinforcement-learning-ed0eb5b58288

slide-9
SLIDE 9

Index

  • Basics of Reinforcement learning
  • Model Based vs Model Free Reinforcement

Learning

  • Autonomous Car collision avoidance
slide-10
SLIDE 10

Model-based

By a model of the environment we mean anything that an agent can use to predict how the environment will respond to its actions[2].

https://towardsdatascience.com/model-based-reinforcement-learning-cb9e41ff1f0d

slide-11
SLIDE 11

Example

Whats Next? :: Now lets sample from it to adjust the policy..

https://towardsdatascience.com/model-based-reinforcement-learning-cb9e41ff1f0d

slide-12
SLIDE 12

Why model-based RL?

Reduced number of interaction with the real environment while learning. Types: Neural Network Model,

Guassian Process Model.. etc

Advantages?

  • Fast
  • Need less data

Problems?

  • What if the model is wrong?
slide-13
SLIDE 13

Model Based+ Model Free

[2]

slide-14
SLIDE 14

Results

[1]

slide-15
SLIDE 15

Why better result?

[1]

slide-16
SLIDE 16

Index

  • Basics of Reinforcement learning
  • Model Based vs Model Free Reinforcement

Learning

  • Autonomous Car Collision Avoidance
slide-17
SLIDE 17

Application: Autonomous Car Why Reinforcement Learning?

Problem with traditional methods

  • Slow
  • Assumptions

Learning in RL

  • Adapting to environment
  • Learning from mistakes
slide-18
SLIDE 18

Generalized Computation Graph

Self-supervised Deep Reinforcement Learning with Generalized Computation Graphs(GCG) for Robot Navigation[3]

  • H=1 : Model-Free
  • H= N (Length of Episode): Model-Based

[3]

slide-19
SLIDE 19

Model Details

  • Deep RNN as Model
  • Model output 1= Current Reward ŷ: Robots speed
  • Model output 2= Future Value to go(value of the state)

^b: Distance travelled before collision

  • Policy Evaluation Function :
  • Policy Evaluation by sampling k random action

sequence and selecting the one with max reward.

slide-20
SLIDE 20

GCG : Algorithm

[3]

slide-21
SLIDE 21

Evaluation and Results

https://www.youtube.com/watch?v=NlFbLVG6LpA

[3]

slide-22
SLIDE 22

Summary

  • Benefits of Reinforcement Learning
  • Model-Free vs Model-Based
  • Combined approach that subsumes Model-free

and Model-based

slide-23
SLIDE 23

References

  • 1. R. Sutton and A. Barto, Reinforcement Learning: An

Introduction

  • 2. R. Sutton, “Dyna, an Integrated Architecture for Learning,

Planning,and Reacting,” in AAAI, 1991.

  • 3. G. Kahn, A. Villaflor, B. Ding, P. Abbeel, and S. Levine. Self-

Supervised Deep ReinforcementLearning with Generalized Computation Graphs for Robot Navigation. InIEEE InternationalConference on Robotics and Automation, 2018.

slide-24
SLIDE 24

Question?