Reinforcement Learning Kevin Spiteri April 21, 2015 n-armed bandit - - PowerPoint PPT Presentation

▶
reinforcement learning
SMART_READER_LITE
LIVE PREVIEW

Reinforcement Learning Kevin Spiteri April 21, 2015 n-armed bandit - - PowerPoint PPT Presentation

Sep 22, 2022 144 likes •1k views

Reinforcement Learning Kevin Spiteri April 21, 2015 n-armed bandit n-armed bandit 0.9 0.5 0.1 n-armed bandit 0.9 0.5 0.1 0.0 0.0 0.0 0.0 estimate n-armed bandit 0.9 0.5 0.1 0 0.0 0.0 0.0 0.0 estimate 0 0 0 0.0 0

slide-1
SLIDE 1

Reinforcement Learning

Kevin Spiteri April 21, 2015

slide-2
SLIDE 2

n-armed bandit

slide-3
SLIDE 3

n-armed bandit

0.9 0.5 0.1

slide-4
SLIDE 4

n-armed bandit

0.9 0.5 0.1 0.0 0.0 0.0 estimate 0.0

slide-5
SLIDE 5

n-armed bandit

0.9 0.5 0.1 0.0 0.0 0.0 estimate 0.0 0.0 attempts 0.0 payoff

slide-6
SLIDE 6

n-armed bandit

0.9 0.5 0.1 0.0 0.0 1.0 0.0 estimate 0.0 0 1 0.0 attempts 0 1 0.0 payoff 1 1 1.0

slide-7
SLIDE 7

n-armed bandit

0.9 0.5 0.1 0.0 1.0 0.5 0.0 estimate 0.0 1 2 0.0 attempts 1 1 0.0 payoff 1 2 0.5

slide-8
SLIDE 8

Exploration

0.9 0.5 0.1 0.0 1.0 0.5 0.0 estimate 0.0 0 1 2 0.0 attempts 0 1 1 0.0 payoff 2 3 0.67

slide-9
SLIDE 9

Going on …

0.9 0.5 0.1 0.9 0.5 0.0 estimate 0.1 280 10 0.0 attempts 10 252 5 0.0 payoff 1 258 300 0.86

slide-10
SLIDE 10

Changing environment

0.7 0.8 0.1 0.9 0.5 0.0 estimate 0.1 280 10 0.0 attempts 10 252 5 0.0 payoff 1 258 300 0.86

slide-11
SLIDE 11

Changing environment

0.7 0.8 0.1 0.8 0.65 0.0 estimate 0.1 560 20 0.0 attempts 20 448 13 0.0 payoff 2 463 600 0.77

slide-12
SLIDE 12

Changing environment

0.7 0.8 0.1 0.74 0.74 0.0 estimate 0.1 1400 50 0.0 attempts 50 1036 37 0.0 payoff 5 1078 1500 0.72

slide-13
SLIDE 13

n-armed bandit

  • Optimal payoff (0.82):

0.9 x 300 + 0.8 x 1200 = 1230

  • Actual payoff (0.72):

0.9 x 280 + 0.5 x 10 + 0.1 x 10 + 0.7 x 1120 + 0.8 x 40 + 0.1 x 40 = 1078

slide-14
SLIDE 14

n-armed bandit

  • Evaluation vs instruction.
  • Discounting.
  • Initial estimates.
  • There is no best way or standard way.
slide-15
SLIDE 15

Markov Decision Process (MDP)

slide-16
SLIDE 16

Markov Decision Process (MDP)

  • States
slide-17
SLIDE 17

Markov Decision Process (MDP)

  • States
slide-18
SLIDE 18

Markov Decision Process (MDP)

  • States
  • Actions

a b c

slide-19
SLIDE 19

Markov Decision Process (MDP)

  • States
  • Actions
  • Model

a 0.25 a 0.75 b c

slide-20
SLIDE 20

Markov Decision Process (MDP)

  • States
  • Actions
  • Model

a 0.25 a 0.75 b c

slide-21
SLIDE 21

Markov Decision Process (MDP)

  • States
  • Actions
  • Model
  • Reward

a 0.25 a 0.75 b c 5

  • 1
slide-22
SLIDE 22

Markov Decision Process (MDP)

  • States
  • Actions
  • Model
  • Reward
  • Policy

a 0.25 a 0.75 b c 5

  • 1
slide-23
SLIDE 23

Markov Decision Process (MDP)

  • States: ball

table hand basket floor

t h t b f

slide-24
SLIDE 24

Markov Decision Process (MDP)

  • States: ball

table hand basket floor

h t b f

slide-25
SLIDE 25

Markov Decision Process (MDP)

  • States: ball

table hand basket floor

  • Actions:

a) attempt b) drop c) wait

a b c h t b f

slide-26
SLIDE 26

Markov Decision Process (MDP)

  • States: ball

table hand basket floor

  • Actions:

a) attempt b) drop c) wait

a 0.25 a 0.75 b c h t b f

slide-27
SLIDE 27

Markov Decision Process (MDP)

  • States: ball

table hand basket floor

  • Actions:

a) attempt b) drop c) wait

a 0.25 a 0.75 b c h t b f

slide-28
SLIDE 28

Markov Decision Process (MDP)

  • States: ball

table hand basket floor

  • Actions:

a) attempt b) drop c) wait

a 0.25 a 0.75 b c 5

  • 1

h t b f

slide-29
SLIDE 29

Markov Decision Process (MDP)

  • States: ball

table hand basket floor

  • Actions:

a) attempt b) drop c) wait

a 0.25 a 0.75 b c 5

  • 1

h t b f

Expected reward per round: 0.25 x 5 + 0.75 x (-1) = 0.5

slide-30
SLIDE 30

Markov Decision Process (MDP)

  • States: ball

table hand basket floor

  • Actions:

a) attempt b) drop c) wait

a 0.25 a 0.75 b c 5

  • 1
  • 1

h t b f

slide-31
SLIDE 31

Markov Decision Process (MDP)

  • States: ball

table hand basket floor

  • Actions:

a) attempt b) drop c) wait

a 0.25 a 0.75 b c 5

  • 1
  • 1

h t b f

slide-32
SLIDE 32

Reinforcement Learning Tools

  • Dynamic Programming
  • Monte Carlo Methods
  • Temporal Difference Learning
slide-33
SLIDE 33

Grid World

Reward: Normal move:

  • 1

Over obstacle:

  • 10

Best reward:

  • 15
slide-34
SLIDE 34

Optimal Policy

slide-35
SLIDE 35

Value Function

  • 15
  • 8
  • 14
  • 9
  • 7
  • 6
  • 1
  • 13
  • 10
  • 12
  • 11
  • 5
  • 2
  • 4
  • 3
slide-36
SLIDE 36

Initial Policy

slide-37
SLIDE 37

Policy Iteration

  • 21
  • 11
  • 22
  • 12
  • 10
  • 11
  • 1
  • 23
  • 13
  • 24
  • 14
  • 12
  • 2
  • 13
  • 3
slide-38
SLIDE 38

Policy Iteration

  • 21
  • 11
  • 22
  • 12
  • 10
  • 11
  • 1
  • 23
  • 13
  • 24
  • 14
  • 12
  • 2
  • 13
  • 3
slide-39
SLIDE 39

Policy Iteration

  • 21
  • 11
  • 22
  • 12
  • 10
  • 11
  • 1
  • 23
  • 13
  • 24
  • 14
  • 12
  • 2
  • 13
  • 3
slide-40
SLIDE 40

Policy Iteration

slide-41
SLIDE 41

Policy Iteration

  • 21
  • 11
  • 22
  • 12
  • 10
  • 11
  • 1
  • 23
  • 13
  • 15
  • 14
  • 12
  • 2
  • 4
  • 3
slide-42
SLIDE 42

Policy Iteration

  • 21
  • 11
  • 22
  • 12
  • 10
  • 11
  • 1
  • 23
  • 13
  • 15
  • 14
  • 12
  • 2
  • 4
  • 3
slide-43
SLIDE 43

Policy Iteration

  • 21
  • 11
  • 22
  • 12
  • 10
  • 11
  • 1
  • 23
  • 13
  • 15
  • 14
  • 12
  • 2
  • 4
  • 3
slide-44
SLIDE 44

Policy Iteration

slide-45
SLIDE 45

Policy Iteration

  • 15
  • 8
  • 14
  • 9
  • 7
  • 6
  • 1
  • 13
  • 10
  • 12
  • 11
  • 5
  • 2
  • 4
  • 3
slide-46
SLIDE 46

Value Iteration

slide-47
SLIDE 47

Value Iteration

  • 1
  • 1
  • 1
  • 1
  • 1
  • 1
  • 1
  • 1
  • 1
  • 1
  • 1
  • 1
  • 1
  • 1
  • 1
slide-48
SLIDE 48

Value Iteration

  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 1
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
slide-49
SLIDE 49

Value Iteration

  • 3
  • 3
  • 3
  • 3
  • 3
  • 3
  • 1
  • 3
  • 3
  • 3
  • 3
  • 3
  • 2
  • 3
  • 3
slide-50
SLIDE 50

Value Iteration

  • 15
  • 8
  • 14
  • 9
  • 7
  • 6
  • 1
  • 13
  • 10
  • 12
  • 11
  • 5
  • 2
  • 4
  • 3
slide-51
SLIDE 51

Stochastic Model

0.025 0.95 0.025

slide-52
SLIDE 52

Value Iteration

  • 19.2 -10.4
  • 18.1 -12.1
  • 9.3
  • 8.2
  • 1.5
  • 17.0 -13.6
  • 15.7 -14.7
  • 6.7
  • 2.9
  • 5.1
  • 4.0

0.95 0.025 0.025

slide-53
SLIDE 53

Value Iteration

  • 19.2 -10.4
  • 18.1 -12.1
  • 9.3
  • 8.2
  • 1.5
  • 17.0 -13.6
  • 15.7 -14.7
  • 6.7
  • 2.9
  • 5.1
  • 4.0

0.95 0.025 0.025

E.g. 13.6: 13.6 = 0.950 x 13.1 + 0.025 x 27.0 + 0.025 x 16.7 16.6 = 0.950 x 16.7 + 0.025 x 13.1 + 0.025 x 15.7

slide-54
SLIDE 54

Richard Bellman

slide-55
SLIDE 55

Bellman Equation

slide-56
SLIDE 56

Reinforcement Learning Tools

  • Dynamic Programming
  • Monte Carlo Methods
  • Temporal Difference Learning
slide-57
SLIDE 57

Monte Carlo Methods

0.025 0.95 0.025

slide-58
SLIDE 58

Monte Carlo Methods

0.95 0.025 0.025

slide-59
SLIDE 59

Monte Carlo Methods

  • 32
  • 22
  • 21
  • 10
  • 11

0.95 0.025 0.025

slide-60
SLIDE 60

Monte Carlo Methods

0.95 0.025 0.025

slide-61
SLIDE 61

Monte Carlo Methods

  • 21
  • 11
  • 10

0.95 0.025 0.025

slide-62
SLIDE 62

Monte Carlo Methods

0.95 0.025 0.025

slide-63
SLIDE 63

Monte Carlo Methods

  • 32
  • 31
  • 21
  • 10
  • 11

0.95 0.025 0.025

slide-64
SLIDE 64

Q-Value

  • 15
  • 10
  • 8
  • 20

0.95 0.025 0.025

slide-65
SLIDE 65

Bellman Equation

  • 15
  • 10
  • 8
  • 20
slide-66
SLIDE 66

Learning Rate

  • We do not replace an old Q value with a

new one.

  • We update at a designed learning rate.
  • Learning rate too small: slow to converge.
  • Learning rate too large: unstable.
  • Will Dabney PhD Thesis:

Adaptive Step-Sizes for Reinforcement Learning.

slide-67
SLIDE 67

Reinforcement Learning Tools

  • Dynamic Programming
  • Monte Carlo Methods
  • Temporal Difference Learning
slide-68
SLIDE 68

Richard Sutton

slide-69
SLIDE 69

Temporal Difference Learning

  • Dynamic Programming:

Learn a guess from other guesses (Bootstrapping).

  • Monte Carlo Methods:

Learn without knowing model.

slide-70
SLIDE 70

Temporal Difference Learning

Temporal Difference:

  • Learn a guess from other guesses

(Bootstrapping).

  • Learn without knowing model.
  • Works with longer episodes than Monte

Carlo methods.

slide-71
SLIDE 71

Temporal Difference Learning

Monte Carlo Methods:

  • First run through whole episode.
  • Update states at end.

Temporal Difference Learning:

  • Update state at each step using earlier

guesses.

slide-72
SLIDE 72

Monte Carlo Methods

  • 32
  • 31
  • 21
  • 10
  • 11

0.95 0.025 0.025

slide-73
SLIDE 73

Monte Carlo Methods

  • 32
  • 31
  • 21
  • 10
  • 11

0.95 0.025 0.025

slide-74
SLIDE 74

Temporal Difference

0.95 0.025 0.025

  • 19
  • 10
  • 22
  • 18
  • 12
slide-75
SLIDE 75

Temporal Difference

  • 23
  • 28
  • 21
  • 10
  • 11

0.95 0.025 0.025

  • 19
  • 10
  • 22
  • 18
  • 12
slide-76
SLIDE 76

Temporal Difference

  • 23
  • 28
  • 21
  • 10
  • 11

0.95 0.025 0.025

  • 19
  • 10
  • 22
  • 18
  • 12

23 = 1 + 22 28 = 10 + 18 21 = 10 + 11 11 = 1 + 10 10 = 10 + 0

slide-77
SLIDE 77

Function Approximation

  • Most problems have large state space.
  • We can generally design an approximation

for the state space.

  • Choosing the correct approximation has a

large influence on system performance.

slide-78
SLIDE 78

Mountain Car Problem

slide-79
SLIDE 79

Mountain Car Problem

  • Car cannot make it to top.
  • Can can swing back and forth to gain

momentum.

  • We know x and ẋ.
  • x and ẋ give an infinite state space.
  • Random – may get to top in 1000 steps.
  • Optimal – may get to top in 102 steps.
slide-80
SLIDE 80

Function Approximation

  • We can partition state space in 200 x 200

grid.

  • Coarse coding – different ways of

partitioning state space.

  • We can approximate V = wT f
  • E.g. f = ( x ẋ height ẋ2 )T
  • We can estimate w to solve problem.
slide-81
SLIDE 81

Problems with Reinforcement Learning

Policy sometimes gets worse:

  • Safe Reinforcement Learning (Phil Thomas)

guarantees an improved policy over the current policy. Very specific to training task:

  • Learning Parameterized Skills

Bruno Castro da Silva PhD Thesis

slide-82
SLIDE 82

Checkers

  • Arthur Samuel (IBM) 1959
slide-83
SLIDE 83

TD-Gammon

  • Neural networks and temporal difference.
  • Current programs play better than human

experts.

  • Expert work

in input selection.

slide-84
SLIDE 84

Deep Learning: Atari

  • Inputs: score and pixels.
  • Deep learning used to discover features.
  • Some games

played at super- human level.

  • Some games

played at mediocre level.