Approximate Q-Learning Update Initialize weight for each feature to - - PowerPoint PPT Presentation

Approximate Q-Learning Update

Initialize weight for each feature to 0. Every time we take an action, perform this update: The Q-value estimate for (s,a) is the weighted sum of its features:

Q(s, a) =

n

X

i=1

fi(s, a)wi

AQL Update Details

• The weighted sum of features is equivalent to a dot

product between the feature and weight vectors:

• The correction term is the same for all features.
• The correction to each feature is weighted by how

“active” that feature was.

Q(s, a) =

n

X

i=1

fi(s, a)wi = ~ w · f(s, a)

Exercise: Feature Q-Update

• Suppose PacMan is considering the up action.

Old Feature Values: wbias = 1 wghosts = -20 wfood = 2 weats = 4 Reward eating food: +10 Reward for losing:

• 500

discount: .95 learning rate: .3

Features:

• bias
• #-of-ghosts-1-step-away
• closest-food
• eats-food

Notes on Approximate Q-Learning

• Learns weights for a tiny number of features.
• Every feature’s value is update every step.
• No longer tracking values for individual (s,a) pairs.
• (s,a) value estimates are calculated from features.
• The weight update is a form of gradient descent.
• We’ve seen this before.
• We’re performing a variant of linear regression.
• Feature extraction is a type of basis change.
• We’ll see these again.

Hypothesis Spaces

PacMan x PacMan y ghost 1 x ghost 1 y ghost 1 scared ghost 2 x ghost 2 y ghost 2 scared food 1 food 2 … food 100 power-up 1 power-up 2 action bias # ghosts closest food eats food action

feature extraction approximate Q-learning linear

value ∈ R

value ∈ R

Q-learning tabular

Plusses and Minuses of Approximation

+Dramatically reduces the size of the Q-table. +States will share many features.

+Allows generalization to unvisited states. +Makes behavior more robust: making similar decisions in similar states.

+Handles continuous state spaces! −Requires feature selection (often must be done by hand). −Restricts the accuracy of the learned rewards.

−The true reward function may not be linear in the features.

Linear Regression

3/26/18

This Week: Supervised Learning

• Data consists of (input, output) pairs.
• We know the right output for each example input.

Sub-categories of supervised learning:

• Regression
• Continuous outputs
• Classification
• Discrete outputs (labels)

Linear Regression Hypothesis Space

Supervised learning

• For every input in the data set, we know the output

Regression

• Outputs are continuous
• A number, not a category label

The learned model:

• A linear function mapping input to output
• A weight for each feature (including bias)

Linear Models

In two dimensions: In d dimensions: We want to find the linear model that fits our data best. When have we seen a model like this before? f(x) = wx + b ~ x ≡      x0 x1 . . . xd      f(~ x) =      wb w0 . . . wd      ·      1 x0 . . . xd     