Reinforcement Learning A (almost) quick (and very incomplete) - - PowerPoint PPT Presentation

Reinforcement Learning

A (almost)quick(and very incomplete) introduction

Slides from David Silver, Dan Klein, Mausam, Dan Weld

Reinforcement Learning

At each time step t:

• Agent executes an action At
• Environment emits a reward Rt
• Agent transitions to state St

Rat Example

Rat Example

• What if agent state = last 3 items in sequence?

Rat Example

• What if agent state = last 3 items in sequence?
• What if agent state = counts for lights, bells and levers?

Rat Example

• What if agent state = last 3 items in sequence?
• What if agent state = counts for lights, bells and levers?
• What if agent state = complete sequence?

Major Components of RL

An RL agent may include one or more of these components:

• Policy: agent’s behaviour function
• Value function: how good is each state and/or action
• Model: agent’s representation of the environment

Policy

• A policy is the agent’s behaviour
• It is a map from state to action
• Deterministic policy: a = π(s)
• Stochastic policy: π(a|s) = P[At = a|St = s]

Value function

• Value function is a prediction of future reward
• Used to evaluate the goodness/badness of states…
• …and therefore to select between actions

Model

• A model predicts what the environment will do next
• It predicts the next state…
• …and predicts the next (immediate) reward

Dimensions of RL

Model-based vs. Model-free

• Model-based: Have/learn action

models (i.e. transition probabilities.

• Uses Dynamic Programming
• Model-free: Skip them and directly

learn what action to do when (without necessarily finding out the exact model of the action)

• e.g. Q-learning

On Policy vs. Off Policy

• On Policy: Makes estimates based on a

policy, and improves it based on estimates.

• Learning on the job.
• e.g. SARSA
• Off Policy: Learn a policy while following

another (or re-using experience from old policy).

• Looking over someone's shoulder
• e.g. Q-learning

Markov Decision Process

• Set of states S = {si}
• Set of actions for each state A(s) = {asi} (often independent of state)
• Transition model T(s -> s’ | a) = Pr(s’ | a, s)
• Reward model R(s, a, s’)
• Discount factor γ

MDP = <S, A, T, R, γ>

Bellman Equation for Value Function

Bellman Equation for Action-Value Function

Q vs V

Exploration vs Exploitation

• Restaurant Selection
• Exploitation: Go to your favourite restaurant
• Exploration: Try a new restaurant
• Online Banner Advertisements
• Exploitation: Show the most successful advert
• Exploration: Show a different advert
• Oil Drilling
• Exploitation: Drill at the best known location
• Exploration: Drill at a new location
• Game Playing
• Exploitation: Play the move you believe is best
• Exploration: Play an experimental move

ε-Greedy solution

• Simplest idea for ensuring continual exploration
• All m actions are tried with non-zero probability
• With probability 1 − ε choose the greedy action
• With probability ε choose an action at random

Off Policy Learning

• Evaluate target policy π(a|s) to compute vπ(s) or qπ(s,a) while following behaviour

policy μ(a|s) {s1,a1,r2,...,sT} ∼ μ

• Why is this important?
• Learn from observing humans or other agents
• Re-use experience generated from old policies π1, π2, ..., πt−1
• Learn about optimal policy while following exploratory policy
• Learn about multiple policies while following one policy

Q - Learning

• We now consider off-policy learning of action-values Q(s,a)
• Next action is chosen using behaviour policy At+1 ∼ μ(·|St)
• But we consider alternative successor action A′ ∼ π(·|St)
• And update Q(St,At) towards value of alternative action

Q - Learning

• We now allow both behaviour and target policies to improve
• The target policy π is greedy w.r.t. Q(s,a)
• The behaviour policy μ is e.g. ε-greedy w.r.t. Q(s,a)
• The Q-learning target then simplifies:

Q - Learning

Q - Learning

Deep RL

• We seek a single agent which can solve any human-level task
• RL defines the objective
• DL gives the mechanism
• RL + DL = general intelligence (David Silver)

Function Approximators

Deep Q-Networks

• Q Learning diverges using neural networks due to:
• Correlations between samples
• Non-stationary targets

Solution: Experience Replay

• Fancy biological analogy
• In reality, quite simple

Solution: Experience Replay

Improving Information Extraction by Acquiring External Evidence with Reinforcement Learning

Karthik Narasimhan, Adam Yala, Regina Barzilay CSAIL, MIT

Slides from Karthik Narasimhan

Why try to reason, when someone else can do it for you

Doubts*

• Algo 1 line# 19. The process should end when "d" == "end_episode" and not q. [Prachi]

Error.

• The dimension of the match vector should be equivalent to the number of columns to ve
• extracted. But Fig 3 has twice the number of dim. [Prachi] Error.
• Is RL the best approach. [Non believers].
• Experience Replay [Anshul]. Hope it is clear now.
• Why is RL-extract better than meta classifier? Explanation provided in paper about "long

tail of noisy, irrelevant documents" is unclear. [Yash]

• The meta-classifier should also cut off at top-20 results per search like the RL system to

be completely fair. [Anshul]

* most mean questions

Discussions

• Experiments
• People are happy!
• Queries
• Cluster documents and learn queries [Yashoteja]
• Many other query formulations [Surag (lowest confidence entity), Barun (LSTM), Gagan (highest confidence entity), DineshR]
• Fixed set of queries [Akshay]
• Simplicity. Search engines are robust.
• Reliance on News articles {Gagan]
• Where else would you get News from?
• Domain limitations
• Too narrow [Barun, Himanshu]. Domain specific [Happy]. Small ontology [Akshay]
• It is not Open IE. It is task specific. Can be applied to any domain.
• Better meta-classifiers [Surag]
• Effect of more sophisticated RL algorithms (A3C, TRPO) [esp. if increasing action space by LSTM queries], and their effect on performance and training time.