CS475/CS675 Lecture 24: July 21, 2016 Open problems CS475/CS675 - - PowerPoint PPT Presentation

▶

Aug 03, 2023 12 likes •185 views

CS475/CS675 Lecture 24: July 21, 2016 Open problems CS475/CS675 (c) 2016 P. Poupart 1 Two Open Problems Kernel methods: how to solve linear systems of equation in less than cubic time Markov decision processes: how to evaluate

SLIDE 1

CS475/CS675 Lecture 24: July 21, 2016

Open problems

SLIDE 2

Two Open Problems

Kernel methods: how to solve linear systems of

equation in less than cubic time

Markov decision processes: how to evaluate

factored policies in less than exponential time

SLIDE 3

Kernel Methods

Class of non‐parametric Machine Learning

techniques that scale with the amount of data

Examples:

– Gaussian processes – Support vector machines – Kernel logistic regression – Kernel principal component analysis – Kernel perceptron

SLIDE 4

Gaussian Process

Quick recall:

– Non‐parametric regression – Infinite dimensional Gaussian

Picture:

SLIDE 5

Kernel

Covariance function is a kernel function

,

Where

is the feature function that defines the kernel

Popular kernels with infinitely many features:

Gaussian kernel:

Exponential kernel:
CS475/CS675 (c) 2016 P. Poupart

SLIDE 6

Common problem

In all kernel methods, a linear system of equations

must be solved:

is an instantiation of the kernel function called the

Gram matrix, i.e.

is a constant positive scalar
is constant vector
is the vector of unknowns

SLIDE 7

Challenge

is an

matrix where is the number of data points in the dataset

Linear system takes

time to solve

This does not scale to large datasets, i.e.,

millions or billions of data points.

How can we reduce the time to
r less?

SLIDE 8

Properties

Gram matrix

is

– Symmetric – Positive semi‐definite – We also know the feature function that is used to create

Can you exploit those properties to reduce

the solution complexity to

r less?

SLIDE 9

Markov Decision Processes

Popular model in Operations Research and Artificial

Intelligence for decision‐theoretic planning

Agent Environment

State Reward Action s0 s1 s2 r0 a0 a1 r1 r2 a2 …

SLIDE 10

Markov Decision Processes

Formally:

Set of states , set of actions , discount ∈ 0,1 Transition function , , Pr |, Reward function , ∈ s0 s1 s2 s3 s4 a0 a1 a2 a3 r1 r2 r3 r4

SLIDE 11

Policy

Policy

(mapping from states to actions)

be the number of states

Transition matrix:

(

)

Reward vector:

(

)

SLIDE 12

Value Function

Value
f a policy

at state :

R s ∑ Pr ,

Pr , ∑ Pr ,

Pr ,

SLIDE 13

Bellman’s Equation

Recursive formula:

R s Pr ,

Matrix form:
Solution: system of linear equation

SLIDE 14

Problem

be the number of states

Transition matrix

Time

which is prohibitive for large state

spaces

SLIDE 15

Factored MDP

be the number of binary features

Each state corresponds to all combinations of binary

features

This yields

states

Time

which is exponential in the number of

features

Challenge: can we reduce the solution to be

polynomial in ?

SLIDE 16

Factored MDP

Factored transition matrix
Additive reward function
CS475/CS675 (c) 2016 P. Poupart

SLIDE 17

Properties

Factored MDP

– Rows of

sum to 1

– Largest eigenvalue of

is 1

–

is factored and is additive

Can you exploit those properties to reduce

CS475/CS675 Lecture 24: July 21, 2016

Open problems

Two Open Problems

equation in less than cubic time

factored policies in less than exponential time

Kernel Methods

techniques that scale with the amount of data

– Gaussian processes – Support vector machines – Kernel logistic regression – Kernel principal component analysis – Kernel perceptron

Gaussian Process

– Non‐parametric regression – Infinite dimensional Gaussian

Kernel

,

is the feature function that defines the kernel

Gaussian kernel:

Common problem

must be solved:

Gram matrix, i.e.

Challenge

matrix where is the number of data points in the dataset

time to solve

millions or billions of data points.

Properties

is

– Symmetric – Positive semi‐definite – We also know the feature function that is used to create

the solution complexity to

Markov Decision Processes

Intelligence for decision‐theoretic planning

Agent Environment

Markov Decision Processes

Formally:

Set of states , set of actions , discount ∈ 0,1 Transition function , , Pr |, Reward function , ∈ s0 s1 s2 s3 s4 a0 a1 a2 a3 r1 r2 r3 r4

Policy

(mapping from states to actions)

be the number of states

)

)

Value Function

at state :

Bellman’s Equation

Problem

be the number of states

spaces

Factored MDP

be the number of binary features

features

features

polynomial in ?

Factored MDP

Properties

– Rows of

– Largest eigenvalue of

–

the time complexity to be polynomial in ?