CS325 Artificial Intelligence Ch. 15,20 Hidden Markov Models and - - PowerPoint PPT Presentation

CS325 Artificial Intelligence

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Cengiz Günay, Emory Univ.

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 1 / 21

Get Rich Fast!

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 2 / 21

Get Rich Fast!

Or go bankrupt?

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 2 / 21

Get Rich Fast!

Or go bankrupt? So, how can we predict time-series data?

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 2 / 21

Hidden Markov Models

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 3 / 21

Hidden Markov Models

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 3 / 21

Entry/Exit Surveys

Exit survey: Reinforcement Learning

What’s the difference between MDPs and Reinforcement Learning? What is the dilemma between exploration and exploitation?

Entry survey: Hidden Markov Models (0.25 points of final grade)

What previous algorithm would you use for time series prediction? What time series do you wish you could predict?

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 4 / 21

Time Series Prediction?

Have we done this before?

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 5 / 21

Time Series Prediction?

Have we done this before? Belief states with action schemas?

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 5 / 21

Time Series Prediction?

Have we done this before? Belief states with action schemas?

Not for continuous variables Goal-based

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 5 / 21

Time Series Prediction?

Have we done this before? Belief states with action schemas?

Not for continuous variables Goal-based

MDPs and RL?

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 5 / 21

Time Series Prediction?

Have we done this before? Belief states with action schemas?

Not for continuous variables Goal-based

MDPs and RL?

Goal-based No time sequence

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 5 / 21

Time Series Prediction with Hidden Markov Models (HMMs)

• Dr. Thrun is very happy – HMMs are his specialty.

HMMs:

analyze & predict time series data can deal with noisy sensors

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 6 / 21

Time Series Prediction with Hidden Markov Models (HMMs)

• Dr. Thrun is very happy – HMMs are his specialty.

HMMs:

analyze & predict time series data can deal with noisy sensors

Example domains: finance (get rich fast!) robotics medical speech and language Alternatives: Recurrent neural networks (not probabilistic)

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 6 / 21

What are HMMs?

Markov chain: Hidden states : S1 → S2 → · · · → Sn ↓ ↓ Measurements : Z1 · · · Zn

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 7 / 21

What are HMMs?

Markov chain: Hidden states : S1 → S2 → · · · → Sn ↓ ↓ Measurements : Z1 · · · Zn It’s essentially a Bayes Net!

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 7 / 21

What are HMMs?

Markov chain: Hidden states : S1 → S2 → · · · → Sn ↓ ↓ Measurements : Z1 · · · Zn It’s essentially a Bayes Net! Implementations: Kalman Filter (see Ch. 15) Particle Filter

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 7 / 21

Video: Lost Robots, Speech Recognition

Future Prediction with Markov Chains

Is tomorrow going to be Rainy or Sunny? R S 0.4 0.2 0.8 0.6

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 9 / 21

Future Prediction with Markov Chains

Is tomorrow going to be Rainy or Sunny? R S 0.4 0.2 0.8 0.6 Start with “today is rainy”: P(R0) = 1, then P(S0) = 0

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 9 / 21

Future Prediction with Markov Chains

Is tomorrow going to be Rainy or Sunny? R S 0.4 0.2 0.8 0.6 Start with “today is rainy”: P(R0) = 1, then P(S0) = 0 What’s P(S1) = ? P(S2) = ? P(S3) = ?

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 9 / 21

Future Prediction with Markov Chains

Is tomorrow going to be Rainy or Sunny? R S 0.4 0.2 0.8 0.6 Start with “today is rainy”: P(R0) = 1, then P(S0) = 0 What’s P(S1) = 0.4 P(S2) = ? P(S3) = ?

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 9 / 21

Future Prediction with Markov Chains

Is tomorrow going to be Rainy or Sunny? R S 0.4 0.2 0.8 0.6 Start with “today is rainy”: P(R0) = 1, then P(S0) = 0 What’s P(S1) = 0.4 P(S2) = 0.56 P(S3) = ?

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 9 / 21

Future Prediction with Markov Chains

Is tomorrow going to be Rainy or Sunny? R S 0.4 0.2 0.8 0.6 Start with “today is rainy”: P(R0) = 1, then P(S0) = 0 What’s P(S1) = 0.4 P(S2) = 0.56 P(S3) = 0.624

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 9 / 21

Future Prediction with Markov Chains

Is tomorrow going to be Rainy or Sunny? R S 0.4 0.2 0.8 0.6 Start with “today is rainy”: P(R0) = 1, then P(S0) = 0 What’s P(S1) = 0.4 P(S2) = 0.56 P(S3) = 0.624 P(St+1) = 0.4 × P(Rt) + 0.8 × P(St)

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 9 / 21

Back to the Future?

How far can we see into the future? P(A∞) =? Until it reaches a stationary state (or limit cycle)

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 10 / 21

Back to the Future?

How far can we see into the future? P(A∞) =? Until it reaches a stationary state (or limit cycle) Use calculus: lim

t→∞ P(At+1) = P(At)

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 10 / 21

Back to the Future?

How far can we see into the future? P(A∞) =? Until it reaches a stationary state (or limit cycle) Use calculus: lim

t→∞ P(At+1) = P(At)

R S 0.4 0.2 0.8 0.6 P(S∞) = ?

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 10 / 21

Back to the Future?

How far can we see into the future? P(A∞) =? Until it reaches a stationary state (or limit cycle) Use calculus: lim

t→∞ P(At+1) = P(At)

R S 0.4 0.2 0.8 0.6 P(S∞) = 2/3

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 10 / 21

Back to the Future?

How far can we see into the future? P(A∞) =? Until it reaches a stationary state (or limit cycle) Use calculus: lim

t→∞ P(At+1) = P(At)

R S 0.4 0.2 0.8 0.6 P(S∞) = 2/3 lim

t→∞ P(St+1) = 0.4 × P(Rt) + 0.8 × P(St),

• subst. x = P(St+1) = P(St) = 1 − P(Rt) = · · ·

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 10 / 21

And How Do We Get The Transition Probabilities?

R S ? ? ? ? Observed sequence in Atlanta : RRSRRRSR Use Maximum Likelihood

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 11 / 21

And How Do We Get The Transition Probabilities?

R S ? ? ? ? Observed sequence in Atlanta : RRSRRRSR Use Maximum Likelihood P(S|S) = ?

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 11 / 21

And How Do We Get The Transition Probabilities?

R S ? ? ? ? Observed sequence in Atlanta : RRSRRRSR Use Maximum Likelihood P(S|S) =

• bserved transitions

total transitions from S = 0 2

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 11 / 21

And How Do We Get The Transition Probabilities?

R S ? ? ? ? Observed sequence in Atlanta : RRSRRRSR Use Maximum Likelihood P(S|S) =

• bserved transitions

total transitions from S = 0 2 P(R|S) = 2/2, P(S|R) = 2/5, P(R|R) = 3/5

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 11 / 21

And How Do We Get The Transition Probabilities?

R S ? ? ? ? Observed sequence in Atlanta : RRSRRRSR Use Maximum Likelihood P(S|S) =

• bserved transitions

total transitions from S = 0 2 P(R|S) = 2/2, P(S|R) = 2/5, P(R|R) = 3/5 Edge effects? P(S|S) = 0?

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 11 / 21

Overcoming Overfitting: Remember Laplacian Smoothing?

Observed sequence in Atlanta : RRSRRRSR Laplacian smoothing K = 1 P(S|S) =

• bserved transitions

total transitions from S

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 12 / 21

Overcoming Overfitting: Remember Laplacian Smoothing?

Observed sequence in Atlanta : RRSRRRSR Laplacian smoothing K = 1 P(S|S) =

• bserved transitions + K

total transitions from S + N

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 12 / 21

Overcoming Overfitting: Remember Laplacian Smoothing?

Observed sequence in Atlanta : RRSRRRSR Laplacian smoothing K = 1 P(S|S) =

• bserved transitions + K

total transitions from S + N = 0 + 1 2 + 2 K, N selected such that 0 ≤ P ≤ 1.

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 12 / 21

Where is Markov Hidden?

R H G S H G 0.4 0.2 0.8 0.6 0.4 0.6 0.9 0.1 Hidden: rainy or sunny Observe: happy or grumpy

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 13 / 21

Where is Markov Hidden?

R H G S H G 0.4 0.2 0.8 0.6 0.4 0.6 0.9 0.1 Hidden: rainy or sunny Observe: happy or grumpy Initial conditions P(R0) = 1/2, P(S0) = 1/2 P(S1|H1) = ?

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 13 / 21

Where is Markov Hidden?

R H G S H G 0.4 0.2 0.8 0.6 0.4 0.6 0.9 0.1 Hidden: rainy or sunny Observe: happy or grumpy Initial conditions P(R0) = 1/2, P(S0) = 1/2 P(S1|H1) = P(H1|S1)P(S1) P(H1) Bayes rule!

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 13 / 21

Congrats, Done with Prediction and State Estimation

What else can we do with HMMs? Localization of the lost robot Blindfolded person

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 14 / 21

Congrats, Done with Prediction and State Estimation

What else can we do with HMMs? Localization of the lost robot Blindfolded person Video: Robot localization

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 14 / 21

HMMs, Formally

Hidden states : S1 → S2 → · · · → Sn ↓ ↓ Measurements : Z1 · · · Zn Question: P(S1|S2) ⊥ P(Sn|S2) ?

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 15 / 21

HMMs, Formally

Hidden states : S1 → S2 → · · · → Sn ↓ ↓ Measurements : Z1 · · · Zn Question: P(S1|S2) ⊥ P(Sn|S2) Yes! Past and future are independent.

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 15 / 21

HMMs, Formally

Hidden states : S1 → S2 → · · · → Sn ↓ ↓ Measurements : Z1 · · · Zn Question: P(S1|S2) ⊥ P(Sn|S2) Yes! Past and future are independent. HMM equations: State estimation: P(S1|Z1) = αP(Z1|S1)P(S1)

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 15 / 21

HMMs, Formally

Hidden states : S1 → S2 → · · · → Sn ↓ ↓ Measurements : Z1 · · · Zn Question: P(S1|S2) ⊥ P(Sn|S2) Yes! Past and future are independent. HMM equations: State estimation: P(S1|Z1) = αP(Z1|S1)P(S1) Prediction: P(S2) =

S1 P(S2|S1)P(S1)

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 15 / 21

HMMs for Localization Example

Robot knows map, but not location: use multiplication and convolution

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 16 / 21

HMMs for Localization Example

Robot knows map, but not location: use multiplication and convolution

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 16 / 21

HMMs for Localization Example

Robot knows map, but not location: use multiplication and convolution

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 16 / 21

HMMs for Localization Example

Robot knows map, but not location: use multiplication and convolution

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 16 / 21

HMMs for Localization Example

Robot knows map, but not location: use multiplication and convolution

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 16 / 21

HMMs for Localization Example

Robot knows map, but not location: use multiplication and convolution

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 16 / 21

Particle Filters: For Clean Water?

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 17 / 21

Particle Filters: For Clean Water?

Nope, but same idea. Video: Robot localization with particle filters

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 17 / 21

Particle Filters: For Clean Water?

Nope, but same idea. Video: Robot localization with particle filters Belief representation Points are hypotheses Particles survive if consistent with measurements Easy implementation!

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 17 / 21

Localization with Particle Filters

Particle filtering: weights show likelihood; pick particles, shift, and repeat

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 18 / 21

Localization with Particle Filters

Particle filtering: weights show likelihood; pick particles, shift, and repeat

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 18 / 21

Localization with Particle Filters

Particle filtering: weights show likelihood; pick particles, shift, and repeat

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 18 / 21

Localization with Particle Filters

Particle filtering: weights show likelihood; pick particles, shift, and repeat

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 18 / 21

Localization with Particle Filters

Particle filtering: weights show likelihood; pick particles, shift, and repeat

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 18 / 21

Localization with Particle Filters

Particle filtering: weights show likelihood; pick particles, shift, and repeat Continuous space! Computational resources used efficiently!

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 18 / 21

Particle Filter Algorithm

S: Particle set {< x, w >, . . .}, U: Control vector (e.g., map), Z: Measure vector S′ = Ø, η = 0 For i=1. . . n sample j ∼ {w} w/ replacement x′ ∼ P(x′|U, Sj) w′ = P(Z|x′) η = η + w′ S′ = S′ ∪ {< x′, w′ >} End For i=1. . . n // Normalization step wi = 1

ηwi

End

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 19 / 21

Particle Filter Pros & Cons

In general works well! Stanley uses it for navigation. Pros: Easy to implement Efficient Complex and changing environments in robotics Cons: Dimensionality problem: need many particles Problems with degenerate conditions (adding noise may help)

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 20 / 21

Time Series Prediction Conclusion

Particle filtering: Most widely used algorithm! Can handle time series and uncertainty Other application areas: Financial prediction Weather Alternative methods: Kalman filters Recurrent neural nets

Günay

• Ch. 15,20 – Hidden Markov Models and Particle Filtering

Spring 2013 21 / 21