t.IPCyilxi.BE o o.E iTNCyilBotp Xi NCpl0 o2I P43 Here dumb 13 - - PDF document

SLIDE 1

Lampley

for

estimation

Idea

Parameter

estimation

via

Simulation

y

Pot P X

t E

ENNIO

02

yn N pot P X OI

Assume

known

P

N

21

Prior

distribution

• ver 13

Suppose

we observe data

D

x

y

10431137

7

Bayed

PCDlB P

PCD

Note

that

D

is

• bserved

P DIB

• t.IPCyilxi.BE

iTNCyilBotp Xi

• .E

P43

NCpl0 o2I

Here

is

a

dumb

algorithm

to

estimate

13

SLIDE 2

For

T

steps A

type

• f

im

smw3

n

e

Samples

to approx

Z

wt

weighted average

132

2 wipe

• ur

s

a

Posterior

• ver

IT

B

Weights

here

are

Unnormalized

hence

2

Note

Can

use

same

Strategy

to

predict Ji

How

Mere

generally Assume

a

data generating distribution

f

W

parameters

and

a

prior

it

fLo TLo do

I

f Os

Os

T

Might be

intractable

approximate by

Sampling

Monte Carlo

Integration

TIO

where

z

Ws

and

Ws

is

an

unnormalized

joint

PCD Os

Obviously

IT

is

critical here

SLIDE 3

The

key

trick

PCOID

PCDlo

Pco

et PLD O

PCD

See

Juptyer

Notebook

For

More

Complex

cases

simple

importance weighting

is

not

going

to

fly

It

would

take

forever

to

find

good

Can

we

be

smarter

about

picking

05

Markov Chan Monte Carlo

MCMC

is

a

method

Hai

tries

to

simulate

draws

from

a

dust

• f

Interest

PC

St

I 01

Os

1

O

Os

Transition probability from current

parameters

Metropolis

Hastings

is

a

particular

version

• f

MCMC

Basically

Start

somewhere

Then

Make

a

proposal

Ott

that

you

accept

• r

reject

with

some

probability

SLIDE 4

r

e

• proba

I y

The

accept

p

should

be

high

if

It is

a

better

fit

Gibbs

Sampling

is

a

simple

recipe where

we

update

a

particular parameter

0g

Conditioned

• n

all

• thers

Gibbs

Sample

Initialize

91 for

T

steps

OI

PCQ.to

• OIi

I

3

c

OE

PCO 10 70

OY

2

s

• Monto

OE

• h

Return Ot

Content

below

derived from

Jordan Boyd Graber

SLIDE 5

Coming

back

to

LDA

13k

Dirichlet M

the

Wd.in Zd n

Discrete43zd.n

For

LDA

we

will

estimate

The

Probability

• f

a

specific

word's

topic

Assignment

Conditioned

• n

all

• ther

assignments

Pkd n

K 4d

n P

K d

ar

all other word topic

assignments

P Zd n

k

2 d.nl P

Q

X

PCE d nlw

K

X

SLIDE 6

Requires

Integrating

• ut

and

13

This

is

a

bit

hairy

but

we

end

with

V

1

Nd K

t KK

kind n

d n

E

nd

taxi

the w't

How much

This

doc

How much Topic

K

likes

This

topic likes

this

word

Count

• f

topic

k

in

duc

d

Court

• f

topic

k

using

word

Wd n

Note

Given

PCad n

for

all

2dm

we

can

derive

13