FrankWood Gatsby UCL - - PowerPoint PPT Presentation

• FrankWood

CedricArchambeau JanGasthaus LancelotJames YeeWhye Teh Gatsby UCL Gatsby HKUST Gatsby

• Model

– SmoothingMarkovmodelofdiscretesequences – ExtensionofhierarchicalPitmanYorprocess[Teh2006]

• Unboundeddepth(contextlength)
• Algorithmsandestimation

– Lineartimesuffix0treegraphicalmodelidentificationandconstruction – StandardChineserestaurantfranchisesampler

• Results

– Maximumcontextualinformationusedduringinference – Competitivelanguagemodellingresults

• Limitofn0gramlanguagemodelasn→∞

– SamecomputationalcostasaBayesianinterpolating50gramlanguage model

• Uses

– Anysituationinwhichalow0orderMarkovmodelofdiscrete sequencesisinsufficient – DropinreplacementforsmoothingMarkovmodel

• Name?

– ‘‘AStochasticMemoizerforSequenceData’’ →Sequence Memoizer(SM)

• Describesposteriorinference[Goodmanetal‘08]

• SequenceMarkovmodelsareusuallyconstructedbytreatinga

sequenceasasetof(exchangeable)observationsinfixed0length contexts

• acac →

     c|ao a|ca c|ac

trigram

• acac →

         a|o c|a a|c c|a

• acac →

              

• |[]

a|[] c|[] a|[] c|[]

• acac →
• a|cao

c|aca

bigram unigram 40gram Increasingcontextlength/orderofMarkovmodel Decreasingnumberofobservations Increasingnumberofconditionaldistributionstoestimate(indexedbycontext) Increasingpowerofmodel

• Example

P(xN) =

N

• i

P(xi|x, . . . xi−) ≈

N

• i

P(xi|xi−n, . . . xi−), n = 2 = P(x)P(x|x)P(x|x)P(x|x) . . . P(oacac) = P(o)P(a|o)P(c|a)P(a|c)P(c|a) = G(o)G(a)G(a)G(c)G(a)

• Discretedistribution↔ vectorofparameters
• Counting/Maximumlikelihoodestimation

– TrainingsequencexN – Predictiveinference

• Example

– Non0smoothedunigrammodel( ǫ)

G

xi

i = 1 : N

ˆ G(X = k) = ˆ πk = {k}

{}

P(Xn|x . . . xN) = ˆ G(Xn) G = [π, . . . , πK], K ∈ |Σ|

!

• Estimation
• Predictiveinference
• Priorsoverdistributions
• Neteffect

– Inferenceis“smoothed” w.r.t.uncertaintyabout unknowndistribution

• Example

– Smoothedunigram( ǫ)

xi

i = 1 : N

P(G|xn) ∝ P(xn|G)P(G) P(Xn|xn) =

• P(Xn|G)P(G|xn)dG

U

G ∼ Dirichlet(U), G ∼ PY(d, c, U)

G

"#

• Toolfortyingtogetherrelateddistributionsinhierarchicalmodels
• Measureovermeasures
• Basemeasureisthe“mean” measure
• AdistributiondrawnfromaPitmanYorprocessisrelatedtoits base

distribution

– (equalwhen =∞ or =1)

G ∼ PY(d, c, Gσ) xi ∼ G

concentration discount base distribution

E[G(dx)] = Gσ(dx)

’

$%&$

• GeneralizationoftheDirichletprocess( =0)

– Different(power0law)properties – Betterfortext[Teh,2006]andimages[Sudderth andJordan,2009]

• Posteriorpredictivedistribution
• Formsthebasisforstraightforward,simplesamplers
• Ruleforstochasticmemoization

P(XN|xN; c, d) ≈

• P(xN|G)P(G|xN; c, d)dG

=

• K

k(mk − d)(φk = XN)

c + N + c + dK c + N Gσ(XN)

• Can’t actually do this integral this way

'!

• Estimation
• Predictiveinference
• Naturallyrelateddistributionstied

together

• Neteffect

– Observationsinonecontextaffect inferenceinothercontext. – Statisticalstrengthissharedbetween similarcontexts

• Example

– Smoothingbi0gram( ǫ ∈ Σ)

xj xi

U

Θ = {G, G, G}, ( = σ() = σ() P(Θ|xN) ∝ P(xN|Θ)P(Θ)

P(XN|xN) =

• P(XN|Θ)P(Θ|xN)dΘ

G

j = 1 : N i = 1 : N

G

G

Gthe United States ∼ PY(d, c, GUnited States)

)'$&$"

Conditional Distributions Posterior Predictive Probabilities Observations

U

G G

G G

*$$+,

Conditional Distributions Posterior Predictive Probabilities Observations CPU

U

G G

G G

*$$+,

Conditional Distributions Posterior Predictive Probabilities Observations CPU CPU

U

G G

G G

'$&$"

• Sharestatisticalstrengthbetween

sequentiallyrelatedpredictive conditionaldistributions

– Estimatesofhighlyspecific conditionaldistributions – Arecoupledwithothersthatare related – Throughasinglecommon,more0 generalsharedancestor

• Correspondsintuitivelytoback0off

G

G

G

G

G

G

• G

G

G

G

G

G

G

G

G

G G G

'$&$

• Bayesiangeneralizationofsmoothingn0gramMarkovmodel
• Languagemodel:outperformsinterpolatedKneser0Ney(KN)smoothing
• Efficientinferencealgorithmsexist

– [Goldwateretal’05;Teh,’06;Teh,Kurihara,Welling,’08]

• Sharingbetweencontextsthatdifferinmostdistantsymbolonly
• Finitedepth

G | d, U ∼ PY(d, 0, U) G | d||, Gσ ∼ PY(d||, 0, Gσ) xi | i− = ∼ G i = 1, . . . , T ∀ ∈ Σn−

’’

"

• Asequencecanbecharacterizedbyasetofsingle
• bservationsinuniquecontextsofgrowinglength

Increasingcontextlength Alwaysasingleobservation Foreshadowing:allsuffixesofthestring“cacao”

• acac →

              

• |[]

a|o c|ao a|cao c|acao

• -.%’’
• Example
• Smoothingessential

– Onlyoneobservationineachcontext!

• Solution

– HierarchicalsharingalaHPYP

P(xN) =

N

• i

P(xi|x, . . . xi−) = P(x)P(x|x)P(x|x, x)P(x|x, . . . x) . . . P(oacac) = P(o)P(a|o)P(c|oa)P(a|oac)P(c|oaca)

• EliminatesMarkovorderselection
• Alwaysusesfullcontextwhenmakingpredictions
• Lineartime,linearspace(inlengthofobservationsequence)graphicalmodel

identification

• Performanceislimitofn0gramasn→∞
• Sameorlessoverallcostas50graminterpolatingKneser Ney

G | d, U ∼ PY(d, 0, U) G | d||, Gσ ∼ PY(d||, 0, Gσ) xi | i− = ∼ G i = 1, . . . , T ∀ ∈ Σ

/,

Observations

• acac →

              

• |[]

a|o c|ao a|cao c|acao

Latent conditional distributions with Pitman Yor priors / stochastic memoizers

00

• acac →

              

• |[]

a|o c|ao a|cao c|acao

All suffixes of the string “cacao”

00

• Deterministicfiniteautomatathatrecognizesall

suffixesofaninputstring.

• RequiresO(N2)timeandspacetobuildandstore

[Ukkonen,95]

• Toointensiveforanypracticalsequencemodelling

application.

00

• Deterministicfiniteautomatathatrecognizesall

suffixesofaninputstring

• Usespathcompressiontoreducestorageand

constructioncomputationalcomplexity.

• RequiresonlyO(N)timeandspacetobuildandstore

[Ukkonen,95]

• Practicalforlargescalesequencemodelling

applications

00

00

/,10

• Thisisagraphicalmodeltransformationunderthe

covers.

• Thesecompressedpathsrequirebeingableto

analyticallymarginalizeoutnodesfromthegraphical model

• Theresultofthismarginalizationcanbethoughtofas

providingadifferentsetofcachingrulestomemoizers

• nthepath0compressededges

• Theorem1:Coagulation

If G|G ∼ PY(d, 0, G) and G|G ∼ PY(d, 0, G) then G|G ∼ PY(dd, 0, G) with G marginalized out.

[Pitman ’99; Ho, James, Lau ’06; W., Archambeau, Gasthaus, James, Teh ‘09]

• →

/,

/,

/,1

• Givenasingleinputsequence

– Ukkonen’s lineartimesuffixtreeconstructionalgorithmis runonitsreversetoproduceaprefixtree – Thisidentifiesthenodesinthegraphicalmodelweneedto represent – Thetreeistraversedandpathcompressedparametersfor thePitmanYorprocessesareassignedtoeachremaining PitmanYorprocess

.1/,

. 2%

!.%/$0

HPYP exceeds SM computational complexity

*

[Mnih&Hinton,2009] 112.1 [Bengioetal.,2003] 109.0 40gramModifiedKneser0Ney[Teh,2006] 102.4 40gramHPYP[Teh,2006] 101.9 SequenceMemoizer(SM) SequenceMemoizer(SM) SequenceMemoizer(SM) SequenceMemoizer(SM) 96.9 96.9 96.9 96.9

APNewsTestPerplexity APNewsTestPerplexity APNewsTestPerplexity APNewsTestPerplexity

• TheSequenceMemoizerisadeep(unbounded)smoothing

Markovmodel

• Itcanbeusedtolearnajointdistributionoverdiscrete

sequencesintimeandspacelinearinthelengthofasingle

• bservationsequence
• Itisequivalenttoasmoothing∞0grambutcostsnomoreto

computethana50gram