161-17 Stochastic Variational Lecture Inference I Kaushal - PowerPoint PPT Presentation

161-17 Stochastic Variational Lecture Inference I Kaushal Panchi Scribes : Jay DeYoung

Recap Vaniatianal Inference : Approximate Goal Posterior : 17,13 ) pH )p( 137 Generative Model 13 ) pc plx × ,z : = , Approx Variational pit ) 1×7 7 ;¢)q( p( 7,13 qc : I ( ELBO ) Lower Bound Evidence Objective : :O )[ lofty , 7,13 ) pcx , ] LC 10,9 ) Etqn = qcp :O , , . , , KL( qczsqlqlpsi ) 11pA ,pl×)) log × ) pl = -

Diniohlet Allocation Model Latent Generative : Generative model : Pu Dirichetlw ) on Qd ~ Bh Yan Oa Dirichlet ( x ) K ~ Discrete ( Od ) % 2- dn~ Zan Discrete ( Pa ) Na 7dn=h ydnl ~ D = ( l§ ) ) § Xdlttd , p( ,p)p(7d p ) p ( 7 x .pl/3u)p(7did)=/dOdp(7d1Od)p(0dj4 , , . )

Parameters Global LDA Local us Generative Model 13 ) , pspttsplp pix pix ) 17 ,z : = , Approx :D ) Variational pet , 131×7 ) qcp get :p : = = ( ¥ ) ) .pl/3u ) , B) pcxdltd pltd p ) pcx 7 , , , . - ( % old ) ) II. :D ! alpa 917419113 ) 917ns - - - , Dui local parameters global old parameters i d ) all ) ( only doc does depend depend I on an

Variational Stochastic Inference Wikipedia Problem 5Mt If has entries : we . wanted VBEM updates then to do need we 114,9 ) £19,7 I ) of = arggnax = angqmax Now full each of updates these requires pass a - the 5Mt documents over I Optimize Solution stochastic with ; gradient descent

Gradient Stochastic Descent E IT get t Optimize of gradient Idea gradient with estimate objective noisy i " It I " LID ) = , I I Approximation Step sire of the Requirement Gradient I estimate should be : unbiased Dfid , LCD ) I ) = Robbins - Monro Requirement 2 conditions ; as as / Finite I [ ( Int ' ft ft Los = as mean variance displacement ) displacement t ) te in = , ,

Stochastic Variation Inference al Approximation : batch Compute for of ELBO does £17 LC 4,9 ) ) = myax , ] an ;¢,qµa[ by # PgYhIIhT$⇒ my = ' [ 68 qq.FI 7 app ) plxa § , # 91Pa ; 9) qaa = mgay 1 log , ] :D # qcp , , , - q( p

Stochastic Variation Inference al Approximation : batch Compute for of ELBO docs ' [ 68 qq.FI Zapp ) { ( mgay plxa 217 , # ) ; 9) 91ps ' = qaa , ] ) , , , I log I :D # "→"9(7bi¢b qcp - q( p " } ) xb~ ( { Choose batch Uniform ' of does x ,x : ... , ( 2^19 ) , llogp ) ] § my year E , 'Yg ; 9(7si¢b µ a , ) ) ) , ,[ logqcp :o) T - ( End Assume ' . ' we can this with VB do

Stochastic Inference Variational For TL DR conditionally : exponential 's conjugate models family compute natural gradient we a can i. -7 ] LCD ) Ing I kit Eq = - , , , Efnglx d) D ) ( ' tea at d , 7 ,7a ) + = , , a [ Sufficient statistics This yields updates gradient It - ft ) i ft Ega ; Mg " it 'd ) It It - ' 'll gtv , fly . ) + = =

Xz Iz Natural Gradients Invariance Coordinate : Suppose have Example equivalent two : we coordinates sets of × × , , ~ ~ ×2 - - × = = 62 1 6 , ~ X x . . - ( ¥i+¥ , ) Kite Llx LCI ) ,xd= ,I.l= , . ,

Gradient Values Change Coordinates with E I I Xz I = , 6 , I • I / = , % 62 x . - ( ¥i + fig ) Kite L( LCI ) x. I ,Ii x. = = . , = 's 2£ 2£ off ¥ ¥ = = - . , 2 6,2 × , , , , ¥ It zxi I Off 2g = = = . . 2×~ 622 , ,

Gradients Invariant Coordinate - Idea Can define notice of steepest descent : we a ? transformations that coordinate Variate under is in ÷¥÷÷÷÷s P×LCx ) SEZ dxt t dxtdx DX anguish 5. = . dx L PIL d dxt DET 0×1 = = Matrix Jacobian partial of derivatives : ⇐ i :÷÷÷÷÷l

Gradients Invariant Coordinate - Observation and gradients : Differentials be transformed the rule chain using can = § J de dxi j de = ; JTJ dxtdx DE DET = 2x Ex f Txt . I = . Qi ; , ; JT Qi Tx =

. ya Gradients Invariant Coordinate - Polar Example coordinates ; .int Xz J t it " Cos ' 1%1=1 : ÷ yo d#TJd i , . O Xz r sin - - Sino & r distance cos G metric dx ? dxtdx dx ' = = + , ) I " , ] I day ) do ! - dr [ on ' dr ? do = n = t

Gradients Invariant Coordinate - Assumption KIKI along de points : (0×214)+(0×141) DE LCE ) 0×1×1 Ifl dx JT x ) = = 't Tx fix DILE DELK J ) ) DE , = = (0×-2679554%4×7) de xt ELK , = = COILED 'T0eLM ) =/

Gradients Invariant Coordinate - a' ET Solution Define natural gradient : By LCE I " ( ELLI JTJ ) LIE T , ) DE = = 8×11×1 7×11×1 dx = = - the IT 6£ ' ) ) - L (E) J J LCE = , ^ \ T (0×26)/55 ' JTT×Lc× , = )T0×[ independent L ( 0×21×1 - C x ) dxt K ) = = → Change five ob.ec of is in Lordi nates of choice

Natural Gradients Vaniational Inference in Distance Metric KL Symmetric divergence ; KL "mld,7 , ,llg9g"h¥aI 't Ear = + Ftaanoillgag "hs¥ts ) KL "m( 7. di ) ddt 6C 7) di � 1 � + = Felli Gilb ) Lt ) 0 , ) =

Gradients Invariant Coordinate - Polar Example coordinates ; -1.1=1 Xz " xi C xi ~ t.n.yx.nl I ' n × . do ' ' J case =/ " - is f sing - cage s - Fa . fusion ° ' =