Pengtao Xie Joint work with Yuntian Deng and Eric Xing Carnegie - PowerPoint PPT Presentation

Mutual Angular Regularization of Latent Variable Models: Theory, Algorithm and Applications Pengtao Xie Joint work with Yuntian Deng and Eric Xing Carnegie Mellon University 1

Latent Variable Models (LVMs) Machine Learning Latent Variable Models Pattern 2

Latent Variable Models Topic Models Gaussian Mixture Model Topics Groups Words Feature vectors Hidden Markov Model, Kalman Filtering, Restricted Boltzmann Machine, Deep Belief Network, Factor Analysis, etc. Neural Network, Sparse Coding, Matrix Factorization, Distance Metric learning, Principal Component Analysis, etc. 3

Latent Variable Models Latent Factors Components Behind Data in LVMs Topics in Documents Topic Models Politics Economics Education Obama GDP University Constitution Bank Knowledge Student Government Marketing Groups in Images Gaussian Mixture Model Tiger Car Food 4

Motivation I: Popularity of latent factors is skewed  Popularity of latent factors follows a power-law distribution Groups in Topics in News Flickr Photos Dominant Topics Long-Tail Topics Dominant Groups Long-Tail Groups Politics Economics Furniture Flower Diamond Painting Obama GDP Sofa Rose Car Food Constitution Bank Closet Tulip Government Marketing Curtain Lily 5

Standard LVMs are insufficient to capture long-tail factors  Latent Dirichlet Allocation (LDA)  “Extremely common words tend to dominate all topics” (Wallach, 2009)  Tencent Peacock LDA, “When learning ≥ 10 5 topics, around 20% ∼ 40% topics have duplicates in practice” (Wang, 2015 )  Restricted Boltzmann Machine Topic 1 Topic 2 Topic 3 president iraq iraq  Ran on 20-Newsgroup dataset clinton united un  Many duplicate topics (e.g., the three iraq un iraqi united weapons lewinsky exemplar topics are all about politics) spkr iraqi saddam  Common words occur repeatedly house nuclear clinton people india baghdad across topics, such as iraq, clinton, lewinsky minister inspectors united, weapons government saddam weapons white military white 6

Standard LVMs are insufficient to capture long-tail factors Latent factors behind data Components in LVM 7

Long-tail factors are important  The amount of long-tail factors is large Long-tail factors  Long-tail factors are more important than dominant factors in some applications  Example: Tencent applied topic models for advertisement and showed that long- tail topics such as “lose weight”, “nursing” improves click-through rate by 40% (Jin, 2015) 8

Diversification Latent factors behind data Components in LVM 9

Motivation II: Tradeoff induced by the number of components k  Tradeoff between Expressiveness and Complexity  Small k: low expressiveness, low complexity  Large k: high expressiveness, high complexity  Can we achieve the best of both worlds?  Small k: high expressiveness, low complexity 10 10

Reduce model complexity without sacrificing expressiveness Without diversification With diversification Data Samples Use components to capture the principal directions of data point cloud Components in LVM 11 11

Mutual Angular Regularization of LVMs  Goal: encourage the components to diversely spread out to (1) improve the coverage of long-tail latent factors; (2) reduce model complexity without compromising expressiveness  Approach:  Define a score based on mutual angles to measure the diversity of components  Use the score to regularize latent variable models and control the geometry of the latent space during learning 12 12

Outline  Mutual Angular Regularizer  Algorithm  Applications  Theory 13 13

Mutual Angular Regularizer  Components are parametrized by vectors  In Latent Dirichlet Allocation, each topic has a multinomial vector  In Sparse Coding, each dictionary item has a real vector  Measure the dissimilarity between two vectors  Measure the diversity of a vector set 14 14

Dissimilarity between two vectors  Invariant to scale, translation, rotation and orientation of the two vectors  Euclidean distance, L1 distance  Distance 𝑒 is variant to scale d d O O  Negative cosine similarity  Negative cosine similarity 𝑏 is variant to orientation O O a=0.6 a=-0.6 15 15

Dissimilarity between two vectors  Non-obtuse angle 𝜄 𝜄 𝜄 𝜄 O O O  Invariant to scale, translation, rotation and orientation of the two vectors  Definition    x y     arccos   x y   16 16

Measure the diversity of a vector set  Based on the pairwise dissimilarity measure between vectors    a K  The diversity of a set of vectors is defined as A  i i 1 2   Mutual    K K K K  K K  1 1 1    a a          ( ) A   Angular i j   arccos    ij ij pq     K K ( 1) K K ( 1) K K ( 1) ij       i 1 j 1 i 1 j 1 p 1 q 1 a a     Regularizer    i j j i j i q p Mean of angles Variance of angles  Mean: summarize how these vectors are different from each other on the whole  Variance: encourage the vectors to evenly spread out 17 17

LVM with Mutual Angular Regularization (MAR-LVM)    max L D ( ; A ) ( ) A A 2   K K K K  K K  1  1  1         ( ) A      ij ij pq   K K ( 1) K K ( 1) K K ( 1)       i 1 j 1 i 1 j 1 p 1 q 1      j i j i q p    a a     i j arccos   ij a a   i j 18 18

Algorithm  Challenge: the mutual angular regularizer is non-smooth and    a K non-convex w.r.t the parameter vectors A  i i 1  Derive a smooth lower bound  The lower bound is easier to derive if the parameter vectors lie on a sphere  Decompose the parameter vectors into magnitudes and directions  Proved that optimizing the lower bound with gradient ascent method can increase the mutual angular regularizer in each iteration 19 19

Optimization Reparametrize     a g a g a a 1 A diag( ) g A i i i i i i = Ω diag(𝐡)𝐁 Ω 𝐁 Magnitude Direction ~ ~    max L ( D ; g A ) ( A ) ~    g , A max L D A ( ; ) ( ) A ~ g A ,    i , a 1 , g 0 s . t . i i Alternating Optimization ~ ~ g g Fix , optimize Fix , optimize A A ~ ~ ~    max ~ max L ( D ; g A ) ( A ) L ( D ; g A ) g A ~     s . t . a i , 1 s . t . i , g 0 i i 20 20

Optimize 𝑩 ~ ~    max ~ g A A L ( D ; ) ( ) A ~   s . t . i , a 1 i Lower bound      2             T T   ( ) A ( ) A arcsin det A A arcsin det A A   2 𝑈 𝑩 is the volume of the parallelipiped Intuition of the lower bound: det 𝑩 . The larger det 𝑩 𝑈 𝑩 is, the more likely that the formed by the vectors in 𝑩 have larger angles (not surely). Γ 𝑩 is an increasing function w.r.t vectors in 𝑩 𝑈 𝑩 . Hence larger Γ 𝑩 is likely to yield larger Ω 𝑩 . det 𝑩 Optimize the lower bound, which is smooth and much more amenable for optimization ~ ~    max ~ L ( D ; g A ) ( A ) A ~   s . t . i , a 1 i 21 21

Close Alignment between the Regularizer and its Lower Bound  If the lower bound is optimized with projected gradient ascent (PGA), the mutual angular regularizer can be increased in each iteration of the PGA procedure  Optimizing the lower bound with PGA can increase the mean of the angles in each iteration  Optimizing the lower bound with PGA can decrease the variance of the angles in each iteration 2   - K K K K  K K  1 1 1           ( ) A     ij ij pq   K K ( 1) K K ( 1) K K ( 1)       i 1 j 1 i 1 j 1 p 1 q 1      j i j i q p Variance Mean 22 22

Geometry Interpretation of the Close Alignment  The gradient of the lower bound w.r.t is orthogonal to all a i     other vectors a a , , , a a 1 2 K i  Move along its gradient direction would enlarge its angle a i with other vectors a a a are parameter vectors g ˆ  2 3 1 1 a a a g is the gradient of 1 a and are orthogonal to 2 1 1 3     a ˆ a a g 1 1 1 1    ˆ     a The angle between and is greater than a 3 1 a a a between and 3 3 1 a 2 23 23