Tensor Methods: A New Paradigm for Probabilistic Models and Feature - - PowerPoint PPT Presentation

Tensor Methods: A New Paradigm for Probabilistic Models and Feature Learning

Anima Anandkumar

U.C. Irvine

Learning with Big Data

Data vs. Information

Data vs. Information

Data vs. Information

Missing observations, gross corruptions, outliers.

Data vs. Information

Missing observations, gross corruptions, outliers. Learning useful information is finding needle in a haystack!

Matrices and Tensors as Data Structures

Multi-modal and multi-relational data. Matrices: pairwise relations. Tensors: higher order relations.

Multi-modal data figure from Lise Getoor slides.

Spectral Decomposition of Tensors

M2 =

i

λiui ⊗ vi

= + ....

Matrix M2 λ1u1 ⊗ v1 λ2u2 ⊗ v2

Spectral Decomposition of Tensors

M2 =

i

λiui ⊗ vi

= + ....

Matrix M2 λ1u1 ⊗ v1 λ2u2 ⊗ v2

M3 =

i

λiui ⊗ vi ⊗ wi

= + ....

Tensor M3 λ1u1 ⊗ v1 ⊗ w1 λ2u2 ⊗ v2 ⊗ w2

We have developed efficient methods to solve tensor decomposition.

Strengths of Tensor Methods

Fast and accurate, orders of magnitude faster than previous methods. Embarrassingly parallel and suited for cloud systems, e.g.Spark. Exploit optimized linear algebra libraries. Exploit parallelism of GPU systems.

Strengths of Tensor Methods

Fast and accurate, orders of magnitude faster than previous methods. Embarrassingly parallel and suited for cloud systems, e.g.Spark. Exploit optimized linear algebra libraries. Exploit parallelism of GPU systems.

10

2

10

3

10

−1

10 10

1

10

2

10

3

10

4

Number of communities k Running time(secs)

MATLAB Tensor Toolbox(CPU) CULA Standard Interface(GPU) CULA Device Interface(GPU) Eigen Sparse(CPU)

Outline

1

Introduction

2

Learning Probabilistic Models

3

Experiments

4

Feature Learning with Tensor Methods

5

Conclusion

Latent variable models

Incorporate hidden or latent variables. Information structures: Relationships between latent variables and

• bserved data.

Latent variable models

Incorporate hidden or latent variables. Information structures: Relationships between latent variables and

• bserved data.

Basic Approach: mixtures/clusters

Hidden variable is categorical.

Latent variable models

Incorporate hidden or latent variables. Information structures: Relationships between latent variables and

• bserved data.

Basic Approach: mixtures/clusters

Hidden variable is categorical.

Advanced: Probabilistic models

Hidden variables have more general distributions. Can model mixed membership/hierarchical groups. x1 x2 x3 x4 x5 h1 h2 h3

Challenges in Learning LVMs

Computational Challenges

Maximum likelihood: non-convex optimization. NP-hard. Practice: Local search approaches such as gradient descent, EM, Variational Bayes have no consistency guarantees. Can get stuck in bad local optima. Poor convergence rates and hard to parallelize.

Tensor methods yield guaranteed learning for LVMs

Unsupervised Learning of LVMs

GMM HMM

h1 h2 h3 x1 x2 x3

ICA

h1 h2 hk x1 x2 xd

Multiview and Topic Models

Overall Framework for Unsupervised Learning

= +

....

Unlabeled Data Probabilistic admixture models Tensor Method Inference

Outline

1

Introduction

2

Learning Probabilistic Models

3

Experiments

4

Feature Learning with Tensor Methods

5

Conclusion

Demo for Learning Gaussian Mixtures

NYTimes Demo

Experimental Results on Yelp and DBLP

Lowest error business categories & largest weight businesses

Rank Category Business Stars Review Counts 1 Latin American Salvadoreno Restaurant 4.0 36 2 Gluten Free P.F. Chang’s China Bistro 3.5 55 3 Hobby Shops Make Meaning 4.5 14 4 Mass Media KJZZ 91.5FM 4.0 13 5 Yoga Sutra Midtown 4.5 31

Experimental Results on Yelp and DBLP

Lowest error business categories & largest weight businesses

Rank Category Business Stars Review Counts 1 Latin American Salvadoreno Restaurant 4.0 36 2 Gluten Free P.F. Chang’s China Bistro 3.5 55 3 Hobby Shops Make Meaning 4.5 14 4 Mass Media KJZZ 91.5FM 4.0 13 5 Yoga Sutra Midtown 4.5 31

Top-5 bridging nodes (businesses)

Business Categories Four Peaks Brewing Restaurants, Bars, American, Nightlife, Food, Pubs, Tempe Pizzeria Bianco Restaurants, Pizza, Phoenix FEZ Restaurants, Bars, American, Nightlife, Mediterranean, Lounges, Phoenix Matt’s Big Breakfast Restaurants, Phoenix, Breakfast& Brunch Cornish Pasty Co Restaurants, Bars, Nightlife, Pubs, Tempe

Experimental Results on Yelp and DBLP

Lowest error business categories & largest weight businesses

Rank Category Business Stars Review Counts 1 Latin American Salvadoreno Restaurant 4.0 36 2 Gluten Free P.F. Chang’s China Bistro 3.5 55 3 Hobby Shops Make Meaning 4.5 14 4 Mass Media KJZZ 91.5FM 4.0 13 5 Yoga Sutra Midtown 4.5 31

Top-5 bridging nodes (businesses)

Business Categories Four Peaks Brewing Restaurants, Bars, American, Nightlife, Food, Pubs, Tempe Pizzeria Bianco Restaurants, Pizza, Phoenix FEZ Restaurants, Bars, American, Nightlife, Mediterranean, Lounges, Phoenix Matt’s Big Breakfast Restaurants, Phoenix, Breakfast& Brunch Cornish Pasty Co Restaurants, Bars, Nightlife, Pubs, Tempe

Error (E) and Recovery ratio (R) Dataset ˆ k Method Running Time E R DBLP sub(n=1e5) 500

• urs

10,157 0.139 89% DBLP sub(n=1e5) 500 variational 558,723 16.38 99% DBLP(n=1e6) 100

• urs

5407 0.105 95%

Discovering Gene Profiles of Neuronal Cell Types

Learning mixture of point processes of cells through tensor methods. Components of mixture are candidates for neuronal cell types.

Discovering Gene Profiles of Neuronal Cell Types

Learning mixture of point processes of cells through tensor methods. Components of mixture are candidates for neuronal cell types.

Hierarchical Tensors for Healthcare Analytics

= + .... = + .... = + .... = + ....

Hierarchical Tensors for Healthcare Analytics

= + .... = + .... = + .... = + ....

CMS dataset: 1.6 million patients, 15.8 million events. Mining disease inferences from patient records.

Outline

1

Introduction

2

Learning Probabilistic Models

3

Experiments

4

Feature Learning with Tensor Methods

5

Conclusion

Feature Learning For Efficient Classification

Find good transformations of input for improved classification

Figures used attributed to Fei-Fei Li, Rob Fergus, Antonio Torralba, et al.

Tensor Methods for Training Neural Networks

First order score function m-th order score function

Input pdf is p(x) Sm(x) := (−1)m∇(m)p(x) p(x) Capture local variations in data.

Algorithm for Training Neural Networks

Estimate score functions using autoencoder. Decompose tensor E[y ⊗ Sm(x)] to obtain weights, for m ≥ 3. Recursively estimate score function of autoencoder and repeat.

Tensor Methods for Training Neural Networks

Second order score function m-th order score function

Input pdf is p(x) Sm(x) := (−1)m∇(m)p(x) p(x) Capture local variations in data.

Algorithm for Training Neural Networks

Estimate score functions using autoencoder. Decompose tensor E[y ⊗ Sm(x)] to obtain weights, for m ≥ 3. Recursively estimate score function of autoencoder and repeat.

Tensor Methods for Training Neural Networks

Third order score function m-th order score function

Input pdf is p(x) Sm(x) := (−1)m∇(m)p(x) p(x) Capture local variations in data.

Algorithm for Training Neural Networks

Estimate score functions using autoencoder. Decompose tensor E[y ⊗ Sm(x)] to obtain weights, for m ≥ 3. Recursively estimate score function of autoencoder and repeat.

Demo: Training Neural Networks

Combining Probabilistic Models with Deep Learning

Multi-object Detection in Computer vision

Deep learning is able to extract good features, but not context. Probabilistic models capture contextual information. Hierarchical models + pre-trained deep learning features. State-of-art results on Microsoft COCO.

Outline

1

Introduction

2

Learning Probabilistic Models

3

Experiments

4

Feature Learning with Tensor Methods

5

Conclusion

Conclusion: Tensor Methods for Learning

Tensor Decomposition

Efficient sample and computational complexities Better performance compared to EM, Variational Bayes etc.

In practice

Scalable and embarrassingly parallel: handle large datasets. Efficient performance: perplexity or ground truth validation.

My Research Group and Resources

Furong H. Majid J. Hanie S. Niranjan U.N. Forough A. Tejaswi N. Hao L. Yang S.

ML summer school lectures available at http://newport.eecs.uci.edu/anandkumar/MLSS.html