Advanced Machine Learning CS 7140 - Spring 2018 Lecture 16: Project - - PowerPoint PPT Presentation

Advanced Machine Learning

CS 7140 - Spring 2018

Lecture 16: Project Discussion

Jan-Willem van de Meent

Post-midterm Feedback

https://goo.gl/forms/TGbXazi7j1lurBf72 Please fill this out (should take <5 mins) (this is unofficial and anonymous)

Project

• Goal: Implement and test one 


state-of-the-art method

• Group size 1-4 members
• Amount of work should be equivalent


to ~2 homework assignments

• 20% of Grade

Grading

• Homework: 30%
• Scribing: 20%
• Exams: 30%
• Project: 20%

Example: LDA

Download one or more datasets

• 20 Newsgroups, NY Times, Wikipedia

Implement and compare algorithms

• Gibbs Sampling,
• Stochastic Variational Inference 


[Hoffman JMLR 2013] Test your Implementations

• Geweke Style Tests
• Reference Implementations

Evaluate results

• Visualize Topics
• Perplexity & Coherence Measures

Example: Automatic Statistician

Goal: Search over Kernels for GP Regression [Duvenaud et al. ICML 2013]

Example: Automatic Statistician

Goal: Search over Kernels for GP Regression [Duvenaud et al. ICML 2013]

Lin × Lin quadratic functions SE × Per locally periodic Lin + Per periodic with trend SE + Per periodic with noise Lin × SE increasing variation Lin × Per growing amplitude

Example: Automatic Statistician

Basic: Automatic Statistician “Light”

• Do basis function regression


(with lots of basis functions) Basic: Test on Standard Datasets

• Airline, CO2, etc…

Advanced: Full Implementation

• Use GPFlow, perform kernel search

Advanced: Real-world Datasets

• Analyze Uber Data

Very Advanced: Model Ensembles

• Use MCMC to sample distribution

• ver possible solutions

Example: Time Series Analysis

Goal: Model commonalities between many short time series. [van de Meent et al. ICML 2013]

Example: Time Series Analysis

Basic: Variational Inference

• Implement VBEM for Hidden Markov Models

Basic: Test on Synthetic Datasets

• Can provide these

Advanced: Stochastic Gradient Version

• Implement Stochastic Variational Inference

Advanced: Full Implementation

• Maximize prior hyperparameters

Example: Variational Autoencoders

784 (28 x 28) 256 Input Images Hidden Units 2-50 Encoding (random) Mean
 Std Dev 256 784 (28 x 28) Hidden Units Reconstructed
 Images

Example: Variational Autoencoders

2-dimensional 50-dimensional (TSNE)

Example: Variational Autoencoders

Basic: Semi-supervised Learning

• Reproduce [Kingma et al NIPS 2014]

Advanced: Autoencoding LDA

• Reproduce [Miao et al ICML 2016]

• r [Srivastava ICLR 2017]

Super Advanced: Grammar VAEs

• Reproduce [Kushner et al. ICML 2017]

Super Advanced: Structured VAEs

• Reproduce [Johnson et al. NIPS 2016]

Super Advanced: Disentangled Representations

• (talk to Babak)

References

Stochastic Variational Inference for LDA

• Hoffman, M. D., Blei, D. M., Wang, C. & Paisley, J. Stochastic variational inference.

Journal of Machine Learning Research 14, 1303–1347 (2013). Automatic Statistican

• Duvenaud, D., Lloyd, J. R., Grosse, R., Tenenbaum, J. B. & Ghahramani, Z.

Structure discovery in nonparametric regression through compositional kernel

• search. ICML (2013).

Time Series Analysis

• van de Meent, J.-W., Bronson, J. E., Wood, F., Gonzalez, R. L. & Wiggins, C. H.

Hierarchically-coupled hidden Markov models for learning kinetic rates from single- molecule data. in International Conference on Machine Learning 28, 361–369 (2013).

References

Semi-Supervised VAEs

• Kingma, D. P., Rezende, D. J., Mohamed, S. & Welling, M. Semi-Supervised

Learning with Deep Generative Models. NIPS (2014). Auto-encoding LDA

• Miao, Y., Yu, L. & Blunsom, P. Neural variational inference for text
• processing. in ICML 1727–1736 (2016).
• Srivastava, A. & Sutton, C. Autoencoding variational inference for topic
• models. ICLR (2017).

Grammar VAEs

• Kusner, M. J., Paige, B. & Hernández-Lobato, J. M. Grammar variational
• autoencoder. ICML (2017).

Structured VAEs

• Johnson, M., Duvenaud, D. K., Wiltschko, A., Adams, R. P. & Datta, S. R.

Composing graphical models with neural networks for structured representations and fast inference. in Advances in neural information processing systems 2946–2954 (2016).