Automatic code rewriting in probabilistic programming Internship - PowerPoint PPT Presentation

Background The transformation in theory Technical aspects Experimental performance Automatic code rewriting in probabilistic programming Internship supervised by Hongseok Yang at the University of Oxford Diane Gallois-Wong September 11, 2015 Diane Gallois-Wong Automatic code rewriting in probabilistic programming 1 / 13

Background The transformation in theory Technical aspects Experimental performance Introduction Probabilistic programming languages: short, intuitive code to describe probabilistic models, built-in inference algorithms LDA model: straightforward, naive implementation: easy to write but bad performance collapsed implementation: complex, better performance Automatic transformation of naive code into collapsed version Formalism with a lambda calculus, program analysis techniques Anglican: a probabilistic programming language (Oxford) Diane Gallois-Wong Automatic code rewriting in probabilistic programming 2 / 13

Background Bayesian statistics The transformation in theory The probabilistic pogramming language Anglican Technical aspects Latent Dirichlet Allocation Experimental performance Bayesian statistics Bayes’ Theorem p ( H | X ) = p ( X | H ) p ( H ) p ( X ) H : hypothesis, X : observation (events) prior probability, posterior probability, likelihood function Diane Gallois-Wong Automatic code rewriting in probabilistic programming 3 / 13

Background Bayesian statistics The transformation in theory The probabilistic pogramming language Anglican Technical aspects Latent Dirichlet Allocation Experimental performance Anglican Probabilistic programming language Integrated in the functional language Clojure (dialect of Lisp) (let [more-heads (sample (flip (/ 1 2))) coin (flip (if more-heads (/ 2 3) (/ 1 3)))] (observe coin true) (observe coin true) (observe coin true) (predict more-heads)) Diane Gallois-Wong Automatic code rewriting in probabilistic programming 4 / 13

Background Bayesian statistics The transformation in theory The probabilistic pogramming language Anglican Technical aspects Latent Dirichlet Allocation Experimental performance Latent Dirichlet Allocation (LDA) Topic Model Input: collection of documents which are collections of words Aim: classify the documents using topics LDA : generative topic model θ d ∼ Dirichlet( α ) prob. vector over topics for each document d ϕ k ∼ Dirichlet( β ) prob. vector over words for each topic k z d , n ∼ Discrete ( θ d ) for each position n in document d , choose a topic z d , n according to θ d w d , n ∼ Discrete ( ϕ z d , n ) then a word w d , n according to ϕ z d , n Diane Gallois-Wong Automatic code rewriting in probabilistic programming 5 / 13

Background The transformation in theory Highly expressive latent variables Technical aspects Conjugate Prior and Dirichlet Process Experimental performance Highly expressive latent variables Naive implementation of LDA: Inputs: a corpus of documents w where w d , n is the word at position n in document d and hyperparameters α and β for each d : θ d = sample Dirichlet( α ) for each k : ϕ k = sample Dirichlet( β ) for each d and each n : z d , n = sample Discrete( θ d ) observe Discrete( ϕ z d , n ) w d , n predict z Diane Gallois-Wong Automatic code rewriting in probabilistic programming 6 / 13

Background The transformation in theory Highly expressive latent variables Technical aspects Conjugate Prior and Dirichlet Process Experimental performance Conjugate Prior θ ∼ Dirichlet( α ) x ∼ Discrete( α ) x | θ ∼ Discrete( θ ) θ | x ∼ Dirichlet( f ( α, x )) Dirichlet is a conjugate prior with a Discrete likelihood. Dirichlet( α ) interpretation: class i observed α i − 1 times. f ( α, x ) = ( α where component with index x incremented) Diane Gallois-Wong Automatic code rewriting in probabilistic programming 7 / 13

Background The transformation in theory Highly expressive latent variables Technical aspects Conjugate Prior and Dirichlet Process Experimental performance Conjugate Prior and Dirichlet Process θ ∼ Dirichlet( α ) x ∼ Discrete( α ) x | θ ∼ Discrete( θ ) θ | x ∼ Dirichlet( f ( α, x )) ∼ Dirichlet( α ) θ x 0 ∼ Discrete( α ) ; α 0 = f ( α, x 0 ) ∼ Discrete( θ ) x 0 x 1 ∼ Discrete( α 0 ) ; α 1 = f ( α 0 , x 1 ) x 1 ∼ Discrete( θ ) x 2 ∼ Discrete( α 1 ) ; α 2 = f ( α 1 , x 2 ) ∼ Discrete( θ ) x 2 ( θ ∼ Dirichlet( α 2 )) θ is marginalised . Diane Gallois-Wong Automatic code rewriting in probabilistic programming 8 / 13

Background Necessary conditions on variables to marginalise The transformation in theory State and modular functions Technical aspects Experimental performance Necessary conditions on variables to marginalise Necessary conditions imposed to make things easier. Main condition to marginalise θ : θ should only appear as argument to a Discrete distribution. Automatically checked through multiple steps of program analysis, including a type and effect system . Diane Gallois-Wong Automatic code rewriting in probabilistic programming 9 / 13

Background Necessary conditions on variables to marginalise The transformation in theory State and modular functions Technical aspects Experimental performance State and modular functions x 1 ∼ Discrete( α 0 ) ; α 1 = f ( α 0 , x 1 ) x 1 ∼ Discrete( θ ) − → x 1 = sample Discrete ( θ ) − → [ x 1 ; S ] = S-sample S-Discrete ( θ ) S Diane Gallois-Wong Automatic code rewriting in probabilistic programming 10 / 13

Background Necessary conditions on variables to marginalise The transformation in theory State and modular functions Technical aspects Experimental performance An example of code Diane Gallois-Wong Automatic code rewriting in probabilistic programming 11 / 13

Background The transformation in theory Technical aspects Experimental performance Vocabulary words: 0, 1, 2. Input is a corpus generated ac- cording to two topics: [0 . 5 ; 0 ; 0 . 5] , [0 ; 0 . 5 ; 0 . 5] Plot estimations of a topic. Naive implementation Transformed code Naive implementation Diane Gallois-Wong Automatic code rewriting in probabilistic programming 12 / 13

Background The transformation in theory Technical aspects Experimental performance Conclusion Formal definition of the automatic transformation and working implementation (small subset of Anglican) Potential improvements: extend the subset, weaken necessary conditions, improve memory management of the state, extend to other conjugate priors Will be used as part of a much more general automatic code transformation developed by my supervisor to be eventually added to Anglican High level transformation: affects the model itself Diane Gallois-Wong Automatic code rewriting in probabilistic programming 13 / 13