SLIDE 1
Nonparametric Deconvolution Models
Allison J.B. Chaney Princeton University
In collaboration with Barbara Engelhardt, Archit Verma and Young-Suk Lee
Nonparametric Deconvolution Models Allison J.B. Chaney Princeton - - PowerPoint PPT Presentation
Nonparametric Deconvolution Models Allison J.B. Chaney Princeton - - PowerPoint PPT Presentation
Nonparametric Deconvolution Models Allison J.B. Chaney Princeton University In collaboration with Barbara Engelhardt, Archit Verma and Young-Suk Lee Objective Model collections of convolved data points Objective Model collections of
SLIDE 2
SLIDE 3
Objective
Model collections of convolved data points
SLIDE 4
Objective
Model collections of convolved data points
SLIDE 5
Objective
Model collections of convolved data points
1 2 2 2 3 3 1 1 3 2 1 2 2 3 2
SLIDE 6
Objective
Model collections of convolved data points
General Voting Bulk RNA-seq Images
- bservation
district vote tally sample image feature issue or candidate gene expression level pixel particle individual voter
- ne cell
light particle factor voting cohort cell type visual pattern
SLIDE 7
“convolution”
individual particles
- bserved together
1 2 2 3 2
signal progressing
convolutional neural nets
SLIDE 8
“convolution”
individual particles
- bserved together
1 2 2 3 2
signal progressing
convolutional neural nets
SLIDE 9
Related Models
- Mixture models assign each
- bservation to one of K clusters, or
factors.
SLIDE 10
Related Models
- Admixture models represent
groups of observations, each with its
- wn mixture of K shared factors.
SLIDE 11
Related Models
- Decomposition models
decompose observations into constituent parts by representing
- bservations as a product between
group representations and factor features.
SLIDE 12
Our Model
- Deconvolution models (this
work) similarly decompose, or deconvolve, observations into constituent parts, but also capture group-specific (or local) fluctuations in factor features.
SLIDE 13
How do usually vote?
1 2 2 2 3 3 1 1 3 2 1 2 2 3 2
SLIDE 14
1 2 2 2 3 3 1 1 3 2 1 2 2 3 2
- How do vote in district A?
A
A B C …
B C D E
SLIDE 15
Our Model
SLIDE 16
Our Model
- global factors
SLIDE 17
Our Model
- local factors
(observation-specific)
SLIDE 18
Our Model
SLIDE 19
Our Model
HDP
SLIDE 20
Our Model
HDP
SLIDE 21
Our Model
HDP
Paisley, 2012
β′
k | α0 ∼ Beta(1,α0)
βk = β′
k k−1
∏
ℓ=1
(1 − β′
ℓ)
SLIDE 22
Our Model
HDP
Paisley, 2012
β′
k | α0 ∼ Beta(1,α0)
βk = β′
k k−1
∏
ℓ=1
(1 − β′
ℓ)
π′
n,k | α, βk ∼ Gamma(αβk,1)
πn,k = π′
n,k
∑∞
ℓ=1 π′ n,ℓ
SLIDE 23
Our Model
μk,m | μ0, σ0 ∼ 𝒪(μ0, σ0)
SLIDE 24
Our Model
μk,m | μ0, σ0 ∼ 𝒪(μ0, σ0) Σk | ν, Ψ ∼ 𝒳−1(Ψ, ν)
SLIDE 25
Our Model
μk,m | μ0, σ0 ∼ 𝒪(μ0, σ0) Σk | ν, Ψ ∼ 𝒳−1(Ψ, ν) ¯ xn,k | πn,k, μk, Σk ∼ 𝒪M (μk, Σk Pnπn,k)
SLIDE 26
Our Model
μk,m | μ0, σ0 ∼ 𝒪(μ0, σ0) Σk | ν, Ψ ∼ 𝒳−1(Ψ, ν) ¯ xn,k | πn,k, μk, Σk ∼ 𝒪M (μk, Σk Pnπn,k) Pn | ρ ∼ Poisson(ρ)
SLIDE 27
Our Model
yn,m | ¯ xn, πn ∼ f g (
∞
∑
k=1
πn,k ¯ xn,k,m)
SLIDE 28
Inference
SLIDE 29
Inference
? ? ? ? ? ?
SLIDE 30
Variational Inference
intractable posterior p
SLIDE 31
easy to compute approximation q
Variational Inference
intractable posterior p
SLIDE 32
easy to compute approximation q
Variational Inference
intractable posterior p black box variational inference (Ranganath, 2014) split-merge procedure (Bryant, 2012) to learn K
SLIDE 33
BBVI overview
Ranganath et al., 2014
∇λ[z]ℒ = 𝔽q [∇λ[z]log q(z | λ[z])(log pz(y, z, …) − log q(z | λ[z]))]
We want to estimate f
z
SLIDE 34
BBVI overview
Ranganath et al., 2014
∇λ[z]ℒ = 𝔽q [∇λ[z]log q(z | λ[z])(log pz(y, z, …) − log q(z | λ[z]))]
We want to estimate f
z
Which has corresponding variational parameter f λ[z]
SLIDE 35
BBVI overview
Ranganath et al., 2014
∇λ[z]ℒ = 𝔽q [∇λ[z]log q(z | λ[z])(log pz(y, z, …) − log q(z | λ[z]))]
We want to estimate f
z
Which has corresponding variational parameter f λ[z]
is the set of all variational parameters
λ
SLIDE 36
BBVI overview
Ranganath et al., 2014
∇λ[z]ℒ = 𝔽q [∇λ[z]log q(z | λ[z])(log pz(y, z, …) − log q(z | λ[z]))]
We want to estimate f
z
Which has corresponding variational parameter f λ[z]
is the set of all variational parameters
λ
The gradient of the ELBO
SLIDE 37
BBVI overview
Ranganath et al., 2014
∇λ[z]ℒ = 𝔽q [∇λ[z]log q(z | λ[z])(log pz(y, z, …) − log q(z | λ[z]))]
We want to estimate f
z
Which has corresponding variational parameter f λ[z]
is the set of all variational parameters
λ
The gradient of the ELBO
≈ ˜ ∇λ[z]ℒ
If we can approximate this gradient, we can use standard stochastic gradient ascent to update . λ[z]
SLIDE 38
BBVI overview
Ranganath et al., 2014
∇λ[z]ℒ = 𝔽q [∇λ[z]log q(z | λ[z])(log pz(y, z, …) − log q(z | λ[z]))]
We want to estimate f
z
Which has corresponding variational parameter f λ[z]
is the set of all variational parameters
λ
The gradient of the ELBO
= 1 S
S
∑
s=1
[∇λ[z]log q(z[s] | λ[z])(log pz(y, z[s], …) − log q(z[s] | λ[z]))] ≈ ˜ ∇λ[z]ℒ
If we can approximate this gradient, we can use standard stochastic gradient ascent to update . λ[z] Average over S samples From the variational distribution
z[s] ∼ q(z | λ[z])
SLIDE 39
split/merge overview
Bryant and Sudderth, 2012 initialize with fixed K iterate until batch convergence consider splitting each factor consider merging some factors full convergence
SLIDE 40
split/merge overview
Bryant and Sudderth, 2012 initialize with fixed K ergence consider splitting each factor consider merging some factors
SLIDE 41
split/merge overview
Bryant and Sudderth, 2012 initialize with fixed K ergence consider splitting each factor consider merging some factors
SLIDE 42
split/merge overview
Bryant and Sudderth, 2012 initialize with fixed K ergence consider splitting each factor consider merging some factors
initialize variational parameters update variational parameters (one iteration) accept / reject based on ELBO
SLIDE 43
split/merge overview
Bryant and Sudderth, 2012 initialize with fixed K ergence consider splitting each factor consider merging some factors
λS[βk′] = ρtλ[βk] λS[βk′′] = (1 − ρt)λ[βk] λ[βk]
SLIDE 44
split/merge overview
Bryant and Sudderth, 2012 initialize with fixed K ergence consider splitting each factor consider merging some factors
λS[πn,k′] = ρtλ[πn,k] λS[πn,k′′] = (1 − ρt)λ[πn,k] λ[πn,k]
SLIDE 45
split/merge overview
Bryant and Sudderth, 2012 initialize with fixed K ergence consider splitting each factor consider merging some factors
λS[μk′] = λ[μk] λS[μk′′] = λ[μk] + ε λ[μk]
SLIDE 46
split/merge overview
Bryant and Sudderth, 2012 initialize with fixed K ergence consider splitting each factor consider merging some factors
λS[Σk′] = λ[Σk] λS[Σk′′] = λ[Σk] λ[Σk]
SLIDE 47
split/merge overview
Bryant and Sudderth, 2012 initialize with fixed K ergence consider splitting each factor consider merging some factors
λS[xn,k′] = λ[xn,k] λS[xn,k′′] = λ[xn,k] λ[xn,k]
SLIDE 48
split/merge overview
Bryant and Sudderth, 2012 initialize with fixed K ergence consider splitting each factor consider merging some factors
λM[βk] = λ[βk′] + λ[βk′′] λM[πn,k] = λ[πn,k′] + λ[πn,k′′] λM[μk] = λ[βk′]λ[μk′] + λ[βk′′]λ[μk′′] λ[βk′] + λ[βk′′]
…
SLIDE 49
set K to an initial value initialize variational parameters repeat until convergence: repeat until batch convergence: update variational parameters for using BBVI update variational parameters for _using analytic updates split/merge latent factors, defining new K and updating variational parameters accordingly
¯ x, π, P, β
μ, Σ
Algorithm Pseudocode
SLIDE 50
SLIDE 51
SLIDE 52
SLIDE 53
SLIDE 54
SLIDE 55
SLIDE 56
SLIDE 57
Results on Simulated Data
SLIDE 58
Results on Simulated Data
SLIDE 59
2016 Election in California
https://github.com/datadesk/california-2016-election-precinct-maps 43.9% registered Democrats 28.9% registered Republicans 27.2% other parties / unregistered caveat: these are very preliminary results
SLIDE 60
SLIDE 61
Prop 63: Background Checks for Ammunition Purchases and Large- Capacity Ammunition Magazine Ban
SLIDE 62
Prop 58: Non-English Languages Allowed in Public Education
SLIDE 63
SLIDE 64
(gun control) (non-english in schools)
SLIDE 65
(gun control) (non-english in schools)
SLIDE 66
Thank you!
This research was supported by an appointment to the Intelligence Community Postdoctoral Research Fellowship Program at Princeton University, administered by Oak Ridge Institute for Science and Education through an interagency agreement between the U.S. Department of Energy and the Office of the Director of National Intelligence.