Finite Approximations of Discrete Random Measures Jonathan H. - - PowerPoint PPT Presentation

Finite Approximations

• f Discrete Random

Measures

Jonathan H. Huggins

Postdoctoral Research Fellow Department of Biostatistics, Harvard

with: Trevor Campbell, Jonathan How, Lorenzo Masoero, Lester Mackey, Tamara Broderick

Bayesian nonparametrics

Bayesian nonparametrics

Need models that can extract new, useful information from unbounded streams of data

Bayesian nonparametrics

e.g. keep learning new topics from a stream of documents Need models that can extract new, useful information from unbounded streams of data

Bayesian nonparametrics

e.g. keep learning new topics from a stream of documents Need models that can extract new, useful information from unbounded streams of data Bayesian nonparametrics: achieves growing model size via infinite parameters

Bayesian nonparametrics

e.g. keep learning new topics from a stream of documents Need models that can extract new, useful information from unbounded streams of data Bayesian nonparametrics: achieves growing model size via infinite parameters

[Gopalan 2014]

movie s

[Teh 2006]

text

[Huang 2014]

medicine

[Michini 2015]

robotics

[Lennox 2010]

genetics

[Prunster 2014]

finance

[Yang 2015]

astronomy

[Yu 2012]

traffic

[Ozaki 2008]

agriculture

[Kottas 2008]

pathology

Bayesian nonparametrics

e.g. keep learning new topics from a stream of documents Need models that can extract new, useful information from unbounded streams of data Bayesian nonparametrics: achieves growing model size via infinite parameters

[Gopalan 2014]

movie s

[Teh 2006]

text

[Huang 2014]

medicine

[Michini 2015]

robotics

[Lennox 2010]

genetics

[Prunster 2014]

finance

[Yang 2015]

astronomy

[Yu 2012]

traffic

[Ozaki 2008]

agriculture

[Kottas 2008]

pathology

Π(dΘ | X) ∝Θ f(X | Θ)Π0(dΘ)

Bayesian nonparametrics

e.g. keep learning new topics from a stream of documents Need models that can extract new, useful information from unbounded streams of data Bayesian nonparametrics: achieves growing model size via infinite parameters

[Gopalan 2014]

movie s

[Teh 2006]

text

[Huang 2014]

medicine

[Michini 2015]

robotics

[Lennox 2010]

genetics

[Prunster 2014]

finance

[Yang 2015]

astronomy

[Yu 2012]

traffic

[Ozaki 2008]

agriculture

[Kottas 2008]

pathology

parameter

Π(dΘ | X) ∝Θ f(X | Θ)Π0(dΘ)

Bayesian nonparametrics

e.g. keep learning new topics from a stream of documents Need models that can extract new, useful information from unbounded streams of data Bayesian nonparametrics: achieves growing model size via infinite parameters

[Gopalan 2014]

movie s

[Teh 2006]

text

[Huang 2014]

medicine

[Michini 2015]

robotics

[Lennox 2010]

genetics

[Prunster 2014]

finance

[Yang 2015]

astronomy

[Yu 2012]

traffic

[Ozaki 2008]

agriculture

[Kottas 2008]

pathology

likelihood parameter

Π(dΘ | X) ∝Θ f(X | Θ)Π0(dΘ)

Bayesian nonparametrics

e.g. keep learning new topics from a stream of documents Need models that can extract new, useful information from unbounded streams of data Bayesian nonparametrics: achieves growing model size via infinite parameters

[Gopalan 2014]

movie s

[Teh 2006]

text

[Huang 2014]

medicine

[Michini 2015]

robotics

[Lennox 2010]

genetics

[Prunster 2014]

finance

[Yang 2015]

astronomy

[Yu 2012]

traffic

[Ozaki 2008]

agriculture

[Kottas 2008]

pathology

data likelihood parameter

Π(dΘ | X) ∝Θ f(X | Θ)Π0(dΘ)

Bayesian nonparametrics

e.g. keep learning new topics from a stream of documents Need models that can extract new, useful information from unbounded streams of data Bayesian nonparametrics: achieves growing model size via infinite parameters

[Gopalan 2014]

movie s

[Teh 2006]

text

[Huang 2014]

medicine

[Michini 2015]

robotics

[Lennox 2010]

genetics

[Prunster 2014]

finance

[Yang 2015]

astronomy

[Yu 2012]

traffic

[Ozaki 2008]

agriculture

[Kottas 2008]

pathology

data likelihood prior parameter

Π(dΘ | X) ∝Θ f(X | Θ)Π0(dΘ)

Bayesian nonparametrics

e.g. keep learning new topics from a stream of documents Need models that can extract new, useful information from unbounded streams of data Bayesian nonparametrics: achieves growing model size via infinite parameters

[Gopalan 2014]

movie s

[Teh 2006]

text

[Huang 2014]

medicine

[Michini 2015]

robotics

[Lennox 2010]

genetics

[Prunster 2014]

finance

[Yang 2015]

astronomy

[Yu 2012]

traffic

[Ozaki 2008]

agriculture

[Kottas 2008]

pathology

posterior data likelihood prior parameter

Π(dΘ | X) ∝Θ f(X | Θ)Π0(dΘ)

Inference in BNP models

π(dΘ | X) ∝Θ f(X | Θ)π0(dΘ)

• Option #1: Integrate out the parameter (CRP, IBP, etc.)


issues: care about the parameters, using certain inference algs. (HMC/VB), distributed computation, discrete latent variables instead

Inference in BNP models

π(dΘ | X) ∝Θ f(X | Θ)π0(dΘ)

Θ

• Option #1: Integrate out the parameter (CRP, IBP, etc.)


issues: care about the parameters, using certain inference algs. (HMC/VB), distributed computation, discrete latent variables instead

• Option #2: use a finite approximation... 


with e.g. variational inference, HMC [Blei 06; Neal 10]

Inference in BNP models

π(dΘ | X) ∝Θ f(X | Θ)π0(dΘ)

Θ

• Option #1: Integrate out the parameter (CRP, IBP, etc.)


issues: care about the parameters, using certain inference algs. (HMC/VB), distributed computation, discrete latent variables instead

• Option #2: use a finite approximation... 


with e.g. variational inference, HMC [Blei 06; Neal 10]

Problem: Wide variety of priors in BNP with
 no or poorly understood finite approximation

Inference in BNP models

π(dΘ | X) ∝Θ f(X | Θ)π0(dΘ)

Θ

• Option #1: Integrate out the parameter (CRP, IBP, etc.)


issues: care about the parameters, using certain inference algs. (HMC/VB), distributed computation, discrete latent variables instead

• Option #2: use a finite approximation... 


with e.g. variational inference, HMC [Blei 06; Neal 10]

Problem: Wide variety of priors in BNP with
 no or poorly understood finite approximation

In this talk:

Inference in BNP models

π(dΘ | X) ∝Θ f(X | Θ)π0(dΘ)

Θ

• Option #1: Integrate out the parameter (CRP, IBP, etc.)


issues: care about the parameters, using certain inference algs. (HMC/VB), distributed computation, discrete latent variables instead

• Option #2: use a finite approximation... 


with e.g. variational inference, HMC [Blei 06; Neal 10]

Problem: Wide variety of priors in BNP with
 no or poorly understood finite approximation

In this talk: 1) Two finite approximation types: truncated and non-nested

Inference in BNP models

π(dΘ | X) ∝Θ f(X | Θ)π0(dΘ)

Θ

• Option #1: Integrate out the parameter (CRP, IBP, etc.)


issues: care about the parameters, using certain inference algs. (HMC/VB), distributed computation, discrete latent variables instead

• Option #2: use a finite approximation... 


with e.g. variational inference, HMC [Blei 06; Neal 10]

Problem: Wide variety of priors in BNP with
 no or poorly understood finite approximation

In this talk: 1) Two finite approximation types: truncated and non-nested 2) Two truncated forms (7 reps total) that allow finite approximation of (normalized) completely random measures [(N)CRMs]

Inference in BNP models

π(dΘ | X) ∝Θ f(X | Θ)π0(dΘ)

Θ

• Option #1: Integrate out the parameter (CRP, IBP, etc.)


issues: care about the parameters, using certain inference algs. (HMC/VB), distributed computation, discrete latent variables instead

• Option #2: use a finite approximation... 


with e.g. variational inference, HMC [Blei 06; Neal 10]

Problem: Wide variety of priors in BNP with
 no or poorly understood finite approximation

In this talk: 1) Two finite approximation types: truncated and non-nested 2) Two truncated forms (7 reps total) that allow finite approximation of (normalized) completely random measures [(N)CRMs] 3) Truncation approximation error analysis

Inference in BNP models

π(dΘ | X) ∝Θ f(X | Θ)π0(dΘ)

Θ

• Option #1: Integrate out the parameter (CRP, IBP, etc.)


issues: care about the parameters, using certain inference algs. (HMC/VB), distributed computation, discrete latent variables instead

• Option #2: use a finite approximation... 


with e.g. variational inference, HMC [Blei 06; Neal 10]

Problem: Wide variety of priors in BNP with
 no or poorly understood finite approximation

In this talk: 1) Two finite approximation types: truncated and non-nested 2) Two truncated forms (7 reps total) that allow finite approximation of (normalized) completely random measures [(N)CRMs] 3) Truncation approximation error analysis 4) One non-nested form for (N)CRMs

Inference in BNP models

π(dΘ | X) ∝Θ f(X | Θ)π0(dΘ)

Θ

Outline

➡ Tractable priors in BNP

• Truncated approximations
• Two forms for sequential representations
• Truncation and error analysis
• Non-nested approximations

Doc 1 (532 words) Doc 2 (210 words) Doc 3 (854 words) Doc 4 (926 words)

The Standard Model in BNP (By Example )

343 189 210 854 342 584

s p

• r

t s politics f

• d

… 0.7 0.5 0.2 …

topic 
 space frequency space Doc 1 (532 words) Doc 2 (210 words) Doc 3 (854 words) Doc 4 (926 words)

The Standard Model in BNP (By Example )

343 189 210 854 342 584

s p

• r

t s politics f

• d

… 0.7 0.5 0.2 …

topic 
 space frequency space Doc 1 (532 words) Doc 2 (210 words) Doc 3 (854 words) Doc 4 (926 words)

The Standard Model in BNP (By Example )

343 189 210 854 342 584

s p

• r

t s politics f

• d

… 0.7 0.5 0.2 …

topic 
 space frequency space Doc 1 (532 words) Doc 2 (210 words) Doc 3 (854 words) Doc 4 (926 words)

The Standard Model in BNP (By Example )

343 189 210 854 342 584

s p

• r

t s politics f

• d

… 0.7 0.5 0.2 …

topic 
 space frequency space Doc 1 (532 words) Doc 2 (210 words) Doc 3 (854 words) Doc 4 (926 words)

The Standard Model in BNP (By Example )

343 189 210 854 342 584

s p

• r

t s politics f

• d

… 0.7 0.5 0.2 … 0.7 sports

topic 
 space frequency space Doc 1 (532 words) Doc 2 (210 words) Doc 3 (854 words) Doc 4 (926 words)

The Standard Model in BNP (By Example )

343 189 210 854 342 584

s p

• r

t s politics f

• d

… 0.7 0.5 0.2 … 0.7 sports

topic 
 space frequency space Doc 1 (532 words) Doc 2 (210 words) Doc 3 (854 words) Doc 4 (926 words)

The Standard Model in BNP (By Example )

343 189 210 854 342 584

s p

• r

t s politics f

• d

… 0.7 0.5 0.2 … 0.7 sports

topic 
 space frequency space Doc 1 (532 words) Doc 2 (210 words) Doc 3 (854 words) Doc 4 (926 words)

The Standard Model in BNP (By Example )

343 189 210 854 342 584

s p

• r

t s politics f

• d

… 0.7 0.5 0.2 … 0.7 sports

topic 
 space frequency space Doc 1 (532 words) Doc 2 (210 words) Doc 3 (854 words) Doc 4 (926 words)

The Standard Model in BNP (By Example )

343 189 210 854 342 584

s p

• r

t s politics f

• d

… 0.7 0.5 0.2 … 0.7 sports

topic 
 space frequency space Doc 1 (532 words) Doc 2 (210 words) Doc 3 (854 words) Doc 4 (926 words)

The Standard Model in BNP (By Example )

343 189 210 854 342 584

s p

• r

t s politics f

• d

… 0.7 0.5 0.2 … 0.7 sports

ϴ is a random discrete measure

• n the topics

Θ

topic 
 space frequency space Doc 1 (532 words) Doc 2 (210 words) Doc 3 (854 words) Doc 4 (926 words)

The Standard Model in BNP (By Example )

343 189 210 854 342 584

s p

• r

t s politics f

• d

… 0.7 0.5 0.2 … 0.7 sports

ϴ is a random discrete measure

• n the topics

Θ

trait
 space rate 
 space Obs 1 Obs 2 Obs 3 Obs 4

The Standard Model in BNP (By Example )

343 189 210 854 342 584 … … “traits” “rates”

ϴ is a random discrete measure

• n the topics

topics traits θ1 θ2 θ3 ψ1 ψ2 ψ3

Θ

trait
 space rate 
 space Obs 1 Obs 2 Obs 3 Obs 4

The Standard Model in BNP (By Example )

343 189 210 854 342 584 … … “traits” “rates”

ϴ is a random discrete measure

• n the topics

topics traits θ1 θ2 θ3 ψ1 ψ2 ψ3

Θ

Poisson processes and (N)CRMs

How do we generate infinitely many trait/rate points ?

trait space rate 
 space

(ψ, θ)

ψ

θ

Poisson processes and (N)CRMs

How do we generate infinitely many trait/rate points ?

Poisson process with intensity measure

[Kingman 93]

trait space rate 
 space

µ(dθ × dψ) = ν(dθ)H(dψ)

(ψ, θ)

ψ

θ

Poisson processes and (N)CRMs

How do we generate infinitely many trait/rate points ?

Poisson process with intensity measure

[Kingman 93]

trait space rate 
 space

µ(dθ × dψ) = ν(dθ)H(dψ)

(ψ, θ)

ψ

θ

Poisson processes and (N)CRMs

How do we generate infinitely many trait/rate points ?

completely random measure (CRM) (e.g. BP, 𝚫P) Poisson process with intensity measure

[Kingman 93]

trait space rate 
 space

Θ

µ(dθ × dψ) = ν(dθ)H(dψ)

(ψ, θ)

Θ = P∞

k=1 θkδψk

ψ

θ

Poisson processes and (N)CRMs

How do we generate infinitely many trait/rate points ?

completely random measure (CRM) (e.g. BP, 𝚫P) Normalize rates: normalized CRM (NCRM) (e.g. DP) Poisson process with intensity measure

[Kingman 93]

trait space rate 
 space

Θ

µ(dθ × dψ) = ν(dθ)H(dψ)

(ψ, θ)

Θ = P∞

k=1 θkδψk

ψ

θ

Poisson processes and (N)CRMs

How do we generate infinitely many trait/rate points ?

completely random measure (CRM) (e.g. BP, 𝚫P) Normalize rates: normalized CRM (NCRM) (e.g. DP) Captures a large class of useful priors in BNP Poisson process with intensity measure

[Kingman 93]

trait space rate 
 space

Θ

µ(dθ × dψ) = ν(dθ)H(dψ)

(ψ, θ)

Θ = P∞

k=1 θkδψk

ψ

θ

Poisson processes and (N)CRMs

How do we generate infinitely many trait/rate points ?

completely random measure (CRM) (e.g. BP, 𝚫P) Normalize rates: normalized CRM (NCRM) (e.g. DP) Captures a large class of useful priors in BNP How do we approximate with finite number of atoms? Poisson process with intensity measure

[Kingman 93]

trait space rate 
 space

Θ

µ(dθ × dψ) = ν(dθ)H(dψ)

(ψ, θ)

Θ = P∞

k=1 θkδψk

ψ

θ

Finite approximation approaches

Ω

Finite approximation approaches

Θ = P∞

k=1 θkδψk

Ω Ω

Finite approximation approaches

ΘK = PK

k=1 θkδψk

Θ = P∞

k=1 θkδψk

Ω Ω

Truncated finite approx.

Finite approximation approaches

ΘK = PK

k=1 θkδψk

ΘK = PK

k=1 θkδψk

Θ = P∞

k=1 θkδψk

Ω Ω

Truncated finite approx. Non-nested finite approx.

Finite approximation approaches

ΘK = PK

k=1 θK,kδψk

ΘK = PK

k=1 θkδψk

ΘK = PK

k=1 θkδψk

Θ = P∞

k=1 θkδψk

Truncated Approximations Truncation Error Bounds Non-nested Approximations DP

✓ ✓ ✓

BP

✓ ✓ ✓

BPP

✓

𝚫P

✓ ✓ ✓

(N)CRM

✓

Past work: finite approximations to BNP priors

Truncated Approximations Truncation Error Bounds Non-nested Approximations DP

✓ ✓ ✓

BP

✓ ✓ ✓

BPP

✓

𝚫P

✓ ✓ ✓

(N)CRM

✓

Past work: finite approximations to BNP priors

[Sethuraman 94] [Roychowdhury 15] [Teh 07] [Paisley 12] [Thibaux 07] [Ferguson 72] [Bondesson 82] [Rosinski 01] [Broderick 14] [Bondesson 82] [Roychowdhury 15] [Ishwaran 01] [Doshi-Velez 09] [Paisley 12] [Broderick 14] [Roychowdhury 15] [Ishwaran 02] [Paisley 16] [Titsias 07]

Truncated Approximations Truncation Error Bounds Non-nested Approximations DP

✓ ✓ ✓

BP

✓ ✓ ✓

BPP

✓

𝚫P

✓ ✓ ✓

(N)CRM

✓

Past work: finite approximations to BNP priors

[Sethuraman 94] [Roychowdhury 15] [Teh 07] [Paisley 12] [Thibaux 07] [Ferguson 72] [Bondesson 82] [Rosinski 01] [Broderick 14] [Bondesson 82] [Roychowdhury 15] [Ishwaran 01] [Doshi-Velez 09] [Paisley 12] [Broderick 14] [Roychowdhury 15] [Ishwaran 02] [Paisley 16] [Titsias 07]

Sparse results for a few priors in BNP

Truncated Approximations Truncation Error Bounds Non-nested Approximations DP

✓ ✓ ✓

BP

✓ ✓ ✓

BPP

✓

𝚫P

✓ ✓ ✓

(N)CRM

✓

Past work: finite approximations to BNP priors

[Sethuraman 94] [Roychowdhury 15] [Teh 07] [Paisley 12] [Thibaux 07] [Ferguson 72] [Bondesson 82] [Rosinski 01] [Broderick 14] [Bondesson 82] [Roychowdhury 15] [Ishwaran 01] [Doshi-Velez 09] [Paisley 12] [Broderick 14] [Roychowdhury 15] [Ishwaran 02] [Paisley 16] [Titsias 07]

Sparse results for a few priors in BNP Incomplete general theory

Outline

• Tractable priors in BNP
• Truncated approximations

➡ Two forms for sequential representations

• Truncation and error analysis
• Non-nested approximations

Ordering of (N)CRM atoms

Ordering of (N)CRM atoms

Θ = P∞

k=1 θkδψk

Ordering of (N)CRM atoms

ΘK = PK

k=1 θkδψk

Θ = P∞

k=1 θkδψk

2 forms for sequential representations

Ordering of (N)CRM atoms

ΘK = PK

k=1 θkδψk

Θ = P∞

k=1 θkδψk

ν(dθ)H(dψ)

Series representation
 function of a homogenous 
 Poisson point process
 (4 versions) 2 forms for sequential representations

Ordering of (N)CRM atoms

[Ferguson and Klass 1972, Bondesson 1982, Rosinski 2001]

ΘK = PK

k=1 θkδψk

Θ = P∞

k=1 θkδψk

ν(dθ)H(dψ)

Series representation
 function of a homogenous 
 Poisson point process
 (4 versions) 2 forms for sequential representations

Ordering of (N)CRM atoms

[Ferguson and Klass 1972, Bondesson 1982, Rosinski 2001]

ΘK = PK

k=1 θkδψk

Θ = P∞

k=1 θkδψk

ν(dθ)H(dψ)

Γk

Series representation
 function of a homogenous 
 Poisson point process
 (4 versions) 2 forms for sequential representations

Ordering of (N)CRM atoms

[Ferguson and Klass 1972, Bondesson 1982, Rosinski 2001]

Vk

i.i.d.

∼ g

ΘK = PK

k=1 θkδψk

Θ = P∞

k=1 θkδψk

ν(dθ)H(dψ)

Γk

Series representation
 function of a homogenous 
 Poisson point process
 (4 versions) 2 forms for sequential representations

Ordering of (N)CRM atoms

[Ferguson and Klass 1972, Bondesson 1982, Rosinski 2001]

Vk

i.i.d.

∼ g

Θ = P∞

k=1 θkδψk

ΘK = PK

k=1 θkδψk

Θ = P∞

k=1 θkδψk

ν(dθ)H(dψ)

Γk

Superposition representation
 infinite sum of CRMs, 
 each with finite # of atoms
 (3 versions) Series representation
 function of a homogenous 
 Poisson point process
 (4 versions) 2 forms for sequential representations

Ordering of (N)CRM atoms

[Ferguson and Klass 1972, Bondesson 1982, Rosinski 2001]

Vk

i.i.d.

∼ g

Θ = P∞

k=1 θkδψk

[James 2014, Broderick et al 2017]

ΘK = PK

k=1 θkδψk

Θ = P∞

k=1 θkδψk

ν(dθ)H(dψ)

Γk

Superposition representation
 infinite sum of CRMs, 
 each with finite # of atoms
 (3 versions) Series representation
 function of a homogenous 
 Poisson point process
 (4 versions) 2 forms for sequential representations

Ordering of (N)CRM atoms

[Ferguson and Klass 1972, Bondesson 1982, Rosinski 2001]

Vk

i.i.d.

∼ g

Θ = P∞

k=1 θkδψk

[James 2014, Broderick et al 2017]

ΘK = PK

k=1 θkδψk

Θ = P∞

k=1 θkδψk

ν(dθ)H(dψ)

Γk

+ + + ⋯ Θ(1)

Θ(2)

Θ(3)

Superposition representation
 infinite sum of CRMs, 
 each with finite # of atoms
 (3 versions) Series representation
 function of a homogenous 
 Poisson point process
 (4 versions) 2 forms for sequential representations

Ordering of (N)CRM atoms

[Ferguson and Klass 1972, Bondesson 1982, Rosinski 2001]

Vk

i.i.d.

∼ g

Θ = P∞

k=1 θkδψk

[James 2014, Broderick et al 2017]

ΘK = PK

k=1 θkδψk

Θ = P∞

k=1 θkδψk

ν(dθ)H(dψ)

Γk

Θ = P∞

k=1 Θ(k)

+ + + ⋯ Θ(1)

Θ(2)

Θ(3)

Series representation
 function of a homogenous 
 Poisson point process
 (4 versions)

Ordering of (N)CRM atoms

Theorem (H., Campbell, How, Broderick).
 Can generate (N)CRMs using all 7 sequential representations ΘK = PK

k=1 θkδψk

Θ = P∞

k=1 θkδψk

2 forms for sequential representations ν(dθ)H(dψ)

Superposition representation
 infinite sum of CRMs, 
 each with finite # of atoms
 (3 versions)

Sequential representation comparison

Why so many representations?

They’re all useful in different circumstances

Sequential representation comparison

Why so many representations?

They’re all useful in different circumstances

Sequential representation comparison

Why so many representations?

Series Reps Superposition Reps B-Rep IL-Rep R-Rep T-Rep DB-Rep PL-Rep SB-Rep Error Bound Decay

✓ ✓ ✓/✗ ✗ ✓ ✓ ✗

Ease of Analysis

✗ ✗✗ ✗ ✗ ✓ ✓ ✓

Generality

✓ ✓ ✓ ✓ ✓ ✓ ✓

Known # Atoms

✓ ✓ ✗ ✗ ✗ ✗ ✗

Given Gamma process:

Sequential representation example

Given Gamma process: Step 1: compute

Sequential representation example

Given Gamma process: Step 1: compute

Sequential representation example

Given Gamma process: Step 1: compute Step 2: compute

Sequential representation example

Given Gamma process: Step 1: compute Step 2: compute

Sequential representation example

Given Gamma process: Exponential(𝜇) density! Step 1: compute Step 2: compute

Sequential representation example

Given Gamma process: Exponential(𝜇) density! Step 3: plug in! Step 1: compute Step 2: compute

Sequential representation example

Outline

✓ Tractable priors in BNP

• Truncated approximations

✓ Two forms for sequential representations ➡ Truncation and error analysis

• Non-nested approximations

How close is our finite approximation?

Choosing between the seven representations

How close is our finite approximation?

Choosing between the seven representations

Π(dΘ | X) ∝Θ f(X | Θ)Π0(dΘ)

How close is our finite approximation?

Choosing between the seven representations

Π(dΘ | X) ∝Θ f(X | Θ)Π0(dΘ)

Truncation error: kpN,∞ pN,Kk1 = 1

2 Z |pN,∞(X) pN,K(X)|dX

How close is our finite approximation?

Choosing between the seven representations

Π(dΘ | X) ∝Θ f(X | Θ)Π0(dΘ)

Truncation error: kpN,∞ pN,Kk1 = 1

2 Z |pN,∞(X) pN,K(X)|dX

truncated full infinite

Θ

ΘK

How close is our finite approximation?

Choosing between the seven representations

Π(dΘ | X) ∝Θ f(X | Θ)Π0(dΘ)

Truncation error: kpN,∞ pN,Kk1 = 1

2 Z |pN,∞(X) pN,K(X)|dX

truncated full infinite

Θ

ΘK

data X data X

f

f

Compare the distribution of the data under full vs. truncated

How close is our finite approximation?

Choosing between the seven representations

Π(dΘ | X) ∝Θ f(X | Θ)Π0(dΘ)

Truncation error: kpN,∞ pN,Kk1 = 1

2 Z |pN,∞(X) pN,K(X)|dX

truncated full infinite

Θ

ΘK

data X data X

f

f

How close is our finite approximation?

Choosing between the seven representations

Π(dΘ | X) ∝Θ f(X | Θ)Π0(dΘ)

Truncation error: kpN,∞ pN,Kk1 = 1

2 Z |pN,∞(X) pN,K(X)|dX

Depends on number of observations N and truncation level K

How close is our finite approximation?

Choosing between the seven representations

Π(dΘ | X) ∝Θ f(X | Θ)Π0(dΘ)

Truncation error: kpN,∞ pN,Kk1 = 1

2 Z |pN,∞(X) pN,K(X)|dX

Depends on number of observations N and truncation level K As N gets larger, error increases

How close is our finite approximation?

Choosing between the seven representations

Π(dΘ | X) ∝Θ f(X | Θ)Π0(dΘ)

Truncation error: kpN,∞ pN,Kk1 = 1

2 Z |pN,∞(X) pN,K(X)|dX

Depends on number of observations N and truncation level K As N gets larger, error increases As K gets larger, error decreases

How close is our finite approximation?

Choosing between the seven representations

Π(dΘ | X) ∝Θ f(X | Θ)Π0(dΘ)

Truncation error: kpN,∞ pN,Kk1 = 1

2 Z |pN,∞(X) pN,K(X)|dX

Depends on number of observations N and truncation level K As N gets larger, error increases As K gets larger, error decreases We develop new upper bounds

How close is our finite approximation?

Choosing between the seven representations

Π(dΘ | X) ∝Θ f(X | Θ)Π0(dΘ)

Truncation error: kpN,∞ pN,Kk1 = 1

2 Z |pN,∞(X) pN,K(X)|dX

Lemma (H., Campbell, How, Broderick).

Protobound

Leads to all the other truncation error bounds in this work

Lemma (H., Campbell, How, Broderick).

Protobound

Leads to all the other truncation error bounds in this work Proposition (HCHB). The protobound is tight

Lemma (H., Campbell, How, Broderick).

Protobound

Leads to all the other truncation error bounds in this work

Lemma (H., Campbell, How, Broderick).

Protobound

Leads to all the other truncation error bounds in this work

Theorem (HCHB). The series

rep error is bounded by

kpN,∞ pN,Kk1  1 e−

R ∞ E[¯ π(τ(V,u+GK))N]du

Lemma (H., Campbell, How, Broderick).

Protobound

Leads to all the other truncation error bounds in this work

Theorem (HCHB). The series

rep error is bounded by

kpN,∞ pN,Kk1  1 e−

R ∞ E[¯ π(τ(V,u+GK))N]du

kpN,∞ pN,Kk1  1 e−

R ∞ ¯ π(θ)Nν+

K(dθ)

Theorem (HCHB). The superposition rep error is bounded by

Given Gamma-Poisson process:

Error bound example

Step 1: bound the integral, where : Given Gamma-Poisson process:

Error bound example

GK ∼ Gamma(K, c)

Step 1: bound the integral, where : Given Gamma-Poisson process:

Integration by parts

Error bound example

GK ∼ Gamma(K, c)

Step 1: bound the integral, where : Given Gamma-Poisson process:

Integration by parts

Error bound example

GK ∼ Gamma(K, c)

Step 1: bound the integral, where : Given Gamma-Poisson process:

Integration by parts Gamma expectation

Error bound example

GK ∼ Gamma(K, c)

Step 1: bound the integral, where : Given Gamma-Poisson process:

Integration by parts Gamma expectation

Step 2: plug in!

Error bound example

1 2kpN,∞ pN,Kk1  1 exp ( Nγ ✓ γλ 1 + γλ ◆K) ⇠ Nγ ✓ γλ 1 + γλ ◆K , K ! 1 GK ∼ Gamma(K, c)

Step 1: bound the integral, where : Given Gamma-Poisson process:

Integration by parts Gamma expectation

Step 2: plug in!

Error bound example

1 2kpN,∞ pN,Kk1  1 exp ( Nγ ✓ γλ 1 + γλ ◆K) ⇠ Nγ ✓ γλ 1 + γλ ◆K , K ! 1 GK ∼ Gamma(K, c) N → ∞, bound → 1

Step 1: bound the integral, where : Given Gamma-Poisson process:

Integration by parts Gamma expectation

Step 2: plug in!

Error bound example

1 2kpN,∞ pN,Kk1  1 exp ( Nγ ✓ γλ 1 + γλ ◆K) ⇠ Nγ ✓ γλ 1 + γλ ◆K , K ! 1 GK ∼ Gamma(K, c) N → ∞, bound → 1

K → ∞, bound → 0

Outline

✓ Tractable priors in BNP ✓ Truncated approximations ✓ Two forms for sequential representations ✓ Truncation and error analysis ➡ Non-nested approximations

Atom weights are independent

Non-nested CRM approximations

ΘK = PK

k=1θK,kδψk,

θK,k

ind

∼ νK

Atom weights are independent

Non-nested CRM approximations

ΘK = PK

k=1θK,kδψk,

θK,k

ind

∼ νK

Intuition: choose νK such that KνK(θ) ≈ ν(θ) and KνK → ν

Atom weights are independent

Non-nested CRM approximations

ΘK = PK

k=1θK,kδψk,

θK,k

ind

∼ νK

0.0 0.5 1.0 1.5 2.0 2.5 3.0 0.5 1 5 10

ν ν ν ν 2ν2 5ν5

Intuition: choose νK such that KνK(θ) ≈ ν(θ) and KνK → ν

Theorem (H., Masoero, Mackey, Broderick). Assume that

Atom weights are independent

Non-nested CRM approximations

ΘK = PK

k=1θK,kδψk,

θK,k

ind

∼ νK

0.0 0.5 1.0 1.5 2.0 2.5 3.0 0.5 1 5 10

ν ν ν ν 2ν2 5ν5

Intuition: choose νK such that KνK(θ) ≈ ν(θ) and KνK → ν

ν(dθ; γ, d, η) = γθ−1−dg(θ)−d

h(θ;η) Z(1−d,η)dθ.

Theorem (H., Masoero, Mackey, Broderick). Assume that

Atom weights are independent

Non-nested CRM approximations

ΘK = PK

k=1θK,kδψk,

θK,k

ind

∼ νK

Then, under mild regularity conditions, when d = 0

νK(dθ) = θ−1+c/Kg(θ)c/K

h(θ;η) Z(c/K,η)dθ,

c , γ h(0;η)

Z(1,η).

where

0.0 0.5 1.0 1.5 2.0 2.5 3.0 0.5 1 5 10

ν ν ν ν 2ν2 5ν5

Intuition: choose νK such that KνK(θ) ≈ ν(θ) and KνK → ν

ν(dθ; γ, d, η) = γθ−1−dg(θ)−d

h(θ;η) Z(1−d,η)dθ.

Theorem (H., Masoero, Mackey, Broderick). Assume that

Atom weights are independent

Non-nested CRM approximations

ΘK = PK

k=1θK,kδψk,

θK,k

ind

∼ νK

Then, under mild regularity conditions, when d = 0

νK(dθ) = θ−1+c/Kg(θ)c/K

h(θ;η) Z(c/K,η)dθ,

c , γ h(0;η)

Z(1,η).

where

0.0 0.5 1.0 1.5 2.0 2.5 3.0 0.5 1 5 10

ν ν ν ν 2ν2 5ν5

Intuition: choose νK such that KνK(θ) ≈ ν(θ) and KνK → ν

ν(dθ; γ, d, η) = γθ−1−dg(θ)−d

h(θ;η) Z(1−d,η)dθ.

Theorem (H., Masoero, Mackey, Broderick). Assume that

Atom weights are independent

Non-nested CRM approximations

ΘK = PK

k=1θK,kδψk,

θK,k

ind

∼ νK

Then, under mild regularity conditions, when d = 0

νK(dθ) = θ−1+c/Kg(θ)c/K

h(θ;η) Z(c/K,η)dθ,

c , γ h(0;η)

Z(1,η).

where

0.0 0.5 1.0 1.5 2.0 2.5 3.0 0.5 1 5 10

ν ν ν ν 2ν2 5ν5

Intuition: choose νK such that KνK(θ) ≈ ν(θ) and KνK → ν

ν(dθ; γ, d, η) = γθ−1−dg(θ)−d

h(θ;η) Z(1−d,η)dθ.

Theorem (H., Masoero, Mackey, Broderick). Assume that

Atom weights are independent

Non-nested CRM approximations

ΘK = PK

k=1θK,kδψk,

θK,k

ind

∼ νK

Then, under mild regularity conditions, when d = 0

νK(dθ) = θ−1+c/Kg(θ)c/K

h(θ;η) Z(c/K,η)dθ,

c , γ h(0;η)

Z(1,η).

where

0.0 0.5 1.0 1.5 2.0 2.5 3.0 0.5 1 5 10

ν ν ν ν 2ν2 5ν5

Intuition: choose νK such that KνK(θ) ≈ ν(θ) and KνK → ν

ν(dθ; γ, d, η) = γθ−1−dg(θ)−d

h(θ;η) Z(1−d,η)dθ.

Outline

✓ Tractable priors in BNP ✓ Truncated approximations ✓ Two forms for sequential representations ✓ Truncation and error analysis ✓ Non-nested approximations

Conclusion

Truncated Approximations Truncation Error Bounds Non-nested Approximations DP

✓ ✓ ✓

BP

✓ ✓ ✓

BPP

✓

𝚫P

✓ ✓ ✓

(N)CRM

✓ Previous Work

Conclusion

Truncated Approximations Truncation Error Bounds Non-nested Approximations DP

✓ ✓ ✓

BP

✓ ✓ ✓

BPP

✓ ✓ ✓

𝚫P

✓ ✓ ✓

(N)CRM

✓ ✓ ✓ Our Work

Conclusion

Truncated Approximations Truncation Error Bounds Non-nested Approximations DP

✓ ✓ ✓

BP

✓ ✓ ✓

BPP

✓ ✓ ✓

𝚫P

✓ ✓ ✓

(N)CRM

✓ ✓ ✓ Our Work

• Large family of BNP priors that admit efficient inference

Conclusion

Truncated Approximations Truncation Error Bounds Non-nested Approximations DP

✓ ✓ ✓

BP

✓ ✓ ✓

BPP

✓ ✓ ✓

𝚫P

✓ ✓ ✓

(N)CRM

✓ ✓ ✓ Our Work

• Large family of BNP priors that admit efficient inference
• Use of “modern” inference methods (e.g. HMC and VB)

Conclusion

Truncated Approximations Truncation Error Bounds Non-nested Approximations DP

✓ ✓ ✓

BP

✓ ✓ ✓

BPP

✓ ✓ ✓

𝚫P

✓ ✓ ✓

(N)CRM

✓ ✓ ✓ Our Work

• Large family of BNP priors that admit efficient inference
• Use of “modern” inference methods (e.g. HMC and VB)
• Trade off computational efficiency and statistical accuracy

Conclusion

• Large family of BNP priors that admit efficient inference
• Use of “modern” inference methods (e.g. HMC and VB)
• Trade off computational efficiency and statistical accuracy
• J. Huggins*, T. Campbell*, J. How, T. Broderick 


Truncated random measures
 Bernoulli, to appear
 Available online: https://arxiv.org/abs/1603.00861

• J. Huggins, L. Masoero, L. Mackey, T. Broderick


Generic finite approximations for practical Bayesian nonparametrics
 NIPS Workshop on Advances in Approximate Bayesian Inference, 2017
 Available online: http://approximateinference.org/2017/accepted/HugginsEtAl2017.pdf