Using Bagged Posteriors for Robust Inference Jonathan Huggins - - PowerPoint PPT Presentation

Jonathan Huggins

Harvard University

Joint work with Jeff Miller

1

Using Bagged Posteriors for Robust Inference

2

Bagged posterior corrects for model misspecification

[H & Miller 2019]

2

Bagged posterior corrects for model misspecification

• Goal: predict future insurance claims based on (real) historic data ⇒

[H & Miller 2019]

2

Bagged posterior corrects for model misspecification

• Goal: predict future insurance claims based on (real) historic data ⇒
• Try Bayesian inference with (non-trivial) model (data is 10 time series) ⇒

standard
 posterior [H & Miller 2019]

2

Bagged posterior corrects for model misspecification

• Goal: predict future insurance claims based on (real) historic data ⇒
• Try Bayesian inference with (non-trivial) model (data is 10 time series) ⇒
• Problem: uncertainty not well-calibrated because model is wrong⇒

true amount standard
 posterior [H & Miller 2019]

2

Bagged posterior corrects for model misspecification

• Goal: predict future insurance claims based on (real) historic data ⇒
• Try Bayesian inference with (non-trivial) model (data is 10 time series) ⇒
• Problem: uncertainty not well-calibrated because model is wrong⇒
• Alternative: the bootstrap ⇒ too little data

true amount standard
 posterior [H & Miller 2019] bootstrap

2

Bagged posterior corrects for model misspecification

• Goal: predict future insurance claims based on (real) historic data ⇒
• Try Bayesian inference with (non-trivial) model (data is 10 time series) ⇒
• Problem: uncertainty not well-calibrated because model is wrong⇒
• Alternative: the bootstrap ⇒ too little data
• Solution: use the bagged posterior (BayesBag)

true amount bagged posterior standard
 posterior [H & Miller 2019] bootstrap

Agenda

3

➡ BayesBag for parameter inference

(and prediction)

• BayesBag theory and methodology
• BayesBag for model selection

Bayesian inference

4

• Goal: learn about unobserved phenomenon

(parameter) of interest 𝜄 [e.g. future claims]

Bayesian inference

4

• Goal: learn about unobserved phenomenon

(parameter) of interest 𝜄 [e.g. future claims]

• Prior beliefs 𝜌₀(𝜄) about the phenomenon

Bayesian inference

4

prior

θ

• Goal: learn about unobserved phenomenon

(parameter) of interest 𝜄 [e.g. future claims]

• Prior beliefs 𝜌₀(𝜄) about the phenomenon
• Observe data Y via model p(Y | 𝜄)

Bayesian inference

4

prior likelihood

θ

• Goal: learn about unobserved phenomenon

(parameter) of interest 𝜄 [e.g. future claims]

• Prior beliefs 𝜌₀(𝜄) about the phenomenon
• Observe data Y via model p(Y | 𝜄)
• Combine prior & likelihood to form posterior:


Bayesian inference

4

π(θ | Y ) ∝ p(Y | θ)π0(θ)

prior likelihood posterior

θ

• Goal: learn about unobserved phenomenon

(parameter) of interest 𝜄 [e.g. future claims]

• Prior beliefs 𝜌₀(𝜄) about the phenomenon
• Observe data Y via model p(Y | 𝜄)
• Combine prior & likelihood to form posterior:


• Benefits: coherent belief updates, uncertainty

quantification, flexible modeling, and more

Bayesian inference

4

π(θ | Y ) ∝ p(Y | θ)π0(θ)

prior likelihood posterior

θ

• Goal: learn about unobserved phenomenon

(parameter) of interest 𝜄 [e.g. future claims]

• Prior beliefs 𝜌₀(𝜄) about the phenomenon
• Observe data Y via model p(Y | 𝜄)
• Combine prior & likelihood to form posterior:


• Benefits: coherent belief updates, uncertainty

quantification, flexible modeling, and more

• Assumption #1: measurement model correct:
• bserved Y has distribution p(Y | 𝜄true)

Bayesian inference

4

π(θ | Y ) ∝ p(Y | θ)π0(θ)

prior likelihood posterior

θ

• Goal: learn about unobserved phenomenon

(parameter) of interest 𝜄 [e.g. future claims]

• Prior beliefs 𝜌₀(𝜄) about the phenomenon
• Observe data Y via model p(Y | 𝜄)
• Combine prior & likelihood to form posterior:


• Benefits: coherent belief updates, uncertainty

quantification, flexible modeling, and more

• Assumption #1: measurement model correct:
• bserved Y has distribution p(Y | 𝜄true)
• Assumption #2: Prior puts sufficient mass on

true parameter 𝜄true

Bayesian inference

4

π(θ | Y ) ∝ p(Y | θ)π0(θ)

prior likelihood posterior

θ

Bootstrapping

Ptrue

[Efron 1979] 5

Yboot

Y

Bootstrapping

Ptrue

[Efron 1979] 5

Yboot

• Data Y = (Y1, …, Yn), where Yi ~ Ptrue

Y

Bootstrapping

Ptrue

[Efron 1979] 5

Yboot

mean(Y )

• Data Y = (Y1, …, Yn), where Yi ~ Ptrue
• Interested in parameter that best

explains distribution [e.g. mean of independent normal observations]

Y

Bootstrapping

Ptrue

[Efron 1979] 5

Yboot

mean(Y )

distribution of… mean(Y )

• Data Y = (Y1, …, Yn), where Yi ~ Ptrue
• Interested in parameter that best

explains distribution [e.g. mean of independent normal observations]

• Want sampling uncertainty [e.g.

distribution of mean(Y) under Ptrue]

Y

Bootstrapping

Ptrue

[Efron 1979] 5

Yboot

mean(Y )

distribution of… mean(Y )

• Data Y = (Y1, …, Yn), where Yi ~ Ptrue
• Interested in parameter that best

explains distribution [e.g. mean of independent normal observations]

• Want sampling uncertainty [e.g.

distribution of mean(Y) under Ptrue]

• Bootstrap: replace Ptrue with Pn

Pn

Y

Bootstrapping

Ptrue

[Efron 1979] 5

Yboot

mean(Y )

distribution of… mean(Y )

• Data Y = (Y1, …, Yn), where Yi ~ Ptrue
• Interested in parameter that best

explains distribution [e.g. mean of independent normal observations]

• Want sampling uncertainty [e.g.

distribution of mean(Y) under Ptrue]

• Bootstrap: replace Ptrue with Pn

Pn

Y

Bootstrapping

Ptrue

[Efron 1979] 5

Yboot

mean(Y )

distribution of… mean(Y )

• Data Y = (Y1, …, Yn), where Yi ~ Ptrue
• Interested in parameter that best

explains distribution [e.g. mean of independent normal observations]

• Want sampling uncertainty [e.g.

distribution of mean(Y) under Ptrue]

• Bootstrap: replace Ptrue with Pn
• Sample B bootstrap datasets to get

empirical distribution [e.g. mean(Yboot)]

Pn

Y

mean(Y (1)

boot)

mean(Y (2)

boot) · · · mean(Y (B) boot)

Bootstrapping

Ptrue

[Efron 1979] 5

Yboot

mean(Y )

distribution of… mean(Y )

• Data Y = (Y1, …, Yn), where Yi ~ Ptrue
• Interested in parameter that best

explains distribution [e.g. mean of independent normal observations]

• Want sampling uncertainty [e.g.

distribution of mean(Y) under Ptrue]

• Bootstrap: replace Ptrue with Pn
• Sample B bootstrap datasets to get

empirical distribution [e.g. mean(Yboot)]

Pn

Y

mean(Y (1)

boot)

mean(Y (2)

boot) · · · mean(Y (B) boot)

mean(Yboot) given Y

Bootstrapping

Ptrue

[Efron 1979] 5

Yboot

mean(Y )

distribution of… mean(Y )

• Data Y = (Y1, …, Yn), where Yi ~ Ptrue
• Interested in parameter that best

explains distribution [e.g. mean of independent normal observations]

• Want sampling uncertainty [e.g.

distribution of mean(Y) under Ptrue]

• Bootstrap: replace Ptrue with Pn
• Sample B bootstrap datasets to get

empirical distribution [e.g. mean(Yboot)]

• Benefits: no assumptions about Ptrue,

easy to use, can parallelize across B

Pn

Y

mean(Y (1)

boot)

mean(Y (2)

boot) · · · mean(Y (B) boot)

mean(Yboot) given Y

Bootstrapping

Ptrue

[Efron 1979] 5

Yboot

mean(Y )

distribution of… mean(Y )

• Data Y = (Y1, …, Yn), where Yi ~ Ptrue
• Interested in parameter that best

explains distribution [e.g. mean of independent normal observations]

• Want sampling uncertainty [e.g.

distribution of mean(Y) under Ptrue]

• Bootstrap: replace Ptrue with Pn
• Sample B bootstrap datasets to get

empirical distribution [e.g. mean(Yboot)]

• Benefits: no assumptions about Ptrue,

easy to use, can parallelize across B

• Challenges: B large (1,000-100,000),

finite-sample properties

Pn

Y

mean(Y (1)

boot)

mean(Y (2)

boot) · · · mean(Y (B) boot)

mean(Yboot) given Y

Bootstrapping Bayes

[Douady et al. 2003, Bühlmann 2014, H & Miller 2019] 6

Ptrue

Yboot

Pn

mean(Y )

Y

Bootstrapping Bayes

[Douady et al. 2003, Bühlmann 2014, H & Miller 2019] 6

• Recall: posterior given data Y is

denoted π(𝜄 | Y )

standard 
 posterior uncertainty about true mean

Ptrue

Yboot

Pn

mean(Y )

Y

Bootstrapping Bayes

[Douady et al. 2003, Bühlmann 2014, H & Miller 2019] 6

• Recall: posterior given data Y is

denoted π(𝜄 | Y )

• BayesBag method: Sample B

bootstrap datasets and average

• ver posteriors


standard 
 posterior uncertainty about true mean bagged posterior

Ptrue

Yboot

Pn

mean(Y )

Y

πBB(θ | Y ) = 1 B

B

X

b=1

π(θ | Y (b)

boot)

Bootstrapping Bayes

[Douady et al. 2003, Bühlmann 2014, H & Miller 2019] 6

• Recall: posterior given data Y is

denoted π(𝜄 | Y )

• BayesBag method: Sample B

bootstrap datasets and average

• ver posteriors


• Same benefits as bootstrap: no

correct model assumption, easy- to-use, can parallelize across B

standard 
 posterior uncertainty about true mean bagged posterior

Ptrue

Yboot

Pn

mean(Y )

Y

πBB(θ | Y ) = 1 B

B

X

b=1

π(θ | Y (b)

boot)

Bootstrapping Bayes

[Douady et al. 2003, Bühlmann 2014, H & Miller 2019] 6

• Recall: posterior given data Y is

denoted π(𝜄 | Y )

• BayesBag method: Sample B

bootstrap datasets and average

• ver posteriors


• Same benefits as bootstrap: no

correct model assumption, easy- to-use, can parallelize across B

• Suffices to take B = 50 or 100

standard 
 posterior uncertainty about true mean bagged posterior

Ptrue

Yboot

Pn

mean(Y )

Y

πBB(θ | Y ) = 1 B

B

X

b=1

π(θ | Y (b)

boot)

Bootstrapping Bayes

[Douady et al. 2003, Bühlmann 2014, H & Miller 2019] 6

• Recall: posterior given data Y is

denoted π(𝜄 | Y )

• BayesBag method: Sample B

bootstrap datasets and average

• ver posteriors


• Same benefits as bootstrap: no

correct model assumption, easy- to-use, can parallelize across B

• Suffices to take B = 50 or 100
• Finite-sample benefits of Bayes

standard 
 posterior uncertainty about true mean bagged posterior

Ptrue

Yboot

Pn

mean(Y )

Y

πBB(θ | Y ) = 1 B

B

X

b=1

π(θ | Y (b)

boot)

• Assumed model: Gaussian linear regression with conjugate priors
• Data-generating distribution Ptrue: can be correct or misspecified
• θopt = optimal parameter that is “closest” to Ptrue
• Performance metric is difference in log posterior density at θopt:

log πBB(θopt | Y ) − log π(θopt | Y )

Better parameter inference with BayesBag

7 [H & Miller 2019]

• Assumed model: Gaussian linear regression with conjugate priors
• Data-generating distribution Ptrue: can be correct or misspecified
• θopt = optimal parameter that is “closest” to Ptrue
• Performance metric is difference in log posterior density at θopt:

log πBB(θopt | Y ) − log π(θopt | Y )

Better parameter inference with BayesBag

7 [H & Miller 2019]

• Assumed model: Gaussian linear regression with conjugate priors
• Data-generating distribution Ptrue: can be correct or misspecified
• θopt = optimal parameter that is “closest” to Ptrue
• Performance metric is difference in log posterior density at θopt:

log πBB(θopt | Y ) − log π(θopt | Y )

Better parameter inference with BayesBag

7 [H & Miller 2019]

• Assumed model: Gaussian linear regression with conjugate priors
• Data-generating distribution Ptrue: can be correct or misspecified
• θopt = optimal parameter that is “closest” to Ptrue
• Performance metric is difference in log posterior density at θopt:

log πBB(θopt | Y ) − log π(θopt | Y )

Better parameter inference with BayesBag

7 [H & Miller 2019]

• Assumed model: Gaussian linear regression with conjugate priors
• Data-generating distribution Ptrue: can be correct or misspecified
• θopt = optimal parameter that is “closest” to Ptrue
• Performance metric is difference in log posterior density at θopt:

log πBB(θopt | Y ) − log π(θopt | Y )

Better parameter inference with BayesBag

7 [H & Miller 2019]

• Assumed model: Gaussian linear regression with conjugate priors
• Data-generating distribution Ptrue: can be correct or misspecified
• θopt = optimal parameter that is “closest” to Ptrue
• Performance metric is difference in log posterior density at θopt:

log πBB(θopt | Y ) − log π(θopt | Y )

Better parameter inference with BayesBag

7

better model correct model incorrect data-generating distribution

[H & Miller 2019]

Agenda

8

• BayesBag for parameter inference

(and prediction)

➡ BayesBag theory and methodology

• BayesBag for model selection

Var(ϑBB | Y ) = E

• Var(ϑBB | Yboot)

| {z } expected posterior variance + Var

• E(ϑBB | Yboot)

| {z } variance of posterior mean

BayesBag incorporates model- and sampling-based uncertainty

9 [H & Miller 2019]

Var(ϑBB | Y ) = E

• Var(ϑBB | Yboot)

| {z } expected posterior variance + Var

• E(ϑBB | Yboot)

| {z } variance of posterior mean

BayesBag incorporates model- and sampling-based uncertainty

9 [H & Miller 2019]

Var{ˆ θ(Yboot)}

Bootstrap variance:

Var(ϑBB | Y ) = E

• Var(ϑBB | Yboot)

| {z } expected posterior variance + Var

• E(ϑBB | Yboot)

| {z } variance of posterior mean

point estimate

BayesBag incorporates model- and sampling-based uncertainty

9 [H & Miller 2019]

Var{ˆ θ(Yboot)}

Bootstrap variance:

Var(ϑBB | Y ) = E

• Var(ϑBB | Yboot)

| {z } expected posterior variance + Var

• E(ϑBB | Yboot)

| {z } variance of posterior mean

point estimate

BayesBag incorporates model- and sampling-based uncertainty

9 [H & Miller 2019]

Var{ˆ θ(Yboot)}

Bootstrap variance:

sampling uncertainty

Var(ϑBB | Y ) = E

• Var(ϑBB | Yboot)

| {z } expected posterior variance + Var

• E(ϑBB | Yboot)

| {z } variance of posterior mean

point estimate

BayesBag incorporates model- and sampling-based uncertainty

9 [H & Miller 2019]

Var{ˆ θ(Yboot)}

Bootstrap variance:

sampling uncertainty

Sample from posterior: ϑ ∼ π(θ | Y )

Var(ϑBB | Y ) = E

• Var(ϑBB | Yboot)

| {z } expected posterior variance + Var

• E(ϑBB | Yboot)

| {z } variance of posterior mean

point estimate

BayesBag incorporates model- and sampling-based uncertainty

9 [H & Miller 2019]

Var{ˆ θ(Yboot)}

Bootstrap variance:

sampling uncertainty

Var(ϑ | Y )

Posterior variance: Sample from posterior: ϑ ∼ π(θ | Y )

Var(ϑBB | Y ) = E

• Var(ϑBB | Yboot)

| {z } expected posterior variance + Var

• E(ϑBB | Yboot)

| {z } variance of posterior mean

point estimate

BayesBag incorporates model- and sampling-based uncertainty

9 [H & Miller 2019]

Var{ˆ θ(Yboot)}

Bootstrap variance:

sampling uncertainty

Var(ϑ | Y )

Posterior variance: Sample from posterior: ϑ ∼ π(θ | Y )

model-based uncertainty

Var(ϑBB | Y ) = E

• Var(ϑBB | Yboot)

| {z } expected posterior variance + Var

• E(ϑBB | Yboot)

| {z } variance of posterior mean

point estimate

BayesBag incorporates model- and sampling-based uncertainty

9 [H & Miller 2019]

ϑBB ∼ πBB(θ | Y )

Sample from BayesBag posterior:

Var{ˆ θ(Yboot)}

Bootstrap variance:

sampling uncertainty

Var(ϑ | Y )

Posterior variance: Sample from posterior: ϑ ∼ π(θ | Y )

model-based uncertainty

Var(ϑBB | Y ) = E

• Var(ϑBB | Yboot)

| {z } expected posterior variance + Var

• E(ϑBB | Yboot)

| {z } variance of posterior mean

point estimate

BayesBag incorporates model- and sampling-based uncertainty

9 [H & Miller 2019]

ϑBB ∼ πBB(θ | Y )

Sample from BayesBag posterior:

Var{ˆ θ(Yboot)}

Bootstrap variance:

sampling uncertainty

Var(ϑ | Y )

Posterior variance: Sample from posterior: ϑ ∼ π(θ | Y )

model-based uncertainty

BayesBag posterior variance:

Var(ϑBB | Y ) = E

• Var(ϑBB | Yboot)

| {z } expected posterior variance + Var

• E(ϑBB | Yboot)

| {z } variance of posterior mean

point estimate

BayesBag incorporates model- and sampling-based uncertainty

9 [H & Miller 2019]

ϑBB ∼ πBB(θ | Y )

Sample from BayesBag posterior:

Var{ˆ θ(Yboot)}

Bootstrap variance:

sampling uncertainty

Var(ϑ | Y )

Posterior variance: Sample from posterior: ϑ ∼ π(θ | Y )

model-based uncertainty

BayesBag posterior variance:

Var(ϑBB | Y ) = E

• Var(ϑBB | Yboot)

| {z } expected posterior variance + Var

• E(ϑBB | Yboot)

| {z } variance of posterior mean

point estimate

BayesBag incorporates model- and sampling-based uncertainty

9 [H & Miller 2019]

ϑBB ∼ πBB(θ | Y )

Sample from BayesBag posterior:

Var{ˆ θ(Yboot)}

Bootstrap variance:

sampling uncertainty

Var(ϑ | Y )

Posterior variance: Sample from posterior: ϑ ∼ π(θ | Y )

model-based uncertainty

BayesBag posterior variance:

Var(ϑBB | Y ) = E

• Var(ϑBB | Yboot)

| {z } expected posterior variance + Var

• E(ϑBB | Yboot)

| {z } variance of posterior mean

point estimate

BayesBag incorporates model- and sampling-based uncertainty

9 [H & Miller 2019]

ϑBB ∼ πBB(θ | Y )

Sample from BayesBag posterior:

Var{ˆ θ(Yboot)}

Bootstrap variance:

sampling uncertainty

Var(ϑ | Y )

Posterior variance: Sample from posterior: ϑ ∼ π(θ | Y )

model-based uncertainty

BayesBag posterior variance:

Var(ϑBB | Y ) = E

• Var(ϑBB | Yboot)

| {z } expected posterior variance + Var

• E(ϑBB | Yboot)

| {z } variance of posterior mean

point estimate

BayesBag incorporates model- and sampling-based uncertainty

9 [H & Miller 2019]

ϑBB ∼ πBB(θ | Y )

Sample from BayesBag posterior:

Var{ˆ θ(Yboot)}

Bootstrap variance:

sampling uncertainty

Var(ϑ | Y )

Posterior variance: Sample from posterior: ϑ ∼ π(θ | Y )

model-based uncertainty

BayesBag posterior variance:

point estimate

Var(ϑBB | Y ) = E

• Var(ϑBB | Yboot)

| {z } expected posterior variance + Var

• E(ϑBB | Yboot)

| {z } variance of posterior mean

point estimate

BayesBag incorporates model- and sampling-based uncertainty

9 [H & Miller 2019]

ϑBB ∼ πBB(θ | Y )

Sample from BayesBag posterior:

Var{ˆ θ(Yboot)}

Bootstrap variance:

sampling uncertainty

Var(ϑ | Y )

Posterior variance: Sample from posterior: ϑ ∼ π(θ | Y )

model-based uncertainty

BayesBag posterior variance:

point estimate

Var(ϑBB | Y ) = E

• Var(ϑBB | Yboot)

| {z } expected posterior variance + Var

• E(ϑBB | Yboot)

| {z } variance of posterior mean

point estimate

BayesBag incorporates model- and sampling-based uncertainty

9 [H & Miller 2019]

ϑBB ∼ πBB(θ | Y )

Sample from BayesBag posterior:

Var{ˆ θ(Yboot)}

Bootstrap variance:

sampling uncertainty

Var(ϑ | Y )

Posterior variance: Sample from posterior: ϑ ∼ π(θ | Y )

model-based uncertainty

BayesBag posterior variance:

BayesBag incorporates model- and sampling-based uncertainty

10 [H & Miller 2019]

BayesBag incorporates model- and sampling-based uncertainty

10 [H & Miller 2019]

• Summarizing the previous slide…

BayesBag incorporates model- and sampling-based uncertainty

10 [H & Miller 2019]

• Summarizing the previous slide…

Posterior variance = model-based uncertainty

BayesBag incorporates model- and sampling-based uncertainty

10 [H & Miller 2019]

• Summarizing the previous slide…

Posterior variance = model-based uncertainty Bootstrap variance = sampling-based uncertainty

BayesBag incorporates model- and sampling-based uncertainty

10 [H & Miller 2019]

• Summarizing the previous slide…

Posterior variance = model-based uncertainty Bootstrap variance = sampling-based uncertainty BayesBag variance = model-based + sampling-based uncertainty

bagged posterior standard 
 posterior bootstrap distribution uncertainty about mean(Yobs)

BayesBag incorporates model- and sampling-based uncertainty

10 [H & Miller 2019]

• Summarizing the previous slide…

Posterior variance = model-based uncertainty Bootstrap variance = sampling-based uncertainty BayesBag variance = model-based + sampling-based uncertainty

bagged posterior standard 
 posterior bootstrap distribution uncertainty about mean(Yobs)

BayesBag incorporates model- and sampling-based uncertainty

10 [H & Miller 2019]

• Summarizing the previous slide…

Posterior variance = model-based uncertainty Bootstrap variance = sampling-based uncertainty BayesBag variance = model-based + sampling-based uncertainty

• Model correct: model-based uncertainty = sampling-based uncertainty

bagged posterior standard 
 posterior bootstrap distribution uncertainty about mean(Yobs)

BayesBag incorporates model- and sampling-based uncertainty

10 [H & Miller 2019]

• Summarizing the previous slide…

Posterior variance = model-based uncertainty Bootstrap variance = sampling-based uncertainty BayesBag variance = model-based + sampling-based uncertainty

• Model correct: model-based uncertainty = sampling-based uncertainty
• Posterior and bootstrap variances correct

bagged posterior standard 
 posterior bootstrap distribution uncertainty about mean(Yobs)

BayesBag incorporates model- and sampling-based uncertainty

10 [H & Miller 2019]

• Summarizing the previous slide…

Posterior variance = model-based uncertainty Bootstrap variance = sampling-based uncertainty BayesBag variance = model-based + sampling-based uncertainty

• Model correct: model-based uncertainty = sampling-based uncertainty
• Posterior and bootstrap variances correct
• BayesBag variance double-counts true uncertainty (conservative)

bagged posterior standard 
 posterior bootstrap distribution uncertainty about mean(Yobs)

BayesBag incorporates model- and sampling-based uncertainty

10 [H & Miller 2019]

• Summarizing the previous slide…

Posterior variance = model-based uncertainty Bootstrap variance = sampling-based uncertainty BayesBag variance = model-based + sampling-based uncertainty

• Model correct: model-based uncertainty = sampling-based uncertainty
• Posterior and bootstrap variances correct
• BayesBag variance double-counts true uncertainty (conservative)
• Model incorrect: model-based uncertainty ≪ sampling-based uncertainty

bagged posterior standard 
 posterior bootstrap distribution uncertainty about mean(Yobs)

BayesBag incorporates model- and sampling-based uncertainty

10 [H & Miller 2019]

• Summarizing the previous slide…

Posterior variance = model-based uncertainty Bootstrap variance = sampling-based uncertainty BayesBag variance = model-based + sampling-based uncertainty

• Model correct: model-based uncertainty = sampling-based uncertainty
• Posterior and bootstrap variances correct
• BayesBag variance double-counts true uncertainty (conservative)
• Model incorrect: model-based uncertainty ≪ sampling-based uncertainty
• Posterior variance far 


too small

bagged posterior standard 
 posterior bootstrap distribution uncertainty about mean(Yobs)

BayesBag incorporates model- and sampling-based uncertainty

10 [H & Miller 2019]

• Summarizing the previous slide…

Posterior variance = model-based uncertainty Bootstrap variance = sampling-based uncertainty BayesBag variance = model-based + sampling-based uncertainty

• Model correct: model-based uncertainty = sampling-based uncertainty
• Posterior and bootstrap variances correct
• BayesBag variance double-counts true uncertainty (conservative)
• Model incorrect: model-based uncertainty ≪ sampling-based uncertainty
• Posterior variance far 


too small

• BayesBag variance


appropriately calibrated

Diagnosing model–data mismatch

11 [H & Miller 2019]

Posterior variance = model-based uncertainty BayesBag variance = model-based + sampling-based uncertainty

• Mismatch index I can diagnose when 


data disagrees with assumed model

Diagnosing model–data mismatch

11 [H & Miller 2019]

Posterior variance = model-based uncertainty BayesBag variance = model-based + sampling-based uncertainty

• Mismatch index I can diagnose when 


data disagrees with assumed model

• -1 < I < 1

Diagnosing model–data mismatch

11 [H & Miller 2019]

Posterior variance = model-based uncertainty BayesBag variance = model-based + sampling-based uncertainty

• Mismatch index I can diagnose when 


data disagrees with assumed model

• -1 < I < 1
• I ≈ 0: no disagreement

Diagnosing model–data mismatch

11 [H & Miller 2019]

Posterior variance = model-based uncertainty BayesBag variance = model-based + sampling-based uncertainty

• Mismatch index I can diagnose when 


data disagrees with assumed model

• -1 < I < 1
• I ≈ 0: no disagreement
• I > 0: posterior overconfident

Diagnosing model–data mismatch

11 [H & Miller 2019]

Posterior variance = model-based uncertainty BayesBag variance = model-based + sampling-based uncertainty

• Mismatch index I can diagnose when 


data disagrees with assumed model

• -1 < I < 1
• I ≈ 0: no disagreement
• I > 0: posterior overconfident
• I < 0: posterior under-confident

Diagnosing model–data mismatch

11 [H & Miller 2019]

Posterior variance = model-based uncertainty BayesBag variance = model-based + sampling-based uncertainty

• Mismatch index I can diagnose when 


data disagrees with assumed model

• -1 < I < 1
• I ≈ 0: no disagreement
• I > 0: posterior overconfident
• I < 0: posterior under-confident

Diagnosing model–data mismatch

11 [H & Miller 2019]

mismatch index ≈ 1 standard
 posterior bagged posterior true amount

Posterior variance = model-based uncertainty BayesBag variance = model-based + sampling-based uncertainty

• Mismatch index I can diagnose when 


data disagrees with assumed model

• -1 < I < 1
• I ≈ 0: no disagreement
• I > 0: posterior overconfident
• I < 0: posterior under-confident
• Model criticism: mismatch index 


indicates when model needs improvement

Diagnosing model–data mismatch

11 [H & Miller 2019]

mismatch index ≈ 1 standard
 posterior bagged posterior true amount

Posterior variance = model-based uncertainty BayesBag variance = model-based + sampling-based uncertainty

• Mismatch index I can diagnose when 


data disagrees with assumed model

• -1 < I < 1
• I ≈ 0: no disagreement
• I > 0: posterior overconfident
• I < 0: posterior under-confident
• Model criticism: mismatch index 


indicates when model needs improvement

• Mismatch index can also detect problems with the prior

Diagnosing model–data mismatch

11 [H & Miller 2019]

mismatch index ≈ 1 standard
 posterior bagged posterior true amount

Posterior variance = model-based uncertainty BayesBag variance = model-based + sampling-based uncertainty

1) compute standard posterior π(· | Y ) – e.g., use MCMC to get approximate samples θ(1), . . . , θ(T) from π(· | Y ) 2) compute bagged posterior πBB(· | Y ) using B ≈ 50 bootstrap datasets – e.g., use MCMC to get approximate samples θ∗

(b,1), . . . , θ∗ (b,T) from π(· | Y ∗ (b))

for b = 1, . . . , B if Gaussian approximation to standard and bagged posteriors decent then 3a) compute mismatch index I 3b) if I ' .2, consider refining the model and returning to step 1 4) output bagged posterior computed in step 2

BayesBag in practice

12 [H & Miller 2019]

1) compute standard posterior π(· | Y ) – e.g., use MCMC to get approximate samples θ(1), . . . , θ(T) from π(· | Y ) 2) compute bagged posterior πBB(· | Y ) using B ≈ 50 bootstrap datasets – e.g., use MCMC to get approximate samples θ∗

(b,1), . . . , θ∗ (b,T) from π(· | Y ∗ (b))

for b = 1, . . . , B if Gaussian approximation to standard and bagged posteriors decent then 3a) compute mismatch index I 3b) if I ' .2, consider refining the model and returning to step 1 4) output bagged posterior computed in step 2

BayesBag in practice

12 [H & Miller 2019]

1) compute standard posterior π(· | Y ) – e.g., use MCMC to get approximate samples θ(1), . . . , θ(T) from π(· | Y ) 2) compute bagged posterior πBB(· | Y ) using B ≈ 50 bootstrap datasets – e.g., use MCMC to get approximate samples θ∗

(b,1), . . . , θ∗ (b,T) from π(· | Y ∗ (b))

for b = 1, . . . , B if Gaussian approximation to standard and bagged posteriors decent then 3a) compute mismatch index I 3b) if I ' .2, consider refining the model and returning to step 1 4) output bagged posterior computed in step 2

BayesBag in practice

12 [H & Miller 2019]

1) compute standard posterior π(· | Y ) – e.g., use MCMC to get approximate samples θ(1), . . . , θ(T) from π(· | Y ) 2) compute bagged posterior πBB(· | Y ) using B ≈ 50 bootstrap datasets – e.g., use MCMC to get approximate samples θ∗

(b,1), . . . , θ∗ (b,T) from π(· | Y ∗ (b))

for b = 1, . . . , B if Gaussian approximation to standard and bagged posteriors decent then 3a) compute mismatch index I 3b) if I ' .2, consider refining the model and returning to step 1 4) output bagged posterior computed in step 2

BayesBag in practice

12 [H & Miller 2019]

1) compute standard posterior π(· | Y ) – e.g., use MCMC to get approximate samples θ(1), . . . , θ(T) from π(· | Y ) 2) compute bagged posterior πBB(· | Y ) using B ≈ 50 bootstrap datasets – e.g., use MCMC to get approximate samples θ∗

(b,1), . . . , θ∗ (b,T) from π(· | Y ∗ (b))

for b = 1, . . . , B if Gaussian approximation to standard and bagged posteriors decent then 3a) compute mismatch index I 3b) if I ' .2, consider refining the model and returning to step 1 4) output bagged posterior computed in step 2

BayesBag in practice

12 [H & Miller 2019]

1) compute standard posterior π(· | Y ) – e.g., use MCMC to get approximate samples θ(1), . . . , θ(T) from π(· | Y ) 2) compute bagged posterior πBB(· | Y ) using B ≈ 50 bootstrap datasets – e.g., use MCMC to get approximate samples θ∗

(b,1), . . . , θ∗ (b,T) from π(· | Y ∗ (b))

for b = 1, . . . , B if Gaussian approximation to standard and bagged posteriors decent then 3a) compute mismatch index I 3b) if I ' .2, consider refining the model and returning to step 1 4) output bagged posterior computed in step 2

BayesBag in practice

12 [H & Miller 2019]

1) compute standard posterior π(· | Y ) – e.g., use MCMC to get approximate samples θ(1), . . . , θ(T) from π(· | Y ) 2) compute bagged posterior πBB(· | Y ) using B ≈ 50 bootstrap datasets – e.g., use MCMC to get approximate samples θ∗

(b,1), . . . , θ∗ (b,T) from π(· | Y ∗ (b))

for b = 1, . . . , B if Gaussian approximation to standard and bagged posteriors decent then 3a) compute mismatch index I 3b) if I ' .2, consider refining the model and returning to step 1 4) output bagged posterior computed in step 2

BayesBag in practice

12 [H & Miller 2019]

1) compute standard posterior π(· | Y ) – e.g., use MCMC to get approximate samples θ(1), . . . , θ(T) from π(· | Y ) 2) compute bagged posterior πBB(· | Y ) using B ≈ 50 bootstrap datasets – e.g., use MCMC to get approximate samples θ∗

(b,1), . . . , θ∗ (b,T) from π(· | Y ∗ (b))

for b = 1, . . . , B if Gaussian approximation to standard and bagged posteriors decent then 3a) compute mismatch index I 3b) if I ' .2, consider refining the model and returning to step 1 4) output bagged posterior computed in step 2

BayesBag in practice

12 [H & Miller 2019]

1) compute standard posterior π(· | Y ) – e.g., use MCMC to get approximate samples θ(1), . . . , θ(T) from π(· | Y ) 2) compute bagged posterior πBB(· | Y ) using B ≈ 50 bootstrap datasets – e.g., use MCMC to get approximate samples θ∗

(b,1), . . . , θ∗ (b,T) from π(· | Y ∗ (b))

for b = 1, . . . , B if Gaussian approximation to standard and bagged posteriors decent then 3a) compute mismatch index I 3b) if I ' .2, consider refining the model and returning to step 1 4) output bagged posterior computed in step 2

BayesBag in practice

12 [H & Miller 2019]

Agenda

13

• BayesBag for parameter inference

(and prediction)

• BayesBag theory and methodology

➡ BayesBag for model selection

Bayesian model selection

14

Bayesian model selection

14

• Goal: based on data Y, select between 


a (finite or countable) set of models 
 M = {m1, m2, …}

Bayesian model selection

14

• Goal: based on data Y, select between 


a (finite or countable) set of models 
 M = {m1, m2, …}

• Example: systematics

Bayesian model selection

14

• Goal: based on data Y, select between 


a (finite or countable) set of models 
 M = {m1, m2, …}

• Example: systematics
• Goal: learn about evolutionary history
• f a set of species [e.g. whales]

Minke whale Grey whale Fin whale Blue whale

m1

Minke whale Grey whale Blue whale Fin whale

m2

. . .

Bayesian model selection

14

• Goal: based on data Y, select between 


a (finite or countable) set of models 
 M = {m1, m2, …}

• Example: systematics
• Goal: learn about evolutionary history
• f a set of species [e.g. whales]
• Approach: infer which phylogenetic

trees are consistent with observed species characteristics Y 
 [e.g. genetic data, physical features such as coloring]

Minke whale Grey whale Fin whale Blue whale

m1

Minke whale Grey whale Blue whale Fin whale

m2

π(m1 | Y ) = .8

π(m2 | Y ) = .1

. . .

Bayesian model selection

14

• Goal: based on data Y, select between 


a (finite or countable) set of models 
 M = {m1, m2, …}

• Example: systematics
• Goal: learn about evolutionary history
• f a set of species [e.g. whales]
• Approach: infer which phylogenetic

trees are consistent with observed species characteristics Y 
 [e.g. genetic data, physical features such as coloring]

• Problem: Bayesian model selection still

assumes some model in M is correct

Minke whale Grey whale Fin whale Blue whale

m1

Minke whale Grey whale Blue whale Fin whale

m2

π(m1 | Y ) = .8

π(m2 | Y ) = .1

. . .

Illustration: two normal models

15 [H & Miller 2019]

• Models are m1 = N(−1, 1) and m2 = N(1, 1)
• True distribution is Ptrue = N(0, 1)
• Generate datasets Y (1), Y (2), . . . of size n = 1000,

where Y (i)

j

∼ N(0, 1).

Illustration: two normal models

15 [H & Miller 2019]

• Models are m1 = N(−1, 1) and m2 = N(1, 1)
• True distribution is Ptrue = N(0, 1)
• Generate datasets Y (1), Y (2), . . . of size n = 1000,

where Y (i)

j

∼ N(0, 1).

Illustration: two normal models

15 [H & Miller 2019]

• Models are m1 = N(−1, 1) and m2 = N(1, 1)
• True distribution is Ptrue = N(0, 1)
• Generate datasets Y (1), Y (2), . . . of size n = 1000,

where Y (i)

j

∼ N(0, 1).

m1

m2

Ptrue

Illustration: two normal models

15 [H & Miller 2019]

• Models are m1 = N(−1, 1) and m2 = N(1, 1)
• True distribution is Ptrue = N(0, 1)
• Generate datasets Y (1), Y (2), . . . of size n = 1000,

where Y (i)

j

∼ N(0, 1).

m1

m2

Ptrue

Illustration: two normal models

15 [H & Miller 2019]

π(m1 | Y (1)) = 1 πBB(m1 | Y (1)) = 0.82

• Models are m1 = N(−1, 1) and m2 = N(1, 1)
• True distribution is Ptrue = N(0, 1)
• Generate datasets Y (1), Y (2), . . . of size n = 1000,

where Y (i)

j

∼ N(0, 1).

m1

m2

Ptrue

Illustration: two normal models

15 [H & Miller 2019]

π(m1 | Y (2)) = 10−5 πBB(m1 | Y (2)) = 0.38

π(m1 | Y (1)) = 1 πBB(m1 | Y (1)) = 0.82

• Models are m1 = N(−1, 1) and m2 = N(1, 1)
• True distribution is Ptrue = N(0, 1)
• Generate datasets Y (1), Y (2), . . . of size n = 1000,

where Y (i)

j

∼ N(0, 1).

m1

m2

Ptrue

Illustration: two normal models

15 [H & Miller 2019]

π(m1 | Y (2)) = 10−5 πBB(m1 | Y (2)) = 0.38

π(m1 | Y (1)) = 1 πBB(m1 | Y (1)) = 0.82

π(m1 | Y (3)) = 1 πBB(m1 | Y (3)) = 0.90

• Models are m1 = N(−1, 1) and m2 = N(1, 1)
• True distribution is Ptrue = N(0, 1)
• Generate datasets Y (1), Y (2), . . . of size n = 1000,

where Y (i)

j

∼ N(0, 1).

m1

m2

Ptrue

BayesBag stabilizes posterior probabilities of similar models

16 [Yang & Zhu 2018, H & Miller 2019]

BayesBag stabilizes posterior probabilities of similar models

16 [Yang & Zhu 2018, H & Miller 2019]

• Assume two models m1 and m2 [e.g. two possible trees]

BayesBag stabilizes posterior probabilities of similar models

16 [Yang & Zhu 2018, H & Miller 2019]

• Assume two models m1 and m2 [e.g. two possible trees]
• If models explain the data-generating distribution equally well, hope

equal posterior probability (with enough data): π(m1 | Y) = π(m2 | Y) = 1/2

Assume m1 and m2 are equally good. 
 Then in the large data limit,

• 1. For the standard posterior,

π(m1 | Y) = 0 or 1 with equal probability

• 2. For the bagged posterior,

πBB(m1 | Y) ~ Uniform[0,1]

Theorem [H & Miller 2019]

BayesBag stabilizes posterior probabilities of similar models

16 [Yang & Zhu 2018, H & Miller 2019]

• Assume two models m1 and m2 [e.g. two possible trees]
• If models explain the data-generating distribution equally well, hope

equal posterior probability (with enough data): π(m1 | Y) = π(m2 | Y) = 1/2 However….

Assume m1 and m2 are equally good. 
 Then in the large data limit,

• 1. For the standard posterior,

π(m1 | Y) = 0 or 1 with equal probability

• 2. For the bagged posterior,

πBB(m1 | Y) ~ Uniform[0,1]

Theorem [H & Miller 2019]

BayesBag stabilizes posterior probabilities of similar models

16 [Yang & Zhu 2018, H & Miller 2019]

• Assume two models m1 and m2 [e.g. two possible trees]
• If models explain the data-generating distribution equally well, hope

equal posterior probability (with enough data): π(m1 | Y) = π(m2 | Y) = 1/2 However….

Assume m1 and m2 are equally good. 
 Then in the large data limit,

• 1. For the standard posterior,

π(m1 | Y) = 0 or 1 with equal probability

• 2. For the bagged posterior,

πBB(m1 | Y) ~ Uniform[0,1]

Theorem [H & Miller 2019]

BayesBag stabilizes posterior probabilities of similar models

16 [Yang & Zhu 2018, H & Miller 2019]

• Assume two models m1 and m2 [e.g. two possible trees]
• If models explain the data-generating distribution equally well, hope

equal posterior probability (with enough data): π(m1 | Y) = π(m2 | Y) = 1/2

Y(1) Y(2) Y(3) Y(4) Y(5) Y(6)

⠇

However….

Assume m1 and m2 are equally good. 
 Then in the large data limit,

• 1. For the standard posterior,

π(m1 | Y) = 0 or 1 with equal probability

• 2. For the bagged posterior,

πBB(m1 | Y) ~ Uniform[0,1]

Theorem [H & Miller 2019]

BayesBag stabilizes posterior probabilities of similar models

16 [Yang & Zhu 2018, H & Miller 2019]

• Assume two models m1 and m2 [e.g. two possible trees]
• If models explain the data-generating distribution equally well, hope

equal posterior probability (with enough data): π(m1 | Y) = π(m2 | Y) = 1/2

Y(1) Y(2) Y(3) Y(4) Y(5) Y(6)

⠇

π(m1|·)

1 1 ⠇

However….

Assume m1 and m2 are equally good. 
 Then in the large data limit,

• 1. For the standard posterior,

π(m1 | Y) = 0 or 1 with equal probability

• 2. For the bagged posterior,

πBB(m1 | Y) ~ Uniform[0,1]

Theorem [H & Miller 2019]

BayesBag stabilizes posterior probabilities of similar models

16 [Yang & Zhu 2018, H & Miller 2019]

• Assume two models m1 and m2 [e.g. two possible trees]
• If models explain the data-generating distribution equally well, hope

equal posterior probability (with enough data): π(m1 | Y) = π(m2 | Y) = 1/2

Y(1) Y(2) Y(3) Y(4) Y(5) Y(6)

⠇

π(m1|·)

1 1 ⠇

All posterior mass

• n a single,

arbitrary model

However….

Assume m1 and m2 are equally good. 
 Then in the large data limit,

• 1. For the standard posterior,

π(m1 | Y) = 0 or 1 with equal probability

• 2. For the bagged posterior,

πBB(m1 | Y) ~ Uniform[0,1]

Theorem [H & Miller 2019]

BayesBag stabilizes posterior probabilities of similar models

16 [Yang & Zhu 2018, H & Miller 2019]

• Assume two models m1 and m2 [e.g. two possible trees]
• If models explain the data-generating distribution equally well, hope

equal posterior probability (with enough data): π(m1 | Y) = π(m2 | Y) = 1/2

Y(1) Y(2) Y(3) Y(4) Y(5) Y(6)

⠇

π(m1|·)

1 1 ⠇

All posterior mass

• n a single,

arbitrary model

However….

Assume m1 and m2 are equally good. 
 Then in the large data limit,

• 1. For the standard posterior,

π(m1 | Y) = 0 or 1 with equal probability

• 2. For the bagged posterior,

πBB(m1 | Y) ~ Uniform[0,1]

Theorem [H & Miller 2019]

BayesBag stabilizes posterior probabilities of similar models

16 [Yang & Zhu 2018, H & Miller 2019]

• Assume two models m1 and m2 [e.g. two possible trees]
• If models explain the data-generating distribution equally well, hope

equal posterior probability (with enough data): π(m1 | Y) = π(m2 | Y) = 1/2

Y(1) Y(2) Y(3) Y(4) Y(5) Y(6)

⠇

π(m1|·)

1 1 ⠇

πBB(m1|·)

0.03 0.78 0.75 0.98 0.95 0.23 ⠇

All posterior mass

• n a single,

arbitrary model

However….

Assume m1 and m2 are equally good. 
 Then in the large data limit,

• 1. For the standard posterior,

π(m1 | Y) = 0 or 1 with equal probability

• 2. For the bagged posterior,

πBB(m1 | Y) ~ Uniform[0,1]

Theorem [H & Miller 2019]

BayesBag stabilizes posterior probabilities of similar models

16 [Yang & Zhu 2018, H & Miller 2019]

• Assume two models m1 and m2 [e.g. two possible trees]
• If models explain the data-generating distribution equally well, hope

equal posterior probability (with enough data): π(m1 | Y) = π(m2 | Y) = 1/2

Y(1) Y(2) Y(3) Y(4) Y(5) Y(6)

⠇

π(m1|·)

1 1 ⠇

πBB(m1|·)

0.03 0.78 0.75 0.98 0.95 0.23 ⠇

All posterior mass

• n a single,

arbitrary model bagged posterior mass more evenly distributed

However….

Reproducible phylogenetic inference

17 [H & Miller 2019]

Reproducible phylogenetic inference

17 [H & Miller 2019]

• Goal: infer phylogeny of 13

whale species from mitochondrial coding DNA

Minke GACCCGAACGTAATAA…ATCCGTTCCCATACTC Blue CACCCCCCCGTACTAT…TGAGTCCGAATTGGAA Fin TGTCTTCTACACTCCA…ACAGGTTGTACGTCAC Grey GGGTCGCTGTAGACCA…GATACCGCTCTCACAT

all

Reproducible phylogenetic inference

17 [H & Miller 2019]

• Goal: infer phylogeny of 13

whale species from mitochondrial coding DNA

Minke GACCCGAACGTAATAA…ATCCGTTCCCATACTC Blue CACCCCCCCGTACTAT…TGAGTCCGAATTGGAA Fin TGTCTTCTACACTCCA…ACAGGTTGTACGTCAC Grey GGGTCGCTGTAGACCA…GATACCGCTCTCACAT

all 1st half

Reproducible phylogenetic inference

17 [H & Miller 2019]

• Goal: infer phylogeny of 13

whale species from mitochondrial coding DNA

Minke GACCCGAACGTAATAA…ATCCGTTCCCATACTC Blue CACCCCCCCGTACTAT…TGAGTCCGAATTGGAA Fin TGTCTTCTACACTCCA…ACAGGTTGTACGTCAC Grey GGGTCGCTGTAGACCA…GATACCGCTCTCACAT

all 1st half 2nd half

Reproducible phylogenetic inference

17 [H & Miller 2019]

1st half all

• Goal: infer phylogeny of 13

whale species from mitochondrial coding DNA

Minke GACCCGAACGTAATAA…ATCCGTTCCCATACTC Blue CACCCCCCCGTACTAT…TGAGTCCGAATTGGAA Fin TGTCTTCTACACTCCA…ACAGGTTGTACGTCAC Grey GGGTCGCTGTAGACCA…GATACCGCTCTCACAT

all 1st half 2nd half

Reproducible phylogenetic inference

17 [H & Miller 2019]

1st half all

• Goal: infer phylogeny of 13

whale species from mitochondrial coding DNA

• Compute posterior tree

probabilities based on all, 
 1st half, and 2nd half

Minke GACCCGAACGTAATAA…ATCCGTTCCCATACTC Blue CACCCCCCCGTACTAT…TGAGTCCGAATTGGAA Fin TGTCTTCTACACTCCA…ACAGGTTGTACGTCAC Grey GGGTCGCTGTAGACCA…GATACCGCTCTCACAT

all 1st half 2nd half

Reproducible phylogenetic inference

17 [H & Miller 2019]

1st half all

• Goal: infer phylogeny of 13

whale species from mitochondrial coding DNA

• Compute posterior tree

probabilities based on all, 
 1st half, and 2nd half

• Compute overlap of 99% 


high probability regions

Minke GACCCGAACGTAATAA…ATCCGTTCCCATACTC Blue CACCCCCCCGTACTAT…TGAGTCCGAATTGGAA Fin TGTCTTCTACACTCCA…ACAGGTTGTACGTCAC Grey GGGTCGCTGTAGACCA…GATACCGCTCTCACAT

all 1st half 2nd half

Reproducible phylogenetic inference

17 [H & Miller 2019]

1st half all

• Goal: infer phylogeny of 13

whale species from mitochondrial coding DNA

• Compute posterior tree

probabilities based on all, 
 1st half, and 2nd half

• Compute overlap of 99% 


high probability regions

Minke GACCCGAACGTAATAA…ATCCGTTCCCATACTC Blue CACCCCCCCGTACTAT…TGAGTCCGAATTGGAA Fin TGTCTTCTACACTCCA…ACAGGTTGTACGTCAC Grey GGGTCGCTGTAGACCA…GATACCGCTCTCACAT

all 1st half 2nd half

0.25 0.5 0.75 1 tree 1 tree 2 tree 3 tree 4 total

all 1st half

• verlap

Reproducible phylogenetic inference

17 [H & Miller 2019]

1st half all

• Goal: infer phylogeny of 13

whale species from mitochondrial coding DNA

• Compute posterior tree

probabilities based on all, 
 1st half, and 2nd half

• Compute overlap of 99% 


high probability regions

Minke GACCCGAACGTAATAA…ATCCGTTCCCATACTC Blue CACCCCCCCGTACTAT…TGAGTCCGAATTGGAA Fin TGTCTTCTACACTCCA…ACAGGTTGTACGTCAC Grey GGGTCGCTGTAGACCA…GATACCGCTCTCACAT

all 1st half 2nd half

0.25 0.5 0.75 1 tree 1 tree 2 tree 3 tree 4 total

all 1st half

• verlap .99

.3 .19 .0 .5

Reproducible phylogenetic inference

17 [H & Miller 2019]

1st half all

• Goal: infer phylogeny of 13

whale species from mitochondrial coding DNA

• Compute posterior tree

probabilities based on all, 
 1st half, and 2nd half

• Compute overlap of 99% 


high probability regions

Minke GACCCGAACGTAATAA…ATCCGTTCCCATACTC Blue CACCCCCCCGTACTAT…TGAGTCCGAATTGGAA Fin TGTCTTCTACACTCCA…ACAGGTTGTACGTCAC Grey GGGTCGCTGTAGACCA…GATACCGCTCTCACAT

all 1st half 2nd half

0.25 0.5 0.75 1 tree 1 tree 2 tree 3 tree 4 total

all 1st half

• verlap

.99 .0 .69 .1 .2 .99 .3 .19 .0 .5

Reproducible phylogenetic inference

17 [H & Miller 2019]

1st half all

• Goal: infer phylogeny of 13

whale species from mitochondrial coding DNA

• Compute posterior tree

probabilities based on all, 
 1st half, and 2nd half

• Compute overlap of 99% 


high probability regions

Minke GACCCGAACGTAATAA…ATCCGTTCCCATACTC Blue CACCCCCCCGTACTAT…TGAGTCCGAATTGGAA Fin TGTCTTCTACACTCCA…ACAGGTTGTACGTCAC Grey GGGTCGCTGTAGACCA…GATACCGCTCTCACAT

all 1st half 2nd half

0.25 0.5 0.75 1 tree 1 tree 2 tree 3 tree 4 total

all 1st half

• verlap

.39 .0 .19 .0 .2 .99 .0 .69 .1 .2 .99 .3 .19 .0 .5

Reproducible phylogenetic inference

17 [H & Miller 2019]

1st half all

• Goal: infer phylogeny of 13

whale species from mitochondrial coding DNA

• Compute posterior tree

probabilities based on all, 
 1st half, and 2nd half

• Compute overlap of 99% 


high probability regions

• 0% overlap = contradiction

Minke GACCCGAACGTAATAA…ATCCGTTCCCATACTC Blue CACCCCCCCGTACTAT…TGAGTCCGAATTGGAA Fin TGTCTTCTACACTCCA…ACAGGTTGTACGTCAC Grey GGGTCGCTGTAGACCA…GATACCGCTCTCACAT

all 1st half 2nd half

0.25 0.5 0.75 1 tree 1 tree 2 tree 3 tree 4 total

all 1st half

• verlap

.39 .0 .19 .0 .2 .99 .0 .69 .1 .2 .99 .3 .19 .0 .5

Stable, reproducible phylogenetic inference with BayesBag

18

• Goal: infer phylogeny of 13 whale species

[H & Miller 2019]

Stable, reproducible phylogenetic inference with BayesBag

18

• Goal: infer phylogeny of 13 whale species

evolutionary model evolutionary model

Standard Posterior Bagged Posterior

1st vs 2nd half all vs 1st half all vs 2nd half

[H & Miller 2019]

• For some evolutionary models, little to no overlap

Stable, reproducible phylogenetic inference with BayesBag

18

• Goal: infer phylogeny of 13 whale species

evolutionary model evolutionary model

Standard Posterior Bagged Posterior

1st vs 2nd half all vs 1st half all vs 2nd half

[H & Miller 2019]

danger
 zone

• For some evolutionary models, little to no overlap

Stable, reproducible phylogenetic inference with BayesBag

18

• Goal: infer phylogeny of 13 whale species

evolutionary model evolutionary model

Standard Posterior Bagged Posterior

1st vs 2nd half all vs 1st half all vs 2nd half

[H & Miller 2019]

zero overlap danger
 zone

• For some evolutionary models, little to no overlap

Stable, reproducible phylogenetic inference with BayesBag

18

• Goal: infer phylogeny of 13 whale species

evolutionary model evolutionary model

Standard Posterior Bagged Posterior

1st vs 2nd half all vs 1st half all vs 2nd half

[H & Miller 2019]

zero overlap danger
 zone

• For some evolutionary models, little to no overlap
• Bayesian model selection is unstable and not reproducible [Wilcox et al.

2002, Alfaro et al. 2003, Douady et al. 2003, …]

Stable, reproducible phylogenetic inference with BayesBag

18

• Goal: infer phylogeny of 13 whale species

evolutionary model evolutionary model

Standard Posterior Bagged Posterior

1st vs 2nd half all vs 1st half all vs 2nd half

[H & Miller 2019]

zero overlap danger
 zone

• For some evolutionary models, little to no overlap
• Bayesian model selection is unstable and not reproducible [Wilcox et al.

2002, Alfaro et al. 2003, Douady et al. 2003, …]

• Same problem comparing evolutionary models with data fixed

Stable, reproducible phylogenetic inference with BayesBag

18

• Goal: infer phylogeny of 13 whale species

evolutionary model evolutionary model

Standard Posterior Bagged Posterior

1st vs 2nd half all vs 1st half all vs 2nd half

[H & Miller 2019]

zero overlap

• For some evolutionary models, little to no overlap
• Bayesian model selection is unstable and not reproducible [Wilcox et al.

2002, Alfaro et al. 2003, Douady et al. 2003, …]

• Same problem comparing evolutionary models with data fixed
• Bagged posterior model probabilities more stable and reproducible

danger
 zone

Stable, reproducible phylogenetic inference with BayesBag

18

• Goal: infer phylogeny of 13 whale species

evolutionary model evolutionary model

Standard Posterior Bagged Posterior

1st vs 2nd half all vs 1st half all vs 2nd half

nonzero

• verlap

[H & Miller 2019]

zero overlap

• For some evolutionary models, little to no overlap
• Bayesian model selection is unstable and not reproducible [Wilcox et al.

2002, Alfaro et al. 2003, Douady et al. 2003, …]

• Same problem comparing evolutionary models with data fixed
• Bagged posterior model probabilities more stable and reproducible

danger
 zone

19

Closing thoughts

19

Closing thoughts

• We show that BayesBag…

19

Closing thoughts

• We show that BayesBag…

1. has provably good statistical robustness properties

19

Closing thoughts

• We show that BayesBag…

1. has provably good statistical robustness properties 2. empirically, demonstrates superior predictive performance (compared to standard Bayes)

19

Closing thoughts

• We show that BayesBag…

1. has provably good statistical robustness properties 2. empirically, demonstrates superior predictive performance (compared to standard Bayes) 3. is easy to use and widely applicable

19

Closing thoughts

• We show that BayesBag…

1. has provably good statistical robustness properties 2. empirically, demonstrates superior predictive performance (compared to standard Bayes) 3. is easy to use and widely applicable 4. combines the flexible modeling features of Bayes with the distributional robustness of frequentist methods

19

Closing thoughts

• We show that BayesBag…

1. has provably good statistical robustness properties 2. empirically, demonstrates superior predictive performance (compared to standard Bayes) 3. is easy to use and widely applicable 4. combines the flexible modeling features of Bayes with the distributional robustness of frequentist methods

• Future work:

19

Closing thoughts

• We show that BayesBag…

1. has provably good statistical robustness properties 2. empirically, demonstrates superior predictive performance (compared to standard Bayes) 3. is easy to use and widely applicable 4. combines the flexible modeling features of Bayes with the distributional robustness of frequentist methods

• Future work:

➡ time series / other structured data

19

Closing thoughts

• We show that BayesBag…

1. has provably good statistical robustness properties 2. empirically, demonstrates superior predictive performance (compared to standard Bayes) 3. is easy to use and widely applicable 4. combines the flexible modeling features of Bayes with the distributional robustness of frequentist methods

• Future work:

➡ time series / other structured data ➡ speeding up computation

19

Closing thoughts

• We show that BayesBag…

1. has provably good statistical robustness properties 2. empirically, demonstrates superior predictive performance (compared to standard Bayes) 3. is easy to use and widely applicable 4. combines the flexible modeling features of Bayes with the distributional robustness of frequentist methods

• Future work:

➡ time series / other structured data ➡ speeding up computation

• On arXiv very soon (if you want a heads up: jhuggins@hsph.harvard.edu)

19

Closing thoughts

• We show that BayesBag…

1. has provably good statistical robustness properties 2. empirically, demonstrates superior predictive performance (compared to standard Bayes) 3. is easy to use and widely applicable 4. combines the flexible modeling features of Bayes with the distributional robustness of frequentist methods

• Future work:

➡ time series / other structured data ➡ speeding up computation

• On arXiv very soon (if you want a heads up: jhuggins@hsph.harvard.edu)

Thank you

• Let ∆n , Pn

i=1 log p(Yi | m1) − log p(Yi | m2)

| {z }

δi

• Then π(m1 | Y ) = (1 + exp{−∆n})−1
• By assumption, E[δi] = 0

but σ2 = Var(δi) > 0

• Hence, ∆n is a random walk

with E[∆2

n] = σ2n

• In other words, with very high

probability, |∆n| = Θ(n1/2)

• Therefore, there is overwhelming evidence of order n1/2 for either m1 or m2

20

Why can Bayesian model selection fail?

• Let ∆n , Pn

i=1 log p(Yi | m1) − log p(Yi | m2)

| {z }

δi

• Then π(m1 | Y ) = (1 + exp{−∆n})−1
• By assumption, E[δi] = 0

but σ2 = Var(δi) > 0

• Hence, ∆n is a random walk

with E[∆2

n] = σ2n

• In other words, with very high

probability, |∆n| = Θ(n1/2)

• Therefore, there is overwhelming evidence of order n1/2 for either m1 or m2

20

Why can Bayesian model selection fail?

• Let ∆n , Pn

i=1 log p(Yi | m1) − log p(Yi | m2)

| {z }

δi

• Then π(m1 | Y ) = (1 + exp{−∆n})−1
• By assumption, E[δi] = 0

but σ2 = Var(δi) > 0

• Hence, ∆n is a random walk

with E[∆2

n] = σ2n

• In other words, with very high

probability, |∆n| = Θ(n1/2)

• Therefore, there is overwhelming evidence of order n1/2 for either m1 or m2

20

Why can Bayesian model selection fail?

• Let ∆n , Pn

i=1 log p(Yi | m1) − log p(Yi | m2)

| {z }

δi

• Then π(m1 | Y ) = (1 + exp{−∆n})−1
• By assumption, E[δi] = 0

but σ2 = Var(δi) > 0

• Hence, ∆n is a random walk

with E[∆2

n] = σ2n

• In other words, with very high

probability, |∆n| = Θ(n1/2)

• Therefore, there is overwhelming evidence of order n1/2 for either m1 or m2

20

Why can Bayesian model selection fail?

• 5

5 10 0.2 0.4 0.6 0.8 1.0 π(m1|Y )

∆n

• Let ∆n , Pn

i=1 log p(Yi | m1) − log p(Yi | m2)

| {z }

δi

• Then π(m1 | Y ) = (1 + exp{−∆n})−1
• By assumption, E[δi] = 0

but σ2 = Var(δi) > 0

• Hence, ∆n is a random walk

with E[∆2

n] = σ2n

• In other words, with very high

probability, |∆n| = Θ(n1/2)

• Therefore, there is overwhelming evidence of order n1/2 for either m1 or m2

20

Why can Bayesian model selection fail?

• 5

5 10 0.2 0.4 0.6 0.8 1.0 π(m1|Y )

∆n

• Let ∆n , Pn

i=1 log p(Yi | m1) − log p(Yi | m2)

| {z }

δi

• Then π(m1 | Y ) = (1 + exp{−∆n})−1
• By assumption, E[δi] = 0

but σ2 = Var(δi) > 0

• Hence, ∆n is a random walk

with E[∆2

n] = σ2n

• In other words, with very high

probability, |∆n| = Θ(n1/2)

• Therefore, there is overwhelming evidence of order n1/2 for either m1 or m2

20

Why can Bayesian model selection fail?

• 5

5 10 0.2 0.4 0.6 0.8 1.0 π(m1|Y )

∆n

• Let ∆n , Pn

i=1 log p(Yi | m1) − log p(Yi | m2)

| {z }

δi

• Then π(m1 | Y ) = (1 + exp{−∆n})−1
• By assumption, E[δi] = 0

but σ2 = Var(δi) > 0

• Hence, ∆n is a random walk

with E[∆2

n] = σ2n

• In other words, with very high

probability, |∆n| = Θ(n1/2)

• Therefore, there is overwhelming evidence of order n1/2 for either m1 or m2

20

Why can Bayesian model selection fail?

• 5

5 10 0.2 0.4 0.6 0.8 1.0 π(m1|Y )

∆n

• Let ∆n , Pn

i=1 log p(Yi | m1) − log p(Yi | m2)

| {z }

δi

• Then π(m1 | Y ) = (1 + exp{−∆n})−1
• By assumption, E[δi] = 0

but σ2 = Var(δi) > 0

• Hence, ∆n is a random walk

with E[∆2

n] = σ2n

• In other words, with very high

probability, |∆n| = Θ(n1/2)

• Therefore, there is overwhelming evidence of order n1/2 for either m1 or m2

20

Why can Bayesian model selection fail?

• 5

5 10 0.2 0.4 0.6 0.8 1.0 π(m1|Y )

∆n

Bootstrap aggregating (bagging)

[Breiman 1995, Bühlmann & Yu 2002] 21

Ptrue

Yboot

Pn

Y

Bootstrap aggregating (bagging)

[Breiman 1995, Bühlmann & Yu 2002] 21

• Have: samples Y = (Y1, …, Yn)

Ptrue

Yboot

Pn

Y

Bootstrap aggregating (bagging)

[Breiman 1995, Bühlmann & Yu 2002] 21

• Have: samples Y = (Y1, …, Yn)
• Goal: predict future outcome

based on Y [i.e. regression]

Ptrue

Yboot

Pn

Yi = (Xi, Zi)

covariates

• utcome

Y

Bootstrap aggregating (bagging)

[Breiman 1995, Bühlmann & Yu 2002] 21

• Have: samples Y = (Y1, …, Yn)
• Goal: predict future outcome

based on Y [i.e. regression]

Ptrue

Yboot

Pn

Yi = (Xi, Zi)

covariates

• utcome

Y

Zpred = f(Xnew; Y )

Bootstrap aggregating (bagging)

[Breiman 1995, Bühlmann & Yu 2002] 21

• Have: samples Y = (Y1, …, Yn)
• Goal: predict future outcome

based on Y [i.e. regression]

• Problem: prediction algorithm is

unstable [e.g. regression trees]

Ptrue

Yboot

Pn

Yi = (Xi, Zi)

covariates

• utcome

Y

Zpred = f(Xnew; Y )

Bootstrap aggregating (bagging)

[Breiman 1995, Bühlmann & Yu 2002] 21

• Have: samples Y = (Y1, …, Yn)
• Goal: predict future outcome

based on Y [i.e. regression]

• Problem: prediction algorithm is

unstable [e.g. regression trees]

• Bagging: stabilize predictions by

aggregating (averaging) over predictions based on bootstrapped datasets

Ptrue

Yboot

Pn

Yi = (Xi, Zi)

covariates

• utcome

Zbag

pred = 1

B

B

X

b=1

f(Xnew; Y (b)

boot)

Y

Zpred = f(Xnew; Y )

Bootstrap aggregating (bagging)

[Breiman 1995, Bühlmann & Yu 2002] 21

• Have: samples Y = (Y1, …, Yn)
• Goal: predict future outcome

based on Y [i.e. regression]

• Problem: prediction algorithm is

unstable [e.g. regression trees]

• Bagging: stabilize predictions by

aggregating (averaging) over predictions based on bootstrapped datasets

• Like bagging, BayesBag seems to

work well with B = 50 or 100

Ptrue

Yboot

Pn

Yi = (Xi, Zi)

covariates

• utcome

Zbag

pred = 1

B

B

X

b=1

f(Xnew; Y (b)

boot)

Y

Zpred = f(Xnew; Y )

πBB(θ | Y ) = 1 B

B

X

b=1

π(θ | Y (b)

boot)