Latent Tree Copulas Sergey Kirshner skirshne@purdue.edu Purdue - - PowerPoint PPT Presentation

▶
latent tree copulas
SMART_READER_LITE
LIVE PREVIEW

Latent Tree Copulas Sergey Kirshner skirshne@purdue.edu Purdue - - PowerPoint PPT Presentation

Mar 05, 2023 397 likes •726 views

Latent Tree Copulas Sergey Kirshner skirshne@purdue.edu Purdue University West Lafayette, IN, USA Granada, Spain, September 19, 2012 Coming Attractions Want to fit density to model multivariate data? and organize real-valued data

slide-1
SLIDE 1

Latent Tree Copulas

Sergey Kirshner

skirshne@purdue.edu Purdue University West Lafayette, IN, USA

Granada, Spain, September 19, 2012

slide-2
SLIDE 2

Coming Attractions

  • Want to fit density to model multivariate

data?

– and organize real-valued data into a hierarchy of features?

  • New density estimation model based on tree-

structured dependence with latent variables

– Distribution = Univariate Marginals + Copula – Hierarchy of variables as a latent tree-copula – Parameter estimation and structure learning

  • Efficient inference for Gaussian copulas (100s of

variables), several structure learning approaches

  • Variational inference for other copulas (10-30 variables)

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

2

slide-3
SLIDE 3
  • 88.5
  • 88
  • 87.5
  • 87
  • 86.5
  • 86
  • 85.5
  • 85
  • 84.5

37.5 38 38.5 39 39.5 40 40.5 41 41.5 42 Longitude Latitude

Building a Hierarchy of Rainfall Stations

State of Indiana (USA) Average monthly

  • bservations for

15 rainfall stations 1951-1996 (47 years)

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

3

slide-4
SLIDE 4

Most Popular Distribution…

  • Interpretable
  • Closed under taking

marginals

  • Generalizes to

multiple dimensions

  • Models pairwise

dependence

  • Tractable
  • 245 pages out of 691

from Continuous Multivariate Distributions by Kotz, Balakrishnan, and Johnson

  • 3
  • 2
  • 1

1 2 3

  • 3
  • 2
  • 1

1 2 3 0.05 0.1 0.15 0.2

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

4

slide-5
SLIDE 5

What If the Data Is NOT Gaussian?

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

5

slide-6
SLIDE 6

Separating Univariate Marginals

univariate marginals, independent variables, multivariate dependence term, copula

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

6

slide-7
SLIDE 7

Monotonic Transformation of the Variables

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

7

slide-8
SLIDE 8

Copula

Copula C is a multivariate distribution (cdf) defined on a unit hypercube with uniform univariate marginals:

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

8

slide-9
SLIDE 9

Sklar’s Theorem

[Sklar 59]

= +

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

9

slide-10
SLIDE 10

Example: Multivariate Gaussian Copula

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

10

slide-11
SLIDE 11

Separating Univariate Marginals

  • 1. Fit univariate marginals (parametric or non-

parametric)

  • 2. Replace data points with cdf’s of the

marginals

  • 3. Estimate copula density

Inference for the margins [Joe and Xu 96]; canonical maximum likelihood [Genest et al 95]

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

11

slide-12
SLIDE 12

Graphical Model Using a Copula

x1 x2 x3 x4 x5 a1 a2 a3 a4 a5 x5 x1 x4 x3 x2

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

12

slide-13
SLIDE 13

Graphical Model Approaches to Estimating Copulas

  • Vines [Bedford and Cooke 02]
  • Trees [Kirshner 08]
  • Nonparanormal model [Liu et al 09]
  • Copula Bayesian networks [Elidan 10]

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

13

slide-14
SLIDE 14

Tree-Structured Densities

x1 x2 x3 x4 x5

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

14

slide-15
SLIDE 15

Tree-Structured Copulas

a1 a2 a3 a4 a5 x5 x1 x4 x3 x2 x1 x2 x3 x4 x5

[Kirshner 08]

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

15

slide-16
SLIDE 16

Using Tree-Structured Copulas

  • Tree-structured copulas are convenient, but

are restrictive

– True distribution may require much larger cliques to decompose

  • Can approximate other dependencies using

latent variables

– Mixtures [Kirshner 08]: discrete latent variables – Latent tree copulas: continuous random variables embedded in copula trees

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

16

slide-17
SLIDE 17

Latent Tree Copulas

  • Defined as a tuple of variables, tree structure,

and bivariate copulas

a1 a2 a3 a4 a5 x5 x1 x4 x3 x2 a6 a8 a7 x1 x2 x3 x4 x5 x6 x8 x7

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

17

slide-18
SLIDE 18

Latent Tree Copulas

  • Defined as a tuple of variables, tree structure,

and bivariate copulas

  • “Siblings” of latent tree models (LTMs) for

categorical variables [e.g., Zhang 02, 04]

a1 a2 a3 a4 a5 x5 x1 x4 x3 x2 a6 a8 a7 x1 x2 x3 x4 x5 x6 x8 x7

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

18

slide-19
SLIDE 19

Inference

  • Good news: posterior distribution is also tree-

structured

– Fairly easy to carry out inference for LTMs

  • Bad news: Latent variables are continuous:

infinite number of possible values

– Need to estimate the joint posterior densities

a1 a2 a3 a4 a5 a6 a8 a7

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

19

slide-20
SLIDE 20

Inference

  • Easy for Gaussian copulas

– Apply inverse standard normal CDF; use belief propagation on jointly Gaussian distribution

  • Difficult for non-Gaussian copulas

– May have no exact form for the posterior!

a1 a2 a3 a4 a5 a6 a8 a7

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

20

slide-21
SLIDE 21

Inference for non-Gaussian Case

  • Variational approach:

– Approximate the posterior distribution using a tree-structured distribution over piece-wise uniform variables – Essentially, approximate using the tree over categorical variables – Use s iterations to find the fixed point – Requires integrating logarithm of bivariate copula pdfs – numerical integration!

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

21

slide-22
SLIDE 22

Parameter Estimation with Known Structure

  • (Variational) EM

– E-step: minimize KL divergence – M-step: maximize the expected compete-data log- likelihood

a1 a2 a3 a4 a5 a6 a8 a7

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

22

slide-23
SLIDE 23

Parameter Estimation with Known Structure

  • Gaussian copula case: EM

– E-step: closed form inference, O(Nt) per iteration – M-step: maximize the expected compete-data log- likelihood

a1 a2 a3 a4 a5 a6 a8 a7

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

23

slide-24
SLIDE 24

Parameter Estimation with Known Structure

  • Non-gaussian copula case: variational EM

– E-step: approximate inference, O(sN|E|k2) + |E|

bivariate integrals per iteration

– M-step: approximate maximization, need to update |E| bivariate copula parameters

a1 a2 a3 a4 a5 a6 a8 a7

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

24

slide-25
SLIDE 25

Unknown Structure

  • Gaussian LTCs: same as for tree-structured

Gaussians

– Size of possible trees can be limited – e.g., can use information distances [Choi et al 2011]

  • Non-Gaussian LTCs: need to restrict the

space of possible models

– Very large space of structures/copula families – Fix the bivariate copula family – Consider only binary latent tree copulas

  • Observed nodes = leafs
  • Motivation: Any Gaussian LTC is equivalent to some

binary LTC

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

25

slide-26
SLIDE 26

Bottom-up Binary LTC Learning

  • Initialize the subtrees to consist of individual

variables (variable = root of a subtree)

  • Iterate until all variables are in one tree

– Estimate mutual informations (Mis) between the root nodes – Pick the pair of roots with the largest MI – Merge the subtrees by creating a new latent root node – Re-estimate parameters (EM)

[Similar to Harmeling and Williams 11]

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

26

slide-27
SLIDE 27

a8 a7 a1 a4 a3

Bottom-up Binary LTC Learning

a2 a5 a6 a9

θ46 θ26 θ37 θ68 θ57 θ78 θ89 θ19

[Similar to Harmeling and Williams 11]

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

27

slide-28
SLIDE 28
  • 88.5
  • 88
  • 87.5
  • 87
  • 86.5
  • 86
  • 85.5
  • 85
  • 84.5

37.5 38 38.5 39 39.5 40 40.5 41 41.5 42 Longitude Latitude

Illustration for Building of Hierarchy of Rainfall Stations

State of Indiana (USA) Average monthly

  • bservations for

15 rainfall stations 1951-1996 (47 years)

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

28

slide-29
SLIDE 29

Experiments: Log-Likelihood on Test Data

UCI ML Repository MAGIC data set 12000 10- dimensional vectors 2000 examples in test sets Average over 10 partitions

50 100 200 500 1000 2000 5000 10000

  • 3.2
  • 3.1
  • 3
  • 2.9
  • 2.8
  • 2.7
  • 2.6

Training set size Log-likelihood per feature Independent KDE Product KDE Gaussian Gaussian Copula Gaussian TCopula Frank TCopula 2-mix of Gaussian TCopulas Gaussian LTC September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

29

slide-30
SLIDE 30

Summary

  • Multivariate distribution = univariate marginals +

copula

  • New model: tree-structured multivariate distribution

with marginally uniform latent variables (latent tree copula, LTC)

– Sufficient to employ only bivariate copula families!

  • Closed form inference for Gaussian copulas (efficient)
  • Variational inference for non-Gaussian copulas (slow)
  • Parameter estimation using the EM algorithm
  • Bottom-up structure learning for bivariate LTCs
  • Can be used for parsimonious multivariate density

estimation or to structure variables into hierarchies

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

30

slide-31
SLIDE 31

Thank you!

http://www.stat.purdue.edu/~skirshne/LTC/index.html

Software: Support: See me at the poster tonight for more details

Questions?

US National Science Foundation Award AGS-1025430

September 19, 2012 Sergey Kirshner, Latent Tree Copulas (PGM 2012)

31