Cmput 651 - Hybrid Networks 24/11/2008 Probabilistic Graphical Models (Cmput 651): Hybrid Network Matthew Brown 24/11/2008 Reading: Handout on Hybrid Networks (Ch. 13 from older version of Koller‐Friedman) 1
Cmput 651 - Hybrid Networks 24/11/2008 Space of topics Semantics Inference Learning Discrete Directed UnDirected Continuous 2
Cmput 651 - Hybrid Networks 24/11/2008 Outline Inference in purely continuous nets Hybrid network semantics Inference in hybrid networks 3
Cmput 651 - Hybrid Networks 24/11/2008 Linear Gaussian Bayesian networks (KF Definition 6.2.1) Definition: A linear Gaussian Bayesian network satisfies: • all variables continuous • all CPDs are linear Gaussians A B C Example: D P ( A ) = N ( µ A , σ 2 A ) P ( B ) = N ( µ B , σ 2 B ) E P ( C ) = N ( µ C , σ 2 C ) P ( D | A, B ) = N ( β D, 0 + β D, 1 A + β D, 2 B, σ 2 D ) P ( E | C, D ) = N ( β E, 0 + β E, 1 C + β E, 2 D, σ 2 E ) 4
Cmput 651 - Hybrid Networks 24/11/2008 Inference in linear Gaussian Bayes nets Recall: linear Gaussian Bayes nets (LGBN) equivalent to multivariate Gaussian distribution To marginalize, could convert LGBN to Gaussian marginalization trivial for Gaussian But ignores structure example ... X 1 X 2 X n p ( X i | X i − 1 ) = N ( β i + α i X i − 1 ; σ 2 i ) LGBN: 3n‐1 parameters Gaussian: n 2 +n parameters bad for large n, eg: > 1000 5
Cmput 651 - Hybrid Networks 24/11/2008 Variable elimination Marginalize out unwanted X using integration rather than sum, as in discrete case Note: Variable elimination gives exact answers for continuous nets (not for hybrid nets) 6
Cmput 651 - Hybrid Networks 24/11/2008 Variable elimination example X 1 X 3 X 4 X 2 � p ( X 4 ) = P ( X 1 , X 2 , X 3 , X 4 ) X 1 ,X 2 ,X 3 � = P ( X 1 ) P ( X 2 ) P ( X 3 | X 1 , X 2 ) P ( X 4 | X 3 ) X 1 ,X 2 ,X 3 � � � = P ( X 1 ) P ( X 2 ) P ( X 3 | X 1 , X 2 ) P ( X 4 | X 3 ) X 1 X 2 X 3 Need a way to represent intermediate factors. Not Gaussian ‐ eg: conditional probabilities not (jointly) Gaussian Need elimination, product, etc. on this representation 7
Cmput 651 - Hybrid Networks 24/11/2008 Canonical forms (KF Handout Def’n 13.2.1) Definition: canonical form Also written 8
Cmput 651 - Hybrid Networks 24/11/2008 Canonical forms and Gaussians (KF Handout 13.2.1) Canonical forms can represent Gaussians: So: 9
Cmput 651 - Hybrid Networks 24/11/2008 Canonical forms and Gaussians (KF Handout 13.2.1) Canonical forms can represent Gaussians Other things (when K ‐1 not defined) eg: linear Gaussian CPDs Can also use conditional forms (multivariate linear Gaussian P(X|Y) ) to represent linear Gaussian CPDs or Gaussians. 10
Cmput 651 - Hybrid Networks 24/11/2008 Operations on canonical forms (KF Handout 13.2.2) Factor product : When scopes don’t overlap, must extend them: Product of and 1st: similarly for product: 11
Cmput 651 - Hybrid Networks 24/11/2008 Operations on canonical forms (KF Handout 13.2.2) Factor division (for belief‐update message passing) Note multiplying or dividing by vacuous canonical form C (0,0,0) has no effect. 12
Cmput 651 - Hybrid Networks 24/11/2008 Operations on canonical forms (KF Handout 13.2.2) Marginalization given over set of variables {X,Y} want require K YY positive definite so that integral is finite marginal 13
Cmput 651 - Hybrid Networks 24/11/2008 Operations on canonical forms (KF Handout 13.2.2) Conditioning given over set of variables {X,Y} want to condition on Y=y Notice: Y no longer part of ‐> canonical form after conditioning (unlike tables). 14
Cmput 651 - Hybrid Networks 24/11/2008 Inference on linear Gaussian Bayesian nets (KF Handout 13.2.3) Factor operations simple, closed form ‐> Variable elimination ‐> Sum‐product message passing ‐> Belief‐update message passing Note on conditioning: conditioned variables disappear from canonical form unlike with factor reduction on table factors ‐> must restrict all factors relevant to inference based on 15 evidence Y=y before doing inference
Cmput 651 - Hybrid Networks 24/11/2008 Inference on linear Gaussian Bayesian nets (KF Handout 13.2.3) Computational performance canonical form operations polynomial in factor scope size n product & division O(n 2 ) marginalization ‐> matrix inversion ≤ O(n 3 ) ‐> inference in LGBNs linear in # cliques cubic in max. clique size for discrete networks factor operations on table factors exponential in scope size 16
Cmput 651 - Hybrid Networks 24/11/2008 Inference on linear Gaussian Bayesian nets (KF Handout 13.2.3) Computational performance (cont’d) ‐ for low dimensionality (small # variables), Gaussian representation can be more efficient ‐ for high dimensionality and low tree width, message passing on LGBN much more efficient 17
Cmput 651 - Hybrid Networks 24/11/2008 Summary Inference on linear Gaussian Bayesian nets: use canonical forms variable elimination or clique tree calibration exact efficient 18
Cmput 651 - Hybrid Networks 24/11/2008 Outline Inference in purely continuous nets Hybrid network semantics Inference in hybrid networks 19
Cmput 651 - Hybrid Networks 24/11/2008 Hybrid networks (KF 5.5.1) Hybrid networks combine discrete and continuous variables 20
Cmput 651 - Hybrid Networks 24/11/2008 Conditional linear Gaussian (CLG) models (KF 5.1) Definition: Given: continuous variable X with discrete parents continuous parents X has a conditional linear Gaussian CPD if for each assignment ∃ coefficients and variance such that 21
Cmput 651 - Hybrid Networks 24/11/2008 Conditional linear Gaussian (CLG) models (KF 5.1) Definition: A Bayesian network is a conditional linear Gaussian network if: • discrete nodes have only discrete parents • continuous nodes have conditional linear Gaussian CPDs ‐ continuous parents cannot have discrete children. ‐ mixture (weighted average) of Gaussians weight = probability of discrete assignment 22
Cmput 651 - Hybrid Networks 24/11/2008 CLG example Country Gender Height Weight is CLG with Weight continuous parent height discrete parents country and gender p ( W | h, c, g ) = N ( β c,g, 0 + β c,g, 1 h ; σ 2 c,g ) 23
Cmput 651 - Hybrid Networks 24/11/2008 Discrete nodes with continuous parents Option 1 ‐ hard threshold: eg: continuous X ‐> discrete Y Y = 0 if X < 3.4 and 1 otherwise hard threshold not differentiable no gradient learning hard threshold often not realistic Option 2 ‐ soft threshold: linear sigmoid (logistic) multivariate logit NOTE: Nonlinearity! 24
Cmput 651 - Hybrid Networks 24/11/2008 Linear sigmoid (Logistic or soft threshold) exp( θ T x ) p ( Y = 1 | x ) = 1 + exp( θ T x ) P(Y=1|x) x 25
Cmput 651 - Hybrid Networks 24/11/2008 Price Multivariate logit Trade Eg: stock trading buy (red) P(trade|price) hold (green) sell (blue) as function of stock price l buy = ‐3*(price‐18) l hold = 1 l sell = 3*(price‐22) 26 Price
Cmput 651 - Hybrid Networks 24/11/2008 Discrete node with discrete & continuous parents Continuous parents’ input filtered through multivariate logit Assignment to discrete parents’ determines coefficients for logit 27
Cmput 651 - Hybrid Networks 24/11/2008 Strategy Price Example hybrid net Trade stock trade (discrete) = {buy, hold, sell} parents: price (continuous) , strategy (discrete) = {1 or 2} strategy 1 (reddish) P(trade|price,strategy) l buy = ‐3*(price‐18) l hold = 1 l sell = 3*(price‐22) strategy 2 (blue/green) l buy = ‐3*(price‐16) l hold = 1 l sell = 1*(price‐26) Price 28
Recommend
More recommend
