Linear Programming in Bounded Tree-width Markov Networks Percy - PowerPoint PPT Presentation

Linear Programming in Bounded Tree-width Markov Networks Percy Liang Nati Srebro MIT U. Toronto Workshop on Mathematical Programming in Data Mining and Machine Learning / June 1, 2005 1

Motivation: Multivariate density estimation Goal: to model the dependencies between a set of random variables ? ? Linear Programming in Bounded Tree-width Markov Networks / Motivation 2

Hypertrees Use Markov networks. Control complexity by limiting tree-width k . k=1 k=2 Weight of a hyperedge (clique) quantifies the importance of modeling the dependencies between the variables in the hyperedge. The maximum hypertree problem: Input: weights of all candidate k -hyperedges Output: a maximum weight k -hypertree Linear Programming in Bounded Tree-width Markov Networks / Motivation 3

1-windmill farms A windmill farm is a subset of the hyperedges of a hypertree. a 1-windmill (star) a tree a 1-windmill farm in the tree a 1-windmill farm in the tree Linear Programming in Bounded Tree-width Markov Networks / Motivation 4

2-windmill farms a 2-windmill a 2-hypertree a 2-windmill farm in a 2-hypertree a 2-windmill farm in a 2-hypertree Linear Programming in Bounded Tree-width Markov Networks / Motivation 5

Using windmills to approximate hypertrees • A linear programming relaxation finds a windmill farm 1 with weight 8 k k ! of the maximum windmill farm 1 • The maximum windmill farm captures at least ( k +1)! of the weight of a hypertree • Conclusion: The LP-based algorithm can find a 1 hypertree with weight 8 k k !( k +1)! of the optimal hypertree Linear Programming in Bounded Tree-width Markov Networks / Motivation 6

Analyzing the windmill coverage ratio C k = the fraction of the weight of a k -hypertree that can be captured by a maximum weight k -windmill farm 1 1 ≤ C k ≤ ( k + 1)! k +1 Previous lower bound Previous upper bound Question: What is C k ? Approach: find the “worst” hypertrees for which the weight of the maximum windmill farm is minimized Linear Programming in Bounded Tree-width Markov Networks / Motivation 7

Analyzing the windmill coverage Assume all weights are non-negative and weight of the hypertree w ( T ) = 1 . 1. Given a weighted hypertree ( T, w ) , find C k ( T, w ) = max F ⊂ T w ( F ) the maximum weight windmill farm F . Linear Programming in Bounded Tree-width Markov Networks / Question 8

Analyzing the windmill coverage Assume all weights are non-negative and weight of the hypertree w ( T ) = 1 . 1. Given a weighted hypertree ( T, w ) , find C k ( T, w ) = max F ⊂ T w ( F ) the maximum weight windmill farm F . 2. Given an unweighted hypertree structure C k ( T ) = min w max F ⊂ T w ( F ) T , find the “worst” weights w . Linear Programming in Bounded Tree-width Markov Networks / Question 8

Analyzing the windmill coverage Assume all weights are non-negative and weight of the hypertree w ( T ) = 1 . 1. Given a weighted hypertree ( T, w ) , find C k ( T, w ) = max F ⊂ T w ( F ) the maximum weight windmill farm F . 2. Given an unweighted hypertree structure C k ( T ) = min w max F ⊂ T w ( F ) T , find the “worst” weights w . 3. Find the “worst” weighted hypertree C k = inf T min w max F ⊂ T w ( F ) ( T, w ) . Linear Programming in Bounded Tree-width Markov Networks / Question 8

Analyzing the windmill coverage Assume all weights are non-negative and weight of the hypertree w ( T ) = 1 . 1. Given a weighted hypertree ( T, w ) , find C k ( T, w ) = max F ⊂ T w ( F ) the maximum weight windmill farm F . 2. Given an unweighted hypertree structure C k ( T ) = min w max F ⊂ T w ( F ) T , find the “worst” weights w . 3. Find the “worst” weighted hypertree C k = inf T min w max F ⊂ T w ( F ) ( T, w ) . Plan: solve problems 1, 2, and 3 for trees and then for hypertrees. Linear Programming in Bounded Tree-width Markov Networks / Question 8

Problem 1: C k =1 ( T, w ) = max F ⊂ T w ( F ) Goal: find the maximum weight windmill farm in a weighted tree. 1 2 4 9 2 6 0 1 4 5 1 2 3 1 4 1 1 1 5 1 1 0 9 w ( F ) = 1 + 4 + 4 + 6 + 2 + 3 + 5 + 9 + 9 + 5 + 1 + 1 = 50 Linear Programming in Bounded Tree-width Markov Networks / Problem 1 (trees) 9

Problem 1: C k =1 ( T, w ) = max F ⊂ T w ( F ) Solve using dynamic programming: e Linear Programming in Bounded Tree-width Markov Networks / Problem 1 (trees) 10

Problem 1: C k =1 ( T, w ) = max F ⊂ T w ( F ) Find the maximum weight windmill farm given the state of the root vertex. 3 vertex states: free regular blocked f v,i,s = maximum weight of a 1-windmill farm in subtree ( v, i ) with vertex v in state s Linear Programming in Bounded Tree-width Markov Networks / Problem 1 (trees) 11

Problem 1: C k =1 ( T, w ) = max F ⊂ T w ( F ) 3 vertex states: free = + regular blocked f v,i, × = f v,i 1 , × + f c i , 1 , ◦ + Linear Programming in Bounded Tree-width Markov Networks / Problem 1 (trees) 12

Problem 1: C k =1 ( T, w ) = max F ⊂ T w ( F ) = max { + , 3 vertex states: free } + + regular blocked f v,i, • = max { f v,i 1 , • + f c i , 1 , ◦ , + f v,i 1 , • + f c i , 1 , × + w v,c i } + Linear Programming in Bounded Tree-width Markov Networks / Problem 1 (trees) 13

Problem 1: C k =1 ( T, w ) = max F ⊂ T w ( F ) = max { , 3 vertex states: + , free regular blocked } + + f v,i, ◦ = max { f v,i, • , f v,i 1 , ◦ + f c i , 1 , ◦ , + f v,i 1 , × + f c i , 1 , • + w v,c i } + Linear Programming in Bounded Tree-width Markov Networks / Problem 1 (trees) 14

Problem 1: C k =1 ( T, w ) = max F ⊂ T w ( F ) Compute all dynamic programming states f v,i,s in O ( | V | ) time: f v,i, × = f v,i 1 , × + f c i , 1 , ◦ + f v,i, • = max { f v,i 1 , • + f c i , 1 , ◦ , + f v,i 1 , • + f c i , 1 , × + w v,c i } + f v,i, ◦ = max { f v,i, • , f v,i 1 , ◦ + f c i , 1 , ◦ , + f v,i 1 , × + f c i , 1 , • + w v,c i } + Linear Programming in Bounded Tree-width Markov Networks / Problem 1 (trees) 15

Problem 2: C k =1 ( T ) = min w max F ⊂ T w ( F ) Preliminary step: convert the dynamic program into an equivalent linear program. Compute f root ( T ) , 1 , ◦ f v,i, × = f v,i 1 , × + f ci, 1 , ◦ + f v,i, • = max { f v,i 1 , • + f ci, 1 , ◦ , + 1 , • + f ci, 1 , × + w v,ci } f v,i + f v,i, ◦ = max { f v,i, • , 1 , ◦ + f ci, 1 , ◦ , f v,i + f v,i 1 , × + f ci, 1 , • + w v,ci } + Linear Programming in Bounded Tree-width Markov Networks / Problem 1 (trees) 16

Problem 2: C k =1 ( T ) = min w max F ⊂ T w ( F ) Preliminary step: convert the dynamic program into an equivalent linear program. Compute f root ( T ) , 1 , ◦ Minimize f root ( T ) , 1 , ◦ f v,i, × = f v,i 1 , × + f ci, 1 , ◦ f v,i, × ≥ f v,i 1 , × + f ci, 1 , ◦ + + f v,i, • = max { f v,i 1 , • + f ci, 1 , ◦ , f v,i, • ≥ f v,i 1 , • + f ci, 1 , ◦ + + ⇔ 1 , • + f ci, 1 , × + w v,ci } f v,i, • ≥ f v,i 1 , • + f ci, 1 , × + w v,ci f v,i + + f v,i, ◦ = max { f v,i, • , f v,i, ◦ ≥ f v,i, • 1 , ◦ + f ci, 1 , ◦ , f v,i, ◦ ≥ f v,i 1 , ◦ + f ci, 1 , ◦ f v,i + + f v,i 1 , × + f ci, 1 , • + w v,ci } f v,i, ◦ ≥ f v,i 1 , × + f ci, 1 , • + w v,ci + + Linear Programming in Bounded Tree-width Markov Networks / Problem 1 (trees) 16

Problem 2: C k =1 ( T ) = min w max F ⊂ T w ( F ) Preliminary step: convert the dynamic program into an equivalent linear program. Compute f root ( T ) , 1 , ◦ Minimize f root ( T ) , 1 , ◦ f v,i, × = f v,i 1 , × + f ci, 1 , ◦ f v,i, × ≥ f v,i 1 , × + f ci, 1 , ◦ + + f v,i, • = max { f v,i 1 , • + f ci, 1 , ◦ , f v,i, • ≥ f v,i 1 , • + f ci, 1 , ◦ + + ⇔ 1 , • + f ci, 1 , × + w v,ci } f v,i, • ≥ f v,i 1 , • + f ci, 1 , × + w v,ci f v,i + + f v,i, ◦ = max { f v,i, • , f v,i, ◦ ≥ f v,i, • 1 , ◦ + f ci, 1 , ◦ , f v,i, ◦ ≥ f v,i 1 , ◦ + f ci, 1 , ◦ f v,i + + f v,i 1 , × + f ci, 1 , • + w v,ci } f v,i, ◦ ≥ f v,i 1 , × + f ci, 1 , • + w v,ci + + max F ⊂ T w ( F ) = Af ≥ Bw f root ( T ) , 1 , ◦ min Linear Programming in Bounded Tree-width Markov Networks / Problem 1 (trees) 16

Problem 2: C k =1 ( T ) = min w max F ⊂ T w ( F ) max F ⊂ T w ( F ) = f : Af ≥ Bw f root ( T ) , 1 , ◦ min Linear Programming in Bounded Tree-width Markov Networks / Problem 2 (trees) 17

Problem 2: C k =1 ( T ) = min w max F ⊂ T w ( F ) min max F ⊂ T w ( F ) = min f : Af ≥ Bw f root ( T ) , 1 , ◦ min w w w ≥ 0; � w i =1 w ≥ 0; � w i =1 Linear Programming in Bounded Tree-width Markov Networks / Problem 2 (trees) 17

Problem 2: C k =1 ( T ) = min w max F ⊂ T w ( F ) min max F ⊂ T w ( F ) = min f : Af ≥ Bw f root ( T ) , 1 , ◦ min w w w ≥ 0; � w i =1 w ≥ 0; � w i =1 A single linear program: min f root ( T ) , 1 , ◦ w,f w ≥ 0; � w i =1; Af ≥ Bw Linear Programming in Bounded Tree-width Markov Networks / Problem 2 (trees) 17

Problem 3: C k =1 = inf T min w max F ⊂ T w ( F ) • Observation: A weighted tree with a weight 0 edge is equivalent to a weighted tree without the edge • Construct a family of tree structures { T b,h | b, h = 1 , 2 , 3 , . . . } (branching factor b , height h ) that contains each tree structure • C k =1 = lim b,h →∞ C k =1 ( T b,h ) We solve the linear program and get C k =1 = 1 2 Linear Programming in Bounded Tree-width Markov Networks / Problem 3 (trees) 18