Clusters and features from combinatorial stochastic processes - PowerPoint PPT Presentation

Exchangeable probability functions “Exchangeable feature probability function” (EFPF)? Counterexample n = 1 P (row = ) = p 1 2 P (row = ) = p 2 ... P (row = ) = p 3 N P (row = ) = p 4 P ( ) � = P ( ) p 1 p 2 � = p 3 p 4 [Broderick, Jordan, Pitman 2012] 9

Exchangeable probability functions “Exchangeable feature probability function” (EFPF)? Counterexample n = 1 P (row = ) = p 1 2 P (row = ) = p 2 ... P (row = ) = p 3 N P (row = ) = p 4 P ( ) � = P ( ) p 1 p 2 � = p 3 p 4 [Broderick, Jordan, Pitman 2012] 9

Exchangeable probability functions Exchangeable cluster distributions = Cluster distributions with EPPFs Exchangeable feature distributions IBP Two-feature example Feature distributions with EFPFs [Broderick, Jordan, Pitman 2012] 10

Paintboxes Exchangeable partition: Kingman paintbox [Kingman 1978] 11

Paintboxes Exchangeable partition: Kingman paintbox [Kingman 1978] 11

Paintboxes Exchangeable partition: Kingman paintbox [Kingman 1978] 11

Paintboxes Exchangeable partition: Kingman paintbox 1 1 [Kingman 1978] 11

Paintboxes Exchangeable partition: Kingman paintbox 2 1 1 2 [Kingman 1978] 11

Paintboxes Exchangeable partition: Kingman paintbox 3 2 1 1 2 3 [Kingman 1978] 11

Paintboxes Exchangeable partition: Kingman paintbox 3 2 1 4 1 2 3 4 [Kingman 1978] 11

Paintboxes Exchangeable partition: Kingman paintbox 3 5 2 1 4 1 2 3 4 5 [Kingman 1978] 11

Paintboxes Exchangeable partition: Kingman paintbox 3 5 2 6 1 4 1 2 3 4 5 6 [Kingman 1978] 11

Paintboxes Exchangeable partition: Kingman paintbox 3 5 2 6 7 1 4 1 2 3 4 5 6 7 [Kingman 1978] 11

Paintboxes Exchangeable partition: Kingman paintbox 3 5 2 6 7 1 4 ... Cat cluster Dog cluster 1 2 3 4 5 6 7 [Kingman 1978] 11

Paintboxes Exchangeable partition: Kingman paintbox 3 5 2 6 7 1 4 Cat cluster Dog cluster Mouse cluster Lizard cluster 1 Sheep cluster 2 Horse cluster 3 4 5 6 7 [Kingman 1978] 12

Paintboxes Cat feature Dog feature Mouse feature Lizard feature Sheep feature Horse feature [Broderick, Pitman, Jordan (submitted)] 13

Paintboxes Exchangeable feature allocation: feature paintbox Cat feature Dog feature Mouse feature Lizard feature Sheep feature Horse feature [Broderick, Pitman, Jordan (submitted)] 13

Paintboxes Exchangeable feature allocation: feature paintbox 1 Cat feature Dog feature Mouse feature Lizard feature 1 Sheep feature Horse feature [Broderick, Pitman, Jordan (submitted)] 13

Paintboxes Exchangeable feature allocation: feature paintbox 2 1 Cat feature Dog feature Mouse feature Lizard feature 1 Sheep feature 2 Horse feature [Broderick, Pitman, Jordan (submitted)] 13

Paintboxes Exchangeable feature allocation: feature paintbox 3 2 1 Cat feature Dog feature Mouse feature Lizard feature 1 Sheep feature 2 Horse feature 3 [Broderick, Pitman, Jordan (submitted)] 13

Paintboxes Exchangeable feature allocation: feature paintbox 3 2 1 4 Cat feature Dog feature Mouse feature Lizard feature 1 Sheep feature 2 Horse feature 3 4 [Broderick, Pitman, Jordan (submitted)] 13

Paintboxes Exchangeable feature allocation: feature paintbox 3 5 2 1 4 Cat feature Dog feature Mouse feature Lizard feature 1 Sheep feature 2 Horse feature 3 4 5 [Broderick, Pitman, Jordan (submitted)] 13

Paintboxes Exchangeable feature allocation: feature paintbox 3 5 2 6 1 4 Cat feature Dog feature Mouse feature Lizard feature 1 Sheep feature 2 Horse feature 3 4 5 6 [Broderick, Pitman, Jordan (submitted)] 13

Paintboxes Exchangeable feature allocation: feature paintbox 3 5 2 6 7 1 4 Cat feature Dog feature Mouse feature Lizard feature 1 Sheep feature 2 Horse feature 3 4 5 6 7 [Broderick, Pitman, Jordan (submitted)] 13

Paintboxes Exchangeable cluster distributions = Cluster distributions with EPPFs Exchangeable feature distributions IBP Two-feature example Feature distributions with EFPFs [Broderick, Jordan, Pitman 2012] 14

Paintboxes Exchangeable cluster distributions Exchangeable feature distributions = Cluster distributions with EPPFs = Feature paintbox allocations = Kingman paintbox partitions IBP Two-feature example Feature distributions with EFPFs [Broderick, Pitman, Jordan (submitted)] 14

Paintboxes Two feature example Feature 1 Feature 2 p 1 p 2 p 3 p 4 P (row = ) = p 1 P (row = ) = p 2 P (row = ) = p 3 P (row = ) = p 4 15

Paintboxes Indian buffet process: beta feature frequencies [Thibaux, Jordan 2007] 16

Paintboxes Indian buffet process: beta feature frequencies For m = 1, 2, ... � � θ 1. Draw K + γ m = Poisson θ + m − 1 m Set � K + K m = m j =1 2. For k = K m − 1 , . . . , K m Draw an atom mass of size q k ∼ Beta(1 , θ + m − 1) [Thibaux, Jordan 2007] 16

Paintboxes Indian buffet process: beta feature frequencies For m = 1, 2, ... � � θ 1. Draw K + γ m = Poisson θ + m − 1 m Set � K + K m = m j =1 2. For k = K m − 1 , . . . , K m Draw a frequency of size q k ∼ Beta(1 , θ + m − 1) [Thibaux, Jordan 2007] 16

Paintboxes Indian buffet process: beta feature frequencies 1 For m = 1, 2, ... � � θ 1. Draw K + γ m = Poisson θ + m − 1 m Set � K + K m = m j =1 2. For k = K m − 1 , . . . , K m Draw a frequency of size q k ∼ Beta(1 , θ + m − 1) 0 [Thibaux, Jordan 2007] 16

Paintboxes Indian buffet process: beta feature frequencies 1 For m = 1, 2, ... � � θ 1. Draw K + γ m = Poisson q 1 θ + m − 1 m Set � K + K m = m j =1 2. For k = K m − 1 , . . . , K m Draw a frequency of size q 2 q k ∼ Beta(1 , θ + m − 1) 0 [Thibaux, Jordan 2007] 16

Paintboxes Indian buffet process: beta feature frequencies 1 For m = 1, 2, ... � � θ 1. Draw K + γ m = Poisson q 1 θ + m − 1 m Set � K + K m = m j =1 2. For k = K m − 1 , . . . , K m Draw a frequency of size q 2 q 3 q k ∼ Beta(1 , θ + m − 1) 0 [Thibaux, Jordan 2007] 16

Paintboxes Indian buffet process: beta feature frequencies 1 For m = 1, 2, ... � � θ 1. Draw K + γ m = Poisson q 1 θ + m − 1 m Set � K + K m = m j =1 2. For k = K m − 1 , . . . , K m Draw a frequency of size q 2 q 3 q k ∼ Beta(1 , θ + m − 1) q 6 q 4 q 5 0 [Thibaux, Jordan 2007] 16

Paintboxes Indian buffet process: beta feature frequencies 1 For m = 1, 2, ... � � θ 1. Draw K + γ m = Poisson q 1 θ + m − 1 m Set � K + K m = m j =1 2. For k = K m − 1 , . . . , K m Draw a frequency of size q 2 q 3 q k ∼ Beta(1 , θ + m − 1) q 6 ... q 4 q 5 0 [Thibaux, Jordan 2007] 16

Paintboxes Indian buffet process: beta feature frequencies 1 For m = 1, 2, ... � � θ 1. Draw K + γ m = Poisson q 1 θ + m − 1 m Set � K + K m = m j =1 2. For k = K m − 1 , . . . , K m Draw a frequency of size q 2 q 3 q k ∼ Beta(1 , θ + m − 1) q 6 ... q 4 q 5 0 [Thibaux, Jordan 2007] 16

Paintboxes Indian buffet process: beta feature frequencies 1 For m = 1, 2, ... � � θ 1. Draw K + γ m = Poisson q 1 θ + m − 1 m Set � K + K m = m j =1 2. For k = K m − 1 , . . . , K m Draw a frequency of size q 2 q 3 q k ∼ Beta(1 , θ + m − 1) q 6 ... q 4 q 5 0 [Thibaux, Jordan 2007] 16

Paintboxes Indian buffet process: beta feature frequencies 1 For m = 1, 2, ... � � θ 1. Draw K + γ m = Poisson q 1 θ + m − 1 m Set � K + K m = m j =1 2. For k = K m − 1 , . . . , K m Draw a frequency of size q 2 q 3 q k ∼ Beta(1 , θ + m − 1) q 6 ... q 4 q 5 0 [Thibaux, Jordan 2007] 16

Paintboxes Indian buffet process: beta feature frequencies 1 For m = 1, 2, ... � � θ 1. Draw K + γ m = Poisson q 1 θ + m − 1 m Set � K + K m = m j =1 2. For k = K m − 1 , . . . , K m Draw a frequency of size q 2 q 3 q k ∼ Beta(1 , θ + m − 1) q 6 ... q 4 q 5 0 [Thibaux, Jordan 2007] 16

Paintboxes Indian buffet process: beta feature frequencies 1 0 q 1 q 2 q 3 q 4 q 5 q 6 ... 17

Paintboxes Indian buffet process: beta feature frequencies 18

Paintboxes Indian buffet process: beta feature frequencies 18

Paintboxes Indian buffet process: beta feature frequencies 18

Paintboxes Indian buffet process: beta feature frequencies 18

Paintboxes Indian buffet process: beta feature frequencies ... 18

Paintboxes 1 q 1 q 2 q 3 q 6 ... q 4 q 5 0 19

Paintboxes 1 q 1 “Frequency models” q 2 q 3 q 6 ... q 4 q 5 0 19 [Broderick, Pitman, Jordan (submitted)]

Paintboxes Two feature example Feature 1 Feature 2 p 1 p 2 p 3 p 4 P (row = ) = p 1 P (row = ) = p 2 P (row = ) = p 3 P (row = ) = p 4 20

Paintboxes Two feature example Not a frequency model Feature 1 Feature 2 p 1 p 2 p 3 p 4 P (row = ) = p 1 P (row = ) = p 2 P (row = ) = p 3 P (row = ) = p 4 20

Paintboxes Exchangeable cluster distributions Exchangeable feature distributions = Cluster distributions with EPPFs = Feature paintbox allocations = Kingman paintbox partitions IBP Two-feature example Feature distributions with EFPFs [Broderick, Pitman, Jordan (submitted)] 21

Paintboxes Exchangeable cluster distributions Exchangeable feature distributions = Cluster distributions with EPPFs = Feature paintbox allocations = Kingman paintbox partitions IBP Two-feature example Frequency models [Broderick, Pitman, Jordan (submitted)] 21

Frequency models: EFPFs? 1 q 1 q 2 q 3 q 6 ... q 4 q 5 0 22

Frequency models: EFPFs? 1 q 1 q 2 q 3 q 6 ... q 4 q 5 0 22

Frequency models: EFPFs? 1 q 1 q 2 q 3 q 6 ... q 4 q 5 0 22

Frequency models: EFPFs? ... k = 1 2 K 1 n = 1 2 ... q 1 N q 2 q 3 q 6 ... q 4 q 5 0 22

Frequency models: EFPFs? ... k = 1 2 K 1 n = 1 2 ) P ( ... q 1 N q 2 q 3 q 6 ... q 4 q 5 0 22

Frequency models: EFPFs? ... k = 1 2 K 1 n = 1 2 ) P ( ... q 1 N q 2 q 3 q 6 ... q 4 q 5 0 22

Frequency models: EFPFs? ... k = 1 2 K 1 n = 1 2 ) P ( ... q 1 N q S N, 1 (1 − q 1 ) N − S N, 1 q 2 q 3 1 q 6 ... q 4 q 5 0 22

Frequency models: EFPFs? ... k = 1 2 K 1 n = 1 2 ) P ( ... q 1 N q S N,k (1 − q k ) N − S N,k q 2 q 3 k q 6 ... q 4 q 5 0 22

Frequency models: EFPFs? ... k = 1 2 K 1 n = 1 2 ) P ( ... q 1 N q S N,k (1 − q i k ) N − S N,k q 2 q 3 i k q 6 ... q 4 q 5 0 23

Frequency models: EFPFs? ... k = 1 2 K 1 n = 1 2 ) P ( ... q 1 N K q S N,k � (1 − q i k ) N − S N,k q 2 q 3 i k k =1 q 6 ... q 4 q 5 0 23

Frequency models: EFPFs? ... k = 1 2 K 1 n = 1 2 ) P ( ... q 1 N K q S N,k � (1 − q i k ) N − S N,k q 2 q 3 i k k =1 � (1 − q j ) N q 6 · ... q 4 ∈ { i k } K j / q 5 k =1 0 23

Frequency models: EFPFs? ... k = 1 2 K 1 n = 1 2 ) P ( ... q 1 N K 1 q S N,k � � (1 − q i k ) N − S N,k = E [ q 2 q 3 i k K ! distinct i k k =1 � (1 − q j ) N ] q 6 · ... q 4 ∈ { i k } K j / q 5 k =1 0 23