GTC 2018 GTC 2018 Motivation Discovering Order in Unordered - PowerPoint PPT Presentation

GTC 2018 GTC 2018 Motivation Discovering Order in Unordered Datasets: GMN Generative Markov Networks Experiments Conclusion Yao-Hung Hubert Tsai † , Han Zhao † , Nebojsa Jojic ‡ , and Ruslan Salakhutdinov † † Machine Learning Department, Carnegie Mellon University ‡ Microsoft Research 1 / 32

A Novel Question GTC 2018 Motivation GMN Experiments Given an unordered dataset where instances Conclusion may be exhibiting an implicit order , can we recover this order? 2 / 32

Motivations GTC 2018 Motivation GMN Experiments Data samples are Conclusion (X) independently identically distributed (O) possessing an unknown order 3 / 32

Motivations GTC 2018 Motivation GMN Experiments Data samples are Conclusion (X) independently identically distributed (O) possessing an unknown order 3 / 32

Motivations GTC 2018 Motivation GMN Experiments Data samples are Conclusion (X) independently identically distributed (O) possessing an unknown order 3 / 32

Motivations (cont’d) GTC 2018 Photos from same person , same place , same dressing , but different days . Motivation GMN Experiments Conclusion Day 1 Day 2 Day 3 Day 4 Day 5 I.i.d. assumption is justified . 4 / 32

Motivations (cont’d) GTC 2018 Photos from same person , same place , same dressing , but different days . Motivation GMN Experiments Conclusion Day 1 Day 2 Day 3 Day 4 Day 5 I.i.d. assumption is justified . 4 / 32

Motivations (cont’d) GTC 2018 After rearrangement.... Motivation GMN Experiments Conclusion Day 5 Day 2 Day 3 Day 4 Day 1 Implicit order is observed. 5 / 32

Motivations (cont’d) GTC 2018 After rearrangement.... Motivation GMN Experiments Conclusion Day 5 Day 2 Day 3 Day 4 Day 1 Implicit order is observed. 5 / 32

Put it Differently.... GTC 2018 Motivation GMN Data generative process that generates data Experiments i.i.d. (X) Conclusion sequentially (O) parameterized with fewer parameters learned more easily Propose a novel Markov network chain generative scheme. 6 / 32

Put it Differently.... GTC 2018 Motivation GMN Data generative process that generates data Experiments i.i.d. (X) Conclusion sequentially (O) parameterized with fewer parameters learned more easily Propose a novel Markov network chain generative scheme. 6 / 32

Put it Differently.... GTC 2018 Motivation GMN Data generative process that generates data Experiments i.i.d. (X) Conclusion sequentially (O) parameterized with fewer parameters learned more easily Propose a novel Markov network chain generative scheme. 6 / 32

Put it Differently.... GTC 2018 Motivation GMN Data generative process that generates data Experiments i.i.d. (X) Conclusion sequentially (O) parameterized with fewer parameters learned more easily Propose a novel Markov network chain generative scheme. 6 / 32

Put it Differently.... GTC 2018 Motivation GMN Data generative process that generates data Experiments i.i.d. (X) Conclusion sequentially (O) parameterized with fewer parameters learned more easily Propose a novel Markov network chain generative scheme. 6 / 32

Put it Differently.... GTC 2018 Motivation GMN Data generative process that generates data Experiments i.i.d. (X) Conclusion sequentially (O) parameterized with fewer parameters learned more easily Propose a novel Markov network chain generative scheme. 6 / 32

Motivations (cont’d) GTC 2018 Motivation GMN Other examples/ applications: Experiments Slow-moving human diseases (i.e., Alzheimers or Conclusion Parkinsons ) understanding Galaxy or star evolution Cellular or molecular biological processes Recover the order from a snapshot of individual samples . 7 / 32

Motivations (cont’d) GTC 2018 Motivation GMN Other examples/ applications: Experiments Slow-moving human diseases (i.e., Alzheimers or Conclusion Parkinsons ) understanding Galaxy or star evolution Cellular or molecular biological processes Recover the order from a snapshot of individual samples . 7 / 32

Motivations (cont’d) GTC 2018 Motivation GMN Other examples/ applications: Experiments Slow-moving human diseases (i.e., Alzheimers or Conclusion Parkinsons ) understanding Galaxy or star evolution Cellular or molecular biological processes Recover the order from a snapshot of individual samples . 7 / 32

Motivations (cont’d) GTC 2018 Motivation GMN Other examples/ applications: Experiments Slow-moving human diseases (i.e., Alzheimers or Conclusion Parkinsons ) understanding Galaxy or star evolution Cellular or molecular biological processes Recover the order from a snapshot of individual samples . 7 / 32

Motivations (cont’d) GTC 2018 Motivation GMN Other examples/ applications: Experiments Slow-moving human diseases (i.e., Alzheimers or Conclusion Parkinsons ) understanding Galaxy or star evolution Cellular or molecular biological processes Recover the order from a snapshot of individual samples . 7 / 32

Datasets Sorting GTC 2018 Motivation GMN Experiments Use predefined distance metric Conclusion Euclidean distance between image pixel values. p − distance between DNA/RNA sequences. Not generalize well. 8 / 32

Datasets Sorting GTC 2018 Motivation GMN Experiments Use predefined distance metric Conclusion Euclidean distance between image pixel values. p − distance between DNA/RNA sequences. Not generalize well. 8 / 32

Datasets Sorting GTC 2018 Motivation GMN Experiments Use predefined distance metric Conclusion Euclidean distance between image pixel values. p − distance between DNA/RNA sequences. Not generalize well. 8 / 32

Datasets Sorting GTC 2018 Motivation GMN Experiments Use predefined distance metric Conclusion Euclidean distance between image pixel values. p − distance between DNA/RNA sequences. Not generalize well. 8 / 32

Our Proposed Method GTC 2018 Motivation GMN Data are sampled from a Markov chain . Experiments s 1 → s 2 → · · · → s n Conclusion T ( s t | s t − 1 ; θ ) Propose a greedy batch-wise permutation scheme . N : total # of data, b : batch # of data O ( N b log b ) ( b is constant) cf. O ( N !) which is NP-Hard 9 / 32

Our Proposed Method GTC 2018 Motivation GMN Data are sampled from a Markov chain . Experiments s 1 → s 2 → · · · → s n Conclusion T ( s t | s t − 1 ; θ ) Propose a greedy batch-wise permutation scheme . N : total # of data, b : batch # of data O ( N b log b ) ( b is constant) cf. O ( N !) which is NP-Hard 9 / 32

Our Proposed Method GTC 2018 Motivation GMN Data are sampled from a Markov chain . Experiments s 1 → s 2 → · · · → s n Conclusion T ( s t | s t − 1 ; θ ) Propose a greedy batch-wise permutation scheme . N : total # of data, b : batch # of data O ( N b log b ) ( b is constant) cf. O ( N !) which is NP-Hard 9 / 32

Our Proposed Method GTC 2018 Motivation GMN Data are sampled from a Markov chain . Experiments s 1 → s 2 → · · · → s n Conclusion T ( s t | s t − 1 ; θ ) Propose a greedy batch-wise permutation scheme . N : total # of data, b : batch # of data O ( N b log b ) ( b is constant) cf. O ( N !) which is NP-Hard 9 / 32

Our Proposed Method GTC 2018 Motivation GMN Data are sampled from a Markov chain . Experiments s 1 → s 2 → · · · → s n Conclusion T ( s t | s t − 1 ; θ ) Propose a greedy batch-wise permutation scheme . N : total # of data, b : batch # of data O ( N b log b ) ( b is constant) cf. O ( N !) which is NP-Hard 9 / 32

Problem Setup GTC 2018 Motivation π : permutation over [ n ] GMN Experiments s π (1) → s π (2) → · · · → s π ( n ) Conclusion Joint log-likelihood estimation problem on θ in transitional operator T ( s ′ | s ; θ ) Optimal π 10 / 32

Problem Setup GTC 2018 Motivation π : permutation over [ n ] GMN Experiments s π (1) → s π (2) → · · · → s π ( n ) Conclusion Joint log-likelihood estimation problem on θ in transitional operator T ( s ′ | s ; θ ) Optimal π 10 / 32

Problem Setup GTC 2018 Motivation π : permutation over [ n ] GMN Experiments s π (1) → s π (2) → · · · → s π ( n ) Conclusion Joint log-likelihood estimation problem on θ in transitional operator T ( s ′ | s ; θ ) Optimal π 10 / 32

Problem Setup GTC 2018 Motivation π : permutation over [ n ] GMN Experiments s π (1) → s π (2) → · · · → s π ( n ) Conclusion Joint log-likelihood estimation problem on θ in transitional operator T ( s ′ | s ; θ ) Optimal π 10 / 32

Problem Setup GTC 2018 Motivation π : permutation over [ n ] GMN Experiments s π (1) → s π (2) → · · · → s π ( n ) Conclusion Joint log-likelihood estimation problem on θ in transitional operator T ( s ′ | s ; θ ) Optimal π 10 / 32

T (transitional operator) Illustration GTC 2018 Motivation GMN Experiments 1 ! (permutation) 2 Simultaneously learn Conclusion Apply Train GMNs for green circle (without order) Apply trained transitional operator to blue circle 11 / 32