Introduction Out with the old ... CSCE 970 CSCE 970 Lecture 8: - PDF document

Introduction Out with the old ... CSCE 970 CSCE 970 Lecture 8: Lecture 8: Structured Structured We now know how to answer the question: CSCE 970 Lecture 8: Prediction Prediction Does this picture contain a cat? Stephen Scott Stephen Scott Structured Prediction and Vinod and Vinod Variyam Variyam Introduction Introduction Definitions Definitions Stephen Scott and Vinod Variyam Applications Applications Graphical Graphical Models Models (Adapted from Sebastian Nowozin and Christoph H. Lampert) Training Training E.g., convolutional layers feeding connected layers feeding softmax sscott@cse.unl.edu 1 / 80 2 / 80 Introduction Outline ... and in with the new. CSCE 970 CSCE 970 Lecture 8: Lecture 8: What we want to know now is: Where are the cats? Structured Structured Prediction Prediction Stephen Scott Stephen Scott and Vinod and Vinod Variyam Variyam Definitions Introduction Introduction Applications Definitions Definitions Graphical modeling of probability distributions Applications Applications Graphical Graphical Training models Models Models Training Training Inference No longer a classification problem; need more sophisticated ( structured ) output 3 / 80 4 / 80 Definitions Definitions Structured Outputs Structured Outputs (2) CSCE 970 CSCE 970 Lecture 8: Lecture 8: Structured Structured Prediction Prediction Stephen Scott Stephen Scott Can think of structured data as consisting of parts, where Most machine learning approaches learn function and Vinod and Vinod Variyam f : X ! R Variyam each part contains information, as well as how they fit together Inputs X are any kind of objects Introduction Introduction Output y is a real number (classification, regression, Definitions Definitions Text: Word sequence matters density estimation, etc.) Applications Applications Structured output learning approaches learn function Hypertext: Links between documents matter Graphical Graphical Models Models f : X ! Y Chemical structures: Relative positions of molecules Training Training Inputs X are any kind of objects matter Outputs y 2 Y are complex (structured) objects Images: Relative positions of pixels matter (images, text, audio, etc.) 5 / 80 6 / 80

Applications Applications Image Processing Image Processing (2) CSCE 970 CSCE 970 Lecture 8: Lecture 8: Structured Structured Prediction Prediction Stephen Scott Stephen Scott and Vinod and Vinod { 0 ,..., 255 } 3 ( m ⇥ n ) R 3 K { 0 ,..., 255 } 3 ( m ⇥ n ) { 0 , 1 } m ⇥ n Variyam Variyam z }| { z }| { z }| { z }| { Pose estimation: f : { images } ! { K positions & angles } Semantic image segmentation: f : { images } ! { masks } Introduction Introduction Definitions Definitions Applications Applications Graphical Graphical Models Models Training Training 7 / 80 8 / 80 Applications Applications Image Processing (3) Image Processing (4) CSCE 970 Point matching: CSCE 970 Lecture 8: Lecture 8: f : { image pairs } ! { mappings between images } Structured Structured Prediction Prediction Stephen Scott Stephen Scott and Vinod and Vinod Variyam Variyam Object localization f : { images } ! { bounding box coordinates } Introduction Introduction Definitions Definitions Applications Applications Graphical Graphical Models Models Training Training 9 / 80 10 / 80 Applications Graphical Models Others Probabilistic Modeling CSCE 970 CSCE 970 Lecture 8: Lecture 8: Structured Structured Prediction Prediction Stephen Scott Stephen Scott Natural language processing (e.g., translation; output is and Vinod and Vinod Variyam Variyam To represent structured outputs, we will often employ sentences) probabilistic modeling Introduction Introduction Bioinformatics (e.g., structure prediction; output is Joint distributions (e.g., P ( A , B , C ) ) Definitions Definitions graphs) Conditional distributions (e.g., P ( A | B , C ) ) Applications Applications Speech processing (e.g., recognition; output is Graphical Graphical Can estimate joint and conditional probabilities by Models Models sentences) counting and normalizing, but have to be careful about Directed Training Undirected Robotics (e.g., planning; output is action plan) representation Energy Separation Image denoising (output is “clean” version of image) Training 11 / 80 12 / 80

Graphical Models Graphical Models Probabilistic Modeling (2) Probabilistic Modeling (3) CSCE 970 CSCE 970 Lecture 8: Lecture 8: E.g., I have a coin with unknown probability p of heads Structured Structured Prediction Prediction I want to estimate the probability of flipping it ten times Stephen Scott Stephen Scott and getting the sequence HHTTHHTTTT and Vinod and Vinod Variyam Variyam Problem: Number of possible outcomes grows One way of representing this joint distribution is a exponentially with number of variables (flips) Introduction Introduction single, big lookup table: ) Most outcomes will have count = 0 , a few with 1, Definitions Definitions probably none with more Applications Applications Each experiment consists of ) Lousy probability estimates Graphical Graphical ten coin flips Outcome Count Models Models .. Ten flips is bad enough, but consider 100 _ Directed Directed 1 TTHHTTHHTH For each outcome, increment Undirected Undirected How would you solve this problem? Energy Energy 0 HHHTHTTTHH its counter Separation Separation 0 Training HTTTTTHHHT Training After n experiments, divide 1 TTHTHTHHTT HHTTHHTTTT ’s counter by n to . . . . get the estimate . . Will this work? 13 / 80 14 / 80 Graphical Models Graphical Models Factoring a Distribution Factoring a Distribution (2) CSCE 970 CSCE 970 Lecture 8: Lecture 8: Structured Structured Prediction Prediction Of course, we recognize that all flips are independent, Stephen Scott Stephen Scott Another example: Relay racing team and Vinod so and Vinod Variyam Variyam Pr [ HHTTHHTTTT ] = p 4 ( 1 � p ) 6 Alice, then Bob, then Carol Introduction Introduction Let t A = Alice’s finish time (in seconds), t B = Bob’s, So we can count n coin flips to estimate p and use the Definitions Definitions t C = Carol’s formula above Applications Applications Want to model the joint distribution Pr [ t A , t B , t C ] I.e., we factor the joint distribution into independent Graphical Graphical Models Models components and multiply the results: Let t C , t B , t A 2 { 1 , . . . , 1000 } Directed Directed Undirected Undirected Energy Pr [ HHTTHHTTTT ] = Pr [ f 1 = H ] Pr [ f 2 = H ] Pr [ f 3 = T ] · · · Pr [ f 10 = T ] Energy How large would the table be for Pr [ t A , t B , t C ] ? Separation Separation Training Training How many races must they run to populate the table? We greatly reduce the number of parameters to estimate 15 / 80 16 / 80 Graphical Models Graphical Models Factoring a Distribution (3) Factoring a Distribution (4) CSCE 970 CSCE 970 Lecture 8: Lecture 8: Structured Structured Prediction Prediction But we can factor this distribution by observing that t A is Stephen Scott Stephen Scott independent of t B and t C and Vinod and Vinod Variyam Variyam ) Can estimate t A on its own Also, t B directly depends on t A , but is independent of t C Introduction Introduction Definitions Definitions t C directly depends on t B , and indirectly on t A Applications Applications Can display this graphically: This directed graphical model (often called a Graphical Graphical Models Models Bayesian network or Bayes net ) represents Directed Directed conditional dependencies among variables Undirected Undirected Energy Energy Separation Separation Makes factoring easy: Training Training Pr [ t A , t B , t C ] = Pr [ t A ] Pr [ t B | t A ] Pr [ t C | t B ] 17 / 80 18 / 80