structured perceptron margin methods
play

Structured Perceptron/ Margin Methods Graham Neubig Site - PowerPoint PPT Presentation

Jul 02, 2023 •312 likes •603 views

CS11-747 Neural Networks for NLP Structured Perceptron/ Margin Methods Graham Neubig Site https://phontron.com/class/nn4nlp2020/ Types of Prediction Two classes ( binary classification ) positive I hate this movie negative

  1. CS11-747 Neural Networks for NLP Structured Perceptron/ Margin Methods Graham Neubig Site https://phontron.com/class/nn4nlp2020/

  2. Types of Prediction • Two classes ( binary classification ) positive I hate this movie negative • Multiple classes ( multi-class classification ) very good good I hate this movie neutral bad very bad • Exponential/infinite labels ( structured prediction ) I hate this movie PRP VBP DT NN I hate this movie kono eiga ga kirai

  3. Many Varieties of Structured Prediction! • Models: • RNN-based decoders Covered • Convolution/self attentional decoders already • CRFs w/ local factors • Training algorithms: • Maximum likelihood w/ teacher forcing • Sequence level likelihood Covered • Structured perceptron, structured large margin today • Reinforcement learning/minimum risk training • Sampling corruptions of data

  4. <latexit sha1_base64="OZ0fwiGra8uyR0OpSCMUh2f+CTQ=">ACNnicbVBSxtBGJ3VamO0GvXoZWgQEilhV4S2B0H04klS2piEbAyzs9/qkJnZWZWCMP+Ky/+DW968aDitT+hkxikTfpg4PHe+/jme1HGmTa+f+8tLH5YWv5YWimvrn1a36hsbp3rNFcUWjTlqepERANnElqGQ6dTAEREYd2NDwZ+1rUJql8pcZdAX5FKyhFinDSonDVrXRwKFuNOHR/iMFGEWriw+Get8wV360VhQ52LgQ0N4zHYboFDJvH5XvEendceFCp+g1/AjxPgimpoimag8pdGKc0FyAN5UTrXuBnpm+JMoxyKMphriEjdEguoeoJAJ0307uLvCuU2KcpMo9afBE/XvCEqH1SEQuKYi50rPeWPyf18tN8q1vmcxyA5K+LUpyjk2KxyXimCmgho8cIVQx91dMr4irzriqy6EYPbkedLab3xvBD8OqkfH0zZKaAd9RjUoK/oCJ2iJmohim7QA3pCz96t9+i9eK9v0QVvOrON/oH3+w9wk6ow</latexit> <latexit sha1_base64="OZ0fwiGra8uyR0OpSCMUh2f+CTQ=">ACNnicbVBSxtBGJ3VamO0GvXoZWgQEilhV4S2B0H04klS2piEbAyzs9/qkJnZWZWCMP+Ky/+DW968aDitT+hkxikTfpg4PHe+/jme1HGmTa+f+8tLH5YWv5YWimvrn1a36hsbp3rNFcUWjTlqepERANnElqGQ6dTAEREYd2NDwZ+1rUJql8pcZdAX5FKyhFinDSonDVrXRwKFuNOHR/iMFGEWriw+Get8wV360VhQ52LgQ0N4zHYboFDJvH5XvEendceFCp+g1/AjxPgimpoimag8pdGKc0FyAN5UTrXuBnpm+JMoxyKMphriEjdEguoeoJAJ0307uLvCuU2KcpMo9afBE/XvCEqH1SEQuKYi50rPeWPyf18tN8q1vmcxyA5K+LUpyjk2KxyXimCmgho8cIVQx91dMr4irzriqy6EYPbkedLab3xvBD8OqkfH0zZKaAd9RjUoK/oCJ2iJmohim7QA3pCz96t9+i9eK9v0QVvOrON/oH3+w9wk6ow</latexit> <latexit sha1_base64="OZ0fwiGra8uyR0OpSCMUh2f+CTQ=">ACNnicbVBSxtBGJ3VamO0GvXoZWgQEilhV4S2B0H04klS2piEbAyzs9/qkJnZWZWCMP+Ky/+DW968aDitT+hkxikTfpg4PHe+/jme1HGmTa+f+8tLH5YWv5YWimvrn1a36hsbp3rNFcUWjTlqepERANnElqGQ6dTAEREYd2NDwZ+1rUJql8pcZdAX5FKyhFinDSonDVrXRwKFuNOHR/iMFGEWriw+Get8wV360VhQ52LgQ0N4zHYboFDJvH5XvEendceFCp+g1/AjxPgimpoimag8pdGKc0FyAN5UTrXuBnpm+JMoxyKMphriEjdEguoeoJAJ0307uLvCuU2KcpMo9afBE/XvCEqH1SEQuKYi50rPeWPyf18tN8q1vmcxyA5K+LUpyjk2KxyXimCmgho8cIVQx91dMr4irzriqy6EYPbkedLab3xvBD8OqkfH0zZKaAd9RjUoK/oCJ2iJmohim7QA3pCz96t9+i9eK9v0QVvOrON/oH3+w9wk6ow</latexit> Reminder: Globally Normalized Models • Locally normalized models: each decision made by the model has a probability that adds to one | Y | e S ( y j | X,y 1 ,...,y j − 1 ) Y P ( Y | X ) = y j ∈ V e S (˜ y j | X,y 1 ,...,y j − 1 ) P ˜ j =1 • Globally normalized models (a.k.a. energy- based models): each sentence has a score, which is not normalized over a particular decision e S ( X,Y ) P ( Y | X ) = Y ∈ V ∗ e S ( X, ˜ Y ) P ˜

  5. Globally Normalized Likelihood

  6. <latexit sha1_base64="OZ0fwiGra8uyR0OpSCMUh2f+CTQ=">ACNnicbVBSxtBGJ3VamO0GvXoZWgQEilhV4S2B0H04klS2piEbAyzs9/qkJnZWZWCMP+Ky/+DW968aDitT+hkxikTfpg4PHe+/jme1HGmTa+f+8tLH5YWv5YWimvrn1a36hsbp3rNFcUWjTlqepERANnElqGQ6dTAEREYd2NDwZ+1rUJql8pcZdAX5FKyhFinDSonDVrXRwKFuNOHR/iMFGEWriw+Get8wV360VhQ52LgQ0N4zHYboFDJvH5XvEendceFCp+g1/AjxPgimpoimag8pdGKc0FyAN5UTrXuBnpm+JMoxyKMphriEjdEguoeoJAJ0307uLvCuU2KcpMo9afBE/XvCEqH1SEQuKYi50rPeWPyf18tN8q1vmcxyA5K+LUpyjk2KxyXimCmgho8cIVQx91dMr4irzriqy6EYPbkedLab3xvBD8OqkfH0zZKaAd9RjUoK/oCJ2iJmohim7QA3pCz96t9+i9eK9v0QVvOrON/oH3+w9wk6ow</latexit> <latexit sha1_base64="OZ0fwiGra8uyR0OpSCMUh2f+CTQ=">ACNnicbVBSxtBGJ3VamO0GvXoZWgQEilhV4S2B0H04klS2piEbAyzs9/qkJnZWZWCMP+Ky/+DW968aDitT+hkxikTfpg4PHe+/jme1HGmTa+f+8tLH5YWv5YWimvrn1a36hsbp3rNFcUWjTlqepERANnElqGQ6dTAEREYd2NDwZ+1rUJql8pcZdAX5FKyhFinDSonDVrXRwKFuNOHR/iMFGEWriw+Get8wV360VhQ52LgQ0N4zHYboFDJvH5XvEendceFCp+g1/AjxPgimpoimag8pdGKc0FyAN5UTrXuBnpm+JMoxyKMphriEjdEguoeoJAJ0307uLvCuU2KcpMo9afBE/XvCEqH1SEQuKYi50rPeWPyf18tN8q1vmcxyA5K+LUpyjk2KxyXimCmgho8cIVQx91dMr4irzriqy6EYPbkedLab3xvBD8OqkfH0zZKaAd9RjUoK/oCJ2iJmohim7QA3pCz96t9+i9eK9v0QVvOrON/oH3+w9wk6ow</latexit> <latexit sha1_base64="OZ0fwiGra8uyR0OpSCMUh2f+CTQ=">ACNnicbVBSxtBGJ3VamO0GvXoZWgQEilhV4S2B0H04klS2piEbAyzs9/qkJnZWZWCMP+Ky/+DW968aDitT+hkxikTfpg4PHe+/jme1HGmTa+f+8tLH5YWv5YWimvrn1a36hsbp3rNFcUWjTlqepERANnElqGQ6dTAEREYd2NDwZ+1rUJql8pcZdAX5FKyhFinDSonDVrXRwKFuNOHR/iMFGEWriw+Get8wV360VhQ52LgQ0N4zHYboFDJvH5XvEendceFCp+g1/AjxPgimpoimag8pdGKc0FyAN5UTrXuBnpm+JMoxyKMphriEjdEguoeoJAJ0307uLvCuU2KcpMo9afBE/XvCEqH1SEQuKYi50rPeWPyf18tN8q1vmcxyA5K+LUpyjk2KxyXimCmgho8cIVQx91dMr4irzriqy6EYPbkedLab3xvBD8OqkfH0zZKaAd9RjUoK/oCJ2iJmohim7QA3pCz96t9+i9eK9v0QVvOrON/oH3+w9wk6ow</latexit> Difficulties Training Globally Normalized Models • Partition function problematic e S ( X,Y ) P ( Y | X ) = Y ∈ V ∗ e S ( X, ˜ Y ) P ˜ • Two options for calculating partition function • Structure model to allow enumeration via dynamic programming, e.g. linear chain CRF, CFG • Estimate partition function through sub-sampling hypothesis space

  7. Two Methods for Approximation • Sampling: • Sample k samples according to the probability distribution • + Unbiased estimator: as k gets large will approach true distribution • - High variance: what if we get low-probability samples? • Beam search: • Search for k best hypotheses • - Biased estimator: may result in systematic differences from true distribution • + Lower variance: more likely to get high-probability outputs

  8. Un-normalized Models: Structured Perceptron

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Machine Learning Fall 2017 Structured Prediction (structured perceptron, HMM, structured SVM)

Machine Learning Fall 2017 Structured Prediction (structured perceptron, HMM, structured SVM)

Machine Learning Fall 2017 Structured Prediction (structured perceptron, HMM, structured SVM) Professor Liang Huang (Chap. 17 of CIML) Structured Prediction x x the man bit the dog the man bit the dog x x DT NN

672 views • 27 slides
Predicting Sequences: Structured Perceptron CS 6355: Structured Prediction 1 Conditional Random

Predicting Sequences: Structured Perceptron CS 6355: Structured Prediction 1 Conditional Random

Predicting Sequences: Structured Perceptron CS 6355: Structured Prediction 1 Conditional Random Fields summary An undirected graphical model Decompose the score over the structure into a collection of factors Each factor assigns a

865 views • 47 slides
lti Is convex Perceptron Boosting Max-Margin Conditional Likelihood MIRA Based on Uses a

lti Is convex Perceptron Boosting Max-Margin Conditional Likelihood MIRA Based on Uses a

Softmax-Margin CRFs: Training Log-Linear Models with Cost Functions Kevin Gimpel and Noah A. Smith lti Is convex Perceptron Boosting Max-Margin Conditional Likelihood MIRA Based on Uses a cost probabilistic function inference

603 views • 26 slides
Structured Perceptron CMSC 470 Marine Carpuat POS tagging Sequence labeling with the perceptron

Structured Perceptron CMSC 470 Marine Carpuat POS tagging Sequence labeling with the perceptron

Sequence Labeling with the Structured Perceptron CMSC 470 Marine Carpuat POS tagging Sequence labeling with the perceptron Sequence labeling problem Structured Perceptron Input: Perceptron algorithm can be used for sequence labeling

654 views • 13 slides
Structured Perceptron with Inexact Search x x the man bit the dog x the man bit

Structured Perceptron with Inexact Search x x the man bit the dog x the man bit

Structured Perceptron with Inexact Search x x the man bit the dog x the man bit the dog x DT NN VBD DT NN y y y=+ 1 y=- 1 Liang Huang Suphan Fayong Yang Guo Information Sciences Institute

1.15k views • 100 slides
Learning as Search Optimization: Approximate Large Margin Methods for Structured Prediction Hal

Learning as Search Optimization: Approximate Large Margin Methods for Structured Prediction Hal

Hal Daum III (hdaume@isi.edu) Learning as Search Optimization: Approximate Large Margin Methods for Structured Prediction Hal Daum III and Daniel Marcu Information Sciences Institute University of Southern California {hdaume,marcu}@isi.edu

400 views • 20 slides
NLP Programming Tutorial 11 - The Structured Perceptron Graham Neubig Nara Institute of Science

NLP Programming Tutorial 11 - The Structured Perceptron Graham Neubig Nara Institute of Science

NLP Programming Tutorial 11 The Structured Perceptron NLP Programming Tutorial 11 - The Structured Perceptron Graham Neubig Nara Institute of Science and Technology (NAIST) 1 NLP Programming Tutorial 11 The Structured Perceptron

447 views • 31 slides
Sequential Data Modeling - The Structured Perceptron Graham Neubig Nara Institute of Science and

Sequential Data Modeling - The Structured Perceptron Graham Neubig Nara Institute of Science and

Sequential Data Modeling The Structured Perceptron Sequential Data Modeling - The Structured Perceptron Graham Neubig Nara Institute of Science and Technology (NAIST) 1 Sequential Data Modeling The Structured Perceptron Prediction

604 views • 46 slides
Large Margin Classification Using the Perceptron Algorithm (Part 2) Henry Tan Georgetown

Large Margin Classification Using the Perceptron Algorithm (Part 2) Henry Tan Georgetown

Large Margin Classification Using the Perceptron Algorithm (Part 2) Henry Tan Georgetown University April 13, 2015 Georgetown University Perceptron 1 Analysis - Theorem 3 Theorem 3 Assume all examples ( x , y ) are generated i.i.d. Let

494 views • 31 slides
TAG, Dynamic Programming, and the Perceptron for Efficent, Feature-Rich Parsing Xavier Carreras,

TAG, Dynamic Programming, and the Perceptron for Efficent, Feature-Rich Parsing Xavier Carreras,

TAG, Dynamic Programming, and the Perceptron for Efficent, Feature-Rich Parsing Xavier Carreras, Michael Collins and Terry Koo MIT CSAIL Discriminative Models for Parsing Structured Prediction methods like CRF or Perceptron train linear models

403 views • 27 slides
Scalable Large-Margin x x the man bit the dog the man bit the dog x x

Scalable Large-Margin x x the man bit the dog the man bit the dog x x

What is Structured Prediction? Scalable Large-Margin x x the man bit the dog the man bit the dog x x Structured Learning: DT NN VBD DT NN S y Theory and Algorithms NP VP y=+ 1 y=- 1 the man bit the

701 views • 22 slides
CS 472 - Perceptron 1 Basic Neuron CS 472 - Perceptron 2 Expanded Neuron CS 472 - Perceptron

CS 472 - Perceptron 1 Basic Neuron CS 472 - Perceptron 2 Expanded Neuron CS 472 - Perceptron

CS 472 - Perceptron 1 Basic Neuron CS 472 - Perceptron 2 Expanded Neuron CS 472 - Perceptron 3 Perceptron Learning Algorithm l First neural network learning model in the 1960s l Simple and limited (single layer models) l Basic concepts

571 views • 53 slides
Part-of-Speech T agging: HMM &amp; structured perceptron CMSC 723 / LING 723 / INST 725 M ARINE

Part-of-Speech T agging: HMM &amp; structured perceptron CMSC 723 / LING 723 / INST 725 M ARINE

Part-of-Speech T agging: HMM &amp; structured perceptron CMSC 723 / LING 723 / INST 725 M ARINE C ARPUAT marine@cs.umd.edu Last time What are parts of speech (POS)? Equivalence classes or categories of words Open class vs.

1.06k views • 61 slides
Joint Event Trigger Identification and Event Coreference Resolution with Structured Perceptron

Joint Event Trigger Identification and Event Coreference Resolution with Structured Perceptron

Joint Event Trigger Identification and Event Coreference Resolution with Structured Perceptron Jun Araki and Teruko Mitamura Language Technologies Institute School of Computer Science Carnegie Mellon University September 21, 2015 EMNLP 2015

204 views • 17 slides
Loss-augmented Structured Prediction CMSC 723 / LING 723 / INST 725 Marine Carpuat Figures,

Loss-augmented Structured Prediction CMSC 723 / LING 723 / INST 725 Marine Carpuat Figures,

Loss-augmented Structured Prediction CMSC 723 / LING 723 / INST 725 Marine Carpuat Figures, algorithms &amp; equations from CIML chap 17 POS tagging Sequence labeling with the perceptron Sequence labeling problem Structured Perceptron

577 views • 40 slides
Structured-Cut: A Max-Margin Feature Selection Framework for Video Segmentation Nikhil S. Naikal

Structured-Cut: A Max-Margin Feature Selection Framework for Video Segmentation Nikhil S. Naikal

Structured-Cut: A Max-Margin Feature Selection Framework for Video Segmentation Nikhil S. Naikal Berkeley EECS Abstract precisely tracking the contour of the object in the 2D image pro- jections. A given video sequence can easily exhibit

56 views • 5 slides
ECON 950 Winter 2020 Prof. James MacKinnon 12. Support Vector Machines These notes are based

ECON 950 Winter 2020 Prof. James MacKinnon 12. Support Vector Machines These notes are based

ECON 950 Winter 2020 Prof. James MacKinnon 12. Support Vector Machines These notes are based on Chapter 9 of ISLR. Support vector machines are a popular method for classification problems where there are two classes. There are extensions

499 views • 16 slides
f able : Estimation of marginal effects with transformed covariates Taking Margins a step further

f able : Estimation of marginal effects with transformed covariates Taking Margins a step further

f able : Estimation of marginal effects with transformed covariates Taking Margins a step further Rios-Avila, Fernando 1 1 friosavi@levy.org Levy Economics Institute UK Stata Conference, September 2020 Rios-Avila (Levy) f able Stata 2020 1 /

884 views • 23 slides
Support Vector Machines INFO-4604, Applied Machine Learning University of Colorado Boulder

Support Vector Machines INFO-4604, Applied Machine Learning University of Colorado Boulder

Support Vector Machines INFO-4604, Applied Machine Learning University of Colorado Boulder September 27, 2018 Prof. Michael Paul Today Two important concepts: Margins Kernels Large Margin Classification Linear Predictions

709 views • 27 slides
CSC 411 Lecture 9: SVMs and Boosting Roger Grosse, Amir-massoud Farahmand, and Juan Carrasquilla

CSC 411 Lecture 9: SVMs and Boosting Roger Grosse, Amir-massoud Farahmand, and Juan Carrasquilla

CSC 411 Lecture 9: SVMs and Boosting Roger Grosse, Amir-massoud Farahmand, and Juan Carrasquilla University of Toronto UofT CSC 411: 09-Classification Odds and Ends 1 / 34 Overview Support Vector Machines Connection between Exponential Loss

387 views • 34 slides
CS145: INTRODUCTION TO DATA MINING 5: Vector Data: Support Vector Machine Instructor: Yizhou Sun

CS145: INTRODUCTION TO DATA MINING 5: Vector Data: Support Vector Machine Instructor: Yizhou Sun

CS145: INTRODUCTION TO DATA MINING 5: Vector Data: Support Vector Machine Instructor: Yizhou Sun yzsun@cs.ucla.edu October 18, 2017 Announcements Homework 1 Due end of the day of this Thursday (11:59pm) Reminder of late submission

2k views • 44 slides
Luke 10:38-42 Martha Mary &amp; Margin OVERCOMING OVERLOAD Mary FOCUSED RELAXED UNCONCERNED

Luke 10:38-42 Martha Mary &amp; Margin OVERCOMING OVERLOAD Mary FOCUSED RELAXED UNCONCERNED

Luke 10:38-42 Martha Mary &amp; Margin OVERCOMING OVERLOAD Mary FOCUSED RELAXED UNCONCERNED PRESENT Martha DISTRACTED ANXIOUS TROUBLED DISTANT Margin THE AMOUNT AVAILABLE BEYOND WHATS NECESSARY Margin SABBATH TITHING

660 views • 17 slides
THIRD QUARTER 2017 REVIEW November 1, 2017 w w w . w e s t e r n g a s . c o m | N Y S E : W E

THIRD QUARTER 2017 REVIEW November 1, 2017 w w w . w e s t e r n g a s . c o m | N Y S E : W E

W E S T E R N G A S I N V E S T O R R E L A T I O N S JON VANDENBRAND Director, Investor Relations (832) 636-1007 THIRD QUARTER 2017 REVIEW November 1, 2017 w w w . w e s t e r n g a s . c o m | N Y S E : W E S , W G P

332 views • 10 slides
ECO 199 GAMES OF STRATEGY Spring Term 2004 April 15 COLLECTIVE ACTION GAMES TWO GENERAL

ECO 199 GAMES OF STRATEGY Spring Term 2004 April 15 COLLECTIVE ACTION GAMES TWO GENERAL

ECO 199 GAMES OF STRATEGY Spring Term 2004 April 15 COLLECTIVE ACTION GAMES TWO GENERAL CONCEPTS 1. SPILLOVERS (EXTERNALITIES) One persons action affects anothers payoff Simplest case: Two players A and B, each has two choices:

322 views • 4 slides

More recommend