an integrated framework for margin based sequential
play

An Integrated Framework for Margin-based Sequential Discriminative - PowerPoint PPT Presentation

Dec 20, 2023 •292 likes •621 views

Overview Background Unification of Margin-modified MPE and MMI An Integrated Framework for Margin-based Sequential Discriminative Training over Lattices using differenced Maximum Mutual Information (dMMI) Erik McDermott - Google Inc.

  1. Overview Background Unification of Margin-modified MPE and MMI An Integrated Framework for Margin-based Sequential Discriminative Training over Lattices using differenced Maximum Mutual Information (dMMI) Erik McDermott - Google Inc. September 14, 2012 1 / 31

  2. Overview Background Unification of Margin-modified MPE and MMI Overview ◮ Error-weighted training using explicit models of error (MPE/MWE/sMBR etc.) ◮ Shifting of loss function: “margin” (MCE, MPE, bMMI) ◮ Make shift proportional to error. ◮ bMMI (Povey et al. 2008): implicit error model , just use error-proportional shift. ◮ Extension of “point” use of margin to integral over margin interval → proposal of “differenced MMI” (dMMI) ◮ dMMI: margin & error-dependent loss smoothing/integration ◮ Unifies margin-modified MMI and MPE ◮ More general than MPE yet allows a simpler implementation using difference of standard Forward-Backward statistics ◮ Bayesian view & further generalization. 2 / 31

  3. Overview Large-scale discriminative training Background Minimum Phone Error Unification of Margin-modified MPE and MMI Margin-based variants; bMMI Integrated system optimization 3 / 31

  4. Overview Large-scale discriminative training Background Minimum Phone Error Unification of Margin-modified MPE and MMI Margin-based variants; bMMI Non-uniform error for discriminative training 4 / 31

  5. Overview Large-scale discriminative training Background Minimum Phone Error Unification of Margin-modified MPE and MMI Margin-based variants; bMMI Minimum Phone Error (Povey 2002); Decision boundaries 5 / 31

  6. Overview Large-scale discriminative training Background Minimum Phone Error Unification of Margin-modified MPE and MMI Margin-based variants; bMMI MPE as multi-dimensional sigmoid 6 / 31

  7. Overview Large-scale discriminative training Background Minimum Phone Error Unification of Margin-modified MPE and MMI Margin-based variants; bMMI MPE derivative - String picture 7 / 31

  8. Overview Large-scale discriminative training Background Minimum Phone Error Unification of Margin-modified MPE and MMI Margin-based variants; bMMI Modified Forward-Backward for MPE over lattices 8 / 31

  9. Overview Large-scale discriminative training Background Minimum Phone Error Unification of Margin-modified MPE and MMI Margin-based variants; bMMI MPE derivative - Arc picture 9 / 31

  10. Overview Large-scale discriminative training Background Minimum Phone Error Unification of Margin-modified MPE and MMI Margin-based variants; bMMI New approaches based on margin ◮ Intuition: improve generalization by making the training problem “harder”. ◮ “Large-margin MCE” (Yu et al., 2007) ◮ Extension of McDermott & Katagiri (2004)’s Parzen window analysis of MCE → iteratively increase MCE sigmoid bias term ◮ Applicable to implicit error models : ◮ “Large-margin HMMs” (Sha & Saul, 2007): Insertion of fine-grained error (e.g. Edit Distance) into the margin term ◮ “Boosted MMI” (Povey et al., Saon & Povey, 2008) ◮ Heigold’s unified theory (2008): bring margin to standard MMI/MPE/MCE approaches 10 / 31

  11. Overview Large-scale discriminative training Background Minimum Phone Error Unification of Margin-modified MPE and MMI Margin-based variants; bMMI Linking ASR and Machine Learning 11 / 31

  12. Overview Large-scale discriminative training Background Minimum Phone Error Unification of Margin-modified MPE and MMI Margin-based variants; bMMI Modifying MPE/MMI with margin term “Boost” likelihoods (Povey & Saon (2008), Heigold (2008)): 12 / 31

  13. Overview Large-scale discriminative training Background Minimum Phone Error Unification of Margin-modified MPE and MMI Margin-based variants; bMMI Margin-modified MPE 13 / 31

  14. Overview Large-scale discriminative training Background Minimum Phone Error Unification of Margin-modified MPE and MMI Margin-based variants; bMMI Effect of margin on MPE loss 14 / 31

  15. Overview Large-scale discriminative training Background Minimum Phone Error Unification of Margin-modified MPE and MMI Margin-based variants; bMMI Margin-modified MMI (Povey & Saon, 2008) 15 / 31

  16. Overview Large-scale discriminative training Background Minimum Phone Error Unification of Margin-modified MPE and MMI Margin-based variants; bMMI Effect of margin on MMI loss 16 / 31

  17. Overview Large-scale discriminative training Background Minimum Phone Error Unification of Margin-modified MPE and MMI Margin-based variants; bMMI ◮ 2300h Arabic Broadcast News (GALE) ◮ 2000h English conversational telephone speech (CTS) 17 / 31

  18. Overview Large-scale discriminative training Background Minimum Phone Error Unification of Margin-modified MPE and MMI Margin-based variants; bMMI Margin-modified MPE & MMI summary “Boost” likelihoods (Povey & Saon (2008), Heigold (2008)): 18 / 31

  19. Overview Integrated framework Background Bayesian view & generalization Unification of Margin-modified MPE and MMI dMMI: the “integrated” framework Margin-space integration of MPE loss via differencing of MMI functionals for generalized error-weighted discriminative training McDermott & Nakamura, Interspeech 2009 ◮ Mathematical link between margin-modified MPE and MMI; ◮ Proposal of “dMMI” 19 / 31

  20. Overview Integrated framework Background Bayesian view & generalization Unification of Margin-modified MPE and MMI MPE is the derivative of modified MMI! 20 / 31

  21. Overview Integrated framework Background Bayesian view & generalization Unification of Margin-modified MPE and MMI dMMI definition Using previous result & Fundamental Theorem of Calculus: 21 / 31

  22. Overview Integrated framework Background Bayesian view & generalization Unification of Margin-modified MPE and MMI dMMI in practice Just use “reverse-boosted” denominator lattice as numerator lattice: 22 / 31

  23. Overview Integrated framework Background Bayesian view & generalization Unification of Margin-modified MPE and MMI Approximating MPE As margin interval is reduced, dMMI converges to MPE Property must hold for any correct implementations of bMMI and MPE! 23 / 31

  24. Overview Integrated framework Background Bayesian view & generalization Unification of Margin-modified MPE and MMI Integrated view of discriminative training 24 / 31

  25. Overview Integrated framework Background Bayesian view & generalization Unification of Margin-modified MPE and MMI Leveraging approximated, shifted hinge functions 25 / 31

  26. Overview Integrated framework Background Bayesian view & generalization Unification of Margin-modified MPE and MMI Gradient-based optimization using dMMI 26 / 31

  27. Overview Integrated framework Background Bayesian view & generalization Unification of Margin-modified MPE and MMI dMMI as integral over margin prior 27 / 31

  28. Overview Integrated framework Background Bayesian view & generalization Unification of Margin-modified MPE and MMI dMMI as building block for modeling general margin priors 28 / 31

  29. Overview Integrated framework Background Bayesian view & generalization Unification of Margin-modified MPE and MMI Numerical approximation of arbitrary margin priors ◮ E.g. prior p ( σ ) = c exp ( − c | σ | ) used for Minimum Relative Entropy Discrimination, Jebara (2004) ◮ Here: use prior in context of standard HMM-based discriminative training ◮ Approximate prior using sum of step functions (cf Lebesgue integration) 29 / 31

  30. Overview Integrated framework Background Bayesian view & generalization Unification of Margin-modified MPE and MMI Buidling margin prior using dMMI 30 / 31

  31. Overview Integrated framework Background Bayesian view & generalization Unification of Margin-modified MPE and MMI Summary ◮ MPE explicitly models non-uniform error, e.g. phone or word error including insertions, deletions & substitutions ◮ Margin-based “Boosted MMI” (bMMI): ◮ super-cheap approach for incorporating non-uniform error into loss function; ◮ however objective is still (modified) Mutual Information, not explicit model of error. ◮ “Differenced MMI” (dMMI) is similarly cheap alternative that ◮ is explicitly linked to error; ◮ generalizes MPE; ◮ possibly offers better performance (Delcroix et al. ICASSP 2012; Kubo et al. Interspeech 2012); ◮ can be further generalized to define arbitrary margin priors for lattice-based discriminative training. 31 / 31

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Maximum Margin based Semi-supervised Spectral Kernel Learning Zenglin Xu, Jianke Zhu, Michael R.

Maximum Margin based Semi-supervised Spectral Kernel Learning Zenglin Xu, Jianke Zhu, Michael R.

Motivation A Framework of Spectral Kernel Learning Experiment and Discussion Conclusion and Future work Question and Answer Maximum Margin based Semi-supervised Spectral Kernel Learning Zenglin Xu, Jianke Zhu, Michael R. Lyu and Irwin King

528 views • 29 slides
lti Is convex Perceptron Boosting Max-Margin Conditional Likelihood MIRA Based on Uses a

lti Is convex Perceptron Boosting Max-Margin Conditional Likelihood MIRA Based on Uses a

Softmax-Margin CRFs: Training Log-Linear Models with Cost Functions Kevin Gimpel and Noah A. Smith lti Is convex Perceptron Boosting Max-Margin Conditional Likelihood MIRA Based on Uses a cost probabilistic function inference

603 views • 26 slides
Structured-Cut: A Max-Margin Feature Selection Framework for Video Segmentation Nikhil S. Naikal

Structured-Cut: A Max-Margin Feature Selection Framework for Video Segmentation Nikhil S. Naikal

Structured-Cut: A Max-Margin Feature Selection Framework for Video Segmentation Nikhil S. Naikal Berkeley EECS Abstract precisely tracking the contour of the object in the 2D image pro- jections. A given video sequence can easily exhibit

56 views • 5 slides
About this class Maximizing the Margin Maximum margin classifiers Picture of large and small

About this class Maximizing the Margin Maximum margin classifiers Picture of large and small

About this class Maximizing the Margin Maximum margin classifiers Picture of large and small margin hyperplanes SVMs: geometric derivation of the primal prob- lem Intuition: large margin condition acts as a reg- ularizer and should generalize

219 views • 9 slides
Getting it Right for Every Child Integrated Assessment Framework East Ayrshire Council NHS

Getting it Right for Every Child Integrated Assessment Framework East Ayrshire Council NHS

Getting it Right for Every Child Integrated Assessment Framework East Ayrshire Council NHS Ayrshire and Arran Background For Scotlands Children Better Integrated Childrens Services (2001) Its Everyones Job to Make Sure Im

410 views • 11 slides
Knowledge is Power: Utilizing the Theory of Margin to Design Effective First Generation Student

Knowledge is Power: Utilizing the Theory of Margin to Design Effective First Generation Student

Knowledge is Power: Utilizing the Theory of Margin to Design Effective First Generation Student Supports By: Dr. Dory Quinn, MSW, EdD Theoretical Framework McCluskys Theory of Margin Adults are driven by both the need for

310 views • 14 slides
Tsybakov noise adap/ve margin-based ac/ve learning Aar$ Singh

Tsybakov noise adap/ve margin-based ac/ve learning Aar$ Singh

Tsybakov noise adap/ve margin-based ac/ve learning Aar$ Singh A. Nico Habermann Associate Professor NIPS workshop on Learning Faster from Easy Data

391 views • 23 slides
An Integrated Framework for Transcription, Modal and q Motivic Analysis of Ma am k

An Integrated Framework for Transcription, Modal and q Motivic Analysis of Ma am k

An Integrated Framework for Transcription, Modal and q Motivic Analysis of Ma am k Improvisation Olivier Lartillot Finnish Centre of Excellence in Interdisciplinary Music Research Mondher Ayari University of Strasbourg, France Cre

326 views • 10 slides
Sequential Files : Outline ! Overview ! Ordered vs. Unordered ! Physical sequential Files !

Sequential Files : Outline ! Overview ! Ordered vs. Unordered ! Physical sequential Files !

Sequential Files : Outline ! Overview ! Ordered vs. Unordered ! Physical sequential Files ! Physical Linked Sequential Files ! COBOL rasitjutrakul Sequential Files " Logically the records are stored consecutively Physical sequential file

335 views • 22 slides
Random Sampling Florian Schoppmann August 24, 2010 Non-Sequential Sequential Sequential with

Random Sampling Florian Schoppmann August 24, 2010 Non-Sequential Sequential Sequential with

Random Sampling Florian Schoppmann August 24, 2010 Non-Sequential Sequential Sequential with Reservoir Sequential With Reservoir and Replacement . . . . . . . . . . . . Sampling Algorithms Input: List [ 1 . . . N ] Length of

555 views • 13 slides
Sequential Circuits Combinational circuits : current input output Sequential circuit :

Sequential Circuits Combinational circuits : current input output Sequential circuit :

14:332:231 DIGITAL LOGIC DESIGN Ivan Marsic, Rutgers University Electrical & Computer Engineering Fall 2013 Lecture #15: Sequential Circuits: Latches Sequential Circuits Combinational circuits : current input output Sequential

483 views • 15 slides
Integrated Approach of Risk-based Compliance-based Performance-based Audit and Surveillance

Integrated Approach of Risk-based Compliance-based Performance-based Audit and Surveillance

Civil Aviation Department The Government of the Hong Kong Special Administrative Region Integrated Approach of Risk-based Compliance-based Performance-based Audit and Surveillance Programme George WAI, Airworthiness Officer Flight Standards

394 views • 26 slides
Internal Control Integrated Framework January 2013 0 Table of Contents Project

Internal Control Integrated Framework January 2013 0 Table of Contents Project

Internal Control Integrated Framework January 2013 0 Table of Contents Project Overview Updates to Internal Control-Integrated Framework Overview of Internal Control over External Financial Reporting: Compendium of

270 views • 23 slides
PAC-learning, VC Dimension and Margin-based Bounds Machine Learning 10701/15781 Carlos

PAC-learning, VC Dimension and Margin-based Bounds Machine Learning 10701/15781 Carlos

More details: General: http://www.learning-with-kernels.org/ Example of more complex bounds: http://www.research.ibm.com/people/t/tzhang/papers/jmlr02_cover.ps.gz PAC-learning, VC Dimension and Margin-based Bounds Machine Learning

919 views • 59 slides
An Integrated Framework for Fog Communications and Computing in Internet of Vehicles Alessio

An Integrated Framework for Fog Communications and Computing in Internet of Vehicles Alessio

University of Florence Department of Information Engineering An Integrated Framework for Fog Communications and Computing in Internet of Vehicles Alessio Bonadio, Francesco Chiti, Romano Fantacci name.surname@unifi.it Outline 1 Introduction

209 views • 18 slides
Max-Margin Markov Networks Ben Taskar Carlos Guestrin Daphne Koller Main Contribution The

Max-Margin Markov Networks Ben Taskar Carlos Guestrin Daphne Koller Main Contribution The

Max-Margin Markov Networks Ben Taskar Carlos Guestrin Daphne Koller Main Contribution The authors combine a graphic model and a discriminative model and apply it in a sequential learning setting. Graphic models: better at

222 views • 8 slides
To: Interested Parties From: GBAO Date: November 16, 2020 Poll Analysis: Michigan Educators On

To: Interested Parties From: GBAO Date: November 16, 2020 Poll Analysis: Michigan Educators On

To: Interested Parties From: GBAO Date: November 16, 2020 Poll Analysis: Michigan Educators On School Reopening A new poll finds that Michigan educators 1 are concerned about the safety of reopening for full in-person learning and a strong

160 views • 13 slides
Lecture 24/Chapter 20 Estimating Proportions with Confidence Example: Importance of Margin of

Lecture 24/Chapter 20 Estimating Proportions with Confidence Example: Importance of Margin of

Lecture 24/Chapter 20 Estimating Proportions with Confidence Example: Importance of Margin of Error From Probability to Confidence Constructing a Confidence Interval Examples Example: What Can We Infer About Population? Background :

723 views • 24 slides
COMPSTAT 2010, Paris Two way classification of a table with non negative entries:

COMPSTAT 2010, Paris Two way classification of a table with non negative entries:

COMPSTAT 2010, Paris Two way classification of a table with non negative entries: Validation of an approach based on Correspondence Analysis and Clustering Antonio Ciampi*, Alina Dyachenko* and Yves Lechevallier** * Department of

606 views • 34 slides
Non-homogeneous random walks on a semi-infinite strip Nicholas Georgiou Joint work with Andrew

Non-homogeneous random walks on a semi-infinite strip Nicholas Georgiou Joint work with Andrew

Non-homogeneous random walks on a semi-infinite strip Nicholas Georgiou Joint work with Andrew Wade Aspects of Random Walks 1st April 2014 Outline Background Lampertis problem Non-homogeneous random walks on strips Model assumptions

741 views • 46 slides
Large Margin Classification Using the Perceptron Algorithm (Part 2) Henry Tan Georgetown

Large Margin Classification Using the Perceptron Algorithm (Part 2) Henry Tan Georgetown

Large Margin Classification Using the Perceptron Algorithm (Part 2) Henry Tan Georgetown University April 13, 2015 Georgetown University Perceptron 1 Analysis - Theorem 3 Theorem 3 Assume all examples ( x , y ) are generated i.i.d. Let

494 views • 31 slides
Statistical and Computational Statistical and Computational Learning Theory Learning Theory

Statistical and Computational Statistical and Computational Learning Theory Learning Theory

Statistical and Computational Statistical and Computational Learning Theory Learning Theory Fundamental Question: Predict Error Rates Fundamental Question: Predict Error Rates Given: Given: The space H of hypotheses The space H of

497 views • 29 slides
CSCE 478/878 Lecture 2: Supervised Learning Supervised Learning Stephen Scott Introduction

CSCE 478/878 Lecture 2: Supervised Learning Supervised Learning Stephen Scott Introduction

CSCE 478/878 Lecture 2: CSCE 478/878 Lecture 2: Supervised Learning Supervised Learning Stephen Scott Introduction Outline Learning a Stephen Scott Class from Examples Noise and Other (Adapted from Ethem Alpaydin) Problems

470 views • 21 slides
Confidence Intervals II 18.05 Spring 2014 Jeremy Orloff and Jonathan Bloom Agenda Polling:

Confidence Intervals II 18.05 Spring 2014 Jeremy Orloff and Jonathan Bloom Agenda Polling:

Confidence Intervals II 18.05 Spring 2014 Jeremy Orloff and Jonathan Bloom Agenda Polling: estimating p in Bernoulli( p ). CLT large sample confidence intervals for the mean. Three views of confidence intervals. Constructing a confidence

562 views • 18 slides

More recommend