generalization and simplification in machine learning
play

Generalization and Simplification in Machine Learning Shay Moran - PowerPoint PPT Presentation

Jan 05, 2024 •2.3k likes •3.11k views

Generalization and Simplification in Machine Learning Shay Moran School of Mathematics, IAS Princeton . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Two dual aspects of

  1. Generalization and Simplification in Machine Learning Shay Moran School of Mathematics, IAS Princeton . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  2. Two dual aspects of “learning” Two aspects: 1. Generalization: Infer new knowledge from existing knowledge. 2. Simplification: Provide simple(r) explanations for existing knowledge. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  3. Interrelations Simplification Generalization e.g. math: simplification generalization simpler proof more general theorem theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  4. Philosophical heuristics Simplification Generalization Simpler (consistent) explanations are better. [Occam’s razor – William of Ockham ≈ 1300]. simplification = ⇒ generalization If I can’t reduce it to a freshman level then I don’t really understand it. [Richard Feynman 1980’s]. when James Gleick (a science reporter) asked him to explain why spin-1/2 particles obey Fermi-Dirac statistics When presented with a complicated proof, Erd¨ os used to reply: “Now, let’s find the book’s proof. . . ” [Paul Erd¨ os] generalization = ⇒ simplification Can these relations be manifested as theorems in learning theory? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  5. This talk ”Simplification ≡ Generalization” in Learning Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  6. Plan Generalization Simplification/compression The “generalization – compression” equivalence Binary classification Multiclass categorization Vapnik’s general setting of learning Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  7. Generalization: General Setting of Learning [Vapnik ’95] . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . .. . . . . .

  8. Intuition Imagine a scientist that performs m experiments with outcomes z 1 , . . . , z m and wishes to predict the outcome of future experiments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  9. Classification example: intervals D – unknown distribution over R c – unknown interval: Given: Training set ( ) ( ) ∼ D m S = x 1 , c ( x 1 ) , . . . , x m , c ( x m ) Goal: Find h = h ( S ) ⊆ R that minimizes the disagreement with c [ ] E x ∼D 1 c ( x ) ̸ = h ( x ) in the Probably (w.p. 1 − δ ) Approximately Correct (up to ϵ ) sense . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  10. Regression example: mean estimation D – unknown distribution over [0 , 1] Given: Training set S = z 1 , . . . , z m ∼ D m 0 . 3 0 . 2 0 . 1 0 0 2 4 6 8 10 Goal: Find h = h ( S ) ∈ [0 , 1] that minimizes ( x − h ) 2 ] [ E x ∼ D in the Probably (w.p. 1 − δ ) Approximately Correct (up to ϵ ) sense . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  11. The General Setting of Learning: definition H hypothesis class D distribution over examples ℓ loss function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  12. The General Setting of Learning: definition H hypothesis class D distribution over examples ℓ loss function Output: h out known to learner: H unknown to learner: D z 1 , . . . , z m i.i.d examples from D Nature Learning algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  13. The General Setting of Learning: definition H hypothesis class D distribution over examples ℓ loss function Output: h out known to learner: H unknown to learner: D z 1 , . . . , z m i.i.d examples from D Nature Learning algorithm Goal: loss of h out ≤ loss of best h ∈ H in the PAC sense classification problems, regression problems, some clustering problems,. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  14. Examples Binary classification: ▶ Z = X × { 0 , 1 } ▶ H – class of X → { 0 , 1 } functions ▶ ℓ ( ) [ ] h , ( x , y ) = 1 h ( x ) ̸ = y . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  15. Examples Binary classification: ▶ Z = X × { 0 , 1 } ▶ H – class of X → { 0 , 1 } functions ▶ ℓ ( ) [ ] h , ( x , y ) = 1 h ( x ) ̸ = y Multiclass categorization: ▶ Z = X × Y ▶ H – class of X → Y functions ▶ ℓ ( ) [ ] h ( x , y ) = 1 h ( x ) ̸ = y . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  16. Examples Binary classification: ▶ Z = X × { 0 , 1 } ▶ H – class of X → { 0 , 1 } functions ▶ ℓ ( ) [ ] h , ( x , y ) = 1 h ( x ) ̸ = y Multiclass categorization: ▶ Z = X × Y ▶ H – class of X → Y functions ▶ ℓ ( ) [ ] h ( x , y ) = 1 h ( x ) ̸ = y Mean estimation: ▶ Z = [0 , 1] ▶ H = [0 , 1] ▶ ℓ ( h , z ) = ( h − z ) 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  17. Examples Binary classification: ▶ Z = X × { 0 , 1 } ▶ H – class of X → { 0 , 1 } functions ▶ ℓ ( ) [ ] h , ( x , y ) = 1 h ( x ) ̸ = y Multiclass categorization: ▶ Z = X × Y ▶ H – class of X → Y functions ▶ ℓ ( ) [ ] h ( x , y ) = 1 h ( x ) ̸ = y Mean estimation: ▶ Z = [0 , 1] ▶ H = [0 , 1] ▶ ℓ ( h , z ) = ( h − z ) 2 Linear regression: ▶ Z = R d × R ▶ H – class of affine R d → R functions ) 2 ▶ ℓ ( ) ( h , ( x , y ) = h ( x ) − y . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  18. Agnostic and realizable-case Learnability H – hypothesis class H is agnostic learnable: ∃ algorithm A , s.t. for every D , if m > n agn ( ϵ, δ ) S ∼D m [ L D ( A ( S )) ≥ min Pr h ∈H L D ( h ) + ϵ ] ≤ δ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  19. Agnostic and realizable-case Learnability H – hypothesis class H is agnostic learnable: ∃ algorithm A , s.t. for every D , if m > n agn ( ϵ, δ ) S ∼D m [ L D ( A ( S )) ≥ min Pr h ∈H L D ( h ) + ϵ ] ≤ δ H is realizable-case learnable: ∃ algorithm A s.t. for every realizable D , if m > n real ( ϵ, δ ) S ∼D m [ L D ( A ( S )) ≥ ϵ ] ≤ δ Pr ▶ D is realizable if there is h ∈ H with L D ( h ) = 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  20. Compression: Sample compression schemes [Littlestone,Warmuth ’86] . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . .. . . . . .

  21. Intuition Imagine a scientist that performs m experiments with outcomes z 1 , . . . z m and wishes to choose d ≪ m of them in a way that allows to explain all other experiments (choose d axioms) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  22. Example: polynomials P – unknown polynomial of degree ≤ d : Input: training set of m evaluations of P ( d ≪ m ) − 2 − 1 0 1 2 3 Compression: Keep d + 1 points − 2 − 1 0 1 2 3 Reconstruction: Lagrange Interpolation Evaluates to the correct value on the whole training set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  23. Compression algorithm: definition [Littlestone,Warmuth ’86] H hypothesis class ℓ loss function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  24. Compression algorithm: definition [Littlestone,Warmuth ’86] H hypothesis class ℓ loss function Compression scheme of size d : S = z 1 , z 2 , . . . , z m Output: h out Input sample z i 1 , . . . , z i d Compression Compressor Reconstructor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  25. Compression algorithm: definition [Littlestone,Warmuth ’86] H hypothesis class ℓ loss function Compression scheme of size d : S = z 1 , z 2 , . . . , z m Output: h out Input sample z i 1 , . . . , z i d Compression Compressor Reconstructor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  26. Compression algorithms examples Compression algorithm for interval approximation of size 2: “output the smallest interval containing the positive examples” input sample output hypothesis Compression algorithm for mean estimation of size 3: “output the average of 3 sample points with minimal empirical error” 0 . 3 0 . 2 0 . 1 0 0 2 4 6 8 10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Good Machine Learning = Generalization Goal of machine learning: build models that generalize

Good Machine Learning = Generalization Goal of machine learning: build models that generalize

Good Machine Learning = Generalization Goal of machine learning: build models that generalize well to predicting new data Overfitting: fitting the training data too well, so we lose generality of model o Example: linear regression vs.

120 views • 11 slides
Some Advice on Applying Machine Learning in Practice CS 760@UW-Madison Its generalization

Some Advice on Applying Machine Learning in Practice CS 760@UW-Madison Its generalization

Some Advice on Applying Machine Learning in Practice CS 760@UW-Madison Its generalization that counts the fundamental goal of machine learning is to generalize beyond the instances in the training set you should rigorously measure

306 views • 17 slides
Computational Learning Theory: The Theory of Generalization Machine Learning 1 Slides based on

Computational Learning Theory: The Theory of Generalization Machine Learning 1 Slides based on

Computational Learning Theory: The Theory of Generalization Machine Learning 1 Slides based on material from Dan Roth, Avrim Blum, Tom Mitchell and others Checkpoint: The bigger picture Supervised learning: instances, concepts, and

895 views • 42 slides
http://www.reframe-d2k.org/ Generalization and reuse of machine learning models over multiple

http://www.reframe-d2k.org/ Generalization and reuse of machine learning models over multiple

http://www.reframe-d2k.org/ Generalization and reuse of machine learning models over multiple contexts (LMCE 2014) Chowdhury Farhan Ahmed University of Strasbourg France ! Introduction and Motivation ! Related Work ! Description of the Bike

293 views • 11 slides
Regression and generalization CE-717: Machine Learning Sharif University of Technology M.

Regression and generalization CE-717: Machine Learning Sharif University of Technology M.

Regression and generalization CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2018 Topics } Beyond linear regression models } Evaluation & model selection } Regularization } Bias-Variance 2 Recall: Linear

1.08k views • 70 slides
Regression and generalization CE-717: Machine Learning Sharif University of Technology M.

Regression and generalization CE-717: Machine Learning Sharif University of Technology M.

Regression and generalization CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2019 Topics } Beyond linear regression models } Evaluation & model selection } Regularization } Bias-Variance 2 Recall: Linear

1.29k views • 72 slides
Neural Networks for Machine Learning Lecture 9a Overview of ways to improve generalization

Neural Networks for Machine Learning Lecture 9a Overview of ways to improve generalization

Neural Networks for Machine Learning Lecture 9a Overview of ways to improve generalization Geoffrey Hinton Nitish Srivastava, Kevin Swersky Tijmen Tieleman Abdel-rahman Mohamed Reminder: Overfitting The training data contains information

1.17k views • 39 slides
Deep learning: Challenges in learning and generalization Tomas Mikolov, Facebook AI What is

Deep learning: Challenges in learning and generalization Tomas Mikolov, Facebook AI What is

Deep learning: Challenges in learning and generalization Tomas Mikolov, Facebook AI What is generalization? Memorization to remember all training examples (Universal Approximator) 2 + 3 = 5, 3 + 2 = 5, ... Generalization to

281 views • 14 slides
Michael Spece Departments of Machine Learning and Statistics Carnegie Mellon University June 11,

Michael Spece Departments of Machine Learning and Statistics Carnegie Mellon University June 11,

Generalization Martingale Bounds Ongoing Work Generalization for Streaming Data Michael Spece Departments of Machine Learning and Statistics Carnegie Mellon University June 11, 2015 1 / 12 Generalization Martingale Bounds Ongoing Work

489 views • 12 slides
Computational Learning Theory: Probably Approximately Correct (PAC) Learning Machine Learning 1

Computational Learning Theory: Probably Approximately Correct (PAC) Learning Machine Learning 1

Computational Learning Theory: Probably Approximately Correct (PAC) Learning Machine Learning 1 Slides based on material from Dan Roth, Avrim Blum, Tom Mitchell and others Computational Learning Theory The Theory of Generalization

411 views • 28 slides
Supervised Learning: The Setup Machine Learning 1 Last lecture We saw What is learning?

Supervised Learning: The Setup Machine Learning 1 Last lecture We saw What is learning?

Supervised Learning: The Setup Machine Learning 1 Last lecture We saw What is learning? Learning as generalization The badges game 2 This lecture More badges Formalizing supervised learning Instance space and features

1.04k views • 80 slides
Machine Learning and Sparsity Klaus-Robert Mller !!et al.!! Todays Talk sensing, sparse

Machine Learning and Sparsity Klaus-Robert Mller !!et al.!! Todays Talk sensing, sparse

Machine Learning and Sparsity Klaus-Robert Mller !!et al.!! Todays Talk sensing, sparse models and generalization interpretabilty and sparse methods explaining for nonlinear methods Sparse Models & Generalization? Machine

458 views • 25 slides
Introduction to Machine Learning Model Validation and Selection Dr. Ilija Bogunovic Learning

Introduction to Machine Learning Model Validation and Selection Dr. Ilija Bogunovic Learning

Introduction to Machine Learning Model Validation and Selection Dr. Ilija Bogunovic Learning and Adaptive Systems (las.ethz.ch) Recap: Achieving generalization Fundamental assumption: Our data set is generated independently and identically

515 views • 28 slides
Computational Learning Theory: Agnostic Learning Machine Learning 1 Slides based on material

Computational Learning Theory: Agnostic Learning Machine Learning 1 Slides based on material

Computational Learning Theory: Agnostic Learning Machine Learning 1 Slides based on material from Dan Roth, Avrim Blum, Tom Mitchell and others This lecture: Computational Learning Theory The Theory of Generalization Probably

653 views • 44 slides
Generalization Error MACH IN E LEARN IN G W ITH TREE-BAS ED MODELS IN P YTH ON Elie Kawerk

Generalization Error MACH IN E LEARN IN G W ITH TREE-BAS ED MODELS IN P YTH ON Elie Kawerk

Generalization Error MACH IN E LEARN IN G W ITH TREE-BAS ED MODELS IN P YTH ON Elie Kawerk Data Scientist Supervised Learning - Under the Hood Supervised Learning: y = f ( x ) , f is unknown. MACHINE LEARNING WITH TREE-BASED MODELS IN PYTHON

601 views • 37 slides
Computational Learning Theory: Positive and negative learnability results Machine Learning 1

Computational Learning Theory: Positive and negative learnability results Machine Learning 1

Computational Learning Theory: Positive and negative learnability results Machine Learning 1 Slides based on material from Dan Roth, Avrim Blum, Tom Mitchell and others Computational Learning Theory The Theory of Generalization Probably

534 views • 40 slides
Learnability and models of decision making under uncertainty Pathikrit Basu Federico Echenique

Learnability and models of decision making under uncertainty Pathikrit Basu Federico Echenique

Learnability and models of decision making under uncertainty Pathikrit Basu Federico Echenique Caltech Virginia Tech DT Workshop April 6, 2018 Pathikrit Basu-Echenique Learnability To think is to forget a difference, to generalize, to

892 views • 41 slides
COL866: Foundations of Data Science Ragesh Jaiswal, IITD Ragesh Jaiswal, IITD COL866:

COL866: Foundations of Data Science Ragesh Jaiswal, IITD Ragesh Jaiswal, IITD COL866:

COL866: Foundations of Data Science Ragesh Jaiswal, IITD Ragesh Jaiswal, IITD COL866: Foundations of Data Science Machine Learning Ragesh Jaiswal, IITD COL866: Foundations of Data Science Machine Learning Generalization bounds One of the

4.63k views • 22 slides
Sigmoid curves and a case for close-to-linear nonlinear models Charles Y. Tan charles

Sigmoid curves and a case for close-to-linear nonlinear models Charles Y. Tan charles

Introduction Nonlinear Models Sigmoid Curves Assess the Approximation Numerical Case Study Conclusions Sigmoid curves and a case for close-to-linear nonlinear models Charles Y. Tan charles tan@merck.com Merck Research Laboratories / MSD

1.1k views • 71 slides
Econometric Evaluation of Social Programs Part I: Counterfactuals, Causality and Structural

Econometric Evaluation of Social Programs Part I: Counterfactuals, Causality and Structural

Econometric Evaluation of Social Programs Part I: Counterfactuals, Causality and Structural Econometric Models James J. Heckman and Edward J. Vytlacil Econ 312, Spring 2019 Heckman and Vytlacil Counterfactuals, Causality and Structural

623 views • 36 slides
Learning grammar(s) statistically Mark Johnson joint work with Sharon Goldwater and Tom Griffiths

Learning grammar(s) statistically Mark Johnson joint work with Sharon Goldwater and Tom Griffiths

Learning grammar(s) statistically Mark Johnson joint work with Sharon Goldwater and Tom Griffiths Cognitive and Linguistic Sciences and Computer Science Brown University Mayfest 2006 Outline Introduction Probabilistic context-free grammars

803 views • 54 slides
Statistical Machine Learning Lecture 07: Clustering and Evaluation Kristian Kersting TU

Statistical Machine Learning Lecture 07: Clustering and Evaluation Kristian Kersting TU

Statistical Machine Learning Lecture 07: Clustering and Evaluation Kristian Kersting TU Darmstadt Summer Term 2020 K. Kersting based on Slides from J. Peters Statistical Machine Learning Summer Term 2020 1 / 38 Todays Objectives Make

545 views • 38 slides
Decision Tree Learning Mitchell, Chapter 3 CptS 570 Machine Learning School of EECS Washington

Decision Tree Learning Mitchell, Chapter 3 CptS 570 Machine Learning School of EECS Washington

Decision Tree Learning Mitchell, Chapter 3 CptS 570 Machine Learning School of EECS Washington State University Outline Decision tree representation ID3 learning algorithm Entropy and information gain Overfitting

540 views • 30 slides
Lecture 3: Loss Functions and Optimization Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 3: Loss Functions and Optimization Fei-Fei Li & Justin Johnson & Serena Yeung

Lecture 3: Loss Functions and Optimization Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 1 April 11, 2017 Administrative Assignment 1 is released: http://cs231n.github.io/assignments2017/assignment1/ Due Thursday April 20 ,

888 views • 85 slides

More recommend