a kernel theory of modern data augmentation
play

A Kernel Theory of Modern Data Augmentation Tr Tri Dao ao , Albert - PowerPoint PPT Presentation

Nov 15, 2022 •223 likes •485 views

th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:309:00 P 9:00 PM A Kernel Theory of Modern Data Augmentation Tr Tri Dao ao , Albert

  1. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM A Kernel Theory of Modern Data Augmentation Tr Tri Dao ao , Albert Gu, Alex Ratner, Virginia Smith, Chris De Sa, Chris Ré

  2. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Data augmentation is important to accuracy…

  3. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Data augmentation is important to accuracy… 3.7 pt. average gain across top ten CIFAR-10 models

  4. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Data augmentation is important to accuracy… 3.7 pt. average gain across top ten CIFAR-10 models 13.9 pt. average gain for CIFAR-100

  5. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Data augmentation is important to accuracy… 3.7 pt. average gain across top ten CIFAR-10 models 13.9 pt. average gain for CIFAR-100 A form of weak supervision: expresses domain knowledge (invariance)

  6. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM … but is not well understood

  7. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM … but is not well understood How does data augmentation affect the model? • Learning process • Parameters and decision surface

  8. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Augmentation as sequence modeling • TANDA [Ratner et al., 2017] • AutoAugment [Cubuk et al., 2018]

  9. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Augmentation as sequence modeling • TANDA [Ratner et al., 2017] • AutoAugment [Cubuk et al., 2018] Model augmentation as a Markov chain

  10. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Augmentation as kernels Base classifier: k-nearest neighbors + Data augmentation = Asymptotic kernel classifier

  11. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Effects of data augmentation on kernel classifiers

  12. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Effects of data augmentation on kernel classifiers o o o o o x x o x x o x x Invariance

  13. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Effects of data augmentation on kernel classifiers o o o o o o o o o o x x x x o o x x x o o x x x x x Regularization Invariance

  14. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Effects of data augmentation on kernel classifiers o o o o o o o o o o x x x x o o x x x o o x x x x x Regularization Invariance Practical utility

  15. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Effects of data augmentation on kernel classifiers o o o o o o o o o o x x x x o o x x x o o x x x x x Regularization Invariance Practical utility speeding up training

  16. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Effects of data augmentation on kernel classifiers o o o o o o o o o o x x x x o o x x x o o x x x x x Regularization Invariance Practical utility as a speeding up diagnostic training

  17. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Model of data augmentation: kernel classifier n 1 X ` ( w > � ( x i )) min Non-augmented: w n Loss function i =1 Feature map

  18. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Model of data augmentation: kernel classifier n 1 X ` ( w > � ( x i )) min Non-augmented: w n Loss function i =1 Feature map n 1 X E z i ⇠ T ( x i ) ` ( w > � ( z i )) min Augmented: w n i =1 Transformed versions of data point

  19. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Data augmentation effects n n 1 E z i ⇠ T ( x i ) ` ( w > � ( z i )) ≈ 1 X X ` ( w > E z i ⇠ T ( x i ) � ( z i )) n n i =1 i =1 Average of augmented features (i.e. kernel mean embedding)

  20. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Data augmentation effects n n 1 E z i ⇠ T ( x i ) ` ( w > � ( z i )) ≈ 1 X X ` ( w > E z i ⇠ T ( x i ) � ( z i )) n n i =1 i =1 Average of augmented features (i.e. kernel mean embedding) 1 st order effect: induces invariance by feature averaging

  21. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Data augmentation effects n n 1 E z i ⇠ T ( x i ) ` ( w > � ( z i )) ≈ 1 X X ` ( w > E z i ⇠ T ( x i ) � ( z i )) n n i =1 i =1 Average of augmented features (i.e. kernel mean embedding) 1 st order effect: 2 nd order effect: reduces induces invariance model complexity by feature via a data-dependent averaging regularization

  22. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM A diagnostic: kernel alignment metric ψ ( x ) = E z ∼ T ( x ) φ ( z ) Averaged features: Kernel target alignment [Cristianini et al., 2002]: how well separated are features from different classes

  23. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM A diagnostic: kernel alignment metric Kernel alignment Kernel alignment MNIST

  24. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM A diagnostic: kernel alignment metric Kernel alignment Kernel alignment MNIST Kernel alignment correlates with accuracy.

  25. th 6:30 11 th ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 6:30—9:00 P 9:00 PM Summary • Data augmentation + k-NN = asymptotic kernel classifier. • Data augmentation induces invariance and regularizes. • Application in speeding up training and diagnostics. Tri Dao trid@stanford.edu Poster #227 on Tuesday Jun 11 th at 6:30pm

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Data Augmentation in NLP 2020-03-21 Xiachong Feng Outline Why we need Data Augmentation?

Data Augmentation in NLP 2020-03-21 Xiachong Feng Outline Why we need Data Augmentation?

Data Augmentation in NLP 2020-03-21 Xiachong Feng Outline Why we need Data Augmentation? Data Augmentation in CV Widely Used Methods EDA Back-Translation Contextual Augmentation Methods based on Pre-trained Language

483 views • 46 slides
Population Based Augmentation Efficient Learning of Augmentation Policy Schedules Daniel Ho , Eric

Population Based Augmentation Efficient Learning of Augmentation Policy Schedules Daniel Ho , Eric

Population Based Augmentation Efficient Learning of Augmentation Policy Schedules Daniel Ho , Eric Liang, Ion Stoica, Pieter Abbeel, Xi Chen Efficiently learn data augmentation policies to improve neural network performance. Data Augmentation

335 views • 8 slides
SwitchOut: An Efficient Data Augmentation for Neural Machine Translation Xinyi Wang , Hieu

SwitchOut: An Efficient Data Augmentation for Neural Machine Translation Xinyi Wang , Hieu

SwitchOut: An Efficient Data Augmentation for Neural Machine Translation Xinyi Wang , Hieu Pham , Zihang Dai, Graham Neubig November 2, 2018 :equal contribution 1 / 41 Data Augmentation Neural models are data hungry, while

782 views • 41 slides
Does Data Augmentation Lead to Positive Margin? Dimitris Po-Ling Loh Shashank Rajput* Zhili

Does Data Augmentation Lead to Positive Margin? Dimitris Po-Ling Loh Shashank Rajput* Zhili

Does Data Augmentation Lead to Positive Margin? Dimitris Po-Ling Loh Shashank Rajput* Zhili Feng* Zachary Charles Papailiopoulos * Equal Contribution Data Augmentation (DA) DA means increasing the training set artificially. Used to

379 views • 15 slides
Machine learning theory Kernel methods Hamid Beigy Sharif university of technology April 20,

Machine learning theory Kernel methods Hamid Beigy Sharif university of technology April 20,

Machine learning theory Kernel methods Hamid Beigy Sharif university of technology April 20, 2020 Table of contents 1. Motivation 2. Kernel methods 3. Basic kernel operations in feature space 4. Kernel-based algorithms 5. Summary 1/24

729 views • 31 slides
In Deep Learning Anima Anandkumar & Zachary Lipton DATA AUGMENTATION To improve

In Deep Learning Anima Anandkumar & Zachary Lipton DATA AUGMENTATION To improve

Addressing Data Scarcity In Deep Learning Anima Anandkumar & Zachary Lipton DATA AUGMENTATION To improve generalization, augment training data. In computer vision. Simple techniques: rotation, cropping, noise In speech

319 views • 27 slides
Learning From Data Lecture 25 The Kernel Trick Learning with only inner products The Kernel M.

Learning From Data Lecture 25 The Kernel Trick Learning with only inner products The Kernel M.

Learning From Data Lecture 25 The Kernel Trick Learning with only inner products The Kernel M. Magdon-Ismail CSCI 4100/6100 recap: Large Margin is Better Controling Overfitting Non-Separable Data 0.08 random hyperplane 2 w t w + C N E

576 views • 18 slides
Modern MDL meets Data Mining Insight, Theory, and Practice Jilles Kenji Vreeken Yamanishi

Modern MDL meets Data Mining Insight, Theory, and Practice Jilles Kenji Vreeken Yamanishi

2019 ACM SIGKDD International Conference on Knowledge Discovery and Data Mining Modern MDL meets Data Mining Insight, Theory, and Practice Jilles Kenji Vreeken Yamanishi CISPA Helmholtz Center The University of Tokyo for Information

636 views • 39 slides
1. Mid-Ocean Ridges 1. Mid-Ocean Ridges How Do We Know? We know about Mid-Ocean Ridges

1. Mid-Ocean Ridges 1. Mid-Ocean Ridges How Do We Know? We know about Mid-Ocean Ridges

Modern Theory The Modern Theory of Plate Tectonics is built on Alfred Wegeners theory of Continental Drift . Modern Theory of Two new discoveries (discoveries over the last 60 years) helped scientists Plate Tectonics develop the

461 views • 9 slides
image-augmentation April 9, 2019 1 Image Augmentation In [1]: % matplotlib inline import d2l

image-augmentation April 9, 2019 1 Image Augmentation In [1]: % matplotlib inline import d2l

image-augmentation April 9, 2019 1 Image Augmentation In [1]: % matplotlib inline import d2l import mxnet as mx from mxnet import autograd, gluon, image, init, nd from mxnet.gluon import data as gdata, loss as gloss, utils as gutils import sys

109 views • 8 slides
1 CPU Events: Interrupts and Exceptions CPU Events: Interrupts and Exceptions Protecting Entry

1 CPU Events: Interrupts and Exceptions CPU Events: Interrupts and Exceptions Protecting Entry

Processes and the Kernel Processes and the Kernel The kernel sets processes up process in private data data execution Processes, Protection and the Kernel: Processes, Protection and the Kernel: virtual contexts to address

540 views • 9 slides
Diverse Paraphrasing and its Effectiveness in Data Augmentation Ashutosh Kumar*, Satwik

Diverse Paraphrasing and its Effectiveness in Data Augmentation Ashutosh Kumar*, Satwik

Diverse Paraphrasing and its Effectiveness in Data Augmentation Ashutosh Kumar*, Satwik Bhattamishra*, Manik Bhandari, Partha Talukdar Machine and Language Learning Lab (MALL) Indian Institute of Science, Bangalore *Equal Contributions

905 views • 87 slides
Convolutional Neural Networks with Data Augmentation against Jitter-Based Countermeasures Eleonora

Convolutional Neural Networks with Data Augmentation against Jitter-Based Countermeasures Eleonora

Convolutional Neural Networks with Data Augmentation against Jitter-Based Countermeasures Convolutional Neural Networks with Data Augmentation against Jitter-Based Countermeasures Eleonora Cagli 1 , 3 ecile Dumas 1 C Emmanuel Prouff 2 , 3 1

1.28k views • 100 slides
Improving Molecular Design by Stochastic Iterative Target Augmentation Kevin Yang, Wengong Jin,

Improving Molecular Design by Stochastic Iterative Target Augmentation Kevin Yang, Wengong Jin,

Improving Molecular Design by Stochastic Iterative Target Augmentation Kevin Yang, Wengong Jin, Kyle Swanson, Regina Barzilay, Tommi Jaakkola 15-Second Overview Data augmentation approach: improve molecular optimization SOTA by > 10%

621 views • 50 slides
Towards Automated Melanoma Detection with Deep Learning: Data Purification and Augmentation

Towards Automated Melanoma Detection with Deep Learning: Data Purification and Augmentation

Motivation Related work Proposed approach Empirical results Conclusion Towards Automated Melanoma Detection with Deep Learning: Data Purification and Augmentation Devansh Bisla , Anna Choromanska, Russell S. Berman, Jennifer A. Stein, David

359 views • 15 slides
Learning From Data Lecture 26 Kernel Machines Popular Kernels The Kernel Measures Similarity

Learning From Data Lecture 26 Kernel Machines Popular Kernels The Kernel Measures Similarity

Learning From Data Lecture 26 Kernel Machines Popular Kernels The Kernel Measures Similarity Kernels in Different Applications M. Magdon-Ismail CSCI 4100/6100 recap: The Kernel Allows Us to Bypass Z -space Solve the QP 1

356 views • 12 slides
Decision support systems and machine learning Lecture 11 Lecture 11 p. 1/24 Neural networks:

Decision support systems and machine learning Lecture 11 Lecture 11 p. 1/24 Neural networks:

Decision support systems and machine learning Lecture 11 Lecture 11 p. 1/24 Neural networks: Biological and artificial Consider humans: Properties of artificial neural nets (ANNs): Neuron switching time 0 . 001 second Many

404 views • 36 slides
CSC 411: Lecture 07: Multiclass Classification Class based on Raquel Urtasun & Rich Zemels

CSC 411: Lecture 07: Multiclass Classification Class based on Raquel Urtasun & Rich Zemels

CSC 411: Lecture 07: Multiclass Classification Class based on Raquel Urtasun & Rich Zemels lectures Sanja Fidler University of Toronto Feb 1, 2016 Urtasun, Zemel, Fidler (UofT) CSC 411: 07-Multiclass Classification Feb 1, 2016 1 / 17

206 views • 17 slides
Non-Bayesian Classifiers Part II: Linear Discriminants and Support Vector Machines Selim Aksoy

Non-Bayesian Classifiers Part II: Linear Discriminants and Support Vector Machines Selim Aksoy

Non-Bayesian Classifiers Part II: Linear Discriminants and Support Vector Machines Selim Aksoy Department of Computer Engineering Bilkent University saksoy@cs.bilkent.edu.tr CS 551, Spring 2019 CS 551, Spring 2019 2019, Selim Aksoy

279 views • 25 slides
Logistic Regression (slides borrowed from Tom Mitchell, Barnabs Pczos & Aarti Singh 1

Logistic Regression (slides borrowed from Tom Mitchell, Barnabs Pczos & Aarti Singh 1

CSCI 4520 - Introduction to Machine Learning Spring 2020 Mehdi Allahyari Georgia Southern University Logistic Regression (slides borrowed from Tom Mitchell, Barnabs Pczos & Aarti Singh 1 Linear Regression & Linear Classification

831 views • 36 slides
Outline Last time: window-based generic object detection Discriminative classifiers

Outline Last time: window-based generic object detection Discriminative classifiers

4/14/2011 Outline Last time: window-based generic object detection Discriminative classifiers basic pipeline for image recognition face detection with boosting as case study Wednesday, April 13 Today: discriminative

365 views • 11 slides
Perceptrons 2-29-16 What is a neural network? activation connection functions A NN is a

Perceptrons 2-29-16 What is a neural network? activation connection functions A NN is a

Perceptrons 2-29-16 What is a neural network? activation connection functions A NN is a directed acyclic graph. weights Nodes are organized into layers. -0.5 Consecutive layers are fully connected. 1.5 0.2 Edges have

147 views • 13 slides
Clean Slate Program: Second Chance Remedies and Preparing Client Declarations June 2020 TAPs

Clean Slate Program: Second Chance Remedies and Preparing Client Declarations June 2020 TAPs

Clean Slate Program: Second Chance Remedies and Preparing Client Declarations June 2020 TAPs Innovative Program Design Large scale volunteer network of pro bono 2 attorneys 3 2.a Increased cloud technology client volunteer management

958 views • 69 slides
Declaration Syntax General form of a declaration: declaration-specifiers declarators ;

Declaration Syntax General form of a declaration: declaration-specifiers declarators ;

Declaration Syntax General form of a declaration: declaration-specifiers declarators ; Declarations Declaration specifiers describe the properties of the variables or functions being declared. Declarators give their names and may

546 views • 7 slides

More recommend