A Kernel Theory of Modern Data Augmentation Tr Tri Dao ao , Albert - - PowerPoint PPT Presentation

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

A Kernel Theory of Modern Data Augmentation

Tr Tri Dao ao, Albert Gu, Alex Ratner, Virginia Smith, Chris De Sa, Chris Ré

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

Data augmentation is important to accuracy…

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

Data augmentation is important to accuracy…

3.7 pt. average gain across top ten CIFAR-10 models

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

Data augmentation is important to accuracy…

3.7 pt. average gain across top ten CIFAR-10 models 13.9 pt. average gain for CIFAR-100

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

Data augmentation is important to accuracy…

3.7 pt. average gain across top ten CIFAR-10 models 13.9 pt. average gain for CIFAR-100 A form of weak supervision: expresses domain knowledge (invariance)

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

… but is not well understood

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

How does data augmentation affect the model?

• Learning process
• Parameters and decision surface

… but is not well understood

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

Augmentation as sequence modeling

• TANDA [Ratner et al., 2017]
• AutoAugment [Cubuk et al., 2018]

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

Augmentation as sequence modeling

Model augmentation as a Markov chain

• TANDA [Ratner et al., 2017]
• AutoAugment [Cubuk et al., 2018]

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

Augmentation as kernels

Base classifier: k-nearest neighbors + Data augmentation = Asymptotic kernel classifier

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

Effects of data augmentation on kernel classifiers

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

Effects of data augmentation on kernel classifiers

• x

x x x

• x

x

Invariance

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

Effects of data augmentation on kernel classifiers

• x

x x x

• x

x

• x

x x x

• x

x

Invariance Regularization

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

Effects of data augmentation on kernel classifiers

• x

x x x

• x

x

• x

x x x

• x

x

Invariance Regularization

Practical utility

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

Effects of data augmentation on kernel classifiers

• x

x x x

• x

x

• x

x x x

• x

x

Invariance Regularization speeding up training

Practical utility

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

Effects of data augmentation on kernel classifiers

• x

x x x

• x

x

• x

x x x

• x

x

Invariance Regularization speeding up training as a diagnostic

Practical utility

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

Non-augmented:

Model of data augmentation: kernel classifier

min

w

1 n

n

X

i=1

`(w>(xi))

Feature map Loss function

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

Non-augmented: Augmented:

Model of data augmentation: kernel classifier

min

w

1 n

n

X

i=1

`(w>(xi))

Feature map Loss function

min

w

1 n

n

X

i=1

Ezi⇠T (xi)`(w>(zi))

Transformed versions of data point

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

Data augmentation effects

Average of augmented features (i.e. kernel mean embedding)

1 n

n

X

i=1

Ezi⇠T (xi)`(w>(zi)) ≈ 1 n

n

X

i=1

`(w>Ezi⇠T (xi)(zi))

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

Data augmentation effects

Average of augmented features (i.e. kernel mean embedding)

1st order effect: induces invariance by feature averaging

1 n

n

X

i=1

Ezi⇠T (xi)`(w>(zi)) ≈ 1 n

n

X

i=1

`(w>Ezi⇠T (xi)(zi))

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

Data augmentation effects

Average of augmented features (i.e. kernel mean embedding)

1st order effect: induces invariance by feature averaging

1 n

n

X

i=1

Ezi⇠T (xi)`(w>(zi)) ≈ 1 n

n

X

i=1

`(w>Ezi⇠T (xi)(zi))

2nd order effect: reduces model complexity via a data-dependent regularization

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

ψ(x) = Ez∼T (x)φ(z)

A diagnostic: kernel alignment metric

Averaged features: Kernel target alignment [Cristianini et al., 2002]: how well separated are features from different classes

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

A diagnostic: kernel alignment metric

MNIST

Kernel alignment Kernel alignment

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

A diagnostic: kernel alignment metric

MNIST

Kernel alignment correlates with accuracy.

Kernel alignment Kernel alignment

ICML Oral 06/11/2019 Post ster #227 | Tue Ju Jun 11 11th

th 6:30

6:30—9:00 P 9:00 PM

• Data augmentation + k-NN = asymptotic kernel classifier.
• Data augmentation induces invariance and regularizes.
• Application in speeding up training and diagnostics.

Summary Poster #227 on Tuesday Jun 11th at 6:30pm

Tri Dao trid@stanford.edu