SLIDE 1
Learning with Marginalized Corrupted Features
- L. van der Maaten, M. Chen, S. Tyree, K. Weinberger
ICML 2013
Jan Gasthaus Tea talk April 11, 2013
1 / 17
2 / 17
Learning with Marginalized Corrupted Features L. van der Maaten, M. - - PowerPoint PPT Presentation
Learning with Marginalized Corrupted Features L. van der Maaten, M. - - PowerPoint PPT Presentation
Learning with Marginalized Corrupted Features L. van der Maaten, M. Chen, S. Tyree, K. Weinberger ICML 2013 Jan Gasthaus Tea talk April 11, 2013 1 / 17 2 / 17 2 / 17 2 / 17 2 / 17 2 / 17 2 / 17 2 / 17 2 / 17 Data Augmentation Secret
SLIDE 2
SLIDE 3
2 / 17
SLIDE 4
2 / 17
SLIDE 5
2 / 17
SLIDE 6
2 / 17
SLIDE 7
2 / 17
SLIDE 8
2 / 17
SLIDE 9
2 / 17
SLIDE 10
Data Augmentation
Secret 4: lots of jittering, mirroring, and color perturbation of the
- riginal images generated on the fly to increase the size of the
training set Yann LeCun on Google+ about Alex Krizhevsky’s ImageNet results
3 / 17
SLIDE 11
Main Idea
Old idea: create artificial additional training data by corrupting it with “noise”
4 / 17
SLIDE 12
Main Idea
Old idea: create artificial additional training data by corrupting it with “noise” One easy way to incorporate domain knowledge (e.g. possible transformations)
4 / 17
SLIDE 13
Main Idea
Old idea: create artificial additional training data by corrupting it with “noise” One easy way to incorporate domain knowledge (e.g. possible transformations) But: additional training data = ⇒ additional computation Idea: Corrupt with known ExpFam noise and integrate it out
4 / 17
SLIDE 14
Explicit vs. Implicit Corruption
Explicit corruption: Take training set D = {(xn, yn)}N
n=1 and
corrupt it M times L(˜ D, Θ) =
N
- n=1
1 M
M
- m=1
L(˜ xnm, yn, Θ) with xnm ∼ p(˜ xnm|xn).
5 / 17
SLIDE 15
Explicit vs. Implicit Corruption
Implicit corruption: Minimize the expected value of the loss under p(˜ xn|xn): L(D, Θ) =
N
- n=1
E [L(˜ xn, yn, Θ)]p(˜
xn|xn)
i.e. replace the empirical average with the expectation.
6 / 17
SLIDE 16
Wait a second . . .
This is so obvious that it must have been done before . . .
7 / 17
SLIDE 17
Wait a second . . .
This is so obvious that it must have been done before . . .
◮ Vicinal Risk Minimization, Chapelle, Weston, Bottou, &
Vapnik, NIPS 2000
7 / 17
SLIDE 18
Wait a second . . .
This is so obvious that it must have been done before . . .
◮ Vicinal Risk Minimization, Chapelle, Weston, Bottou, &
Vapnik, NIPS 2000
Explicitly only consider the case of Gaussian noise distributions
7 / 17
SLIDE 19
Quadratic Loss
8 / 17
SLIDE 20
Quadratic Loss
9 / 17
SLIDE 21
Exponential Loss
10 / 17
SLIDE 22
Logistic Loss
11 / 17
SLIDE 23
MGFs
12 / 17
SLIDE 24
Results
13 / 17
SLIDE 25
Results
14 / 17
SLIDE 26
Results
15 / 17
SLIDE 27
Results
16 / 17
SLIDE 28
Results
17 / 17