SLIDE 1
- Softmax Classifier
- Generalization / Overfitting
- Pytorch
Neural Networks + Backpropagation Last Class Softmax Classifier - - PowerPoint PPT Presentation
Neural Networks + Backpropagation Last Class Softmax Classifier - - PowerPoint PPT Presentation
CS4501: Introduction to Computer Vision Neural Networks + Backpropagation Last Class Softmax Classifier Generalization / Overfitting Pytorch Todays Class Global Features The perceptron model Neural Networks
SLIDE 2
SLIDE 3
- Global Features
- The perceptron model
- Neural Networks – multilayer perceptron model (MLP)
- Backpropagation
Today’s Class
SLIDE 4
Prediction
Supervised Machine Learning Steps
Training Labels Training Images Training
Training
Image Features Image Features
Testing
Test Image Learned model Learned model
Slide credit: D. Hoiem
SLIDE 5
5
Supervised Learning –Softmax Classifier
!" = [!"% !"' !"( !")]
Extract features
+, = -,%!"% + -,'!"' + -,(!"( + -,)!") + /, +0 = -0%!"% + -0'!"' + -0(!"( + -0)!") + /0 +1 = -1%!"% + -1'!"' + -1(!"( + -1)!") + /1 2
, = 345/(345+348 + 349)
2
0 = 348/(345+348 + 349)
2
1 = 349/(345+348 + 349)
Run features through classifier
; <" = [2
, 2 0 2 1]
Get predictions
SLIDE 6
6
Last Class: (mini-batch) Stochastic Gradient Descent (SGD)
!(#, %) = ( −log .
/,01230(#, %) /∈5
6 = 0.01 for e = 0, num_epochs do end Initialize w and b randomly :!(#, %)/:# :!(#, %)/:% Compute: and Update w: Update b: # = # − 6 :!(#, %)/:# % = % − 6 :!(#, %)/:% Print: !(#, %) // Useful to see if this is becoming smaller or not. end for b = 0, num_batches do B is a small set of training examples.
SLIDE 7
Generalization
- Generalization refers to the ability to correctly classify never before
seen examples
- Can be controlled by turning “knobs” that affect the complexity of
the model
Training set (labels known) Test set (labels unknown)
SLIDE 8
Overfitting
!"## $ is high !"## $ is low !"## $ is zero! Overfitting Underfitting High Bias High Variance
% is linear % is cubic % is a polynomial of degree 9
SLIDE 9
Pytorch: Project Assignment 4
- http://vicenteordonez.com/vision/
SLIDE 10
Image Features
- In your Project 4: Nearest Neighbors + Softmax Classifier features are:
Image: 3x32x32 Feature: 3072-dim vector
SLIDE 11
Image Features: Color
Photo by: marielito
slide by Tamara L. Berg
SLIDE 12
Image Features: Color
SLIDE 13
Image Features: Color
However, these are all images of people but the colors in each image are very different.
SLIDE 14
Image Features: HoG
Scikit-image implementation
Paper by Navneet Dalal & Bill Triggs presented at CVPR 2005 for detecting people.
SLIDE 15
Image Features: HoG
Paper by Navneet Dalal & Bill Triggs presented at CVPR 2005 for detecting people. Figure from Zhuolin Jiang, Zhe Lin, Larry S. Davis, ICCV 2009 for human action recognition.
+ Block Normalization
SLIDE 16
Image Features: GIST
The “gist” of a scene: Oliva & Torralba, 2001
SLIDE 17
Image Features: GIST
Oriented edge response at multiple scales (5 spatial scales, 6 edge orientations)
Hays and Efros, SIGGRAPH 2007
SLIDE 18
Image Features: GIST
Aggregated edge responses over 4x4 windows
Hays and Efros, SIGGRAPH 2007
SLIDE 19
Image Features: Bag of (Visual) Words Representation
slide by Fei-fei Li
SLIDE 20
slide by Fei-fei Li
SLIDE 21
Summary: Image Features
- Many other features proposed
- LBP: Local Binary Patterns: Useful for recognizing faces.
- Dense SIFT: SIFT features computed on a grid similar to the HOG features.
- etc.
- Largely replaced by Neural networks
- Still useful to study for inspiration in designing neural networks that
compute features.
SLIDE 22
Perceptron Model
Frank Rosenblatt (1957) - Cornell University More: https://en.wikipedia.org/wiki/Perceptron
! " = $1, if * +,",
- ,./
+ 1 > 0 0,
- therwise
"; "< "= ">
*
+; +< += +> Activation function
SLIDE 23
Perceptron Model
Frank Rosenblatt (1957) - Cornell University More: https://en.wikipedia.org/wiki/Perceptron
! " = $1, if * +,",
- ,./
+ 1 > 0 0,
- therwise
"; "< "= ">
*
+; +< += +>
!?
SLIDE 24
Perceptron Model
Frank Rosenblatt (1957) - Cornell University More: https://en.wikipedia.org/wiki/Perceptron
! " = $1, if * +,",
- ,./
+ 1 > 0 0,
- therwise
"; "< "= ">
*
+; +< += +> Activation function
SLIDE 25
Activation Functions
ReLU(x) = max(0, x) Tanh(x) Sigmoid(x) Step(x)
SLIDE 26
Pytorch - Perceptron
SLIDE 27
Two-layer Multi-layer Perceptron (MLP)
!" !# !$ !%
&
'" '# '$ '%
& & & &
( )" ”hidden" layer (" Loss / Criterion
SLIDE 28
Forward pass
!" !# !$ !%
&
'" '# '$ '%
& & & &
( )" ("
*+ = &
- "+.'+
/ +01
+ 3" !+ = 567896:(*+) =" = &
- #+!+
/ +01
+ 3# (" = 567896:(=+) >9?? = >((", ( )")
SLIDE 29
Backward pass
!" !# !$ !%
&
'" '# '$ '%
& & & &
( )" ("
*+ *,- = * *,- /012304(,-) *+ *!7
89 8:;
=
8 8:; /012304(<-) 89 8= ); 89 8= ); = 8 8= ); +((", (
)") *+ *'7 = ( * *'7 & ?"-@'-
A
- BC
+ E") *+ *,- *+ *?"-@ = *'7 *?"-@ *+ *'7 *+ *!7 = ( * *!7 & ?#-!-
A
- BC
+ E#) *+ *<" *+ *?#- = *!7 *?#- *+ *!7
GradInputs GradParams
SLIDE 30
Pytorch – Two-layer MLP + Regression
SLIDE 31
Pytorch – Two-layer MLP + LogSoftmax
# of Hidden Units
SLIDE 32
Pytorch – Two-layer MLP + LogSoftmax
LogSoftmax + Negative Likelihood Loss
SLIDE 33
Bookmark Opportunity!
SLIDE 34
Questions?
34