Neural Networks + Backpropagation Last Class Softmax Classifier - PowerPoint PPT Presentation

CS4501: Introduction to Computer Vision Neural Networks + Backpropagation

Last Class • Softmax Classifier • Generalization / Overfitting • Pytorch

Today’s Class • Global Features • The perceptron model • Neural Networks – multilayer perceptron model (MLP) • Backpropagation

Supervised Machine Learning Steps Training Training Labels Training Images Image Learned Training Features model Learned model Testing Image Prediction Features Test Image Slide credit: D. Hoiem

Supervised Learning –Softmax Classifier ; < " = [2 , 2 0 2 1 ] Extract features ! " = [! "% ! "' ! "( ! ") ] Run features through classifier + , = - ,% ! "% + - ,' ! "' + - ,( ! "( + - ,) ! ") + / , + 0 = - 0% ! "% + - 0' ! "' + - 0( ! "( + - 0) ! ") + / 0 + 1 = - 1% ! "% + - 1' ! "' + - 1( ! "( + - 1) ! ") + / 1 Get predictions , = 3 4 5 /(3 4 5 +3 4 8 + 3 4 9 ) 2 0 = 3 4 8 /(3 4 5 +3 4 8 + 3 4 9 ) 2 1 = 3 4 9 /(3 4 5 +3 4 8 + 3 4 9 ) 2 5

Last Class: (mini-batch) Stochastic Gradient Descent (SGD) 6 = 0.01 !(#, %) = ( −log . /,01230 (#, %) Initialize w and b randomly /∈5 for e = 0, num_epochs do B is a small for b = 0, num_batches do set of training Compute: and :!(#, %)/:% :!(#, %)/:# examples. Update w: # = # − 6 :!(#, %)/:# Update b: % = % − 6 :!(#, %)/:% // Useful to see if this is becoming smaller or not. Print: !(#, %) end end 6

Generalization Generalization refers to the ability to correctly classify never before • seen examples Can be controlled by turning “knobs” that affect the complexity of • the model Test set (labels unknown) Training set (labels known)

Overfitting % is a polynomial of % is linear % is cubic degree 9 !"## $ is high !"## $ is low !"## $ is zero! Overfitting Underfitting High Bias High Variance

Pytorch: Project Assignment 4 • http://vicenteordonez.com/vision/

Image Features In your Project 4: Nearest Neighbors + Softmax Classifier features are: • Feature: 3072-dim vector Image: 3x32x32

Image Features: Color Photo by: marielito slide by Tamara L. Berg

Image Features: Color

Image Features: Color However, these are all images of people but the colors in each image are very different.

Image Features: HoG Paper by Navneet Dalal & Bill Triggs presented at CVPR 2005 for detecting people. Scikit-image implementation

Image Features: HoG + Block Normalization Paper by Navneet Dalal & Bill Triggs presented at CVPR 2005 for detecting people. Figure from Zhuolin Jiang, Zhe Lin, Larry S. Davis, ICCV 2009 for human action recognition.

Image Features: GIST The “gist” of a scene: Oliva & Torralba, 2001

Image Features: GIST Oriented edge response at multiple scales (5 spatial scales, 6 Hays and Efros, edge orientations) SIGGRAPH 2007

Image Features: GIST Aggregated edge responses over 4x4 windows Hays and Efros, SIGGRAPH 2007

Image Features: Bag of (Visual) Words Representation slide by Fei-fei Li

slide by Fei-fei Li

Summary: Image Features • Largely replaced by Neural networks • Still useful to study for inspiration in designing neural networks that compute features. • Many other features proposed • LBP: Local Binary Patterns: Useful for recognizing faces. • Dense SIFT: SIFT features computed on a grid similar to the HOG features. • etc.

Perceptron Model Frank Rosenblatt (1957) - Cornell University Activation function " ; + ; + < - " < ! " = $1, if * + , " , + 1 > 0 * + = ,./ 0, otherwise " = + > " > More: https://en.wikipedia.org/wiki/Perceptron

Perceptron Model Frank Rosenblatt (1957) - Cornell University !? " ; + ; + < - " < ! " = $1, if * + , " , + 1 > 0 * + = ,./ 0, otherwise " = + > " > More: https://en.wikipedia.org/wiki/Perceptron

Perceptron Model Frank Rosenblatt (1957) - Cornell University Activation function " ; + ; + < - " < ! " = $1, if * + , " , + 1 > 0 * + = ,./ 0, otherwise " = + > " > More: https://en.wikipedia.org/wiki/Perceptron

Activation Functions Sigmoid(x) Step(x) Tanh(x) ReLU(x) = max(0, x)

Pytorch - Perceptron

Two-layer Multi-layer Perceptron (MLP) ”hidden" layer & Loss / Criterion ! " ' " & ! # ' # ( ) " ( " & ! $ ' $ & ! % ' % &

Forward pass / ! + = 567896:(* + ) * + = & - "+. ' + + 3 " +01 / & = " = & - #+ ! + + 3 # +01 ( " = 567896:(= + ) ! " ' " & ! # ' # ( ) " ( " & ! $ ' $ & ! % ' % >9?? = >(( " , ( ) " ) &

Backward pass GradInputs *+ = * /012304(, - ) *+ *+ = ( * A + E " ) *+ & ? "-@ ' - *, - *, - *! 7 *' 7 *' 7 *, - -BC *+ = ( * + E # ) *+ A & ? #- ! - *! 7 *! 7 *< " & -BC 89 8 89 = 8: ; /012304(< - ) ) ; 8: ; 8= ! " ' " & ! # ' # ( ) " ( " & ! $ ' $ & ! % *+ = *! 7 *+ ' % 89 8 ) ; = ) ; +(( " , ( ) " ) *? #- *? #- *! 7 8= 8= & *+ = *' 7 *+ *? "-@ *? "-@ *' 7 GradParams

Pytorch – Two-layer MLP + Regression

Pytorch – Two-layer MLP + LogSoftmax # of Hidden Units

Pytorch – Two-layer MLP + LogSoftmax LogSoftmax + Negative Likelihood Loss

Bookmark Opportunity!

Questions? 34