Convolutional Neural Networks Basics Praveen Krishnan Overview - - PowerPoint PPT Presentation
Convolutional Neural Networks Basics Praveen Krishnan Overview Paradigm Shift Simple Network Convolutional Network Layers Case Study1: Alex Net Training Generalization Visualizations Transfer Learning Case
Paradigm Shift Simple Network Convolutional Network
Layers Case Study1: Alex Net
Training Generalization Visualizations Transfer Learning
Case Study2: JAZ Net
Practical Aspects
Gradient checks. Data GPU Coding/Libraries
Feature Extraction (SIFT, HoG,.) Classifier Feature Learning (CNN, RBM, …) Classifier L1 Sparrow Layers - (Hierarchical decomposition) L2 L4 L3 Coding Sparrow Part Models Pooling
f1 f2 fn-1 fn x1 xn-2 xn-1 x0 xn w1 w2 wn-1 wn Here each output xj depends on previous input xj-1 through a function fj with parameters wj
x00 xn1 W1 x01 x0d xn2 xnc Zooming In Wn
x00 xn1 x01 x0d xn2 xnc
y = [0,0,…,1,…0] z W1 Wn
xn1 xn2 xnc
z Weight updates using back propagation of gradients W1 Wn
Fully connected layer Locally connected layer
200x200x3 3x3x3
Convolutional layer
3x3x3
3 200 # feature maps
3 3 3
Receptive field Convolutional layer
200x200x3 Image size: W1xH1xD1 Receptive field size: FxF #Feature maps: K
200x200x3 Image size: W1xH1xD1 Receptive field size: FxF #Feature maps: K W2=(W1-F)/S+1 H2=(H1-F)/S+1 D2=K It is also better to do zero padding to preserve input size spatially.
Conv. Layer x1
n-1
x2
n-1
x3
n-1
y1
n
y2
n
Here “f” is a non-linear activation function. F= no. of input feature maps n= layer index “*” represents convolution/correlation?
Sigmoid tanh ReLU Leaky ReLU maxout
A typical deep convolutional network Other layers
Pooling Normalization Fully connected etc. CONV POOL NORM CONV POOL NORM FC SOFTMAX
Aggregation over space or feature type. Invariance to image transformation and increases compactness to
Pooling types: Max, Average, L2 etc.
2 8 9 4 3 6 5 7 3 1 6 4 2 5 7 3 8 9 5 7 Pool Size: 2x2 Stride: 2 Type: Max Max pooling
CONV POOL NORM CONV POOL NORM FC SOFTMAX
Local contrast normalization (Jarrett et.al ICCV‟09)
reduce illumination artifacts. performs local subtractive and divisive normalization.
Local response normalization (Krizhevesky et.al. NIPS‟12)
form of lateral inhibition across channels.
Batch normalization (More later)
CONV POOL NORM CONV POOL NORM FC SOFTMAX
Multi layer perceptron Role of an classifier** Generally used in final layers to classify the object
CONV POOL NORM CONV POOL NORM FC SOFTMAX
Winner of ImageNet LSVRC-2012. Trained over 1.2M images using SGD with regularization. Deep architecture (60M parameters.) Optimized GPU implementation (cuda-convnet)
Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." NIPS 2012. - Cited by 11915
Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." NIPS 2012.
CONV 11x11x96 LRN MAX POOL 2x2 CONV 5x5x256 LRN MAX POOL 2x2 CONV 3x3x384 MAX POOL 2x2 FC - 4096 FC - 4096
SOFTMAX - 1000
CONV 3x3x384 MAX POOL 2x2 CONV 3x3x256 MAX POOL 2x2
Learning: Minimizing the loss function (incl.
Mini batch stochastic gradient descent
Sample a batch of data. Forward propagation Backward propagation Parameter update Filter weights
CONV POOL NORM CONV POOL NORM FC LOSS
xn yn
Back propagation
Consider an layer f with parameters w:
CONV POOL NORM CONV POOL NORM FC LOSS
xn yn Recursive eq. which is applicable to each layer
Parameter update
Stochastic gradient descent
Stochastic gradient descent with momentum
CONV POOL NORM CONV POOL NORM FC LOSS
xn yn More in coming slides…
Loss functions.
one vs. rest classification
Soft-max classifier (cross entropy loss)
CONV POOL NORM CONV POOL NORM FC LOSS
xn yn Proof?
Loss functions.
one vs. rest classification
Hinge Loss
CONV POOL NORM CONV POOL NORM FC LOSS
xn yn
Loss functions.
Regression
Euclidean loss / Squared loss
CONV POOL NORM CONV POOL NORM FC LOSS
xn yn
Visualization of loss function
Momentum Step size/learning rate Step direction Initialization Loss θ Typically viewed as highly non-convex function but more recently it‟s believed to have smoother surfaces but with many saddle regions !
Momentum
Better convergence rates. Physical perspective: Affects velocity of the update. Higher velocity in the consistent direction of gradient. Momentum update: Position Velocity
Loss θ
Learning Rates (η)
Controls the kinetic energy
Important to know when
Common methods
Step decay Exponential/log space decay Manual
Adaptive learning methods
Adagrad RMSprop
Figure courtesy: Fei Fei et al. , cs231n
Loss θ
Loss θ
Initialization
Never initialize weights to all zero‟s or same value. (Why?) Popular techniques:-
Random values sampled from N(0,1) Xavier (Glorot et.al JMLR‟10)
Scale of initialization is dependent on the number of input and output
neurons.
Initial weights are sampled from N(0,var(w))
Pre-training
Using RBMs. (Hinton et.al, Science 2006)
Fan-in Fan-out
Generalization
How to prevent?
Underfitting – Deeper n/w‟s Overfitting
Stopping at the right time. Weight penalties.
L1 L2 Max norm
Dropout
Model ensembles
E.g. Same model, different
epoch top5- error training accuracy val-1 accuracy (*) val-2 accuracy (overfitting)
Dropouts
Stochastic regularization. Idea applicable to many other
Dropping out hidden units randomly
While testing the all units are
Dropouts along with max norm
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov, Dropout: a simple way to prevent neural networks from overfitting. JMLR 2014 Before dropout After dropout
Without dropout With dropout
Sparsity Features learned with one hidden layers auto-encoder
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov, Dropout: a simple way to prevent neural networks from overfitting. JMLR 2014
Batch Normalization
Covariate Shift
defined as a change in the distribution of a function‟s domain Mini-batches (randomized) reduces the effect of covariate shift
Internal Covariate Shift
Current layer parameters change the distribution of the input to
Slows down training and careful initialization.
Image Credit: https://gab41.lab41.org/batch-normalization-what-the-hey-d480039a9e3b go water the plants got water in your pants kite bang eat face monkey
Batch Normalization
Fixes the distribution of layer input as training progresses. Faster convergence. Ioffe, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, arxiv'15
Source: Krizhevsky et.al. NIPS‟12 T
GoogLeNet Clarifai AlexNet
CNNs are cool but some of the
How do I interpret the learned filters? What is it that stimulates/excites a
How do I decide the architecture or
To answer we need to probe the
Deconvolutional Networks. [Zeiler
Synthesize images [Simonyan et.al
Zeiler and Fergus , Visualizing and Understanding Convolutional Networks,. ECCV 2014 Visualizing the first conv. layer is possible but how about the later layers. Source: Krizhevsky et.al. NIPS‟12
Deconvnets
Non-parametric approach. Projects the feature activation
Analyses a trained model and
Visualizes a single activation
Helps in understanding the
Zeiler and Fergus , Visualizing and Understanding Convolutional Networks,. ECCV 2014 Source: Zeiler e.t. al. ECCV‟14
Zeiler and Fergus , Visualizing and Understanding Convolutional Networks,. ECCV 2014 Grass ! Source: Zeiler e.t. al. ECCV‟14
Zeiler and Fergus , Visualizing and Understanding Convolutional Networks,. ECCV 2014
stimulates/excites a neuron?
Washing Machine Karen Simonyan, Andrea Vedaldi, Andrew Zisserman, Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. CoRR 2014
Class Model Visualization
Find an L2 normalized image which maximizes the CI class
Initialize with mean image. Back-propagate to update the input pixels, keeping the weights
Karen Simonyan, Andrea Vedaldi, Andrew Zisserman, Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. CoRR 2014 Some more results
Image-Specific Class Saliency
Understanding the spatial support of a class in a specific image.
Karen Simonyan, Andrea Vedaldi, Andrew Zisserman, Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. CoRR 2014
Spatial Support Object localization mask Grab Cut
A key observation that we noticed in visualization:-
Yosinski J, Clune J, BengioY, and Lipson H. How transferable are features in deep neural networks? NIPS ‟14
CONV POOL NORM CONV POOL NORM FC
xn
SOFTMAX CONV POOL NORM
Gabor/Color blobs Dog Face General Specific
A key observation that we noticed in visualization:- Further ques?
Can we quantify the layer generality/specificity? Where does the transition occur? Is the transition sudden or spread over layers? Yosinski J, Clune J, BengioY, and Lipson H. How transferable are features in deep neural networks? NIPS ‟14
CONV POOL NORM CONV POOL NORM FC
xn
SOFTMAX CONV POOL NORM
General Specific
Transfer performance
Task A and B Types of networks
Selffer (BnB/ BnB+) Transfer (AnB+)
Datasets
Random split Dissimilar split
Observations
Higher level neurons are
There exists co-adapted
Yosinski J, Clune J, BengioY, and Lipson H. How transferable are features in deep neural networks? NIPS ‟14 baseB BnB BnB+ AnB AnB+
Take away message
Initializing a network with transferred features almost always
CONV POOL NORM CONV POOL NORM FC
xn
SOFTMAX CONV POOL NORM
Notes If dataset is small retrain the softmax
CONV POOL NORM CONV POOL NORM FC
xn
SOFTMAX CONV POOL NORM
If dataset is reasonable retrain larger portion with fine tuning of initial layers
Benchmarks
Razavian et. al. CVPRW‟2014 Chatfield et. al. BMVC‟2014
Scene text recognition
Recognition of word image
Synthetic generation of training
State of art results in most of
Model 1: Encoding words model Synthetic data engine
Challenges: Huge no of target classes ~90K (Size of english
Incremental training. Uses 9M training data
Jaderberg, Max, et al. "Synthetic data and artificial neural networks for natural scene text recognition." arXivv 2014
Model 2: Sequence
Model 3: Bag of n-grams Jaderberg, Max, et al. "Synthetic data and artificial neural networks for natural scene text recognition." arXivv 2014
Data Preprocessing
(0,1) Normalization Whitening
Data augmentation
Perturbation to image to make it more resilient to variations
Cropping Flipping Jittering Degradation models specific to modalities. (e.g. text )
Applicable at both train/test time
Modern CNN libraries
Theano T
Caffe Matconvnet T
and many more…