SLIDE 1
General Neural Networks Compositions of linear maps and - - PowerPoint PPT Presentation
General Neural Networks Compositions of linear maps and - - PowerPoint PPT Presentation
Pooling and Invariance in Convolutional Neural Networks General Neural Networks Compositions of linear maps and component-wise non- linearities Neural Networks Common Non-Linearities Rectifier Linear Unit Sigmoid Hyperbolic tangent
SLIDE 2
SLIDE 3
Neural Networks
SLIDE 4
Common Non-Linearities
Rectifier Linear Unit Sigmoid Hyperbolic tangent
SLIDE 5
Biological Inspiration
Neurons diagram Rectified Linear Unit
SLIDE 6
Sparse Connections
Not required to be fully connected
SLIDE 7
Backpropagation
SLIDE 8
Representations Learnt
SLIDE 9
Representation Learning
Images far apart in Euclidean space Need to find representation such that members of same class are mapped to similar values
SLIDE 10
Convolutional Neural Network
Compositions of: Convolutions by Linear Filter Thresholding Non-Linearities Spatial Pooling
SLIDE 11
Convolution by Linear Filter
SLIDE 12
Convolution by Linear Filter
SLIDE 13
Example
SLIDE 14
Convolutional Neural Networks
1) Convolution by Linear Filter 2) Apply non-linearity 3) Pooling
SLIDE 15
Convolutional Neural Network
SLIDE 16
Convolutional Neural Networks
SLIDE 17
Imagenet Network
SLIDE 18
Successes
Computer vision Speech Chemistry
SLIDE 19
Object Classification
a
SLIDE 20
Segmentation
SLIDE 21
Object Detection
SLIDE 22
Speech
SLIDE 23
Physical Chemistry
Successfully predict atomization energy, polarizability, frontier orbital eigenvalues, ionization potential, electron affinity and excitation energies from molecular structure.
SLIDE 24
Visualization of First Layer
SLIDE 25
Standard Pooling Mechanisms
Ave pooling Max pooling
SLIDE 26
Example
SLIDE 27
Heterogenous Pooling
Some filters passed to Ave pooling Others filters passed to Max pooling
SLIDE 28
Pooling Continuum
Accordingly, LeCun et al. 2012 ran experiment with variety of “p” values.
SLIDE 29
Results along spectrum
Optimal for this SVHN dataset was p = 4.
SLIDE 30
L_p learnt pooling
Why not learn optimal p for each filter map?
SLIDE 31
Stochastic Pooling
SLIDE 32
Stochastic Pooling
Expectation at Test Time
SLIDE 33
Entropy Pooling
Extend to variable p In particular, Alternative:
SLIDE 34
Max-Out pooling
Pooling across filters Substantial Improvement in performance and allowed depth
SLIDE 35
Example
SLIDE 36
Compete Neurons
Neurons can suppress other responses
SLIDE 37
Visualizations of Filters
Early Layers
SLIDE 38
Visualizations
SLIDE 39
Visualizations
SLIDE 40
Invariance under Rigid Motion
Goodfellow et al. 2009 demonstrated the CNN are invariant under Indeed, depth of NN critical to establishing such invariance
SLIDE 41
Unstable under Deformation
Szegedy et al.
SLIDE 42
Lipshitz Bounds for Layers
Max and ReLU are contractive FC Layers: usual linear
- perator norm
Conv Layers: Parseval’s and DFT yield explicit formula
SLIDE 43
Solutions?
Regularize Lipshitz operator
SLIDE 44
Coding Symmetry
Convolutional Wavelet Networks
SLIDE 45
Architecture
Wavelet convolutions composed with modulus
- perator
SLIDE 46
Gabor Wavelets
Trigonometric function in Gaussian Envelope
SLIDE 47
Group Convolution
SLIDE 48
Group Invariant Scattering
Main Result: Conv. Wavelet Networks are translation invariant functions in L_2(R^2) Furthermore, CWN can be made invariant to action under any compact lie group.
SLIDE 49
Textures
Sifre and Mallat
SLIDE 50
Basis for images
Learns similar representation as Imagenet CNN
SLIDE 51
Learnt Invariances
SLIDE 52
Optimal network
1) Encoded symmetry 2) Regularize Lipshitz coefficients 3) Compete Neurons in final layers
SLIDE 53