Neural Network Training: Old & New Tricks Old: (80s) Stochastic - PowerPoint PPT Presentation

Residual Network Naïve solution • If extra layers are an identity mapping, then training errors can not increase 67

Residual Modelling: Basic idea in image processing • Goal: estimate update between an original image and a changed image Preserving base information Some residual Network can treat perturbation 68

Residual Network • Plain block • Difficult to make identity mapping because of multiple non-linear layers 69

Residual Network • Residual block • If identity were optimal, easy to set weights as 0 • If optimal mapping is closer to identity, easier to find small fluctuations Appropriate for treating perturbation as keeping a base information 70

Residual Network: Deeper is better • Deeper ResNets have lower training error 71

Residual Network: Deeper is better 72

CNNs, 2017: DenseNet Densely Connected Convolutional Networks, CVPR 2017 Gao Huang, Zhuang Liu, Laurens van der Maaten, Kilian Q. Weinberger Recently proposed, better performance/parameter ratio 73

Image-to-Image 74

Graphics: Multiresolution 75

Image-to-image • So far we mapped an image image to a number or label • In graphics, output often is “richer”: • An image • A volume • A 3D mesh • … • Note: “ image ” just placeholder name here for any Eulerian data • Architectures • Fully convolutional • Encoder-Decoder • Skip connections 76

Fully-convolutional Neural Networks FCNN 77

Fully-convolutional Neural Networks FCNN 78

Fully-convolutional Neural Networks FCNN 79

Fully-convolutional Neural Networks FCNN 80

Fully-convolutional Neural Networks FCNN Flexible - works with varying input sizes 81