Introduction to High-Performance Machine Learning: Convolutional - PowerPoint PPT Presentation

Introduction to High-Performance Machine Learning: Convolutional Neural Networks Valeriu Codreanu SURFsara 1 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

SURFsara History: 1971: Founded by the VU, UvA, and CWI 2013: SARA (Stichting Academisch Rekencentrum A’dam) becomes part of SURF Cartesius (Bull supercomputer): 40.960 Ivy Bridge / Haswell cores: 1327 TFLOPS 56GBit/s Infiniband 64 nodes with 2 K40m GPUs each: 210 TFLOPS Broadwell & KNL extension (Nov 2016) 177 BDW and 18 KNL nodes: 284TFLOPS 7.7 PB Lustre parallel file-system Top500 position #45 2014/11 Increasing number of deep learning #142 2017/11 projects! 2 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Outline Today’s lecture: • NNs for Computer vision • High-performance CNN vision models (AlexNet, GoogLeNet, ResNet, …) • High-performance hardware • GPUs, CPUs, special-purpose accelerators • Scaling deep learning vision models • Bottlenecks • Efficiency • Successful results • Today’s hands-on: • Deep learning frameworks • Example: Tensorflow/Horovod and MXNet on Intel and NVIDIA • Example: Keras 3 GPU Programming www.prace-ri.eu

Classical Computer Vision Pipeline CV experts 1.Select / develop features: SURF, HoG, SIFT, RIFT, … 2.Add on top of this Machine Learning for multi-class recognition and train classifier Feature Detection, Extraction: Classification SIFT, HoG... Recognition Classical CV feature definition is domain- specific and time-consuming 4 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Deep Learning–based Vision Pipeline Deep Learning: • Build features automatically based on training data • Combine feature extraction and classification DL experts: define NN topology and train NN Detection, Deep NN ... Deep NN ... Classification Recognition Deep Learning promise: train good feature automatically, same method for different domain 5 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Neural networks 6 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

7 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Object segmentation Figure credit: Dai, He, and Sun, “Instance-aware Semantic Segmentation via Multi-task Network Cascades”, CVPR 2016 8 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Pose estimation Figure credit: Cao et al, “Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields”, arXiv 2016 9 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Image Super-resolution Figure credit: Ledig et al, “Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network”, arXiv 2016 10 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Art generation Gatys, Ecker, and Bethge, “Image Style Transfer using Convolutional Neural Networks”, CVPR 2016 (left) Mordvintsev, Olah, and Tyka, “Inceptionism: Going Deeper into Neural Networks” (upper right) Johnson, Alahi, and Fei-Fei: “Perceptual Losses for Real-Time Style Transfer and Super-Resolution”, ECCV 2016 (bottom left) 11 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Neural networks 12 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

The neuron model Based on slide from Andrew Ng 13 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

14 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks Parallel Computer Architecture and Programming CMU 15-418/15-618, Spring 2017

15 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks Parallel Computer Architecture and Programming CMU 15-418/15-618, Spring 2017

4 easy steps: 1. Sample a batch of data 2. Forward prop it through the graph (network), get loss 3. Backprop to calculate the gradients 4. Update the parameters using the gradient 16 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Fully-connected networks Nonlinearity Matrix Matrix Multiply Multiply x s a ŷ [C 1 ] [C 2 ] [C 2 ] [C 3 ] w 2 w 1 [C 2 ×C 3 ] [C 1 ×C 2 ] 17 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Convolutional neural networks Fully Connected y t i n r a o e i t u n Pooling i l l o n v o n N o C x s a p ŷ C 1 ×H×W C 2 ×H×W C 2 ×H×W C 2 ×H/2×W/2 C 3 w 1 w 2 C 2 × C 1 × k × k C 2 HW/4×C 3 18 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

The convolution operator Sobel operator: 19 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 20 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 21 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 22 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 23 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 24 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 25 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 26 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Convolution as matrix-matrix multiplication Very important factor motivating early GPU usage for neural network training! 27 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 28 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 29 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

l Subsampling pixels will not change the object bird bird Subsampling Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 30 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 31 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

LeNet-5 Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 32 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Neural networks 33 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

AlexNet 5 convolutional layers 3 fully connected layers + soft-max 650K neurons , 60 M weights 34 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

35 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Depth reduction experiment on AlexNet Rob Fergus, NIPS 2013 36 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Depth reduction experiment on AlexNet 37 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Depth reduction experiment on AlexNet 38 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Depth reduction experiment on AlexNet 39 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Depth reduction experiment on AlexNet 40 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Feature visualization Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 41 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks