making convolutional networks shift invariant again
play

Making Convolutional Networks Shift-Invariant Again Richard Zhang - PowerPoint PPT Presentation

Apr 12, 2023 •668 likes •1.36k views

Making Convolutional Networks Shift-Invariant Again Richard Zhang Adobe Research Example classifications P(correct class) P(correct class) not Shift-Invariant Deep Networks are not P(correct class) P(correct class) not Shift-Invariant Deep

  1. Making Convolutional Networks Shift-Invariant Again Richard Zhang Adobe Research

  2. Example classifications P(correct class) P(correct class)

  3. not Shift-Invariant Deep Networks are not P(correct class) P(correct class)

  4. not Shift-Invariant Deep Networks are not P(correct class) P(correct class) Azulay and Weiss. Why y do deep convo volutional networks ks generalize ze so so poorly y to sm small image transf sformations? s? In ArXiv, 2018. Engstrom, Tsipras, Schmidt, Madry. Exp xploring the Landsc scape of Spatial Robust stness. ss. In ICML, 2019.

  5. Why is shift-invariance lost? 5

  6. Why is shift-invariance lost? “ Convo volutions are sh shift-equiva variant ” 6

  7. Why is shift-invariance lost? “ Convo volutions are sh shift-equiva variant ” “ Po Poolin ling builds up sh shift-inva variance ” 7

  8. Why is shift-invariance lost? “ Convo volutions are sh shift-equiva variant ” “ Po Poolin ling builds up sh shift-inva variance ” …but st striding ignores Nyquist sampling theorem and aliase ses 8

  9. Re-examining Max-Pooling 9

  10. Re-examining Max-Pooling max 10

  11. Re-examining Max-Pooling max 11

  12. Re-examining Max-Pooling max 12

  13. Re-examining Max-Pooling max 13

  14. Re-examining Max-Pooling max 14

  15. Re-examining Max-Pooling max 15

  16. Re-examining Max-Pooling max 16

  17. Re-examining Max-Pooling max 17

  18. Re-examining Max-Pooling 18

  19. Re-examining Max-Pooling max 19

  20. Re-examining Max-Pooling max 20

  21. Re-examining Max-Pooling max 21

  22. Re-examining Max-Pooling max 22

  23. Re-examining Max-Pooling max 23

  24. Re-examining Max-Pooling max 24

  25. Re-examining Max-Pooling 25

  26. Re-examining Max-Pooling 26

  27. Re-examining Max-Pooling 27

  28. Re-examining Max-Pooling Max-pooling breaks ks shift-equivariance 28

  29. softmax classifier Shift-equivariance in VGG 1x1 pool5 • CIFAR conv5 • VGG network pool4 • 5 max-pools conv4 pool3 • Test shift-equivariance condition • conv3 pool2 conv2 pool1 conv1 pixels 32x32

  30. softmax classifier Shift-equivariance in VGG 1x1 pool5 • CIFAR conv5 • VGG network pool4 • 5 max-pools conv4 pool3 • Test shift-equivariance condition • conv3 pool2 conv2 pool1 conv1 pixels 32x32

  31. softmax Shift-equivariance, per layer classifier pool5 Large deviation from shift-eq. conv5 pool4 conv4 pool3 conv3 Perfect shift-eq. pool2 conv2 pool1 conv1 v1 31 pixels

  32. softmax Shift-equivariance, per layer classifier pool5 Large deviation from shift-eq. conv5 pool4 conv4 pool3 conv3 Convolution is shift-equivariant Perfect shift-eq. pool2 conv2 pool1 conv1 v1 32 pixels

  33. softmax Shift-equivariance, per layer classifier pool5 Large deviation from shift-eq. conv5 pool4 conv4 pool3 conv3 Perfect shift-eq. pool2 conv2 pool1 po conv1 33 pixels

  34. softmax Shift-equivariance, per layer classifier pool5 Large deviation from shift-eq. conv5 pool4 conv4 pool3 conv3 Pooling breaks shift-equivariance Perfect shift-eq. pool2 conv2 pool1 po conv1 34 pixels

  35. softmax Shift-equivariance, per layer classifier pool5 Large deviation from shift-eq. conv5 pool4 conv4 pool3 conv3 Perfect shift-eq. pool2 po conv2 pool1 conv1 pixels

  36. softmax Shift-equivariance, per layer classifier pool5 Large deviation from shift-eq. conv5 pool4 conv4 pool3 po conv3 Perfect shift-eq. pool2 conv2 pool1 conv1 pixels

  37. softmax Shift-equivariance, per layer classifier pool5 Large deviation from shift-eq. conv5 pool4 po conv4 pool3 conv3 Perfect shift-eq. pool2 conv2 pool1 conv1 pixels

  38. softmax Shift-equivariance, per layer classifier pool5 Large deviation from shift-eq. conv5 po pool4 conv4 pool3 Nyquist theorem ignored when pooling; conv3 sing breaks shift-equivariance aliasi Perfect shift-eq. pool2 conv2 pool1 conv1 pixels

  39. Alternative downsampling methods • Blur+subsample • Antialiasi sing in signal processing; image processing; graphics • Max-pooling • Performs better in deep learning applications [Scherer 2010] 39

  40. Alternative downsampling methods • Blur+subsample • Antialiasi sing in signal processing; image processing; graphics • Max-pooling • Performs better in deep learning applications [Scherer 2010] 40

  41. Alternative downsampling methods • Blur+subsample • Antialiasi sing in signal processing; image processing; graphics • Max-pooling • Performs better in deep learning applications [Scherer 2010] Reconcile antialiasing with max-pooling 41

  42. max( x( ) Baseline (MaxPool) max( x( ) heavy vy aliasi sing

  43. max( x( ) Baseline (MaxPool) max( x( ) heavy vy aliasi sing

  44. max( x( ) max( x( ) Baseline (MaxPool) max( x( ) max( x( ) (1) Max x (dense se eva valuation) heavy vy aliasi sing no aliasi sing

  45. max( x( ) max( x( ) Baseline (MaxPool) max( x( ) max( x( ) (1) Max x (dense se eva valuation) (2) Subsa sampling heavy vy aliasi sing no aliasi sing heavy vy aliasi sing

  46. max( x( ) max( x( ) Baseline (MaxPool) max( x( ) max( x( ) (1) Max x (dense se eva valuation) (2) Subsa sampling heavy vy aliasi sing no aliasi sing heavy vy aliasi sing

  47. max( x( ) max( x( ) Baseline (MaxPool) max( x( ) max( x( ) (1) Max x (dense se eva valuation) (2) Subsa sampling heavy vy aliasi sing no aliasi sing heavy vy aliasi sing max( x( ) Anti-aliased (MaxBlurPool) max( x( ) (1) Max x (dense se eva valuation) no aliasi sing

  48. max( x( ) max( x( ) Baseline (MaxPool) max( x( ) max( x( ) (1) Max x (dense se eva valuation) (2) Subsa sampling heavy vy aliasi sing no aliasi sing heavy vy aliasi sing max( x( ) Anti-aliased Blur Bl (MaxBlurPool) max( x( ) (1) Max x (dense se eva valuation) (2 (2) ) Anti Anti-aliasi sing filter filter no aliasi sing no aliasi sing

  49. max( x( ) max( x( ) Baseline (MaxPool) max( x( ) max( x( ) (1) Max x (dense se eva valuation) (2) Subsa sampling heavy vy aliasi sing no aliasi sing heavy vy aliasi sing max( x( ) Anti-aliased Blur Bl (MaxBlurPool) max( x( ) (1) Max x (dense se eva valuation) (2) (2 ) Anti-aliasi sing filter (3) Subsa sampling no aliasi sing no aliasi sing reduced aliasi sing

  50. max( x( ) max( x( ) Baseline (MaxPool) max( x( ) max( x( ) (1) Max x (dense se eva valuation) (2) Subsa sampling heavy vy aliasi sing no aliasi sing heavy vy aliasi sing max( x( ) Anti-aliased Blur Bl (MaxBlurPool) max( x( ) Evaluated together as “BlurPool” (1) Max x (dense se eva valuation) (2) (2 ) Anti-aliasi sing filter (3) Subsa sampling no aliasi sing no aliasi sing reduced aliasi sing

  51. Antialiasing any downsampling layer • Max Pool • VGG, Alexnet • Strided Convolution • Resnet, MobileNetv2 • Average Pool • DenseNet 51

  52. Antialiasing any downsampling layer • Max Pool • VGG, Alexnet • Strided Convolution • Resnet, MobileNetv2 • Average Pool • DenseNet 52

  53. Antialiasing any downsampling layer • Max Pool • VGG, Alexnet • Strided Convolution • Resnet, MobileNetv2 • Average Pool • DenseNet 53

  54. ImageNet Shift-invariance

  55. ImageNet Shift-invariance Accuracy

  56. ImageNet Shift-invariance Accuracy

  57. ImageNet Baseline Shift-invariance Accuracy

  58. ImageNet Baseline Shift-invariance Antialiased Accuracy

  59. ImageNet Baseline Shift-invariance Antialiased Accuracy

  60. ImageNet Baseline Shift-invariance Antialiased Antialiasing also improves accur accuracy acy Accuracy

  61. Discussion Striding aliases (stride=2) Add antialiasing filter + Improved shift-equivariance + Improved accuracy Additionally + Improved stability to other perturbations + Improved robustness 61

  62. Discussion Striding aliases (stride=2) Add antialiasing filter + Improved shift-equivariance + Improved accuracy Additionally + Improved stability to other perturbations + Improved robustness 62

  63. Discussion Striding aliases (stride=2) Add antialiasing filter + Improved shift-equivariance + Improved accuracy Additionally + Improved stability to other perturbations + Improved robustness 63

  64. Discussion Striding aliases (stride=2) Add antialiasing filter + Improved shift-equivariance + Improved accuracy Additionally + Improved stability to other perturbations + Improved robustness 64

  65. Discussion Striding aliases (stride=2) Add antialiasing filter + Improved shift-equivariance + Improved accuracy Additionally + Improved stability to other perturbations + Improved robustness 65

  66. Discussion Striding aliases (stride=2) Add antialiasing filter + Improved shift-equivariance + Improved accuracy Additionally + Improved stability to other perturbations + Improved robustness 66

  67. Discussion Striding aliases (stride=2) Add antialiasing filter + Improved shift-equivariance + Improved accuracy Additionally + Improved stability to other perturbations + Improved robustness Antialiasing code, pretrained models https://richzhang.github.io/antialiased-cnns/ 67

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Convolutional Neural Networks Basics Praveen Krishnan Overview Paradigm Shift Simple

Convolutional Neural Networks Basics Praveen Krishnan Overview Paradigm Shift Simple

Convolutional Neural Networks Basics Praveen Krishnan Overview Paradigm Shift Simple Network Convolutional Network Layers Case Study1: Alex Net Training Generalization Visualizations Transfer Learning Case

1.02k views • 53 slides
Lecture 22: Linear Shift-Invariant (LSI) Systems and Convolution April 26, 2016. Linear

Lecture 22: Linear Shift-Invariant (LSI) Systems and Convolution April 26, 2016. Linear

Mathematical Tools for Neuroscience (NEU 314) Princeton University, Spring 2016 Jonathan Pillow Lecture 22: Linear Shift-Invariant (LSI) Systems and Convolution April 26, 2016. Linear Shift-Invariant (aka time-invariant) Systems An LSI

302 views • 4 slides
Convolutional Neural Networks ---- Off the shelf top notch performances Convolutional Neural

Convolutional Neural Networks ---- Off the shelf top notch performances Convolutional Neural

Transfer Learning with Convolutional Neural Networks ---- Off the shelf top notch performances Convolutional Neural Networks A breakthough Convolutional Neural Networks VGG-16 example Layers of Convolutional filters Bottleneck

625 views • 23 slides
Shift Invariant Spaces and BMO Morten Nielsen Department of Mathematical Sciences Aalborg

Shift Invariant Spaces and BMO Morten Nielsen Department of Mathematical Sciences Aalborg

Shift Invariant Spaces and BMO Morten Nielsen Department of Mathematical Sciences Aalborg University mnielsen@math.aau.dk Joint work with H. Siki c Fourier Talks, University of Maryland, February 20-21, 2014 M. Nielsen Shift Invariant

621 views • 19 slides
Convolutional Kuan-Ting Lai 2020/3/31 Neural Network Convolutional Neural Networks (CNN)

Convolutional Kuan-Ting Lai 2020/3/31 Neural Network Convolutional Neural Networks (CNN)

Convolutional Kuan-Ting Lai 2020/3/31 Neural Network Convolutional Neural Networks (CNN) A.k.a. CNN or ConvNet Adit Deshpande, A Beginner's Guide To Understanding Convolutional Neural Networks. Digital Images Input array: an images

1.42k views • 72 slides
An Evolutionary View on Reversible Shift-invariant Transformations Luca Mariot, Stjepan Picek,

An Evolutionary View on Reversible Shift-invariant Transformations Luca Mariot, Stjepan Picek,

An Evolutionary View on Reversible Shift-invariant Transformations Luca Mariot, Stjepan Picek, Domagoj Jakobovic, Alberto Leporati l.mariot@tudelft.nl EuroGP 2020, 1517 April 2020 Outline Shift-invariant Transformations and Cellular

430 views • 25 slides
15-780 Graduate Artificial Intelligence: Convolutional and recurrent networks J. Zico Kolter

15-780 Graduate Artificial Intelligence: Convolutional and recurrent networks J. Zico Kolter

15-780 Graduate Artificial Intelligence: Convolutional and recurrent networks J. Zico Kolter (this lecture) and Ariel Procaccia Carnegie Mellon University Spring 2017 1 Outline Convolutional neural networks Applications of convolutional

1.32k views • 32 slides
LSI system Input v v1 x v1 x v2 x v2 x + + L + v3 x + v3 x v4 x + v4 x + Output

LSI system Input v v1 x v1 x v2 x v2 x + + L + v3 x + v3 x v4 x + v4 x + Output

Mathematical Tools for Neural and Cognitive Science Fall semester, 2017 Section 3: Linear Shift-invariant Systems 1 Linear shift-invariant (LSI) systems Linearity (previously discussed): linear combination in, linear combination out

497 views • 24 slides
Convolutional Neural Networks 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing

Convolutional Neural Networks 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing

Convolutional Neural Networks 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing 2016 Prof. Luiz Velho Convolutional Neural Networks 1 Summary & References 08/11 ImageNet Classification with Deep Convolutional Neural Networks

648 views • 26 slides
and Inference for Convolutional Neural Networks 1 2 FFT IFFT 3 4 Mathieu et al.: Fast

and Inference for Convolutional Neural Networks 1 2 FFT IFFT 3 4 Mathieu et al.: Fast

Band-limited Training and Inference for Convolutional Neural Networks 1 2 FFT IFFT 3 4 Mathieu et al.: Fast Training of Convolutional Networks through FFTs Fast Convolutional Nets With fbfft: A GPU Performance Evaluation Data: x

648 views • 46 slides
Convolutional Neural Networks (Part III) 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image

Convolutional Neural Networks (Part III) 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image

Convolutional Neural Networks (Part III) 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing 2016 Prof. Luiz Velho Convolutional Neural Networks 1 Summary & References 08/11 ImageNet Classification with Deep Convolutional

444 views • 27 slides
Convolutional Networks Lecture slides for Chapter 9 of Deep Learning Ian Goodfellow 2016-09-12

Convolutional Networks Lecture slides for Chapter 9 of Deep Learning Ian Goodfellow 2016-09-12

Convolutional Networks Lecture slides for Chapter 9 of Deep Learning Ian Goodfellow 2016-09-12 Convolutional Networks Scale up neural networks to process very large images / video sequences Sparse connections Parameter sharing

552 views • 27 slides
Convolutional Neural Networks in Speech Lecture 20 CS 753 Instructor: Preethi Jyothi

Convolutional Neural Networks in Speech Lecture 20 CS 753 Instructor: Preethi Jyothi

Convolutional Neural Networks in Speech Lecture 20 CS 753 Instructor: Preethi Jyothi Convolutional Neural Networks (CNNs) Fully connected (dense) layers have no awareness of spatial information Key concept behind convolutional layers is

983 views • 41 slides
CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides
Convolutional Neural Nets 4-25-16 Reading Quiz Convolutional neural networks are most commonly

Convolutional Neural Nets 4-25-16 Reading Quiz Convolutional neural networks are most commonly

Convolutional Neural Nets 4-25-16 Reading Quiz Convolutional neural networks are most commonly used for what sort of task? a) Recognizing images b) Transcribing speech c) Translating documents d) Playing games Neural network review

431 views • 13 slides
Convolutional Neural Networks Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline GPUs

Convolutional Neural Networks Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline GPUs

Convolutional Neural Networks Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline GPUs Convolutions Pooling, Padding and Stride Convolutional Neural Networks (LeNet) Deep ConvNets (AlexNet) Networks using Blocks (VGG)

1.91k views • 89 slides
COMP30019 Graphics and Interaction Rasterization and Barycentric Coordinates Adrian Pearce

COMP30019 Graphics and Interaction Rasterization and Barycentric Coordinates Adrian Pearce

Introduction COMP30019 Graphics and Interaction Rasterization and Barycentric Coordinates Adrian Pearce Department of Computing and Information Systems University of Melbourne The University of Melbourne Adrian Pearce University of Melbourne

269 views • 7 slides
Lecture 6 - Light Transport Welcome! , = (, ) ,

Lecture 6 - Light Transport Welcome! , = (, ) ,

INFOMAGR Advanced Graphics Jacco Bikker - November 2017 - February 2018 Lecture 6 - Light Transport Welcome! , = (, ) , + , , ,

1.04k views • 52 slides
1 2 Note approXimate! FXAA does not attempt anything close to the correct solution. Rather

1 2 Note approXimate! FXAA does not attempt anything close to the correct solution. Rather

1 2 Note approXimate! FXAA does not attempt anything close to the correct solution. Rather attempts something fast that looks visually good enough. 3 (1.) MEMORY PROBLEM At huge resolutions with deferred rendering, memory used for render

619 views • 27 slides
GPU-Based Large-Scale Scientific Visualization Johanna Beyer, Harvard University Markus

GPU-Based Large-Scale Scientific Visualization Johanna Beyer, Harvard University Markus

GPU-Based Large-Scale Scientific Visualization Johanna Beyer, Harvard University Markus Hadwiger, KAUST Course Website: http://johanna-b.github.io/LargeSciVis2018/index.html Part 4 - Display-Aware Visualization and Processing MOTIVATION

972 views • 56 slides
Recall: Antialiasing Raster displays have pixels as rectangles Aliasing: Discrete nature of

Recall: Antialiasing Raster displays have pixels as rectangles Aliasing: Discrete nature of

Recall: Antialiasing Raster displays have pixels as rectangles Aliasing: Discrete nature of pixels introduces jaggies Recall: Antialiasing Aliasing effects: Distant objects may disappear entirely Objects can blink on and

795 views • 41 slides
Front-End Electronics based on Waveform Sampling Feature Extraction Grzegorz Pastuszak and

Front-End Electronics based on Waveform Sampling Feature Extraction Grzegorz Pastuszak and

Front-End Electronics based on Waveform Sampling Feature Extraction Grzegorz Pastuszak and Marcin Ziembicki Warsaw University of Technology and AstroCeNT Advanced Workshop on Modern FPGA Based Technology for Scientific Computing ICTP, Trieste,

613 views • 33 slides
Transparency CISC 3620 May 15, 2017 Outline Reading Real-Time Rendering, 5.7 Semitransparent

Transparency CISC 3620 May 15, 2017 Outline Reading Real-Time Rendering, 5.7 Semitransparent

Transparency CISC 3620 May 15, 2017 Outline Reading Real-Time Rendering, 5.7 Semitransparent object View-based effects: semitransparent object itself is rendered Light-based effects: object causes light to be weakened or bent, which

734 views • 17 slides
Effects on Scratched Surfaces Zdravko Velinov 1 , Sebastian Werner 1 , Matthias Hullin 1 1

Effects on Scratched Surfaces Zdravko Velinov 1 , Sebastian Werner 1 , Matthias Hullin 1 1

Real-Time Rendering of Wave-Optical Effects on Scratched Surfaces Zdravko Velinov 1 , Sebastian Werner 1 , Matthias Hullin 1 1 University of Bonn, Germany Motivation Iridescent scratches defined by wave-optical phenomena appear on many

730 views • 28 slides

More recommend