highway networks and residual networks
play

Highway Networks and Residual Networks Renjie Liao University of - PowerPoint PPT Presentation

Feb 07, 2024 •270 likes •685 views

Highway Networks and Residual Networks Renjie Liao University of Toronto Jan 26, 2016 Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 1 / 23 Neural Network A network connecting numerous neurons Renjie Liao (UofT)

  1. Highway Networks and Residual Networks Renjie Liao University of Toronto Jan 26, 2016 Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 1 / 23

  2. Neural Network A network connecting numerous neurons Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 2 / 23

  3. Neural Network A network connecting numerous neurons Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 2 / 23

  4. Analogy Imagine a neural network as a map Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 3 / 23

  5. Analogy Imagine a neural network as a map Imagine a neuron as a place Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 3 / 23

  6. Analogy Imagine a neural network as a map Imagine a neuron as a place Imagine yourself as the information flow Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 3 / 23

  7. Analogy Imagine a neural network as a map Imagine a neuron as a place Imagine yourself as the information flow Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 3 / 23

  8. Analogy Suppose you (information flow) wants to reach Bakery (neuron B) from City Hall (neuron A), what will you do? Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 4 / 23

  9. Analogy Suppose you (information flow) wants to reach Bakery (neuron B) from City Hall (neuron A), what will you do? You have to follow the path of network! Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 4 / 23

  10. Analogy Suppose you (information flow) wants to reach Bakery (neuron B) from City Hall (neuron A), what will you do? You have to follow the path of network! What if there is a highway connecting Bakery and City Hall directly? Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 4 / 23

  11. Highway Networks Allowing direct pass (highway) between neurons in different layers. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 5 / 23

  12. Highway Networks Allowing direct pass (highway) between neurons in different layers. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 5 / 23

  13. Highway Networks Original network: �� � w 1 z 1 = σ n x n + b (1) n =1 Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 6 / 23

  14. Highway Networks Original network: �� � w 1 z 1 = σ n x n + b (1) n =1 Highway network: �� � w 1 z 1 = T σ n x n + b + (1 − T ) x 1 (2) n =1 Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 6 / 23

  15. Highway Networks Original network: �� � w 1 z 1 = σ n x n + b (1) n =1 Highway network: �� � w 1 z 1 = T σ n x n + b + (1 − T ) x 1 (2) n =1 Gating function: �� � T = σ w ′ n x n + b ′ (3) n =1 Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 6 / 23

  16. Highway Networks Remember the shape of sigmoid function. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 7 / 23

  17. Highway Networks Remember the shape of sigmoid function. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 7 / 23

  18. Highway Networks Remember the shape of sigmoid function. We can set bias b ′ to negative values such that gating value T → 0. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 7 / 23

  19. Benefits of Highway networks Enable training of very deep neural networks (e.g., hundreds of layers) “Srivastava, R.K., Greff, K. and Schmidhuber, J., 2015. Highway Networks. arXiv preprint arXiv:1505.00387”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 8 / 23

  20. Benefits of Highway networks Enable training of very deep neural networks (e.g., hundreds of layers) “Srivastava, R.K., Greff, K. and Schmidhuber, J., 2015. Highway Networks. arXiv preprint arXiv:1505.00387”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 9 / 23

  21. Residual Networks Motivation: Does depth matter for deep learning? Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 10 / 23

  22. Residual Networks Motivation: Does depth matter for deep learning? “He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 10 / 23

  23. Residual Networks Motivation: Does depth matter for deep learning? “He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 11 / 23

  24. Residual Networks We need new architecture to make depth matter. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 12 / 23

  25. Residual Networks We need new architecture to make depth matter. Suppose you have a plain 2-layer network H . “He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 12 / 23

  26. Residual Networks We need new architecture to make depth matter. Suppose you have a plain 2-layer network H . We use a new building block which forces the previous 2-layer F to learn the residual H − x . Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 13 / 23

  27. Residual Networks We need new architecture to make depth matter. Suppose you have a plain 2-layer network H . We use a new building block which forces the previous 2-layer F to learn the residual H − x . “He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 13 / 23

  28. Residual Networks What we have done? Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 14 / 23

  29. Residual Networks What we have done? Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 14 / 23

  30. Residual Networks What we have done? “He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 14 / 23

  31. Residual Networks Based on this building block, we can do some crazy things like... Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 15 / 23

  32. Residual Networks Based on this building block, we can do some crazy things like... “He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 15 / 23

  33. Image Net Challenge Results “He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 16 / 23

  34. PASCAL VOC Challenge Results “He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 17 / 23

  35. More Results “He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 18 / 23

  36. More Results “He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 19 / 23

  37. More Results “He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 20 / 23

  38. More Results “He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 21 / 23

  39. More Results “He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 22 / 23

  40. Thanks! Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 23 / 23

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Architecture Perceptron Highway Networks Highway Networks Allows training very deep

Architecture Perceptron Highway Networks Highway Networks Allows training very deep

Highway Networks for Visual Question Answering Aaditya Prakash PhD advisor: James Storer Brandeis University Architecture Perceptron Highway Networks Highway Networks Allows training very deep networks Srivastava et al trained

174 views • 16 slides
Residual Networks (ResNet) Residual Networks (ResNet) In [1]: import d2l from mxnet import gluon,

Residual Networks (ResNet) Residual Networks (ResNet) In [1]: import d2l from mxnet import gluon,

3/8/2019 resnet slides Residual Networks (ResNet) Residual Networks (ResNet) In [1]: import d2l from mxnet import gluon, init, nd from mxnet.gluon import nn http://127.0.0.1:8000/resnet.slides.html?print-pdf/#/ 1/10 3/8/2019 resnet slides

890 views • 10 slides
Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What

Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What

Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What well cover ... f ( x ) computer vision architectures - convolutional networks - data augmentation 1 - residual networks ... ...

1.29k views • 75 slides
Invertible Residual Networks Jens Behrmann * Will Grathwohl* Ricky T. Q. Chen David Duvenaud

Invertible Residual Networks Jens Behrmann * Will Grathwohl* Ricky T. Q. Chen David Duvenaud

Invertible Residual Networks Jens Behrmann * Will Grathwohl* Ricky T. Q. Chen David Duvenaud Jrn-Henrik Jacobsen* (*equal contribution) What are Invertible Neural Networks? Non-invertible Invertible Invertible Neural Networks (INNs) are

330 views • 30 slides
XDN: Towards Efficient Inference of Residual Neural Networks on Cambricon Chips 2019

XDN: Towards Efficient Inference of Residual Neural Networks on Cambricon Chips 2019

(XiaoDianNao) XDN: Towards Efficient Inference of Residual Neural Networks on Cambricon Chips 2019 BenchCouncil International Artificial Intelligence System Challenges Guangli Li , Xueying Wang, Xiu Ma Advisor: Prof. Xiaobing Feng

934 views • 16 slides
AMMI Introduction to Deep Learning 6.5. Residual networks Fran cois Fleuret

AMMI Introduction to Deep Learning 6.5. Residual networks Fran cois Fleuret

AMMI Introduction to Deep Learning 6.5. Residual networks Fran cois Fleuret https://fleuret.org/ammi-2018/ Fri Nov 9 22:38:28 UTC 2018 COLE POLYTECHNIQUE FDRALE DE LAUSANNE Residual networks Fran cois Fleuret AMMI

732 views • 36 slides
Argumentative Link Prediction using Residual Networks and Multi-Objective Learning Galassi Andrea

Argumentative Link Prediction using Residual Networks and Multi-Objective Learning Galassi Andrea

Argumentative Link Prediction using Residual Networks and Multi-Objective Learning Galassi Andrea 1 Departments of 1 Computer Science and Engineering (DISI) Marco Lippi 2 University of Bologna, Italy Paolo Torroni 1 2 Sciences and Methods for

387 views • 20 slides
Reduced-memory training and deployment of deep residual networks by stochastic binary quantization

Reduced-memory training and deployment of deep residual networks by stochastic binary quantization

Reduced-memory training and deployment of deep residual networks by stochastic binary quantization Mark D. McDonnell 1 , Ruchun Wang 2 and Andr van Schaik 2 cls-lab.org 2 BENS Laboratory 1 Computational Learning Systems Laboratory MARCS

256 views • 23 slides
Information Visualization networks without networks, couldn't have any of these: Dataset

Information Visualization networks without networks, couldn't have any of these: Dataset

Network data Applications of networks Information Visualization networks without networks, couldn't have any of these: Dataset Types model relationships Networks & Trees 1/2 between things Tables Networks Fields (Continuous)

263 views • 4 slides
Types of networks (social networks, computer networks, entity- relationship networks, )

Types of networks (social networks, computer networks, entity- relationship networks, )

Networks Types of networks (social networks, computer networks, entity- relationship networks, ) Node-link diagrams Layered Internet architecture (encapsulation) The Internet Structure of the Internet core

556 views • 16 slides
Residual Flows for Invertible Generative Modeling Ricky T. Q. Chen, Jens Behrmann, David

Residual Flows for Invertible Generative Modeling Ricky T. Q. Chen, Jens Behrmann, David

Residual Flows for Invertible Generative Modeling Ricky T. Q. Chen, Jens Behrmann, David Duvenaud, Jrn-Henrik Jacobsen Invertible Residual Networks (i-ResNet) It can be shown that residual blocks can be inverted by fixed-point iteration and

442 views • 16 slides
Social and Technological Networks: Review Rik Sarkar Networks Networks/graphs are

Social and Technological Networks: Review Rik Sarkar Networks Networks/graphs are

Social and Technological Networks: Review Rik Sarkar Networks Networks/graphs are fundamental in computer science And becoming more important As systems, people, computation become more interconnected Why study networks We

223 views • 21 slides
1 Hardware architectures Point-to-point networks Many different types of networks: One

1 Hardware architectures Point-to-point networks Many different types of networks: One

Networks in embedded systems Networking for Embedded Systems Why we use networks. Network abstractions. Example networks. initial processing more processing Properties of wireless networks. PE sensor PE PE actuator PEs

573 views • 7 slides
4-connected shift residual networks ICCV 2019 Neural Architects Workshop Andrew Brown, Pascal

4-connected shift residual networks ICCV 2019 Neural Architects Workshop Andrew Brown, Pascal

4-connected shift residual networks ICCV 2019 Neural Architects Workshop Andrew Brown, Pascal Mettes, Marcel Worring, University of Amsterdam Network costs increasing! Increasing accuracy on ImageNet has come at increasing cost Popular

280 views • 15 slides
Current Network Structure for Pediatrics Hospital Networks Country, state, regional, Academic

Current Network Structure for Pediatrics Hospital Networks Country, state, regional, Academic

Current Network Structure for Pediatrics Hospital Networks Country, state, regional, Academic networks networks Sub-specialty NIH networks networks Foundation networks Trial recruitment networks Disease specific networks Networks for

361 views • 6 slides
Computer Networks I Computer Networks I Networks A networks connection structure is known as

Computer Networks I Computer Networks I Networks A networks connection structure is known as

Computer Networks I Computer Networks I Networks A networks connection structure is known as its network topology . Host nodes source and destination of messages Communication nodes where messages pass through; can be

304 views • 26 slides
PROPOSITION 1 PROPOSITION 1 Map tiles by CartoDB, under CC BY 3.0. Data by OpenStreetMap, under

PROPOSITION 1 PROPOSITION 1 Map tiles by CartoDB, under CC BY 3.0. Data by OpenStreetMap, under

FY 2015 Project Priorities PROPOSITION 1 PROPOSITION 1 Map tiles by CartoDB, under CC BY 3.0. Data by OpenStreetMap, under ODbL. 1 FY2015 Prop 1 Funds FY2015 Prop 1 Funds $1.74 billion deposited in Highway Fund $1.74 billion deposited in

417 views • 10 slides
From data to publication Walk through ATSAS programs and data deposition Al Kikhney EMBL Hamburg

From data to publication Walk through ATSAS programs and data deposition Al Kikhney EMBL Hamburg

From data to publication Walk through ATSAS programs and data deposition Al Kikhney EMBL Hamburg Solution Scattering from Biological Macromolecules July 7, 2020 ATSAS software package 3.0 Over 90 programs Operating systems: Windows

727 views • 60 slides
I SPD 2 0 0 7 Shankar Krishnamoorthy Chief Technical Officer Agenda Physical Design at

I SPD 2 0 0 7 Shankar Krishnamoorthy Chief Technical Officer Agenda Physical Design at

Variation and Litho-driven Physical Design I SPD 2 0 0 7 Shankar Krishnamoorthy Chief Technical Officer Agenda Physical Design at 65nm/45nm Variation challenges and solutions Routing challenges and solutions Summary 2

720 views • 31 slides
Flood Management Task Force November 13, 2020 Welcome and Introductions Thanks for attending!

Flood Management Task Force November 13, 2020 Welcome and Introductions Thanks for attending!

Flood Management Task Force November 13, 2020 Welcome and Introductions Thanks for attending! Please introduce yourself in the chat box. Please mute your line. Unmute your line when you would like to speak during question and

573 views • 41 slides
Slugest 2019 - Hi Res Slides Presentation November 2019 DOI: 10.13140/RG.2.2.12250.90566

Slugest 2019 - Hi Res Slides Presentation November 2019 DOI: 10.13140/RG.2.2.12250.90566

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/337547393 Slugest 2019 - Hi Res Slides Presentation November 2019 DOI: 10.13140/RG.2.2.12250.90566 CITATIONS READS 0 708 1

563 views • 18 slides
Improving the Quality of Your Cyber Security Hires via Pre- Interview Challenges Dinis Cruz

Improving the Quality of Your Cyber Security Hires via Pre- Interview Challenges Dinis Cruz

Improving the Quality of Your Cyber Security Hires via Pre- Interview Challenges Dinis Cruz CISO Photobox Group, 25th Jan 2017 The CV inefficient workflow High Velocity hiring How do you hire 6x senior highly skilful and motivated talent

554 views • 40 slides
High resolution observations of SEGUE stars Sara Lucatello MPA and INAF Osservatorio

High resolution observations of SEGUE stars Sara Lucatello MPA and INAF Osservatorio

High resolution observations of SEGUE stars Sara Lucatello MPA and INAF Osservatorio Astronomico di Padova SEGUE Observations of halo and disk to map position, composition and velocity of >300,000 stars Tremendous amount of

291 views • 16 slides
t ' ! tractable probabilistic inference meeting ! December 11th 2019 - NeurIPS 2019 , Vancouver

t ' ! tractable probabilistic inference meeting ! December 11th 2019 - NeurIPS 2019 , Vancouver

t ' ! tractable probabilistic inference meeting ! December 11th 2019 - NeurIPS 2019 , Vancouver Lets discuss about the current state of flexible , reliable , and efficient probabilistic inference and learning and where we want it to be! 2

547 views • 26 slides

More recommend