Highway Networks and Residual Networks Renjie Liao University of - - PowerPoint PPT Presentation

▶
highway networks and residual networks
SMART_READER_LITE
LIVE PREVIEW

Highway Networks and Residual Networks Renjie Liao University of - - PowerPoint PPT Presentation

Feb 07, 2024 270 likes •691 views

Highway Networks and Residual Networks Renjie Liao University of Toronto Jan 26, 2016 Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 1 / 23 Neural Network A network connecting numerous neurons Renjie Liao (UofT)

slide-1
SLIDE 1

Highway Networks and Residual Networks

Renjie Liao

University of Toronto

Jan 26, 2016

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 1 / 23

slide-2
SLIDE 2

Neural Network

A network connecting numerous neurons

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 2 / 23

slide-3
SLIDE 3

Neural Network

A network connecting numerous neurons

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 2 / 23

slide-4
SLIDE 4

Analogy

Imagine a neural network as a map

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 3 / 23

slide-5
SLIDE 5

Analogy

Imagine a neural network as a map Imagine a neuron as a place

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 3 / 23

slide-6
SLIDE 6

Analogy

Imagine a neural network as a map Imagine a neuron as a place Imagine yourself as the information flow

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 3 / 23

slide-7
SLIDE 7

Analogy

Imagine a neural network as a map Imagine a neuron as a place Imagine yourself as the information flow

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 3 / 23

slide-8
SLIDE 8

Analogy

Suppose you (information flow) wants to reach Bakery (neuron B) from City Hall (neuron A), what will you do?

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 4 / 23

slide-9
SLIDE 9

Analogy

Suppose you (information flow) wants to reach Bakery (neuron B) from City Hall (neuron A), what will you do? You have to follow the path of network!

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 4 / 23

slide-10
SLIDE 10

Analogy

Suppose you (information flow) wants to reach Bakery (neuron B) from City Hall (neuron A), what will you do? You have to follow the path of network! What if there is a highway connecting Bakery and City Hall directly?

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 4 / 23

slide-11
SLIDE 11

Highway Networks

Allowing direct pass (highway) between neurons in different layers.

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 5 / 23

slide-12
SLIDE 12

Highway Networks

Allowing direct pass (highway) between neurons in different layers.

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 5 / 23

slide-13
SLIDE 13

Highway Networks

Original network: z1 = σ

  • n=1

w1

nxn + b

  • (1)

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 6 / 23

slide-14
SLIDE 14

Highway Networks

Original network: z1 = σ

  • n=1

w1

nxn + b

  • (1)

Highway network: z1 = Tσ

  • n=1

w1

nxn + b

  • + (1 − T)x1

(2)

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 6 / 23

slide-15
SLIDE 15

Highway Networks

Original network: z1 = σ

  • n=1

w1

nxn + b

  • (1)

Highway network: z1 = Tσ

  • n=1

w1

nxn + b

  • + (1 − T)x1

(2) Gating function: T = σ

  • n=1

w′

nxn + b′

  • (3)

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 6 / 23

slide-16
SLIDE 16

Highway Networks

Remember the shape of sigmoid function.

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 7 / 23

slide-17
SLIDE 17

Highway Networks

Remember the shape of sigmoid function.

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 7 / 23

slide-18
SLIDE 18

Highway Networks

Remember the shape of sigmoid function. We can set bias b′ to negative values such that gating value T → 0.

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 7 / 23

slide-19
SLIDE 19

Benefits of Highway networks

Enable training of very deep neural networks (e.g., hundreds of layers)

“Srivastava, R.K., Greff, K. and Schmidhuber, J., 2015. Highway Networks. arXiv preprint arXiv:1505.00387”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 8 / 23

slide-20
SLIDE 20

Benefits of Highway networks

Enable training of very deep neural networks (e.g., hundreds of layers)

“Srivastava, R.K., Greff, K. and Schmidhuber, J., 2015. Highway Networks. arXiv preprint arXiv:1505.00387”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 9 / 23

slide-21
SLIDE 21

Residual Networks

Motivation: Does depth matter for deep learning?

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 10 / 23

slide-22
SLIDE 22

Residual Networks

Motivation: Does depth matter for deep learning?

“He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 10 / 23

slide-23
SLIDE 23

Residual Networks

Motivation: Does depth matter for deep learning?

“He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 11 / 23

slide-24
SLIDE 24

Residual Networks

We need new architecture to make depth matter.

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 12 / 23

slide-25
SLIDE 25

Residual Networks

We need new architecture to make depth matter. Suppose you have a plain 2-layer network H.

“He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 12 / 23

slide-26
SLIDE 26

Residual Networks

We need new architecture to make depth matter. Suppose you have a plain 2-layer network H. We use a new building block which forces the previous 2-layer F to learn the residual H − x.

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 13 / 23

slide-27
SLIDE 27

Residual Networks

We need new architecture to make depth matter. Suppose you have a plain 2-layer network H. We use a new building block which forces the previous 2-layer F to learn the residual H − x.

“He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 13 / 23

slide-28
SLIDE 28

Residual Networks

What we have done?

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 14 / 23

slide-29
SLIDE 29

Residual Networks

What we have done?

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 14 / 23

slide-30
SLIDE 30

Residual Networks

What we have done?

“He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 14 / 23

slide-31
SLIDE 31

Residual Networks

Based on this building block, we can do some crazy things like...

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 15 / 23

slide-32
SLIDE 32

Residual Networks

Based on this building block, we can do some crazy things like...

“He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 15 / 23

slide-33
SLIDE 33

Image Net Challenge Results

“He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 16 / 23

slide-34
SLIDE 34

PASCAL VOC Challenge Results

“He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 17 / 23

slide-35
SLIDE 35

More Results

“He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 18 / 23

slide-36
SLIDE 36

More Results

“He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 19 / 23

slide-37
SLIDE 37

More Results

“He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 20 / 23

slide-38
SLIDE 38

More Results

“He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 21 / 23

slide-39
SLIDE 39

More Results

“He, K., Zhang, X., Ren, S. and Sun, J., 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385”. Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 22 / 23

slide-40
SLIDE 40

Thanks!

Renjie Liao (UofT) Highway Networks and Residual Networks Jan 26, 2016 23 / 23