The vanishing gradient problem revisited: Highway and residual - - PowerPoint PPT Presentation
The vanishing gradient problem revisited: Highway and residual connections CS 6956: Deep Learning for NLP Revisiting the vanishing gradient problem Stems from the fact that the derivative of the activation is between zero and one and
1
2
Inputs Outputs Many layers in between Loss
3
Inputs Outputs Many layers in between Loss Gradient vanishes as the depth grows
4
Inputs Outputs Many layers in between Loss Gradient vanishes as the depth grows The loss is no longer influenced by the inputs for very deep networks!
5
Inputs Outputs Many layers in between Loss Gradient vanishes as the depth grows The loss is no longer influenced by the inputs for very deep networks!
6
The t-1th layer is used to calculate the value of the tth layer
7
Instead of a non-linear update that directly calculates the next layer, let us try a linear update
8
Instead of a non-linear update that directly calculates the next layer, let us try a linear update The gradients can be propagated all the way to the input without attenuation
9
[He et al 2015] π"%& π(π"%&π + π"%&) π" Original layer
10
[He et al 2015] π"%& π(π"%&π + π"%&) π" π"%& π(π"%&π + π"%&) + π" Original layer Residual connection
11
[He et al 2015]
12
[Srivastava et al 2015]
13
[Srivastava et al 2015]
14
[Srivastava et al 2015]
15
[Srivastava et al 2015]
16