Neural Networks Luke Zettlemoyer (Slides adapted from Danqi Chen, - - PowerPoint PPT Presentation

neural networks
SMART_READER_LITE
LIVE PREVIEW

Neural Networks Luke Zettlemoyer (Slides adapted from Danqi Chen, - - PowerPoint PPT Presentation

CSEP 517 Natural Language Processing Neural Networks Luke Zettlemoyer (Slides adapted from Danqi Chen, Chris Manning, Dan Jurafsky) Neural networks for NLP Feed-forward NNs Recurrent NNs Transformer Convolutional NNs Always coupled with


slide-1
SLIDE 1

CSEP 517 Natural Language Processing

Neural Networks

Luke Zettlemoyer

(Slides adapted from Danqi Chen, Chris Manning, Dan Jurafsky)

slide-2
SLIDE 2

Neural networks for NLP

Feed-forward NNs

Recurrent NNs

Convolutional NNs

Transformer

Always coupled with word embeddings…

slide-3
SLIDE 3

This Lecture

  • Feedforward Neural Networks
  • Applications
  • Neural Bag-of-Words Models
  • Feedforward Neural Language Models
  • The training algorithm: Back-propagation
slide-4
SLIDE 4

Neural Networks: History

slide-5
SLIDE 5

NN “dark ages”

  • ConvNets: applied to MNIST by LeCun in 1998

Credits: Greg Durrett

  • Long Short-term Memory Networks (LSTMs): Hochreiter

and Schmidhuber 1997

  • Henderson 2003: neural shift-reduce parser, not SOTA
  • Neural network algorithms date from the 80s
slide-6
SLIDE 6

2008-2013: A glimmer of light

  • Collobert and Weston 2011: “NLP (almost) from

Scratch”

  • Feedforward NNs can replace “feature

engineering”

  • 2008 version was marred by bad experiments,

claimed SOTA but wasn’t, 2011 version tied SOTA Credits: Greg Durrett

  • Krizhevskey et al, 2012: AlexNet for ImageNet

Classification

  • Socher 2011-2014: tree-structured

RNNs working okay

slide-7
SLIDE 7

2014: Stuff starts working

  • Kim (2014) + Kalchbrenner et al, 2014: sentence classification
  • ConvNets work for NLP!

Credits: Greg Durrett

  • Sutskever et al, 2014: sequence-to-sequence for neural MT
  • LSTMs work for NLP!
  • Chen and Manning 2014: dependency parsing
  • Even feedforward networks work well for NLP!
  • 2015: explosion of neural networks for everything under the sun
slide-8
SLIDE 8

Why didn’t they work before?

  • Datasets too small: for MT, not really better until you have

1M+ parallel sentences (and really need a lot more)

Credits: Greg Durrett

  • Optimization not well understood: good initialization, per-

feature scaling + momentum (Adagrad/Adam) work best out-of- the-box

  • Regularization: dropout is pretty helpful
  • Computers not big enough: can’t run for enough iterations
  • Inputs: need word embeddings to represent continuous semantics
slide-9
SLIDE 9

The “Promise”

  • Most NLP works in the past focused on human-designed

representations and input features

  • Representation learning attempts to automatically learn

good features and representations

  • Deep learning attempts to learn multiple levels of

representation on increasing complexity/abstraction

slide-10
SLIDE 10

Feed-forward Neural Networks

slide-11
SLIDE 11

Feed-forward NNs

  • Input:
  • Output:

x1, …, xd y ∈ {0,1}

slide-12
SLIDE 12

Neural computation

Computation units: neurons

slide-13
SLIDE 13

An artificial neuron

  • A neuron is a computational unit that has scalar inputs and an
  • utput
  • Each input has an associated weight.
  • The neuron multiples each input by its weight, sums them, applied

a nonlinear function to the result, and passes it to its output.

slide-14
SLIDE 14

Neural networks

  • The neurons are connected to each other, forming a network
  • The output of a neuron may feed into the inputs of other neurons
slide-15
SLIDE 15

A neuron can be a binary logistic regression unit

f(z) = 1 1 + e−z

<latexit sha1_base64="4snGlhjxkE7I7roWcflso17LAcU=">ACBHicbVDLSgMxFM3UV62vUZfdBItQEctEBN0IRTcuK9gHtGPJpJk2NJMZkozQDrNw46+4caGIWz/CnX9j2s5CWw9cOJxzL/fe40WcKe0431ZuaXldS2/XtjY3NresXf3GiqMJaF1EvJQtjysKGeC1jXTnLYiSXHgcdr0htcTv/lApWKhuNOjiLoB7gvmM4K1kbp20S+Pj+Al7PgSkwSlCYLHkN4nJ+M07dolp+JMARcJykgJZKh17a9OLyRxQIUmHCvVRk6k3QRLzQinaETKxphMsR92jZU4IAqN5k+kcJDo/SgH0pTQsOp+nsiwYFSo8AznQHWAzXvTcT/vHas/Qs3YSKNRVktsiPOdQhnCQCe0xSovnIEwkM7dCMsAmDm1yK5gQ0PzLi6RxWkFOBd2elapXWRx5UAQHoAwQOAdVcANqoA4IeATP4BW8WU/Wi/Vufcxac1Y2sw/+wPr8ASGfln4=</latexit><latexit sha1_base64="4snGlhjxkE7I7roWcflso17LAcU=">ACBHicbVDLSgMxFM3UV62vUZfdBItQEctEBN0IRTcuK9gHtGPJpJk2NJMZkozQDrNw46+4caGIWz/CnX9j2s5CWw9cOJxzL/fe40WcKe0431ZuaXldS2/XtjY3NresXf3GiqMJaF1EvJQtjysKGeC1jXTnLYiSXHgcdr0htcTv/lApWKhuNOjiLoB7gvmM4K1kbp20S+Pj+Al7PgSkwSlCYLHkN4nJ+M07dolp+JMARcJykgJZKh17a9OLyRxQIUmHCvVRk6k3QRLzQinaETKxphMsR92jZU4IAqN5k+kcJDo/SgH0pTQsOp+nsiwYFSo8AznQHWAzXvTcT/vHas/Qs3YSKNRVktsiPOdQhnCQCe0xSovnIEwkM7dCMsAmDm1yK5gQ0PzLi6RxWkFOBd2elapXWRx5UAQHoAwQOAdVcANqoA4IeATP4BW8WU/Wi/Vufcxac1Y2sw/+wPr8ASGfln4=</latexit><latexit sha1_base64="4snGlhjxkE7I7roWcflso17LAcU=">ACBHicbVDLSgMxFM3UV62vUZfdBItQEctEBN0IRTcuK9gHtGPJpJk2NJMZkozQDrNw46+4caGIWz/CnX9j2s5CWw9cOJxzL/fe40WcKe0431ZuaXldS2/XtjY3NresXf3GiqMJaF1EvJQtjysKGeC1jXTnLYiSXHgcdr0htcTv/lApWKhuNOjiLoB7gvmM4K1kbp20S+Pj+Al7PgSkwSlCYLHkN4nJ+M07dolp+JMARcJykgJZKh17a9OLyRxQIUmHCvVRk6k3QRLzQinaETKxphMsR92jZU4IAqN5k+kcJDo/SgH0pTQsOp+nsiwYFSo8AznQHWAzXvTcT/vHas/Qs3YSKNRVktsiPOdQhnCQCe0xSovnIEwkM7dCMsAmDm1yK5gQ0PzLi6RxWkFOBd2elapXWRx5UAQHoAwQOAdVcANqoA4IeATP4BW8WU/Wi/Vufcxac1Y2sw/+wPr8ASGfln4=</latexit><latexit sha1_base64="4snGlhjxkE7I7roWcflso17LAcU=">ACBHicbVDLSgMxFM3UV62vUZfdBItQEctEBN0IRTcuK9gHtGPJpJk2NJMZkozQDrNw46+4caGIWz/CnX9j2s5CWw9cOJxzL/fe40WcKe0431ZuaXldS2/XtjY3NresXf3GiqMJaF1EvJQtjysKGeC1jXTnLYiSXHgcdr0htcTv/lApWKhuNOjiLoB7gvmM4K1kbp20S+Pj+Al7PgSkwSlCYLHkN4nJ+M07dolp+JMARcJykgJZKh17a9OLyRxQIUmHCvVRk6k3QRLzQinaETKxphMsR92jZU4IAqN5k+kcJDo/SgH0pTQsOp+nsiwYFSo8AznQHWAzXvTcT/vHas/Qs3YSKNRVktsiPOdQhnCQCe0xSovnIEwkM7dCMsAmDm1yK5gQ0PzLi6RxWkFOBd2elapXWRx5UAQHoAwQOAdVcANqoA4IeATP4BW8WU/Wi/Vufcxac1Y2sw/+wPr8ASGfln4=</latexit>

hw,b(x) = f(w|x + b)

<latexit sha1_base64="6EvxOqBfnKQlVFSWvkhLPUqI7OU=">ACMHicbVDLSgMxFM3UV62vqks3wSK0KGVGBN0IRe6rGAf0I5DJs20oZnMkGTUMswnufFTdKOgiFu/wrQdqbYeCJycy/3uOGjEplmq9GZm5+YXEpu5xbWV1b38hvbtVlEAlMajhgWi6SBJGOakpqhphoIg32Wk4fbPh37jlghJA36tBiGxfdTl1KMYKS05+YueE7d9pHquF98lB9BNij/f+6QET6FXnNg3cZtyRQRGLJlUwX3olpx8wSybI8BZYqWkAFJUnfxTuxPgyCdcYakbFlmqOwYCUxI0muHUkSItxHXdLSlCOfSDseHZzAPa10oBcI/biCI/V3R4x8KQe+qyuHa8pbyj+57Ui5Z3YMeVhpAjH40FexKAK4DA92KGCYMUGmiAsqN4V4h4SCOtUZE6HYE2fPEvqh2XLFtXR4XKWRpHFuyAXVAEFjgGFXAJqAGMHgAz+ANvBuPxovxYXyOSzNG2rMN/sD4+gYOxqoq</latexit><latexit sha1_base64="6EvxOqBfnKQlVFSWvkhLPUqI7OU=">ACMHicbVDLSgMxFM3UV62vqks3wSK0KGVGBN0IRe6rGAf0I5DJs20oZnMkGTUMswnufFTdKOgiFu/wrQdqbYeCJycy/3uOGjEplmq9GZm5+YXEpu5xbWV1b38hvbtVlEAlMajhgWi6SBJGOakpqhphoIg32Wk4fbPh37jlghJA36tBiGxfdTl1KMYKS05+YueE7d9pHquF98lB9BNij/f+6QET6FXnNg3cZtyRQRGLJlUwX3olpx8wSybI8BZYqWkAFJUnfxTuxPgyCdcYakbFlmqOwYCUxI0muHUkSItxHXdLSlCOfSDseHZzAPa10oBcI/biCI/V3R4x8KQe+qyuHa8pbyj+57Ui5Z3YMeVhpAjH40FexKAK4DA92KGCYMUGmiAsqN4V4h4SCOtUZE6HYE2fPEvqh2XLFtXR4XKWRpHFuyAXVAEFjgGFXAJqAGMHgAz+ANvBuPxovxYXyOSzNG2rMN/sD4+gYOxqoq</latexit><latexit sha1_base64="6EvxOqBfnKQlVFSWvkhLPUqI7OU=">ACMHicbVDLSgMxFM3UV62vqks3wSK0KGVGBN0IRe6rGAf0I5DJs20oZnMkGTUMswnufFTdKOgiFu/wrQdqbYeCJycy/3uOGjEplmq9GZm5+YXEpu5xbWV1b38hvbtVlEAlMajhgWi6SBJGOakpqhphoIg32Wk4fbPh37jlghJA36tBiGxfdTl1KMYKS05+YueE7d9pHquF98lB9BNij/f+6QET6FXnNg3cZtyRQRGLJlUwX3olpx8wSybI8BZYqWkAFJUnfxTuxPgyCdcYakbFlmqOwYCUxI0muHUkSItxHXdLSlCOfSDseHZzAPa10oBcI/biCI/V3R4x8KQe+qyuHa8pbyj+57Ui5Z3YMeVhpAjH40FexKAK4DA92KGCYMUGmiAsqN4V4h4SCOtUZE6HYE2fPEvqh2XLFtXR4XKWRpHFuyAXVAEFjgGFXAJqAGMHgAz+ANvBuPxovxYXyOSzNG2rMN/sD4+gYOxqoq</latexit><latexit sha1_base64="6EvxOqBfnKQlVFSWvkhLPUqI7OU=">ACMHicbVDLSgMxFM3UV62vqks3wSK0KGVGBN0IRe6rGAf0I5DJs20oZnMkGTUMswnufFTdKOgiFu/wrQdqbYeCJycy/3uOGjEplmq9GZm5+YXEpu5xbWV1b38hvbtVlEAlMajhgWi6SBJGOakpqhphoIg32Wk4fbPh37jlghJA36tBiGxfdTl1KMYKS05+YueE7d9pHquF98lB9BNij/f+6QET6FXnNg3cZtyRQRGLJlUwX3olpx8wSybI8BZYqWkAFJUnfxTuxPgyCdcYakbFlmqOwYCUxI0muHUkSItxHXdLSlCOfSDseHZzAPa10oBcI/biCI/V3R4x8KQe+qyuHa8pbyj+57Ui5Z3YMeVhpAjH40FexKAK4DA92KGCYMUGmiAsqN4V4h4SCOtUZE6HYE2fPEvqh2XLFtXR4XKWRpHFuyAXVAEFjgGFXAJqAGMHgAz+ANvBuPxovxYXyOSzNG2rMN/sD4+gYOxqoq</latexit>
slide-16
SLIDE 16

A neural network = many layers of classifiers all learned at once, some providing features for others

  • If we feed a vector of inputs through a bunch of logistic regression

functions, then we get a vector of outputs…

  • which we can feed into another logistic regression function
slide-17
SLIDE 17

Mathematical Notation

  • Input layer: x1, …, xd
  • Hidden layer 2: h(2)

1 , h(2) 2 , …, h(2) d2

· · ·

<latexit sha1_base64="C/ThE3LO0bsU2xdfe5suNCnJ9S4=">AB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NKPzM8pWxMh7xnaUJjbvx8fu2UnFklJHSthIkc/X3RE5jYyZxYDtjiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB0oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4ArlePLw=</latexit><latexit sha1_base64="C/ThE3LO0bsU2xdfe5suNCnJ9S4=">AB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NKPzM8pWxMh7xnaUJjbvx8fu2UnFklJHSthIkc/X3RE5jYyZxYDtjiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB0oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4ArlePLw=</latexit><latexit sha1_base64="C/ThE3LO0bsU2xdfe5suNCnJ9S4=">AB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NKPzM8pWxMh7xnaUJjbvx8fu2UnFklJHSthIkc/X3RE5jYyZxYDtjiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB0oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4ArlePLw=</latexit><latexit sha1_base64="C/ThE3LO0bsU2xdfe5suNCnJ9S4=">AB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NKPzM8pWxMh7xnaUJjbvx8fu2UnFklJHSthIkc/X3RE5jYyZxYDtjiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB0oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4ArlePLw=</latexit>

h(2)

1

= f(W (2)

1,1 h(1) 1

+ W (2)

1,2 h(1) 2

+ . . . + W (2)

1,d1h(1) d1 + b(2) 1 )

<latexit sha1_base64="a67o9ygnsA1DfBnoZrjOG8wnOw=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZVMEvQhFLx4r2A9o65LNZtvQ7AdJVihL/6QnPfhXxGy7aj8cCLzPjNMnEizqSyrA/DzG1t7+QLu8W90v7BYfnouCPDWBDaJiEPRc/BknIW0LZitNeJCj2HU67zuQh5d1XKiQLg2c1jejQx6OAeYxgpS27LMYvSbVRm9kI3kGv2s2yBF2hWYrQHF3CJdD4BQ0NBtwNlVytcO2/5iRNHZ+5tTscsWqW/OAmwJlogKyaNnlt4EbktingSIcS9lHVqSGCRaKEU5nxUEsaYTJBI9oX8sA+1QOk/luZvBCOy70QqFPoODcXe5IsC/l1Hd0pY/VWK6z1PyP9WPl3Q4TFkSxogFZDPJiDlUI0VDlwlKFJ9qgYlg+q6QjLHAROnvKOoloPUnb4pOo46sOnq6rjTvs3UwBk4B1WAwA1ogkfQAm1AwDv4MvJGwfg0c2bRLC1KTSPrOQErYZ5+A73GrVI=</latexit><latexit sha1_base64="a67o9ygnsA1DfBnoZrjOG8wnOw=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZVMEvQhFLx4r2A9o65LNZtvQ7AdJVihL/6QnPfhXxGy7aj8cCLzPjNMnEizqSyrA/DzG1t7+QLu8W90v7BYfnouCPDWBDaJiEPRc/BknIW0LZitNeJCj2HU67zuQh5d1XKiQLg2c1jejQx6OAeYxgpS27LMYvSbVRm9kI3kGv2s2yBF2hWYrQHF3CJdD4BQ0NBtwNlVytcO2/5iRNHZ+5tTscsWqW/OAmwJlogKyaNnlt4EbktingSIcS9lHVqSGCRaKEU5nxUEsaYTJBI9oX8sA+1QOk/luZvBCOy70QqFPoODcXe5IsC/l1Hd0pY/VWK6z1PyP9WPl3Q4TFkSxogFZDPJiDlUI0VDlwlKFJ9qgYlg+q6QjLHAROnvKOoloPUnb4pOo46sOnq6rjTvs3UwBk4B1WAwA1ogkfQAm1AwDv4MvJGwfg0c2bRLC1KTSPrOQErYZ5+A73GrVI=</latexit><latexit sha1_base64="a67o9ygnsA1DfBnoZrjOG8wnOw=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZVMEvQhFLx4r2A9o65LNZtvQ7AdJVihL/6QnPfhXxGy7aj8cCLzPjNMnEizqSyrA/DzG1t7+QLu8W90v7BYfnouCPDWBDaJiEPRc/BknIW0LZitNeJCj2HU67zuQh5d1XKiQLg2c1jejQx6OAeYxgpS27LMYvSbVRm9kI3kGv2s2yBF2hWYrQHF3CJdD4BQ0NBtwNlVytcO2/5iRNHZ+5tTscsWqW/OAmwJlogKyaNnlt4EbktingSIcS9lHVqSGCRaKEU5nxUEsaYTJBI9oX8sA+1QOk/luZvBCOy70QqFPoODcXe5IsC/l1Hd0pY/VWK6z1PyP9WPl3Q4TFkSxogFZDPJiDlUI0VDlwlKFJ9qgYlg+q6QjLHAROnvKOoloPUnb4pOo46sOnq6rjTvs3UwBk4B1WAwA1ogkfQAm1AwDv4MvJGwfg0c2bRLC1KTSPrOQErYZ5+A73GrVI=</latexit><latexit sha1_base64="a67o9ygnsA1DfBnoZrjOG8wnOw=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZVMEvQhFLx4r2A9o65LNZtvQ7AdJVihL/6QnPfhXxGy7aj8cCLzPjNMnEizqSyrA/DzG1t7+QLu8W90v7BYfnouCPDWBDaJiEPRc/BknIW0LZitNeJCj2HU67zuQh5d1XKiQLg2c1jejQx6OAeYxgpS27LMYvSbVRm9kI3kGv2s2yBF2hWYrQHF3CJdD4BQ0NBtwNlVytcO2/5iRNHZ+5tTscsWqW/OAmwJlogKyaNnlt4EbktingSIcS9lHVqSGCRaKEU5nxUEsaYTJBI9oX8sA+1QOk/luZvBCOy70QqFPoODcXe5IsC/l1Hd0pY/VWK6z1PyP9WPl3Q4TFkSxogFZDPJiDlUI0VDlwlKFJ9qgYlg+q6QjLHAROnvKOoloPUnb4pOo46sOnq6rjTvs3UwBk4B1WAwA1ogkfQAm1AwDv4MvJGwfg0c2bRLC1KTSPrOQErYZ5+A73GrVI=</latexit>

h(2)

2

= f(W (2)

2,1 h(1) 1

+ W (2)

2,2 h(1) 2

+ . . . + W (2)

2,d1h(1) d1 + b(2) 2 )

<latexit sha1_base64="w4Ns9nbRosQqD3EzcxhGPhYI7VU=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZbMIehGKXjxWsB/Q1iWbzbah2Q+SrFCW/klPevCviNl2/ah1IPDO+8wycSNOZPKst4Ms7CxuVUsbZd3Krt7+9WDw6MEkFoh0Q8En0XS8pZSDuKU7saA4cDntudO7jPeqZAsCh/VLKajAI9D5jOClbacqpg8pXW7MXdseAP9ei/PUvsCzTOEdILgOfwF7G9gazDkXqTkaoXn/DSnWaKx+zWn4VRrVtNaBFwXKBc1kEfbqb4MvYgkAQ0V4VjKAbJiNUqxUIxwOi8PE0ljTKZ4TAdahjigcpQudjOHZ9rxoB8JfUIF+7vjhQHUs4CV1cGWE3kX5aZ/7FBovzrUcrCOFE0JMtBfsKhimC2aOgxQYniMy0wEUzfFZIJFpgo/R1lvQT098nroms3kdVED5e1m2+jhI4AaegDhC4Ai1wD9qgAwh4BR9G0SgZ72bBLJuVZalp5D1HYCXM40/Gfq1X</latexit><latexit sha1_base64="w4Ns9nbRosQqD3EzcxhGPhYI7VU=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZbMIehGKXjxWsB/Q1iWbzbah2Q+SrFCW/klPevCviNl2/ah1IPDO+8wycSNOZPKst4Ms7CxuVUsbZd3Krt7+9WDw6MEkFoh0Q8En0XS8pZSDuKU7saA4cDntudO7jPeqZAsCh/VLKajAI9D5jOClbacqpg8pXW7MXdseAP9ei/PUvsCzTOEdILgOfwF7G9gazDkXqTkaoXn/DSnWaKx+zWn4VRrVtNaBFwXKBc1kEfbqb4MvYgkAQ0V4VjKAbJiNUqxUIxwOi8PE0ljTKZ4TAdahjigcpQudjOHZ9rxoB8JfUIF+7vjhQHUs4CV1cGWE3kX5aZ/7FBovzrUcrCOFE0JMtBfsKhimC2aOgxQYniMy0wEUzfFZIJFpgo/R1lvQT098nroms3kdVED5e1m2+jhI4AaegDhC4Ai1wD9qgAwh4BR9G0SgZ72bBLJuVZalp5D1HYCXM40/Gfq1X</latexit><latexit sha1_base64="w4Ns9nbRosQqD3EzcxhGPhYI7VU=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZbMIehGKXjxWsB/Q1iWbzbah2Q+SrFCW/klPevCviNl2/ah1IPDO+8wycSNOZPKst4Ms7CxuVUsbZd3Krt7+9WDw6MEkFoh0Q8En0XS8pZSDuKU7saA4cDntudO7jPeqZAsCh/VLKajAI9D5jOClbacqpg8pXW7MXdseAP9ei/PUvsCzTOEdILgOfwF7G9gazDkXqTkaoXn/DSnWaKx+zWn4VRrVtNaBFwXKBc1kEfbqb4MvYgkAQ0V4VjKAbJiNUqxUIxwOi8PE0ljTKZ4TAdahjigcpQudjOHZ9rxoB8JfUIF+7vjhQHUs4CV1cGWE3kX5aZ/7FBovzrUcrCOFE0JMtBfsKhimC2aOgxQYniMy0wEUzfFZIJFpgo/R1lvQT098nroms3kdVED5e1m2+jhI4AaegDhC4Ai1wD9qgAwh4BR9G0SgZ72bBLJuVZalp5D1HYCXM40/Gfq1X</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQvEKMXyw=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odBu3wMYA6nMFXEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="Qjb73Ct4Q2f2fZP+NO8a8FTJnvM=">ACUHicbZFdS8MwFIbTt2cU6u3gSHsKGMpjd6IwjeDnBfcA2S5qmWzD9IEmFUfonvdIL/4qYblX34YHAe85zDid54yWcSWXbH4Z2dndq9b26weNw6Nj6TRl3EqCO2RmMdi6GFJOYtoTzHF6TARFIcepwPv5b7g1cqJIujJzVP6CTE04gFjGClS64lZs9Zy2nrgNvYdAalFnmXKG8QEgnCF7CFeD8AkeDMfdjJdc7fPdvOCsSjb2fPW3XatodexFwW6BSNEZXd6G/sxSUMaKcKxlCNkJ2qSYaEY4TSvj1NJE0xe8JSOtIxwSOUkW3iTwtd8WEQC30iBRfV1YkMh1LOQ093hljN5CYriv+xUaqCm0nGoiRVNCLRUHKoYphYT0maBE8bkWmAim7wrJDAtMlP6OujYBbT5W/SdDrI76NEGNXAGzkELIHAN7sAD6IeIOAdfBlVo2Z8mhWzvrTLNErfTsFamI1vSaKskA=</latexit><latexit sha1_base64="Qjb73Ct4Q2f2fZP+NO8a8FTJnvM=">ACUHicbZFdS8MwFIbTt2cU6u3gSHsKGMpjd6IwjeDnBfcA2S5qmWzD9IEmFUfonvdIL/4qYblX34YHAe85zDid54yWcSWXbH4Z2dndq9b26weNw6Nj6TRl3EqCO2RmMdi6GFJOYtoTzHF6TARFIcepwPv5b7g1cqJIujJzVP6CTE04gFjGClS64lZs9Zy2nrgNvYdAalFnmXKG8QEgnCF7CFeD8AkeDMfdjJdc7fPdvOCsSjb2fPW3XatodexFwW6BSNEZXd6G/sxSUMaKcKxlCNkJ2qSYaEY4TSvj1NJE0xe8JSOtIxwSOUkW3iTwtd8WEQC30iBRfV1YkMh1LOQ093hljN5CYriv+xUaqCm0nGoiRVNCLRUHKoYphYT0maBE8bkWmAim7wrJDAtMlP6OujYBbT5W/SdDrI76NEGNXAGzkELIHAN7sAD6IeIOAdfBlVo2Z8mhWzvrTLNErfTsFamI1vSaKskA=</latexit><latexit sha1_base64="WdR2VpbOjV0J9tna7SL9E01jAo=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZbMXvQhFLx4r2A9o65LNZtg9oMkK5Slf9KTHvwrYrZdtR8OBN5nxkmbgxZ1JZ1odhFra2d4ql3fJeZf/gsHp03JVRIgjtkIhHou9iSTkLaUcxWk/FhQHLqc9+U+471XKiSLwic1jekowOQ+YxgpS2nKibPad1uzBwb3kK/3suz1L5CswhnSB4CZeA/QtsDYbci5RcrfCcv+Y0SzR2f+Y0nGrNalrzgJsC5aIG8mg71behF5EkoKEiHEs5QFasRikWihFOZ+VhImMyQse04GWIQ6oHKXz3czghXY86EdCn1DBubvckeJAymng6soAq4lcZ5n5Hxskyr8ZpSyME0VDshjkJxyqCGaLh4TlCg+1QITwfRdIZlgYnS31HWS0DrT94UXbuJrCZ6tGqtu3wdJXAGzkEdIHANWuABtEHEPAOvoyiUTI+zYJZNiuLUtPIe07ASpin38U+rVM=</latexit><latexit sha1_base64="w4Ns9nbRosQqD3EzcxhGPhYI7VU=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZbMIehGKXjxWsB/Q1iWbzbah2Q+SrFCW/klPevCviNl2/ah1IPDO+8wycSNOZPKst4Ms7CxuVUsbZd3Krt7+9WDw6MEkFoh0Q8En0XS8pZSDuKU7saA4cDntudO7jPeqZAsCh/VLKajAI9D5jOClbacqpg8pXW7MXdseAP9ei/PUvsCzTOEdILgOfwF7G9gazDkXqTkaoXn/DSnWaKx+zWn4VRrVtNaBFwXKBc1kEfbqb4MvYgkAQ0V4VjKAbJiNUqxUIxwOi8PE0ljTKZ4TAdahjigcpQudjOHZ9rxoB8JfUIF+7vjhQHUs4CV1cGWE3kX5aZ/7FBovzrUcrCOFE0JMtBfsKhimC2aOgxQYniMy0wEUzfFZIJFpgo/R1lvQT098nroms3kdVED5e1m2+jhI4AaegDhC4Ai1wD9qgAwh4BR9G0SgZ72bBLJuVZalp5D1HYCXM40/Gfq1X</latexit><latexit sha1_base64="w4Ns9nbRosQqD3EzcxhGPhYI7VU=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZbMIehGKXjxWsB/Q1iWbzbah2Q+SrFCW/klPevCviNl2/ah1IPDO+8wycSNOZPKst4Ms7CxuVUsbZd3Krt7+9WDw6MEkFoh0Q8En0XS8pZSDuKU7saA4cDntudO7jPeqZAsCh/VLKajAI9D5jOClbacqpg8pXW7MXdseAP9ei/PUvsCzTOEdILgOfwF7G9gazDkXqTkaoXn/DSnWaKx+zWn4VRrVtNaBFwXKBc1kEfbqb4MvYgkAQ0V4VjKAbJiNUqxUIxwOi8PE0ljTKZ4TAdahjigcpQudjOHZ9rxoB8JfUIF+7vjhQHUs4CV1cGWE3kX5aZ/7FBovzrUcrCOFE0JMtBfsKhimC2aOgxQYniMy0wEUzfFZIJFpgo/R1lvQT098nroms3kdVED5e1m2+jhI4AaegDhC4Ai1wD9qgAwh4BR9G0SgZ72bBLJuVZalp5D1HYCXM40/Gfq1X</latexit><latexit sha1_base64="w4Ns9nbRosQqD3EzcxhGPhYI7VU=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZbMIehGKXjxWsB/Q1iWbzbah2Q+SrFCW/klPevCviNl2/ah1IPDO+8wycSNOZPKst4Ms7CxuVUsbZd3Krt7+9WDw6MEkFoh0Q8En0XS8pZSDuKU7saA4cDntudO7jPeqZAsCh/VLKajAI9D5jOClbacqpg8pXW7MXdseAP9ei/PUvsCzTOEdILgOfwF7G9gazDkXqTkaoXn/DSnWaKx+zWn4VRrVtNaBFwXKBc1kEfbqb4MvYgkAQ0V4VjKAbJiNUqxUIxwOi8PE0ljTKZ4TAdahjigcpQudjOHZ9rxoB8JfUIF+7vjhQHUs4CV1cGWE3kX5aZ/7FBovzrUcrCOFE0JMtBfsKhimC2aOgxQYniMy0wEUzfFZIJFpgo/R1lvQT098nroms3kdVED5e1m2+jhI4AaegDhC4Ai1wD9qgAwh4BR9G0SgZ72bBLJuVZalp5D1HYCXM40/Gfq1X</latexit><latexit sha1_base64="w4Ns9nbRosQqD3EzcxhGPhYI7VU=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZbMIehGKXjxWsB/Q1iWbzbah2Q+SrFCW/klPevCviNl2/ah1IPDO+8wycSNOZPKst4Ms7CxuVUsbZd3Krt7+9WDw6MEkFoh0Q8En0XS8pZSDuKU7saA4cDntudO7jPeqZAsCh/VLKajAI9D5jOClbacqpg8pXW7MXdseAP9ei/PUvsCzTOEdILgOfwF7G9gazDkXqTkaoXn/DSnWaKx+zWn4VRrVtNaBFwXKBc1kEfbqb4MvYgkAQ0V4VjKAbJiNUqxUIxwOi8PE0ljTKZ4TAdahjigcpQudjOHZ9rxoB8JfUIF+7vjhQHUs4CV1cGWE3kX5aZ/7FBovzrUcrCOFE0JMtBfsKhimC2aOgxQYniMy0wEUzfFZIJFpgo/R1lvQT098nroms3kdVED5e1m2+jhI4AaegDhC4Ai1wD9qgAwh4BR9G0SgZ72bBLJuVZalp5D1HYCXM40/Gfq1X</latexit><latexit sha1_base64="w4Ns9nbRosQqD3EzcxhGPhYI7VU=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZbMIehGKXjxWsB/Q1iWbzbah2Q+SrFCW/klPevCviNl2/ah1IPDO+8wycSNOZPKst4Ms7CxuVUsbZd3Krt7+9WDw6MEkFoh0Q8En0XS8pZSDuKU7saA4cDntudO7jPeqZAsCh/VLKajAI9D5jOClbacqpg8pXW7MXdseAP9ei/PUvsCzTOEdILgOfwF7G9gazDkXqTkaoXn/DSnWaKx+zWn4VRrVtNaBFwXKBc1kEfbqb4MvYgkAQ0V4VjKAbJiNUqxUIxwOi8PE0ljTKZ4TAdahjigcpQudjOHZ9rxoB8JfUIF+7vjhQHUs4CV1cGWE3kX5aZ/7FBovzrUcrCOFE0JMtBfsKhimC2aOgxQYniMy0wEUzfFZIJFpgo/R1lvQT098nroms3kdVED5e1m2+jhI4AaegDhC4Ai1wD9qgAwh4BR9G0SgZ72bBLJuVZalp5D1HYCXM40/Gfq1X</latexit><latexit sha1_base64="w4Ns9nbRosQqD3EzcxhGPhYI7VU=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZbMIehGKXjxWsB/Q1iWbzbah2Q+SrFCW/klPevCviNl2/ah1IPDO+8wycSNOZPKst4Ms7CxuVUsbZd3Krt7+9WDw6MEkFoh0Q8En0XS8pZSDuKU7saA4cDntudO7jPeqZAsCh/VLKajAI9D5jOClbacqpg8pXW7MXdseAP9ei/PUvsCzTOEdILgOfwF7G9gazDkXqTkaoXn/DSnWaKx+zWn4VRrVtNaBFwXKBc1kEfbqb4MvYgkAQ0V4VjKAbJiNUqxUIxwOi8PE0ljTKZ4TAdahjigcpQudjOHZ9rxoB8JfUIF+7vjhQHUs4CV1cGWE3kX5aZ/7FBovzrUcrCOFE0JMtBfsKhimC2aOgxQYniMy0wEUzfFZIJFpgo/R1lvQT098nroms3kdVED5e1m2+jhI4AaegDhC4Ai1wD9qgAwh4BR9G0SgZ72bBLJuVZalp5D1HYCXM40/Gfq1X</latexit>
  • Output layer:

y = σ(w(o)

1 h(2) 1

+ w(o)

2 h(2) 2

+ . . . + w(o)

d2 h(2) d2 + b(o))

<latexit sha1_base64="bOumy7ml4JKyWoAXDVdeW0BwETI=">ACT3icbZHNS8MwGMbT+bXNr6lHL8EhbAxGWwS9CEMvHie4D9hmSdN0C6ZNSVJlP2HXvTmv+HFgyJmXR1z84XAL8/zviR54kaMSmWab0ZubX1jcytfKG7v7O7tlw4O25LHApMW5oyLroskYTQkLUVI91IEBS4jHTch+up3kQlIe3qlxRAYBGobUpxgpLTklfwvYV/SYAqT/dJhVcnjgVHmuyUavBXteqrdU+87iSC3biOfZk3jLb1aA7c6tOqWzWzbTgKlgZlEFWTaf02vc4jgMSKsyQlD3LjNQgQUJRzMik2I8liRB+QEPS0xigMhBkuYxgada8aDPhV6hgqm6OJGgQMpx4OrOAKmRXPam4n9eL1b+xSChYRQrEuLZQX7MoOJwGi70qCBYsbEGhAXVd4V4hATCSn9BUYdgLT95Fdp23TLr1u1ZuXGVxZEHx+AEVIAFzkED3IAmaAEMnsE7+ARfxovxYXznstackcER+FO5wg8T4K6n</latexit><latexit sha1_base64="bOumy7ml4JKyWoAXDVdeW0BwETI=">ACT3icbZHNS8MwGMbT+bXNr6lHL8EhbAxGWwS9CEMvHie4D9hmSdN0C6ZNSVJlP2HXvTmv+HFgyJmXR1z84XAL8/zviR54kaMSmWab0ZubX1jcytfKG7v7O7tlw4O25LHApMW5oyLroskYTQkLUVI91IEBS4jHTch+up3kQlIe3qlxRAYBGobUpxgpLTklfwvYV/SYAqT/dJhVcnjgVHmuyUavBXteqrdU+87iSC3biOfZk3jLb1aA7c6tOqWzWzbTgKlgZlEFWTaf02vc4jgMSKsyQlD3LjNQgQUJRzMik2I8liRB+QEPS0xigMhBkuYxgada8aDPhV6hgqm6OJGgQMpx4OrOAKmRXPam4n9eL1b+xSChYRQrEuLZQX7MoOJwGi70qCBYsbEGhAXVd4V4hATCSn9BUYdgLT95Fdp23TLr1u1ZuXGVxZEHx+AEVIAFzkED3IAmaAEMnsE7+ARfxovxYXznstackcER+FO5wg8T4K6n</latexit><latexit sha1_base64="bOumy7ml4JKyWoAXDVdeW0BwETI=">ACT3icbZHNS8MwGMbT+bXNr6lHL8EhbAxGWwS9CEMvHie4D9hmSdN0C6ZNSVJlP2HXvTmv+HFgyJmXR1z84XAL8/zviR54kaMSmWab0ZubX1jcytfKG7v7O7tlw4O25LHApMW5oyLroskYTQkLUVI91IEBS4jHTch+up3kQlIe3qlxRAYBGobUpxgpLTklfwvYV/SYAqT/dJhVcnjgVHmuyUavBXteqrdU+87iSC3biOfZk3jLb1aA7c6tOqWzWzbTgKlgZlEFWTaf02vc4jgMSKsyQlD3LjNQgQUJRzMik2I8liRB+QEPS0xigMhBkuYxgada8aDPhV6hgqm6OJGgQMpx4OrOAKmRXPam4n9eL1b+xSChYRQrEuLZQX7MoOJwGi70qCBYsbEGhAXVd4V4hATCSn9BUYdgLT95Fdp23TLr1u1ZuXGVxZEHx+AEVIAFzkED3IAmaAEMnsE7+ARfxovxYXznstackcER+FO5wg8T4K6n</latexit><latexit sha1_base64="bOumy7ml4JKyWoAXDVdeW0BwETI=">ACT3icbZHNS8MwGMbT+bXNr6lHL8EhbAxGWwS9CEMvHie4D9hmSdN0C6ZNSVJlP2HXvTmv+HFgyJmXR1z84XAL8/zviR54kaMSmWab0ZubX1jcytfKG7v7O7tlw4O25LHApMW5oyLroskYTQkLUVI91IEBS4jHTch+up3kQlIe3qlxRAYBGobUpxgpLTklfwvYV/SYAqT/dJhVcnjgVHmuyUavBXteqrdU+87iSC3biOfZk3jLb1aA7c6tOqWzWzbTgKlgZlEFWTaf02vc4jgMSKsyQlD3LjNQgQUJRzMik2I8liRB+QEPS0xigMhBkuYxgada8aDPhV6hgqm6OJGgQMpx4OrOAKmRXPam4n9eL1b+xSChYRQrEuLZQX7MoOJwGi70qCBYsbEGhAXVd4V4hATCSn9BUYdgLT95Fdp23TLr1u1ZuXGVxZEHx+AEVIAFzkED3IAmaAEMnsE7+ARfxovxYXznstackcER+FO5wg8T4K6n</latexit>
  • Hidden layer 1: h(1)

1 , h(1) 2 , …, h(1) d1

· · ·

<latexit sha1_base64="C/ThE3LO0bsU2xdfe5suNCnJ9S4=">AB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NKPzM8pWxMh7xnaUJjbvx8fu2UnFklJHSthIkc/X3RE5jYyZxYDtjiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB0oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4ArlePLw=</latexit><latexit sha1_base64="C/ThE3LO0bsU2xdfe5suNCnJ9S4=">AB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NKPzM8pWxMh7xnaUJjbvx8fu2UnFklJHSthIkc/X3RE5jYyZxYDtjiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB0oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4ArlePLw=</latexit><latexit sha1_base64="C/ThE3LO0bsU2xdfe5suNCnJ9S4=">AB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NKPzM8pWxMh7xnaUJjbvx8fu2UnFklJHSthIkc/X3RE5jYyZxYDtjiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB0oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4ArlePLw=</latexit><latexit sha1_base64="C/ThE3LO0bsU2xdfe5suNCnJ9S4=">AB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NKPzM8pWxMh7xnaUJjbvx8fu2UnFklJHSthIkc/X3RE5jYyZxYDtjiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB0oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4ArlePLw=</latexit>

h(1)

1

= f(W (1)

1,1 + W (1) 1,2 x2 + . . . + W (1) 1,d xd + b(1) 1 )

<latexit sha1_base64="xyKVt0I2x1mI6pfX2c5pytM/Ws=">ACQHicbVDLSgMxFM34rPU16tJNsAgtSpkUQRcKBTcuK9gHtHXIZDJtaOZBkhHLMJ/mxk9w59qNC0XcujJtR+jDA4Fz7n3JjlOxJlUlvVqLC2vrK6t5zbym1vbO7vm3n5DhrEgtE5CHoqWgyXlLKB1xRSnrUhQ7DucNp3B9chvPlAhWRjcqWFEuz7uBcxjBCst2Wazf58USm1EbyCXrGZVQk6RSk8gVN1JX20K1rqcDdUctZztedqyflbVrLNglW2xoCLBGWkADLUbPOl4Yk9mgCMdStpEVqW6ChWKE0zTfiSWNMBngHm1rGmCfym4yDiCFx1pxoRcKfQIFx+r0RIJ9KYe+ozt9rPpy3huJ/3ntWHkX3YQFUaxoQCYXeTGHKoSjNKHLBCWKDzXBRD9Vkj6WGCidOZ5HQKa/IiaVTKyCqj27NC9TKLIwcOwREoAgTOQRXcgBqoAwKewBv4AJ/Gs/FufBnfk9YlI5s5ADMwfn4BapKpuw=</latexit><latexit sha1_base64="xyKVt0I2x1mI6pfX2c5pytM/Ws=">ACQHicbVDLSgMxFM34rPU16tJNsAgtSpkUQRcKBTcuK9gHtHXIZDJtaOZBkhHLMJ/mxk9w59qNC0XcujJtR+jDA4Fz7n3JjlOxJlUlvVqLC2vrK6t5zbym1vbO7vm3n5DhrEgtE5CHoqWgyXlLKB1xRSnrUhQ7DucNp3B9chvPlAhWRjcqWFEuz7uBcxjBCst2Wazf58USm1EbyCXrGZVQk6RSk8gVN1JX20K1rqcDdUctZztedqyflbVrLNglW2xoCLBGWkADLUbPOl4Yk9mgCMdStpEVqW6ChWKE0zTfiSWNMBngHm1rGmCfym4yDiCFx1pxoRcKfQIFx+r0RIJ9KYe+ozt9rPpy3huJ/3ntWHkX3YQFUaxoQCYXeTGHKoSjNKHLBCWKDzXBRD9Vkj6WGCidOZ5HQKa/IiaVTKyCqj27NC9TKLIwcOwREoAgTOQRXcgBqoAwKewBv4AJ/Gs/FufBnfk9YlI5s5ADMwfn4BapKpuw=</latexit><latexit sha1_base64="xyKVt0I2x1mI6pfX2c5pytM/Ws=">ACQHicbVDLSgMxFM34rPU16tJNsAgtSpkUQRcKBTcuK9gHtHXIZDJtaOZBkhHLMJ/mxk9w59qNC0XcujJtR+jDA4Fz7n3JjlOxJlUlvVqLC2vrK6t5zbym1vbO7vm3n5DhrEgtE5CHoqWgyXlLKB1xRSnrUhQ7DucNp3B9chvPlAhWRjcqWFEuz7uBcxjBCst2Wazf58USm1EbyCXrGZVQk6RSk8gVN1JX20K1rqcDdUctZztedqyflbVrLNglW2xoCLBGWkADLUbPOl4Yk9mgCMdStpEVqW6ChWKE0zTfiSWNMBngHm1rGmCfym4yDiCFx1pxoRcKfQIFx+r0RIJ9KYe+ozt9rPpy3huJ/3ntWHkX3YQFUaxoQCYXeTGHKoSjNKHLBCWKDzXBRD9Vkj6WGCidOZ5HQKa/IiaVTKyCqj27NC9TKLIwcOwREoAgTOQRXcgBqoAwKewBv4AJ/Gs/FufBnfk9YlI5s5ADMwfn4BapKpuw=</latexit><latexit sha1_base64="xyKVt0I2x1mI6pfX2c5pytM/Ws=">ACQHicbVDLSgMxFM34rPU16tJNsAgtSpkUQRcKBTcuK9gHtHXIZDJtaOZBkhHLMJ/mxk9w59qNC0XcujJtR+jDA4Fz7n3JjlOxJlUlvVqLC2vrK6t5zbym1vbO7vm3n5DhrEgtE5CHoqWgyXlLKB1xRSnrUhQ7DucNp3B9chvPlAhWRjcqWFEuz7uBcxjBCst2Wazf58USm1EbyCXrGZVQk6RSk8gVN1JX20K1rqcDdUctZztedqyflbVrLNglW2xoCLBGWkADLUbPOl4Yk9mgCMdStpEVqW6ChWKE0zTfiSWNMBngHm1rGmCfym4yDiCFx1pxoRcKfQIFx+r0RIJ9KYe+ozt9rPpy3huJ/3ntWHkX3YQFUaxoQCYXeTGHKoSjNKHLBCWKDzXBRD9Vkj6WGCidOZ5HQKa/IiaVTKyCqj27NC9TKLIwcOwREoAgTOQRXcgBqoAwKewBv4AJ/Gs/FufBnfk9YlI5s5ADMwfn4BapKpuw=</latexit>

h(1)

2

= f(W (1)

2,1 + W (1) 2,2 x2 + . . . + W (1) 2,d xd + b(1) 2 )

<latexit sha1_base64="OoxWd30Tt1rq58Rry91hetKvsxo=">ACQHicbVDLSgMxFM3UV62vqks3wSK0VMrMIOhCoeDGZQX7gHYcMplMG5p5kGTEMsynufET3Ll240IRt65M2xH68ELg3HPOvUmOEzEqpK6/armV1bX1jfxmYWt7Z3evuH/QEmHMWnikIW84yBGA1IU1LJSCfiBPkOI21neD3W2w+ECxoGd3IUEctH/YB6FCOpKLvYHtwnZaOS2ia8gl65nXWJeWqksApnejN9VKYq7DE3lGJec5XmKsr5W1axiyW9pk8KLgMjAyWQVcMuvTcEMc+CSRmSIiuoUfShCXFDOSFnqxIBHCQ9QnXQUD5BNhJZMAUniGBd6IVcnkHDCzk4kyBdi5DvK6SM5EIvamPxP68bSu7ASGkSxJAGeXuTFDMoQjtOELuUESzZSAGFO1VshHiCOsFSZF1QIxuKXl0HLrBl6zbg9K9Uvszjy4AgcgzIwDmogxvQAE2AwRN4Ax/gU3vW3rUv7XtqzWnZzCGYK+3nF3L2qcA=</latexit><latexit sha1_base64="OoxWd30Tt1rq58Rry91hetKvsxo=">ACQHicbVDLSgMxFM3UV62vqks3wSK0VMrMIOhCoeDGZQX7gHYcMplMG5p5kGTEMsynufET3Ll240IRt65M2xH68ELg3HPOvUmOEzEqpK6/armV1bX1jfxmYWt7Z3evuH/QEmHMWnikIW84yBGA1IU1LJSCfiBPkOI21neD3W2w+ECxoGd3IUEctH/YB6FCOpKLvYHtwnZaOS2ia8gl65nXWJeWqksApnejN9VKYq7DE3lGJec5XmKsr5W1axiyW9pk8KLgMjAyWQVcMuvTcEMc+CSRmSIiuoUfShCXFDOSFnqxIBHCQ9QnXQUD5BNhJZMAUniGBd6IVcnkHDCzk4kyBdi5DvK6SM5EIvamPxP68bSu7ASGkSxJAGeXuTFDMoQjtOELuUESzZSAGFO1VshHiCOsFSZF1QIxuKXl0HLrBl6zbg9K9Uvszjy4AgcgzIwDmogxvQAE2AwRN4Ax/gU3vW3rUv7XtqzWnZzCGYK+3nF3L2qcA=</latexit><latexit sha1_base64="OoxWd30Tt1rq58Rry91hetKvsxo=">ACQHicbVDLSgMxFM3UV62vqks3wSK0VMrMIOhCoeDGZQX7gHYcMplMG5p5kGTEMsynufET3Ll240IRt65M2xH68ELg3HPOvUmOEzEqpK6/armV1bX1jfxmYWt7Z3evuH/QEmHMWnikIW84yBGA1IU1LJSCfiBPkOI21neD3W2w+ECxoGd3IUEctH/YB6FCOpKLvYHtwnZaOS2ia8gl65nXWJeWqksApnejN9VKYq7DE3lGJec5XmKsr5W1axiyW9pk8KLgMjAyWQVcMuvTcEMc+CSRmSIiuoUfShCXFDOSFnqxIBHCQ9QnXQUD5BNhJZMAUniGBd6IVcnkHDCzk4kyBdi5DvK6SM5EIvamPxP68bSu7ASGkSxJAGeXuTFDMoQjtOELuUESzZSAGFO1VshHiCOsFSZF1QIxuKXl0HLrBl6zbg9K9Uvszjy4AgcgzIwDmogxvQAE2AwRN4Ax/gU3vW3rUv7XtqzWnZzCGYK+3nF3L2qcA=</latexit><latexit sha1_base64="OoxWd30Tt1rq58Rry91hetKvsxo=">ACQHicbVDLSgMxFM3UV62vqks3wSK0VMrMIOhCoeDGZQX7gHYcMplMG5p5kGTEMsynufET3Ll240IRt65M2xH68ELg3HPOvUmOEzEqpK6/armV1bX1jfxmYWt7Z3evuH/QEmHMWnikIW84yBGA1IU1LJSCfiBPkOI21neD3W2w+ECxoGd3IUEctH/YB6FCOpKLvYHtwnZaOS2ia8gl65nXWJeWqksApnejN9VKYq7DE3lGJec5XmKsr5W1axiyW9pk8KLgMjAyWQVcMuvTcEMc+CSRmSIiuoUfShCXFDOSFnqxIBHCQ9QnXQUD5BNhJZMAUniGBd6IVcnkHDCzk4kyBdi5DvK6SM5EIvamPxP68bSu7ASGkSxJAGeXuTFDMoQjtOELuUESzZSAGFO1VshHiCOsFSZF1QIxuKXl0HLrBl6zbg9K9Uvszjy4AgcgzIwDmogxvQAE2AwRN4Ax/gU3vW3rUv7XtqzWnZzCGYK+3nF3L2qcA=</latexit>
slide-18
SLIDE 18

Matrix Notation

  • Input layer:
  • Hidden layer 1:

h1 = f(W(1)x + b(1)) ∈ Rd1

<latexit sha1_base64="VM7VTSrhdxVPv8H7p1W7pTPe9Nc=">ACPXicbVDLSgMxFM3UV62vqks3wSK0CGUigm6EohuXVfqCtg6ZNOGZjJDkhHLMD/mxn9w586NC0XcujVtp6CtBwLnMvufe4IWdK2/aLlVlaXldy67nNja3tnfyu3sNFUS0DoJeCBbLlaUM0HrmlOW6Gk2Hc5brDq7HfvKdSsUDU9CikXR/3BfMYwdpITr7W8bEeuF48SBwEL6BXnAnN5C4uolIyqx8SeAxnhZuaJdhIlXd+NaoPQclTr5gl+0J4CJBKSmAFUn/9zpBSTyqdCEY6XayA51N8ZSM8JpkutEioaYDHGftg0V2KeqG0+uT+CRUXrQC6R5QsOJ+nsixr5SI981neM91bw3Fv/z2pH2zrsxE2GkqSDTj7yIQx3AcZSwxyQlmo8MwUQysyskAywx0SbwnAkBzZ+8SBonZWSX0c1poXKZxpEFB+AQFAECZ6ACrkEV1AEBj+AVvIMP68l6sz6tr2lrxkpn9sEfWN8/uPeuVA=</latexit><latexit sha1_base64="VM7VTSrhdxVPv8H7p1W7pTPe9Nc=">ACPXicbVDLSgMxFM3UV62vqks3wSK0CGUigm6EohuXVfqCtg6ZNOGZjJDkhHLMD/mxn9w586NC0XcujVtp6CtBwLnMvufe4IWdK2/aLlVlaXldy67nNja3tnfyu3sNFUS0DoJeCBbLlaUM0HrmlOW6Gk2Hc5brDq7HfvKdSsUDU9CikXR/3BfMYwdpITr7W8bEeuF48SBwEL6BXnAnN5C4uolIyqx8SeAxnhZuaJdhIlXd+NaoPQclTr5gl+0J4CJBKSmAFUn/9zpBSTyqdCEY6XayA51N8ZSM8JpkutEioaYDHGftg0V2KeqG0+uT+CRUXrQC6R5QsOJ+nsixr5SI981neM91bw3Fv/z2pH2zrsxE2GkqSDTj7yIQx3AcZSwxyQlmo8MwUQysyskAywx0SbwnAkBzZ+8SBonZWSX0c1poXKZxpEFB+AQFAECZ6ACrkEV1AEBj+AVvIMP68l6sz6tr2lrxkpn9sEfWN8/uPeuVA=</latexit><latexit sha1_base64="VM7VTSrhdxVPv8H7p1W7pTPe9Nc=">ACPXicbVDLSgMxFM3UV62vqks3wSK0CGUigm6EohuXVfqCtg6ZNOGZjJDkhHLMD/mxn9w586NC0XcujVtp6CtBwLnMvufe4IWdK2/aLlVlaXldy67nNja3tnfyu3sNFUS0DoJeCBbLlaUM0HrmlOW6Gk2Hc5brDq7HfvKdSsUDU9CikXR/3BfMYwdpITr7W8bEeuF48SBwEL6BXnAnN5C4uolIyqx8SeAxnhZuaJdhIlXd+NaoPQclTr5gl+0J4CJBKSmAFUn/9zpBSTyqdCEY6XayA51N8ZSM8JpkutEioaYDHGftg0V2KeqG0+uT+CRUXrQC6R5QsOJ+nsixr5SI981neM91bw3Fv/z2pH2zrsxE2GkqSDTj7yIQx3AcZSwxyQlmo8MwUQysyskAywx0SbwnAkBzZ+8SBonZWSX0c1poXKZxpEFB+AQFAECZ6ACrkEV1AEBj+AVvIMP68l6sz6tr2lrxkpn9sEfWN8/uPeuVA=</latexit><latexit sha1_base64="VM7VTSrhdxVPv8H7p1W7pTPe9Nc=">ACPXicbVDLSgMxFM3UV62vqks3wSK0CGUigm6EohuXVfqCtg6ZNOGZjJDkhHLMD/mxn9w586NC0XcujVtp6CtBwLnMvufe4IWdK2/aLlVlaXldy67nNja3tnfyu3sNFUS0DoJeCBbLlaUM0HrmlOW6Gk2Hc5brDq7HfvKdSsUDU9CikXR/3BfMYwdpITr7W8bEeuF48SBwEL6BXnAnN5C4uolIyqx8SeAxnhZuaJdhIlXd+NaoPQclTr5gl+0J4CJBKSmAFUn/9zpBSTyqdCEY6XayA51N8ZSM8JpkutEioaYDHGftg0V2KeqG0+uT+CRUXrQC6R5QsOJ+nsixr5SI981neM91bw3Fv/z2pH2zrsxE2GkqSDTj7yIQx3AcZSwxyQlmo8MwUQysyskAywx0SbwnAkBzZ+8SBonZWSX0c1poXKZxpEFB+AQFAECZ6ACrkEV1AEBj+AVvIMP68l6sz6tr2lrxkpn9sEfWN8/uPeuVA=</latexit>

W(2) ∈ Rd2×d1, b(2) ∈ Rd2

<latexit sha1_base64="Nn6kKAXiHkWvk29ucTEHXmcjE8=">ACQHicfVDPS8MwGE39OevqUcvwSFMkNEOQY9DLx6nuHWw1pKm6RaWpiVJhVH6p3nxT/Dm2YsHRbx6Mtsq6CY+CDze9z2+l+cnjEplmk/GwuLS8spqa28vrG5tV3Z2e3IOBWYtHMYtH1kSMctJWVDHSTQRBkc+I7Q8vxnP7jghJY36jRglxI9TnNKQYKS15FduJkBr4YWbnt1mtcZRDh3I4Ff3sWouB14COohGRMPCs/Bh+O/x/HLlXqZp1cwI4T6yCVEGBld5dIYpxHhCjMkZc8yE+VmSCiKGcnLTipJgvAQ9UlPU450IDebFJDQ60EMIyFflzBifrTkaFIylHk681xTjk7G4t/zXqpCs/cjPIkVYTj6aEwZVDFcNwmDKgWLGRJgLqrNCPEACYaU7L+sSrNkvz5NOo26ZdevqpNo8L+ogX1wAGrAqegCS5BC7QBvfgGbyCN+PBeDHejY/p6oJRePbALxifX2aAr6o=</latexit><latexit sha1_base64="Nn6kKAXiHkWvk29ucTEHXmcjE8=">ACQHicfVDPS8MwGE39OevqUcvwSFMkNEOQY9DLx6nuHWw1pKm6RaWpiVJhVH6p3nxT/Dm2YsHRbx6Mtsq6CY+CDze9z2+l+cnjEplmk/GwuLS8spqa28vrG5tV3Z2e3IOBWYtHMYtH1kSMctJWVDHSTQRBkc+I7Q8vxnP7jghJY36jRglxI9TnNKQYKS15FduJkBr4YWbnt1mtcZRDh3I4Ff3sWouB14COohGRMPCs/Bh+O/x/HLlXqZp1cwI4T6yCVEGBld5dIYpxHhCjMkZc8yE+VmSCiKGcnLTipJgvAQ9UlPU450IDebFJDQ60EMIyFflzBifrTkaFIylHk681xTjk7G4t/zXqpCs/cjPIkVYTj6aEwZVDFcNwmDKgWLGRJgLqrNCPEACYaU7L+sSrNkvz5NOo26ZdevqpNo8L+ogX1wAGrAqegCS5BC7QBvfgGbyCN+PBeDHejY/p6oJRePbALxifX2aAr6o=</latexit><latexit sha1_base64="Nn6kKAXiHkWvk29ucTEHXmcjE8=">ACQHicfVDPS8MwGE39OevqUcvwSFMkNEOQY9DLx6nuHWw1pKm6RaWpiVJhVH6p3nxT/Dm2YsHRbx6Mtsq6CY+CDze9z2+l+cnjEplmk/GwuLS8spqa28vrG5tV3Z2e3IOBWYtHMYtH1kSMctJWVDHSTQRBkc+I7Q8vxnP7jghJY36jRglxI9TnNKQYKS15FduJkBr4YWbnt1mtcZRDh3I4Ff3sWouB14COohGRMPCs/Bh+O/x/HLlXqZp1cwI4T6yCVEGBld5dIYpxHhCjMkZc8yE+VmSCiKGcnLTipJgvAQ9UlPU450IDebFJDQ60EMIyFflzBifrTkaFIylHk681xTjk7G4t/zXqpCs/cjPIkVYTj6aEwZVDFcNwmDKgWLGRJgLqrNCPEACYaU7L+sSrNkvz5NOo26ZdevqpNo8L+ogX1wAGrAqegCS5BC7QBvfgGbyCN+PBeDHejY/p6oJRePbALxifX2aAr6o=</latexit><latexit sha1_base64="Nn6kKAXiHkWvk29ucTEHXmcjE8=">ACQHicfVDPS8MwGE39OevqUcvwSFMkNEOQY9DLx6nuHWw1pKm6RaWpiVJhVH6p3nxT/Dm2YsHRbx6Mtsq6CY+CDze9z2+l+cnjEplmk/GwuLS8spqa28vrG5tV3Z2e3IOBWYtHMYtH1kSMctJWVDHSTQRBkc+I7Q8vxnP7jghJY36jRglxI9TnNKQYKS15FduJkBr4YWbnt1mtcZRDh3I4Ff3sWouB14COohGRMPCs/Bh+O/x/HLlXqZp1cwI4T6yCVEGBld5dIYpxHhCjMkZc8yE+VmSCiKGcnLTipJgvAQ9UlPU450IDebFJDQ60EMIyFflzBifrTkaFIylHk681xTjk7G4t/zXqpCs/cjPIkVYTj6aEwZVDFcNwmDKgWLGRJgLqrNCPEACYaU7L+sSrNkvz5NOo26ZdevqpNo8L+ogX1wAGrAqegCS5BC7QBvfgGbyCN+PBeDHejY/p6oJRePbALxifX2aAr6o=</latexit>
  • Hidden layer 2:

h2 = f(W(2)h1 + b(2)) ∈ Rd2

<latexit sha1_base64="ti/zp5Hv7rAiTgW9sNoVe9BYoI=">ACP3icbZDNS8MwGMbT+TXn19Sjl+AQNoTRDkEvwtCLxynuA9Z0izdwtK0JKkwSv8zL/4L3rx68aCIV2+mWwdz84XAk9/zvuTN4aMSmWar0ZuZXVtfSO/Wdja3tndK+4ftGQCUyaOGCB6LhIEkY5aSqGOmEgiDfZaTtjq5Tv/1IhKQBv1fjkPR8NODUoxgpjZxiy/aRGrpePEycGryEXnkG2slDXK5VkrkGC57C2dXN7Aq0Kc+oG9p2ndqiVMsmVzUnBZWJkogawaTvHF7gc48glXmCEpu5YZql6MhKYkaRgR5KECI/QgHS15MgnshdP/p/AE0360AuEPlzBCZ2fiJEv5dh3dWe6p1z0Uvif142Ud9GLKQ8jRTiePuRFDKoApmHCPhUEKzbWAmFB9a4QD5FAWOnICzoEa/HLy6JVq1pm1bo9K9Wvsjy4AgcgzKwDmogxvQAE2AwRN4Ax/g03g23o0v43vamjOymUPwp4yfXwG6ruw=</latexit><latexit sha1_base64="ti/zp5Hv7rAiTgW9sNoVe9BYoI=">ACP3icbZDNS8MwGMbT+TXn19Sjl+AQNoTRDkEvwtCLxynuA9Z0izdwtK0JKkwSv8zL/4L3rx68aCIV2+mWwdz84XAk9/zvuTN4aMSmWar0ZuZXVtfSO/Wdja3tndK+4ftGQCUyaOGCB6LhIEkY5aSqGOmEgiDfZaTtjq5Tv/1IhKQBv1fjkPR8NODUoxgpjZxiy/aRGrpePEycGryEXnkG2slDXK5VkrkGC57C2dXN7Aq0Kc+oG9p2ndqiVMsmVzUnBZWJkogawaTvHF7gc48glXmCEpu5YZql6MhKYkaRgR5KECI/QgHS15MgnshdP/p/AE0360AuEPlzBCZ2fiJEv5dh3dWe6p1z0Uvif142Ud9GLKQ8jRTiePuRFDKoApmHCPhUEKzbWAmFB9a4QD5FAWOnICzoEa/HLy6JVq1pm1bo9K9Wvsjy4AgcgzKwDmogxvQAE2AwRN4Ax/g03g23o0v43vamjOymUPwp4yfXwG6ruw=</latexit><latexit sha1_base64="ti/zp5Hv7rAiTgW9sNoVe9BYoI=">ACP3icbZDNS8MwGMbT+TXn19Sjl+AQNoTRDkEvwtCLxynuA9Z0izdwtK0JKkwSv8zL/4L3rx68aCIV2+mWwdz84XAk9/zvuTN4aMSmWar0ZuZXVtfSO/Wdja3tndK+4ftGQCUyaOGCB6LhIEkY5aSqGOmEgiDfZaTtjq5Tv/1IhKQBv1fjkPR8NODUoxgpjZxiy/aRGrpePEycGryEXnkG2slDXK5VkrkGC57C2dXN7Aq0Kc+oG9p2ndqiVMsmVzUnBZWJkogawaTvHF7gc48glXmCEpu5YZql6MhKYkaRgR5KECI/QgHS15MgnshdP/p/AE0360AuEPlzBCZ2fiJEv5dh3dWe6p1z0Uvif142Ud9GLKQ8jRTiePuRFDKoApmHCPhUEKzbWAmFB9a4QD5FAWOnICzoEa/HLy6JVq1pm1bo9K9Wvsjy4AgcgzKwDmogxvQAE2AwRN4Ax/g03g23o0v43vamjOymUPwp4yfXwG6ruw=</latexit><latexit sha1_base64="ti/zp5Hv7rAiTgW9sNoVe9BYoI=">ACP3icbZDNS8MwGMbT+TXn19Sjl+AQNoTRDkEvwtCLxynuA9Z0izdwtK0JKkwSv8zL/4L3rx68aCIV2+mWwdz84XAk9/zvuTN4aMSmWar0ZuZXVtfSO/Wdja3tndK+4ftGQCUyaOGCB6LhIEkY5aSqGOmEgiDfZaTtjq5Tv/1IhKQBv1fjkPR8NODUoxgpjZxiy/aRGrpePEycGryEXnkG2slDXK5VkrkGC57C2dXN7Aq0Kc+oG9p2ndqiVMsmVzUnBZWJkogawaTvHF7gc48glXmCEpu5YZql6MhKYkaRgR5KECI/QgHS15MgnshdP/p/AE0360AuEPlzBCZ2fiJEv5dh3dWe6p1z0Uvif142Ud9GLKQ8jRTiePuRFDKoApmHCPhUEKzbWAmFB9a4QD5FAWOnICzoEa/HLy6JVq1pm1bo9K9Wvsjy4AgcgzKwDmogxvQAE2AwRN4Ax/g03g23o0v43vamjOymUPwp4yfXwG6ruw=</latexit>
  • Output layer:

y = σ(w(o) · h2 + b(o))

<latexit sha1_base64="xwf5NTK8Os9frFEQg9QWL1GKg9Q=">ACInicbVDLSsNAFJ34rPUVdelmsAgtQkmKoC6EohuXFewDmlgm0k7dJIJMxOlhHyLG3/FjQtFXQl+jJM2grYeuHA4517uvceLGJXKsj6NhcWl5ZXVwlpxfWNza9vc2W1JHgtMmpgzLjoekoTRkDQVYx0IkFQ4DHS9kaXmd+I0JSHt6ocUTcA1C6lOMlJZ65tkYnkNH0kGAyk6A1NDzk/v0NinzSgod3OcK/sjDtFeDR9CbmpWeWbKq1gRwntg5KYEcjZ757vQ5jgMSKsyQlF3bipSbIKEoZiQtOrEkEcIjNCBdTUMUEOkmkxdTeKiVPvS50BUqOF/TyQokHIceLozO1fOepn4n9eNlX/qJjSMYkVCPF3kxwqDrO8YJ8KghUba4KwoPpWiIdIKx0qkUdgj378jxp1aq2VbWvj0v1izyOAtgHB6AMbHAC6uAKNEATYPAnsALeDUejWfjzfiYti4Y+cwe+APj6xs+HKLU</latexit><latexit sha1_base64="xwf5NTK8Os9frFEQg9QWL1GKg9Q=">ACInicbVDLSsNAFJ34rPUVdelmsAgtQkmKoC6EohuXFewDmlgm0k7dJIJMxOlhHyLG3/FjQtFXQl+jJM2grYeuHA4517uvceLGJXKsj6NhcWl5ZXVwlpxfWNza9vc2W1JHgtMmpgzLjoekoTRkDQVYx0IkFQ4DHS9kaXmd+I0JSHt6ocUTcA1C6lOMlJZ65tkYnkNH0kGAyk6A1NDzk/v0NinzSgod3OcK/sjDtFeDR9CbmpWeWbKq1gRwntg5KYEcjZ757vQ5jgMSKsyQlF3bipSbIKEoZiQtOrEkEcIjNCBdTUMUEOkmkxdTeKiVPvS50BUqOF/TyQokHIceLozO1fOepn4n9eNlX/qJjSMYkVCPF3kxwqDrO8YJ8KghUba4KwoPpWiIdIKx0qkUdgj378jxp1aq2VbWvj0v1izyOAtgHB6AMbHAC6uAKNEATYPAnsALeDUejWfjzfiYti4Y+cwe+APj6xs+HKLU</latexit><latexit sha1_base64="xwf5NTK8Os9frFEQg9QWL1GKg9Q=">ACInicbVDLSsNAFJ34rPUVdelmsAgtQkmKoC6EohuXFewDmlgm0k7dJIJMxOlhHyLG3/FjQtFXQl+jJM2grYeuHA4517uvceLGJXKsj6NhcWl5ZXVwlpxfWNza9vc2W1JHgtMmpgzLjoekoTRkDQVYx0IkFQ4DHS9kaXmd+I0JSHt6ocUTcA1C6lOMlJZ65tkYnkNH0kGAyk6A1NDzk/v0NinzSgod3OcK/sjDtFeDR9CbmpWeWbKq1gRwntg5KYEcjZ757vQ5jgMSKsyQlF3bipSbIKEoZiQtOrEkEcIjNCBdTUMUEOkmkxdTeKiVPvS50BUqOF/TyQokHIceLozO1fOepn4n9eNlX/qJjSMYkVCPF3kxwqDrO8YJ8KghUba4KwoPpWiIdIKx0qkUdgj378jxp1aq2VbWvj0v1izyOAtgHB6AMbHAC6uAKNEATYPAnsALeDUejWfjzfiYti4Y+cwe+APj6xs+HKLU</latexit><latexit sha1_base64="xwf5NTK8Os9frFEQg9QWL1GKg9Q=">ACInicbVDLSsNAFJ34rPUVdelmsAgtQkmKoC6EohuXFewDmlgm0k7dJIJMxOlhHyLG3/FjQtFXQl+jJM2grYeuHA4517uvceLGJXKsj6NhcWl5ZXVwlpxfWNza9vc2W1JHgtMmpgzLjoekoTRkDQVYx0IkFQ4DHS9kaXmd+I0JSHt6ocUTcA1C6lOMlJZ65tkYnkNH0kGAyk6A1NDzk/v0NinzSgod3OcK/sjDtFeDR9CbmpWeWbKq1gRwntg5KYEcjZ757vQ5jgMSKsyQlF3bipSbIKEoZiQtOrEkEcIjNCBdTUMUEOkmkxdTeKiVPvS50BUqOF/TyQokHIceLozO1fOepn4n9eNlX/qJjSMYkVCPF3kxwqDrO8YJ8KghUba4KwoPpWiIdIKx0qkUdgj378jxp1aq2VbWvj0v1izyOAtgHB6AMbHAC6uAKNEATYPAnsALeDUejWfjzfiYti4Y+cwe+APj6xs+HKLU</latexit>

*: f is applied element-wise

f([z1, z2, z3]) = [f(z1), f(z2), f(z3)]

<latexit sha1_base64="nb0Ya5OXLynEy63m+VxQ8PVH0Sw=">ACFnicbVDLSgMxFM34rPU16tJNsAgt1DLTCroRim5cVrAPmA5DJs20oZnMkGSEtvQr3Pgrblwo4lbc+Tdm2lo64HkHs65l+QeP2ZUKsv6NlZW19Y3NnNb+e2d3b198+CwJaNEYNLEYtEx0eSMpJU1HFSCcWBIU+I21/eJP67QciJI34vRrFxA1Rn9OAYqS05JlnQdEZe3YZjr1qetXcEryCTlDUYqkM01rNaq3kembBqlgzwGViZ6QAMjQ86vbi3ASEq4wQ1I6thUrd4KEopiRab6bSBIjPER94mjKUikO5mtNYWnWunBIBL6cAVn6u+JCQqlHIW+7gyRGshFLxX/85xEBZfuhPI4UYTj+UNBwqCKYJoR7FBsGIjTRAWVP8V4gESCudZF6HYC+uvExa1YptVey780L9OosjB47BCSgCG1yAOrgFDdAEGDyCZ/AK3own48V4Nz7mrStGNnME/sD4/AGnpJqo</latexit><latexit sha1_base64="nb0Ya5OXLynEy63m+VxQ8PVH0Sw=">ACFnicbVDLSgMxFM34rPU16tJNsAgt1DLTCroRim5cVrAPmA5DJs20oZnMkGSEtvQr3Pgrblwo4lbc+Tdm2lo64HkHs65l+QeP2ZUKsv6NlZW19Y3NnNb+e2d3b198+CwJaNEYNLEYtEx0eSMpJU1HFSCcWBIU+I21/eJP67QciJI34vRrFxA1Rn9OAYqS05JlnQdEZe3YZjr1qetXcEryCTlDUYqkM01rNaq3kembBqlgzwGViZ6QAMjQ86vbi3ASEq4wQ1I6thUrd4KEopiRab6bSBIjPER94mjKUikO5mtNYWnWunBIBL6cAVn6u+JCQqlHIW+7gyRGshFLxX/85xEBZfuhPI4UYTj+UNBwqCKYJoR7FBsGIjTRAWVP8V4gESCudZF6HYC+uvExa1YptVey780L9OosjB47BCSgCG1yAOrgFDdAEGDyCZ/AK3own48V4Nz7mrStGNnME/sD4/AGnpJqo</latexit><latexit sha1_base64="nb0Ya5OXLynEy63m+VxQ8PVH0Sw=">ACFnicbVDLSgMxFM34rPU16tJNsAgt1DLTCroRim5cVrAPmA5DJs20oZnMkGSEtvQr3Pgrblwo4lbc+Tdm2lo64HkHs65l+QeP2ZUKsv6NlZW19Y3NnNb+e2d3b198+CwJaNEYNLEYtEx0eSMpJU1HFSCcWBIU+I21/eJP67QciJI34vRrFxA1Rn9OAYqS05JlnQdEZe3YZjr1qetXcEryCTlDUYqkM01rNaq3kembBqlgzwGViZ6QAMjQ86vbi3ASEq4wQ1I6thUrd4KEopiRab6bSBIjPER94mjKUikO5mtNYWnWunBIBL6cAVn6u+JCQqlHIW+7gyRGshFLxX/85xEBZfuhPI4UYTj+UNBwqCKYJoR7FBsGIjTRAWVP8V4gESCudZF6HYC+uvExa1YptVey780L9OosjB47BCSgCG1yAOrgFDdAEGDyCZ/AK3own48V4Nz7mrStGNnME/sD4/AGnpJqo</latexit><latexit sha1_base64="nb0Ya5OXLynEy63m+VxQ8PVH0Sw=">ACFnicbVDLSgMxFM34rPU16tJNsAgt1DLTCroRim5cVrAPmA5DJs20oZnMkGSEtvQr3Pgrblwo4lbc+Tdm2lo64HkHs65l+QeP2ZUKsv6NlZW19Y3NnNb+e2d3b198+CwJaNEYNLEYtEx0eSMpJU1HFSCcWBIU+I21/eJP67QciJI34vRrFxA1Rn9OAYqS05JlnQdEZe3YZjr1qetXcEryCTlDUYqkM01rNaq3kembBqlgzwGViZ6QAMjQ86vbi3ASEq4wQ1I6thUrd4KEopiRab6bSBIjPER94mjKUikO5mtNYWnWunBIBL6cAVn6u+JCQqlHIW+7gyRGshFLxX/85xEBZfuhPI4UYTj+UNBwqCKYJoR7FBsGIjTRAWVP8V4gESCudZF6HYC+uvExa1YptVey780L9OosjB47BCSgCG1yAOrgFDdAEGDyCZ/AK3own48V4Nz7mrStGNnME/sD4/AGnpJqo</latexit>

x ∈ Rd

<latexit sha1_base64="r8cLpKiZJtXS/wgD8iCVci7o1w=">ACBHicbVC7TsMwFL3hWcorwNjFokJiqhKEBANDJRbGguhDakrlOE5r1XEi20FUQYWfoWFAYRY+Qg2/gan7QAtR7J0fM69uvceP+FMacf5tpaWV1bX1ksb5c2t7Z1de2+/peJUEtokMY9lx8eKciZoUzPNaSeRFEc+p21/dFn47XsqFYvFrR4ntBfhgWAhI1gbqW9XvAjroR9mDznymEDTr5/d5HdB3646NWcCtEjcGanCDI2+/eUFMUkjKjThWKmu6yS6l2GpGeE0L3upogkmIzygXUMFjqjqZMjcnRklACFsTRPaDRf3dkOFJqHPmsthRzXuF+J/XTXV43suYSFJNBZkOClOdIyKRFDAJCWajw3BRDKzKyJDLDHRJreyCcGdP3mRtE5qrlNzr0+r9YtZHCWowCEcgwtnUIcraEATCDzCM7zCm/VkvVjv1se0dMma9RzAH1ifP+g+mDo=</latexit><latexit sha1_base64="r8cLpKiZJtXS/wgD8iCVci7o1w=">ACBHicbVC7TsMwFL3hWcorwNjFokJiqhKEBANDJRbGguhDakrlOE5r1XEi20FUQYWfoWFAYRY+Qg2/gan7QAtR7J0fM69uvceP+FMacf5tpaWV1bX1ksb5c2t7Z1de2+/peJUEtokMY9lx8eKciZoUzPNaSeRFEc+p21/dFn47XsqFYvFrR4ntBfhgWAhI1gbqW9XvAjroR9mDznymEDTr5/d5HdB3646NWcCtEjcGanCDI2+/eUFMUkjKjThWKmu6yS6l2GpGeE0L3upogkmIzygXUMFjqjqZMjcnRklACFsTRPaDRf3dkOFJqHPmsthRzXuF+J/XTXV43suYSFJNBZkOClOdIyKRFDAJCWajw3BRDKzKyJDLDHRJreyCcGdP3mRtE5qrlNzr0+r9YtZHCWowCEcgwtnUIcraEATCDzCM7zCm/VkvVjv1se0dMma9RzAH1ifP+g+mDo=</latexit><latexit sha1_base64="r8cLpKiZJtXS/wgD8iCVci7o1w=">ACBHicbVC7TsMwFL3hWcorwNjFokJiqhKEBANDJRbGguhDakrlOE5r1XEi20FUQYWfoWFAYRY+Qg2/gan7QAtR7J0fM69uvceP+FMacf5tpaWV1bX1ksb5c2t7Z1de2+/peJUEtokMY9lx8eKciZoUzPNaSeRFEc+p21/dFn47XsqFYvFrR4ntBfhgWAhI1gbqW9XvAjroR9mDznymEDTr5/d5HdB3646NWcCtEjcGanCDI2+/eUFMUkjKjThWKmu6yS6l2GpGeE0L3upogkmIzygXUMFjqjqZMjcnRklACFsTRPaDRf3dkOFJqHPmsthRzXuF+J/XTXV43suYSFJNBZkOClOdIyKRFDAJCWajw3BRDKzKyJDLDHRJreyCcGdP3mRtE5qrlNzr0+r9YtZHCWowCEcgwtnUIcraEATCDzCM7zCm/VkvVjv1se0dMma9RzAH1ifP+g+mDo=</latexit><latexit sha1_base64="r8cLpKiZJtXS/wgD8iCVci7o1w=">ACBHicbVC7TsMwFL3hWcorwNjFokJiqhKEBANDJRbGguhDakrlOE5r1XEi20FUQYWfoWFAYRY+Qg2/gan7QAtR7J0fM69uvceP+FMacf5tpaWV1bX1ksb5c2t7Z1de2+/peJUEtokMY9lx8eKciZoUzPNaSeRFEc+p21/dFn47XsqFYvFrR4ntBfhgWAhI1gbqW9XvAjroR9mDznymEDTr5/d5HdB3646NWcCtEjcGanCDI2+/eUFMUkjKjThWKmu6yS6l2GpGeE0L3upogkmIzygXUMFjqjqZMjcnRklACFsTRPaDRf3dkOFJqHPmsthRzXuF+J/XTXV43suYSFJNBZkOClOdIyKRFDAJCWajw3BRDKzKyJDLDHRJreyCcGdP3mRtE5qrlNzr0+r9YtZHCWowCEcgwtnUIcraEATCDzCM7zCm/VkvVjv1se0dMma9RzAH1ifP+g+mDo=</latexit>

W(1) ∈ Rd1×d, b(1) ∈ Rd1

<latexit sha1_base64="Av+k0GeUMcXk+fjFEBVogb7nE=">ACPnicfVDPS8MwGE3nrzl/T16CQ5hgoxWBD14GHjxOMVug7WNE23sDQtSqM0r/Mi3+DN49ePCji1aPpNkE38UHg8d7yPc9P2FUKtN8MkoLi0vLK+XVytr6xuZWdXunLeNUYGLjmMWi6yNJGOXEVlQx0k0EQZHPSMcfXhR+54ISWN+o0YJcSPU5zSkGCkteVXbiZAa+GHWyW+zunWYQ4dyOBH97FqLgWdBR9GISBjkR/A7/+Tz71qzWyY8B5Yk1JDUzR8qPThDjNCJcYak7FlmotwMCUxI3nFSVJEB6iPulpypFex83G5+fwQCsBDGOhH1dwrP6cyFAk5SjydbLYU856hfiX10tVeOZmlCepIhxPgpTBlUMiy5hQAXBio0QVhQvSvEAyQVrxi7Bmj15nrSPG5bZsK5Oas3zaR1lsAf2QR1Y4BQ0wSVoARtgcA+ewSt4Mx6MF+Pd+JhES8Z0Zhf8gvH5BQCyrvw=</latexit><latexit sha1_base64="Av+k0GeUMcXk+fjFEBVogb7nE=">ACPnicfVDPS8MwGE3nrzl/T16CQ5hgoxWBD14GHjxOMVug7WNE23sDQtSqM0r/Mi3+DN49ePCji1aPpNkE38UHg8d7yPc9P2FUKtN8MkoLi0vLK+XVytr6xuZWdXunLeNUYGLjmMWi6yNJGOXEVlQx0k0EQZHPSMcfXhR+54ISWN+o0YJcSPU5zSkGCkteVXbiZAa+GHWyW+zunWYQ4dyOBH97FqLgWdBR9GISBjkR/A7/+Tz71qzWyY8B5Yk1JDUzR8qPThDjNCJcYak7FlmotwMCUxI3nFSVJEB6iPulpypFex83G5+fwQCsBDGOhH1dwrP6cyFAk5SjydbLYU856hfiX10tVeOZmlCepIhxPgpTBlUMiy5hQAXBio0QVhQvSvEAyQVrxi7Bmj15nrSPG5bZsK5Oas3zaR1lsAf2QR1Y4BQ0wSVoARtgcA+ewSt4Mx6MF+Pd+JhES8Z0Zhf8gvH5BQCyrvw=</latexit><latexit sha1_base64="Av+k0GeUMcXk+fjFEBVogb7nE=">ACPnicfVDPS8MwGE3nrzl/T16CQ5hgoxWBD14GHjxOMVug7WNE23sDQtSqM0r/Mi3+DN49ePCji1aPpNkE38UHg8d7yPc9P2FUKtN8MkoLi0vLK+XVytr6xuZWdXunLeNUYGLjmMWi6yNJGOXEVlQx0k0EQZHPSMcfXhR+54ISWN+o0YJcSPU5zSkGCkteVXbiZAa+GHWyW+zunWYQ4dyOBH97FqLgWdBR9GISBjkR/A7/+Tz71qzWyY8B5Yk1JDUzR8qPThDjNCJcYak7FlmotwMCUxI3nFSVJEB6iPulpypFex83G5+fwQCsBDGOhH1dwrP6cyFAk5SjydbLYU856hfiX10tVeOZmlCepIhxPgpTBlUMiy5hQAXBio0QVhQvSvEAyQVrxi7Bmj15nrSPG5bZsK5Oas3zaR1lsAf2QR1Y4BQ0wSVoARtgcA+ewSt4Mx6MF+Pd+JhES8Z0Zhf8gvH5BQCyrvw=</latexit><latexit sha1_base64="Av+k0GeUMcXk+fjFEBVogb7nE=">ACPnicfVDPS8MwGE3nrzl/T16CQ5hgoxWBD14GHjxOMVug7WNE23sDQtSqM0r/Mi3+DN49ePCji1aPpNkE38UHg8d7yPc9P2FUKtN8MkoLi0vLK+XVytr6xuZWdXunLeNUYGLjmMWi6yNJGOXEVlQx0k0EQZHPSMcfXhR+54ISWN+o0YJcSPU5zSkGCkteVXbiZAa+GHWyW+zunWYQ4dyOBH97FqLgWdBR9GISBjkR/A7/+Tz71qzWyY8B5Yk1JDUzR8qPThDjNCJcYak7FlmotwMCUxI3nFSVJEB6iPulpypFex83G5+fwQCsBDGOhH1dwrP6cyFAk5SjydbLYU856hfiX10tVeOZmlCepIhxPgpTBlUMiy5hQAXBio0QVhQvSvEAyQVrxi7Bmj15nrSPG5bZsK5Oas3zaR1lsAf2QR1Y4BQ0wSVoARtgcA+ewSt4Mx6MF+Pd+JhES8Z0Zhf8gvH5BQCyrvw=</latexit>
slide-19
SLIDE 19

Mathematical Notation, side by side

  • Input layer: x1, …, xd
  • Hidden layer 2: h(2)

1 , h(2) 2 , …, h(2) d2

· · ·

<latexit sha1_base64="C/ThE3LO0bsU2xdfe5suNCnJ9S4=">AB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NKPzM8pWxMh7xnaUJjbvx8fu2UnFklJHSthIkc/X3RE5jYyZxYDtjiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB0oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4ArlePLw=</latexit><latexit sha1_base64="C/ThE3LO0bsU2xdfe5suNCnJ9S4=">AB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NKPzM8pWxMh7xnaUJjbvx8fu2UnFklJHSthIkc/X3RE5jYyZxYDtjiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB0oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4ArlePLw=</latexit><latexit sha1_base64="C/ThE3LO0bsU2xdfe5suNCnJ9S4=">AB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NKPzM8pWxMh7xnaUJjbvx8fu2UnFklJHSthIkc/X3RE5jYyZxYDtjiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB0oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4ArlePLw=</latexit><latexit sha1_base64="C/ThE3LO0bsU2xdfe5suNCnJ9S4=">AB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NKPzM8pWxMh7xnaUJjbvx8fu2UnFklJHSthIkc/X3RE5jYyZxYDtjiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB0oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4ArlePLw=</latexit>

h(2)

1

= f(W (2)

1,1 h(1) 1

+ W (2)

1,2 h(1) 2

+ . . . + W (2)

1,d1h(1) d1 + b(2) 1 )

<latexit sha1_base64="a67o9ygnsA1DfBnoZrjOG8wnOw=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZVMEvQhFLx4r2A9o65LNZtvQ7AdJVihL/6QnPfhXxGy7aj8cCLzPjNMnEizqSyrA/DzG1t7+QLu8W90v7BYfnouCPDWBDaJiEPRc/BknIW0LZitNeJCj2HU67zuQh5d1XKiQLg2c1jejQx6OAeYxgpS27LMYvSbVRm9kI3kGv2s2yBF2hWYrQHF3CJdD4BQ0NBtwNlVytcO2/5iRNHZ+5tTscsWqW/OAmwJlogKyaNnlt4EbktingSIcS9lHVqSGCRaKEU5nxUEsaYTJBI9oX8sA+1QOk/luZvBCOy70QqFPoODcXe5IsC/l1Hd0pY/VWK6z1PyP9WPl3Q4TFkSxogFZDPJiDlUI0VDlwlKFJ9qgYlg+q6QjLHAROnvKOoloPUnb4pOo46sOnq6rjTvs3UwBk4B1WAwA1ogkfQAm1AwDv4MvJGwfg0c2bRLC1KTSPrOQErYZ5+A73GrVI=</latexit><latexit sha1_base64="a67o9ygnsA1DfBnoZrjOG8wnOw=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZVMEvQhFLx4r2A9o65LNZtvQ7AdJVihL/6QnPfhXxGy7aj8cCLzPjNMnEizqSyrA/DzG1t7+QLu8W90v7BYfnouCPDWBDaJiEPRc/BknIW0LZitNeJCj2HU67zuQh5d1XKiQLg2c1jejQx6OAeYxgpS27LMYvSbVRm9kI3kGv2s2yBF2hWYrQHF3CJdD4BQ0NBtwNlVytcO2/5iRNHZ+5tTscsWqW/OAmwJlogKyaNnlt4EbktingSIcS9lHVqSGCRaKEU5nxUEsaYTJBI9oX8sA+1QOk/luZvBCOy70QqFPoODcXe5IsC/l1Hd0pY/VWK6z1PyP9WPl3Q4TFkSxogFZDPJiDlUI0VDlwlKFJ9qgYlg+q6QjLHAROnvKOoloPUnb4pOo46sOnq6rjTvs3UwBk4B1WAwA1ogkfQAm1AwDv4MvJGwfg0c2bRLC1KTSPrOQErYZ5+A73GrVI=</latexit><latexit sha1_base64="a67o9ygnsA1DfBnoZrjOG8wnOw=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZVMEvQhFLx4r2A9o65LNZtvQ7AdJVihL/6QnPfhXxGy7aj8cCLzPjNMnEizqSyrA/DzG1t7+QLu8W90v7BYfnouCPDWBDaJiEPRc/BknIW0LZitNeJCj2HU67zuQh5d1XKiQLg2c1jejQx6OAeYxgpS27LMYvSbVRm9kI3kGv2s2yBF2hWYrQHF3CJdD4BQ0NBtwNlVytcO2/5iRNHZ+5tTscsWqW/OAmwJlogKyaNnlt4EbktingSIcS9lHVqSGCRaKEU5nxUEsaYTJBI9oX8sA+1QOk/luZvBCOy70QqFPoODcXe5IsC/l1Hd0pY/VWK6z1PyP9WPl3Q4TFkSxogFZDPJiDlUI0VDlwlKFJ9qgYlg+q6QjLHAROnvKOoloPUnb4pOo46sOnq6rjTvs3UwBk4B1WAwA1ogkfQAm1AwDv4MvJGwfg0c2bRLC1KTSPrOQErYZ5+A73GrVI=</latexit><latexit sha1_base64="a67o9ygnsA1DfBnoZrjOG8wnOw=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZVMEvQhFLx4r2A9o65LNZtvQ7AdJVihL/6QnPfhXxGy7aj8cCLzPjNMnEizqSyrA/DzG1t7+QLu8W90v7BYfnouCPDWBDaJiEPRc/BknIW0LZitNeJCj2HU67zuQh5d1XKiQLg2c1jejQx6OAeYxgpS27LMYvSbVRm9kI3kGv2s2yBF2hWYrQHF3CJdD4BQ0NBtwNlVytcO2/5iRNHZ+5tTscsWqW/OAmwJlogKyaNnlt4EbktingSIcS9lHVqSGCRaKEU5nxUEsaYTJBI9oX8sA+1QOk/luZvBCOy70QqFPoODcXe5IsC/l1Hd0pY/VWK6z1PyP9WPl3Q4TFkSxogFZDPJiDlUI0VDlwlKFJ9qgYlg+q6QjLHAROnvKOoloPUnb4pOo46sOnq6rjTvs3UwBk4B1WAwA1ogkfQAm1AwDv4MvJGwfg0c2bRLC1KTSPrOQErYZ5+A73GrVI=</latexit>

h(2)

2

= f(W (2)

2,1 h(1) 1

+ W (2)

2,2 h(1) 2

+ . . . + W (2)

2,d1h(1) d1 + b(2) 2 )

<latexit sha1_base64="w4Ns9nbRosQqD3EzcxhGPhYI7VU=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZbMIehGKXjxWsB/Q1iWbzbah2Q+SrFCW/klPevCviNl2/ah1IPDO+8wycSNOZPKst4Ms7CxuVUsbZd3Krt7+9WDw6MEkFoh0Q8En0XS8pZSDuKU7saA4cDntudO7jPeqZAsCh/VLKajAI9D5jOClbacqpg8pXW7MXdseAP9ei/PUvsCzTOEdILgOfwF7G9gazDkXqTkaoXn/DSnWaKx+zWn4VRrVtNaBFwXKBc1kEfbqb4MvYgkAQ0V4VjKAbJiNUqxUIxwOi8PE0ljTKZ4TAdahjigcpQudjOHZ9rxoB8JfUIF+7vjhQHUs4CV1cGWE3kX5aZ/7FBovzrUcrCOFE0JMtBfsKhimC2aOgxQYniMy0wEUzfFZIJFpgo/R1lvQT098nroms3kdVED5e1m2+jhI4AaegDhC4Ai1wD9qgAwh4BR9G0SgZ72bBLJuVZalp5D1HYCXM40/Gfq1X</latexit><latexit sha1_base64="w4Ns9nbRosQqD3EzcxhGPhYI7VU=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZbMIehGKXjxWsB/Q1iWbzbah2Q+SrFCW/klPevCviNl2/ah1IPDO+8wycSNOZPKst4Ms7CxuVUsbZd3Krt7+9WDw6MEkFoh0Q8En0XS8pZSDuKU7saA4cDntudO7jPeqZAsCh/VLKajAI9D5jOClbacqpg8pXW7MXdseAP9ei/PUvsCzTOEdILgOfwF7G9gazDkXqTkaoXn/DSnWaKx+zWn4VRrVtNaBFwXKBc1kEfbqb4MvYgkAQ0V4VjKAbJiNUqxUIxwOi8PE0ljTKZ4TAdahjigcpQudjOHZ9rxoB8JfUIF+7vjhQHUs4CV1cGWE3kX5aZ/7FBovzrUcrCOFE0JMtBfsKhimC2aOgxQYniMy0wEUzfFZIJFpgo/R1lvQT098nroms3kdVED5e1m2+jhI4AaegDhC4Ai1wD9qgAwh4BR9G0SgZ72bBLJuVZalp5D1HYCXM40/Gfq1X</latexit><latexit sha1_base64="w4Ns9nbRosQqD3EzcxhGPhYI7VU=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZbMIehGKXjxWsB/Q1iWbzbah2Q+SrFCW/klPevCviNl2/ah1IPDO+8wycSNOZPKst4Ms7CxuVUsbZd3Krt7+9WDw6MEkFoh0Q8En0XS8pZSDuKU7saA4cDntudO7jPeqZAsCh/VLKajAI9D5jOClbacqpg8pXW7MXdseAP9ei/PUvsCzTOEdILgOfwF7G9gazDkXqTkaoXn/DSnWaKx+zWn4VRrVtNaBFwXKBc1kEfbqb4MvYgkAQ0V4VjKAbJiNUqxUIxwOi8PE0ljTKZ4TAdahjigcpQudjOHZ9rxoB8JfUIF+7vjhQHUs4CV1cGWE3kX5aZ/7FBovzrUcrCOFE0JMtBfsKhimC2aOgxQYniMy0wEUzfFZIJFpgo/R1lvQT098nroms3kdVED5e1m2+jhI4AaegDhC4Ai1wD9qgAwh4BR9G0SgZ72bBLJuVZalp5D1HYCXM40/Gfq1X</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQvEKMXyw=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odBu3wMYA6nMFXEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="Qjb73Ct4Q2f2fZP+NO8a8FTJnvM=">ACUHicbZFdS8MwFIbTt2cU6u3gSHsKGMpjd6IwjeDnBfcA2S5qmWzD9IEmFUfonvdIL/4qYblX34YHAe85zDid54yWcSWXbH4Z2dndq9b26weNw6Nj6TRl3EqCO2RmMdi6GFJOYtoTzHF6TARFIcepwPv5b7g1cqJIujJzVP6CTE04gFjGClS64lZs9Zy2nrgNvYdAalFnmXKG8QEgnCF7CFeD8AkeDMfdjJdc7fPdvOCsSjb2fPW3XatodexFwW6BSNEZXd6G/sxSUMaKcKxlCNkJ2qSYaEY4TSvj1NJE0xe8JSOtIxwSOUkW3iTwtd8WEQC30iBRfV1YkMh1LOQ093hljN5CYriv+xUaqCm0nGoiRVNCLRUHKoYphYT0maBE8bkWmAim7wrJDAtMlP6OujYBbT5W/SdDrI76NEGNXAGzkELIHAN7sAD6IeIOAdfBlVo2Z8mhWzvrTLNErfTsFamI1vSaKskA=</latexit><latexit sha1_base64="Qjb73Ct4Q2f2fZP+NO8a8FTJnvM=">ACUHicbZFdS8MwFIbTt2cU6u3gSHsKGMpjd6IwjeDnBfcA2S5qmWzD9IEmFUfonvdIL/4qYblX34YHAe85zDid54yWcSWXbH4Z2dndq9b26weNw6Nj6TRl3EqCO2RmMdi6GFJOYtoTzHF6TARFIcepwPv5b7g1cqJIujJzVP6CTE04gFjGClS64lZs9Zy2nrgNvYdAalFnmXKG8QEgnCF7CFeD8AkeDMfdjJdc7fPdvOCsSjb2fPW3XatodexFwW6BSNEZXd6G/sxSUMaKcKxlCNkJ2qSYaEY4TSvj1NJE0xe8JSOtIxwSOUkW3iTwtd8WEQC30iBRfV1YkMh1LOQ093hljN5CYriv+xUaqCm0nGoiRVNCLRUHKoYphYT0maBE8bkWmAim7wrJDAtMlP6OujYBbT5W/SdDrI76NEGNXAGzkELIHAN7sAD6IeIOAdfBlVo2Z8mhWzvrTLNErfTsFamI1vSaKskA=</latexit><latexit sha1_base64="WdR2VpbOjV0J9tna7SL9E01jAo=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZbMXvQhFLx4r2A9o65LNZtg9oMkK5Slf9KTHvwrYrZdtR8OBN5nxkmbgxZ1JZ1odhFra2d4ql3fJeZf/gsHp03JVRIgjtkIhHou9iSTkLaUcxWk/FhQHLqc9+U+471XKiSLwic1jekowOQ+YxgpS2nKibPad1uzBwb3kK/3suz1L5CswhnSB4CZeA/QtsDYbci5RcrfCcv+Y0SzR2f+Y0nGrNalrzgJsC5aIG8mg71behF5EkoKEiHEs5QFasRikWihFOZ+VhImMyQse04GWIQ6oHKXz3czghXY86EdCn1DBubvckeJAymng6soAq4lcZ5n5Hxskyr8ZpSyME0VDshjkJxyqCGaLh4TlCg+1QITwfRdIZlgYnS31HWS0DrT94UXbuJrCZ6tGqtu3wdJXAGzkEdIHANWuABtEHEPAOvoyiUTI+zYJZNiuLUtPIe07ASpin38U+rVM=</latexit><latexit sha1_base64="w4Ns9nbRosQqD3EzcxhGPhYI7VU=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZbMIehGKXjxWsB/Q1iWbzbah2Q+SrFCW/klPevCviNl2/ah1IPDO+8wycSNOZPKst4Ms7CxuVUsbZd3Krt7+9WDw6MEkFoh0Q8En0XS8pZSDuKU7saA4cDntudO7jPeqZAsCh/VLKajAI9D5jOClbacqpg8pXW7MXdseAP9ei/PUvsCzTOEdILgOfwF7G9gazDkXqTkaoXn/DSnWaKx+zWn4VRrVtNaBFwXKBc1kEfbqb4MvYgkAQ0V4VjKAbJiNUqxUIxwOi8PE0ljTKZ4TAdahjigcpQudjOHZ9rxoB8JfUIF+7vjhQHUs4CV1cGWE3kX5aZ/7FBovzrUcrCOFE0JMtBfsKhimC2aOgxQYniMy0wEUzfFZIJFpgo/R1lvQT098nroms3kdVED5e1m2+jhI4AaegDhC4Ai1wD9qgAwh4BR9G0SgZ72bBLJuVZalp5D1HYCXM40/Gfq1X</latexit><latexit sha1_base64="w4Ns9nbRosQqD3EzcxhGPhYI7VU=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZbMIehGKXjxWsB/Q1iWbzbah2Q+SrFCW/klPevCviNl2/ah1IPDO+8wycSNOZPKst4Ms7CxuVUsbZd3Krt7+9WDw6MEkFoh0Q8En0XS8pZSDuKU7saA4cDntudO7jPeqZAsCh/VLKajAI9D5jOClbacqpg8pXW7MXdseAP9ei/PUvsCzTOEdILgOfwF7G9gazDkXqTkaoXn/DSnWaKx+zWn4VRrVtNaBFwXKBc1kEfbqb4MvYgkAQ0V4VjKAbJiNUqxUIxwOi8PE0ljTKZ4TAdahjigcpQudjOHZ9rxoB8JfUIF+7vjhQHUs4CV1cGWE3kX5aZ/7FBovzrUcrCOFE0JMtBfsKhimC2aOgxQYniMy0wEUzfFZIJFpgo/R1lvQT098nroms3kdVED5e1m2+jhI4AaegDhC4Ai1wD9qgAwh4BR9G0SgZ72bBLJuVZalp5D1HYCXM40/Gfq1X</latexit><latexit sha1_base64="w4Ns9nbRosQqD3EzcxhGPhYI7VU=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZbMIehGKXjxWsB/Q1iWbzbah2Q+SrFCW/klPevCviNl2/ah1IPDO+8wycSNOZPKst4Ms7CxuVUsbZd3Krt7+9WDw6MEkFoh0Q8En0XS8pZSDuKU7saA4cDntudO7jPeqZAsCh/VLKajAI9D5jOClbacqpg8pXW7MXdseAP9ei/PUvsCzTOEdILgOfwF7G9gazDkXqTkaoXn/DSnWaKx+zWn4VRrVtNaBFwXKBc1kEfbqb4MvYgkAQ0V4VjKAbJiNUqxUIxwOi8PE0ljTKZ4TAdahjigcpQudjOHZ9rxoB8JfUIF+7vjhQHUs4CV1cGWE3kX5aZ/7FBovzrUcrCOFE0JMtBfsKhimC2aOgxQYniMy0wEUzfFZIJFpgo/R1lvQT098nroms3kdVED5e1m2+jhI4AaegDhC4Ai1wD9qgAwh4BR9G0SgZ72bBLJuVZalp5D1HYCXM40/Gfq1X</latexit><latexit sha1_base64="w4Ns9nbRosQqD3EzcxhGPhYI7VU=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZbMIehGKXjxWsB/Q1iWbzbah2Q+SrFCW/klPevCviNl2/ah1IPDO+8wycSNOZPKst4Ms7CxuVUsbZd3Krt7+9WDw6MEkFoh0Q8En0XS8pZSDuKU7saA4cDntudO7jPeqZAsCh/VLKajAI9D5jOClbacqpg8pXW7MXdseAP9ei/PUvsCzTOEdILgOfwF7G9gazDkXqTkaoXn/DSnWaKx+zWn4VRrVtNaBFwXKBc1kEfbqb4MvYgkAQ0V4VjKAbJiNUqxUIxwOi8PE0ljTKZ4TAdahjigcpQudjOHZ9rxoB8JfUIF+7vjhQHUs4CV1cGWE3kX5aZ/7FBovzrUcrCOFE0JMtBfsKhimC2aOgxQYniMy0wEUzfFZIJFpgo/R1lvQT098nroms3kdVED5e1m2+jhI4AaegDhC4Ai1wD9qgAwh4BR9G0SgZ72bBLJuVZalp5D1HYCXM40/Gfq1X</latexit><latexit sha1_base64="w4Ns9nbRosQqD3EzcxhGPhYI7VU=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZbMIehGKXjxWsB/Q1iWbzbah2Q+SrFCW/klPevCviNl2/ah1IPDO+8wycSNOZPKst4Ms7CxuVUsbZd3Krt7+9WDw6MEkFoh0Q8En0XS8pZSDuKU7saA4cDntudO7jPeqZAsCh/VLKajAI9D5jOClbacqpg8pXW7MXdseAP9ei/PUvsCzTOEdILgOfwF7G9gazDkXqTkaoXn/DSnWaKx+zWn4VRrVtNaBFwXKBc1kEfbqb4MvYgkAQ0V4VjKAbJiNUqxUIxwOi8PE0ljTKZ4TAdahjigcpQudjOHZ9rxoB8JfUIF+7vjhQHUs4CV1cGWE3kX5aZ/7FBovzrUcrCOFE0JMtBfsKhimC2aOgxQYniMy0wEUzfFZIJFpgo/R1lvQT098nroms3kdVED5e1m2+jhI4AaegDhC4Ai1wD9qgAwh4BR9G0SgZ72bBLJuVZalp5D1HYCXM40/Gfq1X</latexit><latexit sha1_base64="w4Ns9nbRosQqD3EzcxhGPhYI7VU=">ACW3icbZFNSwMxEIazW7W1Vq2KJy/BIrQoZbMIehGKXjxWsB/Q1iWbzbah2Q+SrFCW/klPevCviNl2/ah1IPDO+8wycSNOZPKst4Ms7CxuVUsbZd3Krt7+9WDw6MEkFoh0Q8En0XS8pZSDuKU7saA4cDntudO7jPeqZAsCh/VLKajAI9D5jOClbacqpg8pXW7MXdseAP9ei/PUvsCzTOEdILgOfwF7G9gazDkXqTkaoXn/DSnWaKx+zWn4VRrVtNaBFwXKBc1kEfbqb4MvYgkAQ0V4VjKAbJiNUqxUIxwOi8PE0ljTKZ4TAdahjigcpQudjOHZ9rxoB8JfUIF+7vjhQHUs4CV1cGWE3kX5aZ/7FBovzrUcrCOFE0JMtBfsKhimC2aOgxQYniMy0wEUzfFZIJFpgo/R1lvQT098nroms3kdVED5e1m2+jhI4AaegDhC4Ai1wD9qgAwh4BR9G0SgZ72bBLJuVZalp5D1HYCXM40/Gfq1X</latexit>
  • Output layer:

y = σ(w(o)

1 h(2) 1

+ w(o)

2 h(2) 2

+ . . . + w(o)

d2 h(2) d2 + b(o))

<latexit sha1_base64="bOumy7ml4JKyWoAXDVdeW0BwETI=">ACT3icbZHNS8MwGMbT+bXNr6lHL8EhbAxGWwS9CEMvHie4D9hmSdN0C6ZNSVJlP2HXvTmv+HFgyJmXR1z84XAL8/zviR54kaMSmWab0ZubX1jcytfKG7v7O7tlw4O25LHApMW5oyLroskYTQkLUVI91IEBS4jHTch+up3kQlIe3qlxRAYBGobUpxgpLTklfwvYV/SYAqT/dJhVcnjgVHmuyUavBXteqrdU+87iSC3biOfZk3jLb1aA7c6tOqWzWzbTgKlgZlEFWTaf02vc4jgMSKsyQlD3LjNQgQUJRzMik2I8liRB+QEPS0xigMhBkuYxgada8aDPhV6hgqm6OJGgQMpx4OrOAKmRXPam4n9eL1b+xSChYRQrEuLZQX7MoOJwGi70qCBYsbEGhAXVd4V4hATCSn9BUYdgLT95Fdp23TLr1u1ZuXGVxZEHx+AEVIAFzkED3IAmaAEMnsE7+ARfxovxYXznstackcER+FO5wg8T4K6n</latexit><latexit sha1_base64="bOumy7ml4JKyWoAXDVdeW0BwETI=">ACT3icbZHNS8MwGMbT+bXNr6lHL8EhbAxGWwS9CEMvHie4D9hmSdN0C6ZNSVJlP2HXvTmv+HFgyJmXR1z84XAL8/zviR54kaMSmWab0ZubX1jcytfKG7v7O7tlw4O25LHApMW5oyLroskYTQkLUVI91IEBS4jHTch+up3kQlIe3qlxRAYBGobUpxgpLTklfwvYV/SYAqT/dJhVcnjgVHmuyUavBXteqrdU+87iSC3biOfZk3jLb1aA7c6tOqWzWzbTgKlgZlEFWTaf02vc4jgMSKsyQlD3LjNQgQUJRzMik2I8liRB+QEPS0xigMhBkuYxgada8aDPhV6hgqm6OJGgQMpx4OrOAKmRXPam4n9eL1b+xSChYRQrEuLZQX7MoOJwGi70qCBYsbEGhAXVd4V4hATCSn9BUYdgLT95Fdp23TLr1u1ZuXGVxZEHx+AEVIAFzkED3IAmaAEMnsE7+ARfxovxYXznstackcER+FO5wg8T4K6n</latexit><latexit sha1_base64="bOumy7ml4JKyWoAXDVdeW0BwETI=">ACT3icbZHNS8MwGMbT+bXNr6lHL8EhbAxGWwS9CEMvHie4D9hmSdN0C6ZNSVJlP2HXvTmv+HFgyJmXR1z84XAL8/zviR54kaMSmWab0ZubX1jcytfKG7v7O7tlw4O25LHApMW5oyLroskYTQkLUVI91IEBS4jHTch+up3kQlIe3qlxRAYBGobUpxgpLTklfwvYV/SYAqT/dJhVcnjgVHmuyUavBXteqrdU+87iSC3biOfZk3jLb1aA7c6tOqWzWzbTgKlgZlEFWTaf02vc4jgMSKsyQlD3LjNQgQUJRzMik2I8liRB+QEPS0xigMhBkuYxgada8aDPhV6hgqm6OJGgQMpx4OrOAKmRXPam4n9eL1b+xSChYRQrEuLZQX7MoOJwGi70qCBYsbEGhAXVd4V4hATCSn9BUYdgLT95Fdp23TLr1u1ZuXGVxZEHx+AEVIAFzkED3IAmaAEMnsE7+ARfxovxYXznstackcER+FO5wg8T4K6n</latexit><latexit sha1_base64="bOumy7ml4JKyWoAXDVdeW0BwETI=">ACT3icbZHNS8MwGMbT+bXNr6lHL8EhbAxGWwS9CEMvHie4D9hmSdN0C6ZNSVJlP2HXvTmv+HFgyJmXR1z84XAL8/zviR54kaMSmWab0ZubX1jcytfKG7v7O7tlw4O25LHApMW5oyLroskYTQkLUVI91IEBS4jHTch+up3kQlIe3qlxRAYBGobUpxgpLTklfwvYV/SYAqT/dJhVcnjgVHmuyUavBXteqrdU+87iSC3biOfZk3jLb1aA7c6tOqWzWzbTgKlgZlEFWTaf02vc4jgMSKsyQlD3LjNQgQUJRzMik2I8liRB+QEPS0xigMhBkuYxgada8aDPhV6hgqm6OJGgQMpx4OrOAKmRXPam4n9eL1b+xSChYRQrEuLZQX7MoOJwGi70qCBYsbEGhAXVd4V4hATCSn9BUYdgLT95Fdp23TLr1u1ZuXGVxZEHx+AEVIAFzkED3IAmaAEMnsE7+ARfxovxYXznstackcER+FO5wg8T4K6n</latexit>
  • Hidden layer 1: h(1)

1 , h(1) 2 , …, h(1) d1

· · ·

<latexit sha1_base64="C/ThE3LO0bsU2xdfe5suNCnJ9S4=">AB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NKPzM8pWxMh7xnaUJjbvx8fu2UnFklJHSthIkc/X3RE5jYyZxYDtjiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB0oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4ArlePLw=</latexit><latexit sha1_base64="C/ThE3LO0bsU2xdfe5suNCnJ9S4=">AB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NKPzM8pWxMh7xnaUJjbvx8fu2UnFklJHSthIkc/X3RE5jYyZxYDtjiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB0oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4ArlePLw=</latexit><latexit sha1_base64="C/ThE3LO0bsU2xdfe5suNCnJ9S4=">AB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NKPzM8pWxMh7xnaUJjbvx8fu2UnFklJHSthIkc/X3RE5jYyZxYDtjiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB0oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4ArlePLw=</latexit><latexit sha1_base64="C/ThE3LO0bsU2xdfe5suNCnJ9S4=">AB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3WehQjOo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NKPzM8pWxMh7xnaUJjbvx8fu2UnFklJHSthIkc/X3RE5jYyZxYDtjiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB0oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4ArlePLw=</latexit>

h(1)

1

= f(W (1)

1,1 + W (1) 1,2 x2 + . . . + W (1) 1,d xd + b(1) 1 )

<latexit sha1_base64="xyKVt0I2x1mI6pfX2c5pytM/Ws=">ACQHicbVDLSgMxFM34rPU16tJNsAgtSpkUQRcKBTcuK9gHtHXIZDJtaOZBkhHLMJ/mxk9w59qNC0XcujJtR+jDA4Fz7n3JjlOxJlUlvVqLC2vrK6t5zbym1vbO7vm3n5DhrEgtE5CHoqWgyXlLKB1xRSnrUhQ7DucNp3B9chvPlAhWRjcqWFEuz7uBcxjBCst2Wazf58USm1EbyCXrGZVQk6RSk8gVN1JX20K1rqcDdUctZztedqyflbVrLNglW2xoCLBGWkADLUbPOl4Yk9mgCMdStpEVqW6ChWKE0zTfiSWNMBngHm1rGmCfym4yDiCFx1pxoRcKfQIFx+r0RIJ9KYe+ozt9rPpy3huJ/3ntWHkX3YQFUaxoQCYXeTGHKoSjNKHLBCWKDzXBRD9Vkj6WGCidOZ5HQKa/IiaVTKyCqj27NC9TKLIwcOwREoAgTOQRXcgBqoAwKewBv4AJ/Gs/FufBnfk9YlI5s5ADMwfn4BapKpuw=</latexit><latexit sha1_base64="xyKVt0I2x1mI6pfX2c5pytM/Ws=">ACQHicbVDLSgMxFM34rPU16tJNsAgtSpkUQRcKBTcuK9gHtHXIZDJtaOZBkhHLMJ/mxk9w59qNC0XcujJtR+jDA4Fz7n3JjlOxJlUlvVqLC2vrK6t5zbym1vbO7vm3n5DhrEgtE5CHoqWgyXlLKB1xRSnrUhQ7DucNp3B9chvPlAhWRjcqWFEuz7uBcxjBCst2Wazf58USm1EbyCXrGZVQk6RSk8gVN1JX20K1rqcDdUctZztedqyflbVrLNglW2xoCLBGWkADLUbPOl4Yk9mgCMdStpEVqW6ChWKE0zTfiSWNMBngHm1rGmCfym4yDiCFx1pxoRcKfQIFx+r0RIJ9KYe+ozt9rPpy3huJ/3ntWHkX3YQFUaxoQCYXeTGHKoSjNKHLBCWKDzXBRD9Vkj6WGCidOZ5HQKa/IiaVTKyCqj27NC9TKLIwcOwREoAgTOQRXcgBqoAwKewBv4AJ/Gs/FufBnfk9YlI5s5ADMwfn4BapKpuw=</latexit><latexit sha1_base64="xyKVt0I2x1mI6pfX2c5pytM/Ws=">ACQHicbVDLSgMxFM34rPU16tJNsAgtSpkUQRcKBTcuK9gHtHXIZDJtaOZBkhHLMJ/mxk9w59qNC0XcujJtR+jDA4Fz7n3JjlOxJlUlvVqLC2vrK6t5zbym1vbO7vm3n5DhrEgtE5CHoqWgyXlLKB1xRSnrUhQ7DucNp3B9chvPlAhWRjcqWFEuz7uBcxjBCst2Wazf58USm1EbyCXrGZVQk6RSk8gVN1JX20K1rqcDdUctZztedqyflbVrLNglW2xoCLBGWkADLUbPOl4Yk9mgCMdStpEVqW6ChWKE0zTfiSWNMBngHm1rGmCfym4yDiCFx1pxoRcKfQIFx+r0RIJ9KYe+ozt9rPpy3huJ/3ntWHkX3YQFUaxoQCYXeTGHKoSjNKHLBCWKDzXBRD9Vkj6WGCidOZ5HQKa/IiaVTKyCqj27NC9TKLIwcOwREoAgTOQRXcgBqoAwKewBv4AJ/Gs/FufBnfk9YlI5s5ADMwfn4BapKpuw=</latexit><latexit sha1_base64="xyKVt0I2x1mI6pfX2c5pytM/Ws=">ACQHicbVDLSgMxFM34rPU16tJNsAgtSpkUQRcKBTcuK9gHtHXIZDJtaOZBkhHLMJ/mxk9w59qNC0XcujJtR+jDA4Fz7n3JjlOxJlUlvVqLC2vrK6t5zbym1vbO7vm3n5DhrEgtE5CHoqWgyXlLKB1xRSnrUhQ7DucNp3B9chvPlAhWRjcqWFEuz7uBcxjBCst2Wazf58USm1EbyCXrGZVQk6RSk8gVN1JX20K1rqcDdUctZztedqyflbVrLNglW2xoCLBGWkADLUbPOl4Yk9mgCMdStpEVqW6ChWKE0zTfiSWNMBngHm1rGmCfym4yDiCFx1pxoRcKfQIFx+r0RIJ9KYe+ozt9rPpy3huJ/3ntWHkX3YQFUaxoQCYXeTGHKoSjNKHLBCWKDzXBRD9Vkj6WGCidOZ5HQKa/IiaVTKyCqj27NC9TKLIwcOwREoAgTOQRXcgBqoAwKewBv4AJ/Gs/FufBnfk9YlI5s5ADMwfn4BapKpuw=</latexit>

h(1)

2

= f(W (1)

2,1 + W (1) 2,2 x2 + . . . + W (1) 2,d xd + b(1) 2 )

<latexit sha1_base64="OoxWd30Tt1rq58Rry91hetKvsxo=">ACQHicbVDLSgMxFM3UV62vqks3wSK0VMrMIOhCoeDGZQX7gHYcMplMG5p5kGTEMsynufET3Ll240IRt65M2xH68ELg3HPOvUmOEzEqpK6/armV1bX1jfxmYWt7Z3evuH/QEmHMWnikIW84yBGA1IU1LJSCfiBPkOI21neD3W2w+ECxoGd3IUEctH/YB6FCOpKLvYHtwnZaOS2ia8gl65nXWJeWqksApnejN9VKYq7DE3lGJec5XmKsr5W1axiyW9pk8KLgMjAyWQVcMuvTcEMc+CSRmSIiuoUfShCXFDOSFnqxIBHCQ9QnXQUD5BNhJZMAUniGBd6IVcnkHDCzk4kyBdi5DvK6SM5EIvamPxP68bSu7ASGkSxJAGeXuTFDMoQjtOELuUESzZSAGFO1VshHiCOsFSZF1QIxuKXl0HLrBl6zbg9K9Uvszjy4AgcgzIwDmogxvQAE2AwRN4Ax/gU3vW3rUv7XtqzWnZzCGYK+3nF3L2qcA=</latexit><latexit sha1_base64="OoxWd30Tt1rq58Rry91hetKvsxo=">ACQHicbVDLSgMxFM3UV62vqks3wSK0VMrMIOhCoeDGZQX7gHYcMplMG5p5kGTEMsynufET3Ll240IRt65M2xH68ELg3HPOvUmOEzEqpK6/armV1bX1jfxmYWt7Z3evuH/QEmHMWnikIW84yBGA1IU1LJSCfiBPkOI21neD3W2w+ECxoGd3IUEctH/YB6FCOpKLvYHtwnZaOS2ia8gl65nXWJeWqksApnejN9VKYq7DE3lGJec5XmKsr5W1axiyW9pk8KLgMjAyWQVcMuvTcEMc+CSRmSIiuoUfShCXFDOSFnqxIBHCQ9QnXQUD5BNhJZMAUniGBd6IVcnkHDCzk4kyBdi5DvK6SM5EIvamPxP68bSu7ASGkSxJAGeXuTFDMoQjtOELuUESzZSAGFO1VshHiCOsFSZF1QIxuKXl0HLrBl6zbg9K9Uvszjy4AgcgzIwDmogxvQAE2AwRN4Ax/gU3vW3rUv7XtqzWnZzCGYK+3nF3L2qcA=</latexit><latexit sha1_base64="OoxWd30Tt1rq58Rry91hetKvsxo=">ACQHicbVDLSgMxFM3UV62vqks3wSK0VMrMIOhCoeDGZQX7gHYcMplMG5p5kGTEMsynufET3Ll240IRt65M2xH68ELg3HPOvUmOEzEqpK6/armV1bX1jfxmYWt7Z3evuH/QEmHMWnikIW84yBGA1IU1LJSCfiBPkOI21neD3W2w+ECxoGd3IUEctH/YB6FCOpKLvYHtwnZaOS2ia8gl65nXWJeWqksApnejN9VKYq7DE3lGJec5XmKsr5W1axiyW9pk8KLgMjAyWQVcMuvTcEMc+CSRmSIiuoUfShCXFDOSFnqxIBHCQ9QnXQUD5BNhJZMAUniGBd6IVcnkHDCzk4kyBdi5DvK6SM5EIvamPxP68bSu7ASGkSxJAGeXuTFDMoQjtOELuUESzZSAGFO1VshHiCOsFSZF1QIxuKXl0HLrBl6zbg9K9Uvszjy4AgcgzIwDmogxvQAE2AwRN4Ax/gU3vW3rUv7XtqzWnZzCGYK+3nF3L2qcA=</latexit><latexit sha1_base64="OoxWd30Tt1rq58Rry91hetKvsxo=">ACQHicbVDLSgMxFM3UV62vqks3wSK0VMrMIOhCoeDGZQX7gHYcMplMG5p5kGTEMsynufET3Ll240IRt65M2xH68ELg3HPOvUmOEzEqpK6/armV1bX1jfxmYWt7Z3evuH/QEmHMWnikIW84yBGA1IU1LJSCfiBPkOI21neD3W2w+ECxoGd3IUEctH/YB6FCOpKLvYHtwnZaOS2ia8gl65nXWJeWqksApnejN9VKYq7DE3lGJec5XmKsr5W1axiyW9pk8KLgMjAyWQVcMuvTcEMc+CSRmSIiuoUfShCXFDOSFnqxIBHCQ9QnXQUD5BNhJZMAUniGBd6IVcnkHDCzk4kyBdi5DvK6SM5EIvamPxP68bSu7ASGkSxJAGeXuTFDMoQjtOELuUESzZSAGFO1VshHiCOsFSZF1QIxuKXl0HLrBl6zbg9K9Uvszjy4AgcgzIwDmogxvQAE2AwRN4Ax/gU3vW3rUv7XtqzWnZzCGYK+3nF3L2qcA=</latexit>
  • Hidden layer 1:

h1 = f(W(1)x + b(1)) ∈ Rd1

<latexit sha1_base64="VM7VTSrhdxVPv8H7p1W7pTPe9Nc=">ACPXicbVDLSgMxFM3UV62vqks3wSK0CGUigm6EohuXVfqCtg6ZNOGZjJDkhHLMD/mxn9w586NC0XcujVtp6CtBwLnMvufe4IWdK2/aLlVlaXldy67nNja3tnfyu3sNFUS0DoJeCBbLlaUM0HrmlOW6Gk2Hc5brDq7HfvKdSsUDU9CikXR/3BfMYwdpITr7W8bEeuF48SBwEL6BXnAnN5C4uolIyqx8SeAxnhZuaJdhIlXd+NaoPQclTr5gl+0J4CJBKSmAFUn/9zpBSTyqdCEY6XayA51N8ZSM8JpkutEioaYDHGftg0V2KeqG0+uT+CRUXrQC6R5QsOJ+nsixr5SI981neM91bw3Fv/z2pH2zrsxE2GkqSDTj7yIQx3AcZSwxyQlmo8MwUQysyskAywx0SbwnAkBzZ+8SBonZWSX0c1poXKZxpEFB+AQFAECZ6ACrkEV1AEBj+AVvIMP68l6sz6tr2lrxkpn9sEfWN8/uPeuVA=</latexit><latexit sha1_base64="VM7VTSrhdxVPv8H7p1W7pTPe9Nc=">ACPXicbVDLSgMxFM3UV62vqks3wSK0CGUigm6EohuXVfqCtg6ZNOGZjJDkhHLMD/mxn9w586NC0XcujVtp6CtBwLnMvufe4IWdK2/aLlVlaXldy67nNja3tnfyu3sNFUS0DoJeCBbLlaUM0HrmlOW6Gk2Hc5brDq7HfvKdSsUDU9CikXR/3BfMYwdpITr7W8bEeuF48SBwEL6BXnAnN5C4uolIyqx8SeAxnhZuaJdhIlXd+NaoPQclTr5gl+0J4CJBKSmAFUn/9zpBSTyqdCEY6XayA51N8ZSM8JpkutEioaYDHGftg0V2KeqG0+uT+CRUXrQC6R5QsOJ+nsixr5SI981neM91bw3Fv/z2pH2zrsxE2GkqSDTj7yIQx3AcZSwxyQlmo8MwUQysyskAywx0SbwnAkBzZ+8SBonZWSX0c1poXKZxpEFB+AQFAECZ6ACrkEV1AEBj+AVvIMP68l6sz6tr2lrxkpn9sEfWN8/uPeuVA=</latexit><latexit sha1_base64="VM7VTSrhdxVPv8H7p1W7pTPe9Nc=">ACPXicbVDLSgMxFM3UV62vqks3wSK0CGUigm6EohuXVfqCtg6ZNOGZjJDkhHLMD/mxn9w586NC0XcujVtp6CtBwLnMvufe4IWdK2/aLlVlaXldy67nNja3tnfyu3sNFUS0DoJeCBbLlaUM0HrmlOW6Gk2Hc5brDq7HfvKdSsUDU9CikXR/3BfMYwdpITr7W8bEeuF48SBwEL6BXnAnN5C4uolIyqx8SeAxnhZuaJdhIlXd+NaoPQclTr5gl+0J4CJBKSmAFUn/9zpBSTyqdCEY6XayA51N8ZSM8JpkutEioaYDHGftg0V2KeqG0+uT+CRUXrQC6R5QsOJ+nsixr5SI981neM91bw3Fv/z2pH2zrsxE2GkqSDTj7yIQx3AcZSwxyQlmo8MwUQysyskAywx0SbwnAkBzZ+8SBonZWSX0c1poXKZxpEFB+AQFAECZ6ACrkEV1AEBj+AVvIMP68l6sz6tr2lrxkpn9sEfWN8/uPeuVA=</latexit><latexit sha1_base64="VM7VTSrhdxVPv8H7p1W7pTPe9Nc=">ACPXicbVDLSgMxFM3UV62vqks3wSK0CGUigm6EohuXVfqCtg6ZNOGZjJDkhHLMD/mxn9w586NC0XcujVtp6CtBwLnMvufe4IWdK2/aLlVlaXldy67nNja3tnfyu3sNFUS0DoJeCBbLlaUM0HrmlOW6Gk2Hc5brDq7HfvKdSsUDU9CikXR/3BfMYwdpITr7W8bEeuF48SBwEL6BXnAnN5C4uolIyqx8SeAxnhZuaJdhIlXd+NaoPQclTr5gl+0J4CJBKSmAFUn/9zpBSTyqdCEY6XayA51N8ZSM8JpkutEioaYDHGftg0V2KeqG0+uT+CRUXrQC6R5QsOJ+nsixr5SI981neM91bw3Fv/z2pH2zrsxE2GkqSDTj7yIQx3AcZSwxyQlmo8MwUQysyskAywx0SbwnAkBzZ+8SBonZWSX0c1poXKZxpEFB+AQFAECZ6ACrkEV1AEBj+AVvIMP68l6sz6tr2lrxkpn9sEfWN8/uPeuVA=</latexit>

W(2) ∈ Rd2×d1, b(2) ∈ Rd2

<latexit sha1_base64="Nn6kKAXiHkWvk29ucTEHXmcjE8=">ACQHicfVDPS8MwGE39OevqUcvwSFMkNEOQY9DLx6nuHWw1pKm6RaWpiVJhVH6p3nxT/Dm2YsHRbx6Mtsq6CY+CDze9z2+l+cnjEplmk/GwuLS8spqa28vrG5tV3Z2e3IOBWYtHMYtH1kSMctJWVDHSTQRBkc+I7Q8vxnP7jghJY36jRglxI9TnNKQYKS15FduJkBr4YWbnt1mtcZRDh3I4Ff3sWouB14COohGRMPCs/Bh+O/x/HLlXqZp1cwI4T6yCVEGBld5dIYpxHhCjMkZc8yE+VmSCiKGcnLTipJgvAQ9UlPU450IDebFJDQ60EMIyFflzBifrTkaFIylHk681xTjk7G4t/zXqpCs/cjPIkVYTj6aEwZVDFcNwmDKgWLGRJgLqrNCPEACYaU7L+sSrNkvz5NOo26ZdevqpNo8L+ogX1wAGrAqegCS5BC7QBvfgGbyCN+PBeDHejY/p6oJRePbALxifX2aAr6o=</latexit><latexit sha1_base64="Nn6kKAXiHkWvk29ucTEHXmcjE8=">ACQHicfVDPS8MwGE39OevqUcvwSFMkNEOQY9DLx6nuHWw1pKm6RaWpiVJhVH6p3nxT/Dm2YsHRbx6Mtsq6CY+CDze9z2+l+cnjEplmk/GwuLS8spqa28vrG5tV3Z2e3IOBWYtHMYtH1kSMctJWVDHSTQRBkc+I7Q8vxnP7jghJY36jRglxI9TnNKQYKS15FduJkBr4YWbnt1mtcZRDh3I4Ff3sWouB14COohGRMPCs/Bh+O/x/HLlXqZp1cwI4T6yCVEGBld5dIYpxHhCjMkZc8yE+VmSCiKGcnLTipJgvAQ9UlPU450IDebFJDQ60EMIyFflzBifrTkaFIylHk681xTjk7G4t/zXqpCs/cjPIkVYTj6aEwZVDFcNwmDKgWLGRJgLqrNCPEACYaU7L+sSrNkvz5NOo26ZdevqpNo8L+ogX1wAGrAqegCS5BC7QBvfgGbyCN+PBeDHejY/p6oJRePbALxifX2aAr6o=</latexit><latexit sha1_base64="Nn6kKAXiHkWvk29ucTEHXmcjE8=">ACQHicfVDPS8MwGE39OevqUcvwSFMkNEOQY9DLx6nuHWw1pKm6RaWpiVJhVH6p3nxT/Dm2YsHRbx6Mtsq6CY+CDze9z2+l+cnjEplmk/GwuLS8spqa28vrG5tV3Z2e3IOBWYtHMYtH1kSMctJWVDHSTQRBkc+I7Q8vxnP7jghJY36jRglxI9TnNKQYKS15FduJkBr4YWbnt1mtcZRDh3I4Ff3sWouB14COohGRMPCs/Bh+O/x/HLlXqZp1cwI4T6yCVEGBld5dIYpxHhCjMkZc8yE+VmSCiKGcnLTipJgvAQ9UlPU450IDebFJDQ60EMIyFflzBifrTkaFIylHk681xTjk7G4t/zXqpCs/cjPIkVYTj6aEwZVDFcNwmDKgWLGRJgLqrNCPEACYaU7L+sSrNkvz5NOo26ZdevqpNo8L+ogX1wAGrAqegCS5BC7QBvfgGbyCN+PBeDHejY/p6oJRePbALxifX2aAr6o=</latexit><latexit sha1_base64="Nn6kKAXiHkWvk29ucTEHXmcjE8=">ACQHicfVDPS8MwGE39OevqUcvwSFMkNEOQY9DLx6nuHWw1pKm6RaWpiVJhVH6p3nxT/Dm2YsHRbx6Mtsq6CY+CDze9z2+l+cnjEplmk/GwuLS8spqa28vrG5tV3Z2e3IOBWYtHMYtH1kSMctJWVDHSTQRBkc+I7Q8vxnP7jghJY36jRglxI9TnNKQYKS15FduJkBr4YWbnt1mtcZRDh3I4Ff3sWouB14COohGRMPCs/Bh+O/x/HLlXqZp1cwI4T6yCVEGBld5dIYpxHhCjMkZc8yE+VmSCiKGcnLTipJgvAQ9UlPU450IDebFJDQ60EMIyFflzBifrTkaFIylHk681xTjk7G4t/zXqpCs/cjPIkVYTj6aEwZVDFcNwmDKgWLGRJgLqrNCPEACYaU7L+sSrNkvz5NOo26ZdevqpNo8L+ogX1wAGrAqegCS5BC7QBvfgGbyCN+PBeDHejY/p6oJRePbALxifX2aAr6o=</latexit>
  • Hidden layer 2:

h2 = f(W(2)h1 + b(2)) ∈ Rd2

<latexit sha1_base64="ti/zp5Hv7rAiTgW9sNoVe9BYoI=">ACP3icbZDNS8MwGMbT+TXn19Sjl+AQNoTRDkEvwtCLxynuA9Z0izdwtK0JKkwSv8zL/4L3rx68aCIV2+mWwdz84XAk9/zvuTN4aMSmWar0ZuZXVtfSO/Wdja3tndK+4ftGQCUyaOGCB6LhIEkY5aSqGOmEgiDfZaTtjq5Tv/1IhKQBv1fjkPR8NODUoxgpjZxiy/aRGrpePEycGryEXnkG2slDXK5VkrkGC57C2dXN7Aq0Kc+oG9p2ndqiVMsmVzUnBZWJkogawaTvHF7gc48glXmCEpu5YZql6MhKYkaRgR5KECI/QgHS15MgnshdP/p/AE0360AuEPlzBCZ2fiJEv5dh3dWe6p1z0Uvif142Ud9GLKQ8jRTiePuRFDKoApmHCPhUEKzbWAmFB9a4QD5FAWOnICzoEa/HLy6JVq1pm1bo9K9Wvsjy4AgcgzKwDmogxvQAE2AwRN4Ax/g03g23o0v43vamjOymUPwp4yfXwG6ruw=</latexit><latexit sha1_base64="ti/zp5Hv7rAiTgW9sNoVe9BYoI=">ACP3icbZDNS8MwGMbT+TXn19Sjl+AQNoTRDkEvwtCLxynuA9Z0izdwtK0JKkwSv8zL/4L3rx68aCIV2+mWwdz84XAk9/zvuTN4aMSmWar0ZuZXVtfSO/Wdja3tndK+4ftGQCUyaOGCB6LhIEkY5aSqGOmEgiDfZaTtjq5Tv/1IhKQBv1fjkPR8NODUoxgpjZxiy/aRGrpePEycGryEXnkG2slDXK5VkrkGC57C2dXN7Aq0Kc+oG9p2ndqiVMsmVzUnBZWJkogawaTvHF7gc48glXmCEpu5YZql6MhKYkaRgR5KECI/QgHS15MgnshdP/p/AE0360AuEPlzBCZ2fiJEv5dh3dWe6p1z0Uvif142Ud9GLKQ8jRTiePuRFDKoApmHCPhUEKzbWAmFB9a4QD5FAWOnICzoEa/HLy6JVq1pm1bo9K9Wvsjy4AgcgzKwDmogxvQAE2AwRN4Ax/g03g23o0v43vamjOymUPwp4yfXwG6ruw=</latexit><latexit sha1_base64="ti/zp5Hv7rAiTgW9sNoVe9BYoI=">ACP3icbZDNS8MwGMbT+TXn19Sjl+AQNoTRDkEvwtCLxynuA9Z0izdwtK0JKkwSv8zL/4L3rx68aCIV2+mWwdz84XAk9/zvuTN4aMSmWar0ZuZXVtfSO/Wdja3tndK+4ftGQCUyaOGCB6LhIEkY5aSqGOmEgiDfZaTtjq5Tv/1IhKQBv1fjkPR8NODUoxgpjZxiy/aRGrpePEycGryEXnkG2slDXK5VkrkGC57C2dXN7Aq0Kc+oG9p2ndqiVMsmVzUnBZWJkogawaTvHF7gc48glXmCEpu5YZql6MhKYkaRgR5KECI/QgHS15MgnshdP/p/AE0360AuEPlzBCZ2fiJEv5dh3dWe6p1z0Uvif142Ud9GLKQ8jRTiePuRFDKoApmHCPhUEKzbWAmFB9a4QD5FAWOnICzoEa/HLy6JVq1pm1bo9K9Wvsjy4AgcgzKwDmogxvQAE2AwRN4Ax/g03g23o0v43vamjOymUPwp4yfXwG6ruw=</latexit><latexit sha1_base64="ti/zp5Hv7rAiTgW9sNoVe9BYoI=">ACP3icbZDNS8MwGMbT+TXn19Sjl+AQNoTRDkEvwtCLxynuA9Z0izdwtK0JKkwSv8zL/4L3rx68aCIV2+mWwdz84XAk9/zvuTN4aMSmWar0ZuZXVtfSO/Wdja3tndK+4ftGQCUyaOGCB6LhIEkY5aSqGOmEgiDfZaTtjq5Tv/1IhKQBv1fjkPR8NODUoxgpjZxiy/aRGrpePEycGryEXnkG2slDXK5VkrkGC57C2dXN7Aq0Kc+oG9p2ndqiVMsmVzUnBZWJkogawaTvHF7gc48glXmCEpu5YZql6MhKYkaRgR5KECI/QgHS15MgnshdP/p/AE0360AuEPlzBCZ2fiJEv5dh3dWe6p1z0Uvif142Ud9GLKQ8jRTiePuRFDKoApmHCPhUEKzbWAmFB9a4QD5FAWOnICzoEa/HLy6JVq1pm1bo9K9Wvsjy4AgcgzKwDmogxvQAE2AwRN4Ax/g03g23o0v43vamjOymUPwp4yfXwG6ruw=</latexit>
  • Output layer:

y = σ(w(o) · h2 + b(o))

<latexit sha1_base64="xwf5NTK8Os9frFEQg9QWL1GKg9Q=">ACInicbVDLSsNAFJ34rPUVdelmsAgtQkmKoC6EohuXFewDmlgm0k7dJIJMxOlhHyLG3/FjQtFXQl+jJM2grYeuHA4517uvceLGJXKsj6NhcWl5ZXVwlpxfWNza9vc2W1JHgtMmpgzLjoekoTRkDQVYx0IkFQ4DHS9kaXmd+I0JSHt6ocUTcA1C6lOMlJZ65tkYnkNH0kGAyk6A1NDzk/v0NinzSgod3OcK/sjDtFeDR9CbmpWeWbKq1gRwntg5KYEcjZ757vQ5jgMSKsyQlF3bipSbIKEoZiQtOrEkEcIjNCBdTUMUEOkmkxdTeKiVPvS50BUqOF/TyQokHIceLozO1fOepn4n9eNlX/qJjSMYkVCPF3kxwqDrO8YJ8KghUba4KwoPpWiIdIKx0qkUdgj378jxp1aq2VbWvj0v1izyOAtgHB6AMbHAC6uAKNEATYPAnsALeDUejWfjzfiYti4Y+cwe+APj6xs+HKLU</latexit><latexit sha1_base64="xwf5NTK8Os9frFEQg9QWL1GKg9Q=">ACInicbVDLSsNAFJ34rPUVdelmsAgtQkmKoC6EohuXFewDmlgm0k7dJIJMxOlhHyLG3/FjQtFXQl+jJM2grYeuHA4517uvceLGJXKsj6NhcWl5ZXVwlpxfWNza9vc2W1JHgtMmpgzLjoekoTRkDQVYx0IkFQ4DHS9kaXmd+I0JSHt6ocUTcA1C6lOMlJZ65tkYnkNH0kGAyk6A1NDzk/v0NinzSgod3OcK/sjDtFeDR9CbmpWeWbKq1gRwntg5KYEcjZ757vQ5jgMSKsyQlF3bipSbIKEoZiQtOrEkEcIjNCBdTUMUEOkmkxdTeKiVPvS50BUqOF/TyQokHIceLozO1fOepn4n9eNlX/qJjSMYkVCPF3kxwqDrO8YJ8KghUba4KwoPpWiIdIKx0qkUdgj378jxp1aq2VbWvj0v1izyOAtgHB6AMbHAC6uAKNEATYPAnsALeDUejWfjzfiYti4Y+cwe+APj6xs+HKLU</latexit><latexit sha1_base64="xwf5NTK8Os9frFEQg9QWL1GKg9Q=">ACInicbVDLSsNAFJ34rPUVdelmsAgtQkmKoC6EohuXFewDmlgm0k7dJIJMxOlhHyLG3/FjQtFXQl+jJM2grYeuHA4517uvceLGJXKsj6NhcWl5ZXVwlpxfWNza9vc2W1JHgtMmpgzLjoekoTRkDQVYx0IkFQ4DHS9kaXmd+I0JSHt6ocUTcA1C6lOMlJZ65tkYnkNH0kGAyk6A1NDzk/v0NinzSgod3OcK/sjDtFeDR9CbmpWeWbKq1gRwntg5KYEcjZ757vQ5jgMSKsyQlF3bipSbIKEoZiQtOrEkEcIjNCBdTUMUEOkmkxdTeKiVPvS50BUqOF/TyQokHIceLozO1fOepn4n9eNlX/qJjSMYkVCPF3kxwqDrO8YJ8KghUba4KwoPpWiIdIKx0qkUdgj378jxp1aq2VbWvj0v1izyOAtgHB6AMbHAC6uAKNEATYPAnsALeDUejWfjzfiYti4Y+cwe+APj6xs+HKLU</latexit><latexit sha1_base64="xwf5NTK8Os9frFEQg9QWL1GKg9Q=">ACInicbVDLSsNAFJ34rPUVdelmsAgtQkmKoC6EohuXFewDmlgm0k7dJIJMxOlhHyLG3/FjQtFXQl+jJM2grYeuHA4517uvceLGJXKsj6NhcWl5ZXVwlpxfWNza9vc2W1JHgtMmpgzLjoekoTRkDQVYx0IkFQ4DHS9kaXmd+I0JSHt6ocUTcA1C6lOMlJZ65tkYnkNH0kGAyk6A1NDzk/v0NinzSgod3OcK/sjDtFeDR9CbmpWeWbKq1gRwntg5KYEcjZ757vQ5jgMSKsyQlF3bipSbIKEoZiQtOrEkEcIjNCBdTUMUEOkmkxdTeKiVPvS50BUqOF/TyQokHIceLozO1fOepn4n9eNlX/qJjSMYkVCPF3kxwqDrO8YJ8KghUba4KwoPpWiIdIKx0qkUdgj378jxp1aq2VbWvj0v1izyOAtgHB6AMbHAC6uAKNEATYPAnsALeDUejWfjzfiYti4Y+cwe+APj6xs+HKLU</latexit>

x ∈ Rd

<latexit sha1_base64="r8cLpKiZJtXS/wgD8iCVci7o1w=">ACBHicbVC7TsMwFL3hWcorwNjFokJiqhKEBANDJRbGguhDakrlOE5r1XEi20FUQYWfoWFAYRY+Qg2/gan7QAtR7J0fM69uvceP+FMacf5tpaWV1bX1ksb5c2t7Z1de2+/peJUEtokMY9lx8eKciZoUzPNaSeRFEc+p21/dFn47XsqFYvFrR4ntBfhgWAhI1gbqW9XvAjroR9mDznymEDTr5/d5HdB3646NWcCtEjcGanCDI2+/eUFMUkjKjThWKmu6yS6l2GpGeE0L3upogkmIzygXUMFjqjqZMjcnRklACFsTRPaDRf3dkOFJqHPmsthRzXuF+J/XTXV43suYSFJNBZkOClOdIyKRFDAJCWajw3BRDKzKyJDLDHRJreyCcGdP3mRtE5qrlNzr0+r9YtZHCWowCEcgwtnUIcraEATCDzCM7zCm/VkvVjv1se0dMma9RzAH1ifP+g+mDo=</latexit><latexit sha1_base64="r8cLpKiZJtXS/wgD8iCVci7o1w=">ACBHicbVC7TsMwFL3hWcorwNjFokJiqhKEBANDJRbGguhDakrlOE5r1XEi20FUQYWfoWFAYRY+Qg2/gan7QAtR7J0fM69uvceP+FMacf5tpaWV1bX1ksb5c2t7Z1de2+/peJUEtokMY9lx8eKciZoUzPNaSeRFEc+p21/dFn47XsqFYvFrR4ntBfhgWAhI1gbqW9XvAjroR9mDznymEDTr5/d5HdB3646NWcCtEjcGanCDI2+/eUFMUkjKjThWKmu6yS6l2GpGeE0L3upogkmIzygXUMFjqjqZMjcnRklACFsTRPaDRf3dkOFJqHPmsthRzXuF+J/XTXV43suYSFJNBZkOClOdIyKRFDAJCWajw3BRDKzKyJDLDHRJreyCcGdP3mRtE5qrlNzr0+r9YtZHCWowCEcgwtnUIcraEATCDzCM7zCm/VkvVjv1se0dMma9RzAH1ifP+g+mDo=</latexit><latexit sha1_base64="r8cLpKiZJtXS/wgD8iCVci7o1w=">ACBHicbVC7TsMwFL3hWcorwNjFokJiqhKEBANDJRbGguhDakrlOE5r1XEi20FUQYWfoWFAYRY+Qg2/gan7QAtR7J0fM69uvceP+FMacf5tpaWV1bX1ksb5c2t7Z1de2+/peJUEtokMY9lx8eKciZoUzPNaSeRFEc+p21/dFn47XsqFYvFrR4ntBfhgWAhI1gbqW9XvAjroR9mDznymEDTr5/d5HdB3646NWcCtEjcGanCDI2+/eUFMUkjKjThWKmu6yS6l2GpGeE0L3upogkmIzygXUMFjqjqZMjcnRklACFsTRPaDRf3dkOFJqHPmsthRzXuF+J/XTXV43suYSFJNBZkOClOdIyKRFDAJCWajw3BRDKzKyJDLDHRJreyCcGdP3mRtE5qrlNzr0+r9YtZHCWowCEcgwtnUIcraEATCDzCM7zCm/VkvVjv1se0dMma9RzAH1ifP+g+mDo=</latexit><latexit sha1_base64="r8cLpKiZJtXS/wgD8iCVci7o1w=">ACBHicbVC7TsMwFL3hWcorwNjFokJiqhKEBANDJRbGguhDakrlOE5r1XEi20FUQYWfoWFAYRY+Qg2/gan7QAtR7J0fM69uvceP+FMacf5tpaWV1bX1ksb5c2t7Z1de2+/peJUEtokMY9lx8eKciZoUzPNaSeRFEc+p21/dFn47XsqFYvFrR4ntBfhgWAhI1gbqW9XvAjroR9mDznymEDTr5/d5HdB3646NWcCtEjcGanCDI2+/eUFMUkjKjThWKmu6yS6l2GpGeE0L3upogkmIzygXUMFjqjqZMjcnRklACFsTRPaDRf3dkOFJqHPmsthRzXuF+J/XTXV43suYSFJNBZkOClOdIyKRFDAJCWajw3BRDKzKyJDLDHRJreyCcGdP3mRtE5qrlNzr0+r9YtZHCWowCEcgwtnUIcraEATCDzCM7zCm/VkvVjv1se0dMma9RzAH1ifP+g+mDo=</latexit>

W(1) ∈ Rd1×d, b(1) ∈ Rd1

<latexit sha1_base64="Av+k0GeUMcXk+fjFEBVogb7nE=">ACPnicfVDPS8MwGE3nrzl/T16CQ5hgoxWBD14GHjxOMVug7WNE23sDQtSqM0r/Mi3+DN49ePCji1aPpNkE38UHg8d7yPc9P2FUKtN8MkoLi0vLK+XVytr6xuZWdXunLeNUYGLjmMWi6yNJGOXEVlQx0k0EQZHPSMcfXhR+54ISWN+o0YJcSPU5zSkGCkteVXbiZAa+GHWyW+zunWYQ4dyOBH97FqLgWdBR9GISBjkR/A7/+Tz71qzWyY8B5Yk1JDUzR8qPThDjNCJcYak7FlmotwMCUxI3nFSVJEB6iPulpypFex83G5+fwQCsBDGOhH1dwrP6cyFAk5SjydbLYU856hfiX10tVeOZmlCepIhxPgpTBlUMiy5hQAXBio0QVhQvSvEAyQVrxi7Bmj15nrSPG5bZsK5Oas3zaR1lsAf2QR1Y4BQ0wSVoARtgcA+ewSt4Mx6MF+Pd+JhES8Z0Zhf8gvH5BQCyrvw=</latexit><latexit sha1_base64="Av+k0GeUMcXk+fjFEBVogb7nE=">ACPnicfVDPS8MwGE3nrzl/T16CQ5hgoxWBD14GHjxOMVug7WNE23sDQtSqM0r/Mi3+DN49ePCji1aPpNkE38UHg8d7yPc9P2FUKtN8MkoLi0vLK+XVytr6xuZWdXunLeNUYGLjmMWi6yNJGOXEVlQx0k0EQZHPSMcfXhR+54ISWN+o0YJcSPU5zSkGCkteVXbiZAa+GHWyW+zunWYQ4dyOBH97FqLgWdBR9GISBjkR/A7/+Tz71qzWyY8B5Yk1JDUzR8qPThDjNCJcYak7FlmotwMCUxI3nFSVJEB6iPulpypFex83G5+fwQCsBDGOhH1dwrP6cyFAk5SjydbLYU856hfiX10tVeOZmlCepIhxPgpTBlUMiy5hQAXBio0QVhQvSvEAyQVrxi7Bmj15nrSPG5bZsK5Oas3zaR1lsAf2QR1Y4BQ0wSVoARtgcA+ewSt4Mx6MF+Pd+JhES8Z0Zhf8gvH5BQCyrvw=</latexit><latexit sha1_base64="Av+k0GeUMcXk+fjFEBVogb7nE=">ACPnicfVDPS8MwGE3nrzl/T16CQ5hgoxWBD14GHjxOMVug7WNE23sDQtSqM0r/Mi3+DN49ePCji1aPpNkE38UHg8d7yPc9P2FUKtN8MkoLi0vLK+XVytr6xuZWdXunLeNUYGLjmMWi6yNJGOXEVlQx0k0EQZHPSMcfXhR+54ISWN+o0YJcSPU5zSkGCkteVXbiZAa+GHWyW+zunWYQ4dyOBH97FqLgWdBR9GISBjkR/A7/+Tz71qzWyY8B5Yk1JDUzR8qPThDjNCJcYak7FlmotwMCUxI3nFSVJEB6iPulpypFex83G5+fwQCsBDGOhH1dwrP6cyFAk5SjydbLYU856hfiX10tVeOZmlCepIhxPgpTBlUMiy5hQAXBio0QVhQvSvEAyQVrxi7Bmj15nrSPG5bZsK5Oas3zaR1lsAf2QR1Y4BQ0wSVoARtgcA+ewSt4Mx6MF+Pd+JhES8Z0Zhf8gvH5BQCyrvw=</latexit><latexit sha1_base64="Av+k0GeUMcXk+fjFEBVogb7nE=">ACPnicfVDPS8MwGE3nrzl/T16CQ5hgoxWBD14GHjxOMVug7WNE23sDQtSqM0r/Mi3+DN49ePCji1aPpNkE38UHg8d7yPc9P2FUKtN8MkoLi0vLK+XVytr6xuZWdXunLeNUYGLjmMWi6yNJGOXEVlQx0k0EQZHPSMcfXhR+54ISWN+o0YJcSPU5zSkGCkteVXbiZAa+GHWyW+zunWYQ4dyOBH97FqLgWdBR9GISBjkR/A7/+Tz71qzWyY8B5Yk1JDUzR8qPThDjNCJcYak7FlmotwMCUxI3nFSVJEB6iPulpypFex83G5+fwQCsBDGOhH1dwrP6cyFAk5SjydbLYU856hfiX10tVeOZmlCepIhxPgpTBlUMiy5hQAXBio0QVhQvSvEAyQVrxi7Bmj15nrSPG5bZsK5Oas3zaR1lsAf2QR1Y4BQ0wSVoARtgcA+ewSt4Mx6MF+Pd+JhES8Z0Zhf8gvH5BQCyrvw=</latexit>
  • Input layer:
slide-20
SLIDE 20

Why non-linearities?

How if we remove activation function?

  • Neural networks can learn much more complex functions and

nonlinear decision boundaries

The capacity of the network increases with more hidden units and more hidden layers

slide-21
SLIDE 21

Activation functions

f(z) = 1 1 + e−z

<latexit sha1_base64="4snGlhjxkE7I7roWcflso17LAcU=">ACBHicbVDLSgMxFM3UV62vUZfdBItQEctEBN0IRTcuK9gHtGPJpJk2NJMZkozQDrNw46+4caGIWz/CnX9j2s5CWw9cOJxzL/fe40WcKe0431ZuaXldS2/XtjY3NresXf3GiqMJaF1EvJQtjysKGeC1jXTnLYiSXHgcdr0htcTv/lApWKhuNOjiLoB7gvmM4K1kbp20S+Pj+Al7PgSkwSlCYLHkN4nJ+M07dolp+JMARcJykgJZKh17a9OLyRxQIUmHCvVRk6k3QRLzQinaETKxphMsR92jZU4IAqN5k+kcJDo/SgH0pTQsOp+nsiwYFSo8AznQHWAzXvTcT/vHas/Qs3YSKNRVktsiPOdQhnCQCe0xSovnIEwkM7dCMsAmDm1yK5gQ0PzLi6RxWkFOBd2elapXWRx5UAQHoAwQOAdVcANqoA4IeATP4BW8WU/Wi/Vufcxac1Y2sw/+wPr8ASGfln4=</latexit><latexit sha1_base64="4snGlhjxkE7I7roWcflso17LAcU=">ACBHicbVDLSgMxFM3UV62vUZfdBItQEctEBN0IRTcuK9gHtGPJpJk2NJMZkozQDrNw46+4caGIWz/CnX9j2s5CWw9cOJxzL/fe40WcKe0431ZuaXldS2/XtjY3NresXf3GiqMJaF1EvJQtjysKGeC1jXTnLYiSXHgcdr0htcTv/lApWKhuNOjiLoB7gvmM4K1kbp20S+Pj+Al7PgSkwSlCYLHkN4nJ+M07dolp+JMARcJykgJZKh17a9OLyRxQIUmHCvVRk6k3QRLzQinaETKxphMsR92jZU4IAqN5k+kcJDo/SgH0pTQsOp+nsiwYFSo8AznQHWAzXvTcT/vHas/Qs3YSKNRVktsiPOdQhnCQCe0xSovnIEwkM7dCMsAmDm1yK5gQ0PzLi6RxWkFOBd2elapXWRx5UAQHoAwQOAdVcANqoA4IeATP4BW8WU/Wi/Vufcxac1Y2sw/+wPr8ASGfln4=</latexit><latexit sha1_base64="4snGlhjxkE7I7roWcflso17LAcU=">ACBHicbVDLSgMxFM3UV62vUZfdBItQEctEBN0IRTcuK9gHtGPJpJk2NJMZkozQDrNw46+4caGIWz/CnX9j2s5CWw9cOJxzL/fe40WcKe0431ZuaXldS2/XtjY3NresXf3GiqMJaF1EvJQtjysKGeC1jXTnLYiSXHgcdr0htcTv/lApWKhuNOjiLoB7gvmM4K1kbp20S+Pj+Al7PgSkwSlCYLHkN4nJ+M07dolp+JMARcJykgJZKh17a9OLyRxQIUmHCvVRk6k3QRLzQinaETKxphMsR92jZU4IAqN5k+kcJDo/SgH0pTQsOp+nsiwYFSo8AznQHWAzXvTcT/vHas/Qs3YSKNRVktsiPOdQhnCQCe0xSovnIEwkM7dCMsAmDm1yK5gQ0PzLi6RxWkFOBd2elapXWRx5UAQHoAwQOAdVcANqoA4IeATP4BW8WU/Wi/Vufcxac1Y2sw/+wPr8ASGfln4=</latexit><latexit sha1_base64="4snGlhjxkE7I7roWcflso17LAcU=">ACBHicbVDLSgMxFM3UV62vUZfdBItQEctEBN0IRTcuK9gHtGPJpJk2NJMZkozQDrNw46+4caGIWz/CnX9j2s5CWw9cOJxzL/fe40WcKe0431ZuaXldS2/XtjY3NresXf3GiqMJaF1EvJQtjysKGeC1jXTnLYiSXHgcdr0htcTv/lApWKhuNOjiLoB7gvmM4K1kbp20S+Pj+Al7PgSkwSlCYLHkN4nJ+M07dolp+JMARcJykgJZKh17a9OLyRxQIUmHCvVRk6k3QRLzQinaETKxphMsR92jZU4IAqN5k+kcJDo/SgH0pTQsOp+nsiwYFSo8AznQHWAzXvTcT/vHas/Qs3YSKNRVktsiPOdQhnCQCe0xSovnIEwkM7dCMsAmDm1yK5gQ0PzLi6RxWkFOBd2elapXWRx5UAQHoAwQOAdVcANqoA4IeATP4BW8WU/Wi/Vufcxac1Y2sw/+wPr8ASGfln4=</latexit>

sigmoid

f 0(z) = f(z) × (1 − f(z))

<latexit sha1_base64="3GcbAayqBhW+9Kwcpn7Vsrhoc=">ACB3icbVDLSsNAFJ3UV62vqEtBovYLiyJCLoRim5cVrAPaEOZTCft4OTBzI1Q3du/BU3LhRx6y+482+cpFlo64FhDufcy73uJHgCizr2ygsLC4trxRXS2vrG5tb5vZOS4WxpKxJQxHKjksUEzxgTeAgWCeSjPiuYG37ir12/dMKh4GtzCOmOTYcA9TgloqW/ue0eVhyq+wF769YD7TOGKjY8zodo3y1bNyoDniZ2TMsrR6JtfvUFIY58FQAVRqmtbETgJkcCpYJNSL1YsIvSODFlX04DoeU6S3THBh1oZYC+U+gWAM/V3R0J8pca+qyt9AiM16Xif143Bu/cSXgQxcACOh3kxQJDiNQ8IBLRkGMNSFUcr0rpiMiCQUdXUmHYM+ePE9aJzXbqtk3p+X6ZR5HEe2hA1RBNjpDdXSNGqiJKHpEz+gVvRlPxovxbnxMSwtG3rOL/sD4/AFGEZW7</latexit><latexit sha1_base64="3GcbAayqBhW+9Kwcpn7Vsrhoc=">ACB3icbVDLSsNAFJ3UV62vqEtBovYLiyJCLoRim5cVrAPaEOZTCft4OTBzI1Q3du/BU3LhRx6y+482+cpFlo64FhDufcy73uJHgCizr2ygsLC4trxRXS2vrG5tb5vZOS4WxpKxJQxHKjksUEzxgTeAgWCeSjPiuYG37ir12/dMKh4GtzCOmOTYcA9TgloqW/ue0eVhyq+wF769YD7TOGKjY8zodo3y1bNyoDniZ2TMsrR6JtfvUFIY58FQAVRqmtbETgJkcCpYJNSL1YsIvSODFlX04DoeU6S3THBh1oZYC+U+gWAM/V3R0J8pca+qyt9AiM16Xif143Bu/cSXgQxcACOh3kxQJDiNQ8IBLRkGMNSFUcr0rpiMiCQUdXUmHYM+ePE9aJzXbqtk3p+X6ZR5HEe2hA1RBNjpDdXSNGqiJKHpEz+gVvRlPxovxbnxMSwtG3rOL/sD4/AFGEZW7</latexit><latexit sha1_base64="3GcbAayqBhW+9Kwcpn7Vsrhoc=">ACB3icbVDLSsNAFJ3UV62vqEtBovYLiyJCLoRim5cVrAPaEOZTCft4OTBzI1Q3du/BU3LhRx6y+482+cpFlo64FhDufcy73uJHgCizr2ygsLC4trxRXS2vrG5tb5vZOS4WxpKxJQxHKjksUEzxgTeAgWCeSjPiuYG37ir12/dMKh4GtzCOmOTYcA9TgloqW/ue0eVhyq+wF769YD7TOGKjY8zodo3y1bNyoDniZ2TMsrR6JtfvUFIY58FQAVRqmtbETgJkcCpYJNSL1YsIvSODFlX04DoeU6S3THBh1oZYC+U+gWAM/V3R0J8pca+qyt9AiM16Xif143Bu/cSXgQxcACOh3kxQJDiNQ8IBLRkGMNSFUcr0rpiMiCQUdXUmHYM+ePE9aJzXbqtk3p+X6ZR5HEe2hA1RBNjpDdXSNGqiJKHpEz+gVvRlPxovxbnxMSwtG3rOL/sD4/AFGEZW7</latexit><latexit sha1_base64="3GcbAayqBhW+9Kwcpn7Vsrhoc=">ACB3icbVDLSsNAFJ3UV62vqEtBovYLiyJCLoRim5cVrAPaEOZTCft4OTBzI1Q3du/BU3LhRx6y+482+cpFlo64FhDufcy73uJHgCizr2ygsLC4trxRXS2vrG5tb5vZOS4WxpKxJQxHKjksUEzxgTeAgWCeSjPiuYG37ir12/dMKh4GtzCOmOTYcA9TgloqW/ue0eVhyq+wF769YD7TOGKjY8zodo3y1bNyoDniZ2TMsrR6JtfvUFIY58FQAVRqmtbETgJkcCpYJNSL1YsIvSODFlX04DoeU6S3THBh1oZYC+U+gWAM/V3R0J8pca+qyt9AiM16Xif143Bu/cSXgQxcACOh3kxQJDiNQ8IBLRkGMNSFUcr0rpiMiCQUdXUmHYM+ePE9aJzXbqtk3p+X6ZR5HEe2hA1RBNjpDdXSNGqiJKHpEz+gVvRlPxovxbnxMSwtG3rOL/sD4/AFGEZW7</latexit>

tanh

f 0(z) = 1 − f(z)2

<latexit sha1_base64="MqVGA4i/sJsTCsjlSr5qxWmCNA=">AB+3icbZDLSsNAFIZP6q3W6xLN4NFrAtLUgTdCEU3LivYC7SxTKaTduhkEmYmYg19FTcuFHri7jzbZy2Wj1h4GP/5zDOfP7MWdKO86XlVtaXldy68XNja3tnfs3WJTRYktEiHsm2jxXlTNCGZprTdiwpDn1OW/7oalpv3VOpWCRu9TimXogHgWMYG2snl0MjsqPx+gCuegEBQbvqj275FScmdBfcDMoQaZ6z/7s9iOShFRowrFSHdeJtZdiqRnhdFLoJorGmIzwgHYMChxS5aWz2yfo0Dh9FETSPKHRzP05keJQqXHom84Q6FarE3N/2qdRAfnXspEnGgqyHxRkHCkIzQNAvWZpETzsQFMJDO3IjLEhNt4iqYENzFL/+FZrXiOhX35rRUu8ziyM+HEAZXDiDGlxDHRpA4AGe4AVerYn1bL1Z7/PWnJXN7MEvWR/fnxSRkw=</latexit><latexit sha1_base64="MqVGA4i/sJsTCsjlSr5qxWmCNA=">AB+3icbZDLSsNAFIZP6q3W6xLN4NFrAtLUgTdCEU3LivYC7SxTKaTduhkEmYmYg19FTcuFHri7jzbZy2Wj1h4GP/5zDOfP7MWdKO86XlVtaXldy68XNja3tnfs3WJTRYktEiHsm2jxXlTNCGZprTdiwpDn1OW/7oalpv3VOpWCRu9TimXogHgWMYG2snl0MjsqPx+gCuegEBQbvqj275FScmdBfcDMoQaZ6z/7s9iOShFRowrFSHdeJtZdiqRnhdFLoJorGmIzwgHYMChxS5aWz2yfo0Dh9FETSPKHRzP05keJQqXHom84Q6FarE3N/2qdRAfnXspEnGgqyHxRkHCkIzQNAvWZpETzsQFMJDO3IjLEhNt4iqYENzFL/+FZrXiOhX35rRUu8ziyM+HEAZXDiDGlxDHRpA4AGe4AVerYn1bL1Z7/PWnJXN7MEvWR/fnxSRkw=</latexit><latexit sha1_base64="MqVGA4i/sJsTCsjlSr5qxWmCNA=">AB+3icbZDLSsNAFIZP6q3W6xLN4NFrAtLUgTdCEU3LivYC7SxTKaTduhkEmYmYg19FTcuFHri7jzbZy2Wj1h4GP/5zDOfP7MWdKO86XlVtaXldy68XNja3tnfs3WJTRYktEiHsm2jxXlTNCGZprTdiwpDn1OW/7oalpv3VOpWCRu9TimXogHgWMYG2snl0MjsqPx+gCuegEBQbvqj275FScmdBfcDMoQaZ6z/7s9iOShFRowrFSHdeJtZdiqRnhdFLoJorGmIzwgHYMChxS5aWz2yfo0Dh9FETSPKHRzP05keJQqXHom84Q6FarE3N/2qdRAfnXspEnGgqyHxRkHCkIzQNAvWZpETzsQFMJDO3IjLEhNt4iqYENzFL/+FZrXiOhX35rRUu8ziyM+HEAZXDiDGlxDHRpA4AGe4AVerYn1bL1Z7/PWnJXN7MEvWR/fnxSRkw=</latexit><latexit sha1_base64="MqVGA4i/sJsTCsjlSr5qxWmCNA=">AB+3icbZDLSsNAFIZP6q3W6xLN4NFrAtLUgTdCEU3LivYC7SxTKaTduhkEmYmYg19FTcuFHri7jzbZy2Wj1h4GP/5zDOfP7MWdKO86XlVtaXldy68XNja3tnfs3WJTRYktEiHsm2jxXlTNCGZprTdiwpDn1OW/7oalpv3VOpWCRu9TimXogHgWMYG2snl0MjsqPx+gCuegEBQbvqj275FScmdBfcDMoQaZ6z/7s9iOShFRowrFSHdeJtZdiqRnhdFLoJorGmIzwgHYMChxS5aWz2yfo0Dh9FETSPKHRzP05keJQqXHom84Q6FarE3N/2qdRAfnXspEnGgqyHxRkHCkIzQNAvWZpETzsQFMJDO3IjLEhNt4iqYENzFL/+FZrXiOhX35rRUu8ziyM+HEAZXDiDGlxDHRpA4AGe4AVerYn1bL1Z7/PWnJXN7MEvWR/fnxSRkw=</latexit>

f(z) = e2z − 1 e2z + 1

<latexit sha1_base64="G6vDyYAGjQKIG18jOKg0BSVdk0w=">ACDXicbZDLSgMxFIYz9VbrbdSlm2AVKmKZKYJuhKIblxXsBdpaMumZNjRzIckI7TAv4MZXceNCEbfu3fk2pu0I2vpD4OM/53ByfifkTCrL+jIyC4tLyvZ1dza+sbmlrm9U5NBJChUacAD0XCIBM58qCqmODRCAcRzONSdwdW4Xr8HIVng36phCG2P9HzmMkqUtjrmgVsYHeEL3HIFoTHcxaVRgk+wnfzwseaOmbeK1kR4HuwU8ihVpWN+troBjTzwFeVEyqZthaodE6EY5ZDkWpGEkNAB6UFTo08kO14ck2CD7XTxW4g9PMVnri/J2LiSTn0HN3pEdWXs7Wx+V+tGSn3vB0zP4wU+HS6yI04VgEeR4O7TABVfKiBUMH0XzHtE52L0gHmdAj27MnzUCsVbato35zmy5dpHFm0h/ZRAdnoDJXRNaqgKqLoAT2hF/RqPBrPxpvxPm3NGOnMLvoj4+MbIhKZsQ=</latexit><latexit sha1_base64="G6vDyYAGjQKIG18jOKg0BSVdk0w=">ACDXicbZDLSgMxFIYz9VbrbdSlm2AVKmKZKYJuhKIblxXsBdpaMumZNjRzIckI7TAv4MZXceNCEbfu3fk2pu0I2vpD4OM/53ByfifkTCrL+jIyC4tLyvZ1dza+sbmlrm9U5NBJChUacAD0XCIBM58qCqmODRCAcRzONSdwdW4Xr8HIVng36phCG2P9HzmMkqUtjrmgVsYHeEL3HIFoTHcxaVRgk+wnfzwseaOmbeK1kR4HuwU8ihVpWN+troBjTzwFeVEyqZthaodE6EY5ZDkWpGEkNAB6UFTo08kO14ck2CD7XTxW4g9PMVnri/J2LiSTn0HN3pEdWXs7Wx+V+tGSn3vB0zP4wU+HS6yI04VgEeR4O7TABVfKiBUMH0XzHtE52L0gHmdAj27MnzUCsVbato35zmy5dpHFm0h/ZRAdnoDJXRNaqgKqLoAT2hF/RqPBrPxpvxPm3NGOnMLvoj4+MbIhKZsQ=</latexit><latexit sha1_base64="G6vDyYAGjQKIG18jOKg0BSVdk0w=">ACDXicbZDLSgMxFIYz9VbrbdSlm2AVKmKZKYJuhKIblxXsBdpaMumZNjRzIckI7TAv4MZXceNCEbfu3fk2pu0I2vpD4OM/53ByfifkTCrL+jIyC4tLyvZ1dza+sbmlrm9U5NBJChUacAD0XCIBM58qCqmODRCAcRzONSdwdW4Xr8HIVng36phCG2P9HzmMkqUtjrmgVsYHeEL3HIFoTHcxaVRgk+wnfzwseaOmbeK1kR4HuwU8ihVpWN+troBjTzwFeVEyqZthaodE6EY5ZDkWpGEkNAB6UFTo08kO14ck2CD7XTxW4g9PMVnri/J2LiSTn0HN3pEdWXs7Wx+V+tGSn3vB0zP4wU+HS6yI04VgEeR4O7TABVfKiBUMH0XzHtE52L0gHmdAj27MnzUCsVbato35zmy5dpHFm0h/ZRAdnoDJXRNaqgKqLoAT2hF/RqPBrPxpvxPm3NGOnMLvoj4+MbIhKZsQ=</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQvEKMXyw=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odBu3wMYA6nMFXEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="ntnODkgzo3T4kCfyY+K/0V4NGOg=">ACAnicbZDLSgMxGIX/qbdaq45u3QSrUBHLTDe6EQ3LivYC7RjyaSZNjSTGZKM0A7zAm58FTcuFPEh3Pk2phdBWw8EPs5J+PMfP+ZMacf5snIrq2vrG/nNwlZxe2fX3is2VJRIQusk4pFs+VhRzgSta6Y5bcWS4tDntOkPryd584FKxSJxp0cx9ULcFyxgBGtjde2joDw+QZeoE0hMUnqfVscZOkNu9sOnhrt2yak4U6FlcOdQgrlqXfuz04tIElKhCcdKtV0n1l6KpWaE06zQSRSNMRniPm0bFDikykun2To2Dg9FETSHKHR1P39IsWhUqPQNzdDrAdqMZuY/2XtRAcXspEnGgqyGxQkHCkIzSpBvWYpETzkQFMJDN/RWSATS/aFgwJbiLKy9Do1pxnYp760AeDuAQyuDCOVzBDdSgDgQe4Rle4c16sl6s91ldOWve2z78kfXxDYg8mDk=</latexit><latexit sha1_base64="ntnODkgzo3T4kCfyY+K/0V4NGOg=">ACAnicbZDLSgMxGIX/qbdaq45u3QSrUBHLTDe6EQ3LivYC7RjyaSZNjSTGZKM0A7zAm58FTcuFPEh3Pk2phdBWw8EPs5J+PMfP+ZMacf5snIrq2vrG/nNwlZxe2fX3is2VJRIQusk4pFs+VhRzgSta6Y5bcWS4tDntOkPryd584FKxSJxp0cx9ULcFyxgBGtjde2joDw+QZeoE0hMUnqfVscZOkNu9sOnhrt2yak4U6FlcOdQgrlqXfuz04tIElKhCcdKtV0n1l6KpWaE06zQSRSNMRniPm0bFDikykun2To2Dg9FETSHKHR1P39IsWhUqPQNzdDrAdqMZuY/2XtRAcXspEnGgqyGxQkHCkIzSpBvWYpETzkQFMJDN/RWSATS/aFgwJbiLKy9Do1pxnYp760AeDuAQyuDCOVzBDdSgDgQe4Rle4c16sl6s91ldOWve2z78kfXxDYg8mDk=</latexit><latexit sha1_base64="fVWwQ1I2ofWM38dTM+TFfdN8og=">ACDXicbZC7SgNBFIbPxluMt1VLm8EoRMSwm0YbIWhjGcFcIFnD7GQ2GZy9MDMrJMu+gI2vYmOhiK29nW/jJFlBE38Y+PjPOZw5vxtxJpVlfRm5hcWl5ZX8amFtfWNzy9zeacgwFoTWSchD0XKxpJwFtK6Y4rQVCYp9l9Ome3c5rjfvqZAsDG7UMKOj/sB8xjBSltd8ArjY7QOep4ApOE3iaVUYpOkJ3+8LHmrlm0ytZEaB7sDIqQqdY1Pzu9kMQ+DRThWMq2bUXKSbBQjHCaFjqxpBEmd7hP2xoD7FPpJNrUnSonR7yQqFfoNDE/T2RYF/Koe/qTh+rgZytjc3/au1YeWdOwoIoVjQg0VezJEK0Tga1GOCEsWHGjARTP8VkQHWuSgdYEGHYM+ePA+NStm2yva1VaxeZHkYQ/2oQ2nEIVrqAGdSDwAE/wAq/Go/FsvBnv09ackc3swh8ZH98g0pmt</latexit><latexit sha1_base64="G6vDyYAGjQKIG18jOKg0BSVdk0w=">ACDXicbZDLSgMxFIYz9VbrbdSlm2AVKmKZKYJuhKIblxXsBdpaMumZNjRzIckI7TAv4MZXceNCEbfu3fk2pu0I2vpD4OM/53ByfifkTCrL+jIyC4tLyvZ1dza+sbmlrm9U5NBJChUacAD0XCIBM58qCqmODRCAcRzONSdwdW4Xr8HIVng36phCG2P9HzmMkqUtjrmgVsYHeEL3HIFoTHcxaVRgk+wnfzwseaOmbeK1kR4HuwU8ihVpWN+troBjTzwFeVEyqZthaodE6EY5ZDkWpGEkNAB6UFTo08kO14ck2CD7XTxW4g9PMVnri/J2LiSTn0HN3pEdWXs7Wx+V+tGSn3vB0zP4wU+HS6yI04VgEeR4O7TABVfKiBUMH0XzHtE52L0gHmdAj27MnzUCsVbato35zmy5dpHFm0h/ZRAdnoDJXRNaqgKqLoAT2hF/RqPBrPxpvxPm3NGOnMLvoj4+MbIhKZsQ=</latexit><latexit sha1_base64="G6vDyYAGjQKIG18jOKg0BSVdk0w=">ACDXicbZDLSgMxFIYz9VbrbdSlm2AVKmKZKYJuhKIblxXsBdpaMumZNjRzIckI7TAv4MZXceNCEbfu3fk2pu0I2vpD4OM/53ByfifkTCrL+jIyC4tLyvZ1dza+sbmlrm9U5NBJChUacAD0XCIBM58qCqmODRCAcRzONSdwdW4Xr8HIVng36phCG2P9HzmMkqUtjrmgVsYHeEL3HIFoTHcxaVRgk+wnfzwseaOmbeK1kR4HuwU8ihVpWN+troBjTzwFeVEyqZthaodE6EY5ZDkWpGEkNAB6UFTo08kO14ck2CD7XTxW4g9PMVnri/J2LiSTn0HN3pEdWXs7Wx+V+tGSn3vB0zP4wU+HS6yI04VgEeR4O7TABVfKiBUMH0XzHtE52L0gHmdAj27MnzUCsVbato35zmy5dpHFm0h/ZRAdnoDJXRNaqgKqLoAT2hF/RqPBrPxpvxPm3NGOnMLvoj4+MbIhKZsQ=</latexit><latexit sha1_base64="G6vDyYAGjQKIG18jOKg0BSVdk0w=">ACDXicbZDLSgMxFIYz9VbrbdSlm2AVKmKZKYJuhKIblxXsBdpaMumZNjRzIckI7TAv4MZXceNCEbfu3fk2pu0I2vpD4OM/53ByfifkTCrL+jIyC4tLyvZ1dza+sbmlrm9U5NBJChUacAD0XCIBM58qCqmODRCAcRzONSdwdW4Xr8HIVng36phCG2P9HzmMkqUtjrmgVsYHeEL3HIFoTHcxaVRgk+wnfzwseaOmbeK1kR4HuwU8ihVpWN+troBjTzwFeVEyqZthaodE6EY5ZDkWpGEkNAB6UFTo08kO14ck2CD7XTxW4g9PMVnri/J2LiSTn0HN3pEdWXs7Wx+V+tGSn3vB0zP4wU+HS6yI04VgEeR4O7TABVfKiBUMH0XzHtE52L0gHmdAj27MnzUCsVbato35zmy5dpHFm0h/ZRAdnoDJXRNaqgKqLoAT2hF/RqPBrPxpvxPm3NGOnMLvoj4+MbIhKZsQ=</latexit><latexit sha1_base64="G6vDyYAGjQKIG18jOKg0BSVdk0w=">ACDXicbZDLSgMxFIYz9VbrbdSlm2AVKmKZKYJuhKIblxXsBdpaMumZNjRzIckI7TAv4MZXceNCEbfu3fk2pu0I2vpD4OM/53ByfifkTCrL+jIyC4tLyvZ1dza+sbmlrm9U5NBJChUacAD0XCIBM58qCqmODRCAcRzONSdwdW4Xr8HIVng36phCG2P9HzmMkqUtjrmgVsYHeEL3HIFoTHcxaVRgk+wnfzwseaOmbeK1kR4HuwU8ihVpWN+troBjTzwFeVEyqZthaodE6EY5ZDkWpGEkNAB6UFTo08kO14ck2CD7XTxW4g9PMVnri/J2LiSTn0HN3pEdWXs7Wx+V+tGSn3vB0zP4wU+HS6yI04VgEeR4O7TABVfKiBUMH0XzHtE52L0gHmdAj27MnzUCsVbato35zmy5dpHFm0h/ZRAdnoDJXRNaqgKqLoAT2hF/RqPBrPxpvxPm3NGOnMLvoj4+MbIhKZsQ=</latexit><latexit sha1_base64="G6vDyYAGjQKIG18jOKg0BSVdk0w=">ACDXicbZDLSgMxFIYz9VbrbdSlm2AVKmKZKYJuhKIblxXsBdpaMumZNjRzIckI7TAv4MZXceNCEbfu3fk2pu0I2vpD4OM/53ByfifkTCrL+jIyC4tLyvZ1dza+sbmlrm9U5NBJChUacAD0XCIBM58qCqmODRCAcRzONSdwdW4Xr8HIVng36phCG2P9HzmMkqUtjrmgVsYHeEL3HIFoTHcxaVRgk+wnfzwseaOmbeK1kR4HuwU8ihVpWN+troBjTzwFeVEyqZthaodE6EY5ZDkWpGEkNAB6UFTo08kO14ck2CD7XTxW4g9PMVnri/J2LiSTn0HN3pEdWXs7Wx+V+tGSn3vB0zP4wU+HS6yI04VgEeR4O7TABVfKiBUMH0XzHtE52L0gHmdAj27MnzUCsVbato35zmy5dpHFm0h/ZRAdnoDJXRNaqgKqLoAT2hF/RqPBrPxpvxPm3NGOnMLvoj4+MbIhKZsQ=</latexit><latexit sha1_base64="G6vDyYAGjQKIG18jOKg0BSVdk0w=">ACDXicbZDLSgMxFIYz9VbrbdSlm2AVKmKZKYJuhKIblxXsBdpaMumZNjRzIckI7TAv4MZXceNCEbfu3fk2pu0I2vpD4OM/53ByfifkTCrL+jIyC4tLyvZ1dza+sbmlrm9U5NBJChUacAD0XCIBM58qCqmODRCAcRzONSdwdW4Xr8HIVng36phCG2P9HzmMkqUtjrmgVsYHeEL3HIFoTHcxaVRgk+wnfzwseaOmbeK1kR4HuwU8ihVpWN+troBjTzwFeVEyqZthaodE6EY5ZDkWpGEkNAB6UFTo08kO14ck2CD7XTxW4g9PMVnri/J2LiSTn0HN3pEdWXs7Wx+V+tGSn3vB0zP4wU+HS6yI04VgEeR4O7TABVfKiBUMH0XzHtE52L0gHmdAj27MnzUCsVbato35zmy5dpHFm0h/ZRAdnoDJXRNaqgKqLoAT2hF/RqPBrPxpvxPm3NGOnMLvoj4+MbIhKZsQ=</latexit>

f(z) = max(0, z)

<latexit sha1_base64="kxOkDi9OteK9stnv3AMxRIaEm4s=">AB+nicbVBNS8NAEJ3Ur1q/Uj16WSxC1ISEfQiFL14rGA/oA1ls920SzebsLtR29qf4sWDIl79Jd78N27bHLT1wcDjvRlm5vkxZ0o7zreVWVldW9/Ibua2tnd29+z8fl1FiS0RiIeyaPFeVM0JpmtNmLCkOfU4b/uB6jfuqVQsEnd6GFMvxD3BAkawNlLHzgfFUQldonaIH4vOCRqVOnbBKTszoGXipqQAKaod+6vdjUgSUqEJx0q1XCfW3hLzQink1w7UTGZIB7tGWowCFV3nh2+gQdG6WLgkiaEhrN1N8TYxwqNQx90xli3VeL3lT8z2slOrjwxkzEiaCzBcFCUc6QtMcUJdJSjQfGoKJZOZWRPpYqJNWjkTgrv48jKpn5Zdp+zenhUqV2kcWTiEIyiC+dQgRuoQg0IPMAzvMKb9WS9WO/Wx7w1Y6UzB/AH1ucPEqeR7A=</latexit><latexit sha1_base64="kxOkDi9OteK9stnv3AMxRIaEm4s=">AB+nicbVBNS8NAEJ3Ur1q/Uj16WSxC1ISEfQiFL14rGA/oA1ls920SzebsLtR29qf4sWDIl79Jd78N27bHLT1wcDjvRlm5vkxZ0o7zreVWVldW9/Ibua2tnd29+z8fl1FiS0RiIeyaPFeVM0JpmtNmLCkOfU4b/uB6jfuqVQsEnd6GFMvxD3BAkawNlLHzgfFUQldonaIH4vOCRqVOnbBKTszoGXipqQAKaod+6vdjUgSUqEJx0q1XCfW3hLzQink1w7UTGZIB7tGWowCFV3nh2+gQdG6WLgkiaEhrN1N8TYxwqNQx90xli3VeL3lT8z2slOrjwxkzEiaCzBcFCUc6QtMcUJdJSjQfGoKJZOZWRPpYqJNWjkTgrv48jKpn5Zdp+zenhUqV2kcWTiEIyiC+dQgRuoQg0IPMAzvMKb9WS9WO/Wx7w1Y6UzB/AH1ucPEqeR7A=</latexit><latexit sha1_base64="kxOkDi9OteK9stnv3AMxRIaEm4s=">AB+nicbVBNS8NAEJ3Ur1q/Uj16WSxC1ISEfQiFL14rGA/oA1ls920SzebsLtR29qf4sWDIl79Jd78N27bHLT1wcDjvRlm5vkxZ0o7zreVWVldW9/Ibua2tnd29+z8fl1FiS0RiIeyaPFeVM0JpmtNmLCkOfU4b/uB6jfuqVQsEnd6GFMvxD3BAkawNlLHzgfFUQldonaIH4vOCRqVOnbBKTszoGXipqQAKaod+6vdjUgSUqEJx0q1XCfW3hLzQink1w7UTGZIB7tGWowCFV3nh2+gQdG6WLgkiaEhrN1N8TYxwqNQx90xli3VeL3lT8z2slOrjwxkzEiaCzBcFCUc6QtMcUJdJSjQfGoKJZOZWRPpYqJNWjkTgrv48jKpn5Zdp+zenhUqV2kcWTiEIyiC+dQgRuoQg0IPMAzvMKb9WS9WO/Wx7w1Y6UzB/AH1ucPEqeR7A=</latexit><latexit sha1_base64="kxOkDi9OteK9stnv3AMxRIaEm4s=">AB+nicbVBNS8NAEJ3Ur1q/Uj16WSxC1ISEfQiFL14rGA/oA1ls920SzebsLtR29qf4sWDIl79Jd78N27bHLT1wcDjvRlm5vkxZ0o7zreVWVldW9/Ibua2tnd29+z8fl1FiS0RiIeyaPFeVM0JpmtNmLCkOfU4b/uB6jfuqVQsEnd6GFMvxD3BAkawNlLHzgfFUQldonaIH4vOCRqVOnbBKTszoGXipqQAKaod+6vdjUgSUqEJx0q1XCfW3hLzQink1w7UTGZIB7tGWowCFV3nh2+gQdG6WLgkiaEhrN1N8TYxwqNQx90xli3VeL3lT8z2slOrjwxkzEiaCzBcFCUc6QtMcUJdJSjQfGoKJZOZWRPpYqJNWjkTgrv48jKpn5Zdp+zenhUqV2kcWTiEIyiC+dQgRuoQg0IPMAzvMKb9WS9WO/Wx7w1Y6UzB/AH1ucPEqeR7A=</latexit>

ReLU 
 (rectified linear unit)

f 0(z) = ( 1 z > 0 z < 0

<latexit sha1_base64="9/X76zUp6jklbnfvYpAmK6D/r4Q=">ACH3icbZDLSgMxFIYzXmu9V26CRa1bsqMiLpQKbpxWcFeoFNKJj3ThmYyQ5IR2qFv4sZXceNCEXHXtzFtZ6GtBwIf/39OkvN7EWdK2/bIWlhcWl5Zzaxl1zc2t7ZzO7tVFcaSQoWGPJR1jyjgTEBFM82hHkgceh5vXuxn7tCaRioXjU/QiaAekI5jNKtJFauXP/uDA4wdfY9aDRELNXWqIHXyEB/gG26L7QlfGQbRThtaubxdtCeF58FJIY/SKrdy3247pHEAQlNOlGo4dqSbCZGaUQ7DrBsriAjtkQ40DAoSgGomk/2G+NAobeyH0hyh8UT9PZGQKl+4JnOgOiumvXG4n9eI9b+ZTNhIo1CDp9yI851iEeh4XbTALVvG+AUMnMXzHtEkmoNpFmTQjO7MrzUD0tOnbReTjLl27TODJoHx2gAnLQBSqhe1RGFUTRM3pF7+jDerHerE/ra9q6YKUze+hPWaMfrzyfEQ=</latexit><latexit sha1_base64="9/X76zUp6jklbnfvYpAmK6D/r4Q=">ACH3icbZDLSgMxFIYzXmu9V26CRa1bsqMiLpQKbpxWcFeoFNKJj3ThmYyQ5IR2qFv4sZXceNCEXHXtzFtZ6GtBwIf/39OkvN7EWdK2/bIWlhcWl5Zzaxl1zc2t7ZzO7tVFcaSQoWGPJR1jyjgTEBFM82hHkgceh5vXuxn7tCaRioXjU/QiaAekI5jNKtJFauXP/uDA4wdfY9aDRELNXWqIHXyEB/gG26L7QlfGQbRThtaubxdtCeF58FJIY/SKrdy3247pHEAQlNOlGo4dqSbCZGaUQ7DrBsriAjtkQ40DAoSgGomk/2G+NAobeyH0hyh8UT9PZGQKl+4JnOgOiumvXG4n9eI9b+ZTNhIo1CDp9yI851iEeh4XbTALVvG+AUMnMXzHtEkmoNpFmTQjO7MrzUD0tOnbReTjLl27TODJoHx2gAnLQBSqhe1RGFUTRM3pF7+jDerHerE/ra9q6YKUze+hPWaMfrzyfEQ=</latexit><latexit sha1_base64="9/X76zUp6jklbnfvYpAmK6D/r4Q=">ACH3icbZDLSgMxFIYzXmu9V26CRa1bsqMiLpQKbpxWcFeoFNKJj3ThmYyQ5IR2qFv4sZXceNCEXHXtzFtZ6GtBwIf/39OkvN7EWdK2/bIWlhcWl5Zzaxl1zc2t7ZzO7tVFcaSQoWGPJR1jyjgTEBFM82hHkgceh5vXuxn7tCaRioXjU/QiaAekI5jNKtJFauXP/uDA4wdfY9aDRELNXWqIHXyEB/gG26L7QlfGQbRThtaubxdtCeF58FJIY/SKrdy3247pHEAQlNOlGo4dqSbCZGaUQ7DrBsriAjtkQ40DAoSgGomk/2G+NAobeyH0hyh8UT9PZGQKl+4JnOgOiumvXG4n9eI9b+ZTNhIo1CDp9yI851iEeh4XbTALVvG+AUMnMXzHtEkmoNpFmTQjO7MrzUD0tOnbReTjLl27TODJoHx2gAnLQBSqhe1RGFUTRM3pF7+jDerHerE/ra9q6YKUze+hPWaMfrzyfEQ=</latexit><latexit sha1_base64="9/X76zUp6jklbnfvYpAmK6D/r4Q=">ACH3icbZDLSgMxFIYzXmu9V26CRa1bsqMiLpQKbpxWcFeoFNKJj3ThmYyQ5IR2qFv4sZXceNCEXHXtzFtZ6GtBwIf/39OkvN7EWdK2/bIWlhcWl5Zzaxl1zc2t7ZzO7tVFcaSQoWGPJR1jyjgTEBFM82hHkgceh5vXuxn7tCaRioXjU/QiaAekI5jNKtJFauXP/uDA4wdfY9aDRELNXWqIHXyEB/gG26L7QlfGQbRThtaubxdtCeF58FJIY/SKrdy3247pHEAQlNOlGo4dqSbCZGaUQ7DrBsriAjtkQ40DAoSgGomk/2G+NAobeyH0hyh8UT9PZGQKl+4JnOgOiumvXG4n9eI9b+ZTNhIo1CDp9yI851iEeh4XbTALVvG+AUMnMXzHtEkmoNpFmTQjO7MrzUD0tOnbReTjLl27TODJoHx2gAnLQBSqhe1RGFUTRM3pF7+jDerHerE/ra9q6YKUze+hPWaMfrzyfEQ=</latexit>

Advantages of ReLU?

slide-22
SLIDE 22

Activation functions

Problems of ReLU? “dead neurons” Leaky ReLU

f(z) = ( z z ≥ 0 0.01z z < 0

<latexit sha1_base64="hBHq/48/smLgMJ23MnABJb6l6s=">ACJnicbVDLSgMxFM3UV62vqks3waLUTZkRQRcWim5cVrAP6JSe+0oZnMmGSEdujXuPFX3LioiLjzU0zbWjrgQuHc+69yT1exJnStv1lZVZW19Y3spu5re2d3b38/kFdhbGkUKMhD2XTIwo4E1DTHNoRhJI4HFoeIPbqd94AqlYKB70MIJ2QHqC+YwSbaROvuwXR2e4jF0Pekwk1KxSYzCp6bcHjxiG7sutku2M9euse2C6KaNnXzBWDPgZeKkpIBSVDv5idsNaRyA0JQTpVqOHel2QqRmlM458YKIkIHpActQwUJQLWT2ZljfGKULvZDaUpoPFN/TyQkUGoYeKYzILqvFr2p+J/XirV/1U6YiGINgs4f8mOdYinmeEuk0A1HxpCqGTmr5j2iSRUm2RzJgRn8eRlUj8vOXbJub8oVG7SOLoCB2jInLQJaqgO1RFNUTRM3pFE/RuvVhv1of1OW/NWOnMIfoD6/sHSVWh/Q=</latexit><latexit sha1_base64="hBHq/48/smLgMJ23MnABJb6l6s=">ACJnicbVDLSgMxFM3UV62vqks3waLUTZkRQRcWim5cVrAP6JSe+0oZnMmGSEdujXuPFX3LioiLjzU0zbWjrgQuHc+69yT1exJnStv1lZVZW19Y3spu5re2d3b38/kFdhbGkUKMhD2XTIwo4E1DTHNoRhJI4HFoeIPbqd94AqlYKB70MIJ2QHqC+YwSbaROvuwXR2e4jF0Pekwk1KxSYzCp6bcHjxiG7sutku2M9euse2C6KaNnXzBWDPgZeKkpIBSVDv5idsNaRyA0JQTpVqOHel2QqRmlM458YKIkIHpActQwUJQLWT2ZljfGKULvZDaUpoPFN/TyQkUGoYeKYzILqvFr2p+J/XirV/1U6YiGINgs4f8mOdYinmeEuk0A1HxpCqGTmr5j2iSRUm2RzJgRn8eRlUj8vOXbJub8oVG7SOLoCB2jInLQJaqgO1RFNUTRM3pFE/RuvVhv1of1OW/NWOnMIfoD6/sHSVWh/Q=</latexit><latexit sha1_base64="hBHq/48/smLgMJ23MnABJb6l6s=">ACJnicbVDLSgMxFM3UV62vqks3waLUTZkRQRcWim5cVrAP6JSe+0oZnMmGSEdujXuPFX3LioiLjzU0zbWjrgQuHc+69yT1exJnStv1lZVZW19Y3spu5re2d3b38/kFdhbGkUKMhD2XTIwo4E1DTHNoRhJI4HFoeIPbqd94AqlYKB70MIJ2QHqC+YwSbaROvuwXR2e4jF0Pekwk1KxSYzCp6bcHjxiG7sutku2M9euse2C6KaNnXzBWDPgZeKkpIBSVDv5idsNaRyA0JQTpVqOHel2QqRmlM458YKIkIHpActQwUJQLWT2ZljfGKULvZDaUpoPFN/TyQkUGoYeKYzILqvFr2p+J/XirV/1U6YiGINgs4f8mOdYinmeEuk0A1HxpCqGTmr5j2iSRUm2RzJgRn8eRlUj8vOXbJub8oVG7SOLoCB2jInLQJaqgO1RFNUTRM3pFE/RuvVhv1of1OW/NWOnMIfoD6/sHSVWh/Q=</latexit><latexit sha1_base64="hBHq/48/smLgMJ23MnABJb6l6s=">ACJnicbVDLSgMxFM3UV62vqks3waLUTZkRQRcWim5cVrAP6JSe+0oZnMmGSEdujXuPFX3LioiLjzU0zbWjrgQuHc+69yT1exJnStv1lZVZW19Y3spu5re2d3b38/kFdhbGkUKMhD2XTIwo4E1DTHNoRhJI4HFoeIPbqd94AqlYKB70MIJ2QHqC+YwSbaROvuwXR2e4jF0Pekwk1KxSYzCp6bcHjxiG7sutku2M9euse2C6KaNnXzBWDPgZeKkpIBSVDv5idsNaRyA0JQTpVqOHel2QqRmlM458YKIkIHpActQwUJQLWT2ZljfGKULvZDaUpoPFN/TyQkUGoYeKYzILqvFr2p+J/XirV/1U6YiGINgs4f8mOdYinmeEuk0A1HxpCqGTmr5j2iSRUm2RzJgRn8eRlUj8vOXbJub8oVG7SOLoCB2jInLQJaqgO1RFNUTRM3pFE/RuvVhv1of1OW/NWOnMIfoD6/sHSVWh/Q=</latexit>
slide-23
SLIDE 23

Loss functions

  • Binary classification

y = σ(w(o) · h2 + b(o))

<latexit sha1_base64="xwf5NTK8Os9frFEQg9QWL1GKg9Q=">ACInicbVDLSsNAFJ34rPUVdelmsAgtQkmKoC6EohuXFewDmlgm0k7dJIJMxOlhHyLG3/FjQtFXQl+jJM2grYeuHA4517uvceLGJXKsj6NhcWl5ZXVwlpxfWNza9vc2W1JHgtMmpgzLjoekoTRkDQVYx0IkFQ4DHS9kaXmd+I0JSHt6ocUTcA1C6lOMlJZ65tkYnkNH0kGAyk6A1NDzk/v0NinzSgod3OcK/sjDtFeDR9CbmpWeWbKq1gRwntg5KYEcjZ757vQ5jgMSKsyQlF3bipSbIKEoZiQtOrEkEcIjNCBdTUMUEOkmkxdTeKiVPvS50BUqOF/TyQokHIceLozO1fOepn4n9eNlX/qJjSMYkVCPF3kxwqDrO8YJ8KghUba4KwoPpWiIdIKx0qkUdgj378jxp1aq2VbWvj0v1izyOAtgHB6AMbHAC6uAKNEATYPAnsALeDUejWfjzfiYti4Y+cwe+APj6xs+HKLU</latexit><latexit sha1_base64="xwf5NTK8Os9frFEQg9QWL1GKg9Q=">ACInicbVDLSsNAFJ34rPUVdelmsAgtQkmKoC6EohuXFewDmlgm0k7dJIJMxOlhHyLG3/FjQtFXQl+jJM2grYeuHA4517uvceLGJXKsj6NhcWl5ZXVwlpxfWNza9vc2W1JHgtMmpgzLjoekoTRkDQVYx0IkFQ4DHS9kaXmd+I0JSHt6ocUTcA1C6lOMlJZ65tkYnkNH0kGAyk6A1NDzk/v0NinzSgod3OcK/sjDtFeDR9CbmpWeWbKq1gRwntg5KYEcjZ757vQ5jgMSKsyQlF3bipSbIKEoZiQtOrEkEcIjNCBdTUMUEOkmkxdTeKiVPvS50BUqOF/TyQokHIceLozO1fOepn4n9eNlX/qJjSMYkVCPF3kxwqDrO8YJ8KghUba4KwoPpWiIdIKx0qkUdgj378jxp1aq2VbWvj0v1izyOAtgHB6AMbHAC6uAKNEATYPAnsALeDUejWfjzfiYti4Y+cwe+APj6xs+HKLU</latexit><latexit sha1_base64="xwf5NTK8Os9frFEQg9QWL1GKg9Q=">ACInicbVDLSsNAFJ34rPUVdelmsAgtQkmKoC6EohuXFewDmlgm0k7dJIJMxOlhHyLG3/FjQtFXQl+jJM2grYeuHA4517uvceLGJXKsj6NhcWl5ZXVwlpxfWNza9vc2W1JHgtMmpgzLjoekoTRkDQVYx0IkFQ4DHS9kaXmd+I0JSHt6ocUTcA1C6lOMlJZ65tkYnkNH0kGAyk6A1NDzk/v0NinzSgod3OcK/sjDtFeDR9CbmpWeWbKq1gRwntg5KYEcjZ757vQ5jgMSKsyQlF3bipSbIKEoZiQtOrEkEcIjNCBdTUMUEOkmkxdTeKiVPvS50BUqOF/TyQokHIceLozO1fOepn4n9eNlX/qJjSMYkVCPF3kxwqDrO8YJ8KghUba4KwoPpWiIdIKx0qkUdgj378jxp1aq2VbWvj0v1izyOAtgHB6AMbHAC6uAKNEATYPAnsALeDUejWfjzfiYti4Y+cwe+APj6xs+HKLU</latexit><latexit sha1_base64="xwf5NTK8Os9frFEQg9QWL1GKg9Q=">ACInicbVDLSsNAFJ34rPUVdelmsAgtQkmKoC6EohuXFewDmlgm0k7dJIJMxOlhHyLG3/FjQtFXQl+jJM2grYeuHA4517uvceLGJXKsj6NhcWl5ZXVwlpxfWNza9vc2W1JHgtMmpgzLjoekoTRkDQVYx0IkFQ4DHS9kaXmd+I0JSHt6ocUTcA1C6lOMlJZ65tkYnkNH0kGAyk6A1NDzk/v0NinzSgod3OcK/sjDtFeDR9CbmpWeWbKq1gRwntg5KYEcjZ757vQ5jgMSKsyQlF3bipSbIKEoZiQtOrEkEcIjNCBdTUMUEOkmkxdTeKiVPvS50BUqOF/TyQokHIceLozO1fOepn4n9eNlX/qJjSMYkVCPF3kxwqDrO8YJ8KghUba4KwoPpWiIdIKx0qkUdgj378jxp1aq2VbWvj0v1izyOAtgHB6AMbHAC6uAKNEATYPAnsALeDUejWfjzfiYti4Y+cwe+APj6xs+HKLU</latexit>

L(y, y∗) = −y∗ log y − (1 − y∗) log (1 − y)

<latexit sha1_base64="rRgl1nuidQVks0Gj7Z1U34kvg=">ACI3icbVDLSgMxFM3UV62vUZdugkVoxZYZERBKLpx4aKCfUBnLJk0bUMzD5KMAzL278FTculOLGhf9iZjoLbT2Q5OSce0nucQJGhTSML62wtLyulZcL21sbm3v6Lt7beGHJMW9pnPuw4ShFGPtCSVjHQDTpDrMNJxJjep3kiXFDfe5BRQGwXjTw6pBhJfX1S8tFcowRi+SnQCo8fjKryCNXVazB/FUQJrsGKqLXUyaXarJn29bNSNDHCRmDkpgxzNvj61Bj4OXeJzJAQPdMIpB0jLilmJClZoSABwhM0Ij1FPeQSYcfZjAk8UsoADn2ulidhpv7uiJErROQ6qjKdSMx7qfif1wvl8MKOqReEknh49tAwZFD6MA0MDignWLJIEYQ5VX+FeIw4wlLFWlIhmPMjL5L2ad06ub9WblxncdRBAfgEFSACc5BA9yCJmgBDJ7BK3gH9qL9qZNtc9ZaUHLe/bBH2jfP8ZjoJU=</latexit><latexit sha1_base64="rRgl1nuidQVks0Gj7Z1U34kvg=">ACI3icbVDLSgMxFM3UV62vUZdugkVoxZYZERBKLpx4aKCfUBnLJk0bUMzD5KMAzL278FTculOLGhf9iZjoLbT2Q5OSce0nucQJGhTSML62wtLyulZcL21sbm3v6Lt7beGHJMW9pnPuw4ShFGPtCSVjHQDTpDrMNJxJjep3kiXFDfe5BRQGwXjTw6pBhJfX1S8tFcowRi+SnQCo8fjKryCNXVazB/FUQJrsGKqLXUyaXarJn29bNSNDHCRmDkpgxzNvj61Bj4OXeJzJAQPdMIpB0jLilmJClZoSABwhM0Ij1FPeQSYcfZjAk8UsoADn2ulidhpv7uiJErROQ6qjKdSMx7qfif1wvl8MKOqReEknh49tAwZFD6MA0MDignWLJIEYQ5VX+FeIw4wlLFWlIhmPMjL5L2ad06ub9WblxncdRBAfgEFSACc5BA9yCJmgBDJ7BK3gH9qL9qZNtc9ZaUHLe/bBH2jfP8ZjoJU=</latexit><latexit sha1_base64="rRgl1nuidQVks0Gj7Z1U34kvg=">ACI3icbVDLSgMxFM3UV62vUZdugkVoxZYZERBKLpx4aKCfUBnLJk0bUMzD5KMAzL278FTculOLGhf9iZjoLbT2Q5OSce0nucQJGhTSML62wtLyulZcL21sbm3v6Lt7beGHJMW9pnPuw4ShFGPtCSVjHQDTpDrMNJxJjep3kiXFDfe5BRQGwXjTw6pBhJfX1S8tFcowRi+SnQCo8fjKryCNXVazB/FUQJrsGKqLXUyaXarJn29bNSNDHCRmDkpgxzNvj61Bj4OXeJzJAQPdMIpB0jLilmJClZoSABwhM0Ij1FPeQSYcfZjAk8UsoADn2ulidhpv7uiJErROQ6qjKdSMx7qfif1wvl8MKOqReEknh49tAwZFD6MA0MDignWLJIEYQ5VX+FeIw4wlLFWlIhmPMjL5L2ad06ub9WblxncdRBAfgEFSACc5BA9yCJmgBDJ7BK3gH9qL9qZNtc9ZaUHLe/bBH2jfP8ZjoJU=</latexit><latexit sha1_base64="rRgl1nuidQVks0Gj7Z1U34kvg=">ACI3icbVDLSgMxFM3UV62vUZdugkVoxZYZERBKLpx4aKCfUBnLJk0bUMzD5KMAzL278FTculOLGhf9iZjoLbT2Q5OSce0nucQJGhTSML62wtLyulZcL21sbm3v6Lt7beGHJMW9pnPuw4ShFGPtCSVjHQDTpDrMNJxJjep3kiXFDfe5BRQGwXjTw6pBhJfX1S8tFcowRi+SnQCo8fjKryCNXVazB/FUQJrsGKqLXUyaXarJn29bNSNDHCRmDkpgxzNvj61Bj4OXeJzJAQPdMIpB0jLilmJClZoSABwhM0Ij1FPeQSYcfZjAk8UsoADn2ulidhpv7uiJErROQ6qjKdSMx7qfif1wvl8MKOqReEknh49tAwZFD6MA0MDignWLJIEYQ5VX+FeIw4wlLFWlIhmPMjL5L2ad06ub9WblxncdRBAfgEFSACc5BA9yCJmgBDJ7BK3gH9qL9qZNtc9ZaUHLe/bBH2jfP8ZjoJU=</latexit>
  • Regression

y = w(o) · h2 + b(o)

<latexit sha1_base64="wUt6eQRjOkpLTQVKSqhRvpvYR4=">ACGnicbVDLSsNAFJ34rPUVdelmsAgVoSRF0I1QdOygn1AG8NkOmHTjJhZqKEkO9w46+4caGIO3Hj3zhpg2jrgQuHc+7l3nu8iFGpLOvLWFhcWl5ZLa2V1zc2t7bNnd25LHApIU546LrIUkYDUlLUcVINxIEBR4jHW98mfudOyIk5eGNSiLiBGgYUp9ipLTkmnYCz2E/QGrk+el9dptW+VEG+3jA1Y8ytw6PIbe1HTNilWzJoDzxC5IBRouZHf8BxHJBQYak7NlWpJwUCUxI1m5H0sSITxGQ9LTNEQBkU46eS2Dh1oZQJ8LXaGCE/X3RIoCKZPA0535tXLWy8X/vF6s/DMnpWEUKxLi6SI/ZlBxmOcEB1QrFiCcKC6lshHiGBsNJplnUI9uzL86Rdr9lWzb4+qTQuijhKYB8cgCqwSlogCvQBC2AwQN4Ai/g1Xg0no0343aumAUM3vgD4zPb1NOn8Y=</latexit><latexit sha1_base64="wUt6eQRjOkpLTQVKSqhRvpvYR4=">ACGnicbVDLSsNAFJ34rPUVdelmsAgVoSRF0I1QdOygn1AG8NkOmHTjJhZqKEkO9w46+4caGIO3Hj3zhpg2jrgQuHc+7l3nu8iFGpLOvLWFhcWl5ZLa2V1zc2t7bNnd25LHApIU546LrIUkYDUlLUcVINxIEBR4jHW98mfudOyIk5eGNSiLiBGgYUp9ipLTkmnYCz2E/QGrk+el9dptW+VEG+3jA1Y8ytw6PIbe1HTNilWzJoDzxC5IBRouZHf8BxHJBQYak7NlWpJwUCUxI1m5H0sSITxGQ9LTNEQBkU46eS2Dh1oZQJ8LXaGCE/X3RIoCKZPA0535tXLWy8X/vF6s/DMnpWEUKxLi6SI/ZlBxmOcEB1QrFiCcKC6lshHiGBsNJplnUI9uzL86Rdr9lWzb4+qTQuijhKYB8cgCqwSlogCvQBC2AwQN4Ai/g1Xg0no0343aumAUM3vgD4zPb1NOn8Y=</latexit><latexit sha1_base64="wUt6eQRjOkpLTQVKSqhRvpvYR4=">ACGnicbVDLSsNAFJ34rPUVdelmsAgVoSRF0I1QdOygn1AG8NkOmHTjJhZqKEkO9w46+4caGIO3Hj3zhpg2jrgQuHc+7l3nu8iFGpLOvLWFhcWl5ZLa2V1zc2t7bNnd25LHApIU546LrIUkYDUlLUcVINxIEBR4jHW98mfudOyIk5eGNSiLiBGgYUp9ipLTkmnYCz2E/QGrk+el9dptW+VEG+3jA1Y8ytw6PIbe1HTNilWzJoDzxC5IBRouZHf8BxHJBQYak7NlWpJwUCUxI1m5H0sSITxGQ9LTNEQBkU46eS2Dh1oZQJ8LXaGCE/X3RIoCKZPA0535tXLWy8X/vF6s/DMnpWEUKxLi6SI/ZlBxmOcEB1QrFiCcKC6lshHiGBsNJplnUI9uzL86Rdr9lWzb4+qTQuijhKYB8cgCqwSlogCvQBC2AwQN4Ai/g1Xg0no0343aumAUM3vgD4zPb1NOn8Y=</latexit><latexit sha1_base64="wUt6eQRjOkpLTQVKSqhRvpvYR4=">ACGnicbVDLSsNAFJ34rPUVdelmsAgVoSRF0I1QdOygn1AG8NkOmHTjJhZqKEkO9w46+4caGIO3Hj3zhpg2jrgQuHc+7l3nu8iFGpLOvLWFhcWl5ZLa2V1zc2t7bNnd25LHApIU546LrIUkYDUlLUcVINxIEBR4jHW98mfudOyIk5eGNSiLiBGgYUp9ipLTkmnYCz2E/QGrk+el9dptW+VEG+3jA1Y8ytw6PIbe1HTNilWzJoDzxC5IBRouZHf8BxHJBQYak7NlWpJwUCUxI1m5H0sSITxGQ9LTNEQBkU46eS2Dh1oZQJ8LXaGCE/X3RIoCKZPA0535tXLWy8X/vF6s/DMnpWEUKxLi6SI/ZlBxmOcEB1QrFiCcKC6lshHiGBsNJplnUI9uzL86Rdr9lWzb4+qTQuijhKYB8cgCqwSlogCvQBC2AwQN4Ai/g1Xg0no0343aumAUM3vgD4zPb1NOn8Y=</latexit>

LMSE(y, y∗) = (y − y∗)2

<latexit sha1_base64="qXd8yxiBWzOEL57x+yWmNbP8hNE=">ACF3icbVDJSgNBFOxjXGLevTSGIQoGmaCoBchKIHhYhmgWz0dHqSJj0L3W/EYZi/8OKvePGgiFe9+Td2loMmFjQUVe/xusoOBFdgmt/GzOzc/MJiaim9vLK6tp7Z2KwoP5SUlakvfFmziWKCe6wMHASrBZIR1xasavfPB371nknFfe8OoA1XdL1uMpAS21M/mGS6BHiYivknbcAPYA8fXtRZLkogMctfb38CnORfhwyFuFdiZr5s0h8DSxiSLxi1M1+Njk9Dl3lABVGqbpkBNGMigVPBknQjVCwgtE+6rK6pR1ymvEwV4J3tdLBji/18wAP1d8bMXGVilxbTw5SqElvIP7n1UNwTpox94IQmEdHh5xQYPDxoCTc4ZJREJEmhEqu/4pj0hCQVeZ1iVYk5GnSaWQt8y8dXOULZ6N60ihbSDcshCx6iILlEJlRFj+gZvaI348l4Md6Nj9HojDHe2UJ/YHz+AK9WnaI=</latexit><latexit sha1_base64="qXd8yxiBWzOEL57x+yWmNbP8hNE=">ACF3icbVDJSgNBFOxjXGLevTSGIQoGmaCoBchKIHhYhmgWz0dHqSJj0L3W/EYZi/8OKvePGgiFe9+Td2loMmFjQUVe/xusoOBFdgmt/GzOzc/MJiaim9vLK6tp7Z2KwoP5SUlakvfFmziWKCe6wMHASrBZIR1xasavfPB371nknFfe8OoA1XdL1uMpAS21M/mGS6BHiYivknbcAPYA8fXtRZLkogMctfb38CnORfhwyFuFdiZr5s0h8DSxiSLxi1M1+Njk9Dl3lABVGqbpkBNGMigVPBknQjVCwgtE+6rK6pR1ymvEwV4J3tdLBji/18wAP1d8bMXGVilxbTw5SqElvIP7n1UNwTpox94IQmEdHh5xQYPDxoCTc4ZJREJEmhEqu/4pj0hCQVeZ1iVYk5GnSaWQt8y8dXOULZ6N60ihbSDcshCx6iILlEJlRFj+gZvaI348l4Md6Nj9HojDHe2UJ/YHz+AK9WnaI=</latexit><latexit sha1_base64="qXd8yxiBWzOEL57x+yWmNbP8hNE=">ACF3icbVDJSgNBFOxjXGLevTSGIQoGmaCoBchKIHhYhmgWz0dHqSJj0L3W/EYZi/8OKvePGgiFe9+Td2loMmFjQUVe/xusoOBFdgmt/GzOzc/MJiaim9vLK6tp7Z2KwoP5SUlakvfFmziWKCe6wMHASrBZIR1xasavfPB371nknFfe8OoA1XdL1uMpAS21M/mGS6BHiYivknbcAPYA8fXtRZLkogMctfb38CnORfhwyFuFdiZr5s0h8DSxiSLxi1M1+Njk9Dl3lABVGqbpkBNGMigVPBknQjVCwgtE+6rK6pR1ymvEwV4J3tdLBji/18wAP1d8bMXGVilxbTw5SqElvIP7n1UNwTpox94IQmEdHh5xQYPDxoCTc4ZJREJEmhEqu/4pj0hCQVeZ1iVYk5GnSaWQt8y8dXOULZ6N60ihbSDcshCx6iILlEJlRFj+gZvaI348l4Md6Nj9HojDHe2UJ/YHz+AK9WnaI=</latexit><latexit sha1_base64="qXd8yxiBWzOEL57x+yWmNbP8hNE=">ACF3icbVDJSgNBFOxjXGLevTSGIQoGmaCoBchKIHhYhmgWz0dHqSJj0L3W/EYZi/8OKvePGgiFe9+Td2loMmFjQUVe/xusoOBFdgmt/GzOzc/MJiaim9vLK6tp7Z2KwoP5SUlakvfFmziWKCe6wMHASrBZIR1xasavfPB371nknFfe8OoA1XdL1uMpAS21M/mGS6BHiYivknbcAPYA8fXtRZLkogMctfb38CnORfhwyFuFdiZr5s0h8DSxiSLxi1M1+Njk9Dl3lABVGqbpkBNGMigVPBknQjVCwgtE+6rK6pR1ymvEwV4J3tdLBji/18wAP1d8bMXGVilxbTw5SqElvIP7n1UNwTpox94IQmEdHh5xQYPDxoCTc4ZJREJEmhEqu/4pj0hCQVeZ1iVYk5GnSaWQt8y8dXOULZ6N60ihbSDcshCx6iILlEJlRFj+gZvaI348l4Md6Nj9HojDHe2UJ/YHz+AK9WnaI=</latexit>

The question again becomes how to compute: rθL(θ)

<latexit sha1_base64="9sOcX05AFBnmvx4hix7d5k7hXFY=">AC3icbVA9SwNBEN3zM8avqKXNkSDEJtyJoGXQxsIigvmA3HMbTbJkr29Y3dOCEd6G/+KjYUitv4BO/+Ne0kKTXw8Hhvhpl5YSK4Rsf5tlZW19Y3Ngtbxe2d3b390sFhS8epoqxJYxGrTgiaCS5ZEzkK1kUgygUrB2OrnO/cCU5rG8x3HC/AgGkvc5BTRSUCp7EkIBQebhkCFMvAhwSEFkt9WZcjoJShWn5kxhLxN3TipkjkZQ+vJ6MU0jJpEK0LrOgn6GSjkVLBJ0Us1S4COYMC6hkqImPaz6S8T+8QoPbsfK1MS7an6eyKDSOtxFJrO/FS96OXif143xf6ln3GZpMgknS3qp8LG2M6DsXtcMYpibAhQxc2tNh2CAomvqIJwV18eZm0zmquU3Pvziv1q3kcBXJMyqRKXHJB6uSGNEiTUPJInskrebOerBfr3fqYta5Y85kj8gfW5w+XqZtf</latexit><latexit sha1_base64="9sOcX05AFBnmvx4hix7d5k7hXFY=">AC3icbVA9SwNBEN3zM8avqKXNkSDEJtyJoGXQxsIigvmA3HMbTbJkr29Y3dOCEd6G/+KjYUitv4BO/+Ne0kKTXw8Hhvhpl5YSK4Rsf5tlZW19Y3Ngtbxe2d3b390sFhS8epoqxJYxGrTgiaCS5ZEzkK1kUgygUrB2OrnO/cCU5rG8x3HC/AgGkvc5BTRSUCp7EkIBQebhkCFMvAhwSEFkt9WZcjoJShWn5kxhLxN3TipkjkZQ+vJ6MU0jJpEK0LrOgn6GSjkVLBJ0Us1S4COYMC6hkqImPaz6S8T+8QoPbsfK1MS7an6eyKDSOtxFJrO/FS96OXif143xf6ln3GZpMgknS3qp8LG2M6DsXtcMYpibAhQxc2tNh2CAomvqIJwV18eZm0zmquU3Pvziv1q3kcBXJMyqRKXHJB6uSGNEiTUPJInskrebOerBfr3fqYta5Y85kj8gfW5w+XqZtf</latexit><latexit sha1_base64="9sOcX05AFBnmvx4hix7d5k7hXFY=">AC3icbVA9SwNBEN3zM8avqKXNkSDEJtyJoGXQxsIigvmA3HMbTbJkr29Y3dOCEd6G/+KjYUitv4BO/+Ne0kKTXw8Hhvhpl5YSK4Rsf5tlZW19Y3Ngtbxe2d3b390sFhS8epoqxJYxGrTgiaCS5ZEzkK1kUgygUrB2OrnO/cCU5rG8x3HC/AgGkvc5BTRSUCp7EkIBQebhkCFMvAhwSEFkt9WZcjoJShWn5kxhLxN3TipkjkZQ+vJ6MU0jJpEK0LrOgn6GSjkVLBJ0Us1S4COYMC6hkqImPaz6S8T+8QoPbsfK1MS7an6eyKDSOtxFJrO/FS96OXif143xf6ln3GZpMgknS3qp8LG2M6DsXtcMYpibAhQxc2tNh2CAomvqIJwV18eZm0zmquU3Pvziv1q3kcBXJMyqRKXHJB6uSGNEiTUPJInskrebOerBfr3fqYta5Y85kj8gfW5w+XqZtf</latexit><latexit sha1_base64="9sOcX05AFBnmvx4hix7d5k7hXFY=">AC3icbVA9SwNBEN3zM8avqKXNkSDEJtyJoGXQxsIigvmA3HMbTbJkr29Y3dOCEd6G/+KjYUitv4BO/+Ne0kKTXw8Hhvhpl5YSK4Rsf5tlZW19Y3Ngtbxe2d3b390sFhS8epoqxJYxGrTgiaCS5ZEzkK1kUgygUrB2OrnO/cCU5rG8x3HC/AgGkvc5BTRSUCp7EkIBQebhkCFMvAhwSEFkt9WZcjoJShWn5kxhLxN3TipkjkZQ+vJ6MU0jJpEK0LrOgn6GSjkVLBJ0Us1S4COYMC6hkqImPaz6S8T+8QoPbsfK1MS7an6eyKDSOtxFJrO/FS96OXif143xf6ln3GZpMgknS3qp8LG2M6DsXtcMYpibAhQxc2tNh2CAomvqIJwV18eZm0zmquU3Pvziv1q3kcBXJMyqRKXHJB6uSGNEiTUPJInskrebOerBfr3fqYta5Y85kj8gfW5w+XqZtf</latexit>

θ = {W(1), b(1), W(2), b(2), w(o), b(o)}

<latexit sha1_base64="80hiwkos5ckF2sLXO4G6OjDxjMw=">ACV3icbZFdS8MwFIbTbuqcX1MvQkOQUFGK4LeCENvJzgPmCtI81SF0zTkpwqo/RPijf7K95ouk3Q1QOBl+e8h5y8CRLBNTjOzLIr1bX1jdpmfWt7Z3evsX/Q03GqKOvSWMRqEBDNBJesCxwEGySKkSgQrB+83BX9/itTmsfyEaYJ8yPyLHnIKQGDRg3pwYQBwTfYy7yIwCQIs37+lJ26Z/k5/iFBicw9FyXPH/JWkLgwUJ4+ajRdFrOvHBZuEvRMvqjBrv3jimacQkUEG0HrpOAn5GFHAqWF73Us0SQl/IMxsaKUnEtJ/Nc8nxiSFjHMbKHAl4Tn9PZCTSehoFxlmsrFd7BfyvN0whvPYzLpMUmKSLi8JUYIhxETIec8UoiKkRhCpudsV0QhShYL6ibkJwV59cFr2Llu03IfLZvt2GUcNHaFjdIpcdIXa6B51UBdR9IE+rYpVtWbWl71u1xZW21rOHKI/Ze9/AyNus4=</latexit><latexit sha1_base64="80hiwkos5ckF2sLXO4G6OjDxjMw=">ACV3icbZFdS8MwFIbTbuqcX1MvQkOQUFGK4LeCENvJzgPmCtI81SF0zTkpwqo/RPijf7K95ouk3Q1QOBl+e8h5y8CRLBNTjOzLIr1bX1jdpmfWt7Z3evsX/Q03GqKOvSWMRqEBDNBJesCxwEGySKkSgQrB+83BX9/itTmsfyEaYJ8yPyLHnIKQGDRg3pwYQBwTfYy7yIwCQIs37+lJ26Z/k5/iFBicw9FyXPH/JWkLgwUJ4+ajRdFrOvHBZuEvRMvqjBrv3jimacQkUEG0HrpOAn5GFHAqWF73Us0SQl/IMxsaKUnEtJ/Nc8nxiSFjHMbKHAl4Tn9PZCTSehoFxlmsrFd7BfyvN0whvPYzLpMUmKSLi8JUYIhxETIec8UoiKkRhCpudsV0QhShYL6ibkJwV59cFr2Llu03IfLZvt2GUcNHaFjdIpcdIXa6B51UBdR9IE+rYpVtWbWl71u1xZW21rOHKI/Ze9/AyNus4=</latexit><latexit sha1_base64="80hiwkos5ckF2sLXO4G6OjDxjMw=">ACV3icbZFdS8MwFIbTbuqcX1MvQkOQUFGK4LeCENvJzgPmCtI81SF0zTkpwqo/RPijf7K95ouk3Q1QOBl+e8h5y8CRLBNTjOzLIr1bX1jdpmfWt7Z3evsX/Q03GqKOvSWMRqEBDNBJesCxwEGySKkSgQrB+83BX9/itTmsfyEaYJ8yPyLHnIKQGDRg3pwYQBwTfYy7yIwCQIs37+lJ26Z/k5/iFBicw9FyXPH/JWkLgwUJ4+ajRdFrOvHBZuEvRMvqjBrv3jimacQkUEG0HrpOAn5GFHAqWF73Us0SQl/IMxsaKUnEtJ/Nc8nxiSFjHMbKHAl4Tn9PZCTSehoFxlmsrFd7BfyvN0whvPYzLpMUmKSLi8JUYIhxETIec8UoiKkRhCpudsV0QhShYL6ibkJwV59cFr2Llu03IfLZvt2GUcNHaFjdIpcdIXa6B51UBdR9IE+rYpVtWbWl71u1xZW21rOHKI/Ze9/AyNus4=</latexit><latexit sha1_base64="80hiwkos5ckF2sLXO4G6OjDxjMw=">ACV3icbZFdS8MwFIbTbuqcX1MvQkOQUFGK4LeCENvJzgPmCtI81SF0zTkpwqo/RPijf7K95ouk3Q1QOBl+e8h5y8CRLBNTjOzLIr1bX1jdpmfWt7Z3evsX/Q03GqKOvSWMRqEBDNBJesCxwEGySKkSgQrB+83BX9/itTmsfyEaYJ8yPyLHnIKQGDRg3pwYQBwTfYy7yIwCQIs37+lJ26Z/k5/iFBicw9FyXPH/JWkLgwUJ4+ajRdFrOvHBZuEvRMvqjBrv3jimacQkUEG0HrpOAn5GFHAqWF73Us0SQl/IMxsaKUnEtJ/Nc8nxiSFjHMbKHAl4Tn9PZCTSehoFxlmsrFd7BfyvN0whvPYzLpMUmKSLi8JUYIhxETIec8UoiKkRhCpudsV0QhShYL6ibkJwV59cFr2Llu03IfLZvt2GUcNHaFjdIpcdIXa6B51UBdR9IE+rYpVtWbWl71u1xZW21rOHKI/Ze9/AyNus4=</latexit>
  • Multi-class classification ( classes)

C

yi = softmaxi(W(o)h2 + b(o))

<latexit sha1_base64="WFlRAVK0TD10Zz5eyplJUoJ8FXY=">ACMHicbVBNS8NAEN34WetX1aOXxSK0CUpgl6Eogc9KlgrNDFsthu7uMmG3Ym0hPwkL/4UvSgo4tVf4bZG0OqDhTfvzbAzL0gE12Dbz9bU9Mzs3Hxpoby4tLyWlbv9AyVZS1qRSXQZEM8Fj1gYOgl0mipEoEKwT3ByN/M4tU5rL+ByGCfMich3zkFMCRvIrx0Of4wPsAhtApmUIERnkPq+5EYF+EGad/CqryXr+Xfdzv4l38HcZFHbdr1Tthj0G/kucglRgVO/8uD2JE0jFgMVROuYyfgZUQBp4LlZTfVLCH0hlyzrqExiZj2svHBOd42Sg+HUpkXAx6rPycyEmk9jALTOVpUT3oj8T+vm0K472U8TlJgMf36KEwFBolH6eEeV4yCGBpCqOJmV0z7RBEKJuOyCcGZPkvuWg2HLvhnO1W4dFHCW0ibZQDTloD7XQCTpFbUTRHXpEL+jVurerDfr/at1yipmNtAvWB+fypup/g=</latexit><latexit sha1_base64="WFlRAVK0TD10Zz5eyplJUoJ8FXY=">ACMHicbVBNS8NAEN34WetX1aOXxSK0CUpgl6Eogc9KlgrNDFsthu7uMmG3Ym0hPwkL/4UvSgo4tVf4bZG0OqDhTfvzbAzL0gE12Dbz9bU9Mzs3Hxpoby4tLyWlbv9AyVZS1qRSXQZEM8Fj1gYOgl0mipEoEKwT3ByN/M4tU5rL+ByGCfMich3zkFMCRvIrx0Of4wPsAhtApmUIERnkPq+5EYF+EGad/CqryXr+Xfdzv4l38HcZFHbdr1Tthj0G/kucglRgVO/8uD2JE0jFgMVROuYyfgZUQBp4LlZTfVLCH0hlyzrqExiZj2svHBOd42Sg+HUpkXAx6rPycyEmk9jALTOVpUT3oj8T+vm0K472U8TlJgMf36KEwFBolH6eEeV4yCGBpCqOJmV0z7RBEKJuOyCcGZPkvuWg2HLvhnO1W4dFHCW0ibZQDTloD7XQCTpFbUTRHXpEL+jVurerDfr/at1yipmNtAvWB+fypup/g=</latexit><latexit sha1_base64="WFlRAVK0TD10Zz5eyplJUoJ8FXY=">ACMHicbVBNS8NAEN34WetX1aOXxSK0CUpgl6Eogc9KlgrNDFsthu7uMmG3Ym0hPwkL/4UvSgo4tVf4bZG0OqDhTfvzbAzL0gE12Dbz9bU9Mzs3Hxpoby4tLyWlbv9AyVZS1qRSXQZEM8Fj1gYOgl0mipEoEKwT3ByN/M4tU5rL+ByGCfMich3zkFMCRvIrx0Of4wPsAhtApmUIERnkPq+5EYF+EGad/CqryXr+Xfdzv4l38HcZFHbdr1Tthj0G/kucglRgVO/8uD2JE0jFgMVROuYyfgZUQBp4LlZTfVLCH0hlyzrqExiZj2svHBOd42Sg+HUpkXAx6rPycyEmk9jALTOVpUT3oj8T+vm0K472U8TlJgMf36KEwFBolH6eEeV4yCGBpCqOJmV0z7RBEKJuOyCcGZPkvuWg2HLvhnO1W4dFHCW0ibZQDTloD7XQCTpFbUTRHXpEL+jVurerDfr/at1yipmNtAvWB+fypup/g=</latexit><latexit sha1_base64="WFlRAVK0TD10Zz5eyplJUoJ8FXY=">ACMHicbVBNS8NAEN34WetX1aOXxSK0CUpgl6Eogc9KlgrNDFsthu7uMmG3Ym0hPwkL/4UvSgo4tVf4bZG0OqDhTfvzbAzL0gE12Dbz9bU9Mzs3Hxpoby4tLyWlbv9AyVZS1qRSXQZEM8Fj1gYOgl0mipEoEKwT3ByN/M4tU5rL+ByGCfMich3zkFMCRvIrx0Of4wPsAhtApmUIERnkPq+5EYF+EGad/CqryXr+Xfdzv4l38HcZFHbdr1Tthj0G/kucglRgVO/8uD2JE0jFgMVROuYyfgZUQBp4LlZTfVLCH0hlyzrqExiZj2svHBOd42Sg+HUpkXAx6rPycyEmk9jALTOVpUT3oj8T+vm0K472U8TlJgMf36KEwFBolH6eEeV4yCGBpCqOJmV0z7RBEKJuOyCcGZPkvuWg2HLvhnO1W4dFHCW0ibZQDTloD7XQCTpFbUTRHXpEL+jVurerDfr/at1yipmNtAvWB+fypup/g=</latexit>

L(y, y∗) = −

C

X

i=1

y∗

i log yi

<latexit sha1_base64="vO7WpM8DRV37pdQR4k280HdhKGE=">ACHnicbVDLSgMxFM34rPVdekmWIQqWmZE0U2h2I0LFxXsAzrtkEnTNjTzIMkIQ5gvceOvuHGhiOBK/8ZMOwtPRA4nHMvN+e4IaNCmua3sbC4tLymlvLr29sbm0XdnabIog4Jg0csIC3XSQIoz5pSCoZaYecIM9lpOWOa6nfeiBc0MC/l3FIuh4a+nRAMZJacgoXtofkCOmbpNSfALj3vERrMBTW0Seo2jFSnqligtO9RmwVDFDk0Sp1A0y+YEcJ5YGSmCDHWn8Gn3Ax5xJeYISE6lhnKrkJcUsxIkrcjQUKEx2hIOpr6yCOiqybxEniolT4cBFw/X8KJ+ntDIU+I2HP1ZBpGzHqp+J/XieTgquoH0aS+Hh6aBAxKAOYdgX7lBMsWawJwpzqv0I8QhxhqRvN6xKs2cjzpHlWtsydXderF5ndeTAPjgAJWCBS1AFN6AOGgCDR/AMXsGb8WS8GO/Gx3R0wch29sAfGF8/Vf2iCA=</latexit><latexit sha1_base64="vO7WpM8DRV37pdQR4k280HdhKGE=">ACHnicbVDLSgMxFM34rPVdekmWIQqWmZE0U2h2I0LFxXsAzrtkEnTNjTzIMkIQ5gvceOvuHGhiOBK/8ZMOwtPRA4nHMvN+e4IaNCmua3sbC4tLymlvLr29sbm0XdnabIog4Jg0csIC3XSQIoz5pSCoZaYecIM9lpOWOa6nfeiBc0MC/l3FIuh4a+nRAMZJacgoXtofkCOmbpNSfALj3vERrMBTW0Seo2jFSnqligtO9RmwVDFDk0Sp1A0y+YEcJ5YGSmCDHWn8Gn3Ax5xJeYISE6lhnKrkJcUsxIkrcjQUKEx2hIOpr6yCOiqybxEniolT4cBFw/X8KJ+ntDIU+I2HP1ZBpGzHqp+J/XieTgquoH0aS+Hh6aBAxKAOYdgX7lBMsWawJwpzqv0I8QhxhqRvN6xKs2cjzpHlWtsydXderF5ndeTAPjgAJWCBS1AFN6AOGgCDR/AMXsGb8WS8GO/Gx3R0wch29sAfGF8/Vf2iCA=</latexit><latexit sha1_base64="vO7WpM8DRV37pdQR4k280HdhKGE=">ACHnicbVDLSgMxFM34rPVdekmWIQqWmZE0U2h2I0LFxXsAzrtkEnTNjTzIMkIQ5gvceOvuHGhiOBK/8ZMOwtPRA4nHMvN+e4IaNCmua3sbC4tLymlvLr29sbm0XdnabIog4Jg0csIC3XSQIoz5pSCoZaYecIM9lpOWOa6nfeiBc0MC/l3FIuh4a+nRAMZJacgoXtofkCOmbpNSfALj3vERrMBTW0Seo2jFSnqligtO9RmwVDFDk0Sp1A0y+YEcJ5YGSmCDHWn8Gn3Ax5xJeYISE6lhnKrkJcUsxIkrcjQUKEx2hIOpr6yCOiqybxEniolT4cBFw/X8KJ+ntDIU+I2HP1ZBpGzHqp+J/XieTgquoH0aS+Hh6aBAxKAOYdgX7lBMsWawJwpzqv0I8QhxhqRvN6xKs2cjzpHlWtsydXderF5ndeTAPjgAJWCBS1AFN6AOGgCDR/AMXsGb8WS8GO/Gx3R0wch29sAfGF8/Vf2iCA=</latexit><latexit sha1_base64="vO7WpM8DRV37pdQR4k280HdhKGE=">ACHnicbVDLSgMxFM34rPVdekmWIQqWmZE0U2h2I0LFxXsAzrtkEnTNjTzIMkIQ5gvceOvuHGhiOBK/8ZMOwtPRA4nHMvN+e4IaNCmua3sbC4tLymlvLr29sbm0XdnabIog4Jg0csIC3XSQIoz5pSCoZaYecIM9lpOWOa6nfeiBc0MC/l3FIuh4a+nRAMZJacgoXtofkCOmbpNSfALj3vERrMBTW0Seo2jFSnqligtO9RmwVDFDk0Sp1A0y+YEcJ5YGSmCDHWn8Gn3Ax5xJeYISE6lhnKrkJcUsxIkrcjQUKEx2hIOpr6yCOiqybxEniolT4cBFw/X8KJ+ntDIU+I2HP1ZBpGzHqp+J/XieTgquoH0aS+Hh6aBAxKAOYdgX7lBMsWawJwpzqv0I8QhxhqRvN6xKs2cjzpHlWtsydXderF5ndeTAPjgAJWCBS1AFN6AOGgCDR/AMXsGb8WS8GO/Gx3R0wch29sAfGF8/Vf2iCA=</latexit>

W(o) ∈ RC×d2, b(o) ∈ RC

<latexit sha1_base64="l0JO+rOZ6HhCqn5qpiARnWnva8=">ACPHicdVBLSwMxGMz6rPV9eglWIQKUnaLoMdiLx7row/o1iXJZtvQbHZJskJZ9od58Ud48+TFgyJePZs+BG1IDMzEe+b3DMmdK2/WQtLC4tr6zm1vLrG5tb24Wd3aKEklog0Q8km2MFOVM0IZmtN2LCkKMactPKiN/NYdlYpF4kYPY9oNU+wgBGkjeQVrt0Q6T4O0lZ2m5aiowy6TMCJiNMrI9agq1lIFfS9SnYMv/P43zmFYp2R4DzhNnSopgirpXeHT9iCQhFZpwpFTHsWPdTZHUjHCa5d1E0RiRAerRjqECmXW6fj4DB4axYdBJM0TGo7VnxMpCpUahtgkR1uqW8k/uV1Eh2cdVMm4kRTQSYfBQmHOoKjJqHPJCWaDw1BRDKzKyR9JBHRpu+8KcGZPXmeNCtlxy47lyfF6vm0jhzYBwegBxwCqrgAtRBAxBwD57BK3izHqwX6936mEQXrOnMHvgF6/MLCiKumQ=</latexit><latexit sha1_base64="l0JO+rOZ6HhCqn5qpiARnWnva8=">ACPHicdVBLSwMxGMz6rPV9eglWIQKUnaLoMdiLx7row/o1iXJZtvQbHZJskJZ9od58Ud48+TFgyJePZs+BG1IDMzEe+b3DMmdK2/WQtLC4tr6zm1vLrG5tb24Wd3aKEklog0Q8km2MFOVM0IZmtN2LCkKMactPKiN/NYdlYpF4kYPY9oNU+wgBGkjeQVrt0Q6T4O0lZ2m5aiowy6TMCJiNMrI9agq1lIFfS9SnYMv/P43zmFYp2R4DzhNnSopgirpXeHT9iCQhFZpwpFTHsWPdTZHUjHCa5d1E0RiRAerRjqECmXW6fj4DB4axYdBJM0TGo7VnxMpCpUahtgkR1uqW8k/uV1Eh2cdVMm4kRTQSYfBQmHOoKjJqHPJCWaDw1BRDKzKyR9JBHRpu+8KcGZPXmeNCtlxy47lyfF6vm0jhzYBwegBxwCqrgAtRBAxBwD57BK3izHqwX6936mEQXrOnMHvgF6/MLCiKumQ=</latexit><latexit sha1_base64="l0JO+rOZ6HhCqn5qpiARnWnva8=">ACPHicdVBLSwMxGMz6rPV9eglWIQKUnaLoMdiLx7row/o1iXJZtvQbHZJskJZ9od58Ud48+TFgyJePZs+BG1IDMzEe+b3DMmdK2/WQtLC4tr6zm1vLrG5tb24Wd3aKEklog0Q8km2MFOVM0IZmtN2LCkKMactPKiN/NYdlYpF4kYPY9oNU+wgBGkjeQVrt0Q6T4O0lZ2m5aiowy6TMCJiNMrI9agq1lIFfS9SnYMv/P43zmFYp2R4DzhNnSopgirpXeHT9iCQhFZpwpFTHsWPdTZHUjHCa5d1E0RiRAerRjqECmXW6fj4DB4axYdBJM0TGo7VnxMpCpUahtgkR1uqW8k/uV1Eh2cdVMm4kRTQSYfBQmHOoKjJqHPJCWaDw1BRDKzKyR9JBHRpu+8KcGZPXmeNCtlxy47lyfF6vm0jhzYBwegBxwCqrgAtRBAxBwD57BK3izHqwX6936mEQXrOnMHvgF6/MLCiKumQ=</latexit><latexit sha1_base64="l0JO+rOZ6HhCqn5qpiARnWnva8=">ACPHicdVBLSwMxGMz6rPV9eglWIQKUnaLoMdiLx7row/o1iXJZtvQbHZJskJZ9od58Ud48+TFgyJePZs+BG1IDMzEe+b3DMmdK2/WQtLC4tr6zm1vLrG5tb24Wd3aKEklog0Q8km2MFOVM0IZmtN2LCkKMactPKiN/NYdlYpF4kYPY9oNU+wgBGkjeQVrt0Q6T4O0lZ2m5aiowy6TMCJiNMrI9agq1lIFfS9SnYMv/P43zmFYp2R4DzhNnSopgirpXeHT9iCQhFZpwpFTHsWPdTZHUjHCa5d1E0RiRAerRjqECmXW6fj4DB4axYdBJM0TGo7VnxMpCpUahtgkR1uqW8k/uV1Eh2cdVMm4kRTQSYfBQmHOoKjJqHPJCWaDw1BRDKzKyR9JBHRpu+8KcGZPXmeNCtlxy47lyfF6vm0jhzYBwegBxwCqrgAtRBAxBwD57BK3izHqwX6936mEQXrOnMHvgF6/MLCiKumQ=</latexit>

softmaxi(x) = exp(xi) P

j exp(xj)

<latexit sha1_base64="s3M9/5vHZ+rXnSogPG9mjZAO/AY=">ACI3icbVDLSsNAFJ34rPVdelmsAjtpiRSUASh6MZlBfuAJoTJdNJO5OEmYmkhPyLG3/FjQuluHhvzhts9DWAwOHc+7lzjlexKhUpvlrK1vbG5tF3aKu3v7B4elo+O2DGOBSQuHLBRdD0nCaEBaipGupEgiHuMdLzx3czvPBEhaRg8qklEHI4GAfUpRkpLbuna5kgNBU9l6CuOksylaQKb6DtC4RTmyRJXFpNUtGXN3BHNlVM3cUtmsmXPAVWLlpAxyN3S1O6HOYkUJghKXuWGSknRUJRzEhWtGNJIoTHaEB6mgaIE+mk84wZPNdKH/qh0C9QcK7+3kgRl3LCPT05SySXvZn4n9eLlX/lpDSIYkUCvDjkxwyqEM4Kg30qCFZsognCguq/QjxEuhylay3qEqzlyKukfVGz6rX6Q73cuM3rKIBTcAYqwAKXoAHuQRO0AbP4BW8gw/jxXgzpsbnYnTNyHdOwB8Y3z+erKTa</latexit>
slide-24
SLIDE 24

Optimization

  • Logistic regression is convex: one global minimum

θ(t+1) = θ(t) ηrθJ(θ)

<latexit sha1_base64="2xbrEJR+XVhUcysjVyGPSHic0HY=">ACJnicbVDLSgNBEJz1GeMr6tHLYBAiYtgVQS+C6EU8RTBRyK5L72RihszOLjO9QljyNV78FS8eIiLe/BQnD/BZMFBd1U1PV5RKYdB1352p6ZnZufnCQnFxaXltbS23jBJphmvs0Qm+iYCw6VQvI4CJb9JNYc4kvw6p4N/et7ro1I1BX2Uh7EcKdEWzBAK4WlYx87HOE2r+Cut9Onx/RLsOUe9W1BfQWRhDAfe/2LypjshKWyW3VHoH+JNyFlMkEtLA38VsKymCtkEoxpem6KQ4aBZO8X/Qzw1NgXbjTUsVxNwE+ejMPt2Sou2E2fQjpSv0/kEBvTiyPbGQN2zG9vKP7nNTNsHwW5UGmGXLHxonYmKSZ0mBltCc0Zyp4lwLSwf6WsAxoY2mSLNgTv98l/SWO/6rlV7/KgfHI6iaNANskWqRCPHJITck5qpE4YeSBPZEBenEfn2Xl13satU85kZoP8gPxCRNypFM=</latexit><latexit sha1_base64="2xbrEJR+XVhUcysjVyGPSHic0HY=">ACJnicbVDLSgNBEJz1GeMr6tHLYBAiYtgVQS+C6EU8RTBRyK5L72RihszOLjO9QljyNV78FS8eIiLe/BQnD/BZMFBd1U1PV5RKYdB1352p6ZnZufnCQnFxaXltbS23jBJphmvs0Qm+iYCw6VQvI4CJb9JNYc4kvw6p4N/et7ro1I1BX2Uh7EcKdEWzBAK4WlYx87HOE2r+Cut9Onx/RLsOUe9W1BfQWRhDAfe/2LypjshKWyW3VHoH+JNyFlMkEtLA38VsKymCtkEoxpem6KQ4aBZO8X/Qzw1NgXbjTUsVxNwE+ejMPt2Sou2E2fQjpSv0/kEBvTiyPbGQN2zG9vKP7nNTNsHwW5UGmGXLHxonYmKSZ0mBltCc0Zyp4lwLSwf6WsAxoY2mSLNgTv98l/SWO/6rlV7/KgfHI6iaNANskWqRCPHJITck5qpE4YeSBPZEBenEfn2Xl13satU85kZoP8gPxCRNypFM=</latexit><latexit sha1_base64="2xbrEJR+XVhUcysjVyGPSHic0HY=">ACJnicbVDLSgNBEJz1GeMr6tHLYBAiYtgVQS+C6EU8RTBRyK5L72RihszOLjO9QljyNV78FS8eIiLe/BQnD/BZMFBd1U1PV5RKYdB1352p6ZnZufnCQnFxaXltbS23jBJphmvs0Qm+iYCw6VQvI4CJb9JNYc4kvw6p4N/et7ro1I1BX2Uh7EcKdEWzBAK4WlYx87HOE2r+Cut9Onx/RLsOUe9W1BfQWRhDAfe/2LypjshKWyW3VHoH+JNyFlMkEtLA38VsKymCtkEoxpem6KQ4aBZO8X/Qzw1NgXbjTUsVxNwE+ejMPt2Sou2E2fQjpSv0/kEBvTiyPbGQN2zG9vKP7nNTNsHwW5UGmGXLHxonYmKSZ0mBltCc0Zyp4lwLSwf6WsAxoY2mSLNgTv98l/SWO/6rlV7/KgfHI6iaNANskWqRCPHJITck5qpE4YeSBPZEBenEfn2Xl13satU85kZoP8gPxCRNypFM=</latexit><latexit sha1_base64="2xbrEJR+XVhUcysjVyGPSHic0HY=">ACJnicbVDLSgNBEJz1GeMr6tHLYBAiYtgVQS+C6EU8RTBRyK5L72RihszOLjO9QljyNV78FS8eIiLe/BQnD/BZMFBd1U1PV5RKYdB1352p6ZnZufnCQnFxaXltbS23jBJphmvs0Qm+iYCw6VQvI4CJb9JNYc4kvw6p4N/et7ro1I1BX2Uh7EcKdEWzBAK4WlYx87HOE2r+Cut9Onx/RLsOUe9W1BfQWRhDAfe/2LypjshKWyW3VHoH+JNyFlMkEtLA38VsKymCtkEoxpem6KQ4aBZO8X/Qzw1NgXbjTUsVxNwE+ejMPt2Sou2E2fQjpSv0/kEBvTiyPbGQN2zG9vKP7nNTNsHwW5UGmGXLHxonYmKSZ0mBltCc0Zyp4lwLSwf6WsAxoY2mSLNgTv98l/SWO/6rlV7/KgfHI6iaNANskWqRCPHJITck5qpE4YeSBPZEBenEfn2Xl13satU85kZoP8gPxCRNypFM=</latexit>
  • Neural networks are non-convex and not easy to optimize
  • A class of more sophisticated “adaptive” optimizers

that scale the parameter adjustment by an accumulated gradient.

  • Adam
  • Adagrad
  • RMSprop

(Ruder 2016): An overview of gradient descent optimization algorithms

slide-25
SLIDE 25

Dropout

  • Form of regularization NNs
  • Idea: “Handicap” NN by removing hidden

units stochastically

  • set each hidden unit in a layer to 0 with

probability during training ( usually works well)

  • scale outputs by
  • hidden units forced to learn more

general patterns

  • Test time: Simply compute identity

p p = 0.5 1/(1 − p)

slide-26
SLIDE 26

What can we compute with NNs?

slide-27
SLIDE 27

g

Perceptron, linear classification, Boolean functions: xi∈{0,1}

  • Can learn x1 ∨ x2?
  • -0.5 + x1 + x2
  • Can learn x1 ∧ x2?
  • -1.5 + x1 + x2
  • Can learn any conjunction or disjunction?
  • 0.5 + x1 + … + xn
  • (-n+0.5) + x1 + … + xn
  • Can learn majority?
  • (-0.5*n) + x1 + … + xn
  • What are we missing? The dreaded XOR!, etc.
slide-28
SLIDE 28

Going beyond linear classification

Solving the XOR problem y = x1 XOR x2 v1 = (x1 ∧ ¬x2) = -1.5+2x1-x2 v2 = (x2 ∧ ¬x1) = -1.5+2x2-x1 y = v1∨ v2 = -0.5+v1+v2

x1 x2 1 v1 v2 y 1

  • 0.5

1 1

  • 1.5

2

  • 1

2

  • 1
  • 1.5

= (x1 ∧ ¬x2) ∨ (x2 ∧ ¬x1)

slide-29
SLIDE 29

Example data for NN with hidden layer

slide-30
SLIDE 30

Learned weights for hidden layer

slide-31
SLIDE 31

Applications

slide-32
SLIDE 32

Neural Bag-of-Words (NBOW)

  • Deep Averaging Networks (DAN) for Text Classification

(Iyyer et 2015): Deep Unordered Composition Rivals Syntactic Methods for Text Classification

word embeddings averaging

slide-33
SLIDE 33

Word embeddings: re-train or not?

  • Word embeddings can be treated as parameters too!

Eemb

<latexit sha1_base64="xZKtUndX/j5JTnEBM9tx9pw6ToM=">AB+XicdVDLSsNAFJ34rPUVdelmsBRchaSGtu6KIrisYB/QhjCZTtqhk0mYmRKyJ+4caGIW/EnX/jpK2gogcuHM65lzlzgoRqWz7w1hb39jc2i7tlHf39g8OzaPjroxTgUkHxywW/QBJwignHUVI/1EBQFjPSC6Xh92ZESBrzezVPiBehMachxUhpyTfNYTUJAizm9zPSBTkvlmxrctmvebWoW3ZdsOpOQWpNdwLFzpaKVABK7R98304inEaEa4wQ1IOHDtRXoaEopiRvDxMJUkQnqIxGWjKUSkly2S57CqlREMY6GHK7hQv19kKJyrmPBapFT/vYK8S9vkKqw6WUJ6kiHC8fClMGVQyLGuCICoIVm2uCsKA6K8QTJBWuqyLuHrp/B/0q1Zjm05d26ldbWqowROwRk4Bw5ogBa4BW3QARjMwAN4As9GZjwaL8brcnXNWN2cgB8w3j4BXvmUIw=</latexit><latexit sha1_base64="xZKtUndX/j5JTnEBM9tx9pw6ToM=">AB+XicdVDLSsNAFJ34rPUVdelmsBRchaSGtu6KIrisYB/QhjCZTtqhk0mYmRKyJ+4caGIW/EnX/jpK2gogcuHM65lzlzgoRqWz7w1hb39jc2i7tlHf39g8OzaPjroxTgUkHxywW/QBJwignHUVI/1EBQFjPSC6Xh92ZESBrzezVPiBehMachxUhpyTfNYTUJAizm9zPSBTkvlmxrctmvebWoW3ZdsOpOQWpNdwLFzpaKVABK7R98304inEaEa4wQ1IOHDtRXoaEopiRvDxMJUkQnqIxGWjKUSkly2S57CqlREMY6GHK7hQv19kKJyrmPBapFT/vYK8S9vkKqw6WUJ6kiHC8fClMGVQyLGuCICoIVm2uCsKA6K8QTJBWuqyLuHrp/B/0q1Zjm05d26ldbWqowROwRk4Bw5ogBa4BW3QARjMwAN4As9GZjwaL8brcnXNWN2cgB8w3j4BXvmUIw=</latexit><latexit sha1_base64="xZKtUndX/j5JTnEBM9tx9pw6ToM=">AB+XicdVDLSsNAFJ34rPUVdelmsBRchaSGtu6KIrisYB/QhjCZTtqhk0mYmRKyJ+4caGIW/EnX/jpK2gogcuHM65lzlzgoRqWz7w1hb39jc2i7tlHf39g8OzaPjroxTgUkHxywW/QBJwignHUVI/1EBQFjPSC6Xh92ZESBrzezVPiBehMachxUhpyTfNYTUJAizm9zPSBTkvlmxrctmvebWoW3ZdsOpOQWpNdwLFzpaKVABK7R98304inEaEa4wQ1IOHDtRXoaEopiRvDxMJUkQnqIxGWjKUSkly2S57CqlREMY6GHK7hQv19kKJyrmPBapFT/vYK8S9vkKqw6WUJ6kiHC8fClMGVQyLGuCICoIVm2uCsKA6K8QTJBWuqyLuHrp/B/0q1Zjm05d26ldbWqowROwRk4Bw5ogBa4BW3QARjMwAN4As9GZjwaL8brcnXNWN2cgB8w3j4BXvmUIw=</latexit><latexit sha1_base64="xZKtUndX/j5JTnEBM9tx9pw6ToM=">AB+XicdVDLSsNAFJ34rPUVdelmsBRchaSGtu6KIrisYB/QhjCZTtqhk0mYmRKyJ+4caGIW/EnX/jpK2gogcuHM65lzlzgoRqWz7w1hb39jc2i7tlHf39g8OzaPjroxTgUkHxywW/QBJwignHUVI/1EBQFjPSC6Xh92ZESBrzezVPiBehMachxUhpyTfNYTUJAizm9zPSBTkvlmxrctmvebWoW3ZdsOpOQWpNdwLFzpaKVABK7R98304inEaEa4wQ1IOHDtRXoaEopiRvDxMJUkQnqIxGWjKUSkly2S57CqlREMY6GHK7hQv19kKJyrmPBapFT/vYK8S9vkKqw6WUJ6kiHC8fClMGVQyLGuCICoIVm2uCsKA6K8QTJBWuqyLuHrp/B/0q1Zjm05d26ldbWqowROwRk4Bw5ogBa4BW3QARjMwAN4As9GZjwaL8brcnXNWN2cgB8w3j4BXvmUIw=</latexit>

θ = {W(1), b(1), W(2), b(2), w(o), b(o),

<latexit sha1_base64="xkXsZt9OPUHRzD3TCKaXsx7aJsA=">ACVnicbZFLSwMxFIUzU2trfY26dBMsQgUpM0XQjVB047KCfUCnlkyaUMzD5I7Shn6J3WjP8WNmGlHsK0XAofvnktuTrxYcAW2/WmYha3idqm8U9nd2z84tI6OypKJGVtGolI9jyimOAhawMHwXqxZCTwBOt60/us31hUvEofIJZzAYBGYfc5SARkMrcGHCgOBb7KZuQGDi+Wl3/pzWnIv5Jf4l3gZeBobnhXympEoI14uhlbVrtuLwpvCyUV5dUaWm/uKJwEKgijVd+wYBimRwKlg84qbKBYTOiVj1tcyJAFTg3QRyxyfazLCfiT1CQEv6N+JlARKzQJPO7ON1Xovg/1+gn4N4OUh3ECLKTLi/xEYIhwljEeckoiJkWhEqud8V0QiShoH+iokNw1p+8KTqNumPXnceravMuj6OMTtEZqiEHXaMmekAt1EYUvaMvwzQKxofxbRbN0tJqGvnMCVop0/oB0OyFw=</latexit><latexit sha1_base64="xkXsZt9OPUHRzD3TCKaXsx7aJsA=">ACVnicbZFLSwMxFIUzU2trfY26dBMsQgUpM0XQjVB047KCfUCnlkyaUMzD5I7Shn6J3WjP8WNmGlHsK0XAofvnktuTrxYcAW2/WmYha3idqm8U9nd2z84tI6OypKJGVtGolI9jyimOAhawMHwXqxZCTwBOt60/us31hUvEofIJZzAYBGYfc5SARkMrcGHCgOBb7KZuQGDi+Wl3/pzWnIv5Jf4l3gZeBobnhXympEoI14uhlbVrtuLwpvCyUV5dUaWm/uKJwEKgijVd+wYBimRwKlg84qbKBYTOiVj1tcyJAFTg3QRyxyfazLCfiT1CQEv6N+JlARKzQJPO7ON1Xovg/1+gn4N4OUh3ECLKTLi/xEYIhwljEeckoiJkWhEqud8V0QiShoH+iokNw1p+8KTqNumPXnceravMuj6OMTtEZqiEHXaMmekAt1EYUvaMvwzQKxofxbRbN0tJqGvnMCVop0/oB0OyFw=</latexit><latexit sha1_base64="xkXsZt9OPUHRzD3TCKaXsx7aJsA=">ACVnicbZFLSwMxFIUzU2trfY26dBMsQgUpM0XQjVB047KCfUCnlkyaUMzD5I7Shn6J3WjP8WNmGlHsK0XAofvnktuTrxYcAW2/WmYha3idqm8U9nd2z84tI6OypKJGVtGolI9jyimOAhawMHwXqxZCTwBOt60/us31hUvEofIJZzAYBGYfc5SARkMrcGHCgOBb7KZuQGDi+Wl3/pzWnIv5Jf4l3gZeBobnhXympEoI14uhlbVrtuLwpvCyUV5dUaWm/uKJwEKgijVd+wYBimRwKlg84qbKBYTOiVj1tcyJAFTg3QRyxyfazLCfiT1CQEv6N+JlARKzQJPO7ON1Xovg/1+gn4N4OUh3ECLKTLi/xEYIhwljEeckoiJkWhEqud8V0QiShoH+iokNw1p+8KTqNumPXnceravMuj6OMTtEZqiEHXaMmekAt1EYUvaMvwzQKxofxbRbN0tJqGvnMCVop0/oB0OyFw=</latexit><latexit sha1_base64="xkXsZt9OPUHRzD3TCKaXsx7aJsA=">ACVnicbZFLSwMxFIUzU2trfY26dBMsQgUpM0XQjVB047KCfUCnlkyaUMzD5I7Shn6J3WjP8WNmGlHsK0XAofvnktuTrxYcAW2/WmYha3idqm8U9nd2z84tI6OypKJGVtGolI9jyimOAhawMHwXqxZCTwBOt60/us31hUvEofIJZzAYBGYfc5SARkMrcGHCgOBb7KZuQGDi+Wl3/pzWnIv5Jf4l3gZeBobnhXympEoI14uhlbVrtuLwpvCyUV5dUaWm/uKJwEKgijVd+wYBimRwKlg84qbKBYTOiVj1tcyJAFTg3QRyxyfazLCfiT1CQEv6N+JlARKzQJPO7ON1Xovg/1+gn4N4OUh3ECLKTLi/xEYIhwljEeckoiJkWhEqud8V0QiShoH+iokNw1p+8KTqNumPXnceravMuj6OMTtEZqiEHXaMmekAt1EYUvaMvwzQKxofxbRbN0tJqGvnMCVop0/oB0OyFw=</latexit>

}

<latexit sha1_base64="RlH2tdjr0xMRC8QkwnZnHd0h1Fs=">AB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8cq9gPaUDbTbt0swm7E6GE/gMvHhTx6j/y5r9x2+agrQ8GHu/NMDMvTKUw6HnfTmltfWNzq7xd2dnd2z9wD49aJsk0402WyER3Qmq4FIo3UaDknVRzGoeSt8Px7cxvP3FtRKIecZLyIKZDJSLBKFrpoTftu1Wv5s1BVolfkCoUaPTdr94gYVnMFTJjen6XopBTjUKJvm0sMTykb0yHvWqpozE2Qzy+dkjOrDEiUaFsKyVz9PZHT2JhJHNrOmOLILHsz8T+vm2F0HeRCpRlyxRaLokwSTMjsbTIQmjOUE0so08LeStiIasrQhlOxIfjL6+S1kXN92r+/W1flPEUYTOIVz8OEK6nAHDWgCgwie4RXenLHz4rw7H4vWklPMHMfOJ8/noyNZw=</latexit><latexit sha1_base64="RlH2tdjr0xMRC8QkwnZnHd0h1Fs=">AB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8cq9gPaUDbTbt0swm7E6GE/gMvHhTx6j/y5r9x2+agrQ8GHu/NMDMvTKUw6HnfTmltfWNzq7xd2dnd2z9wD49aJsk0402WyER3Qmq4FIo3UaDknVRzGoeSt8Px7cxvP3FtRKIecZLyIKZDJSLBKFrpoTftu1Wv5s1BVolfkCoUaPTdr94gYVnMFTJjen6XopBTjUKJvm0sMTykb0yHvWqpozE2Qzy+dkjOrDEiUaFsKyVz9PZHT2JhJHNrOmOLILHsz8T+vm2F0HeRCpRlyxRaLokwSTMjsbTIQmjOUE0so08LeStiIasrQhlOxIfjL6+S1kXN92r+/W1flPEUYTOIVz8OEK6nAHDWgCgwie4RXenLHz4rw7H4vWklPMHMfOJ8/noyNZw=</latexit><latexit sha1_base64="RlH2tdjr0xMRC8QkwnZnHd0h1Fs=">AB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8cq9gPaUDbTbt0swm7E6GE/gMvHhTx6j/y5r9x2+agrQ8GHu/NMDMvTKUw6HnfTmltfWNzq7xd2dnd2z9wD49aJsk0402WyER3Qmq4FIo3UaDknVRzGoeSt8Px7cxvP3FtRKIecZLyIKZDJSLBKFrpoTftu1Wv5s1BVolfkCoUaPTdr94gYVnMFTJjen6XopBTjUKJvm0sMTykb0yHvWqpozE2Qzy+dkjOrDEiUaFsKyVz9PZHT2JhJHNrOmOLILHsz8T+vm2F0HeRCpRlyxRaLokwSTMjsbTIQmjOUE0so08LeStiIasrQhlOxIfjL6+S1kXN92r+/W1flPEUYTOIVz8OEK6nAHDWgCgwie4RXenLHz4rw7H4vWklPMHMfOJ8/noyNZw=</latexit><latexit sha1_base64="RlH2tdjr0xMRC8QkwnZnHd0h1Fs=">AB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8cq9gPaUDbTbt0swm7E6GE/gMvHhTx6j/y5r9x2+agrQ8GHu/NMDMvTKUw6HnfTmltfWNzq7xd2dnd2z9wD49aJsk0402WyER3Qmq4FIo3UaDknVRzGoeSt8Px7cxvP3FtRKIecZLyIKZDJSLBKFrpoTftu1Wv5s1BVolfkCoUaPTdr94gYVnMFTJjen6XopBTjUKJvm0sMTykb0yHvWqpozE2Qzy+dkjOrDEiUaFsKyVz9PZHT2JhJHNrOmOLILHsz8T+vm2F0HeRCpRlyxRaLokwSTMjsbTIQmjOUE0so08LeStiIasrQhlOxIfjL6+S1kXN92r+/W1flPEUYTOIVz8OEK6nAHDWgCgwie4RXenLHz4rw7H4vWklPMHMfOJ8/noyNZw=</latexit>
  • When the training set is small, don’t re-train word

embeddings (think of them as features!). Why?

  • Most cases: initialize word embeddings using pre-

trained ones (word2vec, Glove) and re-train them for the task

“good” vs “bad”

  • When you have enough data, you can just randomly

initialize them and train from scratch (e.g. machine translation)

slide-34
SLIDE 34

Neural Bag-of-Words (NBOW)

slide-35
SLIDE 35

Feedforward Neural LMs

  • N-gram models:

(Bengio et 2003): A Neural Probabilistic Language Model

P(mat|the cat sat on the)

  • Input layer (context size n = 5):

x = [ethe; ecat; esat; eon; ethe] ∈ Rdn

<latexit sha1_base64="aYKwScrslO2pOcQW2lnrXumNJ2I=">AChnicdVFNj9MwEHVSli1lFwocuVhUSOylSgqoK62QKrhwLIh+SE2oHfSWnWcyJ6sWln5Kfwpbvwb3DZI0JYnWXp+b8YznkKQwGwS/Pbzy4eHjZfNR6fHX95Gn72fOxyUvNYcRzmetpwgxIoWCEAiVMCw0sSyRMkvWnT+5B21Er7htoA4Y0slUsEZOmne/hFlDFdJajcV/UBnf25QzW2EsEGLK6iqO3pquBfOG+Z/Rq7O6/sSMY2Eqs3Efq2+24Wq5u1O0A32oKckrEmH1BjO2z+jRc7LDBRyYyZhUGBsWUaBZdQtaLSQMH4mi1h5qhiGZjY7sdY0dOWdA01+4opHv17wzLMmO2WeIid2aY28nvNmJa3sRWqKBEUPxRKS0kxp7ud0IXQwFuHWFcC9cr5SumGUe3uZYbQnj85VMy7nXDoBt+edcZfKzH0SQvySvyhoSkTwbkMxmSEeFew7vxet5bv+l3/fd+/xDqe3XOC/IP/MFvwdPJSg=</latexit><latexit sha1_base64="aYKwScrslO2pOcQW2lnrXumNJ2I=">AChnicdVFNj9MwEHVSli1lFwocuVhUSOylSgqoK62QKrhwLIh+SE2oHfSWnWcyJ6sWln5Kfwpbvwb3DZI0JYnWXp+b8YznkKQwGwS/Pbzy4eHjZfNR6fHX95Gn72fOxyUvNYcRzmetpwgxIoWCEAiVMCw0sSyRMkvWnT+5B21Er7htoA4Y0slUsEZOmne/hFlDFdJajcV/UBnf25QzW2EsEGLK6iqO3pquBfOG+Z/Rq7O6/sSMY2Eqs3Efq2+24Wq5u1O0A32oKckrEmH1BjO2z+jRc7LDBRyYyZhUGBsWUaBZdQtaLSQMH4mi1h5qhiGZjY7sdY0dOWdA01+4opHv17wzLMmO2WeIid2aY28nvNmJa3sRWqKBEUPxRKS0kxp7ud0IXQwFuHWFcC9cr5SumGUe3uZYbQnj85VMy7nXDoBt+edcZfKzH0SQvySvyhoSkTwbkMxmSEeFew7vxet5bv+l3/fd+/xDqe3XOC/IP/MFvwdPJSg=</latexit><latexit sha1_base64="aYKwScrslO2pOcQW2lnrXumNJ2I=">AChnicdVFNj9MwEHVSli1lFwocuVhUSOylSgqoK62QKrhwLIh+SE2oHfSWnWcyJ6sWln5Kfwpbvwb3DZI0JYnWXp+b8YznkKQwGwS/Pbzy4eHjZfNR6fHX95Gn72fOxyUvNYcRzmetpwgxIoWCEAiVMCw0sSyRMkvWnT+5B21Er7htoA4Y0slUsEZOmne/hFlDFdJajcV/UBnf25QzW2EsEGLK6iqO3pquBfOG+Z/Rq7O6/sSMY2Eqs3Efq2+24Wq5u1O0A32oKckrEmH1BjO2z+jRc7LDBRyYyZhUGBsWUaBZdQtaLSQMH4mi1h5qhiGZjY7sdY0dOWdA01+4opHv17wzLMmO2WeIid2aY28nvNmJa3sRWqKBEUPxRKS0kxp7ud0IXQwFuHWFcC9cr5SumGUe3uZYbQnj85VMy7nXDoBt+edcZfKzH0SQvySvyhoSkTwbkMxmSEeFew7vxet5bv+l3/fd+/xDqe3XOC/IP/MFvwdPJSg=</latexit><latexit sha1_base64="aYKwScrslO2pOcQW2lnrXumNJ2I=">AChnicdVFNj9MwEHVSli1lFwocuVhUSOylSgqoK62QKrhwLIh+SE2oHfSWnWcyJ6sWln5Kfwpbvwb3DZI0JYnWXp+b8YznkKQwGwS/Pbzy4eHjZfNR6fHX95Gn72fOxyUvNYcRzmetpwgxIoWCEAiVMCw0sSyRMkvWnT+5B21Er7htoA4Y0slUsEZOmne/hFlDFdJajcV/UBnf25QzW2EsEGLK6iqO3pquBfOG+Z/Rq7O6/sSMY2Eqs3Efq2+24Wq5u1O0A32oKckrEmH1BjO2z+jRc7LDBRyYyZhUGBsWUaBZdQtaLSQMH4mi1h5qhiGZjY7sdY0dOWdA01+4opHv17wzLMmO2WeIid2aY28nvNmJa3sRWqKBEUPxRKS0kxp7ud0IXQwFuHWFcC9cr5SumGUe3uZYbQnj85VMy7nXDoBt+edcZfKzH0SQvySvyhoSkTwbkMxmSEeFew7vxet5bv+l3/fd+/xDqe3XOC/IP/MFvwdPJSg=</latexit>

concatenation

  • Hidden layer

h = tanh(Wx + b) ∈ Rh

<latexit sha1_base64="m/ig9JANtchQHMKfk0FBlvyK3rE=">ACMnicbVDLSsNAFJ3UV62vqks3g0WoCURQTdC0Y3uqtgHNLFMpNm6GQSZiZiCfkmN36J4EIXirj1I5y2qWjrgYFz72Hufe4EaNSmeaLkZubX1hcyi8XVlbX1jeKm1sNGcYCkzoOWShaLpKEU7qipGWpEgKHAZabr982G/eUeEpCG/UYOIOAHqcepRjJSWOsVLO0DKd73ET+EptBXifnkiNVM4ofcpPgp3HQf2pRntZtcp7fa3imWzIo5ApwlVkZKIEOtU3yuyGOA8IVZkjKtmVGykmQUBQzkhbsWJI4T7qkbamHAVEOsno5BTuaULvVDoxUcqb8dCQqkHASunhxuKad7Q/G/XjtW3omTUB7FinA8/siLGVQhHOYHu1QrNhAE4QF1btC7COBsNIpF3QI1vTJs6RxWLHMinV1VKqeZXHkwQ7YBWVgWNQBRegBuoAgwfwDN7Au/FovBofxud4NGdknm3wB8bXN2O6qts=</latexit><latexit sha1_base64="m/ig9JANtchQHMKfk0FBlvyK3rE=">ACMnicbVDLSsNAFJ3UV62vqks3g0WoCURQTdC0Y3uqtgHNLFMpNm6GQSZiZiCfkmN36J4EIXirj1I5y2qWjrgYFz72Hufe4EaNSmeaLkZubX1hcyi8XVlbX1jeKm1sNGcYCkzoOWShaLpKEU7qipGWpEgKHAZabr982G/eUeEpCG/UYOIOAHqcepRjJSWOsVLO0DKd73ET+EptBXifnkiNVM4ofcpPgp3HQf2pRntZtcp7fa3imWzIo5ApwlVkZKIEOtU3yuyGOA8IVZkjKtmVGykmQUBQzkhbsWJI4T7qkbamHAVEOsno5BTuaULvVDoxUcqb8dCQqkHASunhxuKad7Q/G/XjtW3omTUB7FinA8/siLGVQhHOYHu1QrNhAE4QF1btC7COBsNIpF3QI1vTJs6RxWLHMinV1VKqeZXHkwQ7YBWVgWNQBRegBuoAgwfwDN7Au/FovBofxud4NGdknm3wB8bXN2O6qts=</latexit><latexit sha1_base64="m/ig9JANtchQHMKfk0FBlvyK3rE=">ACMnicbVDLSsNAFJ3UV62vqks3g0WoCURQTdC0Y3uqtgHNLFMpNm6GQSZiZiCfkmN36J4EIXirj1I5y2qWjrgYFz72Hufe4EaNSmeaLkZubX1hcyi8XVlbX1jeKm1sNGcYCkzoOWShaLpKEU7qipGWpEgKHAZabr982G/eUeEpCG/UYOIOAHqcepRjJSWOsVLO0DKd73ET+EptBXifnkiNVM4ofcpPgp3HQf2pRntZtcp7fa3imWzIo5ApwlVkZKIEOtU3yuyGOA8IVZkjKtmVGykmQUBQzkhbsWJI4T7qkbamHAVEOsno5BTuaULvVDoxUcqb8dCQqkHASunhxuKad7Q/G/XjtW3omTUB7FinA8/siLGVQhHOYHu1QrNhAE4QF1btC7COBsNIpF3QI1vTJs6RxWLHMinV1VKqeZXHkwQ7YBWVgWNQBRegBuoAgwfwDN7Au/FovBofxud4NGdknm3wB8bXN2O6qts=</latexit><latexit sha1_base64="m/ig9JANtchQHMKfk0FBlvyK3rE=">ACMnicbVDLSsNAFJ3UV62vqks3g0WoCURQTdC0Y3uqtgHNLFMpNm6GQSZiZiCfkmN36J4EIXirj1I5y2qWjrgYFz72Hufe4EaNSmeaLkZubX1hcyi8XVlbX1jeKm1sNGcYCkzoOWShaLpKEU7qipGWpEgKHAZabr982G/eUeEpCG/UYOIOAHqcepRjJSWOsVLO0DKd73ET+EptBXifnkiNVM4ofcpPgp3HQf2pRntZtcp7fa3imWzIo5ApwlVkZKIEOtU3yuyGOA8IVZkjKtmVGykmQUBQzkhbsWJI4T7qkbamHAVEOsno5BTuaULvVDoxUcqb8dCQqkHASunhxuKad7Q/G/XjtW3omTUB7FinA8/siLGVQhHOYHu1QrNhAE4QF1btC7COBsNIpF3QI1vTJs6RxWLHMinV1VKqeZXHkwQ7YBWVgWNQBRegBuoAgwfwDN7Au/FovBofxud4NGdknm3wB8bXN2O6qts=</latexit>
  • Output layer (softmax)

z = Uh ∈ R|V |

<latexit sha1_base64="NbkraP76q6P/C18JkvOTVHd/w=">ACH3icbVDLSsNAFJ3UV62vqEs3g0VwVRIRdSMU3bisYtpCE8tkOmHTiZhZiLUNH/ixl9x40IRcde/cdqmoK0HBs6cy/3uPHjEplWSOjsLS8srpWXC9tbG5t75i7e3UZJQITB0csEk0fScIoJ46ipFmLAgKfUYafv967DceiZA04vdqEBMvRF1OA4qR0lLbPHNDpHp+kD5l8BLOPk42Y70MupTnhp/eZQ/psD7M2mbZqlgTwEVi56QMctTa5rfbiXASEq4wQ1K2bCtWXoqEopiRrOQmksQI91GXtDTlKCTSyf3ZfBIKx0YREI/ruBE/d2RolDKQejryvGect4bi/95rUQF15KeZwowvF0UJAwqCI4Dgt2qCBYsYEmCAuqd4W4hwTCSkda0iHY8ycvkvpJxbYq9u1puXqVx1EB+AQHAMbnIMquAE14AMnsEreAcfxovxZnwaX9PSgpH37IM/MEY/o0mj+A=</latexit><latexit sha1_base64="NbkraP76q6P/C18JkvOTVHd/w=">ACH3icbVDLSsNAFJ3UV62vqEs3g0VwVRIRdSMU3bisYtpCE8tkOmHTiZhZiLUNH/ixl9x40IRcde/cdqmoK0HBs6cy/3uPHjEplWSOjsLS8srpWXC9tbG5t75i7e3UZJQITB0csEk0fScIoJ46ipFmLAgKfUYafv967DceiZA04vdqEBMvRF1OA4qR0lLbPHNDpHp+kD5l8BLOPk42Y70MupTnhp/eZQ/psD7M2mbZqlgTwEVi56QMctTa5rfbiXASEq4wQ1K2bCtWXoqEopiRrOQmksQI91GXtDTlKCTSyf3ZfBIKx0YREI/ruBE/d2RolDKQejryvGect4bi/95rUQF15KeZwowvF0UJAwqCI4Dgt2qCBYsYEmCAuqd4W4hwTCSkda0iHY8ycvkvpJxbYq9u1puXqVx1EB+AQHAMbnIMquAE14AMnsEreAcfxovxZnwaX9PSgpH37IM/MEY/o0mj+A=</latexit><latexit sha1_base64="NbkraP76q6P/C18JkvOTVHd/w=">ACH3icbVDLSsNAFJ3UV62vqEs3g0VwVRIRdSMU3bisYtpCE8tkOmHTiZhZiLUNH/ixl9x40IRcde/cdqmoK0HBs6cy/3uPHjEplWSOjsLS8srpWXC9tbG5t75i7e3UZJQITB0csEk0fScIoJ46ipFmLAgKfUYafv967DceiZA04vdqEBMvRF1OA4qR0lLbPHNDpHp+kD5l8BLOPk42Y70MupTnhp/eZQ/psD7M2mbZqlgTwEVi56QMctTa5rfbiXASEq4wQ1K2bCtWXoqEopiRrOQmksQI91GXtDTlKCTSyf3ZfBIKx0YREI/ruBE/d2RolDKQejryvGect4bi/95rUQF15KeZwowvF0UJAwqCI4Dgt2qCBYsYEmCAuqd4W4hwTCSkda0iHY8ycvkvpJxbYq9u1puXqVx1EB+AQHAMbnIMquAE14AMnsEreAcfxovxZnwaX9PSgpH37IM/MEY/o0mj+A=</latexit><latexit sha1_base64="NbkraP76q6P/C18JkvOTVHd/w=">ACH3icbVDLSsNAFJ3UV62vqEs3g0VwVRIRdSMU3bisYtpCE8tkOmHTiZhZiLUNH/ixl9x40IRcde/cdqmoK0HBs6cy/3uPHjEplWSOjsLS8srpWXC9tbG5t75i7e3UZJQITB0csEk0fScIoJ46ipFmLAgKfUYafv967DceiZA04vdqEBMvRF1OA4qR0lLbPHNDpHp+kD5l8BLOPk42Y70MupTnhp/eZQ/psD7M2mbZqlgTwEVi56QMctTa5rfbiXASEq4wQ1K2bCtWXoqEopiRrOQmksQI91GXtDTlKCTSyf3ZfBIKx0YREI/ruBE/d2RolDKQejryvGect4bi/95rUQF15KeZwowvF0UJAwqCI4Dgt2qCBYsYEmCAuqd4W4hwTCSkda0iHY8ycvkvpJxbYq9u1puXqVx1EB+AQHAMbnIMquAE14AMnsEreAcfxovxZnwaX9PSgpH37IM/MEY/o0mj+A=</latexit>

P(w = i | context) = softmaxi(z)

<latexit sha1_base64="PfRVpY8ZLMPvgyPpDypIa8e+c9w=">ACHXicbVC7SgNBFJ31GeMramkzGITYhF0JaCMEbSwjmETIhjA7mdXBeSwzdzVxyY/Y+Cs2FopY2Ih/42ySwteBYQ7n3Mu90SJ4BZ8/9ObmZ2bX1gsLBWXV1bX1ksbmy2rU0NZk2qhzUVELBNcsSZwEOwiMYzISLB2dH2S+0bZizX6hyGCetKcql4zCkBJ/VKtUblFh9hjkPJ+zgENoCMapX/oz1nWB2DJIMer4SwFUZ3ejvV6p7Ff9MfBfEkxJGU3R6JXew76mqWQKqCDWdgI/gW5GDHAq2KgYpYlhF6TS9ZxVBHJbDcbXzfCu07p41gb9xTgsfq9IyPS2qGMXGW+ov3t5eJ/XieF+LCbcZWkwBSdDIpTgUHjPCrc54ZRENHCDXc7YrpFTGEgu06EIfp/8l7T2q4FfDc5q5frxNI4C2kY7qICdIDq6BQ1UBNRdI8e0TN68R68J+/Ve5uUznjTni30A97HF1PioX0=</latexit><latexit sha1_base64="PfRVpY8ZLMPvgyPpDypIa8e+c9w=">ACHXicbVC7SgNBFJ31GeMramkzGITYhF0JaCMEbSwjmETIhjA7mdXBeSwzdzVxyY/Y+Cs2FopY2Ih/42ySwteBYQ7n3Mu90SJ4BZ8/9ObmZ2bX1gsLBWXV1bX1ksbmy2rU0NZk2qhzUVELBNcsSZwEOwiMYzISLB2dH2S+0bZizX6hyGCetKcql4zCkBJ/VKtUblFh9hjkPJ+zgENoCMapX/oz1nWB2DJIMer4SwFUZ3ejvV6p7Ff9MfBfEkxJGU3R6JXew76mqWQKqCDWdgI/gW5GDHAq2KgYpYlhF6TS9ZxVBHJbDcbXzfCu07p41gb9xTgsfq9IyPS2qGMXGW+ov3t5eJ/XieF+LCbcZWkwBSdDIpTgUHjPCrc54ZRENHCDXc7YrpFTGEgu06EIfp/8l7T2q4FfDc5q5frxNI4C2kY7qICdIDq6BQ1UBNRdI8e0TN68R68J+/Ve5uUznjTni30A97HF1PioX0=</latexit><latexit sha1_base64="PfRVpY8ZLMPvgyPpDypIa8e+c9w=">ACHXicbVC7SgNBFJ31GeMramkzGITYhF0JaCMEbSwjmETIhjA7mdXBeSwzdzVxyY/Y+Cs2FopY2Ih/42ySwteBYQ7n3Mu90SJ4BZ8/9ObmZ2bX1gsLBWXV1bX1ksbmy2rU0NZk2qhzUVELBNcsSZwEOwiMYzISLB2dH2S+0bZizX6hyGCetKcql4zCkBJ/VKtUblFh9hjkPJ+zgENoCMapX/oz1nWB2DJIMer4SwFUZ3ejvV6p7Ff9MfBfEkxJGU3R6JXew76mqWQKqCDWdgI/gW5GDHAq2KgYpYlhF6TS9ZxVBHJbDcbXzfCu07p41gb9xTgsfq9IyPS2qGMXGW+ov3t5eJ/XieF+LCbcZWkwBSdDIpTgUHjPCrc54ZRENHCDXc7YrpFTGEgu06EIfp/8l7T2q4FfDc5q5frxNI4C2kY7qICdIDq6BQ1UBNRdI8e0TN68R68J+/Ve5uUznjTni30A97HF1PioX0=</latexit><latexit sha1_base64="PfRVpY8ZLMPvgyPpDypIa8e+c9w=">ACHXicbVC7SgNBFJ31GeMramkzGITYhF0JaCMEbSwjmETIhjA7mdXBeSwzdzVxyY/Y+Cs2FopY2Ih/42ySwteBYQ7n3Mu90SJ4BZ8/9ObmZ2bX1gsLBWXV1bX1ksbmy2rU0NZk2qhzUVELBNcsSZwEOwiMYzISLB2dH2S+0bZizX6hyGCetKcql4zCkBJ/VKtUblFh9hjkPJ+zgENoCMapX/oz1nWB2DJIMer4SwFUZ3ejvV6p7Ff9MfBfEkxJGU3R6JXew76mqWQKqCDWdgI/gW5GDHAq2KgYpYlhF6TS9ZxVBHJbDcbXzfCu07p41gb9xTgsfq9IyPS2qGMXGW+ov3t5eJ/XieF+LCbcZWkwBSdDIpTgUHjPCrc54ZRENHCDXc7YrpFTGEgu06EIfp/8l7T2q4FfDc5q5frxNI4C2kY7qICdIDq6BQ1UBNRdI8e0TN68R68J+/Ve5uUznjTni30A97HF1PioX0=</latexit>
slide-36
SLIDE 36

Backpropagation

How to compute gradients?

slide-37
SLIDE 37

Backpropagation

  • It’s taking derivatives and applying chain rule!
  • We’ll re-use derivatives computed for

higher layers in computing derivatives for lower layers so as to minimize computation

  • Good news is that modern automatic

differentiation tools did all for you!

  • Implementing backprop by hand is like

programming in assembly language.

slide-38
SLIDE 38

Deriving gradients for Feedforward NNs

Input: x

<latexit sha1_base64="RVNPG5gvi6bcJf6jbsL3i7LAtbE=">AB8XicbVBNS8NAFHypX7V+VT16WSyCp5KIoAcPBS8eK9hWbEPZbDft0s0m7L6IJfRfePGgiFf/jTf/jZs2B20dWBhm3mPnTZBIYdB1v53Syura+kZ5s7K1vbO7V90/aJs41Yy3WCxjfR9Qw6VQvIUCJb9PNKdRIHknGF/nfueRayNidYeThPsRHSoRCkbRSg+9iOIoCLOnab9ac+vuDGSZeAWpQYFmv/rVG8QsjbhCJqkxXc9N0M+oRsEkn1Z6qeEJZWM65F1LFY248bNZ4ik5scqAhLG2TyGZqb83MhoZM4kCO5knNIteLv7ndVML/1MqCRFrtj8ozCVBGOSn08GQnOGcmIJZVrYrISNqKYMbUkVW4K3ePIyaZ/VPbfu3Z7XGldFHWU4gmM4BQ8uoAE30IQWMFDwDK/w5hjnxXl3PuajJafYOYQ/cD5/APtrkRY=</latexit><latexit sha1_base64="RVNPG5gvi6bcJf6jbsL3i7LAtbE=">AB8XicbVBNS8NAFHypX7V+VT16WSyCp5KIoAcPBS8eK9hWbEPZbDft0s0m7L6IJfRfePGgiFf/jTf/jZs2B20dWBhm3mPnTZBIYdB1v53Syura+kZ5s7K1vbO7V90/aJs41Yy3WCxjfR9Qw6VQvIUCJb9PNKdRIHknGF/nfueRayNidYeThPsRHSoRCkbRSg+9iOIoCLOnab9ac+vuDGSZeAWpQYFmv/rVG8QsjbhCJqkxXc9N0M+oRsEkn1Z6qeEJZWM65F1LFY248bNZ4ik5scqAhLG2TyGZqb83MhoZM4kCO5knNIteLv7ndVML/1MqCRFrtj8ozCVBGOSn08GQnOGcmIJZVrYrISNqKYMbUkVW4K3ePIyaZ/VPbfu3Z7XGldFHWU4gmM4BQ8uoAE30IQWMFDwDK/w5hjnxXl3PuajJafYOYQ/cD5/APtrkRY=</latexit><latexit sha1_base64="RVNPG5gvi6bcJf6jbsL3i7LAtbE=">AB8XicbVBNS8NAFHypX7V+VT16WSyCp5KIoAcPBS8eK9hWbEPZbDft0s0m7L6IJfRfePGgiFf/jTf/jZs2B20dWBhm3mPnTZBIYdB1v53Syura+kZ5s7K1vbO7V90/aJs41Yy3WCxjfR9Qw6VQvIUCJb9PNKdRIHknGF/nfueRayNidYeThPsRHSoRCkbRSg+9iOIoCLOnab9ac+vuDGSZeAWpQYFmv/rVG8QsjbhCJqkxXc9N0M+oRsEkn1Z6qeEJZWM65F1LFY248bNZ4ik5scqAhLG2TyGZqb83MhoZM4kCO5knNIteLv7ndVML/1MqCRFrtj8ozCVBGOSn08GQnOGcmIJZVrYrISNqKYMbUkVW4K3ePIyaZ/VPbfu3Z7XGldFHWU4gmM4BQ8uoAE30IQWMFDwDK/w5hjnxXl3PuajJafYOYQ/cD5/APtrkRY=</latexit><latexit sha1_base64="RVNPG5gvi6bcJf6jbsL3i7LAtbE=">AB8XicbVBNS8NAFHypX7V+VT16WSyCp5KIoAcPBS8eK9hWbEPZbDft0s0m7L6IJfRfePGgiFf/jTf/jZs2B20dWBhm3mPnTZBIYdB1v53Syura+kZ5s7K1vbO7V90/aJs41Yy3WCxjfR9Qw6VQvIUCJb9PNKdRIHknGF/nfueRayNidYeThPsRHSoRCkbRSg+9iOIoCLOnab9ac+vuDGSZeAWpQYFmv/rVG8QsjbhCJqkxXc9N0M+oRsEkn1Z6qeEJZWM65F1LFY248bNZ4ik5scqAhLG2TyGZqb83MhoZM4kCO5knNIteLv7ndVML/1MqCRFrtj8ozCVBGOSn08GQnOGcmIJZVrYrISNqKYMbUkVW4K3ePIyaZ/VPbfu3Z7XGldFHWU4gmM4BQ8uoAE30IQWMFDwDK/w5hjnxXl3PuajJafYOYQ/cD5/APtrkRY=</latexit>

h2 = tanh(W2h1 + b2)

<latexit sha1_base64="6qo1Sk9pMYMRHaF5H3jONO0Lxc=">ACJ3icbVDLSsNAFJ3UV62vqEs3g0WoCUpgi5UCm5cVrAPaEKYTCft0MkzEyEvo3bvwVN4K6NI/cdJGqa0HBs6cy/3uPHjEplWZ9GYWl5ZXWtuF7a2Nza3jF391oySgQmTRyxSHR8JAmjnDQVYx0YkFQ6DPS9ofXmd+J0LSiN+pUzcEPU5DShGSkueWESA38IB2MvRq8hI5CfFD5EduZOFNhw5Pfr6+9Y8sW1VrArhI7JyUQY6GZ74vQgnIeEKMyRl17Zi5aZIKIoZGZecRJIY4SHqk6mHIVEunkzjE80koPBpHQjys4UWc7UhRKOQp9XZktKe9TPzP6yYqOHdTyuNEY6ng4KEQRXBLDTYo4JgxUaICyo3hXiARIKx1tSYdgz5+8SFq1qm1V7dvTcv0ij6MIDsAhqAbnIE6uAEN0AQYPIAn8ArejEfj2Xg3PqalBSPv2Qd/YHx9A/dopVU=</latexit><latexit sha1_base64="6qo1Sk9pMYMRHaF5H3jONO0Lxc=">ACJ3icbVDLSsNAFJ3UV62vqEs3g0WoCUpgi5UCm5cVrAPaEKYTCft0MkzEyEvo3bvwVN4K6NI/cdJGqa0HBs6cy/3uPHjEplWZ9GYWl5ZXWtuF7a2Nza3jF391oySgQmTRyxSHR8JAmjnDQVYx0YkFQ6DPS9ofXmd+J0LSiN+pUzcEPU5DShGSkueWESA38IB2MvRq8hI5CfFD5EduZOFNhw5Pfr6+9Y8sW1VrArhI7JyUQY6GZ74vQgnIeEKMyRl17Zi5aZIKIoZGZecRJIY4SHqk6mHIVEunkzjE80koPBpHQjys4UWc7UhRKOQp9XZktKe9TPzP6yYqOHdTyuNEY6ng4KEQRXBLDTYo4JgxUaICyo3hXiARIKx1tSYdgz5+8SFq1qm1V7dvTcv0ij6MIDsAhqAbnIE6uAEN0AQYPIAn8ArejEfj2Xg3PqalBSPv2Qd/YHx9A/dopVU=</latexit><latexit sha1_base64="6qo1Sk9pMYMRHaF5H3jONO0Lxc=">ACJ3icbVDLSsNAFJ3UV62vqEs3g0WoCUpgi5UCm5cVrAPaEKYTCft0MkzEyEvo3bvwVN4K6NI/cdJGqa0HBs6cy/3uPHjEplWZ9GYWl5ZXWtuF7a2Nza3jF391oySgQmTRyxSHR8JAmjnDQVYx0YkFQ6DPS9ofXmd+J0LSiN+pUzcEPU5DShGSkueWESA38IB2MvRq8hI5CfFD5EduZOFNhw5Pfr6+9Y8sW1VrArhI7JyUQY6GZ74vQgnIeEKMyRl17Zi5aZIKIoZGZecRJIY4SHqk6mHIVEunkzjE80koPBpHQjys4UWc7UhRKOQp9XZktKe9TPzP6yYqOHdTyuNEY6ng4KEQRXBLDTYo4JgxUaICyo3hXiARIKx1tSYdgz5+8SFq1qm1V7dvTcv0ij6MIDsAhqAbnIE6uAEN0AQYPIAn8ArejEfj2Xg3PqalBSPv2Qd/YHx9A/dopVU=</latexit><latexit sha1_base64="6qo1Sk9pMYMRHaF5H3jONO0Lxc=">ACJ3icbVDLSsNAFJ3UV62vqEs3g0WoCUpgi5UCm5cVrAPaEKYTCft0MkzEyEvo3bvwVN4K6NI/cdJGqa0HBs6cy/3uPHjEplWZ9GYWl5ZXWtuF7a2Nza3jF391oySgQmTRyxSHR8JAmjnDQVYx0YkFQ6DPS9ofXmd+J0LSiN+pUzcEPU5DShGSkueWESA38IB2MvRq8hI5CfFD5EduZOFNhw5Pfr6+9Y8sW1VrArhI7JyUQY6GZ74vQgnIeEKMyRl17Zi5aZIKIoZGZecRJIY4SHqk6mHIVEunkzjE80koPBpHQjys4UWc7UhRKOQp9XZktKe9TPzP6yYqOHdTyuNEY6ng4KEQRXBLDTYo4JgxUaICyo3hXiARIKx1tSYdgz5+8SFq1qm1V7dvTcv0ij6MIDsAhqAbnIE6uAEN0AQYPIAn8ArejEfj2Xg3PqalBSPv2Qd/YHx9A/dopVU=</latexit>

h1 = tanh(W1x + b1)

<latexit sha1_base64="tJT8ci0MBQZl4hoeA/0XzNX9tv4=">ACJXicbVDLSsNAFJ3UV62vqEs3g0WoCURQRcVCm5cVrAPaEOYTCft0MkzEzEvIzbvwVNy4sIrjyV5y0qWjrgYFz7mXufd4EaNSWdanUVhZXVvfKG6WtrZ3dvfM/YOWDGOBSROHLBQdD0nCKCdNRUjnUgQFHiMtL3RTea3H4iQNOT3ahwRJ0ADTn2KkdKSa9Z6AVJDz0+GqWvDa9hTiA8rc7GdifPiMYVnP4WnVPXLFtVawq4TOyclEGOhmtOev0QxwHhCjMkZde2IuUkSCiKGUlLvViSCOERGpCuphwFRDrJ9MoUnmilD/1Q6McVnKq/JxIUSDkOPN2ZLSkXvUz8z+vGyr9yEsqjWBGOZx/5MYMqhFlksE8FwYqNUFYUL0rxEMkEFY62JIOwV48eZm0zqu2VbXvLsr1Wh5HERyBY1ABNrgEdXALGqAJMHgCL+ANTIxn49V4Nz5mrQUjnzkEf2B8fQO8cKS+</latexit><latexit sha1_base64="tJT8ci0MBQZl4hoeA/0XzNX9tv4=">ACJXicbVDLSsNAFJ3UV62vqEs3g0WoCURQRcVCm5cVrAPaEOYTCft0MkzEzEvIzbvwVNy4sIrjyV5y0qWjrgYFz7mXufd4EaNSWdanUVhZXVvfKG6WtrZ3dvfM/YOWDGOBSROHLBQdD0nCKCdNRUjnUgQFHiMtL3RTea3H4iQNOT3ahwRJ0ADTn2KkdKSa9Z6AVJDz0+GqWvDa9hTiA8rc7GdifPiMYVnP4WnVPXLFtVawq4TOyclEGOhmtOev0QxwHhCjMkZde2IuUkSCiKGUlLvViSCOERGpCuphwFRDrJ9MoUnmilD/1Q6McVnKq/JxIUSDkOPN2ZLSkXvUz8z+vGyr9yEsqjWBGOZx/5MYMqhFlksE8FwYqNUFYUL0rxEMkEFY62JIOwV48eZm0zqu2VbXvLsr1Wh5HERyBY1ABNrgEdXALGqAJMHgCL+ANTIxn49V4Nz5mrQUjnzkEf2B8fQO8cKS+</latexit><latexit sha1_base64="tJT8ci0MBQZl4hoeA/0XzNX9tv4=">ACJXicbVDLSsNAFJ3UV62vqEs3g0WoCURQRcVCm5cVrAPaEOYTCft0MkzEzEvIzbvwVNy4sIrjyV5y0qWjrgYFz7mXufd4EaNSWdanUVhZXVvfKG6WtrZ3dvfM/YOWDGOBSROHLBQdD0nCKCdNRUjnUgQFHiMtL3RTea3H4iQNOT3ahwRJ0ADTn2KkdKSa9Z6AVJDz0+GqWvDa9hTiA8rc7GdifPiMYVnP4WnVPXLFtVawq4TOyclEGOhmtOev0QxwHhCjMkZde2IuUkSCiKGUlLvViSCOERGpCuphwFRDrJ9MoUnmilD/1Q6McVnKq/JxIUSDkOPN2ZLSkXvUz8z+vGyr9yEsqjWBGOZx/5MYMqhFlksE8FwYqNUFYUL0rxEMkEFY62JIOwV48eZm0zqu2VbXvLsr1Wh5HERyBY1ABNrgEdXALGqAJMHgCL+ANTIxn49V4Nz5mrQUjnzkEf2B8fQO8cKS+</latexit><latexit sha1_base64="tJT8ci0MBQZl4hoeA/0XzNX9tv4=">ACJXicbVDLSsNAFJ3UV62vqEs3g0WoCURQRcVCm5cVrAPaEOYTCft0MkzEzEvIzbvwVNy4sIrjyV5y0qWjrgYFz7mXufd4EaNSWdanUVhZXVvfKG6WtrZ3dvfM/YOWDGOBSROHLBQdD0nCKCdNRUjnUgQFHiMtL3RTea3H4iQNOT3ahwRJ0ADTn2KkdKSa9Z6AVJDz0+GqWvDa9hTiA8rc7GdifPiMYVnP4WnVPXLFtVawq4TOyclEGOhmtOev0QxwHhCjMkZde2IuUkSCiKGUlLvViSCOERGpCuphwFRDrJ9MoUnmilD/1Q6McVnKq/JxIUSDkOPN2ZLSkXvUz8z+vGyr9yEsqjWBGOZx/5MYMqhFlksE8FwYqNUFYUL0rxEMkEFY62JIOwV48eZm0zqu2VbXvLsr1Wh5HERyBY1ABNrgEdXALGqAJMHgCL+ANTIxn49V4Nz5mrQUjnzkEf2B8fQO8cKS+</latexit>

x ∈ Rd

<latexit sha1_base64="r8cLpKiZJtXS/wgD8iCVci7o1w=">ACBHicbVC7TsMwFL3hWcorwNjFokJiqhKEBANDJRbGguhDakrlOE5r1XEi20FUQYWfoWFAYRY+Qg2/gan7QAtR7J0fM69uvceP+FMacf5tpaWV1bX1ksb5c2t7Z1de2+/peJUEtokMY9lx8eKciZoUzPNaSeRFEc+p21/dFn47XsqFYvFrR4ntBfhgWAhI1gbqW9XvAjroR9mDznymEDTr5/d5HdB3646NWcCtEjcGanCDI2+/eUFMUkjKjThWKmu6yS6l2GpGeE0L3upogkmIzygXUMFjqjqZMjcnRklACFsTRPaDRf3dkOFJqHPmsthRzXuF+J/XTXV43suYSFJNBZkOClOdIyKRFDAJCWajw3BRDKzKyJDLDHRJreyCcGdP3mRtE5qrlNzr0+r9YtZHCWowCEcgwtnUIcraEATCDzCM7zCm/VkvVjv1se0dMma9RzAH1ifP+g+mDo=</latexit><latexit sha1_base64="r8cLpKiZJtXS/wgD8iCVci7o1w=">ACBHicbVC7TsMwFL3hWcorwNjFokJiqhKEBANDJRbGguhDakrlOE5r1XEi20FUQYWfoWFAYRY+Qg2/gan7QAtR7J0fM69uvceP+FMacf5tpaWV1bX1ksb5c2t7Z1de2+/peJUEtokMY9lx8eKciZoUzPNaSeRFEc+p21/dFn47XsqFYvFrR4ntBfhgWAhI1gbqW9XvAjroR9mDznymEDTr5/d5HdB3646NWcCtEjcGanCDI2+/eUFMUkjKjThWKmu6yS6l2GpGeE0L3upogkmIzygXUMFjqjqZMjcnRklACFsTRPaDRf3dkOFJqHPmsthRzXuF+J/XTXV43suYSFJNBZkOClOdIyKRFDAJCWajw3BRDKzKyJDLDHRJreyCcGdP3mRtE5qrlNzr0+r9YtZHCWowCEcgwtnUIcraEATCDzCM7zCm/VkvVjv1se0dMma9RzAH1ifP+g+mDo=</latexit><latexit sha1_base64="r8cLpKiZJtXS/wgD8iCVci7o1w=">ACBHicbVC7TsMwFL3hWcorwNjFokJiqhKEBANDJRbGguhDakrlOE5r1XEi20FUQYWfoWFAYRY+Qg2/gan7QAtR7J0fM69uvceP+FMacf5tpaWV1bX1ksb5c2t7Z1de2+/peJUEtokMY9lx8eKciZoUzPNaSeRFEc+p21/dFn47XsqFYvFrR4ntBfhgWAhI1gbqW9XvAjroR9mDznymEDTr5/d5HdB3646NWcCtEjcGanCDI2+/eUFMUkjKjThWKmu6yS6l2GpGeE0L3upogkmIzygXUMFjqjqZMjcnRklACFsTRPaDRf3dkOFJqHPmsthRzXuF+J/XTXV43suYSFJNBZkOClOdIyKRFDAJCWajw3BRDKzKyJDLDHRJreyCcGdP3mRtE5qrlNzr0+r9YtZHCWowCEcgwtnUIcraEATCDzCM7zCm/VkvVjv1se0dMma9RzAH1ifP+g+mDo=</latexit><latexit sha1_base64="r8cLpKiZJtXS/wgD8iCVci7o1w=">ACBHicbVC7TsMwFL3hWcorwNjFokJiqhKEBANDJRbGguhDakrlOE5r1XEi20FUQYWfoWFAYRY+Qg2/gan7QAtR7J0fM69uvceP+FMacf5tpaWV1bX1ksb5c2t7Z1de2+/peJUEtokMY9lx8eKciZoUzPNaSeRFEc+p21/dFn47XsqFYvFrR4ntBfhgWAhI1gbqW9XvAjroR9mDznymEDTr5/d5HdB3646NWcCtEjcGanCDI2+/eUFMUkjKjThWKmu6yS6l2GpGeE0L3upogkmIzygXUMFjqjqZMjcnRklACFsTRPaDRf3dkOFJqHPmsthRzXuF+J/XTXV43suYSFJNBZkOClOdIyKRFDAJCWajw3BRDKzKyJDLDHRJreyCcGdP3mRtE5qrlNzr0+r9YtZHCWowCEcgwtnUIcraEATCDzCM7zCm/VkvVjv1se0dMma9RzAH1ifP+g+mDo=</latexit>

W1 ∈ Rd1×d

<latexit sha1_base64="2wvpmjhpxYNVstsZLu8X4dfHiOc=">ACE3icbVBNS8NAEJ3Ur1q/qh69LBZBPJREBD14KHjxWMV+QFPDZrNpl242YXcjlJD/4MW/4sWDIl69ePfuGl70NYHA4/3ZpiZ5yecKW3b31ZpaXlda28XtnY3Nreqe7utVWcSkJbJOax7PpYUc4EbWmOe0mkuLI57Tj64Kv/NApWKxuNPjhPYjPBAsZARrI3nVEzfCeuiHWSf3HOQygaCn93m91lQaJpFVKEg96o1u25PgBaJMyM1mKHpVb/cICZpRIUmHCvVc+xE9zMsNSOc5hU3VTBZIQHtGeowGZP5v8lKMjowQojKUpodFE/T2R4UipceSbzuJgNe8V4n9eL9XhRT9jIk1FWS6KEw50jEqAkIBk5RoPjYE8nMrYgMscREmxgrJgRn/uVF0j6tO3bduTmrNS5ncZThA7hGBw4hwZcQxNaQOARnuEV3qwn68V6tz6mrSVrNrMPf2B9/gCLVZ3p</latexit><latexit sha1_base64="2wvpmjhpxYNVstsZLu8X4dfHiOc=">ACE3icbVBNS8NAEJ3Ur1q/qh69LBZBPJREBD14KHjxWMV+QFPDZrNpl242YXcjlJD/4MW/4sWDIl69ePfuGl70NYHA4/3ZpiZ5yecKW3b31ZpaXlda28XtnY3Nreqe7utVWcSkJbJOax7PpYUc4EbWmOe0mkuLI57Tj64Kv/NApWKxuNPjhPYjPBAsZARrI3nVEzfCeuiHWSf3HOQygaCn93m91lQaJpFVKEg96o1u25PgBaJMyM1mKHpVb/cICZpRIUmHCvVc+xE9zMsNSOc5hU3VTBZIQHtGeowGZP5v8lKMjowQojKUpodFE/T2R4UipceSbzuJgNe8V4n9eL9XhRT9jIk1FWS6KEw50jEqAkIBk5RoPjYE8nMrYgMscREmxgrJgRn/uVF0j6tO3bduTmrNS5ncZThA7hGBw4hwZcQxNaQOARnuEV3qwn68V6tz6mrSVrNrMPf2B9/gCLVZ3p</latexit><latexit sha1_base64="2wvpmjhpxYNVstsZLu8X4dfHiOc=">ACE3icbVBNS8NAEJ3Ur1q/qh69LBZBPJREBD14KHjxWMV+QFPDZrNpl242YXcjlJD/4MW/4sWDIl69ePfuGl70NYHA4/3ZpiZ5yecKW3b31ZpaXlda28XtnY3Nreqe7utVWcSkJbJOax7PpYUc4EbWmOe0mkuLI57Tj64Kv/NApWKxuNPjhPYjPBAsZARrI3nVEzfCeuiHWSf3HOQygaCn93m91lQaJpFVKEg96o1u25PgBaJMyM1mKHpVb/cICZpRIUmHCvVc+xE9zMsNSOc5hU3VTBZIQHtGeowGZP5v8lKMjowQojKUpodFE/T2R4UipceSbzuJgNe8V4n9eL9XhRT9jIk1FWS6KEw50jEqAkIBk5RoPjYE8nMrYgMscREmxgrJgRn/uVF0j6tO3bduTmrNS5ncZThA7hGBw4hwZcQxNaQOARnuEV3qwn68V6tz6mrSVrNrMPf2B9/gCLVZ3p</latexit><latexit sha1_base64="2wvpmjhpxYNVstsZLu8X4dfHiOc=">ACE3icbVBNS8NAEJ3Ur1q/qh69LBZBPJREBD14KHjxWMV+QFPDZrNpl242YXcjlJD/4MW/4sWDIl69ePfuGl70NYHA4/3ZpiZ5yecKW3b31ZpaXlda28XtnY3Nreqe7utVWcSkJbJOax7PpYUc4EbWmOe0mkuLI57Tj64Kv/NApWKxuNPjhPYjPBAsZARrI3nVEzfCeuiHWSf3HOQygaCn93m91lQaJpFVKEg96o1u25PgBaJMyM1mKHpVb/cICZpRIUmHCvVc+xE9zMsNSOc5hU3VTBZIQHtGeowGZP5v8lKMjowQojKUpodFE/T2R4UipceSbzuJgNe8V4n9eL9XhRT9jIk1FWS6KEw50jEqAkIBk5RoPjYE8nMrYgMscREmxgrJgRn/uVF0j6tO3bduTmrNS5ncZThA7hGBw4hwZcQxNaQOARnuEV3qwn68V6tz6mrSVrNrMPf2B9/gCLVZ3p</latexit>

b1 ∈ Rd1

<latexit sha1_base64="PVi41hGXwKM/EROs5iBVxJnxOBc=">ACnicbVC7TsMwFHXKq5RXgJHFUCExVQlCgoGhEgtjQfQhNSGyHae16jiR7SBVUWYWfoWFAYRY+QI2/gan7QAtR7rS0Tn36t57cMqZ0o7zbVWldW16rtY3Nre0de3evo5JMEtomCU9kDyNFORO0rZnmtJdKimLMaRePrkq/+0ClYom40+OU+jEaCBYxgrSRAvQi5Ee4ijHReBCjwk4FXB+W9znYeAWgV13Gs4EcJG4M1IHM7QC+8sLE5LFVGjCkVJ910m1nyOpGeG0qHmZoikiIzSgfUMFiqny8krBTw2SgijRJoSGk7U3xM5ipUax9h0lneqea8U/P6mY4u/JyJNUkOmiKONQJ7DMBYZMUqL52BEJDO3QjJEhFt0quZENz5lxdJ57ThOg35qzevJzFUQUH4AicABecgya4Bi3QBgQ8gmfwCt6sJ+vFerc+pq0VazazD/7A+vwBEieaeA=</latexit><latexit sha1_base64="PVi41hGXwKM/EROs5iBVxJnxOBc=">ACnicbVC7TsMwFHXKq5RXgJHFUCExVQlCgoGhEgtjQfQhNSGyHae16jiR7SBVUWYWfoWFAYRY+QI2/gan7QAtR7rS0Tn36t57cMqZ0o7zbVWldW16rtY3Nre0de3evo5JMEtomCU9kDyNFORO0rZnmtJdKimLMaRePrkq/+0ClYom40+OU+jEaCBYxgrSRAvQi5Ee4ijHReBCjwk4FXB+W9znYeAWgV13Gs4EcJG4M1IHM7QC+8sLE5LFVGjCkVJ910m1nyOpGeG0qHmZoikiIzSgfUMFiqny8krBTw2SgijRJoSGk7U3xM5ipUax9h0lneqea8U/P6mY4u/JyJNUkOmiKONQJ7DMBYZMUqL52BEJDO3QjJEhFt0quZENz5lxdJ57ThOg35qzevJzFUQUH4AicABecgya4Bi3QBgQ8gmfwCt6sJ+vFerc+pq0VazazD/7A+vwBEieaeA=</latexit><latexit sha1_base64="PVi41hGXwKM/EROs5iBVxJnxOBc=">ACnicbVC7TsMwFHXKq5RXgJHFUCExVQlCgoGhEgtjQfQhNSGyHae16jiR7SBVUWYWfoWFAYRY+QI2/gan7QAtR7rS0Tn36t57cMqZ0o7zbVWldW16rtY3Nre0de3evo5JMEtomCU9kDyNFORO0rZnmtJdKimLMaRePrkq/+0ClYom40+OU+jEaCBYxgrSRAvQi5Ee4ijHReBCjwk4FXB+W9znYeAWgV13Gs4EcJG4M1IHM7QC+8sLE5LFVGjCkVJ910m1nyOpGeG0qHmZoikiIzSgfUMFiqny8krBTw2SgijRJoSGk7U3xM5ipUax9h0lneqea8U/P6mY4u/JyJNUkOmiKONQJ7DMBYZMUqL52BEJDO3QjJEhFt0quZENz5lxdJ57ThOg35qzevJzFUQUH4AicABecgya4Bi3QBgQ8gmfwCt6sJ+vFerc+pq0VazazD/7A+vwBEieaeA=</latexit><latexit sha1_base64="PVi41hGXwKM/EROs5iBVxJnxOBc=">ACnicbVC7TsMwFHXKq5RXgJHFUCExVQlCgoGhEgtjQfQhNSGyHae16jiR7SBVUWYWfoWFAYRY+QI2/gan7QAtR7rS0Tn36t57cMqZ0o7zbVWldW16rtY3Nre0de3evo5JMEtomCU9kDyNFORO0rZnmtJdKimLMaRePrkq/+0ClYom40+OU+jEaCBYxgrSRAvQi5Ee4ijHReBCjwk4FXB+W9znYeAWgV13Gs4EcJG4M1IHM7QC+8sLE5LFVGjCkVJ910m1nyOpGeG0qHmZoikiIzSgfUMFiqny8krBTw2SgijRJoSGk7U3xM5ipUax9h0lneqea8U/P6mY4u/JyJNUkOmiKONQJ7DMBYZMUqL52BEJDO3QjJEhFt0quZENz5lxdJ57ThOg35qzevJzFUQUH4AicABecgya4Bi3QBgQ8gmfwCt6sJ+vFerc+pq0VazazD/7A+vwBEieaeA=</latexit>

b2 ∈ Rd2

<latexit sha1_base64="tvOh1VuDGh+gByDJsf9gRGz3Gs=">ACnicbVC7TsMwFHV4lvIKMLIYKiSmKqmQYGCoxMJYEH1IbYhsx2mtOk5kO0hVlJmFX2FhACFWvoCNv8FpM0DLka50dM69uvcenHCmtON8W0vLK6tr65WN6ubW9s6uvbfUXEqCW2TmMeyh5GinAna1kxz2kskRHmtIvHV4XfaBSsVjc6UlCvQgNBQsZQdpIvn0iJAe4TDud+AybgTMDZbX6fBX4j9+2aU3emgIvELUkNlGj59tcgiEkaUaEJR0r1XSfRXoakZoTvDpIFU0QGaMh7RsqUESVl01fyeGJUQIYxtKU0HCq/p7IUKTUJMKms7hTzXuF+J/XT3V4WVMJKmgswWhSmHOoZFLjBgkhLNJ4YgIpm5FZIRkohok17VhODOv7xIOo269Tdm7Na87KMowIOwTE4BS4B01wDVqgDQh4BM/gFbxZT9aL9W59zFqXrHLmAPyB9fkDFUWaeg=</latexit><latexit sha1_base64="tvOh1VuDGh+gByDJsf9gRGz3Gs=">ACnicbVC7TsMwFHV4lvIKMLIYKiSmKqmQYGCoxMJYEH1IbYhsx2mtOk5kO0hVlJmFX2FhACFWvoCNv8FpM0DLka50dM69uvcenHCmtON8W0vLK6tr65WN6ubW9s6uvbfUXEqCW2TmMeyh5GinAna1kxz2kskRHmtIvHV4XfaBSsVjc6UlCvQgNBQsZQdpIvn0iJAe4TDud+AybgTMDZbX6fBX4j9+2aU3emgIvELUkNlGj59tcgiEkaUaEJR0r1XSfRXoakZoTvDpIFU0QGaMh7RsqUESVl01fyeGJUQIYxtKU0HCq/p7IUKTUJMKms7hTzXuF+J/XT3V4WVMJKmgswWhSmHOoZFLjBgkhLNJ4YgIpm5FZIRkohok17VhODOv7xIOo269Tdm7Na87KMowIOwTE4BS4B01wDVqgDQh4BM/gFbxZT9aL9W59zFqXrHLmAPyB9fkDFUWaeg=</latexit><latexit sha1_base64="tvOh1VuDGh+gByDJsf9gRGz3Gs=">ACnicbVC7TsMwFHV4lvIKMLIYKiSmKqmQYGCoxMJYEH1IbYhsx2mtOk5kO0hVlJmFX2FhACFWvoCNv8FpM0DLka50dM69uvcenHCmtON8W0vLK6tr65WN6ubW9s6uvbfUXEqCW2TmMeyh5GinAna1kxz2kskRHmtIvHV4XfaBSsVjc6UlCvQgNBQsZQdpIvn0iJAe4TDud+AybgTMDZbX6fBX4j9+2aU3emgIvELUkNlGj59tcgiEkaUaEJR0r1XSfRXoakZoTvDpIFU0QGaMh7RsqUESVl01fyeGJUQIYxtKU0HCq/p7IUKTUJMKms7hTzXuF+J/XT3V4WVMJKmgswWhSmHOoZFLjBgkhLNJ4YgIpm5FZIRkohok17VhODOv7xIOo269Tdm7Na87KMowIOwTE4BS4B01wDVqgDQh4BM/gFbxZT9aL9W59zFqXrHLmAPyB9fkDFUWaeg=</latexit><latexit sha1_base64="tvOh1VuDGh+gByDJsf9gRGz3Gs=">ACnicbVC7TsMwFHV4lvIKMLIYKiSmKqmQYGCoxMJYEH1IbYhsx2mtOk5kO0hVlJmFX2FhACFWvoCNv8FpM0DLka50dM69uvcenHCmtON8W0vLK6tr65WN6ubW9s6uvbfUXEqCW2TmMeyh5GinAna1kxz2kskRHmtIvHV4XfaBSsVjc6UlCvQgNBQsZQdpIvn0iJAe4TDud+AybgTMDZbX6fBX4j9+2aU3emgIvELUkNlGj59tcgiEkaUaEJR0r1XSfRXoakZoTvDpIFU0QGaMh7RsqUESVl01fyeGJUQIYxtKU0HCq/p7IUKTUJMKms7hTzXuF+J/XT3V4WVMJKmgswWhSmHOoZFLjBgkhLNJ4YgIpm5FZIRkohok17VhODOv7xIOo269Tdm7Na87KMowIOwTE4BS4B01wDVqgDQh4BM/gFbxZT9aL9W59zFqXrHLmAPyB9fkDFUWaeg=</latexit>

W2 ∈ Rd2×d1

<latexit sha1_base64="Og1rZA4fC86AjGFewrAci9YM3wI=">ACFXicbVBNS8NAEN34WetX1KOXxSJ4kJIUQ8eCl48VrEf0Naw2WzapZtN2J0IJeRPePGvePGgiFfBm/GTduDtj4YeLw3w8w8PxFcg+N8W0vLK6tr6WN8ubW9s6uvbf0nGqKGvSWMSq4xPNBJesCRwE6ySKkcgXrO2Prgq/cCU5rG8g3HC+hEZSB5ySsBIn3aiwgM/TBr514N97jEU8HPbvP7LCg04BHTOPDc3LMrTtWZAC8Sd0YqaIaGZ3/1gpimEZNABdG6zoJ9DOigFPB8nIv1SwhdEQGrGuoJGZTP5t8leNjowQ4jJUpCXi/p7ISKT1OPJNZ3GynvcK8T+vm0J40c+4TFJgk4XhanAEOMiIhxwxSiIsSGEKm5uxXRIFKFgiybENz5lxdJq1Z1nap7c1apX87iKFDdIROkIvOUR1dowZqIoe0TN6RW/Wk/VivVsf09YlazZzgP7A+vwB0rmejw=</latexit><latexit sha1_base64="Og1rZA4fC86AjGFewrAci9YM3wI=">ACFXicbVBNS8NAEN34WetX1KOXxSJ4kJIUQ8eCl48VrEf0Naw2WzapZtN2J0IJeRPePGvePGgiFfBm/GTduDtj4YeLw3w8w8PxFcg+N8W0vLK6tr6WN8ubW9s6uvbf0nGqKGvSWMSq4xPNBJesCRwE6ySKkcgXrO2Prgq/cCU5rG8g3HC+hEZSB5ySsBIn3aiwgM/TBr514N97jEU8HPbvP7LCg04BHTOPDc3LMrTtWZAC8Sd0YqaIaGZ3/1gpimEZNABdG6zoJ9DOigFPB8nIv1SwhdEQGrGuoJGZTP5t8leNjowQ4jJUpCXi/p7ISKT1OPJNZ3GynvcK8T+vm0J40c+4TFJgk4XhanAEOMiIhxwxSiIsSGEKm5uxXRIFKFgiybENz5lxdJq1Z1nap7c1apX87iKFDdIROkIvOUR1dowZqIoe0TN6RW/Wk/VivVsf09YlazZzgP7A+vwB0rmejw=</latexit><latexit sha1_base64="Og1rZA4fC86AjGFewrAci9YM3wI=">ACFXicbVBNS8NAEN34WetX1KOXxSJ4kJIUQ8eCl48VrEf0Naw2WzapZtN2J0IJeRPePGvePGgiFfBm/GTduDtj4YeLw3w8w8PxFcg+N8W0vLK6tr6WN8ubW9s6uvbf0nGqKGvSWMSq4xPNBJesCRwE6ySKkcgXrO2Prgq/cCU5rG8g3HC+hEZSB5ySsBIn3aiwgM/TBr514N97jEU8HPbvP7LCg04BHTOPDc3LMrTtWZAC8Sd0YqaIaGZ3/1gpimEZNABdG6zoJ9DOigFPB8nIv1SwhdEQGrGuoJGZTP5t8leNjowQ4jJUpCXi/p7ISKT1OPJNZ3GynvcK8T+vm0J40c+4TFJgk4XhanAEOMiIhxwxSiIsSGEKm5uxXRIFKFgiybENz5lxdJq1Z1nap7c1apX87iKFDdIROkIvOUR1dowZqIoe0TN6RW/Wk/VivVsf09YlazZzgP7A+vwB0rmejw=</latexit><latexit sha1_base64="Og1rZA4fC86AjGFewrAci9YM3wI=">ACFXicbVBNS8NAEN34WetX1KOXxSJ4kJIUQ8eCl48VrEf0Naw2WzapZtN2J0IJeRPePGvePGgiFfBm/GTduDtj4YeLw3w8w8PxFcg+N8W0vLK6tr6WN8ubW9s6uvbf0nGqKGvSWMSq4xPNBJesCRwE6ySKkcgXrO2Prgq/cCU5rG8g3HC+hEZSB5ySsBIn3aiwgM/TBr514N97jEU8HPbvP7LCg04BHTOPDc3LMrTtWZAC8Sd0YqaIaGZ3/1gpimEZNABdG6zoJ9DOigFPB8nIv1SwhdEQGrGuoJGZTP5t8leNjowQ4jJUpCXi/p7ISKT1OPJNZ3GynvcK8T+vm0J40c+4TFJgk4XhanAEOMiIhxwxSiIsSGEKm5uxXRIFKFgiybENz5lxdJq1Z1nap7c1apX87iKFDdIROkIvOUR1dowZqIoe0TN6RW/Wk/VivVsf09YlazZzgP7A+vwB0rmejw=</latexit>

y = σ(w|h2 + b)

<latexit sha1_base64="9t5JxOFJos7Wnh/aZJ6smWc/vo=">ACG3icbVDLSsNAFJ34rPUVdelmsAgVoSRF0IVCwY3LCvYBTQyT6aQdOpmEmYlSQv7Djb/ixoUirgQX/o2TNoK2HrhwOde7r3HjxmVyrK+jIXFpeWV1dJaeX1jc2vb3NltygRmLRwxCLR9ZEkjHLSUlQx0o0FQaHPSMcfXeZ+54ISN+o8YxcUM04DSgGCkteWZ9DC+gI+kgRFUnRGroB+l9dps6lCsiMGLZjzr06hk8hv6RZ1asmjUBnCd2QSqgQNMzP5x+hJOQcIUZkrJnW7FyUyQUxYxkZSeRJEZ4hAakpylHIZFuOvktg4da6cMgErq4ghP190SKQinHoa8780PlrJeL/3m9RAVnbkp5nCjC8XRkDCoIpgHBftUEKzYWBOEBdW3QjxEAmGdiyzrEOzZl+dJu16zrZp9fVJpnBdxlMA+OABVYINT0ABXoAlaAIMH8ARewKvxaDwb8b7tHXBKGb2wB8Yn9+dqEa</latexit><latexit sha1_base64="9t5JxOFJos7Wnh/aZJ6smWc/vo=">ACG3icbVDLSsNAFJ34rPUVdelmsAgVoSRF0IVCwY3LCvYBTQyT6aQdOpmEmYlSQv7Djb/ixoUirgQX/o2TNoK2HrhwOde7r3HjxmVyrK+jIXFpeWV1dJaeX1jc2vb3NltygRmLRwxCLR9ZEkjHLSUlQx0o0FQaHPSMcfXeZ+54ISN+o8YxcUM04DSgGCkteWZ9DC+gI+kgRFUnRGroB+l9dps6lCsiMGLZjzr06hk8hv6RZ1asmjUBnCd2QSqgQNMzP5x+hJOQcIUZkrJnW7FyUyQUxYxkZSeRJEZ4hAakpylHIZFuOvktg4da6cMgErq4ghP190SKQinHoa8780PlrJeL/3m9RAVnbkp5nCjC8XRkDCoIpgHBftUEKzYWBOEBdW3QjxEAmGdiyzrEOzZl+dJu16zrZp9fVJpnBdxlMA+OABVYINT0ABXoAlaAIMH8ARewKvxaDwb8b7tHXBKGb2wB8Yn9+dqEa</latexit><latexit sha1_base64="9t5JxOFJos7Wnh/aZJ6smWc/vo=">ACG3icbVDLSsNAFJ34rPUVdelmsAgVoSRF0IVCwY3LCvYBTQyT6aQdOpmEmYlSQv7Djb/ixoUirgQX/o2TNoK2HrhwOde7r3HjxmVyrK+jIXFpeWV1dJaeX1jc2vb3NltygRmLRwxCLR9ZEkjHLSUlQx0o0FQaHPSMcfXeZ+54ISN+o8YxcUM04DSgGCkteWZ9DC+gI+kgRFUnRGroB+l9dps6lCsiMGLZjzr06hk8hv6RZ1asmjUBnCd2QSqgQNMzP5x+hJOQcIUZkrJnW7FyUyQUxYxkZSeRJEZ4hAakpylHIZFuOvktg4da6cMgErq4ghP190SKQinHoa8780PlrJeL/3m9RAVnbkp5nCjC8XRkDCoIpgHBftUEKzYWBOEBdW3QjxEAmGdiyzrEOzZl+dJu16zrZp9fVJpnBdxlMA+OABVYINT0ABXoAlaAIMH8ARewKvxaDwb8b7tHXBKGb2wB8Yn9+dqEa</latexit><latexit sha1_base64="9t5JxOFJos7Wnh/aZJ6smWc/vo=">ACG3icbVDLSsNAFJ34rPUVdelmsAgVoSRF0IVCwY3LCvYBTQyT6aQdOpmEmYlSQv7Djb/ixoUirgQX/o2TNoK2HrhwOde7r3HjxmVyrK+jIXFpeWV1dJaeX1jc2vb3NltygRmLRwxCLR9ZEkjHLSUlQx0o0FQaHPSMcfXeZ+54ISN+o8YxcUM04DSgGCkteWZ9DC+gI+kgRFUnRGroB+l9dps6lCsiMGLZjzr06hk8hv6RZ1asmjUBnCd2QSqgQNMzP5x+hJOQcIUZkrJnW7FyUyQUxYxkZSeRJEZ4hAakpylHIZFuOvktg4da6cMgErq4ghP190SKQinHoa8780PlrJeL/3m9RAVnbkp5nCjC8XRkDCoIpgHBftUEKzYWBOEBdW3QjxEAmGdiyzrEOzZl+dJu16zrZp9fVJpnBdxlMA+OABVYINT0ABXoAlaAIMH8ARewKvxaDwb8b7tHXBKGb2wB8Yn9+dqEa</latexit>

L(y, y∗) = −y∗ log y − (1 − y∗) log (1 − y)

<latexit sha1_base64="rRgl1nuidQVks0Gj7Z1U34kvg=">ACI3icbVDLSgMxFM3UV62vUZdugkVoxZYZERBKLpx4aKCfUBnLJk0bUMzD5KMAzL278FTculOLGhf9iZjoLbT2Q5OSce0nucQJGhTSML62wtLyulZcL21sbm3v6Lt7beGHJMW9pnPuw4ShFGPtCSVjHQDTpDrMNJxJjep3kiXFDfe5BRQGwXjTw6pBhJfX1S8tFcowRi+SnQCo8fjKryCNXVazB/FUQJrsGKqLXUyaXarJn29bNSNDHCRmDkpgxzNvj61Bj4OXeJzJAQPdMIpB0jLilmJClZoSABwhM0Ij1FPeQSYcfZjAk8UsoADn2ulidhpv7uiJErROQ6qjKdSMx7qfif1wvl8MKOqReEknh49tAwZFD6MA0MDignWLJIEYQ5VX+FeIw4wlLFWlIhmPMjL5L2ad06ub9WblxncdRBAfgEFSACc5BA9yCJmgBDJ7BK3gH9qL9qZNtc9ZaUHLe/bBH2jfP8ZjoJU=</latexit><latexit sha1_base64="rRgl1nuidQVks0Gj7Z1U34kvg=">ACI3icbVDLSgMxFM3UV62vUZdugkVoxZYZERBKLpx4aKCfUBnLJk0bUMzD5KMAzL278FTculOLGhf9iZjoLbT2Q5OSce0nucQJGhTSML62wtLyulZcL21sbm3v6Lt7beGHJMW9pnPuw4ShFGPtCSVjHQDTpDrMNJxJjep3kiXFDfe5BRQGwXjTw6pBhJfX1S8tFcowRi+SnQCo8fjKryCNXVazB/FUQJrsGKqLXUyaXarJn29bNSNDHCRmDkpgxzNvj61Bj4OXeJzJAQPdMIpB0jLilmJClZoSABwhM0Ij1FPeQSYcfZjAk8UsoADn2ulidhpv7uiJErROQ6qjKdSMx7qfif1wvl8MKOqReEknh49tAwZFD6MA0MDignWLJIEYQ5VX+FeIw4wlLFWlIhmPMjL5L2ad06ub9WblxncdRBAfgEFSACc5BA9yCJmgBDJ7BK3gH9qL9qZNtc9ZaUHLe/bBH2jfP8ZjoJU=</latexit><latexit sha1_base64="rRgl1nuidQVks0Gj7Z1U34kvg=">ACI3icbVDLSgMxFM3UV62vUZdugkVoxZYZERBKLpx4aKCfUBnLJk0bUMzD5KMAzL278FTculOLGhf9iZjoLbT2Q5OSce0nucQJGhTSML62wtLyulZcL21sbm3v6Lt7beGHJMW9pnPuw4ShFGPtCSVjHQDTpDrMNJxJjep3kiXFDfe5BRQGwXjTw6pBhJfX1S8tFcowRi+SnQCo8fjKryCNXVazB/FUQJrsGKqLXUyaXarJn29bNSNDHCRmDkpgxzNvj61Bj4OXeJzJAQPdMIpB0jLilmJClZoSABwhM0Ij1FPeQSYcfZjAk8UsoADn2ulidhpv7uiJErROQ6qjKdSMx7qfif1wvl8MKOqReEknh49tAwZFD6MA0MDignWLJIEYQ5VX+FeIw4wlLFWlIhmPMjL5L2ad06ub9WblxncdRBAfgEFSACc5BA9yCJmgBDJ7BK3gH9qL9qZNtc9ZaUHLe/bBH2jfP8ZjoJU=</latexit><latexit sha1_base64="rRgl1nuidQVks0Gj7Z1U34kvg=">ACI3icbVDLSgMxFM3UV62vUZdugkVoxZYZERBKLpx4aKCfUBnLJk0bUMzD5KMAzL278FTculOLGhf9iZjoLbT2Q5OSce0nucQJGhTSML62wtLyulZcL21sbm3v6Lt7beGHJMW9pnPuw4ShFGPtCSVjHQDTpDrMNJxJjep3kiXFDfe5BRQGwXjTw6pBhJfX1S8tFcowRi+SnQCo8fjKryCNXVazB/FUQJrsGKqLXUyaXarJn29bNSNDHCRmDkpgxzNvj61Bj4OXeJzJAQPdMIpB0jLilmJClZoSABwhM0Ij1FPeQSYcfZjAk8UsoADn2ulidhpv7uiJErROQ6qjKdSMx7qfif1wvl8MKOqReEknh49tAwZFD6MA0MDignWLJIEYQ5VX+FeIw4wlLFWlIhmPMjL5L2ad06ub9WblxncdRBAfgEFSACc5BA9yCJmgBDJ7BK3gH9qL9qZNtc9ZaUHLe/bBH2jfP8ZjoJU=</latexit>

w ∈ Rd2

<latexit sha1_base64="OQGWOHDxod0Lr8suLTzGRzsBoM=">ACHicbVC7TsMwFHXKq5RXgJEBiwqJqUoqJBgYKrEwFkQfUhMix3Faq4T2Q6oijKy8CsDCDEyiew8Tc4bQZoOZKl43Pu1b3+AmjUlnWt1FZWl5ZXau1zY2t7Z3zN29roxTgUkHxywWfR9JwignHUVI/1EBT5jPT8WXh9+6JkDTmt2qSEDdCQ05DipHSkmceOhFSIz/MHnLoUA5nXz+7ye+ywGvmnlm3GtYUcJHYJamDEm3P/HKCGKcR4QozJOXAthLlZkgoihnJa04qSYLwGA3JQFOIiLdbHpIDo+1EsAwFvpxBafq74MRVJOIl9XFnvKea8Q/MGqQrP3YzyJFWE49mgMGVQxbBIBQZUEKzYRBOEBdW7QjxCAmGls6vpEOz5kxdJt9mwrYZ9fVpvXZRxVMEBOAInwAZnoAWuQBt0AaP4Bm8gjfjyXgx3o2PWnFKHv2wR8Ynz/78Znq</latexit><latexit sha1_base64="OQGWOHDxod0Lr8suLTzGRzsBoM=">ACHicbVC7TsMwFHXKq5RXgJEBiwqJqUoqJBgYKrEwFkQfUhMix3Faq4T2Q6oijKy8CsDCDEyiew8Tc4bQZoOZKl43Pu1b3+AmjUlnWt1FZWl5ZXau1zY2t7Z3zN29roxTgUkHxywWfR9JwignHUVI/1EBT5jPT8WXh9+6JkDTmt2qSEDdCQ05DipHSkmceOhFSIz/MHnLoUA5nXz+7ye+ywGvmnlm3GtYUcJHYJamDEm3P/HKCGKcR4QozJOXAthLlZkgoihnJa04qSYLwGA3JQFOIiLdbHpIDo+1EsAwFvpxBafq74MRVJOIl9XFnvKea8Q/MGqQrP3YzyJFWE49mgMGVQxbBIBQZUEKzYRBOEBdW7QjxCAmGls6vpEOz5kxdJt9mwrYZ9fVpvXZRxVMEBOAInwAZnoAWuQBt0AaP4Bm8gjfjyXgx3o2PWnFKHv2wR8Ynz/78Znq</latexit><latexit sha1_base64="OQGWOHDxod0Lr8suLTzGRzsBoM=">ACHicbVC7TsMwFHXKq5RXgJEBiwqJqUoqJBgYKrEwFkQfUhMix3Faq4T2Q6oijKy8CsDCDEyiew8Tc4bQZoOZKl43Pu1b3+AmjUlnWt1FZWl5ZXau1zY2t7Z3zN29roxTgUkHxywWfR9JwignHUVI/1EBT5jPT8WXh9+6JkDTmt2qSEDdCQ05DipHSkmceOhFSIz/MHnLoUA5nXz+7ye+ywGvmnlm3GtYUcJHYJamDEm3P/HKCGKcR4QozJOXAthLlZkgoihnJa04qSYLwGA3JQFOIiLdbHpIDo+1EsAwFvpxBafq74MRVJOIl9XFnvKea8Q/MGqQrP3YzyJFWE49mgMGVQxbBIBQZUEKzYRBOEBdW7QjxCAmGls6vpEOz5kxdJt9mwrYZ9fVpvXZRxVMEBOAInwAZnoAWuQBt0AaP4Bm8gjfjyXgx3o2PWnFKHv2wR8Ynz/78Znq</latexit><latexit sha1_base64="OQGWOHDxod0Lr8suLTzGRzsBoM=">ACHicbVC7TsMwFHXKq5RXgJEBiwqJqUoqJBgYKrEwFkQfUhMix3Faq4T2Q6oijKy8CsDCDEyiew8Tc4bQZoOZKl43Pu1b3+AmjUlnWt1FZWl5ZXau1zY2t7Z3zN29roxTgUkHxywWfR9JwignHUVI/1EBT5jPT8WXh9+6JkDTmt2qSEDdCQ05DipHSkmceOhFSIz/MHnLoUA5nXz+7ye+ywGvmnlm3GtYUcJHYJamDEm3P/HKCGKcR4QozJOXAthLlZkgoihnJa04qSYLwGA3JQFOIiLdbHpIDo+1EsAwFvpxBafq74MRVJOIl9XFnvKea8Q/MGqQrP3YzyJFWE49mgMGVQxbBIBQZUEKzYRBOEBdW7QjxCAmGls6vpEOz5kxdJt9mwrYZ9fVpvXZRxVMEBOAInwAZnoAWuQBt0AaP4Bm8gjfjyXgx3o2PWnFKHv2wR8Ynz/78Znq</latexit>

∂L ∂w =

<latexit sha1_base64="5pB2ylu7QnOzamkCFUHfDzPMs=">ACEnicbVDLSsNAFL2pr1pfUZduBougm5KIoAuFghsXLirYBzShTKaTdujkwcxEKSHf4MZfceNCEbeu3Pk3TtqA2npg4HDOvXfuPV7MmVSW9WUFhaXlfKq5W19Y3NLXN7pyWjRBDaJBGPRMfDknIW0qZitNOLCgOPE7b3ugy9t3VEgWhbdqHFM3wIOQ+YxgpaWeT4ApPUibFQDHN0nf1wJ8Bq6PnpfZahC9Qzq1bNmgDNE7sgVSjQ6JmfTj8iSUBDRTiWsmtbsXLTfDjhNKs4iaQxJiM8oF1NQxQ6aTkzJ0oJU+8iOhX6jQRP3dkeJAynHg6cp8Sznr5eJ/XjdR/pmbsjBOFA3J9CM/4UhFKM8H9ZmgRPGxJpgIpndFZIh1RkqnWNEh2LMnz5PWc2avbNSbV+XsRhj3Yh0Ow4RTqcAUNaAKB3iCF3g1Ho1n4814n5aWjKJnF/7A+PgGmIyd/w=</latexit><latexit sha1_base64="5pB2ylu7QnOzamkCFUHfDzPMs=">ACEnicbVDLSsNAFL2pr1pfUZduBougm5KIoAuFghsXLirYBzShTKaTdujkwcxEKSHf4MZfceNCEbeu3Pk3TtqA2npg4HDOvXfuPV7MmVSW9WUFhaXlfKq5W19Y3NLXN7pyWjRBDaJBGPRMfDknIW0qZitNOLCgOPE7b3ugy9t3VEgWhbdqHFM3wIOQ+YxgpaWeT4ApPUibFQDHN0nf1wJ8Bq6PnpfZahC9Qzq1bNmgDNE7sgVSjQ6JmfTj8iSUBDRTiWsmtbsXLTfDjhNKs4iaQxJiM8oF1NQxQ6aTkzJ0oJU+8iOhX6jQRP3dkeJAynHg6cp8Sznr5eJ/XjdR/pmbsjBOFA3J9CM/4UhFKM8H9ZmgRPGxJpgIpndFZIh1RkqnWNEh2LMnz5PWc2avbNSbV+XsRhj3Yh0Ow4RTqcAUNaAKB3iCF3g1Ho1n4814n5aWjKJnF/7A+PgGmIyd/w=</latexit><latexit sha1_base64="5pB2ylu7QnOzamkCFUHfDzPMs=">ACEnicbVDLSsNAFL2pr1pfUZduBougm5KIoAuFghsXLirYBzShTKaTdujkwcxEKSHf4MZfceNCEbeu3Pk3TtqA2npg4HDOvXfuPV7MmVSW9WUFhaXlfKq5W19Y3NLXN7pyWjRBDaJBGPRMfDknIW0qZitNOLCgOPE7b3ugy9t3VEgWhbdqHFM3wIOQ+YxgpaWeT4ApPUibFQDHN0nf1wJ8Bq6PnpfZahC9Qzq1bNmgDNE7sgVSjQ6JmfTj8iSUBDRTiWsmtbsXLTfDjhNKs4iaQxJiM8oF1NQxQ6aTkzJ0oJU+8iOhX6jQRP3dkeJAynHg6cp8Sznr5eJ/XjdR/pmbsjBOFA3J9CM/4UhFKM8H9ZmgRPGxJpgIpndFZIh1RkqnWNEh2LMnz5PWc2avbNSbV+XsRhj3Yh0Ow4RTqcAUNaAKB3iCF3g1Ho1n4814n5aWjKJnF/7A+PgGmIyd/w=</latexit><latexit sha1_base64="5pB2ylu7QnOzamkCFUHfDzPMs=">ACEnicbVDLSsNAFL2pr1pfUZduBougm5KIoAuFghsXLirYBzShTKaTdujkwcxEKSHf4MZfceNCEbeu3Pk3TtqA2npg4HDOvXfuPV7MmVSW9WUFhaXlfKq5W19Y3NLXN7pyWjRBDaJBGPRMfDknIW0qZitNOLCgOPE7b3ugy9t3VEgWhbdqHFM3wIOQ+YxgpaWeT4ApPUibFQDHN0nf1wJ8Bq6PnpfZahC9Qzq1bNmgDNE7sgVSjQ6JmfTj8iSUBDRTiWsmtbsXLTfDjhNKs4iaQxJiM8oF1NQxQ6aTkzJ0oJU+8iOhX6jQRP3dkeJAynHg6cp8Sznr5eJ/XjdR/pmbsjBOFA3J9CM/4UhFKM8H9ZmgRPGxJpgIpndFZIh1RkqnWNEh2LMnz5PWc2avbNSbV+XsRhj3Yh0Ow4RTqcAUNaAKB3iCF3g1Ho1n4814n5aWjKJnF/7A+PgGmIyd/w=</latexit>

∂L ∂b =

<latexit sha1_base64="Gcz3gTYPh1wbpUR0MFD5vMK9iNY=">ACXicbZDLSsNAFIZPvNZ6i7p0M1gKrkoigi4UCm5cuKhgL9CGMplO2qGTSZiZCVk68ZXceNCEbe+gTvfxkbUFsPDHz8/5yZc34/5kxpx/mylpZXVtfWSxvlza3tnV17b7+lokQS2iQRj2THx4pyJmhTM81pJ5YUhz6nbX98lfvteyoVi8SdnsTUC/FQsIARrI3Ut1EvkJikvRhLzTBHN9kP+xm6RH274tScaFcAuoQFGNv3ZG0QkCanQhGOluq4Tay/N3yScZuVeomiMyRgPadegwCFVXjrdJENVowxQElzhEZT9XdHikOlJqGZrBpiPVLzXi7+53UTHZx7KRNxoqkgs4+ChCMdoTwWNGCSEs0nBjCRzMyKyAibaLQJr2xCcOdXoTWSc1au7taV+UcRgkM4gmNw4QzqcA0NaAKB3iCF3i1Hq1n6816n1dsoqeA/hT1sc38CiZyg=</latexit><latexit sha1_base64="Gcz3gTYPh1wbpUR0MFD5vMK9iNY=">ACXicbZDLSsNAFIZPvNZ6i7p0M1gKrkoigi4UCm5cuKhgL9CGMplO2qGTSZiZCVk68ZXceNCEbe+gTvfxkbUFsPDHz8/5yZc34/5kxpx/mylpZXVtfWSxvlza3tnV17b7+lokQS2iQRj2THx4pyJmhTM81pJ5YUhz6nbX98lfvteyoVi8SdnsTUC/FQsIARrI3Ut1EvkJikvRhLzTBHN9kP+xm6RH274tScaFcAuoQFGNv3ZG0QkCanQhGOluq4Tay/N3yScZuVeomiMyRgPadegwCFVXjrdJENVowxQElzhEZT9XdHikOlJqGZrBpiPVLzXi7+53UTHZx7KRNxoqkgs4+ChCMdoTwWNGCSEs0nBjCRzMyKyAibaLQJr2xCcOdXoTWSc1au7taV+UcRgkM4gmNw4QzqcA0NaAKB3iCF3i1Hq1n6816n1dsoqeA/hT1sc38CiZyg=</latexit><latexit sha1_base64="Gcz3gTYPh1wbpUR0MFD5vMK9iNY=">ACXicbZDLSsNAFIZPvNZ6i7p0M1gKrkoigi4UCm5cuKhgL9CGMplO2qGTSZiZCVk68ZXceNCEbe+gTvfxkbUFsPDHz8/5yZc34/5kxpx/mylpZXVtfWSxvlza3tnV17b7+lokQS2iQRj2THx4pyJmhTM81pJ5YUhz6nbX98lfvteyoVi8SdnsTUC/FQsIARrI3Ut1EvkJikvRhLzTBHN9kP+xm6RH274tScaFcAuoQFGNv3ZG0QkCanQhGOluq4Tay/N3yScZuVeomiMyRgPadegwCFVXjrdJENVowxQElzhEZT9XdHikOlJqGZrBpiPVLzXi7+53UTHZx7KRNxoqkgs4+ChCMdoTwWNGCSEs0nBjCRzMyKyAibaLQJr2xCcOdXoTWSc1au7taV+UcRgkM4gmNw4QzqcA0NaAKB3iCF3i1Hq1n6816n1dsoqeA/hT1sc38CiZyg=</latexit><latexit sha1_base64="Gcz3gTYPh1wbpUR0MFD5vMK9iNY=">ACXicbZDLSsNAFIZPvNZ6i7p0M1gKrkoigi4UCm5cuKhgL9CGMplO2qGTSZiZCVk68ZXceNCEbe+gTvfxkbUFsPDHz8/5yZc34/5kxpx/mylpZXVtfWSxvlza3tnV17b7+lokQS2iQRj2THx4pyJmhTM81pJ5YUhz6nbX98lfvteyoVi8SdnsTUC/FQsIARrI3Ut1EvkJikvRhLzTBHN9kP+xm6RH274tScaFcAuoQFGNv3ZG0QkCanQhGOluq4Tay/N3yScZuVeomiMyRgPadegwCFVXjrdJENVowxQElzhEZT9XdHikOlJqGZrBpiPVLzXi7+53UTHZx7KRNxoqkgs4+ChCMdoTwWNGCSEs0nBjCRzMyKyAibaLQJr2xCcOdXoTWSc1au7taV+UcRgkM4gmNw4QzqcA0NaAKB3iCF3i1Hq1n6816n1dsoqeA/hT1sc38CiZyg=</latexit>

? ?

∂L ∂W2 =

<latexit sha1_base64="pMRkHUBYCknodJGpLhUSRdTS2R8=">ACFHicbVDLSsNAFL2pr1pfUZduBosgCUpgi4UCm5cuKhgH9CEMplO2qGTBzMToYR8hBt/xY0LRdy6cOfOGkDauBgcM596593gxZ1JZ1pdRWlpeWV0r1c2Nre2d8zdvbaMEkFoi0Q8El0PS8pZSFuKU67saA48DjteOr3O/cUyFZFN6pSUzdA9D5jOClZb65onjC0xSJ8ZCMczRTfbDnQCrkenaxfz9Al6ptVq2ZNgRaJXZAqFGj2zU9nEJEkoKEiHEvZs61YuWk+nCaVZxE0hiTMR7SnqYhDqh0+lRGTrSygD5kdAvVGiq/u5IcSDlJPB0Zb6nPdy8T+vlyj/3E1ZGCeKhmT2kZ9wpCKUJ4QGTFCi+EQTATuyIywjolpXOs6BDs+ZMXSbtes62afXtabVwUcZThA7hGw4gwZcQxNaQOABnuAFXo1H49l4M95npSWj6NmHPzA+vgGt1p6E</latexit><latexit sha1_base64="pMRkHUBYCknodJGpLhUSRdTS2R8=">ACFHicbVDLSsNAFL2pr1pfUZduBosgCUpgi4UCm5cuKhgH9CEMplO2qGTBzMToYR8hBt/xY0LRdy6cOfOGkDauBgcM596593gxZ1JZ1pdRWlpeWV0r1c2Nre2d8zdvbaMEkFoi0Q8El0PS8pZSFuKU67saA48DjteOr3O/cUyFZFN6pSUzdA9D5jOClZb65onjC0xSJ8ZCMczRTfbDnQCrkenaxfz9Al6ptVq2ZNgRaJXZAqFGj2zU9nEJEkoKEiHEvZs61YuWk+nCaVZxE0hiTMR7SnqYhDqh0+lRGTrSygD5kdAvVGiq/u5IcSDlJPB0Zb6nPdy8T+vlyj/3E1ZGCeKhmT2kZ9wpCKUJ4QGTFCi+EQTATuyIywjolpXOs6BDs+ZMXSbtes62afXtabVwUcZThA7hGw4gwZcQxNaQOABnuAFXo1H49l4M95npSWj6NmHPzA+vgGt1p6E</latexit><latexit sha1_base64="pMRkHUBYCknodJGpLhUSRdTS2R8=">ACFHicbVDLSsNAFL2pr1pfUZduBosgCUpgi4UCm5cuKhgH9CEMplO2qGTBzMToYR8hBt/xY0LRdy6cOfOGkDauBgcM596593gxZ1JZ1pdRWlpeWV0r1c2Nre2d8zdvbaMEkFoi0Q8El0PS8pZSFuKU67saA48DjteOr3O/cUyFZFN6pSUzdA9D5jOClZb65onjC0xSJ8ZCMczRTfbDnQCrkenaxfz9Al6ptVq2ZNgRaJXZAqFGj2zU9nEJEkoKEiHEvZs61YuWk+nCaVZxE0hiTMR7SnqYhDqh0+lRGTrSygD5kdAvVGiq/u5IcSDlJPB0Zb6nPdy8T+vlyj/3E1ZGCeKhmT2kZ9wpCKUJ4QGTFCi+EQTATuyIywjolpXOs6BDs+ZMXSbtes62afXtabVwUcZThA7hGw4gwZcQxNaQOABnuAFXo1H49l4M95npSWj6NmHPzA+vgGt1p6E</latexit><latexit sha1_base64="pMRkHUBYCknodJGpLhUSRdTS2R8=">ACFHicbVDLSsNAFL2pr1pfUZduBosgCUpgi4UCm5cuKhgH9CEMplO2qGTBzMToYR8hBt/xY0LRdy6cOfOGkDauBgcM596593gxZ1JZ1pdRWlpeWV0r1c2Nre2d8zdvbaMEkFoi0Q8El0PS8pZSFuKU67saA48DjteOr3O/cUyFZFN6pSUzdA9D5jOClZb65onjC0xSJ8ZCMczRTfbDnQCrkenaxfz9Al6ptVq2ZNgRaJXZAqFGj2zU9nEJEkoKEiHEvZs61YuWk+nCaVZxE0hiTMR7SnqYhDqh0+lRGTrSygD5kdAvVGiq/u5IcSDlJPB0Zb6nPdy8T+vlyj/3E1ZGCeKhmT2kZ9wpCKUJ4QGTFCi+EQTATuyIywjolpXOs6BDs+ZMXSbtes62afXtabVwUcZThA7hGw4gwZcQxNaQOABnuAFXo1H49l4M95npSWj6NmHPzA+vgGt1p6E</latexit>

∂L ∂b2 =

<latexit sha1_base64="Ouho/ZMJsF67SF1XIAzLhnRceQ=">ACFHicbVDLSsNAFL2pr1pfVZduBosgCUpgi4UCm5cuKhgH9CEMJlO2qGTBzMToYR8hBt/xY0LRdy6cOfOGkDauBgcM596593gxZ1KZ5pdRWlpeWV0r1c2Nre2d6q7ex0ZJYLQNol4JHoelpSzkLYVU5z2YkFx4Ha9cZXud+9p0KyKLxTk5g6AR6GzGcEKy251RPbF5ikdoyFYpijm+yH2wFWI89PvcxtZOgSudWaWTenQIvEKkgNCrTc6qc9iEgS0FARjqXsW2asnDQfTzjNKnYiaYzJGA9pX9MQB1Q6fSoDB1pZYD8SOgXKjRVf3ekOJByEni6Mt9Tznu5+J/XT5R/7qQsjBNFQzL7yE84UhHKE0IDJihRfKIJoLpXREZYZ2S0jlWdAjW/MmLpNOoW2bduj2tNS+KOMpwAIdwDBacQROuoQVtIPAT/ACr8aj8Wy8Ge+z0pJR9OzDHxgf37Pno8=</latexit><latexit sha1_base64="Ouho/ZMJsF67SF1XIAzLhnRceQ=">ACFHicbVDLSsNAFL2pr1pfVZduBosgCUpgi4UCm5cuKhgH9CEMJlO2qGTBzMToYR8hBt/xY0LRdy6cOfOGkDauBgcM596593gxZ1KZ5pdRWlpeWV0r1c2Nre2d6q7ex0ZJYLQNol4JHoelpSzkLYVU5z2YkFx4Ha9cZXud+9p0KyKLxTk5g6AR6GzGcEKy251RPbF5ikdoyFYpijm+yH2wFWI89PvcxtZOgSudWaWTenQIvEKkgNCrTc6qc9iEgS0FARjqXsW2asnDQfTzjNKnYiaYzJGA9pX9MQB1Q6fSoDB1pZYD8SOgXKjRVf3ekOJByEni6Mt9Tznu5+J/XT5R/7qQsjBNFQzL7yE84UhHKE0IDJihRfKIJoLpXREZYZ2S0jlWdAjW/MmLpNOoW2bduj2tNS+KOMpwAIdwDBacQROuoQVtIPAT/ACr8aj8Wy8Ge+z0pJR9OzDHxgf37Pno8=</latexit><latexit sha1_base64="Ouho/ZMJsF67SF1XIAzLhnRceQ=">ACFHicbVDLSsNAFL2pr1pfVZduBosgCUpgi4UCm5cuKhgH9CEMJlO2qGTBzMToYR8hBt/xY0LRdy6cOfOGkDauBgcM596593gxZ1KZ5pdRWlpeWV0r1c2Nre2d6q7ex0ZJYLQNol4JHoelpSzkLYVU5z2YkFx4Ha9cZXud+9p0KyKLxTk5g6AR6GzGcEKy251RPbF5ikdoyFYpijm+yH2wFWI89PvcxtZOgSudWaWTenQIvEKkgNCrTc6qc9iEgS0FARjqXsW2asnDQfTzjNKnYiaYzJGA9pX9MQB1Q6fSoDB1pZYD8SOgXKjRVf3ekOJByEni6Mt9Tznu5+J/XT5R/7qQsjBNFQzL7yE84UhHKE0IDJihRfKIJoLpXREZYZ2S0jlWdAjW/MmLpNOoW2bduj2tNS+KOMpwAIdwDBacQROuoQVtIPAT/ACr8aj8Wy8Ge+z0pJR9OzDHxgf37Pno8=</latexit><latexit sha1_base64="Ouho/ZMJsF67SF1XIAzLhnRceQ=">ACFHicbVDLSsNAFL2pr1pfVZduBosgCUpgi4UCm5cuKhgH9CEMJlO2qGTBzMToYR8hBt/xY0LRdy6cOfOGkDauBgcM596593gxZ1KZ5pdRWlpeWV0r1c2Nre2d6q7ex0ZJYLQNol4JHoelpSzkLYVU5z2YkFx4Ha9cZXud+9p0KyKLxTk5g6AR6GzGcEKy251RPbF5ikdoyFYpijm+yH2wFWI89PvcxtZOgSudWaWTenQIvEKkgNCrTc6qc9iEgS0FARjqXsW2asnDQfTzjNKnYiaYzJGA9pX9MQB1Q6fSoDB1pZYD8SOgXKjRVf3ekOJByEni6Mt9Tznu5+J/XT5R/7qQsjBNFQzL7yE84UhHKE0IDJihRfKIJoLpXREZYZ2S0jlWdAjW/MmLpNOoW2bduj2tNS+KOMpwAIdwDBacQROuoQVtIPAT/ACr8aj8Wy8Ge+z0pJR9OzDHxgf37Pno8=</latexit>

? ?

∂L ∂W1 =

<latexit sha1_base64="1FU9oIMySt1wauoajnM8XLZ9/C8=">ACFHicbVDLSsNAFL3xWesr6tLNYBEoSQi6EKh4MaFiwr2AU0Ik+mkHTp5MDMRSshHuPFX3LhQxK0Ld/6Nkzagth4YOJxz7517j59wJpVlfRkLi0vLK6uVter6xubWtrmz25ZxKghtkZjHoutjSTmLaEsxWk3ERSHPqcdf3RV+J17KiSLozs1Tqgb4kHEAkaw0pJnHjuBwCRzEiwUwxzd5D/cCbEa+kHWyT07R5fIM2tW3ZoAzRO7JDUo0fTMT6cfkzSkSIcS9mzrUS5WTGecJpXnVTSBJMRHtCephEOqXSzyVE5OtRKHwWx0C9SaKL+7shwKOU49HVlsaec9QrxP6+XquDczViUpIpGZPpRkHKkYlQkhPpMUKL4WBNMBNO7IjLEOiWlc6zqEOzZk+dJ+6RuW3X79rTWuCjqMA+HMAR2HAGDbiGJrSAwAM8wQu8Go/Gs/FmvE9LF4yZw/+wPj4BqxOnoM=</latexit><latexit sha1_base64="1FU9oIMySt1wauoajnM8XLZ9/C8=">ACFHicbVDLSsNAFL3xWesr6tLNYBEoSQi6EKh4MaFiwr2AU0Ik+mkHTp5MDMRSshHuPFX3LhQxK0Ld/6Nkzagth4YOJxz7517j59wJpVlfRkLi0vLK6uVter6xubWtrmz25ZxKghtkZjHoutjSTmLaEsxWk3ERSHPqcdf3RV+J17KiSLozs1Tqgb4kHEAkaw0pJnHjuBwCRzEiwUwxzd5D/cCbEa+kHWyT07R5fIM2tW3ZoAzRO7JDUo0fTMT6cfkzSkSIcS9mzrUS5WTGecJpXnVTSBJMRHtCephEOqXSzyVE5OtRKHwWx0C9SaKL+7shwKOU49HVlsaec9QrxP6+XquDczViUpIpGZPpRkHKkYlQkhPpMUKL4WBNMBNO7IjLEOiWlc6zqEOzZk+dJ+6RuW3X79rTWuCjqMA+HMAR2HAGDbiGJrSAwAM8wQu8Go/Gs/FmvE9LF4yZw/+wPj4BqxOnoM=</latexit><latexit sha1_base64="1FU9oIMySt1wauoajnM8XLZ9/C8=">ACFHicbVDLSsNAFL3xWesr6tLNYBEoSQi6EKh4MaFiwr2AU0Ik+mkHTp5MDMRSshHuPFX3LhQxK0Ld/6Nkzagth4YOJxz7517j59wJpVlfRkLi0vLK6uVter6xubWtrmz25ZxKghtkZjHoutjSTmLaEsxWk3ERSHPqcdf3RV+J17KiSLozs1Tqgb4kHEAkaw0pJnHjuBwCRzEiwUwxzd5D/cCbEa+kHWyT07R5fIM2tW3ZoAzRO7JDUo0fTMT6cfkzSkSIcS9mzrUS5WTGecJpXnVTSBJMRHtCephEOqXSzyVE5OtRKHwWx0C9SaKL+7shwKOU49HVlsaec9QrxP6+XquDczViUpIpGZPpRkHKkYlQkhPpMUKL4WBNMBNO7IjLEOiWlc6zqEOzZk+dJ+6RuW3X79rTWuCjqMA+HMAR2HAGDbiGJrSAwAM8wQu8Go/Gs/FmvE9LF4yZw/+wPj4BqxOnoM=</latexit><latexit sha1_base64="1FU9oIMySt1wauoajnM8XLZ9/C8=">ACFHicbVDLSsNAFL3xWesr6tLNYBEoSQi6EKh4MaFiwr2AU0Ik+mkHTp5MDMRSshHuPFX3LhQxK0Ld/6Nkzagth4YOJxz7517j59wJpVlfRkLi0vLK6uVter6xubWtrmz25ZxKghtkZjHoutjSTmLaEsxWk3ERSHPqcdf3RV+J17KiSLozs1Tqgb4kHEAkaw0pJnHjuBwCRzEiwUwxzd5D/cCbEa+kHWyT07R5fIM2tW3ZoAzRO7JDUo0fTMT6cfkzSkSIcS9mzrUS5WTGecJpXnVTSBJMRHtCephEOqXSzyVE5OtRKHwWx0C9SaKL+7shwKOU49HVlsaec9QrxP6+XquDczViUpIpGZPpRkHKkYlQkhPpMUKL4WBNMBNO7IjLEOiWlc6zqEOzZk+dJ+6RuW3X79rTWuCjqMA+HMAR2HAGDbiGJrSAwAM8wQu8Go/Gs/FmvE9LF4yZw/+wPj4BqxOnoM=</latexit>

∂L ∂b1 =

<latexit sha1_base64="BQiLxk2KtvDrP9eOeoWod8C89LE=">ACFHicbVDLSsNAFL3xWesr6tLNYBEoSQi6EKh4MaFiwr2AU0Ik+mkHTp5MDMRSshHuPFX3LhQxK0Ld/6Nkzagth4YOJxz7517j59wJpVlfRkLi0vLK6uVter6xubWtrmz25ZxKghtkZjHoutjSTmLaEsxWk3ERSHPqcdf3RV+J17KiSLozs1Tqgb4kHEAkaw0pJnHjuBwCRzEiwUwxzd5D/cCbEa+kHm56do0vkmTWrbk2A5oldkhqUaHrmp9OPSRrSBGOpezZVqLcrBhPOM2rTipgskID2hP0wiHVLrZ5KgcHWqlj4JY6BcpNF/d2Q4lHIc+rqy2FPOeoX4n9dLVXDuZixKUkUjMv0oSDlSMSoSQn0mKF8rAkmguldERlinZLSOVZ1CPbsyfOkfVK3rbp9e1prXJRxVGAfDuAIbDiDBlxDE1pA4AGe4AVejUfj2Xgz3qelC0bZswd/YHx8A71Hno4=</latexit><latexit sha1_base64="BQiLxk2KtvDrP9eOeoWod8C89LE=">ACFHicbVDLSsNAFL3xWesr6tLNYBEoSQi6EKh4MaFiwr2AU0Ik+mkHTp5MDMRSshHuPFX3LhQxK0Ld/6Nkzagth4YOJxz7517j59wJpVlfRkLi0vLK6uVter6xubWtrmz25ZxKghtkZjHoutjSTmLaEsxWk3ERSHPqcdf3RV+J17KiSLozs1Tqgb4kHEAkaw0pJnHjuBwCRzEiwUwxzd5D/cCbEa+kHm56do0vkmTWrbk2A5oldkhqUaHrmp9OPSRrSBGOpezZVqLcrBhPOM2rTipgskID2hP0wiHVLrZ5KgcHWqlj4JY6BcpNF/d2Q4lHIc+rqy2FPOeoX4n9dLVXDuZixKUkUjMv0oSDlSMSoSQn0mKF8rAkmguldERlinZLSOVZ1CPbsyfOkfVK3rbp9e1prXJRxVGAfDuAIbDiDBlxDE1pA4AGe4AVejUfj2Xgz3qelC0bZswd/YHx8A71Hno4=</latexit><latexit sha1_base64="BQiLxk2KtvDrP9eOeoWod8C89LE=">ACFHicbVDLSsNAFL3xWesr6tLNYBEoSQi6EKh4MaFiwr2AU0Ik+mkHTp5MDMRSshHuPFX3LhQxK0Ld/6Nkzagth4YOJxz7517j59wJpVlfRkLi0vLK6uVter6xubWtrmz25ZxKghtkZjHoutjSTmLaEsxWk3ERSHPqcdf3RV+J17KiSLozs1Tqgb4kHEAkaw0pJnHjuBwCRzEiwUwxzd5D/cCbEa+kHm56do0vkmTWrbk2A5oldkhqUaHrmp9OPSRrSBGOpezZVqLcrBhPOM2rTipgskID2hP0wiHVLrZ5KgcHWqlj4JY6BcpNF/d2Q4lHIc+rqy2FPOeoX4n9dLVXDuZixKUkUjMv0oSDlSMSoSQn0mKF8rAkmguldERlinZLSOVZ1CPbsyfOkfVK3rbp9e1prXJRxVGAfDuAIbDiDBlxDE1pA4AGe4AVejUfj2Xgz3qelC0bZswd/YHx8A71Hno4=</latexit><latexit sha1_base64="BQiLxk2KtvDrP9eOeoWod8C89LE=">ACFHicbVDLSsNAFL3xWesr6tLNYBEoSQi6EKh4MaFiwr2AU0Ik+mkHTp5MDMRSshHuPFX3LhQxK0Ld/6Nkzagth4YOJxz7517j59wJpVlfRkLi0vLK6uVter6xubWtrmz25ZxKghtkZjHoutjSTmLaEsxWk3ERSHPqcdf3RV+J17KiSLozs1Tqgb4kHEAkaw0pJnHjuBwCRzEiwUwxzd5D/cCbEa+kHm56do0vkmTWrbk2A5oldkhqUaHrmp9OPSRrSBGOpezZVqLcrBhPOM2rTipgskID2hP0wiHVLrZ5KgcHWqlj4JY6BcpNF/d2Q4lHIc+rqy2FPOeoX4n9dLVXDuZixKUkUjMv0oSDlSMSoSQn0mKF8rAkmguldERlinZLSOVZ1CPbsyfOkfVK3rbp9e1prXJRxVGAfDuAIbDiDBlxDE1pA4AGe4AVejUfj2Xgz3qelC0bZswd/YHx8A71Hno4=</latexit>

? ?

slide-39
SLIDE 39

Deriving gradients for Feedforward NNs

Forward Propagation

z1 = W1x + b1

<latexit sha1_base64="sNAfRTR6jAb1N5WltONey3VxJ2w=">ACH3icbVBLS8NAEJ74rPUV9ehlsQiCUBIR9aBQ8OKxgn1AG8pmu2mXbh7sbsQa8k+8+Fe8eFBEvPXfuGlT0daBhe8xw858bsSZVJY1MhYWl5ZXVgtrxfWNza1tc2e3LsNYEFojIQ9F08WSchbQmK02YkKPZdThvu4DrzG/dUSBYGd2oYUcfHvYB5jGClpY51vax6rte8ph2bHSFprSR0Sl5SBE6/mGutjpmySpb40LzwM5BCfKqdsyvdjcksU8DRTiWsmVbkXISLBQjnKbFdixphMkA92hLwD7VDrJ+L4UHWqli7xQ6BcoNFZ/TyTYl3Lou7oz21HOepn4n9eKlXfhJCyIYkUDMvnIizlSIcrCQl0mKF8qAEmguldEeljgYnSkRZ1CPbsyfOgflK2rbJ9e1qXOZxFGAfDuAIbDiHCtxAFWpA4Ale4A3ejWfj1fgwPietC0Y+swd/yh9Ax9Jolw=</latexit><latexit sha1_base64="sNAfRTR6jAb1N5WltONey3VxJ2w=">ACH3icbVBLS8NAEJ74rPUV9ehlsQiCUBIR9aBQ8OKxgn1AG8pmu2mXbh7sbsQa8k+8+Fe8eFBEvPXfuGlT0daBhe8xw858bsSZVJY1MhYWl5ZXVgtrxfWNza1tc2e3LsNYEFojIQ9F08WSchbQmK02YkKPZdThvu4DrzG/dUSBYGd2oYUcfHvYB5jGClpY51vax6rte8ph2bHSFprSR0Sl5SBE6/mGutjpmySpb40LzwM5BCfKqdsyvdjcksU8DRTiWsmVbkXISLBQjnKbFdixphMkA92hLwD7VDrJ+L4UHWqli7xQ6BcoNFZ/TyTYl3Lou7oz21HOepn4n9eKlXfhJCyIYkUDMvnIizlSIcrCQl0mKF8qAEmguldEeljgYnSkRZ1CPbsyfOgflK2rbJ9e1qXOZxFGAfDuAIbDiHCtxAFWpA4Ale4A3ejWfj1fgwPietC0Y+swd/yh9Ax9Jolw=</latexit><latexit sha1_base64="sNAfRTR6jAb1N5WltONey3VxJ2w=">ACH3icbVBLS8NAEJ74rPUV9ehlsQiCUBIR9aBQ8OKxgn1AG8pmu2mXbh7sbsQa8k+8+Fe8eFBEvPXfuGlT0daBhe8xw858bsSZVJY1MhYWl5ZXVgtrxfWNza1tc2e3LsNYEFojIQ9F08WSchbQmK02YkKPZdThvu4DrzG/dUSBYGd2oYUcfHvYB5jGClpY51vax6rte8ph2bHSFprSR0Sl5SBE6/mGutjpmySpb40LzwM5BCfKqdsyvdjcksU8DRTiWsmVbkXISLBQjnKbFdixphMkA92hLwD7VDrJ+L4UHWqli7xQ6BcoNFZ/TyTYl3Lou7oz21HOepn4n9eKlXfhJCyIYkUDMvnIizlSIcrCQl0mKF8qAEmguldEeljgYnSkRZ1CPbsyfOgflK2rbJ9e1qXOZxFGAfDuAIbDiHCtxAFWpA4Ale4A3ejWfj1fgwPietC0Y+swd/yh9Ax9Jolw=</latexit><latexit sha1_base64="sNAfRTR6jAb1N5WltONey3VxJ2w=">ACH3icbVBLS8NAEJ74rPUV9ehlsQiCUBIR9aBQ8OKxgn1AG8pmu2mXbh7sbsQa8k+8+Fe8eFBEvPXfuGlT0daBhe8xw858bsSZVJY1MhYWl5ZXVgtrxfWNza1tc2e3LsNYEFojIQ9F08WSchbQmK02YkKPZdThvu4DrzG/dUSBYGd2oYUcfHvYB5jGClpY51vax6rte8ph2bHSFprSR0Sl5SBE6/mGutjpmySpb40LzwM5BCfKqdsyvdjcksU8DRTiWsmVbkXISLBQjnKbFdixphMkA92hLwD7VDrJ+L4UHWqli7xQ6BcoNFZ/TyTYl3Lou7oz21HOepn4n9eKlXfhJCyIYkUDMvnIizlSIcrCQl0mKF8qAEmguldEeljgYnSkRZ1CPbsyfOgflK2rbJ9e1qXOZxFGAfDuAIbDiHCtxAFWpA4Ale4A3ejWfj1fgwPietC0Y+swd/yh9Ax9Jolw=</latexit>

h1 = tanh(z1)

<latexit sha1_base64="4s+QI7YoAoXKkwMswE+kACGuHhE=">AC3icbZDLSsNAFIYnXmu9RV26GVqEuimJCLpQKLhxWcFeoA1hMp20QyeTMHMi1NC9G1/FjQtF3PoC7nwbp20Ebf1h4Oc75zDn/EiuAbH+bKWldW19YLG8XNre2dXtv6njVFHWoLGIVTsgmgkuWQM4CNZOFCNRIFgrGF5N6q07pjSP5S2MEuZFpC95yCkBg3y71I0IDIwG4x9F1/iLhA5qPzAewOPfbvsVJ2p8KJxc1NGueq+/dntxTSNmAQqiNYd10nAy4gCTgUbF7upZgmhQ9JnHWMliZj2suktY3xkSA+HsTJPAp7S3xMZibQeRYHpnCyp52sT+F+tk0J47mVcJikwSWcfhanAEONJMLjHFaMgRsYQqrjZFdMBUYSCia9oQnDnT140zZOq61Tdm9Ny7SKPo4AOUQlVkIvOUA1dozpqIoe0BN6Qa/Wo/VsvVnvs9YlK585QH9kfXwDBNSaXA=</latexit><latexit sha1_base64="4s+QI7YoAoXKkwMswE+kACGuHhE=">AC3icbZDLSsNAFIYnXmu9RV26GVqEuimJCLpQKLhxWcFeoA1hMp20QyeTMHMi1NC9G1/FjQtF3PoC7nwbp20Ebf1h4Oc75zDn/EiuAbH+bKWldW19YLG8XNre2dXtv6njVFHWoLGIVTsgmgkuWQM4CNZOFCNRIFgrGF5N6q07pjSP5S2MEuZFpC95yCkBg3y71I0IDIwG4x9F1/iLhA5qPzAewOPfbvsVJ2p8KJxc1NGueq+/dntxTSNmAQqiNYd10nAy4gCTgUbF7upZgmhQ9JnHWMliZj2suktY3xkSA+HsTJPAp7S3xMZibQeRYHpnCyp52sT+F+tk0J47mVcJikwSWcfhanAEONJMLjHFaMgRsYQqrjZFdMBUYSCia9oQnDnT140zZOq61Tdm9Ny7SKPo4AOUQlVkIvOUA1dozpqIoe0BN6Qa/Wo/VsvVnvs9YlK585QH9kfXwDBNSaXA=</latexit><latexit sha1_base64="4s+QI7YoAoXKkwMswE+kACGuHhE=">AC3icbZDLSsNAFIYnXmu9RV26GVqEuimJCLpQKLhxWcFeoA1hMp20QyeTMHMi1NC9G1/FjQtF3PoC7nwbp20Ebf1h4Oc75zDn/EiuAbH+bKWldW19YLG8XNre2dXtv6njVFHWoLGIVTsgmgkuWQM4CNZOFCNRIFgrGF5N6q07pjSP5S2MEuZFpC95yCkBg3y71I0IDIwG4x9F1/iLhA5qPzAewOPfbvsVJ2p8KJxc1NGueq+/dntxTSNmAQqiNYd10nAy4gCTgUbF7upZgmhQ9JnHWMliZj2suktY3xkSA+HsTJPAp7S3xMZibQeRYHpnCyp52sT+F+tk0J47mVcJikwSWcfhanAEONJMLjHFaMgRsYQqrjZFdMBUYSCia9oQnDnT140zZOq61Tdm9Ny7SKPo4AOUQlVkIvOUA1dozpqIoe0BN6Qa/Wo/VsvVnvs9YlK585QH9kfXwDBNSaXA=</latexit><latexit sha1_base64="4s+QI7YoAoXKkwMswE+kACGuHhE=">AC3icbZDLSsNAFIYnXmu9RV26GVqEuimJCLpQKLhxWcFeoA1hMp20QyeTMHMi1NC9G1/FjQtF3PoC7nwbp20Ebf1h4Oc75zDn/EiuAbH+bKWldW19YLG8XNre2dXtv6njVFHWoLGIVTsgmgkuWQM4CNZOFCNRIFgrGF5N6q07pjSP5S2MEuZFpC95yCkBg3y71I0IDIwG4x9F1/iLhA5qPzAewOPfbvsVJ2p8KJxc1NGueq+/dntxTSNmAQqiNYd10nAy4gCTgUbF7upZgmhQ9JnHWMliZj2suktY3xkSA+HsTJPAp7S3xMZibQeRYHpnCyp52sT+F+tk0J47mVcJikwSWcfhanAEONJMLjHFaMgRsYQqrjZFdMBUYSCia9oQnDnT140zZOq61Tdm9Ny7SKPo4AOUQlVkIvOUA1dozpqIoe0BN6Qa/Wo/VsvVnvs9YlK585QH9kfXwDBNSaXA=</latexit>

h2 = tanh(z2)

<latexit sha1_base64="ehfJMJ7I8rzQYVl914iHeO3gHU=">AC3icbZDLSsNAFIYn9VbrLerSzdAi1E1JiqALhYIblxXsBZoQJtNJM3QyCTMToYbu3fgqblwo4tYXcOfbOGkjaOsPAz/fOYc5/cTRqWyrC+jtLK6tr5R3qxsbe/s7pn7B10ZpwKTDo5ZLPo+koRTjqKkb6iSAo8hnp+eOrvN67I0LSmN+qSULcCI04DShGSiPrDoRUqEfZOHUa8JL6CjEw/oPvNfwxDNrVsOaCS4buzA1UKjtmZ/OMZpRLjCDEk5sK1EuRkSimJGphUnlSRBeIxGZKAtRxGRbja7ZQqPNRnCIBb6cQVn9PdEhiIpJ5GvO/Ml5WIth/VBqkKzt2M8iRVhOP5R0HKoIphHgwcUkGwYhNtEBZU7wpxiATCSsdX0SHYiycvm26zYVsN+a01ro4iDI1AFdWCDM9AC16ANOgCDB/AEXsCr8Wg8G2/G+7y1ZBQzh+CPjI9vB/OaXg=</latexit><latexit sha1_base64="ehfJMJ7I8rzQYVl914iHeO3gHU=">AC3icbZDLSsNAFIYn9VbrLerSzdAi1E1JiqALhYIblxXsBZoQJtNJM3QyCTMToYbu3fgqblwo4tYXcOfbOGkjaOsPAz/fOYc5/cTRqWyrC+jtLK6tr5R3qxsbe/s7pn7B10ZpwKTDo5ZLPo+koRTjqKkb6iSAo8hnp+eOrvN67I0LSmN+qSULcCI04DShGSiPrDoRUqEfZOHUa8JL6CjEw/oPvNfwxDNrVsOaCS4buzA1UKjtmZ/OMZpRLjCDEk5sK1EuRkSimJGphUnlSRBeIxGZKAtRxGRbja7ZQqPNRnCIBb6cQVn9PdEhiIpJ5GvO/Ml5WIth/VBqkKzt2M8iRVhOP5R0HKoIphHgwcUkGwYhNtEBZU7wpxiATCSsdX0SHYiycvm26zYVsN+a01ro4iDI1AFdWCDM9AC16ANOgCDB/AEXsCr8Wg8G2/G+7y1ZBQzh+CPjI9vB/OaXg=</latexit><latexit sha1_base64="ehfJMJ7I8rzQYVl914iHeO3gHU=">AC3icbZDLSsNAFIYn9VbrLerSzdAi1E1JiqALhYIblxXsBZoQJtNJM3QyCTMToYbu3fgqblwo4tYXcOfbOGkjaOsPAz/fOYc5/cTRqWyrC+jtLK6tr5R3qxsbe/s7pn7B10ZpwKTDo5ZLPo+koRTjqKkb6iSAo8hnp+eOrvN67I0LSmN+qSULcCI04DShGSiPrDoRUqEfZOHUa8JL6CjEw/oPvNfwxDNrVsOaCS4buzA1UKjtmZ/OMZpRLjCDEk5sK1EuRkSimJGphUnlSRBeIxGZKAtRxGRbja7ZQqPNRnCIBb6cQVn9PdEhiIpJ5GvO/Ml5WIth/VBqkKzt2M8iRVhOP5R0HKoIphHgwcUkGwYhNtEBZU7wpxiATCSsdX0SHYiycvm26zYVsN+a01ro4iDI1AFdWCDM9AC16ANOgCDB/AEXsCr8Wg8G2/G+7y1ZBQzh+CPjI9vB/OaXg=</latexit><latexit sha1_base64="ehfJMJ7I8rzQYVl914iHeO3gHU=">AC3icbZDLSsNAFIYn9VbrLerSzdAi1E1JiqALhYIblxXsBZoQJtNJM3QyCTMToYbu3fgqblwo4tYXcOfbOGkjaOsPAz/fOYc5/cTRqWyrC+jtLK6tr5R3qxsbe/s7pn7B10ZpwKTDo5ZLPo+koRTjqKkb6iSAo8hnp+eOrvN67I0LSmN+qSULcCI04DShGSiPrDoRUqEfZOHUa8JL6CjEw/oPvNfwxDNrVsOaCS4buzA1UKjtmZ/OMZpRLjCDEk5sK1EuRkSimJGphUnlSRBeIxGZKAtRxGRbja7ZQqPNRnCIBb6cQVn9PdEhiIpJ5GvO/Ml5WIth/VBqkKzt2M8iRVhOP5R0HKoIphHgwcUkGwYhNtEBZU7wpxiATCSsdX0SHYiycvm26zYVsN+a01ro4iDI1AFdWCDM9AC16ANOgCDB/AEXsCr8Wg8G2/G+7y1ZBQzh+CPjI9vB/OaXg=</latexit>

y = σ(w|h2 + b)

<latexit sha1_base64="9t5JxOFJos7Wnh/aZJ6smWc/vo=">ACG3icbVDLSsNAFJ34rPUVdelmsAgVoSRF0IVCwY3LCvYBTQyT6aQdOpmEmYlSQv7Djb/ixoUirgQX/o2TNoK2HrhwOde7r3HjxmVyrK+jIXFpeWV1dJaeX1jc2vb3NltygRmLRwxCLR9ZEkjHLSUlQx0o0FQaHPSMcfXeZ+54ISN+o8YxcUM04DSgGCkteWZ9DC+gI+kgRFUnRGroB+l9dps6lCsiMGLZjzr06hk8hv6RZ1asmjUBnCd2QSqgQNMzP5x+hJOQcIUZkrJnW7FyUyQUxYxkZSeRJEZ4hAakpylHIZFuOvktg4da6cMgErq4ghP190SKQinHoa8780PlrJeL/3m9RAVnbkp5nCjC8XRkDCoIpgHBftUEKzYWBOEBdW3QjxEAmGdiyzrEOzZl+dJu16zrZp9fVJpnBdxlMA+OABVYINT0ABXoAlaAIMH8ARewKvxaDwb8b7tHXBKGb2wB8Yn9+dqEa</latexit><latexit sha1_base64="9t5JxOFJos7Wnh/aZJ6smWc/vo=">ACG3icbVDLSsNAFJ34rPUVdelmsAgVoSRF0IVCwY3LCvYBTQyT6aQdOpmEmYlSQv7Djb/ixoUirgQX/o2TNoK2HrhwOde7r3HjxmVyrK+jIXFpeWV1dJaeX1jc2vb3NltygRmLRwxCLR9ZEkjHLSUlQx0o0FQaHPSMcfXeZ+54ISN+o8YxcUM04DSgGCkteWZ9DC+gI+kgRFUnRGroB+l9dps6lCsiMGLZjzr06hk8hv6RZ1asmjUBnCd2QSqgQNMzP5x+hJOQcIUZkrJnW7FyUyQUxYxkZSeRJEZ4hAakpylHIZFuOvktg4da6cMgErq4ghP190SKQinHoa8780PlrJeL/3m9RAVnbkp5nCjC8XRkDCoIpgHBftUEKzYWBOEBdW3QjxEAmGdiyzrEOzZl+dJu16zrZp9fVJpnBdxlMA+OABVYINT0ABXoAlaAIMH8ARewKvxaDwb8b7tHXBKGb2wB8Yn9+dqEa</latexit><latexit sha1_base64="9t5JxOFJos7Wnh/aZJ6smWc/vo=">ACG3icbVDLSsNAFJ34rPUVdelmsAgVoSRF0IVCwY3LCvYBTQyT6aQdOpmEmYlSQv7Djb/ixoUirgQX/o2TNoK2HrhwOde7r3HjxmVyrK+jIXFpeWV1dJaeX1jc2vb3NltygRmLRwxCLR9ZEkjHLSUlQx0o0FQaHPSMcfXeZ+54ISN+o8YxcUM04DSgGCkteWZ9DC+gI+kgRFUnRGroB+l9dps6lCsiMGLZjzr06hk8hv6RZ1asmjUBnCd2QSqgQNMzP5x+hJOQcIUZkrJnW7FyUyQUxYxkZSeRJEZ4hAakpylHIZFuOvktg4da6cMgErq4ghP190SKQinHoa8780PlrJeL/3m9RAVnbkp5nCjC8XRkDCoIpgHBftUEKzYWBOEBdW3QjxEAmGdiyzrEOzZl+dJu16zrZp9fVJpnBdxlMA+OABVYINT0ABXoAlaAIMH8ARewKvxaDwb8b7tHXBKGb2wB8Yn9+dqEa</latexit><latexit sha1_base64="9t5JxOFJos7Wnh/aZJ6smWc/vo=">ACG3icbVDLSsNAFJ34rPUVdelmsAgVoSRF0IVCwY3LCvYBTQyT6aQdOpmEmYlSQv7Djb/ixoUirgQX/o2TNoK2HrhwOde7r3HjxmVyrK+jIXFpeWV1dJaeX1jc2vb3NltygRmLRwxCLR9ZEkjHLSUlQx0o0FQaHPSMcfXeZ+54ISN+o8YxcUM04DSgGCkteWZ9DC+gI+kgRFUnRGroB+l9dps6lCsiMGLZjzr06hk8hv6RZ1asmjUBnCd2QSqgQNMzP5x+hJOQcIUZkrJnW7FyUyQUxYxkZSeRJEZ4hAakpylHIZFuOvktg4da6cMgErq4ghP190SKQinHoa8780PlrJeL/3m9RAVnbkp5nCjC8XRkDCoIpgHBftUEKzYWBOEBdW3QjxEAmGdiyzrEOzZl+dJu16zrZp9fVJpnBdxlMA+OABVYINT0ABXoAlaAIMH8ARewKvxaDwb8b7tHXBKGb2wB8Yn9+dqEa</latexit>

z2 = W2h1 + b2

<latexit sha1_base64="YMtT1sm9VIU+T4aPKLOW7IZqw9c=">ACIHicbVBLS8NAEJ74rPUV9ehlsQiCUJIi1INCwYvHCvYBbQmb7aZdunmwuxFqyE/x4l/x4kERvemvcdOmoq0DC9jhp353IgzqSzr01haXldWy9sFDe3tnd2zb39pgxjQWiDhDwUbRdLylAG4opTtuRoNh3OW25o6vMb91RIVkY3KpxRHs+HgTMYwQrLTlmtetjNXS95D51KugSzWgrozMyTB0bnf5QV3uOWbLK1qTQIrBzUIK86o750e2HJPZpoAjHUnZsK1K9BAvFCKdpsRtLGmEywgPa0TDAPpW9ZHJgio610kdeKPQLFJqovycS7Es59l3dme0o571M/M/rxMo7yUsiGJFAzL9yIs5UiHK0kJ9JihRfKwBJoLpXREZYoGJ0pkWdQj2/MmLoFkp21bZvjkr1S7yOApwCEdwAjZUoQbXUIcGEHiAJ3iBV+PReDbejPdp65KRzxzAnzK+vgHvMKLJ</latexit><latexit sha1_base64="YMtT1sm9VIU+T4aPKLOW7IZqw9c=">ACIHicbVBLS8NAEJ74rPUV9ehlsQiCUJIi1INCwYvHCvYBbQmb7aZdunmwuxFqyE/x4l/x4kERvemvcdOmoq0DC9jhp353IgzqSzr01haXldWy9sFDe3tnd2zb39pgxjQWiDhDwUbRdLylAG4opTtuRoNh3OW25o6vMb91RIVkY3KpxRHs+HgTMYwQrLTlmtetjNXS95D51KugSzWgrozMyTB0bnf5QV3uOWbLK1qTQIrBzUIK86o750e2HJPZpoAjHUnZsK1K9BAvFCKdpsRtLGmEywgPa0TDAPpW9ZHJgio610kdeKPQLFJqovycS7Es59l3dme0o571M/M/rxMo7yUsiGJFAzL9yIs5UiHK0kJ9JihRfKwBJoLpXREZYoGJ0pkWdQj2/MmLoFkp21bZvjkr1S7yOApwCEdwAjZUoQbXUIcGEHiAJ3iBV+PReDbejPdp65KRzxzAnzK+vgHvMKLJ</latexit><latexit sha1_base64="YMtT1sm9VIU+T4aPKLOW7IZqw9c=">ACIHicbVBLS8NAEJ74rPUV9ehlsQiCUJIi1INCwYvHCvYBbQmb7aZdunmwuxFqyE/x4l/x4kERvemvcdOmoq0DC9jhp353IgzqSzr01haXldWy9sFDe3tnd2zb39pgxjQWiDhDwUbRdLylAG4opTtuRoNh3OW25o6vMb91RIVkY3KpxRHs+HgTMYwQrLTlmtetjNXS95D51KugSzWgrozMyTB0bnf5QV3uOWbLK1qTQIrBzUIK86o750e2HJPZpoAjHUnZsK1K9BAvFCKdpsRtLGmEywgPa0TDAPpW9ZHJgio610kdeKPQLFJqovycS7Es59l3dme0o571M/M/rxMo7yUsiGJFAzL9yIs5UiHK0kJ9JihRfKwBJoLpXREZYoGJ0pkWdQj2/MmLoFkp21bZvjkr1S7yOApwCEdwAjZUoQbXUIcGEHiAJ3iBV+PReDbejPdp65KRzxzAnzK+vgHvMKLJ</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQvEKMXyw=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odBu3wMYA6nMFXEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="NjSIBTcqwMj6vsN1L82c4MUo9Gg=">ACFXicbVBNS8MwGH7r5xTq1cvwSEIwmh3mRdB8OJxgvuArZQ0S7ewNC1JKszSn+LFv+LFgyJ689+Y7kN084HA85GQ932ChDOlHefLWlvf2NzaLu2Udyt7+wf2YaWt4lQS2iIxj2U3wIpyJmhLM81pN5EURwGnWB8XeSdeyoVi8WdniTUi/BQsJARrI3l241+hPUoCLOH3K+jS7SQnUIuxCj3XT+IwOT+XbVqTlToFXizkV5mj69md/EJM0okITjpXquU6ivQxLzQinebmfKpgMsZD2jNU4IgqL5sumKNT4wxQGEtzhEZT9/eLDEdKTaLA3CxmVMtZYf6X9VIdXngZE0mqSCzj8KUIx2joi0YJISzSeGYCKZmRWREZaYaNp2ZTgLq+8Str1muvU3FsHSnAMJ3AGLjTgCm6gCS0g8AjP8Apv1pP1Yr3P6lqz5r0dwR9YH9/2SKFE</latexit><latexit sha1_base64="NjSIBTcqwMj6vsN1L82c4MUo9Gg=">ACFXicbVBNS8MwGH7r5xTq1cvwSEIwmh3mRdB8OJxgvuArZQ0S7ewNC1JKszSn+LFv+LFgyJ689+Y7kN084HA85GQ932ChDOlHefLWlvf2NzaLu2Udyt7+wf2YaWt4lQS2iIxj2U3wIpyJmhLM81pN5EURwGnWB8XeSdeyoVi8WdniTUi/BQsJARrI3l241+hPUoCLOH3K+jS7SQnUIuxCj3XT+IwOT+XbVqTlToFXizkV5mj69md/EJM0okITjpXquU6ivQxLzQinebmfKpgMsZD2jNU4IgqL5sumKNT4wxQGEtzhEZT9/eLDEdKTaLA3CxmVMtZYf6X9VIdXngZE0mqSCzj8KUIx2joi0YJISzSeGYCKZmRWREZaYaNp2ZTgLq+8Str1muvU3FsHSnAMJ3AGLjTgCm6gCS0g8AjP8Apv1pP1Yr3P6lqz5r0dwR9YH9/2SKFE</latexit><latexit sha1_base64="VcGvSFcQISu2RnXhQ1jpP9qdoe8=">ACIHicbVBLS8NAEJ7UV62vqEcvi0UQhJL0Ug8KBS8eK9gHtCVstpt26ebB7kaoIT/Fi3/FiwdF9Ka/xk2birYOLHyPGXbmcyPOpLKsT6Owsrq2vlHcLG1t7+zumfsHLRnGgtAmCXkoOi6WlLOANhVTnHYiQbHvctp2x1eZ376jQrIwuFWTiPZ9PAyYxwhWnLMWs/HauR6yX3qVNElmtN2RudklDo2OvuhrvYcs2xVrGmhZWDnoAx5NRzozcISezTQBGOpezaVqT6CRaKEU7TUi+WNMJkjIe0q2GAfSr7yfTAFJ1oZYC8UOgXKDRVf08k2Jdy4ru6M9tRLnqZ+J/XjZV3k9YEMWKBmT2kRdzpEKUpYUGTFCi+EQDTATuyIywgITpTMt6RDsxZOXQatasa2KfWOV6xd5HEU4gmM4BRtqUIdraEATCDzAE7zAq/FoPBtvxvustWDkM4fwp4yvb+3wosU=</latexit><latexit sha1_base64="YMtT1sm9VIU+T4aPKLOW7IZqw9c=">ACIHicbVBLS8NAEJ74rPUV9ehlsQiCUJIi1INCwYvHCvYBbQmb7aZdunmwuxFqyE/x4l/x4kERvemvcdOmoq0DC9jhp353IgzqSzr01haXldWy9sFDe3tnd2zb39pgxjQWiDhDwUbRdLylAG4opTtuRoNh3OW25o6vMb91RIVkY3KpxRHs+HgTMYwQrLTlmtetjNXS95D51KugSzWgrozMyTB0bnf5QV3uOWbLK1qTQIrBzUIK86o750e2HJPZpoAjHUnZsK1K9BAvFCKdpsRtLGmEywgPa0TDAPpW9ZHJgio610kdeKPQLFJqovycS7Es59l3dme0o571M/M/rxMo7yUsiGJFAzL9yIs5UiHK0kJ9JihRfKwBJoLpXREZYoGJ0pkWdQj2/MmLoFkp21bZvjkr1S7yOApwCEdwAjZUoQbXUIcGEHiAJ3iBV+PReDbejPdp65KRzxzAnzK+vgHvMKLJ</latexit><latexit sha1_base64="YMtT1sm9VIU+T4aPKLOW7IZqw9c=">ACIHicbVBLS8NAEJ74rPUV9ehlsQiCUJIi1INCwYvHCvYBbQmb7aZdunmwuxFqyE/x4l/x4kERvemvcdOmoq0DC9jhp353IgzqSzr01haXldWy9sFDe3tnd2zb39pgxjQWiDhDwUbRdLylAG4opTtuRoNh3OW25o6vMb91RIVkY3KpxRHs+HgTMYwQrLTlmtetjNXS95D51KugSzWgrozMyTB0bnf5QV3uOWbLK1qTQIrBzUIK86o750e2HJPZpoAjHUnZsK1K9BAvFCKdpsRtLGmEywgPa0TDAPpW9ZHJgio610kdeKPQLFJqovycS7Es59l3dme0o571M/M/rxMo7yUsiGJFAzL9yIs5UiHK0kJ9JihRfKwBJoLpXREZYoGJ0pkWdQj2/MmLoFkp21bZvjkr1S7yOApwCEdwAjZUoQbXUIcGEHiAJ3iBV+PReDbejPdp65KRzxzAnzK+vgHvMKLJ</latexit><latexit sha1_base64="YMtT1sm9VIU+T4aPKLOW7IZqw9c=">ACIHicbVBLS8NAEJ74rPUV9ehlsQiCUJIi1INCwYvHCvYBbQmb7aZdunmwuxFqyE/x4l/x4kERvemvcdOmoq0DC9jhp353IgzqSzr01haXldWy9sFDe3tnd2zb39pgxjQWiDhDwUbRdLylAG4opTtuRoNh3OW25o6vMb91RIVkY3KpxRHs+HgTMYwQrLTlmtetjNXS95D51KugSzWgrozMyTB0bnf5QV3uOWbLK1qTQIrBzUIK86o750e2HJPZpoAjHUnZsK1K9BAvFCKdpsRtLGmEywgPa0TDAPpW9ZHJgio610kdeKPQLFJqovycS7Es59l3dme0o571M/M/rxMo7yUsiGJFAzL9yIs5UiHK0kJ9JihRfKwBJoLpXREZYoGJ0pkWdQj2/MmLoFkp21bZvjkr1S7yOApwCEdwAjZUoQbXUIcGEHiAJ3iBV+PReDbejPdp65KRzxzAnzK+vgHvMKLJ</latexit><latexit sha1_base64="YMtT1sm9VIU+T4aPKLOW7IZqw9c=">ACIHicbVBLS8NAEJ74rPUV9ehlsQiCUJIi1INCwYvHCvYBbQmb7aZdunmwuxFqyE/x4l/x4kERvemvcdOmoq0DC9jhp353IgzqSzr01haXldWy9sFDe3tnd2zb39pgxjQWiDhDwUbRdLylAG4opTtuRoNh3OW25o6vMb91RIVkY3KpxRHs+HgTMYwQrLTlmtetjNXS95D51KugSzWgrozMyTB0bnf5QV3uOWbLK1qTQIrBzUIK86o750e2HJPZpoAjHUnZsK1K9BAvFCKdpsRtLGmEywgPa0TDAPpW9ZHJgio610kdeKPQLFJqovycS7Es59l3dme0o571M/M/rxMo7yUsiGJFAzL9yIs5UiHK0kJ9JihRfKwBJoLpXREZYoGJ0pkWdQj2/MmLoFkp21bZvjkr1S7yOApwCEdwAjZUoQbXUIcGEHiAJ3iBV+PReDbejPdp65KRzxzAnzK+vgHvMKLJ</latexit><latexit sha1_base64="YMtT1sm9VIU+T4aPKLOW7IZqw9c=">ACIHicbVBLS8NAEJ74rPUV9ehlsQiCUJIi1INCwYvHCvYBbQmb7aZdunmwuxFqyE/x4l/x4kERvemvcdOmoq0DC9jhp353IgzqSzr01haXldWy9sFDe3tnd2zb39pgxjQWiDhDwUbRdLylAG4opTtuRoNh3OW25o6vMb91RIVkY3KpxRHs+HgTMYwQrLTlmtetjNXS95D51KugSzWgrozMyTB0bnf5QV3uOWbLK1qTQIrBzUIK86o750e2HJPZpoAjHUnZsK1K9BAvFCKdpsRtLGmEywgPa0TDAPpW9ZHJgio610kdeKPQLFJqovycS7Es59l3dme0o571M/M/rxMo7yUsiGJFAzL9yIs5UiHK0kJ9JihRfKwBJoLpXREZYoGJ0pkWdQj2/MmLoFkp21bZvjkr1S7yOApwCEdwAjZUoQbXUIcGEHiAJ3iBV+PReDbejPdp65KRzxzAnzK+vgHvMKLJ</latexit><latexit sha1_base64="YMtT1sm9VIU+T4aPKLOW7IZqw9c=">ACIHicbVBLS8NAEJ74rPUV9ehlsQiCUJIi1INCwYvHCvYBbQmb7aZdunmwuxFqyE/x4l/x4kERvemvcdOmoq0DC9jhp353IgzqSzr01haXldWy9sFDe3tnd2zb39pgxjQWiDhDwUbRdLylAG4opTtuRoNh3OW25o6vMb91RIVkY3KpxRHs+HgTMYwQrLTlmtetjNXS95D51KugSzWgrozMyTB0bnf5QV3uOWbLK1qTQIrBzUIK86o750e2HJPZpoAjHUnZsK1K9BAvFCKdpsRtLGmEywgPa0TDAPpW9ZHJgio610kdeKPQLFJqovycS7Es59l3dme0o571M/M/rxMo7yUsiGJFAzL9yIs5UiHK0kJ9JihRfKwBJoLpXREZYoGJ0pkWdQj2/MmLoFkp21bZvjkr1S7yOApwCEdwAjZUoQbXUIcGEHiAJ3iBV+PReDbejPdp65KRzxzAnzK+vgHvMKLJ</latexit>

∂L ∂b = y − y∗

<latexit sha1_base64="OeW9dqSOYAJeyghKBtzLlQkW/E=">ACGnicbVDNS8MwHE3n15xfVY9egmMgqMVQ8KAy8ePExwH7DWkWbpFpamJUmFUvp3ePFf8eJBEW/ixf/GdCuomw8Cj/fyS97veRGjUlnWl1FaWFxaXimvVtbWNza3zO2dtgxjgUkLhywUXQ9JwignLUVI91IEBR4jHS8WXud+6JkDTktyqJiBugIac+xUhpqW/aji8QTp0ICURg06A1Agjl5n2Y/qZfACJvAIJneHfbNq1a0J4DyxC1IFBZp98MZhDgOCFeYISl7thUpN82fxoxkFSeWJEJ4jIakpylHAZFuOlktgzWtDKAfCn24ghP190SKAimTQAes5cnlrJeL/3m9WPlnbkp5FCvC8fQjP2ZQhTDvCQ6oIFixRBOEBdVZIR4h3ZXSbVZ0CfbsyvOkfVy3rbp9c1JtnBd1lMEe2AcHwAanoAGuQBO0AYP4Am8gFfj0Xg23oz36dWSUczsgj8wPr8BrQGgiQ=</latexit><latexit sha1_base64="OeW9dqSOYAJeyghKBtzLlQkW/E=">ACGnicbVDNS8MwHE3n15xfVY9egmMgqMVQ8KAy8ePExwH7DWkWbpFpamJUmFUvp3ePFf8eJBEW/ixf/GdCuomw8Cj/fyS97veRGjUlnWl1FaWFxaXimvVtbWNza3zO2dtgxjgUkLhywUXQ9JwignLUVI91IEBR4jHS8WXud+6JkDTktyqJiBugIac+xUhpqW/aji8QTp0ICURg06A1Agjl5n2Y/qZfACJvAIJneHfbNq1a0J4DyxC1IFBZp98MZhDgOCFeYISl7thUpN82fxoxkFSeWJEJ4jIakpylHAZFuOlktgzWtDKAfCn24ghP190SKAimTQAes5cnlrJeL/3m9WPlnbkp5FCvC8fQjP2ZQhTDvCQ6oIFixRBOEBdVZIR4h3ZXSbVZ0CfbsyvOkfVy3rbp9c1JtnBd1lMEe2AcHwAanoAGuQBO0AYP4Am8gFfj0Xg23oz36dWSUczsgj8wPr8BrQGgiQ=</latexit><latexit sha1_base64="OeW9dqSOYAJeyghKBtzLlQkW/E=">ACGnicbVDNS8MwHE3n15xfVY9egmMgqMVQ8KAy8ePExwH7DWkWbpFpamJUmFUvp3ePFf8eJBEW/ixf/GdCuomw8Cj/fyS97veRGjUlnWl1FaWFxaXimvVtbWNza3zO2dtgxjgUkLhywUXQ9JwignLUVI91IEBR4jHS8WXud+6JkDTktyqJiBugIac+xUhpqW/aji8QTp0ICURg06A1Agjl5n2Y/qZfACJvAIJneHfbNq1a0J4DyxC1IFBZp98MZhDgOCFeYISl7thUpN82fxoxkFSeWJEJ4jIakpylHAZFuOlktgzWtDKAfCn24ghP190SKAimTQAes5cnlrJeL/3m9WPlnbkp5FCvC8fQjP2ZQhTDvCQ6oIFixRBOEBdVZIR4h3ZXSbVZ0CfbsyvOkfVy3rbp9c1JtnBd1lMEe2AcHwAanoAGuQBO0AYP4Am8gFfj0Xg23oz36dWSUczsgj8wPr8BrQGgiQ=</latexit><latexit sha1_base64="OeW9dqSOYAJeyghKBtzLlQkW/E=">ACGnicbVDNS8MwHE3n15xfVY9egmMgqMVQ8KAy8ePExwH7DWkWbpFpamJUmFUvp3ePFf8eJBEW/ixf/GdCuomw8Cj/fyS97veRGjUlnWl1FaWFxaXimvVtbWNza3zO2dtgxjgUkLhywUXQ9JwignLUVI91IEBR4jHS8WXud+6JkDTktyqJiBugIac+xUhpqW/aji8QTp0ICURg06A1Agjl5n2Y/qZfACJvAIJneHfbNq1a0J4DyxC1IFBZp98MZhDgOCFeYISl7thUpN82fxoxkFSeWJEJ4jIakpylHAZFuOlktgzWtDKAfCn24ghP190SKAimTQAes5cnlrJeL/3m9WPlnbkp5FCvC8fQjP2ZQhTDvCQ6oIFixRBOEBdVZIR4h3ZXSbVZ0CfbsyvOkfVy3rbp9c1JtnBd1lMEe2AcHwAanoAGuQBO0AYP4Am8gFfj0Xg23oz36dWSUczsgj8wPr8BrQGgiQ=</latexit>

∂L ∂w = (y − y∗)h2

<latexit sha1_base64="FwmsphWbHAFt/AP3juDgRkpZY=">ACMXicbVDLSsNAFJ3UV62vqEs3g0WogiUpgi4UCm6cFHBPqCJYTKdtEMnD2YmSgj5JTf+ibjpQhG3/oSTtoK2HrhwOde7r3HjRgV0jDGWmFpeWV1rbhe2tjc2t7Rd/faIow5Ji0cspB3XSQIowFpSoZ6UacIN9lpOrnO/80C4oGFwJ5OI2D4aBNSjGEklOXrD8jCqRUhLili0PKRHGLE0psm1NdL3MngFKwk8hcn9yfGPMycmqOXjaoxAVwk5oyUwQxNR3+x+iGOfRJIzJAQPdOIpJ3mGzEjWcmKBYkQHqEB6SkaIJ8IO518nMEjpfShF3JVgYQT9fdEinwhEt9VnfmNYt7Lxf+8Xiy9CzulQRLEuDpIi9mUIYwjw/2KSdYskQRhDlVt0I8RCpCqUIuqRDM+ZcXSbtWNY2qeXtWrl/O4iCA3AIKsAE56AOGqAJWgCDJ/AK3sC79qyNtQ/tc9pa0GYz+APtK9voNyqWg=</latexit><latexit sha1_base64="FwmsphWbHAFt/AP3juDgRkpZY=">ACMXicbVDLSsNAFJ3UV62vqEs3g0WogiUpgi4UCm6cFHBPqCJYTKdtEMnD2YmSgj5JTf+ibjpQhG3/oSTtoK2HrhwOde7r3HjRgV0jDGWmFpeWV1rbhe2tjc2t7Rd/faIow5Ji0cspB3XSQIowFpSoZ6UacIN9lpOrnO/80C4oGFwJ5OI2D4aBNSjGEklOXrD8jCqRUhLili0PKRHGLE0psm1NdL3MngFKwk8hcn9yfGPMycmqOXjaoxAVwk5oyUwQxNR3+x+iGOfRJIzJAQPdOIpJ3mGzEjWcmKBYkQHqEB6SkaIJ8IO518nMEjpfShF3JVgYQT9fdEinwhEt9VnfmNYt7Lxf+8Xiy9CzulQRLEuDpIi9mUIYwjw/2KSdYskQRhDlVt0I8RCpCqUIuqRDM+ZcXSbtWNY2qeXtWrl/O4iCA3AIKsAE56AOGqAJWgCDJ/AK3sC79qyNtQ/tc9pa0GYz+APtK9voNyqWg=</latexit><latexit sha1_base64="FwmsphWbHAFt/AP3juDgRkpZY=">ACMXicbVDLSsNAFJ3UV62vqEs3g0WogiUpgi4UCm6cFHBPqCJYTKdtEMnD2YmSgj5JTf+ibjpQhG3/oSTtoK2HrhwOde7r3HjRgV0jDGWmFpeWV1rbhe2tjc2t7Rd/faIow5Ji0cspB3XSQIowFpSoZ6UacIN9lpOrnO/80C4oGFwJ5OI2D4aBNSjGEklOXrD8jCqRUhLili0PKRHGLE0psm1NdL3MngFKwk8hcn9yfGPMycmqOXjaoxAVwk5oyUwQxNR3+x+iGOfRJIzJAQPdOIpJ3mGzEjWcmKBYkQHqEB6SkaIJ8IO518nMEjpfShF3JVgYQT9fdEinwhEt9VnfmNYt7Lxf+8Xiy9CzulQRLEuDpIi9mUIYwjw/2KSdYskQRhDlVt0I8RCpCqUIuqRDM+ZcXSbtWNY2qeXtWrl/O4iCA3AIKsAE56AOGqAJWgCDJ/AK3sC79qyNtQ/tc9pa0GYz+APtK9voNyqWg=</latexit><latexit sha1_base64="FwmsphWbHAFt/AP3juDgRkpZY=">ACMXicbVDLSsNAFJ3UV62vqEs3g0WogiUpgi4UCm6cFHBPqCJYTKdtEMnD2YmSgj5JTf+ibjpQhG3/oSTtoK2HrhwOde7r3HjRgV0jDGWmFpeWV1rbhe2tjc2t7Rd/faIow5Ji0cspB3XSQIowFpSoZ6UacIN9lpOrnO/80C4oGFwJ5OI2D4aBNSjGEklOXrD8jCqRUhLili0PKRHGLE0psm1NdL3MngFKwk8hcn9yfGPMycmqOXjaoxAVwk5oyUwQxNR3+x+iGOfRJIzJAQPdOIpJ3mGzEjWcmKBYkQHqEB6SkaIJ8IO518nMEjpfShF3JVgYQT9fdEinwhEt9VnfmNYt7Lxf+8Xiy9CzulQRLEuDpIi9mUIYwjw/2KSdYskQRhDlVt0I8RCpCqUIuqRDM+ZcXSbtWNY2qeXtWrl/O4iCA3AIKsAE56AOGqAJWgCDJ/AK3sC79qyNtQ/tc9pa0GYz+APtK9voNyqWg=</latexit>

∂L ∂h2 = (y − y∗)w

<latexit sha1_base64="FIVqpGV4oIa+vyctm32ba/uctwA=">ACMXicbVDLSsNAFJ3UV62vqEs3g0WogiUpgi4UCm6cFHBPqCJYTKdtEMnD2YmSgj5JTf+ibjpQhG3/oSTtoK2HrhwOde7r3HjRgV0jDGWmFpeWV1rbhe2tjc2t7Rd/faIow5Ji0cspB3XSQIowFpSoZ6UacIN9lpOrnO/80C4oGFwJ5OI2D4aBNSjGEklOXrD8jCqRUhLili0PKRHGLE0psm1NdLx1mTi2DV7CSwFOY3J8c/xiPmaOXjaoxAVwk5oyUwQxNR3+x+iGOfRJIzJAQPdOIpJ3mGzEjWcmKBYkQHqEB6SkaIJ8IO518nMEjpfShF3JVgYQT9fdEinwhEt9VnfmFYt7Lxf+8Xiy9CzulQRLEuDpIi9mUIYwjw/2KSdYskQRhDlVt0I8RCpCqUIuqRDM+ZcXSbtWNY2qeXtWrl/O4iCA3AIKsAE56AOGqAJWgCDJ/AK3sC79qyNtQ/tc9pa0GYz+APtK9vnayqWg=</latexit><latexit sha1_base64="FIVqpGV4oIa+vyctm32ba/uctwA=">ACMXicbVDLSsNAFJ3UV62vqEs3g0WogiUpgi4UCm6cFHBPqCJYTKdtEMnD2YmSgj5JTf+ibjpQhG3/oSTtoK2HrhwOde7r3HjRgV0jDGWmFpeWV1rbhe2tjc2t7Rd/faIow5Ji0cspB3XSQIowFpSoZ6UacIN9lpOrnO/80C4oGFwJ5OI2D4aBNSjGEklOXrD8jCqRUhLili0PKRHGLE0psm1NdLx1mTi2DV7CSwFOY3J8c/xiPmaOXjaoxAVwk5oyUwQxNR3+x+iGOfRJIzJAQPdOIpJ3mGzEjWcmKBYkQHqEB6SkaIJ8IO518nMEjpfShF3JVgYQT9fdEinwhEt9VnfmFYt7Lxf+8Xiy9CzulQRLEuDpIi9mUIYwjw/2KSdYskQRhDlVt0I8RCpCqUIuqRDM+ZcXSbtWNY2qeXtWrl/O4iCA3AIKsAE56AOGqAJWgCDJ/AK3sC79qyNtQ/tc9pa0GYz+APtK9vnayqWg=</latexit><latexit sha1_base64="FIVqpGV4oIa+vyctm32ba/uctwA=">ACMXicbVDLSsNAFJ3UV62vqEs3g0WogiUpgi4UCm6cFHBPqCJYTKdtEMnD2YmSgj5JTf+ibjpQhG3/oSTtoK2HrhwOde7r3HjRgV0jDGWmFpeWV1rbhe2tjc2t7Rd/faIow5Ji0cspB3XSQIowFpSoZ6UacIN9lpOrnO/80C4oGFwJ5OI2D4aBNSjGEklOXrD8jCqRUhLili0PKRHGLE0psm1NdLx1mTi2DV7CSwFOY3J8c/xiPmaOXjaoxAVwk5oyUwQxNR3+x+iGOfRJIzJAQPdOIpJ3mGzEjWcmKBYkQHqEB6SkaIJ8IO518nMEjpfShF3JVgYQT9fdEinwhEt9VnfmFYt7Lxf+8Xiy9CzulQRLEuDpIi9mUIYwjw/2KSdYskQRhDlVt0I8RCpCqUIuqRDM+ZcXSbtWNY2qeXtWrl/O4iCA3AIKsAE56AOGqAJWgCDJ/AK3sC79qyNtQ/tc9pa0GYz+APtK9vnayqWg=</latexit><latexit sha1_base64="FIVqpGV4oIa+vyctm32ba/uctwA=">ACMXicbVDLSsNAFJ3UV62vqEs3g0WogiUpgi4UCm6cFHBPqCJYTKdtEMnD2YmSgj5JTf+ibjpQhG3/oSTtoK2HrhwOde7r3HjRgV0jDGWmFpeWV1rbhe2tjc2t7Rd/faIow5Ji0cspB3XSQIowFpSoZ6UacIN9lpOrnO/80C4oGFwJ5OI2D4aBNSjGEklOXrD8jCqRUhLili0PKRHGLE0psm1NdLx1mTi2DV7CSwFOY3J8c/xiPmaOXjaoxAVwk5oyUwQxNR3+x+iGOfRJIzJAQPdOIpJ3mGzEjWcmKBYkQHqEB6SkaIJ8IO518nMEjpfShF3JVgYQT9fdEinwhEt9VnfmFYt7Lxf+8Xiy9CzulQRLEuDpIi9mUIYwjw/2KSdYskQRhDlVt0I8RCpCqUIuqRDM+ZcXSbtWNY2qeXtWrl/O4iCA3AIKsAE56AOGqAJWgCDJ/AK3sC79qyNtQ/tc9pa0GYz+APtK9vnayqWg=</latexit>

∂L ∂z2 = (1 h2

2) ∂L

∂h2

<latexit sha1_base64="eTRSpcQfIiLmPuLZMD2ywN71CtY=">ACV3icjVHPS8MwGE27Oef8tenRS3AI8+Boh6AHhYEXDx4mODdYa0mzdAtL05Kkwiz9J8XL/hUvm7DH5sHwQe73fly8vfsyoVJY1M8xCcaO0Wd6qbO/s7u1XawePMkoEJl0csUj0fSQJo5x0FVWM9GNBUOgz0vMnN7nfeyZC0og/qGlM3BCNOA0oRkpLXpU7gUA4dWIkFEUM3mXf3AmRGvtB+pJ5rQxew4YNz7EsRafWqfQwVRg+I8peUPmVetW05oDrhN7SepgiY5XfXWGEU5CwhVmSMqBbcXKTfPZmJGs4iSxAhP0IgMNOUoJNJN57lk8EQrQxhEQh+u4Fz92ZGiUMp6OvKfEm56uXiX94gUcGlm1IeJ4pwvLgoSBhUEcxDhkMqCFZsqgnCgupdIR4jHZHSX1HRIdirT14nj62mbTXt+/N6+2oZRxkcgWPQADa4AG1wCzqgCzB4A+9GwSgaM+PDLJnlRalpLHsOwS+YtU8eBLSK</latexit><latexit sha1_base64="eTRSpcQfIiLmPuLZMD2ywN71CtY=">ACV3icjVHPS8MwGE27Oef8tenRS3AI8+Boh6AHhYEXDx4mODdYa0mzdAtL05Kkwiz9J8XL/hUvm7DH5sHwQe73fly8vfsyoVJY1M8xCcaO0Wd6qbO/s7u1XawePMkoEJl0csUj0fSQJo5x0FVWM9GNBUOgz0vMnN7nfeyZC0og/qGlM3BCNOA0oRkpLXpU7gUA4dWIkFEUM3mXf3AmRGvtB+pJ5rQxew4YNz7EsRafWqfQwVRg+I8peUPmVetW05oDrhN7SepgiY5XfXWGEU5CwhVmSMqBbcXKTfPZmJGs4iSxAhP0IgMNOUoJNJN57lk8EQrQxhEQh+u4Fz92ZGiUMp6OvKfEm56uXiX94gUcGlm1IeJ4pwvLgoSBhUEcxDhkMqCFZsqgnCgupdIR4jHZHSX1HRIdirT14nj62mbTXt+/N6+2oZRxkcgWPQADa4AG1wCzqgCzB4A+9GwSgaM+PDLJnlRalpLHsOwS+YtU8eBLSK</latexit><latexit sha1_base64="eTRSpcQfIiLmPuLZMD2ywN71CtY=">ACV3icjVHPS8MwGE27Oef8tenRS3AI8+Boh6AHhYEXDx4mODdYa0mzdAtL05Kkwiz9J8XL/hUvm7DH5sHwQe73fly8vfsyoVJY1M8xCcaO0Wd6qbO/s7u1XawePMkoEJl0csUj0fSQJo5x0FVWM9GNBUOgz0vMnN7nfeyZC0og/qGlM3BCNOA0oRkpLXpU7gUA4dWIkFEUM3mXf3AmRGvtB+pJ5rQxew4YNz7EsRafWqfQwVRg+I8peUPmVetW05oDrhN7SepgiY5XfXWGEU5CwhVmSMqBbcXKTfPZmJGs4iSxAhP0IgMNOUoJNJN57lk8EQrQxhEQh+u4Fz92ZGiUMp6OvKfEm56uXiX94gUcGlm1IeJ4pwvLgoSBhUEcxDhkMqCFZsqgnCgupdIR4jHZHSX1HRIdirT14nj62mbTXt+/N6+2oZRxkcgWPQADa4AG1wCzqgCzB4A+9GwSgaM+PDLJnlRalpLHsOwS+YtU8eBLSK</latexit><latexit sha1_base64="eTRSpcQfIiLmPuLZMD2ywN71CtY=">ACV3icjVHPS8MwGE27Oef8tenRS3AI8+Boh6AHhYEXDx4mODdYa0mzdAtL05Kkwiz9J8XL/hUvm7DH5sHwQe73fly8vfsyoVJY1M8xCcaO0Wd6qbO/s7u1XawePMkoEJl0csUj0fSQJo5x0FVWM9GNBUOgz0vMnN7nfeyZC0og/qGlM3BCNOA0oRkpLXpU7gUA4dWIkFEUM3mXf3AmRGvtB+pJ5rQxew4YNz7EsRafWqfQwVRg+I8peUPmVetW05oDrhN7SepgiY5XfXWGEU5CwhVmSMqBbcXKTfPZmJGs4iSxAhP0IgMNOUoJNJN57lk8EQrQxhEQh+u4Fz92ZGiUMp6OvKfEm56uXiX94gUcGlm1IeJ4pwvLgoSBhUEcxDhkMqCFZsqgnCgupdIR4jHZHSX1HRIdirT14nj62mbTXt+/N6+2oZRxkcgWPQADa4AG1wCzqgCzB4A+9GwSgaM+PDLJnlRalpLHsOwS+YtU8eBLSK</latexit>

∂L ∂W2 = ∂L ∂z2 h|

1

<latexit sha1_base64="F16bXBCOBpkl5X7y7dfxl5YKis=">ACVHicjVBNS8NAFNykVmvVGvXoZbEInkpSBD0oFLx48FDBfkATw2a7aZduNmF3I9SQH6kHwV/ixYObNuJHPTiwMG/ePN7bCRJGpbLtV8OsrFXN2qb9a3tncautbfl3EqMOnhmMViGCBJGOWkp6hiZJgIgqKAkUEwuyr6gwciJI35nZonxIvQhNOQYqS05FszNxQIZ26ChKIwZv8i7sRUtMgzAa5387hJfyH9bGwfhbT/D5zKVdEYMRy3/Gtpt2yF4CrxClJE5To+tazO45xGhGuMENSjhw7UV5W7MSM5HU3lSRBeIYmZKQpRxGRXrYIJYfHWhnDMBb6cQUX6veJDEVSzqNAO4t75e9eIf7VG6UqPcypNUEY6Xi8KUQRXDImE4poJgxeaICyovhXiKdLR6SBkXYfg/P7yKum3W47dcm5Pm52LMo4aOARH4AQ4Ax0wDXogh7A4Am8GcAwjBfj3ayY1aXVNMqZA/ADZuMDeZS1og=</latexit><latexit sha1_base64="F16bXBCOBpkl5X7y7dfxl5YKis=">ACVHicjVBNS8NAFNykVmvVGvXoZbEInkpSBD0oFLx48FDBfkATw2a7aZduNmF3I9SQH6kHwV/ixYObNuJHPTiwMG/ePN7bCRJGpbLtV8OsrFXN2qb9a3tncautbfl3EqMOnhmMViGCBJGOWkp6hiZJgIgqKAkUEwuyr6gwciJI35nZonxIvQhNOQYqS05FszNxQIZ26ChKIwZv8i7sRUtMgzAa5387hJfyH9bGwfhbT/D5zKVdEYMRy3/Gtpt2yF4CrxClJE5To+tazO45xGhGuMENSjhw7UV5W7MSM5HU3lSRBeIYmZKQpRxGRXrYIJYfHWhnDMBb6cQUX6veJDEVSzqNAO4t75e9eIf7VG6UqPcypNUEY6Xi8KUQRXDImE4poJgxeaICyovhXiKdLR6SBkXYfg/P7yKum3W47dcm5Pm52LMo4aOARH4AQ4Ax0wDXogh7A4Am8GcAwjBfj3ayY1aXVNMqZA/ADZuMDeZS1og=</latexit><latexit sha1_base64="F16bXBCOBpkl5X7y7dfxl5YKis=">ACVHicjVBNS8NAFNykVmvVGvXoZbEInkpSBD0oFLx48FDBfkATw2a7aZduNmF3I9SQH6kHwV/ixYObNuJHPTiwMG/ePN7bCRJGpbLtV8OsrFXN2qb9a3tncautbfl3EqMOnhmMViGCBJGOWkp6hiZJgIgqKAkUEwuyr6gwciJI35nZonxIvQhNOQYqS05FszNxQIZ26ChKIwZv8i7sRUtMgzAa5387hJfyH9bGwfhbT/D5zKVdEYMRy3/Gtpt2yF4CrxClJE5To+tazO45xGhGuMENSjhw7UV5W7MSM5HU3lSRBeIYmZKQpRxGRXrYIJYfHWhnDMBb6cQUX6veJDEVSzqNAO4t75e9eIf7VG6UqPcypNUEY6Xi8KUQRXDImE4poJgxeaICyovhXiKdLR6SBkXYfg/P7yKum3W47dcm5Pm52LMo4aOARH4AQ4Ax0wDXogh7A4Am8GcAwjBfj3ayY1aXVNMqZA/ADZuMDeZS1og=</latexit><latexit sha1_base64="F16bXBCOBpkl5X7y7dfxl5YKis=">ACVHicjVBNS8NAFNykVmvVGvXoZbEInkpSBD0oFLx48FDBfkATw2a7aZduNmF3I9SQH6kHwV/ixYObNuJHPTiwMG/ePN7bCRJGpbLtV8OsrFXN2qb9a3tncautbfl3EqMOnhmMViGCBJGOWkp6hiZJgIgqKAkUEwuyr6gwciJI35nZonxIvQhNOQYqS05FszNxQIZ26ChKIwZv8i7sRUtMgzAa5387hJfyH9bGwfhbT/D5zKVdEYMRy3/Gtpt2yF4CrxClJE5To+tazO45xGhGuMENSjhw7UV5W7MSM5HU3lSRBeIYmZKQpRxGRXrYIJYfHWhnDMBb6cQUX6veJDEVSzqNAO4t75e9eIf7VG6UqPcypNUEY6Xi8KUQRXDImE4poJgxeaICyovhXiKdLR6SBkXYfg/P7yKum3W47dcm5Pm52LMo4aOARH4AQ4Ax0wDXogh7A4Am8GcAwjBfj3ayY1aXVNMqZA/ADZuMDeZS1og=</latexit>

∂L ∂b2 = ∂L ∂z2

<latexit sha1_base64="Ps9aen9kvmpbSPtReCBx7tXpD9g=">ACPHicjVDLSsNAFJ3UV62vqEs3g0VwVZIi6EKh4MaFi4r2AU0Ik+mkHTqZhJmJUEM+zI0f4c6VGxeKuHXtpA2orQsPDBzOPfOvcePGZXKsp6M0sLi0vJKebWytr6xuWVu7RlAhMWjhikej6SBJGOWkpqhjpxoKg0Gek4/O83rnlghJI36jxjFxQzTgNKAYKS15rUTCIRTJ0ZCUcTgZfbNnRCpoR+kfubVM3gG/2G9y62eWbVq1gRwntgFqYICTc98dPoRTkLCFWZIyp5txcpN89mYkaziJLECI/QgPQ05Sgk0k0nx2fwQCt9GERCP67gRP3ZkaJQynHoa2e+pJyt5eJftV6ighM3pTxOFOF4+lGQMKgimCcJ+1QrNhYE4QF1btCPEQ6IqXzrugQ7NmT50m7XrOtmn1VG2cFnGUwR7YB4fABsegAS5AE7QABvfgGbyCN+PBeDHejY+ptWQUPbvgF4zPL1yYr/Y=</latexit><latexit sha1_base64="Ps9aen9kvmpbSPtReCBx7tXpD9g=">ACPHicjVDLSsNAFJ3UV62vqEs3g0VwVZIi6EKh4MaFi4r2AU0Ik+mkHTqZhJmJUEM+zI0f4c6VGxeKuHXtpA2orQsPDBzOPfOvcePGZXKsp6M0sLi0vJKebWytr6xuWVu7RlAhMWjhikej6SBJGOWkpqhjpxoKg0Gek4/O83rnlghJI36jxjFxQzTgNKAYKS15rUTCIRTJ0ZCUcTgZfbNnRCpoR+kfubVM3gG/2G9y62eWbVq1gRwntgFqYICTc98dPoRTkLCFWZIyp5txcpN89mYkaziJLECI/QgPQ05Sgk0k0nx2fwQCt9GERCP67gRP3ZkaJQynHoa2e+pJyt5eJftV6ighM3pTxOFOF4+lGQMKgimCcJ+1QrNhYE4QF1btCPEQ6IqXzrugQ7NmT50m7XrOtmn1VG2cFnGUwR7YB4fABsegAS5AE7QABvfgGbyCN+PBeDHejY+ptWQUPbvgF4zPL1yYr/Y=</latexit><latexit sha1_base64="Ps9aen9kvmpbSPtReCBx7tXpD9g=">ACPHicjVDLSsNAFJ3UV62vqEs3g0VwVZIi6EKh4MaFi4r2AU0Ik+mkHTqZhJmJUEM+zI0f4c6VGxeKuHXtpA2orQsPDBzOPfOvcePGZXKsp6M0sLi0vJKebWytr6xuWVu7RlAhMWjhikej6SBJGOWkpqhjpxoKg0Gek4/O83rnlghJI36jxjFxQzTgNKAYKS15rUTCIRTJ0ZCUcTgZfbNnRCpoR+kfubVM3gG/2G9y62eWbVq1gRwntgFqYICTc98dPoRTkLCFWZIyp5txcpN89mYkaziJLECI/QgPQ05Sgk0k0nx2fwQCt9GERCP67gRP3ZkaJQynHoa2e+pJyt5eJftV6ighM3pTxOFOF4+lGQMKgimCcJ+1QrNhYE4QF1btCPEQ6IqXzrugQ7NmT50m7XrOtmn1VG2cFnGUwR7YB4fABsegAS5AE7QABvfgGbyCN+PBeDHejY+ptWQUPbvgF4zPL1yYr/Y=</latexit><latexit sha1_base64="Ps9aen9kvmpbSPtReCBx7tXpD9g=">ACPHicjVDLSsNAFJ3UV62vqEs3g0VwVZIi6EKh4MaFi4r2AU0Ik+mkHTqZhJmJUEM+zI0f4c6VGxeKuHXtpA2orQsPDBzOPfOvcePGZXKsp6M0sLi0vJKebWytr6xuWVu7RlAhMWjhikej6SBJGOWkpqhjpxoKg0Gek4/O83rnlghJI36jxjFxQzTgNKAYKS15rUTCIRTJ0ZCUcTgZfbNnRCpoR+kfubVM3gG/2G9y62eWbVq1gRwntgFqYICTc98dPoRTkLCFWZIyp5txcpN89mYkaziJLECI/QgPQ05Sgk0k0nx2fwQCt9GERCP67gRP3ZkaJQynHoa2e+pJyt5eJftV6ighM3pTxOFOF4+lGQMKgimCcJ+1QrNhYE4QF1btCPEQ6IqXzrugQ7NmT50m7XrOtmn1VG2cFnGUwR7YB4fABsegAS5AE7QABvfgGbyCN+PBeDHejY+ptWQUPbvgF4zPL1yYr/Y=</latexit>

∂L ∂h1 = W|

2

∂L ∂z2

<latexit sha1_base64="VK7liKpYusHGqowtBJbNBboYhLI=">ACVHicjVDPS8MwGE07p3PqrHr0EhyCp9EOQ8KAy8ePExwP2CtJc3SLSxNS5IKs/SP1IPgX+LFg+lW/DUPfhB4e9X768IGFUKt+NczKWnV9o7Z39reaexae/t9GacCkx6OWSyGAZKEU56ipGhokgKAoYGQSzq0IfPBAhaczv1DwhXoQmnIYUI6Up35q5oUA4cxMkFEUM3uRf2I2QmgZhNs19J4eXn/dBfp+5lCsiMGK53/7HiEdty32rabfsRcFV4JSgCcrq+tazO45xGhGuMENSjhw7UV5WzMaM5HU3lSRBeIYmZKQhRxGRXrYIJYfHmhnDMBb6cAUX7PeODEVSzqNAO4sl5W+tIP/SRqkKz72M8iRVhOPlQ2HKoIphkTAcU0GwYnMNEBZU7wrxFOmIdGCyrkNwfn95FfTbLcduObenzc5FGUcNHIjcAIcAY64Bp0Q9g8ATeDGAYxovxblbM6tJqGmXPAfhRZuMDhLC1og=</latexit><latexit sha1_base64="VK7liKpYusHGqowtBJbNBboYhLI=">ACVHicjVDPS8MwGE07p3PqrHr0EhyCp9EOQ8KAy8ePExwP2CtJc3SLSxNS5IKs/SP1IPgX+LFg+lW/DUPfhB4e9X768IGFUKt+NczKWnV9o7Z39reaexae/t9GacCkx6OWSyGAZKEU56ipGhokgKAoYGQSzq0IfPBAhaczv1DwhXoQmnIYUI6Up35q5oUA4cxMkFEUM3uRf2I2QmgZhNs19J4eXn/dBfp+5lCsiMGK53/7HiEdty32rabfsRcFV4JSgCcrq+tazO45xGhGuMENSjhw7UV5WzMaM5HU3lSRBeIYmZKQhRxGRXrYIJYfHmhnDMBb6cAUX7PeODEVSzqNAO4sl5W+tIP/SRqkKz72M8iRVhOPlQ2HKoIphkTAcU0GwYnMNEBZU7wrxFOmIdGCyrkNwfn95FfTbLcduObenzc5FGUcNHIjcAIcAY64Bp0Q9g8ATeDGAYxovxblbM6tJqGmXPAfhRZuMDhLC1og=</latexit><latexit sha1_base64="VK7liKpYusHGqowtBJbNBboYhLI=">ACVHicjVDPS8MwGE07p3PqrHr0EhyCp9EOQ8KAy8ePExwP2CtJc3SLSxNS5IKs/SP1IPgX+LFg+lW/DUPfhB4e9X768IGFUKt+NczKWnV9o7Z39reaexae/t9GacCkx6OWSyGAZKEU56ipGhokgKAoYGQSzq0IfPBAhaczv1DwhXoQmnIYUI6Up35q5oUA4cxMkFEUM3uRf2I2QmgZhNs19J4eXn/dBfp+5lCsiMGK53/7HiEdty32rabfsRcFV4JSgCcrq+tazO45xGhGuMENSjhw7UV5WzMaM5HU3lSRBeIYmZKQhRxGRXrYIJYfHmhnDMBb6cAUX7PeODEVSzqNAO4sl5W+tIP/SRqkKz72M8iRVhOPlQ2HKoIphkTAcU0GwYnMNEBZU7wrxFOmIdGCyrkNwfn95FfTbLcduObenzc5FGUcNHIjcAIcAY64Bp0Q9g8ATeDGAYxovxblbM6tJqGmXPAfhRZuMDhLC1og=</latexit><latexit sha1_base64="VK7liKpYusHGqowtBJbNBboYhLI=">ACVHicjVDPS8MwGE07p3PqrHr0EhyCp9EOQ8KAy8ePExwP2CtJc3SLSxNS5IKs/SP1IPgX+LFg+lW/DUPfhB4e9X768IGFUKt+NczKWnV9o7Z39reaexae/t9GacCkx6OWSyGAZKEU56ipGhokgKAoYGQSzq0IfPBAhaczv1DwhXoQmnIYUI6Up35q5oUA4cxMkFEUM3uRf2I2QmgZhNs19J4eXn/dBfp+5lCsiMGK53/7HiEdty32rabfsRcFV4JSgCcrq+tazO45xGhGuMENSjhw7UV5WzMaM5HU3lSRBeIYmZKQhRxGRXrYIJYfHmhnDMBb6cAUX7PeODEVSzqNAO4sl5W+tIP/SRqkKz72M8iRVhOPlQ2HKoIphkTAcU0GwYnMNEBZU7wrxFOmIdGCyrkNwfn95FfTbLcduObenzc5FGUcNHIjcAIcAY64Bp0Q9g8ATeDGAYxovxblbM6tJqGmXPAfhRZuMDhLC1og=</latexit>

∂L ∂z1 = (1 h2

1) ∂L

∂h1

<latexit sha1_base64="xm4kFiqB1OrBbsQMbs80K108vI=">ACWXicjVHLSsNAFJ1EqzW+ol26GSxCXViSIuhCoeDGhYsK9gFNDJPpB06eTAzEWrIT7oQxF9x4aQNPloXhg4nHPvmTtn/IRIS3rTdPX1isbm9UtY3tnd2/fPDjsiTjlmHRxzGI+8JEgjEakK6lkZJBwgkKfkb4/vSn0/hPhgsbRg5wlxA3ROKIBxUgqyjMTJ+AIZ06CuKSIwbv8GzshkhM/yJ5z87hNWzY8OyLnOSPWUsJp9DBlGP4D59J4eOZdatpzQuArsEdVBWxzNfnFGM05BEjMkxNC2EulmhTdmJDecVJAE4Skak6GCEQqJcLN5Mjk8UcwIBjFXJ5Jwzv6cyFAoxCz0VWexpFjWCvIvbZjK4NLNaJSkR4cVGQMihjWMQMR5QTLNlMAYQ5VbtCPEqIqk+w1Ah2MtPXgW9VtO2mvb9eb19VcZRBUfgGDSADS5AG9yCDugCDF7Bh1bRNrR3XdOrurFo1bVypgZ+lV7BTAtJQ=</latexit><latexit sha1_base64="xm4kFiqB1OrBbsQMbs80K108vI=">ACWXicjVHLSsNAFJ1EqzW+ol26GSxCXViSIuhCoeDGhYsK9gFNDJPpB06eTAzEWrIT7oQxF9x4aQNPloXhg4nHPvmTtn/IRIS3rTdPX1isbm9UtY3tnd2/fPDjsiTjlmHRxzGI+8JEgjEakK6lkZJBwgkKfkb4/vSn0/hPhgsbRg5wlxA3ROKIBxUgqyjMTJ+AIZ06CuKSIwbv8GzshkhM/yJ5z87hNWzY8OyLnOSPWUsJp9DBlGP4D59J4eOZdatpzQuArsEdVBWxzNfnFGM05BEjMkxNC2EulmhTdmJDecVJAE4Skak6GCEQqJcLN5Mjk8UcwIBjFXJ5Jwzv6cyFAoxCz0VWexpFjWCvIvbZjK4NLNaJSkR4cVGQMihjWMQMR5QTLNlMAYQ5VbtCPEqIqk+w1Ah2MtPXgW9VtO2mvb9eb19VcZRBUfgGDSADS5AG9yCDugCDF7Bh1bRNrR3XdOrurFo1bVypgZ+lV7BTAtJQ=</latexit><latexit sha1_base64="xm4kFiqB1OrBbsQMbs80K108vI=">ACWXicjVHLSsNAFJ1EqzW+ol26GSxCXViSIuhCoeDGhYsK9gFNDJPpB06eTAzEWrIT7oQxF9x4aQNPloXhg4nHPvmTtn/IRIS3rTdPX1isbm9UtY3tnd2/fPDjsiTjlmHRxzGI+8JEgjEakK6lkZJBwgkKfkb4/vSn0/hPhgsbRg5wlxA3ROKIBxUgqyjMTJ+AIZ06CuKSIwbv8GzshkhM/yJ5z87hNWzY8OyLnOSPWUsJp9DBlGP4D59J4eOZdatpzQuArsEdVBWxzNfnFGM05BEjMkxNC2EulmhTdmJDecVJAE4Skak6GCEQqJcLN5Mjk8UcwIBjFXJ5Jwzv6cyFAoxCz0VWexpFjWCvIvbZjK4NLNaJSkR4cVGQMihjWMQMR5QTLNlMAYQ5VbtCPEqIqk+w1Ah2MtPXgW9VtO2mvb9eb19VcZRBUfgGDSADS5AG9yCDugCDF7Bh1bRNrR3XdOrurFo1bVypgZ+lV7BTAtJQ=</latexit><latexit sha1_base64="xm4kFiqB1OrBbsQMbs80K108vI=">ACWXicjVHLSsNAFJ1EqzW+ol26GSxCXViSIuhCoeDGhYsK9gFNDJPpB06eTAzEWrIT7oQxF9x4aQNPloXhg4nHPvmTtn/IRIS3rTdPX1isbm9UtY3tnd2/fPDjsiTjlmHRxzGI+8JEgjEakK6lkZJBwgkKfkb4/vSn0/hPhgsbRg5wlxA3ROKIBxUgqyjMTJ+AIZ06CuKSIwbv8GzshkhM/yJ5z87hNWzY8OyLnOSPWUsJp9DBlGP4D59J4eOZdatpzQuArsEdVBWxzNfnFGM05BEjMkxNC2EulmhTdmJDecVJAE4Skak6GCEQqJcLN5Mjk8UcwIBjFXJ5Jwzv6cyFAoxCz0VWexpFjWCvIvbZjK4NLNaJSkR4cVGQMihjWMQMR5QTLNlMAYQ5VbtCPEqIqk+w1Ah2MtPXgW9VtO2mvb9eb19VcZRBUfgGDSADS5AG9yCDugCDF7Bh1bRNrR3XdOrurFo1bVypgZ+lV7BTAtJQ=</latexit>

∂L ∂W1 = ∂L ∂z1 x|

<latexit sha1_base64="k8gteobUOWK/9RiKcoVzY2LBP8=">ACZ3iclVDLSgMxFM2Mr1ofHa2I4CZaBFdlRgRdKBTcuHBRwT6gU0smzbShmcyQZMQ6xI90596Nf2GmLaKtGw8Ezj3Hu7NCRJGpXLd8teWl5ZXSusFzc2t7ZLzs5uU8apwKSBYxaLdoAkYZSThqKkXYiCIoCRlrB6Cbvt56IkDTmD2qckG6EBpyGFCNlpJ7z6ocC4cxPkFAUMehHSA0xYtmd1nNqEGYt3fM0vIb/Mr0Yj/6unvVj5lOuiDAG3XMqbtWdAC4Sb0YqYIZ6z3nz+zFOI8IVZkjKjucmqpvlKzEjuinkiQIj9CAdAzlKCKym01y0vDEKH0YxsI8ruBE/enIUCTlOArMZH6tnO/l4l+9TqrCy25GeZIqwvF0UZgyqGKYhw7VBCs2NgQhAU1t0I8RCZDE4MsmhC8+S8vkuZ1XOr3v15pXY1i6MADsExOAUeuA1cAvqoAEw+LCKVtnasz7tkr1vH0xHbWvmKYNfsI+AIm3vFw=</latexit><latexit sha1_base64="k8gteobUOWK/9RiKcoVzY2LBP8=">ACZ3iclVDLSgMxFM2Mr1ofHa2I4CZaBFdlRgRdKBTcuHBRwT6gU0smzbShmcyQZMQ6xI90596Nf2GmLaKtGw8Ezj3Hu7NCRJGpXLd8teWl5ZXSusFzc2t7ZLzs5uU8apwKSBYxaLdoAkYZSThqKkXYiCIoCRlrB6Cbvt56IkDTmD2qckG6EBpyGFCNlpJ7z6ocC4cxPkFAUMehHSA0xYtmd1nNqEGYt3fM0vIb/Mr0Yj/6unvVj5lOuiDAG3XMqbtWdAC4Sb0YqYIZ6z3nz+zFOI8IVZkjKjucmqpvlKzEjuinkiQIj9CAdAzlKCKym01y0vDEKH0YxsI8ruBE/enIUCTlOArMZH6tnO/l4l+9TqrCy25GeZIqwvF0UZgyqGKYhw7VBCs2NgQhAU1t0I8RCZDE4MsmhC8+S8vkuZ1XOr3v15pXY1i6MADsExOAUeuA1cAvqoAEw+LCKVtnasz7tkr1vH0xHbWvmKYNfsI+AIm3vFw=</latexit><latexit sha1_base64="k8gteobUOWK/9RiKcoVzY2LBP8=">ACZ3iclVDLSgMxFM2Mr1ofHa2I4CZaBFdlRgRdKBTcuHBRwT6gU0smzbShmcyQZMQ6xI90596Nf2GmLaKtGw8Ezj3Hu7NCRJGpXLd8teWl5ZXSusFzc2t7ZLzs5uU8apwKSBYxaLdoAkYZSThqKkXYiCIoCRlrB6Cbvt56IkDTmD2qckG6EBpyGFCNlpJ7z6ocC4cxPkFAUMehHSA0xYtmd1nNqEGYt3fM0vIb/Mr0Yj/6unvVj5lOuiDAG3XMqbtWdAC4Sb0YqYIZ6z3nz+zFOI8IVZkjKjucmqpvlKzEjuinkiQIj9CAdAzlKCKym01y0vDEKH0YxsI8ruBE/enIUCTlOArMZH6tnO/l4l+9TqrCy25GeZIqwvF0UZgyqGKYhw7VBCs2NgQhAU1t0I8RCZDE4MsmhC8+S8vkuZ1XOr3v15pXY1i6MADsExOAUeuA1cAvqoAEw+LCKVtnasz7tkr1vH0xHbWvmKYNfsI+AIm3vFw=</latexit><latexit sha1_base64="k8gteobUOWK/9RiKcoVzY2LBP8=">ACZ3iclVDLSgMxFM2Mr1ofHa2I4CZaBFdlRgRdKBTcuHBRwT6gU0smzbShmcyQZMQ6xI90596Nf2GmLaKtGw8Ezj3Hu7NCRJGpXLd8teWl5ZXSusFzc2t7ZLzs5uU8apwKSBYxaLdoAkYZSThqKkXYiCIoCRlrB6Cbvt56IkDTmD2qckG6EBpyGFCNlpJ7z6ocC4cxPkFAUMehHSA0xYtmd1nNqEGYt3fM0vIb/Mr0Yj/6unvVj5lOuiDAG3XMqbtWdAC4Sb0YqYIZ6z3nz+zFOI8IVZkjKjucmqpvlKzEjuinkiQIj9CAdAzlKCKym01y0vDEKH0YxsI8ruBE/enIUCTlOArMZH6tnO/l4l+9TqrCy25GeZIqwvF0UZgyqGKYhw7VBCs2NgQhAU1t0I8RCZDE4MsmhC8+S8vkuZ1XOr3v15pXY1i6MADsExOAUeuA1cAvqoAEw+LCKVtnasz7tkr1vH0xHbWvmKYNfsI+AIm3vFw=</latexit>

∂L ∂b1 = ∂L ∂z1

<latexit sha1_base64="BUCvBOQZTAWqUd/o9bO/IRJ+vo=">ACUXiclVHLSsNAFL2Jr9r6iLp0M1gEVyURQRcKBTcuXFSwD2hCmEwn7eDkwcxEqCG/6EJX/ocbF4qTNgt3Xhg4HDuPXfunAlSzqSy7TfDXFldW9+obdYbW9s7u9befk8mSC0SxKeiEGAJeUspl3FKeDVFAcBZz2g4frst5/pEKyJL5X05R6ER7HLGQEKy351sQNBSa5m2KhGObIjbCaEMz26JYUIMwDwrfKdAV+pfpSXsK5FtNu2XPgJaJU5EmVOj41os7SkgW0VgRjqUcOnaqvLwcTjgt6m4maYrJAx7ToaYxjqj08lkiBTrWygiFidAnVmim/nTkOJyGgW6s9xSLtZK8a/aMFPhZezOM0Ujcn8ojDjSCWojBeNmKBE8akmAimd0VkgnVaSn9CXYfgLD5mfROW47dcu7Omu3LKo4aHMIRnIAD59CG+hAFwg8wzt8wpfxanyYJrzVtOoPAfwC2bjGxvKtkU=</latexit><latexit sha1_base64="BUCvBOQZTAWqUd/o9bO/IRJ+vo=">ACUXiclVHLSsNAFL2Jr9r6iLp0M1gEVyURQRcKBTcuXFSwD2hCmEwn7eDkwcxEqCG/6EJX/ocbF4qTNgt3Xhg4HDuPXfunAlSzqSy7TfDXFldW9+obdYbW9s7u9befk8mSC0SxKeiEGAJeUspl3FKeDVFAcBZz2g4frst5/pEKyJL5X05R6ER7HLGQEKy351sQNBSa5m2KhGObIjbCaEMz26JYUIMwDwrfKdAV+pfpSXsK5FtNu2XPgJaJU5EmVOj41os7SkgW0VgRjqUcOnaqvLwcTjgt6m4maYrJAx7ToaYxjqj08lkiBTrWygiFidAnVmim/nTkOJyGgW6s9xSLtZK8a/aMFPhZezOM0Ujcn8ojDjSCWojBeNmKBE8akmAimd0VkgnVaSn9CXYfgLD5mfROW47dcu7Omu3LKo4aHMIRnIAD59CG+hAFwg8wzt8wpfxanyYJrzVtOoPAfwC2bjGxvKtkU=</latexit><latexit sha1_base64="BUCvBOQZTAWqUd/o9bO/IRJ+vo=">ACUXiclVHLSsNAFL2Jr9r6iLp0M1gEVyURQRcKBTcuXFSwD2hCmEwn7eDkwcxEqCG/6EJX/ocbF4qTNgt3Xhg4HDuPXfunAlSzqSy7TfDXFldW9+obdYbW9s7u9befk8mSC0SxKeiEGAJeUspl3FKeDVFAcBZz2g4frst5/pEKyJL5X05R6ER7HLGQEKy351sQNBSa5m2KhGObIjbCaEMz26JYUIMwDwrfKdAV+pfpSXsK5FtNu2XPgJaJU5EmVOj41os7SkgW0VgRjqUcOnaqvLwcTjgt6m4maYrJAx7ToaYxjqj08lkiBTrWygiFidAnVmim/nTkOJyGgW6s9xSLtZK8a/aMFPhZezOM0Ujcn8ojDjSCWojBeNmKBE8akmAimd0VkgnVaSn9CXYfgLD5mfROW47dcu7Omu3LKo4aHMIRnIAD59CG+hAFwg8wzt8wpfxanyYJrzVtOoPAfwC2bjGxvKtkU=</latexit><latexit sha1_base64="BUCvBOQZTAWqUd/o9bO/IRJ+vo=">ACUXiclVHLSsNAFL2Jr9r6iLp0M1gEVyURQRcKBTcuXFSwD2hCmEwn7eDkwcxEqCG/6EJX/ocbF4qTNgt3Xhg4HDuPXfunAlSzqSy7TfDXFldW9+obdYbW9s7u9befk8mSC0SxKeiEGAJeUspl3FKeDVFAcBZz2g4frst5/pEKyJL5X05R6ER7HLGQEKy351sQNBSa5m2KhGObIjbCaEMz26JYUIMwDwrfKdAV+pfpSXsK5FtNu2XPgJaJU5EmVOj41os7SkgW0VgRjqUcOnaqvLwcTjgt6m4maYrJAx7ToaYxjqj08lkiBTrWygiFidAnVmim/nTkOJyGgW6s9xSLtZK8a/aMFPhZezOM0Ujcn8ojDjSCWojBeNmKBE8akmAimd0VkgnVaSn9CXYfgLD5mfROW47dcu7Omu3LKo4aHMIRnIAD59CG+hAFwg8wzt8wpfxanyYJrzVtOoPAfwC2bjGxvKtkU=</latexit>

Backward Propagation

slide-40
SLIDE 40

Computational graphs

Credits: Chris Manning

slide-41
SLIDE 41

An example

Compute the gradients yourself!

slide-42
SLIDE 42

Backpropagation in general computational graph

  • Forward propagation: visit nodes in topological sort order
  • Compute value of node given predecessors
  • Backward propagagation:
  • Initialize output gradient as 1
  • Visit nodes in reverse order and compute gradient wrt each

node using gradient wrt successors {y1, . . . , yn} = successors of x

<latexit sha1_base64="Nho9kGDoYh/19jQ7nqY3iLw3GKc=">ACGHicbVDLSgNBEJz1bXxFPXoZDIKHEHdF0IOC4MWjgolCNiyzk94ODuzPRKliWf4cVf8eJBEa/e/Bsnj4OvgoaiqpvurjiTwqLvf3pT0zOzc/MLi5Wl5ZXVter6Rsvq3HBoci21uYmZBSkUNFGghJvMAEtjCdfx3dnQv74HY4VWV1hk0ElZT4lEcIZOiqp7YVlEQZ2GsqvR1mkRqXBAT2iI0MeS2pxzsFYbS3VCB7QfVWt+wx+B/iXBhNTIBdR9SPsap6noJBLZm078DPslMyg4BIGlTC3kDF+x3rQdlSxFGynHD02oDtO6dJEG1cK6Uj9PlGy1NoijV1nyvDW/vaG4n9eO8fkqFMKleUIio8XJbmkqOkwJdoVBjKwhHGjXC3Un7LDOPosqy4EILfL/8lrf1G4DeCy4Pa6fEkjgWyRbJLgnITkl5+SCNAknD+SJvJBX79F79t6893HrlDeZ2SQ/4H18AUHyn0Q=</latexit><latexit sha1_base64="Nho9kGDoYh/19jQ7nqY3iLw3GKc=">ACGHicbVDLSgNBEJz1bXxFPXoZDIKHEHdF0IOC4MWjgolCNiyzk94ODuzPRKliWf4cVf8eJBEa/e/Bsnj4OvgoaiqpvurjiTwqLvf3pT0zOzc/MLi5Wl5ZXVter6Rsvq3HBoci21uYmZBSkUNFGghJvMAEtjCdfx3dnQv74HY4VWV1hk0ElZT4lEcIZOiqp7YVlEQZ2GsqvR1mkRqXBAT2iI0MeS2pxzsFYbS3VCB7QfVWt+wx+B/iXBhNTIBdR9SPsap6noJBLZm078DPslMyg4BIGlTC3kDF+x3rQdlSxFGynHD02oDtO6dJEG1cK6Uj9PlGy1NoijV1nyvDW/vaG4n9eO8fkqFMKleUIio8XJbmkqOkwJdoVBjKwhHGjXC3Un7LDOPosqy4EILfL/8lrf1G4DeCy4Pa6fEkjgWyRbJLgnITkl5+SCNAknD+SJvJBX79F79t6893HrlDeZ2SQ/4H18AUHyn0Q=</latexit><latexit sha1_base64="Nho9kGDoYh/19jQ7nqY3iLw3GKc=">ACGHicbVDLSgNBEJz1bXxFPXoZDIKHEHdF0IOC4MWjgolCNiyzk94ODuzPRKliWf4cVf8eJBEa/e/Bsnj4OvgoaiqpvurjiTwqLvf3pT0zOzc/MLi5Wl5ZXVter6Rsvq3HBoci21uYmZBSkUNFGghJvMAEtjCdfx3dnQv74HY4VWV1hk0ElZT4lEcIZOiqp7YVlEQZ2GsqvR1mkRqXBAT2iI0MeS2pxzsFYbS3VCB7QfVWt+wx+B/iXBhNTIBdR9SPsap6noJBLZm078DPslMyg4BIGlTC3kDF+x3rQdlSxFGynHD02oDtO6dJEG1cK6Uj9PlGy1NoijV1nyvDW/vaG4n9eO8fkqFMKleUIio8XJbmkqOkwJdoVBjKwhHGjXC3Un7LDOPosqy4EILfL/8lrf1G4DeCy4Pa6fEkjgWyRbJLgnITkl5+SCNAknD+SJvJBX79F79t6893HrlDeZ2SQ/4H18AUHyn0Q=</latexit><latexit sha1_base64="Nho9kGDoYh/19jQ7nqY3iLw3GKc=">ACGHicbVDLSgNBEJz1bXxFPXoZDIKHEHdF0IOC4MWjgolCNiyzk94ODuzPRKliWf4cVf8eJBEa/e/Bsnj4OvgoaiqpvurjiTwqLvf3pT0zOzc/MLi5Wl5ZXVter6Rsvq3HBoci21uYmZBSkUNFGghJvMAEtjCdfx3dnQv74HY4VWV1hk0ElZT4lEcIZOiqp7YVlEQZ2GsqvR1mkRqXBAT2iI0MeS2pxzsFYbS3VCB7QfVWt+wx+B/iXBhNTIBdR9SPsap6noJBLZm078DPslMyg4BIGlTC3kDF+x3rQdlSxFGynHD02oDtO6dJEG1cK6Uj9PlGy1NoijV1nyvDW/vaG4n9eO8fkqFMKleUIio8XJbmkqOkwJdoVBjKwhHGjXC3Un7LDOPosqy4EILfL/8lrf1G4DeCy4Pa6fEkjgWyRbJLgnITkl5+SCNAknD+SJvJBX79F79t6893HrlDeZ2SQ/4H18AUHyn0Q=</latexit>

∂L ∂x =

n

X

i=1

∂L ∂yi ∂yi ∂x

<latexit sha1_base64="m0S6Li+VWDk0V8PNiZ2hHDv4U=">ACYnicbVFNS8MwGE6r0zk/1rmjHoJD8DRaEfTgYODFw4T3Aes6RZuoWlaUlSsZT+SW+evPhDTLeB+/CFwMPzPO+bvE/8mFGpbPvLMPf2SweH5aPK8cnpWdWqnfdlAhMejhikRj6SBJGOekpqhgZxoKg0Gdk4M+fCn3wToSkEX9VaUzGIZpyGlCMlKY8K3UDgXDmxkgoihjs5H/4I4ct6Mok9DLacvK3jGtx0+6GSM0wYlknX2tMPZrDLWfBrY/OPathN+1FwV3grEADrKrWZ/uJMJSLjCDEk5cuxYjbNiImYkr7iJDHCczQlIw05CokcZ4uIcnitmQkMIqEPV3DBrndkKJQyDX3tLFaS21pB/qeNEhU8jDPK40QRjpcXBQmDKoJF3nBCBcGKpRogLKh+K8QzpKNR+lcqOgRne+Vd0L9tOnbTeblrtB9XcZTBbgCN8AB96ANnkEX9AG30bJODOqxo9ZMWtmfWk1jVPHWyUefkLjPi6BA=</latexit><latexit sha1_base64="m0S6Li+VWDk0V8PNiZ2hHDv4U=">ACYnicbVFNS8MwGE6r0zk/1rmjHoJD8DRaEfTgYODFw4T3Aes6RZuoWlaUlSsZT+SW+evPhDTLeB+/CFwMPzPO+bvE/8mFGpbPvLMPf2SweH5aPK8cnpWdWqnfdlAhMejhikRj6SBJGOekpqhgZxoKg0Gdk4M+fCn3wToSkEX9VaUzGIZpyGlCMlKY8K3UDgXDmxkgoihjs5H/4I4ct6Mok9DLacvK3jGtx0+6GSM0wYlknX2tMPZrDLWfBrY/OPathN+1FwV3grEADrKrWZ/uJMJSLjCDEk5cuxYjbNiImYkr7iJDHCczQlIw05CokcZ4uIcnitmQkMIqEPV3DBrndkKJQyDX3tLFaS21pB/qeNEhU8jDPK40QRjpcXBQmDKoJF3nBCBcGKpRogLKh+K8QzpKNR+lcqOgRne+Vd0L9tOnbTeblrtB9XcZTBbgCN8AB96ANnkEX9AG30bJODOqxo9ZMWtmfWk1jVPHWyUefkLjPi6BA=</latexit><latexit sha1_base64="m0S6Li+VWDk0V8PNiZ2hHDv4U=">ACYnicbVFNS8MwGE6r0zk/1rmjHoJD8DRaEfTgYODFw4T3Aes6RZuoWlaUlSsZT+SW+evPhDTLeB+/CFwMPzPO+bvE/8mFGpbPvLMPf2SweH5aPK8cnpWdWqnfdlAhMejhikRj6SBJGOekpqhgZxoKg0Gdk4M+fCn3wToSkEX9VaUzGIZpyGlCMlKY8K3UDgXDmxkgoihjs5H/4I4ct6Mok9DLacvK3jGtx0+6GSM0wYlknX2tMPZrDLWfBrY/OPathN+1FwV3grEADrKrWZ/uJMJSLjCDEk5cuxYjbNiImYkr7iJDHCczQlIw05CokcZ4uIcnitmQkMIqEPV3DBrndkKJQyDX3tLFaS21pB/qeNEhU8jDPK40QRjpcXBQmDKoJF3nBCBcGKpRogLKh+K8QzpKNR+lcqOgRne+Vd0L9tOnbTeblrtB9XcZTBbgCN8AB96ANnkEX9AG30bJODOqxo9ZMWtmfWk1jVPHWyUefkLjPi6BA=</latexit><latexit sha1_base64="m0S6Li+VWDk0V8PNiZ2hHDv4U=">ACYnicbVFNS8MwGE6r0zk/1rmjHoJD8DRaEfTgYODFw4T3Aes6RZuoWlaUlSsZT+SW+evPhDTLeB+/CFwMPzPO+bvE/8mFGpbPvLMPf2SweH5aPK8cnpWdWqnfdlAhMejhikRj6SBJGOekpqhgZxoKg0Gdk4M+fCn3wToSkEX9VaUzGIZpyGlCMlKY8K3UDgXDmxkgoihjs5H/4I4ct6Mok9DLacvK3jGtx0+6GSM0wYlknX2tMPZrDLWfBrY/OPathN+1FwV3grEADrKrWZ/uJMJSLjCDEk5cuxYjbNiImYkr7iJDHCczQlIw05CokcZ4uIcnitmQkMIqEPV3DBrndkKJQyDX3tLFaS21pB/qeNEhU8jDPK40QRjpcXBQmDKoJF3nBCBcGKpRogLKh+K8QzpKNR+lcqOgRne+Vd0L9tOnbTeblrtB9XcZTBbgCN8AB96ANnkEX9AG30bJODOqxo9ZMWtmfWk1jVPHWyUefkLjPi6BA=</latexit>