Neural Networks and Computation Graphs CS 6956: Deep Learning for - - PowerPoint PPT Presentation
Neural Networks and Computation Graphs CS 6956: Deep Learning for NLP Based on slides and material from Geoffrey Hinton, Richard Socher, Yoav Goldberg, and others. The computation graph slides are based on a tutorial Practical Neural
Based on slides and material from Geoffrey Hinton, Richard Socher, Yoav Goldberg, and others. The computation graph slides are based on a tutorial “Practical Neural Networks for NLP” by Chris Dyer, Yoav Goldberg, Graham Neubig in EMNLP 2016
1
2
3
features dot product threshold Prediction 𝑡𝑜 (𝒙'𝒚 + 𝑐) = 𝑡𝑜(∑𝑥/𝑦/ + 𝑐) Learning various algorithms perceptron, SVM, logistic regression,… in general, minimize loss But where do these input features come from? What if the features were outputs of another classifier?
4
5
6
Each of these connections have their own weights as well
7
8
This is a two layer feed forward neural network
9
The output layer The hidden layer The input layer This is a two layer feed forward neural network Think of the hidden layer as learning a good representation of the inputs
10
The dot product followed by the threshold constitutes a neuron Five neurons in this picture (four in hidden layer and one output) This is a two layer feed forward neural network
11
What if the inputs were the outputs of a classifier? The input layer We can make a three layer network…. And so on.
12
Functions that very loosely mimic a biological neuron
13
Dot product Threshold activation Other activations are possible 𝑝𝑣𝑢𝑞𝑣𝑢 = 𝑏𝑑𝑢𝑗𝑤𝑏𝑢𝑗𝑝𝑜(𝒙'𝒚 + 𝑐)
Name of the neuron Activation function: 𝑏𝑑𝑢𝑗𝑤𝑏𝑢𝑗𝑝𝑜 𝑨 Linear unit 𝑨 Threshold/sign unit sgn(𝑨) Sigmoid unit 1 1 + exp (−𝑨) Rectified linear unit (ReLU) max (0, 𝑨) Tanh unit tanh (𝑨)
14
𝑝𝑣𝑢𝑞𝑣𝑢 = 𝑏𝑑𝑢𝑗𝑤𝑏𝑢𝑗𝑝𝑜(𝒙'𝒚 + 𝑐) Many more activation functions exist (sinusoid, sinc, gaussian, polynomial…) Also called transfer functions
15
16
Called the architecture
Typically predefined, part of the design of the classifier Learned from data Input Hidden Output wIJ
K
wIJ
L
17
Input Hidden Output wIJ
K
wIJ
L
18
Called the architecture
Typically predefined, part of the design of the classifier Learned from data Input Hidden Output wIJ
K
wIJ
L
19
See also: http://people.idsia.ch/~juergen/deep-learning-overview.html
[DasGupta et al 1993]
– Exercise: Prove this
– Upper bound: Ο 𝑊 K 𝐹 K – Lower bound: Ω 𝐹 K
20
Exercise: Show that if we have only linear units, then multiple layers does not change the expressiveness
21
This section heavily draws upon the “Practical Neural Networks for NLP” by Chris Dyer, Yoav Goldberg, Graham Neubig in EMNLP 2016
22
23
24
Graph The value is implicitly or explicitly typed. It could represent a
25
f 𝐯 = | 𝐯 |
f 𝐯, 𝐰 = 𝐯Y𝐰
A node with an incoming edge is a function of the the parent node
26
f 𝐯 = | 𝐯 |
f 𝐯, 𝐰 = 𝐯Y𝐰
A node with an incoming edge is a function of the the parent node
27
f 𝐯 = | 𝐯 |
f 𝐯, 𝐰 = 𝐯Y𝐰
A node with an incoming edge is a function of the the parent node
28
f 𝐯, 𝐰 = 𝐯Y𝐰
\𝐯
\𝐯 and \] \𝐰
f 𝐯 = | 𝐯 | Notation: We will write down what that function is next to the node. When we write this, we will use formal arguments (here, the 𝐯 and 𝐰). Think of these as similar to the argument names we use when we declare functions while programming.
29
f 𝐯 = | 𝐯 |
f 𝐯, 𝐰 = 𝐯Y𝐰
\𝐯
\𝐯 and \] \𝐰
30
f 𝐯 = | 𝐯 |
f 𝐯, 𝐰 = 𝐯Y𝐰
31
Graph
f 𝐕, 𝐖 = 𝐕𝐖 f 𝐯 = 𝐯𝐔
32
Graph
f 𝐕, 𝐖 = 𝐕𝐖 f 𝐯 = 𝐯𝐔 f 𝐍, 𝐰 = 𝐍𝐰
33
Graph
f 𝐕, 𝐖 = 𝐕𝐖 f 𝐯 = 𝐯𝐔 f 𝐍, 𝐰 = 𝐍𝐰
f 𝐯, 𝐍 = 𝐯𝐔𝐍𝐯 We could have written the same function with a different graph. Computation graphs are not necessarily unique for a function
34
Graph
f 𝐯, 𝐍 = 𝐯𝐔𝐍𝐯 Remember: The nodes also know how to compute derivatives with respect to each parent
35
Graph
f 𝐯, 𝐍 = 𝐯𝐔𝐍𝐯 Remember: The nodes also know how to compute derivatives with respect to each parent 𝜖𝑔 𝜖𝐯 = 𝐍Y + 𝐍 𝐯 Derivative with respect to this parent
36
Graph
f 𝐯, 𝐍 = 𝐯𝐔𝐍𝐯 Remember: The nodes also know how to compute derivatives with respect to each parent 𝜖𝑔 𝜖𝐍 = 𝐯𝐯' Derivative with respect to this parent
37
Graph
f 𝐯, 𝐍 = 𝐯𝐔𝐍𝐯 Remember: The nodes also know how to compute derivatives with respect to each parent Together, we can compute derivatives of any function with respect to all its inputs, for any value
𝜖𝑔 𝜖𝐯 = 𝐍Y + 𝐍 𝐯 𝜖𝑔 𝜖𝐍 = 𝐯𝐯' 𝜖𝑔 𝜖𝐲 = 𝐁Y + 𝐁 𝐲 𝜖𝑔 𝜖𝐁 = 𝐲𝐲'
38
Graph
f 𝐕, 𝐖 = 𝐕𝐖 f 𝐯 = 𝐯𝐔 f 𝐍, 𝐰 = 𝐍𝐰
f 𝐯, 𝐰 = 𝐯𝐔𝐰 f 𝑦L, 𝑦K, 𝑦g = h 𝑦/
39
Graph
f 𝐕, 𝐖 = 𝐕𝐖 f 𝐯 = 𝐯𝐔 f 𝐍, 𝐰 = 𝐍𝐰
f 𝐯, 𝐰 = 𝐯𝐔𝐰
f 𝑦L, 𝑦K, 𝑦g = h 𝑦/
40
Graph
f 𝐕, 𝐖 = 𝐕𝐖 f 𝐯 = 𝐯𝐔 f 𝐍, 𝐰 = 𝐍𝐰
f 𝐯, 𝐰 = 𝐯𝐔𝐰
f 𝑦L, 𝑦K, 𝑦g = h 𝑦/
We can name variables by labeling nodes
41
42
𝐢 = tanh 𝐗𝐲 + 𝐜 𝒛 = 𝐖𝐢 + 𝐛
43
𝐢 = tanh 𝐗𝐲 + 𝐜 𝒛 = 𝐖𝐢 + 𝐛
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰
𝐠 𝐰 = tanh(𝐰)
44
𝐢 = tanh 𝐗𝐲 + 𝐜 𝒛 = 𝐖𝐢 + 𝐛
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰
𝐠 𝐰 = tanh(𝐰)
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰
L Lqrst (uv)
x L K w'w + 𝐷 ∑ max
45
46
– Given inputs to the graph, compute the value of the function expressed by the graph – Something to think about: Given a node, can we say which nodes are inputs? Which nodes are outputs?
– After computing the function value for an input, compute the gradient of the function at that input – Or equivalently: How does the output change if I make a small change to the input?
– Need an easy-to-use framework to construct graphs – The size of the graph may be input dependent
– Tensorflow, PyTorch are the most popular frameworks today
47
48
– Given inputs to the graph, compute the value of the function expressed by the graph – Something to think about: Given a node, can we say which nodes are inputs? Which nodes are outputs?
– After computing the function value for an input, compute the gradient of the function at that input – Or equivalently: How does the output change if I make a small change to the input?
– Need an easy-to-use framework to construct graphs – The size of the graph may be input dependent
– Tensorflow, PyTorch are the most popular frameworks today
49
50
𝑧 𝑦 𝑣 + 𝑤 𝑣K log 𝑣 𝑣𝑤 h 𝑣/
Conventions:
51
𝑧 𝑦 𝑣 + 𝑤 𝑣K log 𝑣 𝑣𝑤 h 𝑣/
What function does this compute?
52
𝑧 𝑦 𝑣 + 𝑤 𝑣K log 𝑣 𝑣𝑤 h 𝑣/
What function does this compute? Suppose we shade nodes whose values we know (i.e. we have computed).
53
𝑧 𝑦 𝑣 + 𝑤 𝑣K log 𝑣 𝑣𝑤 h 𝑣/
What function does this compute? Suppose we shade nodes whose values we know (i.e. we have computed). 𝑦
54
𝑧 𝑦 𝑣 + 𝑤 𝑣K log 𝑣 𝑣𝑤 h 𝑣/
What function does this compute? Suppose we shade nodes whose values we know (i.e. we have computed). 𝑦 𝑧
55
𝑧 𝑦 𝑣 + 𝑤 𝑣K log 𝑣 𝑣𝑤 h 𝑣/
What function does this compute? Suppose we shade nodes whose values we know (i.e. we have computed). 𝑦 𝑧 We can only compute the value of a node if we know the values of all its inputs
56
𝑧 𝑦 𝑣 + 𝑤 𝑣K log 𝑣 𝑣𝑤 h 𝑣/
What function does this compute? Suppose we shade nodes whose values we know (i.e. we have computed). 𝑦 𝑧 We can only compute the value of a node if we know the values of all its inputs 𝑦 + 𝑧
57
𝑧 𝑦 𝑣 + 𝑤 𝑣K log 𝑣 𝑣𝑤 h 𝑣/
What function does this compute? Suppose we shade nodes whose values we know (i.e. we have computed). 𝑦 𝑧 We can only compute the value of a node if we know the values of all its inputs 𝑦 + 𝑧 𝑧K
58
𝑧 𝑦 𝑣 + 𝑤 𝑣K log 𝑣 𝑣𝑤 h 𝑣/
What function does this compute? Suppose we shade nodes whose values we know (i.e. we have computed). 𝑦 𝑧 We can only compute the value of a node if we know the values of all its inputs 𝑦 + 𝑧 𝑧K 𝑦(𝑦 + 𝑧)
59
𝑧 𝑦 𝑣 + 𝑤 𝑣K log 𝑣 𝑣𝑤 h 𝑣/
What function does this compute? Suppose we shade nodes whose values we know (i.e. we have computed). 𝑦 𝑧 We can only compute the value of a node if we know the values of all its inputs 𝑦 + 𝑧 𝑧K 𝑦(𝑦 + 𝑧) log (𝑦 + 𝑧)
60
𝑧 𝑦 𝑣 + 𝑤 𝑣K log 𝑣 𝑣𝑤 h 𝑣/
What function does this compute? Suppose we shade nodes whose values we know (i.e. we have computed). 𝑦 𝑧 We can only compute the value of a node if we know the values of all its inputs 𝑦 + 𝑧 𝑧K 𝑦(𝑦 + 𝑧) log (𝑦 + 𝑧) x x + y + log 𝑦 + 𝑧 + 𝑧K
61
𝑧 𝑦 𝑣 + 𝑤 𝑣K log 𝑣 𝑣𝑤 h 𝑣/
What function does this compute? Suppose we shade nodes whose values we know (i.e. we have computed). 𝑦 𝑧 We can only compute the value of a node if we know the values of all its inputs 𝑦 + 𝑧 𝑧K 𝑦(𝑦 + 𝑧) log (𝑦 + 𝑧) x x + y + log 𝑦 + 𝑧 + 𝑧K This gives us the function
62
f 𝐕, 𝐖 = 𝐕𝐖 f 𝐯 = 𝐯𝐔 f 𝐍, 𝐰 = 𝐍𝐰
f 𝐯, 𝐰 = 𝐯𝐔𝐰
f 𝑦L, 𝑦K, 𝑦g = h 𝑦/
63
f 𝐕, 𝐖 = 𝐕𝐖 f 𝐯 = 𝐯𝐔 f 𝐍, 𝐰 = 𝐍𝐰
f 𝐯, 𝐰 = 𝐯𝐔𝐰
f 𝑦L, 𝑦K, 𝑦g = h 𝑦/
To compute the function, we need the values of the leaves of this DAG
64
f 𝐕, 𝐖 = 𝐕𝐖 f 𝐯 = 𝐯𝐔 f 𝐍, 𝐰 = 𝐍𝐰
f 𝐯, 𝐰 = 𝐯𝐔𝐰
f 𝑦L, 𝑦K, 𝑦g = h 𝑦/
To compute the function, we need the values of the leaves of this DAG
65
f 𝐕, 𝐖 = 𝐕𝐖 f 𝐯 = 𝐯𝐔 f 𝐍, 𝐰 = 𝐍𝐰
f 𝐯, 𝐰 = 𝐯𝐔𝐰
f 𝑦L, 𝑦K, 𝑦g = h 𝑦/
Let’s also highlight which nodes can be computed using what we know so far
66
f 𝐕, 𝐖 = 𝐕𝐖 f 𝐯 = 𝐯𝐔 f 𝐍, 𝐰 = 𝐍𝐰
f 𝐯, 𝐰 = 𝐯𝐔𝐰
f 𝑦L, 𝑦K, 𝑦g = h 𝑦/
𝐲'
67
f 𝐕, 𝐖 = 𝐕𝐖 f 𝐯 = 𝐯𝐔 f 𝐍, 𝐰 = 𝐍𝐰
f 𝐯, 𝐰 = 𝐯𝐔𝐰
f 𝑦L, 𝑦K, 𝑦g = h 𝑦/
𝐲' 𝐜'𝐲
68
f 𝐕, 𝐖 = 𝐕𝐖 f 𝐯 = 𝐯𝐔 f 𝐍, 𝐰 = 𝐍𝐰
f 𝐯, 𝐰 = 𝐯𝐔𝐰
f 𝑦L, 𝑦K, 𝑦g = h 𝑦/
𝐲' 𝐜'𝐲 𝐲'𝐁
69
f 𝐕, 𝐖 = 𝐕𝐖 f 𝐯 = 𝐯𝐔 f 𝐍, 𝐰 = 𝐍𝐰
f 𝐯, 𝐰 = 𝐯𝐔𝐰
f 𝑦L, 𝑦K, 𝑦g = h 𝑦/
𝐲' 𝐜'𝐲 𝐲'𝐁 𝐲'𝐁𝐲
70
f 𝐕, 𝐖 = 𝐕𝐖 f 𝐯 = 𝐯𝐔 f 𝐍, 𝐰 = 𝐍𝐰
f 𝐯, 𝐰 = 𝐯𝐔𝐰
f 𝑦L, 𝑦K, 𝑦g = h 𝑦/
𝐲' 𝐜'𝐲 𝐲'𝐁 𝐲'𝐁𝐲 𝐲'𝐁𝐲 + 𝐜Y𝐲 + 𝐝
71
72
73
– Given inputs to the graph, compute the value of the function expressed by the graph – Something to think about: Given a node, can we say which nodes are inputs? Which nodes are outputs?
– After computing the function value for an input, compute the gradient of the function at that input – Or equivalently: How does the output change if I make a small change to the input?
– Need an easy-to-use framework to construct graphs – The size of the graph may be input dependent
– Tensorflow, PyTorch are the most popular frameworks today
74
{| {v = 𝑔} 𝑦
75
76
𝑦 𝑨 𝑧 f g
77
𝑦 𝑨 𝑧 f g
78
𝑦 𝑨 𝑧 f g
𝑒𝑧 𝑒𝑨
79
𝑦 𝑨 𝑧 f g
𝑒𝑧 𝑒𝑨 𝑒𝑧 𝑒𝑨 ⋅ 𝑒𝑨 𝑒𝑦
80
𝑦 𝑨 𝑧 𝑔 𝑣 = 1 𝑣 g u = uK 𝑧 = 1 𝑦K
81
𝑦 𝑨 𝑧 𝑔 𝑣 = 1 𝑣 g u = uK 𝑒𝑔 𝑒𝑣 = − 1 𝑣K 𝑒 𝑒𝑣 = 2𝑣 𝑧 = 1 𝑦K Let’s also explicitly write down the derivatives.
82
𝑦 𝑨 𝑧 𝑔 𝑣 = 1 𝑣 g u = uK 𝑒𝑔 𝑒𝑣 = − 1 𝑣K 𝑒 𝑒𝑣 = 2𝑣 𝑧 = 1 𝑦K 𝑒𝑧 𝑒𝑧 = 1 Now, we can proceed backwards from the output At each step, we compute the gradient of the function represented by the graph with respect to the node that we are at.
83
𝑦 𝑨 𝑧 𝑔 𝑣 = 1 𝑣 g u = uK 𝑒𝑔 𝑒𝑣 = − 1 𝑣K 𝑒 𝑒𝑣 = 2𝑣 𝑧 = 1 𝑦K 𝑒𝑧 𝑒𝑧 = 1 𝑒𝑧 𝑒𝑨 = 𝑒𝑧 𝑒𝑧 ̇ ⋅ 𝑒𝑔 𝑒𝑣 „…† = 1 ⋅ − 1 𝑨K = − 1 𝑨K Product of the gradient so far and the derivative computed at this step
84
𝑦 𝑨 𝑧 𝑔 𝑣 = 1 𝑣 g u = uK 𝑒𝑔 𝑒𝑣 = − 1 𝑣K 𝑒 𝑒𝑣 = 2𝑣 𝑧 = 1 𝑦K 𝑒𝑧 𝑒𝑧 = 1 𝑒𝑧 𝑒𝑨 = − 1 𝑨K 𝑒𝑧 𝑒𝑦 = 𝑒𝑧 𝑒𝑨 ⋅ 𝑒 𝑒𝑣 „…v = − 1 𝑨K ⋅ 2𝑦 = − 2x zK
85
𝑦 𝑨 𝑧 𝑔 𝑣 = 1 𝑣 g u = uK 𝑒𝑔 𝑒𝑣 = − 1 𝑣K 𝑒 𝑒𝑣 = 2𝑣 𝑧 = 1 𝑦K 𝑒𝑧 𝑒𝑧 = 1
vˆ 𝑒𝑧 𝑒𝑨 = − 1 𝑨K 𝑒𝑧 𝑒𝑦 = 𝑒𝑧 𝑒𝑨 ⋅ 𝑒 𝑒𝑣 „…v = − 1 𝑨K ⋅ 2𝑦 = − 2x zK
86
𝑦 𝑨 𝑧 𝑔 𝑣, 𝑤 = 𝑤 𝑣 g u = uK 𝑧 = 1 𝑦
87
𝑦 𝑨 𝑧 𝑔 𝑣, 𝑤 = 𝑤 𝑣 g u = uK 𝑧 = 1 𝑦
𝑒𝑔 𝑒𝑣 = − 𝑤 𝑣K 𝑒 𝑒𝑣 = 2𝑣 𝑒𝑔 𝑒𝑤 = 1 𝑣 Let’s also explicitly write down the derivatives. Note that 𝑔 has two derivatives because it has two inputs.
88
𝑦 𝑨 𝑧 𝑔 𝑣, 𝑤 = 𝑤 𝑣 g u = uK 𝑧 = 1 𝑦
𝑒𝑔 𝑒𝑣 = − 𝑤 𝑣K 𝑒 𝑒𝑣 = 2𝑣 𝑒𝑧 𝑒𝑧 = 1 𝑒𝑔 𝑒𝑤 = 1 𝑣
89
𝑦 𝑨 𝑧 𝑔 𝑣, 𝑤 = 𝑤 𝑣 g u = uK 𝑧 = 1 𝑦
𝑒𝑔 𝑒𝑣 = − 𝑤 𝑣K 𝑒 𝑒𝑣 = 2𝑣 𝑒𝑧 𝑒𝑧 = 1 𝑒𝑔 𝑒𝑤 = 1 𝑣 At this point, we can compute the gradient
from y to z. But we can not follow the edge from y to x because all of x’s descendants are not marked as done.
90
𝑦 𝑨 𝑧 𝑔 𝑣, 𝑤 = 𝑤 𝑣 g u = uK 𝑧 = 1 𝑦
𝑒𝑔 𝑒𝑣 = − 𝑤 𝑣K 𝑒 𝑒𝑣 = 2𝑣 𝑒𝑧 𝑒𝑧 = 1 𝑒𝑧 𝑒𝑨 = 𝑒𝑧 𝑒𝑧 ̇ ⋅ 𝑒𝑔 𝑒𝑣 „…† = 1 ⋅ − 𝑦 𝑨K = − 𝑦 𝑨K 𝑒𝑔 𝑒𝑤 = 1 𝑣 Product of the gradient so far and the derivative computed at this step
91
𝑦 𝑨 𝑧 𝑔 𝑣, 𝑤 = 𝑤 𝑣 g u = uK 𝑧 = 1 𝑦
𝑒𝑔 𝑒𝑣 = − 𝑤 𝑣K 𝑒 𝑒𝑣 = 2𝑣 𝑒𝑧 𝑒𝑧 = 1 𝑒𝑧 𝑒𝑨 = 𝑒𝑧 𝑒𝑧 ̇ ⋅ 𝑒𝑔 𝑒𝑣 „…† = 1 ⋅ − 𝑦 𝑨K = − 𝑦 𝑨K 𝑒𝑔 𝑒𝑤 = 1 𝑣 Now we can get to x There are multiple backward paths into x. The general rule: Add the gradients along all the paths.
92
𝑦 𝑨 𝑧 𝑔 𝑣, 𝑤 = 𝑤 𝑣 g u = uK 𝑧 = 1 𝑦
𝑒𝑔 𝑒𝑣 = − 𝑤 𝑣K 𝑒 𝑒𝑣 = 2𝑣 𝑒𝑧 𝑒𝑧 = 1 𝑒𝑧 𝑒𝑨 = − 𝑦 𝑨K 𝑒𝑔 𝑒𝑤 = 1 𝑣 𝑒𝑧 𝑒𝑦 = 𝑒𝑧 𝑒𝑨 ⋅ 𝑒 𝑒𝑣 „…v + 𝑒𝑧 𝑒𝑧 ⋅ 𝑒𝑔 𝑒𝑤 ‰…v There are multiple backward paths into x. The general rule: Add the gradients along all the paths.
93
𝑦 𝑨 𝑧 𝑔 𝑣, 𝑤 = 𝑤 𝑣 g u = uK 𝑧 = 1 𝑦
𝑒𝑔 𝑒𝑣 = − 𝑤 𝑣K 𝑒 𝑒𝑣 = 2𝑣 𝑒𝑧 𝑒𝑧 = 1 𝑒𝑧 𝑒𝑨 = − 𝑦 𝑨K 𝑒𝑔 𝑒𝑤 = 1 𝑣 𝑒𝑧 𝑒𝑦 = 𝑒𝑧 𝑒𝑨 ⋅ 𝑒 𝑒𝑣 „…v + 𝑒𝑧 𝑒𝑧 ⋅ 𝑒𝑔 𝑒𝑤 ‰…v There are multiple backward paths into x. The general rule: Add the gradients along all the paths.
94
𝑦 𝑨 𝑧 𝑔 𝑣, 𝑤 = 𝑤 𝑣 g u = uK 𝑧 = 1 𝑦
𝑒𝑔 𝑒𝑣 = − 𝑤 𝑣K 𝑒 𝑒𝑣 = 2𝑣 𝑒𝑧 𝑒𝑧 = 1 𝑒𝑧 𝑒𝑨 = − 𝑦 𝑨K 𝑒𝑔 𝑒𝑤 = 1 𝑣 𝑒𝑧 𝑒𝑦 = 𝑒𝑧 𝑒𝑨 ⋅ 𝑒 𝑒𝑣 „…v + 𝑒𝑧 𝑒𝑧 ⋅ 𝑒𝑔 𝑒𝑤 ‰…v There are multiple backward paths into x. The general rule: Add the gradients along all the paths.
95
𝑦 𝑨 𝑧 𝑔 𝑣, 𝑤 = 𝑤 𝑣 g u = uK 𝑧 = 1 𝑦
𝑒𝑔 𝑒𝑣 = − 𝑤 𝑣K 𝑒 𝑒𝑣 = 2𝑣 𝑒𝑧 𝑒𝑧 = 1 𝑒𝑔 𝑒𝑤 = 1 𝑣 𝑒𝑧 𝑒𝑦 = 𝑒𝑧 𝑒𝑨 ⋅ 𝑒 𝑒𝑣 „…v + 𝑒𝑧 𝑒𝑧 ⋅ 𝑒𝑔 𝑒𝑤 ‰…v 𝑒𝑧 𝑒𝑦 = − 𝑦 𝑨K ⋅ 2𝑦 + 1 ⋅ 1 𝑨 = − 2𝑦K 𝑨K + 1 𝑨 = − 1 𝑦K 𝑒𝑧 𝑒𝑨 = − 𝑦 𝑨K
96
This is the same two-layer network we saw before. But this time we have added a new loss term at the end. Suppose our goal is to compute the derivative of the loss with respect to 𝐗, 𝐖, 𝐛, 𝐜 𝐢 = tanh 𝐗𝐲 + 𝐜 𝒛 = 𝐖𝐢 + 𝐛 𝑴 = 1 2 𝒛 − 𝒛∗
𝟑
97
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰
𝐠 𝐰 = tanh(𝐰)
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰 𝐢 = tanh 𝐗𝐲 + 𝐜 𝒛 = 𝐖𝐢 + 𝐛 𝑴 = 1 2 𝒛 − 𝒛∗
𝟑
𝒈(𝐯, 𝐰) = 1 2 𝐯 − 𝐰
𝟑
98
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰
𝐠 𝐰 = tanh(𝐰)
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰 𝐢 = tanh 𝐗𝐲 + 𝐜 𝒛 = 𝐖𝐢 + 𝐛 𝑴 = 1 2 𝒛 − 𝒛∗
𝟑
𝒈(𝐯, 𝐰) = 1 2 𝐯 − 𝐰
𝟑
To simplify notation, let us name all the nodes
99
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰
𝐠 𝐰 = tanh(𝐰)
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰 𝐢 = tanh 𝐗𝐲 + 𝐜 𝒛 = 𝐖𝐢 + 𝐛 𝑴 = 1 2 𝒛 − 𝒛∗
𝟑
𝒈(𝐯, 𝐰) = 1 2 𝐯 − 𝐰
𝟑
Let us highlight nodes that are done 𝑒𝑀 𝑒𝑀 = 1
100
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰
𝐠 𝐰 = tanh(𝐰)
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰 𝐢 = tanh 𝐗𝐲 + 𝐜 𝒛 = 𝐖𝐢 + 𝐛 𝑴 = 1 2 𝒛 − 𝒛∗
𝟑
𝒈(𝐯, 𝐰) = 1 2 𝐯 − 𝐰
𝟑
Whenever we have the derivative
some new derivatives can be
𝑒𝑀 𝑒𝑀 = 1
101
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰
𝐠 𝐰 = tanh(𝐰)
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰 𝐢 = tanh 𝐗𝐲 + 𝐜 𝒛 = 𝐖𝐢 + 𝐛 𝑴 = 1 2 𝒛 − 𝒛∗
𝟑
𝒈(𝐯, 𝐰) = 1 2 𝐯 − 𝐰
𝟑
𝑒𝑀 𝑒𝑀 = 1 𝑒𝑀 𝑒𝐳 = 𝑒𝑀 𝑒𝑀 ⋅ 𝑒𝑀 𝑒𝐳 = 1 ⋅ 𝐳 − 𝐳∗
102
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰
𝐠 𝐰 = tanh(𝐰)
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰 𝐢 = tanh 𝐗𝐲 + 𝐜 𝒛 = 𝐖𝐢 + 𝐛 𝑴 = 1 2 𝒛 − 𝒛∗
𝟑
𝒈(𝐯, 𝐰) = 1 2 𝐯 − 𝐰
𝟑
𝑒𝑀 𝑒𝐳 = 𝑒𝑀 𝑒𝑀 ⋅ 𝑒𝑀 𝑒𝐳 = 𝐳 − 𝐳∗ 𝑒𝑀 𝑒𝐛 = 𝑒𝑀 𝑒𝐳 ⋅ 𝑒𝐳 𝑒𝐛 = 𝐳 − 𝐳∗ ⋅ 1
103
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰
𝐠 𝐰 = tanh(𝐰)
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰 𝐢 = tanh 𝐗𝐲 + 𝐜 𝒛 = 𝐖𝐢 + 𝐛 𝑴 = 1 2 𝒛 − 𝒛∗
𝟑
𝒈(𝐯, 𝐰) = 1 2 𝐯 − 𝐰
𝟑
𝑒𝑀 𝑒𝐳 = 𝑒𝑀 𝑒𝑀 ⋅ 𝑒𝑀 𝑒𝐳 = 𝐳 − 𝐳∗ 𝑒𝑀 𝑒𝐴𝟓 = 𝑒𝑀 𝑒𝐳 ⋅ 𝑒𝐳 𝑒𝒜𝟓 = 𝐳 − 𝐳∗ ⋅ 1
104
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰
𝐠 𝐰 = tanh(𝐰)
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰 𝐢 = tanh 𝐗𝐲 + 𝐜 𝒛 = 𝐖𝐢 + 𝐛 𝑴 = 1 2 𝒛 − 𝒛∗
𝟑
𝒈(𝐯, 𝐰) = 1 2 𝐯 − 𝐰
𝟑
𝑒𝑀 𝑒𝐴𝟓 = 𝑒𝑀 𝑒𝐳 ⋅ 𝑒𝐳 𝑒𝒜𝟓 = 𝐳 − 𝐳∗ 𝑒𝑀 𝑒𝐴𝟒 = 𝑒𝑀 𝑒𝐴𝟓 ⋅ 𝑒𝐴𝟓 𝑒𝒜𝟒 = 𝐳 − 𝐳∗ ⋅ 𝐍
105
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰
𝐠 𝐰 = tanh(𝐰)
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰 𝐢 = tanh 𝐗𝐲 + 𝐜 𝒛 = 𝐖𝐢 + 𝐛 𝑴 = 1 2 𝒛 − 𝒛∗
𝟑
𝒈(𝐯, 𝐰) = 1 2 𝐯 − 𝐰
𝟑
𝑒𝑀 𝑒𝐴𝟓 = 𝑒𝑀 𝑒𝐳 ⋅ 𝑒𝐳 𝑒𝒜𝟓 = 𝐳 − 𝐳∗ 𝑒𝑀 𝑒𝐴𝟒 = 𝑒𝑀 𝑒𝐴𝟓 ⋅ 𝑒𝐴𝟓 𝑒𝒜𝟒 = 𝐳 − 𝐳∗ ⋅ 𝐍 Because 𝒊 = 𝒜𝟒 𝑒𝑀 𝑒𝐢 = 𝑒𝑀 𝑒𝐴𝟒
106
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰
𝐠 𝐰 = tanh(𝐰)
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰 𝐢 = tanh 𝐗𝐲 + 𝐜 𝒛 = 𝐖𝐢 + 𝐛 𝑴 = 1 2 𝒛 − 𝒛∗
𝟑
𝒈(𝐯, 𝐰) = 1 2 𝐯 − 𝐰
𝟑
𝑒𝑀 𝑒𝐴𝟓 = 𝑒𝑀 𝑒𝐳 ⋅ 𝑒𝐳 𝑒𝒜𝟓 = 𝐳 − 𝐳∗ 𝑒𝑀 𝑒𝐴𝟒 = 𝑒𝑀 𝑒𝐴𝟓 ⋅ 𝑒𝐴𝟓 𝑒𝒜𝟒 = 𝐳 − 𝐳∗ ⋅ 𝐍 Because 𝒊 = 𝒜𝟒 𝑒𝑀 𝑒𝐢 = 𝑒𝑀 𝑒𝐴𝟒 = 𝐳 − 𝐳∗ ⋅ 𝐍
107
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰
𝐠 𝐰 = tanh(𝐰)
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰 𝐢 = tanh 𝐗𝐲 + 𝐜 𝒛 = 𝐖𝐢 + 𝐛 𝑴 = 1 2 𝒛 − 𝒛∗
𝟑
𝒈(𝐯, 𝐰) = 1 2 𝐯 − 𝐰
𝟑
108
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰
𝐠 𝐰 = tanh(𝐰)
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰 𝐢 = tanh 𝐗𝐲 + 𝐜 𝒛 = 𝐖𝐢 + 𝐛 𝑴 = 1 2 𝒛 − 𝒛∗
𝟑
𝒈(𝐯, 𝐰) = 1 2 𝐯 − 𝐰
𝟑
109
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰
𝐠 𝐰 = tanh(𝐰)
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰 𝐢 = tanh 𝐗𝐲 + 𝐜 𝒛 = 𝐖𝐢 + 𝐛 𝑴 = 1 2 𝒛 − 𝒛∗
𝟑
𝒈(𝐯, 𝐰) = 1 2 𝐯 − 𝐰
𝟑
110
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰
𝐠 𝐰 = tanh(𝐰)
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰 𝐢 = tanh 𝐗𝐲 + 𝐜 𝒛 = 𝐖𝐢 + 𝐛 𝑴 = 1 2 𝒛 − 𝒛∗
𝟑
𝒈(𝐯, 𝐰) = 1 2 𝐯 − 𝐰
𝟑
111
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰
𝐠 𝐰 = tanh(𝐰)
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰 𝐢 = tanh 𝐗𝐲 + 𝐜 𝒛 = 𝐖𝐢 + 𝐛 𝑴 = 1 2 𝒛 − 𝒛∗
𝟑
𝒈(𝐯, 𝐰) = 1 2 𝐯 − 𝐰
𝟑
112
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰
𝐠 𝐰 = tanh(𝐰)
𝐠 𝐍, 𝐰 = 𝐍𝐰
𝐠 𝐯, 𝐰 = 𝐯 + 𝐰 𝐢 = tanh 𝐗𝐲 + 𝐜 𝒛 = 𝐖𝐢 + 𝐛 𝑴 = 1 2 𝒛 − 𝒛∗
𝟑
𝒈(𝐯, 𝐰) = 1 2 𝐯 − 𝐰
𝟑
113
114
– Given inputs to the graph, compute the value of the function expressed by the graph – Something to think about: Given a node, can we say which nodes are inputs? Which nodes are outputs?
– After computing the function value for an input, compute the gradient of the function at that input – Or equivalently: How does the output change if I make a small change to the input?
– Need an easy-to-use framework to construct graphs – The size of the graph may be input dependent
– Tensorflow, PyTorch are the most popular frameworks today
115
116
117
118
119
120