Natural Language Processing with Deep Learning Neural Networks a - - PowerPoint PPT Presentation

Institute of Computational Perception

Natural Language Processing with Deep Learning Neural Networks – a Walkthrough

Navid Rekab-Saz

navid.rekabsaz@jku.at

Agenda

• Introduction
• Non-linearities
• Forward pass & backpropagation
• Softmax & loss function
• Optimization & regularization

Agenda

• Introduction
• Non-linearities
• Forward pass & backpropagation
• Softmax & loss function
• Optimization & regularization

4

Notation § 𝑏 → scalar § 𝒄 → vector

• 𝑗!" element of 𝒄 is the scalar 𝑐#

§ 𝑫 → matrix

• 𝑗!" vector of 𝑫 is 𝒅#
• 𝑘!" element of the 𝑗!" vector of 𝑫 is the scalar 𝑑#,%

§ Tensor: generalization of scalar, vector, matrix to any arbitrary dimension

5

Linear Algebra

6

Linear Algebra – Transpose

§ 𝒃 is in 1×d dimensions → 𝒃𝐔 is in § 𝑩 is in e×d dimensions → 𝑩𝐔 is in

1 2 3 4 5 6

&

=

d×1 dimensions d×e dimensions

1 4 2 5 3 6

7

Linear Algebra – Dot product § 𝒃 2 𝒄' =

• dimensions: 1×d ( d×1 =

1 2 3 2 1 =

§ 𝒃 2 𝑪 =

• dimensions: 1×d ( d×e =

1 2 3 2 3 1 1 −1 =

§ 𝑩 2 𝑪 =

• dimensions: l×m ( m×n =

1 2 3 1 1 5 4 1 2 3 1 1 −1 = § Linear transformation: dot product of a vector to a matrix

𝑑

1

𝒅

1×e

𝑫

l×n 5 5 2 5 2 3 2 5 −5 8 13

8

Probability § Conditional probability

𝑞(𝑧|𝑦)

§ Probability distribution

• For a discrete random variable 𝒜 with 𝐿 states
• 0 ≤ 𝑞 𝑨# ≤ 1
• ∑#01

2

𝑞 𝑨# = 1

• E.g. with 𝐿 = 4 states: 0.2

0.3 0.45 0.05

9

Probability § Expected value

𝔽-~/ 𝑔 = 1 𝑌 ,

• ∈/

𝑔(𝑦)

• Note: This is an imprecise definition. Though, it suffices for our

use in this lecture

10

Artificial Neural Networks § Neural Networks are non-linear functions and universal approximators § They composed of several simple (non-)linear

• perations

§ Neural networks can readily be defined as probabilistic models which estimate 𝑞(𝑧|𝒚; 𝑿)

• Given input vector 𝒚 and the set of parameters 𝑿, estimate the

probability of the output class y

11

A Feedforward network

𝑿(𝟐)

size 3x4

𝑿(𝟑)

size 4x2 input vector

𝒚

parameter matrices

• utput probability

distribution

𝑞 𝑧 𝒚; 𝑿

12

Learning with Neural Networks

§ Design the network’s architecture § Consider proper regularization methods § Initialize parameters § Loop until some exit criteria are met

• Sample a minibatch from training data 𝒠
• Loop over data points in the minibatch
• Forward pass: given input 𝒚 predict output 𝑞 𝑧 𝒚; 𝑿
• Calculate loss function
• Calculate the gradient of each parameter regarding the loss

function using the backpropagation algorithm

• Update parameters using their gradients

Agenda

• Introduction
• Non-linearities
• Forward pass & backpropagation
• Softmax & loss function
• Optimization & regularization

14

Neural Computation

source

15

An Artificial Neuron

source

16

Linear

𝑔 𝑦 = 𝑦

17

Sigmoid

𝑔 𝑦 = 𝜏 𝑦 = 1 1 + 𝑓!"

§ squashes input between 0 and 1 § Output becomes like a probability value

18

Hyperbolic Tangent (Tanh)

𝑔 𝑦 = tanh 𝑦 = 𝑓#" − 1 𝑓#" + 1

§ squashes input between -1 and 1 𝜏

Tanh

19

Rectified Linear Unit (ReLU)

𝑔 𝑦 = max(0, 𝑦)

§ Good for deep architectures, as it prevents vanishing gradient

20

Examples

𝒚 = 1 3 𝑿 = 0.5 −0.5 2 4 −1 § Linear transformation 𝒚𝑿: 𝒚𝑿 = 1 3 0.5 −0.5 2 −1 4 −1 = 𝟏. 𝟔 −𝟏. 𝟔 𝟑 𝟐𝟑 −𝟓 § Non-linear transformation ReLU(𝒚𝑿): ReLU 0.5 −0.5 2 12 −3 = 𝟏. 𝟔 𝟏. 𝟏 𝟑 𝟐𝟑 𝟏. 𝟏 § Non-linear transformation 𝜏(𝒚𝑿): 𝜏 0.5 −0.5 2 12 −3 = 𝟏. 𝟕𝟑 𝟏. 𝟒𝟖 𝟏. 𝟗𝟗 𝟏. 𝟘𝟘 𝟏. 𝟏𝟐𝟗 § Non-linear transformation tanh(𝒚𝑿): tanh 0.5 −0.5 2 12 −3 = 𝟏. 𝟓𝟕 −𝟏. 𝟓𝟕 𝟏. 𝟘𝟕 𝟏. 𝟘𝟘 −𝟏. 𝟘𝟘

Agenda

• Introduction
• Non-linearities
• Forward pass & backpropagation
• Softmax & loss function
• Optimization & regularization

22

Forward pass § Consider this calculation: 𝑨(𝑦; 𝒙) = 2 ∗ 𝑥33 + 𝑦 ∗ 𝑥1 + 𝑥4

where 𝑦 is input and 𝒙 is the set of parameters with the initialization 𝑥" = 1 𝑥# = 3 𝑥$ = 2

§ Let’s break it into intermediary variables: 𝑏 = 𝑦 ∗ 𝑥1 𝑐 = 𝑏 + 𝑥4 𝑑 = 𝑥33 𝑨 = 𝑐 + 2 ∗ 𝑑

23

𝑥! 𝑥! = 1 𝑦 𝑥" 𝑥" = 3 𝑥# 𝑥# = 2 𝑏 = 2 ∗ 𝑦 ∗ 𝑥" 𝑑 = 𝑥## 𝑐 = 𝑏 + 𝑥! z = 𝑐 + 2 ∗ 𝑑

Computational Graph

24

𝑥! 𝑥! = 1 𝑦 𝑥" 𝑥" = 3 𝑥# 𝑥# = 2 𝑏 = 2 ∗ 𝑦 ∗ 𝑥" 𝑑 = 𝑥## 𝑐 = 𝑏 + 𝑥! z = 𝑐 + 2 ∗ 𝑑

Computational Graph

𝜖 = 1 𝜖 = 1 𝜖 = 2 ∗ 𝑥" 𝜖 = 2 𝜖 = 1 𝜖 = 2 ∗ 𝑦 𝜖 = 2 ∗ 𝑥# 𝜖 local derivatives

25

𝑥! 𝑥! = 1 𝑦 𝑥" 𝑥" = 3 𝑥# 𝑥# = 2 𝑏 = 2 ∗ 𝑦 ∗ 𝑥" 𝑑 = 𝑥## 𝑐 = 𝑏 + 𝑥! z = 𝑐 + 2 ∗ 𝑑

Forward pass

𝜖 = 1 𝜖 = 1 𝜖 = 2 𝜖 = 1 𝜖 = 2 ∗ 𝑥# 𝜖 local derivatives 𝑏 = 6 𝑐 = 7 𝑨 = 15 𝑑 = 4 𝑦 = 1 𝜖 = 2 ∗ 𝑥" 𝜖 = 2 ∗ 𝑦

26

𝑥! 𝑥! = 1 𝑦 𝑥" 𝑥" = 3 𝑥# 𝑥# = 2 𝑏 = 2 ∗ 𝑦 ∗ 𝑥" 𝑑 = 𝑥## 𝑐 = 𝑏 + 𝑥! z = 𝑐 + 2 ∗ 𝑑

Backward pass

𝜖 = 1 𝜖 = 1 𝜖 = 2 𝜖 = 1 𝜖 = 2 ∗ 𝑥# 𝜖 local derivatives 𝑏 = 6 𝑐 = 7 𝑨 = 15 𝑑 = 4 𝑦 = 1 𝜖 = 1 𝜖 = 1 𝜖 = 1 𝜖 = 4 𝜖 = 2 𝜖 = 2 ∗ 𝑥" 𝜖 = 2 ∗ 𝑦 𝜖 = 6 𝜖 = 2

27

Gradient & Chain rule § We need the gradient of 𝑨 regarding 𝒙 for optimization ∇𝒙𝑨 = 𝜖𝑨 𝜖𝑥4 𝜖𝑨 𝜖𝑥1 𝜖𝑨 𝜖𝑥3 § We calculate it using chain rule and local derivates:

IJ IK! = IJ IL IL IK! IJ IK" = IJ IL IL IM IM IK" IJ IK# = IJ IN IN IK#

28

Backpropagation

IJ IK! = IJ IL IL IK! = 1 ∗ 1 = 1 IJ IK" = IJ IL IL IM IM IK" = 1 ∗ 1 ∗ 2 = 2 IJ IK# = IJ IN IN IK# = 2 ∗ 4 = 8

Agenda

• Introduction
• Non-linearities
• Forward pass & backpropagation
• Softmax & loss function
• Optimization & regularization

30

Softmax § Given the output vector 𝒜 of a neural networks model with 𝐿 output classes § softmax turns the vector to a probability distribution

softmax(𝒜)O = 𝑓J$ ∑PQR

S

𝑓J%

normalization term

31

Softmax – numeric example § 𝐿 = 4 classes 𝒜 = 1 2 5 6 softmax(𝒜) = 0.004 0.013 0.264 0.717

𝑓% 𝑦 log(𝑦)

32

Softmax characteristics § The exponential function in softmax makes the highest value becomes separated from the others § Softmax identifies the “max” but in a “soft” way! § Softmax makes competition between the predicted

• utput values, so that in the extreme case, “winner

takes all”

• Winner-takes-all: one output is 1 and the rest are 0
• This resembles the competition between nearby neurons in the

cortex

33

Negative Log Likelihood (NLL) Loss § NLL loss function is commonly used in neural networks to optimize classification tasks:

ℒ = −𝔽𝒚,Z~𝒠 log 𝑞 𝑧 𝒚; 𝕏

• 𝒠 the set of (training) data
• 𝒚 input vector
• 𝑧 correct output class

§ NLL is a form of cross entropy loss

34

NLL + Softmax § The choice of output function (such as softmax) is highly related to the selection of loss function. These two should fit with each other! § Softmax and NLL are a good pair § To see why, let’s calculate the final NLL loss function when softmax is used at output layer (next page)

35

NLL + Softmax § Loss function for one data point: ℒ(𝑔 𝒚; 𝒙 , 𝑧) § 𝒜 the output vector of network before applying softmax § 𝑧 the index of the correct class

ℒ(𝑔 𝒚; 𝒙 , 𝑧) = − log 𝑞 𝑧 𝒚; 𝕏 = − log 𝑓J& ∑PQR

S

𝑓J% = −𝑨Z + log ∑PQR

S

𝑓J%

36

NLL + Softmax – example 2 𝒜 = 1 2 0.5 6 § If the correct class is the first one, 𝑧 = 0: ℒ = −1 + log 𝑓1 + 𝑓3 + 𝑓4.D + 𝑓E = −1 + 6.02 = 𝟔. 𝟏𝟑 § If the correct class is the third one, 𝑧 = 2: ℒ = −0.5 + log 𝑓1 + 𝑓3 + 𝑓4.D + 𝑓E = −0.5 + 6.02 = 𝟔. 𝟔𝟑 § If the correct class is the fourth one, 𝑧 = 3: ℒ = −6 + log 𝑓1 + 𝑓3 + 𝑓4.D + 𝑓E = −6 + 6.02 = 𝟏. 𝟏𝟑

37

NLL + Softmax – example 1 𝒜 = 1 2 5 6 § If the correct class is the first one, 𝑧 = 0: ℒ = −1 + log 𝑓1 + 𝑓3 + 𝑓D + 𝑓E = −1 + 6.33 = 𝟔. 𝟒𝟒 § If the correct class is the third one, 𝑧 = 2: ℒ = −5 + log 𝑓1 + 𝑓3 + 𝑓D + 𝑓E = −5 + 6.33 = 𝟐. 𝟒𝟒 § If the correct class is the fourth one, 𝑧 = 3: ℒ = −6 + log 𝑓1 + 𝑓3 + 𝑓D + 𝑓E = −6 + 6.33 = 𝟏. 𝟒𝟒

Agenda

• Introduction
• Non-linearities
• Forward pass & backpropagation
• Softmax & loss function
• Optimization & regularization

39

Stochastic Gradient Descent (SGD)

§ For every 𝑥 ∈ 𝕏 and for 𝑛 training data points 𝑥 ℒ(𝑥)

• ptimum

∇!ℒ(𝑥) −∇!ℒ(𝑥)

𝑢 𝑢 + 1

40

Stochastic Gradient Descent algorithm

§ A set of parameters 𝒙 § A learning rate 𝜃 § Loop until some exit criteria are met

• Sample a minibatch of 𝑛 data points from 𝒠
• Compute gradient (vectors) of parameters:

𝒉 ← 1 𝑛 ∇𝒙 f

#

ℒ(𝑔 𝒚(#); 𝒙 , 𝑧(𝒋))

• Update the parameters by taking a step in the opposite direction
• f the corresponding gradients:

𝒙 ← 𝒙 − 𝜃𝒉

• Reduce learning rate (annealing) if some criteria are met or

based on a schedule

41

Sampling size § If only one data point is used in every step; 𝑛 = 1

• Fast
• learns online
• Training can become unstable with a lot of fluctuations

§ If all data points are used in every step; 𝑛 = 𝑂

• Also called Batch Gradient Descent
• Training can take very long time

§ If 𝑛 is between these

• Also called Mini-Batch Gradient Descent
• Typical setting for training deep learning models

42

Other gradient-based optimizations § Limitations of the mentioned SGD algorithms

• Choosing learning rate is hard
• Choosing annealing method/rate is hard
• Same learning rate is applied to all parameters
• Can get trapped in non-optimal local minima and saddle points

§ Some other commonly used algorithms:

• Nestrov accelerated gradient
• Adagrad
• Adam

43

Regularization techniques for neural networks and deep learning § Parameter norm penalties (discussed in previous lecture) § Early stopping § Dropout § Batch normalization § Transfer learning § Multitask learning § Unsupervised / Semi-supervised pre-training § Noise robustness § Dataset augmentation § Ensemble § Adversarial training

44

Early Stopping

§ Run the model for several steps (epochs), and in each step evaluate the model on the validation set § Store the model if the evaluation results improve § At the end, take the stored model (with best validation results) as the final model

45

Dropout

Srivastava, Nitish, et al. ”Dropout: a simple way to prevent neural networks from overfitting”, JMLR 2014

§ Key idea: prune neural network by removing some hidden units stochastically § At training time for each data point:

• Each hidden unit’s output is multiplied to zero based
• n a dropout probability (like 0.6)

46

Dropout § At test time:

• All hidden units are used
• The output of each hidden is multiplied to the dropout

probability

47

Dropout – characteristics § Computationally inexpensive but a powerful method § Dropout can be viewed as a geometric average of an exponential number of networks → Ensemble § Dropout prevents hidden units from forming co- dependencies amongst each other § Every hidden unit learns to perform well regardless of

• ther units