Neural Networks with Googles TensorFlow Shuo Zhang Computational - - PowerPoint PPT Presentation

Neural Networks with Google’s TensorFlow

Shuo Zhang Computational discourse analysis 11/22/16

Overview

• 1. Neural Networks basics
• 2. Neural Networks specifics
• 3. Neural Networks with Googe’s TensorFlow
• 4. Coreference: Singleton classification example

Resources

• Deep learning course (Google) @ Udacity
• Machine learning course (Stanford, Andrew Ng) @ coursera
• Neural Network course (Geoffrey Hinton) @ coursera

• 1. NN basics

From linear to non-linear classifier

Pros and cons of linear models

Pros:

• Fast
• Numerically stable
• Derivative is constant

Cons:

• Limited to modeling

additive features

• Multiplicative or higher
• rder features leas to

huge parameter space, not suitable for non- linear mapping Conclusion: We want to use parameters within linear functions but able to efficiently do non-linear mapping.

From logistic regression to neural networks

Inserting a non-linear layer: Rectified Linear Unit(ReLU)

Intuition: how NN makes non-linear mapping possible

Type of neural network

• Feed forward
• Feedback
• Self Organizing Map(SOM)
• ..

• 2. NN specifics

Multinomial logistic regression as the basic unit in NN

Softmax – turn outputs of linear functions into probability vectors

One-hot encoding

Cross entropy – measuring similarity between prediction and gold label

Putting it together again

MLR to NN

ReLU – a non-linear activation function to put in the hidden layer

ReLU is one of many choices of a non-linear activation function.

https://en.wikipedia.org/wiki/ Activation_function

Training a neural network

• Basically similar to training

a linear model by optimizing a convex function using a method like gradient descent

• Example cost function for

logistic based activation

Cost function – this is universal for linear classifier or NN

• Cost function is a function of the

parameters that captures the difference between predicted and gold label, therefore we want to minimize it.

• How to minimize? Using gradient

descent, at each iteration, adjust the weights.

• How to adjust weights? Subtracting

gradient (derivative) will move you toward the minimum.

Gradient descent

• Keep in mind that W is a matrix, so we

need to compute partial derivative with respect to each element of W, and sum them up.

Gradient Descent flavors

• Batch GD: classic approach, summing over derivative for all training examples

at each iteration in order to perform one update to weights, very slow, but more stable, almost never used today

• Stochastic GD: only takes one example at each iteration and use the gradient

computed from that example to adjust weights, fast, but less stable behavior

• Mini-batch GD: (in between) takes a mini-batch of examples (such as from 100

to 2000) and sum up those terms derivatives to perform update, balance between stability and speed (also good results), most used today

Neural Network training: forward backward propagation

Intuition from linear classifier: Repeat:

• Compute an output
• Compute error
• Adjust weights

(my implementation in Octave)

• 3. Neural Networks with Googe’s TensorFlow

https://www.youtube.com/watch?v=oZikw5k_2FM

Setup

https://www.tensorflow.org/versions/r0.11/get_started/os_setup.html

Get started

Hyper parameter tuning (loss curve)

• Number of hidden nodes
• Learning rate
• Batch size
• Number of steps
• Overfitting

Google Udacity course example:notMNIST

Example code for notMNIST dataset (Udacity)

• https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/

udacity (This set of ipython notebook is not only partial implementation, since it is meant to be an assignment to be completed. To view a complete implementation, refer to the .ipynb and html files I uploaded on the corpling server).