AI and Predictive Analytics in Data-Center Environments Neural - - PowerPoint PPT Presentation

AI and Predictive Analytics in Data-Center Environments

Neural Networks and Deep Learning

Josep Ll. Berral @BSC

Intel Academic Education Mindshare Initiative for AI

Introduction

“Neural networks attempt to imitate the brain neuron mechanisms” “Actually, mathematical artifacts that learn non-linear models”

Introduction

• Neural networks: allow non-linear models

Dimensions More Dimensions

• NNs: Adjusts input weights after passing data repeatedly

Introduction

Data

Neural Net Predict Get the error Adjust

Repeat this until error is low enough

• NNs: Adjusts input weights after passing data repeatedly

Introduction

Data

Neural Net Predict Get the error Adjust

Repeat this until error is low enough Repeat this with different NN tuning until error is even lower

• NNs: Adjusts input weights after passing data repeatedly

Introduction

Data

Neural Net Predic t Get the error Adjust

Repeat this until error is low enough Repeat this with different NN tuning until error is even lower

Lots of Experiments to RUN! Parallelism is Welcome Accelerators are Welcome Any optimization is Welcome

Introduction

• Schema of Perceptron (basic linear neuron)
• Example...

Inputs x Output g(x) f(x) = ∑(xi · wi + bi ) g(x) = sigm(f(x))

yi = g(f(x)) = sigm(∑(xi · wi + bi))

Introduction

• …but simple neurons can’t learn non-linear mechanisms
• We can create networks!
• Example of XOR (non-linear)
• XOR is not linear, so requires neuron composing

Multi-Layer Perceptrons (MLP)

Input Output “hidden layer”

Training a regular NN

• Iterative process on MLPs

Repeat until acceptable error/reach iteration limit

ε = Ỹ1 - ŷ1 Input Output X ŷ1 ŷ2 ε = Y - ŷ2

Predict → ← Check error & Adjust

• Decisions:
• How many hidden layers?

Composition of NNs

h2 h1 Input Output h1 Input Output h2 h1 Input Output h3

• Decisions:
• How many neurons per layer?

Composition of NNs

h1 Input Output h1 Input Output h1 Input Output

• Decisions:
• Which kind of neuron per layer?
• Linear? Sigmoid? Rectifier? Convolutional? ...

Composition of NNs

h2 h1 Input Output convolutional + pooling linear sigmoid

• Decisions:
• Do we have any kind of “recurrence”? (back connections?)

Composition of NNs

h2 h1 Input Output

• Decisions:
• How many iterations? Any “stop” criteria...?
• Also other hyper-params! (learning rate, momentum...)

Composition of NNs

h2 h1 Input Output Predict → ← Adjust

How many “EPOCHS” (iterations)?!

Composition of NNs

• Most of these issues and decisions:

EXPERIMENTS → TRIAL/ERROR

Executions!

Details of Neural Networks

• Readability
• NNs are hard to “read”
• Can’t know always what’s happening inside
• Sometimes, weights can be visualized
• See the weights of each neuron

Details of Neural Networks

• Example: MNIST
• Hand-written number recognition

Input (28x28) HL (30 neurons) Output (10 neurons)

9

Deep Neural Networks

• Some hard problems can be solved using “Deep” NNs
• … Those are NNs with several hidden layers
• Highly complex datasets require high degree of layers + neurons
• Training require High Performance Computing
• E.g. image recognition scenarios

Input (image) 2000 neurons 1000 neurons 10000 neurons 50000 neurons Output (classes)

“cat”

conv + pool + relu line + relu line + relu + sigm conv + pool + relu conv + pool + relu + flatten

DNN & Transfer Learning

• Some trained NNs can be downloaded and tuned/adapted

Trained NN I trained this

The Internet

Trained NN I have new data

Data

Refined NN Little training

DNN & Transfer Learning

• We can re-train NNs for fine-grain tuning, even parts of a NN
• E.g.: VGG19 (image recognition NN) + tuning “style” layer (hl4)
• Transfer Learning on images:

VGG19 Update style layer Forward + reconstruction

Summary

• Neural Networks:
• Method for Non-linear modeling
• Capabilities:
• Multi-dimensional data
• Transfer Learning
• Difficulties:
• NN smithering and tinkering
• Decide architecture
• Decide hyper-params
• Checking each decision → AN EXPERIMENT TO RUN!