Biomedicine Enrico Grisan enrico.grisan@dei.unipd.it Neuron basics - PowerPoint PPT Presentation

Applied Machine Learning in Biomedicine Enrico Grisan enrico.grisan@dei.unipd.it

Neuron basics

Neuron: real and simulated

A bit of history

From biology to models

Biological models? Careful with brain analogies: Many different types of neurons Dendrites can perform complex non-linear computations Synapses are not single weights but complex dynamical dynamical system Rate code may not be adequate

Single neuron classifier

Neuron and logistic classifier 1 𝑝 𝒚 = 𝜏 𝒙 𝑈 𝒚 = 1 + 𝑓 −(𝑥 0 + 𝑗 𝑥 𝑗 𝑦 𝑗 ) 𝑦 0 𝑥 0 synapse 𝑥 0 𝑦 0 axon from a neuron dendrite cell body x 𝑔 𝑥 𝑗 𝑦 𝑗 + 𝑐 𝑥 1 𝑦 1 𝑧 𝑗 𝑥 𝑗 𝑦 𝑗 + 𝑐 𝑔 output axon 𝑗 activation 𝑥 𝑗 𝑦 𝑗 function Forward flow 𝑧 x

Linking output to input How a change in the output (loss) affects the weights? 𝑦 0 𝑥 0 synapse 𝑥 0 𝑦 0 axon from a neuron dendrite cell body 𝑔 𝑥 𝑗 𝑦 𝑗 + 𝑐 𝑥 1 𝑦 1 𝑧 ) 2 𝑀(𝒙, 𝑧) ≈ (𝑧 − 𝑗 𝑥 𝑗 𝑦 𝑗 + 𝑐 𝑔 output axon 𝑗 activation 𝑥 𝑗 𝑦 𝑗 function Backward flow 𝑧 w

Linking output to input How a change in the output (loss) affects the weights? 𝑦 0 𝑥 0 synapse 𝜖𝑔 𝜖𝑨 axon from a neuron 𝜖𝒜 𝜖𝑥 0 𝜖𝑔 dendrite 𝜖𝑔 𝜖𝑨 cell body 𝜖𝒜 𝜖𝑀 𝜖𝒜 𝜖𝑥 1 𝜖𝑔 𝜖𝑨 𝑔 𝜖𝒜 𝜖𝒜 𝜖𝒙 output axon 𝜖𝑔 𝜖𝑨 activation function 𝜖𝒜 𝜖𝑥 𝑗 Backward flow w y

Activation function Ups 1) Easy analytical derivatives 2) Squashes numbers to range [0,1] 3) Biological interpretation as saturating «firing rate» of a neuron Downs 1 1 + 𝑓 −𝑧(𝒙 𝑈 𝒚) = 𝜏 𝒙 𝑈 𝒚 = 𝜏(𝒜) 1) Saturated neurons kill the gradients 2) Sigmoid output are not zero- centered

Sigmoid backpropagation Assume the input of a neuron is always positive. What about the gradient on 𝒙 ? 𝑔 𝑥 𝑗 𝑦 𝑗 + 𝑐 𝑗 𝒙 𝑢+1 = 𝒙 𝑢 + 𝛼 𝒙 𝑔 Gradient is all positive or all negative!

Improving activation function Ups 1) Still analytical derivatives 2) Squashes numbers to range [-1,1] 3) Zero-centered! tanh 𝑦 = 𝑓 𝑦 − 𝑓 −𝑦 𝑓 𝑦 + 𝑓 −𝑦 Downs 1) Saturated neurons kill the 𝑒 tanh 𝑦 gradients = 1 − 𝑢𝑏𝑜ℎ 2 (𝑦) 𝑒𝑦

Activation function 2 Rectifying Linear Unit: ReLU Ups 1) Does not saturate 2) Computationally efficient 3) Converges faster in practice 𝑔 𝑦 = max(0, 𝑦) Downs 1) What happens for x<0? 𝑒𝑔 𝑒𝑦 = 1 𝑦 > 0 0 𝑦 < 0

ReLU neuron killing

Activation function 3 Leaky ReLU Ups 1) Does not saturate 2) Computationally efficient 3) Converges faster in practice 4) Keep neurons alive! 𝑔 𝑦 = 𝐼 −𝑦 𝛽𝑦 + 𝐼 𝑦 𝑦 Downs 𝑒𝑔 𝑒𝑦 = 1 𝑦 > 0 𝛽 𝑦 < 0

Activation function 4 Ups 1) Does not saturate Maxout 2) Computationally efficient 3) Linear regime 4) Keeps neurons alive! 5) Generalizes ReLU and leaky 𝑔 𝑦 = max(𝒙 1𝑈 𝒚, 𝒙 2𝑈 𝒚) ReLU Downs 1) Is not a dot product 2) Doubles the parameters

Neural Networks: architecture

Neural Networks: architecture 2-layers Neural Network 3-layers Neural Network 1-hidden layer Neural Network 2-hidden layers Neural Network

Neural Networks: architecture Number of neurons? Number of weights? Number of parameters?

Neural Networks: architecture Number of neurons: 4+2=6 Number of weights: 4x3+2x4=20 Number of parameters: 20+6

Neural Networks: architecture Number of neurons: 4+2=6 Number of neurons: 4+4+1=9 Number of weights: 4x3+2x4=20 Number of weights: 4x3+4x4+1x4=32 Number of parameters: 20+6 Number of parameters: 32+9

Neural Networks: architecture Modern CNNs: ~10 million artificial neurons Human Visual Cortex: ~5 billion neurons

ANN representation 𝑥 0,𝑗 𝑥 0,𝑗 𝑥 1,𝑗 𝑥 1,𝑗 𝑥 2,𝑗 𝒙 𝟐,𝒋 = 𝒙 𝟑,𝒋 = 𝑥 2,𝑗 𝑥 3,𝑗 𝑥 3,𝑗 𝑥 4,𝑗 𝒙 1,1 𝒙 2,1 𝑦 1 𝑦 2 𝒙 2,2 𝑦 3 𝒙 1,4 𝑈 𝒚 1 1 𝑦 11 ⋯ 𝑦 13 𝑈 ⋯ 𝑦 23 1 𝑦 21 𝒚 2 ⋯ 𝑦 33 𝒀 = = 1 𝑦 31 𝑈 𝒚 3 ⋯ ⋮ ⋮ ⋮ 1 𝑦 𝑂1 ⋯ 𝑦 𝑂3 𝑈

ANN representation 𝑈 𝒚 1 1 𝑦 11 ⋯ 𝑦 13 𝑈 ⋯ 𝑦 23 1 𝑦 21 𝒚 2 ⋯ 𝑦 33 𝒀 = 1 𝑦 31 = 𝑈 𝒚 3 ⋯ ⋮ ⋮ ⋮ 1 𝑦 𝑂1 ⋯ 𝑦 𝑂3 𝑈 𝒚 𝑂 𝒂 𝟐 𝒀 = 𝑔 1 𝑿 𝟐 𝒀 𝑥 0,1 𝑥 0,2 𝑥 0,3 𝑥 1,1 𝑥 1,2 𝑥 1,3 𝒁 𝒀 = 𝑔 2 𝑿 𝟑 𝒂 1 = 𝑔 2 𝑿 2 𝑔 1 𝑿 1 𝒀 = 𝒙 𝟐,𝟐 𝒙 𝟐,𝟑 𝒙 𝟐,𝟒 𝑿 1 = 𝑥 2,1 𝑥 2,2 𝑥 2,3 𝑥 3,1 𝑥 3,2 𝑥 3,3 𝑿 2 = 𝑥 1,2 𝑥 2,2

ANN becoming popular

ANN training: forward flow Define a loss function 𝑀 𝒙 𝑈 ; 𝑧 Each neuron computes: 𝑏 𝑘 = 𝑗 𝑥 𝑘𝑗 𝑨 𝑗 And pass to the following layer: 𝑨 𝑘 = 𝑔(𝑏 𝑘 )

ANN training: backward flow Need to compute: 𝜖𝑏 𝑘 𝜖𝑀 = 𝜖𝑀 = 𝜀 𝑘 𝑨 𝑗 𝜖𝑥 𝜖𝑏 𝑘 𝜖𝑥 𝑘𝑗 𝑘𝑗 Considering that for the output neurons: 𝜀 𝑙 = 𝑧 𝑙 − 𝑧 𝑙 We get: 𝜀 𝑘 = 𝑔′(𝑏 𝑘 ) 𝑥 𝑙𝑘 𝜀 𝑙 𝑙

ANN training: backpropagation Updating all weights: 𝑘𝑗 𝑢+1 = 𝑥 𝑘𝑗 𝑢 + 𝜃𝜀 𝑥 𝑘

ANN training ex: forward 𝑔 𝑏 = tanh(𝑏) 𝐿 𝑀 𝑜 = 1 𝑧 𝑙 − 𝑧 𝑙 2 2 𝑙=1 𝐸 (1) 𝑦 𝑗 𝑏 𝑘 = 𝑥 𝑗𝑘 𝑗=0 𝑨 𝑘 = tanh(𝑏 𝑘 ) 𝑁 (2) 𝑨 𝑧 𝑙 = 𝑥 𝑙𝑘 𝑘 𝑘=1

ANN training ex: backward 𝜀 𝑙 = 𝑧 𝑙 − 𝑧 𝑙 𝐿 𝑘2 ) 𝜀 𝑘 = (1 − 𝑨 𝑥 𝑙𝑘 𝜀 𝑙 𝑙=1 𝜖𝑀 𝑜 𝑘𝑗(1) = 𝜀 𝑘 𝑦 𝑗 𝜖𝑥 𝜖𝑀 𝑜 𝜖𝑥 𝑙𝑘 (2) = 𝜀 𝑙 𝑨 𝑘

What can ANN represent?

What can ANN classify?

Regularization