Biomedicine Enrico Grisan enrico.grisan@dei.unipd.it Neuron basics - - PowerPoint PPT Presentation

Applied Machine Learning in Biomedicine

Enrico Grisan enrico.grisan@dei.unipd.it

Neuron basics

Neuron: real and simulated

A bit of history

From biology to models

Biological models?

Careful with brain analogies: Many different types of neurons Dendrites can perform complex non-linear computations Synapses are not single weights but complex dynamical dynamical system Rate code may not be adequate

Single neuron classifier

Neuron and logistic classifier

𝑝 𝒚 = 𝜏 𝒙𝑈𝒚 = 1 1 + 𝑓−(𝑥0+ 𝑗 𝑥𝑗𝑦𝑗) Forward flow x x 𝑧

cell body activation function

• utput axon

axon from a neuron synapse dendrite

𝑥𝑗𝑦𝑗

𝑗

𝑥𝑗𝑦𝑗 + 𝑐 𝑔

𝑥0𝑦0 𝑥1𝑦1 𝑦0 𝑥0

𝑔

𝑗

𝑥𝑗𝑦𝑗 + 𝑐

𝑧

Linking output to input

How a change in the output (loss) affects the weights?

𝑔

𝑗

𝑥𝑗𝑦𝑗 + 𝑐 cell body activation function

• utput axon

axon from a neuron synapse dendrite

𝑥𝑗𝑦𝑗

𝑗

𝑥𝑗𝑦𝑗 + 𝑐 𝑔

𝑥0𝑦0 𝑥1𝑦1 𝑦0 𝑥0

Backward flow w 𝑧 𝑀(𝒙, 𝑧) ≈ (𝑧 − 𝑧 ) 2

Linking output to input

cell body activation function

• utput axon

axon from a neuron synapse dendrite 𝑔

𝑦0 𝑥0

Backward flow w y

𝜖𝑔 𝜖𝒜

𝜖𝑔 𝜖𝒜 𝜖𝑨 𝜖𝒙 𝜖𝑔 𝜖𝒜 𝜖𝑨 𝜖𝑥0 𝜖𝑔 𝜖𝒜 𝜖𝑨 𝜖𝑥1 𝜖𝑔 𝜖𝒜 𝜖𝑨 𝜖𝑥𝑗

How a change in the output (loss) affects the weights?

𝜖𝑀 𝜖𝒜

Activation function

1 1 + 𝑓−𝑧(𝒙𝑈𝒚) = 𝜏 𝒙𝑈𝒚 = 𝜏(𝒜) Ups 1) Easy analytical derivatives 2) Squashes numbers to range [0,1] 3) Biological interpretation as saturating «firing rate» of a neuron Downs 1) Saturated neurons kill the gradients 2) Sigmoid output are not zero- centered

Sigmoid backpropagation

Assume the input of a neuron is always positive. What about the gradient on 𝒙? 𝑔

𝑗

𝑥𝑗𝑦𝑗 + 𝑐 𝒙𝑢+1 = 𝒙𝑢 + 𝛼

𝒙𝑔

Gradient is all positive or all negative!

Improving activation function

Ups 1) Still analytical derivatives 2) Squashes numbers to range [-1,1] 3) Zero-centered! Downs 1) Saturated neurons kill the gradients tanh 𝑦 = 𝑓𝑦 − 𝑓−𝑦 𝑓𝑦 + 𝑓−𝑦 𝑒 tanh 𝑦 𝑒𝑦 = 1 − 𝑢𝑏𝑜ℎ2(𝑦)

Activation function 2

Rectifying Linear Unit: ReLU

Ups 1) Does not saturate 2) Computationally efficient 3) Converges faster in practice Downs 1) What happens for x<0? 𝑔 𝑦 = max(0, 𝑦) 𝑒𝑔 𝑒𝑦 = 1 𝑦 > 0 𝑦 < 0

ReLU neuron killing

Activation function 3

Leaky ReLU

Ups 1) Does not saturate 2) Computationally efficient 3) Converges faster in practice 4) Keep neurons alive! Downs 𝑔 𝑦 = 𝐼 −𝑦 𝛽𝑦 + 𝐼 𝑦 𝑦 𝑒𝑔 𝑒𝑦 = 1 𝑦 > 0 𝛽 𝑦 < 0

Activation function 4

Maxout

Ups 1) Does not saturate 2) Computationally efficient 3) Linear regime 4) Keeps neurons alive! 5) Generalizes ReLU and leaky ReLU Downs 1) Is not a dot product 2) Doubles the parameters 𝑔 𝑦 = max(𝒙1𝑈𝒚, 𝒙2𝑈𝒚)

Neural Networks: architecture

Neural Networks: architecture

2-layers Neural Network 1-hidden layer Neural Network 3-layers Neural Network 2-hidden layers Neural Network

Neural Networks: architecture

Number of neurons? Number of weights? Number of parameters?

Neural Networks: architecture

Number of neurons: 4+2=6 Number of weights: 4x3+2x4=20 Number of parameters: 20+6

Neural Networks: architecture

Number of neurons: 4+2=6 Number of weights: 4x3+2x4=20 Number of parameters: 20+6 Number of neurons: 4+4+1=9 Number of weights: 4x3+4x4+1x4=32 Number of parameters: 32+9

Neural Networks: architecture

Modern CNNs: ~10 million artificial neurons Human Visual Cortex: ~5 billion neurons

ANN representation

𝑦1 𝑦2 𝑦3 𝒙1,1 𝒙1,4 𝒙2,1 𝒙2,2

𝒀 = 1 𝑦11 ⋯ 𝑦13 1 𝑦21 1 𝑦31 ⋮ ⋯ ⋯ ⋯ 𝑦23 𝑦33 ⋮ 1 𝑦𝑂1 ⋯ 𝑦𝑂3 = 𝒚1

𝑈

𝒚2

𝑈

𝒚3

𝑈

⋮

𝑈

𝒙𝟐,𝒋 = 𝑥0,𝑗 𝑥1,𝑗 𝑥2,𝑗 𝑥3,𝑗 𝒙𝟑,𝒋 = 𝑥0,𝑗 𝑥1,𝑗 𝑥2,𝑗 𝑥3,𝑗 𝑥4,𝑗

ANN representation

𝒀 = 1 𝑦11 ⋯ 𝑦13 1 𝑦21 1 𝑦31 ⋮ ⋯ ⋯ ⋯ 𝑦23 𝑦33 ⋮ 1 𝑦𝑂1 ⋯ 𝑦𝑂3 = 𝒚1

𝑈

𝒚2

𝑈

𝒚3

𝑈

⋮ 𝒚𝑂

𝑈 𝑿1 =

𝑥0,1 𝑥1,1 𝑥2,1 𝑥3,1 𝑥0,2 𝑥1,2 𝑥2,2 𝑥3,2 𝑥0,3 𝑥1,3 𝑥2,3 𝑥3,3 = 𝒙𝟐,𝟐 𝒙𝟐,𝟑 𝒙𝟐,𝟒 𝑿2 = 𝑥1,2 𝑥2,2 𝒂𝟐 𝒀 = 𝑔

1 𝑿𝟐𝒀

𝒁 𝒀 = 𝑔

2 𝑿𝟑𝒂1 = 𝑔 2 𝑿2𝑔 1 𝑿1𝒀

ANN becoming popular

ANN training: forward flow

Define a loss function 𝑀 𝒙𝑈; 𝑧 Each neuron computes: 𝑏𝑘 = 𝑗 𝑥

𝑘𝑗𝑨𝑗

And pass to the following layer: 𝑨

𝑘 = 𝑔(𝑏𝑘)

ANN training: backward flow

Need to compute: 𝜖𝑀 𝜖𝑥

𝑘𝑗

= 𝜖𝑀 𝜖𝑏𝑘 𝜖𝑏𝑘 𝜖𝑥

𝑘𝑗

= 𝜀

𝑘𝑨𝑗

Considering that for the output neurons: 𝜀𝑙 = 𝑧𝑙 − 𝑧𝑙 We get: 𝜀

𝑘 = 𝑔′(𝑏𝑘) 𝑙

𝑥𝑙𝑘𝜀𝑙

ANN training: backpropagation

Updating all weights: 𝑥

𝑘𝑗𝑢+1 = 𝑥 𝑘𝑗𝑢 + 𝜃𝜀 𝑘

ANN training ex: forward

𝑔 𝑏 = tanh(𝑏) 𝑀𝑜 = 1 2

𝑙=1 𝐿

𝑧𝑙 − 𝑧𝑙 2 𝑏𝑘 =

𝑗=0 𝐸

𝑥𝑗𝑘

(1)𝑦𝑗

𝑨

𝑘 = tanh(𝑏𝑘)

𝑧𝑙 =

𝑘=1 𝑁

𝑥𝑙𝑘

(2)𝑨 𝑘

ANN training ex: backward

𝜀𝑙 = 𝑧𝑙 − 𝑧𝑙 𝜀

𝑘 = (1 − 𝑨 𝑘2) 𝑙=1 𝐿

𝑥𝑙𝑘𝜀𝑙 𝜖𝑀𝑜 𝜖𝑥

𝑘𝑗(1) = 𝜀 𝑘𝑦𝑗

𝜖𝑀𝑜 𝜖𝑥𝑙𝑘(2) = 𝜀𝑙𝑨

𝑘

What can ANN represent?

What can ANN classify?

Regularization