Monte Carlo Methods and Neural Networks Alexander Keller, partially - - PowerPoint PPT Presentation
Monte Carlo Methods and Neural Networks Alexander Keller, partially joint work with Noah Gamboa Artificial Neural Networks in a Nutshell Supervised learning of high dimensional function approximation input layer a 0 , L 1 fully connected
input layer a0, L−1 fully connected hidden layers, and output layer aL
2
input layer a0, L−1 fully connected hidden layers, and output layer aL
– nl rectified linear units (ReLU) al,i = max{0,∑wl,j,i ·al−1,j} in layer l 2
input layer a0, L−1 fully connected hidden layers, and output layer aL
– nl rectified linear units (ReLU) al,i = max{0,∑wl,j,i ·al−1,j} in layer l – backpropagating the error δl−1,i = ∑al,j >0 δl,j ·wl,j,i 2
input layer a0, L−1 fully connected hidden layers, and output layer aL
– nl rectified linear units (ReLU) al,i = max{0,∑wl,j,i ·al−1,j} in layer l – backpropagating the error δl−1,i = ∑al,j >0 δl,j ·wl,j,i, update weights w′
l,j,i = wl,j,i −λ ·δl,j ·al−1,i if al,j > 0
2
convolutional layer: feature map defined by convolution kernel – identical weights across all neural units of one feature map max pooling layer: maximum of tile of neurons in feature map for subsampling ◮ Gradient based learning applied to document recognition ◮ Quasi-Monte Carlo feature maps for shift-invariant kernels 3
rectified linear unit ReLU(x) := max{0,x} as a basic non-linearity 5
rectified linear unit ReLU(x) := max{0,x} as a basic non-linearity for example, leaky ReLU is
5
rectified linear unit ReLU(x) := max{0,x} as a basic non-linearity for example, leaky ReLU is
5
rectified linear unit ReLU(x) := max{0,x} as a basic non-linearity for example, leaky ReLU is
hence the maximum of two values is
5
halfspace H+ with weights ˆ
6
relation to a differential equation by introduction a step size h
l
l
relation to a differential equation by introduction a step size h
l
l
al−al−1 h
7
relation to a differential equation by introduction a step size h
l
l
al−al−1 h
l
l
7
relation to a differential equation by introduction a step size h
l
l
al−al−1 h
l
l
– select your favorite ordinary differential equation to determine W (1)
l
and W (2)
l
◮ Neural networks motivated by partial differential Equations – use your favorite ordinary differential equation solver for both inference and training ◮ A radical new neural network design could overcome big challenges in AI 7
neural unit with ReLU
nl−1−1
i=0
neural unit with ReLU
nl−1−1
i=0
nl−1−1
i=0
8
neural unit with ReLU
nl−1−1
i=0
nl−1−1
i=0
1
0 wl(x,y)max{0,al−1(x)}dx
8
neural unit with ReLU
nl−1−1
i=0
nl−1−1
i=0
1
0 wl(x,y)max{0,al−1(x)}dx
recurrent neural network layer in continuous form alludes to integral equation
l(y) :=
1
0 wl(x,y)max{0,al−1(x)}dx
8
neural unit with ReLU
nl−1−1
i=0
nl−1−1
i=0
1
0 wl(x,y)max{0,al−1(x)}dx
recurrent neural network layer in continuous form alludes to integral equation
l(y) :=
1
0 wl(x,y)max{0,al−1(x)}dx +
1
0 wh l (x,y)max{0,al(x)}dx
– weights wh establish recurrence, e.g. for processing sequences of data 8
structural equivalence of integral equations and reinforcement learning learning integro-approximation from noisy/sampled data 9
structural equivalence of integral equations and reinforcement learning learning integro-approximation from noisy/sampled data examples of random sampling – pseudo-random initialization – training by stochastic gradient descent – regularization by drop-out and drop-connect – random binarization – sampling by generative adversarial networks – fixed pseudo-random matrices for direct feedback alignment ◮ Learning light transport the reinforced way ◮ Machine learning and integral equations ◮ Noise2Noise: Learning image restoration without clean data 9
dropout neuron if threshold 1
P > ξ
– ξ by linear feedback shift register generator (for example)
11
dropout neuron if threshold 1
P > ξ
– ξ by linear feedback shift register generator (for example)
11
dropout neuron if threshold 1
P > ξ
– ξ by linear feedback shift register generator (for example)
11
assign neuron to partition p = ⌊ξ ·P⌋ out of P – less random number generator calls
11
12
select inputs proportional to weight density
14
select inputs proportional to weight density
14
select inputs proportional to weight density
– remember to flip sign accordingly 14
partition of unit interval by sums Pk := ∑k
j=1 |wj| of normalized absolute weights
15
partition of unit interval by sums Pk := ∑k
j=1 |wj| of normalized absolute weights
– using a uniform random variable ξ ∈ [0,1) to
select input i ⇔ Pi−1 ≤ ξ < Pi satisfying Prob({Pi−1 ≤ ξ < Pi}) = |wi|
15
partition of unit interval by sums Pk := ∑k
j=1 |wj| of normalized absolute weights
– using a uniform random variable ξ ∈ [0,1) to
select input i ⇔ Pi−1 ≤ ξ < Pi satisfying Prob({Pi−1 ≤ ξ < Pi}) = |wi|
– use jittered equidistant samples for ξ 15
partition of unit interval by sums Pk := ∑k
j=1 |wj| of normalized absolute weights
– using a uniform random variable ξ ∈ [0,1) to
select input i ⇔ Pi−1 ≤ ξ < Pi satisfying Prob({Pi−1 ≤ ξ < Pi}) = |wi|
– use jittered equidistant samples for ξ in fact derivation of quantization to ternary weights in {−1,0,+1} – integer weights result from neurons referenced more than once – relation to drop connect and drop out 15
97% of accuracy of model by sampling most important 12% of weights
16
number of neural units
L
l=1
18
number of neural units
L
l=1
number of weights
L
l=1
18
number of neural units
L
l=1
number of weights
L
l=1
select constant number of weights per neuron – linear in n, because proportional to nw 18
19
complexity bounded by number of paths times depth L of network – relation to random walks on directed graphs and Markov chains 20
complexity bounded by number of paths times depth L of network – relation to random walks on directed graphs and Markov chains application after training – backwards random walks using sampling proportional to the weights of a neuron – compression and quantization by importance sampling 20
complexity bounded by number of paths times depth L of network – relation to random walks on directed graphs and Markov chains application after training – backwards random walks using sampling proportional to the weights of a neuron – compression and quantization by importance sampling application before training – uniform (bidirectional) random walks to connect inputs and outputs – sparse from scratch 20
sparse from scratch
21
sparse from scratch
21
sparse from scratch
– guaranteed connectivity 21
sparse from scratch
– guaranteed connectivity 21
sparse from scratch
– guaranteed connectivity 21
sparse from scratch
– guaranteed connectivity 21
sparse from scratch
– guaranteed connectivity and coverage 21
22
dropout partitions – using a fraction of the random numbers 24
dropout partitions – using a fraction of the random numbers simulating discrete densities explains {−1,0,1} and integer weights – compression and quantization without retraining 24
dropout partitions – using a fraction of the random numbers simulating discrete densities explains {−1,0,1} and integer weights – compression and quantization without retraining neural networks with linear complexity for both inference and training – sampling paths through neural networks – no more dangling neurons – sparse from scratch ◮ also watch the GTC presentation by Noah Gamboa 24