Improving PixelCNN Vertical stack oblem with this m of masked - - PowerPoint PPT Presentation
Improving PixelCNN Vertical stack oblem with this m of masked convolution. Blind spot Horizontal stack Solution: use two stacks of Stacking layers of masked convolution, convolution creates convolution creates a blindspot a blindspot a
66
m of masked convolution.
Blind spot
Stacking layers of masked convolution creates a blindspot Solution: use two stacks of convolution, convolution creates a blindspot a vertical stack and a horizontal stack
Horizontal stack Vertical stack
67
1 1 1 1 1 1 1 1 1 1 1 1
There is a problem with this form of masked convolution.
m of masked convolution.
Stacking layers of masked convolution creates a blindspot
68
Use more expressive nonlinearity:
essive nonlinearity: hk+1 = tanh(Wk,f ⇤ hk) σ(Wk,g ⇤ hk)
This information flow (between vertical and horizontal stacks) preserves the correct pixel dependencies
Topi Topics: cs: CIFAR-10
69
Topi Topics: cs: CIFAR-10
70
Topi Topics: cs: CIFAR-10
71
72
x1 xi xn
xn2
Convolutional Long
LSTM
Stollenga et al, 2015 Oord, Kalchbrenner, Kavukcuoglu, 2016
Row LSTM
73
x1 xi xn
xn2
Convolutional Long
LSTM
Multiple layers of convolutional LSTM
74
75
76
77
convolutions in the decoder
along the time dimension (during training or scoring)
the past
78
Masked convolution
79
80
No frame dependencies VPN Videos on nal.ai/vpn
81
No frame dependencies VPN Videos on nal.ai/vpn
82
83
Desi sign ch choice ces
rior r on th the late tent t vari riable
− Continuous, Discrete, Gaussian, Bernoulli, Mixture
Likel elihood function
− iid (static), sequential, temporal, spatial
Approximating posterior
− distribution, sequential, spatial
84
F(q) = Eqφ(z)[log pθ(x|z)] KL[qφ(z|x)kp(z)]
Variational Auto-encoder (VAE) Amortised variational inference for latent variable models
Data x
Inference Network q(z |x) z ~ q(z | x) Model p(x |z) x ~ p(x | z) z
For sca scalability and ease se of implementation
cascade of hidden stochastic layers:
85
Each term may denote a h3 h2 h1 v W3 W2 W1
Gen Proces
Input data
Input data
cascade of hidden stochastic layers:
86
Each term may denote a h3 h2 h1 v W3 W2 W1
Gen Proces
Input data
Input data
Process
Generative Process
cascade of hidden stochastic layers:
87
Each term may denote a h3 h2 h1 v W3 W2 W1
Gen Proces
Input data
Input data
Process
Generative Process
stoch chast stic layers.
tractable for each .
aluaRon is tractab ach .
Each term may denote a
cascade of hidden stochastic layers:
88
h3 h2 h1 v W3 W2 W1
Gen Proces
Input data
Input data
Process
Generative Process
stoch chast stic layers.
tractable for each .
aluaRon is tractab ach .
Each term may denote a complicated nonlinear relationship
This term denotes a o
DeterminisRc Layer StochasRc Layer StochasRc Layer
cascade of hidden stochastic layers:
89
Stochastic Layer
stoch chast stic layers.
tractable for each .
aluaRon is tractab ach .
This term denotes a one-layer neural net
Stochastic Layer Deterministic Layer
90
h3 h2 h1 v W3 W2 W1
Input data
91
h3 h2 h1 v W3 W2 W1
Input data
92
h3 h2 h1 v W3 W2 W1
Input data
recognition network (high-variance).
reparameterization trick.
with mean and covariance computed from the state of the hidden units at the previous layer.
93
with mean and covariance computed from the state of the hidden units at the previous layer.
auxi xiliar ary y var variab able:
94
95
Deterministic Encoder Distribution of does not depend on
n
96
Encoder ( ) Decoder ( ) Sample from Encoder ( ) Decoder ( ) Sample from
without reparameterization trick with reparameterization trick
Image: Carl Doersch
97
generaRve:
Gradients can be
Gradients can be computed by backprop The mapping h is a deterministic neural net for fixed
Variational inference turns integration into optimization: Au Automated Tools: :
Differentiation: Theano, Torch7, TensorFlow, Stan.
Message passing: infer.NET
easily be combined.
103
tion tion. s or
Forward pass
Prior p(z) log p(z)
Model p(x |z) log p(x|z)
Inference q(z |x) H[q(z)] z
Data x
Inference q(z |x) Model p(x |z) Prior p(z)
rθ
rφ rφ
Backward pass
Ideally want probabilistic programming using variational inference.
104
φ(x), Σq φ(x))
θ(z), Σp θ(z))
θ (z))
Model p(x |z) log p(x|z)
Prior p(z) log p(z)
Inference q(z |x) H[q(z)] z
Data x
Deep Latent Gaussian Model
F(x, q) = Eq(z)[log p(x|z)] KL[q(z)kp(z)]
All functions are deep networks.
105
Latent space disentangles the input data.
Oxygen/Swimmers Moving Left
3 dimensional latent variable of MNIST
−2 −1 1 2 3 −2 −1 1 2 3
R D p(Y|θ)
0.2 - 0.4 0.01 - 0.2 < 0.01
Factor 2 Factor 1
Latent Factor Embedding
Latent space and likelihood bound gives a visualisation
Representations are useful for strategies such as episodic control.
106
Representation
a R a1 a2 a3
Blundell, Charles, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z. Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis. "Model-Free Episodic Control.” 2016
107
108
p(x|z) = Y
i
p(xi|x<i, z) p(x|z) = Y
i
Bern(xi|f p
θ (x<i, z))
qφ(z) = Y
i
Bern(zi|f q
φ(z<i))
qφ(z) = Y
i
qφ(zi|z<i)
p(z) = Y
i
p(zi|z<i)
p(zi|z<i) = Bern(zi|f(z<i))
Deep Auto-regressive Networks
Model p(x |z) log p(x|z)
Prior p(z) log p(z)
Inference q(z |x) H[q(z)] z
Data x
Gregor, Karol, Ivo Danihelka, Andriy Mnih, Charles Blundell, and Daan Wierstra. "Deep autoregressive
Samples from binarized Atari frames
109
Gregor, Karol, Ivo Danihelka, Andriy Mnih, Charles Blundell, and Daan Wierstra. "Deep autoregressive
Visual Analogies
110
67
2.12 3.3
VAT
0.96
Aux. SS-VAE
1.06
Ladder Net SS-VAE % Classification Error (100 labels)
0.92
SS-GAN
Class Prior p(y) log p(y) Prior p(z) log p(z)
Model p(x |z,y) log p(x|z, y)
Class Inference q(z |x) Latent Inference q(y |x, z)
Data x
% Classification Err
114
p(z) = Y
i
p(zi|z<i)
qφ(z) = Y
i
qφ(zi|z<i) p(x|f p
θ (z))
pθ(x|z) = N(µp
θ(z), Σp θ(z))
Model p(x |z) log p(x|z)
Prior p(z) log p(z)
Inference q(z |x) H[q(z)] z
Data x Gregor, Karol, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. "DRAW: A recurrent neural network for image generation.” ICML 2015
LSTM or GRU networks for state modules
Spatia ial a l attent ntion ion in both the recognition and generation phase using spatial transformers.
model state.
115
Prior p(z1) State h(z) Prior p(z2) State h(z)
Prior p(zT) State h(z)
Inference q(z1⎜x)
Data x
State s(x)
Data x
State s(x) Inference q(z2⎜x) State s(x)
Data x
Inference q(zT⎜x)
Model p(x |z) log p(x|z)
Gregor, Karol, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. "DRAW: A recurrent neural network for image generation.” ICML 2015
116
Gregor, Karol, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. "DRAW: A recurrent neural network for image generation.” ICML 2015
tive Model: Stochastic Recurrent Network, chained sequence of Variational Autoencoders, with a single stochastic layer.
cognition Model: Deterministic Recurrent Network.
117
StochasRc Layer
Variational Autoencoder
(Mansimov, Parisotto, Ba, Salakhutdinov, 2015) (Gregor et al., 2015)
118
(Mansimov, Parisotto, Ba, Salakhutdinov, 2015)
A stop sign is flying in blue skies A pale yellow school bus is flying in blue skies A large commercial airplane is flying in blue skies
119
A yellow school bus parked in the parking lot A red school bus parked in the parking lot A green school bus parked in the parking lot A blue school bus parked in the parking lot
47
120
A very large commercial plane flying in clear skies. A very large commercial plane flying in rainy skies. A herd of elephants walking across a dry grass field. A herd of elephants walking across a green grass field.
48
121