AMMI Introduction to Deep Learning 9.1. Transposed convolutions - PowerPoint PPT Presentation

AMMI – Introduction to Deep Learning 9.1. Transposed convolutions Fran¸ cois Fleuret https://fleuret.org/ammi-2018/ Fri Nov 9 22:39:08 UTC 2018 ÉCOLE POLYTECHNIQUE FÉDÉRALE DE LAUSANNE

Constructing deep generative architectures requires layers to increase the signal dimension, the contrary of what we have done so far with feed-forward networks. Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 1 / 14

Constructing deep generative architectures requires layers to increase the signal dimension, the contrary of what we have done so far with feed-forward networks. Generative processes that consist of optimizing the input rely on back-propagation to expend the signal from a low-dimension representation to the high-dimension signal space. Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 1 / 14

Constructing deep generative architectures requires layers to increase the signal dimension, the contrary of what we have done so far with feed-forward networks. Generative processes that consist of optimizing the input rely on back-propagation to expend the signal from a low-dimension representation to the high-dimension signal space. The same can be done in the forward pass with transposed convolution layers whose forward operation corresponds to a convolution layer backward pass. Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 1 / 14

Consider a 1d convolution with a kernel κ y i = ( x ⊛ κ ) i � = x i + a − 1 κ a a � = x u κ u − i +1 . u Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 2 / 14

Consider a 1d convolution with a kernel κ y i = ( x ⊛ κ ) i � = x i + a − 1 κ a a � = x u κ u − i +1 . u We get � ∂ 퓁 � = ∂ 퓁 ∂ x ∂ x u u ∂ 퓁 ∂ y i � = ∂ y i ∂ x u i ∂ 퓁 � = κ u − i +1 . ∂ y i i which looks a lot like a standard convolution layer, except that the kernel coefficients are visited in reverse order. Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 2 / 14

This is actually the standard convolution operator from signal processing. If ∗ denotes this operation, we have � ( x ∗ κ ) i = x a κ i − a +1 . a Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 3 / 14

This is actually the standard convolution operator from signal processing. If ∗ denotes this operation, we have � ( x ∗ κ ) i = x a κ i − a +1 . a Coming back to the backward pass of the convolution layer, if y = x ⊛ κ then � ∂ 퓁 � ∂ 퓁 � � = ∗ κ. ∂ x ∂ y Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 3 / 14

In the deep-learning field, since it corresponds to transposing the weight matrix of the equivalent fully-connected layer, it is called a transposed convolution . κ 1 0 0 0 0   T 0 0 0 0 0 0 0 κ 1 κ 2 κ 3 κ 2 κ 1     0 κ 1 κ 2 κ 3 0 0 0 κ 3 κ 2 κ 1 0 0       0 0 κ 1 κ 2 κ 3 0 0 = 0 κ 3 κ 2 κ 1 0         0 0 0 κ 1 κ 2 κ 3 0 0 0 κ 3 κ 2 κ 1       0 0 0 0 κ 1 κ 2 κ 3 0 0 0 κ 3 κ 2   0 0 0 0 κ 3 Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 4 / 14

In the deep-learning field, since it corresponds to transposing the weight matrix of the equivalent fully-connected layer, it is called a transposed convolution . κ 1 0 0 0 0   T 0 0 0 0 0 0 0 κ 1 κ 2 κ 3 κ 2 κ 1     0 κ 1 κ 2 κ 3 0 0 0 κ 3 κ 2 κ 1 0 0       0 0 κ 1 κ 2 κ 3 0 0 = 0 κ 3 κ 2 κ 1 0         0 0 0 κ 1 κ 2 κ 3 0 0 0 κ 3 κ 2 κ 1       0 0 0 0 κ 1 κ 2 κ 3 0 0 0 κ 3 κ 2   0 0 0 0 κ 3 While a convolution can be seen as a series of inner products, a transposed convolution can be seen as a weighted sum of translated kernels. Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 4 / 14

Convolution layer Input 1 4 -1 0 2 -2 1 3 3 1 W Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 5 / 14

Convolution layer Input 1 4 -1 0 2 -2 1 3 3 1 W Kernel 1 2 0 -1 w Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 5 / 14

Convolution layer Input 1 4 -1 0 2 -2 1 3 3 1 W 1 2 0 -1 w Output 9 W − w + 1 Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 5 / 14

Convolution layer Input 1 4 -1 0 2 -2 1 3 3 1 W 1 2 0 -1 w Output 9 0 W − w + 1 Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 5 / 14

Convolution layer Input 1 4 -1 0 2 -2 1 3 3 1 W 1 2 0 -1 w Output 9 0 1 W − w + 1 Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 5 / 14

Convolution layer Input 1 4 -1 0 2 -2 1 3 3 1 W 1 2 0 -1 w Output 9 0 1 3 W − w + 1 Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 5 / 14

Convolution layer Input 1 4 -1 0 2 -2 1 3 3 1 W 1 2 0 -1 w Output 9 0 1 3 -5 W − w + 1 Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 5 / 14

Convolution layer Input 1 4 -1 0 2 -2 1 3 3 1 W 1 2 0 -1 w Output 9 0 1 3 -5 -3 W − w + 1 Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 5 / 14

Convolution layer Input 1 4 -1 0 2 -2 1 3 3 1 W 1 2 0 -1 w Output 9 0 1 3 -5 -3 6 W − w + 1 Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 5 / 14

Convolution layer Input 1 4 -1 0 2 -2 1 3 3 1 W Kernel 1 2 0 -1 w Output 9 0 1 3 -5 -3 6 W − w + 1 Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 5 / 14

Transposed convolution layer Input 2 3 0 -1 W Kernel 1 2 -1 w Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 6 / 14

Transposed convolution layer Input 2 3 0 -1 W 1 2 -1 2 4 -2 Output 2 W + w − 1 Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 6 / 14

Transposed convolution layer Input 2 3 0 -1 W 1 2 -1 2 4 -2 3 6 -3 Output 2 7 W + w − 1 Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 6 / 14

Transposed convolution layer Input 2 3 0 -1 W 1 2 -1 2 4 -2 3 6 -3 0 0 0 Output 2 7 4 W + w − 1 Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 6 / 14

Transposed convolution layer Input 2 3 0 -1 W 1 2 -1 2 4 -2 3 6 -3 0 0 0 -1 -2 1 Output 2 7 4 -4 -2 1 W + w − 1 Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 6 / 14

Transposed convolution layer Input 2 3 0 -1 W 2 4 -2 3 6 -3 0 0 0 -1 -2 1 Output 2 7 4 -4 -2 1 W + w − 1 Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 6 / 14

Transposed convolution layer Input 2 3 0 -1 W Kernel 1 2 -1 w Output 2 7 4 -4 -2 1 W + w − 1 Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 6 / 14

torch.nn.functional.conv_transpose1d implements the operation we just described. It takes as input a batch of multi-channel samples, and produces a batch of multi-channel samples. >>> x = torch.tensor([[[0., 0., 1., 0., 0., 0., 0.]]]) >>> k = torch.tensor([[[1., 2., 3.]]]) >>> F.conv1d(x, k) tensor([[[ 3., 2., 1., 0., 0.]]]) = ⊛ Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 7 / 14

torch.nn.functional.conv_transpose1d implements the operation we just described. It takes as input a batch of multi-channel samples, and produces a batch of multi-channel samples. >>> x = torch.tensor([[[0., 0., 1., 0., 0., 0., 0.]]]) >>> k = torch.tensor([[[1., 2., 3.]]]) >>> F.conv1d(x, k) tensor([[[ 3., 2., 1., 0., 0.]]]) = ⊛ >>> F.conv_transpose1d(x, k) tensor([[[ 0., 0., 1., 2., 3., 0., 0., 0., 0.]]]) ∗ = Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 7 / 14

The class torch.nn.ConvTranspose1d embeds that operation into a torch.nn.Module . >>> x = torch.tensor([[[ 2., 3., 0., -1.]]]) >>> m = nn.ConvTranspose1d(1, 1, kernel_size=3) >>> m.bias.data.zero_() tensor([ 0.]) >>> m.weight.data.copy_(Tensor([ 1, 2, -1 ])) tensor([[[ 1., 2., -1.]]]) >>> y = m(x) >>> y tensor([[[ 2., 7., 4., -4., -2., 1.]]]) Fran¸ cois Fleuret AMMI – Introduction to Deep Learning / 9.1. Transposed convolutions 8 / 14