Understanding image representations by measuring their equivariance - - PowerPoint PPT Presentation

Understanding image representations by measuring their equivariance and equivalence

Karel Lenc, Andrea Vedaldi

Visual Geometry Group, Department of Engineering Science

Representations for image understanding

𝜔 image space feature space semantic space

𝜚(𝒚) 𝜚(𝒛) 𝜚(𝒜) 𝒚 𝒛

𝜚 𝜚 𝜚 bike 𝜔 bike 𝜔 dog representation classifier

2 Ultimate goal of a representation: simplify a task such as image classification Many representations

▶

Local image descriptors

SIFT [Lowe 04], HOG [Dalal et al. 05], SURF [Bay et al. 06], LBP [Ojala et al. 02], …

▶

Feature encoders

BoVW [Sivic et al. 02], Fisher Vector [Perronnin et al. 07], VLAD [Jegou et al. 10], sparse coding, …

▶

Deep convolutional neural networks

[Fukushima 1974-1982, LeCun et al. 89, Krizhevsky et al. 12, …]

𝒜

Design of representations

Many designs are empirical, the main theoretical design principle is invariance 3

𝜚 𝑕 𝜚

image space feature space representation

𝒚 𝑕𝒚

Invariant 𝜚 𝒚 = 𝜚(𝑕𝒚)

Design of representations

However, many representations such as HOG are not invariant, even to simple transformations 4

𝑕 𝒚 𝑕𝒚

Not invariant 𝜚 𝒚 ≠ 𝜚(𝑕𝒚)

image space feature space representation HOG HOG

≠

Design of representations

But they often transform in a simple and predictable manner 5

𝑕 𝒚 𝑕𝒚

Equivariant ∀𝒚: 𝜚 𝒚 = 𝑁𝑕 𝜚(𝑕𝒚)

image space feature space representation HOG HOG

Design of representations

But what happens with more complex transformations like affine ones? 6

𝑕 𝒚 𝑕𝒚

image space feature space representation HOG HOG

?

Design of representations

What happens with more complex representations like CNNs?

Invariance of CNN rep. studied in. [Goodfellow et al. 09] or [Zeiler, Fergus 13]

7

𝑕 𝒚 𝑕𝒚

Contribution: transformations in CNNs

image space feature space representation CNN CNN

?

Representation properties Equivariance

How does a representation reflect image transformations? 8 𝜚 𝜚 ?

When are two representations the same?

Learning representations means that there is an endless number of them Variants obtained by learning on different datasets, or different local optima 9

𝒚

Equivalence 𝜚𝐶 𝒚 = 𝐹 𝜚𝐵(𝒚)

representations CNN-A CNN-B

Representation properties Equivariance

How does a representation reflect image transformations? 10 𝜚 𝜚 ? 𝜚𝐶 𝜚𝐵 ?

Equivalence

Do different representations have different meanings?

Regularized linear regression

Finding equivariance empirically

11

𝜚 𝜚 𝑁𝑕

≃

𝑕 𝐵𝑕𝜚 𝒚 + 𝑐𝑕

𝜚(𝐲)

𝑁𝑕 𝜚 𝒚 𝜚 𝑕𝒚 (learned empirically)

Convolutional structure

Finding equivariance empirically

12

𝜚 𝜚 𝑁𝑕 𝑕 𝜚 𝑕𝒚

permutation convolution by 1⨉1 filter bank

∗ 𝐵𝑕 𝜚 𝑕𝒚 𝐵𝑕𝜚 𝒚 + 𝑐𝑕 (learned empirically)

≃

HOG features

Finding equivariance empirically

13

𝜚 𝜚 𝑁𝑕 Rotation 45º 𝜚 𝜚 𝑁𝑕 𝜚 𝜚 𝑁𝑕

≃ ≃ ≃

HOG features – inverse with MIT HOGgles [Vondrick et al. 13]

Finding equivariance empirically

14

𝜚 𝜚 𝑁𝑕 𝜚 𝜚 𝑁𝑕 𝜚 𝜚 𝑁𝑕

≃ ≃ ≃

Rotation 45º

Finding equivariance empirically

15

𝜚 𝜚 𝑁𝑕 𝜚 𝜚 𝑁𝑕 𝜚 𝜚 𝑁𝑕

≃ ≃ ≃

1.25x Upscale HOG features – inverse with MIT HOGgles [Vondrick 13]

Equivariance of representations

Transformations

▶

scaling, rotation, flipping, translation Equivariant representations

▶

HOG Findings 16

Finding equivariance empirically

We run the same analysis on a typical CNN architecture

▶

AlexNet [Krizevsky et al. 12]

▶

5 convolutional layers + fully-connected layers

▶

Trained on ImageNet ILSVRC 17 CNN case convolutional layers fully-connected layers

𝒚

1 2 3 4 5 FC

Label 𝑧 dog 𝜚 𝜔

CNN case

Learning mappings empirically

18

𝒚 𝜚(𝒚)

1 2 3 4 5 FC

1 2 3 4 5 2 3 4 5 FC

Label 𝑧 dog 𝒚

Classif. Loss ℓ

Label 𝑧 𝜚 𝜔

CNN case

Learning mappings empirically

19

𝑕𝒚 𝑁𝑕−1𝜚(𝑕𝒚) 𝒚 𝜚(𝑕𝒚) 𝑕

𝑁𝑕−1

𝐷𝑝𝑜𝑤3

1 2 3 4 5 FC

𝒚

1 2 3 4 5 FC

Label 𝑧 dog

Classif. Loss ℓ

Label 𝑧

learned empirically

Results – Vertical Flip

𝑕

1 2345 12 345 123 45 1234 5 12345

𝑁𝑕−1

𝐷𝑝𝑜𝑤1

𝑁𝑕−1

𝐷𝑝𝑜𝑤2

𝑁𝑕−1

𝐷𝑝𝑜𝑤3

𝑁𝑕−1

𝐷𝑝𝑜𝑤4 𝑁𝑕−1 𝐷𝑝𝑜𝑤5

Original Classif., no TF Original Classif. + TF Before Training

1 2 3 4 5 FC

𝑕𝒚 After Training 𝑁𝑕−1

𝐷𝑝𝑜𝑤3

1 2 3 4 5 FC

𝑕𝒚

1 2 3 4 5 FC

𝒚

1 2 3 4 5 FC

𝑕𝒚 ∗

20

10 20 30 40 50 60 Top-5 Error [%] Original Classif., no TF Original Classif. + TF Before Training After Training

Equivariance of representations

Transformations

▶

scaling, rotation, flipping, translation Equivariant representations

▶

HOG

▶

Early convolutional layers in CNNs Equivariant to a lesser degree

▶

Deeper convolutional layers in CNNs Findings 21

Representation properties Equivariance

How does a representation reflect image transformations? 22 𝜚 𝜚 ? 𝜚𝐶 𝜚𝐵 ?

Equivalence

Do different representations have different meanings?

CNN transplantation crash course

Equivalence

23

CNN-A 𝜚 Are 𝜚 and 𝜚′ equivalent features?

1 2 3 4 5 FC

CNN-B 𝜚′

1 2 3 4 5 FC

AlexNet [Krizhevsky et al. 12], same training data, different parametrization

5 FC 1 2 3 4

24

stitching layer (linear convolution) CNN transplantation crash course

Equivalence

Classif. Loss ℓ

Label 𝑧 Same training data, different parametrization CNN-A

1 2 3 4 5 FC

CNN-B

1 2 3 4 5 FC

Train with SGD ∗ 𝐹

Stitch CNN-A  CNN-B

Franken-network

Training data is the same, but parametrization is entirely different

25

10 20 30 40 50 60 70 80 90 100 Top-5 Error [%]

1 2 3 4 5 FC

Baseline

1 2 3 4 5 FC

Before Training

1 2 3 4 5 FC

𝐹 After Training

CNN-B CNN-B CNN-B CNN-A CNN-A

1 2345 12 345 123 45 1234 5 12345

𝐹𝜚→𝜚′

𝐷𝑝𝑜𝑤1

𝐹𝜚→𝜚′

𝐷𝑝𝑜𝑤2

𝐹𝜚→𝜚′

𝐷𝑝𝑜𝑤3

𝐹𝜚→𝜚′

𝐷𝑝𝑜𝑤4

𝐹𝜚→𝜚′

𝐷𝑝𝑜𝑤5

Compare training on the same or different data

Equivalence of similar architecture

26

Places dataset ILSVRC12 dataset CNN-IMNET CNN-PLACES

1 2 3 4 5 FC 1 2 3 4 5 FC

Stitch CNN-PLACES  CNN-IMNET

Franken-network

Now even the training sets differ 27

10 20 30 40 50 60 70 80 90 100 Top-5 Error [%]

Baseline

CNN-IMNET

1 2 3 4 5 FC

Before Training

CNN-IMNET CNN-PLCS

1 2 3 4 5 FC

𝐹 After Training

CNN-IMNET CNN-PLCS

1 2 3 4 5 FC

1 2345 12 345 123 45 1234 5 12345

𝐹𝜚→𝜚′

𝐷𝑝𝑜𝑤1

𝐹𝜚→𝜚′

𝐷𝑝𝑜𝑤2

𝐹𝜚→𝜚′

𝐷𝑝𝑜𝑤3

𝐹𝜚→𝜚′

𝐷𝑝𝑜𝑤4

𝐹𝜚→𝜚′

𝐷𝑝𝑜𝑤5

Structured-output pose detection

Example application

Equivariant maps – Transform features instead of images Allows significant speedup at test time 28

𝑕∗ = argmax𝑕 ∈ 𝐻 𝒙, 𝜚 𝑕−1𝒚 = argmax𝑕 ∈ 𝐻 𝒙, 𝑁𝑕−1𝜚 𝒚

Conclusions

Representing geometry

▶

Beyond invariance: equivariance

▶

Transforming the image results in a simple and predictable transformation

• f HOG and early CNN layers

▶

Application to accelerated structured output regression Representation equivalence

▶

CNN trained from different random seeds are very different, but only on the surface

▶

Early CNN layers are interchangeable even between tasks General idea

▶

study mathematical properties of representations empirically 29

References

[Lowe 04] Lowe, David G. "Distinctive image features from scale-invariant keypoints." International journal of computer vision 60.2 (2004): 91-110. [Dalal et al. 05] Dalal, Navneet, and Bill Triggs. "Histograms of oriented gradients for human detection." Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on. Vol. 1. IEEE, 2005. [Bay et al. 06] Bay, Herbert, Tinne Tuytelaars, and Luc Van Gool. "Surf: Speeded up robust features." Computer vision– ECCV 2006. Springer Berlin Heidelberg, 2006. 404-417. [Ojala et al. 02] Ojala, Timo, Matti Pietikainen, and Topi Maenpaa. "Multiresolution gray-scale and rotation invariant texture classification with local binary patterns." Pattern Analysis and Machine Intelligence, IEEE Transactions on 24.7 (2002): 971-987. [Sivic et al. 02] Sivic, Josef, and Andrew Zisserman. "Video Google: A text retrieval approach to object matching in videos." Computer Vision, 2003. Proceedings. Ninth IEEE International Conference on. IEEE, 2003. [Perronnin et al. 07] Perronnin, Florent, and Christopher Dance. "Fisher kernels on visual vocabularies for image categorization." Computer Vision and Pattern Recognition, 2007. CVPR'07. IEEE Conference on. IEEE, 2007. [Jegou et al. 10] H. Jegou, M. Douze, C. Schmid, and P. Perez. Aggregating local descriptors into a compact image

• representation. In Proc. CVPR, 2010.

[LeCun et al. 98] LeCun, Yann, et al. "Gradient-based learning applied to document recognition." Proceedings of the IEEE 86.11 (1998): 2278-2324. [Krizhevsky et al. 12] Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems. 2012. [Goodfellow et al. 09] Goodfellow, Ian, et al. "Measuring invariances in deep networks." Advances in neural information processing systems. 2009. [Zeiler, Fergus 13] Zeiler, Matthew D., and Rob Fergus. "Visualizing and Understanding Convolutional Networks." arXiv preprint arXiv:1311.2901 (2013). [Vondrick et al. 13] C. Vondrick, A. Khosla, T. Malisiewicz, A. Torralba. "HOGgles: Visualizing Object Detection Features" International Conference on Computer Vision (ICCV), Sydney, Australia, December 2013. [Simonyan et al. 14] K. Simonyan, A. Vedaldi, A. Zisserman. “Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps.” ICLR Workshop 2014

30