Transformation Equivariance vs. Invariance: Unsupervised Learning of - - PowerPoint PPT Presentation

Transformation Equivariance vs. Invariance: Unsupervised Learning of Visual Representations

Guo-Jun Qi guojunq@gmail.com Laboratory for MAchine Perception and LEarning (MAPLE) & Futurewei Technologies (Huawei Research USA)

1 Laboratory for MAchine Perception and LEarning (MAPLE)

Contents

• TER: Transformation Equivariant Representations
• Definition, Steerability
• AET: AutoEncoding Transformations
• Deterministic approach: AET (AutoEncoding Transformations)
• Probabilistic approach: AVT (Autoencoding Variational Transformations)
• SAT: (Semi-)supervised Autoencoding Transformations
• Conclusions and Future Work
• Unifying the Transformation Equivariance and Invariance

2 Laboratory for MAchine Perception and LEarning (MAPLE)

Contents

• TER: Transformation Equivariant Representations
• Definition, Steerability
• AET: AutoEncoding Transformations
• Deterministic approach: AET (AutoEncoding Transformations)
• Probabilistic approach: AVT (Autoencoding Variational Transformations)
• SAT: (Semi-)supervised Autoencoding Transformations
• Conclusions and Future Work

3 Laboratory for MAchine Perception and LEarning (MAPLE)

Recipe in Success of CNNs

Horse Grass Tree … Classifier Convolution layers Translation Equivariant Representation Fully Connected Classifier

Visual structures Semantic concepts

CNN = Translation-Equivariant Representation + Fully-Connected classifier

4 Laboratory for MAchine Perception and LEarning (MAPLE)

Transformation Equivariant Representations

• Beyond translations: equivariant feature maps under transformations

5

Various transformations Representations

Laboratory for MAchine Perception and LEarning (MAPLE)

Generalize CNNs beyond translations

Horse Grass Tree … FC classifier Representati

• n

Transformation Equivariance Transformation Invariance Spatial Semantic

6 Laboratory for MAchine Perception and LEarning (MAPLE)

Transformation Equivariant Representations + Transformation Invariant Classifiers

Contents

• TER: Transformation Equivariant Representations
• Definition, Steerability
• AET: AutoEncoding Transformations
• Deterministic approach: AET (AutoEncoding Transformations)
• Probabilistic approach: AVT (Autoencoding Variational Transformations)
• SAT: (Semi-)supervised Autoencoding Transformations
• Conclusions and Future Work

7 Laboratory for MAchine Perception and LEarning (MAPLE)

Transformation Equivariance

• Definition of transformation equivariance
• 𝑭 -- the representation of a sample
• 𝐮 -- a transformation on samples
• 𝝇 𝐮 -- the representation transformation corresponding to 𝝇.
• Transformation invariance is a special case of transformation

equivariance when 𝝇 𝐮 is an identity.

𝐹 𝐮 (𝐲) = 𝝇 𝐮 𝐹(𝐲)

8 Laboratory for MAchine Perception and LEarning (MAPLE)

Steerability property

• Steerability: a transformed sample 𝐮 (𝐲) can be represented directly

from the representation 𝐹 𝐲 of original sample, with no access to 𝐲

• 𝝇(𝐮) is a function of the transformation 𝐮, independently of sample.

𝐹 𝐮 (𝐲) = 𝝇 𝐮 [𝐹(𝐲)]

9 Laboratory for MAchine Perception and LEarning (MAPLE)

• For general transformations
• unnecessarily limited to discrete or spatial transformations
• Recoloring, contrasting, etc.
• Nonlinear representations between transformed and original images
• Capturing complex visual structures from transformed images

𝐹 𝐮 (𝐲) = 𝝇 𝐮 [𝐹(𝐲)]

Our Goals

Nonlinear transformations 𝝇 𝐮 on representations

10 Laboratory for MAchine Perception and LEarning (MAPLE)

Contents

• TER: Transformation Equivariant Representations
• Definition, Steerability
• AET: AutoEncoding Transformations
• Deterministic approach: AET (AutoEncoding Transformations)
• Probabilistic approach: AVT (Autoencoding Variational Transformations)
• SAT: (Semi-)supervised Autoencoding Transformations
• Conclusions and Future Work

11 Laboratory for MAchine Perception and LEarning (MAPLE)

A Big Picture: Stack of AET

CNN Group Equivariant CNN SAT AED AET Autoencoders CNNs

SAT: (Semi-)Supervised Autoencding Transformations

Capsule Net

• AET learns a general representation that can be applied everywhere.

AVT

AET: AutoEncoding Transformations

Transformation Equivariant Representations Deterministic AET Deterministic SAT Probabilistic SAT Probabilistic AVT

AVT: Autoencoding Variational Transformations

Stack of Autoencoding Transformations for learning TER

12 Laboratory for MAchine Perception and LEarning (MAPLE)

Contents

• TER: Transformation Equivariant Representations
• Definition, Steerability
• AET: AutoEncoding Transformations
• Deterministic approach: AET (AutoEncoding Transformations)
• Probabilistic approach: AVT (Autoencoding Variational Transformations)
• SAT: (Semi-)supervised Autoencoding Transformations
• Conclusions and Future Work

13 Laboratory for MAchine Perception and LEarning (MAPLE)

Take A Glance: Autoencoding Transformations than Data

E D E(x) E(t(x)) t(x) x 𝐮 𝐮 E E x D E(x) 𝐲 AutoEncoding Data (AED) AutoEncoding Transformations (AET)

Zhang et al., AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transformations rather than Data, in CVPR 2019.

14 Laboratory for MAchine Perception and LEarning (MAPLE)

How does AET work?

• Generative Process:
• An input image x ~ p(x)
• A random transformation t ~ p(t)
• The transformed t(x)
• A representation encoder
• E: x ⟼ E(x), t(x) ⟼ E(t(x))
• A transformation decoder
• D: (x , t(x)) ⟼ D(x , t(x))

E D E(x) E(t(x)) t(x) x 𝐮

𝐮

E

15 Laboratory for MAchine Perception and LEarning (MAPLE)

Decoding Transformations

• A Siamese network individually encodes representations of images
• E.g., Visual structures, and spatial relations among objects
• Decoding by comparing the representations before and after

transformations.

E D E(x) E(t(x)) t(x) x 𝐮 𝐮 E

16 Laboratory for MAchine Perception and LEarning (MAPLE)

AET loss for training

• Parameterized Transformations: 𝒰= {tθ|θ ~ Ө}

E.g. affine or projective: 𝑚 𝐮θ, 𝐮 θ =

1 2 ||𝑁 θ − 𝑁(

θ)||2

2

• GAN-Induced Transformations: transformed image G(x, z)

𝑚 𝐮z, 𝐮 z = 1 2 ||𝐴 − z||2

2

• Non-Parametric Transformations

𝑚 𝐮, 𝐮 = 1 2 𝔽𝐲~𝑌dist(𝐮 𝐲 , 𝐮(𝐲))

17 Laboratory for MAchine Perception and LEarning (MAPLE)

Contents

• TER: Transformation Equivariant Representations
• Definition, Steerability
• AET: AutoEncoding Transformations
• Deterministic approach: AET (AutoEncoding Transformations)
• Probabilistic approach: AVT (Autoencoding Variational Transformations)
• SAT: (Semi-)supervised Autoencoding Transformations
• Conclusions and Future Work

18 Laboratory for MAchine Perception and LEarning (MAPLE)

Steerability of z through 𝝇 𝐮 and 𝐴

Revisit: Steerability of TER

• Obtain the representation 𝐴 of a transformed sample 𝐮(𝐲) from 𝐮 and

𝐴 without accessing x

• Maximizing the mutual information between 𝐴 and (

𝐴, 𝐮)

𝐹 𝐮 (𝐲) = 𝝇 𝐮 [𝐹(𝐲)] 𝐮 𝐴 t(x)

transformation

𝐴

19 Laboratory for MAchine Perception and LEarning (MAPLE)

x

An Information-Theoretical Insight

• Train a TER model 𝜾 by maximizing
• By chain rule of mutual information, we have
• 𝑱𝜾 𝐴;

𝐴, 𝐮 attains its maximum value 𝑱𝜾 𝐴; 𝐴, 𝐮, 𝐲 (the upper bound) when

• Nonlinearity of transformation 𝝇 𝐮 in representations.

max𝜾 𝑱𝜾(𝐴; 𝐴, 𝐮) 𝑱𝜾 𝐴; 𝐴, 𝐮 = 𝑱𝜾 𝐴; 𝐴, 𝐮, 𝐲 − 𝑱𝜾 𝐴; 𝐲| 𝐴, 𝐮 ≤ 𝑱𝜾 𝐴; 𝐴, 𝐮, 𝐲 𝑱𝜾 𝐴; 𝐲| 𝐴, 𝐮 = 𝟏

Steerability: Given ( 𝐴, 𝐮), x contains no more information about z.

x 𝐮 𝐴 t(x) transformation 𝐴

20 Laboratory for MAchine Perception and LEarning (MAPLE)

AVT: Autoencoding Variational Transformations

• Unable to maximize the mutual information directly
• Intractable to evaluate the posterior 𝑞𝜄(𝐮|𝐴, 𝐲)
• Deriving a lower bound by introducing a transformation decoder 𝑟𝝔
• Unsupervised loss to learn AVT

𝑱𝜾 𝐴; 𝐴, 𝐮 ≥ 𝐼(𝐮| 𝐴) + 𝔽𝒒𝜾 𝒖,𝒜,

𝐴 log𝑟𝝔(𝒖|𝒜,

𝐴) max

𝜾,𝝔 𝔽𝒒𝜾 𝐮,𝐴, 𝐴 log𝑟𝝔(𝒖|𝒜,

𝐴)

21

Qi, Learning Generalized Transformation Equivariant Representations via Autoencoding Transformations, preprint.

Laboratory for MAchine Perception and LEarning (MAPLE)

AVT: Autoencoding Variational Transformations

• Generative process
• Given an image x sampled from p(x)
• Sample a transformation t from p(t)
• Apply t to x, resulting in t(x)
• Sample a representation z of t(x) from

𝑞𝜄(𝐴|𝐲, 𝐮)

• 𝐴 is sampled by setting t to an identity
• Decode transformations

𝐮 from 𝑟𝜚(𝐮|𝐴, 𝐴)

t(x) x 𝐮 𝐮 𝑞𝜄(𝐴|𝐲, 𝟐) 𝑟𝜚(𝐮|𝐴, 𝐴) 𝑞𝜄(𝐴|𝐲, 𝐮) 𝐴 𝐴 AVT

22 Laboratory for MAchine Perception and LEarning (MAPLE)

Deterministic vs. Probabilistic Approaches

• AVT can handle uncertainty in
• A probabilistic Representing of images: 𝑞𝜄(𝐴|𝐲, 𝐮)
• Decoding transformations with a probabilistic decoder: 𝑟𝜚(𝐮|𝐴,

𝐴)

t(x) x 𝐮 𝐮 𝑞𝜄(𝐴|𝐲, 𝟐) 𝑟𝜚(𝐮|𝐴, 𝐴) 𝑞𝜄(𝐴|𝐲, 𝐮) 𝐴 𝐴 Probabilistic AVT Deterministic AET E D z=E(t(x)) t(x) x 𝐮 𝐮 E

23 Laboratory for MAchine Perception and LEarning (MAPLE)

𝐴 = 𝐹(𝐲)

Experiments for AET and AVT

• Unsupervised Representation Learning
• AET and AVT
• Evaluation Protocols
• Test learned representations on downstream tasks
• Datasets
• CIFAR10
• ImageNet
• Places

Transformation Equivariant Representations Deterministic AET Deterministic SAT Probabilistic SAT Probabilistic AVT

Stack of Autoencoding Transformations for learning TER

24 Laboratory for MAchine Perception and LEarning (MAPLE)

Transformations in Experiments

• Affine transformations
• Projective transformations

25

Random rotation [-180o, 180o] Translation ±0.2*H/W Scale 0.7 – 1.3 Scale 0.8 – 1.2 Random rotation 0o, 90o,180o, 270o Stretch each corner ±0.125*H/W

Laboratory for MAchine Perception and LEarning (MAPLE)

CIFAR10 and ImageNet

26

Method Error rate Supervised NIN (Lower Bound) 7.20 Random Init. + conv (upper Bound) 27.50 Roto-Scat + SVM 17.7 ExamplarCNN 15.7 DCGAN 17.2 Scattering 15.3 RotNet + FC 10.94 RotNet + Conv 8.84 (Ours) AET-affine + FC 9.77 (Ours) AET-affine + conv 8.05 (Ours) AET-project + FC 9.41 (Ours) AET-project + conv 7.82 (ours) AVT-project + FC 8.96 (ours) AVT-project + conv 7.75 Method Conv4 Conv5 ImageNet Labels (Upper Bound) 59.7 59.7 Random (Lower Bound) 27.1 12.0 Tracking 38.8 29.8 Context 45.6 30.4 Colorization 40.7 35.2 Jigsaw Puzzles 45.3 34.6 BiGAN 41.9 32.2 NAT

• 36.0

DeepCluster

• 44.0

RotNet 50.0 43.8 (Ours) AET-project 53.2 47.0 (Ours) AVT-project 54.2 48.4

Error rates on CIFAR10 Top 1 Accuracy on ImageNet

Projective transformation is slightly better than the affine.

Laboratory for MAchine Perception and LEarning (MAPLE)

Model-Free KNN on Representations

• Purely test the learned representations without training models
• Error rates on CIFAR10

27

K= 3 5 10 15 20 RotNet baseline 25.67 25.01 24.97 25.85 26.00 AET-affine 24.88 23.29 23.07 23.34 23.94 AET-project 23.29 22.40 22.39 23.32 23.75 AVT-project 22.46 21.62 23.70 22.16 21.51

Laboratory for MAchine Perception and LEarning (MAPLE)

PASCAL VOC: Object Detection

• A milestone: even better than fully supervised counterpart

Laboratory for MAchine Perception and LEarning (MAPLE) 28

Contents

• TER: Transformation Equivariant Representations
• Definition, Steerability
• AET: AutoEncoding Transformations
• Deterministic approach: AET (AutoEncoding Transformations)
• Probabilistic approach: AVT (Autoencoding Variational Transformations)
• SAT: (Semi-)supervised Autoencoding Transformations
• Conclusions and Future Work

29 Laboratory for MAchine Perception and LEarning (MAPLE)

Supervised Autoencoding Transformations

• Beyond Translation Equivariance and CNNs
• Transformation Equivariant Representations + Transformation Invariant Classifiers

30

Dog

Laboratory for MAchine Perception and LEarning (MAPLE)

SAT: Semi-Supervised Autoencoding Transformation

• Adding a label decoder 𝑟𝜚(𝐳|

𝐴)to approximate the posterior 𝑞𝜄(𝐳|𝐲)

Horse Grass Tree …

t(x) x 𝐮 𝐮 𝑟𝜚(𝐳| 𝐴) 𝐳 𝑞𝜄(𝐴|𝐲, 𝟐) 𝑟𝜚(𝐮|𝐴, 𝐴) 𝑞𝜄(𝐴|𝐲, 𝐮) 𝐴 𝐴

Encoder Transformation decoder Label decoder

31

Qi, Learning Generalized Transformation Equivariant Representations via Autoencoding Transformations, in preprint.

Laboratory for MAchine Perception and LEarning (MAPLE)

Training SAT

• Seek to maximize the mutual information between (𝐳, 𝐴) and (

𝐴, 𝐮)

• Assumption: (

𝐴, 𝐮) contains sufficient information to decode both label y and representation z of transformed images

• Intractable due to the difficulty in evaluating the posterior

𝑞𝜄(𝐳, 𝐮|𝐴, 𝐴) max

𝜄

𝐽𝜄(𝐳, 𝐴; 𝐴, 𝐮)

32 Laboratory for MAchine Perception and LEarning (MAPLE)

• By introducing label decoder and transformation decoder, we have

𝐽𝜄 𝐳, 𝐴; 𝐴, 𝐮 ≥ 𝔽𝑞𝜄(𝐳,𝐴,

𝐴) log 𝑟𝜚 𝐳 𝐴,

𝐴 + 𝔽𝑞𝜄(𝐮,𝐴,

𝐴) log 𝑟𝜚 𝐮 𝐴,

𝐴

• Jointly maximizing over encoder 𝜄 and decoders 𝜚

max

𝜄,𝜚 𝔽𝑞𝜄(𝐳,𝐴, 𝐴) log 𝑟𝜚 𝐳 𝐴,

𝐴 + 𝔽𝑞𝜄(𝐮,𝐴,

𝐴) log 𝑟𝜚 𝐮 𝐴,

𝐴

Variational Bound

Label decoder Transformation decoder

33 Laboratory for MAchine Perception and LEarning (MAPLE)

A Simple Case: Deterministic SAT

• Replace probabilistic encoder and decoders with deterministic ones
• Supervised AET

Horse Grass Tree …

t(x) x 𝐮 𝐮 𝐷𝜚( 𝑨) 𝐳 𝐹𝜄(𝐲) 𝐸𝜚(𝐴, 𝐴) 𝐹𝜄(𝐮 𝐲 ) 𝐴 𝐴

Encoder Decoders Label decoder

𝐴

max

𝜄,𝜚 𝔽 𝐲,𝐳 𝑚CrossEntropy 𝐳, 𝐷𝜚

𝐴 + 𝔽 𝐮,𝐲 𝑚AET 𝐮, 𝐸𝜚 𝐴, 𝐴

34 Laboratory for MAchine Perception and LEarning (MAPLE)

Experiments on SAT

• (Semi-)supervised learning: SAT
• Evaluation Protocols
• Follow semi-supervised protocols with varying number of labels
• Datasets
• CIFAR10
• SVHN

35

Transformation Equivariant Representations Deterministic AET Deterministic SAT Probabilistic SAT Probabilistic AVT

Stack of Autoencoding Transformations for learning TER

Laboratory for MAchine Perception and LEarning (MAPLE)

Error Rate on CIFAR10

Methods 1000 labels 2000 labels 4000 labels All GAN 18.63 ±2.32 Π model 12.36 ± 0.31 5.56 ± 0.10 Temporal Ensembling 12.16 ± 0.31 5.60 ± 0.10 VAT 10.55 Supervised-only 46.43 ± 1.21 33.94 ± 0.73 20.66 ± 0.57 5.81 ± 0.15 Π model 27.36 ± 1.20 18.02 ± 0.60 13.20 ± 0.27 6.06 ± 0.11 Mean Teacher 21.55 ± 1.48 15.73 ± 0.31 12.31 ± 0.28 5.94 ± 0.15 SAT 14.89 ± 0.38 11.71 ± 0.29 9.85 ± 0.11 4.91 ± 0.13

36 Laboratory for MAchine Perception and LEarning (MAPLE)

Error Rate on SVHN

37

Methods 250 labels 500 labels 1000 labels All GAN 18.63 ±2.32 Π model 12.36 ± 0.31 5.56 ± 0.10 Temporal Ensembling 12.16 ± 0.31 5.60 ± 0.10 VAT 10.55 Supervised-only 46.43 ± 1.21 33.94 ± 0.73 20.66 ± 0.57 5.81 ± 0.15 Π model 27.36 ± 1.20 18.02 ± 0.60 13.20 ± 0.27 6.06 ± 0.11 Mean Teacher 21.55 ± 1.48 15.73 ± 0.31 12.31 ± 0.28 5.94 ± 0.15 SAT 14.89 ± 0.38 11.71 ± 0.29 9.85 ± 0.11 4.91 ± 0.13

Laboratory for MAchine Perception and LEarning (MAPLE)

Contents

• TER: Transformation Equivariant Representations
• Definition, Steerability
• AET: AutoEncoding Transformations
• Deterministic approach: AET (AutoEncoding Transformations)
• Probabilistic approach: AVT (Autoencoding Variational Transformations)
• SAT: (Semi-)supervised Autoencoding Transformations
• Conclusions and Future Work

38 Laboratory for MAchine Perception and LEarning (MAPLE)

Future Works

• Evolutionalize representation learning and various high-level tasks
• More powerful in capturing the (non-)linear equivariant (invariant) visual

structures against transformations

39

Transformation Equivariant Representations Deterministic AET Deterministic SAT Probabilistic SAT Probabilistic AVT

Classification Detection Segmentation Face recogntion Pose estimation Super Resolutions

Representation Learning Foundation High-level tasks

Laboratory for MAchine Perception and LEarning (MAPLE)

Future Work: Unifying Transformation Equivariance vs. Invariance

Laboratory for MAchine Perception and LEarning (MAPLE) 40

equivariance equivariance equivariance Transformation Equivariance discerns the difference between transformed images, thus more advantageous in modeling spatial structures for object detection and semantic segmentation. Push apart

Future Work: Unifying Transformation Equivariance vs. Invariance (cont’d)

Laboratory for MAchine Perception and LEarning (MAPLE) 41

invariance invariance invariance Pull together Transformation invariance (e.g., in contrastive learning) pulls together the embeddings of transformed images, while discriminating between different instances, thus more suitable for image classification tasks.

Future Work: Unifying Transformation Equivariance vs. Invariance (cont’d)

• A naive approach (yet to be done) by combining
• AET loss for transformation equivariance
• Contrastive loss for transformation invariance
• Open questions
• How can both transformation equivariance and invariance reconcile in

learning feature embeddings?

• Will the new approach excel on both spatial sensitive tasks (object detection

and semantic segmentation) and image classification tasks?

Laboratory for MAchine Perception and LEarning (MAPLE) 42

Guo-Jun Qi and Jiebo Luo, Small Data Challenges in Big Data Era: A Survey of Recent Progress on Unsupervised and Semi-Supervised Methods, arXiv:1903.11260

Conclusions

• Transformation Equivariant Representation
• Plays key role in the success of
• Generalizes to any (Continuous, discrete, and beyond spatial)

transformations

• Nonlinear, steerable representations of transformations
• Unsupervised: AutoEncoding Transformations
• Unsupervised Learning: AET and AVT
• (Semi-)supervised learning: SAT
• (Semi-)Supervised: Generalizing CNNs beyond translations
• Transformation Equivariant Representations + Transformation Invariant

Classifier

• With invariance contained in equivariance as a special case

43 Laboratory for MAchine Perception and LEarning (MAPLE)

References

Preprint

• Qi, Learning Generalized Transformation Equivariant Representations

via Autoencoding Transformations, in preprint. (long version) Conference Publications

• Zhang et al., AET vs. AED: Unsupervised Representation Learning by

Auto-Encoding Transformations rather than Data, in CVPR 2019. (AET)

• Qi et al., AVT: Unsupervised Learning of Transformation Equivariant

Representations by Autoencoding Variational Transformations, in ICCV

• 2019. (AVT)

44 Laboratory for MAchine Perception and LEarning (MAPLE)

Thank you! Q&A

45 Laboratory for MAchine Perception and LEarning (MAPLE)

Recent Progresses

Laboratory for MAchine Perception and LEarning (MAPLE) 46

Unsupervised Learning of Transformation Equivariant Representations via Auto-Encoding Node-wise Transformations

Auto-Encoding Node-wise Transformations

• Apply transformations to individual nodes of a graph
• Apply transformations to sampled nodes
• Study different parts of a graph each time
• Reduce the transformation parameters
• Global vs. local transformations
• Isotropic vs. anisotropic transformations

Laboratory for MAchine Perception and LEarning (MAPLE) 47

Auto-Encoding Node-wise Transformations

• Decode the transformations at each

node

• Learn the representation of individual

nodes on both local and global scale

• Global structures are learned by

sampling nodes from different parts of graphs

• Local structures are covered by

integrating information through edge convolutions on nearby nodes

Laboratory for MAchine Perception and LEarning (MAPLE) 48

Results

• Datasets: cloud data of 3D points
• ModelNet40: graph classication
• 12,311 meshed CAD model
• 40 categories
• ShapeNet part: graph segmentations
• 16,881 3D point clouds
• 16 categories
• Two tasks
• Graph classification
• Graph segmentation

Laboratory for MAchine Perception and LEarning (MAPLE) 49

Results

• Classification

Laboratory for MAchine Perception and LEarning (MAPLE) 50

• Segmentation

Results

• Compared with both supervised and unsupervised graph

segmentations, our approach is much more competitive.

Laboratory for MAchine Perception and LEarning (MAPLE) 51

AETv2: AutoEncoding Transformations by Minimizing Geodesic Distances in Lie Groups of Transformations

52 Laboratory for MAchine Perception and LEarning (MAPLE)

Recent Progresses

Mean-Squared Estimation vs. Geodesic Distance Minimization

• (Matrix representation of ) Transformations

stay in a curved manifold of Lie group rather than in a flat Euclidean space

• MSE minimizes the Euclidean space between

predicted and groudtruth transformations

• Not represent how a transformation continuously

evolves to another one in the Lie group

• Geodesic distance minimization (GDM)

represent a “real” distance between transformations

Laboratory for MAchine Perception and LEarning (MAPLE) 53

Geodesic distance between Transformations

• Given two (matrix form of) transformations

𝐔 and 𝐔, the geodesic distance between them is ℓ 𝐔, 𝐔 = 1 2 LogI 𝐔−1 𝐔

F

where LogI is the Riemannian logarithm not the matrix logarithm.

Laboratory for MAchine Perception and LEarning (MAPLE) 54

• Riemannian exp at I maps a point

in tangent space to Lie group

• Riemannian logarithm at I maps

back to the tangent space.

Homography Transformations

• For many spatial transformations, there is no closed-form Riemannian

logarithm.

• Homography transformations achieves the SOTA accuracy in AETv1.
• Solution
• Use a subgroup of transformation with tractable Riemannian logarithm

(which reduces to matrix logarithm)

• Project the transformations to the subgroup
• Minimize the resultant geodesic distance in the subgroup (together with the

projection loss)

Laboratory for MAchine Perception and LEarning (MAPLE) 55

Projection into SO(3) subgroup

• Riemannian logarithm reduces to matrix logarithm in SO(3)
• Project 𝐔−1

𝐔 to SO(3) 𝐐 = ΠSO 3 (𝐔−1 𝐔)

• By Rodrigues’ rotation formula,

𝜄 = arccos[tr 𝐐 − 1 2 ] is the rotation angle around a unit 3D axis in the direction of log 𝐐

Laboratory for MAchine Perception and LEarning (MAPLE) 56

Objective in SO(3) subgroup

• Given two (matrix form of)

transformations 𝐔 and 𝐔, ℓ 𝐔, 𝐔 = 𝜄 + 𝜇 𝐒Π

𝐺 2

Where 𝐒Π = 𝐔−1 𝐔 − ΠSO 3 (𝐔−1 𝐔) is the projection residual

Laboratory for MAchine Perception and LEarning (MAPLE) 57

ImageNet Results

• Better than AETv1 (MSE)

Laboratory for MAchine Perception and LEarning (MAPLE) 58