On Learning Sets of Symmetric Elements [1] [2] [1,3] [3] Haggai - - PowerPoint PPT Presentation

ICML 2020

On Learning Sets of Symmetric Elements

Haggai Maron, Or Litany, Gal Chechik, Ethan Fetaya

[1] Nvidia Research [2] Stanford University [3] Bar-Ilan University

[1] [2] [1,3] [3]

Motivation and Overview

Set Symmetry

{ }

Input set Previous work (DeepSets, PointNet) targeted training a deep network over sets

Deep Net

x1 x2 ⋮ xm , x1 x2 ⋮ xm , x1 x2 ⋮ xm , …

Set+Elements symmetry

{ }

Input set

Main challenge: What architecture is optimal when elements of the set have their own symmetries?

Deep Net

, , …

Both the set and its elements have symmetries.

Deep Symmetric sets

{ }

Input image set Output

Set symmetry: Order invariance/equivariance

{ }

=

Set symmetry: Order invariance/equivariance

{ }

=

{ }

Set symmetry: Order invariance/equivariance

{ }

=

{ }

Element symmetry: Translation invariance/equivariance

{ }

{ } { }

Element symmetry: Translation invariance/equivariance

{ } { }

Element symmetry: Translation invariance/equivariance

Applications

Modalities 1D signals 2D images 3D pointclouds Graph

This paper

A principled approach for learning sets of complex elements (graphs, point clouds, images) Characterize maximally expressive linear layers that respect the symmetries (DSS layers) Prove universality results Experimentally demonstrate that DSS networks outperform baselines

Previous work

Deep sets [Zaheer et al. 2017]

CNN CNN CNN

Siamese

Deep sets [Zaheer et al. 2017]

CNN CNN CNN

Features Siamese

Deep sets [Zaheer et al. 2017]

CNN CNN CNN Deep Sets

Features Siamese Deeps sets block

Deep sets [Zaheer et al.]

Previous work: information sharing

Aittala and Durand, ECCV 2018 Sridhar et al., NeuriPS 2019 Liu et al., ICCV 2019

CNN CNN CNN CNN CNN CNN

Information sharing layer

Our approach

Invariance

Many Learning tasks are invariant to natural transformations (symmetries) More formally. Let be a subgroup: is invariant if , for all e.g. image classification

H ≤ Sn f : ℝn → ℝ f(τ ⋅ x) = f(x) τ ∈ H

τ f f

“Cat”

Equivariance

Let be a subgroup: Equivariant if , e.g. edge detection

H ≤ Sn f(τ ⋅ x) = τ ⋅ f(x)

τ τ f f

Invariant neural networks

⋯

Equivariant FC Invariant

• Invariant by construction

Deep Symmetric Sets

with symmetry group Want to be invariant/equivariant to both and the ordering Formally the symmetry group is

x1, …, xn ∈ ℝd G ≤ Sd G H = Sn × G ≤ Snd

G

Main challenges

• What is the space of linear equivariant layers for specific

?

H = SN × G

• What is the space of linear equivariant layers for a given

?

• Can we compute these operators efficiently?

H = SN × G

Main challenges

• What is the space of linear equivariant layers for a given

?

• Can we compute these operators efficiently?
• Do we lose expressive power?

H = SN × G

• invariant networks

H

• invariant continuous functions

H

Continuous functions

Main challenges

• What is the space of linear equivariant layers for a given

?

• Can we compute these operators efficiently?
• Do we lose expressive power?

H = SN × G

• invariant networks

H

• invariant continuous functions

H

Continuous functions Gap?

Main challenges

Theorem: Any linear −equivariant layer is of the form where are linear

• equivariant functions

We call these layers Deep Sets for Symmetric elements layers (DSS)

SN × G L : ℝn×d → ℝn×d L(X)i = LG

1 (xi) + ∑ j≠i

LG

2 (xj)

LG

1 , LG 2

G

• equivariant layers

H

DSS for images

are images is the group of circular translations

• equivariant layers are convolutions

x1, …, xn G 2D G

Single DSS layer

DSS for images

are images is the group of circular translations

• equivariant layers are convolutions

x1, …, xn G 2D G

CONV CONV CONV

Single DSS layer

DSS for images

are images is the group of circular translations

• equivariant layers are convolutions

x1, …, xn G 2D G

CONV CONV CONV

+

Single DSS layer

DSS for images

are images is the group of circular translations

• equivariant layers are convolutions

x1, …, xn G 2D G

CONV CONV CONV CONV

+

Single DSS layer

DSS for images

Siamese part Information sharing part

CONV CONV CONV CONV

+ +

Single DSS layer

Expressive power

Theorem If G-equivariant networks are universal appoximators for G-equivariant functions, then so are DSS networks for

• equivariant functions.

SN × G

Expressive power

Theorem If G-equivariant networks are universal appoximators for G-equivariant functions, then so are DSS networks for

• equivariant functions.
• Main tool:
• Noether’s Theorem (Invariant theory)
• For any finite group

, the ring of invariant polynomials is finitely generated.

• Generators can be used to create continuous unique encodings for elements in

SN × G H ℝ[x1, . . . , xn]H ℝn×d/H

Results

Signal classification

Image selection

Shape selection

Conclusions

A general framework for learning sets of complex elements Generalizes many previous works Expressivity results Works well in many tasks and data types