SLIDE 1
On the Universality of Invariant Networks
Haggai Maron Ethan Fetaya Nimrod Segol Yaron Lipman
1
Invariant tasks
- Image classification
2
Car Car
On the Universality of Invariant Networks Haggai Maron Ethan - - PowerPoint PPT Presentation
On the Universality of Invariant Networks Haggai Maron Ethan - - PowerPoint PPT Presentation
1 On the Universality of Invariant Networks Haggai Maron Ethan Fetaya Nimrod Segol Yaron Lipman 2 Invariant tasks Image classification Car Car 3 Invariant tasks Image classification Car Car Graph/ hyper-graph
SLIDE 2
SLIDE 3
Invariant tasks
- Image classification
- Graph/ hyper-graph classification
3
!" !# !$ !% !& !' !" !" !' !' !% !% !$ !$ !& !& !# !# !' !# !$ !% !& !" !" !" !' !' !% !% !$ !$ !& !& !# !#
Car Car
SLIDE 4
Invariant tasks
- Image classification
- Graph/ hyper-graph classification
- Point-cloud / set classification
- and many more…
4
!" !# !$ !% !& !' !" !" !' !' !% !% !$ !$ !& !& !# !# !' !# !$ !% !& !" !" !" !' !' !% !% !$ !$ !& !& !# !#
3 " ($%, '%, (%) ($*, '*, (
*)Car Car
SLIDE 5
Goal of this paper
- Invariant neural networks are a common approach for these tasks
- This paper analyzes the expressive power of invariant models
5
!" !# !$ !% !& !' !" !" !' !' !% !% !$ !$ !& !& !# !# !' !# !$ !% !& !" !" !" !' !' !% !% !$ !$ !& !& !# !#
3 " ($%, '%, (%) ($*, '*, (
*)Car Car
SLIDE 6
Formal definition of group action
6
- Let % ≤ '(
- ) ∈ % acts on a vector + ∈ ℝ( by permuting its coordinates:
)+ = (+/01 2 , … , +/01 ( )
SLIDE 7
Formal definition of group action
7
- ! ∈ # acts on a tensor X ∈ ℝ&' by permuting its coordinates in each dimension:
!( )*,…,)' = (./* 0 ,…,./* )'
SLIDE 8
Formal definition of group action
8
- ! ∈ # acts on a tensor X ∈ ℝ&' by permuting its coordinates in each dimension:
!( )*,…,)' = (./* )* ,…,./* )'
SLIDE 9
Invariant and equivariant functions
Definition: A function !: ℝ$ → ℝ is invariant with respect to a group & if: ! '( = ! ( , ∀' ∈ &
9
SLIDE 10
Invariant and equivariant functions
Definition: A function !: ℝ$ → ℝ is invariant with respect to a group & if: ! '( = ! ( , ∀' ∈ & Definition: A function !: ℝ$ → ℝ$ is equivariant with respect to a group & if: ! '( = '! ( , ∀' ∈ &
10
SLIDE 11
!-invariant networks
11
Linear equivariant layers Linear invariant + MLP
SLIDE 12
!-invariant networks
12
Linear equivariant layers
" ∈ ℝ% " ∈ ℝ%& " ∈ ℝ%' " ∈ ℝ%' " ∈ ℝ
Linear invariant + MLP
SLIDE 13
Main question: How expressive are !-invariant networks?
13
SLIDE 14
How expressive are !-invariant networks?
14
Continuous Functions
(approximable with FC networks)
SLIDE 15
How expressive are !-invariant networks?
15
Continuous Functions
(approximable with FC networks)
Continuous !- Invariant functions
SLIDE 16
How expressive are !-invariant networks?
16
!-Invariant networks Continuous Functions
(approximable with FC networks)
Continuous !- Invariant functions
SLIDE 17
How expressive are !-invariant networks?
17
!-Invariant networks Gap? Continuous Functions
(approximable with FC networks)
Continuous !- Invariant functions
SLIDE 18
Theoretical results
18
SLIDE 19
Universality of high-order networks
Theorem 1. !-invariant networks are universal.
19
Tensor order might be as high as " 2
SLIDE 20
Lower bound on network order
Theorem 2. There exists groups ! ≤ #$ for which the tensor order should be at least %(') in order to achieve universality
20
Tensor order must be at least
$)* *
SLIDE 21
Necessary condition for first order networks
Theorem 3. Let ! ∈ #$. If first order !-invariant networks are universal, then | & '/)| < | & '/!| for any strict super-group ! < ) ≤ #$.
21
Tensor order is 1
SLIDE 22
The End
- Support
- ERC Grant (LiftMatch)
- Israel Science Foundation
- Thanks for listening!
23
“Invariant Graph Networks” by Yaron Lipman
Saturday 11am, Grand Ballroom B Learning and Reasoning with Graph-Structured Representations workshop