On the Universality of Invariant Networks Haggai Maron Ethan - PowerPoint PPT Presentation

1 On the Universality of Invariant Networks Haggai Maron Ethan Fetaya Nimrod Segol Yaron Lipman

2 Invariant tasks • Image classification Car Car

3 Invariant tasks • Image classification Car Car • Graph/ hyper-graph classification ! " ! ' ! % ! $ ! & ! # ! ' ! " ! " ! ' ! % ! $ ! & ! # ! " ! " ! " ! ' ! ' ! % ! % ! ' ! % ! % ! $ ! $ ! $ ! $ ! & ! & ! & ! & ! # ! # ! # ! #

4 Invariant tasks • Image classification Car Car • Graph/ hyper-graph classification ! " ! ' ! % ! $ ! & ! # ! ' ! " ! " ! ' ! % ! $ ! & ! # ! " ! " ! " ! ' ! ' ! % ! % ! ' ! % ! % ! $ ! $ ! $ ! $ ! & ! & ! & ! & ! # ! # ! # ! # • Point-cloud / set classification 3 ($ % , ' % , ( % ) " • and many more… ($ * , ' * , ( * )

5 Goal of this paper • Invariant neural networks are a common approach for these tasks • This paper analyzes the expressive power of invariant models 3 ($ % , ' % , ( % ) " ($ * , ' * , ( * ) ! " ! ' ! % ! $ ! & ! # ! ' ! " ! " ! ' ! % ! $ ! & ! # ! " ! " ! " ! ' ! ' ! % ! % ! ' ! % ! % ! $ ! $ ! $ ! $ ! & ! & ! & ! & ! # ! # ! # ! # Car Car

6 Formal definition of group action • Let % ≤ ' ( • ) ∈ % acts on a vector + ∈ ℝ ( by permuting its coordinates: )+ = (+ / 01 2 , … , + / 01 ( )

7 Formal definition of group action • ! ∈ # acts on a tensor X ∈ ℝ & ' by permuting its coordinates in each dimension: !( ) * ,…,) ' = ( . /* 0 ,…,. /* ) '

8 Formal definition of group action • ! ∈ # acts on a tensor X ∈ ℝ & ' by permuting its coordinates in each dimension: !( ) * ,…,) ' = ( . /* ) * ,…,. /* ) '

9 Invariant and equivariant functions Definition: A function !: ℝ $ → ℝ is invariant with respect to a group & if: ! '( = ! ( , ∀' ∈ &

10 Invariant and equivariant functions Definition: A function !: ℝ $ → ℝ is invariant with respect to a group & if: ! '( = ! ( , ∀' ∈ & Definition: A function !: ℝ $ → ℝ $ is equivariant with respect to a group & if: ! '( = '! ( , ∀' ∈ &

11 ! -invariant networks Linear invariant + MLP Linear equivariant layers

12 ! -invariant networks " ∈ ℝ % ' " ∈ ℝ % ' " ∈ ℝ % " ∈ ℝ % & " ∈ ℝ Linear invariant + MLP Linear equivariant layers

13 Main question: How expressive are ! -invariant networks?

14 How expressive are ! -invariant networks? Continuous Functions (approximable with FC networks)

15 How expressive are ! -invariant networks? Continuous Functions Continuous (approximable ! - Invariant with FC networks) functions

16 How expressive are ! -invariant networks? Continuous Functions Continuous ! -Invariant (approximable ! - Invariant networks with FC networks) functions

17 How expressive are ! -invariant networks? Gap? Continuous Functions Continuous ! -Invariant (approximable ! - Invariant networks with FC networks) functions

18 Theoretical results

19 Universality of high-order networks Theorem 1 . ! -invariant networks are universal. Tensor order might be as high as " 2

20 Lower bound on network order Theorem 2 . There exists groups ! ≤ # $ for which the tensor order should be at least %(') in order to achieve universality $)* Tensor order must be at least *

21 Necessary condition for first order networks Theorem 3 . Let ! ∈ # $ . If first order ! -invariant networks are universal, then | & ' /)| < | & ' /!| for any strict super-group ! < ) ≤ # $ . Tensor order is 1

23 The End “Invariant Graph Networks” • Support by Yaron Lipman • ERC Grant (LiftMatch) Saturday 11am, Grand Ballroom B • Israel Science Foundation Learning and Reasoning with Graph-Structured Representations workshop • Thanks for listening!