Approximating Orthogonal Matrices with Effective Givens - PowerPoint PPT Presentation

Approximating Orthogonal Matrices with Effective Givens Factorization Thomas Frerix Technical University of Munich joint work with Joan Bruna (NYU) Poster #164

Givens Factorization of Orthogonal Matrices   1 ··· 0 0 ··· 0 ··· . . . . . ... . . . . . . .    0 ··· cos( α ) ··· − sin( α ) ··· 0    . . . . G T ( i , j , α ) = ...  . . . .  . . . .   0 ··· sin( α ) ··· cos( α ) ··· 0     . . . ... . . . . .   . . . . 0 ··· 0 ··· 0 ··· 1

Givens Factorization of Orthogonal Matrices   1 ··· 0 0 ··· 0 ··· . . . . . ... . . . . . . .    0 ··· cos( α ) ··· − sin( α ) ··· 0    . . . . G T ( i , j , α ) = ...  . . . .  . . . .   0 ··· sin( α ) ··· cos( α ) ··· 0     . . . ... . . . . .   . . . . 0 ··· 0 ··· 0 ··· 1 Exact Givens Factorization N = d ( d − 1) U = G 1 . . . G N 2

Approximate Givens Factorization Approximate Givens Factorization N ≪ d ( d − 1) U ≈ G 1 . . . G N 2 computationally hard problem

Approximate Givens Factorization Approximate Givens Factorization N ≪ d ( d − 1) U ≈ G 1 . . . G N 2 computationally hard problem Our Questions in this Context 1. Which orthogonal matrices can be effectively approximated? (not all of them)

Approximate Givens Factorization Approximate Givens Factorization N ≪ d ( d − 1) U ≈ G 1 . . . G N 2 computationally hard problem Our Questions in this Context 1. Which orthogonal matrices can be effectively approximated? (not all of them) 2. Which principles are behind effective approximation algorithms? (sparsity-inducing algorithms)

Motivation: Unitary Basis Transform / FFT Advantageous Setting Once computed, applied many times

Motivation: Unitary Basis Transform / FFT Advantageous Setting Once computed, applied many times Unitary Basis Transform d 2 � � � � FFT: O → O d log( d )

Motivation: Unitary Basis Transform / FFT Advantageous Setting Once computed, applied many times Unitary Basis Transform d 2 � � � � FFT: O → O d log( d ) Application: Graph Fourier Transform

Which Matrices can be Effectively Approximated? Theorem � d 2 / log( d ) � Let ǫ > 0 . If N = o , then as d → ∞ ,  � � � � �  → 0 , µ U ∈ U ( d ) inf � U − G n � 2 ≤ ǫ  � G 1 ... G N � n where µ is the Haar measure over U ( d ) .

Which Matrices can be Effectively Approximated? Theorem � d 2 / log( d ) � Let ǫ > 0 . If N = o , then as d → ∞ ,  � � � � �  → 0 , µ U ∈ U ( d ) inf � U − G n � 2 ≤ ǫ  � G 1 ... G N � n where µ is the Haar measure over U ( d ) . • proof is based on an ǫ -covering argument • suggests computational-to-statistical gap together with experimental results (details at poster)

K -planted Distribution over SO ( d ) Sample U = G 1 . . . G K • choose subspace ( i k , j k ) uniformly with replacement • choose rotation angle α k ∈ [0 , 2 π ) uniformly

K -planted Distribution over SO ( d ) Sample U = G 1 . . . G K • choose subspace ( i k , j k ) uniformly with replacement • choose rotation angle α k ∈ [0 , 2 π ) uniformly 1 0 . 8 0 . 6 || U || 0 / d 2 K -planted matrices 0 . 4 quickly become dense 0 . 2 0 0 0 . 2 0 . 4 0 . 6 0 . 8 1 K / d log 2 ( d ) 256 512 1024

Minimizing Sparsity-Inducing Norms over O ( d ) ˆ G T N . . . G T N U ≈ I U = G 1 . . . G N

Minimizing Sparsity-Inducing Norms over O ( d ) ˆ G T N . . . G T N U ≈ I U = G 1 . . . G N Approximation criterion � � � � � U − ˆ � U − ˆ U F , sym := min UP � � � � � � P ∈P d F

Minimizing Sparsity-Inducing Norms over O ( d ) ˆ G T N . . . G T N U ≈ I U = G 1 . . . G N Approximation criterion � � � � � U − ˆ � U − ˆ U F , sym := min UP � � � � � � P ∈P d F Better functions to be minimized greedily? d f ( U ) := d − 1 � U � 1 = d − 1 � � � � U ij � i , j =1

Minimizing Sparsity-Inducing Norms over O ( d ) ˆ G T N . . . G T N U ≈ I U = G 1 . . . G N Approximation criterion � � � � � U − ˆ � U − ˆ U F , sym := min UP � � � � � � P ∈P d F Better functions to be minimized greedily? d f ( U ) := d − 1 � U � 1 = d − 1 � � � � U ij � i , j =1 • Non-convex greedy step • global optimum in O ( d 2 ) amortized time complexity

Thank you Poster #164 https://github.com/tfrerix/givens-factorization