cheap orthogonal constraints in neural networks a simple
play

Cheap Orthogonal Constraints in Neural Networks: A Simple - PowerPoint PPT Presentation

Jul 30, 2023 •3.02k likes •3.28k views

Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group Mario Lezcano-Casado David Martnez-Rubio Mathematical Institute Department of Computer Science June 12, 2019 Cheap Orthogonal

  1. Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group Mario Lezcano-Casado David Martínez-Rubio Mathematical Institute Department of Computer Science June 12, 2019

  2. Cheap Orthogonal Constraints in Neural Networks Optimization with orthogonal constraints We study the optimization of neural networks with orthogonal constraints B ∈ R n × n , B ⊺ B = I Visit our poster (#27 on Wednesday) 1 4

  3. Cheap Orthogonal Constraints in Neural Networks Optimization with orthogonal constraints We study the optimization of neural networks with orthogonal constraints B ∈ R n × n , B ⊺ B = I Motivation: Visit our poster (#27 on Wednesday) 1 4

  4. Cheap Orthogonal Constraints in Neural Networks Optimization with orthogonal constraints We study the optimization of neural networks with orthogonal constraints B ∈ R n × n , B ⊺ B = I Motivation: ◮ Orthogonal matrices have eigenvalues with norm 1 . Visit our poster (#27 on Wednesday) 1 4

  5. Cheap Orthogonal Constraints in Neural Networks Optimization with orthogonal constraints We study the optimization of neural networks with orthogonal constraints B ∈ R n × n , B ⊺ B = I Motivation: ◮ Orthogonal matrices have eigenvalues with norm 1 . ◮ Convenient for exploding and vanishing gradient problems within RNN s. ◮ They constitute a implicit regularization method. Visit our poster (#27 on Wednesday) 1 4

  6. Cheap Orthogonal Constraints in Neural Networks Optimization with orthogonal constraints We study the optimization of neural networks with orthogonal constraints B ∈ R n × n , B ⊺ B = I Motivation: ◮ Orthogonal matrices have eigenvalues with norm 1 . ◮ Convenient for exploding and vanishing gradient problems within RNN s. ◮ They constitute a implicit regularization method. ◮ They are the basic building block for matrix factorizations like SVD or QR . Visit our poster (#27 on Wednesday) 1 4

  7. Cheap Orthogonal Constraints in Neural Networks Optimization with orthogonal constraints We study the optimization of neural networks with orthogonal constraints B ∈ R n × n , B ⊺ B = I Motivation: ◮ Orthogonal matrices have eigenvalues with norm 1 . ◮ Convenient for exploding and vanishing gradient problems within RNN s. ◮ They constitute a implicit regularization method. ◮ They are the basic building block for matrix factorizations like SVD or QR . ◮ They allow for the implementation of factorized linear layers. Visit our poster (#27 on Wednesday) 1 4

  8. Cheap Orthogonal Constraints in Neural Networks Optimization with orthogonal constraints B ∈ SO( n ) f ( B ) min is equivalent to solving A ∈ Skew( n ) f (exp ( A )) min Visit our poster (#27 on Wednesday) 2 4

  9. Cheap Orthogonal Constraints in Neural Networks Optimization with orthogonal constraints B ∈ SO( n ) f ( B ) min is equivalent to solving A ∈ Skew( n ) f (exp ( A )) min � �� � � �� � constrained problem. unconstrained problem. Visit our poster (#27 on Wednesday) 2 4

  10. Cheap Orthogonal Constraints in Neural Networks Optimization with orthogonal constraints B ∈ SO( n ) f ( B ) min is equivalent to solving A ∈ Skew( n ) f (exp ( A )) min � �� � � �� � constrained problem. unconstrained problem. ◮ The matrix exponential maps skew-symmetric matrices to orthogonal matrices . Visit our poster (#27 on Wednesday) 2 4

  11. Cheap Orthogonal Constraints in Neural Networks Optimization with orthogonal constraints B ∈ SO( n ) f ( B ) min is equivalent to solving A ∈ Skew( n ) f (exp ( A )) min � �� � � �� � constrained problem. unconstrained problem. ◮ The matrix exponential maps skew-symmetric matrices to orthogonal matrices . ◮ Compute the exponential to optimize over the unconstrained space of skew symmetric matrices. Visit our poster (#27 on Wednesday) 2 4

  12. Cheap Orthogonal Constraints in Neural Networks Optimization with orthogonal constraints B ∈ SO( n ) f ( B ) min is equivalent to solving A ∈ Skew( n ) f (exp ( A )) min � �� � � �� � constrained problem. unconstrained problem. ◮ The matrix exponential maps skew-symmetric matrices to orthogonal matrices . ◮ Compute the exponential to optimize over the unconstrained space of skew symmetric matrices. ◮ No orthogonality needs to be enforced. Visit our poster (#27 on Wednesday) 2 4

  13. Cheap Orthogonal Constraints in Neural Networks Optimization with orthogonal constraints B ∈ SO( n ) f ( B ) min is equivalent to solving A ∈ Skew( n ) f (exp ( A )) min � �� � � �� � constrained problem. unconstrained problem. ◮ The matrix exponential maps skew-symmetric matrices to orthogonal matrices . ◮ Compute the exponential to optimize over the unconstrained space of skew symmetric matrices. ◮ No orthogonality needs to be enforced. ◮ It has negligible overhead in your neural network. Visit our poster (#27 on Wednesday) 2 4

  14. Cheap Orthogonal Constraints in Neural Networks Optimization with orthogonal constraints B ∈ SO( n ) f ( B ) min is equivalent to solving A ∈ Skew( n ) f (exp ( A )) min � �� � � �� � constrained problem. unconstrained problem. ◮ The matrix exponential maps skew-symmetric matrices to orthogonal matrices . ◮ Compute the exponential to optimize over the unconstrained space of skew symmetric matrices. ◮ No orthogonality needs to be enforced. ◮ It has negligible overhead in your neural network. ◮ General purpose optimizers can be used ( SGD , ADAM , ADAGRAD , . . . ). Visit our poster (#27 on Wednesday) 2 4

  15. Cheap Orthogonal Constraints in Neural Networks Optimization with orthogonal constraints B ∈ SO( n ) f ( B ) min is equivalent to solving A ∈ Skew( n ) f (exp ( A )) min � �� � � �� � constrained problem. unconstrained problem. ◮ The matrix exponential maps skew-symmetric matrices to orthogonal matrices . ◮ Compute the exponential to optimize over the unconstrained space of skew symmetric matrices. ◮ No orthogonality needs to be enforced. ◮ It has negligible overhead in your neural network. ◮ General purpose optimizers can be used ( SGD , ADAM , ADAGRAD , . . . ). ◮ No new extremal points are created in the main parametrization region. Visit our poster (#27 on Wednesday) 2 4

  16. Cheap Orthogonal Constraints in Neural Networks 0 . 020 Baseline EURNN LSTM scoRNN expRNN 0 . 015 Cross entropy 0 . 010 0 . 005 0 . 000 0 500 1000 1500 2000 2500 3000 3500 4000 Iterations Cross entropy in the copying problem for L = 2000 . The copying problem uses synthetic data of the form: Random numbers Wait for L steps Recall Input: 14221 ------ :---- Output: ----- ------ 14221 Visit our poster (#27 on Wednesday) 3 4

  17. Cheap Orthogonal Constraints in Neural Networks M ODEL # PARAM V ALID . T EST N 224 ≈ 83 K 5 . 34 5 . 30 EXPRNN 322 ≈ 135 K 4 . 42 4 . 38 EXPRNN 425 ≈ 200 K 5 . 52 5 . 48 EXPRNN 224 ≈ 83 K 9 . 26 8 . 50 SCORNN 322 ≈ 135 K 8 . 48 7 . 82 SCORNN 425 ≈ 200 K 7 . 97 7 . 36 SCORNN ≈ 83 K 15 . 42 14 . 30 LSTM 84 120 ≈ 135 K 13 . 93 12 . 95 LSTM 158 ≈ 200 K 13 . 66 12 . 62 LSTM 158 ≈ 83 K 15 . 57 18 . 51 EURNN ≈ 135 K 15 . 90 15 . 31 EURNN 256 378 ≈ 200 K 16 . 00 15 . 15 EURNN RGD 128 ≈ 83 K 15 . 07 14 . 58 192 ≈ 135 K 15 . 10 14 . 50 RGD ≈ 200 K 14 . 96 14 . 69 RGD 256 RNN s trained on a speech prediction task on the TIMIT dataset. It shows the best validation MSE accuracy. Visit our poster (#27 on Wednesday) 4 4

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Embedding Hard Physical Constraints in Convolutional Neural Networks for 3D Turbulence Dr.

Embedding Hard Physical Constraints in Convolutional Neural Networks for 3D Turbulence Dr.

Embedding Hard Physical Constraints in Convolutional Neural Networks for 3D Turbulence Dr. Arvind T. Mohan Postdoctoral Researcher Center for Nonlinear Studies Computational Physics & Methods Group Los Alamos National Laboratory, New

244 views • 13 slides
Neural Networks with Cheap Differential Operators Ricky T. Q. Chen, David Duvenaud Differential

Neural Networks with Cheap Differential Operators Ricky T. Q. Chen, David Duvenaud Differential

Neural Networks with Cheap Differential Operators Ricky T. Q. Chen, David Duvenaud Differential Operators Want to compute operators such as divergence : d f i ( x ) f = f : d d x i is a neural net. where i

330 views • 11 slides
More Data Mining with Weka Class 5 Lesson 1 Simple neural networks Ian H. Witten Department

More Data Mining with Weka Class 5 Lesson 1 Simple neural networks Ian H. Witten Department

More Data Mining with Weka Class 5 Lesson 1 Simple neural networks Ian H. Witten Department of Computer Science University of Waikato New Zealand weka.waikato.ac.nz Lesson 5.1: Simple neural networks Class 1 Exploring Wekas

698 views • 48 slides
Training Neural Networks CMSC 470 Marine Carpuat Neural Networks so far Powerful non-linear

Training Neural Networks CMSC 470 Marine Carpuat Neural Networks so far Powerful non-linear

Training Neural Networks CMSC 470 Marine Carpuat Neural Networks so far Powerful non-linear models for classification Predictions are made as a sequence of simple operations matrix-vector operations non-linear activation functions

417 views • 14 slides
Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

1.17k views • 91 slides
Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

852 views • 33 slides
CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides
Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.63k views • 119 slides
Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.57k views • 124 slides
Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography Hopfield, J. J., "Neural networks and physical systems with emergent collective computational abilities," Proceedings of the National Academy

367 views • 19 slides
8 Neural MT 2: Attentional Neural MT In the past chapter, we described a simple model for neural

8 Neural MT 2: Attentional Neural MT In the past chapter, we described a simple model for neural

8 Neural MT 2: Attentional Neural MT In the past chapter, we described a simple model for neural machine translation, which uses an encoder to encode sentences as a fixed-length vector. However, in some ways, this view is overly simplified, and

635 views • 9 slides
Lecture 20: Neural Networks for NLP Zubin Pahuja zpahuja2@illinois.edu

Lecture 20: Neural Networks for NLP Zubin Pahuja zpahuja2@illinois.edu

Lecture 20: Neural Networks for NLP Zubin Pahuja zpahuja2@illinois.edu courses.engr.illinois.edu/cs447 CS447: Natural Language Processing 1 Todays Lecture Feed-forward neural networks as classifiers simple architecture in which

724 views • 37 slides
Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides
(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning objectives What are neural networks? What are deep neural networks? How do we train neural networks? What variants of neural network

331 views • 31 slides
How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . .

How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . .

Need for Machine . . . Neural Networks . . . How to Speed Up . . . Neural Networks: A . . . How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . . . Resulting Algorithm Faster by Taking Into How to

464 views • 17 slides
Quiz Let W be a vector space, and let U be a subspace of W . Define the orthogonal complement

Quiz Let W be a vector space, and let U be a subspace of W . Define the orthogonal complement

Quiz Let W be a vector space, and let U be a subspace of W . Define the orthogonal complement of U in W . Let W = R 3 . Let U = Span { [1 , 2 , 0] } . Give a basis for the orthogonal complement of U in W . Give a matrix A such that the

528 views • 17 slides
Final exam: 68:50pm in Clough 152 Cumulative final covers the whole class pretty evenly.

Final exam: 68:50pm in Clough 152 Cumulative final covers the whole class pretty evenly.

Announcements Monday, December 04 Final exam: 68:50pm in Clough 152 Cumulative final covers the whole class pretty evenly. About twice as long as a midterm. Common for all 1553 sections; written collaboratively. Studying

370 views • 14 slides
Lecture 3.1: Fourier series and orthogonality Matthew Macauley Department of Mathematical

Lecture 3.1: Fourier series and orthogonality Matthew Macauley Department of Mathematical

Lecture 3.1: Fourier series and orthogonality Matthew Macauley Department of Mathematical Sciences Clemson University http://www.math.clemson.edu/~macaule/ Math 4340, Advanced Engineering Mathematics M. Macauley (Clemson) Lecture 3.1: Fourier

521 views • 9 slides
Dirichlet problems for Ornstein-Uhlenbeck operators in subsets of Hilbert spaces Alessandra

Dirichlet problems for Ornstein-Uhlenbeck operators in subsets of Hilbert spaces Alessandra

Presentation Weak solutions Open problems Dirichlet problems for Ornstein-Uhlenbeck operators in subsets of Hilbert spaces Alessandra Lunardi in collaboration with G. Da Prato Modena, Sept. 9th, 2010 Alessandra Lunardi in collaboration with

1.28k views • 98 slides
Relations among partitions. I: Partitions of a finite set R. A. Bailey University of St Andrews

Relations among partitions. I: Partitions of a finite set R. A. Bailey University of St Andrews

Relations among partitions. I: Partitions of a finite set R. A. Bailey University of St Andrews Combinatorics Seminar, Shanghai Jiao Tong University, November 2017 Bailey Relations among partitions 1/24 Abstract First I consider a single

1.16k views • 76 slides
Inner Product Spaces and Orthogonality Mongi BLEL King Saud University August 30, 2019 Mongi

Inner Product Spaces and Orthogonality Mongi BLEL King Saud University August 30, 2019 Mongi

Inner Product Spaces and Orthogonality Mongi BLEL King Saud University August 30, 2019 Mongi BLEL Inner Product Spaces and Orthogonality Table of contents Mongi BLEL Inner Product Spaces and Orthogonality Inner Product Definition Let V be

1.15k views • 35 slides
Sets of Orthogonal Hypercubes Gary L. Mullen Penn State University mullen@math.psu.edu Dec.

Sets of Orthogonal Hypercubes Gary L. Mullen Penn State University mullen@math.psu.edu Dec.

Sets of Orthogonal Hypercubes Gary L. Mullen Penn State University mullen@math.psu.edu Dec. 2013 Gary L. Mullen (PSU) Sets of Orthogonal Hypercubes Dec. 2013 1 / 27 Latin Squares A latin square (LS) of order n is an n n array based on n

798 views • 50 slides
Asymptotics of orthogonal polynomials in normal matrix ensemble Seung-Yeop Lee (University of

Asymptotics of orthogonal polynomials in normal matrix ensemble Seung-Yeop Lee (University of

Asymptotics of orthogonal polynomials in normal matrix ensemble Seung-Yeop Lee (University of South Florida) Cincinnati, September 20th 2014 1 / 32 Joint work with Roman Riser. Many discussions with Marco Bertola, Robert Buckingham, Maurice

918 views • 32 slides

More recommend