Lightweight Neural Networks from PCA & LDA Based Distilled Dense - PowerPoint PPT Presentation

Oct 25, 2023 •184 likes •243 views

Lightweight Neural Networks from PCA & LDA Based Distilled Dense Neural Networks ICIP 2020 MEA. Seddik 1 , 2 , , H. Essafi 1 , A. Benzine 1 , 3 , M. Tamaazousti 1 1 CEA List, France 2 CentraleSuplec, L2S, France 3 Sorbonne University,

Lightweight Neural Networks from PCA & LDA Based Distilled Dense Neural Networks ICIP 2020 MEA. Seddik 1 , 2 , ∗ , H. Essafi 1 , A. Benzine 1 , 3 , M. Tamaazousti 1 1 CEA List, France 2 CentraleSupélec, L2S, France 3 Sorbonne University, CNRS, France ∗ http://melaseddik.github.io/ August 21, 2020 1 / 5
/ 2/5 Abstract Context: ◮ Compression of dense neural networks with the teacher-student approach. Motivation: ◮ Build lightweight neural networks that can fit into edge and IoT devices with limited resources (memory and computation). Proposed methods: ◮ We proposed two methods which rely on dimension reduction techniques (PCA and LDA). ◮ The dimension reduction is applied at each layer of the teacher net and then mapped to the layers of the student net using a multi-task loss function. 2 / 5
/ 3/5 Setting Given a Teacher Network (TN) trained on a dataset D with loss L TN � h (0) = x ∈ R p 0 (TN) : � W ( ℓ ) h ( ℓ − 1) + b ( ℓ ) � ∀ ℓ ∈ [ L ] h ( ℓ ) = f ℓ ∈ R p ℓ Construct a Student Network (SN) to train on D � h (0) = x ∈ R p 0 ˜ � b ( ℓ ) � (SN) : ∀ ℓ ∈ [ L ] ˜ h ( ℓ ) = f ℓ W ( ℓ )˜ ˜ h ( ℓ − 1) + ˜ ∈ R k ℓ Such that k ℓ ≪ p ℓ & Performance (SN) � Performance (TN) 3 / 5
/ 4/5 Proposed Methods (Net-PCAD & Net-LDAD) Given (TN) , a data matrix X and (TN) loss function L TN For each layer ℓ : 1. Extract the representations H ℓ of X from (TN) 2. Compute a projection matrix U ℓ ∈ R p ℓ × k ℓ through PCA or LDA on H ℓ Train (SN) as a multi-task 1 problem with L − 1 � ℓ h ( ℓ ) � � h ( ℓ ) , U ⊺ ˜ L SN = e − σ L TN + σ e − σ ℓ L mse + + σ ℓ � �� ℓ =1 Learning Task � �� (SN) Hidden Layers Task where σ and { σ ℓ } L − 1 ℓ =1 are learnable parameters. 1 Using the Homoscedastic loss function: A. Kendall et al. “Multitask learning using uncertainty to weigh losses for scene geometry and semantics” in Proceedings of IEEE CVPR, 2018. 4 / 5
/ 5/5 Experimental Setting & Results Layer (TN) (SN) Dense 1 p 0 × 1024 p 0 × k Dense 2 1024 × 512 k × k Dense 3 512 × 256 k × k Dense 4 256 × 10 k × 10 Table: Networks architectures. (SN) Datasets (TN) k = 50 100 200 MNIST 2 . 23s 0 . 38s 0 . 45s 0 . 65s 98% 97% 97 . 5% 97 . 8% FASHION 2 . 23s 0 . 38s 0 . 45s 0 . 65s 88% 87 . 5% 88 . 5% 88 . 5% CIFAR10 4 . 63s 0 . 75s 0 . 92s 1 . 35s 45% 50% 50 . 1% 50 . 3% Table: Networks performances. ⇒ k ℓ ≪ p ℓ & Performance (SN) � Performance (TN) 5 / 5

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides

Evolving Neural Networks This lecture is based on Xin Yaos tutorial slides From Evolving

Evolving Neural Networks This lecture is based on Xin Yaos tutorial slides From Evolving Single Neural Networks to Evolving Ensembles presented at CEC 2007. Literature: Evolving Artificial Neural Networks , Xin Yao, Proceedings of the

281 views • 16 slides

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

1.17k views • 91 slides

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

852 views • 33 slides

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.63k views • 119 slides

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.57k views • 124 slides

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography Hopfield, J. J., "Neural networks and physical systems with emergent collective computational abilities," Proceedings of the National Academy

367 views • 19 slides

Neural Networks for Time Series Prediction 15-486/782: Artificial Neural Networks Fall 2006

Neural Networks for Time Series Prediction 15-486/782: Artificial Neural Networks Fall 2006 (based on earlier slides by Dave Touretzky and Kornel Laskowski) What is a Time Series? A sequence of vectors (or scalars) which depend on time t . In

975 views • 42 slides

Population Based Training of Neural Networks M. Jaderberg, V. Dalibard, S. Osindero, W.M.

Presented by: Devin Taylor Population Based Training of Neural Networks M. Jaderberg, V. Dalibard, S. Osindero, W.M. Czarnecki November 14, 2018 DeepMind, London, United Kingdom Problem Statement Problem statement Neural networks suffer from

317 views • 17 slides

Neural Networks: Prediction (i.e. the forward pass) Machine Learning Based on slides and

Neural Networks: Prediction (i.e. the forward pass) Machine Learning Based on slides and material from Geoffrey Hinton, Richard Socher, Dan Roth, 1 Yoav Goldberg, Shai Shalev-Shwartz and Shai Ben-David, and others Neural Networks What is a

470 views • 17 slides

Deep Neural Networks and Hidden Markov Models in i-vector-based Text-Dependent Speaker

Introduction HMM based method Deep Neural Networks (DNNs) Experiments and Results Conclusions Deep Neural Networks and Hidden Markov Models in i-vector-based Text-Dependent Speaker Verification Hossein Zeinali 1 , 2 , Luk s Burget 2 ,

301 views • 11 slides

Neural Networks for Time Series Prediction 15-496/782: Artificial Neural Networks Kornel

Neural Networks for Time Series Prediction 15-496/782: Artificial Neural Networks Kornel Laskowski Spring 2004 (based on slides by D. Touretzky, Spring 2002) What is a Time Series? A sequence of vectors (or scalars) which depend on time t . In

719 views • 42 slides

Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The

Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The Problem of Long-Term Dependencies RNN short-term dependencies Language model trying to predict the next word based on the previous ones the clouds

729 views • 27 slides

Neural Networks as Stat Mech Systems Based on arXiv:1710.06570 [stat.ML], A

Neural Networks as Stat Mech Systems Based on arXiv:1710.06570 [stat.ML], A Correspondence Between Random Neural Networks and Statistical Field Theory Yoni Kahn, KICP/UIUC hep-ai 1/8/19 <latexit

462 views • 17 slides

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides

1 last time SIMD (single instruction multiple data) hardware idea: wider ALUs and registers

1 last time SIMD (single instruction multiple data) hardware idea: wider ALUs and registers Intels interface _mm sharing the CPU: context switching context = visible CPU state (registers, condition codes, PC, ) exceptions = OS gets run

1.42k views • 129 slides

CENG3420 Lecture 01: Introduction Bei Yu (Latest update: January 9, 2019) Spring 2019 1 / 50

CENG3420 Lecture 01: Introduction Bei Yu (Latest update: January 9, 2019) Spring 2019 1 / 50 Overview Course Information Background Organization First Glance Summary 2 / 50 Overview Course Information Background Organization

754 views • 60 slides

Introduction 2 A Modern Computer iPhone XS Computer Systems and Networks Spring 2019 3

Computer Systems and Networks ECPE 170 Jeff Shafer University of the Pacific Introduction 2 A Modern Computer iPhone XS Computer Systems and Networks Spring 2019 3 Applications Computer Systems and Networks Spring 2019 4

644 views • 53 slides

CSE141: Introduction to Computer Architecture Hung-Wei Tseng CSE141: Lets say something!

CSE141: Introduction to Computer Architecture Hung-Wei Tseng CSE141: Lets say something! Whats your name? Whats the most exciting thing you did so far this summer? Whats the most interesting computer science topic for you?

947 views • 68 slides

System Structuring with Threads Example: A Transcoding Web Proxy Appliance Proxy clients

System Structuring with Threads Example: A Transcoding Web Proxy Appliance Proxy clients Interposed between Web (HTTP) clients and servers. Masquerade as (represent) the server to the client. Masquerade as (represent) the client to the

617 views • 14 slides

FloWaveNet: A Generative Flow for Raw Audio Sungwon Kim 1 , Sang-gil Lee 1 , Jongyoon Song 1 ,

ICML 2019 FloWaveNet: A Generative Flow for Raw Audio Sungwon Kim 1 , Sang-gil Lee 1 , Jongyoon Song 1 , Jaehyeon Kim 2 , Sungron Yoon 1,3 1 Seoul National University, 2 Kakao Corporation, 3 ASRI, INMC, Institute of Engineering Research, Seoul

346 views • 16 slides

Form over Function Teaching Beginners How to Construct Programs Michael Sperber Collaborators:

Form over Function Teaching Beginners How to Construct Programs Michael Sperber Collaborators: Marcus Crestani, Martin Gasbichler, Herbert Klaeren, Eric Knauel @ University of Tbingen Wednesday, September 12, 12 Back at the Ranch ...

791 views • 58 slides

Another Dynamic Algorithm: Scoreboard Summary Tomasulo Algorithm Speedup 1.7 from compiler;

Another Dynamic Algorithm: Scoreboard Summary Tomasulo Algorithm Speedup 1.7 from compiler; 2.5 by hand For IBM 360/91 about 3 years after CDC 6600 BUT slow memory (no cache) limits benefit Goal: High Performance without special

303 views • 3 slides