Entropy and mutual information in models of deep neural networks - PowerPoint PPT Presentation

Nov 15, 2022 •799 likes •1.16k views

Entropy and mutual information in models of deep neural networks NeurIPS 2018 - Thursday Dec 06th - Spotlight Poster @ Room 210 & 230 AB #110 Marylou Gabri (LPS ENS), Andre Manoel (INRIA Saclay, Owkin), Clment Luneau, Jean Barbier,

Entropy and mutual information in models of deep neural networks NeurIPS 2018 - Thursday Dec 06th - Spotlight Poster @ Room 210 & 230 AB #110 Marylou Gabrié (LPS ENS), Andre Manoel (INRIA Saclay, Owkin), Clément Luneau, Jean Barbier, Nicolas Macris (EPFL), Lenka Zdeborová (CEA Saclay), Florent Krzakala (LPS ENS)
Motivations Information theoretic arguments to deep learning Theory of generalization: Shwartz-Ziv et al. 2017, Saxe et al. 2018, Goldfeld et al. 2018 etc. • • New regularizer: Chalk et al. 2016, Alemi et al. 2017, Kolchinsky et al. 2017, Belghazi et al. 2017, Zhao et al. 2018, Achille et al. 2018, Hjelm et al. 2018, etc. Computing mutual informations in neural networks … is tricky • Intractable Computing or yet more involved • Sampled based estimations (non-parametric, variational methods ) Kraskov et al. 2014, Kolchinsky et al. 2017, Belghazi et al. 2017, Goldfeld et al. 2018 etc. Less and less reliable as networks get large è Need for benchmark controlled case è Here 1) Restrict to models of DNNS, 2) Leveverage statistical physics replica method

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Chapter 2 Entropy, Relative Entropy, and Mutual Infor- mation Peng-Hua Wang Graduate Institute

Chapter 2 Entropy, Relative Entropy, and Mutual Infor- mation Peng-Hua Wang Graduate Institute of Communication Engineering National Taipei University Chapter Outline Chap. 2 Entropy, Relative Entropy, and Mutual Information 2.1 Entropy 2.2

752 views • 51 slides

Information Theory Lecture 1 Course introduction Entropy, relative entropy and mutual

Information Theory Lecture 1 Course introduction Entropy, relative entropy and mutual information: Cover & Thomas (CT) 2.15 Important inequalities: CT2.68, 2.10 Mikael Skoglund, Information Theory 1/26 Information Theory

160 views • 13 slides

Infotheory for Statistics and Learning Lecture 1 Entropy Relative entropy Mutual

Infotheory for Statistics and Learning Lecture 1 Entropy Relative entropy Mutual information f -divergence Mikael Skoglund 1/16 Entropy Over ( R , B ) , consider a discrete RV X with all probability in a countable set X B ,

538 views • 8 slides

Entropy and Shannon information Entropy and Shannon information For a random variable X with

Entropy and Shannon information Entropy and Shannon information For a random variable X with distribution p(x), the entropy is H[ X ] = - S x p( x ) log 2 p( x ) Information is defined as I[ X ] = - log 2 p( x ) Mutual information Typically,

695 views • 47 slides

Towards a Deep and Unified Understanding of Deep Neural Models in NLP Chaoyu Guan* 2 , Xiting

Towards a Deep and Unified Understanding of Deep Neural Models in NLP Chaoyu Guan* 2 , Xiting Wang* 2 , Quanshi Zhang 1 , Runjin Chen 1 , Di He 2 , Xing Xie 2 * Equal Contribution 1 John Hopcroft Center and the MoE Key Lab of Artificial

219 views • 11 slides

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information Engineering The Chinese University of Hong Kong Deep Neural Networks are Everywhere Playing Go Making Medical Decision Understanding Scenes Deep Neural

1.1k views • 46 slides

Mutual information deep regularization for semi- supervised segmentation J. Peng, M. Pedersoli,

MIDL 2020 presentation Mutual information deep regularization for semi- supervised segmentation J. Peng, M. Pedersoli, C. Desrosiers Outline We proposed a semi-supervised segmentation method for medical image regularized by Mutual Information

256 views • 13 slides

Eff Efficient Entr icient Entropy opy Est Estimation imation for for Mutua Mutual l

Efficient entropy estimation for MIA using B-splines Eff Efficient Entr icient Entropy opy Est Estimation imation for for Mutua Mutual l Information Information Analys Analysis is using using B-splines splines Alexandre VENELLI IML

871 views • 25 slides

ECEN 5022 Cryptography Introduction to Information Theory Peter Mathys University of Colorado

Entropy Mutual Information Perfect Cryptosystem ECEN 5022 Cryptography Introduction to Information Theory Peter Mathys University of Colorado Spring 2008 Peter Mathys ECEN 5022 Cryptography Entropy Entropy Mutual Information Binary

484 views • 12 slides

(part 1) CMSC 473/673 UMBC Some slides adapted from 3SLP Outline Maximum Entropy models

Maxent and Neural Language Models (part 1) CMSC 473/673 UMBC Some slides adapted from 3SLP Outline Maximum Entropy models Defining the model Defining the objective Learning: Optimizing the objective Math: gradient derivation Neural

1.9k views • 133 slides

NEURAL CONVERSATIONAL MODELS WITH ENTROPY-BASED DATA FILTERING Richard Csaky 1 , Patrik Purgai 1

IMPROVING NEURAL CONVERSATIONAL MODELS WITH ENTROPY-BASED DATA FILTERING Richard Csaky 1 , Patrik Purgai 1 , Gabor Recski 1,2 1 Budapest University of Technology 2 Sclable AI Introduction Takeaways Better responses by filtering training

410 views • 13 slides

Deep Neural Networks and Hidden Markov Models in i-vector-based Text-Dependent Speaker

Introduction HMM based method Deep Neural Networks (DNNs) Experiments and Results Conclusions Deep Neural Networks and Hidden Markov Models in i-vector-based Text-Dependent Speaker Verification Hossein Zeinali 1 , 2 , Luk s Burget 2 ,

301 views • 11 slides

Maxent Models (III), & Neural Language Models CMSC 473/673 UMBC September 25 th , 2017 Some

Maxent Models (III), & Neural Language Models CMSC 473/673 UMBC September 25 th , 2017 Some slides adapted from 3SLP Recap from last time Maximum Entropy Models a more general language model argmax ) ()

1.46k views • 66 slides

Overview Coding and Information Theory What is information theory? Entropy Coding Chris

Overview Coding and Information Theory What is information theory? Entropy Coding Chris Williams Rate-distortion theory School of Informatics, University of Edinburgh Mutual information Channel capacity November 2007 Reading: Bishop 1.6

389 views • 5 slides

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan Zelener Machine Learning Reading Group January 7 th 2016 The Graduate Center, CUNY Objectives Explain some of the trends of deep learning and

680 views • 44 slides

Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications

Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications Neurons & Nets dimer Learning & Depth G1E model Feature extraction Theory 1 Layer Nets Neural computation

1.32k views • 45 slides

Formally Certifying the Security of Digital Signature Schemes Santiago Zanella 1 , 2 Benjamin

Formally Certifying the Security of Digital Signature Schemes Santiago Zanella 1 , 2 Benjamin Grgoire 1 , 2 Gilles Barthe 3 Federico Olmedo 3 1 Microsoft Research - INRIA Joint Centre, France 2 INRIA Sophia Antipolis - Mditerrane, France 3

837 views • 51 slides

Spontaneous false lumen thrombosis (4%) 2 4/6/2017 Intramural Hematoma (IMH) vs TBAD Intramural

4/6/2017 Intramural Hematoma (IMH) Disclosures Should we treat this aggressively? Speaker / consultant W.L. Gore Endologix Medtronic William J. Quinones-Baldrich MD Professor of Surgery Director UCLA Aortic Center UCLA Medical Center

318 views • 5 slides

T oo Many Knobs to Tune? T owards Faster Database Tuning by Pre-selecting Important Knobs

T oo Many Knobs to Tune? T owards Faster Database Tuning by Pre-selecting Important Knobs Konstantinos Kanellis, Ramnatthan Alagappan, Shivaram Venkataraman Database tuning is important! Realizing high performance requires finding optimal

560 views • 22 slides

Monte Carlo algorithms for Bayesian social network models Alberto Caimo alberto.caimo@usi.ch

Monte Carlo algorithms for Bayesian social network models Alberto Caimo alberto.caimo@usi.ch Social Network Analysis Research Centre InterDisciplinary Institute of Data Science University of Lugano, Switzerland AMS Sectional Meeting, Loyola

471 views • 30 slides

VCMC: Variational Consensus Monte Carlo Maxim Rabinovich, Elaine Angelino, Michael I. Jordan

VCMC: Variational Consensus Monte Carlo Maxim Rabinovich, Elaine Angelino, Michael I. Jordan Berkeley Vision and Learning Center September 22, 2015 probabilistic models! sky fog bridge water grass object tracking & recognition

624 views • 45 slides

Correlated Variational Auto-Encoders Da Tang 1 Dawen Liang 2 Tony Jebara 1 , 2 Nicholas Ruozzi 3 1

Correlated Variational Auto-Encoders Da Tang 1 Dawen Liang 2 Tony Jebara 1 , 2 Nicholas Ruozzi 3 1 Columbia University 2 Netflix Inc. 3 The University of Texas at Dallas June 11, 2019 Variational Auto-Encoders (VAEs) Learn stochastic low

332 views • 19 slides

The Discovery of Asymptotic Freedom & The Emergence of QCD David Gross Nobel Lecture

The Discovery of Asymptotic Freedom & The Emergence of QCD David Gross Nobel Lecture December 8, 2004 The Weak and the Strong The forces operating in the nucleus are of two kinds: WEAK INTERACTIONS STRONG INTERACTIONS Responsible for

748 views • 37 slides

Variational Autoencoders Tom Fletcher March 25, 2019 Talking about this paper: Diederik Kingma

Variational Autoencoders Tom Fletcher March 25, 2019 Talking about this paper: Diederik Kingma and Max Welling, Auto-Encoding Variational Bayes, In International Conference on Learning Representation (ICLR) , 2014. Autoencoders Input Latent

409 views • 15 slides